2024-09-10T18:53:05.157 INFO:root:teuthology version: 1.2.2.dev2+g3752d38 2024-09-10T18:53:05.176 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-09-10T18:53:05.259 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/gabrioux-2024-09-10_17:53:37-orch:cephadm-wip-guits-quincy-2024-09-09-1810-quincy-distro-default-smithi/7900285 branch: wip-guits-quincy-2024-09-09-1810-quincy description: orch:cephadm/thrash/{0-distro/centos_9.stream_runc 1-start 2-thrash 3-tasks/radosbench fixed-2 msgr/async root} email: gabrioux@ibm.com first_in_suite: false job_id: '7900285' kernel: kdb: 1 sha1: distro last_in_suite: false machine_type: smithi name: gabrioux-2024-09-10_17:53:37-orch:cephadm-wip-guits-quincy-2024-09-09-1810-quincy-distro-default-smithi no_nested_subset: false openstack: - volumes: count: 4 size: 10 os_type: centos os_version: 9.stream overrides: admin_socket: branch: wip-guits-quincy-2024-09-09-1810-quincy ceph: conf: client.0: debug ms: 1 debug objecter: 20 debug rados: 20 global: ms type: async mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon min osdmap epochs: 50 mon osdmap full prune interval: 2 mon osdmap full prune min: 15 mon osdmap full prune txsize: 2 paxos service trim min: 10 osd: debug ms: 1 debug osd: 20 osd debug reject backfill probability: 0.3 osd delete sleep: 1 osd max backfills: 3 osd scrub max interval: 120 osd scrub min interval: 60 osd shutdown pgref assert: true osd snap trim sleep: 2 flavor: default log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - but it is still running - objects unfound and apparently lost - \(POOL_APP_NOT_ENABLED\) sha1: 5be57395ea235535910512ae73ba8e7ff25fe588 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} cephadm: cephadm_mode: root install: ceph: flavor: default sha1: 5be57395ea235535910512ae73ba8e7ff25fe588 selinux: whitelist: - scontext=system_u:system_r:logrotate_t:s0 workunit: branch: quincy-orch-centos-9-teuth sha1: fad5356a94e6044ce1062d748daa5a0f2ed53a7d owner: scheduled_gabrioux@teuthology priority: 70 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mon.c - mgr.y - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - ceph.rgw.foo.a - node-exporter.a - alertmanager.a - - mon.b - mgr.x - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - prometheus.a - grafana.a - node-exporter.b - ceph.iscsi.iscsi.a seed: 4625 sha1: 5be57395ea235535910512ae73ba8e7ff25fe588 sleep_before_teardown: 0 subset: 1/8 suite: orch:cephadm suite_branch: quincy-orch-centos-9-teuth suite_path: /home/teuthworker/src/github.com_adk3798_ceph_fad5356a94e6044ce1062d748daa5a0f2ed53a7d/qa suite_relpath: qa suite_repo: https://github.com/adk3798/ceph.git suite_sha1: fad5356a94e6044ce1062d748daa5a0f2ed53a7d targets: smithi183.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLzVHo8ucwdkq2iDwCf1fk/bUYmonpDVhJIrARXyxG5SOBl6C/YOpDBqprc+mo94kB7zcJjV5BdWghgWoRhYwQk= smithi191.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBMgCNZx6+Hjn7UWuYNd7IxjH0+QM/m83rMYXJWHo3pMUWCtp+B1HFpfjj4k53r0KYlS5yBZ1OpnfzSlUW+OPTMM= tasks: - pexec: all: - sudo dnf remove nvme-cli -y - sudo dnf install runc nvmetcli nvme-cli -y - sudo sed -i 's/^#runtime = "crun"/runtime = "runc"/g' /usr/share/containers/containers.conf - sudo sed -i 's/runtime = "crun"/#runtime = "crun"/g' /usr/share/containers/containers.conf - install: null - cephadm: conf: mgr: debug mgr: 20 debug ms: 1 - thrashosds: chance_pgnum_grow: 1 chance_pgnum_shrink: 1 chance_pgpnum_fix: 1 timeout: 1200 - full_sequential: - radosbench: clients: - client.0 time: 90 - radosbench: clients: - client.0 time: 90 - radosbench: clients: - client.0 time: 90 - radosbench: clients: - client.0 time: 90 - radosbench: clients: - client.0 time: 90 teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a timestamp: 2024-09-10_17:53:37 tube: smithi user: gabrioux verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.3401856 2024-09-10T18:53:05.260 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_adk3798_ceph_fad5356a94e6044ce1062d748daa5a0f2ed53a7d/qa; will attempt to use it 2024-09-10T18:53:05.260 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_adk3798_ceph_fad5356a94e6044ce1062d748daa5a0f2ed53a7d/qa/tasks 2024-09-10T18:53:05.260 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-09-10T18:53:05.262 INFO:teuthology.task.internal:Checking packages... 2024-09-10T18:53:05.282 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '5be57395ea235535910512ae73ba8e7ff25fe588' 2024-09-10T18:53:05.282 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-09-10T18:53:05.282 INFO:teuthology.packaging:ref: None 2024-09-10T18:53:05.282 INFO:teuthology.packaging:tag: None 2024-09-10T18:53:05.282 INFO:teuthology.packaging:branch: wip-guits-quincy-2024-09-09-1810-quincy 2024-09-10T18:53:05.282 INFO:teuthology.packaging:sha1: 5be57395ea235535910512ae73ba8e7ff25fe588 2024-09-10T18:53:05.282 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=wip-guits-quincy-2024-09-09-1810-quincy 2024-09-10T18:53:05.420 INFO:teuthology.task.internal:Found packages for ceph version 17.2.7-1606.g5be57395 2024-09-10T18:53:05.421 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-09-10T18:53:05.429 INFO:teuthology.task.internal:no buildpackages task found 2024-09-10T18:53:05.429 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-09-10T18:53:05.435 INFO:teuthology.task.internal:Saving configuration 2024-09-10T18:53:05.447 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-09-10T18:53:05.454 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-09-10T18:53:05.480 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi183.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/gabrioux-2024-09-10_17:53:37-orch:cephadm-wip-guits-quincy-2024-09-09-1810-quincy-distro-default-smithi/7900285', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-09-10 18:48:01.625190', 'locked_by': 'scheduled_gabrioux@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLzVHo8ucwdkq2iDwCf1fk/bUYmonpDVhJIrARXyxG5SOBl6C/YOpDBqprc+mo94kB7zcJjV5BdWghgWoRhYwQk='} 2024-09-10T18:53:05.500 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi191.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/gabrioux-2024-09-10_17:53:37-orch:cephadm-wip-guits-quincy-2024-09-09-1810-quincy-distro-default-smithi/7900285', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-09-10 18:48:01.627063', 'locked_by': 'scheduled_gabrioux@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBMgCNZx6+Hjn7UWuYNd7IxjH0+QM/m83rMYXJWHo3pMUWCtp+B1HFpfjj4k53r0KYlS5yBZ1OpnfzSlUW+OPTMM='} 2024-09-10T18:53:05.500 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-09-10T18:53:05.507 INFO:teuthology.task.internal:roles: ubuntu@smithi183.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'ceph.rgw.foo.a', 'node-exporter.a', 'alertmanager.a'] 2024-09-10T18:53:05.507 INFO:teuthology.task.internal:roles: ubuntu@smithi191.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'grafana.a', 'node-exporter.b', 'ceph.iscsi.iscsi.a'] 2024-09-10T18:53:05.507 INFO:teuthology.run_tasks:Running task console_log... 2024-09-10T18:53:05.569 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7fb7de6b7ac0>, signals=[15]) 2024-09-10T18:53:05.569 INFO:teuthology.run_tasks:Running task internal.connect... 2024-09-10T18:53:05.575 INFO:teuthology.task.internal:Opening connections... 2024-09-10T18:53:05.575 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi183.front.sepia.ceph.com 2024-09-10T18:53:05.576 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi183.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-10T18:53:05.653 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi191.front.sepia.ceph.com 2024-09-10T18:53:05.654 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi191.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-10T18:53:05.727 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-09-10T18:53:05.733 DEBUG:teuthology.orchestra.run.smithi183:> uname -m 2024-09-10T18:53:05.752 INFO:teuthology.orchestra.run.smithi183.stdout:x86_64 2024-09-10T18:53:05.752 DEBUG:teuthology.orchestra.run.smithi183:> cat /etc/os-release 2024-09-10T18:53:05.810 INFO:teuthology.orchestra.run.smithi183.stdout:NAME="CentOS Stream" 2024-09-10T18:53:05.811 INFO:teuthology.orchestra.run.smithi183.stdout:VERSION="9" 2024-09-10T18:53:05.811 INFO:teuthology.orchestra.run.smithi183.stdout:ID="centos" 2024-09-10T18:53:05.811 INFO:teuthology.orchestra.run.smithi183.stdout:ID_LIKE="rhel fedora" 2024-09-10T18:53:05.811 INFO:teuthology.orchestra.run.smithi183.stdout:VERSION_ID="9" 2024-09-10T18:53:05.811 INFO:teuthology.orchestra.run.smithi183.stdout:PLATFORM_ID="platform:el9" 2024-09-10T18:53:05.811 INFO:teuthology.orchestra.run.smithi183.stdout:PRETTY_NAME="CentOS Stream 9" 2024-09-10T18:53:05.811 INFO:teuthology.orchestra.run.smithi183.stdout:ANSI_COLOR="0;31" 2024-09-10T18:53:05.811 INFO:teuthology.orchestra.run.smithi183.stdout:LOGO="fedora-logo-icon" 2024-09-10T18:53:05.811 INFO:teuthology.orchestra.run.smithi183.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-09-10T18:53:05.811 INFO:teuthology.orchestra.run.smithi183.stdout:HOME_URL="https://centos.org/" 2024-09-10T18:53:05.811 INFO:teuthology.orchestra.run.smithi183.stdout:BUG_REPORT_URL="https://issues.redhat.com/" 2024-09-10T18:53:05.811 INFO:teuthology.orchestra.run.smithi183.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-09-10T18:53:05.812 INFO:teuthology.orchestra.run.smithi183.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-09-10T18:53:05.812 INFO:teuthology.lock.ops:Updating smithi183.front.sepia.ceph.com on lock server 2024-09-10T18:53:05.836 DEBUG:teuthology.orchestra.run.smithi191:> uname -m 2024-09-10T18:53:05.854 INFO:teuthology.orchestra.run.smithi191.stdout:x86_64 2024-09-10T18:53:05.854 DEBUG:teuthology.orchestra.run.smithi191:> cat /etc/os-release 2024-09-10T18:53:05.911 INFO:teuthology.orchestra.run.smithi191.stdout:NAME="CentOS Stream" 2024-09-10T18:53:05.911 INFO:teuthology.orchestra.run.smithi191.stdout:VERSION="9" 2024-09-10T18:53:05.911 INFO:teuthology.orchestra.run.smithi191.stdout:ID="centos" 2024-09-10T18:53:05.912 INFO:teuthology.orchestra.run.smithi191.stdout:ID_LIKE="rhel fedora" 2024-09-10T18:53:05.912 INFO:teuthology.orchestra.run.smithi191.stdout:VERSION_ID="9" 2024-09-10T18:53:05.912 INFO:teuthology.orchestra.run.smithi191.stdout:PLATFORM_ID="platform:el9" 2024-09-10T18:53:05.912 INFO:teuthology.orchestra.run.smithi191.stdout:PRETTY_NAME="CentOS Stream 9" 2024-09-10T18:53:05.912 INFO:teuthology.orchestra.run.smithi191.stdout:ANSI_COLOR="0;31" 2024-09-10T18:53:05.912 INFO:teuthology.orchestra.run.smithi191.stdout:LOGO="fedora-logo-icon" 2024-09-10T18:53:05.912 INFO:teuthology.orchestra.run.smithi191.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-09-10T18:53:05.912 INFO:teuthology.orchestra.run.smithi191.stdout:HOME_URL="https://centos.org/" 2024-09-10T18:53:05.912 INFO:teuthology.orchestra.run.smithi191.stdout:BUG_REPORT_URL="https://issues.redhat.com/" 2024-09-10T18:53:05.912 INFO:teuthology.orchestra.run.smithi191.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-09-10T18:53:05.912 INFO:teuthology.orchestra.run.smithi191.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-09-10T18:53:05.912 INFO:teuthology.lock.ops:Updating smithi191.front.sepia.ceph.com on lock server 2024-09-10T18:53:05.936 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-09-10T18:53:05.944 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-09-10T18:53:06.006 INFO:teuthology.task.internal:Checking for old test directory... 2024-09-10T18:53:06.006 DEBUG:teuthology.orchestra.run.smithi183:> test '!' -e /home/ubuntu/cephtest 2024-09-10T18:53:06.009 DEBUG:teuthology.orchestra.run.smithi191:> test '!' -e /home/ubuntu/cephtest 2024-09-10T18:53:06.026 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-09-10T18:53:06.032 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-09-10T18:53:06.032 DEBUG:teuthology.orchestra.run.smithi183:> test -z $(ls -A /var/lib/ceph) 2024-09-10T18:53:06.066 DEBUG:teuthology.orchestra.run.smithi191:> test -z $(ls -A /var/lib/ceph) 2024-09-10T18:53:06.098 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-09-10T18:53:06.153 INFO:teuthology.run_tasks:Running task kernel... 2024-09-10T18:53:06.167 INFO:teuthology.task.kernel:normalize config orig: {'kdb': 1, 'sha1': 'distro'} 2024-09-10T18:53:06.168 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': 1, 'sha1': 'distro'}, 'mon.b': {'kdb': 1, 'sha1': 'distro'}}, timeout 300 2024-09-10T18:53:06.168 DEBUG:teuthology.orchestra.run.smithi183:> test -f /run/.containerenv -o -f /.dockerenv 2024-09-10T18:53:06.168 DEBUG:teuthology.orchestra.run.smithi191:> test -f /run/.containerenv -o -f /.dockerenv 2024-09-10T18:53:06.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:53:06.185 DEBUG:teuthology.orchestra.run.smithi183:> uname -r 2024-09-10T18:53:06.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:53:06.186 DEBUG:teuthology.orchestra.run.smithi191:> uname -r 2024-09-10T18:53:06.243 INFO:teuthology.orchestra.run.smithi183.stdout:5.14.0-503.el9.x86_64 2024-09-10T18:53:06.243 INFO:teuthology.task.kernel:Running kernel on smithi183: 5.14.0-503.el9.x86_64 2024-09-10T18:53:06.244 DEBUG:teuthology.orchestra.run.smithi183:> sudo yum install -y kernel 2024-09-10T18:53:06.244 INFO:teuthology.orchestra.run.smithi191.stdout:5.14.0-503.el9.x86_64 2024-09-10T18:53:06.245 INFO:teuthology.task.kernel:Running kernel on smithi191: 5.14.0-503.el9.x86_64 2024-09-10T18:53:06.245 DEBUG:teuthology.orchestra.run.smithi191:> sudo yum install -y kernel 2024-09-10T18:53:08.630 INFO:teuthology.orchestra.run.smithi191.stdout:CentOS Stream 9 - BaseOS 15 MB/s | 8.2 MB 00:00 2024-09-10T18:53:10.783 INFO:teuthology.orchestra.run.smithi183.stdout:CentOS Stream 9 - BaseOS 3.0 MB/s | 8.2 MB 00:02 2024-09-10T18:53:11.665 INFO:teuthology.orchestra.run.smithi191.stdout:CentOS Stream 9 - AppStream 11 MB/s | 20 MB 00:01 2024-09-10T18:53:12.876 INFO:teuthology.orchestra.run.smithi183.stdout:CentOS Stream 9 - AppStream 21 MB/s | 20 MB 00:00 2024-09-10T18:53:16.949 INFO:teuthology.orchestra.run.smithi191.stdout:CentOS Stream 9 - CRB 4.6 MB/s | 6.5 MB 00:01 2024-09-10T18:53:16.976 INFO:teuthology.orchestra.run.smithi183.stdout:CentOS Stream 9 - CRB 19 MB/s | 6.5 MB 00:00 2024-09-10T18:53:18.417 INFO:teuthology.orchestra.run.smithi191.stdout:CentOS Stream 9 - Extras packages 71 kB/s | 19 kB 00:00 2024-09-10T18:53:18.435 INFO:teuthology.orchestra.run.smithi183.stdout:CentOS Stream 9 - Extras packages 69 kB/s | 19 kB 00:00 2024-09-10T18:53:19.180 INFO:teuthology.orchestra.run.smithi191.stdout:Extra Packages for Enterprise Linux 35 MB/s | 23 MB 00:00 2024-09-10T18:53:19.681 INFO:teuthology.orchestra.run.smithi183.stdout:Extra Packages for Enterprise Linux 20 MB/s | 23 MB 00:01 2024-09-10T18:53:24.644 INFO:teuthology.orchestra.run.smithi191.stdout:lab-extras 33 kB/s | 1.7 kB 00:00 2024-09-10T18:53:25.028 INFO:teuthology.orchestra.run.smithi183.stdout:lab-extras 25 kB/s | 1.7 kB 00:00 2024-09-10T18:53:26.642 INFO:teuthology.orchestra.run.smithi191.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-09-10T18:53:26.643 INFO:teuthology.orchestra.run.smithi191.stdout:Package kernel-5.14.0-503.el9.x86_64 is already installed. 2024-09-10T18:53:26.720 INFO:teuthology.orchestra.run.smithi191.stdout:Dependencies resolved. 2024-09-10T18:53:26.732 INFO:teuthology.orchestra.run.smithi191.stdout:================================================================================ 2024-09-10T18:53:26.733 INFO:teuthology.orchestra.run.smithi191.stdout: Package Architecture Version Repository Size 2024-09-10T18:53:26.733 INFO:teuthology.orchestra.run.smithi191.stdout:================================================================================ 2024-09-10T18:53:26.733 INFO:teuthology.orchestra.run.smithi191.stdout:Installing: 2024-09-10T18:53:26.733 INFO:teuthology.orchestra.run.smithi191.stdout: kernel x86_64 5.14.0-505.el9 baseos 23 k 2024-09-10T18:53:26.733 INFO:teuthology.orchestra.run.smithi191.stdout:Installing dependencies: 2024-09-10T18:53:26.733 INFO:teuthology.orchestra.run.smithi191.stdout: kernel-core x86_64 5.14.0-505.el9 baseos 16 M 2024-09-10T18:53:26.733 INFO:teuthology.orchestra.run.smithi191.stdout: kernel-modules x86_64 5.14.0-505.el9 baseos 35 M 2024-09-10T18:53:26.734 INFO:teuthology.orchestra.run.smithi191.stdout: kernel-modules-core x86_64 5.14.0-505.el9 baseos 29 M 2024-09-10T18:53:26.734 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-10T18:53:26.734 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction Summary 2024-09-10T18:53:26.734 INFO:teuthology.orchestra.run.smithi191.stdout:================================================================================ 2024-09-10T18:53:26.734 INFO:teuthology.orchestra.run.smithi191.stdout:Install 4 Packages 2024-09-10T18:53:26.734 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-10T18:53:26.735 INFO:teuthology.orchestra.run.smithi191.stdout:Total download size: 79 M 2024-09-10T18:53:26.735 INFO:teuthology.orchestra.run.smithi191.stdout:Installed size: 126 M 2024-09-10T18:53:26.736 INFO:teuthology.orchestra.run.smithi191.stdout:Downloading Packages: 2024-09-10T18:53:27.035 INFO:teuthology.orchestra.run.smithi183.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-09-10T18:53:27.035 INFO:teuthology.orchestra.run.smithi183.stdout:Package kernel-5.14.0-503.el9.x86_64 is already installed. 2024-09-10T18:53:27.117 INFO:teuthology.orchestra.run.smithi183.stdout:Dependencies resolved. 2024-09-10T18:53:27.121 INFO:teuthology.orchestra.run.smithi191.stdout:(1/4): kernel-5.14.0-505.el9.x86_64.rpm 211 kB/s | 23 kB 00:00 2024-09-10T18:53:27.129 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2024-09-10T18:53:27.129 INFO:teuthology.orchestra.run.smithi183.stdout: Package Architecture Version Repository Size 2024-09-10T18:53:27.129 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2024-09-10T18:53:27.129 INFO:teuthology.orchestra.run.smithi183.stdout:Installing: 2024-09-10T18:53:27.129 INFO:teuthology.orchestra.run.smithi183.stdout: kernel x86_64 5.14.0-505.el9 baseos 23 k 2024-09-10T18:53:27.130 INFO:teuthology.orchestra.run.smithi183.stdout:Installing dependencies: 2024-09-10T18:53:27.130 INFO:teuthology.orchestra.run.smithi183.stdout: kernel-core x86_64 5.14.0-505.el9 baseos 16 M 2024-09-10T18:53:27.130 INFO:teuthology.orchestra.run.smithi183.stdout: kernel-modules x86_64 5.14.0-505.el9 baseos 35 M 2024-09-10T18:53:27.130 INFO:teuthology.orchestra.run.smithi183.stdout: kernel-modules-core x86_64 5.14.0-505.el9 baseos 29 M 2024-09-10T18:53:27.130 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-09-10T18:53:27.130 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction Summary 2024-09-10T18:53:27.130 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2024-09-10T18:53:27.130 INFO:teuthology.orchestra.run.smithi183.stdout:Install 4 Packages 2024-09-10T18:53:27.130 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-09-10T18:53:27.131 INFO:teuthology.orchestra.run.smithi183.stdout:Total download size: 79 M 2024-09-10T18:53:27.131 INFO:teuthology.orchestra.run.smithi183.stdout:Installed size: 126 M 2024-09-10T18:53:27.131 INFO:teuthology.orchestra.run.smithi183.stdout:Downloading Packages: 2024-09-10T18:53:27.496 INFO:teuthology.orchestra.run.smithi183.stdout:(1/4): kernel-5.14.0-505.el9.x86_64.rpm 210 kB/s | 23 kB 00:00 2024-09-10T18:53:27.896 INFO:teuthology.orchestra.run.smithi183.stdout:(2/4): kernel-core-5.14.0-505.el9.x86_64.rpm 31 MB/s | 16 MB 00:00 2024-09-10T18:53:28.163 INFO:teuthology.orchestra.run.smithi191.stdout:(2/4): kernel-modules-5.14.0-505.el9.x86_64.rpm 30 MB/s | 35 MB 00:01 2024-09-10T18:53:28.255 INFO:teuthology.orchestra.run.smithi183.stdout:(3/4): kernel-modules-core-5.14.0-505.el9.x86_6 38 MB/s | 29 MB 00:00 2024-09-10T18:53:28.513 INFO:teuthology.orchestra.run.smithi191.stdout:(3/4): kernel-core-5.14.0-505.el9.x86_64.rpm 10 MB/s | 16 MB 00:01 2024-09-10T18:53:28.597 INFO:teuthology.orchestra.run.smithi183.stdout:(4/4): kernel-modules-5.14.0-505.el9.x86_64.rpm 29 MB/s | 35 MB 00:01 2024-09-10T18:53:28.597 INFO:teuthology.orchestra.run.smithi183.stdout:-------------------------------------------------------------------------------- 2024-09-10T18:53:28.597 INFO:teuthology.orchestra.run.smithi183.stdout:Total 54 MB/s | 79 MB 00:01 2024-09-10T18:53:29.005 INFO:teuthology.orchestra.run.smithi191.stdout:(4/4): kernel-modules-core-5.14.0-505.el9.x86_6 15 MB/s | 29 MB 00:01 2024-09-10T18:53:29.005 INFO:teuthology.orchestra.run.smithi191.stdout:-------------------------------------------------------------------------------- 2024-09-10T18:53:29.005 INFO:teuthology.orchestra.run.smithi191.stdout:Total 35 MB/s | 79 MB 00:02 2024-09-10T18:53:29.275 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction check 2024-09-10T18:53:29.619 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction check succeeded. 2024-09-10T18:53:29.620 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction test 2024-09-10T18:53:29.682 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction check 2024-09-10T18:53:30.021 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction check succeeded. 2024-09-10T18:53:30.021 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction test 2024-09-10T18:53:30.596 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction test succeeded. 2024-09-10T18:53:30.596 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction 2024-09-10T18:53:30.975 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction test succeeded. 2024-09-10T18:53:30.976 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction 2024-09-10T18:53:33.052 INFO:teuthology.orchestra.run.smithi183.stdout: Preparing : 1/1 2024-09-10T18:53:33.413 INFO:teuthology.orchestra.run.smithi191.stdout: Preparing : 1/1 2024-09-10T18:53:33.897 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : kernel-modules-core-5.14.0-505.el9.x86_64 1/4 2024-09-10T18:53:33.994 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : kernel-core-5.14.0-505.el9.x86_64 2/4 2024-09-10T18:53:34.127 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : kernel-modules-core-5.14.0-505.el9.x86_64 1/4 2024-09-10T18:53:34.213 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : kernel-core-5.14.0-505.el9.x86_64 2/4 2024-09-10T18:53:36.773 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: kernel-core-5.14.0-505.el9.x86_64 2/4 2024-09-10T18:53:36.924 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : kernel-modules-5.14.0-505.el9.x86_64 3/4 2024-09-10T18:53:36.989 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: kernel-core-5.14.0-505.el9.x86_64 2/4 2024-09-10T18:53:37.126 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : kernel-modules-5.14.0-505.el9.x86_64 3/4 2024-09-10T18:53:41.769 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: kernel-modules-5.14.0-505.el9.x86_64 3/4 2024-09-10T18:53:41.865 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : kernel-5.14.0-505.el9.x86_64 4/4 2024-09-10T18:53:41.954 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: kernel-modules-5.14.0-505.el9.x86_64 3/4 2024-09-10T18:53:42.086 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : kernel-5.14.0-505.el9.x86_64 4/4 2024-09-10T18:53:46.731 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: kernel-modules-core-5.14.0-505.el9.x86_64 4/4 2024-09-10T18:53:46.926 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: kernel-modules-core-5.14.0-505.el9.x86_64 4/4 2024-09-10T18:54:24.679 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: kernel-core-5.14.0-505.el9.x86_64 4/4 2024-09-10T18:54:24.686 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: kernel-core-5.14.0-505.el9.x86_64 4/4 2024-09-10T18:54:24.718 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: kernel-modules-5.14.0-505.el9.x86_64 4/4 2024-09-10T18:54:24.726 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: kernel-modules-5.14.0-505.el9.x86_64 4/4 2024-09-10T18:54:24.992 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: kernel-5.14.0-505.el9.x86_64 4/4 2024-09-10T18:54:24.992 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : kernel-5.14.0-505.el9.x86_64 1/4 2024-09-10T18:54:24.992 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : kernel-core-5.14.0-505.el9.x86_64 2/4 2024-09-10T18:54:24.992 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : kernel-modules-5.14.0-505.el9.x86_64 3/4 2024-09-10T18:54:25.062 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: kernel-5.14.0-505.el9.x86_64 4/4 2024-09-10T18:54:25.062 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : kernel-5.14.0-505.el9.x86_64 1/4 2024-09-10T18:54:25.062 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : kernel-core-5.14.0-505.el9.x86_64 2/4 2024-09-10T18:54:25.062 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : kernel-modules-5.14.0-505.el9.x86_64 3/4 2024-09-10T18:54:25.819 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : kernel-modules-core-5.14.0-505.el9.x86_64 4/4 2024-09-10T18:54:25.819 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-10T18:54:25.819 INFO:teuthology.orchestra.run.smithi191.stdout:Installed: 2024-09-10T18:54:25.819 INFO:teuthology.orchestra.run.smithi191.stdout: kernel-5.14.0-505.el9.x86_64 2024-09-10T18:54:25.819 INFO:teuthology.orchestra.run.smithi191.stdout: kernel-core-5.14.0-505.el9.x86_64 2024-09-10T18:54:25.819 INFO:teuthology.orchestra.run.smithi191.stdout: kernel-modules-5.14.0-505.el9.x86_64 2024-09-10T18:54:25.819 INFO:teuthology.orchestra.run.smithi191.stdout: kernel-modules-core-5.14.0-505.el9.x86_64 2024-09-10T18:54:25.819 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-10T18:54:25.819 INFO:teuthology.orchestra.run.smithi191.stdout:Complete! 2024-09-10T18:54:25.926 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : kernel-modules-core-5.14.0-505.el9.x86_64 4/4 2024-09-10T18:54:25.926 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-09-10T18:54:25.926 INFO:teuthology.orchestra.run.smithi183.stdout:Installed: 2024-09-10T18:54:25.927 INFO:teuthology.orchestra.run.smithi183.stdout: kernel-5.14.0-505.el9.x86_64 2024-09-10T18:54:25.927 INFO:teuthology.orchestra.run.smithi183.stdout: kernel-core-5.14.0-505.el9.x86_64 2024-09-10T18:54:25.927 INFO:teuthology.orchestra.run.smithi183.stdout: kernel-modules-5.14.0-505.el9.x86_64 2024-09-10T18:54:25.927 INFO:teuthology.orchestra.run.smithi183.stdout: kernel-modules-core-5.14.0-505.el9.x86_64 2024-09-10T18:54:25.927 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-09-10T18:54:25.927 INFO:teuthology.orchestra.run.smithi183.stdout:Complete! 2024-09-10T18:54:26.177 DEBUG:teuthology.orchestra.run.smithi191:> rpm -q kernel | sort -rV | head -n 1 2024-09-10T18:54:26.271 INFO:teuthology.orchestra.run.smithi191.stdout:kernel-5.14.0-505.el9.x86_64 2024-09-10T18:54:26.271 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-505.el9.x86_64 2024-09-10T18:54:26.271 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-503.el9.x86_64 Expected: 5.14.0-505.el9.x86_64 2024-09-10T18:54:26.271 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-09-10T18:54:26.271 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-09-10T18:54:26.272 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2024-09-10T18:54:26.272 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi191.front.sepia.ceph.com, path=None, version=distro) 2024-09-10T18:54:26.272 DEBUG:teuthology.orchestra.run.smithi191:> rpm -q kernel | sort -rV | head -n 1 2024-09-10T18:54:26.278 DEBUG:teuthology.orchestra.run.smithi183:> rpm -q kernel | sort -rV | head -n 1 2024-09-10T18:54:26.331 INFO:teuthology.orchestra.run.smithi191.stdout:kernel-5.14.0-505.el9.x86_64 2024-09-10T18:54:26.331 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-505.el9.x86_64 2024-09-10T18:54:26.331 DEBUG:teuthology.orchestra.run.smithi191:> sudo rpm -qi grub2-tools 2024-09-10T18:54:26.368 INFO:teuthology.orchestra.run.smithi191.stdout:Name : grub2-tools 2024-09-10T18:54:26.368 INFO:teuthology.orchestra.run.smithi191.stdout:Epoch : 1 2024-09-10T18:54:26.368 INFO:teuthology.orchestra.run.smithi191.stdout:Version : 2.06 2024-09-10T18:54:26.368 INFO:teuthology.orchestra.run.smithi191.stdout:Release : 82.el9 2024-09-10T18:54:26.368 INFO:teuthology.orchestra.run.smithi191.stdout:Architecture: x86_64 2024-09-10T18:54:26.368 INFO:teuthology.orchestra.run.smithi191.stdout:Install Date: Wed 28 Aug 2024 09:06:27 PM UTC 2024-09-10T18:54:26.368 INFO:teuthology.orchestra.run.smithi191.stdout:Group : Unspecified 2024-09-10T18:54:26.368 INFO:teuthology.orchestra.run.smithi191.stdout:Size : 8274670 2024-09-10T18:54:26.368 INFO:teuthology.orchestra.run.smithi191.stdout:License : GPLv3+ 2024-09-10T18:54:26.369 INFO:teuthology.orchestra.run.smithi191.stdout:Signature : RSA/SHA256, Mon 01 Jul 2024 07:46:42 AM UTC, Key ID 05b555b38483c65d 2024-09-10T18:54:26.369 INFO:teuthology.orchestra.run.smithi191.stdout:Source RPM : grub2-2.06-82.el9.src.rpm 2024-09-10T18:54:26.369 INFO:teuthology.orchestra.run.smithi191.stdout:Build Date : Fri 28 Jun 2024 09:24:44 AM UTC 2024-09-10T18:54:26.369 INFO:teuthology.orchestra.run.smithi191.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-09-10T18:54:26.369 INFO:teuthology.orchestra.run.smithi191.stdout:Packager : builder@centos.org 2024-09-10T18:54:26.369 INFO:teuthology.orchestra.run.smithi191.stdout:Vendor : CentOS 2024-09-10T18:54:26.369 INFO:teuthology.orchestra.run.smithi191.stdout:URL : http://www.gnu.org/software/grub/ 2024-09-10T18:54:26.369 INFO:teuthology.orchestra.run.smithi191.stdout:Summary : Support tools for GRUB. 2024-09-10T18:54:26.369 INFO:teuthology.orchestra.run.smithi191.stdout:Description : 2024-09-10T18:54:26.369 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-10T18:54:26.369 INFO:teuthology.orchestra.run.smithi191.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-09-10T18:54:26.369 INFO:teuthology.orchestra.run.smithi191.stdout:customizable bootloader with modular architecture. It supports a rich 2024-09-10T18:54:26.369 INFO:teuthology.orchestra.run.smithi191.stdout:variety of kernel formats, file systems, computer architectures and 2024-09-10T18:54:26.369 INFO:teuthology.orchestra.run.smithi191.stdout:hardware devices. 2024-09-10T18:54:26.370 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-10T18:54:26.370 INFO:teuthology.orchestra.run.smithi191.stdout:This subpackage provides tools for support of all platforms. 2024-09-10T18:54:26.371 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-09-10T18:54:26.371 INFO:teuthology.task.kernel:Updating grub on smithi191 to boot 5.14.0-505.el9.x86_64 2024-09-10T18:54:26.371 DEBUG:teuthology.orchestra.run.smithi191:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-09-10T18:54:26.372 INFO:teuthology.orchestra.run.smithi183.stdout:kernel-5.14.0-505.el9.x86_64 2024-09-10T18:54:26.372 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-505.el9.x86_64 2024-09-10T18:54:26.372 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-503.el9.x86_64 Expected: 5.14.0-505.el9.x86_64 2024-09-10T18:54:26.372 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-09-10T18:54:26.372 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-09-10T18:54:26.372 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-09-10T18:54:26.372 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi183.front.sepia.ceph.com, path=None, version=distro) 2024-09-10T18:54:26.373 DEBUG:teuthology.orchestra.run.smithi183:> rpm -q kernel | sort -rV | head -n 1 2024-09-10T18:54:26.430 INFO:teuthology.orchestra.run.smithi183.stdout:kernel-5.14.0-505.el9.x86_64 2024-09-10T18:54:26.431 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-505.el9.x86_64 2024-09-10T18:54:26.431 DEBUG:teuthology.orchestra.run.smithi183:> sudo rpm -qi grub2-tools 2024-09-10T18:54:26.465 INFO:teuthology.orchestra.run.smithi183.stdout:Name : grub2-tools 2024-09-10T18:54:26.465 INFO:teuthology.orchestra.run.smithi183.stdout:Epoch : 1 2024-09-10T18:54:26.466 INFO:teuthology.orchestra.run.smithi183.stdout:Version : 2.06 2024-09-10T18:54:26.466 INFO:teuthology.orchestra.run.smithi183.stdout:Release : 82.el9 2024-09-10T18:54:26.466 INFO:teuthology.orchestra.run.smithi183.stdout:Architecture: x86_64 2024-09-10T18:54:26.466 INFO:teuthology.orchestra.run.smithi183.stdout:Install Date: Wed 28 Aug 2024 09:06:27 PM UTC 2024-09-10T18:54:26.466 INFO:teuthology.orchestra.run.smithi183.stdout:Group : Unspecified 2024-09-10T18:54:26.466 INFO:teuthology.orchestra.run.smithi183.stdout:Size : 8274670 2024-09-10T18:54:26.466 INFO:teuthology.orchestra.run.smithi183.stdout:License : GPLv3+ 2024-09-10T18:54:26.466 INFO:teuthology.orchestra.run.smithi183.stdout:Signature : RSA/SHA256, Mon 01 Jul 2024 07:46:42 AM UTC, Key ID 05b555b38483c65d 2024-09-10T18:54:26.466 INFO:teuthology.orchestra.run.smithi183.stdout:Source RPM : grub2-2.06-82.el9.src.rpm 2024-09-10T18:54:26.466 INFO:teuthology.orchestra.run.smithi183.stdout:Build Date : Fri 28 Jun 2024 09:24:44 AM UTC 2024-09-10T18:54:26.466 INFO:teuthology.orchestra.run.smithi183.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-09-10T18:54:26.466 INFO:teuthology.orchestra.run.smithi183.stdout:Packager : builder@centos.org 2024-09-10T18:54:26.467 INFO:teuthology.orchestra.run.smithi183.stdout:Vendor : CentOS 2024-09-10T18:54:26.467 INFO:teuthology.orchestra.run.smithi183.stdout:URL : http://www.gnu.org/software/grub/ 2024-09-10T18:54:26.467 INFO:teuthology.orchestra.run.smithi183.stdout:Summary : Support tools for GRUB. 2024-09-10T18:54:26.467 INFO:teuthology.orchestra.run.smithi183.stdout:Description : 2024-09-10T18:54:26.467 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-09-10T18:54:26.467 INFO:teuthology.orchestra.run.smithi183.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-09-10T18:54:26.467 INFO:teuthology.orchestra.run.smithi183.stdout:customizable bootloader with modular architecture. It supports a rich 2024-09-10T18:54:26.467 INFO:teuthology.orchestra.run.smithi183.stdout:variety of kernel formats, file systems, computer architectures and 2024-09-10T18:54:26.467 INFO:teuthology.orchestra.run.smithi183.stdout:hardware devices. 2024-09-10T18:54:26.467 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-09-10T18:54:26.467 INFO:teuthology.orchestra.run.smithi183.stdout:This subpackage provides tools for support of all platforms. 2024-09-10T18:54:26.468 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-09-10T18:54:26.468 INFO:teuthology.task.kernel:Updating grub on smithi183 to boot 5.14.0-505.el9.x86_64 2024-09-10T18:54:26.469 DEBUG:teuthology.orchestra.run.smithi183:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-09-10T18:54:27.042 INFO:teuthology.orchestra.run.smithi191.stderr:Generating grub configuration file ... 2024-09-10T18:54:27.182 INFO:teuthology.orchestra.run.smithi183.stderr:Generating grub configuration file ... 2024-09-10T18:54:28.321 INFO:teuthology.orchestra.run.smithi191.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-09-10T18:54:28.344 INFO:teuthology.orchestra.run.smithi191.stderr:done 2024-09-10T18:54:28.350 DEBUG:teuthology.orchestra.run.smithi191:> mktemp 2024-09-10T18:54:28.365 INFO:teuthology.orchestra.run.smithi191.stdout:/tmp/tmp.ooepKB1CEg 2024-09-10T18:54:28.366 DEBUG:teuthology.orchestra.run.smithi191:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.ooepKB1CEg 2024-09-10T18:54:28.432 DEBUG:teuthology.orchestra.run.smithi191:> sudo chmod 0666 /tmp/tmp.ooepKB1CEg 2024-09-10T18:54:28.567 DEBUG:teuthology.orchestra.remote:smithi191:/tmp/tmp.ooepKB1CEg is 6KB 2024-09-10T18:54:28.585 INFO:teuthology.orchestra.run.smithi183.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-09-10T18:54:28.614 INFO:teuthology.orchestra.run.smithi183.stderr:done 2024-09-10T18:54:28.616 DEBUG:teuthology.orchestra.run.smithi183:> mktemp 2024-09-10T18:54:28.617 DEBUG:teuthology.orchestra.run.smithi191:> rm -fr /tmp/tmp.ooepKB1CEg 2024-09-10T18:54:28.631 INFO:teuthology.orchestra.run.smithi183.stdout:/tmp/tmp.YzmABVrahd 2024-09-10T18:54:28.631 DEBUG:teuthology.orchestra.run.smithi183:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.YzmABVrahd 2024-09-10T18:54:28.632 DEBUG:teuthology.orchestra.run.smithi191:> sudo /bin/ls /boot/loader/entries || true 2024-09-10T18:54:28.699 DEBUG:teuthology.orchestra.run.smithi183:> sudo chmod 0666 /tmp/tmp.YzmABVrahd 2024-09-10T18:54:28.703 INFO:teuthology.orchestra.run.smithi191.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-0-rescue.conf 2024-09-10T18:54:28.703 INFO:teuthology.orchestra.run.smithi191.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-239.el9.x86_64.conf 2024-09-10T18:54:28.703 INFO:teuthology.orchestra.run.smithi191.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-503.el9.x86_64.conf 2024-09-10T18:54:28.703 INFO:teuthology.orchestra.run.smithi191.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-505.el9.x86_64.conf 2024-09-10T18:54:28.705 DEBUG:teuthology.orchestra.run.smithi191:> sudo grub2-set-default fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-505.el9.x86_64 2024-09-10T18:54:28.833 DEBUG:teuthology.orchestra.remote:smithi183:/tmp/tmp.YzmABVrahd is 6KB 2024-09-10T18:54:28.853 DEBUG:teuthology.orchestra.run.smithi183:> rm -fr /tmp/tmp.YzmABVrahd 2024-09-10T18:54:28.866 DEBUG:teuthology.orchestra.run.smithi191:> sudo shutdown -r now 2024-09-10T18:54:28.868 DEBUG:teuthology.orchestra.run.smithi183:> sudo /bin/ls /boot/loader/entries || true 2024-09-10T18:54:28.940 INFO:teuthology.orchestra.run.smithi183.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-0-rescue.conf 2024-09-10T18:54:28.940 INFO:teuthology.orchestra.run.smithi183.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-239.el9.x86_64.conf 2024-09-10T18:54:28.940 INFO:teuthology.orchestra.run.smithi183.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-503.el9.x86_64.conf 2024-09-10T18:54:28.940 INFO:teuthology.orchestra.run.smithi183.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-505.el9.x86_64.conf 2024-09-10T18:54:28.942 DEBUG:teuthology.orchestra.run.smithi183:> sudo grub2-set-default fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-505.el9.x86_64 2024-09-10T18:54:29.114 DEBUG:teuthology.orchestra.run.smithi183:> sudo shutdown -r now 2024-09-10T18:54:58.894 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2024-09-10T18:54:58.894 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi191.front.sepia.ceph.com' 2024-09-10T18:54:58.896 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi191.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-10T18:54:59.117 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-09-10T18:54:59.117 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi183.front.sepia.ceph.com' 2024-09-10T18:54:59.118 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi183.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-10T18:55:17.309 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.191 2024-09-10T18:55:17.565 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.183 2024-09-10T18:55:26.317 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi191.front.sepia.ceph.com' 2024-09-10T18:55:26.318 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi191.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-10T18:55:26.566 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi183.front.sepia.ceph.com' 2024-09-10T18:55:26.566 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi183.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-10T18:55:29.373 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.191 2024-09-10T18:55:29.629 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.183 2024-09-10T18:55:41.385 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi191.front.sepia.ceph.com' 2024-09-10T18:55:41.385 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi191.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-10T18:55:41.630 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi183.front.sepia.ceph.com' 2024-09-10T18:55:41.631 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi183.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-10T18:55:59.805 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.191 2024-09-10T18:56:00.061 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.183 2024-09-10T18:56:14.821 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi191.front.sepia.ceph.com' 2024-09-10T18:56:14.822 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi191.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-10T18:56:15.062 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi183.front.sepia.ceph.com' 2024-09-10T18:56:15.063 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi183.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-10T18:56:15.112 DEBUG:teuthology.orchestra.run.smithi191:> true 2024-09-10T18:56:15.473 DEBUG:teuthology.orchestra.run.smithi183:> true 2024-09-10T18:56:15.509 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi191.front.sepia.ceph.com' 2024-09-10T18:56:15.509 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "5.14.0-505.el9.x86_64"... 2024-09-10T18:56:15.509 DEBUG:teuthology.orchestra.run.smithi191:> uname -r 2024-09-10T18:56:15.524 INFO:teuthology.orchestra.run.smithi191.stdout:5.14.0-505.el9.x86_64 2024-09-10T18:56:15.524 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-505.el9.x86_64 vs 5.14.0-505.el9.x86_64 2024-09-10T18:56:15.524 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-09-10T18:56:15.524 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-09-10T18:56:16.167 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi183.front.sepia.ceph.com' 2024-09-10T18:56:16.167 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-505.el9.x86_64"... 2024-09-10T18:56:16.168 DEBUG:teuthology.orchestra.run.smithi183:> uname -r 2024-09-10T18:56:16.181 INFO:teuthology.orchestra.run.smithi183.stdout:5.14.0-505.el9.x86_64 2024-09-10T18:56:16.182 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-505.el9.x86_64 vs 5.14.0-505.el9.x86_64 2024-09-10T18:56:16.182 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-09-10T18:56:16.182 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-09-10T18:56:16.525 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2024-09-10T18:56:16.526 DEBUG:teuthology.orchestra.run.smithi191:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-09-10T18:56:16.612 INFO:teuthology.orchestra.run.smithi191.stdout:ttyS1 2024-09-10T18:56:16.633 DEBUG:teuthology.parallel:result is None 2024-09-10T18:56:17.183 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-09-10T18:56:17.183 DEBUG:teuthology.orchestra.run.smithi183:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-09-10T18:56:17.269 INFO:teuthology.orchestra.run.smithi183.stdout:ttyS1 2024-09-10T18:56:17.300 DEBUG:teuthology.parallel:result is None 2024-09-10T18:56:17.301 INFO:teuthology.run_tasks:Running task internal.base... 2024-09-10T18:56:17.309 INFO:teuthology.task.internal:Creating test directory... 2024-09-10T18:56:17.309 DEBUG:teuthology.orchestra.run.smithi183:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-09-10T18:56:17.311 DEBUG:teuthology.orchestra.run.smithi191:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-09-10T18:56:17.347 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-09-10T18:56:17.354 INFO:teuthology.run_tasks:Running task internal.archive... 2024-09-10T18:56:17.360 INFO:teuthology.task.internal:Creating archive directory... 2024-09-10T18:56:17.360 DEBUG:teuthology.orchestra.run.smithi183:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-09-10T18:56:17.369 DEBUG:teuthology.orchestra.run.smithi191:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-09-10T18:56:17.465 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-09-10T18:56:17.475 INFO:teuthology.task.internal:Enabling coredump saving... 2024-09-10T18:56:17.475 DEBUG:teuthology.orchestra.run.smithi183:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-09-10T18:56:17.478 DEBUG:teuthology.orchestra.run.smithi191:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-09-10T18:56:17.512 INFO:teuthology.orchestra.run.smithi183.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T18:56:17.523 INFO:teuthology.orchestra.run.smithi183.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T18:56:17.526 INFO:teuthology.orchestra.run.smithi191.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T18:56:17.550 INFO:teuthology.orchestra.run.smithi191.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T18:56:17.555 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-09-10T18:56:17.561 INFO:teuthology.task.internal:Configuring sudo... 2024-09-10T18:56:17.561 DEBUG:teuthology.orchestra.run.smithi183:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-09-10T18:56:17.567 DEBUG:teuthology.orchestra.run.smithi191:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-09-10T18:56:17.619 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-09-10T18:56:17.629 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-09-10T18:56:17.629 DEBUG:teuthology.orchestra.run.smithi183:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-09-10T18:56:17.635 DEBUG:teuthology.orchestra.run.smithi191:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-09-10T18:56:17.666 DEBUG:teuthology.orchestra.run.smithi183:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-10T18:56:17.736 DEBUG:teuthology.orchestra.run.smithi183:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-10T18:56:17.788 DEBUG:teuthology.orchestra.run.smithi183:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-10T18:56:17.872 DEBUG:teuthology.orchestra.run.smithi183:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-10T18:56:17.916 DEBUG:teuthology.orchestra.run.smithi183:> set -ex 2024-09-10T18:56:17.916 DEBUG:teuthology.orchestra.run.smithi183:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-09-10T18:56:17.982 DEBUG:teuthology.orchestra.run.smithi191:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-10T18:56:18.052 DEBUG:teuthology.orchestra.run.smithi191:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-10T18:56:18.115 DEBUG:teuthology.orchestra.run.smithi191:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-10T18:56:18.156 DEBUG:teuthology.orchestra.run.smithi191:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-10T18:56:18.196 DEBUG:teuthology.orchestra.run.smithi191:> set -ex 2024-09-10T18:56:18.196 DEBUG:teuthology.orchestra.run.smithi191:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-09-10T18:56:18.258 DEBUG:teuthology.orchestra.run.smithi183:> sudo service rsyslog restart 2024-09-10T18:56:18.261 DEBUG:teuthology.orchestra.run.smithi191:> sudo service rsyslog restart 2024-09-10T18:56:18.300 INFO:teuthology.orchestra.run.smithi183.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-10T18:56:18.345 INFO:teuthology.orchestra.run.smithi191.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-10T18:56:18.825 INFO:teuthology.run_tasks:Running task internal.timer... 2024-09-10T18:56:18.832 INFO:teuthology.task.internal:Starting timer... 2024-09-10T18:56:18.833 INFO:teuthology.run_tasks:Running task pcp... 2024-09-10T18:56:18.842 INFO:teuthology.run_tasks:Running task selinux... 2024-09-10T18:56:18.854 DEBUG:teuthology.task:Applying overrides for task selinux: {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']} 2024-09-10T18:56:18.854 DEBUG:teuthology.orchestra.run.smithi183:> sudo service auditd rotate 2024-09-10T18:56:18.938 INFO:teuthology.orchestra.run.smithi183.stdout:Rotating logs: 2024-09-10T18:56:18.940 DEBUG:teuthology.orchestra.run.smithi191:> sudo service auditd rotate 2024-09-10T18:56:18.984 INFO:teuthology.orchestra.run.smithi191.stdout:Rotating logs: 2024-09-10T18:56:18.986 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-09-10T18:56:18.986 DEBUG:teuthology.orchestra.run.smithi183:> /usr/sbin/getenforce 2024-09-10T18:56:19.012 INFO:teuthology.orchestra.run.smithi183.stdout:Permissive 2024-09-10T18:56:19.012 DEBUG:teuthology.orchestra.run.smithi191:> /usr/sbin/getenforce 2024-09-10T18:56:19.044 INFO:teuthology.orchestra.run.smithi191.stdout:Permissive 2024-09-10T18:56:19.044 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi183.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi191.front.sepia.ceph.com': 'permissive'} 2024-09-10T18:56:19.045 DEBUG:teuthology.orchestra.run.smithi183:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 2024-09-10T18:56:19.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:56:19.080 DEBUG:teuthology.orchestra.run.smithi191:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 2024-09-10T18:56:19.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T18:56:19.105 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-09-10T18:56:19.105 DEBUG:teuthology.orchestra.run.smithi183:> sudo /usr/sbin/setenforce permissive 2024-09-10T18:56:19.159 DEBUG:teuthology.orchestra.run.smithi191:> sudo /usr/sbin/setenforce permissive 2024-09-10T18:56:19.189 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-09-10T18:56:19.198 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-09-10T18:56:19.209 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-09-10T18:56:19.210 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi183.front.sepia.ceph.com,smithi191.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-09-10T18:59:58.589 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi183.front.sepia.ceph.com'), Remote(name='ubuntu@smithi191.front.sepia.ceph.com')] 2024-09-10T18:59:58.590 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi183.front.sepia.ceph.com' 2024-09-10T18:59:58.592 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi183.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-10T18:59:58.668 DEBUG:teuthology.orchestra.run.smithi183:> true 2024-09-10T18:59:58.753 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi183.front.sepia.ceph.com' 2024-09-10T18:59:58.756 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi191.front.sepia.ceph.com' 2024-09-10T18:59:58.757 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi191.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-10T18:59:58.829 DEBUG:teuthology.orchestra.run.smithi191:> true 2024-09-10T18:59:58.911 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi191.front.sepia.ceph.com' 2024-09-10T18:59:58.912 INFO:teuthology.run_tasks:Running task clock... 2024-09-10T18:59:58.921 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-09-10T18:59:58.921 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-09-10T18:59:58.922 DEBUG:teuthology.orchestra.run.smithi183:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-10T18:59:58.924 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-09-10T18:59:58.924 DEBUG:teuthology.orchestra.run.smithi191:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-10T18:59:58.952 INFO:teuthology.orchestra.run.smithi183.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-09-10T18:59:58.968 INFO:teuthology.orchestra.run.smithi183.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-09-10T18:59:58.980 INFO:teuthology.orchestra.run.smithi191.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-09-10T18:59:58.993 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: ntpd: command not found 2024-09-10T18:59:58.995 INFO:teuthology.orchestra.run.smithi191.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-09-10T18:59:59.004 INFO:teuthology.orchestra.run.smithi183.stdout:506 Cannot talk to daemon 2024-09-10T18:59:59.019 INFO:teuthology.orchestra.run.smithi183.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-09-10T18:59:59.019 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: ntpd: command not found 2024-09-10T18:59:59.030 INFO:teuthology.orchestra.run.smithi191.stdout:506 Cannot talk to daemon 2024-09-10T18:59:59.032 INFO:teuthology.orchestra.run.smithi183.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-09-10T18:59:59.045 INFO:teuthology.orchestra.run.smithi191.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-09-10T18:59:59.059 INFO:teuthology.orchestra.run.smithi191.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-09-10T18:59:59.086 INFO:teuthology.orchestra.run.smithi183.stderr:bash: line 1: ntpq: command not found 2024-09-10T18:59:59.090 INFO:teuthology.orchestra.run.smithi183.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-10T18:59:59.090 INFO:teuthology.orchestra.run.smithi183.stdout:=============================================================================== 2024-09-10T18:59:59.090 INFO:teuthology.orchestra.run.smithi183.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-10T18:59:59.090 INFO:teuthology.orchestra.run.smithi183.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-10T18:59:59.091 INFO:teuthology.orchestra.run.smithi183.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-10T18:59:59.091 INFO:teuthology.orchestra.run.smithi183.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-10T18:59:59.121 INFO:teuthology.orchestra.run.smithi191.stderr:bash: line 1: ntpq: command not found 2024-09-10T18:59:59.125 INFO:teuthology.orchestra.run.smithi191.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-10T18:59:59.126 INFO:teuthology.orchestra.run.smithi191.stdout:=============================================================================== 2024-09-10T18:59:59.126 INFO:teuthology.orchestra.run.smithi191.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-10T18:59:59.126 INFO:teuthology.orchestra.run.smithi191.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-10T18:59:59.126 INFO:teuthology.orchestra.run.smithi191.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-10T18:59:59.126 INFO:teuthology.orchestra.run.smithi191.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-10T18:59:59.126 INFO:teuthology.run_tasks:Running task pexec... 2024-09-10T18:59:59.135 INFO:teuthology.task.pexec:Executing custom commands... 2024-09-10T18:59:59.135 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi183.front.sepia.ceph.com 2024-09-10T18:59:59.136 DEBUG:teuthology.orchestra.run.smithi183:> TESTDIR=/home/ubuntu/cephtest bash -s 2024-09-10T18:59:59.136 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi191.front.sepia.ceph.com 2024-09-10T18:59:59.136 DEBUG:teuthology.orchestra.run.smithi191:> TESTDIR=/home/ubuntu/cephtest bash -s 2024-09-10T18:59:59.486 INFO:teuthology.orchestra.run.smithi183.stdout:Dependencies resolved. 2024-09-10T18:59:59.501 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2024-09-10T18:59:59.501 INFO:teuthology.orchestra.run.smithi183.stdout: Package Architecture Version Repository Size 2024-09-10T18:59:59.501 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2024-09-10T18:59:59.501 INFO:teuthology.orchestra.run.smithi183.stdout:Removing: 2024-09-10T18:59:59.501 INFO:teuthology.orchestra.run.smithi183.stdout: nvme-cli x86_64 2.9.1-6.el9 @baseos 5.3 M 2024-09-10T18:59:59.501 INFO:teuthology.orchestra.run.smithi183.stdout:Removing unused dependencies: 2024-09-10T18:59:59.502 INFO:teuthology.orchestra.run.smithi183.stdout: libnvme x86_64 1.9-3.el9 @baseos 273 k 2024-09-10T18:59:59.502 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-09-10T18:59:59.502 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction Summary 2024-09-10T18:59:59.502 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2024-09-10T18:59:59.502 INFO:teuthology.orchestra.run.smithi183.stdout:Remove 2 Packages 2024-09-10T18:59:59.502 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-09-10T18:59:59.502 INFO:teuthology.orchestra.run.smithi183.stdout:Freed space: 5.5 M 2024-09-10T18:59:59.502 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction check 2024-09-10T18:59:59.505 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction check succeeded. 2024-09-10T18:59:59.506 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction test 2024-09-10T18:59:59.521 INFO:teuthology.orchestra.run.smithi191.stdout:Dependencies resolved. 2024-09-10T18:59:59.536 INFO:teuthology.orchestra.run.smithi191.stdout:================================================================================ 2024-09-10T18:59:59.536 INFO:teuthology.orchestra.run.smithi191.stdout: Package Architecture Version Repository Size 2024-09-10T18:59:59.536 INFO:teuthology.orchestra.run.smithi191.stdout:================================================================================ 2024-09-10T18:59:59.536 INFO:teuthology.orchestra.run.smithi191.stdout:Removing: 2024-09-10T18:59:59.536 INFO:teuthology.orchestra.run.smithi191.stdout: nvme-cli x86_64 2.9.1-6.el9 @baseos 5.3 M 2024-09-10T18:59:59.536 INFO:teuthology.orchestra.run.smithi191.stdout:Removing unused dependencies: 2024-09-10T18:59:59.536 INFO:teuthology.orchestra.run.smithi191.stdout: libnvme x86_64 1.9-3.el9 @baseos 273 k 2024-09-10T18:59:59.536 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-10T18:59:59.537 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction Summary 2024-09-10T18:59:59.537 INFO:teuthology.orchestra.run.smithi191.stdout:================================================================================ 2024-09-10T18:59:59.537 INFO:teuthology.orchestra.run.smithi191.stdout:Remove 2 Packages 2024-09-10T18:59:59.537 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-10T18:59:59.537 INFO:teuthology.orchestra.run.smithi191.stdout:Freed space: 5.5 M 2024-09-10T18:59:59.537 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction check 2024-09-10T18:59:59.541 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction check succeeded. 2024-09-10T18:59:59.541 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction test 2024-09-10T18:59:59.567 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction test succeeded. 2024-09-10T18:59:59.568 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction 2024-09-10T18:59:59.602 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction test succeeded. 2024-09-10T18:59:59.602 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction 2024-09-10T18:59:59.716 INFO:teuthology.orchestra.run.smithi183.stdout: Preparing : 1/1 2024-09-10T18:59:59.755 INFO:teuthology.orchestra.run.smithi191.stdout: Preparing : 1/1 2024-09-10T18:59:59.982 INFO:teuthology.orchestra.run.smithi183.stdout: Erasing : nvme-cli-2.9.1-6.el9.x86_64 1/2 2024-09-10T18:59:59.982 INFO:teuthology.orchestra.run.smithi183.stdout: Erasing : libnvme-1.9-3.el9.x86_64 2/2 2024-09-10T18:59:59.982 INFO:teuthology.orchestra.run.smithi191.stdout: Erasing : nvme-cli-2.9.1-6.el9.x86_64 1/2 2024-09-10T18:59:59.982 INFO:teuthology.orchestra.run.smithi191.stdout: Erasing : libnvme-1.9-3.el9.x86_64 2/2 2024-09-10T19:00:00.325 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: libnvme-1.9-3.el9.x86_64 2/2 2024-09-10T19:00:00.325 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : libnvme-1.9-3.el9.x86_64 1/2 2024-09-10T19:00:00.391 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: libnvme-1.9-3.el9.x86_64 2/2 2024-09-10T19:00:00.392 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : libnvme-1.9-3.el9.x86_64 1/2 2024-09-10T19:00:00.562 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : nvme-cli-2.9.1-6.el9.x86_64 2/2 2024-09-10T19:00:00.562 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-10T19:00:00.563 INFO:teuthology.orchestra.run.smithi191.stdout:Removed: 2024-09-10T19:00:00.563 INFO:teuthology.orchestra.run.smithi191.stdout: libnvme-1.9-3.el9.x86_64 nvme-cli-2.9.1-6.el9.x86_64 2024-09-10T19:00:00.563 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-10T19:00:00.563 INFO:teuthology.orchestra.run.smithi191.stdout:Complete! 2024-09-10T19:00:00.597 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : nvme-cli-2.9.1-6.el9.x86_64 2/2 2024-09-10T19:00:00.597 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-09-10T19:00:00.597 INFO:teuthology.orchestra.run.smithi183.stdout:Removed: 2024-09-10T19:00:00.597 INFO:teuthology.orchestra.run.smithi183.stdout: libnvme-1.9-3.el9.x86_64 nvme-cli-2.9.1-6.el9.x86_64 2024-09-10T19:00:00.598 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-09-10T19:00:00.598 INFO:teuthology.orchestra.run.smithi183.stdout:Complete! 2024-09-10T19:00:01.389 INFO:teuthology.orchestra.run.smithi191.stdout:Last metadata expiration check: 0:00:46 ago on Tue 10 Sep 2024 06:59:15 PM UTC. 2024-09-10T19:00:01.401 INFO:teuthology.orchestra.run.smithi183.stdout:Last metadata expiration check: 0:00:47 ago on Tue 10 Sep 2024 06:59:14 PM UTC. 2024-09-10T19:00:01.524 INFO:teuthology.orchestra.run.smithi191.stdout:Dependencies resolved. 2024-09-10T19:00:01.525 INFO:teuthology.orchestra.run.smithi191.stdout:================================================================================ 2024-09-10T19:00:01.525 INFO:teuthology.orchestra.run.smithi191.stdout: Package Arch Version Repository Size 2024-09-10T19:00:01.525 INFO:teuthology.orchestra.run.smithi191.stdout:================================================================================ 2024-09-10T19:00:01.525 INFO:teuthology.orchestra.run.smithi191.stdout:Installing: 2024-09-10T19:00:01.525 INFO:teuthology.orchestra.run.smithi191.stdout: nvme-cli x86_64 2.9.1-6.el9 baseos 906 k 2024-09-10T19:00:01.525 INFO:teuthology.orchestra.run.smithi191.stdout: nvmetcli noarch 0.7-3.el9 baseos 44 k 2024-09-10T19:00:01.525 INFO:teuthology.orchestra.run.smithi191.stdout: runc x86_64 4:1.1.13-4.el9 appstream 3.2 M 2024-09-10T19:00:01.526 INFO:teuthology.orchestra.run.smithi191.stdout:Installing dependencies: 2024-09-10T19:00:01.526 INFO:teuthology.orchestra.run.smithi191.stdout: libnvme x86_64 1.9-3.el9 baseos 106 k 2024-09-10T19:00:01.526 INFO:teuthology.orchestra.run.smithi191.stdout: python3-configshell noarch 1:1.1.30-1.el9 baseos 72 k 2024-09-10T19:00:01.526 INFO:teuthology.orchestra.run.smithi191.stdout: python3-kmod x86_64 0.9-32.el9 baseos 84 k 2024-09-10T19:00:01.526 INFO:teuthology.orchestra.run.smithi191.stdout: python3-urwid x86_64 2.1.2-4.el9 baseos 837 k 2024-09-10T19:00:01.526 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-10T19:00:01.526 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction Summary 2024-09-10T19:00:01.526 INFO:teuthology.orchestra.run.smithi191.stdout:================================================================================ 2024-09-10T19:00:01.526 INFO:teuthology.orchestra.run.smithi191.stdout:Install 7 Packages 2024-09-10T19:00:01.526 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-10T19:00:01.534 INFO:teuthology.orchestra.run.smithi191.stdout:Total download size: 5.2 M 2024-09-10T19:00:01.534 INFO:teuthology.orchestra.run.smithi191.stdout:Installed size: 19 M 2024-09-10T19:00:01.534 INFO:teuthology.orchestra.run.smithi191.stdout:Downloading Packages: 2024-09-10T19:00:01.536 INFO:teuthology.orchestra.run.smithi183.stdout:Dependencies resolved. 2024-09-10T19:00:01.537 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2024-09-10T19:00:01.537 INFO:teuthology.orchestra.run.smithi183.stdout: Package Arch Version Repository Size 2024-09-10T19:00:01.537 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2024-09-10T19:00:01.537 INFO:teuthology.orchestra.run.smithi183.stdout:Installing: 2024-09-10T19:00:01.537 INFO:teuthology.orchestra.run.smithi183.stdout: nvme-cli x86_64 2.9.1-6.el9 baseos 906 k 2024-09-10T19:00:01.537 INFO:teuthology.orchestra.run.smithi183.stdout: nvmetcli noarch 0.7-3.el9 baseos 44 k 2024-09-10T19:00:01.537 INFO:teuthology.orchestra.run.smithi183.stdout: runc x86_64 4:1.1.13-4.el9 appstream 3.2 M 2024-09-10T19:00:01.537 INFO:teuthology.orchestra.run.smithi183.stdout:Installing dependencies: 2024-09-10T19:00:01.538 INFO:teuthology.orchestra.run.smithi183.stdout: libnvme x86_64 1.9-3.el9 baseos 106 k 2024-09-10T19:00:01.538 INFO:teuthology.orchestra.run.smithi183.stdout: python3-configshell noarch 1:1.1.30-1.el9 baseos 72 k 2024-09-10T19:00:01.538 INFO:teuthology.orchestra.run.smithi183.stdout: python3-kmod x86_64 0.9-32.el9 baseos 84 k 2024-09-10T19:00:01.538 INFO:teuthology.orchestra.run.smithi183.stdout: python3-urwid x86_64 2.1.2-4.el9 baseos 837 k 2024-09-10T19:00:01.538 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-09-10T19:00:01.538 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction Summary 2024-09-10T19:00:01.538 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2024-09-10T19:00:01.538 INFO:teuthology.orchestra.run.smithi183.stdout:Install 7 Packages 2024-09-10T19:00:01.538 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-09-10T19:00:01.550 INFO:teuthology.orchestra.run.smithi183.stdout:Total download size: 5.2 M 2024-09-10T19:00:01.550 INFO:teuthology.orchestra.run.smithi183.stdout:Installed size: 19 M 2024-09-10T19:00:01.550 INFO:teuthology.orchestra.run.smithi183.stdout:Downloading Packages: 2024-09-10T19:00:01.767 INFO:teuthology.orchestra.run.smithi183.stdout:(1/7): nvmetcli-0.7-3.el9.noarch.rpm 281 kB/s | 44 kB 00:00 2024-09-10T19:00:01.792 INFO:teuthology.orchestra.run.smithi183.stdout:(2/7): libnvme-1.9-3.el9.x86_64.rpm 579 kB/s | 106 kB 00:00 2024-09-10T19:00:01.851 INFO:teuthology.orchestra.run.smithi183.stdout:(3/7): python3-kmod-0.9-32.el9.x86_64.rpm 1.4 MB/s | 84 kB 00:00 2024-09-10T19:00:01.876 INFO:teuthology.orchestra.run.smithi183.stdout:(4/7): python3-configshell-1.1.30-1.el9.noarch. 659 kB/s | 72 kB 00:00 2024-09-10T19:00:01.901 INFO:teuthology.orchestra.run.smithi183.stdout:(5/7): nvme-cli-2.9.1-6.el9.x86_64.rpm 3.0 MB/s | 906 kB 00:00 2024-09-10T19:00:02.018 INFO:teuthology.orchestra.run.smithi183.stdout:(6/7): python3-urwid-2.1.2-4.el9.x86_64.rpm 4.9 MB/s | 837 kB 00:00 2024-09-10T19:00:02.185 INFO:teuthology.orchestra.run.smithi183.stdout:(7/7): runc-1.1.13-4.el9.x86_64.rpm 10 MB/s | 3.2 MB 00:00 2024-09-10T19:00:02.186 INFO:teuthology.orchestra.run.smithi183.stdout:-------------------------------------------------------------------------------- 2024-09-10T19:00:02.186 INFO:teuthology.orchestra.run.smithi183.stdout:Total 8.1 MB/s | 5.2 MB 00:00 2024-09-10T19:00:02.296 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction check 2024-09-10T19:00:02.303 INFO:teuthology.orchestra.run.smithi191.stdout:(1/7): nvmetcli-0.7-3.el9.noarch.rpm 116 kB/s | 44 kB 00:00 2024-09-10T19:00:02.311 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction check succeeded. 2024-09-10T19:00:02.311 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction test 2024-09-10T19:00:02.395 INFO:teuthology.orchestra.run.smithi191.stdout:(2/7): libnvme-1.9-3.el9.x86_64.rpm 223 kB/s | 106 kB 00:00 2024-09-10T19:00:02.504 INFO:teuthology.orchestra.run.smithi191.stdout:(3/7): python3-configshell-1.1.30-1.el9.noarch. 358 kB/s | 72 kB 00:00 2024-09-10T19:00:02.529 INFO:teuthology.orchestra.run.smithi191.stdout:(4/7): python3-kmod-0.9-32.el9.x86_64.rpm 631 kB/s | 84 kB 00:00 2024-09-10T19:00:02.657 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction test succeeded. 2024-09-10T19:00:02.657 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction 2024-09-10T19:00:02.737 INFO:teuthology.orchestra.run.smithi191.stdout:(5/7): nvme-cli-2.9.1-6.el9.x86_64.rpm 1.1 MB/s | 906 kB 00:00 2024-09-10T19:00:02.854 INFO:teuthology.orchestra.run.smithi191.stdout:(6/7): runc-1.1.13-4.el9.x86_64.rpm 9.7 MB/s | 3.2 MB 00:00 2024-09-10T19:00:02.904 INFO:teuthology.orchestra.run.smithi191.stdout:(7/7): python3-urwid-2.1.2-4.el9.x86_64.rpm 2.0 MB/s | 837 kB 00:00 2024-09-10T19:00:02.905 INFO:teuthology.orchestra.run.smithi191.stdout:-------------------------------------------------------------------------------- 2024-09-10T19:00:02.905 INFO:teuthology.orchestra.run.smithi191.stdout:Total 3.8 MB/s | 5.2 MB 00:01 2024-09-10T19:00:03.022 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction check 2024-09-10T19:00:03.037 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction check succeeded. 2024-09-10T19:00:03.037 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction test 2024-09-10T19:00:03.134 INFO:teuthology.orchestra.run.smithi183.stdout: Preparing : 1/1 2024-09-10T19:00:03.230 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-urwid-2.1.2-4.el9.x86_64 1/7 2024-09-10T19:00:03.265 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-configshell-1:1.1.30-1.el9.noarch 2/7 2024-09-10T19:00:03.293 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-kmod-0.9-32.el9.x86_64 3/7 2024-09-10T19:00:03.333 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction test succeeded. 2024-09-10T19:00:03.333 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction 2024-09-10T19:00:03.509 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : libnvme-1.9-3.el9.x86_64 4/7 2024-09-10T19:00:03.549 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : nvme-cli-2.9.1-6.el9.x86_64 5/7 2024-09-10T19:00:03.832 INFO:teuthology.orchestra.run.smithi191.stdout: Preparing : 1/1 2024-09-10T19:00:03.932 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-urwid-2.1.2-4.el9.x86_64 1/7 2024-09-10T19:00:03.967 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-configshell-1:1.1.30-1.el9.noarch 2/7 2024-09-10T19:00:03.984 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: nvme-cli-2.9.1-6.el9.x86_64 5/7 2024-09-10T19:00:03.995 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-kmod-0.9-32.el9.x86_64 3/7 2024-09-10T19:00:04.010 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : nvmetcli-0.7-3.el9.noarch 6/7 2024-09-10T19:00:04.129 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: nvmetcli-0.7-3.el9.noarch 6/7 2024-09-10T19:00:04.197 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : runc-4:1.1.13-4.el9.x86_64 7/7 2024-09-10T19:00:04.200 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : libnvme-1.9-3.el9.x86_64 4/7 2024-09-10T19:00:04.235 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : nvme-cli-2.9.1-6.el9.x86_64 5/7 2024-09-10T19:00:04.684 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: nvme-cli-2.9.1-6.el9.x86_64 5/7 2024-09-10T19:00:04.712 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : nvmetcli-0.7-3.el9.noarch 6/7 2024-09-10T19:00:04.827 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: nvmetcli-0.7-3.el9.noarch 6/7 2024-09-10T19:00:04.891 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : runc-4:1.1.13-4.el9.x86_64 7/7 2024-09-10T19:00:05.138 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: runc-4:1.1.13-4.el9.x86_64 7/7 2024-09-10T19:00:05.139 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : libnvme-1.9-3.el9.x86_64 1/7 2024-09-10T19:00:05.139 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : nvme-cli-2.9.1-6.el9.x86_64 2/7 2024-09-10T19:00:05.139 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : nvmetcli-0.7-3.el9.noarch 3/7 2024-09-10T19:00:05.139 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-configshell-1:1.1.30-1.el9.noarch 4/7 2024-09-10T19:00:05.139 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-kmod-0.9-32.el9.x86_64 5/7 2024-09-10T19:00:05.139 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-urwid-2.1.2-4.el9.x86_64 6/7 2024-09-10T19:00:05.481 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : runc-4:1.1.13-4.el9.x86_64 7/7 2024-09-10T19:00:05.481 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-09-10T19:00:05.481 INFO:teuthology.orchestra.run.smithi183.stdout:Installed: 2024-09-10T19:00:05.481 INFO:teuthology.orchestra.run.smithi183.stdout: libnvme-1.9-3.el9.x86_64 nvme-cli-2.9.1-6.el9.x86_64 2024-09-10T19:00:05.482 INFO:teuthology.orchestra.run.smithi183.stdout: nvmetcli-0.7-3.el9.noarch python3-configshell-1:1.1.30-1.el9.noarch 2024-09-10T19:00:05.482 INFO:teuthology.orchestra.run.smithi183.stdout: python3-kmod-0.9-32.el9.x86_64 python3-urwid-2.1.2-4.el9.x86_64 2024-09-10T19:00:05.482 INFO:teuthology.orchestra.run.smithi183.stdout: runc-4:1.1.13-4.el9.x86_64 2024-09-10T19:00:05.482 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-09-10T19:00:05.482 INFO:teuthology.orchestra.run.smithi183.stdout:Complete! 2024-09-10T19:00:05.805 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: runc-4:1.1.13-4.el9.x86_64 7/7 2024-09-10T19:00:05.805 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : libnvme-1.9-3.el9.x86_64 1/7 2024-09-10T19:00:05.805 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : nvme-cli-2.9.1-6.el9.x86_64 2/7 2024-09-10T19:00:05.805 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : nvmetcli-0.7-3.el9.noarch 3/7 2024-09-10T19:00:05.805 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-configshell-1:1.1.30-1.el9.noarch 4/7 2024-09-10T19:00:05.806 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-kmod-0.9-32.el9.x86_64 5/7 2024-09-10T19:00:05.806 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-urwid-2.1.2-4.el9.x86_64 6/7 2024-09-10T19:00:05.852 DEBUG:teuthology.parallel:result is None 2024-09-10T19:00:06.171 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : runc-4:1.1.13-4.el9.x86_64 7/7 2024-09-10T19:00:06.171 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-10T19:00:06.172 INFO:teuthology.orchestra.run.smithi191.stdout:Installed: 2024-09-10T19:00:06.172 INFO:teuthology.orchestra.run.smithi191.stdout: libnvme-1.9-3.el9.x86_64 nvme-cli-2.9.1-6.el9.x86_64 2024-09-10T19:00:06.172 INFO:teuthology.orchestra.run.smithi191.stdout: nvmetcli-0.7-3.el9.noarch python3-configshell-1:1.1.30-1.el9.noarch 2024-09-10T19:00:06.172 INFO:teuthology.orchestra.run.smithi191.stdout: python3-kmod-0.9-32.el9.x86_64 python3-urwid-2.1.2-4.el9.x86_64 2024-09-10T19:00:06.172 INFO:teuthology.orchestra.run.smithi191.stdout: runc-4:1.1.13-4.el9.x86_64 2024-09-10T19:00:06.172 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-10T19:00:06.172 INFO:teuthology.orchestra.run.smithi191.stdout:Complete! 2024-09-10T19:00:06.533 DEBUG:teuthology.parallel:result is None 2024-09-10T19:00:06.533 INFO:teuthology.run_tasks:Running task install... 2024-09-10T19:00:06.542 DEBUG:teuthology.task.install:project ceph 2024-09-10T19:00:06.542 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '5be57395ea235535910512ae73ba8e7ff25fe588'}} 2024-09-10T19:00:06.542 DEBUG:teuthology.task.install:config {'flavor': 'default', 'sha1': '5be57395ea235535910512ae73ba8e7ff25fe588'} 2024-09-10T19:00:06.542 INFO:teuthology.task.install:Using flavor: default 2024-09-10T19:00:06.547 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-09-10T19:00:06.547 INFO:teuthology.task.install:extra packages: [] 2024-09-10T19:00:06.547 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '5be57395ea235535910512ae73ba8e7ff25fe588', 'tag': None, 'wait_for_package': False} 2024-09-10T19:00:06.548 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5be57395ea235535910512ae73ba8e7ff25fe588 2024-09-10T19:00:06.550 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '5be57395ea235535910512ae73ba8e7ff25fe588', 'tag': None, 'wait_for_package': False} 2024-09-10T19:00:06.550 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5be57395ea235535910512ae73ba8e7ff25fe588 2024-09-10T19:00:06.692 INFO:teuthology.task.install.rpm:Pulling from https://4.chacra.ceph.com/r/ceph/wip-guits-quincy-2024-09-09-1810-quincy/5be57395ea235535910512ae73ba8e7ff25fe588/centos/9/flavors/default/ 2024-09-10T19:00:06.692 INFO:teuthology.task.install.rpm:Package version is 17.2.7-1606.g5be57395 2024-09-10T19:00:06.702 INFO:teuthology.task.install.rpm:Pulling from https://4.chacra.ceph.com/r/ceph/wip-guits-quincy-2024-09-09-1810-quincy/5be57395ea235535910512ae73ba8e7ff25fe588/centos/9/flavors/default/ 2024-09-10T19:00:06.702 INFO:teuthology.task.install.rpm:Package version is 17.2.7-1606.g5be57395 2024-09-10T19:00:06.820 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://4.chacra.ceph.com/r/ceph/wip-guits-quincy-2024-09-09-1810-quincy/5be57395ea235535910512ae73ba8e7ff25fe588/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-guits-quincy-2024-09-09-1810-quincy/5be57395ea235535910512ae73ba8e7ff25fe588/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-guits-quincy-2024-09-09-1810-quincy/5be57395ea235535910512ae73ba8e7ff25fe588/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-09-10T19:00:06.820 DEBUG:teuthology.orchestra.run.smithi183:> set -ex 2024-09-10T19:00:06.820 DEBUG:teuthology.orchestra.run.smithi183:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-09-10T19:00:06.826 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://4.chacra.ceph.com/r/ceph/wip-guits-quincy-2024-09-09-1810-quincy/5be57395ea235535910512ae73ba8e7ff25fe588/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-guits-quincy-2024-09-09-1810-quincy/5be57395ea235535910512ae73ba8e7ff25fe588/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-guits-quincy-2024-09-09-1810-quincy/5be57395ea235535910512ae73ba8e7ff25fe588/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-09-10T19:00:06.826 DEBUG:teuthology.orchestra.run.smithi191:> set -ex 2024-09-10T19:00:06.826 DEBUG:teuthology.orchestra.run.smithi191:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-09-10T19:00:06.850 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-09-10T19:00:06.851 DEBUG:teuthology.orchestra.run.smithi183:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/5be57395ea235535910512ae73ba8e7ff25fe588/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-09-10T19:00:06.853 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-09-10T19:00:06.853 DEBUG:teuthology.orchestra.run.smithi191:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/5be57395ea235535910512ae73ba8e7ff25fe588/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-09-10T19:00:06.920 DEBUG:teuthology.orchestra.run.smithi191:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-09-10T19:00:06.921 DEBUG:teuthology.orchestra.run.smithi183:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-09-10T19:00:06.999 DEBUG:teuthology.orchestra.run.smithi191:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-09-10T19:00:07.003 DEBUG:teuthology.orchestra.run.smithi183:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-09-10T19:00:07.071 INFO:teuthology.orchestra.run.smithi183.stdout:check_obsoletes = 1 2024-09-10T19:00:07.073 DEBUG:teuthology.orchestra.run.smithi183:> sudo yum clean all 2024-09-10T19:00:07.075 INFO:teuthology.orchestra.run.smithi191.stdout:check_obsoletes = 1 2024-09-10T19:00:07.077 DEBUG:teuthology.orchestra.run.smithi191:> sudo yum clean all 2024-09-10T19:00:07.306 INFO:teuthology.orchestra.run.smithi191.stdout:45 files removed 2024-09-10T19:00:07.316 INFO:teuthology.orchestra.run.smithi183.stdout:45 files removed 2024-09-10T19:00:07.336 DEBUG:teuthology.orchestra.run.smithi191:> sudo yum -y install ceph-radosgw 2024-09-10T19:00:07.343 DEBUG:teuthology.orchestra.run.smithi183:> sudo yum -y install ceph-radosgw 2024-09-10T19:00:07.925 INFO:teuthology.orchestra.run.smithi183.stdout:ceph packages for x86_64 260 kB/s | 77 kB 00:00 2024-09-10T19:00:07.928 INFO:teuthology.orchestra.run.smithi191.stdout:ceph packages for x86_64 254 kB/s | 77 kB 00:00 2024-09-10T19:00:08.234 INFO:teuthology.orchestra.run.smithi183.stdout:ceph noarch packages 41 kB/s | 11 kB 00:00 2024-09-10T19:00:08.245 INFO:teuthology.orchestra.run.smithi191.stdout:ceph noarch packages 40 kB/s | 11 kB 00:00 2024-09-10T19:00:08.535 INFO:teuthology.orchestra.run.smithi183.stdout:ceph source packages 6.8 kB/s | 1.8 kB 00:00 2024-09-10T19:00:08.554 INFO:teuthology.orchestra.run.smithi191.stdout:ceph source packages 6.6 kB/s | 1.8 kB 00:00 2024-09-10T19:00:10.426 INFO:teuthology.orchestra.run.smithi183.stdout:CentOS Stream 9 - BaseOS 4.4 MB/s | 8.2 MB 00:01 2024-09-10T19:00:10.470 INFO:teuthology.orchestra.run.smithi191.stdout:CentOS Stream 9 - BaseOS 4.4 MB/s | 8.2 MB 00:01 2024-09-10T19:00:12.252 INFO:teuthology.orchestra.run.smithi183.stdout:CentOS Stream 9 - AppStream 29 MB/s | 20 MB 00:00 2024-09-10T19:00:12.497 INFO:teuthology.orchestra.run.smithi191.stdout:CentOS Stream 9 - AppStream 22 MB/s | 20 MB 00:00 2024-09-10T19:00:16.778 INFO:teuthology.orchestra.run.smithi183.stdout:CentOS Stream 9 - CRB 9.1 MB/s | 6.5 MB 00:00 2024-09-10T19:00:16.922 INFO:teuthology.orchestra.run.smithi191.stdout:CentOS Stream 9 - CRB 11 MB/s | 6.5 MB 00:00 2024-09-10T19:00:18.445 INFO:teuthology.orchestra.run.smithi183.stdout:CentOS Stream 9 - Extras packages 62 kB/s | 19 kB 00:00 2024-09-10T19:00:19.275 INFO:teuthology.orchestra.run.smithi183.stdout:Extra Packages for Enterprise Linux 32 MB/s | 23 MB 00:00 2024-09-10T19:00:19.289 INFO:teuthology.orchestra.run.smithi191.stdout:CentOS Stream 9 - Extras packages 19 kB/s | 19 kB 00:00 2024-09-10T19:00:20.085 INFO:teuthology.orchestra.run.smithi191.stdout:Extra Packages for Enterprise Linux 34 MB/s | 23 MB 00:00 2024-09-10T19:00:24.689 INFO:teuthology.orchestra.run.smithi183.stdout:lab-extras 29 kB/s | 1.7 kB 00:00 2024-09-10T19:00:25.566 INFO:teuthology.orchestra.run.smithi191.stdout:lab-extras 46 kB/s | 1.7 kB 00:00 2024-09-10T19:00:27.080 INFO:teuthology.orchestra.run.smithi183.stdout:Dependencies resolved. 2024-09-10T19:00:27.082 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2024-09-10T19:00:27.082 INFO:teuthology.orchestra.run.smithi183.stdout: Package Arch Version Repository Size 2024-09-10T19:00:27.082 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2024-09-10T19:00:27.082 INFO:teuthology.orchestra.run.smithi183.stdout:Installing: 2024-09-10T19:00:27.082 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-radosgw x86_64 2:17.2.7-1606.g5be57395.el9 ceph 14 M 2024-09-10T19:00:27.082 INFO:teuthology.orchestra.run.smithi183.stdout:Upgrading: 2024-09-10T19:00:27.082 INFO:teuthology.orchestra.run.smithi183.stdout: librados2 x86_64 2:17.2.7-1606.g5be57395.el9 ceph 3.1 M 2024-09-10T19:00:27.082 INFO:teuthology.orchestra.run.smithi183.stdout: librbd1 x86_64 2:17.2.7-1606.g5be57395.el9 ceph 2.9 M 2024-09-10T19:00:27.082 INFO:teuthology.orchestra.run.smithi183.stdout:Installing dependencies: 2024-09-10T19:00:27.082 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-base x86_64 2:17.2.7-1606.g5be57395.el9 ceph 5.0 M 2024-09-10T19:00:27.082 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-common x86_64 2:17.2.7-1606.g5be57395.el9 ceph 22 M 2024-09-10T19:00:27.083 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-selinux x86_64 2:17.2.7-1606.g5be57395.el9 ceph 25 k 2024-09-10T19:00:27.083 INFO:teuthology.orchestra.run.smithi183.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-09-10T19:00:27.083 INFO:teuthology.orchestra.run.smithi183.stdout: libcephfs2 x86_64 2:17.2.7-1606.g5be57395.el9 ceph 637 k 2024-09-10T19:00:27.083 INFO:teuthology.orchestra.run.smithi183.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-09-10T19:00:27.083 INFO:teuthology.orchestra.run.smithi183.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-09-10T19:00:27.083 INFO:teuthology.orchestra.run.smithi183.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-09-10T19:00:27.083 INFO:teuthology.orchestra.run.smithi183.stdout: libradosstriper1 x86_64 2:17.2.7-1606.g5be57395.el9 ceph 458 k 2024-09-10T19:00:27.083 INFO:teuthology.orchestra.run.smithi183.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-09-10T19:00:27.083 INFO:teuthology.orchestra.run.smithi183.stdout: librgw2 x86_64 2:17.2.7-1606.g5be57395.el9 ceph 4.9 M 2024-09-10T19:00:27.083 INFO:teuthology.orchestra.run.smithi183.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-09-10T19:00:27.083 INFO:teuthology.orchestra.run.smithi183.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-09-10T19:00:27.083 INFO:teuthology.orchestra.run.smithi183.stdout: python3-ceph-argparse x86_64 2:17.2.7-1606.g5be57395.el9 ceph 45 k 2024-09-10T19:00:27.083 INFO:teuthology.orchestra.run.smithi183.stdout: python3-ceph-common x86_64 2:17.2.7-1606.g5be57395.el9 ceph 120 k 2024-09-10T19:00:27.083 INFO:teuthology.orchestra.run.smithi183.stdout: python3-cephfs x86_64 2:17.2.7-1606.g5be57395.el9 ceph 156 k 2024-09-10T19:00:27.084 INFO:teuthology.orchestra.run.smithi183.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-09-10T19:00:27.084 INFO:teuthology.orchestra.run.smithi183.stdout: python3-rados x86_64 2:17.2.7-1606.g5be57395.el9 ceph 320 k 2024-09-10T19:00:27.084 INFO:teuthology.orchestra.run.smithi183.stdout: python3-rbd x86_64 2:17.2.7-1606.g5be57395.el9 ceph 295 k 2024-09-10T19:00:27.084 INFO:teuthology.orchestra.run.smithi183.stdout: python3-rgw x86_64 2:17.2.7-1606.g5be57395.el9 ceph 99 k 2024-09-10T19:00:27.084 INFO:teuthology.orchestra.run.smithi183.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-09-10T19:00:27.084 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-09-10T19:00:27.084 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction Summary 2024-09-10T19:00:27.084 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2024-09-10T19:00:27.084 INFO:teuthology.orchestra.run.smithi183.stdout:Install 22 Packages 2024-09-10T19:00:27.084 INFO:teuthology.orchestra.run.smithi183.stdout:Upgrade 2 Packages 2024-09-10T19:00:27.084 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-09-10T19:00:27.085 INFO:teuthology.orchestra.run.smithi183.stdout:Total download size: 58 M 2024-09-10T19:00:27.085 INFO:teuthology.orchestra.run.smithi183.stdout:Downloading Packages: 2024-09-10T19:00:27.848 INFO:teuthology.orchestra.run.smithi183.stdout:(1/24): ceph-base-17.2.7-1606.g5be57395.el9.x86 7.3 MB/s | 5.0 MB 00:00 2024-09-10T19:00:27.898 INFO:teuthology.orchestra.run.smithi183.stdout:(2/24): ceph-selinux-17.2.7-1606.g5be57395.el9. 505 kB/s | 25 kB 00:00 2024-09-10T19:00:27.946 INFO:teuthology.orchestra.run.smithi191.stdout:Dependencies resolved. 2024-09-10T19:00:27.948 INFO:teuthology.orchestra.run.smithi191.stdout:================================================================================ 2024-09-10T19:00:27.949 INFO:teuthology.orchestra.run.smithi191.stdout: Package Arch Version Repository Size 2024-09-10T19:00:27.949 INFO:teuthology.orchestra.run.smithi191.stdout:================================================================================ 2024-09-10T19:00:27.949 INFO:teuthology.orchestra.run.smithi191.stdout:Installing: 2024-09-10T19:00:27.949 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-radosgw x86_64 2:17.2.7-1606.g5be57395.el9 ceph 14 M 2024-09-10T19:00:27.949 INFO:teuthology.orchestra.run.smithi191.stdout:Upgrading: 2024-09-10T19:00:27.949 INFO:teuthology.orchestra.run.smithi191.stdout: librados2 x86_64 2:17.2.7-1606.g5be57395.el9 ceph 3.1 M 2024-09-10T19:00:27.949 INFO:teuthology.orchestra.run.smithi191.stdout: librbd1 x86_64 2:17.2.7-1606.g5be57395.el9 ceph 2.9 M 2024-09-10T19:00:27.949 INFO:teuthology.orchestra.run.smithi191.stdout:Installing dependencies: 2024-09-10T19:00:27.949 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-base x86_64 2:17.2.7-1606.g5be57395.el9 ceph 5.0 M 2024-09-10T19:00:27.949 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-common x86_64 2:17.2.7-1606.g5be57395.el9 ceph 22 M 2024-09-10T19:00:27.949 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-selinux x86_64 2:17.2.7-1606.g5be57395.el9 ceph 25 k 2024-09-10T19:00:27.949 INFO:teuthology.orchestra.run.smithi191.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-09-10T19:00:27.949 INFO:teuthology.orchestra.run.smithi191.stdout: libcephfs2 x86_64 2:17.2.7-1606.g5be57395.el9 ceph 637 k 2024-09-10T19:00:27.950 INFO:teuthology.orchestra.run.smithi191.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-09-10T19:00:27.950 INFO:teuthology.orchestra.run.smithi191.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-09-10T19:00:27.950 INFO:teuthology.orchestra.run.smithi191.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-09-10T19:00:27.950 INFO:teuthology.orchestra.run.smithi191.stdout: libradosstriper1 x86_64 2:17.2.7-1606.g5be57395.el9 ceph 458 k 2024-09-10T19:00:27.950 INFO:teuthology.orchestra.run.smithi191.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-09-10T19:00:27.950 INFO:teuthology.orchestra.run.smithi191.stdout: librgw2 x86_64 2:17.2.7-1606.g5be57395.el9 ceph 4.9 M 2024-09-10T19:00:27.950 INFO:teuthology.orchestra.run.smithi191.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-09-10T19:00:27.950 INFO:teuthology.orchestra.run.smithi191.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-09-10T19:00:27.950 INFO:teuthology.orchestra.run.smithi191.stdout: python3-ceph-argparse x86_64 2:17.2.7-1606.g5be57395.el9 ceph 45 k 2024-09-10T19:00:27.950 INFO:teuthology.orchestra.run.smithi191.stdout: python3-ceph-common x86_64 2:17.2.7-1606.g5be57395.el9 ceph 120 k 2024-09-10T19:00:27.950 INFO:teuthology.orchestra.run.smithi191.stdout: python3-cephfs x86_64 2:17.2.7-1606.g5be57395.el9 ceph 156 k 2024-09-10T19:00:27.950 INFO:teuthology.orchestra.run.smithi191.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-09-10T19:00:27.950 INFO:teuthology.orchestra.run.smithi191.stdout: python3-rados x86_64 2:17.2.7-1606.g5be57395.el9 ceph 320 k 2024-09-10T19:00:27.950 INFO:teuthology.orchestra.run.smithi191.stdout: python3-rbd x86_64 2:17.2.7-1606.g5be57395.el9 ceph 295 k 2024-09-10T19:00:27.951 INFO:teuthology.orchestra.run.smithi191.stdout: python3-rgw x86_64 2:17.2.7-1606.g5be57395.el9 ceph 99 k 2024-09-10T19:00:27.951 INFO:teuthology.orchestra.run.smithi191.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-09-10T19:00:27.951 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-10T19:00:27.951 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction Summary 2024-09-10T19:00:27.951 INFO:teuthology.orchestra.run.smithi191.stdout:================================================================================ 2024-09-10T19:00:27.951 INFO:teuthology.orchestra.run.smithi191.stdout:Install 22 Packages 2024-09-10T19:00:27.951 INFO:teuthology.orchestra.run.smithi191.stdout:Upgrade 2 Packages 2024-09-10T19:00:27.951 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-10T19:00:27.951 INFO:teuthology.orchestra.run.smithi191.stdout:Total download size: 58 M 2024-09-10T19:00:27.951 INFO:teuthology.orchestra.run.smithi191.stdout:Downloading Packages: 2024-09-10T19:00:27.973 INFO:teuthology.orchestra.run.smithi183.stdout:(3/24): libcephfs2-17.2.7-1606.g5be57395.el9.x8 8.3 MB/s | 637 kB 00:00 2024-09-10T19:00:28.049 INFO:teuthology.orchestra.run.smithi183.stdout:(4/24): libradosstriper1-17.2.7-1606.g5be57395. 6.0 MB/s | 458 kB 00:00 2024-09-10T19:00:28.424 INFO:teuthology.orchestra.run.smithi183.stdout:(5/24): librgw2-17.2.7-1606.g5be57395.el9.x86_6 13 MB/s | 4.9 MB 00:00 2024-09-10T19:00:28.583 INFO:teuthology.orchestra.run.smithi183.stdout:(6/24): ceph-radosgw-17.2.7-1606.g5be57395.el9. 10 MB/s | 14 MB 00:01 2024-09-10T19:00:28.608 INFO:teuthology.orchestra.run.smithi183.stdout:(7/24): python3-ceph-argparse-17.2.7-1606.g5be5 246 kB/s | 45 kB 00:00 2024-09-10T19:00:28.692 INFO:teuthology.orchestra.run.smithi183.stdout:(8/24): python3-ceph-common-17.2.7-1606.g5be573 1.1 MB/s | 120 kB 00:00 2024-09-10T19:00:28.775 INFO:teuthology.orchestra.run.smithi183.stdout:(9/24): python3-rados-17.2.7-1606.g5be57395.el9 3.8 MB/s | 320 kB 00:00 2024-09-10T19:00:28.825 INFO:teuthology.orchestra.run.smithi183.stdout:(10/24): python3-rbd-17.2.7-1606.g5be57395.el9. 5.7 MB/s | 295 kB 00:00 2024-09-10T19:00:28.842 INFO:teuthology.orchestra.run.smithi191.stdout:(1/24): ceph-base-17.2.7-1606.g5be57395.el9.x86 8.6 MB/s | 5.0 MB 00:00 2024-09-10T19:00:28.900 INFO:teuthology.orchestra.run.smithi191.stdout:(2/24): ceph-selinux-17.2.7-1606.g5be57395.el9. 432 kB/s | 25 kB 00:00 2024-09-10T19:00:28.984 INFO:teuthology.orchestra.run.smithi191.stdout:(3/24): libcephfs2-17.2.7-1606.g5be57395.el9.x8 7.5 MB/s | 637 kB 00:00 2024-09-10T19:00:29.051 INFO:teuthology.orchestra.run.smithi191.stdout:(4/24): libradosstriper1-17.2.7-1606.g5be57395. 6.7 MB/s | 458 kB 00:00 2024-09-10T19:00:29.059 INFO:teuthology.orchestra.run.smithi183.stdout:(11/24): ceph-common-17.2.7-1606.g5be57395.el9. 12 MB/s | 22 MB 00:01 2024-09-10T19:00:29.084 INFO:teuthology.orchestra.run.smithi183.stdout:(12/24): python3-cephfs-17.2.7-1606.g5be57395.e 328 kB/s | 156 kB 00:00 2024-09-10T19:00:29.109 INFO:teuthology.orchestra.run.smithi183.stdout:(13/24): python3-rgw-17.2.7-1606.g5be57395.el9. 350 kB/s | 99 kB 00:00 2024-09-10T19:00:29.226 INFO:teuthology.orchestra.run.smithi183.stdout:(14/24): librabbitmq-0.11.0-7.el9.x86_64.rpm 319 kB/s | 45 kB 00:00 2024-09-10T19:00:29.243 INFO:teuthology.orchestra.run.smithi191.stdout:(5/24): ceph-radosgw-17.2.7-1606.g5be57395.el9. 15 MB/s | 14 MB 00:00 2024-09-10T19:00:29.285 INFO:teuthology.orchestra.run.smithi191.stdout:(6/24): python3-ceph-argparse-17.2.7-1606.g5be5 1.1 MB/s | 45 kB 00:00 2024-09-10T19:00:29.285 INFO:teuthology.orchestra.run.smithi183.stdout:(15/24): libpmemobj-1.12.1-1.el9.x86_64.rpm 711 kB/s | 160 kB 00:00 2024-09-10T19:00:29.344 INFO:teuthology.orchestra.run.smithi183.stdout:(16/24): lttng-ust-2.12.0-6.el9.x86_64.rpm 2.4 MB/s | 292 kB 00:00 2024-09-10T19:00:29.360 INFO:teuthology.orchestra.run.smithi191.stdout:(7/24): librgw2-17.2.7-1606.g5be57395.el9.x86_6 16 MB/s | 4.9 MB 00:00 2024-09-10T19:00:29.369 INFO:teuthology.orchestra.run.smithi183.stdout:(17/24): python3-prettytable-0.7.2-27.el9.noarc 505 kB/s | 42 kB 00:00 2024-09-10T19:00:29.394 INFO:teuthology.orchestra.run.smithi191.stdout:(8/24): python3-ceph-common-17.2.7-1606.g5be573 1.1 MB/s | 120 kB 00:00 2024-09-10T19:00:29.419 INFO:teuthology.orchestra.run.smithi183.stdout:(18/24): gperftools-libs-2.9.1-2.el9.x86_64.rpm 4.0 MB/s | 309 kB 00:00 2024-09-10T19:00:29.419 INFO:teuthology.orchestra.run.smithi191.stdout:(9/24): python3-cephfs-17.2.7-1606.g5be57395.el 2.6 MB/s | 156 kB 00:00 2024-09-10T19:00:29.444 INFO:teuthology.orchestra.run.smithi191.stdout:(10/24): python3-rados-17.2.7-1606.g5be57395.el 6.2 MB/s | 320 kB 00:00 2024-09-10T19:00:29.445 INFO:teuthology.orchestra.run.smithi183.stdout:(19/24): liboath-2.6.7-2.el9.x86_64.rpm 646 kB/s | 49 kB 00:00 2024-09-10T19:00:29.470 INFO:teuthology.orchestra.run.smithi183.stdout:(20/24): librdkafka-1.6.1-102.el9.x86_64.rpm 1.8 MB/s | 662 kB 00:00 2024-09-10T19:00:29.470 INFO:teuthology.orchestra.run.smithi191.stdout:(11/24): python3-rbd-17.2.7-1606.g5be57395.el9. 5.7 MB/s | 295 kB 00:00 2024-09-10T19:00:29.495 INFO:teuthology.orchestra.run.smithi191.stdout:(12/24): python3-rgw-17.2.7-1606.g5be57395.el9. 1.9 MB/s | 99 kB 00:00 2024-09-10T19:00:29.495 INFO:teuthology.orchestra.run.smithi183.stdout:(21/24): libunwind-1.6.2-1.el9.x86_64.rpm 891 kB/s | 67 kB 00:00 2024-09-10T19:00:29.554 INFO:teuthology.orchestra.run.smithi183.stdout:(22/24): thrift-0.15.0-2.el9.x86_64.rpm 15 MB/s | 1.6 MB 00:00 2024-09-10T19:00:29.712 INFO:teuthology.orchestra.run.smithi183.stdout:(23/24): librbd1-17.2.7-1606.g5be57395.el9.x86_ 14 MB/s | 2.9 MB 00:00 2024-09-10T19:00:29.762 INFO:teuthology.orchestra.run.smithi183.stdout:(24/24): librados2-17.2.7-1606.g5be57395.el9.x8 11 MB/s | 3.1 MB 00:00 2024-09-10T19:00:29.767 INFO:teuthology.orchestra.run.smithi183.stdout:-------------------------------------------------------------------------------- 2024-09-10T19:00:29.767 INFO:teuthology.orchestra.run.smithi183.stdout:Total 22 MB/s | 58 MB 00:02 2024-09-10T19:00:29.832 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction check 2024-09-10T19:00:29.865 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction check succeeded. 2024-09-10T19:00:29.865 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction test 2024-09-10T19:00:30.062 INFO:teuthology.orchestra.run.smithi191.stdout:(13/24): librabbitmq-0.11.0-7.el9.x86_64.rpm 80 kB/s | 45 kB 00:00 2024-09-10T19:00:30.296 INFO:teuthology.orchestra.run.smithi191.stdout:(14/24): ceph-common-17.2.7-1606.g5be57395.el9. 11 MB/s | 22 MB 00:02 2024-09-10T19:00:30.321 INFO:teuthology.orchestra.run.smithi191.stdout:(15/24): libpmemobj-1.12.1-1.el9.x86_64.rpm 188 kB/s | 160 kB 00:00 2024-09-10T19:00:30.346 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction test succeeded. 2024-09-10T19:00:30.346 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction 2024-09-10T19:00:30.888 INFO:teuthology.orchestra.run.smithi191.stdout:(16/24): python3-prettytable-0.7.2-27.el9.noarc 75 kB/s | 42 kB 00:00 2024-09-10T19:00:30.955 INFO:teuthology.orchestra.run.smithi191.stdout:(17/24): gperftools-libs-2.9.1-2.el9.x86_64.rpm 4.6 MB/s | 309 kB 00:00 2024-09-10T19:00:30.980 INFO:teuthology.orchestra.run.smithi191.stdout:(18/24): liboath-2.6.7-2.el9.x86_64.rpm 1.9 MB/s | 49 kB 00:00 2024-09-10T19:00:31.005 INFO:teuthology.orchestra.run.smithi191.stdout:(19/24): libunwind-1.6.2-1.el9.x86_64.rpm 2.6 MB/s | 67 kB 00:00 2024-09-10T19:00:31.064 INFO:teuthology.orchestra.run.smithi191.stdout:(20/24): thrift-0.15.0-2.el9.x86_64.rpm 27 MB/s | 1.6 MB 00:00 2024-09-10T19:00:31.114 INFO:teuthology.orchestra.run.smithi191.stdout:(21/24): librdkafka-1.6.1-102.el9.x86_64.rpm 630 kB/s | 662 kB 00:01 2024-09-10T19:00:31.164 INFO:teuthology.orchestra.run.smithi191.stdout:(22/24): lttng-ust-2.12.0-6.el9.x86_64.rpm 337 kB/s | 292 kB 00:00 2024-09-10T19:00:31.314 INFO:teuthology.orchestra.run.smithi191.stdout:(23/24): librbd1-17.2.7-1606.g5be57395.el9.x86_ 15 MB/s | 2.9 MB 00:00 2024-09-10T19:00:31.506 INFO:teuthology.orchestra.run.smithi191.stdout:(24/24): librados2-17.2.7-1606.g5be57395.el9.x8 7.0 MB/s | 3.1 MB 00:00 2024-09-10T19:00:31.511 INFO:teuthology.orchestra.run.smithi191.stdout:-------------------------------------------------------------------------------- 2024-09-10T19:00:31.511 INFO:teuthology.orchestra.run.smithi191.stdout:Total 16 MB/s | 58 MB 00:03 2024-09-10T19:00:31.560 INFO:teuthology.orchestra.run.smithi183.stdout: Preparing : 1/1 2024-09-10T19:00:31.577 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction check 2024-09-10T19:00:31.611 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction check succeeded. 2024-09-10T19:00:31.611 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction test 2024-09-10T19:00:31.951 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/26 2024-09-10T19:00:32.023 INFO:teuthology.orchestra.run.smithi183.stdout: Upgrading : librados2-2:17.2.7-1606.g5be57395.el9.x86_64 2/26 2024-09-10T19:00:32.097 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction test succeeded. 2024-09-10T19:00:32.097 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction 2024-09-10T19:00:32.119 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: librados2-2:17.2.7-1606.g5be57395.el9.x86_64 2/26 2024-09-10T19:00:32.202 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-rados-2:17.2.7-1606.g5be57395.el9.x86_64 3/26 2024-09-10T19:00:32.233 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : libcephfs2-2:17.2.7-1606.g5be57395.el9.x86_64 4/26 2024-09-10T19:00:32.321 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: libcephfs2-2:17.2.7-1606.g5be57395.el9.x86_64 4/26 2024-09-10T19:00:32.367 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : thrift-0.15.0-2.el9.x86_64 5/26 2024-09-10T19:00:32.396 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 6/26 2024-09-10T19:00:32.835 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 7/26 2024-09-10T19:00:32.869 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : librgw2-2:17.2.7-1606.g5be57395.el9.x86_64 8/26 2024-09-10T19:00:32.932 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: librgw2-2:17.2.7-1606.g5be57395.el9.x86_64 8/26 2024-09-10T19:00:32.969 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : liboath-2.6.7-2.el9.x86_64 9/26 2024-09-10T19:00:33.012 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-ceph-argparse-2:17.2.7-1606.g5be57395.el9. 10/26 2024-09-10T19:00:33.049 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-cephfs-2:17.2.7-1606.g5be57395.el9.x86_64 11/26 2024-09-10T19:00:33.110 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-rgw-2:17.2.7-1606.g5be57395.el9.x86_64 12/26 2024-09-10T19:00:33.139 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : libradosstriper1-2:17.2.7-1606.g5be57395.el9.x86_6 13/26 2024-09-10T19:00:33.205 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: libradosstriper1-2:17.2.7-1606.g5be57395.el9.x86_6 13/26 2024-09-10T19:00:33.245 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 14/26 2024-09-10T19:00:33.280 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 15/26 2024-09-10T19:00:33.283 INFO:teuthology.orchestra.run.smithi191.stdout: Preparing : 1/1 2024-09-10T19:00:33.313 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 16/26 2024-09-10T19:00:33.605 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 17/26 2024-09-10T19:00:33.651 INFO:teuthology.orchestra.run.smithi183.stdout: Upgrading : librbd1-2:17.2.7-1606.g5be57395.el9.x86_64 18/26 2024-09-10T19:00:33.653 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/26 2024-09-10T19:00:33.741 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: librbd1-2:17.2.7-1606.g5be57395.el9.x86_64 18/26 2024-09-10T19:00:33.741 INFO:teuthology.orchestra.run.smithi191.stdout: Upgrading : librados2-2:17.2.7-1606.g5be57395.el9.x86_64 2/26 2024-09-10T19:00:33.814 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-rbd-2:17.2.7-1606.g5be57395.el9.x86_64 19/26 2024-09-10T19:00:33.817 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: librados2-2:17.2.7-1606.g5be57395.el9.x86_64 2/26 2024-09-10T19:00:33.845 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-ceph-common-2:17.2.7-1606.g5be57395.el9.x8 20/26 2024-09-10T19:00:33.895 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-rados-2:17.2.7-1606.g5be57395.el9.x86_64 3/26 2024-09-10T19:00:33.918 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : libcephfs2-2:17.2.7-1606.g5be57395.el9.x86_64 4/26 2024-09-10T19:00:34.005 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: libcephfs2-2:17.2.7-1606.g5be57395.el9.x86_64 4/26 2024-09-10T19:00:34.060 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : thrift-0.15.0-2.el9.x86_64 5/26 2024-09-10T19:00:34.094 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 6/26 2024-09-10T19:00:34.526 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 7/26 2024-09-10T19:00:34.554 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : librgw2-2:17.2.7-1606.g5be57395.el9.x86_64 8/26 2024-09-10T19:00:34.617 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: librgw2-2:17.2.7-1606.g5be57395.el9.x86_64 8/26 2024-09-10T19:00:34.645 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : liboath-2.6.7-2.el9.x86_64 9/26 2024-09-10T19:00:34.688 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-ceph-argparse-2:17.2.7-1606.g5be57395.el9. 10/26 2024-09-10T19:00:34.725 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-cephfs-2:17.2.7-1606.g5be57395.el9.x86_64 11/26 2024-09-10T19:00:34.787 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-rgw-2:17.2.7-1606.g5be57395.el9.x86_64 12/26 2024-09-10T19:00:34.815 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : libradosstriper1-2:17.2.7-1606.g5be57395.el9.x86_6 13/26 2024-09-10T19:00:34.880 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: libradosstriper1-2:17.2.7-1606.g5be57395.el9.x86_6 13/26 2024-09-10T19:00:34.922 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 14/26 2024-09-10T19:00:34.957 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 15/26 2024-09-10T19:00:34.997 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 16/26 2024-09-10T19:00:35.293 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 17/26 2024-09-10T19:00:35.344 INFO:teuthology.orchestra.run.smithi191.stdout: Upgrading : librbd1-2:17.2.7-1606.g5be57395.el9.x86_64 18/26 2024-09-10T19:00:35.442 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: librbd1-2:17.2.7-1606.g5be57395.el9.x86_64 18/26 2024-09-10T19:00:35.516 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-rbd-2:17.2.7-1606.g5be57395.el9.x86_64 19/26 2024-09-10T19:00:35.546 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-ceph-common-2:17.2.7-1606.g5be57395.el9.x8 20/26 2024-09-10T19:00:36.313 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: ceph-common-2:17.2.7-1606.g5be57395.el9.x86_64 21/26 2024-09-10T19:00:36.360 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : ceph-common-2:17.2.7-1606.g5be57395.el9.x86_64 21/26 2024-09-10T19:00:36.892 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: ceph-common-2:17.2.7-1606.g5be57395.el9.x86_64 21/26 2024-09-10T19:00:36.930 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : ceph-base-2:17.2.7-1606.g5be57395.el9.x86_64 22/26 2024-09-10T19:00:37.037 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: ceph-base-2:17.2.7-1606.g5be57395.el9.x86_64 22/26 2024-09-10T19:00:37.037 INFO:teuthology.orchestra.run.smithi183.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-09-10T19:00:37.037 INFO:teuthology.orchestra.run.smithi183.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-09-10T19:00:37.037 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-09-10T19:00:37.072 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : ceph-selinux-2:17.2.7-1606.g5be57395.el9.x86_64 23/26 2024-09-10T19:00:38.030 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: ceph-common-2:17.2.7-1606.g5be57395.el9.x86_64 21/26 2024-09-10T19:00:38.069 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : ceph-common-2:17.2.7-1606.g5be57395.el9.x86_64 21/26 2024-09-10T19:00:38.591 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: ceph-common-2:17.2.7-1606.g5be57395.el9.x86_64 21/26 2024-09-10T19:00:38.630 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : ceph-base-2:17.2.7-1606.g5be57395.el9.x86_64 22/26 2024-09-10T19:00:38.730 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: ceph-base-2:17.2.7-1606.g5be57395.el9.x86_64 22/26 2024-09-10T19:00:38.730 INFO:teuthology.orchestra.run.smithi191.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-09-10T19:00:38.730 INFO:teuthology.orchestra.run.smithi191.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-09-10T19:00:38.730 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-10T19:00:38.773 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : ceph-selinux-2:17.2.7-1606.g5be57395.el9.x86_64 23/26 2024-09-10T19:00:47.121 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: ceph-selinux-2:17.2.7-1606.g5be57395.el9.x86_64 23/26 2024-09-10T19:00:47.121 INFO:teuthology.orchestra.run.smithi183.stdout:skipping the directory /sys 2024-09-10T19:00:47.121 INFO:teuthology.orchestra.run.smithi183.stdout:skipping the directory /proc 2024-09-10T19:00:47.121 INFO:teuthology.orchestra.run.smithi183.stdout:skipping the directory /mnt 2024-09-10T19:00:47.121 INFO:teuthology.orchestra.run.smithi183.stdout:skipping the directory /var/tmp 2024-09-10T19:00:47.121 INFO:teuthology.orchestra.run.smithi183.stdout:skipping the directory /home 2024-09-10T19:00:47.121 INFO:teuthology.orchestra.run.smithi183.stdout:skipping the directory /root 2024-09-10T19:00:47.121 INFO:teuthology.orchestra.run.smithi183.stdout:skipping the directory /tmp 2024-09-10T19:00:47.121 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-09-10T19:00:48.421 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : ceph-radosgw-2:17.2.7-1606.g5be57395.el9.x86_64 24/26 2024-09-10T19:00:48.508 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: ceph-radosgw-2:17.2.7-1606.g5be57395.el9.x86_64 24/26 2024-09-10T19:00:48.508 INFO:teuthology.orchestra.run.smithi183.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-10T19:00:48.508 INFO:teuthology.orchestra.run.smithi183.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-09-10T19:00:48.508 INFO:teuthology.orchestra.run.smithi183.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-10T19:00:48.509 INFO:teuthology.orchestra.run.smithi183.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-10T19:00:48.509 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-09-10T19:00:48.510 INFO:teuthology.orchestra.run.smithi183.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 25/26 2024-09-10T19:00:48.607 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 25/26 2024-09-10T19:00:48.607 INFO:teuthology.orchestra.run.smithi183.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 26/26 2024-09-10T19:00:48.764 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: ceph-selinux-2:17.2.7-1606.g5be57395.el9.x86_64 23/26 2024-09-10T19:00:48.764 INFO:teuthology.orchestra.run.smithi191.stdout:skipping the directory /sys 2024-09-10T19:00:48.764 INFO:teuthology.orchestra.run.smithi191.stdout:skipping the directory /proc 2024-09-10T19:00:48.764 INFO:teuthology.orchestra.run.smithi191.stdout:skipping the directory /mnt 2024-09-10T19:00:48.764 INFO:teuthology.orchestra.run.smithi191.stdout:skipping the directory /var/tmp 2024-09-10T19:00:48.764 INFO:teuthology.orchestra.run.smithi191.stdout:skipping the directory /home 2024-09-10T19:00:48.764 INFO:teuthology.orchestra.run.smithi191.stdout:skipping the directory /root 2024-09-10T19:00:48.764 INFO:teuthology.orchestra.run.smithi191.stdout:skipping the directory /tmp 2024-09-10T19:00:48.764 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-10T19:00:49.888 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 26/26 2024-09-10T19:00:49.888 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : ceph-base-2:17.2.7-1606.g5be57395.el9.x86_64 1/26 2024-09-10T19:00:49.888 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : ceph-common-2:17.2.7-1606.g5be57395.el9.x86_64 2/26 2024-09-10T19:00:49.888 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : ceph-radosgw-2:17.2.7-1606.g5be57395.el9.x86_64 3/26 2024-09-10T19:00:49.888 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : ceph-selinux-2:17.2.7-1606.g5be57395.el9.x86_64 4/26 2024-09-10T19:00:49.888 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : libcephfs2-2:17.2.7-1606.g5be57395.el9.x86_64 5/26 2024-09-10T19:00:49.889 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : libradosstriper1-2:17.2.7-1606.g5be57395.el9.x86_6 6/26 2024-09-10T19:00:49.889 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : librgw2-2:17.2.7-1606.g5be57395.el9.x86_64 7/26 2024-09-10T19:00:49.889 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-ceph-argparse-2:17.2.7-1606.g5be57395.el9. 8/26 2024-09-10T19:00:49.889 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-ceph-common-2:17.2.7-1606.g5be57395.el9.x8 9/26 2024-09-10T19:00:49.889 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-cephfs-2:17.2.7-1606.g5be57395.el9.x86_64 10/26 2024-09-10T19:00:49.889 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-rados-2:17.2.7-1606.g5be57395.el9.x86_64 11/26 2024-09-10T19:00:49.889 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-rbd-2:17.2.7-1606.g5be57395.el9.x86_64 12/26 2024-09-10T19:00:49.889 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-rgw-2:17.2.7-1606.g5be57395.el9.x86_64 13/26 2024-09-10T19:00:49.889 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 14/26 2024-09-10T19:00:49.889 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 15/26 2024-09-10T19:00:49.889 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 16/26 2024-09-10T19:00:49.890 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 17/26 2024-09-10T19:00:49.890 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 18/26 2024-09-10T19:00:49.890 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 19/26 2024-09-10T19:00:49.890 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 20/26 2024-09-10T19:00:49.890 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 21/26 2024-09-10T19:00:49.890 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 22/26 2024-09-10T19:00:49.890 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : librados2-2:17.2.7-1606.g5be57395.el9.x86_64 23/26 2024-09-10T19:00:49.890 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 24/26 2024-09-10T19:00:49.890 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : librbd1-2:17.2.7-1606.g5be57395.el9.x86_64 25/26 2024-09-10T19:00:50.054 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : ceph-radosgw-2:17.2.7-1606.g5be57395.el9.x86_64 24/26 2024-09-10T19:00:50.130 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: ceph-radosgw-2:17.2.7-1606.g5be57395.el9.x86_64 24/26 2024-09-10T19:00:50.130 INFO:teuthology.orchestra.run.smithi191.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-10T19:00:50.130 INFO:teuthology.orchestra.run.smithi191.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-09-10T19:00:50.130 INFO:teuthology.orchestra.run.smithi191.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-10T19:00:50.130 INFO:teuthology.orchestra.run.smithi191.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-10T19:00:50.130 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-10T19:00:50.132 INFO:teuthology.orchestra.run.smithi191.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 25/26 2024-09-10T19:00:50.214 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 25/26 2024-09-10T19:00:50.215 INFO:teuthology.orchestra.run.smithi191.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 26/26 2024-09-10T19:00:50.754 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 26/26 2024-09-10T19:00:50.754 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-09-10T19:00:50.755 INFO:teuthology.orchestra.run.smithi183.stdout:Upgraded: 2024-09-10T19:00:50.755 INFO:teuthology.orchestra.run.smithi183.stdout: librados2-2:17.2.7-1606.g5be57395.el9.x86_64 2024-09-10T19:00:50.755 INFO:teuthology.orchestra.run.smithi183.stdout: librbd1-2:17.2.7-1606.g5be57395.el9.x86_64 2024-09-10T19:00:50.755 INFO:teuthology.orchestra.run.smithi183.stdout:Installed: 2024-09-10T19:00:50.755 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-base-2:17.2.7-1606.g5be57395.el9.x86_64 2024-09-10T19:00:50.755 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-common-2:17.2.7-1606.g5be57395.el9.x86_64 2024-09-10T19:00:50.755 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-radosgw-2:17.2.7-1606.g5be57395.el9.x86_64 2024-09-10T19:00:50.755 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-selinux-2:17.2.7-1606.g5be57395.el9.x86_64 2024-09-10T19:00:50.755 INFO:teuthology.orchestra.run.smithi183.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-09-10T19:00:50.755 INFO:teuthology.orchestra.run.smithi183.stdout: libcephfs2-2:17.2.7-1606.g5be57395.el9.x86_64 2024-09-10T19:00:50.755 INFO:teuthology.orchestra.run.smithi183.stdout: liboath-2.6.7-2.el9.x86_64 2024-09-10T19:00:50.755 INFO:teuthology.orchestra.run.smithi183.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-09-10T19:00:50.755 INFO:teuthology.orchestra.run.smithi183.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-09-10T19:00:50.755 INFO:teuthology.orchestra.run.smithi183.stdout: libradosstriper1-2:17.2.7-1606.g5be57395.el9.x86_64 2024-09-10T19:00:50.756 INFO:teuthology.orchestra.run.smithi183.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-09-10T19:00:50.756 INFO:teuthology.orchestra.run.smithi183.stdout: librgw2-2:17.2.7-1606.g5be57395.el9.x86_64 2024-09-10T19:00:50.756 INFO:teuthology.orchestra.run.smithi183.stdout: libunwind-1.6.2-1.el9.x86_64 2024-09-10T19:00:50.756 INFO:teuthology.orchestra.run.smithi183.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-09-10T19:00:50.756 INFO:teuthology.orchestra.run.smithi183.stdout: python3-ceph-argparse-2:17.2.7-1606.g5be57395.el9.x86_64 2024-09-10T19:00:50.756 INFO:teuthology.orchestra.run.smithi183.stdout: python3-ceph-common-2:17.2.7-1606.g5be57395.el9.x86_64 2024-09-10T19:00:50.756 INFO:teuthology.orchestra.run.smithi183.stdout: python3-cephfs-2:17.2.7-1606.g5be57395.el9.x86_64 2024-09-10T19:00:50.756 INFO:teuthology.orchestra.run.smithi183.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-09-10T19:00:50.756 INFO:teuthology.orchestra.run.smithi183.stdout: python3-rados-2:17.2.7-1606.g5be57395.el9.x86_64 2024-09-10T19:00:50.756 INFO:teuthology.orchestra.run.smithi183.stdout: python3-rbd-2:17.2.7-1606.g5be57395.el9.x86_64 2024-09-10T19:00:50.756 INFO:teuthology.orchestra.run.smithi183.stdout: python3-rgw-2:17.2.7-1606.g5be57395.el9.x86_64 2024-09-10T19:00:50.756 INFO:teuthology.orchestra.run.smithi183.stdout: thrift-0.15.0-2.el9.x86_64 2024-09-10T19:00:50.756 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-09-10T19:00:50.756 INFO:teuthology.orchestra.run.smithi183.stdout:Complete! 2024-09-10T19:00:51.176 DEBUG:teuthology.orchestra.run.smithi183:> sudo yum -y install ceph-test 2024-09-10T19:00:51.488 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 26/26 2024-09-10T19:00:51.488 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : ceph-base-2:17.2.7-1606.g5be57395.el9.x86_64 1/26 2024-09-10T19:00:51.489 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : ceph-common-2:17.2.7-1606.g5be57395.el9.x86_64 2/26 2024-09-10T19:00:51.489 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : ceph-radosgw-2:17.2.7-1606.g5be57395.el9.x86_64 3/26 2024-09-10T19:00:51.489 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : ceph-selinux-2:17.2.7-1606.g5be57395.el9.x86_64 4/26 2024-09-10T19:00:51.489 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : libcephfs2-2:17.2.7-1606.g5be57395.el9.x86_64 5/26 2024-09-10T19:00:51.489 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : libradosstriper1-2:17.2.7-1606.g5be57395.el9.x86_6 6/26 2024-09-10T19:00:51.489 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : librgw2-2:17.2.7-1606.g5be57395.el9.x86_64 7/26 2024-09-10T19:00:51.489 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-ceph-argparse-2:17.2.7-1606.g5be57395.el9. 8/26 2024-09-10T19:00:51.489 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-ceph-common-2:17.2.7-1606.g5be57395.el9.x8 9/26 2024-09-10T19:00:51.489 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-cephfs-2:17.2.7-1606.g5be57395.el9.x86_64 10/26 2024-09-10T19:00:51.489 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-rados-2:17.2.7-1606.g5be57395.el9.x86_64 11/26 2024-09-10T19:00:51.489 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-rbd-2:17.2.7-1606.g5be57395.el9.x86_64 12/26 2024-09-10T19:00:51.489 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-rgw-2:17.2.7-1606.g5be57395.el9.x86_64 13/26 2024-09-10T19:00:51.489 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 14/26 2024-09-10T19:00:51.490 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 15/26 2024-09-10T19:00:51.490 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 16/26 2024-09-10T19:00:51.490 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 17/26 2024-09-10T19:00:51.490 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 18/26 2024-09-10T19:00:51.490 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 19/26 2024-09-10T19:00:51.490 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 20/26 2024-09-10T19:00:51.490 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 21/26 2024-09-10T19:00:51.490 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 22/26 2024-09-10T19:00:51.490 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : librados2-2:17.2.7-1606.g5be57395.el9.x86_64 23/26 2024-09-10T19:00:51.490 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 24/26 2024-09-10T19:00:51.490 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : librbd1-2:17.2.7-1606.g5be57395.el9.x86_64 25/26 2024-09-10T19:00:51.751 INFO:teuthology.orchestra.run.smithi183.stdout:Last metadata expiration check: 0:00:27 ago on Tue 10 Sep 2024 07:00:24 PM UTC. 2024-09-10T19:00:51.883 INFO:teuthology.orchestra.run.smithi183.stdout:Dependencies resolved. 2024-09-10T19:00:51.883 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2024-09-10T19:00:51.883 INFO:teuthology.orchestra.run.smithi183.stdout: Package Arch Version Repository Size 2024-09-10T19:00:51.883 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2024-09-10T19:00:51.883 INFO:teuthology.orchestra.run.smithi183.stdout:Installing: 2024-09-10T19:00:51.884 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-test x86_64 2:17.2.7-1606.g5be57395.el9 ceph 48 M 2024-09-10T19:00:51.884 INFO:teuthology.orchestra.run.smithi183.stdout:Installing dependencies: 2024-09-10T19:00:51.884 INFO:teuthology.orchestra.run.smithi183.stdout: libcephsqlite x86_64 2:17.2.7-1606.g5be57395.el9 ceph 163 k 2024-09-10T19:00:51.884 INFO:teuthology.orchestra.run.smithi183.stdout: socat x86_64 1.7.4.1-6.el9 appstream 304 k 2024-09-10T19:00:51.884 INFO:teuthology.orchestra.run.smithi183.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-09-10T19:00:51.884 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-09-10T19:00:51.884 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction Summary 2024-09-10T19:00:51.884 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2024-09-10T19:00:51.884 INFO:teuthology.orchestra.run.smithi183.stdout:Install 4 Packages 2024-09-10T19:00:51.884 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-09-10T19:00:51.885 INFO:teuthology.orchestra.run.smithi183.stdout:Total download size: 48 M 2024-09-10T19:00:51.885 INFO:teuthology.orchestra.run.smithi183.stdout:Installed size: 193 M 2024-09-10T19:00:51.885 INFO:teuthology.orchestra.run.smithi183.stdout:Downloading Packages: 2024-09-10T19:00:52.363 INFO:teuthology.orchestra.run.smithi183.stdout:(1/4): socat-1.7.4.1-6.el9.x86_64.rpm 1.5 MB/s | 304 kB 00:00 2024-09-10T19:00:52.388 INFO:teuthology.orchestra.run.smithi183.stdout:(2/4): libcephsqlite-17.2.7-1606.g5be57395.el9. 727 kB/s | 163 kB 00:00 2024-09-10T19:00:52.414 INFO:teuthology.orchestra.run.smithi183.stdout:(3/4): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.2 MB/s | 64 kB 00:00 2024-09-10T19:00:52.505 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 26/26 2024-09-10T19:00:52.505 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-10T19:00:52.505 INFO:teuthology.orchestra.run.smithi191.stdout:Upgraded: 2024-09-10T19:00:52.505 INFO:teuthology.orchestra.run.smithi191.stdout: librados2-2:17.2.7-1606.g5be57395.el9.x86_64 2024-09-10T19:00:52.505 INFO:teuthology.orchestra.run.smithi191.stdout: librbd1-2:17.2.7-1606.g5be57395.el9.x86_64 2024-09-10T19:00:52.506 INFO:teuthology.orchestra.run.smithi191.stdout:Installed: 2024-09-10T19:00:52.506 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-base-2:17.2.7-1606.g5be57395.el9.x86_64 2024-09-10T19:00:52.506 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-common-2:17.2.7-1606.g5be57395.el9.x86_64 2024-09-10T19:00:52.506 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-radosgw-2:17.2.7-1606.g5be57395.el9.x86_64 2024-09-10T19:00:52.506 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-selinux-2:17.2.7-1606.g5be57395.el9.x86_64 2024-09-10T19:00:52.506 INFO:teuthology.orchestra.run.smithi191.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-09-10T19:00:52.506 INFO:teuthology.orchestra.run.smithi191.stdout: libcephfs2-2:17.2.7-1606.g5be57395.el9.x86_64 2024-09-10T19:00:52.506 INFO:teuthology.orchestra.run.smithi191.stdout: liboath-2.6.7-2.el9.x86_64 2024-09-10T19:00:52.506 INFO:teuthology.orchestra.run.smithi191.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-09-10T19:00:52.506 INFO:teuthology.orchestra.run.smithi191.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-09-10T19:00:52.506 INFO:teuthology.orchestra.run.smithi191.stdout: libradosstriper1-2:17.2.7-1606.g5be57395.el9.x86_64 2024-09-10T19:00:52.506 INFO:teuthology.orchestra.run.smithi191.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-09-10T19:00:52.507 INFO:teuthology.orchestra.run.smithi191.stdout: librgw2-2:17.2.7-1606.g5be57395.el9.x86_64 2024-09-10T19:00:52.507 INFO:teuthology.orchestra.run.smithi191.stdout: libunwind-1.6.2-1.el9.x86_64 2024-09-10T19:00:52.507 INFO:teuthology.orchestra.run.smithi191.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-09-10T19:00:52.507 INFO:teuthology.orchestra.run.smithi191.stdout: python3-ceph-argparse-2:17.2.7-1606.g5be57395.el9.x86_64 2024-09-10T19:00:52.507 INFO:teuthology.orchestra.run.smithi191.stdout: python3-ceph-common-2:17.2.7-1606.g5be57395.el9.x86_64 2024-09-10T19:00:52.507 INFO:teuthology.orchestra.run.smithi191.stdout: python3-cephfs-2:17.2.7-1606.g5be57395.el9.x86_64 2024-09-10T19:00:52.507 INFO:teuthology.orchestra.run.smithi191.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-09-10T19:00:52.507 INFO:teuthology.orchestra.run.smithi191.stdout: python3-rados-2:17.2.7-1606.g5be57395.el9.x86_64 2024-09-10T19:00:52.507 INFO:teuthology.orchestra.run.smithi191.stdout: python3-rbd-2:17.2.7-1606.g5be57395.el9.x86_64 2024-09-10T19:00:52.507 INFO:teuthology.orchestra.run.smithi191.stdout: python3-rgw-2:17.2.7-1606.g5be57395.el9.x86_64 2024-09-10T19:00:52.507 INFO:teuthology.orchestra.run.smithi191.stdout: thrift-0.15.0-2.el9.x86_64 2024-09-10T19:00:52.507 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-10T19:00:52.507 INFO:teuthology.orchestra.run.smithi191.stdout:Complete! 2024-09-10T19:00:52.851 DEBUG:teuthology.orchestra.run.smithi191:> sudo yum -y install ceph-test 2024-09-10T19:00:53.414 INFO:teuthology.orchestra.run.smithi191.stdout:Last metadata expiration check: 0:00:28 ago on Tue 10 Sep 2024 07:00:25 PM UTC. 2024-09-10T19:00:53.546 INFO:teuthology.orchestra.run.smithi191.stdout:Dependencies resolved. 2024-09-10T19:00:53.547 INFO:teuthology.orchestra.run.smithi191.stdout:================================================================================ 2024-09-10T19:00:53.547 INFO:teuthology.orchestra.run.smithi191.stdout: Package Arch Version Repository Size 2024-09-10T19:00:53.547 INFO:teuthology.orchestra.run.smithi191.stdout:================================================================================ 2024-09-10T19:00:53.547 INFO:teuthology.orchestra.run.smithi191.stdout:Installing: 2024-09-10T19:00:53.547 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-test x86_64 2:17.2.7-1606.g5be57395.el9 ceph 48 M 2024-09-10T19:00:53.547 INFO:teuthology.orchestra.run.smithi191.stdout:Installing dependencies: 2024-09-10T19:00:53.548 INFO:teuthology.orchestra.run.smithi191.stdout: libcephsqlite x86_64 2:17.2.7-1606.g5be57395.el9 ceph 163 k 2024-09-10T19:00:53.548 INFO:teuthology.orchestra.run.smithi191.stdout: socat x86_64 1.7.4.1-6.el9 appstream 304 k 2024-09-10T19:00:53.548 INFO:teuthology.orchestra.run.smithi191.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-09-10T19:00:53.548 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-10T19:00:53.548 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction Summary 2024-09-10T19:00:53.548 INFO:teuthology.orchestra.run.smithi191.stdout:================================================================================ 2024-09-10T19:00:53.548 INFO:teuthology.orchestra.run.smithi191.stdout:Install 4 Packages 2024-09-10T19:00:53.548 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-10T19:00:53.549 INFO:teuthology.orchestra.run.smithi191.stdout:Total download size: 48 M 2024-09-10T19:00:53.549 INFO:teuthology.orchestra.run.smithi191.stdout:Installed size: 193 M 2024-09-10T19:00:53.549 INFO:teuthology.orchestra.run.smithi191.stdout:Downloading Packages: 2024-09-10T19:00:53.656 INFO:teuthology.orchestra.run.smithi183.stdout:(4/4): ceph-test-17.2.7-1606.g5be57395.el9.x86_ 32 MB/s | 48 MB 00:01 2024-09-10T19:00:53.658 INFO:teuthology.orchestra.run.smithi183.stdout:-------------------------------------------------------------------------------- 2024-09-10T19:00:53.658 INFO:teuthology.orchestra.run.smithi183.stdout:Total 27 MB/s | 48 MB 00:01 2024-09-10T19:00:53.686 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction check 2024-09-10T19:00:53.705 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction check succeeded. 2024-09-10T19:00:53.705 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction test 2024-09-10T19:00:54.053 INFO:teuthology.orchestra.run.smithi191.stdout:(1/4): libcephsqlite-17.2.7-1606.g5be57395.el9. 726 kB/s | 163 kB 00:00 2024-09-10T19:00:54.073 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction test succeeded. 2024-09-10T19:00:54.073 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction 2024-09-10T19:00:54.078 INFO:teuthology.orchestra.run.smithi191.stdout:(2/4): socat-1.7.4.1-6.el9.x86_64.rpm 1.2 MB/s | 304 kB 00:00 2024-09-10T19:00:54.195 INFO:teuthology.orchestra.run.smithi191.stdout:(3/4): xmlstarlet-1.6.1-20.el9.x86_64.rpm 449 kB/s | 64 kB 00:00 2024-09-10T19:00:54.627 INFO:teuthology.orchestra.run.smithi183.stdout: Preparing : 1/1 2024-09-10T19:00:54.699 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/4 2024-09-10T19:00:54.740 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : socat-1.7.4.1-6.el9.x86_64 2/4 2024-09-10T19:00:54.766 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : libcephsqlite-2:17.2.7-1606.g5be57395.el9.x86_64 3/4 2024-09-10T19:00:55.329 INFO:teuthology.orchestra.run.smithi191.stdout:(4/4): ceph-test-17.2.7-1606.g5be57395.el9.x86_ 32 MB/s | 48 MB 00:01 2024-09-10T19:00:55.331 INFO:teuthology.orchestra.run.smithi191.stdout:-------------------------------------------------------------------------------- 2024-09-10T19:00:55.331 INFO:teuthology.orchestra.run.smithi191.stdout:Total 27 MB/s | 48 MB 00:01 2024-09-10T19:00:55.359 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction check 2024-09-10T19:00:55.378 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction check succeeded. 2024-09-10T19:00:55.378 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction test 2024-09-10T19:00:55.745 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction test succeeded. 2024-09-10T19:00:55.745 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction 2024-09-10T19:00:56.268 INFO:teuthology.orchestra.run.smithi191.stdout: Preparing : 1/1 2024-09-10T19:00:56.360 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/4 2024-09-10T19:00:56.411 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : socat-1.7.4.1-6.el9.x86_64 2/4 2024-09-10T19:00:56.447 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : libcephsqlite-2:17.2.7-1606.g5be57395.el9.x86_64 3/4 2024-09-10T19:00:59.163 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: libcephsqlite-2:17.2.7-1606.g5be57395.el9.x86_64 3/4 2024-09-10T19:00:59.248 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : ceph-test-2:17.2.7-1606.g5be57395.el9.x86_64 4/4 2024-09-10T19:01:00.976 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: libcephsqlite-2:17.2.7-1606.g5be57395.el9.x86_64 3/4 2024-09-10T19:01:01.072 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : ceph-test-2:17.2.7-1606.g5be57395.el9.x86_64 4/4 2024-09-10T19:01:01.397 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: ceph-test-2:17.2.7-1606.g5be57395.el9.x86_64 4/4 2024-09-10T19:01:01.397 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : ceph-test-2:17.2.7-1606.g5be57395.el9.x86_64 1/4 2024-09-10T19:01:01.397 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : libcephsqlite-2:17.2.7-1606.g5be57395.el9.x86_64 2/4 2024-09-10T19:01:01.397 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : socat-1.7.4.1-6.el9.x86_64 3/4 2024-09-10T19:01:01.745 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 4/4 2024-09-10T19:01:01.745 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-09-10T19:01:01.746 INFO:teuthology.orchestra.run.smithi183.stdout:Installed: 2024-09-10T19:01:01.746 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-test-2:17.2.7-1606.g5be57395.el9.x86_64 2024-09-10T19:01:01.746 INFO:teuthology.orchestra.run.smithi183.stdout: libcephsqlite-2:17.2.7-1606.g5be57395.el9.x86_64 2024-09-10T19:01:01.746 INFO:teuthology.orchestra.run.smithi183.stdout: socat-1.7.4.1-6.el9.x86_64 2024-09-10T19:01:01.746 INFO:teuthology.orchestra.run.smithi183.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-09-10T19:01:01.746 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-09-10T19:01:01.746 INFO:teuthology.orchestra.run.smithi183.stdout:Complete! 2024-09-10T19:01:02.087 DEBUG:teuthology.orchestra.run.smithi183:> sudo yum -y install ceph 2024-09-10T19:01:02.664 INFO:teuthology.orchestra.run.smithi183.stdout:Last metadata expiration check: 0:00:38 ago on Tue 10 Sep 2024 07:00:24 PM UTC. 2024-09-10T19:01:02.806 INFO:teuthology.orchestra.run.smithi183.stdout:Dependencies resolved. 2024-09-10T19:01:02.809 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2024-09-10T19:01:02.809 INFO:teuthology.orchestra.run.smithi183.stdout: Package Arch Version Repository Size 2024-09-10T19:01:02.809 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2024-09-10T19:01:02.809 INFO:teuthology.orchestra.run.smithi183.stdout:Installing: 2024-09-10T19:01:02.809 INFO:teuthology.orchestra.run.smithi183.stdout: ceph x86_64 2:17.2.7-1606.g5be57395.el9 ceph 6.5 k 2024-09-10T19:01:02.809 INFO:teuthology.orchestra.run.smithi183.stdout:Installing dependencies: 2024-09-10T19:01:02.809 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-mds x86_64 2:17.2.7-1606.g5be57395.el9 ceph 2.1 M 2024-09-10T19:01:02.809 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-mgr x86_64 2:17.2.7-1606.g5be57395.el9 ceph 1.4 M 2024-09-10T19:01:02.810 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-mgr-modules-core noarch 2:17.2.7-1606.g5be57395.el9 ceph-noarch 240 k 2024-09-10T19:01:02.810 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-mon x86_64 2:17.2.7-1606.g5be57395.el9 ceph 3.9 M 2024-09-10T19:01:02.810 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-osd x86_64 2:17.2.7-1606.g5be57395.el9 ceph 15 M 2024-09-10T19:01:02.810 INFO:teuthology.orchestra.run.smithi183.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-09-10T19:01:02.810 INFO:teuthology.orchestra.run.smithi183.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-09-10T19:01:02.810 INFO:teuthology.orchestra.run.smithi183.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-09-10T19:01:02.810 INFO:teuthology.orchestra.run.smithi183.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-09-10T19:01:02.810 INFO:teuthology.orchestra.run.smithi183.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-09-10T19:01:02.810 INFO:teuthology.orchestra.run.smithi183.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-09-10T19:01:02.810 INFO:teuthology.orchestra.run.smithi183.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-09-10T19:01:02.810 INFO:teuthology.orchestra.run.smithi183.stdout: python3-jaraco-collections 2024-09-10T19:01:02.810 INFO:teuthology.orchestra.run.smithi183.stdout: noarch 3.0.0-8.el9 epel 23 k 2024-09-10T19:01:02.810 INFO:teuthology.orchestra.run.smithi183.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-09-10T19:01:02.810 INFO:teuthology.orchestra.run.smithi183.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-09-10T19:01:02.811 INFO:teuthology.orchestra.run.smithi183.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-09-10T19:01:02.811 INFO:teuthology.orchestra.run.smithi183.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-09-10T19:01:02.811 INFO:teuthology.orchestra.run.smithi183.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-09-10T19:01:02.811 INFO:teuthology.orchestra.run.smithi183.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-09-10T19:01:02.811 INFO:teuthology.orchestra.run.smithi183.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-09-10T19:01:02.811 INFO:teuthology.orchestra.run.smithi183.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-09-10T19:01:02.811 INFO:teuthology.orchestra.run.smithi183.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-09-10T19:01:02.811 INFO:teuthology.orchestra.run.smithi183.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-09-10T19:01:02.811 INFO:teuthology.orchestra.run.smithi183.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-09-10T19:01:02.811 INFO:teuthology.orchestra.run.smithi183.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-09-10T19:01:02.811 INFO:teuthology.orchestra.run.smithi183.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-09-10T19:01:02.811 INFO:teuthology.orchestra.run.smithi183.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-09-10T19:01:02.811 INFO:teuthology.orchestra.run.smithi183.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-09-10T19:01:02.812 INFO:teuthology.orchestra.run.smithi183.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-09-10T19:01:02.812 INFO:teuthology.orchestra.run.smithi183.stdout: python3-urllib3 noarch 1.26.5-6.el9 baseos 215 k 2024-09-10T19:01:02.812 INFO:teuthology.orchestra.run.smithi183.stdout: python3-webob noarch 1.8.8-2.el9 epel 230 k 2024-09-10T19:01:02.812 INFO:teuthology.orchestra.run.smithi183.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-09-10T19:01:02.812 INFO:teuthology.orchestra.run.smithi183.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-09-10T19:01:02.812 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-09-10T19:01:02.812 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction Summary 2024-09-10T19:01:02.812 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2024-09-10T19:01:02.812 INFO:teuthology.orchestra.run.smithi183.stdout:Install 34 Packages 2024-09-10T19:01:02.812 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-09-10T19:01:02.813 INFO:teuthology.orchestra.run.smithi183.stdout:Total download size: 27 M 2024-09-10T19:01:02.813 INFO:teuthology.orchestra.run.smithi183.stdout:Installed size: 96 M 2024-09-10T19:01:02.813 INFO:teuthology.orchestra.run.smithi183.stdout:Downloading Packages: 2024-09-10T19:01:03.242 INFO:teuthology.orchestra.run.smithi183.stdout:(1/34): ceph-17.2.7-1606.g5be57395.el9.x86_64.r 59 kB/s | 6.5 kB 00:00 2024-09-10T19:01:03.428 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: ceph-test-2:17.2.7-1606.g5be57395.el9.x86_64 4/4 2024-09-10T19:01:03.428 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : ceph-test-2:17.2.7-1606.g5be57395.el9.x86_64 1/4 2024-09-10T19:01:03.428 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : libcephsqlite-2:17.2.7-1606.g5be57395.el9.x86_64 2/4 2024-09-10T19:01:03.428 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : socat-1.7.4.1-6.el9.x86_64 3/4 2024-09-10T19:01:03.451 INFO:teuthology.orchestra.run.smithi183.stdout:(2/34): ceph-mds-17.2.7-1606.g5be57395.el9.x86_ 6.5 MB/s | 2.1 MB 00:00 2024-09-10T19:01:03.493 INFO:teuthology.orchestra.run.smithi183.stdout:(3/34): ceph-mgr-17.2.7-1606.g5be57395.el9.x86_ 3.9 MB/s | 1.4 MB 00:00 2024-09-10T19:01:03.577 INFO:teuthology.orchestra.run.smithi183.stdout:(4/34): ceph-mgr-modules-core-17.2.7-1606.g5be5 2.8 MB/s | 240 kB 00:00 2024-09-10T19:01:03.652 INFO:teuthology.orchestra.run.smithi183.stdout:(5/34): ceph-mon-17.2.7-1606.g5be57395.el9.x86_ 9.6 MB/s | 3.9 MB 00:00 2024-09-10T19:01:03.736 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 4/4 2024-09-10T19:01:03.736 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-10T19:01:03.736 INFO:teuthology.orchestra.run.smithi191.stdout:Installed: 2024-09-10T19:01:03.736 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-test-2:17.2.7-1606.g5be57395.el9.x86_64 2024-09-10T19:01:03.737 INFO:teuthology.orchestra.run.smithi191.stdout: libcephsqlite-2:17.2.7-1606.g5be57395.el9.x86_64 2024-09-10T19:01:03.737 INFO:teuthology.orchestra.run.smithi191.stdout: socat-1.7.4.1-6.el9.x86_64 2024-09-10T19:01:03.737 INFO:teuthology.orchestra.run.smithi191.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-09-10T19:01:03.737 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-10T19:01:03.737 INFO:teuthology.orchestra.run.smithi191.stdout:Complete! 2024-09-10T19:01:03.828 INFO:teuthology.orchestra.run.smithi183.stdout:(6/34): python3-cffi-1.14.5-5.el9.x86_64.rpm 1.0 MB/s | 253 kB 00:00 2024-09-10T19:01:03.978 INFO:teuthology.orchestra.run.smithi183.stdout:(7/34): ceph-osd-17.2.7-1606.g5be57395.el9.x86_ 29 MB/s | 15 MB 00:00 2024-09-10T19:01:03.998 DEBUG:teuthology.orchestra.run.smithi191:> sudo yum -y install ceph 2024-09-10T19:01:04.003 INFO:teuthology.orchestra.run.smithi183.stdout:(8/34): python3-ply-3.11-14.el9.noarch.rpm 608 kB/s | 106 kB 00:00 2024-09-10T19:01:04.061 INFO:teuthology.orchestra.run.smithi183.stdout:(9/34): python3-requests-2.25.1-8.el9.noarch.rp 2.1 MB/s | 125 kB 00:00 2024-09-10T19:01:04.095 INFO:teuthology.orchestra.run.smithi183.stdout:(10/34): python3-cryptography-36.0.1-4.el9.x86_ 2.8 MB/s | 1.2 MB 00:00 2024-09-10T19:01:04.154 INFO:teuthology.orchestra.run.smithi183.stdout:(11/34): python3-urllib3-1.26.5-6.el9.noarch.rp 2.3 MB/s | 215 kB 00:00 2024-09-10T19:01:04.179 INFO:teuthology.orchestra.run.smithi183.stdout:(12/34): python3-pycparser-2.20-6.el9.noarch.rp 673 kB/s | 135 kB 00:00 2024-09-10T19:01:04.229 INFO:teuthology.orchestra.run.smithi183.stdout:(13/34): python3-mako-1.1.4-6.el9.noarch.rpm 1.3 MB/s | 172 kB 00:00 2024-09-10T19:01:04.254 INFO:teuthology.orchestra.run.smithi183.stdout:(14/34): python3-markupsafe-1.1.1-12.el9.x86_64 346 kB/s | 35 kB 00:00 2024-09-10T19:01:04.279 INFO:teuthology.orchestra.run.smithi183.stdout:(15/34): python3-pytz-2021.1-5.el9.noarch.rpm 1.0 MB/s | 51 kB 00:00 2024-09-10T19:01:04.305 INFO:teuthology.orchestra.run.smithi183.stdout:(16/34): python3-packaging-20.9-5.el9.noarch.rp 613 kB/s | 77 kB 00:00 2024-09-10T19:01:04.330 INFO:teuthology.orchestra.run.smithi183.stdout:(17/34): python3-toml-0.10.2-6.el9.noarch.rpm 551 kB/s | 42 kB 00:00 2024-09-10T19:01:04.380 INFO:teuthology.orchestra.run.smithi183.stdout:(18/34): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 432 kB/s | 43 kB 00:00 2024-09-10T19:01:04.406 INFO:teuthology.orchestra.run.smithi183.stdout:(19/34): python3-jaraco-8.2.1-3.el9.noarch.rpm 458 kB/s | 11 kB 00:00 2024-09-10T19:01:04.431 INFO:teuthology.orchestra.run.smithi183.stdout:(20/34): python3-cheroot-8.6.0-4.el9.noarch.rpm 1.3 MB/s | 172 kB 00:00 2024-09-10T19:01:04.456 INFO:teuthology.orchestra.run.smithi183.stdout:(21/34): python3-jaraco-classes-3.2.1-5.el9.noa 352 kB/s | 18 kB 00:00 2024-09-10T19:01:04.481 INFO:teuthology.orchestra.run.smithi183.stdout:(22/34): python3-jaraco-collections-3.0.0-8.el9 462 kB/s | 23 kB 00:00 2024-09-10T19:01:04.507 INFO:teuthology.orchestra.run.smithi183.stdout:(23/34): python3-jaraco-functools-3.5.0-2.el9.n 388 kB/s | 19 kB 00:00 2024-09-10T19:01:04.532 INFO:teuthology.orchestra.run.smithi183.stdout:(24/34): python3-jaraco-text-3.2.0-6.el9.noarch 391 kB/s | 20 kB 00:00 2024-09-10T19:01:04.557 INFO:teuthology.orchestra.run.smithi183.stdout:(25/34): python3-logutils-0.3.5-21.el9.noarch.r 923 kB/s | 46 kB 00:00 2024-09-10T19:01:04.564 INFO:teuthology.orchestra.run.smithi191.stdout:Last metadata expiration check: 0:00:39 ago on Tue 10 Sep 2024 07:00:25 PM UTC. 2024-09-10T19:01:04.582 INFO:teuthology.orchestra.run.smithi183.stdout:(26/34): python3-more-itertools-8.12.0-2.el9.no 1.5 MB/s | 79 kB 00:00 2024-09-10T19:01:04.608 INFO:teuthology.orchestra.run.smithi183.stdout:(27/34): python3-cherrypy-18.6.1-2.el9.noarch.r 1.3 MB/s | 358 kB 00:00 2024-09-10T19:01:04.633 INFO:teuthology.orchestra.run.smithi183.stdout:(28/34): python3-portend-3.1.0-2.el9.noarch.rpm 327 kB/s | 16 kB 00:00 2024-09-10T19:01:04.659 INFO:teuthology.orchestra.run.smithi183.stdout:(29/34): python3-pyOpenSSL-21.0.0-1.el9.noarch. 1.7 MB/s | 90 kB 00:00 2024-09-10T19:01:04.684 INFO:teuthology.orchestra.run.smithi183.stdout:(30/34): python3-tempora-5.0.0-2.el9.noarch.rpm 709 kB/s | 36 kB 00:00 2024-09-10T19:01:04.699 INFO:teuthology.orchestra.run.smithi191.stdout:Dependencies resolved. 2024-09-10T19:01:04.702 INFO:teuthology.orchestra.run.smithi191.stdout:================================================================================ 2024-09-10T19:01:04.702 INFO:teuthology.orchestra.run.smithi191.stdout: Package Arch Version Repository Size 2024-09-10T19:01:04.702 INFO:teuthology.orchestra.run.smithi191.stdout:================================================================================ 2024-09-10T19:01:04.702 INFO:teuthology.orchestra.run.smithi191.stdout:Installing: 2024-09-10T19:01:04.702 INFO:teuthology.orchestra.run.smithi191.stdout: ceph x86_64 2:17.2.7-1606.g5be57395.el9 ceph 6.5 k 2024-09-10T19:01:04.702 INFO:teuthology.orchestra.run.smithi191.stdout:Installing dependencies: 2024-09-10T19:01:04.702 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-mds x86_64 2:17.2.7-1606.g5be57395.el9 ceph 2.1 M 2024-09-10T19:01:04.702 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-mgr x86_64 2:17.2.7-1606.g5be57395.el9 ceph 1.4 M 2024-09-10T19:01:04.703 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-mgr-modules-core noarch 2:17.2.7-1606.g5be57395.el9 ceph-noarch 240 k 2024-09-10T19:01:04.703 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-mon x86_64 2:17.2.7-1606.g5be57395.el9 ceph 3.9 M 2024-09-10T19:01:04.703 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-osd x86_64 2:17.2.7-1606.g5be57395.el9 ceph 15 M 2024-09-10T19:01:04.703 INFO:teuthology.orchestra.run.smithi191.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-09-10T19:01:04.703 INFO:teuthology.orchestra.run.smithi191.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-09-10T19:01:04.703 INFO:teuthology.orchestra.run.smithi191.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-09-10T19:01:04.703 INFO:teuthology.orchestra.run.smithi191.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-09-10T19:01:04.703 INFO:teuthology.orchestra.run.smithi191.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-09-10T19:01:04.703 INFO:teuthology.orchestra.run.smithi191.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-09-10T19:01:04.703 INFO:teuthology.orchestra.run.smithi191.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-09-10T19:01:04.703 INFO:teuthology.orchestra.run.smithi191.stdout: python3-jaraco-collections 2024-09-10T19:01:04.703 INFO:teuthology.orchestra.run.smithi191.stdout: noarch 3.0.0-8.el9 epel 23 k 2024-09-10T19:01:04.703 INFO:teuthology.orchestra.run.smithi191.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-09-10T19:01:04.704 INFO:teuthology.orchestra.run.smithi191.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-09-10T19:01:04.704 INFO:teuthology.orchestra.run.smithi191.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-09-10T19:01:04.704 INFO:teuthology.orchestra.run.smithi191.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-09-10T19:01:04.704 INFO:teuthology.orchestra.run.smithi191.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-09-10T19:01:04.704 INFO:teuthology.orchestra.run.smithi191.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-09-10T19:01:04.704 INFO:teuthology.orchestra.run.smithi191.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-09-10T19:01:04.704 INFO:teuthology.orchestra.run.smithi191.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-09-10T19:01:04.704 INFO:teuthology.orchestra.run.smithi191.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-09-10T19:01:04.704 INFO:teuthology.orchestra.run.smithi191.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-09-10T19:01:04.704 INFO:teuthology.orchestra.run.smithi191.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-09-10T19:01:04.704 INFO:teuthology.orchestra.run.smithi191.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-09-10T19:01:04.704 INFO:teuthology.orchestra.run.smithi191.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-09-10T19:01:04.704 INFO:teuthology.orchestra.run.smithi191.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-09-10T19:01:04.704 INFO:teuthology.orchestra.run.smithi191.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-09-10T19:01:04.705 INFO:teuthology.orchestra.run.smithi191.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-09-10T19:01:04.705 INFO:teuthology.orchestra.run.smithi191.stdout: python3-urllib3 noarch 1.26.5-6.el9 baseos 215 k 2024-09-10T19:01:04.705 INFO:teuthology.orchestra.run.smithi191.stdout: python3-webob noarch 1.8.8-2.el9 epel 230 k 2024-09-10T19:01:04.705 INFO:teuthology.orchestra.run.smithi191.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-09-10T19:01:04.705 INFO:teuthology.orchestra.run.smithi191.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-09-10T19:01:04.705 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-10T19:01:04.705 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction Summary 2024-09-10T19:01:04.705 INFO:teuthology.orchestra.run.smithi191.stdout:================================================================================ 2024-09-10T19:01:04.705 INFO:teuthology.orchestra.run.smithi191.stdout:Install 34 Packages 2024-09-10T19:01:04.705 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-10T19:01:04.706 INFO:teuthology.orchestra.run.smithi191.stdout:Total download size: 27 M 2024-09-10T19:01:04.706 INFO:teuthology.orchestra.run.smithi191.stdout:Installed size: 96 M 2024-09-10T19:01:04.706 INFO:teuthology.orchestra.run.smithi191.stdout:Downloading Packages: 2024-09-10T19:01:04.709 INFO:teuthology.orchestra.run.smithi183.stdout:(31/34): python3-pecan-1.4.2-3.el9.noarch.rpm 1.8 MB/s | 272 kB 00:00 2024-09-10T19:01:04.734 INFO:teuthology.orchestra.run.smithi183.stdout:(32/34): python3-zc-lockfile-2.0-10.el9.noarch. 799 kB/s | 20 kB 00:00 2024-09-10T19:01:04.759 INFO:teuthology.orchestra.run.smithi183.stdout:(33/34): python3-webob-1.8.8-2.el9.noarch.rpm 2.2 MB/s | 230 kB 00:00 2024-09-10T19:01:04.793 INFO:teuthology.orchestra.run.smithi183.stdout:(34/34): python3-werkzeug-2.0.3-3.el9.1.noarch. 3.8 MB/s | 427 kB 00:00 2024-09-10T19:01:04.797 INFO:teuthology.orchestra.run.smithi183.stdout:-------------------------------------------------------------------------------- 2024-09-10T19:01:04.797 INFO:teuthology.orchestra.run.smithi183.stdout:Total 14 MB/s | 27 MB 00:01 2024-09-10T19:01:04.921 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction check 2024-09-10T19:01:04.957 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction check succeeded. 2024-09-10T19:01:04.957 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction test 2024-09-10T19:01:05.336 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction test succeeded. 2024-09-10T19:01:05.336 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction 2024-09-10T19:01:05.681 INFO:teuthology.orchestra.run.smithi191.stdout:(1/34): ceph-17.2.7-1606.g5be57395.el9.x86_64.r 51 kB/s | 6.5 kB 00:00 2024-09-10T19:01:05.873 INFO:teuthology.orchestra.run.smithi191.stdout:(2/34): ceph-mgr-17.2.7-1606.g5be57395.el9.x86_ 4.4 MB/s | 1.4 MB 00:00 2024-09-10T19:01:05.924 INFO:teuthology.orchestra.run.smithi191.stdout:(3/34): ceph-mds-17.2.7-1606.g5be57395.el9.x86_ 5.6 MB/s | 2.1 MB 00:00 2024-09-10T19:01:05.999 INFO:teuthology.orchestra.run.smithi191.stdout:(4/34): ceph-mon-17.2.7-1606.g5be57395.el9.x86_ 12 MB/s | 3.9 MB 00:00 2024-09-10T19:01:06.024 INFO:teuthology.orchestra.run.smithi191.stdout:(5/34): ceph-mgr-modules-core-17.2.7-1606.g5be5 2.3 MB/s | 240 kB 00:00 2024-09-10T19:01:06.177 INFO:teuthology.orchestra.run.smithi191.stdout:(6/34): python3-cffi-1.14.5-5.el9.x86_64.rpm 1.4 MB/s | 253 kB 00:00 2024-09-10T19:01:06.219 INFO:teuthology.orchestra.run.smithi191.stdout:(7/34): python3-ply-3.11-14.el9.noarch.rpm 2.5 MB/s | 106 kB 00:00 2024-09-10T19:01:06.394 INFO:teuthology.orchestra.run.smithi191.stdout:(8/34): ceph-osd-17.2.7-1606.g5be57395.el9.x86_ 29 MB/s | 15 MB 00:00 2024-09-10T19:01:06.419 INFO:teuthology.orchestra.run.smithi191.stdout:(9/34): python3-pycparser-2.20-6.el9.noarch.rpm 675 kB/s | 135 kB 00:00 2024-09-10T19:01:06.461 INFO:teuthology.orchestra.run.smithi191.stdout:(10/34): python3-cryptography-36.0.1-4.el9.x86_ 2.8 MB/s | 1.2 MB 00:00 2024-09-10T19:01:06.486 INFO:teuthology.orchestra.run.smithi191.stdout:(11/34): python3-urllib3-1.26.5-6.el9.noarch.rp 3.1 MB/s | 215 kB 00:00 2024-09-10T19:01:06.545 INFO:teuthology.orchestra.run.smithi191.stdout:(12/34): python3-requests-2.25.1-8.el9.noarch.r 829 kB/s | 125 kB 00:00 2024-09-10T19:01:06.595 INFO:teuthology.orchestra.run.smithi191.stdout:(13/34): python3-markupsafe-1.1.1-12.el9.x86_64 320 kB/s | 35 kB 00:00 2024-09-10T19:01:06.608 INFO:teuthology.orchestra.run.smithi183.stdout: Preparing : 1/1 2024-09-10T19:01:06.662 INFO:teuthology.orchestra.run.smithi191.stdout:(14/34): python3-mako-1.1.4-6.el9.noarch.rpm 858 kB/s | 172 kB 00:00 2024-09-10T19:01:06.663 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/34 2024-09-10T19:01:06.687 INFO:teuthology.orchestra.run.smithi191.stdout:(15/34): python3-pytz-2021.1-5.el9.noarch.rpm 556 kB/s | 51 kB 00:00 2024-09-10T19:01:06.695 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/34 2024-09-10T19:01:06.721 INFO:teuthology.orchestra.run.smithi191.stdout:(16/34): python3-toml-0.10.2-6.el9.noarch.rpm 711 kB/s | 42 kB 00:00 2024-09-10T19:01:06.728 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 3/34 2024-09-10T19:01:06.746 INFO:teuthology.orchestra.run.smithi191.stdout:(17/34): python3-packaging-20.9-5.el9.noarch.rp 384 kB/s | 77 kB 00:00 2024-09-10T19:01:06.780 INFO:teuthology.orchestra.run.smithi191.stdout:(18/34): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 472 kB/s | 43 kB 00:00 2024-09-10T19:01:06.813 INFO:teuthology.orchestra.run.smithi191.stdout:(19/34): python3-jaraco-8.2.1-3.el9.noarch.rpm 447 kB/s | 11 kB 00:00 2024-09-10T19:01:06.819 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 4/34 2024-09-10T19:01:06.838 INFO:teuthology.orchestra.run.smithi191.stdout:(20/34): python3-cheroot-8.6.0-4.el9.noarch.rpm 1.5 MB/s | 172 kB 00:00 2024-09-10T19:01:06.864 INFO:teuthology.orchestra.run.smithi191.stdout:(21/34): python3-jaraco-classes-3.2.1-5.el9.noa 364 kB/s | 18 kB 00:00 2024-09-10T19:01:06.889 INFO:teuthology.orchestra.run.smithi191.stdout:(22/34): python3-jaraco-collections-3.0.0-8.el9 464 kB/s | 23 kB 00:00 2024-09-10T19:01:06.907 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 5/34 2024-09-10T19:01:06.914 INFO:teuthology.orchestra.run.smithi191.stdout:(23/34): python3-jaraco-functools-3.5.0-2.el9.n 389 kB/s | 19 kB 00:00 2024-09-10T19:01:06.939 INFO:teuthology.orchestra.run.smithi191.stdout:(24/34): python3-jaraco-text-3.2.0-6.el9.noarch 392 kB/s | 20 kB 00:00 2024-09-10T19:01:06.947 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-webob-1.8.8-2.el9.noarch 6/34 2024-09-10T19:01:06.965 INFO:teuthology.orchestra.run.smithi191.stdout:(25/34): python3-logutils-0.3.5-21.el9.noarch.r 923 kB/s | 46 kB 00:00 2024-09-10T19:01:06.990 INFO:teuthology.orchestra.run.smithi191.stdout:(26/34): python3-more-itertools-8.12.0-2.el9.no 1.5 MB/s | 79 kB 00:00 2024-09-10T19:01:06.992 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 7/34 2024-09-10T19:01:07.015 INFO:teuthology.orchestra.run.smithi191.stdout:(27/34): python3-portend-3.1.0-2.el9.noarch.rpm 659 kB/s | 16 kB 00:00 2024-09-10T19:01:07.022 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 8/34 2024-09-10T19:01:07.041 INFO:teuthology.orchestra.run.smithi191.stdout:(28/34): python3-cherrypy-18.6.1-2.el9.noarch.r 1.2 MB/s | 358 kB 00:00 2024-09-10T19:01:07.056 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 9/34 2024-09-10T19:01:07.066 INFO:teuthology.orchestra.run.smithi191.stdout:(29/34): python3-pyOpenSSL-21.0.0-1.el9.noarch. 1.7 MB/s | 90 kB 00:00 2024-09-10T19:01:07.090 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 10/34 2024-09-10T19:01:07.091 INFO:teuthology.orchestra.run.smithi191.stdout:(30/34): python3-tempora-5.0.0-2.el9.noarch.rpm 712 kB/s | 36 kB 00:00 2024-09-10T19:01:07.117 INFO:teuthology.orchestra.run.smithi191.stdout:(31/34): python3-pecan-1.4.2-3.el9.noarch.rpm 1.8 MB/s | 272 kB 00:00 2024-09-10T19:01:07.130 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 11/34 2024-09-10T19:01:07.142 INFO:teuthology.orchestra.run.smithi191.stdout:(32/34): python3-zc-lockfile-2.0-10.el9.noarch. 801 kB/s | 20 kB 00:00 2024-09-10T19:01:07.167 INFO:teuthology.orchestra.run.smithi191.stdout:(33/34): python3-webob-1.8.8-2.el9.noarch.rpm 2.2 MB/s | 230 kB 00:00 2024-09-10T19:01:07.173 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 12/34 2024-09-10T19:01:07.200 INFO:teuthology.orchestra.run.smithi191.stdout:(34/34): python3-werkzeug-2.0.3-3.el9.1.noarch. 3.8 MB/s | 427 kB 00:00 2024-09-10T19:01:07.206 INFO:teuthology.orchestra.run.smithi191.stdout:-------------------------------------------------------------------------------- 2024-09-10T19:01:07.206 INFO:teuthology.orchestra.run.smithi191.stdout:Total 11 MB/s | 27 MB 00:02 2024-09-10T19:01:07.210 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 13/34 2024-09-10T19:01:07.258 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 14/34 2024-09-10T19:01:07.295 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-packaging-20.9-5.el9.noarch 15/34 2024-09-10T19:01:07.340 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction check 2024-09-10T19:01:07.372 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 16/34 2024-09-10T19:01:07.377 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction check succeeded. 2024-09-10T19:01:07.377 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction test 2024-09-10T19:01:07.512 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 17/34 2024-09-10T19:01:07.598 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 18/34 2024-09-10T19:01:07.652 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-urllib3-1.26.5-6.el9.noarch 19/34 2024-09-10T19:01:07.691 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 20/34 2024-09-10T19:01:07.748 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-ply-3.11-14.el9.noarch 21/34 2024-09-10T19:01:07.759 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction test succeeded. 2024-09-10T19:01:07.760 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction 2024-09-10T19:01:07.802 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 22/34 2024-09-10T19:01:07.988 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 23/34 2024-09-10T19:01:08.045 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 24/34 2024-09-10T19:01:08.124 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 25/34 2024-09-10T19:01:08.219 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 26/34 2024-09-10T19:01:08.263 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 27/34 2024-09-10T19:01:08.385 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 28/34 2024-09-10T19:01:08.530 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : ceph-mgr-modules-core-2:17.2.7-1606.g5be57395.el9. 29/34 2024-09-10T19:01:08.554 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : ceph-mgr-2:17.2.7-1606.g5be57395.el9.x86_64 30/34 2024-09-10T19:01:08.580 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: ceph-mgr-2:17.2.7-1606.g5be57395.el9.x86_64 30/34 2024-09-10T19:01:08.580 INFO:teuthology.orchestra.run.smithi183.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-10T19:01:08.580 INFO:teuthology.orchestra.run.smithi183.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-09-10T19:01:08.580 INFO:teuthology.orchestra.run.smithi183.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-10T19:01:08.580 INFO:teuthology.orchestra.run.smithi183.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-10T19:01:08.580 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-09-10T19:01:09.010 INFO:teuthology.orchestra.run.smithi191.stdout: Preparing : 1/1 2024-09-10T19:01:09.070 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/34 2024-09-10T19:01:09.102 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/34 2024-09-10T19:01:09.135 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 3/34 2024-09-10T19:01:09.227 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 4/34 2024-09-10T19:01:09.330 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 5/34 2024-09-10T19:01:09.372 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-webob-1.8.8-2.el9.noarch 6/34 2024-09-10T19:01:09.415 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 7/34 2024-09-10T19:01:09.445 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 8/34 2024-09-10T19:01:09.480 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 9/34 2024-09-10T19:01:09.513 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 10/34 2024-09-10T19:01:09.553 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 11/34 2024-09-10T19:01:09.588 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 12/34 2024-09-10T19:01:09.623 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 13/34 2024-09-10T19:01:09.671 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 14/34 2024-09-10T19:01:09.710 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-packaging-20.9-5.el9.noarch 15/34 2024-09-10T19:01:09.794 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 16/34 2024-09-10T19:01:09.931 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : ceph-osd-2:17.2.7-1606.g5be57395.el9.x86_64 31/34 2024-09-10T19:01:09.931 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 17/34 2024-09-10T19:01:09.960 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: ceph-osd-2:17.2.7-1606.g5be57395.el9.x86_64 31/34 2024-09-10T19:01:09.960 INFO:teuthology.orchestra.run.smithi183.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-10T19:01:09.960 INFO:teuthology.orchestra.run.smithi183.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-09-10T19:01:09.960 INFO:teuthology.orchestra.run.smithi183.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-10T19:01:09.960 INFO:teuthology.orchestra.run.smithi183.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-10T19:01:09.960 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-09-10T19:01:10.020 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 18/34 2024-09-10T19:01:10.097 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-urllib3-1.26.5-6.el9.noarch 19/34 2024-09-10T19:01:10.136 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 20/34 2024-09-10T19:01:10.185 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-ply-3.11-14.el9.noarch 21/34 2024-09-10T19:01:10.241 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 22/34 2024-09-10T19:01:10.357 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : ceph-mon-2:17.2.7-1606.g5be57395.el9.x86_64 32/34 2024-09-10T19:01:10.380 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: ceph-mon-2:17.2.7-1606.g5be57395.el9.x86_64 32/34 2024-09-10T19:01:10.381 INFO:teuthology.orchestra.run.smithi183.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-10T19:01:10.381 INFO:teuthology.orchestra.run.smithi183.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-09-10T19:01:10.381 INFO:teuthology.orchestra.run.smithi183.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-10T19:01:10.381 INFO:teuthology.orchestra.run.smithi183.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-10T19:01:10.381 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-09-10T19:01:10.426 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 23/34 2024-09-10T19:01:10.482 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 24/34 2024-09-10T19:01:10.552 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 25/34 2024-09-10T19:01:10.582 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : ceph-mds-2:17.2.7-1606.g5be57395.el9.x86_64 33/34 2024-09-10T19:01:10.605 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: ceph-mds-2:17.2.7-1606.g5be57395.el9.x86_64 33/34 2024-09-10T19:01:10.605 INFO:teuthology.orchestra.run.smithi183.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-10T19:01:10.605 INFO:teuthology.orchestra.run.smithi183.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-09-10T19:01:10.606 INFO:teuthology.orchestra.run.smithi183.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-10T19:01:10.606 INFO:teuthology.orchestra.run.smithi183.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-10T19:01:10.606 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-09-10T19:01:10.639 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 26/34 2024-09-10T19:01:10.670 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : ceph-2:17.2.7-1606.g5be57395.el9.x86_64 34/34 2024-09-10T19:01:10.685 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 27/34 2024-09-10T19:01:10.805 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 28/34 2024-09-10T19:01:10.952 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : ceph-mgr-modules-core-2:17.2.7-1606.g5be57395.el9. 29/34 2024-09-10T19:01:10.983 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : ceph-mgr-2:17.2.7-1606.g5be57395.el9.x86_64 30/34 2024-09-10T19:01:11.009 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: ceph-mgr-2:17.2.7-1606.g5be57395.el9.x86_64 30/34 2024-09-10T19:01:11.009 INFO:teuthology.orchestra.run.smithi191.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-10T19:01:11.009 INFO:teuthology.orchestra.run.smithi191.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-09-10T19:01:11.009 INFO:teuthology.orchestra.run.smithi191.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-10T19:01:11.009 INFO:teuthology.orchestra.run.smithi191.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-10T19:01:11.009 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-10T19:01:12.350 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: ceph-2:17.2.7-1606.g5be57395.el9.x86_64 34/34 2024-09-10T19:01:12.350 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : ceph-2:17.2.7-1606.g5be57395.el9.x86_64 1/34 2024-09-10T19:01:12.350 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : ceph-mds-2:17.2.7-1606.g5be57395.el9.x86_64 2/34 2024-09-10T19:01:12.350 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : ceph-mgr-2:17.2.7-1606.g5be57395.el9.x86_64 3/34 2024-09-10T19:01:12.351 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : ceph-mon-2:17.2.7-1606.g5be57395.el9.x86_64 4/34 2024-09-10T19:01:12.351 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : ceph-osd-2:17.2.7-1606.g5be57395.el9.x86_64 5/34 2024-09-10T19:01:12.351 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : ceph-mgr-modules-core-2:17.2.7-1606.g5be57395.el9. 6/34 2024-09-10T19:01:12.351 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 7/34 2024-09-10T19:01:12.351 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 8/34 2024-09-10T19:01:12.351 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-ply-3.11-14.el9.noarch 9/34 2024-09-10T19:01:12.351 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 10/34 2024-09-10T19:01:12.351 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 11/34 2024-09-10T19:01:12.351 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-urllib3-1.26.5-6.el9.noarch 12/34 2024-09-10T19:01:12.351 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 13/34 2024-09-10T19:01:12.351 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 14/34 2024-09-10T19:01:12.351 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 15/34 2024-09-10T19:01:12.351 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 16/34 2024-09-10T19:01:12.352 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 17/34 2024-09-10T19:01:12.352 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 18/34 2024-09-10T19:01:12.352 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 19/34 2024-09-10T19:01:12.352 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 20/34 2024-09-10T19:01:12.352 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 21/34 2024-09-10T19:01:12.352 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 22/34 2024-09-10T19:01:12.352 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 23/34 2024-09-10T19:01:12.352 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 24/34 2024-09-10T19:01:12.352 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 25/34 2024-09-10T19:01:12.352 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 26/34 2024-09-10T19:01:12.352 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 27/34 2024-09-10T19:01:12.352 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 28/34 2024-09-10T19:01:12.353 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 29/34 2024-09-10T19:01:12.353 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 30/34 2024-09-10T19:01:12.354 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 31/34 2024-09-10T19:01:12.354 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-webob-1.8.8-2.el9.noarch 32/34 2024-09-10T19:01:12.354 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 33/34 2024-09-10T19:01:12.385 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : ceph-osd-2:17.2.7-1606.g5be57395.el9.x86_64 31/34 2024-09-10T19:01:12.413 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: ceph-osd-2:17.2.7-1606.g5be57395.el9.x86_64 31/34 2024-09-10T19:01:12.413 INFO:teuthology.orchestra.run.smithi191.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-10T19:01:12.413 INFO:teuthology.orchestra.run.smithi191.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-09-10T19:01:12.414 INFO:teuthology.orchestra.run.smithi191.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-10T19:01:12.414 INFO:teuthology.orchestra.run.smithi191.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-10T19:01:12.414 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-10T19:01:12.785 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : ceph-mon-2:17.2.7-1606.g5be57395.el9.x86_64 32/34 2024-09-10T19:01:12.810 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: ceph-mon-2:17.2.7-1606.g5be57395.el9.x86_64 32/34 2024-09-10T19:01:12.810 INFO:teuthology.orchestra.run.smithi191.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-10T19:01:12.810 INFO:teuthology.orchestra.run.smithi191.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-09-10T19:01:12.810 INFO:teuthology.orchestra.run.smithi191.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-10T19:01:12.810 INFO:teuthology.orchestra.run.smithi191.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-10T19:01:12.810 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-10T19:01:13.011 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : ceph-mds-2:17.2.7-1606.g5be57395.el9.x86_64 33/34 2024-09-10T19:01:13.035 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: ceph-mds-2:17.2.7-1606.g5be57395.el9.x86_64 33/34 2024-09-10T19:01:13.036 INFO:teuthology.orchestra.run.smithi191.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-10T19:01:13.036 INFO:teuthology.orchestra.run.smithi191.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-09-10T19:01:13.036 INFO:teuthology.orchestra.run.smithi191.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-10T19:01:13.036 INFO:teuthology.orchestra.run.smithi191.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-10T19:01:13.036 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-10T19:01:13.100 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : ceph-2:17.2.7-1606.g5be57395.el9.x86_64 34/34 2024-09-10T19:01:13.241 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 34/34 2024-09-10T19:01:13.241 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-09-10T19:01:13.241 INFO:teuthology.orchestra.run.smithi183.stdout:Installed: 2024-09-10T19:01:13.241 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-2:17.2.7-1606.g5be57395.el9.x86_64 2024-09-10T19:01:13.241 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-mds-2:17.2.7-1606.g5be57395.el9.x86_64 2024-09-10T19:01:13.241 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-mgr-2:17.2.7-1606.g5be57395.el9.x86_64 2024-09-10T19:01:13.241 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-mgr-modules-core-2:17.2.7-1606.g5be57395.el9.noarch 2024-09-10T19:01:13.241 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-mon-2:17.2.7-1606.g5be57395.el9.x86_64 2024-09-10T19:01:13.241 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-osd-2:17.2.7-1606.g5be57395.el9.x86_64 2024-09-10T19:01:13.242 INFO:teuthology.orchestra.run.smithi183.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-09-10T19:01:13.242 INFO:teuthology.orchestra.run.smithi183.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-09-10T19:01:13.242 INFO:teuthology.orchestra.run.smithi183.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-09-10T19:01:13.242 INFO:teuthology.orchestra.run.smithi183.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-09-10T19:01:13.242 INFO:teuthology.orchestra.run.smithi183.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-09-10T19:01:13.242 INFO:teuthology.orchestra.run.smithi183.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-09-10T19:01:13.242 INFO:teuthology.orchestra.run.smithi183.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-09-10T19:01:13.242 INFO:teuthology.orchestra.run.smithi183.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-09-10T19:01:13.242 INFO:teuthology.orchestra.run.smithi183.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-09-10T19:01:13.242 INFO:teuthology.orchestra.run.smithi183.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-09-10T19:01:13.242 INFO:teuthology.orchestra.run.smithi183.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-09-10T19:01:13.242 INFO:teuthology.orchestra.run.smithi183.stdout: python3-mako-1.1.4-6.el9.noarch 2024-09-10T19:01:13.243 INFO:teuthology.orchestra.run.smithi183.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-09-10T19:01:13.243 INFO:teuthology.orchestra.run.smithi183.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-09-10T19:01:13.243 INFO:teuthology.orchestra.run.smithi183.stdout: python3-packaging-20.9-5.el9.noarch 2024-09-10T19:01:13.243 INFO:teuthology.orchestra.run.smithi183.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-09-10T19:01:13.243 INFO:teuthology.orchestra.run.smithi183.stdout: python3-ply-3.11-14.el9.noarch 2024-09-10T19:01:13.243 INFO:teuthology.orchestra.run.smithi183.stdout: python3-portend-3.1.0-2.el9.noarch 2024-09-10T19:01:13.243 INFO:teuthology.orchestra.run.smithi183.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-09-10T19:01:13.243 INFO:teuthology.orchestra.run.smithi183.stdout: python3-pycparser-2.20-6.el9.noarch 2024-09-10T19:01:13.243 INFO:teuthology.orchestra.run.smithi183.stdout: python3-pytz-2021.1-5.el9.noarch 2024-09-10T19:01:13.243 INFO:teuthology.orchestra.run.smithi183.stdout: python3-requests-2.25.1-8.el9.noarch 2024-09-10T19:01:13.243 INFO:teuthology.orchestra.run.smithi183.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-09-10T19:01:13.243 INFO:teuthology.orchestra.run.smithi183.stdout: python3-toml-0.10.2-6.el9.noarch 2024-09-10T19:01:13.243 INFO:teuthology.orchestra.run.smithi183.stdout: python3-urllib3-1.26.5-6.el9.noarch 2024-09-10T19:01:13.244 INFO:teuthology.orchestra.run.smithi183.stdout: python3-webob-1.8.8-2.el9.noarch 2024-09-10T19:01:13.244 INFO:teuthology.orchestra.run.smithi183.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-09-10T19:01:13.244 INFO:teuthology.orchestra.run.smithi183.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-09-10T19:01:13.244 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-09-10T19:01:13.244 INFO:teuthology.orchestra.run.smithi183.stdout:Complete! 2024-09-10T19:01:13.647 DEBUG:teuthology.orchestra.run.smithi183:> sudo yum -y install ceph-base 2024-09-10T19:01:14.211 INFO:teuthology.orchestra.run.smithi183.stdout:Last metadata expiration check: 0:00:50 ago on Tue 10 Sep 2024 07:00:24 PM UTC. 2024-09-10T19:01:14.301 INFO:teuthology.orchestra.run.smithi183.stdout:Package ceph-base-2:17.2.7-1606.g5be57395.el9.x86_64 is already installed. 2024-09-10T19:01:14.342 INFO:teuthology.orchestra.run.smithi183.stdout:Dependencies resolved. 2024-09-10T19:01:14.343 INFO:teuthology.orchestra.run.smithi183.stdout:Nothing to do. 2024-09-10T19:01:14.343 INFO:teuthology.orchestra.run.smithi183.stdout:Complete! 2024-09-10T19:01:14.405 DEBUG:teuthology.orchestra.run.smithi183:> sudo yum -y install cephadm 2024-09-10T19:01:14.838 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: ceph-2:17.2.7-1606.g5be57395.el9.x86_64 34/34 2024-09-10T19:01:14.838 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : ceph-2:17.2.7-1606.g5be57395.el9.x86_64 1/34 2024-09-10T19:01:14.838 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : ceph-mds-2:17.2.7-1606.g5be57395.el9.x86_64 2/34 2024-09-10T19:01:14.838 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : ceph-mgr-2:17.2.7-1606.g5be57395.el9.x86_64 3/34 2024-09-10T19:01:14.838 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : ceph-mon-2:17.2.7-1606.g5be57395.el9.x86_64 4/34 2024-09-10T19:01:14.838 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : ceph-osd-2:17.2.7-1606.g5be57395.el9.x86_64 5/34 2024-09-10T19:01:14.839 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : ceph-mgr-modules-core-2:17.2.7-1606.g5be57395.el9. 6/34 2024-09-10T19:01:14.839 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 7/34 2024-09-10T19:01:14.839 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 8/34 2024-09-10T19:01:14.839 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-ply-3.11-14.el9.noarch 9/34 2024-09-10T19:01:14.839 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 10/34 2024-09-10T19:01:14.839 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 11/34 2024-09-10T19:01:14.839 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-urllib3-1.26.5-6.el9.noarch 12/34 2024-09-10T19:01:14.839 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 13/34 2024-09-10T19:01:14.839 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 14/34 2024-09-10T19:01:14.839 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 15/34 2024-09-10T19:01:14.839 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 16/34 2024-09-10T19:01:14.839 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 17/34 2024-09-10T19:01:14.839 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 18/34 2024-09-10T19:01:14.839 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 19/34 2024-09-10T19:01:14.840 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 20/34 2024-09-10T19:01:14.840 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 21/34 2024-09-10T19:01:14.840 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 22/34 2024-09-10T19:01:14.840 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 23/34 2024-09-10T19:01:14.840 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 24/34 2024-09-10T19:01:14.840 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 25/34 2024-09-10T19:01:14.840 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 26/34 2024-09-10T19:01:14.841 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 27/34 2024-09-10T19:01:14.841 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 28/34 2024-09-10T19:01:14.841 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 29/34 2024-09-10T19:01:14.841 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 30/34 2024-09-10T19:01:14.841 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 31/34 2024-09-10T19:01:14.841 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-webob-1.8.8-2.el9.noarch 32/34 2024-09-10T19:01:14.841 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 33/34 2024-09-10T19:01:14.959 INFO:teuthology.orchestra.run.smithi183.stdout:Last metadata expiration check: 0:00:50 ago on Tue 10 Sep 2024 07:00:24 PM UTC. 2024-09-10T19:01:15.091 INFO:teuthology.orchestra.run.smithi183.stdout:Dependencies resolved. 2024-09-10T19:01:15.092 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2024-09-10T19:01:15.092 INFO:teuthology.orchestra.run.smithi183.stdout: Package Arch Version Repository Size 2024-09-10T19:01:15.092 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2024-09-10T19:01:15.092 INFO:teuthology.orchestra.run.smithi183.stdout:Installing: 2024-09-10T19:01:15.092 INFO:teuthology.orchestra.run.smithi183.stdout: cephadm noarch 2:17.2.7-1606.g5be57395.el9 ceph-noarch 83 k 2024-09-10T19:01:15.093 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-09-10T19:01:15.093 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction Summary 2024-09-10T19:01:15.093 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2024-09-10T19:01:15.093 INFO:teuthology.orchestra.run.smithi183.stdout:Install 1 Package 2024-09-10T19:01:15.093 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-09-10T19:01:15.093 INFO:teuthology.orchestra.run.smithi183.stdout:Total download size: 83 k 2024-09-10T19:01:15.093 INFO:teuthology.orchestra.run.smithi183.stdout:Installed size: 375 k 2024-09-10T19:01:15.093 INFO:teuthology.orchestra.run.smithi183.stdout:Downloading Packages: 2024-09-10T19:01:15.281 INFO:teuthology.orchestra.run.smithi183.stdout:cephadm-17.2.7-1606.g5be57395.el9.noarch.rpm 440 kB/s | 83 kB 00:00 2024-09-10T19:01:15.282 INFO:teuthology.orchestra.run.smithi183.stdout:-------------------------------------------------------------------------------- 2024-09-10T19:01:15.282 INFO:teuthology.orchestra.run.smithi183.stdout:Total 436 kB/s | 83 kB 00:00 2024-09-10T19:01:15.282 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction check 2024-09-10T19:01:15.287 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction check succeeded. 2024-09-10T19:01:15.287 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction test 2024-09-10T19:01:15.289 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction test succeeded. 2024-09-10T19:01:15.289 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction 2024-09-10T19:01:15.366 INFO:teuthology.orchestra.run.smithi183.stdout: Preparing : 1/1 2024-09-10T19:01:15.737 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 34/34 2024-09-10T19:01:15.737 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-10T19:01:15.737 INFO:teuthology.orchestra.run.smithi191.stdout:Installed: 2024-09-10T19:01:15.737 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-2:17.2.7-1606.g5be57395.el9.x86_64 2024-09-10T19:01:15.737 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-mds-2:17.2.7-1606.g5be57395.el9.x86_64 2024-09-10T19:01:15.737 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-mgr-2:17.2.7-1606.g5be57395.el9.x86_64 2024-09-10T19:01:15.737 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-mgr-modules-core-2:17.2.7-1606.g5be57395.el9.noarch 2024-09-10T19:01:15.737 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-mon-2:17.2.7-1606.g5be57395.el9.x86_64 2024-09-10T19:01:15.737 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-osd-2:17.2.7-1606.g5be57395.el9.x86_64 2024-09-10T19:01:15.738 INFO:teuthology.orchestra.run.smithi191.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-09-10T19:01:15.738 INFO:teuthology.orchestra.run.smithi191.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-09-10T19:01:15.738 INFO:teuthology.orchestra.run.smithi191.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-09-10T19:01:15.738 INFO:teuthology.orchestra.run.smithi191.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-09-10T19:01:15.738 INFO:teuthology.orchestra.run.smithi191.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-09-10T19:01:15.738 INFO:teuthology.orchestra.run.smithi191.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-09-10T19:01:15.738 INFO:teuthology.orchestra.run.smithi191.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-09-10T19:01:15.738 INFO:teuthology.orchestra.run.smithi191.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-09-10T19:01:15.738 INFO:teuthology.orchestra.run.smithi191.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-09-10T19:01:15.738 INFO:teuthology.orchestra.run.smithi191.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-09-10T19:01:15.738 INFO:teuthology.orchestra.run.smithi191.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-09-10T19:01:15.738 INFO:teuthology.orchestra.run.smithi191.stdout: python3-mako-1.1.4-6.el9.noarch 2024-09-10T19:01:15.738 INFO:teuthology.orchestra.run.smithi191.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-09-10T19:01:15.739 INFO:teuthology.orchestra.run.smithi191.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-09-10T19:01:15.739 INFO:teuthology.orchestra.run.smithi191.stdout: python3-packaging-20.9-5.el9.noarch 2024-09-10T19:01:15.739 INFO:teuthology.orchestra.run.smithi191.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-09-10T19:01:15.739 INFO:teuthology.orchestra.run.smithi191.stdout: python3-ply-3.11-14.el9.noarch 2024-09-10T19:01:15.739 INFO:teuthology.orchestra.run.smithi191.stdout: python3-portend-3.1.0-2.el9.noarch 2024-09-10T19:01:15.739 INFO:teuthology.orchestra.run.smithi191.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-09-10T19:01:15.739 INFO:teuthology.orchestra.run.smithi191.stdout: python3-pycparser-2.20-6.el9.noarch 2024-09-10T19:01:15.739 INFO:teuthology.orchestra.run.smithi191.stdout: python3-pytz-2021.1-5.el9.noarch 2024-09-10T19:01:15.739 INFO:teuthology.orchestra.run.smithi191.stdout: python3-requests-2.25.1-8.el9.noarch 2024-09-10T19:01:15.739 INFO:teuthology.orchestra.run.smithi191.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-09-10T19:01:15.739 INFO:teuthology.orchestra.run.smithi191.stdout: python3-toml-0.10.2-6.el9.noarch 2024-09-10T19:01:15.739 INFO:teuthology.orchestra.run.smithi191.stdout: python3-urllib3-1.26.5-6.el9.noarch 2024-09-10T19:01:15.739 INFO:teuthology.orchestra.run.smithi191.stdout: python3-webob-1.8.8-2.el9.noarch 2024-09-10T19:01:15.739 INFO:teuthology.orchestra.run.smithi191.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-09-10T19:01:15.740 INFO:teuthology.orchestra.run.smithi191.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-09-10T19:01:15.740 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-10T19:01:15.740 INFO:teuthology.orchestra.run.smithi191.stdout:Complete! 2024-09-10T19:01:15.806 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: cephadm-2:17.2.7-1606.g5be57395.el9.noarch 1/1 2024-09-10T19:01:15.902 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : cephadm-2:17.2.7-1606.g5be57395.el9.noarch 1/1 2024-09-10T19:01:16.098 DEBUG:teuthology.orchestra.run.smithi191:> sudo yum -y install ceph-base 2024-09-10T19:01:16.274 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: cephadm-2:17.2.7-1606.g5be57395.el9.noarch 1/1 2024-09-10T19:01:16.478 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : cephadm-2:17.2.7-1606.g5be57395.el9.noarch 1/1 2024-09-10T19:01:16.478 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-09-10T19:01:16.478 INFO:teuthology.orchestra.run.smithi183.stdout:Installed: 2024-09-10T19:01:16.478 INFO:teuthology.orchestra.run.smithi183.stdout: cephadm-2:17.2.7-1606.g5be57395.el9.noarch 2024-09-10T19:01:16.478 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-09-10T19:01:16.478 INFO:teuthology.orchestra.run.smithi183.stdout:Complete! 2024-09-10T19:01:16.666 INFO:teuthology.orchestra.run.smithi191.stdout:Last metadata expiration check: 0:00:51 ago on Tue 10 Sep 2024 07:00:25 PM UTC. 2024-09-10T19:01:16.703 DEBUG:teuthology.orchestra.run.smithi183:> sudo yum -y install ceph-immutable-object-cache 2024-09-10T19:01:16.756 INFO:teuthology.orchestra.run.smithi191.stdout:Package ceph-base-2:17.2.7-1606.g5be57395.el9.x86_64 is already installed. 2024-09-10T19:01:16.799 INFO:teuthology.orchestra.run.smithi191.stdout:Dependencies resolved. 2024-09-10T19:01:16.799 INFO:teuthology.orchestra.run.smithi191.stdout:Nothing to do. 2024-09-10T19:01:16.800 INFO:teuthology.orchestra.run.smithi191.stdout:Complete! 2024-09-10T19:01:16.854 DEBUG:teuthology.orchestra.run.smithi191:> sudo yum -y install cephadm 2024-09-10T19:01:17.283 INFO:teuthology.orchestra.run.smithi183.stdout:Last metadata expiration check: 0:00:53 ago on Tue 10 Sep 2024 07:00:24 PM UTC. 2024-09-10T19:01:17.416 INFO:teuthology.orchestra.run.smithi191.stdout:Last metadata expiration check: 0:00:52 ago on Tue 10 Sep 2024 07:00:25 PM UTC. 2024-09-10T19:01:17.416 INFO:teuthology.orchestra.run.smithi183.stdout:Dependencies resolved. 2024-09-10T19:01:17.416 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2024-09-10T19:01:17.417 INFO:teuthology.orchestra.run.smithi183.stdout: Package Arch Version Repo Size 2024-09-10T19:01:17.417 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2024-09-10T19:01:17.417 INFO:teuthology.orchestra.run.smithi183.stdout:Installing: 2024-09-10T19:01:17.417 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-immutable-object-cache x86_64 2:17.2.7-1606.g5be57395.el9 ceph 137 k 2024-09-10T19:01:17.417 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-09-10T19:01:17.417 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction Summary 2024-09-10T19:01:17.417 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2024-09-10T19:01:17.417 INFO:teuthology.orchestra.run.smithi183.stdout:Install 1 Package 2024-09-10T19:01:17.417 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-09-10T19:01:17.417 INFO:teuthology.orchestra.run.smithi183.stdout:Total download size: 137 k 2024-09-10T19:01:17.417 INFO:teuthology.orchestra.run.smithi183.stdout:Installed size: 379 k 2024-09-10T19:01:17.417 INFO:teuthology.orchestra.run.smithi183.stdout:Downloading Packages: 2024-09-10T19:01:17.547 INFO:teuthology.orchestra.run.smithi191.stdout:Dependencies resolved. 2024-09-10T19:01:17.548 INFO:teuthology.orchestra.run.smithi191.stdout:================================================================================ 2024-09-10T19:01:17.548 INFO:teuthology.orchestra.run.smithi191.stdout: Package Arch Version Repository Size 2024-09-10T19:01:17.548 INFO:teuthology.orchestra.run.smithi191.stdout:================================================================================ 2024-09-10T19:01:17.548 INFO:teuthology.orchestra.run.smithi191.stdout:Installing: 2024-09-10T19:01:17.548 INFO:teuthology.orchestra.run.smithi191.stdout: cephadm noarch 2:17.2.7-1606.g5be57395.el9 ceph-noarch 83 k 2024-09-10T19:01:17.548 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-10T19:01:17.548 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction Summary 2024-09-10T19:01:17.548 INFO:teuthology.orchestra.run.smithi191.stdout:================================================================================ 2024-09-10T19:01:17.548 INFO:teuthology.orchestra.run.smithi191.stdout:Install 1 Package 2024-09-10T19:01:17.548 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-10T19:01:17.548 INFO:teuthology.orchestra.run.smithi191.stdout:Total download size: 83 k 2024-09-10T19:01:17.548 INFO:teuthology.orchestra.run.smithi191.stdout:Installed size: 375 k 2024-09-10T19:01:17.549 INFO:teuthology.orchestra.run.smithi191.stdout:Downloading Packages: 2024-09-10T19:01:17.620 INFO:teuthology.orchestra.run.smithi183.stdout:ceph-immutable-object-cache-17.2.7-1606.g5be573 678 kB/s | 137 kB 00:00 2024-09-10T19:01:17.621 INFO:teuthology.orchestra.run.smithi183.stdout:-------------------------------------------------------------------------------- 2024-09-10T19:01:17.622 INFO:teuthology.orchestra.run.smithi183.stdout:Total 672 kB/s | 137 kB 00:00 2024-09-10T19:01:17.622 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction check 2024-09-10T19:01:17.629 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction check succeeded. 2024-09-10T19:01:17.629 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction test 2024-09-10T19:01:17.691 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction test succeeded. 2024-09-10T19:01:17.691 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction 2024-09-10T19:01:17.750 INFO:teuthology.orchestra.run.smithi191.stdout:cephadm-17.2.7-1606.g5be57395.el9.noarch.rpm 410 kB/s | 83 kB 00:00 2024-09-10T19:01:17.751 INFO:teuthology.orchestra.run.smithi191.stdout:-------------------------------------------------------------------------------- 2024-09-10T19:01:17.751 INFO:teuthology.orchestra.run.smithi191.stdout:Total 407 kB/s | 83 kB 00:00 2024-09-10T19:01:17.751 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction check 2024-09-10T19:01:17.757 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction check succeeded. 2024-09-10T19:01:17.757 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction test 2024-09-10T19:01:17.759 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction test succeeded. 2024-09-10T19:01:17.759 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction 2024-09-10T19:01:17.839 INFO:teuthology.orchestra.run.smithi191.stdout: Preparing : 1/1 2024-09-10T19:01:17.856 INFO:teuthology.orchestra.run.smithi183.stdout: Preparing : 1/1 2024-09-10T19:01:17.922 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : ceph-immutable-object-cache-2:17.2.7-1606.g5be57395. 1/1 2024-09-10T19:01:17.946 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: ceph-immutable-object-cache-2:17.2.7-1606.g5be57395. 1/1 2024-09-10T19:01:17.946 INFO:teuthology.orchestra.run.smithi183.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-10T19:01:17.946 INFO:teuthology.orchestra.run.smithi183.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-09-10T19:01:17.946 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-09-10T19:01:18.293 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: cephadm-2:17.2.7-1606.g5be57395.el9.noarch 1/1 2024-09-10T19:01:18.396 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : cephadm-2:17.2.7-1606.g5be57395.el9.noarch 1/1 2024-09-10T19:01:18.719 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: cephadm-2:17.2.7-1606.g5be57395.el9.noarch 1/1 2024-09-10T19:01:18.836 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : ceph-immutable-object-cache-2:17.2.7-1606.g5be57395. 1/1 2024-09-10T19:01:18.836 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-09-10T19:01:18.836 INFO:teuthology.orchestra.run.smithi183.stdout:Installed: 2024-09-10T19:01:18.836 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-immutable-object-cache-2:17.2.7-1606.g5be57395.el9.x86_64 2024-09-10T19:01:18.836 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-09-10T19:01:18.837 INFO:teuthology.orchestra.run.smithi183.stdout:Complete! 2024-09-10T19:01:18.997 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : cephadm-2:17.2.7-1606.g5be57395.el9.noarch 1/1 2024-09-10T19:01:18.997 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-10T19:01:18.997 INFO:teuthology.orchestra.run.smithi191.stdout:Installed: 2024-09-10T19:01:18.997 INFO:teuthology.orchestra.run.smithi191.stdout: cephadm-2:17.2.7-1606.g5be57395.el9.noarch 2024-09-10T19:01:18.997 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-10T19:01:18.997 INFO:teuthology.orchestra.run.smithi191.stdout:Complete! 2024-09-10T19:01:19.063 DEBUG:teuthology.orchestra.run.smithi183:> sudo yum -y install ceph-mgr 2024-09-10T19:01:19.199 DEBUG:teuthology.orchestra.run.smithi191:> sudo yum -y install ceph-immutable-object-cache 2024-09-10T19:01:19.634 INFO:teuthology.orchestra.run.smithi183.stdout:Last metadata expiration check: 0:00:55 ago on Tue 10 Sep 2024 07:00:24 PM UTC. 2024-09-10T19:01:19.728 INFO:teuthology.orchestra.run.smithi183.stdout:Package ceph-mgr-2:17.2.7-1606.g5be57395.el9.x86_64 is already installed. 2024-09-10T19:01:19.771 INFO:teuthology.orchestra.run.smithi183.stdout:Dependencies resolved. 2024-09-10T19:01:19.772 INFO:teuthology.orchestra.run.smithi183.stdout:Nothing to do. 2024-09-10T19:01:19.773 INFO:teuthology.orchestra.run.smithi183.stdout:Complete! 2024-09-10T19:01:19.773 INFO:teuthology.orchestra.run.smithi191.stdout:Last metadata expiration check: 0:00:54 ago on Tue 10 Sep 2024 07:00:25 PM UTC. 2024-09-10T19:01:19.839 DEBUG:teuthology.orchestra.run.smithi183:> sudo yum -y install ceph-mgr-dashboard 2024-09-10T19:01:19.907 INFO:teuthology.orchestra.run.smithi191.stdout:Dependencies resolved. 2024-09-10T19:01:19.908 INFO:teuthology.orchestra.run.smithi191.stdout:================================================================================ 2024-09-10T19:01:19.908 INFO:teuthology.orchestra.run.smithi191.stdout: Package Arch Version Repo Size 2024-09-10T19:01:19.908 INFO:teuthology.orchestra.run.smithi191.stdout:================================================================================ 2024-09-10T19:01:19.909 INFO:teuthology.orchestra.run.smithi191.stdout:Installing: 2024-09-10T19:01:19.909 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-immutable-object-cache x86_64 2:17.2.7-1606.g5be57395.el9 ceph 137 k 2024-09-10T19:01:19.909 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-10T19:01:19.909 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction Summary 2024-09-10T19:01:19.909 INFO:teuthology.orchestra.run.smithi191.stdout:================================================================================ 2024-09-10T19:01:19.909 INFO:teuthology.orchestra.run.smithi191.stdout:Install 1 Package 2024-09-10T19:01:19.909 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-10T19:01:19.909 INFO:teuthology.orchestra.run.smithi191.stdout:Total download size: 137 k 2024-09-10T19:01:19.909 INFO:teuthology.orchestra.run.smithi191.stdout:Installed size: 379 k 2024-09-10T19:01:19.909 INFO:teuthology.orchestra.run.smithi191.stdout:Downloading Packages: 2024-09-10T19:01:20.107 INFO:teuthology.orchestra.run.smithi191.stdout:ceph-immutable-object-cache-17.2.7-1606.g5be573 698 kB/s | 137 kB 00:00 2024-09-10T19:01:20.107 INFO:teuthology.orchestra.run.smithi191.stdout:-------------------------------------------------------------------------------- 2024-09-10T19:01:20.108 INFO:teuthology.orchestra.run.smithi191.stdout:Total 692 kB/s | 137 kB 00:00 2024-09-10T19:01:20.108 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction check 2024-09-10T19:01:20.115 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction check succeeded. 2024-09-10T19:01:20.115 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction test 2024-09-10T19:01:20.177 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction test succeeded. 2024-09-10T19:01:20.177 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction 2024-09-10T19:01:20.327 INFO:teuthology.orchestra.run.smithi191.stdout: Preparing : 1/1 2024-09-10T19:01:20.408 INFO:teuthology.orchestra.run.smithi183.stdout:Last metadata expiration check: 0:00:56 ago on Tue 10 Sep 2024 07:00:24 PM UTC. 2024-09-10T19:01:20.416 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : ceph-immutable-object-cache-2:17.2.7-1606.g5be57395. 1/1 2024-09-10T19:01:20.441 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: ceph-immutable-object-cache-2:17.2.7-1606.g5be57395. 1/1 2024-09-10T19:01:20.442 INFO:teuthology.orchestra.run.smithi191.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-10T19:01:20.442 INFO:teuthology.orchestra.run.smithi191.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-09-10T19:01:20.442 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-10T19:01:20.541 INFO:teuthology.orchestra.run.smithi183.stdout:Dependencies resolved. 2024-09-10T19:01:20.542 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2024-09-10T19:01:20.544 INFO:teuthology.orchestra.run.smithi183.stdout: Package Arch Version Repository Size 2024-09-10T19:01:20.544 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2024-09-10T19:01:20.544 INFO:teuthology.orchestra.run.smithi183.stdout:Installing: 2024-09-10T19:01:20.544 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-mgr-dashboard noarch 2:17.2.7-1606.g5be57395.el9 ceph-noarch 3.4 M 2024-09-10T19:01:20.544 INFO:teuthology.orchestra.run.smithi183.stdout:Installing dependencies: 2024-09-10T19:01:20.544 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-grafana-dashboards noarch 2:17.2.7-1606.g5be57395.el9 ceph-noarch 24 k 2024-09-10T19:01:20.544 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-prometheus-alerts noarch 2:17.2.7-1606.g5be57395.el9 ceph-noarch 15 k 2024-09-10T19:01:20.544 INFO:teuthology.orchestra.run.smithi183.stdout: python3-jwt noarch 2.4.0-1.el9 epel 41 k 2024-09-10T19:01:20.544 INFO:teuthology.orchestra.run.smithi183.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-09-10T19:01:20.544 INFO:teuthology.orchestra.run.smithi183.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-09-10T19:01:20.545 INFO:teuthology.orchestra.run.smithi183.stdout:Installing weak dependencies: 2024-09-10T19:01:20.545 INFO:teuthology.orchestra.run.smithi183.stdout: python3-jwt+crypto noarch 2.4.0-1.el9 epel 9.0 k 2024-09-10T19:01:20.545 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-09-10T19:01:20.545 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction Summary 2024-09-10T19:01:20.545 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2024-09-10T19:01:20.545 INFO:teuthology.orchestra.run.smithi183.stdout:Install 7 Packages 2024-09-10T19:01:20.545 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-09-10T19:01:20.546 INFO:teuthology.orchestra.run.smithi183.stdout:Total download size: 3.7 M 2024-09-10T19:01:20.546 INFO:teuthology.orchestra.run.smithi183.stdout:Installed size: 80 M 2024-09-10T19:01:20.546 INFO:teuthology.orchestra.run.smithi183.stdout:Downloading Packages: 2024-09-10T19:01:20.758 INFO:teuthology.orchestra.run.smithi183.stdout:(1/7): ceph-prometheus-alerts-17.2.7-1606.g5be5 94 kB/s | 15 kB 00:00 2024-09-10T19:01:20.783 INFO:teuthology.orchestra.run.smithi183.stdout:(2/7): ceph-grafana-dashboards-17.2.7-1606.g5be 136 kB/s | 24 kB 00:00 2024-09-10T19:01:20.825 INFO:teuthology.orchestra.run.smithi183.stdout:(3/7): python3-jwt+crypto-2.4.0-1.el9.noarch.rp 135 kB/s | 9.0 kB 00:00 2024-09-10T19:01:20.850 INFO:teuthology.orchestra.run.smithi183.stdout:(4/7): python3-repoze-lru-0.7-16.el9.noarch.rpm 1.3 MB/s | 31 kB 00:00 2024-09-10T19:01:20.875 INFO:teuthology.orchestra.run.smithi183.stdout:(5/7): python3-jwt-2.4.0-1.el9.noarch.rpm 443 kB/s | 41 kB 00:00 2024-09-10T19:01:20.901 INFO:teuthology.orchestra.run.smithi183.stdout:(6/7): python3-routes-2.5.1-5.el9.noarch.rpm 3.7 MB/s | 188 kB 00:00 2024-09-10T19:01:20.968 INFO:teuthology.orchestra.run.smithi183.stdout:(7/7): ceph-mgr-dashboard-17.2.7-1606.g5be57395 9.2 MB/s | 3.4 MB 00:00 2024-09-10T19:01:20.971 INFO:teuthology.orchestra.run.smithi183.stdout:-------------------------------------------------------------------------------- 2024-09-10T19:01:20.971 INFO:teuthology.orchestra.run.smithi183.stdout:Total 8.5 MB/s | 3.7 MB 00:00 2024-09-10T19:01:20.971 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction check 2024-09-10T19:01:20.979 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction check succeeded. 2024-09-10T19:01:20.979 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction test 2024-09-10T19:01:21.084 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction test succeeded. 2024-09-10T19:01:21.085 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction 2024-09-10T19:01:21.347 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : ceph-immutable-object-cache-2:17.2.7-1606.g5be57395. 1/1 2024-09-10T19:01:21.348 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-10T19:01:21.348 INFO:teuthology.orchestra.run.smithi191.stdout:Installed: 2024-09-10T19:01:21.348 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-immutable-object-cache-2:17.2.7-1606.g5be57395.el9.x86_64 2024-09-10T19:01:21.348 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-10T19:01:21.348 INFO:teuthology.orchestra.run.smithi191.stdout:Complete! 2024-09-10T19:01:21.411 INFO:teuthology.orchestra.run.smithi183.stdout: Preparing : 1/1 2024-09-10T19:01:21.485 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-jwt-2.4.0-1.el9.noarch 1/7 2024-09-10T19:01:21.520 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-jwt+crypto-2.4.0-1.el9.noarch 2/7 2024-09-10T19:01:21.551 DEBUG:teuthology.orchestra.run.smithi191:> sudo yum -y install ceph-mgr 2024-09-10T19:01:21.568 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/7 2024-09-10T19:01:21.606 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/7 2024-09-10T19:01:21.635 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : ceph-prometheus-alerts-2:17.2.7-1606.g5be57395.el9.n 5/7 2024-09-10T19:01:22.129 INFO:teuthology.orchestra.run.smithi191.stdout:Last metadata expiration check: 0:00:57 ago on Tue 10 Sep 2024 07:00:25 PM UTC. 2024-09-10T19:01:22.223 INFO:teuthology.orchestra.run.smithi191.stdout:Package ceph-mgr-2:17.2.7-1606.g5be57395.el9.x86_64 is already installed. 2024-09-10T19:01:22.266 INFO:teuthology.orchestra.run.smithi191.stdout:Dependencies resolved. 2024-09-10T19:01:22.267 INFO:teuthology.orchestra.run.smithi191.stdout:Nothing to do. 2024-09-10T19:01:22.268 INFO:teuthology.orchestra.run.smithi191.stdout:Complete! 2024-09-10T19:01:22.327 DEBUG:teuthology.orchestra.run.smithi191:> sudo yum -y install ceph-mgr-dashboard 2024-09-10T19:01:22.631 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : ceph-grafana-dashboards-2:17.2.7-1606.g5be57395.el9. 6/7 2024-09-10T19:01:22.672 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : ceph-mgr-dashboard-2:17.2.7-1606.g5be57395.el9.noarc 7/7 2024-09-10T19:01:22.887 INFO:teuthology.orchestra.run.smithi191.stdout:Last metadata expiration check: 0:00:57 ago on Tue 10 Sep 2024 07:00:25 PM UTC. 2024-09-10T19:01:23.019 INFO:teuthology.orchestra.run.smithi191.stdout:Dependencies resolved. 2024-09-10T19:01:23.020 INFO:teuthology.orchestra.run.smithi191.stdout:================================================================================ 2024-09-10T19:01:23.020 INFO:teuthology.orchestra.run.smithi191.stdout: Package Arch Version Repository Size 2024-09-10T19:01:23.020 INFO:teuthology.orchestra.run.smithi191.stdout:================================================================================ 2024-09-10T19:01:23.020 INFO:teuthology.orchestra.run.smithi191.stdout:Installing: 2024-09-10T19:01:23.020 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-mgr-dashboard noarch 2:17.2.7-1606.g5be57395.el9 ceph-noarch 3.4 M 2024-09-10T19:01:23.020 INFO:teuthology.orchestra.run.smithi191.stdout:Installing dependencies: 2024-09-10T19:01:23.020 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-grafana-dashboards noarch 2:17.2.7-1606.g5be57395.el9 ceph-noarch 24 k 2024-09-10T19:01:23.020 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-prometheus-alerts noarch 2:17.2.7-1606.g5be57395.el9 ceph-noarch 15 k 2024-09-10T19:01:23.021 INFO:teuthology.orchestra.run.smithi191.stdout: python3-jwt noarch 2.4.0-1.el9 epel 41 k 2024-09-10T19:01:23.021 INFO:teuthology.orchestra.run.smithi191.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-09-10T19:01:23.021 INFO:teuthology.orchestra.run.smithi191.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-09-10T19:01:23.021 INFO:teuthology.orchestra.run.smithi191.stdout:Installing weak dependencies: 2024-09-10T19:01:23.021 INFO:teuthology.orchestra.run.smithi191.stdout: python3-jwt+crypto noarch 2.4.0-1.el9 epel 9.0 k 2024-09-10T19:01:23.021 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-10T19:01:23.021 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction Summary 2024-09-10T19:01:23.021 INFO:teuthology.orchestra.run.smithi191.stdout:================================================================================ 2024-09-10T19:01:23.021 INFO:teuthology.orchestra.run.smithi191.stdout:Install 7 Packages 2024-09-10T19:01:23.021 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-10T19:01:23.022 INFO:teuthology.orchestra.run.smithi191.stdout:Total download size: 3.7 M 2024-09-10T19:01:23.022 INFO:teuthology.orchestra.run.smithi191.stdout:Installed size: 80 M 2024-09-10T19:01:23.022 INFO:teuthology.orchestra.run.smithi191.stdout:Downloading Packages: 2024-09-10T19:01:23.235 INFO:teuthology.orchestra.run.smithi191.stdout:(1/7): ceph-prometheus-alerts-17.2.7-1606.g5be5 95 kB/s | 15 kB 00:00 2024-09-10T19:01:23.252 INFO:teuthology.orchestra.run.smithi191.stdout:(2/7): ceph-grafana-dashboards-17.2.7-1606.g5be 142 kB/s | 24 kB 00:00 2024-09-10T19:01:23.294 INFO:teuthology.orchestra.run.smithi191.stdout:(3/7): python3-jwt+crypto-2.4.0-1.el9.noarch.rp 154 kB/s | 9.0 kB 00:00 2024-09-10T19:01:23.319 INFO:teuthology.orchestra.run.smithi191.stdout:(4/7): python3-repoze-lru-0.7-16.el9.noarch.rpm 1.3 MB/s | 31 kB 00:00 2024-09-10T19:01:23.344 INFO:teuthology.orchestra.run.smithi191.stdout:(5/7): python3-jwt-2.4.0-1.el9.noarch.rpm 444 kB/s | 41 kB 00:00 2024-09-10T19:01:23.370 INFO:teuthology.orchestra.run.smithi191.stdout:(6/7): python3-routes-2.5.1-5.el9.noarch.rpm 3.7 MB/s | 188 kB 00:00 2024-09-10T19:01:23.437 INFO:teuthology.orchestra.run.smithi191.stdout:(7/7): ceph-mgr-dashboard-17.2.7-1606.g5be57395 9.4 MB/s | 3.4 MB 00:00 2024-09-10T19:01:23.440 INFO:teuthology.orchestra.run.smithi191.stdout:-------------------------------------------------------------------------------- 2024-09-10T19:01:23.440 INFO:teuthology.orchestra.run.smithi191.stdout:Total 8.7 MB/s | 3.7 MB 00:00 2024-09-10T19:01:23.441 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction check 2024-09-10T19:01:23.449 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction check succeeded. 2024-09-10T19:01:23.449 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction test 2024-09-10T19:01:23.527 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction test succeeded. 2024-09-10T19:01:23.528 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction 2024-09-10T19:01:23.756 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: ceph-mgr-dashboard-2:17.2.7-1606.g5be57395.el9.noarc 7/7 2024-09-10T19:01:23.756 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : ceph-grafana-dashboards-2:17.2.7-1606.g5be57395.el9. 1/7 2024-09-10T19:01:23.756 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : ceph-mgr-dashboard-2:17.2.7-1606.g5be57395.el9.noarc 2/7 2024-09-10T19:01:23.756 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : ceph-prometheus-alerts-2:17.2.7-1606.g5be57395.el9.n 3/7 2024-09-10T19:01:23.756 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-jwt+crypto-2.4.0-1.el9.noarch 4/7 2024-09-10T19:01:23.756 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-jwt-2.4.0-1.el9.noarch 5/7 2024-09-10T19:01:23.756 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 6/7 2024-09-10T19:01:23.844 INFO:teuthology.orchestra.run.smithi191.stdout: Preparing : 1/1 2024-09-10T19:01:23.904 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-jwt-2.4.0-1.el9.noarch 1/7 2024-09-10T19:01:23.947 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-jwt+crypto-2.4.0-1.el9.noarch 2/7 2024-09-10T19:01:24.001 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/7 2024-09-10T19:01:24.036 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 7/7 2024-09-10T19:01:24.036 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-09-10T19:01:24.036 INFO:teuthology.orchestra.run.smithi183.stdout:Installed: 2024-09-10T19:01:24.036 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-grafana-dashboards-2:17.2.7-1606.g5be57395.el9.noarch 2024-09-10T19:01:24.036 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-mgr-dashboard-2:17.2.7-1606.g5be57395.el9.noarch 2024-09-10T19:01:24.037 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-prometheus-alerts-2:17.2.7-1606.g5be57395.el9.noarch 2024-09-10T19:01:24.037 INFO:teuthology.orchestra.run.smithi183.stdout: python3-jwt-2.4.0-1.el9.noarch 2024-09-10T19:01:24.037 INFO:teuthology.orchestra.run.smithi183.stdout: python3-jwt+crypto-2.4.0-1.el9.noarch 2024-09-10T19:01:24.037 INFO:teuthology.orchestra.run.smithi183.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-09-10T19:01:24.037 INFO:teuthology.orchestra.run.smithi183.stdout: python3-routes-2.5.1-5.el9.noarch 2024-09-10T19:01:24.037 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-09-10T19:01:24.037 INFO:teuthology.orchestra.run.smithi183.stdout:Complete! 2024-09-10T19:01:24.042 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/7 2024-09-10T19:01:24.070 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : ceph-prometheus-alerts-2:17.2.7-1606.g5be57395.el9.n 5/7 2024-09-10T19:01:24.260 DEBUG:teuthology.orchestra.run.smithi183:> sudo yum -y install ceph-mgr-diskprediction-local 2024-09-10T19:01:24.824 INFO:teuthology.orchestra.run.smithi183.stdout:Last metadata expiration check: 0:01:00 ago on Tue 10 Sep 2024 07:00:24 PM UTC. 2024-09-10T19:01:24.957 INFO:teuthology.orchestra.run.smithi183.stdout:Dependencies resolved. 2024-09-10T19:01:24.957 INFO:teuthology.orchestra.run.smithi183.stdout:======================================================================================= 2024-09-10T19:01:24.958 INFO:teuthology.orchestra.run.smithi183.stdout: Package Arch Version Repository Size 2024-09-10T19:01:24.958 INFO:teuthology.orchestra.run.smithi183.stdout:======================================================================================= 2024-09-10T19:01:24.958 INFO:teuthology.orchestra.run.smithi183.stdout:Installing: 2024-09-10T19:01:24.958 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-mgr-diskprediction-local noarch 2:17.2.7-1606.g5be57395.el9 ceph-noarch 7.4 M 2024-09-10T19:01:24.958 INFO:teuthology.orchestra.run.smithi183.stdout:Installing dependencies: 2024-09-10T19:01:24.958 INFO:teuthology.orchestra.run.smithi183.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-09-10T19:01:24.958 INFO:teuthology.orchestra.run.smithi183.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-09-10T19:01:24.958 INFO:teuthology.orchestra.run.smithi183.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-09-10T19:01:24.958 INFO:teuthology.orchestra.run.smithi183.stdout: libgfortran x86_64 11.5.0-2.el9 baseos 801 k 2024-09-10T19:01:24.958 INFO:teuthology.orchestra.run.smithi183.stdout: libquadmath x86_64 11.5.0-2.el9 baseos 191 k 2024-09-10T19:01:24.959 INFO:teuthology.orchestra.run.smithi183.stdout: openblas x86_64 0.3.26-2.el9 appstream 38 k 2024-09-10T19:01:24.959 INFO:teuthology.orchestra.run.smithi183.stdout: openblas-openmp x86_64 0.3.26-2.el9 appstream 4.9 M 2024-09-10T19:01:24.959 INFO:teuthology.orchestra.run.smithi183.stdout: python3-devel x86_64 3.9.19-8.el9 appstream 245 k 2024-09-10T19:01:24.959 INFO:teuthology.orchestra.run.smithi183.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-09-10T19:01:24.959 INFO:teuthology.orchestra.run.smithi183.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-09-10T19:01:24.959 INFO:teuthology.orchestra.run.smithi183.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-09-10T19:01:24.959 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-09-10T19:01:24.959 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction Summary 2024-09-10T19:01:24.959 INFO:teuthology.orchestra.run.smithi183.stdout:======================================================================================= 2024-09-10T19:01:24.959 INFO:teuthology.orchestra.run.smithi183.stdout:Install 12 Packages 2024-09-10T19:01:24.959 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-09-10T19:01:24.960 INFO:teuthology.orchestra.run.smithi183.stdout:Total download size: 38 M 2024-09-10T19:01:24.960 INFO:teuthology.orchestra.run.smithi183.stdout:Installed size: 212 M 2024-09-10T19:01:24.960 INFO:teuthology.orchestra.run.smithi183.stdout:Downloading Packages: 2024-09-10T19:01:25.049 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : ceph-grafana-dashboards-2:17.2.7-1606.g5be57395.el9. 6/7 2024-09-10T19:01:25.091 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : ceph-mgr-dashboard-2:17.2.7-1606.g5be57395.el9.noarc 7/7 2024-09-10T19:01:25.376 INFO:teuthology.orchestra.run.smithi183.stdout:(1/12): libquadmath-11.5.0-2.el9.x86_64.rpm 1.1 MB/s | 191 kB 00:00 2024-09-10T19:01:25.459 INFO:teuthology.orchestra.run.smithi183.stdout:(2/12): libgfortran-11.5.0-2.el9.x86_64.rpm 3.0 MB/s | 801 kB 00:00 2024-09-10T19:01:25.693 INFO:teuthology.orchestra.run.smithi183.stdout:(3/12): ceph-mgr-diskprediction-local-17.2.7-16 15 MB/s | 7.4 MB 00:00 2024-09-10T19:01:25.718 INFO:teuthology.orchestra.run.smithi183.stdout:(4/12): flexiblas-3.0.4-8.el9.x86_64.rpm 95 kB/s | 33 kB 00:00 2024-09-10T19:01:25.918 INFO:teuthology.orchestra.run.smithi183.stdout:(5/12): openblas-0.3.26-2.el9.x86_64.rpm 192 kB/s | 38 kB 00:00 2024-09-10T19:01:25.977 INFO:teuthology.orchestra.run.smithi183.stdout:(6/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 64 kB/s | 18 kB 00:00 2024-09-10T19:01:26.191 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: ceph-mgr-dashboard-2:17.2.7-1606.g5be57395.el9.noarc 7/7 2024-09-10T19:01:26.191 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : ceph-grafana-dashboards-2:17.2.7-1606.g5be57395.el9. 1/7 2024-09-10T19:01:26.191 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : ceph-mgr-dashboard-2:17.2.7-1606.g5be57395.el9.noarc 2/7 2024-09-10T19:01:26.191 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : ceph-prometheus-alerts-2:17.2.7-1606.g5be57395.el9.n 3/7 2024-09-10T19:01:26.191 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-jwt+crypto-2.4.0-1.el9.noarch 4/7 2024-09-10T19:01:26.191 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-jwt-2.4.0-1.el9.noarch 5/7 2024-09-10T19:01:26.191 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 6/7 2024-09-10T19:01:26.444 INFO:teuthology.orchestra.run.smithi183.stdout:(7/12): python3-devel-3.9.19-8.el9.x86_64.rpm 525 kB/s | 245 kB 00:00 2024-09-10T19:01:26.464 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 7/7 2024-09-10T19:01:26.464 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-10T19:01:26.464 INFO:teuthology.orchestra.run.smithi191.stdout:Installed: 2024-09-10T19:01:26.465 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-grafana-dashboards-2:17.2.7-1606.g5be57395.el9.noarch 2024-09-10T19:01:26.465 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-mgr-dashboard-2:17.2.7-1606.g5be57395.el9.noarch 2024-09-10T19:01:26.468 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-prometheus-alerts-2:17.2.7-1606.g5be57395.el9.noarch 2024-09-10T19:01:26.468 INFO:teuthology.orchestra.run.smithi191.stdout: python3-jwt-2.4.0-1.el9.noarch 2024-09-10T19:01:26.468 INFO:teuthology.orchestra.run.smithi191.stdout: python3-jwt+crypto-2.4.0-1.el9.noarch 2024-09-10T19:01:26.468 INFO:teuthology.orchestra.run.smithi191.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-09-10T19:01:26.468 INFO:teuthology.orchestra.run.smithi191.stdout: python3-routes-2.5.1-5.el9.noarch 2024-09-10T19:01:26.468 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-10T19:01:26.468 INFO:teuthology.orchestra.run.smithi191.stdout:Complete! 2024-09-10T19:01:26.668 DEBUG:teuthology.orchestra.run.smithi191:> sudo yum -y install ceph-mgr-diskprediction-local 2024-09-10T19:01:26.669 INFO:teuthology.orchestra.run.smithi183.stdout:(8/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 2.5 MB/s | 3.0 MB 00:01 2024-09-10T19:01:26.778 INFO:teuthology.orchestra.run.smithi183.stdout:(9/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 2.3 MB/s | 250 kB 00:00 2024-09-10T19:01:27.212 INFO:teuthology.orchestra.run.smithi183.stdout:(10/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 6.6 MB/s | 5.1 MB 00:00 2024-09-10T19:01:27.232 INFO:teuthology.orchestra.run.smithi191.stdout:Last metadata expiration check: 0:01:02 ago on Tue 10 Sep 2024 07:00:25 PM UTC. 2024-09-10T19:01:27.366 INFO:teuthology.orchestra.run.smithi191.stdout:Dependencies resolved. 2024-09-10T19:01:27.367 INFO:teuthology.orchestra.run.smithi191.stdout:======================================================================================= 2024-09-10T19:01:27.367 INFO:teuthology.orchestra.run.smithi191.stdout: Package Arch Version Repository Size 2024-09-10T19:01:27.367 INFO:teuthology.orchestra.run.smithi191.stdout:======================================================================================= 2024-09-10T19:01:27.367 INFO:teuthology.orchestra.run.smithi191.stdout:Installing: 2024-09-10T19:01:27.367 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-mgr-diskprediction-local noarch 2:17.2.7-1606.g5be57395.el9 ceph-noarch 7.4 M 2024-09-10T19:01:27.367 INFO:teuthology.orchestra.run.smithi191.stdout:Installing dependencies: 2024-09-10T19:01:27.367 INFO:teuthology.orchestra.run.smithi191.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-09-10T19:01:27.368 INFO:teuthology.orchestra.run.smithi191.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-09-10T19:01:27.368 INFO:teuthology.orchestra.run.smithi191.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-09-10T19:01:27.368 INFO:teuthology.orchestra.run.smithi191.stdout: libgfortran x86_64 11.5.0-2.el9 baseos 801 k 2024-09-10T19:01:27.368 INFO:teuthology.orchestra.run.smithi191.stdout: libquadmath x86_64 11.5.0-2.el9 baseos 191 k 2024-09-10T19:01:27.368 INFO:teuthology.orchestra.run.smithi191.stdout: openblas x86_64 0.3.26-2.el9 appstream 38 k 2024-09-10T19:01:27.368 INFO:teuthology.orchestra.run.smithi191.stdout: openblas-openmp x86_64 0.3.26-2.el9 appstream 4.9 M 2024-09-10T19:01:27.368 INFO:teuthology.orchestra.run.smithi191.stdout: python3-devel x86_64 3.9.19-8.el9 appstream 245 k 2024-09-10T19:01:27.368 INFO:teuthology.orchestra.run.smithi191.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-09-10T19:01:27.368 INFO:teuthology.orchestra.run.smithi191.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-09-10T19:01:27.368 INFO:teuthology.orchestra.run.smithi191.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-09-10T19:01:27.368 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-10T19:01:27.368 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction Summary 2024-09-10T19:01:27.368 INFO:teuthology.orchestra.run.smithi191.stdout:======================================================================================= 2024-09-10T19:01:27.368 INFO:teuthology.orchestra.run.smithi191.stdout:Install 12 Packages 2024-09-10T19:01:27.369 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-10T19:01:27.369 INFO:teuthology.orchestra.run.smithi191.stdout:Total download size: 38 M 2024-09-10T19:01:27.369 INFO:teuthology.orchestra.run.smithi191.stdout:Installed size: 212 M 2024-09-10T19:01:27.369 INFO:teuthology.orchestra.run.smithi191.stdout:Downloading Packages: 2024-09-10T19:01:28.312 INFO:teuthology.orchestra.run.smithi183.stdout:(11/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 11 MB/s | 16 MB 00:01 2024-09-10T19:01:29.078 INFO:teuthology.orchestra.run.smithi183.stdout:(12/12): openblas-openmp-0.3.26-2.el9.x86_64.rp 1.6 MB/s | 4.9 MB 00:03 2024-09-10T19:01:29.079 INFO:teuthology.orchestra.run.smithi183.stdout:-------------------------------------------------------------------------------- 2024-09-10T19:01:29.080 INFO:teuthology.orchestra.run.smithi183.stdout:Total 9.3 MB/s | 38 MB 00:04 2024-09-10T19:01:29.435 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction check 2024-09-10T19:01:29.460 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction check succeeded. 2024-09-10T19:01:29.460 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction test 2024-09-10T19:01:29.975 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction test succeeded. 2024-09-10T19:01:29.976 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction 2024-09-10T19:01:30.144 INFO:teuthology.orchestra.run.smithi191.stdout:(1/12): libquadmath-11.5.0-2.el9.x86_64.rpm 1.1 MB/s | 191 kB 00:00 2024-09-10T19:01:30.194 INFO:teuthology.orchestra.run.smithi191.stdout:(2/12): libgfortran-11.5.0-2.el9.x86_64.rpm 3.5 MB/s | 801 kB 00:00 2024-09-10T19:01:30.445 INFO:teuthology.orchestra.run.smithi191.stdout:(3/12): ceph-mgr-diskprediction-local-17.2.7-16 16 MB/s | 7.4 MB 00:00 2024-09-10T19:01:30.470 INFO:teuthology.orchestra.run.smithi191.stdout:(4/12): flexiblas-3.0.4-8.el9.x86_64.rpm 100 kB/s | 33 kB 00:00 2024-09-10T19:01:30.670 INFO:teuthology.orchestra.run.smithi191.stdout:(5/12): openblas-0.3.26-2.el9.x86_64.rpm 193 kB/s | 38 kB 00:00 2024-09-10T19:01:30.729 INFO:teuthology.orchestra.run.smithi191.stdout:(6/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 64 kB/s | 18 kB 00:00 2024-09-10T19:01:30.731 INFO:teuthology.orchestra.run.smithi183.stdout: Preparing : 1/1 2024-09-10T19:01:30.791 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 1/12 2024-09-10T19:01:30.840 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : libquadmath-11.5.0-2.el9.x86_64 2/12 2024-09-10T19:01:30.945 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : libgfortran-11.5.0-2.el9.x86_64 3/12 2024-09-10T19:01:30.978 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-devel-3.9.19-8.el9.x86_64 4/12 2024-09-10T19:01:31.104 INFO:teuthology.orchestra.run.smithi191.stdout:(7/12): python3-devel-3.9.19-8.el9.x86_64.rpm 653 kB/s | 245 kB 00:00 2024-09-10T19:01:31.170 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : openblas-0.3.26-2.el9.x86_64 5/12 2024-09-10T19:01:31.201 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : openblas-openmp-0.3.26-2.el9.x86_64 6/12 2024-09-10T19:01:31.283 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-09-10T19:01:31.437 INFO:teuthology.orchestra.run.smithi191.stdout:(8/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 2.4 MB/s | 3.0 MB 00:01 2024-09-10T19:01:31.554 INFO:teuthology.orchestra.run.smithi191.stdout:(9/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 2.1 MB/s | 250 kB 00:00 2024-09-10T19:01:31.671 INFO:teuthology.orchestra.run.smithi191.stdout:(10/12): openblas-openmp-0.3.26-2.el9.x86_64.rp 4.9 MB/s | 4.9 MB 00:01 2024-09-10T19:01:31.913 INFO:teuthology.orchestra.run.smithi191.stdout:(11/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 6.3 MB/s | 5.1 MB 00:00 2024-09-10T19:01:31.933 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-09-10T19:01:32.048 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-09-10T19:01:33.311 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-09-10T19:01:33.838 INFO:teuthology.orchestra.run.smithi191.stdout:(12/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 7.1 MB/s | 16 MB 00:02 2024-09-10T19:01:33.839 INFO:teuthology.orchestra.run.smithi191.stdout:-------------------------------------------------------------------------------- 2024-09-10T19:01:33.839 INFO:teuthology.orchestra.run.smithi191.stdout:Total 5.9 MB/s | 38 MB 00:06 2024-09-10T19:01:34.196 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction check 2024-09-10T19:01:34.222 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction check succeeded. 2024-09-10T19:01:34.222 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction test 2024-09-10T19:01:34.348 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-09-10T19:01:34.375 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : ceph-mgr-diskprediction-local-2:17.2.7-1606.g5be57 12/12 2024-09-10T19:01:34.711 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction test succeeded. 2024-09-10T19:01:34.712 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction 2024-09-10T19:01:35.464 INFO:teuthology.orchestra.run.smithi191.stdout: Preparing : 1/1 2024-09-10T19:01:35.530 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 1/12 2024-09-10T19:01:35.578 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : libquadmath-11.5.0-2.el9.x86_64 2/12 2024-09-10T19:01:35.678 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : libgfortran-11.5.0-2.el9.x86_64 3/12 2024-09-10T19:01:35.720 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-devel-3.9.19-8.el9.x86_64 4/12 2024-09-10T19:01:35.906 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : openblas-0.3.26-2.el9.x86_64 5/12 2024-09-10T19:01:35.935 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : openblas-openmp-0.3.26-2.el9.x86_64 6/12 2024-09-10T19:01:36.013 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-09-10T19:01:36.657 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-09-10T19:01:36.773 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-09-10T19:01:36.963 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:17.2.7-1606.g5be57 12/12 2024-09-10T19:01:36.963 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : ceph-mgr-diskprediction-local-2:17.2.7-1606.g5be57 1/12 2024-09-10T19:01:36.963 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : libgfortran-11.5.0-2.el9.x86_64 2/12 2024-09-10T19:01:36.963 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : libquadmath-11.5.0-2.el9.x86_64 3/12 2024-09-10T19:01:36.963 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 4/12 2024-09-10T19:01:36.964 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 5/12 2024-09-10T19:01:36.964 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 6/12 2024-09-10T19:01:36.964 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : openblas-0.3.26-2.el9.x86_64 7/12 2024-09-10T19:01:36.964 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : openblas-openmp-0.3.26-2.el9.x86_64 8/12 2024-09-10T19:01:36.964 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-devel-3.9.19-8.el9.x86_64 9/12 2024-09-10T19:01:36.964 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 10/12 2024-09-10T19:01:36.964 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 11/12 2024-09-10T19:01:37.379 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 12/12 2024-09-10T19:01:37.379 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-09-10T19:01:37.379 INFO:teuthology.orchestra.run.smithi183.stdout:Installed: 2024-09-10T19:01:37.379 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-mgr-diskprediction-local-2:17.2.7-1606.g5be57395.el9.noarch 2024-09-10T19:01:37.379 INFO:teuthology.orchestra.run.smithi183.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-09-10T19:01:37.379 INFO:teuthology.orchestra.run.smithi183.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-09-10T19:01:37.380 INFO:teuthology.orchestra.run.smithi183.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-09-10T19:01:37.380 INFO:teuthology.orchestra.run.smithi183.stdout: libgfortran-11.5.0-2.el9.x86_64 2024-09-10T19:01:37.380 INFO:teuthology.orchestra.run.smithi183.stdout: libquadmath-11.5.0-2.el9.x86_64 2024-09-10T19:01:37.380 INFO:teuthology.orchestra.run.smithi183.stdout: openblas-0.3.26-2.el9.x86_64 2024-09-10T19:01:37.380 INFO:teuthology.orchestra.run.smithi183.stdout: openblas-openmp-0.3.26-2.el9.x86_64 2024-09-10T19:01:37.380 INFO:teuthology.orchestra.run.smithi183.stdout: python3-devel-3.9.19-8.el9.x86_64 2024-09-10T19:01:37.380 INFO:teuthology.orchestra.run.smithi183.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-09-10T19:01:37.380 INFO:teuthology.orchestra.run.smithi183.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-09-10T19:01:37.380 INFO:teuthology.orchestra.run.smithi183.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-09-10T19:01:37.380 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-09-10T19:01:37.380 INFO:teuthology.orchestra.run.smithi183.stdout:Complete! 2024-09-10T19:01:37.725 DEBUG:teuthology.orchestra.run.smithi183:> sudo yum -y install ceph-mgr-rook 2024-09-10T19:01:38.027 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-09-10T19:01:38.310 INFO:teuthology.orchestra.run.smithi183.stdout:Last metadata expiration check: 0:01:14 ago on Tue 10 Sep 2024 07:00:24 PM UTC. 2024-09-10T19:01:38.454 INFO:teuthology.orchestra.run.smithi183.stdout:Dependencies resolved. 2024-09-10T19:01:38.456 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2024-09-10T19:01:38.456 INFO:teuthology.orchestra.run.smithi183.stdout: Package Arch Version Repository Size 2024-09-10T19:01:38.456 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2024-09-10T19:01:38.456 INFO:teuthology.orchestra.run.smithi183.stdout:Installing: 2024-09-10T19:01:38.456 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-mgr-rook noarch 2:17.2.7-1606.g5be57395.el9 ceph-noarch 47 k 2024-09-10T19:01:38.456 INFO:teuthology.orchestra.run.smithi183.stdout:Installing dependencies: 2024-09-10T19:01:38.456 INFO:teuthology.orchestra.run.smithi183.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-09-10T19:01:38.456 INFO:teuthology.orchestra.run.smithi183.stdout: python3-certifi noarch 2023.05.07-4.el9 epel 14 k 2024-09-10T19:01:38.456 INFO:teuthology.orchestra.run.smithi183.stdout: python3-google-auth noarch 1:2.34.0-1.el9 epel 220 k 2024-09-10T19:01:38.456 INFO:teuthology.orchestra.run.smithi183.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-09-10T19:01:38.457 INFO:teuthology.orchestra.run.smithi183.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-09-10T19:01:38.457 INFO:teuthology.orchestra.run.smithi183.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-09-10T19:01:38.457 INFO:teuthology.orchestra.run.smithi183.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-09-10T19:01:38.457 INFO:teuthology.orchestra.run.smithi183.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-09-10T19:01:38.457 INFO:teuthology.orchestra.run.smithi183.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-09-10T19:01:38.457 INFO:teuthology.orchestra.run.smithi183.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-09-10T19:01:38.457 INFO:teuthology.orchestra.run.smithi183.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-09-10T19:01:38.457 INFO:teuthology.orchestra.run.smithi183.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-09-10T19:01:38.457 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-09-10T19:01:38.457 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction Summary 2024-09-10T19:01:38.457 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2024-09-10T19:01:38.457 INFO:teuthology.orchestra.run.smithi183.stdout:Install 13 Packages 2024-09-10T19:01:38.457 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-09-10T19:01:38.458 INFO:teuthology.orchestra.run.smithi183.stdout:Total download size: 2.2 M 2024-09-10T19:01:38.458 INFO:teuthology.orchestra.run.smithi183.stdout:Installed size: 26 M 2024-09-10T19:01:38.458 INFO:teuthology.orchestra.run.smithi183.stdout:Downloading Packages: 2024-09-10T19:01:39.065 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-09-10T19:01:39.093 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : ceph-mgr-diskprediction-local-2:17.2.7-1606.g5be57 12/12 2024-09-10T19:01:39.267 INFO:teuthology.orchestra.run.smithi183.stdout:(1/13): python3-jsonpatch-1.21-16.el9.noarch.rp 228 kB/s | 26 kB 00:00 2024-09-10T19:01:39.284 INFO:teuthology.orchestra.run.smithi183.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 148 kB/s | 19 kB 00:00 2024-09-10T19:01:39.309 INFO:teuthology.orchestra.run.smithi183.stdout:(3/13): ceph-mgr-rook-17.2.7-1606.g5be57395.el9 307 kB/s | 47 kB 00:00 2024-09-10T19:01:39.410 INFO:teuthology.orchestra.run.smithi183.stdout:(4/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 1.7 MB/s | 222 kB 00:00 2024-09-10T19:01:39.435 INFO:teuthology.orchestra.run.smithi183.stdout:(5/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.0 MB/s | 159 kB 00:00 2024-09-10T19:01:39.460 INFO:teuthology.orchestra.run.smithi183.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 1.0 MB/s | 54 kB 00:00 2024-09-10T19:01:39.494 INFO:teuthology.orchestra.run.smithi183.stdout:(7/13): python3-cachetools-4.2.4-1.el9.noarch.r 547 kB/s | 32 kB 00:00 2024-09-10T19:01:39.519 INFO:teuthology.orchestra.run.smithi183.stdout:(8/13): python3-pyasn1-modules-0.4.8-6.el9.noar 1.3 MB/s | 279 kB 00:00 2024-09-10T19:01:39.536 INFO:teuthology.orchestra.run.smithi183.stdout:(9/13): python3-certifi-2023.05.07-4.el9.noarch 190 kB/s | 14 kB 00:00 2024-09-10T19:01:39.570 INFO:teuthology.orchestra.run.smithi183.stdout:(10/13): python3-google-auth-2.34.0-1.el9.noarc 2.9 MB/s | 220 kB 00:00 2024-09-10T19:01:39.595 INFO:teuthology.orchestra.run.smithi183.stdout:(11/13): python3-rsa-4.9-2.el9.noarch.rpm 1.1 MB/s | 59 kB 00:00 2024-09-10T19:01:39.620 INFO:teuthology.orchestra.run.smithi183.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 1.8 MB/s | 90 kB 00:00 2024-09-10T19:01:39.662 INFO:teuthology.orchestra.run.smithi183.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 7.2 MB/s | 1.0 MB 00:00 2024-09-10T19:01:39.665 INFO:teuthology.orchestra.run.smithi183.stdout:-------------------------------------------------------------------------------- 2024-09-10T19:01:39.665 INFO:teuthology.orchestra.run.smithi183.stdout:Total 1.8 MB/s | 2.2 MB 00:01 2024-09-10T19:01:39.728 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction check 2024-09-10T19:01:39.743 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction check succeeded. 2024-09-10T19:01:39.743 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction test 2024-09-10T19:01:39.830 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction test succeeded. 2024-09-10T19:01:39.831 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction 2024-09-10T19:01:40.333 INFO:teuthology.orchestra.run.smithi183.stdout: Preparing : 1/1 2024-09-10T19:01:40.509 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-09-10T19:01:40.563 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-09-10T19:01:40.618 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-09-10T19:01:40.656 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-09-10T19:01:40.697 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-certifi-2023.05.07-4.el9.noarch 5/13 2024-09-10T19:01:40.806 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-09-10T19:01:40.916 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-google-auth-1:2.34.0-1.el9.noarch 7/13 2024-09-10T19:01:40.965 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-09-10T19:01:41.661 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:17.2.7-1606.g5be57 12/12 2024-09-10T19:01:41.661 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : ceph-mgr-diskprediction-local-2:17.2.7-1606.g5be57 1/12 2024-09-10T19:01:41.661 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : libgfortran-11.5.0-2.el9.x86_64 2/12 2024-09-10T19:01:41.661 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : libquadmath-11.5.0-2.el9.x86_64 3/12 2024-09-10T19:01:41.661 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 4/12 2024-09-10T19:01:41.661 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 5/12 2024-09-10T19:01:41.661 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 6/12 2024-09-10T19:01:41.661 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : openblas-0.3.26-2.el9.x86_64 7/12 2024-09-10T19:01:41.661 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : openblas-openmp-0.3.26-2.el9.x86_64 8/12 2024-09-10T19:01:41.661 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-devel-3.9.19-8.el9.x86_64 9/12 2024-09-10T19:01:41.662 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 10/12 2024-09-10T19:01:41.662 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 11/12 2024-09-10T19:01:41.669 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-09-10T19:01:41.721 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-09-10T19:01:41.755 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-09-10T19:01:41.800 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-09-10T19:01:41.825 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : ceph-mgr-rook-2:17.2.7-1606.g5be57395.el9.noarch 13/13 2024-09-10T19:01:42.087 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 12/12 2024-09-10T19:01:42.087 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-10T19:01:42.088 INFO:teuthology.orchestra.run.smithi191.stdout:Installed: 2024-09-10T19:01:42.088 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-mgr-diskprediction-local-2:17.2.7-1606.g5be57395.el9.noarch 2024-09-10T19:01:42.088 INFO:teuthology.orchestra.run.smithi191.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-09-10T19:01:42.088 INFO:teuthology.orchestra.run.smithi191.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-09-10T19:01:42.088 INFO:teuthology.orchestra.run.smithi191.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-09-10T19:01:42.088 INFO:teuthology.orchestra.run.smithi191.stdout: libgfortran-11.5.0-2.el9.x86_64 2024-09-10T19:01:42.088 INFO:teuthology.orchestra.run.smithi191.stdout: libquadmath-11.5.0-2.el9.x86_64 2024-09-10T19:01:42.088 INFO:teuthology.orchestra.run.smithi191.stdout: openblas-0.3.26-2.el9.x86_64 2024-09-10T19:01:42.088 INFO:teuthology.orchestra.run.smithi191.stdout: openblas-openmp-0.3.26-2.el9.x86_64 2024-09-10T19:01:42.088 INFO:teuthology.orchestra.run.smithi191.stdout: python3-devel-3.9.19-8.el9.x86_64 2024-09-10T19:01:42.088 INFO:teuthology.orchestra.run.smithi191.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-09-10T19:01:42.088 INFO:teuthology.orchestra.run.smithi191.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-09-10T19:01:42.089 INFO:teuthology.orchestra.run.smithi191.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-09-10T19:01:42.089 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-10T19:01:42.089 INFO:teuthology.orchestra.run.smithi191.stdout:Complete! 2024-09-10T19:01:42.437 DEBUG:teuthology.orchestra.run.smithi191:> sudo yum -y install ceph-mgr-rook 2024-09-10T19:01:42.619 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: ceph-mgr-rook-2:17.2.7-1606.g5be57395.el9.noarch 13/13 2024-09-10T19:01:42.619 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : ceph-mgr-rook-2:17.2.7-1606.g5be57395.el9.noarch 1/13 2024-09-10T19:01:42.619 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-09-10T19:01:42.619 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-09-10T19:01:42.619 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-09-10T19:01:42.619 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-09-10T19:01:42.619 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-09-10T19:01:42.620 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-09-10T19:01:42.620 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-09-10T19:01:42.620 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-certifi-2023.05.07-4.el9.noarch 9/13 2024-09-10T19:01:42.620 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-google-auth-1:2.34.0-1.el9.noarch 10/13 2024-09-10T19:01:42.620 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-09-10T19:01:42.620 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-09-10T19:01:42.966 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-09-10T19:01:42.966 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-09-10T19:01:42.966 INFO:teuthology.orchestra.run.smithi183.stdout:Installed: 2024-09-10T19:01:42.966 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-mgr-rook-2:17.2.7-1606.g5be57395.el9.noarch 2024-09-10T19:01:42.967 INFO:teuthology.orchestra.run.smithi183.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-09-10T19:01:42.967 INFO:teuthology.orchestra.run.smithi183.stdout: python3-certifi-2023.05.07-4.el9.noarch 2024-09-10T19:01:42.967 INFO:teuthology.orchestra.run.smithi183.stdout: python3-google-auth-1:2.34.0-1.el9.noarch 2024-09-10T19:01:42.967 INFO:teuthology.orchestra.run.smithi183.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-09-10T19:01:42.967 INFO:teuthology.orchestra.run.smithi183.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-09-10T19:01:42.967 INFO:teuthology.orchestra.run.smithi183.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-09-10T19:01:42.967 INFO:teuthology.orchestra.run.smithi183.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-09-10T19:01:42.967 INFO:teuthology.orchestra.run.smithi183.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-09-10T19:01:42.967 INFO:teuthology.orchestra.run.smithi183.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-09-10T19:01:42.967 INFO:teuthology.orchestra.run.smithi183.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-09-10T19:01:42.967 INFO:teuthology.orchestra.run.smithi183.stdout: python3-rsa-4.9-2.el9.noarch 2024-09-10T19:01:42.967 INFO:teuthology.orchestra.run.smithi183.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-09-10T19:01:42.967 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-09-10T19:01:42.967 INFO:teuthology.orchestra.run.smithi183.stdout:Complete! 2024-09-10T19:01:43.030 INFO:teuthology.orchestra.run.smithi191.stdout:Last metadata expiration check: 0:01:18 ago on Tue 10 Sep 2024 07:00:25 PM UTC. 2024-09-10T19:01:43.177 INFO:teuthology.orchestra.run.smithi191.stdout:Dependencies resolved. 2024-09-10T19:01:43.178 INFO:teuthology.orchestra.run.smithi191.stdout:================================================================================ 2024-09-10T19:01:43.178 INFO:teuthology.orchestra.run.smithi191.stdout: Package Arch Version Repository Size 2024-09-10T19:01:43.178 INFO:teuthology.orchestra.run.smithi191.stdout:================================================================================ 2024-09-10T19:01:43.178 INFO:teuthology.orchestra.run.smithi191.stdout:Installing: 2024-09-10T19:01:43.178 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-mgr-rook noarch 2:17.2.7-1606.g5be57395.el9 ceph-noarch 47 k 2024-09-10T19:01:43.179 INFO:teuthology.orchestra.run.smithi191.stdout:Installing dependencies: 2024-09-10T19:01:43.179 INFO:teuthology.orchestra.run.smithi191.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-09-10T19:01:43.179 INFO:teuthology.orchestra.run.smithi191.stdout: python3-certifi noarch 2023.05.07-4.el9 epel 14 k 2024-09-10T19:01:43.179 INFO:teuthology.orchestra.run.smithi191.stdout: python3-google-auth noarch 1:2.34.0-1.el9 epel 220 k 2024-09-10T19:01:43.179 INFO:teuthology.orchestra.run.smithi191.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-09-10T19:01:43.179 INFO:teuthology.orchestra.run.smithi191.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-09-10T19:01:43.179 INFO:teuthology.orchestra.run.smithi191.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-09-10T19:01:43.179 INFO:teuthology.orchestra.run.smithi191.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-09-10T19:01:43.179 INFO:teuthology.orchestra.run.smithi191.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-09-10T19:01:43.179 INFO:teuthology.orchestra.run.smithi191.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-09-10T19:01:43.179 INFO:teuthology.orchestra.run.smithi191.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-09-10T19:01:43.179 INFO:teuthology.orchestra.run.smithi191.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-09-10T19:01:43.179 INFO:teuthology.orchestra.run.smithi191.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-09-10T19:01:43.180 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-10T19:01:43.180 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction Summary 2024-09-10T19:01:43.180 INFO:teuthology.orchestra.run.smithi191.stdout:================================================================================ 2024-09-10T19:01:43.180 INFO:teuthology.orchestra.run.smithi191.stdout:Install 13 Packages 2024-09-10T19:01:43.180 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-10T19:01:43.180 INFO:teuthology.orchestra.run.smithi191.stdout:Total download size: 2.2 M 2024-09-10T19:01:43.180 INFO:teuthology.orchestra.run.smithi191.stdout:Installed size: 26 M 2024-09-10T19:01:43.180 INFO:teuthology.orchestra.run.smithi191.stdout:Downloading Packages: 2024-09-10T19:01:43.226 DEBUG:teuthology.orchestra.run.smithi183:> sudo yum -y install ceph-mgr-cephadm 2024-09-10T19:01:43.668 INFO:teuthology.orchestra.run.smithi191.stdout:(1/13): ceph-mgr-rook-17.2.7-1606.g5be57395.el9 274 kB/s | 47 kB 00:00 2024-09-10T19:01:43.785 INFO:teuthology.orchestra.run.smithi191.stdout:(2/13): python3-jsonpatch-1.21-16.el9.noarch.rp 88 kB/s | 26 kB 00:00 2024-09-10T19:01:43.790 INFO:teuthology.orchestra.run.smithi183.stdout:Last metadata expiration check: 0:01:19 ago on Tue 10 Sep 2024 07:00:24 PM UTC. 2024-09-10T19:01:43.810 INFO:teuthology.orchestra.run.smithi191.stdout:(3/13): python3-jsonpointer-2.0-4.el9.noarch.rp 61 kB/s | 19 kB 00:00 2024-09-10T19:01:43.925 INFO:teuthology.orchestra.run.smithi183.stdout:Dependencies resolved. 2024-09-10T19:01:43.925 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2024-09-10T19:01:43.925 INFO:teuthology.orchestra.run.smithi183.stdout: Package Arch Version Repository Size 2024-09-10T19:01:43.926 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2024-09-10T19:01:43.926 INFO:teuthology.orchestra.run.smithi183.stdout:Installing: 2024-09-10T19:01:43.926 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-mgr-cephadm noarch 2:17.2.7-1606.g5be57395.el9 ceph-noarch 124 k 2024-09-10T19:01:43.926 INFO:teuthology.orchestra.run.smithi183.stdout:Installing dependencies: 2024-09-10T19:01:43.926 INFO:teuthology.orchestra.run.smithi183.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-09-10T19:01:43.926 INFO:teuthology.orchestra.run.smithi183.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-09-10T19:01:43.926 INFO:teuthology.orchestra.run.smithi183.stdout: python3-jinja2 noarch 2.11.3-6.el9 appstream 248 k 2024-09-10T19:01:43.926 INFO:teuthology.orchestra.run.smithi183.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-09-10T19:01:43.926 INFO:teuthology.orchestra.run.smithi183.stdout: python3-typing-extensions noarch 4.12.2-2.el9 epel 74 k 2024-09-10T19:01:43.926 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-09-10T19:01:43.926 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction Summary 2024-09-10T19:01:43.926 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2024-09-10T19:01:43.927 INFO:teuthology.orchestra.run.smithi183.stdout:Install 6 Packages 2024-09-10T19:01:43.927 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-09-10T19:01:43.927 INFO:teuthology.orchestra.run.smithi183.stdout:Total download size: 7.0 M 2024-09-10T19:01:43.927 INFO:teuthology.orchestra.run.smithi183.stdout:Installed size: 33 M 2024-09-10T19:01:43.927 INFO:teuthology.orchestra.run.smithi183.stdout:Downloading Packages: 2024-09-10T19:01:44.252 INFO:teuthology.orchestra.run.smithi191.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 340 kB/s | 159 kB 00:00 2024-09-10T19:01:44.277 INFO:teuthology.orchestra.run.smithi191.stdout:(5/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 365 kB/s | 222 kB 00:00 2024-09-10T19:01:44.281 INFO:teuthology.orchestra.run.smithi183.stdout:(1/6): python3-jinja2-2.11.3-6.el9.noarch.rpm 1.6 MB/s | 248 kB 00:00 2024-09-10T19:01:44.302 INFO:teuthology.orchestra.run.smithi191.stdout:(6/13): python3-pyasn1-modules-0.4.8-6.el9.noar 566 kB/s | 279 kB 00:00 2024-09-10T19:01:44.331 INFO:teuthology.orchestra.run.smithi183.stdout:(2/6): ceph-mgr-cephadm-17.2.7-1606.g5be57395.e 619 kB/s | 124 kB 00:00 2024-09-10T19:01:44.344 INFO:teuthology.orchestra.run.smithi191.stdout:(7/13): python3-cachetools-4.2.4-1.el9.noarch.r 480 kB/s | 32 kB 00:00 2024-09-10T19:01:44.361 INFO:teuthology.orchestra.run.smithi191.stdout:(8/13): python3-certifi-2023.05.07-4.el9.noarch 241 kB/s | 14 kB 00:00 2024-09-10T19:01:44.378 INFO:teuthology.orchestra.run.smithi191.stdout:(9/13): python3-requests-oauthlib-1.3.0-12.el9. 426 kB/s | 54 kB 00:00 2024-09-10T19:01:44.420 INFO:teuthology.orchestra.run.smithi191.stdout:(10/13): python3-google-auth-2.34.0-1.el9.noarc 2.9 MB/s | 220 kB 00:00 2024-09-10T19:01:44.432 INFO:teuthology.orchestra.run.smithi183.stdout:(3/6): python3-babel-2.9.1-2.el9.noarch.rpm 20 MB/s | 6.0 MB 00:00 2024-09-10T19:01:44.445 INFO:teuthology.orchestra.run.smithi191.stdout:(11/13): python3-rsa-4.9-2.el9.noarch.rpm 884 kB/s | 59 kB 00:00 2024-09-10T19:01:44.457 INFO:teuthology.orchestra.run.smithi183.stdout:(4/6): python3-natsort-7.1.1-5.el9.noarch.rpm 460 kB/s | 58 kB 00:00 2024-09-10T19:01:44.470 INFO:teuthology.orchestra.run.smithi191.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 1.8 MB/s | 90 kB 00:00 2024-09-10T19:01:44.491 INFO:teuthology.orchestra.run.smithi183.stdout:(5/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 2.6 MB/s | 548 kB 00:00 2024-09-10T19:01:44.504 INFO:teuthology.orchestra.run.smithi191.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 7.2 MB/s | 1.0 MB 00:00 2024-09-10T19:01:44.507 INFO:teuthology.orchestra.run.smithi191.stdout:-------------------------------------------------------------------------------- 2024-09-10T19:01:44.507 INFO:teuthology.orchestra.run.smithi191.stdout:Total 1.7 MB/s | 2.2 MB 00:01 2024-09-10T19:01:44.524 INFO:teuthology.orchestra.run.smithi183.stdout:(6/6): python3-typing-extensions-4.12.2-2.el9.n 811 kB/s | 74 kB 00:00 2024-09-10T19:01:44.527 INFO:teuthology.orchestra.run.smithi183.stdout:-------------------------------------------------------------------------------- 2024-09-10T19:01:44.527 INFO:teuthology.orchestra.run.smithi183.stdout:Total 12 MB/s | 7.0 MB 00:00 2024-09-10T19:01:44.570 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction check 2024-09-10T19:01:44.586 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction check succeeded. 2024-09-10T19:01:44.586 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction test 2024-09-10T19:01:44.598 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction check 2024-09-10T19:01:44.609 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction check succeeded. 2024-09-10T19:01:44.609 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction test 2024-09-10T19:01:44.672 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction test succeeded. 2024-09-10T19:01:44.677 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction 2024-09-10T19:01:44.709 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction test succeeded. 2024-09-10T19:01:44.710 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction 2024-09-10T19:01:45.013 INFO:teuthology.orchestra.run.smithi183.stdout: Preparing : 1/1 2024-09-10T19:01:45.188 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-typing-extensions-4.12.2-2.el9.noarch 1/6 2024-09-10T19:01:45.208 INFO:teuthology.orchestra.run.smithi191.stdout: Preparing : 1/1 2024-09-10T19:01:45.234 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-09-10T19:01:45.387 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-09-10T19:01:45.438 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-09-10T19:01:45.494 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-09-10T19:01:45.531 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-09-10T19:01:45.573 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-certifi-2023.05.07-4.el9.noarch 5/13 2024-09-10T19:01:45.674 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-09-10T19:01:45.760 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-09-10T19:01:45.777 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-google-auth-1:2.34.0-1.el9.noarch 7/13 2024-09-10T19:01:45.825 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-09-10T19:01:45.846 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-09-10T19:01:45.909 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-jinja2-2.11.3-6.el9.noarch 5/6 2024-09-10T19:01:45.938 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : ceph-mgr-cephadm-2:17.2.7-1606.g5be57395.el9.noarch 6/6 2024-09-10T19:01:46.541 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-09-10T19:01:46.604 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-09-10T19:01:46.638 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-09-10T19:01:46.675 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-09-10T19:01:46.691 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: ceph-mgr-cephadm-2:17.2.7-1606.g5be57395.el9.noarch 6/6 2024-09-10T19:01:46.691 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : ceph-mgr-cephadm-2:17.2.7-1606.g5be57395.el9.noarch 1/6 2024-09-10T19:01:46.691 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-09-10T19:01:46.691 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-jinja2-2.11.3-6.el9.noarch 3/6 2024-09-10T19:01:46.691 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-09-10T19:01:46.691 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-09-10T19:01:46.708 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : ceph-mgr-rook-2:17.2.7-1606.g5be57395.el9.noarch 13/13 2024-09-10T19:01:46.956 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-typing-extensions-4.12.2-2.el9.noarch 6/6 2024-09-10T19:01:46.957 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-09-10T19:01:46.957 INFO:teuthology.orchestra.run.smithi183.stdout:Installed: 2024-09-10T19:01:46.957 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-mgr-cephadm-2:17.2.7-1606.g5be57395.el9.noarch 2024-09-10T19:01:46.957 INFO:teuthology.orchestra.run.smithi183.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-09-10T19:01:46.957 INFO:teuthology.orchestra.run.smithi183.stdout: python3-babel-2.9.1-2.el9.noarch 2024-09-10T19:01:46.957 INFO:teuthology.orchestra.run.smithi183.stdout: python3-jinja2-2.11.3-6.el9.noarch 2024-09-10T19:01:46.957 INFO:teuthology.orchestra.run.smithi183.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-09-10T19:01:46.957 INFO:teuthology.orchestra.run.smithi183.stdout: python3-typing-extensions-4.12.2-2.el9.noarch 2024-09-10T19:01:46.957 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-09-10T19:01:46.957 INFO:teuthology.orchestra.run.smithi183.stdout:Complete! 2024-09-10T19:01:47.188 DEBUG:teuthology.orchestra.run.smithi183:> sudo yum -y install ceph-fuse 2024-09-10T19:01:47.478 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: ceph-mgr-rook-2:17.2.7-1606.g5be57395.el9.noarch 13/13 2024-09-10T19:01:47.478 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : ceph-mgr-rook-2:17.2.7-1606.g5be57395.el9.noarch 1/13 2024-09-10T19:01:47.478 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-09-10T19:01:47.478 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-09-10T19:01:47.478 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-09-10T19:01:47.478 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-09-10T19:01:47.478 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-09-10T19:01:47.478 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-09-10T19:01:47.479 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-09-10T19:01:47.479 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-certifi-2023.05.07-4.el9.noarch 9/13 2024-09-10T19:01:47.479 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-google-auth-1:2.34.0-1.el9.noarch 10/13 2024-09-10T19:01:47.479 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-09-10T19:01:47.479 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-09-10T19:01:47.753 INFO:teuthology.orchestra.run.smithi183.stdout:Last metadata expiration check: 0:01:23 ago on Tue 10 Sep 2024 07:00:24 PM UTC. 2024-09-10T19:01:47.810 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-09-10T19:01:47.810 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-10T19:01:47.810 INFO:teuthology.orchestra.run.smithi191.stdout:Installed: 2024-09-10T19:01:47.810 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-mgr-rook-2:17.2.7-1606.g5be57395.el9.noarch 2024-09-10T19:01:47.810 INFO:teuthology.orchestra.run.smithi191.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-09-10T19:01:47.810 INFO:teuthology.orchestra.run.smithi191.stdout: python3-certifi-2023.05.07-4.el9.noarch 2024-09-10T19:01:47.810 INFO:teuthology.orchestra.run.smithi191.stdout: python3-google-auth-1:2.34.0-1.el9.noarch 2024-09-10T19:01:47.810 INFO:teuthology.orchestra.run.smithi191.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-09-10T19:01:47.811 INFO:teuthology.orchestra.run.smithi191.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-09-10T19:01:47.811 INFO:teuthology.orchestra.run.smithi191.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-09-10T19:01:47.811 INFO:teuthology.orchestra.run.smithi191.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-09-10T19:01:47.811 INFO:teuthology.orchestra.run.smithi191.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-09-10T19:01:47.811 INFO:teuthology.orchestra.run.smithi191.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-09-10T19:01:47.811 INFO:teuthology.orchestra.run.smithi191.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-09-10T19:01:47.811 INFO:teuthology.orchestra.run.smithi191.stdout: python3-rsa-4.9-2.el9.noarch 2024-09-10T19:01:47.811 INFO:teuthology.orchestra.run.smithi191.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-09-10T19:01:47.811 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-10T19:01:47.811 INFO:teuthology.orchestra.run.smithi191.stdout:Complete! 2024-09-10T19:01:47.884 INFO:teuthology.orchestra.run.smithi183.stdout:Dependencies resolved. 2024-09-10T19:01:47.885 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2024-09-10T19:01:47.885 INFO:teuthology.orchestra.run.smithi183.stdout: Package Arch Version Repository Size 2024-09-10T19:01:47.885 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2024-09-10T19:01:47.885 INFO:teuthology.orchestra.run.smithi183.stdout:Installing: 2024-09-10T19:01:47.885 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-fuse x86_64 2:17.2.7-1606.g5be57395.el9 ceph 813 k 2024-09-10T19:01:47.885 INFO:teuthology.orchestra.run.smithi183.stdout:Installing dependencies: 2024-09-10T19:01:47.885 INFO:teuthology.orchestra.run.smithi183.stdout: fuse x86_64 2.9.9-16.el9 baseos 80 k 2024-09-10T19:01:47.885 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-09-10T19:01:47.885 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction Summary 2024-09-10T19:01:47.885 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2024-09-10T19:01:47.885 INFO:teuthology.orchestra.run.smithi183.stdout:Install 2 Packages 2024-09-10T19:01:47.886 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-09-10T19:01:47.886 INFO:teuthology.orchestra.run.smithi183.stdout:Total download size: 893 k 2024-09-10T19:01:47.886 INFO:teuthology.orchestra.run.smithi183.stdout:Installed size: 2.6 M 2024-09-10T19:01:47.886 INFO:teuthology.orchestra.run.smithi183.stdout:Downloading Packages: 2024-09-10T19:01:48.060 INFO:teuthology.orchestra.run.smithi183.stdout:(1/2): fuse-2.9.9-16.el9.x86_64.rpm 524 kB/s | 80 kB 00:00 2024-09-10T19:01:48.085 DEBUG:teuthology.orchestra.run.smithi191:> sudo yum -y install ceph-mgr-cephadm 2024-09-10T19:01:48.193 INFO:teuthology.orchestra.run.smithi183.stdout:(2/2): ceph-fuse-17.2.7-1606.g5be57395.el9.x86_ 2.8 MB/s | 813 kB 00:00 2024-09-10T19:01:48.193 INFO:teuthology.orchestra.run.smithi183.stdout:-------------------------------------------------------------------------------- 2024-09-10T19:01:48.194 INFO:teuthology.orchestra.run.smithi183.stdout:Total 2.8 MB/s | 893 kB 00:00 2024-09-10T19:01:48.206 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction check 2024-09-10T19:01:48.227 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction check succeeded. 2024-09-10T19:01:48.227 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction test 2024-09-10T19:01:48.307 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction test succeeded. 2024-09-10T19:01:48.307 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction 2024-09-10T19:01:48.498 INFO:teuthology.orchestra.run.smithi183.stdout: Preparing : 1/1 2024-09-10T19:01:48.638 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : fuse-2.9.9-16.el9.x86_64 1/2 2024-09-10T19:01:48.645 INFO:teuthology.orchestra.run.smithi191.stdout:Last metadata expiration check: 0:01:23 ago on Tue 10 Sep 2024 07:00:25 PM UTC. 2024-09-10T19:01:48.702 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : ceph-fuse-2:17.2.7-1606.g5be57395.el9.x86_64 2/2 2024-09-10T19:01:48.782 INFO:teuthology.orchestra.run.smithi191.stdout:Dependencies resolved. 2024-09-10T19:01:48.782 INFO:teuthology.orchestra.run.smithi191.stdout:================================================================================ 2024-09-10T19:01:48.783 INFO:teuthology.orchestra.run.smithi191.stdout: Package Arch Version Repository Size 2024-09-10T19:01:48.783 INFO:teuthology.orchestra.run.smithi191.stdout:================================================================================ 2024-09-10T19:01:48.783 INFO:teuthology.orchestra.run.smithi191.stdout:Installing: 2024-09-10T19:01:48.783 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-mgr-cephadm noarch 2:17.2.7-1606.g5be57395.el9 ceph-noarch 124 k 2024-09-10T19:01:48.783 INFO:teuthology.orchestra.run.smithi191.stdout:Installing dependencies: 2024-09-10T19:01:48.784 INFO:teuthology.orchestra.run.smithi191.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-09-10T19:01:48.784 INFO:teuthology.orchestra.run.smithi191.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-09-10T19:01:48.784 INFO:teuthology.orchestra.run.smithi191.stdout: python3-jinja2 noarch 2.11.3-6.el9 appstream 248 k 2024-09-10T19:01:48.784 INFO:teuthology.orchestra.run.smithi191.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-09-10T19:01:48.785 INFO:teuthology.orchestra.run.smithi191.stdout: python3-typing-extensions noarch 4.12.2-2.el9 epel 74 k 2024-09-10T19:01:48.785 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-10T19:01:48.785 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction Summary 2024-09-10T19:01:48.786 INFO:teuthology.orchestra.run.smithi191.stdout:================================================================================ 2024-09-10T19:01:48.786 INFO:teuthology.orchestra.run.smithi191.stdout:Install 6 Packages 2024-09-10T19:01:48.786 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-10T19:01:48.787 INFO:teuthology.orchestra.run.smithi191.stdout:Total download size: 7.0 M 2024-09-10T19:01:48.787 INFO:teuthology.orchestra.run.smithi191.stdout:Installed size: 33 M 2024-09-10T19:01:48.787 INFO:teuthology.orchestra.run.smithi191.stdout:Downloading Packages: 2024-09-10T19:01:49.297 INFO:teuthology.orchestra.run.smithi191.stdout:(1/6): ceph-mgr-cephadm-17.2.7-1606.g5be57395.e 584 kB/s | 124 kB 00:00 2024-09-10T19:01:49.348 INFO:teuthology.orchestra.run.smithi191.stdout:(2/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 11 MB/s | 548 kB 00:00 2024-09-10T19:01:49.372 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: ceph-fuse-2:17.2.7-1606.g5be57395.el9.x86_64 2/2 2024-09-10T19:01:49.372 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : ceph-fuse-2:17.2.7-1606.g5be57395.el9.x86_64 1/2 2024-09-10T19:01:49.373 INFO:teuthology.orchestra.run.smithi191.stdout:(3/6): python3-natsort-7.1.1-5.el9.noarch.rpm 2.3 MB/s | 58 kB 00:00 2024-09-10T19:01:49.398 INFO:teuthology.orchestra.run.smithi191.stdout:(4/6): python3-typing-extensions-4.12.2-2.el9.n 2.9 MB/s | 74 kB 00:00 2024-09-10T19:01:49.598 INFO:teuthology.orchestra.run.smithi191.stdout:(5/6): python3-jinja2-2.11.3-6.el9.noarch.rpm 484 kB/s | 248 kB 00:00 2024-09-10T19:01:49.647 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : fuse-2.9.9-16.el9.x86_64 2/2 2024-09-10T19:01:49.647 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-09-10T19:01:49.647 INFO:teuthology.orchestra.run.smithi183.stdout:Installed: 2024-09-10T19:01:49.647 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-fuse-2:17.2.7-1606.g5be57395.el9.x86_64 fuse-2.9.9-16.el9.x86_64 2024-09-10T19:01:49.647 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-09-10T19:01:49.647 INFO:teuthology.orchestra.run.smithi183.stdout:Complete! 2024-09-10T19:01:49.940 DEBUG:teuthology.orchestra.run.smithi183:> sudo yum -y install ceph-volume 2024-09-10T19:01:50.473 INFO:teuthology.orchestra.run.smithi191.stdout:(6/6): python3-babel-2.9.1-2.el9.noarch.rpm 4.3 MB/s | 6.0 MB 00:01 2024-09-10T19:01:50.475 INFO:teuthology.orchestra.run.smithi191.stdout:-------------------------------------------------------------------------------- 2024-09-10T19:01:50.476 INFO:teuthology.orchestra.run.smithi191.stdout:Total 4.1 MB/s | 7.0 MB 00:01 2024-09-10T19:01:50.520 INFO:teuthology.orchestra.run.smithi183.stdout:Last metadata expiration check: 0:01:26 ago on Tue 10 Sep 2024 07:00:24 PM UTC. 2024-09-10T19:01:50.549 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction check 2024-09-10T19:01:50.561 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction check succeeded. 2024-09-10T19:01:50.561 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction test 2024-09-10T19:01:50.660 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction test succeeded. 2024-09-10T19:01:50.661 INFO:teuthology.orchestra.run.smithi183.stdout:Dependencies resolved. 2024-09-10T19:01:50.661 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction 2024-09-10T19:01:50.662 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2024-09-10T19:01:50.662 INFO:teuthology.orchestra.run.smithi183.stdout: Package Arch Version Repository Size 2024-09-10T19:01:50.662 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2024-09-10T19:01:50.662 INFO:teuthology.orchestra.run.smithi183.stdout:Installing: 2024-09-10T19:01:50.662 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-volume noarch 2:17.2.7-1606.g5be57395.el9 ceph-noarch 270 k 2024-09-10T19:01:50.662 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-09-10T19:01:50.662 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction Summary 2024-09-10T19:01:50.662 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2024-09-10T19:01:50.663 INFO:teuthology.orchestra.run.smithi183.stdout:Install 1 Package 2024-09-10T19:01:50.663 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-09-10T19:01:50.663 INFO:teuthology.orchestra.run.smithi183.stdout:Total download size: 270 k 2024-09-10T19:01:50.663 INFO:teuthology.orchestra.run.smithi183.stdout:Installed size: 1.3 M 2024-09-10T19:01:50.663 INFO:teuthology.orchestra.run.smithi183.stdout:Downloading Packages: 2024-09-10T19:01:50.889 INFO:teuthology.orchestra.run.smithi183.stdout:ceph-volume-17.2.7-1606.g5be57395.el9.noarch.rp 1.2 MB/s | 270 kB 00:00 2024-09-10T19:01:50.890 INFO:teuthology.orchestra.run.smithi183.stdout:-------------------------------------------------------------------------------- 2024-09-10T19:01:50.890 INFO:teuthology.orchestra.run.smithi183.stdout:Total 1.2 MB/s | 270 kB 00:00 2024-09-10T19:01:50.890 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction check 2024-09-10T19:01:50.898 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction check succeeded. 2024-09-10T19:01:50.898 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction test 2024-09-10T19:01:50.929 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction test succeeded. 2024-09-10T19:01:50.930 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction 2024-09-10T19:01:50.961 INFO:teuthology.orchestra.run.smithi191.stdout: Preparing : 1/1 2024-09-10T19:01:51.113 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-typing-extensions-4.12.2-2.el9.noarch 1/6 2024-09-10T19:01:51.141 INFO:teuthology.orchestra.run.smithi183.stdout: Preparing : 1/1 2024-09-10T19:01:51.158 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-09-10T19:01:51.208 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : ceph-volume-2:17.2.7-1606.g5be57395.el9.noarch 1/1 2024-09-10T19:01:51.224 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: ceph-volume-2:17.2.7-1606.g5be57395.el9.noarch 1/1 2024-09-10T19:01:51.224 INFO:teuthology.orchestra.run.smithi183.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-10T19:01:51.224 INFO:teuthology.orchestra.run.smithi183.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-09-10T19:01:51.225 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-09-10T19:01:51.677 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-09-10T19:01:51.760 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-09-10T19:01:51.816 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-jinja2-2.11.3-6.el9.noarch 5/6 2024-09-10T19:01:51.845 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : ceph-mgr-cephadm-2:17.2.7-1606.g5be57395.el9.noarch 6/6 2024-09-10T19:01:52.198 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : ceph-volume-2:17.2.7-1606.g5be57395.el9.noarch 1/1 2024-09-10T19:01:52.199 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-09-10T19:01:52.199 INFO:teuthology.orchestra.run.smithi183.stdout:Installed: 2024-09-10T19:01:52.199 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-volume-2:17.2.7-1606.g5be57395.el9.noarch 2024-09-10T19:01:52.199 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-09-10T19:01:52.199 INFO:teuthology.orchestra.run.smithi183.stdout:Complete! 2024-09-10T19:01:52.479 DEBUG:teuthology.orchestra.run.smithi183:> sudo yum -y install librados-devel 2024-09-10T19:01:52.640 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: ceph-mgr-cephadm-2:17.2.7-1606.g5be57395.el9.noarch 6/6 2024-09-10T19:01:52.640 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : ceph-mgr-cephadm-2:17.2.7-1606.g5be57395.el9.noarch 1/6 2024-09-10T19:01:52.640 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-09-10T19:01:52.640 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-jinja2-2.11.3-6.el9.noarch 3/6 2024-09-10T19:01:52.640 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-09-10T19:01:52.640 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-09-10T19:01:52.909 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-typing-extensions-4.12.2-2.el9.noarch 6/6 2024-09-10T19:01:52.909 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-10T19:01:52.909 INFO:teuthology.orchestra.run.smithi191.stdout:Installed: 2024-09-10T19:01:52.909 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-mgr-cephadm-2:17.2.7-1606.g5be57395.el9.noarch 2024-09-10T19:01:52.909 INFO:teuthology.orchestra.run.smithi191.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-09-10T19:01:52.909 INFO:teuthology.orchestra.run.smithi191.stdout: python3-babel-2.9.1-2.el9.noarch 2024-09-10T19:01:52.909 INFO:teuthology.orchestra.run.smithi191.stdout: python3-jinja2-2.11.3-6.el9.noarch 2024-09-10T19:01:52.909 INFO:teuthology.orchestra.run.smithi191.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-09-10T19:01:52.909 INFO:teuthology.orchestra.run.smithi191.stdout: python3-typing-extensions-4.12.2-2.el9.noarch 2024-09-10T19:01:52.910 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-10T19:01:52.910 INFO:teuthology.orchestra.run.smithi191.stdout:Complete! 2024-09-10T19:01:53.063 INFO:teuthology.orchestra.run.smithi183.stdout:Last metadata expiration check: 0:01:29 ago on Tue 10 Sep 2024 07:00:24 PM UTC. 2024-09-10T19:01:53.165 DEBUG:teuthology.orchestra.run.smithi191:> sudo yum -y install ceph-fuse 2024-09-10T19:01:53.201 INFO:teuthology.orchestra.run.smithi183.stdout:Dependencies resolved. 2024-09-10T19:01:53.202 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2024-09-10T19:01:53.202 INFO:teuthology.orchestra.run.smithi183.stdout: Package Arch Version Repo Size 2024-09-10T19:01:53.202 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2024-09-10T19:01:53.202 INFO:teuthology.orchestra.run.smithi183.stdout:Installing: 2024-09-10T19:01:53.202 INFO:teuthology.orchestra.run.smithi183.stdout: librados-devel x86_64 2:17.2.7-1606.g5be57395.el9 ceph 126 k 2024-09-10T19:01:53.202 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-09-10T19:01:53.202 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction Summary 2024-09-10T19:01:53.202 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2024-09-10T19:01:53.202 INFO:teuthology.orchestra.run.smithi183.stdout:Install 1 Package 2024-09-10T19:01:53.203 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-09-10T19:01:53.203 INFO:teuthology.orchestra.run.smithi183.stdout:Total download size: 126 k 2024-09-10T19:01:53.203 INFO:teuthology.orchestra.run.smithi183.stdout:Installed size: 448 k 2024-09-10T19:01:53.203 INFO:teuthology.orchestra.run.smithi183.stdout:Downloading Packages: 2024-09-10T19:01:53.395 INFO:teuthology.orchestra.run.smithi183.stdout:librados-devel-17.2.7-1606.g5be57395.el9.x86_64 658 kB/s | 126 kB 00:00 2024-09-10T19:01:53.396 INFO:teuthology.orchestra.run.smithi183.stdout:-------------------------------------------------------------------------------- 2024-09-10T19:01:53.396 INFO:teuthology.orchestra.run.smithi183.stdout:Total 654 kB/s | 126 kB 00:00 2024-09-10T19:01:53.396 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction check 2024-09-10T19:01:53.401 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction check succeeded. 2024-09-10T19:01:53.401 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction test 2024-09-10T19:01:53.465 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction test succeeded. 2024-09-10T19:01:53.465 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction 2024-09-10T19:01:53.628 INFO:teuthology.orchestra.run.smithi183.stdout: Preparing : 1/1 2024-09-10T19:01:53.718 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : librados-devel-2:17.2.7-1606.g5be57395.el9.x86_64 1/1 2024-09-10T19:01:53.741 INFO:teuthology.orchestra.run.smithi191.stdout:Last metadata expiration check: 0:01:28 ago on Tue 10 Sep 2024 07:00:25 PM UTC. 2024-09-10T19:01:53.874 INFO:teuthology.orchestra.run.smithi191.stdout:Dependencies resolved. 2024-09-10T19:01:53.875 INFO:teuthology.orchestra.run.smithi191.stdout:================================================================================ 2024-09-10T19:01:53.875 INFO:teuthology.orchestra.run.smithi191.stdout: Package Arch Version Repository Size 2024-09-10T19:01:53.875 INFO:teuthology.orchestra.run.smithi191.stdout:================================================================================ 2024-09-10T19:01:53.875 INFO:teuthology.orchestra.run.smithi191.stdout:Installing: 2024-09-10T19:01:53.875 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-fuse x86_64 2:17.2.7-1606.g5be57395.el9 ceph 813 k 2024-09-10T19:01:53.875 INFO:teuthology.orchestra.run.smithi191.stdout:Installing dependencies: 2024-09-10T19:01:53.875 INFO:teuthology.orchestra.run.smithi191.stdout: fuse x86_64 2.9.9-16.el9 baseos 80 k 2024-09-10T19:01:53.875 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-10T19:01:53.875 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction Summary 2024-09-10T19:01:53.876 INFO:teuthology.orchestra.run.smithi191.stdout:================================================================================ 2024-09-10T19:01:53.876 INFO:teuthology.orchestra.run.smithi191.stdout:Install 2 Packages 2024-09-10T19:01:53.876 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-10T19:01:53.876 INFO:teuthology.orchestra.run.smithi191.stdout:Total download size: 893 k 2024-09-10T19:01:53.876 INFO:teuthology.orchestra.run.smithi191.stdout:Installed size: 2.6 M 2024-09-10T19:01:53.876 INFO:teuthology.orchestra.run.smithi191.stdout:Downloading Packages: 2024-09-10T19:01:54.041 INFO:teuthology.orchestra.run.smithi191.stdout:(1/2): fuse-2.9.9-16.el9.x86_64.rpm 554 kB/s | 80 kB 00:00 2024-09-10T19:01:54.166 INFO:teuthology.orchestra.run.smithi191.stdout:(2/2): ceph-fuse-17.2.7-1606.g5be57395.el9.x86_ 3.0 MB/s | 813 kB 00:00 2024-09-10T19:01:54.167 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: librados-devel-2:17.2.7-1606.g5be57395.el9.x86_64 1/1 2024-09-10T19:01:54.167 INFO:teuthology.orchestra.run.smithi191.stdout:-------------------------------------------------------------------------------- 2024-09-10T19:01:54.167 INFO:teuthology.orchestra.run.smithi191.stdout:Total 3.0 MB/s | 893 kB 00:00 2024-09-10T19:01:54.178 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction check 2024-09-10T19:01:54.200 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction check succeeded. 2024-09-10T19:01:54.200 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction test 2024-09-10T19:01:54.280 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction test succeeded. 2024-09-10T19:01:54.280 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction 2024-09-10T19:01:54.422 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : librados-devel-2:17.2.7-1606.g5be57395.el9.x86_64 1/1 2024-09-10T19:01:54.422 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-09-10T19:01:54.422 INFO:teuthology.orchestra.run.smithi183.stdout:Installed: 2024-09-10T19:01:54.422 INFO:teuthology.orchestra.run.smithi183.stdout: librados-devel-2:17.2.7-1606.g5be57395.el9.x86_64 2024-09-10T19:01:54.422 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-09-10T19:01:54.422 INFO:teuthology.orchestra.run.smithi183.stdout:Complete! 2024-09-10T19:01:54.464 INFO:teuthology.orchestra.run.smithi191.stdout: Preparing : 1/1 2024-09-10T19:01:54.596 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : fuse-2.9.9-16.el9.x86_64 1/2 2024-09-10T19:01:54.665 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : ceph-fuse-2:17.2.7-1606.g5be57395.el9.x86_64 2/2 2024-09-10T19:01:54.681 DEBUG:teuthology.orchestra.run.smithi183:> sudo yum -y install libcephfs2 2024-09-10T19:01:55.257 INFO:teuthology.orchestra.run.smithi183.stdout:Last metadata expiration check: 0:01:31 ago on Tue 10 Sep 2024 07:00:24 PM UTC. 2024-09-10T19:01:55.349 INFO:teuthology.orchestra.run.smithi183.stdout:Package libcephfs2-2:17.2.7-1606.g5be57395.el9.x86_64 is already installed. 2024-09-10T19:01:55.358 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: ceph-fuse-2:17.2.7-1606.g5be57395.el9.x86_64 2/2 2024-09-10T19:01:55.358 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : ceph-fuse-2:17.2.7-1606.g5be57395.el9.x86_64 1/2 2024-09-10T19:01:55.394 INFO:teuthology.orchestra.run.smithi183.stdout:Dependencies resolved. 2024-09-10T19:01:55.396 INFO:teuthology.orchestra.run.smithi183.stdout:Nothing to do. 2024-09-10T19:01:55.396 INFO:teuthology.orchestra.run.smithi183.stdout:Complete! 2024-09-10T19:01:55.469 DEBUG:teuthology.orchestra.run.smithi183:> sudo yum -y install libcephfs-devel 2024-09-10T19:01:55.646 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : fuse-2.9.9-16.el9.x86_64 2/2 2024-09-10T19:01:55.646 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-10T19:01:55.646 INFO:teuthology.orchestra.run.smithi191.stdout:Installed: 2024-09-10T19:01:55.646 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-fuse-2:17.2.7-1606.g5be57395.el9.x86_64 fuse-2.9.9-16.el9.x86_64 2024-09-10T19:01:55.646 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-10T19:01:55.646 INFO:teuthology.orchestra.run.smithi191.stdout:Complete! 2024-09-10T19:01:55.915 DEBUG:teuthology.orchestra.run.smithi191:> sudo yum -y install ceph-volume 2024-09-10T19:01:56.035 INFO:teuthology.orchestra.run.smithi183.stdout:Last metadata expiration check: 0:01:32 ago on Tue 10 Sep 2024 07:00:24 PM UTC. 2024-09-10T19:01:56.169 INFO:teuthology.orchestra.run.smithi183.stdout:Dependencies resolved. 2024-09-10T19:01:56.169 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2024-09-10T19:01:56.169 INFO:teuthology.orchestra.run.smithi183.stdout: Package Arch Version Repo Size 2024-09-10T19:01:56.169 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2024-09-10T19:01:56.170 INFO:teuthology.orchestra.run.smithi183.stdout:Installing: 2024-09-10T19:01:56.170 INFO:teuthology.orchestra.run.smithi183.stdout: libcephfs-devel x86_64 2:17.2.7-1606.g5be57395.el9 ceph 25 k 2024-09-10T19:01:56.170 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-09-10T19:01:56.170 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction Summary 2024-09-10T19:01:56.170 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2024-09-10T19:01:56.170 INFO:teuthology.orchestra.run.smithi183.stdout:Install 1 Package 2024-09-10T19:01:56.170 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-09-10T19:01:56.170 INFO:teuthology.orchestra.run.smithi183.stdout:Total download size: 25 k 2024-09-10T19:01:56.170 INFO:teuthology.orchestra.run.smithi183.stdout:Installed size: 110 k 2024-09-10T19:01:56.170 INFO:teuthology.orchestra.run.smithi183.stdout:Downloading Packages: 2024-09-10T19:01:56.315 INFO:teuthology.orchestra.run.smithi183.stdout:libcephfs-devel-17.2.7-1606.g5be57395.el9.x86_6 173 kB/s | 25 kB 00:00 2024-09-10T19:01:56.315 INFO:teuthology.orchestra.run.smithi183.stdout:-------------------------------------------------------------------------------- 2024-09-10T19:01:56.315 INFO:teuthology.orchestra.run.smithi183.stdout:Total 171 kB/s | 25 kB 00:00 2024-09-10T19:01:56.316 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction check 2024-09-10T19:01:56.320 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction check succeeded. 2024-09-10T19:01:56.321 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction test 2024-09-10T19:01:56.321 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction test succeeded. 2024-09-10T19:01:56.321 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction 2024-09-10T19:01:56.400 INFO:teuthology.orchestra.run.smithi183.stdout: Preparing : 1/1 2024-09-10T19:01:56.495 INFO:teuthology.orchestra.run.smithi191.stdout:Last metadata expiration check: 0:01:31 ago on Tue 10 Sep 2024 07:00:25 PM UTC. 2024-09-10T19:01:56.495 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : libcephfs-devel-2:17.2.7-1606.g5be57395.el9.x86_64 1/1 2024-09-10T19:01:56.629 INFO:teuthology.orchestra.run.smithi191.stdout:Dependencies resolved. 2024-09-10T19:01:56.630 INFO:teuthology.orchestra.run.smithi191.stdout:================================================================================ 2024-09-10T19:01:56.630 INFO:teuthology.orchestra.run.smithi191.stdout: Package Arch Version Repository Size 2024-09-10T19:01:56.630 INFO:teuthology.orchestra.run.smithi191.stdout:================================================================================ 2024-09-10T19:01:56.630 INFO:teuthology.orchestra.run.smithi191.stdout:Installing: 2024-09-10T19:01:56.630 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-volume noarch 2:17.2.7-1606.g5be57395.el9 ceph-noarch 270 k 2024-09-10T19:01:56.630 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-10T19:01:56.631 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction Summary 2024-09-10T19:01:56.631 INFO:teuthology.orchestra.run.smithi191.stdout:================================================================================ 2024-09-10T19:01:56.631 INFO:teuthology.orchestra.run.smithi191.stdout:Install 1 Package 2024-09-10T19:01:56.631 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-10T19:01:56.631 INFO:teuthology.orchestra.run.smithi191.stdout:Total download size: 270 k 2024-09-10T19:01:56.631 INFO:teuthology.orchestra.run.smithi191.stdout:Installed size: 1.3 M 2024-09-10T19:01:56.631 INFO:teuthology.orchestra.run.smithi191.stdout:Downloading Packages: 2024-09-10T19:01:56.838 INFO:teuthology.orchestra.run.smithi191.stdout:ceph-volume-17.2.7-1606.g5be57395.el9.noarch.rp 1.3 MB/s | 270 kB 00:00 2024-09-10T19:01:56.838 INFO:teuthology.orchestra.run.smithi191.stdout:-------------------------------------------------------------------------------- 2024-09-10T19:01:56.838 INFO:teuthology.orchestra.run.smithi191.stdout:Total 1.3 MB/s | 270 kB 00:00 2024-09-10T19:01:56.838 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction check 2024-09-10T19:01:56.847 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction check succeeded. 2024-09-10T19:01:56.847 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction test 2024-09-10T19:01:56.865 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: libcephfs-devel-2:17.2.7-1606.g5be57395.el9.x86_64 1/1 2024-09-10T19:01:56.873 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction test succeeded. 2024-09-10T19:01:56.874 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction 2024-09-10T19:01:57.092 INFO:teuthology.orchestra.run.smithi191.stdout: Preparing : 1/1 2024-09-10T19:01:57.094 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : libcephfs-devel-2:17.2.7-1606.g5be57395.el9.x86_64 1/1 2024-09-10T19:01:57.094 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-09-10T19:01:57.095 INFO:teuthology.orchestra.run.smithi183.stdout:Installed: 2024-09-10T19:01:57.095 INFO:teuthology.orchestra.run.smithi183.stdout: libcephfs-devel-2:17.2.7-1606.g5be57395.el9.x86_64 2024-09-10T19:01:57.095 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-09-10T19:01:57.095 INFO:teuthology.orchestra.run.smithi183.stdout:Complete! 2024-09-10T19:01:57.156 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : ceph-volume-2:17.2.7-1606.g5be57395.el9.noarch 1/1 2024-09-10T19:01:57.173 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: ceph-volume-2:17.2.7-1606.g5be57395.el9.noarch 1/1 2024-09-10T19:01:57.173 INFO:teuthology.orchestra.run.smithi191.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-10T19:01:57.173 INFO:teuthology.orchestra.run.smithi191.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-09-10T19:01:57.173 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-10T19:01:57.278 DEBUG:teuthology.orchestra.run.smithi183:> sudo yum -y install librados2 2024-09-10T19:01:57.840 INFO:teuthology.orchestra.run.smithi183.stdout:Last metadata expiration check: 0:01:33 ago on Tue 10 Sep 2024 07:00:24 PM UTC. 2024-09-10T19:01:57.930 INFO:teuthology.orchestra.run.smithi183.stdout:Package librados2-2:17.2.7-1606.g5be57395.el9.x86_64 is already installed. 2024-09-10T19:01:57.973 INFO:teuthology.orchestra.run.smithi183.stdout:Dependencies resolved. 2024-09-10T19:01:57.974 INFO:teuthology.orchestra.run.smithi183.stdout:Nothing to do. 2024-09-10T19:01:57.974 INFO:teuthology.orchestra.run.smithi183.stdout:Complete! 2024-09-10T19:01:58.039 DEBUG:teuthology.orchestra.run.smithi183:> sudo yum -y install librbd1 2024-09-10T19:01:58.216 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : ceph-volume-2:17.2.7-1606.g5be57395.el9.noarch 1/1 2024-09-10T19:01:58.216 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-10T19:01:58.217 INFO:teuthology.orchestra.run.smithi191.stdout:Installed: 2024-09-10T19:01:58.217 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-volume-2:17.2.7-1606.g5be57395.el9.noarch 2024-09-10T19:01:58.217 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-10T19:01:58.217 INFO:teuthology.orchestra.run.smithi191.stdout:Complete! 2024-09-10T19:01:58.492 DEBUG:teuthology.orchestra.run.smithi191:> sudo yum -y install librados-devel 2024-09-10T19:01:58.604 INFO:teuthology.orchestra.run.smithi183.stdout:Last metadata expiration check: 0:01:34 ago on Tue 10 Sep 2024 07:00:24 PM UTC. 2024-09-10T19:01:58.693 INFO:teuthology.orchestra.run.smithi183.stdout:Package librbd1-2:17.2.7-1606.g5be57395.el9.x86_64 is already installed. 2024-09-10T19:01:58.737 INFO:teuthology.orchestra.run.smithi183.stdout:Dependencies resolved. 2024-09-10T19:01:58.738 INFO:teuthology.orchestra.run.smithi183.stdout:Nothing to do. 2024-09-10T19:01:58.738 INFO:teuthology.orchestra.run.smithi183.stdout:Complete! 2024-09-10T19:01:58.809 DEBUG:teuthology.orchestra.run.smithi183:> sudo yum -y install python3-rados 2024-09-10T19:01:59.081 INFO:teuthology.orchestra.run.smithi191.stdout:Last metadata expiration check: 0:01:34 ago on Tue 10 Sep 2024 07:00:25 PM UTC. 2024-09-10T19:01:59.216 INFO:teuthology.orchestra.run.smithi191.stdout:Dependencies resolved. 2024-09-10T19:01:59.217 INFO:teuthology.orchestra.run.smithi191.stdout:================================================================================ 2024-09-10T19:01:59.217 INFO:teuthology.orchestra.run.smithi191.stdout: Package Arch Version Repo Size 2024-09-10T19:01:59.217 INFO:teuthology.orchestra.run.smithi191.stdout:================================================================================ 2024-09-10T19:01:59.217 INFO:teuthology.orchestra.run.smithi191.stdout:Installing: 2024-09-10T19:01:59.217 INFO:teuthology.orchestra.run.smithi191.stdout: librados-devel x86_64 2:17.2.7-1606.g5be57395.el9 ceph 126 k 2024-09-10T19:01:59.217 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-10T19:01:59.217 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction Summary 2024-09-10T19:01:59.217 INFO:teuthology.orchestra.run.smithi191.stdout:================================================================================ 2024-09-10T19:01:59.217 INFO:teuthology.orchestra.run.smithi191.stdout:Install 1 Package 2024-09-10T19:01:59.218 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-10T19:01:59.218 INFO:teuthology.orchestra.run.smithi191.stdout:Total download size: 126 k 2024-09-10T19:01:59.218 INFO:teuthology.orchestra.run.smithi191.stdout:Installed size: 448 k 2024-09-10T19:01:59.218 INFO:teuthology.orchestra.run.smithi191.stdout:Downloading Packages: 2024-09-10T19:01:59.372 INFO:teuthology.orchestra.run.smithi183.stdout:Last metadata expiration check: 0:01:35 ago on Tue 10 Sep 2024 07:00:24 PM UTC. 2024-09-10T19:01:59.418 INFO:teuthology.orchestra.run.smithi191.stdout:librados-devel-17.2.7-1606.g5be57395.el9.x86_64 633 kB/s | 126 kB 00:00 2024-09-10T19:01:59.418 INFO:teuthology.orchestra.run.smithi191.stdout:-------------------------------------------------------------------------------- 2024-09-10T19:01:59.418 INFO:teuthology.orchestra.run.smithi191.stdout:Total 627 kB/s | 126 kB 00:00 2024-09-10T19:01:59.419 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction check 2024-09-10T19:01:59.424 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction check succeeded. 2024-09-10T19:01:59.424 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction test 2024-09-10T19:01:59.462 INFO:teuthology.orchestra.run.smithi183.stdout:Package python3-rados-2:17.2.7-1606.g5be57395.el9.x86_64 is already installed. 2024-09-10T19:01:59.486 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction test succeeded. 2024-09-10T19:01:59.487 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction 2024-09-10T19:01:59.504 INFO:teuthology.orchestra.run.smithi183.stdout:Dependencies resolved. 2024-09-10T19:01:59.505 INFO:teuthology.orchestra.run.smithi183.stdout:Nothing to do. 2024-09-10T19:01:59.506 INFO:teuthology.orchestra.run.smithi183.stdout:Complete! 2024-09-10T19:01:59.618 DEBUG:teuthology.orchestra.run.smithi183:> sudo yum -y install python3-rgw 2024-09-10T19:01:59.647 INFO:teuthology.orchestra.run.smithi191.stdout: Preparing : 1/1 2024-09-10T19:01:59.756 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : librados-devel-2:17.2.7-1606.g5be57395.el9.x86_64 1/1 2024-09-10T19:02:00.185 INFO:teuthology.orchestra.run.smithi183.stdout:Last metadata expiration check: 0:01:36 ago on Tue 10 Sep 2024 07:00:24 PM UTC. 2024-09-10T19:02:00.207 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: librados-devel-2:17.2.7-1606.g5be57395.el9.x86_64 1/1 2024-09-10T19:02:00.276 INFO:teuthology.orchestra.run.smithi183.stdout:Package python3-rgw-2:17.2.7-1606.g5be57395.el9.x86_64 is already installed. 2024-09-10T19:02:00.319 INFO:teuthology.orchestra.run.smithi183.stdout:Dependencies resolved. 2024-09-10T19:02:00.320 INFO:teuthology.orchestra.run.smithi183.stdout:Nothing to do. 2024-09-10T19:02:00.320 INFO:teuthology.orchestra.run.smithi183.stdout:Complete! 2024-09-10T19:02:00.395 DEBUG:teuthology.orchestra.run.smithi183:> sudo yum -y install python3-cephfs 2024-09-10T19:02:00.449 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : librados-devel-2:17.2.7-1606.g5be57395.el9.x86_64 1/1 2024-09-10T19:02:00.449 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-10T19:02:00.449 INFO:teuthology.orchestra.run.smithi191.stdout:Installed: 2024-09-10T19:02:00.449 INFO:teuthology.orchestra.run.smithi191.stdout: librados-devel-2:17.2.7-1606.g5be57395.el9.x86_64 2024-09-10T19:02:00.449 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-10T19:02:00.450 INFO:teuthology.orchestra.run.smithi191.stdout:Complete! 2024-09-10T19:02:00.722 DEBUG:teuthology.orchestra.run.smithi191:> sudo yum -y install libcephfs2 2024-09-10T19:02:00.956 INFO:teuthology.orchestra.run.smithi183.stdout:Last metadata expiration check: 0:01:36 ago on Tue 10 Sep 2024 07:00:24 PM UTC. 2024-09-10T19:02:01.047 INFO:teuthology.orchestra.run.smithi183.stdout:Package python3-cephfs-2:17.2.7-1606.g5be57395.el9.x86_64 is already installed. 2024-09-10T19:02:01.091 INFO:teuthology.orchestra.run.smithi183.stdout:Dependencies resolved. 2024-09-10T19:02:01.092 INFO:teuthology.orchestra.run.smithi183.stdout:Nothing to do. 2024-09-10T19:02:01.092 INFO:teuthology.orchestra.run.smithi183.stdout:Complete! 2024-09-10T19:02:01.158 DEBUG:teuthology.orchestra.run.smithi183:> sudo yum -y install python3-rbd 2024-09-10T19:02:01.296 INFO:teuthology.orchestra.run.smithi191.stdout:Last metadata expiration check: 0:01:36 ago on Tue 10 Sep 2024 07:00:25 PM UTC. 2024-09-10T19:02:01.387 INFO:teuthology.orchestra.run.smithi191.stdout:Package libcephfs2-2:17.2.7-1606.g5be57395.el9.x86_64 is already installed. 2024-09-10T19:02:01.433 INFO:teuthology.orchestra.run.smithi191.stdout:Dependencies resolved. 2024-09-10T19:02:01.433 INFO:teuthology.orchestra.run.smithi191.stdout:Nothing to do. 2024-09-10T19:02:01.433 INFO:teuthology.orchestra.run.smithi191.stdout:Complete! 2024-09-10T19:02:01.487 DEBUG:teuthology.orchestra.run.smithi191:> sudo yum -y install libcephfs-devel 2024-09-10T19:02:01.723 INFO:teuthology.orchestra.run.smithi183.stdout:Last metadata expiration check: 0:01:37 ago on Tue 10 Sep 2024 07:00:24 PM UTC. 2024-09-10T19:02:01.812 INFO:teuthology.orchestra.run.smithi183.stdout:Package python3-rbd-2:17.2.7-1606.g5be57395.el9.x86_64 is already installed. 2024-09-10T19:02:01.855 INFO:teuthology.orchestra.run.smithi183.stdout:Dependencies resolved. 2024-09-10T19:02:01.855 INFO:teuthology.orchestra.run.smithi183.stdout:Nothing to do. 2024-09-10T19:02:01.856 INFO:teuthology.orchestra.run.smithi183.stdout:Complete! 2024-09-10T19:02:01.942 DEBUG:teuthology.orchestra.run.smithi183:> sudo yum -y install rbd-fuse 2024-09-10T19:02:02.052 INFO:teuthology.orchestra.run.smithi191.stdout:Last metadata expiration check: 0:01:37 ago on Tue 10 Sep 2024 07:00:25 PM UTC. 2024-09-10T19:02:02.186 INFO:teuthology.orchestra.run.smithi191.stdout:Dependencies resolved. 2024-09-10T19:02:02.187 INFO:teuthology.orchestra.run.smithi191.stdout:================================================================================ 2024-09-10T19:02:02.187 INFO:teuthology.orchestra.run.smithi191.stdout: Package Arch Version Repo Size 2024-09-10T19:02:02.187 INFO:teuthology.orchestra.run.smithi191.stdout:================================================================================ 2024-09-10T19:02:02.187 INFO:teuthology.orchestra.run.smithi191.stdout:Installing: 2024-09-10T19:02:02.187 INFO:teuthology.orchestra.run.smithi191.stdout: libcephfs-devel x86_64 2:17.2.7-1606.g5be57395.el9 ceph 25 k 2024-09-10T19:02:02.187 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-10T19:02:02.188 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction Summary 2024-09-10T19:02:02.188 INFO:teuthology.orchestra.run.smithi191.stdout:================================================================================ 2024-09-10T19:02:02.188 INFO:teuthology.orchestra.run.smithi191.stdout:Install 1 Package 2024-09-10T19:02:02.188 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-10T19:02:02.188 INFO:teuthology.orchestra.run.smithi191.stdout:Total download size: 25 k 2024-09-10T19:02:02.188 INFO:teuthology.orchestra.run.smithi191.stdout:Installed size: 110 k 2024-09-10T19:02:02.188 INFO:teuthology.orchestra.run.smithi191.stdout:Downloading Packages: 2024-09-10T19:02:02.329 INFO:teuthology.orchestra.run.smithi191.stdout:libcephfs-devel-17.2.7-1606.g5be57395.el9.x86_6 176 kB/s | 25 kB 00:00 2024-09-10T19:02:02.330 INFO:teuthology.orchestra.run.smithi191.stdout:-------------------------------------------------------------------------------- 2024-09-10T19:02:02.330 INFO:teuthology.orchestra.run.smithi191.stdout:Total 175 kB/s | 25 kB 00:00 2024-09-10T19:02:02.330 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction check 2024-09-10T19:02:02.334 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction check succeeded. 2024-09-10T19:02:02.334 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction test 2024-09-10T19:02:02.335 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction test succeeded. 2024-09-10T19:02:02.335 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction 2024-09-10T19:02:02.424 INFO:teuthology.orchestra.run.smithi191.stdout: Preparing : 1/1 2024-09-10T19:02:02.507 INFO:teuthology.orchestra.run.smithi183.stdout:Last metadata expiration check: 0:01:38 ago on Tue 10 Sep 2024 07:00:24 PM UTC. 2024-09-10T19:02:02.526 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : libcephfs-devel-2:17.2.7-1606.g5be57395.el9.x86_64 1/1 2024-09-10T19:02:02.639 INFO:teuthology.orchestra.run.smithi183.stdout:Dependencies resolved. 2024-09-10T19:02:02.640 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2024-09-10T19:02:02.640 INFO:teuthology.orchestra.run.smithi183.stdout: Package Architecture Version Repository Size 2024-09-10T19:02:02.640 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2024-09-10T19:02:02.640 INFO:teuthology.orchestra.run.smithi183.stdout:Installing: 2024-09-10T19:02:02.640 INFO:teuthology.orchestra.run.smithi183.stdout: rbd-fuse x86_64 2:17.2.7-1606.g5be57395.el9 ceph 82 k 2024-09-10T19:02:02.640 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-09-10T19:02:02.640 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction Summary 2024-09-10T19:02:02.640 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2024-09-10T19:02:02.640 INFO:teuthology.orchestra.run.smithi183.stdout:Install 1 Package 2024-09-10T19:02:02.640 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-09-10T19:02:02.640 INFO:teuthology.orchestra.run.smithi183.stdout:Total download size: 82 k 2024-09-10T19:02:02.641 INFO:teuthology.orchestra.run.smithi183.stdout:Installed size: 222 k 2024-09-10T19:02:02.641 INFO:teuthology.orchestra.run.smithi183.stdout:Downloading Packages: 2024-09-10T19:02:02.835 INFO:teuthology.orchestra.run.smithi183.stdout:rbd-fuse-17.2.7-1606.g5be57395.el9.x86_64.rpm 421 kB/s | 82 kB 00:00 2024-09-10T19:02:02.835 INFO:teuthology.orchestra.run.smithi183.stdout:-------------------------------------------------------------------------------- 2024-09-10T19:02:02.835 INFO:teuthology.orchestra.run.smithi183.stdout:Total 418 kB/s | 82 kB 00:00 2024-09-10T19:02:02.836 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction check 2024-09-10T19:02:02.842 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction check succeeded. 2024-09-10T19:02:02.842 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction test 2024-09-10T19:02:02.883 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: libcephfs-devel-2:17.2.7-1606.g5be57395.el9.x86_64 1/1 2024-09-10T19:02:02.906 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction test succeeded. 2024-09-10T19:02:02.906 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction 2024-09-10T19:02:03.058 INFO:teuthology.orchestra.run.smithi183.stdout: Preparing : 1/1 2024-09-10T19:02:03.113 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : libcephfs-devel-2:17.2.7-1606.g5be57395.el9.x86_64 1/1 2024-09-10T19:02:03.113 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-10T19:02:03.113 INFO:teuthology.orchestra.run.smithi191.stdout:Installed: 2024-09-10T19:02:03.114 INFO:teuthology.orchestra.run.smithi191.stdout: libcephfs-devel-2:17.2.7-1606.g5be57395.el9.x86_64 2024-09-10T19:02:03.114 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-10T19:02:03.114 INFO:teuthology.orchestra.run.smithi191.stdout:Complete! 2024-09-10T19:02:03.148 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : rbd-fuse-2:17.2.7-1606.g5be57395.el9.x86_64 1/1 2024-09-10T19:02:03.293 DEBUG:teuthology.orchestra.run.smithi191:> sudo yum -y install librados2 2024-09-10T19:02:03.527 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: rbd-fuse-2:17.2.7-1606.g5be57395.el9.x86_64 1/1 2024-09-10T19:02:03.785 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : rbd-fuse-2:17.2.7-1606.g5be57395.el9.x86_64 1/1 2024-09-10T19:02:03.785 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-09-10T19:02:03.785 INFO:teuthology.orchestra.run.smithi183.stdout:Installed: 2024-09-10T19:02:03.785 INFO:teuthology.orchestra.run.smithi183.stdout: rbd-fuse-2:17.2.7-1606.g5be57395.el9.x86_64 2024-09-10T19:02:03.785 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-09-10T19:02:03.785 INFO:teuthology.orchestra.run.smithi183.stdout:Complete! 2024-09-10T19:02:03.855 INFO:teuthology.orchestra.run.smithi191.stdout:Last metadata expiration check: 0:01:38 ago on Tue 10 Sep 2024 07:00:25 PM UTC. 2024-09-10T19:02:03.947 INFO:teuthology.orchestra.run.smithi191.stdout:Package librados2-2:17.2.7-1606.g5be57395.el9.x86_64 is already installed. 2024-09-10T19:02:03.990 INFO:teuthology.orchestra.run.smithi191.stdout:Dependencies resolved. 2024-09-10T19:02:03.991 INFO:teuthology.orchestra.run.smithi191.stdout:Nothing to do. 2024-09-10T19:02:03.991 INFO:teuthology.orchestra.run.smithi191.stdout:Complete! 2024-09-10T19:02:04.053 DEBUG:teuthology.orchestra.run.smithi191:> sudo yum -y install librbd1 2024-09-10T19:02:04.063 DEBUG:teuthology.orchestra.run.smithi183:> sudo yum -y install rbd-mirror 2024-09-10T19:02:04.619 INFO:teuthology.orchestra.run.smithi191.stdout:Last metadata expiration check: 0:01:39 ago on Tue 10 Sep 2024 07:00:25 PM UTC. 2024-09-10T19:02:04.637 INFO:teuthology.orchestra.run.smithi183.stdout:Last metadata expiration check: 0:01:40 ago on Tue 10 Sep 2024 07:00:24 PM UTC. 2024-09-10T19:02:04.708 INFO:teuthology.orchestra.run.smithi191.stdout:Package librbd1-2:17.2.7-1606.g5be57395.el9.x86_64 is already installed. 2024-09-10T19:02:04.751 INFO:teuthology.orchestra.run.smithi191.stdout:Dependencies resolved. 2024-09-10T19:02:04.752 INFO:teuthology.orchestra.run.smithi191.stdout:Nothing to do. 2024-09-10T19:02:04.752 INFO:teuthology.orchestra.run.smithi191.stdout:Complete! 2024-09-10T19:02:04.772 INFO:teuthology.orchestra.run.smithi183.stdout:Dependencies resolved. 2024-09-10T19:02:04.772 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2024-09-10T19:02:04.772 INFO:teuthology.orchestra.run.smithi183.stdout: Package Arch Version Repo Size 2024-09-10T19:02:04.773 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2024-09-10T19:02:04.773 INFO:teuthology.orchestra.run.smithi183.stdout:Installing: 2024-09-10T19:02:04.773 INFO:teuthology.orchestra.run.smithi183.stdout: rbd-mirror x86_64 2:17.2.7-1606.g5be57395.el9 ceph 2.9 M 2024-09-10T19:02:04.773 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-09-10T19:02:04.773 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction Summary 2024-09-10T19:02:04.773 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2024-09-10T19:02:04.773 INFO:teuthology.orchestra.run.smithi183.stdout:Install 1 Package 2024-09-10T19:02:04.773 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-09-10T19:02:04.774 INFO:teuthology.orchestra.run.smithi183.stdout:Total download size: 2.9 M 2024-09-10T19:02:04.774 INFO:teuthology.orchestra.run.smithi183.stdout:Installed size: 12 M 2024-09-10T19:02:04.774 INFO:teuthology.orchestra.run.smithi183.stdout:Downloading Packages: 2024-09-10T19:02:04.814 DEBUG:teuthology.orchestra.run.smithi191:> sudo yum -y install python3-rados 2024-09-10T19:02:05.096 INFO:teuthology.orchestra.run.smithi183.stdout:rbd-mirror-17.2.7-1606.g5be57395.el9.x86_64.rpm 9.1 MB/s | 2.9 MB 00:00 2024-09-10T19:02:05.097 INFO:teuthology.orchestra.run.smithi183.stdout:-------------------------------------------------------------------------------- 2024-09-10T19:02:05.097 INFO:teuthology.orchestra.run.smithi183.stdout:Total 9.1 MB/s | 2.9 MB 00:00 2024-09-10T19:02:05.097 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction check 2024-09-10T19:02:05.106 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction check succeeded. 2024-09-10T19:02:05.106 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction test 2024-09-10T19:02:05.185 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction test succeeded. 2024-09-10T19:02:05.186 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction 2024-09-10T19:02:05.379 INFO:teuthology.orchestra.run.smithi191.stdout:Last metadata expiration check: 0:01:40 ago on Tue 10 Sep 2024 07:00:25 PM UTC. 2024-09-10T19:02:05.469 INFO:teuthology.orchestra.run.smithi191.stdout:Package python3-rados-2:17.2.7-1606.g5be57395.el9.x86_64 is already installed. 2024-09-10T19:02:05.512 INFO:teuthology.orchestra.run.smithi191.stdout:Dependencies resolved. 2024-09-10T19:02:05.513 INFO:teuthology.orchestra.run.smithi191.stdout:Nothing to do. 2024-09-10T19:02:05.513 INFO:teuthology.orchestra.run.smithi191.stdout:Complete! 2024-09-10T19:02:05.567 DEBUG:teuthology.orchestra.run.smithi191:> sudo yum -y install python3-rgw 2024-09-10T19:02:05.600 INFO:teuthology.orchestra.run.smithi183.stdout: Preparing : 1/1 2024-09-10T19:02:05.672 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : rbd-mirror-2:17.2.7-1606.g5be57395.el9.x86_64 1/1 2024-09-10T19:02:05.697 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: rbd-mirror-2:17.2.7-1606.g5be57395.el9.x86_64 1/1 2024-09-10T19:02:05.697 INFO:teuthology.orchestra.run.smithi183.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-10T19:02:05.697 INFO:teuthology.orchestra.run.smithi183.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-09-10T19:02:05.698 INFO:teuthology.orchestra.run.smithi183.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-10T19:02:05.698 INFO:teuthology.orchestra.run.smithi183.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-10T19:02:05.698 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-09-10T19:02:06.141 INFO:teuthology.orchestra.run.smithi191.stdout:Last metadata expiration check: 0:01:41 ago on Tue 10 Sep 2024 07:00:25 PM UTC. 2024-09-10T19:02:06.232 INFO:teuthology.orchestra.run.smithi191.stdout:Package python3-rgw-2:17.2.7-1606.g5be57395.el9.x86_64 is already installed. 2024-09-10T19:02:06.275 INFO:teuthology.orchestra.run.smithi191.stdout:Dependencies resolved. 2024-09-10T19:02:06.276 INFO:teuthology.orchestra.run.smithi191.stdout:Nothing to do. 2024-09-10T19:02:06.276 INFO:teuthology.orchestra.run.smithi191.stdout:Complete! 2024-09-10T19:02:06.335 DEBUG:teuthology.orchestra.run.smithi191:> sudo yum -y install python3-cephfs 2024-09-10T19:02:06.745 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : rbd-mirror-2:17.2.7-1606.g5be57395.el9.x86_64 1/1 2024-09-10T19:02:06.745 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-09-10T19:02:06.745 INFO:teuthology.orchestra.run.smithi183.stdout:Installed: 2024-09-10T19:02:06.745 INFO:teuthology.orchestra.run.smithi183.stdout: rbd-mirror-2:17.2.7-1606.g5be57395.el9.x86_64 2024-09-10T19:02:06.745 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-09-10T19:02:06.745 INFO:teuthology.orchestra.run.smithi183.stdout:Complete! 2024-09-10T19:02:06.908 INFO:teuthology.orchestra.run.smithi191.stdout:Last metadata expiration check: 0:01:41 ago on Tue 10 Sep 2024 07:00:25 PM UTC. 2024-09-10T19:02:06.974 DEBUG:teuthology.orchestra.run.smithi183:> sudo yum -y install rbd-nbd 2024-09-10T19:02:06.999 INFO:teuthology.orchestra.run.smithi191.stdout:Package python3-cephfs-2:17.2.7-1606.g5be57395.el9.x86_64 is already installed. 2024-09-10T19:02:07.043 INFO:teuthology.orchestra.run.smithi191.stdout:Dependencies resolved. 2024-09-10T19:02:07.044 INFO:teuthology.orchestra.run.smithi191.stdout:Nothing to do. 2024-09-10T19:02:07.044 INFO:teuthology.orchestra.run.smithi191.stdout:Complete! 2024-09-10T19:02:07.104 DEBUG:teuthology.orchestra.run.smithi191:> sudo yum -y install python3-rbd 2024-09-10T19:02:07.553 INFO:teuthology.orchestra.run.smithi183.stdout:Last metadata expiration check: 0:01:43 ago on Tue 10 Sep 2024 07:00:24 PM UTC. 2024-09-10T19:02:07.675 INFO:teuthology.orchestra.run.smithi191.stdout:Last metadata expiration check: 0:01:42 ago on Tue 10 Sep 2024 07:00:25 PM UTC. 2024-09-10T19:02:07.690 INFO:teuthology.orchestra.run.smithi183.stdout:Dependencies resolved. 2024-09-10T19:02:07.690 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2024-09-10T19:02:07.690 INFO:teuthology.orchestra.run.smithi183.stdout: Package Architecture Version Repository Size 2024-09-10T19:02:07.691 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2024-09-10T19:02:07.691 INFO:teuthology.orchestra.run.smithi183.stdout:Installing: 2024-09-10T19:02:07.691 INFO:teuthology.orchestra.run.smithi183.stdout: rbd-nbd x86_64 2:17.2.7-1606.g5be57395.el9 ceph 165 k 2024-09-10T19:02:07.691 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-09-10T19:02:07.691 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction Summary 2024-09-10T19:02:07.691 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2024-09-10T19:02:07.691 INFO:teuthology.orchestra.run.smithi183.stdout:Install 1 Package 2024-09-10T19:02:07.691 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-09-10T19:02:07.692 INFO:teuthology.orchestra.run.smithi183.stdout:Total download size: 165 k 2024-09-10T19:02:07.692 INFO:teuthology.orchestra.run.smithi183.stdout:Installed size: 477 k 2024-09-10T19:02:07.692 INFO:teuthology.orchestra.run.smithi183.stdout:Downloading Packages: 2024-09-10T19:02:07.766 INFO:teuthology.orchestra.run.smithi191.stdout:Package python3-rbd-2:17.2.7-1606.g5be57395.el9.x86_64 is already installed. 2024-09-10T19:02:07.809 INFO:teuthology.orchestra.run.smithi191.stdout:Dependencies resolved. 2024-09-10T19:02:07.810 INFO:teuthology.orchestra.run.smithi191.stdout:Nothing to do. 2024-09-10T19:02:07.810 INFO:teuthology.orchestra.run.smithi191.stdout:Complete! 2024-09-10T19:02:07.866 DEBUG:teuthology.orchestra.run.smithi191:> sudo yum -y install rbd-fuse 2024-09-10T19:02:07.891 INFO:teuthology.orchestra.run.smithi183.stdout:rbd-nbd-17.2.7-1606.g5be57395.el9.x86_64.rpm 829 kB/s | 165 kB 00:00 2024-09-10T19:02:07.891 INFO:teuthology.orchestra.run.smithi183.stdout:-------------------------------------------------------------------------------- 2024-09-10T19:02:07.892 INFO:teuthology.orchestra.run.smithi183.stdout:Total 824 kB/s | 165 kB 00:00 2024-09-10T19:02:07.892 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction check 2024-09-10T19:02:07.899 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction check succeeded. 2024-09-10T19:02:07.899 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction test 2024-09-10T19:02:07.962 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction test succeeded. 2024-09-10T19:02:07.963 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction 2024-09-10T19:02:08.116 INFO:teuthology.orchestra.run.smithi183.stdout: Preparing : 1/1 2024-09-10T19:02:08.221 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : rbd-nbd-2:17.2.7-1606.g5be57395.el9.x86_64 1/1 2024-09-10T19:02:08.431 INFO:teuthology.orchestra.run.smithi191.stdout:Last metadata expiration check: 0:01:43 ago on Tue 10 Sep 2024 07:00:25 PM UTC. 2024-09-10T19:02:08.564 INFO:teuthology.orchestra.run.smithi191.stdout:Dependencies resolved. 2024-09-10T19:02:08.565 INFO:teuthology.orchestra.run.smithi191.stdout:================================================================================ 2024-09-10T19:02:08.565 INFO:teuthology.orchestra.run.smithi191.stdout: Package Architecture Version Repository Size 2024-09-10T19:02:08.565 INFO:teuthology.orchestra.run.smithi191.stdout:================================================================================ 2024-09-10T19:02:08.566 INFO:teuthology.orchestra.run.smithi191.stdout:Installing: 2024-09-10T19:02:08.566 INFO:teuthology.orchestra.run.smithi191.stdout: rbd-fuse x86_64 2:17.2.7-1606.g5be57395.el9 ceph 82 k 2024-09-10T19:02:08.566 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-10T19:02:08.566 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction Summary 2024-09-10T19:02:08.566 INFO:teuthology.orchestra.run.smithi191.stdout:================================================================================ 2024-09-10T19:02:08.566 INFO:teuthology.orchestra.run.smithi191.stdout:Install 1 Package 2024-09-10T19:02:08.566 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-10T19:02:08.566 INFO:teuthology.orchestra.run.smithi191.stdout:Total download size: 82 k 2024-09-10T19:02:08.566 INFO:teuthology.orchestra.run.smithi191.stdout:Installed size: 222 k 2024-09-10T19:02:08.566 INFO:teuthology.orchestra.run.smithi191.stdout:Downloading Packages: 2024-09-10T19:02:08.688 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: rbd-nbd-2:17.2.7-1606.g5be57395.el9.x86_64 1/1 2024-09-10T19:02:08.757 INFO:teuthology.orchestra.run.smithi191.stdout:rbd-fuse-17.2.7-1606.g5be57395.el9.x86_64.rpm 428 kB/s | 82 kB 00:00 2024-09-10T19:02:08.758 INFO:teuthology.orchestra.run.smithi191.stdout:-------------------------------------------------------------------------------- 2024-09-10T19:02:08.758 INFO:teuthology.orchestra.run.smithi191.stdout:Total 425 kB/s | 82 kB 00:00 2024-09-10T19:02:08.758 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction check 2024-09-10T19:02:08.765 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction check succeeded. 2024-09-10T19:02:08.765 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction test 2024-09-10T19:02:08.828 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction test succeeded. 2024-09-10T19:02:08.829 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction 2024-09-10T19:02:08.964 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : rbd-nbd-2:17.2.7-1606.g5be57395.el9.x86_64 1/1 2024-09-10T19:02:08.964 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-09-10T19:02:08.964 INFO:teuthology.orchestra.run.smithi183.stdout:Installed: 2024-09-10T19:02:08.964 INFO:teuthology.orchestra.run.smithi183.stdout: rbd-nbd-2:17.2.7-1606.g5be57395.el9.x86_64 2024-09-10T19:02:08.964 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-09-10T19:02:08.964 INFO:teuthology.orchestra.run.smithi183.stdout:Complete! 2024-09-10T19:02:08.971 INFO:teuthology.orchestra.run.smithi191.stdout: Preparing : 1/1 2024-09-10T19:02:09.063 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : rbd-fuse-2:17.2.7-1606.g5be57395.el9.x86_64 1/1 2024-09-10T19:02:09.192 DEBUG:teuthology.parallel:result is None 2024-09-10T19:02:09.439 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: rbd-fuse-2:17.2.7-1606.g5be57395.el9.x86_64 1/1 2024-09-10T19:02:09.688 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : rbd-fuse-2:17.2.7-1606.g5be57395.el9.x86_64 1/1 2024-09-10T19:02:09.688 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-10T19:02:09.688 INFO:teuthology.orchestra.run.smithi191.stdout:Installed: 2024-09-10T19:02:09.688 INFO:teuthology.orchestra.run.smithi191.stdout: rbd-fuse-2:17.2.7-1606.g5be57395.el9.x86_64 2024-09-10T19:02:09.688 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-10T19:02:09.688 INFO:teuthology.orchestra.run.smithi191.stdout:Complete! 2024-09-10T19:02:09.931 DEBUG:teuthology.orchestra.run.smithi191:> sudo yum -y install rbd-mirror 2024-09-10T19:02:10.499 INFO:teuthology.orchestra.run.smithi191.stdout:Last metadata expiration check: 0:01:45 ago on Tue 10 Sep 2024 07:00:25 PM UTC. 2024-09-10T19:02:10.633 INFO:teuthology.orchestra.run.smithi191.stdout:Dependencies resolved. 2024-09-10T19:02:10.633 INFO:teuthology.orchestra.run.smithi191.stdout:================================================================================ 2024-09-10T19:02:10.633 INFO:teuthology.orchestra.run.smithi191.stdout: Package Arch Version Repo Size 2024-09-10T19:02:10.634 INFO:teuthology.orchestra.run.smithi191.stdout:================================================================================ 2024-09-10T19:02:10.634 INFO:teuthology.orchestra.run.smithi191.stdout:Installing: 2024-09-10T19:02:10.634 INFO:teuthology.orchestra.run.smithi191.stdout: rbd-mirror x86_64 2:17.2.7-1606.g5be57395.el9 ceph 2.9 M 2024-09-10T19:02:10.634 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-10T19:02:10.634 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction Summary 2024-09-10T19:02:10.634 INFO:teuthology.orchestra.run.smithi191.stdout:================================================================================ 2024-09-10T19:02:10.634 INFO:teuthology.orchestra.run.smithi191.stdout:Install 1 Package 2024-09-10T19:02:10.634 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-10T19:02:10.634 INFO:teuthology.orchestra.run.smithi191.stdout:Total download size: 2.9 M 2024-09-10T19:02:10.635 INFO:teuthology.orchestra.run.smithi191.stdout:Installed size: 12 M 2024-09-10T19:02:10.635 INFO:teuthology.orchestra.run.smithi191.stdout:Downloading Packages: 2024-09-10T19:02:10.952 INFO:teuthology.orchestra.run.smithi191.stdout:rbd-mirror-17.2.7-1606.g5be57395.el9.x86_64.rpm 9.3 MB/s | 2.9 MB 00:00 2024-09-10T19:02:10.952 INFO:teuthology.orchestra.run.smithi191.stdout:-------------------------------------------------------------------------------- 2024-09-10T19:02:10.953 INFO:teuthology.orchestra.run.smithi191.stdout:Total 9.2 MB/s | 2.9 MB 00:00 2024-09-10T19:02:10.953 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction check 2024-09-10T19:02:10.961 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction check succeeded. 2024-09-10T19:02:10.962 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction test 2024-09-10T19:02:11.043 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction test succeeded. 2024-09-10T19:02:11.043 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction 2024-09-10T19:02:11.455 INFO:teuthology.orchestra.run.smithi191.stdout: Preparing : 1/1 2024-09-10T19:02:11.519 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : rbd-mirror-2:17.2.7-1606.g5be57395.el9.x86_64 1/1 2024-09-10T19:02:11.544 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: rbd-mirror-2:17.2.7-1606.g5be57395.el9.x86_64 1/1 2024-09-10T19:02:11.544 INFO:teuthology.orchestra.run.smithi191.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-10T19:02:11.544 INFO:teuthology.orchestra.run.smithi191.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-09-10T19:02:11.544 INFO:teuthology.orchestra.run.smithi191.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-10T19:02:11.545 INFO:teuthology.orchestra.run.smithi191.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-10T19:02:11.545 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-10T19:02:12.638 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : rbd-mirror-2:17.2.7-1606.g5be57395.el9.x86_64 1/1 2024-09-10T19:02:12.638 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-10T19:02:12.638 INFO:teuthology.orchestra.run.smithi191.stdout:Installed: 2024-09-10T19:02:12.638 INFO:teuthology.orchestra.run.smithi191.stdout: rbd-mirror-2:17.2.7-1606.g5be57395.el9.x86_64 2024-09-10T19:02:12.638 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-10T19:02:12.638 INFO:teuthology.orchestra.run.smithi191.stdout:Complete! 2024-09-10T19:02:12.878 DEBUG:teuthology.orchestra.run.smithi191:> sudo yum -y install rbd-nbd 2024-09-10T19:02:13.459 INFO:teuthology.orchestra.run.smithi191.stdout:Last metadata expiration check: 0:01:48 ago on Tue 10 Sep 2024 07:00:25 PM UTC. 2024-09-10T19:02:13.600 INFO:teuthology.orchestra.run.smithi191.stdout:Dependencies resolved. 2024-09-10T19:02:13.601 INFO:teuthology.orchestra.run.smithi191.stdout:================================================================================ 2024-09-10T19:02:13.601 INFO:teuthology.orchestra.run.smithi191.stdout: Package Architecture Version Repository Size 2024-09-10T19:02:13.601 INFO:teuthology.orchestra.run.smithi191.stdout:================================================================================ 2024-09-10T19:02:13.601 INFO:teuthology.orchestra.run.smithi191.stdout:Installing: 2024-09-10T19:02:13.602 INFO:teuthology.orchestra.run.smithi191.stdout: rbd-nbd x86_64 2:17.2.7-1606.g5be57395.el9 ceph 165 k 2024-09-10T19:02:13.602 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-10T19:02:13.602 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction Summary 2024-09-10T19:02:13.602 INFO:teuthology.orchestra.run.smithi191.stdout:================================================================================ 2024-09-10T19:02:13.602 INFO:teuthology.orchestra.run.smithi191.stdout:Install 1 Package 2024-09-10T19:02:13.602 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-10T19:02:13.602 INFO:teuthology.orchestra.run.smithi191.stdout:Total download size: 165 k 2024-09-10T19:02:13.602 INFO:teuthology.orchestra.run.smithi191.stdout:Installed size: 477 k 2024-09-10T19:02:13.602 INFO:teuthology.orchestra.run.smithi191.stdout:Downloading Packages: 2024-09-10T19:02:13.805 INFO:teuthology.orchestra.run.smithi191.stdout:rbd-nbd-17.2.7-1606.g5be57395.el9.x86_64.rpm 814 kB/s | 165 kB 00:00 2024-09-10T19:02:13.806 INFO:teuthology.orchestra.run.smithi191.stdout:-------------------------------------------------------------------------------- 2024-09-10T19:02:13.806 INFO:teuthology.orchestra.run.smithi191.stdout:Total 806 kB/s | 165 kB 00:00 2024-09-10T19:02:13.807 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction check 2024-09-10T19:02:13.814 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction check succeeded. 2024-09-10T19:02:13.814 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction test 2024-09-10T19:02:13.882 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction test succeeded. 2024-09-10T19:02:13.883 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction 2024-09-10T19:02:14.042 INFO:teuthology.orchestra.run.smithi191.stdout: Preparing : 1/1 2024-09-10T19:02:14.146 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : rbd-nbd-2:17.2.7-1606.g5be57395.el9.x86_64 1/1 2024-09-10T19:02:14.601 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: rbd-nbd-2:17.2.7-1606.g5be57395.el9.x86_64 1/1 2024-09-10T19:02:14.860 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : rbd-nbd-2:17.2.7-1606.g5be57395.el9.x86_64 1/1 2024-09-10T19:02:14.860 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-10T19:02:14.860 INFO:teuthology.orchestra.run.smithi191.stdout:Installed: 2024-09-10T19:02:14.860 INFO:teuthology.orchestra.run.smithi191.stdout: rbd-nbd-2:17.2.7-1606.g5be57395.el9.x86_64 2024-09-10T19:02:14.860 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-10T19:02:14.860 INFO:teuthology.orchestra.run.smithi191.stdout:Complete! 2024-09-10T19:02:15.102 DEBUG:teuthology.parallel:result is None 2024-09-10T19:02:15.103 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5be57395ea235535910512ae73ba8e7ff25fe588 2024-09-10T19:02:15.247 DEBUG:teuthology.orchestra.run.smithi183:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-09-10T19:02:15.273 INFO:teuthology.orchestra.run.smithi183.stdout:17.2.7-1606.g5be57395.el9 2024-09-10T19:02:15.273 INFO:teuthology.packaging:The installed version of ceph is 17.2.7-1606.g5be57395.el9 2024-09-10T19:02:15.274 INFO:teuthology.task.install:The correct ceph version 17.2.7-1606.g5be57395 is installed. 2024-09-10T19:02:15.275 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5be57395ea235535910512ae73ba8e7ff25fe588 2024-09-10T19:02:15.418 DEBUG:teuthology.orchestra.run.smithi191:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-09-10T19:02:15.441 INFO:teuthology.orchestra.run.smithi191.stdout:17.2.7-1606.g5be57395.el9 2024-09-10T19:02:15.441 INFO:teuthology.packaging:The installed version of ceph is 17.2.7-1606.g5be57395.el9 2024-09-10T19:02:15.441 INFO:teuthology.task.install:The correct ceph version 17.2.7-1606.g5be57395 is installed. 2024-09-10T19:02:15.443 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-09-10T19:02:15.443 DEBUG:teuthology.orchestra.run.smithi183:> set -ex 2024-09-10T19:02:15.443 DEBUG:teuthology.orchestra.run.smithi183:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-09-10T19:02:15.476 DEBUG:teuthology.orchestra.run.smithi191:> set -ex 2024-09-10T19:02:15.476 DEBUG:teuthology.orchestra.run.smithi191:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-09-10T19:02:15.507 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-09-10T19:02:15.508 DEBUG:teuthology.orchestra.run.smithi183:> set -ex 2024-09-10T19:02:15.508 DEBUG:teuthology.orchestra.run.smithi183:> sudo dd of=/usr/bin/daemon-helper 2024-09-10T19:02:15.544 DEBUG:teuthology.orchestra.run.smithi183:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-09-10T19:02:15.611 DEBUG:teuthology.orchestra.run.smithi191:> set -ex 2024-09-10T19:02:15.611 DEBUG:teuthology.orchestra.run.smithi191:> sudo dd of=/usr/bin/daemon-helper 2024-09-10T19:02:15.637 DEBUG:teuthology.orchestra.run.smithi191:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-09-10T19:02:15.701 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-09-10T19:02:15.702 DEBUG:teuthology.orchestra.run.smithi183:> set -ex 2024-09-10T19:02:15.702 DEBUG:teuthology.orchestra.run.smithi183:> sudo dd of=/usr/bin/adjust-ulimits 2024-09-10T19:02:15.729 DEBUG:teuthology.orchestra.run.smithi183:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-09-10T19:02:15.797 DEBUG:teuthology.orchestra.run.smithi191:> set -ex 2024-09-10T19:02:15.797 DEBUG:teuthology.orchestra.run.smithi191:> sudo dd of=/usr/bin/adjust-ulimits 2024-09-10T19:02:15.842 DEBUG:teuthology.orchestra.run.smithi191:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-09-10T19:02:15.907 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-09-10T19:02:15.907 DEBUG:teuthology.orchestra.run.smithi183:> set -ex 2024-09-10T19:02:15.907 DEBUG:teuthology.orchestra.run.smithi183:> sudo dd of=/usr/bin/stdin-killer 2024-09-10T19:02:15.937 DEBUG:teuthology.orchestra.run.smithi183:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-09-10T19:02:16.005 DEBUG:teuthology.orchestra.run.smithi191:> set -ex 2024-09-10T19:02:16.005 DEBUG:teuthology.orchestra.run.smithi191:> sudo dd of=/usr/bin/stdin-killer 2024-09-10T19:02:16.030 DEBUG:teuthology.orchestra.run.smithi191:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-09-10T19:02:16.094 INFO:teuthology.run_tasks:Running task cephadm... 2024-09-10T19:02:16.186 INFO:tasks.cephadm:Config: {'conf': {'mgr': {'debug mgr': 20, 'debug ms': 1}, 'client.0': {'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'global': {'ms type': 'async'}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'paxos service trim min': 10}, 'osd': {'debug ms': 1, 'debug osd': 20, 'osd debug reject backfill probability': 0.3, 'osd delete sleep': 1, 'osd max backfills': 3, 'osd scrub max interval': 120, 'osd scrub min interval': 60, 'osd shutdown pgref assert': True, 'osd snap trim sleep': 2}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'but it is still running', 'objects unfound and apparently lost', '\\(POOL_APP_NOT_ENABLED\\)'], 'sha1': '5be57395ea235535910512ae73ba8e7ff25fe588', 'cephadm_mode': 'root'} 2024-09-10T19:02:16.186 INFO:tasks.cephadm:Cluster image is quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 2024-09-10T19:02:16.186 INFO:tasks.cephadm:Cluster fsid is 320f6df8-6fa7-11ef-bcea-c7b262605968 2024-09-10T19:02:16.187 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2024-09-10T19:02:16.187 INFO:tasks.cephadm:Monitor IPs: {'mon.a': '172.21.15.183', 'mon.c': '[v2:172.21.15.183:3301,v1:172.21.15.183:6790]', 'mon.b': '172.21.15.191'} 2024-09-10T19:02:16.187 INFO:tasks.cephadm:First mon is mon.a on smithi183 2024-09-10T19:02:16.187 INFO:tasks.cephadm:First mgr is y 2024-09-10T19:02:16.187 INFO:tasks.cephadm:Normalizing hostnames... 2024-09-10T19:02:16.187 DEBUG:teuthology.orchestra.run.smithi183:> sudo hostname $(hostname -s) 2024-09-10T19:02:16.217 DEBUG:teuthology.orchestra.run.smithi191:> sudo hostname $(hostname -s) 2024-09-10T19:02:16.242 INFO:tasks.cephadm:Downloading cephadm (repo https://git.ceph.com/ceph-ci.git ref 5be57395ea235535910512ae73ba8e7ff25fe588)... 2024-09-10T19:02:16.242 DEBUG:teuthology.orchestra.run.smithi183:> git clone https://git.ceph.com/ceph-ci.git testrepo && cd testrepo && git show 5be57395ea235535910512ae73ba8e7ff25fe588:src/cephadm/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-09-10T19:02:16.275 INFO:teuthology.orchestra.run.smithi183.stderr:Cloning into 'testrepo'... 2024-09-10T19:03:13.118 INFO:teuthology.orchestra.run.smithi183.stdout:-rw-r--r--. 1 ubuntu ubuntu 378585 Sep 10 19:03 /home/ubuntu/cephtest/cephadm 2024-09-10T19:03:13.119 DEBUG:teuthology.orchestra.run.smithi191:> git clone https://git.ceph.com/ceph-ci.git testrepo && cd testrepo && git show 5be57395ea235535910512ae73ba8e7ff25fe588:src/cephadm/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-09-10T19:03:13.138 INFO:teuthology.orchestra.run.smithi191.stderr:Cloning into 'testrepo'... 2024-09-10T19:04:10.868 INFO:teuthology.orchestra.run.smithi191.stdout:-rw-r--r--. 1 ubuntu ubuntu 378585 Sep 10 19:04 /home/ubuntu/cephtest/cephadm 2024-09-10T19:04:10.869 DEBUG:teuthology.orchestra.run.smithi183:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-09-10T19:04:10.889 DEBUG:teuthology.orchestra.run.smithi191:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-09-10T19:04:10.934 INFO:tasks.cephadm:Pulling image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 on all hosts... 2024-09-10T19:04:10.934 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 pull 2024-09-10T19:04:10.937 DEBUG:teuthology.orchestra.run.smithi191:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 pull 2024-09-10T19:04:12.516 INFO:teuthology.orchestra.run.smithi183.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588... 2024-09-10T19:04:14.776 INFO:teuthology.orchestra.run.smithi191.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588... 2024-09-10T19:04:38.552 INFO:teuthology.orchestra.run.smithi183.stdout:{ 2024-09-10T19:04:38.552 INFO:teuthology.orchestra.run.smithi183.stdout: "ceph_version": "ceph version 17.2.7-1606-g5be57395 (5be57395ea235535910512ae73ba8e7ff25fe588) quincy (stable)", 2024-09-10T19:04:38.552 INFO:teuthology.orchestra.run.smithi183.stdout: "image_id": "4c55dd4096115e48502f368cff1eab06769f71f030e1692e2611bbc6c6f637e9", 2024-09-10T19:04:38.553 INFO:teuthology.orchestra.run.smithi183.stdout: "repo_digests": [ 2024-09-10T19:04:38.553 INFO:teuthology.orchestra.run.smithi183.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3bfeaf4417585c821cab9ca76e4d95f390b0e769807122a45463640820b5cd16" 2024-09-10T19:04:38.553 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:04:38.553 INFO:teuthology.orchestra.run.smithi183.stdout:} 2024-09-10T19:04:43.838 INFO:teuthology.orchestra.run.smithi191.stdout:{ 2024-09-10T19:04:43.838 INFO:teuthology.orchestra.run.smithi191.stdout: "ceph_version": "ceph version 17.2.7-1606-g5be57395 (5be57395ea235535910512ae73ba8e7ff25fe588) quincy (stable)", 2024-09-10T19:04:43.838 INFO:teuthology.orchestra.run.smithi191.stdout: "image_id": "4c55dd4096115e48502f368cff1eab06769f71f030e1692e2611bbc6c6f637e9", 2024-09-10T19:04:43.838 INFO:teuthology.orchestra.run.smithi191.stdout: "repo_digests": [ 2024-09-10T19:04:43.838 INFO:teuthology.orchestra.run.smithi191.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3bfeaf4417585c821cab9ca76e4d95f390b0e769807122a45463640820b5cd16" 2024-09-10T19:04:43.838 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-10T19:04:43.838 INFO:teuthology.orchestra.run.smithi191.stdout:} 2024-09-10T19:04:43.858 DEBUG:teuthology.orchestra.run.smithi183:> sudo mkdir -p /etc/ceph 2024-09-10T19:04:43.891 DEBUG:teuthology.orchestra.run.smithi191:> sudo mkdir -p /etc/ceph 2024-09-10T19:04:43.919 DEBUG:teuthology.orchestra.run.smithi183:> sudo chmod 777 /etc/ceph 2024-09-10T19:04:43.959 DEBUG:teuthology.orchestra.run.smithi191:> sudo chmod 777 /etc/ceph 2024-09-10T19:04:43.985 INFO:tasks.cephadm:Writing seed config... 2024-09-10T19:04:43.986 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2024-09-10T19:04:43.986 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2024-09-10T19:04:43.986 INFO:tasks.cephadm: override: [client.0] debug ms = 1 2024-09-10T19:04:43.987 INFO:tasks.cephadm: override: [client.0] debug objecter = 20 2024-09-10T19:04:43.987 INFO:tasks.cephadm: override: [client.0] debug rados = 20 2024-09-10T19:04:43.987 INFO:tasks.cephadm: override: [global] ms type = async 2024-09-10T19:04:43.987 INFO:tasks.cephadm: override: [mon] debug mon = 20 2024-09-10T19:04:43.987 INFO:tasks.cephadm: override: [mon] debug ms = 1 2024-09-10T19:04:43.987 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2024-09-10T19:04:43.987 INFO:tasks.cephadm: override: [mon] mon min osdmap epochs = 50 2024-09-10T19:04:43.987 INFO:tasks.cephadm: override: [mon] mon osdmap full prune interval = 2 2024-09-10T19:04:43.987 INFO:tasks.cephadm: override: [mon] mon osdmap full prune min = 15 2024-09-10T19:04:43.987 INFO:tasks.cephadm: override: [mon] mon osdmap full prune txsize = 2 2024-09-10T19:04:43.987 INFO:tasks.cephadm: override: [mon] paxos service trim min = 10 2024-09-10T19:04:43.987 INFO:tasks.cephadm: override: [osd] debug ms = 1 2024-09-10T19:04:43.987 INFO:tasks.cephadm: override: [osd] debug osd = 20 2024-09-10T19:04:43.987 INFO:tasks.cephadm: override: [osd] osd debug reject backfill probability = 0.3 2024-09-10T19:04:43.988 INFO:tasks.cephadm: override: [osd] osd delete sleep = 1 2024-09-10T19:04:43.988 INFO:tasks.cephadm: override: [osd] osd max backfills = 3 2024-09-10T19:04:43.988 INFO:tasks.cephadm: override: [osd] osd scrub max interval = 120 2024-09-10T19:04:43.988 INFO:tasks.cephadm: override: [osd] osd scrub min interval = 60 2024-09-10T19:04:43.988 INFO:tasks.cephadm: override: [osd] osd shutdown pgref assert = True 2024-09-10T19:04:43.988 INFO:tasks.cephadm: override: [osd] osd snap trim sleep = 2 2024-09-10T19:04:43.989 DEBUG:teuthology.orchestra.run.smithi183:> set -ex 2024-09-10T19:04:43.989 DEBUG:teuthology.orchestra.run.smithi183:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2024-09-10T19:04:44.017 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = 320f6df8-6fa7-11ef-bcea-c7b262605968 ms type = async [osd] osd scrub load threshold = 5.0 osd scrub max interval = 120 osd mclock profile = high_recovery_ops osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = True bdev debug aio = true osd sloppy crc = true debug ms = 1 debug osd = 20 osd debug reject backfill probability = 0.3 osd delete sleep = 1 osd max backfills = 3 osd scrub min interval = 60 osd snap trim sleep = 2 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false debug mgr = 20 debug ms = 1 [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 mon min osdmap epochs = 50 mon osdmap full prune interval = 2 mon osdmap full prune min = 15 mon osdmap full prune txsize = 2 paxos service trim min = 10 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true [client.0] debug ms = 1 debug objecter = 20 debug rados = 20 2024-09-10T19:04:44.017 DEBUG:teuthology.orchestra.run.smithi183:mon.a> sudo journalctl -f -n 0 -u ceph-320f6df8-6fa7-11ef-bcea-c7b262605968@mon.a.service 2024-09-10T19:04:44.060 DEBUG:teuthology.orchestra.run.smithi183:mgr.y> sudo journalctl -f -n 0 -u ceph-320f6df8-6fa7-11ef-bcea-c7b262605968@mgr.y.service 2024-09-10T19:04:44.103 INFO:tasks.cephadm:Bootstrapping... 2024-09-10T19:04:44.103 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 -v bootstrap --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id y --orphan-initial-daemons --skip-monitoring-stack --mon-ip 172.21.15.183 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2024-09-10T19:04:44.312 INFO:teuthology.orchestra.run.smithi183.stdout:-------------------------------------------------------------------------------- 2024-09-10T19:04:44.312 INFO:teuthology.orchestra.run.smithi183.stdout:cephadm ['--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588', '-v', 'bootstrap', '--fsid', '320f6df8-6fa7-11ef-bcea-c7b262605968', '--config', '/home/ubuntu/cephtest/seed.ceph.conf', '--output-config', '/etc/ceph/ceph.conf', '--output-keyring', '/etc/ceph/ceph.client.admin.keyring', '--output-pub-ssh-key', '/home/ubuntu/cephtest/ceph.pub', '--mon-id', 'a', '--mgr-id', 'y', '--orphan-initial-daemons', '--skip-monitoring-stack', '--mon-ip', '172.21.15.183', '--skip-admin-label'] 2024-09-10T19:04:44.339 INFO:teuthology.orchestra.run.smithi183.stdout:/bin/podman: stdout 5.2.2 2024-09-10T19:04:44.340 INFO:teuthology.orchestra.run.smithi183.stderr:Specifying an fsid for your cluster offers no advantages and may increase the likelihood of fsid conflicts. 2024-09-10T19:04:44.340 INFO:teuthology.orchestra.run.smithi183.stdout:Verifying podman|docker is present... 2024-09-10T19:04:44.366 INFO:teuthology.orchestra.run.smithi183.stdout:/bin/podman: stdout 5.2.2 2024-09-10T19:04:44.366 INFO:teuthology.orchestra.run.smithi183.stdout:Verifying lvm2 is present... 2024-09-10T19:04:44.366 INFO:teuthology.orchestra.run.smithi183.stdout:Verifying time synchronization is in place... 2024-09-10T19:04:44.375 INFO:teuthology.orchestra.run.smithi183.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-09-10T19:04:44.375 INFO:teuthology.orchestra.run.smithi183.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-09-10T19:04:44.382 INFO:teuthology.orchestra.run.smithi183.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-09-10T19:04:44.382 INFO:teuthology.orchestra.run.smithi183.stdout:systemctl: stdout inactive 2024-09-10T19:04:44.389 INFO:teuthology.orchestra.run.smithi183.stdout:systemctl: stdout enabled 2024-09-10T19:04:44.397 INFO:teuthology.orchestra.run.smithi183.stdout:systemctl: stdout active 2024-09-10T19:04:44.397 INFO:teuthology.orchestra.run.smithi183.stdout:Unit chronyd.service is enabled and running 2024-09-10T19:04:44.397 INFO:teuthology.orchestra.run.smithi183.stdout:Repeating the final host check... 2024-09-10T19:04:44.427 INFO:teuthology.orchestra.run.smithi183.stdout:/bin/podman: stdout 5.2.2 2024-09-10T19:04:44.427 INFO:teuthology.orchestra.run.smithi183.stdout:podman (/bin/podman) version 5.2.2 is present 2024-09-10T19:04:44.427 INFO:teuthology.orchestra.run.smithi183.stdout:systemctl is present 2024-09-10T19:04:44.427 INFO:teuthology.orchestra.run.smithi183.stdout:lvcreate is present 2024-09-10T19:04:44.435 INFO:teuthology.orchestra.run.smithi183.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-09-10T19:04:44.435 INFO:teuthology.orchestra.run.smithi183.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-09-10T19:04:44.443 INFO:teuthology.orchestra.run.smithi183.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-09-10T19:04:44.443 INFO:teuthology.orchestra.run.smithi183.stdout:systemctl: stdout inactive 2024-09-10T19:04:44.451 INFO:teuthology.orchestra.run.smithi183.stdout:systemctl: stdout enabled 2024-09-10T19:04:44.459 INFO:teuthology.orchestra.run.smithi183.stdout:systemctl: stdout active 2024-09-10T19:04:44.459 INFO:teuthology.orchestra.run.smithi183.stdout:Unit chronyd.service is enabled and running 2024-09-10T19:04:44.460 INFO:teuthology.orchestra.run.smithi183.stdout:Host looks OK 2024-09-10T19:04:44.460 INFO:teuthology.orchestra.run.smithi183.stdout:Cluster fsid: 320f6df8-6fa7-11ef-bcea-c7b262605968 2024-09-10T19:04:44.460 INFO:teuthology.orchestra.run.smithi183.stdout:Acquiring lock 139710081938336 on /run/cephadm/320f6df8-6fa7-11ef-bcea-c7b262605968.lock 2024-09-10T19:04:44.460 INFO:teuthology.orchestra.run.smithi183.stdout:Lock 139710081938336 acquired on /run/cephadm/320f6df8-6fa7-11ef-bcea-c7b262605968.lock 2024-09-10T19:04:44.460 INFO:teuthology.orchestra.run.smithi183.stdout:Verifying IP 172.21.15.183 port 3300 ... 2024-09-10T19:04:44.461 INFO:teuthology.orchestra.run.smithi183.stdout:Verifying IP 172.21.15.183 port 6789 ... 2024-09-10T19:04:44.461 INFO:teuthology.orchestra.run.smithi183.stdout:Base mon IP(s) is [172.21.15.183:3300, 172.21.15.183:6789], mon addrv is [v2:172.21.15.183:3300,v1:172.21.15.183:6789] 2024-09-10T19:04:44.466 INFO:teuthology.orchestra.run.smithi183.stdout:/sbin/ip: stdout default via 172.21.15.254 dev enp3s0f1 proto dhcp src 172.21.15.183 metric 100 2024-09-10T19:04:44.466 INFO:teuthology.orchestra.run.smithi183.stdout:/sbin/ip: stdout 172.21.0.0/20 dev enp3s0f1 proto kernel scope link src 172.21.15.183 metric 100 2024-09-10T19:04:44.470 INFO:teuthology.orchestra.run.smithi183.stdout:/sbin/ip: stdout ::1 dev lo proto kernel metric 256 pref medium 2024-09-10T19:04:44.470 INFO:teuthology.orchestra.run.smithi183.stdout:/sbin/ip: stdout fe80::/64 dev enp3s0f1 proto kernel metric 1024 pref medium 2024-09-10T19:04:44.470 INFO:teuthology.orchestra.run.smithi183.stdout:/sbin/ip: stdout default via fe80::327c:5e00:6487:71e0 dev enp3s0f1 proto ra metric 100 pref medium 2024-09-10T19:04:44.473 INFO:teuthology.orchestra.run.smithi183.stdout:/sbin/ip: stdout 1: lo: mtu 65536 state UNKNOWN qlen 1000 2024-09-10T19:04:44.474 INFO:teuthology.orchestra.run.smithi183.stdout:/sbin/ip: stdout inet6 ::1/128 scope host 2024-09-10T19:04:44.474 INFO:teuthology.orchestra.run.smithi183.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-09-10T19:04:44.474 INFO:teuthology.orchestra.run.smithi183.stdout:/sbin/ip: stdout 5: enp3s0f1: mtu 1500 state UP qlen 1000 2024-09-10T19:04:44.474 INFO:teuthology.orchestra.run.smithi183.stdout:/sbin/ip: stdout inet6 fe80::ec4:7aff:fe88:8085/64 scope link noprefixroute 2024-09-10T19:04:44.474 INFO:teuthology.orchestra.run.smithi183.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-09-10T19:04:44.475 INFO:teuthology.orchestra.run.smithi183.stdout:Mon IP `172.21.15.183` is in CIDR network `172.21.0.0/20` 2024-09-10T19:04:44.475 INFO:teuthology.orchestra.run.smithi183.stdout:Mon IP `172.21.15.183` is in CIDR network `172.21.0.0/20` 2024-09-10T19:04:44.475 INFO:teuthology.orchestra.run.smithi183.stdout:Inferred mon public CIDR from local network configuration ['172.21.0.0/20', '172.21.0.0/20'] 2024-09-10T19:04:44.476 INFO:teuthology.orchestra.run.smithi183.stdout:Internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2024-09-10T19:04:44.477 INFO:teuthology.orchestra.run.smithi183.stdout:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588... 2024-09-10T19:04:44.892 INFO:teuthology.orchestra.run.smithi183.stdout:/bin/podman: stdout 4c55dd4096115e48502f368cff1eab06769f71f030e1692e2611bbc6c6f637e9 2024-09-10T19:04:44.892 INFO:teuthology.orchestra.run.smithi183.stdout:/bin/podman: stderr Trying to pull quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588... 2024-09-10T19:04:44.892 INFO:teuthology.orchestra.run.smithi183.stdout:/bin/podman: stderr Getting image source signatures 2024-09-10T19:04:44.892 INFO:teuthology.orchestra.run.smithi183.stdout:/bin/podman: stderr Copying blob sha256:5df13064c03763eb21a89b455ab82ba4ec435f51a3f340cae76514aaf2996d4c 2024-09-10T19:04:44.892 INFO:teuthology.orchestra.run.smithi183.stdout:/bin/podman: stderr Copying blob sha256:e8b54c863393b7a8216a71737771b73b16a8e43ec7acf927c7faa175c255e551 2024-09-10T19:04:44.892 INFO:teuthology.orchestra.run.smithi183.stdout:/bin/podman: stderr Copying config sha256:4c55dd4096115e48502f368cff1eab06769f71f030e1692e2611bbc6c6f637e9 2024-09-10T19:04:44.892 INFO:teuthology.orchestra.run.smithi183.stdout:/bin/podman: stderr Writing manifest to image destination 2024-09-10T19:04:45.921 INFO:teuthology.orchestra.run.smithi183.stdout:ceph: stdout ceph version 17.2.7-1606-g5be57395 (5be57395ea235535910512ae73ba8e7ff25fe588) quincy (stable) 2024-09-10T19:04:45.922 INFO:teuthology.orchestra.run.smithi183.stdout:Ceph version: ceph version 17.2.7-1606-g5be57395 (5be57395ea235535910512ae73ba8e7ff25fe588) quincy (stable) 2024-09-10T19:04:45.922 INFO:teuthology.orchestra.run.smithi183.stdout:Extracting ceph user uid/gid from container image... 2024-09-10T19:04:46.859 INFO:teuthology.orchestra.run.smithi183.stdout:stat: stdout 167 167 2024-09-10T19:04:46.859 INFO:teuthology.orchestra.run.smithi183.stdout:Creating initial keys... 2024-09-10T19:04:47.823 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph-authtool: stdout AQBPmOBm55TeDRAAeqXYy1qhUY1OlK3cp+F8bg== 2024-09-10T19:04:54.517 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph-authtool: stdout AQBQmOBmwCIvCRAAWG8jDEyWd/j0G0Qp+nFK4Q== 2024-09-10T19:05:00.215 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph-authtool: stdout AQBZmOBm9o3TERAAY2oqK/9uIoXekDYP3693VA== 2024-09-10T19:05:00.216 INFO:teuthology.orchestra.run.smithi183.stdout:Creating initial monmap... 2024-09-10T19:05:01.179 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: monmap file /tmp/monmap 2024-09-10T19:05:01.179 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/monmaptool: stdout setting min_mon_release = octopus 2024-09-10T19:05:01.179 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: set fsid to 320f6df8-6fa7-11ef-bcea-c7b262605968 2024-09-10T19:05:01.179 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-09-10T19:05:01.179 INFO:teuthology.orchestra.run.smithi183.stdout:monmaptool for a [v2:172.21.15.183:3300,v1:172.21.15.183:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2024-09-10T19:05:01.179 INFO:teuthology.orchestra.run.smithi183.stdout:setting min_mon_release = octopus 2024-09-10T19:05:01.180 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/monmaptool: set fsid to 320f6df8-6fa7-11ef-bcea-c7b262605968 2024-09-10T19:05:01.180 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-09-10T19:05:01.180 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-09-10T19:05:01.180 INFO:teuthology.orchestra.run.smithi183.stdout:Creating mon... 2024-09-10T19:05:02.434 INFO:teuthology.orchestra.run.smithi183.stdout:create mon.a on 2024-09-10T19:05:02.636 INFO:teuthology.orchestra.run.smithi183.stdout:systemctl: stderr Removed "/etc/systemd/system/multi-user.target.wants/ceph.target". 2024-09-10T19:05:02.779 INFO:teuthology.orchestra.run.smithi183.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2024-09-10T19:05:02.928 INFO:teuthology.orchestra.run.smithi183.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph-320f6df8-6fa7-11ef-bcea-c7b262605968.target → /etc/systemd/system/ceph-320f6df8-6fa7-11ef-bcea-c7b262605968.target. 2024-09-10T19:05:02.928 INFO:teuthology.orchestra.run.smithi183.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph.target.wants/ceph-320f6df8-6fa7-11ef-bcea-c7b262605968.target → /etc/systemd/system/ceph-320f6df8-6fa7-11ef-bcea-c7b262605968.target. 2024-09-10T19:05:03.097 INFO:teuthology.orchestra.run.smithi183.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-320f6df8-6fa7-11ef-bcea-c7b262605968@mon.a 2024-09-10T19:05:03.097 INFO:teuthology.orchestra.run.smithi183.stdout:systemctl: stderr Failed to reset failed state of unit ceph-320f6df8-6fa7-11ef-bcea-c7b262605968@mon.a.service: Unit ceph-320f6df8-6fa7-11ef-bcea-c7b262605968@mon.a.service not loaded. 2024-09-10T19:05:03.259 INFO:teuthology.orchestra.run.smithi183.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-320f6df8-6fa7-11ef-bcea-c7b262605968.target.wants/ceph-320f6df8-6fa7-11ef-bcea-c7b262605968@mon.a.service → /etc/systemd/system/ceph-320f6df8-6fa7-11ef-bcea-c7b262605968@.service. 2024-09-10T19:05:03.752 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:05:03 smithi183 ceph-mon[29269]: mon.a is new leader, mons a in quorum (ranks 0) 2024-09-10T19:05:03.765 INFO:teuthology.orchestra.run.smithi183.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-09-10T19:05:03.765 INFO:teuthology.orchestra.run.smithi183.stdout:systemctl: stdout disabled 2024-09-10T19:05:03.772 INFO:teuthology.orchestra.run.smithi183.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-09-10T19:05:03.772 INFO:teuthology.orchestra.run.smithi183.stdout:systemctl: stdout inactive 2024-09-10T19:05:03.772 INFO:teuthology.orchestra.run.smithi183.stdout:firewalld.service is not enabled 2024-09-10T19:05:03.772 INFO:teuthology.orchestra.run.smithi183.stdout:Not possible to enable service . firewalld.service is not available 2024-09-10T19:05:03.772 INFO:teuthology.orchestra.run.smithi183.stdout:Waiting for mon to start... 2024-09-10T19:05:03.773 INFO:teuthology.orchestra.run.smithi183.stdout:Waiting for mon... 2024-09-10T19:05:04.013 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:05:03 smithi183 bash[29231]: a2a4e8d7d71d1482851f2eeaaae64993971eda56fcf061ebad699fad4c2310b3 2024-09-10T19:05:04.013 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:05:03 smithi183 systemd[1]: Started Ceph mon.a for 320f6df8-6fa7-11ef-bcea-c7b262605968. 2024-09-10T19:05:04.770 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout cluster: 2024-09-10T19:05:04.770 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout id: 320f6df8-6fa7-11ef-bcea-c7b262605968 2024-09-10T19:05:04.770 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout health: HEALTH_OK 2024-09-10T19:05:04.770 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout 2024-09-10T19:05:04.770 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout services: 2024-09-10T19:05:04.770 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout mon: 1 daemons, quorum a (age 0.564506s) 2024-09-10T19:05:04.770 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout mgr: no daemons active 2024-09-10T19:05:04.771 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout osd: 0 osds: 0 up, 0 in 2024-09-10T19:05:04.771 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout 2024-09-10T19:05:04.771 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout data: 2024-09-10T19:05:04.771 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout pools: 0 pools, 0 pgs 2024-09-10T19:05:04.771 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout objects: 0 objects, 0 B 2024-09-10T19:05:04.771 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout usage: 0 B used, 0 B / 0 B avail 2024-09-10T19:05:04.771 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout pgs: 2024-09-10T19:05:04.771 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout 2024-09-10T19:05:04.771 INFO:teuthology.orchestra.run.smithi183.stdout:mon is available 2024-09-10T19:05:04.771 INFO:teuthology.orchestra.run.smithi183.stdout:Assimilating anything we can from ceph.conf... 2024-09-10T19:05:04.884 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:05:04 smithi183 ceph-mon[29269]: mon.a is new leader, mons a in quorum (ranks 0) 2024-09-10T19:05:04.884 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:05:04 smithi183 ceph-mon[29269]: monmap e1: 1 mons at {a=[v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-09-10T19:05:04.884 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:05:04 smithi183 ceph-mon[29269]: fsmap 2024-09-10T19:05:04.884 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:05:04 smithi183 ceph-mon[29269]: osdmap e1: 0 total, 0 up, 0 in 2024-09-10T19:05:04.884 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:05:04 smithi183 ceph-mon[29269]: mgrmap e1: no daemons active 2024-09-10T19:05:04.884 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:05:04 smithi183 ceph-mon[29269]: from='client.? 172.21.15.183:0/415305886' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-09-10T19:05:05.931 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:05:05 smithi183 ceph-mon[29269]: from='client.? 172.21.15.183:0/4089025000' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-09-10T19:05:05.932 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:05:05 smithi183 ceph-mon[29269]: from='client.? 172.21.15.183:0/4089025000' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-09-10T19:05:06.849 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout 2024-09-10T19:05:06.850 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout [global] 2024-09-10T19:05:06.850 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout fsid = 320f6df8-6fa7-11ef-bcea-c7b262605968 2024-09-10T19:05:06.850 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout mon_cluster_log_file_level = debug 2024-09-10T19:05:06.850 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout mon_host = [v2:172.21.15.183:3300,v1:172.21.15.183:6789] 2024-09-10T19:05:06.850 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-09-10T19:05:06.850 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-09-10T19:05:06.850 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-09-10T19:05:06.850 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-09-10T19:05:06.850 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout 2024-09-10T19:05:06.850 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout [mgr] 2024-09-10T19:05:06.850 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-09-10T19:05:06.850 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout 2024-09-10T19:05:06.850 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout [osd] 2024-09-10T19:05:06.851 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-09-10T19:05:06.851 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-09-10T19:05:06.851 INFO:teuthology.orchestra.run.smithi183.stdout:Generating new minimal ceph.conf... 2024-09-10T19:05:07.681 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:05:07 smithi183 ceph-mon[29269]: from='client.? 172.21.15.183:0/2225438421' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T19:05:08.003 INFO:teuthology.orchestra.run.smithi183.stdout:Restarting the monitor... 2024-09-10T19:05:08.256 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:05:08 smithi183 systemd[1]: Stopping Ceph mon.a for 320f6df8-6fa7-11ef-bcea-c7b262605968... 2024-09-10T19:05:08.587 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:05:08 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mon-a[29246]: 2024-09-10T19:05:08.253+0000 7f9fe90db640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-09-10T19:05:08.587 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:05:08 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mon-a[29246]: 2024-09-10T19:05:08.253+0000 7f9fe90db640 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2024-09-10T19:05:08.587 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:05:08 smithi183 podman[29587]: 2024-09-10 19:05:08.346610776 +0000 UTC m=+0.186990186 container died a2a4e8d7d71d1482851f2eeaaae64993971eda56fcf061ebad699fad4c2310b3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588, name=ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mon-a, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, ceph=True, RELEASE=wip-guits-quincy-2024-09-09-1810-quincy-5be5739, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_CLEAN=True, org.label-schema.schema-version=1.0) 2024-09-10T19:05:08.931 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:05:08 smithi183 podman[29587]: 2024-09-10 19:05:08.694321032 +0000 UTC m=+0.534700439 container cleanup a2a4e8d7d71d1482851f2eeaaae64993971eda56fcf061ebad699fad4c2310b3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588, name=ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mon-a, GIT_CLEAN=True, org.label-schema.vendor=CentOS, RELEASE=wip-guits-quincy-2024-09-09-1810-quincy-5be5739, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, ceph=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-10T19:05:08.931 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:05:08 smithi183 bash[29587]: ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mon-a 2024-09-10T19:05:09.297 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:05:08 smithi183 podman[29603]: 2024-09-10 19:05:08.95347548 +0000 UTC m=+0.610541251 container remove a2a4e8d7d71d1482851f2eeaaae64993971eda56fcf061ebad699fad4c2310b3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588, name=ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mon-a, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, RELEASE=wip-guits-quincy-2024-09-09-1810-quincy-5be5739, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD) 2024-09-10T19:05:09.297 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:05:09 smithi183 systemd[1]: ceph-320f6df8-6fa7-11ef-bcea-c7b262605968@mon.a.service: Deactivated successfully. 2024-09-10T19:05:09.297 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:05:09 smithi183 systemd[1]: Stopped Ceph mon.a for 320f6df8-6fa7-11ef-bcea-c7b262605968. 2024-09-10T19:05:09.297 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:05:09 smithi183 systemd[1]: Starting Ceph mon.a for 320f6df8-6fa7-11ef-bcea-c7b262605968... 2024-09-10T19:05:09.564 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:05:09 smithi183 podman[29698]: 2024-09-10 19:05:09.296133824 +0000 UTC m=+0.096104157 container create fae323eed8e937f6d021ee78cc6ea5e4e5e25230228ac9d0174430e87674bb76 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588, name=ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mon-a, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , RELEASE=wip-guits-quincy-2024-09-09-1810-quincy-5be5739, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2) 2024-09-10T19:05:09.564 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:05:09 smithi183 podman[29698]: 2024-09-10 19:05:09.215771638 +0000 UTC m=+0.015741928 image pull 4c55dd4096115e48502f368cff1eab06769f71f030e1692e2611bbc6c6f637e9 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 2024-09-10T19:05:09.564 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:05:09 smithi183 podman[29698]: 2024-09-10 19:05:09.540023997 +0000 UTC m=+0.339994290 container init fae323eed8e937f6d021ee78cc6ea5e4e5e25230228ac9d0174430e87674bb76 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588, name=ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mon-a, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , RELEASE=wip-guits-quincy-2024-09-09-1810-quincy-5be5739, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731) 2024-09-10T19:05:09.564 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:05:09 smithi183 podman[29698]: 2024-09-10 19:05:09.544949985 +0000 UTC m=+0.344920276 container start fae323eed8e937f6d021ee78cc6ea5e4e5e25230228ac9d0174430e87674bb76 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588, name=ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mon-a, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, GIT_CLEAN=True, RELEASE=wip-guits-quincy-2024-09-09-1810-quincy-5be5739, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=) 2024-09-10T19:05:09.670 INFO:teuthology.orchestra.run.smithi183.stdout:Setting public_network to 172.21.0.0/20 in mon config section 2024-09-10T19:05:09.891 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:05:09 smithi183 ceph-mon[29736]: set uid:gid to 167:167 (ceph:ceph) 2024-09-10T19:05:09.891 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:05:09 smithi183 ceph-mon[29736]: ceph version 17.2.7-1606-g5be57395 (5be57395ea235535910512ae73ba8e7ff25fe588) quincy (stable), process ceph-mon, pid 7 2024-09-10T19:05:09.891 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:05:09 smithi183 ceph-mon[29736]: pidfile_write: ignore empty --pid-file 2024-09-10T19:05:09.891 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:05:09 smithi183 ceph-mon[29736]: load: jerasure load: lrc 2024-09-10T19:05:09.891 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:05:09 smithi183 ceph-mon[29736]: rocksdb: RocksDB version: 6.15.5 2024-09-10T19:05:09.891 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:05:09 smithi183 ceph-mon[29736]: rocksdb: Git sha rocksdb_build_git_sha:@0@ 2024-09-10T19:05:09.892 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:05:09 smithi183 ceph-mon[29736]: rocksdb: Compile date Sep 9 2024 2024-09-10T19:05:09.892 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:05:09 smithi183 ceph-mon[29736]: rocksdb: DB SUMMARY 2024-09-10T19:05:09.892 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:05:09 smithi183 ceph-mon[29736]: rocksdb: DB Session ID: U0Q5F17B8FHO0OY4JC2E 2024-09-10T19:05:09.892 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:05:09 smithi183 ceph-mon[29736]: rocksdb: CURRENT file: CURRENT 2024-09-10T19:05:09.892 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:05:09 smithi183 ceph-mon[29736]: rocksdb: IDENTITY file: IDENTITY 2024-09-10T19:05:09.892 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:05:09 smithi183 ceph-mon[29736]: rocksdb: MANIFEST file: MANIFEST-000009 size: 131 Bytes 2024-09-10T19:05:09.892 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:05:09 smithi183 ceph-mon[29736]: rocksdb: SST files in /var/lib/ceph/mon/ceph-a/store.db dir, Total Num: 1, files: 000008.sst 2024-09-10T19:05:09.892 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:05:09 smithi183 ceph-mon[29736]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-a/store.db: 000010.log size: 89074 ; 2024-09-10T19:05:09.892 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:05:09 smithi183 ceph-mon[29736]: rocksdb: Options.error_if_exists: 0 2024-09-10T19:05:09.892 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:05:09 smithi183 ceph-mon[29736]: rocksdb: Options.create_if_missing: 0 2024-09-10T19:05:09.892 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:05:09 smithi183 ceph-mon[29736]: rocksdb: Options.paranoid_checks: 1 2024-09-10T19:05:09.892 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:05:09 smithi183 ceph-mon[29736]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-09-10T19:05:09.893 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:05:09 smithi183 ceph-mon[29736]: rocksdb: Options.env: 0x55f36e1b2440 2024-09-10T19:05:09.893 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:05:09 smithi183 ceph-mon[29736]: rocksdb: Options.fs: Posix File System 2024-09-10T19:05:09.893 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:05:09 smithi183 ceph-mon[29736]: rocksdb: Options.info_log: 0x55f370376ec0 2024-09-10T19:05:09.893 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:05:09 smithi183 ceph-mon[29736]: rocksdb: Options.max_file_opening_threads: 16 2024-09-10T19:05:09.893 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:05:09 smithi183 ceph-mon[29736]: rocksdb: Options.statistics: (nil) 2024-09-10T19:05:09.893 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:05:09 smithi183 ceph-mon[29736]: rocksdb: Options.use_fsync: 0 2024-09-10T19:05:09.893 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:05:09 smithi183 ceph-mon[29736]: rocksdb: Options.max_log_file_size: 0 2024-09-10T19:05:09.893 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:05:09 smithi183 ceph-mon[29736]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-09-10T19:05:09.893 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:05:09 smithi183 ceph-mon[29736]: rocksdb: Options.log_file_time_to_roll: 0 2024-09-10T19:05:09.893 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:05:09 smithi183 ceph-mon[29736]: rocksdb: Options.keep_log_file_num: 1000 2024-09-10T19:05:09.893 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:05:09 smithi183 ceph-mon[29736]: rocksdb: Options.recycle_log_file_num: 0 2024-09-10T19:05:09.893 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:05:09 smithi183 ceph-mon[29736]: rocksdb: Options.allow_fallocate: 1 2024-09-10T19:05:09.893 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:05:09 smithi183 ceph-mon[29736]: rocksdb: Options.allow_mmap_reads: 0 2024-09-10T19:05:09.893 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:05:09 smithi183 ceph-mon[29736]: rocksdb: Options.allow_mmap_writes: 0 2024-09-10T19:05:09.894 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:05:09 smithi183 ceph-mon[29736]: rocksdb: Options.use_direct_reads: 0 2024-09-10T19:05:09.894 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:05:09 smithi183 ceph-mon[29736]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-09-10T19:05:09.894 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:05:09 smithi183 ceph-mon[29736]: rocksdb: Options.create_missing_column_families: 0 2024-09-10T19:05:09.894 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:05:09 smithi183 ceph-mon[29736]: rocksdb: Options.db_log_dir: 2024-09-10T19:05:09.894 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:05:09 smithi183 ceph-mon[29736]: rocksdb: Options.wal_dir: /var/lib/ceph/mon/ceph-a/store.db 2024-09-10T19:05:09.894 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:05:09 smithi183 ceph-mon[29736]: rocksdb: Options.table_cache_numshardbits: 6 2024-09-10T19:05:09.894 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:05:09 smithi183 ceph-mon[29736]: rocksdb: Options.WAL_ttl_seconds: 0 2024-09-10T19:05:09.895 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:05:09 smithi183 ceph-mon[29736]: rocksdb: Options.WAL_size_limit_MB: 0 2024-09-10T19:05:09.896 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:05:09 smithi183 ceph-mon[29736]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-09-10T19:05:09.896 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:05:09 smithi183 ceph-mon[29736]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-09-10T19:05:09.896 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:05:09 smithi183 ceph-mon[29736]: rocksdb: Options.is_fd_close_on_exec: 1 2024-09-10T19:05:09.896 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:05:09 smithi183 ceph-mon[29736]: rocksdb: Options.advise_random_on_open: 1 2024-09-10T19:05:09.896 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:05:09 smithi183 ceph-mon[29736]: rocksdb: Options.db_write_buffer_size: 0 2024-09-10T19:05:09.896 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:05:09 smithi183 ceph-mon[29736]: rocksdb: Options.write_buffer_manager: 0x55f370434780 2024-09-10T19:05:09.896 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:05:09 smithi183 ceph-mon[29736]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-09-10T19:05:09.896 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:05:09 smithi183 ceph-mon[29736]: rocksdb: Options.new_table_reader_for_compaction_inputs: 0 2024-09-10T19:05:09.896 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:05:09 smithi183 ceph-mon[29736]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-09-10T19:05:09.896 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:05:09 smithi183 ceph-mon[29736]: rocksdb: Options.use_adaptive_mutex: 0 2024-09-10T19:05:09.896 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:05:09 smithi183 ceph-mon[29736]: rocksdb: Options.rate_limiter: (nil) 2024-09-10T19:05:09.896 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:05:09 smithi183 ceph-mon[29736]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-09-10T19:05:09.896 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:05:09 smithi183 ceph-mon[29736]: rocksdb: Options.wal_recovery_mode: 2 2024-09-10T19:05:09.897 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:05:09 smithi183 ceph-mon[29736]: rocksdb: Options.enable_thread_tracking: 0 2024-09-10T19:05:09.897 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:05:09 smithi183 ceph-mon[29736]: rocksdb: Options.enable_pipelined_write: 0 2024-09-10T19:05:09.897 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:05:09 smithi183 ceph-mon[29736]: rocksdb: Options.unordered_write: 0 2024-09-10T19:05:09.897 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:05:09 smithi183 ceph-mon[29736]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-09-10T19:05:09.897 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:05:09 smithi183 ceph-mon[29736]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-09-10T19:05:09.897 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:05:09 smithi183 ceph-mon[29736]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-09-10T19:05:09.897 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:05:09 smithi183 ceph-mon[29736]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-09-10T19:05:09.897 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:05:09 smithi183 ceph-mon[29736]: rocksdb: Options.row_cache: None 2024-09-10T19:05:09.897 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:05:09 smithi183 ceph-mon[29736]: rocksdb: Options.wal_filter: None 2024-09-10T19:05:09.897 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:05:09 smithi183 ceph-mon[29736]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-09-10T19:05:09.897 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:05:09 smithi183 ceph-mon[29736]: rocksdb: Options.allow_ingest_behind: 0 2024-09-10T19:05:09.897 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:05:09 smithi183 ceph-mon[29736]: rocksdb: Options.preserve_deletes: 0 2024-09-10T19:05:09.897 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:05:09 smithi183 ceph-mon[29736]: rocksdb: Options.two_write_queues: 0 2024-09-10T19:05:09.898 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:05:09 smithi183 ceph-mon[29736]: rocksdb: Options.manual_wal_flush: 0 2024-09-10T19:05:09.898 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:05:09 smithi183 ceph-mon[29736]: rocksdb: Options.atomic_flush: 0 2024-09-10T19:05:09.898 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:05:09 smithi183 ceph-mon[29736]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-09-10T19:05:09.898 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:05:09 smithi183 ceph-mon[29736]: rocksdb: Options.persist_stats_to_disk: 0 2024-09-10T19:05:09.898 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:05:09 smithi183 ceph-mon[29736]: rocksdb: Options.write_dbid_to_manifest: 0 2024-09-10T19:05:09.898 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:05:09 smithi183 ceph-mon[29736]: rocksdb: Options.log_readahead_size: 0 2024-09-10T19:05:09.898 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:05:09 smithi183 ceph-mon[29736]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-09-10T19:05:09.898 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:05:09 smithi183 ceph-mon[29736]: rocksdb: Options.best_efforts_recovery: 0 2024-09-10T19:05:09.898 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:05:09 smithi183 ceph-mon[29736]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-09-10T19:05:09.898 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:05:09 smithi183 ceph-mon[29736]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-09-10T19:05:09.898 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:05:09 smithi183 ceph-mon[29736]: rocksdb: Options.allow_data_in_errors: 0 2024-09-10T19:05:09.898 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:05:09 smithi183 ceph-mon[29736]: rocksdb: Options.db_host_id: __hostname__ 2024-09-10T19:05:09.898 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:05:09 smithi183 ceph-mon[29736]: rocksdb: Options.max_background_jobs: 2 2024-09-10T19:05:09.899 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:05:09 smithi183 ceph-mon[29736]: rocksdb: Options.max_background_compactions: -1 2024-09-10T19:05:09.899 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:05:09 smithi183 ceph-mon[29736]: rocksdb: Options.max_subcompactions: 1 2024-09-10T19:05:09.899 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:05:09 smithi183 ceph-mon[29736]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-09-10T19:05:09.899 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:05:09 smithi183 ceph-mon[29736]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-09-10T19:05:09.899 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:05:09 smithi183 ceph-mon[29736]: rocksdb: Options.delayed_write_rate : 16777216 2024-09-10T19:05:09.899 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:05:09 smithi183 ceph-mon[29736]: rocksdb: Options.max_total_wal_size: 0 2024-09-10T19:05:09.899 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:05:09 smithi183 ceph-mon[29736]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-09-10T19:05:09.899 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:05:09 smithi183 ceph-mon[29736]: rocksdb: Options.stats_dump_period_sec: 600 2024-09-10T19:05:09.899 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:05:09 smithi183 ceph-mon[29736]: rocksdb: Options.stats_persist_period_sec: 600 2024-09-10T19:05:09.899 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:05:09 smithi183 ceph-mon[29736]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-09-10T19:05:09.899 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:05:09 smithi183 ceph-mon[29736]: rocksdb: Options.max_open_files: -1 2024-09-10T19:05:09.899 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:05:09 smithi183 ceph-mon[29736]: rocksdb: Options.bytes_per_sync: 0 2024-09-10T19:05:09.899 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:05:09 smithi183 ceph-mon[29736]: rocksdb: Options.wal_bytes_per_sync: 0 2024-09-10T19:05:09.900 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:05:09 smithi183 ceph-mon[29736]: rocksdb: Options.strict_bytes_per_sync: 0 2024-09-10T19:05:09.900 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:05:09 smithi183 ceph-mon[29736]: rocksdb: Options.compaction_readahead_size: 0 2024-09-10T19:05:09.900 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:05:09 smithi183 ceph-mon[29736]: rocksdb: Options.max_background_flushes: -1 2024-09-10T19:05:09.900 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:05:09 smithi183 ceph-mon[29736]: rocksdb: Compression algorithms supported: 2024-09-10T19:05:09.900 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:05:09 smithi183 ceph-mon[29736]: rocksdb: kZSTD supported: 0 2024-09-10T19:05:09.900 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:05:09 smithi183 ceph-mon[29736]: rocksdb: kXpressCompression supported: 0 2024-09-10T19:05:09.900 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:05:09 smithi183 ceph-mon[29736]: rocksdb: kBZip2Compression supported: 0 2024-09-10T19:05:09.900 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:05:09 smithi183 ceph-mon[29736]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-09-10T19:05:09.900 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:05:09 smithi183 ceph-mon[29736]: rocksdb: kLZ4Compression supported: 1 2024-09-10T19:05:09.900 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:05:09 smithi183 ceph-mon[29736]: rocksdb: kZlibCompression supported: 1 2024-09-10T19:05:09.900 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:05:09 smithi183 ceph-mon[29736]: rocksdb: kLZ4HCCompression supported: 1 2024-09-10T19:05:09.900 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:05:09 smithi183 ceph-mon[29736]: rocksdb: kSnappyCompression supported: 1 2024-09-10T19:05:09.900 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:05:09 smithi183 ceph-mon[29736]: rocksdb: Fast CRC32 supported: Supported on x86 2024-09-10T19:05:09.900 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:05:09 smithi183 ceph-mon[29736]: rocksdb: [db/version_set.cc:4724] Recovering from manifest file: /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000009 2024-09-10T19:05:09.901 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:05:09 smithi183 ceph-mon[29736]: rocksdb: [db/column_family.cc:595] --------------- Options for column family [default]: 2024-09-10T19:05:09.901 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:05:09 smithi183 ceph-mon[29736]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-09-10T19:05:09.901 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:05:09 smithi183 ceph-mon[29736]: rocksdb: Options.merge_operator: 2024-09-10T19:05:09.901 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:05:09 smithi183 ceph-mon[29736]: rocksdb: Options.compaction_filter: None 2024-09-10T19:05:09.901 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:05:09 smithi183 ceph-mon[29736]: rocksdb: Options.compaction_filter_factory: None 2024-09-10T19:05:09.901 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:05:09 smithi183 ceph-mon[29736]: rocksdb: Options.sst_partitioner_factory: None 2024-09-10T19:05:09.901 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:05:09 smithi183 ceph-mon[29736]: rocksdb: Options.memtable_factory: SkipListFactory 2024-09-10T19:05:09.901 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:05:09 smithi183 ceph-mon[29736]: rocksdb: Options.table_factory: BlockBasedTable 2024-09-10T19:05:09.901 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:05:09 smithi183 ceph-mon[29736]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55f37031da38) 2024-09-10T19:05:09.901 INFO:journalctl@ceph.mon.a.smithi183.stdout: cache_index_and_filter_blocks: 1 2024-09-10T19:05:09.901 INFO:journalctl@ceph.mon.a.smithi183.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-09-10T19:05:09.901 INFO:journalctl@ceph.mon.a.smithi183.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-09-10T19:05:09.901 INFO:journalctl@ceph.mon.a.smithi183.stdout: pin_top_level_index_and_filter: 1 2024-09-10T19:05:09.901 INFO:journalctl@ceph.mon.a.smithi183.stdout: index_type: 0 2024-09-10T19:05:09.902 INFO:journalctl@ceph.mon.a.smithi183.stdout: data_block_index_type: 0 2024-09-10T19:05:09.902 INFO:journalctl@ceph.mon.a.smithi183.stdout: index_shortening: 1 2024-09-10T19:05:09.902 INFO:journalctl@ceph.mon.a.smithi183.stdout: data_block_hash_table_util_ratio: 0.750000 2024-09-10T19:05:09.902 INFO:journalctl@ceph.mon.a.smithi183.stdout: hash_index_allow_collision: 1 2024-09-10T19:05:09.902 INFO:journalctl@ceph.mon.a.smithi183.stdout: checksum: 1 2024-09-10T19:05:09.902 INFO:journalctl@ceph.mon.a.smithi183.stdout: no_block_cache: 0 2024-09-10T19:05:09.902 INFO:journalctl@ceph.mon.a.smithi183.stdout: block_cache: 0x55f3703531f0 2024-09-10T19:05:09.902 INFO:journalctl@ceph.mon.a.smithi183.stdout: block_cache_name: BinnedLRUCache 2024-09-10T19:05:09.902 INFO:journalctl@ceph.mon.a.smithi183.stdout: block_cache_options: 2024-09-10T19:05:09.902 INFO:journalctl@ceph.mon.a.smithi183.stdout: capacity : 536870912 2024-09-10T19:05:09.902 INFO:journalctl@ceph.mon.a.smithi183.stdout: num_shard_bits : 4 2024-09-10T19:05:09.902 INFO:journalctl@ceph.mon.a.smithi183.stdout: strict_capacity_limit : 0 2024-09-10T19:05:09.902 INFO:journalctl@ceph.mon.a.smithi183.stdout: high_pri_pool_ratio: 0.000 2024-09-10T19:05:09.902 INFO:journalctl@ceph.mon.a.smithi183.stdout: block_cache_compressed: (nil) 2024-09-10T19:05:09.903 INFO:journalctl@ceph.mon.a.smithi183.stdout: persistent_cache: (nil) 2024-09-10T19:05:09.903 INFO:journalctl@ceph.mon.a.smithi183.stdout: block_size: 4096 2024-09-10T19:05:09.903 INFO:journalctl@ceph.mon.a.smithi183.stdout: block_size_deviation: 10 2024-09-10T19:05:09.903 INFO:journalctl@ceph.mon.a.smithi183.stdout: block_restart_interval: 16 2024-09-10T19:05:09.903 INFO:journalctl@ceph.mon.a.smithi183.stdout: index_block_restart_interval: 1 2024-09-10T19:05:09.903 INFO:journalctl@ceph.mon.a.smithi183.stdout: metadata_block_size: 4096 2024-09-10T19:05:09.903 INFO:journalctl@ceph.mon.a.smithi183.stdout: partition_filters: 0 2024-09-10T19:05:09.903 INFO:journalctl@ceph.mon.a.smithi183.stdout: use_delta_encoding: 1 2024-09-10T19:05:09.903 INFO:journalctl@ceph.mon.a.smithi183.stdout: filter_policy: rocksdb.BuiltinBloomFilter 2024-09-10T19:05:09.903 INFO:journalctl@ceph.mon.a.smithi183.stdout: whole_key_filtering: 1 2024-09-10T19:05:09.903 INFO:journalctl@ceph.mon.a.smithi183.stdout: verify_compression: 0 2024-09-10T19:05:09.903 INFO:journalctl@ceph.mon.a.smithi183.stdout: read_amp_bytes_per_bit: 0 2024-09-10T19:05:09.903 INFO:journalctl@ceph.mon.a.smithi183.stdout: format_version: 4 2024-09-10T19:05:09.903 INFO:journalctl@ceph.mon.a.smithi183.stdout: enable_index_compression: 1 2024-09-10T19:05:09.904 INFO:journalctl@ceph.mon.a.smithi183.stdout: block_align: 0 2024-09-10T19:05:09.904 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:05:09 smithi183 ceph-mon[29736]: rocksdb: Options.write_buffer_size: 33554432 2024-09-10T19:05:09.904 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:05:09 smithi183 ceph-mon[29736]: rocksdb: Options.max_write_buffer_number: 2 2024-09-10T19:05:09.904 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:05:09 smithi183 ceph-mon[29736]: rocksdb: Options.compression: NoCompression 2024-09-10T19:05:09.904 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:05:09 smithi183 ceph-mon[29736]: rocksdb: Options.bottommost_compression: Disabled 2024-09-10T19:05:09.904 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:05:09 smithi183 ceph-mon[29736]: rocksdb: Options.prefix_extractor: nullptr 2024-09-10T19:05:09.904 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:05:09 smithi183 ceph-mon[29736]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-09-10T19:05:09.904 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:05:09 smithi183 ceph-mon[29736]: rocksdb: Options.num_levels: 7 2024-09-10T19:05:09.904 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:05:09 smithi183 ceph-mon[29736]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-09-10T19:05:09.904 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:05:09 smithi183 ceph-mon[29736]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-09-10T19:05:09.904 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:05:09 smithi183 ceph-mon[29736]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-09-10T19:05:09.904 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:05:09 smithi183 ceph-mon[29736]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-09-10T19:05:09.904 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:05:09 smithi183 ceph-mon[29736]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-09-10T19:05:09.904 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:05:09 smithi183 ceph-mon[29736]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-09-10T19:05:09.905 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:05:09 smithi183 ceph-mon[29736]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-09-10T19:05:09.905 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:05:09 smithi183 ceph-mon[29736]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-09-10T19:05:09.905 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:05:09 smithi183 ceph-mon[29736]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-09-10T19:05:09.905 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:05:09 smithi183 ceph-mon[29736]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-09-10T19:05:09.905 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:05:09 smithi183 ceph-mon[29736]: rocksdb: Options.compression_opts.window_bits: -14 2024-09-10T19:05:09.905 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:05:09 smithi183 ceph-mon[29736]: rocksdb: Options.compression_opts.level: 32767 2024-09-10T19:05:09.905 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:05:09 smithi183 ceph-mon[29736]: rocksdb: Options.compression_opts.strategy: 0 2024-09-10T19:05:09.905 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:05:09 smithi183 ceph-mon[29736]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-09-10T19:05:09.905 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:05:09 smithi183 ceph-mon[29736]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-09-10T19:05:09.905 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:05:09 smithi183 ceph-mon[29736]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-09-10T19:05:09.905 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:05:09 smithi183 ceph-mon[29736]: rocksdb: Options.compression_opts.enabled: false 2024-09-10T19:05:09.905 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:05:09 smithi183 ceph-mon[29736]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-09-10T19:05:09.905 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:05:09 smithi183 ceph-mon[29736]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-09-10T19:05:09.906 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:05:09 smithi183 ceph-mon[29736]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-09-10T19:05:09.906 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:05:09 smithi183 ceph-mon[29736]: rocksdb: Options.target_file_size_base: 67108864 2024-09-10T19:05:09.906 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:05:09 smithi183 ceph-mon[29736]: rocksdb: Options.target_file_size_multiplier: 1 2024-09-10T19:05:09.906 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:05:09 smithi183 ceph-mon[29736]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-09-10T19:05:09.906 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:05:09 smithi183 ceph-mon[29736]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-09-10T19:05:09.906 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:05:09 smithi183 ceph-mon[29736]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-09-10T19:05:09.906 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:05:09 smithi183 ceph-mon[29736]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-09-10T19:05:09.906 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:05:09 smithi183 ceph-mon[29736]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-09-10T19:05:09.906 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:05:09 smithi183 ceph-mon[29736]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-09-10T19:05:09.906 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:05:09 smithi183 ceph-mon[29736]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-09-10T19:05:09.906 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:05:09 smithi183 ceph-mon[29736]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-09-10T19:05:09.906 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:05:09 smithi183 ceph-mon[29736]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-09-10T19:05:09.906 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:05:09 smithi183 ceph-mon[29736]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-09-10T19:05:09.907 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:05:09 smithi183 ceph-mon[29736]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-09-10T19:05:09.907 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:05:09 smithi183 ceph-mon[29736]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-09-10T19:05:09.907 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:05:09 smithi183 ceph-mon[29736]: rocksdb: Options.arena_block_size: 4194304 2024-09-10T19:05:09.907 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:05:09 smithi183 ceph-mon[29736]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-09-10T19:05:09.907 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:05:09 smithi183 ceph-mon[29736]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-09-10T19:05:09.907 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:05:09 smithi183 ceph-mon[29736]: rocksdb: Options.rate_limit_delay_max_milliseconds: 100 2024-09-10T19:05:09.907 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:05:09 smithi183 ceph-mon[29736]: rocksdb: Options.disable_auto_compactions: 0 2024-09-10T19:05:09.907 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:05:09 smithi183 ceph-mon[29736]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-09-10T19:05:09.907 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:05:09 smithi183 ceph-mon[29736]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-09-10T19:05:09.907 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:05:09 smithi183 ceph-mon[29736]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-09-10T19:05:09.907 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:05:09 smithi183 ceph-mon[29736]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-09-10T19:05:09.907 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:05:09 smithi183 ceph-mon[29736]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-09-10T19:05:09.907 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:05:09 smithi183 ceph-mon[29736]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-09-10T19:05:09.908 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:05:09 smithi183 ceph-mon[29736]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-09-10T19:05:09.908 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:05:09 smithi183 ceph-mon[29736]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-09-10T19:05:09.908 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:05:09 smithi183 ceph-mon[29736]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-09-10T19:05:09.908 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:05:09 smithi183 ceph-mon[29736]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-09-10T19:05:09.908 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:05:09 smithi183 ceph-mon[29736]: rocksdb: Options.table_properties_collectors: CompactOnDeletionCollector (Sliding window size = 32768 Deletion trigger = 16384 Deletion ratio = 0); 2024-09-10T19:05:09.908 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:05:09 smithi183 ceph-mon[29736]: rocksdb: Options.inplace_update_support: 0 2024-09-10T19:05:09.908 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:05:09 smithi183 ceph-mon[29736]: rocksdb: Options.inplace_update_num_locks: 10000 2024-09-10T19:05:09.908 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:05:09 smithi183 ceph-mon[29736]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-09-10T19:05:09.908 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:05:09 smithi183 ceph-mon[29736]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-09-10T19:05:09.908 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:05:09 smithi183 ceph-mon[29736]: rocksdb: Options.memtable_huge_page_size: 0 2024-09-10T19:05:09.908 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:05:09 smithi183 ceph-mon[29736]: rocksdb: Options.bloom_locality: 0 2024-09-10T19:05:09.908 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:05:09 smithi183 ceph-mon[29736]: rocksdb: Options.max_successive_merges: 0 2024-09-10T19:05:09.908 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:05:09 smithi183 ceph-mon[29736]: rocksdb: Options.optimize_filters_for_hits: 0 2024-09-10T19:05:09.908 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:05:09 smithi183 ceph-mon[29736]: rocksdb: Options.paranoid_file_checks: 0 2024-09-10T19:05:09.909 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:05:09 smithi183 ceph-mon[29736]: rocksdb: Options.force_consistency_checks: 1 2024-09-10T19:05:09.909 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:05:09 smithi183 ceph-mon[29736]: rocksdb: Options.report_bg_io_stats: 0 2024-09-10T19:05:09.909 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:05:09 smithi183 ceph-mon[29736]: rocksdb: Options.ttl: 2592000 2024-09-10T19:05:09.909 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:05:09 smithi183 ceph-mon[29736]: rocksdb: Options.periodic_compaction_seconds: 0 2024-09-10T19:05:09.909 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:05:09 smithi183 ceph-mon[29736]: rocksdb: Options.enable_blob_files: false 2024-09-10T19:05:09.909 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:05:09 smithi183 ceph-mon[29736]: rocksdb: Options.min_blob_size: 0 2024-09-10T19:05:09.909 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:05:09 smithi183 ceph-mon[29736]: rocksdb: Options.blob_file_size: 268435456 2024-09-10T19:05:09.909 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:05:09 smithi183 ceph-mon[29736]: rocksdb: Options.blob_compression_type: NoCompression 2024-09-10T19:05:09.909 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:05:09 smithi183 ceph-mon[29736]: rocksdb: Options.enable_blob_garbage_collection: false 2024-09-10T19:05:09.909 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:05:09 smithi183 ceph-mon[29736]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-09-10T19:05:09.909 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:05:09 smithi183 ceph-mon[29736]: rocksdb: [db/version_set.cc:4764] Recovered from manifest file:/var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000009 succeeded,manifest_file_number is 9, next_file_number is 11, last_sequence is 5, log_number is 5,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-09-10T19:05:09.909 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:05:09 smithi183 ceph-mon[29736]: rocksdb: [db/version_set.cc:4779] Column family [default] (ID 0), log number is 5 2024-09-10T19:05:09.909 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:05:09 smithi183 ceph-mon[29736]: rocksdb: [db/version_set.cc:4082] Creating manifest 13 2024-09-10T19:05:09.909 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:05:09 smithi183 ceph-mon[29736]: rocksdb: EVENT_LOG_v1 {"time_micros": 1725995109568843, "job": 1, "event": "recovery_started", "wal_files": [10]} 2024-09-10T19:05:09.910 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:05:09 smithi183 ceph-mon[29736]: rocksdb: [db/db_impl/db_impl_open.cc:845] Recovering log #10 mode 2 2024-09-10T19:05:09.910 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:05:09 smithi183 ceph-mon[29736]: rocksdb: [table/block_based/filter_policy.cc:991] Using legacy Bloom filter with high (20) bits/key. Dramatic filter space and/or accuracy improvement is available with format_version>=5. 2024-09-10T19:05:09.910 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:05:09 smithi183 ceph-mon[29736]: rocksdb: EVENT_LOG_v1 {"time_micros": 1725995109569597, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 14, "file_size": 85219, "file_checksum": "", "file_checksum_func_name": "Unknown", "table_properties": {"data_size": 83310, "index_size": 243, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 709, "raw_key_size": 11312, "raw_average_key_size": 48, "raw_value_size": 77020, "raw_average_value_size": 330, "num_data_blocks": 10, "num_entries": 233, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "rocksdb.BuiltinBloomFilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[CompactOnDeletionCollector]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; ", "creation_time": 1725995109, "oldest_key_time": 0, "file_creation_time": 0, "db_id": "cd026548-f58b-486c-bae0-f5d99bdeb6d6", "db_session_id": "U0Q5F17B8FHO0OY4JC2E"}} 2024-09-10T19:05:09.910 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:05:09 smithi183 ceph-mon[29736]: rocksdb: [db/version_set.cc:4082] Creating manifest 15 2024-09-10T19:05:09.910 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:05:09 smithi183 ceph-mon[29736]: rocksdb: EVENT_LOG_v1 {"time_micros": 1725995109569943, "job": 1, "event": "recovery_finished"} 2024-09-10T19:05:09.910 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:05:09 smithi183 ceph-mon[29736]: rocksdb: [file/delete_scheduler.cc:69] Deleted file /var/lib/ceph/mon/ceph-a/store.db/000010.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-09-10T19:05:09.910 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:05:09 smithi183 ceph-mon[29736]: rocksdb: [db/db_impl/db_impl_open.cc:1700] SstFileManager instance 0x55f370386700 2024-09-10T19:05:09.910 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:05:09 smithi183 ceph-mon[29736]: rocksdb: DB pointer 0x55f370356000 2024-09-10T19:05:09.910 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:05:09 smithi183 ceph-mon[29736]: rocksdb: [db/db_impl/db_impl.cc:901] ------- DUMPING STATS ------- 2024-09-10T19:05:09.910 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:05:09 smithi183 ceph-mon[29736]: rocksdb: [db/db_impl/db_impl.cc:903] 2024-09-10T19:05:09.910 INFO:journalctl@ceph.mon.a.smithi183.stdout: ** DB Stats ** 2024-09-10T19:05:09.910 INFO:journalctl@ceph.mon.a.smithi183.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-09-10T19:05:09.910 INFO:journalctl@ceph.mon.a.smithi183.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-09-10T19:05:09.910 INFO:journalctl@ceph.mon.a.smithi183.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-09-10T19:05:09.911 INFO:journalctl@ceph.mon.a.smithi183.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-09-10T19:05:09.911 INFO:journalctl@ceph.mon.a.smithi183.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-09-10T19:05:09.911 INFO:journalctl@ceph.mon.a.smithi183.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s 2024-09-10T19:05:09.911 INFO:journalctl@ceph.mon.a.smithi183.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-09-10T19:05:09.911 INFO:journalctl@ceph.mon.a.smithi183.stdout: 2024-09-10T19:05:09.911 INFO:journalctl@ceph.mon.a.smithi183.stdout: ** Compaction Stats [default] ** 2024-09-10T19:05:09.911 INFO:journalctl@ceph.mon.a.smithi183.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-09-10T19:05:09.911 INFO:journalctl@ceph.mon.a.smithi183.stdout: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-09-10T19:05:09.911 INFO:journalctl@ceph.mon.a.smithi183.stdout: L0 2/0 85.01 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 161.3 0.00 0.00 1 0.001 0 0 2024-09-10T19:05:09.911 INFO:journalctl@ceph.mon.a.smithi183.stdout: Sum 2/0 85.01 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 161.3 0.00 0.00 1 0.001 0 0 2024-09-10T19:05:09.911 INFO:journalctl@ceph.mon.a.smithi183.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 161.3 0.00 0.00 1 0.001 0 0 2024-09-10T19:05:09.911 INFO:journalctl@ceph.mon.a.smithi183.stdout: 2024-09-10T19:05:09.911 INFO:journalctl@ceph.mon.a.smithi183.stdout: ** Compaction Stats [default] ** 2024-09-10T19:05:09.911 INFO:journalctl@ceph.mon.a.smithi183.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-09-10T19:05:09.912 INFO:journalctl@ceph.mon.a.smithi183.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-09-10T19:05:09.912 INFO:journalctl@ceph.mon.a.smithi183.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 161.3 0.00 0.00 1 0.001 0 0 2024-09-10T19:05:09.912 INFO:journalctl@ceph.mon.a.smithi183.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-09-10T19:05:09.912 INFO:journalctl@ceph.mon.a.smithi183.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-09-10T19:05:09.912 INFO:journalctl@ceph.mon.a.smithi183.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-09-10T19:05:09.912 INFO:journalctl@ceph.mon.a.smithi183.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-09-10T19:05:09.912 INFO:journalctl@ceph.mon.a.smithi183.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-09-10T19:05:09.912 INFO:journalctl@ceph.mon.a.smithi183.stdout: AddFile(Keys): cumulative 0, interval 0 2024-09-10T19:05:09.912 INFO:journalctl@ceph.mon.a.smithi183.stdout: Cumulative compaction: 0.00 GB write, 26.41 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-10T19:05:09.912 INFO:journalctl@ceph.mon.a.smithi183.stdout: Interval compaction: 0.00 GB write, 26.41 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-10T19:05:09.912 INFO:journalctl@ceph.mon.a.smithi183.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-09-10T19:05:09.912 INFO:journalctl@ceph.mon.a.smithi183.stdout: 2024-09-10T19:05:09.913 INFO:journalctl@ceph.mon.a.smithi183.stdout: ** File Read Latency Histogram By Level [default] ** 2024-09-10T19:05:09.913 INFO:journalctl@ceph.mon.a.smithi183.stdout: 2024-09-10T19:05:09.913 INFO:journalctl@ceph.mon.a.smithi183.stdout: ** Compaction Stats [default] ** 2024-09-10T19:05:09.913 INFO:journalctl@ceph.mon.a.smithi183.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-09-10T19:05:09.913 INFO:journalctl@ceph.mon.a.smithi183.stdout: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-09-10T19:05:09.913 INFO:journalctl@ceph.mon.a.smithi183.stdout: L0 2/0 85.01 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 161.3 0.00 0.00 1 0.001 0 0 2024-09-10T19:05:09.913 INFO:journalctl@ceph.mon.a.smithi183.stdout: Sum 2/0 85.01 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 161.3 0.00 0.00 1 0.001 0 0 2024-09-10T19:05:09.913 INFO:journalctl@ceph.mon.a.smithi183.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2024-09-10T19:05:09.913 INFO:journalctl@ceph.mon.a.smithi183.stdout: 2024-09-10T19:05:09.913 INFO:journalctl@ceph.mon.a.smithi183.stdout: ** Compaction Stats [default] ** 2024-09-10T19:05:09.913 INFO:journalctl@ceph.mon.a.smithi183.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-09-10T19:05:09.913 INFO:journalctl@ceph.mon.a.smithi183.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-09-10T19:05:09.913 INFO:journalctl@ceph.mon.a.smithi183.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 161.3 0.00 0.00 1 0.001 0 0 2024-09-10T19:05:09.914 INFO:journalctl@ceph.mon.a.smithi183.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-09-10T19:05:09.914 INFO:journalctl@ceph.mon.a.smithi183.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-09-10T19:05:09.914 INFO:journalctl@ceph.mon.a.smithi183.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-09-10T19:05:09.914 INFO:journalctl@ceph.mon.a.smithi183.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-09-10T19:05:09.914 INFO:journalctl@ceph.mon.a.smithi183.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-09-10T19:05:09.914 INFO:journalctl@ceph.mon.a.smithi183.stdout: AddFile(Keys): cumulative 0, interval 0 2024-09-10T19:05:09.914 INFO:journalctl@ceph.mon.a.smithi183.stdout: Cumulative compaction: 0.00 GB write, 26.15 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-10T19:05:09.914 INFO:journalctl@ceph.mon.a.smithi183.stdout: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-10T19:05:09.914 INFO:journalctl@ceph.mon.a.smithi183.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-09-10T19:05:09.914 INFO:journalctl@ceph.mon.a.smithi183.stdout: 2024-09-10T19:05:09.914 INFO:journalctl@ceph.mon.a.smithi183.stdout: ** File Read Latency Histogram By Level [default] ** 2024-09-10T19:05:09.914 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:05:09 smithi183 ceph-mon[29736]: starting mon.a rank 0 at public addrs [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] at bind addrs [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] mon_data /var/lib/ceph/mon/ceph-a fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 2024-09-10T19:05:09.914 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:05:09 smithi183 ceph-mon[29736]: mon.a@-1(???) e1 preinit fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 2024-09-10T19:05:09.914 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:05:09 smithi183 ceph-mon[29736]: mon.a@-1(???).mds e1 new map 2024-09-10T19:05:09.914 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:05:09 smithi183 ceph-mon[29736]: mon.a@-1(???).mds e1 print_map 2024-09-10T19:05:09.915 INFO:journalctl@ceph.mon.a.smithi183.stdout: e1 2024-09-10T19:05:09.915 INFO:journalctl@ceph.mon.a.smithi183.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-09-10T19:05:09.915 INFO:journalctl@ceph.mon.a.smithi183.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-09-10T19:05:09.915 INFO:journalctl@ceph.mon.a.smithi183.stdout: legacy client fscid: -1 2024-09-10T19:05:09.915 INFO:journalctl@ceph.mon.a.smithi183.stdout: 2024-09-10T19:05:09.915 INFO:journalctl@ceph.mon.a.smithi183.stdout: No filesystems configured 2024-09-10T19:05:09.915 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:05:09 smithi183 ceph-mon[29736]: mon.a@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2024-09-10T19:05:09.915 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:05:09 smithi183 ceph-mon[29736]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-09-10T19:05:09.915 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:05:09 smithi183 ceph-mon[29736]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-09-10T19:05:09.915 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:05:09 smithi183 ceph-mon[29736]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-09-10T19:05:09.915 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:05:09 smithi183 ceph-mon[29736]: mon.a@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2024-09-10T19:05:09.915 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:05:09 smithi183 ceph-mon[29736]: mon.a is new leader, mons a in quorum (ranks 0) 2024-09-10T19:05:09.915 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:05:09 smithi183 ceph-mon[29736]: monmap e1: 1 mons at {a=[v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-09-10T19:05:09.915 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:05:09 smithi183 ceph-mon[29736]: fsmap 2024-09-10T19:05:09.915 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:05:09 smithi183 ceph-mon[29736]: osdmap e1: 0 total, 0 up, 0 in 2024-09-10T19:05:09.916 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:05:09 smithi183 ceph-mon[29736]: mgrmap e1: no daemons active 2024-09-10T19:05:09.916 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:05:09 smithi183 bash[29698]: fae323eed8e937f6d021ee78cc6ea5e4e5e25230228ac9d0174430e87674bb76 2024-09-10T19:05:09.916 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:05:09 smithi183 systemd[1]: Started Ceph mon.a for 320f6df8-6fa7-11ef-bcea-c7b262605968. 2024-09-10T19:05:10.934 INFO:teuthology.orchestra.run.smithi183.stdout:Wrote config to /etc/ceph/ceph.conf 2024-09-10T19:05:10.935 INFO:teuthology.orchestra.run.smithi183.stdout:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2024-09-10T19:05:10.935 INFO:teuthology.orchestra.run.smithi183.stdout:Creating mgr... 2024-09-10T19:05:10.935 INFO:teuthology.orchestra.run.smithi183.stdout:Verifying port 0.0.0.0:9283 ... 2024-09-10T19:05:10.935 INFO:teuthology.orchestra.run.smithi183.stdout:Verifying port 0.0.0.0:8765 ... 2024-09-10T19:05:11.152 INFO:teuthology.orchestra.run.smithi183.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-320f6df8-6fa7-11ef-bcea-c7b262605968@mgr.y 2024-09-10T19:05:11.153 INFO:teuthology.orchestra.run.smithi183.stdout:systemctl: stderr Failed to reset failed state of unit ceph-320f6df8-6fa7-11ef-bcea-c7b262605968@mgr.y.service: Unit ceph-320f6df8-6fa7-11ef-bcea-c7b262605968@mgr.y.service not loaded. 2024-09-10T19:05:11.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:05:11 smithi183 ceph-mon[29736]: from='client.? 172.21.15.183:0/1633432940' entity='client.admin' 2024-09-10T19:05:11.336 INFO:teuthology.orchestra.run.smithi183.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-320f6df8-6fa7-11ef-bcea-c7b262605968.target.wants/ceph-320f6df8-6fa7-11ef-bcea-c7b262605968@mgr.y.service → /etc/systemd/system/ceph-320f6df8-6fa7-11ef-bcea-c7b262605968@.service. 2024-09-10T19:05:11.681 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Sep 10 19:05:11 smithi183 podman[29974]: 2024-09-10 19:05:11.589126517 +0000 UTC m=+0.122395476 container create 73c5cf4fd854ee8514163d2ca51c956c5145ae3f18ab431c08031417b0232ade (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588, name=ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-y, io.buildah.version=1.37.2, RELEASE=wip-guits-quincy-2024-09-09-1810-quincy-5be5739, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.vendor=CentOS, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, ceph=True, GIT_BRANCH=HEAD) 2024-09-10T19:05:11.923 INFO:teuthology.orchestra.run.smithi183.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-09-10T19:05:11.923 INFO:teuthology.orchestra.run.smithi183.stdout:systemctl: stdout disabled 2024-09-10T19:05:11.931 INFO:teuthology.orchestra.run.smithi183.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-09-10T19:05:11.931 INFO:teuthology.orchestra.run.smithi183.stdout:systemctl: stdout inactive 2024-09-10T19:05:11.931 INFO:teuthology.orchestra.run.smithi183.stdout:firewalld.service is not enabled 2024-09-10T19:05:11.931 INFO:teuthology.orchestra.run.smithi183.stdout:Not possible to enable service . firewalld.service is not available 2024-09-10T19:05:11.940 INFO:teuthology.orchestra.run.smithi183.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-09-10T19:05:11.940 INFO:teuthology.orchestra.run.smithi183.stdout:systemctl: stdout disabled 2024-09-10T19:05:11.948 INFO:teuthology.orchestra.run.smithi183.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-09-10T19:05:11.948 INFO:teuthology.orchestra.run.smithi183.stdout:systemctl: stdout inactive 2024-09-10T19:05:11.948 INFO:teuthology.orchestra.run.smithi183.stdout:firewalld.service is not enabled 2024-09-10T19:05:11.948 INFO:teuthology.orchestra.run.smithi183.stdout:Not possible to open ports <[9283, 8765]>. firewalld.service is not available 2024-09-10T19:05:11.948 INFO:teuthology.orchestra.run.smithi183.stdout:Waiting for mgr to start... 2024-09-10T19:05:11.948 INFO:teuthology.orchestra.run.smithi183.stdout:Waiting for mgr... 2024-09-10T19:05:12.072 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Sep 10 19:05:11 smithi183 podman[29974]: 2024-09-10 19:05:11.79997021 +0000 UTC m=+0.333239169 container init 73c5cf4fd854ee8514163d2ca51c956c5145ae3f18ab431c08031417b0232ade (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588, name=ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-y, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, ceph=True, RELEASE=wip-guits-quincy-2024-09-09-1810-quincy-5be5739, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS) 2024-09-10T19:05:12.073 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Sep 10 19:05:11 smithi183 podman[29974]: 2024-09-10 19:05:11.805250592 +0000 UTC m=+0.338519551 container start 73c5cf4fd854ee8514163d2ca51c956c5145ae3f18ab431c08031417b0232ade (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588, name=ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-y, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, ceph=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-guits-quincy-2024-09-09-1810-quincy-5be5739, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7) 2024-09-10T19:05:12.073 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Sep 10 19:05:11 smithi183 bash[29974]: 73c5cf4fd854ee8514163d2ca51c956c5145ae3f18ab431c08031417b0232ade 2024-09-10T19:05:12.073 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Sep 10 19:05:11 smithi183 systemd[1]: Started Ceph mgr.y for 320f6df8-6fa7-11ef-bcea-c7b262605968. 2024-09-10T19:05:12.073 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Sep 10 19:05:11 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-y[29989]: 2024-09-10T19:05:11.917+0000 7f6c66a6fec0 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-09-10T19:05:12.073 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Sep 10 19:05:11 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-y[29989]: 2024-09-10T19:05:11.976+0000 7f6c66a6fec0 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-09-10T19:05:12.073 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Sep 10 19:05:12 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-y[29989]: 2024-09-10T19:05:12.041+0000 7f6c66a6fec0 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-09-10T19:05:12.337 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Sep 10 19:05:12 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-y[29989]: 2024-09-10T19:05:12.168+0000 7f6c66a6fec0 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-09-10T19:05:12.338 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Sep 10 19:05:12 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-y[29989]: 2024-09-10T19:05:12.308+0000 7f6c66a6fec0 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-09-10T19:05:12.640 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:05:12 smithi183 ceph-mon[29736]: from='client.? 172.21.15.183:0/1665738154' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-09-10T19:05:12.641 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Sep 10 19:05:12 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-y[29989]: 2024-09-10T19:05:12.579+0000 7f6c66a6fec0 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-09-10T19:05:12.641 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Sep 10 19:05:12 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-y[29989]: 2024-09-10T19:05:12.638+0000 7f6c66a6fec0 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-09-10T19:05:12.931 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Sep 10 19:05:12 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-y[29989]: 2024-09-10T19:05:12.808+0000 7f6c66a6fec0 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-09-10T19:05:13.062 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout 2024-09-10T19:05:13.062 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout { 2024-09-10T19:05:13.062 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout "fsid": "320f6df8-6fa7-11ef-bcea-c7b262605968", 2024-09-10T19:05:13.062 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout "health": { 2024-09-10T19:05:13.062 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-09-10T19:05:13.062 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-09-10T19:05:13.062 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-09-10T19:05:13.062 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout }, 2024-09-10T19:05:13.063 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-09-10T19:05:13.063 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-09-10T19:05:13.063 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout 0 2024-09-10T19:05:13.063 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout ], 2024-09-10T19:05:13.063 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-09-10T19:05:13.063 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout "a" 2024-09-10T19:05:13.063 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout ], 2024-09-10T19:05:13.063 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout "quorum_age": 2, 2024-09-10T19:05:13.063 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout "monmap": { 2024-09-10T19:05:13.063 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-10T19:05:13.063 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "quincy", 2024-09-10T19:05:13.063 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-09-10T19:05:13.063 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout }, 2024-09-10T19:05:13.064 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-09-10T19:05:13.064 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-10T19:05:13.064 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-09-10T19:05:13.064 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-09-10T19:05:13.064 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-09-10T19:05:13.064 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-09-10T19:05:13.064 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-09-10T19:05:13.064 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-09-10T19:05:13.067 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout }, 2024-09-10T19:05:13.068 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-09-10T19:05:13.068 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-09-10T19:05:13.068 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-09-10T19:05:13.068 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-09-10T19:05:13.068 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-09-10T19:05:13.068 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-09-10T19:05:13.068 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-09-10T19:05:13.068 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-09-10T19:05:13.068 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-09-10T19:05:13.068 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout }, 2024-09-10T19:05:13.068 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-09-10T19:05:13.068 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-10T19:05:13.068 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-09-10T19:05:13.069 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-09-10T19:05:13.069 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout }, 2024-09-10T19:05:13.069 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-09-10T19:05:13.069 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout "available": false, 2024-09-10T19:05:13.069 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-09-10T19:05:13.069 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout "modules": [ 2024-09-10T19:05:13.069 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout "iostat", 2024-09-10T19:05:13.069 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout "nfs", 2024-09-10T19:05:13.069 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout "restful" 2024-09-10T19:05:13.069 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout ], 2024-09-10T19:05:13.069 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout "services": {} 2024-09-10T19:05:13.069 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout }, 2024-09-10T19:05:13.069 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-09-10T19:05:13.070 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-10T19:05:13.070 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout "modified": "2024-09-10T19:05:03.688982+0000", 2024-09-10T19:05:13.070 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout "services": {} 2024-09-10T19:05:13.070 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout }, 2024-09-10T19:05:13.070 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-09-10T19:05:13.070 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout } 2024-09-10T19:05:13.070 INFO:teuthology.orchestra.run.smithi183.stdout:mgr not available, waiting (1/15)... 2024-09-10T19:05:13.931 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Sep 10 19:05:13 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-y[29989]: 2024-09-10T19:05:13.641+0000 7f6c66a6fec0 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-09-10T19:05:13.931 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Sep 10 19:05:13 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-y[29989]: 2024-09-10T19:05:13.704+0000 7f6c66a6fec0 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-09-10T19:05:14.681 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Sep 10 19:05:14 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-y[29989]: 2024-09-10T19:05:14.399+0000 7f6c66a6fec0 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-09-10T19:05:14.681 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Sep 10 19:05:14 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-y[29989]: 2024-09-10T19:05:14.507+0000 7f6c66a6fec0 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-09-10T19:05:14.681 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Sep 10 19:05:14 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-y[29989]: 2024-09-10T19:05:14.624+0000 7f6c66a6fec0 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-09-10T19:05:14.938 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Sep 10 19:05:14 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-y[29989]: 2024-09-10T19:05:14.682+0000 7f6c66a6fec0 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-09-10T19:05:14.938 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Sep 10 19:05:14 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-y[29989]: 2024-09-10T19:05:14.810+0000 7f6c66a6fec0 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-09-10T19:05:14.938 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Sep 10 19:05:14 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-y[29989]: 2024-09-10T19:05:14.877+0000 7f6c66a6fec0 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-09-10T19:05:14.938 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Sep 10 19:05:14 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-y[29989]: 2024-09-10T19:05:14.936+0000 7f6c66a6fec0 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-09-10T19:05:15.190 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Sep 10 19:05:15 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-y[29989]: 2024-09-10T19:05:15.075+0000 7f6c66a6fec0 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-09-10T19:05:15.190 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Sep 10 19:05:15 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-y[29989]: 2024-09-10T19:05:15.144+0000 7f6c66a6fec0 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-09-10T19:05:15.451 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Sep 10 19:05:15 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-y[29989]: 2024-09-10T19:05:15.214+0000 7f6c66a6fec0 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-09-10T19:05:15.451 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Sep 10 19:05:15 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-y[29989]: 2024-09-10T19:05:15.301+0000 7f6c66a6fec0 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-09-10T19:05:15.452 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Sep 10 19:05:15 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-y[29989]: 2024-09-10T19:05:15.362+0000 7f6c66a6fec0 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-09-10T19:05:15.452 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Sep 10 19:05:15 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-y[29989]: 2024-09-10T19:05:15.421+0000 7f6c66a6fec0 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-09-10T19:05:15.904 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:05:15 smithi183 ceph-mon[29736]: from='client.? 172.21.15.183:0/4244224475' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-09-10T19:05:16.227 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout 2024-09-10T19:05:16.227 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout { 2024-09-10T19:05:16.227 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout "fsid": "320f6df8-6fa7-11ef-bcea-c7b262605968", 2024-09-10T19:05:16.227 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout "health": { 2024-09-10T19:05:16.227 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-09-10T19:05:16.227 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-09-10T19:05:16.227 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-09-10T19:05:16.227 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout }, 2024-09-10T19:05:16.227 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-09-10T19:05:16.227 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-09-10T19:05:16.227 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout 0 2024-09-10T19:05:16.228 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout ], 2024-09-10T19:05:16.228 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-09-10T19:05:16.228 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout "a" 2024-09-10T19:05:16.228 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout ], 2024-09-10T19:05:16.228 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout "quorum_age": 6, 2024-09-10T19:05:16.228 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout "monmap": { 2024-09-10T19:05:16.228 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-10T19:05:16.228 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "quincy", 2024-09-10T19:05:16.228 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-09-10T19:05:16.228 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout }, 2024-09-10T19:05:16.228 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-09-10T19:05:16.228 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-10T19:05:16.228 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-09-10T19:05:16.228 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-09-10T19:05:16.229 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-09-10T19:05:16.229 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-09-10T19:05:16.229 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-09-10T19:05:16.229 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-09-10T19:05:16.229 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout }, 2024-09-10T19:05:16.229 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-09-10T19:05:16.229 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-09-10T19:05:16.229 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-09-10T19:05:16.229 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-09-10T19:05:16.229 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-09-10T19:05:16.229 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-09-10T19:05:16.229 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-09-10T19:05:16.229 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-09-10T19:05:16.229 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-09-10T19:05:16.230 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout }, 2024-09-10T19:05:16.230 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-09-10T19:05:16.230 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-10T19:05:16.230 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-09-10T19:05:16.230 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-09-10T19:05:16.230 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout }, 2024-09-10T19:05:16.230 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-09-10T19:05:16.230 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout "available": false, 2024-09-10T19:05:16.230 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-09-10T19:05:16.230 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout "modules": [ 2024-09-10T19:05:16.230 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout "iostat", 2024-09-10T19:05:16.230 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout "nfs", 2024-09-10T19:05:16.230 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout "restful" 2024-09-10T19:05:16.230 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout ], 2024-09-10T19:05:16.230 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout "services": {} 2024-09-10T19:05:16.231 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout }, 2024-09-10T19:05:16.231 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-09-10T19:05:16.231 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-10T19:05:16.231 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout "modified": "2024-09-10T19:05:03.688982+0000", 2024-09-10T19:05:16.231 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout "services": {} 2024-09-10T19:05:16.231 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout }, 2024-09-10T19:05:16.231 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-09-10T19:05:16.231 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout } 2024-09-10T19:05:16.231 INFO:teuthology.orchestra.run.smithi183.stdout:mgr not available, waiting (2/15)... 2024-09-10T19:05:16.646 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Sep 10 19:05:16 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-y[29989]: 2024-09-10T19:05:16.195+0000 7f6c66a6fec0 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-09-10T19:05:16.931 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:05:16 smithi183 ceph-mon[29736]: Activating manager daemon y 2024-09-10T19:05:16.931 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:05:16 smithi183 ceph-mon[29736]: mgrmap e2: y(active, starting, since 0.00276834s) 2024-09-10T19:05:16.931 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:05:16 smithi183 ceph-mon[29736]: from='mgr.14100 172.21.15.183:0/3657357378' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-09-10T19:05:16.931 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:05:16 smithi183 ceph-mon[29736]: from='mgr.14100 172.21.15.183:0/3657357378' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-09-10T19:05:16.931 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:05:16 smithi183 ceph-mon[29736]: from='mgr.14100 172.21.15.183:0/3657357378' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-09-10T19:05:16.931 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:05:16 smithi183 ceph-mon[29736]: from='mgr.14100 172.21.15.183:0/3657357378' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-10T19:05:16.932 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:05:16 smithi183 ceph-mon[29736]: from='mgr.14100 172.21.15.183:0/3657357378' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-09-10T19:05:16.932 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:05:16 smithi183 ceph-mon[29736]: Manager daemon y is now available 2024-09-10T19:05:16.932 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:05:16 smithi183 ceph-mon[29736]: from='mgr.14100 172.21.15.183:0/3657357378' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-09-10T19:05:16.932 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:05:16 smithi183 ceph-mon[29736]: from='mgr.14100 172.21.15.183:0/3657357378' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-09-10T19:05:16.932 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:05:16 smithi183 ceph-mon[29736]: from='mgr.14100 172.21.15.183:0/3657357378' entity='mgr.y' 2024-09-10T19:05:16.932 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:05:16 smithi183 ceph-mon[29736]: from='mgr.14100 172.21.15.183:0/3657357378' entity='mgr.y' 2024-09-10T19:05:16.932 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:05:16 smithi183 ceph-mon[29736]: from='mgr.14100 172.21.15.183:0/3657357378' entity='mgr.y' 2024-09-10T19:05:18.461 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:05:18 smithi183 ceph-mon[29736]: mgrmap e3: y(active, since 1.007s) 2024-09-10T19:05:19.419 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:05:19 smithi183 ceph-mon[29736]: mgrmap e4: y(active, since 2s) 2024-09-10T19:05:19.419 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:05:19 smithi183 ceph-mon[29736]: from='client.? 172.21.15.183:0/3660863660' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-09-10T19:05:19.423 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout 2024-09-10T19:05:19.424 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout { 2024-09-10T19:05:19.424 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout "fsid": "320f6df8-6fa7-11ef-bcea-c7b262605968", 2024-09-10T19:05:19.424 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout "health": { 2024-09-10T19:05:19.424 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-09-10T19:05:19.424 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-09-10T19:05:19.424 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-09-10T19:05:19.424 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout }, 2024-09-10T19:05:19.424 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-09-10T19:05:19.424 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-09-10T19:05:19.424 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout 0 2024-09-10T19:05:19.424 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout ], 2024-09-10T19:05:19.424 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-09-10T19:05:19.425 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout "a" 2024-09-10T19:05:19.425 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout ], 2024-09-10T19:05:19.425 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout "quorum_age": 9, 2024-09-10T19:05:19.425 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout "monmap": { 2024-09-10T19:05:19.425 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-10T19:05:19.425 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "quincy", 2024-09-10T19:05:19.425 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-09-10T19:05:19.425 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout }, 2024-09-10T19:05:19.425 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-09-10T19:05:19.425 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-10T19:05:19.425 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-09-10T19:05:19.425 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-09-10T19:05:19.425 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-09-10T19:05:19.425 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-09-10T19:05:19.426 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-09-10T19:05:19.426 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-09-10T19:05:19.429 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout }, 2024-09-10T19:05:19.429 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-09-10T19:05:19.429 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-09-10T19:05:19.429 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-09-10T19:05:19.429 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-09-10T19:05:19.429 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-09-10T19:05:19.429 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-09-10T19:05:19.429 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-09-10T19:05:19.429 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-09-10T19:05:19.429 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-09-10T19:05:19.430 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout }, 2024-09-10T19:05:19.430 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-09-10T19:05:19.430 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-10T19:05:19.430 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-09-10T19:05:19.430 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-09-10T19:05:19.430 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout }, 2024-09-10T19:05:19.430 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-09-10T19:05:19.430 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout "available": true, 2024-09-10T19:05:19.430 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-09-10T19:05:19.430 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout "modules": [ 2024-09-10T19:05:19.430 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout "iostat", 2024-09-10T19:05:19.430 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout "nfs", 2024-09-10T19:05:19.430 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout "restful" 2024-09-10T19:05:19.430 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout ], 2024-09-10T19:05:19.431 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout "services": {} 2024-09-10T19:05:19.431 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout }, 2024-09-10T19:05:19.431 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-09-10T19:05:19.431 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-10T19:05:19.431 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout "modified": "2024-09-10T19:05:03.688982+0000", 2024-09-10T19:05:19.431 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout "services": {} 2024-09-10T19:05:19.431 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout }, 2024-09-10T19:05:19.431 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-09-10T19:05:19.431 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout } 2024-09-10T19:05:19.431 INFO:teuthology.orchestra.run.smithi183.stdout:mgr is available 2024-09-10T19:05:20.341 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:05:20 smithi183 ceph-mon[29736]: from='client.? 172.21.15.183:0/3103672091' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-09-10T19:05:20.629 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout 2024-09-10T19:05:20.629 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout [global] 2024-09-10T19:05:20.630 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout fsid = 320f6df8-6fa7-11ef-bcea-c7b262605968 2024-09-10T19:05:20.630 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout mon_cluster_log_file_level = debug 2024-09-10T19:05:20.630 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-09-10T19:05:20.630 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-09-10T19:05:20.630 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-09-10T19:05:20.630 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-09-10T19:05:20.630 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout 2024-09-10T19:05:20.630 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout [mgr] 2024-09-10T19:05:20.630 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-09-10T19:05:20.630 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout 2024-09-10T19:05:20.630 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout [osd] 2024-09-10T19:05:20.630 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-09-10T19:05:20.630 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-09-10T19:05:20.631 INFO:teuthology.orchestra.run.smithi183.stdout:Enabling cephadm module... 2024-09-10T19:05:21.622 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:05:21 smithi183 ceph-mon[29736]: from='client.? 172.21.15.183:0/1707210434' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2024-09-10T19:05:21.623 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Sep 10 19:05:21 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-y[29989]: ignoring --setuser ceph since I am not root 2024-09-10T19:05:21.623 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Sep 10 19:05:21 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-y[29989]: ignoring --setgroup ceph since I am not root 2024-09-10T19:05:21.623 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Sep 10 19:05:21 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-y[29989]: 2024-09-10T19:05:21.463+0000 7fa7cd79bec0 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-09-10T19:05:21.623 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Sep 10 19:05:21 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-y[29989]: 2024-09-10T19:05:21.517+0000 7fa7cd79bec0 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-09-10T19:05:21.623 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Sep 10 19:05:21 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-y[29989]: 2024-09-10T19:05:21.584+0000 7fa7cd79bec0 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-09-10T19:05:21.881 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Sep 10 19:05:21 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-y[29989]: 2024-09-10T19:05:21.709+0000 7fa7cd79bec0 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-09-10T19:05:21.881 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Sep 10 19:05:21 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-y[29989]: 2024-09-10T19:05:21.847+0000 7fa7cd79bec0 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-09-10T19:05:22.174 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Sep 10 19:05:22 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-y[29989]: 2024-09-10T19:05:22.112+0000 7fa7cd79bec0 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-09-10T19:05:22.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:05:22 smithi183 ceph-mon[29736]: from='client.? 172.21.15.183:0/1707210434' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2024-09-10T19:05:22.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:05:22 smithi183 ceph-mon[29736]: mgrmap e5: y(active, since 5s) 2024-09-10T19:05:22.431 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Sep 10 19:05:22 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-y[29989]: 2024-09-10T19:05:22.173+0000 7fa7cd79bec0 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-09-10T19:05:22.431 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Sep 10 19:05:22 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-y[29989]: 2024-09-10T19:05:22.343+0000 7fa7cd79bec0 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-09-10T19:05:23.107 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout { 2024-09-10T19:05:23.108 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout "epoch": 5, 2024-09-10T19:05:23.108 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout "available": true, 2024-09-10T19:05:23.108 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout "active_name": "y", 2024-09-10T19:05:23.108 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-09-10T19:05:23.108 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout } 2024-09-10T19:05:23.108 INFO:teuthology.orchestra.run.smithi183.stdout:Waiting for the mgr to restart... 2024-09-10T19:05:23.108 INFO:teuthology.orchestra.run.smithi183.stdout:Waiting for mgr epoch 5... 2024-09-10T19:05:23.423 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:05:23 smithi183 ceph-mon[29736]: from='client.? 172.21.15.183:0/3143406484' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-09-10T19:05:23.423 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Sep 10 19:05:23 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-y[29989]: 2024-09-10T19:05:23.197+0000 7fa7cd79bec0 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-09-10T19:05:23.423 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Sep 10 19:05:23 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-y[29989]: 2024-09-10T19:05:23.262+0000 7fa7cd79bec0 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-09-10T19:05:24.257 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Sep 10 19:05:23 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-y[29989]: 2024-09-10T19:05:23.970+0000 7fa7cd79bec0 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-09-10T19:05:24.258 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Sep 10 19:05:24 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-y[29989]: 2024-09-10T19:05:24.079+0000 7fa7cd79bec0 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-09-10T19:05:24.258 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Sep 10 19:05:24 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-y[29989]: 2024-09-10T19:05:24.198+0000 7fa7cd79bec0 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-09-10T19:05:24.258 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Sep 10 19:05:24 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-y[29989]: 2024-09-10T19:05:24.256+0000 7fa7cd79bec0 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-09-10T19:05:24.637 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Sep 10 19:05:24 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-y[29989]: 2024-09-10T19:05:24.380+0000 7fa7cd79bec0 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-09-10T19:05:24.638 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Sep 10 19:05:24 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-y[29989]: 2024-09-10T19:05:24.445+0000 7fa7cd79bec0 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-09-10T19:05:24.638 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Sep 10 19:05:24 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-y[29989]: 2024-09-10T19:05:24.502+0000 7fa7cd79bec0 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-09-10T19:05:24.638 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Sep 10 19:05:24 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-y[29989]: 2024-09-10T19:05:24.636+0000 7fa7cd79bec0 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-09-10T19:05:24.901 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Sep 10 19:05:24 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-y[29989]: 2024-09-10T19:05:24.699+0000 7fa7cd79bec0 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-09-10T19:05:24.901 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Sep 10 19:05:24 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-y[29989]: 2024-09-10T19:05:24.762+0000 7fa7cd79bec0 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-09-10T19:05:24.901 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Sep 10 19:05:24 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-y[29989]: 2024-09-10T19:05:24.841+0000 7fa7cd79bec0 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-09-10T19:05:24.901 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Sep 10 19:05:24 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-y[29989]: 2024-09-10T19:05:24.899+0000 7fa7cd79bec0 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-09-10T19:05:25.181 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Sep 10 19:05:24 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-y[29989]: 2024-09-10T19:05:24.956+0000 7fa7cd79bec0 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-09-10T19:05:26.024 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Sep 10 19:05:25 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-y[29989]: 2024-09-10T19:05:25.700+0000 7fa7cd79bec0 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-09-10T19:05:26.024 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:05:25 smithi183 ceph-mon[29736]: Active manager daemon y restarted 2024-09-10T19:05:26.025 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:05:25 smithi183 ceph-mon[29736]: Activating manager daemon y 2024-09-10T19:05:26.025 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:05:25 smithi183 ceph-mon[29736]: osdmap e2: 0 total, 0 up, 0 in 2024-09-10T19:05:26.025 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:05:25 smithi183 ceph-mon[29736]: mgrmap e6: y(active, starting, since 0.00364353s) 2024-09-10T19:05:26.025 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:05:25 smithi183 ceph-mon[29736]: from='mgr.14118 172.21.15.183:0/243006257' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-10T19:05:26.025 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:05:25 smithi183 ceph-mon[29736]: from='mgr.14118 172.21.15.183:0/243006257' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-09-10T19:05:26.025 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:05:25 smithi183 ceph-mon[29736]: from='mgr.14118 172.21.15.183:0/243006257' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-09-10T19:05:26.025 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:05:25 smithi183 ceph-mon[29736]: from='mgr.14118 172.21.15.183:0/243006257' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-09-10T19:05:26.025 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:05:25 smithi183 ceph-mon[29736]: from='mgr.14118 172.21.15.183:0/243006257' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-09-10T19:05:26.025 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:05:25 smithi183 ceph-mon[29736]: Manager daemon y is now available 2024-09-10T19:05:26.025 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:05:25 smithi183 ceph-mon[29736]: from='mgr.14118 172.21.15.183:0/243006257' entity='mgr.y' 2024-09-10T19:05:26.025 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:05:25 smithi183 ceph-mon[29736]: from='mgr.14118 172.21.15.183:0/243006257' entity='mgr.y' 2024-09-10T19:05:26.025 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:05:25 smithi183 ceph-mon[29736]: from='mgr.14118 172.21.15.183:0/243006257' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T19:05:26.025 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:05:25 smithi183 ceph-mon[29736]: from='mgr.14118 172.21.15.183:0/243006257' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T19:05:26.026 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:05:25 smithi183 ceph-mon[29736]: from='mgr.14118 172.21.15.183:0/243006257' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-09-10T19:05:26.026 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:05:25 smithi183 ceph-mon[29736]: from='mgr.14118 172.21.15.183:0/243006257' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-09-10T19:05:27.067 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:05:26 smithi183 ceph-mon[29736]: Found migration_current of "None". Setting to last migration. 2024-09-10T19:05:27.068 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:05:26 smithi183 ceph-mon[29736]: mgrmap e7: y(active, since 1.00568s) 2024-09-10T19:05:27.352 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Sep 10 19:05:27 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-y[29989]: 2024-09-10T19:05:27.142+0000 7fa77fb89640 -1 log_channel(cephadm) log [ERR] : [10/Sep/2024:19:05:27] ENGINE Error in HTTPServer.serve 2024-09-10T19:05:27.352 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Sep 10 19:05:27 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-y[29989]: Traceback (most recent call last): 2024-09-10T19:05:27.352 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Sep 10 19:05:27 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-y[29989]: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-10T19:05:27.352 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Sep 10 19:05:27 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-y[29989]: self._connections.run(self.expiration_interval) 2024-09-10T19:05:27.353 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Sep 10 19:05:27 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-y[29989]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-10T19:05:27.353 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Sep 10 19:05:27 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-y[29989]: self._run(expiration_interval) 2024-09-10T19:05:27.353 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Sep 10 19:05:27 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-y[29989]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-10T19:05:27.353 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Sep 10 19:05:27 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-y[29989]: new_conn = self._from_server_socket(self.server.socket) 2024-09-10T19:05:27.353 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Sep 10 19:05:27 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-y[29989]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-10T19:05:27.353 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Sep 10 19:05:27 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-y[29989]: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-10T19:05:27.353 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Sep 10 19:05:27 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-y[29989]: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-10T19:05:27.353 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Sep 10 19:05:27 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-y[29989]: s = self.context.wrap_socket( 2024-09-10T19:05:27.353 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Sep 10 19:05:27 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-y[29989]: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-10T19:05:27.353 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Sep 10 19:05:27 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-y[29989]: return self.sslsocket_class._create( 2024-09-10T19:05:27.353 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Sep 10 19:05:27 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-y[29989]: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-10T19:05:27.354 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Sep 10 19:05:27 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-y[29989]: self.do_handshake() 2024-09-10T19:05:27.354 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Sep 10 19:05:27 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-y[29989]: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-10T19:05:27.354 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Sep 10 19:05:27 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-y[29989]: self._sslobj.do_handshake() 2024-09-10T19:05:27.354 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Sep 10 19:05:27 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-y[29989]: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-10T19:05:27.354 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Sep 10 19:05:27 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-y[29989]: 2024-09-10T19:05:27.355 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout { 2024-09-10T19:05:27.355 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 7, 2024-09-10T19:05:27.355 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout "initialized": true 2024-09-10T19:05:27.355 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout } 2024-09-10T19:05:27.355 INFO:teuthology.orchestra.run.smithi183.stdout:mgr epoch 5 is available 2024-09-10T19:05:27.355 INFO:teuthology.orchestra.run.smithi183.stdout:Setting orchestrator backend to cephadm... 2024-09-10T19:05:28.265 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:05:28 smithi183 ceph-mon[29736]: from='client.14122 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-09-10T19:05:28.265 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:05:28 smithi183 ceph-mon[29736]: from='client.14122 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-09-10T19:05:28.265 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:05:28 smithi183 ceph-mon[29736]: [10/Sep/2024:19:05:27] ENGINE Bus STARTING 2024-09-10T19:05:28.265 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:05:28 smithi183 ceph-mon[29736]: [10/Sep/2024:19:05:27] ENGINE Serving on https://172.21.15.183:7150 2024-09-10T19:05:28.265 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:05:28 smithi183 ceph-mon[29736]: [10/Sep/2024:19:05:27] ENGINE Bus STARTED 2024-09-10T19:05:28.265 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:05:28 smithi183 ceph-mon[29736]: [10/Sep/2024:19:05:27] ENGINE Error in HTTPServer.serve 2024-09-10T19:05:28.265 INFO:journalctl@ceph.mon.a.smithi183.stdout: Traceback (most recent call last): 2024-09-10T19:05:28.266 INFO:journalctl@ceph.mon.a.smithi183.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-10T19:05:28.266 INFO:journalctl@ceph.mon.a.smithi183.stdout: self._connections.run(self.expiration_interval) 2024-09-10T19:05:28.266 INFO:journalctl@ceph.mon.a.smithi183.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-10T19:05:28.266 INFO:journalctl@ceph.mon.a.smithi183.stdout: self._run(expiration_interval) 2024-09-10T19:05:28.266 INFO:journalctl@ceph.mon.a.smithi183.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-10T19:05:28.266 INFO:journalctl@ceph.mon.a.smithi183.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-09-10T19:05:28.266 INFO:journalctl@ceph.mon.a.smithi183.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-10T19:05:28.266 INFO:journalctl@ceph.mon.a.smithi183.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-10T19:05:28.266 INFO:journalctl@ceph.mon.a.smithi183.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-10T19:05:28.266 INFO:journalctl@ceph.mon.a.smithi183.stdout: s = self.context.wrap_socket( 2024-09-10T19:05:28.266 INFO:journalctl@ceph.mon.a.smithi183.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-10T19:05:28.266 INFO:journalctl@ceph.mon.a.smithi183.stdout: return self.sslsocket_class._create( 2024-09-10T19:05:28.266 INFO:journalctl@ceph.mon.a.smithi183.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-10T19:05:28.266 INFO:journalctl@ceph.mon.a.smithi183.stdout: self.do_handshake() 2024-09-10T19:05:28.267 INFO:journalctl@ceph.mon.a.smithi183.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-10T19:05:28.267 INFO:journalctl@ceph.mon.a.smithi183.stdout: self._sslobj.do_handshake() 2024-09-10T19:05:28.267 INFO:journalctl@ceph.mon.a.smithi183.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-10T19:05:28.267 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:05:28 smithi183 ceph-mon[29736]: from='mgr.14118 172.21.15.183:0/243006257' entity='mgr.y' 2024-09-10T19:05:28.267 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:05:28 smithi183 ceph-mon[29736]: from='mgr.14118 172.21.15.183:0/243006257' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T19:05:28.267 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:05:28 smithi183 ceph-mon[29736]: from='mgr.14118 172.21.15.183:0/243006257' entity='mgr.y' 2024-09-10T19:05:28.267 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:05:28 smithi183 ceph-mon[29736]: from='mgr.14118 172.21.15.183:0/243006257' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T19:05:29.165 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:05:29 smithi183 ceph-mon[29736]: from='client.14130 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2024-09-10T19:05:29.165 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:05:29 smithi183 ceph-mon[29736]: mgrmap e8: y(active, since 2s) 2024-09-10T19:05:29.668 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout value unchanged 2024-09-10T19:05:29.669 INFO:teuthology.orchestra.run.smithi183.stdout:Generating ssh key... 2024-09-10T19:05:30.362 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:05:30 smithi183 ceph-mon[29736]: from='client.14132 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2024-09-10T19:05:30.652 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Sep 10 19:05:30 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-y[29989]: Generating public/private rsa key pair. 2024-09-10T19:05:30.653 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Sep 10 19:05:30 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-y[29989]: Your identification has been saved in /tmp/tmpapeeo5y8/key 2024-09-10T19:05:30.653 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Sep 10 19:05:30 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-y[29989]: Your public key has been saved in /tmp/tmpapeeo5y8/key.pub 2024-09-10T19:05:30.653 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Sep 10 19:05:30 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-y[29989]: The key fingerprint is: 2024-09-10T19:05:30.653 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Sep 10 19:05:30 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-y[29989]: SHA256:trs4OIrdhxPcca4oWCt7DdvFzIZL6V8x+3f3ZuTQ7DA ceph-320f6df8-6fa7-11ef-bcea-c7b262605968 2024-09-10T19:05:30.653 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Sep 10 19:05:30 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-y[29989]: The key's randomart image is: 2024-09-10T19:05:30.653 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Sep 10 19:05:30 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-y[29989]: +---[RSA 3072]----+ 2024-09-10T19:05:30.653 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Sep 10 19:05:30 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-y[29989]: | | 2024-09-10T19:05:30.653 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Sep 10 19:05:30 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-y[29989]: | | 2024-09-10T19:05:30.653 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Sep 10 19:05:30 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-y[29989]: | | 2024-09-10T19:05:30.653 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Sep 10 19:05:30 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-y[29989]: | . . | 2024-09-10T19:05:30.653 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Sep 10 19:05:30 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-y[29989]: | . B +S o | 2024-09-10T19:05:30.653 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Sep 10 19:05:30 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-y[29989]: | o * B..= E +| 2024-09-10T19:05:30.653 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Sep 10 19:05:30 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-y[29989]: | o O O .+ B | 2024-09-10T19:05:30.653 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Sep 10 19:05:30 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-y[29989]: |oo=.@ +o o . . *| 2024-09-10T19:05:30.654 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Sep 10 19:05:30 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-y[29989]: |o+oo.=o.o... . +o| 2024-09-10T19:05:30.654 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Sep 10 19:05:30 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-y[29989]: +----[SHA256]-----+ 2024-09-10T19:05:31.594 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:05:31 smithi183 ceph-mon[29736]: from='client.14134 -' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2024-09-10T19:05:31.594 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:05:31 smithi183 ceph-mon[29736]: Generating ssh key... 2024-09-10T19:05:31.594 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:05:31 smithi183 ceph-mon[29736]: from='mgr.14118 172.21.15.183:0/243006257' entity='mgr.y' 2024-09-10T19:05:31.594 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:05:31 smithi183 ceph-mon[29736]: from='mgr.14118 172.21.15.183:0/243006257' entity='mgr.y' 2024-09-10T19:05:32.169 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDDCDuybm9NlW8u+FnMe9qOmlWItt6cpklNVQB8Bm/EoJp7GTnzGWY3IGtFW5IChb2wAuYl/jWWzfeZfO1zwhnaiBsJoRIP3NZK+1tm9QaXs+NOcK57sRthsUebOHk6NB6vaVPdv1nM5O09COnXq2gtkF2pw9hZ58W4olLejnvHCPXsxEM+/o8KXlcyZHop4eJRRfmMG8ccr9O5Nv2HpWvnAfDLXtzoMeAAdnx/XdQCMwYkxLdbrQc76WQsxkEZ+BRc6WOLpW9459YF0EcrTnA+//WK+ZiZ5WnM7RsewVgKta1Xi30TZqgTk+fnu2dW5jcOYP0PmzoYZhP9tJVJYA44KROdMIcLLzw1ehkDyz/Xy0iAxeI/x/HOlDfSwi+3kH8MtcDqKqv7UkGKLbE/MD9E/t68bJ8TqA74lZSs136z9DCSyF0PTOLBRvjNOAxALl1nhwO+COBGBod13oZzV3JjLNaJPP+2d9sDNYoLZ6GNRwdsm/u90k0L3vJdlUZ9uvc= ceph-320f6df8-6fa7-11ef-bcea-c7b262605968 2024-09-10T19:05:32.170 INFO:teuthology.orchestra.run.smithi183.stdout:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2024-09-10T19:05:32.170 INFO:teuthology.orchestra.run.smithi183.stdout:Adding key to root@localhost authorized_keys... 2024-09-10T19:05:32.177 INFO:teuthology.orchestra.run.smithi183.stdout:Adding host smithi183... 2024-09-10T19:05:32.681 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:05:32 smithi183 ceph-mon[29736]: from='client.14136 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2024-09-10T19:05:33.560 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:05:33 smithi183 ceph-mon[29736]: from='client.14138 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi183", "addr": "172.21.15.183", "target": ["mon-mgr", ""]}]: dispatch 2024-09-10T19:05:34.502 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:05:34 smithi183 ceph-mon[29736]: Deploying cephadm binary to smithi183 2024-09-10T19:05:36.010 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout Added host 'smithi183' with addr '172.21.15.183' 2024-09-10T19:05:36.010 INFO:teuthology.orchestra.run.smithi183.stdout:Deploying unmanaged mon service... 2024-09-10T19:05:36.673 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:05:36 smithi183 ceph-mon[29736]: from='mgr.14118 172.21.15.183:0/243006257' entity='mgr.y' 2024-09-10T19:05:36.673 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:05:36 smithi183 ceph-mon[29736]: Added host smithi183 2024-09-10T19:05:36.673 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:05:36 smithi183 ceph-mon[29736]: from='mgr.14118 172.21.15.183:0/243006257' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T19:05:37.210 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout Scheduled mon update... 2024-09-10T19:05:37.210 INFO:teuthology.orchestra.run.smithi183.stdout:Deploying unmanaged mgr service... 2024-09-10T19:05:37.931 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:05:37 smithi183 ceph-mon[29736]: from='client.14140 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-09-10T19:05:37.931 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:05:37 smithi183 ceph-mon[29736]: Saving service mon spec with placement count:5 2024-09-10T19:05:37.931 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:05:37 smithi183 ceph-mon[29736]: from='mgr.14118 172.21.15.183:0/243006257' entity='mgr.y' 2024-09-10T19:05:39.372 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout Scheduled mgr update... 2024-09-10T19:05:39.681 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:05:39 smithi183 ceph-mon[29736]: from='mgr.14118 172.21.15.183:0/243006257' entity='mgr.y' 2024-09-10T19:05:39.681 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:05:39 smithi183 ceph-mon[29736]: from='mgr.14118 172.21.15.183:0/243006257' entity='mgr.y' 2024-09-10T19:05:39.681 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:05:39 smithi183 ceph-mon[29736]: from='client.14142 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-09-10T19:05:39.682 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:05:39 smithi183 ceph-mon[29736]: Saving service mgr spec with placement count:2 2024-09-10T19:05:39.682 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:05:39 smithi183 ceph-mon[29736]: from='mgr.14118 172.21.15.183:0/243006257' entity='mgr.y' 2024-09-10T19:05:41.237 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:05:40 smithi183 ceph-mon[29736]: from='client.? 172.21.15.183:0/3521556912' entity='client.admin' 2024-09-10T19:05:41.237 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:05:40 smithi183 ceph-mon[29736]: from='mgr.14118 172.21.15.183:0/243006257' entity='mgr.y' 2024-09-10T19:05:42.429 INFO:teuthology.orchestra.run.smithi183.stdout:Enabling the dashboard module... 2024-09-10T19:05:43.080 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:05:42 smithi183 ceph-mon[29736]: from='mgr.14118 172.21.15.183:0/243006257' entity='mgr.y' 2024-09-10T19:05:43.080 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:05:42 smithi183 ceph-mon[29736]: from='client.? 172.21.15.183:0/81218880' entity='client.admin' 2024-09-10T19:05:43.992 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:05:43 smithi183 ceph-mon[29736]: from='client.? 172.21.15.183:0/1855493826' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2024-09-10T19:05:43.993 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Sep 10 19:05:43 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-y[29989]: ignoring --setuser ceph since I am not root 2024-09-10T19:05:43.993 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Sep 10 19:05:43 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-y[29989]: ignoring --setgroup ceph since I am not root 2024-09-10T19:05:43.993 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Sep 10 19:05:43 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-y[29989]: 2024-09-10T19:05:43.872+0000 7ff7eebecec0 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-09-10T19:05:43.993 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Sep 10 19:05:43 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-y[29989]: 2024-09-10T19:05:43.926+0000 7ff7eebecec0 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-09-10T19:05:43.993 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Sep 10 19:05:43 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-y[29989]: 2024-09-10T19:05:43.990+0000 7ff7eebecec0 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-09-10T19:05:44.251 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Sep 10 19:05:44 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-y[29989]: 2024-09-10T19:05:44.113+0000 7ff7eebecec0 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-09-10T19:05:44.252 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Sep 10 19:05:44 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-y[29989]: 2024-09-10T19:05:44.248+0000 7ff7eebecec0 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-09-10T19:05:44.789 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:05:44 smithi183 ceph-mon[29736]: from='client.? 172.21.15.183:0/1855493826' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2024-09-10T19:05:44.789 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:05:44 smithi183 ceph-mon[29736]: mgrmap e9: y(active, since 18s) 2024-09-10T19:05:44.790 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Sep 10 19:05:44 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-y[29989]: 2024-09-10T19:05:44.502+0000 7ff7eebecec0 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-09-10T19:05:44.790 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Sep 10 19:05:44 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-y[29989]: 2024-09-10T19:05:44.562+0000 7ff7eebecec0 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-09-10T19:05:44.790 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Sep 10 19:05:44 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-y[29989]: 2024-09-10T19:05:44.724+0000 7ff7eebecec0 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-09-10T19:05:45.734 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Sep 10 19:05:45 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-y[29989]: 2024-09-10T19:05:45.613+0000 7ff7eebecec0 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-09-10T19:05:45.734 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Sep 10 19:05:45 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-y[29989]: 2024-09-10T19:05:45.683+0000 7ff7eebecec0 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-09-10T19:05:45.996 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:05:45 smithi183 ceph-mon[29736]: from='client.? 172.21.15.183:0/953008174' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-09-10T19:05:45.999 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout { 2024-09-10T19:05:46.000 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout "epoch": 9, 2024-09-10T19:05:46.000 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout "available": true, 2024-09-10T19:05:46.000 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout "active_name": "y", 2024-09-10T19:05:46.000 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-09-10T19:05:46.000 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout } 2024-09-10T19:05:46.000 INFO:teuthology.orchestra.run.smithi183.stdout:Waiting for the mgr to restart... 2024-09-10T19:05:46.000 INFO:teuthology.orchestra.run.smithi183.stdout:Waiting for mgr epoch 9... 2024-09-10T19:05:46.506 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Sep 10 19:05:46 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-y[29989]: 2024-09-10T19:05:46.422+0000 7ff7eebecec0 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-09-10T19:05:46.840 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Sep 10 19:05:46 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-y[29989]: 2024-09-10T19:05:46.537+0000 7ff7eebecec0 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-09-10T19:05:46.840 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Sep 10 19:05:46 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-y[29989]: 2024-09-10T19:05:46.654+0000 7ff7eebecec0 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-09-10T19:05:46.841 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Sep 10 19:05:46 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-y[29989]: 2024-09-10T19:05:46.714+0000 7ff7eebecec0 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-09-10T19:05:46.841 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Sep 10 19:05:46 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-y[29989]: 2024-09-10T19:05:46.838+0000 7ff7eebecec0 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-09-10T19:05:47.095 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Sep 10 19:05:46 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-y[29989]: 2024-09-10T19:05:46.905+0000 7ff7eebecec0 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-09-10T19:05:47.095 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Sep 10 19:05:46 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-y[29989]: 2024-09-10T19:05:46.962+0000 7ff7eebecec0 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-09-10T19:05:47.095 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Sep 10 19:05:47 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-y[29989]: 2024-09-10T19:05:47.093+0000 7ff7eebecec0 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-09-10T19:05:47.360 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Sep 10 19:05:47 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-y[29989]: 2024-09-10T19:05:47.157+0000 7ff7eebecec0 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-09-10T19:05:47.360 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Sep 10 19:05:47 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-y[29989]: 2024-09-10T19:05:47.221+0000 7ff7eebecec0 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-09-10T19:05:47.360 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Sep 10 19:05:47 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-y[29989]: 2024-09-10T19:05:47.300+0000 7ff7eebecec0 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-09-10T19:05:47.360 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Sep 10 19:05:47 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-y[29989]: 2024-09-10T19:05:47.358+0000 7ff7eebecec0 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-09-10T19:05:47.681 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Sep 10 19:05:47 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-y[29989]: 2024-09-10T19:05:47.415+0000 7ff7eebecec0 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-09-10T19:05:48.431 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Sep 10 19:05:48 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-y[29989]: 2024-09-10T19:05:48.151+0000 7ff7eebecec0 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-09-10T19:05:48.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:05:48 smithi183 ceph-mon[29736]: Active manager daemon y restarted 2024-09-10T19:05:48.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:05:48 smithi183 ceph-mon[29736]: Activating manager daemon y 2024-09-10T19:05:48.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:05:48 smithi183 ceph-mon[29736]: osdmap e3: 0 total, 0 up, 0 in 2024-09-10T19:05:48.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:05:48 smithi183 ceph-mon[29736]: mgrmap e10: y(active, starting, since 0.00307085s) 2024-09-10T19:05:48.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:05:48 smithi183 ceph-mon[29736]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-10T19:05:48.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:05:48 smithi183 ceph-mon[29736]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-09-10T19:05:48.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:05:48 smithi183 ceph-mon[29736]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-09-10T19:05:48.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:05:48 smithi183 ceph-mon[29736]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-09-10T19:05:48.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:05:48 smithi183 ceph-mon[29736]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-09-10T19:05:48.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:05:48 smithi183 ceph-mon[29736]: Manager daemon y is now available 2024-09-10T19:05:48.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:05:48 smithi183 ceph-mon[29736]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T19:05:48.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:05:48 smithi183 ceph-mon[29736]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-09-10T19:05:48.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:05:48 smithi183 ceph-mon[29736]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-09-10T19:05:49.403 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Sep 10 19:05:49 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-y[29989]: 2024-09-10T19:05:49.229+0000 7ff79f749640 -1 log_channel(cephadm) log [ERR] : [10/Sep/2024:19:05:49] ENGINE Error in HTTPServer.serve 2024-09-10T19:05:49.403 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Sep 10 19:05:49 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-y[29989]: Traceback (most recent call last): 2024-09-10T19:05:49.403 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Sep 10 19:05:49 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-y[29989]: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-10T19:05:49.403 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Sep 10 19:05:49 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-y[29989]: self._connections.run(self.expiration_interval) 2024-09-10T19:05:49.403 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Sep 10 19:05:49 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-y[29989]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-10T19:05:49.403 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Sep 10 19:05:49 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-y[29989]: self._run(expiration_interval) 2024-09-10T19:05:49.404 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Sep 10 19:05:49 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-y[29989]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-10T19:05:49.404 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Sep 10 19:05:49 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-y[29989]: new_conn = self._from_server_socket(self.server.socket) 2024-09-10T19:05:49.404 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Sep 10 19:05:49 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-y[29989]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-10T19:05:49.404 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Sep 10 19:05:49 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-y[29989]: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-10T19:05:49.404 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Sep 10 19:05:49 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-y[29989]: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-10T19:05:49.404 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Sep 10 19:05:49 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-y[29989]: s = self.context.wrap_socket( 2024-09-10T19:05:49.404 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Sep 10 19:05:49 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-y[29989]: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-10T19:05:49.404 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Sep 10 19:05:49 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-y[29989]: return self.sslsocket_class._create( 2024-09-10T19:05:49.404 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Sep 10 19:05:49 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-y[29989]: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-10T19:05:49.404 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Sep 10 19:05:49 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-y[29989]: self.do_handshake() 2024-09-10T19:05:49.404 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Sep 10 19:05:49 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-y[29989]: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-10T19:05:49.404 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Sep 10 19:05:49 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-y[29989]: self._sslobj.do_handshake() 2024-09-10T19:05:49.404 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Sep 10 19:05:49 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-y[29989]: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-10T19:05:49.405 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Sep 10 19:05:49 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-y[29989]: 2024-09-10T19:05:49.883 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout { 2024-09-10T19:05:49.883 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 11, 2024-09-10T19:05:49.883 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout "initialized": true 2024-09-10T19:05:49.883 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout } 2024-09-10T19:05:49.883 INFO:teuthology.orchestra.run.smithi183.stdout:mgr epoch 9 is available 2024-09-10T19:05:49.883 INFO:teuthology.orchestra.run.smithi183.stdout:Generating a dashboard self-signed certificate... 2024-09-10T19:05:50.317 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:05:50 smithi183 ceph-mon[29736]: [10/Sep/2024:19:05:49] ENGINE Bus STARTING 2024-09-10T19:05:50.317 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:05:50 smithi183 ceph-mon[29736]: from='client.14154 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-09-10T19:05:50.317 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:05:50 smithi183 ceph-mon[29736]: mgrmap e11: y(active, since 1.00558s) 2024-09-10T19:05:50.317 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:05:50 smithi183 ceph-mon[29736]: from='client.14154 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-09-10T19:05:50.317 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:05:50 smithi183 ceph-mon[29736]: [10/Sep/2024:19:05:49] ENGINE Serving on https://172.21.15.183:7150 2024-09-10T19:05:50.317 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:05:50 smithi183 ceph-mon[29736]: [10/Sep/2024:19:05:49] ENGINE Bus STARTED 2024-09-10T19:05:50.317 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:05:50 smithi183 ceph-mon[29736]: [10/Sep/2024:19:05:49] ENGINE Error in HTTPServer.serve 2024-09-10T19:05:50.317 INFO:journalctl@ceph.mon.a.smithi183.stdout: Traceback (most recent call last): 2024-09-10T19:05:50.317 INFO:journalctl@ceph.mon.a.smithi183.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-10T19:05:50.318 INFO:journalctl@ceph.mon.a.smithi183.stdout: self._connections.run(self.expiration_interval) 2024-09-10T19:05:50.318 INFO:journalctl@ceph.mon.a.smithi183.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-10T19:05:50.318 INFO:journalctl@ceph.mon.a.smithi183.stdout: self._run(expiration_interval) 2024-09-10T19:05:50.318 INFO:journalctl@ceph.mon.a.smithi183.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-10T19:05:50.318 INFO:journalctl@ceph.mon.a.smithi183.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-09-10T19:05:50.318 INFO:journalctl@ceph.mon.a.smithi183.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-10T19:05:50.318 INFO:journalctl@ceph.mon.a.smithi183.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-10T19:05:50.318 INFO:journalctl@ceph.mon.a.smithi183.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-10T19:05:50.318 INFO:journalctl@ceph.mon.a.smithi183.stdout: s = self.context.wrap_socket( 2024-09-10T19:05:50.318 INFO:journalctl@ceph.mon.a.smithi183.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-10T19:05:50.318 INFO:journalctl@ceph.mon.a.smithi183.stdout: return self.sslsocket_class._create( 2024-09-10T19:05:50.318 INFO:journalctl@ceph.mon.a.smithi183.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-10T19:05:50.318 INFO:journalctl@ceph.mon.a.smithi183.stdout: self.do_handshake() 2024-09-10T19:05:50.319 INFO:journalctl@ceph.mon.a.smithi183.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-10T19:05:50.319 INFO:journalctl@ceph.mon.a.smithi183.stdout: self._sslobj.do_handshake() 2024-09-10T19:05:50.319 INFO:journalctl@ceph.mon.a.smithi183.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-10T19:05:50.319 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:05:50 smithi183 ceph-mon[29736]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:05:50.319 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:05:50 smithi183 ceph-mon[29736]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:05:51.305 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout Self-signed certificate created 2024-09-10T19:05:51.305 INFO:teuthology.orchestra.run.smithi183.stdout:Creating initial admin user... 2024-09-10T19:05:51.614 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:05:51 smithi183 ceph-mon[29736]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:05:51.614 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:05:51 smithi183 ceph-mon[29736]: mgrmap e12: y(active, since 2s) 2024-09-10T19:05:51.614 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:05:51 smithi183 ceph-mon[29736]: from='client.14162 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2024-09-10T19:05:51.614 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:05:51 smithi183 ceph-mon[29736]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:05:51.614 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:05:51 smithi183 ceph-mon[29736]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:05:53.217 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout {"username": "admin", "password": "$2b$12$s9LyQLvNUuzwj1Z2fwG8nu9ukykC5bbh2ip3clv0LxdiX2CP9lZ3m", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1725995152, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2024-09-10T19:05:53.217 INFO:teuthology.orchestra.run.smithi183.stdout:Fetching dashboard port number... 2024-09-10T19:05:53.641 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:05:53 smithi183 ceph-mon[29736]: from='client.14164 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2024-09-10T19:05:53.642 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:05:53 smithi183 ceph-mon[29736]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:05:54.422 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stdout 8443 2024-09-10T19:05:54.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:05:54 smithi183 ceph-mon[29736]: from='client.? 172.21.15.183:0/3786468928' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2024-09-10T19:05:54.432 INFO:teuthology.orchestra.run.smithi183.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-09-10T19:05:54.432 INFO:teuthology.orchestra.run.smithi183.stdout:systemctl: stdout disabled 2024-09-10T19:05:54.440 INFO:teuthology.orchestra.run.smithi183.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-09-10T19:05:54.440 INFO:teuthology.orchestra.run.smithi183.stdout:systemctl: stdout inactive 2024-09-10T19:05:54.440 INFO:teuthology.orchestra.run.smithi183.stdout:firewalld.service is not enabled 2024-09-10T19:05:54.440 INFO:teuthology.orchestra.run.smithi183.stdout:Not possible to open ports <[8443]>. firewalld.service is not available 2024-09-10T19:05:54.442 INFO:teuthology.orchestra.run.smithi183.stdout:Ceph Dashboard is now available at: 2024-09-10T19:05:54.442 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-09-10T19:05:54.443 INFO:teuthology.orchestra.run.smithi183.stdout: URL: https://smithi183.front.sepia.ceph.com:8443/ 2024-09-10T19:05:54.443 INFO:teuthology.orchestra.run.smithi183.stdout: User: admin 2024-09-10T19:05:54.443 INFO:teuthology.orchestra.run.smithi183.stdout: Password: sr589fonl9 2024-09-10T19:05:54.443 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-09-10T19:05:54.443 INFO:teuthology.orchestra.run.smithi183.stdout:Saving cluster configuration to /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/config directory 2024-09-10T19:05:54.443 INFO:teuthology.orchestra.run.smithi183.stdout:Enabling autotune for osd_memory_target 2024-09-10T19:05:55.668 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:05:55 smithi183 ceph-mon[29736]: mgrmap e13: y(active, since 6s) 2024-09-10T19:05:56.801 INFO:teuthology.orchestra.run.smithi183.stdout:/usr/bin/ceph: stderr set mgr/dashboard/cluster/status 2024-09-10T19:05:56.801 INFO:teuthology.orchestra.run.smithi183.stdout:You can access the Ceph CLI as following in case of multi-cluster or non-default config: 2024-09-10T19:05:56.801 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-09-10T19:05:56.801 INFO:teuthology.orchestra.run.smithi183.stdout: sudo /home/ubuntu/cephtest/cephadm shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2024-09-10T19:05:56.801 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-09-10T19:05:56.801 INFO:teuthology.orchestra.run.smithi183.stdout:Or, if you are only running a single cluster on this host: 2024-09-10T19:05:56.801 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-09-10T19:05:56.801 INFO:teuthology.orchestra.run.smithi183.stdout: sudo /home/ubuntu/cephtest/cephadm shell 2024-09-10T19:05:56.802 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-09-10T19:05:56.802 INFO:teuthology.orchestra.run.smithi183.stdout:Please consider enabling telemetry to help improve Ceph: 2024-09-10T19:05:56.802 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-09-10T19:05:56.802 INFO:teuthology.orchestra.run.smithi183.stdout: ceph telemetry on 2024-09-10T19:05:56.802 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-09-10T19:05:56.802 INFO:teuthology.orchestra.run.smithi183.stdout:For more information see: 2024-09-10T19:05:56.802 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-09-10T19:05:56.802 INFO:teuthology.orchestra.run.smithi183.stdout: https://docs.ceph.com/docs/master/mgr/telemetry/ 2024-09-10T19:05:56.802 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-09-10T19:05:56.802 INFO:teuthology.orchestra.run.smithi183.stdout:Bootstrap complete. 2024-09-10T19:05:56.835 INFO:tasks.cephadm:Fetching config... 2024-09-10T19:05:56.835 DEBUG:teuthology.orchestra.run.smithi183:> set -ex 2024-09-10T19:05:56.835 DEBUG:teuthology.orchestra.run.smithi183:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2024-09-10T19:05:56.851 INFO:tasks.cephadm:Fetching client.admin keyring... 2024-09-10T19:05:56.851 DEBUG:teuthology.orchestra.run.smithi183:> set -ex 2024-09-10T19:05:56.851 DEBUG:teuthology.orchestra.run.smithi183:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2024-09-10T19:05:56.907 INFO:tasks.cephadm:Fetching mon keyring... 2024-09-10T19:05:56.907 DEBUG:teuthology.orchestra.run.smithi183:> set -ex 2024-09-10T19:05:56.908 DEBUG:teuthology.orchestra.run.smithi183:> sudo dd if=/var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.a/keyring of=/dev/stdout 2024-09-10T19:05:56.971 INFO:tasks.cephadm:Fetching pub ssh key... 2024-09-10T19:05:56.971 DEBUG:teuthology.orchestra.run.smithi183:> set -ex 2024-09-10T19:05:56.972 DEBUG:teuthology.orchestra.run.smithi183:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2024-09-10T19:05:57.026 INFO:tasks.cephadm:Installing pub ssh key for root users... 2024-09-10T19:05:57.027 DEBUG:teuthology.orchestra.run.smithi183:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDDCDuybm9NlW8u+FnMe9qOmlWItt6cpklNVQB8Bm/EoJp7GTnzGWY3IGtFW5IChb2wAuYl/jWWzfeZfO1zwhnaiBsJoRIP3NZK+1tm9QaXs+NOcK57sRthsUebOHk6NB6vaVPdv1nM5O09COnXq2gtkF2pw9hZ58W4olLejnvHCPXsxEM+/o8KXlcyZHop4eJRRfmMG8ccr9O5Nv2HpWvnAfDLXtzoMeAAdnx/XdQCMwYkxLdbrQc76WQsxkEZ+BRc6WOLpW9459YF0EcrTnA+//WK+ZiZ5WnM7RsewVgKta1Xi30TZqgTk+fnu2dW5jcOYP0PmzoYZhP9tJVJYA44KROdMIcLLzw1ehkDyz/Xy0iAxeI/x/HOlDfSwi+3kH8MtcDqKqv7UkGKLbE/MD9E/t68bJ8TqA74lZSs136z9DCSyF0PTOLBRvjNOAxALl1nhwO+COBGBod13oZzV3JjLNaJPP+2d9sDNYoLZ6GNRwdsm/u90k0L3vJdlUZ9uvc= ceph-320f6df8-6fa7-11ef-bcea-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-09-10T19:05:57.101 INFO:teuthology.orchestra.run.smithi183.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDDCDuybm9NlW8u+FnMe9qOmlWItt6cpklNVQB8Bm/EoJp7GTnzGWY3IGtFW5IChb2wAuYl/jWWzfeZfO1zwhnaiBsJoRIP3NZK+1tm9QaXs+NOcK57sRthsUebOHk6NB6vaVPdv1nM5O09COnXq2gtkF2pw9hZ58W4olLejnvHCPXsxEM+/o8KXlcyZHop4eJRRfmMG8ccr9O5Nv2HpWvnAfDLXtzoMeAAdnx/XdQCMwYkxLdbrQc76WQsxkEZ+BRc6WOLpW9459YF0EcrTnA+//WK+ZiZ5WnM7RsewVgKta1Xi30TZqgTk+fnu2dW5jcOYP0PmzoYZhP9tJVJYA44KROdMIcLLzw1ehkDyz/Xy0iAxeI/x/HOlDfSwi+3kH8MtcDqKqv7UkGKLbE/MD9E/t68bJ8TqA74lZSs136z9DCSyF0PTOLBRvjNOAxALl1nhwO+COBGBod13oZzV3JjLNaJPP+2d9sDNYoLZ6GNRwdsm/u90k0L3vJdlUZ9uvc= ceph-320f6df8-6fa7-11ef-bcea-c7b262605968 2024-09-10T19:05:57.112 DEBUG:teuthology.orchestra.run.smithi191:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDDCDuybm9NlW8u+FnMe9qOmlWItt6cpklNVQB8Bm/EoJp7GTnzGWY3IGtFW5IChb2wAuYl/jWWzfeZfO1zwhnaiBsJoRIP3NZK+1tm9QaXs+NOcK57sRthsUebOHk6NB6vaVPdv1nM5O09COnXq2gtkF2pw9hZ58W4olLejnvHCPXsxEM+/o8KXlcyZHop4eJRRfmMG8ccr9O5Nv2HpWvnAfDLXtzoMeAAdnx/XdQCMwYkxLdbrQc76WQsxkEZ+BRc6WOLpW9459YF0EcrTnA+//WK+ZiZ5WnM7RsewVgKta1Xi30TZqgTk+fnu2dW5jcOYP0PmzoYZhP9tJVJYA44KROdMIcLLzw1ehkDyz/Xy0iAxeI/x/HOlDfSwi+3kH8MtcDqKqv7UkGKLbE/MD9E/t68bJ8TqA74lZSs136z9DCSyF0PTOLBRvjNOAxALl1nhwO+COBGBod13oZzV3JjLNaJPP+2d9sDNYoLZ6GNRwdsm/u90k0L3vJdlUZ9uvc= ceph-320f6df8-6fa7-11ef-bcea-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-09-10T19:05:57.162 INFO:teuthology.orchestra.run.smithi191.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDDCDuybm9NlW8u+FnMe9qOmlWItt6cpklNVQB8Bm/EoJp7GTnzGWY3IGtFW5IChb2wAuYl/jWWzfeZfO1zwhnaiBsJoRIP3NZK+1tm9QaXs+NOcK57sRthsUebOHk6NB6vaVPdv1nM5O09COnXq2gtkF2pw9hZ58W4olLejnvHCPXsxEM+/o8KXlcyZHop4eJRRfmMG8ccr9O5Nv2HpWvnAfDLXtzoMeAAdnx/XdQCMwYkxLdbrQc76WQsxkEZ+BRc6WOLpW9459YF0EcrTnA+//WK+ZiZ5WnM7RsewVgKta1Xi30TZqgTk+fnu2dW5jcOYP0PmzoYZhP9tJVJYA44KROdMIcLLzw1ehkDyz/Xy0iAxeI/x/HOlDfSwi+3kH8MtcDqKqv7UkGKLbE/MD9E/t68bJ8TqA74lZSs136z9DCSyF0PTOLBRvjNOAxALl1nhwO+COBGBod13oZzV3JjLNaJPP+2d9sDNYoLZ6GNRwdsm/u90k0L3vJdlUZ9uvc= ceph-320f6df8-6fa7-11ef-bcea-c7b262605968 2024-09-10T19:05:57.175 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2024-09-10T19:05:57.351 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:05:57 smithi183 ceph-mon[29736]: from='client.? 172.21.15.183:0/637071036' entity='client.admin' 2024-09-10T19:05:57.417 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:05:59.759 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:05:59 smithi183 ceph-mon[29736]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:05:59.760 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:05:59 smithi183 ceph-mon[29736]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:05:59.760 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:05:59 smithi183 ceph-mon[29736]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi183", "name": "osd_memory_target"}]: dispatch 2024-09-10T19:05:59.760 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:05:59 smithi183 ceph-mon[29736]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:05:59.760 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:05:59 smithi183 ceph-mon[29736]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T19:05:59.760 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:05:59 smithi183 ceph-mon[29736]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:06:00.422 INFO:tasks.cephadm:Distributing conf and client.admin keyring to all hosts + 0755 2024-09-10T19:06:00.422 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph orch client-keyring set client.admin '*' --mode 0755 2024-09-10T19:06:00.742 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:06:00.931 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:06:00 smithi183 ceph-mon[29736]: from='client.? 172.21.15.183:0/921963443' entity='client.admin' 2024-09-10T19:06:00.931 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:06:00 smithi183 ceph-mon[29736]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T19:06:00.931 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:06:00 smithi183 ceph-mon[29736]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:06:03.816 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi191 2024-09-10T19:06:03.817 DEBUG:teuthology.orchestra.run.smithi191:> set -ex 2024-09-10T19:06:03.817 DEBUG:teuthology.orchestra.run.smithi191:> dd of=/etc/ceph/ceph.conf 2024-09-10T19:06:03.837 DEBUG:teuthology.orchestra.run.smithi191:> set -ex 2024-09-10T19:06:03.837 DEBUG:teuthology.orchestra.run.smithi191:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-09-10T19:06:03.895 INFO:tasks.cephadm:Adding host smithi191 to orchestrator... 2024-09-10T19:06:03.895 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph orch host add smithi191 2024-09-10T19:06:04.136 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:06:04.389 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:06:04 smithi183 ceph-mon[29736]: from='client.14174 -' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2024-09-10T19:06:04.389 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:06:04 smithi183 ceph-mon[29736]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:06:04.389 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:06:04 smithi183 ceph-mon[29736]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T19:06:04.389 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:06:04 smithi183 ceph-mon[29736]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T19:06:04.389 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:06:04 smithi183 ceph-mon[29736]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T19:06:04.389 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:06:04 smithi183 ceph-mon[29736]: Updating smithi183:/etc/ceph/ceph.conf 2024-09-10T19:06:04.389 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:06:04 smithi183 ceph-mon[29736]: Updating smithi183:/var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/config/ceph.conf 2024-09-10T19:06:04.389 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:06:04 smithi183 ceph-mon[29736]: Updating smithi183:/etc/ceph/ceph.client.admin.keyring 2024-09-10T19:06:04.389 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:06:04 smithi183 ceph-mon[29736]: Updating smithi183:/var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/config/ceph.client.admin.keyring 2024-09-10T19:06:04.389 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:06:04 smithi183 ceph-mon[29736]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:06:04.389 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:06:04 smithi183 ceph-mon[29736]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:06:04.389 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:06:04 smithi183 ceph-mon[29736]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:06:07.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:06:06 smithi183 ceph-mon[29736]: from='client.14176 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi191", "target": ["mon-mgr", ""]}]: dispatch 2024-09-10T19:06:08.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:06:07 smithi183 ceph-mon[29736]: Deploying cephadm binary to smithi191 2024-09-10T19:06:08.575 INFO:teuthology.orchestra.run.smithi183.stdout:Added host 'smithi191' with addr '172.21.15.191' 2024-09-10T19:06:09.145 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph orch host ls --format=json 2024-09-10T19:06:09.381 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:06:09.770 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:06:09 smithi183 ceph-mon[29736]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T19:06:09.771 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:06:09 smithi183 ceph-mon[29736]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:06:09.771 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:06:09 smithi183 ceph-mon[29736]: Added host smithi191 2024-09-10T19:06:09.771 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:06:09 smithi183 ceph-mon[29736]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T19:06:09.771 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:06:09 smithi183 ceph-mon[29736]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:06:09.771 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:06:09 smithi183 ceph-mon[29736]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:06:11.122 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-09-10T19:06:11.123 INFO:teuthology.orchestra.run.smithi183.stdout:[{"addr": "172.21.15.183", "hostname": "smithi183", "labels": [], "status": ""}, {"addr": "172.21.15.191", "hostname": "smithi191", "labels": [], "status": ""}] 2024-09-10T19:06:11.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:06:10 smithi183 ceph-mon[29736]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:06:11.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:06:10 smithi183 ceph-mon[29736]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T19:06:11.688 INFO:tasks.cephadm:Setting crush tunables to default 2024-09-10T19:06:11.688 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph osd crush tunables default 2024-09-10T19:06:11.924 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:06:12.171 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:06:11 smithi183 ceph-mon[29736]: from='client.14178 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-10T19:06:13.931 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:06:13 smithi183 ceph-mon[29736]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T19:06:13.931 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:06:13 smithi183 ceph-mon[29736]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:06:13.931 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:06:13 smithi183 ceph-mon[29736]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:06:13.931 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:06:13 smithi183 ceph-mon[29736]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:06:13.932 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:06:13 smithi183 ceph-mon[29736]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:06:13.932 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:06:13 smithi183 ceph-mon[29736]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi191", "name": "osd_memory_target"}]: dispatch 2024-09-10T19:06:13.932 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:06:13 smithi183 ceph-mon[29736]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T19:06:13.932 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:06:13 smithi183 ceph-mon[29736]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T19:06:13.932 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:06:13 smithi183 ceph-mon[29736]: Updating smithi191:/etc/ceph/ceph.conf 2024-09-10T19:06:13.932 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:06:13 smithi183 ceph-mon[29736]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:06:13.932 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:06:13 smithi183 ceph-mon[29736]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:06:13.932 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:06:13 smithi183 ceph-mon[29736]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:06:14.691 INFO:teuthology.orchestra.run.smithi183.stderr:adjusted tunables profile to default 2024-09-10T19:06:14.931 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:06:14 smithi183 ceph-mon[29736]: Updating smithi191:/var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/config/ceph.conf 2024-09-10T19:06:14.931 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:06:14 smithi183 ceph-mon[29736]: Updating smithi191:/etc/ceph/ceph.client.admin.keyring 2024-09-10T19:06:14.931 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:06:14 smithi183 ceph-mon[29736]: Updating smithi191:/var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/config/ceph.client.admin.keyring 2024-09-10T19:06:14.931 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:06:14 smithi183 ceph-mon[29736]: from='client.? 172.21.15.183:0/4062954633' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-09-10T19:06:15.346 INFO:tasks.cephadm:Adding mon.a on smithi183 2024-09-10T19:06:15.346 INFO:tasks.cephadm:Adding mon.c on smithi183 2024-09-10T19:06:15.346 INFO:tasks.cephadm:Adding mon.b on smithi191 2024-09-10T19:06:15.346 DEBUG:teuthology.orchestra.run.smithi191:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph orch apply mon '3;smithi183:172.21.15.183=a;smithi183:[v2:172.21.15.183:3301,v1:172.21.15.183:6790]=c;smithi191:172.21.15.191=b' 2024-09-10T19:06:15.587 INFO:teuthology.orchestra.run.smithi191.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/config/ceph.conf 2024-09-10T19:06:16.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:06:15 smithi183 ceph-mon[29736]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T19:06:16.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:06:15 smithi183 ceph-mon[29736]: from='client.? 172.21.15.183:0/4062954633' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-09-10T19:06:16.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:06:15 smithi183 ceph-mon[29736]: osdmap e4: 0 total, 0 up, 0 in 2024-09-10T19:06:17.275 INFO:teuthology.orchestra.run.smithi191.stdout:Scheduled mon update... 2024-09-10T19:06:17.799 DEBUG:teuthology.orchestra.run.smithi183:mon.c> sudo journalctl -f -n 0 -u ceph-320f6df8-6fa7-11ef-bcea-c7b262605968@mon.c.service 2024-09-10T19:06:17.803 DEBUG:teuthology.orchestra.run.smithi191:mon.b> sudo journalctl -f -n 0 -u ceph-320f6df8-6fa7-11ef-bcea-c7b262605968@mon.b.service 2024-09-10T19:06:17.806 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-09-10T19:06:17.806 DEBUG:teuthology.orchestra.run.smithi191:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph mon dump -f json 2024-09-10T19:06:17.824 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:06:17 smithi183 ceph-mon[29736]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T19:06:17.824 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:06:17 smithi183 ceph-mon[29736]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:06:17.825 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:06:17 smithi183 ceph-mon[29736]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T19:06:17.825 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:06:17 smithi183 ceph-mon[29736]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T19:06:17.825 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:06:17 smithi183 ceph-mon[29736]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T19:06:17.825 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:06:17 smithi183 ceph-mon[29736]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:06:17.825 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:06:17 smithi183 ceph-mon[29736]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-10T19:06:17.826 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:06:17 smithi183 ceph-mon[29736]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T19:06:18.523 INFO:teuthology.orchestra.run.smithi191.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/config/ceph.conf 2024-09-10T19:06:19.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:06:18 smithi183 ceph-mon[29736]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi183:172.21.15.183=a;smithi183:[v2:172.21.15.183:3301,v1:172.21.15.183:6790]=c;smithi191:172.21.15.191=b", "target": ["mon-mgr", ""]}]: dispatch 2024-09-10T19:06:19.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:06:18 smithi183 ceph-mon[29736]: Saving service mon spec with placement smithi183:172.21.15.183=a;smithi183:[v2:172.21.15.183:3301,v1:172.21.15.183:6790]=c;smithi191:172.21.15.191=b;count:3 2024-09-10T19:06:19.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:06:18 smithi183 ceph-mon[29736]: Deploying daemon mon.b on smithi191 2024-09-10T19:06:20.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:06:19 smithi183 ceph-mon[29736]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T19:06:21.803 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-10T19:06:21.803 INFO:teuthology.orchestra.run.smithi191.stdout:{"epoch":1,"fsid":"320f6df8-6fa7-11ef-bcea-c7b262605968","modified":"2024-09-10T19:05:00.576431Z","created":"2024-09-10T19:05:00.576431Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:3300","nonce":0},{"type":"v1","addr":"172.21.15.183:6789","nonce":0}]},"addr":"172.21.15.183:6789/0","public_addr":"172.21.15.183:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-09-10T19:06:21.804 INFO:teuthology.orchestra.run.smithi191.stderr:dumped monmap epoch 1 2024-09-10T19:06:22.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:06:21 smithi183 ceph-mon[29736]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T19:06:23.614 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-09-10T19:06:23.615 DEBUG:teuthology.orchestra.run.smithi191:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph mon dump -f json 2024-09-10T19:06:23.874 INFO:teuthology.orchestra.run.smithi191.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.b/config 2024-09-10T19:06:27.392 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:06:27 smithi183 ceph-mon[29736]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-10T19:06:27.393 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:06:27 smithi183 ceph-mon[29736]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-10T19:06:27.393 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:06:27 smithi183 ceph-mon[29736]: mon.a calling monitor election 2024-09-10T19:06:27.393 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:06:27 smithi183 ceph-mon[29736]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T19:06:27.393 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:06:27 smithi183 ceph-mon[29736]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-10T19:06:27.393 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:06:27 smithi183 ceph-mon[29736]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-10T19:06:27.393 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:06:27 smithi183 ceph-mon[29736]: mon.b calling monitor election 2024-09-10T19:06:27.393 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:06:27 smithi183 ceph-mon[29736]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T19:06:27.393 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:06:27 smithi183 ceph-mon[29736]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-10T19:06:27.393 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:06:27 smithi183 ceph-mon[29736]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-10T19:06:27.394 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:06:27 smithi183 ceph-mon[29736]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T19:06:27.394 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:06:27 smithi183 ceph-mon[29736]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-10T19:06:27.394 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:06:27 smithi183 ceph-mon[29736]: mon.a is new leader, mons a,b in quorum (ranks 0,1) 2024-09-10T19:06:27.394 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:06:27 smithi183 ceph-mon[29736]: monmap e2: 2 mons at {a=[v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0],b=[v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-09-10T19:06:27.394 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:06:27 smithi183 ceph-mon[29736]: fsmap 2024-09-10T19:06:27.394 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:06:27 smithi183 ceph-mon[29736]: osdmap e4: 0 total, 0 up, 0 in 2024-09-10T19:06:27.394 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:06:27 smithi183 ceph-mon[29736]: mgrmap e13: y(active, since 38s) 2024-09-10T19:06:27.394 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:06:27 smithi183 ceph-mon[29736]: overall HEALTH_OK 2024-09-10T19:06:27.394 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:06:27 smithi183 ceph-mon[29736]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:06:27.394 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:06:27 smithi183 ceph-mon[29736]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:06:27.394 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:06:27 smithi183 ceph-mon[29736]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:06:27.394 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:06:27 smithi183 ceph-mon[29736]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-10T19:06:27.394 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:06:27 smithi183 ceph-mon[29736]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T19:06:27.465 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:06:27 smithi191 ceph-mon[32884]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T19:06:28.239 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:06:28 smithi183 ceph-mon[29736]: Deploying daemon mon.c on smithi183 2024-09-10T19:06:28.239 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:06:28 smithi183 ceph-mon[29736]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-10T19:06:28.465 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:06:28 smithi191 ceph-mon[32884]: Deploying daemon mon.c on smithi183 2024-09-10T19:06:28.465 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:06:28 smithi191 ceph-mon[32884]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-10T19:06:28.606 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-10T19:06:28.606 INFO:teuthology.orchestra.run.smithi191.stdout:{"epoch":2,"fsid":"320f6df8-6fa7-11ef-bcea-c7b262605968","modified":"2024-09-10T19:06:21.964058Z","created":"2024-09-10T19:05:00.576431Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:3300","nonce":0},{"type":"v1","addr":"172.21.15.183:6789","nonce":0}]},"addr":"172.21.15.183:6789/0","public_addr":"172.21.15.183:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:3300","nonce":0},{"type":"v1","addr":"172.21.15.191:6789","nonce":0}]},"addr":"172.21.15.191:6789/0","public_addr":"172.21.15.191:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-09-10T19:06:28.606 INFO:teuthology.orchestra.run.smithi191.stderr:dumped monmap epoch 2 2024-09-10T19:06:29.187 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:06:29 smithi191 ceph-mon[32884]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T19:06:29.187 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:06:29 smithi191 ceph-mon[32884]: from='client.? 172.21.15.191:0/469899973' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-10T19:06:29.278 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:06:29 smithi183 ceph-mon[29736]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T19:06:29.278 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:06:29 smithi183 ceph-mon[29736]: from='client.? 172.21.15.191:0/469899973' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-10T19:06:29.931 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:06:29 smithi183 podman[41525]: 2024-09-10 19:06:29.845678522 +0000 UTC m=+0.123593941 container create 193339ca1848d4955e928b9db4037cffbdebba48883ee47aee9f02dee955d48f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3bfeaf4417585c821cab9ca76e4d95f390b0e769807122a45463640820b5cd16, name=ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mon-c, RELEASE=wip-guits-quincy-2024-09-09-1810-quincy-5be5739, CEPH_POINT_RELEASE=, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, ceph=True, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-10T19:06:30.211 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-09-10T19:06:30.211 DEBUG:teuthology.orchestra.run.smithi191:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph mon dump -f json 2024-09-10T19:06:30.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:06:30 smithi183 podman[41525]: 2024-09-10 19:06:30.072716252 +0000 UTC m=+0.350631675 container init 193339ca1848d4955e928b9db4037cffbdebba48883ee47aee9f02dee955d48f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3bfeaf4417585c821cab9ca76e4d95f390b0e769807122a45463640820b5cd16, name=ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mon-c, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-guits-quincy-2024-09-09-1810-quincy-5be5739, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.schema-version=1.0, ceph=True, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS) 2024-09-10T19:06:30.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:06:30 smithi183 podman[41525]: 2024-09-10 19:06:30.077287081 +0000 UTC m=+0.355202500 container start 193339ca1848d4955e928b9db4037cffbdebba48883ee47aee9f02dee955d48f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3bfeaf4417585c821cab9ca76e4d95f390b0e769807122a45463640820b5cd16, name=ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mon-c, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, RELEASE=wip-guits-quincy-2024-09-09-1810-quincy-5be5739, ceph=True, maintainer=Guillaume Abrioux , GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731) 2024-09-10T19:06:30.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:06:30 smithi183 ceph-mon[41562]: set uid:gid to 167:167 (ceph:ceph) 2024-09-10T19:06:30.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:06:30 smithi183 ceph-mon[41562]: ceph version 17.2.7-1606-g5be57395 (5be57395ea235535910512ae73ba8e7ff25fe588) quincy (stable), process ceph-mon, pid 6 2024-09-10T19:06:30.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:06:30 smithi183 ceph-mon[41562]: pidfile_write: ignore empty --pid-file 2024-09-10T19:06:30.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:06:30 smithi183 ceph-mon[41562]: load: jerasure load: lrc 2024-09-10T19:06:30.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:06:30 smithi183 ceph-mon[41562]: rocksdb: RocksDB version: 6.15.5 2024-09-10T19:06:30.433 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:06:30 smithi183 ceph-mon[41562]: rocksdb: Git sha rocksdb_build_git_sha:@0@ 2024-09-10T19:06:30.433 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:06:30 smithi183 ceph-mon[41562]: rocksdb: Compile date Sep 9 2024 2024-09-10T19:06:30.433 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:06:30 smithi183 ceph-mon[41562]: rocksdb: DB SUMMARY 2024-09-10T19:06:30.433 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:06:30 smithi183 ceph-mon[41562]: rocksdb: DB Session ID: K4N6UDBR9OREKPZFL9LC 2024-09-10T19:06:30.433 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:06:30 smithi183 ceph-mon[41562]: rocksdb: CURRENT file: CURRENT 2024-09-10T19:06:30.433 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:06:30 smithi183 ceph-mon[41562]: rocksdb: IDENTITY file: IDENTITY 2024-09-10T19:06:30.433 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:06:30 smithi183 ceph-mon[41562]: rocksdb: MANIFEST file: MANIFEST-000003 size: 57 Bytes 2024-09-10T19:06:30.433 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:06:30 smithi183 ceph-mon[41562]: rocksdb: SST files in /var/lib/ceph/mon/ceph-c/store.db dir, Total Num: 0, files: 2024-09-10T19:06:30.433 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:06:30 smithi183 ceph-mon[41562]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-c/store.db: 000004.log size: 636 ; 2024-09-10T19:06:30.434 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:06:30 smithi183 ceph-mon[41562]: rocksdb: Options.error_if_exists: 0 2024-09-10T19:06:30.434 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:06:30 smithi183 ceph-mon[41562]: rocksdb: Options.create_if_missing: 0 2024-09-10T19:06:30.434 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:06:30 smithi183 ceph-mon[41562]: rocksdb: Options.paranoid_checks: 1 2024-09-10T19:06:30.434 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:06:30 smithi183 ceph-mon[41562]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-09-10T19:06:30.434 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:06:30 smithi183 ceph-mon[41562]: rocksdb: Options.env: 0x564d46554440 2024-09-10T19:06:30.434 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:06:30 smithi183 ceph-mon[41562]: rocksdb: Options.fs: Posix File System 2024-09-10T19:06:30.435 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:06:30 smithi183 ceph-mon[41562]: rocksdb: Options.info_log: 0x564d47392ec0 2024-09-10T19:06:30.435 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:06:30 smithi183 ceph-mon[41562]: rocksdb: Options.max_file_opening_threads: 16 2024-09-10T19:06:30.435 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:06:30 smithi183 ceph-mon[41562]: rocksdb: Options.statistics: (nil) 2024-09-10T19:06:30.435 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:06:30 smithi183 ceph-mon[41562]: rocksdb: Options.use_fsync: 0 2024-09-10T19:06:30.435 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:06:30 smithi183 ceph-mon[41562]: rocksdb: Options.max_log_file_size: 0 2024-09-10T19:06:30.435 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:06:30 smithi183 ceph-mon[41562]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-09-10T19:06:30.435 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:06:30 smithi183 ceph-mon[41562]: rocksdb: Options.log_file_time_to_roll: 0 2024-09-10T19:06:30.435 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:06:30 smithi183 ceph-mon[41562]: rocksdb: Options.keep_log_file_num: 1000 2024-09-10T19:06:30.436 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:06:30 smithi183 ceph-mon[41562]: rocksdb: Options.recycle_log_file_num: 0 2024-09-10T19:06:30.436 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:06:30 smithi183 ceph-mon[41562]: rocksdb: Options.allow_fallocate: 1 2024-09-10T19:06:30.436 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:06:30 smithi183 ceph-mon[41562]: rocksdb: Options.allow_mmap_reads: 0 2024-09-10T19:06:30.436 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:06:30 smithi183 ceph-mon[41562]: rocksdb: Options.allow_mmap_writes: 0 2024-09-10T19:06:30.436 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:06:30 smithi183 ceph-mon[41562]: rocksdb: Options.use_direct_reads: 0 2024-09-10T19:06:30.436 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:06:30 smithi183 ceph-mon[41562]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-09-10T19:06:30.436 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:06:30 smithi183 ceph-mon[41562]: rocksdb: Options.create_missing_column_families: 0 2024-09-10T19:06:30.436 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:06:30 smithi183 ceph-mon[41562]: rocksdb: Options.db_log_dir: 2024-09-10T19:06:30.436 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:06:30 smithi183 ceph-mon[41562]: rocksdb: Options.wal_dir: /var/lib/ceph/mon/ceph-c/store.db 2024-09-10T19:06:30.437 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:06:30 smithi183 ceph-mon[41562]: rocksdb: Options.table_cache_numshardbits: 6 2024-09-10T19:06:30.437 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:06:30 smithi183 ceph-mon[41562]: rocksdb: Options.WAL_ttl_seconds: 0 2024-09-10T19:06:30.437 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:06:30 smithi183 ceph-mon[41562]: rocksdb: Options.WAL_size_limit_MB: 0 2024-09-10T19:06:30.437 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:06:30 smithi183 ceph-mon[41562]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-09-10T19:06:30.437 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:06:30 smithi183 ceph-mon[41562]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-09-10T19:06:30.437 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:06:30 smithi183 ceph-mon[41562]: rocksdb: Options.is_fd_close_on_exec: 1 2024-09-10T19:06:30.437 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:06:30 smithi183 ceph-mon[41562]: rocksdb: Options.advise_random_on_open: 1 2024-09-10T19:06:30.437 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:06:30 smithi183 ceph-mon[41562]: rocksdb: Options.db_write_buffer_size: 0 2024-09-10T19:06:30.438 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:06:30 smithi183 ceph-mon[41562]: rocksdb: Options.write_buffer_manager: 0x564d474507b0 2024-09-10T19:06:30.438 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:06:30 smithi183 ceph-mon[41562]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-09-10T19:06:30.438 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:06:30 smithi183 ceph-mon[41562]: rocksdb: Options.new_table_reader_for_compaction_inputs: 0 2024-09-10T19:06:30.438 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:06:30 smithi183 ceph-mon[41562]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-09-10T19:06:30.438 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:06:30 smithi183 ceph-mon[41562]: rocksdb: Options.use_adaptive_mutex: 0 2024-09-10T19:06:30.438 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:06:30 smithi183 ceph-mon[41562]: rocksdb: Options.rate_limiter: (nil) 2024-09-10T19:06:30.438 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:06:30 smithi183 ceph-mon[41562]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-09-10T19:06:30.438 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:06:30 smithi183 ceph-mon[41562]: rocksdb: Options.wal_recovery_mode: 2 2024-09-10T19:06:30.439 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:06:30 smithi183 ceph-mon[41562]: rocksdb: Options.enable_thread_tracking: 0 2024-09-10T19:06:30.439 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:06:30 smithi183 ceph-mon[41562]: rocksdb: Options.enable_pipelined_write: 0 2024-09-10T19:06:30.439 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:06:30 smithi183 ceph-mon[41562]: rocksdb: Options.unordered_write: 0 2024-09-10T19:06:30.439 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:06:30 smithi183 ceph-mon[41562]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-09-10T19:06:30.439 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:06:30 smithi183 ceph-mon[41562]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-09-10T19:06:30.439 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:06:30 smithi183 ceph-mon[41562]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-09-10T19:06:30.439 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:06:30 smithi183 ceph-mon[41562]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-09-10T19:06:30.439 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:06:30 smithi183 ceph-mon[41562]: rocksdb: Options.row_cache: None 2024-09-10T19:06:30.439 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:06:30 smithi183 ceph-mon[41562]: rocksdb: Options.wal_filter: None 2024-09-10T19:06:30.442 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:06:30 smithi183 ceph-mon[41562]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-09-10T19:06:30.442 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:06:30 smithi183 ceph-mon[41562]: rocksdb: Options.allow_ingest_behind: 0 2024-09-10T19:06:30.442 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:06:30 smithi183 ceph-mon[41562]: rocksdb: Options.preserve_deletes: 0 2024-09-10T19:06:30.442 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:06:30 smithi183 ceph-mon[41562]: rocksdb: Options.two_write_queues: 0 2024-09-10T19:06:30.442 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:06:30 smithi183 ceph-mon[41562]: rocksdb: Options.manual_wal_flush: 0 2024-09-10T19:06:30.442 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:06:30 smithi183 ceph-mon[41562]: rocksdb: Options.atomic_flush: 0 2024-09-10T19:06:30.442 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:06:30 smithi183 ceph-mon[41562]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-09-10T19:06:30.443 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:06:30 smithi183 ceph-mon[41562]: rocksdb: Options.persist_stats_to_disk: 0 2024-09-10T19:06:30.443 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:06:30 smithi183 ceph-mon[41562]: rocksdb: Options.write_dbid_to_manifest: 0 2024-09-10T19:06:30.443 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:06:30 smithi183 ceph-mon[41562]: rocksdb: Options.log_readahead_size: 0 2024-09-10T19:06:30.443 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:06:30 smithi183 ceph-mon[41562]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-09-10T19:06:30.443 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:06:30 smithi183 ceph-mon[41562]: rocksdb: Options.best_efforts_recovery: 0 2024-09-10T19:06:30.443 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:06:30 smithi183 ceph-mon[41562]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-09-10T19:06:30.443 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:06:30 smithi183 ceph-mon[41562]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-09-10T19:06:30.443 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:06:30 smithi183 ceph-mon[41562]: rocksdb: Options.allow_data_in_errors: 0 2024-09-10T19:06:30.444 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:06:30 smithi183 ceph-mon[41562]: rocksdb: Options.db_host_id: __hostname__ 2024-09-10T19:06:30.444 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:06:30 smithi183 ceph-mon[41562]: rocksdb: Options.max_background_jobs: 2 2024-09-10T19:06:30.444 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:06:30 smithi183 ceph-mon[41562]: rocksdb: Options.max_background_compactions: -1 2024-09-10T19:06:30.444 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:06:30 smithi183 ceph-mon[41562]: rocksdb: Options.max_subcompactions: 1 2024-09-10T19:06:30.444 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:06:30 smithi183 ceph-mon[41562]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-09-10T19:06:30.444 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:06:30 smithi183 ceph-mon[41562]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-09-10T19:06:30.444 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:06:30 smithi183 ceph-mon[41562]: rocksdb: Options.delayed_write_rate : 16777216 2024-09-10T19:06:30.444 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:06:30 smithi183 ceph-mon[41562]: rocksdb: Options.max_total_wal_size: 0 2024-09-10T19:06:30.445 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:06:30 smithi183 ceph-mon[41562]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-09-10T19:06:30.445 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:06:30 smithi183 ceph-mon[41562]: rocksdb: Options.stats_dump_period_sec: 600 2024-09-10T19:06:30.445 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:06:30 smithi183 ceph-mon[41562]: rocksdb: Options.stats_persist_period_sec: 600 2024-09-10T19:06:30.445 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:06:30 smithi183 ceph-mon[41562]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-09-10T19:06:30.445 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:06:30 smithi183 ceph-mon[41562]: rocksdb: Options.max_open_files: -1 2024-09-10T19:06:30.445 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:06:30 smithi183 ceph-mon[41562]: rocksdb: Options.bytes_per_sync: 0 2024-09-10T19:06:30.445 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:06:30 smithi183 ceph-mon[41562]: rocksdb: Options.wal_bytes_per_sync: 0 2024-09-10T19:06:30.445 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:06:30 smithi183 ceph-mon[41562]: rocksdb: Options.strict_bytes_per_sync: 0 2024-09-10T19:06:30.446 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:06:30 smithi183 ceph-mon[41562]: rocksdb: Options.compaction_readahead_size: 0 2024-09-10T19:06:30.446 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:06:30 smithi183 ceph-mon[41562]: rocksdb: Options.max_background_flushes: -1 2024-09-10T19:06:30.446 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:06:30 smithi183 ceph-mon[41562]: rocksdb: Compression algorithms supported: 2024-09-10T19:06:30.446 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:06:30 smithi183 ceph-mon[41562]: rocksdb: kZSTD supported: 0 2024-09-10T19:06:30.446 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:06:30 smithi183 ceph-mon[41562]: rocksdb: kXpressCompression supported: 0 2024-09-10T19:06:30.446 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:06:30 smithi183 ceph-mon[41562]: rocksdb: kBZip2Compression supported: 0 2024-09-10T19:06:30.446 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:06:30 smithi183 ceph-mon[41562]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-09-10T19:06:30.446 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:06:30 smithi183 ceph-mon[41562]: rocksdb: kLZ4Compression supported: 1 2024-09-10T19:06:30.446 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:06:30 smithi183 ceph-mon[41562]: rocksdb: kZlibCompression supported: 1 2024-09-10T19:06:30.447 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:06:30 smithi183 ceph-mon[41562]: rocksdb: kLZ4HCCompression supported: 1 2024-09-10T19:06:30.447 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:06:30 smithi183 ceph-mon[41562]: rocksdb: kSnappyCompression supported: 1 2024-09-10T19:06:30.447 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:06:30 smithi183 ceph-mon[41562]: rocksdb: Fast CRC32 supported: Supported on x86 2024-09-10T19:06:30.447 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:06:30 smithi183 ceph-mon[41562]: rocksdb: [db/version_set.cc:4724] Recovering from manifest file: /var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000003 2024-09-10T19:06:30.447 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:06:30 smithi183 ceph-mon[41562]: rocksdb: [db/column_family.cc:595] --------------- Options for column family [default]: 2024-09-10T19:06:30.447 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:06:30 smithi183 ceph-mon[41562]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-09-10T19:06:30.447 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:06:30 smithi183 ceph-mon[41562]: rocksdb: Options.merge_operator: 2024-09-10T19:06:30.447 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:06:30 smithi183 ceph-mon[41562]: rocksdb: Options.compaction_filter: None 2024-09-10T19:06:30.448 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:06:30 smithi183 ceph-mon[41562]: rocksdb: Options.compaction_filter_factory: None 2024-09-10T19:06:30.448 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:06:30 smithi183 ceph-mon[41562]: rocksdb: Options.sst_partitioner_factory: None 2024-09-10T19:06:30.448 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:06:30 smithi183 ceph-mon[41562]: rocksdb: Options.memtable_factory: SkipListFactory 2024-09-10T19:06:30.448 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:06:30 smithi183 ceph-mon[41562]: rocksdb: Options.table_factory: BlockBasedTable 2024-09-10T19:06:30.448 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:06:30 smithi183 ceph-mon[41562]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x564d47339a38) 2024-09-10T19:06:30.448 INFO:journalctl@ceph.mon.c.smithi183.stdout: cache_index_and_filter_blocks: 1 2024-09-10T19:06:30.448 INFO:journalctl@ceph.mon.c.smithi183.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-09-10T19:06:30.448 INFO:journalctl@ceph.mon.c.smithi183.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-09-10T19:06:30.449 INFO:journalctl@ceph.mon.c.smithi183.stdout: pin_top_level_index_and_filter: 1 2024-09-10T19:06:30.449 INFO:journalctl@ceph.mon.c.smithi183.stdout: index_type: 0 2024-09-10T19:06:30.449 INFO:journalctl@ceph.mon.c.smithi183.stdout: data_block_index_type: 0 2024-09-10T19:06:30.449 INFO:journalctl@ceph.mon.c.smithi183.stdout: index_shortening: 1 2024-09-10T19:06:30.449 INFO:journalctl@ceph.mon.c.smithi183.stdout: data_block_hash_table_util_ratio: 0.750000 2024-09-10T19:06:30.449 INFO:journalctl@ceph.mon.c.smithi183.stdout: hash_index_allow_collision: 1 2024-09-10T19:06:30.449 INFO:journalctl@ceph.mon.c.smithi183.stdout: checksum: 1 2024-09-10T19:06:30.449 INFO:journalctl@ceph.mon.c.smithi183.stdout: no_block_cache: 0 2024-09-10T19:06:30.449 INFO:journalctl@ceph.mon.c.smithi183.stdout: block_cache: 0x564d4736f1f0 2024-09-10T19:06:30.450 INFO:journalctl@ceph.mon.c.smithi183.stdout: block_cache_name: BinnedLRUCache 2024-09-10T19:06:30.450 INFO:journalctl@ceph.mon.c.smithi183.stdout: block_cache_options: 2024-09-10T19:06:30.450 INFO:journalctl@ceph.mon.c.smithi183.stdout: capacity : 536870912 2024-09-10T19:06:30.450 INFO:journalctl@ceph.mon.c.smithi183.stdout: num_shard_bits : 4 2024-09-10T19:06:30.450 INFO:journalctl@ceph.mon.c.smithi183.stdout: strict_capacity_limit : 0 2024-09-10T19:06:30.450 INFO:journalctl@ceph.mon.c.smithi183.stdout: high_pri_pool_ratio: 0.000 2024-09-10T19:06:30.450 INFO:journalctl@ceph.mon.c.smithi183.stdout: block_cache_compressed: (nil) 2024-09-10T19:06:30.450 INFO:journalctl@ceph.mon.c.smithi183.stdout: persistent_cache: (nil) 2024-09-10T19:06:30.451 INFO:journalctl@ceph.mon.c.smithi183.stdout: block_size: 4096 2024-09-10T19:06:30.451 INFO:journalctl@ceph.mon.c.smithi183.stdout: block_size_deviation: 10 2024-09-10T19:06:30.451 INFO:journalctl@ceph.mon.c.smithi183.stdout: block_restart_interval: 16 2024-09-10T19:06:30.451 INFO:journalctl@ceph.mon.c.smithi183.stdout: index_block_restart_interval: 1 2024-09-10T19:06:30.451 INFO:journalctl@ceph.mon.c.smithi183.stdout: metadata_block_size: 4096 2024-09-10T19:06:30.451 INFO:journalctl@ceph.mon.c.smithi183.stdout: partition_filters: 0 2024-09-10T19:06:30.451 INFO:journalctl@ceph.mon.c.smithi183.stdout: use_delta_encoding: 1 2024-09-10T19:06:30.451 INFO:journalctl@ceph.mon.c.smithi183.stdout: filter_policy: rocksdb.BuiltinBloomFilter 2024-09-10T19:06:30.451 INFO:journalctl@ceph.mon.c.smithi183.stdout: whole_key_filtering: 1 2024-09-10T19:06:30.452 INFO:journalctl@ceph.mon.c.smithi183.stdout: verify_compression: 0 2024-09-10T19:06:30.452 INFO:journalctl@ceph.mon.c.smithi183.stdout: read_amp_bytes_per_bit: 0 2024-09-10T19:06:30.452 INFO:journalctl@ceph.mon.c.smithi183.stdout: format_version: 4 2024-09-10T19:06:30.452 INFO:journalctl@ceph.mon.c.smithi183.stdout: enable_index_compression: 1 2024-09-10T19:06:30.452 INFO:journalctl@ceph.mon.c.smithi183.stdout: block_align: 0 2024-09-10T19:06:30.452 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:06:30 smithi183 ceph-mon[41562]: rocksdb: Options.write_buffer_size: 33554432 2024-09-10T19:06:30.452 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:06:30 smithi183 ceph-mon[41562]: rocksdb: Options.max_write_buffer_number: 2 2024-09-10T19:06:30.452 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:06:30 smithi183 ceph-mon[41562]: rocksdb: Options.compression: NoCompression 2024-09-10T19:06:30.452 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:06:30 smithi183 ceph-mon[41562]: rocksdb: Options.bottommost_compression: Disabled 2024-09-10T19:06:30.452 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:06:30 smithi183 ceph-mon[41562]: rocksdb: Options.prefix_extractor: nullptr 2024-09-10T19:06:30.452 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:06:30 smithi183 ceph-mon[41562]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-09-10T19:06:30.453 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:06:30 smithi183 ceph-mon[41562]: rocksdb: Options.num_levels: 7 2024-09-10T19:06:30.453 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:06:30 smithi183 ceph-mon[41562]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-09-10T19:06:30.453 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:06:30 smithi183 ceph-mon[41562]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-09-10T19:06:30.453 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:06:30 smithi183 ceph-mon[41562]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-09-10T19:06:30.453 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:06:30 smithi183 ceph-mon[41562]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-09-10T19:06:30.453 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:06:30 smithi183 ceph-mon[41562]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-09-10T19:06:30.453 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:06:30 smithi183 ceph-mon[41562]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-09-10T19:06:30.453 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:06:30 smithi183 ceph-mon[41562]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-09-10T19:06:30.453 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:06:30 smithi183 ceph-mon[41562]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-09-10T19:06:30.453 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:06:30 smithi183 ceph-mon[41562]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-09-10T19:06:30.453 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:06:30 smithi183 ceph-mon[41562]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-09-10T19:06:30.453 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:06:30 smithi183 ceph-mon[41562]: rocksdb: Options.compression_opts.window_bits: -14 2024-09-10T19:06:30.453 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:06:30 smithi183 ceph-mon[41562]: rocksdb: Options.compression_opts.level: 32767 2024-09-10T19:06:30.454 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:06:30 smithi183 ceph-mon[41562]: rocksdb: Options.compression_opts.strategy: 0 2024-09-10T19:06:30.454 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:06:30 smithi183 ceph-mon[41562]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-09-10T19:06:30.454 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:06:30 smithi183 ceph-mon[41562]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-09-10T19:06:30.454 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:06:30 smithi183 ceph-mon[41562]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-09-10T19:06:30.454 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:06:30 smithi183 ceph-mon[41562]: rocksdb: Options.compression_opts.enabled: false 2024-09-10T19:06:30.454 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:06:30 smithi183 ceph-mon[41562]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-09-10T19:06:30.454 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:06:30 smithi183 ceph-mon[41562]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-09-10T19:06:30.454 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:06:30 smithi183 ceph-mon[41562]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-09-10T19:06:30.454 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:06:30 smithi183 ceph-mon[41562]: rocksdb: Options.target_file_size_base: 67108864 2024-09-10T19:06:30.454 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:06:30 smithi183 ceph-mon[41562]: rocksdb: Options.target_file_size_multiplier: 1 2024-09-10T19:06:30.455 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:06:30 smithi183 ceph-mon[41562]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-09-10T19:06:30.455 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:06:30 smithi183 ceph-mon[41562]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-09-10T19:06:30.455 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:06:30 smithi183 ceph-mon[41562]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-09-10T19:06:30.455 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:06:30 smithi183 ceph-mon[41562]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-09-10T19:06:30.455 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:06:30 smithi183 ceph-mon[41562]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-09-10T19:06:30.455 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:06:30 smithi183 ceph-mon[41562]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-09-10T19:06:30.455 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:06:30 smithi183 ceph-mon[41562]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-09-10T19:06:30.455 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:06:30 smithi183 ceph-mon[41562]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-09-10T19:06:30.455 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:06:30 smithi183 ceph-mon[41562]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-09-10T19:06:30.455 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:06:30 smithi183 ceph-mon[41562]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-09-10T19:06:30.455 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:06:30 smithi183 ceph-mon[41562]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-09-10T19:06:30.455 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:06:30 smithi183 ceph-mon[41562]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-09-10T19:06:30.455 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:06:30 smithi183 ceph-mon[41562]: rocksdb: Options.arena_block_size: 4194304 2024-09-10T19:06:30.455 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:06:30 smithi183 ceph-mon[41562]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-09-10T19:06:30.456 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:06:30 smithi183 ceph-mon[41562]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-09-10T19:06:30.456 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:06:30 smithi183 ceph-mon[41562]: rocksdb: Options.rate_limit_delay_max_milliseconds: 100 2024-09-10T19:06:30.456 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:06:30 smithi183 ceph-mon[41562]: rocksdb: Options.disable_auto_compactions: 0 2024-09-10T19:06:30.456 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:06:30 smithi183 ceph-mon[41562]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-09-10T19:06:30.456 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:06:30 smithi183 ceph-mon[41562]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-09-10T19:06:30.456 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:06:30 smithi183 ceph-mon[41562]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-09-10T19:06:30.456 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:06:30 smithi183 ceph-mon[41562]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-09-10T19:06:30.456 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:06:30 smithi183 ceph-mon[41562]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-09-10T19:06:30.456 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:06:30 smithi183 ceph-mon[41562]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-09-10T19:06:30.456 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:06:30 smithi183 ceph-mon[41562]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-09-10T19:06:30.456 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:06:30 smithi183 ceph-mon[41562]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-09-10T19:06:30.456 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:06:30 smithi183 ceph-mon[41562]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-09-10T19:06:30.457 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:06:30 smithi183 ceph-mon[41562]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-09-10T19:06:30.457 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:06:30 smithi183 ceph-mon[41562]: rocksdb: Options.table_properties_collectors: CompactOnDeletionCollector (Sliding window size = 32768 Deletion trigger = 16384 Deletion ratio = 0); 2024-09-10T19:06:30.457 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:06:30 smithi183 ceph-mon[41562]: rocksdb: Options.inplace_update_support: 0 2024-09-10T19:06:30.457 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:06:30 smithi183 ceph-mon[41562]: rocksdb: Options.inplace_update_num_locks: 10000 2024-09-10T19:06:30.457 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:06:30 smithi183 ceph-mon[41562]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-09-10T19:06:30.457 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:06:30 smithi183 ceph-mon[41562]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-09-10T19:06:30.457 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:06:30 smithi183 ceph-mon[41562]: rocksdb: Options.memtable_huge_page_size: 0 2024-09-10T19:06:30.457 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:06:30 smithi183 ceph-mon[41562]: rocksdb: Options.bloom_locality: 0 2024-09-10T19:06:30.457 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:06:30 smithi183 ceph-mon[41562]: rocksdb: Options.max_successive_merges: 0 2024-09-10T19:06:30.457 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:06:30 smithi183 ceph-mon[41562]: rocksdb: Options.optimize_filters_for_hits: 0 2024-09-10T19:06:30.457 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:06:30 smithi183 ceph-mon[41562]: rocksdb: Options.paranoid_file_checks: 0 2024-09-10T19:06:30.457 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:06:30 smithi183 ceph-mon[41562]: rocksdb: Options.force_consistency_checks: 1 2024-09-10T19:06:30.458 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:06:30 smithi183 ceph-mon[41562]: rocksdb: Options.report_bg_io_stats: 0 2024-09-10T19:06:30.458 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:06:30 smithi183 ceph-mon[41562]: rocksdb: Options.ttl: 2592000 2024-09-10T19:06:30.458 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:06:30 smithi183 ceph-mon[41562]: rocksdb: Options.periodic_compaction_seconds: 0 2024-09-10T19:06:30.458 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:06:30 smithi183 ceph-mon[41562]: rocksdb: Options.enable_blob_files: false 2024-09-10T19:06:30.458 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:06:30 smithi183 ceph-mon[41562]: rocksdb: Options.min_blob_size: 0 2024-09-10T19:06:30.458 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:06:30 smithi183 ceph-mon[41562]: rocksdb: Options.blob_file_size: 268435456 2024-09-10T19:06:30.458 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:06:30 smithi183 ceph-mon[41562]: rocksdb: Options.blob_compression_type: NoCompression 2024-09-10T19:06:30.458 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:06:30 smithi183 ceph-mon[41562]: rocksdb: Options.enable_blob_garbage_collection: false 2024-09-10T19:06:30.458 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:06:30 smithi183 ceph-mon[41562]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-09-10T19:06:30.458 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:06:30 smithi183 ceph-mon[41562]: rocksdb: [db/version_set.cc:4764] Recovered from manifest file:/var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000003 succeeded,manifest_file_number is 3, next_file_number is 5, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-09-10T19:06:30.458 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:06:30 smithi183 ceph-mon[41562]: rocksdb: [db/version_set.cc:4779] Column family [default] (ID 0), log number is 0 2024-09-10T19:06:30.458 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:06:30 smithi183 ceph-mon[41562]: rocksdb: [db/version_set.cc:4082] Creating manifest 7 2024-09-10T19:06:30.458 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:06:30 smithi183 ceph-mon[41562]: rocksdb: EVENT_LOG_v1 {"time_micros": 1725995190101181, "job": 1, "event": "recovery_started", "wal_files": [4]} 2024-09-10T19:06:30.458 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:06:30 smithi183 ceph-mon[41562]: rocksdb: [db/db_impl/db_impl_open.cc:845] Recovering log #4 mode 2 2024-09-10T19:06:30.459 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:06:30 smithi183 ceph-mon[41562]: rocksdb: [table/block_based/filter_policy.cc:991] Using legacy Bloom filter with high (20) bits/key. Dramatic filter space and/or accuracy improvement is available with format_version>=5. 2024-09-10T19:06:30.459 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:06:30 smithi183 ceph-mon[41562]: rocksdb: EVENT_LOG_v1 {"time_micros": 1725995190101473, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 8, "file_size": 1696, "file_checksum": "", "file_checksum_func_name": "Unknown", "table_properties": {"data_size": 648, "index_size": 31, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 526, "raw_average_value_size": 105, "num_data_blocks": 1, "num_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "rocksdb.BuiltinBloomFilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[CompactOnDeletionCollector]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; ", "creation_time": 1725995190, "oldest_key_time": 0, "file_creation_time": 0, "db_id": "3743253e-2396-41a5-bc90-df6dc6e10d61", "db_session_id": "K4N6UDBR9OREKPZFL9LC"}} 2024-09-10T19:06:30.459 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:06:30 smithi183 ceph-mon[41562]: rocksdb: [db/version_set.cc:4082] Creating manifest 9 2024-09-10T19:06:30.459 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:06:30 smithi183 ceph-mon[41562]: rocksdb: EVENT_LOG_v1 {"time_micros": 1725995190101822, "job": 1, "event": "recovery_finished"} 2024-09-10T19:06:30.459 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:06:30 smithi183 ceph-mon[41562]: rocksdb: [file/delete_scheduler.cc:69] Deleted file /var/lib/ceph/mon/ceph-c/store.db/000004.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-09-10T19:06:30.459 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:06:30 smithi183 ceph-mon[41562]: rocksdb: [db/db_impl/db_impl_open.cc:1700] SstFileManager instance 0x564d473a2700 2024-09-10T19:06:30.459 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:06:30 smithi183 ceph-mon[41562]: rocksdb: DB pointer 0x564d47372000 2024-09-10T19:06:30.459 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:06:30 smithi183 ceph-mon[41562]: rocksdb: [db/db_impl/db_impl.cc:901] ------- DUMPING STATS ------- 2024-09-10T19:06:30.459 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:06:30 smithi183 ceph-mon[41562]: rocksdb: [db/db_impl/db_impl.cc:903] 2024-09-10T19:06:30.459 INFO:journalctl@ceph.mon.c.smithi183.stdout: ** DB Stats ** 2024-09-10T19:06:30.459 INFO:journalctl@ceph.mon.c.smithi183.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-09-10T19:06:30.459 INFO:journalctl@ceph.mon.c.smithi183.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-09-10T19:06:30.459 INFO:journalctl@ceph.mon.c.smithi183.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-09-10T19:06:30.460 INFO:journalctl@ceph.mon.c.smithi183.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-09-10T19:06:30.460 INFO:journalctl@ceph.mon.c.smithi183.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-09-10T19:06:30.460 INFO:journalctl@ceph.mon.c.smithi183.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s 2024-09-10T19:06:30.460 INFO:journalctl@ceph.mon.c.smithi183.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-09-10T19:06:30.460 INFO:journalctl@ceph.mon.c.smithi183.stdout: 2024-09-10T19:06:30.460 INFO:journalctl@ceph.mon.c.smithi183.stdout: ** Compaction Stats [default] ** 2024-09-10T19:06:30.460 INFO:journalctl@ceph.mon.c.smithi183.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-09-10T19:06:30.460 INFO:journalctl@ceph.mon.c.smithi183.stdout: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-09-10T19:06:30.460 INFO:journalctl@ceph.mon.c.smithi183.stdout: L0 1/0 1.66 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 5.8 0.00 0.00 1 0.000 0 0 2024-09-10T19:06:30.460 INFO:journalctl@ceph.mon.c.smithi183.stdout: Sum 1/0 1.66 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 5.8 0.00 0.00 1 0.000 0 0 2024-09-10T19:06:30.460 INFO:journalctl@ceph.mon.c.smithi183.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 5.8 0.00 0.00 1 0.000 0 0 2024-09-10T19:06:30.460 INFO:journalctl@ceph.mon.c.smithi183.stdout: 2024-09-10T19:06:30.460 INFO:journalctl@ceph.mon.c.smithi183.stdout: ** Compaction Stats [default] ** 2024-09-10T19:06:30.461 INFO:journalctl@ceph.mon.c.smithi183.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-09-10T19:06:30.461 INFO:journalctl@ceph.mon.c.smithi183.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-09-10T19:06:30.461 INFO:journalctl@ceph.mon.c.smithi183.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 5.8 0.00 0.00 1 0.000 0 0 2024-09-10T19:06:30.461 INFO:journalctl@ceph.mon.c.smithi183.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-09-10T19:06:30.461 INFO:journalctl@ceph.mon.c.smithi183.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-09-10T19:06:30.461 INFO:journalctl@ceph.mon.c.smithi183.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-09-10T19:06:30.461 INFO:journalctl@ceph.mon.c.smithi183.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-09-10T19:06:30.461 INFO:journalctl@ceph.mon.c.smithi183.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-09-10T19:06:30.461 INFO:journalctl@ceph.mon.c.smithi183.stdout: AddFile(Keys): cumulative 0, interval 0 2024-09-10T19:06:30.461 INFO:journalctl@ceph.mon.c.smithi183.stdout: Cumulative compaction: 0.00 GB write, 0.57 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-10T19:06:30.461 INFO:journalctl@ceph.mon.c.smithi183.stdout: Interval compaction: 0.00 GB write, 0.57 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-10T19:06:30.461 INFO:journalctl@ceph.mon.c.smithi183.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-09-10T19:06:30.461 INFO:journalctl@ceph.mon.c.smithi183.stdout: 2024-09-10T19:06:30.461 INFO:journalctl@ceph.mon.c.smithi183.stdout: ** File Read Latency Histogram By Level [default] ** 2024-09-10T19:06:30.462 INFO:journalctl@ceph.mon.c.smithi183.stdout: 2024-09-10T19:06:30.462 INFO:journalctl@ceph.mon.c.smithi183.stdout: ** Compaction Stats [default] ** 2024-09-10T19:06:30.462 INFO:journalctl@ceph.mon.c.smithi183.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-09-10T19:06:30.462 INFO:journalctl@ceph.mon.c.smithi183.stdout: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-09-10T19:06:30.462 INFO:journalctl@ceph.mon.c.smithi183.stdout: L0 1/0 1.66 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 5.8 0.00 0.00 1 0.000 0 0 2024-09-10T19:06:30.462 INFO:journalctl@ceph.mon.c.smithi183.stdout: Sum 1/0 1.66 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 5.8 0.00 0.00 1 0.000 0 0 2024-09-10T19:06:30.462 INFO:journalctl@ceph.mon.c.smithi183.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2024-09-10T19:06:30.462 INFO:journalctl@ceph.mon.c.smithi183.stdout: 2024-09-10T19:06:30.462 INFO:journalctl@ceph.mon.c.smithi183.stdout: ** Compaction Stats [default] ** 2024-09-10T19:06:30.462 INFO:journalctl@ceph.mon.c.smithi183.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-09-10T19:06:30.462 INFO:journalctl@ceph.mon.c.smithi183.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-09-10T19:06:30.462 INFO:journalctl@ceph.mon.c.smithi183.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 5.8 0.00 0.00 1 0.000 0 0 2024-09-10T19:06:30.462 INFO:journalctl@ceph.mon.c.smithi183.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-09-10T19:06:30.463 INFO:journalctl@ceph.mon.c.smithi183.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-09-10T19:06:30.463 INFO:journalctl@ceph.mon.c.smithi183.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-09-10T19:06:30.463 INFO:journalctl@ceph.mon.c.smithi183.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-09-10T19:06:30.463 INFO:journalctl@ceph.mon.c.smithi183.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-09-10T19:06:30.463 INFO:journalctl@ceph.mon.c.smithi183.stdout: AddFile(Keys): cumulative 0, interval 0 2024-09-10T19:06:30.463 INFO:journalctl@ceph.mon.c.smithi183.stdout: Cumulative compaction: 0.00 GB write, 0.57 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-10T19:06:30.463 INFO:journalctl@ceph.mon.c.smithi183.stdout: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-10T19:06:30.463 INFO:journalctl@ceph.mon.c.smithi183.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-09-10T19:06:30.463 INFO:journalctl@ceph.mon.c.smithi183.stdout: 2024-09-10T19:06:30.463 INFO:journalctl@ceph.mon.c.smithi183.stdout: ** File Read Latency Histogram By Level [default] ** 2024-09-10T19:06:30.463 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:06:30 smithi183 ceph-mon[41562]: mon.c does not exist in monmap, will attempt to join an existing cluster 2024-09-10T19:06:30.463 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:06:30 smithi183 ceph-mon[41562]: using public_addrv [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] 2024-09-10T19:06:30.463 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:06:30 smithi183 ceph-mon[41562]: starting mon.c rank -1 at public addrs [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] at bind addrs [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] mon_data /var/lib/ceph/mon/ceph-c fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 2024-09-10T19:06:30.463 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:06:30 smithi183 ceph-mon[41562]: mon.c@-1(???) e0 preinit fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 2024-09-10T19:06:30.464 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:06:30 smithi183 ceph-mon[41562]: mon.c@-1(synchronizing).mds e1 new map 2024-09-10T19:06:30.464 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:06:30 smithi183 ceph-mon[41562]: mon.c@-1(synchronizing).mds e1 print_map 2024-09-10T19:06:30.464 INFO:journalctl@ceph.mon.c.smithi183.stdout: e1 2024-09-10T19:06:30.464 INFO:journalctl@ceph.mon.c.smithi183.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-09-10T19:06:30.464 INFO:journalctl@ceph.mon.c.smithi183.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-09-10T19:06:30.464 INFO:journalctl@ceph.mon.c.smithi183.stdout: legacy client fscid: -1 2024-09-10T19:06:30.464 INFO:journalctl@ceph.mon.c.smithi183.stdout: 2024-09-10T19:06:30.464 INFO:journalctl@ceph.mon.c.smithi183.stdout: No filesystems configured 2024-09-10T19:06:30.464 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:06:30 smithi183 ceph-mon[41562]: mon.c@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2024-09-10T19:06:30.464 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:06:30 smithi183 ceph-mon[41562]: mon.c@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2024-09-10T19:06:30.464 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:06:30 smithi183 ceph-mon[41562]: mon.c@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2024-09-10T19:06:30.465 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:06:30 smithi183 ceph-mon[41562]: mon.c@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2024-09-10T19:06:30.465 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:06:30 smithi183 ceph-mon[41562]: mon.c@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2024-09-10T19:06:30.465 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:06:30 smithi183 ceph-mon[41562]: mon.c@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2024-09-10T19:06:30.465 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:06:30 smithi183 ceph-mon[41562]: mon.c@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2024-09-10T19:06:30.465 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:06:30 smithi183 ceph-mon[41562]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-09-10T19:06:30.465 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:06:30 smithi183 ceph-mon[41562]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-09-10T19:06:30.465 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:06:30 smithi183 ceph-mon[41562]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-09-10T19:06:30.465 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:06:30 smithi183 ceph-mon[41562]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T19:06:30.465 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:06:30 smithi183 ceph-mon[41562]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:06:30.465 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:06:30 smithi183 ceph-mon[41562]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T19:06:30.465 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:06:30 smithi183 ceph-mon[41562]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T19:06:30.466 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:06:30 smithi183 ceph-mon[41562]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T19:06:30.466 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:06:30 smithi183 ceph-mon[41562]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:06:30.466 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:06:30 smithi183 ceph-mon[41562]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-10T19:06:30.466 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:06:30 smithi183 ceph-mon[41562]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T19:06:30.466 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:06:30 smithi183 ceph-mon[41562]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi183:172.21.15.183=a;smithi183:[v2:172.21.15.183:3301,v1:172.21.15.183:6790]=c;smithi191:172.21.15.191=b", "target": ["mon-mgr", ""]}]: dispatch 2024-09-10T19:06:30.466 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:06:30 smithi183 ceph-mon[41562]: Saving service mon spec with placement smithi183:172.21.15.183=a;smithi183:[v2:172.21.15.183:3301,v1:172.21.15.183:6790]=c;smithi191:172.21.15.191=b;count:3 2024-09-10T19:06:30.466 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:06:30 smithi183 ceph-mon[41562]: Deploying daemon mon.b on smithi191 2024-09-10T19:06:30.466 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:06:30 smithi183 ceph-mon[41562]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T19:06:30.466 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:06:30 smithi183 ceph-mon[41562]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T19:06:30.466 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:06:30 smithi183 ceph-mon[41562]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-10T19:06:30.466 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:06:30 smithi183 ceph-mon[41562]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-10T19:06:30.466 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:06:30 smithi183 ceph-mon[41562]: mon.a calling monitor election 2024-09-10T19:06:30.466 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:06:30 smithi183 ceph-mon[41562]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T19:06:30.466 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:06:30 smithi183 ceph-mon[41562]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-10T19:06:30.467 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:06:30 smithi183 ceph-mon[41562]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-10T19:06:30.467 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:06:30 smithi183 ceph-mon[41562]: mon.b calling monitor election 2024-09-10T19:06:30.467 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:06:30 smithi183 ceph-mon[41562]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T19:06:30.467 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:06:30 smithi183 ceph-mon[41562]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-10T19:06:30.467 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:06:30 smithi183 ceph-mon[41562]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-10T19:06:30.467 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:06:30 smithi183 ceph-mon[41562]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T19:06:30.467 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:06:30 smithi183 ceph-mon[41562]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-10T19:06:30.467 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:06:30 smithi183 ceph-mon[41562]: mon.a is new leader, mons a,b in quorum (ranks 0,1) 2024-09-10T19:06:30.467 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:06:30 smithi183 ceph-mon[41562]: monmap e2: 2 mons at {a=[v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0],b=[v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-09-10T19:06:30.467 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:06:30 smithi183 ceph-mon[41562]: fsmap 2024-09-10T19:06:30.467 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:06:30 smithi183 ceph-mon[41562]: osdmap e4: 0 total, 0 up, 0 in 2024-09-10T19:06:30.467 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:06:30 smithi183 ceph-mon[41562]: mgrmap e13: y(active, since 38s) 2024-09-10T19:06:30.467 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:06:30 smithi183 ceph-mon[41562]: overall HEALTH_OK 2024-09-10T19:06:30.467 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:06:30 smithi183 ceph-mon[41562]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:06:30.468 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:06:30 smithi183 ceph-mon[41562]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:06:30.468 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:06:30 smithi183 ceph-mon[41562]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:06:30.468 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:06:30 smithi183 ceph-mon[41562]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-10T19:06:30.468 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:06:30 smithi183 ceph-mon[41562]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T19:06:30.468 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:06:30 smithi183 ceph-mon[41562]: Deploying daemon mon.c on smithi183 2024-09-10T19:06:30.468 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:06:30 smithi183 ceph-mon[41562]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-10T19:06:30.468 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:06:30 smithi183 ceph-mon[41562]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T19:06:30.468 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:06:30 smithi183 ceph-mon[41562]: from='client.? 172.21.15.191:0/469899973' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-10T19:06:30.468 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:06:30 smithi183 ceph-mon[41562]: mon.c@-1(synchronizing).paxosservice(auth 1..3) refresh upgraded, format 0 -> 3 2024-09-10T19:06:30.469 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:06:30 smithi183 bash[41525]: 193339ca1848d4955e928b9db4037cffbdebba48883ee47aee9f02dee955d48f 2024-09-10T19:06:30.469 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:06:30 smithi183 systemd[1]: Started Ceph mon.c for 320f6df8-6fa7-11ef-bcea-c7b262605968. 2024-09-10T19:06:30.469 INFO:teuthology.orchestra.run.smithi191.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.b/config 2024-09-10T19:06:35.465 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:06:35 smithi191 ceph-mon[32884]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-10T19:06:35.465 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:06:35 smithi191 ceph-mon[32884]: mon.a calling monitor election 2024-09-10T19:06:35.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:06:35 smithi191 ceph-mon[32884]: mon.b calling monitor election 2024-09-10T19:06:35.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:06:35 smithi191 ceph-mon[32884]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-10T19:06:35.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:06:35 smithi191 ceph-mon[32884]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-10T19:06:35.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:06:35 smithi191 ceph-mon[32884]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T19:06:35.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:06:35 smithi191 ceph-mon[32884]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-10T19:06:35.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:06:35 smithi191 ceph-mon[32884]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-10T19:06:35.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:06:35 smithi191 ceph-mon[32884]: mon.c calling monitor election 2024-09-10T19:06:35.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:06:35 smithi191 ceph-mon[32884]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T19:06:35.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:06:35 smithi191 ceph-mon[32884]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-10T19:06:35.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:06:35 smithi191 ceph-mon[32884]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-10T19:06:35.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:06:35 smithi191 ceph-mon[32884]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T19:06:35.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:06:35 smithi191 ceph-mon[32884]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-10T19:06:35.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:06:35 smithi191 ceph-mon[32884]: mon.a is new leader, mons a,b,c in quorum (ranks 0,1,2) 2024-09-10T19:06:35.467 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:06:35 smithi191 ceph-mon[32884]: monmap e3: 3 mons at {a=[v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0],b=[v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0],c=[v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0]} removed_ranks: {} disallowed_leaders: {} 2024-09-10T19:06:35.467 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:06:35 smithi191 ceph-mon[32884]: fsmap 2024-09-10T19:06:35.467 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:06:35 smithi191 ceph-mon[32884]: osdmap e4: 0 total, 0 up, 0 in 2024-09-10T19:06:35.467 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:06:35 smithi191 ceph-mon[32884]: mgrmap e13: y(active, since 46s) 2024-09-10T19:06:35.467 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:06:35 smithi191 ceph-mon[32884]: overall HEALTH_OK 2024-09-10T19:06:35.467 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:06:35 smithi191 ceph-mon[32884]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:06:35.467 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:06:35 smithi191 ceph-mon[32884]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:06:35.467 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:06:35 smithi191 ceph-mon[32884]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:06:35.467 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:06:35 smithi191 ceph-mon[32884]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:06:35.681 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:06:35 smithi183 ceph-mon[29736]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-10T19:06:35.681 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:06:35 smithi183 ceph-mon[29736]: mon.a calling monitor election 2024-09-10T19:06:35.682 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:06:35 smithi183 ceph-mon[29736]: mon.b calling monitor election 2024-09-10T19:06:35.682 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:06:35 smithi183 ceph-mon[29736]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-10T19:06:35.682 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:06:35 smithi183 ceph-mon[29736]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-10T19:06:35.682 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:06:35 smithi183 ceph-mon[29736]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T19:06:35.682 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:06:35 smithi183 ceph-mon[29736]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-10T19:06:35.682 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:06:35 smithi183 ceph-mon[29736]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-10T19:06:35.682 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:06:35 smithi183 ceph-mon[29736]: mon.c calling monitor election 2024-09-10T19:06:35.682 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:06:35 smithi183 ceph-mon[29736]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T19:06:35.682 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:06:35 smithi183 ceph-mon[29736]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-10T19:06:35.682 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:06:35 smithi183 ceph-mon[29736]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-10T19:06:35.682 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:06:35 smithi183 ceph-mon[29736]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T19:06:35.682 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:06:35 smithi183 ceph-mon[29736]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-10T19:06:35.682 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:06:35 smithi183 ceph-mon[29736]: mon.a is new leader, mons a,b,c in quorum (ranks 0,1,2) 2024-09-10T19:06:35.683 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:06:35 smithi183 ceph-mon[29736]: monmap e3: 3 mons at {a=[v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0],b=[v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0],c=[v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0]} removed_ranks: {} disallowed_leaders: {} 2024-09-10T19:06:35.683 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:06:35 smithi183 ceph-mon[29736]: fsmap 2024-09-10T19:06:35.683 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:06:35 smithi183 ceph-mon[29736]: osdmap e4: 0 total, 0 up, 0 in 2024-09-10T19:06:35.683 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:06:35 smithi183 ceph-mon[29736]: mgrmap e13: y(active, since 46s) 2024-09-10T19:06:35.683 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:06:35 smithi183 ceph-mon[29736]: overall HEALTH_OK 2024-09-10T19:06:35.683 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:06:35 smithi183 ceph-mon[29736]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:06:35.683 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:06:35 smithi183 ceph-mon[29736]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:06:35.683 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:06:35 smithi183 ceph-mon[29736]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:06:35.683 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:06:35 smithi183 ceph-mon[29736]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:06:35.683 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:06:35 smithi183 ceph-mon[41562]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-10T19:06:35.683 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:06:35 smithi183 ceph-mon[41562]: mon.a calling monitor election 2024-09-10T19:06:35.684 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:06:35 smithi183 ceph-mon[41562]: mon.b calling monitor election 2024-09-10T19:06:35.684 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:06:35 smithi183 ceph-mon[41562]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-10T19:06:35.684 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:06:35 smithi183 ceph-mon[41562]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-10T19:06:35.684 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:06:35 smithi183 ceph-mon[41562]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T19:06:35.684 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:06:35 smithi183 ceph-mon[41562]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-10T19:06:35.684 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:06:35 smithi183 ceph-mon[41562]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-10T19:06:35.684 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:06:35 smithi183 ceph-mon[41562]: mon.c calling monitor election 2024-09-10T19:06:35.684 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:06:35 smithi183 ceph-mon[41562]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T19:06:35.684 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:06:35 smithi183 ceph-mon[41562]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-10T19:06:35.684 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:06:35 smithi183 ceph-mon[41562]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-10T19:06:35.684 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:06:35 smithi183 ceph-mon[41562]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T19:06:35.684 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:06:35 smithi183 ceph-mon[41562]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-10T19:06:35.685 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:06:35 smithi183 ceph-mon[41562]: mon.a is new leader, mons a,b,c in quorum (ranks 0,1,2) 2024-09-10T19:06:35.685 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:06:35 smithi183 ceph-mon[41562]: monmap e3: 3 mons at {a=[v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0],b=[v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0],c=[v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0]} removed_ranks: {} disallowed_leaders: {} 2024-09-10T19:06:35.685 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:06:35 smithi183 ceph-mon[41562]: fsmap 2024-09-10T19:06:35.685 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:06:35 smithi183 ceph-mon[41562]: osdmap e4: 0 total, 0 up, 0 in 2024-09-10T19:06:35.685 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:06:35 smithi183 ceph-mon[41562]: mgrmap e13: y(active, since 46s) 2024-09-10T19:06:35.685 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:06:35 smithi183 ceph-mon[41562]: overall HEALTH_OK 2024-09-10T19:06:35.685 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:06:35 smithi183 ceph-mon[41562]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:06:35.685 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:06:35 smithi183 ceph-mon[41562]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:06:35.685 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:06:35 smithi183 ceph-mon[41562]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:06:35.685 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:06:35 smithi183 ceph-mon[41562]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:06:35.704 INFO:teuthology.orchestra.run.smithi191.stdout: 2024-09-10T19:06:35.704 INFO:teuthology.orchestra.run.smithi191.stdout:{"epoch":3,"fsid":"320f6df8-6fa7-11ef-bcea-c7b262605968","modified":"2024-09-10T19:06:30.124734Z","created":"2024-09-10T19:05:00.576431Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:3300","nonce":0},{"type":"v1","addr":"172.21.15.183:6789","nonce":0}]},"addr":"172.21.15.183:6789/0","public_addr":"172.21.15.183:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:3300","nonce":0},{"type":"v1","addr":"172.21.15.191:6789","nonce":0}]},"addr":"172.21.15.191:6789/0","public_addr":"172.21.15.191:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:3301","nonce":0},{"type":"v1","addr":"172.21.15.183:6790","nonce":0}]},"addr":"172.21.15.183:6790/0","public_addr":"172.21.15.183:6790/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1,2]} 2024-09-10T19:06:35.704 INFO:teuthology.orchestra.run.smithi191.stderr:dumped monmap epoch 3 2024-09-10T19:06:36.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:06:36 smithi183 ceph-mon[29736]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T19:06:36.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:06:36 smithi183 ceph-mon[29736]: from='client.? 172.21.15.191:0/108374370' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-10T19:06:36.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:06:36 smithi183 ceph-mon[29736]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-10T19:06:36.431 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:06:36 smithi183 ceph-mon[41562]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T19:06:36.431 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:06:36 smithi183 ceph-mon[41562]: from='client.? 172.21.15.191:0/108374370' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-10T19:06:36.431 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:06:36 smithi183 ceph-mon[41562]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-10T19:06:36.465 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:06:36 smithi191 ceph-mon[32884]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T19:06:36.465 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:06:36 smithi191 ceph-mon[32884]: from='client.? 172.21.15.191:0/108374370' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-10T19:06:36.465 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:06:36 smithi191 ceph-mon[32884]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-10T19:06:36.896 INFO:tasks.cephadm:Generating final ceph.conf file... 2024-09-10T19:06:36.896 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph config generate-minimal-conf 2024-09-10T19:06:37.147 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:06:37.522 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:06:37 smithi183 ceph-mon[29736]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T19:06:37.523 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:06:37 smithi183 ceph-mon[29736]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:06:37.523 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:06:37 smithi183 ceph-mon[29736]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:06:37.523 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:06:37 smithi183 ceph-mon[29736]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:06:37.523 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:06:37 smithi183 ceph-mon[29736]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:06:37.523 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:06:37 smithi183 ceph-mon[29736]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T19:06:37.524 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:06:37 smithi183 ceph-mon[29736]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T19:06:37.524 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:06:37 smithi183 ceph-mon[29736]: Updating smithi183:/etc/ceph/ceph.conf 2024-09-10T19:06:37.524 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:06:37 smithi183 ceph-mon[29736]: Updating smithi191:/etc/ceph/ceph.conf 2024-09-10T19:06:37.524 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:06:37 smithi183 ceph-mon[29736]: Updating smithi191:/var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/config/ceph.conf 2024-09-10T19:06:37.524 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:06:37 smithi183 ceph-mon[29736]: Updating smithi183:/var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/config/ceph.conf 2024-09-10T19:06:37.525 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:06:37 smithi183 ceph-mon[29736]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:06:37.525 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:06:37 smithi183 ceph-mon[29736]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:06:37.525 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:06:37 smithi183 ceph-mon[29736]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:06:37.525 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:06:37 smithi183 ceph-mon[29736]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:06:37.525 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:06:37 smithi183 ceph-mon[29736]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:06:37.525 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:06:37 smithi183 ceph-mon[29736]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-10T19:06:37.525 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:06:37 smithi183 ceph-mon[29736]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-10T19:06:37.525 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:06:37 smithi183 ceph-mon[29736]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T19:06:37.526 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:06:37 smithi183 ceph-mon[41562]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T19:06:37.526 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:06:37 smithi183 ceph-mon[41562]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:06:37.526 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:06:37 smithi183 ceph-mon[41562]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:06:37.526 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:06:37 smithi183 ceph-mon[41562]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:06:37.526 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:06:37 smithi183 ceph-mon[41562]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:06:37.526 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:06:37 smithi183 ceph-mon[41562]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T19:06:37.526 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:06:37 smithi183 ceph-mon[41562]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T19:06:37.526 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:06:37 smithi183 ceph-mon[41562]: Updating smithi183:/etc/ceph/ceph.conf 2024-09-10T19:06:37.526 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:06:37 smithi183 ceph-mon[41562]: Updating smithi191:/etc/ceph/ceph.conf 2024-09-10T19:06:37.526 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:06:37 smithi183 ceph-mon[41562]: Updating smithi191:/var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/config/ceph.conf 2024-09-10T19:06:37.526 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:06:37 smithi183 ceph-mon[41562]: Updating smithi183:/var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/config/ceph.conf 2024-09-10T19:06:37.526 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:06:37 smithi183 ceph-mon[41562]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:06:37.527 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:06:37 smithi183 ceph-mon[41562]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:06:37.527 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:06:37 smithi183 ceph-mon[41562]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:06:37.527 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:06:37 smithi183 ceph-mon[41562]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:06:37.527 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:06:37 smithi183 ceph-mon[41562]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:06:37.527 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:06:37 smithi183 ceph-mon[41562]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-10T19:06:37.527 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:06:37 smithi183 ceph-mon[41562]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-10T19:06:37.527 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:06:37 smithi183 ceph-mon[41562]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T19:06:37.715 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:06:37 smithi191 ceph-mon[32884]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T19:06:37.715 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:06:37 smithi191 ceph-mon[32884]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:06:37.715 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:06:37 smithi191 ceph-mon[32884]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:06:37.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:06:37 smithi191 ceph-mon[32884]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:06:37.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:06:37 smithi191 ceph-mon[32884]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:06:37.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:06:37 smithi191 ceph-mon[32884]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T19:06:37.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:06:37 smithi191 ceph-mon[32884]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T19:06:37.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:06:37 smithi191 ceph-mon[32884]: Updating smithi183:/etc/ceph/ceph.conf 2024-09-10T19:06:37.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:06:37 smithi191 ceph-mon[32884]: Updating smithi191:/etc/ceph/ceph.conf 2024-09-10T19:06:37.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:06:37 smithi191 ceph-mon[32884]: Updating smithi191:/var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/config/ceph.conf 2024-09-10T19:06:37.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:06:37 smithi191 ceph-mon[32884]: Updating smithi183:/var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/config/ceph.conf 2024-09-10T19:06:37.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:06:37 smithi191 ceph-mon[32884]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:06:37.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:06:37 smithi191 ceph-mon[32884]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:06:37.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:06:37 smithi191 ceph-mon[32884]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:06:37.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:06:37 smithi191 ceph-mon[32884]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:06:37.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:06:37 smithi191 ceph-mon[32884]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:06:37.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:06:37 smithi191 ceph-mon[32884]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-10T19:06:37.717 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:06:37 smithi191 ceph-mon[32884]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-10T19:06:37.717 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:06:37 smithi191 ceph-mon[32884]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T19:06:38.715 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:06:38 smithi191 ceph-mon[32884]: Reconfiguring mon.a (unknown last config time)... 2024-09-10T19:06:38.715 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:06:38 smithi191 ceph-mon[32884]: Reconfiguring daemon mon.a on smithi183 2024-09-10T19:06:38.715 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:06:38 smithi191 ceph-mon[32884]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:06:38.931 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:06:38 smithi183 ceph-mon[29736]: Reconfiguring mon.a (unknown last config time)... 2024-09-10T19:06:38.931 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:06:38 smithi183 ceph-mon[29736]: Reconfiguring daemon mon.a on smithi183 2024-09-10T19:06:38.931 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:06:38 smithi183 ceph-mon[29736]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:06:38.931 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:06:38 smithi183 ceph-mon[41562]: Reconfiguring mon.a (unknown last config time)... 2024-09-10T19:06:38.932 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:06:38 smithi183 ceph-mon[41562]: Reconfiguring daemon mon.a on smithi183 2024-09-10T19:06:38.932 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:06:38 smithi183 ceph-mon[41562]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:06:39.465 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:06:39 smithi191 ceph-mon[32884]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T19:06:39.725 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:06:39 smithi183 ceph-mon[29736]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T19:06:39.726 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:06:39 smithi183 ceph-mon[41562]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T19:06:39.902 INFO:teuthology.orchestra.run.smithi183.stdout:# minimal ceph.conf for 320f6df8-6fa7-11ef-bcea-c7b262605968 2024-09-10T19:06:39.902 INFO:teuthology.orchestra.run.smithi183.stdout:[global] 2024-09-10T19:06:39.902 INFO:teuthology.orchestra.run.smithi183.stdout: fsid = 320f6df8-6fa7-11ef-bcea-c7b262605968 2024-09-10T19:06:39.902 INFO:teuthology.orchestra.run.smithi183.stdout: mon_host = [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] 2024-09-10T19:06:40.681 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:06:40 smithi183 ceph-mon[29736]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:06:40.681 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:06:40 smithi183 ceph-mon[29736]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:06:40.681 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:06:40 smithi183 ceph-mon[29736]: Reconfiguring mon.c (monmap changed)... 2024-09-10T19:06:40.681 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:06:40 smithi183 ceph-mon[29736]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-10T19:06:40.681 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:06:40 smithi183 ceph-mon[29736]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-10T19:06:40.681 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:06:40 smithi183 ceph-mon[29736]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T19:06:40.682 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:06:40 smithi183 ceph-mon[29736]: Reconfiguring daemon mon.c on smithi183 2024-09-10T19:06:40.682 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:06:40 smithi183 ceph-mon[29736]: from='client.? 172.21.15.183:0/3843763429' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T19:06:40.682 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:06:40 smithi183 ceph-mon[41562]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:06:40.682 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:06:40 smithi183 ceph-mon[41562]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:06:40.682 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:06:40 smithi183 ceph-mon[41562]: Reconfiguring mon.c (monmap changed)... 2024-09-10T19:06:40.682 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:06:40 smithi183 ceph-mon[41562]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-10T19:06:40.682 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:06:40 smithi183 ceph-mon[41562]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-10T19:06:40.682 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:06:40 smithi183 ceph-mon[41562]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T19:06:40.682 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:06:40 smithi183 ceph-mon[41562]: Reconfiguring daemon mon.c on smithi183 2024-09-10T19:06:40.682 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:06:40 smithi183 ceph-mon[41562]: from='client.? 172.21.15.183:0/3843763429' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T19:06:40.965 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:06:40 smithi191 ceph-mon[32884]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:06:40.965 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:06:40 smithi191 ceph-mon[32884]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:06:40.965 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:06:40 smithi191 ceph-mon[32884]: Reconfiguring mon.c (monmap changed)... 2024-09-10T19:06:40.966 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:06:40 smithi191 ceph-mon[32884]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-10T19:06:40.966 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:06:40 smithi191 ceph-mon[32884]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-10T19:06:40.966 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:06:40 smithi191 ceph-mon[32884]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T19:06:40.966 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:06:40 smithi191 ceph-mon[32884]: Reconfiguring daemon mon.c on smithi183 2024-09-10T19:06:40.966 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:06:40 smithi191 ceph-mon[32884]: from='client.? 172.21.15.183:0/3843763429' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T19:06:41.034 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2024-09-10T19:06:41.034 DEBUG:teuthology.orchestra.run.smithi183:> set -ex 2024-09-10T19:06:41.034 DEBUG:teuthology.orchestra.run.smithi183:> sudo dd of=/etc/ceph/ceph.conf 2024-09-10T19:06:41.064 DEBUG:teuthology.orchestra.run.smithi183:> set -ex 2024-09-10T19:06:41.065 DEBUG:teuthology.orchestra.run.smithi183:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-09-10T19:06:41.131 DEBUG:teuthology.orchestra.run.smithi191:> set -ex 2024-09-10T19:06:41.131 DEBUG:teuthology.orchestra.run.smithi191:> sudo dd of=/etc/ceph/ceph.conf 2024-09-10T19:06:41.164 DEBUG:teuthology.orchestra.run.smithi191:> set -ex 2024-09-10T19:06:41.164 DEBUG:teuthology.orchestra.run.smithi191:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-09-10T19:06:41.233 INFO:tasks.cephadm:Adding mgr.y on smithi183 2024-09-10T19:06:41.233 INFO:tasks.cephadm:Adding mgr.x on smithi191 2024-09-10T19:06:41.234 DEBUG:teuthology.orchestra.run.smithi191:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph orch apply mgr '2;smithi183=y;smithi191=x' 2024-09-10T19:06:41.514 INFO:teuthology.orchestra.run.smithi191.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.b/config 2024-09-10T19:06:41.891 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:06:41 smithi191 ceph-mon[32884]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T19:06:41.891 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:06:41 smithi191 ceph-mon[32884]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:06:41.891 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:06:41 smithi191 ceph-mon[32884]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:06:41.891 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:06:41 smithi191 ceph-mon[32884]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-10T19:06:41.891 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:06:41 smithi191 ceph-mon[32884]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-10T19:06:41.891 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:06:41 smithi191 ceph-mon[32884]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T19:06:41.931 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:06:41 smithi183 ceph-mon[29736]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T19:06:41.931 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:06:41 smithi183 ceph-mon[29736]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:06:41.931 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:06:41 smithi183 ceph-mon[29736]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:06:41.931 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:06:41 smithi183 ceph-mon[29736]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-10T19:06:41.931 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:06:41 smithi183 ceph-mon[29736]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-10T19:06:41.932 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:06:41 smithi183 ceph-mon[29736]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T19:06:41.932 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:06:41 smithi183 ceph-mon[41562]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T19:06:41.932 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:06:41 smithi183 ceph-mon[41562]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:06:41.932 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:06:41 smithi183 ceph-mon[41562]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:06:41.932 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:06:41 smithi183 ceph-mon[41562]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-10T19:06:41.932 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:06:41 smithi183 ceph-mon[41562]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-10T19:06:41.932 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:06:41 smithi183 ceph-mon[41562]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T19:06:42.854 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:06:42 smithi191 ceph-mon[32884]: Reconfiguring mon.b (monmap changed)... 2024-09-10T19:06:42.854 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:06:42 smithi191 ceph-mon[32884]: Reconfiguring daemon mon.b on smithi191 2024-09-10T19:06:42.931 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:06:42 smithi183 ceph-mon[29736]: Reconfiguring mon.b (monmap changed)... 2024-09-10T19:06:42.931 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:06:42 smithi183 ceph-mon[29736]: Reconfiguring daemon mon.b on smithi191 2024-09-10T19:06:42.931 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:06:42 smithi183 ceph-mon[41562]: Reconfiguring mon.b (monmap changed)... 2024-09-10T19:06:42.931 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:06:42 smithi183 ceph-mon[41562]: Reconfiguring daemon mon.b on smithi191 2024-09-10T19:06:43.848 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:06:43 smithi191 ceph-mon[32884]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T19:06:43.848 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:06:43 smithi191 ceph-mon[32884]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:06:43.848 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:06:43 smithi191 ceph-mon[32884]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:06:43.848 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:06:43 smithi191 ceph-mon[32884]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T19:06:43.849 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:06:43 smithi191 ceph-mon[32884]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T19:06:43.849 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:06:43 smithi191 ceph-mon[32884]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T19:06:43.849 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:06:43 smithi191 ceph-mon[32884]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:06:43.932 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:06:43 smithi183 ceph-mon[29736]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T19:06:43.932 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:06:43 smithi183 ceph-mon[29736]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:06:43.932 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:06:43 smithi183 ceph-mon[29736]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:06:43.932 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:06:43 smithi183 ceph-mon[29736]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T19:06:43.932 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:06:43 smithi183 ceph-mon[29736]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T19:06:43.933 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:06:43 smithi183 ceph-mon[29736]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T19:06:43.933 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:06:43 smithi183 ceph-mon[29736]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:06:43.933 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:06:43 smithi183 ceph-mon[41562]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T19:06:43.933 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:06:43 smithi183 ceph-mon[41562]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:06:43.933 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:06:43 smithi183 ceph-mon[41562]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:06:43.933 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:06:43 smithi183 ceph-mon[41562]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T19:06:43.933 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:06:43 smithi183 ceph-mon[41562]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T19:06:43.933 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:06:43 smithi183 ceph-mon[41562]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T19:06:43.933 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:06:43 smithi183 ceph-mon[41562]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:06:44.226 INFO:teuthology.orchestra.run.smithi191.stdout:Scheduled mgr update... 2024-09-10T19:06:44.799 DEBUG:teuthology.orchestra.run.smithi191:mgr.x> sudo journalctl -f -n 0 -u ceph-320f6df8-6fa7-11ef-bcea-c7b262605968@mgr.x.service 2024-09-10T19:06:44.801 INFO:tasks.cephadm:Deploying OSDs... 2024-09-10T19:06:44.801 DEBUG:teuthology.orchestra.run.smithi183:> set -ex 2024-09-10T19:06:44.801 DEBUG:teuthology.orchestra.run.smithi183:> dd if=/scratch_devs of=/dev/stdout 2024-09-10T19:06:44.820 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-09-10T19:06:44.821 DEBUG:teuthology.orchestra.run.smithi183:> stat /dev/vg_nvme/lv_1 2024-09-10T19:06:44.878 INFO:teuthology.orchestra.run.smithi183.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-09-10T19:06:44.878 INFO:teuthology.orchestra.run.smithi183.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-10T19:06:44.878 INFO:teuthology.orchestra.run.smithi183.stdout:Device: 5h/5d Inode: 967 Links: 1 2024-09-10T19:06:44.878 INFO:teuthology.orchestra.run.smithi183.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-10T19:06:44.878 INFO:teuthology.orchestra.run.smithi183.stdout:Context: system_u:object_r:device_t:s0 2024-09-10T19:06:44.878 INFO:teuthology.orchestra.run.smithi183.stdout:Access: 2024-09-10 19:06:29.568690490 +0000 2024-09-10T19:06:44.878 INFO:teuthology.orchestra.run.smithi183.stdout:Modify: 2024-09-10 19:00:04.095438874 +0000 2024-09-10T19:06:44.879 INFO:teuthology.orchestra.run.smithi183.stdout:Change: 2024-09-10 19:00:04.095438874 +0000 2024-09-10T19:06:44.879 INFO:teuthology.orchestra.run.smithi183.stdout: Birth: 2024-09-10 19:00:04.095438874 +0000 2024-09-10T19:06:44.879 DEBUG:teuthology.orchestra.run.smithi183:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-09-10T19:06:44.946 INFO:teuthology.orchestra.run.smithi183.stderr:1+0 records in 2024-09-10T19:06:44.946 INFO:teuthology.orchestra.run.smithi183.stderr:1+0 records out 2024-09-10T19:06:44.946 INFO:teuthology.orchestra.run.smithi183.stderr:512 bytes copied, 0.000303101 s, 1.7 MB/s 2024-09-10T19:06:44.948 DEBUG:teuthology.orchestra.run.smithi183:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-09-10T19:06:45.007 DEBUG:teuthology.orchestra.run.smithi183:> stat /dev/vg_nvme/lv_2 2024-09-10T19:06:45.064 INFO:teuthology.orchestra.run.smithi183.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-09-10T19:06:45.065 INFO:teuthology.orchestra.run.smithi183.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-10T19:06:45.065 INFO:teuthology.orchestra.run.smithi183.stdout:Device: 5h/5d Inode: 979 Links: 1 2024-09-10T19:06:45.065 INFO:teuthology.orchestra.run.smithi183.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-10T19:06:45.065 INFO:teuthology.orchestra.run.smithi183.stdout:Context: system_u:object_r:device_t:s0 2024-09-10T19:06:45.065 INFO:teuthology.orchestra.run.smithi183.stdout:Access: 2024-09-10 19:06:29.568690490 +0000 2024-09-10T19:06:45.065 INFO:teuthology.orchestra.run.smithi183.stdout:Modify: 2024-09-10 19:00:04.095438874 +0000 2024-09-10T19:06:45.065 INFO:teuthology.orchestra.run.smithi183.stdout:Change: 2024-09-10 19:00:04.095438874 +0000 2024-09-10T19:06:45.065 INFO:teuthology.orchestra.run.smithi183.stdout: Birth: 2024-09-10 19:00:04.095438874 +0000 2024-09-10T19:06:45.065 DEBUG:teuthology.orchestra.run.smithi183:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-09-10T19:06:45.130 INFO:teuthology.orchestra.run.smithi183.stderr:1+0 records in 2024-09-10T19:06:45.130 INFO:teuthology.orchestra.run.smithi183.stderr:1+0 records out 2024-09-10T19:06:45.130 INFO:teuthology.orchestra.run.smithi183.stderr:512 bytes copied, 0.00020964 s, 2.4 MB/s 2024-09-10T19:06:45.131 DEBUG:teuthology.orchestra.run.smithi183:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-09-10T19:06:45.189 DEBUG:teuthology.orchestra.run.smithi183:> stat /dev/vg_nvme/lv_3 2024-09-10T19:06:45.246 INFO:teuthology.orchestra.run.smithi183.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-09-10T19:06:45.247 INFO:teuthology.orchestra.run.smithi183.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-10T19:06:45.247 INFO:teuthology.orchestra.run.smithi183.stdout:Device: 5h/5d Inode: 972 Links: 1 2024-09-10T19:06:45.247 INFO:teuthology.orchestra.run.smithi183.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-10T19:06:45.247 INFO:teuthology.orchestra.run.smithi183.stdout:Context: system_u:object_r:device_t:s0 2024-09-10T19:06:45.247 INFO:teuthology.orchestra.run.smithi183.stdout:Access: 2024-09-10 19:06:29.568690490 +0000 2024-09-10T19:06:45.247 INFO:teuthology.orchestra.run.smithi183.stdout:Modify: 2024-09-10 19:00:04.095438874 +0000 2024-09-10T19:06:45.247 INFO:teuthology.orchestra.run.smithi183.stdout:Change: 2024-09-10 19:00:04.095438874 +0000 2024-09-10T19:06:45.247 INFO:teuthology.orchestra.run.smithi183.stdout: Birth: 2024-09-10 19:00:04.095438874 +0000 2024-09-10T19:06:45.247 DEBUG:teuthology.orchestra.run.smithi183:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-09-10T19:06:45.312 INFO:teuthology.orchestra.run.smithi183.stderr:1+0 records in 2024-09-10T19:06:45.312 INFO:teuthology.orchestra.run.smithi183.stderr:1+0 records out 2024-09-10T19:06:45.312 INFO:teuthology.orchestra.run.smithi183.stderr:512 bytes copied, 0.00018898 s, 2.7 MB/s 2024-09-10T19:06:45.313 DEBUG:teuthology.orchestra.run.smithi183:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-09-10T19:06:45.370 DEBUG:teuthology.orchestra.run.smithi183:> stat /dev/vg_nvme/lv_4 2024-09-10T19:06:45.425 INFO:teuthology.orchestra.run.smithi183.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-09-10T19:06:45.425 INFO:teuthology.orchestra.run.smithi183.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-10T19:06:45.425 INFO:teuthology.orchestra.run.smithi183.stdout:Device: 5h/5d Inode: 984 Links: 1 2024-09-10T19:06:45.426 INFO:teuthology.orchestra.run.smithi183.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-10T19:06:45.426 INFO:teuthology.orchestra.run.smithi183.stdout:Context: system_u:object_r:device_t:s0 2024-09-10T19:06:45.426 INFO:teuthology.orchestra.run.smithi183.stdout:Access: 2024-09-10 19:06:29.568690490 +0000 2024-09-10T19:06:45.426 INFO:teuthology.orchestra.run.smithi183.stdout:Modify: 2024-09-10 19:00:04.096438649 +0000 2024-09-10T19:06:45.426 INFO:teuthology.orchestra.run.smithi183.stdout:Change: 2024-09-10 19:00:04.096438649 +0000 2024-09-10T19:06:45.426 INFO:teuthology.orchestra.run.smithi183.stdout: Birth: 2024-09-10 19:00:04.096438649 +0000 2024-09-10T19:06:45.426 DEBUG:teuthology.orchestra.run.smithi183:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-09-10T19:06:45.465 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:06:45 smithi191 ceph-mon[32884]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T19:06:45.465 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:06:45 smithi191 ceph-mon[32884]: from='client.24109 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi183=y;smithi191=x", "target": ["mon-mgr", ""]}]: dispatch 2024-09-10T19:06:45.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:06:45 smithi191 ceph-mon[32884]: Saving service mgr spec with placement smithi183=y;smithi191=x;count:2 2024-09-10T19:06:45.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:06:45 smithi191 ceph-mon[32884]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:06:45.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:06:45 smithi191 ceph-mon[32884]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T19:06:45.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:06:45 smithi191 ceph-mon[32884]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T19:06:45.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:06:45 smithi191 ceph-mon[32884]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T19:06:45.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:06:45 smithi191 ceph-mon[32884]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:06:45.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:06:45 smithi191 ceph-mon[32884]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-10T19:06:45.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:06:45 smithi191 ceph-mon[32884]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-09-10T19:06:45.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:06:45 smithi191 ceph-mon[32884]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-10T19:06:45.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:06:45 smithi191 ceph-mon[32884]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T19:06:45.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:06:45 smithi191 ceph-mon[32884]: Deploying daemon mgr.x on smithi191 2024-09-10T19:06:45.487 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:06:45 smithi183 ceph-mon[41562]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T19:06:45.487 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:06:45 smithi183 ceph-mon[41562]: from='client.24109 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi183=y;smithi191=x", "target": ["mon-mgr", ""]}]: dispatch 2024-09-10T19:06:45.488 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:06:45 smithi183 ceph-mon[41562]: Saving service mgr spec with placement smithi183=y;smithi191=x;count:2 2024-09-10T19:06:45.488 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:06:45 smithi183 ceph-mon[41562]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:06:45.488 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:06:45 smithi183 ceph-mon[41562]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T19:06:45.488 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:06:45 smithi183 ceph-mon[41562]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T19:06:45.488 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:06:45 smithi183 ceph-mon[41562]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T19:06:45.488 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:06:45 smithi183 ceph-mon[41562]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:06:45.488 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:06:45 smithi183 ceph-mon[41562]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-10T19:06:45.488 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:06:45 smithi183 ceph-mon[41562]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-09-10T19:06:45.488 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:06:45 smithi183 ceph-mon[41562]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-10T19:06:45.488 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:06:45 smithi183 ceph-mon[41562]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T19:06:45.488 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:06:45 smithi183 ceph-mon[41562]: Deploying daemon mgr.x on smithi191 2024-09-10T19:06:45.489 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:06:45 smithi183 ceph-mon[29736]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T19:06:45.489 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:06:45 smithi183 ceph-mon[29736]: from='client.24109 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi183=y;smithi191=x", "target": ["mon-mgr", ""]}]: dispatch 2024-09-10T19:06:45.489 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:06:45 smithi183 ceph-mon[29736]: Saving service mgr spec with placement smithi183=y;smithi191=x;count:2 2024-09-10T19:06:45.489 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:06:45 smithi183 ceph-mon[29736]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:06:45.489 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:06:45 smithi183 ceph-mon[29736]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T19:06:45.489 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:06:45 smithi183 ceph-mon[29736]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T19:06:45.489 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:06:45 smithi183 ceph-mon[29736]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T19:06:45.489 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:06:45 smithi183 ceph-mon[29736]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:06:45.489 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:06:45 smithi183 ceph-mon[29736]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-10T19:06:45.489 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:06:45 smithi183 ceph-mon[29736]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-09-10T19:06:45.489 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:06:45 smithi183 ceph-mon[29736]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-10T19:06:45.489 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:06:45 smithi183 ceph-mon[29736]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T19:06:45.489 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:06:45 smithi183 ceph-mon[29736]: Deploying daemon mgr.x on smithi191 2024-09-10T19:06:45.490 INFO:teuthology.orchestra.run.smithi183.stderr:1+0 records in 2024-09-10T19:06:45.490 INFO:teuthology.orchestra.run.smithi183.stderr:1+0 records out 2024-09-10T19:06:45.490 INFO:teuthology.orchestra.run.smithi183.stderr:512 bytes copied, 0.000193728 s, 2.6 MB/s 2024-09-10T19:06:45.491 DEBUG:teuthology.orchestra.run.smithi183:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-09-10T19:06:45.548 DEBUG:teuthology.orchestra.run.smithi191:> set -ex 2024-09-10T19:06:45.548 DEBUG:teuthology.orchestra.run.smithi191:> dd if=/scratch_devs of=/dev/stdout 2024-09-10T19:06:45.564 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-09-10T19:06:45.565 DEBUG:teuthology.orchestra.run.smithi191:> stat /dev/vg_nvme/lv_1 2024-09-10T19:06:45.621 INFO:teuthology.orchestra.run.smithi191.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-09-10T19:06:45.621 INFO:teuthology.orchestra.run.smithi191.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-10T19:06:45.621 INFO:teuthology.orchestra.run.smithi191.stdout:Device: 5h/5d Inode: 981 Links: 1 2024-09-10T19:06:45.621 INFO:teuthology.orchestra.run.smithi191.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-10T19:06:45.621 INFO:teuthology.orchestra.run.smithi191.stdout:Context: system_u:object_r:device_t:s0 2024-09-10T19:06:45.621 INFO:teuthology.orchestra.run.smithi191.stdout:Access: 2024-09-10 19:06:20.760515487 +0000 2024-09-10T19:06:45.621 INFO:teuthology.orchestra.run.smithi191.stdout:Modify: 2024-09-10 19:00:04.790688895 +0000 2024-09-10T19:06:45.621 INFO:teuthology.orchestra.run.smithi191.stdout:Change: 2024-09-10 19:00:04.790688895 +0000 2024-09-10T19:06:45.621 INFO:teuthology.orchestra.run.smithi191.stdout: Birth: 2024-09-10 19:00:04.790688895 +0000 2024-09-10T19:06:45.622 DEBUG:teuthology.orchestra.run.smithi191:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-09-10T19:06:45.686 INFO:teuthology.orchestra.run.smithi191.stderr:1+0 records in 2024-09-10T19:06:45.686 INFO:teuthology.orchestra.run.smithi191.stderr:1+0 records out 2024-09-10T19:06:45.686 INFO:teuthology.orchestra.run.smithi191.stderr:512 bytes copied, 0.000209793 s, 2.4 MB/s 2024-09-10T19:06:45.687 DEBUG:teuthology.orchestra.run.smithi191:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-09-10T19:06:45.743 DEBUG:teuthology.orchestra.run.smithi191:> stat /dev/vg_nvme/lv_2 2024-09-10T19:06:45.799 INFO:teuthology.orchestra.run.smithi191.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-09-10T19:06:45.799 INFO:teuthology.orchestra.run.smithi191.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-10T19:06:45.799 INFO:teuthology.orchestra.run.smithi191.stdout:Device: 5h/5d Inode: 966 Links: 1 2024-09-10T19:06:45.799 INFO:teuthology.orchestra.run.smithi191.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-10T19:06:45.799 INFO:teuthology.orchestra.run.smithi191.stdout:Context: system_u:object_r:device_t:s0 2024-09-10T19:06:45.799 INFO:teuthology.orchestra.run.smithi191.stdout:Access: 2024-09-10 19:06:20.760515487 +0000 2024-09-10T19:06:45.799 INFO:teuthology.orchestra.run.smithi191.stdout:Modify: 2024-09-10 19:00:04.789688704 +0000 2024-09-10T19:06:45.799 INFO:teuthology.orchestra.run.smithi191.stdout:Change: 2024-09-10 19:00:04.789688704 +0000 2024-09-10T19:06:45.799 INFO:teuthology.orchestra.run.smithi191.stdout: Birth: 2024-09-10 19:00:04.789688704 +0000 2024-09-10T19:06:45.800 DEBUG:teuthology.orchestra.run.smithi191:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-09-10T19:06:45.922 INFO:teuthology.orchestra.run.smithi191.stderr:1+0 records in 2024-09-10T19:06:45.922 INFO:teuthology.orchestra.run.smithi191.stderr:1+0 records out 2024-09-10T19:06:45.922 INFO:teuthology.orchestra.run.smithi191.stderr:512 bytes copied, 0.000217509 s, 2.4 MB/s 2024-09-10T19:06:45.923 DEBUG:teuthology.orchestra.run.smithi191:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-09-10T19:06:45.938 DEBUG:teuthology.orchestra.run.smithi191:> stat /dev/vg_nvme/lv_3 2024-09-10T19:06:45.998 INFO:teuthology.orchestra.run.smithi191.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-09-10T19:06:45.998 INFO:teuthology.orchestra.run.smithi191.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-10T19:06:45.998 INFO:teuthology.orchestra.run.smithi191.stdout:Device: 5h/5d Inode: 969 Links: 1 2024-09-10T19:06:45.998 INFO:teuthology.orchestra.run.smithi191.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-10T19:06:45.998 INFO:teuthology.orchestra.run.smithi191.stdout:Context: system_u:object_r:device_t:s0 2024-09-10T19:06:45.999 INFO:teuthology.orchestra.run.smithi191.stdout:Access: 2024-09-10 19:06:45.875104679 +0000 2024-09-10T19:06:45.999 INFO:teuthology.orchestra.run.smithi191.stdout:Modify: 2024-09-10 19:00:04.789688704 +0000 2024-09-10T19:06:45.999 INFO:teuthology.orchestra.run.smithi191.stdout:Change: 2024-09-10 19:00:04.789688704 +0000 2024-09-10T19:06:45.999 INFO:teuthology.orchestra.run.smithi191.stdout: Birth: 2024-09-10 19:00:04.789688704 +0000 2024-09-10T19:06:45.999 DEBUG:teuthology.orchestra.run.smithi191:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-09-10T19:06:46.112 INFO:teuthology.orchestra.run.smithi191.stderr:1+0 records in 2024-09-10T19:06:46.112 INFO:teuthology.orchestra.run.smithi191.stderr:1+0 records out 2024-09-10T19:06:46.112 INFO:teuthology.orchestra.run.smithi191.stderr:512 bytes copied, 0.000201544 s, 2.5 MB/s 2024-09-10T19:06:46.113 DEBUG:teuthology.orchestra.run.smithi191:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-09-10T19:06:46.129 DEBUG:teuthology.orchestra.run.smithi191:> stat /dev/vg_nvme/lv_4 2024-09-10T19:06:46.188 INFO:teuthology.orchestra.run.smithi191.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-09-10T19:06:46.189 INFO:teuthology.orchestra.run.smithi191.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-10T19:06:46.189 INFO:teuthology.orchestra.run.smithi191.stdout:Device: 5h/5d Inode: 976 Links: 1 2024-09-10T19:06:46.189 INFO:teuthology.orchestra.run.smithi191.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-10T19:06:46.189 INFO:teuthology.orchestra.run.smithi191.stdout:Context: system_u:object_r:device_t:s0 2024-09-10T19:06:46.189 INFO:teuthology.orchestra.run.smithi191.stdout:Access: 2024-09-10 19:06:46.067101541 +0000 2024-09-10T19:06:46.189 INFO:teuthology.orchestra.run.smithi191.stdout:Modify: 2024-09-10 19:00:04.789688704 +0000 2024-09-10T19:06:46.189 INFO:teuthology.orchestra.run.smithi191.stdout:Change: 2024-09-10 19:00:04.789688704 +0000 2024-09-10T19:06:46.189 INFO:teuthology.orchestra.run.smithi191.stdout: Birth: 2024-09-10 19:00:04.789688704 +0000 2024-09-10T19:06:46.190 DEBUG:teuthology.orchestra.run.smithi191:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-09-10T19:06:46.254 INFO:teuthology.orchestra.run.smithi191.stderr:1+0 records in 2024-09-10T19:06:46.254 INFO:teuthology.orchestra.run.smithi191.stderr:1+0 records out 2024-09-10T19:06:46.254 INFO:teuthology.orchestra.run.smithi191.stderr:512 bytes copied, 0.000196329 s, 2.6 MB/s 2024-09-10T19:06:46.256 DEBUG:teuthology.orchestra.run.smithi191:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-09-10T19:06:46.312 INFO:tasks.cephadm:Deploying osd.0 on smithi183 with /dev/vg_nvme/lv_4... 2024-09-10T19:06:46.313 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-09-10T19:06:46.554 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:06:46.595 INFO:journalctl@ceph.mgr.x.smithi191.stdout:Sep 10 19:06:46 smithi191 podman[34915]: 2024-09-10 19:06:46.572513867 +0000 UTC m=+0.347903712 container start 70400de28ce5b4cb0bc449ee51c920241cb369ae4214f43c28a3d7ce06bc20c2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3bfeaf4417585c821cab9ca76e4d95f390b0e769807122a45463640820b5cd16, name=ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-x, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, GIT_CLEAN=True, RELEASE=wip-guits-quincy-2024-09-09-1810-quincy-5be5739, org.label-schema.build-date=20240731) 2024-09-10T19:06:46.949 INFO:journalctl@ceph.mgr.x.smithi191.stdout:Sep 10 19:06:46 smithi191 ceph-mgr[34990]: -- 172.21.15.191:0/351907175 <== mon.1 v2:172.21.15.191:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x561bd1db2340 con 0x561bd1dbe800 2024-09-10T19:06:46.949 INFO:journalctl@ceph.mgr.x.smithi191.stdout:Sep 10 19:06:46 smithi191 bash[34915]: 70400de28ce5b4cb0bc449ee51c920241cb369ae4214f43c28a3d7ce06bc20c2 2024-09-10T19:06:46.950 INFO:journalctl@ceph.mgr.x.smithi191.stdout:Sep 10 19:06:46 smithi191 systemd[1]: Started Ceph mgr.x for 320f6df8-6fa7-11ef-bcea-c7b262605968. 2024-09-10T19:06:46.950 INFO:journalctl@ceph.mgr.x.smithi191.stdout:Sep 10 19:06:46 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-x[34967]: 2024-09-10T19:06:46.687+0000 7f2f64f62ec0 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-09-10T19:06:46.950 INFO:journalctl@ceph.mgr.x.smithi191.stdout:Sep 10 19:06:46 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-x[34967]: 2024-09-10T19:06:46.751+0000 7f2f64f62ec0 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-09-10T19:06:46.950 INFO:journalctl@ceph.mgr.x.smithi191.stdout:Sep 10 19:06:46 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-x[34967]: 2024-09-10T19:06:46.818+0000 7f2f64f62ec0 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-09-10T19:06:46.950 INFO:journalctl@ceph.mgr.x.smithi191.stdout:Sep 10 19:06:46 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-x[34967]: 2024-09-10T19:06:46.947+0000 7f2f64f62ec0 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-09-10T19:06:47.215 INFO:journalctl@ceph.mgr.x.smithi191.stdout:Sep 10 19:06:47 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-x[34967]: 2024-09-10T19:06:47.093+0000 7f2f64f62ec0 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-09-10T19:06:47.622 INFO:journalctl@ceph.mgr.x.smithi191.stdout:Sep 10 19:06:47 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-x[34967]: 2024-09-10T19:06:47.366+0000 7f2f64f62ec0 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-09-10T19:06:47.622 INFO:journalctl@ceph.mgr.x.smithi191.stdout:Sep 10 19:06:47 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-x[34967]: 2024-09-10T19:06:47.431+0000 7f2f64f62ec0 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-09-10T19:06:47.622 INFO:journalctl@ceph.mgr.x.smithi191.stdout:Sep 10 19:06:47 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-x[34967]: 2024-09-10T19:06:47.600+0000 7f2f64f62ec0 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-09-10T19:06:47.965 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:06:47 smithi191 ceph-mon[32884]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T19:06:47.965 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:06:47 smithi191 ceph-mon[32884]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:06:47.965 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:06:47 smithi191 ceph-mon[32884]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:06:47.965 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:06:47 smithi191 ceph-mon[32884]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:06:47.965 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:06:47 smithi191 ceph-mon[32884]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:06:47.965 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:06:47 smithi191 ceph-mon[32884]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T19:06:47.992 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:06:47 smithi183 ceph-mon[29736]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T19:06:47.992 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:06:47 smithi183 ceph-mon[29736]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:06:47.992 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:06:47 smithi183 ceph-mon[29736]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:06:47.992 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:06:47 smithi183 ceph-mon[29736]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:06:47.992 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:06:47 smithi183 ceph-mon[29736]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:06:47.992 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:06:47 smithi183 ceph-mon[29736]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T19:06:47.992 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:06:47 smithi183 ceph-mon[41562]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T19:06:47.992 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:06:47 smithi183 ceph-mon[41562]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:06:47.992 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:06:47 smithi183 ceph-mon[41562]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:06:47.993 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:06:47 smithi183 ceph-mon[41562]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:06:47.993 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:06:47 smithi183 ceph-mon[41562]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:06:47.993 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:06:47 smithi183 ceph-mon[41562]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T19:06:48.231 INFO:teuthology.orchestra.run.smithi183.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-09-10T19:06:48.314 INFO:teuthology.orchestra.run.smithi183.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-09-10T19:06:48.329 INFO:teuthology.orchestra.run.smithi183.stderr: stderr: 10+0 records in 2024-09-10T19:06:48.330 INFO:teuthology.orchestra.run.smithi183.stderr:10+0 records out 2024-09-10T19:06:48.330 INFO:teuthology.orchestra.run.smithi183.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0135566 s, 773 MB/s 2024-09-10T19:06:48.330 INFO:teuthology.orchestra.run.smithi183.stderr:--> Zapping successful for: 2024-09-10T19:06:48.715 INFO:journalctl@ceph.mgr.x.smithi191.stdout:Sep 10 19:06:48 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-x[34967]: 2024-09-10T19:06:48.452+0000 7f2f64f62ec0 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-09-10T19:06:48.715 INFO:journalctl@ceph.mgr.x.smithi191.stdout:Sep 10 19:06:48 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-x[34967]: 2024-09-10T19:06:48.516+0000 7f2f64f62ec0 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-09-10T19:06:49.052 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:06:48 smithi183 ceph-mon[41562]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:06:49.052 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:06:48 smithi183 ceph-mon[41562]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:06:49.052 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:06:48 smithi183 ceph-mon[41562]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T19:06:49.052 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:06:48 smithi183 ceph-mon[41562]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T19:06:49.052 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:06:48 smithi183 ceph-mon[41562]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:06:49.052 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:06:48 smithi183 ceph-mon[41562]: Reconfiguring mgr.y (unknown last config time)... 2024-09-10T19:06:49.052 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:06:48 smithi183 ceph-mon[41562]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-10T19:06:49.053 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:06:48 smithi183 ceph-mon[41562]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-10T19:06:49.053 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:06:48 smithi183 ceph-mon[41562]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T19:06:49.053 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:06:48 smithi183 ceph-mon[41562]: Reconfiguring daemon mgr.y on smithi183 2024-09-10T19:06:49.053 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:06:48 smithi183 ceph-mon[41562]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T19:06:49.053 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:06:48 smithi183 ceph-mon[41562]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:06:49.053 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:06:48 smithi183 ceph-mon[29736]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:06:49.053 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:06:48 smithi183 ceph-mon[29736]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:06:49.053 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:06:48 smithi183 ceph-mon[29736]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T19:06:49.054 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:06:48 smithi183 ceph-mon[29736]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T19:06:49.054 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:06:48 smithi183 ceph-mon[29736]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:06:49.054 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:06:48 smithi183 ceph-mon[29736]: Reconfiguring mgr.y (unknown last config time)... 2024-09-10T19:06:49.054 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:06:48 smithi183 ceph-mon[29736]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-10T19:06:49.054 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:06:48 smithi183 ceph-mon[29736]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-10T19:06:49.054 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:06:48 smithi183 ceph-mon[29736]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T19:06:49.054 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:06:48 smithi183 ceph-mon[29736]: Reconfiguring daemon mgr.y on smithi183 2024-09-10T19:06:49.054 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:06:48 smithi183 ceph-mon[29736]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T19:06:49.054 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:06:48 smithi183 ceph-mon[29736]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:06:49.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:06:48 smithi191 ceph-mon[32884]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:06:49.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:06:48 smithi191 ceph-mon[32884]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:06:49.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:06:48 smithi191 ceph-mon[32884]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T19:06:49.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:06:48 smithi191 ceph-mon[32884]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T19:06:49.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:06:48 smithi191 ceph-mon[32884]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:06:49.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:06:48 smithi191 ceph-mon[32884]: Reconfiguring mgr.y (unknown last config time)... 2024-09-10T19:06:49.217 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:06:48 smithi191 ceph-mon[32884]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-10T19:06:49.217 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:06:48 smithi191 ceph-mon[32884]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-10T19:06:49.217 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:06:48 smithi191 ceph-mon[32884]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T19:06:49.217 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:06:48 smithi191 ceph-mon[32884]: Reconfiguring daemon mgr.y on smithi183 2024-09-10T19:06:49.217 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:06:48 smithi191 ceph-mon[32884]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T19:06:49.217 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:06:48 smithi191 ceph-mon[32884]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:06:49.334 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph orch daemon add osd smithi183:vg_nvme/lv_4 2024-09-10T19:06:49.515 INFO:journalctl@ceph.mgr.x.smithi191.stdout:Sep 10 19:06:49 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-x[34967]: 2024-09-10T19:06:49.223+0000 7f2f64f62ec0 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-09-10T19:06:49.515 INFO:journalctl@ceph.mgr.x.smithi191.stdout:Sep 10 19:06:49 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-x[34967]: 2024-09-10T19:06:49.334+0000 7f2f64f62ec0 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-09-10T19:06:49.515 INFO:journalctl@ceph.mgr.x.smithi191.stdout:Sep 10 19:06:49 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-x[34967]: 2024-09-10T19:06:49.455+0000 7f2f64f62ec0 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-09-10T19:06:49.515 INFO:journalctl@ceph.mgr.x.smithi191.stdout:Sep 10 19:06:49 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-x[34967]: 2024-09-10T19:06:49.513+0000 7f2f64f62ec0 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-09-10T19:06:49.768 INFO:journalctl@ceph.mgr.x.smithi191.stdout:Sep 10 19:06:49 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-x[34967]: 2024-09-10T19:06:49.642+0000 7f2f64f62ec0 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-09-10T19:06:49.768 INFO:journalctl@ceph.mgr.x.smithi191.stdout:Sep 10 19:06:49 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-x[34967]: 2024-09-10T19:06:49.708+0000 7f2f64f62ec0 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-09-10T19:06:49.768 INFO:journalctl@ceph.mgr.x.smithi191.stdout:Sep 10 19:06:49 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-x[34967]: 2024-09-10T19:06:49.766+0000 7f2f64f62ec0 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-09-10T19:06:49.907 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:06:50.037 INFO:journalctl@ceph.mgr.x.smithi191.stdout:Sep 10 19:06:49 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-x[34967]: 2024-09-10T19:06:49.906+0000 7f2f64f62ec0 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-09-10T19:06:50.037 INFO:journalctl@ceph.mgr.x.smithi191.stdout:Sep 10 19:06:49 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-x[34967]: 2024-09-10T19:06:49.971+0000 7f2f64f62ec0 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-09-10T19:06:50.037 INFO:journalctl@ceph.mgr.x.smithi191.stdout:Sep 10 19:06:50 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-x[34967]: 2024-09-10T19:06:50.036+0000 7f2f64f62ec0 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-09-10T19:06:50.465 INFO:journalctl@ceph.mgr.x.smithi191.stdout:Sep 10 19:06:50 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-x[34967]: 2024-09-10T19:06:50.115+0000 7f2f64f62ec0 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-09-10T19:06:50.465 INFO:journalctl@ceph.mgr.x.smithi191.stdout:Sep 10 19:06:50 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-x[34967]: 2024-09-10T19:06:50.171+0000 7f2f64f62ec0 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-09-10T19:06:50.466 INFO:journalctl@ceph.mgr.x.smithi191.stdout:Sep 10 19:06:50 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-x[34967]: 2024-09-10T19:06:50.227+0000 7f2f64f62ec0 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-09-10T19:06:50.931 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:06:50 smithi183 ceph-mon[29736]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:06:50.931 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:06:50 smithi183 ceph-mon[29736]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:06:50.931 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:06:50 smithi183 ceph-mon[29736]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T19:06:50.931 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:06:50 smithi183 ceph-mon[29736]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T19:06:50.932 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:06:50 smithi183 ceph-mon[29736]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T19:06:50.932 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:06:50 smithi183 ceph-mon[29736]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:06:50.932 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:06:50 smithi183 ceph-mon[29736]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T19:06:50.932 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:06:50 smithi183 ceph-mon[41562]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:06:50.932 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:06:50 smithi183 ceph-mon[41562]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:06:50.932 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:06:50 smithi183 ceph-mon[41562]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T19:06:50.932 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:06:50 smithi183 ceph-mon[41562]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T19:06:50.932 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:06:50 smithi183 ceph-mon[41562]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T19:06:50.932 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:06:50 smithi183 ceph-mon[41562]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:06:50.932 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:06:50 smithi183 ceph-mon[41562]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T19:06:51.215 INFO:journalctl@ceph.mgr.x.smithi191.stdout:Sep 10 19:06:50 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-x[34967]: 2024-09-10T19:06:50.956+0000 7f2f64f62ec0 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-09-10T19:06:51.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:06:50 smithi191 ceph-mon[32884]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:06:51.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:06:50 smithi191 ceph-mon[32884]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:06:51.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:06:50 smithi191 ceph-mon[32884]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T19:06:51.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:06:50 smithi191 ceph-mon[32884]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T19:06:51.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:06:50 smithi191 ceph-mon[32884]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T19:06:51.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:06:50 smithi191 ceph-mon[32884]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:06:51.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:06:50 smithi191 ceph-mon[32884]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T19:06:52.118 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:06:51 smithi183 ceph-mon[41562]: Standby manager daemon x started 2024-09-10T19:06:52.118 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:06:51 smithi183 ceph-mon[41562]: from='mgr.? 172.21.15.191:0/2226855499' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-09-10T19:06:52.118 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:06:51 smithi183 ceph-mon[41562]: from='mgr.? 172.21.15.191:0/2226855499' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-10T19:06:52.118 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:06:51 smithi183 ceph-mon[41562]: from='mgr.? 172.21.15.191:0/2226855499' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-09-10T19:06:52.118 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:06:51 smithi183 ceph-mon[41562]: from='mgr.? 172.21.15.191:0/2226855499' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-10T19:06:52.118 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:06:51 smithi183 ceph-mon[41562]: from='client.24119 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi183:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-09-10T19:06:52.118 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:06:51 smithi183 ceph-mon[41562]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-10T19:06:52.118 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:06:51 smithi183 ceph-mon[41562]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-10T19:06:52.118 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:06:51 smithi183 ceph-mon[41562]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T19:06:52.119 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:06:51 smithi183 ceph-mon[29736]: Standby manager daemon x started 2024-09-10T19:06:52.119 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:06:51 smithi183 ceph-mon[29736]: from='mgr.? 172.21.15.191:0/2226855499' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-09-10T19:06:52.119 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:06:51 smithi183 ceph-mon[29736]: from='mgr.? 172.21.15.191:0/2226855499' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-10T19:06:52.119 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:06:51 smithi183 ceph-mon[29736]: from='mgr.? 172.21.15.191:0/2226855499' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-09-10T19:06:52.119 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:06:51 smithi183 ceph-mon[29736]: from='mgr.? 172.21.15.191:0/2226855499' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-10T19:06:52.119 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:06:51 smithi183 ceph-mon[29736]: from='client.24119 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi183:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-09-10T19:06:52.119 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:06:51 smithi183 ceph-mon[29736]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-10T19:06:52.119 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:06:51 smithi183 ceph-mon[29736]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-10T19:06:52.119 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:06:51 smithi183 ceph-mon[29736]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T19:06:52.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:06:51 smithi191 ceph-mon[32884]: Standby manager daemon x started 2024-09-10T19:06:52.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:06:51 smithi191 ceph-mon[32884]: from='mgr.? 172.21.15.191:0/2226855499' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-09-10T19:06:52.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:06:51 smithi191 ceph-mon[32884]: from='mgr.? 172.21.15.191:0/2226855499' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-10T19:06:52.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:06:51 smithi191 ceph-mon[32884]: from='mgr.? 172.21.15.191:0/2226855499' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-09-10T19:06:52.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:06:51 smithi191 ceph-mon[32884]: from='mgr.? 172.21.15.191:0/2226855499' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-10T19:06:52.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:06:51 smithi191 ceph-mon[32884]: from='client.24119 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi183:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-09-10T19:06:52.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:06:51 smithi191 ceph-mon[32884]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-10T19:06:52.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:06:51 smithi191 ceph-mon[32884]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-10T19:06:52.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:06:51 smithi191 ceph-mon[32884]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T19:06:52.968 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:06:52 smithi183 ceph-mon[29736]: mgrmap e14: y(active, since 63s), standbys: x 2024-09-10T19:06:52.968 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:06:52 smithi183 ceph-mon[29736]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-09-10T19:06:52.968 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:06:52 smithi183 ceph-mon[29736]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T19:06:52.969 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:06:52 smithi183 ceph-mon[41562]: mgrmap e14: y(active, since 63s), standbys: x 2024-09-10T19:06:52.969 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:06:52 smithi183 ceph-mon[41562]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-09-10T19:06:52.969 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:06:52 smithi183 ceph-mon[41562]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T19:06:53.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:06:52 smithi191 ceph-mon[32884]: mgrmap e14: y(active, since 63s), standbys: x 2024-09-10T19:06:53.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:06:52 smithi191 ceph-mon[32884]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-09-10T19:06:53.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:06:52 smithi191 ceph-mon[32884]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T19:06:54.431 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:06:54 smithi183 ceph-mon[41562]: from='client.? 172.21.15.183:0/1200124877' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9d323010-394a-49aa-ba9b-244d87faf36e"}]: dispatch 2024-09-10T19:06:54.431 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:06:54 smithi183 ceph-mon[41562]: from='client.? 172.21.15.183:0/1200124877' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "9d323010-394a-49aa-ba9b-244d87faf36e"}]': finished 2024-09-10T19:06:54.431 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:06:54 smithi183 ceph-mon[41562]: osdmap e5: 1 total, 0 up, 1 in 2024-09-10T19:06:54.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:06:54 smithi183 ceph-mon[41562]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-10T19:06:54.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:06:54 smithi183 ceph-mon[29736]: from='client.? 172.21.15.183:0/1200124877' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9d323010-394a-49aa-ba9b-244d87faf36e"}]: dispatch 2024-09-10T19:06:54.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:06:54 smithi183 ceph-mon[29736]: from='client.? 172.21.15.183:0/1200124877' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "9d323010-394a-49aa-ba9b-244d87faf36e"}]': finished 2024-09-10T19:06:54.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:06:54 smithi183 ceph-mon[29736]: osdmap e5: 1 total, 0 up, 1 in 2024-09-10T19:06:54.433 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:06:54 smithi183 ceph-mon[29736]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-10T19:06:54.465 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:06:54 smithi191 ceph-mon[32884]: from='client.? 172.21.15.183:0/1200124877' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9d323010-394a-49aa-ba9b-244d87faf36e"}]: dispatch 2024-09-10T19:06:54.465 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:06:54 smithi191 ceph-mon[32884]: from='client.? 172.21.15.183:0/1200124877' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "9d323010-394a-49aa-ba9b-244d87faf36e"}]': finished 2024-09-10T19:06:54.465 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:06:54 smithi191 ceph-mon[32884]: osdmap e5: 1 total, 0 up, 1 in 2024-09-10T19:06:54.465 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:06:54 smithi191 ceph-mon[32884]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-10T19:06:55.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:06:55 smithi183 ceph-mon[29736]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T19:06:55.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:06:55 smithi183 ceph-mon[29736]: from='client.? 172.21.15.183:0/2779250942' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-10T19:06:55.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:06:55 smithi183 ceph-mon[41562]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T19:06:55.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:06:55 smithi183 ceph-mon[41562]: from='client.? 172.21.15.183:0/2779250942' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-10T19:06:55.465 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:06:55 smithi191 ceph-mon[32884]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T19:06:55.465 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:06:55 smithi191 ceph-mon[32884]: from='client.? 172.21.15.183:0/2779250942' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-10T19:06:57.410 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:06:57 smithi183 ceph-mon[29736]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T19:06:57.410 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:06:57 smithi183 ceph-mon[41562]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T19:06:57.531 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:06:57 smithi191 ceph-mon[32884]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T19:06:59.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:06:59 smithi183 ceph-mon[29736]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T19:06:59.431 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:06:59 smithi183 ceph-mon[41562]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T19:06:59.715 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:06:59 smithi191 ceph-mon[32884]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T19:07:00.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:07:00 smithi183 ceph-mon[29736]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-09-10T19:07:00.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:07:00 smithi183 ceph-mon[29736]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T19:07:00.431 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:07:00 smithi183 ceph-mon[41562]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-09-10T19:07:00.431 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:07:00 smithi183 ceph-mon[41562]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T19:07:00.715 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:07:00 smithi191 ceph-mon[32884]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-09-10T19:07:00.715 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:07:00 smithi191 ceph-mon[32884]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T19:07:01.519 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:07:01 smithi183 ceph-mon[29736]: Deploying daemon osd.0 on smithi183 2024-09-10T19:07:01.519 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:07:01 smithi183 ceph-mon[29736]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T19:07:01.519 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:07:01 smithi183 ceph-mon[41562]: Deploying daemon osd.0 on smithi183 2024-09-10T19:07:01.519 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:07:01 smithi183 ceph-mon[41562]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T19:07:01.715 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:07:01 smithi191 ceph-mon[32884]: Deploying daemon osd.0 on smithi183 2024-09-10T19:07:01.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:07:01 smithi191 ceph-mon[32884]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T19:07:03.562 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:07:03 smithi183 ceph-mon[29736]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T19:07:03.563 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:07:03 smithi183 ceph-mon[41562]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T19:07:03.715 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:07:03 smithi191 ceph-mon[32884]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T19:07:05.503 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:07:05 smithi183 ceph-mon[29736]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T19:07:05.503 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:07:05 smithi183 ceph-mon[29736]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T19:07:05.503 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:07:05 smithi183 ceph-mon[29736]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:07:05.504 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:07:05 smithi183 ceph-mon[29736]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:07:05.504 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:07:05 smithi183 ceph-mon[41562]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T19:07:05.504 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:07:05 smithi183 ceph-mon[41562]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T19:07:05.504 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:07:05 smithi183 ceph-mon[41562]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:07:05.504 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:07:05 smithi183 ceph-mon[41562]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:07:05.715 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:07:05 smithi191 ceph-mon[32884]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T19:07:05.715 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:07:05 smithi191 ceph-mon[32884]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T19:07:05.715 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:07:05 smithi191 ceph-mon[32884]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:07:05.715 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:07:05 smithi191 ceph-mon[32884]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:07:07.329 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:07:07 smithi183 ceph-mon[29736]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T19:07:07.330 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:07:07 smithi183 ceph-mon[41562]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T19:07:07.715 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:07:07 smithi191 ceph-mon[32884]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T19:07:08.444 INFO:teuthology.orchestra.run.smithi183.stdout:Created osd(s) 0 on host 'smithi183' 2024-09-10T19:07:08.965 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:07:08 smithi191 ceph-mon[32884]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:07:08.965 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:07:08 smithi191 ceph-mon[32884]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:07:08.965 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:07:08 smithi191 ceph-mon[32884]: from='osd.0 [v2:172.21.15.183:6802/2813074952,v1:172.21.15.183:6803/2813074952]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-09-10T19:07:08.966 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:07:08 smithi191 ceph-mon[32884]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-09-10T19:07:08.966 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:07:08 smithi191 ceph-mon[32884]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T19:07:08.966 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:07:08 smithi191 ceph-mon[32884]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T19:07:08.966 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:07:08 smithi191 ceph-mon[32884]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:07:08.966 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:07:08 smithi191 ceph-mon[32884]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T19:07:08.966 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:07:08 smithi191 ceph-mon[32884]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:07:08.966 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:07:08 smithi191 ceph-mon[32884]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:07:09.030 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:07:08 smithi183 ceph-mon[29736]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:07:09.030 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:07:08 smithi183 ceph-mon[29736]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:07:09.030 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:07:08 smithi183 ceph-mon[29736]: from='osd.0 [v2:172.21.15.183:6802/2813074952,v1:172.21.15.183:6803/2813074952]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-09-10T19:07:09.030 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:07:08 smithi183 ceph-mon[29736]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-09-10T19:07:09.030 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:07:08 smithi183 ceph-mon[29736]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T19:07:09.030 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:07:08 smithi183 ceph-mon[29736]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T19:07:09.030 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:07:08 smithi183 ceph-mon[29736]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:07:09.031 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:07:08 smithi183 ceph-mon[29736]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T19:07:09.031 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:07:08 smithi183 ceph-mon[29736]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:07:09.031 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:07:08 smithi183 ceph-mon[29736]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:07:09.031 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:07:08 smithi183 ceph-mon[41562]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:07:09.031 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:07:08 smithi183 ceph-mon[41562]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:07:09.031 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:07:08 smithi183 ceph-mon[41562]: from='osd.0 [v2:172.21.15.183:6802/2813074952,v1:172.21.15.183:6803/2813074952]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-09-10T19:07:09.031 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:07:08 smithi183 ceph-mon[41562]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-09-10T19:07:09.031 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:07:08 smithi183 ceph-mon[41562]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T19:07:09.031 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:07:08 smithi183 ceph-mon[41562]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T19:07:09.031 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:07:08 smithi183 ceph-mon[41562]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:07:09.031 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:07:08 smithi183 ceph-mon[41562]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T19:07:09.032 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:07:08 smithi183 ceph-mon[41562]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:07:09.032 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:07:08 smithi183 ceph-mon[41562]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:07:09.053 DEBUG:teuthology.orchestra.run.smithi183:osd.0> sudo journalctl -f -n 0 -u ceph-320f6df8-6fa7-11ef-bcea-c7b262605968@osd.0.service 2024-09-10T19:07:09.056 INFO:tasks.cephadm:Deploying osd.1 on smithi183 with /dev/vg_nvme/lv_3... 2024-09-10T19:07:09.056 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-09-10T19:07:09.805 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:07:10.061 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:07:09 smithi183 ceph-mon[41562]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T19:07:10.061 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:07:09 smithi183 ceph-mon[41562]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-09-10T19:07:10.061 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:07:09 smithi183 ceph-mon[41562]: osdmap e6: 1 total, 0 up, 1 in 2024-09-10T19:07:10.061 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:07:09 smithi183 ceph-mon[41562]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-10T19:07:10.061 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:07:09 smithi183 ceph-mon[41562]: from='osd.0 [v2:172.21.15.183:6802/2813074952,v1:172.21.15.183:6803/2813074952]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi183", "root=default"]}]: dispatch 2024-09-10T19:07:10.062 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:07:09 smithi183 ceph-mon[41562]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi183", "root=default"]}]: dispatch 2024-09-10T19:07:10.062 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:07:09 smithi183 ceph-mon[29736]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T19:07:10.062 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:07:09 smithi183 ceph-mon[29736]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-09-10T19:07:10.062 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:07:09 smithi183 ceph-mon[29736]: osdmap e6: 1 total, 0 up, 1 in 2024-09-10T19:07:10.063 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:07:09 smithi183 ceph-mon[29736]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-10T19:07:10.063 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:07:09 smithi183 ceph-mon[29736]: from='osd.0 [v2:172.21.15.183:6802/2813074952,v1:172.21.15.183:6803/2813074952]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi183", "root=default"]}]: dispatch 2024-09-10T19:07:10.063 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:07:09 smithi183 ceph-mon[29736]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi183", "root=default"]}]: dispatch 2024-09-10T19:07:10.063 INFO:journalctl@ceph.osd.0.smithi183.stdout:Sep 10 19:07:09 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd-0[45861]: 2024-09-10T19:07:09.772+0000 7f96e646f640 -1 osd.0 0 waiting for initial osdmap 2024-09-10T19:07:10.063 INFO:journalctl@ceph.osd.0.smithi183.stdout:Sep 10 19:07:09 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd-0[45861]: 2024-09-10T19:07:09.775+0000 7f96e1284640 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-10T19:07:10.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:07:09 smithi191 ceph-mon[32884]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T19:07:10.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:07:09 smithi191 ceph-mon[32884]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-09-10T19:07:10.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:07:09 smithi191 ceph-mon[32884]: osdmap e6: 1 total, 0 up, 1 in 2024-09-10T19:07:10.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:07:09 smithi191 ceph-mon[32884]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-10T19:07:10.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:07:09 smithi191 ceph-mon[32884]: from='osd.0 [v2:172.21.15.183:6802/2813074952,v1:172.21.15.183:6803/2813074952]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi183", "root=default"]}]: dispatch 2024-09-10T19:07:10.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:07:09 smithi191 ceph-mon[32884]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi183", "root=default"]}]: dispatch 2024-09-10T19:07:11.167 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:07:10 smithi183 ceph-mon[41562]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi183", "root=default"]}]': finished 2024-09-10T19:07:11.167 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:07:10 smithi183 ceph-mon[41562]: osdmap e7: 1 total, 0 up, 1 in 2024-09-10T19:07:11.167 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:07:10 smithi183 ceph-mon[41562]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-10T19:07:11.167 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:07:10 smithi183 ceph-mon[41562]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-10T19:07:11.168 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:07:10 smithi183 ceph-mon[41562]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-10T19:07:11.168 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:07:10 smithi183 ceph-mon[29736]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi183", "root=default"]}]': finished 2024-09-10T19:07:11.168 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:07:10 smithi183 ceph-mon[29736]: osdmap e7: 1 total, 0 up, 1 in 2024-09-10T19:07:11.168 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:07:10 smithi183 ceph-mon[29736]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-10T19:07:11.168 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:07:10 smithi183 ceph-mon[29736]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-10T19:07:11.168 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:07:10 smithi183 ceph-mon[29736]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-10T19:07:11.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:07:10 smithi191 ceph-mon[32884]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi183", "root=default"]}]': finished 2024-09-10T19:07:11.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:07:10 smithi191 ceph-mon[32884]: osdmap e7: 1 total, 0 up, 1 in 2024-09-10T19:07:11.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:07:10 smithi191 ceph-mon[32884]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-10T19:07:11.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:07:10 smithi191 ceph-mon[32884]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-10T19:07:11.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:07:10 smithi191 ceph-mon[32884]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-10T19:07:12.072 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:07:11 smithi183 ceph-mon[41562]: purged_snaps scrub starts 2024-09-10T19:07:12.072 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:07:11 smithi183 ceph-mon[41562]: purged_snaps scrub ok 2024-09-10T19:07:12.072 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:07:11 smithi183 ceph-mon[41562]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T19:07:12.072 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:07:11 smithi183 ceph-mon[41562]: osd.0 [v2:172.21.15.183:6802/2813074952,v1:172.21.15.183:6803/2813074952] boot 2024-09-10T19:07:12.072 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:07:11 smithi183 ceph-mon[41562]: osdmap e8: 1 total, 1 up, 1 in 2024-09-10T19:07:12.072 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:07:11 smithi183 ceph-mon[41562]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-10T19:07:12.072 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:07:11 smithi183 ceph-mon[29736]: purged_snaps scrub starts 2024-09-10T19:07:12.073 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:07:11 smithi183 ceph-mon[29736]: purged_snaps scrub ok 2024-09-10T19:07:12.073 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:07:11 smithi183 ceph-mon[29736]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T19:07:12.073 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:07:11 smithi183 ceph-mon[29736]: osd.0 [v2:172.21.15.183:6802/2813074952,v1:172.21.15.183:6803/2813074952] boot 2024-09-10T19:07:12.073 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:07:11 smithi183 ceph-mon[29736]: osdmap e8: 1 total, 1 up, 1 in 2024-09-10T19:07:12.073 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:07:11 smithi183 ceph-mon[29736]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-10T19:07:12.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:07:11 smithi191 ceph-mon[32884]: purged_snaps scrub starts 2024-09-10T19:07:12.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:07:11 smithi191 ceph-mon[32884]: purged_snaps scrub ok 2024-09-10T19:07:12.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:07:11 smithi191 ceph-mon[32884]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-10T19:07:12.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:07:11 smithi191 ceph-mon[32884]: osd.0 [v2:172.21.15.183:6802/2813074952,v1:172.21.15.183:6803/2813074952] boot 2024-09-10T19:07:12.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:07:11 smithi191 ceph-mon[32884]: osdmap e8: 1 total, 1 up, 1 in 2024-09-10T19:07:12.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:07:11 smithi191 ceph-mon[32884]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-10T19:07:12.315 INFO:teuthology.orchestra.run.smithi183.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-09-10T19:07:12.423 INFO:teuthology.orchestra.run.smithi183.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-09-10T19:07:12.440 INFO:teuthology.orchestra.run.smithi183.stderr: stderr: 10+0 records in 2024-09-10T19:07:12.440 INFO:teuthology.orchestra.run.smithi183.stderr:10+0 records out 2024-09-10T19:07:12.440 INFO:teuthology.orchestra.run.smithi183.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0150338 s, 697 MB/s 2024-09-10T19:07:12.440 INFO:teuthology.orchestra.run.smithi183.stderr:--> Zapping successful for: 2024-09-10T19:07:13.119 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:07:12 smithi183 ceph-mon[29736]: osdmap e9: 1 total, 1 up, 1 in 2024-09-10T19:07:13.119 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:07:12 smithi183 ceph-mon[41562]: osdmap e9: 1 total, 1 up, 1 in 2024-09-10T19:07:13.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:07:12 smithi191 ceph-mon[32884]: osdmap e9: 1 total, 1 up, 1 in 2024-09-10T19:07:13.444 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph orch daemon add osd smithi183:vg_nvme/lv_3 2024-09-10T19:07:13.683 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:07:14.056 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:07:13 smithi183 ceph-mon[41562]: pgmap v41: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-10T19:07:14.056 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:07:13 smithi183 ceph-mon[41562]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:07:14.057 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:07:13 smithi183 ceph-mon[41562]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:07:14.057 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:07:13 smithi183 ceph-mon[41562]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-10T19:07:14.057 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:07:13 smithi183 ceph-mon[41562]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:07:14.057 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:07:13 smithi183 ceph-mon[41562]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T19:07:14.057 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:07:13 smithi183 ceph-mon[41562]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T19:07:14.057 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:07:13 smithi183 ceph-mon[41562]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:07:14.057 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:07:13 smithi183 ceph-mon[29736]: pgmap v41: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-10T19:07:14.057 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:07:13 smithi183 ceph-mon[29736]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:07:14.057 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:07:13 smithi183 ceph-mon[29736]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:07:14.057 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:07:13 smithi183 ceph-mon[29736]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-10T19:07:14.057 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:07:13 smithi183 ceph-mon[29736]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:07:14.058 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:07:13 smithi183 ceph-mon[29736]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T19:07:14.058 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:07:13 smithi183 ceph-mon[29736]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T19:07:14.058 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:07:13 smithi183 ceph-mon[29736]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:07:14.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:07:13 smithi191 ceph-mon[32884]: pgmap v41: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-10T19:07:14.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:07:13 smithi191 ceph-mon[32884]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:07:14.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:07:13 smithi191 ceph-mon[32884]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:07:14.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:07:13 smithi191 ceph-mon[32884]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-10T19:07:14.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:07:13 smithi191 ceph-mon[32884]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:07:14.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:07:13 smithi191 ceph-mon[32884]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T19:07:14.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:07:13 smithi191 ceph-mon[32884]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T19:07:14.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:07:13 smithi191 ceph-mon[32884]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:07:15.063 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:07:14 smithi183 ceph-mon[29736]: Detected new or changed devices on smithi183 2024-09-10T19:07:15.063 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:07:14 smithi183 ceph-mon[29736]: Adjusting osd_memory_target on smithi183 to 16064M 2024-09-10T19:07:15.063 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:07:14 smithi183 ceph-mon[41562]: Detected new or changed devices on smithi183 2024-09-10T19:07:15.063 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:07:14 smithi183 ceph-mon[41562]: Adjusting osd_memory_target on smithi183 to 16064M 2024-09-10T19:07:15.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:07:14 smithi191 ceph-mon[32884]: Detected new or changed devices on smithi183 2024-09-10T19:07:15.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:07:14 smithi191 ceph-mon[32884]: Adjusting osd_memory_target on smithi183 to 16064M 2024-09-10T19:07:16.125 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:07:15 smithi183 ceph-mon[29736]: pgmap v42: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-10T19:07:16.126 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:07:15 smithi183 ceph-mon[29736]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-10T19:07:16.126 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:07:15 smithi183 ceph-mon[29736]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-10T19:07:16.126 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:07:15 smithi183 ceph-mon[29736]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T19:07:16.126 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:07:15 smithi183 ceph-mon[41562]: pgmap v42: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-10T19:07:16.126 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:07:15 smithi183 ceph-mon[41562]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-10T19:07:16.126 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:07:15 smithi183 ceph-mon[41562]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-10T19:07:16.126 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:07:15 smithi183 ceph-mon[41562]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T19:07:16.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:07:15 smithi191 ceph-mon[32884]: pgmap v42: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-10T19:07:16.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:07:15 smithi191 ceph-mon[32884]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-10T19:07:16.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:07:15 smithi191 ceph-mon[32884]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-10T19:07:16.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:07:15 smithi191 ceph-mon[32884]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T19:07:17.073 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:07:16 smithi183 ceph-mon[29736]: from='client.24146 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi183:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-09-10T19:07:17.074 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:07:16 smithi183 ceph-mon[41562]: from='client.24146 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi183:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-09-10T19:07:17.214 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:07:16 smithi191 ceph-mon[32884]: from='client.24146 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi183:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-09-10T19:07:18.181 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:07:17 smithi183 ceph-mon[41562]: pgmap v43: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-10T19:07:18.181 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:07:17 smithi183 ceph-mon[41562]: from='client.? 172.21.15.183:0/3328875564' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "71a6c5dc-5109-4284-8d20-187d8e7afc9e"}]: dispatch 2024-09-10T19:07:18.181 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:07:17 smithi183 ceph-mon[41562]: from='client.? 172.21.15.183:0/3328875564' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "71a6c5dc-5109-4284-8d20-187d8e7afc9e"}]': finished 2024-09-10T19:07:18.181 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:07:17 smithi183 ceph-mon[41562]: osdmap e10: 2 total, 1 up, 2 in 2024-09-10T19:07:18.181 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:07:17 smithi183 ceph-mon[41562]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-10T19:07:18.182 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:07:17 smithi183 ceph-mon[29736]: pgmap v43: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-10T19:07:18.182 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:07:17 smithi183 ceph-mon[29736]: from='client.? 172.21.15.183:0/3328875564' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "71a6c5dc-5109-4284-8d20-187d8e7afc9e"}]: dispatch 2024-09-10T19:07:18.182 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:07:17 smithi183 ceph-mon[29736]: from='client.? 172.21.15.183:0/3328875564' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "71a6c5dc-5109-4284-8d20-187d8e7afc9e"}]': finished 2024-09-10T19:07:18.182 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:07:17 smithi183 ceph-mon[29736]: osdmap e10: 2 total, 1 up, 2 in 2024-09-10T19:07:18.182 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:07:17 smithi183 ceph-mon[29736]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-10T19:07:18.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:07:17 smithi191 ceph-mon[32884]: pgmap v43: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-10T19:07:18.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:07:17 smithi191 ceph-mon[32884]: from='client.? 172.21.15.183:0/3328875564' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "71a6c5dc-5109-4284-8d20-187d8e7afc9e"}]: dispatch 2024-09-10T19:07:18.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:07:17 smithi191 ceph-mon[32884]: from='client.? 172.21.15.183:0/3328875564' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "71a6c5dc-5109-4284-8d20-187d8e7afc9e"}]': finished 2024-09-10T19:07:18.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:07:17 smithi191 ceph-mon[32884]: osdmap e10: 2 total, 1 up, 2 in 2024-09-10T19:07:18.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:07:17 smithi191 ceph-mon[32884]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-10T19:07:19.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:07:18 smithi183 ceph-mon[29736]: from='client.? 172.21.15.183:0/151402008' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-10T19:07:19.181 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:07:18 smithi183 ceph-mon[41562]: from='client.? 172.21.15.183:0/151402008' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-10T19:07:19.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:07:18 smithi191 ceph-mon[32884]: from='client.? 172.21.15.183:0/151402008' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-10T19:07:20.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:07:19 smithi183 ceph-mon[29736]: pgmap v45: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-10T19:07:20.181 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:07:19 smithi183 ceph-mon[41562]: pgmap v45: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-10T19:07:20.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:07:19 smithi191 ceph-mon[32884]: pgmap v45: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-10T19:07:21.077 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:07:20 smithi183 ceph-mon[29736]: pgmap v46: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-10T19:07:21.078 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:07:20 smithi183 ceph-mon[41562]: pgmap v46: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-10T19:07:21.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:07:20 smithi191 ceph-mon[32884]: pgmap v46: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-10T19:07:23.397 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:07:23 smithi183 ceph-mon[29736]: pgmap v47: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-10T19:07:23.398 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:07:23 smithi183 ceph-mon[41562]: pgmap v47: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-10T19:07:23.715 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:07:23 smithi191 ceph-mon[32884]: pgmap v47: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-10T19:07:24.417 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:07:24 smithi183 ceph-mon[29736]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-09-10T19:07:24.417 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:07:24 smithi183 ceph-mon[29736]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T19:07:24.418 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:07:24 smithi183 ceph-mon[29736]: Deploying daemon osd.1 on smithi183 2024-09-10T19:07:24.418 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:07:24 smithi183 ceph-mon[41562]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-09-10T19:07:24.418 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:07:24 smithi183 ceph-mon[41562]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T19:07:24.418 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:07:24 smithi183 ceph-mon[41562]: Deploying daemon osd.1 on smithi183 2024-09-10T19:07:24.715 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:07:24 smithi191 ceph-mon[32884]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-09-10T19:07:24.715 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:07:24 smithi191 ceph-mon[32884]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T19:07:24.715 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:07:24 smithi191 ceph-mon[32884]: Deploying daemon osd.1 on smithi183 2024-09-10T19:07:25.420 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:07:25 smithi183 ceph-mon[29736]: pgmap v48: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-10T19:07:25.421 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:07:25 smithi183 ceph-mon[41562]: pgmap v48: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-10T19:07:25.715 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:07:25 smithi191 ceph-mon[32884]: pgmap v48: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-10T19:07:27.517 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:07:27 smithi191 ceph-mon[32884]: pgmap v49: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-10T19:07:27.681 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:07:27 smithi183 ceph-mon[29736]: pgmap v49: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-10T19:07:27.682 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:07:27 smithi183 ceph-mon[41562]: pgmap v49: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-10T19:07:29.431 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:07:29 smithi183 ceph-mon[41562]: pgmap v50: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-10T19:07:29.431 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:07:29 smithi183 ceph-mon[41562]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T19:07:29.431 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:07:29 smithi183 ceph-mon[41562]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:07:29.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:07:29 smithi183 ceph-mon[41562]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:07:29.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:07:29 smithi183 ceph-mon[29736]: pgmap v50: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-10T19:07:29.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:07:29 smithi183 ceph-mon[29736]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T19:07:29.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:07:29 smithi183 ceph-mon[29736]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:07:29.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:07:29 smithi183 ceph-mon[29736]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:07:29.715 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:07:29 smithi191 ceph-mon[32884]: pgmap v50: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-10T19:07:29.715 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:07:29 smithi191 ceph-mon[32884]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T19:07:29.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:07:29 smithi191 ceph-mon[32884]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:07:29.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:07:29 smithi191 ceph-mon[32884]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:07:31.883 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:07:31 smithi183 ceph-mon[29736]: pgmap v51: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-10T19:07:31.883 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:07:31 smithi183 ceph-mon[41562]: pgmap v51: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-10T19:07:31.965 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:07:31 smithi191 ceph-mon[32884]: pgmap v51: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-10T19:07:32.061 INFO:teuthology.orchestra.run.smithi183.stdout:Created osd(s) 1 on host 'smithi183' 2024-09-10T19:07:32.595 DEBUG:teuthology.orchestra.run.smithi183:osd.1> sudo journalctl -f -n 0 -u ceph-320f6df8-6fa7-11ef-bcea-c7b262605968@osd.1.service 2024-09-10T19:07:32.597 INFO:tasks.cephadm:Deploying osd.2 on smithi183 with /dev/vg_nvme/lv_2... 2024-09-10T19:07:32.598 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-09-10T19:07:32.892 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:07:32.931 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:07:32 smithi183 ceph-mon[29736]: from='osd.1 [v2:172.21.15.183:6810/3880626114,v1:172.21.15.183:6811/3880626114]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-09-10T19:07:32.931 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:07:32 smithi183 ceph-mon[29736]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:07:32.931 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:07:32 smithi183 ceph-mon[29736]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:07:32.932 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:07:32 smithi183 ceph-mon[41562]: from='osd.1 [v2:172.21.15.183:6810/3880626114,v1:172.21.15.183:6811/3880626114]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-09-10T19:07:32.932 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:07:32 smithi183 ceph-mon[41562]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:07:32.932 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:07:32 smithi183 ceph-mon[41562]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:07:32.965 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:07:32 smithi191 ceph-mon[32884]: from='osd.1 [v2:172.21.15.183:6810/3880626114,v1:172.21.15.183:6811/3880626114]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-09-10T19:07:32.965 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:07:32 smithi191 ceph-mon[32884]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:07:32.965 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:07:32 smithi191 ceph-mon[32884]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:07:33.931 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:07:33 smithi183 ceph-mon[29736]: pgmap v52: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-10T19:07:33.931 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:07:33 smithi183 ceph-mon[29736]: from='osd.1 [v2:172.21.15.183:6810/3880626114,v1:172.21.15.183:6811/3880626114]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-09-10T19:07:33.931 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:07:33 smithi183 ceph-mon[29736]: osdmap e11: 2 total, 1 up, 2 in 2024-09-10T19:07:33.931 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:07:33 smithi183 ceph-mon[29736]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-10T19:07:33.931 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:07:33 smithi183 ceph-mon[29736]: from='osd.1 [v2:172.21.15.183:6810/3880626114,v1:172.21.15.183:6811/3880626114]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi183", "root=default"]}]: dispatch 2024-09-10T19:07:33.932 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:07:33 smithi183 ceph-mon[29736]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:07:33.932 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:07:33 smithi183 ceph-mon[29736]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:07:33.932 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:07:33 smithi183 ceph-mon[41562]: pgmap v52: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-10T19:07:33.932 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:07:33 smithi183 ceph-mon[41562]: from='osd.1 [v2:172.21.15.183:6810/3880626114,v1:172.21.15.183:6811/3880626114]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-09-10T19:07:33.932 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:07:33 smithi183 ceph-mon[41562]: osdmap e11: 2 total, 1 up, 2 in 2024-09-10T19:07:33.932 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:07:33 smithi183 ceph-mon[41562]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-10T19:07:33.932 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:07:33 smithi183 ceph-mon[41562]: from='osd.1 [v2:172.21.15.183:6810/3880626114,v1:172.21.15.183:6811/3880626114]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi183", "root=default"]}]: dispatch 2024-09-10T19:07:33.932 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:07:33 smithi183 ceph-mon[41562]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:07:33.932 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:07:33 smithi183 ceph-mon[41562]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:07:33.933 INFO:journalctl@ceph.osd.1.smithi183.stdout:Sep 10 19:07:33 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd-1[52280]: 2024-09-10T19:07:33.611+0000 7f77b159d640 -1 osd.1 0 waiting for initial osdmap 2024-09-10T19:07:33.933 INFO:journalctl@ceph.osd.1.smithi183.stdout:Sep 10 19:07:33 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd-1[52280]: 2024-09-10T19:07:33.616+0000 7f77acbd5640 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-10T19:07:33.965 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:07:33 smithi191 ceph-mon[32884]: pgmap v52: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-10T19:07:33.965 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:07:33 smithi191 ceph-mon[32884]: from='osd.1 [v2:172.21.15.183:6810/3880626114,v1:172.21.15.183:6811/3880626114]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-09-10T19:07:33.966 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:07:33 smithi191 ceph-mon[32884]: osdmap e11: 2 total, 1 up, 2 in 2024-09-10T19:07:33.966 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:07:33 smithi191 ceph-mon[32884]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-10T19:07:33.966 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:07:33 smithi191 ceph-mon[32884]: from='osd.1 [v2:172.21.15.183:6810/3880626114,v1:172.21.15.183:6811/3880626114]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi183", "root=default"]}]: dispatch 2024-09-10T19:07:33.966 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:07:33 smithi191 ceph-mon[32884]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:07:33.966 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:07:33 smithi191 ceph-mon[32884]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:07:34.681 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:07:34 smithi183 ceph-mon[29736]: from='osd.1 [v2:172.21.15.183:6810/3880626114,v1:172.21.15.183:6811/3880626114]' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi183", "root=default"]}]': finished 2024-09-10T19:07:34.681 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:07:34 smithi183 ceph-mon[29736]: osdmap e12: 2 total, 1 up, 2 in 2024-09-10T19:07:34.681 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:07:34 smithi183 ceph-mon[29736]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-10T19:07:34.681 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:07:34 smithi183 ceph-mon[29736]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-10T19:07:34.682 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:07:34 smithi183 ceph-mon[41562]: from='osd.1 [v2:172.21.15.183:6810/3880626114,v1:172.21.15.183:6811/3880626114]' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi183", "root=default"]}]': finished 2024-09-10T19:07:34.682 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:07:34 smithi183 ceph-mon[41562]: osdmap e12: 2 total, 1 up, 2 in 2024-09-10T19:07:34.682 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:07:34 smithi183 ceph-mon[41562]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-10T19:07:34.682 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:07:34 smithi183 ceph-mon[41562]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-10T19:07:34.965 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:07:34 smithi191 ceph-mon[32884]: from='osd.1 [v2:172.21.15.183:6810/3880626114,v1:172.21.15.183:6811/3880626114]' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi183", "root=default"]}]': finished 2024-09-10T19:07:34.966 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:07:34 smithi191 ceph-mon[32884]: osdmap e12: 2 total, 1 up, 2 in 2024-09-10T19:07:34.966 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:07:34 smithi191 ceph-mon[32884]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-10T19:07:34.966 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:07:34 smithi191 ceph-mon[32884]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-10T19:07:35.779 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:07:35 smithi183 ceph-mon[29736]: purged_snaps scrub starts 2024-09-10T19:07:35.779 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:07:35 smithi183 ceph-mon[29736]: purged_snaps scrub ok 2024-09-10T19:07:35.779 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:07:35 smithi183 ceph-mon[29736]: pgmap v55: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-10T19:07:35.780 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:07:35 smithi183 ceph-mon[29736]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-10T19:07:35.780 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:07:35 smithi183 ceph-mon[29736]: osd.1 [v2:172.21.15.183:6810/3880626114,v1:172.21.15.183:6811/3880626114] boot 2024-09-10T19:07:35.780 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:07:35 smithi183 ceph-mon[29736]: osdmap e13: 2 total, 2 up, 2 in 2024-09-10T19:07:35.780 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:07:35 smithi183 ceph-mon[29736]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-10T19:07:35.780 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:07:35 smithi183 ceph-mon[41562]: purged_snaps scrub starts 2024-09-10T19:07:35.780 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:07:35 smithi183 ceph-mon[41562]: purged_snaps scrub ok 2024-09-10T19:07:35.780 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:07:35 smithi183 ceph-mon[41562]: pgmap v55: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-10T19:07:35.780 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:07:35 smithi183 ceph-mon[41562]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-10T19:07:35.780 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:07:35 smithi183 ceph-mon[41562]: osd.1 [v2:172.21.15.183:6810/3880626114,v1:172.21.15.183:6811/3880626114] boot 2024-09-10T19:07:35.780 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:07:35 smithi183 ceph-mon[41562]: osdmap e13: 2 total, 2 up, 2 in 2024-09-10T19:07:35.780 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:07:35 smithi183 ceph-mon[41562]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-10T19:07:35.965 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:07:35 smithi191 ceph-mon[32884]: purged_snaps scrub starts 2024-09-10T19:07:35.965 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:07:35 smithi191 ceph-mon[32884]: purged_snaps scrub ok 2024-09-10T19:07:35.965 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:07:35 smithi191 ceph-mon[32884]: pgmap v55: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-09-10T19:07:35.965 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:07:35 smithi191 ceph-mon[32884]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-10T19:07:35.966 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:07:35 smithi191 ceph-mon[32884]: osd.1 [v2:172.21.15.183:6810/3880626114,v1:172.21.15.183:6811/3880626114] boot 2024-09-10T19:07:35.966 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:07:35 smithi191 ceph-mon[32884]: osdmap e13: 2 total, 2 up, 2 in 2024-09-10T19:07:35.966 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:07:35 smithi191 ceph-mon[32884]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-10T19:07:36.154 INFO:teuthology.orchestra.run.smithi183.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-09-10T19:07:36.238 INFO:teuthology.orchestra.run.smithi183.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-09-10T19:07:36.252 INFO:teuthology.orchestra.run.smithi183.stderr: stderr: 10+0 records in 2024-09-10T19:07:36.253 INFO:teuthology.orchestra.run.smithi183.stderr:10+0 records out 2024-09-10T19:07:36.253 INFO:teuthology.orchestra.run.smithi183.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0126449 s, 829 MB/s 2024-09-10T19:07:36.253 INFO:teuthology.orchestra.run.smithi183.stderr:--> Zapping successful for: 2024-09-10T19:07:36.931 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:07:36 smithi183 ceph-mon[29736]: osdmap e14: 2 total, 2 up, 2 in 2024-09-10T19:07:36.931 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:07:36 smithi183 ceph-mon[41562]: osdmap e14: 2 total, 2 up, 2 in 2024-09-10T19:07:36.965 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:07:36 smithi191 ceph-mon[32884]: osdmap e14: 2 total, 2 up, 2 in 2024-09-10T19:07:37.093 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph orch daemon add osd smithi183:vg_nvme/lv_2 2024-09-10T19:07:37.573 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:07:37.806 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:07:37 smithi183 ceph-mon[29736]: pgmap v58: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-10T19:07:37.806 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:07:37 smithi183 ceph-mon[29736]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:07:37.806 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:07:37 smithi183 ceph-mon[29736]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:07:37.806 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:07:37 smithi183 ceph-mon[29736]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-10T19:07:37.806 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:07:37 smithi183 ceph-mon[29736]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-10T19:07:37.806 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:07:37 smithi183 ceph-mon[29736]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:07:37.806 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:07:37 smithi183 ceph-mon[29736]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T19:07:37.806 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:07:37 smithi183 ceph-mon[29736]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T19:07:37.806 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:07:37 smithi183 ceph-mon[29736]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:07:37.807 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:07:37 smithi183 ceph-mon[29736]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T19:07:37.807 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:07:37 smithi183 ceph-mon[29736]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T19:07:37.807 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:07:37 smithi183 ceph-mon[29736]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T19:07:37.807 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:07:37 smithi183 ceph-mon[29736]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:07:37.807 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:07:37 smithi183 ceph-mon[41562]: pgmap v58: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-10T19:07:37.807 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:07:37 smithi183 ceph-mon[41562]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:07:37.807 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:07:37 smithi183 ceph-mon[41562]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:07:37.807 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:07:37 smithi183 ceph-mon[41562]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-10T19:07:37.807 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:07:37 smithi183 ceph-mon[41562]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-10T19:07:37.807 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:07:37 smithi183 ceph-mon[41562]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:07:37.807 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:07:37 smithi183 ceph-mon[41562]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T19:07:37.807 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:07:37 smithi183 ceph-mon[41562]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T19:07:37.808 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:07:37 smithi183 ceph-mon[41562]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:07:37.808 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:07:37 smithi183 ceph-mon[41562]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T19:07:37.808 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:07:37 smithi183 ceph-mon[41562]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T19:07:37.808 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:07:37 smithi183 ceph-mon[41562]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T19:07:37.808 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:07:37 smithi183 ceph-mon[41562]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:07:37.965 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:07:37 smithi191 ceph-mon[32884]: pgmap v58: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-10T19:07:37.965 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:07:37 smithi191 ceph-mon[32884]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:07:37.965 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:07:37 smithi191 ceph-mon[32884]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:07:37.966 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:07:37 smithi191 ceph-mon[32884]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-10T19:07:37.966 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:07:37 smithi191 ceph-mon[32884]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-10T19:07:37.966 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:07:37 smithi191 ceph-mon[32884]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:07:37.966 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:07:37 smithi191 ceph-mon[32884]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T19:07:37.966 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:07:37 smithi191 ceph-mon[32884]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T19:07:37.966 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:07:37 smithi191 ceph-mon[32884]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:07:37.966 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:07:37 smithi191 ceph-mon[32884]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T19:07:37.966 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:07:37 smithi191 ceph-mon[32884]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T19:07:37.966 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:07:37 smithi191 ceph-mon[32884]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T19:07:37.966 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:07:37 smithi191 ceph-mon[32884]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:07:38.931 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:07:38 smithi183 ceph-mon[29736]: Detected new or changed devices on smithi183 2024-09-10T19:07:38.931 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:07:38 smithi183 ceph-mon[29736]: Adjusting osd_memory_target on smithi183 to 8032M 2024-09-10T19:07:38.931 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:07:38 smithi183 ceph-mon[41562]: Detected new or changed devices on smithi183 2024-09-10T19:07:38.931 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:07:38 smithi183 ceph-mon[41562]: Adjusting osd_memory_target on smithi183 to 8032M 2024-09-10T19:07:38.965 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:07:38 smithi191 ceph-mon[32884]: Detected new or changed devices on smithi183 2024-09-10T19:07:38.965 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:07:38 smithi191 ceph-mon[32884]: Adjusting osd_memory_target on smithi183 to 8032M 2024-09-10T19:07:39.931 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:07:39 smithi183 ceph-mon[29736]: pgmap v59: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-10T19:07:39.931 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:07:39 smithi183 ceph-mon[29736]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-10T19:07:39.931 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:07:39 smithi183 ceph-mon[29736]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-10T19:07:39.931 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:07:39 smithi183 ceph-mon[29736]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T19:07:39.931 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:07:39 smithi183 ceph-mon[41562]: pgmap v59: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-10T19:07:39.931 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:07:39 smithi183 ceph-mon[41562]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-10T19:07:39.932 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:07:39 smithi183 ceph-mon[41562]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-10T19:07:39.932 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:07:39 smithi183 ceph-mon[41562]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T19:07:39.965 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:07:39 smithi191 ceph-mon[32884]: pgmap v59: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-10T19:07:39.965 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:07:39 smithi191 ceph-mon[32884]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-10T19:07:39.965 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:07:39 smithi191 ceph-mon[32884]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-10T19:07:39.966 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:07:39 smithi191 ceph-mon[32884]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T19:07:40.681 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:07:40 smithi183 ceph-mon[29736]: from='client.14268 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi183:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-09-10T19:07:40.681 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:07:40 smithi183 ceph-mon[41562]: from='client.14268 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi183:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-09-10T19:07:40.965 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:07:40 smithi191 ceph-mon[32884]: from='client.14268 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi183:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-09-10T19:07:41.931 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:07:41 smithi183 ceph-mon[29736]: pgmap v60: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-10T19:07:41.931 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:07:41 smithi183 ceph-mon[29736]: from='client.? 172.21.15.183:0/1260676335' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "fc310d5b-b89b-4655-9989-7e10ba356509"}]: dispatch 2024-09-10T19:07:41.931 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:07:41 smithi183 ceph-mon[29736]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "fc310d5b-b89b-4655-9989-7e10ba356509"}]: dispatch 2024-09-10T19:07:41.931 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:07:41 smithi183 ceph-mon[29736]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "fc310d5b-b89b-4655-9989-7e10ba356509"}]': finished 2024-09-10T19:07:41.931 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:07:41 smithi183 ceph-mon[29736]: osdmap e15: 3 total, 2 up, 3 in 2024-09-10T19:07:41.931 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:07:41 smithi183 ceph-mon[29736]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-10T19:07:41.932 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:07:41 smithi183 ceph-mon[41562]: pgmap v60: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-10T19:07:41.932 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:07:41 smithi183 ceph-mon[41562]: from='client.? 172.21.15.183:0/1260676335' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "fc310d5b-b89b-4655-9989-7e10ba356509"}]: dispatch 2024-09-10T19:07:41.932 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:07:41 smithi183 ceph-mon[41562]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "fc310d5b-b89b-4655-9989-7e10ba356509"}]: dispatch 2024-09-10T19:07:41.932 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:07:41 smithi183 ceph-mon[41562]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "fc310d5b-b89b-4655-9989-7e10ba356509"}]': finished 2024-09-10T19:07:41.932 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:07:41 smithi183 ceph-mon[41562]: osdmap e15: 3 total, 2 up, 3 in 2024-09-10T19:07:41.932 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:07:41 smithi183 ceph-mon[41562]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-10T19:07:41.965 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:07:41 smithi191 ceph-mon[32884]: pgmap v60: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-10T19:07:41.965 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:07:41 smithi191 ceph-mon[32884]: from='client.? 172.21.15.183:0/1260676335' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "fc310d5b-b89b-4655-9989-7e10ba356509"}]: dispatch 2024-09-10T19:07:41.966 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:07:41 smithi191 ceph-mon[32884]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "fc310d5b-b89b-4655-9989-7e10ba356509"}]: dispatch 2024-09-10T19:07:41.966 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:07:41 smithi191 ceph-mon[32884]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "fc310d5b-b89b-4655-9989-7e10ba356509"}]': finished 2024-09-10T19:07:41.966 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:07:41 smithi191 ceph-mon[32884]: osdmap e15: 3 total, 2 up, 3 in 2024-09-10T19:07:41.966 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:07:41 smithi191 ceph-mon[32884]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-10T19:07:42.931 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:07:42 smithi183 ceph-mon[29736]: from='client.? 172.21.15.183:0/333900344' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-10T19:07:42.931 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:07:42 smithi183 ceph-mon[41562]: from='client.? 172.21.15.183:0/333900344' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-10T19:07:42.965 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:07:42 smithi191 ceph-mon[32884]: from='client.? 172.21.15.183:0/333900344' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-10T19:07:43.931 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:07:43 smithi183 ceph-mon[29736]: pgmap v62: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-10T19:07:43.931 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:07:43 smithi183 ceph-mon[41562]: pgmap v62: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-10T19:07:43.965 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:07:43 smithi191 ceph-mon[32884]: pgmap v62: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-10T19:07:45.803 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:07:45 smithi183 ceph-mon[29736]: pgmap v63: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-10T19:07:45.804 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:07:45 smithi183 ceph-mon[41562]: pgmap v63: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-10T19:07:45.965 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:07:45 smithi191 ceph-mon[32884]: pgmap v63: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-10T19:07:47.681 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:07:47 smithi183 ceph-mon[29736]: pgmap v64: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-10T19:07:47.682 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:07:47 smithi183 ceph-mon[41562]: pgmap v64: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-10T19:07:47.965 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:07:47 smithi191 ceph-mon[32884]: pgmap v64: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-10T19:07:48.684 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:07:48 smithi183 ceph-mon[29736]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-09-10T19:07:48.684 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:07:48 smithi183 ceph-mon[29736]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T19:07:48.684 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:07:48 smithi183 ceph-mon[29736]: Deploying daemon osd.2 on smithi183 2024-09-10T19:07:48.685 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:07:48 smithi183 ceph-mon[41562]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-09-10T19:07:48.685 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:07:48 smithi183 ceph-mon[41562]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T19:07:48.685 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:07:48 smithi183 ceph-mon[41562]: Deploying daemon osd.2 on smithi183 2024-09-10T19:07:48.965 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:07:48 smithi191 ceph-mon[32884]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-09-10T19:07:48.966 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:07:48 smithi191 ceph-mon[32884]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T19:07:48.966 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:07:48 smithi191 ceph-mon[32884]: Deploying daemon osd.2 on smithi183 2024-09-10T19:07:49.931 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:07:49 smithi183 ceph-mon[29736]: pgmap v65: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-10T19:07:49.931 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:07:49 smithi183 ceph-mon[41562]: pgmap v65: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-10T19:07:49.965 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:07:49 smithi191 ceph-mon[32884]: pgmap v65: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-10T19:07:51.807 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:07:51 smithi183 ceph-mon[29736]: pgmap v66: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-10T19:07:51.807 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:07:51 smithi183 ceph-mon[41562]: pgmap v66: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-10T19:07:51.965 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:07:51 smithi191 ceph-mon[32884]: pgmap v66: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-10T19:07:52.931 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:07:52 smithi183 ceph-mon[29736]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T19:07:52.931 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:07:52 smithi183 ceph-mon[29736]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:07:52.931 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:07:52 smithi183 ceph-mon[29736]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:07:52.932 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:07:52 smithi183 ceph-mon[41562]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T19:07:52.932 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:07:52 smithi183 ceph-mon[41562]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:07:52.932 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:07:52 smithi183 ceph-mon[41562]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:07:52.965 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:07:52 smithi191 ceph-mon[32884]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T19:07:52.965 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:07:52 smithi191 ceph-mon[32884]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:07:52.965 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:07:52 smithi191 ceph-mon[32884]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:07:53.931 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:07:53 smithi183 ceph-mon[29736]: pgmap v67: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-10T19:07:53.932 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:07:53 smithi183 ceph-mon[41562]: pgmap v67: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-10T19:07:53.965 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:07:53 smithi191 ceph-mon[32884]: pgmap v67: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-10T19:07:55.767 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:07:55 smithi183 ceph-mon[41562]: pgmap v68: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-10T19:07:55.768 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:07:55 smithi183 ceph-mon[41562]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:07:55.768 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:07:55 smithi183 ceph-mon[41562]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:07:55.768 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:07:55 smithi183 ceph-mon[41562]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T19:07:55.768 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:07:55 smithi183 ceph-mon[41562]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T19:07:55.768 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:07:55 smithi183 ceph-mon[41562]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:07:55.768 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:07:55 smithi183 ceph-mon[29736]: pgmap v68: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-10T19:07:55.768 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:07:55 smithi183 ceph-mon[29736]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:07:55.768 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:07:55 smithi183 ceph-mon[29736]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:07:55.768 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:07:55 smithi183 ceph-mon[29736]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T19:07:55.768 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:07:55 smithi183 ceph-mon[29736]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T19:07:55.768 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:07:55 smithi183 ceph-mon[29736]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:07:55.965 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:07:55 smithi191 ceph-mon[32884]: pgmap v68: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-10T19:07:55.965 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:07:55 smithi191 ceph-mon[32884]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:07:55.965 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:07:55 smithi191 ceph-mon[32884]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:07:55.965 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:07:55 smithi191 ceph-mon[32884]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T19:07:55.966 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:07:55 smithi191 ceph-mon[32884]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T19:07:55.966 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:07:55 smithi191 ceph-mon[32884]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:07:56.026 INFO:teuthology.orchestra.run.smithi183.stdout:Created osd(s) 2 on host 'smithi183' 2024-09-10T19:07:56.659 DEBUG:teuthology.orchestra.run.smithi183:osd.2> sudo journalctl -f -n 0 -u ceph-320f6df8-6fa7-11ef-bcea-c7b262605968@osd.2.service 2024-09-10T19:07:56.662 INFO:tasks.cephadm:Deploying osd.3 on smithi183 with /dev/vg_nvme/lv_1... 2024-09-10T19:07:56.662 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-09-10T19:07:56.931 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:07:56 smithi183 ceph-mon[29736]: from='osd.2 [v2:172.21.15.183:6818/3730013786,v1:172.21.15.183:6819/3730013786]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-09-10T19:07:56.931 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:07:56 smithi183 ceph-mon[29736]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T19:07:56.931 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:07:56 smithi183 ceph-mon[29736]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:07:56.932 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:07:56 smithi183 ceph-mon[29736]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:07:56.932 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:07:56 smithi183 ceph-mon[41562]: from='osd.2 [v2:172.21.15.183:6818/3730013786,v1:172.21.15.183:6819/3730013786]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-09-10T19:07:56.932 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:07:56 smithi183 ceph-mon[41562]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T19:07:56.932 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:07:56 smithi183 ceph-mon[41562]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:07:56.932 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:07:56 smithi183 ceph-mon[41562]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:07:56.965 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:07:56 smithi191 ceph-mon[32884]: from='osd.2 [v2:172.21.15.183:6818/3730013786,v1:172.21.15.183:6819/3730013786]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-09-10T19:07:56.965 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:07:56 smithi191 ceph-mon[32884]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T19:07:56.965 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:07:56 smithi191 ceph-mon[32884]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:07:56.965 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:07:56 smithi191 ceph-mon[32884]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:07:57.449 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:07:57.820 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:07:57 smithi183 ceph-mon[29736]: pgmap v69: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-10T19:07:57.820 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:07:57 smithi183 ceph-mon[29736]: from='osd.2 [v2:172.21.15.183:6818/3730013786,v1:172.21.15.183:6819/3730013786]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-09-10T19:07:57.820 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:07:57 smithi183 ceph-mon[29736]: osdmap e16: 3 total, 2 up, 3 in 2024-09-10T19:07:57.820 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:07:57 smithi183 ceph-mon[29736]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-10T19:07:57.821 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:07:57 smithi183 ceph-mon[29736]: from='osd.2 [v2:172.21.15.183:6818/3730013786,v1:172.21.15.183:6819/3730013786]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi183", "root=default"]}]: dispatch 2024-09-10T19:07:57.821 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:07:57 smithi183 ceph-mon[29736]: from='osd.2 [v2:172.21.15.183:6818/3730013786,v1:172.21.15.183:6819/3730013786]' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi183", "root=default"]}]': finished 2024-09-10T19:07:57.821 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:07:57 smithi183 ceph-mon[29736]: osdmap e17: 3 total, 2 up, 3 in 2024-09-10T19:07:57.821 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:07:57 smithi183 ceph-mon[29736]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-10T19:07:57.821 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:07:57 smithi183 ceph-mon[41562]: pgmap v69: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-10T19:07:57.821 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:07:57 smithi183 ceph-mon[41562]: from='osd.2 [v2:172.21.15.183:6818/3730013786,v1:172.21.15.183:6819/3730013786]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-09-10T19:07:57.821 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:07:57 smithi183 ceph-mon[41562]: osdmap e16: 3 total, 2 up, 3 in 2024-09-10T19:07:57.821 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:07:57 smithi183 ceph-mon[41562]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-10T19:07:57.821 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:07:57 smithi183 ceph-mon[41562]: from='osd.2 [v2:172.21.15.183:6818/3730013786,v1:172.21.15.183:6819/3730013786]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi183", "root=default"]}]: dispatch 2024-09-10T19:07:57.822 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:07:57 smithi183 ceph-mon[41562]: from='osd.2 [v2:172.21.15.183:6818/3730013786,v1:172.21.15.183:6819/3730013786]' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi183", "root=default"]}]': finished 2024-09-10T19:07:57.822 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:07:57 smithi183 ceph-mon[41562]: osdmap e17: 3 total, 2 up, 3 in 2024-09-10T19:07:57.822 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:07:57 smithi183 ceph-mon[41562]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-10T19:07:57.822 INFO:journalctl@ceph.osd.2.smithi183.stdout:Sep 10 19:07:57 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd-2[58691]: 2024-09-10T19:07:57.685+0000 7f395e2a1640 -1 osd.2 0 waiting for initial osdmap 2024-09-10T19:07:57.822 INFO:journalctl@ceph.osd.2.smithi183.stdout:Sep 10 19:07:57 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd-2[58691]: 2024-09-10T19:07:57.689+0000 7f39590d8640 -1 osd.2 17 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-10T19:07:57.966 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:07:57 smithi191 ceph-mon[32884]: pgmap v69: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-10T19:07:57.966 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:07:57 smithi191 ceph-mon[32884]: from='osd.2 [v2:172.21.15.183:6818/3730013786,v1:172.21.15.183:6819/3730013786]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-09-10T19:07:57.966 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:07:57 smithi191 ceph-mon[32884]: osdmap e16: 3 total, 2 up, 3 in 2024-09-10T19:07:57.966 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:07:57 smithi191 ceph-mon[32884]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-10T19:07:57.966 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:07:57 smithi191 ceph-mon[32884]: from='osd.2 [v2:172.21.15.183:6818/3730013786,v1:172.21.15.183:6819/3730013786]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi183", "root=default"]}]: dispatch 2024-09-10T19:07:57.966 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:07:57 smithi191 ceph-mon[32884]: from='osd.2 [v2:172.21.15.183:6818/3730013786,v1:172.21.15.183:6819/3730013786]' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi183", "root=default"]}]': finished 2024-09-10T19:07:57.966 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:07:57 smithi191 ceph-mon[32884]: osdmap e17: 3 total, 2 up, 3 in 2024-09-10T19:07:57.966 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:07:57 smithi191 ceph-mon[32884]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-10T19:07:58.965 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:07:58 smithi191 ceph-mon[32884]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-10T19:07:58.965 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:07:58 smithi191 ceph-mon[32884]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-10T19:07:59.067 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:07:58 smithi183 ceph-mon[29736]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-10T19:07:59.067 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:07:58 smithi183 ceph-mon[29736]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-10T19:07:59.067 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:07:58 smithi183 ceph-mon[41562]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-10T19:07:59.067 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:07:58 smithi183 ceph-mon[41562]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-10T19:07:59.968 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:07:59 smithi191 ceph-mon[32884]: purged_snaps scrub starts 2024-09-10T19:07:59.969 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:07:59 smithi191 ceph-mon[32884]: purged_snaps scrub ok 2024-09-10T19:07:59.969 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:07:59 smithi191 ceph-mon[32884]: pgmap v72: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-10T19:07:59.969 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:07:59 smithi191 ceph-mon[32884]: osd.2 [v2:172.21.15.183:6818/3730013786,v1:172.21.15.183:6819/3730013786] boot 2024-09-10T19:07:59.969 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:07:59 smithi191 ceph-mon[32884]: osdmap e18: 3 total, 3 up, 3 in 2024-09-10T19:07:59.969 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:07:59 smithi191 ceph-mon[32884]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-10T19:08:00.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:07:59 smithi183 ceph-mon[29736]: purged_snaps scrub starts 2024-09-10T19:08:00.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:07:59 smithi183 ceph-mon[29736]: purged_snaps scrub ok 2024-09-10T19:08:00.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:07:59 smithi183 ceph-mon[29736]: pgmap v72: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-10T19:08:00.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:07:59 smithi183 ceph-mon[29736]: osd.2 [v2:172.21.15.183:6818/3730013786,v1:172.21.15.183:6819/3730013786] boot 2024-09-10T19:08:00.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:07:59 smithi183 ceph-mon[29736]: osdmap e18: 3 total, 3 up, 3 in 2024-09-10T19:08:00.182 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:07:59 smithi183 ceph-mon[29736]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-10T19:08:00.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:07:59 smithi183 ceph-mon[41562]: purged_snaps scrub starts 2024-09-10T19:08:00.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:07:59 smithi183 ceph-mon[41562]: purged_snaps scrub ok 2024-09-10T19:08:00.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:07:59 smithi183 ceph-mon[41562]: pgmap v72: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-09-10T19:08:00.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:07:59 smithi183 ceph-mon[41562]: osd.2 [v2:172.21.15.183:6818/3730013786,v1:172.21.15.183:6819/3730013786] boot 2024-09-10T19:08:00.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:07:59 smithi183 ceph-mon[41562]: osdmap e18: 3 total, 3 up, 3 in 2024-09-10T19:08:00.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:07:59 smithi183 ceph-mon[41562]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-10T19:08:00.463 INFO:teuthology.orchestra.run.smithi183.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-09-10T19:08:00.562 INFO:teuthology.orchestra.run.smithi183.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-09-10T19:08:00.578 INFO:teuthology.orchestra.run.smithi183.stderr: stderr: 10+0 records in 2024-09-10T19:08:00.578 INFO:teuthology.orchestra.run.smithi183.stderr:10+0 records out 2024-09-10T19:08:00.578 INFO:teuthology.orchestra.run.smithi183.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0139876 s, 750 MB/s 2024-09-10T19:08:00.578 INFO:teuthology.orchestra.run.smithi183.stderr:--> Zapping successful for: 2024-09-10T19:08:00.810 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:08:00 smithi183 ceph-mon[29736]: osdmap e19: 3 total, 3 up, 3 in 2024-09-10T19:08:00.810 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:08:00 smithi183 ceph-mon[29736]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]: dispatch 2024-09-10T19:08:00.811 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:08:00 smithi183 ceph-mon[41562]: osdmap e19: 3 total, 3 up, 3 in 2024-09-10T19:08:00.811 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:08:00 smithi183 ceph-mon[41562]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]: dispatch 2024-09-10T19:08:00.965 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:08:00 smithi191 ceph-mon[32884]: osdmap e19: 3 total, 3 up, 3 in 2024-09-10T19:08:00.965 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:08:00 smithi191 ceph-mon[32884]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]: dispatch 2024-09-10T19:08:01.422 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph orch daemon add osd smithi183:vg_nvme/lv_1 2024-09-10T19:08:01.663 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:08:01.965 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:08:01 smithi191 ceph-mon[32884]: pgmap v75: 0 pgs: ; 0 B data, 872 MiB used, 267 GiB / 268 GiB avail 2024-09-10T19:08:01.965 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:08:01 smithi191 ceph-mon[32884]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]': finished 2024-09-10T19:08:01.966 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:08:01 smithi191 ceph-mon[32884]: osdmap e20: 3 total, 3 up, 3 in 2024-09-10T19:08:01.966 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:08:01 smithi191 ceph-mon[32884]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:08:01.966 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:08:01 smithi191 ceph-mon[32884]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:08:01.966 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:08:01 smithi191 ceph-mon[32884]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:08:01.966 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:08:01 smithi191 ceph-mon[32884]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-10T19:08:01.966 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:08:01 smithi191 ceph-mon[32884]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-10T19:08:01.966 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:08:01 smithi191 ceph-mon[32884]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-10T19:08:01.966 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:08:01 smithi191 ceph-mon[32884]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:08:01.966 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:08:01 smithi191 ceph-mon[32884]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T19:08:01.966 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:08:01 smithi191 ceph-mon[32884]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T19:08:01.966 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:08:01 smithi191 ceph-mon[32884]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:08:01.990 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:08:01 smithi183 ceph-mon[29736]: pgmap v75: 0 pgs: ; 0 B data, 872 MiB used, 267 GiB / 268 GiB avail 2024-09-10T19:08:01.990 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:08:01 smithi183 ceph-mon[29736]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]': finished 2024-09-10T19:08:01.991 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:08:01 smithi183 ceph-mon[29736]: osdmap e20: 3 total, 3 up, 3 in 2024-09-10T19:08:01.991 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:08:01 smithi183 ceph-mon[29736]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:08:01.991 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:08:01 smithi183 ceph-mon[29736]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:08:01.991 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:08:01 smithi183 ceph-mon[29736]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:08:01.991 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:08:01 smithi183 ceph-mon[29736]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-10T19:08:01.991 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:08:01 smithi183 ceph-mon[29736]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-10T19:08:01.991 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:08:01 smithi183 ceph-mon[29736]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-10T19:08:01.991 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:08:01 smithi183 ceph-mon[29736]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:08:01.991 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:08:01 smithi183 ceph-mon[29736]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T19:08:01.991 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:08:01 smithi183 ceph-mon[29736]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T19:08:01.991 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:08:01 smithi183 ceph-mon[29736]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:08:01.992 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:08:01 smithi183 ceph-mon[41562]: pgmap v75: 0 pgs: ; 0 B data, 872 MiB used, 267 GiB / 268 GiB avail 2024-09-10T19:08:01.992 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:08:01 smithi183 ceph-mon[41562]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]': finished 2024-09-10T19:08:01.992 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:08:01 smithi183 ceph-mon[41562]: osdmap e20: 3 total, 3 up, 3 in 2024-09-10T19:08:01.992 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:08:01 smithi183 ceph-mon[41562]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:08:01.992 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:08:01 smithi183 ceph-mon[41562]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:08:01.992 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:08:01 smithi183 ceph-mon[41562]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:08:01.992 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:08:01 smithi183 ceph-mon[41562]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-10T19:08:01.992 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:08:01 smithi183 ceph-mon[41562]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-10T19:08:01.992 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:08:01 smithi183 ceph-mon[41562]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-10T19:08:01.992 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:08:01 smithi183 ceph-mon[41562]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:08:01.992 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:08:01 smithi183 ceph-mon[41562]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T19:08:01.992 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:08:01 smithi183 ceph-mon[41562]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T19:08:01.993 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:08:01 smithi183 ceph-mon[41562]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:08:02.248 INFO:journalctl@ceph.osd.0.smithi183.stdout:Sep 10 19:08:01 smithi183 sudo[62999]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-09-10T19:08:02.249 INFO:journalctl@ceph.osd.0.smithi183.stdout:Sep 10 19:08:01 smithi183 sudo[62999]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-10T19:08:02.249 INFO:journalctl@ceph.osd.0.smithi183.stdout:Sep 10 19:08:01 smithi183 sudo[62999]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-10T19:08:02.249 INFO:journalctl@ceph.osd.0.smithi183.stdout:Sep 10 19:08:02 smithi183 sudo[62999]: pam_unix(sudo:session): session closed for user root 2024-09-10T19:08:02.515 INFO:journalctl@ceph.osd.0.smithi183.stdout:Sep 10 19:08:02 smithi183 sudo[63020]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-09-10T19:08:02.515 INFO:journalctl@ceph.osd.0.smithi183.stdout:Sep 10 19:08:02 smithi183 sudo[63020]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-10T19:08:02.515 INFO:journalctl@ceph.osd.0.smithi183.stdout:Sep 10 19:08:02 smithi183 sudo[63020]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-10T19:08:02.515 INFO:journalctl@ceph.osd.0.smithi183.stdout:Sep 10 19:08:02 smithi183 sudo[63020]: pam_unix(sudo:session): session closed for user root 2024-09-10T19:08:02.775 INFO:journalctl@ceph.osd.1.smithi183.stdout:Sep 10 19:08:02 smithi183 sudo[63029]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-09-10T19:08:02.775 INFO:journalctl@ceph.osd.1.smithi183.stdout:Sep 10 19:08:02 smithi183 sudo[63029]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-10T19:08:02.775 INFO:journalctl@ceph.osd.1.smithi183.stdout:Sep 10 19:08:02 smithi183 sudo[63029]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-10T19:08:02.775 INFO:journalctl@ceph.osd.1.smithi183.stdout:Sep 10 19:08:02 smithi183 sudo[63029]: pam_unix(sudo:session): session closed for user root 2024-09-10T19:08:02.776 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:08:02 smithi183 ceph-mon[41562]: Detected new or changed devices on smithi183 2024-09-10T19:08:02.776 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:08:02 smithi183 ceph-mon[41562]: Adjusting osd_memory_target on smithi183 to 5354M 2024-09-10T19:08:02.776 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:08:02 smithi183 ceph-mon[41562]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-09-10T19:08:02.776 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:08:02 smithi183 ceph-mon[41562]: osdmap e21: 3 total, 3 up, 3 in 2024-09-10T19:08:02.776 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:08:02 smithi183 ceph-mon[41562]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:08:02.776 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:08:02 smithi183 ceph-mon[29736]: Detected new or changed devices on smithi183 2024-09-10T19:08:02.776 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:08:02 smithi183 ceph-mon[29736]: Adjusting osd_memory_target on smithi183 to 5354M 2024-09-10T19:08:02.776 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:08:02 smithi183 ceph-mon[29736]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-09-10T19:08:02.777 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:08:02 smithi183 ceph-mon[29736]: osdmap e21: 3 total, 3 up, 3 in 2024-09-10T19:08:02.777 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:08:02 smithi183 ceph-mon[29736]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:08:03.027 INFO:journalctl@ceph.osd.1.smithi183.stdout:Sep 10 19:08:02 smithi183 sudo[63032]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-09-10T19:08:03.027 INFO:journalctl@ceph.osd.1.smithi183.stdout:Sep 10 19:08:02 smithi183 sudo[63032]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-10T19:08:03.027 INFO:journalctl@ceph.osd.1.smithi183.stdout:Sep 10 19:08:02 smithi183 sudo[63032]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-10T19:08:03.027 INFO:journalctl@ceph.osd.1.smithi183.stdout:Sep 10 19:08:02 smithi183 sudo[63032]: pam_unix(sudo:session): session closed for user root 2024-09-10T19:08:03.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:08:02 smithi191 ceph-mon[32884]: Detected new or changed devices on smithi183 2024-09-10T19:08:03.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:08:02 smithi191 ceph-mon[32884]: Adjusting osd_memory_target on smithi183 to 5354M 2024-09-10T19:08:03.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:08:02 smithi191 ceph-mon[32884]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-09-10T19:08:03.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:08:02 smithi191 ceph-mon[32884]: osdmap e21: 3 total, 3 up, 3 in 2024-09-10T19:08:03.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:08:02 smithi191 ceph-mon[32884]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:08:03.287 INFO:journalctl@ceph.osd.2.smithi183.stdout:Sep 10 19:08:03 smithi183 sudo[63055]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-09-10T19:08:03.287 INFO:journalctl@ceph.osd.2.smithi183.stdout:Sep 10 19:08:03 smithi183 sudo[63055]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-10T19:08:03.287 INFO:journalctl@ceph.osd.2.smithi183.stdout:Sep 10 19:08:03 smithi183 sudo[63055]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-10T19:08:03.287 INFO:journalctl@ceph.osd.2.smithi183.stdout:Sep 10 19:08:03 smithi183 sudo[63055]: pam_unix(sudo:session): session closed for user root 2024-09-10T19:08:03.537 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:08:03 smithi183 sudo[63107]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-09-10T19:08:03.538 INFO:journalctl@ceph.osd.2.smithi183.stdout:Sep 10 19:08:03 smithi183 sudo[63078]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-09-10T19:08:03.538 INFO:journalctl@ceph.osd.2.smithi183.stdout:Sep 10 19:08:03 smithi183 sudo[63078]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-10T19:08:03.538 INFO:journalctl@ceph.osd.2.smithi183.stdout:Sep 10 19:08:03 smithi183 sudo[63078]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-10T19:08:03.538 INFO:journalctl@ceph.osd.2.smithi183.stdout:Sep 10 19:08:03 smithi183 sudo[63078]: pam_unix(sudo:session): session closed for user root 2024-09-10T19:08:03.795 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:08:03 smithi183 sudo[63107]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-10T19:08:03.795 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:08:03 smithi183 sudo[63107]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-10T19:08:03.795 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:08:03 smithi183 sudo[63107]: pam_unix(sudo:session): session closed for user root 2024-09-10T19:08:04.060 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:08:03 smithi191 ceph-mon[32884]: pgmap v78: 1 pgs: 1 unknown; 0 B data, 872 MiB used, 267 GiB / 268 GiB avail 2024-09-10T19:08:04.060 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:08:03 smithi191 ceph-mon[32884]: osdmap e22: 3 total, 3 up, 3 in 2024-09-10T19:08:04.060 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:08:03 smithi191 ceph-mon[32884]: mgrmap e15: y(active, since 2m), standbys: x 2024-09-10T19:08:04.061 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:08:04 smithi191 sudo[35641]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-09-10T19:08:04.061 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:08:04 smithi191 sudo[35641]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-10T19:08:04.061 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:08:04 smithi191 sudo[35641]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-10T19:08:04.061 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:08:04 smithi191 sudo[35641]: pam_unix(sudo:session): session closed for user root 2024-09-10T19:08:04.181 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:08:03 smithi183 ceph-mon[41562]: pgmap v78: 1 pgs: 1 unknown; 0 B data, 872 MiB used, 267 GiB / 268 GiB avail 2024-09-10T19:08:04.181 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:08:03 smithi183 ceph-mon[41562]: osdmap e22: 3 total, 3 up, 3 in 2024-09-10T19:08:04.181 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:08:03 smithi183 ceph-mon[41562]: mgrmap e15: y(active, since 2m), standbys: x 2024-09-10T19:08:04.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:08:03 smithi183 sudo[63111]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-09-10T19:08:04.182 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:08:03 smithi183 sudo[63111]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-10T19:08:04.182 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:08:03 smithi183 sudo[63111]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-10T19:08:04.182 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:08:03 smithi183 sudo[63111]: pam_unix(sudo:session): session closed for user root 2024-09-10T19:08:04.182 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:08:03 smithi183 ceph-mon[29736]: pgmap v78: 1 pgs: 1 unknown; 0 B data, 872 MiB used, 267 GiB / 268 GiB avail 2024-09-10T19:08:04.182 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:08:03 smithi183 ceph-mon[29736]: osdmap e22: 3 total, 3 up, 3 in 2024-09-10T19:08:04.182 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:08:03 smithi183 ceph-mon[29736]: mgrmap e15: y(active, since 2m), standbys: x 2024-09-10T19:08:04.465 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:08:04 smithi191 sudo[35644]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-09-10T19:08:04.465 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:08:04 smithi191 sudo[35644]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-10T19:08:04.465 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:08:04 smithi191 sudo[35644]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-10T19:08:04.465 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:08:04 smithi191 sudo[35644]: pam_unix(sudo:session): session closed for user root 2024-09-10T19:08:04.825 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:08:04 smithi183 sudo[63273]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-09-10T19:08:04.825 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:08:04 smithi183 sudo[63273]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-10T19:08:04.825 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:08:04 smithi183 sudo[63273]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-10T19:08:04.825 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:08:04 smithi183 sudo[63273]: pam_unix(sudo:session): session closed for user root 2024-09-10T19:08:05.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:08:04 smithi183 ceph-mon[29736]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-10T19:08:05.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:08:04 smithi183 ceph-mon[29736]: from='client.24206 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi183:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-09-10T19:08:05.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:08:04 smithi183 ceph-mon[29736]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-10T19:08:05.182 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:08:04 smithi183 ceph-mon[29736]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-10T19:08:05.182 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:08:04 smithi183 ceph-mon[29736]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-10T19:08:05.182 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:08:04 smithi183 ceph-mon[29736]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-10T19:08:05.182 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:08:04 smithi183 ceph-mon[29736]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-10T19:08:05.182 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:08:04 smithi183 ceph-mon[29736]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-10T19:08:05.182 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:08:04 smithi183 ceph-mon[29736]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-10T19:08:05.182 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:08:04 smithi183 ceph-mon[29736]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-10T19:08:05.182 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:08:04 smithi183 ceph-mon[29736]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-10T19:08:05.182 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:08:04 smithi183 ceph-mon[29736]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-10T19:08:05.182 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:08:04 smithi183 ceph-mon[29736]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T19:08:05.182 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:08:04 smithi183 ceph-mon[29736]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-10T19:08:05.182 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:08:04 smithi183 ceph-mon[29736]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:08:05.182 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:08:04 smithi183 ceph-mon[29736]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-10T19:08:05.183 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:08:04 smithi183 ceph-mon[29736]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-10T19:08:05.183 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:08:04 smithi183 ceph-mon[29736]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-10T19:08:05.183 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:08:04 smithi183 sudo[63282]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-09-10T19:08:05.183 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:08:04 smithi183 sudo[63282]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-10T19:08:05.183 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:08:04 smithi183 sudo[63282]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-10T19:08:05.183 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:08:04 smithi183 sudo[63282]: pam_unix(sudo:session): session closed for user root 2024-09-10T19:08:05.183 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:08:04 smithi183 ceph-mon[41562]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-10T19:08:05.183 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:08:04 smithi183 ceph-mon[41562]: from='client.24206 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi183:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-09-10T19:08:05.183 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:08:04 smithi183 ceph-mon[41562]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-10T19:08:05.183 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:08:04 smithi183 ceph-mon[41562]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-10T19:08:05.183 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:08:04 smithi183 ceph-mon[41562]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-10T19:08:05.184 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:08:04 smithi183 ceph-mon[41562]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-10T19:08:05.184 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:08:04 smithi183 ceph-mon[41562]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-10T19:08:05.184 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:08:04 smithi183 ceph-mon[41562]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-10T19:08:05.184 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:08:04 smithi183 ceph-mon[41562]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-10T19:08:05.184 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:08:04 smithi183 ceph-mon[41562]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-10T19:08:05.184 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:08:04 smithi183 ceph-mon[41562]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-10T19:08:05.184 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:08:04 smithi183 ceph-mon[41562]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-10T19:08:05.184 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:08:04 smithi183 ceph-mon[41562]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T19:08:05.184 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:08:04 smithi183 ceph-mon[41562]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-10T19:08:05.184 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:08:04 smithi183 ceph-mon[41562]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:08:05.184 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:08:04 smithi183 ceph-mon[41562]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-10T19:08:05.184 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:08:04 smithi183 ceph-mon[41562]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-10T19:08:05.184 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:08:04 smithi183 ceph-mon[41562]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-10T19:08:05.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:08:04 smithi191 ceph-mon[32884]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-10T19:08:05.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:08:04 smithi191 ceph-mon[32884]: from='client.24206 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi183:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-09-10T19:08:05.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:08:04 smithi191 ceph-mon[32884]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-10T19:08:05.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:08:04 smithi191 ceph-mon[32884]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-10T19:08:05.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:08:04 smithi191 ceph-mon[32884]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-10T19:08:05.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:08:04 smithi191 ceph-mon[32884]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-10T19:08:05.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:08:04 smithi191 ceph-mon[32884]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-10T19:08:05.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:08:04 smithi191 ceph-mon[32884]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-10T19:08:05.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:08:04 smithi191 ceph-mon[32884]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-10T19:08:05.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:08:04 smithi191 ceph-mon[32884]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-10T19:08:05.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:08:04 smithi191 ceph-mon[32884]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-10T19:08:05.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:08:04 smithi191 ceph-mon[32884]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-10T19:08:05.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:08:04 smithi191 ceph-mon[32884]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T19:08:05.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:08:04 smithi191 ceph-mon[32884]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-10T19:08:05.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:08:04 smithi191 ceph-mon[32884]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:08:05.217 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:08:04 smithi191 ceph-mon[32884]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-10T19:08:05.217 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:08:04 smithi191 ceph-mon[32884]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-10T19:08:05.217 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:08:04 smithi191 ceph-mon[32884]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-10T19:08:06.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:08:05 smithi183 ceph-mon[29736]: pgmap v80: 1 pgs: 1 unknown; 0 B data, 872 MiB used, 267 GiB / 268 GiB avail 2024-09-10T19:08:06.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:08:05 smithi183 ceph-mon[29736]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-10T19:08:06.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:08:05 smithi183 ceph-mon[29736]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-10T19:08:06.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:08:05 smithi183 ceph-mon[41562]: pgmap v80: 1 pgs: 1 unknown; 0 B data, 872 MiB used, 267 GiB / 268 GiB avail 2024-09-10T19:08:06.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:08:05 smithi183 ceph-mon[41562]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-10T19:08:06.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:08:05 smithi183 ceph-mon[41562]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-10T19:08:06.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:08:05 smithi191 ceph-mon[32884]: pgmap v80: 1 pgs: 1 unknown; 0 B data, 872 MiB used, 267 GiB / 268 GiB avail 2024-09-10T19:08:06.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:08:05 smithi191 ceph-mon[32884]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-10T19:08:06.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:08:05 smithi191 ceph-mon[32884]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-10T19:08:07.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:08:06 smithi183 ceph-mon[29736]: from='client.? 172.21.15.183:0/3687727747' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7fd07a95-ad22-43f5-a968-539552283e89"}]: dispatch 2024-09-10T19:08:07.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:08:06 smithi183 ceph-mon[29736]: from='client.? 172.21.15.183:0/3687727747' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "7fd07a95-ad22-43f5-a968-539552283e89"}]': finished 2024-09-10T19:08:07.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:08:06 smithi183 ceph-mon[29736]: osdmap e23: 4 total, 3 up, 4 in 2024-09-10T19:08:07.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:08:06 smithi183 ceph-mon[29736]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-10T19:08:07.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:08:06 smithi183 ceph-mon[29736]: pgmap v82: 1 pgs: 1 active+clean; 577 KiB data, 873 MiB used, 267 GiB / 268 GiB avail 2024-09-10T19:08:07.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:08:06 smithi183 ceph-mon[29736]: from='client.? 172.21.15.183:0/2896654297' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-10T19:08:07.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:08:06 smithi183 ceph-mon[41562]: from='client.? 172.21.15.183:0/3687727747' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7fd07a95-ad22-43f5-a968-539552283e89"}]: dispatch 2024-09-10T19:08:07.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:08:06 smithi183 ceph-mon[41562]: from='client.? 172.21.15.183:0/3687727747' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "7fd07a95-ad22-43f5-a968-539552283e89"}]': finished 2024-09-10T19:08:07.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:08:06 smithi183 ceph-mon[41562]: osdmap e23: 4 total, 3 up, 4 in 2024-09-10T19:08:07.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:08:06 smithi183 ceph-mon[41562]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-10T19:08:07.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:08:06 smithi183 ceph-mon[41562]: pgmap v82: 1 pgs: 1 active+clean; 577 KiB data, 873 MiB used, 267 GiB / 268 GiB avail 2024-09-10T19:08:07.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:08:06 smithi183 ceph-mon[41562]: from='client.? 172.21.15.183:0/2896654297' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-10T19:08:07.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:08:06 smithi191 ceph-mon[32884]: from='client.? 172.21.15.183:0/3687727747' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7fd07a95-ad22-43f5-a968-539552283e89"}]: dispatch 2024-09-10T19:08:07.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:08:06 smithi191 ceph-mon[32884]: from='client.? 172.21.15.183:0/3687727747' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "7fd07a95-ad22-43f5-a968-539552283e89"}]': finished 2024-09-10T19:08:07.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:08:06 smithi191 ceph-mon[32884]: osdmap e23: 4 total, 3 up, 4 in 2024-09-10T19:08:07.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:08:06 smithi191 ceph-mon[32884]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-10T19:08:07.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:08:06 smithi191 ceph-mon[32884]: pgmap v82: 1 pgs: 1 active+clean; 577 KiB data, 873 MiB used, 267 GiB / 268 GiB avail 2024-09-10T19:08:07.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:08:06 smithi191 ceph-mon[32884]: from='client.? 172.21.15.183:0/2896654297' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-10T19:08:09.555 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:08:09 smithi183 ceph-mon[29736]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-09-10T19:08:09.555 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:08:09 smithi183 ceph-mon[41562]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-09-10T19:08:09.715 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:08:09 smithi191 ceph-mon[32884]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-09-10T19:08:11.931 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:08:11 smithi183 ceph-mon[29736]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-09-10T19:08:11.931 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:08:11 smithi183 ceph-mon[41562]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-09-10T19:08:11.965 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:08:11 smithi191 ceph-mon[32884]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-09-10T19:08:12.904 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:08:12 smithi183 ceph-mon[29736]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-09-10T19:08:12.904 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:08:12 smithi183 ceph-mon[29736]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T19:08:12.904 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:08:12 smithi183 ceph-mon[41562]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-09-10T19:08:12.904 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:08:12 smithi183 ceph-mon[41562]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T19:08:12.965 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:08:12 smithi191 ceph-mon[32884]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-09-10T19:08:12.965 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:08:12 smithi191 ceph-mon[32884]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T19:08:13.731 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:08:13 smithi183 ceph-mon[29736]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-09-10T19:08:13.731 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:08:13 smithi183 ceph-mon[29736]: Deploying daemon osd.3 on smithi183 2024-09-10T19:08:13.731 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:08:13 smithi183 ceph-mon[41562]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-09-10T19:08:13.731 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:08:13 smithi183 ceph-mon[41562]: Deploying daemon osd.3 on smithi183 2024-09-10T19:08:13.965 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:08:13 smithi191 ceph-mon[32884]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-09-10T19:08:13.965 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:08:13 smithi191 ceph-mon[32884]: Deploying daemon osd.3 on smithi183 2024-09-10T19:08:15.885 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:08:15 smithi183 ceph-mon[29736]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-09-10T19:08:15.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:08:15 smithi183 ceph-mon[41562]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-09-10T19:08:15.965 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:08:15 smithi191 ceph-mon[32884]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-09-10T19:08:17.681 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:08:17 smithi183 ceph-mon[29736]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-09-10T19:08:17.681 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:08:17 smithi183 ceph-mon[29736]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T19:08:17.681 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:08:17 smithi183 ceph-mon[29736]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:08:17.681 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:08:17 smithi183 ceph-mon[29736]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:08:17.682 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:08:17 smithi183 ceph-mon[41562]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-09-10T19:08:17.682 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:08:17 smithi183 ceph-mon[41562]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T19:08:17.682 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:08:17 smithi183 ceph-mon[41562]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:08:17.682 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:08:17 smithi183 ceph-mon[41562]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:08:17.965 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:08:17 smithi191 ceph-mon[32884]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-09-10T19:08:17.965 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:08:17 smithi191 ceph-mon[32884]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T19:08:17.966 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:08:17 smithi191 ceph-mon[32884]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:08:17.966 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:08:17 smithi191 ceph-mon[32884]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:08:19.906 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:08:19 smithi183 ceph-mon[29736]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-09-10T19:08:19.906 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:08:19 smithi183 ceph-mon[41562]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-09-10T19:08:19.965 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:08:19 smithi191 ceph-mon[32884]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-09-10T19:08:20.818 INFO:teuthology.orchestra.run.smithi183.stdout:Created osd(s) 3 on host 'smithi183' 2024-09-10T19:08:21.414 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:08:21 smithi183 ceph-mon[29736]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:08:21.414 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:08:21 smithi183 ceph-mon[29736]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-09-10T19:08:21.414 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:08:21 smithi183 ceph-mon[29736]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:08:21.414 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:08:21 smithi183 ceph-mon[29736]: from='osd.3 [v2:172.21.15.183:6826/3522836984,v1:172.21.15.183:6827/3522836984]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-09-10T19:08:21.414 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:08:21 smithi183 ceph-mon[29736]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-09-10T19:08:21.414 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:08:21 smithi183 ceph-mon[29736]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:08:21.414 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:08:21 smithi183 ceph-mon[29736]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:08:21.414 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:08:21 smithi183 ceph-mon[41562]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:08:21.414 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:08:21 smithi183 ceph-mon[41562]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-09-10T19:08:21.414 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:08:21 smithi183 ceph-mon[41562]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:08:21.415 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:08:21 smithi183 ceph-mon[41562]: from='osd.3 [v2:172.21.15.183:6826/3522836984,v1:172.21.15.183:6827/3522836984]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-09-10T19:08:21.415 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:08:21 smithi183 ceph-mon[41562]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-09-10T19:08:21.415 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:08:21 smithi183 ceph-mon[41562]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:08:21.415 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:08:21 smithi183 ceph-mon[41562]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:08:21.439 DEBUG:teuthology.orchestra.run.smithi183:osd.3> sudo journalctl -f -n 0 -u ceph-320f6df8-6fa7-11ef-bcea-c7b262605968@osd.3.service 2024-09-10T19:08:21.441 INFO:tasks.cephadm:Deploying osd.4 on smithi191 with /dev/vg_nvme/lv_4... 2024-09-10T19:08:21.441 DEBUG:teuthology.orchestra.run.smithi191:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-09-10T19:08:21.482 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:08:21 smithi191 ceph-mon[32884]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:08:21.482 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:08:21 smithi191 ceph-mon[32884]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-09-10T19:08:21.482 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:08:21 smithi191 ceph-mon[32884]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:08:21.482 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:08:21 smithi191 ceph-mon[32884]: from='osd.3 [v2:172.21.15.183:6826/3522836984,v1:172.21.15.183:6827/3522836984]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-09-10T19:08:21.482 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:08:21 smithi191 ceph-mon[32884]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-09-10T19:08:21.482 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:08:21 smithi191 ceph-mon[32884]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:08:21.483 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:08:21 smithi191 ceph-mon[32884]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:08:21.729 INFO:teuthology.orchestra.run.smithi191.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.b/config 2024-09-10T19:08:22.465 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:08:22 smithi191 ceph-mon[32884]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-09-10T19:08:22.465 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:08:22 smithi191 ceph-mon[32884]: osdmap e24: 4 total, 3 up, 4 in 2024-09-10T19:08:22.465 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:08:22 smithi191 ceph-mon[32884]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-10T19:08:22.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:08:22 smithi191 ceph-mon[32884]: from='osd.3 [v2:172.21.15.183:6826/3522836984,v1:172.21.15.183:6827/3522836984]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi183", "root=default"]}]: dispatch 2024-09-10T19:08:22.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:08:22 smithi191 ceph-mon[32884]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi183", "root=default"]}]: dispatch 2024-09-10T19:08:22.595 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:08:22 smithi183 ceph-mon[29736]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-09-10T19:08:22.595 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:08:22 smithi183 ceph-mon[29736]: osdmap e24: 4 total, 3 up, 4 in 2024-09-10T19:08:22.595 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:08:22 smithi183 ceph-mon[29736]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-10T19:08:22.595 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:08:22 smithi183 ceph-mon[29736]: from='osd.3 [v2:172.21.15.183:6826/3522836984,v1:172.21.15.183:6827/3522836984]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi183", "root=default"]}]: dispatch 2024-09-10T19:08:22.595 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:08:22 smithi183 ceph-mon[29736]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi183", "root=default"]}]: dispatch 2024-09-10T19:08:22.595 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:08:22 smithi183 ceph-mon[41562]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-09-10T19:08:22.595 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:08:22 smithi183 ceph-mon[41562]: osdmap e24: 4 total, 3 up, 4 in 2024-09-10T19:08:22.596 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:08:22 smithi183 ceph-mon[41562]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-10T19:08:22.596 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:08:22 smithi183 ceph-mon[41562]: from='osd.3 [v2:172.21.15.183:6826/3522836984,v1:172.21.15.183:6827/3522836984]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi183", "root=default"]}]: dispatch 2024-09-10T19:08:22.596 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:08:22 smithi183 ceph-mon[41562]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi183", "root=default"]}]: dispatch 2024-09-10T19:08:22.596 INFO:journalctl@ceph.osd.3.smithi183.stdout:Sep 10 19:08:22 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd-3[65168]: 2024-09-10T19:08:22.194+0000 7f3535b27640 -1 osd.3 0 waiting for initial osdmap 2024-09-10T19:08:22.596 INFO:journalctl@ceph.osd.3.smithi183.stdout:Sep 10 19:08:22 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd-3[65168]: 2024-09-10T19:08:22.197+0000 7f352d135640 -1 osd.3 25 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-10T19:08:23.368 INFO:teuthology.orchestra.run.smithi191.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-09-10T19:08:23.450 INFO:teuthology.orchestra.run.smithi191.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-09-10T19:08:23.464 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:08:23 smithi191 ceph-mon[32884]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-09-10T19:08:23.464 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:08:23 smithi191 ceph-mon[32884]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi183", "root=default"]}]': finished 2024-09-10T19:08:23.464 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:08:23 smithi191 ceph-mon[32884]: osdmap e25: 4 total, 3 up, 4 in 2024-09-10T19:08:23.464 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:08:23 smithi191 ceph-mon[32884]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-10T19:08:23.464 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:08:23 smithi191 ceph-mon[32884]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-10T19:08:23.465 INFO:teuthology.orchestra.run.smithi191.stderr: stderr: 10+0 records in 2024-09-10T19:08:23.465 INFO:teuthology.orchestra.run.smithi191.stderr:10+0 records out 2024-09-10T19:08:23.465 INFO:teuthology.orchestra.run.smithi191.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0114532 s, 916 MB/s 2024-09-10T19:08:23.465 INFO:teuthology.orchestra.run.smithi191.stderr:--> Zapping successful for: 2024-09-10T19:08:23.611 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:08:23 smithi183 ceph-mon[41562]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-09-10T19:08:23.611 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:08:23 smithi183 ceph-mon[41562]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi183", "root=default"]}]': finished 2024-09-10T19:08:23.611 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:08:23 smithi183 ceph-mon[41562]: osdmap e25: 4 total, 3 up, 4 in 2024-09-10T19:08:23.611 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:08:23 smithi183 ceph-mon[41562]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-10T19:08:23.611 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:08:23 smithi183 ceph-mon[41562]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-10T19:08:23.611 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:08:23 smithi183 ceph-mon[29736]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-09-10T19:08:23.612 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:08:23 smithi183 ceph-mon[29736]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi183", "root=default"]}]': finished 2024-09-10T19:08:23.612 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:08:23 smithi183 ceph-mon[29736]: osdmap e25: 4 total, 3 up, 4 in 2024-09-10T19:08:23.612 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:08:23 smithi183 ceph-mon[29736]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-10T19:08:23.612 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:08:23 smithi183 ceph-mon[29736]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-10T19:08:24.060 DEBUG:teuthology.orchestra.run.smithi191:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph orch daemon add osd smithi191:vg_nvme/lv_4 2024-09-10T19:08:24.308 INFO:teuthology.orchestra.run.smithi191.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.b/config 2024-09-10T19:08:24.428 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:08:24 smithi191 ceph-mon[32884]: purged_snaps scrub starts 2024-09-10T19:08:24.429 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:08:24 smithi191 ceph-mon[32884]: purged_snaps scrub ok 2024-09-10T19:08:24.429 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:08:24 smithi191 ceph-mon[32884]: osd.3 [v2:172.21.15.183:6826/3522836984,v1:172.21.15.183:6827/3522836984] boot 2024-09-10T19:08:24.429 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:08:24 smithi191 ceph-mon[32884]: osdmap e26: 4 total, 4 up, 4 in 2024-09-10T19:08:24.429 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:08:24 smithi191 ceph-mon[32884]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-10T19:08:24.681 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:08:24 smithi183 ceph-mon[41562]: purged_snaps scrub starts 2024-09-10T19:08:24.681 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:08:24 smithi183 ceph-mon[41562]: purged_snaps scrub ok 2024-09-10T19:08:24.681 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:08:24 smithi183 ceph-mon[41562]: osd.3 [v2:172.21.15.183:6826/3522836984,v1:172.21.15.183:6827/3522836984] boot 2024-09-10T19:08:24.681 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:08:24 smithi183 ceph-mon[41562]: osdmap e26: 4 total, 4 up, 4 in 2024-09-10T19:08:24.681 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:08:24 smithi183 ceph-mon[41562]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-10T19:08:24.682 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:08:24 smithi183 ceph-mon[29736]: purged_snaps scrub starts 2024-09-10T19:08:24.682 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:08:24 smithi183 ceph-mon[29736]: purged_snaps scrub ok 2024-09-10T19:08:24.682 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:08:24 smithi183 ceph-mon[29736]: osd.3 [v2:172.21.15.183:6826/3522836984,v1:172.21.15.183:6827/3522836984] boot 2024-09-10T19:08:24.682 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:08:24 smithi183 ceph-mon[29736]: osdmap e26: 4 total, 4 up, 4 in 2024-09-10T19:08:24.682 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:08:24 smithi183 ceph-mon[29736]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-10T19:08:25.465 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:08:25 smithi191 ceph-mon[32884]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-09-10T19:08:25.465 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:08:25 smithi191 ceph-mon[32884]: osdmap e27: 4 total, 4 up, 4 in 2024-09-10T19:08:25.465 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:08:25 smithi191 ceph-mon[32884]: Detected new or changed devices on smithi183 2024-09-10T19:08:25.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:08:25 smithi191 ceph-mon[32884]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:08:25.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:08:25 smithi191 ceph-mon[32884]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:08:25.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:08:25 smithi191 ceph-mon[32884]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-10T19:08:25.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:08:25 smithi191 ceph-mon[32884]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-10T19:08:25.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:08:25 smithi191 ceph-mon[32884]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-10T19:08:25.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:08:25 smithi191 ceph-mon[32884]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-10T19:08:25.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:08:25 smithi191 ceph-mon[32884]: Adjusting osd_memory_target on smithi183 to 4016M 2024-09-10T19:08:25.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:08:25 smithi191 ceph-mon[32884]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:08:25.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:08:25 smithi191 ceph-mon[32884]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T19:08:25.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:08:25 smithi191 ceph-mon[32884]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T19:08:25.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:08:25 smithi191 ceph-mon[32884]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:08:25.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:08:25 smithi191 ceph-mon[32884]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T19:08:25.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:08:25 smithi191 ceph-mon[32884]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T19:08:25.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:08:25 smithi191 ceph-mon[32884]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T19:08:25.467 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:08:25 smithi191 ceph-mon[32884]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:08:25.681 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:08:25 smithi183 ceph-mon[41562]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-09-10T19:08:25.681 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:08:25 smithi183 ceph-mon[41562]: osdmap e27: 4 total, 4 up, 4 in 2024-09-10T19:08:25.681 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:08:25 smithi183 ceph-mon[41562]: Detected new or changed devices on smithi183 2024-09-10T19:08:25.681 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:08:25 smithi183 ceph-mon[41562]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:08:25.681 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:08:25 smithi183 ceph-mon[41562]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:08:25.681 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:08:25 smithi183 ceph-mon[41562]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-10T19:08:25.682 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:08:25 smithi183 ceph-mon[41562]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-10T19:08:25.682 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:08:25 smithi183 ceph-mon[41562]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-10T19:08:25.682 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:08:25 smithi183 ceph-mon[41562]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-10T19:08:25.682 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:08:25 smithi183 ceph-mon[41562]: Adjusting osd_memory_target on smithi183 to 4016M 2024-09-10T19:08:25.682 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:08:25 smithi183 ceph-mon[41562]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:08:25.682 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:08:25 smithi183 ceph-mon[41562]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T19:08:25.682 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:08:25 smithi183 ceph-mon[41562]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T19:08:25.682 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:08:25 smithi183 ceph-mon[41562]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:08:25.682 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:08:25 smithi183 ceph-mon[41562]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T19:08:25.682 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:08:25 smithi183 ceph-mon[41562]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T19:08:25.682 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:08:25 smithi183 ceph-mon[41562]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T19:08:25.682 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:08:25 smithi183 ceph-mon[41562]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:08:25.683 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:08:25 smithi183 ceph-mon[29736]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-09-10T19:08:25.683 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:08:25 smithi183 ceph-mon[29736]: osdmap e27: 4 total, 4 up, 4 in 2024-09-10T19:08:25.683 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:08:25 smithi183 ceph-mon[29736]: Detected new or changed devices on smithi183 2024-09-10T19:08:25.683 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:08:25 smithi183 ceph-mon[29736]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:08:25.683 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:08:25 smithi183 ceph-mon[29736]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:08:25.683 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:08:25 smithi183 ceph-mon[29736]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-10T19:08:25.683 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:08:25 smithi183 ceph-mon[29736]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-10T19:08:25.683 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:08:25 smithi183 ceph-mon[29736]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-10T19:08:25.683 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:08:25 smithi183 ceph-mon[29736]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-10T19:08:25.683 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:08:25 smithi183 ceph-mon[29736]: Adjusting osd_memory_target on smithi183 to 4016M 2024-09-10T19:08:25.683 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:08:25 smithi183 ceph-mon[29736]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:08:25.683 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:08:25 smithi183 ceph-mon[29736]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T19:08:25.683 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:08:25 smithi183 ceph-mon[29736]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T19:08:25.683 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:08:25 smithi183 ceph-mon[29736]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:08:25.684 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:08:25 smithi183 ceph-mon[29736]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T19:08:25.684 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:08:25 smithi183 ceph-mon[29736]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T19:08:25.684 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:08:25 smithi183 ceph-mon[29736]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T19:08:25.684 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:08:25 smithi183 ceph-mon[29736]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:08:26.551 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:08:26 smithi191 ceph-mon[32884]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-10T19:08:26.551 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:08:26 smithi191 ceph-mon[32884]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-10T19:08:26.551 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:08:26 smithi191 ceph-mon[32884]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T19:08:26.681 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:08:26 smithi183 ceph-mon[29736]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-10T19:08:26.682 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:08:26 smithi183 ceph-mon[29736]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-10T19:08:26.682 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:08:26 smithi183 ceph-mon[29736]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T19:08:26.682 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:08:26 smithi183 ceph-mon[41562]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-10T19:08:26.682 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:08:26 smithi183 ceph-mon[41562]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-10T19:08:26.682 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:08:26 smithi183 ceph-mon[41562]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T19:08:27.376 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:08:27 smithi191 ceph-mon[32884]: from='client.24190 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi191:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-09-10T19:08:27.376 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:08:27 smithi191 ceph-mon[32884]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-09-10T19:08:27.536 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:08:27 smithi183 ceph-mon[29736]: from='client.24190 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi191:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-09-10T19:08:27.536 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:08:27 smithi183 ceph-mon[29736]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-09-10T19:08:27.536 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:08:27 smithi183 ceph-mon[41562]: from='client.24190 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi191:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-09-10T19:08:27.536 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:08:27 smithi183 ceph-mon[41562]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-09-10T19:08:28.509 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:08:28 smithi191 ceph-mon[32884]: purged_snaps scrub starts 2024-09-10T19:08:28.509 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:08:28 smithi191 ceph-mon[32884]: purged_snaps scrub ok 2024-09-10T19:08:28.681 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:08:28 smithi183 ceph-mon[29736]: purged_snaps scrub starts 2024-09-10T19:08:28.681 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:08:28 smithi183 ceph-mon[29736]: purged_snaps scrub ok 2024-09-10T19:08:28.681 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:08:28 smithi183 ceph-mon[41562]: purged_snaps scrub starts 2024-09-10T19:08:28.681 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:08:28 smithi183 ceph-mon[41562]: purged_snaps scrub ok 2024-09-10T19:08:29.465 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:08:29 smithi191 ceph-mon[32884]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-09-10T19:08:29.465 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:08:29 smithi191 ceph-mon[32884]: osdmap e28: 4 total, 4 up, 4 in 2024-09-10T19:08:29.465 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:08:29 smithi191 ceph-mon[32884]: from='client.? 172.21.15.191:0/2964600450' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "631083fb-e503-44da-9b6b-154e88539ee7"}]: dispatch 2024-09-10T19:08:29.465 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:08:29 smithi191 ceph-mon[32884]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "631083fb-e503-44da-9b6b-154e88539ee7"}]: dispatch 2024-09-10T19:08:29.465 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:08:29 smithi191 ceph-mon[32884]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "631083fb-e503-44da-9b6b-154e88539ee7"}]': finished 2024-09-10T19:08:29.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:08:29 smithi191 ceph-mon[32884]: osdmap e29: 5 total, 4 up, 5 in 2024-09-10T19:08:29.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:08:29 smithi191 ceph-mon[32884]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-10T19:08:29.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:08:29 smithi191 ceph-mon[32884]: from='client.? 172.21.15.191:0/287160228' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-10T19:08:29.681 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:08:29 smithi183 ceph-mon[41562]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-09-10T19:08:29.681 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:08:29 smithi183 ceph-mon[41562]: osdmap e28: 4 total, 4 up, 4 in 2024-09-10T19:08:29.681 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:08:29 smithi183 ceph-mon[41562]: from='client.? 172.21.15.191:0/2964600450' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "631083fb-e503-44da-9b6b-154e88539ee7"}]: dispatch 2024-09-10T19:08:29.681 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:08:29 smithi183 ceph-mon[41562]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "631083fb-e503-44da-9b6b-154e88539ee7"}]: dispatch 2024-09-10T19:08:29.681 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:08:29 smithi183 ceph-mon[41562]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "631083fb-e503-44da-9b6b-154e88539ee7"}]': finished 2024-09-10T19:08:29.682 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:08:29 smithi183 ceph-mon[41562]: osdmap e29: 5 total, 4 up, 5 in 2024-09-10T19:08:29.682 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:08:29 smithi183 ceph-mon[41562]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-10T19:08:29.682 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:08:29 smithi183 ceph-mon[41562]: from='client.? 172.21.15.191:0/287160228' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-10T19:08:29.682 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:08:29 smithi183 ceph-mon[29736]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-09-10T19:08:29.682 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:08:29 smithi183 ceph-mon[29736]: osdmap e28: 4 total, 4 up, 4 in 2024-09-10T19:08:29.682 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:08:29 smithi183 ceph-mon[29736]: from='client.? 172.21.15.191:0/2964600450' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "631083fb-e503-44da-9b6b-154e88539ee7"}]: dispatch 2024-09-10T19:08:29.682 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:08:29 smithi183 ceph-mon[29736]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "631083fb-e503-44da-9b6b-154e88539ee7"}]: dispatch 2024-09-10T19:08:29.682 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:08:29 smithi183 ceph-mon[29736]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "631083fb-e503-44da-9b6b-154e88539ee7"}]': finished 2024-09-10T19:08:29.682 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:08:29 smithi183 ceph-mon[29736]: osdmap e29: 5 total, 4 up, 5 in 2024-09-10T19:08:29.682 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:08:29 smithi183 ceph-mon[29736]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-10T19:08:29.682 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:08:29 smithi183 ceph-mon[29736]: from='client.? 172.21.15.191:0/287160228' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-10T19:08:31.715 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:08:31 smithi191 ceph-mon[32884]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-09-10T19:08:31.931 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:08:31 smithi183 ceph-mon[29736]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-09-10T19:08:31.931 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:08:31 smithi183 ceph-mon[41562]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-09-10T19:08:33.715 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:08:33 smithi191 ceph-mon[32884]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-09-10T19:08:33.931 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:08:33 smithi183 ceph-mon[29736]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-09-10T19:08:33.931 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:08:33 smithi183 ceph-mon[41562]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-09-10T19:08:34.877 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:08:34 smithi191 ceph-mon[32884]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-09-10T19:08:34.877 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:08:34 smithi191 ceph-mon[32884]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T19:08:34.931 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:08:34 smithi183 ceph-mon[29736]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-09-10T19:08:34.931 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:08:34 smithi183 ceph-mon[29736]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T19:08:34.931 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:08:34 smithi183 ceph-mon[41562]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-09-10T19:08:34.931 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:08:34 smithi183 ceph-mon[41562]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T19:08:35.872 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:08:35 smithi191 ceph-mon[32884]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-09-10T19:08:35.872 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:08:35 smithi191 ceph-mon[32884]: Deploying daemon osd.4 on smithi191 2024-09-10T19:08:35.931 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:08:35 smithi183 ceph-mon[29736]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-09-10T19:08:35.931 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:08:35 smithi183 ceph-mon[29736]: Deploying daemon osd.4 on smithi191 2024-09-10T19:08:35.931 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:08:35 smithi183 ceph-mon[41562]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-09-10T19:08:35.931 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:08:35 smithi183 ceph-mon[41562]: Deploying daemon osd.4 on smithi191 2024-09-10T19:08:37.931 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:08:37 smithi183 ceph-mon[29736]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-09-10T19:08:37.931 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:08:37 smithi183 ceph-mon[41562]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-09-10T19:08:37.966 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:08:37 smithi191 ceph-mon[32884]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-09-10T19:08:39.931 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:08:39 smithi183 ceph-mon[29736]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-09-10T19:08:39.931 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:08:39 smithi183 ceph-mon[29736]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T19:08:39.932 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:08:39 smithi183 ceph-mon[29736]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:08:39.932 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:08:39 smithi183 ceph-mon[29736]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:08:39.932 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:08:39 smithi183 ceph-mon[41562]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-09-10T19:08:39.932 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:08:39 smithi183 ceph-mon[41562]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T19:08:39.932 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:08:39 smithi183 ceph-mon[41562]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:08:39.932 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:08:39 smithi183 ceph-mon[41562]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:08:39.965 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:08:39 smithi191 ceph-mon[32884]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-09-10T19:08:39.966 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:08:39 smithi191 ceph-mon[32884]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T19:08:39.966 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:08:39 smithi191 ceph-mon[32884]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:08:39.966 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:08:39 smithi191 ceph-mon[32884]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:08:41.743 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:08:41 smithi191 ceph-mon[32884]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-09-10T19:08:41.776 INFO:teuthology.orchestra.run.smithi191.stdout:Created osd(s) 4 on host 'smithi191' 2024-09-10T19:08:41.931 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:08:41 smithi183 ceph-mon[29736]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-09-10T19:08:41.931 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:08:41 smithi183 ceph-mon[41562]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-09-10T19:08:42.437 DEBUG:teuthology.orchestra.run.smithi191:osd.4> sudo journalctl -f -n 0 -u ceph-320f6df8-6fa7-11ef-bcea-c7b262605968@osd.4.service 2024-09-10T19:08:42.439 INFO:tasks.cephadm:Deploying osd.5 on smithi191 with /dev/vg_nvme/lv_3... 2024-09-10T19:08:42.439 DEBUG:teuthology.orchestra.run.smithi191:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-09-10T19:08:42.728 INFO:teuthology.orchestra.run.smithi191.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.b/config 2024-09-10T19:08:43.083 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:08:42 smithi191 ceph-mon[32884]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:08:43.083 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:08:42 smithi191 ceph-mon[32884]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:08:43.084 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:08:42 smithi191 ceph-mon[32884]: from='osd.4 [v2:172.21.15.191:6800/2674828539,v1:172.21.15.191:6801/2674828539]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-09-10T19:08:43.084 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:08:42 smithi191 ceph-mon[32884]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-09-10T19:08:43.084 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:08:42 smithi191 ceph-mon[32884]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:08:43.084 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:08:42 smithi191 ceph-mon[32884]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:08:43.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:08:42 smithi183 ceph-mon[29736]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:08:43.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:08:42 smithi183 ceph-mon[29736]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:08:43.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:08:42 smithi183 ceph-mon[29736]: from='osd.4 [v2:172.21.15.191:6800/2674828539,v1:172.21.15.191:6801/2674828539]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-09-10T19:08:43.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:08:42 smithi183 ceph-mon[29736]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-09-10T19:08:43.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:08:42 smithi183 ceph-mon[29736]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:08:43.182 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:08:42 smithi183 ceph-mon[29736]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:08:43.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:08:42 smithi183 ceph-mon[41562]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:08:43.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:08:42 smithi183 ceph-mon[41562]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:08:43.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:08:42 smithi183 ceph-mon[41562]: from='osd.4 [v2:172.21.15.191:6800/2674828539,v1:172.21.15.191:6801/2674828539]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-09-10T19:08:43.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:08:42 smithi183 ceph-mon[41562]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-09-10T19:08:43.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:08:42 smithi183 ceph-mon[41562]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:08:43.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:08:42 smithi183 ceph-mon[41562]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:08:43.799 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:08:43 smithi191 ceph-mon[32884]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-09-10T19:08:43.800 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:08:43 smithi191 ceph-mon[32884]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-09-10T19:08:43.800 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:08:43 smithi191 ceph-mon[32884]: osdmap e30: 5 total, 4 up, 5 in 2024-09-10T19:08:43.800 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:08:43 smithi191 ceph-mon[32884]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-10T19:08:43.800 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:08:43 smithi191 ceph-mon[32884]: from='osd.4 [v2:172.21.15.191:6800/2674828539,v1:172.21.15.191:6801/2674828539]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi191", "root=default"]}]: dispatch 2024-09-10T19:08:43.800 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:08:43 smithi191 ceph-mon[32884]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi191", "root=default"]}]: dispatch 2024-09-10T19:08:44.058 INFO:journalctl@ceph.osd.4.smithi191.stdout:Sep 10 19:08:43 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd-4[38197]: 2024-09-10T19:08:43.797+0000 7fb6048c0640 -1 osd.4 0 waiting for initial osdmap 2024-09-10T19:08:44.058 INFO:journalctl@ceph.osd.4.smithi191.stdout:Sep 10 19:08:43 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd-4[38197]: 2024-09-10T19:08:43.804+0000 7fb5ffef8640 -1 osd.4 31 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-10T19:08:44.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:08:43 smithi183 ceph-mon[29736]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-09-10T19:08:44.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:08:43 smithi183 ceph-mon[29736]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-09-10T19:08:44.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:08:43 smithi183 ceph-mon[29736]: osdmap e30: 5 total, 4 up, 5 in 2024-09-10T19:08:44.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:08:43 smithi183 ceph-mon[29736]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-10T19:08:44.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:08:43 smithi183 ceph-mon[29736]: from='osd.4 [v2:172.21.15.191:6800/2674828539,v1:172.21.15.191:6801/2674828539]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi191", "root=default"]}]: dispatch 2024-09-10T19:08:44.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:08:43 smithi183 ceph-mon[29736]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi191", "root=default"]}]: dispatch 2024-09-10T19:08:44.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:08:43 smithi183 ceph-mon[41562]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-09-10T19:08:44.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:08:43 smithi183 ceph-mon[41562]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-09-10T19:08:44.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:08:43 smithi183 ceph-mon[41562]: osdmap e30: 5 total, 4 up, 5 in 2024-09-10T19:08:44.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:08:43 smithi183 ceph-mon[41562]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-10T19:08:44.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:08:43 smithi183 ceph-mon[41562]: from='osd.4 [v2:172.21.15.191:6800/2674828539,v1:172.21.15.191:6801/2674828539]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi191", "root=default"]}]: dispatch 2024-09-10T19:08:44.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:08:43 smithi183 ceph-mon[41562]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi191", "root=default"]}]: dispatch 2024-09-10T19:08:44.571 INFO:teuthology.orchestra.run.smithi191.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-09-10T19:08:44.651 INFO:teuthology.orchestra.run.smithi191.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-09-10T19:08:44.667 INFO:teuthology.orchestra.run.smithi191.stderr: stderr: 10+0 records in 2024-09-10T19:08:44.667 INFO:teuthology.orchestra.run.smithi191.stderr:10+0 records out 2024-09-10T19:08:44.667 INFO:teuthology.orchestra.run.smithi191.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0141673 s, 740 MB/s 2024-09-10T19:08:44.667 INFO:teuthology.orchestra.run.smithi191.stderr:--> Zapping successful for: 2024-09-10T19:08:44.913 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:08:44 smithi191 ceph-mon[32884]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi191", "root=default"]}]': finished 2024-09-10T19:08:44.913 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:08:44 smithi191 ceph-mon[32884]: osdmap e31: 5 total, 4 up, 5 in 2024-09-10T19:08:44.913 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:08:44 smithi191 ceph-mon[32884]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-10T19:08:44.913 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:08:44 smithi191 ceph-mon[32884]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-10T19:08:44.914 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:08:44 smithi191 ceph-mon[32884]: osd.4 [v2:172.21.15.191:6800/2674828539,v1:172.21.15.191:6801/2674828539] boot 2024-09-10T19:08:44.914 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:08:44 smithi191 ceph-mon[32884]: osdmap e32: 5 total, 5 up, 5 in 2024-09-10T19:08:44.914 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:08:44 smithi191 ceph-mon[32884]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-10T19:08:45.082 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:08:44 smithi183 ceph-mon[41562]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi191", "root=default"]}]': finished 2024-09-10T19:08:45.083 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:08:44 smithi183 ceph-mon[41562]: osdmap e31: 5 total, 4 up, 5 in 2024-09-10T19:08:45.083 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:08:44 smithi183 ceph-mon[41562]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-10T19:08:45.083 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:08:44 smithi183 ceph-mon[41562]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-10T19:08:45.083 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:08:44 smithi183 ceph-mon[41562]: osd.4 [v2:172.21.15.191:6800/2674828539,v1:172.21.15.191:6801/2674828539] boot 2024-09-10T19:08:45.083 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:08:44 smithi183 ceph-mon[41562]: osdmap e32: 5 total, 5 up, 5 in 2024-09-10T19:08:45.083 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:08:44 smithi183 ceph-mon[41562]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-10T19:08:45.083 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:08:44 smithi183 ceph-mon[29736]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi191", "root=default"]}]': finished 2024-09-10T19:08:45.083 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:08:44 smithi183 ceph-mon[29736]: osdmap e31: 5 total, 4 up, 5 in 2024-09-10T19:08:45.083 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:08:44 smithi183 ceph-mon[29736]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-10T19:08:45.083 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:08:44 smithi183 ceph-mon[29736]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-10T19:08:45.083 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:08:44 smithi183 ceph-mon[29736]: osd.4 [v2:172.21.15.191:6800/2674828539,v1:172.21.15.191:6801/2674828539] boot 2024-09-10T19:08:45.084 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:08:44 smithi183 ceph-mon[29736]: osdmap e32: 5 total, 5 up, 5 in 2024-09-10T19:08:45.084 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:08:44 smithi183 ceph-mon[29736]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-10T19:08:45.456 DEBUG:teuthology.orchestra.run.smithi191:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph orch daemon add osd smithi191:vg_nvme/lv_3 2024-09-10T19:08:46.096 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:08:45 smithi191 ceph-mon[32884]: purged_snaps scrub starts 2024-09-10T19:08:46.096 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:08:45 smithi191 ceph-mon[32884]: purged_snaps scrub ok 2024-09-10T19:08:46.096 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:08:45 smithi191 ceph-mon[32884]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-09-10T19:08:46.096 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:08:45 smithi191 ceph-mon[32884]: osdmap e33: 5 total, 5 up, 5 in 2024-09-10T19:08:46.097 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:08:45 smithi183 ceph-mon[29736]: purged_snaps scrub starts 2024-09-10T19:08:46.098 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:08:45 smithi183 ceph-mon[29736]: purged_snaps scrub ok 2024-09-10T19:08:46.098 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:08:45 smithi183 ceph-mon[29736]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-09-10T19:08:46.098 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:08:45 smithi183 ceph-mon[29736]: osdmap e33: 5 total, 5 up, 5 in 2024-09-10T19:08:46.098 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:08:45 smithi183 ceph-mon[41562]: purged_snaps scrub starts 2024-09-10T19:08:46.098 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:08:45 smithi183 ceph-mon[41562]: purged_snaps scrub ok 2024-09-10T19:08:46.098 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:08:45 smithi183 ceph-mon[41562]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-09-10T19:08:46.098 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:08:45 smithi183 ceph-mon[41562]: osdmap e33: 5 total, 5 up, 5 in 2024-09-10T19:08:46.325 INFO:teuthology.orchestra.run.smithi191.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.b/config 2024-09-10T19:08:47.931 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:08:47 smithi183 ceph-mon[29736]: pgmap v112: 1 pgs: 1 peering; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-09-10T19:08:47.931 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:08:47 smithi183 ceph-mon[29736]: osdmap e34: 5 total, 5 up, 5 in 2024-09-10T19:08:47.931 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:08:47 smithi183 ceph-mon[29736]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:08:47.931 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:08:47 smithi183 ceph-mon[29736]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:08:47.931 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:08:47 smithi183 ceph-mon[29736]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-10T19:08:47.931 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:08:47 smithi183 ceph-mon[29736]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:08:47.932 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:08:47 smithi183 ceph-mon[29736]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T19:08:47.932 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:08:47 smithi183 ceph-mon[29736]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T19:08:47.932 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:08:47 smithi183 ceph-mon[41562]: pgmap v112: 1 pgs: 1 peering; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-09-10T19:08:47.932 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:08:47 smithi183 ceph-mon[41562]: osdmap e34: 5 total, 5 up, 5 in 2024-09-10T19:08:47.932 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:08:47 smithi183 ceph-mon[41562]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:08:47.932 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:08:47 smithi183 ceph-mon[41562]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:08:47.932 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:08:47 smithi183 ceph-mon[41562]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-10T19:08:47.932 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:08:47 smithi183 ceph-mon[41562]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:08:47.932 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:08:47 smithi183 ceph-mon[41562]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T19:08:47.933 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:08:47 smithi183 ceph-mon[41562]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T19:08:47.957 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:08:47 smithi191 ceph-mon[32884]: pgmap v112: 1 pgs: 1 peering; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-09-10T19:08:47.957 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:08:47 smithi191 ceph-mon[32884]: osdmap e34: 5 total, 5 up, 5 in 2024-09-10T19:08:47.958 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:08:47 smithi191 ceph-mon[32884]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:08:47.958 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:08:47 smithi191 ceph-mon[32884]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:08:47.958 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:08:47 smithi191 ceph-mon[32884]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-10T19:08:47.958 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:08:47 smithi191 ceph-mon[32884]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:08:47.958 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:08:47 smithi191 ceph-mon[32884]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T19:08:47.958 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:08:47 smithi191 ceph-mon[32884]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T19:08:48.622 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:08:48 smithi191 ceph-mon[32884]: Detected new or changed devices on smithi191 2024-09-10T19:08:48.623 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:08:48 smithi191 ceph-mon[32884]: Adjusting osd_memory_target on smithi191 to 17088M 2024-09-10T19:08:48.623 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:08:48 smithi191 ceph-mon[32884]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:08:48.931 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:08:48 smithi183 ceph-mon[29736]: Detected new or changed devices on smithi191 2024-09-10T19:08:48.931 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:08:48 smithi183 ceph-mon[29736]: Adjusting osd_memory_target on smithi191 to 17088M 2024-09-10T19:08:48.931 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:08:48 smithi183 ceph-mon[29736]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:08:48.931 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:08:48 smithi183 ceph-mon[29736]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T19:08:48.931 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:08:48 smithi183 ceph-mon[29736]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T19:08:48.931 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:08:48 smithi183 ceph-mon[29736]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T19:08:48.932 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:08:48 smithi183 ceph-mon[29736]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:08:48.932 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:08:48 smithi183 ceph-mon[29736]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-10T19:08:48.932 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:08:48 smithi183 ceph-mon[29736]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-10T19:08:48.932 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:08:48 smithi183 ceph-mon[29736]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T19:08:48.932 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:08:48 smithi183 ceph-mon[41562]: Detected new or changed devices on smithi191 2024-09-10T19:08:48.932 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:08:48 smithi183 ceph-mon[41562]: Adjusting osd_memory_target on smithi191 to 17088M 2024-09-10T19:08:48.932 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:08:48 smithi183 ceph-mon[41562]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:08:48.932 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:08:48 smithi183 ceph-mon[41562]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T19:08:48.932 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:08:48 smithi183 ceph-mon[41562]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T19:08:48.932 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:08:48 smithi183 ceph-mon[41562]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T19:08:48.933 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:08:48 smithi183 ceph-mon[41562]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:08:48.933 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:08:48 smithi183 ceph-mon[41562]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-10T19:08:48.933 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:08:48 smithi183 ceph-mon[41562]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-10T19:08:48.933 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:08:48 smithi183 ceph-mon[41562]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T19:08:48.965 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:08:48 smithi191 ceph-mon[32884]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T19:08:48.965 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:08:48 smithi191 ceph-mon[32884]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T19:08:48.965 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:08:48 smithi191 ceph-mon[32884]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T19:08:48.965 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:08:48 smithi191 ceph-mon[32884]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:08:48.965 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:08:48 smithi191 ceph-mon[32884]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-10T19:08:48.965 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:08:48 smithi191 ceph-mon[32884]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-10T19:08:48.965 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:08:48 smithi191 ceph-mon[32884]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T19:08:49.917 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:08:49 smithi191 ceph-mon[32884]: pgmap v114: 1 pgs: 1 peering; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-09-10T19:08:49.918 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:08:49 smithi191 ceph-mon[32884]: from='client.24217 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi191:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-09-10T19:08:49.918 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:08:49 smithi191 ceph-mon[32884]: purged_snaps scrub starts 2024-09-10T19:08:49.918 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:08:49 smithi191 ceph-mon[32884]: purged_snaps scrub ok 2024-09-10T19:08:50.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:08:49 smithi183 ceph-mon[29736]: pgmap v114: 1 pgs: 1 peering; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-09-10T19:08:50.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:08:49 smithi183 ceph-mon[29736]: from='client.24217 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi191:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-09-10T19:08:50.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:08:49 smithi183 ceph-mon[29736]: purged_snaps scrub starts 2024-09-10T19:08:50.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:08:49 smithi183 ceph-mon[29736]: purged_snaps scrub ok 2024-09-10T19:08:50.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:08:49 smithi183 ceph-mon[41562]: pgmap v114: 1 pgs: 1 peering; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-09-10T19:08:50.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:08:49 smithi183 ceph-mon[41562]: from='client.24217 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi191:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-09-10T19:08:50.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:08:49 smithi183 ceph-mon[41562]: purged_snaps scrub starts 2024-09-10T19:08:50.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:08:49 smithi183 ceph-mon[41562]: purged_snaps scrub ok 2024-09-10T19:08:50.965 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:08:50 smithi191 ceph-mon[32884]: osdmap e35: 5 total, 5 up, 5 in 2024-09-10T19:08:51.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:08:50 smithi183 ceph-mon[29736]: osdmap e35: 5 total, 5 up, 5 in 2024-09-10T19:08:51.181 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:08:50 smithi183 ceph-mon[41562]: osdmap e35: 5 total, 5 up, 5 in 2024-09-10T19:08:51.965 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:08:51 smithi191 ceph-mon[32884]: pgmap v116: 1 pgs: 1 peering; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-09-10T19:08:51.965 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:08:51 smithi191 ceph-mon[32884]: from='client.? 172.21.15.191:0/891802727' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6b963695-3eca-4e6e-9207-05b8010ebff9"}]: dispatch 2024-09-10T19:08:51.965 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:08:51 smithi191 ceph-mon[32884]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6b963695-3eca-4e6e-9207-05b8010ebff9"}]: dispatch 2024-09-10T19:08:51.966 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:08:51 smithi191 ceph-mon[32884]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "6b963695-3eca-4e6e-9207-05b8010ebff9"}]': finished 2024-09-10T19:08:51.966 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:08:51 smithi191 ceph-mon[32884]: osdmap e36: 6 total, 5 up, 6 in 2024-09-10T19:08:51.966 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:08:51 smithi191 ceph-mon[32884]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-10T19:08:51.966 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:08:51 smithi191 ceph-mon[32884]: from='client.? 172.21.15.191:0/2297538199' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-10T19:08:52.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:08:51 smithi183 ceph-mon[29736]: pgmap v116: 1 pgs: 1 peering; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-09-10T19:08:52.193 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:08:51 smithi183 ceph-mon[29736]: from='client.? 172.21.15.191:0/891802727' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6b963695-3eca-4e6e-9207-05b8010ebff9"}]: dispatch 2024-09-10T19:08:52.193 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:08:51 smithi183 ceph-mon[29736]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6b963695-3eca-4e6e-9207-05b8010ebff9"}]: dispatch 2024-09-10T19:08:52.193 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:08:51 smithi183 ceph-mon[29736]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "6b963695-3eca-4e6e-9207-05b8010ebff9"}]': finished 2024-09-10T19:08:52.193 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:08:51 smithi183 ceph-mon[29736]: osdmap e36: 6 total, 5 up, 6 in 2024-09-10T19:08:52.193 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:08:51 smithi183 ceph-mon[29736]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-10T19:08:52.193 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:08:51 smithi183 ceph-mon[29736]: from='client.? 172.21.15.191:0/2297538199' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-10T19:08:52.194 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:08:51 smithi183 ceph-mon[41562]: pgmap v116: 1 pgs: 1 peering; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-09-10T19:08:52.194 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:08:51 smithi183 ceph-mon[41562]: from='client.? 172.21.15.191:0/891802727' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6b963695-3eca-4e6e-9207-05b8010ebff9"}]: dispatch 2024-09-10T19:08:52.194 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:08:51 smithi183 ceph-mon[41562]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6b963695-3eca-4e6e-9207-05b8010ebff9"}]: dispatch 2024-09-10T19:08:52.194 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:08:51 smithi183 ceph-mon[41562]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "6b963695-3eca-4e6e-9207-05b8010ebff9"}]': finished 2024-09-10T19:08:52.194 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:08:51 smithi183 ceph-mon[41562]: osdmap e36: 6 total, 5 up, 6 in 2024-09-10T19:08:52.194 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:08:51 smithi183 ceph-mon[41562]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-10T19:08:52.194 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:08:51 smithi183 ceph-mon[41562]: from='client.? 172.21.15.191:0/2297538199' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-10T19:08:53.965 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:08:53 smithi191 ceph-mon[32884]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-09-10T19:08:54.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:08:53 smithi183 ceph-mon[29736]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-09-10T19:08:54.181 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:08:53 smithi183 ceph-mon[41562]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-09-10T19:08:55.965 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:08:55 smithi191 ceph-mon[32884]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail; 76 KiB/s, 0 objects/s recovering 2024-09-10T19:08:56.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:08:55 smithi183 ceph-mon[29736]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail; 76 KiB/s, 0 objects/s recovering 2024-09-10T19:08:56.181 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:08:55 smithi183 ceph-mon[41562]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail; 76 KiB/s, 0 objects/s recovering 2024-09-10T19:08:57.922 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:08:57 smithi191 ceph-mon[32884]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-09-10T19:08:57.922 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:08:57 smithi191 ceph-mon[32884]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-09-10T19:08:57.922 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:08:57 smithi191 ceph-mon[32884]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T19:08:58.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:08:57 smithi183 ceph-mon[29736]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-09-10T19:08:58.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:08:57 smithi183 ceph-mon[29736]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-09-10T19:08:58.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:08:57 smithi183 ceph-mon[29736]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T19:08:58.181 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:08:57 smithi183 ceph-mon[41562]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-09-10T19:08:58.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:08:57 smithi183 ceph-mon[41562]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-09-10T19:08:58.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:08:57 smithi183 ceph-mon[41562]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T19:08:58.965 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:08:58 smithi191 ceph-mon[32884]: Deploying daemon osd.5 on smithi191 2024-09-10T19:08:59.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:08:58 smithi183 ceph-mon[29736]: Deploying daemon osd.5 on smithi191 2024-09-10T19:08:59.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:08:58 smithi183 ceph-mon[41562]: Deploying daemon osd.5 on smithi191 2024-09-10T19:08:59.936 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:08:59 smithi191 ceph-mon[32884]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail; 68 KiB/s, 0 objects/s recovering 2024-09-10T19:09:00.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:08:59 smithi183 ceph-mon[29736]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail; 68 KiB/s, 0 objects/s recovering 2024-09-10T19:09:00.181 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:08:59 smithi183 ceph-mon[41562]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail; 68 KiB/s, 0 objects/s recovering 2024-09-10T19:09:02.033 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:09:01 smithi191 ceph-mon[32884]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-09-10T19:09:02.034 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:09:01 smithi191 ceph-mon[32884]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T19:09:02.034 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:09:01 smithi191 ceph-mon[32884]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:09:02.034 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:09:01 smithi191 ceph-mon[32884]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:09:02.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:09:01 smithi183 ceph-mon[29736]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-09-10T19:09:02.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:09:01 smithi183 ceph-mon[29736]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T19:09:02.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:09:01 smithi183 ceph-mon[29736]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:09:02.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:09:01 smithi183 ceph-mon[29736]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:09:02.181 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:09:01 smithi183 ceph-mon[41562]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-09-10T19:09:02.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:09:01 smithi183 ceph-mon[41562]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T19:09:02.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:09:01 smithi183 ceph-mon[41562]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:09:02.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:09:01 smithi183 ceph-mon[41562]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:09:03.977 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:09:03 smithi191 ceph-mon[32884]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-09-10T19:09:04.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:09:03 smithi183 ceph-mon[29736]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-09-10T19:09:04.181 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:09:03 smithi183 ceph-mon[41562]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-09-10T19:09:04.865 INFO:teuthology.orchestra.run.smithi191.stdout:Created osd(s) 5 on host 'smithi191' 2024-09-10T19:09:05.468 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:09:05 smithi191 ceph-mon[32884]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-09-10T19:09:05.468 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:09:05 smithi191 ceph-mon[32884]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:09:05.469 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:09:05 smithi191 ceph-mon[32884]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:09:05.469 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:09:05 smithi191 ceph-mon[32884]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T19:09:05.469 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:09:05 smithi191 ceph-mon[32884]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T19:09:05.469 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:09:05 smithi191 ceph-mon[32884]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:09:05.469 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:09:05 smithi191 ceph-mon[32884]: from='osd.5 [v2:172.21.15.191:6808/681733378,v1:172.21.15.191:6809/681733378]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-09-10T19:09:05.469 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:09:05 smithi191 ceph-mon[32884]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-09-10T19:09:05.469 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:09:05 smithi191 ceph-mon[32884]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T19:09:05.470 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:09:05 smithi191 ceph-mon[32884]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:09:05.470 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:09:05 smithi191 ceph-mon[32884]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:09:05.495 DEBUG:teuthology.orchestra.run.smithi191:osd.5> sudo journalctl -f -n 0 -u ceph-320f6df8-6fa7-11ef-bcea-c7b262605968@osd.5.service 2024-09-10T19:09:05.499 INFO:tasks.cephadm:Deploying osd.6 on smithi191 with /dev/vg_nvme/lv_2... 2024-09-10T19:09:05.499 DEBUG:teuthology.orchestra.run.smithi191:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-09-10T19:09:05.681 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:09:05 smithi183 ceph-mon[29736]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-09-10T19:09:05.682 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:09:05 smithi183 ceph-mon[29736]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:09:05.682 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:09:05 smithi183 ceph-mon[29736]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:09:05.682 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:09:05 smithi183 ceph-mon[29736]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T19:09:05.682 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:09:05 smithi183 ceph-mon[29736]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T19:09:05.682 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:09:05 smithi183 ceph-mon[29736]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:09:05.682 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:09:05 smithi183 ceph-mon[29736]: from='osd.5 [v2:172.21.15.191:6808/681733378,v1:172.21.15.191:6809/681733378]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-09-10T19:09:05.682 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:09:05 smithi183 ceph-mon[29736]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-09-10T19:09:05.683 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:09:05 smithi183 ceph-mon[29736]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T19:09:05.683 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:09:05 smithi183 ceph-mon[29736]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:09:05.683 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:09:05 smithi183 ceph-mon[29736]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:09:05.683 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:09:05 smithi183 ceph-mon[41562]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-09-10T19:09:05.683 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:09:05 smithi183 ceph-mon[41562]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:09:05.683 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:09:05 smithi183 ceph-mon[41562]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:09:05.683 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:09:05 smithi183 ceph-mon[41562]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T19:09:05.683 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:09:05 smithi183 ceph-mon[41562]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T19:09:05.683 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:09:05 smithi183 ceph-mon[41562]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:09:05.684 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:09:05 smithi183 ceph-mon[41562]: from='osd.5 [v2:172.21.15.191:6808/681733378,v1:172.21.15.191:6809/681733378]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-09-10T19:09:05.684 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:09:05 smithi183 ceph-mon[41562]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-09-10T19:09:05.684 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:09:05 smithi183 ceph-mon[41562]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T19:09:05.684 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:09:05 smithi183 ceph-mon[41562]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:09:05.684 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:09:05 smithi183 ceph-mon[41562]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:09:06.265 INFO:teuthology.orchestra.run.smithi191.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.b/config 2024-09-10T19:09:06.492 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:09:06 smithi191 ceph-mon[32884]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-09-10T19:09:06.492 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:09:06 smithi191 ceph-mon[32884]: osdmap e37: 6 total, 5 up, 6 in 2024-09-10T19:09:06.492 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:09:06 smithi191 ceph-mon[32884]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-10T19:09:06.492 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:09:06 smithi191 ceph-mon[32884]: from='osd.5 [v2:172.21.15.191:6808/681733378,v1:172.21.15.191:6809/681733378]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi191", "root=default"]}]: dispatch 2024-09-10T19:09:06.492 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:09:06 smithi191 ceph-mon[32884]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi191", "root=default"]}]: dispatch 2024-09-10T19:09:06.492 INFO:journalctl@ceph.osd.5.smithi191.stdout:Sep 10 19:09:06 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd-5[44138]: 2024-09-10T19:09:06.224+0000 7f7dd2b3e640 -1 osd.5 0 waiting for initial osdmap 2024-09-10T19:09:06.492 INFO:journalctl@ceph.osd.5.smithi191.stdout:Sep 10 19:09:06 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd-5[44138]: 2024-09-10T19:09:06.230+0000 7f7dcd152640 -1 osd.5 38 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-10T19:09:06.681 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:09:06 smithi183 ceph-mon[29736]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-09-10T19:09:06.681 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:09:06 smithi183 ceph-mon[29736]: osdmap e37: 6 total, 5 up, 6 in 2024-09-10T19:09:06.681 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:09:06 smithi183 ceph-mon[29736]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-10T19:09:06.681 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:09:06 smithi183 ceph-mon[29736]: from='osd.5 [v2:172.21.15.191:6808/681733378,v1:172.21.15.191:6809/681733378]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi191", "root=default"]}]: dispatch 2024-09-10T19:09:06.681 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:09:06 smithi183 ceph-mon[29736]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi191", "root=default"]}]: dispatch 2024-09-10T19:09:06.682 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:09:06 smithi183 ceph-mon[41562]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-09-10T19:09:06.682 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:09:06 smithi183 ceph-mon[41562]: osdmap e37: 6 total, 5 up, 6 in 2024-09-10T19:09:06.682 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:09:06 smithi183 ceph-mon[41562]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-10T19:09:06.682 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:09:06 smithi183 ceph-mon[41562]: from='osd.5 [v2:172.21.15.191:6808/681733378,v1:172.21.15.191:6809/681733378]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi191", "root=default"]}]: dispatch 2024-09-10T19:09:06.682 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:09:06 smithi183 ceph-mon[41562]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi191", "root=default"]}]: dispatch 2024-09-10T19:09:07.562 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:09:07 smithi191 ceph-mon[32884]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-09-10T19:09:07.562 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:09:07 smithi191 ceph-mon[32884]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi191", "root=default"]}]': finished 2024-09-10T19:09:07.562 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:09:07 smithi191 ceph-mon[32884]: osdmap e38: 6 total, 5 up, 6 in 2024-09-10T19:09:07.562 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:09:07 smithi191 ceph-mon[32884]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-10T19:09:07.562 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:09:07 smithi191 ceph-mon[32884]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-10T19:09:07.681 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:09:07 smithi183 ceph-mon[29736]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-09-10T19:09:07.681 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:09:07 smithi183 ceph-mon[29736]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi191", "root=default"]}]': finished 2024-09-10T19:09:07.681 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:09:07 smithi183 ceph-mon[29736]: osdmap e38: 6 total, 5 up, 6 in 2024-09-10T19:09:07.681 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:09:07 smithi183 ceph-mon[29736]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-10T19:09:07.681 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:09:07 smithi183 ceph-mon[29736]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-10T19:09:07.681 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:09:07 smithi183 ceph-mon[41562]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-09-10T19:09:07.682 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:09:07 smithi183 ceph-mon[41562]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi191", "root=default"]}]': finished 2024-09-10T19:09:07.682 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:09:07 smithi183 ceph-mon[41562]: osdmap e38: 6 total, 5 up, 6 in 2024-09-10T19:09:07.682 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:09:07 smithi183 ceph-mon[41562]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-10T19:09:07.682 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:09:07 smithi183 ceph-mon[41562]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-10T19:09:08.422 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:09:08 smithi191 ceph-mon[32884]: purged_snaps scrub starts 2024-09-10T19:09:08.422 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:09:08 smithi191 ceph-mon[32884]: purged_snaps scrub ok 2024-09-10T19:09:08.422 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:09:08 smithi191 ceph-mon[32884]: osd.5 [v2:172.21.15.191:6808/681733378,v1:172.21.15.191:6809/681733378] boot 2024-09-10T19:09:08.422 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:09:08 smithi191 ceph-mon[32884]: osdmap e39: 6 total, 6 up, 6 in 2024-09-10T19:09:08.422 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:09:08 smithi191 ceph-mon[32884]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-10T19:09:08.681 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:09:08 smithi183 ceph-mon[29736]: purged_snaps scrub starts 2024-09-10T19:09:08.681 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:09:08 smithi183 ceph-mon[29736]: purged_snaps scrub ok 2024-09-10T19:09:08.682 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:09:08 smithi183 ceph-mon[29736]: osd.5 [v2:172.21.15.191:6808/681733378,v1:172.21.15.191:6809/681733378] boot 2024-09-10T19:09:08.682 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:09:08 smithi183 ceph-mon[29736]: osdmap e39: 6 total, 6 up, 6 in 2024-09-10T19:09:08.682 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:09:08 smithi183 ceph-mon[29736]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-10T19:09:08.682 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:09:08 smithi183 ceph-mon[41562]: purged_snaps scrub starts 2024-09-10T19:09:08.682 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:09:08 smithi183 ceph-mon[41562]: purged_snaps scrub ok 2024-09-10T19:09:08.682 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:09:08 smithi183 ceph-mon[41562]: osd.5 [v2:172.21.15.191:6808/681733378,v1:172.21.15.191:6809/681733378] boot 2024-09-10T19:09:08.682 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:09:08 smithi183 ceph-mon[41562]: osdmap e39: 6 total, 6 up, 6 in 2024-09-10T19:09:08.682 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:09:08 smithi183 ceph-mon[41562]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-10T19:09:08.913 INFO:teuthology.orchestra.run.smithi191.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-09-10T19:09:09.003 INFO:teuthology.orchestra.run.smithi191.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-09-10T19:09:09.022 INFO:teuthology.orchestra.run.smithi191.stderr: stderr: 10+0 records in 2024-09-10T19:09:09.022 INFO:teuthology.orchestra.run.smithi191.stderr:10+0 records out 2024-09-10T19:09:09.022 INFO:teuthology.orchestra.run.smithi191.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0168777 s, 621 MB/s 2024-09-10T19:09:09.022 INFO:teuthology.orchestra.run.smithi191.stderr:--> Zapping successful for: 2024-09-10T19:09:09.355 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:09:09 smithi191 ceph-mon[32884]: pgmap v129: 1 pgs: 1 peering; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-09-10T19:09:09.356 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:09:09 smithi191 ceph-mon[32884]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-09-10T19:09:09.356 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:09:09 smithi191 ceph-mon[32884]: osdmap e40: 6 total, 6 up, 6 in 2024-09-10T19:09:09.681 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:09:09 smithi183 ceph-mon[29736]: pgmap v129: 1 pgs: 1 peering; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-09-10T19:09:09.681 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:09:09 smithi183 ceph-mon[29736]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-09-10T19:09:09.681 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:09:09 smithi183 ceph-mon[29736]: osdmap e40: 6 total, 6 up, 6 in 2024-09-10T19:09:09.681 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:09:09 smithi183 ceph-mon[41562]: pgmap v129: 1 pgs: 1 peering; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-09-10T19:09:09.681 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:09:09 smithi183 ceph-mon[41562]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-09-10T19:09:09.682 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:09:09 smithi183 ceph-mon[41562]: osdmap e40: 6 total, 6 up, 6 in 2024-09-10T19:09:09.942 DEBUG:teuthology.orchestra.run.smithi191:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph orch daemon add osd smithi191:vg_nvme/lv_2 2024-09-10T19:09:10.184 INFO:teuthology.orchestra.run.smithi191.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.b/config 2024-09-10T19:09:10.465 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:09:10 smithi191 ceph-mon[32884]: osdmap e41: 6 total, 6 up, 6 in 2024-09-10T19:09:10.465 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:09:10 smithi191 ceph-mon[32884]: Detected new or changed devices on smithi191 2024-09-10T19:09:10.465 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:09:10 smithi191 ceph-mon[32884]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:09:10.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:09:10 smithi191 ceph-mon[32884]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:09:10.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:09:10 smithi191 ceph-mon[32884]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-10T19:09:10.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:09:10 smithi191 ceph-mon[32884]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-10T19:09:10.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:09:10 smithi191 ceph-mon[32884]: Adjusting osd_memory_target on smithi191 to 8544M 2024-09-10T19:09:10.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:09:10 smithi191 ceph-mon[32884]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:09:10.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:09:10 smithi191 ceph-mon[32884]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T19:09:10.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:09:10 smithi191 ceph-mon[32884]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T19:09:10.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:09:10 smithi191 ceph-mon[32884]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:09:10.681 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:09:10 smithi183 ceph-mon[29736]: osdmap e41: 6 total, 6 up, 6 in 2024-09-10T19:09:10.681 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:09:10 smithi183 ceph-mon[29736]: Detected new or changed devices on smithi191 2024-09-10T19:09:10.681 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:09:10 smithi183 ceph-mon[29736]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:09:10.682 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:09:10 smithi183 ceph-mon[29736]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:09:10.682 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:09:10 smithi183 ceph-mon[29736]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-10T19:09:10.682 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:09:10 smithi183 ceph-mon[29736]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-10T19:09:10.682 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:09:10 smithi183 ceph-mon[29736]: Adjusting osd_memory_target on smithi191 to 8544M 2024-09-10T19:09:10.682 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:09:10 smithi183 ceph-mon[29736]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:09:10.682 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:09:10 smithi183 ceph-mon[29736]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T19:09:10.682 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:09:10 smithi183 ceph-mon[29736]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T19:09:10.682 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:09:10 smithi183 ceph-mon[29736]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:09:10.682 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:09:10 smithi183 ceph-mon[41562]: osdmap e41: 6 total, 6 up, 6 in 2024-09-10T19:09:10.682 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:09:10 smithi183 ceph-mon[41562]: Detected new or changed devices on smithi191 2024-09-10T19:09:10.682 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:09:10 smithi183 ceph-mon[41562]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:09:10.682 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:09:10 smithi183 ceph-mon[41562]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:09:10.683 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:09:10 smithi183 ceph-mon[41562]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-10T19:09:10.683 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:09:10 smithi183 ceph-mon[41562]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-10T19:09:10.683 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:09:10 smithi183 ceph-mon[41562]: Adjusting osd_memory_target on smithi191 to 8544M 2024-09-10T19:09:10.683 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:09:10 smithi183 ceph-mon[41562]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:09:10.683 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:09:10 smithi183 ceph-mon[41562]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T19:09:10.683 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:09:10 smithi183 ceph-mon[41562]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T19:09:10.683 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:09:10 smithi183 ceph-mon[41562]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:09:11.931 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:09:11 smithi183 ceph-mon[29736]: pgmap v132: 1 pgs: 1 peering; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-09-10T19:09:11.931 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:09:11 smithi183 ceph-mon[41562]: pgmap v132: 1 pgs: 1 peering; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-09-10T19:09:11.965 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:09:11 smithi191 ceph-mon[32884]: pgmap v132: 1 pgs: 1 peering; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-09-10T19:09:12.931 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:09:12 smithi183 ceph-mon[29736]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-10T19:09:12.931 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:09:12 smithi183 ceph-mon[29736]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-10T19:09:12.931 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:09:12 smithi183 ceph-mon[29736]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T19:09:12.932 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:09:12 smithi183 ceph-mon[41562]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-10T19:09:12.932 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:09:12 smithi183 ceph-mon[41562]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-10T19:09:12.932 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:09:12 smithi183 ceph-mon[41562]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T19:09:12.965 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:09:12 smithi191 ceph-mon[32884]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-10T19:09:12.965 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:09:12 smithi191 ceph-mon[32884]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-10T19:09:12.965 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:09:12 smithi191 ceph-mon[32884]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T19:09:13.931 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:09:13 smithi183 ceph-mon[29736]: from='client.24244 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi191:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-09-10T19:09:13.931 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:09:13 smithi183 ceph-mon[29736]: pgmap v133: 1 pgs: 1 peering; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-09-10T19:09:13.931 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:09:13 smithi183 ceph-mon[41562]: from='client.24244 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi191:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-09-10T19:09:13.932 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:09:13 smithi183 ceph-mon[41562]: pgmap v133: 1 pgs: 1 peering; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-09-10T19:09:13.941 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:09:13 smithi191 ceph-mon[32884]: from='client.24244 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi191:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-09-10T19:09:13.941 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:09:13 smithi191 ceph-mon[32884]: pgmap v133: 1 pgs: 1 peering; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-09-10T19:09:14.786 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:09:14 smithi191 ceph-mon[32884]: from='client.? 172.21.15.191:0/3570598512' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2dc9754f-6aef-4e19-8605-2f1591921811"}]: dispatch 2024-09-10T19:09:14.786 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:09:14 smithi191 ceph-mon[32884]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2dc9754f-6aef-4e19-8605-2f1591921811"}]: dispatch 2024-09-10T19:09:14.787 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:09:14 smithi191 ceph-mon[32884]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "2dc9754f-6aef-4e19-8605-2f1591921811"}]': finished 2024-09-10T19:09:14.787 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:09:14 smithi191 ceph-mon[32884]: osdmap e42: 7 total, 6 up, 7 in 2024-09-10T19:09:14.787 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:09:14 smithi191 ceph-mon[32884]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-10T19:09:14.931 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:09:14 smithi183 ceph-mon[29736]: from='client.? 172.21.15.191:0/3570598512' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2dc9754f-6aef-4e19-8605-2f1591921811"}]: dispatch 2024-09-10T19:09:14.931 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:09:14 smithi183 ceph-mon[29736]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2dc9754f-6aef-4e19-8605-2f1591921811"}]: dispatch 2024-09-10T19:09:14.931 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:09:14 smithi183 ceph-mon[29736]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "2dc9754f-6aef-4e19-8605-2f1591921811"}]': finished 2024-09-10T19:09:14.931 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:09:14 smithi183 ceph-mon[29736]: osdmap e42: 7 total, 6 up, 7 in 2024-09-10T19:09:14.931 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:09:14 smithi183 ceph-mon[29736]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-10T19:09:14.932 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:09:14 smithi183 ceph-mon[41562]: from='client.? 172.21.15.191:0/3570598512' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2dc9754f-6aef-4e19-8605-2f1591921811"}]: dispatch 2024-09-10T19:09:14.932 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:09:14 smithi183 ceph-mon[41562]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2dc9754f-6aef-4e19-8605-2f1591921811"}]: dispatch 2024-09-10T19:09:14.932 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:09:14 smithi183 ceph-mon[41562]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "2dc9754f-6aef-4e19-8605-2f1591921811"}]': finished 2024-09-10T19:09:14.932 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:09:14 smithi183 ceph-mon[41562]: osdmap e42: 7 total, 6 up, 7 in 2024-09-10T19:09:14.932 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:09:14 smithi183 ceph-mon[41562]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-10T19:09:15.931 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:09:15 smithi183 ceph-mon[29736]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 83 KiB/s, 0 objects/s recovering 2024-09-10T19:09:15.931 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:09:15 smithi183 ceph-mon[29736]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-09-10T19:09:15.931 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:09:15 smithi183 ceph-mon[29736]: Cluster is now healthy 2024-09-10T19:09:15.931 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:09:15 smithi183 ceph-mon[29736]: from='client.? 172.21.15.191:0/3119591730' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-10T19:09:15.932 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:09:15 smithi183 ceph-mon[41562]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 83 KiB/s, 0 objects/s recovering 2024-09-10T19:09:15.932 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:09:15 smithi183 ceph-mon[41562]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-09-10T19:09:15.932 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:09:15 smithi183 ceph-mon[41562]: Cluster is now healthy 2024-09-10T19:09:15.932 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:09:15 smithi183 ceph-mon[41562]: from='client.? 172.21.15.191:0/3119591730' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-10T19:09:15.965 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:09:15 smithi191 ceph-mon[32884]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 83 KiB/s, 0 objects/s recovering 2024-09-10T19:09:15.965 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:09:15 smithi191 ceph-mon[32884]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-09-10T19:09:15.965 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:09:15 smithi191 ceph-mon[32884]: Cluster is now healthy 2024-09-10T19:09:15.965 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:09:15 smithi191 ceph-mon[32884]: from='client.? 172.21.15.191:0/3119591730' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-10T19:09:17.931 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:09:17 smithi183 ceph-mon[29736]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-09-10T19:09:17.931 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:09:17 smithi183 ceph-mon[41562]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-09-10T19:09:17.965 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:09:17 smithi191 ceph-mon[32884]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-09-10T19:09:19.931 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:09:19 smithi183 ceph-mon[29736]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 64 KiB/s, 0 objects/s recovering 2024-09-10T19:09:19.931 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:09:19 smithi183 ceph-mon[41562]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 64 KiB/s, 0 objects/s recovering 2024-09-10T19:09:19.965 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:09:19 smithi191 ceph-mon[32884]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 64 KiB/s, 0 objects/s recovering 2024-09-10T19:09:20.931 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:09:20 smithi183 ceph-mon[29736]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-09-10T19:09:20.931 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:09:20 smithi183 ceph-mon[29736]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T19:09:20.932 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:09:20 smithi183 ceph-mon[41562]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-09-10T19:09:20.932 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:09:20 smithi183 ceph-mon[41562]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T19:09:20.965 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:09:20 smithi191 ceph-mon[32884]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-09-10T19:09:20.965 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:09:20 smithi191 ceph-mon[32884]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T19:09:21.916 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:09:21 smithi191 ceph-mon[32884]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-09-10T19:09:21.917 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:09:21 smithi191 ceph-mon[32884]: Deploying daemon osd.6 on smithi191 2024-09-10T19:09:21.931 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:09:21 smithi183 ceph-mon[29736]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-09-10T19:09:21.931 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:09:21 smithi183 ceph-mon[29736]: Deploying daemon osd.6 on smithi191 2024-09-10T19:09:21.931 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:09:21 smithi183 ceph-mon[41562]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-09-10T19:09:21.931 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:09:21 smithi183 ceph-mon[41562]: Deploying daemon osd.6 on smithi191 2024-09-10T19:09:23.932 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:09:23 smithi183 ceph-mon[29736]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-09-10T19:09:23.932 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:09:23 smithi183 ceph-mon[41562]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-09-10T19:09:23.953 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:09:23 smithi191 ceph-mon[32884]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-09-10T19:09:25.779 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:09:25 smithi191 ceph-mon[32884]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-09-10T19:09:25.780 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:09:25 smithi191 ceph-mon[32884]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T19:09:25.780 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:09:25 smithi191 ceph-mon[32884]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:09:25.780 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:09:25 smithi191 ceph-mon[32884]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:09:25.931 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:09:25 smithi183 ceph-mon[29736]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-09-10T19:09:25.931 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:09:25 smithi183 ceph-mon[29736]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T19:09:25.931 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:09:25 smithi183 ceph-mon[29736]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:09:25.931 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:09:25 smithi183 ceph-mon[29736]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:09:25.931 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:09:25 smithi183 ceph-mon[41562]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-09-10T19:09:25.931 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:09:25 smithi183 ceph-mon[41562]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T19:09:25.932 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:09:25 smithi183 ceph-mon[41562]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:09:25.932 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:09:25 smithi183 ceph-mon[41562]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:09:27.757 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:09:27 smithi191 ceph-mon[32884]: purged_snaps scrub starts 2024-09-10T19:09:27.757 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:09:27 smithi191 ceph-mon[32884]: purged_snaps scrub ok 2024-09-10T19:09:27.757 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:09:27 smithi191 ceph-mon[32884]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-09-10T19:09:27.757 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:09:27 smithi191 ceph-mon[32884]: 1.0 scrub starts 2024-09-10T19:09:27.757 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:09:27 smithi191 ceph-mon[32884]: 1.0 scrub ok 2024-09-10T19:09:27.757 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:09:27 smithi191 ceph-mon[32884]: osdmap e43: 7 total, 6 up, 7 in 2024-09-10T19:09:27.758 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:09:27 smithi191 ceph-mon[32884]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-10T19:09:27.931 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:09:27 smithi183 ceph-mon[41562]: purged_snaps scrub starts 2024-09-10T19:09:27.931 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:09:27 smithi183 ceph-mon[41562]: purged_snaps scrub ok 2024-09-10T19:09:27.931 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:09:27 smithi183 ceph-mon[41562]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-09-10T19:09:27.931 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:09:27 smithi183 ceph-mon[41562]: 1.0 scrub starts 2024-09-10T19:09:27.931 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:09:27 smithi183 ceph-mon[41562]: 1.0 scrub ok 2024-09-10T19:09:27.931 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:09:27 smithi183 ceph-mon[41562]: osdmap e43: 7 total, 6 up, 7 in 2024-09-10T19:09:27.932 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:09:27 smithi183 ceph-mon[41562]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-10T19:09:27.932 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:09:27 smithi183 ceph-mon[29736]: purged_snaps scrub starts 2024-09-10T19:09:27.932 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:09:27 smithi183 ceph-mon[29736]: purged_snaps scrub ok 2024-09-10T19:09:27.932 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:09:27 smithi183 ceph-mon[29736]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-09-10T19:09:27.932 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:09:27 smithi183 ceph-mon[29736]: 1.0 scrub starts 2024-09-10T19:09:27.932 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:09:27 smithi183 ceph-mon[29736]: 1.0 scrub ok 2024-09-10T19:09:27.932 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:09:27 smithi183 ceph-mon[29736]: osdmap e43: 7 total, 6 up, 7 in 2024-09-10T19:09:27.932 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:09:27 smithi183 ceph-mon[29736]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-10T19:09:28.590 INFO:teuthology.orchestra.run.smithi191.stdout:Created osd(s) 6 on host 'smithi191' 2024-09-10T19:09:28.856 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:09:28 smithi191 ceph-mon[32884]: purged_snaps scrub starts 2024-09-10T19:09:28.856 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:09:28 smithi191 ceph-mon[32884]: purged_snaps scrub ok 2024-09-10T19:09:28.857 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:09:28 smithi191 ceph-mon[32884]: osdmap e44: 7 total, 6 up, 7 in 2024-09-10T19:09:28.857 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:09:28 smithi191 ceph-mon[32884]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-10T19:09:28.857 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:09:28 smithi191 ceph-mon[32884]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:09:28.857 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:09:28 smithi191 ceph-mon[32884]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:09:28.857 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:09:28 smithi191 ceph-mon[32884]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T19:09:28.857 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:09:28 smithi191 ceph-mon[32884]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T19:09:28.857 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:09:28 smithi191 ceph-mon[32884]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:09:28.857 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:09:28 smithi191 ceph-mon[32884]: from='osd.6 [v2:172.21.15.191:6816/2825813477,v1:172.21.15.191:6817/2825813477]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-09-10T19:09:28.857 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:09:28 smithi191 ceph-mon[32884]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-09-10T19:09:28.857 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:09:28 smithi191 ceph-mon[32884]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T19:09:28.857 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:09:28 smithi191 ceph-mon[32884]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:09:28.857 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:09:28 smithi191 ceph-mon[32884]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:09:28.931 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:09:28 smithi183 ceph-mon[29736]: purged_snaps scrub starts 2024-09-10T19:09:28.931 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:09:28 smithi183 ceph-mon[29736]: purged_snaps scrub ok 2024-09-10T19:09:28.931 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:09:28 smithi183 ceph-mon[29736]: osdmap e44: 7 total, 6 up, 7 in 2024-09-10T19:09:28.931 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:09:28 smithi183 ceph-mon[29736]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-10T19:09:28.931 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:09:28 smithi183 ceph-mon[29736]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:09:28.931 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:09:28 smithi183 ceph-mon[29736]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:09:28.932 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:09:28 smithi183 ceph-mon[29736]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T19:09:28.932 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:09:28 smithi183 ceph-mon[29736]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T19:09:28.932 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:09:28 smithi183 ceph-mon[29736]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:09:28.932 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:09:28 smithi183 ceph-mon[29736]: from='osd.6 [v2:172.21.15.191:6816/2825813477,v1:172.21.15.191:6817/2825813477]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-09-10T19:09:28.932 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:09:28 smithi183 ceph-mon[29736]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-09-10T19:09:28.932 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:09:28 smithi183 ceph-mon[29736]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T19:09:28.932 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:09:28 smithi183 ceph-mon[29736]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:09:28.932 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:09:28 smithi183 ceph-mon[29736]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:09:28.932 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:09:28 smithi183 ceph-mon[41562]: purged_snaps scrub starts 2024-09-10T19:09:28.932 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:09:28 smithi183 ceph-mon[41562]: purged_snaps scrub ok 2024-09-10T19:09:28.932 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:09:28 smithi183 ceph-mon[41562]: osdmap e44: 7 total, 6 up, 7 in 2024-09-10T19:09:28.932 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:09:28 smithi183 ceph-mon[41562]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-10T19:09:28.932 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:09:28 smithi183 ceph-mon[41562]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:09:28.933 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:09:28 smithi183 ceph-mon[41562]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:09:28.933 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:09:28 smithi183 ceph-mon[41562]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T19:09:28.933 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:09:28 smithi183 ceph-mon[41562]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T19:09:28.933 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:09:28 smithi183 ceph-mon[41562]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:09:28.933 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:09:28 smithi183 ceph-mon[41562]: from='osd.6 [v2:172.21.15.191:6816/2825813477,v1:172.21.15.191:6817/2825813477]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-09-10T19:09:28.933 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:09:28 smithi183 ceph-mon[41562]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-09-10T19:09:28.933 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:09:28 smithi183 ceph-mon[41562]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T19:09:28.933 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:09:28 smithi183 ceph-mon[41562]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:09:28.933 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:09:28 smithi183 ceph-mon[41562]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:09:29.189 DEBUG:teuthology.orchestra.run.smithi191:osd.6> sudo journalctl -f -n 0 -u ceph-320f6df8-6fa7-11ef-bcea-c7b262605968@osd.6.service 2024-09-10T19:09:29.191 INFO:tasks.cephadm:Deploying osd.7 on smithi191 with /dev/vg_nvme/lv_1... 2024-09-10T19:09:29.191 DEBUG:teuthology.orchestra.run.smithi191:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-09-10T19:09:29.931 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:09:29 smithi183 ceph-mon[29736]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-09-10T19:09:29.931 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:09:29 smithi183 ceph-mon[29736]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-09-10T19:09:29.931 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:09:29 smithi183 ceph-mon[29736]: osdmap e45: 7 total, 6 up, 7 in 2024-09-10T19:09:29.931 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:09:29 smithi183 ceph-mon[29736]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-10T19:09:29.931 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:09:29 smithi183 ceph-mon[29736]: from='osd.6 [v2:172.21.15.191:6816/2825813477,v1:172.21.15.191:6817/2825813477]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi191", "root=default"]}]: dispatch 2024-09-10T19:09:29.931 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:09:29 smithi183 ceph-mon[29736]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi191", "root=default"]}]: dispatch 2024-09-10T19:09:29.932 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:09:29 smithi183 ceph-mon[41562]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-09-10T19:09:29.932 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:09:29 smithi183 ceph-mon[41562]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-09-10T19:09:29.932 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:09:29 smithi183 ceph-mon[41562]: osdmap e45: 7 total, 6 up, 7 in 2024-09-10T19:09:29.932 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:09:29 smithi183 ceph-mon[41562]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-10T19:09:29.932 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:09:29 smithi183 ceph-mon[41562]: from='osd.6 [v2:172.21.15.191:6816/2825813477,v1:172.21.15.191:6817/2825813477]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi191", "root=default"]}]: dispatch 2024-09-10T19:09:29.932 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:09:29 smithi183 ceph-mon[41562]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi191", "root=default"]}]: dispatch 2024-09-10T19:09:29.934 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:09:29 smithi191 ceph-mon[32884]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-09-10T19:09:29.934 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:09:29 smithi191 ceph-mon[32884]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-09-10T19:09:29.934 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:09:29 smithi191 ceph-mon[32884]: osdmap e45: 7 total, 6 up, 7 in 2024-09-10T19:09:29.934 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:09:29 smithi191 ceph-mon[32884]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-10T19:09:29.935 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:09:29 smithi191 ceph-mon[32884]: from='osd.6 [v2:172.21.15.191:6816/2825813477,v1:172.21.15.191:6817/2825813477]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi191", "root=default"]}]: dispatch 2024-09-10T19:09:29.935 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:09:29 smithi191 ceph-mon[32884]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi191", "root=default"]}]: dispatch 2024-09-10T19:09:30.215 INFO:journalctl@ceph.osd.6.smithi191.stdout:Sep 10 19:09:29 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd-6[50347]: 2024-09-10T19:09:29.932+0000 7f4abc472640 -1 osd.6 0 waiting for initial osdmap 2024-09-10T19:09:30.216 INFO:journalctl@ceph.osd.6.smithi191.stdout:Sep 10 19:09:29 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd-6[50347]: 2024-09-10T19:09:29.938+0000 7f4ab5a84640 -1 osd.6 46 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-10T19:09:30.788 INFO:teuthology.orchestra.run.smithi191.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.b/config 2024-09-10T19:09:31.162 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:09:30 smithi191 ceph-mon[32884]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi191", "root=default"]}]': finished 2024-09-10T19:09:31.162 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:09:30 smithi191 ceph-mon[32884]: osdmap e46: 7 total, 6 up, 7 in 2024-09-10T19:09:31.162 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:09:30 smithi191 ceph-mon[32884]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-10T19:09:31.162 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:09:30 smithi191 ceph-mon[32884]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-10T19:09:31.162 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:09:30 smithi191 ceph-mon[32884]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-09-10T19:09:31.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:09:30 smithi183 ceph-mon[29736]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi191", "root=default"]}]': finished 2024-09-10T19:09:31.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:09:30 smithi183 ceph-mon[29736]: osdmap e46: 7 total, 6 up, 7 in 2024-09-10T19:09:31.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:09:30 smithi183 ceph-mon[29736]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-10T19:09:31.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:09:30 smithi183 ceph-mon[29736]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-10T19:09:31.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:09:30 smithi183 ceph-mon[29736]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-09-10T19:09:31.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:09:30 smithi183 ceph-mon[41562]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi191", "root=default"]}]': finished 2024-09-10T19:09:31.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:09:30 smithi183 ceph-mon[41562]: osdmap e46: 7 total, 6 up, 7 in 2024-09-10T19:09:31.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:09:30 smithi183 ceph-mon[41562]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-10T19:09:31.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:09:30 smithi183 ceph-mon[41562]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-10T19:09:31.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:09:30 smithi183 ceph-mon[41562]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-09-10T19:09:32.060 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:09:31 smithi191 ceph-mon[32884]: purged_snaps scrub starts 2024-09-10T19:09:32.060 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:09:31 smithi191 ceph-mon[32884]: purged_snaps scrub ok 2024-09-10T19:09:32.060 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:09:31 smithi191 ceph-mon[32884]: osd.6 [v2:172.21.15.191:6816/2825813477,v1:172.21.15.191:6817/2825813477] boot 2024-09-10T19:09:32.060 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:09:31 smithi191 ceph-mon[32884]: osdmap e47: 7 total, 7 up, 7 in 2024-09-10T19:09:32.060 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:09:31 smithi191 ceph-mon[32884]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-10T19:09:32.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:09:31 smithi183 ceph-mon[29736]: purged_snaps scrub starts 2024-09-10T19:09:32.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:09:31 smithi183 ceph-mon[29736]: purged_snaps scrub ok 2024-09-10T19:09:32.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:09:31 smithi183 ceph-mon[29736]: osd.6 [v2:172.21.15.191:6816/2825813477,v1:172.21.15.191:6817/2825813477] boot 2024-09-10T19:09:32.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:09:31 smithi183 ceph-mon[29736]: osdmap e47: 7 total, 7 up, 7 in 2024-09-10T19:09:32.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:09:31 smithi183 ceph-mon[29736]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-10T19:09:32.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:09:31 smithi183 ceph-mon[41562]: purged_snaps scrub starts 2024-09-10T19:09:32.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:09:31 smithi183 ceph-mon[41562]: purged_snaps scrub ok 2024-09-10T19:09:32.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:09:31 smithi183 ceph-mon[41562]: osd.6 [v2:172.21.15.191:6816/2825813477,v1:172.21.15.191:6817/2825813477] boot 2024-09-10T19:09:32.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:09:31 smithi183 ceph-mon[41562]: osdmap e47: 7 total, 7 up, 7 in 2024-09-10T19:09:32.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:09:31 smithi183 ceph-mon[41562]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-10T19:09:33.153 INFO:teuthology.orchestra.run.smithi191.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-09-10T19:09:33.153 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:09:32 smithi191 ceph-mon[32884]: osdmap e48: 7 total, 7 up, 7 in 2024-09-10T19:09:33.154 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:09:32 smithi191 ceph-mon[32884]: pgmap v150: 1 pgs: 1 remapped+peering; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-10T19:09:33.154 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:09:32 smithi191 ceph-mon[32884]: Detected new or changed devices on smithi191 2024-09-10T19:09:33.154 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:09:32 smithi191 ceph-mon[32884]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:09:33.154 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:09:32 smithi191 ceph-mon[32884]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:09:33.154 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:09:32 smithi191 ceph-mon[32884]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-10T19:09:33.154 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:09:32 smithi191 ceph-mon[32884]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-10T19:09:33.154 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:09:32 smithi191 ceph-mon[32884]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-10T19:09:33.154 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:09:32 smithi191 ceph-mon[32884]: Adjusting osd_memory_target on smithi191 to 5696M 2024-09-10T19:09:33.154 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:09:32 smithi191 ceph-mon[32884]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:09:33.154 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:09:32 smithi191 ceph-mon[32884]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T19:09:33.154 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:09:32 smithi191 ceph-mon[32884]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T19:09:33.154 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:09:32 smithi191 ceph-mon[32884]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:09:33.244 INFO:teuthology.orchestra.run.smithi191.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-09-10T19:09:33.258 INFO:teuthology.orchestra.run.smithi191.stderr: stderr: 10+0 records in 2024-09-10T19:09:33.258 INFO:teuthology.orchestra.run.smithi191.stderr:10+0 records out 2024-09-10T19:09:33.258 INFO:teuthology.orchestra.run.smithi191.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0132775 s, 790 MB/s 2024-09-10T19:09:33.258 INFO:teuthology.orchestra.run.smithi191.stderr:--> Zapping successful for: 2024-09-10T19:09:33.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:09:32 smithi183 ceph-mon[29736]: osdmap e48: 7 total, 7 up, 7 in 2024-09-10T19:09:33.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:09:32 smithi183 ceph-mon[29736]: pgmap v150: 1 pgs: 1 remapped+peering; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-10T19:09:33.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:09:32 smithi183 ceph-mon[29736]: Detected new or changed devices on smithi191 2024-09-10T19:09:33.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:09:32 smithi183 ceph-mon[29736]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:09:33.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:09:32 smithi183 ceph-mon[29736]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:09:33.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:09:32 smithi183 ceph-mon[29736]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-10T19:09:33.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:09:32 smithi183 ceph-mon[29736]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-10T19:09:33.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:09:32 smithi183 ceph-mon[29736]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-10T19:09:33.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:09:32 smithi183 ceph-mon[29736]: Adjusting osd_memory_target on smithi191 to 5696M 2024-09-10T19:09:33.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:09:32 smithi183 ceph-mon[29736]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:09:33.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:09:32 smithi183 ceph-mon[29736]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T19:09:33.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:09:32 smithi183 ceph-mon[29736]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T19:09:33.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:09:32 smithi183 ceph-mon[29736]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:09:33.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:09:32 smithi183 ceph-mon[41562]: osdmap e48: 7 total, 7 up, 7 in 2024-09-10T19:09:33.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:09:32 smithi183 ceph-mon[41562]: pgmap v150: 1 pgs: 1 remapped+peering; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-10T19:09:33.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:09:32 smithi183 ceph-mon[41562]: Detected new or changed devices on smithi191 2024-09-10T19:09:33.433 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:09:32 smithi183 ceph-mon[41562]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:09:33.433 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:09:32 smithi183 ceph-mon[41562]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:09:33.433 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:09:32 smithi183 ceph-mon[41562]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-10T19:09:33.433 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:09:32 smithi183 ceph-mon[41562]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-10T19:09:33.433 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:09:32 smithi183 ceph-mon[41562]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-10T19:09:33.433 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:09:32 smithi183 ceph-mon[41562]: Adjusting osd_memory_target on smithi191 to 5696M 2024-09-10T19:09:33.433 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:09:32 smithi183 ceph-mon[41562]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:09:33.433 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:09:32 smithi183 ceph-mon[41562]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T19:09:33.433 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:09:32 smithi183 ceph-mon[41562]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T19:09:33.433 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:09:32 smithi183 ceph-mon[41562]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:09:33.862 DEBUG:teuthology.orchestra.run.smithi191:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph orch daemon add osd smithi191:vg_nvme/lv_1 2024-09-10T19:09:34.105 INFO:teuthology.orchestra.run.smithi191.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.b/config 2024-09-10T19:09:34.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:09:33 smithi191 ceph-mon[32884]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-09-10T19:09:34.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:09:33 smithi191 ceph-mon[32884]: osdmap e49: 7 total, 7 up, 7 in 2024-09-10T19:09:34.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:09:33 smithi183 ceph-mon[29736]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-09-10T19:09:34.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:09:33 smithi183 ceph-mon[29736]: osdmap e49: 7 total, 7 up, 7 in 2024-09-10T19:09:34.431 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:09:33 smithi183 ceph-mon[41562]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-09-10T19:09:34.431 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:09:33 smithi183 ceph-mon[41562]: osdmap e49: 7 total, 7 up, 7 in 2024-09-10T19:09:35.726 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:09:35 smithi191 ceph-mon[32884]: pgmap v152: 1 pgs: 1 remapped+peering; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-10T19:09:35.931 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:09:35 smithi183 ceph-mon[29736]: pgmap v152: 1 pgs: 1 remapped+peering; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-10T19:09:35.931 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:09:35 smithi183 ceph-mon[41562]: pgmap v152: 1 pgs: 1 remapped+peering; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-10T19:09:36.931 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:09:36 smithi183 ceph-mon[29736]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-10T19:09:36.931 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:09:36 smithi183 ceph-mon[29736]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-10T19:09:36.932 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:09:36 smithi183 ceph-mon[29736]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T19:09:36.932 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:09:36 smithi183 ceph-mon[41562]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-10T19:09:36.932 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:09:36 smithi183 ceph-mon[41562]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-10T19:09:36.932 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:09:36 smithi183 ceph-mon[41562]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T19:09:36.965 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:09:36 smithi191 ceph-mon[32884]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-10T19:09:36.965 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:09:36 smithi191 ceph-mon[32884]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-10T19:09:36.966 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:09:36 smithi191 ceph-mon[32884]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T19:09:37.897 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:09:37 smithi191 ceph-mon[32884]: from='client.24271 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi191:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-09-10T19:09:37.897 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:09:37 smithi191 ceph-mon[32884]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-10T19:09:37.897 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:09:37 smithi191 ceph-mon[32884]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-09-10T19:09:37.897 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:09:37 smithi191 ceph-mon[32884]: Cluster is now healthy 2024-09-10T19:09:37.931 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:09:37 smithi183 ceph-mon[29736]: from='client.24271 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi191:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-09-10T19:09:37.931 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:09:37 smithi183 ceph-mon[29736]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-10T19:09:37.931 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:09:37 smithi183 ceph-mon[29736]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-09-10T19:09:37.931 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:09:37 smithi183 ceph-mon[29736]: Cluster is now healthy 2024-09-10T19:09:37.932 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:09:37 smithi183 ceph-mon[41562]: from='client.24271 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi191:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-09-10T19:09:37.932 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:09:37 smithi183 ceph-mon[41562]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-10T19:09:37.932 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:09:37 smithi183 ceph-mon[41562]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-09-10T19:09:37.932 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:09:37 smithi183 ceph-mon[41562]: Cluster is now healthy 2024-09-10T19:09:38.715 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:09:38 smithi191 ceph-mon[32884]: from='client.? 172.21.15.191:0/2425654221' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1a9a299e-d5ba-4b9a-85d5-7f6a5bc754e1"}]: dispatch 2024-09-10T19:09:38.715 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:09:38 smithi191 ceph-mon[32884]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1a9a299e-d5ba-4b9a-85d5-7f6a5bc754e1"}]: dispatch 2024-09-10T19:09:38.715 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:09:38 smithi191 ceph-mon[32884]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "1a9a299e-d5ba-4b9a-85d5-7f6a5bc754e1"}]': finished 2024-09-10T19:09:38.715 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:09:38 smithi191 ceph-mon[32884]: osdmap e50: 8 total, 7 up, 8 in 2024-09-10T19:09:38.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:09:38 smithi191 ceph-mon[32884]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-10T19:09:38.931 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:09:38 smithi183 ceph-mon[29736]: from='client.? 172.21.15.191:0/2425654221' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1a9a299e-d5ba-4b9a-85d5-7f6a5bc754e1"}]: dispatch 2024-09-10T19:09:38.931 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:09:38 smithi183 ceph-mon[29736]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1a9a299e-d5ba-4b9a-85d5-7f6a5bc754e1"}]: dispatch 2024-09-10T19:09:38.931 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:09:38 smithi183 ceph-mon[29736]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "1a9a299e-d5ba-4b9a-85d5-7f6a5bc754e1"}]': finished 2024-09-10T19:09:38.931 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:09:38 smithi183 ceph-mon[29736]: osdmap e50: 8 total, 7 up, 8 in 2024-09-10T19:09:38.931 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:09:38 smithi183 ceph-mon[29736]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-10T19:09:38.932 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:09:38 smithi183 ceph-mon[41562]: from='client.? 172.21.15.191:0/2425654221' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1a9a299e-d5ba-4b9a-85d5-7f6a5bc754e1"}]: dispatch 2024-09-10T19:09:38.932 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:09:38 smithi183 ceph-mon[41562]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1a9a299e-d5ba-4b9a-85d5-7f6a5bc754e1"}]: dispatch 2024-09-10T19:09:38.932 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:09:38 smithi183 ceph-mon[41562]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "1a9a299e-d5ba-4b9a-85d5-7f6a5bc754e1"}]': finished 2024-09-10T19:09:38.932 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:09:38 smithi183 ceph-mon[41562]: osdmap e50: 8 total, 7 up, 8 in 2024-09-10T19:09:38.932 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:09:38 smithi183 ceph-mon[41562]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-10T19:09:39.931 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:09:39 smithi183 ceph-mon[29736]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-10T19:09:39.931 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:09:39 smithi183 ceph-mon[29736]: from='client.? 172.21.15.191:0/3882415916' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-10T19:09:39.931 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:09:39 smithi183 ceph-mon[41562]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-10T19:09:39.931 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:09:39 smithi183 ceph-mon[41562]: from='client.? 172.21.15.191:0/3882415916' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-10T19:09:39.965 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:09:39 smithi191 ceph-mon[32884]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-10T19:09:39.965 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:09:39 smithi191 ceph-mon[32884]: from='client.? 172.21.15.191:0/3882415916' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-10T19:09:41.931 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:09:41 smithi183 ceph-mon[29736]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-09-10T19:09:41.932 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:09:41 smithi183 ceph-mon[41562]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-09-10T19:09:41.942 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:09:41 smithi191 ceph-mon[32884]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-09-10T19:09:43.931 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:09:43 smithi183 ceph-mon[29736]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 62 KiB/s, 0 objects/s recovering 2024-09-10T19:09:43.931 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:09:43 smithi183 ceph-mon[41562]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 62 KiB/s, 0 objects/s recovering 2024-09-10T19:09:43.965 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:09:43 smithi191 ceph-mon[32884]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 62 KiB/s, 0 objects/s recovering 2024-09-10T19:09:44.913 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:09:44 smithi191 ceph-mon[32884]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-09-10T19:09:44.913 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:09:44 smithi191 ceph-mon[32884]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T19:09:44.931 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:09:44 smithi183 ceph-mon[29736]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-09-10T19:09:44.931 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:09:44 smithi183 ceph-mon[29736]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T19:09:44.931 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:09:44 smithi183 ceph-mon[41562]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-09-10T19:09:44.931 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:09:44 smithi183 ceph-mon[41562]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T19:09:45.903 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:09:45 smithi191 ceph-mon[32884]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-09-10T19:09:45.903 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:09:45 smithi191 ceph-mon[32884]: Deploying daemon osd.7 on smithi191 2024-09-10T19:09:45.931 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:09:45 smithi183 ceph-mon[41562]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-09-10T19:09:45.931 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:09:45 smithi183 ceph-mon[41562]: Deploying daemon osd.7 on smithi191 2024-09-10T19:09:45.931 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:09:45 smithi183 ceph-mon[29736]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-09-10T19:09:45.931 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:09:45 smithi183 ceph-mon[29736]: Deploying daemon osd.7 on smithi191 2024-09-10T19:09:47.931 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:09:47 smithi183 ceph-mon[29736]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-10T19:09:47.931 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:09:47 smithi183 ceph-mon[41562]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-10T19:09:47.966 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:09:47 smithi191 ceph-mon[32884]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-10T19:09:48.876 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:09:48 smithi191 ceph-mon[32884]: purged_snaps scrub starts 2024-09-10T19:09:48.877 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:09:48 smithi191 ceph-mon[32884]: purged_snaps scrub ok 2024-09-10T19:09:48.931 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:09:48 smithi183 ceph-mon[29736]: purged_snaps scrub starts 2024-09-10T19:09:48.931 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:09:48 smithi183 ceph-mon[29736]: purged_snaps scrub ok 2024-09-10T19:09:48.931 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:09:48 smithi183 ceph-mon[41562]: purged_snaps scrub starts 2024-09-10T19:09:48.931 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:09:48 smithi183 ceph-mon[41562]: purged_snaps scrub ok 2024-09-10T19:09:49.931 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:09:49 smithi183 ceph-mon[29736]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-10T19:09:49.931 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:09:49 smithi183 ceph-mon[29736]: osdmap e51: 8 total, 7 up, 8 in 2024-09-10T19:09:49.931 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:09:49 smithi183 ceph-mon[29736]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-10T19:09:49.931 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:09:49 smithi183 ceph-mon[29736]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T19:09:49.931 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:09:49 smithi183 ceph-mon[29736]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:09:49.931 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:09:49 smithi183 ceph-mon[29736]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:09:49.932 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:09:49 smithi183 ceph-mon[41562]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-10T19:09:49.932 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:09:49 smithi183 ceph-mon[41562]: osdmap e51: 8 total, 7 up, 8 in 2024-09-10T19:09:49.932 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:09:49 smithi183 ceph-mon[41562]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-10T19:09:49.932 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:09:49 smithi183 ceph-mon[41562]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T19:09:49.932 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:09:49 smithi183 ceph-mon[41562]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:09:49.932 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:09:49 smithi183 ceph-mon[41562]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:09:49.966 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:09:49 smithi191 ceph-mon[32884]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-10T19:09:49.966 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:09:49 smithi191 ceph-mon[32884]: osdmap e51: 8 total, 7 up, 8 in 2024-09-10T19:09:49.966 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:09:49 smithi191 ceph-mon[32884]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-10T19:09:49.966 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:09:49 smithi191 ceph-mon[32884]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T19:09:49.966 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:09:49 smithi191 ceph-mon[32884]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:09:49.966 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:09:49 smithi191 ceph-mon[32884]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:09:51.931 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:09:51 smithi183 ceph-mon[29736]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-10T19:09:51.931 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:09:51 smithi183 ceph-mon[41562]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-10T19:09:51.965 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:09:51 smithi191 ceph-mon[32884]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-10T19:09:52.495 INFO:teuthology.orchestra.run.smithi191.stdout:Created osd(s) 7 on host 'smithi191' 2024-09-10T19:09:53.097 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:09:52 smithi191 ceph-mon[32884]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:09:53.097 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:09:52 smithi191 ceph-mon[32884]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:09:53.098 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:09:52 smithi191 ceph-mon[32884]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T19:09:53.098 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:09:52 smithi191 ceph-mon[32884]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T19:09:53.098 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:09:52 smithi191 ceph-mon[32884]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:09:53.098 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:09:52 smithi191 ceph-mon[32884]: from='osd.7 [v2:172.21.15.191:6824/1422610021,v1:172.21.15.191:6825/1422610021]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-09-10T19:09:53.098 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:09:52 smithi191 ceph-mon[32884]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-09-10T19:09:53.098 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:09:52 smithi191 ceph-mon[32884]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T19:09:53.098 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:09:52 smithi191 ceph-mon[32884]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:09:53.098 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:09:52 smithi191 ceph-mon[32884]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:09:53.121 DEBUG:teuthology.orchestra.run.smithi191:osd.7> sudo journalctl -f -n 0 -u ceph-320f6df8-6fa7-11ef-bcea-c7b262605968@osd.7.service 2024-09-10T19:09:53.123 INFO:tasks.cephadm:Waiting for 8 OSDs to come up... 2024-09-10T19:09:53.123 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph osd stat -f json 2024-09-10T19:09:53.152 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:09:52 smithi183 ceph-mon[29736]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:09:53.152 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:09:52 smithi183 ceph-mon[29736]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:09:53.152 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:09:52 smithi183 ceph-mon[29736]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T19:09:53.152 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:09:52 smithi183 ceph-mon[29736]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T19:09:53.152 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:09:52 smithi183 ceph-mon[29736]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:09:53.152 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:09:52 smithi183 ceph-mon[29736]: from='osd.7 [v2:172.21.15.191:6824/1422610021,v1:172.21.15.191:6825/1422610021]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-09-10T19:09:53.152 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:09:52 smithi183 ceph-mon[29736]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-09-10T19:09:53.152 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:09:52 smithi183 ceph-mon[29736]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T19:09:53.153 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:09:52 smithi183 ceph-mon[29736]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:09:53.153 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:09:52 smithi183 ceph-mon[29736]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:09:53.153 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:09:52 smithi183 ceph-mon[41562]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:09:53.153 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:09:52 smithi183 ceph-mon[41562]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:09:53.153 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:09:52 smithi183 ceph-mon[41562]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T19:09:53.153 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:09:52 smithi183 ceph-mon[41562]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T19:09:53.153 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:09:52 smithi183 ceph-mon[41562]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:09:53.153 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:09:52 smithi183 ceph-mon[41562]: from='osd.7 [v2:172.21.15.191:6824/1422610021,v1:172.21.15.191:6825/1422610021]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-09-10T19:09:53.153 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:09:52 smithi183 ceph-mon[41562]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-09-10T19:09:53.153 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:09:52 smithi183 ceph-mon[41562]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T19:09:53.154 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:09:52 smithi183 ceph-mon[41562]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:09:53.154 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:09:52 smithi183 ceph-mon[41562]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:09:53.378 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:09:54.177 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:09:53 smithi191 ceph-mon[32884]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-10T19:09:54.178 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:09:53 smithi191 ceph-mon[32884]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-09-10T19:09:54.178 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:09:53 smithi191 ceph-mon[32884]: osdmap e52: 8 total, 7 up, 8 in 2024-09-10T19:09:54.178 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:09:53 smithi191 ceph-mon[32884]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-10T19:09:54.178 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:09:53 smithi191 ceph-mon[32884]: from='osd.7 [v2:172.21.15.191:6824/1422610021,v1:172.21.15.191:6825/1422610021]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi191", "root=default"]}]: dispatch 2024-09-10T19:09:54.178 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:09:53 smithi191 ceph-mon[32884]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi191", "root=default"]}]: dispatch 2024-09-10T19:09:54.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:09:53 smithi183 ceph-mon[29736]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-10T19:09:54.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:09:53 smithi183 ceph-mon[29736]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-09-10T19:09:54.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:09:53 smithi183 ceph-mon[29736]: osdmap e52: 8 total, 7 up, 8 in 2024-09-10T19:09:54.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:09:53 smithi183 ceph-mon[29736]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-10T19:09:54.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:09:53 smithi183 ceph-mon[29736]: from='osd.7 [v2:172.21.15.191:6824/1422610021,v1:172.21.15.191:6825/1422610021]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi191", "root=default"]}]: dispatch 2024-09-10T19:09:54.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:09:53 smithi183 ceph-mon[29736]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi191", "root=default"]}]: dispatch 2024-09-10T19:09:54.181 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:09:53 smithi183 ceph-mon[41562]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-10T19:09:54.181 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:09:53 smithi183 ceph-mon[41562]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-09-10T19:09:54.181 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:09:53 smithi183 ceph-mon[41562]: osdmap e52: 8 total, 7 up, 8 in 2024-09-10T19:09:54.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:09:53 smithi183 ceph-mon[41562]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-10T19:09:54.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:09:53 smithi183 ceph-mon[41562]: from='osd.7 [v2:172.21.15.191:6824/1422610021,v1:172.21.15.191:6825/1422610021]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi191", "root=default"]}]: dispatch 2024-09-10T19:09:54.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:09:53 smithi183 ceph-mon[41562]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi191", "root=default"]}]: dispatch 2024-09-10T19:09:54.465 INFO:journalctl@ceph.osd.7.smithi191.stdout:Sep 10 19:09:54 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd-7[56482]: 2024-09-10T19:09:54.175+0000 7f9efec89640 -1 osd.7 0 waiting for initial osdmap 2024-09-10T19:09:54.465 INFO:journalctl@ceph.osd.7.smithi191.stdout:Sep 10 19:09:54 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd-7[56482]: 2024-09-10T19:09:54.183+0000 7f9ef6297640 -1 osd.7 53 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-10T19:09:55.256 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-09-10T19:09:55.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:09:55 smithi183 ceph-mon[29736]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi191", "root=default"]}]': finished 2024-09-10T19:09:55.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:09:55 smithi183 ceph-mon[29736]: osdmap e53: 8 total, 7 up, 8 in 2024-09-10T19:09:55.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:09:55 smithi183 ceph-mon[29736]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-10T19:09:55.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:09:55 smithi183 ceph-mon[29736]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-10T19:09:55.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:09:55 smithi183 ceph-mon[29736]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-10T19:09:55.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:09:55 smithi183 ceph-mon[29736]: osd.7 [v2:172.21.15.191:6824/1422610021,v1:172.21.15.191:6825/1422610021] boot 2024-09-10T19:09:55.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:09:55 smithi183 ceph-mon[29736]: osdmap e54: 8 total, 8 up, 8 in 2024-09-10T19:09:55.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:09:55 smithi183 ceph-mon[29736]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-10T19:09:55.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:09:55 smithi183 ceph-mon[41562]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi191", "root=default"]}]': finished 2024-09-10T19:09:55.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:09:55 smithi183 ceph-mon[41562]: osdmap e53: 8 total, 7 up, 8 in 2024-09-10T19:09:55.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:09:55 smithi183 ceph-mon[41562]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-10T19:09:55.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:09:55 smithi183 ceph-mon[41562]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-10T19:09:55.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:09:55 smithi183 ceph-mon[41562]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-10T19:09:55.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:09:55 smithi183 ceph-mon[41562]: osd.7 [v2:172.21.15.191:6824/1422610021,v1:172.21.15.191:6825/1422610021] boot 2024-09-10T19:09:55.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:09:55 smithi183 ceph-mon[41562]: osdmap e54: 8 total, 8 up, 8 in 2024-09-10T19:09:55.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:09:55 smithi183 ceph-mon[41562]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-10T19:09:55.465 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:09:55 smithi191 ceph-mon[32884]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi191", "root=default"]}]': finished 2024-09-10T19:09:55.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:09:55 smithi191 ceph-mon[32884]: osdmap e53: 8 total, 7 up, 8 in 2024-09-10T19:09:55.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:09:55 smithi191 ceph-mon[32884]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-10T19:09:55.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:09:55 smithi191 ceph-mon[32884]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-10T19:09:55.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:09:55 smithi191 ceph-mon[32884]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail 2024-09-10T19:09:55.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:09:55 smithi191 ceph-mon[32884]: osd.7 [v2:172.21.15.191:6824/1422610021,v1:172.21.15.191:6825/1422610021] boot 2024-09-10T19:09:55.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:09:55 smithi191 ceph-mon[32884]: osdmap e54: 8 total, 8 up, 8 in 2024-09-10T19:09:55.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:09:55 smithi191 ceph-mon[32884]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-10T19:09:55.835 INFO:teuthology.orchestra.run.smithi183.stdout:{"epoch":54,"num_osds":8,"num_up_osds":8,"osd_up_since":1725995394,"num_in_osds":8,"osd_in_since":1725995378,"num_remapped_pgs":1} 2024-09-10T19:09:55.836 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph osd dump --format=json 2024-09-10T19:09:56.112 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:09:56.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:09:56 smithi183 ceph-mon[29736]: purged_snaps scrub starts 2024-09-10T19:09:56.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:09:56 smithi183 ceph-mon[29736]: purged_snaps scrub ok 2024-09-10T19:09:56.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:09:56 smithi183 ceph-mon[29736]: from='client.? 172.21.15.183:0/366567282' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-09-10T19:09:56.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:09:56 smithi183 ceph-mon[29736]: purged_snaps scrub starts 2024-09-10T19:09:56.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:09:56 smithi183 ceph-mon[29736]: purged_snaps scrub ok 2024-09-10T19:09:56.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:09:56 smithi183 ceph-mon[29736]: osdmap e55: 8 total, 8 up, 8 in 2024-09-10T19:09:56.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:09:56 smithi183 ceph-mon[29736]: Detected new or changed devices on smithi191 2024-09-10T19:09:56.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:09:56 smithi183 ceph-mon[29736]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:09:56.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:09:56 smithi183 ceph-mon[29736]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:09:56.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:09:56 smithi183 ceph-mon[29736]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-10T19:09:56.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:09:56 smithi183 ceph-mon[29736]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-10T19:09:56.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:09:56 smithi183 ceph-mon[29736]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-10T19:09:56.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:09:56 smithi183 ceph-mon[29736]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-10T19:09:56.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:09:56 smithi183 ceph-mon[29736]: Adjusting osd_memory_target on smithi191 to 4272M 2024-09-10T19:09:56.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:09:56 smithi183 ceph-mon[29736]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:09:56.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:09:56 smithi183 ceph-mon[29736]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T19:09:56.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:09:56 smithi183 ceph-mon[29736]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T19:09:56.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:09:56 smithi183 ceph-mon[29736]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:09:56.433 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:09:56 smithi183 ceph-mon[41562]: purged_snaps scrub starts 2024-09-10T19:09:56.433 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:09:56 smithi183 ceph-mon[41562]: purged_snaps scrub ok 2024-09-10T19:09:56.433 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:09:56 smithi183 ceph-mon[41562]: from='client.? 172.21.15.183:0/366567282' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-09-10T19:09:56.433 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:09:56 smithi183 ceph-mon[41562]: purged_snaps scrub starts 2024-09-10T19:09:56.433 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:09:56 smithi183 ceph-mon[41562]: purged_snaps scrub ok 2024-09-10T19:09:56.433 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:09:56 smithi183 ceph-mon[41562]: osdmap e55: 8 total, 8 up, 8 in 2024-09-10T19:09:56.433 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:09:56 smithi183 ceph-mon[41562]: Detected new or changed devices on smithi191 2024-09-10T19:09:56.433 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:09:56 smithi183 ceph-mon[41562]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:09:56.433 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:09:56 smithi183 ceph-mon[41562]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:09:56.433 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:09:56 smithi183 ceph-mon[41562]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-10T19:09:56.433 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:09:56 smithi183 ceph-mon[41562]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-10T19:09:56.434 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:09:56 smithi183 ceph-mon[41562]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-10T19:09:56.434 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:09:56 smithi183 ceph-mon[41562]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-10T19:09:56.434 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:09:56 smithi183 ceph-mon[41562]: Adjusting osd_memory_target on smithi191 to 4272M 2024-09-10T19:09:56.434 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:09:56 smithi183 ceph-mon[41562]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:09:56.434 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:09:56 smithi183 ceph-mon[41562]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T19:09:56.434 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:09:56 smithi183 ceph-mon[41562]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T19:09:56.434 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:09:56 smithi183 ceph-mon[41562]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:09:56.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:09:56 smithi191 ceph-mon[32884]: purged_snaps scrub starts 2024-09-10T19:09:56.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:09:56 smithi191 ceph-mon[32884]: purged_snaps scrub ok 2024-09-10T19:09:56.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:09:56 smithi191 ceph-mon[32884]: from='client.? 172.21.15.183:0/366567282' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-09-10T19:09:56.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:09:56 smithi191 ceph-mon[32884]: purged_snaps scrub starts 2024-09-10T19:09:56.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:09:56 smithi191 ceph-mon[32884]: purged_snaps scrub ok 2024-09-10T19:09:56.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:09:56 smithi191 ceph-mon[32884]: osdmap e55: 8 total, 8 up, 8 in 2024-09-10T19:09:56.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:09:56 smithi191 ceph-mon[32884]: Detected new or changed devices on smithi191 2024-09-10T19:09:56.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:09:56 smithi191 ceph-mon[32884]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:09:56.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:09:56 smithi191 ceph-mon[32884]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:09:56.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:09:56 smithi191 ceph-mon[32884]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-10T19:09:56.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:09:56 smithi191 ceph-mon[32884]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-10T19:09:56.467 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:09:56 smithi191 ceph-mon[32884]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-10T19:09:56.467 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:09:56 smithi191 ceph-mon[32884]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-10T19:09:56.467 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:09:56 smithi191 ceph-mon[32884]: Adjusting osd_memory_target on smithi191 to 4272M 2024-09-10T19:09:56.467 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:09:56 smithi191 ceph-mon[32884]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:09:56.467 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:09:56 smithi191 ceph-mon[32884]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T19:09:56.467 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:09:56 smithi191 ceph-mon[32884]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T19:09:56.467 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:09:56 smithi191 ceph-mon[32884]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:09:57.884 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-09-10T19:09:57.885 INFO:teuthology.orchestra.run.smithi183.stdout:{"epoch":56,"fsid":"320f6df8-6fa7-11ef-bcea-c7b262605968","created":"2024-09-10T19:05:03.688531+0000","modified":"2024-09-10T19:09:56.627114+0000","last_up_change":"2024-09-10T19:09:54.624393+0000","last_in_change":"2024-09-10T19:09:38.195646+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-10T19:08:00.456786+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"9d323010-394a-49aa-ba9b-244d87faf36e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6802","nonce":2813074952},{"type":"v1","addr":"172.21.15.183:6803","nonce":2813074952}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6804","nonce":2813074952},{"type":"v1","addr":"172.21.15.183:6805","nonce":2813074952}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6808","nonce":2813074952},{"type":"v1","addr":"172.21.15.183:6809","nonce":2813074952}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6806","nonce":2813074952},{"type":"v1","addr":"172.21.15.183:6807","nonce":2813074952}]},"public_addr":"172.21.15.183:6803/2813074952","cluster_addr":"172.21.15.183:6805/2813074952","heartbeat_back_addr":"172.21.15.183:6809/2813074952","heartbeat_front_addr":"172.21.15.183:6807/2813074952","state":["exists","up"]},{"osd":1,"uuid":"71a6c5dc-5109-4284-8d20-187d8e7afc9e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":33,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6810","nonce":3880626114},{"type":"v1","addr":"172.21.15.183:6811","nonce":3880626114}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6812","nonce":3880626114},{"type":"v1","addr":"172.21.15.183:6813","nonce":3880626114}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6816","nonce":3880626114},{"type":"v1","addr":"172.21.15.183:6817","nonce":3880626114}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6814","nonce":3880626114},{"type":"v1","addr":"172.21.15.183:6815","nonce":3880626114}]},"public_addr":"172.21.15.183:6811/3880626114","cluster_addr":"172.21.15.183:6813/3880626114","heartbeat_back_addr":"172.21.15.183:6817/3880626114","heartbeat_front_addr":"172.21.15.183:6815/3880626114","state":["exists","up"]},{"osd":2,"uuid":"fc310d5b-b89b-4655-9989-7e10ba356509","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6818","nonce":3730013786},{"type":"v1","addr":"172.21.15.183:6819","nonce":3730013786}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6820","nonce":3730013786},{"type":"v1","addr":"172.21.15.183:6821","nonce":3730013786}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6824","nonce":3730013786},{"type":"v1","addr":"172.21.15.183:6825","nonce":3730013786}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6822","nonce":3730013786},{"type":"v1","addr":"172.21.15.183:6823","nonce":3730013786}]},"public_addr":"172.21.15.183:6819/3730013786","cluster_addr":"172.21.15.183:6821/3730013786","heartbeat_back_addr":"172.21.15.183:6825/3730013786","heartbeat_front_addr":"172.21.15.183:6823/3730013786","state":["exists","up"]},{"osd":3,"uuid":"7fd07a95-ad22-43f5-a968-539552283e89","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6826","nonce":3522836984},{"type":"v1","addr":"172.21.15.183:6827","nonce":3522836984}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6828","nonce":3522836984},{"type":"v1","addr":"172.21.15.183:6829","nonce":3522836984}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6832","nonce":3522836984},{"type":"v1","addr":"172.21.15.183:6833","nonce":3522836984}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6830","nonce":3522836984},{"type":"v1","addr":"172.21.15.183:6831","nonce":3522836984}]},"public_addr":"172.21.15.183:6827/3522836984","cluster_addr":"172.21.15.183:6829/3522836984","heartbeat_back_addr":"172.21.15.183:6833/3522836984","heartbeat_front_addr":"172.21.15.183:6831/3522836984","state":["exists","up"]},{"osd":4,"uuid":"631083fb-e503-44da-9b6b-154e88539ee7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6800","nonce":2674828539},{"type":"v1","addr":"172.21.15.191:6801","nonce":2674828539}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6802","nonce":2674828539},{"type":"v1","addr":"172.21.15.191:6803","nonce":2674828539}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6806","nonce":2674828539},{"type":"v1","addr":"172.21.15.191:6807","nonce":2674828539}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6804","nonce":2674828539},{"type":"v1","addr":"172.21.15.191:6805","nonce":2674828539}]},"public_addr":"172.21.15.191:6801/2674828539","cluster_addr":"172.21.15.191:6803/2674828539","heartbeat_back_addr":"172.21.15.191:6807/2674828539","heartbeat_front_addr":"172.21.15.191:6805/2674828539","state":["exists","up"]},{"osd":5,"uuid":"6b963695-3eca-4e6e-9207-05b8010ebff9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6808","nonce":681733378},{"type":"v1","addr":"172.21.15.191:6809","nonce":681733378}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6810","nonce":681733378},{"type":"v1","addr":"172.21.15.191:6811","nonce":681733378}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6814","nonce":681733378},{"type":"v1","addr":"172.21.15.191:6815","nonce":681733378}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6812","nonce":681733378},{"type":"v1","addr":"172.21.15.191:6813","nonce":681733378}]},"public_addr":"172.21.15.191:6809/681733378","cluster_addr":"172.21.15.191:6811/681733378","heartbeat_back_addr":"172.21.15.191:6815/681733378","heartbeat_front_addr":"172.21.15.191:6813/681733378","state":["exists","up"]},{"osd":6,"uuid":"2dc9754f-6aef-4e19-8605-2f1591921811","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6816","nonce":2825813477},{"type":"v1","addr":"172.21.15.191:6817","nonce":2825813477}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6818","nonce":2825813477},{"type":"v1","addr":"172.21.15.191:6819","nonce":2825813477}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6822","nonce":2825813477},{"type":"v1","addr":"172.21.15.191:6823","nonce":2825813477}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6820","nonce":2825813477},{"type":"v1","addr":"172.21.15.191:6821","nonce":2825813477}]},"public_addr":"172.21.15.191:6817/2825813477","cluster_addr":"172.21.15.191:6819/2825813477","heartbeat_back_addr":"172.21.15.191:6823/2825813477","heartbeat_front_addr":"172.21.15.191:6821/2825813477","state":["exists","up"]},{"osd":7,"uuid":"1a9a299e-d5ba-4b9a-85d5-7f6a5bc754e1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6824","nonce":1422610021},{"type":"v1","addr":"172.21.15.191:6825","nonce":1422610021}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6826","nonce":1422610021},{"type":"v1","addr":"172.21.15.191:6827","nonce":1422610021}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6830","nonce":1422610021},{"type":"v1","addr":"172.21.15.191:6831","nonce":1422610021}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6828","nonce":1422610021},{"type":"v1","addr":"172.21.15.191:6829","nonce":1422610021}]},"public_addr":"172.21.15.191:6825/1422610021","cluster_addr":"172.21.15.191:6827/1422610021","heartbeat_back_addr":"172.21.15.191:6831/1422610021","heartbeat_front_addr":"172.21.15.191:6829/1422610021","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T19:09:27.306221+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T19:09:55.356115+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T19:09:25.792445+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T19:09:47.698553+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T19:08:42.942791+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T19:09:05.612741+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T19:09:29.209723+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T19:09:53.205672+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.183:6800/659745765":"2024-09-11T19:05:48.153879+0000","172.21.15.183:0/1012899549":"2024-09-11T19:05:48.153879+0000","172.21.15.183:6800/3747320570":"2024-09-11T19:05:25.702354+0000","172.21.15.183:6801/659745765":"2024-09-11T19:05:48.153879+0000","172.21.15.183:0/2436014455":"2024-09-11T19:05:25.702354+0000","172.21.15.183:6801/3747320570":"2024-09-11T19:05:25.702354+0000","172.21.15.183:0/3044170088":"2024-09-11T19:05:25.702354+0000","172.21.15.183:0/1766036658":"2024-09-11T19:05:48.153879+0000","172.21.15.183:0/3528769414":"2024-09-11T19:05:48.153879+0000","172.21.15.183:0/3037333778":"2024-09-11T19:05:25.702354+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-10T19:09:57.885 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:09:57 smithi183 ceph-mon[29736]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:09:57.885 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:09:57 smithi183 ceph-mon[29736]: osdmap e56: 8 total, 8 up, 8 in 2024-09-10T19:09:57.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:09:57 smithi183 ceph-mon[41562]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:09:57.885 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:09:57 smithi183 ceph-mon[41562]: osdmap e56: 8 total, 8 up, 8 in 2024-09-10T19:09:57.966 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:09:57 smithi191 ceph-mon[32884]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:09:57.966 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:09:57 smithi191 ceph-mon[32884]: osdmap e56: 8 total, 8 up, 8 in 2024-09-10T19:09:58.761 INFO:tasks.cephadm.ceph_manager.ceph:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-10T19:08:00.456786+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}] 2024-09-10T19:09:58.761 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph osd pool get .mgr pg_num 2024-09-10T19:09:58.931 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:09:58 smithi183 ceph-mon[29736]: from='client.? 172.21.15.183:0/3843189407' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-10T19:09:58.931 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:09:58 smithi183 ceph-mon[41562]: from='client.? 172.21.15.183:0/3843189407' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-10T19:09:58.965 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:09:58 smithi191 ceph-mon[32884]: from='client.? 172.21.15.183:0/3843189407' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-10T19:09:59.005 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:09:59.931 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:09:59 smithi183 ceph-mon[29736]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:09:59.932 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:09:59 smithi183 ceph-mon[41562]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:09:59.965 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:09:59 smithi191 ceph-mon[32884]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:10:00.756 INFO:teuthology.orchestra.run.smithi183.stdout:pg_num: 1 2024-09-10T19:10:00.931 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:10:00 smithi183 ceph-mon[29736]: overall HEALTH_OK 2024-09-10T19:10:00.931 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:10:00 smithi183 ceph-mon[41562]: overall HEALTH_OK 2024-09-10T19:10:00.965 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:10:00 smithi191 ceph-mon[32884]: overall HEALTH_OK 2024-09-10T19:10:01.462 INFO:tasks.cephadm:Adding ceph.rgw.foo.a on smithi183 2024-09-10T19:10:01.463 DEBUG:teuthology.orchestra.run.smithi191:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph orch apply rgw foo.a --placement '1;smithi183=foo.a' 2024-09-10T19:10:01.720 INFO:teuthology.orchestra.run.smithi191.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.b/config 2024-09-10T19:10:01.840 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:10:01 smithi191 ceph-mon[32884]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:10:01.841 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:10:01 smithi191 ceph-mon[32884]: purged_snaps scrub starts 2024-09-10T19:10:01.841 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:10:01 smithi191 ceph-mon[32884]: purged_snaps scrub ok 2024-09-10T19:10:01.841 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:10:01 smithi191 ceph-mon[32884]: osdmap e57: 8 total, 8 up, 8 in 2024-09-10T19:10:01.841 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:10:01 smithi191 ceph-mon[32884]: from='client.? 172.21.15.183:0/2136217565' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-09-10T19:10:01.931 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:10:01 smithi183 ceph-mon[41562]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:10:01.931 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:10:01 smithi183 ceph-mon[41562]: purged_snaps scrub starts 2024-09-10T19:10:01.931 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:10:01 smithi183 ceph-mon[41562]: purged_snaps scrub ok 2024-09-10T19:10:01.932 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:10:01 smithi183 ceph-mon[41562]: osdmap e57: 8 total, 8 up, 8 in 2024-09-10T19:10:01.932 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:10:01 smithi183 ceph-mon[41562]: from='client.? 172.21.15.183:0/2136217565' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-09-10T19:10:01.932 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:10:01 smithi183 ceph-mon[29736]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:10:01.932 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:10:01 smithi183 ceph-mon[29736]: purged_snaps scrub starts 2024-09-10T19:10:01.932 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:10:01 smithi183 ceph-mon[29736]: purged_snaps scrub ok 2024-09-10T19:10:01.932 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:10:01 smithi183 ceph-mon[29736]: osdmap e57: 8 total, 8 up, 8 in 2024-09-10T19:10:01.932 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:10:01 smithi183 ceph-mon[29736]: from='client.? 172.21.15.183:0/2136217565' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-09-10T19:10:03.380 INFO:teuthology.orchestra.run.smithi191.stdout:Scheduled rgw.foo.a update... 2024-09-10T19:10:03.929 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:10:03 smithi183 ceph-mon[29736]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:10:03.929 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:10:03 smithi183 ceph-mon[29736]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:10:03.929 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:10:03 smithi183 ceph-mon[29736]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T19:10:03.929 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:10:03 smithi183 ceph-mon[29736]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T19:10:03.929 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:10:03 smithi183 ceph-mon[29736]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T19:10:03.929 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:10:03 smithi183 ceph-mon[29736]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:10:03.929 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:10:03 smithi183 ceph-mon[29736]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-09-10T19:10:03.930 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:10:03 smithi183 ceph-mon[29736]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-09-10T19:10:03.930 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:10:03 smithi183 ceph-mon[29736]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:10:03.930 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:10:03 smithi183 ceph-mon[29736]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T19:10:03.930 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:10:03 smithi183 ceph-mon[41562]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:10:03.930 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:10:03 smithi183 ceph-mon[41562]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:10:03.930 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:10:03 smithi183 ceph-mon[41562]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T19:10:03.930 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:10:03 smithi183 ceph-mon[41562]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T19:10:03.930 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:10:03 smithi183 ceph-mon[41562]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T19:10:03.930 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:10:03 smithi183 ceph-mon[41562]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:10:03.930 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:10:03 smithi183 ceph-mon[41562]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-09-10T19:10:03.930 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:10:03 smithi183 ceph-mon[41562]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-09-10T19:10:03.931 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:10:03 smithi183 ceph-mon[41562]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:10:03.931 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:10:03 smithi183 ceph-mon[41562]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T19:10:03.934 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:10:03 smithi191 ceph-mon[32884]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:10:03.934 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:10:03 smithi191 ceph-mon[32884]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:10:03.934 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:10:03 smithi191 ceph-mon[32884]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T19:10:03.934 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:10:03 smithi191 ceph-mon[32884]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T19:10:03.934 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:10:03 smithi191 ceph-mon[32884]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T19:10:03.934 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:10:03 smithi191 ceph-mon[32884]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:10:03.935 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:10:03 smithi191 ceph-mon[32884]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-09-10T19:10:03.935 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:10:03 smithi191 ceph-mon[32884]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-09-10T19:10:03.935 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:10:03 smithi191 ceph-mon[32884]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:10:03.935 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:10:03 smithi191 ceph-mon[32884]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T19:10:03.958 DEBUG:teuthology.orchestra.run.smithi183:rgw.foo.a> sudo journalctl -f -n 0 -u ceph-320f6df8-6fa7-11ef-bcea-c7b262605968@rgw.foo.a.service 2024-09-10T19:10:03.961 INFO:tasks.cephadm:Adding ceph.iscsi.iscsi.a on smithi191 2024-09-10T19:10:03.961 DEBUG:teuthology.orchestra.run.smithi191:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph osd pool create datapool 3 3 replicated 2024-09-10T19:10:04.200 INFO:teuthology.orchestra.run.smithi191.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.b/config 2024-09-10T19:10:04.877 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:10:04 smithi183 ceph-mon[29736]: from='client.24310 -' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "foo.a", "placement": "1;smithi183=foo.a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-10T19:10:04.878 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:10:04 smithi183 ceph-mon[29736]: Saving service rgw.foo.a spec with placement smithi183=foo.a;count:1 2024-09-10T19:10:04.878 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:10:04 smithi183 ceph-mon[29736]: Deploying daemon rgw.foo.a on smithi183 2024-09-10T19:10:04.878 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:10:04 smithi183 ceph-mon[41562]: from='client.24310 -' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "foo.a", "placement": "1;smithi183=foo.a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-10T19:10:04.878 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:10:04 smithi183 ceph-mon[41562]: Saving service rgw.foo.a spec with placement smithi183=foo.a;count:1 2024-09-10T19:10:04.878 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:10:04 smithi183 ceph-mon[41562]: Deploying daemon rgw.foo.a on smithi183 2024-09-10T19:10:04.965 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:10:04 smithi191 ceph-mon[32884]: from='client.24310 -' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "foo.a", "placement": "1;smithi183=foo.a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-10T19:10:04.965 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:10:04 smithi191 ceph-mon[32884]: Saving service rgw.foo.a spec with placement smithi183=foo.a;count:1 2024-09-10T19:10:04.966 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:10:04 smithi191 ceph-mon[32884]: Deploying daemon rgw.foo.a on smithi183 2024-09-10T19:10:05.619 INFO:journalctl@ceph.rgw.foo.a.smithi183.stdout:Sep 10 19:10:05 smithi183 podman[70558]: 2024-09-10 19:10:05.503449896 +0000 UTC m=+0.124815536 container create 13ddc2616ff7061707e71a81e5b7090a646126088d968f8f6d8cbb1b10ad530e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3bfeaf4417585c821cab9ca76e4d95f390b0e769807122a45463640820b5cd16, name=ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-rgw-foo-a, org.label-schema.license=GPLv2, ceph=True, GIT_BRANCH=HEAD, RELEASE=wip-guits-quincy-2024-09-09-1810-quincy-5be5739, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, GIT_CLEAN=True) 2024-09-10T19:10:05.931 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:10:05 smithi183 ceph-mon[29736]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:10:05.931 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:10:05 smithi183 ceph-mon[41562]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:10:05.931 INFO:journalctl@ceph.rgw.foo.a.smithi183.stdout:Sep 10 19:10:05 smithi183 podman[70558]: 2024-09-10 19:10:05.714230919 +0000 UTC m=+0.335596564 container init 13ddc2616ff7061707e71a81e5b7090a646126088d968f8f6d8cbb1b10ad530e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3bfeaf4417585c821cab9ca76e4d95f390b0e769807122a45463640820b5cd16, name=ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-rgw-foo-a, org.label-schema.license=GPLv2, RELEASE=wip-guits-quincy-2024-09-09-1810-quincy-5be5739, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.build-date=20240731, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-10T19:10:05.931 INFO:journalctl@ceph.rgw.foo.a.smithi183.stdout:Sep 10 19:10:05 smithi183 podman[70558]: 2024-09-10 19:10:05.718790685 +0000 UTC m=+0.340156327 container start 13ddc2616ff7061707e71a81e5b7090a646126088d968f8f6d8cbb1b10ad530e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3bfeaf4417585c821cab9ca76e4d95f390b0e769807122a45463640820b5cd16, name=ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-rgw-foo-a, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, ceph=True, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, RELEASE=wip-guits-quincy-2024-09-09-1810-quincy-5be5739) 2024-09-10T19:10:05.932 INFO:journalctl@ceph.rgw.foo.a.smithi183.stdout:Sep 10 19:10:05 smithi183 bash[70558]: 13ddc2616ff7061707e71a81e5b7090a646126088d968f8f6d8cbb1b10ad530e 2024-09-10T19:10:05.932 INFO:journalctl@ceph.rgw.foo.a.smithi183.stdout:Sep 10 19:10:05 smithi183 systemd[1]: Started Ceph rgw.foo.a for 320f6df8-6fa7-11ef-bcea-c7b262605968. 2024-09-10T19:10:05.965 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:10:05 smithi191 ceph-mon[32884]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:10:06.676 INFO:teuthology.orchestra.run.smithi191.stderr:pool 'datapool' created 2024-09-10T19:10:06.918 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:10:06 smithi183 ceph-mon[41562]: from='client.? 172.21.15.191:0/128383219' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]: dispatch 2024-09-10T19:10:06.918 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:10:06 smithi183 ceph-mon[41562]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]: dispatch 2024-09-10T19:10:06.918 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:10:06 smithi183 ceph-mon[41562]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:10:06.918 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:10:06 smithi183 ceph-mon[41562]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:10:06.918 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:10:06 smithi183 ceph-mon[41562]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:10:06.918 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:10:06 smithi183 ceph-mon[41562]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:10:06.918 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:10:06 smithi183 ceph-mon[41562]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:10:06.918 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:10:06 smithi183 ceph-mon[41562]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T19:10:06.919 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:10:06 smithi183 ceph-mon[29736]: from='client.? 172.21.15.191:0/128383219' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]: dispatch 2024-09-10T19:10:06.919 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:10:06 smithi183 ceph-mon[29736]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]: dispatch 2024-09-10T19:10:06.919 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:10:06 smithi183 ceph-mon[29736]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:10:06.919 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:10:06 smithi183 ceph-mon[29736]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:10:06.919 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:10:06 smithi183 ceph-mon[29736]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:10:06.919 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:10:06 smithi183 ceph-mon[29736]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:10:06.919 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:10:06 smithi183 ceph-mon[29736]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:10:06.919 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:10:06 smithi183 ceph-mon[29736]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T19:10:06.965 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:10:06 smithi191 ceph-mon[32884]: from='client.? 172.21.15.191:0/128383219' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]: dispatch 2024-09-10T19:10:06.966 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:10:06 smithi191 ceph-mon[32884]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]: dispatch 2024-09-10T19:10:06.966 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:10:06 smithi191 ceph-mon[32884]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:10:06.966 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:10:06 smithi191 ceph-mon[32884]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:10:06.966 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:10:06 smithi191 ceph-mon[32884]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:10:06.966 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:10:06 smithi191 ceph-mon[32884]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:10:06.966 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:10:06 smithi191 ceph-mon[32884]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:10:06.966 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:10:06 smithi191 ceph-mon[32884]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T19:10:07.312 DEBUG:teuthology.orchestra.run.smithi191:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- rbd pool init datapool 2024-09-10T19:10:07.550 INFO:teuthology.orchestra.run.smithi191.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.b/config 2024-09-10T19:10:07.913 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:10:07 smithi191 ceph-mon[32884]: pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:10:07.913 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:10:07 smithi191 ceph-mon[32884]: Saving service rgw.foo.a spec with placement smithi183=foo.a;count:1 2024-09-10T19:10:07.913 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:10:07 smithi191 ceph-mon[32884]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]': finished 2024-09-10T19:10:07.913 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:10:07 smithi191 ceph-mon[32884]: osdmap e58: 8 total, 8 up, 8 in 2024-09-10T19:10:07.913 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:10:07 smithi191 ceph-mon[32884]: from='client.? 172.21.15.183:0/3843093913' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-09-10T19:10:07.914 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:10:07 smithi191 ceph-mon[32884]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:10:07.914 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:10:07 smithi191 ceph-mon[32884]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:10:07.914 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:10:07 smithi191 ceph-mon[32884]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T19:10:07.914 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:10:07 smithi191 ceph-mon[32884]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T19:10:08.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:10:07 smithi183 ceph-mon[29736]: pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:10:08.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:10:07 smithi183 ceph-mon[29736]: Saving service rgw.foo.a spec with placement smithi183=foo.a;count:1 2024-09-10T19:10:08.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:10:07 smithi183 ceph-mon[29736]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]': finished 2024-09-10T19:10:08.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:10:07 smithi183 ceph-mon[29736]: osdmap e58: 8 total, 8 up, 8 in 2024-09-10T19:10:08.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:10:07 smithi183 ceph-mon[29736]: from='client.? 172.21.15.183:0/3843093913' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-09-10T19:10:08.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:10:07 smithi183 ceph-mon[29736]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:10:08.182 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:10:07 smithi183 ceph-mon[29736]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:10:08.182 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:10:07 smithi183 ceph-mon[29736]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T19:10:08.182 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:10:07 smithi183 ceph-mon[29736]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T19:10:08.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:10:07 smithi183 ceph-mon[41562]: pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:10:08.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:10:07 smithi183 ceph-mon[41562]: Saving service rgw.foo.a spec with placement smithi183=foo.a;count:1 2024-09-10T19:10:08.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:10:07 smithi183 ceph-mon[41562]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]': finished 2024-09-10T19:10:08.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:10:07 smithi183 ceph-mon[41562]: osdmap e58: 8 total, 8 up, 8 in 2024-09-10T19:10:08.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:10:07 smithi183 ceph-mon[41562]: from='client.? 172.21.15.183:0/3843093913' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-09-10T19:10:08.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:10:07 smithi183 ceph-mon[41562]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:10:08.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:10:07 smithi183 ceph-mon[41562]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:10:08.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:10:07 smithi183 ceph-mon[41562]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T19:10:08.183 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:10:07 smithi183 ceph-mon[41562]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T19:10:08.951 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:10:08 smithi191 ceph-mon[32884]: Checking dashboard <-> RGW credentials 2024-09-10T19:10:08.951 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:10:08 smithi191 ceph-mon[32884]: from='client.? 172.21.15.183:0/3843093913' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2024-09-10T19:10:08.951 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:10:08 smithi191 ceph-mon[32884]: osdmap e59: 8 total, 8 up, 8 in 2024-09-10T19:10:08.951 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:10:08 smithi191 ceph-mon[32884]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:10:08.952 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:10:08 smithi191 ceph-mon[32884]: osdmap e60: 8 total, 8 up, 8 in 2024-09-10T19:10:09.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:10:08 smithi183 ceph-mon[29736]: Checking dashboard <-> RGW credentials 2024-09-10T19:10:09.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:10:08 smithi183 ceph-mon[29736]: from='client.? 172.21.15.183:0/3843093913' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2024-09-10T19:10:09.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:10:08 smithi183 ceph-mon[29736]: osdmap e59: 8 total, 8 up, 8 in 2024-09-10T19:10:09.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:10:08 smithi183 ceph-mon[29736]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:10:09.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:10:08 smithi183 ceph-mon[29736]: osdmap e60: 8 total, 8 up, 8 in 2024-09-10T19:10:09.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:10:08 smithi183 ceph-mon[41562]: Checking dashboard <-> RGW credentials 2024-09-10T19:10:09.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:10:08 smithi183 ceph-mon[41562]: from='client.? 172.21.15.183:0/3843093913' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2024-09-10T19:10:09.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:10:08 smithi183 ceph-mon[41562]: osdmap e59: 8 total, 8 up, 8 in 2024-09-10T19:10:09.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:10:08 smithi183 ceph-mon[41562]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:10:09.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:10:08 smithi183 ceph-mon[41562]: osdmap e60: 8 total, 8 up, 8 in 2024-09-10T19:10:10.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:10:09 smithi191 ceph-mon[32884]: pgmap v179: 36 pgs: 8 creating+peering, 27 unknown, 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:10:10.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:10:09 smithi191 ceph-mon[32884]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T19:10:10.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:10:09 smithi191 ceph-mon[32884]: from='client.? 172.21.15.183:0/2551009346' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-09-10T19:10:10.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:10:09 smithi191 ceph-mon[32884]: from='client.? 172.21.15.183:0/3843093913' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-09-10T19:10:10.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:10:09 smithi191 ceph-mon[32884]: from='client.? ' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-09-10T19:10:10.217 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:10:09 smithi191 ceph-mon[32884]: from='client.? 172.21.15.191:0/1627397251' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]: dispatch 2024-09-10T19:10:10.217 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:10:09 smithi191 ceph-mon[32884]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]: dispatch 2024-09-10T19:10:10.217 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:10:09 smithi191 ceph-mon[32884]: from='client.? 172.21.15.183:0/3843093913' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-09-10T19:10:10.217 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:10:09 smithi191 ceph-mon[32884]: from='client.? ' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-09-10T19:10:10.217 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:10:09 smithi191 ceph-mon[32884]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]': finished 2024-09-10T19:10:10.217 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:10:09 smithi191 ceph-mon[32884]: osdmap e61: 8 total, 8 up, 8 in 2024-09-10T19:10:10.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:10:09 smithi183 ceph-mon[29736]: pgmap v179: 36 pgs: 8 creating+peering, 27 unknown, 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:10:10.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:10:09 smithi183 ceph-mon[29736]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T19:10:10.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:10:09 smithi183 ceph-mon[29736]: from='client.? 172.21.15.183:0/2551009346' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-09-10T19:10:10.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:10:09 smithi183 ceph-mon[29736]: from='client.? 172.21.15.183:0/3843093913' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-09-10T19:10:10.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:10:09 smithi183 ceph-mon[29736]: from='client.? ' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-09-10T19:10:10.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:10:09 smithi183 ceph-mon[29736]: from='client.? 172.21.15.191:0/1627397251' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]: dispatch 2024-09-10T19:10:10.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:10:09 smithi183 ceph-mon[29736]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]: dispatch 2024-09-10T19:10:10.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:10:09 smithi183 ceph-mon[29736]: from='client.? 172.21.15.183:0/3843093913' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-09-10T19:10:10.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:10:09 smithi183 ceph-mon[29736]: from='client.? ' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-09-10T19:10:10.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:10:09 smithi183 ceph-mon[29736]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]': finished 2024-09-10T19:10:10.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:10:09 smithi183 ceph-mon[29736]: osdmap e61: 8 total, 8 up, 8 in 2024-09-10T19:10:10.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:10:09 smithi183 ceph-mon[41562]: pgmap v179: 36 pgs: 8 creating+peering, 27 unknown, 1 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:10:10.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:10:09 smithi183 ceph-mon[41562]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T19:10:10.433 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:10:09 smithi183 ceph-mon[41562]: from='client.? 172.21.15.183:0/2551009346' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-09-10T19:10:10.433 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:10:09 smithi183 ceph-mon[41562]: from='client.? 172.21.15.183:0/3843093913' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-09-10T19:10:10.433 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:10:09 smithi183 ceph-mon[41562]: from='client.? ' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-09-10T19:10:10.433 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:10:09 smithi183 ceph-mon[41562]: from='client.? 172.21.15.191:0/1627397251' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]: dispatch 2024-09-10T19:10:10.433 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:10:09 smithi183 ceph-mon[41562]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]: dispatch 2024-09-10T19:10:10.433 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:10:09 smithi183 ceph-mon[41562]: from='client.? 172.21.15.183:0/3843093913' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-09-10T19:10:10.433 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:10:09 smithi183 ceph-mon[41562]: from='client.? ' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-09-10T19:10:10.433 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:10:09 smithi183 ceph-mon[41562]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]': finished 2024-09-10T19:10:10.433 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:10:09 smithi183 ceph-mon[41562]: osdmap e61: 8 total, 8 up, 8 in 2024-09-10T19:10:11.965 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:10:11 smithi191 ceph-mon[32884]: pgmap v182: 68 pgs: 15 creating+peering, 27 unknown, 26 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.5 KiB/s rd, 767 B/s wr, 4 op/s 2024-09-10T19:10:11.965 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:10:11 smithi191 ceph-mon[32884]: osdmap e62: 8 total, 8 up, 8 in 2024-09-10T19:10:11.965 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:10:11 smithi191 ceph-mon[32884]: from='client.? 172.21.15.183:0/3843093913' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-09-10T19:10:11.966 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:10:11 smithi191 ceph-mon[32884]: from='client.? 172.21.15.183:0/2551009346' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-09-10T19:10:11.966 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:10:11 smithi191 ceph-mon[32884]: from='client.? ' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-09-10T19:10:11.966 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:10:11 smithi191 ceph-mon[32884]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-10T19:10:11.966 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:10:11 smithi191 ceph-mon[32884]: Cluster is now healthy 2024-09-10T19:10:12.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:10:11 smithi183 ceph-mon[29736]: pgmap v182: 68 pgs: 15 creating+peering, 27 unknown, 26 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.5 KiB/s rd, 767 B/s wr, 4 op/s 2024-09-10T19:10:12.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:10:11 smithi183 ceph-mon[29736]: osdmap e62: 8 total, 8 up, 8 in 2024-09-10T19:10:12.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:10:11 smithi183 ceph-mon[29736]: from='client.? 172.21.15.183:0/3843093913' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-09-10T19:10:12.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:10:11 smithi183 ceph-mon[29736]: from='client.? 172.21.15.183:0/2551009346' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-09-10T19:10:12.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:10:11 smithi183 ceph-mon[29736]: from='client.? ' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-09-10T19:10:12.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:10:11 smithi183 ceph-mon[29736]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-10T19:10:12.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:10:11 smithi183 ceph-mon[29736]: Cluster is now healthy 2024-09-10T19:10:12.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:10:11 smithi183 ceph-mon[41562]: pgmap v182: 68 pgs: 15 creating+peering, 27 unknown, 26 active+clean; 577 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 3.5 KiB/s rd, 767 B/s wr, 4 op/s 2024-09-10T19:10:12.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:10:11 smithi183 ceph-mon[41562]: osdmap e62: 8 total, 8 up, 8 in 2024-09-10T19:10:12.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:10:11 smithi183 ceph-mon[41562]: from='client.? 172.21.15.183:0/3843093913' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-09-10T19:10:12.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:10:11 smithi183 ceph-mon[41562]: from='client.? 172.21.15.183:0/2551009346' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-09-10T19:10:12.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:10:11 smithi183 ceph-mon[41562]: from='client.? ' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-09-10T19:10:12.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:10:11 smithi183 ceph-mon[41562]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-10T19:10:12.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:10:11 smithi183 ceph-mon[41562]: Cluster is now healthy 2024-09-10T19:10:12.324 DEBUG:teuthology.orchestra.run.smithi191:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph orch apply iscsi datapool admin admin --trusted_ip_list 172.21.15.191 --placement '1;smithi191=iscsi.a' 2024-09-10T19:10:12.566 INFO:teuthology.orchestra.run.smithi191.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.b/config 2024-09-10T19:10:12.926 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:10:12 smithi191 ceph-mon[32884]: from='client.? 172.21.15.183:0/3843093913' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-09-10T19:10:12.926 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:10:12 smithi191 ceph-mon[32884]: from='client.? ' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-09-10T19:10:12.926 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:10:12 smithi191 ceph-mon[32884]: osdmap e63: 8 total, 8 up, 8 in 2024-09-10T19:10:13.181 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:10:12 smithi183 ceph-mon[41562]: from='client.? 172.21.15.183:0/3843093913' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-09-10T19:10:13.181 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:10:12 smithi183 ceph-mon[41562]: from='client.? ' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-09-10T19:10:13.181 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:10:12 smithi183 ceph-mon[41562]: osdmap e63: 8 total, 8 up, 8 in 2024-09-10T19:10:13.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:10:12 smithi183 ceph-mon[29736]: from='client.? 172.21.15.183:0/3843093913' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-09-10T19:10:13.182 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:10:12 smithi183 ceph-mon[29736]: from='client.? ' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-09-10T19:10:13.182 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:10:12 smithi183 ceph-mon[29736]: osdmap e63: 8 total, 8 up, 8 in 2024-09-10T19:10:13.970 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:10:13 smithi191 ceph-mon[32884]: pgmap v185: 100 pgs: 15 creating+peering, 28 unknown, 57 active+clean; 578 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 4.5 KiB/s rd, 1023 B/s wr, 5 op/s 2024-09-10T19:10:13.970 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:10:13 smithi191 ceph-mon[32884]: osdmap e64: 8 total, 8 up, 8 in 2024-09-10T19:10:13.970 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:10:13 smithi191 ceph-mon[32884]: from='client.? 172.21.15.183:0/2540042807' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-09-10T19:10:13.970 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:10:13 smithi191 ceph-mon[32884]: from='client.? 172.21.15.183:0/904613203' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-09-10T19:10:13.970 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:10:13 smithi191 ceph-mon[32884]: from='client.? ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-09-10T19:10:14.181 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:10:13 smithi183 ceph-mon[41562]: pgmap v185: 100 pgs: 15 creating+peering, 28 unknown, 57 active+clean; 578 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 4.5 KiB/s rd, 1023 B/s wr, 5 op/s 2024-09-10T19:10:14.181 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:10:13 smithi183 ceph-mon[41562]: osdmap e64: 8 total, 8 up, 8 in 2024-09-10T19:10:14.181 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:10:13 smithi183 ceph-mon[41562]: from='client.? 172.21.15.183:0/2540042807' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-09-10T19:10:14.181 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:10:13 smithi183 ceph-mon[41562]: from='client.? 172.21.15.183:0/904613203' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-09-10T19:10:14.181 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:10:13 smithi183 ceph-mon[41562]: from='client.? ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-09-10T19:10:14.182 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:10:13 smithi183 ceph-mon[29736]: pgmap v185: 100 pgs: 15 creating+peering, 28 unknown, 57 active+clean; 578 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 4.5 KiB/s rd, 1023 B/s wr, 5 op/s 2024-09-10T19:10:14.182 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:10:13 smithi183 ceph-mon[29736]: osdmap e64: 8 total, 8 up, 8 in 2024-09-10T19:10:14.182 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:10:13 smithi183 ceph-mon[29736]: from='client.? 172.21.15.183:0/2540042807' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-09-10T19:10:14.182 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:10:13 smithi183 ceph-mon[29736]: from='client.? 172.21.15.183:0/904613203' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-09-10T19:10:14.182 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:10:13 smithi183 ceph-mon[29736]: from='client.? ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-09-10T19:10:14.411 INFO:teuthology.orchestra.run.smithi191.stdout:Scheduled iscsi.datapool update... 2024-09-10T19:10:15.051 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:10:14 smithi191 ceph-mon[32884]: from='client.? 172.21.15.183:0/2540042807' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-09-10T19:10:15.052 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:10:14 smithi191 ceph-mon[32884]: from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-09-10T19:10:15.052 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:10:14 smithi191 ceph-mon[32884]: osdmap e65: 8 total, 8 up, 8 in 2024-09-10T19:10:15.052 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:10:14 smithi191 ceph-mon[32884]: from='client.? 172.21.15.183:0/2540042807' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-09-10T19:10:15.052 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:10:14 smithi191 ceph-mon[32884]: from='client.? 172.21.15.183:0/904613203' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-09-10T19:10:15.052 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:10:14 smithi191 ceph-mon[32884]: from='client.? ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-09-10T19:10:15.052 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:10:14 smithi191 ceph-mon[32884]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:10:15.052 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:10:14 smithi191 ceph-mon[32884]: from='client.? 172.21.15.183:0/2540042807' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-09-10T19:10:15.052 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:10:14 smithi191 ceph-mon[32884]: from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-09-10T19:10:15.053 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:10:14 smithi191 ceph-mon[32884]: osdmap e66: 8 total, 8 up, 8 in 2024-09-10T19:10:15.075 INFO:tasks.cephadm:Distributing iscsi-gateway.cfg... 2024-09-10T19:10:15.075 DEBUG:teuthology.orchestra.run.smithi183:> set -ex 2024-09-10T19:10:15.076 DEBUG:teuthology.orchestra.run.smithi183:> sudo dd of=/etc/ceph/iscsi-gateway.cfg 2024-09-10T19:10:15.081 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:10:14 smithi183 ceph-mon[29736]: from='client.? 172.21.15.183:0/2540042807' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-09-10T19:10:15.081 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:10:14 smithi183 ceph-mon[29736]: from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-09-10T19:10:15.082 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:10:14 smithi183 ceph-mon[29736]: osdmap e65: 8 total, 8 up, 8 in 2024-09-10T19:10:15.082 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:10:14 smithi183 ceph-mon[29736]: from='client.? 172.21.15.183:0/2540042807' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-09-10T19:10:15.082 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:10:14 smithi183 ceph-mon[29736]: from='client.? 172.21.15.183:0/904613203' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-09-10T19:10:15.082 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:10:14 smithi183 ceph-mon[29736]: from='client.? ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-09-10T19:10:15.082 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:10:14 smithi183 ceph-mon[29736]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:10:15.082 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:10:14 smithi183 ceph-mon[29736]: from='client.? 172.21.15.183:0/2540042807' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-09-10T19:10:15.082 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:10:14 smithi183 ceph-mon[29736]: from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-09-10T19:10:15.082 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:10:14 smithi183 ceph-mon[29736]: osdmap e66: 8 total, 8 up, 8 in 2024-09-10T19:10:15.082 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:10:14 smithi183 ceph-mon[41562]: from='client.? 172.21.15.183:0/2540042807' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-09-10T19:10:15.082 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:10:14 smithi183 ceph-mon[41562]: from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-09-10T19:10:15.082 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:10:14 smithi183 ceph-mon[41562]: osdmap e65: 8 total, 8 up, 8 in 2024-09-10T19:10:15.083 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:10:14 smithi183 ceph-mon[41562]: from='client.? 172.21.15.183:0/2540042807' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-09-10T19:10:15.083 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:10:14 smithi183 ceph-mon[41562]: from='client.? 172.21.15.183:0/904613203' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-09-10T19:10:15.083 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:10:14 smithi183 ceph-mon[41562]: from='client.? ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-09-10T19:10:15.083 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:10:14 smithi183 ceph-mon[41562]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:10:15.083 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:10:14 smithi183 ceph-mon[41562]: from='client.? 172.21.15.183:0/2540042807' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-09-10T19:10:15.083 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:10:14 smithi183 ceph-mon[41562]: from='client.? ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-09-10T19:10:15.083 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:10:14 smithi183 ceph-mon[41562]: osdmap e66: 8 total, 8 up, 8 in 2024-09-10T19:10:15.115 DEBUG:teuthology.orchestra.run.smithi191:> set -ex 2024-09-10T19:10:15.115 DEBUG:teuthology.orchestra.run.smithi191:> sudo dd of=/etc/ceph/iscsi-gateway.cfg 2024-09-10T19:10:15.143 DEBUG:teuthology.orchestra.run.smithi191:iscsi.iscsi.a> sudo journalctl -f -n 0 -u ceph-320f6df8-6fa7-11ef-bcea-c7b262605968@iscsi.iscsi.a.service 2024-09-10T19:10:15.186 INFO:tasks.cephadm:Adding prometheus.a on smithi191 2024-09-10T19:10:15.186 DEBUG:teuthology.orchestra.run.smithi191:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph orch apply prometheus '1;smithi191=a' 2024-09-10T19:10:15.943 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:10:15 smithi191 ceph-mon[32884]: pgmap v188: 132 pgs: 10 creating+peering, 53 unknown, 69 active+clean; 578 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 255 B/s wr, 1 op/s 2024-09-10T19:10:15.943 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:10:15 smithi191 ceph-mon[32884]: from='client.24367 -' entity='client.admin' cmd=[{"prefix": "orch apply iscsi", "pool": "datapool", "api_user": "admin", "api_password": "admin", "trusted_ip_list": "172.21.15.191", "placement": "1;smithi191=iscsi.a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-10T19:10:15.943 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:10:15 smithi191 ceph-mon[32884]: Saving service iscsi.datapool spec with placement smithi191=iscsi.a;count:1 2024-09-10T19:10:15.943 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:10:15 smithi191 ceph-mon[32884]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:10:15.943 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:10:15 smithi191 ceph-mon[32884]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:10:15.943 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:10:15 smithi191 ceph-mon[32884]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:10:15.943 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:10:15 smithi191 ceph-mon[32884]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-09-10T19:10:15.943 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:10:15 smithi191 ceph-mon[32884]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]': finished 2024-09-10T19:10:15.943 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:10:15 smithi191 ceph-mon[32884]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T19:10:16.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:10:15 smithi183 ceph-mon[29736]: pgmap v188: 132 pgs: 10 creating+peering, 53 unknown, 69 active+clean; 578 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 255 B/s wr, 1 op/s 2024-09-10T19:10:16.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:10:15 smithi183 ceph-mon[29736]: from='client.24367 -' entity='client.admin' cmd=[{"prefix": "orch apply iscsi", "pool": "datapool", "api_user": "admin", "api_password": "admin", "trusted_ip_list": "172.21.15.191", "placement": "1;smithi191=iscsi.a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-10T19:10:16.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:10:15 smithi183 ceph-mon[29736]: Saving service iscsi.datapool spec with placement smithi191=iscsi.a;count:1 2024-09-10T19:10:16.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:10:15 smithi183 ceph-mon[29736]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:10:16.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:10:15 smithi183 ceph-mon[29736]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:10:16.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:10:15 smithi183 ceph-mon[29736]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:10:16.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:10:15 smithi183 ceph-mon[29736]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-09-10T19:10:16.182 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:10:15 smithi183 ceph-mon[29736]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]': finished 2024-09-10T19:10:16.182 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:10:15 smithi183 ceph-mon[29736]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T19:10:16.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:10:15 smithi183 ceph-mon[41562]: pgmap v188: 132 pgs: 10 creating+peering, 53 unknown, 69 active+clean; 578 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 255 B/s wr, 1 op/s 2024-09-10T19:10:16.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:10:15 smithi183 ceph-mon[41562]: from='client.24367 -' entity='client.admin' cmd=[{"prefix": "orch apply iscsi", "pool": "datapool", "api_user": "admin", "api_password": "admin", "trusted_ip_list": "172.21.15.191", "placement": "1;smithi191=iscsi.a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-10T19:10:16.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:10:15 smithi183 ceph-mon[41562]: Saving service iscsi.datapool spec with placement smithi191=iscsi.a;count:1 2024-09-10T19:10:16.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:10:15 smithi183 ceph-mon[41562]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:10:16.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:10:15 smithi183 ceph-mon[41562]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:10:16.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:10:15 smithi183 ceph-mon[41562]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:10:16.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:10:15 smithi183 ceph-mon[41562]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-09-10T19:10:16.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:10:15 smithi183 ceph-mon[41562]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]': finished 2024-09-10T19:10:16.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:10:15 smithi183 ceph-mon[41562]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T19:10:16.219 INFO:teuthology.orchestra.run.smithi191.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.b/config 2024-09-10T19:10:16.965 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:10:16 smithi191 ceph-mon[32884]: Deploying daemon iscsi.iscsi.a on smithi191 2024-09-10T19:10:17.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:10:16 smithi183 ceph-mon[29736]: Deploying daemon iscsi.iscsi.a on smithi191 2024-09-10T19:10:17.181 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:10:16 smithi183 ceph-mon[41562]: Deploying daemon iscsi.iscsi.a on smithi191 2024-09-10T19:10:17.819 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:10:17 smithi191 ceph-mon[32884]: pgmap v190: 132 pgs: 9 unknown, 123 active+clean; 580 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 38 KiB/s rd, 5.1 KiB/s wr, 96 op/s 2024-09-10T19:10:17.820 INFO:journalctl@ceph.iscsi.iscsi.a.smithi191.stdout:Sep 10 19:10:17 smithi191 podman[61667]: 2024-09-10 19:10:17.804631767 +0000 UTC m=+0.354983089 container start a859d96610538fc85ac9d2cdf1ca966867d8747ed5198589630e4e8e98200703 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3bfeaf4417585c821cab9ca76e4d95f390b0e769807122a45463640820b5cd16, name=ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-iscsi-iscsi-a-tcmu, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, ceph=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , RELEASE=wip-guits-quincy-2024-09-09-1810-quincy-5be5739) 2024-09-10T19:10:17.820 INFO:journalctl@ceph.iscsi.iscsi.a.smithi191.stdout:Sep 10 19:10:17 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-iscsi-iscsi-a-tcmu[61719]: log file path now is '/var/log/tcmu-runner.log' 2024-09-10T19:10:18.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:10:17 smithi183 ceph-mon[29736]: pgmap v190: 132 pgs: 9 unknown, 123 active+clean; 580 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 38 KiB/s rd, 5.1 KiB/s wr, 96 op/s 2024-09-10T19:10:18.181 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:10:17 smithi183 ceph-mon[41562]: pgmap v190: 132 pgs: 9 unknown, 123 active+clean; 580 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 38 KiB/s rd, 5.1 KiB/s wr, 96 op/s 2024-09-10T19:10:18.340 INFO:journalctl@ceph.iscsi.iscsi.a.smithi191.stdout:Sep 10 19:10:18 smithi191 podman[61783]: 2024-09-10 19:10:18.067339878 +0000 UTC m=+0.017678296 image pull 4c55dd4096115e48502f368cff1eab06769f71f030e1692e2611bbc6c6f637e9 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3bfeaf4417585c821cab9ca76e4d95f390b0e769807122a45463640820b5cd16 2024-09-10T19:10:18.340 INFO:journalctl@ceph.iscsi.iscsi.a.smithi191.stdout:Sep 10 19:10:18 smithi191 podman[61783]: 2024-09-10 19:10:18.204542666 +0000 UTC m=+0.154881083 container create 63021e878f8c78349c9334cc6961d041522f70a598f9d5b52df2cc5408c49d05 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3bfeaf4417585c821cab9ca76e4d95f390b0e769807122a45463640820b5cd16, name=ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-iscsi-iscsi-a, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, ceph=True, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.schema-version=1.0, RELEASE=wip-guits-quincy-2024-09-09-1810-quincy-5be5739) 2024-09-10T19:10:18.652 INFO:journalctl@ceph.iscsi.iscsi.a.smithi191.stdout:Sep 10 19:10:18 smithi191 podman[61783]: 2024-09-10 19:10:18.475271431 +0000 UTC m=+0.425609865 container init 63021e878f8c78349c9334cc6961d041522f70a598f9d5b52df2cc5408c49d05 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3bfeaf4417585c821cab9ca76e4d95f390b0e769807122a45463640820b5cd16, name=ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-iscsi-iscsi-a, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, RELEASE=wip-guits-quincy-2024-09-09-1810-quincy-5be5739, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_BRANCH=HEAD, ceph=True, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0) 2024-09-10T19:10:18.653 INFO:journalctl@ceph.iscsi.iscsi.a.smithi191.stdout:Sep 10 19:10:18 smithi191 podman[61783]: 2024-09-10 19:10:18.48196066 +0000 UTC m=+0.432299080 container start 63021e878f8c78349c9334cc6961d041522f70a598f9d5b52df2cc5408c49d05 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3bfeaf4417585c821cab9ca76e4d95f390b0e769807122a45463640820b5cd16, name=ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-iscsi-iscsi-a, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-guits-quincy-2024-09-09-1810-quincy-5be5739, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, org.label-schema.license=GPLv2) 2024-09-10T19:10:18.734 INFO:teuthology.orchestra.run.smithi191.stdout:Scheduled prometheus update... 2024-09-10T19:10:18.965 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:10:18 smithi191 ceph-mon[32884]: purged_snaps scrub starts 2024-09-10T19:10:18.965 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:10:18 smithi191 ceph-mon[32884]: purged_snaps scrub ok 2024-09-10T19:10:18.965 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:10:18 smithi191 ceph-mon[32884]: osdmap e67: 8 total, 8 up, 8 in 2024-09-10T19:10:18.966 INFO:journalctl@ceph.iscsi.iscsi.a.smithi191.stdout:Sep 10 19:10:18 smithi191 bash[61783]: 63021e878f8c78349c9334cc6961d041522f70a598f9d5b52df2cc5408c49d05 2024-09-10T19:10:18.966 INFO:journalctl@ceph.iscsi.iscsi.a.smithi191.stdout:Sep 10 19:10:18 smithi191 systemd[1]: Started Ceph iscsi.iscsi.a for 320f6df8-6fa7-11ef-bcea-c7b262605968. 2024-09-10T19:10:18.966 INFO:journalctl@ceph.iscsi.iscsi.a.smithi191.stdout:Sep 10 19:10:18 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-iscsi-iscsi-a[61816]: debug Started the configuration object watcher 2024-09-10T19:10:18.966 INFO:journalctl@ceph.iscsi.iscsi.a.smithi191.stdout:Sep 10 19:10:18 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-iscsi-iscsi-a[61816]: debug Checking for config object changes every 1s 2024-09-10T19:10:18.966 INFO:journalctl@ceph.iscsi.iscsi.a.smithi191.stdout:Sep 10 19:10:18 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-iscsi-iscsi-a[61816]: debug Processing osd blocklist entries for this node 2024-09-10T19:10:18.966 INFO:journalctl@ceph.iscsi.iscsi.a.smithi191.stdout:Sep 10 19:10:18 smithi191 bash[61667]: a859d96610538fc85ac9d2cdf1ca966867d8747ed5198589630e4e8e98200703 2024-09-10T19:10:19.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:10:18 smithi183 ceph-mon[29736]: purged_snaps scrub starts 2024-09-10T19:10:19.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:10:18 smithi183 ceph-mon[29736]: purged_snaps scrub ok 2024-09-10T19:10:19.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:10:18 smithi183 ceph-mon[29736]: osdmap e67: 8 total, 8 up, 8 in 2024-09-10T19:10:19.181 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:10:18 smithi183 ceph-mon[41562]: purged_snaps scrub starts 2024-09-10T19:10:19.181 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:10:18 smithi183 ceph-mon[41562]: purged_snaps scrub ok 2024-09-10T19:10:19.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:10:18 smithi183 ceph-mon[41562]: osdmap e67: 8 total, 8 up, 8 in 2024-09-10T19:10:19.365 INFO:journalctl@ceph.iscsi.iscsi.a.smithi191.stdout:Sep 10 19:10:19 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-iscsi-iscsi-a[61816]: debug Reading the configuration object to update local LIO configuration 2024-09-10T19:10:19.365 INFO:journalctl@ceph.iscsi.iscsi.a.smithi191.stdout:Sep 10 19:10:19 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-iscsi-iscsi-a[61816]: debug Configuration does not have an entry for this host(smithi191.front.sepia.ceph.com) - nothing to define to LIO 2024-09-10T19:10:19.365 INFO:journalctl@ceph.iscsi.iscsi.a.smithi191.stdout:Sep 10 19:10:19 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-iscsi-iscsi-a[61816]: * Serving Flask app 'rbd-target-api' (lazy loading) 2024-09-10T19:10:19.366 INFO:journalctl@ceph.iscsi.iscsi.a.smithi191.stdout:Sep 10 19:10:19 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-iscsi-iscsi-a[61816]: * Environment: production 2024-09-10T19:10:19.366 INFO:journalctl@ceph.iscsi.iscsi.a.smithi191.stdout:Sep 10 19:10:19 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-iscsi-iscsi-a[61816]: WARNING: This is a development server. Do not use it in a production deployment. 2024-09-10T19:10:19.366 INFO:journalctl@ceph.iscsi.iscsi.a.smithi191.stdout:Sep 10 19:10:19 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-iscsi-iscsi-a[61816]: Use a production WSGI server instead. 2024-09-10T19:10:19.366 INFO:journalctl@ceph.iscsi.iscsi.a.smithi191.stdout:Sep 10 19:10:19 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-iscsi-iscsi-a[61816]: * Debug mode: off 2024-09-10T19:10:19.366 INFO:journalctl@ceph.iscsi.iscsi.a.smithi191.stdout:Sep 10 19:10:19 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-iscsi-iscsi-a[61816]: debug * Running on all addresses. 2024-09-10T19:10:19.366 INFO:journalctl@ceph.iscsi.iscsi.a.smithi191.stdout:Sep 10 19:10:19 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-iscsi-iscsi-a[61816]: WARNING: This is a development server. Do not use it in a production deployment. 2024-09-10T19:10:19.367 INFO:journalctl@ceph.iscsi.iscsi.a.smithi191.stdout:Sep 10 19:10:19 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-iscsi-iscsi-a[61816]: * Running on all addresses. 2024-09-10T19:10:19.367 INFO:journalctl@ceph.iscsi.iscsi.a.smithi191.stdout:Sep 10 19:10:19 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-iscsi-iscsi-a[61816]: WARNING: This is a development server. Do not use it in a production deployment. 2024-09-10T19:10:19.367 INFO:journalctl@ceph.iscsi.iscsi.a.smithi191.stdout:Sep 10 19:10:19 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-iscsi-iscsi-a[61816]: debug * Running on http://[fe80::ec4:7aff:fe88:7211]:5000/ (Press CTRL+C to quit) 2024-09-10T19:10:19.367 INFO:journalctl@ceph.iscsi.iscsi.a.smithi191.stdout:Sep 10 19:10:19 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-iscsi-iscsi-a[61816]: * Running on http://[fe80::ec4:7aff:fe88:7211]:5000/ (Press CTRL+C to quit) 2024-09-10T19:10:19.684 DEBUG:teuthology.orchestra.run.smithi191:prometheus.a> sudo journalctl -f -n 0 -u ceph-320f6df8-6fa7-11ef-bcea-c7b262605968@prometheus.a.service 2024-09-10T19:10:19.687 INFO:tasks.cephadm:Adding node-exporter.a on smithi183 2024-09-10T19:10:19.687 INFO:tasks.cephadm:Adding node-exporter.b on smithi191 2024-09-10T19:10:19.687 DEBUG:teuthology.orchestra.run.smithi191:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph orch apply node-exporter '2;smithi183=a;smithi191=b' 2024-09-10T19:10:19.965 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:10:19 smithi191 ceph-mon[32884]: pgmap v192: 132 pgs: 132 active+clean; 581 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 49 KiB/s rd, 5.5 KiB/s wr, 123 op/s 2024-09-10T19:10:19.966 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:10:19 smithi191 ceph-mon[32884]: from='client.24388 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi191=a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-10T19:10:19.966 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:10:19 smithi191 ceph-mon[32884]: Saving service prometheus spec with placement smithi191=a;count:1 2024-09-10T19:10:19.966 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:10:19 smithi191 ceph-mon[32884]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:10:19.966 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:10:19 smithi191 ceph-mon[32884]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:10:19.966 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:10:19 smithi191 ceph-mon[32884]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:10:19.966 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:10:19 smithi191 ceph-mon[32884]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:10:19.966 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:10:19 smithi191 ceph-mon[32884]: Checking pool "datapool" exists for service iscsi.datapool 2024-09-10T19:10:19.966 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:10:19 smithi191 ceph-mon[32884]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:10:19.966 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:10:19 smithi191 ceph-mon[32884]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T19:10:19.966 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:10:19 smithi191 ceph-mon[32884]: from='client.? 172.21.15.191:0/735712572' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-09-10T19:10:19.966 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:10:19 smithi191 ceph-mon[32884]: osdmap e68: 8 total, 8 up, 8 in 2024-09-10T19:10:20.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:10:19 smithi183 ceph-mon[29736]: pgmap v192: 132 pgs: 132 active+clean; 581 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 49 KiB/s rd, 5.5 KiB/s wr, 123 op/s 2024-09-10T19:10:20.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:10:19 smithi183 ceph-mon[29736]: from='client.24388 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi191=a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-10T19:10:20.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:10:19 smithi183 ceph-mon[29736]: Saving service prometheus spec with placement smithi191=a;count:1 2024-09-10T19:10:20.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:10:19 smithi183 ceph-mon[29736]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:10:20.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:10:19 smithi183 ceph-mon[29736]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:10:20.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:10:19 smithi183 ceph-mon[29736]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:10:20.182 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:10:19 smithi183 ceph-mon[29736]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:10:20.182 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:10:19 smithi183 ceph-mon[29736]: Checking pool "datapool" exists for service iscsi.datapool 2024-09-10T19:10:20.182 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:10:19 smithi183 ceph-mon[29736]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:10:20.182 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:10:19 smithi183 ceph-mon[29736]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T19:10:20.182 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:10:19 smithi183 ceph-mon[29736]: from='client.? 172.21.15.191:0/735712572' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-09-10T19:10:20.182 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:10:19 smithi183 ceph-mon[29736]: osdmap e68: 8 total, 8 up, 8 in 2024-09-10T19:10:20.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:10:19 smithi183 ceph-mon[41562]: pgmap v192: 132 pgs: 132 active+clean; 581 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 49 KiB/s rd, 5.5 KiB/s wr, 123 op/s 2024-09-10T19:10:20.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:10:19 smithi183 ceph-mon[41562]: from='client.24388 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi191=a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-10T19:10:20.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:10:19 smithi183 ceph-mon[41562]: Saving service prometheus spec with placement smithi191=a;count:1 2024-09-10T19:10:20.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:10:19 smithi183 ceph-mon[41562]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:10:20.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:10:19 smithi183 ceph-mon[41562]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:10:20.183 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:10:19 smithi183 ceph-mon[41562]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:10:20.183 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:10:19 smithi183 ceph-mon[41562]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:10:20.183 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:10:19 smithi183 ceph-mon[41562]: Checking pool "datapool" exists for service iscsi.datapool 2024-09-10T19:10:20.183 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:10:19 smithi183 ceph-mon[41562]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:10:20.183 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:10:19 smithi183 ceph-mon[41562]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T19:10:20.183 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:10:19 smithi183 ceph-mon[41562]: from='client.? 172.21.15.191:0/735712572' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-09-10T19:10:20.183 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:10:19 smithi183 ceph-mon[41562]: osdmap e68: 8 total, 8 up, 8 in 2024-09-10T19:10:20.505 INFO:teuthology.orchestra.run.smithi191.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.b/config 2024-09-10T19:10:22.086 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:10:21 smithi191 ceph-mon[32884]: pgmap v194: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 77 KiB/s rd, 6.3 KiB/s wr, 188 op/s 2024-09-10T19:10:22.086 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:10:21 smithi191 ceph-mon[32884]: mgrmap e16: y(active, since 4m), standbys: x 2024-09-10T19:10:22.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:10:21 smithi183 ceph-mon[29736]: pgmap v194: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 77 KiB/s rd, 6.3 KiB/s wr, 188 op/s 2024-09-10T19:10:22.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:10:21 smithi183 ceph-mon[29736]: mgrmap e16: y(active, since 4m), standbys: x 2024-09-10T19:10:22.181 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:10:21 smithi183 ceph-mon[41562]: pgmap v194: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 77 KiB/s rd, 6.3 KiB/s wr, 188 op/s 2024-09-10T19:10:22.181 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:10:21 smithi183 ceph-mon[41562]: mgrmap e16: y(active, since 4m), standbys: x 2024-09-10T19:10:22.247 INFO:teuthology.orchestra.run.smithi191.stdout:Scheduled node-exporter update... 2024-09-10T19:10:22.949 DEBUG:teuthology.orchestra.run.smithi183:node-exporter.a> sudo journalctl -f -n 0 -u ceph-320f6df8-6fa7-11ef-bcea-c7b262605968@node-exporter.a.service 2024-09-10T19:10:22.952 DEBUG:teuthology.orchestra.run.smithi191:node-exporter.b> sudo journalctl -f -n 0 -u ceph-320f6df8-6fa7-11ef-bcea-c7b262605968@node-exporter.b.service 2024-09-10T19:10:22.954 INFO:tasks.cephadm:Adding alertmanager.a on smithi183 2024-09-10T19:10:22.954 DEBUG:teuthology.orchestra.run.smithi191:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph orch apply alertmanager '1;smithi183=a' 2024-09-10T19:10:23.571 INFO:teuthology.orchestra.run.smithi191.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.b/config 2024-09-10T19:10:23.681 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:10:23 smithi183 ceph-mon[29736]: pgmap v195: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 62 KiB/s rd, 4.9 KiB/s wr, 151 op/s 2024-09-10T19:10:23.681 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:10:23 smithi183 ceph-mon[29736]: from='client.24421 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "placement": "2;smithi183=a;smithi191=b", "target": ["mon-mgr", ""]}]: dispatch 2024-09-10T19:10:23.681 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:10:23 smithi183 ceph-mon[29736]: Saving service node-exporter spec with placement smithi183=a;smithi191=b;count:2 2024-09-10T19:10:23.681 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:10:23 smithi183 ceph-mon[29736]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:10:23.681 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:10:23 smithi183 ceph-mon[29736]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:10:23.681 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:10:23 smithi183 ceph-mon[41562]: pgmap v195: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 62 KiB/s rd, 4.9 KiB/s wr, 151 op/s 2024-09-10T19:10:23.682 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:10:23 smithi183 ceph-mon[41562]: from='client.24421 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "placement": "2;smithi183=a;smithi191=b", "target": ["mon-mgr", ""]}]: dispatch 2024-09-10T19:10:23.682 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:10:23 smithi183 ceph-mon[41562]: Saving service node-exporter spec with placement smithi183=a;smithi191=b;count:2 2024-09-10T19:10:23.682 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:10:23 smithi183 ceph-mon[41562]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:10:23.682 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:10:23 smithi183 ceph-mon[41562]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:10:23.692 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:10:23 smithi191 ceph-mon[32884]: pgmap v195: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 62 KiB/s rd, 4.9 KiB/s wr, 151 op/s 2024-09-10T19:10:23.692 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:10:23 smithi191 ceph-mon[32884]: from='client.24421 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "placement": "2;smithi183=a;smithi191=b", "target": ["mon-mgr", ""]}]: dispatch 2024-09-10T19:10:23.692 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:10:23 smithi191 ceph-mon[32884]: Saving service node-exporter spec with placement smithi183=a;smithi191=b;count:2 2024-09-10T19:10:23.692 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:10:23 smithi191 ceph-mon[32884]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:10:23.692 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:10:23 smithi191 ceph-mon[32884]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:10:24.608 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:10:24 smithi191 ceph-mon[32884]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:10:24.608 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:10:24 smithi191 ceph-mon[32884]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:10:24.609 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:10:24 smithi191 ceph-mon[32884]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T19:10:24.609 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:10:24 smithi191 ceph-mon[32884]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T19:10:24.609 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:10:24 smithi191 ceph-mon[32884]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:10:24.609 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:10:24 smithi191 ceph-mon[32884]: Deploying daemon prometheus.a on smithi191 2024-09-10T19:10:24.931 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:10:24 smithi183 ceph-mon[29736]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:10:24.931 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:10:24 smithi183 ceph-mon[29736]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:10:24.931 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:10:24 smithi183 ceph-mon[29736]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T19:10:24.931 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:10:24 smithi183 ceph-mon[29736]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T19:10:24.932 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:10:24 smithi183 ceph-mon[29736]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:10:24.932 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:10:24 smithi183 ceph-mon[29736]: Deploying daemon prometheus.a on smithi191 2024-09-10T19:10:24.932 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:10:24 smithi183 ceph-mon[41562]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:10:24.932 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:10:24 smithi183 ceph-mon[41562]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:10:24.932 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:10:24 smithi183 ceph-mon[41562]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T19:10:24.932 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:10:24 smithi183 ceph-mon[41562]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T19:10:24.932 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:10:24 smithi183 ceph-mon[41562]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:10:24.932 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:10:24 smithi183 ceph-mon[41562]: Deploying daemon prometheus.a on smithi191 2024-09-10T19:10:25.422 INFO:teuthology.orchestra.run.smithi191.stdout:Scheduled alertmanager update... 2024-09-10T19:10:25.931 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:10:25 smithi183 ceph-mon[29736]: pgmap v196: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 37 KiB/s rd, 2.0 KiB/s wr, 87 op/s 2024-09-10T19:10:25.931 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:10:25 smithi183 ceph-mon[29736]: osdmap e69: 8 total, 8 up, 8 in 2024-09-10T19:10:25.931 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:10:25 smithi183 ceph-mon[29736]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:10:25.932 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:10:25 smithi183 ceph-mon[41562]: pgmap v196: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 37 KiB/s rd, 2.0 KiB/s wr, 87 op/s 2024-09-10T19:10:25.932 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:10:25 smithi183 ceph-mon[41562]: osdmap e69: 8 total, 8 up, 8 in 2024-09-10T19:10:25.932 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:10:25 smithi183 ceph-mon[41562]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:10:25.965 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:10:25 smithi191 ceph-mon[32884]: pgmap v196: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 37 KiB/s rd, 2.0 KiB/s wr, 87 op/s 2024-09-10T19:10:25.965 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:10:25 smithi191 ceph-mon[32884]: osdmap e69: 8 total, 8 up, 8 in 2024-09-10T19:10:25.965 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:10:25 smithi191 ceph-mon[32884]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:10:26.181 DEBUG:teuthology.orchestra.run.smithi183:alertmanager.a> sudo journalctl -f -n 0 -u ceph-320f6df8-6fa7-11ef-bcea-c7b262605968@alertmanager.a.service 2024-09-10T19:10:26.184 INFO:tasks.cephadm:Adding grafana.a on smithi191 2024-09-10T19:10:26.184 DEBUG:teuthology.orchestra.run.smithi191:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph orch apply grafana '1;smithi191=a' 2024-09-10T19:10:26.465 INFO:teuthology.orchestra.run.smithi191.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.b/config 2024-09-10T19:10:26.910 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:10:26 smithi191 ceph-mon[32884]: from='client.24427 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi183=a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-10T19:10:26.910 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:10:26 smithi191 ceph-mon[32884]: Saving service alertmanager spec with placement smithi183=a;count:1 2024-09-10T19:10:26.931 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:10:26 smithi183 ceph-mon[41562]: from='client.24427 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi183=a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-10T19:10:26.931 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:10:26 smithi183 ceph-mon[41562]: Saving service alertmanager spec with placement smithi183=a;count:1 2024-09-10T19:10:26.931 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:10:26 smithi183 ceph-mon[29736]: from='client.24427 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi183=a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-10T19:10:26.931 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:10:26 smithi183 ceph-mon[29736]: Saving service alertmanager spec with placement smithi183=a;count:1 2024-09-10T19:10:27.931 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:10:27 smithi183 ceph-mon[29736]: pgmap v198: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 25 KiB/s rd, 1.1 KiB/s wr, 58 op/s 2024-09-10T19:10:27.931 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:10:27 smithi183 ceph-mon[41562]: pgmap v198: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 25 KiB/s rd, 1.1 KiB/s wr, 58 op/s 2024-09-10T19:10:27.965 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:10:27 smithi191 ceph-mon[32884]: pgmap v198: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 25 KiB/s rd, 1.1 KiB/s wr, 58 op/s 2024-09-10T19:10:28.966 INFO:journalctl@ceph.iscsi.iscsi.a.smithi191.stdout:Sep 10 19:10:28 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-iscsi-iscsi-a[61816]: debug there is no tcmu-runner data available 2024-09-10T19:10:29.014 INFO:teuthology.orchestra.run.smithi191.stdout:Scheduled grafana update... 2024-09-10T19:10:29.931 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:10:29 smithi183 ceph-mon[29736]: pgmap v199: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 13 KiB/s rd, 358 B/s wr, 29 op/s 2024-09-10T19:10:29.931 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:10:29 smithi183 ceph-mon[29736]: from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:10:29.931 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:10:29 smithi183 ceph-mon[29736]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:10:29.931 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:10:29 smithi183 ceph-mon[29736]: osdmap e70: 8 total, 8 up, 8 in 2024-09-10T19:10:29.932 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:10:29 smithi183 ceph-mon[41562]: pgmap v199: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 13 KiB/s rd, 358 B/s wr, 29 op/s 2024-09-10T19:10:29.932 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:10:29 smithi183 ceph-mon[41562]: from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:10:29.932 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:10:29 smithi183 ceph-mon[41562]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:10:29.932 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:10:29 smithi183 ceph-mon[41562]: osdmap e70: 8 total, 8 up, 8 in 2024-09-10T19:10:29.965 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:10:29 smithi191 ceph-mon[32884]: pgmap v199: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 13 KiB/s rd, 358 B/s wr, 29 op/s 2024-09-10T19:10:29.965 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:10:29 smithi191 ceph-mon[32884]: from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:10:29.965 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:10:29 smithi191 ceph-mon[32884]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:10:29.965 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:10:29 smithi191 ceph-mon[32884]: osdmap e70: 8 total, 8 up, 8 in 2024-09-10T19:10:30.834 DEBUG:teuthology.orchestra.run.smithi191:grafana.a> sudo journalctl -f -n 0 -u ceph-320f6df8-6fa7-11ef-bcea-c7b262605968@grafana.a.service 2024-09-10T19:10:30.837 INFO:tasks.cephadm:Setting up client nodes... 2024-09-10T19:10:30.837 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph auth get-or-create client.0 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-09-10T19:10:30.931 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:10:30 smithi183 ceph-mon[29736]: from='client.24433 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi191=a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-10T19:10:30.931 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:10:30 smithi183 ceph-mon[29736]: Saving service grafana spec with placement smithi191=a;count:1 2024-09-10T19:10:30.931 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:10:30 smithi183 ceph-mon[41562]: from='client.24433 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi191=a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-10T19:10:30.931 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:10:30 smithi183 ceph-mon[41562]: Saving service grafana spec with placement smithi191=a;count:1 2024-09-10T19:10:30.965 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:10:30 smithi191 ceph-mon[32884]: from='client.24433 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi191=a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-10T19:10:30.965 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:10:30 smithi191 ceph-mon[32884]: Saving service grafana spec with placement smithi191=a;count:1 2024-09-10T19:10:31.090 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:10:31.921 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:10:31 smithi191 ceph-mon[32884]: pgmap v201: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-10T19:10:31.931 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:10:31 smithi183 ceph-mon[29736]: pgmap v201: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-10T19:10:31.932 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:10:31 smithi183 ceph-mon[41562]: pgmap v201: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-10T19:10:32.924 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:10:32 smithi191 ceph-mon[32884]: purged_snaps scrub starts 2024-09-10T19:10:32.924 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:10:32 smithi191 ceph-mon[32884]: purged_snaps scrub ok 2024-09-10T19:10:32.924 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:10:32 smithi191 ceph-mon[32884]: osdmap e71: 8 total, 8 up, 8 in 2024-09-10T19:10:32.931 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:10:32 smithi183 ceph-mon[29736]: purged_snaps scrub starts 2024-09-10T19:10:32.931 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:10:32 smithi183 ceph-mon[29736]: purged_snaps scrub ok 2024-09-10T19:10:32.931 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:10:32 smithi183 ceph-mon[29736]: osdmap e71: 8 total, 8 up, 8 in 2024-09-10T19:10:32.931 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:10:32 smithi183 ceph-mon[41562]: purged_snaps scrub starts 2024-09-10T19:10:32.931 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:10:32 smithi183 ceph-mon[41562]: purged_snaps scrub ok 2024-09-10T19:10:32.931 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:10:32 smithi183 ceph-mon[41562]: osdmap e71: 8 total, 8 up, 8 in 2024-09-10T19:10:33.152 INFO:teuthology.orchestra.run.smithi183.stdout:[client.0] 2024-09-10T19:10:33.152 INFO:teuthology.orchestra.run.smithi183.stdout: key = AQCpmeBmcwDECBAA4Z1QyY7SjDNdsVemuwmreg== 2024-09-10T19:10:33.746 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:10:33 smithi183 ceph-mon[29736]: pgmap v203: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-09-10T19:10:33.746 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:10:33 smithi183 ceph-mon[29736]: from='client.? 172.21.15.183:0/2212326962' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-09-10T19:10:33.746 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:10:33 smithi183 ceph-mon[29736]: from='client.? 172.21.15.183:0/2212326962' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-09-10T19:10:33.746 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:10:33 smithi183 ceph-mon[41562]: pgmap v203: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-09-10T19:10:33.747 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:10:33 smithi183 ceph-mon[41562]: from='client.? 172.21.15.183:0/2212326962' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-09-10T19:10:33.747 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:10:33 smithi183 ceph-mon[41562]: from='client.? 172.21.15.183:0/2212326962' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-09-10T19:10:33.747 DEBUG:teuthology.orchestra.run.smithi183:> set -ex 2024-09-10T19:10:33.748 DEBUG:teuthology.orchestra.run.smithi183:> sudo dd of=/etc/ceph/ceph.client.0.keyring 2024-09-10T19:10:33.748 DEBUG:teuthology.orchestra.run.smithi183:> sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-09-10T19:10:33.785 DEBUG:teuthology.orchestra.run.smithi191:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph auth get-or-create client.1 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-09-10T19:10:33.915 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:10:33 smithi191 ceph-mon[32884]: pgmap v203: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-09-10T19:10:33.915 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:10:33 smithi191 ceph-mon[32884]: from='client.? 172.21.15.183:0/2212326962' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-09-10T19:10:33.915 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:10:33 smithi191 ceph-mon[32884]: from='client.? 172.21.15.183:0/2212326962' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-09-10T19:10:34.469 INFO:teuthology.orchestra.run.smithi191.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.b/config 2024-09-10T19:10:35.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:10:35 smithi183 ceph-mon[29736]: purged_snaps scrub starts 2024-09-10T19:10:35.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:10:35 smithi183 ceph-mon[29736]: purged_snaps scrub ok 2024-09-10T19:10:35.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:10:35 smithi183 ceph-mon[29736]: osdmap e72: 8 total, 8 up, 8 in 2024-09-10T19:10:35.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:10:35 smithi183 ceph-mon[29736]: pgmap v205: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-10T19:10:35.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:10:35 smithi183 ceph-mon[41562]: purged_snaps scrub starts 2024-09-10T19:10:35.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:10:35 smithi183 ceph-mon[41562]: purged_snaps scrub ok 2024-09-10T19:10:35.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:10:35 smithi183 ceph-mon[41562]: osdmap e72: 8 total, 8 up, 8 in 2024-09-10T19:10:35.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:10:35 smithi183 ceph-mon[41562]: pgmap v205: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-10T19:10:35.465 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:10:35 smithi191 ceph-mon[32884]: purged_snaps scrub starts 2024-09-10T19:10:35.465 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:10:35 smithi191 ceph-mon[32884]: purged_snaps scrub ok 2024-09-10T19:10:35.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:10:35 smithi191 ceph-mon[32884]: osdmap e72: 8 total, 8 up, 8 in 2024-09-10T19:10:35.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:10:35 smithi191 ceph-mon[32884]: pgmap v205: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-10T19:10:36.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:10:36 smithi183 ceph-mon[29736]: 1.0 scrub starts 2024-09-10T19:10:36.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:10:36 smithi183 ceph-mon[29736]: 1.0 scrub ok 2024-09-10T19:10:36.431 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:10:36 smithi183 ceph-mon[41562]: 1.0 scrub starts 2024-09-10T19:10:36.431 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:10:36 smithi183 ceph-mon[41562]: 1.0 scrub ok 2024-09-10T19:10:36.465 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:10:36 smithi191 ceph-mon[32884]: 1.0 scrub starts 2024-09-10T19:10:36.465 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:10:36 smithi191 ceph-mon[32884]: 1.0 scrub ok 2024-09-10T19:10:37.334 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:10:37 smithi191 ceph-mon[32884]: pgmap v206: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-09-10T19:10:37.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:10:37 smithi183 ceph-mon[29736]: pgmap v206: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-09-10T19:10:37.431 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:10:37 smithi183 ceph-mon[41562]: pgmap v206: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-09-10T19:10:38.232 INFO:teuthology.orchestra.run.smithi191.stdout:[client.1] 2024-09-10T19:10:38.232 INFO:teuthology.orchestra.run.smithi191.stdout: key = AQCumeBm6bBWDRAAkrmp9T7z5P9wh3E8uPfciQ== 2024-09-10T19:10:38.908 INFO:journalctl@ceph.iscsi.iscsi.a.smithi191.stdout:Sep 10 19:10:38 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-iscsi-iscsi-a[61816]: debug there is no tcmu-runner data available 2024-09-10T19:10:38.933 DEBUG:teuthology.orchestra.run.smithi191:> set -ex 2024-09-10T19:10:38.933 DEBUG:teuthology.orchestra.run.smithi191:> sudo dd of=/etc/ceph/ceph.client.1.keyring 2024-09-10T19:10:38.933 DEBUG:teuthology.orchestra.run.smithi191:> sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-09-10T19:10:38.971 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-09-10T19:10:38.971 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2024-09-10T19:10:38.971 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph mgr dump --format=json 2024-09-10T19:10:39.215 INFO:journalctl@ceph.mgr.x.smithi191.stdout:Sep 10 19:10:39 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-x[34967]: ignoring --setuser ceph since I am not root 2024-09-10T19:10:39.216 INFO:journalctl@ceph.mgr.x.smithi191.stdout:Sep 10 19:10:39 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-x[34967]: ignoring --setgroup ceph since I am not root 2024-09-10T19:10:39.216 INFO:journalctl@ceph.mgr.x.smithi191.stdout:Sep 10 19:10:39 smithi191 ceph-mgr[34990]: -- 172.21.15.191:0/2815404526 <== mon.1 v2:172.21.15.191:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x558491bec340 con 0x558491bf9000 2024-09-10T19:10:39.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:10:39 smithi191 ceph-mon[32884]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:10:39.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:10:39 smithi191 ceph-mon[32884]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:10:39.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:10:39 smithi191 ceph-mon[32884]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:10:39.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:10:39 smithi191 ceph-mon[32884]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-09-10T19:10:39.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:10:39 smithi191 ceph-mon[32884]: pgmap v207: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:10:39.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:10:39 smithi191 ceph-mon[32884]: from='client.? 172.21.15.191:0/3407231305' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-09-10T19:10:39.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:10:39 smithi191 ceph-mon[32884]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-09-10T19:10:39.217 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:10:39 smithi191 ceph-mon[32884]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-09-10T19:10:39.217 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:10:39 smithi191 ceph-mon[32884]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:10:39.217 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:10:39 smithi191 ceph-mon[32884]: from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:10:39.249 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:10:39 smithi183 ceph-mon[41562]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:10:39.249 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:10:39 smithi183 ceph-mon[41562]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:10:39.249 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:10:39 smithi183 ceph-mon[41562]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:10:39.250 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:10:39 smithi183 ceph-mon[41562]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-09-10T19:10:39.250 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:10:39 smithi183 ceph-mon[41562]: pgmap v207: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:10:39.250 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:10:39 smithi183 ceph-mon[41562]: from='client.? 172.21.15.191:0/3407231305' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-09-10T19:10:39.250 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:10:39 smithi183 ceph-mon[41562]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-09-10T19:10:39.250 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:10:39 smithi183 ceph-mon[41562]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-09-10T19:10:39.250 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:10:39 smithi183 ceph-mon[41562]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:10:39.250 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:10:39 smithi183 ceph-mon[41562]: from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:10:39.250 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:10:39 smithi183 ceph-mon[29736]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:10:39.250 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:10:39 smithi183 ceph-mon[29736]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:10:39.250 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:10:39 smithi183 ceph-mon[29736]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:10:39.251 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:10:39 smithi183 ceph-mon[29736]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-09-10T19:10:39.251 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:10:39 smithi183 ceph-mon[29736]: pgmap v207: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:10:39.251 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:10:39 smithi183 ceph-mon[29736]: from='client.? 172.21.15.191:0/3407231305' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-09-10T19:10:39.251 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:10:39 smithi183 ceph-mon[29736]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-09-10T19:10:39.251 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:10:39 smithi183 ceph-mon[29736]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-09-10T19:10:39.251 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:10:39 smithi183 ceph-mon[29736]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' 2024-09-10T19:10:39.251 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:10:39 smithi183 ceph-mon[29736]: from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:10:39.251 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Sep 10 19:10:39 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-y[29989]: ignoring --setuser ceph since I am not root 2024-09-10T19:10:39.251 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Sep 10 19:10:39 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-y[29989]: ignoring --setgroup ceph since I am not root 2024-09-10T19:10:39.251 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Sep 10 19:10:39 smithi183 ceph-mgr[30013]: -- 172.21.15.183:0/28262827 <== mon.0 v2:172.21.15.183:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55ffdf0a2340 con 0x55ffdf0ae800 2024-09-10T19:10:39.251 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Sep 10 19:10:39 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-y[29989]: 2024-09-10T19:10:39.247+0000 7f9feab4cec0 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-09-10T19:10:39.260 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:10:39.501 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Sep 10 19:10:39 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-y[29989]: 2024-09-10T19:10:39.307+0000 7f9feab4cec0 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-09-10T19:10:39.501 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Sep 10 19:10:39 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-y[29989]: 2024-09-10T19:10:39.372+0000 7f9feab4cec0 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-09-10T19:10:39.501 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Sep 10 19:10:39 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-y[29989]: 2024-09-10T19:10:39.499+0000 7f9feab4cec0 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-09-10T19:10:39.590 INFO:journalctl@ceph.mgr.x.smithi191.stdout:Sep 10 19:10:39 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-x[34967]: 2024-09-10T19:10:39.216+0000 7fc8cb4d9ec0 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-09-10T19:10:39.591 INFO:journalctl@ceph.mgr.x.smithi191.stdout:Sep 10 19:10:39 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-x[34967]: 2024-09-10T19:10:39.270+0000 7fc8cb4d9ec0 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-09-10T19:10:39.591 INFO:journalctl@ceph.mgr.x.smithi191.stdout:Sep 10 19:10:39 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-x[34967]: 2024-09-10T19:10:39.336+0000 7fc8cb4d9ec0 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-09-10T19:10:39.591 INFO:journalctl@ceph.mgr.x.smithi191.stdout:Sep 10 19:10:39 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-x[34967]: 2024-09-10T19:10:39.455+0000 7fc8cb4d9ec0 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-09-10T19:10:39.591 INFO:journalctl@ceph.mgr.x.smithi191.stdout:Sep 10 19:10:39 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-x[34967]: 2024-09-10T19:10:39.588+0000 7fc8cb4d9ec0 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-09-10T19:10:39.769 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Sep 10 19:10:39 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-y[29989]: 2024-09-10T19:10:39.642+0000 7f9feab4cec0 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-09-10T19:10:40.082 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Sep 10 19:10:39 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-y[29989]: 2024-09-10T19:10:39.905+0000 7f9feab4cec0 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-09-10T19:10:40.082 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Sep 10 19:10:39 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-y[29989]: 2024-09-10T19:10:39.965+0000 7f9feab4cec0 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-09-10T19:10:40.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:10:40 smithi191 ceph-mon[32884]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-09-10T19:10:40.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:10:40 smithi191 ceph-mon[32884]: mgrmap e17: y(active, since 4m), standbys: x 2024-09-10T19:10:40.216 INFO:journalctl@ceph.mgr.x.smithi191.stdout:Sep 10 19:10:39 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-x[34967]: 2024-09-10T19:10:39.844+0000 7fc8cb4d9ec0 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-09-10T19:10:40.216 INFO:journalctl@ceph.mgr.x.smithi191.stdout:Sep 10 19:10:39 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-x[34967]: 2024-09-10T19:10:39.903+0000 7fc8cb4d9ec0 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-09-10T19:10:40.216 INFO:journalctl@ceph.mgr.x.smithi191.stdout:Sep 10 19:10:40 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-x[34967]: 2024-09-10T19:10:40.061+0000 7fc8cb4d9ec0 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-09-10T19:10:40.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:10:40 smithi183 ceph-mon[29736]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-09-10T19:10:40.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:10:40 smithi183 ceph-mon[29736]: mgrmap e17: y(active, since 4m), standbys: x 2024-09-10T19:10:40.431 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:10:40 smithi183 ceph-mon[41562]: from='mgr.14150 172.21.15.183:0/3810900293' entity='mgr.y' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-09-10T19:10:40.431 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:10:40 smithi183 ceph-mon[41562]: mgrmap e17: y(active, since 4m), standbys: x 2024-09-10T19:10:40.432 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Sep 10 19:10:40 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-y[29989]: 2024-09-10T19:10:40.131+0000 7f9feab4cec0 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-09-10T19:10:41.181 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Sep 10 19:10:40 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-y[29989]: 2024-09-10T19:10:40.991+0000 7f9feab4cec0 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-09-10T19:10:41.181 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Sep 10 19:10:41 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-y[29989]: 2024-09-10T19:10:41.059+0000 7f9feab4cec0 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-09-10T19:10:41.215 INFO:journalctl@ceph.mgr.x.smithi191.stdout:Sep 10 19:10:40 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-x[34967]: 2024-09-10T19:10:40.893+0000 7fc8cb4d9ec0 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-09-10T19:10:41.216 INFO:journalctl@ceph.mgr.x.smithi191.stdout:Sep 10 19:10:40 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-x[34967]: 2024-09-10T19:10:40.956+0000 7fc8cb4d9ec0 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-09-10T19:10:41.256 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-09-10T19:10:41.655 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:10:41 smithi191 ceph-mon[32884]: from='client.? 172.21.15.183:0/572509389' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-09-10T19:10:41.681 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:10:41 smithi183 ceph-mon[29736]: from='client.? 172.21.15.183:0/572509389' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-09-10T19:10:41.681 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:10:41 smithi183 ceph-mon[41562]: from='client.? 172.21.15.183:0/572509389' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-09-10T19:10:41.944 INFO:journalctl@ceph.mgr.x.smithi191.stdout:Sep 10 19:10:41 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-x[34967]: 2024-09-10T19:10:41.653+0000 7fc8cb4d9ec0 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-09-10T19:10:41.944 INFO:journalctl@ceph.mgr.x.smithi191.stdout:Sep 10 19:10:41 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-x[34967]: 2024-09-10T19:10:41.765+0000 7fc8cb4d9ec0 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-09-10T19:10:41.944 INFO:journalctl@ceph.mgr.x.smithi191.stdout:Sep 10 19:10:41 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-x[34967]: 2024-09-10T19:10:41.884+0000 7fc8cb4d9ec0 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-09-10T19:10:42.022 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Sep 10 19:10:41 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-y[29989]: 2024-09-10T19:10:41.789+0000 7f9feab4cec0 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-09-10T19:10:42.022 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Sep 10 19:10:41 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-y[29989]: 2024-09-10T19:10:41.901+0000 7f9feab4cec0 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-09-10T19:10:42.022 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Sep 10 19:10:42 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-y[29989]: 2024-09-10T19:10:42.020+0000 7f9feab4cec0 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-09-10T19:10:42.207 INFO:teuthology.orchestra.run.smithi183.stdout:{"epoch":17,"flags":0,"active_gid":14150,"active_name":"y","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6800","nonce":3311783362},{"type":"v1","addr":"172.21.15.183:6801","nonce":3311783362}]},"active_addr":"172.21.15.183:6801/3311783362","active_change":"2024-09-10T19:05:48.153993+0000","active_mgr_features":4540138320759226367,"available":true,"standbys":[{"gid":24115,"name":"x","mgr_features":4540138320759226367,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.1.5","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","nfs","prometheus","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.1.5","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://172.21.15.183:8443/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":3,"active_clients":[{"name":"cephadm","addrvec":[{"type":"v2","addr":"172.21.15.183:0","nonce":1652576282}]},{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.183:0","nonce":3982386906}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.183:0","nonce":3636171401}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.183:0","nonce":4088553922}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.183:0","nonce":412044400}]}]} 2024-09-10T19:10:42.211 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2024-09-10T19:10:42.211 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-09-10T19:10:42.211 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph osd dump --format=json 2024-09-10T19:10:42.215 INFO:journalctl@ceph.mgr.x.smithi191.stdout:Sep 10 19:10:41 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-x[34967]: 2024-09-10T19:10:41.942+0000 7fc8cb4d9ec0 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-09-10T19:10:42.215 INFO:journalctl@ceph.mgr.x.smithi191.stdout:Sep 10 19:10:42 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-x[34967]: 2024-09-10T19:10:42.067+0000 7fc8cb4d9ec0 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-09-10T19:10:42.215 INFO:journalctl@ceph.mgr.x.smithi191.stdout:Sep 10 19:10:42 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-x[34967]: 2024-09-10T19:10:42.133+0000 7fc8cb4d9ec0 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-09-10T19:10:42.215 INFO:journalctl@ceph.mgr.x.smithi191.stdout:Sep 10 19:10:42 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-x[34967]: 2024-09-10T19:10:42.190+0000 7fc8cb4d9ec0 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-09-10T19:10:42.281 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Sep 10 19:10:42 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-y[29989]: 2024-09-10T19:10:42.079+0000 7f9feab4cec0 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-09-10T19:10:42.281 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Sep 10 19:10:42 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-y[29989]: 2024-09-10T19:10:42.208+0000 7f9feab4cec0 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-09-10T19:10:42.281 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Sep 10 19:10:42 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-y[29989]: 2024-09-10T19:10:42.279+0000 7f9feab4cec0 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-09-10T19:10:42.463 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:10:42.554 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Sep 10 19:10:42 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-y[29989]: 2024-09-10T19:10:42.340+0000 7f9feab4cec0 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-09-10T19:10:42.554 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Sep 10 19:10:42 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-y[29989]: 2024-09-10T19:10:42.488+0000 7f9feab4cec0 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-09-10T19:10:42.554 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Sep 10 19:10:42 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-y[29989]: 2024-09-10T19:10:42.552+0000 7f9feab4cec0 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-09-10T19:10:42.585 INFO:journalctl@ceph.mgr.x.smithi191.stdout:Sep 10 19:10:42 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-x[34967]: 2024-09-10T19:10:42.323+0000 7fc8cb4d9ec0 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-09-10T19:10:42.585 INFO:journalctl@ceph.mgr.x.smithi191.stdout:Sep 10 19:10:42 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-x[34967]: 2024-09-10T19:10:42.385+0000 7fc8cb4d9ec0 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-09-10T19:10:42.585 INFO:journalctl@ceph.mgr.x.smithi191.stdout:Sep 10 19:10:42 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-x[34967]: 2024-09-10T19:10:42.448+0000 7fc8cb4d9ec0 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-09-10T19:10:42.585 INFO:journalctl@ceph.mgr.x.smithi191.stdout:Sep 10 19:10:42 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-x[34967]: 2024-09-10T19:10:42.527+0000 7fc8cb4d9ec0 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-09-10T19:10:42.585 INFO:journalctl@ceph.mgr.x.smithi191.stdout:Sep 10 19:10:42 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-x[34967]: 2024-09-10T19:10:42.583+0000 7fc8cb4d9ec0 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-09-10T19:10:42.822 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Sep 10 19:10:42 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-y[29989]: 2024-09-10T19:10:42.617+0000 7f9feab4cec0 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-09-10T19:10:42.822 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Sep 10 19:10:42 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-y[29989]: 2024-09-10T19:10:42.699+0000 7f9feab4cec0 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-09-10T19:10:42.822 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Sep 10 19:10:42 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-y[29989]: 2024-09-10T19:10:42.761+0000 7f9feab4cec0 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-09-10T19:10:42.965 INFO:journalctl@ceph.mgr.x.smithi191.stdout:Sep 10 19:10:42 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-x[34967]: 2024-09-10T19:10:42.640+0000 7fc8cb4d9ec0 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-09-10T19:10:43.181 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Sep 10 19:10:42 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-y[29989]: 2024-09-10T19:10:42.820+0000 7f9feab4cec0 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-09-10T19:10:43.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:10:43 smithi191 ceph-mon[32884]: Standby manager daemon x restarted 2024-09-10T19:10:43.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:10:43 smithi191 ceph-mon[32884]: Standby manager daemon x started 2024-09-10T19:10:43.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:10:43 smithi191 ceph-mon[32884]: from='mgr.? 172.21.15.191:0/2051198039' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-09-10T19:10:43.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:10:43 smithi191 ceph-mon[32884]: from='mgr.? 172.21.15.191:0/2051198039' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-10T19:10:43.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:10:43 smithi191 ceph-mon[32884]: from='mgr.? 172.21.15.191:0/2051198039' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-09-10T19:10:43.717 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:10:43 smithi191 ceph-mon[32884]: from='mgr.? 172.21.15.191:0/2051198039' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-10T19:10:43.717 INFO:journalctl@ceph.mgr.x.smithi191.stdout:Sep 10 19:10:43 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-x[34967]: 2024-09-10T19:10:43.370+0000 7fc8cb4d9ec0 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-09-10T19:10:43.717 INFO:journalctl@ceph.mgr.x.smithi191.stdout:Sep 10 19:10:43 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-x[34967]: [10/Sep/2024:19:10:43] ENGINE Bus STARTING 2024-09-10T19:10:43.717 INFO:journalctl@ceph.mgr.x.smithi191.stdout:Sep 10 19:10:43 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-x[34967]: CherryPy Checker: 2024-09-10T19:10:43.718 INFO:journalctl@ceph.mgr.x.smithi191.stdout:Sep 10 19:10:43 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-x[34967]: The Application mounted at '' has an empty config. 2024-09-10T19:10:43.718 INFO:journalctl@ceph.mgr.x.smithi191.stdout:Sep 10 19:10:43 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-x[34967]: 2024-09-10T19:10:43.718 INFO:journalctl@ceph.mgr.x.smithi191.stdout:Sep 10 19:10:43 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-x[34967]: [10/Sep/2024:19:10:43] ENGINE Serving on http://:::9283 2024-09-10T19:10:43.718 INFO:journalctl@ceph.mgr.x.smithi191.stdout:Sep 10 19:10:43 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-x[34967]: [10/Sep/2024:19:10:43] ENGINE Bus STARTED 2024-09-10T19:10:43.756 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:10:43 smithi183 ceph-mon[29736]: Standby manager daemon x restarted 2024-09-10T19:10:43.756 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:10:43 smithi183 ceph-mon[29736]: Standby manager daemon x started 2024-09-10T19:10:43.756 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:10:43 smithi183 ceph-mon[29736]: from='mgr.? 172.21.15.191:0/2051198039' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-09-10T19:10:43.756 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:10:43 smithi183 ceph-mon[29736]: from='mgr.? 172.21.15.191:0/2051198039' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-10T19:10:43.756 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:10:43 smithi183 ceph-mon[29736]: from='mgr.? 172.21.15.191:0/2051198039' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-09-10T19:10:43.756 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:10:43 smithi183 ceph-mon[29736]: from='mgr.? 172.21.15.191:0/2051198039' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-10T19:10:43.757 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:10:43 smithi183 ceph-mon[41562]: Standby manager daemon x restarted 2024-09-10T19:10:43.757 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:10:43 smithi183 ceph-mon[41562]: Standby manager daemon x started 2024-09-10T19:10:43.757 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:10:43 smithi183 ceph-mon[41562]: from='mgr.? 172.21.15.191:0/2051198039' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-09-10T19:10:43.757 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:10:43 smithi183 ceph-mon[41562]: from='mgr.? 172.21.15.191:0/2051198039' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-10T19:10:43.757 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:10:43 smithi183 ceph-mon[41562]: from='mgr.? 172.21.15.191:0/2051198039' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-09-10T19:10:43.757 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:10:43 smithi183 ceph-mon[41562]: from='mgr.? 172.21.15.191:0/2051198039' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-10T19:10:43.757 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Sep 10 19:10:43 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-y[29989]: 2024-09-10T19:10:43.574+0000 7f9feab4cec0 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-09-10T19:10:43.757 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Sep 10 19:10:43 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-y[29989]: [10/Sep/2024:19:10:43] ENGINE Bus STARTING 2024-09-10T19:10:43.757 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Sep 10 19:10:43 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-y[29989]: CherryPy Checker: 2024-09-10T19:10:43.758 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Sep 10 19:10:43 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-y[29989]: The Application mounted at '' has an empty config. 2024-09-10T19:10:43.758 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Sep 10 19:10:43 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-y[29989]: 2024-09-10T19:10:43.758 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Sep 10 19:10:43 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-y[29989]: [10/Sep/2024:19:10:43] ENGINE Serving on http://:::9283 2024-09-10T19:10:44.027 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Sep 10 19:10:43 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-y[29989]: [10/Sep/2024:19:10:43] ENGINE Bus STARTED 2024-09-10T19:10:44.600 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-09-10T19:10:44.600 INFO:teuthology.orchestra.run.smithi183.stdout:{"epoch":73,"fsid":"320f6df8-6fa7-11ef-bcea-c7b262605968","created":"2024-09-10T19:05:03.688531+0000","modified":"2024-09-10T19:10:43.576984+0000","last_up_change":"2024-09-10T19:09:54.624393+0000","last_in_change":"2024-09-10T19:09:38.195646+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-10T19:08:00.456786+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":".rgw.root","create_time":"2024-09-10T19:10:05.993763+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"60","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}},{"pool":3,"pool_name":"datapool","create_time":"2024-09-10T19:10:06.224267+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":3,"pg_placement_num":3,"pg_placement_num_target":3,"pg_num_target":3,"pg_num_pending":3,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"63","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":3,"snap_epoch":63,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":4,"pool_name":"default.rgw.log","create_time":"2024-09-10T19:10:07.958284+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"62","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}},{"pool":5,"pool_name":"default.rgw.control","create_time":"2024-09-10T19:10:09.933596+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"64","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}},{"pool":6,"pool_name":"default.rgw.meta","create_time":"2024-09-10T19:10:11.993712+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"66","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4},"application_metadata":{"rgw":{}}}],"osds":[{"osd":0,"uuid":"9d323010-394a-49aa-ba9b-244d87faf36e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":64,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6802","nonce":2813074952},{"type":"v1","addr":"172.21.15.183:6803","nonce":2813074952}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6804","nonce":2813074952},{"type":"v1","addr":"172.21.15.183:6805","nonce":2813074952}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6808","nonce":2813074952},{"type":"v1","addr":"172.21.15.183:6809","nonce":2813074952}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6806","nonce":2813074952},{"type":"v1","addr":"172.21.15.183:6807","nonce":2813074952}]},"public_addr":"172.21.15.183:6803/2813074952","cluster_addr":"172.21.15.183:6805/2813074952","heartbeat_back_addr":"172.21.15.183:6809/2813074952","heartbeat_front_addr":"172.21.15.183:6807/2813074952","state":["exists","up"]},{"osd":1,"uuid":"71a6c5dc-5109-4284-8d20-187d8e7afc9e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":64,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6810","nonce":3880626114},{"type":"v1","addr":"172.21.15.183:6811","nonce":3880626114}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6812","nonce":3880626114},{"type":"v1","addr":"172.21.15.183:6813","nonce":3880626114}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6816","nonce":3880626114},{"type":"v1","addr":"172.21.15.183:6817","nonce":3880626114}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6814","nonce":3880626114},{"type":"v1","addr":"172.21.15.183:6815","nonce":3880626114}]},"public_addr":"172.21.15.183:6811/3880626114","cluster_addr":"172.21.15.183:6813/3880626114","heartbeat_back_addr":"172.21.15.183:6817/3880626114","heartbeat_front_addr":"172.21.15.183:6815/3880626114","state":["exists","up"]},{"osd":2,"uuid":"fc310d5b-b89b-4655-9989-7e10ba356509","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":64,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6818","nonce":3730013786},{"type":"v1","addr":"172.21.15.183:6819","nonce":3730013786}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6820","nonce":3730013786},{"type":"v1","addr":"172.21.15.183:6821","nonce":3730013786}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6824","nonce":3730013786},{"type":"v1","addr":"172.21.15.183:6825","nonce":3730013786}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6822","nonce":3730013786},{"type":"v1","addr":"172.21.15.183:6823","nonce":3730013786}]},"public_addr":"172.21.15.183:6819/3730013786","cluster_addr":"172.21.15.183:6821/3730013786","heartbeat_back_addr":"172.21.15.183:6825/3730013786","heartbeat_front_addr":"172.21.15.183:6823/3730013786","state":["exists","up"]},{"osd":3,"uuid":"7fd07a95-ad22-43f5-a968-539552283e89","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":64,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6826","nonce":3522836984},{"type":"v1","addr":"172.21.15.183:6827","nonce":3522836984}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6828","nonce":3522836984},{"type":"v1","addr":"172.21.15.183:6829","nonce":3522836984}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6832","nonce":3522836984},{"type":"v1","addr":"172.21.15.183:6833","nonce":3522836984}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6830","nonce":3522836984},{"type":"v1","addr":"172.21.15.183:6831","nonce":3522836984}]},"public_addr":"172.21.15.183:6827/3522836984","cluster_addr":"172.21.15.183:6829/3522836984","heartbeat_back_addr":"172.21.15.183:6833/3522836984","heartbeat_front_addr":"172.21.15.183:6831/3522836984","state":["exists","up"]},{"osd":4,"uuid":"631083fb-e503-44da-9b6b-154e88539ee7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":64,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6800","nonce":2674828539},{"type":"v1","addr":"172.21.15.191:6801","nonce":2674828539}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6802","nonce":2674828539},{"type":"v1","addr":"172.21.15.191:6803","nonce":2674828539}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6806","nonce":2674828539},{"type":"v1","addr":"172.21.15.191:6807","nonce":2674828539}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6804","nonce":2674828539},{"type":"v1","addr":"172.21.15.191:6805","nonce":2674828539}]},"public_addr":"172.21.15.191:6801/2674828539","cluster_addr":"172.21.15.191:6803/2674828539","heartbeat_back_addr":"172.21.15.191:6807/2674828539","heartbeat_front_addr":"172.21.15.191:6805/2674828539","state":["exists","up"]},{"osd":5,"uuid":"6b963695-3eca-4e6e-9207-05b8010ebff9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":64,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6808","nonce":681733378},{"type":"v1","addr":"172.21.15.191:6809","nonce":681733378}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6810","nonce":681733378},{"type":"v1","addr":"172.21.15.191:6811","nonce":681733378}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6814","nonce":681733378},{"type":"v1","addr":"172.21.15.191:6815","nonce":681733378}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6812","nonce":681733378},{"type":"v1","addr":"172.21.15.191:6813","nonce":681733378}]},"public_addr":"172.21.15.191:6809/681733378","cluster_addr":"172.21.15.191:6811/681733378","heartbeat_back_addr":"172.21.15.191:6815/681733378","heartbeat_front_addr":"172.21.15.191:6813/681733378","state":["exists","up"]},{"osd":6,"uuid":"2dc9754f-6aef-4e19-8605-2f1591921811","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":62,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6816","nonce":2825813477},{"type":"v1","addr":"172.21.15.191:6817","nonce":2825813477}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6818","nonce":2825813477},{"type":"v1","addr":"172.21.15.191:6819","nonce":2825813477}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6822","nonce":2825813477},{"type":"v1","addr":"172.21.15.191:6823","nonce":2825813477}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6820","nonce":2825813477},{"type":"v1","addr":"172.21.15.191:6821","nonce":2825813477}]},"public_addr":"172.21.15.191:6817/2825813477","cluster_addr":"172.21.15.191:6819/2825813477","heartbeat_back_addr":"172.21.15.191:6823/2825813477","heartbeat_front_addr":"172.21.15.191:6821/2825813477","state":["exists","up"]},{"osd":7,"uuid":"1a9a299e-d5ba-4b9a-85d5-7f6a5bc754e1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":64,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6824","nonce":1422610021},{"type":"v1","addr":"172.21.15.191:6825","nonce":1422610021}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6826","nonce":1422610021},{"type":"v1","addr":"172.21.15.191:6827","nonce":1422610021}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6830","nonce":1422610021},{"type":"v1","addr":"172.21.15.191:6831","nonce":1422610021}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6828","nonce":1422610021},{"type":"v1","addr":"172.21.15.191:6829","nonce":1422610021}]},"public_addr":"172.21.15.191:6825/1422610021","cluster_addr":"172.21.15.191:6827/1422610021","heartbeat_back_addr":"172.21.15.191:6831/1422610021","heartbeat_front_addr":"172.21.15.191:6829/1422610021","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T19:10:33.174805+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T19:09:55.356115+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T19:10:31.650672+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T19:09:47.698553+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T19:10:00.287014+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T19:10:17.655583+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T19:09:29.209723+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T19:09:53.205672+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.183:6801/3311783362":"2024-09-11T19:10:43.576943+0000","172.21.15.183:0/4088553922":"2024-09-11T19:10:43.576943+0000","172.21.15.183:0/3528769414":"2024-09-11T19:05:48.153879+0000","172.21.15.183:0/1766036658":"2024-09-11T19:05:48.153879+0000","172.21.15.183:0/3044170088":"2024-09-11T19:05:25.702354+0000","172.21.15.183:0/3037333778":"2024-09-11T19:05:25.702354+0000","172.21.15.183:6801/3747320570":"2024-09-11T19:05:25.702354+0000","172.21.15.183:0/3982386906":"2024-09-11T19:10:43.576943+0000","172.21.15.183:0/2436014455":"2024-09-11T19:05:25.702354+0000","172.21.15.183:6801/659745765":"2024-09-11T19:05:48.153879+0000","172.21.15.183:6800/3747320570":"2024-09-11T19:05:25.702354+0000","172.21.15.183:0/1652576282":"2024-09-11T19:10:43.576943+0000","172.21.15.183:6800/3311783362":"2024-09-11T19:10:43.576943+0000","172.21.15.183:0/1012899549":"2024-09-11T19:05:48.153879+0000","172.21.15.183:6800/659745765":"2024-09-11T19:05:48.153879+0000","172.21.15.183:0/412044400":"2024-09-11T19:10:43.576943+0000","172.21.15.183:0/3636171401":"2024-09-11T19:10:43.576943+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-10T19:10:44.600 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:10:44 smithi183 ceph-mon[29736]: mgrmap e18: y(active, since 4m), standbys: x 2024-09-10T19:10:44.600 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:10:44 smithi183 ceph-mon[29736]: Active manager daemon y restarted 2024-09-10T19:10:44.600 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:10:44 smithi183 ceph-mon[29736]: Activating manager daemon y 2024-09-10T19:10:44.601 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:10:44 smithi183 ceph-mon[29736]: osdmap e73: 8 total, 8 up, 8 in 2024-09-10T19:10:44.601 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:10:44 smithi183 ceph-mon[29736]: mgrmap e19: y(active, starting, since 0.00965688s), standbys: x 2024-09-10T19:10:44.601 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:10:44 smithi183 ceph-mon[29736]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-10T19:10:44.601 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:10:44 smithi183 ceph-mon[29736]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-10T19:10:44.601 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:10:44 smithi183 ceph-mon[29736]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-10T19:10:44.601 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:10:44 smithi183 ceph-mon[29736]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-09-10T19:10:44.601 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:10:44 smithi183 ceph-mon[29736]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-09-10T19:10:44.601 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:10:44 smithi183 ceph-mon[29736]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-10T19:10:44.601 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:10:44 smithi183 ceph-mon[29736]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-10T19:10:44.601 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:10:44 smithi183 ceph-mon[29736]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-10T19:10:44.601 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:10:44 smithi183 ceph-mon[29736]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-10T19:10:44.601 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:10:44 smithi183 ceph-mon[29736]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-10T19:10:44.601 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:10:44 smithi183 ceph-mon[29736]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-10T19:10:44.602 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:10:44 smithi183 ceph-mon[29736]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-10T19:10:44.602 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:10:44 smithi183 ceph-mon[29736]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-10T19:10:44.602 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:10:44 smithi183 ceph-mon[29736]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-09-10T19:10:44.602 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:10:44 smithi183 ceph-mon[29736]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-09-10T19:10:44.602 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:10:44 smithi183 ceph-mon[29736]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-09-10T19:10:44.602 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:10:44 smithi183 ceph-mon[29736]: Manager daemon y is now available 2024-09-10T19:10:44.602 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:10:44 smithi183 ceph-mon[29736]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T19:10:44.602 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:10:44 smithi183 ceph-mon[29736]: from='mgr.14556 ' entity='mgr.y' 2024-09-10T19:10:44.602 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:10:44 smithi183 ceph-mon[29736]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:10:44.602 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:10:44 smithi183 ceph-mon[29736]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-09-10T19:10:44.602 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:10:44 smithi183 ceph-mon[29736]: from='mgr.14556 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-09-10T19:10:44.602 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:10:44 smithi183 ceph-mon[29736]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-09-10T19:10:44.602 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:10:44 smithi183 ceph-mon[29736]: from='mgr.14556 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-09-10T19:10:44.603 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:10:44 smithi183 ceph-mon[41562]: mgrmap e18: y(active, since 4m), standbys: x 2024-09-10T19:10:44.603 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:10:44 smithi183 ceph-mon[41562]: Active manager daemon y restarted 2024-09-10T19:10:44.603 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:10:44 smithi183 ceph-mon[41562]: Activating manager daemon y 2024-09-10T19:10:44.603 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:10:44 smithi183 ceph-mon[41562]: osdmap e73: 8 total, 8 up, 8 in 2024-09-10T19:10:44.603 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:10:44 smithi183 ceph-mon[41562]: mgrmap e19: y(active, starting, since 0.00965688s), standbys: x 2024-09-10T19:10:44.603 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:10:44 smithi183 ceph-mon[41562]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-10T19:10:44.603 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:10:44 smithi183 ceph-mon[41562]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-10T19:10:44.603 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:10:44 smithi183 ceph-mon[41562]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-10T19:10:44.603 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:10:44 smithi183 ceph-mon[41562]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-09-10T19:10:44.603 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:10:44 smithi183 ceph-mon[41562]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-09-10T19:10:44.604 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:10:44 smithi183 ceph-mon[41562]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-10T19:10:44.604 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:10:44 smithi183 ceph-mon[41562]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-10T19:10:44.604 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:10:44 smithi183 ceph-mon[41562]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-10T19:10:44.604 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:10:44 smithi183 ceph-mon[41562]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-10T19:10:44.604 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:10:44 smithi183 ceph-mon[41562]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-10T19:10:44.604 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:10:44 smithi183 ceph-mon[41562]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-10T19:10:44.604 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:10:44 smithi183 ceph-mon[41562]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-10T19:10:44.604 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:10:44 smithi183 ceph-mon[41562]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-10T19:10:44.604 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:10:44 smithi183 ceph-mon[41562]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-09-10T19:10:44.604 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:10:44 smithi183 ceph-mon[41562]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-09-10T19:10:44.604 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:10:44 smithi183 ceph-mon[41562]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-09-10T19:10:44.604 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:10:44 smithi183 ceph-mon[41562]: Manager daemon y is now available 2024-09-10T19:10:44.604 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:10:44 smithi183 ceph-mon[41562]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T19:10:44.605 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:10:44 smithi183 ceph-mon[41562]: from='mgr.14556 ' entity='mgr.y' 2024-09-10T19:10:44.605 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:10:44 smithi183 ceph-mon[41562]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:10:44.605 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:10:44 smithi183 ceph-mon[41562]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-09-10T19:10:44.605 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:10:44 smithi183 ceph-mon[41562]: from='mgr.14556 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-09-10T19:10:44.605 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:10:44 smithi183 ceph-mon[41562]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-09-10T19:10:44.605 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:10:44 smithi183 ceph-mon[41562]: from='mgr.14556 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-09-10T19:10:44.715 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:10:44 smithi191 ceph-mon[32884]: mgrmap e18: y(active, since 4m), standbys: x 2024-09-10T19:10:44.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:10:44 smithi191 ceph-mon[32884]: Active manager daemon y restarted 2024-09-10T19:10:44.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:10:44 smithi191 ceph-mon[32884]: Activating manager daemon y 2024-09-10T19:10:44.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:10:44 smithi191 ceph-mon[32884]: osdmap e73: 8 total, 8 up, 8 in 2024-09-10T19:10:44.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:10:44 smithi191 ceph-mon[32884]: mgrmap e19: y(active, starting, since 0.00965688s), standbys: x 2024-09-10T19:10:44.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:10:44 smithi191 ceph-mon[32884]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-10T19:10:44.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:10:44 smithi191 ceph-mon[32884]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-10T19:10:44.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:10:44 smithi191 ceph-mon[32884]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-10T19:10:44.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:10:44 smithi191 ceph-mon[32884]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-09-10T19:10:44.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:10:44 smithi191 ceph-mon[32884]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-09-10T19:10:44.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:10:44 smithi191 ceph-mon[32884]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-10T19:10:44.717 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:10:44 smithi191 ceph-mon[32884]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-10T19:10:44.717 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:10:44 smithi191 ceph-mon[32884]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-10T19:10:44.717 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:10:44 smithi191 ceph-mon[32884]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-10T19:10:44.717 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:10:44 smithi191 ceph-mon[32884]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-10T19:10:44.717 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:10:44 smithi191 ceph-mon[32884]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-10T19:10:44.717 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:10:44 smithi191 ceph-mon[32884]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-10T19:10:44.717 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:10:44 smithi191 ceph-mon[32884]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-10T19:10:44.717 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:10:44 smithi191 ceph-mon[32884]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-09-10T19:10:44.717 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:10:44 smithi191 ceph-mon[32884]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-09-10T19:10:44.717 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:10:44 smithi191 ceph-mon[32884]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-09-10T19:10:44.717 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:10:44 smithi191 ceph-mon[32884]: Manager daemon y is now available 2024-09-10T19:10:44.717 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:10:44 smithi191 ceph-mon[32884]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T19:10:44.717 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:10:44 smithi191 ceph-mon[32884]: from='mgr.14556 ' entity='mgr.y' 2024-09-10T19:10:44.717 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:10:44 smithi191 ceph-mon[32884]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:10:44.718 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:10:44 smithi191 ceph-mon[32884]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-09-10T19:10:44.718 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:10:44 smithi191 ceph-mon[32884]: from='mgr.14556 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-09-10T19:10:44.718 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:10:44 smithi191 ceph-mon[32884]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-09-10T19:10:44.718 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:10:44 smithi191 ceph-mon[32884]: from='mgr.14556 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-09-10T19:10:44.932 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Sep 10 19:10:44 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-y[29989]: 2024-09-10T19:10:44.661+0000 7f9f8d8ce640 -1 log_channel(cephadm) log [ERR] : [10/Sep/2024:19:10:44] ENGINE Error in HTTPServer.serve 2024-09-10T19:10:44.932 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Sep 10 19:10:44 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-y[29989]: Traceback (most recent call last): 2024-09-10T19:10:44.932 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Sep 10 19:10:44 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-y[29989]: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-10T19:10:44.932 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Sep 10 19:10:44 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-y[29989]: self._connections.run(self.expiration_interval) 2024-09-10T19:10:44.932 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Sep 10 19:10:44 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-y[29989]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-10T19:10:44.932 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Sep 10 19:10:44 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-y[29989]: self._run(expiration_interval) 2024-09-10T19:10:44.932 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Sep 10 19:10:44 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-y[29989]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-10T19:10:44.932 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Sep 10 19:10:44 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-y[29989]: new_conn = self._from_server_socket(self.server.socket) 2024-09-10T19:10:44.933 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Sep 10 19:10:44 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-y[29989]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-10T19:10:44.933 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Sep 10 19:10:44 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-y[29989]: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-10T19:10:44.933 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Sep 10 19:10:44 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-y[29989]: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-10T19:10:44.933 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Sep 10 19:10:44 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-y[29989]: s = self.context.wrap_socket( 2024-09-10T19:10:44.933 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Sep 10 19:10:44 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-y[29989]: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-10T19:10:44.933 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Sep 10 19:10:44 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-y[29989]: return self.sslsocket_class._create( 2024-09-10T19:10:44.933 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Sep 10 19:10:44 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-y[29989]: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-10T19:10:44.933 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Sep 10 19:10:44 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-y[29989]: self.do_handshake() 2024-09-10T19:10:44.933 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Sep 10 19:10:44 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-y[29989]: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-10T19:10:44.933 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Sep 10 19:10:44 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-y[29989]: self._sslobj.do_handshake() 2024-09-10T19:10:44.933 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Sep 10 19:10:44 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-y[29989]: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-10T19:10:44.933 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Sep 10 19:10:44 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-y[29989]: 2024-09-10T19:10:45.583 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-09-10T19:10:45.583 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph osd dump --format=json 2024-09-10T19:10:45.719 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:10:45 smithi183 ceph-mon[29736]: [10/Sep/2024:19:10:44] ENGINE Bus STARTING 2024-09-10T19:10:45.719 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:10:45 smithi183 ceph-mon[29736]: mgrmap e20: y(active, since 1.01146s), standbys: x 2024-09-10T19:10:45.719 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:10:45 smithi183 ceph-mon[29736]: pgmap v3: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:10:45.719 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:10:45 smithi183 ceph-mon[29736]: from='client.? 172.21.15.183:0/4103699768' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-10T19:10:45.719 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:10:45 smithi183 ceph-mon[29736]: osdmap e74: 8 total, 8 up, 8 in 2024-09-10T19:10:45.719 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:10:45 smithi183 ceph-mon[29736]: [10/Sep/2024:19:10:44] ENGINE Serving on https://172.21.15.183:7150 2024-09-10T19:10:45.719 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:10:45 smithi183 ceph-mon[29736]: [10/Sep/2024:19:10:44] ENGINE Bus STARTED 2024-09-10T19:10:45.720 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:10:45 smithi183 ceph-mon[29736]: [10/Sep/2024:19:10:44] ENGINE Error in HTTPServer.serve 2024-09-10T19:10:45.720 INFO:journalctl@ceph.mon.a.smithi183.stdout: Traceback (most recent call last): 2024-09-10T19:10:45.720 INFO:journalctl@ceph.mon.a.smithi183.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-10T19:10:45.720 INFO:journalctl@ceph.mon.a.smithi183.stdout: self._connections.run(self.expiration_interval) 2024-09-10T19:10:45.720 INFO:journalctl@ceph.mon.a.smithi183.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-10T19:10:45.720 INFO:journalctl@ceph.mon.a.smithi183.stdout: self._run(expiration_interval) 2024-09-10T19:10:45.720 INFO:journalctl@ceph.mon.a.smithi183.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-10T19:10:45.720 INFO:journalctl@ceph.mon.a.smithi183.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-09-10T19:10:45.720 INFO:journalctl@ceph.mon.a.smithi183.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-10T19:10:45.721 INFO:journalctl@ceph.mon.a.smithi183.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-10T19:10:45.721 INFO:journalctl@ceph.mon.a.smithi183.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-10T19:10:45.721 INFO:journalctl@ceph.mon.a.smithi183.stdout: s = self.context.wrap_socket( 2024-09-10T19:10:45.721 INFO:journalctl@ceph.mon.a.smithi183.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-10T19:10:45.721 INFO:journalctl@ceph.mon.a.smithi183.stdout: return self.sslsocket_class._create( 2024-09-10T19:10:45.721 INFO:journalctl@ceph.mon.a.smithi183.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-10T19:10:45.722 INFO:journalctl@ceph.mon.a.smithi183.stdout: self.do_handshake() 2024-09-10T19:10:45.722 INFO:journalctl@ceph.mon.a.smithi183.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-10T19:10:45.722 INFO:journalctl@ceph.mon.a.smithi183.stdout: self._sslobj.do_handshake() 2024-09-10T19:10:45.722 INFO:journalctl@ceph.mon.a.smithi183.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-10T19:10:45.722 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:10:45 smithi183 ceph-mon[29736]: from='mgr.14556 ' entity='mgr.y' 2024-09-10T19:10:45.723 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:10:45 smithi183 ceph-mon[41562]: [10/Sep/2024:19:10:44] ENGINE Bus STARTING 2024-09-10T19:10:45.723 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:10:45 smithi183 ceph-mon[41562]: mgrmap e20: y(active, since 1.01146s), standbys: x 2024-09-10T19:10:45.723 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:10:45 smithi183 ceph-mon[41562]: pgmap v3: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:10:45.723 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:10:45 smithi183 ceph-mon[41562]: from='client.? 172.21.15.183:0/4103699768' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-10T19:10:45.723 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:10:45 smithi183 ceph-mon[41562]: osdmap e74: 8 total, 8 up, 8 in 2024-09-10T19:10:45.723 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:10:45 smithi183 ceph-mon[41562]: [10/Sep/2024:19:10:44] ENGINE Serving on https://172.21.15.183:7150 2024-09-10T19:10:45.723 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:10:45 smithi183 ceph-mon[41562]: [10/Sep/2024:19:10:44] ENGINE Bus STARTED 2024-09-10T19:10:45.723 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:10:45 smithi183 ceph-mon[41562]: [10/Sep/2024:19:10:44] ENGINE Error in HTTPServer.serve 2024-09-10T19:10:45.723 INFO:journalctl@ceph.mon.c.smithi183.stdout: Traceback (most recent call last): 2024-09-10T19:10:45.723 INFO:journalctl@ceph.mon.c.smithi183.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-10T19:10:45.724 INFO:journalctl@ceph.mon.c.smithi183.stdout: self._connections.run(self.expiration_interval) 2024-09-10T19:10:45.724 INFO:journalctl@ceph.mon.c.smithi183.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-10T19:10:45.724 INFO:journalctl@ceph.mon.c.smithi183.stdout: self._run(expiration_interval) 2024-09-10T19:10:45.724 INFO:journalctl@ceph.mon.c.smithi183.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-10T19:10:45.724 INFO:journalctl@ceph.mon.c.smithi183.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-09-10T19:10:45.724 INFO:journalctl@ceph.mon.c.smithi183.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-10T19:10:45.724 INFO:journalctl@ceph.mon.c.smithi183.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-10T19:10:45.724 INFO:journalctl@ceph.mon.c.smithi183.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-10T19:10:45.724 INFO:journalctl@ceph.mon.c.smithi183.stdout: s = self.context.wrap_socket( 2024-09-10T19:10:45.724 INFO:journalctl@ceph.mon.c.smithi183.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-10T19:10:45.724 INFO:journalctl@ceph.mon.c.smithi183.stdout: return self.sslsocket_class._create( 2024-09-10T19:10:45.724 INFO:journalctl@ceph.mon.c.smithi183.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-10T19:10:45.724 INFO:journalctl@ceph.mon.c.smithi183.stdout: self.do_handshake() 2024-09-10T19:10:45.725 INFO:journalctl@ceph.mon.c.smithi183.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-10T19:10:45.725 INFO:journalctl@ceph.mon.c.smithi183.stdout: self._sslobj.do_handshake() 2024-09-10T19:10:45.725 INFO:journalctl@ceph.mon.c.smithi183.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-10T19:10:45.725 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:10:45 smithi183 ceph-mon[41562]: from='mgr.14556 ' entity='mgr.y' 2024-09-10T19:10:45.965 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:10:45 smithi191 ceph-mon[32884]: [10/Sep/2024:19:10:44] ENGINE Bus STARTING 2024-09-10T19:10:45.966 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:10:45 smithi191 ceph-mon[32884]: mgrmap e20: y(active, since 1.01146s), standbys: x 2024-09-10T19:10:45.966 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:10:45 smithi191 ceph-mon[32884]: pgmap v3: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:10:45.966 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:10:45 smithi191 ceph-mon[32884]: from='client.? 172.21.15.183:0/4103699768' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-10T19:10:45.966 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:10:45 smithi191 ceph-mon[32884]: osdmap e74: 8 total, 8 up, 8 in 2024-09-10T19:10:45.966 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:10:45 smithi191 ceph-mon[32884]: [10/Sep/2024:19:10:44] ENGINE Serving on https://172.21.15.183:7150 2024-09-10T19:10:45.966 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:10:45 smithi191 ceph-mon[32884]: [10/Sep/2024:19:10:44] ENGINE Bus STARTED 2024-09-10T19:10:45.966 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:10:45 smithi191 ceph-mon[32884]: [10/Sep/2024:19:10:44] ENGINE Error in HTTPServer.serve 2024-09-10T19:10:45.966 INFO:journalctl@ceph.mon.b.smithi191.stdout: Traceback (most recent call last): 2024-09-10T19:10:45.966 INFO:journalctl@ceph.mon.b.smithi191.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-10T19:10:45.966 INFO:journalctl@ceph.mon.b.smithi191.stdout: self._connections.run(self.expiration_interval) 2024-09-10T19:10:45.966 INFO:journalctl@ceph.mon.b.smithi191.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-10T19:10:45.966 INFO:journalctl@ceph.mon.b.smithi191.stdout: self._run(expiration_interval) 2024-09-10T19:10:45.966 INFO:journalctl@ceph.mon.b.smithi191.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-10T19:10:45.967 INFO:journalctl@ceph.mon.b.smithi191.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-09-10T19:10:45.967 INFO:journalctl@ceph.mon.b.smithi191.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-10T19:10:45.967 INFO:journalctl@ceph.mon.b.smithi191.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-10T19:10:45.967 INFO:journalctl@ceph.mon.b.smithi191.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-10T19:10:45.967 INFO:journalctl@ceph.mon.b.smithi191.stdout: s = self.context.wrap_socket( 2024-09-10T19:10:45.967 INFO:journalctl@ceph.mon.b.smithi191.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-10T19:10:45.967 INFO:journalctl@ceph.mon.b.smithi191.stdout: return self.sslsocket_class._create( 2024-09-10T19:10:45.967 INFO:journalctl@ceph.mon.b.smithi191.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-10T19:10:45.967 INFO:journalctl@ceph.mon.b.smithi191.stdout: self.do_handshake() 2024-09-10T19:10:45.967 INFO:journalctl@ceph.mon.b.smithi191.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-10T19:10:45.967 INFO:journalctl@ceph.mon.b.smithi191.stdout: self._sslobj.do_handshake() 2024-09-10T19:10:45.967 INFO:journalctl@ceph.mon.b.smithi191.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-10T19:10:45.967 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:10:45 smithi191 ceph-mon[32884]: from='mgr.14556 ' entity='mgr.y' 2024-09-10T19:10:46.170 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:10:46.931 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:10:46 smithi183 ceph-mon[29736]: pgmap v5: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:10:46.931 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:10:46 smithi183 ceph-mon[29736]: mgrmap e21: y(active, since 2s), standbys: x 2024-09-10T19:10:46.931 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:10:46 smithi183 ceph-mon[41562]: pgmap v5: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:10:46.931 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:10:46 smithi183 ceph-mon[41562]: mgrmap e21: y(active, since 2s), standbys: x 2024-09-10T19:10:46.965 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:10:46 smithi191 ceph-mon[32884]: pgmap v5: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:10:46.965 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:10:46 smithi191 ceph-mon[32884]: mgrmap e21: y(active, since 2s), standbys: x 2024-09-10T19:10:47.701 INFO:journalctl@ceph.mgr.x.smithi191.stdout:Sep 10 19:10:47 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-x[34967]: ::ffff:172.21.15.191 - - [10/Sep/2024:19:10:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T19:10:47.965 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:10:47 smithi191 ceph-mon[32884]: from='mgr.14556 ' entity='mgr.y' 2024-09-10T19:10:47.966 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:10:47 smithi191 ceph-mon[32884]: from='mgr.14556 ' entity='mgr.y' 2024-09-10T19:10:47.966 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:10:47 smithi191 ceph-mon[32884]: from='mgr.14556 ' entity='mgr.y' 2024-09-10T19:10:47.966 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:10:47 smithi191 ceph-mon[32884]: from='mgr.14556 ' entity='mgr.y' 2024-09-10T19:10:47.966 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:10:47 smithi191 ceph-mon[32884]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-10T19:10:47.966 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:10:47 smithi191 ceph-mon[32884]: from='mgr.14556 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-10T19:10:47.966 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:10:47 smithi191 ceph-mon[32884]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-10T19:10:47.966 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:10:47 smithi191 ceph-mon[32884]: from='mgr.14556 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-10T19:10:47.966 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:10:47 smithi191 ceph-mon[32884]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-10T19:10:47.966 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:10:47 smithi191 ceph-mon[32884]: from='mgr.14556 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-10T19:10:47.966 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:10:47 smithi191 ceph-mon[32884]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-10T19:10:47.966 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:10:47 smithi191 ceph-mon[32884]: from='mgr.14556 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-10T19:10:47.967 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:10:47 smithi191 ceph-mon[32884]: from='mgr.14556 ' entity='mgr.y' 2024-09-10T19:10:47.967 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:10:47 smithi191 ceph-mon[32884]: from='mgr.14556 ' entity='mgr.y' 2024-09-10T19:10:47.967 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:10:47 smithi191 ceph-mon[32884]: from='mgr.14556 ' entity='mgr.y' 2024-09-10T19:10:47.972 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:10:47 smithi183 ceph-mon[29736]: from='mgr.14556 ' entity='mgr.y' 2024-09-10T19:10:47.972 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:10:47 smithi183 ceph-mon[29736]: from='mgr.14556 ' entity='mgr.y' 2024-09-10T19:10:47.972 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:10:47 smithi183 ceph-mon[29736]: from='mgr.14556 ' entity='mgr.y' 2024-09-10T19:10:47.972 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:10:47 smithi183 ceph-mon[29736]: from='mgr.14556 ' entity='mgr.y' 2024-09-10T19:10:47.972 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:10:47 smithi183 ceph-mon[29736]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-10T19:10:47.972 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:10:47 smithi183 ceph-mon[29736]: from='mgr.14556 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-10T19:10:47.972 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:10:47 smithi183 ceph-mon[29736]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-10T19:10:47.972 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:10:47 smithi183 ceph-mon[29736]: from='mgr.14556 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-10T19:10:47.973 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:10:47 smithi183 ceph-mon[29736]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-10T19:10:47.973 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:10:47 smithi183 ceph-mon[29736]: from='mgr.14556 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-10T19:10:47.973 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:10:47 smithi183 ceph-mon[29736]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-10T19:10:47.973 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:10:47 smithi183 ceph-mon[29736]: from='mgr.14556 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-10T19:10:47.973 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:10:47 smithi183 ceph-mon[29736]: from='mgr.14556 ' entity='mgr.y' 2024-09-10T19:10:47.973 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:10:47 smithi183 ceph-mon[29736]: from='mgr.14556 ' entity='mgr.y' 2024-09-10T19:10:47.973 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:10:47 smithi183 ceph-mon[29736]: from='mgr.14556 ' entity='mgr.y' 2024-09-10T19:10:47.973 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:10:47 smithi183 ceph-mon[41562]: from='mgr.14556 ' entity='mgr.y' 2024-09-10T19:10:47.973 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:10:47 smithi183 ceph-mon[41562]: from='mgr.14556 ' entity='mgr.y' 2024-09-10T19:10:47.973 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:10:47 smithi183 ceph-mon[41562]: from='mgr.14556 ' entity='mgr.y' 2024-09-10T19:10:47.973 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:10:47 smithi183 ceph-mon[41562]: from='mgr.14556 ' entity='mgr.y' 2024-09-10T19:10:47.974 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:10:47 smithi183 ceph-mon[41562]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-10T19:10:47.974 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:10:47 smithi183 ceph-mon[41562]: from='mgr.14556 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-10T19:10:47.974 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:10:47 smithi183 ceph-mon[41562]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-10T19:10:47.974 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:10:47 smithi183 ceph-mon[41562]: from='mgr.14556 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-10T19:10:47.974 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:10:47 smithi183 ceph-mon[41562]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-10T19:10:47.974 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:10:47 smithi183 ceph-mon[41562]: from='mgr.14556 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-10T19:10:47.974 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:10:47 smithi183 ceph-mon[41562]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-10T19:10:47.974 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:10:47 smithi183 ceph-mon[41562]: from='mgr.14556 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-10T19:10:47.974 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:10:47 smithi183 ceph-mon[41562]: from='mgr.14556 ' entity='mgr.y' 2024-09-10T19:10:47.975 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:10:47 smithi183 ceph-mon[41562]: from='mgr.14556 ' entity='mgr.y' 2024-09-10T19:10:47.975 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:10:47 smithi183 ceph-mon[41562]: from='mgr.14556 ' entity='mgr.y' 2024-09-10T19:10:48.135 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-09-10T19:10:48.136 INFO:teuthology.orchestra.run.smithi183.stdout:{"epoch":74,"fsid":"320f6df8-6fa7-11ef-bcea-c7b262605968","created":"2024-09-10T19:05:03.688531+0000","modified":"2024-09-10T19:10:44.636496+0000","last_up_change":"2024-09-10T19:09:54.624393+0000","last_in_change":"2024-09-10T19:09:38.195646+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-10T19:08:00.456786+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":".rgw.root","create_time":"2024-09-10T19:10:05.993763+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"60","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}},{"pool":3,"pool_name":"datapool","create_time":"2024-09-10T19:10:06.224267+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":3,"pg_placement_num":3,"pg_placement_num_target":3,"pg_num_target":3,"pg_num_pending":3,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"63","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":3,"snap_epoch":63,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":4,"pool_name":"default.rgw.log","create_time":"2024-09-10T19:10:07.958284+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"62","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}},{"pool":5,"pool_name":"default.rgw.control","create_time":"2024-09-10T19:10:09.933596+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"64","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}},{"pool":6,"pool_name":"default.rgw.meta","create_time":"2024-09-10T19:10:11.993712+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"66","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4},"application_metadata":{"rgw":{}}}],"osds":[{"osd":0,"uuid":"9d323010-394a-49aa-ba9b-244d87faf36e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":64,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6802","nonce":2813074952},{"type":"v1","addr":"172.21.15.183:6803","nonce":2813074952}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6804","nonce":2813074952},{"type":"v1","addr":"172.21.15.183:6805","nonce":2813074952}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6808","nonce":2813074952},{"type":"v1","addr":"172.21.15.183:6809","nonce":2813074952}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6806","nonce":2813074952},{"type":"v1","addr":"172.21.15.183:6807","nonce":2813074952}]},"public_addr":"172.21.15.183:6803/2813074952","cluster_addr":"172.21.15.183:6805/2813074952","heartbeat_back_addr":"172.21.15.183:6809/2813074952","heartbeat_front_addr":"172.21.15.183:6807/2813074952","state":["exists","up"]},{"osd":1,"uuid":"71a6c5dc-5109-4284-8d20-187d8e7afc9e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":64,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6810","nonce":3880626114},{"type":"v1","addr":"172.21.15.183:6811","nonce":3880626114}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6812","nonce":3880626114},{"type":"v1","addr":"172.21.15.183:6813","nonce":3880626114}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6816","nonce":3880626114},{"type":"v1","addr":"172.21.15.183:6817","nonce":3880626114}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6814","nonce":3880626114},{"type":"v1","addr":"172.21.15.183:6815","nonce":3880626114}]},"public_addr":"172.21.15.183:6811/3880626114","cluster_addr":"172.21.15.183:6813/3880626114","heartbeat_back_addr":"172.21.15.183:6817/3880626114","heartbeat_front_addr":"172.21.15.183:6815/3880626114","state":["exists","up"]},{"osd":2,"uuid":"fc310d5b-b89b-4655-9989-7e10ba356509","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":64,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6818","nonce":3730013786},{"type":"v1","addr":"172.21.15.183:6819","nonce":3730013786}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6820","nonce":3730013786},{"type":"v1","addr":"172.21.15.183:6821","nonce":3730013786}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6824","nonce":3730013786},{"type":"v1","addr":"172.21.15.183:6825","nonce":3730013786}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6822","nonce":3730013786},{"type":"v1","addr":"172.21.15.183:6823","nonce":3730013786}]},"public_addr":"172.21.15.183:6819/3730013786","cluster_addr":"172.21.15.183:6821/3730013786","heartbeat_back_addr":"172.21.15.183:6825/3730013786","heartbeat_front_addr":"172.21.15.183:6823/3730013786","state":["exists","up"]},{"osd":3,"uuid":"7fd07a95-ad22-43f5-a968-539552283e89","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":64,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6826","nonce":3522836984},{"type":"v1","addr":"172.21.15.183:6827","nonce":3522836984}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6828","nonce":3522836984},{"type":"v1","addr":"172.21.15.183:6829","nonce":3522836984}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6832","nonce":3522836984},{"type":"v1","addr":"172.21.15.183:6833","nonce":3522836984}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6830","nonce":3522836984},{"type":"v1","addr":"172.21.15.183:6831","nonce":3522836984}]},"public_addr":"172.21.15.183:6827/3522836984","cluster_addr":"172.21.15.183:6829/3522836984","heartbeat_back_addr":"172.21.15.183:6833/3522836984","heartbeat_front_addr":"172.21.15.183:6831/3522836984","state":["exists","up"]},{"osd":4,"uuid":"631083fb-e503-44da-9b6b-154e88539ee7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":64,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6800","nonce":2674828539},{"type":"v1","addr":"172.21.15.191:6801","nonce":2674828539}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6802","nonce":2674828539},{"type":"v1","addr":"172.21.15.191:6803","nonce":2674828539}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6806","nonce":2674828539},{"type":"v1","addr":"172.21.15.191:6807","nonce":2674828539}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6804","nonce":2674828539},{"type":"v1","addr":"172.21.15.191:6805","nonce":2674828539}]},"public_addr":"172.21.15.191:6801/2674828539","cluster_addr":"172.21.15.191:6803/2674828539","heartbeat_back_addr":"172.21.15.191:6807/2674828539","heartbeat_front_addr":"172.21.15.191:6805/2674828539","state":["exists","up"]},{"osd":5,"uuid":"6b963695-3eca-4e6e-9207-05b8010ebff9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":64,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6808","nonce":681733378},{"type":"v1","addr":"172.21.15.191:6809","nonce":681733378}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6810","nonce":681733378},{"type":"v1","addr":"172.21.15.191:6811","nonce":681733378}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6814","nonce":681733378},{"type":"v1","addr":"172.21.15.191:6815","nonce":681733378}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6812","nonce":681733378},{"type":"v1","addr":"172.21.15.191:6813","nonce":681733378}]},"public_addr":"172.21.15.191:6809/681733378","cluster_addr":"172.21.15.191:6811/681733378","heartbeat_back_addr":"172.21.15.191:6815/681733378","heartbeat_front_addr":"172.21.15.191:6813/681733378","state":["exists","up"]},{"osd":6,"uuid":"2dc9754f-6aef-4e19-8605-2f1591921811","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":62,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6816","nonce":2825813477},{"type":"v1","addr":"172.21.15.191:6817","nonce":2825813477}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6818","nonce":2825813477},{"type":"v1","addr":"172.21.15.191:6819","nonce":2825813477}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6822","nonce":2825813477},{"type":"v1","addr":"172.21.15.191:6823","nonce":2825813477}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6820","nonce":2825813477},{"type":"v1","addr":"172.21.15.191:6821","nonce":2825813477}]},"public_addr":"172.21.15.191:6817/2825813477","cluster_addr":"172.21.15.191:6819/2825813477","heartbeat_back_addr":"172.21.15.191:6823/2825813477","heartbeat_front_addr":"172.21.15.191:6821/2825813477","state":["exists","up"]},{"osd":7,"uuid":"1a9a299e-d5ba-4b9a-85d5-7f6a5bc754e1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":64,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6824","nonce":1422610021},{"type":"v1","addr":"172.21.15.191:6825","nonce":1422610021}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6826","nonce":1422610021},{"type":"v1","addr":"172.21.15.191:6827","nonce":1422610021}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6830","nonce":1422610021},{"type":"v1","addr":"172.21.15.191:6831","nonce":1422610021}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6828","nonce":1422610021},{"type":"v1","addr":"172.21.15.191:6829","nonce":1422610021}]},"public_addr":"172.21.15.191:6825/1422610021","cluster_addr":"172.21.15.191:6827/1422610021","heartbeat_back_addr":"172.21.15.191:6831/1422610021","heartbeat_front_addr":"172.21.15.191:6829/1422610021","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T19:10:33.174805+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T19:09:55.356115+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T19:10:31.650672+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T19:09:47.698553+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T19:10:00.287014+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T19:10:17.655583+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T19:09:29.209723+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T19:09:53.205672+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.183:6801/3311783362":"2024-09-11T19:10:43.576943+0000","172.21.15.183:0/4088553922":"2024-09-11T19:10:43.576943+0000","172.21.15.183:0/3528769414":"2024-09-11T19:05:48.153879+0000","172.21.15.183:0/1766036658":"2024-09-11T19:05:48.153879+0000","172.21.15.183:0/3044170088":"2024-09-11T19:05:25.702354+0000","172.21.15.183:0/3037333778":"2024-09-11T19:05:25.702354+0000","172.21.15.183:6801/3747320570":"2024-09-11T19:05:25.702354+0000","172.21.15.183:0/3982386906":"2024-09-11T19:10:43.576943+0000","172.21.15.183:0/2436014455":"2024-09-11T19:05:25.702354+0000","172.21.15.183:6801/659745765":"2024-09-11T19:05:48.153879+0000","172.21.15.183:6800/3747320570":"2024-09-11T19:05:25.702354+0000","172.21.15.183:0/1652576282":"2024-09-11T19:10:43.576943+0000","172.21.15.183:6800/3311783362":"2024-09-11T19:10:43.576943+0000","172.21.15.183:0/1012899549":"2024-09-11T19:05:48.153879+0000","172.21.15.183:6800/659745765":"2024-09-11T19:05:48.153879+0000","172.21.15.183:0/412044400":"2024-09-11T19:10:43.576943+0000","172.21.15.183:0/3636171401":"2024-09-11T19:10:43.576943+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-10T19:10:48.830 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph tell osd.0 flush_pg_stats 2024-09-10T19:10:48.831 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph tell osd.1 flush_pg_stats 2024-09-10T19:10:48.831 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph tell osd.2 flush_pg_stats 2024-09-10T19:10:48.832 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph tell osd.3 flush_pg_stats 2024-09-10T19:10:48.832 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph tell osd.4 flush_pg_stats 2024-09-10T19:10:48.833 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph tell osd.5 flush_pg_stats 2024-09-10T19:10:48.833 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph tell osd.6 flush_pg_stats 2024-09-10T19:10:48.834 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph tell osd.7 flush_pg_stats 2024-09-10T19:10:49.103 INFO:journalctl@ceph.iscsi.iscsi.a.smithi191.stdout:Sep 10 19:10:48 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-iscsi-iscsi-a[61816]: debug there is no tcmu-runner data available 2024-09-10T19:10:49.321 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:10:49.325 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:10:49.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:10:49 smithi183 ceph-mon[29736]: Adjusting osd_memory_target on smithi191 to 3760M 2024-09-10T19:10:49.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:10:49 smithi183 ceph-mon[29736]: pgmap v6: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:10:49.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:10:49 smithi183 ceph-mon[29736]: from='mgr.14556 ' entity='mgr.y' 2024-09-10T19:10:49.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:10:49 smithi183 ceph-mon[29736]: from='mgr.14556 ' entity='mgr.y' 2024-09-10T19:10:49.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:10:49 smithi183 ceph-mon[29736]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-10T19:10:49.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:10:49 smithi183 ceph-mon[29736]: from='mgr.14556 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-10T19:10:49.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:10:49 smithi183 ceph-mon[29736]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-10T19:10:49.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:10:49 smithi183 ceph-mon[29736]: from='mgr.14556 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-10T19:10:49.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:10:49 smithi183 ceph-mon[29736]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-10T19:10:49.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:10:49 smithi183 ceph-mon[29736]: from='mgr.14556 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-10T19:10:49.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:10:49 smithi183 ceph-mon[29736]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-10T19:10:49.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:10:49 smithi183 ceph-mon[29736]: from='mgr.14556 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-10T19:10:49.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:10:49 smithi183 ceph-mon[29736]: Adjusting osd_memory_target on smithi183 to 3760M 2024-09-10T19:10:49.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:10:49 smithi183 ceph-mon[29736]: from='mgr.14556 ' entity='mgr.y' 2024-09-10T19:10:49.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:10:49 smithi183 ceph-mon[29736]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T19:10:49.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:10:49 smithi183 ceph-mon[29736]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T19:10:49.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:10:49 smithi183 ceph-mon[29736]: Updating smithi183:/etc/ceph/ceph.conf 2024-09-10T19:10:49.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:10:49 smithi183 ceph-mon[29736]: Updating smithi191:/etc/ceph/ceph.conf 2024-09-10T19:10:49.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:10:49 smithi183 ceph-mon[29736]: from='client.? 172.21.15.183:0/2459723025' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-10T19:10:49.433 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:10:49 smithi183 ceph-mon[29736]: mgrmap e22: y(active, since 4s), standbys: x 2024-09-10T19:10:49.433 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:10:49 smithi183 ceph-mon[29736]: from='mgr.14556 ' entity='mgr.y' 2024-09-10T19:10:49.433 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:10:49 smithi183 ceph-mon[29736]: from='mgr.14556 ' entity='mgr.y' 2024-09-10T19:10:49.433 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:10:49 smithi183 ceph-mon[41562]: Adjusting osd_memory_target on smithi191 to 3760M 2024-09-10T19:10:49.433 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:10:49 smithi183 ceph-mon[41562]: pgmap v6: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:10:49.433 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:10:49 smithi183 ceph-mon[41562]: from='mgr.14556 ' entity='mgr.y' 2024-09-10T19:10:49.433 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:10:49 smithi183 ceph-mon[41562]: from='mgr.14556 ' entity='mgr.y' 2024-09-10T19:10:49.433 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:10:49 smithi183 ceph-mon[41562]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-10T19:10:49.433 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:10:49 smithi183 ceph-mon[41562]: from='mgr.14556 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-10T19:10:49.433 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:10:49 smithi183 ceph-mon[41562]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-10T19:10:49.433 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:10:49 smithi183 ceph-mon[41562]: from='mgr.14556 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-10T19:10:49.433 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:10:49 smithi183 ceph-mon[41562]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-10T19:10:49.434 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:10:49 smithi183 ceph-mon[41562]: from='mgr.14556 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-10T19:10:49.434 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:10:49 smithi183 ceph-mon[41562]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-10T19:10:49.434 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:10:49 smithi183 ceph-mon[41562]: from='mgr.14556 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-10T19:10:49.434 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:10:49 smithi183 ceph-mon[41562]: Adjusting osd_memory_target on smithi183 to 3760M 2024-09-10T19:10:49.434 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:10:49 smithi183 ceph-mon[41562]: from='mgr.14556 ' entity='mgr.y' 2024-09-10T19:10:49.434 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:10:49 smithi183 ceph-mon[41562]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T19:10:49.434 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:10:49 smithi183 ceph-mon[41562]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T19:10:49.434 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:10:49 smithi183 ceph-mon[41562]: Updating smithi183:/etc/ceph/ceph.conf 2024-09-10T19:10:49.434 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:10:49 smithi183 ceph-mon[41562]: Updating smithi191:/etc/ceph/ceph.conf 2024-09-10T19:10:49.434 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:10:49 smithi183 ceph-mon[41562]: from='client.? 172.21.15.183:0/2459723025' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-10T19:10:49.434 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:10:49 smithi183 ceph-mon[41562]: mgrmap e22: y(active, since 4s), standbys: x 2024-09-10T19:10:49.434 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:10:49 smithi183 ceph-mon[41562]: from='mgr.14556 ' entity='mgr.y' 2024-09-10T19:10:49.434 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:10:49 smithi183 ceph-mon[41562]: from='mgr.14556 ' entity='mgr.y' 2024-09-10T19:10:49.465 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:10:49 smithi191 ceph-mon[32884]: Adjusting osd_memory_target on smithi191 to 3760M 2024-09-10T19:10:49.465 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:10:49 smithi191 ceph-mon[32884]: pgmap v6: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:10:49.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:10:49 smithi191 ceph-mon[32884]: from='mgr.14556 ' entity='mgr.y' 2024-09-10T19:10:49.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:10:49 smithi191 ceph-mon[32884]: from='mgr.14556 ' entity='mgr.y' 2024-09-10T19:10:49.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:10:49 smithi191 ceph-mon[32884]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-10T19:10:49.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:10:49 smithi191 ceph-mon[32884]: from='mgr.14556 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-10T19:10:49.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:10:49 smithi191 ceph-mon[32884]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-10T19:10:49.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:10:49 smithi191 ceph-mon[32884]: from='mgr.14556 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-10T19:10:49.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:10:49 smithi191 ceph-mon[32884]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-10T19:10:49.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:10:49 smithi191 ceph-mon[32884]: from='mgr.14556 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-10T19:10:49.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:10:49 smithi191 ceph-mon[32884]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-10T19:10:49.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:10:49 smithi191 ceph-mon[32884]: from='mgr.14556 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-10T19:10:49.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:10:49 smithi191 ceph-mon[32884]: Adjusting osd_memory_target on smithi183 to 3760M 2024-09-10T19:10:49.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:10:49 smithi191 ceph-mon[32884]: from='mgr.14556 ' entity='mgr.y' 2024-09-10T19:10:49.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:10:49 smithi191 ceph-mon[32884]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T19:10:49.467 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:10:49 smithi191 ceph-mon[32884]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T19:10:49.467 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:10:49 smithi191 ceph-mon[32884]: Updating smithi183:/etc/ceph/ceph.conf 2024-09-10T19:10:49.467 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:10:49 smithi191 ceph-mon[32884]: Updating smithi191:/etc/ceph/ceph.conf 2024-09-10T19:10:49.467 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:10:49 smithi191 ceph-mon[32884]: from='client.? 172.21.15.183:0/2459723025' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-10T19:10:49.467 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:10:49 smithi191 ceph-mon[32884]: mgrmap e22: y(active, since 4s), standbys: x 2024-09-10T19:10:49.467 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:10:49 smithi191 ceph-mon[32884]: from='mgr.14556 ' entity='mgr.y' 2024-09-10T19:10:49.467 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:10:49 smithi191 ceph-mon[32884]: from='mgr.14556 ' entity='mgr.y' 2024-09-10T19:10:50.332 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:10:50.333 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:10:50.333 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:10:50.365 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:10:50 smithi183 ceph-mon[29736]: Updating smithi183:/var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/config/ceph.conf 2024-09-10T19:10:50.365 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:10:50 smithi183 ceph-mon[29736]: Updating smithi191:/var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/config/ceph.conf 2024-09-10T19:10:50.365 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:10:50 smithi183 ceph-mon[29736]: Updating smithi183:/etc/ceph/ceph.client.admin.keyring 2024-09-10T19:10:50.365 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:10:50 smithi183 ceph-mon[29736]: Updating smithi191:/etc/ceph/ceph.client.admin.keyring 2024-09-10T19:10:50.365 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:10:50 smithi183 ceph-mon[29736]: Updating smithi183:/var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/config/ceph.client.admin.keyring 2024-09-10T19:10:50.365 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:10:50 smithi183 ceph-mon[29736]: Updating smithi191:/var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/config/ceph.client.admin.keyring 2024-09-10T19:10:50.365 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:10:50 smithi183 ceph-mon[29736]: from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:10:50.365 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:10:50 smithi183 ceph-mon[29736]: from='mgr.14556 ' entity='mgr.y' 2024-09-10T19:10:50.366 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:10:50 smithi183 ceph-mon[29736]: from='mgr.14556 ' entity='mgr.y' 2024-09-10T19:10:50.366 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:10:50 smithi183 ceph-mon[29736]: from='mgr.14556 ' entity='mgr.y' 2024-09-10T19:10:50.366 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:10:50 smithi183 ceph-mon[41562]: Updating smithi183:/var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/config/ceph.conf 2024-09-10T19:10:50.366 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:10:50 smithi183 ceph-mon[41562]: Updating smithi191:/var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/config/ceph.conf 2024-09-10T19:10:50.366 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:10:50 smithi183 ceph-mon[41562]: Updating smithi183:/etc/ceph/ceph.client.admin.keyring 2024-09-10T19:10:50.366 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:10:50 smithi183 ceph-mon[41562]: Updating smithi191:/etc/ceph/ceph.client.admin.keyring 2024-09-10T19:10:50.366 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:10:50 smithi183 ceph-mon[41562]: Updating smithi183:/var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/config/ceph.client.admin.keyring 2024-09-10T19:10:50.366 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:10:50 smithi183 ceph-mon[41562]: Updating smithi191:/var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/config/ceph.client.admin.keyring 2024-09-10T19:10:50.366 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:10:50 smithi183 ceph-mon[41562]: from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:10:50.367 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:10:50 smithi183 ceph-mon[41562]: from='mgr.14556 ' entity='mgr.y' 2024-09-10T19:10:50.367 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:10:50 smithi183 ceph-mon[41562]: from='mgr.14556 ' entity='mgr.y' 2024-09-10T19:10:50.367 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:10:50 smithi183 ceph-mon[41562]: from='mgr.14556 ' entity='mgr.y' 2024-09-10T19:10:50.465 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:10:50 smithi191 ceph-mon[32884]: Updating smithi183:/var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/config/ceph.conf 2024-09-10T19:10:50.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:10:50 smithi191 ceph-mon[32884]: Updating smithi191:/var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/config/ceph.conf 2024-09-10T19:10:50.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:10:50 smithi191 ceph-mon[32884]: Updating smithi183:/etc/ceph/ceph.client.admin.keyring 2024-09-10T19:10:50.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:10:50 smithi191 ceph-mon[32884]: Updating smithi191:/etc/ceph/ceph.client.admin.keyring 2024-09-10T19:10:50.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:10:50 smithi191 ceph-mon[32884]: Updating smithi183:/var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/config/ceph.client.admin.keyring 2024-09-10T19:10:50.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:10:50 smithi191 ceph-mon[32884]: Updating smithi191:/var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/config/ceph.client.admin.keyring 2024-09-10T19:10:50.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:10:50 smithi191 ceph-mon[32884]: from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:10:50.467 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:10:50 smithi191 ceph-mon[32884]: from='mgr.14556 ' entity='mgr.y' 2024-09-10T19:10:50.467 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:10:50 smithi191 ceph-mon[32884]: from='mgr.14556 ' entity='mgr.y' 2024-09-10T19:10:50.467 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:10:50 smithi191 ceph-mon[32884]: from='mgr.14556 ' entity='mgr.y' 2024-09-10T19:10:51.150 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:10:51 smithi183 ceph-mon[29736]: Deploying daemon alertmanager.a on smithi183 2024-09-10T19:10:51.151 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:10:51 smithi183 ceph-mon[29736]: pgmap v7: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:10:51.151 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:10:51 smithi183 ceph-mon[41562]: Deploying daemon alertmanager.a on smithi183 2024-09-10T19:10:51.151 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:10:51 smithi183 ceph-mon[41562]: pgmap v7: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:10:51.469 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:10:51 smithi191 ceph-mon[32884]: Deploying daemon alertmanager.a on smithi183 2024-09-10T19:10:51.469 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:10:51 smithi191 ceph-mon[32884]: pgmap v7: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:10:53.076 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:10:53.077 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:10:53.366 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:10:53 smithi183 ceph-mon[29736]: pgmap v8: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 26 KiB/s rd, 0 B/s wr, 11 op/s 2024-09-10T19:10:53.366 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:10:53 smithi183 ceph-mon[41562]: pgmap v8: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 26 KiB/s rd, 0 B/s wr, 11 op/s 2024-09-10T19:10:53.465 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:10:53 smithi191 ceph-mon[32884]: pgmap v8: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 26 KiB/s rd, 0 B/s wr, 11 op/s 2024-09-10T19:10:55.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:10:55 smithi183 ceph-mon[29736]: pgmap v9: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 8 op/s 2024-09-10T19:10:55.431 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:10:55 smithi183 ceph-mon[41562]: pgmap v9: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 8 op/s 2024-09-10T19:10:55.465 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:10:55 smithi191 ceph-mon[32884]: pgmap v9: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 8 op/s 2024-09-10T19:10:57.205 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:10:57 smithi183 ceph-mon[29736]: pgmap v10: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 17 KiB/s rd, 0 B/s wr, 7 op/s 2024-09-10T19:10:57.205 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:10:57 smithi183 ceph-mon[29736]: purged_snaps scrub starts 2024-09-10T19:10:57.205 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:10:57 smithi183 ceph-mon[29736]: purged_snaps scrub ok 2024-09-10T19:10:57.205 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:10:57 smithi183 ceph-mon[41562]: pgmap v10: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 17 KiB/s rd, 0 B/s wr, 7 op/s 2024-09-10T19:10:57.205 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:10:57 smithi183 ceph-mon[41562]: purged_snaps scrub starts 2024-09-10T19:10:57.206 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:10:57 smithi183 ceph-mon[41562]: purged_snaps scrub ok 2024-09-10T19:10:57.206 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:10:57.450 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:10:57 smithi191 ceph-mon[32884]: pgmap v10: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 17 KiB/s rd, 0 B/s wr, 7 op/s 2024-09-10T19:10:57.450 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:10:57 smithi191 ceph-mon[32884]: purged_snaps scrub starts 2024-09-10T19:10:57.450 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:10:57 smithi191 ceph-mon[32884]: purged_snaps scrub ok 2024-09-10T19:10:57.715 INFO:journalctl@ceph.mgr.x.smithi191.stdout:Sep 10 19:10:57 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-x[34967]: ::ffff:172.21.15.191 - - [10/Sep/2024:19:10:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T19:10:58.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:10:58 smithi183 ceph-mon[29736]: osdmap e75: 8 total, 8 up, 8 in 2024-09-10T19:10:58.431 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:10:58 smithi183 ceph-mon[41562]: osdmap e75: 8 total, 8 up, 8 in 2024-09-10T19:10:58.465 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:10:58 smithi191 ceph-mon[32884]: osdmap e75: 8 total, 8 up, 8 in 2024-09-10T19:10:59.123 INFO:journalctl@ceph.iscsi.iscsi.a.smithi191.stdout:Sep 10 19:10:58 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-iscsi-iscsi-a[61816]: debug there is no tcmu-runner data available 2024-09-10T19:10:59.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:10:59 smithi183 ceph-mon[29736]: pgmap v12: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 19 KiB/s rd, 0 B/s wr, 8 op/s 2024-09-10T19:10:59.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:10:59 smithi183 ceph-mon[29736]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:10:59.431 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:10:59 smithi183 ceph-mon[41562]: pgmap v12: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 19 KiB/s rd, 0 B/s wr, 8 op/s 2024-09-10T19:10:59.431 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:10:59 smithi183 ceph-mon[41562]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:10:59.465 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:10:59 smithi191 ceph-mon[32884]: pgmap v12: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 19 KiB/s rd, 0 B/s wr, 8 op/s 2024-09-10T19:10:59.465 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:10:59 smithi191 ceph-mon[32884]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:11:00.426 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:11:00 smithi183 ceph-mon[29736]: from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:11:00.426 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:11:00 smithi183 ceph-mon[29736]: osdmap e76: 8 total, 8 up, 8 in 2024-09-10T19:11:00.426 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:11:00 smithi183 ceph-mon[41562]: from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:11:00.426 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:11:00 smithi183 ceph-mon[41562]: osdmap e76: 8 total, 8 up, 8 in 2024-09-10T19:11:00.465 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:11:00 smithi191 ceph-mon[32884]: from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:11:00.465 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:11:00 smithi191 ceph-mon[32884]: osdmap e76: 8 total, 8 up, 8 in 2024-09-10T19:11:01.406 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:11:01 smithi183 ceph-mon[29736]: purged_snaps scrub starts 2024-09-10T19:11:01.407 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:11:01 smithi183 ceph-mon[29736]: purged_snaps scrub ok 2024-09-10T19:11:01.407 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:11:01 smithi183 ceph-mon[29736]: pgmap v13: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 19 KiB/s rd, 0 B/s wr, 8 op/s 2024-09-10T19:11:01.407 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:11:01 smithi183 ceph-mon[41562]: purged_snaps scrub starts 2024-09-10T19:11:01.407 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:11:01 smithi183 ceph-mon[41562]: purged_snaps scrub ok 2024-09-10T19:11:01.407 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:11:01 smithi183 ceph-mon[41562]: pgmap v13: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 19 KiB/s rd, 0 B/s wr, 8 op/s 2024-09-10T19:11:01.435 INFO:teuthology.orchestra.run.smithi183.stdout:201863462931 2024-09-10T19:11:01.435 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph osd last-stat-seq osd.6 2024-09-10T19:11:01.465 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:11:01 smithi191 ceph-mon[32884]: purged_snaps scrub starts 2024-09-10T19:11:01.465 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:11:01 smithi191 ceph-mon[32884]: purged_snaps scrub ok 2024-09-10T19:11:01.465 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:11:01 smithi191 ceph-mon[32884]: pgmap v13: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 19 KiB/s rd, 0 B/s wr, 8 op/s 2024-09-10T19:11:03.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:11:03 smithi183 ceph-mon[29736]: pgmap v15: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:11:03.431 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:11:03 smithi183 ceph-mon[41562]: pgmap v15: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:11:03.465 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:11:03 smithi191 ceph-mon[32884]: pgmap v15: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:11:05.032 INFO:teuthology.orchestra.run.smithi183.stdout:77309411367 2024-09-10T19:11:05.033 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph osd last-stat-seq osd.2 2024-09-10T19:11:05.271 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:11:05 smithi183 ceph-mon[29736]: pgmap v16: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-10T19:11:05.271 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:11:05 smithi183 ceph-mon[41562]: pgmap v16: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-10T19:11:05.465 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:11:05 smithi191 ceph-mon[32884]: pgmap v16: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-10T19:11:06.416 INFO:teuthology.orchestra.run.smithi183.stdout:231928233998 2024-09-10T19:11:06.416 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph osd last-stat-seq osd.7 2024-09-10T19:11:07.270 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:11:07 smithi183 ceph-mon[29736]: pgmap v17: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:11:07.271 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:11:07 smithi183 ceph-mon[41562]: pgmap v17: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:11:07.297 INFO:teuthology.orchestra.run.smithi183.stdout:55834574892 2024-09-10T19:11:07.298 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph osd last-stat-seq osd.1 2024-09-10T19:11:07.450 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:11:07 smithi191 ceph-mon[32884]: pgmap v17: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:11:07.711 INFO:teuthology.orchestra.run.smithi183.stdout:167503724570 2024-09-10T19:11:07.711 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph osd last-stat-seq osd.5 2024-09-10T19:11:07.715 INFO:journalctl@ceph.mgr.x.smithi191.stdout:Sep 10 19:11:07 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-x[34967]: ::ffff:172.21.15.191 - - [10/Sep/2024:19:11:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T19:11:08.061 INFO:teuthology.orchestra.run.smithi183.stdout:34359738418 2024-09-10T19:11:08.061 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph osd last-stat-seq osd.0 2024-09-10T19:11:08.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:11:08 smithi183 ceph-mon[29736]: pgmap v18: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-10T19:11:08.431 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:11:08 smithi183 ceph-mon[41562]: pgmap v18: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-10T19:11:08.465 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:11:08 smithi191 ceph-mon[32884]: pgmap v18: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-10T19:11:08.561 INFO:teuthology.orchestra.run.smithi183.stdout:137438953501 2024-09-10T19:11:08.562 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph osd last-stat-seq osd.4 2024-09-10T19:11:09.145 INFO:journalctl@ceph.iscsi.iscsi.a.smithi191.stdout:Sep 10 19:11:08 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-iscsi-iscsi-a[61816]: debug there is no tcmu-runner data available 2024-09-10T19:11:09.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:11:09 smithi183 ceph-mon[29736]: from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:11:09.431 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:11:09 smithi183 ceph-mon[41562]: from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:11:09.465 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:11:09 smithi191 ceph-mon[32884]: from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:11:10.299 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:11:10 smithi183 ceph-mon[29736]: pgmap v19: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-09-10T19:11:10.299 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:11:10 smithi183 ceph-mon[41562]: pgmap v19: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-09-10T19:11:10.465 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:11:10 smithi191 ceph-mon[32884]: pgmap v19: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-09-10T19:11:10.746 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:11:10.746 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:11:11.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:11:11 smithi183 ceph-mon[29736]: 2.15 deep-scrub starts 2024-09-10T19:11:11.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:11:11 smithi183 ceph-mon[29736]: 2.15 deep-scrub ok 2024-09-10T19:11:11.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:11:11 smithi183 ceph-mon[41562]: 2.15 deep-scrub starts 2024-09-10T19:11:11.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:11:11 smithi183 ceph-mon[41562]: 2.15 deep-scrub ok 2024-09-10T19:11:11.465 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:11:11 smithi191 ceph-mon[32884]: 2.15 deep-scrub starts 2024-09-10T19:11:11.465 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:11:11 smithi191 ceph-mon[32884]: 2.15 deep-scrub ok 2024-09-10T19:11:12.391 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:11:12 smithi183 ceph-mon[29736]: 2.9 scrub starts 2024-09-10T19:11:12.391 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:11:12 smithi183 ceph-mon[29736]: 2.9 scrub ok 2024-09-10T19:11:12.391 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:11:12 smithi183 ceph-mon[29736]: pgmap v20: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 856 B/s rd, 0 op/s 2024-09-10T19:11:12.391 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:11:12 smithi183 ceph-mon[41562]: 2.9 scrub starts 2024-09-10T19:11:12.392 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:11:12 smithi183 ceph-mon[41562]: 2.9 scrub ok 2024-09-10T19:11:12.392 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:11:12 smithi183 ceph-mon[41562]: pgmap v20: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 856 B/s rd, 0 op/s 2024-09-10T19:11:12.465 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:11:12 smithi191 ceph-mon[32884]: 2.9 scrub starts 2024-09-10T19:11:12.465 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:11:12 smithi191 ceph-mon[32884]: 2.9 scrub ok 2024-09-10T19:11:12.465 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:11:12 smithi191 ceph-mon[32884]: pgmap v20: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 856 B/s rd, 0 op/s 2024-09-10T19:11:13.159 INFO:journalctl@ceph.alertmanager.a.smithi183.stdout:Sep 10 19:11:12 smithi183 podman[76822]: 2024-09-10 19:11:12.959038317 +0000 UTC m=+0.182888911 container create 17002da0f9cee8dc2b9a40d67c12f814adebfa79bafcb6177550784f70b8327a (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-09-10T19:11:13.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:11:13 smithi183 ceph-mon[29736]: 4.16 scrub starts 2024-09-10T19:11:13.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:11:13 smithi183 ceph-mon[29736]: 4.16 scrub ok 2024-09-10T19:11:13.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:11:13 smithi183 ceph-mon[29736]: 2.12 scrub starts 2024-09-10T19:11:13.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:11:13 smithi183 ceph-mon[29736]: 2.12 scrub ok 2024-09-10T19:11:13.431 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:11:13 smithi183 ceph-mon[41562]: 4.16 scrub starts 2024-09-10T19:11:13.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:11:13 smithi183 ceph-mon[41562]: 4.16 scrub ok 2024-09-10T19:11:13.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:11:13 smithi183 ceph-mon[41562]: 2.12 scrub starts 2024-09-10T19:11:13.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:11:13 smithi183 ceph-mon[41562]: 2.12 scrub ok 2024-09-10T19:11:13.465 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:11:13 smithi191 ceph-mon[32884]: 4.16 scrub starts 2024-09-10T19:11:13.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:11:13 smithi191 ceph-mon[32884]: 4.16 scrub ok 2024-09-10T19:11:13.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:11:13 smithi191 ceph-mon[32884]: 2.12 scrub starts 2024-09-10T19:11:13.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:11:13 smithi191 ceph-mon[32884]: 2.12 scrub ok 2024-09-10T19:11:14.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:11:14 smithi183 ceph-mon[29736]: 6.18 scrub starts 2024-09-10T19:11:14.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:11:14 smithi183 ceph-mon[29736]: 6.18 scrub ok 2024-09-10T19:11:14.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:11:14 smithi183 ceph-mon[29736]: pgmap v21: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-10T19:11:14.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:11:14 smithi183 ceph-mon[29736]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:11:14.181 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:11:14 smithi183 ceph-mon[41562]: 6.18 scrub starts 2024-09-10T19:11:14.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:11:14 smithi183 ceph-mon[41562]: 6.18 scrub ok 2024-09-10T19:11:14.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:11:14 smithi183 ceph-mon[41562]: pgmap v21: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-10T19:11:14.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:11:14 smithi183 ceph-mon[41562]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:11:14.465 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:11:14 smithi191 ceph-mon[32884]: 6.18 scrub starts 2024-09-10T19:11:14.465 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:11:14 smithi191 ceph-mon[32884]: 6.18 scrub ok 2024-09-10T19:11:14.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:11:14 smithi191 ceph-mon[32884]: pgmap v21: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-10T19:11:14.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:11:14 smithi191 ceph-mon[32884]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:11:14.496 INFO:teuthology.orchestra.run.smithi183.stdout:111669149732 2024-09-10T19:11:14.496 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph osd last-stat-seq osd.3 2024-09-10T19:11:14.854 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:11:14.855 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:11:15.098 INFO:teuthology.orchestra.run.smithi183.stdout:201863462934 2024-09-10T19:11:15.368 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:11:15 smithi183 ceph-mon[29736]: 6.15 scrub starts 2024-09-10T19:11:15.368 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:11:15 smithi183 ceph-mon[29736]: 6.15 scrub ok 2024-09-10T19:11:15.368 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:11:15 smithi183 ceph-mon[29736]: 4.d scrub starts 2024-09-10T19:11:15.368 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:11:15 smithi183 ceph-mon[29736]: 4.d scrub ok 2024-09-10T19:11:15.368 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:11:15 smithi183 ceph-mon[29736]: 5.5 scrub starts 2024-09-10T19:11:15.368 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:11:15 smithi183 ceph-mon[29736]: 5.5 scrub ok 2024-09-10T19:11:15.368 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:11:15 smithi183 ceph-mon[29736]: from='client.? 172.21.15.183:0/3541568489' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-09-10T19:11:15.369 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:11:15 smithi183 ceph-mon[41562]: 6.15 scrub starts 2024-09-10T19:11:15.369 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:11:15 smithi183 ceph-mon[41562]: 6.15 scrub ok 2024-09-10T19:11:15.369 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:11:15 smithi183 ceph-mon[41562]: 4.d scrub starts 2024-09-10T19:11:15.369 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:11:15 smithi183 ceph-mon[41562]: 4.d scrub ok 2024-09-10T19:11:15.369 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:11:15 smithi183 ceph-mon[41562]: 5.5 scrub starts 2024-09-10T19:11:15.369 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:11:15 smithi183 ceph-mon[41562]: 5.5 scrub ok 2024-09-10T19:11:15.369 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:11:15 smithi183 ceph-mon[41562]: from='client.? 172.21.15.183:0/3541568489' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-09-10T19:11:15.384 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:11:15.385 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:11:15.465 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:11:15 smithi191 ceph-mon[32884]: 6.15 scrub starts 2024-09-10T19:11:15.465 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:11:15 smithi191 ceph-mon[32884]: 6.15 scrub ok 2024-09-10T19:11:15.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:11:15 smithi191 ceph-mon[32884]: 4.d scrub starts 2024-09-10T19:11:15.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:11:15 smithi191 ceph-mon[32884]: 4.d scrub ok 2024-09-10T19:11:15.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:11:15 smithi191 ceph-mon[32884]: 5.5 scrub starts 2024-09-10T19:11:15.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:11:15 smithi191 ceph-mon[32884]: 5.5 scrub ok 2024-09-10T19:11:15.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:11:15 smithi191 ceph-mon[32884]: from='client.? 172.21.15.183:0/3541568489' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-09-10T19:11:16.034 INFO:journalctl@ceph.alertmanager.a.smithi183.stdout:Sep 10 19:11:15 smithi183 podman[76822]: 2024-09-10 19:11:15.723466139 +0000 UTC m=+2.947316690 container init 17002da0f9cee8dc2b9a40d67c12f814adebfa79bafcb6177550784f70b8327a (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-09-10T19:11:16.034 INFO:journalctl@ceph.alertmanager.a.smithi183.stdout:Sep 10 19:11:15 smithi183 podman[76822]: 2024-09-10 19:11:15.729809765 +0000 UTC m=+2.953660323 container start 17002da0f9cee8dc2b9a40d67c12f814adebfa79bafcb6177550784f70b8327a (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-09-10T19:11:16.034 INFO:journalctl@ceph.alertmanager.a.smithi183.stdout:Sep 10 19:11:15 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-alertmanager-a[76837]: ts=2024-09-10T19:11:15.741Z caller=main.go:240 level=info msg="Starting Alertmanager" version="(version=0.25.0, branch=HEAD, revision=258fab7cdd551f2cf251ed0348f0ad7289aee789)" 2024-09-10T19:11:16.034 INFO:journalctl@ceph.alertmanager.a.smithi183.stdout:Sep 10 19:11:15 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-alertmanager-a[76837]: ts=2024-09-10T19:11:15.741Z caller=main.go:241 level=info build_context="(go=go1.19.4, user=root@abe866dd5717, date=20221222-14:51:36)" 2024-09-10T19:11:16.034 INFO:journalctl@ceph.alertmanager.a.smithi183.stdout:Sep 10 19:11:15 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-alertmanager-a[76837]: ts=2024-09-10T19:11:15.743Z caller=cluster.go:185 level=info component=cluster msg="setting advertise address explicitly" addr=172.21.15.183 port=9094 2024-09-10T19:11:16.034 INFO:journalctl@ceph.alertmanager.a.smithi183.stdout:Sep 10 19:11:15 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-alertmanager-a[76837]: ts=2024-09-10T19:11:15.745Z caller=cluster.go:681 level=info component=cluster msg="Waiting for gossip to settle..." interval=2s 2024-09-10T19:11:16.035 INFO:journalctl@ceph.alertmanager.a.smithi183.stdout:Sep 10 19:11:15 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-alertmanager-a[76837]: ts=2024-09-10T19:11:15.773Z caller=coordinator.go:113 level=info component=configuration msg="Loading configuration file" file=/etc/alertmanager/alertmanager.yml 2024-09-10T19:11:16.035 INFO:journalctl@ceph.alertmanager.a.smithi183.stdout:Sep 10 19:11:15 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-alertmanager-a[76837]: ts=2024-09-10T19:11:15.773Z caller=coordinator.go:126 level=info component=configuration msg="Completed loading of configuration file" file=/etc/alertmanager/alertmanager.yml 2024-09-10T19:11:16.035 INFO:journalctl@ceph.alertmanager.a.smithi183.stdout:Sep 10 19:11:15 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-alertmanager-a[76837]: ts=2024-09-10T19:11:15.776Z caller=tls_config.go:232 level=info msg="Listening on" address=[::]:9093 2024-09-10T19:11:16.035 INFO:journalctl@ceph.alertmanager.a.smithi183.stdout:Sep 10 19:11:15 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-alertmanager-a[76837]: ts=2024-09-10T19:11:15.776Z caller=tls_config.go:235 level=info msg="TLS is disabled." http2=false address=[::]:9093 2024-09-10T19:11:16.035 INFO:journalctl@ceph.alertmanager.a.smithi183.stdout:Sep 10 19:11:15 smithi183 bash[76822]: 17002da0f9cee8dc2b9a40d67c12f814adebfa79bafcb6177550784f70b8327a 2024-09-10T19:11:16.035 INFO:journalctl@ceph.alertmanager.a.smithi183.stdout:Sep 10 19:11:15 smithi183 systemd[1]: Started Ceph alertmanager.a for 320f6df8-6fa7-11ef-bcea-c7b262605968. 2024-09-10T19:11:16.295 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Sep 10 19:11:16 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-y[29989]: [10/Sep/2024:19:11:16] ENGINE Bus STOPPING 2024-09-10T19:11:16.295 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Sep 10 19:11:16 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-y[29989]: [10/Sep/2024:19:11:16] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-09-10T19:11:16.295 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:11:16 smithi183 ceph-mon[41562]: 6.12 scrub starts 2024-09-10T19:11:16.295 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:11:16 smithi183 ceph-mon[41562]: 6.12 scrub ok 2024-09-10T19:11:16.296 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:11:16 smithi183 ceph-mon[41562]: 4.10 scrub starts 2024-09-10T19:11:16.296 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:11:16 smithi183 ceph-mon[41562]: 4.10 scrub ok 2024-09-10T19:11:16.296 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:11:16 smithi183 ceph-mon[41562]: 4.c scrub starts 2024-09-10T19:11:16.296 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:11:16 smithi183 ceph-mon[41562]: 4.c scrub ok 2024-09-10T19:11:16.296 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:11:16 smithi183 ceph-mon[41562]: 2.e scrub starts 2024-09-10T19:11:16.297 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:11:16 smithi183 ceph-mon[41562]: 2.e scrub ok 2024-09-10T19:11:16.297 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:11:16 smithi183 ceph-mon[41562]: pgmap v22: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:11:16.297 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:11:16 smithi183 ceph-mon[41562]: from='mgr.14556 ' entity='mgr.y' 2024-09-10T19:11:16.297 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:11:16 smithi183 ceph-mon[41562]: from='mgr.14556 ' entity='mgr.y' 2024-09-10T19:11:16.297 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:11:16 smithi183 ceph-mon[41562]: from='mgr.14556 ' entity='mgr.y' 2024-09-10T19:11:16.297 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:11:16 smithi183 ceph-mon[41562]: from='mgr.14556 ' entity='mgr.y' 2024-09-10T19:11:16.297 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:11:16 smithi183 ceph-mon[41562]: from='mgr.14556 ' entity='mgr.y' 2024-09-10T19:11:16.298 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:11:16 smithi183 ceph-mon[41562]: from='mgr.14556 ' entity='mgr.y' 2024-09-10T19:11:16.298 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:11:16 smithi183 ceph-mon[41562]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-09-10T19:11:16.298 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:11:16 smithi183 ceph-mon[41562]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-09-10T19:11:16.298 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:11:16 smithi183 ceph-mon[41562]: from='mgr.14556 ' entity='mgr.y' 2024-09-10T19:11:16.298 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:11:16 smithi183 ceph-mon[41562]: Deploying daemon grafana.a on smithi191 2024-09-10T19:11:16.299 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:11:16 smithi183 ceph-mon[29736]: 6.12 scrub starts 2024-09-10T19:11:16.299 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:11:16 smithi183 ceph-mon[29736]: 6.12 scrub ok 2024-09-10T19:11:16.299 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:11:16 smithi183 ceph-mon[29736]: 4.10 scrub starts 2024-09-10T19:11:16.299 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:11:16 smithi183 ceph-mon[29736]: 4.10 scrub ok 2024-09-10T19:11:16.299 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:11:16 smithi183 ceph-mon[29736]: 4.c scrub starts 2024-09-10T19:11:16.299 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:11:16 smithi183 ceph-mon[29736]: 4.c scrub ok 2024-09-10T19:11:16.299 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:11:16 smithi183 ceph-mon[29736]: 2.e scrub starts 2024-09-10T19:11:16.299 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:11:16 smithi183 ceph-mon[29736]: 2.e scrub ok 2024-09-10T19:11:16.300 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:11:16 smithi183 ceph-mon[29736]: pgmap v22: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:11:16.300 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:11:16 smithi183 ceph-mon[29736]: from='mgr.14556 ' entity='mgr.y' 2024-09-10T19:11:16.300 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:11:16 smithi183 ceph-mon[29736]: from='mgr.14556 ' entity='mgr.y' 2024-09-10T19:11:16.300 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:11:16 smithi183 ceph-mon[29736]: from='mgr.14556 ' entity='mgr.y' 2024-09-10T19:11:16.300 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:11:16 smithi183 ceph-mon[29736]: from='mgr.14556 ' entity='mgr.y' 2024-09-10T19:11:16.300 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:11:16 smithi183 ceph-mon[29736]: from='mgr.14556 ' entity='mgr.y' 2024-09-10T19:11:16.300 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:11:16 smithi183 ceph-mon[29736]: from='mgr.14556 ' entity='mgr.y' 2024-09-10T19:11:16.301 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:11:16 smithi183 ceph-mon[29736]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-09-10T19:11:16.301 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:11:16 smithi183 ceph-mon[29736]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-09-10T19:11:16.301 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:11:16 smithi183 ceph-mon[29736]: from='mgr.14556 ' entity='mgr.y' 2024-09-10T19:11:16.301 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:11:16 smithi183 ceph-mon[29736]: Deploying daemon grafana.a on smithi191 2024-09-10T19:11:16.363 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:11:16.465 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:11:16 smithi191 ceph-mon[32884]: 6.12 scrub starts 2024-09-10T19:11:16.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:11:16 smithi191 ceph-mon[32884]: 6.12 scrub ok 2024-09-10T19:11:16.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:11:16 smithi191 ceph-mon[32884]: 4.10 scrub starts 2024-09-10T19:11:16.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:11:16 smithi191 ceph-mon[32884]: 4.10 scrub ok 2024-09-10T19:11:16.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:11:16 smithi191 ceph-mon[32884]: 4.c scrub starts 2024-09-10T19:11:16.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:11:16 smithi191 ceph-mon[32884]: 4.c scrub ok 2024-09-10T19:11:16.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:11:16 smithi191 ceph-mon[32884]: 2.e scrub starts 2024-09-10T19:11:16.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:11:16 smithi191 ceph-mon[32884]: 2.e scrub ok 2024-09-10T19:11:16.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:11:16 smithi191 ceph-mon[32884]: pgmap v22: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:11:16.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:11:16 smithi191 ceph-mon[32884]: from='mgr.14556 ' entity='mgr.y' 2024-09-10T19:11:16.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:11:16 smithi191 ceph-mon[32884]: from='mgr.14556 ' entity='mgr.y' 2024-09-10T19:11:16.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:11:16 smithi191 ceph-mon[32884]: from='mgr.14556 ' entity='mgr.y' 2024-09-10T19:11:16.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:11:16 smithi191 ceph-mon[32884]: from='mgr.14556 ' entity='mgr.y' 2024-09-10T19:11:16.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:11:16 smithi191 ceph-mon[32884]: from='mgr.14556 ' entity='mgr.y' 2024-09-10T19:11:16.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:11:16 smithi191 ceph-mon[32884]: from='mgr.14556 ' entity='mgr.y' 2024-09-10T19:11:16.467 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:11:16 smithi191 ceph-mon[32884]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-09-10T19:11:16.467 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:11:16 smithi191 ceph-mon[32884]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-09-10T19:11:16.467 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:11:16 smithi191 ceph-mon[32884]: from='mgr.14556 ' entity='mgr.y' 2024-09-10T19:11:16.467 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:11:16 smithi191 ceph-mon[32884]: Deploying daemon grafana.a on smithi191 2024-09-10T19:11:16.566 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Sep 10 19:11:16 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-y[29989]: [10/Sep/2024:19:11:16] ENGINE Bus STOPPED 2024-09-10T19:11:16.566 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Sep 10 19:11:16 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-y[29989]: [10/Sep/2024:19:11:16] ENGINE Bus STARTING 2024-09-10T19:11:16.566 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Sep 10 19:11:16 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-y[29989]: [10/Sep/2024:19:11:16] ENGINE Serving on http://:::9283 2024-09-10T19:11:16.566 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Sep 10 19:11:16 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-y[29989]: [10/Sep/2024:19:11:16] ENGINE Bus STARTED 2024-09-10T19:11:16.798 INFO:tasks.cephadm.ceph_manager.ceph:need seq 201863462931 got 201863462934 for osd.6 2024-09-10T19:11:16.799 DEBUG:teuthology.parallel:result is None 2024-09-10T19:11:17.430 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:11:17 smithi183 ceph-mon[29736]: purged_snaps scrub starts 2024-09-10T19:11:17.430 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:11:17 smithi183 ceph-mon[29736]: purged_snaps scrub ok 2024-09-10T19:11:17.430 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:11:17 smithi183 ceph-mon[29736]: 2.1e scrub starts 2024-09-10T19:11:17.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:11:17 smithi183 ceph-mon[29736]: 2.1e scrub ok 2024-09-10T19:11:17.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:11:17 smithi183 ceph-mon[29736]: 3.0 scrub starts 2024-09-10T19:11:17.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:11:17 smithi183 ceph-mon[29736]: 3.0 scrub ok 2024-09-10T19:11:17.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:11:17 smithi183 ceph-mon[29736]: 5.7 scrub starts 2024-09-10T19:11:17.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:11:17 smithi183 ceph-mon[29736]: 5.7 scrub ok 2024-09-10T19:11:17.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:11:17 smithi183 ceph-mon[29736]: osdmap e77: 8 total, 8 up, 8 in 2024-09-10T19:11:17.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:11:17 smithi183 ceph-mon[41562]: purged_snaps scrub starts 2024-09-10T19:11:17.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:11:17 smithi183 ceph-mon[41562]: purged_snaps scrub ok 2024-09-10T19:11:17.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:11:17 smithi183 ceph-mon[41562]: 2.1e scrub starts 2024-09-10T19:11:17.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:11:17 smithi183 ceph-mon[41562]: 2.1e scrub ok 2024-09-10T19:11:17.433 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:11:17 smithi183 ceph-mon[41562]: 3.0 scrub starts 2024-09-10T19:11:17.433 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:11:17 smithi183 ceph-mon[41562]: 3.0 scrub ok 2024-09-10T19:11:17.433 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:11:17 smithi183 ceph-mon[41562]: 5.7 scrub starts 2024-09-10T19:11:17.433 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:11:17 smithi183 ceph-mon[41562]: 5.7 scrub ok 2024-09-10T19:11:17.433 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:11:17 smithi183 ceph-mon[41562]: osdmap e77: 8 total, 8 up, 8 in 2024-09-10T19:11:17.450 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:11:17 smithi191 ceph-mon[32884]: purged_snaps scrub starts 2024-09-10T19:11:17.450 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:11:17 smithi191 ceph-mon[32884]: purged_snaps scrub ok 2024-09-10T19:11:17.450 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:11:17 smithi191 ceph-mon[32884]: 2.1e scrub starts 2024-09-10T19:11:17.451 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:11:17 smithi191 ceph-mon[32884]: 2.1e scrub ok 2024-09-10T19:11:17.451 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:11:17 smithi191 ceph-mon[32884]: 3.0 scrub starts 2024-09-10T19:11:17.451 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:11:17 smithi191 ceph-mon[32884]: 3.0 scrub ok 2024-09-10T19:11:17.451 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:11:17 smithi191 ceph-mon[32884]: 5.7 scrub starts 2024-09-10T19:11:17.451 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:11:17 smithi191 ceph-mon[32884]: 5.7 scrub ok 2024-09-10T19:11:17.451 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:11:17 smithi191 ceph-mon[32884]: osdmap e77: 8 total, 8 up, 8 in 2024-09-10T19:11:17.715 INFO:journalctl@ceph.mgr.x.smithi191.stdout:Sep 10 19:11:17 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-x[34967]: ::ffff:172.21.15.191 - - [10/Sep/2024:19:11:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T19:11:18.117 INFO:journalctl@ceph.alertmanager.a.smithi183.stdout:Sep 10 19:11:17 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-alertmanager-a[76837]: ts=2024-09-10T19:11:17.745Z caller=cluster.go:706 level=info component=cluster msg="gossip not settled" polls=0 before=0 now=1 elapsed=2.000196049s 2024-09-10T19:11:18.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:11:18 smithi183 ceph-mon[29736]: 6.11 scrub starts 2024-09-10T19:11:18.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:11:18 smithi183 ceph-mon[29736]: 6.11 scrub ok 2024-09-10T19:11:18.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:11:18 smithi183 ceph-mon[29736]: 4.1 scrub starts 2024-09-10T19:11:18.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:11:18 smithi183 ceph-mon[29736]: 4.1 scrub ok 2024-09-10T19:11:18.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:11:18 smithi183 ceph-mon[29736]: 4.4 scrub starts 2024-09-10T19:11:18.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:11:18 smithi183 ceph-mon[29736]: 4.4 scrub ok 2024-09-10T19:11:18.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:11:18 smithi183 ceph-mon[29736]: 2.3 scrub starts 2024-09-10T19:11:18.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:11:18 smithi183 ceph-mon[29736]: 2.3 scrub ok 2024-09-10T19:11:18.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:11:18 smithi183 ceph-mon[29736]: pgmap v24: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-10T19:11:18.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:11:18 smithi183 ceph-mon[29736]: 6.9 scrub starts 2024-09-10T19:11:18.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:11:18 smithi183 ceph-mon[29736]: 6.9 scrub ok 2024-09-10T19:11:18.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:11:18 smithi183 ceph-mon[41562]: 6.11 scrub starts 2024-09-10T19:11:18.433 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:11:18 smithi183 ceph-mon[41562]: 6.11 scrub ok 2024-09-10T19:11:18.433 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:11:18 smithi183 ceph-mon[41562]: 4.1 scrub starts 2024-09-10T19:11:18.433 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:11:18 smithi183 ceph-mon[41562]: 4.1 scrub ok 2024-09-10T19:11:18.433 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:11:18 smithi183 ceph-mon[41562]: 4.4 scrub starts 2024-09-10T19:11:18.433 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:11:18 smithi183 ceph-mon[41562]: 4.4 scrub ok 2024-09-10T19:11:18.433 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:11:18 smithi183 ceph-mon[41562]: 2.3 scrub starts 2024-09-10T19:11:18.433 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:11:18 smithi183 ceph-mon[41562]: 2.3 scrub ok 2024-09-10T19:11:18.433 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:11:18 smithi183 ceph-mon[41562]: pgmap v24: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-10T19:11:18.433 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:11:18 smithi183 ceph-mon[41562]: 6.9 scrub starts 2024-09-10T19:11:18.433 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:11:18 smithi183 ceph-mon[41562]: 6.9 scrub ok 2024-09-10T19:11:18.465 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:11:18 smithi191 ceph-mon[32884]: 6.11 scrub starts 2024-09-10T19:11:18.465 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:11:18 smithi191 ceph-mon[32884]: 6.11 scrub ok 2024-09-10T19:11:18.465 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:11:18 smithi191 ceph-mon[32884]: 4.1 scrub starts 2024-09-10T19:11:18.465 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:11:18 smithi191 ceph-mon[32884]: 4.1 scrub ok 2024-09-10T19:11:18.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:11:18 smithi191 ceph-mon[32884]: 4.4 scrub starts 2024-09-10T19:11:18.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:11:18 smithi191 ceph-mon[32884]: 4.4 scrub ok 2024-09-10T19:11:18.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:11:18 smithi191 ceph-mon[32884]: 2.3 scrub starts 2024-09-10T19:11:18.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:11:18 smithi191 ceph-mon[32884]: 2.3 scrub ok 2024-09-10T19:11:18.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:11:18 smithi191 ceph-mon[32884]: pgmap v24: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-10T19:11:18.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:11:18 smithi191 ceph-mon[32884]: 6.9 scrub starts 2024-09-10T19:11:18.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:11:18 smithi191 ceph-mon[32884]: 6.9 scrub ok 2024-09-10T19:11:19.181 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:11:19 smithi191 ceph-mon[32884]: 6.d scrub starts 2024-09-10T19:11:19.181 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:11:19 smithi191 ceph-mon[32884]: 6.d scrub ok 2024-09-10T19:11:19.181 INFO:journalctl@ceph.iscsi.iscsi.a.smithi191.stdout:Sep 10 19:11:18 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-iscsi-iscsi-a[61816]: debug there is no tcmu-runner data available 2024-09-10T19:11:19.326 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:11:19 smithi183 ceph-mon[29736]: 6.d scrub starts 2024-09-10T19:11:19.326 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:11:19 smithi183 ceph-mon[29736]: 6.d scrub ok 2024-09-10T19:11:19.326 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:11:19 smithi183 ceph-mon[29736]: from='mgr.14556 ' entity='mgr.y' 2024-09-10T19:11:19.326 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:11:19 smithi183 ceph-mon[29736]: from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:11:19.326 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:11:19 smithi183 ceph-mon[41562]: 6.d scrub starts 2024-09-10T19:11:19.326 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:11:19 smithi183 ceph-mon[41562]: 6.d scrub ok 2024-09-10T19:11:19.327 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:11:19 smithi183 ceph-mon[41562]: from='mgr.14556 ' entity='mgr.y' 2024-09-10T19:11:19.327 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:11:19 smithi183 ceph-mon[41562]: from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:11:19.465 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:11:19 smithi191 ceph-mon[32884]: from='mgr.14556 ' entity='mgr.y' 2024-09-10T19:11:19.465 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:11:19 smithi191 ceph-mon[32884]: from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:11:20.461 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:11:20 smithi183 ceph-mon[29736]: purged_snaps scrub starts 2024-09-10T19:11:20.461 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:11:20 smithi183 ceph-mon[29736]: purged_snaps scrub ok 2024-09-10T19:11:20.461 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:11:20 smithi183 ceph-mon[29736]: 5.b scrub starts 2024-09-10T19:11:20.461 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:11:20 smithi183 ceph-mon[29736]: 5.b scrub ok 2024-09-10T19:11:20.461 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:11:20 smithi183 ceph-mon[29736]: pgmap v25: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-10T19:11:20.462 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:11:20 smithi183 ceph-mon[29736]: osdmap e78: 8 total, 8 up, 8 in 2024-09-10T19:11:20.462 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:11:20 smithi183 ceph-mon[41562]: purged_snaps scrub starts 2024-09-10T19:11:20.462 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:11:20 smithi183 ceph-mon[41562]: purged_snaps scrub ok 2024-09-10T19:11:20.462 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:11:20 smithi183 ceph-mon[41562]: 5.b scrub starts 2024-09-10T19:11:20.462 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:11:20 smithi183 ceph-mon[41562]: 5.b scrub ok 2024-09-10T19:11:20.462 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:11:20 smithi183 ceph-mon[41562]: pgmap v25: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-10T19:11:20.462 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:11:20 smithi183 ceph-mon[41562]: osdmap e78: 8 total, 8 up, 8 in 2024-09-10T19:11:20.465 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:11:20 smithi191 ceph-mon[32884]: purged_snaps scrub starts 2024-09-10T19:11:20.465 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:11:20 smithi191 ceph-mon[32884]: purged_snaps scrub ok 2024-09-10T19:11:20.465 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:11:20 smithi191 ceph-mon[32884]: 5.b scrub starts 2024-09-10T19:11:20.465 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:11:20 smithi191 ceph-mon[32884]: 5.b scrub ok 2024-09-10T19:11:20.465 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:11:20 smithi191 ceph-mon[32884]: pgmap v25: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-10T19:11:20.465 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:11:20 smithi191 ceph-mon[32884]: osdmap e78: 8 total, 8 up, 8 in 2024-09-10T19:11:20.540 INFO:teuthology.orchestra.run.smithi183.stdout:55834574896 2024-09-10T19:11:20.708 INFO:teuthology.orchestra.run.smithi183.stdout:231928234003 2024-09-10T19:11:21.293 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:11:21 smithi183 ceph-mon[29736]: 2.1d scrub starts 2024-09-10T19:11:21.293 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:11:21 smithi183 ceph-mon[29736]: 2.1d scrub ok 2024-09-10T19:11:21.293 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:11:21 smithi183 ceph-mon[29736]: 4.0 scrub starts 2024-09-10T19:11:21.293 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:11:21 smithi183 ceph-mon[29736]: 4.0 scrub ok 2024-09-10T19:11:21.294 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:11:21 smithi183 ceph-mon[29736]: 6.a scrub starts 2024-09-10T19:11:21.294 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:11:21 smithi183 ceph-mon[29736]: 6.a scrub ok 2024-09-10T19:11:21.294 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:11:21 smithi183 ceph-mon[29736]: from='client.? 172.21.15.183:0/2051479436' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-09-10T19:11:21.294 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:11:21 smithi183 ceph-mon[29736]: from='client.? 172.21.15.183:0/1013423769' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-09-10T19:11:21.294 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:11:21 smithi183 ceph-mon[29736]: 6.16 scrub starts 2024-09-10T19:11:21.294 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:11:21 smithi183 ceph-mon[29736]: 6.16 scrub ok 2024-09-10T19:11:21.294 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:11:21 smithi183 ceph-mon[41562]: 2.1d scrub starts 2024-09-10T19:11:21.294 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:11:21 smithi183 ceph-mon[41562]: 2.1d scrub ok 2024-09-10T19:11:21.294 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:11:21 smithi183 ceph-mon[41562]: 4.0 scrub starts 2024-09-10T19:11:21.294 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:11:21 smithi183 ceph-mon[41562]: 4.0 scrub ok 2024-09-10T19:11:21.294 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:11:21 smithi183 ceph-mon[41562]: 6.a scrub starts 2024-09-10T19:11:21.294 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:11:21 smithi183 ceph-mon[41562]: 6.a scrub ok 2024-09-10T19:11:21.295 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:11:21 smithi183 ceph-mon[41562]: from='client.? 172.21.15.183:0/2051479436' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-09-10T19:11:21.295 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:11:21 smithi183 ceph-mon[41562]: from='client.? 172.21.15.183:0/1013423769' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-09-10T19:11:21.295 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:11:21 smithi183 ceph-mon[41562]: 6.16 scrub starts 2024-09-10T19:11:21.295 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:11:21 smithi183 ceph-mon[41562]: 6.16 scrub ok 2024-09-10T19:11:21.314 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:11:21.465 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:11:21 smithi191 ceph-mon[32884]: 2.1d scrub starts 2024-09-10T19:11:21.465 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:11:21 smithi191 ceph-mon[32884]: 2.1d scrub ok 2024-09-10T19:11:21.465 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:11:21 smithi191 ceph-mon[32884]: 4.0 scrub starts 2024-09-10T19:11:21.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:11:21 smithi191 ceph-mon[32884]: 4.0 scrub ok 2024-09-10T19:11:21.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:11:21 smithi191 ceph-mon[32884]: 6.a scrub starts 2024-09-10T19:11:21.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:11:21 smithi191 ceph-mon[32884]: 6.a scrub ok 2024-09-10T19:11:21.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:11:21 smithi191 ceph-mon[32884]: from='client.? 172.21.15.183:0/2051479436' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-09-10T19:11:21.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:11:21 smithi191 ceph-mon[32884]: from='client.? 172.21.15.183:0/1013423769' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-09-10T19:11:21.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:11:21 smithi191 ceph-mon[32884]: 6.16 scrub starts 2024-09-10T19:11:21.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:11:21 smithi191 ceph-mon[32884]: 6.16 scrub ok 2024-09-10T19:11:21.732 INFO:tasks.cephadm.ceph_manager.ceph:need seq 55834574892 got 55834574896 for osd.1 2024-09-10T19:11:21.733 DEBUG:teuthology.parallel:result is None 2024-09-10T19:11:22.374 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:11:22 smithi183 ceph-mon[41562]: purged_snaps scrub starts 2024-09-10T19:11:22.374 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:11:22 smithi183 ceph-mon[41562]: purged_snaps scrub ok 2024-09-10T19:11:22.374 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:11:22 smithi183 ceph-mon[41562]: 4.1c deep-scrub starts 2024-09-10T19:11:22.374 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:11:22 smithi183 ceph-mon[41562]: 4.1c deep-scrub ok 2024-09-10T19:11:22.374 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:11:22 smithi183 ceph-mon[41562]: osdmap e79: 8 total, 8 up, 8 in 2024-09-10T19:11:22.374 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:11:22 smithi183 ceph-mon[41562]: pgmap v28: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-10T19:11:22.374 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:11:22 smithi183 ceph-mon[29736]: purged_snaps scrub starts 2024-09-10T19:11:22.374 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:11:22 smithi183 ceph-mon[29736]: purged_snaps scrub ok 2024-09-10T19:11:22.374 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:11:22 smithi183 ceph-mon[29736]: 4.1c deep-scrub starts 2024-09-10T19:11:22.375 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:11:22 smithi183 ceph-mon[29736]: 4.1c deep-scrub ok 2024-09-10T19:11:22.375 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:11:22 smithi183 ceph-mon[29736]: osdmap e79: 8 total, 8 up, 8 in 2024-09-10T19:11:22.375 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:11:22 smithi183 ceph-mon[29736]: pgmap v28: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-10T19:11:22.465 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:11:22 smithi191 ceph-mon[32884]: purged_snaps scrub starts 2024-09-10T19:11:22.465 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:11:22 smithi191 ceph-mon[32884]: purged_snaps scrub ok 2024-09-10T19:11:22.465 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:11:22 smithi191 ceph-mon[32884]: 4.1c deep-scrub starts 2024-09-10T19:11:22.465 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:11:22 smithi191 ceph-mon[32884]: 4.1c deep-scrub ok 2024-09-10T19:11:22.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:11:22 smithi191 ceph-mon[32884]: osdmap e79: 8 total, 8 up, 8 in 2024-09-10T19:11:22.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:11:22 smithi191 ceph-mon[32884]: pgmap v28: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-10T19:11:22.881 INFO:tasks.cephadm.ceph_manager.ceph:need seq 231928233998 got 231928234003 for osd.7 2024-09-10T19:11:22.881 DEBUG:teuthology.parallel:result is None 2024-09-10T19:11:23.465 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:11:23 smithi191 ceph-mon[32884]: 2.0 scrub starts 2024-09-10T19:11:23.465 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:11:23 smithi191 ceph-mon[32884]: 2.0 scrub ok 2024-09-10T19:11:23.465 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:11:23 smithi191 ceph-mon[32884]: 4.12 deep-scrub starts 2024-09-10T19:11:23.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:11:23 smithi191 ceph-mon[32884]: 4.12 deep-scrub ok 2024-09-10T19:11:23.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:11:23 smithi191 ceph-mon[32884]: purged_snaps scrub starts 2024-09-10T19:11:23.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:11:23 smithi191 ceph-mon[32884]: purged_snaps scrub ok 2024-09-10T19:11:23.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:11:23 smithi191 ceph-mon[32884]: 4.b scrub starts 2024-09-10T19:11:23.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:11:23 smithi191 ceph-mon[32884]: 4.b scrub ok 2024-09-10T19:11:23.530 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:11:23 smithi183 ceph-mon[29736]: 2.0 scrub starts 2024-09-10T19:11:23.531 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:11:23 smithi183 ceph-mon[29736]: 2.0 scrub ok 2024-09-10T19:11:23.531 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:11:23 smithi183 ceph-mon[29736]: 4.12 deep-scrub starts 2024-09-10T19:11:23.531 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:11:23 smithi183 ceph-mon[29736]: 4.12 deep-scrub ok 2024-09-10T19:11:23.531 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:11:23 smithi183 ceph-mon[29736]: purged_snaps scrub starts 2024-09-10T19:11:23.531 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:11:23 smithi183 ceph-mon[29736]: purged_snaps scrub ok 2024-09-10T19:11:23.531 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:11:23 smithi183 ceph-mon[29736]: 4.b scrub starts 2024-09-10T19:11:23.531 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:11:23 smithi183 ceph-mon[29736]: 4.b scrub ok 2024-09-10T19:11:23.531 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:11:23 smithi183 ceph-mon[41562]: 2.0 scrub starts 2024-09-10T19:11:23.531 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:11:23 smithi183 ceph-mon[41562]: 2.0 scrub ok 2024-09-10T19:11:23.532 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:11:23 smithi183 ceph-mon[41562]: 4.12 deep-scrub starts 2024-09-10T19:11:23.532 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:11:23 smithi183 ceph-mon[41562]: 4.12 deep-scrub ok 2024-09-10T19:11:23.532 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:11:23 smithi183 ceph-mon[41562]: purged_snaps scrub starts 2024-09-10T19:11:23.532 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:11:23 smithi183 ceph-mon[41562]: purged_snaps scrub ok 2024-09-10T19:11:23.532 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:11:23 smithi183 ceph-mon[41562]: 4.b scrub starts 2024-09-10T19:11:23.532 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:11:23 smithi183 ceph-mon[41562]: 4.b scrub ok 2024-09-10T19:11:23.969 INFO:teuthology.orchestra.run.smithi183.stdout:137438953505 2024-09-10T19:11:24.465 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:11:24 smithi191 ceph-mon[32884]: 2.b deep-scrub starts 2024-09-10T19:11:24.465 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:11:24 smithi191 ceph-mon[32884]: 2.b deep-scrub ok 2024-09-10T19:11:24.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:11:24 smithi191 ceph-mon[32884]: 2.7 scrub starts 2024-09-10T19:11:24.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:11:24 smithi191 ceph-mon[32884]: 2.7 scrub ok 2024-09-10T19:11:24.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:11:24 smithi191 ceph-mon[32884]: 5.18 scrub starts 2024-09-10T19:11:24.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:11:24 smithi191 ceph-mon[32884]: 5.18 scrub ok 2024-09-10T19:11:24.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:11:24 smithi191 ceph-mon[32884]: osdmap e80: 8 total, 8 up, 8 in 2024-09-10T19:11:24.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:11:24 smithi191 ceph-mon[32884]: 5.16 deep-scrub starts 2024-09-10T19:11:24.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:11:24 smithi191 ceph-mon[32884]: 5.16 deep-scrub ok 2024-09-10T19:11:24.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:11:24 smithi191 ceph-mon[32884]: pgmap v30: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-10T19:11:24.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:11:24 smithi191 ceph-mon[32884]: from='client.? 172.21.15.183:0/3072328869' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-09-10T19:11:24.510 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:11:24 smithi183 ceph-mon[29736]: 2.b deep-scrub starts 2024-09-10T19:11:24.510 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:11:24 smithi183 ceph-mon[29736]: 2.b deep-scrub ok 2024-09-10T19:11:24.510 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:11:24 smithi183 ceph-mon[29736]: 2.7 scrub starts 2024-09-10T19:11:24.510 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:11:24 smithi183 ceph-mon[29736]: 2.7 scrub ok 2024-09-10T19:11:24.510 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:11:24 smithi183 ceph-mon[29736]: 5.18 scrub starts 2024-09-10T19:11:24.510 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:11:24 smithi183 ceph-mon[29736]: 5.18 scrub ok 2024-09-10T19:11:24.510 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:11:24 smithi183 ceph-mon[29736]: osdmap e80: 8 total, 8 up, 8 in 2024-09-10T19:11:24.511 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:11:24 smithi183 ceph-mon[29736]: 5.16 deep-scrub starts 2024-09-10T19:11:24.511 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:11:24 smithi183 ceph-mon[29736]: 5.16 deep-scrub ok 2024-09-10T19:11:24.511 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:11:24 smithi183 ceph-mon[29736]: pgmap v30: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-10T19:11:24.511 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:11:24 smithi183 ceph-mon[29736]: from='client.? 172.21.15.183:0/3072328869' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-09-10T19:11:24.511 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:11:24 smithi183 ceph-mon[41562]: 2.b deep-scrub starts 2024-09-10T19:11:24.511 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:11:24 smithi183 ceph-mon[41562]: 2.b deep-scrub ok 2024-09-10T19:11:24.511 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:11:24 smithi183 ceph-mon[41562]: 2.7 scrub starts 2024-09-10T19:11:24.511 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:11:24 smithi183 ceph-mon[41562]: 2.7 scrub ok 2024-09-10T19:11:24.511 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:11:24 smithi183 ceph-mon[41562]: 5.18 scrub starts 2024-09-10T19:11:24.511 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:11:24 smithi183 ceph-mon[41562]: 5.18 scrub ok 2024-09-10T19:11:24.511 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:11:24 smithi183 ceph-mon[41562]: osdmap e80: 8 total, 8 up, 8 in 2024-09-10T19:11:24.512 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:11:24 smithi183 ceph-mon[41562]: 5.16 deep-scrub starts 2024-09-10T19:11:24.512 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:11:24 smithi183 ceph-mon[41562]: 5.16 deep-scrub ok 2024-09-10T19:11:24.512 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:11:24 smithi183 ceph-mon[41562]: pgmap v30: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-10T19:11:24.512 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:11:24 smithi183 ceph-mon[41562]: from='client.? 172.21.15.183:0/3072328869' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-09-10T19:11:24.533 INFO:tasks.cephadm.ceph_manager.ceph:need seq 137438953501 got 137438953505 for osd.4 2024-09-10T19:11:24.534 DEBUG:teuthology.parallel:result is None 2024-09-10T19:11:25.349 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:11:25 smithi191 ceph-mon[32884]: 5.2 scrub starts 2024-09-10T19:11:25.349 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:11:25 smithi191 ceph-mon[32884]: 5.2 scrub ok 2024-09-10T19:11:25.349 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:11:25 smithi191 ceph-mon[32884]: 5.14 scrub starts 2024-09-10T19:11:25.349 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:11:25 smithi191 ceph-mon[32884]: 5.14 scrub ok 2024-09-10T19:11:25.349 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:11:25 smithi191 ceph-mon[32884]: 4.1b scrub starts 2024-09-10T19:11:25.349 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:11:25 smithi191 ceph-mon[32884]: 4.1b scrub ok 2024-09-10T19:11:25.349 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:11:25 smithi191 ceph-mon[32884]: 6.f scrub starts 2024-09-10T19:11:25.349 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:11:25 smithi191 ceph-mon[32884]: 6.f scrub ok 2024-09-10T19:11:25.349 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:11:25 smithi191 ceph-mon[32884]: 2.16 scrub starts 2024-09-10T19:11:25.350 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:11:25 smithi191 ceph-mon[32884]: 2.16 scrub ok 2024-09-10T19:11:25.417 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:11:25 smithi183 ceph-mon[29736]: 5.2 scrub starts 2024-09-10T19:11:25.417 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:11:25 smithi183 ceph-mon[29736]: 5.2 scrub ok 2024-09-10T19:11:25.417 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:11:25 smithi183 ceph-mon[29736]: 5.14 scrub starts 2024-09-10T19:11:25.417 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:11:25 smithi183 ceph-mon[29736]: 5.14 scrub ok 2024-09-10T19:11:25.417 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:11:25 smithi183 ceph-mon[29736]: 4.1b scrub starts 2024-09-10T19:11:25.417 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:11:25 smithi183 ceph-mon[29736]: 4.1b scrub ok 2024-09-10T19:11:25.417 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:11:25 smithi183 ceph-mon[29736]: 6.f scrub starts 2024-09-10T19:11:25.417 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:11:25 smithi183 ceph-mon[29736]: 6.f scrub ok 2024-09-10T19:11:25.417 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:11:25 smithi183 ceph-mon[29736]: 2.16 scrub starts 2024-09-10T19:11:25.418 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:11:25 smithi183 ceph-mon[29736]: 2.16 scrub ok 2024-09-10T19:11:25.418 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:11:25 smithi183 ceph-mon[41562]: 5.2 scrub starts 2024-09-10T19:11:25.418 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:11:25 smithi183 ceph-mon[41562]: 5.2 scrub ok 2024-09-10T19:11:25.418 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:11:25 smithi183 ceph-mon[41562]: 5.14 scrub starts 2024-09-10T19:11:25.418 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:11:25 smithi183 ceph-mon[41562]: 5.14 scrub ok 2024-09-10T19:11:25.418 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:11:25 smithi183 ceph-mon[41562]: 4.1b scrub starts 2024-09-10T19:11:25.418 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:11:25 smithi183 ceph-mon[41562]: 4.1b scrub ok 2024-09-10T19:11:25.418 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:11:25 smithi183 ceph-mon[41562]: 6.f scrub starts 2024-09-10T19:11:25.418 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:11:25 smithi183 ceph-mon[41562]: 6.f scrub ok 2024-09-10T19:11:25.418 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:11:25 smithi183 ceph-mon[41562]: 2.16 scrub starts 2024-09-10T19:11:25.418 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:11:25 smithi183 ceph-mon[41562]: 2.16 scrub ok 2024-09-10T19:11:25.663 INFO:teuthology.orchestra.run.smithi183.stdout:77309411371 2024-09-10T19:11:25.931 INFO:journalctl@ceph.alertmanager.a.smithi183.stdout:Sep 10 19:11:25 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-alertmanager-a[76837]: ts=2024-09-10T19:11:25.746Z caller=cluster.go:698 level=info component=cluster msg="gossip settled; proceeding" elapsed=10.00168091s 2024-09-10T19:11:26.307 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:11:26 smithi183 ceph-mon[41562]: 6.1f scrub starts 2024-09-10T19:11:26.307 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:11:26 smithi183 ceph-mon[41562]: 6.1f scrub ok 2024-09-10T19:11:26.307 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:11:26 smithi183 ceph-mon[41562]: 2.f scrub starts 2024-09-10T19:11:26.307 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:11:26 smithi183 ceph-mon[41562]: 2.f scrub ok 2024-09-10T19:11:26.307 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:11:26 smithi183 ceph-mon[41562]: 5.d deep-scrub starts 2024-09-10T19:11:26.307 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:11:26 smithi183 ceph-mon[41562]: 5.d deep-scrub ok 2024-09-10T19:11:26.307 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:11:26 smithi183 ceph-mon[41562]: 6.7 scrub starts 2024-09-10T19:11:26.307 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:11:26 smithi183 ceph-mon[41562]: 6.7 scrub ok 2024-09-10T19:11:26.307 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:11:26 smithi183 ceph-mon[41562]: pgmap v31: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-10T19:11:26.308 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:11:26 smithi183 ceph-mon[41562]: from='client.? 172.21.15.183:0/2499479932' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-09-10T19:11:26.308 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:11:26 smithi183 ceph-mon[41562]: 4.1e deep-scrub starts 2024-09-10T19:11:26.308 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:11:26 smithi183 ceph-mon[41562]: 4.1e deep-scrub ok 2024-09-10T19:11:26.308 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:11:26 smithi183 ceph-mon[29736]: 6.1f scrub starts 2024-09-10T19:11:26.308 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:11:26 smithi183 ceph-mon[29736]: 6.1f scrub ok 2024-09-10T19:11:26.308 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:11:26 smithi183 ceph-mon[29736]: 2.f scrub starts 2024-09-10T19:11:26.308 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:11:26 smithi183 ceph-mon[29736]: 2.f scrub ok 2024-09-10T19:11:26.308 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:11:26 smithi183 ceph-mon[29736]: 5.d deep-scrub starts 2024-09-10T19:11:26.308 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:11:26 smithi183 ceph-mon[29736]: 5.d deep-scrub ok 2024-09-10T19:11:26.308 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:11:26 smithi183 ceph-mon[29736]: 6.7 scrub starts 2024-09-10T19:11:26.308 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:11:26 smithi183 ceph-mon[29736]: 6.7 scrub ok 2024-09-10T19:11:26.309 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:11:26 smithi183 ceph-mon[29736]: pgmap v31: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-10T19:11:26.309 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:11:26 smithi183 ceph-mon[29736]: from='client.? 172.21.15.183:0/2499479932' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-09-10T19:11:26.309 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:11:26 smithi183 ceph-mon[29736]: 4.1e deep-scrub starts 2024-09-10T19:11:26.309 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:11:26 smithi183 ceph-mon[29736]: 4.1e deep-scrub ok 2024-09-10T19:11:26.332 INFO:tasks.cephadm.ceph_manager.ceph:need seq 77309411367 got 77309411371 for osd.2 2024-09-10T19:11:26.332 DEBUG:teuthology.parallel:result is None 2024-09-10T19:11:26.465 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:11:26 smithi191 ceph-mon[32884]: 6.1f scrub starts 2024-09-10T19:11:26.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:11:26 smithi191 ceph-mon[32884]: 6.1f scrub ok 2024-09-10T19:11:26.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:11:26 smithi191 ceph-mon[32884]: 2.f scrub starts 2024-09-10T19:11:26.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:11:26 smithi191 ceph-mon[32884]: 2.f scrub ok 2024-09-10T19:11:26.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:11:26 smithi191 ceph-mon[32884]: 5.d deep-scrub starts 2024-09-10T19:11:26.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:11:26 smithi191 ceph-mon[32884]: 5.d deep-scrub ok 2024-09-10T19:11:26.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:11:26 smithi191 ceph-mon[32884]: 6.7 scrub starts 2024-09-10T19:11:26.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:11:26 smithi191 ceph-mon[32884]: 6.7 scrub ok 2024-09-10T19:11:26.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:11:26 smithi191 ceph-mon[32884]: pgmap v31: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-10T19:11:26.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:11:26 smithi191 ceph-mon[32884]: from='client.? 172.21.15.183:0/2499479932' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-09-10T19:11:26.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:11:26 smithi191 ceph-mon[32884]: 4.1e deep-scrub starts 2024-09-10T19:11:26.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:11:26 smithi191 ceph-mon[32884]: 4.1e deep-scrub ok 2024-09-10T19:11:26.693 INFO:teuthology.orchestra.run.smithi183.stdout:34359738422 2024-09-10T19:11:26.919 INFO:teuthology.orchestra.run.smithi183.stdout:167503724574 2024-09-10T19:11:27.488 INFO:journalctl@ceph.mgr.x.smithi191.stdout:Sep 10 19:11:27 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-x[34967]: ::ffff:172.21.15.191 - - [10/Sep/2024:19:11:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T19:11:27.488 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:11:27 smithi191 ceph-mon[32884]: 2.5 scrub starts 2024-09-10T19:11:27.488 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:11:27 smithi191 ceph-mon[32884]: 2.5 scrub ok 2024-09-10T19:11:27.488 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:11:27 smithi191 ceph-mon[32884]: 2.1b scrub starts 2024-09-10T19:11:27.488 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:11:27 smithi191 ceph-mon[32884]: 2.1b scrub ok 2024-09-10T19:11:27.488 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:11:27 smithi191 ceph-mon[32884]: 6.e scrub starts 2024-09-10T19:11:27.488 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:11:27 smithi191 ceph-mon[32884]: 6.e scrub ok 2024-09-10T19:11:27.489 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:11:27 smithi191 ceph-mon[32884]: 2.a scrub starts 2024-09-10T19:11:27.489 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:11:27 smithi191 ceph-mon[32884]: 2.a scrub ok 2024-09-10T19:11:27.489 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:11:27 smithi191 ceph-mon[32884]: from='client.? 172.21.15.183:0/3651865564' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-09-10T19:11:27.489 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:11:27 smithi191 ceph-mon[32884]: from='client.? 172.21.15.183:0/1784562906' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-09-10T19:11:27.489 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:27 smithi191 podman[65752]: 2024-09-10 19:11:27.46474438 +0000 UTC m=+0.091135921 container create 217ef440a5ace4277b60a45237debb4aa0177045ca3690ae58f2684fe8bca300 (image=quay.io/ceph/ceph-grafana:9.4.7, name=ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a, summary=Grafana Container configured for Ceph mgr/dashboard integration, com.redhat.component=ubi8-container, name=ubi8, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, architecture=x86_64, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, release=239.1651231664, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664, com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com, vendor=Red Hat, Inc., io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.buildah.version=1.32.0, distribution-scope=public, vcs-type=git, io.openshift.expose-services=, build-date=2022-04-29T11:48:11.055897, io.k8s.display-name=Red Hat Universal Base Image 8, description=Ceph Grafana Container, io.openshift.tags=base rhel8, maintainer=Paul Cuzner , version=8.5) 2024-09-10T19:11:27.489 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:27 smithi191 podman[65752]: 2024-09-10 19:11:27.388559181 +0000 UTC m=+0.014945863 image pull 954c08fa618887fe9a6fe572ae541197c77e625b05f4193df7898f5ac649ada9 quay.io/ceph/ceph-grafana:9.4.7 2024-09-10T19:11:27.621 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:11:27 smithi183 ceph-mon[29736]: 2.5 scrub starts 2024-09-10T19:11:27.621 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:11:27 smithi183 ceph-mon[29736]: 2.5 scrub ok 2024-09-10T19:11:27.621 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:11:27 smithi183 ceph-mon[29736]: 2.1b scrub starts 2024-09-10T19:11:27.621 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:11:27 smithi183 ceph-mon[29736]: 2.1b scrub ok 2024-09-10T19:11:27.622 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:11:27 smithi183 ceph-mon[29736]: 6.e scrub starts 2024-09-10T19:11:27.622 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:11:27 smithi183 ceph-mon[29736]: 6.e scrub ok 2024-09-10T19:11:27.622 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:11:27 smithi183 ceph-mon[29736]: 2.a scrub starts 2024-09-10T19:11:27.622 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:11:27 smithi183 ceph-mon[29736]: 2.a scrub ok 2024-09-10T19:11:27.622 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:11:27 smithi183 ceph-mon[29736]: from='client.? 172.21.15.183:0/3651865564' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-09-10T19:11:27.622 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:11:27 smithi183 ceph-mon[29736]: from='client.? 172.21.15.183:0/1784562906' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-09-10T19:11:27.622 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:11:27 smithi183 ceph-mon[41562]: 2.5 scrub starts 2024-09-10T19:11:27.622 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:11:27 smithi183 ceph-mon[41562]: 2.5 scrub ok 2024-09-10T19:11:27.622 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:11:27 smithi183 ceph-mon[41562]: 2.1b scrub starts 2024-09-10T19:11:27.622 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:11:27 smithi183 ceph-mon[41562]: 2.1b scrub ok 2024-09-10T19:11:27.622 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:11:27 smithi183 ceph-mon[41562]: 6.e scrub starts 2024-09-10T19:11:27.622 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:11:27 smithi183 ceph-mon[41562]: 6.e scrub ok 2024-09-10T19:11:27.623 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:11:27 smithi183 ceph-mon[41562]: 2.a scrub starts 2024-09-10T19:11:27.623 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:11:27 smithi183 ceph-mon[41562]: 2.a scrub ok 2024-09-10T19:11:27.623 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:11:27 smithi183 ceph-mon[41562]: from='client.? 172.21.15.183:0/3651865564' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-09-10T19:11:27.623 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:11:27 smithi183 ceph-mon[41562]: from='client.? 172.21.15.183:0/1784562906' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-09-10T19:11:27.648 INFO:tasks.cephadm.ceph_manager.ceph:need seq 34359738418 got 34359738422 for osd.0 2024-09-10T19:11:27.648 DEBUG:teuthology.parallel:result is None 2024-09-10T19:11:27.785 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:27 smithi191 podman[65752]: 2024-09-10 19:11:27.700892116 +0000 UTC m=+0.327278788 container init 217ef440a5ace4277b60a45237debb4aa0177045ca3690ae58f2684fe8bca300 (image=quay.io/ceph/ceph-grafana:9.4.7, name=ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a, com.redhat.component=ubi8-container, build-date=2022-04-29T11:48:11.055897, vendor=Red Hat, Inc., io.openshift.tags=base rhel8, description=Ceph Grafana Container, summary=Grafana Container configured for Ceph mgr/dashboard integration, architecture=x86_64, com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com, release=239.1651231664, distribution-scope=public, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664, vcs-type=git, io.openshift.expose-services=, io.k8s.display-name=Red Hat Universal Base Image 8, name=ubi8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, version=8.5, maintainer=Paul Cuzner , io.buildah.version=1.32.0) 2024-09-10T19:11:27.785 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:27 smithi191 podman[65752]: 2024-09-10 19:11:27.705659512 +0000 UTC m=+0.332046186 container start 217ef440a5ace4277b60a45237debb4aa0177045ca3690ae58f2684fe8bca300 (image=quay.io/ceph/ceph-grafana:9.4.7, name=ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a, version=8.5, architecture=x86_64, com.redhat.component=ubi8-container, name=ubi8, io.k8s.display-name=Red Hat Universal Base Image 8, com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com, release=239.1651231664, io.openshift.tags=base rhel8, vcs-type=git, maintainer=Paul Cuzner , io.buildah.version=1.32.0, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, io.openshift.expose-services=, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., description=Ceph Grafana Container, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664, distribution-scope=public, summary=Grafana Container configured for Ceph mgr/dashboard integration, build-date=2022-04-29T11:48:11.055897, vendor=Red Hat, Inc.) 2024-09-10T19:11:28.036 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:27 smithi191 bash[65752]: 217ef440a5ace4277b60a45237debb4aa0177045ca3690ae58f2684fe8bca300 2024-09-10T19:11:28.036 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:27 smithi191 systemd[1]: Started Ceph grafana.a for 320f6df8-6fa7-11ef-bcea-c7b262605968. 2024-09-10T19:11:28.036 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:27 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=settings t=2024-09-10T19:11:27.858668449Z level=info msg="Starting Grafana" version=9.4.7 commit=4add91f03d branch=HEAD compiled=2023-03-16T23:56:52Z 2024-09-10T19:11:28.037 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:27 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=settings t=2024-09-10T19:11:27.858881104Z level=warn msg="\"sentry\" frontend logging provider is deprecated and will be removed in the next major version. Use \"grafana\" provider instead." 2024-09-10T19:11:28.037 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:27 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=settings t=2024-09-10T19:11:27.85890805Z level=info msg="Config loaded from" file=/usr/share/grafana/conf/defaults.ini 2024-09-10T19:11:28.037 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:27 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=settings t=2024-09-10T19:11:27.858921548Z level=info msg="Config loaded from" file=/etc/grafana/grafana.ini 2024-09-10T19:11:28.037 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:27 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=settings t=2024-09-10T19:11:27.858932883Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_DATA=/var/lib/grafana" 2024-09-10T19:11:28.037 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:27 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=settings t=2024-09-10T19:11:27.858944803Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_LOGS=/var/log/grafana" 2024-09-10T19:11:28.037 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:27 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=settings t=2024-09-10T19:11:27.858955578Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_PLUGINS=/var/lib/grafana/plugins" 2024-09-10T19:11:28.037 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:27 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=settings t=2024-09-10T19:11:27.858965909Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_PROVISIONING=/etc/grafana/provisioning" 2024-09-10T19:11:28.037 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:27 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=settings t=2024-09-10T19:11:27.858977194Z level=info msg="Path Home" path=/usr/share/grafana 2024-09-10T19:11:28.037 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:27 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=settings t=2024-09-10T19:11:27.858988448Z level=info msg="Path Data" path=/var/lib/grafana 2024-09-10T19:11:28.037 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:27 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=settings t=2024-09-10T19:11:27.858999213Z level=info msg="Path Logs" path=/var/log/grafana 2024-09-10T19:11:28.037 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:27 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=settings t=2024-09-10T19:11:27.859009203Z level=info msg="Path Plugins" path=/var/lib/grafana/plugins 2024-09-10T19:11:28.037 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:27 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=settings t=2024-09-10T19:11:27.859019477Z level=info msg="Path Provisioning" path=/etc/grafana/provisioning 2024-09-10T19:11:28.037 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:27 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=settings t=2024-09-10T19:11:27.859029826Z level=info msg="App mode production" 2024-09-10T19:11:28.038 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:27 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=sqlstore t=2024-09-10T19:11:27.859083406Z level=info msg="Connecting to DB" dbtype=sqlite3 2024-09-10T19:11:28.038 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:27 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=sqlstore t=2024-09-10T19:11:27.859106394Z level=warn msg="SQLite database file has broader permissions than it should" path=/var/lib/grafana/grafana.db mode=-rw-r--r-- expected=-rw-r----- 2024-09-10T19:11:28.038 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:27 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:27.859976261Z level=info msg="Starting DB migrations" 2024-09-10T19:11:28.038 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:27 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:27.860061779Z level=info msg="Executing migration" id="create migration_log table" 2024-09-10T19:11:28.038 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:27 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:27.860578232Z level=info msg="Executing migration" id="create user table" 2024-09-10T19:11:28.038 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:27 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:27.861031441Z level=info msg="Executing migration" id="add unique index user.login" 2024-09-10T19:11:28.038 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:27 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:27.861866896Z level=info msg="Executing migration" id="add unique index user.email" 2024-09-10T19:11:28.038 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:27 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:27.862779931Z level=info msg="Executing migration" id="drop index UQE_user_login - v1" 2024-09-10T19:11:28.038 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:27 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:27.863314312Z level=info msg="Executing migration" id="drop index UQE_user_email - v1" 2024-09-10T19:11:28.038 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:27 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:27.864170305Z level=info msg="Executing migration" id="Rename table user to user_v1 - v1" 2024-09-10T19:11:28.038 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:27 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:27.865200338Z level=info msg="Executing migration" id="create user table v2" 2024-09-10T19:11:28.038 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:27 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:27.865890776Z level=info msg="Executing migration" id="create index UQE_user_login - v2" 2024-09-10T19:11:28.038 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:27 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:27.866903739Z level=info msg="Executing migration" id="create index UQE_user_email - v2" 2024-09-10T19:11:28.039 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:27 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:27.86766495Z level=info msg="Executing migration" id="copy data_source v1 to v2" 2024-09-10T19:11:28.039 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:27 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:27.86805578Z level=info msg="Executing migration" id="Drop old table user_v1" 2024-09-10T19:11:28.039 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:27 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:27.868954028Z level=info msg="Executing migration" id="Add column help_flags1 to user table" 2024-09-10T19:11:28.039 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:27 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:27.869799581Z level=info msg="Executing migration" id="Update user table charset" 2024-09-10T19:11:28.039 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:27 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:27.870614508Z level=info msg="Executing migration" id="Add last_seen_at column to user" 2024-09-10T19:11:28.039 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:27 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:27.871482524Z level=info msg="Executing migration" id="Add missing user data" 2024-09-10T19:11:28.039 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:27 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:27.872432876Z level=info msg="Executing migration" id="Add is_disabled column to user" 2024-09-10T19:11:28.039 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:27 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:27.872952483Z level=info msg="Executing migration" id="Add index user.login/user.email" 2024-09-10T19:11:28.039 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:27 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:27.875125705Z level=info msg="Executing migration" id="Add is_service_account column to user" 2024-09-10T19:11:28.039 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:27 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:27.877822518Z level=info msg="Executing migration" id="Update is_service_account column to nullable" 2024-09-10T19:11:28.040 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:27 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:27.880519659Z level=info msg="Executing migration" id="create temp user table v1-7" 2024-09-10T19:11:28.040 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:27 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:27.883521382Z level=info msg="Executing migration" id="create index IDX_temp_user_email - v1-7" 2024-09-10T19:11:28.040 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:27 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:27.885867859Z level=info msg="Executing migration" id="create index IDX_temp_user_org_id - v1-7" 2024-09-10T19:11:28.040 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:27 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:27.888534859Z level=info msg="Executing migration" id="create index IDX_temp_user_code - v1-7" 2024-09-10T19:11:28.040 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:27 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:27.891286712Z level=info msg="Executing migration" id="create index IDX_temp_user_status - v1-7" 2024-09-10T19:11:28.040 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:27 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:27.894398465Z level=info msg="Executing migration" id="Update temp_user table charset" 2024-09-10T19:11:28.040 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:27 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:27.899830888Z level=info msg="Executing migration" id="drop index IDX_temp_user_email - v1" 2024-09-10T19:11:28.041 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:27 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:27.900279988Z level=info msg="Executing migration" id="drop index IDX_temp_user_org_id - v1" 2024-09-10T19:11:28.041 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:27 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:27.902993416Z level=info msg="Executing migration" id="drop index IDX_temp_user_code - v1" 2024-09-10T19:11:28.041 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:27 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:27.903399477Z level=info msg="Executing migration" id="drop index IDX_temp_user_status - v1" 2024-09-10T19:11:28.041 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:27 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:27.905541596Z level=info msg="Executing migration" id="Rename table temp_user to temp_user_tmp_qwerty - v1" 2024-09-10T19:11:28.041 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:27 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:27.906253891Z level=info msg="Executing migration" id="create temp_user v2" 2024-09-10T19:11:28.041 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:27 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:27.908459582Z level=info msg="Executing migration" id="create index IDX_temp_user_email - v2" 2024-09-10T19:11:28.041 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:27 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:27.908980591Z level=info msg="Executing migration" id="create index IDX_temp_user_org_id - v2" 2024-09-10T19:11:28.041 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:27 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:27.911783126Z level=info msg="Executing migration" id="create index IDX_temp_user_code - v2" 2024-09-10T19:11:28.041 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:27 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:27.914222726Z level=info msg="Executing migration" id="create index IDX_temp_user_status - v2" 2024-09-10T19:11:28.041 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:27 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:27.916991598Z level=info msg="Executing migration" id="copy temp_user v1 to v2" 2024-09-10T19:11:28.041 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:27 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:27.917359352Z level=info msg="Executing migration" id="drop temp_user_tmp_qwerty" 2024-09-10T19:11:28.041 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:27 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:27.919929838Z level=info msg="Executing migration" id="Set created for temp users that will otherwise prematurely expire" 2024-09-10T19:11:28.041 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:27 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:27.920293795Z level=info msg="Executing migration" id="create star table" 2024-09-10T19:11:28.042 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:27 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:27.922741682Z level=info msg="Executing migration" id="add unique index star.user_id_dashboard_id" 2024-09-10T19:11:28.042 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:27 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:27.923310412Z level=info msg="Executing migration" id="create org table v1" 2024-09-10T19:11:28.042 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:27 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:27.925377936Z level=info msg="Executing migration" id="create index UQE_org_name - v1" 2024-09-10T19:11:28.042 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:27 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:27.928040899Z level=info msg="Executing migration" id="create org_user table v1" 2024-09-10T19:11:28.042 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:27 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:27.930869063Z level=info msg="Executing migration" id="create index IDX_org_user_org_id - v1" 2024-09-10T19:11:28.042 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:27 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:27.933699637Z level=info msg="Executing migration" id="create index UQE_org_user_org_id_user_id - v1" 2024-09-10T19:11:28.042 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:27 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:27.934350713Z level=info msg="Executing migration" id="create index IDX_org_user_user_id - v1" 2024-09-10T19:11:28.042 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:27 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:27.934831625Z level=info msg="Executing migration" id="Update org table charset" 2024-09-10T19:11:28.042 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:27 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:27.935610121Z level=info msg="Executing migration" id="Update org_user table charset" 2024-09-10T19:11:28.042 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:27 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:27.936535062Z level=info msg="Executing migration" id="Migrate all Read Only Viewers to Viewers" 2024-09-10T19:11:28.042 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:27 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:27.938444493Z level=info msg="Executing migration" id="create dashboard table" 2024-09-10T19:11:28.042 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:27 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:27.940487624Z level=info msg="Executing migration" id="add index dashboard.account_id" 2024-09-10T19:11:28.042 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:27 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:27.94237624Z level=info msg="Executing migration" id="add unique index dashboard_account_id_slug" 2024-09-10T19:11:28.043 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:27 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:27.943373244Z level=info msg="Executing migration" id="create dashboard_tag table" 2024-09-10T19:11:28.043 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:27 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:27.94420351Z level=info msg="Executing migration" id="add unique index dashboard_tag.dasboard_id_term" 2024-09-10T19:11:28.043 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:27 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:27.944952117Z level=info msg="Executing migration" id="drop index UQE_dashboard_tag_dashboard_id_term - v1" 2024-09-10T19:11:28.043 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:27 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:27.945803705Z level=info msg="Executing migration" id="Rename table dashboard to dashboard_v1 - v1" 2024-09-10T19:11:28.043 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:27 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:27.946673885Z level=info msg="Executing migration" id="create dashboard v2" 2024-09-10T19:11:28.043 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:27 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:27.947560574Z level=info msg="Executing migration" id="create index IDX_dashboard_org_id - v2" 2024-09-10T19:11:28.043 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:27 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:27.948425205Z level=info msg="Executing migration" id="create index UQE_dashboard_org_id_slug - v2" 2024-09-10T19:11:28.043 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:27 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:27.949402452Z level=info msg="Executing migration" id="copy dashboard v1 to v2" 2024-09-10T19:11:28.043 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:27 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:27.950142333Z level=info msg="Executing migration" id="drop table dashboard_v1" 2024-09-10T19:11:28.043 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:27 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:27.950996557Z level=info msg="Executing migration" id="alter dashboard.data to mediumtext v1" 2024-09-10T19:11:28.043 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:27 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:27.951870952Z level=info msg="Executing migration" id="Add column updated_by in dashboard - v2" 2024-09-10T19:11:28.043 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:27 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:27.95270508Z level=info msg="Executing migration" id="Add column created_by in dashboard - v2" 2024-09-10T19:11:28.043 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:27 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:27.953677073Z level=info msg="Executing migration" id="Add column gnetId in dashboard" 2024-09-10T19:11:28.043 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:27 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:27.95440873Z level=info msg="Executing migration" id="Add index for gnetId in dashboard" 2024-09-10T19:11:28.044 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:27 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:27.955326666Z level=info msg="Executing migration" id="Add column plugin_id in dashboard" 2024-09-10T19:11:28.044 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:27 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:27.95598418Z level=info msg="Executing migration" id="Add index for plugin_id in dashboard" 2024-09-10T19:11:28.044 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:27 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:27.956408346Z level=info msg="Executing migration" id="Add index for dashboard_id in dashboard_tag" 2024-09-10T19:11:28.044 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:27 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:27.959162007Z level=info msg="Executing migration" id="Update dashboard table charset" 2024-09-10T19:11:28.044 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:27 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:27.961967152Z level=info msg="Executing migration" id="Update dashboard_tag table charset" 2024-09-10T19:11:28.044 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:27 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:27.964743816Z level=info msg="Executing migration" id="Add column folder_id in dashboard" 2024-09-10T19:11:28.044 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:27 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:27.968360536Z level=info msg="Executing migration" id="Add column isFolder in dashboard" 2024-09-10T19:11:28.044 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:27 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:27.96898664Z level=info msg="Executing migration" id="Add column has_acl in dashboard" 2024-09-10T19:11:28.044 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:27 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:27.971848902Z level=info msg="Executing migration" id="Add column uid in dashboard" 2024-09-10T19:11:28.044 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:27 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:27.974306064Z level=info msg="Executing migration" id="Update uid column values in dashboard" 2024-09-10T19:11:28.044 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:27 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:27.977269163Z level=info msg="Executing migration" id="Add unique index dashboard_org_id_uid" 2024-09-10T19:11:28.044 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:27 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:27.979902172Z level=info msg="Executing migration" id="Remove unique index org_id_slug" 2024-09-10T19:11:28.044 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:27 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:27.982526767Z level=info msg="Executing migration" id="Update dashboard title length" 2024-09-10T19:11:28.045 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:27 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:27.985305617Z level=info msg="Executing migration" id="Add unique index for dashboard_org_id_title_folder_id" 2024-09-10T19:11:28.045 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:27 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:27.987912282Z level=info msg="Executing migration" id="create dashboard_provisioning" 2024-09-10T19:11:28.045 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:27 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:27.99088678Z level=info msg="Executing migration" id="Rename table dashboard_provisioning to dashboard_provisioning_tmp_qwerty - v1" 2024-09-10T19:11:28.045 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:27 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:27.993730855Z level=info msg="Executing migration" id="create dashboard_provisioning v2" 2024-09-10T19:11:28.045 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:27 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:27.996510812Z level=info msg="Executing migration" id="create index IDX_dashboard_provisioning_dashboard_id - v2" 2024-09-10T19:11:28.045 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:27 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:27.999142024Z level=info msg="Executing migration" id="create index IDX_dashboard_provisioning_dashboard_id_name - v2" 2024-09-10T19:11:28.045 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:28 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:28.002744754Z level=info msg="Executing migration" id="copy dashboard_provisioning v1 to v2" 2024-09-10T19:11:28.045 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:28 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:28.005466135Z level=info msg="Executing migration" id="drop dashboard_provisioning_tmp_qwerty" 2024-09-10T19:11:28.045 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:28 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:28.008069882Z level=info msg="Executing migration" id="Add check_sum column" 2024-09-10T19:11:28.045 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:28 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:28.010688149Z level=info msg="Executing migration" id="Add index for dashboard_title" 2024-09-10T19:11:28.045 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:28 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:28.013286427Z level=info msg="Executing migration" id="delete tags for deleted dashboards" 2024-09-10T19:11:28.045 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:28 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:28.015997219Z level=info msg="Executing migration" id="delete stars for deleted dashboards" 2024-09-10T19:11:28.045 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:28 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:28.018651953Z level=info msg="Executing migration" id="Add index for dashboard_is_folder" 2024-09-10T19:11:28.045 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:28 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:28.021090532Z level=info msg="Executing migration" id="Add isPublic for dashboard" 2024-09-10T19:11:28.046 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:28 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:28.023764017Z level=info msg="Executing migration" id="create data_source table" 2024-09-10T19:11:28.046 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:28 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:28.026408253Z level=info msg="Executing migration" id="add index data_source.account_id" 2024-09-10T19:11:28.046 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:28 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:28.029025217Z level=info msg="Executing migration" id="add unique index data_source.account_id_name" 2024-09-10T19:11:28.046 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:28 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:28.029517905Z level=info msg="Executing migration" id="drop index IDX_data_source_account_id - v1" 2024-09-10T19:11:28.046 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:28 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:28.032307649Z level=info msg="Executing migration" id="drop index UQE_data_source_account_id_name - v1" 2024-09-10T19:11:28.046 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:28 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:28.03499205Z level=info msg="Executing migration" id="Rename table data_source to data_source_v1 - v1" 2024-09-10T19:11:28.046 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:28 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:28.037663182Z level=info msg="Executing migration" id="create data_source table v2" 2024-09-10T19:11:28.286 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:11:28 smithi191 ceph-mon[32884]: 3.2 scrub starts 2024-09-10T19:11:28.286 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:11:28 smithi191 ceph-mon[32884]: 3.2 scrub ok 2024-09-10T19:11:28.287 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:11:28 smithi191 ceph-mon[32884]: 5.1b scrub starts 2024-09-10T19:11:28.287 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:11:28 smithi191 ceph-mon[32884]: 5.1b scrub ok 2024-09-10T19:11:28.287 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:11:28 smithi191 ceph-mon[32884]: pgmap v32: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 642 B/s rd, 0 op/s 2024-09-10T19:11:28.287 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:11:28 smithi191 ceph-mon[32884]: from='mgr.14556 ' entity='mgr.y' 2024-09-10T19:11:28.287 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:11:28 smithi191 ceph-mon[32884]: from='mgr.14556 ' entity='mgr.y' 2024-09-10T19:11:28.287 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:11:28 smithi191 ceph-mon[32884]: from='mgr.14556 ' entity='mgr.y' 2024-09-10T19:11:28.287 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:11:28 smithi191 ceph-mon[32884]: from='mgr.14556 ' entity='mgr.y' 2024-09-10T19:11:28.287 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:11:28 smithi191 ceph-mon[32884]: Deploying daemon node-exporter.a on smithi183 2024-09-10T19:11:28.288 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:28 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:28.040325924Z level=info msg="Executing migration" id="create index IDX_data_source_org_id - v2" 2024-09-10T19:11:28.288 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:28 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:28.042939473Z level=info msg="Executing migration" id="create index UQE_data_source_org_id_name - v2" 2024-09-10T19:11:28.288 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:28 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:28.045645668Z level=info msg="Executing migration" id="Drop old table data_source_v1 #2" 2024-09-10T19:11:28.288 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:28 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:28.048219209Z level=info msg="Executing migration" id="Add column with_credentials" 2024-09-10T19:11:28.288 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:28 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:28.050892943Z level=info msg="Executing migration" id="Add secure json data column" 2024-09-10T19:11:28.289 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:28 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:28.051870259Z level=info msg="Executing migration" id="Update data_source table charset" 2024-09-10T19:11:28.289 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:28 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:28.052549202Z level=info msg="Executing migration" id="Update initial version to 1" 2024-09-10T19:11:28.289 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:28 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:28.053340382Z level=info msg="Executing migration" id="Add read_only data column" 2024-09-10T19:11:28.289 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:28 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:28.053903865Z level=info msg="Executing migration" id="Migrate logging ds to loki ds" 2024-09-10T19:11:28.289 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:28 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:28.054236516Z level=info msg="Executing migration" id="Update json_data with nulls" 2024-09-10T19:11:28.289 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:28 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:28.055072784Z level=info msg="Executing migration" id="Add uid column" 2024-09-10T19:11:28.289 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:28 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:28.0556954Z level=info msg="Executing migration" id="Update uid value" 2024-09-10T19:11:28.289 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:28 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:28.056031466Z level=info msg="Executing migration" id="Add unique index datasource_org_id_uid" 2024-09-10T19:11:28.289 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:28 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:28.056892388Z level=info msg="Executing migration" id="add unique index datasource_org_id_is_default" 2024-09-10T19:11:28.289 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:28 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:28.057793767Z level=info msg="Executing migration" id="create api_key table" 2024-09-10T19:11:28.289 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:28 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:28.058582217Z level=info msg="Executing migration" id="add index api_key.account_id" 2024-09-10T19:11:28.289 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:28 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:28.059536887Z level=info msg="Executing migration" id="add index api_key.key" 2024-09-10T19:11:28.289 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:28 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:28.060320329Z level=info msg="Executing migration" id="add index api_key.account_id_name" 2024-09-10T19:11:28.290 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:28 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:28.06131749Z level=info msg="Executing migration" id="drop index IDX_api_key_account_id - v1" 2024-09-10T19:11:28.290 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:28 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:28.06212763Z level=info msg="Executing migration" id="drop index UQE_api_key_key - v1" 2024-09-10T19:11:28.290 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:28 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:28.062840852Z level=info msg="Executing migration" id="drop index UQE_api_key_account_id_name - v1" 2024-09-10T19:11:28.290 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:28 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:28.063712135Z level=info msg="Executing migration" id="Rename table api_key to api_key_v1 - v1" 2024-09-10T19:11:28.290 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:28 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:28.064781585Z level=info msg="Executing migration" id="create api_key table v2" 2024-09-10T19:11:28.290 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:28 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:28.065190039Z level=info msg="Executing migration" id="create index IDX_api_key_org_id - v2" 2024-09-10T19:11:28.290 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:28 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:28.065587564Z level=info msg="Executing migration" id="create index UQE_api_key_key - v2" 2024-09-10T19:11:28.290 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:28 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:28.066005819Z level=info msg="Executing migration" id="create index UQE_api_key_org_id_name - v2" 2024-09-10T19:11:28.290 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:28 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:28.066458951Z level=info msg="Executing migration" id="copy api_key v1 to v2" 2024-09-10T19:11:28.290 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:28 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:28.066766902Z level=info msg="Executing migration" id="Drop old table api_key_v1" 2024-09-10T19:11:28.290 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:28 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:28.067133681Z level=info msg="Executing migration" id="Update api_key table charset" 2024-09-10T19:11:28.290 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:28 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:28.067415466Z level=info msg="Executing migration" id="Add expires to api_key table" 2024-09-10T19:11:28.290 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:28 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:28.067964682Z level=info msg="Executing migration" id="Add service account foreign key" 2024-09-10T19:11:28.291 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:28 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:28.068501269Z level=info msg="Executing migration" id="set service account foreign key to nil if 0" 2024-09-10T19:11:28.291 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:28 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:28.068804833Z level=info msg="Executing migration" id="Add last_used_at to api_key table" 2024-09-10T19:11:28.291 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:28 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:28.069350215Z level=info msg="Executing migration" id="Add is_revoked column to api_key table" 2024-09-10T19:11:28.291 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:28 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:28.06992205Z level=info msg="Executing migration" id="create dashboard_snapshot table v4" 2024-09-10T19:11:28.291 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:28 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:28.070347573Z level=info msg="Executing migration" id="drop table dashboard_snapshot_v4 #1" 2024-09-10T19:11:28.291 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:28 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:28.070689127Z level=info msg="Executing migration" id="create dashboard_snapshot table v5 #2" 2024-09-10T19:11:28.291 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:28 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:28.07111267Z level=info msg="Executing migration" id="create index UQE_dashboard_snapshot_key - v5" 2024-09-10T19:11:28.291 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:28 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:28.071546558Z level=info msg="Executing migration" id="create index UQE_dashboard_snapshot_delete_key - v5" 2024-09-10T19:11:28.291 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:28 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:28.071965864Z level=info msg="Executing migration" id="create index IDX_dashboard_snapshot_user_id - v5" 2024-09-10T19:11:28.291 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:28 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:28.072377104Z level=info msg="Executing migration" id="alter dashboard_snapshot to mediumtext v2" 2024-09-10T19:11:28.291 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:28 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:28.072628224Z level=info msg="Executing migration" id="Update dashboard_snapshot table charset" 2024-09-10T19:11:28.291 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:28 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:28.072911469Z level=info msg="Executing migration" id="Add column external_delete_url to dashboard_snapshots table" 2024-09-10T19:11:28.291 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:28 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:28.073448631Z level=info msg="Executing migration" id="Add encrypted dashboard json column" 2024-09-10T19:11:28.291 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:28 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:28.07398952Z level=info msg="Executing migration" id="Change dashboard_encrypted column to MEDIUMBLOB" 2024-09-10T19:11:28.292 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:28 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:28.074282075Z level=info msg="Executing migration" id="create quota table v1" 2024-09-10T19:11:28.292 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:28 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:28.074651189Z level=info msg="Executing migration" id="create index UQE_quota_org_id_user_id_target - v1" 2024-09-10T19:11:28.292 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:28 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:28.075059267Z level=info msg="Executing migration" id="Update quota table charset" 2024-09-10T19:11:28.292 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:28 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:28.075338899Z level=info msg="Executing migration" id="create plugin_setting table" 2024-09-10T19:11:28.292 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:28 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:28.075716195Z level=info msg="Executing migration" id="create index UQE_plugin_setting_org_id_plugin_id - v1" 2024-09-10T19:11:28.292 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:28 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:28.076131208Z level=info msg="Executing migration" id="Add column plugin_version to plugin_settings" 2024-09-10T19:11:28.292 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:28 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:28.076654259Z level=info msg="Executing migration" id="Update plugin_setting table charset" 2024-09-10T19:11:28.292 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:28 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:28.076966194Z level=info msg="Executing migration" id="create session table" 2024-09-10T19:11:28.292 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:28 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:28.077370227Z level=info msg="Executing migration" id="Drop old table playlist table" 2024-09-10T19:11:28.292 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:28 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:28.077621951Z level=info msg="Executing migration" id="Drop old table playlist_item table" 2024-09-10T19:11:28.292 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:28 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:28.077919946Z level=info msg="Executing migration" id="create playlist table v2" 2024-09-10T19:11:28.292 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:28 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:28.078295669Z level=info msg="Executing migration" id="create playlist item table v2" 2024-09-10T19:11:28.292 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:28 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:28.078656114Z level=info msg="Executing migration" id="Update playlist table charset" 2024-09-10T19:11:28.293 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:28 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:28.078935732Z level=info msg="Executing migration" id="Update playlist_item table charset" 2024-09-10T19:11:28.293 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:28 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:28.079205983Z level=info msg="Executing migration" id="drop preferences table v2" 2024-09-10T19:11:28.293 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:28 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:28.079476087Z level=info msg="Executing migration" id="drop preferences table v3" 2024-09-10T19:11:28.293 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:28 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:28.079734734Z level=info msg="Executing migration" id="create preferences table v3" 2024-09-10T19:11:28.293 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:28 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:28.080120578Z level=info msg="Executing migration" id="Update preferences table charset" 2024-09-10T19:11:28.293 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:28 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:28.080381373Z level=info msg="Executing migration" id="Add column team_id in preferences" 2024-09-10T19:11:28.293 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:28 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:28.080992662Z level=info msg="Executing migration" id="Update team_id column values in preferences" 2024-09-10T19:11:28.293 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:28 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:28.081334358Z level=info msg="Executing migration" id="Add column week_start in preferences" 2024-09-10T19:11:28.293 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:28 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:28.081933352Z level=info msg="Executing migration" id="Add column preferences.json_data" 2024-09-10T19:11:28.293 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:28 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:28.082562697Z level=info msg="Executing migration" id="alter preferences.json_data to mediumtext v1" 2024-09-10T19:11:28.293 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:28 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:28.082843875Z level=info msg="Executing migration" id="Add preferences index org_id" 2024-09-10T19:11:28.293 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:28 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:28.083292459Z level=info msg="Executing migration" id="Add preferences index user_id" 2024-09-10T19:11:28.293 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:28 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:28.083793288Z level=info msg="Executing migration" id="create alert table v1" 2024-09-10T19:11:28.293 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:28 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:28.084312331Z level=info msg="Executing migration" id="add index alert org_id & id " 2024-09-10T19:11:28.294 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:28 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:28.085143792Z level=info msg="Executing migration" id="add index alert state" 2024-09-10T19:11:28.294 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:28 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:28.085970908Z level=info msg="Executing migration" id="add index alert dashboard_id" 2024-09-10T19:11:28.294 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:28 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:28.086414346Z level=info msg="Executing migration" id="Create alert_rule_tag table v1" 2024-09-10T19:11:28.294 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:28 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:28.086810049Z level=info msg="Executing migration" id="Add unique index alert_rule_tag.alert_id_tag_id" 2024-09-10T19:11:28.294 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:28 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:28.08728059Z level=info msg="Executing migration" id="drop index UQE_alert_rule_tag_alert_id_tag_id - v1" 2024-09-10T19:11:28.294 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:28 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:28.08766382Z level=info msg="Executing migration" id="Rename table alert_rule_tag to alert_rule_tag_v1 - v1" 2024-09-10T19:11:28.294 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:28 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:28.089082371Z level=info msg="Executing migration" id="Create alert_rule_tag table v2" 2024-09-10T19:11:28.294 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:28 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:28.089446955Z level=info msg="Executing migration" id="create index UQE_alert_rule_tag_alert_id_tag_id - Add unique index alert_rule_tag.alert_id_tag_id V2" 2024-09-10T19:11:28.294 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:28 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:28.089848633Z level=info msg="Executing migration" id="copy alert_rule_tag v1 to v2" 2024-09-10T19:11:28.294 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:28 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:28.090155831Z level=info msg="Executing migration" id="drop table alert_rule_tag_v1" 2024-09-10T19:11:28.294 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:28 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:28.090473742Z level=info msg="Executing migration" id="create alert_notification table v1" 2024-09-10T19:11:28.294 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:28 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:28.090814018Z level=info msg="Executing migration" id="Add column is_default" 2024-09-10T19:11:28.294 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:28 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:28.091415751Z level=info msg="Executing migration" id="Add column frequency" 2024-09-10T19:11:28.294 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:28 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:28.092030766Z level=info msg="Executing migration" id="Add column send_reminder" 2024-09-10T19:11:28.295 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:28 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:28.09263944Z level=info msg="Executing migration" id="Add column disable_resolve_message" 2024-09-10T19:11:28.295 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:28 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:28.093309481Z level=info msg="Executing migration" id="add index alert_notification org_id & name" 2024-09-10T19:11:28.295 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:28 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:28.093712943Z level=info msg="Executing migration" id="Update alert table charset" 2024-09-10T19:11:28.295 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:28 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:28.093998849Z level=info msg="Executing migration" id="Update alert_notification table charset" 2024-09-10T19:11:28.295 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:28 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:28.09425479Z level=info msg="Executing migration" id="create notification_journal table v1" 2024-09-10T19:11:28.295 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:28 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:28.094615149Z level=info msg="Executing migration" id="add index notification_journal org_id & alert_id & notifier_id" 2024-09-10T19:11:28.295 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:28 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:28.095122106Z level=info msg="Executing migration" id="drop alert_notification_journal" 2024-09-10T19:11:28.295 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:28 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:28.096110834Z level=info msg="Executing migration" id="create alert_notification_state table v1" 2024-09-10T19:11:28.295 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:28 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:28.09672172Z level=info msg="Executing migration" id="add index alert_notification_state org_id & alert_id & notifier_id" 2024-09-10T19:11:28.295 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:28 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:28.097665438Z level=info msg="Executing migration" id="Add for to alert table" 2024-09-10T19:11:28.296 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:28 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:28.098374241Z level=info msg="Executing migration" id="Add column uid in alert_notification" 2024-09-10T19:11:28.297 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:28 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:28.099154728Z level=info msg="Executing migration" id="Update uid column values in alert_notification" 2024-09-10T19:11:28.297 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:28 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:28.099993223Z level=info msg="Executing migration" id="Add unique index alert_notification_org_id_uid" 2024-09-10T19:11:28.297 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:28 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:28.10041471Z level=info msg="Executing migration" id="Remove unique index org_id_name" 2024-09-10T19:11:28.297 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:28 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:28.101194661Z level=info msg="Executing migration" id="Add column secure_settings in alert_notification" 2024-09-10T19:11:28.297 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:28 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:28.101852415Z level=info msg="Executing migration" id="alter alert.settings to mediumtext" 2024-09-10T19:11:28.297 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:28 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:28.102154127Z level=info msg="Executing migration" id="Add non-unique index alert_notification_state_alert_id" 2024-09-10T19:11:28.297 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:28 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:28.102907637Z level=info msg="Executing migration" id="Add non-unique index alert_rule_tag_alert_id" 2024-09-10T19:11:28.297 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:28 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:28.103317171Z level=info msg="Executing migration" id="Drop old annotation table v4" 2024-09-10T19:11:28.297 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:28 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:28.103618966Z level=info msg="Executing migration" id="create annotation table v5" 2024-09-10T19:11:28.297 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:28 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:28.10406026Z level=info msg="Executing migration" id="add index annotation 0 v3" 2024-09-10T19:11:28.297 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:28 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:28.104471627Z level=info msg="Executing migration" id="add index annotation 1 v3" 2024-09-10T19:11:28.297 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:28 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:28.105450686Z level=info msg="Executing migration" id="add index annotation 2 v3" 2024-09-10T19:11:28.298 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:28 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:28.106153322Z level=info msg="Executing migration" id="add index annotation 3 v3" 2024-09-10T19:11:28.298 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:28 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:28.106587321Z level=info msg="Executing migration" id="add index annotation 4 v3" 2024-09-10T19:11:28.298 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:28 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:28.107402623Z level=info msg="Executing migration" id="Update annotation table charset" 2024-09-10T19:11:28.298 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:28 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:28.108359831Z level=info msg="Executing migration" id="Add column region_id to annotation table" 2024-09-10T19:11:28.298 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:28 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:28.109121659Z level=info msg="Executing migration" id="Drop category_id index" 2024-09-10T19:11:28.298 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:28 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:28.109526595Z level=info msg="Executing migration" id="Add column tags to annotation table" 2024-09-10T19:11:28.298 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:28 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:28.110323832Z level=info msg="Executing migration" id="Create annotation_tag table v2" 2024-09-10T19:11:28.298 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:28 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:28.111296071Z level=info msg="Executing migration" id="Add unique index annotation_tag.annotation_id_tag_id" 2024-09-10T19:11:28.298 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:28 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:28.112030624Z level=info msg="Executing migration" id="drop index UQE_annotation_tag_annotation_id_tag_id - v2" 2024-09-10T19:11:28.298 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:28 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:28.112424075Z level=info msg="Executing migration" id="Rename table annotation_tag to annotation_tag_v2 - v2" 2024-09-10T19:11:28.298 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:28 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:28.114008076Z level=info msg="Executing migration" id="Create annotation_tag table v3" 2024-09-10T19:11:28.298 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:28 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:28.114857185Z level=info msg="Executing migration" id="create index UQE_annotation_tag_annotation_id_tag_id - Add unique index annotation_tag.annotation_id_tag_id V3" 2024-09-10T19:11:28.298 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:28 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:28.115823825Z level=info msg="Executing migration" id="copy annotation_tag v2 to v3" 2024-09-10T19:11:28.298 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:28 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:28.116216365Z level=info msg="Executing migration" id="drop table annotation_tag_v2" 2024-09-10T19:11:28.299 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:28 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:28.117057233Z level=info msg="Executing migration" id="Update alert annotations and set TEXT to empty" 2024-09-10T19:11:28.299 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:28 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:28.117411808Z level=info msg="Executing migration" id="Add created time to annotation table" 2024-09-10T19:11:28.299 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:28 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:28.118099252Z level=info msg="Executing migration" id="Add updated time to annotation table" 2024-09-10T19:11:28.299 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:28 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:28.120969775Z level=info msg="Executing migration" id="Add index for created in annotation table" 2024-09-10T19:11:28.299 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:28 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:28.121537181Z level=info msg="Executing migration" id="Add index for updated in annotation table" 2024-09-10T19:11:28.299 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:28 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:28.124292796Z level=info msg="Executing migration" id="Convert existing annotations from seconds to milliseconds" 2024-09-10T19:11:28.299 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:28 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:28.126854821Z level=info msg="Executing migration" id="Add epoch_end column" 2024-09-10T19:11:28.299 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:28 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:28.129665848Z level=info msg="Executing migration" id="Add index for epoch_end" 2024-09-10T19:11:28.299 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:28 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:28.130139299Z level=info msg="Executing migration" id="Make epoch_end the same as epoch" 2024-09-10T19:11:28.299 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:28 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:28.132736071Z level=info msg="Executing migration" id="Move region to single row" 2024-09-10T19:11:28.299 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:28 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:28.135652061Z level=info msg="Executing migration" id="Remove index org_id_epoch from annotation table" 2024-09-10T19:11:28.299 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:28 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:28.138063079Z level=info msg="Executing migration" id="Remove index org_id_dashboard_id_panel_id_epoch from annotation table" 2024-09-10T19:11:28.300 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:28 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:28.138510359Z level=info msg="Executing migration" id="Add index for org_id_dashboard_id_epoch_end_epoch on annotation table" 2024-09-10T19:11:28.300 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:28 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:28.141351561Z level=info msg="Executing migration" id="Add index for org_id_epoch_end_epoch on annotation table" 2024-09-10T19:11:28.300 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:28 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:28.143858463Z level=info msg="Executing migration" id="Remove index org_id_epoch_epoch_end from annotation table" 2024-09-10T19:11:28.300 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:28 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:28.146744989Z level=info msg="Executing migration" id="Add index for alert_id on annotation table" 2024-09-10T19:11:28.300 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:28 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:28.147180347Z level=info msg="Executing migration" id="Increase tags column to length 4096" 2024-09-10T19:11:28.300 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:28 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:28.150033338Z level=info msg="Executing migration" id="create test_data table" 2024-09-10T19:11:28.300 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:28 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:28.150504174Z level=info msg="Executing migration" id="create dashboard_version table v1" 2024-09-10T19:11:28.300 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:28 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:28.152984119Z level=info msg="Executing migration" id="add index dashboard_version.dashboard_id" 2024-09-10T19:11:28.300 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:28 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:28.155784377Z level=info msg="Executing migration" id="add unique index dashboard_version.dashboard_id and dashboard_version.version" 2024-09-10T19:11:28.300 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:28 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:28.156233221Z level=info msg="Executing migration" id="Set dashboard version to 1 where 0" 2024-09-10T19:11:28.300 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:28 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:28.158835494Z level=info msg="Executing migration" id="save existing dashboard data in dashboard_version table v1" 2024-09-10T19:11:28.300 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:28 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:28.159249088Z level=info msg="Executing migration" id="alter dashboard_version.data to mediumtext v1" 2024-09-10T19:11:28.300 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:28 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:28.161394947Z level=info msg="Executing migration" id="create team table" 2024-09-10T19:11:28.301 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:28 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:28.161842769Z level=info msg="Executing migration" id="add index team.org_id" 2024-09-10T19:11:28.301 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:28 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:28.1640164Z level=info msg="Executing migration" id="add unique index team_org_id_name" 2024-09-10T19:11:28.301 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:28 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:28.164551541Z level=info msg="Executing migration" id="create team member table" 2024-09-10T19:11:28.301 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:28 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:28.167234795Z level=info msg="Executing migration" id="add index team_member.org_id" 2024-09-10T19:11:28.301 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:28 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:28.169976146Z level=info msg="Executing migration" id="add unique index team_member_org_id_team_id_user_id" 2024-09-10T19:11:28.301 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:28 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:28.172795819Z level=info msg="Executing migration" id="add index team_member.team_id" 2024-09-10T19:11:28.301 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:28 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:28.173416969Z level=info msg="Executing migration" id="Add column email to team table" 2024-09-10T19:11:28.301 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:28 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:28.176192341Z level=info msg="Executing migration" id="Add column external to team_member table" 2024-09-10T19:11:28.301 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:28 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:28.17884641Z level=info msg="Executing migration" id="Add column permission to team_member table" 2024-09-10T19:11:28.301 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:28 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:28.181355167Z level=info msg="Executing migration" id="create dashboard acl table" 2024-09-10T19:11:28.301 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:28 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:28.181800303Z level=info msg="Executing migration" id="add index dashboard_acl_dashboard_id" 2024-09-10T19:11:28.301 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:28 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:28.184522997Z level=info msg="Executing migration" id="add unique index dashboard_acl_dashboard_id_user_id" 2024-09-10T19:11:28.301 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:28 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:28.187090016Z level=info msg="Executing migration" id="add unique index dashboard_acl_dashboard_id_team_id" 2024-09-10T19:11:28.301 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:28 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:28.187968856Z level=info msg="Executing migration" id="add index dashboard_acl_user_id" 2024-09-10T19:11:28.302 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:28 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:28.188507406Z level=info msg="Executing migration" id="add index dashboard_acl_team_id" 2024-09-10T19:11:28.302 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:28 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:28.189275731Z level=info msg="Executing migration" id="add index dashboard_acl_org_id_role" 2024-09-10T19:11:28.302 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:28 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:28.190152419Z level=info msg="Executing migration" id="add index dashboard_permission" 2024-09-10T19:11:28.302 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:28 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:28.191027314Z level=info msg="Executing migration" id="save default acl rules in dashboard_acl table" 2024-09-10T19:11:28.302 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:28 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:28.191740069Z level=info msg="Executing migration" id="delete acl rules for deleted dashboards and folders" 2024-09-10T19:11:28.302 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:28 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:28.192488663Z level=info msg="Executing migration" id="create tag table" 2024-09-10T19:11:28.302 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:28 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:28.193329503Z level=info msg="Executing migration" id="add index tag.key_value" 2024-09-10T19:11:28.302 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:28 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:28.194143437Z level=info msg="Executing migration" id="create login attempt table" 2024-09-10T19:11:28.302 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:28 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:28.194965405Z level=info msg="Executing migration" id="add index login_attempt.username" 2024-09-10T19:11:28.302 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:28 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:28.195918866Z level=info msg="Executing migration" id="drop index IDX_login_attempt_username - v1" 2024-09-10T19:11:28.302 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:28 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:28.196560759Z level=info msg="Executing migration" id="Rename table login_attempt to login_attempt_tmp_qwerty - v1" 2024-09-10T19:11:28.302 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:28 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:28.198448118Z level=info msg="Executing migration" id="create login_attempt v2" 2024-09-10T19:11:28.302 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:28 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:28.199251147Z level=info msg="Executing migration" id="create index IDX_login_attempt_username - v2" 2024-09-10T19:11:28.303 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:28 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:28.200192804Z level=info msg="Executing migration" id="copy login_attempt v1 to v2" 2024-09-10T19:11:28.303 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:28 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:28.200839573Z level=info msg="Executing migration" id="drop login_attempt_tmp_qwerty" 2024-09-10T19:11:28.303 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:28 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:28.201676447Z level=info msg="Executing migration" id="create user auth table" 2024-09-10T19:11:28.303 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:28 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:28.202656659Z level=info msg="Executing migration" id="create index IDX_user_auth_auth_module_auth_id - v1" 2024-09-10T19:11:28.303 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:28 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:28.203370121Z level=info msg="Executing migration" id="alter user_auth.auth_id to length 190" 2024-09-10T19:11:28.303 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:28 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:28.204307814Z level=info msg="Executing migration" id="Add OAuth access token to user_auth" 2024-09-10T19:11:28.303 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:28 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:28.205293384Z level=info msg="Executing migration" id="Add OAuth refresh token to user_auth" 2024-09-10T19:11:28.303 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:28 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:28.206027647Z level=info msg="Executing migration" id="Add OAuth token type to user_auth" 2024-09-10T19:11:28.303 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:28 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:28.206886726Z level=info msg="Executing migration" id="Add OAuth expiry to user_auth" 2024-09-10T19:11:28.303 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:28 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:28.207673271Z level=info msg="Executing migration" id="Add index to user_id column in user_auth" 2024-09-10T19:11:28.303 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:28 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:28.208654086Z level=info msg="Executing migration" id="Add OAuth ID token to user_auth" 2024-09-10T19:11:28.303 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:28 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:28.20942014Z level=info msg="Executing migration" id="create server_lock table" 2024-09-10T19:11:28.303 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:28 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:28.210333567Z level=info msg="Executing migration" id="add index server_lock.operation_uid" 2024-09-10T19:11:28.303 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:28 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:28.211341531Z level=info msg="Executing migration" id="create user auth token table" 2024-09-10T19:11:28.304 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:28 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:28.212035369Z level=info msg="Executing migration" id="add unique index user_auth_token.auth_token" 2024-09-10T19:11:28.304 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:28 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:28.212870041Z level=info msg="Executing migration" id="add unique index user_auth_token.prev_auth_token" 2024-09-10T19:11:28.304 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:28 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:28.213741761Z level=info msg="Executing migration" id="add index user_auth_token.user_id" 2024-09-10T19:11:28.304 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:28 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:28.214720345Z level=info msg="Executing migration" id="Add revoked_at to the user auth token" 2024-09-10T19:11:28.304 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:28 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:28.215652183Z level=info msg="Executing migration" id="create cache_data table" 2024-09-10T19:11:28.304 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:28 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:28.216633122Z level=info msg="Executing migration" id="add unique index cache_data.cache_key" 2024-09-10T19:11:28.304 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:28 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:28.217396849Z level=info msg="Executing migration" id="create short_url table v1" 2024-09-10T19:11:28.304 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:28 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:28.220185187Z level=info msg="Executing migration" id="add index short_url.org_id-uid" 2024-09-10T19:11:28.304 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:28 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:28.225349195Z level=info msg="Executing migration" id="alter table short_url alter column created_by type to bigint" 2024-09-10T19:11:28.304 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:28 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:28.225645687Z level=info msg="Executing migration" id="delete alert_definition table" 2024-09-10T19:11:28.304 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:28 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:28.228416828Z level=info msg="Executing migration" id="recreate alert_definition table" 2024-09-10T19:11:28.304 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:28 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:28.23383678Z level=info msg="Executing migration" id="add index in alert_definition on org_id and title columns" 2024-09-10T19:11:28.304 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:28 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:28.234291395Z level=info msg="Executing migration" id="add index in alert_definition on org_id and uid columns" 2024-09-10T19:11:28.305 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:28 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:28.236874819Z level=info msg="Executing migration" id="alter alert_definition table data column to mediumtext in mysql" 2024-09-10T19:11:28.305 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:28 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:28.23952949Z level=info msg="Executing migration" id="drop index in alert_definition on org_id and title columns" 2024-09-10T19:11:28.305 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:28 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:28.242713574Z level=info msg="Executing migration" id="drop index in alert_definition on org_id and uid columns" 2024-09-10T19:11:28.305 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:28 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:28.245471545Z level=info msg="Executing migration" id="add unique index in alert_definition on org_id and title columns" 2024-09-10T19:11:28.305 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:28 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:28.248294566Z level=info msg="Executing migration" id="add unique index in alert_definition on org_id and uid columns" 2024-09-10T19:11:28.305 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:28 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:28.251082861Z level=info msg="Executing migration" id="Add column paused in alert_definition" 2024-09-10T19:11:28.305 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:28 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:28.253730155Z level=info msg="Executing migration" id="drop alert_definition table" 2024-09-10T19:11:28.305 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:28 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:28.256332027Z level=info msg="Executing migration" id="delete alert_definition_version table" 2024-09-10T19:11:28.305 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:28 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:28.259220676Z level=info msg="Executing migration" id="recreate alert_definition_version table" 2024-09-10T19:11:28.305 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:28 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:28.261947347Z level=info msg="Executing migration" id="add index in alert_definition_version table on alert_definition_id and version columns" 2024-09-10T19:11:28.305 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:28 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:28.26469393Z level=info msg="Executing migration" id="add index in alert_definition_version table on alert_definition_uid and version columns" 2024-09-10T19:11:28.305 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:28 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:28.267370125Z level=info msg="Executing migration" id="alter alert_definition_version table data column to mediumtext in mysql" 2024-09-10T19:11:28.305 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:28 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:28.269810022Z level=info msg="Executing migration" id="drop alert_definition_version table" 2024-09-10T19:11:28.306 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:28 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:28.270262157Z level=info msg="Executing migration" id="create alert_instance table" 2024-09-10T19:11:28.306 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:28 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:28.270645321Z level=info msg="Executing migration" id="add index in alert_instance table on def_org_id, def_uid and current_state columns" 2024-09-10T19:11:28.306 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:28 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:28.271075186Z level=info msg="Executing migration" id="add index in alert_instance table on def_org_id, current_state columns" 2024-09-10T19:11:28.306 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:28 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:28.271546465Z level=info msg="Executing migration" id="add column current_state_end to alert_instance" 2024-09-10T19:11:28.306 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:28 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:28.272370147Z level=info msg="Executing migration" id="remove index def_org_id, def_uid, current_state on alert_instance" 2024-09-10T19:11:28.306 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:28 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:28.272813856Z level=info msg="Executing migration" id="remove index def_org_id, current_state on alert_instance" 2024-09-10T19:11:28.306 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:28 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:28.273278689Z level=info msg="Executing migration" id="rename def_org_id to rule_org_id in alert_instance" 2024-09-10T19:11:28.306 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:28 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:28.276730468Z level=info msg="Executing migration" id="rename def_uid to rule_uid in alert_instance" 2024-09-10T19:11:28.306 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:28 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:28.280063833Z level=info msg="Executing migration" id="add index rule_org_id, rule_uid, current_state on alert_instance" 2024-09-10T19:11:28.306 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:28 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:28.280549295Z level=info msg="Executing migration" id="add index rule_org_id, current_state on alert_instance" 2024-09-10T19:11:28.306 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:28 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:28.280985336Z level=info msg="Executing migration" id="add current_reason column related to current_state" 2024-09-10T19:11:28.306 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:28 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:28.281823744Z level=info msg="Executing migration" id="create alert_rule table" 2024-09-10T19:11:28.306 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:28 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:28.282310163Z level=info msg="Executing migration" id="add index in alert_rule on org_id and title columns" 2024-09-10T19:11:28.306 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:28 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:28.283154151Z level=info msg="Executing migration" id="add index in alert_rule on org_id and uid columns" 2024-09-10T19:11:28.307 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:28 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:28.283968833Z level=info msg="Executing migration" id="add index in alert_rule on org_id, namespace_uid, group_uid columns" 2024-09-10T19:11:28.307 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:28 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:28.284963032Z level=info msg="Executing migration" id="alter alert_rule table data column to mediumtext in mysql" 2024-09-10T19:11:28.307 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:28 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:28.2856822Z level=info msg="Executing migration" id="add column for to alert_rule" 2024-09-10T19:11:28.307 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:28 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:28.286561823Z level=info msg="Executing migration" id="add column annotations to alert_rule" 2024-09-10T19:11:28.307 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:28 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:28.287457058Z level=info msg="Executing migration" id="add column labels to alert_rule" 2024-09-10T19:11:28.307 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:28 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:28.288310761Z level=info msg="Executing migration" id="remove unique index from alert_rule on org_id, title columns" 2024-09-10T19:11:28.307 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:28 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:28.289166089Z level=info msg="Executing migration" id="add index in alert_rule on org_id, namespase_uid and title columns" 2024-09-10T19:11:28.307 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:28 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:28.290016444Z level=info msg="Executing migration" id="add dashboard_uid column to alert_rule" 2024-09-10T19:11:28.307 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:28 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:28.290920698Z level=info msg="Executing migration" id="add panel_id column to alert_rule" 2024-09-10T19:11:28.372 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:11:28 smithi183 ceph-mon[29736]: 3.2 scrub starts 2024-09-10T19:11:28.372 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:11:28 smithi183 ceph-mon[29736]: 3.2 scrub ok 2024-09-10T19:11:28.372 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:11:28 smithi183 ceph-mon[29736]: 5.1b scrub starts 2024-09-10T19:11:28.372 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:11:28 smithi183 ceph-mon[29736]: 5.1b scrub ok 2024-09-10T19:11:28.372 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:11:28 smithi183 ceph-mon[29736]: pgmap v32: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 642 B/s rd, 0 op/s 2024-09-10T19:11:28.372 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:11:28 smithi183 ceph-mon[29736]: from='mgr.14556 ' entity='mgr.y' 2024-09-10T19:11:28.372 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:11:28 smithi183 ceph-mon[29736]: from='mgr.14556 ' entity='mgr.y' 2024-09-10T19:11:28.372 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:11:28 smithi183 ceph-mon[29736]: from='mgr.14556 ' entity='mgr.y' 2024-09-10T19:11:28.373 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:11:28 smithi183 ceph-mon[29736]: from='mgr.14556 ' entity='mgr.y' 2024-09-10T19:11:28.373 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:11:28 smithi183 ceph-mon[29736]: Deploying daemon node-exporter.a on smithi183 2024-09-10T19:11:28.373 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:11:28 smithi183 ceph-mon[41562]: 3.2 scrub starts 2024-09-10T19:11:28.373 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:11:28 smithi183 ceph-mon[41562]: 3.2 scrub ok 2024-09-10T19:11:28.373 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:11:28 smithi183 ceph-mon[41562]: 5.1b scrub starts 2024-09-10T19:11:28.373 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:11:28 smithi183 ceph-mon[41562]: 5.1b scrub ok 2024-09-10T19:11:28.373 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:11:28 smithi183 ceph-mon[41562]: pgmap v32: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 642 B/s rd, 0 op/s 2024-09-10T19:11:28.373 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:11:28 smithi183 ceph-mon[41562]: from='mgr.14556 ' entity='mgr.y' 2024-09-10T19:11:28.373 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:11:28 smithi183 ceph-mon[41562]: from='mgr.14556 ' entity='mgr.y' 2024-09-10T19:11:28.373 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:11:28 smithi183 ceph-mon[41562]: from='mgr.14556 ' entity='mgr.y' 2024-09-10T19:11:28.373 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:11:28 smithi183 ceph-mon[41562]: from='mgr.14556 ' entity='mgr.y' 2024-09-10T19:11:28.374 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:11:28 smithi183 ceph-mon[41562]: Deploying daemon node-exporter.a on smithi183 2024-09-10T19:11:28.583 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:28 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:28.29374353Z level=info msg="Executing migration" id="add index in alert_rule on org_id, dashboard_uid and panel_id columns" 2024-09-10T19:11:28.584 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:28 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:28.296520061Z level=info msg="Executing migration" id="add rule_group_idx column to alert_rule" 2024-09-10T19:11:28.584 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:28 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:28.299304197Z level=info msg="Executing migration" id="add is_paused column to alert_rule table" 2024-09-10T19:11:28.584 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:28 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:28.302133956Z level=info msg="Executing migration" id="fix is_paused column for alert_rule table" 2024-09-10T19:11:28.584 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:28 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:28.305388191Z level=info msg="Executing migration" id="create alert_rule_version table" 2024-09-10T19:11:28.584 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:28 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:28.307975358Z level=info msg="Executing migration" id="add index in alert_rule_version table on rule_org_id, rule_uid and version columns" 2024-09-10T19:11:28.584 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:28 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:28.310600495Z level=info msg="Executing migration" id="add index in alert_rule_version table on rule_org_id, rule_namespace_uid and rule_group columns" 2024-09-10T19:11:28.584 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:28 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:28.313496498Z level=info msg="Executing migration" id="alter alert_rule_version table data column to mediumtext in mysql" 2024-09-10T19:11:28.584 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:28 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:28.313874414Z level=info msg="Executing migration" id="add column for to alert_rule_version" 2024-09-10T19:11:28.584 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:28 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:28.316719104Z level=info msg="Executing migration" id="add column annotations to alert_rule_version" 2024-09-10T19:11:28.584 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:28 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:28.319666197Z level=info msg="Executing migration" id="add column labels to alert_rule_version" 2024-09-10T19:11:28.584 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:28 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:28.322412688Z level=info msg="Executing migration" id="add rule_group_idx column to alert_rule_version" 2024-09-10T19:11:28.585 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:28 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:28.325236566Z level=info msg="Executing migration" id="add is_paused column to alert_rule_versions table" 2024-09-10T19:11:28.585 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:28 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:28.328293552Z level=info msg="Executing migration" id="fix is_paused column for alert_rule_version table" 2024-09-10T19:11:28.585 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:28 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:28.331143724Z level=info msg="Executing migration" id=create_alert_configuration_table 2024-09-10T19:11:28.585 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:28 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:28.334055852Z level=info msg="Executing migration" id="Add column default in alert_configuration" 2024-09-10T19:11:28.585 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:28 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:28.336914232Z level=info msg="Executing migration" id="alert alert_configuration alertmanager_configuration column from TEXT to MEDIUMTEXT if mysql" 2024-09-10T19:11:28.585 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:28 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:28.339969672Z level=info msg="Executing migration" id="add column org_id in alert_configuration" 2024-09-10T19:11:28.585 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:28 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:28.342615647Z level=info msg="Executing migration" id="add index in alert_configuration table on org_id column" 2024-09-10T19:11:28.585 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:28 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:28.345422513Z level=info msg="Executing migration" id="add configuration_hash column to alert_configuration" 2024-09-10T19:11:28.585 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:28 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:28.346328591Z level=info msg="Executing migration" id=create_ngalert_configuration_table 2024-09-10T19:11:28.585 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:28 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:28.349166357Z level=info msg="Executing migration" id="add index in ngalert_configuration on org_id column" 2024-09-10T19:11:28.585 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:28 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:28.351908734Z level=info msg="Executing migration" id="add column send_alerts_to in ngalert_configuration" 2024-09-10T19:11:28.585 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:28 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:28.354665817Z level=info msg="Executing migration" id="create provenance_type table" 2024-09-10T19:11:28.585 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:28 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:28.355077908Z level=info msg="Executing migration" id="add index to uniquify (record_key, record_type, org_id) columns" 2024-09-10T19:11:28.586 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:28 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:28.357954292Z level=info msg="Executing migration" id="create alert_image table" 2024-09-10T19:11:28.586 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:28 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:28.360474498Z level=info msg="Executing migration" id="add unique index on token to alert_image table" 2024-09-10T19:11:28.586 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:28 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:28.363196548Z level=info msg="Executing migration" id="support longer URLs in alert_image table" 2024-09-10T19:11:28.586 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:28 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:28.363516487Z level=info msg="Executing migration" id=create_alert_configuration_history_table 2024-09-10T19:11:28.586 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:28 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:28.366243837Z level=info msg="Executing migration" id="drop non-unique orgID index on alert_configuration" 2024-09-10T19:11:28.586 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:28 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:28.366666704Z level=info msg="Executing migration" id="drop unique orgID index on alert_configuration if exists" 2024-09-10T19:11:28.586 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:28 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:28.366737768Z level=warn msg="Skipping migration: Already executed, but not recorded in migration log" id="drop unique orgID index on alert_configuration if exists" 2024-09-10T19:11:28.586 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:28 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:28.369494561Z level=info msg="Executing migration" id="extract alertmanager configuration history to separate table" 2024-09-10T19:11:28.586 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:28 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:28.369871629Z level=info msg="Executing migration" id="add unique index on orgID to alert_configuration" 2024-09-10T19:11:28.586 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:28 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:28.372660283Z level=info msg="Executing migration" id="clear migration entry \"remove unified alerting data\"" 2024-09-10T19:11:28.587 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:28 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:28.372794939Z level=info msg="Executing migration" id="move dashboard alerts to unified alerting" 2024-09-10T19:11:28.587 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:28 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:28.372908293Z level=info msg="alerts found to migrate" alerts=0 2024-09-10T19:11:28.587 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:28 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:28.375457359Z level=info msg="Executing migration" id="create library_element table v1" 2024-09-10T19:11:28.588 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:28 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:28.378038848Z level=info msg="Executing migration" id="add index library_element org_id-folder_id-name-kind" 2024-09-10T19:11:28.588 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:28 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:28.378513613Z level=info msg="Executing migration" id="create library_element_connection table v1" 2024-09-10T19:11:28.588 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:28 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:28.379353108Z level=info msg="Executing migration" id="add index library_element_connection element_id-kind-connection_id" 2024-09-10T19:11:28.588 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:28 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:28.380197176Z level=info msg="Executing migration" id="add unique index library_element org_id_uid" 2024-09-10T19:11:28.588 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:28 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:28.380952578Z level=info msg="Executing migration" id="increase max description length to 2048" 2024-09-10T19:11:28.588 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:28 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:28.381237164Z level=info msg="Executing migration" id="clone move dashboard alerts to unified alerting" 2024-09-10T19:11:28.588 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:28 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:28.382154511Z level=info msg="Executing migration" id="create data_keys table" 2024-09-10T19:11:28.588 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:28 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:28.382831666Z level=info msg="Executing migration" id="create secrets table" 2024-09-10T19:11:28.588 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:28 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:28.383252172Z level=info msg="Executing migration" id="rename data_keys name column to id" 2024-09-10T19:11:28.588 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:28 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:28.38758271Z level=info msg="Executing migration" id="add name column into data_keys" 2024-09-10T19:11:28.588 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:28 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:28.388514016Z level=info msg="Executing migration" id="copy data_keys id column values into name" 2024-09-10T19:11:28.588 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:28 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:28.389489398Z level=info msg="Executing migration" id="rename data_keys name column to label" 2024-09-10T19:11:28.589 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:28 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:28.393808025Z level=info msg="Executing migration" id="rename data_keys id column back to name" 2024-09-10T19:11:28.589 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:28 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:28.39807119Z level=info msg="Executing migration" id="create kv_store table v1" 2024-09-10T19:11:28.589 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:28 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:28.398484695Z level=info msg="Executing migration" id="add index kv_store.org_id-namespace-key" 2024-09-10T19:11:28.589 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:28 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:28.399312711Z level=info msg="Executing migration" id="update dashboard_uid and panel_id from existing annotations" 2024-09-10T19:11:28.589 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:28 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:28.400171704Z level=info msg="Executing migration" id="create permission table" 2024-09-10T19:11:28.589 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:28 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:28.400966677Z level=info msg="Executing migration" id="add unique index permission.role_id" 2024-09-10T19:11:28.589 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:28 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:28.401411515Z level=info msg="Executing migration" id="add unique index role_id_action_scope" 2024-09-10T19:11:28.589 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:28 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:28.402450351Z level=info msg="Executing migration" id="create role table" 2024-09-10T19:11:28.589 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:28 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:28.403170459Z level=info msg="Executing migration" id="add column display_name" 2024-09-10T19:11:28.589 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:28 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:28.404130884Z level=info msg="Executing migration" id="add column group_name" 2024-09-10T19:11:28.589 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:28 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:28.405051364Z level=info msg="Executing migration" id="add index role.org_id" 2024-09-10T19:11:28.589 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:28 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:28.405935339Z level=info msg="Executing migration" id="add unique index role_org_id_name" 2024-09-10T19:11:28.589 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:28 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:28.406852257Z level=info msg="Executing migration" id="add index role_org_id_uid" 2024-09-10T19:11:28.589 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:28 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:28.407661102Z level=info msg="Executing migration" id="create team role table" 2024-09-10T19:11:28.590 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:28 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:28.408514204Z level=info msg="Executing migration" id="add index team_role.org_id" 2024-09-10T19:11:28.590 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:28 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:28.40936228Z level=info msg="Executing migration" id="add unique index team_role_org_id_team_id_role_id" 2024-09-10T19:11:28.590 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:28 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:28.410324147Z level=info msg="Executing migration" id="add index team_role.team_id" 2024-09-10T19:11:28.590 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:28 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:28.411034871Z level=info msg="Executing migration" id="create user role table" 2024-09-10T19:11:28.590 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:28 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:28.411436188Z level=info msg="Executing migration" id="add index user_role.org_id" 2024-09-10T19:11:28.590 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:28 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:28.412432368Z level=info msg="Executing migration" id="add unique index user_role_org_id_user_id_role_id" 2024-09-10T19:11:28.590 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:28 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:28.41312986Z level=info msg="Executing migration" id="add index user_role.user_id" 2024-09-10T19:11:28.590 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:28 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:28.413977105Z level=info msg="Executing migration" id="create builtin role table" 2024-09-10T19:11:28.590 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:28 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:28.414474416Z level=info msg="Executing migration" id="add index builtin_role.role_id" 2024-09-10T19:11:28.590 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:28 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:28.415275925Z level=info msg="Executing migration" id="add index builtin_role.name" 2024-09-10T19:11:28.591 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:28 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:28.416161198Z level=info msg="Executing migration" id="Add column org_id to builtin_role table" 2024-09-10T19:11:28.591 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:28 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:28.417150825Z level=info msg="Executing migration" id="add index builtin_role.org_id" 2024-09-10T19:11:28.591 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:28 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:28.418883553Z level=info msg="Executing migration" id="add unique index builtin_role_org_id_role_id_role" 2024-09-10T19:11:28.591 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:28 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:28.421568388Z level=info msg="Executing migration" id="Remove unique index role_org_id_uid" 2024-09-10T19:11:28.591 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:28 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:28.422036411Z level=info msg="Executing migration" id="add unique index role.uid" 2024-09-10T19:11:28.591 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:28 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:28.422492945Z level=info msg="Executing migration" id="create seed assignment table" 2024-09-10T19:11:28.591 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:28 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:28.422881768Z level=info msg="Executing migration" id="add unique index builtin_role_role_name" 2024-09-10T19:11:28.591 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:28 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:28.42334814Z level=info msg="Executing migration" id="add column hidden to role table" 2024-09-10T19:11:28.591 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:28 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:28.42440297Z level=info msg="Executing migration" id="create query_history table v1" 2024-09-10T19:11:28.591 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:28 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:28.42481508Z level=info msg="Executing migration" id="add index query_history.org_id-created_by-datasource_uid" 2024-09-10T19:11:28.591 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:28 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:28.425273676Z level=info msg="Executing migration" id="alter table query_history alter column created_by type to bigint" 2024-09-10T19:11:28.591 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:28 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:28.425543292Z level=info msg="Executing migration" id="teams permissions migration" 2024-09-10T19:11:28.591 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:28 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:28.425960104Z level=info msg="Executing migration" id="dashboard permissions" 2024-09-10T19:11:28.591 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:28 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:28.426407083Z level=info msg="Executing migration" id="dashboard permissions uid scopes" 2024-09-10T19:11:28.592 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:28 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:28.426736243Z level=info msg="Executing migration" id="drop managed folder create actions" 2024-09-10T19:11:28.592 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:28 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:28.427065924Z level=info msg="Executing migration" id="alerting notification permissions" 2024-09-10T19:11:28.592 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:28 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:28.427381885Z level=info msg="Executing migration" id="create query_history_star table v1" 2024-09-10T19:11:28.592 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:28 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:28.427768172Z level=info msg="Executing migration" id="add index query_history.user_id-query_uid" 2024-09-10T19:11:28.592 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:28 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:28.428215235Z level=info msg="Executing migration" id="add column org_id in query_history_star" 2024-09-10T19:11:28.592 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:28 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:28.429247748Z level=info msg="Executing migration" id="alter table query_history_star_mig column user_id type to bigint" 2024-09-10T19:11:28.592 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:28 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:28.42953911Z level=info msg="Executing migration" id="create correlation table v1" 2024-09-10T19:11:28.592 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:28 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:28.429957125Z level=info msg="Executing migration" id="add index correlations.uid" 2024-09-10T19:11:28.592 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:28 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:28.430372369Z level=info msg="Executing migration" id="add index correlations.source_uid" 2024-09-10T19:11:28.592 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:28 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:28.430793619Z level=info msg="Executing migration" id="add correlation config column" 2024-09-10T19:11:28.592 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:28 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:28.431776667Z level=info msg="Executing migration" id="create entity_events table" 2024-09-10T19:11:28.592 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:28 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:28.432215934Z level=info msg="Executing migration" id="create dashboard public config v1" 2024-09-10T19:11:28.592 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:28 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:28.434884922Z level=info msg="Executing migration" id="drop index UQE_dashboard_public_config_uid - v1" 2024-09-10T19:11:28.592 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:28 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:28.434981924Z level=warn msg="Skipping migration: Already executed, but not recorded in migration log" id="drop index UQE_dashboard_public_config_uid - v1" 2024-09-10T19:11:28.593 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:28 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:28.435257304Z level=info msg="Executing migration" id="drop index IDX_dashboard_public_config_org_id_dashboard_uid - v1" 2024-09-10T19:11:28.593 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:28 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:28.435333597Z level=warn msg="Skipping migration: Already executed, but not recorded in migration log" id="drop index IDX_dashboard_public_config_org_id_dashboard_uid - v1" 2024-09-10T19:11:28.593 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:28 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:28.435598198Z level=info msg="Executing migration" id="Drop old dashboard public config table" 2024-09-10T19:11:28.593 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:28 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:28.436019977Z level=info msg="Executing migration" id="recreate dashboard public config v1" 2024-09-10T19:11:28.593 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:28 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:28.436426905Z level=info msg="Executing migration" id="create index UQE_dashboard_public_config_uid - v1" 2024-09-10T19:11:28.593 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:28 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:28.43925879Z level=info msg="Executing migration" id="create index IDX_dashboard_public_config_org_id_dashboard_uid - v1" 2024-09-10T19:11:28.593 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:28 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:28.442084885Z level=info msg="Executing migration" id="drop index UQE_dashboard_public_config_uid - v2" 2024-09-10T19:11:28.593 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:28 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:28.444968354Z level=info msg="Executing migration" id="drop index IDX_dashboard_public_config_org_id_dashboard_uid - v2" 2024-09-10T19:11:28.593 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:28 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:28.445429617Z level=info msg="Executing migration" id="Drop public config table" 2024-09-10T19:11:28.593 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:28 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:28.448332009Z level=info msg="Executing migration" id="Recreate dashboard public config v2" 2024-09-10T19:11:28.593 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:28 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:28.448998097Z level=info msg="Executing migration" id="create index UQE_dashboard_public_config_uid - v2" 2024-09-10T19:11:28.593 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:28 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:28.450964435Z level=info msg="Executing migration" id="create index IDX_dashboard_public_config_org_id_dashboard_uid - v2" 2024-09-10T19:11:28.593 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:28 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:28.45382526Z level=info msg="Executing migration" id="create index UQE_dashboard_public_config_access_token - v2" 2024-09-10T19:11:28.594 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:28 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:28.454283476Z level=info msg="Executing migration" id="Rename table dashboard_public_config to dashboard_public - v2" 2024-09-10T19:11:28.594 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:28 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:28.457347712Z level=info msg="Executing migration" id="add annotations_enabled column" 2024-09-10T19:11:28.594 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:28 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:28.460200144Z level=info msg="Executing migration" id="add time_selection_enabled column" 2024-09-10T19:11:28.594 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:28 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:28.463010982Z level=info msg="Executing migration" id="delete orphaned public dashboards" 2024-09-10T19:11:28.594 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:28 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:28.463380293Z level=info msg="Executing migration" id="add share column" 2024-09-10T19:11:28.594 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:28 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:28.466286174Z level=info msg="Executing migration" id="create default alerting folders" 2024-09-10T19:11:28.594 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:28 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:28.468867786Z level=info msg="Executing migration" id="create file table" 2024-09-10T19:11:28.594 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:28 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:28.471510467Z level=info msg="Executing migration" id="file table idx: path natural pk" 2024-09-10T19:11:28.594 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:28 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:28.471969174Z level=info msg="Executing migration" id="file table idx: parent_folder_path_hash fast folder retrieval" 2024-09-10T19:11:28.594 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:28 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:28.474776337Z level=info msg="Executing migration" id="create file_meta table" 2024-09-10T19:11:28.594 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:28 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:28.477288001Z level=info msg="Executing migration" id="file table idx: path key" 2024-09-10T19:11:28.594 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:28 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:28.479879673Z level=info msg="Executing migration" id="set path collation in file table" 2024-09-10T19:11:28.594 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:28 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:28.482545753Z level=info msg="Executing migration" id="managed permissions migration" 2024-09-10T19:11:28.595 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:28 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:28.4851153Z level=info msg="Executing migration" id="managed folder permissions alert actions migration" 2024-09-10T19:11:28.595 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:28 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:28.487978279Z level=info msg="Executing migration" id="RBAC action name migrator" 2024-09-10T19:11:28.595 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:28 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:28.490690311Z level=info msg="Executing migration" id="Add UID column to playlist" 2024-09-10T19:11:28.595 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:28 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:28.491740794Z level=info msg="Executing migration" id="Update uid column values in playlist" 2024-09-10T19:11:28.595 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:28 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:28.494529068Z level=info msg="Executing migration" id="Add index for uid in playlist" 2024-09-10T19:11:28.595 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:28 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:28.497059937Z level=info msg="Executing migration" id="update group index for alert rules" 2024-09-10T19:11:28.595 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:28 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:28.499707477Z level=info msg="Executing migration" id="managed folder permissions alert actions repeated migration" 2024-09-10T19:11:28.595 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:28 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:28.500108997Z level=info msg="Executing migration" id="admin only folder/dashboard permission" 2024-09-10T19:11:28.595 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:28 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:28.502826818Z level=info msg="Executing migration" id="add action column to seed_assignment" 2024-09-10T19:11:28.595 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:28 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:28.503900636Z level=info msg="Executing migration" id="add scope column to seed_assignment" 2024-09-10T19:11:28.595 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:28 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:28.505573129Z level=info msg="Executing migration" id="remove unique index builtin_role_role_name before nullable update" 2024-09-10T19:11:28.595 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:28 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:28.506266446Z level=info msg="Executing migration" id="update seed_assignment role_name column to nullable" 2024-09-10T19:11:28.595 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:28 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:28.51619743Z level=info msg="Executing migration" id="add unique index builtin_role_name back" 2024-09-10T19:11:28.595 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:28 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:28.516975083Z level=info msg="Executing migration" id="add unique index builtin_role_action_scope" 2024-09-10T19:11:28.596 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:28 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:28.51792842Z level=info msg="Executing migration" id="add primary key to seed_assigment" 2024-09-10T19:11:28.596 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:28 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:28.521552969Z level=info msg="Executing migration" id="managed folder permissions alert actions repeated fixed migration" 2024-09-10T19:11:28.596 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:28 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:28.522267008Z level=info msg="Executing migration" id="migrate external alertmanagers to datsourcse" 2024-09-10T19:11:28.596 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:28 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:28.523211611Z level=info msg="Executing migration" id="create folder table" 2024-09-10T19:11:28.596 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:28 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:28.523847164Z level=info msg="Executing migration" id="Add index for parent_uid" 2024-09-10T19:11:28.596 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:28 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:28.524796011Z level=info msg="Executing migration" id="Add unique index for folder.uid and folder.org_id" 2024-09-10T19:11:28.596 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:28 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:28.525397007Z level=info msg="Executing migration" id="Update folder title length" 2024-09-10T19:11:28.596 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:28 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:28.526138372Z level=info msg="Executing migration" id="Add unique index for folder.title and folder.parent_uid" 2024-09-10T19:11:28.596 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:28 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=migrator t=2024-09-10T19:11:28.526563788Z level=info msg="migrations completed" performed=485 skipped=0 duration=666.522516ms 2024-09-10T19:11:28.596 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:28 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=sqlstore t=2024-09-10T19:11:28.526986841Z level=info msg="Created default organization" 2024-09-10T19:11:28.596 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:28 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=plugin.loader t=2024-09-10T19:11:28.582325117Z level=info msg="Plugin registered" pluginID=input 2024-09-10T19:11:28.860 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:28 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=plugin.loader t=2024-09-10T19:11:28.610727795Z level=info msg="Plugin registered" pluginID=vonage-status-panel 2024-09-10T19:11:28.860 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:28 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=plugin.loader t=2024-09-10T19:11:28.610746561Z level=info msg="Plugin registered" pluginID=grafana-piechart-panel 2024-09-10T19:11:28.860 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:28 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=secrets t=2024-09-10T19:11:28.610803806Z level=info msg="Envelope encryption state" enabled=true currentprovider=secretKey.v1 2024-09-10T19:11:28.860 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:28 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=query_data t=2024-09-10T19:11:28.612468362Z level=info msg="Query Service initialization" 2024-09-10T19:11:28.860 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:28 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=live.push_http t=2024-09-10T19:11:28.614766822Z level=info msg="Live Push Gateway initialization" 2024-09-10T19:11:29.037 INFO:tasks.cephadm.ceph_manager.ceph:need seq 167503724570 got 167503724574 for osd.5 2024-09-10T19:11:29.038 DEBUG:teuthology.parallel:result is None 2024-09-10T19:11:29.215 INFO:journalctl@ceph.iscsi.iscsi.a.smithi191.stdout:Sep 10 19:11:28 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-iscsi-iscsi-a[61816]: debug there is no tcmu-runner data available 2024-09-10T19:11:29.292 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:11:29 smithi183 ceph-mon[29736]: 4.2 scrub starts 2024-09-10T19:11:29.292 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:11:29 smithi183 ceph-mon[29736]: 4.2 scrub ok 2024-09-10T19:11:29.292 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:11:29 smithi183 ceph-mon[29736]: 6.14 scrub starts 2024-09-10T19:11:29.292 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:11:29 smithi183 ceph-mon[29736]: 6.14 scrub ok 2024-09-10T19:11:29.292 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:11:29 smithi183 ceph-mon[29736]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:11:29.292 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:11:29 smithi183 ceph-mon[29736]: from='mgr.14556 ' entity='mgr.y' 2024-09-10T19:11:29.292 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:11:29 smithi183 ceph-mon[29736]: 5.1e deep-scrub starts 2024-09-10T19:11:29.292 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:11:29 smithi183 ceph-mon[29736]: 5.1e deep-scrub ok 2024-09-10T19:11:29.292 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:11:29 smithi183 ceph-mon[29736]: from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:11:29.293 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:11:29 smithi183 ceph-mon[41562]: 4.2 scrub starts 2024-09-10T19:11:29.293 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:11:29 smithi183 ceph-mon[41562]: 4.2 scrub ok 2024-09-10T19:11:29.293 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:11:29 smithi183 ceph-mon[41562]: 6.14 scrub starts 2024-09-10T19:11:29.293 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:11:29 smithi183 ceph-mon[41562]: 6.14 scrub ok 2024-09-10T19:11:29.293 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:11:29 smithi183 ceph-mon[41562]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:11:29.293 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:11:29 smithi183 ceph-mon[41562]: from='mgr.14556 ' entity='mgr.y' 2024-09-10T19:11:29.293 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:11:29 smithi183 ceph-mon[41562]: 5.1e deep-scrub starts 2024-09-10T19:11:29.293 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:11:29 smithi183 ceph-mon[41562]: 5.1e deep-scrub ok 2024-09-10T19:11:29.293 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:11:29 smithi183 ceph-mon[41562]: from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:11:29.412 INFO:teuthology.orchestra.run.smithi183.stdout:111669149735 2024-09-10T19:11:29.492 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:11:29 smithi191 ceph-mon[32884]: 4.2 scrub starts 2024-09-10T19:11:29.492 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:11:29 smithi191 ceph-mon[32884]: 4.2 scrub ok 2024-09-10T19:11:29.492 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:11:29 smithi191 ceph-mon[32884]: 6.14 scrub starts 2024-09-10T19:11:29.492 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:11:29 smithi191 ceph-mon[32884]: 6.14 scrub ok 2024-09-10T19:11:29.492 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:11:29 smithi191 ceph-mon[32884]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:11:29.492 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:11:29 smithi191 ceph-mon[32884]: from='mgr.14556 ' entity='mgr.y' 2024-09-10T19:11:29.492 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:11:29 smithi191 ceph-mon[32884]: 5.1e deep-scrub starts 2024-09-10T19:11:29.493 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:11:29 smithi191 ceph-mon[32884]: 5.1e deep-scrub ok 2024-09-10T19:11:29.493 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:11:29 smithi191 ceph-mon[32884]: from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:11:29.493 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:29 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=infra.usagestats.collector t=2024-09-10T19:11:29.325194712Z level=info msg="registering usage stat providers" usageStatsProvidersLen=2 2024-09-10T19:11:29.493 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:29 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=provisioning.datasources t=2024-09-10T19:11:29.325624257Z level=warn msg="[Deprecated] the datasource provisioning config is outdated. please upgrade" filename=/etc/grafana/provisioning/datasources/ceph-dashboard.yml 2024-09-10T19:11:29.493 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:29 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=provisioning.datasources t=2024-09-10T19:11:29.358119225Z level=info msg="inserting datasource from configuration " name=Dashboard1 uid=P43CA22E17D0F9596 2024-09-10T19:11:29.493 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:29 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=provisioning.datasources t=2024-09-10T19:11:29.389613078Z level=info msg="inserting datasource from configuration " name=Loki uid=P8E80F9AEF21F6940 2024-09-10T19:11:29.493 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:29 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=provisioning.alerting t=2024-09-10T19:11:29.411129918Z level=info msg="starting to provision alerting" 2024-09-10T19:11:29.493 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:29 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=provisioning.alerting t=2024-09-10T19:11:29.411155113Z level=info msg="finished to provision alerting" 2024-09-10T19:11:29.493 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:29 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=grafanaStorageLogger t=2024-09-10T19:11:29.411378173Z level=info msg="storage starting" 2024-09-10T19:11:29.493 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:29 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=ngalert.state.manager t=2024-09-10T19:11:29.411412171Z level=info msg="Warming state cache for startup" 2024-09-10T19:11:29.493 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:29 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=http.server t=2024-09-10T19:11:29.414917401Z level=info msg="HTTP Server Listen" address=[::]:3000 protocol=https subUrl= socket= 2024-09-10T19:11:29.494 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:29 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=sqlstore.transactions t=2024-09-10T19:11:29.421589228Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=0 code="database is locked" 2024-09-10T19:11:29.494 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:29 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=ngalert.state.manager t=2024-09-10T19:11:29.423686801Z level=info msg="State cache has been initialized" states=0 duration=12.273813ms 2024-09-10T19:11:29.494 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:29 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=ticker t=2024-09-10T19:11:29.42374776Z level=info msg=starting first_tick=2024-09-10T19:11:30Z 2024-09-10T19:11:29.494 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:29 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=ngalert.multiorg.alertmanager t=2024-09-10T19:11:29.423748441Z level=info msg="starting MultiOrg Alertmanager" 2024-09-10T19:11:29.494 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:11:29 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=sqlstore.transactions t=2024-09-10T19:11:29.491234992Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=0 code="database is locked" 2024-09-10T19:11:30.077 INFO:tasks.cephadm.ceph_manager.ceph:need seq 111669149732 got 111669149735 for osd.3 2024-09-10T19:11:30.077 DEBUG:teuthology.parallel:result is None 2024-09-10T19:11:30.077 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-09-10T19:11:30.078 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph pg dump --format=json 2024-09-10T19:11:30.335 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:11:30 smithi183 ceph-mon[29736]: 5.1 scrub starts 2024-09-10T19:11:30.335 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:11:30 smithi183 ceph-mon[29736]: 5.1 scrub ok 2024-09-10T19:11:30.335 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:11:30 smithi183 ceph-mon[29736]: from='client.? 172.21.15.183:0/1509718342' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-09-10T19:11:30.335 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:11:30 smithi183 ceph-mon[29736]: pgmap v33: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:11:30.335 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:11:30 smithi183 ceph-mon[29736]: 2.13 scrub starts 2024-09-10T19:11:30.336 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:11:30 smithi183 ceph-mon[29736]: 2.13 scrub ok 2024-09-10T19:11:30.336 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:11:30 smithi183 ceph-mon[41562]: 5.1 scrub starts 2024-09-10T19:11:30.336 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:11:30 smithi183 ceph-mon[41562]: 5.1 scrub ok 2024-09-10T19:11:30.336 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:11:30 smithi183 ceph-mon[41562]: from='client.? 172.21.15.183:0/1509718342' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-09-10T19:11:30.336 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:11:30 smithi183 ceph-mon[41562]: pgmap v33: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:11:30.336 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:11:30 smithi183 ceph-mon[41562]: 2.13 scrub starts 2024-09-10T19:11:30.336 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:11:30 smithi183 ceph-mon[41562]: 2.13 scrub ok 2024-09-10T19:11:30.358 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:11:30.715 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:11:30 smithi191 ceph-mon[32884]: 5.1 scrub starts 2024-09-10T19:11:30.715 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:11:30 smithi191 ceph-mon[32884]: 5.1 scrub ok 2024-09-10T19:11:30.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:11:30 smithi191 ceph-mon[32884]: from='client.? 172.21.15.183:0/1509718342' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-09-10T19:11:30.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:11:30 smithi191 ceph-mon[32884]: pgmap v33: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:11:30.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:11:30 smithi191 ceph-mon[32884]: 2.13 scrub starts 2024-09-10T19:11:30.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:11:30 smithi191 ceph-mon[32884]: 2.13 scrub ok 2024-09-10T19:11:31.465 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:11:31 smithi191 ceph-mon[32884]: 5.0 scrub starts 2024-09-10T19:11:31.465 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:11:31 smithi191 ceph-mon[32884]: 5.0 scrub ok 2024-09-10T19:11:31.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:11:31 smithi191 ceph-mon[32884]: 6.1d scrub starts 2024-09-10T19:11:31.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:11:31 smithi191 ceph-mon[32884]: 6.1d scrub ok 2024-09-10T19:11:31.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:11:31 smithi191 ceph-mon[32884]: 6.1c deep-scrub starts 2024-09-10T19:11:31.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:11:31 smithi191 ceph-mon[32884]: 6.1c deep-scrub ok 2024-09-10T19:11:31.681 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:11:31 smithi183 ceph-mon[29736]: 5.0 scrub starts 2024-09-10T19:11:31.681 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:11:31 smithi183 ceph-mon[29736]: 5.0 scrub ok 2024-09-10T19:11:31.681 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:11:31 smithi183 ceph-mon[29736]: 6.1d scrub starts 2024-09-10T19:11:31.681 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:11:31 smithi183 ceph-mon[29736]: 6.1d scrub ok 2024-09-10T19:11:31.681 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:11:31 smithi183 ceph-mon[29736]: 6.1c deep-scrub starts 2024-09-10T19:11:31.682 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:11:31 smithi183 ceph-mon[29736]: 6.1c deep-scrub ok 2024-09-10T19:11:31.682 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:11:31 smithi183 ceph-mon[41562]: 5.0 scrub starts 2024-09-10T19:11:31.682 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:11:31 smithi183 ceph-mon[41562]: 5.0 scrub ok 2024-09-10T19:11:31.682 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:11:31 smithi183 ceph-mon[41562]: 6.1d scrub starts 2024-09-10T19:11:31.682 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:11:31 smithi183 ceph-mon[41562]: 6.1d scrub ok 2024-09-10T19:11:31.682 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:11:31 smithi183 ceph-mon[41562]: 6.1c deep-scrub starts 2024-09-10T19:11:31.682 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:11:31 smithi183 ceph-mon[41562]: 6.1c deep-scrub ok 2024-09-10T19:11:32.368 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:11:32 smithi183 ceph-mon[29736]: 5.1f scrub starts 2024-09-10T19:11:32.369 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:11:32 smithi183 ceph-mon[29736]: 5.1f scrub ok 2024-09-10T19:11:32.369 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:11:32 smithi183 ceph-mon[29736]: 4.1a scrub starts 2024-09-10T19:11:32.369 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:11:32 smithi183 ceph-mon[29736]: 4.1a scrub ok 2024-09-10T19:11:32.369 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:11:32 smithi183 ceph-mon[29736]: 5.12 scrub starts 2024-09-10T19:11:32.369 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:11:32 smithi183 ceph-mon[29736]: 5.12 scrub ok 2024-09-10T19:11:32.369 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:11:32 smithi183 ceph-mon[29736]: pgmap v34: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-10T19:11:32.369 INFO:journalctl@ceph.node-exporter.a.smithi183.stdout:Sep 10 19:11:32 smithi183 bash[78288]: Copying config sha256:0da6a335fe1356545476b749c68f022c897de3a2139e8f0054f6937349ee2b83 2024-09-10T19:11:32.370 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:11:32 smithi183 ceph-mon[41562]: 5.1f scrub starts 2024-09-10T19:11:32.370 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:11:32 smithi183 ceph-mon[41562]: 5.1f scrub ok 2024-09-10T19:11:32.370 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:11:32 smithi183 ceph-mon[41562]: 4.1a scrub starts 2024-09-10T19:11:32.370 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:11:32 smithi183 ceph-mon[41562]: 4.1a scrub ok 2024-09-10T19:11:32.370 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:11:32 smithi183 ceph-mon[41562]: 5.12 scrub starts 2024-09-10T19:11:32.370 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:11:32 smithi183 ceph-mon[41562]: 5.12 scrub ok 2024-09-10T19:11:32.370 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:11:32 smithi183 ceph-mon[41562]: pgmap v34: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-10T19:11:32.630 INFO:journalctl@ceph.node-exporter.a.smithi183.stdout:Sep 10 19:11:32 smithi183 bash[78288]: Writing manifest to image destination 2024-09-10T19:11:32.715 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:11:32 smithi191 ceph-mon[32884]: 5.1f scrub starts 2024-09-10T19:11:32.715 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:11:32 smithi191 ceph-mon[32884]: 5.1f scrub ok 2024-09-10T19:11:32.715 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:11:32 smithi191 ceph-mon[32884]: 4.1a scrub starts 2024-09-10T19:11:32.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:11:32 smithi191 ceph-mon[32884]: 4.1a scrub ok 2024-09-10T19:11:32.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:11:32 smithi191 ceph-mon[32884]: 5.12 scrub starts 2024-09-10T19:11:32.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:11:32 smithi191 ceph-mon[32884]: 5.12 scrub ok 2024-09-10T19:11:32.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:11:32 smithi191 ceph-mon[32884]: pgmap v34: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-10T19:11:33.038 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-09-10T19:11:33.040 INFO:teuthology.orchestra.run.smithi183.stderr:dumped all 2024-09-10T19:11:33.172 INFO:journalctl@ceph.node-exporter.a.smithi183.stdout:Sep 10 19:11:33 smithi183 podman[78288]: 2024-09-10 19:11:32.964844391 +0000 UTC m=+2.740857917 image pull 0da6a335fe1356545476b749c68f022c897de3a2139e8f0054f6937349ee2b83 quay.io/prometheus/node-exporter:v1.5.0 2024-09-10T19:11:33.172 INFO:journalctl@ceph.node-exporter.a.smithi183.stdout:Sep 10 19:11:33 smithi183 podman[78288]: 2024-09-10 19:11:33.063399575 +0000 UTC m=+2.839413090 container create 518c3989dce0713f1a8b2dd14da9174ec8c56045558be06bf3147ed2c8c4f680 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-node-exporter-a, maintainer=The Prometheus Authors ) 2024-09-10T19:11:33.431 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:11:33 smithi183 ceph-mon[41562]: 5.11 deep-scrub starts 2024-09-10T19:11:33.431 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:11:33 smithi183 ceph-mon[41562]: 5.11 deep-scrub ok 2024-09-10T19:11:33.431 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:11:33 smithi183 ceph-mon[41562]: 6.1a scrub starts 2024-09-10T19:11:33.431 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:11:33 smithi183 ceph-mon[41562]: 6.1a scrub ok 2024-09-10T19:11:33.431 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:11:33 smithi183 ceph-mon[41562]: 2.11 scrub starts 2024-09-10T19:11:33.431 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:11:33 smithi183 ceph-mon[41562]: 2.11 scrub ok 2024-09-10T19:11:33.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:11:33 smithi183 ceph-mon[41562]: from='client.24602 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-10T19:11:33.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:11:33 smithi183 ceph-mon[29736]: 5.11 deep-scrub starts 2024-09-10T19:11:33.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:11:33 smithi183 ceph-mon[29736]: 5.11 deep-scrub ok 2024-09-10T19:11:33.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:11:33 smithi183 ceph-mon[29736]: 6.1a scrub starts 2024-09-10T19:11:33.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:11:33 smithi183 ceph-mon[29736]: 6.1a scrub ok 2024-09-10T19:11:33.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:11:33 smithi183 ceph-mon[29736]: 2.11 scrub starts 2024-09-10T19:11:33.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:11:33 smithi183 ceph-mon[29736]: 2.11 scrub ok 2024-09-10T19:11:33.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:11:33 smithi183 ceph-mon[29736]: from='client.24602 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-10T19:11:33.715 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:11:33 smithi191 ceph-mon[32884]: 5.11 deep-scrub starts 2024-09-10T19:11:33.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:11:33 smithi191 ceph-mon[32884]: 5.11 deep-scrub ok 2024-09-10T19:11:33.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:11:33 smithi191 ceph-mon[32884]: 6.1a scrub starts 2024-09-10T19:11:33.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:11:33 smithi191 ceph-mon[32884]: 6.1a scrub ok 2024-09-10T19:11:33.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:11:33 smithi191 ceph-mon[32884]: 2.11 scrub starts 2024-09-10T19:11:33.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:11:33 smithi191 ceph-mon[32884]: 2.11 scrub ok 2024-09-10T19:11:33.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:11:33 smithi191 ceph-mon[32884]: from='client.24602 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-10T19:11:33.796 INFO:teuthology.orchestra.run.smithi183.stdout:{"pg_ready":true,"pg_map":{"version":34,"stamp":"2024-09-10T19:11:31.601901+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596188,"num_objects":196,"num_object_clones":0,"num_object_copies":588,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":196,"num_whiteouts":0,"num_read":927,"num_read_kb":784,"num_write":543,"num_write_kb":1671,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":565,"ondisk_log_size":565,"up":396,"acting":396,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":396,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":2387576,"kb_used_data":8760,"kb_used_omap":13,"kb_used_meta":2378738,"kb_avail":747573640,"statfs":{"total":767960285184,"available":765515407360,"internally_reserved":0,"allocated":8970240,"data_stored":4963216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13632,"internal_metadata":2435828416},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.002338"},"pg_stats":[{"pgid":"6.1b","version":"0'0","reported_seq":24,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-09-10T19:11:21.194696+0000","last_change":"2024-09-10T19:10:14.056737+0000","last_active":"2024-09-10T19:11:21.194696+0000","last_peered":"2024-09-10T19:11:21.194696+0000","last_clean":"2024-09-10T19:11:21.194696+0000","last_became_active":"2024-09-10T19:10:14.056187+0000","last_became_peered":"2024-09-10T19:10:14.056187+0000","last_unstale":"2024-09-10T19:11:21.194696+0000","last_undegraded":"2024-09-10T19:11:21.194696+0000","last_fullsized":"2024-09-10T19:11:21.194696+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:10:12.693598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:11:30.090418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":30,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-09-10T19:11:21.302847+0000","last_change":"2024-09-10T19:10:07.688713+0000","last_active":"2024-09-10T19:11:21.302847+0000","last_peered":"2024-09-10T19:11:21.302847+0000","last_clean":"2024-09-10T19:11:21.302847+0000","last_became_active":"2024-09-10T19:10:07.688372+0000","last_became_peered":"2024-09-10T19:10:07.688372+0000","last_unstale":"2024-09-10T19:11:21.302847+0000","last_undegraded":"2024-09-10T19:11:21.302847+0000","last_fullsized":"2024-09-10T19:11:21.302847+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:10:06.664709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:11:33.189992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.19","version":"66'15","reported_seq":53,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-09-10T19:11:21.194822+0000","last_change":"2024-09-10T19:10:09.688886+0000","last_active":"2024-09-10T19:11:21.194822+0000","last_peered":"2024-09-10T19:11:21.194822+0000","last_clean":"2024-09-10T19:11:21.194822+0000","last_became_active":"2024-09-10T19:10:09.688432+0000","last_became_peered":"2024-09-10T19:10:09.688432+0000","last_unstale":"2024-09-10T19:11:21.194822+0000","last_undegraded":"2024-09-10T19:11:21.194822+0000","last_fullsized":"2024-09-10T19:11:21.194822+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:10:08.671214+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:11:30.404066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":36,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-09-10T19:11:23.086323+0000","last_change":"2024-09-10T19:11:23.086288+0000","last_active":"2024-09-10T19:11:23.086323+0000","last_peered":"2024-09-10T19:11:23.086323+0000","last_clean":"2024-09-10T19:11:23.086323+0000","last_became_active":"2024-09-10T19:10:11.697523+0000","last_became_peered":"2024-09-10T19:10:11.697523+0000","last_unstale":"2024-09-10T19:11:23.086323+0000","last_undegraded":"2024-09-10T19:11:23.086323+0000","last_fullsized":"2024-09-10T19:11:23.086323+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:11:23.086191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:11:23.086191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:12:31.131189+0000","scrub_duration":0.0016087969999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"59'1","reported_seq":48,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-09-10T19:11:21.195475+0000","last_change":"2024-09-10T19:11:15.782493+0000","last_active":"2024-09-10T19:11:21.195475+0000","last_peered":"2024-09-10T19:11:21.195475+0000","last_clean":"2024-09-10T19:11:21.195475+0000","last_became_active":"2024-09-10T19:10:07.686143+0000","last_became_peered":"2024-09-10T19:10:07.686143+0000","last_unstale":"2024-09-10T19:11:21.195475+0000","last_undegraded":"2024-09-10T19:11:21.195475+0000","last_fullsized":"2024-09-10T19:11:21.195475+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'1","last_scrub_stamp":"2024-09-10T19:11:15.782393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:11:15.782393+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:12:21.959971+0000","scrub_duration":0.00183947,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":874,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"66'9","reported_seq":44,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-09-10T19:11:21.815200+0000","last_change":"2024-09-10T19:10:09.692950+0000","last_active":"2024-09-10T19:11:21.815200+0000","last_peered":"2024-09-10T19:11:21.815200+0000","last_clean":"2024-09-10T19:11:21.815200+0000","last_became_active":"2024-09-10T19:10:09.692681+0000","last_became_peered":"2024-09-10T19:10:09.692681+0000","last_unstale":"2024-09-10T19:11:21.815200+0000","last_undegraded":"2024-09-10T19:11:21.815200+0000","last_fullsized":"2024-09-10T19:11:21.815200+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:10:08.671214+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:11:38.074554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":26,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-09-10T19:11:21.189235+0000","last_change":"2024-09-10T19:10:11.702830+0000","last_active":"2024-09-10T19:11:21.189235+0000","last_peered":"2024-09-10T19:11:21.189235+0000","last_clean":"2024-09-10T19:11:21.189235+0000","last_became_active":"2024-09-10T19:10:11.702546+0000","last_became_peered":"2024-09-10T19:10:11.702546+0000","last_unstale":"2024-09-10T19:11:21.189235+0000","last_undegraded":"2024-09-10T19:11:21.189235+0000","last_fullsized":"2024-09-10T19:11:21.189235+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:10:10.676876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:11:31.938904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":24,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-09-10T19:11:21.816873+0000","last_change":"2024-09-10T19:10:13.716342+0000","last_active":"2024-09-10T19:11:21.816873+0000","last_peered":"2024-09-10T19:11:21.816873+0000","last_clean":"2024-09-10T19:11:21.816873+0000","last_became_active":"2024-09-10T19:10:13.716022+0000","last_became_peered":"2024-09-10T19:10:13.716022+0000","last_unstale":"2024-09-10T19:11:21.816873+0000","last_undegraded":"2024-09-10T19:11:21.816873+0000","last_fullsized":"2024-09-10T19:11:21.816873+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:10:12.693598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":40,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-09-10T19:11:21.301148+0000","last_change":"2024-09-10T19:11:19.275407+0000","last_active":"2024-09-10T19:11:21.301148+0000","last_peered":"2024-09-10T19:11:21.301148+0000","last_clean":"2024-09-10T19:11:21.301148+0000","last_became_active":"2024-09-10T19:10:07.690728+0000","last_became_peered":"2024-09-10T19:10:07.690728+0000","last_unstale":"2024-09-10T19:11:21.301148+0000","last_undegraded":"2024-09-10T19:11:21.301148+0000","last_fullsized":"2024-09-10T19:11:21.301148+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:11:19.275331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:11:19.275331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:12:30.648634+0000","scrub_duration":0.001355258,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1b","version":"66'5","reported_seq":48,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-10T19:11:24.101637+0000","last_change":"2024-09-10T19:11:24.101604+0000","last_active":"2024-09-10T19:11:24.101637+0000","last_peered":"2024-09-10T19:11:24.101637+0000","last_clean":"2024-09-10T19:11:24.101637+0000","last_became_active":"2024-09-10T19:10:09.693690+0000","last_became_peered":"2024-09-10T19:10:09.693690+0000","last_unstale":"2024-09-10T19:11:24.101637+0000","last_undegraded":"2024-09-10T19:11:24.101637+0000","last_fullsized":"2024-09-10T19:11:24.101637+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'5","last_scrub_stamp":"2024-09-10T19:11:24.101551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:11:24.101551+0000","objects_scrubbed":2,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:12:31.538341+0000","scrub_duration":0.0019539829999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":26,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-09-10T19:11:21.300812+0000","last_change":"2024-09-10T19:10:11.697342+0000","last_active":"2024-09-10T19:11:21.300812+0000","last_peered":"2024-09-10T19:11:21.300812+0000","last_clean":"2024-09-10T19:11:21.300812+0000","last_became_active":"2024-09-10T19:10:11.697088+0000","last_became_peered":"2024-09-10T19:10:11.697088+0000","last_unstale":"2024-09-10T19:11:21.300812+0000","last_undegraded":"2024-09-10T19:11:21.300812+0000","last_fullsized":"2024-09-10T19:11:21.300812+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:10:10.676876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:11:27.002523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":24,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-09-10T19:11:21.186715+0000","last_change":"2024-09-10T19:10:13.724478+0000","last_active":"2024-09-10T19:11:21.186715+0000","last_peered":"2024-09-10T19:11:21.186715+0000","last_clean":"2024-09-10T19:11:21.186715+0000","last_became_active":"2024-09-10T19:10:13.724255+0000","last_became_peered":"2024-09-10T19:10:13.724255+0000","last_unstale":"2024-09-10T19:11:21.186715+0000","last_undegraded":"2024-09-10T19:11:21.186715+0000","last_fullsized":"2024-09-10T19:11:21.186715+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:10:12.693598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:11:28.931682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":30,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-09-10T19:11:21.301209+0000","last_change":"2024-09-10T19:10:07.690458+0000","last_active":"2024-09-10T19:11:21.301209+0000","last_peered":"2024-09-10T19:11:21.301209+0000","last_clean":"2024-09-10T19:11:21.301209+0000","last_became_active":"2024-09-10T19:10:07.690174+0000","last_became_peered":"2024-09-10T19:10:07.690174+0000","last_unstale":"2024-09-10T19:11:21.301209+0000","last_undegraded":"2024-09-10T19:11:21.301209+0000","last_fullsized":"2024-09-10T19:11:21.301209+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:10:06.664709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1a","version":"66'9","reported_seq":44,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-09-10T19:11:21.815695+0000","last_change":"2024-09-10T19:10:09.689170+0000","last_active":"2024-09-10T19:11:21.815695+0000","last_peered":"2024-09-10T19:11:21.815695+0000","last_clean":"2024-09-10T19:11:21.815695+0000","last_became_active":"2024-09-10T19:10:09.688987+0000","last_became_peered":"2024-09-10T19:10:09.688987+0000","last_unstale":"2024-09-10T19:11:21.815695+0000","last_undegraded":"2024-09-10T19:11:21.815695+0000","last_fullsized":"2024-09-10T19:11:21.815695+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:10:08.671214+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":36,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-10T19:11:27.317920+0000","last_change":"2024-09-10T19:11:27.317868+0000","last_active":"2024-09-10T19:11:27.317920+0000","last_peered":"2024-09-10T19:11:27.317920+0000","last_clean":"2024-09-10T19:11:27.317920+0000","last_became_active":"2024-09-10T19:10:11.701799+0000","last_became_peered":"2024-09-10T19:10:11.701799+0000","last_unstale":"2024-09-10T19:11:27.317920+0000","last_undegraded":"2024-09-10T19:11:27.317920+0000","last_fullsized":"2024-09-10T19:11:27.317920+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:11:27.317779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:11:27.317779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:12:44.343114+0000","scrub_duration":0.001509584,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":34,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-09-10T19:11:21.302551+0000","last_change":"2024-09-10T19:11:12.970227+0000","last_active":"2024-09-10T19:11:21.302551+0000","last_peered":"2024-09-10T19:11:21.302551+0000","last_clean":"2024-09-10T19:11:21.302551+0000","last_became_active":"2024-09-10T19:10:13.717310+0000","last_became_peered":"2024-09-10T19:10:13.717310+0000","last_unstale":"2024-09-10T19:11:21.302551+0000","last_undegraded":"2024-09-10T19:11:21.302551+0000","last_fullsized":"2024-09-10T19:11:21.302551+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:11:12.970173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:11:12.970173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:12:27.060515+0000","scrub_duration":0.001527219,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.d","version":"66'17","reported_seq":66,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-09-10T19:11:21.815465+0000","last_change":"2024-09-10T19:11:14.007973+0000","last_active":"2024-09-10T19:11:21.815465+0000","last_peered":"2024-09-10T19:11:21.815465+0000","last_clean":"2024-09-10T19:11:21.815465+0000","last_became_active":"2024-09-10T19:10:09.694016+0000","last_became_peered":"2024-09-10T19:10:09.694016+0000","last_unstale":"2024-09-10T19:11:21.815465+0000","last_undegraded":"2024-09-10T19:11:21.815465+0000","last_fullsized":"2024-09-10T19:11:21.815465+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'17","last_scrub_stamp":"2024-09-10T19:11:14.007902+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:11:14.007902+0000","objects_scrubbed":8,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:12:21.164915+0000","scrub_duration":0.0024520879999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":40,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-09-10T19:11:22.280126+0000","last_change":"2024-09-10T19:11:22.280067+0000","last_active":"2024-09-10T19:11:22.280126+0000","last_peered":"2024-09-10T19:11:22.280126+0000","last_clean":"2024-09-10T19:11:22.280126+0000","last_became_active":"2024-09-10T19:10:07.691119+0000","last_became_peered":"2024-09-10T19:10:07.691119+0000","last_unstale":"2024-09-10T19:11:22.280126+0000","last_undegraded":"2024-09-10T19:11:22.280126+0000","last_fullsized":"2024-09-10T19:11:22.280126+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:11:22.280002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:11:22.280002+0000","last_clean_scrub_stamp":"2024-09-10T19:11:22.280002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:12:27.848752+0000","scrub_duration":0.0013828040000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":26,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-09-10T19:11:21.189594+0000","last_change":"2024-09-10T19:10:11.695393+0000","last_active":"2024-09-10T19:11:21.189594+0000","last_peered":"2024-09-10T19:11:21.189594+0000","last_clean":"2024-09-10T19:11:21.189594+0000","last_became_active":"2024-09-10T19:10:11.695141+0000","last_became_peered":"2024-09-10T19:10:11.695141+0000","last_unstale":"2024-09-10T19:11:21.189594+0000","last_undegraded":"2024-09-10T19:11:21.189594+0000","last_fullsized":"2024-09-10T19:11:21.189594+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:10:10.676876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:11:37.843556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":34,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-10T19:11:24.145082+0000","last_change":"2024-09-10T19:11:24.145056+0000","last_active":"2024-09-10T19:11:24.145082+0000","last_peered":"2024-09-10T19:11:24.145082+0000","last_clean":"2024-09-10T19:11:24.145082+0000","last_became_active":"2024-09-10T19:10:13.722894+0000","last_became_peered":"2024-09-10T19:10:13.722894+0000","last_unstale":"2024-09-10T19:11:24.145082+0000","last_undegraded":"2024-09-10T19:11:24.145082+0000","last_fullsized":"2024-09-10T19:11:24.145082+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:11:24.145003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:11:24.145003+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:12:26.405847+0000","scrub_duration":0.0013083789999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.c","version":"66'10","reported_seq":53,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-09-10T19:11:21.816215+0000","last_change":"2024-09-10T19:11:14.987470+0000","last_active":"2024-09-10T19:11:21.816215+0000","last_peered":"2024-09-10T19:11:21.816215+0000","last_clean":"2024-09-10T19:11:21.816215+0000","last_became_active":"2024-09-10T19:10:09.688321+0000","last_became_peered":"2024-09-10T19:10:09.688321+0000","last_unstale":"2024-09-10T19:11:21.816215+0000","last_undegraded":"2024-09-10T19:11:21.816215+0000","last_fullsized":"2024-09-10T19:11:21.816215+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'10","last_scrub_stamp":"2024-09-10T19:11:14.987403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:11:14.987403+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:12:21.734174+0000","scrub_duration":0.0018747600000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":40,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-10T19:11:26.110254+0000","last_change":"2024-09-10T19:11:26.110216+0000","last_active":"2024-09-10T19:11:26.110254+0000","last_peered":"2024-09-10T19:11:26.110254+0000","last_clean":"2024-09-10T19:11:26.110254+0000","last_became_active":"2024-09-10T19:10:07.680803+0000","last_became_peered":"2024-09-10T19:10:07.680803+0000","last_unstale":"2024-09-10T19:11:26.110254+0000","last_undegraded":"2024-09-10T19:11:26.110254+0000","last_fullsized":"2024-09-10T19:11:26.110254+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:11:26.110138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:11:26.110138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:12:35.580206+0000","scrub_duration":0.0011581549999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.d","version":"80'9","reported_seq":64,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-10T19:11:25.156549+0000","last_change":"2024-09-10T19:11:25.156512+0000","last_active":"2024-09-10T19:11:25.156549+0000","last_peered":"2024-09-10T19:11:25.156549+0000","last_clean":"2024-09-10T19:11:25.156549+0000","last_became_active":"2024-09-10T19:10:11.698547+0000","last_became_peered":"2024-09-10T19:10:11.698547+0000","last_unstale":"2024-09-10T19:11:25.156549+0000","last_undegraded":"2024-09-10T19:11:25.156549+0000","last_fullsized":"2024-09-10T19:11:25.156549+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"80'9","last_scrub_stamp":"2024-09-10T19:11:25.156465+0000","last_deep_scrub":"80'9","last_deep_scrub_stamp":"2024-09-10T19:11:25.156465+0000","last_clean_scrub_stamp":"2024-09-10T19:11:25.156465+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:12:26.762818+0000","scrub_duration":0.0022299569999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":34,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-10T19:11:26.096176+0000","last_change":"2024-09-10T19:11:26.096147+0000","last_active":"2024-09-10T19:11:26.096176+0000","last_peered":"2024-09-10T19:11:26.096176+0000","last_clean":"2024-09-10T19:11:26.096176+0000","last_became_active":"2024-09-10T19:10:13.718242+0000","last_became_peered":"2024-09-10T19:10:13.718242+0000","last_unstale":"2024-09-10T19:11:26.096176+0000","last_undegraded":"2024-09-10T19:11:26.096176+0000","last_fullsized":"2024-09-10T19:11:26.096176+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:11:26.096090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:11:26.096090+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:12:49.347064+0000","scrub_duration":0.0015599909999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.f","version":"66'15","reported_seq":53,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-09-10T19:11:21.189954+0000","last_change":"2024-09-10T19:10:09.691727+0000","last_active":"2024-09-10T19:11:21.189954+0000","last_peered":"2024-09-10T19:11:21.189954+0000","last_clean":"2024-09-10T19:11:21.189954+0000","last_became_active":"2024-09-10T19:10:09.691230+0000","last_became_peered":"2024-09-10T19:10:09.691230+0000","last_unstale":"2024-09-10T19:11:21.189954+0000","last_undegraded":"2024-09-10T19:11:21.189954+0000","last_fullsized":"2024-09-10T19:11:21.189954+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:10:08.671214+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:11:34.422017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":40,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-09-10T19:11:21.189383+0000","last_change":"2024-09-10T19:11:11.036017+0000","last_active":"2024-09-10T19:11:21.189383+0000","last_peered":"2024-09-10T19:11:21.189383+0000","last_clean":"2024-09-10T19:11:21.189383+0000","last_became_active":"2024-09-10T19:10:07.679646+0000","last_became_peered":"2024-09-10T19:10:07.679646+0000","last_unstale":"2024-09-10T19:11:21.189383+0000","last_undegraded":"2024-09-10T19:11:21.189383+0000","last_fullsized":"2024-09-10T19:11:21.189383+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:11:11.035954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:11:11.035954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:12:35.469086+0000","scrub_duration":0.001618024,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.e","version":"66'8","reported_seq":50,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-10T19:11:23.237861+0000","last_change":"2024-09-10T19:10:11.699487+0000","last_active":"2024-09-10T19:11:23.237861+0000","last_peered":"2024-09-10T19:11:23.237861+0000","last_clean":"2024-09-10T19:11:23.237861+0000","last_became_active":"2024-09-10T19:10:11.699232+0000","last_became_peered":"2024-09-10T19:10:11.699232+0000","last_unstale":"2024-09-10T19:11:23.237861+0000","last_undegraded":"2024-09-10T19:11:23.237861+0000","last_fullsized":"2024-09-10T19:11:23.237861+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:10:10.676876+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:11:36.171473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":34,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-09-10T19:11:21.187406+0000","last_change":"2024-09-10T19:11:18.208847+0000","last_active":"2024-09-10T19:11:21.187406+0000","last_peered":"2024-09-10T19:11:21.187406+0000","last_clean":"2024-09-10T19:11:21.187406+0000","last_became_active":"2024-09-10T19:10:13.715274+0000","last_became_peered":"2024-09-10T19:10:13.715274+0000","last_unstale":"2024-09-10T19:11:21.187406+0000","last_undegraded":"2024-09-10T19:11:21.187406+0000","last_fullsized":"2024-09-10T19:11:21.187406+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:11:18.208782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:11:18.208782+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:12:43.016878+0000","scrub_duration":0.001904481,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.e","version":"66'11","reported_seq":47,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-09-10T19:11:21.816254+0000","last_change":"2024-09-10T19:10:09.692995+0000","last_active":"2024-09-10T19:11:21.816254+0000","last_peered":"2024-09-10T19:11:21.816254+0000","last_clean":"2024-09-10T19:11:21.816254+0000","last_became_active":"2024-09-10T19:10:09.692675+0000","last_became_peered":"2024-09-10T19:10:09.692675+0000","last_unstale":"2024-09-10T19:11:21.816254+0000","last_undegraded":"2024-09-10T19:11:21.816254+0000","last_fullsized":"2024-09-10T19:11:21.816254+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:10:08.671214+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:11:34.182028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":30,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-09-10T19:11:21.300365+0000","last_change":"2024-09-10T19:10:07.690701+0000","last_active":"2024-09-10T19:11:21.300365+0000","last_peered":"2024-09-10T19:11:21.300365+0000","last_clean":"2024-09-10T19:11:21.300365+0000","last_became_active":"2024-09-10T19:10:07.690434+0000","last_became_peered":"2024-09-10T19:10:07.690434+0000","last_unstale":"2024-09-10T19:11:21.300365+0000","last_undegraded":"2024-09-10T19:11:21.300365+0000","last_fullsized":"2024-09-10T19:11:21.300365+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:10:06.664709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:11:28.466436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":26,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-09-10T19:11:21.187698+0000","last_change":"2024-09-10T19:10:11.692483+0000","last_active":"2024-09-10T19:11:21.187698+0000","last_peered":"2024-09-10T19:11:21.187698+0000","last_clean":"2024-09-10T19:11:21.187698+0000","last_became_active":"2024-09-10T19:10:11.692215+0000","last_became_peered":"2024-09-10T19:10:11.692215+0000","last_unstale":"2024-09-10T19:11:21.187698+0000","last_undegraded":"2024-09-10T19:11:21.187698+0000","last_fullsized":"2024-09-10T19:11:21.187698+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:10:10.676876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:11:38.454128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":24,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-09-10T19:11:21.194837+0000","last_change":"2024-09-10T19:10:14.054106+0000","last_active":"2024-09-10T19:11:21.194837+0000","last_peered":"2024-09-10T19:11:21.194837+0000","last_clean":"2024-09-10T19:11:21.194837+0000","last_became_active":"2024-09-10T19:10:14.053736+0000","last_became_peered":"2024-09-10T19:10:14.053736+0000","last_unstale":"2024-09-10T19:11:21.194837+0000","last_undegraded":"2024-09-10T19:11:21.194837+0000","last_fullsized":"2024-09-10T19:11:21.194837+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:10:12.693598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:11:32.848291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"66'14","reported_seq":59,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-09-10T19:11:21.815965+0000","last_change":"2024-09-10T19:11:16.976833+0000","last_active":"2024-09-10T19:11:21.815965+0000","last_peered":"2024-09-10T19:11:21.815965+0000","last_clean":"2024-09-10T19:11:21.815965+0000","last_became_active":"2024-09-10T19:10:09.686534+0000","last_became_peered":"2024-09-10T19:10:09.686534+0000","last_unstale":"2024-09-10T19:11:21.815965+0000","last_undegraded":"2024-09-10T19:11:21.815965+0000","last_fullsized":"2024-09-10T19:11:21.815965+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'14","last_scrub_stamp":"2024-09-10T19:11:16.976776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:11:16.976776+0000","objects_scrubbed":7,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:12:29.125897+0000","scrub_duration":0.001777144,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":40,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-09-10T19:11:22.801241+0000","last_change":"2024-09-10T19:11:22.801187+0000","last_active":"2024-09-10T19:11:22.801241+0000","last_peered":"2024-09-10T19:11:22.801241+0000","last_clean":"2024-09-10T19:11:22.801241+0000","last_became_active":"2024-09-10T19:10:07.684864+0000","last_became_peered":"2024-09-10T19:10:07.684864+0000","last_unstale":"2024-09-10T19:11:22.801241+0000","last_undegraded":"2024-09-10T19:11:22.801241+0000","last_fullsized":"2024-09-10T19:11:22.801241+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:11:22.801123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:11:22.801123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:12:47.860716+0000","scrub_duration":0.0015474410000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.0","version":"66'8","reported_seq":50,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-10T19:11:23.206948+0000","last_change":"2024-09-10T19:10:11.697408+0000","last_active":"2024-09-10T19:11:23.206948+0000","last_peered":"2024-09-10T19:11:23.206948+0000","last_clean":"2024-09-10T19:11:23.206948+0000","last_became_active":"2024-09-10T19:10:11.697106+0000","last_became_peered":"2024-09-10T19:10:11.697106+0000","last_unstale":"2024-09-10T19:11:23.206948+0000","last_undegraded":"2024-09-10T19:11:23.206948+0000","last_fullsized":"2024-09-10T19:11:23.206948+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:10:10.676876+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":24,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-09-10T19:11:21.300549+0000","last_change":"2024-09-10T19:10:14.054712+0000","last_active":"2024-09-10T19:11:21.300549+0000","last_peered":"2024-09-10T19:11:21.300549+0000","last_clean":"2024-09-10T19:11:21.300549+0000","last_became_active":"2024-09-10T19:10:14.054456+0000","last_became_peered":"2024-09-10T19:10:14.054456+0000","last_unstale":"2024-09-10T19:11:21.300549+0000","last_undegraded":"2024-09-10T19:11:21.300549+0000","last_fullsized":"2024-09-10T19:11:21.300549+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:10:12.693598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:11:24.269111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.0","version":"66'18","reported_seq":70,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-09-10T19:11:21.194693+0000","last_change":"2024-09-10T19:11:19.848186+0000","last_active":"2024-09-10T19:11:21.194693+0000","last_peered":"2024-09-10T19:11:21.194693+0000","last_clean":"2024-09-10T19:11:21.194693+0000","last_became_active":"2024-09-10T19:10:09.821836+0000","last_became_peered":"2024-09-10T19:10:09.821836+0000","last_unstale":"2024-09-10T19:11:21.194693+0000","last_undegraded":"2024-09-10T19:11:21.194693+0000","last_fullsized":"2024-09-10T19:11:21.194693+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'18","last_scrub_stamp":"2024-09-10T19:11:19.848132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:11:19.848132+0000","objects_scrubbed":8,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:12:26.692213+0000","scrub_duration":0.0034247819999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":30,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-09-10T19:11:21.189816+0000","last_change":"2024-09-10T19:10:07.682359+0000","last_active":"2024-09-10T19:11:21.189816+0000","last_peered":"2024-09-10T19:11:21.189816+0000","last_clean":"2024-09-10T19:11:21.189816+0000","last_became_active":"2024-09-10T19:10:07.682081+0000","last_became_peered":"2024-09-10T19:10:07.682081+0000","last_unstale":"2024-09-10T19:11:21.189816+0000","last_undegraded":"2024-09-10T19:11:21.189816+0000","last_fullsized":"2024-09-10T19:11:21.189816+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:10:06.664709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:11:25.212065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":36,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-10T19:11:29.069453+0000","last_change":"2024-09-10T19:11:29.069421+0000","last_active":"2024-09-10T19:11:29.069453+0000","last_peered":"2024-09-10T19:11:29.069453+0000","last_clean":"2024-09-10T19:11:29.069453+0000","last_became_active":"2024-09-10T19:10:11.698054+0000","last_became_peered":"2024-09-10T19:10:11.698054+0000","last_unstale":"2024-09-10T19:11:29.069453+0000","last_undegraded":"2024-09-10T19:11:29.069453+0000","last_fullsized":"2024-09-10T19:11:29.069453+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:11:29.069369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:11:29.069369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:12:55.817612+0000","scrub_duration":0.001327562,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":24,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-09-10T19:11:21.816993+0000","last_change":"2024-09-10T19:10:13.723183+0000","last_active":"2024-09-10T19:11:21.816993+0000","last_peered":"2024-09-10T19:11:21.816993+0000","last_clean":"2024-09-10T19:11:21.816993+0000","last_became_active":"2024-09-10T19:10:13.722784+0000","last_became_peered":"2024-09-10T19:10:13.722784+0000","last_unstale":"2024-09-10T19:11:21.816993+0000","last_undegraded":"2024-09-10T19:11:21.816993+0000","last_fullsized":"2024-09-10T19:11:21.816993+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:10:12.693598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:11:27.261267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.3","version":"66'19","reported_seq":64,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-09-10T19:11:21.302758+0000","last_change":"2024-09-10T19:10:09.823263+0000","last_active":"2024-09-10T19:11:21.302758+0000","last_peered":"2024-09-10T19:11:21.302758+0000","last_clean":"2024-09-10T19:11:21.302758+0000","last_became_active":"2024-09-10T19:10:09.823003+0000","last_became_peered":"2024-09-10T19:10:09.823003+0000","last_unstale":"2024-09-10T19:11:21.302758+0000","last_undegraded":"2024-09-10T19:11:21.302758+0000","last_fullsized":"2024-09-10T19:11:21.302758+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:10:08.671214+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:11:34.837626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":40,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-10T19:11:25.290175+0000","last_change":"2024-09-10T19:11:25.290147+0000","last_active":"2024-09-10T19:11:25.290175+0000","last_peered":"2024-09-10T19:11:25.290175+0000","last_clean":"2024-09-10T19:11:25.290175+0000","last_became_active":"2024-09-10T19:10:07.690325+0000","last_became_peered":"2024-09-10T19:10:07.690325+0000","last_unstale":"2024-09-10T19:11:25.290175+0000","last_undegraded":"2024-09-10T19:11:25.290175+0000","last_fullsized":"2024-09-10T19:11:25.290175+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:11:25.290088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:11:25.290088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:12:31.270129+0000","scrub_duration":0.0013977760000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":36,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-10T19:11:23.843357+0000","last_change":"2024-09-10T19:11:23.843315+0000","last_active":"2024-09-10T19:11:23.843357+0000","last_peered":"2024-09-10T19:11:23.843357+0000","last_clean":"2024-09-10T19:11:23.843357+0000","last_became_active":"2024-09-10T19:10:11.700370+0000","last_became_peered":"2024-09-10T19:10:11.700370+0000","last_unstale":"2024-09-10T19:11:23.843357+0000","last_undegraded":"2024-09-10T19:11:23.843357+0000","last_fullsized":"2024-09-10T19:11:23.843357+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:11:23.843228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:11:23.843228+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:12:29.095733+0000","scrub_duration":0.0013052070000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":24,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-09-10T19:11:21.189202+0000","last_change":"2024-09-10T19:10:14.054654+0000","last_active":"2024-09-10T19:11:21.189202+0000","last_peered":"2024-09-10T19:11:21.189202+0000","last_clean":"2024-09-10T19:11:21.189202+0000","last_became_active":"2024-09-10T19:10:14.054318+0000","last_became_peered":"2024-09-10T19:10:14.054318+0000","last_unstale":"2024-09-10T19:11:21.189202+0000","last_undegraded":"2024-09-10T19:11:21.189202+0000","last_fullsized":"2024-09-10T19:11:21.189202+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:10:12.693598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:11:26.731659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"66'10","reported_seq":53,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-10T19:11:28.132964+0000","last_change":"2024-09-10T19:11:28.132910+0000","last_active":"2024-09-10T19:11:28.132964+0000","last_peered":"2024-09-10T19:11:28.132964+0000","last_clean":"2024-09-10T19:11:28.132964+0000","last_became_active":"2024-09-10T19:10:09.689815+0000","last_became_peered":"2024-09-10T19:10:09.689815+0000","last_unstale":"2024-09-10T19:11:28.132964+0000","last_undegraded":"2024-09-10T19:11:28.132964+0000","last_fullsized":"2024-09-10T19:11:28.132964+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'10","last_scrub_stamp":"2024-09-10T19:11:28.132850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:11:28.132850+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:12:42.376407+0000","scrub_duration":0.0018173429999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":30,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-09-10T19:11:21.189857+0000","last_change":"2024-09-10T19:10:07.682243+0000","last_active":"2024-09-10T19:11:21.189857+0000","last_peered":"2024-09-10T19:11:21.189857+0000","last_clean":"2024-09-10T19:11:21.189857+0000","last_became_active":"2024-09-10T19:10:07.681965+0000","last_became_peered":"2024-09-10T19:10:07.681965+0000","last_unstale":"2024-09-10T19:11:21.189857+0000","last_undegraded":"2024-09-10T19:11:21.189857+0000","last_fullsized":"2024-09-10T19:11:21.189857+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:10:06.664709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:11:28.605002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"66'8","reported_seq":50,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-10T19:11:23.236436+0000","last_change":"2024-09-10T19:10:11.699975+0000","last_active":"2024-09-10T19:11:23.236436+0000","last_peered":"2024-09-10T19:11:23.236436+0000","last_clean":"2024-09-10T19:11:23.236436+0000","last_became_active":"2024-09-10T19:10:11.699690+0000","last_became_peered":"2024-09-10T19:10:11.699690+0000","last_unstale":"2024-09-10T19:11:23.236436+0000","last_undegraded":"2024-09-10T19:11:23.236436+0000","last_fullsized":"2024-09-10T19:11:23.236436+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:10:10.676876+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:11:32.470332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":24,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-09-10T19:11:21.302269+0000","last_change":"2024-09-10T19:10:13.720657+0000","last_active":"2024-09-10T19:11:21.302269+0000","last_peered":"2024-09-10T19:11:21.302269+0000","last_clean":"2024-09-10T19:11:21.302269+0000","last_became_active":"2024-09-10T19:10:13.720403+0000","last_became_peered":"2024-09-10T19:10:13.720403+0000","last_unstale":"2024-09-10T19:11:21.302269+0000","last_undegraded":"2024-09-10T19:11:21.302269+0000","last_fullsized":"2024-09-10T19:11:21.302269+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:10:12.693598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:11:28.077386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"66'28","reported_seq":102,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-10T19:11:23.237045+0000","last_change":"2024-09-10T19:11:17.050501+0000","last_active":"2024-09-10T19:11:23.237045+0000","last_peered":"2024-09-10T19:11:23.237045+0000","last_clean":"2024-09-10T19:11:23.237045+0000","last_became_active":"2024-09-10T19:10:09.692663+0000","last_became_peered":"2024-09-10T19:10:09.692663+0000","last_unstale":"2024-09-10T19:11:23.237045+0000","last_undegraded":"2024-09-10T19:11:23.237045+0000","last_fullsized":"2024-09-10T19:11:23.237045+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'28","last_scrub_stamp":"2024-09-10T19:11:17.050442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:11:17.050442+0000","objects_scrubbed":10,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:12:31.661637+0000","scrub_duration":0.001985559,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":30,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-09-10T19:11:21.187081+0000","last_change":"2024-09-10T19:10:07.683032+0000","last_active":"2024-09-10T19:11:21.187081+0000","last_peered":"2024-09-10T19:11:21.187081+0000","last_clean":"2024-09-10T19:11:21.187081+0000","last_became_active":"2024-09-10T19:10:07.682724+0000","last_became_peered":"2024-09-10T19:10:07.682724+0000","last_unstale":"2024-09-10T19:11:21.187081+0000","last_undegraded":"2024-09-10T19:11:21.187081+0000","last_fullsized":"2024-09-10T19:11:21.187081+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:10:06.664709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:11:28.664405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":36,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-09-10T19:11:21.303045+0000","last_change":"2024-09-10T19:11:14.985340+0000","last_active":"2024-09-10T19:11:21.303045+0000","last_peered":"2024-09-10T19:11:21.303045+0000","last_clean":"2024-09-10T19:11:21.303045+0000","last_became_active":"2024-09-10T19:10:11.698825+0000","last_became_peered":"2024-09-10T19:10:11.698825+0000","last_unstale":"2024-09-10T19:11:21.303045+0000","last_undegraded":"2024-09-10T19:11:21.303045+0000","last_fullsized":"2024-09-10T19:11:21.303045+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:11:14.985249+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:11:14.985249+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:12:30.155012+0000","scrub_duration":0.001460161,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":24,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-09-10T19:11:21.195389+0000","last_change":"2024-09-10T19:10:13.722038+0000","last_active":"2024-09-10T19:11:21.195389+0000","last_peered":"2024-09-10T19:11:21.195389+0000","last_clean":"2024-09-10T19:11:21.195389+0000","last_became_active":"2024-09-10T19:10:13.721744+0000","last_became_peered":"2024-09-10T19:10:13.721744+0000","last_unstale":"2024-09-10T19:11:21.195389+0000","last_undegraded":"2024-09-10T19:11:21.195389+0000","last_fullsized":"2024-09-10T19:11:21.195389+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:10:12.693598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:11:42.038674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.7","version":"66'13","reported_seq":55,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-09-10T19:11:21.190003+0000","last_change":"2024-09-10T19:10:09.693625+0000","last_active":"2024-09-10T19:11:21.190003+0000","last_peered":"2024-09-10T19:11:21.190003+0000","last_clean":"2024-09-10T19:11:21.190003+0000","last_became_active":"2024-09-10T19:10:09.693346+0000","last_became_peered":"2024-09-10T19:10:09.693346+0000","last_unstale":"2024-09-10T19:11:21.190003+0000","last_undegraded":"2024-09-10T19:11:21.190003+0000","last_fullsized":"2024-09-10T19:11:21.190003+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:10:08.671214+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:11:26.671345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":19,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"67'5","reported_seq":124,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-10T19:11:27.898869+0000","last_change":"2024-09-10T19:11:16.007355+0000","last_active":"2024-09-10T19:11:27.898869+0000","last_peered":"2024-09-10T19:11:27.898869+0000","last_clean":"2024-09-10T19:11:27.898869+0000","last_became_active":"2024-09-10T19:10:07.682125+0000","last_became_peered":"2024-09-10T19:10:07.682125+0000","last_unstale":"2024-09-10T19:11:27.898869+0000","last_undegraded":"2024-09-10T19:11:27.898869+0000","last_fullsized":"2024-09-10T19:11:27.898869+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'5","last_scrub_stamp":"2024-09-10T19:11:16.007261+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:11:16.007261+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:12:39.490457+0000","scrub_duration":0.0019059680000000001,"objects_trimmed":0,"snaptrim_duration":0.00034454499999999997,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":76,"num_read_kb":71,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":30,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-09-10T19:11:21.815131+0000","last_change":"2024-09-10T19:10:07.682382+0000","last_active":"2024-09-10T19:11:21.815131+0000","last_peered":"2024-09-10T19:11:21.815131+0000","last_clean":"2024-09-10T19:11:21.815131+0000","last_became_active":"2024-09-10T19:10:07.681456+0000","last_became_peered":"2024-09-10T19:10:07.681456+0000","last_unstale":"2024-09-10T19:11:21.815131+0000","last_undegraded":"2024-09-10T19:11:21.815131+0000","last_fullsized":"2024-09-10T19:11:21.815131+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:10:06.664709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:11:24.198210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":26,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-09-10T19:11:21.815541+0000","last_change":"2024-09-10T19:10:11.695520+0000","last_active":"2024-09-10T19:11:21.815541+0000","last_peered":"2024-09-10T19:11:21.815541+0000","last_clean":"2024-09-10T19:11:21.815541+0000","last_became_active":"2024-09-10T19:10:11.695205+0000","last_became_peered":"2024-09-10T19:10:11.695205+0000","last_unstale":"2024-09-10T19:11:21.815541+0000","last_undegraded":"2024-09-10T19:11:21.815541+0000","last_fullsized":"2024-09-10T19:11:21.815541+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:10:10.676876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:11:28.918766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":24,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-09-10T19:11:21.300908+0000","last_change":"2024-09-10T19:10:14.054047+0000","last_active":"2024-09-10T19:11:21.300908+0000","last_peered":"2024-09-10T19:11:21.300908+0000","last_clean":"2024-09-10T19:11:21.300908+0000","last_became_active":"2024-09-10T19:10:14.053697+0000","last_became_peered":"2024-09-10T19:10:14.053697+0000","last_unstale":"2024-09-10T19:11:21.300908+0000","last_undegraded":"2024-09-10T19:11:21.300908+0000","last_fullsized":"2024-09-10T19:11:21.300908+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:10:12.693598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:11:40.555328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"66'12","reported_seq":46,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-09-10T19:11:21.302911+0000","last_change":"2024-09-10T19:10:09.688409+0000","last_active":"2024-09-10T19:11:21.302911+0000","last_peered":"2024-09-10T19:11:21.302911+0000","last_clean":"2024-09-10T19:11:21.302911+0000","last_became_active":"2024-09-10T19:10:09.688091+0000","last_became_peered":"2024-09-10T19:10:09.688091+0000","last_unstale":"2024-09-10T19:11:21.302911+0000","last_undegraded":"2024-09-10T19:11:21.302911+0000","last_fullsized":"2024-09-10T19:11:21.302911+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:10:08.671214+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:11:26.174026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"61'1","reported_seq":36,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-09-10T19:11:21.302500+0000","last_change":"2024-09-10T19:10:11.693049+0000","last_active":"2024-09-10T19:11:21.302500+0000","last_peered":"2024-09-10T19:11:21.302500+0000","last_clean":"2024-09-10T19:11:21.302500+0000","last_became_active":"2024-09-10T19:10:07.683259+0000","last_became_peered":"2024-09-10T19:10:07.683259+0000","last_unstale":"2024-09-10T19:11:21.302500+0000","last_undegraded":"2024-09-10T19:11:21.302500+0000","last_fullsized":"2024-09-10T19:11:21.302500+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:10:06.664709+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:11:33.524399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014341130000000001,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":40,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-09-10T19:11:21.301199+0000","last_change":"2024-09-10T19:11:21.301130+0000","last_active":"2024-09-10T19:11:21.301199+0000","last_peered":"2024-09-10T19:11:21.301199+0000","last_clean":"2024-09-10T19:11:21.301199+0000","last_became_active":"2024-09-10T19:10:07.690549+0000","last_became_peered":"2024-09-10T19:10:07.690549+0000","last_unstale":"2024-09-10T19:11:21.301199+0000","last_undegraded":"2024-09-10T19:11:21.301199+0000","last_fullsized":"2024-09-10T19:11:21.301199+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:11:21.301058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:11:21.301058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:12:41.356053+0000","scrub_duration":0.0021074710000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":36,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-09-10T19:11:21.187730+0000","last_change":"2024-09-10T19:11:16.156929+0000","last_active":"2024-09-10T19:11:21.187730+0000","last_peered":"2024-09-10T19:11:21.187730+0000","last_clean":"2024-09-10T19:11:21.187730+0000","last_became_active":"2024-09-10T19:10:11.699798+0000","last_became_peered":"2024-09-10T19:10:11.699798+0000","last_unstale":"2024-09-10T19:11:21.187730+0000","last_undegraded":"2024-09-10T19:11:21.187730+0000","last_fullsized":"2024-09-10T19:11:21.187730+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:11:16.156819+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:11:16.156819+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:12:16.166187+0000","scrub_duration":0.001649059,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":24,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-09-10T19:11:21.189585+0000","last_change":"2024-09-10T19:10:13.722695+0000","last_active":"2024-09-10T19:11:21.189585+0000","last_peered":"2024-09-10T19:11:21.189585+0000","last_clean":"2024-09-10T19:11:21.189585+0000","last_became_active":"2024-09-10T19:10:13.722368+0000","last_became_peered":"2024-09-10T19:10:13.722368+0000","last_unstale":"2024-09-10T19:11:21.189585+0000","last_undegraded":"2024-09-10T19:11:21.189585+0000","last_fullsized":"2024-09-10T19:11:21.189585+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:10:12.693598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:11:39.250623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"66'14","reported_seq":50,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-09-10T19:11:21.816239+0000","last_change":"2024-09-10T19:10:09.822467+0000","last_active":"2024-09-10T19:11:21.816239+0000","last_peered":"2024-09-10T19:11:21.816239+0000","last_clean":"2024-09-10T19:11:21.816239+0000","last_became_active":"2024-09-10T19:10:09.822184+0000","last_became_peered":"2024-09-10T19:10:09.822184+0000","last_unstale":"2024-09-10T19:11:21.816239+0000","last_undegraded":"2024-09-10T19:11:21.816239+0000","last_fullsized":"2024-09-10T19:11:21.816239+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:10:08.671214+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:11:31.203195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"63'2","reported_seq":47,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-10T19:11:26.824211+0000","last_change":"2024-09-10T19:11:26.824164+0000","last_active":"2024-09-10T19:11:26.824211+0000","last_peered":"2024-09-10T19:11:26.824211+0000","last_clean":"2024-09-10T19:11:26.824211+0000","last_became_active":"2024-09-10T19:10:07.684784+0000","last_became_peered":"2024-09-10T19:10:07.684784+0000","last_unstale":"2024-09-10T19:11:26.824211+0000","last_undegraded":"2024-09-10T19:11:26.824211+0000","last_fullsized":"2024-09-10T19:11:26.824211+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'2","last_scrub_stamp":"2024-09-10T19:11:26.824103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:11:26.824103+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:12:48.229817+0000","scrub_duration":0.0018538929999999999,"objects_trimmed":0,"snaptrim_duration":0.00075522999999999996,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":40,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-09-10T19:11:21.187274+0000","last_change":"2024-09-10T19:11:17.171319+0000","last_active":"2024-09-10T19:11:21.187274+0000","last_peered":"2024-09-10T19:11:21.187274+0000","last_clean":"2024-09-10T19:11:21.187274+0000","last_became_active":"2024-09-10T19:10:07.687596+0000","last_became_peered":"2024-09-10T19:10:07.687596+0000","last_unstale":"2024-09-10T19:11:21.187274+0000","last_undegraded":"2024-09-10T19:11:21.187274+0000","last_fullsized":"2024-09-10T19:11:21.187274+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:11:17.171251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:11:17.171251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:12:21.736124+0000","scrub_duration":0.0016247670000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"74'79","reported_seq":75,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-09-10T19:11:21.300547+0000","last_change":"2024-09-10T19:10:34.385577+0000","last_active":"2024-09-10T19:11:21.300547+0000","last_peered":"2024-09-10T19:11:21.300547+0000","last_clean":"2024-09-10T19:11:21.300547+0000","last_became_active":"2024-09-10T19:09:56.634919+0000","last_became_peered":"2024-09-10T19:09:56.634919+0000","last_unstale":"2024-09-10T19:11:21.300547+0000","last_undegraded":"2024-09-10T19:11:21.300547+0000","last_fullsized":"2024-09-10T19:11:21.300547+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'72","last_scrub_stamp":"2024-09-10T19:10:34.385519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:08:00.699315+0000","last_clean_scrub_stamp":"2024-09-10T19:10:34.385519+0000","objects_scrubbed":2,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:11:51.371427+0000","scrub_duration":0.0030997160000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":26,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-09-10T19:11:21.300623+0000","last_change":"2024-09-10T19:10:11.695316+0000","last_active":"2024-09-10T19:11:21.300623+0000","last_peered":"2024-09-10T19:11:21.300623+0000","last_clean":"2024-09-10T19:11:21.300623+0000","last_became_active":"2024-09-10T19:10:11.694986+0000","last_became_peered":"2024-09-10T19:10:11.694986+0000","last_unstale":"2024-09-10T19:11:21.300623+0000","last_undegraded":"2024-09-10T19:11:21.300623+0000","last_fullsized":"2024-09-10T19:11:21.300623+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:10:10.676876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:11:38.724621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":34,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-10T19:11:25.312442+0000","last_change":"2024-09-10T19:11:25.312410+0000","last_active":"2024-09-10T19:11:25.312442+0000","last_peered":"2024-09-10T19:11:25.312442+0000","last_clean":"2024-09-10T19:11:25.312442+0000","last_became_active":"2024-09-10T19:10:13.720647+0000","last_became_peered":"2024-09-10T19:10:13.720647+0000","last_unstale":"2024-09-10T19:11:25.312442+0000","last_undegraded":"2024-09-10T19:11:25.312442+0000","last_fullsized":"2024-09-10T19:11:25.312442+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:11:25.312356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:11:25.312356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:12:54.878799+0000","scrub_duration":0.001690642,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.a","version":"66'19","reported_seq":59,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-09-10T19:11:21.815959+0000","last_change":"2024-09-10T19:10:09.821179+0000","last_active":"2024-09-10T19:11:21.815959+0000","last_peered":"2024-09-10T19:11:21.815959+0000","last_clean":"2024-09-10T19:11:21.815959+0000","last_became_active":"2024-09-10T19:10:09.820880+0000","last_became_peered":"2024-09-10T19:10:09.820880+0000","last_unstale":"2024-09-10T19:11:21.815959+0000","last_undegraded":"2024-09-10T19:11:21.815959+0000","last_fullsized":"2024-09-10T19:11:21.815959+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:10:08.671214+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:11:33.941959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":30,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-09-10T19:11:21.815568+0000","last_change":"2024-09-10T19:10:07.686475+0000","last_active":"2024-09-10T19:11:21.815568+0000","last_peered":"2024-09-10T19:11:21.815568+0000","last_clean":"2024-09-10T19:11:21.815568+0000","last_became_active":"2024-09-10T19:10:07.686236+0000","last_became_peered":"2024-09-10T19:10:07.686236+0000","last_unstale":"2024-09-10T19:11:21.815568+0000","last_undegraded":"2024-09-10T19:11:21.815568+0000","last_fullsized":"2024-09-10T19:11:21.815568+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:10:06.664709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:11:30.324556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":36,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-09-10T19:11:21.815224+0000","last_change":"2024-09-10T19:11:19.119050+0000","last_active":"2024-09-10T19:11:21.815224+0000","last_peered":"2024-09-10T19:11:21.815224+0000","last_clean":"2024-09-10T19:11:21.815224+0000","last_became_active":"2024-09-10T19:10:11.699745+0000","last_became_peered":"2024-09-10T19:10:11.699745+0000","last_unstale":"2024-09-10T19:11:21.815224+0000","last_undegraded":"2024-09-10T19:11:21.815224+0000","last_fullsized":"2024-09-10T19:11:21.815224+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:11:19.118987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:11:19.118987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:12:48.847171+0000","scrub_duration":0.0012170379999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":24,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-09-10T19:11:21.300446+0000","last_change":"2024-09-10T19:10:13.720270+0000","last_active":"2024-09-10T19:11:21.300446+0000","last_peered":"2024-09-10T19:11:21.300446+0000","last_clean":"2024-09-10T19:11:21.300446+0000","last_became_active":"2024-09-10T19:10:13.719973+0000","last_became_peered":"2024-09-10T19:10:13.719973+0000","last_unstale":"2024-09-10T19:11:21.300446+0000","last_undegraded":"2024-09-10T19:11:21.300446+0000","last_fullsized":"2024-09-10T19:11:21.300446+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:10:12.693598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:11:24.140279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"66'9","reported_seq":54,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-09-10T19:11:22.870476+0000","last_change":"2024-09-10T19:11:22.870432+0000","last_active":"2024-09-10T19:11:22.870476+0000","last_peered":"2024-09-10T19:11:22.870476+0000","last_clean":"2024-09-10T19:11:22.870476+0000","last_became_active":"2024-09-10T19:10:09.689117+0000","last_became_peered":"2024-09-10T19:10:09.689117+0000","last_unstale":"2024-09-10T19:11:22.870476+0000","last_undegraded":"2024-09-10T19:11:22.870476+0000","last_fullsized":"2024-09-10T19:11:22.870476+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'9","last_scrub_stamp":"2024-09-10T19:11:22.870369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:11:22.870369+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:12:27.485364+0000","scrub_duration":0.0017468729999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":30,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-09-10T19:11:21.189991+0000","last_change":"2024-09-10T19:10:07.683195+0000","last_active":"2024-09-10T19:11:21.189991+0000","last_peered":"2024-09-10T19:11:21.189991+0000","last_clean":"2024-09-10T19:11:21.189991+0000","last_became_active":"2024-09-10T19:10:07.682834+0000","last_became_peered":"2024-09-10T19:10:07.682834+0000","last_unstale":"2024-09-10T19:11:21.189991+0000","last_undegraded":"2024-09-10T19:11:21.189991+0000","last_fullsized":"2024-09-10T19:11:21.189991+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:10:06.664709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:11:26.248552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":26,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-09-10T19:11:21.815169+0000","last_change":"2024-09-10T19:10:11.694697+0000","last_active":"2024-09-10T19:11:21.815169+0000","last_peered":"2024-09-10T19:11:21.815169+0000","last_clean":"2024-09-10T19:11:21.815169+0000","last_became_active":"2024-09-10T19:10:11.694405+0000","last_became_peered":"2024-09-10T19:10:11.694405+0000","last_unstale":"2024-09-10T19:11:21.815169+0000","last_undegraded":"2024-09-10T19:11:21.815169+0000","last_fullsized":"2024-09-10T19:11:21.815169+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:10:10.676876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:11:32.816040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":34,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-09-10T19:11:21.302407+0000","last_change":"2024-09-10T19:11:17.974993+0000","last_active":"2024-09-10T19:11:21.302407+0000","last_peered":"2024-09-10T19:11:21.302407+0000","last_clean":"2024-09-10T19:11:21.302407+0000","last_became_active":"2024-09-10T19:10:13.714952+0000","last_became_peered":"2024-09-10T19:10:13.714952+0000","last_unstale":"2024-09-10T19:11:21.302407+0000","last_undegraded":"2024-09-10T19:11:21.302407+0000","last_fullsized":"2024-09-10T19:11:21.302407+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:11:17.974937+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:11:17.974937+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:12:32.960908+0000","scrub_duration":0.001345834,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.8","version":"66'15","reported_seq":53,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-09-10T19:11:21.186632+0000","last_change":"2024-09-10T19:10:09.821306+0000","last_active":"2024-09-10T19:11:21.186632+0000","last_peered":"2024-09-10T19:11:21.186632+0000","last_clean":"2024-09-10T19:11:21.186632+0000","last_became_active":"2024-09-10T19:10:09.820950+0000","last_became_peered":"2024-09-10T19:10:09.820950+0000","last_unstale":"2024-09-10T19:11:21.186632+0000","last_undegraded":"2024-09-10T19:11:21.186632+0000","last_fullsized":"2024-09-10T19:11:21.186632+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:10:08.671214+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:11:25.564861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":40,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-09-10T19:11:21.815596+0000","last_change":"2024-09-10T19:11:15.197602+0000","last_active":"2024-09-10T19:11:21.815596+0000","last_peered":"2024-09-10T19:11:21.815596+0000","last_clean":"2024-09-10T19:11:21.815596+0000","last_became_active":"2024-09-10T19:10:07.682286+0000","last_became_peered":"2024-09-10T19:10:07.682286+0000","last_unstale":"2024-09-10T19:11:21.815596+0000","last_undegraded":"2024-09-10T19:11:21.815596+0000","last_fullsized":"2024-09-10T19:11:21.815596+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:11:15.197497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:11:15.197497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:12:23.597035+0000","scrub_duration":0.0019717929999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.9","version":"66'8","reported_seq":50,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-10T19:11:23.237424+0000","last_change":"2024-09-10T19:10:11.703391+0000","last_active":"2024-09-10T19:11:23.237424+0000","last_peered":"2024-09-10T19:11:23.237424+0000","last_clean":"2024-09-10T19:11:23.237424+0000","last_became_active":"2024-09-10T19:10:11.702878+0000","last_became_peered":"2024-09-10T19:10:11.702878+0000","last_unstale":"2024-09-10T19:11:23.237424+0000","last_undegraded":"2024-09-10T19:11:23.237424+0000","last_fullsized":"2024-09-10T19:11:23.237424+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:10:10.676876+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":34,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-09-10T19:11:21.186724+0000","last_change":"2024-09-10T19:11:20.184218+0000","last_active":"2024-09-10T19:11:21.186724+0000","last_peered":"2024-09-10T19:11:21.186724+0000","last_clean":"2024-09-10T19:11:21.186724+0000","last_became_active":"2024-09-10T19:10:14.053875+0000","last_became_peered":"2024-09-10T19:10:14.053875+0000","last_unstale":"2024-09-10T19:11:21.186724+0000","last_undegraded":"2024-09-10T19:11:21.186724+0000","last_fullsized":"2024-09-10T19:11:21.186724+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:11:20.184132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:11:20.184132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:12:50.058323+0000","scrub_duration":0.002119522,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"66'12","reported_seq":51,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-09-10T19:11:21.816081+0000","last_change":"2024-09-10T19:10:09.693243+0000","last_active":"2024-09-10T19:11:21.816081+0000","last_peered":"2024-09-10T19:11:21.816081+0000","last_clean":"2024-09-10T19:11:21.816081+0000","last_became_active":"2024-09-10T19:10:09.692627+0000","last_became_peered":"2024-09-10T19:10:09.692627+0000","last_unstale":"2024-09-10T19:11:21.816081+0000","last_undegraded":"2024-09-10T19:11:21.816081+0000","last_fullsized":"2024-09-10T19:11:21.816081+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:10:08.671214+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:11:32.186259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.f","version":"59'2","reported_seq":56,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-10T19:11:25.111519+0000","last_change":"2024-09-10T19:11:25.111486+0000","last_active":"2024-09-10T19:11:25.111519+0000","last_peered":"2024-09-10T19:11:25.111519+0000","last_clean":"2024-09-10T19:11:25.111519+0000","last_became_active":"2024-09-10T19:10:07.681719+0000","last_became_peered":"2024-09-10T19:10:07.681719+0000","last_unstale":"2024-09-10T19:11:25.111519+0000","last_undegraded":"2024-09-10T19:11:25.111519+0000","last_fullsized":"2024-09-10T19:11:25.111519+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'2","last_scrub_stamp":"2024-09-10T19:11:25.111429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:11:25.111429+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:12:31.423912+0000","scrub_duration":0.0014791139999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":14,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":26,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-09-10T19:11:21.815057+0000","last_change":"2024-09-10T19:10:11.698643+0000","last_active":"2024-09-10T19:11:21.815057+0000","last_peered":"2024-09-10T19:11:21.815057+0000","last_clean":"2024-09-10T19:11:21.815057+0000","last_became_active":"2024-09-10T19:10:11.698354+0000","last_became_peered":"2024-09-10T19:10:11.698354+0000","last_unstale":"2024-09-10T19:11:21.815057+0000","last_undegraded":"2024-09-10T19:11:21.815057+0000","last_fullsized":"2024-09-10T19:11:21.815057+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:10:10.676876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:11:36.386854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":24,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-09-10T19:11:21.194939+0000","last_change":"2024-09-10T19:10:13.719272+0000","last_active":"2024-09-10T19:11:21.194939+0000","last_peered":"2024-09-10T19:11:21.194939+0000","last_clean":"2024-09-10T19:11:21.194939+0000","last_became_active":"2024-09-10T19:10:13.718962+0000","last_became_peered":"2024-09-10T19:10:13.718962+0000","last_unstale":"2024-09-10T19:11:21.194939+0000","last_undegraded":"2024-09-10T19:11:21.194939+0000","last_fullsized":"2024-09-10T19:11:21.194939+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:10:12.693598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.16","version":"66'9","reported_seq":54,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-09-10T19:11:21.302864+0000","last_change":"2024-09-10T19:11:11.962017+0000","last_active":"2024-09-10T19:11:21.302864+0000","last_peered":"2024-09-10T19:11:21.302864+0000","last_clean":"2024-09-10T19:11:21.302864+0000","last_became_active":"2024-09-10T19:10:09.822994+0000","last_became_peered":"2024-09-10T19:10:09.822994+0000","last_unstale":"2024-09-10T19:11:21.302864+0000","last_undegraded":"2024-09-10T19:11:21.302864+0000","last_fullsized":"2024-09-10T19:11:21.302864+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'9","last_scrub_stamp":"2024-09-10T19:11:11.961960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:11:11.961960+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:12:16.945811+0000","scrub_duration":0.001903287,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":30,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-09-10T19:11:21.815137+0000","last_change":"2024-09-10T19:10:07.678348+0000","last_active":"2024-09-10T19:11:21.815137+0000","last_peered":"2024-09-10T19:11:21.815137+0000","last_clean":"2024-09-10T19:11:21.815137+0000","last_became_active":"2024-09-10T19:10:07.678076+0000","last_became_peered":"2024-09-10T19:10:07.678076+0000","last_unstale":"2024-09-10T19:11:21.815137+0000","last_undegraded":"2024-09-10T19:11:21.815137+0000","last_fullsized":"2024-09-10T19:11:21.815137+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:10:06.664709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:11:30.763897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":26,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-09-10T19:11:21.195518+0000","last_change":"2024-09-10T19:10:11.693961+0000","last_active":"2024-09-10T19:11:21.195518+0000","last_peered":"2024-09-10T19:11:21.195518+0000","last_clean":"2024-09-10T19:11:21.195518+0000","last_became_active":"2024-09-10T19:10:11.693407+0000","last_became_peered":"2024-09-10T19:10:11.693407+0000","last_unstale":"2024-09-10T19:11:21.195518+0000","last_undegraded":"2024-09-10T19:11:21.195518+0000","last_fullsized":"2024-09-10T19:11:21.195518+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:10:10.676876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:11:32.367716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"66'1","reported_seq":35,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-10T19:11:28.192820+0000","last_change":"2024-09-10T19:11:28.192794+0000","last_active":"2024-09-10T19:11:28.192820+0000","last_peered":"2024-09-10T19:11:28.192820+0000","last_clean":"2024-09-10T19:11:28.192820+0000","last_became_active":"2024-09-10T19:10:13.717001+0000","last_became_peered":"2024-09-10T19:10:13.717001+0000","last_unstale":"2024-09-10T19:11:28.192820+0000","last_undegraded":"2024-09-10T19:11:28.192820+0000","last_fullsized":"2024-09-10T19:11:28.192820+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'1","last_scrub_stamp":"2024-09-10T19:11:28.192745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:11:28.192745+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:12:37.727108+0000","scrub_duration":0.001276323,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":30,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-09-10T19:11:21.816484+0000","last_change":"2024-09-10T19:10:07.683787+0000","last_active":"2024-09-10T19:11:21.816484+0000","last_peered":"2024-09-10T19:11:21.816484+0000","last_clean":"2024-09-10T19:11:21.816484+0000","last_became_active":"2024-09-10T19:10:07.682963+0000","last_became_peered":"2024-09-10T19:10:07.682963+0000","last_unstale":"2024-09-10T19:11:21.816484+0000","last_undegraded":"2024-09-10T19:11:21.816484+0000","last_fullsized":"2024-09-10T19:11:21.816484+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:10:06.664709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:11:29.067558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"66'6","reported_seq":37,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-09-10T19:11:21.195577+0000","last_change":"2024-09-10T19:10:09.822283+0000","last_active":"2024-09-10T19:11:21.195577+0000","last_peered":"2024-09-10T19:11:21.195577+0000","last_clean":"2024-09-10T19:11:21.195577+0000","last_became_active":"2024-09-10T19:10:09.821906+0000","last_became_peered":"2024-09-10T19:10:09.821906+0000","last_unstale":"2024-09-10T19:11:21.195577+0000","last_undegraded":"2024-09-10T19:11:21.195577+0000","last_fullsized":"2024-09-10T19:11:21.195577+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:10:08.671214+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:11:35.174496+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":36,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-10T19:11:23.264917+0000","last_change":"2024-09-10T19:11:23.264870+0000","last_active":"2024-09-10T19:11:23.264917+0000","last_peered":"2024-09-10T19:11:23.264917+0000","last_clean":"2024-09-10T19:11:23.264917+0000","last_became_active":"2024-09-10T19:10:11.692943+0000","last_became_peered":"2024-09-10T19:10:11.692943+0000","last_unstale":"2024-09-10T19:11:23.264917+0000","last_undegraded":"2024-09-10T19:11:23.264917+0000","last_fullsized":"2024-09-10T19:11:23.264917+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:11:23.264793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:11:23.264793+0000","last_clean_scrub_stamp":"2024-09-10T19:11:23.264793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:12:44.248872+0000","scrub_duration":0.0015950649999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":34,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-09-10T19:11:21.300839+0000","last_change":"2024-09-10T19:11:13.342070+0000","last_active":"2024-09-10T19:11:21.300839+0000","last_peered":"2024-09-10T19:11:21.300839+0000","last_clean":"2024-09-10T19:11:21.300839+0000","last_became_active":"2024-09-10T19:10:14.054360+0000","last_became_peered":"2024-09-10T19:10:14.054360+0000","last_unstale":"2024-09-10T19:11:21.300839+0000","last_undegraded":"2024-09-10T19:11:21.300839+0000","last_fullsized":"2024-09-10T19:11:21.300839+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:11:13.341970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:11:13.341970+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:12:30.833886+0000","scrub_duration":0.001537725,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"66'10","reported_seq":43,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-09-10T19:11:21.195212+0000","last_change":"2024-09-10T19:10:09.822478+0000","last_active":"2024-09-10T19:11:21.195212+0000","last_peered":"2024-09-10T19:11:21.195212+0000","last_clean":"2024-09-10T19:11:21.195212+0000","last_became_active":"2024-09-10T19:10:09.821810+0000","last_became_peered":"2024-09-10T19:10:09.821810+0000","last_unstale":"2024-09-10T19:11:21.195212+0000","last_undegraded":"2024-09-10T19:11:21.195212+0000","last_fullsized":"2024-09-10T19:11:21.195212+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:10:08.671214+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":40,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-09-10T19:11:21.187053+0000","last_change":"2024-09-10T19:11:12.081772+0000","last_active":"2024-09-10T19:11:21.187053+0000","last_peered":"2024-09-10T19:11:21.187053+0000","last_clean":"2024-09-10T19:11:21.187053+0000","last_became_active":"2024-09-10T19:10:07.685870+0000","last_became_peered":"2024-09-10T19:10:07.685870+0000","last_unstale":"2024-09-10T19:11:21.187053+0000","last_undegraded":"2024-09-10T19:11:21.187053+0000","last_fullsized":"2024-09-10T19:11:21.187053+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:11:12.081691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:11:12.081691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:12:41.469111+0000","scrub_duration":0.001629903,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"66'8","reported_seq":50,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-10T19:11:23.236949+0000","last_change":"2024-09-10T19:10:11.700895+0000","last_active":"2024-09-10T19:11:23.236949+0000","last_peered":"2024-09-10T19:11:23.236949+0000","last_clean":"2024-09-10T19:11:23.236949+0000","last_became_active":"2024-09-10T19:10:11.700584+0000","last_became_peered":"2024-09-10T19:10:11.700584+0000","last_unstale":"2024-09-10T19:11:23.236949+0000","last_undegraded":"2024-09-10T19:11:23.236949+0000","last_fullsized":"2024-09-10T19:11:23.236949+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:10:10.676876+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:11:34.335843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":34,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-09-10T19:11:21.302650+0000","last_change":"2024-09-10T19:11:20.881495+0000","last_active":"2024-09-10T19:11:21.302650+0000","last_peered":"2024-09-10T19:11:21.302650+0000","last_clean":"2024-09-10T19:11:21.302650+0000","last_became_active":"2024-09-10T19:10:13.721035+0000","last_became_peered":"2024-09-10T19:10:13.721035+0000","last_unstale":"2024-09-10T19:11:21.302650+0000","last_undegraded":"2024-09-10T19:11:21.302650+0000","last_fullsized":"2024-09-10T19:11:21.302650+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:11:20.881436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:11:20.881436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:12:49.668407+0000","scrub_duration":0.0014551099999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"66'9","reported_seq":44,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-09-10T19:11:21.187595+0000","last_change":"2024-09-10T19:10:09.823289+0000","last_active":"2024-09-10T19:11:21.187595+0000","last_peered":"2024-09-10T19:11:21.187595+0000","last_clean":"2024-09-10T19:11:21.187595+0000","last_became_active":"2024-09-10T19:10:09.823006+0000","last_became_peered":"2024-09-10T19:10:09.823006+0000","last_unstale":"2024-09-10T19:11:21.187595+0000","last_undegraded":"2024-09-10T19:11:21.187595+0000","last_fullsized":"2024-09-10T19:11:21.187595+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:10:08.671214+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:11:26.960895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":30,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-09-10T19:11:21.302829+0000","last_change":"2024-09-10T19:10:07.688840+0000","last_active":"2024-09-10T19:11:21.302829+0000","last_peered":"2024-09-10T19:11:21.302829+0000","last_clean":"2024-09-10T19:11:21.302829+0000","last_became_active":"2024-09-10T19:10:07.688499+0000","last_became_peered":"2024-09-10T19:10:07.688499+0000","last_unstale":"2024-09-10T19:11:21.302829+0000","last_undegraded":"2024-09-10T19:11:21.302829+0000","last_fullsized":"2024-09-10T19:11:21.302829+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:10:06.664709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:11:24.704063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"66'8","reported_seq":60,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-10T19:11:23.870183+0000","last_change":"2024-09-10T19:11:23.870113+0000","last_active":"2024-09-10T19:11:23.870183+0000","last_peered":"2024-09-10T19:11:23.870183+0000","last_clean":"2024-09-10T19:11:23.870183+0000","last_became_active":"2024-09-10T19:10:11.692359+0000","last_became_peered":"2024-09-10T19:10:11.692359+0000","last_unstale":"2024-09-10T19:11:23.870183+0000","last_undegraded":"2024-09-10T19:11:23.870183+0000","last_fullsized":"2024-09-10T19:11:23.870183+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'8","last_scrub_stamp":"2024-09-10T19:11:23.870010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:11:23.870010+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:12:47.854105+0000","scrub_duration":0.0017621309999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":24,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-09-10T19:11:21.817375+0000","last_change":"2024-09-10T19:10:13.718617+0000","last_active":"2024-09-10T19:11:21.817375+0000","last_peered":"2024-09-10T19:11:21.817375+0000","last_clean":"2024-09-10T19:11:21.817375+0000","last_became_active":"2024-09-10T19:10:13.718285+0000","last_became_peered":"2024-09-10T19:10:13.718285+0000","last_unstale":"2024-09-10T19:11:21.817375+0000","last_undegraded":"2024-09-10T19:11:21.817375+0000","last_fullsized":"2024-09-10T19:11:21.817375+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:10:12.693598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:11:30.971974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.12","version":"79'13","reported_seq":58,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-09-10T19:11:22.072557+0000","last_change":"2024-09-10T19:11:22.072464+0000","last_active":"2024-09-10T19:11:22.072557+0000","last_peered":"2024-09-10T19:11:22.072557+0000","last_clean":"2024-09-10T19:11:22.072557+0000","last_became_active":"2024-09-10T19:10:09.689440+0000","last_became_peered":"2024-09-10T19:10:09.689440+0000","last_unstale":"2024-09-10T19:11:22.072557+0000","last_undegraded":"2024-09-10T19:11:22.072557+0000","last_fullsized":"2024-09-10T19:11:22.072557+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'13","last_scrub_stamp":"2024-09-10T19:11:22.072400+0000","last_deep_scrub":"79'13","last_deep_scrub_stamp":"2024-09-10T19:11:22.072400+0000","last_clean_scrub_stamp":"2024-09-10T19:11:22.072400+0000","objects_scrubbed":4,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:12:33.209191+0000","scrub_duration":0.0032983940000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"59'1","reported_seq":37,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-09-10T19:11:21.816042+0000","last_change":"2024-09-10T19:10:07.683567+0000","last_active":"2024-09-10T19:11:21.816042+0000","last_peered":"2024-09-10T19:11:21.816042+0000","last_clean":"2024-09-10T19:11:21.816042+0000","last_became_active":"2024-09-10T19:10:07.683263+0000","last_became_peered":"2024-09-10T19:10:07.683263+0000","last_unstale":"2024-09-10T19:11:21.816042+0000","last_undegraded":"2024-09-10T19:11:21.816042+0000","last_fullsized":"2024-09-10T19:11:21.816042+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:10:06.664709+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:11:35.421853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":362,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":26,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-09-10T19:11:21.194980+0000","last_change":"2024-09-10T19:10:11.701754+0000","last_active":"2024-09-10T19:11:21.194980+0000","last_peered":"2024-09-10T19:11:21.194980+0000","last_clean":"2024-09-10T19:11:21.194980+0000","last_became_active":"2024-09-10T19:10:11.701416+0000","last_became_peered":"2024-09-10T19:10:11.701416+0000","last_unstale":"2024-09-10T19:11:21.194980+0000","last_undegraded":"2024-09-10T19:11:21.194980+0000","last_fullsized":"2024-09-10T19:11:21.194980+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:10:10.676876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:11:30.081463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":24,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-09-10T19:11:21.302167+0000","last_change":"2024-09-10T19:10:13.716464+0000","last_active":"2024-09-10T19:11:21.302167+0000","last_peered":"2024-09-10T19:11:21.302167+0000","last_clean":"2024-09-10T19:11:21.302167+0000","last_became_active":"2024-09-10T19:10:13.716198+0000","last_became_peered":"2024-09-10T19:10:13.716198+0000","last_unstale":"2024-09-10T19:11:21.302167+0000","last_undegraded":"2024-09-10T19:11:21.302167+0000","last_fullsized":"2024-09-10T19:11:21.302167+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:10:12.693598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:11:29.980159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"66'11","reported_seq":47,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-09-10T19:11:21.815272+0000","last_change":"2024-09-10T19:10:09.694117+0000","last_active":"2024-09-10T19:11:21.815272+0000","last_peered":"2024-09-10T19:11:21.815272+0000","last_clean":"2024-09-10T19:11:21.815272+0000","last_became_active":"2024-09-10T19:10:09.693856+0000","last_became_peered":"2024-09-10T19:10:09.693856+0000","last_unstale":"2024-09-10T19:11:21.815272+0000","last_undegraded":"2024-09-10T19:11:21.815272+0000","last_fullsized":"2024-09-10T19:11:21.815272+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:10:08.671214+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:11:32.166737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":40,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-09-10T19:11:21.189948+0000","last_change":"2024-09-10T19:11:10.031299+0000","last_active":"2024-09-10T19:11:21.189948+0000","last_peered":"2024-09-10T19:11:21.189948+0000","last_clean":"2024-09-10T19:11:21.189948+0000","last_became_active":"2024-09-10T19:10:07.685339+0000","last_became_peered":"2024-09-10T19:10:07.685339+0000","last_unstale":"2024-09-10T19:11:21.189948+0000","last_undegraded":"2024-09-10T19:11:21.189948+0000","last_fullsized":"2024-09-10T19:11:21.189948+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:11:10.031245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:11:10.031245+0000","last_clean_scrub_stamp":"2024-09-10T19:11:10.031245+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:12:27.675011+0000","scrub_duration":0.0014113439999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":26,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-09-10T19:11:21.189285+0000","last_change":"2024-09-10T19:10:11.694827+0000","last_active":"2024-09-10T19:11:21.189285+0000","last_peered":"2024-09-10T19:11:21.189285+0000","last_clean":"2024-09-10T19:11:21.189285+0000","last_became_active":"2024-09-10T19:10:11.694564+0000","last_became_peered":"2024-09-10T19:10:11.694564+0000","last_unstale":"2024-09-10T19:11:21.189285+0000","last_undegraded":"2024-09-10T19:11:21.189285+0000","last_fullsized":"2024-09-10T19:11:21.189285+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:10:10.676876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:11:24.555440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":34,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-09-10T19:11:21.194595+0000","last_change":"2024-09-10T19:11:16.813629+0000","last_active":"2024-09-10T19:11:21.194595+0000","last_peered":"2024-09-10T19:11:21.194595+0000","last_clean":"2024-09-10T19:11:21.194595+0000","last_became_active":"2024-09-10T19:10:13.724201+0000","last_became_peered":"2024-09-10T19:10:13.724201+0000","last_unstale":"2024-09-10T19:11:21.194595+0000","last_undegraded":"2024-09-10T19:11:21.194595+0000","last_fullsized":"2024-09-10T19:11:21.194595+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:11:16.813575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:11:16.813575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:12:44.427906+0000","scrub_duration":0.001237312,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"66'4","reported_seq":44,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-09-10T19:11:21.194577+0000","last_change":"2024-09-10T19:11:14.816693+0000","last_active":"2024-09-10T19:11:21.194577+0000","last_peered":"2024-09-10T19:11:21.194577+0000","last_clean":"2024-09-10T19:11:21.194577+0000","last_became_active":"2024-09-10T19:10:09.692751+0000","last_became_peered":"2024-09-10T19:10:09.692751+0000","last_unstale":"2024-09-10T19:11:21.194577+0000","last_undegraded":"2024-09-10T19:11:21.194577+0000","last_fullsized":"2024-09-10T19:11:21.194577+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'4","last_scrub_stamp":"2024-09-10T19:11:14.816632+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:11:14.816632+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:12:42.695105+0000","scrub_duration":0.001902504,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":40,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-10T19:11:24.275089+0000","last_change":"2024-09-10T19:11:24.275061+0000","last_active":"2024-09-10T19:11:24.275089+0000","last_peered":"2024-09-10T19:11:24.275089+0000","last_clean":"2024-09-10T19:11:24.275089+0000","last_became_active":"2024-09-10T19:10:07.687723+0000","last_became_peered":"2024-09-10T19:10:07.687723+0000","last_unstale":"2024-09-10T19:11:24.275089+0000","last_undegraded":"2024-09-10T19:11:24.275089+0000","last_fullsized":"2024-09-10T19:11:24.275089+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:11:24.275004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:11:24.275004+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:12:41.723440+0000","scrub_duration":0.001588014,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":26,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-09-10T19:11:21.816290+0000","last_change":"2024-09-10T19:10:11.698910+0000","last_active":"2024-09-10T19:11:21.816290+0000","last_peered":"2024-09-10T19:11:21.816290+0000","last_clean":"2024-09-10T19:11:21.816290+0000","last_became_active":"2024-09-10T19:10:11.698536+0000","last_became_peered":"2024-09-10T19:10:11.698536+0000","last_unstale":"2024-09-10T19:11:21.816290+0000","last_undegraded":"2024-09-10T19:11:21.816290+0000","last_fullsized":"2024-09-10T19:11:21.816290+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:10:10.676876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:11:24.876541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":34,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-09-10T19:11:21.300729+0000","last_change":"2024-09-10T19:11:14.298616+0000","last_active":"2024-09-10T19:11:21.300729+0000","last_peered":"2024-09-10T19:11:21.300729+0000","last_clean":"2024-09-10T19:11:21.300729+0000","last_became_active":"2024-09-10T19:10:13.718271+0000","last_became_peered":"2024-09-10T19:10:13.718271+0000","last_unstale":"2024-09-10T19:11:21.300729+0000","last_undegraded":"2024-09-10T19:11:21.300729+0000","last_fullsized":"2024-09-10T19:11:21.300729+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:11:14.298545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:11:14.298545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:12:15.177239+0000","scrub_duration":0.001561799,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.11","version":"66'11","reported_seq":47,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-09-10T19:11:21.194692+0000","last_change":"2024-09-10T19:10:09.821652+0000","last_active":"2024-09-10T19:11:21.194692+0000","last_peered":"2024-09-10T19:11:21.194692+0000","last_clean":"2024-09-10T19:11:21.194692+0000","last_became_active":"2024-09-10T19:10:09.821357+0000","last_became_peered":"2024-09-10T19:10:09.821357+0000","last_unstale":"2024-09-10T19:11:21.194692+0000","last_undegraded":"2024-09-10T19:11:21.194692+0000","last_fullsized":"2024-09-10T19:11:21.194692+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:10:08.671214+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":30,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-09-10T19:11:21.815770+0000","last_change":"2024-09-10T19:10:07.685503+0000","last_active":"2024-09-10T19:11:21.815770+0000","last_peered":"2024-09-10T19:11:21.815770+0000","last_clean":"2024-09-10T19:11:21.815770+0000","last_became_active":"2024-09-10T19:10:07.685223+0000","last_became_peered":"2024-09-10T19:10:07.685223+0000","last_unstale":"2024-09-10T19:11:21.815770+0000","last_undegraded":"2024-09-10T19:11:21.815770+0000","last_fullsized":"2024-09-10T19:11:21.815770+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:10:06.664709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:11:30.851917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":26,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-09-10T19:11:21.300206+0000","last_change":"2024-09-10T19:10:11.699248+0000","last_active":"2024-09-10T19:11:21.300206+0000","last_peered":"2024-09-10T19:11:21.300206+0000","last_clean":"2024-09-10T19:11:21.300206+0000","last_became_active":"2024-09-10T19:10:11.699013+0000","last_became_peered":"2024-09-10T19:10:11.699013+0000","last_unstale":"2024-09-10T19:11:21.300206+0000","last_undegraded":"2024-09-10T19:11:21.300206+0000","last_fullsized":"2024-09-10T19:11:21.300206+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:10:10.676876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:11:37.771138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":24,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-09-10T19:11:21.194830+0000","last_change":"2024-09-10T19:10:14.056524+0000","last_active":"2024-09-10T19:11:21.194830+0000","last_peered":"2024-09-10T19:11:21.194830+0000","last_clean":"2024-09-10T19:11:21.194830+0000","last_became_active":"2024-09-10T19:10:14.056172+0000","last_became_peered":"2024-09-10T19:10:14.056172+0000","last_unstale":"2024-09-10T19:11:21.194830+0000","last_undegraded":"2024-09-10T19:11:21.194830+0000","last_fullsized":"2024-09-10T19:11:21.194830+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:10:12.693598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:11:24.815438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"66'1","reported_seq":25,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-09-10T19:11:21.300549+0000","last_change":"2024-09-10T19:10:13.719166+0000","last_active":"2024-09-10T19:11:21.300549+0000","last_peered":"2024-09-10T19:11:21.300549+0000","last_clean":"2024-09-10T19:11:21.300549+0000","last_became_active":"2024-09-10T19:10:13.718875+0000","last_became_peered":"2024-09-10T19:10:13.718875+0000","last_unstale":"2024-09-10T19:11:21.300549+0000","last_undegraded":"2024-09-10T19:11:21.300549+0000","last_fullsized":"2024-09-10T19:11:21.300549+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:10:12.693598+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":369,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":30,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-09-10T19:11:21.186687+0000","last_change":"2024-09-10T19:10:07.690146+0000","last_active":"2024-09-10T19:11:21.186687+0000","last_peered":"2024-09-10T19:11:21.186687+0000","last_clean":"2024-09-10T19:11:21.186687+0000","last_became_active":"2024-09-10T19:10:07.689877+0000","last_became_peered":"2024-09-10T19:10:07.689877+0000","last_unstale":"2024-09-10T19:11:21.186687+0000","last_undegraded":"2024-09-10T19:11:21.186687+0000","last_fullsized":"2024-09-10T19:11:21.186687+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:10:06.664709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:11:33.455191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1e","version":"80'15","reported_seq":58,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-10T19:11:25.884049+0000","last_change":"2024-09-10T19:11:25.884008+0000","last_active":"2024-09-10T19:11:25.884049+0000","last_peered":"2024-09-10T19:11:25.884049+0000","last_clean":"2024-09-10T19:11:25.884049+0000","last_became_active":"2024-09-10T19:10:09.823061+0000","last_became_peered":"2024-09-10T19:10:09.823061+0000","last_unstale":"2024-09-10T19:11:25.884049+0000","last_undegraded":"2024-09-10T19:11:25.884049+0000","last_fullsized":"2024-09-10T19:11:25.884049+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"80'15","last_scrub_stamp":"2024-09-10T19:11:25.883919+0000","last_deep_scrub":"80'15","last_deep_scrub_stamp":"2024-09-10T19:11:25.883919+0000","last_clean_scrub_stamp":"2024-09-10T19:11:25.883919+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:12:31.561992+0000","scrub_duration":0.0034715229999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1f","version":"66'8","reported_seq":53,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-10T19:11:23.207454+0000","last_change":"2024-09-10T19:10:11.695734+0000","last_active":"2024-09-10T19:11:23.207454+0000","last_peered":"2024-09-10T19:11:23.207454+0000","last_clean":"2024-09-10T19:11:23.207454+0000","last_became_active":"2024-09-10T19:10:11.695427+0000","last_became_peered":"2024-09-10T19:10:11.695427+0000","last_unstale":"2024-09-10T19:11:23.207454+0000","last_undegraded":"2024-09-10T19:11:23.207454+0000","last_fullsized":"2024-09-10T19:11:23.207454+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:10:10.676876+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":24,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-09-10T19:11:21.189619+0000","last_change":"2024-09-10T19:10:13.715720+0000","last_active":"2024-09-10T19:11:21.189619+0000","last_peered":"2024-09-10T19:11:21.189619+0000","last_clean":"2024-09-10T19:11:21.189619+0000","last_became_active":"2024-09-10T19:10:13.715419+0000","last_became_peered":"2024-09-10T19:10:13.715419+0000","last_unstale":"2024-09-10T19:11:21.189619+0000","last_undegraded":"2024-09-10T19:11:21.189619+0000","last_fullsized":"2024-09-10T19:11:21.189619+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:10:12.693598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:11:23.479548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":30,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-09-10T19:11:21.195149+0000","last_change":"2024-09-10T19:10:07.684238+0000","last_active":"2024-09-10T19:11:21.195149+0000","last_peered":"2024-09-10T19:11:21.195149+0000","last_clean":"2024-09-10T19:11:21.195149+0000","last_became_active":"2024-09-10T19:10:07.682689+0000","last_became_peered":"2024-09-10T19:10:07.682689+0000","last_unstale":"2024-09-10T19:11:21.195149+0000","last_undegraded":"2024-09-10T19:11:21.195149+0000","last_fullsized":"2024-09-10T19:11:21.195149+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:10:06.664709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"66'11","reported_seq":47,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-09-10T19:11:21.815432+0000","last_change":"2024-09-10T19:10:09.688145+0000","last_active":"2024-09-10T19:11:21.815432+0000","last_peered":"2024-09-10T19:11:21.815432+0000","last_clean":"2024-09-10T19:11:21.815432+0000","last_became_active":"2024-09-10T19:10:09.687851+0000","last_became_peered":"2024-09-10T19:10:09.687851+0000","last_unstale":"2024-09-10T19:11:21.815432+0000","last_undegraded":"2024-09-10T19:11:21.815432+0000","last_fullsized":"2024-09-10T19:11:21.815432+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:10:08.671214+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:11:33.211646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":36,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-10T19:11:28.822429+0000","last_change":"2024-09-10T19:11:28.822385+0000","last_active":"2024-09-10T19:11:28.822429+0000","last_peered":"2024-09-10T19:11:28.822429+0000","last_clean":"2024-09-10T19:11:28.822429+0000","last_became_active":"2024-09-10T19:10:11.701735+0000","last_became_peered":"2024-09-10T19:10:11.701735+0000","last_unstale":"2024-09-10T19:11:28.822429+0000","last_undegraded":"2024-09-10T19:11:28.822429+0000","last_fullsized":"2024-09-10T19:11:28.822429+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:11:28.822327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:11:28.822327+0000","last_clean_scrub_stamp":"2024-09-10T19:11:28.822327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:12:45.908873+0000","scrub_duration":0.0012591119999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":24,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-09-10T19:11:21.817462+0000","last_change":"2024-09-10T19:10:14.053927+0000","last_active":"2024-09-10T19:11:21.817462+0000","last_peered":"2024-09-10T19:11:21.817462+0000","last_clean":"2024-09-10T19:11:21.817462+0000","last_became_active":"2024-09-10T19:10:14.053619+0000","last_became_peered":"2024-09-10T19:10:14.053619+0000","last_unstale":"2024-09-10T19:11:21.817462+0000","last_undegraded":"2024-09-10T19:11:21.817462+0000","last_fullsized":"2024-09-10T19:11:21.817462+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:10:12.693598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:11:29.510159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":30,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-09-10T19:11:21.816060+0000","last_change":"2024-09-10T19:10:07.686443+0000","last_active":"2024-09-10T19:11:21.816060+0000","last_peered":"2024-09-10T19:11:21.816060+0000","last_clean":"2024-09-10T19:11:21.816060+0000","last_became_active":"2024-09-10T19:10:07.685997+0000","last_became_peered":"2024-09-10T19:10:07.685997+0000","last_unstale":"2024-09-10T19:11:21.816060+0000","last_undegraded":"2024-09-10T19:11:21.816060+0000","last_fullsized":"2024-09-10T19:11:21.816060+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:10:06.664709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:11:36.019712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1c","version":"78'22","reported_seq":70,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-09-10T19:11:21.814940+0000","last_change":"2024-09-10T19:11:21.119003+0000","last_active":"2024-09-10T19:11:21.814940+0000","last_peered":"2024-09-10T19:11:21.814940+0000","last_clean":"2024-09-10T19:11:21.814940+0000","last_became_active":"2024-09-10T19:10:09.690328+0000","last_became_peered":"2024-09-10T19:10:09.690328+0000","last_unstale":"2024-09-10T19:11:21.814940+0000","last_undegraded":"2024-09-10T19:11:21.814940+0000","last_fullsized":"2024-09-10T19:11:21.814940+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"78'22","last_scrub_stamp":"2024-09-10T19:11:21.118937+0000","last_deep_scrub":"78'22","last_deep_scrub_stamp":"2024-09-10T19:11:21.118937+0000","last_clean_scrub_stamp":"2024-09-10T19:11:21.118937+0000","objects_scrubbed":7,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:12:21.535383+0000","scrub_duration":0.0048819759999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":26,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-09-10T19:11:21.189835+0000","last_change":"2024-09-10T19:10:11.700300+0000","last_active":"2024-09-10T19:11:21.189835+0000","last_peered":"2024-09-10T19:11:21.189835+0000","last_clean":"2024-09-10T19:11:21.189835+0000","last_became_active":"2024-09-10T19:10:11.700037+0000","last_became_peered":"2024-09-10T19:10:11.700037+0000","last_unstale":"2024-09-10T19:11:21.189835+0000","last_undegraded":"2024-09-10T19:11:21.189835+0000","last_fullsized":"2024-09-10T19:11:21.189835+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:10:10.676876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:11:25.509401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":34,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-10T19:11:24.828009+0000","last_change":"2024-09-10T19:11:24.827983+0000","last_active":"2024-09-10T19:11:24.828009+0000","last_peered":"2024-09-10T19:11:24.828009+0000","last_clean":"2024-09-10T19:11:24.828009+0000","last_became_active":"2024-09-10T19:10:14.053698+0000","last_became_peered":"2024-09-10T19:10:14.053698+0000","last_unstale":"2024-09-10T19:11:24.828009+0000","last_undegraded":"2024-09-10T19:11:24.828009+0000","last_fullsized":"2024-09-10T19:11:24.828009+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:11:24.827916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:11:24.827916+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:12:36.645047+0000","scrub_duration":0.0014104860000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":40,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-10T19:11:25.870618+0000","last_change":"2024-09-10T19:11:25.870573+0000","last_active":"2024-09-10T19:11:25.870618+0000","last_peered":"2024-09-10T19:11:25.870618+0000","last_clean":"2024-09-10T19:11:25.870618+0000","last_became_active":"2024-09-10T19:10:07.682811+0000","last_became_peered":"2024-09-10T19:10:07.682811+0000","last_unstale":"2024-09-10T19:11:25.870618+0000","last_undegraded":"2024-09-10T19:11:25.870618+0000","last_fullsized":"2024-09-10T19:11:25.870618+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:11:25.870516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:11:25.870516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:12:33.650020+0000","scrub_duration":0.001292568,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1d","version":"66'12","reported_seq":51,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-09-10T19:11:21.195335+0000","last_change":"2024-09-10T19:10:09.693108+0000","last_active":"2024-09-10T19:11:21.195335+0000","last_peered":"2024-09-10T19:11:21.195335+0000","last_clean":"2024-09-10T19:11:21.195335+0000","last_became_active":"2024-09-10T19:10:09.692781+0000","last_became_peered":"2024-09-10T19:10:09.692781+0000","last_unstale":"2024-09-10T19:11:21.195335+0000","last_undegraded":"2024-09-10T19:11:21.195335+0000","last_fullsized":"2024-09-10T19:11:21.195335+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:10:08.671214+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:11:25.822393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":26,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-09-10T19:11:21.817165+0000","last_change":"2024-09-10T19:10:11.701419+0000","last_active":"2024-09-10T19:11:21.817165+0000","last_peered":"2024-09-10T19:11:21.817165+0000","last_clean":"2024-09-10T19:11:21.817165+0000","last_became_active":"2024-09-10T19:10:11.701155+0000","last_became_peered":"2024-09-10T19:10:11.701155+0000","last_unstale":"2024-09-10T19:11:21.817165+0000","last_undegraded":"2024-09-10T19:11:21.817165+0000","last_fullsized":"2024-09-10T19:11:21.817165+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:10:10.676876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:11:33.357988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":65,"ondisk_log_size":65,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":177,"num_object_clones":0,"num_object_copies":531,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":177,"num_whiteouts":0,"num_read":698,"num_read_kb":455,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":407,"ondisk_log_size":407,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":76,"num_read_kb":71,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1328,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":27,"num_read_kb":27,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":3984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4,"ondisk_log_size":4,"up":96,"acting":96,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79,"ondisk_log_size":79,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":54,"seq":231928234005,"num_pgs":57,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298804,"kb_used_data":1452,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93446348,"statfs":{"total":95995035648,"available":95689060352,"internally_reserved":0,"allocated":1486848,"data_stored":988678,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 19:10:56 2024","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.53500000000000003},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.70499999999999996,"5min":0.70499999999999996,"15min":0.70499999999999996},"last":0.42799999999999999},{"interface":"front","average":{"1min":0.53800000000000003,"5min":0.53800000000000003,"15min":0.53800000000000003},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.76200000000000001,"5min":0.76200000000000001,"15min":0.76200000000000001},"last":0.498}]},{"osd":1,"last update":"Tue Sep 10 19:10:56 2024","interfaces":[{"interface":"back","average":{"1min":0.52400000000000002,"5min":0.52400000000000002,"15min":0.52400000000000002},"min":{"1min":0.41299999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":0.79400000000000004,"5min":0.79400000000000004,"15min":0.79400000000000004},"last":0.40799999999999997},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"min":{"1min":0.314,"5min":0.314,"15min":0.314},"max":{"1min":0.71799999999999997,"5min":0.71799999999999997,"15min":0.71799999999999997},"last":0.48199999999999998}]},{"osd":2,"last update":"Tue Sep 10 19:10:56 2024","interfaces":[{"interface":"back","average":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"min":{"1min":0.29199999999999998,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.84399999999999997,"5min":0.84399999999999997,"15min":0.84399999999999997},"last":0.36399999999999999},{"interface":"front","average":{"1min":0.55000000000000004,"5min":0.55000000000000004,"15min":0.55000000000000004},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.77300000000000002,"5min":0.77300000000000002,"15min":0.77300000000000002},"last":0.47099999999999997}]},{"osd":3,"last update":"Tue Sep 10 19:10:56 2024","interfaces":[{"interface":"back","average":{"1min":0.53600000000000003,"5min":0.53600000000000003,"15min":0.53600000000000003},"min":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":0.86399999999999999,"5min":0.86399999999999999,"15min":0.86399999999999999},"last":0.44600000000000001},{"interface":"front","average":{"1min":0.52000000000000002,"5min":0.52000000000000002,"15min":0.52000000000000002},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.77600000000000002,"5min":0.77600000000000002,"15min":0.77600000000000002},"last":0.35599999999999998}]},{"osd":4,"last update":"Tue Sep 10 19:10:56 2024","interfaces":[{"interface":"back","average":{"1min":0.45500000000000002,"5min":0.45500000000000002,"15min":0.45500000000000002},"min":{"1min":0.318,"5min":0.318,"15min":0.318},"max":{"1min":0.65000000000000002,"5min":0.65000000000000002,"15min":0.65000000000000002},"last":0.46000000000000002},{"interface":"front","average":{"1min":0.46000000000000002,"5min":0.46000000000000002,"15min":0.46000000000000002},"min":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.60599999999999998,"5min":0.60599999999999998,"15min":0.60599999999999998},"last":0.32900000000000001}]},{"osd":5,"last update":"Tue Sep 10 19:10:56 2024","interfaces":[{"interface":"back","average":{"1min":0.47999999999999998,"5min":0.47999999999999998,"15min":0.47999999999999998},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.76100000000000001,"5min":0.76100000000000001,"15min":0.76100000000000001},"last":0.313},{"interface":"front","average":{"1min":0.47599999999999998,"5min":0.47599999999999998,"15min":0.47599999999999998},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.68600000000000005,"5min":0.68600000000000005,"15min":0.68600000000000005},"last":0.41699999999999998}]},{"osd":6,"last update":"Tue Sep 10 19:10:56 2024","interfaces":[{"interface":"back","average":{"1min":0.5,"5min":0.5,"15min":0.5},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":0.65900000000000003,"5min":0.65900000000000003,"15min":0.65900000000000003},"last":0.373},{"interface":"front","average":{"1min":0.48899999999999999,"5min":0.48899999999999999,"15min":0.48899999999999999},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.67000000000000004,"5min":0.67000000000000004,"15min":0.67000000000000004},"last":0.39100000000000001}]}]},{"osd":6,"up_from":47,"seq":201863462938,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298792,"kb_used_data":1440,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93446360,"statfs":{"total":95995035648,"available":95689072640,"internally_reserved":0,"allocated":1474560,"data_stored":988804,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 19:10:35 2024","interfaces":[{"interface":"back","average":{"1min":0.52200000000000002,"5min":0.52200000000000002,"15min":0.52200000000000002},"min":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.621,"5min":0.621,"15min":0.621},"last":0.38100000000000001},{"interface":"front","average":{"1min":0.54400000000000004,"5min":0.54400000000000004,"15min":0.54400000000000004},"min":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":0.67200000000000004,"5min":0.67200000000000004,"15min":0.67200000000000004},"last":0.40500000000000003}]},{"osd":1,"last update":"Tue Sep 10 19:10:35 2024","interfaces":[{"interface":"back","average":{"1min":0.52900000000000003,"5min":0.52900000000000003,"15min":0.52900000000000003},"min":{"1min":0.42599999999999999,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.67400000000000004,"5min":0.67400000000000004,"15min":0.67400000000000004},"last":0.436},{"interface":"front","average":{"1min":0.53100000000000003,"5min":0.53100000000000003,"15min":0.53100000000000003},"min":{"1min":0.42399999999999999,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":0.65800000000000003,"5min":0.65800000000000003,"15min":0.65800000000000003},"last":0.34899999999999998}]},{"osd":2,"last update":"Tue Sep 10 19:10:35 2024","interfaces":[{"interface":"back","average":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"min":{"1min":0.45500000000000002,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":0.66400000000000003,"5min":0.66400000000000003,"15min":0.66400000000000003},"last":0.42599999999999999},{"interface":"front","average":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.51000000000000001},"min":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.64600000000000002,"5min":0.64600000000000002,"15min":0.64600000000000002},"last":0.45000000000000001}]},{"osd":3,"last update":"Tue Sep 10 19:10:35 2024","interfaces":[{"interface":"back","average":{"1min":0.51600000000000001,"5min":0.51600000000000001,"15min":0.51600000000000001},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.68100000000000005,"5min":0.68100000000000005,"15min":0.68100000000000005},"last":0.39900000000000002},{"interface":"front","average":{"1min":0.56599999999999995,"5min":0.56599999999999995,"15min":0.56599999999999995},"min":{"1min":0.434,"5min":0.434,"15min":0.434},"max":{"1min":0.70199999999999996,"5min":0.70199999999999996,"15min":0.70199999999999996},"last":0.48599999999999999}]},{"osd":4,"last update":"Tue Sep 10 19:10:35 2024","interfaces":[{"interface":"back","average":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"min":{"1min":0.27600000000000002,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.46500000000000002,"5min":0.46500000000000002,"15min":0.46500000000000002},"last":0.47699999999999998},{"interface":"front","average":{"1min":0.375,"5min":0.375,"15min":0.375},"min":{"1min":0.315,"5min":0.315,"15min":0.315},"max":{"1min":0.54400000000000004,"5min":0.54400000000000004,"15min":0.54400000000000004},"last":0.32100000000000001}]},{"osd":5,"last update":"Tue Sep 10 19:10:35 2024","interfaces":[{"interface":"back","average":{"1min":0.375,"5min":0.375,"15min":0.375},"min":{"1min":0.307,"5min":0.307,"15min":0.307},"max":{"1min":0.52200000000000002,"5min":0.52200000000000002,"15min":0.52200000000000002},"last":0.33200000000000002},{"interface":"front","average":{"1min":0.374,"5min":0.374,"15min":0.374},"min":{"1min":0.29999999999999999,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.51200000000000001,"5min":0.51200000000000001,"15min":0.51200000000000001},"last":0.45900000000000002}]},{"osd":7,"last update":"Tue Sep 10 19:10:54 2024","interfaces":[{"interface":"back","average":{"1min":0.44700000000000001,"5min":0.44700000000000001,"15min":0.44700000000000001},"min":{"1min":0.29899999999999999,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.68799999999999994,"5min":0.68799999999999994,"15min":0.68799999999999994},"last":0.46700000000000003},{"interface":"front","average":{"1min":0.44500000000000001,"5min":0.44500000000000001,"15min":0.44500000000000001},"min":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.55000000000000004,"5min":0.55000000000000004,"15min":0.55000000000000004},"last":0.36399999999999999}]}]},{"osd":5,"up_from":39,"seq":167503724575,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298216,"kb_used_data":864,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93446936,"statfs":{"total":95995035648,"available":95689662464,"internally_reserved":0,"allocated":884736,"data_stored":399317,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 19:11:08 2024","interfaces":[{"interface":"back","average":{"1min":0.51900000000000002,"5min":0.48299999999999998,"15min":0.47699999999999998},"min":{"1min":0.38200000000000001,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.67200000000000004,"5min":0.67200000000000004,"15min":0.67200000000000004},"last":0.38200000000000001},{"interface":"front","average":{"1min":0.53600000000000003,"5min":0.48399999999999999,"15min":0.47499999999999998},"min":{"1min":0.38300000000000001,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.75600000000000001,"5min":0.75600000000000001,"15min":0.75600000000000001},"last":0.42199999999999999}]},{"osd":1,"last update":"Tue Sep 10 19:11:08 2024","interfaces":[{"interface":"back","average":{"1min":0.58399999999999996,"5min":0.51900000000000002,"15min":0.50800000000000001},"min":{"1min":0.46800000000000003,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.72799999999999998,"5min":0.72799999999999998,"15min":0.72799999999999998},"last":0.39100000000000001},{"interface":"front","average":{"1min":0.50700000000000001,"5min":0.47899999999999998,"15min":0.47399999999999998},"min":{"1min":0.39900000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.67600000000000005,"5min":0.68799999999999994,"15min":0.68799999999999994},"last":0.39900000000000002}]},{"osd":2,"last update":"Tue Sep 10 19:11:08 2024","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.48599999999999999,"15min":0.47799999999999998},"min":{"1min":0.39900000000000002,"5min":0.309,"15min":0.309},"max":{"1min":0.70699999999999996,"5min":0.70699999999999996,"15min":0.70699999999999996},"last":0.432},{"interface":"front","average":{"1min":0.54100000000000004,"5min":0.498,"15min":0.49099999999999999},"min":{"1min":0.39200000000000002,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.70799999999999996,"5min":0.70799999999999996,"15min":0.70799999999999996},"last":0.372}]},{"osd":3,"last update":"Tue Sep 10 19:11:08 2024","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.52200000000000002,"15min":0.51400000000000001},"min":{"1min":0.437,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.74099999999999999,"5min":0.74099999999999999,"15min":0.74099999999999999},"last":0.44700000000000001},{"interface":"front","average":{"1min":0.59899999999999998,"5min":0.53800000000000003,"15min":0.52800000000000002},"min":{"1min":0.44700000000000001,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.747,"5min":0.747,"15min":0.747},"last":0.442}]},{"osd":4,"last update":"Tue Sep 10 19:11:08 2024","interfaces":[{"interface":"back","average":{"1min":0.44,"5min":0.37,"15min":0.35799999999999998},"min":{"1min":0.27800000000000002,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":0.624,"5min":0.624,"15min":0.624},"last":0.313},{"interface":"front","average":{"1min":0.46500000000000002,"5min":0.38100000000000001,"15min":0.36699999999999999},"min":{"1min":0.27800000000000002,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.76800000000000002,"5min":0.76800000000000002,"15min":0.76800000000000002},"last":0.41199999999999998}]},{"osd":6,"last update":"Tue Sep 10 19:10:34 2024","interfaces":[{"interface":"back","average":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"min":{"1min":0.223,"5min":0.223,"15min":0.223},"max":{"1min":0.61499999999999999,"5min":0.61499999999999999,"15min":0.61499999999999999},"last":0.32600000000000001},{"interface":"front","average":{"1min":0.40600000000000003,"5min":0.40600000000000003,"15min":0.40600000000000003},"min":{"1min":0.25,"5min":0.25,"15min":0.25},"max":{"1min":0.52800000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"last":0.29399999999999998}]},{"osd":7,"last update":"Tue Sep 10 19:10:55 2024","interfaces":[{"interface":"back","average":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.51000000000000001},"min":{"1min":0.29099999999999998,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.72699999999999998,"5min":0.72699999999999998,"15min":0.72699999999999998},"last":0.34200000000000003},{"interface":"front","average":{"1min":0.47899999999999998,"5min":0.47899999999999998,"15min":0.47899999999999998},"min":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.64600000000000002,"5min":0.64600000000000002,"15min":0.64600000000000002},"last":0.35199999999999998}]}]},{"osd":4,"up_from":32,"seq":137438953507,"num_pgs":52,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298232,"kb_used_data":880,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93446920,"statfs":{"total":95995035648,"available":95689646080,"internally_reserved":0,"allocated":901120,"data_stored":398526,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 19:10:48 2024","interfaces":[{"interface":"back","average":{"1min":0.49399999999999999,"5min":0.53200000000000003,"15min":0.53800000000000003},"min":{"1min":0.32500000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.61699999999999999,"5min":0.91500000000000004,"15min":0.91500000000000004},"last":0.46600000000000003},{"interface":"front","average":{"1min":0.47599999999999998,"5min":0.52000000000000002,"15min":0.52700000000000002},"min":{"1min":0.34499999999999997,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.63200000000000001,"5min":0.78200000000000003,"15min":0.78200000000000003},"last":0.49299999999999999}]},{"osd":1,"last update":"Tue Sep 10 19:10:48 2024","interfaces":[{"interface":"back","average":{"1min":0.5,"5min":0.51000000000000001,"15min":0.51200000000000001},"min":{"1min":0.33300000000000002,"5min":0.313,"15min":0.313},"max":{"1min":0.68799999999999994,"5min":0.83499999999999996,"15min":0.83499999999999996},"last":0.432},{"interface":"front","average":{"1min":0.504,"5min":0.52000000000000002,"15min":0.52300000000000002},"min":{"1min":0.374,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.77100000000000002,"5min":0.77100000000000002,"15min":0.77100000000000002},"last":0.59399999999999997}]},{"osd":2,"last update":"Tue Sep 10 19:10:48 2024","interfaces":[{"interface":"back","average":{"1min":0.51100000000000001,"5min":0.53300000000000003,"15min":0.53700000000000003},"min":{"1min":0.39300000000000002,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.69599999999999995,"5min":1.0649999999999999,"15min":1.0649999999999999},"last":0.55200000000000005},{"interface":"front","average":{"1min":0.48499999999999999,"5min":0.52700000000000002,"15min":0.53400000000000003},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.66000000000000003,"5min":0.86899999999999999,"15min":0.86899999999999999},"last":0.42299999999999999}]},{"osd":3,"last update":"Tue Sep 10 19:10:48 2024","interfaces":[{"interface":"back","average":{"1min":0.498,"5min":0.52100000000000002,"15min":0.52500000000000002},"min":{"1min":0.373,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.67700000000000005,"5min":0.76500000000000001,"15min":0.76500000000000001},"last":0.38},{"interface":"front","average":{"1min":0.52300000000000002,"5min":0.54600000000000004,"15min":0.55000000000000004},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.92200000000000004,"5min":1.0249999999999999,"15min":1.0249999999999999},"last":0.52400000000000002}]},{"osd":5,"last update":"Tue Sep 10 19:11:10 2024","interfaces":[{"interface":"back","average":{"1min":0.46400000000000002,"5min":0.39200000000000002,"15min":0.38},"min":{"1min":0.34999999999999998,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.61599999999999999,"5min":0.61599999999999999,"15min":0.61599999999999999},"last":0.57899999999999996},{"interface":"front","average":{"1min":0.47099999999999997,"5min":0.40899999999999997,"15min":0.39900000000000002},"min":{"1min":0.30599999999999999,"5min":0.223,"15min":0.223},"max":{"1min":0.63,"5min":0.89100000000000001,"15min":0.89100000000000001},"last":0.51400000000000001}]},{"osd":6,"last update":"Tue Sep 10 19:10:33 2024","interfaces":[{"interface":"back","average":{"1min":0.41299999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"min":{"1min":0.26100000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.58999999999999997,"5min":0.58999999999999997,"15min":0.58999999999999997},"last":0.627},{"interface":"front","average":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"min":{"1min":0.25,"5min":0.25,"15min":0.25},"max":{"1min":0.62,"5min":0.62,"15min":0.62},"last":0.64700000000000002}]},{"osd":7,"last update":"Tue Sep 10 19:10:58 2024","interfaces":[{"interface":"back","average":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.72699999999999998,"5min":0.72699999999999998,"15min":0.72699999999999998},"last":0.61399999999999999},{"interface":"front","average":{"1min":0.503,"5min":0.503,"15min":0.503},"min":{"1min":0.29099999999999998,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.69899999999999995,"5min":0.69899999999999995,"15min":0.69899999999999995},"last":0.56699999999999995}]}]},{"osd":3,"up_from":26,"seq":111669149736,"num_pgs":55,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298252,"kb_used_data":900,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93446900,"statfs":{"total":95995035648,"available":95689625600,"internally_reserved":0,"allocated":921600,"data_stored":400100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 19:11:25 2024","interfaces":[{"interface":"back","average":{"1min":0.315,"5min":0.38700000000000001,"15min":0.40000000000000002},"min":{"1min":0.223,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":0.46800000000000003,"5min":0.61499999999999999,"15min":0.61499999999999999},"last":0.34300000000000003},{"interface":"front","average":{"1min":0.33000000000000002,"5min":0.39700000000000002,"15min":0.41199999999999998},"min":{"1min":0.24399999999999999,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":0.42299999999999999,"5min":0.63600000000000001,"15min":0.63600000000000001},"last":0.376}]},{"osd":1,"last update":"Tue Sep 10 19:11:25 2024","interfaces":[{"interface":"back","average":{"1min":0.35499999999999998,"5min":0.42999999999999999,"15min":0.44700000000000001},"min":{"1min":0.253,"5min":0.253,"15min":0.253},"max":{"1min":0.505,"5min":0.67600000000000005,"15min":0.67600000000000005},"last":0.28000000000000003},{"interface":"front","average":{"1min":0.32900000000000001,"5min":0.38700000000000001,"15min":0.39400000000000002},"min":{"1min":0.26300000000000001,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.441,"5min":0.622,"15min":0.622},"last":0.30099999999999999}]},{"osd":2,"last update":"Tue Sep 10 19:11:25 2024","interfaces":[{"interface":"back","average":{"1min":0.38100000000000001,"5min":0.42499999999999999,"15min":0.434},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.51200000000000001,"5min":0.56599999999999995,"15min":0.56599999999999995},"last":0.36499999999999999},{"interface":"front","average":{"1min":0.38600000000000001,"5min":0.40799999999999997,"15min":0.41399999999999998},"min":{"1min":0.25600000000000001,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":0.66800000000000004,"5min":0.66800000000000004,"15min":0.66800000000000004},"last":0.39400000000000002}]},{"osd":4,"last update":"Tue Sep 10 19:10:50 2024","interfaces":[{"interface":"back","average":{"1min":0.54300000000000004,"5min":0.54400000000000004,"15min":0.54400000000000004},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.74399999999999999,"5min":0.74399999999999999,"15min":0.74399999999999999},"last":0.64800000000000002},{"interface":"front","average":{"1min":0.55400000000000005,"5min":0.54900000000000004,"15min":0.54800000000000004},"min":{"1min":0.433,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":0.81699999999999995,"5min":0.81699999999999995,"15min":0.81699999999999995},"last":0.44800000000000001}]},{"osd":5,"last update":"Tue Sep 10 19:11:07 2024","interfaces":[{"interface":"back","average":{"1min":0.58799999999999997,"5min":0.56999999999999995,"15min":0.56699999999999995},"min":{"1min":0.441,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.72599999999999998,"5min":0.80400000000000005,"15min":0.80400000000000005},"last":0.59199999999999997},{"interface":"front","average":{"1min":0.56200000000000006,"5min":0.56100000000000005,"15min":0.56100000000000005},"min":{"1min":0.40799999999999997,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.89100000000000001,"5min":0.89100000000000001,"15min":0.89100000000000001},"last":0.47099999999999997}]},{"osd":6,"last update":"Tue Sep 10 19:10:31 2024","interfaces":[{"interface":"back","average":{"1min":0.57899999999999996,"5min":0.57899999999999996,"15min":0.57899999999999996},"min":{"1min":0.46600000000000003,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":0.77000000000000002,"5min":0.77000000000000002,"15min":0.77000000000000002},"last":0.67000000000000004},{"interface":"front","average":{"1min":0.55700000000000005,"5min":0.55700000000000005,"15min":0.55700000000000005},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.90800000000000003,"5min":0.90800000000000003,"15min":0.90800000000000003},"last":0.437}]},{"osd":7,"last update":"Tue Sep 10 19:11:00 2024","interfaces":[{"interface":"back","average":{"1min":0.57999999999999996,"5min":0.57999999999999996,"15min":0.57999999999999996},"min":{"1min":0.45400000000000001,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":0.75600000000000001,"5min":0.75600000000000001,"15min":0.75600000000000001},"last":0.46200000000000002},{"interface":"front","average":{"1min":0.58399999999999996,"5min":0.58399999999999996,"15min":0.58399999999999996},"min":{"1min":0.443,"5min":0.443,"15min":0.443},"max":{"1min":0.749,"5min":0.749,"15min":0.749},"last":0.621}]}]},{"osd":2,"up_from":18,"seq":77309411373,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298212,"kb_used_data":860,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93446940,"statfs":{"total":95995035648,"available":95689666560,"internally_reserved":0,"allocated":880640,"data_stored":398516,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 19:11:08 2024","interfaces":[{"interface":"back","average":{"1min":0.33800000000000002,"5min":0.34899999999999998,"15min":0.34300000000000003},"min":{"1min":0.20100000000000001,"5min":0.20100000000000001,"15min":0.20100000000000001},"max":{"1min":0.45100000000000001,"5min":0.52300000000000002,"15min":0.52300000000000002},"last":0.36099999999999999},{"interface":"front","average":{"1min":0.33200000000000002,"5min":0.374,"15min":0.36799999999999999},"min":{"1min":0.23100000000000001,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":0.44700000000000001,"5min":0.64200000000000002,"15min":0.64200000000000002},"last":0.40100000000000002}]},{"osd":1,"last update":"Tue Sep 10 19:11:08 2024","interfaces":[{"interface":"back","average":{"1min":0.38200000000000001,"5min":0.38900000000000001,"15min":0.379},"min":{"1min":0.24099999999999999,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.63,"5min":0.69799999999999995,"15min":0.69799999999999995},"last":0.31900000000000001},{"interface":"front","average":{"1min":0.35799999999999998,"5min":0.39700000000000002,"15min":0.39100000000000001},"min":{"1min":0.28499999999999998,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.48299999999999998,"5min":0.63400000000000001,"15min":0.63400000000000001},"last":0.373}]},{"osd":3,"last update":"Tue Sep 10 19:10:30 2024","interfaces":[{"interface":"back","average":{"1min":0.46200000000000002,"5min":0.45400000000000001,"15min":0.45200000000000001},"min":{"1min":0.27300000000000002,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.63900000000000001,"5min":0.63900000000000001,"15min":0.63900000000000001},"last":0.38700000000000001},{"interface":"front","average":{"1min":0.45400000000000001,"5min":0.44900000000000001,"15min":0.44800000000000001},"min":{"1min":0.32300000000000001,"5min":0.312,"15min":0.312},"max":{"1min":0.65800000000000003,"5min":0.65800000000000003,"15min":0.65800000000000003},"last":0.34599999999999997}]},{"osd":4,"last update":"Tue Sep 10 19:10:50 2024","interfaces":[{"interface":"back","average":{"1min":0.59099999999999997,"5min":0.58299999999999996,"15min":0.58099999999999996},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.81999999999999995,"5min":0.81999999999999995,"15min":0.81999999999999995},"last":0.45300000000000001},{"interface":"front","average":{"1min":0.55600000000000005,"5min":0.56799999999999995,"15min":0.56999999999999995},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.749,"5min":0.80600000000000005,"15min":0.80600000000000005},"last":0.48599999999999999}]},{"osd":5,"last update":"Tue Sep 10 19:11:08 2024","interfaces":[{"interface":"back","average":{"1min":0.54800000000000004,"5min":0.52900000000000003,"15min":0.52600000000000002},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.73599999999999999,"5min":0.73899999999999999,"15min":0.73899999999999999},"last":0.44},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.57899999999999996,"15min":0.57899999999999996},"min":{"1min":0.443,"5min":0.443,"15min":0.443},"max":{"1min":0.70699999999999996,"5min":0.82699999999999996,"15min":0.82699999999999996},"last":0.51600000000000001}]},{"osd":6,"last update":"Tue Sep 10 19:10:32 2024","interfaces":[{"interface":"back","average":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"min":{"1min":0.432,"5min":0.432,"15min":0.432},"max":{"1min":0.73199999999999998,"5min":0.73199999999999998,"15min":0.73199999999999998},"last":0.53700000000000003},{"interface":"front","average":{"1min":0.55100000000000005,"5min":0.55100000000000005,"15min":0.55100000000000005},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.748,"5min":0.748,"15min":0.748},"last":0.60299999999999998}]},{"osd":7,"last update":"Tue Sep 10 19:11:00 2024","interfaces":[{"interface":"back","average":{"1min":0.60599999999999998,"5min":0.60599999999999998,"15min":0.60599999999999998},"min":{"1min":0.44700000000000001,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":0.79700000000000004,"5min":0.79700000000000004,"15min":0.79700000000000004},"last":0.47799999999999998},{"interface":"front","average":{"1min":0.54900000000000004,"5min":0.54900000000000004,"15min":0.54900000000000004},"min":{"1min":0.40300000000000002,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":0.72599999999999998,"5min":0.72599999999999998,"15min":0.72599999999999998},"last":0.625}]}]},{"osd":1,"up_from":13,"seq":55834574898,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298260,"kb_used_data":908,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93446892,"statfs":{"total":95995035648,"available":95689617408,"internally_reserved":0,"allocated":929792,"data_stored":399674,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 19:10:39 2024","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.36499999999999999,"15min":0.33400000000000002},"min":{"1min":0.188,"5min":0.188,"15min":0.188},"max":{"1min":2.7730000000000001,"5min":2.7730000000000001,"15min":2.7730000000000001},"last":0.246},{"interface":"front","average":{"1min":0.47999999999999998,"5min":0.373,"15min":0.34499999999999997},"min":{"1min":0.20599999999999999,"5min":0.20599999999999999,"15min":0.20599999999999999},"max":{"1min":2.8380000000000001,"5min":2.8380000000000001,"15min":2.8380000000000001},"last":0.26000000000000001}]},{"osd":2,"last update":"Tue Sep 10 19:11:01 2024","interfaces":[{"interface":"back","average":{"1min":0.42399999999999999,"5min":0.38500000000000001,"15min":0.373},"min":{"1min":0.251,"5min":0.251,"15min":0.251},"max":{"1min":1.1779999999999999,"5min":1.1779999999999999,"15min":1.1779999999999999},"last":0.33300000000000002},{"interface":"front","average":{"1min":0.42899999999999999,"5min":0.38,"15min":0.36799999999999999},"min":{"1min":0.254,"5min":0.254,"15min":0.254},"max":{"1min":1.103,"5min":1.103,"15min":1.103},"last":0.47399999999999998}]},{"osd":3,"last update":"Tue Sep 10 19:11:29 2024","interfaces":[{"interface":"back","average":{"1min":0.44700000000000001,"5min":0.45600000000000002,"15min":0.44900000000000001},"min":{"1min":0.27500000000000002,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":1.093,"5min":2.0670000000000002,"15min":2.0670000000000002},"last":0.27800000000000002},{"interface":"front","average":{"1min":0.40100000000000002,"5min":0.42299999999999999,"15min":0.42099999999999999},"min":{"1min":0.30299999999999999,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":0.54600000000000004,"5min":1.956,"15min":1.956},"last":0.35299999999999998}]},{"osd":4,"last update":"Tue Sep 10 19:10:49 2024","interfaces":[{"interface":"back","average":{"1min":0.61299999999999999,"5min":0.52200000000000002,"15min":0.50700000000000001},"min":{"1min":0.28499999999999998,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":2.665,"5min":2.665,"15min":2.665},"last":0.42899999999999999},{"interface":"front","average":{"1min":0.65200000000000002,"5min":0.56200000000000006,"15min":0.54700000000000004},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":2.7330000000000001,"5min":2.7330000000000001,"15min":2.7330000000000001},"last":0.64500000000000002}]},{"osd":5,"last update":"Tue Sep 10 19:11:11 2024","interfaces":[{"interface":"back","average":{"1min":0.58299999999999996,"5min":0.53800000000000003,"15min":0.53000000000000003},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":1.851,"5min":1.851,"15min":1.851},"last":0.54400000000000004},{"interface":"front","average":{"1min":0.63500000000000001,"5min":0.55100000000000005,"15min":0.53700000000000003},"min":{"1min":0.31900000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":2.706,"5min":2.706,"15min":2.706},"last":0.50700000000000001}]},{"osd":6,"last update":"Tue Sep 10 19:10:31 2024","interfaces":[{"interface":"back","average":{"1min":0.61899999999999999,"5min":0.61899999999999999,"15min":0.61899999999999999},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":1.9950000000000001,"5min":1.9950000000000001,"15min":1.9950000000000001},"last":0.41599999999999998},{"interface":"front","average":{"1min":0.59799999999999998,"5min":0.59799999999999998,"15min":0.59799999999999998},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":1.9399999999999999,"5min":1.9399999999999999,"15min":1.9399999999999999},"last":0.39300000000000002}]},{"osd":7,"last update":"Tue Sep 10 19:10:55 2024","interfaces":[{"interface":"back","average":{"1min":0.66600000000000004,"5min":0.66600000000000004,"15min":0.66600000000000004},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":3.4620000000000002,"5min":3.4620000000000002,"15min":3.4620000000000002},"last":0.45200000000000001},{"interface":"front","average":{"1min":0.621,"5min":0.621,"15min":0.621},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":1.514,"5min":1.514,"15min":1.514},"last":0.46100000000000002}]}]},{"osd":0,"up_from":8,"seq":34359738423,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298808,"kb_used_data":1456,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93446344,"statfs":{"total":95995035648,"available":95689056256,"internally_reserved":0,"allocated":1490944,"data_stored":989601,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Sep 10 19:10:42 2024","interfaces":[{"interface":"back","average":{"1min":0.373,"5min":0.34899999999999998,"15min":0.33100000000000002},"min":{"1min":0.221,"5min":0.221,"15min":0.221},"max":{"1min":0.53100000000000003,"5min":0.63300000000000001,"15min":0.63300000000000001},"last":0.31},{"interface":"front","average":{"1min":0.36499999999999999,"5min":0.34999999999999998,"15min":0.33400000000000002},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":0.51900000000000002,"5min":0.61899999999999999,"15min":0.61899999999999999},"last":0.33700000000000002}]},{"osd":2,"last update":"Tue Sep 10 19:11:08 2024","interfaces":[{"interface":"back","average":{"1min":0.375,"5min":0.38700000000000001,"15min":0.38300000000000001},"min":{"1min":0.27600000000000002,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":0.50700000000000001,"5min":0.57399999999999995,"15min":0.57399999999999995},"last":0.27100000000000002},{"interface":"front","average":{"1min":0.36499999999999999,"5min":0.375,"15min":0.373},"min":{"1min":0.22900000000000001,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":0.53000000000000003,"5min":0.54100000000000004,"15min":0.54100000000000004},"last":0.32400000000000001}]},{"osd":3,"last update":"Tue Sep 10 19:10:27 2024","interfaces":[{"interface":"back","average":{"1min":0.44600000000000001,"5min":0.42799999999999999,"15min":0.42499999999999999},"min":{"1min":0.27300000000000002,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.65000000000000002,"5min":0.65000000000000002,"15min":0.65000000000000002},"last":0.29499999999999998},{"interface":"front","average":{"1min":0.42499999999999999,"5min":0.41199999999999998,"15min":0.40999999999999998},"min":{"1min":0.246,"5min":0.246,"15min":0.246},"max":{"1min":0.55300000000000005,"5min":0.58799999999999997,"15min":0.58799999999999997},"last":0.35499999999999998}]},{"osd":4,"last update":"Tue Sep 10 19:10:46 2024","interfaces":[{"interface":"back","average":{"1min":0.54700000000000004,"5min":0.56200000000000006,"15min":0.56399999999999995},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.76000000000000001,"5min":0.76000000000000001,"15min":0.76000000000000001},"last":0.44700000000000001},{"interface":"front","average":{"1min":0.56999999999999995,"5min":0.56299999999999994,"15min":0.56200000000000006},"min":{"1min":0.40200000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.73699999999999999,"5min":0.77000000000000002,"15min":0.77000000000000002},"last":0.38400000000000001}]},{"osd":5,"last update":"Tue Sep 10 19:11:08 2024","interfaces":[{"interface":"back","average":{"1min":0.58399999999999996,"5min":0.58399999999999996,"15min":0.58399999999999996},"min":{"1min":0.44800000000000001,"5min":0.378,"15min":0.378},"max":{"1min":0.82599999999999996,"5min":0.82599999999999996,"15min":0.82599999999999996},"last":0.39900000000000002},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.57399999999999995,"15min":0.57499999999999996},"min":{"1min":0.42699999999999999,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.77900000000000003,"5min":0.82899999999999996,"15min":0.82899999999999996},"last":0.50900000000000001}]},{"osd":6,"last update":"Tue Sep 10 19:10:33 2024","interfaces":[{"interface":"back","average":{"1min":0.56699999999999995,"5min":0.56699999999999995,"15min":0.56699999999999995},"min":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":0.71799999999999997,"5min":0.71799999999999997,"15min":0.71799999999999997},"last":0.46500000000000002},{"interface":"front","average":{"1min":0.58699999999999997,"5min":0.58699999999999997,"15min":0.58699999999999997},"min":{"1min":0.44,"5min":0.44,"15min":0.44},"max":{"1min":0.76100000000000001,"5min":0.76100000000000001,"15min":0.76100000000000001},"last":0.42699999999999999}]},{"osd":7,"last update":"Tue Sep 10 19:10:58 2024","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.57799999999999996,"15min":0.57799999999999996},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":0.76900000000000002,"5min":0.76900000000000002,"15min":0.76900000000000002},"last":0.36799999999999999},{"interface":"front","average":{"1min":0.56200000000000006,"5min":0.56200000000000006,"15min":0.56200000000000006},"min":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.72599999999999998,"5min":0.72599999999999998,"15min":0.72599999999999998},"last":0.53200000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":966,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1236,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1236,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":362,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":2338,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":798,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T19:11:33.804 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph pg dump --format=json 2024-09-10T19:11:34.060 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:11:34.181 INFO:journalctl@ceph.node-exporter.a.smithi183.stdout:Sep 10 19:11:33 smithi183 podman[78288]: 2024-09-10 19:11:33.911743694 +0000 UTC m=+3.687757215 container init 518c3989dce0713f1a8b2dd14da9174ec8c56045558be06bf3147ed2c8c4f680 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-node-exporter-a, maintainer=The Prometheus Authors ) 2024-09-10T19:11:34.182 INFO:journalctl@ceph.node-exporter.a.smithi183.stdout:Sep 10 19:11:33 smithi183 podman[78288]: 2024-09-10 19:11:33.917628766 +0000 UTC m=+3.693642283 container start 518c3989dce0713f1a8b2dd14da9174ec8c56045558be06bf3147ed2c8c4f680 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-node-exporter-a, maintainer=The Prometheus Authors ) 2024-09-10T19:11:34.182 INFO:journalctl@ceph.node-exporter.a.smithi183.stdout:Sep 10 19:11:33 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-node-exporter-a[78542]: ts=2024-09-10T19:11:33.921Z caller=node_exporter.go:180 level=info msg="Starting node_exporter" version="(version=1.5.0, branch=HEAD, revision=1b48970ffcf5630534fb00bb0687d73c66d1c959)" 2024-09-10T19:11:34.182 INFO:journalctl@ceph.node-exporter.a.smithi183.stdout:Sep 10 19:11:33 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-node-exporter-a[78542]: ts=2024-09-10T19:11:33.921Z caller=node_exporter.go:181 level=info msg="Build context" build_context="(go=go1.19.3, user=root@6e7732a7b81b, date=20221129-18:59:09)" 2024-09-10T19:11:34.182 INFO:journalctl@ceph.node-exporter.a.smithi183.stdout:Sep 10 19:11:33 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-node-exporter-a[78542]: ts=2024-09-10T19:11:33.922Z caller=diskstats_common.go:111 level=info collector=diskstats msg="Parsed flag --collector.diskstats.device-exclude" flag=^(ram|loop|fd|(h|s|v|xv)d[a-z]|nvme\d+n\d+p)\d+$ 2024-09-10T19:11:34.182 INFO:journalctl@ceph.node-exporter.a.smithi183.stdout:Sep 10 19:11:33 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-node-exporter-a[78542]: ts=2024-09-10T19:11:33.922Z caller=diskstats_linux.go:264 level=error collector=diskstats msg="Failed to open directory, disabling udev device properties" path=/run/udev/data 2024-09-10T19:11:34.182 INFO:journalctl@ceph.node-exporter.a.smithi183.stdout:Sep 10 19:11:33 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-node-exporter-a[78542]: ts=2024-09-10T19:11:33.922Z caller=filesystem_common.go:111 level=info collector=filesystem msg="Parsed flag --collector.filesystem.mount-points-exclude" flag=^/(dev|proc|run/credentials/.+|sys|var/lib/docker/.+|var/lib/containers/storage/.+)($|/) 2024-09-10T19:11:34.182 INFO:journalctl@ceph.node-exporter.a.smithi183.stdout:Sep 10 19:11:33 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-node-exporter-a[78542]: ts=2024-09-10T19:11:33.922Z caller=filesystem_common.go:113 level=info collector=filesystem msg="Parsed flag --collector.filesystem.fs-types-exclude" flag=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$ 2024-09-10T19:11:34.182 INFO:journalctl@ceph.node-exporter.a.smithi183.stdout:Sep 10 19:11:33 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-node-exporter-a[78542]: ts=2024-09-10T19:11:33.923Z caller=node_exporter.go:110 level=info msg="Enabled collectors" 2024-09-10T19:11:34.182 INFO:journalctl@ceph.node-exporter.a.smithi183.stdout:Sep 10 19:11:33 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-node-exporter-a[78542]: ts=2024-09-10T19:11:33.923Z caller=node_exporter.go:117 level=info collector=arp 2024-09-10T19:11:34.182 INFO:journalctl@ceph.node-exporter.a.smithi183.stdout:Sep 10 19:11:33 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-node-exporter-a[78542]: ts=2024-09-10T19:11:33.923Z caller=node_exporter.go:117 level=info collector=bcache 2024-09-10T19:11:34.182 INFO:journalctl@ceph.node-exporter.a.smithi183.stdout:Sep 10 19:11:33 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-node-exporter-a[78542]: ts=2024-09-10T19:11:33.923Z caller=node_exporter.go:117 level=info collector=bonding 2024-09-10T19:11:34.182 INFO:journalctl@ceph.node-exporter.a.smithi183.stdout:Sep 10 19:11:33 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-node-exporter-a[78542]: ts=2024-09-10T19:11:33.923Z caller=node_exporter.go:117 level=info collector=btrfs 2024-09-10T19:11:34.183 INFO:journalctl@ceph.node-exporter.a.smithi183.stdout:Sep 10 19:11:33 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-node-exporter-a[78542]: ts=2024-09-10T19:11:33.923Z caller=node_exporter.go:117 level=info collector=conntrack 2024-09-10T19:11:34.183 INFO:journalctl@ceph.node-exporter.a.smithi183.stdout:Sep 10 19:11:33 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-node-exporter-a[78542]: ts=2024-09-10T19:11:33.923Z caller=node_exporter.go:117 level=info collector=cpu 2024-09-10T19:11:34.183 INFO:journalctl@ceph.node-exporter.a.smithi183.stdout:Sep 10 19:11:33 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-node-exporter-a[78542]: ts=2024-09-10T19:11:33.923Z caller=node_exporter.go:117 level=info collector=cpufreq 2024-09-10T19:11:34.183 INFO:journalctl@ceph.node-exporter.a.smithi183.stdout:Sep 10 19:11:33 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-node-exporter-a[78542]: ts=2024-09-10T19:11:33.923Z caller=node_exporter.go:117 level=info collector=diskstats 2024-09-10T19:11:34.183 INFO:journalctl@ceph.node-exporter.a.smithi183.stdout:Sep 10 19:11:33 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-node-exporter-a[78542]: ts=2024-09-10T19:11:33.923Z caller=node_exporter.go:117 level=info collector=dmi 2024-09-10T19:11:34.183 INFO:journalctl@ceph.node-exporter.a.smithi183.stdout:Sep 10 19:11:33 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-node-exporter-a[78542]: ts=2024-09-10T19:11:33.923Z caller=node_exporter.go:117 level=info collector=edac 2024-09-10T19:11:34.183 INFO:journalctl@ceph.node-exporter.a.smithi183.stdout:Sep 10 19:11:33 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-node-exporter-a[78542]: ts=2024-09-10T19:11:33.923Z caller=node_exporter.go:117 level=info collector=entropy 2024-09-10T19:11:34.184 INFO:journalctl@ceph.node-exporter.a.smithi183.stdout:Sep 10 19:11:33 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-node-exporter-a[78542]: ts=2024-09-10T19:11:33.923Z caller=node_exporter.go:117 level=info collector=fibrechannel 2024-09-10T19:11:34.184 INFO:journalctl@ceph.node-exporter.a.smithi183.stdout:Sep 10 19:11:33 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-node-exporter-a[78542]: ts=2024-09-10T19:11:33.923Z caller=node_exporter.go:117 level=info collector=filefd 2024-09-10T19:11:34.184 INFO:journalctl@ceph.node-exporter.a.smithi183.stdout:Sep 10 19:11:33 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-node-exporter-a[78542]: ts=2024-09-10T19:11:33.923Z caller=node_exporter.go:117 level=info collector=filesystem 2024-09-10T19:11:34.184 INFO:journalctl@ceph.node-exporter.a.smithi183.stdout:Sep 10 19:11:33 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-node-exporter-a[78542]: ts=2024-09-10T19:11:33.923Z caller=node_exporter.go:117 level=info collector=hwmon 2024-09-10T19:11:34.184 INFO:journalctl@ceph.node-exporter.a.smithi183.stdout:Sep 10 19:11:33 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-node-exporter-a[78542]: ts=2024-09-10T19:11:33.923Z caller=node_exporter.go:117 level=info collector=infiniband 2024-09-10T19:11:34.184 INFO:journalctl@ceph.node-exporter.a.smithi183.stdout:Sep 10 19:11:33 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-node-exporter-a[78542]: ts=2024-09-10T19:11:33.923Z caller=node_exporter.go:117 level=info collector=ipvs 2024-09-10T19:11:34.184 INFO:journalctl@ceph.node-exporter.a.smithi183.stdout:Sep 10 19:11:33 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-node-exporter-a[78542]: ts=2024-09-10T19:11:33.923Z caller=node_exporter.go:117 level=info collector=loadavg 2024-09-10T19:11:34.184 INFO:journalctl@ceph.node-exporter.a.smithi183.stdout:Sep 10 19:11:33 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-node-exporter-a[78542]: ts=2024-09-10T19:11:33.923Z caller=node_exporter.go:117 level=info collector=mdadm 2024-09-10T19:11:34.184 INFO:journalctl@ceph.node-exporter.a.smithi183.stdout:Sep 10 19:11:33 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-node-exporter-a[78542]: ts=2024-09-10T19:11:33.923Z caller=node_exporter.go:117 level=info collector=meminfo 2024-09-10T19:11:34.184 INFO:journalctl@ceph.node-exporter.a.smithi183.stdout:Sep 10 19:11:33 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-node-exporter-a[78542]: ts=2024-09-10T19:11:33.923Z caller=node_exporter.go:117 level=info collector=netclass 2024-09-10T19:11:34.184 INFO:journalctl@ceph.node-exporter.a.smithi183.stdout:Sep 10 19:11:33 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-node-exporter-a[78542]: ts=2024-09-10T19:11:33.923Z caller=node_exporter.go:117 level=info collector=netdev 2024-09-10T19:11:34.184 INFO:journalctl@ceph.node-exporter.a.smithi183.stdout:Sep 10 19:11:33 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-node-exporter-a[78542]: ts=2024-09-10T19:11:33.923Z caller=node_exporter.go:117 level=info collector=netstat 2024-09-10T19:11:34.184 INFO:journalctl@ceph.node-exporter.a.smithi183.stdout:Sep 10 19:11:33 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-node-exporter-a[78542]: ts=2024-09-10T19:11:33.923Z caller=node_exporter.go:117 level=info collector=nfs 2024-09-10T19:11:34.184 INFO:journalctl@ceph.node-exporter.a.smithi183.stdout:Sep 10 19:11:33 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-node-exporter-a[78542]: ts=2024-09-10T19:11:33.923Z caller=node_exporter.go:117 level=info collector=nfsd 2024-09-10T19:11:34.185 INFO:journalctl@ceph.node-exporter.a.smithi183.stdout:Sep 10 19:11:33 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-node-exporter-a[78542]: ts=2024-09-10T19:11:33.923Z caller=node_exporter.go:117 level=info collector=nvme 2024-09-10T19:11:34.185 INFO:journalctl@ceph.node-exporter.a.smithi183.stdout:Sep 10 19:11:33 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-node-exporter-a[78542]: ts=2024-09-10T19:11:33.923Z caller=node_exporter.go:117 level=info collector=os 2024-09-10T19:11:34.185 INFO:journalctl@ceph.node-exporter.a.smithi183.stdout:Sep 10 19:11:33 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-node-exporter-a[78542]: ts=2024-09-10T19:11:33.923Z caller=node_exporter.go:117 level=info collector=powersupplyclass 2024-09-10T19:11:34.185 INFO:journalctl@ceph.node-exporter.a.smithi183.stdout:Sep 10 19:11:33 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-node-exporter-a[78542]: ts=2024-09-10T19:11:33.923Z caller=node_exporter.go:117 level=info collector=pressure 2024-09-10T19:11:34.185 INFO:journalctl@ceph.node-exporter.a.smithi183.stdout:Sep 10 19:11:33 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-node-exporter-a[78542]: ts=2024-09-10T19:11:33.923Z caller=node_exporter.go:117 level=info collector=rapl 2024-09-10T19:11:34.185 INFO:journalctl@ceph.node-exporter.a.smithi183.stdout:Sep 10 19:11:33 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-node-exporter-a[78542]: ts=2024-09-10T19:11:33.923Z caller=node_exporter.go:117 level=info collector=schedstat 2024-09-10T19:11:34.185 INFO:journalctl@ceph.node-exporter.a.smithi183.stdout:Sep 10 19:11:33 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-node-exporter-a[78542]: ts=2024-09-10T19:11:33.923Z caller=node_exporter.go:117 level=info collector=selinux 2024-09-10T19:11:34.185 INFO:journalctl@ceph.node-exporter.a.smithi183.stdout:Sep 10 19:11:33 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-node-exporter-a[78542]: ts=2024-09-10T19:11:33.923Z caller=node_exporter.go:117 level=info collector=sockstat 2024-09-10T19:11:34.185 INFO:journalctl@ceph.node-exporter.a.smithi183.stdout:Sep 10 19:11:33 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-node-exporter-a[78542]: ts=2024-09-10T19:11:33.923Z caller=node_exporter.go:117 level=info collector=softnet 2024-09-10T19:11:34.185 INFO:journalctl@ceph.node-exporter.a.smithi183.stdout:Sep 10 19:11:33 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-node-exporter-a[78542]: ts=2024-09-10T19:11:33.923Z caller=node_exporter.go:117 level=info collector=stat 2024-09-10T19:11:34.185 INFO:journalctl@ceph.node-exporter.a.smithi183.stdout:Sep 10 19:11:33 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-node-exporter-a[78542]: ts=2024-09-10T19:11:33.923Z caller=node_exporter.go:117 level=info collector=tapestats 2024-09-10T19:11:34.185 INFO:journalctl@ceph.node-exporter.a.smithi183.stdout:Sep 10 19:11:33 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-node-exporter-a[78542]: ts=2024-09-10T19:11:33.923Z caller=node_exporter.go:117 level=info collector=textfile 2024-09-10T19:11:34.185 INFO:journalctl@ceph.node-exporter.a.smithi183.stdout:Sep 10 19:11:33 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-node-exporter-a[78542]: ts=2024-09-10T19:11:33.923Z caller=node_exporter.go:117 level=info collector=thermal_zone 2024-09-10T19:11:34.186 INFO:journalctl@ceph.node-exporter.a.smithi183.stdout:Sep 10 19:11:33 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-node-exporter-a[78542]: ts=2024-09-10T19:11:33.923Z caller=node_exporter.go:117 level=info collector=time 2024-09-10T19:11:34.186 INFO:journalctl@ceph.node-exporter.a.smithi183.stdout:Sep 10 19:11:33 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-node-exporter-a[78542]: ts=2024-09-10T19:11:33.923Z caller=node_exporter.go:117 level=info collector=udp_queues 2024-09-10T19:11:34.186 INFO:journalctl@ceph.node-exporter.a.smithi183.stdout:Sep 10 19:11:33 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-node-exporter-a[78542]: ts=2024-09-10T19:11:33.923Z caller=node_exporter.go:117 level=info collector=uname 2024-09-10T19:11:34.186 INFO:journalctl@ceph.node-exporter.a.smithi183.stdout:Sep 10 19:11:33 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-node-exporter-a[78542]: ts=2024-09-10T19:11:33.923Z caller=node_exporter.go:117 level=info collector=vmstat 2024-09-10T19:11:34.186 INFO:journalctl@ceph.node-exporter.a.smithi183.stdout:Sep 10 19:11:33 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-node-exporter-a[78542]: ts=2024-09-10T19:11:33.923Z caller=node_exporter.go:117 level=info collector=xfs 2024-09-10T19:11:34.186 INFO:journalctl@ceph.node-exporter.a.smithi183.stdout:Sep 10 19:11:33 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-node-exporter-a[78542]: ts=2024-09-10T19:11:33.923Z caller=node_exporter.go:117 level=info collector=zfs 2024-09-10T19:11:34.186 INFO:journalctl@ceph.node-exporter.a.smithi183.stdout:Sep 10 19:11:33 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-node-exporter-a[78542]: ts=2024-09-10T19:11:33.923Z caller=tls_config.go:232 level=info msg="Listening on" address=[::]:9100 2024-09-10T19:11:34.186 INFO:journalctl@ceph.node-exporter.a.smithi183.stdout:Sep 10 19:11:33 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-node-exporter-a[78542]: ts=2024-09-10T19:11:33.923Z caller=tls_config.go:235 level=info msg="TLS is disabled." http2=false address=[::]:9100 2024-09-10T19:11:34.186 INFO:journalctl@ceph.node-exporter.a.smithi183.stdout:Sep 10 19:11:33 smithi183 bash[78288]: 518c3989dce0713f1a8b2dd14da9174ec8c56045558be06bf3147ed2c8c4f680 2024-09-10T19:11:34.186 INFO:journalctl@ceph.node-exporter.a.smithi183.stdout:Sep 10 19:11:33 smithi183 systemd[1]: Started Ceph node-exporter.a for 320f6df8-6fa7-11ef-bcea-c7b262605968. 2024-09-10T19:11:34.519 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:11:34 smithi183 ceph-mon[29736]: 2.1c scrub starts 2024-09-10T19:11:34.519 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:11:34 smithi183 ceph-mon[29736]: 2.1c scrub ok 2024-09-10T19:11:34.519 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:11:34 smithi183 ceph-mon[29736]: 2.1 scrub starts 2024-09-10T19:11:34.519 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:11:34 smithi183 ceph-mon[29736]: 2.1 scrub ok 2024-09-10T19:11:34.519 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:11:34 smithi183 ceph-mon[29736]: 4.8 scrub starts 2024-09-10T19:11:34.519 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:11:34 smithi183 ceph-mon[29736]: 4.8 scrub ok 2024-09-10T19:11:34.519 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:11:34 smithi183 ceph-mon[29736]: pgmap v35: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 984 B/s rd, 0 op/s 2024-09-10T19:11:34.519 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:11:34 smithi183 ceph-mon[29736]: from='mgr.14556 ' entity='mgr.y' 2024-09-10T19:11:34.519 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:11:34 smithi183 ceph-mon[29736]: from='mgr.14556 ' entity='mgr.y' 2024-09-10T19:11:34.519 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:11:34 smithi183 ceph-mon[29736]: from='mgr.14556 ' entity='mgr.y' 2024-09-10T19:11:34.519 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:11:34 smithi183 ceph-mon[29736]: Deploying daemon node-exporter.b on smithi191 2024-09-10T19:11:34.520 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:11:34 smithi183 ceph-mon[41562]: 2.1c scrub starts 2024-09-10T19:11:34.520 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:11:34 smithi183 ceph-mon[41562]: 2.1c scrub ok 2024-09-10T19:11:34.520 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:11:34 smithi183 ceph-mon[41562]: 2.1 scrub starts 2024-09-10T19:11:34.520 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:11:34 smithi183 ceph-mon[41562]: 2.1 scrub ok 2024-09-10T19:11:34.520 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:11:34 smithi183 ceph-mon[41562]: 4.8 scrub starts 2024-09-10T19:11:34.520 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:11:34 smithi183 ceph-mon[41562]: 4.8 scrub ok 2024-09-10T19:11:34.520 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:11:34 smithi183 ceph-mon[41562]: pgmap v35: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 984 B/s rd, 0 op/s 2024-09-10T19:11:34.520 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:11:34 smithi183 ceph-mon[41562]: from='mgr.14556 ' entity='mgr.y' 2024-09-10T19:11:34.520 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:11:34 smithi183 ceph-mon[41562]: from='mgr.14556 ' entity='mgr.y' 2024-09-10T19:11:34.520 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:11:34 smithi183 ceph-mon[41562]: from='mgr.14556 ' entity='mgr.y' 2024-09-10T19:11:34.520 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:11:34 smithi183 ceph-mon[41562]: Deploying daemon node-exporter.b on smithi191 2024-09-10T19:11:34.575 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:11:34 smithi191 ceph-mon[32884]: 2.1c scrub starts 2024-09-10T19:11:34.575 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:11:34 smithi191 ceph-mon[32884]: 2.1c scrub ok 2024-09-10T19:11:34.575 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:11:34 smithi191 ceph-mon[32884]: 2.1 scrub starts 2024-09-10T19:11:34.576 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:11:34 smithi191 ceph-mon[32884]: 2.1 scrub ok 2024-09-10T19:11:34.576 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:11:34 smithi191 ceph-mon[32884]: 4.8 scrub starts 2024-09-10T19:11:34.576 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:11:34 smithi191 ceph-mon[32884]: 4.8 scrub ok 2024-09-10T19:11:34.576 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:11:34 smithi191 ceph-mon[32884]: pgmap v35: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 984 B/s rd, 0 op/s 2024-09-10T19:11:34.576 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:11:34 smithi191 ceph-mon[32884]: from='mgr.14556 ' entity='mgr.y' 2024-09-10T19:11:34.576 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:11:34 smithi191 ceph-mon[32884]: from='mgr.14556 ' entity='mgr.y' 2024-09-10T19:11:34.576 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:11:34 smithi191 ceph-mon[32884]: from='mgr.14556 ' entity='mgr.y' 2024-09-10T19:11:34.576 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:11:34 smithi191 ceph-mon[32884]: Deploying daemon node-exporter.b on smithi191 2024-09-10T19:11:34.952 INFO:journalctl@ceph.node-exporter.b.smithi191.stdout:Sep 10 19:11:34 smithi191 systemd[1]: Starting Ceph node-exporter.b for 320f6df8-6fa7-11ef-bcea-c7b262605968... 2024-09-10T19:11:35.216 INFO:journalctl@ceph.node-exporter.b.smithi191.stdout:Sep 10 19:11:34 smithi191 bash[66046]: Trying to pull quay.io/prometheus/node-exporter:v1.5.0... 2024-09-10T19:11:35.655 INFO:journalctl@ceph.node-exporter.b.smithi191.stdout:Sep 10 19:11:35 smithi191 bash[66046]: Getting image source signatures 2024-09-10T19:11:35.655 INFO:journalctl@ceph.node-exporter.b.smithi191.stdout:Sep 10 19:11:35 smithi191 bash[66046]: Copying blob sha256:c0e87333d3805a7b7e1c25783dd25f27da0fe70a8e262eee7ed190c8dc843fa7 2024-09-10T19:11:35.655 INFO:journalctl@ceph.node-exporter.b.smithi191.stdout:Sep 10 19:11:35 smithi191 bash[66046]: Copying blob sha256:22b70bddd3acadc892fca4c2af4260629bfda5dfd11ebc106a93ce24e752b5ed 2024-09-10T19:11:35.655 INFO:journalctl@ceph.node-exporter.b.smithi191.stdout:Sep 10 19:11:35 smithi191 bash[66046]: Copying blob sha256:5c12815fee558b157dee7f7509dedbaba0a8379098858a65ec869e1f1526ea0c 2024-09-10T19:11:35.890 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:11:35 smithi183 ceph-mon[29736]: 6.2 scrub starts 2024-09-10T19:11:35.890 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:11:35 smithi183 ceph-mon[29736]: 6.2 scrub ok 2024-09-10T19:11:35.890 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:11:35 smithi183 ceph-mon[29736]: 2.19 scrub starts 2024-09-10T19:11:35.890 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:11:35 smithi183 ceph-mon[29736]: 2.19 scrub ok 2024-09-10T19:11:35.890 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:11:35 smithi183 ceph-mon[29736]: 2.17 scrub starts 2024-09-10T19:11:35.890 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:11:35 smithi183 ceph-mon[29736]: 2.17 scrub ok 2024-09-10T19:11:35.890 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:11:35 smithi183 ceph-mon[29736]: 5.9 deep-scrub starts 2024-09-10T19:11:35.890 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:11:35 smithi183 ceph-mon[29736]: 5.9 deep-scrub ok 2024-09-10T19:11:35.890 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:11:35 smithi183 ceph-mon[41562]: 6.2 scrub starts 2024-09-10T19:11:35.891 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:11:35 smithi183 ceph-mon[41562]: 6.2 scrub ok 2024-09-10T19:11:35.891 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:11:35 smithi183 ceph-mon[41562]: 2.19 scrub starts 2024-09-10T19:11:35.891 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:11:35 smithi183 ceph-mon[41562]: 2.19 scrub ok 2024-09-10T19:11:35.891 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:11:35 smithi183 ceph-mon[41562]: 2.17 scrub starts 2024-09-10T19:11:35.891 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:11:35 smithi183 ceph-mon[41562]: 2.17 scrub ok 2024-09-10T19:11:35.891 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:11:35 smithi183 ceph-mon[41562]: 5.9 deep-scrub starts 2024-09-10T19:11:35.891 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:11:35 smithi183 ceph-mon[41562]: 5.9 deep-scrub ok 2024-09-10T19:11:35.965 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:11:35 smithi191 ceph-mon[32884]: 6.2 scrub starts 2024-09-10T19:11:35.965 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:11:35 smithi191 ceph-mon[32884]: 6.2 scrub ok 2024-09-10T19:11:35.965 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:11:35 smithi191 ceph-mon[32884]: 2.19 scrub starts 2024-09-10T19:11:35.965 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:11:35 smithi191 ceph-mon[32884]: 2.19 scrub ok 2024-09-10T19:11:35.965 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:11:35 smithi191 ceph-mon[32884]: 2.17 scrub starts 2024-09-10T19:11:35.966 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:11:35 smithi191 ceph-mon[32884]: 2.17 scrub ok 2024-09-10T19:11:35.966 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:11:35 smithi191 ceph-mon[32884]: 5.9 deep-scrub starts 2024-09-10T19:11:35.966 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:11:35 smithi191 ceph-mon[32884]: 5.9 deep-scrub ok 2024-09-10T19:11:36.031 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-09-10T19:11:36.033 INFO:teuthology.orchestra.run.smithi183.stderr:dumped all 2024-09-10T19:11:36.694 INFO:teuthology.orchestra.run.smithi183.stdout:{"pg_ready":true,"pg_map":{"version":36,"stamp":"2024-09-10T19:11:35.602519+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596188,"num_objects":196,"num_object_clones":0,"num_object_copies":588,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":196,"num_whiteouts":0,"num_read":932,"num_read_kb":789,"num_write":543,"num_write_kb":1671,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":566,"ondisk_log_size":566,"up":396,"acting":396,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":396,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":2387576,"kb_used_data":8760,"kb_used_omap":13,"kb_used_meta":2378738,"kb_avail":747573640,"statfs":{"total":767960285184,"available":765515407360,"internally_reserved":0,"allocated":8970240,"data_stored":4963216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13632,"internal_metadata":2435828416},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":15,"num_read_kb":15,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002396"},"pg_stats":[{"pgid":"6.1b","version":"0'0","reported_seq":24,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-09-10T19:11:21.194696+0000","last_change":"2024-09-10T19:10:14.056737+0000","last_active":"2024-09-10T19:11:21.194696+0000","last_peered":"2024-09-10T19:11:21.194696+0000","last_clean":"2024-09-10T19:11:21.194696+0000","last_became_active":"2024-09-10T19:10:14.056187+0000","last_became_peered":"2024-09-10T19:10:14.056187+0000","last_unstale":"2024-09-10T19:11:21.194696+0000","last_undegraded":"2024-09-10T19:11:21.194696+0000","last_fullsized":"2024-09-10T19:11:21.194696+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:10:12.693598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:11:30.090418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":30,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-09-10T19:11:21.302847+0000","last_change":"2024-09-10T19:10:07.688713+0000","last_active":"2024-09-10T19:11:21.302847+0000","last_peered":"2024-09-10T19:11:21.302847+0000","last_clean":"2024-09-10T19:11:21.302847+0000","last_became_active":"2024-09-10T19:10:07.688372+0000","last_became_peered":"2024-09-10T19:10:07.688372+0000","last_unstale":"2024-09-10T19:11:21.302847+0000","last_undegraded":"2024-09-10T19:11:21.302847+0000","last_fullsized":"2024-09-10T19:11:21.302847+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:10:06.664709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:11:33.189992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.19","version":"66'15","reported_seq":53,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-09-10T19:11:21.194822+0000","last_change":"2024-09-10T19:10:09.688886+0000","last_active":"2024-09-10T19:11:21.194822+0000","last_peered":"2024-09-10T19:11:21.194822+0000","last_clean":"2024-09-10T19:11:21.194822+0000","last_became_active":"2024-09-10T19:10:09.688432+0000","last_became_peered":"2024-09-10T19:10:09.688432+0000","last_unstale":"2024-09-10T19:11:21.194822+0000","last_undegraded":"2024-09-10T19:11:21.194822+0000","last_fullsized":"2024-09-10T19:11:21.194822+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:10:08.671214+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:11:30.404066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":36,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-09-10T19:11:23.086323+0000","last_change":"2024-09-10T19:11:23.086288+0000","last_active":"2024-09-10T19:11:23.086323+0000","last_peered":"2024-09-10T19:11:23.086323+0000","last_clean":"2024-09-10T19:11:23.086323+0000","last_became_active":"2024-09-10T19:10:11.697523+0000","last_became_peered":"2024-09-10T19:10:11.697523+0000","last_unstale":"2024-09-10T19:11:23.086323+0000","last_undegraded":"2024-09-10T19:11:23.086323+0000","last_fullsized":"2024-09-10T19:11:23.086323+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:11:23.086191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:11:23.086191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:12:31.131189+0000","scrub_duration":0.0016087969999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"59'1","reported_seq":48,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-09-10T19:11:21.195475+0000","last_change":"2024-09-10T19:11:15.782493+0000","last_active":"2024-09-10T19:11:21.195475+0000","last_peered":"2024-09-10T19:11:21.195475+0000","last_clean":"2024-09-10T19:11:21.195475+0000","last_became_active":"2024-09-10T19:10:07.686143+0000","last_became_peered":"2024-09-10T19:10:07.686143+0000","last_unstale":"2024-09-10T19:11:21.195475+0000","last_undegraded":"2024-09-10T19:11:21.195475+0000","last_fullsized":"2024-09-10T19:11:21.195475+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'1","last_scrub_stamp":"2024-09-10T19:11:15.782393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:11:15.782393+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:12:21.959971+0000","scrub_duration":0.00183947,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":874,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"66'9","reported_seq":44,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-09-10T19:11:21.815200+0000","last_change":"2024-09-10T19:10:09.692950+0000","last_active":"2024-09-10T19:11:21.815200+0000","last_peered":"2024-09-10T19:11:21.815200+0000","last_clean":"2024-09-10T19:11:21.815200+0000","last_became_active":"2024-09-10T19:10:09.692681+0000","last_became_peered":"2024-09-10T19:10:09.692681+0000","last_unstale":"2024-09-10T19:11:21.815200+0000","last_undegraded":"2024-09-10T19:11:21.815200+0000","last_fullsized":"2024-09-10T19:11:21.815200+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:10:08.671214+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:11:38.074554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":26,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-09-10T19:11:21.189235+0000","last_change":"2024-09-10T19:10:11.702830+0000","last_active":"2024-09-10T19:11:21.189235+0000","last_peered":"2024-09-10T19:11:21.189235+0000","last_clean":"2024-09-10T19:11:21.189235+0000","last_became_active":"2024-09-10T19:10:11.702546+0000","last_became_peered":"2024-09-10T19:10:11.702546+0000","last_unstale":"2024-09-10T19:11:21.189235+0000","last_undegraded":"2024-09-10T19:11:21.189235+0000","last_fullsized":"2024-09-10T19:11:21.189235+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:10:10.676876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:11:31.938904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":34,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-10T19:11:32.056376+0000","last_change":"2024-09-10T19:11:32.056320+0000","last_active":"2024-09-10T19:11:32.056376+0000","last_peered":"2024-09-10T19:11:32.056376+0000","last_clean":"2024-09-10T19:11:32.056376+0000","last_became_active":"2024-09-10T19:10:13.716022+0000","last_became_peered":"2024-09-10T19:10:13.716022+0000","last_unstale":"2024-09-10T19:11:32.056376+0000","last_undegraded":"2024-09-10T19:11:32.056376+0000","last_fullsized":"2024-09-10T19:11:32.056376+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:11:32.056254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:11:32.056254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:12:39.600944+0000","scrub_duration":0.0013010680000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":40,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-09-10T19:11:21.301148+0000","last_change":"2024-09-10T19:11:19.275407+0000","last_active":"2024-09-10T19:11:21.301148+0000","last_peered":"2024-09-10T19:11:21.301148+0000","last_clean":"2024-09-10T19:11:21.301148+0000","last_became_active":"2024-09-10T19:10:07.690728+0000","last_became_peered":"2024-09-10T19:10:07.690728+0000","last_unstale":"2024-09-10T19:11:21.301148+0000","last_undegraded":"2024-09-10T19:11:21.301148+0000","last_fullsized":"2024-09-10T19:11:21.301148+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:11:19.275331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:11:19.275331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:12:30.648634+0000","scrub_duration":0.001355258,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1b","version":"66'5","reported_seq":48,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-10T19:11:24.101637+0000","last_change":"2024-09-10T19:11:24.101604+0000","last_active":"2024-09-10T19:11:24.101637+0000","last_peered":"2024-09-10T19:11:24.101637+0000","last_clean":"2024-09-10T19:11:24.101637+0000","last_became_active":"2024-09-10T19:10:09.693690+0000","last_became_peered":"2024-09-10T19:10:09.693690+0000","last_unstale":"2024-09-10T19:11:24.101637+0000","last_undegraded":"2024-09-10T19:11:24.101637+0000","last_fullsized":"2024-09-10T19:11:24.101637+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'5","last_scrub_stamp":"2024-09-10T19:11:24.101551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:11:24.101551+0000","objects_scrubbed":2,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:12:31.538341+0000","scrub_duration":0.0019539829999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":26,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-09-10T19:11:21.300812+0000","last_change":"2024-09-10T19:10:11.697342+0000","last_active":"2024-09-10T19:11:21.300812+0000","last_peered":"2024-09-10T19:11:21.300812+0000","last_clean":"2024-09-10T19:11:21.300812+0000","last_became_active":"2024-09-10T19:10:11.697088+0000","last_became_peered":"2024-09-10T19:10:11.697088+0000","last_unstale":"2024-09-10T19:11:21.300812+0000","last_undegraded":"2024-09-10T19:11:21.300812+0000","last_fullsized":"2024-09-10T19:11:21.300812+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:10:10.676876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:11:27.002523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":24,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-09-10T19:11:21.186715+0000","last_change":"2024-09-10T19:10:13.724478+0000","last_active":"2024-09-10T19:11:21.186715+0000","last_peered":"2024-09-10T19:11:21.186715+0000","last_clean":"2024-09-10T19:11:21.186715+0000","last_became_active":"2024-09-10T19:10:13.724255+0000","last_became_peered":"2024-09-10T19:10:13.724255+0000","last_unstale":"2024-09-10T19:11:21.186715+0000","last_undegraded":"2024-09-10T19:11:21.186715+0000","last_fullsized":"2024-09-10T19:11:21.186715+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:10:12.693598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:11:28.931682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":40,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-10T19:11:33.268393+0000","last_change":"2024-09-10T19:11:33.268356+0000","last_active":"2024-09-10T19:11:33.268393+0000","last_peered":"2024-09-10T19:11:33.268393+0000","last_clean":"2024-09-10T19:11:33.268393+0000","last_became_active":"2024-09-10T19:10:07.690174+0000","last_became_peered":"2024-09-10T19:10:07.690174+0000","last_unstale":"2024-09-10T19:11:33.268393+0000","last_undegraded":"2024-09-10T19:11:33.268393+0000","last_fullsized":"2024-09-10T19:11:33.268393+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:11:33.268282+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:11:33.268282+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:12:35.423155+0000","scrub_duration":0.0015798069999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1a","version":"66'9","reported_seq":54,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-10T19:11:31.035993+0000","last_change":"2024-09-10T19:11:31.035964+0000","last_active":"2024-09-10T19:11:31.035993+0000","last_peered":"2024-09-10T19:11:31.035993+0000","last_clean":"2024-09-10T19:11:31.035993+0000","last_became_active":"2024-09-10T19:10:09.688987+0000","last_became_peered":"2024-09-10T19:10:09.688987+0000","last_unstale":"2024-09-10T19:11:31.035993+0000","last_undegraded":"2024-09-10T19:11:31.035993+0000","last_fullsized":"2024-09-10T19:11:31.035993+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'9","last_scrub_stamp":"2024-09-10T19:11:31.035912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:11:31.035912+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:12:57.913355+0000","scrub_duration":0.001674916,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":36,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-10T19:11:27.317920+0000","last_change":"2024-09-10T19:11:27.317868+0000","last_active":"2024-09-10T19:11:27.317920+0000","last_peered":"2024-09-10T19:11:27.317920+0000","last_clean":"2024-09-10T19:11:27.317920+0000","last_became_active":"2024-09-10T19:10:11.701799+0000","last_became_peered":"2024-09-10T19:10:11.701799+0000","last_unstale":"2024-09-10T19:11:27.317920+0000","last_undegraded":"2024-09-10T19:11:27.317920+0000","last_fullsized":"2024-09-10T19:11:27.317920+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:11:27.317779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:11:27.317779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:12:44.343114+0000","scrub_duration":0.001509584,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":34,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-09-10T19:11:21.302551+0000","last_change":"2024-09-10T19:11:12.970227+0000","last_active":"2024-09-10T19:11:21.302551+0000","last_peered":"2024-09-10T19:11:21.302551+0000","last_clean":"2024-09-10T19:11:21.302551+0000","last_became_active":"2024-09-10T19:10:13.717310+0000","last_became_peered":"2024-09-10T19:10:13.717310+0000","last_unstale":"2024-09-10T19:11:21.302551+0000","last_undegraded":"2024-09-10T19:11:21.302551+0000","last_fullsized":"2024-09-10T19:11:21.302551+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:11:12.970173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:11:12.970173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:12:27.060515+0000","scrub_duration":0.001527219,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.d","version":"66'17","reported_seq":66,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-09-10T19:11:21.815465+0000","last_change":"2024-09-10T19:11:14.007973+0000","last_active":"2024-09-10T19:11:21.815465+0000","last_peered":"2024-09-10T19:11:21.815465+0000","last_clean":"2024-09-10T19:11:21.815465+0000","last_became_active":"2024-09-10T19:10:09.694016+0000","last_became_peered":"2024-09-10T19:10:09.694016+0000","last_unstale":"2024-09-10T19:11:21.815465+0000","last_undegraded":"2024-09-10T19:11:21.815465+0000","last_fullsized":"2024-09-10T19:11:21.815465+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'17","last_scrub_stamp":"2024-09-10T19:11:14.007902+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:11:14.007902+0000","objects_scrubbed":8,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:12:21.164915+0000","scrub_duration":0.0024520879999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":40,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-09-10T19:11:22.280126+0000","last_change":"2024-09-10T19:11:22.280067+0000","last_active":"2024-09-10T19:11:22.280126+0000","last_peered":"2024-09-10T19:11:22.280126+0000","last_clean":"2024-09-10T19:11:22.280126+0000","last_became_active":"2024-09-10T19:10:07.691119+0000","last_became_peered":"2024-09-10T19:10:07.691119+0000","last_unstale":"2024-09-10T19:11:22.280126+0000","last_undegraded":"2024-09-10T19:11:22.280126+0000","last_fullsized":"2024-09-10T19:11:22.280126+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:11:22.280002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:11:22.280002+0000","last_clean_scrub_stamp":"2024-09-10T19:11:22.280002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:12:27.848752+0000","scrub_duration":0.0013828040000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":26,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-09-10T19:11:21.189594+0000","last_change":"2024-09-10T19:10:11.695393+0000","last_active":"2024-09-10T19:11:21.189594+0000","last_peered":"2024-09-10T19:11:21.189594+0000","last_clean":"2024-09-10T19:11:21.189594+0000","last_became_active":"2024-09-10T19:10:11.695141+0000","last_became_peered":"2024-09-10T19:10:11.695141+0000","last_unstale":"2024-09-10T19:11:21.189594+0000","last_undegraded":"2024-09-10T19:11:21.189594+0000","last_fullsized":"2024-09-10T19:11:21.189594+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:10:10.676876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:11:37.843556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":34,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-10T19:11:24.145082+0000","last_change":"2024-09-10T19:11:24.145056+0000","last_active":"2024-09-10T19:11:24.145082+0000","last_peered":"2024-09-10T19:11:24.145082+0000","last_clean":"2024-09-10T19:11:24.145082+0000","last_became_active":"2024-09-10T19:10:13.722894+0000","last_became_peered":"2024-09-10T19:10:13.722894+0000","last_unstale":"2024-09-10T19:11:24.145082+0000","last_undegraded":"2024-09-10T19:11:24.145082+0000","last_fullsized":"2024-09-10T19:11:24.145082+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:11:24.145003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:11:24.145003+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:12:26.405847+0000","scrub_duration":0.0013083789999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.c","version":"66'10","reported_seq":53,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-09-10T19:11:21.816215+0000","last_change":"2024-09-10T19:11:14.987470+0000","last_active":"2024-09-10T19:11:21.816215+0000","last_peered":"2024-09-10T19:11:21.816215+0000","last_clean":"2024-09-10T19:11:21.816215+0000","last_became_active":"2024-09-10T19:10:09.688321+0000","last_became_peered":"2024-09-10T19:10:09.688321+0000","last_unstale":"2024-09-10T19:11:21.816215+0000","last_undegraded":"2024-09-10T19:11:21.816215+0000","last_fullsized":"2024-09-10T19:11:21.816215+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'10","last_scrub_stamp":"2024-09-10T19:11:14.987403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:11:14.987403+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:12:21.734174+0000","scrub_duration":0.0018747600000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":40,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-10T19:11:26.110254+0000","last_change":"2024-09-10T19:11:26.110216+0000","last_active":"2024-09-10T19:11:26.110254+0000","last_peered":"2024-09-10T19:11:26.110254+0000","last_clean":"2024-09-10T19:11:26.110254+0000","last_became_active":"2024-09-10T19:10:07.680803+0000","last_became_peered":"2024-09-10T19:10:07.680803+0000","last_unstale":"2024-09-10T19:11:26.110254+0000","last_undegraded":"2024-09-10T19:11:26.110254+0000","last_fullsized":"2024-09-10T19:11:26.110254+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:11:26.110138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:11:26.110138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:12:35.580206+0000","scrub_duration":0.0011581549999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.d","version":"80'9","reported_seq":65,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-10T19:11:25.764663+0000","last_change":"2024-09-10T19:11:25.156512+0000","last_active":"2024-09-10T19:11:25.764663+0000","last_peered":"2024-09-10T19:11:25.764663+0000","last_clean":"2024-09-10T19:11:25.764663+0000","last_became_active":"2024-09-10T19:10:11.698547+0000","last_became_peered":"2024-09-10T19:10:11.698547+0000","last_unstale":"2024-09-10T19:11:25.764663+0000","last_undegraded":"2024-09-10T19:11:25.764663+0000","last_fullsized":"2024-09-10T19:11:25.764663+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"80'9","last_scrub_stamp":"2024-09-10T19:11:25.156465+0000","last_deep_scrub":"80'9","last_deep_scrub_stamp":"2024-09-10T19:11:25.156465+0000","last_clean_scrub_stamp":"2024-09-10T19:11:25.156465+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:12:26.762818+0000","scrub_duration":0.0022299569999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":34,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-10T19:11:26.096176+0000","last_change":"2024-09-10T19:11:26.096147+0000","last_active":"2024-09-10T19:11:26.096176+0000","last_peered":"2024-09-10T19:11:26.096176+0000","last_clean":"2024-09-10T19:11:26.096176+0000","last_became_active":"2024-09-10T19:10:13.718242+0000","last_became_peered":"2024-09-10T19:10:13.718242+0000","last_unstale":"2024-09-10T19:11:26.096176+0000","last_undegraded":"2024-09-10T19:11:26.096176+0000","last_fullsized":"2024-09-10T19:11:26.096176+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:11:26.096090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:11:26.096090+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:12:49.347064+0000","scrub_duration":0.0015599909999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.f","version":"66'15","reported_seq":53,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-09-10T19:11:21.189954+0000","last_change":"2024-09-10T19:10:09.691727+0000","last_active":"2024-09-10T19:11:21.189954+0000","last_peered":"2024-09-10T19:11:21.189954+0000","last_clean":"2024-09-10T19:11:21.189954+0000","last_became_active":"2024-09-10T19:10:09.691230+0000","last_became_peered":"2024-09-10T19:10:09.691230+0000","last_unstale":"2024-09-10T19:11:21.189954+0000","last_undegraded":"2024-09-10T19:11:21.189954+0000","last_fullsized":"2024-09-10T19:11:21.189954+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:10:08.671214+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:11:34.422017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":40,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-09-10T19:11:21.189383+0000","last_change":"2024-09-10T19:11:11.036017+0000","last_active":"2024-09-10T19:11:21.189383+0000","last_peered":"2024-09-10T19:11:21.189383+0000","last_clean":"2024-09-10T19:11:21.189383+0000","last_became_active":"2024-09-10T19:10:07.679646+0000","last_became_peered":"2024-09-10T19:10:07.679646+0000","last_unstale":"2024-09-10T19:11:21.189383+0000","last_undegraded":"2024-09-10T19:11:21.189383+0000","last_fullsized":"2024-09-10T19:11:21.189383+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:11:11.035954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:11:11.035954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:12:35.469086+0000","scrub_duration":0.001618024,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.e","version":"66'8","reported_seq":51,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-10T19:11:25.764916+0000","last_change":"2024-09-10T19:10:11.699487+0000","last_active":"2024-09-10T19:11:25.764916+0000","last_peered":"2024-09-10T19:11:25.764916+0000","last_clean":"2024-09-10T19:11:25.764916+0000","last_became_active":"2024-09-10T19:10:11.699232+0000","last_became_peered":"2024-09-10T19:10:11.699232+0000","last_unstale":"2024-09-10T19:11:25.764916+0000","last_undegraded":"2024-09-10T19:11:25.764916+0000","last_fullsized":"2024-09-10T19:11:25.764916+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:10:10.676876+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:11:36.171473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":34,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-09-10T19:11:21.187406+0000","last_change":"2024-09-10T19:11:18.208847+0000","last_active":"2024-09-10T19:11:21.187406+0000","last_peered":"2024-09-10T19:11:21.187406+0000","last_clean":"2024-09-10T19:11:21.187406+0000","last_became_active":"2024-09-10T19:10:13.715274+0000","last_became_peered":"2024-09-10T19:10:13.715274+0000","last_unstale":"2024-09-10T19:11:21.187406+0000","last_undegraded":"2024-09-10T19:11:21.187406+0000","last_fullsized":"2024-09-10T19:11:21.187406+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:11:18.208782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:11:18.208782+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:12:43.016878+0000","scrub_duration":0.001904481,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.e","version":"66'11","reported_seq":47,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-09-10T19:11:21.816254+0000","last_change":"2024-09-10T19:10:09.692995+0000","last_active":"2024-09-10T19:11:21.816254+0000","last_peered":"2024-09-10T19:11:21.816254+0000","last_clean":"2024-09-10T19:11:21.816254+0000","last_became_active":"2024-09-10T19:10:09.692675+0000","last_became_peered":"2024-09-10T19:10:09.692675+0000","last_unstale":"2024-09-10T19:11:21.816254+0000","last_undegraded":"2024-09-10T19:11:21.816254+0000","last_fullsized":"2024-09-10T19:11:21.816254+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:10:08.671214+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:11:34.182028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":30,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-09-10T19:11:21.300365+0000","last_change":"2024-09-10T19:10:07.690701+0000","last_active":"2024-09-10T19:11:21.300365+0000","last_peered":"2024-09-10T19:11:21.300365+0000","last_clean":"2024-09-10T19:11:21.300365+0000","last_became_active":"2024-09-10T19:10:07.690434+0000","last_became_peered":"2024-09-10T19:10:07.690434+0000","last_unstale":"2024-09-10T19:11:21.300365+0000","last_undegraded":"2024-09-10T19:11:21.300365+0000","last_fullsized":"2024-09-10T19:11:21.300365+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:10:06.664709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:11:28.466436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":26,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-09-10T19:11:21.187698+0000","last_change":"2024-09-10T19:10:11.692483+0000","last_active":"2024-09-10T19:11:21.187698+0000","last_peered":"2024-09-10T19:11:21.187698+0000","last_clean":"2024-09-10T19:11:21.187698+0000","last_became_active":"2024-09-10T19:10:11.692215+0000","last_became_peered":"2024-09-10T19:10:11.692215+0000","last_unstale":"2024-09-10T19:11:21.187698+0000","last_undegraded":"2024-09-10T19:11:21.187698+0000","last_fullsized":"2024-09-10T19:11:21.187698+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:10:10.676876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:11:38.454128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":24,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-09-10T19:11:21.194837+0000","last_change":"2024-09-10T19:10:14.054106+0000","last_active":"2024-09-10T19:11:21.194837+0000","last_peered":"2024-09-10T19:11:21.194837+0000","last_clean":"2024-09-10T19:11:21.194837+0000","last_became_active":"2024-09-10T19:10:14.053736+0000","last_became_peered":"2024-09-10T19:10:14.053736+0000","last_unstale":"2024-09-10T19:11:21.194837+0000","last_undegraded":"2024-09-10T19:11:21.194837+0000","last_fullsized":"2024-09-10T19:11:21.194837+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:10:12.693598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:11:32.848291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"66'14","reported_seq":59,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-09-10T19:11:21.815965+0000","last_change":"2024-09-10T19:11:16.976833+0000","last_active":"2024-09-10T19:11:21.815965+0000","last_peered":"2024-09-10T19:11:21.815965+0000","last_clean":"2024-09-10T19:11:21.815965+0000","last_became_active":"2024-09-10T19:10:09.686534+0000","last_became_peered":"2024-09-10T19:10:09.686534+0000","last_unstale":"2024-09-10T19:11:21.815965+0000","last_undegraded":"2024-09-10T19:11:21.815965+0000","last_fullsized":"2024-09-10T19:11:21.815965+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'14","last_scrub_stamp":"2024-09-10T19:11:16.976776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:11:16.976776+0000","objects_scrubbed":7,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:12:29.125897+0000","scrub_duration":0.001777144,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":40,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-09-10T19:11:22.801241+0000","last_change":"2024-09-10T19:11:22.801187+0000","last_active":"2024-09-10T19:11:22.801241+0000","last_peered":"2024-09-10T19:11:22.801241+0000","last_clean":"2024-09-10T19:11:22.801241+0000","last_became_active":"2024-09-10T19:10:07.684864+0000","last_became_peered":"2024-09-10T19:10:07.684864+0000","last_unstale":"2024-09-10T19:11:22.801241+0000","last_undegraded":"2024-09-10T19:11:22.801241+0000","last_fullsized":"2024-09-10T19:11:22.801241+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:11:22.801123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:11:22.801123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:12:47.860716+0000","scrub_duration":0.0015474410000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.0","version":"66'8","reported_seq":61,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-10T19:11:29.884830+0000","last_change":"2024-09-10T19:11:29.884792+0000","last_active":"2024-09-10T19:11:29.884830+0000","last_peered":"2024-09-10T19:11:29.884830+0000","last_clean":"2024-09-10T19:11:29.884830+0000","last_became_active":"2024-09-10T19:10:11.697106+0000","last_became_peered":"2024-09-10T19:10:11.697106+0000","last_unstale":"2024-09-10T19:11:29.884830+0000","last_undegraded":"2024-09-10T19:11:29.884830+0000","last_fullsized":"2024-09-10T19:11:29.884830+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'8","last_scrub_stamp":"2024-09-10T19:11:29.884739+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:11:29.884739+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:12:37.090103+0000","scrub_duration":0.0014372650000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":24,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-09-10T19:11:21.300549+0000","last_change":"2024-09-10T19:10:14.054712+0000","last_active":"2024-09-10T19:11:21.300549+0000","last_peered":"2024-09-10T19:11:21.300549+0000","last_clean":"2024-09-10T19:11:21.300549+0000","last_became_active":"2024-09-10T19:10:14.054456+0000","last_became_peered":"2024-09-10T19:10:14.054456+0000","last_unstale":"2024-09-10T19:11:21.300549+0000","last_undegraded":"2024-09-10T19:11:21.300549+0000","last_fullsized":"2024-09-10T19:11:21.300549+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:10:12.693598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:11:24.269111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.0","version":"66'18","reported_seq":70,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-09-10T19:11:21.194693+0000","last_change":"2024-09-10T19:11:19.848186+0000","last_active":"2024-09-10T19:11:21.194693+0000","last_peered":"2024-09-10T19:11:21.194693+0000","last_clean":"2024-09-10T19:11:21.194693+0000","last_became_active":"2024-09-10T19:10:09.821836+0000","last_became_peered":"2024-09-10T19:10:09.821836+0000","last_unstale":"2024-09-10T19:11:21.194693+0000","last_undegraded":"2024-09-10T19:11:21.194693+0000","last_fullsized":"2024-09-10T19:11:21.194693+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'18","last_scrub_stamp":"2024-09-10T19:11:19.848132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:11:19.848132+0000","objects_scrubbed":8,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:12:26.692213+0000","scrub_duration":0.0034247819999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":30,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-09-10T19:11:21.189816+0000","last_change":"2024-09-10T19:10:07.682359+0000","last_active":"2024-09-10T19:11:21.189816+0000","last_peered":"2024-09-10T19:11:21.189816+0000","last_clean":"2024-09-10T19:11:21.189816+0000","last_became_active":"2024-09-10T19:10:07.682081+0000","last_became_peered":"2024-09-10T19:10:07.682081+0000","last_unstale":"2024-09-10T19:11:21.189816+0000","last_undegraded":"2024-09-10T19:11:21.189816+0000","last_fullsized":"2024-09-10T19:11:21.189816+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:10:06.664709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:11:25.212065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":36,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-10T19:11:29.069453+0000","last_change":"2024-09-10T19:11:29.069421+0000","last_active":"2024-09-10T19:11:29.069453+0000","last_peered":"2024-09-10T19:11:29.069453+0000","last_clean":"2024-09-10T19:11:29.069453+0000","last_became_active":"2024-09-10T19:10:11.698054+0000","last_became_peered":"2024-09-10T19:10:11.698054+0000","last_unstale":"2024-09-10T19:11:29.069453+0000","last_undegraded":"2024-09-10T19:11:29.069453+0000","last_fullsized":"2024-09-10T19:11:29.069453+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:11:29.069369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:11:29.069369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:12:55.817612+0000","scrub_duration":0.001327562,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":34,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-10T19:11:34.139417+0000","last_change":"2024-09-10T19:11:34.139371+0000","last_active":"2024-09-10T19:11:34.139417+0000","last_peered":"2024-09-10T19:11:34.139417+0000","last_clean":"2024-09-10T19:11:34.139417+0000","last_became_active":"2024-09-10T19:10:13.722784+0000","last_became_peered":"2024-09-10T19:10:13.722784+0000","last_unstale":"2024-09-10T19:11:34.139417+0000","last_undegraded":"2024-09-10T19:11:34.139417+0000","last_fullsized":"2024-09-10T19:11:34.139417+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:11:34.139306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:11:34.139306+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:12:45.529547+0000","scrub_duration":0.00133668,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.3","version":"66'19","reported_seq":64,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-09-10T19:11:21.302758+0000","last_change":"2024-09-10T19:10:09.823263+0000","last_active":"2024-09-10T19:11:21.302758+0000","last_peered":"2024-09-10T19:11:21.302758+0000","last_clean":"2024-09-10T19:11:21.302758+0000","last_became_active":"2024-09-10T19:10:09.823003+0000","last_became_peered":"2024-09-10T19:10:09.823003+0000","last_unstale":"2024-09-10T19:11:21.302758+0000","last_undegraded":"2024-09-10T19:11:21.302758+0000","last_fullsized":"2024-09-10T19:11:21.302758+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:10:08.671214+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:11:34.837626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":40,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-10T19:11:25.290175+0000","last_change":"2024-09-10T19:11:25.290147+0000","last_active":"2024-09-10T19:11:25.290175+0000","last_peered":"2024-09-10T19:11:25.290175+0000","last_clean":"2024-09-10T19:11:25.290175+0000","last_became_active":"2024-09-10T19:10:07.690325+0000","last_became_peered":"2024-09-10T19:10:07.690325+0000","last_unstale":"2024-09-10T19:11:25.290175+0000","last_undegraded":"2024-09-10T19:11:25.290175+0000","last_fullsized":"2024-09-10T19:11:25.290175+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:11:25.290088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:11:25.290088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:12:31.270129+0000","scrub_duration":0.0013977760000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":36,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-10T19:11:23.843357+0000","last_change":"2024-09-10T19:11:23.843315+0000","last_active":"2024-09-10T19:11:23.843357+0000","last_peered":"2024-09-10T19:11:23.843357+0000","last_clean":"2024-09-10T19:11:23.843357+0000","last_became_active":"2024-09-10T19:10:11.700370+0000","last_became_peered":"2024-09-10T19:10:11.700370+0000","last_unstale":"2024-09-10T19:11:23.843357+0000","last_undegraded":"2024-09-10T19:11:23.843357+0000","last_fullsized":"2024-09-10T19:11:23.843357+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:11:23.843228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:11:23.843228+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:12:29.095733+0000","scrub_duration":0.0013052070000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":24,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-09-10T19:11:21.189202+0000","last_change":"2024-09-10T19:10:14.054654+0000","last_active":"2024-09-10T19:11:21.189202+0000","last_peered":"2024-09-10T19:11:21.189202+0000","last_clean":"2024-09-10T19:11:21.189202+0000","last_became_active":"2024-09-10T19:10:14.054318+0000","last_became_peered":"2024-09-10T19:10:14.054318+0000","last_unstale":"2024-09-10T19:11:21.189202+0000","last_undegraded":"2024-09-10T19:11:21.189202+0000","last_fullsized":"2024-09-10T19:11:21.189202+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:10:12.693598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:11:26.731659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"66'10","reported_seq":53,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-10T19:11:28.132964+0000","last_change":"2024-09-10T19:11:28.132910+0000","last_active":"2024-09-10T19:11:28.132964+0000","last_peered":"2024-09-10T19:11:28.132964+0000","last_clean":"2024-09-10T19:11:28.132964+0000","last_became_active":"2024-09-10T19:10:09.689815+0000","last_became_peered":"2024-09-10T19:10:09.689815+0000","last_unstale":"2024-09-10T19:11:28.132964+0000","last_undegraded":"2024-09-10T19:11:28.132964+0000","last_fullsized":"2024-09-10T19:11:28.132964+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'10","last_scrub_stamp":"2024-09-10T19:11:28.132850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:11:28.132850+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:12:42.376407+0000","scrub_duration":0.0018173429999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":30,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-09-10T19:11:21.189857+0000","last_change":"2024-09-10T19:10:07.682243+0000","last_active":"2024-09-10T19:11:21.189857+0000","last_peered":"2024-09-10T19:11:21.189857+0000","last_clean":"2024-09-10T19:11:21.189857+0000","last_became_active":"2024-09-10T19:10:07.681965+0000","last_became_peered":"2024-09-10T19:10:07.681965+0000","last_unstale":"2024-09-10T19:11:21.189857+0000","last_undegraded":"2024-09-10T19:11:21.189857+0000","last_fullsized":"2024-09-10T19:11:21.189857+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:10:06.664709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:11:28.605002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"66'8","reported_seq":51,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-10T19:11:25.764868+0000","last_change":"2024-09-10T19:10:11.699975+0000","last_active":"2024-09-10T19:11:25.764868+0000","last_peered":"2024-09-10T19:11:25.764868+0000","last_clean":"2024-09-10T19:11:25.764868+0000","last_became_active":"2024-09-10T19:10:11.699690+0000","last_became_peered":"2024-09-10T19:10:11.699690+0000","last_unstale":"2024-09-10T19:11:25.764868+0000","last_undegraded":"2024-09-10T19:11:25.764868+0000","last_fullsized":"2024-09-10T19:11:25.764868+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:10:10.676876+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:11:32.470332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":24,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-09-10T19:11:21.302269+0000","last_change":"2024-09-10T19:10:13.720657+0000","last_active":"2024-09-10T19:11:21.302269+0000","last_peered":"2024-09-10T19:11:21.302269+0000","last_clean":"2024-09-10T19:11:21.302269+0000","last_became_active":"2024-09-10T19:10:13.720403+0000","last_became_peered":"2024-09-10T19:10:13.720403+0000","last_unstale":"2024-09-10T19:11:21.302269+0000","last_undegraded":"2024-09-10T19:11:21.302269+0000","last_fullsized":"2024-09-10T19:11:21.302269+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:10:12.693598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:11:28.077386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"66'28","reported_seq":103,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-10T19:11:25.764591+0000","last_change":"2024-09-10T19:11:17.050501+0000","last_active":"2024-09-10T19:11:25.764591+0000","last_peered":"2024-09-10T19:11:25.764591+0000","last_clean":"2024-09-10T19:11:25.764591+0000","last_became_active":"2024-09-10T19:10:09.692663+0000","last_became_peered":"2024-09-10T19:10:09.692663+0000","last_unstale":"2024-09-10T19:11:25.764591+0000","last_undegraded":"2024-09-10T19:11:25.764591+0000","last_fullsized":"2024-09-10T19:11:25.764591+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'28","last_scrub_stamp":"2024-09-10T19:11:17.050442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:11:17.050442+0000","objects_scrubbed":10,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:12:31.661637+0000","scrub_duration":0.001985559,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":30,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-09-10T19:11:21.187081+0000","last_change":"2024-09-10T19:10:07.683032+0000","last_active":"2024-09-10T19:11:21.187081+0000","last_peered":"2024-09-10T19:11:21.187081+0000","last_clean":"2024-09-10T19:11:21.187081+0000","last_became_active":"2024-09-10T19:10:07.682724+0000","last_became_peered":"2024-09-10T19:10:07.682724+0000","last_unstale":"2024-09-10T19:11:21.187081+0000","last_undegraded":"2024-09-10T19:11:21.187081+0000","last_fullsized":"2024-09-10T19:11:21.187081+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:10:06.664709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:11:28.664405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":36,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-09-10T19:11:21.303045+0000","last_change":"2024-09-10T19:11:14.985340+0000","last_active":"2024-09-10T19:11:21.303045+0000","last_peered":"2024-09-10T19:11:21.303045+0000","last_clean":"2024-09-10T19:11:21.303045+0000","last_became_active":"2024-09-10T19:10:11.698825+0000","last_became_peered":"2024-09-10T19:10:11.698825+0000","last_unstale":"2024-09-10T19:11:21.303045+0000","last_undegraded":"2024-09-10T19:11:21.303045+0000","last_fullsized":"2024-09-10T19:11:21.303045+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:11:14.985249+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:11:14.985249+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:12:30.155012+0000","scrub_duration":0.001460161,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":24,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-09-10T19:11:21.195389+0000","last_change":"2024-09-10T19:10:13.722038+0000","last_active":"2024-09-10T19:11:21.195389+0000","last_peered":"2024-09-10T19:11:21.195389+0000","last_clean":"2024-09-10T19:11:21.195389+0000","last_became_active":"2024-09-10T19:10:13.721744+0000","last_became_peered":"2024-09-10T19:10:13.721744+0000","last_unstale":"2024-09-10T19:11:21.195389+0000","last_undegraded":"2024-09-10T19:11:21.195389+0000","last_fullsized":"2024-09-10T19:11:21.195389+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:10:12.693598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:11:42.038674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.7","version":"66'13","reported_seq":55,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-09-10T19:11:21.190003+0000","last_change":"2024-09-10T19:10:09.693625+0000","last_active":"2024-09-10T19:11:21.190003+0000","last_peered":"2024-09-10T19:11:21.190003+0000","last_clean":"2024-09-10T19:11:21.190003+0000","last_became_active":"2024-09-10T19:10:09.693346+0000","last_became_peered":"2024-09-10T19:10:09.693346+0000","last_unstale":"2024-09-10T19:11:21.190003+0000","last_undegraded":"2024-09-10T19:11:21.190003+0000","last_fullsized":"2024-09-10T19:11:21.190003+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:10:08.671214+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:11:26.671345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":19,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"67'5","reported_seq":129,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-10T19:11:32.907502+0000","last_change":"2024-09-10T19:11:16.007355+0000","last_active":"2024-09-10T19:11:32.907502+0000","last_peered":"2024-09-10T19:11:32.907502+0000","last_clean":"2024-09-10T19:11:32.907502+0000","last_became_active":"2024-09-10T19:10:07.682125+0000","last_became_peered":"2024-09-10T19:10:07.682125+0000","last_unstale":"2024-09-10T19:11:32.907502+0000","last_undegraded":"2024-09-10T19:11:32.907502+0000","last_fullsized":"2024-09-10T19:11:32.907502+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'5","last_scrub_stamp":"2024-09-10T19:11:16.007261+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:11:16.007261+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:12:39.490457+0000","scrub_duration":0.0019059680000000001,"objects_trimmed":0,"snaptrim_duration":0.00034454499999999997,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":81,"num_read_kb":76,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":40,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-10T19:11:33.340819+0000","last_change":"2024-09-10T19:11:33.340773+0000","last_active":"2024-09-10T19:11:33.340819+0000","last_peered":"2024-09-10T19:11:33.340819+0000","last_clean":"2024-09-10T19:11:33.340819+0000","last_became_active":"2024-09-10T19:10:07.681456+0000","last_became_peered":"2024-09-10T19:10:07.681456+0000","last_unstale":"2024-09-10T19:11:33.340819+0000","last_undegraded":"2024-09-10T19:11:33.340819+0000","last_fullsized":"2024-09-10T19:11:33.340819+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:11:33.340715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:11:33.340715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:12:37.277130+0000","scrub_duration":0.0012840099999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":26,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-09-10T19:11:21.815541+0000","last_change":"2024-09-10T19:10:11.695520+0000","last_active":"2024-09-10T19:11:21.815541+0000","last_peered":"2024-09-10T19:11:21.815541+0000","last_clean":"2024-09-10T19:11:21.815541+0000","last_became_active":"2024-09-10T19:10:11.695205+0000","last_became_peered":"2024-09-10T19:10:11.695205+0000","last_unstale":"2024-09-10T19:11:21.815541+0000","last_undegraded":"2024-09-10T19:11:21.815541+0000","last_fullsized":"2024-09-10T19:11:21.815541+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:10:10.676876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:11:28.918766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":24,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-09-10T19:11:21.300908+0000","last_change":"2024-09-10T19:10:14.054047+0000","last_active":"2024-09-10T19:11:21.300908+0000","last_peered":"2024-09-10T19:11:21.300908+0000","last_clean":"2024-09-10T19:11:21.300908+0000","last_became_active":"2024-09-10T19:10:14.053697+0000","last_became_peered":"2024-09-10T19:10:14.053697+0000","last_unstale":"2024-09-10T19:11:21.300908+0000","last_undegraded":"2024-09-10T19:11:21.300908+0000","last_fullsized":"2024-09-10T19:11:21.300908+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:10:12.693598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:11:40.555328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"66'12","reported_seq":46,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-09-10T19:11:21.302911+0000","last_change":"2024-09-10T19:10:09.688409+0000","last_active":"2024-09-10T19:11:21.302911+0000","last_peered":"2024-09-10T19:11:21.302911+0000","last_clean":"2024-09-10T19:11:21.302911+0000","last_became_active":"2024-09-10T19:10:09.688091+0000","last_became_peered":"2024-09-10T19:10:09.688091+0000","last_unstale":"2024-09-10T19:11:21.302911+0000","last_undegraded":"2024-09-10T19:11:21.302911+0000","last_fullsized":"2024-09-10T19:11:21.302911+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:10:08.671214+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:11:26.174026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"61'1","reported_seq":36,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-09-10T19:11:21.302500+0000","last_change":"2024-09-10T19:10:11.693049+0000","last_active":"2024-09-10T19:11:21.302500+0000","last_peered":"2024-09-10T19:11:21.302500+0000","last_clean":"2024-09-10T19:11:21.302500+0000","last_became_active":"2024-09-10T19:10:07.683259+0000","last_became_peered":"2024-09-10T19:10:07.683259+0000","last_unstale":"2024-09-10T19:11:21.302500+0000","last_undegraded":"2024-09-10T19:11:21.302500+0000","last_fullsized":"2024-09-10T19:11:21.302500+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:10:06.664709+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:11:33.524399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014341130000000001,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":40,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-09-10T19:11:21.301199+0000","last_change":"2024-09-10T19:11:21.301130+0000","last_active":"2024-09-10T19:11:21.301199+0000","last_peered":"2024-09-10T19:11:21.301199+0000","last_clean":"2024-09-10T19:11:21.301199+0000","last_became_active":"2024-09-10T19:10:07.690549+0000","last_became_peered":"2024-09-10T19:10:07.690549+0000","last_unstale":"2024-09-10T19:11:21.301199+0000","last_undegraded":"2024-09-10T19:11:21.301199+0000","last_fullsized":"2024-09-10T19:11:21.301199+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:11:21.301058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:11:21.301058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:12:41.356053+0000","scrub_duration":0.0021074710000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":36,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-09-10T19:11:21.187730+0000","last_change":"2024-09-10T19:11:16.156929+0000","last_active":"2024-09-10T19:11:21.187730+0000","last_peered":"2024-09-10T19:11:21.187730+0000","last_clean":"2024-09-10T19:11:21.187730+0000","last_became_active":"2024-09-10T19:10:11.699798+0000","last_became_peered":"2024-09-10T19:10:11.699798+0000","last_unstale":"2024-09-10T19:11:21.187730+0000","last_undegraded":"2024-09-10T19:11:21.187730+0000","last_fullsized":"2024-09-10T19:11:21.187730+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:11:16.156819+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:11:16.156819+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:12:16.166187+0000","scrub_duration":0.001649059,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":24,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-09-10T19:11:21.189585+0000","last_change":"2024-09-10T19:10:13.722695+0000","last_active":"2024-09-10T19:11:21.189585+0000","last_peered":"2024-09-10T19:11:21.189585+0000","last_clean":"2024-09-10T19:11:21.189585+0000","last_became_active":"2024-09-10T19:10:13.722368+0000","last_became_peered":"2024-09-10T19:10:13.722368+0000","last_unstale":"2024-09-10T19:11:21.189585+0000","last_undegraded":"2024-09-10T19:11:21.189585+0000","last_fullsized":"2024-09-10T19:11:21.189585+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:10:12.693598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:11:39.250623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"66'14","reported_seq":50,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-09-10T19:11:21.816239+0000","last_change":"2024-09-10T19:10:09.822467+0000","last_active":"2024-09-10T19:11:21.816239+0000","last_peered":"2024-09-10T19:11:21.816239+0000","last_clean":"2024-09-10T19:11:21.816239+0000","last_became_active":"2024-09-10T19:10:09.822184+0000","last_became_peered":"2024-09-10T19:10:09.822184+0000","last_unstale":"2024-09-10T19:11:21.816239+0000","last_undegraded":"2024-09-10T19:11:21.816239+0000","last_fullsized":"2024-09-10T19:11:21.816239+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:10:08.671214+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:11:31.203195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"63'2","reported_seq":47,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-10T19:11:26.824211+0000","last_change":"2024-09-10T19:11:26.824164+0000","last_active":"2024-09-10T19:11:26.824211+0000","last_peered":"2024-09-10T19:11:26.824211+0000","last_clean":"2024-09-10T19:11:26.824211+0000","last_became_active":"2024-09-10T19:10:07.684784+0000","last_became_peered":"2024-09-10T19:10:07.684784+0000","last_unstale":"2024-09-10T19:11:26.824211+0000","last_undegraded":"2024-09-10T19:11:26.824211+0000","last_fullsized":"2024-09-10T19:11:26.824211+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'2","last_scrub_stamp":"2024-09-10T19:11:26.824103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:11:26.824103+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:12:48.229817+0000","scrub_duration":0.0018538929999999999,"objects_trimmed":0,"snaptrim_duration":0.00075522999999999996,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":40,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-09-10T19:11:21.187274+0000","last_change":"2024-09-10T19:11:17.171319+0000","last_active":"2024-09-10T19:11:21.187274+0000","last_peered":"2024-09-10T19:11:21.187274+0000","last_clean":"2024-09-10T19:11:21.187274+0000","last_became_active":"2024-09-10T19:10:07.687596+0000","last_became_peered":"2024-09-10T19:10:07.687596+0000","last_unstale":"2024-09-10T19:11:21.187274+0000","last_undegraded":"2024-09-10T19:11:21.187274+0000","last_fullsized":"2024-09-10T19:11:21.187274+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:11:17.171251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:11:17.171251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:12:21.736124+0000","scrub_duration":0.0016247670000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"74'79","reported_seq":75,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-09-10T19:11:21.300547+0000","last_change":"2024-09-10T19:10:34.385577+0000","last_active":"2024-09-10T19:11:21.300547+0000","last_peered":"2024-09-10T19:11:21.300547+0000","last_clean":"2024-09-10T19:11:21.300547+0000","last_became_active":"2024-09-10T19:09:56.634919+0000","last_became_peered":"2024-09-10T19:09:56.634919+0000","last_unstale":"2024-09-10T19:11:21.300547+0000","last_undegraded":"2024-09-10T19:11:21.300547+0000","last_fullsized":"2024-09-10T19:11:21.300547+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'72","last_scrub_stamp":"2024-09-10T19:10:34.385519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:08:00.699315+0000","last_clean_scrub_stamp":"2024-09-10T19:10:34.385519+0000","objects_scrubbed":2,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:11:51.371427+0000","scrub_duration":0.0030997160000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":26,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-09-10T19:11:21.300623+0000","last_change":"2024-09-10T19:10:11.695316+0000","last_active":"2024-09-10T19:11:21.300623+0000","last_peered":"2024-09-10T19:11:21.300623+0000","last_clean":"2024-09-10T19:11:21.300623+0000","last_became_active":"2024-09-10T19:10:11.694986+0000","last_became_peered":"2024-09-10T19:10:11.694986+0000","last_unstale":"2024-09-10T19:11:21.300623+0000","last_undegraded":"2024-09-10T19:11:21.300623+0000","last_fullsized":"2024-09-10T19:11:21.300623+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:10:10.676876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:11:38.724621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":34,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-10T19:11:25.312442+0000","last_change":"2024-09-10T19:11:25.312410+0000","last_active":"2024-09-10T19:11:25.312442+0000","last_peered":"2024-09-10T19:11:25.312442+0000","last_clean":"2024-09-10T19:11:25.312442+0000","last_became_active":"2024-09-10T19:10:13.720647+0000","last_became_peered":"2024-09-10T19:10:13.720647+0000","last_unstale":"2024-09-10T19:11:25.312442+0000","last_undegraded":"2024-09-10T19:11:25.312442+0000","last_fullsized":"2024-09-10T19:11:25.312442+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:11:25.312356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:11:25.312356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:12:54.878799+0000","scrub_duration":0.001690642,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.a","version":"66'19","reported_seq":59,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-09-10T19:11:21.815959+0000","last_change":"2024-09-10T19:10:09.821179+0000","last_active":"2024-09-10T19:11:21.815959+0000","last_peered":"2024-09-10T19:11:21.815959+0000","last_clean":"2024-09-10T19:11:21.815959+0000","last_became_active":"2024-09-10T19:10:09.820880+0000","last_became_peered":"2024-09-10T19:10:09.820880+0000","last_unstale":"2024-09-10T19:11:21.815959+0000","last_undegraded":"2024-09-10T19:11:21.815959+0000","last_fullsized":"2024-09-10T19:11:21.815959+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:10:08.671214+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:11:33.941959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":30,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-09-10T19:11:21.815568+0000","last_change":"2024-09-10T19:10:07.686475+0000","last_active":"2024-09-10T19:11:21.815568+0000","last_peered":"2024-09-10T19:11:21.815568+0000","last_clean":"2024-09-10T19:11:21.815568+0000","last_became_active":"2024-09-10T19:10:07.686236+0000","last_became_peered":"2024-09-10T19:10:07.686236+0000","last_unstale":"2024-09-10T19:11:21.815568+0000","last_undegraded":"2024-09-10T19:11:21.815568+0000","last_fullsized":"2024-09-10T19:11:21.815568+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:10:06.664709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:11:30.324556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":36,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-09-10T19:11:21.815224+0000","last_change":"2024-09-10T19:11:19.119050+0000","last_active":"2024-09-10T19:11:21.815224+0000","last_peered":"2024-09-10T19:11:21.815224+0000","last_clean":"2024-09-10T19:11:21.815224+0000","last_became_active":"2024-09-10T19:10:11.699745+0000","last_became_peered":"2024-09-10T19:10:11.699745+0000","last_unstale":"2024-09-10T19:11:21.815224+0000","last_undegraded":"2024-09-10T19:11:21.815224+0000","last_fullsized":"2024-09-10T19:11:21.815224+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:11:19.118987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:11:19.118987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:12:48.847171+0000","scrub_duration":0.0012170379999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":24,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-09-10T19:11:21.300446+0000","last_change":"2024-09-10T19:10:13.720270+0000","last_active":"2024-09-10T19:11:21.300446+0000","last_peered":"2024-09-10T19:11:21.300446+0000","last_clean":"2024-09-10T19:11:21.300446+0000","last_became_active":"2024-09-10T19:10:13.719973+0000","last_became_peered":"2024-09-10T19:10:13.719973+0000","last_unstale":"2024-09-10T19:11:21.300446+0000","last_undegraded":"2024-09-10T19:11:21.300446+0000","last_fullsized":"2024-09-10T19:11:21.300446+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:10:12.693598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:11:24.140279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"66'9","reported_seq":54,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-09-10T19:11:22.870476+0000","last_change":"2024-09-10T19:11:22.870432+0000","last_active":"2024-09-10T19:11:22.870476+0000","last_peered":"2024-09-10T19:11:22.870476+0000","last_clean":"2024-09-10T19:11:22.870476+0000","last_became_active":"2024-09-10T19:10:09.689117+0000","last_became_peered":"2024-09-10T19:10:09.689117+0000","last_unstale":"2024-09-10T19:11:22.870476+0000","last_undegraded":"2024-09-10T19:11:22.870476+0000","last_fullsized":"2024-09-10T19:11:22.870476+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'9","last_scrub_stamp":"2024-09-10T19:11:22.870369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:11:22.870369+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:12:27.485364+0000","scrub_duration":0.0017468729999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":30,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-09-10T19:11:21.189991+0000","last_change":"2024-09-10T19:10:07.683195+0000","last_active":"2024-09-10T19:11:21.189991+0000","last_peered":"2024-09-10T19:11:21.189991+0000","last_clean":"2024-09-10T19:11:21.189991+0000","last_became_active":"2024-09-10T19:10:07.682834+0000","last_became_peered":"2024-09-10T19:10:07.682834+0000","last_unstale":"2024-09-10T19:11:21.189991+0000","last_undegraded":"2024-09-10T19:11:21.189991+0000","last_fullsized":"2024-09-10T19:11:21.189991+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:10:06.664709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:11:26.248552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":26,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-09-10T19:11:21.815169+0000","last_change":"2024-09-10T19:10:11.694697+0000","last_active":"2024-09-10T19:11:21.815169+0000","last_peered":"2024-09-10T19:11:21.815169+0000","last_clean":"2024-09-10T19:11:21.815169+0000","last_became_active":"2024-09-10T19:10:11.694405+0000","last_became_peered":"2024-09-10T19:10:11.694405+0000","last_unstale":"2024-09-10T19:11:21.815169+0000","last_undegraded":"2024-09-10T19:11:21.815169+0000","last_fullsized":"2024-09-10T19:11:21.815169+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:10:10.676876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:11:32.816040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":34,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-09-10T19:11:21.302407+0000","last_change":"2024-09-10T19:11:17.974993+0000","last_active":"2024-09-10T19:11:21.302407+0000","last_peered":"2024-09-10T19:11:21.302407+0000","last_clean":"2024-09-10T19:11:21.302407+0000","last_became_active":"2024-09-10T19:10:13.714952+0000","last_became_peered":"2024-09-10T19:10:13.714952+0000","last_unstale":"2024-09-10T19:11:21.302407+0000","last_undegraded":"2024-09-10T19:11:21.302407+0000","last_fullsized":"2024-09-10T19:11:21.302407+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:11:17.974937+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:11:17.974937+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:12:32.960908+0000","scrub_duration":0.001345834,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.8","version":"66'15","reported_seq":63,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-10T19:11:33.481861+0000","last_change":"2024-09-10T19:11:33.481823+0000","last_active":"2024-09-10T19:11:33.481861+0000","last_peered":"2024-09-10T19:11:33.481861+0000","last_clean":"2024-09-10T19:11:33.481861+0000","last_became_active":"2024-09-10T19:10:09.820950+0000","last_became_peered":"2024-09-10T19:10:09.820950+0000","last_unstale":"2024-09-10T19:11:33.481861+0000","last_undegraded":"2024-09-10T19:11:33.481861+0000","last_fullsized":"2024-09-10T19:11:33.481861+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'15","last_scrub_stamp":"2024-09-10T19:11:33.481743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:11:33.481743+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:12:42.712551+0000","scrub_duration":0.002098094,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":40,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-09-10T19:11:21.815596+0000","last_change":"2024-09-10T19:11:15.197602+0000","last_active":"2024-09-10T19:11:21.815596+0000","last_peered":"2024-09-10T19:11:21.815596+0000","last_clean":"2024-09-10T19:11:21.815596+0000","last_became_active":"2024-09-10T19:10:07.682286+0000","last_became_peered":"2024-09-10T19:10:07.682286+0000","last_unstale":"2024-09-10T19:11:21.815596+0000","last_undegraded":"2024-09-10T19:11:21.815596+0000","last_fullsized":"2024-09-10T19:11:21.815596+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:11:15.197497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:11:15.197497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:12:23.597035+0000","scrub_duration":0.0019717929999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.9","version":"66'8","reported_seq":51,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-10T19:11:25.764708+0000","last_change":"2024-09-10T19:10:11.703391+0000","last_active":"2024-09-10T19:11:25.764708+0000","last_peered":"2024-09-10T19:11:25.764708+0000","last_clean":"2024-09-10T19:11:25.764708+0000","last_became_active":"2024-09-10T19:10:11.702878+0000","last_became_peered":"2024-09-10T19:10:11.702878+0000","last_unstale":"2024-09-10T19:11:25.764708+0000","last_undegraded":"2024-09-10T19:11:25.764708+0000","last_fullsized":"2024-09-10T19:11:25.764708+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:10:10.676876+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":34,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-09-10T19:11:21.186724+0000","last_change":"2024-09-10T19:11:20.184218+0000","last_active":"2024-09-10T19:11:21.186724+0000","last_peered":"2024-09-10T19:11:21.186724+0000","last_clean":"2024-09-10T19:11:21.186724+0000","last_became_active":"2024-09-10T19:10:14.053875+0000","last_became_peered":"2024-09-10T19:10:14.053875+0000","last_unstale":"2024-09-10T19:11:21.186724+0000","last_undegraded":"2024-09-10T19:11:21.186724+0000","last_fullsized":"2024-09-10T19:11:21.186724+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:11:20.184132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:11:20.184132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:12:50.058323+0000","scrub_duration":0.002119522,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"66'12","reported_seq":51,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-09-10T19:11:21.816081+0000","last_change":"2024-09-10T19:10:09.693243+0000","last_active":"2024-09-10T19:11:21.816081+0000","last_peered":"2024-09-10T19:11:21.816081+0000","last_clean":"2024-09-10T19:11:21.816081+0000","last_became_active":"2024-09-10T19:10:09.692627+0000","last_became_peered":"2024-09-10T19:10:09.692627+0000","last_unstale":"2024-09-10T19:11:21.816081+0000","last_undegraded":"2024-09-10T19:11:21.816081+0000","last_fullsized":"2024-09-10T19:11:21.816081+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:10:08.671214+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:11:32.186259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.f","version":"59'2","reported_seq":56,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-10T19:11:25.111519+0000","last_change":"2024-09-10T19:11:25.111486+0000","last_active":"2024-09-10T19:11:25.111519+0000","last_peered":"2024-09-10T19:11:25.111519+0000","last_clean":"2024-09-10T19:11:25.111519+0000","last_became_active":"2024-09-10T19:10:07.681719+0000","last_became_peered":"2024-09-10T19:10:07.681719+0000","last_unstale":"2024-09-10T19:11:25.111519+0000","last_undegraded":"2024-09-10T19:11:25.111519+0000","last_fullsized":"2024-09-10T19:11:25.111519+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'2","last_scrub_stamp":"2024-09-10T19:11:25.111429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:11:25.111429+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:12:31.423912+0000","scrub_duration":0.0014791139999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":14,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":26,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-09-10T19:11:21.815057+0000","last_change":"2024-09-10T19:10:11.698643+0000","last_active":"2024-09-10T19:11:21.815057+0000","last_peered":"2024-09-10T19:11:21.815057+0000","last_clean":"2024-09-10T19:11:21.815057+0000","last_became_active":"2024-09-10T19:10:11.698354+0000","last_became_peered":"2024-09-10T19:10:11.698354+0000","last_unstale":"2024-09-10T19:11:21.815057+0000","last_undegraded":"2024-09-10T19:11:21.815057+0000","last_fullsized":"2024-09-10T19:11:21.815057+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:10:10.676876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:11:36.386854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":24,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-09-10T19:11:21.194939+0000","last_change":"2024-09-10T19:10:13.719272+0000","last_active":"2024-09-10T19:11:21.194939+0000","last_peered":"2024-09-10T19:11:21.194939+0000","last_clean":"2024-09-10T19:11:21.194939+0000","last_became_active":"2024-09-10T19:10:13.718962+0000","last_became_peered":"2024-09-10T19:10:13.718962+0000","last_unstale":"2024-09-10T19:11:21.194939+0000","last_undegraded":"2024-09-10T19:11:21.194939+0000","last_fullsized":"2024-09-10T19:11:21.194939+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:10:12.693598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.16","version":"66'9","reported_seq":54,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-09-10T19:11:21.302864+0000","last_change":"2024-09-10T19:11:11.962017+0000","last_active":"2024-09-10T19:11:21.302864+0000","last_peered":"2024-09-10T19:11:21.302864+0000","last_clean":"2024-09-10T19:11:21.302864+0000","last_became_active":"2024-09-10T19:10:09.822994+0000","last_became_peered":"2024-09-10T19:10:09.822994+0000","last_unstale":"2024-09-10T19:11:21.302864+0000","last_undegraded":"2024-09-10T19:11:21.302864+0000","last_fullsized":"2024-09-10T19:11:21.302864+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'9","last_scrub_stamp":"2024-09-10T19:11:11.961960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:11:11.961960+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:12:16.945811+0000","scrub_duration":0.001903287,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":30,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-09-10T19:11:21.815137+0000","last_change":"2024-09-10T19:10:07.678348+0000","last_active":"2024-09-10T19:11:21.815137+0000","last_peered":"2024-09-10T19:11:21.815137+0000","last_clean":"2024-09-10T19:11:21.815137+0000","last_became_active":"2024-09-10T19:10:07.678076+0000","last_became_peered":"2024-09-10T19:10:07.678076+0000","last_unstale":"2024-09-10T19:11:21.815137+0000","last_undegraded":"2024-09-10T19:11:21.815137+0000","last_fullsized":"2024-09-10T19:11:21.815137+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:10:06.664709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:11:30.763897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":26,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-09-10T19:11:21.195518+0000","last_change":"2024-09-10T19:10:11.693961+0000","last_active":"2024-09-10T19:11:21.195518+0000","last_peered":"2024-09-10T19:11:21.195518+0000","last_clean":"2024-09-10T19:11:21.195518+0000","last_became_active":"2024-09-10T19:10:11.693407+0000","last_became_peered":"2024-09-10T19:10:11.693407+0000","last_unstale":"2024-09-10T19:11:21.195518+0000","last_undegraded":"2024-09-10T19:11:21.195518+0000","last_fullsized":"2024-09-10T19:11:21.195518+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:10:10.676876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:11:32.367716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"66'1","reported_seq":35,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-10T19:11:28.192820+0000","last_change":"2024-09-10T19:11:28.192794+0000","last_active":"2024-09-10T19:11:28.192820+0000","last_peered":"2024-09-10T19:11:28.192820+0000","last_clean":"2024-09-10T19:11:28.192820+0000","last_became_active":"2024-09-10T19:10:13.717001+0000","last_became_peered":"2024-09-10T19:10:13.717001+0000","last_unstale":"2024-09-10T19:11:28.192820+0000","last_undegraded":"2024-09-10T19:11:28.192820+0000","last_fullsized":"2024-09-10T19:11:28.192820+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'1","last_scrub_stamp":"2024-09-10T19:11:28.192745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:11:28.192745+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:12:37.727108+0000","scrub_duration":0.001276323,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":40,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-10T19:11:32.859211+0000","last_change":"2024-09-10T19:11:32.859175+0000","last_active":"2024-09-10T19:11:32.859211+0000","last_peered":"2024-09-10T19:11:32.859211+0000","last_clean":"2024-09-10T19:11:32.859211+0000","last_became_active":"2024-09-10T19:10:07.682963+0000","last_became_peered":"2024-09-10T19:10:07.682963+0000","last_unstale":"2024-09-10T19:11:32.859211+0000","last_undegraded":"2024-09-10T19:11:32.859211+0000","last_fullsized":"2024-09-10T19:11:32.859211+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:11:32.859106+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:11:32.859106+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:12:47.279562+0000","scrub_duration":0.0014403980000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"66'6","reported_seq":37,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-09-10T19:11:21.195577+0000","last_change":"2024-09-10T19:10:09.822283+0000","last_active":"2024-09-10T19:11:21.195577+0000","last_peered":"2024-09-10T19:11:21.195577+0000","last_clean":"2024-09-10T19:11:21.195577+0000","last_became_active":"2024-09-10T19:10:09.821906+0000","last_became_peered":"2024-09-10T19:10:09.821906+0000","last_unstale":"2024-09-10T19:11:21.195577+0000","last_undegraded":"2024-09-10T19:11:21.195577+0000","last_fullsized":"2024-09-10T19:11:21.195577+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:10:08.671214+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:11:35.174496+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":36,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-10T19:11:23.264917+0000","last_change":"2024-09-10T19:11:23.264870+0000","last_active":"2024-09-10T19:11:23.264917+0000","last_peered":"2024-09-10T19:11:23.264917+0000","last_clean":"2024-09-10T19:11:23.264917+0000","last_became_active":"2024-09-10T19:10:11.692943+0000","last_became_peered":"2024-09-10T19:10:11.692943+0000","last_unstale":"2024-09-10T19:11:23.264917+0000","last_undegraded":"2024-09-10T19:11:23.264917+0000","last_fullsized":"2024-09-10T19:11:23.264917+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:11:23.264793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:11:23.264793+0000","last_clean_scrub_stamp":"2024-09-10T19:11:23.264793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:12:44.248872+0000","scrub_duration":0.0015950649999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":34,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-09-10T19:11:21.300839+0000","last_change":"2024-09-10T19:11:13.342070+0000","last_active":"2024-09-10T19:11:21.300839+0000","last_peered":"2024-09-10T19:11:21.300839+0000","last_clean":"2024-09-10T19:11:21.300839+0000","last_became_active":"2024-09-10T19:10:14.054360+0000","last_became_peered":"2024-09-10T19:10:14.054360+0000","last_unstale":"2024-09-10T19:11:21.300839+0000","last_undegraded":"2024-09-10T19:11:21.300839+0000","last_fullsized":"2024-09-10T19:11:21.300839+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:11:13.341970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:11:13.341970+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:12:30.833886+0000","scrub_duration":0.001537725,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"66'10","reported_seq":43,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-09-10T19:11:21.195212+0000","last_change":"2024-09-10T19:10:09.822478+0000","last_active":"2024-09-10T19:11:21.195212+0000","last_peered":"2024-09-10T19:11:21.195212+0000","last_clean":"2024-09-10T19:11:21.195212+0000","last_became_active":"2024-09-10T19:10:09.821810+0000","last_became_peered":"2024-09-10T19:10:09.821810+0000","last_unstale":"2024-09-10T19:11:21.195212+0000","last_undegraded":"2024-09-10T19:11:21.195212+0000","last_fullsized":"2024-09-10T19:11:21.195212+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:10:08.671214+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":40,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-09-10T19:11:21.187053+0000","last_change":"2024-09-10T19:11:12.081772+0000","last_active":"2024-09-10T19:11:21.187053+0000","last_peered":"2024-09-10T19:11:21.187053+0000","last_clean":"2024-09-10T19:11:21.187053+0000","last_became_active":"2024-09-10T19:10:07.685870+0000","last_became_peered":"2024-09-10T19:10:07.685870+0000","last_unstale":"2024-09-10T19:11:21.187053+0000","last_undegraded":"2024-09-10T19:11:21.187053+0000","last_fullsized":"2024-09-10T19:11:21.187053+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:11:12.081691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:11:12.081691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:12:41.469111+0000","scrub_duration":0.001629903,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"66'8","reported_seq":51,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-10T19:11:25.764871+0000","last_change":"2024-09-10T19:10:11.700895+0000","last_active":"2024-09-10T19:11:25.764871+0000","last_peered":"2024-09-10T19:11:25.764871+0000","last_clean":"2024-09-10T19:11:25.764871+0000","last_became_active":"2024-09-10T19:10:11.700584+0000","last_became_peered":"2024-09-10T19:10:11.700584+0000","last_unstale":"2024-09-10T19:11:25.764871+0000","last_undegraded":"2024-09-10T19:11:25.764871+0000","last_fullsized":"2024-09-10T19:11:25.764871+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:10:10.676876+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:11:34.335843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":34,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-09-10T19:11:21.302650+0000","last_change":"2024-09-10T19:11:20.881495+0000","last_active":"2024-09-10T19:11:21.302650+0000","last_peered":"2024-09-10T19:11:21.302650+0000","last_clean":"2024-09-10T19:11:21.302650+0000","last_became_active":"2024-09-10T19:10:13.721035+0000","last_became_peered":"2024-09-10T19:10:13.721035+0000","last_unstale":"2024-09-10T19:11:21.302650+0000","last_undegraded":"2024-09-10T19:11:21.302650+0000","last_fullsized":"2024-09-10T19:11:21.302650+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:11:20.881436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:11:20.881436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:12:49.668407+0000","scrub_duration":0.0014551099999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"66'9","reported_seq":44,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-09-10T19:11:21.187595+0000","last_change":"2024-09-10T19:10:09.823289+0000","last_active":"2024-09-10T19:11:21.187595+0000","last_peered":"2024-09-10T19:11:21.187595+0000","last_clean":"2024-09-10T19:11:21.187595+0000","last_became_active":"2024-09-10T19:10:09.823006+0000","last_became_peered":"2024-09-10T19:10:09.823006+0000","last_unstale":"2024-09-10T19:11:21.187595+0000","last_undegraded":"2024-09-10T19:11:21.187595+0000","last_fullsized":"2024-09-10T19:11:21.187595+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:10:08.671214+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:11:26.960895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":40,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-10T19:11:29.796508+0000","last_change":"2024-09-10T19:11:29.796484+0000","last_active":"2024-09-10T19:11:29.796508+0000","last_peered":"2024-09-10T19:11:29.796508+0000","last_clean":"2024-09-10T19:11:29.796508+0000","last_became_active":"2024-09-10T19:10:07.688499+0000","last_became_peered":"2024-09-10T19:10:07.688499+0000","last_unstale":"2024-09-10T19:11:29.796508+0000","last_undegraded":"2024-09-10T19:11:29.796508+0000","last_fullsized":"2024-09-10T19:11:29.796508+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:11:29.796429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:11:29.796429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:12:53.847123+0000","scrub_duration":0.0012420479999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"66'8","reported_seq":61,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-10T19:11:25.764903+0000","last_change":"2024-09-10T19:11:23.870113+0000","last_active":"2024-09-10T19:11:25.764903+0000","last_peered":"2024-09-10T19:11:25.764903+0000","last_clean":"2024-09-10T19:11:25.764903+0000","last_became_active":"2024-09-10T19:10:11.692359+0000","last_became_peered":"2024-09-10T19:10:11.692359+0000","last_unstale":"2024-09-10T19:11:25.764903+0000","last_undegraded":"2024-09-10T19:11:25.764903+0000","last_fullsized":"2024-09-10T19:11:25.764903+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'8","last_scrub_stamp":"2024-09-10T19:11:23.870010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:11:23.870010+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:12:47.854105+0000","scrub_duration":0.0017621309999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":24,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-09-10T19:11:21.817375+0000","last_change":"2024-09-10T19:10:13.718617+0000","last_active":"2024-09-10T19:11:21.817375+0000","last_peered":"2024-09-10T19:11:21.817375+0000","last_clean":"2024-09-10T19:11:21.817375+0000","last_became_active":"2024-09-10T19:10:13.718285+0000","last_became_peered":"2024-09-10T19:10:13.718285+0000","last_unstale":"2024-09-10T19:11:21.817375+0000","last_undegraded":"2024-09-10T19:11:21.817375+0000","last_fullsized":"2024-09-10T19:11:21.817375+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:10:12.693598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:11:30.971974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.12","version":"79'13","reported_seq":58,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-09-10T19:11:22.072557+0000","last_change":"2024-09-10T19:11:22.072464+0000","last_active":"2024-09-10T19:11:22.072557+0000","last_peered":"2024-09-10T19:11:22.072557+0000","last_clean":"2024-09-10T19:11:22.072557+0000","last_became_active":"2024-09-10T19:10:09.689440+0000","last_became_peered":"2024-09-10T19:10:09.689440+0000","last_unstale":"2024-09-10T19:11:22.072557+0000","last_undegraded":"2024-09-10T19:11:22.072557+0000","last_fullsized":"2024-09-10T19:11:22.072557+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'13","last_scrub_stamp":"2024-09-10T19:11:22.072400+0000","last_deep_scrub":"79'13","last_deep_scrub_stamp":"2024-09-10T19:11:22.072400+0000","last_clean_scrub_stamp":"2024-09-10T19:11:22.072400+0000","objects_scrubbed":4,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:12:33.209191+0000","scrub_duration":0.0032983940000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"59'1","reported_seq":37,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-09-10T19:11:21.816042+0000","last_change":"2024-09-10T19:10:07.683567+0000","last_active":"2024-09-10T19:11:21.816042+0000","last_peered":"2024-09-10T19:11:21.816042+0000","last_clean":"2024-09-10T19:11:21.816042+0000","last_became_active":"2024-09-10T19:10:07.683263+0000","last_became_peered":"2024-09-10T19:10:07.683263+0000","last_unstale":"2024-09-10T19:11:21.816042+0000","last_undegraded":"2024-09-10T19:11:21.816042+0000","last_fullsized":"2024-09-10T19:11:21.816042+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:10:06.664709+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:11:35.421853+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":362,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":26,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-09-10T19:11:21.194980+0000","last_change":"2024-09-10T19:10:11.701754+0000","last_active":"2024-09-10T19:11:21.194980+0000","last_peered":"2024-09-10T19:11:21.194980+0000","last_clean":"2024-09-10T19:11:21.194980+0000","last_became_active":"2024-09-10T19:10:11.701416+0000","last_became_peered":"2024-09-10T19:10:11.701416+0000","last_unstale":"2024-09-10T19:11:21.194980+0000","last_undegraded":"2024-09-10T19:11:21.194980+0000","last_fullsized":"2024-09-10T19:11:21.194980+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:10:10.676876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:11:30.081463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":24,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-09-10T19:11:21.302167+0000","last_change":"2024-09-10T19:10:13.716464+0000","last_active":"2024-09-10T19:11:21.302167+0000","last_peered":"2024-09-10T19:11:21.302167+0000","last_clean":"2024-09-10T19:11:21.302167+0000","last_became_active":"2024-09-10T19:10:13.716198+0000","last_became_peered":"2024-09-10T19:10:13.716198+0000","last_unstale":"2024-09-10T19:11:21.302167+0000","last_undegraded":"2024-09-10T19:11:21.302167+0000","last_fullsized":"2024-09-10T19:11:21.302167+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:10:12.693598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:11:29.980159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"66'11","reported_seq":47,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-09-10T19:11:21.815272+0000","last_change":"2024-09-10T19:10:09.694117+0000","last_active":"2024-09-10T19:11:21.815272+0000","last_peered":"2024-09-10T19:11:21.815272+0000","last_clean":"2024-09-10T19:11:21.815272+0000","last_became_active":"2024-09-10T19:10:09.693856+0000","last_became_peered":"2024-09-10T19:10:09.693856+0000","last_unstale":"2024-09-10T19:11:21.815272+0000","last_undegraded":"2024-09-10T19:11:21.815272+0000","last_fullsized":"2024-09-10T19:11:21.815272+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:10:08.671214+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:11:32.166737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":40,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-09-10T19:11:21.189948+0000","last_change":"2024-09-10T19:11:10.031299+0000","last_active":"2024-09-10T19:11:21.189948+0000","last_peered":"2024-09-10T19:11:21.189948+0000","last_clean":"2024-09-10T19:11:21.189948+0000","last_became_active":"2024-09-10T19:10:07.685339+0000","last_became_peered":"2024-09-10T19:10:07.685339+0000","last_unstale":"2024-09-10T19:11:21.189948+0000","last_undegraded":"2024-09-10T19:11:21.189948+0000","last_fullsized":"2024-09-10T19:11:21.189948+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:11:10.031245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:11:10.031245+0000","last_clean_scrub_stamp":"2024-09-10T19:11:10.031245+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:12:27.675011+0000","scrub_duration":0.0014113439999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":36,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-10T19:11:31.249426+0000","last_change":"2024-09-10T19:11:31.249400+0000","last_active":"2024-09-10T19:11:31.249426+0000","last_peered":"2024-09-10T19:11:31.249426+0000","last_clean":"2024-09-10T19:11:31.249426+0000","last_became_active":"2024-09-10T19:10:11.694564+0000","last_became_peered":"2024-09-10T19:10:11.694564+0000","last_unstale":"2024-09-10T19:11:31.249426+0000","last_undegraded":"2024-09-10T19:11:31.249426+0000","last_fullsized":"2024-09-10T19:11:31.249426+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:11:31.249343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:11:31.249343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:12:58.372067+0000","scrub_duration":0.001093644,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":34,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-09-10T19:11:21.194595+0000","last_change":"2024-09-10T19:11:16.813629+0000","last_active":"2024-09-10T19:11:21.194595+0000","last_peered":"2024-09-10T19:11:21.194595+0000","last_clean":"2024-09-10T19:11:21.194595+0000","last_became_active":"2024-09-10T19:10:13.724201+0000","last_became_peered":"2024-09-10T19:10:13.724201+0000","last_unstale":"2024-09-10T19:11:21.194595+0000","last_undegraded":"2024-09-10T19:11:21.194595+0000","last_fullsized":"2024-09-10T19:11:21.194595+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:11:16.813575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:11:16.813575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:12:44.427906+0000","scrub_duration":0.001237312,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"66'4","reported_seq":44,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-09-10T19:11:21.194577+0000","last_change":"2024-09-10T19:11:14.816693+0000","last_active":"2024-09-10T19:11:21.194577+0000","last_peered":"2024-09-10T19:11:21.194577+0000","last_clean":"2024-09-10T19:11:21.194577+0000","last_became_active":"2024-09-10T19:10:09.692751+0000","last_became_peered":"2024-09-10T19:10:09.692751+0000","last_unstale":"2024-09-10T19:11:21.194577+0000","last_undegraded":"2024-09-10T19:11:21.194577+0000","last_fullsized":"2024-09-10T19:11:21.194577+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'4","last_scrub_stamp":"2024-09-10T19:11:14.816632+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:11:14.816632+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:12:42.695105+0000","scrub_duration":0.001902504,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":40,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-10T19:11:24.275089+0000","last_change":"2024-09-10T19:11:24.275061+0000","last_active":"2024-09-10T19:11:24.275089+0000","last_peered":"2024-09-10T19:11:24.275089+0000","last_clean":"2024-09-10T19:11:24.275089+0000","last_became_active":"2024-09-10T19:10:07.687723+0000","last_became_peered":"2024-09-10T19:10:07.687723+0000","last_unstale":"2024-09-10T19:11:24.275089+0000","last_undegraded":"2024-09-10T19:11:24.275089+0000","last_fullsized":"2024-09-10T19:11:24.275089+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:11:24.275004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:11:24.275004+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:12:41.723440+0000","scrub_duration":0.001588014,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":36,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-10T19:11:31.838035+0000","last_change":"2024-09-10T19:11:31.838005+0000","last_active":"2024-09-10T19:11:31.838035+0000","last_peered":"2024-09-10T19:11:31.838035+0000","last_clean":"2024-09-10T19:11:31.838035+0000","last_became_active":"2024-09-10T19:10:11.698536+0000","last_became_peered":"2024-09-10T19:10:11.698536+0000","last_unstale":"2024-09-10T19:11:31.838035+0000","last_undegraded":"2024-09-10T19:11:31.838035+0000","last_fullsized":"2024-09-10T19:11:31.838035+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:11:31.837944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:11:31.837944+0000","last_clean_scrub_stamp":"2024-09-10T19:11:31.837944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:12:53.987457+0000","scrub_duration":0.001263639,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":34,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-09-10T19:11:21.300729+0000","last_change":"2024-09-10T19:11:14.298616+0000","last_active":"2024-09-10T19:11:21.300729+0000","last_peered":"2024-09-10T19:11:21.300729+0000","last_clean":"2024-09-10T19:11:21.300729+0000","last_became_active":"2024-09-10T19:10:13.718271+0000","last_became_peered":"2024-09-10T19:10:13.718271+0000","last_unstale":"2024-09-10T19:11:21.300729+0000","last_undegraded":"2024-09-10T19:11:21.300729+0000","last_fullsized":"2024-09-10T19:11:21.300729+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:11:14.298545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:11:14.298545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:12:15.177239+0000","scrub_duration":0.001561799,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.11","version":"66'11","reported_seq":47,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-09-10T19:11:21.194692+0000","last_change":"2024-09-10T19:10:09.821652+0000","last_active":"2024-09-10T19:11:21.194692+0000","last_peered":"2024-09-10T19:11:21.194692+0000","last_clean":"2024-09-10T19:11:21.194692+0000","last_became_active":"2024-09-10T19:10:09.821357+0000","last_became_peered":"2024-09-10T19:10:09.821357+0000","last_unstale":"2024-09-10T19:11:21.194692+0000","last_undegraded":"2024-09-10T19:11:21.194692+0000","last_fullsized":"2024-09-10T19:11:21.194692+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:10:08.671214+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":30,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-09-10T19:11:21.815770+0000","last_change":"2024-09-10T19:10:07.685503+0000","last_active":"2024-09-10T19:11:21.815770+0000","last_peered":"2024-09-10T19:11:21.815770+0000","last_clean":"2024-09-10T19:11:21.815770+0000","last_became_active":"2024-09-10T19:10:07.685223+0000","last_became_peered":"2024-09-10T19:10:07.685223+0000","last_unstale":"2024-09-10T19:11:21.815770+0000","last_undegraded":"2024-09-10T19:11:21.815770+0000","last_fullsized":"2024-09-10T19:11:21.815770+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:10:06.664709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:11:30.851917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":26,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-09-10T19:11:21.300206+0000","last_change":"2024-09-10T19:10:11.699248+0000","last_active":"2024-09-10T19:11:21.300206+0000","last_peered":"2024-09-10T19:11:21.300206+0000","last_clean":"2024-09-10T19:11:21.300206+0000","last_became_active":"2024-09-10T19:10:11.699013+0000","last_became_peered":"2024-09-10T19:10:11.699013+0000","last_unstale":"2024-09-10T19:11:21.300206+0000","last_undegraded":"2024-09-10T19:11:21.300206+0000","last_fullsized":"2024-09-10T19:11:21.300206+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:10:10.676876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:11:37.771138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":24,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-09-10T19:11:21.194830+0000","last_change":"2024-09-10T19:10:14.056524+0000","last_active":"2024-09-10T19:11:21.194830+0000","last_peered":"2024-09-10T19:11:21.194830+0000","last_clean":"2024-09-10T19:11:21.194830+0000","last_became_active":"2024-09-10T19:10:14.056172+0000","last_became_peered":"2024-09-10T19:10:14.056172+0000","last_unstale":"2024-09-10T19:11:21.194830+0000","last_undegraded":"2024-09-10T19:11:21.194830+0000","last_fullsized":"2024-09-10T19:11:21.194830+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:10:12.693598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:11:24.815438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"80'2","reported_seq":36,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-10T19:11:30.272124+0000","last_change":"2024-09-10T19:11:30.272094+0000","last_active":"2024-09-10T19:11:30.272124+0000","last_peered":"2024-09-10T19:11:30.272124+0000","last_clean":"2024-09-10T19:11:30.272124+0000","last_became_active":"2024-09-10T19:10:13.718875+0000","last_became_peered":"2024-09-10T19:10:13.718875+0000","last_unstale":"2024-09-10T19:11:30.272124+0000","last_undegraded":"2024-09-10T19:11:30.272124+0000","last_fullsized":"2024-09-10T19:11:30.272124+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"80'2","last_scrub_stamp":"2024-09-10T19:11:30.272043+0000","last_deep_scrub":"80'2","last_deep_scrub_stamp":"2024-09-10T19:11:30.272043+0000","last_clean_scrub_stamp":"2024-09-10T19:11:30.272043+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:12:37.490688+0000","scrub_duration":0.0022929460000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":369,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":30,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-09-10T19:11:21.186687+0000","last_change":"2024-09-10T19:10:07.690146+0000","last_active":"2024-09-10T19:11:21.186687+0000","last_peered":"2024-09-10T19:11:21.186687+0000","last_clean":"2024-09-10T19:11:21.186687+0000","last_became_active":"2024-09-10T19:10:07.689877+0000","last_became_peered":"2024-09-10T19:10:07.689877+0000","last_unstale":"2024-09-10T19:11:21.186687+0000","last_undegraded":"2024-09-10T19:11:21.186687+0000","last_fullsized":"2024-09-10T19:11:21.186687+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:10:06.664709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:11:33.455191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1e","version":"80'15","reported_seq":58,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-10T19:11:25.884049+0000","last_change":"2024-09-10T19:11:25.884008+0000","last_active":"2024-09-10T19:11:25.884049+0000","last_peered":"2024-09-10T19:11:25.884049+0000","last_clean":"2024-09-10T19:11:25.884049+0000","last_became_active":"2024-09-10T19:10:09.823061+0000","last_became_peered":"2024-09-10T19:10:09.823061+0000","last_unstale":"2024-09-10T19:11:25.884049+0000","last_undegraded":"2024-09-10T19:11:25.884049+0000","last_fullsized":"2024-09-10T19:11:25.884049+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"80'15","last_scrub_stamp":"2024-09-10T19:11:25.883919+0000","last_deep_scrub":"80'15","last_deep_scrub_stamp":"2024-09-10T19:11:25.883919+0000","last_clean_scrub_stamp":"2024-09-10T19:11:25.883919+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:12:31.561992+0000","scrub_duration":0.0034715229999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1f","version":"66'8","reported_seq":64,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-10T19:11:30.835462+0000","last_change":"2024-09-10T19:11:30.835414+0000","last_active":"2024-09-10T19:11:30.835462+0000","last_peered":"2024-09-10T19:11:30.835462+0000","last_clean":"2024-09-10T19:11:30.835462+0000","last_became_active":"2024-09-10T19:10:11.695427+0000","last_became_peered":"2024-09-10T19:10:11.695427+0000","last_unstale":"2024-09-10T19:11:30.835462+0000","last_undegraded":"2024-09-10T19:11:30.835462+0000","last_fullsized":"2024-09-10T19:11:30.835462+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'8","last_scrub_stamp":"2024-09-10T19:11:30.835354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:11:30.835354+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:12:42.690611+0000","scrub_duration":0.0013545090000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":34,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-10T19:11:30.210683+0000","last_change":"2024-09-10T19:11:30.210657+0000","last_active":"2024-09-10T19:11:30.210683+0000","last_peered":"2024-09-10T19:11:30.210683+0000","last_clean":"2024-09-10T19:11:30.210683+0000","last_became_active":"2024-09-10T19:10:13.715419+0000","last_became_peered":"2024-09-10T19:10:13.715419+0000","last_unstale":"2024-09-10T19:11:30.210683+0000","last_undegraded":"2024-09-10T19:11:30.210683+0000","last_fullsized":"2024-09-10T19:11:30.210683+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:11:30.210603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:11:30.210603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:12:40.846927+0000","scrub_duration":0.0012894740000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":30,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-09-10T19:11:21.195149+0000","last_change":"2024-09-10T19:10:07.684238+0000","last_active":"2024-09-10T19:11:21.195149+0000","last_peered":"2024-09-10T19:11:21.195149+0000","last_clean":"2024-09-10T19:11:21.195149+0000","last_became_active":"2024-09-10T19:10:07.682689+0000","last_became_peered":"2024-09-10T19:10:07.682689+0000","last_unstale":"2024-09-10T19:11:21.195149+0000","last_undegraded":"2024-09-10T19:11:21.195149+0000","last_fullsized":"2024-09-10T19:11:21.195149+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:10:06.664709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"66'11","reported_seq":47,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-09-10T19:11:21.815432+0000","last_change":"2024-09-10T19:10:09.688145+0000","last_active":"2024-09-10T19:11:21.815432+0000","last_peered":"2024-09-10T19:11:21.815432+0000","last_clean":"2024-09-10T19:11:21.815432+0000","last_became_active":"2024-09-10T19:10:09.687851+0000","last_became_peered":"2024-09-10T19:10:09.687851+0000","last_unstale":"2024-09-10T19:11:21.815432+0000","last_undegraded":"2024-09-10T19:11:21.815432+0000","last_fullsized":"2024-09-10T19:11:21.815432+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:10:08.671214+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:11:33.211646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":36,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-10T19:11:28.822429+0000","last_change":"2024-09-10T19:11:28.822385+0000","last_active":"2024-09-10T19:11:28.822429+0000","last_peered":"2024-09-10T19:11:28.822429+0000","last_clean":"2024-09-10T19:11:28.822429+0000","last_became_active":"2024-09-10T19:10:11.701735+0000","last_became_peered":"2024-09-10T19:10:11.701735+0000","last_unstale":"2024-09-10T19:11:28.822429+0000","last_undegraded":"2024-09-10T19:11:28.822429+0000","last_fullsized":"2024-09-10T19:11:28.822429+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:11:28.822327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:11:28.822327+0000","last_clean_scrub_stamp":"2024-09-10T19:11:28.822327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:12:45.908873+0000","scrub_duration":0.0012591119999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":24,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-09-10T19:11:21.817462+0000","last_change":"2024-09-10T19:10:14.053927+0000","last_active":"2024-09-10T19:11:21.817462+0000","last_peered":"2024-09-10T19:11:21.817462+0000","last_clean":"2024-09-10T19:11:21.817462+0000","last_became_active":"2024-09-10T19:10:14.053619+0000","last_became_peered":"2024-09-10T19:10:14.053619+0000","last_unstale":"2024-09-10T19:11:21.817462+0000","last_undegraded":"2024-09-10T19:11:21.817462+0000","last_fullsized":"2024-09-10T19:11:21.817462+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:10:12.693598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:11:29.510159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":30,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-09-10T19:11:21.816060+0000","last_change":"2024-09-10T19:10:07.686443+0000","last_active":"2024-09-10T19:11:21.816060+0000","last_peered":"2024-09-10T19:11:21.816060+0000","last_clean":"2024-09-10T19:11:21.816060+0000","last_became_active":"2024-09-10T19:10:07.685997+0000","last_became_peered":"2024-09-10T19:10:07.685997+0000","last_unstale":"2024-09-10T19:11:21.816060+0000","last_undegraded":"2024-09-10T19:11:21.816060+0000","last_fullsized":"2024-09-10T19:11:21.816060+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:10:06.664709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:11:36.019712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1c","version":"78'22","reported_seq":70,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-09-10T19:11:21.814940+0000","last_change":"2024-09-10T19:11:21.119003+0000","last_active":"2024-09-10T19:11:21.814940+0000","last_peered":"2024-09-10T19:11:21.814940+0000","last_clean":"2024-09-10T19:11:21.814940+0000","last_became_active":"2024-09-10T19:10:09.690328+0000","last_became_peered":"2024-09-10T19:10:09.690328+0000","last_unstale":"2024-09-10T19:11:21.814940+0000","last_undegraded":"2024-09-10T19:11:21.814940+0000","last_fullsized":"2024-09-10T19:11:21.814940+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"78'22","last_scrub_stamp":"2024-09-10T19:11:21.118937+0000","last_deep_scrub":"78'22","last_deep_scrub_stamp":"2024-09-10T19:11:21.118937+0000","last_clean_scrub_stamp":"2024-09-10T19:11:21.118937+0000","objects_scrubbed":7,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:12:21.535383+0000","scrub_duration":0.0048819759999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":26,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-09-10T19:11:21.189835+0000","last_change":"2024-09-10T19:10:11.700300+0000","last_active":"2024-09-10T19:11:21.189835+0000","last_peered":"2024-09-10T19:11:21.189835+0000","last_clean":"2024-09-10T19:11:21.189835+0000","last_became_active":"2024-09-10T19:10:11.700037+0000","last_became_peered":"2024-09-10T19:10:11.700037+0000","last_unstale":"2024-09-10T19:11:21.189835+0000","last_undegraded":"2024-09-10T19:11:21.189835+0000","last_fullsized":"2024-09-10T19:11:21.189835+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:10:10.676876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:11:25.509401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":34,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-10T19:11:24.828009+0000","last_change":"2024-09-10T19:11:24.827983+0000","last_active":"2024-09-10T19:11:24.828009+0000","last_peered":"2024-09-10T19:11:24.828009+0000","last_clean":"2024-09-10T19:11:24.828009+0000","last_became_active":"2024-09-10T19:10:14.053698+0000","last_became_peered":"2024-09-10T19:10:14.053698+0000","last_unstale":"2024-09-10T19:11:24.828009+0000","last_undegraded":"2024-09-10T19:11:24.828009+0000","last_fullsized":"2024-09-10T19:11:24.828009+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:11:24.827916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:11:24.827916+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:12:36.645047+0000","scrub_duration":0.0014104860000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":40,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-10T19:11:25.870618+0000","last_change":"2024-09-10T19:11:25.870573+0000","last_active":"2024-09-10T19:11:25.870618+0000","last_peered":"2024-09-10T19:11:25.870618+0000","last_clean":"2024-09-10T19:11:25.870618+0000","last_became_active":"2024-09-10T19:10:07.682811+0000","last_became_peered":"2024-09-10T19:10:07.682811+0000","last_unstale":"2024-09-10T19:11:25.870618+0000","last_undegraded":"2024-09-10T19:11:25.870618+0000","last_fullsized":"2024-09-10T19:11:25.870618+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:11:25.870516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:11:25.870516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:12:33.650020+0000","scrub_duration":0.001292568,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1d","version":"66'12","reported_seq":51,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-09-10T19:11:21.195335+0000","last_change":"2024-09-10T19:10:09.693108+0000","last_active":"2024-09-10T19:11:21.195335+0000","last_peered":"2024-09-10T19:11:21.195335+0000","last_clean":"2024-09-10T19:11:21.195335+0000","last_became_active":"2024-09-10T19:10:09.692781+0000","last_became_peered":"2024-09-10T19:10:09.692781+0000","last_unstale":"2024-09-10T19:11:21.195335+0000","last_undegraded":"2024-09-10T19:11:21.195335+0000","last_fullsized":"2024-09-10T19:11:21.195335+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:10:08.671214+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:11:25.822393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":26,"reported_epoch":79,"state":"active+clean","last_fresh":"2024-09-10T19:11:21.817165+0000","last_change":"2024-09-10T19:10:11.701419+0000","last_active":"2024-09-10T19:11:21.817165+0000","last_peered":"2024-09-10T19:11:21.817165+0000","last_clean":"2024-09-10T19:11:21.817165+0000","last_became_active":"2024-09-10T19:10:11.701155+0000","last_became_peered":"2024-09-10T19:10:11.701155+0000","last_unstale":"2024-09-10T19:11:21.817165+0000","last_undegraded":"2024-09-10T19:11:21.817165+0000","last_fullsized":"2024-09-10T19:11:21.817165+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:10:10.676876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:11:33.357988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":65,"ondisk_log_size":65,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":177,"num_object_clones":0,"num_object_copies":531,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":177,"num_whiteouts":0,"num_read":698,"num_read_kb":455,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":407,"ondisk_log_size":407,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":81,"num_read_kb":76,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1328,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":27,"num_read_kb":27,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":3984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4,"ondisk_log_size":4,"up":96,"acting":96,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79,"ondisk_log_size":79,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":54,"seq":231928234006,"num_pgs":57,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298804,"kb_used_data":1452,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93446348,"statfs":{"total":95995035648,"available":95689060352,"internally_reserved":0,"allocated":1486848,"data_stored":988678,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 19:10:56 2024","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.53500000000000003},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.70499999999999996,"5min":0.70499999999999996,"15min":0.70499999999999996},"last":0.51600000000000001},{"interface":"front","average":{"1min":0.53800000000000003,"5min":0.53800000000000003,"15min":0.53800000000000003},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.76200000000000001,"5min":0.76200000000000001,"15min":0.76200000000000001},"last":0.44600000000000001}]},{"osd":1,"last update":"Tue Sep 10 19:10:56 2024","interfaces":[{"interface":"back","average":{"1min":0.52400000000000002,"5min":0.52400000000000002,"15min":0.52400000000000002},"min":{"1min":0.41299999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":0.79400000000000004,"5min":0.79400000000000004,"15min":0.79400000000000004},"last":0.57399999999999995},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"min":{"1min":0.314,"5min":0.314,"15min":0.314},"max":{"1min":0.71799999999999997,"5min":0.71799999999999997,"15min":0.71799999999999997},"last":0.39100000000000001}]},{"osd":2,"last update":"Tue Sep 10 19:10:56 2024","interfaces":[{"interface":"back","average":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"min":{"1min":0.29199999999999998,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.84399999999999997,"5min":0.84399999999999997,"15min":0.84399999999999997},"last":0.48799999999999999},{"interface":"front","average":{"1min":0.55000000000000004,"5min":0.55000000000000004,"15min":0.55000000000000004},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.77300000000000002,"5min":0.77300000000000002,"15min":0.77300000000000002},"last":0.46200000000000002}]},{"osd":3,"last update":"Tue Sep 10 19:10:56 2024","interfaces":[{"interface":"back","average":{"1min":0.53600000000000003,"5min":0.53600000000000003,"15min":0.53600000000000003},"min":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":0.86399999999999999,"5min":0.86399999999999999,"15min":0.86399999999999999},"last":0.42799999999999999},{"interface":"front","average":{"1min":0.52000000000000002,"5min":0.52000000000000002,"15min":0.52000000000000002},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.77600000000000002,"5min":0.77600000000000002,"15min":0.77600000000000002},"last":0.60499999999999998}]},{"osd":4,"last update":"Tue Sep 10 19:10:56 2024","interfaces":[{"interface":"back","average":{"1min":0.45500000000000002,"5min":0.45500000000000002,"15min":0.45500000000000002},"min":{"1min":0.318,"5min":0.318,"15min":0.318},"max":{"1min":0.65000000000000002,"5min":0.65000000000000002,"15min":0.65000000000000002},"last":0.50600000000000001},{"interface":"front","average":{"1min":0.46000000000000002,"5min":0.46000000000000002,"15min":0.46000000000000002},"min":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.60599999999999998,"5min":0.60599999999999998,"15min":0.60599999999999998},"last":0.58699999999999997}]},{"osd":5,"last update":"Tue Sep 10 19:10:56 2024","interfaces":[{"interface":"back","average":{"1min":0.47999999999999998,"5min":0.47999999999999998,"15min":0.47999999999999998},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.76100000000000001,"5min":0.76100000000000001,"15min":0.76100000000000001},"last":0.626},{"interface":"front","average":{"1min":0.47599999999999998,"5min":0.47599999999999998,"15min":0.47599999999999998},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.68600000000000005,"5min":0.68600000000000005,"15min":0.68600000000000005},"last":0.47599999999999998}]},{"osd":6,"last update":"Tue Sep 10 19:10:56 2024","interfaces":[{"interface":"back","average":{"1min":0.5,"5min":0.5,"15min":0.5},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":0.65900000000000003,"5min":0.65900000000000003,"15min":0.65900000000000003},"last":0.54300000000000004},{"interface":"front","average":{"1min":0.48899999999999999,"5min":0.48899999999999999,"15min":0.48899999999999999},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.67000000000000004,"5min":0.67000000000000004,"15min":0.67000000000000004},"last":0.55600000000000005}]}]},{"osd":6,"up_from":47,"seq":201863462939,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298792,"kb_used_data":1440,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93446360,"statfs":{"total":95995035648,"available":95689072640,"internally_reserved":0,"allocated":1474560,"data_stored":988804,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 19:10:35 2024","interfaces":[{"interface":"back","average":{"1min":0.52200000000000002,"5min":0.52200000000000002,"15min":0.52200000000000002},"min":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.621,"5min":0.621,"15min":0.621},"last":0.42599999999999999},{"interface":"front","average":{"1min":0.54400000000000004,"5min":0.54400000000000004,"15min":0.54400000000000004},"min":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":0.67200000000000004,"5min":0.67200000000000004,"15min":0.67200000000000004},"last":0.58999999999999997}]},{"osd":1,"last update":"Tue Sep 10 19:10:35 2024","interfaces":[{"interface":"back","average":{"1min":0.52900000000000003,"5min":0.52900000000000003,"15min":0.52900000000000003},"min":{"1min":0.42599999999999999,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.67400000000000004,"5min":0.67400000000000004,"15min":0.67400000000000004},"last":0.57799999999999996},{"interface":"front","average":{"1min":0.53100000000000003,"5min":0.53100000000000003,"15min":0.53100000000000003},"min":{"1min":0.42399999999999999,"5min":0.42399999999999999,"15min":0.42399999999999999},"max":{"1min":0.65800000000000003,"5min":0.65800000000000003,"15min":0.65800000000000003},"last":0.53800000000000003}]},{"osd":2,"last update":"Tue Sep 10 19:10:35 2024","interfaces":[{"interface":"back","average":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"min":{"1min":0.45500000000000002,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":0.66400000000000003,"5min":0.66400000000000003,"15min":0.66400000000000003},"last":0.47999999999999998},{"interface":"front","average":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.51000000000000001},"min":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.64600000000000002,"5min":0.64600000000000002,"15min":0.64600000000000002},"last":0.504}]},{"osd":3,"last update":"Tue Sep 10 19:10:35 2024","interfaces":[{"interface":"back","average":{"1min":0.51600000000000001,"5min":0.51600000000000001,"15min":0.51600000000000001},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.68100000000000005,"5min":0.68100000000000005,"15min":0.68100000000000005},"last":0.45900000000000002},{"interface":"front","average":{"1min":0.56599999999999995,"5min":0.56599999999999995,"15min":0.56599999999999995},"min":{"1min":0.434,"5min":0.434,"15min":0.434},"max":{"1min":0.70199999999999996,"5min":0.70199999999999996,"15min":0.70199999999999996},"last":0.56399999999999995}]},{"osd":4,"last update":"Tue Sep 10 19:10:35 2024","interfaces":[{"interface":"back","average":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"min":{"1min":0.27600000000000002,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.46500000000000002,"5min":0.46500000000000002,"15min":0.46500000000000002},"last":0.377},{"interface":"front","average":{"1min":0.375,"5min":0.375,"15min":0.375},"min":{"1min":0.315,"5min":0.315,"15min":0.315},"max":{"1min":0.54400000000000004,"5min":0.54400000000000004,"15min":0.54400000000000004},"last":0.52400000000000002}]},{"osd":5,"last update":"Tue Sep 10 19:10:35 2024","interfaces":[{"interface":"back","average":{"1min":0.375,"5min":0.375,"15min":0.375},"min":{"1min":0.307,"5min":0.307,"15min":0.307},"max":{"1min":0.52200000000000002,"5min":0.52200000000000002,"15min":0.52200000000000002},"last":0.44400000000000001},{"interface":"front","average":{"1min":0.374,"5min":0.374,"15min":0.374},"min":{"1min":0.29999999999999999,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.51200000000000001,"5min":0.51200000000000001,"15min":0.51200000000000001},"last":0.61899999999999999}]},{"osd":7,"last update":"Tue Sep 10 19:10:54 2024","interfaces":[{"interface":"back","average":{"1min":0.44700000000000001,"5min":0.44700000000000001,"15min":0.44700000000000001},"min":{"1min":0.29899999999999999,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.68799999999999994,"5min":0.68799999999999994,"15min":0.68799999999999994},"last":0.60499999999999998},{"interface":"front","average":{"1min":0.44500000000000001,"5min":0.44500000000000001,"15min":0.44500000000000001},"min":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.55000000000000004,"5min":0.55000000000000004,"15min":0.55000000000000004},"last":0.55200000000000005}]}]},{"osd":5,"up_from":39,"seq":167503724576,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298216,"kb_used_data":864,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93446936,"statfs":{"total":95995035648,"available":95689662464,"internally_reserved":0,"allocated":884736,"data_stored":399317,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 19:11:08 2024","interfaces":[{"interface":"back","average":{"1min":0.51900000000000002,"5min":0.48299999999999998,"15min":0.47699999999999998},"min":{"1min":0.38200000000000001,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.67200000000000004,"5min":0.67200000000000004,"15min":0.67200000000000004},"last":0.33500000000000002},{"interface":"front","average":{"1min":0.53600000000000003,"5min":0.48399999999999999,"15min":0.47499999999999998},"min":{"1min":0.38300000000000001,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.75600000000000001,"5min":0.75600000000000001,"15min":0.75600000000000001},"last":0.35099999999999998}]},{"osd":1,"last update":"Tue Sep 10 19:11:08 2024","interfaces":[{"interface":"back","average":{"1min":0.58399999999999996,"5min":0.51900000000000002,"15min":0.50800000000000001},"min":{"1min":0.46800000000000003,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.72799999999999998,"5min":0.72799999999999998,"15min":0.72799999999999998},"last":0.40300000000000002},{"interface":"front","average":{"1min":0.50700000000000001,"5min":0.47899999999999998,"15min":0.47399999999999998},"min":{"1min":0.39900000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.67600000000000005,"5min":0.68799999999999994,"15min":0.68799999999999994},"last":0.34399999999999997}]},{"osd":2,"last update":"Tue Sep 10 19:11:08 2024","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.48599999999999999,"15min":0.47799999999999998},"min":{"1min":0.39900000000000002,"5min":0.309,"15min":0.309},"max":{"1min":0.70699999999999996,"5min":0.70699999999999996,"15min":0.70699999999999996},"last":0.35999999999999999},{"interface":"front","average":{"1min":0.54100000000000004,"5min":0.498,"15min":0.49099999999999999},"min":{"1min":0.39200000000000002,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.70799999999999996,"5min":0.70799999999999996,"15min":0.70799999999999996},"last":0.38600000000000001}]},{"osd":3,"last update":"Tue Sep 10 19:11:08 2024","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.52200000000000002,"15min":0.51400000000000001},"min":{"1min":0.437,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.74099999999999999,"5min":0.74099999999999999,"15min":0.74099999999999999},"last":0.39400000000000002},{"interface":"front","average":{"1min":0.59899999999999998,"5min":0.53800000000000003,"15min":0.52800000000000002},"min":{"1min":0.44700000000000001,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.747,"5min":0.747,"15min":0.747},"last":0.36799999999999999}]},{"osd":4,"last update":"Tue Sep 10 19:11:08 2024","interfaces":[{"interface":"back","average":{"1min":0.44,"5min":0.37,"15min":0.35799999999999998},"min":{"1min":0.27800000000000002,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":0.624,"5min":0.624,"15min":0.624},"last":0.32400000000000001},{"interface":"front","average":{"1min":0.46500000000000002,"5min":0.38100000000000001,"15min":0.36699999999999999},"min":{"1min":0.27800000000000002,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.76800000000000002,"5min":0.76800000000000002,"15min":0.76800000000000002},"last":0.28299999999999997}]},{"osd":6,"last update":"Tue Sep 10 19:10:34 2024","interfaces":[{"interface":"back","average":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"min":{"1min":0.223,"5min":0.223,"15min":0.223},"max":{"1min":0.61499999999999999,"5min":0.61499999999999999,"15min":0.61499999999999999},"last":0.374},{"interface":"front","average":{"1min":0.40600000000000003,"5min":0.40600000000000003,"15min":0.40600000000000003},"min":{"1min":0.25,"5min":0.25,"15min":0.25},"max":{"1min":0.52800000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"last":0.26800000000000002}]},{"osd":7,"last update":"Tue Sep 10 19:10:55 2024","interfaces":[{"interface":"back","average":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.51000000000000001},"min":{"1min":0.29099999999999998,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.72699999999999998,"5min":0.72699999999999998,"15min":0.72699999999999998},"last":0.31},{"interface":"front","average":{"1min":0.47899999999999998,"5min":0.47899999999999998,"15min":0.47899999999999998},"min":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.64600000000000002,"5min":0.64600000000000002,"15min":0.64600000000000002},"last":0.29699999999999999}]}]},{"osd":4,"up_from":32,"seq":137438953508,"num_pgs":52,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298232,"kb_used_data":880,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93446920,"statfs":{"total":95995035648,"available":95689646080,"internally_reserved":0,"allocated":901120,"data_stored":398526,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 19:10:48 2024","interfaces":[{"interface":"back","average":{"1min":0.49399999999999999,"5min":0.53200000000000003,"15min":0.53800000000000003},"min":{"1min":0.32500000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.61699999999999999,"5min":0.91500000000000004,"15min":0.91500000000000004},"last":0.45100000000000001},{"interface":"front","average":{"1min":0.47599999999999998,"5min":0.52000000000000002,"15min":0.52700000000000002},"min":{"1min":0.34499999999999997,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.63200000000000001,"5min":0.78200000000000003,"15min":0.78200000000000003},"last":0.51900000000000002}]},{"osd":1,"last update":"Tue Sep 10 19:10:48 2024","interfaces":[{"interface":"back","average":{"1min":0.5,"5min":0.51000000000000001,"15min":0.51200000000000001},"min":{"1min":0.33300000000000002,"5min":0.313,"15min":0.313},"max":{"1min":0.68799999999999994,"5min":0.83499999999999996,"15min":0.83499999999999996},"last":0.53000000000000003},{"interface":"front","average":{"1min":0.504,"5min":0.52000000000000002,"15min":0.52300000000000002},"min":{"1min":0.374,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.77100000000000002,"5min":0.77100000000000002,"15min":0.77100000000000002},"last":0.49299999999999999}]},{"osd":2,"last update":"Tue Sep 10 19:10:48 2024","interfaces":[{"interface":"back","average":{"1min":0.51100000000000001,"5min":0.53300000000000003,"15min":0.53700000000000003},"min":{"1min":0.39300000000000002,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.69599999999999995,"5min":1.0649999999999999,"15min":1.0649999999999999},"last":0.51100000000000001},{"interface":"front","average":{"1min":0.48499999999999999,"5min":0.52700000000000002,"15min":0.53400000000000003},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.66000000000000003,"5min":0.86899999999999999,"15min":0.86899999999999999},"last":0.48299999999999998}]},{"osd":3,"last update":"Tue Sep 10 19:10:48 2024","interfaces":[{"interface":"back","average":{"1min":0.498,"5min":0.52100000000000002,"15min":0.52500000000000002},"min":{"1min":0.373,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.67700000000000005,"5min":0.76500000000000001,"15min":0.76500000000000001},"last":0.54700000000000004},{"interface":"front","average":{"1min":0.52300000000000002,"5min":0.54600000000000004,"15min":0.55000000000000004},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.92200000000000004,"5min":1.0249999999999999,"15min":1.0249999999999999},"last":0.47399999999999998}]},{"osd":5,"last update":"Tue Sep 10 19:11:10 2024","interfaces":[{"interface":"back","average":{"1min":0.46400000000000002,"5min":0.39200000000000002,"15min":0.38},"min":{"1min":0.34999999999999998,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.61599999999999999,"5min":0.61599999999999999,"15min":0.61599999999999999},"last":0.35799999999999998},{"interface":"front","average":{"1min":0.47099999999999997,"5min":0.40899999999999997,"15min":0.39900000000000002},"min":{"1min":0.30599999999999999,"5min":0.223,"15min":0.223},"max":{"1min":0.63,"5min":0.89100000000000001,"15min":0.89100000000000001},"last":0.37}]},{"osd":6,"last update":"Tue Sep 10 19:10:33 2024","interfaces":[{"interface":"back","average":{"1min":0.41299999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"min":{"1min":0.26100000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.58999999999999997,"5min":0.58999999999999997,"15min":0.58999999999999997},"last":0.32400000000000001},{"interface":"front","average":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"min":{"1min":0.25,"5min":0.25,"15min":0.25},"max":{"1min":0.62,"5min":0.62,"15min":0.62},"last":0.31}]},{"osd":7,"last update":"Tue Sep 10 19:10:58 2024","interfaces":[{"interface":"back","average":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.72699999999999998,"5min":0.72699999999999998,"15min":0.72699999999999998},"last":0.38100000000000001},{"interface":"front","average":{"1min":0.503,"5min":0.503,"15min":0.503},"min":{"1min":0.29099999999999998,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.69899999999999995,"5min":0.69899999999999995,"15min":0.69899999999999995},"last":0.34100000000000003}]}]},{"osd":3,"up_from":26,"seq":111669149737,"num_pgs":55,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298252,"kb_used_data":900,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93446900,"statfs":{"total":95995035648,"available":95689625600,"internally_reserved":0,"allocated":921600,"data_stored":400100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 19:11:25 2024","interfaces":[{"interface":"back","average":{"1min":0.315,"5min":0.38700000000000001,"15min":0.40000000000000002},"min":{"1min":0.223,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":0.46800000000000003,"5min":0.61499999999999999,"15min":0.61499999999999999},"last":0.33100000000000002},{"interface":"front","average":{"1min":0.33000000000000002,"5min":0.39700000000000002,"15min":0.41199999999999998},"min":{"1min":0.24399999999999999,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":0.42299999999999999,"5min":0.63600000000000001,"15min":0.63600000000000001},"last":0.28699999999999998}]},{"osd":1,"last update":"Tue Sep 10 19:11:25 2024","interfaces":[{"interface":"back","average":{"1min":0.35499999999999998,"5min":0.42999999999999999,"15min":0.44700000000000001},"min":{"1min":0.253,"5min":0.253,"15min":0.253},"max":{"1min":0.505,"5min":0.67600000000000005,"15min":0.67600000000000005},"last":0.32200000000000001},{"interface":"front","average":{"1min":0.32900000000000001,"5min":0.38700000000000001,"15min":0.39400000000000002},"min":{"1min":0.26300000000000001,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.441,"5min":0.622,"15min":0.622},"last":0.34100000000000003}]},{"osd":2,"last update":"Tue Sep 10 19:11:25 2024","interfaces":[{"interface":"back","average":{"1min":0.38100000000000001,"5min":0.42499999999999999,"15min":0.434},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.51200000000000001,"5min":0.56599999999999995,"15min":0.56599999999999995},"last":0.26400000000000001},{"interface":"front","average":{"1min":0.38600000000000001,"5min":0.40799999999999997,"15min":0.41399999999999998},"min":{"1min":0.25600000000000001,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":0.66800000000000004,"5min":0.66800000000000004,"15min":0.66800000000000004},"last":0.29999999999999999}]},{"osd":4,"last update":"Tue Sep 10 19:10:50 2024","interfaces":[{"interface":"back","average":{"1min":0.54300000000000004,"5min":0.54400000000000004,"15min":0.54400000000000004},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.74399999999999999,"5min":0.74399999999999999,"15min":0.74399999999999999},"last":0.35299999999999998},{"interface":"front","average":{"1min":0.55400000000000005,"5min":0.54900000000000004,"15min":0.54800000000000004},"min":{"1min":0.433,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":0.81699999999999995,"5min":0.81699999999999995,"15min":0.81699999999999995},"last":0.40300000000000002}]},{"osd":5,"last update":"Tue Sep 10 19:11:07 2024","interfaces":[{"interface":"back","average":{"1min":0.58799999999999997,"5min":0.56999999999999995,"15min":0.56699999999999995},"min":{"1min":0.441,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.72599999999999998,"5min":0.80400000000000005,"15min":0.80400000000000005},"last":0.376},{"interface":"front","average":{"1min":0.56200000000000006,"5min":0.56100000000000005,"15min":0.56100000000000005},"min":{"1min":0.40799999999999997,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.89100000000000001,"5min":0.89100000000000001,"15min":0.89100000000000001},"last":0.36799999999999999}]},{"osd":6,"last update":"Tue Sep 10 19:11:34 2024","interfaces":[{"interface":"back","average":{"1min":0.53400000000000003,"5min":0.56999999999999995,"15min":0.57599999999999996},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.67000000000000004,"5min":0.77000000000000002,"15min":0.77000000000000002},"last":0.39600000000000002},{"interface":"front","average":{"1min":0.55100000000000005,"5min":0.55500000000000005,"15min":0.55600000000000005},"min":{"1min":0.41399999999999998,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.71799999999999997,"5min":0.90800000000000003,"15min":0.90800000000000003},"last":0.45200000000000001}]},{"osd":7,"last update":"Tue Sep 10 19:11:00 2024","interfaces":[{"interface":"back","average":{"1min":0.57999999999999996,"5min":0.57999999999999996,"15min":0.57999999999999996},"min":{"1min":0.45400000000000001,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":0.75600000000000001,"5min":0.75600000000000001,"15min":0.75600000000000001},"last":0.47099999999999997},{"interface":"front","average":{"1min":0.58399999999999996,"5min":0.58399999999999996,"15min":0.58399999999999996},"min":{"1min":0.443,"5min":0.443,"15min":0.443},"max":{"1min":0.749,"5min":0.749,"15min":0.749},"last":0.51100000000000001}]}]},{"osd":2,"up_from":18,"seq":77309411374,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298212,"kb_used_data":860,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93446940,"statfs":{"total":95995035648,"available":95689666560,"internally_reserved":0,"allocated":880640,"data_stored":398516,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 19:11:08 2024","interfaces":[{"interface":"back","average":{"1min":0.33800000000000002,"5min":0.34899999999999998,"15min":0.34300000000000003},"min":{"1min":0.20100000000000001,"5min":0.20100000000000001,"15min":0.20100000000000001},"max":{"1min":0.45100000000000001,"5min":0.52300000000000002,"15min":0.52300000000000002},"last":0.20100000000000001},{"interface":"front","average":{"1min":0.33200000000000002,"5min":0.374,"15min":0.36799999999999999},"min":{"1min":0.23100000000000001,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":0.44700000000000001,"5min":0.64200000000000002,"15min":0.64200000000000002},"last":0.34799999999999998}]},{"osd":1,"last update":"Tue Sep 10 19:11:08 2024","interfaces":[{"interface":"back","average":{"1min":0.38200000000000001,"5min":0.38900000000000001,"15min":0.379},"min":{"1min":0.24099999999999999,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.63,"5min":0.69799999999999995,"15min":0.69799999999999995},"last":0.252},{"interface":"front","average":{"1min":0.35799999999999998,"5min":0.39700000000000002,"15min":0.39100000000000001},"min":{"1min":0.28499999999999998,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.48299999999999998,"5min":0.63400000000000001,"15min":0.63400000000000001},"last":0.26400000000000001}]},{"osd":3,"last update":"Tue Sep 10 19:11:30 2024","interfaces":[{"interface":"back","average":{"1min":0.35699999999999998,"5min":0.435,"15min":0.44600000000000001},"min":{"1min":0.23699999999999999,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":0.49099999999999999,"5min":0.63900000000000001,"15min":0.63900000000000001},"last":0.28299999999999997},{"interface":"front","average":{"1min":0.36199999999999999,"5min":0.432,"15min":0.442},"min":{"1min":0.27100000000000002,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.438,"5min":0.65800000000000003,"15min":0.65800000000000003},"last":0.308}]},{"osd":4,"last update":"Tue Sep 10 19:10:50 2024","interfaces":[{"interface":"back","average":{"1min":0.59099999999999997,"5min":0.58299999999999996,"15min":0.58099999999999996},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.81999999999999995,"5min":0.81999999999999995,"15min":0.81999999999999995},"last":0.30099999999999999},{"interface":"front","average":{"1min":0.55600000000000005,"5min":0.56799999999999995,"15min":0.56999999999999995},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.749,"5min":0.80600000000000005,"15min":0.80600000000000005},"last":0.432}]},{"osd":5,"last update":"Tue Sep 10 19:11:08 2024","interfaces":[{"interface":"back","average":{"1min":0.54800000000000004,"5min":0.52900000000000003,"15min":0.52600000000000002},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.73599999999999999,"5min":0.73899999999999999,"15min":0.73899999999999999},"last":0.32200000000000001},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.57899999999999996,"15min":0.57899999999999996},"min":{"1min":0.443,"5min":0.443,"15min":0.443},"max":{"1min":0.70699999999999996,"5min":0.82699999999999996,"15min":0.82699999999999996},"last":0.33500000000000002}]},{"osd":6,"last update":"Tue Sep 10 19:10:32 2024","interfaces":[{"interface":"back","average":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"min":{"1min":0.432,"5min":0.432,"15min":0.432},"max":{"1min":0.73199999999999998,"5min":0.73199999999999998,"15min":0.73199999999999998},"last":0.37},{"interface":"front","average":{"1min":0.55100000000000005,"5min":0.55100000000000005,"15min":0.55100000000000005},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.748,"5min":0.748,"15min":0.748},"last":0.39100000000000001}]},{"osd":7,"last update":"Tue Sep 10 19:11:00 2024","interfaces":[{"interface":"back","average":{"1min":0.60599999999999998,"5min":0.60599999999999998,"15min":0.60599999999999998},"min":{"1min":0.44700000000000001,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":0.79700000000000004,"5min":0.79700000000000004,"15min":0.79700000000000004},"last":0.41399999999999998},{"interface":"front","average":{"1min":0.54900000000000004,"5min":0.54900000000000004,"15min":0.54900000000000004},"min":{"1min":0.40300000000000002,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":0.72599999999999998,"5min":0.72599999999999998,"15min":0.72599999999999998},"last":0.45800000000000002}]}]},{"osd":1,"up_from":13,"seq":55834574899,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298260,"kb_used_data":908,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93446892,"statfs":{"total":95995035648,"available":95689617408,"internally_reserved":0,"allocated":929792,"data_stored":399674,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 19:10:39 2024","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.36499999999999999,"15min":0.33400000000000002},"min":{"1min":0.188,"5min":0.188,"15min":0.188},"max":{"1min":2.7730000000000001,"5min":2.7730000000000001,"15min":2.7730000000000001},"last":0.246},{"interface":"front","average":{"1min":0.47999999999999998,"5min":0.373,"15min":0.34499999999999997},"min":{"1min":0.20599999999999999,"5min":0.20599999999999999,"15min":0.20599999999999999},"max":{"1min":2.8380000000000001,"5min":2.8380000000000001,"15min":2.8380000000000001},"last":0.26000000000000001}]},{"osd":2,"last update":"Tue Sep 10 19:11:01 2024","interfaces":[{"interface":"back","average":{"1min":0.42399999999999999,"5min":0.38500000000000001,"15min":0.373},"min":{"1min":0.251,"5min":0.251,"15min":0.251},"max":{"1min":1.1779999999999999,"5min":1.1779999999999999,"15min":1.1779999999999999},"last":0.33300000000000002},{"interface":"front","average":{"1min":0.42899999999999999,"5min":0.38,"15min":0.36799999999999999},"min":{"1min":0.254,"5min":0.254,"15min":0.254},"max":{"1min":1.103,"5min":1.103,"15min":1.103},"last":0.47399999999999998}]},{"osd":3,"last update":"Tue Sep 10 19:11:29 2024","interfaces":[{"interface":"back","average":{"1min":0.44700000000000001,"5min":0.45600000000000002,"15min":0.44900000000000001},"min":{"1min":0.27500000000000002,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":1.093,"5min":2.0670000000000002,"15min":2.0670000000000002},"last":0.27800000000000002},{"interface":"front","average":{"1min":0.40100000000000002,"5min":0.42299999999999999,"15min":0.42099999999999999},"min":{"1min":0.30299999999999999,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":0.54600000000000004,"5min":1.956,"15min":1.956},"last":0.35299999999999998}]},{"osd":4,"last update":"Tue Sep 10 19:10:49 2024","interfaces":[{"interface":"back","average":{"1min":0.61299999999999999,"5min":0.52200000000000002,"15min":0.50700000000000001},"min":{"1min":0.28499999999999998,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":2.665,"5min":2.665,"15min":2.665},"last":0.42899999999999999},{"interface":"front","average":{"1min":0.65200000000000002,"5min":0.56200000000000006,"15min":0.54700000000000004},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":2.7330000000000001,"5min":2.7330000000000001,"15min":2.7330000000000001},"last":0.64500000000000002}]},{"osd":5,"last update":"Tue Sep 10 19:11:11 2024","interfaces":[{"interface":"back","average":{"1min":0.58299999999999996,"5min":0.53800000000000003,"15min":0.53000000000000003},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":1.851,"5min":1.851,"15min":1.851},"last":0.54400000000000004},{"interface":"front","average":{"1min":0.63500000000000001,"5min":0.55100000000000005,"15min":0.53700000000000003},"min":{"1min":0.31900000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":2.706,"5min":2.706,"15min":2.706},"last":0.50700000000000001}]},{"osd":6,"last update":"Tue Sep 10 19:10:31 2024","interfaces":[{"interface":"back","average":{"1min":0.61899999999999999,"5min":0.61899999999999999,"15min":0.61899999999999999},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":1.9950000000000001,"5min":1.9950000000000001,"15min":1.9950000000000001},"last":0.41599999999999998},{"interface":"front","average":{"1min":0.59799999999999998,"5min":0.59799999999999998,"15min":0.59799999999999998},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":1.9399999999999999,"5min":1.9399999999999999,"15min":1.9399999999999999},"last":0.39300000000000002}]},{"osd":7,"last update":"Tue Sep 10 19:10:55 2024","interfaces":[{"interface":"back","average":{"1min":0.66600000000000004,"5min":0.66600000000000004,"15min":0.66600000000000004},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":3.4620000000000002,"5min":3.4620000000000002,"15min":3.4620000000000002},"last":0.45200000000000001},{"interface":"front","average":{"1min":0.621,"5min":0.621,"15min":0.621},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":1.514,"5min":1.514,"15min":1.514},"last":0.46100000000000002}]}]},{"osd":0,"up_from":8,"seq":34359738424,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298808,"kb_used_data":1456,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93446344,"statfs":{"total":95995035648,"available":95689056256,"internally_reserved":0,"allocated":1490944,"data_stored":989601,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Sep 10 19:10:42 2024","interfaces":[{"interface":"back","average":{"1min":0.373,"5min":0.34899999999999998,"15min":0.33100000000000002},"min":{"1min":0.221,"5min":0.221,"15min":0.221},"max":{"1min":0.53100000000000003,"5min":0.63300000000000001,"15min":0.63300000000000001},"last":0.27400000000000002},{"interface":"front","average":{"1min":0.36499999999999999,"5min":0.34999999999999998,"15min":0.33400000000000002},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":0.51900000000000002,"5min":0.61899999999999999,"15min":0.61899999999999999},"last":0.378}]},{"osd":2,"last update":"Tue Sep 10 19:11:08 2024","interfaces":[{"interface":"back","average":{"1min":0.375,"5min":0.38700000000000001,"15min":0.38300000000000001},"min":{"1min":0.27600000000000002,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":0.50700000000000001,"5min":0.57399999999999995,"15min":0.57399999999999995},"last":0.34100000000000003},{"interface":"front","average":{"1min":0.36499999999999999,"5min":0.375,"15min":0.373},"min":{"1min":0.22900000000000001,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":0.53000000000000003,"5min":0.54100000000000004,"15min":0.54100000000000004},"last":0.312}]},{"osd":3,"last update":"Tue Sep 10 19:11:30 2024","interfaces":[{"interface":"back","average":{"1min":0.35999999999999999,"5min":0.41499999999999998,"15min":0.42099999999999999},"min":{"1min":0.29499999999999998,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.45500000000000002,"5min":0.65000000000000002,"15min":0.65000000000000002},"last":0.36099999999999999},{"interface":"front","average":{"1min":0.34499999999999997,"5min":0.39900000000000002,"15min":0.40500000000000003},"min":{"1min":0.27800000000000002,"5min":0.246,"15min":0.246},"max":{"1min":0.45900000000000002,"5min":0.58799999999999997,"15min":0.58799999999999997},"last":0.32900000000000001}]},{"osd":4,"last update":"Tue Sep 10 19:10:46 2024","interfaces":[{"interface":"back","average":{"1min":0.54700000000000004,"5min":0.56200000000000006,"15min":0.56399999999999995},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.76000000000000001,"5min":0.76000000000000001,"15min":0.76000000000000001},"last":0.59099999999999997},{"interface":"front","average":{"1min":0.56999999999999995,"5min":0.56299999999999994,"15min":0.56200000000000006},"min":{"1min":0.40200000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.73699999999999999,"5min":0.77000000000000002,"15min":0.77000000000000002},"last":0.52900000000000003}]},{"osd":5,"last update":"Tue Sep 10 19:11:08 2024","interfaces":[{"interface":"back","average":{"1min":0.58399999999999996,"5min":0.58399999999999996,"15min":0.58399999999999996},"min":{"1min":0.44800000000000001,"5min":0.378,"15min":0.378},"max":{"1min":0.82599999999999996,"5min":0.82599999999999996,"15min":0.82599999999999996},"last":0.51100000000000001},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.57399999999999995,"15min":0.57499999999999996},"min":{"1min":0.42699999999999999,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.77900000000000003,"5min":0.82899999999999996,"15min":0.82899999999999996},"last":0.61099999999999999}]},{"osd":6,"last update":"Tue Sep 10 19:11:33 2024","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":0.55800000000000005,"15min":0.56399999999999995},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.78900000000000003,"5min":0.78900000000000003,"15min":0.78900000000000003},"last":0.503},{"interface":"front","average":{"1min":0.54800000000000004,"5min":0.57899999999999996,"15min":0.58399999999999996},"min":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":0.70499999999999996,"5min":0.76100000000000001,"15min":0.76100000000000001},"last":0.56599999999999995}]},{"osd":7,"last update":"Tue Sep 10 19:10:58 2024","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.57799999999999996,"15min":0.57799999999999996},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":0.76900000000000002,"5min":0.76900000000000002,"15min":0.76900000000000002},"last":0.54900000000000004},{"interface":"front","average":{"1min":0.56200000000000006,"5min":0.56200000000000006,"15min":0.56200000000000006},"min":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.72599999999999998,"5min":0.72599999999999998,"15min":0.72599999999999998},"last":0.57899999999999996}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":966,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1236,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1236,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":362,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":2338,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":798,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T19:11:36.700 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-09-10T19:11:36.700 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-09-10T19:11:36.700 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2024-09-10T19:11:36.700 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph health --format=json 2024-09-10T19:11:36.931 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:11:36 smithi183 ceph-mon[29736]: 6.1e scrub starts 2024-09-10T19:11:36.931 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:11:36 smithi183 ceph-mon[29736]: 6.1e scrub ok 2024-09-10T19:11:36.931 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:11:36 smithi183 ceph-mon[29736]: pgmap v36: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:11:36.931 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:11:36 smithi183 ceph-mon[29736]: from='client.24608 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-10T19:11:36.931 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:11:36 smithi183 ceph-mon[41562]: 6.1e scrub starts 2024-09-10T19:11:36.931 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:11:36 smithi183 ceph-mon[41562]: 6.1e scrub ok 2024-09-10T19:11:36.931 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:11:36 smithi183 ceph-mon[41562]: pgmap v36: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:11:36.932 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:11:36 smithi183 ceph-mon[41562]: from='client.24608 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-10T19:11:36.944 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:11:36.965 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:11:36 smithi191 ceph-mon[32884]: 6.1e scrub starts 2024-09-10T19:11:36.965 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:11:36 smithi191 ceph-mon[32884]: 6.1e scrub ok 2024-09-10T19:11:36.965 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:11:36 smithi191 ceph-mon[32884]: pgmap v36: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:11:36.966 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:11:36 smithi191 ceph-mon[32884]: from='client.24608 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-10T19:11:36.966 INFO:journalctl@ceph.node-exporter.b.smithi191.stdout:Sep 10 19:11:36 smithi191 bash[66046]: Copying config sha256:0da6a335fe1356545476b749c68f022c897de3a2139e8f0054f6937349ee2b83 2024-09-10T19:11:36.966 INFO:journalctl@ceph.node-exporter.b.smithi191.stdout:Sep 10 19:11:36 smithi191 bash[66046]: Writing manifest to image destination 2024-09-10T19:11:37.273 INFO:journalctl@ceph.node-exporter.b.smithi191.stdout:Sep 10 19:11:36 smithi191 podman[66046]: 2024-09-10 19:11:36.946405153 +0000 UTC m=+2.089386166 image pull 0da6a335fe1356545476b749c68f022c897de3a2139e8f0054f6937349ee2b83 quay.io/prometheus/node-exporter:v1.5.0 2024-09-10T19:11:37.273 INFO:journalctl@ceph.node-exporter.b.smithi191.stdout:Sep 10 19:11:37 smithi191 podman[66046]: 2024-09-10 19:11:37.061536496 +0000 UTC m=+2.204517500 container create 44646e3eeb19f15281735e23d6f999ef803b33a4eab2191dc67439562f12264a (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-node-exporter-b, maintainer=The Prometheus Authors ) 2024-09-10T19:11:37.273 INFO:journalctl@ceph.node-exporter.b.smithi191.stdout:Sep 10 19:11:37 smithi191 podman[66046]: 2024-09-10 19:11:37.272105358 +0000 UTC m=+2.415086361 container init 44646e3eeb19f15281735e23d6f999ef803b33a4eab2191dc67439562f12264a (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-node-exporter-b, maintainer=The Prometheus Authors ) 2024-09-10T19:11:37.659 INFO:journalctl@ceph.mgr.x.smithi191.stdout:Sep 10 19:11:37 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-x[34967]: ::ffff:172.21.15.191 - - [10/Sep/2024:19:11:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T19:11:37.660 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:11:37 smithi183 ceph-mon[29736]: 5.6 scrub starts 2024-09-10T19:11:37.661 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:11:37 smithi183 ceph-mon[29736]: 5.6 scrub ok 2024-09-10T19:11:37.661 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:11:37 smithi183 ceph-mon[29736]: 4.11 scrub starts 2024-09-10T19:11:37.661 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:11:37 smithi183 ceph-mon[29736]: 4.11 scrub ok 2024-09-10T19:11:37.661 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:11:37 smithi183 ceph-mon[29736]: 4.15 scrub starts 2024-09-10T19:11:37.661 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:11:37 smithi183 ceph-mon[29736]: 4.15 scrub ok 2024-09-10T19:11:37.661 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:11:37 smithi183 ceph-mon[29736]: from='mgr.14556 ' entity='mgr.y' 2024-09-10T19:11:37.661 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:11:37 smithi183 ceph-mon[29736]: from='mgr.14556 ' entity='mgr.y' 2024-09-10T19:11:37.661 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:11:37 smithi183 ceph-mon[29736]: from='mgr.14556 ' entity='mgr.y' 2024-09-10T19:11:37.661 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:11:37 smithi183 ceph-mon[29736]: from='mgr.14556 ' entity='mgr.y' 2024-09-10T19:11:37.661 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:11:37 smithi183 ceph-mon[29736]: from='mgr.14556 ' entity='mgr.y' 2024-09-10T19:11:37.661 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:11:37 smithi183 ceph-mon[29736]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T19:11:37.662 INFO:journalctl@ceph.node-exporter.b.smithi191.stdout:Sep 10 19:11:37 smithi191 podman[66046]: 2024-09-10 19:11:37.276709609 +0000 UTC m=+2.419690612 container start 44646e3eeb19f15281735e23d6f999ef803b33a4eab2191dc67439562f12264a (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-node-exporter-b, maintainer=The Prometheus Authors ) 2024-09-10T19:11:37.662 INFO:journalctl@ceph.node-exporter.b.smithi191.stdout:Sep 10 19:11:37 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-node-exporter-b[66120]: ts=2024-09-10T19:11:37.280Z caller=node_exporter.go:180 level=info msg="Starting node_exporter" version="(version=1.5.0, branch=HEAD, revision=1b48970ffcf5630534fb00bb0687d73c66d1c959)" 2024-09-10T19:11:37.662 INFO:journalctl@ceph.node-exporter.b.smithi191.stdout:Sep 10 19:11:37 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-node-exporter-b[66120]: ts=2024-09-10T19:11:37.280Z caller=node_exporter.go:181 level=info msg="Build context" build_context="(go=go1.19.3, user=root@6e7732a7b81b, date=20221129-18:59:09)" 2024-09-10T19:11:37.662 INFO:journalctl@ceph.node-exporter.b.smithi191.stdout:Sep 10 19:11:37 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-node-exporter-b[66120]: ts=2024-09-10T19:11:37.280Z caller=filesystem_common.go:111 level=info collector=filesystem msg="Parsed flag --collector.filesystem.mount-points-exclude" flag=^/(dev|proc|run/credentials/.+|sys|var/lib/docker/.+|var/lib/containers/storage/.+)($|/) 2024-09-10T19:11:37.662 INFO:journalctl@ceph.node-exporter.b.smithi191.stdout:Sep 10 19:11:37 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-node-exporter-b[66120]: ts=2024-09-10T19:11:37.280Z caller=filesystem_common.go:113 level=info collector=filesystem msg="Parsed flag --collector.filesystem.fs-types-exclude" flag=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$ 2024-09-10T19:11:37.662 INFO:journalctl@ceph.node-exporter.b.smithi191.stdout:Sep 10 19:11:37 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-node-exporter-b[66120]: ts=2024-09-10T19:11:37.281Z caller=diskstats_common.go:111 level=info collector=diskstats msg="Parsed flag --collector.diskstats.device-exclude" flag=^(ram|loop|fd|(h|s|v|xv)d[a-z]|nvme\d+n\d+p)\d+$ 2024-09-10T19:11:37.662 INFO:journalctl@ceph.node-exporter.b.smithi191.stdout:Sep 10 19:11:37 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-node-exporter-b[66120]: ts=2024-09-10T19:11:37.281Z caller=diskstats_linux.go:264 level=error collector=diskstats msg="Failed to open directory, disabling udev device properties" path=/run/udev/data 2024-09-10T19:11:37.662 INFO:journalctl@ceph.node-exporter.b.smithi191.stdout:Sep 10 19:11:37 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-node-exporter-b[66120]: ts=2024-09-10T19:11:37.281Z caller=node_exporter.go:110 level=info msg="Enabled collectors" 2024-09-10T19:11:37.662 INFO:journalctl@ceph.node-exporter.b.smithi191.stdout:Sep 10 19:11:37 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-node-exporter-b[66120]: ts=2024-09-10T19:11:37.281Z caller=node_exporter.go:117 level=info collector=arp 2024-09-10T19:11:37.662 INFO:journalctl@ceph.node-exporter.b.smithi191.stdout:Sep 10 19:11:37 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-node-exporter-b[66120]: ts=2024-09-10T19:11:37.281Z caller=node_exporter.go:117 level=info collector=bcache 2024-09-10T19:11:37.663 INFO:journalctl@ceph.node-exporter.b.smithi191.stdout:Sep 10 19:11:37 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-node-exporter-b[66120]: ts=2024-09-10T19:11:37.281Z caller=node_exporter.go:117 level=info collector=bonding 2024-09-10T19:11:37.663 INFO:journalctl@ceph.node-exporter.b.smithi191.stdout:Sep 10 19:11:37 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-node-exporter-b[66120]: ts=2024-09-10T19:11:37.281Z caller=node_exporter.go:117 level=info collector=btrfs 2024-09-10T19:11:37.663 INFO:journalctl@ceph.node-exporter.b.smithi191.stdout:Sep 10 19:11:37 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-node-exporter-b[66120]: ts=2024-09-10T19:11:37.281Z caller=node_exporter.go:117 level=info collector=conntrack 2024-09-10T19:11:37.663 INFO:journalctl@ceph.node-exporter.b.smithi191.stdout:Sep 10 19:11:37 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-node-exporter-b[66120]: ts=2024-09-10T19:11:37.281Z caller=node_exporter.go:117 level=info collector=cpu 2024-09-10T19:11:37.663 INFO:journalctl@ceph.node-exporter.b.smithi191.stdout:Sep 10 19:11:37 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-node-exporter-b[66120]: ts=2024-09-10T19:11:37.281Z caller=node_exporter.go:117 level=info collector=cpufreq 2024-09-10T19:11:37.663 INFO:journalctl@ceph.node-exporter.b.smithi191.stdout:Sep 10 19:11:37 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-node-exporter-b[66120]: ts=2024-09-10T19:11:37.281Z caller=node_exporter.go:117 level=info collector=diskstats 2024-09-10T19:11:37.663 INFO:journalctl@ceph.node-exporter.b.smithi191.stdout:Sep 10 19:11:37 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-node-exporter-b[66120]: ts=2024-09-10T19:11:37.281Z caller=node_exporter.go:117 level=info collector=dmi 2024-09-10T19:11:37.663 INFO:journalctl@ceph.node-exporter.b.smithi191.stdout:Sep 10 19:11:37 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-node-exporter-b[66120]: ts=2024-09-10T19:11:37.281Z caller=node_exporter.go:117 level=info collector=edac 2024-09-10T19:11:37.663 INFO:journalctl@ceph.node-exporter.b.smithi191.stdout:Sep 10 19:11:37 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-node-exporter-b[66120]: ts=2024-09-10T19:11:37.281Z caller=node_exporter.go:117 level=info collector=entropy 2024-09-10T19:11:37.663 INFO:journalctl@ceph.node-exporter.b.smithi191.stdout:Sep 10 19:11:37 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-node-exporter-b[66120]: ts=2024-09-10T19:11:37.281Z caller=node_exporter.go:117 level=info collector=fibrechannel 2024-09-10T19:11:37.663 INFO:journalctl@ceph.node-exporter.b.smithi191.stdout:Sep 10 19:11:37 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-node-exporter-b[66120]: ts=2024-09-10T19:11:37.281Z caller=node_exporter.go:117 level=info collector=filefd 2024-09-10T19:11:37.663 INFO:journalctl@ceph.node-exporter.b.smithi191.stdout:Sep 10 19:11:37 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-node-exporter-b[66120]: ts=2024-09-10T19:11:37.281Z caller=node_exporter.go:117 level=info collector=filesystem 2024-09-10T19:11:37.663 INFO:journalctl@ceph.node-exporter.b.smithi191.stdout:Sep 10 19:11:37 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-node-exporter-b[66120]: ts=2024-09-10T19:11:37.281Z caller=node_exporter.go:117 level=info collector=hwmon 2024-09-10T19:11:37.664 INFO:journalctl@ceph.node-exporter.b.smithi191.stdout:Sep 10 19:11:37 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-node-exporter-b[66120]: ts=2024-09-10T19:11:37.281Z caller=node_exporter.go:117 level=info collector=infiniband 2024-09-10T19:11:37.664 INFO:journalctl@ceph.node-exporter.b.smithi191.stdout:Sep 10 19:11:37 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-node-exporter-b[66120]: ts=2024-09-10T19:11:37.281Z caller=node_exporter.go:117 level=info collector=ipvs 2024-09-10T19:11:37.664 INFO:journalctl@ceph.node-exporter.b.smithi191.stdout:Sep 10 19:11:37 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-node-exporter-b[66120]: ts=2024-09-10T19:11:37.282Z caller=node_exporter.go:117 level=info collector=loadavg 2024-09-10T19:11:37.664 INFO:journalctl@ceph.node-exporter.b.smithi191.stdout:Sep 10 19:11:37 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-node-exporter-b[66120]: ts=2024-09-10T19:11:37.282Z caller=node_exporter.go:117 level=info collector=mdadm 2024-09-10T19:11:37.664 INFO:journalctl@ceph.node-exporter.b.smithi191.stdout:Sep 10 19:11:37 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-node-exporter-b[66120]: ts=2024-09-10T19:11:37.282Z caller=node_exporter.go:117 level=info collector=meminfo 2024-09-10T19:11:37.664 INFO:journalctl@ceph.node-exporter.b.smithi191.stdout:Sep 10 19:11:37 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-node-exporter-b[66120]: ts=2024-09-10T19:11:37.282Z caller=node_exporter.go:117 level=info collector=netclass 2024-09-10T19:11:37.664 INFO:journalctl@ceph.node-exporter.b.smithi191.stdout:Sep 10 19:11:37 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-node-exporter-b[66120]: ts=2024-09-10T19:11:37.282Z caller=node_exporter.go:117 level=info collector=netdev 2024-09-10T19:11:37.664 INFO:journalctl@ceph.node-exporter.b.smithi191.stdout:Sep 10 19:11:37 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-node-exporter-b[66120]: ts=2024-09-10T19:11:37.282Z caller=node_exporter.go:117 level=info collector=netstat 2024-09-10T19:11:37.665 INFO:journalctl@ceph.node-exporter.b.smithi191.stdout:Sep 10 19:11:37 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-node-exporter-b[66120]: ts=2024-09-10T19:11:37.282Z caller=node_exporter.go:117 level=info collector=nfs 2024-09-10T19:11:37.665 INFO:journalctl@ceph.node-exporter.b.smithi191.stdout:Sep 10 19:11:37 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-node-exporter-b[66120]: ts=2024-09-10T19:11:37.282Z caller=node_exporter.go:117 level=info collector=nfsd 2024-09-10T19:11:37.665 INFO:journalctl@ceph.node-exporter.b.smithi191.stdout:Sep 10 19:11:37 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-node-exporter-b[66120]: ts=2024-09-10T19:11:37.282Z caller=node_exporter.go:117 level=info collector=nvme 2024-09-10T19:11:37.665 INFO:journalctl@ceph.node-exporter.b.smithi191.stdout:Sep 10 19:11:37 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-node-exporter-b[66120]: ts=2024-09-10T19:11:37.282Z caller=node_exporter.go:117 level=info collector=os 2024-09-10T19:11:37.665 INFO:journalctl@ceph.node-exporter.b.smithi191.stdout:Sep 10 19:11:37 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-node-exporter-b[66120]: ts=2024-09-10T19:11:37.282Z caller=node_exporter.go:117 level=info collector=powersupplyclass 2024-09-10T19:11:37.665 INFO:journalctl@ceph.node-exporter.b.smithi191.stdout:Sep 10 19:11:37 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-node-exporter-b[66120]: ts=2024-09-10T19:11:37.282Z caller=node_exporter.go:117 level=info collector=pressure 2024-09-10T19:11:37.665 INFO:journalctl@ceph.node-exporter.b.smithi191.stdout:Sep 10 19:11:37 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-node-exporter-b[66120]: ts=2024-09-10T19:11:37.282Z caller=node_exporter.go:117 level=info collector=rapl 2024-09-10T19:11:37.665 INFO:journalctl@ceph.node-exporter.b.smithi191.stdout:Sep 10 19:11:37 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-node-exporter-b[66120]: ts=2024-09-10T19:11:37.282Z caller=node_exporter.go:117 level=info collector=schedstat 2024-09-10T19:11:37.665 INFO:journalctl@ceph.node-exporter.b.smithi191.stdout:Sep 10 19:11:37 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-node-exporter-b[66120]: ts=2024-09-10T19:11:37.282Z caller=node_exporter.go:117 level=info collector=selinux 2024-09-10T19:11:37.665 INFO:journalctl@ceph.node-exporter.b.smithi191.stdout:Sep 10 19:11:37 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-node-exporter-b[66120]: ts=2024-09-10T19:11:37.282Z caller=node_exporter.go:117 level=info collector=sockstat 2024-09-10T19:11:37.665 INFO:journalctl@ceph.node-exporter.b.smithi191.stdout:Sep 10 19:11:37 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-node-exporter-b[66120]: ts=2024-09-10T19:11:37.282Z caller=node_exporter.go:117 level=info collector=softnet 2024-09-10T19:11:37.665 INFO:journalctl@ceph.node-exporter.b.smithi191.stdout:Sep 10 19:11:37 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-node-exporter-b[66120]: ts=2024-09-10T19:11:37.282Z caller=node_exporter.go:117 level=info collector=stat 2024-09-10T19:11:37.666 INFO:journalctl@ceph.node-exporter.b.smithi191.stdout:Sep 10 19:11:37 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-node-exporter-b[66120]: ts=2024-09-10T19:11:37.282Z caller=node_exporter.go:117 level=info collector=tapestats 2024-09-10T19:11:37.666 INFO:journalctl@ceph.node-exporter.b.smithi191.stdout:Sep 10 19:11:37 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-node-exporter-b[66120]: ts=2024-09-10T19:11:37.282Z caller=node_exporter.go:117 level=info collector=textfile 2024-09-10T19:11:37.666 INFO:journalctl@ceph.node-exporter.b.smithi191.stdout:Sep 10 19:11:37 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-node-exporter-b[66120]: ts=2024-09-10T19:11:37.282Z caller=node_exporter.go:117 level=info collector=thermal_zone 2024-09-10T19:11:37.666 INFO:journalctl@ceph.node-exporter.b.smithi191.stdout:Sep 10 19:11:37 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-node-exporter-b[66120]: ts=2024-09-10T19:11:37.282Z caller=node_exporter.go:117 level=info collector=time 2024-09-10T19:11:37.666 INFO:journalctl@ceph.node-exporter.b.smithi191.stdout:Sep 10 19:11:37 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-node-exporter-b[66120]: ts=2024-09-10T19:11:37.282Z caller=node_exporter.go:117 level=info collector=udp_queues 2024-09-10T19:11:37.666 INFO:journalctl@ceph.node-exporter.b.smithi191.stdout:Sep 10 19:11:37 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-node-exporter-b[66120]: ts=2024-09-10T19:11:37.282Z caller=node_exporter.go:117 level=info collector=uname 2024-09-10T19:11:37.666 INFO:journalctl@ceph.node-exporter.b.smithi191.stdout:Sep 10 19:11:37 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-node-exporter-b[66120]: ts=2024-09-10T19:11:37.282Z caller=node_exporter.go:117 level=info collector=vmstat 2024-09-10T19:11:37.666 INFO:journalctl@ceph.node-exporter.b.smithi191.stdout:Sep 10 19:11:37 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-node-exporter-b[66120]: ts=2024-09-10T19:11:37.282Z caller=node_exporter.go:117 level=info collector=xfs 2024-09-10T19:11:37.666 INFO:journalctl@ceph.node-exporter.b.smithi191.stdout:Sep 10 19:11:37 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-node-exporter-b[66120]: ts=2024-09-10T19:11:37.282Z caller=node_exporter.go:117 level=info collector=zfs 2024-09-10T19:11:37.666 INFO:journalctl@ceph.node-exporter.b.smithi191.stdout:Sep 10 19:11:37 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-node-exporter-b[66120]: ts=2024-09-10T19:11:37.282Z caller=tls_config.go:232 level=info msg="Listening on" address=[::]:9100 2024-09-10T19:11:37.666 INFO:journalctl@ceph.node-exporter.b.smithi191.stdout:Sep 10 19:11:37 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-node-exporter-b[66120]: ts=2024-09-10T19:11:37.282Z caller=tls_config.go:235 level=info msg="TLS is disabled." http2=false address=[::]:9100 2024-09-10T19:11:37.666 INFO:journalctl@ceph.node-exporter.b.smithi191.stdout:Sep 10 19:11:37 smithi191 bash[66046]: 44646e3eeb19f15281735e23d6f999ef803b33a4eab2191dc67439562f12264a 2024-09-10T19:11:37.666 INFO:journalctl@ceph.node-exporter.b.smithi191.stdout:Sep 10 19:11:37 smithi191 systemd[1]: Started Ceph node-exporter.b for 320f6df8-6fa7-11ef-bcea-c7b262605968. 2024-09-10T19:11:37.931 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:11:37 smithi183 ceph-mon[41562]: 5.6 scrub starts 2024-09-10T19:11:37.931 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:11:37 smithi183 ceph-mon[41562]: 5.6 scrub ok 2024-09-10T19:11:37.932 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:11:37 smithi183 ceph-mon[41562]: 4.11 scrub starts 2024-09-10T19:11:37.932 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:11:37 smithi183 ceph-mon[41562]: 4.11 scrub ok 2024-09-10T19:11:37.932 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:11:37 smithi183 ceph-mon[41562]: 4.15 scrub starts 2024-09-10T19:11:37.932 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:11:37 smithi183 ceph-mon[41562]: 4.15 scrub ok 2024-09-10T19:11:37.932 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:11:37 smithi183 ceph-mon[41562]: from='mgr.14556 ' entity='mgr.y' 2024-09-10T19:11:37.932 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:11:37 smithi183 ceph-mon[41562]: from='mgr.14556 ' entity='mgr.y' 2024-09-10T19:11:37.932 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:11:37 smithi183 ceph-mon[41562]: from='mgr.14556 ' entity='mgr.y' 2024-09-10T19:11:37.932 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:11:37 smithi183 ceph-mon[41562]: from='mgr.14556 ' entity='mgr.y' 2024-09-10T19:11:37.932 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:11:37 smithi183 ceph-mon[41562]: from='mgr.14556 ' entity='mgr.y' 2024-09-10T19:11:37.932 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:11:37 smithi183 ceph-mon[41562]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T19:11:37.965 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:11:37 smithi191 ceph-mon[32884]: 5.6 scrub starts 2024-09-10T19:11:37.965 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:11:37 smithi191 ceph-mon[32884]: 5.6 scrub ok 2024-09-10T19:11:37.965 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:11:37 smithi191 ceph-mon[32884]: 4.11 scrub starts 2024-09-10T19:11:37.966 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:11:37 smithi191 ceph-mon[32884]: 4.11 scrub ok 2024-09-10T19:11:37.966 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:11:37 smithi191 ceph-mon[32884]: 4.15 scrub starts 2024-09-10T19:11:37.966 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:11:37 smithi191 ceph-mon[32884]: 4.15 scrub ok 2024-09-10T19:11:37.966 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:11:37 smithi191 ceph-mon[32884]: from='mgr.14556 ' entity='mgr.y' 2024-09-10T19:11:37.966 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:11:37 smithi191 ceph-mon[32884]: from='mgr.14556 ' entity='mgr.y' 2024-09-10T19:11:37.966 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:11:37 smithi191 ceph-mon[32884]: from='mgr.14556 ' entity='mgr.y' 2024-09-10T19:11:37.966 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:11:37 smithi191 ceph-mon[32884]: from='mgr.14556 ' entity='mgr.y' 2024-09-10T19:11:37.966 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:11:37 smithi191 ceph-mon[32884]: from='mgr.14556 ' entity='mgr.y' 2024-09-10T19:11:37.966 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:11:37 smithi191 ceph-mon[32884]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T19:11:38.897 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-09-10T19:11:38.897 INFO:teuthology.orchestra.run.smithi183.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-09-10T19:11:38.897 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:11:38 smithi183 ceph-mon[29736]: pgmap v37: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-10T19:11:38.897 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:11:38 smithi183 ceph-mon[29736]: 6.8 scrub starts 2024-09-10T19:11:38.897 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:11:38 smithi183 ceph-mon[29736]: 6.8 scrub ok 2024-09-10T19:11:38.897 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:11:38 smithi183 ceph-mon[29736]: from='mgr.14556 ' entity='mgr.y' 2024-09-10T19:11:38.898 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:11:38 smithi183 ceph-mon[41562]: pgmap v37: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-10T19:11:38.898 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:11:38 smithi183 ceph-mon[41562]: 6.8 scrub starts 2024-09-10T19:11:38.898 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:11:38 smithi183 ceph-mon[41562]: 6.8 scrub ok 2024-09-10T19:11:38.898 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:11:38 smithi183 ceph-mon[41562]: from='mgr.14556 ' entity='mgr.y' 2024-09-10T19:11:38.965 INFO:journalctl@ceph.iscsi.iscsi.a.smithi191.stdout:Sep 10 19:11:38 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-iscsi-iscsi-a[61816]: debug there is no tcmu-runner data available 2024-09-10T19:11:38.965 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:11:38 smithi191 ceph-mon[32884]: pgmap v37: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-10T19:11:38.965 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:11:38 smithi191 ceph-mon[32884]: 6.8 scrub starts 2024-09-10T19:11:38.966 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:11:38 smithi191 ceph-mon[32884]: 6.8 scrub ok 2024-09-10T19:11:38.966 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:11:38 smithi191 ceph-mon[32884]: from='mgr.14556 ' entity='mgr.y' 2024-09-10T19:11:39.737 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy done 2024-09-10T19:11:39.737 INFO:tasks.cephadm:Setup complete, yielding 2024-09-10T19:11:39.738 INFO:teuthology.run_tasks:Running task thrashosds... 2024-09-10T19:11:39.749 INFO:tasks.thrashosds:config is {'chance_pgnum_grow': 1, 'chance_pgnum_shrink': 1, 'chance_pgpnum_fix': 1, 'timeout': 1200, 'sighup_delay': 0.1, 'optrack_toggle_delay': 2.0, 'dump_ops_enable': 'true', 'noscrub_toggle_delay': 2.0, 'random_eio': 0.0} 2024-09-10T19:11:39.749 INFO:tasks.thrashosds:overrides is {'admin_socket': {'branch': 'wip-guits-quincy-2024-09-09-1810-quincy'}, 'ceph': {'conf': {'client.0': {'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'global': {'ms type': 'async'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'paxos service trim min': 10}, 'osd': {'debug ms': 1, 'debug osd': 20, 'osd debug reject backfill probability': 0.3, 'osd delete sleep': 1, 'osd max backfills': 3, 'osd scrub max interval': 120, 'osd scrub min interval': 60, 'osd shutdown pgref assert': True, 'osd snap trim sleep': 2}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'but it is still running', 'objects unfound and apparently lost', '\\(POOL_APP_NOT_ENABLED\\)'], 'sha1': '5be57395ea235535910512ae73ba8e7ff25fe588'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'cephadm': {'cephadm_mode': 'root'}, 'install': {'ceph': {'flavor': 'default', 'sha1': '5be57395ea235535910512ae73ba8e7ff25fe588'}}, 'selinux': {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']}, 'workunit': {'branch': 'quincy-orch-centos-9-teuth', 'sha1': 'fad5356a94e6044ce1062d748daa5a0f2ed53a7d'}} 2024-09-10T19:11:39.749 INFO:tasks.thrashosds:config is {'chance_pgnum_grow': 1, 'chance_pgnum_shrink': 1, 'chance_pgpnum_fix': 1, 'timeout': 1200, 'sighup_delay': 0.1, 'optrack_toggle_delay': 2.0, 'dump_ops_enable': 'true', 'noscrub_toggle_delay': 2.0, 'random_eio': 0.0} 2024-09-10T19:11:39.749 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph config set mgr mgr_debug_aggressive_pg_num_changes true 2024-09-10T19:11:39.931 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:11:39 smithi183 ceph-mon[41562]: from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:11:39.931 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:11:39 smithi183 ceph-mon[41562]: from='client.? 172.21.15.183:0/3367172146' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-09-10T19:11:39.931 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:11:39 smithi183 ceph-mon[41562]: 4.6 scrub starts 2024-09-10T19:11:39.931 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:11:39 smithi183 ceph-mon[41562]: 4.6 scrub ok 2024-09-10T19:11:39.931 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:11:39 smithi183 ceph-mon[29736]: from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:11:39.932 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:11:39 smithi183 ceph-mon[29736]: from='client.? 172.21.15.183:0/3367172146' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-09-10T19:11:39.932 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:11:39 smithi183 ceph-mon[29736]: 4.6 scrub starts 2024-09-10T19:11:39.932 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:11:39 smithi183 ceph-mon[29736]: 4.6 scrub ok 2024-09-10T19:11:39.965 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:11:39 smithi191 ceph-mon[32884]: from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:11:39.966 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:11:39 smithi191 ceph-mon[32884]: from='client.? 172.21.15.183:0/3367172146' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-09-10T19:11:39.966 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:11:39 smithi191 ceph-mon[32884]: 4.6 scrub starts 2024-09-10T19:11:39.966 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:11:39 smithi191 ceph-mon[32884]: 4.6 scrub ok 2024-09-10T19:11:40.489 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:11:40.900 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:11:40 smithi183 ceph-mon[29736]: pgmap v38: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:11:40.900 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:11:40 smithi183 ceph-mon[29736]: 4.5 deep-scrub starts 2024-09-10T19:11:40.900 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:11:40 smithi183 ceph-mon[29736]: 4.5 deep-scrub ok 2024-09-10T19:11:40.901 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:11:40 smithi183 ceph-mon[41562]: pgmap v38: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:11:40.901 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:11:40 smithi183 ceph-mon[41562]: 4.5 deep-scrub starts 2024-09-10T19:11:40.901 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:11:40 smithi183 ceph-mon[41562]: 4.5 deep-scrub ok 2024-09-10T19:11:40.965 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:11:40 smithi191 ceph-mon[32884]: pgmap v38: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:11:40.965 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:11:40 smithi191 ceph-mon[32884]: 4.5 deep-scrub starts 2024-09-10T19:11:40.965 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:11:40 smithi191 ceph-mon[32884]: 4.5 deep-scrub ok 2024-09-10T19:11:41.931 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:11:41 smithi183 ceph-mon[29736]: 2.c scrub starts 2024-09-10T19:11:41.931 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:11:41 smithi183 ceph-mon[29736]: 2.c scrub ok 2024-09-10T19:11:41.932 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:11:41 smithi183 ceph-mon[29736]: 4.1f deep-scrub starts 2024-09-10T19:11:41.932 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:11:41 smithi183 ceph-mon[29736]: 4.1f deep-scrub ok 2024-09-10T19:11:41.932 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:11:41 smithi183 ceph-mon[29736]: 6.3 deep-scrub starts 2024-09-10T19:11:41.932 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:11:41 smithi183 ceph-mon[29736]: 6.3 deep-scrub ok 2024-09-10T19:11:41.932 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:11:41 smithi183 ceph-mon[29736]: 2.2 scrub starts 2024-09-10T19:11:41.932 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:11:41 smithi183 ceph-mon[29736]: 2.2 scrub ok 2024-09-10T19:11:41.933 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:11:41 smithi183 ceph-mon[41562]: 2.c scrub starts 2024-09-10T19:11:41.933 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:11:41 smithi183 ceph-mon[41562]: 2.c scrub ok 2024-09-10T19:11:41.933 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:11:41 smithi183 ceph-mon[41562]: 4.1f deep-scrub starts 2024-09-10T19:11:41.933 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:11:41 smithi183 ceph-mon[41562]: 4.1f deep-scrub ok 2024-09-10T19:11:41.933 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:11:41 smithi183 ceph-mon[41562]: 6.3 deep-scrub starts 2024-09-10T19:11:41.933 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:11:41 smithi183 ceph-mon[41562]: 6.3 deep-scrub ok 2024-09-10T19:11:41.933 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:11:41 smithi183 ceph-mon[41562]: 2.2 scrub starts 2024-09-10T19:11:41.934 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:11:41 smithi183 ceph-mon[41562]: 2.2 scrub ok 2024-09-10T19:11:41.965 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:11:41 smithi191 ceph-mon[32884]: 2.c scrub starts 2024-09-10T19:11:41.966 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:11:41 smithi191 ceph-mon[32884]: 2.c scrub ok 2024-09-10T19:11:41.966 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:11:41 smithi191 ceph-mon[32884]: 4.1f deep-scrub starts 2024-09-10T19:11:41.966 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:11:41 smithi191 ceph-mon[32884]: 4.1f deep-scrub ok 2024-09-10T19:11:41.966 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:11:41 smithi191 ceph-mon[32884]: 6.3 deep-scrub starts 2024-09-10T19:11:41.966 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:11:41 smithi191 ceph-mon[32884]: 6.3 deep-scrub ok 2024-09-10T19:11:41.966 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:11:41 smithi191 ceph-mon[32884]: 2.2 scrub starts 2024-09-10T19:11:41.966 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:11:41 smithi191 ceph-mon[32884]: 2.2 scrub ok 2024-09-10T19:11:42.681 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Sep 10 19:11:42 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-y[29989]: [10/Sep/2024:19:11:42] ENGINE Bus STOPPING 2024-09-10T19:11:42.681 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Sep 10 19:11:42 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-y[29989]: [10/Sep/2024:19:11:42] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-09-10T19:11:42.681 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Sep 10 19:11:42 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-y[29989]: [10/Sep/2024:19:11:42] ENGINE Bus STOPPED 2024-09-10T19:11:42.681 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Sep 10 19:11:42 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-y[29989]: [10/Sep/2024:19:11:42] ENGINE Bus STARTING 2024-09-10T19:11:42.682 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Sep 10 19:11:42 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-y[29989]: [10/Sep/2024:19:11:42] ENGINE Serving on http://:::9283 2024-09-10T19:11:42.682 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Sep 10 19:11:42 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-y[29989]: [10/Sep/2024:19:11:42] ENGINE Bus STARTED 2024-09-10T19:11:43.114 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:11:42 smithi183 ceph-mon[29736]: 2.10 deep-scrub starts 2024-09-10T19:11:43.114 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:11:42 smithi183 ceph-mon[29736]: 2.10 deep-scrub ok 2024-09-10T19:11:43.114 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:11:42 smithi183 ceph-mon[29736]: pgmap v39: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-10T19:11:43.115 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:11:42 smithi183 ceph-mon[29736]: 4.14 deep-scrub starts 2024-09-10T19:11:43.115 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:11:42 smithi183 ceph-mon[29736]: 4.14 deep-scrub ok 2024-09-10T19:11:43.115 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:11:42 smithi183 ceph-mon[29736]: from='mgr.14556 ' entity='mgr.y' 2024-09-10T19:11:43.115 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:11:42 smithi183 ceph-mon[29736]: from='mgr.14556 ' entity='mgr.y' 2024-09-10T19:11:43.115 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:11:42 smithi183 ceph-mon[29736]: from='client.? 172.21.15.183:0/3827737139' entity='client.admin' 2024-09-10T19:11:43.115 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:11:42 smithi183 ceph-mon[41562]: 2.10 deep-scrub starts 2024-09-10T19:11:43.115 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:11:42 smithi183 ceph-mon[41562]: 2.10 deep-scrub ok 2024-09-10T19:11:43.115 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:11:42 smithi183 ceph-mon[41562]: pgmap v39: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-10T19:11:43.115 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:11:42 smithi183 ceph-mon[41562]: 4.14 deep-scrub starts 2024-09-10T19:11:43.115 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:11:42 smithi183 ceph-mon[41562]: 4.14 deep-scrub ok 2024-09-10T19:11:43.116 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:11:42 smithi183 ceph-mon[41562]: from='mgr.14556 ' entity='mgr.y' 2024-09-10T19:11:43.116 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:11:42 smithi183 ceph-mon[41562]: from='mgr.14556 ' entity='mgr.y' 2024-09-10T19:11:43.116 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:11:42 smithi183 ceph-mon[41562]: from='client.? 172.21.15.183:0/3827737139' entity='client.admin' 2024-09-10T19:11:43.138 INFO:tasks.thrashosds:Beginning thrashosds... 2024-09-10T19:11:43.138 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-09-10T19:11:43.138 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph pg dump --format=json 2024-09-10T19:11:43.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:11:42 smithi191 ceph-mon[32884]: 2.10 deep-scrub starts 2024-09-10T19:11:43.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:11:42 smithi191 ceph-mon[32884]: 2.10 deep-scrub ok 2024-09-10T19:11:43.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:11:42 smithi191 ceph-mon[32884]: pgmap v39: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-10T19:11:43.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:11:42 smithi191 ceph-mon[32884]: 4.14 deep-scrub starts 2024-09-10T19:11:43.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:11:42 smithi191 ceph-mon[32884]: 4.14 deep-scrub ok 2024-09-10T19:11:43.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:11:42 smithi191 ceph-mon[32884]: from='mgr.14556 ' entity='mgr.y' 2024-09-10T19:11:43.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:11:42 smithi191 ceph-mon[32884]: from='mgr.14556 ' entity='mgr.y' 2024-09-10T19:11:43.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:11:42 smithi191 ceph-mon[32884]: from='client.? 172.21.15.183:0/3827737139' entity='client.admin' 2024-09-10T19:11:43.575 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:11:43.977 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:11:43 smithi183 ceph-mon[29736]: 2.6 scrub starts 2024-09-10T19:11:43.978 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:11:43 smithi183 ceph-mon[29736]: 2.6 scrub ok 2024-09-10T19:11:43.978 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:11:43 smithi183 ceph-mon[29736]: 5.a scrub starts 2024-09-10T19:11:43.978 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:11:43 smithi183 ceph-mon[29736]: 5.a scrub ok 2024-09-10T19:11:43.978 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:11:43 smithi183 ceph-mon[29736]: 6.b scrub starts 2024-09-10T19:11:43.978 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:11:43 smithi183 ceph-mon[29736]: 6.b scrub ok 2024-09-10T19:11:43.978 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:11:43 smithi183 ceph-mon[29736]: 4.a scrub starts 2024-09-10T19:11:43.978 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:11:43 smithi183 ceph-mon[29736]: 4.a scrub ok 2024-09-10T19:11:43.978 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:11:43 smithi183 ceph-mon[29736]: 5.1a scrub starts 2024-09-10T19:11:43.978 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:11:43 smithi183 ceph-mon[29736]: 5.1a scrub ok 2024-09-10T19:11:43.978 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:11:43 smithi183 ceph-mon[29736]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1f", "id": [7, 2]}]: dispatch 2024-09-10T19:11:43.978 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:11:43 smithi183 ceph-mon[29736]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.2", "id": [1, 2]}]: dispatch 2024-09-10T19:11:43.978 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:11:43 smithi183 ceph-mon[29736]: from='mgr.14556 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1f", "id": [7, 2]}]: dispatch 2024-09-10T19:11:43.978 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:11:43 smithi183 ceph-mon[29736]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1b", "id": [1, 2]}]: dispatch 2024-09-10T19:11:43.979 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:11:43 smithi183 ceph-mon[29736]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1f", "id": [1, 2]}]: dispatch 2024-09-10T19:11:43.979 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:11:43 smithi183 ceph-mon[29736]: from='mgr.14556 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.2", "id": [1, 2]}]: dispatch 2024-09-10T19:11:43.979 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:11:43 smithi183 ceph-mon[29736]: from='mgr.14556 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1b", "id": [1, 2]}]: dispatch 2024-09-10T19:11:43.979 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:11:43 smithi183 ceph-mon[29736]: from='mgr.14556 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1f", "id": [1, 2]}]: dispatch 2024-09-10T19:11:43.979 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:11:43 smithi183 ceph-mon[29736]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:11:43.979 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:11:43 smithi183 ceph-mon[41562]: 2.6 scrub starts 2024-09-10T19:11:43.979 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:11:43 smithi183 ceph-mon[41562]: 2.6 scrub ok 2024-09-10T19:11:43.979 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:11:43 smithi183 ceph-mon[41562]: 5.a scrub starts 2024-09-10T19:11:43.979 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:11:43 smithi183 ceph-mon[41562]: 5.a scrub ok 2024-09-10T19:11:43.979 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:11:43 smithi183 ceph-mon[41562]: 6.b scrub starts 2024-09-10T19:11:43.980 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:11:43 smithi183 ceph-mon[41562]: 6.b scrub ok 2024-09-10T19:11:43.980 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:11:43 smithi183 ceph-mon[41562]: 4.a scrub starts 2024-09-10T19:11:43.980 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:11:43 smithi183 ceph-mon[41562]: 4.a scrub ok 2024-09-10T19:11:43.980 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:11:43 smithi183 ceph-mon[41562]: 5.1a scrub starts 2024-09-10T19:11:43.980 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:11:43 smithi183 ceph-mon[41562]: 5.1a scrub ok 2024-09-10T19:11:43.980 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:11:43 smithi183 ceph-mon[41562]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1f", "id": [7, 2]}]: dispatch 2024-09-10T19:11:43.980 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:11:43 smithi183 ceph-mon[41562]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.2", "id": [1, 2]}]: dispatch 2024-09-10T19:11:43.980 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:11:43 smithi183 ceph-mon[41562]: from='mgr.14556 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1f", "id": [7, 2]}]: dispatch 2024-09-10T19:11:43.980 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:11:43 smithi183 ceph-mon[41562]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1b", "id": [1, 2]}]: dispatch 2024-09-10T19:11:43.980 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:11:43 smithi183 ceph-mon[41562]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1f", "id": [1, 2]}]: dispatch 2024-09-10T19:11:43.981 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:11:43 smithi183 ceph-mon[41562]: from='mgr.14556 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.2", "id": [1, 2]}]: dispatch 2024-09-10T19:11:43.981 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:11:43 smithi183 ceph-mon[41562]: from='mgr.14556 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1b", "id": [1, 2]}]: dispatch 2024-09-10T19:11:43.981 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:11:43 smithi183 ceph-mon[41562]: from='mgr.14556 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1f", "id": [1, 2]}]: dispatch 2024-09-10T19:11:43.981 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:11:43 smithi183 ceph-mon[41562]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:11:44.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:11:43 smithi191 ceph-mon[32884]: 2.6 scrub starts 2024-09-10T19:11:44.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:11:43 smithi191 ceph-mon[32884]: 2.6 scrub ok 2024-09-10T19:11:44.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:11:43 smithi191 ceph-mon[32884]: 5.a scrub starts 2024-09-10T19:11:44.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:11:43 smithi191 ceph-mon[32884]: 5.a scrub ok 2024-09-10T19:11:44.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:11:43 smithi191 ceph-mon[32884]: 6.b scrub starts 2024-09-10T19:11:44.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:11:43 smithi191 ceph-mon[32884]: 6.b scrub ok 2024-09-10T19:11:44.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:11:43 smithi191 ceph-mon[32884]: 4.a scrub starts 2024-09-10T19:11:44.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:11:43 smithi191 ceph-mon[32884]: 4.a scrub ok 2024-09-10T19:11:44.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:11:43 smithi191 ceph-mon[32884]: 5.1a scrub starts 2024-09-10T19:11:44.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:11:43 smithi191 ceph-mon[32884]: 5.1a scrub ok 2024-09-10T19:11:44.217 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:11:43 smithi191 ceph-mon[32884]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1f", "id": [7, 2]}]: dispatch 2024-09-10T19:11:44.217 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:11:43 smithi191 ceph-mon[32884]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.2", "id": [1, 2]}]: dispatch 2024-09-10T19:11:44.217 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:11:43 smithi191 ceph-mon[32884]: from='mgr.14556 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1f", "id": [7, 2]}]: dispatch 2024-09-10T19:11:44.217 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:11:43 smithi191 ceph-mon[32884]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1b", "id": [1, 2]}]: dispatch 2024-09-10T19:11:44.217 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:11:43 smithi191 ceph-mon[32884]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1f", "id": [1, 2]}]: dispatch 2024-09-10T19:11:44.217 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:11:43 smithi191 ceph-mon[32884]: from='mgr.14556 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.2", "id": [1, 2]}]: dispatch 2024-09-10T19:11:44.217 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:11:43 smithi191 ceph-mon[32884]: from='mgr.14556 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1b", "id": [1, 2]}]: dispatch 2024-09-10T19:11:44.217 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:11:43 smithi191 ceph-mon[32884]: from='mgr.14556 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1f", "id": [1, 2]}]: dispatch 2024-09-10T19:11:44.217 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:11:43 smithi191 ceph-mon[32884]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:11:45.102 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:11:44 smithi183 ceph-mon[29736]: 6.17 scrub starts 2024-09-10T19:11:45.102 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:11:44 smithi183 ceph-mon[29736]: 6.17 scrub ok 2024-09-10T19:11:45.103 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:11:44 smithi183 ceph-mon[29736]: 5.1d scrub starts 2024-09-10T19:11:45.103 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:11:44 smithi183 ceph-mon[29736]: 5.1d scrub ok 2024-09-10T19:11:45.103 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:11:44 smithi183 ceph-mon[29736]: pgmap v40: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-10T19:11:45.103 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:11:44 smithi183 ceph-mon[29736]: from='mgr.14556 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1f", "id": [7, 2]}]': finished 2024-09-10T19:11:45.103 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:11:44 smithi183 ceph-mon[29736]: from='mgr.14556 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.2", "id": [1, 2]}]': finished 2024-09-10T19:11:45.103 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:11:44 smithi183 ceph-mon[29736]: from='mgr.14556 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1b", "id": [1, 2]}]': finished 2024-09-10T19:11:45.103 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:11:44 smithi183 ceph-mon[29736]: from='mgr.14556 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1f", "id": [1, 2]}]': finished 2024-09-10T19:11:45.103 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:11:44 smithi183 ceph-mon[29736]: osdmap e81: 8 total, 8 up, 8 in 2024-09-10T19:11:45.103 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:11:44 smithi183 ceph-mon[29736]: 6.0 scrub starts 2024-09-10T19:11:45.103 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:11:44 smithi183 ceph-mon[29736]: 6.0 scrub ok 2024-09-10T19:11:45.103 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:11:44 smithi183 ceph-mon[29736]: from='mgr.14556 ' entity='mgr.y' 2024-09-10T19:11:45.103 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:11:44 smithi183 ceph-mon[29736]: from='mgr.14556 ' entity='mgr.y' 2024-09-10T19:11:45.103 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:11:44 smithi183 ceph-mon[29736]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T19:11:45.103 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:11:44 smithi183 ceph-mon[29736]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T19:11:45.103 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:11:44 smithi183 ceph-mon[29736]: from='mgr.14556 ' entity='mgr.y' 2024-09-10T19:11:45.104 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:11:44 smithi183 ceph-mon[41562]: 6.17 scrub starts 2024-09-10T19:11:45.104 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:11:44 smithi183 ceph-mon[41562]: 6.17 scrub ok 2024-09-10T19:11:45.104 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:11:44 smithi183 ceph-mon[41562]: 5.1d scrub starts 2024-09-10T19:11:45.104 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:11:44 smithi183 ceph-mon[41562]: 5.1d scrub ok 2024-09-10T19:11:45.104 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:11:44 smithi183 ceph-mon[41562]: pgmap v40: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-10T19:11:45.104 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:11:44 smithi183 ceph-mon[41562]: from='mgr.14556 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1f", "id": [7, 2]}]': finished 2024-09-10T19:11:45.104 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:11:44 smithi183 ceph-mon[41562]: from='mgr.14556 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.2", "id": [1, 2]}]': finished 2024-09-10T19:11:45.104 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:11:44 smithi183 ceph-mon[41562]: from='mgr.14556 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1b", "id": [1, 2]}]': finished 2024-09-10T19:11:45.104 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:11:44 smithi183 ceph-mon[41562]: from='mgr.14556 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1f", "id": [1, 2]}]': finished 2024-09-10T19:11:45.104 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:11:44 smithi183 ceph-mon[41562]: osdmap e81: 8 total, 8 up, 8 in 2024-09-10T19:11:45.104 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:11:44 smithi183 ceph-mon[41562]: 6.0 scrub starts 2024-09-10T19:11:45.104 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:11:44 smithi183 ceph-mon[41562]: 6.0 scrub ok 2024-09-10T19:11:45.105 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:11:44 smithi183 ceph-mon[41562]: from='mgr.14556 ' entity='mgr.y' 2024-09-10T19:11:45.105 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:11:44 smithi183 ceph-mon[41562]: from='mgr.14556 ' entity='mgr.y' 2024-09-10T19:11:45.105 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:11:44 smithi183 ceph-mon[41562]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T19:11:45.105 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:11:44 smithi183 ceph-mon[41562]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T19:11:45.105 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:11:44 smithi183 ceph-mon[41562]: from='mgr.14556 ' entity='mgr.y' 2024-09-10T19:11:45.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:11:44 smithi191 ceph-mon[32884]: 6.17 scrub starts 2024-09-10T19:11:45.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:11:44 smithi191 ceph-mon[32884]: 6.17 scrub ok 2024-09-10T19:11:45.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:11:44 smithi191 ceph-mon[32884]: 5.1d scrub starts 2024-09-10T19:11:45.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:11:44 smithi191 ceph-mon[32884]: 5.1d scrub ok 2024-09-10T19:11:45.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:11:44 smithi191 ceph-mon[32884]: pgmap v40: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-10T19:11:45.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:11:44 smithi191 ceph-mon[32884]: from='mgr.14556 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1f", "id": [7, 2]}]': finished 2024-09-10T19:11:45.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:11:44 smithi191 ceph-mon[32884]: from='mgr.14556 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.2", "id": [1, 2]}]': finished 2024-09-10T19:11:45.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:11:44 smithi191 ceph-mon[32884]: from='mgr.14556 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1b", "id": [1, 2]}]': finished 2024-09-10T19:11:45.217 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:11:44 smithi191 ceph-mon[32884]: from='mgr.14556 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1f", "id": [1, 2]}]': finished 2024-09-10T19:11:45.217 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:11:44 smithi191 ceph-mon[32884]: osdmap e81: 8 total, 8 up, 8 in 2024-09-10T19:11:45.217 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:11:44 smithi191 ceph-mon[32884]: 6.0 scrub starts 2024-09-10T19:11:45.217 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:11:44 smithi191 ceph-mon[32884]: 6.0 scrub ok 2024-09-10T19:11:45.217 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:11:44 smithi191 ceph-mon[32884]: from='mgr.14556 ' entity='mgr.y' 2024-09-10T19:11:45.217 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:11:44 smithi191 ceph-mon[32884]: from='mgr.14556 ' entity='mgr.y' 2024-09-10T19:11:45.217 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:11:44 smithi191 ceph-mon[32884]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T19:11:45.217 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:11:44 smithi191 ceph-mon[32884]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T19:11:45.217 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:11:44 smithi191 ceph-mon[32884]: from='mgr.14556 ' entity='mgr.y' 2024-09-10T19:11:46.004 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:11:45 smithi183 ceph-mon[29736]: 4.13 scrub starts 2024-09-10T19:11:46.004 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:11:45 smithi183 ceph-mon[29736]: 4.13 scrub ok 2024-09-10T19:11:46.004 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:11:45 smithi183 ceph-mon[29736]: 2.d deep-scrub starts 2024-09-10T19:11:46.005 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:11:45 smithi183 ceph-mon[29736]: 2.d deep-scrub ok 2024-09-10T19:11:46.005 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:11:45 smithi183 ceph-mon[29736]: osdmap e82: 8 total, 8 up, 8 in 2024-09-10T19:11:46.005 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:11:45 smithi183 ceph-mon[29736]: Reconfiguring alertmanager.a (dependencies changed)... 2024-09-10T19:11:46.005 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:11:45 smithi183 ceph-mon[29736]: Reconfiguring daemon alertmanager.a on smithi183 2024-09-10T19:11:46.005 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:11:45 smithi183 ceph-mon[29736]: 6.19 scrub starts 2024-09-10T19:11:46.005 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:11:45 smithi183 ceph-mon[29736]: 6.19 scrub ok 2024-09-10T19:11:46.006 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:11:45 smithi183 ceph-mon[41562]: 4.13 scrub starts 2024-09-10T19:11:46.006 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:11:45 smithi183 ceph-mon[41562]: 4.13 scrub ok 2024-09-10T19:11:46.006 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:11:45 smithi183 ceph-mon[41562]: 2.d deep-scrub starts 2024-09-10T19:11:46.006 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:11:45 smithi183 ceph-mon[41562]: 2.d deep-scrub ok 2024-09-10T19:11:46.006 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:11:45 smithi183 ceph-mon[41562]: osdmap e82: 8 total, 8 up, 8 in 2024-09-10T19:11:46.007 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:11:45 smithi183 ceph-mon[41562]: Reconfiguring alertmanager.a (dependencies changed)... 2024-09-10T19:11:46.007 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:11:45 smithi183 ceph-mon[41562]: Reconfiguring daemon alertmanager.a on smithi183 2024-09-10T19:11:46.007 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:11:45 smithi183 ceph-mon[41562]: 6.19 scrub starts 2024-09-10T19:11:46.007 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:11:45 smithi183 ceph-mon[41562]: 6.19 scrub ok 2024-09-10T19:11:46.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:11:45 smithi191 ceph-mon[32884]: 4.13 scrub starts 2024-09-10T19:11:46.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:11:45 smithi191 ceph-mon[32884]: 4.13 scrub ok 2024-09-10T19:11:46.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:11:45 smithi191 ceph-mon[32884]: 2.d deep-scrub starts 2024-09-10T19:11:46.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:11:45 smithi191 ceph-mon[32884]: 2.d deep-scrub ok 2024-09-10T19:11:46.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:11:45 smithi191 ceph-mon[32884]: osdmap e82: 8 total, 8 up, 8 in 2024-09-10T19:11:46.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:11:45 smithi191 ceph-mon[32884]: Reconfiguring alertmanager.a (dependencies changed)... 2024-09-10T19:11:46.217 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:11:45 smithi191 ceph-mon[32884]: Reconfiguring daemon alertmanager.a on smithi183 2024-09-10T19:11:46.217 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:11:45 smithi191 ceph-mon[32884]: 6.19 scrub starts 2024-09-10T19:11:46.217 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:11:45 smithi191 ceph-mon[32884]: 6.19 scrub ok 2024-09-10T19:11:47.022 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:11:46 smithi183 ceph-mon[29736]: 4.9 scrub starts 2024-09-10T19:11:47.022 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:11:46 smithi183 ceph-mon[29736]: 4.9 scrub ok 2024-09-10T19:11:47.023 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:11:46 smithi183 ceph-mon[29736]: pgmap v43: 132 pgs: 4 peering, 128 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:11:47.023 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:11:46 smithi183 ceph-mon[41562]: 4.9 scrub starts 2024-09-10T19:11:47.023 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:11:46 smithi183 ceph-mon[41562]: 4.9 scrub ok 2024-09-10T19:11:47.023 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:11:46 smithi183 ceph-mon[41562]: pgmap v43: 132 pgs: 4 peering, 128 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:11:47.075 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-09-10T19:11:47.077 INFO:teuthology.orchestra.run.smithi183.stderr:dumped all 2024-09-10T19:11:47.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:11:46 smithi191 ceph-mon[32884]: 4.9 scrub starts 2024-09-10T19:11:47.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:11:46 smithi191 ceph-mon[32884]: 4.9 scrub ok 2024-09-10T19:11:47.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:11:46 smithi191 ceph-mon[32884]: pgmap v43: 132 pgs: 4 peering, 128 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:11:47.715 INFO:journalctl@ceph.mgr.x.smithi191.stdout:Sep 10 19:11:47 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-x[34967]: ::ffff:172.21.15.191 - - [10/Sep/2024:19:11:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T19:11:47.931 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:11:47 smithi183 ceph-mon[29736]: 5.8 deep-scrub starts 2024-09-10T19:11:47.931 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:11:47 smithi183 ceph-mon[29736]: 5.8 deep-scrub ok 2024-09-10T19:11:47.931 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:11:47 smithi183 ceph-mon[29736]: 6.13 scrub starts 2024-09-10T19:11:47.931 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:11:47 smithi183 ceph-mon[29736]: 6.13 scrub ok 2024-09-10T19:11:47.931 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:11:47 smithi183 ceph-mon[29736]: 2.1f scrub starts 2024-09-10T19:11:47.931 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:11:47 smithi183 ceph-mon[29736]: 2.1f scrub ok 2024-09-10T19:11:47.931 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:11:47 smithi183 ceph-mon[29736]: from='client.24626 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-10T19:11:47.932 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:11:47 smithi183 ceph-mon[41562]: 5.8 deep-scrub starts 2024-09-10T19:11:47.932 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:11:47 smithi183 ceph-mon[41562]: 5.8 deep-scrub ok 2024-09-10T19:11:47.932 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:11:47 smithi183 ceph-mon[41562]: 6.13 scrub starts 2024-09-10T19:11:47.932 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:11:47 smithi183 ceph-mon[41562]: 6.13 scrub ok 2024-09-10T19:11:47.932 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:11:47 smithi183 ceph-mon[41562]: 2.1f scrub starts 2024-09-10T19:11:47.932 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:11:47 smithi183 ceph-mon[41562]: 2.1f scrub ok 2024-09-10T19:11:47.932 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:11:47 smithi183 ceph-mon[41562]: from='client.24626 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-10T19:11:48.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:11:47 smithi191 ceph-mon[32884]: 5.8 deep-scrub starts 2024-09-10T19:11:48.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:11:47 smithi191 ceph-mon[32884]: 5.8 deep-scrub ok 2024-09-10T19:11:48.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:11:47 smithi191 ceph-mon[32884]: 6.13 scrub starts 2024-09-10T19:11:48.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:11:47 smithi191 ceph-mon[32884]: 6.13 scrub ok 2024-09-10T19:11:48.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:11:47 smithi191 ceph-mon[32884]: 2.1f scrub starts 2024-09-10T19:11:48.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:11:47 smithi191 ceph-mon[32884]: 2.1f scrub ok 2024-09-10T19:11:48.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:11:47 smithi191 ceph-mon[32884]: from='client.24626 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-10T19:11:49.063 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:11:48 smithi183 ceph-mon[29736]: pgmap v44: 132 pgs: 4 peering, 128 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:11:49.063 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:11:48 smithi183 ceph-mon[29736]: 2.14 scrub starts 2024-09-10T19:11:49.063 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:11:48 smithi183 ceph-mon[29736]: 2.14 scrub ok 2024-09-10T19:11:49.063 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:11:48 smithi183 ceph-mon[41562]: pgmap v44: 132 pgs: 4 peering, 128 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:11:49.063 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:11:48 smithi183 ceph-mon[41562]: 2.14 scrub starts 2024-09-10T19:11:49.063 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:11:48 smithi183 ceph-mon[41562]: 2.14 scrub ok 2024-09-10T19:11:49.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:11:48 smithi191 ceph-mon[32884]: pgmap v44: 132 pgs: 4 peering, 128 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:11:49.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:11:48 smithi191 ceph-mon[32884]: 2.14 scrub starts 2024-09-10T19:11:49.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:11:48 smithi191 ceph-mon[32884]: 2.14 scrub ok 2024-09-10T19:11:49.216 INFO:journalctl@ceph.iscsi.iscsi.a.smithi191.stdout:Sep 10 19:11:48 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-iscsi-iscsi-a[61816]: debug there is no tcmu-runner data available 2024-09-10T19:11:49.431 INFO:journalctl@ceph.alertmanager.a.smithi183.stdout:Sep 10 19:11:49 smithi183 systemd[1]: Stopping Ceph alertmanager.a for 320f6df8-6fa7-11ef-bcea-c7b262605968... 2024-09-10T19:11:49.563 INFO:teuthology.orchestra.run.smithi183.stdout:{"pg_ready":true,"pg_map":{"version":43,"stamp":"2024-09-10T19:11:45.604877+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596188,"num_objects":191,"num_object_clones":0,"num_object_copies":573,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":191,"num_whiteouts":0,"num_read":927,"num_read_kb":789,"num_write":533,"num_write_kb":1671,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":574,"ondisk_log_size":574,"up":396,"acting":396,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":396,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":2387704,"kb_used_data":7000,"kb_used_omap":13,"kb_used_meta":2378738,"kb_avail":747573512,"statfs":{"total":767960285184,"available":765515276288,"internally_reserved":0,"allocated":7168000,"data_stored":4376520,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13632,"internal_metadata":2435828416},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":-5,"num_object_clones":0,"num_object_copies":-15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-5,"num_whiteouts":0,"num_read":-5,"num_read_kb":0,"num_write":-10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001793"},"pg_stats":[{"pgid":"6.1b","version":"0'0","reported_seq":25,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-10T19:11:23.207586+0000","last_change":"2024-09-10T19:10:14.056737+0000","last_active":"2024-09-10T19:11:23.207586+0000","last_peered":"2024-09-10T19:11:23.207586+0000","last_clean":"2024-09-10T19:11:23.207586+0000","last_became_active":"2024-09-10T19:10:14.056187+0000","last_became_peered":"2024-09-10T19:10:14.056187+0000","last_unstale":"2024-09-10T19:11:23.207586+0000","last_undegraded":"2024-09-10T19:11:23.207586+0000","last_fullsized":"2024-09-10T19:11:23.207586+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:10:12.693598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":33,"reported_epoch":81,"state":"peering","last_fresh":"2024-09-10T19:11:43.735804+0000","last_change":"2024-09-10T19:11:43.736017+0000","last_active":"2024-09-10T19:11:43.735775+0000","last_peered":"2024-09-10T19:11:23.236643+0000","last_clean":"2024-09-10T19:11:23.236643+0000","last_became_active":"2024-09-10T19:10:07.688372+0000","last_became_peered":"2024-09-10T19:10:07.688372+0000","last_unstale":"2024-09-10T19:11:43.735804+0000","last_undegraded":"2024-09-10T19:11:43.735804+0000","last_fullsized":"2024-09-10T19:11:43.735804+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:10:06.664709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2,4,7],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.19","version":"66'15","reported_seq":54,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-10T19:11:23.207153+0000","last_change":"2024-09-10T19:10:09.688886+0000","last_active":"2024-09-10T19:11:23.207153+0000","last_peered":"2024-09-10T19:11:23.207153+0000","last_clean":"2024-09-10T19:11:23.207153+0000","last_became_active":"2024-09-10T19:10:09.688432+0000","last_became_peered":"2024-09-10T19:10:09.688432+0000","last_unstale":"2024-09-10T19:11:23.207153+0000","last_undegraded":"2024-09-10T19:11:23.207153+0000","last_fullsized":"2024-09-10T19:11:23.207153+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:10:08.671214+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":37,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-10T19:11:23.236252+0000","last_change":"2024-09-10T19:11:23.086288+0000","last_active":"2024-09-10T19:11:23.236252+0000","last_peered":"2024-09-10T19:11:23.236252+0000","last_clean":"2024-09-10T19:11:23.236252+0000","last_became_active":"2024-09-10T19:10:11.697523+0000","last_became_peered":"2024-09-10T19:10:11.697523+0000","last_unstale":"2024-09-10T19:11:23.236252+0000","last_undegraded":"2024-09-10T19:11:23.236252+0000","last_fullsized":"2024-09-10T19:11:23.236252+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:11:23.086191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:11:23.086191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:12:31.131189+0000","scrub_duration":0.0016087969999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"59'1","reported_seq":49,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-10T19:11:23.208417+0000","last_change":"2024-09-10T19:11:15.782493+0000","last_active":"2024-09-10T19:11:23.208417+0000","last_peered":"2024-09-10T19:11:23.208417+0000","last_clean":"2024-09-10T19:11:23.208417+0000","last_became_active":"2024-09-10T19:10:07.686143+0000","last_became_peered":"2024-09-10T19:10:07.686143+0000","last_unstale":"2024-09-10T19:11:23.208417+0000","last_undegraded":"2024-09-10T19:11:23.208417+0000","last_fullsized":"2024-09-10T19:11:23.208417+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'1","last_scrub_stamp":"2024-09-10T19:11:15.782393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:11:15.782393+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:12:21.959971+0000","scrub_duration":0.00183947,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":874,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"66'9","reported_seq":45,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-10T19:11:23.236329+0000","last_change":"2024-09-10T19:10:09.692950+0000","last_active":"2024-09-10T19:11:23.236329+0000","last_peered":"2024-09-10T19:11:23.236329+0000","last_clean":"2024-09-10T19:11:23.236329+0000","last_became_active":"2024-09-10T19:10:09.692681+0000","last_became_peered":"2024-09-10T19:10:09.692681+0000","last_unstale":"2024-09-10T19:11:23.236329+0000","last_undegraded":"2024-09-10T19:11:23.236329+0000","last_fullsized":"2024-09-10T19:11:23.236329+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:10:08.671214+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":27,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-10T19:11:23.236255+0000","last_change":"2024-09-10T19:10:11.702830+0000","last_active":"2024-09-10T19:11:23.236255+0000","last_peered":"2024-09-10T19:11:23.236255+0000","last_clean":"2024-09-10T19:11:23.236255+0000","last_became_active":"2024-09-10T19:10:11.702546+0000","last_became_peered":"2024-09-10T19:10:11.702546+0000","last_unstale":"2024-09-10T19:11:23.236255+0000","last_undegraded":"2024-09-10T19:11:23.236255+0000","last_fullsized":"2024-09-10T19:11:23.236255+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:10:10.676876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":35,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-10T19:11:32.056391+0000","last_change":"2024-09-10T19:11:32.056320+0000","last_active":"2024-09-10T19:11:32.056391+0000","last_peered":"2024-09-10T19:11:32.056391+0000","last_clean":"2024-09-10T19:11:32.056391+0000","last_became_active":"2024-09-10T19:10:13.716022+0000","last_became_peered":"2024-09-10T19:10:13.716022+0000","last_unstale":"2024-09-10T19:11:32.056391+0000","last_undegraded":"2024-09-10T19:11:32.056391+0000","last_fullsized":"2024-09-10T19:11:32.056391+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:11:32.056254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:11:32.056254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:12:39.600944+0000","scrub_duration":0.0013010680000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":41,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-10T19:11:23.238329+0000","last_change":"2024-09-10T19:11:19.275407+0000","last_active":"2024-09-10T19:11:23.238329+0000","last_peered":"2024-09-10T19:11:23.238329+0000","last_clean":"2024-09-10T19:11:23.238329+0000","last_became_active":"2024-09-10T19:10:07.690728+0000","last_became_peered":"2024-09-10T19:10:07.690728+0000","last_unstale":"2024-09-10T19:11:23.238329+0000","last_undegraded":"2024-09-10T19:11:23.238329+0000","last_fullsized":"2024-09-10T19:11:23.238329+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:11:19.275331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:11:19.275331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:12:30.648634+0000","scrub_duration":0.001355258,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1b","version":"66'5","reported_seq":51,"reported_epoch":81,"state":"peering","last_fresh":"2024-09-10T19:11:43.729822+0000","last_change":"2024-09-10T19:11:43.730212+0000","last_active":"2024-09-10T19:11:43.729785+0000","last_peered":"2024-09-10T19:11:24.101661+0000","last_clean":"2024-09-10T19:11:24.101661+0000","last_became_active":"2024-09-10T19:10:09.693690+0000","last_became_peered":"2024-09-10T19:10:09.693690+0000","last_unstale":"2024-09-10T19:11:43.729822+0000","last_undegraded":"2024-09-10T19:11:43.729822+0000","last_fullsized":"2024-09-10T19:11:43.729822+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'5","last_scrub_stamp":"2024-09-10T19:11:24.101551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:11:24.101551+0000","objects_scrubbed":2,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:12:29.369870+0000","scrub_duration":0.0019539829999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1,2,3],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":37,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-10T19:11:43.142879+0000","last_change":"2024-09-10T19:11:43.142813+0000","last_active":"2024-09-10T19:11:43.142879+0000","last_peered":"2024-09-10T19:11:43.142879+0000","last_clean":"2024-09-10T19:11:43.142879+0000","last_became_active":"2024-09-10T19:10:11.697088+0000","last_became_peered":"2024-09-10T19:10:11.697088+0000","last_unstale":"2024-09-10T19:11:43.142879+0000","last_undegraded":"2024-09-10T19:11:43.142879+0000","last_fullsized":"2024-09-10T19:11:43.142879+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:11:43.142734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:11:43.142734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:12:50.945873+0000","scrub_duration":0.0013612330000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":25,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-10T19:11:23.237151+0000","last_change":"2024-09-10T19:10:13.724478+0000","last_active":"2024-09-10T19:11:23.237151+0000","last_peered":"2024-09-10T19:11:23.237151+0000","last_clean":"2024-09-10T19:11:23.237151+0000","last_became_active":"2024-09-10T19:10:13.724255+0000","last_became_peered":"2024-09-10T19:10:13.724255+0000","last_unstale":"2024-09-10T19:11:23.237151+0000","last_undegraded":"2024-09-10T19:11:23.237151+0000","last_fullsized":"2024-09-10T19:11:23.237151+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:10:12.693598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":41,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-10T19:11:33.268414+0000","last_change":"2024-09-10T19:11:33.268356+0000","last_active":"2024-09-10T19:11:33.268414+0000","last_peered":"2024-09-10T19:11:33.268414+0000","last_clean":"2024-09-10T19:11:33.268414+0000","last_became_active":"2024-09-10T19:10:07.690174+0000","last_became_peered":"2024-09-10T19:10:07.690174+0000","last_unstale":"2024-09-10T19:11:33.268414+0000","last_undegraded":"2024-09-10T19:11:33.268414+0000","last_fullsized":"2024-09-10T19:11:33.268414+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:11:33.268282+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:11:33.268282+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:12:35.423155+0000","scrub_duration":0.0015798069999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1a","version":"66'9","reported_seq":55,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-10T19:11:31.036010+0000","last_change":"2024-09-10T19:11:31.035964+0000","last_active":"2024-09-10T19:11:31.036010+0000","last_peered":"2024-09-10T19:11:31.036010+0000","last_clean":"2024-09-10T19:11:31.036010+0000","last_became_active":"2024-09-10T19:10:09.688987+0000","last_became_peered":"2024-09-10T19:10:09.688987+0000","last_unstale":"2024-09-10T19:11:31.036010+0000","last_undegraded":"2024-09-10T19:11:31.036010+0000","last_fullsized":"2024-09-10T19:11:31.036010+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'9","last_scrub_stamp":"2024-09-10T19:11:31.035912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:11:31.035912+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:12:57.913355+0000","scrub_duration":0.001674916,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":37,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-10T19:11:27.317940+0000","last_change":"2024-09-10T19:11:27.317868+0000","last_active":"2024-09-10T19:11:27.317940+0000","last_peered":"2024-09-10T19:11:27.317940+0000","last_clean":"2024-09-10T19:11:27.317940+0000","last_became_active":"2024-09-10T19:10:11.701799+0000","last_became_peered":"2024-09-10T19:10:11.701799+0000","last_unstale":"2024-09-10T19:11:27.317940+0000","last_undegraded":"2024-09-10T19:11:27.317940+0000","last_fullsized":"2024-09-10T19:11:27.317940+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:11:27.317779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:11:27.317779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:12:44.343114+0000","scrub_duration":0.001509584,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":35,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-10T19:11:23.236049+0000","last_change":"2024-09-10T19:11:12.970227+0000","last_active":"2024-09-10T19:11:23.236049+0000","last_peered":"2024-09-10T19:11:23.236049+0000","last_clean":"2024-09-10T19:11:23.236049+0000","last_became_active":"2024-09-10T19:10:13.717310+0000","last_became_peered":"2024-09-10T19:10:13.717310+0000","last_unstale":"2024-09-10T19:11:23.236049+0000","last_undegraded":"2024-09-10T19:11:23.236049+0000","last_fullsized":"2024-09-10T19:11:23.236049+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:11:12.970173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:11:12.970173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:12:27.060515+0000","scrub_duration":0.001527219,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.d","version":"66'17","reported_seq":67,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-10T19:11:23.237470+0000","last_change":"2024-09-10T19:11:14.007973+0000","last_active":"2024-09-10T19:11:23.237470+0000","last_peered":"2024-09-10T19:11:23.237470+0000","last_clean":"2024-09-10T19:11:23.237470+0000","last_became_active":"2024-09-10T19:10:09.694016+0000","last_became_peered":"2024-09-10T19:10:09.694016+0000","last_unstale":"2024-09-10T19:11:23.237470+0000","last_undegraded":"2024-09-10T19:11:23.237470+0000","last_fullsized":"2024-09-10T19:11:23.237470+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'17","last_scrub_stamp":"2024-09-10T19:11:14.007902+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:11:14.007902+0000","objects_scrubbed":8,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:12:21.164915+0000","scrub_duration":0.0024520879999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":41,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-10T19:11:23.238251+0000","last_change":"2024-09-10T19:11:22.280067+0000","last_active":"2024-09-10T19:11:23.238251+0000","last_peered":"2024-09-10T19:11:23.238251+0000","last_clean":"2024-09-10T19:11:23.238251+0000","last_became_active":"2024-09-10T19:10:07.691119+0000","last_became_peered":"2024-09-10T19:10:07.691119+0000","last_unstale":"2024-09-10T19:11:23.238251+0000","last_undegraded":"2024-09-10T19:11:23.238251+0000","last_fullsized":"2024-09-10T19:11:23.238251+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:11:22.280002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:11:22.280002+0000","last_clean_scrub_stamp":"2024-09-10T19:11:22.280002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:12:27.848752+0000","scrub_duration":0.0013828040000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":27,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-10T19:11:23.236896+0000","last_change":"2024-09-10T19:10:11.695393+0000","last_active":"2024-09-10T19:11:23.236896+0000","last_peered":"2024-09-10T19:11:23.236896+0000","last_clean":"2024-09-10T19:11:23.236896+0000","last_became_active":"2024-09-10T19:10:11.695141+0000","last_became_peered":"2024-09-10T19:10:11.695141+0000","last_unstale":"2024-09-10T19:11:23.236896+0000","last_undegraded":"2024-09-10T19:11:23.236896+0000","last_fullsized":"2024-09-10T19:11:23.236896+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:10:10.676876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":35,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-10T19:11:24.145092+0000","last_change":"2024-09-10T19:11:24.145056+0000","last_active":"2024-09-10T19:11:24.145092+0000","last_peered":"2024-09-10T19:11:24.145092+0000","last_clean":"2024-09-10T19:11:24.145092+0000","last_became_active":"2024-09-10T19:10:13.722894+0000","last_became_peered":"2024-09-10T19:10:13.722894+0000","last_unstale":"2024-09-10T19:11:24.145092+0000","last_undegraded":"2024-09-10T19:11:24.145092+0000","last_fullsized":"2024-09-10T19:11:24.145092+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:11:24.145003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:11:24.145003+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:12:26.405847+0000","scrub_duration":0.0013083789999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.c","version":"66'10","reported_seq":54,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-10T19:11:23.237079+0000","last_change":"2024-09-10T19:11:14.987470+0000","last_active":"2024-09-10T19:11:23.237079+0000","last_peered":"2024-09-10T19:11:23.237079+0000","last_clean":"2024-09-10T19:11:23.237079+0000","last_became_active":"2024-09-10T19:10:09.688321+0000","last_became_peered":"2024-09-10T19:10:09.688321+0000","last_unstale":"2024-09-10T19:11:23.237079+0000","last_undegraded":"2024-09-10T19:11:23.237079+0000","last_fullsized":"2024-09-10T19:11:23.237079+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'10","last_scrub_stamp":"2024-09-10T19:11:14.987403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:11:14.987403+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:12:21.734174+0000","scrub_duration":0.0018747600000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":41,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-10T19:11:26.110270+0000","last_change":"2024-09-10T19:11:26.110216+0000","last_active":"2024-09-10T19:11:26.110270+0000","last_peered":"2024-09-10T19:11:26.110270+0000","last_clean":"2024-09-10T19:11:26.110270+0000","last_became_active":"2024-09-10T19:10:07.680803+0000","last_became_peered":"2024-09-10T19:10:07.680803+0000","last_unstale":"2024-09-10T19:11:26.110270+0000","last_undegraded":"2024-09-10T19:11:26.110270+0000","last_fullsized":"2024-09-10T19:11:26.110270+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:11:26.110138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:11:26.110138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:12:35.580206+0000","scrub_duration":0.0011581549999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.d","version":"80'9","reported_seq":68,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-10T19:11:40.767439+0000","last_change":"2024-09-10T19:11:25.156512+0000","last_active":"2024-09-10T19:11:40.767439+0000","last_peered":"2024-09-10T19:11:40.767439+0000","last_clean":"2024-09-10T19:11:40.767439+0000","last_became_active":"2024-09-10T19:10:11.698547+0000","last_became_peered":"2024-09-10T19:10:11.698547+0000","last_unstale":"2024-09-10T19:11:40.767439+0000","last_undegraded":"2024-09-10T19:11:40.767439+0000","last_fullsized":"2024-09-10T19:11:40.767439+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"80'9","last_scrub_stamp":"2024-09-10T19:11:25.156465+0000","last_deep_scrub":"80'9","last_deep_scrub_stamp":"2024-09-10T19:11:25.156465+0000","last_clean_scrub_stamp":"2024-09-10T19:11:25.156465+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:12:26.762818+0000","scrub_duration":0.0022299569999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":35,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-10T19:11:26.096187+0000","last_change":"2024-09-10T19:11:26.096147+0000","last_active":"2024-09-10T19:11:26.096187+0000","last_peered":"2024-09-10T19:11:26.096187+0000","last_clean":"2024-09-10T19:11:26.096187+0000","last_became_active":"2024-09-10T19:10:13.718242+0000","last_became_peered":"2024-09-10T19:10:13.718242+0000","last_unstale":"2024-09-10T19:11:26.096187+0000","last_undegraded":"2024-09-10T19:11:26.096187+0000","last_fullsized":"2024-09-10T19:11:26.096187+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:11:26.096090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:11:26.096090+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:12:49.347064+0000","scrub_duration":0.0015599909999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.f","version":"66'15","reported_seq":54,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-10T19:11:23.237179+0000","last_change":"2024-09-10T19:10:09.691727+0000","last_active":"2024-09-10T19:11:23.237179+0000","last_peered":"2024-09-10T19:11:23.237179+0000","last_clean":"2024-09-10T19:11:23.237179+0000","last_became_active":"2024-09-10T19:10:09.691230+0000","last_became_peered":"2024-09-10T19:10:09.691230+0000","last_unstale":"2024-09-10T19:11:23.237179+0000","last_undegraded":"2024-09-10T19:11:23.237179+0000","last_fullsized":"2024-09-10T19:11:23.237179+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:10:08.671214+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":41,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-10T19:11:23.236396+0000","last_change":"2024-09-10T19:11:11.036017+0000","last_active":"2024-09-10T19:11:23.236396+0000","last_peered":"2024-09-10T19:11:23.236396+0000","last_clean":"2024-09-10T19:11:23.236396+0000","last_became_active":"2024-09-10T19:10:07.679646+0000","last_became_peered":"2024-09-10T19:10:07.679646+0000","last_unstale":"2024-09-10T19:11:23.236396+0000","last_undegraded":"2024-09-10T19:11:23.236396+0000","last_fullsized":"2024-09-10T19:11:23.236396+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:11:11.035954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:11:11.035954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:12:35.469086+0000","scrub_duration":0.001618024,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.e","version":"66'8","reported_seq":54,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-10T19:11:40.767449+0000","last_change":"2024-09-10T19:10:11.699487+0000","last_active":"2024-09-10T19:11:40.767449+0000","last_peered":"2024-09-10T19:11:40.767449+0000","last_clean":"2024-09-10T19:11:40.767449+0000","last_became_active":"2024-09-10T19:10:11.699232+0000","last_became_peered":"2024-09-10T19:10:11.699232+0000","last_unstale":"2024-09-10T19:11:40.767449+0000","last_undegraded":"2024-09-10T19:11:40.767449+0000","last_fullsized":"2024-09-10T19:11:40.767449+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:10:10.676876+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":35,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-10T19:11:23.236898+0000","last_change":"2024-09-10T19:11:18.208847+0000","last_active":"2024-09-10T19:11:23.236898+0000","last_peered":"2024-09-10T19:11:23.236898+0000","last_clean":"2024-09-10T19:11:23.236898+0000","last_became_active":"2024-09-10T19:10:13.715274+0000","last_became_peered":"2024-09-10T19:10:13.715274+0000","last_unstale":"2024-09-10T19:11:23.236898+0000","last_undegraded":"2024-09-10T19:11:23.236898+0000","last_fullsized":"2024-09-10T19:11:23.236898+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:11:18.208782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:11:18.208782+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:12:43.016878+0000","scrub_duration":0.001904481,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.e","version":"66'11","reported_seq":48,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-10T19:11:23.237941+0000","last_change":"2024-09-10T19:10:09.692995+0000","last_active":"2024-09-10T19:11:23.237941+0000","last_peered":"2024-09-10T19:11:23.237941+0000","last_clean":"2024-09-10T19:11:23.237941+0000","last_became_active":"2024-09-10T19:10:09.692675+0000","last_became_peered":"2024-09-10T19:10:09.692675+0000","last_unstale":"2024-09-10T19:11:23.237941+0000","last_undegraded":"2024-09-10T19:11:23.237941+0000","last_fullsized":"2024-09-10T19:11:23.237941+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:10:08.671214+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":31,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-10T19:11:23.237673+0000","last_change":"2024-09-10T19:10:07.690701+0000","last_active":"2024-09-10T19:11:23.237673+0000","last_peered":"2024-09-10T19:11:23.237673+0000","last_clean":"2024-09-10T19:11:23.237673+0000","last_became_active":"2024-09-10T19:10:07.690434+0000","last_became_peered":"2024-09-10T19:10:07.690434+0000","last_unstale":"2024-09-10T19:11:23.237673+0000","last_undegraded":"2024-09-10T19:11:23.237673+0000","last_fullsized":"2024-09-10T19:11:23.237673+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:10:06.664709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":27,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-10T19:11:23.237149+0000","last_change":"2024-09-10T19:10:11.692483+0000","last_active":"2024-09-10T19:11:23.237149+0000","last_peered":"2024-09-10T19:11:23.237149+0000","last_clean":"2024-09-10T19:11:23.237149+0000","last_became_active":"2024-09-10T19:10:11.692215+0000","last_became_peered":"2024-09-10T19:10:11.692215+0000","last_unstale":"2024-09-10T19:11:23.237149+0000","last_undegraded":"2024-09-10T19:11:23.237149+0000","last_fullsized":"2024-09-10T19:11:23.237149+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:10:10.676876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":25,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-10T19:11:23.208012+0000","last_change":"2024-09-10T19:10:14.054106+0000","last_active":"2024-09-10T19:11:23.208012+0000","last_peered":"2024-09-10T19:11:23.208012+0000","last_clean":"2024-09-10T19:11:23.208012+0000","last_became_active":"2024-09-10T19:10:14.053736+0000","last_became_peered":"2024-09-10T19:10:14.053736+0000","last_unstale":"2024-09-10T19:11:23.208012+0000","last_undegraded":"2024-09-10T19:11:23.208012+0000","last_fullsized":"2024-09-10T19:11:23.208012+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:10:12.693598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"66'14","reported_seq":60,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-10T19:11:23.236488+0000","last_change":"2024-09-10T19:11:16.976833+0000","last_active":"2024-09-10T19:11:23.236488+0000","last_peered":"2024-09-10T19:11:23.236488+0000","last_clean":"2024-09-10T19:11:23.236488+0000","last_became_active":"2024-09-10T19:10:09.686534+0000","last_became_peered":"2024-09-10T19:10:09.686534+0000","last_unstale":"2024-09-10T19:11:23.236488+0000","last_undegraded":"2024-09-10T19:11:23.236488+0000","last_fullsized":"2024-09-10T19:11:23.236488+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'14","last_scrub_stamp":"2024-09-10T19:11:16.976776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:11:16.976776+0000","objects_scrubbed":7,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:12:29.125897+0000","scrub_duration":0.001777144,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":41,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-10T19:11:23.207677+0000","last_change":"2024-09-10T19:11:22.801187+0000","last_active":"2024-09-10T19:11:23.207677+0000","last_peered":"2024-09-10T19:11:23.207677+0000","last_clean":"2024-09-10T19:11:23.207677+0000","last_became_active":"2024-09-10T19:10:07.684864+0000","last_became_peered":"2024-09-10T19:10:07.684864+0000","last_unstale":"2024-09-10T19:11:23.207677+0000","last_undegraded":"2024-09-10T19:11:23.207677+0000","last_fullsized":"2024-09-10T19:11:23.207677+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:11:22.801123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:11:22.801123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:12:47.860716+0000","scrub_duration":0.0015474410000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.0","version":"66'8","reported_seq":64,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-10T19:11:40.767437+0000","last_change":"2024-09-10T19:11:29.884792+0000","last_active":"2024-09-10T19:11:40.767437+0000","last_peered":"2024-09-10T19:11:40.767437+0000","last_clean":"2024-09-10T19:11:40.767437+0000","last_became_active":"2024-09-10T19:10:11.697106+0000","last_became_peered":"2024-09-10T19:10:11.697106+0000","last_unstale":"2024-09-10T19:11:40.767437+0000","last_undegraded":"2024-09-10T19:11:40.767437+0000","last_fullsized":"2024-09-10T19:11:40.767437+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'8","last_scrub_stamp":"2024-09-10T19:11:29.884739+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:11:29.884739+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:12:37.090103+0000","scrub_duration":0.0014372650000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":35,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-10T19:11:41.073938+0000","last_change":"2024-09-10T19:11:41.073902+0000","last_active":"2024-09-10T19:11:41.073938+0000","last_peered":"2024-09-10T19:11:41.073938+0000","last_clean":"2024-09-10T19:11:41.073938+0000","last_became_active":"2024-09-10T19:10:14.054456+0000","last_became_peered":"2024-09-10T19:10:14.054456+0000","last_unstale":"2024-09-10T19:11:41.073938+0000","last_undegraded":"2024-09-10T19:11:41.073938+0000","last_fullsized":"2024-09-10T19:11:41.073938+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:11:41.073833+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:11:41.073833+0000","last_clean_scrub_stamp":"2024-09-10T19:11:41.073833+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:13:00.467462+0000","scrub_duration":0.001716567,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.0","version":"66'18","reported_seq":71,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-10T19:11:23.207100+0000","last_change":"2024-09-10T19:11:19.848186+0000","last_active":"2024-09-10T19:11:23.207100+0000","last_peered":"2024-09-10T19:11:23.207100+0000","last_clean":"2024-09-10T19:11:23.207100+0000","last_became_active":"2024-09-10T19:10:09.821836+0000","last_became_peered":"2024-09-10T19:10:09.821836+0000","last_unstale":"2024-09-10T19:11:23.207100+0000","last_undegraded":"2024-09-10T19:11:23.207100+0000","last_fullsized":"2024-09-10T19:11:23.207100+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'18","last_scrub_stamp":"2024-09-10T19:11:19.848132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:11:19.848132+0000","objects_scrubbed":8,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:12:26.692213+0000","scrub_duration":0.0034247819999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":41,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-10T19:11:42.202780+0000","last_change":"2024-09-10T19:11:42.202747+0000","last_active":"2024-09-10T19:11:42.202780+0000","last_peered":"2024-09-10T19:11:42.202780+0000","last_clean":"2024-09-10T19:11:42.202780+0000","last_became_active":"2024-09-10T19:10:07.682081+0000","last_became_peered":"2024-09-10T19:10:07.682081+0000","last_unstale":"2024-09-10T19:11:42.202780+0000","last_undegraded":"2024-09-10T19:11:42.202780+0000","last_fullsized":"2024-09-10T19:11:42.202780+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:11:42.202689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:11:42.202689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:12:53.708170+0000","scrub_duration":0.00127655,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":37,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-10T19:11:29.069470+0000","last_change":"2024-09-10T19:11:29.069421+0000","last_active":"2024-09-10T19:11:29.069470+0000","last_peered":"2024-09-10T19:11:29.069470+0000","last_clean":"2024-09-10T19:11:29.069470+0000","last_became_active":"2024-09-10T19:10:11.698054+0000","last_became_peered":"2024-09-10T19:10:11.698054+0000","last_unstale":"2024-09-10T19:11:29.069470+0000","last_undegraded":"2024-09-10T19:11:29.069470+0000","last_fullsized":"2024-09-10T19:11:29.069470+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:11:29.069369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:11:29.069369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:12:55.817612+0000","scrub_duration":0.001327562,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":35,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-10T19:11:34.139436+0000","last_change":"2024-09-10T19:11:34.139371+0000","last_active":"2024-09-10T19:11:34.139436+0000","last_peered":"2024-09-10T19:11:34.139436+0000","last_clean":"2024-09-10T19:11:34.139436+0000","last_became_active":"2024-09-10T19:10:13.722784+0000","last_became_peered":"2024-09-10T19:10:13.722784+0000","last_unstale":"2024-09-10T19:11:34.139436+0000","last_undegraded":"2024-09-10T19:11:34.139436+0000","last_fullsized":"2024-09-10T19:11:34.139436+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:11:34.139306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:11:34.139306+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:12:45.529547+0000","scrub_duration":0.00133668,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.3","version":"66'19","reported_seq":65,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-10T19:11:23.236589+0000","last_change":"2024-09-10T19:10:09.823263+0000","last_active":"2024-09-10T19:11:23.236589+0000","last_peered":"2024-09-10T19:11:23.236589+0000","last_clean":"2024-09-10T19:11:23.236589+0000","last_became_active":"2024-09-10T19:10:09.823003+0000","last_became_peered":"2024-09-10T19:10:09.823003+0000","last_unstale":"2024-09-10T19:11:23.236589+0000","last_undegraded":"2024-09-10T19:11:23.236589+0000","last_fullsized":"2024-09-10T19:11:23.236589+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:10:08.671214+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":41,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-10T19:11:25.290185+0000","last_change":"2024-09-10T19:11:25.290147+0000","last_active":"2024-09-10T19:11:25.290185+0000","last_peered":"2024-09-10T19:11:25.290185+0000","last_clean":"2024-09-10T19:11:25.290185+0000","last_became_active":"2024-09-10T19:10:07.690325+0000","last_became_peered":"2024-09-10T19:10:07.690325+0000","last_unstale":"2024-09-10T19:11:25.290185+0000","last_undegraded":"2024-09-10T19:11:25.290185+0000","last_fullsized":"2024-09-10T19:11:25.290185+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:11:25.290088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:11:25.290088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:12:31.270129+0000","scrub_duration":0.0013977760000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":37,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-10T19:11:23.843378+0000","last_change":"2024-09-10T19:11:23.843315+0000","last_active":"2024-09-10T19:11:23.843378+0000","last_peered":"2024-09-10T19:11:23.843378+0000","last_clean":"2024-09-10T19:11:23.843378+0000","last_became_active":"2024-09-10T19:10:11.700370+0000","last_became_peered":"2024-09-10T19:10:11.700370+0000","last_unstale":"2024-09-10T19:11:23.843378+0000","last_undegraded":"2024-09-10T19:11:23.843378+0000","last_fullsized":"2024-09-10T19:11:23.843378+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:11:23.843228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:11:23.843228+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:12:29.095733+0000","scrub_duration":0.0013052070000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":25,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-10T19:11:23.236235+0000","last_change":"2024-09-10T19:10:14.054654+0000","last_active":"2024-09-10T19:11:23.236235+0000","last_peered":"2024-09-10T19:11:23.236235+0000","last_clean":"2024-09-10T19:11:23.236235+0000","last_became_active":"2024-09-10T19:10:14.054318+0000","last_became_peered":"2024-09-10T19:10:14.054318+0000","last_unstale":"2024-09-10T19:11:23.236235+0000","last_undegraded":"2024-09-10T19:11:23.236235+0000","last_fullsized":"2024-09-10T19:11:23.236235+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:10:12.693598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":1,"reported_epoch":81,"state":"peering","last_fresh":"2024-09-10T19:11:43.736440+0000","last_change":"2024-09-10T19:11:43.736440+0000","last_active":"0.000000","last_peered":"0.000000","last_clean":"0.000000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-10T19:11:43.736440+0000","last_undegraded":"2024-09-10T19:11:43.736440+0000","last_fullsized":"2024-09-10T19:11:43.736440+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'10","last_scrub_stamp":"2024-09-10T19:11:28.132850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:11:28.132850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:12:49.690133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":31,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-10T19:11:23.237074+0000","last_change":"2024-09-10T19:10:07.682243+0000","last_active":"2024-09-10T19:11:23.237074+0000","last_peered":"2024-09-10T19:11:23.237074+0000","last_clean":"2024-09-10T19:11:23.237074+0000","last_became_active":"2024-09-10T19:10:07.681965+0000","last_became_peered":"2024-09-10T19:10:07.681965+0000","last_unstale":"2024-09-10T19:11:23.237074+0000","last_undegraded":"2024-09-10T19:11:23.237074+0000","last_fullsized":"2024-09-10T19:11:23.237074+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:10:06.664709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"66'8","reported_seq":54,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-10T19:11:40.767497+0000","last_change":"2024-09-10T19:10:11.699975+0000","last_active":"2024-09-10T19:11:40.767497+0000","last_peered":"2024-09-10T19:11:40.767497+0000","last_clean":"2024-09-10T19:11:40.767497+0000","last_became_active":"2024-09-10T19:10:11.699690+0000","last_became_peered":"2024-09-10T19:10:11.699690+0000","last_unstale":"2024-09-10T19:11:40.767497+0000","last_undegraded":"2024-09-10T19:11:40.767497+0000","last_fullsized":"2024-09-10T19:11:40.767497+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:10:10.676876+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":35,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-10T19:11:43.922609+0000","last_change":"2024-09-10T19:11:43.922577+0000","last_active":"2024-09-10T19:11:43.922609+0000","last_peered":"2024-09-10T19:11:43.922609+0000","last_clean":"2024-09-10T19:11:43.922609+0000","last_became_active":"2024-09-10T19:10:13.720403+0000","last_became_peered":"2024-09-10T19:10:13.720403+0000","last_unstale":"2024-09-10T19:11:43.922609+0000","last_undegraded":"2024-09-10T19:11:43.922609+0000","last_fullsized":"2024-09-10T19:11:43.922609+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:11:43.922522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:11:43.922522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:13:13.573808+0000","scrub_duration":0.0012347289999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"66'28","reported_seq":106,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-10T19:11:40.767444+0000","last_change":"2024-09-10T19:11:17.050501+0000","last_active":"2024-09-10T19:11:40.767444+0000","last_peered":"2024-09-10T19:11:40.767444+0000","last_clean":"2024-09-10T19:11:40.767444+0000","last_became_active":"2024-09-10T19:10:09.692663+0000","last_became_peered":"2024-09-10T19:10:09.692663+0000","last_unstale":"2024-09-10T19:11:40.767444+0000","last_undegraded":"2024-09-10T19:11:40.767444+0000","last_fullsized":"2024-09-10T19:11:40.767444+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'28","last_scrub_stamp":"2024-09-10T19:11:17.050442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:11:17.050442+0000","objects_scrubbed":10,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:12:31.661637+0000","scrub_duration":0.001985559,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":41,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-10T19:11:41.429510+0000","last_change":"2024-09-10T19:11:41.429457+0000","last_active":"2024-09-10T19:11:41.429510+0000","last_peered":"2024-09-10T19:11:41.429510+0000","last_clean":"2024-09-10T19:11:41.429510+0000","last_became_active":"2024-09-10T19:10:07.682724+0000","last_became_peered":"2024-09-10T19:10:07.682724+0000","last_unstale":"2024-09-10T19:11:41.429510+0000","last_undegraded":"2024-09-10T19:11:41.429510+0000","last_fullsized":"2024-09-10T19:11:41.429510+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:11:41.429402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:11:41.429402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:12:58.343046+0000","scrub_duration":0.0014226359999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":37,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-10T19:11:23.236895+0000","last_change":"2024-09-10T19:11:14.985340+0000","last_active":"2024-09-10T19:11:23.236895+0000","last_peered":"2024-09-10T19:11:23.236895+0000","last_clean":"2024-09-10T19:11:23.236895+0000","last_became_active":"2024-09-10T19:10:11.698825+0000","last_became_peered":"2024-09-10T19:10:11.698825+0000","last_unstale":"2024-09-10T19:11:23.236895+0000","last_undegraded":"2024-09-10T19:11:23.236895+0000","last_fullsized":"2024-09-10T19:11:23.236895+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:11:14.985249+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:11:14.985249+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:12:30.155012+0000","scrub_duration":0.001460161,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":25,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-10T19:11:23.208309+0000","last_change":"2024-09-10T19:10:13.722038+0000","last_active":"2024-09-10T19:11:23.208309+0000","last_peered":"2024-09-10T19:11:23.208309+0000","last_clean":"2024-09-10T19:11:23.208309+0000","last_became_active":"2024-09-10T19:10:13.721744+0000","last_became_peered":"2024-09-10T19:10:13.721744+0000","last_unstale":"2024-09-10T19:11:23.208309+0000","last_undegraded":"2024-09-10T19:11:23.208309+0000","last_fullsized":"2024-09-10T19:11:23.208309+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:10:12.693598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.7","version":"66'13","reported_seq":56,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-10T19:11:23.237191+0000","last_change":"2024-09-10T19:10:09.693625+0000","last_active":"2024-09-10T19:11:23.237191+0000","last_peered":"2024-09-10T19:11:23.237191+0000","last_clean":"2024-09-10T19:11:23.237191+0000","last_became_active":"2024-09-10T19:10:09.693346+0000","last_became_peered":"2024-09-10T19:10:09.693346+0000","last_unstale":"2024-09-10T19:11:23.237191+0000","last_undegraded":"2024-09-10T19:11:23.237191+0000","last_fullsized":"2024-09-10T19:11:23.237191+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:10:08.671214+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":19,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"67'5","reported_seq":140,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-10T19:11:43.739607+0000","last_change":"2024-09-10T19:11:16.007355+0000","last_active":"2024-09-10T19:11:43.739607+0000","last_peered":"2024-09-10T19:11:43.739607+0000","last_clean":"2024-09-10T19:11:43.739607+0000","last_became_active":"2024-09-10T19:10:07.682125+0000","last_became_peered":"2024-09-10T19:10:07.682125+0000","last_unstale":"2024-09-10T19:11:43.739607+0000","last_undegraded":"2024-09-10T19:11:43.739607+0000","last_fullsized":"2024-09-10T19:11:43.739607+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'5","last_scrub_stamp":"2024-09-10T19:11:16.007261+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:11:16.007261+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:12:39.490457+0000","scrub_duration":0.0019059680000000001,"objects_trimmed":0,"snaptrim_duration":0.00034454499999999997,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":91,"num_read_kb":86,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":41,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-10T19:11:33.340830+0000","last_change":"2024-09-10T19:11:33.340773+0000","last_active":"2024-09-10T19:11:33.340830+0000","last_peered":"2024-09-10T19:11:33.340830+0000","last_clean":"2024-09-10T19:11:33.340830+0000","last_became_active":"2024-09-10T19:10:07.681456+0000","last_became_peered":"2024-09-10T19:10:07.681456+0000","last_unstale":"2024-09-10T19:11:33.340830+0000","last_undegraded":"2024-09-10T19:11:33.340830+0000","last_fullsized":"2024-09-10T19:11:33.340830+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:11:33.340715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:11:33.340715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:12:37.277130+0000","scrub_duration":0.0012840099999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":37,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-10T19:11:36.443446+0000","last_change":"2024-09-10T19:11:36.443407+0000","last_active":"2024-09-10T19:11:36.443446+0000","last_peered":"2024-09-10T19:11:36.443446+0000","last_clean":"2024-09-10T19:11:36.443446+0000","last_became_active":"2024-09-10T19:10:11.695205+0000","last_became_peered":"2024-09-10T19:10:11.695205+0000","last_unstale":"2024-09-10T19:11:36.443446+0000","last_undegraded":"2024-09-10T19:11:36.443446+0000","last_fullsized":"2024-09-10T19:11:36.443446+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:11:36.443352+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:11:36.443352+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:12:59.383628+0000","scrub_duration":0.001511814,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":25,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-10T19:11:23.238117+0000","last_change":"2024-09-10T19:10:14.054047+0000","last_active":"2024-09-10T19:11:23.238117+0000","last_peered":"2024-09-10T19:11:23.238117+0000","last_clean":"2024-09-10T19:11:23.238117+0000","last_became_active":"2024-09-10T19:10:14.053697+0000","last_became_peered":"2024-09-10T19:10:14.053697+0000","last_unstale":"2024-09-10T19:11:23.238117+0000","last_undegraded":"2024-09-10T19:11:23.238117+0000","last_fullsized":"2024-09-10T19:11:23.238117+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:10:12.693598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"66'12","reported_seq":57,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-10T19:11:38.936966+0000","last_change":"2024-09-10T19:11:38.936913+0000","last_active":"2024-09-10T19:11:38.936966+0000","last_peered":"2024-09-10T19:11:38.936966+0000","last_clean":"2024-09-10T19:11:38.936966+0000","last_became_active":"2024-09-10T19:10:09.688091+0000","last_became_peered":"2024-09-10T19:10:09.688091+0000","last_unstale":"2024-09-10T19:11:38.936966+0000","last_undegraded":"2024-09-10T19:11:38.936966+0000","last_fullsized":"2024-09-10T19:11:38.936966+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'12","last_scrub_stamp":"2024-09-10T19:11:38.936857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:11:38.936857+0000","objects_scrubbed":6,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:12:55.623292+0000","scrub_duration":0.0016916909999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"61'1","reported_seq":37,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-10T19:11:23.236290+0000","last_change":"2024-09-10T19:10:11.693049+0000","last_active":"2024-09-10T19:11:23.236290+0000","last_peered":"2024-09-10T19:11:23.236290+0000","last_clean":"2024-09-10T19:11:23.236290+0000","last_became_active":"2024-09-10T19:10:07.683259+0000","last_became_peered":"2024-09-10T19:10:07.683259+0000","last_unstale":"2024-09-10T19:11:23.236290+0000","last_undegraded":"2024-09-10T19:11:23.236290+0000","last_fullsized":"2024-09-10T19:11:23.236290+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:10:06.664709+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014341130000000001,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":41,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-10T19:11:23.237674+0000","last_change":"2024-09-10T19:11:21.301130+0000","last_active":"2024-09-10T19:11:23.237674+0000","last_peered":"2024-09-10T19:11:23.237674+0000","last_clean":"2024-09-10T19:11:23.237674+0000","last_became_active":"2024-09-10T19:10:07.690549+0000","last_became_peered":"2024-09-10T19:10:07.690549+0000","last_unstale":"2024-09-10T19:11:23.237674+0000","last_undegraded":"2024-09-10T19:11:23.237674+0000","last_fullsized":"2024-09-10T19:11:23.237674+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:11:21.301058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:11:21.301058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:12:41.356053+0000","scrub_duration":0.0021074710000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":37,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-10T19:11:23.237199+0000","last_change":"2024-09-10T19:11:16.156929+0000","last_active":"2024-09-10T19:11:23.237199+0000","last_peered":"2024-09-10T19:11:23.237199+0000","last_clean":"2024-09-10T19:11:23.237199+0000","last_became_active":"2024-09-10T19:10:11.699798+0000","last_became_peered":"2024-09-10T19:10:11.699798+0000","last_unstale":"2024-09-10T19:11:23.237199+0000","last_undegraded":"2024-09-10T19:11:23.237199+0000","last_fullsized":"2024-09-10T19:11:23.237199+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:11:16.156819+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:11:16.156819+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:12:16.166187+0000","scrub_duration":0.001649059,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":25,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-10T19:11:23.236821+0000","last_change":"2024-09-10T19:10:13.722695+0000","last_active":"2024-09-10T19:11:23.236821+0000","last_peered":"2024-09-10T19:11:23.236821+0000","last_clean":"2024-09-10T19:11:23.236821+0000","last_became_active":"2024-09-10T19:10:13.722368+0000","last_became_peered":"2024-09-10T19:10:13.722368+0000","last_unstale":"2024-09-10T19:11:23.236821+0000","last_undegraded":"2024-09-10T19:11:23.236821+0000","last_fullsized":"2024-09-10T19:11:23.236821+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:10:12.693598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"80'21","reported_seq":68,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-10T19:11:39.836243+0000","last_change":"2024-09-10T19:11:39.836197+0000","last_active":"2024-09-10T19:11:39.836243+0000","last_peered":"2024-09-10T19:11:39.836243+0000","last_clean":"2024-09-10T19:11:39.836243+0000","last_became_active":"2024-09-10T19:10:09.822184+0000","last_became_peered":"2024-09-10T19:10:09.822184+0000","last_unstale":"2024-09-10T19:11:39.836243+0000","last_undegraded":"2024-09-10T19:11:39.836243+0000","last_fullsized":"2024-09-10T19:11:39.836243+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"80'21","last_scrub_stamp":"2024-09-10T19:11:39.836144+0000","last_deep_scrub":"80'21","last_deep_scrub_stamp":"2024-09-10T19:11:39.836144+0000","last_clean_scrub_stamp":"2024-09-10T19:11:39.836144+0000","objects_scrubbed":7,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:12:48.908909+0000","scrub_duration":0.0049231029999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"63'2","reported_seq":48,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-10T19:11:26.824234+0000","last_change":"2024-09-10T19:11:26.824164+0000","last_active":"2024-09-10T19:11:26.824234+0000","last_peered":"2024-09-10T19:11:26.824234+0000","last_clean":"2024-09-10T19:11:26.824234+0000","last_became_active":"2024-09-10T19:10:07.684784+0000","last_became_peered":"2024-09-10T19:10:07.684784+0000","last_unstale":"2024-09-10T19:11:26.824234+0000","last_undegraded":"2024-09-10T19:11:26.824234+0000","last_fullsized":"2024-09-10T19:11:26.824234+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'2","last_scrub_stamp":"2024-09-10T19:11:26.824103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:11:26.824103+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:12:48.229817+0000","scrub_duration":0.0018538929999999999,"objects_trimmed":0,"snaptrim_duration":0.00075522999999999996,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":41,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-10T19:11:23.236859+0000","last_change":"2024-09-10T19:11:17.171319+0000","last_active":"2024-09-10T19:11:23.236859+0000","last_peered":"2024-09-10T19:11:23.236859+0000","last_clean":"2024-09-10T19:11:23.236859+0000","last_became_active":"2024-09-10T19:10:07.687596+0000","last_became_peered":"2024-09-10T19:10:07.687596+0000","last_unstale":"2024-09-10T19:11:23.236859+0000","last_undegraded":"2024-09-10T19:11:23.236859+0000","last_fullsized":"2024-09-10T19:11:23.236859+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:11:17.171251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:11:17.171251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:12:21.736124+0000","scrub_duration":0.0016247670000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"74'79","reported_seq":76,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-10T19:11:23.237772+0000","last_change":"2024-09-10T19:10:34.385577+0000","last_active":"2024-09-10T19:11:23.237772+0000","last_peered":"2024-09-10T19:11:23.237772+0000","last_clean":"2024-09-10T19:11:23.237772+0000","last_became_active":"2024-09-10T19:09:56.634919+0000","last_became_peered":"2024-09-10T19:09:56.634919+0000","last_unstale":"2024-09-10T19:11:23.237772+0000","last_undegraded":"2024-09-10T19:11:23.237772+0000","last_fullsized":"2024-09-10T19:11:23.237772+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'72","last_scrub_stamp":"2024-09-10T19:10:34.385519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:08:00.699315+0000","last_clean_scrub_stamp":"2024-09-10T19:10:34.385519+0000","objects_scrubbed":2,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:11:51.371427+0000","scrub_duration":0.0030997160000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":27,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-10T19:11:23.237889+0000","last_change":"2024-09-10T19:10:11.695316+0000","last_active":"2024-09-10T19:11:23.237889+0000","last_peered":"2024-09-10T19:11:23.237889+0000","last_clean":"2024-09-10T19:11:23.237889+0000","last_became_active":"2024-09-10T19:10:11.694986+0000","last_became_peered":"2024-09-10T19:10:11.694986+0000","last_unstale":"2024-09-10T19:11:23.237889+0000","last_undegraded":"2024-09-10T19:11:23.237889+0000","last_fullsized":"2024-09-10T19:11:23.237889+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:10:10.676876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":35,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-10T19:11:25.312459+0000","last_change":"2024-09-10T19:11:25.312410+0000","last_active":"2024-09-10T19:11:25.312459+0000","last_peered":"2024-09-10T19:11:25.312459+0000","last_clean":"2024-09-10T19:11:25.312459+0000","last_became_active":"2024-09-10T19:10:13.720647+0000","last_became_peered":"2024-09-10T19:10:13.720647+0000","last_unstale":"2024-09-10T19:11:25.312459+0000","last_undegraded":"2024-09-10T19:11:25.312459+0000","last_fullsized":"2024-09-10T19:11:25.312459+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:11:25.312356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:11:25.312356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:12:54.878799+0000","scrub_duration":0.001690642,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.a","version":"66'19","reported_seq":70,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-10T19:11:42.941535+0000","last_change":"2024-09-10T19:11:42.941440+0000","last_active":"2024-09-10T19:11:42.941535+0000","last_peered":"2024-09-10T19:11:42.941535+0000","last_clean":"2024-09-10T19:11:42.941535+0000","last_became_active":"2024-09-10T19:10:09.820880+0000","last_became_peered":"2024-09-10T19:10:09.820880+0000","last_unstale":"2024-09-10T19:11:42.941535+0000","last_undegraded":"2024-09-10T19:11:42.941535+0000","last_fullsized":"2024-09-10T19:11:42.941535+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'19","last_scrub_stamp":"2024-09-10T19:11:42.941375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:11:42.941375+0000","objects_scrubbed":9,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:13:12.267779+0000","scrub_duration":0.0021647599999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":41,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-10T19:11:40.484457+0000","last_change":"2024-09-10T19:11:40.484394+0000","last_active":"2024-09-10T19:11:40.484457+0000","last_peered":"2024-09-10T19:11:40.484457+0000","last_clean":"2024-09-10T19:11:40.484457+0000","last_became_active":"2024-09-10T19:10:07.686236+0000","last_became_peered":"2024-09-10T19:10:07.686236+0000","last_unstale":"2024-09-10T19:11:40.484457+0000","last_undegraded":"2024-09-10T19:11:40.484457+0000","last_fullsized":"2024-09-10T19:11:40.484457+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:11:40.484333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:11:40.484333+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:13:05.193849+0000","scrub_duration":0.0013800889999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":37,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-10T19:11:23.206714+0000","last_change":"2024-09-10T19:11:19.119050+0000","last_active":"2024-09-10T19:11:23.206714+0000","last_peered":"2024-09-10T19:11:23.206714+0000","last_clean":"2024-09-10T19:11:23.206714+0000","last_became_active":"2024-09-10T19:10:11.699745+0000","last_became_peered":"2024-09-10T19:10:11.699745+0000","last_unstale":"2024-09-10T19:11:23.206714+0000","last_undegraded":"2024-09-10T19:11:23.206714+0000","last_fullsized":"2024-09-10T19:11:23.206714+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:11:19.118987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:11:19.118987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:12:48.847171+0000","scrub_duration":0.0012170379999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":35,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-10T19:11:38.102895+0000","last_change":"2024-09-10T19:11:38.102859+0000","last_active":"2024-09-10T19:11:38.102895+0000","last_peered":"2024-09-10T19:11:38.102895+0000","last_clean":"2024-09-10T19:11:38.102895+0000","last_became_active":"2024-09-10T19:10:13.719973+0000","last_became_peered":"2024-09-10T19:10:13.719973+0000","last_unstale":"2024-09-10T19:11:38.102895+0000","last_undegraded":"2024-09-10T19:11:38.102895+0000","last_fullsized":"2024-09-10T19:11:38.102895+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:11:38.102803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:11:38.102803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:12:52.451650+0000","scrub_duration":0.001577272,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"66'9","reported_seq":55,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-10T19:11:23.236512+0000","last_change":"2024-09-10T19:11:22.870432+0000","last_active":"2024-09-10T19:11:23.236512+0000","last_peered":"2024-09-10T19:11:23.236512+0000","last_clean":"2024-09-10T19:11:23.236512+0000","last_became_active":"2024-09-10T19:10:09.689117+0000","last_became_peered":"2024-09-10T19:10:09.689117+0000","last_unstale":"2024-09-10T19:11:23.236512+0000","last_undegraded":"2024-09-10T19:11:23.236512+0000","last_fullsized":"2024-09-10T19:11:23.236512+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'9","last_scrub_stamp":"2024-09-10T19:11:22.870369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:11:22.870369+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:12:27.485364+0000","scrub_duration":0.0017468729999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":41,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-10T19:11:44.254598+0000","last_change":"2024-09-10T19:11:44.254567+0000","last_active":"2024-09-10T19:11:44.254598+0000","last_peered":"2024-09-10T19:11:44.254598+0000","last_clean":"2024-09-10T19:11:44.254598+0000","last_became_active":"2024-09-10T19:10:07.682834+0000","last_became_peered":"2024-09-10T19:10:07.682834+0000","last_unstale":"2024-09-10T19:11:44.254598+0000","last_undegraded":"2024-09-10T19:11:44.254598+0000","last_fullsized":"2024-09-10T19:11:44.254598+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:11:44.254513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:11:44.254513+0000","last_clean_scrub_stamp":"2024-09-10T19:11:44.254513+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:13:01.679588+0000","scrub_duration":0.0013179349999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":37,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-10T19:11:42.480578+0000","last_change":"2024-09-10T19:11:42.480542+0000","last_active":"2024-09-10T19:11:42.480578+0000","last_peered":"2024-09-10T19:11:42.480578+0000","last_clean":"2024-09-10T19:11:42.480578+0000","last_became_active":"2024-09-10T19:10:11.694405+0000","last_became_peered":"2024-09-10T19:10:11.694405+0000","last_unstale":"2024-09-10T19:11:42.480578+0000","last_undegraded":"2024-09-10T19:11:42.480578+0000","last_fullsized":"2024-09-10T19:11:42.480578+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:11:42.480487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:11:42.480487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:13:07.856129+0000","scrub_duration":0.0012878259999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":35,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-10T19:11:23.236216+0000","last_change":"2024-09-10T19:11:17.974993+0000","last_active":"2024-09-10T19:11:23.236216+0000","last_peered":"2024-09-10T19:11:23.236216+0000","last_clean":"2024-09-10T19:11:23.236216+0000","last_became_active":"2024-09-10T19:10:13.714952+0000","last_became_peered":"2024-09-10T19:10:13.714952+0000","last_unstale":"2024-09-10T19:11:23.236216+0000","last_undegraded":"2024-09-10T19:11:23.236216+0000","last_fullsized":"2024-09-10T19:11:23.236216+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:11:17.974937+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:11:17.974937+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:12:32.960908+0000","scrub_duration":0.001345834,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.8","version":"66'15","reported_seq":64,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-10T19:11:33.481880+0000","last_change":"2024-09-10T19:11:33.481823+0000","last_active":"2024-09-10T19:11:33.481880+0000","last_peered":"2024-09-10T19:11:33.481880+0000","last_clean":"2024-09-10T19:11:33.481880+0000","last_became_active":"2024-09-10T19:10:09.820950+0000","last_became_peered":"2024-09-10T19:10:09.820950+0000","last_unstale":"2024-09-10T19:11:33.481880+0000","last_undegraded":"2024-09-10T19:11:33.481880+0000","last_fullsized":"2024-09-10T19:11:33.481880+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'15","last_scrub_stamp":"2024-09-10T19:11:33.481743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:11:33.481743+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:12:42.712551+0000","scrub_duration":0.002098094,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":41,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-10T19:11:23.207082+0000","last_change":"2024-09-10T19:11:15.197602+0000","last_active":"2024-09-10T19:11:23.207082+0000","last_peered":"2024-09-10T19:11:23.207082+0000","last_clean":"2024-09-10T19:11:23.207082+0000","last_became_active":"2024-09-10T19:10:07.682286+0000","last_became_peered":"2024-09-10T19:10:07.682286+0000","last_unstale":"2024-09-10T19:11:23.207082+0000","last_undegraded":"2024-09-10T19:11:23.207082+0000","last_fullsized":"2024-09-10T19:11:23.207082+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:11:15.197497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:11:15.197497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:12:23.597035+0000","scrub_duration":0.0019717929999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.9","version":"80'9","reported_seq":65,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-10T19:11:40.767780+0000","last_change":"2024-09-10T19:11:35.174273+0000","last_active":"2024-09-10T19:11:40.767780+0000","last_peered":"2024-09-10T19:11:40.767780+0000","last_clean":"2024-09-10T19:11:40.767780+0000","last_became_active":"2024-09-10T19:10:11.702878+0000","last_became_peered":"2024-09-10T19:10:11.702878+0000","last_unstale":"2024-09-10T19:11:40.767780+0000","last_undegraded":"2024-09-10T19:11:40.767780+0000","last_fullsized":"2024-09-10T19:11:40.767780+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"80'9","last_scrub_stamp":"2024-09-10T19:11:35.174218+0000","last_deep_scrub":"80'9","last_deep_scrub_stamp":"2024-09-10T19:11:35.174218+0000","last_clean_scrub_stamp":"2024-09-10T19:11:35.174218+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:13:04.007735+0000","scrub_duration":0.0019812380000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":35,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-10T19:11:23.236522+0000","last_change":"2024-09-10T19:11:20.184218+0000","last_active":"2024-09-10T19:11:23.236522+0000","last_peered":"2024-09-10T19:11:23.236522+0000","last_clean":"2024-09-10T19:11:23.236522+0000","last_became_active":"2024-09-10T19:10:14.053875+0000","last_became_peered":"2024-09-10T19:10:14.053875+0000","last_unstale":"2024-09-10T19:11:23.236522+0000","last_undegraded":"2024-09-10T19:11:23.236522+0000","last_fullsized":"2024-09-10T19:11:23.236522+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:11:20.184132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:11:20.184132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:12:50.058323+0000","scrub_duration":0.002119522,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"66'12","reported_seq":52,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-10T19:11:23.236409+0000","last_change":"2024-09-10T19:10:09.693243+0000","last_active":"2024-09-10T19:11:23.236409+0000","last_peered":"2024-09-10T19:11:23.236409+0000","last_clean":"2024-09-10T19:11:23.236409+0000","last_became_active":"2024-09-10T19:10:09.692627+0000","last_became_peered":"2024-09-10T19:10:09.692627+0000","last_unstale":"2024-09-10T19:11:23.236409+0000","last_undegraded":"2024-09-10T19:11:23.236409+0000","last_fullsized":"2024-09-10T19:11:23.236409+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:10:08.671214+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.f","version":"59'2","reported_seq":57,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-10T19:11:25.111544+0000","last_change":"2024-09-10T19:11:25.111486+0000","last_active":"2024-09-10T19:11:25.111544+0000","last_peered":"2024-09-10T19:11:25.111544+0000","last_clean":"2024-09-10T19:11:25.111544+0000","last_became_active":"2024-09-10T19:10:07.681719+0000","last_became_peered":"2024-09-10T19:10:07.681719+0000","last_unstale":"2024-09-10T19:11:25.111544+0000","last_undegraded":"2024-09-10T19:11:25.111544+0000","last_fullsized":"2024-09-10T19:11:25.111544+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'2","last_scrub_stamp":"2024-09-10T19:11:25.111429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:11:25.111429+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:12:31.423912+0000","scrub_duration":0.0014791139999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":14,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":27,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-10T19:11:23.206496+0000","last_change":"2024-09-10T19:10:11.698643+0000","last_active":"2024-09-10T19:11:23.206496+0000","last_peered":"2024-09-10T19:11:23.206496+0000","last_clean":"2024-09-10T19:11:23.206496+0000","last_became_active":"2024-09-10T19:10:11.698354+0000","last_became_peered":"2024-09-10T19:10:11.698354+0000","last_unstale":"2024-09-10T19:11:23.206496+0000","last_undegraded":"2024-09-10T19:11:23.206496+0000","last_fullsized":"2024-09-10T19:11:23.206496+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:10:10.676876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":35,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-10T19:11:42.723597+0000","last_change":"2024-09-10T19:11:42.723535+0000","last_active":"2024-09-10T19:11:42.723597+0000","last_peered":"2024-09-10T19:11:42.723597+0000","last_clean":"2024-09-10T19:11:42.723597+0000","last_became_active":"2024-09-10T19:10:13.718962+0000","last_became_peered":"2024-09-10T19:10:13.718962+0000","last_unstale":"2024-09-10T19:11:42.723597+0000","last_undegraded":"2024-09-10T19:11:42.723597+0000","last_fullsized":"2024-09-10T19:11:42.723597+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:11:42.723437+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:11:42.723437+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:13:12.077483+0000","scrub_duration":0.001385631,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.16","version":"66'9","reported_seq":55,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-10T19:11:23.236785+0000","last_change":"2024-09-10T19:11:11.962017+0000","last_active":"2024-09-10T19:11:23.236785+0000","last_peered":"2024-09-10T19:11:23.236785+0000","last_clean":"2024-09-10T19:11:23.236785+0000","last_became_active":"2024-09-10T19:10:09.822994+0000","last_became_peered":"2024-09-10T19:10:09.822994+0000","last_unstale":"2024-09-10T19:11:23.236785+0000","last_undegraded":"2024-09-10T19:11:23.236785+0000","last_fullsized":"2024-09-10T19:11:23.236785+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'9","last_scrub_stamp":"2024-09-10T19:11:11.961960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:11:11.961960+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:12:16.945811+0000","scrub_duration":0.001903287,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":41,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-10T19:11:41.458628+0000","last_change":"2024-09-10T19:11:41.458577+0000","last_active":"2024-09-10T19:11:41.458628+0000","last_peered":"2024-09-10T19:11:41.458628+0000","last_clean":"2024-09-10T19:11:41.458628+0000","last_became_active":"2024-09-10T19:10:07.678076+0000","last_became_peered":"2024-09-10T19:10:07.678076+0000","last_unstale":"2024-09-10T19:11:41.458628+0000","last_undegraded":"2024-09-10T19:11:41.458628+0000","last_fullsized":"2024-09-10T19:11:41.458628+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:11:41.458523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:11:41.458523+0000","last_clean_scrub_stamp":"2024-09-10T19:11:41.458523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:12:47.189060+0000","scrub_duration":0.0012499830000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":27,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-10T19:11:23.208376+0000","last_change":"2024-09-10T19:10:11.693961+0000","last_active":"2024-09-10T19:11:23.208376+0000","last_peered":"2024-09-10T19:11:23.208376+0000","last_clean":"2024-09-10T19:11:23.208376+0000","last_became_active":"2024-09-10T19:10:11.693407+0000","last_became_peered":"2024-09-10T19:10:11.693407+0000","last_unstale":"2024-09-10T19:11:23.208376+0000","last_undegraded":"2024-09-10T19:11:23.208376+0000","last_fullsized":"2024-09-10T19:11:23.208376+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:10:10.676876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"66'1","reported_seq":36,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-10T19:11:28.192833+0000","last_change":"2024-09-10T19:11:28.192794+0000","last_active":"2024-09-10T19:11:28.192833+0000","last_peered":"2024-09-10T19:11:28.192833+0000","last_clean":"2024-09-10T19:11:28.192833+0000","last_became_active":"2024-09-10T19:10:13.717001+0000","last_became_peered":"2024-09-10T19:10:13.717001+0000","last_unstale":"2024-09-10T19:11:28.192833+0000","last_undegraded":"2024-09-10T19:11:28.192833+0000","last_fullsized":"2024-09-10T19:11:28.192833+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'1","last_scrub_stamp":"2024-09-10T19:11:28.192745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:11:28.192745+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:12:37.727108+0000","scrub_duration":0.001276323,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":41,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-10T19:11:32.859223+0000","last_change":"2024-09-10T19:11:32.859175+0000","last_active":"2024-09-10T19:11:32.859223+0000","last_peered":"2024-09-10T19:11:32.859223+0000","last_clean":"2024-09-10T19:11:32.859223+0000","last_became_active":"2024-09-10T19:10:07.682963+0000","last_became_peered":"2024-09-10T19:10:07.682963+0000","last_unstale":"2024-09-10T19:11:32.859223+0000","last_undegraded":"2024-09-10T19:11:32.859223+0000","last_fullsized":"2024-09-10T19:11:32.859223+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:11:32.859106+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:11:32.859106+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:12:47.279562+0000","scrub_duration":0.0014403980000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"66'6","reported_seq":38,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-10T19:11:23.208440+0000","last_change":"2024-09-10T19:10:09.822283+0000","last_active":"2024-09-10T19:11:23.208440+0000","last_peered":"2024-09-10T19:11:23.208440+0000","last_clean":"2024-09-10T19:11:23.208440+0000","last_became_active":"2024-09-10T19:10:09.821906+0000","last_became_peered":"2024-09-10T19:10:09.821906+0000","last_unstale":"2024-09-10T19:11:23.208440+0000","last_undegraded":"2024-09-10T19:11:23.208440+0000","last_fullsized":"2024-09-10T19:11:23.208440+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:10:08.671214+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":37,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-10T19:11:23.264929+0000","last_change":"2024-09-10T19:11:23.264870+0000","last_active":"2024-09-10T19:11:23.264929+0000","last_peered":"2024-09-10T19:11:23.264929+0000","last_clean":"2024-09-10T19:11:23.264929+0000","last_became_active":"2024-09-10T19:10:11.692943+0000","last_became_peered":"2024-09-10T19:10:11.692943+0000","last_unstale":"2024-09-10T19:11:23.264929+0000","last_undegraded":"2024-09-10T19:11:23.264929+0000","last_fullsized":"2024-09-10T19:11:23.264929+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:11:23.264793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:11:23.264793+0000","last_clean_scrub_stamp":"2024-09-10T19:11:23.264793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:12:44.248872+0000","scrub_duration":0.0015950649999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":35,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-10T19:11:23.238051+0000","last_change":"2024-09-10T19:11:13.342070+0000","last_active":"2024-09-10T19:11:23.238051+0000","last_peered":"2024-09-10T19:11:23.238051+0000","last_clean":"2024-09-10T19:11:23.238051+0000","last_became_active":"2024-09-10T19:10:14.054360+0000","last_became_peered":"2024-09-10T19:10:14.054360+0000","last_unstale":"2024-09-10T19:11:23.238051+0000","last_undegraded":"2024-09-10T19:11:23.238051+0000","last_fullsized":"2024-09-10T19:11:23.238051+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:11:13.341970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:11:13.341970+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:12:30.833886+0000","scrub_duration":0.001537725,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"80'15","reported_seq":59,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-10T19:11:41.719437+0000","last_change":"2024-09-10T19:11:41.719367+0000","last_active":"2024-09-10T19:11:41.719437+0000","last_peered":"2024-09-10T19:11:41.719437+0000","last_clean":"2024-09-10T19:11:41.719437+0000","last_became_active":"2024-09-10T19:10:09.821810+0000","last_became_peered":"2024-09-10T19:10:09.821810+0000","last_unstale":"2024-09-10T19:11:41.719437+0000","last_undegraded":"2024-09-10T19:11:41.719437+0000","last_fullsized":"2024-09-10T19:11:41.719437+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"80'15","last_scrub_stamp":"2024-09-10T19:11:41.719310+0000","last_deep_scrub":"80'15","last_deep_scrub_stamp":"2024-09-10T19:11:41.719310+0000","last_clean_scrub_stamp":"2024-09-10T19:11:41.719310+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:12:47.895535+0000","scrub_duration":0.0046706430000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":41,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-10T19:11:23.236652+0000","last_change":"2024-09-10T19:11:12.081772+0000","last_active":"2024-09-10T19:11:23.236652+0000","last_peered":"2024-09-10T19:11:23.236652+0000","last_clean":"2024-09-10T19:11:23.236652+0000","last_became_active":"2024-09-10T19:10:07.685870+0000","last_became_peered":"2024-09-10T19:10:07.685870+0000","last_unstale":"2024-09-10T19:11:23.236652+0000","last_undegraded":"2024-09-10T19:11:23.236652+0000","last_fullsized":"2024-09-10T19:11:23.236652+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:11:12.081691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:11:12.081691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:12:41.469111+0000","scrub_duration":0.001629903,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"66'8","reported_seq":54,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-10T19:11:40.767910+0000","last_change":"2024-09-10T19:10:11.700895+0000","last_active":"2024-09-10T19:11:40.767910+0000","last_peered":"2024-09-10T19:11:40.767910+0000","last_clean":"2024-09-10T19:11:40.767910+0000","last_became_active":"2024-09-10T19:10:11.700584+0000","last_became_peered":"2024-09-10T19:10:11.700584+0000","last_unstale":"2024-09-10T19:11:40.767910+0000","last_undegraded":"2024-09-10T19:11:40.767910+0000","last_fullsized":"2024-09-10T19:11:40.767910+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:10:10.676876+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":35,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-10T19:11:23.236504+0000","last_change":"2024-09-10T19:11:20.881495+0000","last_active":"2024-09-10T19:11:23.236504+0000","last_peered":"2024-09-10T19:11:23.236504+0000","last_clean":"2024-09-10T19:11:23.236504+0000","last_became_active":"2024-09-10T19:10:13.721035+0000","last_became_peered":"2024-09-10T19:10:13.721035+0000","last_unstale":"2024-09-10T19:11:23.236504+0000","last_undegraded":"2024-09-10T19:11:23.236504+0000","last_fullsized":"2024-09-10T19:11:23.236504+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:11:20.881436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:11:20.881436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:12:49.668407+0000","scrub_duration":0.0014551099999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"66'9","reported_seq":55,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-10T19:11:37.394672+0000","last_change":"2024-09-10T19:11:37.394628+0000","last_active":"2024-09-10T19:11:37.394672+0000","last_peered":"2024-09-10T19:11:37.394672+0000","last_clean":"2024-09-10T19:11:37.394672+0000","last_became_active":"2024-09-10T19:10:09.823006+0000","last_became_peered":"2024-09-10T19:10:09.823006+0000","last_unstale":"2024-09-10T19:11:37.394672+0000","last_undegraded":"2024-09-10T19:11:37.394672+0000","last_fullsized":"2024-09-10T19:11:37.394672+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'9","last_scrub_stamp":"2024-09-10T19:11:37.394573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:11:37.394573+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:12:49.673447+0000","scrub_duration":0.00176055,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":41,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-10T19:11:29.796517+0000","last_change":"2024-09-10T19:11:29.796484+0000","last_active":"2024-09-10T19:11:29.796517+0000","last_peered":"2024-09-10T19:11:29.796517+0000","last_clean":"2024-09-10T19:11:29.796517+0000","last_became_active":"2024-09-10T19:10:07.688499+0000","last_became_peered":"2024-09-10T19:10:07.688499+0000","last_unstale":"2024-09-10T19:11:29.796517+0000","last_undegraded":"2024-09-10T19:11:29.796517+0000","last_fullsized":"2024-09-10T19:11:29.796517+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:11:29.796429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:11:29.796429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:12:53.847123+0000","scrub_duration":0.0012420479999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"66'8","reported_seq":64,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-10T19:11:40.767762+0000","last_change":"2024-09-10T19:11:23.870113+0000","last_active":"2024-09-10T19:11:40.767762+0000","last_peered":"2024-09-10T19:11:40.767762+0000","last_clean":"2024-09-10T19:11:40.767762+0000","last_became_active":"2024-09-10T19:10:11.692359+0000","last_became_peered":"2024-09-10T19:10:11.692359+0000","last_unstale":"2024-09-10T19:11:40.767762+0000","last_undegraded":"2024-09-10T19:11:40.767762+0000","last_fullsized":"2024-09-10T19:11:40.767762+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'8","last_scrub_stamp":"2024-09-10T19:11:23.870010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:11:23.870010+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:12:47.854105+0000","scrub_duration":0.0017621309999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":35,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-10T19:11:43.071854+0000","last_change":"2024-09-10T19:11:43.071796+0000","last_active":"2024-09-10T19:11:43.071854+0000","last_peered":"2024-09-10T19:11:43.071854+0000","last_clean":"2024-09-10T19:11:43.071854+0000","last_became_active":"2024-09-10T19:10:13.718285+0000","last_became_peered":"2024-09-10T19:10:13.718285+0000","last_unstale":"2024-09-10T19:11:43.071854+0000","last_undegraded":"2024-09-10T19:11:43.071854+0000","last_fullsized":"2024-09-10T19:11:43.071854+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:11:43.071727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:11:43.071727+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:13:09.326836+0000","scrub_duration":0.00142789,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.12","version":"79'13","reported_seq":59,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-10T19:11:23.236505+0000","last_change":"2024-09-10T19:11:22.072464+0000","last_active":"2024-09-10T19:11:23.236505+0000","last_peered":"2024-09-10T19:11:23.236505+0000","last_clean":"2024-09-10T19:11:23.236505+0000","last_became_active":"2024-09-10T19:10:09.689440+0000","last_became_peered":"2024-09-10T19:10:09.689440+0000","last_unstale":"2024-09-10T19:11:23.236505+0000","last_undegraded":"2024-09-10T19:11:23.236505+0000","last_fullsized":"2024-09-10T19:11:23.236505+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'13","last_scrub_stamp":"2024-09-10T19:11:22.072400+0000","last_deep_scrub":"79'13","last_deep_scrub_stamp":"2024-09-10T19:11:22.072400+0000","last_clean_scrub_stamp":"2024-09-10T19:11:22.072400+0000","objects_scrubbed":4,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:12:33.209191+0000","scrub_duration":0.0032983940000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"59'1","reported_seq":38,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-10T19:11:23.207149+0000","last_change":"2024-09-10T19:10:07.683567+0000","last_active":"2024-09-10T19:11:23.207149+0000","last_peered":"2024-09-10T19:11:23.207149+0000","last_clean":"2024-09-10T19:11:23.207149+0000","last_became_active":"2024-09-10T19:10:07.683263+0000","last_became_peered":"2024-09-10T19:10:07.683263+0000","last_unstale":"2024-09-10T19:11:23.207149+0000","last_undegraded":"2024-09-10T19:11:23.207149+0000","last_fullsized":"2024-09-10T19:11:23.207149+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:10:06.664709+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":362,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":27,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-10T19:11:23.208305+0000","last_change":"2024-09-10T19:10:11.701754+0000","last_active":"2024-09-10T19:11:23.208305+0000","last_peered":"2024-09-10T19:11:23.208305+0000","last_clean":"2024-09-10T19:11:23.208305+0000","last_became_active":"2024-09-10T19:10:11.701416+0000","last_became_peered":"2024-09-10T19:10:11.701416+0000","last_unstale":"2024-09-10T19:11:23.208305+0000","last_undegraded":"2024-09-10T19:11:23.208305+0000","last_fullsized":"2024-09-10T19:11:23.208305+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:10:10.676876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":25,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-10T19:11:23.235949+0000","last_change":"2024-09-10T19:10:13.716464+0000","last_active":"2024-09-10T19:11:23.235949+0000","last_peered":"2024-09-10T19:11:23.235949+0000","last_clean":"2024-09-10T19:11:23.235949+0000","last_became_active":"2024-09-10T19:10:13.716198+0000","last_became_peered":"2024-09-10T19:10:13.716198+0000","last_unstale":"2024-09-10T19:11:23.235949+0000","last_undegraded":"2024-09-10T19:11:23.235949+0000","last_fullsized":"2024-09-10T19:11:23.235949+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:10:12.693598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"66'11","reported_seq":58,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-10T19:11:44.025460+0000","last_change":"2024-09-10T19:11:44.025426+0000","last_active":"2024-09-10T19:11:44.025460+0000","last_peered":"2024-09-10T19:11:44.025460+0000","last_clean":"2024-09-10T19:11:44.025460+0000","last_became_active":"2024-09-10T19:10:09.693856+0000","last_became_peered":"2024-09-10T19:10:09.693856+0000","last_unstale":"2024-09-10T19:11:44.025460+0000","last_undegraded":"2024-09-10T19:11:44.025460+0000","last_fullsized":"2024-09-10T19:11:44.025460+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'11","last_scrub_stamp":"2024-09-10T19:11:44.025361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:11:44.025361+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:13:09.366530+0000","scrub_duration":0.002036894,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":41,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-10T19:11:23.236972+0000","last_change":"2024-09-10T19:11:10.031299+0000","last_active":"2024-09-10T19:11:23.236972+0000","last_peered":"2024-09-10T19:11:23.236972+0000","last_clean":"2024-09-10T19:11:23.236972+0000","last_became_active":"2024-09-10T19:10:07.685339+0000","last_became_peered":"2024-09-10T19:10:07.685339+0000","last_unstale":"2024-09-10T19:11:23.236972+0000","last_undegraded":"2024-09-10T19:11:23.236972+0000","last_fullsized":"2024-09-10T19:11:23.236972+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:11:10.031245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:11:10.031245+0000","last_clean_scrub_stamp":"2024-09-10T19:11:10.031245+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:12:27.675011+0000","scrub_duration":0.0014113439999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":37,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-10T19:11:31.249435+0000","last_change":"2024-09-10T19:11:31.249400+0000","last_active":"2024-09-10T19:11:31.249435+0000","last_peered":"2024-09-10T19:11:31.249435+0000","last_clean":"2024-09-10T19:11:31.249435+0000","last_became_active":"2024-09-10T19:10:11.694564+0000","last_became_peered":"2024-09-10T19:10:11.694564+0000","last_unstale":"2024-09-10T19:11:31.249435+0000","last_undegraded":"2024-09-10T19:11:31.249435+0000","last_fullsized":"2024-09-10T19:11:31.249435+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:11:31.249343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:11:31.249343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:12:58.372067+0000","scrub_duration":0.001093644,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":35,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-10T19:11:23.206824+0000","last_change":"2024-09-10T19:11:16.813629+0000","last_active":"2024-09-10T19:11:23.206824+0000","last_peered":"2024-09-10T19:11:23.206824+0000","last_clean":"2024-09-10T19:11:23.206824+0000","last_became_active":"2024-09-10T19:10:13.724201+0000","last_became_peered":"2024-09-10T19:10:13.724201+0000","last_unstale":"2024-09-10T19:11:23.206824+0000","last_undegraded":"2024-09-10T19:11:23.206824+0000","last_fullsized":"2024-09-10T19:11:23.206824+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:11:16.813575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:11:16.813575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:12:44.427906+0000","scrub_duration":0.001237312,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"66'4","reported_seq":45,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-10T19:11:23.207029+0000","last_change":"2024-09-10T19:11:14.816693+0000","last_active":"2024-09-10T19:11:23.207029+0000","last_peered":"2024-09-10T19:11:23.207029+0000","last_clean":"2024-09-10T19:11:23.207029+0000","last_became_active":"2024-09-10T19:10:09.692751+0000","last_became_peered":"2024-09-10T19:10:09.692751+0000","last_unstale":"2024-09-10T19:11:23.207029+0000","last_undegraded":"2024-09-10T19:11:23.207029+0000","last_fullsized":"2024-09-10T19:11:23.207029+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'4","last_scrub_stamp":"2024-09-10T19:11:14.816632+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:11:14.816632+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:12:42.695105+0000","scrub_duration":0.001902504,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":41,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-10T19:11:24.275104+0000","last_change":"2024-09-10T19:11:24.275061+0000","last_active":"2024-09-10T19:11:24.275104+0000","last_peered":"2024-09-10T19:11:24.275104+0000","last_clean":"2024-09-10T19:11:24.275104+0000","last_became_active":"2024-09-10T19:10:07.687723+0000","last_became_peered":"2024-09-10T19:10:07.687723+0000","last_unstale":"2024-09-10T19:11:24.275104+0000","last_undegraded":"2024-09-10T19:11:24.275104+0000","last_fullsized":"2024-09-10T19:11:24.275104+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:11:24.275004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:11:24.275004+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:12:41.723440+0000","scrub_duration":0.001588014,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":37,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-10T19:11:31.838045+0000","last_change":"2024-09-10T19:11:31.838005+0000","last_active":"2024-09-10T19:11:31.838045+0000","last_peered":"2024-09-10T19:11:31.838045+0000","last_clean":"2024-09-10T19:11:31.838045+0000","last_became_active":"2024-09-10T19:10:11.698536+0000","last_became_peered":"2024-09-10T19:10:11.698536+0000","last_unstale":"2024-09-10T19:11:31.838045+0000","last_undegraded":"2024-09-10T19:11:31.838045+0000","last_fullsized":"2024-09-10T19:11:31.838045+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:11:31.837944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:11:31.837944+0000","last_clean_scrub_stamp":"2024-09-10T19:11:31.837944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:12:53.987457+0000","scrub_duration":0.001263639,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":35,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-10T19:11:23.237565+0000","last_change":"2024-09-10T19:11:14.298616+0000","last_active":"2024-09-10T19:11:23.237565+0000","last_peered":"2024-09-10T19:11:23.237565+0000","last_clean":"2024-09-10T19:11:23.237565+0000","last_became_active":"2024-09-10T19:10:13.718271+0000","last_became_peered":"2024-09-10T19:10:13.718271+0000","last_unstale":"2024-09-10T19:11:23.237565+0000","last_undegraded":"2024-09-10T19:11:23.237565+0000","last_fullsized":"2024-09-10T19:11:23.237565+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:11:14.298545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:11:14.298545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:12:15.177239+0000","scrub_duration":0.001561799,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.11","version":"66'11","reported_seq":58,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-10T19:11:36.743566+0000","last_change":"2024-09-10T19:11:36.743509+0000","last_active":"2024-09-10T19:11:36.743566+0000","last_peered":"2024-09-10T19:11:36.743566+0000","last_clean":"2024-09-10T19:11:36.743566+0000","last_became_active":"2024-09-10T19:10:09.821357+0000","last_became_peered":"2024-09-10T19:10:09.821357+0000","last_unstale":"2024-09-10T19:11:36.743566+0000","last_undegraded":"2024-09-10T19:11:36.743566+0000","last_fullsized":"2024-09-10T19:11:36.743566+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'11","last_scrub_stamp":"2024-09-10T19:11:36.743445+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:11:36.743445+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:13:04.557447+0000","scrub_duration":0.001722374,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":41,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-10T19:11:34.868035+0000","last_change":"2024-09-10T19:11:34.867991+0000","last_active":"2024-09-10T19:11:34.868035+0000","last_peered":"2024-09-10T19:11:34.868035+0000","last_clean":"2024-09-10T19:11:34.868035+0000","last_became_active":"2024-09-10T19:10:07.685223+0000","last_became_peered":"2024-09-10T19:10:07.685223+0000","last_unstale":"2024-09-10T19:11:34.868035+0000","last_undegraded":"2024-09-10T19:11:34.868035+0000","last_fullsized":"2024-09-10T19:11:34.868035+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:11:34.867935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:11:34.867935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:12:56.224806+0000","scrub_duration":0.001504086,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":27,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-10T19:11:23.237471+0000","last_change":"2024-09-10T19:10:11.699248+0000","last_active":"2024-09-10T19:11:23.237471+0000","last_peered":"2024-09-10T19:11:23.237471+0000","last_clean":"2024-09-10T19:11:23.237471+0000","last_became_active":"2024-09-10T19:10:11.699013+0000","last_became_peered":"2024-09-10T19:10:11.699013+0000","last_unstale":"2024-09-10T19:11:23.237471+0000","last_undegraded":"2024-09-10T19:11:23.237471+0000","last_fullsized":"2024-09-10T19:11:23.237471+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:10:10.676876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":25,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-10T19:11:23.207655+0000","last_change":"2024-09-10T19:10:14.056524+0000","last_active":"2024-09-10T19:11:23.207655+0000","last_peered":"2024-09-10T19:11:23.207655+0000","last_clean":"2024-09-10T19:11:23.207655+0000","last_became_active":"2024-09-10T19:10:14.056172+0000","last_became_peered":"2024-09-10T19:10:14.056172+0000","last_unstale":"2024-09-10T19:11:23.207655+0000","last_undegraded":"2024-09-10T19:11:23.207655+0000","last_fullsized":"2024-09-10T19:11:23.207655+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:10:12.693598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"80'2","reported_seq":37,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-10T19:11:30.272144+0000","last_change":"2024-09-10T19:11:30.272094+0000","last_active":"2024-09-10T19:11:30.272144+0000","last_peered":"2024-09-10T19:11:30.272144+0000","last_clean":"2024-09-10T19:11:30.272144+0000","last_became_active":"2024-09-10T19:10:13.718875+0000","last_became_peered":"2024-09-10T19:10:13.718875+0000","last_unstale":"2024-09-10T19:11:30.272144+0000","last_undegraded":"2024-09-10T19:11:30.272144+0000","last_fullsized":"2024-09-10T19:11:30.272144+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"80'2","last_scrub_stamp":"2024-09-10T19:11:30.272043+0000","last_deep_scrub":"80'2","last_deep_scrub_stamp":"2024-09-10T19:11:30.272043+0000","last_clean_scrub_stamp":"2024-09-10T19:11:30.272043+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:12:37.490688+0000","scrub_duration":0.0022929460000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":369,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":31,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-10T19:11:23.236211+0000","last_change":"2024-09-10T19:10:07.690146+0000","last_active":"2024-09-10T19:11:23.236211+0000","last_peered":"2024-09-10T19:11:23.236211+0000","last_clean":"2024-09-10T19:11:23.236211+0000","last_became_active":"2024-09-10T19:10:07.689877+0000","last_became_peered":"2024-09-10T19:10:07.689877+0000","last_unstale":"2024-09-10T19:11:23.236211+0000","last_undegraded":"2024-09-10T19:11:23.236211+0000","last_fullsized":"2024-09-10T19:11:23.236211+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:10:06.664709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1e","version":"80'15","reported_seq":59,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-10T19:11:25.884077+0000","last_change":"2024-09-10T19:11:25.884008+0000","last_active":"2024-09-10T19:11:25.884077+0000","last_peered":"2024-09-10T19:11:25.884077+0000","last_clean":"2024-09-10T19:11:25.884077+0000","last_became_active":"2024-09-10T19:10:09.823061+0000","last_became_peered":"2024-09-10T19:10:09.823061+0000","last_unstale":"2024-09-10T19:11:25.884077+0000","last_undegraded":"2024-09-10T19:11:25.884077+0000","last_fullsized":"2024-09-10T19:11:25.884077+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"80'15","last_scrub_stamp":"2024-09-10T19:11:25.883919+0000","last_deep_scrub":"80'15","last_deep_scrub_stamp":"2024-09-10T19:11:25.883919+0000","last_clean_scrub_stamp":"2024-09-10T19:11:25.883919+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:12:31.561992+0000","scrub_duration":0.0034715229999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1f","version":"66'8","reported_seq":67,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-10T19:11:40.767449+0000","last_change":"2024-09-10T19:11:30.835414+0000","last_active":"2024-09-10T19:11:40.767449+0000","last_peered":"2024-09-10T19:11:40.767449+0000","last_clean":"2024-09-10T19:11:40.767449+0000","last_became_active":"2024-09-10T19:10:11.695427+0000","last_became_peered":"2024-09-10T19:10:11.695427+0000","last_unstale":"2024-09-10T19:11:40.767449+0000","last_undegraded":"2024-09-10T19:11:40.767449+0000","last_fullsized":"2024-09-10T19:11:40.767449+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'8","last_scrub_stamp":"2024-09-10T19:11:30.835354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:11:30.835354+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:12:42.690611+0000","scrub_duration":0.0013545090000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":35,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-10T19:11:30.210698+0000","last_change":"2024-09-10T19:11:30.210657+0000","last_active":"2024-09-10T19:11:30.210698+0000","last_peered":"2024-09-10T19:11:30.210698+0000","last_clean":"2024-09-10T19:11:30.210698+0000","last_became_active":"2024-09-10T19:10:13.715419+0000","last_became_peered":"2024-09-10T19:10:13.715419+0000","last_unstale":"2024-09-10T19:11:30.210698+0000","last_undegraded":"2024-09-10T19:11:30.210698+0000","last_fullsized":"2024-09-10T19:11:30.210698+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:11:30.210603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:11:30.210603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:12:40.846927+0000","scrub_duration":0.0012894740000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":41,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-10T19:11:34.815887+0000","last_change":"2024-09-10T19:11:34.815849+0000","last_active":"2024-09-10T19:11:34.815887+0000","last_peered":"2024-09-10T19:11:34.815887+0000","last_clean":"2024-09-10T19:11:34.815887+0000","last_became_active":"2024-09-10T19:10:07.682689+0000","last_became_peered":"2024-09-10T19:10:07.682689+0000","last_unstale":"2024-09-10T19:11:34.815887+0000","last_undegraded":"2024-09-10T19:11:34.815887+0000","last_fullsized":"2024-09-10T19:11:34.815887+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:11:34.815794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:11:34.815794+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:13:02.432072+0000","scrub_duration":0.001252501,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"80'16","reported_seq":65,"reported_epoch":81,"state":"peering","last_fresh":"2024-09-10T19:11:43.741742+0000","last_change":"2024-09-10T19:11:43.742044+0000","last_active":"2024-09-10T19:11:43.741708+0000","last_peered":"2024-09-10T19:11:40.853711+0000","last_clean":"2024-09-10T19:11:40.853711+0000","last_became_active":"2024-09-10T19:10:09.687851+0000","last_became_peered":"2024-09-10T19:10:09.687851+0000","last_unstale":"2024-09-10T19:11:43.741742+0000","last_undegraded":"2024-09-10T19:11:43.741742+0000","last_fullsized":"2024-09-10T19:11:43.741742+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"80'16","last_scrub_stamp":"2024-09-10T19:11:40.853611+0000","last_deep_scrub":"80'16","last_deep_scrub_stamp":"2024-09-10T19:11:40.853611+0000","last_clean_scrub_stamp":"2024-09-10T19:11:40.853611+0000","objects_scrubbed":5,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:12:56.523219+0000","scrub_duration":0.003886877,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1,2,5],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":37,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-10T19:11:28.822448+0000","last_change":"2024-09-10T19:11:28.822385+0000","last_active":"2024-09-10T19:11:28.822448+0000","last_peered":"2024-09-10T19:11:28.822448+0000","last_clean":"2024-09-10T19:11:28.822448+0000","last_became_active":"2024-09-10T19:10:11.701735+0000","last_became_peered":"2024-09-10T19:10:11.701735+0000","last_unstale":"2024-09-10T19:11:28.822448+0000","last_undegraded":"2024-09-10T19:11:28.822448+0000","last_fullsized":"2024-09-10T19:11:28.822448+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:11:28.822327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:11:28.822327+0000","last_clean_scrub_stamp":"2024-09-10T19:11:28.822327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:12:45.908873+0000","scrub_duration":0.0012591119999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":35,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-10T19:11:35.145058+0000","last_change":"2024-09-10T19:11:35.145006+0000","last_active":"2024-09-10T19:11:35.145058+0000","last_peered":"2024-09-10T19:11:35.145058+0000","last_clean":"2024-09-10T19:11:35.145058+0000","last_became_active":"2024-09-10T19:10:14.053619+0000","last_became_peered":"2024-09-10T19:10:14.053619+0000","last_unstale":"2024-09-10T19:11:35.145058+0000","last_undegraded":"2024-09-10T19:11:35.145058+0000","last_fullsized":"2024-09-10T19:11:35.145058+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:11:35.144953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:11:35.144953+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:12:55.849066+0000","scrub_duration":0.001302503,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":31,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-10T19:11:23.207581+0000","last_change":"2024-09-10T19:10:07.686443+0000","last_active":"2024-09-10T19:11:23.207581+0000","last_peered":"2024-09-10T19:11:23.207581+0000","last_clean":"2024-09-10T19:11:23.207581+0000","last_became_active":"2024-09-10T19:10:07.685997+0000","last_became_peered":"2024-09-10T19:10:07.685997+0000","last_unstale":"2024-09-10T19:11:23.207581+0000","last_undegraded":"2024-09-10T19:11:23.207581+0000","last_fullsized":"2024-09-10T19:11:23.207581+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:10:06.664709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1c","version":"78'22","reported_seq":71,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-10T19:11:23.207577+0000","last_change":"2024-09-10T19:11:21.119003+0000","last_active":"2024-09-10T19:11:23.207577+0000","last_peered":"2024-09-10T19:11:23.207577+0000","last_clean":"2024-09-10T19:11:23.207577+0000","last_became_active":"2024-09-10T19:10:09.690328+0000","last_became_peered":"2024-09-10T19:10:09.690328+0000","last_unstale":"2024-09-10T19:11:23.207577+0000","last_undegraded":"2024-09-10T19:11:23.207577+0000","last_fullsized":"2024-09-10T19:11:23.207577+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"78'22","last_scrub_stamp":"2024-09-10T19:11:21.118937+0000","last_deep_scrub":"78'22","last_deep_scrub_stamp":"2024-09-10T19:11:21.118937+0000","last_clean_scrub_stamp":"2024-09-10T19:11:21.118937+0000","objects_scrubbed":7,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:12:21.535383+0000","scrub_duration":0.0048819759999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":37,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-10T19:11:43.221410+0000","last_change":"2024-09-10T19:11:43.221376+0000","last_active":"2024-09-10T19:11:43.221410+0000","last_peered":"2024-09-10T19:11:43.221410+0000","last_clean":"2024-09-10T19:11:43.221410+0000","last_became_active":"2024-09-10T19:10:11.700037+0000","last_became_peered":"2024-09-10T19:10:11.700037+0000","last_unstale":"2024-09-10T19:11:43.221410+0000","last_undegraded":"2024-09-10T19:11:43.221410+0000","last_fullsized":"2024-09-10T19:11:43.221410+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:11:43.221323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:11:43.221323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:12:47.537742+0000","scrub_duration":0.001173189,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":35,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-10T19:11:24.828019+0000","last_change":"2024-09-10T19:11:24.827983+0000","last_active":"2024-09-10T19:11:24.828019+0000","last_peered":"2024-09-10T19:11:24.828019+0000","last_clean":"2024-09-10T19:11:24.828019+0000","last_became_active":"2024-09-10T19:10:14.053698+0000","last_became_peered":"2024-09-10T19:10:14.053698+0000","last_unstale":"2024-09-10T19:11:24.828019+0000","last_undegraded":"2024-09-10T19:11:24.828019+0000","last_fullsized":"2024-09-10T19:11:24.828019+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:11:24.827916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:11:24.827916+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:12:36.645047+0000","scrub_duration":0.0014104860000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":41,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-10T19:11:25.870632+0000","last_change":"2024-09-10T19:11:25.870573+0000","last_active":"2024-09-10T19:11:25.870632+0000","last_peered":"2024-09-10T19:11:25.870632+0000","last_clean":"2024-09-10T19:11:25.870632+0000","last_became_active":"2024-09-10T19:10:07.682811+0000","last_became_peered":"2024-09-10T19:10:07.682811+0000","last_unstale":"2024-09-10T19:11:25.870632+0000","last_undegraded":"2024-09-10T19:11:25.870632+0000","last_fullsized":"2024-09-10T19:11:25.870632+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:11:25.870516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:11:25.870516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:12:33.650020+0000","scrub_duration":0.001292568,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1d","version":"66'12","reported_seq":52,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-10T19:11:23.208096+0000","last_change":"2024-09-10T19:10:09.693108+0000","last_active":"2024-09-10T19:11:23.208096+0000","last_peered":"2024-09-10T19:11:23.208096+0000","last_clean":"2024-09-10T19:11:23.208096+0000","last_became_active":"2024-09-10T19:10:09.692781+0000","last_became_peered":"2024-09-10T19:10:09.692781+0000","last_unstale":"2024-09-10T19:11:23.208096+0000","last_undegraded":"2024-09-10T19:11:23.208096+0000","last_fullsized":"2024-09-10T19:11:23.208096+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:10:08.671214+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":27,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-10T19:11:23.237002+0000","last_change":"2024-09-10T19:10:11.701419+0000","last_active":"2024-09-10T19:11:23.237002+0000","last_peered":"2024-09-10T19:11:23.237002+0000","last_clean":"2024-09-10T19:11:23.237002+0000","last_became_active":"2024-09-10T19:10:11.701155+0000","last_became_peered":"2024-09-10T19:10:11.701155+0000","last_unstale":"2024-09-10T19:11:23.237002+0000","last_undegraded":"2024-09-10T19:11:23.237002+0000","last_fullsized":"2024-09-10T19:11:23.237002+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:10:10.676876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":66,"ondisk_log_size":66,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":172,"num_object_clones":0,"num_object_copies":516,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":172,"num_whiteouts":0,"num_read":683,"num_read_kb":445,"num_write":407,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":414,"ondisk_log_size":414,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":91,"num_read_kb":86,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1328,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":27,"num_read_kb":27,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":3984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4,"ondisk_log_size":4,"up":96,"acting":96,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79,"ondisk_log_size":79,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":54,"seq":231928234008,"num_pgs":57,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298820,"kb_used_data":1232,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93446332,"statfs":{"total":95995035648,"available":95689043968,"internally_reserved":0,"allocated":1261568,"data_stored":915341,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 19:10:56 2024","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.53500000000000003},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.70499999999999996,"5min":0.70499999999999996,"15min":0.70499999999999996},"last":0.46000000000000002},{"interface":"front","average":{"1min":0.53800000000000003,"5min":0.53800000000000003,"15min":0.53800000000000003},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.76200000000000001,"5min":0.76200000000000001,"15min":0.76200000000000001},"last":0.41799999999999998}]},{"osd":1,"last update":"Tue Sep 10 19:10:56 2024","interfaces":[{"interface":"back","average":{"1min":0.52400000000000002,"5min":0.52400000000000002,"15min":0.52400000000000002},"min":{"1min":0.41299999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":0.79400000000000004,"5min":0.79400000000000004,"15min":0.79400000000000004},"last":0.36199999999999999},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"min":{"1min":0.314,"5min":0.314,"15min":0.314},"max":{"1min":0.71799999999999997,"5min":0.71799999999999997,"15min":0.71799999999999997},"last":0.44800000000000001}]},{"osd":2,"last update":"Tue Sep 10 19:10:56 2024","interfaces":[{"interface":"back","average":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"min":{"1min":0.29199999999999998,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.84399999999999997,"5min":0.84399999999999997,"15min":0.84399999999999997},"last":0.46999999999999997},{"interface":"front","average":{"1min":0.55000000000000004,"5min":0.55000000000000004,"15min":0.55000000000000004},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.77300000000000002,"5min":0.77300000000000002,"15min":0.77300000000000002},"last":0.437}]},{"osd":3,"last update":"Tue Sep 10 19:10:56 2024","interfaces":[{"interface":"back","average":{"1min":0.53600000000000003,"5min":0.53600000000000003,"15min":0.53600000000000003},"min":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":0.86399999999999999,"5min":0.86399999999999999,"15min":0.86399999999999999},"last":0.42699999999999999},{"interface":"front","average":{"1min":0.52000000000000002,"5min":0.52000000000000002,"15min":0.52000000000000002},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.77600000000000002,"5min":0.77600000000000002,"15min":0.77600000000000002},"last":0.40899999999999997}]},{"osd":4,"last update":"Tue Sep 10 19:10:56 2024","interfaces":[{"interface":"back","average":{"1min":0.45500000000000002,"5min":0.45500000000000002,"15min":0.45500000000000002},"min":{"1min":0.318,"5min":0.318,"15min":0.318},"max":{"1min":0.65000000000000002,"5min":0.65000000000000002,"15min":0.65000000000000002},"last":0.26900000000000002},{"interface":"front","average":{"1min":0.46000000000000002,"5min":0.46000000000000002,"15min":0.46000000000000002},"min":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.60599999999999998,"5min":0.60599999999999998,"15min":0.60599999999999998},"last":0.39800000000000002}]},{"osd":5,"last update":"Tue Sep 10 19:10:56 2024","interfaces":[{"interface":"back","average":{"1min":0.47999999999999998,"5min":0.47999999999999998,"15min":0.47999999999999998},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.76100000000000001,"5min":0.76100000000000001,"15min":0.76100000000000001},"last":0.48399999999999999},{"interface":"front","average":{"1min":0.47599999999999998,"5min":0.47599999999999998,"15min":0.47599999999999998},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.68600000000000005,"5min":0.68600000000000005,"15min":0.68600000000000005},"last":0.30199999999999999}]},{"osd":6,"last update":"Tue Sep 10 19:10:56 2024","interfaces":[{"interface":"back","average":{"1min":0.5,"5min":0.5,"15min":0.5},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":0.65900000000000003,"5min":0.65900000000000003,"15min":0.65900000000000003},"last":0.374},{"interface":"front","average":{"1min":0.48899999999999999,"5min":0.48899999999999999,"15min":0.48899999999999999},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.67000000000000004,"5min":0.67000000000000004,"15min":0.67000000000000004},"last":0.34300000000000003}]}]},{"osd":6,"up_from":47,"seq":201863462941,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298808,"kb_used_data":1220,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93446344,"statfs":{"total":95995035648,"available":95689056256,"internally_reserved":0,"allocated":1249280,"data_stored":915467,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 19:11:36 2024","interfaces":[{"interface":"back","average":{"1min":0.48399999999999999,"5min":0.51400000000000001,"15min":0.51900000000000002},"min":{"1min":0.309,"5min":0.309,"15min":0.309},"max":{"1min":0.67800000000000005,"5min":0.67800000000000005,"15min":0.67800000000000005},"last":0.377},{"interface":"front","average":{"1min":0.47699999999999998,"5min":0.53000000000000003,"15min":0.53900000000000003},"min":{"1min":0.308,"5min":0.308,"15min":0.308},"max":{"1min":0.59999999999999998,"5min":0.67200000000000004,"15min":0.67200000000000004},"last":0.56100000000000005}]},{"osd":1,"last update":"Tue Sep 10 19:11:36 2024","interfaces":[{"interface":"back","average":{"1min":0.501,"5min":0.52300000000000002,"15min":0.52700000000000002},"min":{"1min":0.318,"5min":0.318,"15min":0.318},"max":{"1min":0.63900000000000001,"5min":0.67400000000000004,"15min":0.67400000000000004},"last":0.39600000000000002},{"interface":"front","average":{"1min":0.48499999999999999,"5min":0.52100000000000002,"15min":0.52700000000000002},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.63,"5min":0.65800000000000003,"15min":0.65800000000000003},"last":0.46000000000000002}]},{"osd":2,"last update":"Tue Sep 10 19:11:36 2024","interfaces":[{"interface":"back","average":{"1min":0.5,"5min":0.53600000000000003,"15min":0.54200000000000004},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.65100000000000002,"5min":0.66400000000000003,"15min":0.66400000000000003},"last":0.50600000000000001},{"interface":"front","average":{"1min":0.501,"5min":0.50800000000000001,"15min":0.50900000000000001},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.78600000000000003,"5min":0.78600000000000003,"15min":0.78600000000000003},"last":0.36099999999999999}]},{"osd":3,"last update":"Tue Sep 10 19:11:36 2024","interfaces":[{"interface":"back","average":{"1min":0.47799999999999998,"5min":0.50800000000000001,"15min":0.51300000000000001},"min":{"1min":0.32100000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.64600000000000002,"5min":0.68100000000000005,"15min":0.68100000000000005},"last":0.34000000000000002},{"interface":"front","average":{"1min":0.52900000000000003,"5min":0.55800000000000005,"15min":0.56299999999999994},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.69499999999999995,"5min":0.70199999999999996,"15min":0.70199999999999996},"last":0.52900000000000003}]},{"osd":4,"last update":"Tue Sep 10 19:11:36 2024","interfaces":[{"interface":"back","average":{"1min":0.40899999999999997,"5min":0.35999999999999999,"15min":0.35199999999999998},"min":{"1min":0.25700000000000001,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":0.64700000000000002,"5min":0.64700000000000002,"15min":0.64700000000000002},"last":0.41299999999999998},{"interface":"front","average":{"1min":0.40899999999999997,"5min":0.38100000000000001,"15min":0.377},"min":{"1min":0.27800000000000002,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.58999999999999997,"5min":0.58999999999999997,"15min":0.58999999999999997},"last":0.433}]},{"osd":5,"last update":"Tue Sep 10 19:11:36 2024","interfaces":[{"interface":"back","average":{"1min":0.433,"5min":0.38600000000000001,"15min":0.378},"min":{"1min":0.27100000000000002,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.60699999999999998,"5min":0.60699999999999998,"15min":0.60699999999999998},"last":0.58399999999999996},{"interface":"front","average":{"1min":0.46700000000000003,"5min":0.39200000000000002,"15min":0.38},"min":{"1min":0.318,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.66000000000000003,"5min":0.66000000000000003,"15min":0.66000000000000003},"last":0.51900000000000002}]},{"osd":7,"last update":"Tue Sep 10 19:10:54 2024","interfaces":[{"interface":"back","average":{"1min":0.44700000000000001,"5min":0.44700000000000001,"15min":0.44700000000000001},"min":{"1min":0.29899999999999999,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.68799999999999994,"5min":0.68799999999999994,"15min":0.68799999999999994},"last":0.54900000000000004},{"interface":"front","average":{"1min":0.44500000000000001,"5min":0.44500000000000001,"15min":0.44500000000000001},"min":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.55000000000000004,"5min":0.55000000000000004,"15min":0.55000000000000004},"last":0.47799999999999998}]}]},{"osd":5,"up_from":39,"seq":167503724578,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298232,"kb_used_data":644,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93446920,"statfs":{"total":95995035648,"available":95689646080,"internally_reserved":0,"allocated":659456,"data_stored":325980,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 19:11:08 2024","interfaces":[{"interface":"back","average":{"1min":0.51900000000000002,"5min":0.48299999999999998,"15min":0.47699999999999998},"min":{"1min":0.38200000000000001,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.67200000000000004,"5min":0.67200000000000004,"15min":0.67200000000000004},"last":0.39400000000000002},{"interface":"front","average":{"1min":0.53600000000000003,"5min":0.48399999999999999,"15min":0.47499999999999998},"min":{"1min":0.38300000000000001,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.75600000000000001,"5min":0.75600000000000001,"15min":0.75600000000000001},"last":0.40100000000000002}]},{"osd":1,"last update":"Tue Sep 10 19:11:08 2024","interfaces":[{"interface":"back","average":{"1min":0.58399999999999996,"5min":0.51900000000000002,"15min":0.50800000000000001},"min":{"1min":0.46800000000000003,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.72799999999999998,"5min":0.72799999999999998,"15min":0.72799999999999998},"last":0.33700000000000002},{"interface":"front","average":{"1min":0.50700000000000001,"5min":0.47899999999999998,"15min":0.47399999999999998},"min":{"1min":0.39900000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.67600000000000005,"5min":0.68799999999999994,"15min":0.68799999999999994},"last":0.38300000000000001}]},{"osd":2,"last update":"Tue Sep 10 19:11:08 2024","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.48599999999999999,"15min":0.47799999999999998},"min":{"1min":0.39900000000000002,"5min":0.309,"15min":0.309},"max":{"1min":0.70699999999999996,"5min":0.70699999999999996,"15min":0.70699999999999996},"last":0.374},{"interface":"front","average":{"1min":0.54100000000000004,"5min":0.498,"15min":0.49099999999999999},"min":{"1min":0.39200000000000002,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.70799999999999996,"5min":0.70799999999999996,"15min":0.70799999999999996},"last":0.36399999999999999}]},{"osd":3,"last update":"Tue Sep 10 19:11:08 2024","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.52200000000000002,"15min":0.51400000000000001},"min":{"1min":0.437,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.74099999999999999,"5min":0.74099999999999999,"15min":0.74099999999999999},"last":0.41499999999999998},{"interface":"front","average":{"1min":0.59899999999999998,"5min":0.53800000000000003,"15min":0.52800000000000002},"min":{"1min":0.44700000000000001,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.747,"5min":0.747,"15min":0.747},"last":0.42999999999999999}]},{"osd":4,"last update":"Tue Sep 10 19:11:08 2024","interfaces":[{"interface":"back","average":{"1min":0.44,"5min":0.37,"15min":0.35799999999999998},"min":{"1min":0.27800000000000002,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":0.624,"5min":0.624,"15min":0.624},"last":0.27500000000000002},{"interface":"front","average":{"1min":0.46500000000000002,"5min":0.38100000000000001,"15min":0.36699999999999999},"min":{"1min":0.27800000000000002,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.76800000000000002,"5min":0.76800000000000002,"15min":0.76800000000000002},"last":0.318}]},{"osd":6,"last update":"Tue Sep 10 19:11:35 2024","interfaces":[{"interface":"back","average":{"1min":0.48399999999999999,"5min":0.42499999999999999,"15min":0.41499999999999998},"min":{"1min":0.307,"5min":0.223,"15min":0.223},"max":{"1min":0.74099999999999999,"5min":0.74099999999999999,"15min":0.74099999999999999},"last":0.441},{"interface":"front","average":{"1min":0.48099999999999998,"5min":0.42099999999999999,"15min":0.41099999999999998},"min":{"1min":0.26800000000000002,"5min":0.25,"15min":0.25},"max":{"1min":0.78000000000000003,"5min":0.78000000000000003,"15min":0.78000000000000003},"last":0.30099999999999999}]},{"osd":7,"last update":"Tue Sep 10 19:10:55 2024","interfaces":[{"interface":"back","average":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.51000000000000001},"min":{"1min":0.29099999999999998,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.72699999999999998,"5min":0.72699999999999998,"15min":0.72699999999999998},"last":0.45600000000000002},{"interface":"front","average":{"1min":0.47899999999999998,"5min":0.47899999999999998,"15min":0.47899999999999998},"min":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.64600000000000002,"5min":0.64600000000000002,"15min":0.64600000000000002},"last":0.35099999999999998}]}]},{"osd":4,"up_from":32,"seq":137438953510,"num_pgs":52,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298248,"kb_used_data":660,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93446904,"statfs":{"total":95995035648,"available":95689629696,"internally_reserved":0,"allocated":675840,"data_stored":325189,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 19:10:48 2024","interfaces":[{"interface":"back","average":{"1min":0.49399999999999999,"5min":0.53200000000000003,"15min":0.53800000000000003},"min":{"1min":0.32500000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.61699999999999999,"5min":0.91500000000000004,"15min":0.91500000000000004},"last":0.45100000000000001},{"interface":"front","average":{"1min":0.47599999999999998,"5min":0.52000000000000002,"15min":0.52700000000000002},"min":{"1min":0.34499999999999997,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.63200000000000001,"5min":0.78200000000000003,"15min":0.78200000000000003},"last":0.372}]},{"osd":1,"last update":"Tue Sep 10 19:10:48 2024","interfaces":[{"interface":"back","average":{"1min":0.5,"5min":0.51000000000000001,"15min":0.51200000000000001},"min":{"1min":0.33300000000000002,"5min":0.313,"15min":0.313},"max":{"1min":0.68799999999999994,"5min":0.83499999999999996,"15min":0.83499999999999996},"last":0.29999999999999999},{"interface":"front","average":{"1min":0.504,"5min":0.52000000000000002,"15min":0.52300000000000002},"min":{"1min":0.374,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.77100000000000002,"5min":0.77100000000000002,"15min":0.77100000000000002},"last":0.41499999999999998}]},{"osd":2,"last update":"Tue Sep 10 19:10:48 2024","interfaces":[{"interface":"back","average":{"1min":0.51100000000000001,"5min":0.53300000000000003,"15min":0.53700000000000003},"min":{"1min":0.39300000000000002,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.69599999999999995,"5min":1.0649999999999999,"15min":1.0649999999999999},"last":0.38600000000000001},{"interface":"front","average":{"1min":0.48499999999999999,"5min":0.52700000000000002,"15min":0.53400000000000003},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.66000000000000003,"5min":0.86899999999999999,"15min":0.86899999999999999},"last":0.434}]},{"osd":3,"last update":"Tue Sep 10 19:10:48 2024","interfaces":[{"interface":"back","average":{"1min":0.498,"5min":0.52100000000000002,"15min":0.52500000000000002},"min":{"1min":0.373,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.67700000000000005,"5min":0.76500000000000001,"15min":0.76500000000000001},"last":0.39700000000000002},{"interface":"front","average":{"1min":0.52300000000000002,"5min":0.54600000000000004,"15min":0.55000000000000004},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.92200000000000004,"5min":1.0249999999999999,"15min":1.0249999999999999},"last":0.44400000000000001}]},{"osd":5,"last update":"Tue Sep 10 19:11:10 2024","interfaces":[{"interface":"back","average":{"1min":0.46400000000000002,"5min":0.39200000000000002,"15min":0.38},"min":{"1min":0.34999999999999998,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.61599999999999999,"5min":0.61599999999999999,"15min":0.61599999999999999},"last":0.34799999999999998},{"interface":"front","average":{"1min":0.47099999999999997,"5min":0.40899999999999997,"15min":0.39900000000000002},"min":{"1min":0.30599999999999999,"5min":0.223,"15min":0.223},"max":{"1min":0.63,"5min":0.89100000000000001,"15min":0.89100000000000001},"last":0.28499999999999998}]},{"osd":6,"last update":"Tue Sep 10 19:11:36 2024","interfaces":[{"interface":"back","average":{"1min":0.46800000000000003,"5min":0.42399999999999999,"15min":0.41599999999999998},"min":{"1min":0.29799999999999999,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.627,"5min":0.627,"15min":0.627},"last":0.32100000000000001},{"interface":"front","average":{"1min":0.46899999999999997,"5min":0.42499999999999999,"15min":0.41699999999999998},"min":{"1min":0.28999999999999998,"5min":0.25,"15min":0.25},"max":{"1min":0.65500000000000003,"5min":0.65500000000000003,"15min":0.65500000000000003},"last":0.42499999999999999}]},{"osd":7,"last update":"Tue Sep 10 19:10:58 2024","interfaces":[{"interface":"back","average":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.72699999999999998,"5min":0.72699999999999998,"15min":0.72699999999999998},"last":0.40600000000000003},{"interface":"front","average":{"1min":0.503,"5min":0.503,"15min":0.503},"min":{"1min":0.29099999999999998,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.69899999999999995,"5min":0.69899999999999995,"15min":0.69899999999999995},"last":0.45700000000000002}]}]},{"osd":3,"up_from":26,"seq":111669149739,"num_pgs":55,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298268,"kb_used_data":680,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93446884,"statfs":{"total":95995035648,"available":95689609216,"internally_reserved":0,"allocated":696320,"data_stored":326763,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 19:11:25 2024","interfaces":[{"interface":"back","average":{"1min":0.315,"5min":0.38700000000000001,"15min":0.40000000000000002},"min":{"1min":0.223,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":0.46800000000000003,"5min":0.61499999999999999,"15min":0.61499999999999999},"last":0.36599999999999999},{"interface":"front","average":{"1min":0.33000000000000002,"5min":0.39700000000000002,"15min":0.41199999999999998},"min":{"1min":0.24399999999999999,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":0.42299999999999999,"5min":0.63600000000000001,"15min":0.63600000000000001},"last":0.44400000000000001}]},{"osd":1,"last update":"Tue Sep 10 19:11:25 2024","interfaces":[{"interface":"back","average":{"1min":0.35499999999999998,"5min":0.42999999999999999,"15min":0.44700000000000001},"min":{"1min":0.253,"5min":0.253,"15min":0.253},"max":{"1min":0.505,"5min":0.67600000000000005,"15min":0.67600000000000005},"last":0.39200000000000002},{"interface":"front","average":{"1min":0.32900000000000001,"5min":0.38700000000000001,"15min":0.39400000000000002},"min":{"1min":0.26300000000000001,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.441,"5min":0.622,"15min":0.622},"last":0.40500000000000003}]},{"osd":2,"last update":"Tue Sep 10 19:11:25 2024","interfaces":[{"interface":"back","average":{"1min":0.38100000000000001,"5min":0.42499999999999999,"15min":0.434},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.51200000000000001,"5min":0.56599999999999995,"15min":0.56599999999999995},"last":0.435},{"interface":"front","average":{"1min":0.38600000000000001,"5min":0.40799999999999997,"15min":0.41399999999999998},"min":{"1min":0.25600000000000001,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":0.66800000000000004,"5min":0.66800000000000004,"15min":0.66800000000000004},"last":0.42599999999999999}]},{"osd":4,"last update":"Tue Sep 10 19:10:50 2024","interfaces":[{"interface":"back","average":{"1min":0.54300000000000004,"5min":0.54400000000000004,"15min":0.54400000000000004},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.74399999999999999,"5min":0.74399999999999999,"15min":0.74399999999999999},"last":0.504},{"interface":"front","average":{"1min":0.55400000000000005,"5min":0.54900000000000004,"15min":0.54800000000000004},"min":{"1min":0.433,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":0.81699999999999995,"5min":0.81699999999999995,"15min":0.81699999999999995},"last":0.57999999999999996}]},{"osd":5,"last update":"Tue Sep 10 19:11:07 2024","interfaces":[{"interface":"back","average":{"1min":0.58799999999999997,"5min":0.56999999999999995,"15min":0.56699999999999995},"min":{"1min":0.441,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.72599999999999998,"5min":0.80400000000000005,"15min":0.80400000000000005},"last":0.52800000000000002},{"interface":"front","average":{"1min":0.56200000000000006,"5min":0.56100000000000005,"15min":0.56100000000000005},"min":{"1min":0.40799999999999997,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.89100000000000001,"5min":0.89100000000000001,"15min":0.89100000000000001},"last":0.621}]},{"osd":6,"last update":"Tue Sep 10 19:11:34 2024","interfaces":[{"interface":"back","average":{"1min":0.53400000000000003,"5min":0.56999999999999995,"15min":0.57599999999999996},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.67000000000000004,"5min":0.77000000000000002,"15min":0.77000000000000002},"last":0.55700000000000005},{"interface":"front","average":{"1min":0.55100000000000005,"5min":0.55500000000000005,"15min":0.55600000000000005},"min":{"1min":0.41399999999999998,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.71799999999999997,"5min":0.90800000000000003,"15min":0.90800000000000003},"last":0.54100000000000004}]},{"osd":7,"last update":"Tue Sep 10 19:11:00 2024","interfaces":[{"interface":"back","average":{"1min":0.57999999999999996,"5min":0.57999999999999996,"15min":0.57999999999999996},"min":{"1min":0.45400000000000001,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":0.75600000000000001,"5min":0.75600000000000001,"15min":0.75600000000000001},"last":0.57299999999999995},{"interface":"front","average":{"1min":0.58399999999999996,"5min":0.58399999999999996,"15min":0.58399999999999996},"min":{"1min":0.443,"5min":0.443,"15min":0.443},"max":{"1min":0.749,"5min":0.749,"15min":0.749},"last":0.60099999999999998}]}]},{"osd":2,"up_from":18,"seq":77309411376,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298228,"kb_used_data":640,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93446924,"statfs":{"total":95995035648,"available":95689650176,"internally_reserved":0,"allocated":655360,"data_stored":325179,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 19:11:08 2024","interfaces":[{"interface":"back","average":{"1min":0.33800000000000002,"5min":0.34899999999999998,"15min":0.34300000000000003},"min":{"1min":0.20100000000000001,"5min":0.20100000000000001,"15min":0.20100000000000001},"max":{"1min":0.45100000000000001,"5min":0.52300000000000002,"15min":0.52300000000000002},"last":0.34899999999999998},{"interface":"front","average":{"1min":0.33200000000000002,"5min":0.374,"15min":0.36799999999999999},"min":{"1min":0.23100000000000001,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":0.44700000000000001,"5min":0.64200000000000002,"15min":0.64200000000000002},"last":0.40999999999999998}]},{"osd":1,"last update":"Tue Sep 10 19:11:08 2024","interfaces":[{"interface":"back","average":{"1min":0.38200000000000001,"5min":0.38900000000000001,"15min":0.379},"min":{"1min":0.24099999999999999,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.63,"5min":0.69799999999999995,"15min":0.69799999999999995},"last":0.375},{"interface":"front","average":{"1min":0.35799999999999998,"5min":0.39700000000000002,"15min":0.39100000000000001},"min":{"1min":0.28499999999999998,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.48299999999999998,"5min":0.63400000000000001,"15min":0.63400000000000001},"last":0.33300000000000002}]},{"osd":3,"last update":"Tue Sep 10 19:11:30 2024","interfaces":[{"interface":"back","average":{"1min":0.35699999999999998,"5min":0.435,"15min":0.44600000000000001},"min":{"1min":0.23699999999999999,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":0.49099999999999999,"5min":0.63900000000000001,"15min":0.63900000000000001},"last":0.39600000000000002},{"interface":"front","average":{"1min":0.36199999999999999,"5min":0.432,"15min":0.442},"min":{"1min":0.27100000000000002,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.438,"5min":0.65800000000000003,"15min":0.65800000000000003},"last":0.38900000000000001}]},{"osd":4,"last update":"Tue Sep 10 19:10:50 2024","interfaces":[{"interface":"back","average":{"1min":0.59099999999999997,"5min":0.58299999999999996,"15min":0.58099999999999996},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.81999999999999995,"5min":0.81999999999999995,"15min":0.81999999999999995},"last":0.59899999999999998},{"interface":"front","average":{"1min":0.55600000000000005,"5min":0.56799999999999995,"15min":0.56999999999999995},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.749,"5min":0.80600000000000005,"15min":0.80600000000000005},"last":0.48599999999999999}]},{"osd":5,"last update":"Tue Sep 10 19:11:08 2024","interfaces":[{"interface":"back","average":{"1min":0.54800000000000004,"5min":0.52900000000000003,"15min":0.52600000000000002},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.73599999999999999,"5min":0.73899999999999999,"15min":0.73899999999999999},"last":0.51000000000000001},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.57899999999999996,"15min":0.57899999999999996},"min":{"1min":0.443,"5min":0.443,"15min":0.443},"max":{"1min":0.70699999999999996,"5min":0.82699999999999996,"15min":0.82699999999999996},"last":0.63700000000000001}]},{"osd":6,"last update":"Tue Sep 10 19:11:34 2024","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":0.55200000000000005,"15min":0.55600000000000005},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":0.67300000000000004,"5min":0.73199999999999998,"15min":0.73199999999999998},"last":0.65100000000000002},{"interface":"front","average":{"1min":0.54800000000000004,"5min":0.55000000000000004,"15min":0.55000000000000004},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.72299999999999998,"5min":0.748,"15min":0.748},"last":0.58899999999999997}]},{"osd":7,"last update":"Tue Sep 10 19:11:00 2024","interfaces":[{"interface":"back","average":{"1min":0.60599999999999998,"5min":0.60599999999999998,"15min":0.60599999999999998},"min":{"1min":0.44700000000000001,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":0.79700000000000004,"5min":0.79700000000000004,"15min":0.79700000000000004},"last":0.621},{"interface":"front","average":{"1min":0.54900000000000004,"5min":0.54900000000000004,"15min":0.54900000000000004},"min":{"1min":0.40300000000000002,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":0.72599999999999998,"5min":0.72599999999999998,"15min":0.72599999999999998},"last":0.51900000000000002}]}]},{"osd":1,"up_from":13,"seq":55834574901,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298276,"kb_used_data":688,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93446876,"statfs":{"total":95995035648,"available":95689601024,"internally_reserved":0,"allocated":704512,"data_stored":326337,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 19:11:42 2024","interfaces":[{"interface":"back","average":{"1min":0.38500000000000001,"5min":0.378,"15min":0.33800000000000002},"min":{"1min":0.21099999999999999,"5min":0.188,"15min":0.188},"max":{"1min":1.0629999999999999,"5min":2.7730000000000001,"15min":2.7730000000000001},"last":0.35399999999999998},{"interface":"front","average":{"1min":0.36699999999999999,"5min":0.38,"15min":0.34699999999999998},"min":{"1min":0.20699999999999999,"5min":0.20599999999999999,"15min":0.20599999999999999},"max":{"1min":0.53700000000000003,"5min":2.8380000000000001,"15min":2.8380000000000001},"last":0.33000000000000002}]},{"osd":2,"last update":"Tue Sep 10 19:11:01 2024","interfaces":[{"interface":"back","average":{"1min":0.42399999999999999,"5min":0.38500000000000001,"15min":0.373},"min":{"1min":0.251,"5min":0.251,"15min":0.251},"max":{"1min":1.1779999999999999,"5min":1.1779999999999999,"15min":1.1779999999999999},"last":0.30099999999999999},{"interface":"front","average":{"1min":0.42899999999999999,"5min":0.38,"15min":0.36799999999999999},"min":{"1min":0.254,"5min":0.254,"15min":0.254},"max":{"1min":1.103,"5min":1.103,"15min":1.103},"last":0.38500000000000001}]},{"osd":3,"last update":"Tue Sep 10 19:11:29 2024","interfaces":[{"interface":"back","average":{"1min":0.44700000000000001,"5min":0.45600000000000002,"15min":0.44900000000000001},"min":{"1min":0.27500000000000002,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":1.093,"5min":2.0670000000000002,"15min":2.0670000000000002},"last":0.40000000000000002},{"interface":"front","average":{"1min":0.40100000000000002,"5min":0.42299999999999999,"15min":0.42099999999999999},"min":{"1min":0.30299999999999999,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":0.54600000000000004,"5min":1.956,"15min":1.956},"last":0.374}]},{"osd":4,"last update":"Tue Sep 10 19:10:49 2024","interfaces":[{"interface":"back","average":{"1min":0.61299999999999999,"5min":0.52200000000000002,"15min":0.50700000000000001},"min":{"1min":0.28499999999999998,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":2.665,"5min":2.665,"15min":2.665},"last":0.46400000000000002},{"interface":"front","average":{"1min":0.65200000000000002,"5min":0.56200000000000006,"15min":0.54700000000000004},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":2.7330000000000001,"5min":2.7330000000000001,"15min":2.7330000000000001},"last":0.53800000000000003}]},{"osd":5,"last update":"Tue Sep 10 19:11:11 2024","interfaces":[{"interface":"back","average":{"1min":0.58299999999999996,"5min":0.53800000000000003,"15min":0.53000000000000003},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":1.851,"5min":1.851,"15min":1.851},"last":0.59199999999999997},{"interface":"front","average":{"1min":0.63500000000000001,"5min":0.55100000000000005,"15min":0.53700000000000003},"min":{"1min":0.31900000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":2.706,"5min":2.706,"15min":2.706},"last":0.52300000000000002}]},{"osd":6,"last update":"Tue Sep 10 19:11:35 2024","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.60199999999999998,"15min":0.61299999999999999},"min":{"1min":0.41599999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.68500000000000005,"5min":1.9950000000000001,"15min":1.9950000000000001},"last":0.436},{"interface":"front","average":{"1min":0.54200000000000004,"5min":0.58599999999999997,"15min":0.59399999999999997},"min":{"1min":0.39300000000000002,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.90600000000000003,"5min":1.9399999999999999,"15min":1.9399999999999999},"last":0.60199999999999998}]},{"osd":7,"last update":"Tue Sep 10 19:10:55 2024","interfaces":[{"interface":"back","average":{"1min":0.66600000000000004,"5min":0.66600000000000004,"15min":0.66600000000000004},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":3.4620000000000002,"5min":3.4620000000000002,"15min":3.4620000000000002},"last":0.49199999999999999},{"interface":"front","average":{"1min":0.621,"5min":0.621,"15min":0.621},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":1.514,"5min":1.514,"15min":1.514},"last":0.56200000000000006}]}]},{"osd":0,"up_from":8,"seq":34359738426,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298824,"kb_used_data":1236,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93446328,"statfs":{"total":95995035648,"available":95689039872,"internally_reserved":0,"allocated":1265664,"data_stored":916264,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Sep 10 19:11:42 2024","interfaces":[{"interface":"back","average":{"1min":0.318,"5min":0.34799999999999998,"15min":0.33100000000000002},"min":{"1min":0.23999999999999999,"5min":0.221,"15min":0.221},"max":{"1min":0.40799999999999997,"5min":0.63300000000000001,"15min":0.63300000000000001},"last":0.32400000000000001},{"interface":"front","average":{"1min":0.33600000000000002,"5min":0.35199999999999998,"15min":0.33500000000000002},"min":{"1min":0.23799999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":0.46899999999999997,"5min":0.61899999999999999,"15min":0.61899999999999999},"last":0.23799999999999999}]},{"osd":2,"last update":"Tue Sep 10 19:11:08 2024","interfaces":[{"interface":"back","average":{"1min":0.375,"5min":0.38700000000000001,"15min":0.38300000000000001},"min":{"1min":0.27600000000000002,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":0.50700000000000001,"5min":0.57399999999999995,"15min":0.57399999999999995},"last":0.26900000000000002},{"interface":"front","average":{"1min":0.36499999999999999,"5min":0.375,"15min":0.373},"min":{"1min":0.22900000000000001,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":0.53000000000000003,"5min":0.54100000000000004,"15min":0.54100000000000004},"last":0.28599999999999998}]},{"osd":3,"last update":"Tue Sep 10 19:11:30 2024","interfaces":[{"interface":"back","average":{"1min":0.35999999999999999,"5min":0.41499999999999998,"15min":0.42099999999999999},"min":{"1min":0.29499999999999998,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.45500000000000002,"5min":0.65000000000000002,"15min":0.65000000000000002},"last":0.313},{"interface":"front","average":{"1min":0.34499999999999997,"5min":0.39900000000000002,"15min":0.40500000000000003},"min":{"1min":0.27800000000000002,"5min":0.246,"15min":0.246},"max":{"1min":0.45900000000000002,"5min":0.58799999999999997,"15min":0.58799999999999997},"last":0.251}]},{"osd":4,"last update":"Tue Sep 10 19:10:46 2024","interfaces":[{"interface":"back","average":{"1min":0.54700000000000004,"5min":0.56200000000000006,"15min":0.56399999999999995},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.76000000000000001,"5min":0.76000000000000001,"15min":0.76000000000000001},"last":0.34300000000000003},{"interface":"front","average":{"1min":0.56999999999999995,"5min":0.56299999999999994,"15min":0.56200000000000006},"min":{"1min":0.40200000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.73699999999999999,"5min":0.77000000000000002,"15min":0.77000000000000002},"last":0.40300000000000002}]},{"osd":5,"last update":"Tue Sep 10 19:11:08 2024","interfaces":[{"interface":"back","average":{"1min":0.58399999999999996,"5min":0.58399999999999996,"15min":0.58399999999999996},"min":{"1min":0.44800000000000001,"5min":0.378,"15min":0.378},"max":{"1min":0.82599999999999996,"5min":0.82599999999999996,"15min":0.82599999999999996},"last":0.378},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.57399999999999995,"15min":0.57499999999999996},"min":{"1min":0.42699999999999999,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.77900000000000003,"5min":0.82899999999999996,"15min":0.82899999999999996},"last":0.41799999999999998}]},{"osd":6,"last update":"Tue Sep 10 19:11:33 2024","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":0.55800000000000005,"15min":0.56399999999999995},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.78900000000000003,"5min":0.78900000000000003,"15min":0.78900000000000003},"last":0.434},{"interface":"front","average":{"1min":0.54800000000000004,"5min":0.57899999999999996,"15min":0.58399999999999996},"min":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":0.70499999999999996,"5min":0.76100000000000001,"15min":0.76100000000000001},"last":0.36099999999999999}]},{"osd":7,"last update":"Tue Sep 10 19:10:58 2024","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.57799999999999996,"15min":0.57799999999999996},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":0.76900000000000002,"5min":0.76900000000000002,"15min":0.76900000000000002},"last":0.42699999999999999},{"interface":"front","average":{"1min":0.56200000000000006,"5min":0.56200000000000006,"15min":0.56200000000000006},"min":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.72599999999999998,"5min":0.72599999999999998,"15min":0.72599999999999998},"last":0.36899999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":966,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1236,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1236,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":362,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":2338,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":798,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T19:11:49.570 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph pg dump --format=json 2024-09-10T19:11:49.858 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:11:49 smithi183 ceph-mon[29736]: from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:11:49.858 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:11:49 smithi183 ceph-mon[29736]: 2.18 scrub starts 2024-09-10T19:11:49.858 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:11:49 smithi183 ceph-mon[29736]: 2.18 scrub ok 2024-09-10T19:11:49.858 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:11:49 smithi183 ceph-mon[41562]: from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:11:49.859 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:11:49 smithi183 ceph-mon[41562]: 2.18 scrub starts 2024-09-10T19:11:49.859 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:11:49 smithi183 ceph-mon[41562]: 2.18 scrub ok 2024-09-10T19:11:50.140 INFO:journalctl@ceph.alertmanager.a.smithi183.stdout:Sep 10 19:11:49 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-alertmanager-a[76837]: ts=2024-09-10T19:11:49.856Z caller=main.go:583 level=info msg="Received SIGTERM, exiting gracefully..." 2024-09-10T19:11:50.141 INFO:journalctl@ceph.alertmanager.a.smithi183.stdout:Sep 10 19:11:49 smithi183 podman[80223]: 2024-09-10 19:11:49.87745422 +0000 UTC m=+0.134637341 container died 17002da0f9cee8dc2b9a40d67c12f814adebfa79bafcb6177550784f70b8327a (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-09-10T19:11:50.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:11:49 smithi191 ceph-mon[32884]: from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:11:50.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:11:49 smithi191 ceph-mon[32884]: 2.18 scrub starts 2024-09-10T19:11:50.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:11:49 smithi191 ceph-mon[32884]: 2.18 scrub ok 2024-09-10T19:11:50.431 INFO:journalctl@ceph.alertmanager.a.smithi183.stdout:Sep 10 19:11:50 smithi183 podman[80223]: 2024-09-10 19:11:50.32162798 +0000 UTC m=+0.578811137 container cleanup 17002da0f9cee8dc2b9a40d67c12f814adebfa79bafcb6177550784f70b8327a (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-09-10T19:11:50.431 INFO:journalctl@ceph.alertmanager.a.smithi183.stdout:Sep 10 19:11:50 smithi183 bash[80223]: ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-alertmanager-a 2024-09-10T19:11:50.792 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:11:50.915 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:11:50 smithi183 ceph-mon[41562]: 4.7 deep-scrub starts 2024-09-10T19:11:50.915 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:11:50 smithi183 ceph-mon[41562]: 4.7 deep-scrub ok 2024-09-10T19:11:50.916 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:11:50 smithi183 ceph-mon[41562]: pgmap v45: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 27 B/s, 1 objects/s recovering 2024-09-10T19:11:50.916 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:11:50 smithi183 ceph-mon[41562]: 5.15 scrub starts 2024-09-10T19:11:50.916 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:11:50 smithi183 ceph-mon[41562]: 5.15 scrub ok 2024-09-10T19:11:50.916 INFO:journalctl@ceph.alertmanager.a.smithi183.stdout:Sep 10 19:11:50 smithi183 podman[80256]: 2024-09-10 19:11:50.64818755 +0000 UTC m=+0.767939036 container remove 17002da0f9cee8dc2b9a40d67c12f814adebfa79bafcb6177550784f70b8327a (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-09-10T19:11:50.916 INFO:journalctl@ceph.alertmanager.a.smithi183.stdout:Sep 10 19:11:50 smithi183 podman[80256]: 2024-09-10 19:11:50.748735827 +0000 UTC m=+0.868487312 volume remove 8bb6558239c6b9737e87d03c846d245069ff64df96ebd0418243f36fd67d4fee 2024-09-10T19:11:50.916 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:11:50 smithi183 ceph-mon[29736]: 4.7 deep-scrub starts 2024-09-10T19:11:50.916 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:11:50 smithi183 ceph-mon[29736]: 4.7 deep-scrub ok 2024-09-10T19:11:50.916 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:11:50 smithi183 ceph-mon[29736]: pgmap v45: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 27 B/s, 1 objects/s recovering 2024-09-10T19:11:50.917 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:11:50 smithi183 ceph-mon[29736]: 5.15 scrub starts 2024-09-10T19:11:50.917 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:11:50 smithi183 ceph-mon[29736]: 5.15 scrub ok 2024-09-10T19:11:51.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:11:50 smithi191 ceph-mon[32884]: 4.7 deep-scrub starts 2024-09-10T19:11:51.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:11:50 smithi191 ceph-mon[32884]: 4.7 deep-scrub ok 2024-09-10T19:11:51.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:11:50 smithi191 ceph-mon[32884]: pgmap v45: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 27 B/s, 1 objects/s recovering 2024-09-10T19:11:51.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:11:50 smithi191 ceph-mon[32884]: 5.15 scrub starts 2024-09-10T19:11:51.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:11:50 smithi191 ceph-mon[32884]: 5.15 scrub ok 2024-09-10T19:11:51.423 INFO:journalctl@ceph.alertmanager.a.smithi183.stdout:Sep 10 19:11:51 smithi183 podman[80314]: 2024-09-10 19:11:51.270364976 +0000 UTC m=+0.489427807 container died ebd2a33d7bca959e0bdc6dec819e48a06bf1aaca46507179848e6f593f615e22 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588, name=tender_williamson, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, RELEASE=wip-guits-quincy-2024-09-09-1810-quincy-5be5739, GIT_CLEAN=True, CEPH_POINT_RELEASE=) 2024-09-10T19:11:51.757 INFO:journalctl@ceph.alertmanager.a.smithi183.stdout:Sep 10 19:11:51 smithi183 systemd[1]: ceph-320f6df8-6fa7-11ef-bcea-c7b262605968@alertmanager.a.service: Deactivated successfully. 2024-09-10T19:11:51.757 INFO:journalctl@ceph.alertmanager.a.smithi183.stdout:Sep 10 19:11:51 smithi183 systemd[1]: Stopped Ceph alertmanager.a for 320f6df8-6fa7-11ef-bcea-c7b262605968. 2024-09-10T19:11:51.757 INFO:journalctl@ceph.alertmanager.a.smithi183.stdout:Sep 10 19:11:51 smithi183 systemd[1]: Starting Ceph alertmanager.a for 320f6df8-6fa7-11ef-bcea-c7b262605968... 2024-09-10T19:11:52.048 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:11:51 smithi183 ceph-mon[29736]: 5.f scrub starts 2024-09-10T19:11:52.048 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:11:51 smithi183 ceph-mon[29736]: 5.f scrub ok 2024-09-10T19:11:52.048 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:11:51 smithi183 ceph-mon[41562]: 5.f scrub starts 2024-09-10T19:11:52.048 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:11:51 smithi183 ceph-mon[41562]: 5.f scrub ok 2024-09-10T19:11:52.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:11:51 smithi191 ceph-mon[32884]: 5.f scrub starts 2024-09-10T19:11:52.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:11:51 smithi191 ceph-mon[32884]: 5.f scrub ok 2024-09-10T19:11:52.420 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-09-10T19:11:52.422 INFO:teuthology.orchestra.run.smithi183.stderr:dumped all 2024-09-10T19:11:52.681 INFO:journalctl@ceph.alertmanager.a.smithi183.stdout:Sep 10 19:11:52 smithi183 podman[80503]: 2024-09-10 19:11:52.294678772 +0000 UTC m=+0.016526985 image pull c8568f914cd25b2062c44e9f79f9c18da6e3b85fe0c47a12a2191c61426c2b19 quay.io/prometheus/alertmanager:v0.25.0 2024-09-10T19:11:52.681 INFO:journalctl@ceph.alertmanager.a.smithi183.stdout:Sep 10 19:11:52 smithi183 podman[80503]: 2024-09-10 19:11:52.420872947 +0000 UTC m=+0.142721160 volume create d4fb3fd11a5f4d7a217c91b4fe698c6bca5f6057348618c30cb85b7187ae5960 2024-09-10T19:11:52.681 INFO:journalctl@ceph.alertmanager.a.smithi183.stdout:Sep 10 19:11:52 smithi183 podman[80503]: 2024-09-10 19:11:52.53812455 +0000 UTC m=+0.259972796 container create 1693fe6a126991cd9e9bae41ec0e86c2aed52a7bf9b9695f6083f915cbdc7440 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-09-10T19:11:53.044 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:11:52 smithi183 ceph-mon[29736]: 6.1 scrub starts 2024-09-10T19:11:53.044 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:11:52 smithi183 ceph-mon[29736]: 6.1 scrub ok 2024-09-10T19:11:53.044 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:11:52 smithi183 ceph-mon[29736]: pgmap v46: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 27 B/s, 1 objects/s recovering 2024-09-10T19:11:53.045 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:11:52 smithi183 ceph-mon[29736]: 6.10 scrub starts 2024-09-10T19:11:53.045 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:11:52 smithi183 ceph-mon[29736]: 6.10 scrub ok 2024-09-10T19:11:53.045 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:11:52 smithi183 ceph-mon[41562]: 6.1 scrub starts 2024-09-10T19:11:53.045 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:11:52 smithi183 ceph-mon[41562]: 6.1 scrub ok 2024-09-10T19:11:53.045 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:11:52 smithi183 ceph-mon[41562]: pgmap v46: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 27 B/s, 1 objects/s recovering 2024-09-10T19:11:53.045 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:11:52 smithi183 ceph-mon[41562]: 6.10 scrub starts 2024-09-10T19:11:53.045 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:11:52 smithi183 ceph-mon[41562]: 6.10 scrub ok 2024-09-10T19:11:53.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:11:52 smithi191 ceph-mon[32884]: 6.1 scrub starts 2024-09-10T19:11:53.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:11:52 smithi191 ceph-mon[32884]: 6.1 scrub ok 2024-09-10T19:11:53.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:11:52 smithi191 ceph-mon[32884]: pgmap v46: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 27 B/s, 1 objects/s recovering 2024-09-10T19:11:53.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:11:52 smithi191 ceph-mon[32884]: 6.10 scrub starts 2024-09-10T19:11:53.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:11:52 smithi191 ceph-mon[32884]: 6.10 scrub ok 2024-09-10T19:11:53.486 INFO:teuthology.orchestra.run.smithi183.stdout:{"pg_ready":true,"pg_map":{"version":46,"stamp":"2024-09-10T19:11:51.606076+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596188,"num_objects":196,"num_object_clones":0,"num_object_copies":588,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":196,"num_whiteouts":0,"num_read":947,"num_read_kb":804,"num_write":543,"num_write_kb":1671,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":590588,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":589,"ondisk_log_size":589,"up":396,"acting":396,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":396,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":2386400,"kb_used_data":7096,"kb_used_omap":13,"kb_used_meta":2378738,"kb_avail":747574816,"statfs":{"total":767960285184,"available":765516611584,"internally_reserved":0,"allocated":7266304,"data_stored":4438576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13632,"internal_metadata":2435828416},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001530"},"pg_stats":[{"pgid":"6.1b","version":"0'0","reported_seq":26,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-10T19:11:43.737383+0000","last_change":"2024-09-10T19:10:14.056737+0000","last_active":"2024-09-10T19:11:43.737383+0000","last_peered":"2024-09-10T19:11:43.737383+0000","last_clean":"2024-09-10T19:11:43.737383+0000","last_became_active":"2024-09-10T19:10:14.056187+0000","last_became_peered":"2024-09-10T19:10:14.056187+0000","last_unstale":"2024-09-10T19:11:43.737383+0000","last_undegraded":"2024-09-10T19:11:43.737383+0000","last_fullsized":"2024-09-10T19:11:43.737383+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:10:12.693598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":52,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-09-10T19:11:46.882547+0000","last_change":"2024-09-10T19:11:46.882500+0000","last_active":"2024-09-10T19:11:46.882547+0000","last_peered":"2024-09-10T19:11:46.882547+0000","last_clean":"2024-09-10T19:11:46.882547+0000","last_became_active":"2024-09-10T19:11:44.748696+0000","last_became_peered":"2024-09-10T19:11:44.748696+0000","last_unstale":"2024-09-10T19:11:46.882547+0000","last_undegraded":"2024-09-10T19:11:46.882547+0000","last_fullsized":"2024-09-10T19:11:46.882547+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:11:46.882412+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:11:46.882412+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:13:12.297001+0000","scrub_duration":0.0015255100000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.19","version":"66'15","reported_seq":55,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-10T19:11:43.735743+0000","last_change":"2024-09-10T19:10:09.688886+0000","last_active":"2024-09-10T19:11:43.735743+0000","last_peered":"2024-09-10T19:11:43.735743+0000","last_clean":"2024-09-10T19:11:43.735743+0000","last_became_active":"2024-09-10T19:10:09.688432+0000","last_became_peered":"2024-09-10T19:10:09.688432+0000","last_unstale":"2024-09-10T19:11:43.735743+0000","last_undegraded":"2024-09-10T19:11:43.735743+0000","last_fullsized":"2024-09-10T19:11:43.735743+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:10:08.671214+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":38,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-10T19:11:43.728937+0000","last_change":"2024-09-10T19:11:23.086288+0000","last_active":"2024-09-10T19:11:43.728937+0000","last_peered":"2024-09-10T19:11:43.728937+0000","last_clean":"2024-09-10T19:11:43.728937+0000","last_became_active":"2024-09-10T19:10:11.697523+0000","last_became_peered":"2024-09-10T19:10:11.697523+0000","last_unstale":"2024-09-10T19:11:43.728937+0000","last_undegraded":"2024-09-10T19:11:43.728937+0000","last_fullsized":"2024-09-10T19:11:43.728937+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:11:23.086191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:11:23.086191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:12:31.131189+0000","scrub_duration":0.0016087969999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"59'1","reported_seq":50,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-10T19:11:43.738716+0000","last_change":"2024-09-10T19:11:15.782493+0000","last_active":"2024-09-10T19:11:43.738716+0000","last_peered":"2024-09-10T19:11:43.738716+0000","last_clean":"2024-09-10T19:11:43.738716+0000","last_became_active":"2024-09-10T19:10:07.686143+0000","last_became_peered":"2024-09-10T19:10:07.686143+0000","last_unstale":"2024-09-10T19:11:43.738716+0000","last_undegraded":"2024-09-10T19:11:43.738716+0000","last_fullsized":"2024-09-10T19:11:43.738716+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'1","last_scrub_stamp":"2024-09-10T19:11:15.782393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:11:15.782393+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:12:21.959971+0000","scrub_duration":0.00183947,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":874,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"66'9","reported_seq":46,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-10T19:11:43.729014+0000","last_change":"2024-09-10T19:10:09.692950+0000","last_active":"2024-09-10T19:11:43.729014+0000","last_peered":"2024-09-10T19:11:43.729014+0000","last_clean":"2024-09-10T19:11:43.729014+0000","last_became_active":"2024-09-10T19:10:09.692681+0000","last_became_peered":"2024-09-10T19:10:09.692681+0000","last_unstale":"2024-09-10T19:11:43.729014+0000","last_undegraded":"2024-09-10T19:11:43.729014+0000","last_fullsized":"2024-09-10T19:11:43.729014+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:10:08.671214+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":28,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-10T19:11:43.738865+0000","last_change":"2024-09-10T19:10:11.702830+0000","last_active":"2024-09-10T19:11:43.738865+0000","last_peered":"2024-09-10T19:11:43.738865+0000","last_clean":"2024-09-10T19:11:43.738865+0000","last_became_active":"2024-09-10T19:10:11.702546+0000","last_became_peered":"2024-09-10T19:10:11.702546+0000","last_unstale":"2024-09-10T19:11:43.738865+0000","last_undegraded":"2024-09-10T19:11:43.738865+0000","last_fullsized":"2024-09-10T19:11:43.738865+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:10:10.676876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":36,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-10T19:11:43.729161+0000","last_change":"2024-09-10T19:11:32.056320+0000","last_active":"2024-09-10T19:11:43.729161+0000","last_peered":"2024-09-10T19:11:43.729161+0000","last_clean":"2024-09-10T19:11:43.729161+0000","last_became_active":"2024-09-10T19:10:13.716022+0000","last_became_peered":"2024-09-10T19:10:13.716022+0000","last_unstale":"2024-09-10T19:11:43.729161+0000","last_undegraded":"2024-09-10T19:11:43.729161+0000","last_fullsized":"2024-09-10T19:11:43.729161+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:11:32.056254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:11:32.056254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:12:39.600944+0000","scrub_duration":0.0013010680000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":42,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-10T19:11:43.739287+0000","last_change":"2024-09-10T19:11:19.275407+0000","last_active":"2024-09-10T19:11:43.739287+0000","last_peered":"2024-09-10T19:11:43.739287+0000","last_clean":"2024-09-10T19:11:43.739287+0000","last_became_active":"2024-09-10T19:10:07.690728+0000","last_became_peered":"2024-09-10T19:10:07.690728+0000","last_unstale":"2024-09-10T19:11:43.739287+0000","last_undegraded":"2024-09-10T19:11:43.739287+0000","last_fullsized":"2024-09-10T19:11:43.739287+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:11:19.275331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:11:19.275331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:12:30.648634+0000","scrub_duration":0.001355258,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1b","version":"66'5","reported_seq":66,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-09-10T19:11:44.752160+0000","last_change":"2024-09-10T19:11:44.752160+0000","last_active":"2024-09-10T19:11:44.752160+0000","last_peered":"2024-09-10T19:11:44.752160+0000","last_clean":"2024-09-10T19:11:44.752160+0000","last_became_active":"2024-09-10T19:11:44.748296+0000","last_became_peered":"2024-09-10T19:11:44.748296+0000","last_unstale":"2024-09-10T19:11:44.752160+0000","last_undegraded":"2024-09-10T19:11:44.752160+0000","last_fullsized":"2024-09-10T19:11:44.752160+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'5","last_scrub_stamp":"2024-09-10T19:11:24.101551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:11:24.101551+0000","objects_scrubbed":2,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:12:33.998803+0000","scrub_duration":0.0019539829999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":38,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-10T19:11:43.738152+0000","last_change":"2024-09-10T19:11:43.142813+0000","last_active":"2024-09-10T19:11:43.738152+0000","last_peered":"2024-09-10T19:11:43.738152+0000","last_clean":"2024-09-10T19:11:43.738152+0000","last_became_active":"2024-09-10T19:10:11.697088+0000","last_became_peered":"2024-09-10T19:10:11.697088+0000","last_unstale":"2024-09-10T19:11:43.738152+0000","last_undegraded":"2024-09-10T19:11:43.738152+0000","last_fullsized":"2024-09-10T19:11:43.738152+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:11:43.142734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:11:43.142734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:12:50.945873+0000","scrub_duration":0.0013612330000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":36,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-09-10T19:11:45.411428+0000","last_change":"2024-09-10T19:11:45.411391+0000","last_active":"2024-09-10T19:11:45.411428+0000","last_peered":"2024-09-10T19:11:45.411428+0000","last_clean":"2024-09-10T19:11:45.411428+0000","last_became_active":"2024-09-10T19:10:13.724255+0000","last_became_peered":"2024-09-10T19:10:13.724255+0000","last_unstale":"2024-09-10T19:11:45.411428+0000","last_undegraded":"2024-09-10T19:11:45.411428+0000","last_fullsized":"2024-09-10T19:11:45.411428+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:11:45.411321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:11:45.411321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:12:49.122352+0000","scrub_duration":0.001536576,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":42,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-10T19:11:43.739238+0000","last_change":"2024-09-10T19:11:33.268356+0000","last_active":"2024-09-10T19:11:43.739238+0000","last_peered":"2024-09-10T19:11:43.739238+0000","last_clean":"2024-09-10T19:11:43.739238+0000","last_became_active":"2024-09-10T19:10:07.690174+0000","last_became_peered":"2024-09-10T19:10:07.690174+0000","last_unstale":"2024-09-10T19:11:43.739238+0000","last_undegraded":"2024-09-10T19:11:43.739238+0000","last_fullsized":"2024-09-10T19:11:43.739238+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:11:33.268282+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:11:33.268282+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:12:35.423155+0000","scrub_duration":0.0015798069999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1a","version":"66'9","reported_seq":56,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-10T19:11:43.729429+0000","last_change":"2024-09-10T19:11:31.035964+0000","last_active":"2024-09-10T19:11:43.729429+0000","last_peered":"2024-09-10T19:11:43.729429+0000","last_clean":"2024-09-10T19:11:43.729429+0000","last_became_active":"2024-09-10T19:10:09.688987+0000","last_became_peered":"2024-09-10T19:10:09.688987+0000","last_unstale":"2024-09-10T19:11:43.729429+0000","last_undegraded":"2024-09-10T19:11:43.729429+0000","last_fullsized":"2024-09-10T19:11:43.729429+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'9","last_scrub_stamp":"2024-09-10T19:11:31.035912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:11:31.035912+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:12:57.913355+0000","scrub_duration":0.001674916,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":38,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-10T19:11:43.739446+0000","last_change":"2024-09-10T19:11:27.317868+0000","last_active":"2024-09-10T19:11:43.739446+0000","last_peered":"2024-09-10T19:11:43.739446+0000","last_clean":"2024-09-10T19:11:43.739446+0000","last_became_active":"2024-09-10T19:10:11.701799+0000","last_became_peered":"2024-09-10T19:10:11.701799+0000","last_unstale":"2024-09-10T19:11:43.739446+0000","last_undegraded":"2024-09-10T19:11:43.739446+0000","last_fullsized":"2024-09-10T19:11:43.739446+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:11:27.317779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:11:27.317779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:12:44.343114+0000","scrub_duration":0.001509584,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":36,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-10T19:11:43.736436+0000","last_change":"2024-09-10T19:11:12.970227+0000","last_active":"2024-09-10T19:11:43.736436+0000","last_peered":"2024-09-10T19:11:43.736436+0000","last_clean":"2024-09-10T19:11:43.736436+0000","last_became_active":"2024-09-10T19:10:13.717310+0000","last_became_peered":"2024-09-10T19:10:13.717310+0000","last_unstale":"2024-09-10T19:11:43.736436+0000","last_undegraded":"2024-09-10T19:11:43.736436+0000","last_fullsized":"2024-09-10T19:11:43.736436+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:11:12.970173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:11:12.970173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:12:27.060515+0000","scrub_duration":0.001527219,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.d","version":"66'17","reported_seq":68,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-10T19:11:43.729713+0000","last_change":"2024-09-10T19:11:14.007973+0000","last_active":"2024-09-10T19:11:43.729713+0000","last_peered":"2024-09-10T19:11:43.729713+0000","last_clean":"2024-09-10T19:11:43.729713+0000","last_became_active":"2024-09-10T19:10:09.694016+0000","last_became_peered":"2024-09-10T19:10:09.694016+0000","last_unstale":"2024-09-10T19:11:43.729713+0000","last_undegraded":"2024-09-10T19:11:43.729713+0000","last_fullsized":"2024-09-10T19:11:43.729713+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'17","last_scrub_stamp":"2024-09-10T19:11:14.007902+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:11:14.007902+0000","objects_scrubbed":8,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:12:21.164915+0000","scrub_duration":0.0024520879999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":42,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-10T19:11:43.738682+0000","last_change":"2024-09-10T19:11:22.280067+0000","last_active":"2024-09-10T19:11:43.738682+0000","last_peered":"2024-09-10T19:11:43.738682+0000","last_clean":"2024-09-10T19:11:43.738682+0000","last_became_active":"2024-09-10T19:10:07.691119+0000","last_became_peered":"2024-09-10T19:10:07.691119+0000","last_unstale":"2024-09-10T19:11:43.738682+0000","last_undegraded":"2024-09-10T19:11:43.738682+0000","last_fullsized":"2024-09-10T19:11:43.738682+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:11:22.280002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:11:22.280002+0000","last_clean_scrub_stamp":"2024-09-10T19:11:22.280002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:12:27.848752+0000","scrub_duration":0.0013828040000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":28,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-10T19:11:43.739551+0000","last_change":"2024-09-10T19:10:11.695393+0000","last_active":"2024-09-10T19:11:43.739551+0000","last_peered":"2024-09-10T19:11:43.739551+0000","last_clean":"2024-09-10T19:11:43.739551+0000","last_became_active":"2024-09-10T19:10:11.695141+0000","last_became_peered":"2024-09-10T19:10:11.695141+0000","last_unstale":"2024-09-10T19:11:43.739551+0000","last_undegraded":"2024-09-10T19:11:43.739551+0000","last_fullsized":"2024-09-10T19:11:43.739551+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:10:10.676876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":36,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-10T19:11:43.735069+0000","last_change":"2024-09-10T19:11:24.145056+0000","last_active":"2024-09-10T19:11:43.735069+0000","last_peered":"2024-09-10T19:11:43.735069+0000","last_clean":"2024-09-10T19:11:43.735069+0000","last_became_active":"2024-09-10T19:10:13.722894+0000","last_became_peered":"2024-09-10T19:10:13.722894+0000","last_unstale":"2024-09-10T19:11:43.735069+0000","last_undegraded":"2024-09-10T19:11:43.735069+0000","last_fullsized":"2024-09-10T19:11:43.735069+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:11:24.145003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:11:24.145003+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:12:26.405847+0000","scrub_duration":0.0013083789999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.c","version":"66'10","reported_seq":55,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-10T19:11:43.729601+0000","last_change":"2024-09-10T19:11:14.987470+0000","last_active":"2024-09-10T19:11:43.729601+0000","last_peered":"2024-09-10T19:11:43.729601+0000","last_clean":"2024-09-10T19:11:43.729601+0000","last_became_active":"2024-09-10T19:10:09.688321+0000","last_became_peered":"2024-09-10T19:10:09.688321+0000","last_unstale":"2024-09-10T19:11:43.729601+0000","last_undegraded":"2024-09-10T19:11:43.729601+0000","last_fullsized":"2024-09-10T19:11:43.729601+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'10","last_scrub_stamp":"2024-09-10T19:11:14.987403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:11:14.987403+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:12:21.734174+0000","scrub_duration":0.0018747600000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":42,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-10T19:11:43.739823+0000","last_change":"2024-09-10T19:11:26.110216+0000","last_active":"2024-09-10T19:11:43.739823+0000","last_peered":"2024-09-10T19:11:43.739823+0000","last_clean":"2024-09-10T19:11:43.739823+0000","last_became_active":"2024-09-10T19:10:07.680803+0000","last_became_peered":"2024-09-10T19:10:07.680803+0000","last_unstale":"2024-09-10T19:11:43.739823+0000","last_undegraded":"2024-09-10T19:11:43.739823+0000","last_fullsized":"2024-09-10T19:11:43.739823+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:11:26.110138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:11:26.110138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:12:35.580206+0000","scrub_duration":0.0011581549999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.d","version":"80'9","reported_seq":70,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-09-10T19:11:44.745373+0000","last_change":"2024-09-10T19:11:25.156512+0000","last_active":"2024-09-10T19:11:44.745373+0000","last_peered":"2024-09-10T19:11:44.745373+0000","last_clean":"2024-09-10T19:11:44.745373+0000","last_became_active":"2024-09-10T19:10:11.698547+0000","last_became_peered":"2024-09-10T19:10:11.698547+0000","last_unstale":"2024-09-10T19:11:44.745373+0000","last_undegraded":"2024-09-10T19:11:44.745373+0000","last_fullsized":"2024-09-10T19:11:44.745373+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"80'9","last_scrub_stamp":"2024-09-10T19:11:25.156465+0000","last_deep_scrub":"80'9","last_deep_scrub_stamp":"2024-09-10T19:11:25.156465+0000","last_clean_scrub_stamp":"2024-09-10T19:11:25.156465+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:12:26.762818+0000","scrub_duration":0.0022299569999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":36,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-10T19:11:43.729808+0000","last_change":"2024-09-10T19:11:26.096147+0000","last_active":"2024-09-10T19:11:43.729808+0000","last_peered":"2024-09-10T19:11:43.729808+0000","last_clean":"2024-09-10T19:11:43.729808+0000","last_became_active":"2024-09-10T19:10:13.718242+0000","last_became_peered":"2024-09-10T19:10:13.718242+0000","last_unstale":"2024-09-10T19:11:43.729808+0000","last_undegraded":"2024-09-10T19:11:43.729808+0000","last_fullsized":"2024-09-10T19:11:43.729808+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:11:26.096090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:11:26.096090+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:12:49.347064+0000","scrub_duration":0.0015599909999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.f","version":"66'15","reported_seq":55,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-10T19:11:43.739407+0000","last_change":"2024-09-10T19:10:09.691727+0000","last_active":"2024-09-10T19:11:43.739407+0000","last_peered":"2024-09-10T19:11:43.739407+0000","last_clean":"2024-09-10T19:11:43.739407+0000","last_became_active":"2024-09-10T19:10:09.691230+0000","last_became_peered":"2024-09-10T19:10:09.691230+0000","last_unstale":"2024-09-10T19:11:43.739407+0000","last_undegraded":"2024-09-10T19:11:43.739407+0000","last_fullsized":"2024-09-10T19:11:43.739407+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:10:08.671214+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":42,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-10T19:11:43.739102+0000","last_change":"2024-09-10T19:11:11.036017+0000","last_active":"2024-09-10T19:11:43.739102+0000","last_peered":"2024-09-10T19:11:43.739102+0000","last_clean":"2024-09-10T19:11:43.739102+0000","last_became_active":"2024-09-10T19:10:07.679646+0000","last_became_peered":"2024-09-10T19:10:07.679646+0000","last_unstale":"2024-09-10T19:11:43.739102+0000","last_undegraded":"2024-09-10T19:11:43.739102+0000","last_fullsized":"2024-09-10T19:11:43.739102+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:11:11.035954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:11:11.035954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:12:35.469086+0000","scrub_duration":0.001618024,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.e","version":"66'8","reported_seq":56,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-09-10T19:11:44.741458+0000","last_change":"2024-09-10T19:10:11.699487+0000","last_active":"2024-09-10T19:11:44.741458+0000","last_peered":"2024-09-10T19:11:44.741458+0000","last_clean":"2024-09-10T19:11:44.741458+0000","last_became_active":"2024-09-10T19:10:11.699232+0000","last_became_peered":"2024-09-10T19:10:11.699232+0000","last_unstale":"2024-09-10T19:11:44.741458+0000","last_undegraded":"2024-09-10T19:11:44.741458+0000","last_fullsized":"2024-09-10T19:11:44.741458+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:10:10.676876+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":36,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-10T19:11:43.739496+0000","last_change":"2024-09-10T19:11:18.208847+0000","last_active":"2024-09-10T19:11:43.739496+0000","last_peered":"2024-09-10T19:11:43.739496+0000","last_clean":"2024-09-10T19:11:43.739496+0000","last_became_active":"2024-09-10T19:10:13.715274+0000","last_became_peered":"2024-09-10T19:10:13.715274+0000","last_unstale":"2024-09-10T19:11:43.739496+0000","last_undegraded":"2024-09-10T19:11:43.739496+0000","last_fullsized":"2024-09-10T19:11:43.739496+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:11:18.208782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:11:18.208782+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:12:43.016878+0000","scrub_duration":0.001904481,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.e","version":"66'11","reported_seq":49,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-10T19:11:43.730037+0000","last_change":"2024-09-10T19:10:09.692995+0000","last_active":"2024-09-10T19:11:43.730037+0000","last_peered":"2024-09-10T19:11:43.730037+0000","last_clean":"2024-09-10T19:11:43.730037+0000","last_became_active":"2024-09-10T19:10:09.692675+0000","last_became_peered":"2024-09-10T19:10:09.692675+0000","last_unstale":"2024-09-10T19:11:43.730037+0000","last_undegraded":"2024-09-10T19:11:43.730037+0000","last_fullsized":"2024-09-10T19:11:43.730037+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:10:08.671214+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":32,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-10T19:11:43.738164+0000","last_change":"2024-09-10T19:10:07.690701+0000","last_active":"2024-09-10T19:11:43.738164+0000","last_peered":"2024-09-10T19:11:43.738164+0000","last_clean":"2024-09-10T19:11:43.738164+0000","last_became_active":"2024-09-10T19:10:07.690434+0000","last_became_peered":"2024-09-10T19:10:07.690434+0000","last_unstale":"2024-09-10T19:11:43.738164+0000","last_undegraded":"2024-09-10T19:11:43.738164+0000","last_fullsized":"2024-09-10T19:11:43.738164+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:10:06.664709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":28,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-10T19:11:43.739676+0000","last_change":"2024-09-10T19:10:11.692483+0000","last_active":"2024-09-10T19:11:43.739676+0000","last_peered":"2024-09-10T19:11:43.739676+0000","last_clean":"2024-09-10T19:11:43.739676+0000","last_became_active":"2024-09-10T19:10:11.692215+0000","last_became_peered":"2024-09-10T19:10:11.692215+0000","last_unstale":"2024-09-10T19:11:43.739676+0000","last_undegraded":"2024-09-10T19:11:43.739676+0000","last_fullsized":"2024-09-10T19:11:43.739676+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:10:10.676876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":26,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-10T19:11:43.737683+0000","last_change":"2024-09-10T19:10:14.054106+0000","last_active":"2024-09-10T19:11:43.737683+0000","last_peered":"2024-09-10T19:11:43.737683+0000","last_clean":"2024-09-10T19:11:43.737683+0000","last_became_active":"2024-09-10T19:10:14.053736+0000","last_became_peered":"2024-09-10T19:10:14.053736+0000","last_unstale":"2024-09-10T19:11:43.737683+0000","last_undegraded":"2024-09-10T19:11:43.737683+0000","last_fullsized":"2024-09-10T19:11:43.737683+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:10:12.693598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"66'14","reported_seq":61,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-10T19:11:43.729138+0000","last_change":"2024-09-10T19:11:16.976833+0000","last_active":"2024-09-10T19:11:43.729138+0000","last_peered":"2024-09-10T19:11:43.729138+0000","last_clean":"2024-09-10T19:11:43.729138+0000","last_became_active":"2024-09-10T19:10:09.686534+0000","last_became_peered":"2024-09-10T19:10:09.686534+0000","last_unstale":"2024-09-10T19:11:43.729138+0000","last_undegraded":"2024-09-10T19:11:43.729138+0000","last_fullsized":"2024-09-10T19:11:43.729138+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'14","last_scrub_stamp":"2024-09-10T19:11:16.976776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:11:16.976776+0000","objects_scrubbed":7,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:12:29.125897+0000","scrub_duration":0.001777144,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":42,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-10T19:11:43.742564+0000","last_change":"2024-09-10T19:11:22.801187+0000","last_active":"2024-09-10T19:11:43.742564+0000","last_peered":"2024-09-10T19:11:43.742564+0000","last_clean":"2024-09-10T19:11:43.742564+0000","last_became_active":"2024-09-10T19:10:07.684864+0000","last_became_peered":"2024-09-10T19:10:07.684864+0000","last_unstale":"2024-09-10T19:11:43.742564+0000","last_undegraded":"2024-09-10T19:11:43.742564+0000","last_fullsized":"2024-09-10T19:11:43.742564+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:11:22.801123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:11:22.801123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:12:47.860716+0000","scrub_duration":0.0015474410000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.0","version":"66'8","reported_seq":66,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-09-10T19:11:44.742953+0000","last_change":"2024-09-10T19:11:29.884792+0000","last_active":"2024-09-10T19:11:44.742953+0000","last_peered":"2024-09-10T19:11:44.742953+0000","last_clean":"2024-09-10T19:11:44.742953+0000","last_became_active":"2024-09-10T19:10:11.697106+0000","last_became_peered":"2024-09-10T19:10:11.697106+0000","last_unstale":"2024-09-10T19:11:44.742953+0000","last_undegraded":"2024-09-10T19:11:44.742953+0000","last_fullsized":"2024-09-10T19:11:44.742953+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'8","last_scrub_stamp":"2024-09-10T19:11:29.884739+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:11:29.884739+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:12:37.090103+0000","scrub_duration":0.0014372650000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":36,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-10T19:11:43.738407+0000","last_change":"2024-09-10T19:11:41.073902+0000","last_active":"2024-09-10T19:11:43.738407+0000","last_peered":"2024-09-10T19:11:43.738407+0000","last_clean":"2024-09-10T19:11:43.738407+0000","last_became_active":"2024-09-10T19:10:14.054456+0000","last_became_peered":"2024-09-10T19:10:14.054456+0000","last_unstale":"2024-09-10T19:11:43.738407+0000","last_undegraded":"2024-09-10T19:11:43.738407+0000","last_fullsized":"2024-09-10T19:11:43.738407+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:11:41.073833+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:11:41.073833+0000","last_clean_scrub_stamp":"2024-09-10T19:11:41.073833+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:13:00.467462+0000","scrub_duration":0.001716567,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.0","version":"66'18","reported_seq":72,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-10T19:11:43.735236+0000","last_change":"2024-09-10T19:11:19.848186+0000","last_active":"2024-09-10T19:11:43.735236+0000","last_peered":"2024-09-10T19:11:43.735236+0000","last_clean":"2024-09-10T19:11:43.735236+0000","last_became_active":"2024-09-10T19:10:09.821836+0000","last_became_peered":"2024-09-10T19:10:09.821836+0000","last_unstale":"2024-09-10T19:11:43.735236+0000","last_undegraded":"2024-09-10T19:11:43.735236+0000","last_fullsized":"2024-09-10T19:11:43.735236+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'18","last_scrub_stamp":"2024-09-10T19:11:19.848132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:11:19.848132+0000","objects_scrubbed":8,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:12:26.692213+0000","scrub_duration":0.0034247819999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":42,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-10T19:11:43.739400+0000","last_change":"2024-09-10T19:11:42.202747+0000","last_active":"2024-09-10T19:11:43.739400+0000","last_peered":"2024-09-10T19:11:43.739400+0000","last_clean":"2024-09-10T19:11:43.739400+0000","last_became_active":"2024-09-10T19:10:07.682081+0000","last_became_peered":"2024-09-10T19:10:07.682081+0000","last_unstale":"2024-09-10T19:11:43.739400+0000","last_undegraded":"2024-09-10T19:11:43.739400+0000","last_fullsized":"2024-09-10T19:11:43.739400+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:11:42.202689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:11:42.202689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:12:53.708170+0000","scrub_duration":0.00127655,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":38,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-10T19:11:43.729038+0000","last_change":"2024-09-10T19:11:29.069421+0000","last_active":"2024-09-10T19:11:43.729038+0000","last_peered":"2024-09-10T19:11:43.729038+0000","last_clean":"2024-09-10T19:11:43.729038+0000","last_became_active":"2024-09-10T19:10:11.698054+0000","last_became_peered":"2024-09-10T19:10:11.698054+0000","last_unstale":"2024-09-10T19:11:43.729038+0000","last_undegraded":"2024-09-10T19:11:43.729038+0000","last_fullsized":"2024-09-10T19:11:43.729038+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:11:29.069369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:11:29.069369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:12:55.817612+0000","scrub_duration":0.001327562,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":36,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-10T19:11:43.729325+0000","last_change":"2024-09-10T19:11:34.139371+0000","last_active":"2024-09-10T19:11:43.729325+0000","last_peered":"2024-09-10T19:11:43.729325+0000","last_clean":"2024-09-10T19:11:43.729325+0000","last_became_active":"2024-09-10T19:10:13.722784+0000","last_became_peered":"2024-09-10T19:10:13.722784+0000","last_unstale":"2024-09-10T19:11:43.729325+0000","last_undegraded":"2024-09-10T19:11:43.729325+0000","last_fullsized":"2024-09-10T19:11:43.729325+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:11:34.139306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:11:34.139306+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:12:45.529547+0000","scrub_duration":0.00133668,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.3","version":"66'19","reported_seq":66,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-10T19:11:43.737039+0000","last_change":"2024-09-10T19:10:09.823263+0000","last_active":"2024-09-10T19:11:43.737039+0000","last_peered":"2024-09-10T19:11:43.737039+0000","last_clean":"2024-09-10T19:11:43.737039+0000","last_became_active":"2024-09-10T19:10:09.823003+0000","last_became_peered":"2024-09-10T19:10:09.823003+0000","last_unstale":"2024-09-10T19:11:43.737039+0000","last_undegraded":"2024-09-10T19:11:43.737039+0000","last_fullsized":"2024-09-10T19:11:43.737039+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:10:08.671214+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":42,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-10T19:11:43.739372+0000","last_change":"2024-09-10T19:11:25.290147+0000","last_active":"2024-09-10T19:11:43.739372+0000","last_peered":"2024-09-10T19:11:43.739372+0000","last_clean":"2024-09-10T19:11:43.739372+0000","last_became_active":"2024-09-10T19:10:07.690325+0000","last_became_peered":"2024-09-10T19:10:07.690325+0000","last_unstale":"2024-09-10T19:11:43.739372+0000","last_undegraded":"2024-09-10T19:11:43.739372+0000","last_fullsized":"2024-09-10T19:11:43.739372+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:11:25.290088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:11:25.290088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:12:31.270129+0000","scrub_duration":0.0013977760000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":38,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-10T19:11:43.741525+0000","last_change":"2024-09-10T19:11:23.843315+0000","last_active":"2024-09-10T19:11:43.741525+0000","last_peered":"2024-09-10T19:11:43.741525+0000","last_clean":"2024-09-10T19:11:43.741525+0000","last_became_active":"2024-09-10T19:10:11.700370+0000","last_became_peered":"2024-09-10T19:10:11.700370+0000","last_unstale":"2024-09-10T19:11:43.741525+0000","last_undegraded":"2024-09-10T19:11:43.741525+0000","last_fullsized":"2024-09-10T19:11:43.741525+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:11:23.843228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:11:23.843228+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:12:29.095733+0000","scrub_duration":0.0013052070000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":26,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-10T19:11:43.738796+0000","last_change":"2024-09-10T19:10:14.054654+0000","last_active":"2024-09-10T19:11:43.738796+0000","last_peered":"2024-09-10T19:11:43.738796+0000","last_clean":"2024-09-10T19:11:43.738796+0000","last_became_active":"2024-09-10T19:10:14.054318+0000","last_became_peered":"2024-09-10T19:10:14.054318+0000","last_unstale":"2024-09-10T19:11:43.738796+0000","last_undegraded":"2024-09-10T19:11:43.738796+0000","last_fullsized":"2024-09-10T19:11:43.738796+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:10:12.693598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"66'10","reported_seq":22,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-09-10T19:11:44.754488+0000","last_change":"2024-09-10T19:11:44.754488+0000","last_active":"2024-09-10T19:11:44.754488+0000","last_peered":"2024-09-10T19:11:44.754488+0000","last_clean":"2024-09-10T19:11:44.754488+0000","last_became_active":"2024-09-10T19:11:44.748517+0000","last_became_peered":"2024-09-10T19:11:44.748517+0000","last_unstale":"2024-09-10T19:11:44.754488+0000","last_undegraded":"2024-09-10T19:11:44.754488+0000","last_fullsized":"2024-09-10T19:11:44.754488+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'10","last_scrub_stamp":"2024-09-10T19:11:28.132850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:11:28.132850+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:12:49.124860+0000","scrub_duration":0.0018173429999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":32,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-10T19:11:43.739836+0000","last_change":"2024-09-10T19:10:07.682243+0000","last_active":"2024-09-10T19:11:43.739836+0000","last_peered":"2024-09-10T19:11:43.739836+0000","last_clean":"2024-09-10T19:11:43.739836+0000","last_became_active":"2024-09-10T19:10:07.681965+0000","last_became_peered":"2024-09-10T19:10:07.681965+0000","last_unstale":"2024-09-10T19:11:43.739836+0000","last_undegraded":"2024-09-10T19:11:43.739836+0000","last_fullsized":"2024-09-10T19:11:43.739836+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:10:06.664709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"66'8","reported_seq":56,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-09-10T19:11:44.747227+0000","last_change":"2024-09-10T19:10:11.699975+0000","last_active":"2024-09-10T19:11:44.747227+0000","last_peered":"2024-09-10T19:11:44.747227+0000","last_clean":"2024-09-10T19:11:44.747227+0000","last_became_active":"2024-09-10T19:10:11.699690+0000","last_became_peered":"2024-09-10T19:10:11.699690+0000","last_unstale":"2024-09-10T19:11:44.747227+0000","last_undegraded":"2024-09-10T19:11:44.747227+0000","last_fullsized":"2024-09-10T19:11:44.747227+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:10:10.676876+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":36,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-10T19:11:43.922626+0000","last_change":"2024-09-10T19:11:43.922577+0000","last_active":"2024-09-10T19:11:43.922626+0000","last_peered":"2024-09-10T19:11:43.922626+0000","last_clean":"2024-09-10T19:11:43.922626+0000","last_became_active":"2024-09-10T19:10:13.720403+0000","last_became_peered":"2024-09-10T19:10:13.720403+0000","last_unstale":"2024-09-10T19:11:43.922626+0000","last_undegraded":"2024-09-10T19:11:43.922626+0000","last_fullsized":"2024-09-10T19:11:43.922626+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:11:43.922522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:11:43.922522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:13:13.573808+0000","scrub_duration":0.0012347289999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"66'28","reported_seq":108,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-09-10T19:11:44.743733+0000","last_change":"2024-09-10T19:11:17.050501+0000","last_active":"2024-09-10T19:11:44.743733+0000","last_peered":"2024-09-10T19:11:44.743733+0000","last_clean":"2024-09-10T19:11:44.743733+0000","last_became_active":"2024-09-10T19:10:09.692663+0000","last_became_peered":"2024-09-10T19:10:09.692663+0000","last_unstale":"2024-09-10T19:11:44.743733+0000","last_undegraded":"2024-09-10T19:11:44.743733+0000","last_fullsized":"2024-09-10T19:11:44.743733+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'28","last_scrub_stamp":"2024-09-10T19:11:17.050442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:11:17.050442+0000","objects_scrubbed":10,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:12:31.661637+0000","scrub_duration":0.001985559,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":42,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-10T19:11:43.740343+0000","last_change":"2024-09-10T19:11:41.429457+0000","last_active":"2024-09-10T19:11:43.740343+0000","last_peered":"2024-09-10T19:11:43.740343+0000","last_clean":"2024-09-10T19:11:43.740343+0000","last_became_active":"2024-09-10T19:10:07.682724+0000","last_became_peered":"2024-09-10T19:10:07.682724+0000","last_unstale":"2024-09-10T19:11:43.740343+0000","last_undegraded":"2024-09-10T19:11:43.740343+0000","last_fullsized":"2024-09-10T19:11:43.740343+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:11:41.429402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:11:41.429402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:12:58.343046+0000","scrub_duration":0.0014226359999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":38,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-10T19:11:43.737081+0000","last_change":"2024-09-10T19:11:14.985340+0000","last_active":"2024-09-10T19:11:43.737081+0000","last_peered":"2024-09-10T19:11:43.737081+0000","last_clean":"2024-09-10T19:11:43.737081+0000","last_became_active":"2024-09-10T19:10:11.698825+0000","last_became_peered":"2024-09-10T19:10:11.698825+0000","last_unstale":"2024-09-10T19:11:43.737081+0000","last_undegraded":"2024-09-10T19:11:43.737081+0000","last_fullsized":"2024-09-10T19:11:43.737081+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:11:14.985249+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:11:14.985249+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:12:30.155012+0000","scrub_duration":0.001460161,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":26,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-10T19:11:43.738476+0000","last_change":"2024-09-10T19:10:13.722038+0000","last_active":"2024-09-10T19:11:43.738476+0000","last_peered":"2024-09-10T19:11:43.738476+0000","last_clean":"2024-09-10T19:11:43.738476+0000","last_became_active":"2024-09-10T19:10:13.721744+0000","last_became_peered":"2024-09-10T19:10:13.721744+0000","last_unstale":"2024-09-10T19:11:43.738476+0000","last_undegraded":"2024-09-10T19:11:43.738476+0000","last_fullsized":"2024-09-10T19:11:43.738476+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:10:12.693598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.7","version":"82'18","reported_seq":72,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-09-10T19:11:49.207404+0000","last_change":"2024-09-10T19:11:49.207365+0000","last_active":"2024-09-10T19:11:49.207404+0000","last_peered":"2024-09-10T19:11:49.207404+0000","last_clean":"2024-09-10T19:11:49.207404+0000","last_became_active":"2024-09-10T19:10:09.693346+0000","last_became_peered":"2024-09-10T19:10:09.693346+0000","last_unstale":"2024-09-10T19:11:49.207404+0000","last_undegraded":"2024-09-10T19:11:49.207404+0000","last_fullsized":"2024-09-10T19:11:49.207404+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'18","last_scrub_stamp":"2024-09-10T19:11:49.207309+0000","last_deep_scrub":"82'18","last_deep_scrub_stamp":"2024-09-10T19:11:49.207309+0000","last_clean_scrub_stamp":"2024-09-10T19:11:49.207309+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:13:10.129757+0000","scrub_duration":0.0046031240000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":19,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"67'5","reported_seq":146,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-09-10T19:11:47.930213+0000","last_change":"2024-09-10T19:11:16.007355+0000","last_active":"2024-09-10T19:11:47.930213+0000","last_peered":"2024-09-10T19:11:47.930213+0000","last_clean":"2024-09-10T19:11:47.930213+0000","last_became_active":"2024-09-10T19:10:07.682125+0000","last_became_peered":"2024-09-10T19:10:07.682125+0000","last_unstale":"2024-09-10T19:11:47.930213+0000","last_undegraded":"2024-09-10T19:11:47.930213+0000","last_fullsized":"2024-09-10T19:11:47.930213+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'5","last_scrub_stamp":"2024-09-10T19:11:16.007261+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:11:16.007261+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:12:39.490457+0000","scrub_duration":0.0019059680000000001,"objects_trimmed":0,"snaptrim_duration":0.00034454499999999997,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":96,"num_read_kb":91,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":42,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-10T19:11:43.735419+0000","last_change":"2024-09-10T19:11:33.340773+0000","last_active":"2024-09-10T19:11:43.735419+0000","last_peered":"2024-09-10T19:11:43.735419+0000","last_clean":"2024-09-10T19:11:43.735419+0000","last_became_active":"2024-09-10T19:10:07.681456+0000","last_became_peered":"2024-09-10T19:10:07.681456+0000","last_unstale":"2024-09-10T19:11:43.735419+0000","last_undegraded":"2024-09-10T19:11:43.735419+0000","last_fullsized":"2024-09-10T19:11:43.735419+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:11:33.340715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:11:33.340715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:12:37.277130+0000","scrub_duration":0.0012840099999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":38,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-10T19:11:43.735523+0000","last_change":"2024-09-10T19:11:36.443407+0000","last_active":"2024-09-10T19:11:43.735523+0000","last_peered":"2024-09-10T19:11:43.735523+0000","last_clean":"2024-09-10T19:11:43.735523+0000","last_became_active":"2024-09-10T19:10:11.695205+0000","last_became_peered":"2024-09-10T19:10:11.695205+0000","last_unstale":"2024-09-10T19:11:43.735523+0000","last_undegraded":"2024-09-10T19:11:43.735523+0000","last_fullsized":"2024-09-10T19:11:43.735523+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:11:36.443352+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:11:36.443352+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:12:59.383628+0000","scrub_duration":0.001511814,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":26,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-10T19:11:43.738922+0000","last_change":"2024-09-10T19:10:14.054047+0000","last_active":"2024-09-10T19:11:43.738922+0000","last_peered":"2024-09-10T19:11:43.738922+0000","last_clean":"2024-09-10T19:11:43.738922+0000","last_became_active":"2024-09-10T19:10:14.053697+0000","last_became_peered":"2024-09-10T19:10:14.053697+0000","last_unstale":"2024-09-10T19:11:43.738922+0000","last_undegraded":"2024-09-10T19:11:43.738922+0000","last_fullsized":"2024-09-10T19:11:43.738922+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:10:12.693598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"66'12","reported_seq":58,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-10T19:11:43.735969+0000","last_change":"2024-09-10T19:11:38.936913+0000","last_active":"2024-09-10T19:11:43.735969+0000","last_peered":"2024-09-10T19:11:43.735969+0000","last_clean":"2024-09-10T19:11:43.735969+0000","last_became_active":"2024-09-10T19:10:09.688091+0000","last_became_peered":"2024-09-10T19:10:09.688091+0000","last_unstale":"2024-09-10T19:11:43.735969+0000","last_undegraded":"2024-09-10T19:11:43.735969+0000","last_fullsized":"2024-09-10T19:11:43.735969+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'12","last_scrub_stamp":"2024-09-10T19:11:38.936857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:11:38.936857+0000","objects_scrubbed":6,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:12:55.623292+0000","scrub_duration":0.0016916909999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"61'1","reported_seq":38,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-10T19:11:43.735361+0000","last_change":"2024-09-10T19:10:11.693049+0000","last_active":"2024-09-10T19:11:43.735361+0000","last_peered":"2024-09-10T19:11:43.735361+0000","last_clean":"2024-09-10T19:11:43.735361+0000","last_became_active":"2024-09-10T19:10:07.683259+0000","last_became_peered":"2024-09-10T19:10:07.683259+0000","last_unstale":"2024-09-10T19:11:43.735361+0000","last_undegraded":"2024-09-10T19:11:43.735361+0000","last_fullsized":"2024-09-10T19:11:43.735361+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:10:06.664709+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014341130000000001,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":42,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-10T19:11:43.738245+0000","last_change":"2024-09-10T19:11:21.301130+0000","last_active":"2024-09-10T19:11:43.738245+0000","last_peered":"2024-09-10T19:11:43.738245+0000","last_clean":"2024-09-10T19:11:43.738245+0000","last_became_active":"2024-09-10T19:10:07.690549+0000","last_became_peered":"2024-09-10T19:10:07.690549+0000","last_unstale":"2024-09-10T19:11:43.738245+0000","last_undegraded":"2024-09-10T19:11:43.738245+0000","last_fullsized":"2024-09-10T19:11:43.738245+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:11:21.301058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:11:21.301058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:12:41.356053+0000","scrub_duration":0.0021074710000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":38,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-10T19:11:43.739783+0000","last_change":"2024-09-10T19:11:16.156929+0000","last_active":"2024-09-10T19:11:43.739783+0000","last_peered":"2024-09-10T19:11:43.739783+0000","last_clean":"2024-09-10T19:11:43.739783+0000","last_became_active":"2024-09-10T19:10:11.699798+0000","last_became_peered":"2024-09-10T19:10:11.699798+0000","last_unstale":"2024-09-10T19:11:43.739783+0000","last_undegraded":"2024-09-10T19:11:43.739783+0000","last_fullsized":"2024-09-10T19:11:43.739783+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:11:16.156819+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:11:16.156819+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:12:16.166187+0000","scrub_duration":0.001649059,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":26,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-10T19:11:43.739481+0000","last_change":"2024-09-10T19:10:13.722695+0000","last_active":"2024-09-10T19:11:43.739481+0000","last_peered":"2024-09-10T19:11:43.739481+0000","last_clean":"2024-09-10T19:11:43.739481+0000","last_became_active":"2024-09-10T19:10:13.722368+0000","last_became_peered":"2024-09-10T19:10:13.722368+0000","last_unstale":"2024-09-10T19:11:43.739481+0000","last_undegraded":"2024-09-10T19:11:43.739481+0000","last_fullsized":"2024-09-10T19:11:43.739481+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:10:12.693598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"80'21","reported_seq":69,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-10T19:11:43.741989+0000","last_change":"2024-09-10T19:11:39.836197+0000","last_active":"2024-09-10T19:11:43.741989+0000","last_peered":"2024-09-10T19:11:43.741989+0000","last_clean":"2024-09-10T19:11:43.741989+0000","last_became_active":"2024-09-10T19:10:09.822184+0000","last_became_peered":"2024-09-10T19:10:09.822184+0000","last_unstale":"2024-09-10T19:11:43.741989+0000","last_undegraded":"2024-09-10T19:11:43.741989+0000","last_fullsized":"2024-09-10T19:11:43.741989+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"80'21","last_scrub_stamp":"2024-09-10T19:11:39.836144+0000","last_deep_scrub":"80'21","last_deep_scrub_stamp":"2024-09-10T19:11:39.836144+0000","last_clean_scrub_stamp":"2024-09-10T19:11:39.836144+0000","objects_scrubbed":7,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:12:48.908909+0000","scrub_duration":0.0049231029999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"63'2","reported_seq":49,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-10T19:11:43.737535+0000","last_change":"2024-09-10T19:11:26.824164+0000","last_active":"2024-09-10T19:11:43.737535+0000","last_peered":"2024-09-10T19:11:43.737535+0000","last_clean":"2024-09-10T19:11:43.737535+0000","last_became_active":"2024-09-10T19:10:07.684784+0000","last_became_peered":"2024-09-10T19:10:07.684784+0000","last_unstale":"2024-09-10T19:11:43.737535+0000","last_undegraded":"2024-09-10T19:11:43.737535+0000","last_fullsized":"2024-09-10T19:11:43.737535+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'2","last_scrub_stamp":"2024-09-10T19:11:26.824103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:11:26.824103+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:12:48.229817+0000","scrub_duration":0.0018538929999999999,"objects_trimmed":0,"snaptrim_duration":0.00075522999999999996,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":42,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-10T19:11:43.739801+0000","last_change":"2024-09-10T19:11:17.171319+0000","last_active":"2024-09-10T19:11:43.739801+0000","last_peered":"2024-09-10T19:11:43.739801+0000","last_clean":"2024-09-10T19:11:43.739801+0000","last_became_active":"2024-09-10T19:10:07.687596+0000","last_became_peered":"2024-09-10T19:10:07.687596+0000","last_unstale":"2024-09-10T19:11:43.739801+0000","last_undegraded":"2024-09-10T19:11:43.739801+0000","last_fullsized":"2024-09-10T19:11:43.739801+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:11:17.171251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:11:17.171251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:12:21.736124+0000","scrub_duration":0.0016247670000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"74'79","reported_seq":77,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-10T19:11:43.738429+0000","last_change":"2024-09-10T19:10:34.385577+0000","last_active":"2024-09-10T19:11:43.738429+0000","last_peered":"2024-09-10T19:11:43.738429+0000","last_clean":"2024-09-10T19:11:43.738429+0000","last_became_active":"2024-09-10T19:09:56.634919+0000","last_became_peered":"2024-09-10T19:09:56.634919+0000","last_unstale":"2024-09-10T19:11:43.738429+0000","last_undegraded":"2024-09-10T19:11:43.738429+0000","last_fullsized":"2024-09-10T19:11:43.738429+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"22'72","last_scrub_stamp":"2024-09-10T19:10:34.385519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:08:00.699315+0000","last_clean_scrub_stamp":"2024-09-10T19:10:34.385519+0000","objects_scrubbed":2,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:11:51.371427+0000","scrub_duration":0.0030997160000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":28,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-10T19:11:43.738655+0000","last_change":"2024-09-10T19:10:11.695316+0000","last_active":"2024-09-10T19:11:43.738655+0000","last_peered":"2024-09-10T19:11:43.738655+0000","last_clean":"2024-09-10T19:11:43.738655+0000","last_became_active":"2024-09-10T19:10:11.694986+0000","last_became_peered":"2024-09-10T19:10:11.694986+0000","last_unstale":"2024-09-10T19:11:43.738655+0000","last_undegraded":"2024-09-10T19:11:43.738655+0000","last_fullsized":"2024-09-10T19:11:43.738655+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:10:10.676876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":36,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-10T19:11:43.739982+0000","last_change":"2024-09-10T19:11:25.312410+0000","last_active":"2024-09-10T19:11:43.739982+0000","last_peered":"2024-09-10T19:11:43.739982+0000","last_clean":"2024-09-10T19:11:43.739982+0000","last_became_active":"2024-09-10T19:10:13.720647+0000","last_became_peered":"2024-09-10T19:10:13.720647+0000","last_unstale":"2024-09-10T19:11:43.739982+0000","last_undegraded":"2024-09-10T19:11:43.739982+0000","last_fullsized":"2024-09-10T19:11:43.739982+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:11:25.312356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:11:25.312356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:12:54.878799+0000","scrub_duration":0.001690642,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.a","version":"66'19","reported_seq":71,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-10T19:11:43.741673+0000","last_change":"2024-09-10T19:11:42.941440+0000","last_active":"2024-09-10T19:11:43.741673+0000","last_peered":"2024-09-10T19:11:43.741673+0000","last_clean":"2024-09-10T19:11:43.741673+0000","last_became_active":"2024-09-10T19:10:09.820880+0000","last_became_peered":"2024-09-10T19:10:09.820880+0000","last_unstale":"2024-09-10T19:11:43.741673+0000","last_undegraded":"2024-09-10T19:11:43.741673+0000","last_fullsized":"2024-09-10T19:11:43.741673+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'19","last_scrub_stamp":"2024-09-10T19:11:42.941375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:11:42.941375+0000","objects_scrubbed":9,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:13:12.267779+0000","scrub_duration":0.0021647599999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":42,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-10T19:11:43.735995+0000","last_change":"2024-09-10T19:11:40.484394+0000","last_active":"2024-09-10T19:11:43.735995+0000","last_peered":"2024-09-10T19:11:43.735995+0000","last_clean":"2024-09-10T19:11:43.735995+0000","last_became_active":"2024-09-10T19:10:07.686236+0000","last_became_peered":"2024-09-10T19:10:07.686236+0000","last_unstale":"2024-09-10T19:11:43.735995+0000","last_undegraded":"2024-09-10T19:11:43.735995+0000","last_fullsized":"2024-09-10T19:11:43.735995+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:11:40.484333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:11:40.484333+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:13:05.193849+0000","scrub_duration":0.0013800889999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":38,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-10T19:11:43.735278+0000","last_change":"2024-09-10T19:11:19.119050+0000","last_active":"2024-09-10T19:11:43.735278+0000","last_peered":"2024-09-10T19:11:43.735278+0000","last_clean":"2024-09-10T19:11:43.735278+0000","last_became_active":"2024-09-10T19:10:11.699745+0000","last_became_peered":"2024-09-10T19:10:11.699745+0000","last_unstale":"2024-09-10T19:11:43.735278+0000","last_undegraded":"2024-09-10T19:11:43.735278+0000","last_fullsized":"2024-09-10T19:11:43.735278+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:11:19.118987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:11:19.118987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:12:48.847171+0000","scrub_duration":0.0012170379999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":36,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-10T19:11:43.738420+0000","last_change":"2024-09-10T19:11:38.102859+0000","last_active":"2024-09-10T19:11:43.738420+0000","last_peered":"2024-09-10T19:11:43.738420+0000","last_clean":"2024-09-10T19:11:43.738420+0000","last_became_active":"2024-09-10T19:10:13.719973+0000","last_became_peered":"2024-09-10T19:10:13.719973+0000","last_unstale":"2024-09-10T19:11:43.738420+0000","last_undegraded":"2024-09-10T19:11:43.738420+0000","last_fullsized":"2024-09-10T19:11:43.738420+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:11:38.102803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:11:38.102803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:12:52.451650+0000","scrub_duration":0.001577272,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"66'9","reported_seq":56,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-10T19:11:43.736961+0000","last_change":"2024-09-10T19:11:22.870432+0000","last_active":"2024-09-10T19:11:43.736961+0000","last_peered":"2024-09-10T19:11:43.736961+0000","last_clean":"2024-09-10T19:11:43.736961+0000","last_became_active":"2024-09-10T19:10:09.689117+0000","last_became_peered":"2024-09-10T19:10:09.689117+0000","last_unstale":"2024-09-10T19:11:43.736961+0000","last_undegraded":"2024-09-10T19:11:43.736961+0000","last_fullsized":"2024-09-10T19:11:43.736961+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'9","last_scrub_stamp":"2024-09-10T19:11:22.870369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:11:22.870369+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:12:27.485364+0000","scrub_duration":0.0017468729999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":42,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-10T19:11:44.254608+0000","last_change":"2024-09-10T19:11:44.254567+0000","last_active":"2024-09-10T19:11:44.254608+0000","last_peered":"2024-09-10T19:11:44.254608+0000","last_clean":"2024-09-10T19:11:44.254608+0000","last_became_active":"2024-09-10T19:10:07.682834+0000","last_became_peered":"2024-09-10T19:10:07.682834+0000","last_unstale":"2024-09-10T19:11:44.254608+0000","last_undegraded":"2024-09-10T19:11:44.254608+0000","last_fullsized":"2024-09-10T19:11:44.254608+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:11:44.254513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:11:44.254513+0000","last_clean_scrub_stamp":"2024-09-10T19:11:44.254513+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:13:01.679588+0000","scrub_duration":0.0013179349999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":38,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-10T19:11:43.736914+0000","last_change":"2024-09-10T19:11:42.480542+0000","last_active":"2024-09-10T19:11:43.736914+0000","last_peered":"2024-09-10T19:11:43.736914+0000","last_clean":"2024-09-10T19:11:43.736914+0000","last_became_active":"2024-09-10T19:10:11.694405+0000","last_became_peered":"2024-09-10T19:10:11.694405+0000","last_unstale":"2024-09-10T19:11:43.736914+0000","last_undegraded":"2024-09-10T19:11:43.736914+0000","last_fullsized":"2024-09-10T19:11:43.736914+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:11:42.480487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:11:42.480487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:13:07.856129+0000","scrub_duration":0.0012878259999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":36,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-10T19:11:43.735288+0000","last_change":"2024-09-10T19:11:17.974993+0000","last_active":"2024-09-10T19:11:43.735288+0000","last_peered":"2024-09-10T19:11:43.735288+0000","last_clean":"2024-09-10T19:11:43.735288+0000","last_became_active":"2024-09-10T19:10:13.714952+0000","last_became_peered":"2024-09-10T19:10:13.714952+0000","last_unstale":"2024-09-10T19:11:43.735288+0000","last_undegraded":"2024-09-10T19:11:43.735288+0000","last_fullsized":"2024-09-10T19:11:43.735288+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:11:17.974937+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:11:17.974937+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:12:32.960908+0000","scrub_duration":0.001345834,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.8","version":"66'15","reported_seq":65,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-10T19:11:43.739607+0000","last_change":"2024-09-10T19:11:33.481823+0000","last_active":"2024-09-10T19:11:43.739607+0000","last_peered":"2024-09-10T19:11:43.739607+0000","last_clean":"2024-09-10T19:11:43.739607+0000","last_became_active":"2024-09-10T19:10:09.820950+0000","last_became_peered":"2024-09-10T19:10:09.820950+0000","last_unstale":"2024-09-10T19:11:43.739607+0000","last_undegraded":"2024-09-10T19:11:43.739607+0000","last_fullsized":"2024-09-10T19:11:43.739607+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'15","last_scrub_stamp":"2024-09-10T19:11:33.481743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:11:33.481743+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:12:42.712551+0000","scrub_duration":0.002098094,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":42,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-10T19:11:43.735691+0000","last_change":"2024-09-10T19:11:15.197602+0000","last_active":"2024-09-10T19:11:43.735691+0000","last_peered":"2024-09-10T19:11:43.735691+0000","last_clean":"2024-09-10T19:11:43.735691+0000","last_became_active":"2024-09-10T19:10:07.682286+0000","last_became_peered":"2024-09-10T19:10:07.682286+0000","last_unstale":"2024-09-10T19:11:43.735691+0000","last_undegraded":"2024-09-10T19:11:43.735691+0000","last_fullsized":"2024-09-10T19:11:43.735691+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:11:15.197497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:11:15.197497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:12:23.597035+0000","scrub_duration":0.0019717929999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.9","version":"80'9","reported_seq":67,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-09-10T19:11:44.927715+0000","last_change":"2024-09-10T19:11:35.174273+0000","last_active":"2024-09-10T19:11:44.927715+0000","last_peered":"2024-09-10T19:11:44.927715+0000","last_clean":"2024-09-10T19:11:44.927715+0000","last_became_active":"2024-09-10T19:10:11.702878+0000","last_became_peered":"2024-09-10T19:10:11.702878+0000","last_unstale":"2024-09-10T19:11:44.927715+0000","last_undegraded":"2024-09-10T19:11:44.927715+0000","last_fullsized":"2024-09-10T19:11:44.927715+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"80'9","last_scrub_stamp":"2024-09-10T19:11:35.174218+0000","last_deep_scrub":"80'9","last_deep_scrub_stamp":"2024-09-10T19:11:35.174218+0000","last_clean_scrub_stamp":"2024-09-10T19:11:35.174218+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:13:04.007735+0000","scrub_duration":0.0019812380000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":36,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-10T19:11:43.740056+0000","last_change":"2024-09-10T19:11:20.184218+0000","last_active":"2024-09-10T19:11:43.740056+0000","last_peered":"2024-09-10T19:11:43.740056+0000","last_clean":"2024-09-10T19:11:43.740056+0000","last_became_active":"2024-09-10T19:10:14.053875+0000","last_became_peered":"2024-09-10T19:10:14.053875+0000","last_unstale":"2024-09-10T19:11:43.740056+0000","last_undegraded":"2024-09-10T19:11:43.740056+0000","last_fullsized":"2024-09-10T19:11:43.740056+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:11:20.184132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:11:20.184132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:12:50.058323+0000","scrub_duration":0.002119522,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"66'12","reported_seq":63,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-09-10T19:11:45.012501+0000","last_change":"2024-09-10T19:11:45.012450+0000","last_active":"2024-09-10T19:11:45.012501+0000","last_peered":"2024-09-10T19:11:45.012501+0000","last_clean":"2024-09-10T19:11:45.012501+0000","last_became_active":"2024-09-10T19:10:09.692627+0000","last_became_peered":"2024-09-10T19:10:09.692627+0000","last_unstale":"2024-09-10T19:11:45.012501+0000","last_undegraded":"2024-09-10T19:11:45.012501+0000","last_fullsized":"2024-09-10T19:11:45.012501+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'12","last_scrub_stamp":"2024-09-10T19:11:45.012377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:11:45.012377+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:13:02.868370+0000","scrub_duration":0.0019105979999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.f","version":"59'2","reported_seq":58,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-10T19:11:43.730251+0000","last_change":"2024-09-10T19:11:25.111486+0000","last_active":"2024-09-10T19:11:43.730251+0000","last_peered":"2024-09-10T19:11:43.730251+0000","last_clean":"2024-09-10T19:11:43.730251+0000","last_became_active":"2024-09-10T19:10:07.681719+0000","last_became_peered":"2024-09-10T19:10:07.681719+0000","last_unstale":"2024-09-10T19:11:43.730251+0000","last_undegraded":"2024-09-10T19:11:43.730251+0000","last_fullsized":"2024-09-10T19:11:43.730251+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'2","last_scrub_stamp":"2024-09-10T19:11:25.111429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:11:25.111429+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:12:31.423912+0000","scrub_duration":0.0014791139999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":14,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":38,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-09-10T19:11:46.587968+0000","last_change":"2024-09-10T19:11:46.587917+0000","last_active":"2024-09-10T19:11:46.587968+0000","last_peered":"2024-09-10T19:11:46.587968+0000","last_clean":"2024-09-10T19:11:46.587968+0000","last_became_active":"2024-09-10T19:10:11.698354+0000","last_became_peered":"2024-09-10T19:10:11.698354+0000","last_unstale":"2024-09-10T19:11:46.587968+0000","last_undegraded":"2024-09-10T19:11:46.587968+0000","last_fullsized":"2024-09-10T19:11:46.587968+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:11:46.587846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:11:46.587846+0000","last_clean_scrub_stamp":"2024-09-10T19:11:46.587846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:12:49.697286+0000","scrub_duration":0.001267142,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":36,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-10T19:11:43.737643+0000","last_change":"2024-09-10T19:11:42.723535+0000","last_active":"2024-09-10T19:11:43.737643+0000","last_peered":"2024-09-10T19:11:43.737643+0000","last_clean":"2024-09-10T19:11:43.737643+0000","last_became_active":"2024-09-10T19:10:13.718962+0000","last_became_peered":"2024-09-10T19:10:13.718962+0000","last_unstale":"2024-09-10T19:11:43.737643+0000","last_undegraded":"2024-09-10T19:11:43.737643+0000","last_fullsized":"2024-09-10T19:11:43.737643+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:11:42.723437+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:11:42.723437+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:13:12.077483+0000","scrub_duration":0.001385631,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.16","version":"66'9","reported_seq":56,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-10T19:11:43.735886+0000","last_change":"2024-09-10T19:11:11.962017+0000","last_active":"2024-09-10T19:11:43.735886+0000","last_peered":"2024-09-10T19:11:43.735886+0000","last_clean":"2024-09-10T19:11:43.735886+0000","last_became_active":"2024-09-10T19:10:09.822994+0000","last_became_peered":"2024-09-10T19:10:09.822994+0000","last_unstale":"2024-09-10T19:11:43.735886+0000","last_undegraded":"2024-09-10T19:11:43.735886+0000","last_fullsized":"2024-09-10T19:11:43.735886+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'9","last_scrub_stamp":"2024-09-10T19:11:11.961960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:11:11.961960+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:12:16.945811+0000","scrub_duration":0.001903287,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":42,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-10T19:11:43.735965+0000","last_change":"2024-09-10T19:11:41.458577+0000","last_active":"2024-09-10T19:11:43.735965+0000","last_peered":"2024-09-10T19:11:43.735965+0000","last_clean":"2024-09-10T19:11:43.735965+0000","last_became_active":"2024-09-10T19:10:07.678076+0000","last_became_peered":"2024-09-10T19:10:07.678076+0000","last_unstale":"2024-09-10T19:11:43.735965+0000","last_undegraded":"2024-09-10T19:11:43.735965+0000","last_fullsized":"2024-09-10T19:11:43.735965+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:11:41.458523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:11:41.458523+0000","last_clean_scrub_stamp":"2024-09-10T19:11:41.458523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:12:47.189060+0000","scrub_duration":0.0012499830000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":28,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-10T19:11:43.738581+0000","last_change":"2024-09-10T19:10:11.693961+0000","last_active":"2024-09-10T19:11:43.738581+0000","last_peered":"2024-09-10T19:11:43.738581+0000","last_clean":"2024-09-10T19:11:43.738581+0000","last_became_active":"2024-09-10T19:10:11.693407+0000","last_became_peered":"2024-09-10T19:10:11.693407+0000","last_unstale":"2024-09-10T19:11:43.738581+0000","last_undegraded":"2024-09-10T19:11:43.738581+0000","last_fullsized":"2024-09-10T19:11:43.738581+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:10:10.676876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"66'1","reported_seq":37,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-10T19:11:43.735521+0000","last_change":"2024-09-10T19:11:28.192794+0000","last_active":"2024-09-10T19:11:43.735521+0000","last_peered":"2024-09-10T19:11:43.735521+0000","last_clean":"2024-09-10T19:11:43.735521+0000","last_became_active":"2024-09-10T19:10:13.717001+0000","last_became_peered":"2024-09-10T19:10:13.717001+0000","last_unstale":"2024-09-10T19:11:43.735521+0000","last_undegraded":"2024-09-10T19:11:43.735521+0000","last_fullsized":"2024-09-10T19:11:43.735521+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'1","last_scrub_stamp":"2024-09-10T19:11:28.192745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:11:28.192745+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:12:37.727108+0000","scrub_duration":0.001276323,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":42,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-10T19:11:43.741655+0000","last_change":"2024-09-10T19:11:32.859175+0000","last_active":"2024-09-10T19:11:43.741655+0000","last_peered":"2024-09-10T19:11:43.741655+0000","last_clean":"2024-09-10T19:11:43.741655+0000","last_became_active":"2024-09-10T19:10:07.682963+0000","last_became_peered":"2024-09-10T19:10:07.682963+0000","last_unstale":"2024-09-10T19:11:43.741655+0000","last_undegraded":"2024-09-10T19:11:43.741655+0000","last_fullsized":"2024-09-10T19:11:43.741655+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:11:32.859106+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:11:32.859106+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:12:47.279562+0000","scrub_duration":0.0014403980000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"66'6","reported_seq":39,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-10T19:11:43.738713+0000","last_change":"2024-09-10T19:10:09.822283+0000","last_active":"2024-09-10T19:11:43.738713+0000","last_peered":"2024-09-10T19:11:43.738713+0000","last_clean":"2024-09-10T19:11:43.738713+0000","last_became_active":"2024-09-10T19:10:09.821906+0000","last_became_peered":"2024-09-10T19:10:09.821906+0000","last_unstale":"2024-09-10T19:11:43.738713+0000","last_undegraded":"2024-09-10T19:11:43.738713+0000","last_fullsized":"2024-09-10T19:11:43.738713+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:10:08.671214+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":38,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-10T19:11:43.739580+0000","last_change":"2024-09-10T19:11:23.264870+0000","last_active":"2024-09-10T19:11:43.739580+0000","last_peered":"2024-09-10T19:11:43.739580+0000","last_clean":"2024-09-10T19:11:43.739580+0000","last_became_active":"2024-09-10T19:10:11.692943+0000","last_became_peered":"2024-09-10T19:10:11.692943+0000","last_unstale":"2024-09-10T19:11:43.739580+0000","last_undegraded":"2024-09-10T19:11:43.739580+0000","last_fullsized":"2024-09-10T19:11:43.739580+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:11:23.264793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:11:23.264793+0000","last_clean_scrub_stamp":"2024-09-10T19:11:23.264793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:12:44.248872+0000","scrub_duration":0.0015950649999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":36,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-10T19:11:43.738807+0000","last_change":"2024-09-10T19:11:13.342070+0000","last_active":"2024-09-10T19:11:43.738807+0000","last_peered":"2024-09-10T19:11:43.738807+0000","last_clean":"2024-09-10T19:11:43.738807+0000","last_became_active":"2024-09-10T19:10:14.054360+0000","last_became_peered":"2024-09-10T19:10:14.054360+0000","last_unstale":"2024-09-10T19:11:43.738807+0000","last_undegraded":"2024-09-10T19:11:43.738807+0000","last_fullsized":"2024-09-10T19:11:43.738807+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:11:13.341970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:11:13.341970+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:12:30.833886+0000","scrub_duration":0.001537725,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"80'15","reported_seq":60,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-10T19:11:43.738012+0000","last_change":"2024-09-10T19:11:41.719367+0000","last_active":"2024-09-10T19:11:43.738012+0000","last_peered":"2024-09-10T19:11:43.738012+0000","last_clean":"2024-09-10T19:11:43.738012+0000","last_became_active":"2024-09-10T19:10:09.821810+0000","last_became_peered":"2024-09-10T19:10:09.821810+0000","last_unstale":"2024-09-10T19:11:43.738012+0000","last_undegraded":"2024-09-10T19:11:43.738012+0000","last_fullsized":"2024-09-10T19:11:43.738012+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"80'15","last_scrub_stamp":"2024-09-10T19:11:41.719310+0000","last_deep_scrub":"80'15","last_deep_scrub_stamp":"2024-09-10T19:11:41.719310+0000","last_clean_scrub_stamp":"2024-09-10T19:11:41.719310+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:12:47.895535+0000","scrub_duration":0.0046706430000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":42,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-10T19:11:43.740276+0000","last_change":"2024-09-10T19:11:12.081772+0000","last_active":"2024-09-10T19:11:43.740276+0000","last_peered":"2024-09-10T19:11:43.740276+0000","last_clean":"2024-09-10T19:11:43.740276+0000","last_became_active":"2024-09-10T19:10:07.685870+0000","last_became_peered":"2024-09-10T19:10:07.685870+0000","last_unstale":"2024-09-10T19:11:43.740276+0000","last_undegraded":"2024-09-10T19:11:43.740276+0000","last_fullsized":"2024-09-10T19:11:43.740276+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:11:12.081691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:11:12.081691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:12:41.469111+0000","scrub_duration":0.001629903,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"66'8","reported_seq":56,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-09-10T19:11:44.748332+0000","last_change":"2024-09-10T19:10:11.700895+0000","last_active":"2024-09-10T19:11:44.748332+0000","last_peered":"2024-09-10T19:11:44.748332+0000","last_clean":"2024-09-10T19:11:44.748332+0000","last_became_active":"2024-09-10T19:10:11.700584+0000","last_became_peered":"2024-09-10T19:10:11.700584+0000","last_unstale":"2024-09-10T19:11:44.748332+0000","last_undegraded":"2024-09-10T19:11:44.748332+0000","last_fullsized":"2024-09-10T19:11:44.748332+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:10:10.676876+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":36,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-10T19:11:43.735617+0000","last_change":"2024-09-10T19:11:20.881495+0000","last_active":"2024-09-10T19:11:43.735617+0000","last_peered":"2024-09-10T19:11:43.735617+0000","last_clean":"2024-09-10T19:11:43.735617+0000","last_became_active":"2024-09-10T19:10:13.721035+0000","last_became_peered":"2024-09-10T19:10:13.721035+0000","last_unstale":"2024-09-10T19:11:43.735617+0000","last_undegraded":"2024-09-10T19:11:43.735617+0000","last_fullsized":"2024-09-10T19:11:43.735617+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:11:20.881436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:11:20.881436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:12:49.668407+0000","scrub_duration":0.0014551099999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"66'9","reported_seq":56,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-10T19:11:43.739657+0000","last_change":"2024-09-10T19:11:37.394628+0000","last_active":"2024-09-10T19:11:43.739657+0000","last_peered":"2024-09-10T19:11:43.739657+0000","last_clean":"2024-09-10T19:11:43.739657+0000","last_became_active":"2024-09-10T19:10:09.823006+0000","last_became_peered":"2024-09-10T19:10:09.823006+0000","last_unstale":"2024-09-10T19:11:43.739657+0000","last_undegraded":"2024-09-10T19:11:43.739657+0000","last_fullsized":"2024-09-10T19:11:43.739657+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'9","last_scrub_stamp":"2024-09-10T19:11:37.394573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:11:37.394573+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:12:49.673447+0000","scrub_duration":0.00176055,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":42,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-10T19:11:43.737224+0000","last_change":"2024-09-10T19:11:29.796484+0000","last_active":"2024-09-10T19:11:43.737224+0000","last_peered":"2024-09-10T19:11:43.737224+0000","last_clean":"2024-09-10T19:11:43.737224+0000","last_became_active":"2024-09-10T19:10:07.688499+0000","last_became_peered":"2024-09-10T19:10:07.688499+0000","last_unstale":"2024-09-10T19:11:43.737224+0000","last_undegraded":"2024-09-10T19:11:43.737224+0000","last_fullsized":"2024-09-10T19:11:43.737224+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:11:29.796429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:11:29.796429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:12:53.847123+0000","scrub_duration":0.0012420479999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"66'8","reported_seq":66,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-09-10T19:11:44.744078+0000","last_change":"2024-09-10T19:11:23.870113+0000","last_active":"2024-09-10T19:11:44.744078+0000","last_peered":"2024-09-10T19:11:44.744078+0000","last_clean":"2024-09-10T19:11:44.744078+0000","last_became_active":"2024-09-10T19:10:11.692359+0000","last_became_peered":"2024-09-10T19:10:11.692359+0000","last_unstale":"2024-09-10T19:11:44.744078+0000","last_undegraded":"2024-09-10T19:11:44.744078+0000","last_fullsized":"2024-09-10T19:11:44.744078+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'8","last_scrub_stamp":"2024-09-10T19:11:23.870010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:11:23.870010+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:12:47.854105+0000","scrub_duration":0.0017621309999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":36,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-10T19:11:43.729942+0000","last_change":"2024-09-10T19:11:43.071796+0000","last_active":"2024-09-10T19:11:43.729942+0000","last_peered":"2024-09-10T19:11:43.729942+0000","last_clean":"2024-09-10T19:11:43.729942+0000","last_became_active":"2024-09-10T19:10:13.718285+0000","last_became_peered":"2024-09-10T19:10:13.718285+0000","last_unstale":"2024-09-10T19:11:43.729942+0000","last_undegraded":"2024-09-10T19:11:43.729942+0000","last_fullsized":"2024-09-10T19:11:43.729942+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:11:43.071727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:11:43.071727+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:13:09.326836+0000","scrub_duration":0.00142789,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.12","version":"79'13","reported_seq":60,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-10T19:11:43.739737+0000","last_change":"2024-09-10T19:11:22.072464+0000","last_active":"2024-09-10T19:11:43.739737+0000","last_peered":"2024-09-10T19:11:43.739737+0000","last_clean":"2024-09-10T19:11:43.739737+0000","last_became_active":"2024-09-10T19:10:09.689440+0000","last_became_peered":"2024-09-10T19:10:09.689440+0000","last_unstale":"2024-09-10T19:11:43.739737+0000","last_undegraded":"2024-09-10T19:11:43.739737+0000","last_fullsized":"2024-09-10T19:11:43.739737+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'13","last_scrub_stamp":"2024-09-10T19:11:22.072400+0000","last_deep_scrub":"79'13","last_deep_scrub_stamp":"2024-09-10T19:11:22.072400+0000","last_clean_scrub_stamp":"2024-09-10T19:11:22.072400+0000","objects_scrubbed":4,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:12:33.209191+0000","scrub_duration":0.0032983940000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"59'1","reported_seq":49,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-09-10T19:11:47.926618+0000","last_change":"2024-09-10T19:11:47.926583+0000","last_active":"2024-09-10T19:11:47.926618+0000","last_peered":"2024-09-10T19:11:47.926618+0000","last_clean":"2024-09-10T19:11:47.926618+0000","last_became_active":"2024-09-10T19:10:07.683263+0000","last_became_peered":"2024-09-10T19:10:07.683263+0000","last_unstale":"2024-09-10T19:11:47.926618+0000","last_undegraded":"2024-09-10T19:11:47.926618+0000","last_fullsized":"2024-09-10T19:11:47.926618+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'1","last_scrub_stamp":"2024-09-10T19:11:47.926513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:11:47.926513+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:13:03.515009+0000","scrub_duration":0.001771999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":362,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":28,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-10T19:11:43.737717+0000","last_change":"2024-09-10T19:10:11.701754+0000","last_active":"2024-09-10T19:11:43.737717+0000","last_peered":"2024-09-10T19:11:43.737717+0000","last_clean":"2024-09-10T19:11:43.737717+0000","last_became_active":"2024-09-10T19:10:11.701416+0000","last_became_peered":"2024-09-10T19:10:11.701416+0000","last_unstale":"2024-09-10T19:11:43.737717+0000","last_undegraded":"2024-09-10T19:11:43.737717+0000","last_fullsized":"2024-09-10T19:11:43.737717+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:10:10.676876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":26,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-10T19:11:43.736256+0000","last_change":"2024-09-10T19:10:13.716464+0000","last_active":"2024-09-10T19:11:43.736256+0000","last_peered":"2024-09-10T19:11:43.736256+0000","last_clean":"2024-09-10T19:11:43.736256+0000","last_became_active":"2024-09-10T19:10:13.716198+0000","last_became_peered":"2024-09-10T19:10:13.716198+0000","last_unstale":"2024-09-10T19:11:43.736256+0000","last_undegraded":"2024-09-10T19:11:43.736256+0000","last_fullsized":"2024-09-10T19:11:43.736256+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:10:12.693598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"66'11","reported_seq":59,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-10T19:11:44.025481+0000","last_change":"2024-09-10T19:11:44.025426+0000","last_active":"2024-09-10T19:11:44.025481+0000","last_peered":"2024-09-10T19:11:44.025481+0000","last_clean":"2024-09-10T19:11:44.025481+0000","last_became_active":"2024-09-10T19:10:09.693856+0000","last_became_peered":"2024-09-10T19:10:09.693856+0000","last_unstale":"2024-09-10T19:11:44.025481+0000","last_undegraded":"2024-09-10T19:11:44.025481+0000","last_fullsized":"2024-09-10T19:11:44.025481+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'11","last_scrub_stamp":"2024-09-10T19:11:44.025361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:11:44.025361+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:13:09.366530+0000","scrub_duration":0.002036894,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":42,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-10T19:11:43.740261+0000","last_change":"2024-09-10T19:11:10.031299+0000","last_active":"2024-09-10T19:11:43.740261+0000","last_peered":"2024-09-10T19:11:43.740261+0000","last_clean":"2024-09-10T19:11:43.740261+0000","last_became_active":"2024-09-10T19:10:07.685339+0000","last_became_peered":"2024-09-10T19:10:07.685339+0000","last_unstale":"2024-09-10T19:11:43.740261+0000","last_undegraded":"2024-09-10T19:11:43.740261+0000","last_fullsized":"2024-09-10T19:11:43.740261+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:11:10.031245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:11:10.031245+0000","last_clean_scrub_stamp":"2024-09-10T19:11:10.031245+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:12:27.675011+0000","scrub_duration":0.0014113439999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":38,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-10T19:11:43.739655+0000","last_change":"2024-09-10T19:11:31.249400+0000","last_active":"2024-09-10T19:11:43.739655+0000","last_peered":"2024-09-10T19:11:43.739655+0000","last_clean":"2024-09-10T19:11:43.739655+0000","last_became_active":"2024-09-10T19:10:11.694564+0000","last_became_peered":"2024-09-10T19:10:11.694564+0000","last_unstale":"2024-09-10T19:11:43.739655+0000","last_undegraded":"2024-09-10T19:11:43.739655+0000","last_fullsized":"2024-09-10T19:11:43.739655+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:11:31.249343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:11:31.249343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:12:58.372067+0000","scrub_duration":0.001093644,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":36,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-10T19:11:43.735374+0000","last_change":"2024-09-10T19:11:16.813629+0000","last_active":"2024-09-10T19:11:43.735374+0000","last_peered":"2024-09-10T19:11:43.735374+0000","last_clean":"2024-09-10T19:11:43.735374+0000","last_became_active":"2024-09-10T19:10:13.724201+0000","last_became_peered":"2024-09-10T19:10:13.724201+0000","last_unstale":"2024-09-10T19:11:43.735374+0000","last_undegraded":"2024-09-10T19:11:43.735374+0000","last_fullsized":"2024-09-10T19:11:43.735374+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:11:16.813575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:11:16.813575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:12:44.427906+0000","scrub_duration":0.001237312,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"66'4","reported_seq":46,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-10T19:11:43.735343+0000","last_change":"2024-09-10T19:11:14.816693+0000","last_active":"2024-09-10T19:11:43.735343+0000","last_peered":"2024-09-10T19:11:43.735343+0000","last_clean":"2024-09-10T19:11:43.735343+0000","last_became_active":"2024-09-10T19:10:09.692751+0000","last_became_peered":"2024-09-10T19:10:09.692751+0000","last_unstale":"2024-09-10T19:11:43.735343+0000","last_undegraded":"2024-09-10T19:11:43.735343+0000","last_fullsized":"2024-09-10T19:11:43.735343+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'4","last_scrub_stamp":"2024-09-10T19:11:14.816632+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:11:14.816632+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:12:42.695105+0000","scrub_duration":0.001902504,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":42,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-10T19:11:43.739794+0000","last_change":"2024-09-10T19:11:24.275061+0000","last_active":"2024-09-10T19:11:43.739794+0000","last_peered":"2024-09-10T19:11:43.739794+0000","last_clean":"2024-09-10T19:11:43.739794+0000","last_became_active":"2024-09-10T19:10:07.687723+0000","last_became_peered":"2024-09-10T19:10:07.687723+0000","last_unstale":"2024-09-10T19:11:43.739794+0000","last_undegraded":"2024-09-10T19:11:43.739794+0000","last_fullsized":"2024-09-10T19:11:43.739794+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:11:24.275004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:11:24.275004+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:12:41.723440+0000","scrub_duration":0.001588014,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":38,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-10T19:11:43.741495+0000","last_change":"2024-09-10T19:11:31.838005+0000","last_active":"2024-09-10T19:11:43.741495+0000","last_peered":"2024-09-10T19:11:43.741495+0000","last_clean":"2024-09-10T19:11:43.741495+0000","last_became_active":"2024-09-10T19:10:11.698536+0000","last_became_peered":"2024-09-10T19:10:11.698536+0000","last_unstale":"2024-09-10T19:11:43.741495+0000","last_undegraded":"2024-09-10T19:11:43.741495+0000","last_fullsized":"2024-09-10T19:11:43.741495+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:11:31.837944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:11:31.837944+0000","last_clean_scrub_stamp":"2024-09-10T19:11:31.837944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:12:53.987457+0000","scrub_duration":0.001263639,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":36,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-10T19:11:43.738179+0000","last_change":"2024-09-10T19:11:14.298616+0000","last_active":"2024-09-10T19:11:43.738179+0000","last_peered":"2024-09-10T19:11:43.738179+0000","last_clean":"2024-09-10T19:11:43.738179+0000","last_became_active":"2024-09-10T19:10:13.718271+0000","last_became_peered":"2024-09-10T19:10:13.718271+0000","last_unstale":"2024-09-10T19:11:43.738179+0000","last_undegraded":"2024-09-10T19:11:43.738179+0000","last_fullsized":"2024-09-10T19:11:43.738179+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:11:14.298545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:11:14.298545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:12:15.177239+0000","scrub_duration":0.001561799,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.11","version":"66'11","reported_seq":59,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-10T19:11:43.735544+0000","last_change":"2024-09-10T19:11:36.743509+0000","last_active":"2024-09-10T19:11:43.735544+0000","last_peered":"2024-09-10T19:11:43.735544+0000","last_clean":"2024-09-10T19:11:43.735544+0000","last_became_active":"2024-09-10T19:10:09.821357+0000","last_became_peered":"2024-09-10T19:10:09.821357+0000","last_unstale":"2024-09-10T19:11:43.735544+0000","last_undegraded":"2024-09-10T19:11:43.735544+0000","last_fullsized":"2024-09-10T19:11:43.735544+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'11","last_scrub_stamp":"2024-09-10T19:11:36.743445+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:11:36.743445+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:13:04.557447+0000","scrub_duration":0.001722374,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":42,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-10T19:11:43.742523+0000","last_change":"2024-09-10T19:11:34.867991+0000","last_active":"2024-09-10T19:11:43.742523+0000","last_peered":"2024-09-10T19:11:43.742523+0000","last_clean":"2024-09-10T19:11:43.742523+0000","last_became_active":"2024-09-10T19:10:07.685223+0000","last_became_peered":"2024-09-10T19:10:07.685223+0000","last_unstale":"2024-09-10T19:11:43.742523+0000","last_undegraded":"2024-09-10T19:11:43.742523+0000","last_fullsized":"2024-09-10T19:11:43.742523+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:11:34.867935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:11:34.867935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:12:56.224806+0000","scrub_duration":0.001504086,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":28,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-10T19:11:43.737945+0000","last_change":"2024-09-10T19:10:11.699248+0000","last_active":"2024-09-10T19:11:43.737945+0000","last_peered":"2024-09-10T19:11:43.737945+0000","last_clean":"2024-09-10T19:11:43.737945+0000","last_became_active":"2024-09-10T19:10:11.699013+0000","last_became_peered":"2024-09-10T19:10:11.699013+0000","last_unstale":"2024-09-10T19:11:43.737945+0000","last_undegraded":"2024-09-10T19:11:43.737945+0000","last_fullsized":"2024-09-10T19:11:43.737945+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:10:10.676876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":36,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-09-10T19:11:46.720372+0000","last_change":"2024-09-10T19:11:46.720343+0000","last_active":"2024-09-10T19:11:46.720372+0000","last_peered":"2024-09-10T19:11:46.720372+0000","last_clean":"2024-09-10T19:11:46.720372+0000","last_became_active":"2024-09-10T19:10:14.056172+0000","last_became_peered":"2024-09-10T19:10:14.056172+0000","last_unstale":"2024-09-10T19:11:46.720372+0000","last_undegraded":"2024-09-10T19:11:46.720372+0000","last_fullsized":"2024-09-10T19:11:46.720372+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:11:46.720280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:11:46.720280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:13:09.566661+0000","scrub_duration":0.0014580260000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"80'2","reported_seq":38,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-10T19:11:43.738604+0000","last_change":"2024-09-10T19:11:30.272094+0000","last_active":"2024-09-10T19:11:43.738604+0000","last_peered":"2024-09-10T19:11:43.738604+0000","last_clean":"2024-09-10T19:11:43.738604+0000","last_became_active":"2024-09-10T19:10:13.718875+0000","last_became_peered":"2024-09-10T19:10:13.718875+0000","last_unstale":"2024-09-10T19:11:43.738604+0000","last_undegraded":"2024-09-10T19:11:43.738604+0000","last_fullsized":"2024-09-10T19:11:43.738604+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"80'2","last_scrub_stamp":"2024-09-10T19:11:30.272043+0000","last_deep_scrub":"80'2","last_deep_scrub_stamp":"2024-09-10T19:11:30.272043+0000","last_clean_scrub_stamp":"2024-09-10T19:11:30.272043+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:12:37.490688+0000","scrub_duration":0.0022929460000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":369,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":42,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-09-10T19:11:49.356450+0000","last_change":"2024-09-10T19:11:49.356398+0000","last_active":"2024-09-10T19:11:49.356450+0000","last_peered":"2024-09-10T19:11:49.356450+0000","last_clean":"2024-09-10T19:11:49.356450+0000","last_became_active":"2024-09-10T19:10:07.689877+0000","last_became_peered":"2024-09-10T19:10:07.689877+0000","last_unstale":"2024-09-10T19:11:49.356450+0000","last_undegraded":"2024-09-10T19:11:49.356450+0000","last_fullsized":"2024-09-10T19:11:49.356450+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:11:49.356336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:11:49.356336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:13:14.019219+0000","scrub_duration":0.001371527,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1e","version":"80'15","reported_seq":60,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-10T19:11:43.735813+0000","last_change":"2024-09-10T19:11:25.884008+0000","last_active":"2024-09-10T19:11:43.735813+0000","last_peered":"2024-09-10T19:11:43.735813+0000","last_clean":"2024-09-10T19:11:43.735813+0000","last_became_active":"2024-09-10T19:10:09.823061+0000","last_became_peered":"2024-09-10T19:10:09.823061+0000","last_unstale":"2024-09-10T19:11:43.735813+0000","last_undegraded":"2024-09-10T19:11:43.735813+0000","last_fullsized":"2024-09-10T19:11:43.735813+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"80'15","last_scrub_stamp":"2024-09-10T19:11:25.883919+0000","last_deep_scrub":"80'15","last_deep_scrub_stamp":"2024-09-10T19:11:25.883919+0000","last_clean_scrub_stamp":"2024-09-10T19:11:25.883919+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:12:31.561992+0000","scrub_duration":0.0034715229999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1f","version":"66'8","reported_seq":69,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-09-10T19:11:44.747245+0000","last_change":"2024-09-10T19:11:30.835414+0000","last_active":"2024-09-10T19:11:44.747245+0000","last_peered":"2024-09-10T19:11:44.747245+0000","last_clean":"2024-09-10T19:11:44.747245+0000","last_became_active":"2024-09-10T19:10:11.695427+0000","last_became_peered":"2024-09-10T19:10:11.695427+0000","last_unstale":"2024-09-10T19:11:44.747245+0000","last_undegraded":"2024-09-10T19:11:44.747245+0000","last_fullsized":"2024-09-10T19:11:44.747245+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'8","last_scrub_stamp":"2024-09-10T19:11:30.835354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:11:30.835354+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:12:42.690611+0000","scrub_duration":0.0013545090000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":36,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-10T19:11:43.739820+0000","last_change":"2024-09-10T19:11:30.210657+0000","last_active":"2024-09-10T19:11:43.739820+0000","last_peered":"2024-09-10T19:11:43.739820+0000","last_clean":"2024-09-10T19:11:43.739820+0000","last_became_active":"2024-09-10T19:10:13.715419+0000","last_became_peered":"2024-09-10T19:10:13.715419+0000","last_unstale":"2024-09-10T19:11:43.739820+0000","last_undegraded":"2024-09-10T19:11:43.739820+0000","last_fullsized":"2024-09-10T19:11:43.739820+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:11:30.210603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:11:30.210603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:12:40.846927+0000","scrub_duration":0.0012894740000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":42,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-10T19:11:43.735856+0000","last_change":"2024-09-10T19:11:34.815849+0000","last_active":"2024-09-10T19:11:43.735856+0000","last_peered":"2024-09-10T19:11:43.735856+0000","last_clean":"2024-09-10T19:11:43.735856+0000","last_became_active":"2024-09-10T19:10:07.682689+0000","last_became_peered":"2024-09-10T19:10:07.682689+0000","last_unstale":"2024-09-10T19:11:43.735856+0000","last_undegraded":"2024-09-10T19:11:43.735856+0000","last_fullsized":"2024-09-10T19:11:43.735856+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:11:34.815794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:11:34.815794+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:13:02.432072+0000","scrub_duration":0.001252501,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"80'16","reported_seq":86,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-09-10T19:11:44.755688+0000","last_change":"2024-09-10T19:11:44.755688+0000","last_active":"2024-09-10T19:11:44.755688+0000","last_peered":"2024-09-10T19:11:44.755688+0000","last_clean":"2024-09-10T19:11:44.755688+0000","last_became_active":"2024-09-10T19:11:44.750039+0000","last_became_peered":"2024-09-10T19:11:44.750039+0000","last_unstale":"2024-09-10T19:11:44.755688+0000","last_undegraded":"2024-09-10T19:11:44.755688+0000","last_fullsized":"2024-09-10T19:11:44.755688+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"80'16","last_scrub_stamp":"2024-09-10T19:11:40.853611+0000","last_deep_scrub":"80'16","last_deep_scrub_stamp":"2024-09-10T19:11:40.853611+0000","last_clean_scrub_stamp":"2024-09-10T19:11:40.853611+0000","objects_scrubbed":5,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:12:46.019580+0000","scrub_duration":0.003886877,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":38,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-10T19:11:43.735741+0000","last_change":"2024-09-10T19:11:28.822385+0000","last_active":"2024-09-10T19:11:43.735741+0000","last_peered":"2024-09-10T19:11:43.735741+0000","last_clean":"2024-09-10T19:11:43.735741+0000","last_became_active":"2024-09-10T19:10:11.701735+0000","last_became_peered":"2024-09-10T19:10:11.701735+0000","last_unstale":"2024-09-10T19:11:43.735741+0000","last_undegraded":"2024-09-10T19:11:43.735741+0000","last_fullsized":"2024-09-10T19:11:43.735741+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:11:28.822327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:11:28.822327+0000","last_clean_scrub_stamp":"2024-09-10T19:11:28.822327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:12:45.908873+0000","scrub_duration":0.0012591119999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":36,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-10T19:11:43.729911+0000","last_change":"2024-09-10T19:11:35.145006+0000","last_active":"2024-09-10T19:11:43.729911+0000","last_peered":"2024-09-10T19:11:43.729911+0000","last_clean":"2024-09-10T19:11:43.729911+0000","last_became_active":"2024-09-10T19:10:14.053619+0000","last_became_peered":"2024-09-10T19:10:14.053619+0000","last_unstale":"2024-09-10T19:11:43.729911+0000","last_undegraded":"2024-09-10T19:11:43.729911+0000","last_fullsized":"2024-09-10T19:11:43.729911+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:11:35.144953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:11:35.144953+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:12:55.849066+0000","scrub_duration":0.001302503,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":32,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-10T19:11:43.741785+0000","last_change":"2024-09-10T19:10:07.686443+0000","last_active":"2024-09-10T19:11:43.741785+0000","last_peered":"2024-09-10T19:11:43.741785+0000","last_clean":"2024-09-10T19:11:43.741785+0000","last_became_active":"2024-09-10T19:10:07.685997+0000","last_became_peered":"2024-09-10T19:10:07.685997+0000","last_unstale":"2024-09-10T19:11:43.741785+0000","last_undegraded":"2024-09-10T19:11:43.741785+0000","last_fullsized":"2024-09-10T19:11:43.741785+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:10:06.664709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1c","version":"78'22","reported_seq":72,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-10T19:11:43.735763+0000","last_change":"2024-09-10T19:11:21.119003+0000","last_active":"2024-09-10T19:11:43.735763+0000","last_peered":"2024-09-10T19:11:43.735763+0000","last_clean":"2024-09-10T19:11:43.735763+0000","last_became_active":"2024-09-10T19:10:09.690328+0000","last_became_peered":"2024-09-10T19:10:09.690328+0000","last_unstale":"2024-09-10T19:11:43.735763+0000","last_undegraded":"2024-09-10T19:11:43.735763+0000","last_fullsized":"2024-09-10T19:11:43.735763+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"78'22","last_scrub_stamp":"2024-09-10T19:11:21.118937+0000","last_deep_scrub":"78'22","last_deep_scrub_stamp":"2024-09-10T19:11:21.118937+0000","last_clean_scrub_stamp":"2024-09-10T19:11:21.118937+0000","objects_scrubbed":7,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:12:21.535383+0000","scrub_duration":0.0048819759999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":38,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-10T19:11:43.739991+0000","last_change":"2024-09-10T19:11:43.221376+0000","last_active":"2024-09-10T19:11:43.739991+0000","last_peered":"2024-09-10T19:11:43.739991+0000","last_clean":"2024-09-10T19:11:43.739991+0000","last_became_active":"2024-09-10T19:10:11.700037+0000","last_became_peered":"2024-09-10T19:10:11.700037+0000","last_unstale":"2024-09-10T19:11:43.739991+0000","last_undegraded":"2024-09-10T19:11:43.739991+0000","last_fullsized":"2024-09-10T19:11:43.739991+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:11:43.221323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:11:43.221323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:12:47.537742+0000","scrub_duration":0.001173189,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":36,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-10T19:11:43.738512+0000","last_change":"2024-09-10T19:11:24.827983+0000","last_active":"2024-09-10T19:11:43.738512+0000","last_peered":"2024-09-10T19:11:43.738512+0000","last_clean":"2024-09-10T19:11:43.738512+0000","last_became_active":"2024-09-10T19:10:14.053698+0000","last_became_peered":"2024-09-10T19:10:14.053698+0000","last_unstale":"2024-09-10T19:11:43.738512+0000","last_undegraded":"2024-09-10T19:11:43.738512+0000","last_fullsized":"2024-09-10T19:11:43.738512+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:11:24.827916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:11:24.827916+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:12:36.645047+0000","scrub_duration":0.0014104860000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":42,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-10T19:11:43.737863+0000","last_change":"2024-09-10T19:11:25.870573+0000","last_active":"2024-09-10T19:11:43.737863+0000","last_peered":"2024-09-10T19:11:43.737863+0000","last_clean":"2024-09-10T19:11:43.737863+0000","last_became_active":"2024-09-10T19:10:07.682811+0000","last_became_peered":"2024-09-10T19:10:07.682811+0000","last_unstale":"2024-09-10T19:11:43.737863+0000","last_undegraded":"2024-09-10T19:11:43.737863+0000","last_fullsized":"2024-09-10T19:11:43.737863+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:11:25.870516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:11:25.870516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:12:33.650020+0000","scrub_duration":0.001292568,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1d","version":"66'12","reported_seq":53,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-10T19:11:43.738206+0000","last_change":"2024-09-10T19:10:09.693108+0000","last_active":"2024-09-10T19:11:43.738206+0000","last_peered":"2024-09-10T19:11:43.738206+0000","last_clean":"2024-09-10T19:11:43.738206+0000","last_became_active":"2024-09-10T19:10:09.692781+0000","last_became_peered":"2024-09-10T19:10:09.692781+0000","last_unstale":"2024-09-10T19:11:43.738206+0000","last_undegraded":"2024-09-10T19:11:43.738206+0000","last_fullsized":"2024-09-10T19:11:43.738206+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:10:08.671214+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":28,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-10T19:11:43.729485+0000","last_change":"2024-09-10T19:10:11.701419+0000","last_active":"2024-09-10T19:11:43.729485+0000","last_peered":"2024-09-10T19:11:43.729485+0000","last_clean":"2024-09-10T19:11:43.729485+0000","last_became_active":"2024-09-10T19:10:11.701155+0000","last_became_peered":"2024-09-10T19:10:11.701155+0000","last_unstale":"2024-09-10T19:11:43.729485+0000","last_undegraded":"2024-09-10T19:11:43.729485+0000","last_fullsized":"2024-09-10T19:11:43.729485+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:10:10.676876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":66,"ondisk_log_size":66,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":177,"num_object_clones":0,"num_object_copies":531,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":177,"num_whiteouts":0,"num_read":698,"num_read_kb":455,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":429,"ondisk_log_size":429,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":96,"num_read_kb":91,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1328,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":27,"num_read_kb":27,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":3984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4,"ondisk_log_size":4,"up":96,"acting":96,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79,"ondisk_log_size":79,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":54,"seq":231928234009,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298836,"kb_used_data":1244,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93446316,"statfs":{"total":95995035648,"available":95689027584,"internally_reserved":0,"allocated":1273856,"data_stored":923098,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 19:10:56 2024","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.53500000000000003},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.70499999999999996,"5min":0.70499999999999996,"15min":0.70499999999999996},"last":0.443},{"interface":"front","average":{"1min":0.53800000000000003,"5min":0.53800000000000003,"15min":0.53800000000000003},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.76200000000000001,"5min":0.76200000000000001,"15min":0.76200000000000001},"last":0.47799999999999998}]},{"osd":1,"last update":"Tue Sep 10 19:10:56 2024","interfaces":[{"interface":"back","average":{"1min":0.52400000000000002,"5min":0.52400000000000002,"15min":0.52400000000000002},"min":{"1min":0.41299999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":0.79400000000000004,"5min":0.79400000000000004,"15min":0.79400000000000004},"last":0.53500000000000003},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"min":{"1min":0.314,"5min":0.314,"15min":0.314},"max":{"1min":0.71799999999999997,"5min":0.71799999999999997,"15min":0.71799999999999997},"last":0.40799999999999997}]},{"osd":2,"last update":"Tue Sep 10 19:10:56 2024","interfaces":[{"interface":"back","average":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"min":{"1min":0.29199999999999998,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.84399999999999997,"5min":0.84399999999999997,"15min":0.84399999999999997},"last":0.60699999999999998},{"interface":"front","average":{"1min":0.55000000000000004,"5min":0.55000000000000004,"15min":0.55000000000000004},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.77300000000000002,"5min":0.77300000000000002,"15min":0.77300000000000002},"last":0.50600000000000001}]},{"osd":3,"last update":"Tue Sep 10 19:10:56 2024","interfaces":[{"interface":"back","average":{"1min":0.53600000000000003,"5min":0.53600000000000003,"15min":0.53600000000000003},"min":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":0.86399999999999999,"5min":0.86399999999999999,"15min":0.86399999999999999},"last":0.57899999999999996},{"interface":"front","average":{"1min":0.52000000000000002,"5min":0.52000000000000002,"15min":0.52000000000000002},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.77600000000000002,"5min":0.77600000000000002,"15min":0.77600000000000002},"last":0.56899999999999995}]},{"osd":4,"last update":"Tue Sep 10 19:10:56 2024","interfaces":[{"interface":"back","average":{"1min":0.45500000000000002,"5min":0.45500000000000002,"15min":0.45500000000000002},"min":{"1min":0.318,"5min":0.318,"15min":0.318},"max":{"1min":0.65000000000000002,"5min":0.65000000000000002,"15min":0.65000000000000002},"last":0.35399999999999998},{"interface":"front","average":{"1min":0.46000000000000002,"5min":0.46000000000000002,"15min":0.46000000000000002},"min":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.60599999999999998,"5min":0.60599999999999998,"15min":0.60599999999999998},"last":0.45700000000000002}]},{"osd":5,"last update":"Tue Sep 10 19:10:56 2024","interfaces":[{"interface":"back","average":{"1min":0.47999999999999998,"5min":0.47999999999999998,"15min":0.47999999999999998},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.76100000000000001,"5min":0.76100000000000001,"15min":0.76100000000000001},"last":0.38300000000000001},{"interface":"front","average":{"1min":0.47599999999999998,"5min":0.47599999999999998,"15min":0.47599999999999998},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.68600000000000005,"5min":0.68600000000000005,"15min":0.68600000000000005},"last":0.54700000000000004}]},{"osd":6,"last update":"Tue Sep 10 19:10:56 2024","interfaces":[{"interface":"back","average":{"1min":0.5,"5min":0.5,"15min":0.5},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":0.65900000000000003,"5min":0.65900000000000003,"15min":0.65900000000000003},"last":0.59199999999999997},{"interface":"front","average":{"1min":0.48899999999999999,"5min":0.48899999999999999,"15min":0.48899999999999999},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.67000000000000004,"5min":0.67000000000000004,"15min":0.67000000000000004},"last":0.52300000000000002}]}]},{"osd":6,"up_from":47,"seq":201863462942,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298584,"kb_used_data":1232,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93446568,"statfs":{"total":95995035648,"available":95689285632,"internally_reserved":0,"allocated":1261568,"data_stored":923224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 19:11:36 2024","interfaces":[{"interface":"back","average":{"1min":0.48399999999999999,"5min":0.51400000000000001,"15min":0.51900000000000002},"min":{"1min":0.309,"5min":0.309,"15min":0.309},"max":{"1min":0.67800000000000005,"5min":0.67800000000000005,"15min":0.67800000000000005},"last":0.44},{"interface":"front","average":{"1min":0.47699999999999998,"5min":0.53000000000000003,"15min":0.53900000000000003},"min":{"1min":0.308,"5min":0.308,"15min":0.308},"max":{"1min":0.59999999999999998,"5min":0.67200000000000004,"15min":0.67200000000000004},"last":0.46300000000000002}]},{"osd":1,"last update":"Tue Sep 10 19:11:36 2024","interfaces":[{"interface":"back","average":{"1min":0.501,"5min":0.52300000000000002,"15min":0.52700000000000002},"min":{"1min":0.318,"5min":0.318,"15min":0.318},"max":{"1min":0.63900000000000001,"5min":0.67400000000000004,"15min":0.67400000000000004},"last":0.59999999999999998},{"interface":"front","average":{"1min":0.48499999999999999,"5min":0.52100000000000002,"15min":0.52700000000000002},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.63,"5min":0.65800000000000003,"15min":0.65800000000000003},"last":0.52000000000000002}]},{"osd":2,"last update":"Tue Sep 10 19:11:36 2024","interfaces":[{"interface":"back","average":{"1min":0.5,"5min":0.53600000000000003,"15min":0.54200000000000004},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.65100000000000002,"5min":0.66400000000000003,"15min":0.66400000000000003},"last":0.35499999999999998},{"interface":"front","average":{"1min":0.501,"5min":0.50800000000000001,"15min":0.50900000000000001},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.78600000000000003,"5min":0.78600000000000003,"15min":0.78600000000000003},"last":0.53400000000000003}]},{"osd":3,"last update":"Tue Sep 10 19:11:36 2024","interfaces":[{"interface":"back","average":{"1min":0.47799999999999998,"5min":0.50800000000000001,"15min":0.51300000000000001},"min":{"1min":0.32100000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.64600000000000002,"5min":0.68100000000000005,"15min":0.68100000000000005},"last":0.41299999999999998},{"interface":"front","average":{"1min":0.52900000000000003,"5min":0.55800000000000005,"15min":0.56299999999999994},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.69499999999999995,"5min":0.70199999999999996,"15min":0.70199999999999996},"last":0.47899999999999998}]},{"osd":4,"last update":"Tue Sep 10 19:11:36 2024","interfaces":[{"interface":"back","average":{"1min":0.40899999999999997,"5min":0.35999999999999999,"15min":0.35199999999999998},"min":{"1min":0.25700000000000001,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":0.64700000000000002,"5min":0.64700000000000002,"15min":0.64700000000000002},"last":0.378},{"interface":"front","average":{"1min":0.40899999999999997,"5min":0.38100000000000001,"15min":0.377},"min":{"1min":0.27800000000000002,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.58999999999999997,"5min":0.58999999999999997,"15min":0.58999999999999997},"last":0.55200000000000005}]},{"osd":5,"last update":"Tue Sep 10 19:11:36 2024","interfaces":[{"interface":"back","average":{"1min":0.433,"5min":0.38600000000000001,"15min":0.378},"min":{"1min":0.27100000000000002,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.60699999999999998,"5min":0.60699999999999998,"15min":0.60699999999999998},"last":0.42799999999999999},{"interface":"front","average":{"1min":0.46700000000000003,"5min":0.39200000000000002,"15min":0.38},"min":{"1min":0.318,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.66000000000000003,"5min":0.66000000000000003,"15min":0.66000000000000003},"last":0.625}]},{"osd":7,"last update":"Tue Sep 10 19:10:54 2024","interfaces":[{"interface":"back","average":{"1min":0.44700000000000001,"5min":0.44700000000000001,"15min":0.44700000000000001},"min":{"1min":0.29899999999999999,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.68799999999999994,"5min":0.68799999999999994,"15min":0.68799999999999994},"last":0.57399999999999995},{"interface":"front","average":{"1min":0.44500000000000001,"5min":0.44500000000000001,"15min":0.44500000000000001},"min":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.55000000000000004,"5min":0.55000000000000004,"15min":0.55000000000000004},"last":0.64400000000000002}]}]},{"osd":5,"up_from":39,"seq":167503724579,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298008,"kb_used_data":656,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93447144,"statfs":{"total":95995035648,"available":95689875456,"internally_reserved":0,"allocated":671744,"data_stored":333737,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 19:11:08 2024","interfaces":[{"interface":"back","average":{"1min":0.51900000000000002,"5min":0.48299999999999998,"15min":0.47699999999999998},"min":{"1min":0.38200000000000001,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.67200000000000004,"5min":0.67200000000000004,"15min":0.67200000000000004},"last":0.67600000000000005},{"interface":"front","average":{"1min":0.53600000000000003,"5min":0.48399999999999999,"15min":0.47499999999999998},"min":{"1min":0.38300000000000001,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.75600000000000001,"5min":0.75600000000000001,"15min":0.75600000000000001},"last":0.57899999999999996}]},{"osd":1,"last update":"Tue Sep 10 19:11:08 2024","interfaces":[{"interface":"back","average":{"1min":0.58399999999999996,"5min":0.51900000000000002,"15min":0.50800000000000001},"min":{"1min":0.46800000000000003,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.72799999999999998,"5min":0.72799999999999998,"15min":0.72799999999999998},"last":0.61699999999999999},{"interface":"front","average":{"1min":0.50700000000000001,"5min":0.47899999999999998,"15min":0.47399999999999998},"min":{"1min":0.39900000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.67600000000000005,"5min":0.68799999999999994,"15min":0.68799999999999994},"last":0.51500000000000001}]},{"osd":2,"last update":"Tue Sep 10 19:11:08 2024","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.48599999999999999,"15min":0.47799999999999998},"min":{"1min":0.39900000000000002,"5min":0.309,"15min":0.309},"max":{"1min":0.70699999999999996,"5min":0.70699999999999996,"15min":0.70699999999999996},"last":0.70699999999999996},{"interface":"front","average":{"1min":0.54100000000000004,"5min":0.498,"15min":0.49099999999999999},"min":{"1min":0.39200000000000002,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.70799999999999996,"5min":0.70799999999999996,"15min":0.70799999999999996},"last":0.56100000000000005}]},{"osd":3,"last update":"Tue Sep 10 19:11:08 2024","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.52200000000000002,"15min":0.51400000000000001},"min":{"1min":0.437,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.74099999999999999,"5min":0.74099999999999999,"15min":0.74099999999999999},"last":0.64200000000000002},{"interface":"front","average":{"1min":0.59899999999999998,"5min":0.53800000000000003,"15min":0.52800000000000002},"min":{"1min":0.44700000000000001,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.747,"5min":0.747,"15min":0.747},"last":0.65300000000000002}]},{"osd":4,"last update":"Tue Sep 10 19:11:08 2024","interfaces":[{"interface":"back","average":{"1min":0.44,"5min":0.37,"15min":0.35799999999999998},"min":{"1min":0.27800000000000002,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":0.624,"5min":0.624,"15min":0.624},"last":0.47299999999999998},{"interface":"front","average":{"1min":0.46500000000000002,"5min":0.38100000000000001,"15min":0.36699999999999999},"min":{"1min":0.27800000000000002,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.76800000000000002,"5min":0.76800000000000002,"15min":0.76800000000000002},"last":0.35699999999999998}]},{"osd":6,"last update":"Tue Sep 10 19:11:35 2024","interfaces":[{"interface":"back","average":{"1min":0.48399999999999999,"5min":0.42499999999999999,"15min":0.41499999999999998},"min":{"1min":0.307,"5min":0.223,"15min":0.223},"max":{"1min":0.74099999999999999,"5min":0.74099999999999999,"15min":0.74099999999999999},"last":0.40500000000000003},{"interface":"front","average":{"1min":0.48099999999999998,"5min":0.42099999999999999,"15min":0.41099999999999998},"min":{"1min":0.26800000000000002,"5min":0.25,"15min":0.25},"max":{"1min":0.78000000000000003,"5min":0.78000000000000003,"15min":0.78000000000000003},"last":0.53800000000000003}]},{"osd":7,"last update":"Tue Sep 10 19:10:55 2024","interfaces":[{"interface":"back","average":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.51000000000000001},"min":{"1min":0.29099999999999998,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.72699999999999998,"5min":0.72699999999999998,"15min":0.72699999999999998},"last":0.48899999999999999},{"interface":"front","average":{"1min":0.47899999999999998,"5min":0.47899999999999998,"15min":0.47899999999999998},"min":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.64600000000000002,"5min":0.64600000000000002,"15min":0.64600000000000002},"last":0.42099999999999999}]}]},{"osd":4,"up_from":32,"seq":137438953511,"num_pgs":52,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298264,"kb_used_data":672,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93446888,"statfs":{"total":95995035648,"available":95689613312,"internally_reserved":0,"allocated":688128,"data_stored":332946,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 19:11:49 2024","interfaces":[{"interface":"back","average":{"1min":0.48299999999999998,"5min":0.52000000000000002,"15min":0.53400000000000003},"min":{"1min":0.35299999999999998,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.626,"5min":0.91500000000000004,"15min":0.91500000000000004},"last":0.61099999999999999},{"interface":"front","average":{"1min":0.48399999999999999,"5min":0.51000000000000001,"15min":0.52400000000000002},"min":{"1min":0.30099999999999999,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.71899999999999997,"5min":0.78200000000000003,"15min":0.78200000000000003},"last":0.58299999999999996}]},{"osd":1,"last update":"Tue Sep 10 19:11:49 2024","interfaces":[{"interface":"back","average":{"1min":0.47999999999999998,"5min":0.503,"15min":0.50900000000000001},"min":{"1min":0.29999999999999999,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.72199999999999998,"5min":0.83499999999999996,"15min":0.83499999999999996},"last":0.625},{"interface":"front","average":{"1min":0.51000000000000001,"5min":0.51700000000000002,"15min":0.52200000000000002},"min":{"1min":0.27500000000000002,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.68999999999999995,"5min":0.77100000000000002,"15min":0.77100000000000002},"last":0.51200000000000001}]},{"osd":2,"last update":"Tue Sep 10 19:11:49 2024","interfaces":[{"interface":"back","average":{"1min":0.49199999999999999,"5min":0.52400000000000002,"15min":0.53400000000000003},"min":{"1min":0.30399999999999999,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.65600000000000003,"5min":1.0649999999999999,"15min":1.0649999999999999},"last":0.65600000000000003},{"interface":"front","average":{"1min":0.47599999999999998,"5min":0.51500000000000001,"15min":0.53000000000000003},"min":{"1min":0.33700000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.73699999999999999,"5min":0.86899999999999999,"15min":0.86899999999999999},"last":0.499}]},{"osd":3,"last update":"Tue Sep 10 19:11:49 2024","interfaces":[{"interface":"back","average":{"1min":0.47899999999999998,"5min":0.51100000000000001,"15min":0.52100000000000002},"min":{"1min":0.34999999999999998,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.67000000000000004,"5min":0.76500000000000001,"15min":0.76500000000000001},"last":0.64100000000000001},{"interface":"front","average":{"1min":0.505,"5min":0.53600000000000003,"15min":0.54600000000000004},"min":{"1min":0.373,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.73099999999999998,"5min":1.0249999999999999,"15min":1.0249999999999999},"last":0.59999999999999998}]},{"osd":5,"last update":"Tue Sep 10 19:11:10 2024","interfaces":[{"interface":"back","average":{"1min":0.46400000000000002,"5min":0.39200000000000002,"15min":0.38},"min":{"1min":0.34999999999999998,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.61599999999999999,"5min":0.61599999999999999,"15min":0.61599999999999999},"last":0.45100000000000001},{"interface":"front","average":{"1min":0.47099999999999997,"5min":0.40899999999999997,"15min":0.39900000000000002},"min":{"1min":0.30599999999999999,"5min":0.223,"15min":0.223},"max":{"1min":0.63,"5min":0.89100000000000001,"15min":0.89100000000000001},"last":0.42799999999999999}]},{"osd":6,"last update":"Tue Sep 10 19:11:36 2024","interfaces":[{"interface":"back","average":{"1min":0.46800000000000003,"5min":0.42399999999999999,"15min":0.41599999999999998},"min":{"1min":0.29799999999999999,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.627,"5min":0.627,"15min":0.627},"last":0.55300000000000005},{"interface":"front","average":{"1min":0.46899999999999997,"5min":0.42499999999999999,"15min":0.41699999999999998},"min":{"1min":0.28999999999999998,"5min":0.25,"15min":0.25},"max":{"1min":0.65500000000000003,"5min":0.65500000000000003,"15min":0.65500000000000003},"last":0.53900000000000003}]},{"osd":7,"last update":"Tue Sep 10 19:10:58 2024","interfaces":[{"interface":"back","average":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.72699999999999998,"5min":0.72699999999999998,"15min":0.72699999999999998},"last":0.57199999999999995},{"interface":"front","average":{"1min":0.503,"5min":0.503,"15min":0.503},"min":{"1min":0.29099999999999998,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.69899999999999995,"5min":0.69899999999999995,"15min":0.69899999999999995},"last":0.67100000000000004}]}]},{"osd":3,"up_from":26,"seq":111669149740,"num_pgs":55,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298044,"kb_used_data":692,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93447108,"statfs":{"total":95995035648,"available":95689838592,"internally_reserved":0,"allocated":708608,"data_stored":334520,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 19:11:25 2024","interfaces":[{"interface":"back","average":{"1min":0.315,"5min":0.38700000000000001,"15min":0.40000000000000002},"min":{"1min":0.223,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":0.46800000000000003,"5min":0.61499999999999999,"15min":0.61499999999999999},"last":0.33700000000000002},{"interface":"front","average":{"1min":0.33000000000000002,"5min":0.39700000000000002,"15min":0.41199999999999998},"min":{"1min":0.24399999999999999,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":0.42299999999999999,"5min":0.63600000000000001,"15min":0.63600000000000001},"last":0.26600000000000001}]},{"osd":1,"last update":"Tue Sep 10 19:11:25 2024","interfaces":[{"interface":"back","average":{"1min":0.35499999999999998,"5min":0.42999999999999999,"15min":0.44700000000000001},"min":{"1min":0.253,"5min":0.253,"15min":0.253},"max":{"1min":0.505,"5min":0.67600000000000005,"15min":0.67600000000000005},"last":0.29599999999999999},{"interface":"front","average":{"1min":0.32900000000000001,"5min":0.38700000000000001,"15min":0.39400000000000002},"min":{"1min":0.26300000000000001,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.441,"5min":0.622,"15min":0.622},"last":0.311}]},{"osd":2,"last update":"Tue Sep 10 19:11:25 2024","interfaces":[{"interface":"back","average":{"1min":0.38100000000000001,"5min":0.42499999999999999,"15min":0.434},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.51200000000000001,"5min":0.56599999999999995,"15min":0.56599999999999995},"last":0.39500000000000002},{"interface":"front","average":{"1min":0.38600000000000001,"5min":0.40799999999999997,"15min":0.41399999999999998},"min":{"1min":0.25600000000000001,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":0.66800000000000004,"5min":0.66800000000000004,"15min":0.66800000000000004},"last":0.38400000000000001}]},{"osd":4,"last update":"Tue Sep 10 19:10:50 2024","interfaces":[{"interface":"back","average":{"1min":0.54300000000000004,"5min":0.54400000000000004,"15min":0.54400000000000004},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.74399999999999999,"5min":0.74399999999999999,"15min":0.74399999999999999},"last":0.61499999999999999},{"interface":"front","average":{"1min":0.55400000000000005,"5min":0.54900000000000004,"15min":0.54800000000000004},"min":{"1min":0.433,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":0.81699999999999995,"5min":0.81699999999999995,"15min":0.81699999999999995},"last":0.60299999999999998}]},{"osd":5,"last update":"Tue Sep 10 19:11:07 2024","interfaces":[{"interface":"back","average":{"1min":0.58799999999999997,"5min":0.56999999999999995,"15min":0.56699999999999995},"min":{"1min":0.441,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.72599999999999998,"5min":0.80400000000000005,"15min":0.80400000000000005},"last":0.58699999999999997},{"interface":"front","average":{"1min":0.56200000000000006,"5min":0.56100000000000005,"15min":0.56100000000000005},"min":{"1min":0.40799999999999997,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.89100000000000001,"5min":0.89100000000000001,"15min":0.89100000000000001},"last":0.64300000000000002}]},{"osd":6,"last update":"Tue Sep 10 19:11:34 2024","interfaces":[{"interface":"back","average":{"1min":0.53400000000000003,"5min":0.56999999999999995,"15min":0.57599999999999996},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.67000000000000004,"5min":0.77000000000000002,"15min":0.77000000000000002},"last":0.67900000000000005},{"interface":"front","average":{"1min":0.55100000000000005,"5min":0.55500000000000005,"15min":0.55600000000000005},"min":{"1min":0.41399999999999998,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.71799999999999997,"5min":0.90800000000000003,"15min":0.90800000000000003},"last":0.71799999999999997}]},{"osd":7,"last update":"Tue Sep 10 19:11:00 2024","interfaces":[{"interface":"back","average":{"1min":0.57999999999999996,"5min":0.57999999999999996,"15min":0.57999999999999996},"min":{"1min":0.45400000000000001,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":0.75600000000000001,"5min":0.75600000000000001,"15min":0.75600000000000001},"last":0.59499999999999997},{"interface":"front","average":{"1min":0.58399999999999996,"5min":0.58399999999999996,"15min":0.58399999999999996},"min":{"1min":0.443,"5min":0.443,"15min":0.443},"max":{"1min":0.749,"5min":0.749,"15min":0.749},"last":0.63600000000000001}]}]},{"osd":2,"up_from":18,"seq":77309411377,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298012,"kb_used_data":660,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93447140,"statfs":{"total":95995035648,"available":95689871360,"internally_reserved":0,"allocated":675840,"data_stored":333156,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 19:11:08 2024","interfaces":[{"interface":"back","average":{"1min":0.33800000000000002,"5min":0.34899999999999998,"15min":0.34300000000000003},"min":{"1min":0.20100000000000001,"5min":0.20100000000000001,"15min":0.20100000000000001},"max":{"1min":0.45100000000000001,"5min":0.52300000000000002,"15min":0.52300000000000002},"last":0.29999999999999999},{"interface":"front","average":{"1min":0.33200000000000002,"5min":0.374,"15min":0.36799999999999999},"min":{"1min":0.23100000000000001,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":0.44700000000000001,"5min":0.64200000000000002,"15min":0.64200000000000002},"last":0.36699999999999999}]},{"osd":1,"last update":"Tue Sep 10 19:11:08 2024","interfaces":[{"interface":"back","average":{"1min":0.38200000000000001,"5min":0.38900000000000001,"15min":0.379},"min":{"1min":0.24099999999999999,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.63,"5min":0.69799999999999995,"15min":0.69799999999999995},"last":0.38800000000000001},{"interface":"front","average":{"1min":0.35799999999999998,"5min":0.39700000000000002,"15min":0.39100000000000001},"min":{"1min":0.28499999999999998,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.48299999999999998,"5min":0.63400000000000001,"15min":0.63400000000000001},"last":0.33500000000000002}]},{"osd":3,"last update":"Tue Sep 10 19:11:30 2024","interfaces":[{"interface":"back","average":{"1min":0.35699999999999998,"5min":0.435,"15min":0.44600000000000001},"min":{"1min":0.23699999999999999,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":0.49099999999999999,"5min":0.63900000000000001,"15min":0.63900000000000001},"last":0.34999999999999998},{"interface":"front","average":{"1min":0.36199999999999999,"5min":0.432,"15min":0.442},"min":{"1min":0.27100000000000002,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.438,"5min":0.65800000000000003,"15min":0.65800000000000003},"last":0.40400000000000003}]},{"osd":4,"last update":"Tue Sep 10 19:10:50 2024","interfaces":[{"interface":"back","average":{"1min":0.59099999999999997,"5min":0.58299999999999996,"15min":0.58099999999999996},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.81999999999999995,"5min":0.81999999999999995,"15min":0.81999999999999995},"last":0.56699999999999995},{"interface":"front","average":{"1min":0.55600000000000005,"5min":0.56799999999999995,"15min":0.56999999999999995},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.749,"5min":0.80600000000000005,"15min":0.80600000000000005},"last":0.55900000000000005}]},{"osd":5,"last update":"Tue Sep 10 19:11:08 2024","interfaces":[{"interface":"back","average":{"1min":0.54800000000000004,"5min":0.52900000000000003,"15min":0.52600000000000002},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.73599999999999999,"5min":0.73899999999999999,"15min":0.73899999999999999},"last":0.53700000000000003},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.57899999999999996,"15min":0.57899999999999996},"min":{"1min":0.443,"5min":0.443,"15min":0.443},"max":{"1min":0.70699999999999996,"5min":0.82699999999999996,"15min":0.82699999999999996},"last":0.52700000000000002}]},{"osd":6,"last update":"Tue Sep 10 19:11:34 2024","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":0.55200000000000005,"15min":0.55600000000000005},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":0.67300000000000004,"5min":0.73199999999999998,"15min":0.73199999999999998},"last":0.58799999999999997},{"interface":"front","average":{"1min":0.54800000000000004,"5min":0.55000000000000004,"15min":0.55000000000000004},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.72299999999999998,"5min":0.748,"15min":0.748},"last":0.59599999999999997}]},{"osd":7,"last update":"Tue Sep 10 19:11:00 2024","interfaces":[{"interface":"back","average":{"1min":0.60599999999999998,"5min":0.60599999999999998,"15min":0.60599999999999998},"min":{"1min":0.44700000000000001,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":0.79700000000000004,"5min":0.79700000000000004,"15min":0.79700000000000004},"last":0.45900000000000002},{"interface":"front","average":{"1min":0.54900000000000004,"5min":0.54900000000000004,"15min":0.54900000000000004},"min":{"1min":0.40300000000000002,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":0.72599999999999998,"5min":0.72599999999999998,"15min":0.72599999999999998},"last":0.51600000000000001}]}]},{"osd":1,"up_from":13,"seq":55834574902,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298048,"kb_used_data":692,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93447104,"statfs":{"total":95995035648,"available":95689834496,"internally_reserved":0,"allocated":708608,"data_stored":333874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 19:11:42 2024","interfaces":[{"interface":"back","average":{"1min":0.38500000000000001,"5min":0.378,"15min":0.33800000000000002},"min":{"1min":0.21099999999999999,"5min":0.188,"15min":0.188},"max":{"1min":1.0629999999999999,"5min":2.7730000000000001,"15min":2.7730000000000001},"last":0.32200000000000001},{"interface":"front","average":{"1min":0.36699999999999999,"5min":0.38,"15min":0.34699999999999998},"min":{"1min":0.20699999999999999,"5min":0.20599999999999999,"15min":0.20599999999999999},"max":{"1min":0.53700000000000003,"5min":2.8380000000000001,"15min":2.8380000000000001},"last":0.28799999999999998}]},{"osd":2,"last update":"Tue Sep 10 19:11:01 2024","interfaces":[{"interface":"back","average":{"1min":0.42399999999999999,"5min":0.38500000000000001,"15min":0.373},"min":{"1min":0.251,"5min":0.251,"15min":0.251},"max":{"1min":1.1779999999999999,"5min":1.1779999999999999,"15min":1.1779999999999999},"last":0.34999999999999998},{"interface":"front","average":{"1min":0.42899999999999999,"5min":0.38,"15min":0.36799999999999999},"min":{"1min":0.254,"5min":0.254,"15min":0.254},"max":{"1min":1.103,"5min":1.103,"15min":1.103},"last":0.309}]},{"osd":3,"last update":"Tue Sep 10 19:11:29 2024","interfaces":[{"interface":"back","average":{"1min":0.44700000000000001,"5min":0.45600000000000002,"15min":0.44900000000000001},"min":{"1min":0.27500000000000002,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":1.093,"5min":2.0670000000000002,"15min":2.0670000000000002},"last":0.36899999999999999},{"interface":"front","average":{"1min":0.40100000000000002,"5min":0.42299999999999999,"15min":0.42099999999999999},"min":{"1min":0.30299999999999999,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":0.54600000000000004,"5min":1.956,"15min":1.956},"last":0.33600000000000002}]},{"osd":4,"last update":"Tue Sep 10 19:10:49 2024","interfaces":[{"interface":"back","average":{"1min":0.61299999999999999,"5min":0.52200000000000002,"15min":0.50700000000000001},"min":{"1min":0.28499999999999998,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":2.665,"5min":2.665,"15min":2.665},"last":0.71799999999999997},{"interface":"front","average":{"1min":0.65200000000000002,"5min":0.56200000000000006,"15min":0.54700000000000004},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":2.7330000000000001,"5min":2.7330000000000001,"15min":2.7330000000000001},"last":0.76400000000000001}]},{"osd":5,"last update":"Tue Sep 10 19:11:11 2024","interfaces":[{"interface":"back","average":{"1min":0.58299999999999996,"5min":0.53800000000000003,"15min":0.53000000000000003},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":1.851,"5min":1.851,"15min":1.851},"last":0.81899999999999995},{"interface":"front","average":{"1min":0.63500000000000001,"5min":0.55100000000000005,"15min":0.53700000000000003},"min":{"1min":0.31900000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":2.706,"5min":2.706,"15min":2.706},"last":0.60299999999999998}]},{"osd":6,"last update":"Tue Sep 10 19:11:35 2024","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.60199999999999998,"15min":0.61299999999999999},"min":{"1min":0.41599999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.68500000000000005,"5min":1.9950000000000001,"15min":1.9950000000000001},"last":0.73399999999999999},{"interface":"front","average":{"1min":0.54200000000000004,"5min":0.58599999999999997,"15min":0.59399999999999997},"min":{"1min":0.39300000000000002,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.90600000000000003,"5min":1.9399999999999999,"15min":1.9399999999999999},"last":0.67600000000000005}]},{"osd":7,"last update":"Tue Sep 10 19:10:55 2024","interfaces":[{"interface":"back","average":{"1min":0.66600000000000004,"5min":0.66600000000000004,"15min":0.66600000000000004},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":3.4620000000000002,"5min":3.4620000000000002,"15min":3.4620000000000002},"last":0.79000000000000004},{"interface":"front","average":{"1min":0.621,"5min":0.621,"15min":0.621},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":1.514,"5min":1.514,"15min":1.514},"last":0.58499999999999996}]}]},{"osd":0,"up_from":8,"seq":34359738427,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298604,"kb_used_data":1248,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93446548,"statfs":{"total":95995035648,"available":95689265152,"internally_reserved":0,"allocated":1277952,"data_stored":924021,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Sep 10 19:11:42 2024","interfaces":[{"interface":"back","average":{"1min":0.318,"5min":0.34799999999999998,"15min":0.33100000000000002},"min":{"1min":0.23999999999999999,"5min":0.221,"15min":0.221},"max":{"1min":0.40799999999999997,"5min":0.63300000000000001,"15min":0.63300000000000001},"last":0.34999999999999998},{"interface":"front","average":{"1min":0.33600000000000002,"5min":0.35199999999999998,"15min":0.33500000000000002},"min":{"1min":0.23799999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":0.46899999999999997,"5min":0.61899999999999999,"15min":0.61899999999999999},"last":0.437}]},{"osd":2,"last update":"Tue Sep 10 19:11:08 2024","interfaces":[{"interface":"back","average":{"1min":0.375,"5min":0.38700000000000001,"15min":0.38300000000000001},"min":{"1min":0.27600000000000002,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":0.50700000000000001,"5min":0.57399999999999995,"15min":0.57399999999999995},"last":0.46500000000000002},{"interface":"front","average":{"1min":0.36499999999999999,"5min":0.375,"15min":0.373},"min":{"1min":0.22900000000000001,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":0.53000000000000003,"5min":0.54100000000000004,"15min":0.54100000000000004},"last":0.36399999999999999}]},{"osd":3,"last update":"Tue Sep 10 19:11:30 2024","interfaces":[{"interface":"back","average":{"1min":0.35999999999999999,"5min":0.41499999999999998,"15min":0.42099999999999999},"min":{"1min":0.29499999999999998,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.45500000000000002,"5min":0.65000000000000002,"15min":0.65000000000000002},"last":0.39500000000000002},{"interface":"front","average":{"1min":0.34499999999999997,"5min":0.39900000000000002,"15min":0.40500000000000003},"min":{"1min":0.27800000000000002,"5min":0.246,"15min":0.246},"max":{"1min":0.45900000000000002,"5min":0.58799999999999997,"15min":0.58799999999999997},"last":0.315}]},{"osd":4,"last update":"Tue Sep 10 19:11:46 2024","interfaces":[{"interface":"back","average":{"1min":0.496,"5min":0.54800000000000004,"15min":0.56000000000000005},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.81499999999999995,"5min":0.81499999999999995,"15min":0.81499999999999995},"last":0.51900000000000002},{"interface":"front","average":{"1min":0.51200000000000001,"5min":0.55300000000000005,"15min":0.55900000000000005},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.71599999999999997,"5min":0.77000000000000002,"15min":0.77000000000000002},"last":0.59299999999999997}]},{"osd":5,"last update":"Tue Sep 10 19:11:08 2024","interfaces":[{"interface":"back","average":{"1min":0.58399999999999996,"5min":0.58399999999999996,"15min":0.58399999999999996},"min":{"1min":0.44800000000000001,"5min":0.378,"15min":0.378},"max":{"1min":0.82599999999999996,"5min":0.82599999999999996,"15min":0.82599999999999996},"last":0.48799999999999999},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.57399999999999995,"15min":0.57499999999999996},"min":{"1min":0.42699999999999999,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.77900000000000003,"5min":0.82899999999999996,"15min":0.82899999999999996},"last":0.54800000000000004}]},{"osd":6,"last update":"Tue Sep 10 19:11:33 2024","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":0.55800000000000005,"15min":0.56399999999999995},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.78900000000000003,"5min":0.78900000000000003,"15min":0.78900000000000003},"last":0.57799999999999996},{"interface":"front","average":{"1min":0.54800000000000004,"5min":0.57899999999999996,"15min":0.58399999999999996},"min":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":0.70499999999999996,"5min":0.76100000000000001,"15min":0.76100000000000001},"last":0.53400000000000003}]},{"osd":7,"last update":"Tue Sep 10 19:10:58 2024","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.57799999999999996,"15min":0.57799999999999996},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":0.76900000000000002,"5min":0.76900000000000002,"15min":0.76900000000000002},"last":0.56499999999999995},{"interface":"front","average":{"1min":0.56200000000000006,"5min":0.56200000000000006,"15min":0.56200000000000006},"min":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.72599999999999998,"5min":0.72599999999999998,"15min":0.72599999999999998},"last":0.503}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":966,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1236,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1236,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":362,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":2118,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1018,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T19:11:53.494 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-09-10T19:11:53.495 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph osd dump 2024-09-10T19:11:53.775 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:11:53.777 INFO:journalctl@ceph.alertmanager.a.smithi183.stdout:Sep 10 19:11:53 smithi183 podman[80503]: 2024-09-10 19:11:53.603416081 +0000 UTC m=+1.325264285 container init 1693fe6a126991cd9e9bae41ec0e86c2aed52a7bf9b9695f6083f915cbdc7440 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-09-10T19:11:53.777 INFO:journalctl@ceph.alertmanager.a.smithi183.stdout:Sep 10 19:11:53 smithi183 podman[80503]: 2024-09-10 19:11:53.609423124 +0000 UTC m=+1.331271328 container start 1693fe6a126991cd9e9bae41ec0e86c2aed52a7bf9b9695f6083f915cbdc7440 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-09-10T19:11:53.777 INFO:journalctl@ceph.alertmanager.a.smithi183.stdout:Sep 10 19:11:53 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-alertmanager-a[80531]: ts=2024-09-10T19:11:53.621Z caller=main.go:240 level=info msg="Starting Alertmanager" version="(version=0.25.0, branch=HEAD, revision=258fab7cdd551f2cf251ed0348f0ad7289aee789)" 2024-09-10T19:11:53.777 INFO:journalctl@ceph.alertmanager.a.smithi183.stdout:Sep 10 19:11:53 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-alertmanager-a[80531]: ts=2024-09-10T19:11:53.621Z caller=main.go:241 level=info build_context="(go=go1.19.4, user=root@abe866dd5717, date=20221222-14:51:36)" 2024-09-10T19:11:53.777 INFO:journalctl@ceph.alertmanager.a.smithi183.stdout:Sep 10 19:11:53 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-alertmanager-a[80531]: ts=2024-09-10T19:11:53.623Z caller=cluster.go:185 level=info component=cluster msg="setting advertise address explicitly" addr=172.21.15.183 port=9094 2024-09-10T19:11:53.777 INFO:journalctl@ceph.alertmanager.a.smithi183.stdout:Sep 10 19:11:53 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-alertmanager-a[80531]: ts=2024-09-10T19:11:53.624Z caller=cluster.go:681 level=info component=cluster msg="Waiting for gossip to settle..." interval=2s 2024-09-10T19:11:53.777 INFO:journalctl@ceph.alertmanager.a.smithi183.stdout:Sep 10 19:11:53 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-alertmanager-a[80531]: ts=2024-09-10T19:11:53.646Z caller=coordinator.go:113 level=info component=configuration msg="Loading configuration file" file=/etc/alertmanager/alertmanager.yml 2024-09-10T19:11:53.778 INFO:journalctl@ceph.alertmanager.a.smithi183.stdout:Sep 10 19:11:53 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-alertmanager-a[80531]: ts=2024-09-10T19:11:53.647Z caller=coordinator.go:126 level=info component=configuration msg="Completed loading of configuration file" file=/etc/alertmanager/alertmanager.yml 2024-09-10T19:11:53.778 INFO:journalctl@ceph.alertmanager.a.smithi183.stdout:Sep 10 19:11:53 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-alertmanager-a[80531]: ts=2024-09-10T19:11:53.650Z caller=tls_config.go:232 level=info msg="Listening on" address=[::]:9093 2024-09-10T19:11:53.778 INFO:journalctl@ceph.alertmanager.a.smithi183.stdout:Sep 10 19:11:53 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-alertmanager-a[80531]: ts=2024-09-10T19:11:53.650Z caller=tls_config.go:235 level=info msg="TLS is disabled." http2=false address=[::]:9093 2024-09-10T19:11:53.778 INFO:journalctl@ceph.alertmanager.a.smithi183.stdout:Sep 10 19:11:53 smithi183 bash[80503]: 1693fe6a126991cd9e9bae41ec0e86c2aed52a7bf9b9695f6083f915cbdc7440 2024-09-10T19:11:53.778 INFO:journalctl@ceph.alertmanager.a.smithi183.stdout:Sep 10 19:11:53 smithi183 systemd[1]: Started Ceph alertmanager.a for 320f6df8-6fa7-11ef-bcea-c7b262605968. 2024-09-10T19:11:54.044 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:11:53 smithi183 ceph-mon[41562]: purged_snaps scrub starts 2024-09-10T19:11:54.044 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:11:53 smithi183 ceph-mon[41562]: purged_snaps scrub ok 2024-09-10T19:11:54.044 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:11:53 smithi183 ceph-mon[41562]: from='client.14730 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-10T19:11:54.044 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:11:53 smithi183 ceph-mon[41562]: 4.1d scrub starts 2024-09-10T19:11:54.044 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:11:53 smithi183 ceph-mon[41562]: 4.1d scrub ok 2024-09-10T19:11:54.045 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:11:53 smithi183 ceph-mon[41562]: osdmap e83: 8 total, 8 up, 8 in 2024-09-10T19:11:54.045 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:11:53 smithi183 ceph-mon[41562]: 2.8 deep-scrub starts 2024-09-10T19:11:54.045 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:11:53 smithi183 ceph-mon[41562]: 2.8 deep-scrub ok 2024-09-10T19:11:54.045 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:11:53 smithi183 ceph-mon[41562]: from='mgr.14556 ' entity='mgr.y' 2024-09-10T19:11:54.045 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:11:53 smithi183 ceph-mon[41562]: from='mgr.14556 ' entity='mgr.y' 2024-09-10T19:11:54.045 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:11:53 smithi183 ceph-mon[29736]: purged_snaps scrub starts 2024-09-10T19:11:54.045 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:11:53 smithi183 ceph-mon[29736]: purged_snaps scrub ok 2024-09-10T19:11:54.045 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:11:53 smithi183 ceph-mon[29736]: from='client.14730 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-10T19:11:54.045 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:11:53 smithi183 ceph-mon[29736]: 4.1d scrub starts 2024-09-10T19:11:54.045 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:11:53 smithi183 ceph-mon[29736]: 4.1d scrub ok 2024-09-10T19:11:54.045 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:11:53 smithi183 ceph-mon[29736]: osdmap e83: 8 total, 8 up, 8 in 2024-09-10T19:11:54.046 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:11:53 smithi183 ceph-mon[29736]: 2.8 deep-scrub starts 2024-09-10T19:11:54.046 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:11:53 smithi183 ceph-mon[29736]: 2.8 deep-scrub ok 2024-09-10T19:11:54.046 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:11:53 smithi183 ceph-mon[29736]: from='mgr.14556 ' entity='mgr.y' 2024-09-10T19:11:54.046 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:11:53 smithi183 ceph-mon[29736]: from='mgr.14556 ' entity='mgr.y' 2024-09-10T19:11:54.119 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:11:53 smithi191 ceph-mon[32884]: purged_snaps scrub starts 2024-09-10T19:11:54.119 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:11:53 smithi191 ceph-mon[32884]: purged_snaps scrub ok 2024-09-10T19:11:54.119 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:11:53 smithi191 ceph-mon[32884]: from='client.14730 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-10T19:11:54.119 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:11:53 smithi191 ceph-mon[32884]: 4.1d scrub starts 2024-09-10T19:11:54.119 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:11:53 smithi191 ceph-mon[32884]: 4.1d scrub ok 2024-09-10T19:11:54.119 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:11:53 smithi191 ceph-mon[32884]: osdmap e83: 8 total, 8 up, 8 in 2024-09-10T19:11:54.120 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:11:53 smithi191 ceph-mon[32884]: 2.8 deep-scrub starts 2024-09-10T19:11:54.120 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:11:53 smithi191 ceph-mon[32884]: 2.8 deep-scrub ok 2024-09-10T19:11:54.120 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:11:53 smithi191 ceph-mon[32884]: from='mgr.14556 ' entity='mgr.y' 2024-09-10T19:11:54.120 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:11:53 smithi191 ceph-mon[32884]: from='mgr.14556 ' entity='mgr.y' 2024-09-10T19:11:54.931 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:11:54 smithi183 ceph-mon[41562]: 5.1c scrub starts 2024-09-10T19:11:54.931 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:11:54 smithi183 ceph-mon[41562]: 5.1c scrub ok 2024-09-10T19:11:54.931 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:11:54 smithi183 ceph-mon[41562]: purged_snaps scrub starts 2024-09-10T19:11:54.931 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:11:54 smithi183 ceph-mon[41562]: purged_snaps scrub ok 2024-09-10T19:11:54.932 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:11:54 smithi183 ceph-mon[41562]: pgmap v48: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 3 op/s; 24 B/s, 1 objects/s recovering 2024-09-10T19:11:54.932 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:11:54 smithi183 ceph-mon[41562]: Reconfiguring prometheus.a (dependencies changed)... 2024-09-10T19:11:54.932 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:11:54 smithi183 ceph-mon[41562]: Reconfiguring daemon prometheus.a on smithi191 2024-09-10T19:11:54.932 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:11:54 smithi183 ceph-mon[41562]: osdmap e84: 8 total, 8 up, 8 in 2024-09-10T19:11:54.932 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:11:54 smithi183 ceph-mon[41562]: 5.3 deep-scrub starts 2024-09-10T19:11:54.932 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:11:54 smithi183 ceph-mon[41562]: 5.3 deep-scrub ok 2024-09-10T19:11:54.932 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:11:54 smithi183 ceph-mon[29736]: 5.1c scrub starts 2024-09-10T19:11:54.932 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:11:54 smithi183 ceph-mon[29736]: 5.1c scrub ok 2024-09-10T19:11:54.932 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:11:54 smithi183 ceph-mon[29736]: purged_snaps scrub starts 2024-09-10T19:11:54.932 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:11:54 smithi183 ceph-mon[29736]: purged_snaps scrub ok 2024-09-10T19:11:54.932 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:11:54 smithi183 ceph-mon[29736]: pgmap v48: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 3 op/s; 24 B/s, 1 objects/s recovering 2024-09-10T19:11:54.933 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:11:54 smithi183 ceph-mon[29736]: Reconfiguring prometheus.a (dependencies changed)... 2024-09-10T19:11:54.933 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:11:54 smithi183 ceph-mon[29736]: Reconfiguring daemon prometheus.a on smithi191 2024-09-10T19:11:54.933 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:11:54 smithi183 ceph-mon[29736]: osdmap e84: 8 total, 8 up, 8 in 2024-09-10T19:11:54.933 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:11:54 smithi183 ceph-mon[29736]: 5.3 deep-scrub starts 2024-09-10T19:11:54.933 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:11:54 smithi183 ceph-mon[29736]: 5.3 deep-scrub ok 2024-09-10T19:11:55.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:11:54 smithi191 ceph-mon[32884]: 5.1c scrub starts 2024-09-10T19:11:55.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:11:54 smithi191 ceph-mon[32884]: 5.1c scrub ok 2024-09-10T19:11:55.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:11:54 smithi191 ceph-mon[32884]: purged_snaps scrub starts 2024-09-10T19:11:55.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:11:54 smithi191 ceph-mon[32884]: purged_snaps scrub ok 2024-09-10T19:11:55.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:11:54 smithi191 ceph-mon[32884]: pgmap v48: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 3 op/s; 24 B/s, 1 objects/s recovering 2024-09-10T19:11:55.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:11:54 smithi191 ceph-mon[32884]: Reconfiguring prometheus.a (dependencies changed)... 2024-09-10T19:11:55.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:11:54 smithi191 ceph-mon[32884]: Reconfiguring daemon prometheus.a on smithi191 2024-09-10T19:11:55.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:11:54 smithi191 ceph-mon[32884]: osdmap e84: 8 total, 8 up, 8 in 2024-09-10T19:11:55.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:11:54 smithi191 ceph-mon[32884]: 5.3 deep-scrub starts 2024-09-10T19:11:55.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:11:54 smithi191 ceph-mon[32884]: 5.3 deep-scrub ok 2024-09-10T19:11:55.616 INFO:teuthology.orchestra.run.smithi183.stdout:epoch 84 2024-09-10T19:11:55.616 INFO:teuthology.orchestra.run.smithi183.stdout:fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 2024-09-10T19:11:55.616 INFO:teuthology.orchestra.run.smithi183.stdout:created 2024-09-10T19:05:03.688531+0000 2024-09-10T19:11:55.616 INFO:teuthology.orchestra.run.smithi183.stdout:modified 2024-09-10T19:11:53.771768+0000 2024-09-10T19:11:55.616 INFO:teuthology.orchestra.run.smithi183.stdout:flags sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit 2024-09-10T19:11:55.616 INFO:teuthology.orchestra.run.smithi183.stdout:crush_version 18 2024-09-10T19:11:55.616 INFO:teuthology.orchestra.run.smithi183.stdout:full_ratio 0.95 2024-09-10T19:11:55.616 INFO:teuthology.orchestra.run.smithi183.stdout:backfillfull_ratio 0.9 2024-09-10T19:11:55.616 INFO:teuthology.orchestra.run.smithi183.stdout:nearfull_ratio 0.85 2024-09-10T19:11:55.616 INFO:teuthology.orchestra.run.smithi183.stdout:require_min_compat_client luminous 2024-09-10T19:11:55.617 INFO:teuthology.orchestra.run.smithi183.stdout:min_compat_client luminous 2024-09-10T19:11:55.617 INFO:teuthology.orchestra.run.smithi183.stdout:require_osd_release quincy 2024-09-10T19:11:55.617 INFO:teuthology.orchestra.run.smithi183.stdout:stretch_mode_enabled false 2024-09-10T19:11:55.617 INFO:teuthology.orchestra.run.smithi183.stdout:pool 1 '.mgr' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode off last_change 22 flags hashpspool stripe_width 0 pg_num_max 32 pg_num_min 1 application mgr 2024-09-10T19:11:55.617 INFO:teuthology.orchestra.run.smithi183.stdout:pool 2 '.rgw.root' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 60 flags hashpspool stripe_width 0 application rgw 2024-09-10T19:11:55.617 INFO:teuthology.orchestra.run.smithi183.stdout:pool 3 'datapool' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 3 pgp_num 3 autoscale_mode off last_change 63 flags hashpspool,selfmanaged_snaps stripe_width 0 application rbd 2024-09-10T19:11:55.617 INFO:teuthology.orchestra.run.smithi183.stdout:pool 4 'default.rgw.log' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 62 flags hashpspool stripe_width 0 application rgw 2024-09-10T19:11:55.617 INFO:teuthology.orchestra.run.smithi183.stdout:pool 5 'default.rgw.control' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 64 flags hashpspool stripe_width 0 application rgw 2024-09-10T19:11:55.617 INFO:teuthology.orchestra.run.smithi183.stdout:pool 6 'default.rgw.meta' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 66 flags hashpspool stripe_width 0 pg_autoscale_bias 4 application rgw 2024-09-10T19:11:55.617 INFO:teuthology.orchestra.run.smithi183.stdout:max_osd 8 2024-09-10T19:11:55.617 INFO:teuthology.orchestra.run.smithi183.stdout:osd.0 up in weight 1 up_from 8 up_thru 81 down_at 0 last_clean_interval [0,0) [v2:172.21.15.183:6802/2813074952,v1:172.21.15.183:6803/2813074952] [v2:172.21.15.183:6804/2813074952,v1:172.21.15.183:6805/2813074952] exists,up 9d323010-394a-49aa-ba9b-244d87faf36e 2024-09-10T19:11:55.617 INFO:teuthology.orchestra.run.smithi183.stdout:osd.1 up in weight 1 up_from 13 up_thru 64 down_at 0 last_clean_interval [0,0) [v2:172.21.15.183:6810/3880626114,v1:172.21.15.183:6811/3880626114] [v2:172.21.15.183:6812/3880626114,v1:172.21.15.183:6813/3880626114] exists,up 71a6c5dc-5109-4284-8d20-187d8e7afc9e 2024-09-10T19:11:55.617 INFO:teuthology.orchestra.run.smithi183.stdout:osd.2 up in weight 1 up_from 18 up_thru 81 down_at 0 last_clean_interval [0,0) [v2:172.21.15.183:6818/3730013786,v1:172.21.15.183:6819/3730013786] [v2:172.21.15.183:6820/3730013786,v1:172.21.15.183:6821/3730013786] exists,up fc310d5b-b89b-4655-9989-7e10ba356509 2024-09-10T19:11:55.618 INFO:teuthology.orchestra.run.smithi183.stdout:osd.3 up in weight 1 up_from 26 up_thru 64 down_at 0 last_clean_interval [0,0) [v2:172.21.15.183:6826/3522836984,v1:172.21.15.183:6827/3522836984] [v2:172.21.15.183:6828/3522836984,v1:172.21.15.183:6829/3522836984] exists,up 7fd07a95-ad22-43f5-a968-539552283e89 2024-09-10T19:11:55.618 INFO:teuthology.orchestra.run.smithi183.stdout:osd.4 up in weight 1 up_from 32 up_thru 81 down_at 0 last_clean_interval [0,0) [v2:172.21.15.191:6800/2674828539,v1:172.21.15.191:6801/2674828539] [v2:172.21.15.191:6802/2674828539,v1:172.21.15.191:6803/2674828539] exists,up 631083fb-e503-44da-9b6b-154e88539ee7 2024-09-10T19:11:55.618 INFO:teuthology.orchestra.run.smithi183.stdout:osd.5 up in weight 1 up_from 39 up_thru 64 down_at 0 last_clean_interval [0,0) [v2:172.21.15.191:6808/681733378,v1:172.21.15.191:6809/681733378] [v2:172.21.15.191:6810/681733378,v1:172.21.15.191:6811/681733378] exists,up 6b963695-3eca-4e6e-9207-05b8010ebff9 2024-09-10T19:11:55.618 INFO:teuthology.orchestra.run.smithi183.stdout:osd.6 up in weight 1 up_from 47 up_thru 81 down_at 0 last_clean_interval [0,0) [v2:172.21.15.191:6816/2825813477,v1:172.21.15.191:6817/2825813477] [v2:172.21.15.191:6818/2825813477,v1:172.21.15.191:6819/2825813477] exists,up 2dc9754f-6aef-4e19-8605-2f1591921811 2024-09-10T19:11:55.618 INFO:teuthology.orchestra.run.smithi183.stdout:osd.7 up in weight 1 up_from 54 up_thru 64 down_at 0 last_clean_interval [0,0) [v2:172.21.15.191:6824/1422610021,v1:172.21.15.191:6825/1422610021] [v2:172.21.15.191:6826/1422610021,v1:172.21.15.191:6827/1422610021] exists,up 1a9a299e-d5ba-4b9a-85d5-7f6a5bc754e1 2024-09-10T19:11:55.618 INFO:teuthology.orchestra.run.smithi183.stdout:pg_upmap_items 2.1f [7,2] 2024-09-10T19:11:55.618 INFO:teuthology.orchestra.run.smithi183.stdout:pg_upmap_items 4.2 [1,2] 2024-09-10T19:11:55.618 INFO:teuthology.orchestra.run.smithi183.stdout:pg_upmap_items 4.1b [1,2] 2024-09-10T19:11:55.618 INFO:teuthology.orchestra.run.smithi183.stdout:pg_upmap_items 4.1f [1,2] 2024-09-10T19:11:55.618 INFO:teuthology.orchestra.run.smithi183.stdout:blocklist 172.21.15.183:6801/3311783362 expires 2024-09-11T19:10:43.576943+0000 2024-09-10T19:11:55.619 INFO:teuthology.orchestra.run.smithi183.stdout:blocklist 172.21.15.183:0/4088553922 expires 2024-09-11T19:10:43.576943+0000 2024-09-10T19:11:55.619 INFO:teuthology.orchestra.run.smithi183.stdout:blocklist 172.21.15.183:0/3528769414 expires 2024-09-11T19:05:48.153879+0000 2024-09-10T19:11:55.619 INFO:teuthology.orchestra.run.smithi183.stdout:blocklist 172.21.15.183:0/1766036658 expires 2024-09-11T19:05:48.153879+0000 2024-09-10T19:11:55.619 INFO:teuthology.orchestra.run.smithi183.stdout:blocklist 172.21.15.183:0/3044170088 expires 2024-09-11T19:05:25.702354+0000 2024-09-10T19:11:55.619 INFO:teuthology.orchestra.run.smithi183.stdout:blocklist 172.21.15.183:0/3037333778 expires 2024-09-11T19:05:25.702354+0000 2024-09-10T19:11:55.619 INFO:teuthology.orchestra.run.smithi183.stdout:blocklist 172.21.15.183:6801/3747320570 expires 2024-09-11T19:05:25.702354+0000 2024-09-10T19:11:55.619 INFO:teuthology.orchestra.run.smithi183.stdout:blocklist 172.21.15.183:0/3982386906 expires 2024-09-11T19:10:43.576943+0000 2024-09-10T19:11:55.619 INFO:teuthology.orchestra.run.smithi183.stdout:blocklist 172.21.15.183:0/2436014455 expires 2024-09-11T19:05:25.702354+0000 2024-09-10T19:11:55.619 INFO:teuthology.orchestra.run.smithi183.stdout:blocklist 172.21.15.183:6801/659745765 expires 2024-09-11T19:05:48.153879+0000 2024-09-10T19:11:55.619 INFO:teuthology.orchestra.run.smithi183.stdout:blocklist 172.21.15.183:6800/3747320570 expires 2024-09-11T19:05:25.702354+0000 2024-09-10T19:11:55.619 INFO:teuthology.orchestra.run.smithi183.stdout:blocklist 172.21.15.183:0/1652576282 expires 2024-09-11T19:10:43.576943+0000 2024-09-10T19:11:55.620 INFO:teuthology.orchestra.run.smithi183.stdout:blocklist 172.21.15.183:6800/3311783362 expires 2024-09-11T19:10:43.576943+0000 2024-09-10T19:11:55.620 INFO:teuthology.orchestra.run.smithi183.stdout:blocklist 172.21.15.183:0/1012899549 expires 2024-09-11T19:05:48.153879+0000 2024-09-10T19:11:55.620 INFO:teuthology.orchestra.run.smithi183.stdout:blocklist 172.21.15.183:6800/659745765 expires 2024-09-11T19:05:48.153879+0000 2024-09-10T19:11:55.620 INFO:teuthology.orchestra.run.smithi183.stdout:blocklist 172.21.15.183:0/412044400 expires 2024-09-11T19:10:43.576943+0000 2024-09-10T19:11:55.620 INFO:teuthology.orchestra.run.smithi183.stdout:blocklist 172.21.15.183:0/3636171401 expires 2024-09-11T19:10:43.576943+0000 2024-09-10T19:11:55.924 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:11:55 smithi183 ceph-mon[29736]: 3.1 scrub starts 2024-09-10T19:11:55.924 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:11:55 smithi183 ceph-mon[29736]: 3.1 scrub ok 2024-09-10T19:11:55.924 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:11:55 smithi183 ceph-mon[29736]: from='client.? 172.21.15.183:0/788708675' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-09-10T19:11:55.925 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:11:55 smithi183 ceph-mon[41562]: 3.1 scrub starts 2024-09-10T19:11:55.925 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:11:55 smithi183 ceph-mon[41562]: 3.1 scrub ok 2024-09-10T19:11:55.925 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:11:55 smithi183 ceph-mon[41562]: from='client.? 172.21.15.183:0/788708675' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-09-10T19:11:55.925 INFO:journalctl@ceph.alertmanager.a.smithi183.stdout:Sep 10 19:11:55 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-alertmanager-a[80531]: ts=2024-09-10T19:11:55.624Z caller=cluster.go:706 level=info component=cluster msg="gossip not settled" polls=0 before=0 now=1 elapsed=2.000147301s 2024-09-10T19:11:56.085 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:11:55 smithi191 ceph-mon[32884]: 3.1 scrub starts 2024-09-10T19:11:56.085 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:11:55 smithi191 ceph-mon[32884]: 3.1 scrub ok 2024-09-10T19:11:56.085 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:11:55 smithi191 ceph-mon[32884]: from='client.? 172.21.15.183:0/788708675' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-09-10T19:11:56.308 INFO:tasks.cephadm.ceph_manager.ceph:['osd.0 up in weight 1 up_from 8 up_thru 81 down_at 0 last_clean_interval [0,0) [v2:172.21.15.183:6802/2813074952,v1:172.21.15.183:6803/2813074952] [v2:172.21.15.183:6804/2813074952,v1:172.21.15.183:6805/2813074952] exists,up 9d323010-394a-49aa-ba9b-244d87faf36e', 'osd.1 up in weight 1 up_from 13 up_thru 64 down_at 0 last_clean_interval [0,0) [v2:172.21.15.183:6810/3880626114,v1:172.21.15.183:6811/3880626114] [v2:172.21.15.183:6812/3880626114,v1:172.21.15.183:6813/3880626114] exists,up 71a6c5dc-5109-4284-8d20-187d8e7afc9e', 'osd.2 up in weight 1 up_from 18 up_thru 81 down_at 0 last_clean_interval [0,0) [v2:172.21.15.183:6818/3730013786,v1:172.21.15.183:6819/3730013786] [v2:172.21.15.183:6820/3730013786,v1:172.21.15.183:6821/3730013786] exists,up fc310d5b-b89b-4655-9989-7e10ba356509', 'osd.3 up in weight 1 up_from 26 up_thru 64 down_at 0 last_clean_interval [0,0) [v2:172.21.15.183:6826/3522836984,v1:172.21.15.183:6827/3522836984] [v2:172.21.15.183:6828/3522836984,v1:172.21.15.183:6829/3522836984] exists,up 7fd07a95-ad22-43f5-a968-539552283e89', 'osd.4 up in weight 1 up_from 32 up_thru 81 down_at 0 last_clean_interval [0,0) [v2:172.21.15.191:6800/2674828539,v1:172.21.15.191:6801/2674828539] [v2:172.21.15.191:6802/2674828539,v1:172.21.15.191:6803/2674828539] exists,up 631083fb-e503-44da-9b6b-154e88539ee7', 'osd.5 up in weight 1 up_from 39 up_thru 64 down_at 0 last_clean_interval [0,0) [v2:172.21.15.191:6808/681733378,v1:172.21.15.191:6809/681733378] [v2:172.21.15.191:6810/681733378,v1:172.21.15.191:6811/681733378] exists,up 6b963695-3eca-4e6e-9207-05b8010ebff9', 'osd.6 up in weight 1 up_from 47 up_thru 81 down_at 0 last_clean_interval [0,0) [v2:172.21.15.191:6816/2825813477,v1:172.21.15.191:6817/2825813477] [v2:172.21.15.191:6818/2825813477,v1:172.21.15.191:6819/2825813477] exists,up 2dc9754f-6aef-4e19-8605-2f1591921811', 'osd.7 up in weight 1 up_from 54 up_thru 64 down_at 0 last_clean_interval [0,0) [v2:172.21.15.191:6824/1422610021,v1:172.21.15.191:6825/1422610021] [v2:172.21.15.191:6826/1422610021,v1:172.21.15.191:6827/1422610021] exists,up 1a9a299e-d5ba-4b9a-85d5-7f6a5bc754e1'] 2024-09-10T19:11:56.308 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph daemon mon.a config show 2024-09-10T19:11:56.551 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:11:56.929 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:11:56 smithi183 ceph-mon[29736]: 2.4 scrub starts 2024-09-10T19:11:56.929 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:11:56 smithi183 ceph-mon[29736]: 2.4 scrub ok 2024-09-10T19:11:56.929 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:11:56 smithi183 ceph-mon[29736]: pgmap v50: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 0 B/s wr, 4 op/s; 27 B/s, 1 objects/s recovering 2024-09-10T19:11:56.929 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:11:56 smithi183 ceph-mon[29736]: 2.1a scrub starts 2024-09-10T19:11:56.929 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:11:56 smithi183 ceph-mon[29736]: 2.1a scrub ok 2024-09-10T19:11:56.929 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:11:56 smithi183 ceph-mon[29736]: 4.3 scrub starts 2024-09-10T19:11:56.929 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:11:56 smithi183 ceph-mon[29736]: 4.3 scrub ok 2024-09-10T19:11:56.929 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:11:56 smithi183 ceph-mon[41562]: 2.4 scrub starts 2024-09-10T19:11:56.929 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:11:56 smithi183 ceph-mon[41562]: 2.4 scrub ok 2024-09-10T19:11:56.930 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:11:56 smithi183 ceph-mon[41562]: pgmap v50: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 0 B/s wr, 4 op/s; 27 B/s, 1 objects/s recovering 2024-09-10T19:11:56.930 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:11:56 smithi183 ceph-mon[41562]: 2.1a scrub starts 2024-09-10T19:11:56.930 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:11:56 smithi183 ceph-mon[41562]: 2.1a scrub ok 2024-09-10T19:11:56.930 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:11:56 smithi183 ceph-mon[41562]: 4.3 scrub starts 2024-09-10T19:11:56.930 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:11:56 smithi183 ceph-mon[41562]: 4.3 scrub ok 2024-09-10T19:11:57.115 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:11:56 smithi191 ceph-mon[32884]: 2.4 scrub starts 2024-09-10T19:11:57.115 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:11:56 smithi191 ceph-mon[32884]: 2.4 scrub ok 2024-09-10T19:11:57.115 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:11:56 smithi191 ceph-mon[32884]: pgmap v50: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 2.5 KiB/s rd, 0 B/s wr, 4 op/s; 27 B/s, 1 objects/s recovering 2024-09-10T19:11:57.115 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:11:56 smithi191 ceph-mon[32884]: 2.1a scrub starts 2024-09-10T19:11:57.115 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:11:56 smithi191 ceph-mon[32884]: 2.1a scrub ok 2024-09-10T19:11:57.115 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:11:56 smithi191 ceph-mon[32884]: 4.3 scrub starts 2024-09-10T19:11:57.116 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:11:56 smithi191 ceph-mon[32884]: 4.3 scrub ok 2024-09-10T19:11:57.715 INFO:journalctl@ceph.mgr.x.smithi191.stdout:Sep 10 19:11:57 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-x[34967]: ::ffff:172.21.15.191 - - [10/Sep/2024:19:11:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T19:11:58.022 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:11:57 smithi183 ceph-mon[29736]: 4.e deep-scrub starts 2024-09-10T19:11:58.022 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:11:57 smithi183 ceph-mon[29736]: 4.e deep-scrub ok 2024-09-10T19:11:58.022 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:11:57 smithi183 ceph-mon[29736]: 5.19 scrub starts 2024-09-10T19:11:58.022 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:11:57 smithi183 ceph-mon[29736]: 5.19 scrub ok 2024-09-10T19:11:58.022 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:11:57 smithi183 ceph-mon[29736]: 5.10 deep-scrub starts 2024-09-10T19:11:58.022 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:11:57 smithi183 ceph-mon[29736]: 5.10 deep-scrub ok 2024-09-10T19:11:58.022 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:11:57 smithi183 ceph-mon[41562]: 4.e deep-scrub starts 2024-09-10T19:11:58.022 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:11:57 smithi183 ceph-mon[41562]: 4.e deep-scrub ok 2024-09-10T19:11:58.022 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:11:57 smithi183 ceph-mon[41562]: 5.19 scrub starts 2024-09-10T19:11:58.023 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:11:57 smithi183 ceph-mon[41562]: 5.19 scrub ok 2024-09-10T19:11:58.023 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:11:57 smithi183 ceph-mon[41562]: 5.10 deep-scrub starts 2024-09-10T19:11:58.023 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:11:57 smithi183 ceph-mon[41562]: 5.10 deep-scrub ok 2024-09-10T19:11:58.216 INFO:journalctl@ceph.prometheus.a.smithi191.stdout:Sep 10 19:11:57 smithi191 systemd[1]: Stopping Ceph prometheus.a for 320f6df8-6fa7-11ef-bcea-c7b262605968... 2024-09-10T19:11:58.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:11:57 smithi191 ceph-mon[32884]: 4.e deep-scrub starts 2024-09-10T19:11:58.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:11:57 smithi191 ceph-mon[32884]: 4.e deep-scrub ok 2024-09-10T19:11:58.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:11:57 smithi191 ceph-mon[32884]: 5.19 scrub starts 2024-09-10T19:11:58.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:11:57 smithi191 ceph-mon[32884]: 5.19 scrub ok 2024-09-10T19:11:58.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:11:57 smithi191 ceph-mon[32884]: 5.10 deep-scrub starts 2024-09-10T19:11:58.217 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:11:57 smithi191 ceph-mon[32884]: 5.10 deep-scrub ok 2024-09-10T19:11:58.262 INFO:teuthology.orchestra.run.smithi183.stdout:{ 2024-09-10T19:11:58.262 INFO:teuthology.orchestra.run.smithi183.stdout: "name": "mon.a", 2024-09-10T19:11:58.262 INFO:teuthology.orchestra.run.smithi183.stdout: "cluster": "ceph", 2024-09-10T19:11:58.263 INFO:teuthology.orchestra.run.smithi183.stdout: "admin_socket": "/var/run/ceph/ceph-mon.a.asok", 2024-09-10T19:11:58.263 INFO:teuthology.orchestra.run.smithi183.stdout: "admin_socket_mode": "", 2024-09-10T19:11:58.263 INFO:teuthology.orchestra.run.smithi183.stdout: "auth_allow_insecure_global_id_reclaim": "false", 2024-09-10T19:11:58.263 INFO:teuthology.orchestra.run.smithi183.stdout: "auth_client_required": "cephx, none", 2024-09-10T19:11:58.263 INFO:teuthology.orchestra.run.smithi183.stdout: "auth_cluster_required": "cephx", 2024-09-10T19:11:58.263 INFO:teuthology.orchestra.run.smithi183.stdout: "auth_debug": "true", 2024-09-10T19:11:58.263 INFO:teuthology.orchestra.run.smithi183.stdout: "auth_expose_insecure_global_id_reclaim": "true", 2024-09-10T19:11:58.263 INFO:teuthology.orchestra.run.smithi183.stdout: "auth_mon_ticket_ttl": "660.000000", 2024-09-10T19:11:58.263 INFO:teuthology.orchestra.run.smithi183.stdout: "auth_service_required": "cephx", 2024-09-10T19:11:58.263 INFO:teuthology.orchestra.run.smithi183.stdout: "auth_service_ticket_ttl": "240.000000", 2024-09-10T19:11:58.263 INFO:teuthology.orchestra.run.smithi183.stdout: "auth_supported": "", 2024-09-10T19:11:58.263 INFO:teuthology.orchestra.run.smithi183.stdout: "bdev_aio": "true", 2024-09-10T19:11:58.263 INFO:teuthology.orchestra.run.smithi183.stdout: "bdev_aio_max_queue_depth": "1024", 2024-09-10T19:11:58.264 INFO:teuthology.orchestra.run.smithi183.stdout: "bdev_aio_poll_ms": "250", 2024-09-10T19:11:58.264 INFO:teuthology.orchestra.run.smithi183.stdout: "bdev_aio_reap_max": "16", 2024-09-10T19:11:58.264 INFO:teuthology.orchestra.run.smithi183.stdout: "bdev_async_discard": "false", 2024-09-10T19:11:58.264 INFO:teuthology.orchestra.run.smithi183.stdout: "bdev_block_size": "4096", 2024-09-10T19:11:58.264 INFO:teuthology.orchestra.run.smithi183.stdout: "bdev_debug_aio": "false", 2024-09-10T19:11:58.264 INFO:teuthology.orchestra.run.smithi183.stdout: "bdev_debug_aio_log_age": "5.000000", 2024-09-10T19:11:58.264 INFO:teuthology.orchestra.run.smithi183.stdout: "bdev_debug_aio_suicide_timeout": "60.000000", 2024-09-10T19:11:58.264 INFO:teuthology.orchestra.run.smithi183.stdout: "bdev_debug_inflight_ios": "false", 2024-09-10T19:11:58.264 INFO:teuthology.orchestra.run.smithi183.stdout: "bdev_enable_discard": "false", 2024-09-10T19:11:58.264 INFO:teuthology.orchestra.run.smithi183.stdout: "bdev_flock_retry": "3", 2024-09-10T19:11:58.264 INFO:teuthology.orchestra.run.smithi183.stdout: "bdev_flock_retry_interval": "0.100000", 2024-09-10T19:11:58.264 INFO:teuthology.orchestra.run.smithi183.stdout: "bdev_inject_crash": "0", 2024-09-10T19:11:58.264 INFO:teuthology.orchestra.run.smithi183.stdout: "bdev_inject_crash_flush_delay": "2", 2024-09-10T19:11:58.265 INFO:teuthology.orchestra.run.smithi183.stdout: "bdev_ioring": "false", 2024-09-10T19:11:58.265 INFO:teuthology.orchestra.run.smithi183.stdout: "bdev_ioring_hipri": "false", 2024-09-10T19:11:58.265 INFO:teuthology.orchestra.run.smithi183.stdout: "bdev_ioring_sqthread_poll": "false", 2024-09-10T19:11:58.265 INFO:teuthology.orchestra.run.smithi183.stdout: "bdev_nvme_unbind_from_kernel": "false", 2024-09-10T19:11:58.265 INFO:teuthology.orchestra.run.smithi183.stdout: "bdev_read_buffer_alignment": "4096", 2024-09-10T19:11:58.266 INFO:teuthology.orchestra.run.smithi183.stdout: "bdev_read_preallocated_huge_buffers": "", 2024-09-10T19:11:58.266 INFO:teuthology.orchestra.run.smithi183.stdout: "bdev_type": "", 2024-09-10T19:11:58.266 INFO:teuthology.orchestra.run.smithi183.stdout: "bluefs_alloc_size": "1048576", 2024-09-10T19:11:58.266 INFO:teuthology.orchestra.run.smithi183.stdout: "bluefs_allocator": "hybrid", 2024-09-10T19:11:58.266 INFO:teuthology.orchestra.run.smithi183.stdout: "bluefs_buffered_io": "true", 2024-09-10T19:11:58.266 INFO:teuthology.orchestra.run.smithi183.stdout: "bluefs_check_for_zeros": "false", 2024-09-10T19:11:58.266 INFO:teuthology.orchestra.run.smithi183.stdout: "bluefs_check_volume_selector_often": "false", 2024-09-10T19:11:58.266 INFO:teuthology.orchestra.run.smithi183.stdout: "bluefs_check_volume_selector_on_umount": "false", 2024-09-10T19:11:58.266 INFO:teuthology.orchestra.run.smithi183.stdout: "bluefs_compact_log_sync": "false", 2024-09-10T19:11:58.267 INFO:teuthology.orchestra.run.smithi183.stdout: "bluefs_failed_shared_alloc_cooldown": "600.000000", 2024-09-10T19:11:58.267 INFO:teuthology.orchestra.run.smithi183.stdout: "bluefs_log_compact_min_ratio": "5.000000", 2024-09-10T19:11:58.267 INFO:teuthology.orchestra.run.smithi183.stdout: "bluefs_log_compact_min_size": "16777216", 2024-09-10T19:11:58.267 INFO:teuthology.orchestra.run.smithi183.stdout: "bluefs_log_replay_check_allocations": "true", 2024-09-10T19:11:58.267 INFO:teuthology.orchestra.run.smithi183.stdout: "bluefs_max_log_runway": "4194304", 2024-09-10T19:11:58.267 INFO:teuthology.orchestra.run.smithi183.stdout: "bluefs_max_prefetch": "1048576", 2024-09-10T19:11:58.267 INFO:teuthology.orchestra.run.smithi183.stdout: "bluefs_min_flush_size": "524288", 2024-09-10T19:11:58.267 INFO:teuthology.orchestra.run.smithi183.stdout: "bluefs_min_log_runway": "1048576", 2024-09-10T19:11:58.267 INFO:teuthology.orchestra.run.smithi183.stdout: "bluefs_replay_recovery": "false", 2024-09-10T19:11:58.267 INFO:teuthology.orchestra.run.smithi183.stdout: "bluefs_replay_recovery_disable_compact": "false", 2024-09-10T19:11:58.267 INFO:teuthology.orchestra.run.smithi183.stdout: "bluefs_shared_alloc_size": "65536", 2024-09-10T19:11:58.267 INFO:teuthology.orchestra.run.smithi183.stdout: "bluefs_sync_write": "false", 2024-09-10T19:11:58.267 INFO:teuthology.orchestra.run.smithi183.stdout: "bluestore_2q_cache_kin_ratio": "0.500000", 2024-09-10T19:11:58.267 INFO:teuthology.orchestra.run.smithi183.stdout: "bluestore_2q_cache_kout_ratio": "0.500000", 2024-09-10T19:11:58.268 INFO:teuthology.orchestra.run.smithi183.stdout: "bluestore_alloc_stats_dump_interval": "86400.000000", 2024-09-10T19:11:58.268 INFO:teuthology.orchestra.run.smithi183.stdout: "bluestore_allocation_from_file": "true", 2024-09-10T19:11:58.268 INFO:teuthology.orchestra.run.smithi183.stdout: "bluestore_allocator": "hybrid", 2024-09-10T19:11:58.268 INFO:teuthology.orchestra.run.smithi183.stdout: "bluestore_async_db_compaction": "true", 2024-09-10T19:11:58.268 INFO:teuthology.orchestra.run.smithi183.stdout: "bluestore_avl_alloc_bf_free_pct": "4", 2024-09-10T19:11:58.268 INFO:teuthology.orchestra.run.smithi183.stdout: "bluestore_avl_alloc_bf_threshold": "131072", 2024-09-10T19:11:58.268 INFO:teuthology.orchestra.run.smithi183.stdout: "bluestore_avl_alloc_ff_max_search_bytes": "16777216", 2024-09-10T19:11:58.268 INFO:teuthology.orchestra.run.smithi183.stdout: "bluestore_avl_alloc_ff_max_search_count": "100", 2024-09-10T19:11:58.268 INFO:teuthology.orchestra.run.smithi183.stdout: "bluestore_bitmapallocator_blocks_per_zone": "1024", 2024-09-10T19:11:58.268 INFO:teuthology.orchestra.run.smithi183.stdout: "bluestore_bitmapallocator_span_size": "1024", 2024-09-10T19:11:58.268 INFO:teuthology.orchestra.run.smithi183.stdout: "bluestore_blobid_prealloc": "10240", 2024-09-10T19:11:58.268 INFO:teuthology.orchestra.run.smithi183.stdout: "bluestore_block_create": "true", 2024-09-10T19:11:58.268 INFO:teuthology.orchestra.run.smithi183.stdout: "bluestore_block_db_create": "false", 2024-09-10T19:11:58.269 INFO:teuthology.orchestra.run.smithi183.stdout: "bluestore_block_db_path": "", 2024-09-10T19:11:58.269 INFO:teuthology.orchestra.run.smithi183.stdout: "bluestore_block_db_size": "0", 2024-09-10T19:11:58.269 INFO:teuthology.orchestra.run.smithi183.stdout: "bluestore_block_path": "", 2024-09-10T19:11:58.269 INFO:teuthology.orchestra.run.smithi183.stdout: "bluestore_block_preallocate_file": "false", 2024-09-10T19:11:58.269 INFO:teuthology.orchestra.run.smithi183.stdout: "bluestore_block_size": "107374182400", 2024-09-10T19:11:58.269 INFO:teuthology.orchestra.run.smithi183.stdout: "bluestore_block_wal_create": "false", 2024-09-10T19:11:58.269 INFO:teuthology.orchestra.run.smithi183.stdout: "bluestore_block_wal_path": "", 2024-09-10T19:11:58.269 INFO:teuthology.orchestra.run.smithi183.stdout: "bluestore_block_wal_size": "100663296", 2024-09-10T19:11:58.269 INFO:teuthology.orchestra.run.smithi183.stdout: "bluestore_bluefs": "true", 2024-09-10T19:11:58.269 INFO:teuthology.orchestra.run.smithi183.stdout: "bluestore_bluefs_alloc_failure_dump_interval": "0.000000", 2024-09-10T19:11:58.269 INFO:teuthology.orchestra.run.smithi183.stdout: "bluestore_bluefs_env_mirror": "false", 2024-09-10T19:11:58.269 INFO:teuthology.orchestra.run.smithi183.stdout: "bluestore_bluefs_max_free": "10737418240", 2024-09-10T19:11:58.269 INFO:teuthology.orchestra.run.smithi183.stdout: "bluestore_cache_age_bin_interval": "1.000000", 2024-09-10T19:11:58.270 INFO:teuthology.orchestra.run.smithi183.stdout: "bluestore_cache_age_bins_data": "1 2 6 24 120 720 0 0 0 0", 2024-09-10T19:11:58.270 INFO:teuthology.orchestra.run.smithi183.stdout: "bluestore_cache_age_bins_kv": "1 2 6 24 120 720 0 0 0 0", 2024-09-10T19:11:58.270 INFO:teuthology.orchestra.run.smithi183.stdout: "bluestore_cache_age_bins_kv_onode": "0 0 0 0 0 0 0 0 0 720", 2024-09-10T19:11:58.270 INFO:teuthology.orchestra.run.smithi183.stdout: "bluestore_cache_age_bins_meta": "1 2 6 24 120 720 0 0 0 0", 2024-09-10T19:11:58.270 INFO:teuthology.orchestra.run.smithi183.stdout: "bluestore_cache_autotune": "true", 2024-09-10T19:11:58.270 INFO:teuthology.orchestra.run.smithi183.stdout: "bluestore_cache_autotune_interval": "5.000000", 2024-09-10T19:11:58.270 INFO:teuthology.orchestra.run.smithi183.stdout: "bluestore_cache_kv_onode_ratio": "0.040000", 2024-09-10T19:11:58.270 INFO:teuthology.orchestra.run.smithi183.stdout: "bluestore_cache_kv_ratio": "0.450000", 2024-09-10T19:11:58.270 INFO:teuthology.orchestra.run.smithi183.stdout: "bluestore_cache_meta_ratio": "0.450000", 2024-09-10T19:11:58.270 INFO:teuthology.orchestra.run.smithi183.stdout: "bluestore_cache_size": "0", 2024-09-10T19:11:58.270 INFO:teuthology.orchestra.run.smithi183.stdout: "bluestore_cache_size_hdd": "1073741824", 2024-09-10T19:11:58.270 INFO:teuthology.orchestra.run.smithi183.stdout: "bluestore_cache_size_ssd": "3221225472", 2024-09-10T19:11:58.270 INFO:teuthology.orchestra.run.smithi183.stdout: "bluestore_cache_trim_interval": "0.050000", 2024-09-10T19:11:58.271 INFO:teuthology.orchestra.run.smithi183.stdout: "bluestore_cache_trim_max_skip_pinned": "1000", 2024-09-10T19:11:58.271 INFO:teuthology.orchestra.run.smithi183.stdout: "bluestore_cache_type": "2q", 2024-09-10T19:11:58.271 INFO:teuthology.orchestra.run.smithi183.stdout: "bluestore_cleaner_sleep_interval": "5.000000", 2024-09-10T19:11:58.271 INFO:teuthology.orchestra.run.smithi183.stdout: "bluestore_clone_cow": "true", 2024-09-10T19:11:58.271 INFO:teuthology.orchestra.run.smithi183.stdout: "bluestore_compression_algorithm": "snappy", 2024-09-10T19:11:58.271 INFO:teuthology.orchestra.run.smithi183.stdout: "bluestore_compression_max_blob_size": "0", 2024-09-10T19:11:58.271 INFO:teuthology.orchestra.run.smithi183.stdout: "bluestore_compression_max_blob_size_hdd": "65536", 2024-09-10T19:11:58.271 INFO:teuthology.orchestra.run.smithi183.stdout: "bluestore_compression_max_blob_size_ssd": "65536", 2024-09-10T19:11:58.271 INFO:teuthology.orchestra.run.smithi183.stdout: "bluestore_compression_min_blob_size": "0", 2024-09-10T19:11:58.271 INFO:teuthology.orchestra.run.smithi183.stdout: "bluestore_compression_min_blob_size_hdd": "8192", 2024-09-10T19:11:58.271 INFO:teuthology.orchestra.run.smithi183.stdout: "bluestore_compression_min_blob_size_ssd": "65536", 2024-09-10T19:11:58.271 INFO:teuthology.orchestra.run.smithi183.stdout: "bluestore_compression_mode": "none", 2024-09-10T19:11:58.271 INFO:teuthology.orchestra.run.smithi183.stdout: "bluestore_compression_required_ratio": "0.875000", 2024-09-10T19:11:58.271 INFO:teuthology.orchestra.run.smithi183.stdout: "bluestore_csum_type": "crc32c", 2024-09-10T19:11:58.272 INFO:teuthology.orchestra.run.smithi183.stdout: "bluestore_debug_enforce_settings": "default", 2024-09-10T19:11:58.272 INFO:teuthology.orchestra.run.smithi183.stdout: "bluestore_debug_freelist": "false", 2024-09-10T19:11:58.272 INFO:teuthology.orchestra.run.smithi183.stdout: "bluestore_debug_fsck_abort": "false", 2024-09-10T19:11:58.272 INFO:teuthology.orchestra.run.smithi183.stdout: "bluestore_debug_inject_allocation_from_file_failure": "0.000000", 2024-09-10T19:11:58.272 INFO:teuthology.orchestra.run.smithi183.stdout: "bluestore_debug_inject_bug21040": "false", 2024-09-10T19:11:58.272 INFO:teuthology.orchestra.run.smithi183.stdout: "bluestore_debug_inject_csum_err_probability": "0.000000", 2024-09-10T19:11:58.272 INFO:teuthology.orchestra.run.smithi183.stdout: "bluestore_debug_inject_read_err": "false", 2024-09-10T19:11:58.272 INFO:teuthology.orchestra.run.smithi183.stdout: "bluestore_debug_legacy_omap": "false", 2024-09-10T19:11:58.272 INFO:teuthology.orchestra.run.smithi183.stdout: "bluestore_debug_no_reuse_blocks": "false", 2024-09-10T19:11:58.273 INFO:teuthology.orchestra.run.smithi183.stdout: "bluestore_debug_omit_block_device_write": "false", 2024-09-10T19:11:58.273 INFO:teuthology.orchestra.run.smithi183.stdout: "bluestore_debug_omit_kv_commit": "false", 2024-09-10T19:11:58.273 INFO:teuthology.orchestra.run.smithi183.stdout: "bluestore_debug_permit_any_bdev_label": "false", 2024-09-10T19:11:58.273 INFO:teuthology.orchestra.run.smithi183.stdout: "bluestore_debug_prefill": "0.000000", 2024-09-10T19:11:58.273 INFO:teuthology.orchestra.run.smithi183.stdout: "bluestore_debug_prefragment_max": "1048576", 2024-09-10T19:11:58.273 INFO:teuthology.orchestra.run.smithi183.stdout: "bluestore_debug_random_read_err": "0.000000", 2024-09-10T19:11:58.273 INFO:teuthology.orchestra.run.smithi183.stdout: "bluestore_debug_randomize_serial_transaction": "0", 2024-09-10T19:11:58.273 INFO:teuthology.orchestra.run.smithi183.stdout: "bluestore_debug_small_allocations": "0", 2024-09-10T19:11:58.273 INFO:teuthology.orchestra.run.smithi183.stdout: "bluestore_debug_too_many_blobs_threshold": "24576", 2024-09-10T19:11:58.273 INFO:teuthology.orchestra.run.smithi183.stdout: "bluestore_default_buffered_read": "true", 2024-09-10T19:11:58.274 INFO:teuthology.orchestra.run.smithi183.stdout: "bluestore_default_buffered_write": "false", 2024-09-10T19:11:58.274 INFO:teuthology.orchestra.run.smithi183.stdout: "bluestore_deferred_batch_ops": "0", 2024-09-10T19:11:58.274 INFO:teuthology.orchestra.run.smithi183.stdout: "bluestore_deferred_batch_ops_hdd": "64", 2024-09-10T19:11:58.274 INFO:teuthology.orchestra.run.smithi183.stdout: "bluestore_deferred_batch_ops_ssd": "16", 2024-09-10T19:11:58.274 INFO:teuthology.orchestra.run.smithi183.stdout: "bluestore_extent_map_inline_shard_prealloc_size": "256", 2024-09-10T19:11:58.274 INFO:teuthology.orchestra.run.smithi183.stdout: "bluestore_extent_map_shard_max_size": "1200", 2024-09-10T19:11:58.274 INFO:teuthology.orchestra.run.smithi183.stdout: "bluestore_extent_map_shard_min_size": "150", 2024-09-10T19:11:58.274 INFO:teuthology.orchestra.run.smithi183.stdout: "bluestore_extent_map_shard_target_size": "500", 2024-09-10T19:11:58.274 INFO:teuthology.orchestra.run.smithi183.stdout: "bluestore_extent_map_shard_target_size_slop": "0.200000", 2024-09-10T19:11:58.274 INFO:teuthology.orchestra.run.smithi183.stdout: "bluestore_fail_eio": "false", 2024-09-10T19:11:58.274 INFO:teuthology.orchestra.run.smithi183.stdout: "bluestore_freelist_blocks_per_key": "128", 2024-09-10T19:11:58.275 INFO:teuthology.orchestra.run.smithi183.stdout: "bluestore_fsck_error_on_no_per_pg_omap": "false", 2024-09-10T19:11:58.275 INFO:teuthology.orchestra.run.smithi183.stdout: "bluestore_fsck_error_on_no_per_pool_omap": "false", 2024-09-10T19:11:58.275 INFO:teuthology.orchestra.run.smithi183.stdout: "bluestore_fsck_error_on_no_per_pool_stats": "false", 2024-09-10T19:11:58.276 INFO:teuthology.orchestra.run.smithi183.stdout: "bluestore_fsck_on_mkfs": "true", 2024-09-10T19:11:58.276 INFO:teuthology.orchestra.run.smithi183.stdout: "bluestore_fsck_on_mkfs_deep": "false", 2024-09-10T19:11:58.276 INFO:teuthology.orchestra.run.smithi183.stdout: "bluestore_fsck_on_mount": "false", 2024-09-10T19:11:58.276 INFO:teuthology.orchestra.run.smithi183.stdout: "bluestore_fsck_on_mount_deep": "false", 2024-09-10T19:11:58.276 INFO:teuthology.orchestra.run.smithi183.stdout: "bluestore_fsck_on_umount": "false", 2024-09-10T19:11:58.276 INFO:teuthology.orchestra.run.smithi183.stdout: "bluestore_fsck_on_umount_deep": "false", 2024-09-10T19:11:58.276 INFO:teuthology.orchestra.run.smithi183.stdout: "bluestore_fsck_quick_fix_on_mount": "false", 2024-09-10T19:11:58.276 INFO:teuthology.orchestra.run.smithi183.stdout: "bluestore_fsck_quick_fix_threads": "2", 2024-09-10T19:11:58.276 INFO:teuthology.orchestra.run.smithi183.stdout: "bluestore_fsck_read_bytes_cap": "67108864", 2024-09-10T19:11:58.276 INFO:teuthology.orchestra.run.smithi183.stdout: "bluestore_fsck_shared_blob_tracker_size": "0.031250", 2024-09-10T19:11:58.276 INFO:teuthology.orchestra.run.smithi183.stdout: "bluestore_gc_enable_blob_threshold": "0", 2024-09-10T19:11:58.276 INFO:teuthology.orchestra.run.smithi183.stdout: "bluestore_gc_enable_total_threshold": "0", 2024-09-10T19:11:58.276 INFO:teuthology.orchestra.run.smithi183.stdout: "bluestore_hybrid_alloc_mem_cap": "67108864", 2024-09-10T19:11:58.277 INFO:teuthology.orchestra.run.smithi183.stdout: "bluestore_ignore_data_csum": "false", 2024-09-10T19:11:58.277 INFO:teuthology.orchestra.run.smithi183.stdout: "bluestore_kv_sync_util_logging_s": "10.000000", 2024-09-10T19:11:58.277 INFO:teuthology.orchestra.run.smithi183.stdout: "bluestore_kvbackend": "rocksdb", 2024-09-10T19:11:58.277 INFO:teuthology.orchestra.run.smithi183.stdout: "bluestore_log_collection_list_age": "60.000000", 2024-09-10T19:11:58.277 INFO:teuthology.orchestra.run.smithi183.stdout: "bluestore_log_omap_iterator_age": "5.000000", 2024-09-10T19:11:58.277 INFO:teuthology.orchestra.run.smithi183.stdout: "bluestore_log_op_age": "5.000000", 2024-09-10T19:11:58.277 INFO:teuthology.orchestra.run.smithi183.stdout: "bluestore_max_alloc_size": "0", 2024-09-10T19:11:58.277 INFO:teuthology.orchestra.run.smithi183.stdout: "bluestore_max_blob_size": "0", 2024-09-10T19:11:58.277 INFO:teuthology.orchestra.run.smithi183.stdout: "bluestore_max_blob_size_hdd": "65536", 2024-09-10T19:11:58.277 INFO:teuthology.orchestra.run.smithi183.stdout: "bluestore_max_blob_size_ssd": "65536", 2024-09-10T19:11:58.277 INFO:teuthology.orchestra.run.smithi183.stdout: "bluestore_max_defer_interval": "3.000000", 2024-09-10T19:11:58.277 INFO:teuthology.orchestra.run.smithi183.stdout: "bluestore_max_deferred_txc": "32", 2024-09-10T19:11:58.277 INFO:teuthology.orchestra.run.smithi183.stdout: "bluestore_min_alloc_size": "0", 2024-09-10T19:11:58.278 INFO:teuthology.orchestra.run.smithi183.stdout: "bluestore_min_alloc_size_hdd": "4096", 2024-09-10T19:11:58.278 INFO:teuthology.orchestra.run.smithi183.stdout: "bluestore_min_alloc_size_ssd": "4096", 2024-09-10T19:11:58.278 INFO:teuthology.orchestra.run.smithi183.stdout: "bluestore_nid_prealloc": "1024", 2024-09-10T19:11:58.278 INFO:teuthology.orchestra.run.smithi183.stdout: "bluestore_prefer_deferred_size": "0", 2024-09-10T19:11:58.278 INFO:teuthology.orchestra.run.smithi183.stdout: "bluestore_prefer_deferred_size_hdd": "65536", 2024-09-10T19:11:58.278 INFO:teuthology.orchestra.run.smithi183.stdout: "bluestore_prefer_deferred_size_ssd": "0", 2024-09-10T19:11:58.278 INFO:teuthology.orchestra.run.smithi183.stdout: "bluestore_qfsck_on_mount": "true", 2024-09-10T19:11:58.278 INFO:teuthology.orchestra.run.smithi183.stdout: "bluestore_retry_disk_reads": "3", 2024-09-10T19:11:58.278 INFO:teuthology.orchestra.run.smithi183.stdout: "bluestore_rocksdb_cf": "true", 2024-09-10T19:11:58.278 INFO:teuthology.orchestra.run.smithi183.stdout: "bluestore_rocksdb_cfs": "m(3) p(3,0-12) O(3,0-13)=block_cache={type=binned_lru} L P", 2024-09-10T19:11:58.278 INFO:teuthology.orchestra.run.smithi183.stdout: "bluestore_rocksdb_options": "compression=kNoCompression,max_write_buffer_number=4,min_write_buffer_number_to_merge=1,recycle_log_file_num=4,write_buffer_size=268435456,writable_file_max_buffer_size=0,compaction_readahead_size=2097152,max_background_compactions=2,max_total_wal_size=1073741824", 2024-09-10T19:11:58.278 INFO:teuthology.orchestra.run.smithi183.stdout: "bluestore_rocksdb_options_annex": "", 2024-09-10T19:11:58.278 INFO:teuthology.orchestra.run.smithi183.stdout: "bluestore_spdk_coremask": "0x1", 2024-09-10T19:11:58.279 INFO:teuthology.orchestra.run.smithi183.stdout: "bluestore_spdk_io_sleep": "5", 2024-09-10T19:11:58.279 INFO:teuthology.orchestra.run.smithi183.stdout: "bluestore_spdk_max_io_completion": "0", 2024-09-10T19:11:58.279 INFO:teuthology.orchestra.run.smithi183.stdout: "bluestore_spdk_mem": "512", 2024-09-10T19:11:58.279 INFO:teuthology.orchestra.run.smithi183.stdout: "bluestore_sync_submit_transaction": "false", 2024-09-10T19:11:58.279 INFO:teuthology.orchestra.run.smithi183.stdout: "bluestore_throttle_bytes": "67108864", 2024-09-10T19:11:58.279 INFO:teuthology.orchestra.run.smithi183.stdout: "bluestore_throttle_cost_per_io": "0", 2024-09-10T19:11:58.279 INFO:teuthology.orchestra.run.smithi183.stdout: "bluestore_throttle_cost_per_io_hdd": "670000", 2024-09-10T19:11:58.279 INFO:teuthology.orchestra.run.smithi183.stdout: "bluestore_throttle_cost_per_io_ssd": "4000", 2024-09-10T19:11:58.279 INFO:teuthology.orchestra.run.smithi183.stdout: "bluestore_throttle_deferred_bytes": "134217728", 2024-09-10T19:11:58.279 INFO:teuthology.orchestra.run.smithi183.stdout: "bluestore_throttle_trace_rate": "0.000000", 2024-09-10T19:11:58.279 INFO:teuthology.orchestra.run.smithi183.stdout: "bluestore_tracing": "false", 2024-09-10T19:11:58.279 INFO:teuthology.orchestra.run.smithi183.stdout: "bluestore_use_optimal_io_size_for_min_alloc_size": "false", 2024-09-10T19:11:58.279 INFO:teuthology.orchestra.run.smithi183.stdout: "bluestore_volume_selection_policy": "use_some_extra", 2024-09-10T19:11:58.279 INFO:teuthology.orchestra.run.smithi183.stdout: "bluestore_volume_selection_reserved": "0", 2024-09-10T19:11:58.279 INFO:teuthology.orchestra.run.smithi183.stdout: "bluestore_volume_selection_reserved_factor": "2.000000", 2024-09-10T19:11:58.280 INFO:teuthology.orchestra.run.smithi183.stdout: "bluestore_warn_on_bluefs_spillover": "true", 2024-09-10T19:11:58.280 INFO:teuthology.orchestra.run.smithi183.stdout: "bluestore_warn_on_legacy_statfs": "true", 2024-09-10T19:11:58.280 INFO:teuthology.orchestra.run.smithi183.stdout: "bluestore_warn_on_no_per_pg_omap": "false", 2024-09-10T19:11:58.280 INFO:teuthology.orchestra.run.smithi183.stdout: "bluestore_warn_on_no_per_pool_omap": "true", 2024-09-10T19:11:58.280 INFO:teuthology.orchestra.run.smithi183.stdout: "bluestore_warn_on_spurious_read_errors": "true", 2024-09-10T19:11:58.280 INFO:teuthology.orchestra.run.smithi183.stdout: "bluestore_zero_block_detection": "false", 2024-09-10T19:11:58.280 INFO:teuthology.orchestra.run.smithi183.stdout: "cephadm_path": "/usr/sbin/cephadm", 2024-09-10T19:11:58.280 INFO:teuthology.orchestra.run.smithi183.stdout: "cephfs_mirror_action_update_interval": "2", 2024-09-10T19:11:58.280 INFO:teuthology.orchestra.run.smithi183.stdout: "cephfs_mirror_directory_scan_interval": "10", 2024-09-10T19:11:58.280 INFO:teuthology.orchestra.run.smithi183.stdout: "cephfs_mirror_max_concurrent_directory_syncs": "3", 2024-09-10T19:11:58.280 INFO:teuthology.orchestra.run.smithi183.stdout: "cephfs_mirror_max_consecutive_failures_per_directory": "10", 2024-09-10T19:11:58.280 INFO:teuthology.orchestra.run.smithi183.stdout: "cephfs_mirror_max_snapshot_sync_per_cycle": "3", 2024-09-10T19:11:58.280 INFO:teuthology.orchestra.run.smithi183.stdout: "cephfs_mirror_mount_timeout": "10", 2024-09-10T19:11:58.281 INFO:teuthology.orchestra.run.smithi183.stdout: "cephfs_mirror_restart_mirror_on_blocklist_interval": "30", 2024-09-10T19:11:58.281 INFO:teuthology.orchestra.run.smithi183.stdout: "cephfs_mirror_restart_mirror_on_failure_interval": "20", 2024-09-10T19:11:58.281 INFO:teuthology.orchestra.run.smithi183.stdout: "cephfs_mirror_retry_failed_directories_interval": "60", 2024-09-10T19:11:58.281 INFO:teuthology.orchestra.run.smithi183.stdout: "cephsqlite_blocklist_dead_locker": "true", 2024-09-10T19:11:58.281 INFO:teuthology.orchestra.run.smithi183.stdout: "cephsqlite_lock_renewal_interval": "2000", 2024-09-10T19:11:58.281 INFO:teuthology.orchestra.run.smithi183.stdout: "cephsqlite_lock_renewal_timeout": "30000", 2024-09-10T19:11:58.281 INFO:teuthology.orchestra.run.smithi183.stdout: "cephx_cluster_require_signatures": "false", 2024-09-10T19:11:58.281 INFO:teuthology.orchestra.run.smithi183.stdout: "cephx_cluster_require_version": "2", 2024-09-10T19:11:58.281 INFO:teuthology.orchestra.run.smithi183.stdout: "cephx_require_signatures": "false", 2024-09-10T19:11:58.281 INFO:teuthology.orchestra.run.smithi183.stdout: "cephx_require_version": "2", 2024-09-10T19:11:58.281 INFO:teuthology.orchestra.run.smithi183.stdout: "cephx_service_require_signatures": "false", 2024-09-10T19:11:58.281 INFO:teuthology.orchestra.run.smithi183.stdout: "cephx_service_require_version": "2", 2024-09-10T19:11:58.281 INFO:teuthology.orchestra.run.smithi183.stdout: "cephx_sign_messages": "true", 2024-09-10T19:11:58.281 INFO:teuthology.orchestra.run.smithi183.stdout: "chdir": "", 2024-09-10T19:11:58.282 INFO:teuthology.orchestra.run.smithi183.stdout: "client_acl_type": "", 2024-09-10T19:11:58.282 INFO:teuthology.orchestra.run.smithi183.stdout: "client_asio_thread_count": "2", 2024-09-10T19:11:58.282 INFO:teuthology.orchestra.run.smithi183.stdout: "client_cache_mid": "0.750000", 2024-09-10T19:11:58.282 INFO:teuthology.orchestra.run.smithi183.stdout: "client_cache_size": "16384", 2024-09-10T19:11:58.282 INFO:teuthology.orchestra.run.smithi183.stdout: "client_caps_release_delay": "5", 2024-09-10T19:11:58.282 INFO:teuthology.orchestra.run.smithi183.stdout: "client_check_pool_perm": "true", 2024-09-10T19:11:58.282 INFO:teuthology.orchestra.run.smithi183.stdout: "client_collect_and_send_global_metrics": "false", 2024-09-10T19:11:58.282 INFO:teuthology.orchestra.run.smithi183.stdout: "client_debug_force_sync_read": "false", 2024-09-10T19:11:58.282 INFO:teuthology.orchestra.run.smithi183.stdout: "client_debug_getattr_caps": "false", 2024-09-10T19:11:58.283 INFO:teuthology.orchestra.run.smithi183.stdout: "client_debug_inject_features": "", 2024-09-10T19:11:58.283 INFO:teuthology.orchestra.run.smithi183.stdout: "client_debug_inject_tick_delay": "0", 2024-09-10T19:11:58.283 INFO:teuthology.orchestra.run.smithi183.stdout: "client_die_on_failed_dentry_invalidate": "true", 2024-09-10T19:11:58.283 INFO:teuthology.orchestra.run.smithi183.stdout: "client_die_on_failed_remount": "false", 2024-09-10T19:11:58.283 INFO:teuthology.orchestra.run.smithi183.stdout: "client_dirsize_rbytes": "true", 2024-09-10T19:11:58.283 INFO:teuthology.orchestra.run.smithi183.stdout: "client_force_lazyio": "false", 2024-09-10T19:11:58.283 INFO:teuthology.orchestra.run.smithi183.stdout: "client_fs": "", 2024-09-10T19:11:58.283 INFO:teuthology.orchestra.run.smithi183.stdout: "client_inject_fixed_oldest_tid": "false", 2024-09-10T19:11:58.283 INFO:teuthology.orchestra.run.smithi183.stdout: "client_inject_release_failure": "false", 2024-09-10T19:11:58.283 INFO:teuthology.orchestra.run.smithi183.stdout: "client_max_inline_size": "4096", 2024-09-10T19:11:58.283 INFO:teuthology.orchestra.run.smithi183.stdout: "client_max_retries_on_remount_failure": "5", 2024-09-10T19:11:58.283 INFO:teuthology.orchestra.run.smithi183.stdout: "client_mds_namespace": "", 2024-09-10T19:11:58.284 INFO:teuthology.orchestra.run.smithi183.stdout: "client_metadata": "", 2024-09-10T19:11:58.284 INFO:teuthology.orchestra.run.smithi183.stdout: "client_mount_gid": "-1", 2024-09-10T19:11:58.284 INFO:teuthology.orchestra.run.smithi183.stdout: "client_mount_timeout": "300", 2024-09-10T19:11:58.284 INFO:teuthology.orchestra.run.smithi183.stdout: "client_mount_uid": "-1", 2024-09-10T19:11:58.284 INFO:teuthology.orchestra.run.smithi183.stdout: "client_mountpoint": "/", 2024-09-10T19:11:58.284 INFO:teuthology.orchestra.run.smithi183.stdout: "client_notify_timeout": "10", 2024-09-10T19:11:58.284 INFO:teuthology.orchestra.run.smithi183.stdout: "client_oc": "true", 2024-09-10T19:11:58.284 INFO:teuthology.orchestra.run.smithi183.stdout: "client_oc_max_dirty": "104857600", 2024-09-10T19:11:58.284 INFO:teuthology.orchestra.run.smithi183.stdout: "client_oc_max_dirty_age": "5.000000", 2024-09-10T19:11:58.284 INFO:teuthology.orchestra.run.smithi183.stdout: "client_oc_max_objects": "1000", 2024-09-10T19:11:58.284 INFO:teuthology.orchestra.run.smithi183.stdout: "client_oc_size": "209715200", 2024-09-10T19:11:58.284 INFO:teuthology.orchestra.run.smithi183.stdout: "client_oc_target_dirty": "8388608", 2024-09-10T19:11:58.284 INFO:teuthology.orchestra.run.smithi183.stdout: "client_permissions": "true", 2024-09-10T19:11:58.284 INFO:teuthology.orchestra.run.smithi183.stdout: "client_quota": "true", 2024-09-10T19:11:58.285 INFO:teuthology.orchestra.run.smithi183.stdout: "client_quota_df": "true", 2024-09-10T19:11:58.285 INFO:teuthology.orchestra.run.smithi183.stdout: "client_readahead_max_bytes": "0", 2024-09-10T19:11:58.285 INFO:teuthology.orchestra.run.smithi183.stdout: "client_readahead_max_periods": "4", 2024-09-10T19:11:58.285 INFO:teuthology.orchestra.run.smithi183.stdout: "client_readahead_min": "131072", 2024-09-10T19:11:58.285 INFO:teuthology.orchestra.run.smithi183.stdout: "client_reconnect_stale": "false", 2024-09-10T19:11:58.285 INFO:teuthology.orchestra.run.smithi183.stdout: "client_shutdown_timeout": "30", 2024-09-10T19:11:58.285 INFO:teuthology.orchestra.run.smithi183.stdout: "client_snapdir": ".snap", 2024-09-10T19:11:58.285 INFO:teuthology.orchestra.run.smithi183.stdout: "client_tick_interval": "1", 2024-09-10T19:11:58.285 INFO:teuthology.orchestra.run.smithi183.stdout: "client_trace": "", 2024-09-10T19:11:58.285 INFO:teuthology.orchestra.run.smithi183.stdout: "client_try_dentry_invalidate": "false", 2024-09-10T19:11:58.286 INFO:teuthology.orchestra.run.smithi183.stdout: "client_use_faked_inos": "false", 2024-09-10T19:11:58.286 INFO:teuthology.orchestra.run.smithi183.stdout: "client_use_random_mds": "false", 2024-09-10T19:11:58.286 INFO:teuthology.orchestra.run.smithi183.stdout: "clog_to_graylog": "false", 2024-09-10T19:11:58.286 INFO:teuthology.orchestra.run.smithi183.stdout: "clog_to_graylog_host": "127.0.0.1", 2024-09-10T19:11:58.286 INFO:teuthology.orchestra.run.smithi183.stdout: "clog_to_graylog_port": "12201", 2024-09-10T19:11:58.286 INFO:teuthology.orchestra.run.smithi183.stdout: "clog_to_monitors": "default=true", 2024-09-10T19:11:58.286 INFO:teuthology.orchestra.run.smithi183.stdout: "clog_to_syslog": "false", 2024-09-10T19:11:58.287 INFO:teuthology.orchestra.run.smithi183.stdout: "clog_to_syslog_facility": "default=daemon audit=local0", 2024-09-10T19:11:58.287 INFO:teuthology.orchestra.run.smithi183.stdout: "clog_to_syslog_level": "info", 2024-09-10T19:11:58.287 INFO:teuthology.orchestra.run.smithi183.stdout: "cluster_addr": "-", 2024-09-10T19:11:58.287 INFO:teuthology.orchestra.run.smithi183.stdout: "cluster_network": "", 2024-09-10T19:11:58.287 INFO:teuthology.orchestra.run.smithi183.stdout: "cluster_network_interface": "", 2024-09-10T19:11:58.287 INFO:teuthology.orchestra.run.smithi183.stdout: "compressor_zlib_isal": "false", 2024-09-10T19:11:58.287 INFO:teuthology.orchestra.run.smithi183.stdout: "compressor_zlib_level": "5", 2024-09-10T19:11:58.287 INFO:teuthology.orchestra.run.smithi183.stdout: "compressor_zlib_winsize": "-15", 2024-09-10T19:11:58.287 INFO:teuthology.orchestra.run.smithi183.stdout: "compressor_zstd_level": "1", 2024-09-10T19:11:58.287 INFO:teuthology.orchestra.run.smithi183.stdout: "container_image": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3bfeaf4417585c821cab9ca76e4d95f390b0e769807122a45463640820b5cd16", 2024-09-10T19:11:58.288 INFO:teuthology.orchestra.run.smithi183.stdout: "crash_dir": "/var/lib/ceph/crash", 2024-09-10T19:11:58.288 INFO:teuthology.orchestra.run.smithi183.stdout: "crimson_alien_op_num_threads": "6", 2024-09-10T19:11:58.288 INFO:teuthology.orchestra.run.smithi183.stdout: "crimson_alien_thread_cpu_cores": "", 2024-09-10T19:11:58.288 INFO:teuthology.orchestra.run.smithi183.stdout: "crimson_osd_obc_lru_size": "10", 2024-09-10T19:11:58.288 INFO:teuthology.orchestra.run.smithi183.stdout: "crimson_osd_scheduler_concurrency": "0", 2024-09-10T19:11:58.288 INFO:teuthology.orchestra.run.smithi183.stdout: "crush_location": "", 2024-09-10T19:11:58.288 INFO:teuthology.orchestra.run.smithi183.stdout: "crush_location_hook": "", 2024-09-10T19:11:58.288 INFO:teuthology.orchestra.run.smithi183.stdout: "crush_location_hook_timeout": "10", 2024-09-10T19:11:58.288 INFO:teuthology.orchestra.run.smithi183.stdout: "daemonize": "false", 2024-09-10T19:11:58.288 INFO:teuthology.orchestra.run.smithi183.stdout: "debug_alienstore": "0/5", 2024-09-10T19:11:58.288 INFO:teuthology.orchestra.run.smithi183.stdout: "debug_allow_any_pool_priority": "false", 2024-09-10T19:11:58.288 INFO:teuthology.orchestra.run.smithi183.stdout: "debug_asok": "1/5", 2024-09-10T19:11:58.288 INFO:teuthology.orchestra.run.smithi183.stdout: "debug_asok_assert_abort": "false", 2024-09-10T19:11:58.289 INFO:teuthology.orchestra.run.smithi183.stdout: "debug_asserts_on_shutdown": "true", 2024-09-10T19:11:58.289 INFO:teuthology.orchestra.run.smithi183.stdout: "debug_auth": "1/5", 2024-09-10T19:11:58.289 INFO:teuthology.orchestra.run.smithi183.stdout: "debug_bdev": "1/3", 2024-09-10T19:11:58.289 INFO:teuthology.orchestra.run.smithi183.stdout: "debug_bluefs": "1/5", 2024-09-10T19:11:58.289 INFO:teuthology.orchestra.run.smithi183.stdout: "debug_bluestore": "1/5", 2024-09-10T19:11:58.289 INFO:teuthology.orchestra.run.smithi183.stdout: "debug_buffer": "0/1", 2024-09-10T19:11:58.289 INFO:teuthology.orchestra.run.smithi183.stdout: "debug_ceph_exporter": "1/5", 2024-09-10T19:11:58.289 INFO:teuthology.orchestra.run.smithi183.stdout: "debug_cephfs_mirror": "0/5", 2024-09-10T19:11:58.289 INFO:teuthology.orchestra.run.smithi183.stdout: "debug_cephsqlite": "0/5", 2024-09-10T19:11:58.289 INFO:teuthology.orchestra.run.smithi183.stdout: "debug_civetweb": "1/10", 2024-09-10T19:11:58.289 INFO:teuthology.orchestra.run.smithi183.stdout: "debug_client": "0/5", 2024-09-10T19:11:58.289 INFO:teuthology.orchestra.run.smithi183.stdout: "debug_compressor": "1/5", 2024-09-10T19:11:58.289 INFO:teuthology.orchestra.run.smithi183.stdout: "debug_context": "0/1", 2024-09-10T19:11:58.289 INFO:teuthology.orchestra.run.smithi183.stdout: "debug_crush": "1/1", 2024-09-10T19:11:58.290 INFO:teuthology.orchestra.run.smithi183.stdout: "debug_crypto": "1/5", 2024-09-10T19:11:58.290 INFO:teuthology.orchestra.run.smithi183.stdout: "debug_deliberately_leak_memory": "false", 2024-09-10T19:11:58.290 INFO:teuthology.orchestra.run.smithi183.stdout: "debug_disable_randomized_ping": "false", 2024-09-10T19:11:58.290 INFO:teuthology.orchestra.run.smithi183.stdout: "debug_dpdk": "1/5", 2024-09-10T19:11:58.290 INFO:teuthology.orchestra.run.smithi183.stdout: "debug_eventtrace": "1/5", 2024-09-10T19:11:58.290 INFO:teuthology.orchestra.run.smithi183.stdout: "debug_filer": "0/1", 2024-09-10T19:11:58.290 INFO:teuthology.orchestra.run.smithi183.stdout: "debug_filestore": "1/3", 2024-09-10T19:11:58.290 INFO:teuthology.orchestra.run.smithi183.stdout: "debug_finisher": "1/1", 2024-09-10T19:11:58.290 INFO:teuthology.orchestra.run.smithi183.stdout: "debug_fuse": "1/5", 2024-09-10T19:11:58.290 INFO:teuthology.orchestra.run.smithi183.stdout: "debug_heartbeat_testing_span": "0", 2024-09-10T19:11:58.290 INFO:teuthology.orchestra.run.smithi183.stdout: "debug_heartbeatmap": "1/5", 2024-09-10T19:11:58.290 INFO:teuthology.orchestra.run.smithi183.stdout: "debug_immutable_obj_cache": "0/5", 2024-09-10T19:11:58.290 INFO:teuthology.orchestra.run.smithi183.stdout: "debug_javaclient": "1/5", 2024-09-10T19:11:58.290 INFO:teuthology.orchestra.run.smithi183.stdout: "debug_journal": "1/3", 2024-09-10T19:11:58.290 INFO:teuthology.orchestra.run.smithi183.stdout: "debug_journaler": "0/5", 2024-09-10T19:11:58.291 INFO:teuthology.orchestra.run.smithi183.stdout: "debug_kstore": "1/5", 2024-09-10T19:11:58.291 INFO:teuthology.orchestra.run.smithi183.stdout: "debug_leveldb": "4/5", 2024-09-10T19:11:58.291 INFO:teuthology.orchestra.run.smithi183.stdout: "debug_lockdep": "0/1", 2024-09-10T19:11:58.291 INFO:teuthology.orchestra.run.smithi183.stdout: "debug_mclock": "1/5", 2024-09-10T19:11:58.291 INFO:teuthology.orchestra.run.smithi183.stdout: "debug_mds": "1/5", 2024-09-10T19:11:58.291 INFO:teuthology.orchestra.run.smithi183.stdout: "debug_mds_balancer": "1/5", 2024-09-10T19:11:58.291 INFO:teuthology.orchestra.run.smithi183.stdout: "debug_mds_locker": "1/5", 2024-09-10T19:11:58.291 INFO:teuthology.orchestra.run.smithi183.stdout: "debug_mds_log": "1/5", 2024-09-10T19:11:58.291 INFO:teuthology.orchestra.run.smithi183.stdout: "debug_mds_log_expire": "1/5", 2024-09-10T19:11:58.291 INFO:teuthology.orchestra.run.smithi183.stdout: "debug_mds_migrator": "1/5", 2024-09-10T19:11:58.291 INFO:teuthology.orchestra.run.smithi183.stdout: "debug_memdb": "4/5", 2024-09-10T19:11:58.291 INFO:teuthology.orchestra.run.smithi183.stdout: "debug_mgr": "2/5", 2024-09-10T19:11:58.292 INFO:teuthology.orchestra.run.smithi183.stdout: "debug_mgrc": "1/5", 2024-09-10T19:11:58.292 INFO:teuthology.orchestra.run.smithi183.stdout: "debug_mon": "20/20", 2024-09-10T19:11:58.292 INFO:teuthology.orchestra.run.smithi183.stdout: "debug_monc": "0/10", 2024-09-10T19:11:58.292 INFO:teuthology.orchestra.run.smithi183.stdout: "debug_ms": "1/1", 2024-09-10T19:11:58.292 INFO:teuthology.orchestra.run.smithi183.stdout: "debug_none": "0/5", 2024-09-10T19:11:58.292 INFO:teuthology.orchestra.run.smithi183.stdout: "debug_objclass": "0/5", 2024-09-10T19:11:58.292 INFO:teuthology.orchestra.run.smithi183.stdout: "debug_objectcacher": "0/5", 2024-09-10T19:11:58.292 INFO:teuthology.orchestra.run.smithi183.stdout: "debug_objecter": "0/1", 2024-09-10T19:11:58.292 INFO:teuthology.orchestra.run.smithi183.stdout: "debug_optracker": "0/5", 2024-09-10T19:11:58.293 INFO:teuthology.orchestra.run.smithi183.stdout: "debug_osd": "1/5", 2024-09-10T19:11:58.293 INFO:teuthology.orchestra.run.smithi183.stdout: "debug_paxos": "20/20", 2024-09-10T19:11:58.293 INFO:teuthology.orchestra.run.smithi183.stdout: "debug_perfcounter": "1/5", 2024-09-10T19:11:58.293 INFO:teuthology.orchestra.run.smithi183.stdout: "debug_prioritycache": "1/5", 2024-09-10T19:11:58.293 INFO:teuthology.orchestra.run.smithi183.stdout: "debug_rados": "0/5", 2024-09-10T19:11:58.293 INFO:teuthology.orchestra.run.smithi183.stdout: "debug_rbd": "0/5", 2024-09-10T19:11:58.293 INFO:teuthology.orchestra.run.smithi183.stdout: "debug_rbd_mirror": "0/5", 2024-09-10T19:11:58.293 INFO:teuthology.orchestra.run.smithi183.stdout: "debug_rbd_pwl": "0/5", 2024-09-10T19:11:58.293 INFO:teuthology.orchestra.run.smithi183.stdout: "debug_rbd_replay": "0/5", 2024-09-10T19:11:58.293 INFO:teuthology.orchestra.run.smithi183.stdout: "debug_refs": "0/0", 2024-09-10T19:11:58.293 INFO:teuthology.orchestra.run.smithi183.stdout: "debug_reserver": "1/1", 2024-09-10T19:11:58.293 INFO:teuthology.orchestra.run.smithi183.stdout: "debug_rgw": "1/5", 2024-09-10T19:11:58.293 INFO:teuthology.orchestra.run.smithi183.stdout: "debug_rgw_datacache": "1/5", 2024-09-10T19:11:58.293 INFO:teuthology.orchestra.run.smithi183.stdout: "debug_rgw_sync": "1/5", 2024-09-10T19:11:58.294 INFO:teuthology.orchestra.run.smithi183.stdout: "debug_rocksdb": "4/5", 2024-09-10T19:11:58.294 INFO:teuthology.orchestra.run.smithi183.stdout: "debug_seastore": "0/5", 2024-09-10T19:11:58.294 INFO:teuthology.orchestra.run.smithi183.stdout: "debug_seastore_cache": "0/5", 2024-09-10T19:11:58.294 INFO:teuthology.orchestra.run.smithi183.stdout: "debug_seastore_cleaner": "0/5", 2024-09-10T19:11:58.294 INFO:teuthology.orchestra.run.smithi183.stdout: "debug_seastore_device": "0/5", 2024-09-10T19:11:58.294 INFO:teuthology.orchestra.run.smithi183.stdout: "debug_seastore_journal": "0/5", 2024-09-10T19:11:58.294 INFO:teuthology.orchestra.run.smithi183.stdout: "debug_seastore_lba": "0/5", 2024-09-10T19:11:58.294 INFO:teuthology.orchestra.run.smithi183.stdout: "debug_seastore_odata": "0/5", 2024-09-10T19:11:58.294 INFO:teuthology.orchestra.run.smithi183.stdout: "debug_seastore_omap": "0/5", 2024-09-10T19:11:58.294 INFO:teuthology.orchestra.run.smithi183.stdout: "debug_seastore_onode": "0/5", 2024-09-10T19:11:58.294 INFO:teuthology.orchestra.run.smithi183.stdout: "debug_seastore_tm": "0/5", 2024-09-10T19:11:58.294 INFO:teuthology.orchestra.run.smithi183.stdout: "debug_striper": "0/1", 2024-09-10T19:11:58.294 INFO:teuthology.orchestra.run.smithi183.stdout: "debug_test": "0/5", 2024-09-10T19:11:58.294 INFO:teuthology.orchestra.run.smithi183.stdout: "debug_throttle": "1/1", 2024-09-10T19:11:58.295 INFO:teuthology.orchestra.run.smithi183.stdout: "debug_timer": "0/1", 2024-09-10T19:11:58.295 INFO:teuthology.orchestra.run.smithi183.stdout: "debug_tp": "0/5", 2024-09-10T19:11:58.295 INFO:teuthology.orchestra.run.smithi183.stdout: "defer_client_eviction_on_laggy_osds": "false", 2024-09-10T19:11:58.295 INFO:teuthology.orchestra.run.smithi183.stdout: "device_failure_prediction_mode": "none", 2024-09-10T19:11:58.295 INFO:teuthology.orchestra.run.smithi183.stdout: "enable_experimental_unrecoverable_data_corrupting_features": "", 2024-09-10T19:11:58.295 INFO:teuthology.orchestra.run.smithi183.stdout: "erasure_code_dir": "/usr/lib64/ceph/erasure-code", 2024-09-10T19:11:58.295 INFO:teuthology.orchestra.run.smithi183.stdout: "err_to_graylog": "false", 2024-09-10T19:11:58.295 INFO:teuthology.orchestra.run.smithi183.stdout: "err_to_journald": "false", 2024-09-10T19:11:58.295 INFO:teuthology.orchestra.run.smithi183.stdout: "err_to_stderr": "true", 2024-09-10T19:11:58.295 INFO:teuthology.orchestra.run.smithi183.stdout: "err_to_syslog": "false", 2024-09-10T19:11:58.295 INFO:teuthology.orchestra.run.smithi183.stdout: "event_tracing": "false", 2024-09-10T19:11:58.295 INFO:teuthology.orchestra.run.smithi183.stdout: "exporter_addr": "0.0.0.0", 2024-09-10T19:11:58.295 INFO:teuthology.orchestra.run.smithi183.stdout: "exporter_http_port": "9926", 2024-09-10T19:11:58.295 INFO:teuthology.orchestra.run.smithi183.stdout: "exporter_prio_limit": "5", 2024-09-10T19:11:58.296 INFO:teuthology.orchestra.run.smithi183.stdout: "exporter_sock_dir": "/var/run/ceph/", 2024-09-10T19:11:58.296 INFO:teuthology.orchestra.run.smithi183.stdout: "exporter_sort_metrics": "true", 2024-09-10T19:11:58.296 INFO:teuthology.orchestra.run.smithi183.stdout: "exporter_stats_period": "5", 2024-09-10T19:11:58.296 INFO:teuthology.orchestra.run.smithi183.stdout: "fake_statfs_for_testing": "0", 2024-09-10T19:11:58.296 INFO:teuthology.orchestra.run.smithi183.stdout: "fatal_signal_handlers": "true", 2024-09-10T19:11:58.296 INFO:teuthology.orchestra.run.smithi183.stdout: "filer_max_purge_ops": "10", 2024-09-10T19:11:58.296 INFO:teuthology.orchestra.run.smithi183.stdout: "filer_max_truncate_ops": "128", 2024-09-10T19:11:58.296 INFO:teuthology.orchestra.run.smithi183.stdout: "filestore_apply_finisher_threads": "1", 2024-09-10T19:11:58.296 INFO:teuthology.orchestra.run.smithi183.stdout: "filestore_blackhole": "false", 2024-09-10T19:11:58.297 INFO:teuthology.orchestra.run.smithi183.stdout: "filestore_btrfs_clone_range": "true", 2024-09-10T19:11:58.297 INFO:teuthology.orchestra.run.smithi183.stdout: "filestore_btrfs_snap": "true", 2024-09-10T19:11:58.297 INFO:teuthology.orchestra.run.smithi183.stdout: "filestore_caller_concurrency": "10", 2024-09-10T19:11:58.297 INFO:teuthology.orchestra.run.smithi183.stdout: "filestore_collect_device_partition_information": "true", 2024-09-10T19:11:58.297 INFO:teuthology.orchestra.run.smithi183.stdout: "filestore_commit_timeout": "600.000000", 2024-09-10T19:11:58.297 INFO:teuthology.orchestra.run.smithi183.stdout: "filestore_debug_inject_read_err": "false", 2024-09-10T19:11:58.297 INFO:teuthology.orchestra.run.smithi183.stdout: "filestore_debug_omap_check": "false", 2024-09-10T19:11:58.297 INFO:teuthology.orchestra.run.smithi183.stdout: "filestore_debug_random_read_err": "0.000000", 2024-09-10T19:11:58.297 INFO:teuthology.orchestra.run.smithi183.stdout: "filestore_debug_verify_split": "false", 2024-09-10T19:11:58.297 INFO:teuthology.orchestra.run.smithi183.stdout: "filestore_dump_file": "", 2024-09-10T19:11:58.297 INFO:teuthology.orchestra.run.smithi183.stdout: "filestore_expected_throughput_bytes": "209715200.000000", 2024-09-10T19:11:58.297 INFO:teuthology.orchestra.run.smithi183.stdout: "filestore_expected_throughput_ops": "200.000000", 2024-09-10T19:11:58.297 INFO:teuthology.orchestra.run.smithi183.stdout: "filestore_fadvise": "true", 2024-09-10T19:11:58.297 INFO:teuthology.orchestra.run.smithi183.stdout: "filestore_fail_eio": "true", 2024-09-10T19:11:58.298 INFO:teuthology.orchestra.run.smithi183.stdout: "filestore_fd_cache_shards": "16", 2024-09-10T19:11:58.298 INFO:teuthology.orchestra.run.smithi183.stdout: "filestore_fd_cache_size": "128", 2024-09-10T19:11:58.298 INFO:teuthology.orchestra.run.smithi183.stdout: "filestore_fiemap": "false", 2024-09-10T19:11:58.298 INFO:teuthology.orchestra.run.smithi183.stdout: "filestore_fiemap_threshold": "4096", 2024-09-10T19:11:58.298 INFO:teuthology.orchestra.run.smithi183.stdout: "filestore_fsync_flushes_journal_data": "false", 2024-09-10T19:11:58.298 INFO:teuthology.orchestra.run.smithi183.stdout: "filestore_index_retry_probability": "0.000000", 2024-09-10T19:11:58.298 INFO:teuthology.orchestra.run.smithi183.stdout: "filestore_inject_stall": "0", 2024-09-10T19:11:58.299 INFO:teuthology.orchestra.run.smithi183.stdout: "filestore_journal_parallel": "false", 2024-09-10T19:11:58.299 INFO:teuthology.orchestra.run.smithi183.stdout: "filestore_journal_trailing": "false", 2024-09-10T19:11:58.299 INFO:teuthology.orchestra.run.smithi183.stdout: "filestore_journal_writeahead": "false", 2024-09-10T19:11:58.299 INFO:teuthology.orchestra.run.smithi183.stdout: "filestore_kill_at": "0", 2024-09-10T19:11:58.299 INFO:teuthology.orchestra.run.smithi183.stdout: "filestore_max_alloc_hint_size": "1048576", 2024-09-10T19:11:58.299 INFO:teuthology.orchestra.run.smithi183.stdout: "filestore_max_inline_xattr_size": "0", 2024-09-10T19:11:58.299 INFO:teuthology.orchestra.run.smithi183.stdout: "filestore_max_inline_xattr_size_btrfs": "2048", 2024-09-10T19:11:58.299 INFO:teuthology.orchestra.run.smithi183.stdout: "filestore_max_inline_xattr_size_other": "512", 2024-09-10T19:11:58.299 INFO:teuthology.orchestra.run.smithi183.stdout: "filestore_max_inline_xattr_size_xfs": "65536", 2024-09-10T19:11:58.299 INFO:teuthology.orchestra.run.smithi183.stdout: "filestore_max_inline_xattrs": "0", 2024-09-10T19:11:58.300 INFO:teuthology.orchestra.run.smithi183.stdout: "filestore_max_inline_xattrs_btrfs": "10", 2024-09-10T19:11:58.300 INFO:teuthology.orchestra.run.smithi183.stdout: "filestore_max_inline_xattrs_other": "2", 2024-09-10T19:11:58.300 INFO:teuthology.orchestra.run.smithi183.stdout: "filestore_max_inline_xattrs_xfs": "10", 2024-09-10T19:11:58.300 INFO:teuthology.orchestra.run.smithi183.stdout: "filestore_max_sync_interval": "5.000000", 2024-09-10T19:11:58.300 INFO:teuthology.orchestra.run.smithi183.stdout: "filestore_max_xattr_value_size": "0", 2024-09-10T19:11:58.300 INFO:teuthology.orchestra.run.smithi183.stdout: "filestore_max_xattr_value_size_btrfs": "65536", 2024-09-10T19:11:58.300 INFO:teuthology.orchestra.run.smithi183.stdout: "filestore_max_xattr_value_size_other": "1024", 2024-09-10T19:11:58.300 INFO:teuthology.orchestra.run.smithi183.stdout: "filestore_max_xattr_value_size_xfs": "65536", 2024-09-10T19:11:58.300 INFO:teuthology.orchestra.run.smithi183.stdout: "filestore_merge_threshold": "-10", 2024-09-10T19:11:58.300 INFO:teuthology.orchestra.run.smithi183.stdout: "filestore_min_sync_interval": "0.010000", 2024-09-10T19:11:58.300 INFO:teuthology.orchestra.run.smithi183.stdout: "filestore_odsync_write": "false", 2024-09-10T19:11:58.300 INFO:teuthology.orchestra.run.smithi183.stdout: "filestore_omap_backend": "rocksdb", 2024-09-10T19:11:58.300 INFO:teuthology.orchestra.run.smithi183.stdout: "filestore_omap_backend_path": "", 2024-09-10T19:11:58.301 INFO:teuthology.orchestra.run.smithi183.stdout: "filestore_omap_header_cache_size": "1024", 2024-09-10T19:11:58.301 INFO:teuthology.orchestra.run.smithi183.stdout: "filestore_ondisk_finisher_threads": "1", 2024-09-10T19:11:58.301 INFO:teuthology.orchestra.run.smithi183.stdout: "filestore_op_thread_suicide_timeout": "180", 2024-09-10T19:11:58.301 INFO:teuthology.orchestra.run.smithi183.stdout: "filestore_op_thread_timeout": "60", 2024-09-10T19:11:58.301 INFO:teuthology.orchestra.run.smithi183.stdout: "filestore_op_threads": "2", 2024-09-10T19:11:58.301 INFO:teuthology.orchestra.run.smithi183.stdout: "filestore_punch_hole": "false", 2024-09-10T19:11:58.301 INFO:teuthology.orchestra.run.smithi183.stdout: "filestore_queue_high_delay_multiple": "0.000000", 2024-09-10T19:11:58.301 INFO:teuthology.orchestra.run.smithi183.stdout: "filestore_queue_high_delay_multiple_bytes": "0.000000", 2024-09-10T19:11:58.301 INFO:teuthology.orchestra.run.smithi183.stdout: "filestore_queue_high_delay_multiple_ops": "0.000000", 2024-09-10T19:11:58.301 INFO:teuthology.orchestra.run.smithi183.stdout: "filestore_queue_high_threshhold": "0.900000", 2024-09-10T19:11:58.301 INFO:teuthology.orchestra.run.smithi183.stdout: "filestore_queue_low_threshhold": "0.300000", 2024-09-10T19:11:58.301 INFO:teuthology.orchestra.run.smithi183.stdout: "filestore_queue_max_bytes": "104857600", 2024-09-10T19:11:58.301 INFO:teuthology.orchestra.run.smithi183.stdout: "filestore_queue_max_delay_multiple": "0.000000", 2024-09-10T19:11:58.301 INFO:teuthology.orchestra.run.smithi183.stdout: "filestore_queue_max_delay_multiple_bytes": "0.000000", 2024-09-10T19:11:58.302 INFO:teuthology.orchestra.run.smithi183.stdout: "filestore_queue_max_delay_multiple_ops": "0.000000", 2024-09-10T19:11:58.302 INFO:teuthology.orchestra.run.smithi183.stdout: "filestore_queue_max_ops": "50", 2024-09-10T19:11:58.302 INFO:teuthology.orchestra.run.smithi183.stdout: "filestore_rocksdb_options": "max_background_jobs=10,compaction_readahead_size=2097152,compression=kNoCompression", 2024-09-10T19:11:58.302 INFO:teuthology.orchestra.run.smithi183.stdout: "filestore_seek_data_hole": "false", 2024-09-10T19:11:58.302 INFO:teuthology.orchestra.run.smithi183.stdout: "filestore_sloppy_crc": "false", 2024-09-10T19:11:58.302 INFO:teuthology.orchestra.run.smithi183.stdout: "filestore_sloppy_crc_block_size": "65536", 2024-09-10T19:11:58.302 INFO:teuthology.orchestra.run.smithi183.stdout: "filestore_splice": "false", 2024-09-10T19:11:58.302 INFO:teuthology.orchestra.run.smithi183.stdout: "filestore_split_multiple": "2", 2024-09-10T19:11:58.302 INFO:teuthology.orchestra.run.smithi183.stdout: "filestore_split_rand_factor": "20", 2024-09-10T19:11:58.302 INFO:teuthology.orchestra.run.smithi183.stdout: "filestore_update_to": "1000", 2024-09-10T19:11:58.302 INFO:teuthology.orchestra.run.smithi183.stdout: "filestore_wbthrottle_btrfs_bytes_hard_limit": "419430400", 2024-09-10T19:11:58.302 INFO:teuthology.orchestra.run.smithi183.stdout: "filestore_wbthrottle_btrfs_bytes_start_flusher": "41943040", 2024-09-10T19:11:58.302 INFO:teuthology.orchestra.run.smithi183.stdout: "filestore_wbthrottle_btrfs_inodes_hard_limit": "5000", 2024-09-10T19:11:58.302 INFO:teuthology.orchestra.run.smithi183.stdout: "filestore_wbthrottle_btrfs_inodes_start_flusher": "500", 2024-09-10T19:11:58.303 INFO:teuthology.orchestra.run.smithi183.stdout: "filestore_wbthrottle_btrfs_ios_hard_limit": "5000", 2024-09-10T19:11:58.303 INFO:teuthology.orchestra.run.smithi183.stdout: "filestore_wbthrottle_btrfs_ios_start_flusher": "500", 2024-09-10T19:11:58.303 INFO:teuthology.orchestra.run.smithi183.stdout: "filestore_wbthrottle_enable": "true", 2024-09-10T19:11:58.303 INFO:teuthology.orchestra.run.smithi183.stdout: "filestore_wbthrottle_xfs_bytes_hard_limit": "419430400", 2024-09-10T19:11:58.303 INFO:teuthology.orchestra.run.smithi183.stdout: "filestore_wbthrottle_xfs_bytes_start_flusher": "41943040", 2024-09-10T19:11:58.303 INFO:teuthology.orchestra.run.smithi183.stdout: "filestore_wbthrottle_xfs_inodes_hard_limit": "5000", 2024-09-10T19:11:58.303 INFO:teuthology.orchestra.run.smithi183.stdout: "filestore_wbthrottle_xfs_inodes_start_flusher": "500", 2024-09-10T19:11:58.303 INFO:teuthology.orchestra.run.smithi183.stdout: "filestore_wbthrottle_xfs_ios_hard_limit": "5000", 2024-09-10T19:11:58.303 INFO:teuthology.orchestra.run.smithi183.stdout: "filestore_wbthrottle_xfs_ios_start_flusher": "500", 2024-09-10T19:11:58.303 INFO:teuthology.orchestra.run.smithi183.stdout: "filestore_xfs_extsize": "false", 2024-09-10T19:11:58.303 INFO:teuthology.orchestra.run.smithi183.stdout: "filestore_zfs_snap": "false", 2024-09-10T19:11:58.303 INFO:teuthology.orchestra.run.smithi183.stdout: "fio_dir": "/tmp/fio", 2024-09-10T19:11:58.303 INFO:teuthology.orchestra.run.smithi183.stdout: "fsid": "320f6df8-6fa7-11ef-bcea-c7b262605968", 2024-09-10T19:11:58.303 INFO:teuthology.orchestra.run.smithi183.stdout: "fuse_allow_other": "true", 2024-09-10T19:11:58.304 INFO:teuthology.orchestra.run.smithi183.stdout: "fuse_atomic_o_trunc": "true", 2024-09-10T19:11:58.304 INFO:teuthology.orchestra.run.smithi183.stdout: "fuse_big_writes": "true", 2024-09-10T19:11:58.304 INFO:teuthology.orchestra.run.smithi183.stdout: "fuse_debug": "false", 2024-09-10T19:11:58.304 INFO:teuthology.orchestra.run.smithi183.stdout: "fuse_default_permissions": "false", 2024-09-10T19:11:58.304 INFO:teuthology.orchestra.run.smithi183.stdout: "fuse_disable_pagecache": "false", 2024-09-10T19:11:58.304 INFO:teuthology.orchestra.run.smithi183.stdout: "fuse_max_write": "0", 2024-09-10T19:11:58.304 INFO:teuthology.orchestra.run.smithi183.stdout: "fuse_multithreaded": "true", 2024-09-10T19:11:58.304 INFO:teuthology.orchestra.run.smithi183.stdout: "fuse_require_active_mds": "true", 2024-09-10T19:11:58.304 INFO:teuthology.orchestra.run.smithi183.stdout: "fuse_set_user_groups": "true", 2024-09-10T19:11:58.304 INFO:teuthology.orchestra.run.smithi183.stdout: "fuse_splice_move": "true", 2024-09-10T19:11:58.304 INFO:teuthology.orchestra.run.smithi183.stdout: "fuse_splice_read": "true", 2024-09-10T19:11:58.304 INFO:teuthology.orchestra.run.smithi183.stdout: "fuse_splice_write": "true", 2024-09-10T19:11:58.304 INFO:teuthology.orchestra.run.smithi183.stdout: "fuse_syncfs_on_mksnap": "true", 2024-09-10T19:11:58.305 INFO:teuthology.orchestra.run.smithi183.stdout: "fuse_use_invalidate_cb": "true", 2024-09-10T19:11:58.305 INFO:teuthology.orchestra.run.smithi183.stdout: "gss_ktab_client_file": "/var/lib/ceph/mon.a/gss_client_mon.a.ktab", 2024-09-10T19:11:58.305 INFO:teuthology.orchestra.run.smithi183.stdout: "gss_target_name": "ceph", 2024-09-10T19:11:58.305 INFO:teuthology.orchestra.run.smithi183.stdout: "heartbeat_file": "", 2024-09-10T19:11:58.305 INFO:teuthology.orchestra.run.smithi183.stdout: "heartbeat_inject_failure": "0", 2024-09-10T19:11:58.305 INFO:teuthology.orchestra.run.smithi183.stdout: "heartbeat_interval": "5", 2024-09-10T19:11:58.305 INFO:teuthology.orchestra.run.smithi183.stdout: "host": "smithi183", 2024-09-10T19:11:58.305 INFO:teuthology.orchestra.run.smithi183.stdout: "immutable_object_cache_client_dedicated_thread_num": "2", 2024-09-10T19:11:58.305 INFO:teuthology.orchestra.run.smithi183.stdout: "immutable_object_cache_max_inflight_ops": "128", 2024-09-10T19:11:58.305 INFO:teuthology.orchestra.run.smithi183.stdout: "immutable_object_cache_max_size": "1073741824", 2024-09-10T19:11:58.305 INFO:teuthology.orchestra.run.smithi183.stdout: "immutable_object_cache_path": "/tmp/ceph_immutable_object_cache", 2024-09-10T19:11:58.305 INFO:teuthology.orchestra.run.smithi183.stdout: "immutable_object_cache_qos_bps_burst": "0", 2024-09-10T19:11:58.305 INFO:teuthology.orchestra.run.smithi183.stdout: "immutable_object_cache_qos_bps_burst_seconds": "1", 2024-09-10T19:11:58.305 INFO:teuthology.orchestra.run.smithi183.stdout: "immutable_object_cache_qos_bps_limit": "0", 2024-09-10T19:11:58.305 INFO:teuthology.orchestra.run.smithi183.stdout: "immutable_object_cache_qos_iops_burst": "0", 2024-09-10T19:11:58.306 INFO:teuthology.orchestra.run.smithi183.stdout: "immutable_object_cache_qos_iops_burst_seconds": "1", 2024-09-10T19:11:58.306 INFO:teuthology.orchestra.run.smithi183.stdout: "immutable_object_cache_qos_iops_limit": "0", 2024-09-10T19:11:58.306 INFO:teuthology.orchestra.run.smithi183.stdout: "immutable_object_cache_qos_schedule_tick_min": "50", 2024-09-10T19:11:58.306 INFO:teuthology.orchestra.run.smithi183.stdout: "immutable_object_cache_sock": "/var/run/ceph/immutable_object_cache_sock", 2024-09-10T19:11:58.306 INFO:teuthology.orchestra.run.smithi183.stdout: "immutable_object_cache_watermark": "0.900000", 2024-09-10T19:11:58.306 INFO:teuthology.orchestra.run.smithi183.stdout: "inject_early_sigterm": "false", 2024-09-10T19:11:58.306 INFO:teuthology.orchestra.run.smithi183.stdout: "jaeger_tracing_enable": "false", 2024-09-10T19:11:58.306 INFO:teuthology.orchestra.run.smithi183.stdout: "journal_aio": "true", 2024-09-10T19:11:58.306 INFO:teuthology.orchestra.run.smithi183.stdout: "journal_align_min_size": "65536", 2024-09-10T19:11:58.306 INFO:teuthology.orchestra.run.smithi183.stdout: "journal_block_align": "true", 2024-09-10T19:11:58.306 INFO:teuthology.orchestra.run.smithi183.stdout: "journal_block_size": "4096", 2024-09-10T19:11:58.306 INFO:teuthology.orchestra.run.smithi183.stdout: "journal_dio": "true", 2024-09-10T19:11:58.306 INFO:teuthology.orchestra.run.smithi183.stdout: "journal_discard": "false", 2024-09-10T19:11:58.306 INFO:teuthology.orchestra.run.smithi183.stdout: "journal_force_aio": "false", 2024-09-10T19:11:58.307 INFO:teuthology.orchestra.run.smithi183.stdout: "journal_ignore_corruption": "false", 2024-09-10T19:11:58.307 INFO:teuthology.orchestra.run.smithi183.stdout: "journal_max_write_bytes": "10485760", 2024-09-10T19:11:58.307 INFO:teuthology.orchestra.run.smithi183.stdout: "journal_max_write_entries": "100", 2024-09-10T19:11:58.307 INFO:teuthology.orchestra.run.smithi183.stdout: "journal_replay_from": "0", 2024-09-10T19:11:58.307 INFO:teuthology.orchestra.run.smithi183.stdout: "journal_throttle_high_multiple": "0.000000", 2024-09-10T19:11:58.307 INFO:teuthology.orchestra.run.smithi183.stdout: "journal_throttle_high_threshhold": "0.900000", 2024-09-10T19:11:58.307 INFO:teuthology.orchestra.run.smithi183.stdout: "journal_throttle_low_threshhold": "0.600000", 2024-09-10T19:11:58.307 INFO:teuthology.orchestra.run.smithi183.stdout: "journal_throttle_max_multiple": "0.000000", 2024-09-10T19:11:58.307 INFO:teuthology.orchestra.run.smithi183.stdout: "journal_write_header_frequency": "0", 2024-09-10T19:11:58.307 INFO:teuthology.orchestra.run.smithi183.stdout: "journal_zero_on_create": "false", 2024-09-10T19:11:58.307 INFO:teuthology.orchestra.run.smithi183.stdout: "journaler_prefetch_periods": "10", 2024-09-10T19:11:58.307 INFO:teuthology.orchestra.run.smithi183.stdout: "journaler_prezero_periods": "5", 2024-09-10T19:11:58.307 INFO:teuthology.orchestra.run.smithi183.stdout: "journaler_write_head_interval": "15", 2024-09-10T19:11:58.307 INFO:teuthology.orchestra.run.smithi183.stdout: "key": "", 2024-09-10T19:11:58.308 INFO:teuthology.orchestra.run.smithi183.stdout: "keyfile": "", 2024-09-10T19:11:58.308 INFO:teuthology.orchestra.run.smithi183.stdout: "keyring": "/var/lib/ceph/mon/ceph-a/keyring", 2024-09-10T19:11:58.308 INFO:teuthology.orchestra.run.smithi183.stdout: "kstore_backend": "rocksdb", 2024-09-10T19:11:58.308 INFO:teuthology.orchestra.run.smithi183.stdout: "kstore_default_stripe_size": "65536", 2024-09-10T19:11:58.308 INFO:teuthology.orchestra.run.smithi183.stdout: "kstore_fsck_on_mount": "false", 2024-09-10T19:11:58.308 INFO:teuthology.orchestra.run.smithi183.stdout: "kstore_fsck_on_mount_deep": "true", 2024-09-10T19:11:58.308 INFO:teuthology.orchestra.run.smithi183.stdout: "kstore_max_bytes": "67108864", 2024-09-10T19:11:58.308 INFO:teuthology.orchestra.run.smithi183.stdout: "kstore_max_ops": "512", 2024-09-10T19:11:58.308 INFO:teuthology.orchestra.run.smithi183.stdout: "kstore_nid_prealloc": "1024", 2024-09-10T19:11:58.308 INFO:teuthology.orchestra.run.smithi183.stdout: "kstore_onode_map_size": "1024", 2024-09-10T19:11:58.308 INFO:teuthology.orchestra.run.smithi183.stdout: "kstore_rocksdb_options": "compression=kNoCompression", 2024-09-10T19:11:58.308 INFO:teuthology.orchestra.run.smithi183.stdout: "kstore_sync_submit_transaction": "false", 2024-09-10T19:11:58.308 INFO:teuthology.orchestra.run.smithi183.stdout: "kstore_sync_transaction": "false", 2024-09-10T19:11:58.308 INFO:teuthology.orchestra.run.smithi183.stdout: "leveldb_block_size": "65536", 2024-09-10T19:11:58.309 INFO:teuthology.orchestra.run.smithi183.stdout: "leveldb_bloom_size": "0", 2024-09-10T19:11:58.309 INFO:teuthology.orchestra.run.smithi183.stdout: "leveldb_cache_size": "536870912", 2024-09-10T19:11:58.309 INFO:teuthology.orchestra.run.smithi183.stdout: "leveldb_compact_on_mount": "false", 2024-09-10T19:11:58.309 INFO:teuthology.orchestra.run.smithi183.stdout: "leveldb_compression": "false", 2024-09-10T19:11:58.309 INFO:teuthology.orchestra.run.smithi183.stdout: "leveldb_log": "", 2024-09-10T19:11:58.309 INFO:teuthology.orchestra.run.smithi183.stdout: "leveldb_log_to_ceph_log": "true", 2024-09-10T19:11:58.309 INFO:teuthology.orchestra.run.smithi183.stdout: "leveldb_max_open_files": "0", 2024-09-10T19:11:58.309 INFO:teuthology.orchestra.run.smithi183.stdout: "leveldb_paranoid": "false", 2024-09-10T19:11:58.309 INFO:teuthology.orchestra.run.smithi183.stdout: "leveldb_write_buffer_size": "33554432", 2024-09-10T19:11:58.309 INFO:teuthology.orchestra.run.smithi183.stdout: "librados_thread_count": "2", 2024-09-10T19:11:58.309 INFO:teuthology.orchestra.run.smithi183.stdout: "lockdep": "false", 2024-09-10T19:11:58.309 INFO:teuthology.orchestra.run.smithi183.stdout: "lockdep_force_backtrace": "false", 2024-09-10T19:11:58.309 INFO:teuthology.orchestra.run.smithi183.stdout: "log_coarse_timestamps": "true", 2024-09-10T19:11:58.310 INFO:teuthology.orchestra.run.smithi183.stdout: "log_file": "/var/log/ceph/ceph-mon.a.log", 2024-09-10T19:11:58.310 INFO:teuthology.orchestra.run.smithi183.stdout: "log_flush_on_exit": "false", 2024-09-10T19:11:58.310 INFO:teuthology.orchestra.run.smithi183.stdout: "log_graylog_host": "127.0.0.1", 2024-09-10T19:11:58.310 INFO:teuthology.orchestra.run.smithi183.stdout: "log_graylog_port": "12201", 2024-09-10T19:11:58.310 INFO:teuthology.orchestra.run.smithi183.stdout: "log_max_new": "1000", 2024-09-10T19:11:58.310 INFO:teuthology.orchestra.run.smithi183.stdout: "log_max_recent": "10000", 2024-09-10T19:11:58.310 INFO:teuthology.orchestra.run.smithi183.stdout: "log_stderr_prefix": "", 2024-09-10T19:11:58.310 INFO:teuthology.orchestra.run.smithi183.stdout: "log_stop_at_utilization": "0.970000", 2024-09-10T19:11:58.310 INFO:teuthology.orchestra.run.smithi183.stdout: "log_to_file": "true", 2024-09-10T19:11:58.310 INFO:teuthology.orchestra.run.smithi183.stdout: "log_to_graylog": "false", 2024-09-10T19:11:58.310 INFO:teuthology.orchestra.run.smithi183.stdout: "log_to_journald": "false", 2024-09-10T19:11:58.310 INFO:teuthology.orchestra.run.smithi183.stdout: "log_to_stderr": "false", 2024-09-10T19:11:58.310 INFO:teuthology.orchestra.run.smithi183.stdout: "log_to_syslog": "false", 2024-09-10T19:11:58.310 INFO:teuthology.orchestra.run.smithi183.stdout: "max_rotating_auth_attempts": "10", 2024-09-10T19:11:58.311 INFO:teuthology.orchestra.run.smithi183.stdout: "mds_abort_on_newly_corrupt_dentry": "true", 2024-09-10T19:11:58.311 INFO:teuthology.orchestra.run.smithi183.stdout: "mds_action_on_write_error": "1", 2024-09-10T19:11:58.311 INFO:teuthology.orchestra.run.smithi183.stdout: "mds_alternate_name_max": "8192", 2024-09-10T19:11:58.311 INFO:teuthology.orchestra.run.smithi183.stdout: "mds_asio_thread_count": "2", 2024-09-10T19:11:58.311 INFO:teuthology.orchestra.run.smithi183.stdout: "mds_bal_export_pin": "true", 2024-09-10T19:11:58.311 INFO:teuthology.orchestra.run.smithi183.stdout: "mds_bal_fragment_dirs": "true", 2024-09-10T19:11:58.311 INFO:teuthology.orchestra.run.smithi183.stdout: "mds_bal_fragment_fast_factor": "1.500000", 2024-09-10T19:11:58.311 INFO:teuthology.orchestra.run.smithi183.stdout: "mds_bal_fragment_interval": "5", 2024-09-10T19:11:58.311 INFO:teuthology.orchestra.run.smithi183.stdout: "mds_bal_fragment_size_max": "100000", 2024-09-10T19:11:58.311 INFO:teuthology.orchestra.run.smithi183.stdout: "mds_bal_idle_threshold": "0.000000", 2024-09-10T19:11:58.311 INFO:teuthology.orchestra.run.smithi183.stdout: "mds_bal_interval": "10", 2024-09-10T19:11:58.311 INFO:teuthology.orchestra.run.smithi183.stdout: "mds_bal_max": "-1", 2024-09-10T19:11:58.311 INFO:teuthology.orchestra.run.smithi183.stdout: "mds_bal_max_until": "-1", 2024-09-10T19:11:58.312 INFO:teuthology.orchestra.run.smithi183.stdout: "mds_bal_merge_size": "50", 2024-09-10T19:11:58.312 INFO:teuthology.orchestra.run.smithi183.stdout: "mds_bal_midchunk": "0.300000", 2024-09-10T19:11:58.312 INFO:teuthology.orchestra.run.smithi183.stdout: "mds_bal_min_rebalance": "0.100000", 2024-09-10T19:11:58.312 INFO:teuthology.orchestra.run.smithi183.stdout: "mds_bal_min_start": "0.200000", 2024-09-10T19:11:58.312 INFO:teuthology.orchestra.run.smithi183.stdout: "mds_bal_minchunk": "0.001000", 2024-09-10T19:11:58.312 INFO:teuthology.orchestra.run.smithi183.stdout: "mds_bal_mode": "0", 2024-09-10T19:11:58.312 INFO:teuthology.orchestra.run.smithi183.stdout: "mds_bal_need_max": "1.200000", 2024-09-10T19:11:58.312 INFO:teuthology.orchestra.run.smithi183.stdout: "mds_bal_need_min": "0.800000", 2024-09-10T19:11:58.312 INFO:teuthology.orchestra.run.smithi183.stdout: "mds_bal_replicate_threshold": "8000.000000", 2024-09-10T19:11:58.312 INFO:teuthology.orchestra.run.smithi183.stdout: "mds_bal_sample_interval": "3.000000", 2024-09-10T19:11:58.313 INFO:teuthology.orchestra.run.smithi183.stdout: "mds_bal_split_bits": "3", 2024-09-10T19:11:58.313 INFO:teuthology.orchestra.run.smithi183.stdout: "mds_bal_split_rd": "25000.000000", 2024-09-10T19:11:58.313 INFO:teuthology.orchestra.run.smithi183.stdout: "mds_bal_split_size": "10000", 2024-09-10T19:11:58.313 INFO:teuthology.orchestra.run.smithi183.stdout: "mds_bal_split_wr": "10000.000000", 2024-09-10T19:11:58.313 INFO:teuthology.orchestra.run.smithi183.stdout: "mds_bal_target_decay": "10.000000", 2024-09-10T19:11:58.313 INFO:teuthology.orchestra.run.smithi183.stdout: "mds_bal_unreplicate_threshold": "0.000000", 2024-09-10T19:11:58.313 INFO:teuthology.orchestra.run.smithi183.stdout: "mds_beacon_grace": "15.000000", 2024-09-10T19:11:58.313 INFO:teuthology.orchestra.run.smithi183.stdout: "mds_beacon_interval": "4.000000", 2024-09-10T19:11:58.313 INFO:teuthology.orchestra.run.smithi183.stdout: "mds_beacon_mon_down_grace": "60", 2024-09-10T19:11:58.313 INFO:teuthology.orchestra.run.smithi183.stdout: "mds_cache_memory_limit": "4294967296", 2024-09-10T19:11:58.313 INFO:teuthology.orchestra.run.smithi183.stdout: "mds_cache_mid": "0.700000", 2024-09-10T19:11:58.313 INFO:teuthology.orchestra.run.smithi183.stdout: "mds_cache_release_free_interval": "10", 2024-09-10T19:11:58.313 INFO:teuthology.orchestra.run.smithi183.stdout: "mds_cache_reservation": "0.050000", 2024-09-10T19:11:58.313 INFO:teuthology.orchestra.run.smithi183.stdout: "mds_cache_trim_decay_rate": "1.000000", 2024-09-10T19:11:58.314 INFO:teuthology.orchestra.run.smithi183.stdout: "mds_cache_trim_interval": "1", 2024-09-10T19:11:58.314 INFO:teuthology.orchestra.run.smithi183.stdout: "mds_cache_trim_threshold": "262144", 2024-09-10T19:11:58.314 INFO:teuthology.orchestra.run.smithi183.stdout: "mds_cap_acquisition_throttle_retry_request_timeout": "0.500000", 2024-09-10T19:11:58.314 INFO:teuthology.orchestra.run.smithi183.stdout: "mds_cap_revoke_eviction_timeout": "0.000000", 2024-09-10T19:11:58.314 INFO:teuthology.orchestra.run.smithi183.stdout: "mds_client_delegate_inos_pct": "50", 2024-09-10T19:11:58.314 INFO:teuthology.orchestra.run.smithi183.stdout: "mds_client_prealloc_inos": "1000", 2024-09-10T19:11:58.314 INFO:teuthology.orchestra.run.smithi183.stdout: "mds_client_writeable_range_max_inc_objs": "1024", 2024-09-10T19:11:58.314 INFO:teuthology.orchestra.run.smithi183.stdout: "mds_connect_bootstrapping": "false", 2024-09-10T19:11:58.314 INFO:teuthology.orchestra.run.smithi183.stdout: "mds_damage_table_max_entries": "10000", 2024-09-10T19:11:58.314 INFO:teuthology.orchestra.run.smithi183.stdout: "mds_data": "/var/lib/ceph/mds/ceph-a", 2024-09-10T19:11:58.314 INFO:teuthology.orchestra.run.smithi183.stdout: "mds_debug_auth_pins": "false", 2024-09-10T19:11:58.314 INFO:teuthology.orchestra.run.smithi183.stdout: "mds_debug_frag": "false", 2024-09-10T19:11:58.314 INFO:teuthology.orchestra.run.smithi183.stdout: "mds_debug_scatterstat": "false", 2024-09-10T19:11:58.314 INFO:teuthology.orchestra.run.smithi183.stdout: "mds_debug_subtrees": "false", 2024-09-10T19:11:58.314 INFO:teuthology.orchestra.run.smithi183.stdout: "mds_decay_halflife": "5.000000", 2024-09-10T19:11:58.315 INFO:teuthology.orchestra.run.smithi183.stdout: "mds_default_dir_hash": "2", 2024-09-10T19:11:58.315 INFO:teuthology.orchestra.run.smithi183.stdout: "mds_defer_session_stale": "true", 2024-09-10T19:11:58.315 INFO:teuthology.orchestra.run.smithi183.stdout: "mds_deny_all_reconnect": "false", 2024-09-10T19:11:58.315 INFO:teuthology.orchestra.run.smithi183.stdout: "mds_dir_keys_per_op": "16384", 2024-09-10T19:11:58.315 INFO:teuthology.orchestra.run.smithi183.stdout: "mds_dir_max_commit_size": "10", 2024-09-10T19:11:58.315 INFO:teuthology.orchestra.run.smithi183.stdout: "mds_dir_max_entries": "0", 2024-09-10T19:11:58.315 INFO:teuthology.orchestra.run.smithi183.stdout: "mds_dirstat_min_interval": "1.000000", 2024-09-10T19:11:58.315 INFO:teuthology.orchestra.run.smithi183.stdout: "mds_dump_cache_after_rejoin": "false", 2024-09-10T19:11:58.315 INFO:teuthology.orchestra.run.smithi183.stdout: "mds_dump_cache_on_map": "false", 2024-09-10T19:11:58.315 INFO:teuthology.orchestra.run.smithi183.stdout: "mds_dump_cache_threshold_file": "0", 2024-09-10T19:11:58.315 INFO:teuthology.orchestra.run.smithi183.stdout: "mds_dump_cache_threshold_formatter": "1073741824", 2024-09-10T19:11:58.315 INFO:teuthology.orchestra.run.smithi183.stdout: "mds_early_reply": "true", 2024-09-10T19:11:58.315 INFO:teuthology.orchestra.run.smithi183.stdout: "mds_enable_op_tracker": "true", 2024-09-10T19:11:58.315 INFO:teuthology.orchestra.run.smithi183.stdout: "mds_enforce_unique_name": "true", 2024-09-10T19:11:58.316 INFO:teuthology.orchestra.run.smithi183.stdout: "mds_export_ephemeral_distributed": "true", 2024-09-10T19:11:58.316 INFO:teuthology.orchestra.run.smithi183.stdout: "mds_export_ephemeral_distributed_factor": "2.000000", 2024-09-10T19:11:58.316 INFO:teuthology.orchestra.run.smithi183.stdout: "mds_export_ephemeral_random": "true", 2024-09-10T19:11:58.316 INFO:teuthology.orchestra.run.smithi183.stdout: "mds_export_ephemeral_random_max": "0.010000", 2024-09-10T19:11:58.316 INFO:teuthology.orchestra.run.smithi183.stdout: "mds_forward_all_requests_to_auth": "false", 2024-09-10T19:11:58.316 INFO:teuthology.orchestra.run.smithi183.stdout: "mds_freeze_tree_timeout": "30.000000", 2024-09-10T19:11:58.316 INFO:teuthology.orchestra.run.smithi183.stdout: "mds_go_bad_corrupt_dentry": "true", 2024-09-10T19:11:58.316 INFO:teuthology.orchestra.run.smithi183.stdout: "mds_hack_allow_loading_invalid_metadata": "false", 2024-09-10T19:11:58.316 INFO:teuthology.orchestra.run.smithi183.stdout: "mds_health_cache_threshold": "1.500000", 2024-09-10T19:11:58.316 INFO:teuthology.orchestra.run.smithi183.stdout: "mds_health_summarize_threshold": "10", 2024-09-10T19:11:58.316 INFO:teuthology.orchestra.run.smithi183.stdout: "mds_heartbeat_grace": "15.000000", 2024-09-10T19:11:58.316 INFO:teuthology.orchestra.run.smithi183.stdout: "mds_heartbeat_reset_grace": "1000", 2024-09-10T19:11:58.316 INFO:teuthology.orchestra.run.smithi183.stdout: "mds_inject_health_dummy": "false", 2024-09-10T19:11:58.316 INFO:teuthology.orchestra.run.smithi183.stdout: "mds_inject_journal_corrupt_dentry_first": "0.000000", 2024-09-10T19:11:58.317 INFO:teuthology.orchestra.run.smithi183.stdout: "mds_inject_migrator_session_race": "false", 2024-09-10T19:11:58.317 INFO:teuthology.orchestra.run.smithi183.stdout: "mds_inject_rename_corrupt_dentry_first": "0.000000", 2024-09-10T19:11:58.317 INFO:teuthology.orchestra.run.smithi183.stdout: "mds_inject_skip_replaying_inotable": "false", 2024-09-10T19:11:58.317 INFO:teuthology.orchestra.run.smithi183.stdout: "mds_inject_traceless_reply_probability": "0.000000", 2024-09-10T19:11:58.317 INFO:teuthology.orchestra.run.smithi183.stdout: "mds_join_fs": "", 2024-09-10T19:11:58.317 INFO:teuthology.orchestra.run.smithi183.stdout: "mds_journal_format": "1", 2024-09-10T19:11:58.317 INFO:teuthology.orchestra.run.smithi183.stdout: "mds_kill_create_at": "0", 2024-09-10T19:11:58.317 INFO:teuthology.orchestra.run.smithi183.stdout: "mds_kill_export_at": "0", 2024-09-10T19:11:58.317 INFO:teuthology.orchestra.run.smithi183.stdout: "mds_kill_import_at": "0", 2024-09-10T19:11:58.317 INFO:teuthology.orchestra.run.smithi183.stdout: "mds_kill_journal_at": "0", 2024-09-10T19:11:58.317 INFO:teuthology.orchestra.run.smithi183.stdout: "mds_kill_journal_expire_at": "0", 2024-09-10T19:11:58.317 INFO:teuthology.orchestra.run.smithi183.stdout: "mds_kill_journal_replay_at": "0", 2024-09-10T19:11:58.317 INFO:teuthology.orchestra.run.smithi183.stdout: "mds_kill_link_at": "0", 2024-09-10T19:11:58.318 INFO:teuthology.orchestra.run.smithi183.stdout: "mds_kill_mdstable_at": "0", 2024-09-10T19:11:58.318 INFO:teuthology.orchestra.run.smithi183.stdout: "mds_kill_openc_at": "0", 2024-09-10T19:11:58.318 INFO:teuthology.orchestra.run.smithi183.stdout: "mds_kill_rename_at": "0", 2024-09-10T19:11:58.318 INFO:teuthology.orchestra.run.smithi183.stdout: "mds_kill_skip_replaying_inotable": "false", 2024-09-10T19:11:58.318 INFO:teuthology.orchestra.run.smithi183.stdout: "mds_log_events_per_segment": "1024", 2024-09-10T19:11:58.318 INFO:teuthology.orchestra.run.smithi183.stdout: "mds_log_max_events": "-1", 2024-09-10T19:11:58.318 INFO:teuthology.orchestra.run.smithi183.stdout: "mds_log_max_segments": "128", 2024-09-10T19:11:58.318 INFO:teuthology.orchestra.run.smithi183.stdout: "mds_log_pause": "false", 2024-09-10T19:11:58.318 INFO:teuthology.orchestra.run.smithi183.stdout: "mds_log_segment_size": "0", 2024-09-10T19:11:58.318 INFO:teuthology.orchestra.run.smithi183.stdout: "mds_log_skip_corrupt_events": "false", 2024-09-10T19:11:58.318 INFO:teuthology.orchestra.run.smithi183.stdout: "mds_log_warn_factor": "2.000000", 2024-09-10T19:11:58.318 INFO:teuthology.orchestra.run.smithi183.stdout: "mds_max_caps_per_client": "1048576", 2024-09-10T19:11:58.318 INFO:teuthology.orchestra.run.smithi183.stdout: "mds_max_completed_flushes": "100000", 2024-09-10T19:11:58.318 INFO:teuthology.orchestra.run.smithi183.stdout: "mds_max_completed_requests": "100000", 2024-09-10T19:11:58.318 INFO:teuthology.orchestra.run.smithi183.stdout: "mds_max_export_size": "20971520", 2024-09-10T19:11:58.319 INFO:teuthology.orchestra.run.smithi183.stdout: "mds_max_file_recover": "32", 2024-09-10T19:11:58.319 INFO:teuthology.orchestra.run.smithi183.stdout: "mds_max_purge_files": "64", 2024-09-10T19:11:58.319 INFO:teuthology.orchestra.run.smithi183.stdout: "mds_max_purge_ops": "8192", 2024-09-10T19:11:58.319 INFO:teuthology.orchestra.run.smithi183.stdout: "mds_max_purge_ops_per_pg": "0.500000", 2024-09-10T19:11:58.319 INFO:teuthology.orchestra.run.smithi183.stdout: "mds_max_scrub_ops_in_progress": "5", 2024-09-10T19:11:58.319 INFO:teuthology.orchestra.run.smithi183.stdout: "mds_max_snaps_per_dir": "100", 2024-09-10T19:11:58.319 INFO:teuthology.orchestra.run.smithi183.stdout: "mds_max_xattr_pairs_size": "65536", 2024-09-10T19:11:58.319 INFO:teuthology.orchestra.run.smithi183.stdout: "mds_metrics_update_interval": "2", 2024-09-10T19:11:58.319 INFO:teuthology.orchestra.run.smithi183.stdout: "mds_min_caps_per_client": "100", 2024-09-10T19:11:58.319 INFO:teuthology.orchestra.run.smithi183.stdout: "mds_min_caps_working_set": "10000", 2024-09-10T19:11:58.319 INFO:teuthology.orchestra.run.smithi183.stdout: "mds_mon_shutdown_timeout": "5.000000", 2024-09-10T19:11:58.319 INFO:teuthology.orchestra.run.smithi183.stdout: "mds_numa_node": "-1", 2024-09-10T19:11:58.319 INFO:teuthology.orchestra.run.smithi183.stdout: "mds_oft_prefetch_dirfrags": "false", 2024-09-10T19:11:58.319 INFO:teuthology.orchestra.run.smithi183.stdout: "mds_op_complaint_time": "30.000000", 2024-09-10T19:11:58.320 INFO:teuthology.orchestra.run.smithi183.stdout: "mds_op_history_duration": "600", 2024-09-10T19:11:58.320 INFO:teuthology.orchestra.run.smithi183.stdout: "mds_op_history_size": "20", 2024-09-10T19:11:58.320 INFO:teuthology.orchestra.run.smithi183.stdout: "mds_op_history_slow_op_size": "20", 2024-09-10T19:11:58.320 INFO:teuthology.orchestra.run.smithi183.stdout: "mds_op_history_slow_op_threshold": "10", 2024-09-10T19:11:58.320 INFO:teuthology.orchestra.run.smithi183.stdout: "mds_op_log_threshold": "5", 2024-09-10T19:11:58.320 INFO:teuthology.orchestra.run.smithi183.stdout: "mds_ping_grace": "15", 2024-09-10T19:11:58.320 INFO:teuthology.orchestra.run.smithi183.stdout: "mds_ping_interval": "5", 2024-09-10T19:11:58.320 INFO:teuthology.orchestra.run.smithi183.stdout: "mds_purge_queue_busy_flush_period": "1.000000", 2024-09-10T19:11:58.320 INFO:teuthology.orchestra.run.smithi183.stdout: "mds_recall_global_max_decay_threshold": "131072", 2024-09-10T19:11:58.320 INFO:teuthology.orchestra.run.smithi183.stdout: "mds_recall_max_caps": "30000", 2024-09-10T19:11:58.321 INFO:teuthology.orchestra.run.smithi183.stdout: "mds_recall_max_decay_rate": "1.500000", 2024-09-10T19:11:58.321 INFO:teuthology.orchestra.run.smithi183.stdout: "mds_recall_max_decay_threshold": "131072", 2024-09-10T19:11:58.321 INFO:teuthology.orchestra.run.smithi183.stdout: "mds_recall_warning_decay_rate": "60.000000", 2024-09-10T19:11:58.321 INFO:teuthology.orchestra.run.smithi183.stdout: "mds_recall_warning_threshold": "262144", 2024-09-10T19:11:58.321 INFO:teuthology.orchestra.run.smithi183.stdout: "mds_reconnect_timeout": "45.000000", 2024-09-10T19:11:58.321 INFO:teuthology.orchestra.run.smithi183.stdout: "mds_replay_interval": "1.000000", 2024-09-10T19:11:58.321 INFO:teuthology.orchestra.run.smithi183.stdout: "mds_replay_unsafe_with_closed_session": "false", 2024-09-10T19:11:58.321 INFO:teuthology.orchestra.run.smithi183.stdout: "mds_request_load_average_decay_rate": "60.000000", 2024-09-10T19:11:58.321 INFO:teuthology.orchestra.run.smithi183.stdout: "mds_root_ino_gid": "0", 2024-09-10T19:11:58.321 INFO:teuthology.orchestra.run.smithi183.stdout: "mds_root_ino_uid": "0", 2024-09-10T19:11:58.321 INFO:teuthology.orchestra.run.smithi183.stdout: "mds_scatter_nudge_interval": "5.000000", 2024-09-10T19:11:58.321 INFO:teuthology.orchestra.run.smithi183.stdout: "mds_session_blocklist_on_evict": "true", 2024-09-10T19:11:58.321 INFO:teuthology.orchestra.run.smithi183.stdout: "mds_session_blocklist_on_timeout": "true", 2024-09-10T19:11:58.321 INFO:teuthology.orchestra.run.smithi183.stdout: "mds_session_cache_liveness_decay_rate": "300.000000", 2024-09-10T19:11:58.322 INFO:teuthology.orchestra.run.smithi183.stdout: "mds_session_cache_liveness_magnitude": "10", 2024-09-10T19:11:58.322 INFO:teuthology.orchestra.run.smithi183.stdout: "mds_session_cap_acquisition_decay_rate": "30.000000", 2024-09-10T19:11:58.322 INFO:teuthology.orchestra.run.smithi183.stdout: "mds_session_cap_acquisition_throttle": "100000", 2024-09-10T19:11:58.322 INFO:teuthology.orchestra.run.smithi183.stdout: "mds_session_max_caps_throttle_ratio": "1.100000", 2024-09-10T19:11:58.322 INFO:teuthology.orchestra.run.smithi183.stdout: "mds_session_metadata_threshold": "16777216", 2024-09-10T19:11:58.322 INFO:teuthology.orchestra.run.smithi183.stdout: "mds_sessionmap_keys_per_op": "1024", 2024-09-10T19:11:58.322 INFO:teuthology.orchestra.run.smithi183.stdout: "mds_shutdown_check": "0", 2024-09-10T19:11:58.322 INFO:teuthology.orchestra.run.smithi183.stdout: "mds_skip_ino": "0", 2024-09-10T19:11:58.322 INFO:teuthology.orchestra.run.smithi183.stdout: "mds_sleep_rank_change": "0.000000", 2024-09-10T19:11:58.322 INFO:teuthology.orchestra.run.smithi183.stdout: "mds_snap_max_uid": "4294967294", 2024-09-10T19:11:58.322 INFO:teuthology.orchestra.run.smithi183.stdout: "mds_snap_min_uid": "0", 2024-09-10T19:11:58.322 INFO:teuthology.orchestra.run.smithi183.stdout: "mds_snap_rstat": "false", 2024-09-10T19:11:58.322 INFO:teuthology.orchestra.run.smithi183.stdout: "mds_standby_replay_damaged": "false", 2024-09-10T19:11:58.322 INFO:teuthology.orchestra.run.smithi183.stdout: "mds_symlink_recovery": "true", 2024-09-10T19:11:58.323 INFO:teuthology.orchestra.run.smithi183.stdout: "mds_task_status_update_interval": "2.000000", 2024-09-10T19:11:58.323 INFO:teuthology.orchestra.run.smithi183.stdout: "mds_thrash_exports": "0", 2024-09-10T19:11:58.323 INFO:teuthology.orchestra.run.smithi183.stdout: "mds_thrash_fragments": "0", 2024-09-10T19:11:58.323 INFO:teuthology.orchestra.run.smithi183.stdout: "mds_tick_interval": "5.000000", 2024-09-10T19:11:58.323 INFO:teuthology.orchestra.run.smithi183.stdout: "mds_valgrind_exit": "false", 2024-09-10T19:11:58.323 INFO:teuthology.orchestra.run.smithi183.stdout: "mds_verify_backtrace": "1", 2024-09-10T19:11:58.323 INFO:teuthology.orchestra.run.smithi183.stdout: "mds_verify_scatter": "false", 2024-09-10T19:11:58.323 INFO:teuthology.orchestra.run.smithi183.stdout: "mds_wipe_ino_prealloc": "false", 2024-09-10T19:11:58.323 INFO:teuthology.orchestra.run.smithi183.stdout: "mds_wipe_sessions": "false", 2024-09-10T19:11:58.323 INFO:teuthology.orchestra.run.smithi183.stdout: "mempool_debug": "false", 2024-09-10T19:11:58.323 INFO:teuthology.orchestra.run.smithi183.stdout: "memstore_debug_omit_block_device_write": "false", 2024-09-10T19:11:58.323 INFO:teuthology.orchestra.run.smithi183.stdout: "memstore_device_bytes": "1073741824", 2024-09-10T19:11:58.323 INFO:teuthology.orchestra.run.smithi183.stdout: "memstore_page_set": "false", 2024-09-10T19:11:58.323 INFO:teuthology.orchestra.run.smithi183.stdout: "memstore_page_size": "65536", 2024-09-10T19:11:58.323 INFO:teuthology.orchestra.run.smithi183.stdout: "mgr_client_bytes": "134217728", 2024-09-10T19:11:58.324 INFO:teuthology.orchestra.run.smithi183.stdout: "mgr_client_messages": "512", 2024-09-10T19:11:58.324 INFO:teuthology.orchestra.run.smithi183.stdout: "mgr_client_service_daemon_unregister_timeout": "1.000000", 2024-09-10T19:11:58.324 INFO:teuthology.orchestra.run.smithi183.stdout: "mgr_connect_retry_interval": "1.000000", 2024-09-10T19:11:58.324 INFO:teuthology.orchestra.run.smithi183.stdout: "mgr_data": "/var/lib/ceph/mgr/ceph-a", 2024-09-10T19:11:58.324 INFO:teuthology.orchestra.run.smithi183.stdout: "mgr_debug_aggressive_pg_num_changes": "false", 2024-09-10T19:11:58.324 INFO:teuthology.orchestra.run.smithi183.stdout: "mgr_disabled_modules": "diskprediction_local", 2024-09-10T19:11:58.324 INFO:teuthology.orchestra.run.smithi183.stdout: "mgr_initial_modules": "restful iostat nfs", 2024-09-10T19:11:58.324 INFO:teuthology.orchestra.run.smithi183.stdout: "mgr_max_pg_num_change": "128", 2024-09-10T19:11:58.324 INFO:teuthology.orchestra.run.smithi183.stdout: "mgr_mds_bytes": "134217728", 2024-09-10T19:11:58.324 INFO:teuthology.orchestra.run.smithi183.stdout: "mgr_mds_messages": "128", 2024-09-10T19:11:58.324 INFO:teuthology.orchestra.run.smithi183.stdout: "mgr_module_path": "/usr/share/ceph/mgr", 2024-09-10T19:11:58.324 INFO:teuthology.orchestra.run.smithi183.stdout: "mgr_mon_bytes": "134217728", 2024-09-10T19:11:58.324 INFO:teuthology.orchestra.run.smithi183.stdout: "mgr_mon_messages": "128", 2024-09-10T19:11:58.325 INFO:teuthology.orchestra.run.smithi183.stdout: "mgr_osd_bytes": "536870912", 2024-09-10T19:11:58.325 INFO:teuthology.orchestra.run.smithi183.stdout: "mgr_osd_messages": "8192", 2024-09-10T19:11:58.325 INFO:teuthology.orchestra.run.smithi183.stdout: "mgr_pool": "true", 2024-09-10T19:11:58.325 INFO:teuthology.orchestra.run.smithi183.stdout: "mgr_service_beacon_grace": "60.000000", 2024-09-10T19:11:58.325 INFO:teuthology.orchestra.run.smithi183.stdout: "mgr_standby_modules": "true", 2024-09-10T19:11:58.325 INFO:teuthology.orchestra.run.smithi183.stdout: "mgr_stats_period": "5", 2024-09-10T19:11:58.325 INFO:teuthology.orchestra.run.smithi183.stdout: "mgr_stats_threshold": "5", 2024-09-10T19:11:58.325 INFO:teuthology.orchestra.run.smithi183.stdout: "mgr_tick_period": "2", 2024-09-10T19:11:58.325 INFO:teuthology.orchestra.run.smithi183.stdout: "mgr_ttl_cache_expire_seconds": "0", 2024-09-10T19:11:58.325 INFO:teuthology.orchestra.run.smithi183.stdout: "mon_accept_timeout_factor": "2.000000", 2024-09-10T19:11:58.325 INFO:teuthology.orchestra.run.smithi183.stdout: "mon_allow_pool_delete": "true", 2024-09-10T19:11:58.326 INFO:teuthology.orchestra.run.smithi183.stdout: "mon_allow_pool_size_one": "false", 2024-09-10T19:11:58.326 INFO:teuthology.orchestra.run.smithi183.stdout: "mon_auth_validate_all_caps": "true", 2024-09-10T19:11:58.326 INFO:teuthology.orchestra.run.smithi183.stdout: "mon_cache_target_full_warn_ratio": "0.660000", 2024-09-10T19:11:58.326 INFO:teuthology.orchestra.run.smithi183.stdout: "mon_clean_pg_upmaps_per_chunk": "256", 2024-09-10T19:11:58.326 INFO:teuthology.orchestra.run.smithi183.stdout: "mon_client_bytes": "104857600", 2024-09-10T19:11:58.326 INFO:teuthology.orchestra.run.smithi183.stdout: "mon_client_directed_command_retry": "2", 2024-09-10T19:11:58.326 INFO:teuthology.orchestra.run.smithi183.stdout: "mon_client_hunt_interval": "3.000000", 2024-09-10T19:11:58.327 INFO:teuthology.orchestra.run.smithi183.stdout: "mon_client_hunt_interval_backoff": "1.500000", 2024-09-10T19:11:58.327 INFO:teuthology.orchestra.run.smithi183.stdout: "mon_client_hunt_interval_max_multiple": "10.000000", 2024-09-10T19:11:58.328 INFO:teuthology.orchestra.run.smithi183.stdout: "mon_client_hunt_interval_min_multiple": "1.000000", 2024-09-10T19:11:58.328 INFO:teuthology.orchestra.run.smithi183.stdout: "mon_client_hunt_parallel": "3", 2024-09-10T19:11:58.328 INFO:teuthology.orchestra.run.smithi183.stdout: "mon_client_log_interval": "1.000000", 2024-09-10T19:11:58.328 INFO:teuthology.orchestra.run.smithi183.stdout: "mon_client_max_log_entries_per_message": "1000", 2024-09-10T19:11:58.328 INFO:teuthology.orchestra.run.smithi183.stdout: "mon_client_ping_interval": "10.000000", 2024-09-10T19:11:58.328 INFO:teuthology.orchestra.run.smithi183.stdout: "mon_client_ping_timeout": "30.000000", 2024-09-10T19:11:58.328 INFO:teuthology.orchestra.run.smithi183.stdout: "mon_clock_drift_allowed": "1.000000", 2024-09-10T19:11:58.328 INFO:teuthology.orchestra.run.smithi183.stdout: "mon_clock_drift_warn_backoff": "5.000000", 2024-09-10T19:11:58.328 INFO:teuthology.orchestra.run.smithi183.stdout: "mon_cluster_log_file": "default=/var/log/ceph/ceph.$channel.log cluster=/var/log/ceph/ceph.log", 2024-09-10T19:11:58.328 INFO:teuthology.orchestra.run.smithi183.stdout: "mon_cluster_log_level": "debug", 2024-09-10T19:11:58.328 INFO:teuthology.orchestra.run.smithi183.stdout: "mon_cluster_log_to_file": "false", 2024-09-10T19:11:58.328 INFO:teuthology.orchestra.run.smithi183.stdout: "mon_cluster_log_to_graylog": "false", 2024-09-10T19:11:58.328 INFO:teuthology.orchestra.run.smithi183.stdout: "mon_cluster_log_to_graylog_host": "127.0.0.1", 2024-09-10T19:11:58.328 INFO:teuthology.orchestra.run.smithi183.stdout: "mon_cluster_log_to_graylog_port": "12201", 2024-09-10T19:11:58.329 INFO:teuthology.orchestra.run.smithi183.stdout: "mon_cluster_log_to_journald": "true", 2024-09-10T19:11:58.329 INFO:teuthology.orchestra.run.smithi183.stdout: "mon_cluster_log_to_stderr": "false", 2024-09-10T19:11:58.329 INFO:teuthology.orchestra.run.smithi183.stdout: "mon_cluster_log_to_syslog": "default=false", 2024-09-10T19:11:58.329 INFO:teuthology.orchestra.run.smithi183.stdout: "mon_cluster_log_to_syslog_facility": "daemon", 2024-09-10T19:11:58.329 INFO:teuthology.orchestra.run.smithi183.stdout: "mon_compact_on_bootstrap": "false", 2024-09-10T19:11:58.329 INFO:teuthology.orchestra.run.smithi183.stdout: "mon_compact_on_start": "false", 2024-09-10T19:11:58.329 INFO:teuthology.orchestra.run.smithi183.stdout: "mon_compact_on_trim": "true", 2024-09-10T19:11:58.329 INFO:teuthology.orchestra.run.smithi183.stdout: "mon_con_tracker_persist_interval": "10", 2024-09-10T19:11:58.329 INFO:teuthology.orchestra.run.smithi183.stdout: "mon_con_tracker_score_halflife": "43200", 2024-09-10T19:11:58.329 INFO:teuthology.orchestra.run.smithi183.stdout: "mon_config_key_max_entry_size": "65536", 2024-09-10T19:11:58.329 INFO:teuthology.orchestra.run.smithi183.stdout: "mon_cpu_threads": "4", 2024-09-10T19:11:58.329 INFO:teuthology.orchestra.run.smithi183.stdout: "mon_crush_min_required_version": "hammer", 2024-09-10T19:11:58.329 INFO:teuthology.orchestra.run.smithi183.stdout: "mon_daemon_bytes": "419430400", 2024-09-10T19:11:58.330 INFO:teuthology.orchestra.run.smithi183.stdout: "mon_data": "/var/lib/ceph/mon/ceph-a", 2024-09-10T19:11:58.330 INFO:teuthology.orchestra.run.smithi183.stdout: "mon_data_avail_crit": "5", 2024-09-10T19:11:58.330 INFO:teuthology.orchestra.run.smithi183.stdout: "mon_data_avail_warn": "5", 2024-09-10T19:11:58.330 INFO:teuthology.orchestra.run.smithi183.stdout: "mon_data_size_warn": "16106127360", 2024-09-10T19:11:58.330 INFO:teuthology.orchestra.run.smithi183.stdout: "mon_debug_block_osdmap_trim": "false", 2024-09-10T19:11:58.330 INFO:teuthology.orchestra.run.smithi183.stdout: "mon_debug_deprecated_as_obsolete": "false", 2024-09-10T19:11:58.330 INFO:teuthology.orchestra.run.smithi183.stdout: "mon_debug_dump_json": "false", 2024-09-10T19:11:58.330 INFO:teuthology.orchestra.run.smithi183.stdout: "mon_debug_dump_location": "/var/log/ceph/ceph-mon.a.tdump", 2024-09-10T19:11:58.330 INFO:teuthology.orchestra.run.smithi183.stdout: "mon_debug_dump_transactions": "false", 2024-09-10T19:11:58.330 INFO:teuthology.orchestra.run.smithi183.stdout: "mon_debug_extra_checks": "false", 2024-09-10T19:11:58.330 INFO:teuthology.orchestra.run.smithi183.stdout: "mon_debug_no_initial_persistent_features": "false", 2024-09-10T19:11:58.330 INFO:teuthology.orchestra.run.smithi183.stdout: "mon_debug_no_require_bluestore_for_ec_overwrites": "false", 2024-09-10T19:11:58.330 INFO:teuthology.orchestra.run.smithi183.stdout: "mon_debug_no_require_pacific": "false", 2024-09-10T19:11:58.330 INFO:teuthology.orchestra.run.smithi183.stdout: "mon_debug_no_require_quincy": "false", 2024-09-10T19:11:58.331 INFO:teuthology.orchestra.run.smithi183.stdout: "mon_debug_unsafe_allow_tier_with_nonempty_snaps": "false", 2024-09-10T19:11:58.331 INFO:teuthology.orchestra.run.smithi183.stdout: "mon_delta_reset_interval": "10.000000", 2024-09-10T19:11:58.331 INFO:teuthology.orchestra.run.smithi183.stdout: "mon_dns_srv_name": "ceph-mon", 2024-09-10T19:11:58.331 INFO:teuthology.orchestra.run.smithi183.stdout: "mon_down_mkfs_grace": "60", 2024-09-10T19:11:58.331 INFO:teuthology.orchestra.run.smithi183.stdout: "mon_election_default_strategy": "1", 2024-09-10T19:11:58.331 INFO:teuthology.orchestra.run.smithi183.stdout: "mon_election_timeout": "5.000000", 2024-09-10T19:11:58.331 INFO:teuthology.orchestra.run.smithi183.stdout: "mon_elector_ignore_propose_margin": "0.000500", 2024-09-10T19:11:58.331 INFO:teuthology.orchestra.run.smithi183.stdout: "mon_elector_ping_divisor": "2", 2024-09-10T19:11:58.331 INFO:teuthology.orchestra.run.smithi183.stdout: "mon_elector_ping_timeout": "2.000000", 2024-09-10T19:11:58.331 INFO:teuthology.orchestra.run.smithi183.stdout: "mon_enable_op_tracker": "true", 2024-09-10T19:11:58.331 INFO:teuthology.orchestra.run.smithi183.stdout: "mon_fake_pool_delete": "false", 2024-09-10T19:11:58.331 INFO:teuthology.orchestra.run.smithi183.stdout: "mon_force_quorum_join": "false", 2024-09-10T19:11:58.331 INFO:teuthology.orchestra.run.smithi183.stdout: "mon_globalid_prealloc": "10000", 2024-09-10T19:11:58.331 INFO:teuthology.orchestra.run.smithi183.stdout: "mon_health_detail_to_clog": "true", 2024-09-10T19:11:58.331 INFO:teuthology.orchestra.run.smithi183.stdout: "mon_health_log_update_period": "5", 2024-09-10T19:11:58.332 INFO:teuthology.orchestra.run.smithi183.stdout: "mon_health_max_detail": "50", 2024-09-10T19:11:58.332 INFO:teuthology.orchestra.run.smithi183.stdout: "mon_health_to_clog": "true", 2024-09-10T19:11:58.332 INFO:teuthology.orchestra.run.smithi183.stdout: "mon_health_to_clog_interval": "600", 2024-09-10T19:11:58.332 INFO:teuthology.orchestra.run.smithi183.stdout: "mon_health_to_clog_tick_interval": "60.000000", 2024-09-10T19:11:58.332 INFO:teuthology.orchestra.run.smithi183.stdout: "mon_host": "[v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0]", 2024-09-10T19:11:58.332 INFO:teuthology.orchestra.run.smithi183.stdout: "mon_host_override": "", 2024-09-10T19:11:58.332 INFO:teuthology.orchestra.run.smithi183.stdout: "mon_initial_members": "", 2024-09-10T19:11:58.332 INFO:teuthology.orchestra.run.smithi183.stdout: "mon_inject_pg_merge_bounce_probability": "0.000000", 2024-09-10T19:11:58.332 INFO:teuthology.orchestra.run.smithi183.stdout: "mon_inject_sync_get_chunk_delay": "0.000000", 2024-09-10T19:11:58.332 INFO:teuthology.orchestra.run.smithi183.stdout: "mon_inject_transaction_delay_max": "10.000000", 2024-09-10T19:11:58.332 INFO:teuthology.orchestra.run.smithi183.stdout: "mon_inject_transaction_delay_probability": "0.000000", 2024-09-10T19:11:58.332 INFO:teuthology.orchestra.run.smithi183.stdout: "mon_keyvaluedb": "rocksdb", 2024-09-10T19:11:58.332 INFO:teuthology.orchestra.run.smithi183.stdout: "mon_lease": "5.000000", 2024-09-10T19:11:58.333 INFO:teuthology.orchestra.run.smithi183.stdout: "mon_lease_ack_timeout_factor": "2.000000", 2024-09-10T19:11:58.333 INFO:teuthology.orchestra.run.smithi183.stdout: "mon_lease_renew_interval_factor": "0.600000", 2024-09-10T19:11:58.333 INFO:teuthology.orchestra.run.smithi183.stdout: "mon_log_full_interval": "50", 2024-09-10T19:11:58.333 INFO:teuthology.orchestra.run.smithi183.stdout: "mon_log_max": "10000", 2024-09-10T19:11:58.333 INFO:teuthology.orchestra.run.smithi183.stdout: "mon_log_max_summary": "50", 2024-09-10T19:11:58.333 INFO:teuthology.orchestra.run.smithi183.stdout: "mon_max_log_entries_per_event": "4096", 2024-09-10T19:11:58.333 INFO:teuthology.orchestra.run.smithi183.stdout: "mon_max_log_epochs": "500", 2024-09-10T19:11:58.333 INFO:teuthology.orchestra.run.smithi183.stdout: "mon_max_mdsmap_epochs": "500", 2024-09-10T19:11:58.333 INFO:teuthology.orchestra.run.smithi183.stdout: "mon_max_mgrmap_epochs": "500", 2024-09-10T19:11:58.333 INFO:teuthology.orchestra.run.smithi183.stdout: "mon_max_osd": "10000", 2024-09-10T19:11:58.333 INFO:teuthology.orchestra.run.smithi183.stdout: "mon_max_pg_per_osd": "10000", 2024-09-10T19:11:58.333 INFO:teuthology.orchestra.run.smithi183.stdout: "mon_max_pool_pg_num": "65536", 2024-09-10T19:11:58.333 INFO:teuthology.orchestra.run.smithi183.stdout: "mon_max_snap_prune_per_epoch": "100", 2024-09-10T19:11:58.333 INFO:teuthology.orchestra.run.smithi183.stdout: "mon_mds_blocklist_interval": "86400.000000", 2024-09-10T19:11:58.334 INFO:teuthology.orchestra.run.smithi183.stdout: "mon_mds_force_trim_to": "0", 2024-09-10T19:11:58.334 INFO:teuthology.orchestra.run.smithi183.stdout: "mon_mds_skip_sanity": "false", 2024-09-10T19:11:58.334 INFO:teuthology.orchestra.run.smithi183.stdout: "mon_memory_autotune": "true", 2024-09-10T19:11:58.334 INFO:teuthology.orchestra.run.smithi183.stdout: "mon_memory_target": "2147483648", 2024-09-10T19:11:58.334 INFO:teuthology.orchestra.run.smithi183.stdout: "mon_mgr_beacon_grace": "30", 2024-09-10T19:11:58.334 INFO:teuthology.orchestra.run.smithi183.stdout: "mon_mgr_blocklist_interval": "86400.000000", 2024-09-10T19:11:58.334 INFO:teuthology.orchestra.run.smithi183.stdout: "mon_mgr_digest_period": "5", 2024-09-10T19:11:58.334 INFO:teuthology.orchestra.run.smithi183.stdout: "mon_mgr_inactive_grace": "60", 2024-09-10T19:11:58.334 INFO:teuthology.orchestra.run.smithi183.stdout: "mon_mgr_mkfs_grace": "240", 2024-09-10T19:11:58.334 INFO:teuthology.orchestra.run.smithi183.stdout: "mon_mgr_proxy_client_bytes_ratio": "0.300000", 2024-09-10T19:11:58.334 INFO:teuthology.orchestra.run.smithi183.stdout: "mon_min_osdmap_epochs": "50", 2024-09-10T19:11:58.334 INFO:teuthology.orchestra.run.smithi183.stdout: "mon_op_complaint_time": "30", 2024-09-10T19:11:58.334 INFO:teuthology.orchestra.run.smithi183.stdout: "mon_op_history_duration": "600", 2024-09-10T19:11:58.334 INFO:teuthology.orchestra.run.smithi183.stdout: "mon_op_history_size": "20", 2024-09-10T19:11:58.334 INFO:teuthology.orchestra.run.smithi183.stdout: "mon_op_history_slow_op_size": "20", 2024-09-10T19:11:58.335 INFO:teuthology.orchestra.run.smithi183.stdout: "mon_op_history_slow_op_threshold": "10", 2024-09-10T19:11:58.335 INFO:teuthology.orchestra.run.smithi183.stdout: "mon_op_log_threshold": "5", 2024-09-10T19:11:58.335 INFO:teuthology.orchestra.run.smithi183.stdout: "mon_osd_adjust_down_out_interval": "true", 2024-09-10T19:11:58.335 INFO:teuthology.orchestra.run.smithi183.stdout: "mon_osd_adjust_heartbeat_grace": "true", 2024-09-10T19:11:58.335 INFO:teuthology.orchestra.run.smithi183.stdout: "mon_osd_auto_mark_auto_out_in": "true", 2024-09-10T19:11:58.335 INFO:teuthology.orchestra.run.smithi183.stdout: "mon_osd_auto_mark_in": "false", 2024-09-10T19:11:58.335 INFO:teuthology.orchestra.run.smithi183.stdout: "mon_osd_auto_mark_new_in": "true", 2024-09-10T19:11:58.335 INFO:teuthology.orchestra.run.smithi183.stdout: "mon_osd_backfillfull_ratio": "0.900000", 2024-09-10T19:11:58.335 INFO:teuthology.orchestra.run.smithi183.stdout: "mon_osd_blocklist_default_expire": "3600.000000", 2024-09-10T19:11:58.335 INFO:teuthology.orchestra.run.smithi183.stdout: "mon_osd_cache_size": "500", 2024-09-10T19:11:58.335 INFO:teuthology.orchestra.run.smithi183.stdout: "mon_osd_cache_size_min": "134217728", 2024-09-10T19:11:58.335 INFO:teuthology.orchestra.run.smithi183.stdout: "mon_osd_crush_smoke_test": "true", 2024-09-10T19:11:58.335 INFO:teuthology.orchestra.run.smithi183.stdout: "mon_osd_destroyed_out_interval": "600", 2024-09-10T19:11:58.335 INFO:teuthology.orchestra.run.smithi183.stdout: "mon_osd_down_out_interval": "600", 2024-09-10T19:11:58.336 INFO:teuthology.orchestra.run.smithi183.stdout: "mon_osd_down_out_subtree_limit": "rack", 2024-09-10T19:11:58.336 INFO:teuthology.orchestra.run.smithi183.stdout: "mon_osd_err_op_age_ratio": "128.000000", 2024-09-10T19:11:58.336 INFO:teuthology.orchestra.run.smithi183.stdout: "mon_osd_force_trim_to": "0", 2024-09-10T19:11:58.336 INFO:teuthology.orchestra.run.smithi183.stdout: "mon_osd_full_ratio": "0.950000", 2024-09-10T19:11:58.336 INFO:teuthology.orchestra.run.smithi183.stdout: "mon_osd_initial_require_min_compat_client": "luminous", 2024-09-10T19:11:58.336 INFO:teuthology.orchestra.run.smithi183.stdout: "mon_osd_laggy_halflife": "3600", 2024-09-10T19:11:58.336 INFO:teuthology.orchestra.run.smithi183.stdout: "mon_osd_laggy_max_interval": "300", 2024-09-10T19:11:58.336 INFO:teuthology.orchestra.run.smithi183.stdout: "mon_osd_laggy_weight": "0.300000", 2024-09-10T19:11:58.336 INFO:teuthology.orchestra.run.smithi183.stdout: "mon_osd_mapping_pgs_per_chunk": "4096", 2024-09-10T19:11:58.336 INFO:teuthology.orchestra.run.smithi183.stdout: "mon_osd_max_creating_pgs": "1024", 2024-09-10T19:11:58.336 INFO:teuthology.orchestra.run.smithi183.stdout: "mon_osd_max_initial_pgs": "1024", 2024-09-10T19:11:58.336 INFO:teuthology.orchestra.run.smithi183.stdout: "mon_osd_min_down_reporters": "2", 2024-09-10T19:11:58.336 INFO:teuthology.orchestra.run.smithi183.stdout: "mon_osd_min_in_ratio": "0.750000", 2024-09-10T19:11:58.337 INFO:teuthology.orchestra.run.smithi183.stdout: "mon_osd_min_up_ratio": "0.300000", 2024-09-10T19:11:58.337 INFO:teuthology.orchestra.run.smithi183.stdout: "mon_osd_nearfull_ratio": "0.850000", 2024-09-10T19:11:58.337 INFO:teuthology.orchestra.run.smithi183.stdout: "mon_osd_prime_pg_temp": "true", 2024-09-10T19:11:58.337 INFO:teuthology.orchestra.run.smithi183.stdout: "mon_osd_prime_pg_temp_max_estimate": "0.250000", 2024-09-10T19:11:58.337 INFO:teuthology.orchestra.run.smithi183.stdout: "mon_osd_prime_pg_temp_max_time": "0.500000", 2024-09-10T19:11:58.337 INFO:teuthology.orchestra.run.smithi183.stdout: "mon_osd_report_timeout": "900", 2024-09-10T19:11:58.337 INFO:teuthology.orchestra.run.smithi183.stdout: "mon_osd_reporter_subtree_level": "osd", 2024-09-10T19:11:58.337 INFO:teuthology.orchestra.run.smithi183.stdout: "mon_osd_snap_trim_queue_warn_on": "32768", 2024-09-10T19:11:58.337 INFO:teuthology.orchestra.run.smithi183.stdout: "mon_osd_warn_num_repaired": "10", 2024-09-10T19:11:58.337 INFO:teuthology.orchestra.run.smithi183.stdout: "mon_osd_warn_op_age": "32.000000", 2024-09-10T19:11:58.337 INFO:teuthology.orchestra.run.smithi183.stdout: "mon_osdmap_full_prune_enabled": "true", 2024-09-10T19:11:58.337 INFO:teuthology.orchestra.run.smithi183.stdout: "mon_osdmap_full_prune_interval": "2", 2024-09-10T19:11:58.337 INFO:teuthology.orchestra.run.smithi183.stdout: "mon_osdmap_full_prune_min": "15", 2024-09-10T19:11:58.337 INFO:teuthology.orchestra.run.smithi183.stdout: "mon_osdmap_full_prune_txsize": "2", 2024-09-10T19:11:58.338 INFO:teuthology.orchestra.run.smithi183.stdout: "mon_pg_check_down_all_threshold": "0.500000", 2024-09-10T19:11:58.338 INFO:teuthology.orchestra.run.smithi183.stdout: "mon_pg_stuck_threshold": "60", 2024-09-10T19:11:58.338 INFO:teuthology.orchestra.run.smithi183.stdout: "mon_pg_warn_max_object_skew": "0.000000", 2024-09-10T19:11:58.338 INFO:teuthology.orchestra.run.smithi183.stdout: "mon_pg_warn_min_objects": "10000", 2024-09-10T19:11:58.338 INFO:teuthology.orchestra.run.smithi183.stdout: "mon_pg_warn_min_per_osd": "0", 2024-09-10T19:11:58.338 INFO:teuthology.orchestra.run.smithi183.stdout: "mon_pg_warn_min_pool_objects": "1000", 2024-09-10T19:11:58.338 INFO:teuthology.orchestra.run.smithi183.stdout: "mon_pool_quota_crit_threshold": "0", 2024-09-10T19:11:58.338 INFO:teuthology.orchestra.run.smithi183.stdout: "mon_pool_quota_warn_threshold": "0", 2024-09-10T19:11:58.338 INFO:teuthology.orchestra.run.smithi183.stdout: "mon_probe_timeout": "2.000000", 2024-09-10T19:11:58.338 INFO:teuthology.orchestra.run.smithi183.stdout: "mon_reweight_max_change": "0.050000", 2024-09-10T19:11:58.338 INFO:teuthology.orchestra.run.smithi183.stdout: "mon_reweight_max_osds": "4", 2024-09-10T19:11:58.338 INFO:teuthology.orchestra.run.smithi183.stdout: "mon_reweight_min_bytes_per_osd": "10", 2024-09-10T19:11:58.338 INFO:teuthology.orchestra.run.smithi183.stdout: "mon_reweight_min_pgs_per_osd": "4", 2024-09-10T19:11:58.338 INFO:teuthology.orchestra.run.smithi183.stdout: "mon_rocksdb_options": "write_buffer_size=33554432,compression=kNoCompression,level_compaction_dynamic_level_bytes=true", 2024-09-10T19:11:58.339 INFO:teuthology.orchestra.run.smithi183.stdout: "mon_scrub_inject_crc_mismatch": "0.000000", 2024-09-10T19:11:58.339 INFO:teuthology.orchestra.run.smithi183.stdout: "mon_scrub_inject_missing_keys": "0.000000", 2024-09-10T19:11:58.339 INFO:teuthology.orchestra.run.smithi183.stdout: "mon_scrub_interval": "86400", 2024-09-10T19:11:58.339 INFO:teuthology.orchestra.run.smithi183.stdout: "mon_scrub_max_keys": "100", 2024-09-10T19:11:58.339 INFO:teuthology.orchestra.run.smithi183.stdout: "mon_scrub_timeout": "300", 2024-09-10T19:11:58.339 INFO:teuthology.orchestra.run.smithi183.stdout: "mon_session_timeout": "300", 2024-09-10T19:11:58.339 INFO:teuthology.orchestra.run.smithi183.stdout: "mon_smart_report_timeout": "5", 2024-09-10T19:11:58.339 INFO:teuthology.orchestra.run.smithi183.stdout: "mon_stat_smooth_intervals": "6", 2024-09-10T19:11:58.339 INFO:teuthology.orchestra.run.smithi183.stdout: "mon_stretch_cluster_recovery_ratio": "0.600000", 2024-09-10T19:11:58.339 INFO:teuthology.orchestra.run.smithi183.stdout: "mon_stretch_pool_min_size": "2", 2024-09-10T19:11:58.339 INFO:teuthology.orchestra.run.smithi183.stdout: "mon_stretch_pool_size": "4", 2024-09-10T19:11:58.339 INFO:teuthology.orchestra.run.smithi183.stdout: "mon_stretch_recovery_min_wait": "15.000000", 2024-09-10T19:11:58.339 INFO:teuthology.orchestra.run.smithi183.stdout: "mon_subscribe_interval": "86400.000000", 2024-09-10T19:11:58.339 INFO:teuthology.orchestra.run.smithi183.stdout: "mon_sync_debug": "false", 2024-09-10T19:11:58.340 INFO:teuthology.orchestra.run.smithi183.stdout: "mon_sync_max_payload_keys": "2000", 2024-09-10T19:11:58.340 INFO:teuthology.orchestra.run.smithi183.stdout: "mon_sync_max_payload_size": "1048576", 2024-09-10T19:11:58.340 INFO:teuthology.orchestra.run.smithi183.stdout: "mon_sync_provider_kill_at": "0", 2024-09-10T19:11:58.340 INFO:teuthology.orchestra.run.smithi183.stdout: "mon_sync_requester_kill_at": "0", 2024-09-10T19:11:58.340 INFO:teuthology.orchestra.run.smithi183.stdout: "mon_sync_timeout": "60.000000", 2024-09-10T19:11:58.340 INFO:teuthology.orchestra.run.smithi183.stdout: "mon_target_pg_per_osd": "100", 2024-09-10T19:11:58.340 INFO:teuthology.orchestra.run.smithi183.stdout: "mon_tick_interval": "5", 2024-09-10T19:11:58.340 INFO:teuthology.orchestra.run.smithi183.stdout: "mon_timecheck_interval": "300.000000", 2024-09-10T19:11:58.340 INFO:teuthology.orchestra.run.smithi183.stdout: "mon_timecheck_skew_interval": "30.000000", 2024-09-10T19:11:58.340 INFO:teuthology.orchestra.run.smithi183.stdout: "mon_warn_older_version_delay": "604800", 2024-09-10T19:11:58.340 INFO:teuthology.orchestra.run.smithi183.stdout: "mon_warn_on_cache_pools_without_hit_sets": "true", 2024-09-10T19:11:58.340 INFO:teuthology.orchestra.run.smithi183.stdout: "mon_warn_on_crush_straw_calc_version_zero": "false", 2024-09-10T19:11:58.340 INFO:teuthology.orchestra.run.smithi183.stdout: "mon_warn_on_degraded_stretch_mode": "true", 2024-09-10T19:11:58.340 INFO:teuthology.orchestra.run.smithi183.stdout: "mon_warn_on_filestore_osds": "true", 2024-09-10T19:11:58.341 INFO:teuthology.orchestra.run.smithi183.stdout: "mon_warn_on_insecure_global_id_reclaim": "false", 2024-09-10T19:11:58.341 INFO:teuthology.orchestra.run.smithi183.stdout: "mon_warn_on_insecure_global_id_reclaim_allowed": "false", 2024-09-10T19:11:58.341 INFO:teuthology.orchestra.run.smithi183.stdout: "mon_warn_on_legacy_crush_tunables": "false", 2024-09-10T19:11:58.341 INFO:teuthology.orchestra.run.smithi183.stdout: "mon_warn_on_misplaced": "false", 2024-09-10T19:11:58.341 INFO:teuthology.orchestra.run.smithi183.stdout: "mon_warn_on_msgr2_not_enabled": "true", 2024-09-10T19:11:58.341 INFO:teuthology.orchestra.run.smithi183.stdout: "mon_warn_on_older_version": "true", 2024-09-10T19:11:58.341 INFO:teuthology.orchestra.run.smithi183.stdout: "mon_warn_on_osd_down_out_interval_zero": "false", 2024-09-10T19:11:58.341 INFO:teuthology.orchestra.run.smithi183.stdout: "mon_warn_on_pool_no_app": "true", 2024-09-10T19:11:58.341 INFO:teuthology.orchestra.run.smithi183.stdout: "mon_warn_on_pool_no_redundancy": "true", 2024-09-10T19:11:58.341 INFO:teuthology.orchestra.run.smithi183.stdout: "mon_warn_on_pool_pg_num_not_power_of_two": "false", 2024-09-10T19:11:58.341 INFO:teuthology.orchestra.run.smithi183.stdout: "mon_warn_on_slow_ping_ratio": "0.050000", 2024-09-10T19:11:58.341 INFO:teuthology.orchestra.run.smithi183.stdout: "mon_warn_on_slow_ping_time": "0.000000", 2024-09-10T19:11:58.341 INFO:teuthology.orchestra.run.smithi183.stdout: "mon_warn_on_too_few_osds": "false", 2024-09-10T19:11:58.341 INFO:teuthology.orchestra.run.smithi183.stdout: "mon_warn_pg_not_deep_scrubbed_ratio": "0.750000", 2024-09-10T19:11:58.342 INFO:teuthology.orchestra.run.smithi183.stdout: "mon_warn_pg_not_scrubbed_ratio": "0.500000", 2024-09-10T19:11:58.342 INFO:teuthology.orchestra.run.smithi183.stdout: "monmap": "", 2024-09-10T19:11:58.342 INFO:teuthology.orchestra.run.smithi183.stdout: "ms_async_op_threads": "3", 2024-09-10T19:11:58.342 INFO:teuthology.orchestra.run.smithi183.stdout: "ms_async_rdma_buffer_size": "131072", 2024-09-10T19:11:58.342 INFO:teuthology.orchestra.run.smithi183.stdout: "ms_async_rdma_cm": "false", 2024-09-10T19:11:58.342 INFO:teuthology.orchestra.run.smithi183.stdout: "ms_async_rdma_device_name": "", 2024-09-10T19:11:58.342 INFO:teuthology.orchestra.run.smithi183.stdout: "ms_async_rdma_dscp": "96", 2024-09-10T19:11:58.342 INFO:teuthology.orchestra.run.smithi183.stdout: "ms_async_rdma_enable_hugepage": "false", 2024-09-10T19:11:58.342 INFO:teuthology.orchestra.run.smithi183.stdout: "ms_async_rdma_gid_idx": "0", 2024-09-10T19:11:58.342 INFO:teuthology.orchestra.run.smithi183.stdout: "ms_async_rdma_local_gid": "", 2024-09-10T19:11:58.342 INFO:teuthology.orchestra.run.smithi183.stdout: "ms_async_rdma_polling_us": "1000", 2024-09-10T19:11:58.342 INFO:teuthology.orchestra.run.smithi183.stdout: "ms_async_rdma_port_num": "1", 2024-09-10T19:11:58.342 INFO:teuthology.orchestra.run.smithi183.stdout: "ms_async_rdma_receive_buffers": "32768", 2024-09-10T19:11:58.342 INFO:teuthology.orchestra.run.smithi183.stdout: "ms_async_rdma_receive_queue_len": "4096", 2024-09-10T19:11:58.343 INFO:teuthology.orchestra.run.smithi183.stdout: "ms_async_rdma_roce_ver": "1", 2024-09-10T19:11:58.343 INFO:teuthology.orchestra.run.smithi183.stdout: "ms_async_rdma_send_buffers": "1024", 2024-09-10T19:11:58.343 INFO:teuthology.orchestra.run.smithi183.stdout: "ms_async_rdma_sl": "3", 2024-09-10T19:11:58.343 INFO:teuthology.orchestra.run.smithi183.stdout: "ms_async_rdma_support_srq": "true", 2024-09-10T19:11:58.343 INFO:teuthology.orchestra.run.smithi183.stdout: "ms_async_rdma_type": "ib", 2024-09-10T19:11:58.343 INFO:teuthology.orchestra.run.smithi183.stdout: "ms_async_reap_threshold": "5", 2024-09-10T19:11:58.343 INFO:teuthology.orchestra.run.smithi183.stdout: "ms_bind_before_connect": "false", 2024-09-10T19:11:58.343 INFO:teuthology.orchestra.run.smithi183.stdout: "ms_bind_ipv4": "true", 2024-09-10T19:11:58.343 INFO:teuthology.orchestra.run.smithi183.stdout: "ms_bind_ipv6": "false", 2024-09-10T19:11:58.343 INFO:teuthology.orchestra.run.smithi183.stdout: "ms_bind_msgr1": "true", 2024-09-10T19:11:58.343 INFO:teuthology.orchestra.run.smithi183.stdout: "ms_bind_msgr2": "true", 2024-09-10T19:11:58.343 INFO:teuthology.orchestra.run.smithi183.stdout: "ms_bind_port_max": "7568", 2024-09-10T19:11:58.343 INFO:teuthology.orchestra.run.smithi183.stdout: "ms_bind_port_min": "6800", 2024-09-10T19:11:58.343 INFO:teuthology.orchestra.run.smithi183.stdout: "ms_bind_prefer_ipv4": "false", 2024-09-10T19:11:58.343 INFO:teuthology.orchestra.run.smithi183.stdout: "ms_bind_retry_count": "3", 2024-09-10T19:11:58.344 INFO:teuthology.orchestra.run.smithi183.stdout: "ms_bind_retry_delay": "5", 2024-09-10T19:11:58.344 INFO:teuthology.orchestra.run.smithi183.stdout: "ms_blackhole_client": "false", 2024-09-10T19:11:58.344 INFO:teuthology.orchestra.run.smithi183.stdout: "ms_blackhole_mds": "false", 2024-09-10T19:11:58.344 INFO:teuthology.orchestra.run.smithi183.stdout: "ms_blackhole_mgr": "false", 2024-09-10T19:11:58.344 INFO:teuthology.orchestra.run.smithi183.stdout: "ms_blackhole_mon": "false", 2024-09-10T19:11:58.344 INFO:teuthology.orchestra.run.smithi183.stdout: "ms_blackhole_osd": "false", 2024-09-10T19:11:58.344 INFO:teuthology.orchestra.run.smithi183.stdout: "ms_client_mode": "crc secure", 2024-09-10T19:11:58.344 INFO:teuthology.orchestra.run.smithi183.stdout: "ms_cluster_mode": "crc secure", 2024-09-10T19:11:58.344 INFO:teuthology.orchestra.run.smithi183.stdout: "ms_cluster_type": "", 2024-09-10T19:11:58.344 INFO:teuthology.orchestra.run.smithi183.stdout: "ms_compress_secure": "false", 2024-09-10T19:11:58.344 INFO:teuthology.orchestra.run.smithi183.stdout: "ms_connection_idle_timeout": "900", 2024-09-10T19:11:58.344 INFO:teuthology.orchestra.run.smithi183.stdout: "ms_connection_ready_timeout": "10", 2024-09-10T19:11:58.344 INFO:teuthology.orchestra.run.smithi183.stdout: "ms_crc_data": "true", 2024-09-10T19:11:58.345 INFO:teuthology.orchestra.run.smithi183.stdout: "ms_crc_header": "true", 2024-09-10T19:11:58.345 INFO:teuthology.orchestra.run.smithi183.stdout: "ms_die_on_bad_msg": "false", 2024-09-10T19:11:58.345 INFO:teuthology.orchestra.run.smithi183.stdout: "ms_die_on_bug": "true", 2024-09-10T19:11:58.345 INFO:teuthology.orchestra.run.smithi183.stdout: "ms_die_on_old_message": "true", 2024-09-10T19:11:58.345 INFO:teuthology.orchestra.run.smithi183.stdout: "ms_die_on_skipped_message": "false", 2024-09-10T19:11:58.345 INFO:teuthology.orchestra.run.smithi183.stdout: "ms_die_on_unhandled_msg": "false", 2024-09-10T19:11:58.345 INFO:teuthology.orchestra.run.smithi183.stdout: "ms_dispatch_throttle_bytes": "104857600", 2024-09-10T19:11:58.345 INFO:teuthology.orchestra.run.smithi183.stdout: "ms_dpdk_coremask": "0xF", 2024-09-10T19:11:58.345 INFO:teuthology.orchestra.run.smithi183.stdout: "ms_dpdk_debug_allow_loopback": "false", 2024-09-10T19:11:58.346 INFO:teuthology.orchestra.run.smithi183.stdout: "ms_dpdk_devs_allowlist": "", 2024-09-10T19:11:58.346 INFO:teuthology.orchestra.run.smithi183.stdout: "ms_dpdk_enable_tso": "true", 2024-09-10T19:11:58.346 INFO:teuthology.orchestra.run.smithi183.stdout: "ms_dpdk_gateway_ipv4_addr": "", 2024-09-10T19:11:58.346 INFO:teuthology.orchestra.run.smithi183.stdout: "ms_dpdk_host_ipv4_addr": "", 2024-09-10T19:11:58.346 INFO:teuthology.orchestra.run.smithi183.stdout: "ms_dpdk_hugepages": "", 2024-09-10T19:11:58.346 INFO:teuthology.orchestra.run.smithi183.stdout: "ms_dpdk_hw_flow_control": "true", 2024-09-10T19:11:58.346 INFO:teuthology.orchestra.run.smithi183.stdout: "ms_dpdk_hw_queue_weight": "1.000000", 2024-09-10T19:11:58.346 INFO:teuthology.orchestra.run.smithi183.stdout: "ms_dpdk_lro": "true", 2024-09-10T19:11:58.346 INFO:teuthology.orchestra.run.smithi183.stdout: "ms_dpdk_memory_channel": "4", 2024-09-10T19:11:58.346 INFO:teuthology.orchestra.run.smithi183.stdout: "ms_dpdk_netmask_ipv4_addr": "", 2024-09-10T19:11:58.346 INFO:teuthology.orchestra.run.smithi183.stdout: "ms_dpdk_pmd": "", 2024-09-10T19:11:58.346 INFO:teuthology.orchestra.run.smithi183.stdout: "ms_dpdk_port_id": "0", 2024-09-10T19:11:58.346 INFO:teuthology.orchestra.run.smithi183.stdout: "ms_dpdk_rx_buffer_count_per_core": "8192", 2024-09-10T19:11:58.346 INFO:teuthology.orchestra.run.smithi183.stdout: "ms_dump_corrupt_message_level": "1", 2024-09-10T19:11:58.347 INFO:teuthology.orchestra.run.smithi183.stdout: "ms_dump_on_send": "false", 2024-09-10T19:11:58.347 INFO:teuthology.orchestra.run.smithi183.stdout: "ms_initial_backoff": "0.200000", 2024-09-10T19:11:58.347 INFO:teuthology.orchestra.run.smithi183.stdout: "ms_inject_delay_max": "1.000000", 2024-09-10T19:11:58.347 INFO:teuthology.orchestra.run.smithi183.stdout: "ms_inject_delay_probability": "0.000000", 2024-09-10T19:11:58.347 INFO:teuthology.orchestra.run.smithi183.stdout: "ms_inject_delay_type": "", 2024-09-10T19:11:58.347 INFO:teuthology.orchestra.run.smithi183.stdout: "ms_inject_internal_delays": "0.000000", 2024-09-10T19:11:58.347 INFO:teuthology.orchestra.run.smithi183.stdout: "ms_inject_socket_failures": "0", 2024-09-10T19:11:58.347 INFO:teuthology.orchestra.run.smithi183.stdout: "ms_learn_addr_from_peer": "true", 2024-09-10T19:11:58.347 INFO:teuthology.orchestra.run.smithi183.stdout: "ms_max_accept_failures": "4", 2024-09-10T19:11:58.347 INFO:teuthology.orchestra.run.smithi183.stdout: "ms_max_backoff": "15.000000", 2024-09-10T19:11:58.347 INFO:teuthology.orchestra.run.smithi183.stdout: "ms_mon_client_mode": "secure crc", 2024-09-10T19:11:58.347 INFO:teuthology.orchestra.run.smithi183.stdout: "ms_mon_cluster_mode": "secure crc", 2024-09-10T19:11:58.347 INFO:teuthology.orchestra.run.smithi183.stdout: "ms_mon_service_mode": "secure crc", 2024-09-10T19:11:58.347 INFO:teuthology.orchestra.run.smithi183.stdout: "ms_osd_compress_min_size": "1024", 2024-09-10T19:11:58.348 INFO:teuthology.orchestra.run.smithi183.stdout: "ms_osd_compress_mode": "none", 2024-09-10T19:11:58.348 INFO:teuthology.orchestra.run.smithi183.stdout: "ms_osd_compression_algorithm": "snappy zlib zstd lz4", 2024-09-10T19:11:58.348 INFO:teuthology.orchestra.run.smithi183.stdout: "ms_pq_max_tokens_per_priority": "16777216", 2024-09-10T19:11:58.348 INFO:teuthology.orchestra.run.smithi183.stdout: "ms_pq_min_cost": "65536", 2024-09-10T19:11:58.348 INFO:teuthology.orchestra.run.smithi183.stdout: "ms_public_type": "", 2024-09-10T19:11:58.348 INFO:teuthology.orchestra.run.smithi183.stdout: "ms_service_mode": "crc secure", 2024-09-10T19:11:58.348 INFO:teuthology.orchestra.run.smithi183.stdout: "ms_tcp_listen_backlog": "512", 2024-09-10T19:11:58.348 INFO:teuthology.orchestra.run.smithi183.stdout: "ms_tcp_nodelay": "true", 2024-09-10T19:11:58.348 INFO:teuthology.orchestra.run.smithi183.stdout: "ms_tcp_prefetch_max_size": "4096", 2024-09-10T19:11:58.348 INFO:teuthology.orchestra.run.smithi183.stdout: "ms_tcp_rcvbuf": "0", 2024-09-10T19:11:58.348 INFO:teuthology.orchestra.run.smithi183.stdout: "ms_type": "async+posix", 2024-09-10T19:11:58.348 INFO:teuthology.orchestra.run.smithi183.stdout: "no_config_file": "false", 2024-09-10T19:11:58.348 INFO:teuthology.orchestra.run.smithi183.stdout: "objecter_completion_locks_per_session": "32", 2024-09-10T19:11:58.348 INFO:teuthology.orchestra.run.smithi183.stdout: "objecter_debug_inject_relock_delay": "false", 2024-09-10T19:11:58.349 INFO:teuthology.orchestra.run.smithi183.stdout: "objecter_inflight_op_bytes": "104857600", 2024-09-10T19:11:58.349 INFO:teuthology.orchestra.run.smithi183.stdout: "objecter_inflight_ops": "1024", 2024-09-10T19:11:58.349 INFO:teuthology.orchestra.run.smithi183.stdout: "objecter_inject_no_watch_ping": "false", 2024-09-10T19:11:58.349 INFO:teuthology.orchestra.run.smithi183.stdout: "objecter_retry_writes_after_first_reply": "false", 2024-09-10T19:11:58.349 INFO:teuthology.orchestra.run.smithi183.stdout: "objecter_tick_interval": "5.000000", 2024-09-10T19:11:58.349 INFO:teuthology.orchestra.run.smithi183.stdout: "objecter_timeout": "10.000000", 2024-09-10T19:11:58.349 INFO:teuthology.orchestra.run.smithi183.stdout: "objectstore_blackhole": "false", 2024-09-10T19:11:58.349 INFO:teuthology.orchestra.run.smithi183.stdout: "openssl_engine_opts": "", 2024-09-10T19:11:58.349 INFO:teuthology.orchestra.run.smithi183.stdout: "osd_agent_delay_time": "5.000000", 2024-09-10T19:11:58.349 INFO:teuthology.orchestra.run.smithi183.stdout: "osd_agent_hist_halflife": "1000", 2024-09-10T19:11:58.349 INFO:teuthology.orchestra.run.smithi183.stdout: "osd_agent_max_low_ops": "2", 2024-09-10T19:11:58.349 INFO:teuthology.orchestra.run.smithi183.stdout: "osd_agent_max_ops": "4", 2024-09-10T19:11:58.349 INFO:teuthology.orchestra.run.smithi183.stdout: "osd_agent_min_evict_effort": "0.100000", 2024-09-10T19:11:58.349 INFO:teuthology.orchestra.run.smithi183.stdout: "osd_agent_quantize_effort": "0.100000", 2024-09-10T19:11:58.350 INFO:teuthology.orchestra.run.smithi183.stdout: "osd_agent_slop": "0.020000", 2024-09-10T19:11:58.350 INFO:teuthology.orchestra.run.smithi183.stdout: "osd_aggregated_slow_ops_logging": "true", 2024-09-10T19:11:58.350 INFO:teuthology.orchestra.run.smithi183.stdout: "osd_allow_recovery_below_min_size": "true", 2024-09-10T19:11:58.350 INFO:teuthology.orchestra.run.smithi183.stdout: "osd_asio_thread_count": "2", 2024-09-10T19:11:58.350 INFO:teuthology.orchestra.run.smithi183.stdout: "osd_async_recovery_min_cost": "100", 2024-09-10T19:11:58.350 INFO:teuthology.orchestra.run.smithi183.stdout: "osd_auto_mark_unfound_lost": "false", 2024-09-10T19:11:58.350 INFO:teuthology.orchestra.run.smithi183.stdout: "osd_backfill_retry_interval": "30.000000", 2024-09-10T19:11:58.350 INFO:teuthology.orchestra.run.smithi183.stdout: "osd_backfill_scan_max": "512", 2024-09-10T19:11:58.350 INFO:teuthology.orchestra.run.smithi183.stdout: "osd_backfill_scan_min": "64", 2024-09-10T19:11:58.350 INFO:teuthology.orchestra.run.smithi183.stdout: "osd_backoff_on_degraded": "false", 2024-09-10T19:11:58.350 INFO:teuthology.orchestra.run.smithi183.stdout: "osd_backoff_on_peering": "false", 2024-09-10T19:11:58.350 INFO:teuthology.orchestra.run.smithi183.stdout: "osd_backoff_on_unfound": "true", 2024-09-10T19:11:58.350 INFO:teuthology.orchestra.run.smithi183.stdout: "osd_beacon_report_interval": "300", 2024-09-10T19:11:58.350 INFO:teuthology.orchestra.run.smithi183.stdout: "osd_bench_duration": "30", 2024-09-10T19:11:58.351 INFO:teuthology.orchestra.run.smithi183.stdout: "osd_bench_large_size_max_throughput": "104857600", 2024-09-10T19:11:58.351 INFO:teuthology.orchestra.run.smithi183.stdout: "osd_bench_max_block_size": "67108864", 2024-09-10T19:11:58.351 INFO:teuthology.orchestra.run.smithi183.stdout: "osd_bench_small_size_max_iops": "100", 2024-09-10T19:11:58.351 INFO:teuthology.orchestra.run.smithi183.stdout: "osd_blkin_trace_all": "false", 2024-09-10T19:11:58.351 INFO:teuthology.orchestra.run.smithi183.stdout: "osd_calc_pg_upmaps_aggressively": "true", 2024-09-10T19:11:58.351 INFO:teuthology.orchestra.run.smithi183.stdout: "osd_calc_pg_upmaps_local_fallback_retries": "100", 2024-09-10T19:11:58.351 INFO:teuthology.orchestra.run.smithi183.stdout: "osd_check_for_log_corruption": "false", 2024-09-10T19:11:58.351 INFO:teuthology.orchestra.run.smithi183.stdout: "osd_check_max_object_name_len_on_startup": "true", 2024-09-10T19:11:58.351 INFO:teuthology.orchestra.run.smithi183.stdout: "osd_class_default_list": "cephfs hello journal lock log numops otp rbd refcount rgw rgw_gc timeindex user version cas cmpomap queue 2pc_queue fifo", 2024-09-10T19:11:58.351 INFO:teuthology.orchestra.run.smithi183.stdout: "osd_class_dir": "/usr/lib64/rados-classes", 2024-09-10T19:11:58.351 INFO:teuthology.orchestra.run.smithi183.stdout: "osd_class_load_list": "cephfs hello journal lock log numops otp rbd refcount rgw rgw_gc timeindex user version cas cmpomap queue 2pc_queue fifo", 2024-09-10T19:11:58.351 INFO:teuthology.orchestra.run.smithi183.stdout: "osd_class_update_on_start": "true", 2024-09-10T19:11:58.351 INFO:teuthology.orchestra.run.smithi183.stdout: "osd_client_message_cap": "256", 2024-09-10T19:11:58.351 INFO:teuthology.orchestra.run.smithi183.stdout: "osd_client_message_size_cap": "524288000", 2024-09-10T19:11:58.352 INFO:teuthology.orchestra.run.smithi183.stdout: "osd_client_op_priority": "63", 2024-09-10T19:11:58.352 INFO:teuthology.orchestra.run.smithi183.stdout: "osd_client_watch_timeout": "30", 2024-09-10T19:11:58.352 INFO:teuthology.orchestra.run.smithi183.stdout: "osd_command_max_records": "256", 2024-09-10T19:11:58.352 INFO:teuthology.orchestra.run.smithi183.stdout: "osd_compact_on_start": "false", 2024-09-10T19:11:58.352 INFO:teuthology.orchestra.run.smithi183.stdout: "osd_copyfrom_max_chunk": "8388608", 2024-09-10T19:11:58.352 INFO:teuthology.orchestra.run.smithi183.stdout: "osd_crush_chooseleaf_type": "1", 2024-09-10T19:11:58.352 INFO:teuthology.orchestra.run.smithi183.stdout: "osd_crush_initial_weight": "-1.000000", 2024-09-10T19:11:58.352 INFO:teuthology.orchestra.run.smithi183.stdout: "osd_crush_update_on_start": "true", 2024-09-10T19:11:58.352 INFO:teuthology.orchestra.run.smithi183.stdout: "osd_crush_update_weight_set": "true", 2024-09-10T19:11:58.352 INFO:teuthology.orchestra.run.smithi183.stdout: "osd_data": "/var/lib/ceph/osd/ceph-a", 2024-09-10T19:11:58.352 INFO:teuthology.orchestra.run.smithi183.stdout: "osd_debug_crash_on_ignored_backoff": "false", 2024-09-10T19:11:58.352 INFO:teuthology.orchestra.run.smithi183.stdout: "osd_debug_deep_scrub_sleep": "0.000000", 2024-09-10T19:11:58.352 INFO:teuthology.orchestra.run.smithi183.stdout: "osd_debug_drop_ping_duration": "0", 2024-09-10T19:11:58.352 INFO:teuthology.orchestra.run.smithi183.stdout: "osd_debug_drop_ping_probability": "0.000000", 2024-09-10T19:11:58.353 INFO:teuthology.orchestra.run.smithi183.stdout: "osd_debug_feed_pullee": "-1", 2024-09-10T19:11:58.353 INFO:teuthology.orchestra.run.smithi183.stdout: "osd_debug_inject_copyfrom_error": "false", 2024-09-10T19:11:58.353 INFO:teuthology.orchestra.run.smithi183.stdout: "osd_debug_inject_dispatch_delay_duration": "0.100000", 2024-09-10T19:11:58.353 INFO:teuthology.orchestra.run.smithi183.stdout: "osd_debug_inject_dispatch_delay_probability": "0.000000", 2024-09-10T19:11:58.353 INFO:teuthology.orchestra.run.smithi183.stdout: "osd_debug_misdirected_ops": "false", 2024-09-10T19:11:58.353 INFO:teuthology.orchestra.run.smithi183.stdout: "osd_debug_no_acting_change": "false", 2024-09-10T19:11:58.353 INFO:teuthology.orchestra.run.smithi183.stdout: "osd_debug_no_purge_strays": "false", 2024-09-10T19:11:58.353 INFO:teuthology.orchestra.run.smithi183.stdout: "osd_debug_op_order": "false", 2024-09-10T19:11:58.353 INFO:teuthology.orchestra.run.smithi183.stdout: "osd_debug_pg_log_writeout": "false", 2024-09-10T19:11:58.353 INFO:teuthology.orchestra.run.smithi183.stdout: "osd_debug_pretend_recovery_active": "false", 2024-09-10T19:11:58.353 INFO:teuthology.orchestra.run.smithi183.stdout: "osd_debug_random_push_read_error": "0.000000", 2024-09-10T19:11:58.353 INFO:teuthology.orchestra.run.smithi183.stdout: "osd_debug_reject_backfill_probability": "0.000000", 2024-09-10T19:11:58.353 INFO:teuthology.orchestra.run.smithi183.stdout: "osd_debug_shutdown": "false", 2024-09-10T19:11:58.353 INFO:teuthology.orchestra.run.smithi183.stdout: "osd_debug_skip_full_check_in_backfill_reservation": "false", 2024-09-10T19:11:58.354 INFO:teuthology.orchestra.run.smithi183.stdout: "osd_debug_skip_full_check_in_recovery": "false", 2024-09-10T19:11:58.354 INFO:teuthology.orchestra.run.smithi183.stdout: "osd_debug_verify_cached_snaps": "false", 2024-09-10T19:11:58.354 INFO:teuthology.orchestra.run.smithi183.stdout: "osd_debug_verify_missing_on_start": "false", 2024-09-10T19:11:58.354 INFO:teuthology.orchestra.run.smithi183.stdout: "osd_debug_verify_snaps": "false", 2024-09-10T19:11:58.354 INFO:teuthology.orchestra.run.smithi183.stdout: "osd_debug_verify_stray_on_activate": "false", 2024-09-10T19:11:58.354 INFO:teuthology.orchestra.run.smithi183.stdout: "osd_deep_scrub_interval": "604800.000000", 2024-09-10T19:11:58.354 INFO:teuthology.orchestra.run.smithi183.stdout: "osd_deep_scrub_keys": "1024", 2024-09-10T19:11:58.354 INFO:teuthology.orchestra.run.smithi183.stdout: "osd_deep_scrub_large_omap_object_key_threshold": "200000", 2024-09-10T19:11:58.354 INFO:teuthology.orchestra.run.smithi183.stdout: "osd_deep_scrub_large_omap_object_value_sum_threshold": "1073741824", 2024-09-10T19:11:58.354 INFO:teuthology.orchestra.run.smithi183.stdout: "osd_deep_scrub_randomize_ratio": "0.150000", 2024-09-10T19:11:58.354 INFO:teuthology.orchestra.run.smithi183.stdout: "osd_deep_scrub_stride": "524288", 2024-09-10T19:11:58.354 INFO:teuthology.orchestra.run.smithi183.stdout: "osd_deep_scrub_update_digest_min_age": "7200", 2024-09-10T19:11:58.354 INFO:teuthology.orchestra.run.smithi183.stdout: "osd_default_data_pool_replay_window": "45", 2024-09-10T19:11:58.354 INFO:teuthology.orchestra.run.smithi183.stdout: "osd_default_notify_timeout": "30", 2024-09-10T19:11:58.355 INFO:teuthology.orchestra.run.smithi183.stdout: "osd_delete_sleep": "0.000000", 2024-09-10T19:11:58.355 INFO:teuthology.orchestra.run.smithi183.stdout: "osd_delete_sleep_hdd": "5.000000", 2024-09-10T19:11:58.355 INFO:teuthology.orchestra.run.smithi183.stdout: "osd_delete_sleep_hybrid": "1.000000", 2024-09-10T19:11:58.355 INFO:teuthology.orchestra.run.smithi183.stdout: "osd_delete_sleep_ssd": "1.000000", 2024-09-10T19:11:58.355 INFO:teuthology.orchestra.run.smithi183.stdout: "osd_discard_disconnected_ops": "true", 2024-09-10T19:11:58.355 INFO:teuthology.orchestra.run.smithi183.stdout: "osd_enable_op_tracker": "true", 2024-09-10T19:11:58.355 INFO:teuthology.orchestra.run.smithi183.stdout: "osd_erasure_code_plugins": "jerasure lrc", 2024-09-10T19:11:58.355 INFO:teuthology.orchestra.run.smithi183.stdout: "osd_failsafe_full_ratio": "0.970000", 2024-09-10T19:11:58.355 INFO:teuthology.orchestra.run.smithi183.stdout: "osd_fast_fail_on_connection_refused": "true", 2024-09-10T19:11:58.355 INFO:teuthology.orchestra.run.smithi183.stdout: "osd_fast_info": "true", 2024-09-10T19:11:58.355 INFO:teuthology.orchestra.run.smithi183.stdout: "osd_fast_shutdown": "true", 2024-09-10T19:11:58.355 INFO:teuthology.orchestra.run.smithi183.stdout: "osd_fast_shutdown_notify_mon": "true", 2024-09-10T19:11:58.355 INFO:teuthology.orchestra.run.smithi183.stdout: "osd_fast_shutdown_timeout": "15", 2024-09-10T19:11:58.355 INFO:teuthology.orchestra.run.smithi183.stdout: "osd_find_best_info_ignore_history_les": "false", 2024-09-10T19:11:58.356 INFO:teuthology.orchestra.run.smithi183.stdout: "osd_force_auth_primary_missing_objects": "100", 2024-09-10T19:11:58.356 INFO:teuthology.orchestra.run.smithi183.stdout: "osd_force_recovery_pg_log_entries_factor": "1.300000", 2024-09-10T19:11:58.356 INFO:teuthology.orchestra.run.smithi183.stdout: "osd_function_tracing": "false", 2024-09-10T19:11:58.356 INFO:teuthology.orchestra.run.smithi183.stdout: "osd_heartbeat_grace": "20", 2024-09-10T19:11:58.356 INFO:teuthology.orchestra.run.smithi183.stdout: "osd_heartbeat_interval": "6", 2024-09-10T19:11:58.356 INFO:teuthology.orchestra.run.smithi183.stdout: "osd_heartbeat_min_healthy_ratio": "0.330000", 2024-09-10T19:11:58.356 INFO:teuthology.orchestra.run.smithi183.stdout: "osd_heartbeat_min_peers": "10", 2024-09-10T19:11:58.356 INFO:teuthology.orchestra.run.smithi183.stdout: "osd_heartbeat_min_size": "2000", 2024-09-10T19:11:58.356 INFO:teuthology.orchestra.run.smithi183.stdout: "osd_heartbeat_stale": "600", 2024-09-10T19:11:58.356 INFO:teuthology.orchestra.run.smithi183.stdout: "osd_heartbeat_use_min_delay_socket": "false", 2024-09-10T19:11:58.356 INFO:teuthology.orchestra.run.smithi183.stdout: "osd_hit_set_max_size": "100000", 2024-09-10T19:11:58.356 INFO:teuthology.orchestra.run.smithi183.stdout: "osd_hit_set_min_size": "1000", 2024-09-10T19:11:58.356 INFO:teuthology.orchestra.run.smithi183.stdout: "osd_hit_set_namespace": ".ceph-internal", 2024-09-10T19:11:58.356 INFO:teuthology.orchestra.run.smithi183.stdout: "osd_ignore_stale_divergent_priors": "false", 2024-09-10T19:11:58.357 INFO:teuthology.orchestra.run.smithi183.stdout: "osd_inject_bad_map_crc_probability": "0.000000", 2024-09-10T19:11:58.357 INFO:teuthology.orchestra.run.smithi183.stdout: "osd_inject_failure_on_pg_removal": "false", 2024-09-10T19:11:58.357 INFO:teuthology.orchestra.run.smithi183.stdout: "osd_journal": "/var/lib/ceph/osd/ceph-a/journal", 2024-09-10T19:11:58.357 INFO:teuthology.orchestra.run.smithi183.stdout: "osd_journal_flush_on_shutdown": "true", 2024-09-10T19:11:58.357 INFO:teuthology.orchestra.run.smithi183.stdout: "osd_journal_size": "5120", 2024-09-10T19:11:58.357 INFO:teuthology.orchestra.run.smithi183.stdout: "osd_kill_backfill_at": "0", 2024-09-10T19:11:58.357 INFO:teuthology.orchestra.run.smithi183.stdout: "osd_loop_before_reset_tphandle": "64", 2024-09-10T19:11:58.357 INFO:teuthology.orchestra.run.smithi183.stdout: "osd_map_cache_size": "50", 2024-09-10T19:11:58.357 INFO:teuthology.orchestra.run.smithi183.stdout: "osd_map_dedup": "true", 2024-09-10T19:11:58.357 INFO:teuthology.orchestra.run.smithi183.stdout: "osd_map_message_max": "40", 2024-09-10T19:11:58.357 INFO:teuthology.orchestra.run.smithi183.stdout: "osd_map_message_max_bytes": "10485760", 2024-09-10T19:11:58.357 INFO:teuthology.orchestra.run.smithi183.stdout: "osd_map_share_max_epochs": "40", 2024-09-10T19:11:58.357 INFO:teuthology.orchestra.run.smithi183.stdout: "osd_max_attr_name_len": "100", 2024-09-10T19:11:58.357 INFO:teuthology.orchestra.run.smithi183.stdout: "osd_max_attr_size": "0", 2024-09-10T19:11:58.358 INFO:teuthology.orchestra.run.smithi183.stdout: "osd_max_backfills": "1", 2024-09-10T19:11:58.358 INFO:teuthology.orchestra.run.smithi183.stdout: "osd_max_markdown_count": "5", 2024-09-10T19:11:58.358 INFO:teuthology.orchestra.run.smithi183.stdout: "osd_max_markdown_period": "600", 2024-09-10T19:11:58.358 INFO:teuthology.orchestra.run.smithi183.stdout: "osd_max_object_name_len": "2048", 2024-09-10T19:11:58.358 INFO:teuthology.orchestra.run.smithi183.stdout: "osd_max_object_namespace_len": "256", 2024-09-10T19:11:58.358 INFO:teuthology.orchestra.run.smithi183.stdout: "osd_max_object_size": "134217728", 2024-09-10T19:11:58.358 INFO:teuthology.orchestra.run.smithi183.stdout: "osd_max_omap_bytes_per_request": "1073741824", 2024-09-10T19:11:58.358 INFO:teuthology.orchestra.run.smithi183.stdout: "osd_max_omap_entries_per_request": "1024", 2024-09-10T19:11:58.358 INFO:teuthology.orchestra.run.smithi183.stdout: "osd_max_pg_blocked_by": "16", 2024-09-10T19:11:58.358 INFO:teuthology.orchestra.run.smithi183.stdout: "osd_max_pg_log_entries": "10000", 2024-09-10T19:11:58.358 INFO:teuthology.orchestra.run.smithi183.stdout: "osd_max_pg_per_osd_hard_ratio": "3.000000", 2024-09-10T19:11:58.358 INFO:teuthology.orchestra.run.smithi183.stdout: "osd_max_pgls": "1024", 2024-09-10T19:11:58.358 INFO:teuthology.orchestra.run.smithi183.stdout: "osd_max_push_cost": "8388608", 2024-09-10T19:11:58.358 INFO:teuthology.orchestra.run.smithi183.stdout: "osd_max_push_objects": "10", 2024-09-10T19:11:58.359 INFO:teuthology.orchestra.run.smithi183.stdout: "osd_max_scrubs": "3", 2024-09-10T19:11:58.359 INFO:teuthology.orchestra.run.smithi183.stdout: "osd_max_snap_prune_intervals_per_epoch": "512", 2024-09-10T19:11:58.359 INFO:teuthology.orchestra.run.smithi183.stdout: "osd_max_trimming_pgs": "2", 2024-09-10T19:11:58.359 INFO:teuthology.orchestra.run.smithi183.stdout: "osd_max_write_op_reply_len": "64", 2024-09-10T19:11:58.359 INFO:teuthology.orchestra.run.smithi183.stdout: "osd_max_write_size": "90", 2024-09-10T19:11:58.359 INFO:teuthology.orchestra.run.smithi183.stdout: "osd_mclock_force_run_benchmark_on_init": "false", 2024-09-10T19:11:58.359 INFO:teuthology.orchestra.run.smithi183.stdout: "osd_mclock_iops_capacity_threshold_hdd": "500.000000", 2024-09-10T19:11:58.359 INFO:teuthology.orchestra.run.smithi183.stdout: "osd_mclock_iops_capacity_threshold_ssd": "80000.000000", 2024-09-10T19:11:58.359 INFO:teuthology.orchestra.run.smithi183.stdout: "osd_mclock_max_capacity_iops_hdd": "315.000000", 2024-09-10T19:11:58.359 INFO:teuthology.orchestra.run.smithi183.stdout: "osd_mclock_max_capacity_iops_ssd": "21500.000000", 2024-09-10T19:11:58.359 INFO:teuthology.orchestra.run.smithi183.stdout: "osd_mclock_max_sequential_bandwidth_hdd": "157286400", 2024-09-10T19:11:58.359 INFO:teuthology.orchestra.run.smithi183.stdout: "osd_mclock_max_sequential_bandwidth_ssd": "1258291200", 2024-09-10T19:11:58.359 INFO:teuthology.orchestra.run.smithi183.stdout: "osd_mclock_override_recovery_settings": "false", 2024-09-10T19:11:58.359 INFO:teuthology.orchestra.run.smithi183.stdout: "osd_mclock_profile": "balanced", 2024-09-10T19:11:58.360 INFO:teuthology.orchestra.run.smithi183.stdout: "osd_mclock_scheduler_anticipation_timeout": "0.000000", 2024-09-10T19:11:58.360 INFO:teuthology.orchestra.run.smithi183.stdout: "osd_mclock_scheduler_background_best_effort_lim": "0.000000", 2024-09-10T19:11:58.360 INFO:teuthology.orchestra.run.smithi183.stdout: "osd_mclock_scheduler_background_best_effort_res": "0.000000", 2024-09-10T19:11:58.360 INFO:teuthology.orchestra.run.smithi183.stdout: "osd_mclock_scheduler_background_best_effort_wgt": "1", 2024-09-10T19:11:58.360 INFO:teuthology.orchestra.run.smithi183.stdout: "osd_mclock_scheduler_background_recovery_lim": "0.000000", 2024-09-10T19:11:58.360 INFO:teuthology.orchestra.run.smithi183.stdout: "osd_mclock_scheduler_background_recovery_res": "0.000000", 2024-09-10T19:11:58.360 INFO:teuthology.orchestra.run.smithi183.stdout: "osd_mclock_scheduler_background_recovery_wgt": "1", 2024-09-10T19:11:58.360 INFO:teuthology.orchestra.run.smithi183.stdout: "osd_mclock_scheduler_client_lim": "0.000000", 2024-09-10T19:11:58.360 INFO:teuthology.orchestra.run.smithi183.stdout: "osd_mclock_scheduler_client_res": "0.000000", 2024-09-10T19:11:58.360 INFO:teuthology.orchestra.run.smithi183.stdout: "osd_mclock_scheduler_client_wgt": "1", 2024-09-10T19:11:58.360 INFO:teuthology.orchestra.run.smithi183.stdout: "osd_mclock_skip_benchmark": "false", 2024-09-10T19:11:58.360 INFO:teuthology.orchestra.run.smithi183.stdout: "osd_memory_base": "805306368", 2024-09-10T19:11:58.360 INFO:teuthology.orchestra.run.smithi183.stdout: "osd_memory_cache_min": "134217728", 2024-09-10T19:11:58.360 INFO:teuthology.orchestra.run.smithi183.stdout: "osd_memory_cache_resize_interval": "1.000000", 2024-09-10T19:11:58.361 INFO:teuthology.orchestra.run.smithi183.stdout: "osd_memory_expected_fragmentation": "0.150000", 2024-09-10T19:11:58.361 INFO:teuthology.orchestra.run.smithi183.stdout: "osd_memory_target": "4294967296", 2024-09-10T19:11:58.361 INFO:teuthology.orchestra.run.smithi183.stdout: "osd_memory_target_autotune": "false", 2024-09-10T19:11:58.361 INFO:teuthology.orchestra.run.smithi183.stdout: "osd_memory_target_cgroup_limit_ratio": "0.800000", 2024-09-10T19:11:58.361 INFO:teuthology.orchestra.run.smithi183.stdout: "osd_min_pg_log_entries": "250", 2024-09-10T19:11:58.361 INFO:teuthology.orchestra.run.smithi183.stdout: "osd_min_recovery_priority": "0", 2024-09-10T19:11:58.361 INFO:teuthology.orchestra.run.smithi183.stdout: "osd_mon_heartbeat_interval": "30", 2024-09-10T19:11:58.361 INFO:teuthology.orchestra.run.smithi183.stdout: "osd_mon_heartbeat_stat_stale": "3600", 2024-09-10T19:11:58.361 INFO:teuthology.orchestra.run.smithi183.stdout: "osd_mon_report_interval": "5", 2024-09-10T19:11:58.361 INFO:teuthology.orchestra.run.smithi183.stdout: "osd_mon_report_max_in_flight": "2", 2024-09-10T19:11:58.361 INFO:teuthology.orchestra.run.smithi183.stdout: "osd_mon_shutdown_timeout": "5.000000", 2024-09-10T19:11:58.361 INFO:teuthology.orchestra.run.smithi183.stdout: "osd_num_cache_shards": "32", 2024-09-10T19:11:58.361 INFO:teuthology.orchestra.run.smithi183.stdout: "osd_num_op_tracker_shard": "32", 2024-09-10T19:11:58.361 INFO:teuthology.orchestra.run.smithi183.stdout: "osd_numa_auto_affinity": "true", 2024-09-10T19:11:58.362 INFO:teuthology.orchestra.run.smithi183.stdout: "osd_numa_node": "-1", 2024-09-10T19:11:58.362 INFO:teuthology.orchestra.run.smithi183.stdout: "osd_numa_prefer_iface": "true", 2024-09-10T19:11:58.362 INFO:teuthology.orchestra.run.smithi183.stdout: "osd_object_clean_region_max_num_intervals": "10", 2024-09-10T19:11:58.362 INFO:teuthology.orchestra.run.smithi183.stdout: "osd_objecter_finishers": "1", 2024-09-10T19:11:58.362 INFO:teuthology.orchestra.run.smithi183.stdout: "osd_objectstore": "bluestore", 2024-09-10T19:11:58.362 INFO:teuthology.orchestra.run.smithi183.stdout: "osd_objectstore_fuse": "false", 2024-09-10T19:11:58.362 INFO:teuthology.orchestra.run.smithi183.stdout: "osd_objectstore_tracing": "false", 2024-09-10T19:11:58.362 INFO:teuthology.orchestra.run.smithi183.stdout: "osd_op_complaint_time": "30.000000", 2024-09-10T19:11:58.362 INFO:teuthology.orchestra.run.smithi183.stdout: "osd_op_history_duration": "600", 2024-09-10T19:11:58.362 INFO:teuthology.orchestra.run.smithi183.stdout: "osd_op_history_size": "20", 2024-09-10T19:11:58.362 INFO:teuthology.orchestra.run.smithi183.stdout: "osd_op_history_slow_op_size": "20", 2024-09-10T19:11:58.362 INFO:teuthology.orchestra.run.smithi183.stdout: "osd_op_history_slow_op_threshold": "10.000000", 2024-09-10T19:11:58.362 INFO:teuthology.orchestra.run.smithi183.stdout: "osd_op_log_threshold": "5", 2024-09-10T19:11:58.362 INFO:teuthology.orchestra.run.smithi183.stdout: "osd_op_num_shards": "0", 2024-09-10T19:11:58.363 INFO:teuthology.orchestra.run.smithi183.stdout: "osd_op_num_shards_hdd": "5", 2024-09-10T19:11:58.363 INFO:teuthology.orchestra.run.smithi183.stdout: "osd_op_num_shards_ssd": "8", 2024-09-10T19:11:58.363 INFO:teuthology.orchestra.run.smithi183.stdout: "osd_op_num_threads_per_shard": "0", 2024-09-10T19:11:58.363 INFO:teuthology.orchestra.run.smithi183.stdout: "osd_op_num_threads_per_shard_hdd": "1", 2024-09-10T19:11:58.363 INFO:teuthology.orchestra.run.smithi183.stdout: "osd_op_num_threads_per_shard_ssd": "2", 2024-09-10T19:11:58.363 INFO:teuthology.orchestra.run.smithi183.stdout: "osd_op_pq_max_tokens_per_priority": "4194304", 2024-09-10T19:11:58.363 INFO:teuthology.orchestra.run.smithi183.stdout: "osd_op_pq_min_cost": "65536", 2024-09-10T19:11:58.363 INFO:teuthology.orchestra.run.smithi183.stdout: "osd_op_queue": "mclock_scheduler", 2024-09-10T19:11:58.363 INFO:teuthology.orchestra.run.smithi183.stdout: "osd_op_queue_cut_off": "high", 2024-09-10T19:11:58.363 INFO:teuthology.orchestra.run.smithi183.stdout: "osd_op_thread_suicide_timeout": "150", 2024-09-10T19:11:58.363 INFO:teuthology.orchestra.run.smithi183.stdout: "osd_op_thread_timeout": "15", 2024-09-10T19:11:58.363 INFO:teuthology.orchestra.run.smithi183.stdout: "osd_open_classes_on_start": "true", 2024-09-10T19:11:58.363 INFO:teuthology.orchestra.run.smithi183.stdout: "osd_os_flags": "0", 2024-09-10T19:11:58.363 INFO:teuthology.orchestra.run.smithi183.stdout: "osd_peering_op_priority": "255", 2024-09-10T19:11:58.364 INFO:teuthology.orchestra.run.smithi183.stdout: "osd_pg_delete_cost": "1048576", 2024-09-10T19:11:58.364 INFO:teuthology.orchestra.run.smithi183.stdout: "osd_pg_delete_priority": "5", 2024-09-10T19:11:58.364 INFO:teuthology.orchestra.run.smithi183.stdout: "osd_pg_epoch_max_lag_factor": "2.000000", 2024-09-10T19:11:58.364 INFO:teuthology.orchestra.run.smithi183.stdout: "osd_pg_epoch_persisted_max_stale": "40", 2024-09-10T19:11:58.364 INFO:teuthology.orchestra.run.smithi183.stdout: "osd_pg_log_dups_tracked": "3000", 2024-09-10T19:11:58.364 INFO:teuthology.orchestra.run.smithi183.stdout: "osd_pg_log_trim_max": "10000", 2024-09-10T19:11:58.364 INFO:teuthology.orchestra.run.smithi183.stdout: "osd_pg_log_trim_min": "100", 2024-09-10T19:11:58.364 INFO:teuthology.orchestra.run.smithi183.stdout: "osd_pg_max_concurrent_snap_trims": "2", 2024-09-10T19:11:58.364 INFO:teuthology.orchestra.run.smithi183.stdout: "osd_pg_object_context_cache_count": "64", 2024-09-10T19:11:58.364 INFO:teuthology.orchestra.run.smithi183.stdout: "osd_pg_stat_report_interval_max": "500", 2024-09-10T19:11:58.364 INFO:teuthology.orchestra.run.smithi183.stdout: "osd_pool_default_cache_max_evict_check_size": "10", 2024-09-10T19:11:58.364 INFO:teuthology.orchestra.run.smithi183.stdout: "osd_pool_default_cache_min_evict_age": "0", 2024-09-10T19:11:58.364 INFO:teuthology.orchestra.run.smithi183.stdout: "osd_pool_default_cache_min_flush_age": "0", 2024-09-10T19:11:58.364 INFO:teuthology.orchestra.run.smithi183.stdout: "osd_pool_default_cache_target_dirty_high_ratio": "0.600000", 2024-09-10T19:11:58.365 INFO:teuthology.orchestra.run.smithi183.stdout: "osd_pool_default_cache_target_dirty_ratio": "0.400000", 2024-09-10T19:11:58.365 INFO:teuthology.orchestra.run.smithi183.stdout: "osd_pool_default_cache_target_full_ratio": "0.800000", 2024-09-10T19:11:58.365 INFO:teuthology.orchestra.run.smithi183.stdout: "osd_pool_default_crush_rule": "-1", 2024-09-10T19:11:58.365 INFO:teuthology.orchestra.run.smithi183.stdout: "osd_pool_default_ec_fast_read": "false", 2024-09-10T19:11:58.365 INFO:teuthology.orchestra.run.smithi183.stdout: "osd_pool_default_erasure_code_profile": "plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd", 2024-09-10T19:11:58.365 INFO:teuthology.orchestra.run.smithi183.stdout: "osd_pool_default_flag_bulk": "false", 2024-09-10T19:11:58.365 INFO:teuthology.orchestra.run.smithi183.stdout: "osd_pool_default_flag_hashpspool": "true", 2024-09-10T19:11:58.365 INFO:teuthology.orchestra.run.smithi183.stdout: "osd_pool_default_flag_nodelete": "false", 2024-09-10T19:11:58.365 INFO:teuthology.orchestra.run.smithi183.stdout: "osd_pool_default_flag_nopgchange": "false", 2024-09-10T19:11:58.365 INFO:teuthology.orchestra.run.smithi183.stdout: "osd_pool_default_flag_nosizechange": "false", 2024-09-10T19:11:58.365 INFO:teuthology.orchestra.run.smithi183.stdout: "osd_pool_default_flags": "0", 2024-09-10T19:11:58.365 INFO:teuthology.orchestra.run.smithi183.stdout: "osd_pool_default_hit_set_bloom_fpp": "0.050000", 2024-09-10T19:11:58.365 INFO:teuthology.orchestra.run.smithi183.stdout: "osd_pool_default_min_size": "0", 2024-09-10T19:11:58.365 INFO:teuthology.orchestra.run.smithi183.stdout: "osd_pool_default_pg_autoscale_mode": "off", 2024-09-10T19:11:58.365 INFO:teuthology.orchestra.run.smithi183.stdout: "osd_pool_default_pg_num": "32", 2024-09-10T19:11:58.366 INFO:teuthology.orchestra.run.smithi183.stdout: "osd_pool_default_pgp_num": "0", 2024-09-10T19:11:58.366 INFO:teuthology.orchestra.run.smithi183.stdout: "osd_pool_default_read_lease_ratio": "0.800000", 2024-09-10T19:11:58.366 INFO:teuthology.orchestra.run.smithi183.stdout: "osd_pool_default_size": "3", 2024-09-10T19:11:58.366 INFO:teuthology.orchestra.run.smithi183.stdout: "osd_pool_default_type": "replicated", 2024-09-10T19:11:58.366 INFO:teuthology.orchestra.run.smithi183.stdout: "osd_pool_erasure_code_stripe_unit": "4096", 2024-09-10T19:11:58.366 INFO:teuthology.orchestra.run.smithi183.stdout: "osd_pool_use_gmt_hitset": "true", 2024-09-10T19:11:58.366 INFO:teuthology.orchestra.run.smithi183.stdout: "osd_push_per_object_cost": "1000", 2024-09-10T19:11:58.368 INFO:teuthology.orchestra.run.smithi183.stdout: "osd_read_ec_check_for_errors": "false", 2024-09-10T19:11:58.368 INFO:teuthology.orchestra.run.smithi183.stdout: "osd_recover_clone_overlap": "true", 2024-09-10T19:11:58.368 INFO:teuthology.orchestra.run.smithi183.stdout: "osd_recover_clone_overlap_limit": "10", 2024-09-10T19:11:58.368 INFO:teuthology.orchestra.run.smithi183.stdout: "osd_recovery_cost": "20971520", 2024-09-10T19:11:58.368 INFO:teuthology.orchestra.run.smithi183.stdout: "osd_recovery_delay_start": "0.000000", 2024-09-10T19:11:58.368 INFO:teuthology.orchestra.run.smithi183.stdout: "osd_recovery_max_active": "0", 2024-09-10T19:11:58.368 INFO:teuthology.orchestra.run.smithi183.stdout: "osd_recovery_max_active_hdd": "3", 2024-09-10T19:11:58.369 INFO:teuthology.orchestra.run.smithi183.stdout: "osd_recovery_max_active_ssd": "10", 2024-09-10T19:11:58.369 INFO:teuthology.orchestra.run.smithi183.stdout: "osd_recovery_max_chunk": "8388608", 2024-09-10T19:11:58.369 INFO:teuthology.orchestra.run.smithi183.stdout: "osd_recovery_max_omap_entries_per_chunk": "8096", 2024-09-10T19:11:58.369 INFO:teuthology.orchestra.run.smithi183.stdout: "osd_recovery_max_single_start": "1", 2024-09-10T19:11:58.369 INFO:teuthology.orchestra.run.smithi183.stdout: "osd_recovery_op_priority": "3", 2024-09-10T19:11:58.369 INFO:teuthology.orchestra.run.smithi183.stdout: "osd_recovery_op_warn_multiple": "16", 2024-09-10T19:11:58.369 INFO:teuthology.orchestra.run.smithi183.stdout: "osd_recovery_priority": "5", 2024-09-10T19:11:58.369 INFO:teuthology.orchestra.run.smithi183.stdout: "osd_recovery_retry_interval": "30.000000", 2024-09-10T19:11:58.369 INFO:teuthology.orchestra.run.smithi183.stdout: "osd_recovery_sleep": "0.000000", 2024-09-10T19:11:58.369 INFO:teuthology.orchestra.run.smithi183.stdout: "osd_recovery_sleep_hdd": "0.100000", 2024-09-10T19:11:58.369 INFO:teuthology.orchestra.run.smithi183.stdout: "osd_recovery_sleep_hybrid": "0.025000", 2024-09-10T19:11:58.370 INFO:teuthology.orchestra.run.smithi183.stdout: "osd_recovery_sleep_ssd": "0.000000", 2024-09-10T19:11:58.370 INFO:teuthology.orchestra.run.smithi183.stdout: "osd_repair_during_recovery": "false", 2024-09-10T19:11:58.370 INFO:teuthology.orchestra.run.smithi183.stdout: "osd_requested_scrub_priority": "120", 2024-09-10T19:11:58.370 INFO:teuthology.orchestra.run.smithi183.stdout: "osd_rocksdb_iterator_bounds_enabled": "true", 2024-09-10T19:11:58.370 INFO:teuthology.orchestra.run.smithi183.stdout: "osd_rollback_to_cluster_snap": "", 2024-09-10T19:11:58.370 INFO:teuthology.orchestra.run.smithi183.stdout: "osd_scrub_auto_repair": "false", 2024-09-10T19:11:58.370 INFO:teuthology.orchestra.run.smithi183.stdout: "osd_scrub_auto_repair_num_errors": "5", 2024-09-10T19:11:58.370 INFO:teuthology.orchestra.run.smithi183.stdout: "osd_scrub_backoff_ratio": "0.660000", 2024-09-10T19:11:58.370 INFO:teuthology.orchestra.run.smithi183.stdout: "osd_scrub_begin_hour": "0", 2024-09-10T19:11:58.370 INFO:teuthology.orchestra.run.smithi183.stdout: "osd_scrub_begin_week_day": "0", 2024-09-10T19:11:58.371 INFO:teuthology.orchestra.run.smithi183.stdout: "osd_scrub_chunk_max": "25", 2024-09-10T19:11:58.371 INFO:teuthology.orchestra.run.smithi183.stdout: "osd_scrub_chunk_min": "5", 2024-09-10T19:11:58.371 INFO:teuthology.orchestra.run.smithi183.stdout: "osd_scrub_cost": "52428800", 2024-09-10T19:11:58.371 INFO:teuthology.orchestra.run.smithi183.stdout: "osd_scrub_during_recovery": "false", 2024-09-10T19:11:58.371 INFO:teuthology.orchestra.run.smithi183.stdout: "osd_scrub_end_hour": "0", 2024-09-10T19:11:58.371 INFO:teuthology.orchestra.run.smithi183.stdout: "osd_scrub_end_week_day": "0", 2024-09-10T19:11:58.371 INFO:teuthology.orchestra.run.smithi183.stdout: "osd_scrub_event_cost": "4096", 2024-09-10T19:11:58.371 INFO:teuthology.orchestra.run.smithi183.stdout: "osd_scrub_extended_sleep": "0.000000", 2024-09-10T19:11:58.371 INFO:teuthology.orchestra.run.smithi183.stdout: "osd_scrub_interval_randomize_ratio": "0.500000", 2024-09-10T19:11:58.371 INFO:teuthology.orchestra.run.smithi183.stdout: "osd_scrub_invalid_stats": "true", 2024-09-10T19:11:58.371 INFO:teuthology.orchestra.run.smithi183.stdout: "osd_scrub_load_threshold": "0.500000", 2024-09-10T19:11:58.371 INFO:teuthology.orchestra.run.smithi183.stdout: "osd_scrub_max_interval": "604800.000000", 2024-09-10T19:11:58.371 INFO:teuthology.orchestra.run.smithi183.stdout: "osd_scrub_max_preemptions": "5", 2024-09-10T19:11:58.372 INFO:teuthology.orchestra.run.smithi183.stdout: "osd_scrub_min_interval": "86400.000000", 2024-09-10T19:11:58.372 INFO:teuthology.orchestra.run.smithi183.stdout: "osd_scrub_priority": "5", 2024-09-10T19:11:58.372 INFO:teuthology.orchestra.run.smithi183.stdout: "osd_scrub_sleep": "0.000000", 2024-09-10T19:11:58.372 INFO:teuthology.orchestra.run.smithi183.stdout: "osd_shutdown_pgref_assert": "false", 2024-09-10T19:11:58.372 INFO:teuthology.orchestra.run.smithi183.stdout: "osd_skip_data_digest": "false", 2024-09-10T19:11:58.372 INFO:teuthology.orchestra.run.smithi183.stdout: "osd_smart_report_timeout": "5", 2024-09-10T19:11:58.372 INFO:teuthology.orchestra.run.smithi183.stdout: "osd_snap_trim_cost": "1048576", 2024-09-10T19:11:58.372 INFO:teuthology.orchestra.run.smithi183.stdout: "osd_snap_trim_priority": "5", 2024-09-10T19:11:58.372 INFO:teuthology.orchestra.run.smithi183.stdout: "osd_snap_trim_sleep": "0.000000", 2024-09-10T19:11:58.372 INFO:teuthology.orchestra.run.smithi183.stdout: "osd_snap_trim_sleep_hdd": "5.000000", 2024-09-10T19:11:58.372 INFO:teuthology.orchestra.run.smithi183.stdout: "osd_snap_trim_sleep_hybrid": "2.000000", 2024-09-10T19:11:58.372 INFO:teuthology.orchestra.run.smithi183.stdout: "osd_snap_trim_sleep_ssd": "0.000000", 2024-09-10T19:11:58.372 INFO:teuthology.orchestra.run.smithi183.stdout: "osd_target_pg_log_entries_per_osd": "300000", 2024-09-10T19:11:58.373 INFO:teuthology.orchestra.run.smithi183.stdout: "osd_target_transaction_size": "30", 2024-09-10T19:11:58.373 INFO:teuthology.orchestra.run.smithi183.stdout: "osd_tier_default_cache_hit_set_count": "4", 2024-09-10T19:11:58.373 INFO:teuthology.orchestra.run.smithi183.stdout: "osd_tier_default_cache_hit_set_grade_decay_rate": "20", 2024-09-10T19:11:58.373 INFO:teuthology.orchestra.run.smithi183.stdout: "osd_tier_default_cache_hit_set_period": "1200", 2024-09-10T19:11:58.373 INFO:teuthology.orchestra.run.smithi183.stdout: "osd_tier_default_cache_hit_set_search_last_n": "1", 2024-09-10T19:11:58.373 INFO:teuthology.orchestra.run.smithi183.stdout: "osd_tier_default_cache_hit_set_type": "bloom", 2024-09-10T19:11:58.373 INFO:teuthology.orchestra.run.smithi183.stdout: "osd_tier_default_cache_min_read_recency_for_promote": "1", 2024-09-10T19:11:58.373 INFO:teuthology.orchestra.run.smithi183.stdout: "osd_tier_default_cache_min_write_recency_for_promote": "1", 2024-09-10T19:11:58.373 INFO:teuthology.orchestra.run.smithi183.stdout: "osd_tier_default_cache_mode": "writeback", 2024-09-10T19:11:58.373 INFO:teuthology.orchestra.run.smithi183.stdout: "osd_tier_promote_max_bytes_sec": "5242880", 2024-09-10T19:11:58.373 INFO:teuthology.orchestra.run.smithi183.stdout: "osd_tier_promote_max_objects_sec": "25", 2024-09-10T19:11:58.373 INFO:teuthology.orchestra.run.smithi183.stdout: "osd_tracing": "false", 2024-09-10T19:11:58.373 INFO:teuthology.orchestra.run.smithi183.stdout: "osd_use_stale_snap": "false", 2024-09-10T19:11:58.374 INFO:teuthology.orchestra.run.smithi183.stdout: "osd_uuid": "00000000-0000-0000-0000-000000000000", 2024-09-10T19:11:58.374 INFO:teuthology.orchestra.run.smithi183.stdout: "osdc_blkin_trace_all": "false", 2024-09-10T19:11:58.374 INFO:teuthology.orchestra.run.smithi183.stdout: "paxos_kill_at": "0", 2024-09-10T19:11:58.374 INFO:teuthology.orchestra.run.smithi183.stdout: "paxos_max_join_drift": "10", 2024-09-10T19:11:58.374 INFO:teuthology.orchestra.run.smithi183.stdout: "paxos_min": "500", 2024-09-10T19:11:58.374 INFO:teuthology.orchestra.run.smithi183.stdout: "paxos_min_wait": "0.050000", 2024-09-10T19:11:58.374 INFO:teuthology.orchestra.run.smithi183.stdout: "paxos_propose_interval": "1.000000", 2024-09-10T19:11:58.374 INFO:teuthology.orchestra.run.smithi183.stdout: "paxos_service_trim_max": "500", 2024-09-10T19:11:58.374 INFO:teuthology.orchestra.run.smithi183.stdout: "paxos_service_trim_max_multiplier": "20", 2024-09-10T19:11:58.374 INFO:teuthology.orchestra.run.smithi183.stdout: "paxos_service_trim_min": "10", 2024-09-10T19:11:58.374 INFO:teuthology.orchestra.run.smithi183.stdout: "paxos_stash_full_interval": "25", 2024-09-10T19:11:58.374 INFO:teuthology.orchestra.run.smithi183.stdout: "paxos_trim_max": "500", 2024-09-10T19:11:58.374 INFO:teuthology.orchestra.run.smithi183.stdout: "paxos_trim_min": "250", 2024-09-10T19:11:58.374 INFO:teuthology.orchestra.run.smithi183.stdout: "perf": "true", 2024-09-10T19:11:58.374 INFO:teuthology.orchestra.run.smithi183.stdout: "pid_file": "", 2024-09-10T19:11:58.375 INFO:teuthology.orchestra.run.smithi183.stdout: "plugin_crypto_accelerator": "crypto_isal", 2024-09-10T19:11:58.375 INFO:teuthology.orchestra.run.smithi183.stdout: "plugin_dir": "/usr/lib64/ceph", 2024-09-10T19:11:58.375 INFO:teuthology.orchestra.run.smithi183.stdout: "public_addr": "-", 2024-09-10T19:11:58.375 INFO:teuthology.orchestra.run.smithi183.stdout: "public_addrv": "", 2024-09-10T19:11:58.375 INFO:teuthology.orchestra.run.smithi183.stdout: "public_bind_addr": "-", 2024-09-10T19:11:58.375 INFO:teuthology.orchestra.run.smithi183.stdout: "public_network": "", 2024-09-10T19:11:58.375 INFO:teuthology.orchestra.run.smithi183.stdout: "public_network_interface": "", 2024-09-10T19:11:58.375 INFO:teuthology.orchestra.run.smithi183.stdout: "qat_compressor_enabled": "false", 2024-09-10T19:11:58.375 INFO:teuthology.orchestra.run.smithi183.stdout: "rados_mon_op_timeout": "0", 2024-09-10T19:11:58.375 INFO:teuthology.orchestra.run.smithi183.stdout: "rados_osd_op_timeout": "0", 2024-09-10T19:11:58.375 INFO:teuthology.orchestra.run.smithi183.stdout: "rados_tracing": "false", 2024-09-10T19:11:58.375 INFO:teuthology.orchestra.run.smithi183.stdout: "rbd_atime_update_interval": "60", 2024-09-10T19:11:58.375 INFO:teuthology.orchestra.run.smithi183.stdout: "rbd_auto_exclusive_lock_until_manual_request": "true", 2024-09-10T19:11:58.375 INFO:teuthology.orchestra.run.smithi183.stdout: "rbd_balance_parent_reads": "false", 2024-09-10T19:11:58.376 INFO:teuthology.orchestra.run.smithi183.stdout: "rbd_balance_snap_reads": "false", 2024-09-10T19:11:58.376 INFO:teuthology.orchestra.run.smithi183.stdout: "rbd_blkin_trace_all": "false", 2024-09-10T19:11:58.376 INFO:teuthology.orchestra.run.smithi183.stdout: "rbd_blocklist_expire_seconds": "0", 2024-09-10T19:11:58.376 INFO:teuthology.orchestra.run.smithi183.stdout: "rbd_blocklist_on_break_lock": "true", 2024-09-10T19:11:58.376 INFO:teuthology.orchestra.run.smithi183.stdout: "rbd_cache": "true", 2024-09-10T19:11:58.376 INFO:teuthology.orchestra.run.smithi183.stdout: "rbd_cache_block_writes_upfront": "false", 2024-09-10T19:11:58.376 INFO:teuthology.orchestra.run.smithi183.stdout: "rbd_cache_max_dirty": "25165824", 2024-09-10T19:11:58.376 INFO:teuthology.orchestra.run.smithi183.stdout: "rbd_cache_max_dirty_age": "1.000000", 2024-09-10T19:11:58.376 INFO:teuthology.orchestra.run.smithi183.stdout: "rbd_cache_max_dirty_object": "0", 2024-09-10T19:11:58.376 INFO:teuthology.orchestra.run.smithi183.stdout: "rbd_cache_policy": "writearound", 2024-09-10T19:11:58.376 INFO:teuthology.orchestra.run.smithi183.stdout: "rbd_cache_size": "33554432", 2024-09-10T19:11:58.376 INFO:teuthology.orchestra.run.smithi183.stdout: "rbd_cache_target_dirty": "16777216", 2024-09-10T19:11:58.376 INFO:teuthology.orchestra.run.smithi183.stdout: "rbd_cache_writethrough_until_flush": "true", 2024-09-10T19:11:58.376 INFO:teuthology.orchestra.run.smithi183.stdout: "rbd_clone_copy_on_read": "false", 2024-09-10T19:11:58.377 INFO:teuthology.orchestra.run.smithi183.stdout: "rbd_compression_hint": "none", 2024-09-10T19:11:58.377 INFO:teuthology.orchestra.run.smithi183.stdout: "rbd_concurrent_management_ops": "10", 2024-09-10T19:11:58.377 INFO:teuthology.orchestra.run.smithi183.stdout: "rbd_config_pool_override_update_timestamp": "0", 2024-09-10T19:11:58.377 INFO:teuthology.orchestra.run.smithi183.stdout: "rbd_default_clone_format": "auto", 2024-09-10T19:11:58.377 INFO:teuthology.orchestra.run.smithi183.stdout: "rbd_default_data_pool": "", 2024-09-10T19:11:58.377 INFO:teuthology.orchestra.run.smithi183.stdout: "rbd_default_features": "61", 2024-09-10T19:11:58.377 INFO:teuthology.orchestra.run.smithi183.stdout: "rbd_default_format": "2", 2024-09-10T19:11:58.377 INFO:teuthology.orchestra.run.smithi183.stdout: "rbd_default_map_options": "", 2024-09-10T19:11:58.377 INFO:teuthology.orchestra.run.smithi183.stdout: "rbd_default_order": "22", 2024-09-10T19:11:58.377 INFO:teuthology.orchestra.run.smithi183.stdout: "rbd_default_pool": "rbd", 2024-09-10T19:11:58.377 INFO:teuthology.orchestra.run.smithi183.stdout: "rbd_default_snapshot_quiesce_mode": "required", 2024-09-10T19:11:58.377 INFO:teuthology.orchestra.run.smithi183.stdout: "rbd_default_stripe_count": "0", 2024-09-10T19:11:58.377 INFO:teuthology.orchestra.run.smithi183.stdout: "rbd_default_stripe_unit": "0", 2024-09-10T19:11:58.377 INFO:teuthology.orchestra.run.smithi183.stdout: "rbd_disable_zero_copy_writes": "true", 2024-09-10T19:11:58.378 INFO:teuthology.orchestra.run.smithi183.stdout: "rbd_discard_granularity_bytes": "65536", 2024-09-10T19:11:58.378 INFO:teuthology.orchestra.run.smithi183.stdout: "rbd_discard_on_zeroed_write_same": "true", 2024-09-10T19:11:58.378 INFO:teuthology.orchestra.run.smithi183.stdout: "rbd_enable_alloc_hint": "true", 2024-09-10T19:11:58.378 INFO:teuthology.orchestra.run.smithi183.stdout: "rbd_invalidate_object_map_on_timeout": "true", 2024-09-10T19:11:58.378 INFO:teuthology.orchestra.run.smithi183.stdout: "rbd_io_scheduler": "simple", 2024-09-10T19:11:58.378 INFO:teuthology.orchestra.run.smithi183.stdout: "rbd_io_scheduler_simple_max_delay": "0", 2024-09-10T19:11:58.378 INFO:teuthology.orchestra.run.smithi183.stdout: "rbd_journal_commit_age": "5.000000", 2024-09-10T19:11:58.378 INFO:teuthology.orchestra.run.smithi183.stdout: "rbd_journal_max_concurrent_object_sets": "0", 2024-09-10T19:11:58.378 INFO:teuthology.orchestra.run.smithi183.stdout: "rbd_journal_max_payload_bytes": "16384", 2024-09-10T19:11:58.378 INFO:teuthology.orchestra.run.smithi183.stdout: "rbd_journal_object_flush_age": "0.000000", 2024-09-10T19:11:58.378 INFO:teuthology.orchestra.run.smithi183.stdout: "rbd_journal_object_flush_bytes": "1048576", 2024-09-10T19:11:58.378 INFO:teuthology.orchestra.run.smithi183.stdout: "rbd_journal_object_flush_interval": "0", 2024-09-10T19:11:58.378 INFO:teuthology.orchestra.run.smithi183.stdout: "rbd_journal_object_max_in_flight_appends": "0", 2024-09-10T19:11:58.378 INFO:teuthology.orchestra.run.smithi183.stdout: "rbd_journal_object_writethrough_until_flush": "true", 2024-09-10T19:11:58.379 INFO:teuthology.orchestra.run.smithi183.stdout: "rbd_journal_order": "24", 2024-09-10T19:11:58.379 INFO:teuthology.orchestra.run.smithi183.stdout: "rbd_journal_pool": "", 2024-09-10T19:11:58.379 INFO:teuthology.orchestra.run.smithi183.stdout: "rbd_journal_splay_width": "4", 2024-09-10T19:11:58.379 INFO:teuthology.orchestra.run.smithi183.stdout: "rbd_localize_parent_reads": "false", 2024-09-10T19:11:58.379 INFO:teuthology.orchestra.run.smithi183.stdout: "rbd_localize_snap_reads": "false", 2024-09-10T19:11:58.379 INFO:teuthology.orchestra.run.smithi183.stdout: "rbd_mirror_concurrent_image_deletions": "1", 2024-09-10T19:11:58.379 INFO:teuthology.orchestra.run.smithi183.stdout: "rbd_mirror_concurrent_image_syncs": "5", 2024-09-10T19:11:58.379 INFO:teuthology.orchestra.run.smithi183.stdout: "rbd_mirror_delete_retry_interval": "30.000000", 2024-09-10T19:11:58.379 INFO:teuthology.orchestra.run.smithi183.stdout: "rbd_mirror_image_perf_stats_prio": "5", 2024-09-10T19:11:58.379 INFO:teuthology.orchestra.run.smithi183.stdout: "rbd_mirror_image_policy_migration_throttle": "300", 2024-09-10T19:11:58.379 INFO:teuthology.orchestra.run.smithi183.stdout: "rbd_mirror_image_policy_rebalance_timeout": "0.000000", 2024-09-10T19:11:58.379 INFO:teuthology.orchestra.run.smithi183.stdout: "rbd_mirror_image_policy_type": "simple", 2024-09-10T19:11:58.379 INFO:teuthology.orchestra.run.smithi183.stdout: "rbd_mirror_image_policy_update_throttle_interval": "1.000000", 2024-09-10T19:11:58.379 INFO:teuthology.orchestra.run.smithi183.stdout: "rbd_mirror_image_state_check_interval": "30", 2024-09-10T19:11:58.380 INFO:teuthology.orchestra.run.smithi183.stdout: "rbd_mirror_journal_commit_age": "5.000000", 2024-09-10T19:11:58.380 INFO:teuthology.orchestra.run.smithi183.stdout: "rbd_mirror_journal_poll_age": "5.000000", 2024-09-10T19:11:58.380 INFO:teuthology.orchestra.run.smithi183.stdout: "rbd_mirror_leader_heartbeat_interval": "5", 2024-09-10T19:11:58.380 INFO:teuthology.orchestra.run.smithi183.stdout: "rbd_mirror_leader_max_acquire_attempts_before_break": "3", 2024-09-10T19:11:58.380 INFO:teuthology.orchestra.run.smithi183.stdout: "rbd_mirror_leader_max_missed_heartbeats": "2", 2024-09-10T19:11:58.380 INFO:teuthology.orchestra.run.smithi183.stdout: "rbd_mirror_memory_autotune": "true", 2024-09-10T19:11:58.380 INFO:teuthology.orchestra.run.smithi183.stdout: "rbd_mirror_memory_base": "805306368", 2024-09-10T19:11:58.380 INFO:teuthology.orchestra.run.smithi183.stdout: "rbd_mirror_memory_cache_autotune_interval": "30.000000", 2024-09-10T19:11:58.380 INFO:teuthology.orchestra.run.smithi183.stdout: "rbd_mirror_memory_cache_min": "134217728", 2024-09-10T19:11:58.380 INFO:teuthology.orchestra.run.smithi183.stdout: "rbd_mirror_memory_cache_resize_interval": "5.000000", 2024-09-10T19:11:58.380 INFO:teuthology.orchestra.run.smithi183.stdout: "rbd_mirror_memory_expected_fragmentation": "0.150000", 2024-09-10T19:11:58.380 INFO:teuthology.orchestra.run.smithi183.stdout: "rbd_mirror_memory_target": "4294967296", 2024-09-10T19:11:58.380 INFO:teuthology.orchestra.run.smithi183.stdout: "rbd_mirror_perf_stats_prio": "5", 2024-09-10T19:11:58.380 INFO:teuthology.orchestra.run.smithi183.stdout: "rbd_mirror_pool_replayers_refresh_interval": "30", 2024-09-10T19:11:58.381 INFO:teuthology.orchestra.run.smithi183.stdout: "rbd_mirror_sync_point_update_age": "30.000000", 2024-09-10T19:11:58.381 INFO:teuthology.orchestra.run.smithi183.stdout: "rbd_mirroring_delete_delay": "0", 2024-09-10T19:11:58.381 INFO:teuthology.orchestra.run.smithi183.stdout: "rbd_mirroring_max_mirroring_snapshots": "5", 2024-09-10T19:11:58.381 INFO:teuthology.orchestra.run.smithi183.stdout: "rbd_mirroring_replay_delay": "0", 2024-09-10T19:11:58.381 INFO:teuthology.orchestra.run.smithi183.stdout: "rbd_mirroring_resync_after_disconnect": "false", 2024-09-10T19:11:58.381 INFO:teuthology.orchestra.run.smithi183.stdout: "rbd_move_parent_to_trash_on_remove": "false", 2024-09-10T19:11:58.381 INFO:teuthology.orchestra.run.smithi183.stdout: "rbd_move_to_trash_on_remove": "false", 2024-09-10T19:11:58.381 INFO:teuthology.orchestra.run.smithi183.stdout: "rbd_move_to_trash_on_remove_expire_seconds": "0", 2024-09-10T19:11:58.381 INFO:teuthology.orchestra.run.smithi183.stdout: "rbd_mtime_update_interval": "60", 2024-09-10T19:11:58.381 INFO:teuthology.orchestra.run.smithi183.stdout: "rbd_non_blocking_aio": "true", 2024-09-10T19:11:58.381 INFO:teuthology.orchestra.run.smithi183.stdout: "rbd_op_thread_timeout": "60", 2024-09-10T19:11:58.381 INFO:teuthology.orchestra.run.smithi183.stdout: "rbd_op_threads": "1", 2024-09-10T19:11:58.381 INFO:teuthology.orchestra.run.smithi183.stdout: "rbd_parent_cache_enabled": "false", 2024-09-10T19:11:58.381 INFO:teuthology.orchestra.run.smithi183.stdout: "rbd_persistent_cache_mode": "disabled", 2024-09-10T19:11:58.382 INFO:teuthology.orchestra.run.smithi183.stdout: "rbd_persistent_cache_path": "/tmp", 2024-09-10T19:11:58.382 INFO:teuthology.orchestra.run.smithi183.stdout: "rbd_persistent_cache_size": "1073741824", 2024-09-10T19:11:58.382 INFO:teuthology.orchestra.run.smithi183.stdout: "rbd_plugins": "", 2024-09-10T19:11:58.382 INFO:teuthology.orchestra.run.smithi183.stdout: "rbd_qos_bps_burst": "0", 2024-09-10T19:11:58.382 INFO:teuthology.orchestra.run.smithi183.stdout: "rbd_qos_bps_burst_seconds": "1", 2024-09-10T19:11:58.382 INFO:teuthology.orchestra.run.smithi183.stdout: "rbd_qos_bps_limit": "0", 2024-09-10T19:11:58.382 INFO:teuthology.orchestra.run.smithi183.stdout: "rbd_qos_exclude_ops": "0", 2024-09-10T19:11:58.382 INFO:teuthology.orchestra.run.smithi183.stdout: "rbd_qos_iops_burst": "0", 2024-09-10T19:11:58.382 INFO:teuthology.orchestra.run.smithi183.stdout: "rbd_qos_iops_burst_seconds": "1", 2024-09-10T19:11:58.382 INFO:teuthology.orchestra.run.smithi183.stdout: "rbd_qos_iops_limit": "0", 2024-09-10T19:11:58.382 INFO:teuthology.orchestra.run.smithi183.stdout: "rbd_qos_read_bps_burst": "0", 2024-09-10T19:11:58.382 INFO:teuthology.orchestra.run.smithi183.stdout: "rbd_qos_read_bps_burst_seconds": "1", 2024-09-10T19:11:58.382 INFO:teuthology.orchestra.run.smithi183.stdout: "rbd_qos_read_bps_limit": "0", 2024-09-10T19:11:58.382 INFO:teuthology.orchestra.run.smithi183.stdout: "rbd_qos_read_iops_burst": "0", 2024-09-10T19:11:58.382 INFO:teuthology.orchestra.run.smithi183.stdout: "rbd_qos_read_iops_burst_seconds": "1", 2024-09-10T19:11:58.383 INFO:teuthology.orchestra.run.smithi183.stdout: "rbd_qos_read_iops_limit": "0", 2024-09-10T19:11:58.383 INFO:teuthology.orchestra.run.smithi183.stdout: "rbd_qos_schedule_tick_min": "50", 2024-09-10T19:11:58.383 INFO:teuthology.orchestra.run.smithi183.stdout: "rbd_qos_write_bps_burst": "0", 2024-09-10T19:11:58.383 INFO:teuthology.orchestra.run.smithi183.stdout: "rbd_qos_write_bps_burst_seconds": "1", 2024-09-10T19:11:58.383 INFO:teuthology.orchestra.run.smithi183.stdout: "rbd_qos_write_bps_limit": "0", 2024-09-10T19:11:58.383 INFO:teuthology.orchestra.run.smithi183.stdout: "rbd_qos_write_iops_burst": "0", 2024-09-10T19:11:58.383 INFO:teuthology.orchestra.run.smithi183.stdout: "rbd_qos_write_iops_burst_seconds": "1", 2024-09-10T19:11:58.383 INFO:teuthology.orchestra.run.smithi183.stdout: "rbd_qos_write_iops_limit": "0", 2024-09-10T19:11:58.383 INFO:teuthology.orchestra.run.smithi183.stdout: "rbd_quiesce_notification_attempts": "10", 2024-09-10T19:11:58.383 INFO:teuthology.orchestra.run.smithi183.stdout: "rbd_read_from_replica_policy": "default", 2024-09-10T19:11:58.383 INFO:teuthology.orchestra.run.smithi183.stdout: "rbd_readahead_disable_after_bytes": "52428800", 2024-09-10T19:11:58.383 INFO:teuthology.orchestra.run.smithi183.stdout: "rbd_readahead_max_bytes": "524288", 2024-09-10T19:11:58.383 INFO:teuthology.orchestra.run.smithi183.stdout: "rbd_readahead_trigger_requests": "10", 2024-09-10T19:11:58.383 INFO:teuthology.orchestra.run.smithi183.stdout: "rbd_request_timed_out_seconds": "30", 2024-09-10T19:11:58.384 INFO:teuthology.orchestra.run.smithi183.stdout: "rbd_skip_partial_discard": "true", 2024-09-10T19:11:58.384 INFO:teuthology.orchestra.run.smithi183.stdout: "rbd_sparse_read_threshold_bytes": "65536", 2024-09-10T19:11:58.384 INFO:teuthology.orchestra.run.smithi183.stdout: "rbd_tracing": "false", 2024-09-10T19:11:58.384 INFO:teuthology.orchestra.run.smithi183.stdout: "rbd_validate_names": "true", 2024-09-10T19:11:58.384 INFO:teuthology.orchestra.run.smithi183.stdout: "rbd_validate_pool": "true", 2024-09-10T19:11:58.384 INFO:teuthology.orchestra.run.smithi183.stdout: "restapi_base_url": "", 2024-09-10T19:11:58.384 INFO:teuthology.orchestra.run.smithi183.stdout: "restapi_log_level": "", 2024-09-10T19:11:58.384 INFO:teuthology.orchestra.run.smithi183.stdout: "rgw_acl_grants_max_num": "100", 2024-09-10T19:11:58.384 INFO:teuthology.orchestra.run.smithi183.stdout: "rgw_admin_entry": "admin", 2024-09-10T19:11:58.384 INFO:teuthology.orchestra.run.smithi183.stdout: "rgw_backend_store": "rados", 2024-09-10T19:11:58.384 INFO:teuthology.orchestra.run.smithi183.stdout: "rgw_barbican_url": "", 2024-09-10T19:11:58.384 INFO:teuthology.orchestra.run.smithi183.stdout: "rgw_beast_enable_async": "true", 2024-09-10T19:11:58.384 INFO:teuthology.orchestra.run.smithi183.stdout: "rgw_bucket_default_quota_max_objects": "-1", 2024-09-10T19:11:58.384 INFO:teuthology.orchestra.run.smithi183.stdout: "rgw_bucket_default_quota_max_size": "-1", 2024-09-10T19:11:58.385 INFO:teuthology.orchestra.run.smithi183.stdout: "rgw_bucket_index_max_aio": "128", 2024-09-10T19:11:58.385 INFO:teuthology.orchestra.run.smithi183.stdout: "rgw_bucket_quota_cache_size": "10000", 2024-09-10T19:11:58.385 INFO:teuthology.orchestra.run.smithi183.stdout: "rgw_bucket_quota_ttl": "600", 2024-09-10T19:11:58.385 INFO:teuthology.orchestra.run.smithi183.stdout: "rgw_cache_enabled": "true", 2024-09-10T19:11:58.385 INFO:teuthology.orchestra.run.smithi183.stdout: "rgw_cache_expiry_interval": "900", 2024-09-10T19:11:58.385 INFO:teuthology.orchestra.run.smithi183.stdout: "rgw_cache_lru_size": "10000", 2024-09-10T19:11:58.385 INFO:teuthology.orchestra.run.smithi183.stdout: "rgw_content_length_compat": "false", 2024-09-10T19:11:58.385 INFO:teuthology.orchestra.run.smithi183.stdout: "rgw_copy_obj_progress": "true", 2024-09-10T19:11:58.385 INFO:teuthology.orchestra.run.smithi183.stdout: "rgw_copy_obj_progress_every_bytes": "1048576", 2024-09-10T19:11:58.385 INFO:teuthology.orchestra.run.smithi183.stdout: "rgw_cors_rules_max_num": "100", 2024-09-10T19:11:58.385 INFO:teuthology.orchestra.run.smithi183.stdout: "rgw_cross_domain_policy": "", 2024-09-10T19:11:58.385 INFO:teuthology.orchestra.run.smithi183.stdout: "rgw_crypt_default_encryption_key": "", 2024-09-10T19:11:58.385 INFO:teuthology.orchestra.run.smithi183.stdout: "rgw_crypt_kmip_addr": "", 2024-09-10T19:11:58.385 INFO:teuthology.orchestra.run.smithi183.stdout: "rgw_crypt_kmip_ca_path": "", 2024-09-10T19:11:58.386 INFO:teuthology.orchestra.run.smithi183.stdout: "rgw_crypt_kmip_client_cert": "", 2024-09-10T19:11:58.386 INFO:teuthology.orchestra.run.smithi183.stdout: "rgw_crypt_kmip_client_key": "", 2024-09-10T19:11:58.386 INFO:teuthology.orchestra.run.smithi183.stdout: "rgw_crypt_kmip_kms_key_template": "", 2024-09-10T19:11:58.386 INFO:teuthology.orchestra.run.smithi183.stdout: "rgw_crypt_kmip_password": "", 2024-09-10T19:11:58.386 INFO:teuthology.orchestra.run.smithi183.stdout: "rgw_crypt_kmip_s3_key_template": "$keyid", 2024-09-10T19:11:58.386 INFO:teuthology.orchestra.run.smithi183.stdout: "rgw_crypt_kmip_username": "", 2024-09-10T19:11:58.386 INFO:teuthology.orchestra.run.smithi183.stdout: "rgw_crypt_require_ssl": "true", 2024-09-10T19:11:58.386 INFO:teuthology.orchestra.run.smithi183.stdout: "rgw_crypt_s3_kms_backend": "barbican", 2024-09-10T19:11:58.386 INFO:teuthology.orchestra.run.smithi183.stdout: "rgw_crypt_s3_kms_encryption_keys": "", 2024-09-10T19:11:58.386 INFO:teuthology.orchestra.run.smithi183.stdout: "rgw_crypt_sse_s3_backend": "vault", 2024-09-10T19:11:58.386 INFO:teuthology.orchestra.run.smithi183.stdout: "rgw_crypt_sse_s3_key_template": "%bucket_id", 2024-09-10T19:11:58.386 INFO:teuthology.orchestra.run.smithi183.stdout: "rgw_crypt_sse_s3_vault_addr": "", 2024-09-10T19:11:58.386 INFO:teuthology.orchestra.run.smithi183.stdout: "rgw_crypt_sse_s3_vault_auth": "token", 2024-09-10T19:11:58.386 INFO:teuthology.orchestra.run.smithi183.stdout: "rgw_crypt_sse_s3_vault_namespace": "", 2024-09-10T19:11:58.387 INFO:teuthology.orchestra.run.smithi183.stdout: "rgw_crypt_sse_s3_vault_prefix": "", 2024-09-10T19:11:58.387 INFO:teuthology.orchestra.run.smithi183.stdout: "rgw_crypt_sse_s3_vault_secret_engine": "transit", 2024-09-10T19:11:58.387 INFO:teuthology.orchestra.run.smithi183.stdout: "rgw_crypt_sse_s3_vault_ssl_cacert": "", 2024-09-10T19:11:58.387 INFO:teuthology.orchestra.run.smithi183.stdout: "rgw_crypt_sse_s3_vault_ssl_clientcert": "", 2024-09-10T19:11:58.387 INFO:teuthology.orchestra.run.smithi183.stdout: "rgw_crypt_sse_s3_vault_ssl_clientkey": "", 2024-09-10T19:11:58.387 INFO:teuthology.orchestra.run.smithi183.stdout: "rgw_crypt_sse_s3_vault_token_file": "", 2024-09-10T19:11:58.387 INFO:teuthology.orchestra.run.smithi183.stdout: "rgw_crypt_sse_s3_vault_verify_ssl": "true", 2024-09-10T19:11:58.387 INFO:teuthology.orchestra.run.smithi183.stdout: "rgw_crypt_suppress_logs": "true", 2024-09-10T19:11:58.387 INFO:teuthology.orchestra.run.smithi183.stdout: "rgw_crypt_vault_addr": "", 2024-09-10T19:11:58.387 INFO:teuthology.orchestra.run.smithi183.stdout: "rgw_crypt_vault_auth": "token", 2024-09-10T19:11:58.388 INFO:teuthology.orchestra.run.smithi183.stdout: "rgw_crypt_vault_namespace": "", 2024-09-10T19:11:58.388 INFO:teuthology.orchestra.run.smithi183.stdout: "rgw_crypt_vault_prefix": "", 2024-09-10T19:11:58.388 INFO:teuthology.orchestra.run.smithi183.stdout: "rgw_crypt_vault_secret_engine": "transit", 2024-09-10T19:11:58.388 INFO:teuthology.orchestra.run.smithi183.stdout: "rgw_crypt_vault_ssl_cacert": "", 2024-09-10T19:11:58.388 INFO:teuthology.orchestra.run.smithi183.stdout: "rgw_crypt_vault_ssl_clientcert": "", 2024-09-10T19:11:58.388 INFO:teuthology.orchestra.run.smithi183.stdout: "rgw_crypt_vault_ssl_clientkey": "", 2024-09-10T19:11:58.388 INFO:teuthology.orchestra.run.smithi183.stdout: "rgw_crypt_vault_token_file": "", 2024-09-10T19:11:58.388 INFO:teuthology.orchestra.run.smithi183.stdout: "rgw_crypt_vault_verify_ssl": "true", 2024-09-10T19:11:58.388 INFO:teuthology.orchestra.run.smithi183.stdout: "rgw_curl_buffersize": "524288", 2024-09-10T19:11:58.388 INFO:teuthology.orchestra.run.smithi183.stdout: "rgw_curl_low_speed_limit": "1024", 2024-09-10T19:11:58.388 INFO:teuthology.orchestra.run.smithi183.stdout: "rgw_curl_low_speed_time": "300", 2024-09-10T19:11:58.388 INFO:teuthology.orchestra.run.smithi183.stdout: "rgw_curl_wait_timeout_ms": "1000", 2024-09-10T19:11:58.388 INFO:teuthology.orchestra.run.smithi183.stdout: "rgw_d3n_l1_datacache_persistent_path": "/tmp/rgw_datacache/", 2024-09-10T19:11:58.389 INFO:teuthology.orchestra.run.smithi183.stdout: "rgw_d3n_l1_datacache_size": "1073741824", 2024-09-10T19:11:58.389 INFO:teuthology.orchestra.run.smithi183.stdout: "rgw_d3n_l1_evict_cache_on_start": "true", 2024-09-10T19:11:58.389 INFO:teuthology.orchestra.run.smithi183.stdout: "rgw_d3n_l1_eviction_policy": "lru", 2024-09-10T19:11:58.389 INFO:teuthology.orchestra.run.smithi183.stdout: "rgw_d3n_l1_fadvise": "4", 2024-09-10T19:11:58.389 INFO:teuthology.orchestra.run.smithi183.stdout: "rgw_d3n_l1_local_datacache_enabled": "false", 2024-09-10T19:11:58.389 INFO:teuthology.orchestra.run.smithi183.stdout: "rgw_d3n_libaio_aio_num": "64", 2024-09-10T19:11:58.389 INFO:teuthology.orchestra.run.smithi183.stdout: "rgw_d3n_libaio_aio_threads": "20", 2024-09-10T19:11:58.389 INFO:teuthology.orchestra.run.smithi183.stdout: "rgw_data": "/var/lib/ceph/radosgw/ceph-a", 2024-09-10T19:11:58.389 INFO:teuthology.orchestra.run.smithi183.stdout: "rgw_data_log_changes_size": "1000", 2024-09-10T19:11:58.389 INFO:teuthology.orchestra.run.smithi183.stdout: "rgw_data_log_num_shards": "128", 2024-09-10T19:11:58.389 INFO:teuthology.orchestra.run.smithi183.stdout: "rgw_data_log_obj_prefix": "data_log", 2024-09-10T19:11:58.389 INFO:teuthology.orchestra.run.smithi183.stdout: "rgw_data_log_window": "30", 2024-09-10T19:11:58.389 INFO:teuthology.orchestra.run.smithi183.stdout: "rgw_data_notify_interval_msec": "200", 2024-09-10T19:11:58.389 INFO:teuthology.orchestra.run.smithi183.stdout: "rgw_debug_inject_latency_bi_unlink": "0", 2024-09-10T19:11:58.390 INFO:teuthology.orchestra.run.smithi183.stdout: "rgw_debug_inject_olh_cancel_modification_err": "false", 2024-09-10T19:11:58.390 INFO:teuthology.orchestra.run.smithi183.stdout: "rgw_debug_inject_set_olh_err": "0", 2024-09-10T19:11:58.390 INFO:teuthology.orchestra.run.smithi183.stdout: "rgw_default_data_log_backing": "fifo", 2024-09-10T19:11:58.390 INFO:teuthology.orchestra.run.smithi183.stdout: "rgw_default_realm_info_oid": "default.realm", 2024-09-10T19:11:58.390 INFO:teuthology.orchestra.run.smithi183.stdout: "rgw_default_region_info_oid": "default.region", 2024-09-10T19:11:58.390 INFO:teuthology.orchestra.run.smithi183.stdout: "rgw_default_zone_info_oid": "default.zone", 2024-09-10T19:11:58.390 INFO:teuthology.orchestra.run.smithi183.stdout: "rgw_default_zonegroup_info_oid": "default.zonegroup", 2024-09-10T19:11:58.390 INFO:teuthology.orchestra.run.smithi183.stdout: "rgw_defer_to_bucket_acls": "", 2024-09-10T19:11:58.390 INFO:teuthology.orchestra.run.smithi183.stdout: "rgw_delete_multi_obj_max_num": "1000", 2024-09-10T19:11:58.390 INFO:teuthology.orchestra.run.smithi183.stdout: "rgw_dmclock_admin_lim": "0.000000", 2024-09-10T19:11:58.390 INFO:teuthology.orchestra.run.smithi183.stdout: "rgw_dmclock_admin_res": "100.000000", 2024-09-10T19:11:58.390 INFO:teuthology.orchestra.run.smithi183.stdout: "rgw_dmclock_admin_wgt": "100.000000", 2024-09-10T19:11:58.390 INFO:teuthology.orchestra.run.smithi183.stdout: "rgw_dmclock_auth_lim": "0.000000", 2024-09-10T19:11:58.390 INFO:teuthology.orchestra.run.smithi183.stdout: "rgw_dmclock_auth_res": "200.000000", 2024-09-10T19:11:58.390 INFO:teuthology.orchestra.run.smithi183.stdout: "rgw_dmclock_auth_wgt": "100.000000", 2024-09-10T19:11:58.391 INFO:teuthology.orchestra.run.smithi183.stdout: "rgw_dmclock_data_lim": "0.000000", 2024-09-10T19:11:58.391 INFO:teuthology.orchestra.run.smithi183.stdout: "rgw_dmclock_data_res": "500.000000", 2024-09-10T19:11:58.391 INFO:teuthology.orchestra.run.smithi183.stdout: "rgw_dmclock_data_wgt": "500.000000", 2024-09-10T19:11:58.391 INFO:teuthology.orchestra.run.smithi183.stdout: "rgw_dmclock_metadata_lim": "0.000000", 2024-09-10T19:11:58.391 INFO:teuthology.orchestra.run.smithi183.stdout: "rgw_dmclock_metadata_res": "500.000000", 2024-09-10T19:11:58.391 INFO:teuthology.orchestra.run.smithi183.stdout: "rgw_dmclock_metadata_wgt": "500.000000", 2024-09-10T19:11:58.391 INFO:teuthology.orchestra.run.smithi183.stdout: "rgw_dns_name": "", 2024-09-10T19:11:58.391 INFO:teuthology.orchestra.run.smithi183.stdout: "rgw_dns_s3website_name": "", 2024-09-10T19:11:58.391 INFO:teuthology.orchestra.run.smithi183.stdout: "rgw_dynamic_resharding": "true", 2024-09-10T19:11:58.391 INFO:teuthology.orchestra.run.smithi183.stdout: "rgw_enable_apis": "s3, s3website, swift, swift_auth, admin, sts, iam, notifications", 2024-09-10T19:11:58.391 INFO:teuthology.orchestra.run.smithi183.stdout: "rgw_enable_gc_threads": "true", 2024-09-10T19:11:58.392 INFO:teuthology.orchestra.run.smithi183.stdout: "rgw_enable_lc_threads": "true", 2024-09-10T19:11:58.392 INFO:teuthology.orchestra.run.smithi183.stdout: "rgw_enable_ops_log": "false", 2024-09-10T19:11:58.392 INFO:teuthology.orchestra.run.smithi183.stdout: "rgw_enable_quota_threads": "true", 2024-09-10T19:11:58.392 INFO:teuthology.orchestra.run.smithi183.stdout: "rgw_enable_static_website": "false", 2024-09-10T19:11:58.392 INFO:teuthology.orchestra.run.smithi183.stdout: "rgw_enable_usage_log": "false", 2024-09-10T19:11:58.392 INFO:teuthology.orchestra.run.smithi183.stdout: "rgw_enforce_swift_acls": "true", 2024-09-10T19:11:58.392 INFO:teuthology.orchestra.run.smithi183.stdout: "rgw_exit_timeout_secs": "120", 2024-09-10T19:11:58.392 INFO:teuthology.orchestra.run.smithi183.stdout: "rgw_expose_bucket": "false", 2024-09-10T19:11:58.392 INFO:teuthology.orchestra.run.smithi183.stdout: "rgw_extended_http_attrs": "", 2024-09-10T19:11:58.392 INFO:teuthology.orchestra.run.smithi183.stdout: "rgw_frontend_defaults": "beast ssl_certificate=config://rgw/cert/$realm/$zone.crt ssl_private_key=config://rgw/cert/$realm/$zone.key", 2024-09-10T19:11:58.392 INFO:teuthology.orchestra.run.smithi183.stdout: "rgw_frontends": "beast port=7480", 2024-09-10T19:11:58.392 INFO:teuthology.orchestra.run.smithi183.stdout: "rgw_gc_max_concurrent_io": "10", 2024-09-10T19:11:58.393 INFO:teuthology.orchestra.run.smithi183.stdout: "rgw_gc_max_deferred": "50", 2024-09-10T19:11:58.393 INFO:teuthology.orchestra.run.smithi183.stdout: "rgw_gc_max_deferred_entries_size": "3072", 2024-09-10T19:11:58.393 INFO:teuthology.orchestra.run.smithi183.stdout: "rgw_gc_max_objs": "32", 2024-09-10T19:11:58.393 INFO:teuthology.orchestra.run.smithi183.stdout: "rgw_gc_max_queue_size": "134213632", 2024-09-10T19:11:58.393 INFO:teuthology.orchestra.run.smithi183.stdout: "rgw_gc_max_trim_chunk": "16", 2024-09-10T19:11:58.393 INFO:teuthology.orchestra.run.smithi183.stdout: "rgw_gc_obj_min_wait": "7200", 2024-09-10T19:11:58.393 INFO:teuthology.orchestra.run.smithi183.stdout: "rgw_gc_processor_max_time": "3600", 2024-09-10T19:11:58.393 INFO:teuthology.orchestra.run.smithi183.stdout: "rgw_gc_processor_period": "3600", 2024-09-10T19:11:58.393 INFO:teuthology.orchestra.run.smithi183.stdout: "rgw_get_obj_max_req_size": "4194304", 2024-09-10T19:11:58.393 INFO:teuthology.orchestra.run.smithi183.stdout: "rgw_get_obj_window_size": "16777216", 2024-09-10T19:11:58.393 INFO:teuthology.orchestra.run.smithi183.stdout: "rgw_healthcheck_disabling_path": "", 2024-09-10T19:11:58.393 INFO:teuthology.orchestra.run.smithi183.stdout: "rgw_ignore_get_invalid_range": "false", 2024-09-10T19:11:58.393 INFO:teuthology.orchestra.run.smithi183.stdout: "rgw_init_timeout": "300", 2024-09-10T19:11:58.393 INFO:teuthology.orchestra.run.smithi183.stdout: "rgw_inject_notify_timeout_probability": "0.000000", 2024-09-10T19:11:58.394 INFO:teuthology.orchestra.run.smithi183.stdout: "rgw_kafka_connection_idle": "300", 2024-09-10T19:11:58.394 INFO:teuthology.orchestra.run.smithi183.stdout: "rgw_kafka_message_timeout": "5000", 2024-09-10T19:11:58.394 INFO:teuthology.orchestra.run.smithi183.stdout: "rgw_keystone_accepted_admin_roles": "", 2024-09-10T19:11:58.394 INFO:teuthology.orchestra.run.smithi183.stdout: "rgw_keystone_accepted_roles": "Member, admin", 2024-09-10T19:11:58.394 INFO:teuthology.orchestra.run.smithi183.stdout: "rgw_keystone_admin_domain": "", 2024-09-10T19:11:58.394 INFO:teuthology.orchestra.run.smithi183.stdout: "rgw_keystone_admin_password": "", 2024-09-10T19:11:58.394 INFO:teuthology.orchestra.run.smithi183.stdout: "rgw_keystone_admin_password_path": "", 2024-09-10T19:11:58.394 INFO:teuthology.orchestra.run.smithi183.stdout: "rgw_keystone_admin_project": "", 2024-09-10T19:11:58.394 INFO:teuthology.orchestra.run.smithi183.stdout: "rgw_keystone_admin_tenant": "", 2024-09-10T19:11:58.394 INFO:teuthology.orchestra.run.smithi183.stdout: "rgw_keystone_admin_token": "", 2024-09-10T19:11:58.394 INFO:teuthology.orchestra.run.smithi183.stdout: "rgw_keystone_admin_token_path": "", 2024-09-10T19:11:58.394 INFO:teuthology.orchestra.run.smithi183.stdout: "rgw_keystone_admin_user": "", 2024-09-10T19:11:58.394 INFO:teuthology.orchestra.run.smithi183.stdout: "rgw_keystone_api_version": "2", 2024-09-10T19:11:58.394 INFO:teuthology.orchestra.run.smithi183.stdout: "rgw_keystone_barbican_domain": "", 2024-09-10T19:11:58.395 INFO:teuthology.orchestra.run.smithi183.stdout: "rgw_keystone_barbican_password": "", 2024-09-10T19:11:58.395 INFO:teuthology.orchestra.run.smithi183.stdout: "rgw_keystone_barbican_project": "", 2024-09-10T19:11:58.395 INFO:teuthology.orchestra.run.smithi183.stdout: "rgw_keystone_barbican_tenant": "", 2024-09-10T19:11:58.395 INFO:teuthology.orchestra.run.smithi183.stdout: "rgw_keystone_barbican_user": "", 2024-09-10T19:11:58.395 INFO:teuthology.orchestra.run.smithi183.stdout: "rgw_keystone_expired_token_cache_expiration": "3600", 2024-09-10T19:11:58.395 INFO:teuthology.orchestra.run.smithi183.stdout: "rgw_keystone_implicit_tenants": "false", 2024-09-10T19:11:58.395 INFO:teuthology.orchestra.run.smithi183.stdout: "rgw_keystone_service_token_accepted_roles": "admin", 2024-09-10T19:11:58.395 INFO:teuthology.orchestra.run.smithi183.stdout: "rgw_keystone_service_token_enabled": "false", 2024-09-10T19:11:58.395 INFO:teuthology.orchestra.run.smithi183.stdout: "rgw_keystone_token_cache_size": "10000", 2024-09-10T19:11:58.395 INFO:teuthology.orchestra.run.smithi183.stdout: "rgw_keystone_url": "", 2024-09-10T19:11:58.395 INFO:teuthology.orchestra.run.smithi183.stdout: "rgw_keystone_verify_ssl": "true", 2024-09-10T19:11:58.395 INFO:teuthology.orchestra.run.smithi183.stdout: "rgw_lc_debug_interval": "-1", 2024-09-10T19:11:58.395 INFO:teuthology.orchestra.run.smithi183.stdout: "rgw_lc_lock_max_time": "90", 2024-09-10T19:11:58.395 INFO:teuthology.orchestra.run.smithi183.stdout: "rgw_lc_max_objs": "32", 2024-09-10T19:11:58.396 INFO:teuthology.orchestra.run.smithi183.stdout: "rgw_lc_max_rules": "1000", 2024-09-10T19:11:58.396 INFO:teuthology.orchestra.run.smithi183.stdout: "rgw_lc_max_worker": "3", 2024-09-10T19:11:58.396 INFO:teuthology.orchestra.run.smithi183.stdout: "rgw_lc_max_wp_worker": "3", 2024-09-10T19:11:58.396 INFO:teuthology.orchestra.run.smithi183.stdout: "rgw_lc_thread_delay": "0", 2024-09-10T19:11:58.396 INFO:teuthology.orchestra.run.smithi183.stdout: "rgw_ldap_binddn": "uid=admin,cn=users,dc=example,dc=com", 2024-09-10T19:11:58.396 INFO:teuthology.orchestra.run.smithi183.stdout: "rgw_ldap_dnattr": "uid", 2024-09-10T19:11:58.396 INFO:teuthology.orchestra.run.smithi183.stdout: "rgw_ldap_searchdn": "cn=users,cn=accounts,dc=example,dc=com", 2024-09-10T19:11:58.396 INFO:teuthology.orchestra.run.smithi183.stdout: "rgw_ldap_searchfilter": "", 2024-09-10T19:11:58.396 INFO:teuthology.orchestra.run.smithi183.stdout: "rgw_ldap_secret": "/etc/openldap/secret", 2024-09-10T19:11:58.396 INFO:teuthology.orchestra.run.smithi183.stdout: "rgw_ldap_uri": "", 2024-09-10T19:11:58.396 INFO:teuthology.orchestra.run.smithi183.stdout: "rgw_lifecycle_work_time": "00:00-06:00", 2024-09-10T19:11:58.396 INFO:teuthology.orchestra.run.smithi183.stdout: "rgw_list_bucket_min_readahead": "1000", 2024-09-10T19:11:58.396 INFO:teuthology.orchestra.run.smithi183.stdout: "rgw_list_buckets_max_chunk": "1000", 2024-09-10T19:11:58.396 INFO:teuthology.orchestra.run.smithi183.stdout: "rgw_log_http_headers": "", 2024-09-10T19:11:58.397 INFO:teuthology.orchestra.run.smithi183.stdout: "rgw_log_nonexistent_bucket": "false", 2024-09-10T19:11:58.397 INFO:teuthology.orchestra.run.smithi183.stdout: "rgw_log_object_name": "%Y-%m-%d-%H-%i-%n", 2024-09-10T19:11:58.397 INFO:teuthology.orchestra.run.smithi183.stdout: "rgw_log_object_name_utc": "false", 2024-09-10T19:11:58.397 INFO:teuthology.orchestra.run.smithi183.stdout: "rgw_luarocks_location": "/tmp/luarocks", 2024-09-10T19:11:58.397 INFO:teuthology.orchestra.run.smithi183.stdout: "rgw_max_attr_name_len": "0", 2024-09-10T19:11:58.397 INFO:teuthology.orchestra.run.smithi183.stdout: "rgw_max_attr_size": "0", 2024-09-10T19:11:58.397 INFO:teuthology.orchestra.run.smithi183.stdout: "rgw_max_attrs_num_in_req": "0", 2024-09-10T19:11:58.397 INFO:teuthology.orchestra.run.smithi183.stdout: "rgw_max_chunk_size": "4194304", 2024-09-10T19:11:58.397 INFO:teuthology.orchestra.run.smithi183.stdout: "rgw_max_concurrent_requests": "1024", 2024-09-10T19:11:58.397 INFO:teuthology.orchestra.run.smithi183.stdout: "rgw_max_dynamic_shards": "1999", 2024-09-10T19:11:58.398 INFO:teuthology.orchestra.run.smithi183.stdout: "rgw_max_listing_results": "1000", 2024-09-10T19:11:58.398 INFO:teuthology.orchestra.run.smithi183.stdout: "rgw_max_notify_retries": "10", 2024-09-10T19:11:58.398 INFO:teuthology.orchestra.run.smithi183.stdout: "rgw_max_objs_per_shard": "100000", 2024-09-10T19:11:58.398 INFO:teuthology.orchestra.run.smithi183.stdout: "rgw_max_put_param_size": "1048576", 2024-09-10T19:11:58.398 INFO:teuthology.orchestra.run.smithi183.stdout: "rgw_max_put_size": "5368709120", 2024-09-10T19:11:58.398 INFO:teuthology.orchestra.run.smithi183.stdout: "rgw_max_slo_entries": "1000", 2024-09-10T19:11:58.398 INFO:teuthology.orchestra.run.smithi183.stdout: "rgw_md_log_max_shards": "64", 2024-09-10T19:11:58.398 INFO:teuthology.orchestra.run.smithi183.stdout: "rgw_md_notify_interval_msec": "200", 2024-09-10T19:11:58.398 INFO:teuthology.orchestra.run.smithi183.stdout: "rgw_mime_types_file": "/etc/mime.types", 2024-09-10T19:11:58.398 INFO:teuthology.orchestra.run.smithi183.stdout: "rgw_mp_lock_max_time": "600", 2024-09-10T19:11:58.399 INFO:teuthology.orchestra.run.smithi183.stdout: "rgw_multi_obj_del_max_aio": "16", 2024-09-10T19:11:58.399 INFO:teuthology.orchestra.run.smithi183.stdout: "rgw_multipart_min_part_size": "5242880", 2024-09-10T19:11:58.399 INFO:teuthology.orchestra.run.smithi183.stdout: "rgw_multipart_part_upload_limit": "10000", 2024-09-10T19:11:58.399 INFO:teuthology.orchestra.run.smithi183.stdout: "rgw_nfs_fhcache_partitions": "3", 2024-09-10T19:11:58.399 INFO:teuthology.orchestra.run.smithi183.stdout: "rgw_nfs_fhcache_size": "2017", 2024-09-10T19:11:58.399 INFO:teuthology.orchestra.run.smithi183.stdout: "rgw_nfs_lru_lane_hiwat": "911", 2024-09-10T19:11:58.399 INFO:teuthology.orchestra.run.smithi183.stdout: "rgw_nfs_lru_lanes": "5", 2024-09-10T19:11:58.399 INFO:teuthology.orchestra.run.smithi183.stdout: "rgw_nfs_max_gc": "300", 2024-09-10T19:11:58.399 INFO:teuthology.orchestra.run.smithi183.stdout: "rgw_nfs_namespace_expire_secs": "300", 2024-09-10T19:11:58.399 INFO:teuthology.orchestra.run.smithi183.stdout: "rgw_nfs_run_gc_threads": "false", 2024-09-10T19:11:58.399 INFO:teuthology.orchestra.run.smithi183.stdout: "rgw_nfs_run_lc_threads": "false", 2024-09-10T19:11:58.399 INFO:teuthology.orchestra.run.smithi183.stdout: "rgw_nfs_run_quota_threads": "false", 2024-09-10T19:11:58.399 INFO:teuthology.orchestra.run.smithi183.stdout: "rgw_nfs_run_sync_thread": "false", 2024-09-10T19:11:58.399 INFO:teuthology.orchestra.run.smithi183.stdout: "rgw_nfs_s3_fast_attrs": "false", 2024-09-10T19:11:58.400 INFO:teuthology.orchestra.run.smithi183.stdout: "rgw_nfs_write_completion_interval_s": "10", 2024-09-10T19:11:58.400 INFO:teuthology.orchestra.run.smithi183.stdout: "rgw_num_async_rados_threads": "32", 2024-09-10T19:11:58.400 INFO:teuthology.orchestra.run.smithi183.stdout: "rgw_num_control_oids": "8", 2024-09-10T19:11:58.400 INFO:teuthology.orchestra.run.smithi183.stdout: "rgw_numa_node": "-1", 2024-09-10T19:11:58.400 INFO:teuthology.orchestra.run.smithi183.stdout: "rgw_obj_stripe_size": "4194304", 2024-09-10T19:11:58.400 INFO:teuthology.orchestra.run.smithi183.stdout: "rgw_obj_tombstone_cache_size": "1000", 2024-09-10T19:11:58.400 INFO:teuthology.orchestra.run.smithi183.stdout: "rgw_objexp_chunk_size": "100", 2024-09-10T19:11:58.400 INFO:teuthology.orchestra.run.smithi183.stdout: "rgw_objexp_gc_interval": "600", 2024-09-10T19:11:58.400 INFO:teuthology.orchestra.run.smithi183.stdout: "rgw_objexp_hints_num_shards": "127", 2024-09-10T19:11:58.400 INFO:teuthology.orchestra.run.smithi183.stdout: "rgw_olh_pending_timeout_sec": "3600", 2024-09-10T19:11:58.400 INFO:teuthology.orchestra.run.smithi183.stdout: "rgw_op_thread_suicide_timeout": "0", 2024-09-10T19:11:58.400 INFO:teuthology.orchestra.run.smithi183.stdout: "rgw_op_thread_timeout": "600", 2024-09-10T19:11:58.400 INFO:teuthology.orchestra.run.smithi183.stdout: "rgw_op_tracing": "false", 2024-09-10T19:11:58.401 INFO:teuthology.orchestra.run.smithi183.stdout: "rgw_opa_token": "", 2024-09-10T19:11:58.401 INFO:teuthology.orchestra.run.smithi183.stdout: "rgw_opa_url": "", 2024-09-10T19:11:58.401 INFO:teuthology.orchestra.run.smithi183.stdout: "rgw_opa_verify_ssl": "true", 2024-09-10T19:11:58.401 INFO:teuthology.orchestra.run.smithi183.stdout: "rgw_ops_log_data_backlog": "5242880", 2024-09-10T19:11:58.401 INFO:teuthology.orchestra.run.smithi183.stdout: "rgw_ops_log_file_path": "", 2024-09-10T19:11:58.401 INFO:teuthology.orchestra.run.smithi183.stdout: "rgw_ops_log_rados": "true", 2024-09-10T19:11:58.401 INFO:teuthology.orchestra.run.smithi183.stdout: "rgw_ops_log_socket_path": "", 2024-09-10T19:11:58.401 INFO:teuthology.orchestra.run.smithi183.stdout: "rgw_override_bucket_index_max_shards": "0", 2024-09-10T19:11:58.401 INFO:teuthology.orchestra.run.smithi183.stdout: "rgw_period_latest_epoch_info_oid": ".latest_epoch", 2024-09-10T19:11:58.401 INFO:teuthology.orchestra.run.smithi183.stdout: "rgw_period_push_interval": "2.000000", 2024-09-10T19:11:58.401 INFO:teuthology.orchestra.run.smithi183.stdout: "rgw_period_push_interval_max": "30.000000", 2024-09-10T19:11:58.401 INFO:teuthology.orchestra.run.smithi183.stdout: "rgw_period_root_pool": ".rgw.root", 2024-09-10T19:11:58.401 INFO:teuthology.orchestra.run.smithi183.stdout: "rgw_print_continue": "true", 2024-09-10T19:11:58.401 INFO:teuthology.orchestra.run.smithi183.stdout: "rgw_print_prohibited_content_length": "false", 2024-09-10T19:11:58.402 INFO:teuthology.orchestra.run.smithi183.stdout: "rgw_put_obj_max_window_size": "67108864", 2024-09-10T19:11:58.402 INFO:teuthology.orchestra.run.smithi183.stdout: "rgw_put_obj_min_window_size": "16777216", 2024-09-10T19:11:58.402 INFO:teuthology.orchestra.run.smithi183.stdout: "rgw_rados_pool_autoscale_bias": "4.000000", 2024-09-10T19:11:58.402 INFO:teuthology.orchestra.run.smithi183.stdout: "rgw_rados_pool_recovery_priority": "5", 2024-09-10T19:11:58.402 INFO:teuthology.orchestra.run.smithi183.stdout: "rgw_rados_tracing": "false", 2024-09-10T19:11:58.402 INFO:teuthology.orchestra.run.smithi183.stdout: "rgw_realm": "", 2024-09-10T19:11:58.402 INFO:teuthology.orchestra.run.smithi183.stdout: "rgw_realm_id": "", 2024-09-10T19:11:58.402 INFO:teuthology.orchestra.run.smithi183.stdout: "rgw_realm_root_pool": ".rgw.root", 2024-09-10T19:11:58.402 INFO:teuthology.orchestra.run.smithi183.stdout: "rgw_region": "", 2024-09-10T19:11:58.402 INFO:teuthology.orchestra.run.smithi183.stdout: "rgw_region_root_pool": ".rgw.root", 2024-09-10T19:11:58.402 INFO:teuthology.orchestra.run.smithi183.stdout: "rgw_relaxed_region_enforcement": "false", 2024-09-10T19:11:58.402 INFO:teuthology.orchestra.run.smithi183.stdout: "rgw_relaxed_s3_bucket_names": "false", 2024-09-10T19:11:58.402 INFO:teuthology.orchestra.run.smithi183.stdout: "rgw_remote_addr_param": "REMOTE_ADDR", 2024-09-10T19:11:58.402 INFO:teuthology.orchestra.run.smithi183.stdout: "rgw_request_uri": "", 2024-09-10T19:11:58.403 INFO:teuthology.orchestra.run.smithi183.stdout: "rgw_reshard_batch_size": "64", 2024-09-10T19:11:58.403 INFO:teuthology.orchestra.run.smithi183.stdout: "rgw_reshard_bucket_lock_duration": "360", 2024-09-10T19:11:58.403 INFO:teuthology.orchestra.run.smithi183.stdout: "rgw_reshard_max_aio": "128", 2024-09-10T19:11:58.403 INFO:teuthology.orchestra.run.smithi183.stdout: "rgw_reshard_num_logs": "16", 2024-09-10T19:11:58.403 INFO:teuthology.orchestra.run.smithi183.stdout: "rgw_reshard_thread_interval": "600", 2024-09-10T19:11:58.403 INFO:teuthology.orchestra.run.smithi183.stdout: "rgw_resolve_cname": "false", 2024-09-10T19:11:58.403 INFO:teuthology.orchestra.run.smithi183.stdout: "rgw_rest_getusage_op_compat": "false", 2024-09-10T19:11:58.403 INFO:teuthology.orchestra.run.smithi183.stdout: "rgw_run_sync_thread": "true", 2024-09-10T19:11:58.403 INFO:teuthology.orchestra.run.smithi183.stdout: "rgw_s3_auth_disable_signature_url": "false", 2024-09-10T19:11:58.403 INFO:teuthology.orchestra.run.smithi183.stdout: "rgw_s3_auth_order": "sts, external, local", 2024-09-10T19:11:58.403 INFO:teuthology.orchestra.run.smithi183.stdout: "rgw_s3_auth_use_keystone": "false", 2024-09-10T19:11:58.403 INFO:teuthology.orchestra.run.smithi183.stdout: "rgw_s3_auth_use_ldap": "false", 2024-09-10T19:11:58.403 INFO:teuthology.orchestra.run.smithi183.stdout: "rgw_s3_auth_use_rados": "true", 2024-09-10T19:11:58.404 INFO:teuthology.orchestra.run.smithi183.stdout: "rgw_s3_auth_use_sts": "false", 2024-09-10T19:11:58.404 INFO:teuthology.orchestra.run.smithi183.stdout: "rgw_s3_client_max_sig_ver": "-1", 2024-09-10T19:11:58.404 INFO:teuthology.orchestra.run.smithi183.stdout: "rgw_s3_success_create_obj_status": "0", 2024-09-10T19:11:58.404 INFO:teuthology.orchestra.run.smithi183.stdout: "rgw_safe_max_objects_per_shard": "102400", 2024-09-10T19:11:58.404 INFO:teuthology.orchestra.run.smithi183.stdout: "rgw_scheduler_type": "throttler", 2024-09-10T19:11:58.404 INFO:teuthology.orchestra.run.smithi183.stdout: "rgw_script_uri": "", 2024-09-10T19:11:58.404 INFO:teuthology.orchestra.run.smithi183.stdout: "rgw_service_provider_name": "", 2024-09-10T19:11:58.404 INFO:teuthology.orchestra.run.smithi183.stdout: "rgw_shard_warning_threshold": "90.000000", 2024-09-10T19:11:58.404 INFO:teuthology.orchestra.run.smithi183.stdout: "rgw_sts_client_id": "", 2024-09-10T19:11:58.404 INFO:teuthology.orchestra.run.smithi183.stdout: "rgw_sts_client_secret": "", 2024-09-10T19:11:58.404 INFO:teuthology.orchestra.run.smithi183.stdout: "rgw_sts_entry": "sts", 2024-09-10T19:11:58.404 INFO:teuthology.orchestra.run.smithi183.stdout: "rgw_sts_key": "sts", 2024-09-10T19:11:58.404 INFO:teuthology.orchestra.run.smithi183.stdout: "rgw_sts_max_session_duration": "43200", 2024-09-10T19:11:58.405 INFO:teuthology.orchestra.run.smithi183.stdout: "rgw_sts_min_session_duration": "900", 2024-09-10T19:11:58.405 INFO:teuthology.orchestra.run.smithi183.stdout: "rgw_sts_token_introspection_url": "", 2024-09-10T19:11:58.405 INFO:teuthology.orchestra.run.smithi183.stdout: "rgw_swift_account_in_url": "false", 2024-09-10T19:11:58.405 INFO:teuthology.orchestra.run.smithi183.stdout: "rgw_swift_auth_entry": "auth", 2024-09-10T19:11:58.405 INFO:teuthology.orchestra.run.smithi183.stdout: "rgw_swift_auth_url": "", 2024-09-10T19:11:58.405 INFO:teuthology.orchestra.run.smithi183.stdout: "rgw_swift_custom_header": "", 2024-09-10T19:11:58.405 INFO:teuthology.orchestra.run.smithi183.stdout: "rgw_swift_enforce_content_length": "false", 2024-09-10T19:11:58.405 INFO:teuthology.orchestra.run.smithi183.stdout: "rgw_swift_need_stats": "true", 2024-09-10T19:11:58.405 INFO:teuthology.orchestra.run.smithi183.stdout: "rgw_swift_tenant_name": "", 2024-09-10T19:11:58.405 INFO:teuthology.orchestra.run.smithi183.stdout: "rgw_swift_token_expiration": "86400", 2024-09-10T19:11:58.405 INFO:teuthology.orchestra.run.smithi183.stdout: "rgw_swift_url": "", 2024-09-10T19:11:58.405 INFO:teuthology.orchestra.run.smithi183.stdout: "rgw_swift_url_prefix": "swift", 2024-09-10T19:11:58.405 INFO:teuthology.orchestra.run.smithi183.stdout: "rgw_swift_versioning_enabled": "false", 2024-09-10T19:11:58.405 INFO:teuthology.orchestra.run.smithi183.stdout: "rgw_sync_data_inject_err_probability": "0.000000", 2024-09-10T19:11:58.406 INFO:teuthology.orchestra.run.smithi183.stdout: "rgw_sync_lease_period": "120", 2024-09-10T19:11:58.406 INFO:teuthology.orchestra.run.smithi183.stdout: "rgw_sync_log_trim_concurrent_buckets": "4", 2024-09-10T19:11:58.406 INFO:teuthology.orchestra.run.smithi183.stdout: "rgw_sync_log_trim_interval": "1200", 2024-09-10T19:11:58.406 INFO:teuthology.orchestra.run.smithi183.stdout: "rgw_sync_log_trim_max_buckets": "16", 2024-09-10T19:11:58.406 INFO:teuthology.orchestra.run.smithi183.stdout: "rgw_sync_log_trim_min_cold_buckets": "4", 2024-09-10T19:11:58.406 INFO:teuthology.orchestra.run.smithi183.stdout: "rgw_sync_meta_inject_err_probability": "0.000000", 2024-09-10T19:11:58.406 INFO:teuthology.orchestra.run.smithi183.stdout: "rgw_sync_obj_etag_verify": "false", 2024-09-10T19:11:58.406 INFO:teuthology.orchestra.run.smithi183.stdout: "rgw_sync_trace_history_size": "4096", 2024-09-10T19:11:58.406 INFO:teuthology.orchestra.run.smithi183.stdout: "rgw_sync_trace_per_node_log_size": "32", 2024-09-10T19:11:58.406 INFO:teuthology.orchestra.run.smithi183.stdout: "rgw_sync_trace_servicemap_update_interval": "10", 2024-09-10T19:11:58.406 INFO:teuthology.orchestra.run.smithi183.stdout: "rgw_thread_pool_size": "512", 2024-09-10T19:11:58.406 INFO:teuthology.orchestra.run.smithi183.stdout: "rgw_torrent_comment": "", 2024-09-10T19:11:58.406 INFO:teuthology.orchestra.run.smithi183.stdout: "rgw_torrent_createby": "", 2024-09-10T19:11:58.406 INFO:teuthology.orchestra.run.smithi183.stdout: "rgw_torrent_encoding": "", 2024-09-10T19:11:58.407 INFO:teuthology.orchestra.run.smithi183.stdout: "rgw_torrent_flag": "false", 2024-09-10T19:11:58.407 INFO:teuthology.orchestra.run.smithi183.stdout: "rgw_torrent_origin": "", 2024-09-10T19:11:58.407 INFO:teuthology.orchestra.run.smithi183.stdout: "rgw_torrent_sha_unit": "524288", 2024-09-10T19:11:58.407 INFO:teuthology.orchestra.run.smithi183.stdout: "rgw_torrent_tracker": "", 2024-09-10T19:11:58.407 INFO:teuthology.orchestra.run.smithi183.stdout: "rgw_trust_forwarded_https": "false", 2024-09-10T19:11:58.407 INFO:teuthology.orchestra.run.smithi183.stdout: "rgw_usage_log_flush_threshold": "1024", 2024-09-10T19:11:58.407 INFO:teuthology.orchestra.run.smithi183.stdout: "rgw_usage_log_tick_interval": "30", 2024-09-10T19:11:58.407 INFO:teuthology.orchestra.run.smithi183.stdout: "rgw_usage_max_shards": "32", 2024-09-10T19:11:58.407 INFO:teuthology.orchestra.run.smithi183.stdout: "rgw_usage_max_user_shards": "1", 2024-09-10T19:11:58.407 INFO:teuthology.orchestra.run.smithi183.stdout: "rgw_use_opa_authz": "false", 2024-09-10T19:11:58.407 INFO:teuthology.orchestra.run.smithi183.stdout: "rgw_user_default_quota_max_objects": "-1", 2024-09-10T19:11:58.407 INFO:teuthology.orchestra.run.smithi183.stdout: "rgw_user_default_quota_max_size": "-1", 2024-09-10T19:11:58.407 INFO:teuthology.orchestra.run.smithi183.stdout: "rgw_user_max_buckets": "1000", 2024-09-10T19:11:58.407 INFO:teuthology.orchestra.run.smithi183.stdout: "rgw_user_quota_bucket_sync_interval": "180", 2024-09-10T19:11:58.407 INFO:teuthology.orchestra.run.smithi183.stdout: "rgw_user_quota_sync_idle_users": "false", 2024-09-10T19:11:58.408 INFO:teuthology.orchestra.run.smithi183.stdout: "rgw_user_quota_sync_interval": "86400", 2024-09-10T19:11:58.408 INFO:teuthology.orchestra.run.smithi183.stdout: "rgw_user_quota_sync_wait_time": "86400", 2024-09-10T19:11:58.408 INFO:teuthology.orchestra.run.smithi183.stdout: "rgw_user_unique_email": "true", 2024-09-10T19:11:58.408 INFO:teuthology.orchestra.run.smithi183.stdout: "rgw_verify_ssl": "true", 2024-09-10T19:11:58.408 INFO:teuthology.orchestra.run.smithi183.stdout: "rgw_website_routing_rules_max_num": "50", 2024-09-10T19:11:58.408 INFO:teuthology.orchestra.run.smithi183.stdout: "rgw_zone": "", 2024-09-10T19:11:58.408 INFO:teuthology.orchestra.run.smithi183.stdout: "rgw_zone_id": "", 2024-09-10T19:11:58.408 INFO:teuthology.orchestra.run.smithi183.stdout: "rgw_zone_root_pool": ".rgw.root", 2024-09-10T19:11:58.408 INFO:teuthology.orchestra.run.smithi183.stdout: "rgw_zonegroup": "", 2024-09-10T19:11:58.408 INFO:teuthology.orchestra.run.smithi183.stdout: "rgw_zonegroup_id": "", 2024-09-10T19:11:58.408 INFO:teuthology.orchestra.run.smithi183.stdout: "rgw_zonegroup_root_pool": ".rgw.root", 2024-09-10T19:11:58.408 INFO:teuthology.orchestra.run.smithi183.stdout: "rocksdb_block_size": "4096", 2024-09-10T19:11:58.408 INFO:teuthology.orchestra.run.smithi183.stdout: "rocksdb_bloom_bits_per_key": "20", 2024-09-10T19:11:58.408 INFO:teuthology.orchestra.run.smithi183.stdout: "rocksdb_cache_index_and_filter_blocks": "true", 2024-09-10T19:11:58.409 INFO:teuthology.orchestra.run.smithi183.stdout: "rocksdb_cache_index_and_filter_blocks_with_high_priority": "false", 2024-09-10T19:11:58.409 INFO:teuthology.orchestra.run.smithi183.stdout: "rocksdb_cache_row_ratio": "0.000000", 2024-09-10T19:11:58.409 INFO:teuthology.orchestra.run.smithi183.stdout: "rocksdb_cache_shard_bits": "4", 2024-09-10T19:11:58.409 INFO:teuthology.orchestra.run.smithi183.stdout: "rocksdb_cache_size": "536870912", 2024-09-10T19:11:58.409 INFO:teuthology.orchestra.run.smithi183.stdout: "rocksdb_cache_type": "binned_lru", 2024-09-10T19:11:58.409 INFO:teuthology.orchestra.run.smithi183.stdout: "rocksdb_cf_compact_on_deletion": "true", 2024-09-10T19:11:58.409 INFO:teuthology.orchestra.run.smithi183.stdout: "rocksdb_cf_compact_on_deletion_sliding_window": "32768", 2024-09-10T19:11:58.409 INFO:teuthology.orchestra.run.smithi183.stdout: "rocksdb_cf_compact_on_deletion_trigger": "16384", 2024-09-10T19:11:58.409 INFO:teuthology.orchestra.run.smithi183.stdout: "rocksdb_collect_compaction_stats": "false", 2024-09-10T19:11:58.409 INFO:teuthology.orchestra.run.smithi183.stdout: "rocksdb_collect_extended_stats": "false", 2024-09-10T19:11:58.409 INFO:teuthology.orchestra.run.smithi183.stdout: "rocksdb_collect_memory_stats": "false", 2024-09-10T19:11:58.409 INFO:teuthology.orchestra.run.smithi183.stdout: "rocksdb_delete_range_threshold": "1048576", 2024-09-10T19:11:58.409 INFO:teuthology.orchestra.run.smithi183.stdout: "rocksdb_index_type": "binary_search", 2024-09-10T19:11:58.409 INFO:teuthology.orchestra.run.smithi183.stdout: "rocksdb_log_to_ceph_log": "true", 2024-09-10T19:11:58.410 INFO:teuthology.orchestra.run.smithi183.stdout: "rocksdb_metadata_block_size": "4096", 2024-09-10T19:11:58.410 INFO:teuthology.orchestra.run.smithi183.stdout: "rocksdb_partition_filters": "false", 2024-09-10T19:11:58.410 INFO:teuthology.orchestra.run.smithi183.stdout: "rocksdb_perf": "false", 2024-09-10T19:11:58.410 INFO:teuthology.orchestra.run.smithi183.stdout: "rocksdb_pin_l0_filter_and_index_blocks_in_cache": "false", 2024-09-10T19:11:58.410 INFO:teuthology.orchestra.run.smithi183.stdout: "rotating_keys_bootstrap_timeout": "30", 2024-09-10T19:11:58.410 INFO:teuthology.orchestra.run.smithi183.stdout: "rotating_keys_renewal_timeout": "10", 2024-09-10T19:11:58.410 INFO:teuthology.orchestra.run.smithi183.stdout: "run_dir": "/var/run/ceph", 2024-09-10T19:11:58.410 INFO:teuthology.orchestra.run.smithi183.stdout: "seastore_block_create": "true", 2024-09-10T19:11:58.410 INFO:teuthology.orchestra.run.smithi183.stdout: "seastore_cache_lru_size": "67108864", 2024-09-10T19:11:58.410 INFO:teuthology.orchestra.run.smithi183.stdout: "seastore_default_max_object_size": "16777216", 2024-09-10T19:11:58.410 INFO:teuthology.orchestra.run.smithi183.stdout: "seastore_default_object_metadata_reservation": "16777216", 2024-09-10T19:11:58.410 INFO:teuthology.orchestra.run.smithi183.stdout: "seastore_device_size": "107374182400", 2024-09-10T19:11:58.410 INFO:teuthology.orchestra.run.smithi183.stdout: "seastore_init_rewrite_segments_num_per_device": "6", 2024-09-10T19:11:58.410 INFO:teuthology.orchestra.run.smithi183.stdout: "seastore_journal_batch_capacity": "16", 2024-09-10T19:11:58.411 INFO:teuthology.orchestra.run.smithi183.stdout: "seastore_journal_batch_flush_size": "16777216", 2024-09-10T19:11:58.411 INFO:teuthology.orchestra.run.smithi183.stdout: "seastore_journal_batch_preferred_fullness": "0.950000", 2024-09-10T19:11:58.411 INFO:teuthology.orchestra.run.smithi183.stdout: "seastore_journal_iodepth_limit": "5", 2024-09-10T19:11:58.411 INFO:teuthology.orchestra.run.smithi183.stdout: "seastore_segment_size": "67108864", 2024-09-10T19:11:58.411 INFO:teuthology.orchestra.run.smithi183.stdout: "setgroup": "ceph", 2024-09-10T19:11:58.411 INFO:teuthology.orchestra.run.smithi183.stdout: "setuser": "ceph", 2024-09-10T19:11:58.411 INFO:teuthology.orchestra.run.smithi183.stdout: "setuser_match_path": "", 2024-09-10T19:11:58.411 INFO:teuthology.orchestra.run.smithi183.stdout: "target_max_misplaced_ratio": "0.050000", 2024-09-10T19:11:58.411 INFO:teuthology.orchestra.run.smithi183.stdout: "thp": "false", 2024-09-10T19:11:58.411 INFO:teuthology.orchestra.run.smithi183.stdout: "threadpool_default_timeout": "60", 2024-09-10T19:11:58.411 INFO:teuthology.orchestra.run.smithi183.stdout: "threadpool_empty_queue_max_wait": "2", 2024-09-10T19:11:58.411 INFO:teuthology.orchestra.run.smithi183.stdout: "throttler_perf_counter": "true" 2024-09-10T19:11:58.411 INFO:teuthology.orchestra.run.smithi183.stdout:} 2024-09-10T19:11:58.547 INFO:journalctl@ceph.prometheus.a.smithi191.stdout:Sep 10 19:11:58 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-prometheus-a[63737]: ts=2024-09-10T19:11:58.264Z caller=main.go:840 level=warn msg="Received SIGTERM, exiting gracefully..." 2024-09-10T19:11:58.548 INFO:journalctl@ceph.prometheus.a.smithi191.stdout:Sep 10 19:11:58 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-prometheus-a[63737]: ts=2024-09-10T19:11:58.264Z caller=main.go:864 level=info msg="Stopping scrape discovery manager..." 2024-09-10T19:11:58.548 INFO:journalctl@ceph.prometheus.a.smithi191.stdout:Sep 10 19:11:58 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-prometheus-a[63737]: ts=2024-09-10T19:11:58.264Z caller=main.go:878 level=info msg="Stopping notify discovery manager..." 2024-09-10T19:11:58.548 INFO:journalctl@ceph.prometheus.a.smithi191.stdout:Sep 10 19:11:58 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-prometheus-a[63737]: ts=2024-09-10T19:11:58.264Z caller=manager.go:988 level=info component="rule manager" msg="Stopping rule manager..." 2024-09-10T19:11:58.548 INFO:journalctl@ceph.prometheus.a.smithi191.stdout:Sep 10 19:11:58 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-prometheus-a[63737]: ts=2024-09-10T19:11:58.264Z caller=main.go:860 level=info msg="Scrape discovery manager stopped" 2024-09-10T19:11:58.548 INFO:journalctl@ceph.prometheus.a.smithi191.stdout:Sep 10 19:11:58 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-prometheus-a[63737]: ts=2024-09-10T19:11:58.265Z caller=main.go:874 level=info msg="Notify discovery manager stopped" 2024-09-10T19:11:58.548 INFO:journalctl@ceph.prometheus.a.smithi191.stdout:Sep 10 19:11:58 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-prometheus-a[63737]: ts=2024-09-10T19:11:58.265Z caller=manager.go:998 level=info component="rule manager" msg="Rule manager stopped" 2024-09-10T19:11:58.548 INFO:journalctl@ceph.prometheus.a.smithi191.stdout:Sep 10 19:11:58 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-prometheus-a[63737]: ts=2024-09-10T19:11:58.265Z caller=main.go:915 level=info msg="Stopping scrape manager..." 2024-09-10T19:11:58.548 INFO:journalctl@ceph.prometheus.a.smithi191.stdout:Sep 10 19:11:58 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-prometheus-a[63737]: ts=2024-09-10T19:11:58.265Z caller=main.go:907 level=info msg="Scrape manager stopped" 2024-09-10T19:11:58.548 INFO:journalctl@ceph.prometheus.a.smithi191.stdout:Sep 10 19:11:58 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-prometheus-a[63737]: ts=2024-09-10T19:11:58.265Z caller=notifier.go:606 level=info component=notifier msg="Stopping notification manager..." 2024-09-10T19:11:58.548 INFO:journalctl@ceph.prometheus.a.smithi191.stdout:Sep 10 19:11:58 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-prometheus-a[63737]: ts=2024-09-10T19:11:58.265Z caller=main.go:1135 level=info msg="Notifier manager stopped" 2024-09-10T19:11:58.548 INFO:journalctl@ceph.prometheus.a.smithi191.stdout:Sep 10 19:11:58 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-prometheus-a[63737]: ts=2024-09-10T19:11:58.265Z caller=main.go:1147 level=info msg="See you next time!" 2024-09-10T19:11:58.549 INFO:journalctl@ceph.prometheus.a.smithi191.stdout:Sep 10 19:11:58 smithi191 podman[67045]: 2024-09-10 19:11:58.275413753 +0000 UTC m=+0.134977648 container died 1c55fcb84c302ee4599106b913bfc205582a7e266a8f3420268d4d3bcbb6bda8 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-10T19:11:58.893 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:11:58 smithi191 ceph-mon[32884]: pgmap v51: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-10T19:11:58.893 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:11:58 smithi191 ceph-mon[32884]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:11:58.893 INFO:journalctl@ceph.prometheus.a.smithi191.stdout:Sep 10 19:11:58 smithi191 podman[67045]: 2024-09-10 19:11:58.645726178 +0000 UTC m=+0.505290113 container cleanup 1c55fcb84c302ee4599106b913bfc205582a7e266a8f3420268d4d3bcbb6bda8 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-10T19:11:58.894 INFO:journalctl@ceph.prometheus.a.smithi191.stdout:Sep 10 19:11:58 smithi191 bash[67045]: ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-prometheus-a 2024-09-10T19:11:59.185 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:11:58 smithi183 ceph-mon[29736]: pgmap v51: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-10T19:11:59.185 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:11:58 smithi183 ceph-mon[29736]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:11:59.185 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:11:58 smithi183 ceph-mon[41562]: pgmap v51: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-10T19:11:59.186 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:11:58 smithi183 ceph-mon[41562]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:11:59.215 INFO:journalctl@ceph.iscsi.iscsi.a.smithi191.stdout:Sep 10 19:11:58 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-iscsi-iscsi-a[61816]: debug there is no tcmu-runner data available 2024-09-10T19:11:59.215 INFO:journalctl@ceph.prometheus.a.smithi191.stdout:Sep 10 19:11:58 smithi191 podman[67061]: 2024-09-10 19:11:58.896931142 +0000 UTC m=+0.623747998 container remove 1c55fcb84c302ee4599106b913bfc205582a7e266a8f3420268d4d3bcbb6bda8 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-10T19:11:59.216 INFO:journalctl@ceph.prometheus.a.smithi191.stdout:Sep 10 19:11:59 smithi191 systemd[1]: ceph-320f6df8-6fa7-11ef-bcea-c7b262605968@prometheus.a.service: Deactivated successfully. 2024-09-10T19:11:59.216 INFO:journalctl@ceph.prometheus.a.smithi191.stdout:Sep 10 19:11:59 smithi191 systemd[1]: Stopped Ceph prometheus.a for 320f6df8-6fa7-11ef-bcea-c7b262605968. 2024-09-10T19:11:59.216 INFO:journalctl@ceph.prometheus.a.smithi191.stdout:Sep 10 19:11:59 smithi191 systemd[1]: ceph-320f6df8-6fa7-11ef-bcea-c7b262605968@prometheus.a.service: Consumed 1.106s CPU time. 2024-09-10T19:11:59.216 INFO:journalctl@ceph.prometheus.a.smithi191.stdout:Sep 10 19:11:59 smithi191 systemd[1]: Starting Ceph prometheus.a for 320f6df8-6fa7-11ef-bcea-c7b262605968... 2024-09-10T19:11:59.507 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph -- tell 'mon.*' injectargs --mon_osd_down_out_interval=0 2024-09-10T19:11:59.559 INFO:journalctl@ceph.prometheus.a.smithi191.stdout:Sep 10 19:11:59 smithi191 podman[67159]: 2024-09-10 19:11:59.174566404 +0000 UTC m=+0.016961777 image pull a07b618ecd1dce142bce4c52f0e80982eaf1f14265a2415c2d35978ccaa0a464 quay.io/prometheus/prometheus:v2.43.0 2024-09-10T19:11:59.560 INFO:journalctl@ceph.prometheus.a.smithi191.stdout:Sep 10 19:11:59 smithi191 podman[67159]: 2024-09-10 19:11:59.298027179 +0000 UTC m=+0.140422545 container create e9f7fcd20861732a5b312cee595b5fc835c9ca9d5e4eb4b6b509833063076edf (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-10T19:11:59.760 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:11:59.965 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:11:59 smithi191 ceph-mon[32884]: from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:11:59.966 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:11:59 smithi191 ceph-mon[32884]: from='mgr.14556 ' entity='mgr.y' 2024-09-10T19:11:59.966 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:11:59 smithi191 ceph-mon[32884]: from='mgr.14556 ' entity='mgr.y' 2024-09-10T19:11:59.966 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:11:59 smithi191 ceph-mon[32884]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-09-10T19:11:59.966 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:11:59 smithi191 ceph-mon[32884]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi183.front.sepia.ceph.com:9093"}]: dispatch 2024-09-10T19:11:59.966 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:11:59 smithi191 ceph-mon[32884]: from='mgr.14556 ' entity='mgr.y' 2024-09-10T19:11:59.966 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:11:59 smithi191 ceph-mon[32884]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-09-10T19:11:59.966 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:11:59 smithi191 ceph-mon[32884]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi191.front.sepia.ceph.com:9095"}]: dispatch 2024-09-10T19:11:59.966 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:11:59 smithi191 ceph-mon[32884]: from='mgr.14556 ' entity='mgr.y' 2024-09-10T19:11:59.966 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:11:59 smithi191 ceph-mon[32884]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-09-10T19:11:59.966 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:11:59 smithi191 ceph-mon[32884]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi191.front.sepia.ceph.com:3000"}]: dispatch 2024-09-10T19:11:59.966 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:11:59 smithi191 ceph-mon[32884]: from='mgr.14556 ' entity='mgr.y' 2024-09-10T19:11:59.967 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:11:59 smithi191 ceph-mon[32884]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T19:11:59.967 INFO:journalctl@ceph.prometheus.a.smithi191.stdout:Sep 10 19:11:59 smithi191 podman[67159]: 2024-09-10 19:11:59.585227594 +0000 UTC m=+0.427622966 container init e9f7fcd20861732a5b312cee595b5fc835c9ca9d5e4eb4b6b509833063076edf (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-10T19:11:59.967 INFO:journalctl@ceph.prometheus.a.smithi191.stdout:Sep 10 19:11:59 smithi191 podman[67159]: 2024-09-10 19:11:59.590079803 +0000 UTC m=+0.432475166 container start e9f7fcd20861732a5b312cee595b5fc835c9ca9d5e4eb4b6b509833063076edf (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-10T19:11:59.967 INFO:journalctl@ceph.prometheus.a.smithi191.stdout:Sep 10 19:11:59 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-prometheus-a[67174]: ts=2024-09-10T19:11:59.614Z caller=main.go:564 level=info msg="Starting Prometheus Server" mode=server version="(version=2.43.0, branch=HEAD, revision=edfc3bcd025dd6fe296c167a14a216cab1e552ee)" 2024-09-10T19:11:59.967 INFO:journalctl@ceph.prometheus.a.smithi191.stdout:Sep 10 19:11:59 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-prometheus-a[67174]: ts=2024-09-10T19:11:59.614Z caller=main.go:569 level=info build_context="(go=go1.19.7, platform=linux/amd64, user=root@8a0ee342e522, date=20230321-12:56:07, tags=netgo,builtinassets)" 2024-09-10T19:11:59.967 INFO:journalctl@ceph.prometheus.a.smithi191.stdout:Sep 10 19:11:59 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-prometheus-a[67174]: ts=2024-09-10T19:11:59.614Z caller=main.go:570 level=info host_details="(Linux 5.14.0-505.el9.x86_64 #1 SMP PREEMPT_DYNAMIC Thu Sep 5 07:54:07 UTC 2024 x86_64 smithi191 (none))" 2024-09-10T19:11:59.967 INFO:journalctl@ceph.prometheus.a.smithi191.stdout:Sep 10 19:11:59 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-prometheus-a[67174]: ts=2024-09-10T19:11:59.614Z caller=main.go:571 level=info fd_limits="(soft=1048576, hard=1048576)" 2024-09-10T19:11:59.967 INFO:journalctl@ceph.prometheus.a.smithi191.stdout:Sep 10 19:11:59 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-prometheus-a[67174]: ts=2024-09-10T19:11:59.614Z caller=main.go:572 level=info vm_limits="(soft=unlimited, hard=unlimited)" 2024-09-10T19:11:59.968 INFO:journalctl@ceph.prometheus.a.smithi191.stdout:Sep 10 19:11:59 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-prometheus-a[67174]: ts=2024-09-10T19:11:59.615Z caller=web.go:561 level=info component=web msg="Start listening for connections" address=:9095 2024-09-10T19:11:59.968 INFO:journalctl@ceph.prometheus.a.smithi191.stdout:Sep 10 19:11:59 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-prometheus-a[67174]: ts=2024-09-10T19:11:59.615Z caller=main.go:1005 level=info msg="Starting TSDB ..." 2024-09-10T19:11:59.968 INFO:journalctl@ceph.prometheus.a.smithi191.stdout:Sep 10 19:11:59 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-prometheus-a[67174]: ts=2024-09-10T19:11:59.618Z caller=tls_config.go:232 level=info component=web msg="Listening on" address=[::]:9095 2024-09-10T19:11:59.968 INFO:journalctl@ceph.prometheus.a.smithi191.stdout:Sep 10 19:11:59 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-prometheus-a[67174]: ts=2024-09-10T19:11:59.618Z caller=tls_config.go:235 level=info component=web msg="TLS is disabled." http2=false address=[::]:9095 2024-09-10T19:11:59.968 INFO:journalctl@ceph.prometheus.a.smithi191.stdout:Sep 10 19:11:59 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-prometheus-a[67174]: ts=2024-09-10T19:11:59.619Z caller=head.go:587 level=info component=tsdb msg="Replaying on-disk memory mappable chunks if any" 2024-09-10T19:11:59.968 INFO:journalctl@ceph.prometheus.a.smithi191.stdout:Sep 10 19:11:59 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-prometheus-a[67174]: ts=2024-09-10T19:11:59.619Z caller=head.go:658 level=info component=tsdb msg="On-disk memory mappable chunks replay completed" duration=1.637µs 2024-09-10T19:11:59.968 INFO:journalctl@ceph.prometheus.a.smithi191.stdout:Sep 10 19:11:59 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-prometheus-a[67174]: ts=2024-09-10T19:11:59.619Z caller=head.go:664 level=info component=tsdb msg="Replaying WAL, this may take a while" 2024-09-10T19:11:59.968 INFO:journalctl@ceph.prometheus.a.smithi191.stdout:Sep 10 19:11:59 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-prometheus-a[67174]: ts=2024-09-10T19:11:59.619Z caller=head.go:735 level=info component=tsdb msg="WAL segment loaded" segment=0 maxSegment=1 2024-09-10T19:11:59.968 INFO:journalctl@ceph.prometheus.a.smithi191.stdout:Sep 10 19:11:59 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-prometheus-a[67174]: ts=2024-09-10T19:11:59.619Z caller=head.go:735 level=info component=tsdb msg="WAL segment loaded" segment=1 maxSegment=1 2024-09-10T19:11:59.968 INFO:journalctl@ceph.prometheus.a.smithi191.stdout:Sep 10 19:11:59 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-prometheus-a[67174]: ts=2024-09-10T19:11:59.619Z caller=head.go:772 level=info component=tsdb msg="WAL replay completed" checkpoint_replay_duration=18.631µs wal_replay_duration=634.129µs wbl_replay_duration=88ns total_replay_duration=670.373µs 2024-09-10T19:11:59.968 INFO:journalctl@ceph.prometheus.a.smithi191.stdout:Sep 10 19:11:59 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-prometheus-a[67174]: ts=2024-09-10T19:11:59.620Z caller=main.go:1026 level=info fs_type=XFS_SUPER_MAGIC 2024-09-10T19:11:59.968 INFO:journalctl@ceph.prometheus.a.smithi191.stdout:Sep 10 19:11:59 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-prometheus-a[67174]: ts=2024-09-10T19:11:59.620Z caller=main.go:1029 level=info msg="TSDB started" 2024-09-10T19:11:59.968 INFO:journalctl@ceph.prometheus.a.smithi191.stdout:Sep 10 19:11:59 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-prometheus-a[67174]: ts=2024-09-10T19:11:59.620Z caller=main.go:1209 level=info msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml 2024-09-10T19:11:59.968 INFO:journalctl@ceph.prometheus.a.smithi191.stdout:Sep 10 19:11:59 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-prometheus-a[67174]: ts=2024-09-10T19:11:59.634Z caller=main.go:1246 level=info msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml totalDuration=14.391765ms db_storage=797ns remote_storage=1.26µs web_handler=291ns query_engine=502ns scrape=394.915µs scrape_sd=32.007µs notify=30.414µs notify_sd=10.31µs rules=13.678052ms tracing=5.875µs 2024-09-10T19:11:59.969 INFO:journalctl@ceph.prometheus.a.smithi191.stdout:Sep 10 19:11:59 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-prometheus-a[67174]: ts=2024-09-10T19:11:59.634Z caller=main.go:990 level=info msg="Server is ready to receive web requests." 2024-09-10T19:11:59.969 INFO:journalctl@ceph.prometheus.a.smithi191.stdout:Sep 10 19:11:59 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-prometheus-a[67174]: ts=2024-09-10T19:11:59.634Z caller=manager.go:974 level=info component="rule manager" msg="Starting rule manager..." 2024-09-10T19:11:59.969 INFO:journalctl@ceph.prometheus.a.smithi191.stdout:Sep 10 19:11:59 smithi191 bash[67159]: e9f7fcd20861732a5b312cee595b5fc835c9ca9d5e4eb4b6b509833063076edf 2024-09-10T19:11:59.969 INFO:journalctl@ceph.prometheus.a.smithi191.stdout:Sep 10 19:11:59 smithi191 systemd[1]: Started Ceph prometheus.a for 320f6df8-6fa7-11ef-bcea-c7b262605968. 2024-09-10T19:12:00.016 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Sep 10 19:11:59 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-y[29989]: [10/Sep/2024:19:11:59] ENGINE Bus STOPPING 2024-09-10T19:12:00.016 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:11:59 smithi183 ceph-mon[29736]: from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:12:00.016 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:11:59 smithi183 ceph-mon[29736]: from='mgr.14556 ' entity='mgr.y' 2024-09-10T19:12:00.016 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:11:59 smithi183 ceph-mon[29736]: from='mgr.14556 ' entity='mgr.y' 2024-09-10T19:12:00.016 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:11:59 smithi183 ceph-mon[29736]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-09-10T19:12:00.017 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:11:59 smithi183 ceph-mon[29736]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi183.front.sepia.ceph.com:9093"}]: dispatch 2024-09-10T19:12:00.017 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:11:59 smithi183 ceph-mon[29736]: from='mgr.14556 ' entity='mgr.y' 2024-09-10T19:12:00.017 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:11:59 smithi183 ceph-mon[29736]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-09-10T19:12:00.017 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:11:59 smithi183 ceph-mon[29736]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi191.front.sepia.ceph.com:9095"}]: dispatch 2024-09-10T19:12:00.017 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:11:59 smithi183 ceph-mon[29736]: from='mgr.14556 ' entity='mgr.y' 2024-09-10T19:12:00.017 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:11:59 smithi183 ceph-mon[29736]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-09-10T19:12:00.017 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:11:59 smithi183 ceph-mon[29736]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi191.front.sepia.ceph.com:3000"}]: dispatch 2024-09-10T19:12:00.017 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:11:59 smithi183 ceph-mon[29736]: from='mgr.14556 ' entity='mgr.y' 2024-09-10T19:12:00.017 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:11:59 smithi183 ceph-mon[29736]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T19:12:00.017 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:11:59 smithi183 ceph-mon[41562]: from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:12:00.018 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:11:59 smithi183 ceph-mon[41562]: from='mgr.14556 ' entity='mgr.y' 2024-09-10T19:12:00.018 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:11:59 smithi183 ceph-mon[41562]: from='mgr.14556 ' entity='mgr.y' 2024-09-10T19:12:00.018 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:11:59 smithi183 ceph-mon[41562]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-09-10T19:12:00.018 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:11:59 smithi183 ceph-mon[41562]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi183.front.sepia.ceph.com:9093"}]: dispatch 2024-09-10T19:12:00.018 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:11:59 smithi183 ceph-mon[41562]: from='mgr.14556 ' entity='mgr.y' 2024-09-10T19:12:00.018 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:11:59 smithi183 ceph-mon[41562]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-09-10T19:12:00.018 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:11:59 smithi183 ceph-mon[41562]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi191.front.sepia.ceph.com:9095"}]: dispatch 2024-09-10T19:12:00.018 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:11:59 smithi183 ceph-mon[41562]: from='mgr.14556 ' entity='mgr.y' 2024-09-10T19:12:00.018 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:11:59 smithi183 ceph-mon[41562]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-09-10T19:12:00.018 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:11:59 smithi183 ceph-mon[41562]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi191.front.sepia.ceph.com:3000"}]: dispatch 2024-09-10T19:12:00.018 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:11:59 smithi183 ceph-mon[41562]: from='mgr.14556 ' entity='mgr.y' 2024-09-10T19:12:00.018 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:11:59 smithi183 ceph-mon[41562]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T19:12:00.431 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Sep 10 19:12:00 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-y[29989]: [10/Sep/2024:19:12:00] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-09-10T19:12:00.431 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Sep 10 19:12:00 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-y[29989]: [10/Sep/2024:19:12:00] ENGINE Bus STOPPED 2024-09-10T19:12:00.431 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Sep 10 19:12:00 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-y[29989]: [10/Sep/2024:19:12:00] ENGINE Bus STARTING 2024-09-10T19:12:00.431 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Sep 10 19:12:00 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-y[29989]: [10/Sep/2024:19:12:00] ENGINE Serving on http://:::9283 2024-09-10T19:12:00.431 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Sep 10 19:12:00 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-y[29989]: [10/Sep/2024:19:12:00] ENGINE Bus STARTED 2024-09-10T19:12:00.431 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Sep 10 19:12:00 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-y[29989]: [10/Sep/2024:19:12:00] ENGINE Bus STOPPING 2024-09-10T19:12:00.931 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:12:00 smithi183 ceph-mon[29736]: 4.f scrub starts 2024-09-10T19:12:00.931 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:12:00 smithi183 ceph-mon[29736]: 4.f scrub ok 2024-09-10T19:12:00.931 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:12:00 smithi183 ceph-mon[29736]: pgmap v52: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:12:00.931 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:12:00 smithi183 ceph-mon[29736]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-09-10T19:12:00.931 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:12:00 smithi183 ceph-mon[29736]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi183.front.sepia.ceph.com:9093"}]: dispatch 2024-09-10T19:12:00.932 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:12:00 smithi183 ceph-mon[29736]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-09-10T19:12:00.932 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:12:00 smithi183 ceph-mon[29736]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi191.front.sepia.ceph.com:9095"}]: dispatch 2024-09-10T19:12:00.932 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:12:00 smithi183 ceph-mon[29736]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-09-10T19:12:00.932 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:12:00 smithi183 ceph-mon[29736]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi191.front.sepia.ceph.com:3000"}]: dispatch 2024-09-10T19:12:00.932 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Sep 10 19:12:00 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-y[29989]: [10/Sep/2024:19:12:00] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-09-10T19:12:00.932 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Sep 10 19:12:00 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-y[29989]: [10/Sep/2024:19:12:00] ENGINE Bus STOPPED 2024-09-10T19:12:00.932 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Sep 10 19:12:00 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-y[29989]: [10/Sep/2024:19:12:00] ENGINE Bus STARTING 2024-09-10T19:12:00.932 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Sep 10 19:12:00 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-y[29989]: [10/Sep/2024:19:12:00] ENGINE Serving on http://:::9283 2024-09-10T19:12:00.932 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Sep 10 19:12:00 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-y[29989]: [10/Sep/2024:19:12:00] ENGINE Bus STARTED 2024-09-10T19:12:00.932 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Sep 10 19:12:00 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-y[29989]: [10/Sep/2024:19:12:00] ENGINE Bus STOPPING 2024-09-10T19:12:00.933 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:12:00 smithi183 ceph-mon[41562]: 4.f scrub starts 2024-09-10T19:12:00.933 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:12:00 smithi183 ceph-mon[41562]: 4.f scrub ok 2024-09-10T19:12:00.933 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:12:00 smithi183 ceph-mon[41562]: pgmap v52: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:12:00.933 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:12:00 smithi183 ceph-mon[41562]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-09-10T19:12:00.933 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:12:00 smithi183 ceph-mon[41562]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi183.front.sepia.ceph.com:9093"}]: dispatch 2024-09-10T19:12:00.933 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:12:00 smithi183 ceph-mon[41562]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-09-10T19:12:00.933 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:12:00 smithi183 ceph-mon[41562]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi191.front.sepia.ceph.com:9095"}]: dispatch 2024-09-10T19:12:00.933 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:12:00 smithi183 ceph-mon[41562]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-09-10T19:12:00.933 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:12:00 smithi183 ceph-mon[41562]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi191.front.sepia.ceph.com:3000"}]: dispatch 2024-09-10T19:12:00.965 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:12:00 smithi191 ceph-mon[32884]: 4.f scrub starts 2024-09-10T19:12:00.966 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:12:00 smithi191 ceph-mon[32884]: 4.f scrub ok 2024-09-10T19:12:00.966 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:12:00 smithi191 ceph-mon[32884]: pgmap v52: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:12:00.966 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:12:00 smithi191 ceph-mon[32884]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-09-10T19:12:00.966 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:12:00 smithi191 ceph-mon[32884]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi183.front.sepia.ceph.com:9093"}]: dispatch 2024-09-10T19:12:00.966 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:12:00 smithi191 ceph-mon[32884]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-09-10T19:12:00.966 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:12:00 smithi191 ceph-mon[32884]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi191.front.sepia.ceph.com:9095"}]: dispatch 2024-09-10T19:12:00.966 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:12:00 smithi191 ceph-mon[32884]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-09-10T19:12:00.966 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:12:00 smithi191 ceph-mon[32884]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi191.front.sepia.ceph.com:3000"}]: dispatch 2024-09-10T19:12:01.577 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Sep 10 19:12:01 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-y[29989]: [10/Sep/2024:19:12:01] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-09-10T19:12:01.577 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Sep 10 19:12:01 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-y[29989]: [10/Sep/2024:19:12:01] ENGINE Bus STOPPED 2024-09-10T19:12:01.577 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Sep 10 19:12:01 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-y[29989]: [10/Sep/2024:19:12:01] ENGINE Bus STARTING 2024-09-10T19:12:01.577 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Sep 10 19:12:01 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-y[29989]: [10/Sep/2024:19:12:01] ENGINE Serving on http://:::9283 2024-09-10T19:12:01.577 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Sep 10 19:12:01 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-y[29989]: [10/Sep/2024:19:12:01] ENGINE Bus STARTED 2024-09-10T19:12:01.792 INFO:teuthology.orchestra.run.smithi183.stdout:mon.a: {} 2024-09-10T19:12:01.793 INFO:teuthology.orchestra.run.smithi183.stdout:mon.b: {} 2024-09-10T19:12:01.793 INFO:teuthology.orchestra.run.smithi183.stdout:mon.c: {} 2024-09-10T19:12:01.793 INFO:teuthology.orchestra.run.smithi183.stderr:mon.a: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-09-10T19:12:01.793 INFO:teuthology.orchestra.run.smithi183.stderr:mon.b: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-09-10T19:12:01.793 INFO:teuthology.orchestra.run.smithi183.stderr:mon.c: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-09-10T19:12:01.931 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:12:01 smithi183 ceph-mon[29736]: 5.e scrub starts 2024-09-10T19:12:01.931 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:12:01 smithi183 ceph-mon[29736]: 5.e scrub ok 2024-09-10T19:12:01.931 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:12:01 smithi183 ceph-mon[29736]: 5.c scrub starts 2024-09-10T19:12:01.932 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:12:01 smithi183 ceph-mon[29736]: 5.c scrub ok 2024-09-10T19:12:01.932 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:12:01 smithi183 ceph-mon[29736]: 5.13 scrub starts 2024-09-10T19:12:01.932 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:12:01 smithi183 ceph-mon[29736]: 5.13 scrub ok 2024-09-10T19:12:01.932 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:12:01 smithi183 ceph-mon[29736]: from='client.? 172.21.15.183:0/803387613' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-10T19:12:01.932 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:12:01 smithi183 ceph-mon[41562]: 5.e scrub starts 2024-09-10T19:12:01.932 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:12:01 smithi183 ceph-mon[41562]: 5.e scrub ok 2024-09-10T19:12:01.932 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:12:01 smithi183 ceph-mon[41562]: 5.c scrub starts 2024-09-10T19:12:01.932 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:12:01 smithi183 ceph-mon[41562]: 5.c scrub ok 2024-09-10T19:12:01.932 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:12:01 smithi183 ceph-mon[41562]: 5.13 scrub starts 2024-09-10T19:12:01.932 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:12:01 smithi183 ceph-mon[41562]: 5.13 scrub ok 2024-09-10T19:12:01.933 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:12:01 smithi183 ceph-mon[41562]: from='client.? 172.21.15.183:0/803387613' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-10T19:12:02.136 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:12:01 smithi191 ceph-mon[32884]: 5.e scrub starts 2024-09-10T19:12:02.137 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:12:01 smithi191 ceph-mon[32884]: 5.e scrub ok 2024-09-10T19:12:02.137 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:12:01 smithi191 ceph-mon[32884]: 5.c scrub starts 2024-09-10T19:12:02.137 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:12:01 smithi191 ceph-mon[32884]: 5.c scrub ok 2024-09-10T19:12:02.137 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:12:01 smithi191 ceph-mon[32884]: 5.13 scrub starts 2024-09-10T19:12:02.137 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:12:01 smithi191 ceph-mon[32884]: 5.13 scrub ok 2024-09-10T19:12:02.137 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:12:01 smithi191 ceph-mon[32884]: from='client.? 172.21.15.183:0/803387613' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-10T19:12:02.573 INFO:teuthology.run_tasks:Running task full_sequential... 2024-09-10T19:12:02.584 INFO:teuthology.task.full_sequential:In full_sequential, running task radosbench... 2024-09-10T19:12:02.585 INFO:tasks.radosbench:Beginning radosbench... 2024-09-10T19:12:02.585 INFO:tasks.cephadm.ceph_manager.ceph:creating pool_name unique_pool_0 2024-09-10T19:12:02.586 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph osd pool create unique_pool_0 16 2024-09-10T19:12:02.586 INFO:tasks.thrashosds.thrasher:starting do_thrash 2024-09-10T19:12:02.586 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-10T19:12:02.586 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-10T19:12:02.586 INFO:tasks.thrashosds.thrasher:starting do_sighup with a delay of 0.1 2024-09-10T19:12:02.587 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:12:02.587 INFO:tasks.thrashosds.thrasher:starting do_optrack_toggle with a delay of 2.0 2024-09-10T19:12:02.587 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-10T19:12:02.588 INFO:tasks.thrashosds.thrasher:starting do_dump_ops 2024-09-10T19:12:02.588 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph daemon osd.0 dump_ops_in_flight 2024-09-10T19:12:02.588 INFO:tasks.thrashosds.thrasher:starting do_noscrub_toggle with a delay of 2.0 2024-09-10T19:12:02.588 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph osd set noscrub 2024-09-10T19:12:02.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:12:02.615 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:12:02.615 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:12:02.716 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:12:02.739 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:12:02.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:12:02.740 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:12:02.841 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:12:02.863 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:12:02.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:12:02.864 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:12:02.965 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:12:02.986 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:12:02.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:12:02.987 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:12:03.060 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:12:03.071 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:12:03.088 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:12:03.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:12:03.110 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:12:03.110 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:12:03.120 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:12:03.120 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:12:03.181 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:12:02 smithi183 ceph-mon[41562]: pgmap v53: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-10T19:12:03.181 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:12:02 smithi183 ceph-mon[41562]: 5.4 scrub starts 2024-09-10T19:12:03.181 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:12:02 smithi183 ceph-mon[41562]: 5.4 scrub ok 2024-09-10T19:12:03.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:12:02 smithi183 ceph-mon[29736]: pgmap v53: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-10T19:12:03.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:12:02 smithi183 ceph-mon[29736]: 5.4 scrub starts 2024-09-10T19:12:03.182 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:12:02 smithi183 ceph-mon[29736]: 5.4 scrub ok 2024-09-10T19:12:03.211 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:12:03.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:12:02 smithi191 ceph-mon[32884]: pgmap v53: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-10T19:12:03.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:12:02 smithi191 ceph-mon[32884]: 5.4 scrub starts 2024-09-10T19:12:03.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:12:02 smithi191 ceph-mon[32884]: 5.4 scrub ok 2024-09-10T19:12:03.233 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:12:03.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:12:03.234 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:12:03.334 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:12:03.357 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:12:03.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:12:03.358 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:12:03.459 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:12:03.481 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:12:03.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:12:03.482 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:12:03.583 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:12:03.604 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:12:03.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:12:03.605 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:12:03.706 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:12:03.727 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:12:03.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:12:03.728 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:12:03.804 INFO:journalctl@ceph.alertmanager.a.smithi183.stdout:Sep 10 19:12:03 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-alertmanager-a[80531]: ts=2024-09-10T19:12:03.627Z caller=cluster.go:698 level=info component=cluster msg="gossip settled; proceeding" elapsed=10.003430204s 2024-09-10T19:12:03.829 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:12:03.851 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:12:03.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:12:03.852 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:12:03.953 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:12:03.975 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:12:03.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:12:03.976 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:12:04.076 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:12:04.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:12:04.099 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:12:04.100 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:12:04.202 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:12:04.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:12:04.224 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:12:04.224 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:12:04.325 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:12:04.347 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:12:04.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:12:04.347 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:12:04.448 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:12:04.470 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:12:04.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:12:04.471 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:12:04.572 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:12:04.593 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:12:04.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:12:04.594 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:12:04.695 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:12:04.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:12:04.717 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:12:04.718 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:12:04.818 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:12:04.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:12:04.841 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:12:04.841 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:12:04.942 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:12:04.964 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:12:04.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:12:04.966 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:12:05.067 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:12:05.089 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:12:05.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:12:05.090 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:12:05.191 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:12:05.213 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:12:05.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:12:05.215 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:12:05.316 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:12:05.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:12:05.339 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:12:05.340 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:12:05.348 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:12:04 smithi183 ceph-mon[29736]: pgmap v54: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-10T19:12:05.348 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:12:04 smithi183 ceph-mon[29736]: from='mgr.14556 ' entity='mgr.y' 2024-09-10T19:12:05.348 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:12:04 smithi183 ceph-mon[29736]: from='mgr.14556 ' entity='mgr.y' 2024-09-10T19:12:05.349 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:12:04 smithi183 ceph-mon[29736]: 4.18 scrub starts 2024-09-10T19:12:05.349 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:12:04 smithi183 ceph-mon[29736]: 4.18 scrub ok 2024-09-10T19:12:05.349 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:12:04 smithi183 ceph-mon[41562]: pgmap v54: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-10T19:12:05.350 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:12:04 smithi183 ceph-mon[41562]: from='mgr.14556 ' entity='mgr.y' 2024-09-10T19:12:05.350 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:12:04 smithi183 ceph-mon[41562]: from='mgr.14556 ' entity='mgr.y' 2024-09-10T19:12:05.350 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:12:04 smithi183 ceph-mon[41562]: 4.18 scrub starts 2024-09-10T19:12:05.350 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:12:04 smithi183 ceph-mon[41562]: 4.18 scrub ok 2024-09-10T19:12:05.441 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:12:05.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:12:05.464 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:12:05.464 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:12:05.465 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:12:04 smithi191 ceph-mon[32884]: pgmap v54: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-10T19:12:05.465 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:12:04 smithi191 ceph-mon[32884]: from='mgr.14556 ' entity='mgr.y' 2024-09-10T19:12:05.465 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:12:04 smithi191 ceph-mon[32884]: from='mgr.14556 ' entity='mgr.y' 2024-09-10T19:12:05.465 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:12:04 smithi191 ceph-mon[32884]: 4.18 scrub starts 2024-09-10T19:12:05.465 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:12:04 smithi191 ceph-mon[32884]: 4.18 scrub ok 2024-09-10T19:12:05.565 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:12:05.587 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:12:05.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:12:05.588 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:12:05.688 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:12:05.710 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:12:05.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:12:05.711 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:12:05.812 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:12:05.834 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:12:05.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:12:05.835 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:12:05.935 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:12:05.957 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:12:05.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:12:05.958 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:12:06.059 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:12:06.082 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:12:06.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:12:06.083 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:12:06.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:12:05 smithi183 ceph-mon[29736]: 6.4 scrub starts 2024-09-10T19:12:06.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:12:05 smithi183 ceph-mon[29736]: 6.4 scrub ok 2024-09-10T19:12:06.181 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:12:05 smithi183 ceph-mon[41562]: 6.4 scrub starts 2024-09-10T19:12:06.181 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:12:05 smithi183 ceph-mon[41562]: 6.4 scrub ok 2024-09-10T19:12:06.184 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:12:06.206 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:12:06.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:12:06.207 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:12:06.308 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:12:06.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:12:06.330 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:12:06.330 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:12:06.431 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:12:06.452 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:12:06.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:12:06.453 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:12:06.465 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:12:05 smithi191 ceph-mon[32884]: 6.4 scrub starts 2024-09-10T19:12:06.465 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:12:05 smithi191 ceph-mon[32884]: 6.4 scrub ok 2024-09-10T19:12:06.556 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:12:06.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:12:06.580 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:12:06.580 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:12:06.681 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:12:06.704 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:12:06.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:12:06.705 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:12:06.806 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:12:06.829 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:12:06.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:12:06.830 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:12:06.930 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:12:06.954 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:12:06.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:12:06.955 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:12:07.056 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:12:07.077 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:12:07.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:12:07.078 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:12:07.098 INFO:teuthology.orchestra.run.smithi183.stdout:osd.0: {} 2024-09-10T19:12:07.098 INFO:teuthology.orchestra.run.smithi183.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-10T19:12:07.120 INFO:teuthology.orchestra.run.smithi183.stdout:osd.1: {} 2024-09-10T19:12:07.120 INFO:teuthology.orchestra.run.smithi183.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-10T19:12:07.136 INFO:teuthology.orchestra.run.smithi183.stdout:osd.2: {} 2024-09-10T19:12:07.136 INFO:teuthology.orchestra.run.smithi183.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-10T19:12:07.151 INFO:teuthology.orchestra.run.smithi183.stdout:osd.3: {} 2024-09-10T19:12:07.152 INFO:teuthology.orchestra.run.smithi183.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-10T19:12:07.167 INFO:teuthology.orchestra.run.smithi183.stdout:osd.4: {} 2024-09-10T19:12:07.168 INFO:teuthology.orchestra.run.smithi183.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-10T19:12:07.179 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:12:07.183 INFO:teuthology.orchestra.run.smithi183.stdout:osd.5: {} 2024-09-10T19:12:07.183 INFO:teuthology.orchestra.run.smithi183.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-10T19:12:07.200 INFO:teuthology.orchestra.run.smithi183.stdout:osd.6: {} 2024-09-10T19:12:07.200 INFO:teuthology.orchestra.run.smithi183.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-10T19:12:07.204 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:12:07.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:12:07.205 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:12:07.217 INFO:teuthology.orchestra.run.smithi183.stdout:osd.7: {} 2024-09-10T19:12:07.217 INFO:teuthology.orchestra.run.smithi183.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-10T19:12:07.277 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:12:06 smithi183 ceph-mon[29736]: pgmap v55: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-09-10T19:12:07.277 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:12:06 smithi183 ceph-mon[29736]: 6.1b scrub starts 2024-09-10T19:12:07.277 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:12:06 smithi183 ceph-mon[29736]: 6.1b scrub ok 2024-09-10T19:12:07.277 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:12:06 smithi183 ceph-mon[29736]: 6.5 scrub starts 2024-09-10T19:12:07.277 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:12:06 smithi183 ceph-mon[29736]: 6.5 scrub ok 2024-09-10T19:12:07.277 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:12:06 smithi183 ceph-mon[41562]: pgmap v55: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-09-10T19:12:07.277 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:12:06 smithi183 ceph-mon[41562]: 6.1b scrub starts 2024-09-10T19:12:07.277 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:12:06 smithi183 ceph-mon[41562]: 6.1b scrub ok 2024-09-10T19:12:07.277 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:12:06 smithi183 ceph-mon[41562]: 6.5 scrub starts 2024-09-10T19:12:07.277 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:12:06 smithi183 ceph-mon[41562]: 6.5 scrub ok 2024-09-10T19:12:07.305 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:12:07.330 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:12:07.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:12:07.331 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:12:07.431 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:12:07.451 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:12:06 smithi191 ceph-mon[32884]: pgmap v55: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-09-10T19:12:07.451 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:12:06 smithi191 ceph-mon[32884]: 6.1b scrub starts 2024-09-10T19:12:07.451 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:12:06 smithi191 ceph-mon[32884]: 6.1b scrub ok 2024-09-10T19:12:07.451 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:12:06 smithi191 ceph-mon[32884]: 6.5 scrub starts 2024-09-10T19:12:07.451 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:12:06 smithi191 ceph-mon[32884]: 6.5 scrub ok 2024-09-10T19:12:07.453 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:12:07.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:12:07.454 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:12:07.555 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:12:07.576 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:12:07.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:12:07.577 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:12:07.678 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:12:07.701 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:12:07.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:12:07.702 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:12:07.715 INFO:journalctl@ceph.mgr.x.smithi191.stdout:Sep 10 19:12:07 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-x[34967]: ::ffff:172.21.15.191 - - [10/Sep/2024:19:12:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T19:12:07.802 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:12:07.829 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:12:07.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:12:07.830 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:12:07.931 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:12:07.956 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:12:07.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:12:07.957 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:12:08.058 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:12:08.079 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:12:08.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:12:08.080 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:12:08.181 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:12:08.202 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:12:08.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:12:08.203 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:12:08.304 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:12:08.326 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:12:08.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:12:08.327 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:12:08.359 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:12:07 smithi183 ceph-mon[29736]: from='client.? 172.21.15.183:0/2024328028' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T19:12:08.359 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:12:07 smithi183 ceph-mon[41562]: from='client.? 172.21.15.183:0/2024328028' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T19:12:08.428 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:12:08.449 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:12:08.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:12:08.450 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:12:08.465 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:12:07 smithi191 ceph-mon[32884]: from='client.? 172.21.15.183:0/2024328028' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T19:12:08.551 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:12:08.577 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:12:08.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:12:08.578 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:12:08.678 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:12:08.703 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:12:08.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:12:08.705 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:12:08.805 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:12:08.831 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:12:08.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:12:08.832 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:12:08.933 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:12:08.955 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:12:08.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:12:08.956 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:12:09.057 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:12:09.077 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:12:09 smithi183 ceph-mon[29736]: pgmap v56: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-10T19:12:09.078 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:12:09 smithi183 ceph-mon[29736]: purged_snaps scrub starts 2024-09-10T19:12:09.078 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:12:09 smithi183 ceph-mon[29736]: purged_snaps scrub ok 2024-09-10T19:12:09.078 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:12:09 smithi183 ceph-mon[41562]: pgmap v56: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-10T19:12:09.078 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:12:09 smithi183 ceph-mon[41562]: purged_snaps scrub starts 2024-09-10T19:12:09.078 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:12:09 smithi183 ceph-mon[41562]: purged_snaps scrub ok 2024-09-10T19:12:09.080 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:12:09.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:12:09.081 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:12:09.182 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:12:09.205 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:12:09.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:12:09.206 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:12:09.216 INFO:journalctl@ceph.iscsi.iscsi.a.smithi191.stdout:Sep 10 19:12:08 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-iscsi-iscsi-a[61816]: debug there is no tcmu-runner data available 2024-09-10T19:12:09.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:12:09 smithi191 ceph-mon[32884]: pgmap v56: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-10T19:12:09.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:12:09 smithi191 ceph-mon[32884]: purged_snaps scrub starts 2024-09-10T19:12:09.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:12:09 smithi191 ceph-mon[32884]: purged_snaps scrub ok 2024-09-10T19:12:09.307 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:12:09.329 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:12:09.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:12:09.330 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:12:09.431 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:12:09.453 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:12:09.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:12:09.454 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:12:09.555 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:12:09.576 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:12:09.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:12:09.577 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:12:09.678 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:12:09.704 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:12:09.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:12:09.705 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:12:09.741 INFO:teuthology.orchestra.run.smithi183.stdout:ERROR: (22) Invalid argument 2024-09-10T19:12:09.742 INFO:teuthology.orchestra.run.smithi183.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-10T19:12:09.806 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:12:09.837 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:12:09.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:12:09.838 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:12:09.938 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:12:09.962 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:12:09.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:12:09.963 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:12:10.019 INFO:teuthology.orchestra.run.smithi183.stderr:pool 'unique_pool_0' created 2024-09-10T19:12:10.063 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:12:10.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:12:10.087 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:12:10.087 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:12:10.188 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:12:10.210 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:12:10.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:12:10.211 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:12:10.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:12:10 smithi191 ceph-mon[32884]: from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:12:10.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:12:10 smithi191 ceph-mon[32884]: osdmap e85: 8 total, 8 up, 8 in 2024-09-10T19:12:10.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:12:10 smithi191 ceph-mon[32884]: from='client.? 172.21.15.183:0/948872065' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16}]: dispatch 2024-09-10T19:12:10.312 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:12:10.333 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:12:10.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:12:10.334 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:12:10.431 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:12:10 smithi183 ceph-mon[41562]: from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:12:10.431 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:12:10 smithi183 ceph-mon[41562]: osdmap e85: 8 total, 8 up, 8 in 2024-09-10T19:12:10.431 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:12:10 smithi183 ceph-mon[41562]: from='client.? 172.21.15.183:0/948872065' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16}]: dispatch 2024-09-10T19:12:10.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:12:10 smithi183 ceph-mon[29736]: from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:12:10.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:12:10 smithi183 ceph-mon[29736]: osdmap e85: 8 total, 8 up, 8 in 2024-09-10T19:12:10.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:12:10 smithi183 ceph-mon[29736]: from='client.? 172.21.15.183:0/948872065' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16}]: dispatch 2024-09-10T19:12:10.435 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:12:10.457 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:12:10.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:12:10.458 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:12:10.558 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:12:10.579 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:12:10.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:12:10.580 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:12:10.681 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:12:10.703 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:12:10.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:12:10.704 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:12:10.722 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph daemon osd.0 dump_blocked_ops 2024-09-10T19:12:10.805 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:12:10.826 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:12:10.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:12:10.827 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:12:10.928 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:12:10.950 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:12:10.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:12:10.951 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:12:11.051 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:12:11.072 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:12:11.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:12:11.073 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:12:11.174 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:12:11.200 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:12:11.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:12:11.201 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:12:11.204 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph osd pool application enable unique_pool_0 rados --yes-i-really-mean-it || true 2024-09-10T19:12:11.302 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:12:11.311 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:12:11 smithi183 ceph-mon[29736]: pgmap v58: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-10T19:12:11.311 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:12:11 smithi183 ceph-mon[29736]: 1.0 scrub starts 2024-09-10T19:12:11.311 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:12:11 smithi183 ceph-mon[29736]: 1.0 scrub ok 2024-09-10T19:12:11.311 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:12:11 smithi183 ceph-mon[29736]: from='client.? 172.21.15.183:0/948872065' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16}]': finished 2024-09-10T19:12:11.311 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:12:11 smithi183 ceph-mon[29736]: osdmap e86: 8 total, 8 up, 8 in 2024-09-10T19:12:11.312 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:12:11 smithi183 ceph-mon[41562]: pgmap v58: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-10T19:12:11.312 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:12:11 smithi183 ceph-mon[41562]: 1.0 scrub starts 2024-09-10T19:12:11.312 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:12:11 smithi183 ceph-mon[41562]: 1.0 scrub ok 2024-09-10T19:12:11.312 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:12:11 smithi183 ceph-mon[41562]: from='client.? 172.21.15.183:0/948872065' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16}]': finished 2024-09-10T19:12:11.312 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:12:11 smithi183 ceph-mon[41562]: osdmap e86: 8 total, 8 up, 8 in 2024-09-10T19:12:11.324 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:12:11.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:12:11.325 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:12:11.426 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:12:11.447 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:12:11.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:12:11.448 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:12:11.465 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:12:11 smithi191 ceph-mon[32884]: pgmap v58: 132 pgs: 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-10T19:12:11.465 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:12:11 smithi191 ceph-mon[32884]: 1.0 scrub starts 2024-09-10T19:12:11.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:12:11 smithi191 ceph-mon[32884]: 1.0 scrub ok 2024-09-10T19:12:11.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:12:11 smithi191 ceph-mon[32884]: from='client.? 172.21.15.183:0/948872065' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16}]': finished 2024-09-10T19:12:11.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:12:11 smithi191 ceph-mon[32884]: osdmap e86: 8 total, 8 up, 8 in 2024-09-10T19:12:11.548 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:12:11.569 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:12:11.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:12:11.570 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:12:11.672 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:12:11.681 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Sep 10 19:12:11 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-y[29989]: ::ffff:172.21.15.191 - - [10/Sep/2024:19:12:11] "GET /metrics HTTP/1.1" 200 220070 "" "Prometheus/2.43.0" 2024-09-10T19:12:11.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:12:11.694 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:12:11.694 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:12:11.795 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:12:11.821 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:12:11.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:12:11.822 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:12:11.923 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:12:11.946 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:12:11.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:12:11.947 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:12:12.047 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:12:12.070 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:12:12.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:12:12.071 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:12:12.171 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:12:12.197 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:12:12.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:12:12.198 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:12:12.275 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:12:12 smithi183 ceph-mon[29736]: 4.19 scrub starts 2024-09-10T19:12:12.275 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:12:12 smithi183 ceph-mon[29736]: 4.19 scrub ok 2024-09-10T19:12:12.275 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:12:12 smithi183 ceph-mon[29736]: osdmap e87: 8 total, 8 up, 8 in 2024-09-10T19:12:12.276 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:12:12 smithi183 ceph-mon[41562]: 4.19 scrub starts 2024-09-10T19:12:12.276 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:12:12 smithi183 ceph-mon[41562]: 4.19 scrub ok 2024-09-10T19:12:12.276 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:12:12 smithi183 ceph-mon[41562]: osdmap e87: 8 total, 8 up, 8 in 2024-09-10T19:12:12.299 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:12:12.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:12:12.326 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:12:12.326 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:12:12.426 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:12:12.453 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:12:12.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:12:12.454 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:12:12.465 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:12:12 smithi191 ceph-mon[32884]: 4.19 scrub starts 2024-09-10T19:12:12.465 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:12:12 smithi191 ceph-mon[32884]: 4.19 scrub ok 2024-09-10T19:12:12.465 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:12:12 smithi191 ceph-mon[32884]: osdmap e87: 8 total, 8 up, 8 in 2024-09-10T19:12:12.530 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:12:12.531 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:12:12.554 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:12:12.578 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:12:12.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:12:12.579 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:12:12.680 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:12:12.702 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:12:12.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:12:12.703 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:12:12.803 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:12:12.828 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:12:12.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:12:12.829 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:12:12.930 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:12:12.951 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:12:12.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:12:12.952 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:12:13.038 INFO:teuthology.orchestra.run.smithi183.stderr:noscrub is set 2024-09-10T19:12:13.053 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:12:13.074 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:12:13.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:12:13.075 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:12:13.176 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:12:13.202 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:12:13.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:12:13.203 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:12:13.244 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:12:13 smithi183 ceph-mon[29736]: pgmap v61: 148 pgs: 16 unknown, 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-10T19:12:13.244 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:12:13 smithi183 ceph-mon[29736]: 5.17 scrub starts 2024-09-10T19:12:13.244 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:12:13 smithi183 ceph-mon[29736]: 5.17 scrub ok 2024-09-10T19:12:13.244 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:12:13 smithi183 ceph-mon[29736]: purged_snaps scrub starts 2024-09-10T19:12:13.244 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:12:13 smithi183 ceph-mon[29736]: purged_snaps scrub ok 2024-09-10T19:12:13.244 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:12:13 smithi183 ceph-mon[29736]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T19:12:13.244 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:12:13 smithi183 ceph-mon[29736]: osdmap e88: 8 total, 8 up, 8 in 2024-09-10T19:12:13.244 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:12:13 smithi183 ceph-mon[29736]: from='client.? 172.21.15.183:0/3815681929' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-10T19:12:13.245 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:12:13 smithi183 ceph-mon[41562]: pgmap v61: 148 pgs: 16 unknown, 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-10T19:12:13.245 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:12:13 smithi183 ceph-mon[41562]: 5.17 scrub starts 2024-09-10T19:12:13.245 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:12:13 smithi183 ceph-mon[41562]: 5.17 scrub ok 2024-09-10T19:12:13.245 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:12:13 smithi183 ceph-mon[41562]: purged_snaps scrub starts 2024-09-10T19:12:13.245 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:12:13 smithi183 ceph-mon[41562]: purged_snaps scrub ok 2024-09-10T19:12:13.245 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:12:13 smithi183 ceph-mon[41562]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T19:12:13.245 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:12:13 smithi183 ceph-mon[41562]: osdmap e88: 8 total, 8 up, 8 in 2024-09-10T19:12:13.245 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:12:13 smithi183 ceph-mon[41562]: from='client.? 172.21.15.183:0/3815681929' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-10T19:12:13.304 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:12:13.329 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:12:13.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:12:13.330 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:12:13.431 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:12:13.456 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:12:13.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:12:13.457 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:12:13.465 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:12:13 smithi191 ceph-mon[32884]: pgmap v61: 148 pgs: 16 unknown, 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-10T19:12:13.465 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:12:13 smithi191 ceph-mon[32884]: 5.17 scrub starts 2024-09-10T19:12:13.465 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:12:13 smithi191 ceph-mon[32884]: 5.17 scrub ok 2024-09-10T19:12:13.465 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:12:13 smithi191 ceph-mon[32884]: purged_snaps scrub starts 2024-09-10T19:12:13.465 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:12:13 smithi191 ceph-mon[32884]: purged_snaps scrub ok 2024-09-10T19:12:13.465 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:12:13 smithi191 ceph-mon[32884]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T19:12:13.465 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:12:13 smithi191 ceph-mon[32884]: osdmap e88: 8 total, 8 up, 8 in 2024-09-10T19:12:13.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:12:13 smithi191 ceph-mon[32884]: from='client.? 172.21.15.183:0/3815681929' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-10T19:12:13.558 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:12:13.583 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:12:13.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:12:13.584 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:12:13.685 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:12:13.706 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:12:13.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:12:13.707 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:12:13.728 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-10T19:12:13.808 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:12:13.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:12:13.831 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:12:13.831 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:12:13.932 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:12:13.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:12:13.954 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:12:13.954 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:12:14.055 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:12:14.076 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:12:14.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:12:14.077 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:12:14.178 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:12:14.201 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:12:14.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:12:14.202 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:12:14.303 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:12:14.313 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:12:14 smithi183 ceph-mon[29736]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T19:12:14.313 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:12:14 smithi183 ceph-mon[29736]: from='client.? 172.21.15.183:0/3815681929' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-10T19:12:14.313 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:12:14 smithi183 ceph-mon[29736]: osdmap e89: 8 total, 8 up, 8 in 2024-09-10T19:12:14.313 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:12:14 smithi183 ceph-mon[29736]: from='mgr.14556 ' entity='mgr.y' 2024-09-10T19:12:14.313 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:12:14 smithi183 ceph-mon[29736]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:12:14.313 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:12:14 smithi183 ceph-mon[41562]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T19:12:14.313 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:12:14 smithi183 ceph-mon[41562]: from='client.? 172.21.15.183:0/3815681929' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-10T19:12:14.313 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:12:14 smithi183 ceph-mon[41562]: osdmap e89: 8 total, 8 up, 8 in 2024-09-10T19:12:14.313 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:12:14 smithi183 ceph-mon[41562]: from='mgr.14556 ' entity='mgr.y' 2024-09-10T19:12:14.314 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:12:14 smithi183 ceph-mon[41562]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:12:14.329 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:12:14.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:12:14.331 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:12:14.431 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:12:14.453 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:12:14.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:12:14.454 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:12:14.465 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:12:14 smithi191 ceph-mon[32884]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T19:12:14.465 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:12:14 smithi191 ceph-mon[32884]: from='client.? 172.21.15.183:0/3815681929' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-10T19:12:14.465 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:12:14 smithi191 ceph-mon[32884]: osdmap e89: 8 total, 8 up, 8 in 2024-09-10T19:12:14.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:12:14 smithi191 ceph-mon[32884]: from='mgr.14556 ' entity='mgr.y' 2024-09-10T19:12:14.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:12:14 smithi191 ceph-mon[32884]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:12:14.554 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:12:14.580 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:12:14.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:12:14.581 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:12:14.682 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:12:14.704 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:12:14.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:12:14.705 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:12:14.806 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:12:14.827 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:12:14.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:12:14.828 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:12:14.929 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:12:14.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:12:14.955 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:12:14.955 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:12:15.056 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:12:15.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:12:15.079 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:12:15.079 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:12:15.180 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:12:15.201 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:12:15.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:12:15.202 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:12:15.303 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:12:15.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:12:15.325 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:12:15.326 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:12:15.418 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:12:15 smithi183 ceph-mon[29736]: pgmap v64: 148 pgs: 16 unknown, 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:12:15.418 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:12:15 smithi183 ceph-mon[41562]: pgmap v64: 148 pgs: 16 unknown, 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:12:15.427 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:12:15.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:12:15.450 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:12:15.451 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:12:15.465 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:12:15 smithi191 ceph-mon[32884]: pgmap v64: 148 pgs: 16 unknown, 132 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail 2024-09-10T19:12:15.551 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:12:15.578 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:12:15.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:12:15.579 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:12:15.679 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:12:15.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:12:15.702 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:12:15.703 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:12:15.804 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:12:15.829 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:12:15.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:12:15.830 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:12:15.931 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:12:15.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:12:15.959 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:12:15.959 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:12:16.060 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:12:16.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:12:16.085 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:12:16.085 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:12:16.186 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:12:16.212 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-10T19:12:16.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:12:16.214 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:12:16.214 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:12:16.315 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:12:16.339 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:12:16.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:12:16.340 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:12:16.441 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:12:16.466 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:12:16.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:12:16.468 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:12:16.568 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:12:16.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:12:16.591 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:12:16.591 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:12:16.692 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:12:16.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:12:16.714 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:12:16.714 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:12:16.815 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:12:16.837 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:12:16.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:12:16.838 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:12:16.938 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:12:16.961 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:12:16.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:12:16.962 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:12:17.050 INFO:teuthology.orchestra.run.smithi183.stderr:enabled application 'rados' on pool 'unique_pool_0' 2024-09-10T19:12:17.062 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:12:17.088 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:12:17.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:12:17.089 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:12:17.190 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:12:17.215 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:12:17.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:12:17.216 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:12:17.317 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:12:17.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:12:17.338 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:12:17.339 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:12:17.351 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:12:17 smithi183 ceph-mon[41562]: pgmap v65: 148 pgs: 148 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 914 B/s rd, 0 op/s 2024-09-10T19:12:17.352 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:12:17 smithi183 ceph-mon[41562]: from='client.? 172.21.15.183:0/3115605172' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:12:17.352 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:12:17 smithi183 ceph-mon[41562]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:12:17.352 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:12:17 smithi183 ceph-mon[29736]: pgmap v65: 148 pgs: 148 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 914 B/s rd, 0 op/s 2024-09-10T19:12:17.352 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:12:17 smithi183 ceph-mon[29736]: from='client.? 172.21.15.183:0/3115605172' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:12:17.352 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:12:17 smithi183 ceph-mon[29736]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:12:17.440 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:12:17.449 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:12:17 smithi191 ceph-mon[32884]: pgmap v65: 148 pgs: 148 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 914 B/s rd, 0 op/s 2024-09-10T19:12:17.450 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:12:17 smithi191 ceph-mon[32884]: from='client.? 172.21.15.183:0/3115605172' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:12:17.450 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:12:17 smithi191 ceph-mon[32884]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:12:17.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:12:17.463 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:12:17.463 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:12:17.564 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:12:17.590 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:12:17.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:12:17.591 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:12:17.692 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:12:17.715 INFO:journalctl@ceph.mgr.x.smithi191.stdout:Sep 10 19:12:17 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-x[34967]: ::ffff:172.21.15.191 - - [10/Sep/2024:19:12:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T19:12:17.718 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:12:17.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:12:17.719 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:12:17.820 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:12:17.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:12:17.847 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:12:17.848 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:12:17.948 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:12:17.975 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:12:17.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:12:17.976 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:12:18.043 INFO:teuthology.orchestra.run.smithi183.stdout:ERROR: (22) Invalid argument 2024-09-10T19:12:18.043 INFO:teuthology.orchestra.run.smithi183.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-10T19:12:18.077 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:12:18.103 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:12:18.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:12:18.104 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:12:18.205 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:12:18.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:12:18.231 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:12:18.232 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:12:18.333 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:12:18.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:12:18.361 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:12:18.362 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:12:18.364 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:12:18 smithi183 ceph-mon[29736]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-09-10T19:12:18.364 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:12:18 smithi183 ceph-mon[29736]: osdmap e90: 8 total, 8 up, 8 in 2024-09-10T19:12:18.365 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:12:18 smithi183 ceph-mon[41562]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-09-10T19:12:18.365 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:12:18 smithi183 ceph-mon[41562]: osdmap e90: 8 total, 8 up, 8 in 2024-09-10T19:12:18.462 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:12:18.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:12:18 smithi191 ceph-mon[32884]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-09-10T19:12:18.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:12:18 smithi191 ceph-mon[32884]: osdmap e90: 8 total, 8 up, 8 in 2024-09-10T19:12:18.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:12:18.489 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:12:18.489 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:12:18.590 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:12:18.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:12:18.612 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:12:18.613 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:12:18.671 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph daemon osd.0 dump_historic_ops 2024-09-10T19:12:18.714 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:12:18.735 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:12:18.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:12:18.737 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:12:18.826 DEBUG:teuthology.orchestra.run.smithi183:> /bin/sh -c 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --no-log-to-stderr --name client.0 -b 65536 --object-size 65536 -p unique_pool_0 bench 90 write' 2024-09-10T19:12:18.827 INFO:tasks.thrashosds.thrasher:Shrinking pool unique_pool_0 2024-09-10T19:12:18.827 INFO:tasks.cephadm.ceph_manager.ceph:contract_pool unique_pool_0 by 10 min 8 2024-09-10T19:12:18.828 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph pg dump --format=json 2024-09-10T19:12:18.830 INFO:tasks.radosbench:joining radosbench (timing out after 3000s) 2024-09-10T19:12:18.831 INFO:teuthology.orchestra.run:waiting for 3000 2024-09-10T19:12:18.838 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:12:18.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:12:18.897 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:12:18.897 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:12:18.898 INFO:tasks.radosbench.radosbench.0.smithi183.stdout:hints = 1 2024-09-10T19:12:18.898 INFO:tasks.radosbench.radosbench.0.smithi183.stdout:Maintaining 16 concurrent writes of 65536 bytes to objects of size 65536 for up to 90 seconds or 0 objects 2024-09-10T19:12:18.898 INFO:tasks.radosbench.radosbench.0.smithi183.stdout:Object prefix: benchmark_data_smithi183_84160 2024-09-10T19:12:18.899 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-10T19:12:18.899 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: 0 0 0 0 0 0 - 0 2024-09-10T19:12:18.998 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:12:19.027 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:12:19.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:12:19.029 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:12:19.129 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:12:19.131 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:12:19.168 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:12:19.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:12:19.202 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:12:19.202 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:12:19.215 INFO:journalctl@ceph.iscsi.iscsi.a.smithi191.stdout:Sep 10 19:12:18 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-iscsi-iscsi-a[61816]: debug there is no tcmu-runner data available 2024-09-10T19:12:19.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:12:19 smithi191 ceph-mon[32884]: pgmap v67: 148 pgs: 148 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-10T19:12:19.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:12:19 smithi191 ceph-mon[32884]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-10T19:12:19.304 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:12:19.327 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:12:19 smithi183 ceph-mon[29736]: pgmap v67: 148 pgs: 148 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-10T19:12:19.327 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:12:19 smithi183 ceph-mon[29736]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-10T19:12:19.328 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:12:19 smithi183 ceph-mon[41562]: pgmap v67: 148 pgs: 148 active+clean; 582 KiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-10T19:12:19.328 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:12:19 smithi183 ceph-mon[41562]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-10T19:12:19.358 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:12:19.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:12:19.359 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:12:19.460 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:12:19.505 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:12:19.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:12:19.507 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:12:19.607 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:12:19.636 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:12:19.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:12:19.637 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:12:19.737 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:12:19.765 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:12:19.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:12:19.766 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:12:19.867 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:12:19.899 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: 1 16 3085 3069 191.801 191.812 0.00714097 0.00518279 2024-09-10T19:12:19.926 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:12:19.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:12:19.927 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:12:20.028 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:12:20.059 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:12:20.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:12:20.061 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:12:20.162 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:12:20.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:12:20.204 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:12:20.205 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:12:20.306 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:12:20.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:12:20.348 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:12:20.348 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:12:20.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:12:20 smithi183 ceph-mon[29736]: from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:12:20.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:12:20 smithi183 ceph-mon[29736]: from='mgr.14556 ' entity='mgr.y' 2024-09-10T19:12:20.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:12:20 smithi183 ceph-mon[29736]: from='mgr.14556 ' entity='mgr.y' 2024-09-10T19:12:20.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:12:20 smithi183 ceph-mon[29736]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T19:12:20.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:12:20 smithi183 ceph-mon[29736]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T19:12:20.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:12:20 smithi183 ceph-mon[29736]: from='mgr.14556 ' entity='mgr.y' 2024-09-10T19:12:20.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:12:20 smithi183 ceph-mon[29736]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T19:12:20.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:12:20 smithi183 ceph-mon[41562]: from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:12:20.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:12:20 smithi183 ceph-mon[41562]: from='mgr.14556 ' entity='mgr.y' 2024-09-10T19:12:20.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:12:20 smithi183 ceph-mon[41562]: from='mgr.14556 ' entity='mgr.y' 2024-09-10T19:12:20.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:12:20 smithi183 ceph-mon[41562]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T19:12:20.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:12:20 smithi183 ceph-mon[41562]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T19:12:20.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:12:20 smithi183 ceph-mon[41562]: from='mgr.14556 ' entity='mgr.y' 2024-09-10T19:12:20.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:12:20 smithi183 ceph-mon[41562]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T19:12:20.450 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:12:20.465 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:12:20 smithi191 ceph-mon[32884]: from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:12:20.465 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:12:20 smithi191 ceph-mon[32884]: from='mgr.14556 ' entity='mgr.y' 2024-09-10T19:12:20.465 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:12:20 smithi191 ceph-mon[32884]: from='mgr.14556 ' entity='mgr.y' 2024-09-10T19:12:20.465 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:12:20 smithi191 ceph-mon[32884]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T19:12:20.465 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:12:20 smithi191 ceph-mon[32884]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T19:12:20.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:12:20 smithi191 ceph-mon[32884]: from='mgr.14556 ' entity='mgr.y' 2024-09-10T19:12:20.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:12:20 smithi191 ceph-mon[32884]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T19:12:20.481 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:12:20.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:12:20.484 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:12:20.584 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:12:20.622 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:12:20.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:12:20.623 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:12:20.724 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:12:20.755 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:12:20.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:12:20.756 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:12:20.857 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:12:20.896 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:12:20.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:12:20.897 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:12:20.900 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: 2 16 6396 6380 199.36 206.938 0.00295868 0.00500693 2024-09-10T19:12:20.998 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:12:21.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:12:21.037 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:12:21.038 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:12:21.140 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:12:21.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:12:21.172 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:12:21.172 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:12:21.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:12:21 smithi183 ceph-mon[29736]: pgmap v68: 148 pgs: 148 active+clean; 133 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.3 KiB/s rd, 18 MiB/s wr, 281 op/s 2024-09-10T19:12:21.181 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:12:21 smithi183 ceph-mon[41562]: pgmap v68: 148 pgs: 148 active+clean; 133 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.3 KiB/s rd, 18 MiB/s wr, 281 op/s 2024-09-10T19:12:21.273 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:12:21.302 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:12:21.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:12:21.303 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:12:21.404 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:12:21.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:12:21.448 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:12:21.448 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:12:21.465 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:12:21 smithi191 ceph-mon[32884]: pgmap v68: 148 pgs: 148 active+clean; 133 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.3 KiB/s rd, 18 MiB/s wr, 281 op/s 2024-09-10T19:12:21.549 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:12:21.575 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Sep 10 19:12:21 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-y[29989]: ::ffff:172.21.15.191 - - [10/Sep/2024:19:12:21] "GET /metrics HTTP/1.1" 200 222491 "" "Prometheus/2.43.0" 2024-09-10T19:12:21.592 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:12:21.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:12:21.593 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:12:21.694 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:12:21.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:12:21.738 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:12:21.738 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:12:21.839 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:12:21.879 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:12:21.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:12:21.881 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:12:21.882 INFO:teuthology.orchestra.run.smithi183.stdout:osd.0: {} 2024-09-10T19:12:21.882 INFO:teuthology.orchestra.run.smithi183.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-10T19:12:21.899 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: 3 16 9539 9523 198.38 196.438 0.00605479 0.00503071 2024-09-10T19:12:21.915 INFO:teuthology.orchestra.run.smithi183.stdout:osd.1: {} 2024-09-10T19:12:21.916 INFO:teuthology.orchestra.run.smithi183.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-10T19:12:21.952 INFO:teuthology.orchestra.run.smithi183.stdout:osd.2: {} 2024-09-10T19:12:21.952 INFO:teuthology.orchestra.run.smithi183.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-10T19:12:21.982 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:12:21.989 INFO:teuthology.orchestra.run.smithi183.stdout:osd.3: {} 2024-09-10T19:12:21.989 INFO:teuthology.orchestra.run.smithi183.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-10T19:12:22.018 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:12:22.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:12:22.020 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:12:22.020 INFO:teuthology.orchestra.run.smithi183.stdout:osd.4: {} 2024-09-10T19:12:22.020 INFO:teuthology.orchestra.run.smithi183.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-10T19:12:22.052 INFO:teuthology.orchestra.run.smithi183.stdout:osd.5: {} 2024-09-10T19:12:22.052 INFO:teuthology.orchestra.run.smithi183.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-10T19:12:22.085 INFO:teuthology.orchestra.run.smithi183.stdout:osd.6: {} 2024-09-10T19:12:22.086 INFO:teuthology.orchestra.run.smithi183.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-10T19:12:22.120 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:12:22.121 INFO:teuthology.orchestra.run.smithi183.stdout:osd.7: {} 2024-09-10T19:12:22.121 INFO:teuthology.orchestra.run.smithi183.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-10T19:12:22.149 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:12:22.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:12:22.150 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:12:22.251 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:12:22.276 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:12:22.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:12:22.277 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:12:22.378 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:12:22.408 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:12:22.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:12:22.409 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:12:22.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:12:22 smithi183 ceph-mon[29736]: from='client.? 172.21.15.183:0/3700954635' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T19:12:22.431 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:12:22 smithi183 ceph-mon[41562]: from='client.? 172.21.15.183:0/3700954635' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T19:12:22.465 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:12:22 smithi191 ceph-mon[32884]: from='client.? 172.21.15.183:0/3700954635' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T19:12:22.511 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:12:22.542 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:12:22.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:12:22.545 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:12:22.646 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:12:22.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:12:22.667 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:12:22.668 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:12:22.768 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:12:22.816 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:12:22.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:12:22.817 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:12:22.899 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: 4 16 12275 12259 191.509 171 0.00744161 0.00521256 2024-09-10T19:12:22.918 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:12:22.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:12:22.940 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:12:22.940 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:12:23.041 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:12:23.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:12:23.073 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:12:23.073 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:12:23.174 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:12:23.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:12:23.216 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:12:23.216 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:12:23.317 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:12:23.317 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:12:23.325 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:12:23.377 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:12:23.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:12:23.378 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:12:23.421 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:12:23 smithi183 ceph-mon[29736]: pgmap v69: 148 pgs: 148 active+clean; 133 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 15 MiB/s wr, 248 op/s 2024-09-10T19:12:23.422 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:12:23 smithi183 ceph-mon[41562]: pgmap v69: 148 pgs: 148 active+clean; 133 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 15 MiB/s wr, 248 op/s 2024-09-10T19:12:23.445 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:12:23 smithi191 ceph-mon[32884]: pgmap v69: 148 pgs: 148 active+clean; 133 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1.2 KiB/s rd, 15 MiB/s wr, 248 op/s 2024-09-10T19:12:23.479 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:12:23.519 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:12:23.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:12:23.520 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:12:23.621 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:12:23.652 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:12:23.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:12:23.653 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:12:23.754 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:12:23.792 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:12:23.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:12:23.793 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:12:23.894 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:12:23.900 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: 5 16 15070 15054 188.142 174.688 0.0084702 0.0053079 2024-09-10T19:12:23.933 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:12:23.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:12:23.934 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:12:24.035 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:12:24.072 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:12:24.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:12:24.073 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:12:24.174 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:12:24.202 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:12:24.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:12:24.203 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:12:24.304 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:12:24.343 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:12:24.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:12:24.344 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:12:24.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:12:24 smithi183 ceph-mon[29736]: pgmap v70: 148 pgs: 148 active+clean; 133 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 13 MiB/s wr, 213 op/s 2024-09-10T19:12:24.431 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:12:24 smithi183 ceph-mon[41562]: pgmap v70: 148 pgs: 148 active+clean; 133 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 13 MiB/s wr, 213 op/s 2024-09-10T19:12:24.445 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:12:24.465 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:12:24 smithi191 ceph-mon[32884]: pgmap v70: 148 pgs: 148 active+clean; 133 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 13 MiB/s wr, 213 op/s 2024-09-10T19:12:24.468 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:12:24.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:12:24.469 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:12:24.570 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:12:24.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:12:24.609 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:12:24.610 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:12:24.710 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:12:24.732 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:12:24.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:12:24.733 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:12:24.833 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:12:24.872 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:12:24.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:12:24.873 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:12:24.899 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: 6 16 17385 17369 180.898 144.688 0.0068397 0.0055223 2024-09-10T19:12:24.973 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:12:25.004 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:12:25.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:12:25.005 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:12:25.106 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:12:25.134 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:12:25.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:12:25.135 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:12:25.198 INFO:teuthology.orchestra.run.smithi183.stderr:nodeep-scrub is set 2024-09-10T19:12:25.236 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:12:25.254 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:12:25 smithi183 ceph-mon[41562]: from='client.? 172.21.15.183:0/2867383587' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-10T19:12:25.255 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:12:25 smithi183 ceph-mon[29736]: from='client.? 172.21.15.183:0/2867383587' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-10T19:12:25.280 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:12:25.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:12:25.281 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:12:25.382 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:12:25.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:12:25.422 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:12:25.422 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:12:25.425 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:12:25 smithi191 ceph-mon[32884]: from='client.? 172.21.15.183:0/2867383587' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-10T19:12:25.523 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:12:25.560 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:12:25.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:12:25.561 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:12:25.662 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:12:25.701 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:12:25.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:12:25.702 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:12:25.802 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:12:25.840 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:12:25.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:12:25.841 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:12:25.900 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: 7 16 19960 19944 178.045 160.938 0.00328984 0.00560988 2024-09-10T19:12:25.907 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-10T19:12:25.942 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:12:25.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:12:26.000 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:12:26.000 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:12:26.101 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:12:26.151 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:12:26.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:12:26.153 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:12:26.251 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:12:26 smithi183 ceph-mon[41562]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T19:12:26.251 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:12:26 smithi183 ceph-mon[41562]: from='client.? 172.21.15.183:0/2867383587' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-10T19:12:26.251 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:12:26 smithi183 ceph-mon[41562]: osdmap e91: 8 total, 8 up, 8 in 2024-09-10T19:12:26.251 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:12:26 smithi183 ceph-mon[41562]: pgmap v72: 148 pgs: 148 active+clean; 1.0 GiB data, 5.2 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 123 MiB/s wr, 1.97k op/s 2024-09-10T19:12:26.251 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:12:26 smithi183 ceph-mon[41562]: from='mgr.14556 ' entity='mgr.y' 2024-09-10T19:12:26.251 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:12:26 smithi183 ceph-mon[41562]: from='mgr.14556 ' entity='mgr.y' 2024-09-10T19:12:26.252 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:12:26 smithi183 ceph-mon[29736]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T19:12:26.252 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:12:26 smithi183 ceph-mon[29736]: from='client.? 172.21.15.183:0/2867383587' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-10T19:12:26.252 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:12:26 smithi183 ceph-mon[29736]: osdmap e91: 8 total, 8 up, 8 in 2024-09-10T19:12:26.252 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:12:26 smithi183 ceph-mon[29736]: pgmap v72: 148 pgs: 148 active+clean; 1.0 GiB data, 5.2 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 123 MiB/s wr, 1.97k op/s 2024-09-10T19:12:26.252 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:12:26 smithi183 ceph-mon[29736]: from='mgr.14556 ' entity='mgr.y' 2024-09-10T19:12:26.252 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:12:26 smithi183 ceph-mon[29736]: from='mgr.14556 ' entity='mgr.y' 2024-09-10T19:12:26.253 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:12:26.281 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:12:26.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:12:26.282 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:12:26.383 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:12:26.411 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:12:26.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:12:26.412 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:12:26.465 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:12:26 smithi191 ceph-mon[32884]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T19:12:26.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:12:26 smithi191 ceph-mon[32884]: from='client.? 172.21.15.183:0/2867383587' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-10T19:12:26.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:12:26 smithi191 ceph-mon[32884]: osdmap e91: 8 total, 8 up, 8 in 2024-09-10T19:12:26.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:12:26 smithi191 ceph-mon[32884]: pgmap v72: 148 pgs: 148 active+clean; 1.0 GiB data, 5.2 GiB used, 710 GiB / 715 GiB avail; 1.2 KiB/s rd, 123 MiB/s wr, 1.97k op/s 2024-09-10T19:12:26.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:12:26 smithi191 ceph-mon[32884]: from='mgr.14556 ' entity='mgr.y' 2024-09-10T19:12:26.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:12:26 smithi191 ceph-mon[32884]: from='mgr.14556 ' entity='mgr.y' 2024-09-10T19:12:26.512 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:12:26.560 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:12:26.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:12:26.561 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:12:26.662 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:12:26.690 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:12:26.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:12:26.691 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:12:26.792 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:12:26.836 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:12:26.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:12:26.837 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:12:26.900 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: 8 15 22359 22344 174.537 150 0.00669996 0.00572257 2024-09-10T19:12:26.938 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:12:26.968 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:12:26.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:12:26.969 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:12:27.070 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:12:27.099 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:12:27.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:12:27.101 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:12:27.201 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:12:27.230 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:12:27.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:12:27.231 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:12:27.332 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:12:27.361 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:12:27.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:12:27.362 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:12:27.464 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:12:27.485 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:12:27.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:12:27.486 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:12:27.587 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:12:27.625 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:12:27.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:12:27.626 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:12:27.715 INFO:journalctl@ceph.mgr.x.smithi191.stdout:Sep 10 19:12:27 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-x[34967]: ::ffff:172.21.15.191 - - [10/Sep/2024:19:12:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T19:12:27.727 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:12:27.765 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:12:27.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:12:27.766 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:12:27.866 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:12:27.899 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:12:27.900 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: 9 16 24414 24398 169.407 128.375 0.00457102 0.00589765 2024-09-10T19:12:27.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:12:27.901 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:12:28.002 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:12:28.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:12:28.029 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:12:28.030 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:12:28.130 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:12:28.167 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:12:28.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:12:28.168 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:12:28.269 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:12:28.293 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:12:28.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:12:28.294 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:12:28.395 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:12:28.436 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:12:28.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:12:28.437 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:12:28.538 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:12:28.559 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:12:28.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:12:28.560 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:12:28.661 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:12:28.692 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:12:28.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:12:28.693 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:12:28.794 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:12:28.831 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:12:28.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:12:28.832 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:12:28.900 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: 10 16 26414 26398 164.965 125 0.00584294 0.0060562 2024-09-10T19:12:28.931 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:12:28 smithi183 ceph-mon[41562]: pgmap v73: 148 pgs: 148 active+clean; 1.0 GiB data, 5.2 GiB used, 710 GiB / 715 GiB avail; 1023 B/s rd, 105 MiB/s wr, 1.68k op/s 2024-09-10T19:12:28.931 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:12:28 smithi183 ceph-mon[41562]: from='mgr.14556 ' entity='mgr.y' 2024-09-10T19:12:28.931 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:12:28 smithi183 ceph-mon[41562]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:12:28.931 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:12:28 smithi183 ceph-mon[29736]: pgmap v73: 148 pgs: 148 active+clean; 1.0 GiB data, 5.2 GiB used, 710 GiB / 715 GiB avail; 1023 B/s rd, 105 MiB/s wr, 1.68k op/s 2024-09-10T19:12:28.931 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:12:28 smithi183 ceph-mon[29736]: from='mgr.14556 ' entity='mgr.y' 2024-09-10T19:12:28.932 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:12:28 smithi183 ceph-mon[29736]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:12:28.933 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:12:28.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:12:28.955 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:12:28.955 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:12:28.965 INFO:journalctl@ceph.iscsi.iscsi.a.smithi191.stdout:Sep 10 19:12:28 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-iscsi-iscsi-a[61816]: debug there is no tcmu-runner data available 2024-09-10T19:12:28.965 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:12:28 smithi191 ceph-mon[32884]: pgmap v73: 148 pgs: 148 active+clean; 1.0 GiB data, 5.2 GiB used, 710 GiB / 715 GiB avail; 1023 B/s rd, 105 MiB/s wr, 1.68k op/s 2024-09-10T19:12:28.966 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:12:28 smithi191 ceph-mon[32884]: from='mgr.14556 ' entity='mgr.y' 2024-09-10T19:12:28.966 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:12:28 smithi191 ceph-mon[32884]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:12:29.056 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:12:29.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:12:29.094 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:12:29.095 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:12:29.195 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:12:29.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:12:29.228 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:12:29.228 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:12:29.329 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:12:29.367 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:12:29.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:12:29.368 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:12:29.469 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:12:29.499 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:12:29.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:12:29.500 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:12:29.601 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:12:29.639 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:12:29.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:12:29.640 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:12:29.741 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:12:29.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:12:29.780 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:12:29.780 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:12:29.881 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:12:29.900 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: 11 16 28459 28443 161.587 127.812 0.0056355 0.00618267 2024-09-10T19:12:29.911 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:12:29.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:12:29.912 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:12:29.933 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:12:29 smithi183 ceph-mon[29736]: from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:12:29.934 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:12:29 smithi183 ceph-mon[41562]: from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:12:29.965 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:12:29 smithi191 ceph-mon[32884]: from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:12:30.014 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:12:30.052 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:12:30.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:12:30.053 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:12:30.154 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:12:30.192 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:12:30.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:12:30.193 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:12:30.294 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:12:30.336 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:12:30.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:12:30.337 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:12:30.438 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:12:30.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:12:30.464 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:12:30.464 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:12:30.565 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:12:30.603 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:12:30.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:12:30.605 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:12:30.705 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:12:30.733 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:12:30.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:12:30.734 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:12:30.836 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:12:30.873 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:12:30.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:12:30.875 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:12:30.901 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: 12 16 30343 30327 157.926 117.75 0.00936889 0.00632565 2024-09-10T19:12:30.943 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:12:30 smithi183 ceph-mon[29736]: pgmap v74: 148 pgs: 148 active+clean; 1.7 GiB data, 7.2 GiB used, 708 GiB / 715 GiB avail; 1023 B/s rd, 161 MiB/s wr, 2.57k op/s 2024-09-10T19:12:30.944 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:12:30 smithi183 ceph-mon[41562]: pgmap v74: 148 pgs: 148 active+clean; 1.7 GiB data, 7.2 GiB used, 708 GiB / 715 GiB avail; 1023 B/s rd, 161 MiB/s wr, 2.57k op/s 2024-09-10T19:12:30.965 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:12:30 smithi191 ceph-mon[32884]: pgmap v74: 148 pgs: 148 active+clean; 1.7 GiB data, 7.2 GiB used, 708 GiB / 715 GiB avail; 1023 B/s rd, 161 MiB/s wr, 2.57k op/s 2024-09-10T19:12:30.975 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:12:31.003 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:12:31.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:12:31.004 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:12:31.105 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:12:31.142 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:12:31.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:12:31.143 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:12:31.243 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:12:31.273 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:12:31.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:12:31.275 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:12:31.376 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:12:31.412 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:12:31.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:12:31.413 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:12:31.514 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:12:31.547 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:12:31.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:12:31.548 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:12:31.649 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:12:31.681 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Sep 10 19:12:31 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-y[29989]: ::ffff:172.21.15.191 - - [10/Sep/2024:19:12:31] "GET /metrics HTTP/1.1" 200 223218 "" "Prometheus/2.43.0" 2024-09-10T19:12:31.684 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:12:31.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:12:31.685 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:12:31.786 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:12:31.823 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:12:31.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:12:31.825 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:12:31.858 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph osd unset noscrub 2024-09-10T19:12:31.901 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: 13 16 32316 32300 155.263 123.312 0.01103 0.0064349 2024-09-10T19:12:31.926 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:12:31.965 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:12:31.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:12:31.966 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:12:32.067 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:12:32.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:12:32.108 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:12:32.108 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:12:32.209 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:12:32.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:12:32.239 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:12:32.240 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:12:32.341 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:12:32.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:12:32.420 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:12:32.420 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:12:32.521 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:12:32.561 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:12:32.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:12:32.562 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:12:32.662 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:12:32.691 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:12:32.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:12:32.692 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:12:32.793 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:12:32.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:12:32.832 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:12:32.832 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:12:32.901 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: 14 16 34305 34289 153.051 124.312 0.00814406 0.00652859 2024-09-10T19:12:32.934 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:12:32.959 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:12:32.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:12:32.960 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:12:32.965 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:12:32 smithi191 ceph-mon[32884]: purged_snaps scrub starts 2024-09-10T19:12:32.965 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:12:32 smithi191 ceph-mon[32884]: purged_snaps scrub ok 2024-09-10T19:12:32.965 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:12:32 smithi191 ceph-mon[32884]: pgmap v75: 148 pgs: 148 active+clean; 1.7 GiB data, 7.2 GiB used, 708 GiB / 715 GiB avail; 1023 B/s rd, 160 MiB/s wr, 2.57k op/s 2024-09-10T19:12:32.965 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:12:32 smithi191 ceph-mon[32884]: osdmap e92: 8 total, 8 up, 8 in 2024-09-10T19:12:32.966 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:12:32 smithi191 ceph-mon[32884]: purged_snaps scrub starts 2024-09-10T19:12:32.966 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:12:32 smithi191 ceph-mon[32884]: purged_snaps scrub ok 2024-09-10T19:12:33.061 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:12:33.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:12:33.100 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:12:33.100 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:12:33.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:12:32 smithi183 ceph-mon[29736]: purged_snaps scrub starts 2024-09-10T19:12:33.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:12:32 smithi183 ceph-mon[29736]: purged_snaps scrub ok 2024-09-10T19:12:33.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:12:32 smithi183 ceph-mon[29736]: pgmap v75: 148 pgs: 148 active+clean; 1.7 GiB data, 7.2 GiB used, 708 GiB / 715 GiB avail; 1023 B/s rd, 160 MiB/s wr, 2.57k op/s 2024-09-10T19:12:33.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:12:32 smithi183 ceph-mon[29736]: osdmap e92: 8 total, 8 up, 8 in 2024-09-10T19:12:33.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:12:32 smithi183 ceph-mon[29736]: purged_snaps scrub starts 2024-09-10T19:12:33.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:12:32 smithi183 ceph-mon[29736]: purged_snaps scrub ok 2024-09-10T19:12:33.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:12:32 smithi183 ceph-mon[41562]: purged_snaps scrub starts 2024-09-10T19:12:33.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:12:32 smithi183 ceph-mon[41562]: purged_snaps scrub ok 2024-09-10T19:12:33.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:12:32 smithi183 ceph-mon[41562]: pgmap v75: 148 pgs: 148 active+clean; 1.7 GiB data, 7.2 GiB used, 708 GiB / 715 GiB avail; 1023 B/s rd, 160 MiB/s wr, 2.57k op/s 2024-09-10T19:12:33.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:12:32 smithi183 ceph-mon[41562]: osdmap e92: 8 total, 8 up, 8 in 2024-09-10T19:12:33.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:12:32 smithi183 ceph-mon[41562]: purged_snaps scrub starts 2024-09-10T19:12:33.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:12:32 smithi183 ceph-mon[41562]: purged_snaps scrub ok 2024-09-10T19:12:33.201 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:12:33.228 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:12:33.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:12:33.229 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:12:33.330 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:12:33.360 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:12:33.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:12:33.361 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:12:33.406 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-09-10T19:12:33.409 INFO:teuthology.orchestra.run.smithi183.stderr:dumped all 2024-09-10T19:12:33.461 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:12:33.507 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:12:33.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:12:33.508 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:12:33.609 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:12:33.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:12:33.650 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:12:33.650 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:12:33.751 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:12:33.787 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:12:33.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:12:33.788 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:12:33.889 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:12:33.902 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: 15 16 36146 36130 150.509 115.062 0.00469046 0.00663797 2024-09-10T19:12:33.927 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:12:33.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:12:33.928 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:12:33.955 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:12:33 smithi183 ceph-mon[29736]: osdmap e93: 8 total, 8 up, 8 in 2024-09-10T19:12:33.956 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:12:33 smithi183 ceph-mon[41562]: osdmap e93: 8 total, 8 up, 8 in 2024-09-10T19:12:34.029 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:12:34.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:12:34.081 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:12:34.081 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:12:34.182 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:12:34.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:12:34.211 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:12:34.212 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:12:34.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:12:33 smithi191 ceph-mon[32884]: osdmap e93: 8 total, 8 up, 8 in 2024-09-10T19:12:34.313 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:12:34.354 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:12:34.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:12:34.355 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:12:34.456 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:12:34.498 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:12:34.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:12:34.499 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:12:34.600 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:12:34.642 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:12:34.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:12:34.644 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:12:34.744 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:12:34.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:12:34.772 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:12:34.772 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:12:34.873 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:12:34.895 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:12:34 smithi183 ceph-mon[29736]: from='client.24655 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-10T19:12:34.896 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:12:34 smithi183 ceph-mon[29736]: pgmap v78: 148 pgs: 148 active+clean; 1.7 GiB data, 7.2 GiB used, 708 GiB / 715 GiB avail; 639 B/s rd, 86 MiB/s wr, 1.37k op/s 2024-09-10T19:12:34.896 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:12:34 smithi183 ceph-mon[41562]: from='client.24655 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-10T19:12:34.896 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:12:34 smithi183 ceph-mon[41562]: pgmap v78: 148 pgs: 148 active+clean; 1.7 GiB data, 7.2 GiB used, 708 GiB / 715 GiB avail; 639 B/s rd, 86 MiB/s wr, 1.37k op/s 2024-09-10T19:12:34.900 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:12:34.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:12:34.901 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:12:34.902 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: 16 16 37982 37966 148.274 114.75 0.00919269 0.00673746 2024-09-10T19:12:35.001 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:12:35.031 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:12:35.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:12:35.032 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:12:35.133 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:12:35.176 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:12:35.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:12:35.177 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:12:35.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:12:34 smithi191 ceph-mon[32884]: from='client.24655 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-10T19:12:35.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:12:34 smithi191 ceph-mon[32884]: pgmap v78: 148 pgs: 148 active+clean; 1.7 GiB data, 7.2 GiB used, 708 GiB / 715 GiB avail; 639 B/s rd, 86 MiB/s wr, 1.37k op/s 2024-09-10T19:12:35.278 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:12:35.306 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:12:35.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:12:35.308 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:12:35.408 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:12:35.434 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:12:35.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:12:35.435 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:12:35.453 INFO:teuthology.orchestra.run.smithi183.stdout:{"pg_ready":true,"pg_map":{"version":77,"stamp":"2024-09-10T19:12:32.697701+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1823414492,"num_objects":28010,"num_object_clones":0,"num_object_copies":84030,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28010,"num_whiteouts":0,"num_read":987,"num_read_kb":844,"num_write":28357,"num_write_kb":1781767,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":590588,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":28409,"ondisk_log_size":28409,"up":444,"acting":444,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":444,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":7554776,"kb_used_data":5142616,"kb_used_omap":13,"kb_used_meta":2412018,"kb_avail":742406440,"statfs":{"total":767960285184,"available":760224194560,"internally_reserved":0,"allocated":5266038784,"data_stored":5262910048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13632,"internal_metadata":2469907136},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[6,6,5,0,0,0,0,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":719126528,"num_objects":10973,"num_object_clones":0,"num_object_copies":32919,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10973,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":10973,"num_write_kb":702272,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.507872"},"pg_stats":[{"pgid":"6.1b","version":"0'0","reported_seq":45,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-10T19:12:17.048668+0000","last_change":"2024-09-10T19:12:05.630214+0000","last_active":"2024-09-10T19:12:17.048668+0000","last_peered":"2024-09-10T19:12:17.048668+0000","last_clean":"2024-09-10T19:12:17.048668+0000","last_became_active":"2024-09-10T19:10:14.056187+0000","last_became_peered":"2024-09-10T19:10:14.056187+0000","last_unstale":"2024-09-10T19:12:17.048668+0000","last_undegraded":"2024-09-10T19:12:17.048668+0000","last_fullsized":"2024-09-10T19:12:17.048668+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:12:05.630153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:12:05.630153+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:13:06.643368+0000","scrub_duration":0.001661771,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":61,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-10T19:12:18.123208+0000","last_change":"2024-09-10T19:11:46.882500+0000","last_active":"2024-09-10T19:12:18.123208+0000","last_peered":"2024-09-10T19:12:18.123208+0000","last_clean":"2024-09-10T19:12:18.123208+0000","last_became_active":"2024-09-10T19:11:44.748696+0000","last_became_peered":"2024-09-10T19:11:44.748696+0000","last_unstale":"2024-09-10T19:12:18.123208+0000","last_undegraded":"2024-09-10T19:12:18.123208+0000","last_fullsized":"2024-09-10T19:12:18.123208+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:11:46.882412+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:11:46.882412+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:13:12.297001+0000","scrub_duration":0.0015255100000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.19","version":"66'15","reported_seq":74,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-10T19:12:17.048666+0000","last_change":"2024-09-10T19:12:10.638999+0000","last_active":"2024-09-10T19:12:17.048666+0000","last_peered":"2024-09-10T19:12:17.048666+0000","last_clean":"2024-09-10T19:12:17.048666+0000","last_became_active":"2024-09-10T19:10:09.688432+0000","last_became_peered":"2024-09-10T19:10:09.688432+0000","last_unstale":"2024-09-10T19:12:17.048666+0000","last_undegraded":"2024-09-10T19:12:17.048666+0000","last_fullsized":"2024-09-10T19:12:17.048666+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'15","last_scrub_stamp":"2024-09-10T19:12:10.638939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:12:10.638939+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:13:25.707571+0000","scrub_duration":0.0020969259999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":46,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-10T19:12:18.294169+0000","last_change":"2024-09-10T19:11:23.086288+0000","last_active":"2024-09-10T19:12:18.294169+0000","last_peered":"2024-09-10T19:12:18.294169+0000","last_clean":"2024-09-10T19:12:18.294169+0000","last_became_active":"2024-09-10T19:10:11.697523+0000","last_became_peered":"2024-09-10T19:10:11.697523+0000","last_unstale":"2024-09-10T19:12:18.294169+0000","last_undegraded":"2024-09-10T19:12:18.294169+0000","last_fullsized":"2024-09-10T19:12:18.294169+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:11:23.086191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:11:23.086191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:12:31.131189+0000","scrub_duration":0.0016087969999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"59'1","reported_seq":59,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-10T19:12:17.049152+0000","last_change":"2024-09-10T19:11:15.782493+0000","last_active":"2024-09-10T19:12:17.049152+0000","last_peered":"2024-09-10T19:12:17.049152+0000","last_clean":"2024-09-10T19:12:17.049152+0000","last_became_active":"2024-09-10T19:10:07.686143+0000","last_became_peered":"2024-09-10T19:10:07.686143+0000","last_unstale":"2024-09-10T19:12:17.049152+0000","last_undegraded":"2024-09-10T19:12:17.049152+0000","last_fullsized":"2024-09-10T19:12:17.049152+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'1","last_scrub_stamp":"2024-09-10T19:11:15.782393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:11:15.782393+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.00183947,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":874,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"66'9","reported_seq":64,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-10T19:12:18.294310+0000","last_change":"2024-09-10T19:12:04.079318+0000","last_active":"2024-09-10T19:12:18.294310+0000","last_peered":"2024-09-10T19:12:18.294310+0000","last_clean":"2024-09-10T19:12:18.294310+0000","last_became_active":"2024-09-10T19:10:09.692681+0000","last_became_peered":"2024-09-10T19:10:09.692681+0000","last_unstale":"2024-09-10T19:12:18.294310+0000","last_undegraded":"2024-09-10T19:12:18.294310+0000","last_fullsized":"2024-09-10T19:12:18.294310+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'9","last_scrub_stamp":"2024-09-10T19:12:04.079264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:12:04.079264+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:13:16.653517+0000","scrub_duration":0.0015358819999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":46,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-10T19:12:18.203439+0000","last_change":"2024-09-10T19:11:56.186416+0000","last_active":"2024-09-10T19:12:18.203439+0000","last_peered":"2024-09-10T19:12:18.203439+0000","last_clean":"2024-09-10T19:12:18.203439+0000","last_became_active":"2024-09-10T19:10:11.702546+0000","last_became_peered":"2024-09-10T19:10:11.702546+0000","last_unstale":"2024-09-10T19:12:18.203439+0000","last_undegraded":"2024-09-10T19:12:18.203439+0000","last_fullsized":"2024-09-10T19:12:18.203439+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:11:56.186365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:11:56.186365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:13:05.311138+0000","scrub_duration":0.001284161,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":44,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-10T19:12:18.294185+0000","last_change":"2024-09-10T19:11:32.056320+0000","last_active":"2024-09-10T19:12:18.294185+0000","last_peered":"2024-09-10T19:12:18.294185+0000","last_clean":"2024-09-10T19:12:18.294185+0000","last_became_active":"2024-09-10T19:10:13.716022+0000","last_became_peered":"2024-09-10T19:10:13.716022+0000","last_unstale":"2024-09-10T19:12:18.294185+0000","last_undegraded":"2024-09-10T19:12:18.294185+0000","last_fullsized":"2024-09-10T19:12:18.294185+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:11:32.056254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:11:32.056254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:12:39.600944+0000","scrub_duration":0.0013010680000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":51,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-10T19:12:18.444819+0000","last_change":"2024-09-10T19:11:19.275407+0000","last_active":"2024-09-10T19:12:18.444819+0000","last_peered":"2024-09-10T19:12:18.444819+0000","last_clean":"2024-09-10T19:12:18.444819+0000","last_became_active":"2024-09-10T19:10:07.690728+0000","last_became_peered":"2024-09-10T19:10:07.690728+0000","last_unstale":"2024-09-10T19:12:18.444819+0000","last_undegraded":"2024-09-10T19:12:18.444819+0000","last_fullsized":"2024-09-10T19:12:18.444819+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:11:19.275331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:11:19.275331+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:12:30.648634+0000","scrub_duration":0.001355258,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1b","version":"66'5","reported_seq":74,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-10T19:12:18.294256+0000","last_change":"2024-09-10T19:11:44.752160+0000","last_active":"2024-09-10T19:12:18.294256+0000","last_peered":"2024-09-10T19:12:18.294256+0000","last_clean":"2024-09-10T19:12:18.294256+0000","last_became_active":"2024-09-10T19:11:44.748296+0000","last_became_peered":"2024-09-10T19:11:44.748296+0000","last_unstale":"2024-09-10T19:12:18.294256+0000","last_undegraded":"2024-09-10T19:12:18.294256+0000","last_fullsized":"2024-09-10T19:12:18.294256+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'5","last_scrub_stamp":"2024-09-10T19:11:24.101551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:11:24.101551+0000","objects_scrubbed":2,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:12:33.998803+0000","scrub_duration":0.0019539829999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-10T19:12:18.444307+0000","last_change":"2024-09-10T19:11:43.142813+0000","last_active":"2024-09-10T19:12:18.444307+0000","last_peered":"2024-09-10T19:12:18.444307+0000","last_clean":"2024-09-10T19:12:18.444307+0000","last_became_active":"2024-09-10T19:10:11.697088+0000","last_became_peered":"2024-09-10T19:10:11.697088+0000","last_unstale":"2024-09-10T19:12:18.444307+0000","last_undegraded":"2024-09-10T19:12:18.444307+0000","last_fullsized":"2024-09-10T19:12:18.444307+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:11:43.142734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:11:43.142734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:12:50.945873+0000","scrub_duration":0.0013612330000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":45,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-10T19:12:17.046746+0000","last_change":"2024-09-10T19:11:45.411391+0000","last_active":"2024-09-10T19:12:17.046746+0000","last_peered":"2024-09-10T19:12:17.046746+0000","last_clean":"2024-09-10T19:12:17.046746+0000","last_became_active":"2024-09-10T19:10:13.724255+0000","last_became_peered":"2024-09-10T19:10:13.724255+0000","last_unstale":"2024-09-10T19:12:17.046746+0000","last_undegraded":"2024-09-10T19:12:17.046746+0000","last_fullsized":"2024-09-10T19:12:17.046746+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:11:45.411321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:11:45.411321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:12:49.122352+0000","scrub_duration":0.001536576,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":51,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-10T19:12:18.444674+0000","last_change":"2024-09-10T19:11:33.268356+0000","last_active":"2024-09-10T19:12:18.444674+0000","last_peered":"2024-09-10T19:12:18.444674+0000","last_clean":"2024-09-10T19:12:18.444674+0000","last_became_active":"2024-09-10T19:10:07.690174+0000","last_became_peered":"2024-09-10T19:10:07.690174+0000","last_unstale":"2024-09-10T19:12:18.444674+0000","last_undegraded":"2024-09-10T19:12:18.444674+0000","last_fullsized":"2024-09-10T19:12:18.444674+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:11:33.268282+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:11:33.268282+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:12:35.423155+0000","scrub_duration":0.0015798069999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1a","version":"66'9","reported_seq":64,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-10T19:12:18.293862+0000","last_change":"2024-09-10T19:11:31.035964+0000","last_active":"2024-09-10T19:12:18.293862+0000","last_peered":"2024-09-10T19:12:18.293862+0000","last_clean":"2024-09-10T19:12:18.293862+0000","last_became_active":"2024-09-10T19:10:09.688987+0000","last_became_peered":"2024-09-10T19:10:09.688987+0000","last_unstale":"2024-09-10T19:12:18.293862+0000","last_undegraded":"2024-09-10T19:12:18.293862+0000","last_fullsized":"2024-09-10T19:12:18.293862+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'9","last_scrub_stamp":"2024-09-10T19:11:31.035912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:11:31.035912+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:12:57.913355+0000","scrub_duration":0.001674916,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-10T19:12:17.047029+0000","last_change":"2024-09-10T19:11:27.317868+0000","last_active":"2024-09-10T19:12:17.047029+0000","last_peered":"2024-09-10T19:12:17.047029+0000","last_clean":"2024-09-10T19:12:17.047029+0000","last_became_active":"2024-09-10T19:10:11.701799+0000","last_became_peered":"2024-09-10T19:10:11.701799+0000","last_unstale":"2024-09-10T19:12:17.047029+0000","last_undegraded":"2024-09-10T19:12:17.047029+0000","last_fullsized":"2024-09-10T19:12:17.047029+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:11:27.317779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:11:27.317779+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:12:44.343114+0000","scrub_duration":0.001509584,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":45,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-10T19:12:18.122506+0000","last_change":"2024-09-10T19:11:12.970227+0000","last_active":"2024-09-10T19:12:18.122506+0000","last_peered":"2024-09-10T19:12:18.122506+0000","last_clean":"2024-09-10T19:12:18.122506+0000","last_became_active":"2024-09-10T19:10:13.717310+0000","last_became_peered":"2024-09-10T19:10:13.717310+0000","last_unstale":"2024-09-10T19:12:18.122506+0000","last_undegraded":"2024-09-10T19:12:18.122506+0000","last_fullsized":"2024-09-10T19:12:18.122506+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:11:12.970173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:11:12.970173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:12:27.060515+0000","scrub_duration":0.001527219,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.e","version":"91'1764","reported_seq":1777,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-10T19:12:29.592910+0000","last_change":"2024-09-10T19:12:11.033809+0000","last_active":"2024-09-10T19:12:29.592910+0000","last_peered":"2024-09-10T19:12:29.592910+0000","last_clean":"2024-09-10T19:12:29.592910+0000","last_became_active":"2024-09-10T19:12:11.032901+0000","last_became_peered":"2024-09-10T19:12:11.032901+0000","last_unstale":"2024-09-10T19:12:29.592910+0000","last_undegraded":"2024-09-10T19:12:29.592910+0000","last_fullsized":"2024-09-10T19:12:29.592910+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:12:10.007465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:12:10.007465+0000","last_clean_scrub_stamp":"2024-09-10T19:12:10.007465+0000","objects_scrubbed":0,"log_size":1764,"log_dups_size":0,"ondisk_log_size":1764,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:13:24.088652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":115605504,"num_objects":1764,"num_object_clones":0,"num_object_copies":5292,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1764,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1764,"num_write_kb":112896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.d","version":"66'17","reported_seq":76,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-10T19:12:18.293924+0000","last_change":"2024-09-10T19:11:14.007973+0000","last_active":"2024-09-10T19:12:18.293924+0000","last_peered":"2024-09-10T19:12:18.293924+0000","last_clean":"2024-09-10T19:12:18.293924+0000","last_became_active":"2024-09-10T19:10:09.694016+0000","last_became_peered":"2024-09-10T19:10:09.694016+0000","last_unstale":"2024-09-10T19:12:18.293924+0000","last_undegraded":"2024-09-10T19:12:18.293924+0000","last_fullsized":"2024-09-10T19:12:18.293924+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'17","last_scrub_stamp":"2024-09-10T19:11:14.007902+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:11:14.007902+0000","objects_scrubbed":8,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0024520879999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":51,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-10T19:12:18.444585+0000","last_change":"2024-09-10T19:11:22.280067+0000","last_active":"2024-09-10T19:12:18.444585+0000","last_peered":"2024-09-10T19:12:18.444585+0000","last_clean":"2024-09-10T19:12:18.444585+0000","last_became_active":"2024-09-10T19:10:07.691119+0000","last_became_peered":"2024-09-10T19:10:07.691119+0000","last_unstale":"2024-09-10T19:12:18.444585+0000","last_undegraded":"2024-09-10T19:12:18.444585+0000","last_fullsized":"2024-09-10T19:12:18.444585+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:11:22.280002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:11:22.280002+0000","last_clean_scrub_stamp":"2024-09-10T19:11:22.280002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:12:27.848752+0000","scrub_duration":0.0013828040000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":46,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-10T19:12:18.203739+0000","last_change":"2024-09-10T19:12:00.240895+0000","last_active":"2024-09-10T19:12:18.203739+0000","last_peered":"2024-09-10T19:12:18.203739+0000","last_clean":"2024-09-10T19:12:18.203739+0000","last_became_active":"2024-09-10T19:10:11.695141+0000","last_became_peered":"2024-09-10T19:10:11.695141+0000","last_unstale":"2024-09-10T19:12:18.203739+0000","last_undegraded":"2024-09-10T19:12:18.203739+0000","last_fullsized":"2024-09-10T19:12:18.203739+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:12:00.240841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:12:00.240841+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:13:25.944035+0000","scrub_duration":0.001239841,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":45,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-10T19:12:17.048065+0000","last_change":"2024-09-10T19:11:24.145056+0000","last_active":"2024-09-10T19:12:17.048065+0000","last_peered":"2024-09-10T19:12:17.048065+0000","last_clean":"2024-09-10T19:12:17.048065+0000","last_became_active":"2024-09-10T19:10:13.722894+0000","last_became_peered":"2024-09-10T19:10:13.722894+0000","last_unstale":"2024-09-10T19:12:17.048065+0000","last_undegraded":"2024-09-10T19:12:17.048065+0000","last_fullsized":"2024-09-10T19:12:17.048065+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:11:24.145003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:11:24.145003+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:12:26.405847+0000","scrub_duration":0.0013083789999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.f","version":"91'1751","reported_seq":1764,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-10T19:12:29.569280+0000","last_change":"2024-09-10T19:12:11.036448+0000","last_active":"2024-09-10T19:12:29.569280+0000","last_peered":"2024-09-10T19:12:29.569280+0000","last_clean":"2024-09-10T19:12:29.569280+0000","last_became_active":"2024-09-10T19:12:11.036181+0000","last_became_peered":"2024-09-10T19:12:11.036181+0000","last_unstale":"2024-09-10T19:12:29.569280+0000","last_undegraded":"2024-09-10T19:12:29.569280+0000","last_fullsized":"2024-09-10T19:12:29.569280+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:12:10.007465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:12:10.007465+0000","last_clean_scrub_stamp":"2024-09-10T19:12:10.007465+0000","objects_scrubbed":0,"log_size":1751,"log_dups_size":0,"ondisk_log_size":1751,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:13:19.194684+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":114753536,"num_objects":1751,"num_object_clones":0,"num_object_copies":5253,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1751,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1751,"num_write_kb":112064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.c","version":"66'10","reported_seq":63,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-10T19:12:18.293941+0000","last_change":"2024-09-10T19:11:14.987470+0000","last_active":"2024-09-10T19:12:18.293941+0000","last_peered":"2024-09-10T19:12:18.293941+0000","last_clean":"2024-09-10T19:12:18.293941+0000","last_became_active":"2024-09-10T19:10:09.688321+0000","last_became_peered":"2024-09-10T19:10:09.688321+0000","last_unstale":"2024-09-10T19:12:18.293941+0000","last_undegraded":"2024-09-10T19:12:18.293941+0000","last_fullsized":"2024-09-10T19:12:18.293941+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'10","last_scrub_stamp":"2024-09-10T19:11:14.987403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:11:14.987403+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0018747600000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":50,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-10T19:12:18.203874+0000","last_change":"2024-09-10T19:11:26.110216+0000","last_active":"2024-09-10T19:12:18.203874+0000","last_peered":"2024-09-10T19:12:18.203874+0000","last_clean":"2024-09-10T19:12:18.203874+0000","last_became_active":"2024-09-10T19:10:07.680803+0000","last_became_peered":"2024-09-10T19:10:07.680803+0000","last_unstale":"2024-09-10T19:12:18.203874+0000","last_undegraded":"2024-09-10T19:12:18.203874+0000","last_fullsized":"2024-09-10T19:12:18.203874+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:11:26.110138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:11:26.110138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:12:35.580206+0000","scrub_duration":0.0011581549999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.d","version":"80'9","reported_seq":87,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-10T19:12:25.195276+0000","last_change":"2024-09-10T19:11:25.156512+0000","last_active":"2024-09-10T19:12:25.195276+0000","last_peered":"2024-09-10T19:12:25.195276+0000","last_clean":"2024-09-10T19:12:25.195276+0000","last_became_active":"2024-09-10T19:10:11.698547+0000","last_became_peered":"2024-09-10T19:10:11.698547+0000","last_unstale":"2024-09-10T19:12:25.195276+0000","last_undegraded":"2024-09-10T19:12:25.195276+0000","last_fullsized":"2024-09-10T19:12:25.195276+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"80'9","last_scrub_stamp":"2024-09-10T19:11:25.156465+0000","last_deep_scrub":"80'9","last_deep_scrub_stamp":"2024-09-10T19:11:25.156465+0000","last_clean_scrub_stamp":"2024-09-10T19:11:25.156465+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:12:26.762818+0000","scrub_duration":0.0022299569999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":44,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-10T19:12:18.294710+0000","last_change":"2024-09-10T19:11:26.096147+0000","last_active":"2024-09-10T19:12:18.294710+0000","last_peered":"2024-09-10T19:12:18.294710+0000","last_clean":"2024-09-10T19:12:18.294710+0000","last_became_active":"2024-09-10T19:10:13.718242+0000","last_became_peered":"2024-09-10T19:10:13.718242+0000","last_unstale":"2024-09-10T19:12:18.294710+0000","last_undegraded":"2024-09-10T19:12:18.294710+0000","last_fullsized":"2024-09-10T19:12:18.294710+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:11:26.096090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:11:26.096090+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:12:49.347064+0000","scrub_duration":0.0015599909999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.c","version":"91'1711","reported_seq":1724,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-10T19:12:29.558466+0000","last_change":"2024-09-10T19:12:11.030171+0000","last_active":"2024-09-10T19:12:29.558466+0000","last_peered":"2024-09-10T19:12:29.558466+0000","last_clean":"2024-09-10T19:12:29.558466+0000","last_became_active":"2024-09-10T19:12:11.029860+0000","last_became_peered":"2024-09-10T19:12:11.029860+0000","last_unstale":"2024-09-10T19:12:29.558466+0000","last_undegraded":"2024-09-10T19:12:29.558466+0000","last_fullsized":"2024-09-10T19:12:29.558466+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:12:10.007465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:12:10.007465+0000","last_clean_scrub_stamp":"2024-09-10T19:12:10.007465+0000","objects_scrubbed":0,"log_size":1711,"log_dups_size":0,"ondisk_log_size":1711,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:13:31.161702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112132096,"num_objects":1711,"num_object_clones":0,"num_object_copies":5133,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1711,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1711,"num_write_kb":109504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.f","version":"66'15","reported_seq":73,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-10T19:12:18.203202+0000","last_change":"2024-09-10T19:11:59.232203+0000","last_active":"2024-09-10T19:12:18.203202+0000","last_peered":"2024-09-10T19:12:18.203202+0000","last_clean":"2024-09-10T19:12:18.203202+0000","last_became_active":"2024-09-10T19:10:09.691230+0000","last_became_peered":"2024-09-10T19:10:09.691230+0000","last_unstale":"2024-09-10T19:12:18.203202+0000","last_undegraded":"2024-09-10T19:12:18.203202+0000","last_fullsized":"2024-09-10T19:12:18.203202+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'15","last_scrub_stamp":"2024-09-10T19:11:59.232145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:11:59.232145+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:13:03.230026+0000","scrub_duration":0.001721729,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":50,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-10T19:12:18.203582+0000","last_change":"2024-09-10T19:11:11.036017+0000","last_active":"2024-09-10T19:12:18.203582+0000","last_peered":"2024-09-10T19:12:18.203582+0000","last_clean":"2024-09-10T19:12:18.203582+0000","last_became_active":"2024-09-10T19:10:07.679646+0000","last_became_peered":"2024-09-10T19:10:07.679646+0000","last_unstale":"2024-09-10T19:12:18.203582+0000","last_undegraded":"2024-09-10T19:12:18.203582+0000","last_fullsized":"2024-09-10T19:12:18.203582+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:11:11.035954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:11:11.035954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:12:35.469086+0000","scrub_duration":0.001618024,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.e","version":"66'8","reported_seq":82,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-10T19:12:25.191106+0000","last_change":"2024-09-10T19:12:00.134716+0000","last_active":"2024-09-10T19:12:25.191106+0000","last_peered":"2024-09-10T19:12:25.191106+0000","last_clean":"2024-09-10T19:12:25.191106+0000","last_became_active":"2024-09-10T19:10:11.699232+0000","last_became_peered":"2024-09-10T19:10:11.699232+0000","last_unstale":"2024-09-10T19:12:25.191106+0000","last_undegraded":"2024-09-10T19:12:25.191106+0000","last_fullsized":"2024-09-10T19:12:25.191106+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'8","last_scrub_stamp":"2024-09-10T19:12:00.134657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:12:00.134657+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:13:18.798101+0000","scrub_duration":0.0017750890000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":45,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-10T19:12:17.047248+0000","last_change":"2024-09-10T19:11:18.208847+0000","last_active":"2024-09-10T19:12:17.047248+0000","last_peered":"2024-09-10T19:12:17.047248+0000","last_clean":"2024-09-10T19:12:17.047248+0000","last_became_active":"2024-09-10T19:10:13.715274+0000","last_became_peered":"2024-09-10T19:10:13.715274+0000","last_unstale":"2024-09-10T19:12:17.047248+0000","last_undegraded":"2024-09-10T19:12:17.047248+0000","last_fullsized":"2024-09-10T19:12:17.047248+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:11:18.208782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:11:18.208782+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:12:43.016878+0000","scrub_duration":0.001904481,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.d","version":"91'1744","reported_seq":1756,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-10T19:12:29.593330+0000","last_change":"2024-09-10T19:12:11.033180+0000","last_active":"2024-09-10T19:12:29.593330+0000","last_peered":"2024-09-10T19:12:29.593330+0000","last_clean":"2024-09-10T19:12:29.593330+0000","last_became_active":"2024-09-10T19:12:11.032872+0000","last_became_peered":"2024-09-10T19:12:11.032872+0000","last_unstale":"2024-09-10T19:12:29.593330+0000","last_undegraded":"2024-09-10T19:12:29.593330+0000","last_fullsized":"2024-09-10T19:12:29.593330+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:12:10.007465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:12:10.007465+0000","last_clean_scrub_stamp":"2024-09-10T19:12:10.007465+0000","objects_scrubbed":0,"log_size":1744,"log_dups_size":0,"ondisk_log_size":1744,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:13:10.028740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":114294784,"num_objects":1744,"num_object_clones":0,"num_object_copies":5232,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1744,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1744,"num_write_kb":111616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.e","version":"84'16","reported_seq":72,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-10T19:12:18.294885+0000","last_change":"2024-09-10T19:11:56.158077+0000","last_active":"2024-09-10T19:12:18.294885+0000","last_peered":"2024-09-10T19:12:18.294885+0000","last_clean":"2024-09-10T19:12:18.294885+0000","last_became_active":"2024-09-10T19:10:09.692675+0000","last_became_peered":"2024-09-10T19:10:09.692675+0000","last_unstale":"2024-09-10T19:12:18.294885+0000","last_undegraded":"2024-09-10T19:12:18.294885+0000","last_fullsized":"2024-09-10T19:12:18.294885+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'16","last_scrub_stamp":"2024-09-10T19:11:56.157975+0000","last_deep_scrub":"84'16","last_deep_scrub_stamp":"2024-09-10T19:11:56.157975+0000","last_clean_scrub_stamp":"2024-09-10T19:11:56.157975+0000","objects_scrubbed":5,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:13:03.795623+0000","scrub_duration":0.0045462239999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":51,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-10T19:12:18.444220+0000","last_change":"2024-09-10T19:11:53.066918+0000","last_active":"2024-09-10T19:12:18.444220+0000","last_peered":"2024-09-10T19:12:18.444220+0000","last_clean":"2024-09-10T19:12:18.444220+0000","last_became_active":"2024-09-10T19:10:07.690434+0000","last_became_peered":"2024-09-10T19:10:07.690434+0000","last_unstale":"2024-09-10T19:12:18.444220+0000","last_undegraded":"2024-09-10T19:12:18.444220+0000","last_fullsized":"2024-09-10T19:12:18.444220+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:11:53.066849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:11:53.066849+0000","last_clean_scrub_stamp":"2024-09-10T19:11:53.066849+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:13:16.562614+0000","scrub_duration":0.0024992679999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-10T19:12:17.047601+0000","last_change":"2024-09-10T19:11:51.345883+0000","last_active":"2024-09-10T19:12:17.047601+0000","last_peered":"2024-09-10T19:12:17.047601+0000","last_clean":"2024-09-10T19:12:17.047601+0000","last_became_active":"2024-09-10T19:10:11.692215+0000","last_became_peered":"2024-09-10T19:10:11.692215+0000","last_unstale":"2024-09-10T19:12:17.047601+0000","last_undegraded":"2024-09-10T19:12:17.047601+0000","last_fullsized":"2024-09-10T19:12:17.047601+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:11:51.345791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:11:51.345791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:13:02.401462+0000","scrub_duration":0.001653964,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":35,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-10T19:12:17.048966+0000","last_change":"2024-09-10T19:10:14.054106+0000","last_active":"2024-09-10T19:12:17.048966+0000","last_peered":"2024-09-10T19:12:17.048966+0000","last_clean":"2024-09-10T19:12:17.048966+0000","last_became_active":"2024-09-10T19:10:14.053736+0000","last_became_peered":"2024-09-10T19:10:14.053736+0000","last_unstale":"2024-09-10T19:12:17.048966+0000","last_undegraded":"2024-09-10T19:12:17.048966+0000","last_fullsized":"2024-09-10T19:12:17.048966+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:10:12.693598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.2","version":"91'1711","reported_seq":1724,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-10T19:12:29.576350+0000","last_change":"2024-09-10T19:12:11.035410+0000","last_active":"2024-09-10T19:12:29.576350+0000","last_peered":"2024-09-10T19:12:29.576350+0000","last_clean":"2024-09-10T19:12:29.576350+0000","last_became_active":"2024-09-10T19:12:11.032482+0000","last_became_peered":"2024-09-10T19:12:11.032482+0000","last_unstale":"2024-09-10T19:12:29.576350+0000","last_undegraded":"2024-09-10T19:12:29.576350+0000","last_fullsized":"2024-09-10T19:12:29.576350+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:12:10.007465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:12:10.007465+0000","last_clean_scrub_stamp":"2024-09-10T19:12:10.007465+0000","objects_scrubbed":0,"log_size":1711,"log_dups_size":0,"ondisk_log_size":1711,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:13:12.709062+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112132096,"num_objects":1711,"num_object_clones":0,"num_object_copies":5133,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1711,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1711,"num_write_kb":109504,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"66'14","reported_seq":69,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-10T19:12:18.293855+0000","last_change":"2024-09-10T19:11:16.976833+0000","last_active":"2024-09-10T19:12:18.293855+0000","last_peered":"2024-09-10T19:12:18.293855+0000","last_clean":"2024-09-10T19:12:18.293855+0000","last_became_active":"2024-09-10T19:10:09.686534+0000","last_became_peered":"2024-09-10T19:10:09.686534+0000","last_unstale":"2024-09-10T19:12:18.293855+0000","last_undegraded":"2024-09-10T19:12:18.293855+0000","last_fullsized":"2024-09-10T19:12:18.293855+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'14","last_scrub_stamp":"2024-09-10T19:11:16.976776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:11:16.976776+0000","objects_scrubbed":7,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:12:29.125897+0000","scrub_duration":0.001777144,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":51,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-10T19:12:17.961521+0000","last_change":"2024-09-10T19:11:22.801187+0000","last_active":"2024-09-10T19:12:17.961521+0000","last_peered":"2024-09-10T19:12:17.961521+0000","last_clean":"2024-09-10T19:12:17.961521+0000","last_became_active":"2024-09-10T19:10:07.684864+0000","last_became_peered":"2024-09-10T19:10:07.684864+0000","last_unstale":"2024-09-10T19:12:17.961521+0000","last_undegraded":"2024-09-10T19:12:17.961521+0000","last_fullsized":"2024-09-10T19:12:17.961521+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:11:22.801123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:11:22.801123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:12:47.860716+0000","scrub_duration":0.0015474410000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.0","version":"66'8","reported_seq":83,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-10T19:12:25.198474+0000","last_change":"2024-09-10T19:11:29.884792+0000","last_active":"2024-09-10T19:12:25.198474+0000","last_peered":"2024-09-10T19:12:25.198474+0000","last_clean":"2024-09-10T19:12:25.198474+0000","last_became_active":"2024-09-10T19:10:11.697106+0000","last_became_peered":"2024-09-10T19:10:11.697106+0000","last_unstale":"2024-09-10T19:12:25.198474+0000","last_undegraded":"2024-09-10T19:12:25.198474+0000","last_fullsized":"2024-09-10T19:12:25.198474+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'8","last_scrub_stamp":"2024-09-10T19:11:29.884739+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:11:29.884739+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:12:37.090103+0000","scrub_duration":0.0014372650000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":45,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-10T19:12:18.444401+0000","last_change":"2024-09-10T19:11:41.073902+0000","last_active":"2024-09-10T19:12:18.444401+0000","last_peered":"2024-09-10T19:12:18.444401+0000","last_clean":"2024-09-10T19:12:18.444401+0000","last_became_active":"2024-09-10T19:10:14.054456+0000","last_became_peered":"2024-09-10T19:10:14.054456+0000","last_unstale":"2024-09-10T19:12:18.444401+0000","last_undegraded":"2024-09-10T19:12:18.444401+0000","last_fullsized":"2024-09-10T19:12:18.444401+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:11:41.073833+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:11:41.073833+0000","last_clean_scrub_stamp":"2024-09-10T19:11:41.073833+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:13:00.467462+0000","scrub_duration":0.001716567,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.3","version":"91'1750","reported_seq":1763,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-10T19:12:29.589297+0000","last_change":"2024-09-10T19:12:11.030188+0000","last_active":"2024-09-10T19:12:29.589297+0000","last_peered":"2024-09-10T19:12:29.589297+0000","last_clean":"2024-09-10T19:12:29.589297+0000","last_became_active":"2024-09-10T19:12:11.029906+0000","last_became_peered":"2024-09-10T19:12:11.029906+0000","last_unstale":"2024-09-10T19:12:29.589297+0000","last_undegraded":"2024-09-10T19:12:29.589297+0000","last_fullsized":"2024-09-10T19:12:29.589297+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:12:10.007465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:12:10.007465+0000","last_clean_scrub_stamp":"2024-09-10T19:12:10.007465+0000","objects_scrubbed":0,"log_size":1750,"log_dups_size":0,"ondisk_log_size":1750,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:13:16.954937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":114688000,"num_objects":1750,"num_object_clones":0,"num_object_copies":5250,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1750,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1750,"num_write_kb":112000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"66'18","reported_seq":81,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-10T19:12:17.047329+0000","last_change":"2024-09-10T19:11:19.848186+0000","last_active":"2024-09-10T19:12:17.047329+0000","last_peered":"2024-09-10T19:12:17.047329+0000","last_clean":"2024-09-10T19:12:17.047329+0000","last_became_active":"2024-09-10T19:10:09.821836+0000","last_became_peered":"2024-09-10T19:10:09.821836+0000","last_unstale":"2024-09-10T19:12:17.047329+0000","last_undegraded":"2024-09-10T19:12:17.047329+0000","last_fullsized":"2024-09-10T19:12:17.047329+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'18","last_scrub_stamp":"2024-09-10T19:11:19.848132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:11:19.848132+0000","objects_scrubbed":8,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:12:26.692213+0000","scrub_duration":0.0034247819999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":50,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-10T19:12:18.204064+0000","last_change":"2024-09-10T19:11:42.202747+0000","last_active":"2024-09-10T19:12:18.204064+0000","last_peered":"2024-09-10T19:12:18.204064+0000","last_clean":"2024-09-10T19:12:18.204064+0000","last_became_active":"2024-09-10T19:10:07.682081+0000","last_became_peered":"2024-09-10T19:10:07.682081+0000","last_unstale":"2024-09-10T19:12:18.204064+0000","last_undegraded":"2024-09-10T19:12:18.204064+0000","last_fullsized":"2024-09-10T19:12:18.204064+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:11:42.202689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:11:42.202689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:12:53.708170+0000","scrub_duration":0.00127655,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":46,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-10T19:12:18.294217+0000","last_change":"2024-09-10T19:11:29.069421+0000","last_active":"2024-09-10T19:12:18.294217+0000","last_peered":"2024-09-10T19:12:18.294217+0000","last_clean":"2024-09-10T19:12:18.294217+0000","last_became_active":"2024-09-10T19:10:11.698054+0000","last_became_peered":"2024-09-10T19:10:11.698054+0000","last_unstale":"2024-09-10T19:12:18.294217+0000","last_undegraded":"2024-09-10T19:12:18.294217+0000","last_fullsized":"2024-09-10T19:12:18.294217+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:11:29.069369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:11:29.069369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:12:55.817612+0000","scrub_duration":0.001327562,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":44,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-10T19:12:18.294319+0000","last_change":"2024-09-10T19:11:34.139371+0000","last_active":"2024-09-10T19:12:18.294319+0000","last_peered":"2024-09-10T19:12:18.294319+0000","last_clean":"2024-09-10T19:12:18.294319+0000","last_became_active":"2024-09-10T19:10:13.722784+0000","last_became_peered":"2024-09-10T19:10:13.722784+0000","last_unstale":"2024-09-10T19:12:18.294319+0000","last_undegraded":"2024-09-10T19:12:18.294319+0000","last_fullsized":"2024-09-10T19:12:18.294319+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:11:34.139306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:11:34.139306+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:12:45.529547+0000","scrub_duration":0.00133668,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.0","version":"91'1746","reported_seq":1759,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-10T19:12:29.554910+0000","last_change":"2024-09-10T19:12:11.030329+0000","last_active":"2024-09-10T19:12:29.554910+0000","last_peered":"2024-09-10T19:12:29.554910+0000","last_clean":"2024-09-10T19:12:29.554910+0000","last_became_active":"2024-09-10T19:12:11.030025+0000","last_became_peered":"2024-09-10T19:12:11.030025+0000","last_unstale":"2024-09-10T19:12:29.554910+0000","last_undegraded":"2024-09-10T19:12:29.554910+0000","last_fullsized":"2024-09-10T19:12:29.554910+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:12:10.007465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:12:10.007465+0000","last_clean_scrub_stamp":"2024-09-10T19:12:10.007465+0000","objects_scrubbed":0,"log_size":1746,"log_dups_size":0,"ondisk_log_size":1746,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:13:27.037283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":114425856,"num_objects":1746,"num_object_clones":0,"num_object_copies":5238,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1746,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1746,"num_write_kb":111744,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.3","version":"66'19","reported_seq":85,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-10T19:12:18.122279+0000","last_change":"2024-09-10T19:11:55.955294+0000","last_active":"2024-09-10T19:12:18.122279+0000","last_peered":"2024-09-10T19:12:18.122279+0000","last_clean":"2024-09-10T19:12:18.122279+0000","last_became_active":"2024-09-10T19:10:09.823003+0000","last_became_peered":"2024-09-10T19:10:09.823003+0000","last_unstale":"2024-09-10T19:12:18.122279+0000","last_undegraded":"2024-09-10T19:12:18.122279+0000","last_fullsized":"2024-09-10T19:12:18.122279+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'19","last_scrub_stamp":"2024-09-10T19:11:55.955238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:11:55.955238+0000","objects_scrubbed":8,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:12:59.464484+0000","scrub_duration":0.0021230300000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":51,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-10T19:12:18.444844+0000","last_change":"2024-09-10T19:11:25.290147+0000","last_active":"2024-09-10T19:12:18.444844+0000","last_peered":"2024-09-10T19:12:18.444844+0000","last_clean":"2024-09-10T19:12:18.444844+0000","last_became_active":"2024-09-10T19:10:07.690325+0000","last_became_peered":"2024-09-10T19:10:07.690325+0000","last_unstale":"2024-09-10T19:12:18.444844+0000","last_undegraded":"2024-09-10T19:12:18.444844+0000","last_fullsized":"2024-09-10T19:12:18.444844+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:11:25.290088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:11:25.290088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:12:31.270129+0000","scrub_duration":0.0013977760000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-10T19:12:17.960535+0000","last_change":"2024-09-10T19:11:23.843315+0000","last_active":"2024-09-10T19:12:17.960535+0000","last_peered":"2024-09-10T19:12:17.960535+0000","last_clean":"2024-09-10T19:12:17.960535+0000","last_became_active":"2024-09-10T19:10:11.700370+0000","last_became_peered":"2024-09-10T19:10:11.700370+0000","last_unstale":"2024-09-10T19:12:17.960535+0000","last_undegraded":"2024-09-10T19:12:17.960535+0000","last_fullsized":"2024-09-10T19:12:17.960535+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:11:23.843228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:11:23.843228+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:12:29.095733+0000","scrub_duration":0.0013052070000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":44,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-10T19:12:18.203423+0000","last_change":"2024-09-10T19:11:51.187167+0000","last_active":"2024-09-10T19:12:18.203423+0000","last_peered":"2024-09-10T19:12:18.203423+0000","last_clean":"2024-09-10T19:12:18.203423+0000","last_became_active":"2024-09-10T19:10:14.054318+0000","last_became_peered":"2024-09-10T19:10:14.054318+0000","last_unstale":"2024-09-10T19:12:18.203423+0000","last_undegraded":"2024-09-10T19:12:18.203423+0000","last_fullsized":"2024-09-10T19:12:18.203423+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:11:51.187109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:11:51.187109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:12:53.163034+0000","scrub_duration":0.0017758800000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.1","version":"91'1754","reported_seq":1767,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-10T19:12:29.597798+0000","last_change":"2024-09-10T19:12:11.030716+0000","last_active":"2024-09-10T19:12:29.597798+0000","last_peered":"2024-09-10T19:12:29.597798+0000","last_clean":"2024-09-10T19:12:29.597798+0000","last_became_active":"2024-09-10T19:12:11.030399+0000","last_became_peered":"2024-09-10T19:12:11.030399+0000","last_unstale":"2024-09-10T19:12:29.597798+0000","last_undegraded":"2024-09-10T19:12:29.597798+0000","last_fullsized":"2024-09-10T19:12:29.597798+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:12:10.007465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:12:10.007465+0000","last_clean_scrub_stamp":"2024-09-10T19:12:10.007465+0000","objects_scrubbed":0,"log_size":1754,"log_dups_size":0,"ondisk_log_size":1754,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:13:28.881179+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":114950144,"num_objects":1754,"num_object_clones":0,"num_object_copies":5262,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1754,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1754,"num_write_kb":112256,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.2","version":"66'10","reported_seq":31,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-10T19:12:17.047580+0000","last_change":"2024-09-10T19:11:44.754488+0000","last_active":"2024-09-10T19:12:17.047580+0000","last_peered":"2024-09-10T19:12:17.047580+0000","last_clean":"2024-09-10T19:12:17.047580+0000","last_became_active":"2024-09-10T19:11:44.748517+0000","last_became_peered":"2024-09-10T19:11:44.748517+0000","last_unstale":"2024-09-10T19:12:17.047580+0000","last_undegraded":"2024-09-10T19:12:17.047580+0000","last_fullsized":"2024-09-10T19:12:17.047580+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'10","last_scrub_stamp":"2024-09-10T19:11:28.132850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:11:28.132850+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:12:49.124860+0000","scrub_duration":0.0018173429999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":50,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-10T19:12:18.203865+0000","last_change":"2024-09-10T19:11:55.227417+0000","last_active":"2024-09-10T19:12:18.203865+0000","last_peered":"2024-09-10T19:12:18.203865+0000","last_clean":"2024-09-10T19:12:18.203865+0000","last_became_active":"2024-09-10T19:10:07.681965+0000","last_became_peered":"2024-09-10T19:10:07.681965+0000","last_unstale":"2024-09-10T19:12:18.203865+0000","last_undegraded":"2024-09-10T19:12:18.203865+0000","last_fullsized":"2024-09-10T19:12:18.203865+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:11:55.227355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:11:55.227355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:13:12.543009+0000","scrub_duration":0.001737775,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"84'9","reported_seq":84,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-10T19:12:25.196364+0000","last_change":"2024-09-10T19:11:53.879325+0000","last_active":"2024-09-10T19:12:25.196364+0000","last_peered":"2024-09-10T19:12:25.196364+0000","last_clean":"2024-09-10T19:12:25.196364+0000","last_became_active":"2024-09-10T19:10:11.699690+0000","last_became_peered":"2024-09-10T19:10:11.699690+0000","last_unstale":"2024-09-10T19:12:25.196364+0000","last_undegraded":"2024-09-10T19:12:25.196364+0000","last_fullsized":"2024-09-10T19:12:25.196364+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'9","last_scrub_stamp":"2024-09-10T19:11:53.879272+0000","last_deep_scrub":"84'9","last_deep_scrub_stamp":"2024-09-10T19:11:53.879272+0000","last_clean_scrub_stamp":"2024-09-10T19:11:53.879272+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:13:20.209734+0000","scrub_duration":0.0044029450000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":45,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-10T19:12:18.122419+0000","last_change":"2024-09-10T19:11:43.922577+0000","last_active":"2024-09-10T19:12:18.122419+0000","last_peered":"2024-09-10T19:12:18.122419+0000","last_clean":"2024-09-10T19:12:18.122419+0000","last_became_active":"2024-09-10T19:10:13.720403+0000","last_became_peered":"2024-09-10T19:10:13.720403+0000","last_unstale":"2024-09-10T19:12:18.122419+0000","last_undegraded":"2024-09-10T19:12:18.122419+0000","last_fullsized":"2024-09-10T19:12:18.122419+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:11:43.922522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:11:43.922522+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:13:13.573808+0000","scrub_duration":0.0012347289999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.7","version":"91'1790","reported_seq":1803,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-10T19:12:29.594476+0000","last_change":"2024-09-10T19:12:11.033186+0000","last_active":"2024-09-10T19:12:29.594476+0000","last_peered":"2024-09-10T19:12:29.594476+0000","last_clean":"2024-09-10T19:12:29.594476+0000","last_became_active":"2024-09-10T19:12:11.032857+0000","last_became_peered":"2024-09-10T19:12:11.032857+0000","last_unstale":"2024-09-10T19:12:29.594476+0000","last_undegraded":"2024-09-10T19:12:29.594476+0000","last_fullsized":"2024-09-10T19:12:29.594476+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:12:10.007465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:12:10.007465+0000","last_clean_scrub_stamp":"2024-09-10T19:12:10.007465+0000","objects_scrubbed":0,"log_size":1790,"log_dups_size":0,"ondisk_log_size":1790,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:13:38.922063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117309440,"num_objects":1790,"num_object_clones":0,"num_object_copies":5370,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1790,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1790,"num_write_kb":114560,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.4","version":"66'28","reported_seq":124,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-10T19:12:25.192618+0000","last_change":"2024-09-10T19:11:17.050501+0000","last_active":"2024-09-10T19:12:25.192618+0000","last_peered":"2024-09-10T19:12:25.192618+0000","last_clean":"2024-09-10T19:12:25.192618+0000","last_became_active":"2024-09-10T19:10:09.692663+0000","last_became_peered":"2024-09-10T19:10:09.692663+0000","last_unstale":"2024-09-10T19:12:25.192618+0000","last_undegraded":"2024-09-10T19:12:25.192618+0000","last_fullsized":"2024-09-10T19:12:25.192618+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'28","last_scrub_stamp":"2024-09-10T19:11:17.050442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:11:17.050442+0000","objects_scrubbed":10,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:12:31.661637+0000","scrub_duration":0.001985559,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":51,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-10T19:12:17.047045+0000","last_change":"2024-09-10T19:11:41.429457+0000","last_active":"2024-09-10T19:12:17.047045+0000","last_peered":"2024-09-10T19:12:17.047045+0000","last_clean":"2024-09-10T19:12:17.047045+0000","last_became_active":"2024-09-10T19:10:07.682724+0000","last_became_peered":"2024-09-10T19:10:07.682724+0000","last_unstale":"2024-09-10T19:12:17.047045+0000","last_undegraded":"2024-09-10T19:12:17.047045+0000","last_fullsized":"2024-09-10T19:12:17.047045+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:11:41.429402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:11:41.429402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:12:58.343046+0000","scrub_duration":0.0014226359999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-10T19:12:18.122998+0000","last_change":"2024-09-10T19:11:14.985340+0000","last_active":"2024-09-10T19:12:18.122998+0000","last_peered":"2024-09-10T19:12:18.122998+0000","last_clean":"2024-09-10T19:12:18.122998+0000","last_became_active":"2024-09-10T19:10:11.698825+0000","last_became_peered":"2024-09-10T19:10:11.698825+0000","last_unstale":"2024-09-10T19:12:18.122998+0000","last_undegraded":"2024-09-10T19:12:18.122998+0000","last_fullsized":"2024-09-10T19:12:18.122998+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:11:14.985249+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:11:14.985249+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:12:30.155012+0000","scrub_duration":0.001460161,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":35,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-10T19:12:17.048964+0000","last_change":"2024-09-10T19:10:13.722038+0000","last_active":"2024-09-10T19:12:17.048964+0000","last_peered":"2024-09-10T19:12:17.048964+0000","last_clean":"2024-09-10T19:12:17.048964+0000","last_became_active":"2024-09-10T19:10:13.721744+0000","last_became_peered":"2024-09-10T19:10:13.721744+0000","last_unstale":"2024-09-10T19:12:17.048964+0000","last_undegraded":"2024-09-10T19:12:17.048964+0000","last_fullsized":"2024-09-10T19:12:17.048964+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:10:12.693598+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.4","version":"91'1765","reported_seq":1778,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-10T19:12:29.585953+0000","last_change":"2024-09-10T19:12:11.033251+0000","last_active":"2024-09-10T19:12:29.585953+0000","last_peered":"2024-09-10T19:12:29.585953+0000","last_clean":"2024-09-10T19:12:29.585953+0000","last_became_active":"2024-09-10T19:12:11.032935+0000","last_became_peered":"2024-09-10T19:12:11.032935+0000","last_unstale":"2024-09-10T19:12:29.585953+0000","last_undegraded":"2024-09-10T19:12:29.585953+0000","last_fullsized":"2024-09-10T19:12:29.585953+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:12:10.007465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:12:10.007465+0000","last_clean_scrub_stamp":"2024-09-10T19:12:10.007465+0000","objects_scrubbed":0,"log_size":1765,"log_dups_size":0,"ondisk_log_size":1765,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:13:37.144463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":115671040,"num_objects":1765,"num_object_clones":0,"num_object_copies":5295,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1765,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1765,"num_write_kb":112960,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.7","version":"82'18","reported_seq":80,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-10T19:12:18.203208+0000","last_change":"2024-09-10T19:11:49.207365+0000","last_active":"2024-09-10T19:12:18.203208+0000","last_peered":"2024-09-10T19:12:18.203208+0000","last_clean":"2024-09-10T19:12:18.203208+0000","last_became_active":"2024-09-10T19:10:09.693346+0000","last_became_peered":"2024-09-10T19:10:09.693346+0000","last_unstale":"2024-09-10T19:12:18.203208+0000","last_undegraded":"2024-09-10T19:12:18.203208+0000","last_fullsized":"2024-09-10T19:12:18.203208+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'18","last_scrub_stamp":"2024-09-10T19:11:49.207309+0000","last_deep_scrub":"82'18","last_deep_scrub_stamp":"2024-09-10T19:11:49.207309+0000","last_clean_scrub_stamp":"2024-09-10T19:11:49.207309+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:13:10.129757+0000","scrub_duration":0.0046031240000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":19,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"67'5","reported_seq":194,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-10T19:12:27.992370+0000","last_change":"2024-09-10T19:11:16.007355+0000","last_active":"2024-09-10T19:12:27.992370+0000","last_peered":"2024-09-10T19:12:27.992370+0000","last_clean":"2024-09-10T19:12:27.992370+0000","last_became_active":"2024-09-10T19:10:07.682125+0000","last_became_peered":"2024-09-10T19:10:07.682125+0000","last_unstale":"2024-09-10T19:12:27.992370+0000","last_undegraded":"2024-09-10T19:12:27.992370+0000","last_fullsized":"2024-09-10T19:12:27.992370+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'5","last_scrub_stamp":"2024-09-10T19:11:16.007261+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:11:16.007261+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:12:39.490457+0000","scrub_duration":0.0019059680000000001,"objects_trimmed":0,"snaptrim_duration":0.00034454499999999997,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":136,"num_read_kb":131,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":51,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-10T19:12:17.047384+0000","last_change":"2024-09-10T19:11:33.340773+0000","last_active":"2024-09-10T19:12:17.047384+0000","last_peered":"2024-09-10T19:12:17.047384+0000","last_clean":"2024-09-10T19:12:17.047384+0000","last_became_active":"2024-09-10T19:10:07.681456+0000","last_became_peered":"2024-09-10T19:10:07.681456+0000","last_unstale":"2024-09-10T19:12:17.047384+0000","last_undegraded":"2024-09-10T19:12:17.047384+0000","last_fullsized":"2024-09-10T19:12:17.047384+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:11:33.340715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:11:33.340715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:12:37.277130+0000","scrub_duration":0.0012840099999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-10T19:12:17.047794+0000","last_change":"2024-09-10T19:11:36.443407+0000","last_active":"2024-09-10T19:12:17.047794+0000","last_peered":"2024-09-10T19:12:17.047794+0000","last_clean":"2024-09-10T19:12:17.047794+0000","last_became_active":"2024-09-10T19:10:11.695205+0000","last_became_peered":"2024-09-10T19:10:11.695205+0000","last_unstale":"2024-09-10T19:12:17.047794+0000","last_undegraded":"2024-09-10T19:12:17.047794+0000","last_fullsized":"2024-09-10T19:12:17.047794+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:11:36.443352+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:11:36.443352+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:12:59.383628+0000","scrub_duration":0.001511814,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":45,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-10T19:12:18.444635+0000","last_change":"2024-09-10T19:12:06.040047+0000","last_active":"2024-09-10T19:12:18.444635+0000","last_peered":"2024-09-10T19:12:18.444635+0000","last_clean":"2024-09-10T19:12:18.444635+0000","last_became_active":"2024-09-10T19:10:14.053697+0000","last_became_peered":"2024-09-10T19:10:14.053697+0000","last_unstale":"2024-09-10T19:12:18.444635+0000","last_undegraded":"2024-09-10T19:12:18.444635+0000","last_fullsized":"2024-09-10T19:12:18.444635+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:12:06.039981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:12:06.039981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:13:09.353692+0000","scrub_duration":0.0013437499999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.5","version":"91'1712","reported_seq":1725,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-10T19:12:29.593503+0000","last_change":"2024-09-10T19:12:11.033491+0000","last_active":"2024-09-10T19:12:29.593503+0000","last_peered":"2024-09-10T19:12:29.593503+0000","last_clean":"2024-09-10T19:12:29.593503+0000","last_became_active":"2024-09-10T19:12:11.032880+0000","last_became_peered":"2024-09-10T19:12:11.032880+0000","last_unstale":"2024-09-10T19:12:29.593503+0000","last_undegraded":"2024-09-10T19:12:29.593503+0000","last_fullsized":"2024-09-10T19:12:29.593503+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:12:10.007465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:12:10.007465+0000","last_clean_scrub_stamp":"2024-09-10T19:12:10.007465+0000","objects_scrubbed":0,"log_size":1712,"log_dups_size":0,"ondisk_log_size":1712,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:13:37.808289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112197632,"num_objects":1712,"num_object_clones":0,"num_object_copies":5136,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1712,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1712,"num_write_kb":109568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"66'12","reported_seq":67,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-10T19:12:18.122341+0000","last_change":"2024-09-10T19:11:38.936913+0000","last_active":"2024-09-10T19:12:18.122341+0000","last_peered":"2024-09-10T19:12:18.122341+0000","last_clean":"2024-09-10T19:12:18.122341+0000","last_became_active":"2024-09-10T19:10:09.688091+0000","last_became_peered":"2024-09-10T19:10:09.688091+0000","last_unstale":"2024-09-10T19:12:18.122341+0000","last_undegraded":"2024-09-10T19:12:18.122341+0000","last_fullsized":"2024-09-10T19:12:18.122341+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'12","last_scrub_stamp":"2024-09-10T19:11:38.936857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:11:38.936857+0000","objects_scrubbed":6,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:12:55.623292+0000","scrub_duration":0.0016916909999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"61'1","reported_seq":57,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-10T19:12:18.122705+0000","last_change":"2024-09-10T19:11:54.921281+0000","last_active":"2024-09-10T19:12:18.122705+0000","last_peered":"2024-09-10T19:12:18.122705+0000","last_clean":"2024-09-10T19:12:18.122705+0000","last_became_active":"2024-09-10T19:10:07.683259+0000","last_became_peered":"2024-09-10T19:10:07.683259+0000","last_unstale":"2024-09-10T19:12:18.122705+0000","last_undegraded":"2024-09-10T19:12:18.122705+0000","last_fullsized":"2024-09-10T19:12:18.122705+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'1","last_scrub_stamp":"2024-09-10T19:11:54.921214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:11:54.921214+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:13:04.518575+0000","scrub_duration":0.001623153,"objects_trimmed":0,"snaptrim_duration":0.0014341130000000001,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":51,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-10T19:12:18.444309+0000","last_change":"2024-09-10T19:11:21.301130+0000","last_active":"2024-09-10T19:12:18.444309+0000","last_peered":"2024-09-10T19:12:18.444309+0000","last_clean":"2024-09-10T19:12:18.444309+0000","last_became_active":"2024-09-10T19:10:07.690549+0000","last_became_peered":"2024-09-10T19:10:07.690549+0000","last_unstale":"2024-09-10T19:12:18.444309+0000","last_undegraded":"2024-09-10T19:12:18.444309+0000","last_fullsized":"2024-09-10T19:12:18.444309+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:11:21.301058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:11:21.301058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:12:41.356053+0000","scrub_duration":0.0021074710000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-10T19:12:17.047628+0000","last_change":"2024-09-10T19:11:16.156929+0000","last_active":"2024-09-10T19:12:17.047628+0000","last_peered":"2024-09-10T19:12:17.047628+0000","last_clean":"2024-09-10T19:12:17.047628+0000","last_became_active":"2024-09-10T19:10:11.699798+0000","last_became_peered":"2024-09-10T19:10:11.699798+0000","last_unstale":"2024-09-10T19:12:17.047628+0000","last_undegraded":"2024-09-10T19:12:17.047628+0000","last_fullsized":"2024-09-10T19:12:17.047628+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:11:16.156819+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:11:16.156819+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001649059,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":44,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-10T19:12:18.203696+0000","last_change":"2024-09-10T19:12:05.165558+0000","last_active":"2024-09-10T19:12:18.203696+0000","last_peered":"2024-09-10T19:12:18.203696+0000","last_clean":"2024-09-10T19:12:18.203696+0000","last_became_active":"2024-09-10T19:10:13.722368+0000","last_became_peered":"2024-09-10T19:10:13.722368+0000","last_unstale":"2024-09-10T19:12:18.203696+0000","last_undegraded":"2024-09-10T19:12:18.203696+0000","last_fullsized":"2024-09-10T19:12:18.203696+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:12:05.165500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:12:05.165500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:13:30.706182+0000","scrub_duration":0.001257396,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.6","version":"91'1755","reported_seq":1766,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-10T19:12:29.588565+0000","last_change":"2024-09-10T19:12:11.035560+0000","last_active":"2024-09-10T19:12:29.588565+0000","last_peered":"2024-09-10T19:12:29.588565+0000","last_clean":"2024-09-10T19:12:29.588565+0000","last_became_active":"2024-09-10T19:12:11.035290+0000","last_became_peered":"2024-09-10T19:12:11.035290+0000","last_unstale":"2024-09-10T19:12:29.588565+0000","last_undegraded":"2024-09-10T19:12:29.588565+0000","last_fullsized":"2024-09-10T19:12:29.588565+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:12:10.007465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:12:10.007465+0000","last_clean_scrub_stamp":"2024-09-10T19:12:10.007465+0000","objects_scrubbed":0,"log_size":1755,"log_dups_size":0,"ondisk_log_size":1755,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:13:16.864506+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":115015680,"num_objects":1755,"num_object_clones":0,"num_object_copies":5265,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1755,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1755,"num_write_kb":112320,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.5","version":"80'21","reported_seq":78,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-10T19:12:17.961163+0000","last_change":"2024-09-10T19:11:39.836197+0000","last_active":"2024-09-10T19:12:17.961163+0000","last_peered":"2024-09-10T19:12:17.961163+0000","last_clean":"2024-09-10T19:12:17.961163+0000","last_became_active":"2024-09-10T19:10:09.822184+0000","last_became_peered":"2024-09-10T19:10:09.822184+0000","last_unstale":"2024-09-10T19:12:17.961163+0000","last_undegraded":"2024-09-10T19:12:17.961163+0000","last_fullsized":"2024-09-10T19:12:17.961163+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"80'21","last_scrub_stamp":"2024-09-10T19:11:39.836144+0000","last_deep_scrub":"80'21","last_deep_scrub_stamp":"2024-09-10T19:11:39.836144+0000","last_clean_scrub_stamp":"2024-09-10T19:11:39.836144+0000","objects_scrubbed":7,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:12:48.908909+0000","scrub_duration":0.0049231029999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"63'2","reported_seq":58,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-10T19:12:17.048830+0000","last_change":"2024-09-10T19:11:26.824164+0000","last_active":"2024-09-10T19:12:17.048830+0000","last_peered":"2024-09-10T19:12:17.048830+0000","last_clean":"2024-09-10T19:12:17.048830+0000","last_became_active":"2024-09-10T19:10:07.684784+0000","last_became_peered":"2024-09-10T19:10:07.684784+0000","last_unstale":"2024-09-10T19:12:17.048830+0000","last_undegraded":"2024-09-10T19:12:17.048830+0000","last_fullsized":"2024-09-10T19:12:17.048830+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'2","last_scrub_stamp":"2024-09-10T19:11:26.824103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:11:26.824103+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:12:48.229817+0000","scrub_duration":0.0018538929999999999,"objects_trimmed":0,"snaptrim_duration":0.00075522999999999996,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":51,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-10T19:12:17.047184+0000","last_change":"2024-09-10T19:11:17.171319+0000","last_active":"2024-09-10T19:12:17.047184+0000","last_peered":"2024-09-10T19:12:17.047184+0000","last_clean":"2024-09-10T19:12:17.047184+0000","last_became_active":"2024-09-10T19:10:07.687596+0000","last_became_peered":"2024-09-10T19:10:07.687596+0000","last_unstale":"2024-09-10T19:12:17.047184+0000","last_undegraded":"2024-09-10T19:12:17.047184+0000","last_fullsized":"2024-09-10T19:12:17.047184+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:11:17.171251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:11:17.171251+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0016247670000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"74'79","reported_seq":96,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-10T19:12:18.444494+0000","last_change":"2024-09-10T19:12:09.938638+0000","last_active":"2024-09-10T19:12:18.444494+0000","last_peered":"2024-09-10T19:12:18.444494+0000","last_clean":"2024-09-10T19:12:18.444494+0000","last_became_active":"2024-09-10T19:09:56.634919+0000","last_became_peered":"2024-09-10T19:09:56.634919+0000","last_unstale":"2024-09-10T19:12:18.444494+0000","last_undegraded":"2024-09-10T19:12:18.444494+0000","last_fullsized":"2024-09-10T19:12:18.444494+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'79","last_scrub_stamp":"2024-09-10T19:12:09.938546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:08:00.699315+0000","last_clean_scrub_stamp":"2024-09-10T19:12:09.938546+0000","objects_scrubbed":2,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:13:12.056941+0000","scrub_duration":0.002371565,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-10T19:12:18.444442+0000","last_change":"2024-09-10T19:12:02.140246+0000","last_active":"2024-09-10T19:12:18.444442+0000","last_peered":"2024-09-10T19:12:18.444442+0000","last_clean":"2024-09-10T19:12:18.444442+0000","last_became_active":"2024-09-10T19:10:11.694986+0000","last_became_peered":"2024-09-10T19:10:11.694986+0000","last_unstale":"2024-09-10T19:12:18.444442+0000","last_undegraded":"2024-09-10T19:12:18.444442+0000","last_fullsized":"2024-09-10T19:12:18.444442+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:12:02.140193+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:12:02.140193+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:13:18.306894+0000","scrub_duration":0.0015805999999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":45,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-10T19:12:17.047717+0000","last_change":"2024-09-10T19:11:25.312410+0000","last_active":"2024-09-10T19:12:17.047717+0000","last_peered":"2024-09-10T19:12:17.047717+0000","last_clean":"2024-09-10T19:12:17.047717+0000","last_became_active":"2024-09-10T19:10:13.720647+0000","last_became_peered":"2024-09-10T19:10:13.720647+0000","last_unstale":"2024-09-10T19:12:17.047717+0000","last_undegraded":"2024-09-10T19:12:17.047717+0000","last_fullsized":"2024-09-10T19:12:17.047717+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:11:25.312356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:11:25.312356+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:12:54.878799+0000","scrub_duration":0.001690642,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.9","version":"91'1730","reported_seq":1743,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-10T19:12:29.603622+0000","last_change":"2024-09-10T19:12:11.033287+0000","last_active":"2024-09-10T19:12:29.603622+0000","last_peered":"2024-09-10T19:12:29.603622+0000","last_clean":"2024-09-10T19:12:29.603622+0000","last_became_active":"2024-09-10T19:12:11.032912+0000","last_became_peered":"2024-09-10T19:12:11.032912+0000","last_unstale":"2024-09-10T19:12:29.603622+0000","last_undegraded":"2024-09-10T19:12:29.603622+0000","last_fullsized":"2024-09-10T19:12:29.603622+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:12:10.007465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:12:10.007465+0000","last_clean_scrub_stamp":"2024-09-10T19:12:10.007465+0000","objects_scrubbed":0,"log_size":1730,"log_dups_size":0,"ondisk_log_size":1730,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:13:37.548647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":113377280,"num_objects":1730,"num_object_clones":0,"num_object_copies":5190,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1730,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1730,"num_write_kb":110720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"66'19","reported_seq":80,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-10T19:12:17.960920+0000","last_change":"2024-09-10T19:11:42.941440+0000","last_active":"2024-09-10T19:12:17.960920+0000","last_peered":"2024-09-10T19:12:17.960920+0000","last_clean":"2024-09-10T19:12:17.960920+0000","last_became_active":"2024-09-10T19:10:09.820880+0000","last_became_peered":"2024-09-10T19:10:09.820880+0000","last_unstale":"2024-09-10T19:12:17.960920+0000","last_undegraded":"2024-09-10T19:12:17.960920+0000","last_fullsized":"2024-09-10T19:12:17.960920+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'19","last_scrub_stamp":"2024-09-10T19:11:42.941375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:11:42.941375+0000","objects_scrubbed":9,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:13:12.267779+0000","scrub_duration":0.0021647599999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":51,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-10T19:12:17.048420+0000","last_change":"2024-09-10T19:11:40.484394+0000","last_active":"2024-09-10T19:12:17.048420+0000","last_peered":"2024-09-10T19:12:17.048420+0000","last_clean":"2024-09-10T19:12:17.048420+0000","last_became_active":"2024-09-10T19:10:07.686236+0000","last_became_peered":"2024-09-10T19:10:07.686236+0000","last_unstale":"2024-09-10T19:12:17.048420+0000","last_undegraded":"2024-09-10T19:12:17.048420+0000","last_fullsized":"2024-09-10T19:12:17.048420+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:11:40.484333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:11:40.484333+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:13:05.193849+0000","scrub_duration":0.0013800889999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-10T19:12:17.047325+0000","last_change":"2024-09-10T19:11:19.119050+0000","last_active":"2024-09-10T19:12:17.047325+0000","last_peered":"2024-09-10T19:12:17.047325+0000","last_clean":"2024-09-10T19:12:17.047325+0000","last_became_active":"2024-09-10T19:10:11.699745+0000","last_became_peered":"2024-09-10T19:10:11.699745+0000","last_unstale":"2024-09-10T19:12:17.047325+0000","last_undegraded":"2024-09-10T19:12:17.047325+0000","last_fullsized":"2024-09-10T19:12:17.047325+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:11:19.118987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:11:19.118987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:12:48.847171+0000","scrub_duration":0.0012170379999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":45,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-10T19:12:18.444398+0000","last_change":"2024-09-10T19:11:38.102859+0000","last_active":"2024-09-10T19:12:18.444398+0000","last_peered":"2024-09-10T19:12:18.444398+0000","last_clean":"2024-09-10T19:12:18.444398+0000","last_became_active":"2024-09-10T19:10:13.719973+0000","last_became_peered":"2024-09-10T19:10:13.719973+0000","last_unstale":"2024-09-10T19:12:18.444398+0000","last_undegraded":"2024-09-10T19:12:18.444398+0000","last_fullsized":"2024-09-10T19:12:18.444398+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:11:38.102803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:11:38.102803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:12:52.451650+0000","scrub_duration":0.001577272,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.8","version":"91'1784","reported_seq":1797,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-10T19:12:29.597215+0000","last_change":"2024-09-10T19:12:11.031707+0000","last_active":"2024-09-10T19:12:29.597215+0000","last_peered":"2024-09-10T19:12:29.597215+0000","last_clean":"2024-09-10T19:12:29.597215+0000","last_became_active":"2024-09-10T19:12:11.031441+0000","last_became_peered":"2024-09-10T19:12:11.031441+0000","last_unstale":"2024-09-10T19:12:29.597215+0000","last_undegraded":"2024-09-10T19:12:29.597215+0000","last_fullsized":"2024-09-10T19:12:29.597215+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:12:10.007465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:12:10.007465+0000","last_clean_scrub_stamp":"2024-09-10T19:12:10.007465+0000","objects_scrubbed":0,"log_size":1784,"log_dups_size":0,"ondisk_log_size":1784,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:13:19.074522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":116916224,"num_objects":1784,"num_object_clones":0,"num_object_copies":5352,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1784,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1784,"num_write_kb":114176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.b","version":"66'9","reported_seq":65,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-10T19:12:18.122831+0000","last_change":"2024-09-10T19:11:22.870432+0000","last_active":"2024-09-10T19:12:18.122831+0000","last_peered":"2024-09-10T19:12:18.122831+0000","last_clean":"2024-09-10T19:12:18.122831+0000","last_became_active":"2024-09-10T19:10:09.689117+0000","last_became_peered":"2024-09-10T19:10:09.689117+0000","last_unstale":"2024-09-10T19:12:18.122831+0000","last_undegraded":"2024-09-10T19:12:18.122831+0000","last_fullsized":"2024-09-10T19:12:18.122831+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'9","last_scrub_stamp":"2024-09-10T19:11:22.870369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:11:22.870369+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:12:27.485364+0000","scrub_duration":0.0017468729999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":50,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-10T19:12:18.204120+0000","last_change":"2024-09-10T19:11:44.254567+0000","last_active":"2024-09-10T19:12:18.204120+0000","last_peered":"2024-09-10T19:12:18.204120+0000","last_clean":"2024-09-10T19:12:18.204120+0000","last_became_active":"2024-09-10T19:10:07.682834+0000","last_became_peered":"2024-09-10T19:10:07.682834+0000","last_unstale":"2024-09-10T19:12:18.204120+0000","last_undegraded":"2024-09-10T19:12:18.204120+0000","last_fullsized":"2024-09-10T19:12:18.204120+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:11:44.254513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:11:44.254513+0000","last_clean_scrub_stamp":"2024-09-10T19:11:44.254513+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:13:01.679588+0000","scrub_duration":0.0013179349999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-10T19:12:17.047747+0000","last_change":"2024-09-10T19:11:42.480542+0000","last_active":"2024-09-10T19:12:17.047747+0000","last_peered":"2024-09-10T19:12:17.047747+0000","last_clean":"2024-09-10T19:12:17.047747+0000","last_became_active":"2024-09-10T19:10:11.694405+0000","last_became_peered":"2024-09-10T19:10:11.694405+0000","last_unstale":"2024-09-10T19:12:17.047747+0000","last_undegraded":"2024-09-10T19:12:17.047747+0000","last_fullsized":"2024-09-10T19:12:17.047747+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:11:42.480487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:11:42.480487+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:13:07.856129+0000","scrub_duration":0.0012878259999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":45,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-10T19:12:18.122675+0000","last_change":"2024-09-10T19:11:17.974993+0000","last_active":"2024-09-10T19:12:18.122675+0000","last_peered":"2024-09-10T19:12:18.122675+0000","last_clean":"2024-09-10T19:12:18.122675+0000","last_became_active":"2024-09-10T19:10:13.714952+0000","last_became_peered":"2024-09-10T19:10:13.714952+0000","last_unstale":"2024-09-10T19:12:18.122675+0000","last_undegraded":"2024-09-10T19:12:18.122675+0000","last_fullsized":"2024-09-10T19:12:18.122675+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:11:17.974937+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:11:17.974937+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:12:32.960908+0000","scrub_duration":0.001345834,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.b","version":"91'1618","reported_seq":1631,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-10T19:12:29.598683+0000","last_change":"2024-09-10T19:12:11.030218+0000","last_active":"2024-09-10T19:12:29.598683+0000","last_peered":"2024-09-10T19:12:29.598683+0000","last_clean":"2024-09-10T19:12:29.598683+0000","last_became_active":"2024-09-10T19:12:11.029910+0000","last_became_peered":"2024-09-10T19:12:11.029910+0000","last_unstale":"2024-09-10T19:12:29.598683+0000","last_undegraded":"2024-09-10T19:12:29.598683+0000","last_fullsized":"2024-09-10T19:12:29.598683+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:12:10.007465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:12:10.007465+0000","last_clean_scrub_stamp":"2024-09-10T19:12:10.007465+0000","objects_scrubbed":0,"log_size":1618,"log_dups_size":0,"ondisk_log_size":1618,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:13:29.534851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":106037248,"num_objects":1618,"num_object_clones":0,"num_object_copies":4854,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1618,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1618,"num_write_kb":103552,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.8","version":"66'15","reported_seq":74,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-10T19:12:17.046805+0000","last_change":"2024-09-10T19:11:33.481823+0000","last_active":"2024-09-10T19:12:17.046805+0000","last_peered":"2024-09-10T19:12:17.046805+0000","last_clean":"2024-09-10T19:12:17.046805+0000","last_became_active":"2024-09-10T19:10:09.820950+0000","last_became_peered":"2024-09-10T19:10:09.820950+0000","last_unstale":"2024-09-10T19:12:17.046805+0000","last_undegraded":"2024-09-10T19:12:17.046805+0000","last_fullsized":"2024-09-10T19:12:17.046805+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'15","last_scrub_stamp":"2024-09-10T19:11:33.481743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:11:33.481743+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:12:42.712551+0000","scrub_duration":0.002098094,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":51,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-10T19:12:17.047966+0000","last_change":"2024-09-10T19:11:15.197602+0000","last_active":"2024-09-10T19:12:17.047966+0000","last_peered":"2024-09-10T19:12:17.047966+0000","last_clean":"2024-09-10T19:12:17.047966+0000","last_became_active":"2024-09-10T19:10:07.682286+0000","last_became_peered":"2024-09-10T19:10:07.682286+0000","last_unstale":"2024-09-10T19:12:17.047966+0000","last_undegraded":"2024-09-10T19:12:17.047966+0000","last_fullsized":"2024-09-10T19:12:17.047966+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:11:15.197497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:11:15.197497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0019717929999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.9","version":"80'9","reported_seq":84,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-10T19:12:25.200219+0000","last_change":"2024-09-10T19:11:35.174273+0000","last_active":"2024-09-10T19:12:25.200219+0000","last_peered":"2024-09-10T19:12:25.200219+0000","last_clean":"2024-09-10T19:12:25.200219+0000","last_became_active":"2024-09-10T19:10:11.702878+0000","last_became_peered":"2024-09-10T19:10:11.702878+0000","last_unstale":"2024-09-10T19:12:25.200219+0000","last_undegraded":"2024-09-10T19:12:25.200219+0000","last_fullsized":"2024-09-10T19:12:25.200219+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"80'9","last_scrub_stamp":"2024-09-10T19:11:35.174218+0000","last_deep_scrub":"80'9","last_deep_scrub_stamp":"2024-09-10T19:11:35.174218+0000","last_clean_scrub_stamp":"2024-09-10T19:11:35.174218+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:13:04.007735+0000","scrub_duration":0.0019812380000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":45,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-10T19:12:17.046725+0000","last_change":"2024-09-10T19:11:20.184218+0000","last_active":"2024-09-10T19:12:17.046725+0000","last_peered":"2024-09-10T19:12:17.046725+0000","last_clean":"2024-09-10T19:12:17.046725+0000","last_became_active":"2024-09-10T19:10:14.053875+0000","last_became_peered":"2024-09-10T19:10:14.053875+0000","last_unstale":"2024-09-10T19:12:17.046725+0000","last_undegraded":"2024-09-10T19:12:17.046725+0000","last_fullsized":"2024-09-10T19:12:17.046725+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:11:20.184132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:11:20.184132+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:12:50.058323+0000","scrub_duration":0.002119522,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.a","version":"91'1729","reported_seq":1742,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-10T19:12:29.594685+0000","last_change":"2024-09-10T19:12:11.035541+0000","last_active":"2024-09-10T19:12:29.594685+0000","last_peered":"2024-09-10T19:12:29.594685+0000","last_clean":"2024-09-10T19:12:29.594685+0000","last_became_active":"2024-09-10T19:12:11.034838+0000","last_became_peered":"2024-09-10T19:12:11.034838+0000","last_unstale":"2024-09-10T19:12:29.594685+0000","last_undegraded":"2024-09-10T19:12:29.594685+0000","last_fullsized":"2024-09-10T19:12:29.594685+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:12:10.007465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:12:10.007465+0000","last_clean_scrub_stamp":"2024-09-10T19:12:10.007465+0000","objects_scrubbed":0,"log_size":1729,"log_dups_size":0,"ondisk_log_size":1729,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:13:16.711659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":113311744,"num_objects":1729,"num_object_clones":0,"num_object_copies":5187,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1729,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1729,"num_write_kb":110656,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.9","version":"66'12","reported_seq":71,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-10T19:12:18.293898+0000","last_change":"2024-09-10T19:11:45.012450+0000","last_active":"2024-09-10T19:12:18.293898+0000","last_peered":"2024-09-10T19:12:18.293898+0000","last_clean":"2024-09-10T19:12:18.293898+0000","last_became_active":"2024-09-10T19:10:09.692627+0000","last_became_peered":"2024-09-10T19:10:09.692627+0000","last_unstale":"2024-09-10T19:12:18.293898+0000","last_undegraded":"2024-09-10T19:12:18.293898+0000","last_fullsized":"2024-09-10T19:12:18.293898+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'12","last_scrub_stamp":"2024-09-10T19:11:45.012377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:11:45.012377+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:13:02.868370+0000","scrub_duration":0.0019105979999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.f","version":"59'2","reported_seq":66,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-10T19:12:18.294999+0000","last_change":"2024-09-10T19:11:25.111486+0000","last_active":"2024-09-10T19:12:18.294999+0000","last_peered":"2024-09-10T19:12:18.294999+0000","last_clean":"2024-09-10T19:12:18.294999+0000","last_became_active":"2024-09-10T19:10:07.681719+0000","last_became_peered":"2024-09-10T19:10:07.681719+0000","last_unstale":"2024-09-10T19:12:18.294999+0000","last_undegraded":"2024-09-10T19:12:18.294999+0000","last_fullsized":"2024-09-10T19:12:18.294999+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'2","last_scrub_stamp":"2024-09-10T19:11:25.111429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:11:25.111429+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:12:31.423912+0000","scrub_duration":0.0014791139999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":14,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-10T19:12:17.047878+0000","last_change":"2024-09-10T19:11:46.587917+0000","last_active":"2024-09-10T19:12:17.047878+0000","last_peered":"2024-09-10T19:12:17.047878+0000","last_clean":"2024-09-10T19:12:17.047878+0000","last_became_active":"2024-09-10T19:10:11.698354+0000","last_became_peered":"2024-09-10T19:10:11.698354+0000","last_unstale":"2024-09-10T19:12:17.047878+0000","last_undegraded":"2024-09-10T19:12:17.047878+0000","last_fullsized":"2024-09-10T19:12:17.047878+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:11:46.587846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:11:46.587846+0000","last_clean_scrub_stamp":"2024-09-10T19:11:46.587846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:12:49.697286+0000","scrub_duration":0.001267142,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":45,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-10T19:12:17.048853+0000","last_change":"2024-09-10T19:11:42.723535+0000","last_active":"2024-09-10T19:12:17.048853+0000","last_peered":"2024-09-10T19:12:17.048853+0000","last_clean":"2024-09-10T19:12:17.048853+0000","last_became_active":"2024-09-10T19:10:13.718962+0000","last_became_peered":"2024-09-10T19:10:13.718962+0000","last_unstale":"2024-09-10T19:12:17.048853+0000","last_undegraded":"2024-09-10T19:12:17.048853+0000","last_fullsized":"2024-09-10T19:12:17.048853+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:11:42.723437+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:11:42.723437+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:13:12.077483+0000","scrub_duration":0.001385631,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.16","version":"66'9","reported_seq":65,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-10T19:12:18.122346+0000","last_change":"2024-09-10T19:11:11.962017+0000","last_active":"2024-09-10T19:12:18.122346+0000","last_peered":"2024-09-10T19:12:18.122346+0000","last_clean":"2024-09-10T19:12:18.122346+0000","last_became_active":"2024-09-10T19:10:09.822994+0000","last_became_peered":"2024-09-10T19:10:09.822994+0000","last_unstale":"2024-09-10T19:12:18.122346+0000","last_undegraded":"2024-09-10T19:12:18.122346+0000","last_fullsized":"2024-09-10T19:12:18.122346+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'9","last_scrub_stamp":"2024-09-10T19:11:11.961960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:11:11.961960+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001903287,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":51,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-10T19:12:17.048017+0000","last_change":"2024-09-10T19:11:41.458577+0000","last_active":"2024-09-10T19:12:17.048017+0000","last_peered":"2024-09-10T19:12:17.048017+0000","last_clean":"2024-09-10T19:12:17.048017+0000","last_became_active":"2024-09-10T19:10:07.678076+0000","last_became_peered":"2024-09-10T19:10:07.678076+0000","last_unstale":"2024-09-10T19:12:17.048017+0000","last_undegraded":"2024-09-10T19:12:17.048017+0000","last_fullsized":"2024-09-10T19:12:17.048017+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:11:41.458523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:11:41.458523+0000","last_clean_scrub_stamp":"2024-09-10T19:11:41.458523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:12:47.189060+0000","scrub_duration":0.0012499830000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-10T19:12:17.049399+0000","last_change":"2024-09-10T19:12:11.619262+0000","last_active":"2024-09-10T19:12:17.049399+0000","last_peered":"2024-09-10T19:12:17.049399+0000","last_clean":"2024-09-10T19:12:17.049399+0000","last_became_active":"2024-09-10T19:10:11.693407+0000","last_became_peered":"2024-09-10T19:10:11.693407+0000","last_unstale":"2024-09-10T19:12:17.049399+0000","last_undegraded":"2024-09-10T19:12:17.049399+0000","last_fullsized":"2024-09-10T19:12:17.049399+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:12:11.619202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:12:11.619202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:13:33.915758+0000","scrub_duration":0.001950834,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"66'1","reported_seq":46,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-10T19:12:17.048064+0000","last_change":"2024-09-10T19:11:28.192794+0000","last_active":"2024-09-10T19:12:17.048064+0000","last_peered":"2024-09-10T19:12:17.048064+0000","last_clean":"2024-09-10T19:12:17.048064+0000","last_became_active":"2024-09-10T19:10:13.717001+0000","last_became_peered":"2024-09-10T19:10:13.717001+0000","last_unstale":"2024-09-10T19:12:17.048064+0000","last_undegraded":"2024-09-10T19:12:17.048064+0000","last_fullsized":"2024-09-10T19:12:17.048064+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'1","last_scrub_stamp":"2024-09-10T19:11:28.192745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:11:28.192745+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:12:37.727108+0000","scrub_duration":0.001276323,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":51,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-10T19:12:17.960677+0000","last_change":"2024-09-10T19:11:32.859175+0000","last_active":"2024-09-10T19:12:17.960677+0000","last_peered":"2024-09-10T19:12:17.960677+0000","last_clean":"2024-09-10T19:12:17.960677+0000","last_became_active":"2024-09-10T19:10:07.682963+0000","last_became_peered":"2024-09-10T19:10:07.682963+0000","last_unstale":"2024-09-10T19:12:17.960677+0000","last_undegraded":"2024-09-10T19:12:17.960677+0000","last_fullsized":"2024-09-10T19:12:17.960677+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:11:32.859106+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:11:32.859106+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:12:47.279562+0000","scrub_duration":0.0014403980000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"66'6","reported_seq":48,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-10T19:12:17.049468+0000","last_change":"2024-09-10T19:10:09.822283+0000","last_active":"2024-09-10T19:12:17.049468+0000","last_peered":"2024-09-10T19:12:17.049468+0000","last_clean":"2024-09-10T19:12:17.049468+0000","last_became_active":"2024-09-10T19:10:09.821906+0000","last_became_peered":"2024-09-10T19:10:09.821906+0000","last_unstale":"2024-09-10T19:12:17.049468+0000","last_undegraded":"2024-09-10T19:12:17.049468+0000","last_fullsized":"2024-09-10T19:12:17.049468+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:10:08.671214+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-10T19:12:17.047499+0000","last_change":"2024-09-10T19:11:23.264870+0000","last_active":"2024-09-10T19:12:17.047499+0000","last_peered":"2024-09-10T19:12:17.047499+0000","last_clean":"2024-09-10T19:12:17.047499+0000","last_became_active":"2024-09-10T19:10:11.692943+0000","last_became_peered":"2024-09-10T19:10:11.692943+0000","last_unstale":"2024-09-10T19:12:17.047499+0000","last_undegraded":"2024-09-10T19:12:17.047499+0000","last_fullsized":"2024-09-10T19:12:17.047499+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:11:23.264793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:11:23.264793+0000","last_clean_scrub_stamp":"2024-09-10T19:11:23.264793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:12:44.248872+0000","scrub_duration":0.0015950649999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":45,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-10T19:12:18.444494+0000","last_change":"2024-09-10T19:11:13.342070+0000","last_active":"2024-09-10T19:12:18.444494+0000","last_peered":"2024-09-10T19:12:18.444494+0000","last_clean":"2024-09-10T19:12:18.444494+0000","last_became_active":"2024-09-10T19:10:14.054360+0000","last_became_peered":"2024-09-10T19:10:14.054360+0000","last_unstale":"2024-09-10T19:12:18.444494+0000","last_undegraded":"2024-09-10T19:12:18.444494+0000","last_fullsized":"2024-09-10T19:12:18.444494+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:11:13.341970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:11:13.341970+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:12:30.833886+0000","scrub_duration":0.001537725,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"80'15","reported_seq":69,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-10T19:12:17.049564+0000","last_change":"2024-09-10T19:11:41.719367+0000","last_active":"2024-09-10T19:12:17.049564+0000","last_peered":"2024-09-10T19:12:17.049564+0000","last_clean":"2024-09-10T19:12:17.049564+0000","last_became_active":"2024-09-10T19:10:09.821810+0000","last_became_peered":"2024-09-10T19:10:09.821810+0000","last_unstale":"2024-09-10T19:12:17.049564+0000","last_undegraded":"2024-09-10T19:12:17.049564+0000","last_fullsized":"2024-09-10T19:12:17.049564+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"80'15","last_scrub_stamp":"2024-09-10T19:11:41.719310+0000","last_deep_scrub":"80'15","last_deep_scrub_stamp":"2024-09-10T19:11:41.719310+0000","last_clean_scrub_stamp":"2024-09-10T19:11:41.719310+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:12:47.895535+0000","scrub_duration":0.0046706430000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":51,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-10T19:12:17.046976+0000","last_change":"2024-09-10T19:11:12.081772+0000","last_active":"2024-09-10T19:12:17.046976+0000","last_peered":"2024-09-10T19:12:17.046976+0000","last_clean":"2024-09-10T19:12:17.046976+0000","last_became_active":"2024-09-10T19:10:07.685870+0000","last_became_peered":"2024-09-10T19:10:07.685870+0000","last_unstale":"2024-09-10T19:12:17.046976+0000","last_undegraded":"2024-09-10T19:12:17.046976+0000","last_fullsized":"2024-09-10T19:12:17.046976+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:11:12.081691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:11:12.081691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:12:41.469111+0000","scrub_duration":0.001629903,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"66'8","reported_seq":83,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-10T19:12:25.197740+0000","last_change":"2024-09-10T19:11:50.330376+0000","last_active":"2024-09-10T19:12:25.197740+0000","last_peered":"2024-09-10T19:12:25.197740+0000","last_clean":"2024-09-10T19:12:25.197740+0000","last_became_active":"2024-09-10T19:10:11.700584+0000","last_became_peered":"2024-09-10T19:10:11.700584+0000","last_unstale":"2024-09-10T19:12:25.197740+0000","last_undegraded":"2024-09-10T19:12:25.197740+0000","last_fullsized":"2024-09-10T19:12:25.197740+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'8","last_scrub_stamp":"2024-09-10T19:11:50.330303+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:11:50.330303+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:13:18.803273+0000","scrub_duration":0.0020032069999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":45,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-10T19:12:18.122963+0000","last_change":"2024-09-10T19:11:20.881495+0000","last_active":"2024-09-10T19:12:18.122963+0000","last_peered":"2024-09-10T19:12:18.122963+0000","last_clean":"2024-09-10T19:12:18.122963+0000","last_became_active":"2024-09-10T19:10:13.721035+0000","last_became_peered":"2024-09-10T19:10:13.721035+0000","last_unstale":"2024-09-10T19:12:18.122963+0000","last_undegraded":"2024-09-10T19:12:18.122963+0000","last_fullsized":"2024-09-10T19:12:18.122963+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:11:20.881436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:11:20.881436+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:12:49.668407+0000","scrub_duration":0.0014551099999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"66'9","reported_seq":65,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-10T19:12:17.047383+0000","last_change":"2024-09-10T19:11:37.394628+0000","last_active":"2024-09-10T19:12:17.047383+0000","last_peered":"2024-09-10T19:12:17.047383+0000","last_clean":"2024-09-10T19:12:17.047383+0000","last_became_active":"2024-09-10T19:10:09.823006+0000","last_became_peered":"2024-09-10T19:10:09.823006+0000","last_unstale":"2024-09-10T19:12:17.047383+0000","last_undegraded":"2024-09-10T19:12:17.047383+0000","last_fullsized":"2024-09-10T19:12:17.047383+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'9","last_scrub_stamp":"2024-09-10T19:11:37.394573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:11:37.394573+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:12:49.673447+0000","scrub_duration":0.00176055,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":51,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-10T19:12:18.122914+0000","last_change":"2024-09-10T19:11:29.796484+0000","last_active":"2024-09-10T19:12:18.122914+0000","last_peered":"2024-09-10T19:12:18.122914+0000","last_clean":"2024-09-10T19:12:18.122914+0000","last_became_active":"2024-09-10T19:10:07.688499+0000","last_became_peered":"2024-09-10T19:10:07.688499+0000","last_unstale":"2024-09-10T19:12:18.122914+0000","last_undegraded":"2024-09-10T19:12:18.122914+0000","last_fullsized":"2024-09-10T19:12:18.122914+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:11:29.796429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:11:29.796429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:12:53.847123+0000","scrub_duration":0.0012420479999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"66'8","reported_seq":83,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-10T19:12:25.199184+0000","last_change":"2024-09-10T19:11:23.870113+0000","last_active":"2024-09-10T19:12:25.199184+0000","last_peered":"2024-09-10T19:12:25.199184+0000","last_clean":"2024-09-10T19:12:25.199184+0000","last_became_active":"2024-09-10T19:10:11.692359+0000","last_became_peered":"2024-09-10T19:10:11.692359+0000","last_unstale":"2024-09-10T19:12:25.199184+0000","last_undegraded":"2024-09-10T19:12:25.199184+0000","last_fullsized":"2024-09-10T19:12:25.199184+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'8","last_scrub_stamp":"2024-09-10T19:11:23.870010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:11:23.870010+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:12:47.854105+0000","scrub_duration":0.0017621309999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":44,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-10T19:12:18.294857+0000","last_change":"2024-09-10T19:11:43.071796+0000","last_active":"2024-09-10T19:12:18.294857+0000","last_peered":"2024-09-10T19:12:18.294857+0000","last_clean":"2024-09-10T19:12:18.294857+0000","last_became_active":"2024-09-10T19:10:13.718285+0000","last_became_peered":"2024-09-10T19:10:13.718285+0000","last_unstale":"2024-09-10T19:12:18.294857+0000","last_undegraded":"2024-09-10T19:12:18.294857+0000","last_fullsized":"2024-09-10T19:12:18.294857+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:11:43.071727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:11:43.071727+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:13:09.326836+0000","scrub_duration":0.00142789,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.12","version":"79'13","reported_seq":68,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-10T19:12:18.203124+0000","last_change":"2024-09-10T19:11:22.072464+0000","last_active":"2024-09-10T19:12:18.203124+0000","last_peered":"2024-09-10T19:12:18.203124+0000","last_clean":"2024-09-10T19:12:18.203124+0000","last_became_active":"2024-09-10T19:10:09.689440+0000","last_became_peered":"2024-09-10T19:10:09.689440+0000","last_unstale":"2024-09-10T19:12:18.203124+0000","last_undegraded":"2024-09-10T19:12:18.203124+0000","last_fullsized":"2024-09-10T19:12:18.203124+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'13","last_scrub_stamp":"2024-09-10T19:11:22.072400+0000","last_deep_scrub":"79'13","last_deep_scrub_stamp":"2024-09-10T19:11:22.072400+0000","last_clean_scrub_stamp":"2024-09-10T19:11:22.072400+0000","objects_scrubbed":4,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:12:33.209191+0000","scrub_duration":0.0032983940000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"59'1","reported_seq":58,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-10T19:12:17.961015+0000","last_change":"2024-09-10T19:11:47.926583+0000","last_active":"2024-09-10T19:12:17.961015+0000","last_peered":"2024-09-10T19:12:17.961015+0000","last_clean":"2024-09-10T19:12:17.961015+0000","last_became_active":"2024-09-10T19:10:07.683263+0000","last_became_peered":"2024-09-10T19:10:07.683263+0000","last_unstale":"2024-09-10T19:12:17.961015+0000","last_undegraded":"2024-09-10T19:12:17.961015+0000","last_fullsized":"2024-09-10T19:12:17.961015+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'1","last_scrub_stamp":"2024-09-10T19:11:47.926513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:11:47.926513+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:13:03.515009+0000","scrub_duration":0.001771999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":362,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-10T19:12:17.048919+0000","last_change":"2024-09-10T19:12:00.660341+0000","last_active":"2024-09-10T19:12:17.048919+0000","last_peered":"2024-09-10T19:12:17.048919+0000","last_clean":"2024-09-10T19:12:17.048919+0000","last_became_active":"2024-09-10T19:10:11.701416+0000","last_became_peered":"2024-09-10T19:10:11.701416+0000","last_unstale":"2024-09-10T19:12:17.048919+0000","last_undegraded":"2024-09-10T19:12:17.048919+0000","last_fullsized":"2024-09-10T19:12:17.048919+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:12:00.660284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:12:00.660284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:13:21.090430+0000","scrub_duration":0.0012510729999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":45,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-10T19:12:18.122416+0000","last_change":"2024-09-10T19:11:51.920101+0000","last_active":"2024-09-10T19:12:18.122416+0000","last_peered":"2024-09-10T19:12:18.122416+0000","last_clean":"2024-09-10T19:12:18.122416+0000","last_became_active":"2024-09-10T19:10:13.716198+0000","last_became_peered":"2024-09-10T19:10:13.716198+0000","last_unstale":"2024-09-10T19:12:18.122416+0000","last_undegraded":"2024-09-10T19:12:18.122416+0000","last_fullsized":"2024-09-10T19:12:18.122416+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:11:51.920006+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:11:51.920006+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:13:19.065723+0000","scrub_duration":0.001865058,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"66'11","reported_seq":67,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-10T19:12:18.294318+0000","last_change":"2024-09-10T19:11:44.025426+0000","last_active":"2024-09-10T19:12:18.294318+0000","last_peered":"2024-09-10T19:12:18.294318+0000","last_clean":"2024-09-10T19:12:18.294318+0000","last_became_active":"2024-09-10T19:10:09.693856+0000","last_became_peered":"2024-09-10T19:10:09.693856+0000","last_unstale":"2024-09-10T19:12:18.294318+0000","last_undegraded":"2024-09-10T19:12:18.294318+0000","last_fullsized":"2024-09-10T19:12:18.294318+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'11","last_scrub_stamp":"2024-09-10T19:11:44.025361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:11:44.025361+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:13:09.366530+0000","scrub_duration":0.002036894,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":50,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-10T19:12:18.204092+0000","last_change":"2024-09-10T19:11:10.031299+0000","last_active":"2024-09-10T19:12:18.204092+0000","last_peered":"2024-09-10T19:12:18.204092+0000","last_clean":"2024-09-10T19:12:18.204092+0000","last_became_active":"2024-09-10T19:10:07.685339+0000","last_became_peered":"2024-09-10T19:10:07.685339+0000","last_unstale":"2024-09-10T19:12:18.204092+0000","last_undegraded":"2024-09-10T19:12:18.204092+0000","last_fullsized":"2024-09-10T19:12:18.204092+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:11:10.031245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:11:10.031245+0000","last_clean_scrub_stamp":"2024-09-10T19:11:10.031245+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:12:27.675011+0000","scrub_duration":0.0014113439999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":46,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-10T19:12:18.203657+0000","last_change":"2024-09-10T19:11:31.249400+0000","last_active":"2024-09-10T19:12:18.203657+0000","last_peered":"2024-09-10T19:12:18.203657+0000","last_clean":"2024-09-10T19:12:18.203657+0000","last_became_active":"2024-09-10T19:10:11.694564+0000","last_became_peered":"2024-09-10T19:10:11.694564+0000","last_unstale":"2024-09-10T19:12:18.203657+0000","last_undegraded":"2024-09-10T19:12:18.203657+0000","last_fullsized":"2024-09-10T19:12:18.203657+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:11:31.249343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:11:31.249343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:12:58.372067+0000","scrub_duration":0.001093644,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":45,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-10T19:12:17.048290+0000","last_change":"2024-09-10T19:11:16.813629+0000","last_active":"2024-09-10T19:12:17.048290+0000","last_peered":"2024-09-10T19:12:17.048290+0000","last_clean":"2024-09-10T19:12:17.048290+0000","last_became_active":"2024-09-10T19:10:13.724201+0000","last_became_peered":"2024-09-10T19:10:13.724201+0000","last_unstale":"2024-09-10T19:12:17.048290+0000","last_undegraded":"2024-09-10T19:12:17.048290+0000","last_fullsized":"2024-09-10T19:12:17.048290+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:11:16.813575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:11:16.813575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:12:44.427906+0000","scrub_duration":0.001237312,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"66'4","reported_seq":55,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-10T19:12:17.047290+0000","last_change":"2024-09-10T19:11:14.816693+0000","last_active":"2024-09-10T19:12:17.047290+0000","last_peered":"2024-09-10T19:12:17.047290+0000","last_clean":"2024-09-10T19:12:17.047290+0000","last_became_active":"2024-09-10T19:10:09.692751+0000","last_became_peered":"2024-09-10T19:10:09.692751+0000","last_unstale":"2024-09-10T19:12:17.047290+0000","last_undegraded":"2024-09-10T19:12:17.047290+0000","last_fullsized":"2024-09-10T19:12:17.047290+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'4","last_scrub_stamp":"2024-09-10T19:11:14.816632+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:11:14.816632+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:12:42.695105+0000","scrub_duration":0.001902504,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":51,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-10T19:12:17.047575+0000","last_change":"2024-09-10T19:11:24.275061+0000","last_active":"2024-09-10T19:12:17.047575+0000","last_peered":"2024-09-10T19:12:17.047575+0000","last_clean":"2024-09-10T19:12:17.047575+0000","last_became_active":"2024-09-10T19:10:07.687723+0000","last_became_peered":"2024-09-10T19:10:07.687723+0000","last_unstale":"2024-09-10T19:12:17.047575+0000","last_undegraded":"2024-09-10T19:12:17.047575+0000","last_fullsized":"2024-09-10T19:12:17.047575+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:11:24.275004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:11:24.275004+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:12:41.723440+0000","scrub_duration":0.001588014,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-10T19:12:17.960511+0000","last_change":"2024-09-10T19:11:31.838005+0000","last_active":"2024-09-10T19:12:17.960511+0000","last_peered":"2024-09-10T19:12:17.960511+0000","last_clean":"2024-09-10T19:12:17.960511+0000","last_became_active":"2024-09-10T19:10:11.698536+0000","last_became_peered":"2024-09-10T19:10:11.698536+0000","last_unstale":"2024-09-10T19:12:17.960511+0000","last_undegraded":"2024-09-10T19:12:17.960511+0000","last_fullsized":"2024-09-10T19:12:17.960511+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:11:31.837944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:11:31.837944+0000","last_clean_scrub_stamp":"2024-09-10T19:11:31.837944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:12:53.987457+0000","scrub_duration":0.001263639,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":45,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-10T19:12:18.444208+0000","last_change":"2024-09-10T19:11:14.298616+0000","last_active":"2024-09-10T19:12:18.444208+0000","last_peered":"2024-09-10T19:12:18.444208+0000","last_clean":"2024-09-10T19:12:18.444208+0000","last_became_active":"2024-09-10T19:10:13.718271+0000","last_became_peered":"2024-09-10T19:10:13.718271+0000","last_unstale":"2024-09-10T19:12:18.444208+0000","last_undegraded":"2024-09-10T19:12:18.444208+0000","last_fullsized":"2024-09-10T19:12:18.444208+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:11:14.298545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:11:14.298545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001561799,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.11","version":"66'11","reported_seq":68,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-10T19:12:17.048477+0000","last_change":"2024-09-10T19:11:36.743509+0000","last_active":"2024-09-10T19:12:17.048477+0000","last_peered":"2024-09-10T19:12:17.048477+0000","last_clean":"2024-09-10T19:12:17.048477+0000","last_became_active":"2024-09-10T19:10:09.821357+0000","last_became_peered":"2024-09-10T19:10:09.821357+0000","last_unstale":"2024-09-10T19:12:17.048477+0000","last_undegraded":"2024-09-10T19:12:17.048477+0000","last_fullsized":"2024-09-10T19:12:17.048477+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'11","last_scrub_stamp":"2024-09-10T19:11:36.743445+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:11:36.743445+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:13:04.557447+0000","scrub_duration":0.001722374,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":51,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-10T19:12:17.961443+0000","last_change":"2024-09-10T19:11:34.867991+0000","last_active":"2024-09-10T19:12:17.961443+0000","last_peered":"2024-09-10T19:12:17.961443+0000","last_clean":"2024-09-10T19:12:17.961443+0000","last_became_active":"2024-09-10T19:10:07.685223+0000","last_became_peered":"2024-09-10T19:10:07.685223+0000","last_unstale":"2024-09-10T19:12:17.961443+0000","last_undegraded":"2024-09-10T19:12:17.961443+0000","last_fullsized":"2024-09-10T19:12:17.961443+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:11:34.867935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:11:34.867935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:12:56.224806+0000","scrub_duration":0.001504086,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-10T19:12:18.444094+0000","last_change":"2024-09-10T19:11:57.112153+0000","last_active":"2024-09-10T19:12:18.444094+0000","last_peered":"2024-09-10T19:12:18.444094+0000","last_clean":"2024-09-10T19:12:18.444094+0000","last_became_active":"2024-09-10T19:10:11.699013+0000","last_became_peered":"2024-09-10T19:10:11.699013+0000","last_unstale":"2024-09-10T19:12:18.444094+0000","last_undegraded":"2024-09-10T19:12:18.444094+0000","last_fullsized":"2024-09-10T19:12:18.444094+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:11:57.112092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:11:57.112092+0000","last_clean_scrub_stamp":"2024-09-10T19:11:57.112092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:13:02.946526+0000","scrub_duration":0.001443653,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":45,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-10T19:12:17.048756+0000","last_change":"2024-09-10T19:11:46.720343+0000","last_active":"2024-09-10T19:12:17.048756+0000","last_peered":"2024-09-10T19:12:17.048756+0000","last_clean":"2024-09-10T19:12:17.048756+0000","last_became_active":"2024-09-10T19:10:14.056172+0000","last_became_peered":"2024-09-10T19:10:14.056172+0000","last_unstale":"2024-09-10T19:12:17.048756+0000","last_undegraded":"2024-09-10T19:12:17.048756+0000","last_fullsized":"2024-09-10T19:12:17.048756+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:11:46.720280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:11:46.720280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:13:09.566661+0000","scrub_duration":0.0014580260000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"80'2","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-10T19:12:18.444464+0000","last_change":"2024-09-10T19:11:30.272094+0000","last_active":"2024-09-10T19:12:18.444464+0000","last_peered":"2024-09-10T19:12:18.444464+0000","last_clean":"2024-09-10T19:12:18.444464+0000","last_became_active":"2024-09-10T19:10:13.718875+0000","last_became_peered":"2024-09-10T19:10:13.718875+0000","last_unstale":"2024-09-10T19:12:18.444464+0000","last_undegraded":"2024-09-10T19:12:18.444464+0000","last_fullsized":"2024-09-10T19:12:18.444464+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"80'2","last_scrub_stamp":"2024-09-10T19:11:30.272043+0000","last_deep_scrub":"80'2","last_deep_scrub_stamp":"2024-09-10T19:11:30.272043+0000","last_clean_scrub_stamp":"2024-09-10T19:11:30.272043+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:12:37.490688+0000","scrub_duration":0.0022929460000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":369,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":51,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-10T19:12:17.046850+0000","last_change":"2024-09-10T19:11:49.356398+0000","last_active":"2024-09-10T19:12:17.046850+0000","last_peered":"2024-09-10T19:12:17.046850+0000","last_clean":"2024-09-10T19:12:17.046850+0000","last_became_active":"2024-09-10T19:10:07.689877+0000","last_became_peered":"2024-09-10T19:10:07.689877+0000","last_unstale":"2024-09-10T19:12:17.046850+0000","last_undegraded":"2024-09-10T19:12:17.046850+0000","last_fullsized":"2024-09-10T19:12:17.046850+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:11:49.356336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:11:49.356336+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:13:14.019219+0000","scrub_duration":0.001371527,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1e","version":"80'15","reported_seq":69,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-10T19:12:18.122444+0000","last_change":"2024-09-10T19:11:25.884008+0000","last_active":"2024-09-10T19:12:18.122444+0000","last_peered":"2024-09-10T19:12:18.122444+0000","last_clean":"2024-09-10T19:12:18.122444+0000","last_became_active":"2024-09-10T19:10:09.823061+0000","last_became_peered":"2024-09-10T19:10:09.823061+0000","last_unstale":"2024-09-10T19:12:18.122444+0000","last_undegraded":"2024-09-10T19:12:18.122444+0000","last_fullsized":"2024-09-10T19:12:18.122444+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"80'15","last_scrub_stamp":"2024-09-10T19:11:25.883919+0000","last_deep_scrub":"80'15","last_deep_scrub_stamp":"2024-09-10T19:11:25.883919+0000","last_clean_scrub_stamp":"2024-09-10T19:11:25.883919+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:12:31.561992+0000","scrub_duration":0.0034715229999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1f","version":"66'8","reported_seq":86,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-09-10T19:12:25.195402+0000","last_change":"2024-09-10T19:11:30.835414+0000","last_active":"2024-09-10T19:12:25.195402+0000","last_peered":"2024-09-10T19:12:25.195402+0000","last_clean":"2024-09-10T19:12:25.195402+0000","last_became_active":"2024-09-10T19:10:11.695427+0000","last_became_peered":"2024-09-10T19:10:11.695427+0000","last_unstale":"2024-09-10T19:12:25.195402+0000","last_undegraded":"2024-09-10T19:12:25.195402+0000","last_fullsized":"2024-09-10T19:12:25.195402+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'8","last_scrub_stamp":"2024-09-10T19:11:30.835354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:11:30.835354+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:12:42.690611+0000","scrub_duration":0.0013545090000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":44,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-10T19:12:18.203750+0000","last_change":"2024-09-10T19:11:30.210657+0000","last_active":"2024-09-10T19:12:18.203750+0000","last_peered":"2024-09-10T19:12:18.203750+0000","last_clean":"2024-09-10T19:12:18.203750+0000","last_became_active":"2024-09-10T19:10:13.715419+0000","last_became_peered":"2024-09-10T19:10:13.715419+0000","last_unstale":"2024-09-10T19:12:18.203750+0000","last_undegraded":"2024-09-10T19:12:18.203750+0000","last_fullsized":"2024-09-10T19:12:18.203750+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:11:30.210603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:11:30.210603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:12:40.846927+0000","scrub_duration":0.0012894740000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":51,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-10T19:12:17.048831+0000","last_change":"2024-09-10T19:11:34.815849+0000","last_active":"2024-09-10T19:12:17.048831+0000","last_peered":"2024-09-10T19:12:17.048831+0000","last_clean":"2024-09-10T19:12:17.048831+0000","last_became_active":"2024-09-10T19:10:07.682689+0000","last_became_peered":"2024-09-10T19:10:07.682689+0000","last_unstale":"2024-09-10T19:12:17.048831+0000","last_undegraded":"2024-09-10T19:12:17.048831+0000","last_fullsized":"2024-09-10T19:12:17.048831+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:11:34.815794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:11:34.815794+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:13:02.432072+0000","scrub_duration":0.001252501,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"80'16","reported_seq":95,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-10T19:12:17.961372+0000","last_change":"2024-09-10T19:11:44.755688+0000","last_active":"2024-09-10T19:12:17.961372+0000","last_peered":"2024-09-10T19:12:17.961372+0000","last_clean":"2024-09-10T19:12:17.961372+0000","last_became_active":"2024-09-10T19:11:44.750039+0000","last_became_peered":"2024-09-10T19:11:44.750039+0000","last_unstale":"2024-09-10T19:12:17.961372+0000","last_undegraded":"2024-09-10T19:12:17.961372+0000","last_fullsized":"2024-09-10T19:12:17.961372+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"80'16","last_scrub_stamp":"2024-09-10T19:11:40.853611+0000","last_deep_scrub":"80'16","last_deep_scrub_stamp":"2024-09-10T19:11:40.853611+0000","last_clean_scrub_stamp":"2024-09-10T19:11:40.853611+0000","objects_scrubbed":5,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:12:46.019580+0000","scrub_duration":0.003886877,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":47,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-10T19:12:18.123060+0000","last_change":"2024-09-10T19:11:28.822385+0000","last_active":"2024-09-10T19:12:18.123060+0000","last_peered":"2024-09-10T19:12:18.123060+0000","last_clean":"2024-09-10T19:12:18.123060+0000","last_became_active":"2024-09-10T19:10:11.701735+0000","last_became_peered":"2024-09-10T19:10:11.701735+0000","last_unstale":"2024-09-10T19:12:18.123060+0000","last_undegraded":"2024-09-10T19:12:18.123060+0000","last_fullsized":"2024-09-10T19:12:18.123060+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:11:28.822327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:11:28.822327+0000","last_clean_scrub_stamp":"2024-09-10T19:11:28.822327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:12:45.908873+0000","scrub_duration":0.0012591119999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":44,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-10T19:12:18.294790+0000","last_change":"2024-09-10T19:11:35.145006+0000","last_active":"2024-09-10T19:12:18.294790+0000","last_peered":"2024-09-10T19:12:18.294790+0000","last_clean":"2024-09-10T19:12:18.294790+0000","last_became_active":"2024-09-10T19:10:14.053619+0000","last_became_peered":"2024-09-10T19:10:14.053619+0000","last_unstale":"2024-09-10T19:12:18.294790+0000","last_undegraded":"2024-09-10T19:12:18.294790+0000","last_fullsized":"2024-09-10T19:12:18.294790+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:11:35.144953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:11:35.144953+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:12:55.849066+0000","scrub_duration":0.001302503,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":51,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-10T19:12:17.961069+0000","last_change":"2024-09-10T19:11:55.930125+0000","last_active":"2024-09-10T19:12:17.961069+0000","last_peered":"2024-09-10T19:12:17.961069+0000","last_clean":"2024-09-10T19:12:17.961069+0000","last_became_active":"2024-09-10T19:10:07.685997+0000","last_became_peered":"2024-09-10T19:10:07.685997+0000","last_unstale":"2024-09-10T19:12:17.961069+0000","last_undegraded":"2024-09-10T19:12:17.961069+0000","last_fullsized":"2024-09-10T19:12:17.961069+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:11:55.930055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:11:55.930055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:13:20.276257+0000","scrub_duration":0.0013233539999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1c","version":"78'22","reported_seq":81,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-10T19:12:17.048248+0000","last_change":"2024-09-10T19:11:21.119003+0000","last_active":"2024-09-10T19:12:17.048248+0000","last_peered":"2024-09-10T19:12:17.048248+0000","last_clean":"2024-09-10T19:12:17.048248+0000","last_became_active":"2024-09-10T19:10:09.690328+0000","last_became_peered":"2024-09-10T19:10:09.690328+0000","last_unstale":"2024-09-10T19:12:17.048248+0000","last_undegraded":"2024-09-10T19:12:17.048248+0000","last_fullsized":"2024-09-10T19:12:17.048248+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"78'22","last_scrub_stamp":"2024-09-10T19:11:21.118937+0000","last_deep_scrub":"78'22","last_deep_scrub_stamp":"2024-09-10T19:11:21.118937+0000","last_clean_scrub_stamp":"2024-09-10T19:11:21.118937+0000","objects_scrubbed":7,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0048819759999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":46,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-10T19:12:18.203977+0000","last_change":"2024-09-10T19:11:43.221376+0000","last_active":"2024-09-10T19:12:18.203977+0000","last_peered":"2024-09-10T19:12:18.203977+0000","last_clean":"2024-09-10T19:12:18.203977+0000","last_became_active":"2024-09-10T19:10:11.700037+0000","last_became_peered":"2024-09-10T19:10:11.700037+0000","last_unstale":"2024-09-10T19:12:18.203977+0000","last_undegraded":"2024-09-10T19:12:18.203977+0000","last_fullsized":"2024-09-10T19:12:18.203977+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:11:43.221323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:11:43.221323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:12:47.537742+0000","scrub_duration":0.001173189,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":45,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-10T19:12:17.049357+0000","last_change":"2024-09-10T19:11:24.827983+0000","last_active":"2024-09-10T19:12:17.049357+0000","last_peered":"2024-09-10T19:12:17.049357+0000","last_clean":"2024-09-10T19:12:17.049357+0000","last_became_active":"2024-09-10T19:10:14.053698+0000","last_became_peered":"2024-09-10T19:10:14.053698+0000","last_unstale":"2024-09-10T19:12:17.049357+0000","last_undegraded":"2024-09-10T19:12:17.049357+0000","last_fullsized":"2024-09-10T19:12:17.049357+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:11:24.827916+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:11:24.827916+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:12:36.645047+0000","scrub_duration":0.0014104860000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":51,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-10T19:12:17.049178+0000","last_change":"2024-09-10T19:11:25.870573+0000","last_active":"2024-09-10T19:12:17.049178+0000","last_peered":"2024-09-10T19:12:17.049178+0000","last_clean":"2024-09-10T19:12:17.049178+0000","last_became_active":"2024-09-10T19:10:07.682811+0000","last_became_peered":"2024-09-10T19:10:07.682811+0000","last_unstale":"2024-09-10T19:12:17.049178+0000","last_undegraded":"2024-09-10T19:12:17.049178+0000","last_fullsized":"2024-09-10T19:12:17.049178+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:11:25.870516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:11:25.870516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:12:33.650020+0000","scrub_duration":0.001292568,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1d","version":"66'12","reported_seq":72,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-10T19:12:17.048666+0000","last_change":"2024-09-10T19:11:52.663716+0000","last_active":"2024-09-10T19:12:17.048666+0000","last_peered":"2024-09-10T19:12:17.048666+0000","last_clean":"2024-09-10T19:12:17.048666+0000","last_became_active":"2024-09-10T19:10:09.692781+0000","last_became_peered":"2024-09-10T19:10:09.692781+0000","last_unstale":"2024-09-10T19:12:17.048666+0000","last_undegraded":"2024-09-10T19:12:17.048666+0000","last_fullsized":"2024-09-10T19:12:17.048666+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'12","last_scrub_stamp":"2024-09-10T19:11:52.663657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:11:52.663657+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:13:11.446956+0000","scrub_duration":0.0019136909999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":46,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-09-10T19:12:18.294642+0000","last_change":"2024-09-10T19:11:53.089520+0000","last_active":"2024-09-10T19:12:18.294642+0000","last_peered":"2024-09-10T19:12:18.294642+0000","last_clean":"2024-09-10T19:12:18.294642+0000","last_became_active":"2024-09-10T19:10:11.701155+0000","last_became_peered":"2024-09-10T19:10:11.701155+0000","last_unstale":"2024-09-10T19:12:18.294642+0000","last_undegraded":"2024-09-10T19:12:18.294642+0000","last_fullsized":"2024-09-10T19:12:18.294642+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:11:53.089456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:11:53.089456+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:13:10.833624+0000","scrub_duration":0.0024862920000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":16,"stat_sum":{"num_bytes":1822818304,"num_objects":27814,"num_object_clones":0,"num_object_copies":83442,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27814,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":27814,"num_write_kb":1780096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":5469831168,"data_stored":5469831168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":27814,"ondisk_log_size":27814,"up":48,"acting":48,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":67,"ondisk_log_size":67,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":177,"num_object_clones":0,"num_object_copies":531,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":177,"num_whiteouts":0,"num_read":698,"num_read_kb":455,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":434,"ondisk_log_size":434,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":136,"num_read_kb":131,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1328,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":27,"num_read_kb":27,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":3984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4,"ondisk_log_size":4,"up":96,"acting":96,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79,"ondisk_log_size":79,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":54,"seq":231928234017,"num_pgs":61,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":820624,"kb_used_data":523272,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":92924528,"statfs":{"total":95995035648,"available":95154716672,"internally_reserved":0,"allocated":535830528,"data_stored":535442144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 19:11:56 2024","interfaces":[{"interface":"back","average":{"1min":0.496,"5min":0.52700000000000002,"15min":0.53200000000000003},"min":{"1min":0.32100000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.76000000000000001,"5min":0.76000000000000001,"15min":0.76000000000000001},"last":0.25600000000000001},{"interface":"front","average":{"1min":0.5,"5min":0.53000000000000003,"15min":0.53500000000000003},"min":{"1min":0.223,"5min":0.223,"15min":0.223},"max":{"1min":0.64500000000000002,"5min":0.76200000000000001,"15min":0.76200000000000001},"last":0.70199999999999996}]},{"osd":1,"last update":"Tue Sep 10 19:11:56 2024","interfaces":[{"interface":"back","average":{"1min":0.52400000000000002,"5min":0.52400000000000002,"15min":0.52400000000000002},"min":{"1min":0.36199999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.70199999999999996,"5min":0.79400000000000004,"15min":0.79400000000000004},"last":0.51300000000000001},{"interface":"front","average":{"1min":0.49199999999999999,"5min":0.51200000000000001,"15min":0.51500000000000001},"min":{"1min":0.28999999999999998,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.68500000000000005,"5min":0.71799999999999997,"15min":0.71799999999999997},"last":0.56000000000000005}]},{"osd":2,"last update":"Tue Sep 10 19:11:56 2024","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.51800000000000002,"15min":0.52200000000000002},"min":{"1min":0.311,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.71899999999999997,"5min":0.84399999999999997,"15min":0.84399999999999997},"last":0.32000000000000001},{"interface":"front","average":{"1min":0.52000000000000002,"5min":0.54400000000000004,"15min":0.54800000000000004},"min":{"1min":0.42499999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.74399999999999999,"5min":0.77300000000000002,"15min":0.77300000000000002},"last":1.3799999999999999}]},{"osd":3,"last update":"Tue Sep 10 19:11:56 2024","interfaces":[{"interface":"back","average":{"1min":0.5,"5min":0.52800000000000002,"15min":0.53300000000000003},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.61299999999999999,"5min":0.86399999999999999,"15min":0.86399999999999999},"last":1.1160000000000001},{"interface":"front","average":{"1min":0.53500000000000003,"5min":0.52300000000000002,"15min":0.52100000000000002},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.80300000000000005,"5min":0.80300000000000005,"15min":0.80300000000000005},"last":0.57999999999999996}]},{"osd":4,"last update":"Tue Sep 10 19:11:56 2024","interfaces":[{"interface":"back","average":{"1min":0.45700000000000002,"5min":0.45500000000000002,"15min":0.45500000000000002},"min":{"1min":0.245,"5min":0.245,"15min":0.245},"max":{"1min":0.68999999999999995,"5min":0.68999999999999995,"15min":0.68999999999999995},"last":0.30399999999999999},{"interface":"front","average":{"1min":0.47399999999999998,"5min":0.46200000000000002,"15min":0.46000000000000002},"min":{"1min":0.31,"5min":0.31,"15min":0.31},"max":{"1min":0.71399999999999997,"5min":0.71399999999999997,"15min":0.71399999999999997},"last":0.38500000000000001}]},{"osd":5,"last update":"Tue Sep 10 19:11:56 2024","interfaces":[{"interface":"back","average":{"1min":0.52500000000000002,"5min":0.48899999999999999,"15min":0.48299999999999998},"min":{"1min":0.26300000000000001,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.82899999999999996,"5min":0.82899999999999996,"15min":0.82899999999999996},"last":0.28799999999999998},{"interface":"front","average":{"1min":0.52000000000000002,"5min":0.48399999999999999,"15min":0.47799999999999998},"min":{"1min":0.29499999999999998,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.73299999999999998,"5min":0.73299999999999998,"15min":0.73299999999999998},"last":0.33900000000000002}]},{"osd":6,"last update":"Tue Sep 10 19:11:56 2024","interfaces":[{"interface":"back","average":{"1min":0.55000000000000004,"5min":0.51000000000000001,"15min":0.503},"min":{"1min":0.29299999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.78100000000000003,"5min":0.78100000000000003,"15min":0.78100000000000003},"last":0.36899999999999999},{"interface":"front","average":{"1min":0.52800000000000002,"5min":0.496,"15min":0.49099999999999999},"min":{"1min":0.312,"5min":0.312,"15min":0.312},"max":{"1min":0.75,"5min":0.75,"15min":0.75},"last":0.46800000000000003}]}]},{"osd":6,"up_from":47,"seq":201863462950,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":822084,"kb_used_data":524732,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":92923068,"statfs":{"total":95995035648,"available":95153221632,"internally_reserved":0,"allocated":537325568,"data_stored":536949598,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 19:11:36 2024","interfaces":[{"interface":"back","average":{"1min":0.48399999999999999,"5min":0.51400000000000001,"15min":0.51900000000000002},"min":{"1min":0.309,"5min":0.309,"15min":0.309},"max":{"1min":0.67800000000000005,"5min":0.67800000000000005,"15min":0.67800000000000005},"last":0.23400000000000001},{"interface":"front","average":{"1min":0.47699999999999998,"5min":0.53000000000000003,"15min":0.53900000000000003},"min":{"1min":0.308,"5min":0.308,"15min":0.308},"max":{"1min":0.59999999999999998,"5min":0.67200000000000004,"15min":0.67200000000000004},"last":0.245}]},{"osd":1,"last update":"Tue Sep 10 19:11:36 2024","interfaces":[{"interface":"back","average":{"1min":0.501,"5min":0.52300000000000002,"15min":0.52700000000000002},"min":{"1min":0.318,"5min":0.318,"15min":0.318},"max":{"1min":0.63900000000000001,"5min":0.67400000000000004,"15min":0.67400000000000004},"last":1.29},{"interface":"front","average":{"1min":0.48499999999999999,"5min":0.52100000000000002,"15min":0.52700000000000002},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.63,"5min":0.65800000000000003,"15min":0.65800000000000003},"last":0.39700000000000002}]},{"osd":2,"last update":"Tue Sep 10 19:11:36 2024","interfaces":[{"interface":"back","average":{"1min":0.5,"5min":0.53600000000000003,"15min":0.54200000000000004},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.65100000000000002,"5min":0.66400000000000003,"15min":0.66400000000000003},"last":0.41999999999999998},{"interface":"front","average":{"1min":0.501,"5min":0.50800000000000001,"15min":0.50900000000000001},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.78600000000000003,"5min":0.78600000000000003,"15min":0.78600000000000003},"last":1.268}]},{"osd":3,"last update":"Tue Sep 10 19:11:36 2024","interfaces":[{"interface":"back","average":{"1min":0.47799999999999998,"5min":0.50800000000000001,"15min":0.51300000000000001},"min":{"1min":0.32100000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.64600000000000002,"5min":0.68100000000000005,"15min":0.68100000000000005},"last":1.3919999999999999},{"interface":"front","average":{"1min":0.52900000000000003,"5min":0.55800000000000005,"15min":0.56299999999999994},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.69499999999999995,"5min":0.70199999999999996,"15min":0.70199999999999996},"last":0.93100000000000005}]},{"osd":4,"last update":"Tue Sep 10 19:11:36 2024","interfaces":[{"interface":"back","average":{"1min":0.40899999999999997,"5min":0.35999999999999999,"15min":0.35199999999999998},"min":{"1min":0.25700000000000001,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":0.64700000000000002,"5min":0.64700000000000002,"15min":0.64700000000000002},"last":0.28499999999999998},{"interface":"front","average":{"1min":0.40899999999999997,"5min":0.38100000000000001,"15min":0.377},"min":{"1min":0.27800000000000002,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.58999999999999997,"5min":0.58999999999999997,"15min":0.58999999999999997},"last":1.369}]},{"osd":5,"last update":"Tue Sep 10 19:11:36 2024","interfaces":[{"interface":"back","average":{"1min":0.433,"5min":0.38600000000000001,"15min":0.378},"min":{"1min":0.27100000000000002,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.60699999999999998,"5min":0.60699999999999998,"15min":0.60699999999999998},"last":0.311},{"interface":"front","average":{"1min":0.46700000000000003,"5min":0.39200000000000002,"15min":0.38},"min":{"1min":0.318,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.66000000000000003,"5min":0.66000000000000003,"15min":0.66000000000000003},"last":1.639}]},{"osd":7,"last update":"Tue Sep 10 19:11:55 2024","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.46100000000000002,"15min":0.45100000000000001},"min":{"1min":0.34300000000000003,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.746,"5min":0.746,"15min":0.746},"last":0.438},{"interface":"front","average":{"1min":0.51100000000000001,"5min":0.45800000000000002,"15min":0.44900000000000001},"min":{"1min":0.36399999999999999,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.71799999999999997,"5min":0.71799999999999997,"15min":0.71799999999999997},"last":1.6739999999999999}]}]},{"osd":5,"up_from":39,"seq":167503724587,"num_pgs":52,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":744004,"kb_used_data":434172,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":93001148,"statfs":{"total":95995035648,"available":95233175552,"internally_reserved":0,"allocated":444592128,"data_stored":444216495,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 19:12:09 2024","interfaces":[{"interface":"back","average":{"1min":0.47499999999999998,"5min":0.48299999999999998,"15min":0.47699999999999998},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.67600000000000005,"5min":0.67600000000000005,"15min":0.67600000000000005},"last":0.57399999999999995},{"interface":"front","average":{"1min":0.48399999999999999,"5min":0.48599999999999999,"15min":0.47599999999999998},"min":{"1min":0.32700000000000001,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.68000000000000005,"5min":0.75600000000000001,"15min":0.75600000000000001},"last":0.42099999999999999}]},{"osd":1,"last update":"Tue Sep 10 19:12:09 2024","interfaces":[{"interface":"back","average":{"1min":0.48199999999999998,"5min":0.51500000000000001,"15min":0.50700000000000001},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.64600000000000002,"5min":0.72799999999999998,"15min":0.72799999999999998},"last":0.93700000000000006},{"interface":"front","average":{"1min":0.47299999999999998,"5min":0.47899999999999998,"15min":0.47399999999999998},"min":{"1min":0.34399999999999997,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.61599999999999999,"5min":0.68799999999999994,"15min":0.68799999999999994},"last":1.6359999999999999}]},{"osd":2,"last update":"Tue Sep 10 19:12:09 2024","interfaces":[{"interface":"back","average":{"1min":0.502,"5min":0.49099999999999999,"15min":0.47899999999999998},"min":{"1min":0.35999999999999999,"5min":0.309,"15min":0.309},"max":{"1min":0.75800000000000001,"5min":0.75800000000000001,"15min":0.75800000000000001},"last":0.33200000000000002},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.504,"15min":0.49299999999999999},"min":{"1min":0.36399999999999999,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.80800000000000005,"5min":0.80800000000000005,"15min":0.80800000000000005},"last":0.90700000000000003}]},{"osd":3,"last update":"Tue Sep 10 19:12:09 2024","interfaces":[{"interface":"back","average":{"1min":0.50700000000000001,"5min":0.52100000000000002,"15min":0.51400000000000001},"min":{"1min":0.38200000000000001,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.64400000000000002,"5min":0.74099999999999999,"15min":0.74099999999999999},"last":0.28100000000000003},{"interface":"front","average":{"1min":0.54300000000000004,"5min":0.54200000000000004,"15min":0.52900000000000003},"min":{"1min":0.36799999999999999,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.73699999999999999,"5min":0.747,"15min":0.747},"last":0.34899999999999998}]},{"osd":4,"last update":"Tue Sep 10 19:12:09 2024","interfaces":[{"interface":"back","average":{"1min":0.40600000000000003,"5min":0.38100000000000001,"15min":0.36199999999999999},"min":{"1min":0.27500000000000002,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":0.66400000000000003,"5min":0.66400000000000003,"15min":0.66400000000000003},"last":0.38100000000000001},{"interface":"front","average":{"1min":0.433,"5min":0.39600000000000002,"15min":0.372},"min":{"1min":0.28299999999999997,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.61799999999999999,"5min":0.76800000000000002,"15min":0.76800000000000002},"last":0.309}]},{"osd":6,"last update":"Tue Sep 10 19:11:35 2024","interfaces":[{"interface":"back","average":{"1min":0.48399999999999999,"5min":0.42499999999999999,"15min":0.41499999999999998},"min":{"1min":0.307,"5min":0.223,"15min":0.223},"max":{"1min":0.74099999999999999,"5min":0.74099999999999999,"15min":0.74099999999999999},"last":0.68200000000000005},{"interface":"front","average":{"1min":0.48099999999999998,"5min":0.42099999999999999,"15min":0.41099999999999998},"min":{"1min":0.26800000000000002,"5min":0.25,"15min":0.25},"max":{"1min":0.78000000000000003,"5min":0.78000000000000003,"15min":0.78000000000000003},"last":0.59199999999999997}]},{"osd":7,"last update":"Tue Sep 10 19:11:55 2024","interfaces":[{"interface":"back","average":{"1min":0.48399999999999999,"5min":0.504,"15min":0.50800000000000001},"min":{"1min":0.31,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.68700000000000006,"5min":0.72699999999999998,"15min":0.72699999999999998},"last":0.32200000000000001},{"interface":"front","average":{"1min":0.47599999999999998,"5min":0.47799999999999998,"15min":0.47799999999999998},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.67400000000000004,"5min":0.67400000000000004,"15min":0.67400000000000004},"last":0.47299999999999998}]}]},{"osd":4,"up_from":32,"seq":137438953519,"num_pgs":57,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":846420,"kb_used_data":549068,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":92898732,"statfs":{"total":95995035648,"available":95128301568,"internally_reserved":0,"allocated":562245632,"data_stored":561852824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 19:11:49 2024","interfaces":[{"interface":"back","average":{"1min":0.48299999999999998,"5min":0.52000000000000002,"15min":0.53400000000000003},"min":{"1min":0.35299999999999998,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.626,"5min":0.91500000000000004,"15min":0.91500000000000004},"last":0.57799999999999996},{"interface":"front","average":{"1min":0.48399999999999999,"5min":0.51000000000000001,"15min":0.52400000000000002},"min":{"1min":0.30099999999999999,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.71899999999999997,"5min":0.78200000000000003,"15min":0.78200000000000003},"last":0.40899999999999997}]},{"osd":1,"last update":"Tue Sep 10 19:11:49 2024","interfaces":[{"interface":"back","average":{"1min":0.47999999999999998,"5min":0.503,"15min":0.50900000000000001},"min":{"1min":0.29999999999999999,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.72199999999999998,"5min":0.83499999999999996,"15min":0.83499999999999996},"last":0.56799999999999995},{"interface":"front","average":{"1min":0.51000000000000001,"5min":0.51700000000000002,"15min":0.52200000000000002},"min":{"1min":0.27500000000000002,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.68999999999999995,"5min":0.77100000000000002,"15min":0.77100000000000002},"last":0.29799999999999999}]},{"osd":2,"last update":"Tue Sep 10 19:11:49 2024","interfaces":[{"interface":"back","average":{"1min":0.49199999999999999,"5min":0.52400000000000002,"15min":0.53400000000000003},"min":{"1min":0.30399999999999999,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.65600000000000003,"5min":1.0649999999999999,"15min":1.0649999999999999},"last":0.26600000000000001},{"interface":"front","average":{"1min":0.47599999999999998,"5min":0.51500000000000001,"15min":0.53000000000000003},"min":{"1min":0.33700000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.73699999999999999,"5min":0.86899999999999999,"15min":0.86899999999999999},"last":1.0229999999999999}]},{"osd":3,"last update":"Tue Sep 10 19:11:49 2024","interfaces":[{"interface":"back","average":{"1min":0.47899999999999998,"5min":0.51100000000000001,"15min":0.52100000000000002},"min":{"1min":0.34999999999999998,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.67000000000000004,"5min":0.76500000000000001,"15min":0.76500000000000001},"last":0.33800000000000002},{"interface":"front","average":{"1min":0.505,"5min":0.53600000000000003,"15min":0.54600000000000004},"min":{"1min":0.373,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.73099999999999998,"5min":1.0249999999999999,"15min":1.0249999999999999},"last":1.4330000000000001}]},{"osd":5,"last update":"Tue Sep 10 19:12:12 2024","interfaces":[{"interface":"back","average":{"1min":0.41999999999999998,"5min":0.40100000000000002,"15min":0.38300000000000001},"min":{"1min":0.245,"5min":0.245,"15min":0.245},"max":{"1min":0.61299999999999999,"5min":0.61599999999999999,"15min":0.61599999999999999},"last":0.35299999999999998},{"interface":"front","average":{"1min":0.41799999999999998,"5min":0.41399999999999998,"15min":0.40000000000000002},"min":{"1min":0.28499999999999998,"5min":0.223,"15min":0.223},"max":{"1min":0.70899999999999996,"5min":0.89100000000000001,"15min":0.89100000000000001},"last":0.32500000000000001}]},{"osd":6,"last update":"Tue Sep 10 19:11:36 2024","interfaces":[{"interface":"back","average":{"1min":0.46800000000000003,"5min":0.42399999999999999,"15min":0.41599999999999998},"min":{"1min":0.29799999999999999,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.627,"5min":0.627,"15min":0.627},"last":0.41899999999999998},{"interface":"front","average":{"1min":0.46899999999999997,"5min":0.42499999999999999,"15min":0.41699999999999998},"min":{"1min":0.28999999999999998,"5min":0.25,"15min":0.25},"max":{"1min":0.65500000000000003,"5min":0.65500000000000003,"15min":0.65500000000000003},"last":0.45000000000000001}]},{"osd":7,"last update":"Tue Sep 10 19:12:00 2024","interfaces":[{"interface":"back","average":{"1min":0.5,"5min":0.52000000000000002,"15min":0.52300000000000002},"min":{"1min":0.28100000000000003,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":0.70699999999999996,"5min":0.72699999999999998,"15min":0.72699999999999998},"last":0.39800000000000002},{"interface":"front","average":{"1min":0.46300000000000002,"5min":0.495,"15min":0.5},"min":{"1min":0.33800000000000002,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.67100000000000004,"5min":0.69899999999999995,"15min":0.69899999999999995},"last":0.38400000000000001}]}]},{"osd":3,"up_from":26,"seq":111669149748,"num_pgs":64,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1237544,"kb_used_data":931872,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":92507608,"statfs":{"total":95995035648,"available":94727790592,"internally_reserved":0,"allocated":954236928,"data_stored":953825214,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,2],"upper_bound":256},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 19:12:27 2024","interfaces":[{"interface":"back","average":{"1min":0.47799999999999998,"5min":0.40100000000000002,"15min":0.40500000000000003},"min":{"1min":0.20000000000000001,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":2.1080000000000001,"5min":2.1080000000000001,"15min":2.1080000000000001},"last":1.47},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.42099999999999999,"15min":0.41999999999999998},"min":{"1min":0.26100000000000001,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":3.585,"5min":3.585,"15min":3.585},"last":2.6120000000000001}]},{"osd":1,"last update":"Tue Sep 10 19:12:27 2024","interfaces":[{"interface":"back","average":{"1min":0.52800000000000002,"5min":0.44400000000000001,"15min":0.45200000000000001},"min":{"1min":0.216,"5min":0.216,"15min":0.216},"max":{"1min":3.5150000000000001,"5min":3.5150000000000001,"15min":3.5150000000000001},"last":2.3460000000000001},{"interface":"front","average":{"1min":0.46600000000000003,"5min":0.40000000000000002,"15min":0.39800000000000002},"min":{"1min":0.187,"5min":0.187,"15min":0.187},"max":{"1min":1.3640000000000001,"5min":1.3640000000000001,"15min":1.3640000000000001},"last":0.22500000000000001}]},{"osd":2,"last update":"Tue Sep 10 19:12:27 2024","interfaces":[{"interface":"back","average":{"1min":0.49299999999999999,"5min":0.436,"15min":0.438},"min":{"1min":0.22800000000000001,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":1.417,"5min":1.417,"15min":1.417},"last":0.41899999999999998},{"interface":"front","average":{"1min":0.47499999999999998,"5min":0.41899999999999998,"15min":0.41699999999999998},"min":{"1min":0.19800000000000001,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":1.988,"5min":1.988,"15min":1.988},"last":0.54700000000000004}]},{"osd":4,"last update":"Tue Sep 10 19:11:51 2024","interfaces":[{"interface":"back","average":{"1min":0.56200000000000006,"5min":0.54800000000000004,"15min":0.54600000000000004},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.69799999999999995,"5min":0.74399999999999999,"15min":0.74399999999999999},"last":0.57499999999999996},{"interface":"front","average":{"1min":0.51600000000000001,"5min":0.54200000000000004,"15min":0.54600000000000004},"min":{"1min":0.40300000000000002,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":0.68000000000000005,"5min":0.81699999999999995,"15min":0.81699999999999995},"last":2.3759999999999999}]},{"osd":5,"last update":"Tue Sep 10 19:12:11 2024","interfaces":[{"interface":"back","average":{"1min":0.52500000000000002,"5min":0.56200000000000006,"15min":0.56399999999999995},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":0.67500000000000004,"5min":0.80400000000000005,"15min":0.80400000000000005},"last":0.59299999999999997},{"interface":"front","average":{"1min":0.55400000000000005,"5min":0.55900000000000005,"15min":0.56000000000000005},"min":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.69499999999999995,"5min":0.89100000000000001,"15min":0.89100000000000001},"last":2.3980000000000001}]},{"osd":6,"last update":"Tue Sep 10 19:11:34 2024","interfaces":[{"interface":"back","average":{"1min":0.53400000000000003,"5min":0.56999999999999995,"15min":0.57599999999999996},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.67000000000000004,"5min":0.77000000000000002,"15min":0.77000000000000002},"last":0.60999999999999999},{"interface":"front","average":{"1min":0.55100000000000005,"5min":0.55500000000000005,"15min":0.55600000000000005},"min":{"1min":0.41399999999999998,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.71799999999999997,"5min":0.90800000000000003,"15min":0.90800000000000003},"last":0.378}]},{"osd":7,"last update":"Tue Sep 10 19:12:02 2024","interfaces":[{"interface":"back","average":{"1min":0.55900000000000005,"5min":0.57499999999999996,"15min":0.57799999999999996},"min":{"1min":0.41799999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"max":{"1min":0.67800000000000005,"5min":0.75600000000000001,"15min":0.75600000000000001},"last":0.40100000000000002},{"interface":"front","average":{"1min":0.58699999999999997,"5min":0.58399999999999996,"15min":0.58399999999999996},"min":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":0.72499999999999998,"5min":0.749,"15min":0.749},"last":0.629}]}]},{"osd":2,"up_from":18,"seq":77309411385,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":965512,"kb_used_data":663936,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":92779640,"statfs":{"total":95995035648,"available":95006351360,"internally_reserved":0,"allocated":679870464,"data_stored":679490154,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 19:12:08 2024","interfaces":[{"interface":"back","average":{"1min":0.29999999999999999,"5min":0.34100000000000003,"15min":0.34000000000000002},"min":{"1min":0.188,"5min":0.188,"15min":0.188},"max":{"1min":0.46999999999999997,"5min":0.52300000000000002,"15min":0.52300000000000002},"last":1.24},{"interface":"front","average":{"1min":0.33700000000000002,"5min":0.36899999999999999,"15min":0.36599999999999999},"min":{"1min":0.23400000000000001,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":0.497,"5min":0.64200000000000002,"15min":0.64200000000000002},"last":0.22600000000000001}]},{"osd":1,"last update":"Tue Sep 10 19:12:08 2024","interfaces":[{"interface":"back","average":{"1min":0.317,"5min":0.378,"15min":0.375},"min":{"1min":0.23400000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.46800000000000003,"5min":0.69799999999999995,"15min":0.69799999999999995},"last":1.5569999999999999},{"interface":"front","average":{"1min":0.33300000000000002,"5min":0.38600000000000001,"15min":0.38800000000000001},"min":{"1min":0.255,"5min":0.255,"15min":0.255},"max":{"1min":0.499,"5min":0.63400000000000001,"15min":0.63400000000000001},"last":1.9750000000000001}]},{"osd":3,"last update":"Tue Sep 10 19:11:30 2024","interfaces":[{"interface":"back","average":{"1min":0.35699999999999998,"5min":0.435,"15min":0.44600000000000001},"min":{"1min":0.23699999999999999,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":0.49099999999999999,"5min":0.63900000000000001,"15min":0.63900000000000001},"last":1.401},{"interface":"front","average":{"1min":0.36199999999999999,"5min":0.432,"15min":0.442},"min":{"1min":0.27100000000000002,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.438,"5min":0.65800000000000003,"15min":0.65800000000000003},"last":0.377}]},{"osd":4,"last update":"Tue Sep 10 19:11:52 2024","interfaces":[{"interface":"back","average":{"1min":0.52300000000000002,"5min":0.57099999999999995,"15min":0.57699999999999996},"min":{"1min":0.30099999999999999,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.67600000000000005,"5min":0.81999999999999995,"15min":0.81999999999999995},"last":1.206},{"interface":"front","average":{"1min":0.52900000000000003,"5min":0.56000000000000005,"15min":0.56799999999999995},"min":{"1min":0.41899999999999998,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.77600000000000002,"5min":0.80600000000000005,"15min":0.80600000000000005},"last":0.40500000000000003}]},{"osd":5,"last update":"Tue Sep 10 19:12:08 2024","interfaces":[{"interface":"back","average":{"1min":0.49299999999999999,"5min":0.52300000000000002,"15min":0.52400000000000002},"min":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.73599999999999999,"5min":0.73899999999999999,"15min":0.73899999999999999},"last":0.45900000000000002},{"interface":"front","average":{"1min":0.54300000000000004,"5min":0.57099999999999995,"15min":0.57599999999999996},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.79400000000000004,"5min":0.82699999999999996,"15min":0.82699999999999996},"last":1.2629999999999999}]},{"osd":6,"last update":"Tue Sep 10 19:11:34 2024","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":0.55200000000000005,"15min":0.55600000000000005},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":0.67300000000000004,"5min":0.73199999999999998,"15min":0.73199999999999998},"last":1.4350000000000001},{"interface":"front","average":{"1min":0.54800000000000004,"5min":0.55000000000000004,"15min":0.55000000000000004},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.72299999999999998,"5min":0.748,"15min":0.748},"last":0.432}]},{"osd":7,"last update":"Tue Sep 10 19:12:01 2024","interfaces":[{"interface":"back","average":{"1min":0.53300000000000003,"5min":0.59099999999999997,"15min":0.60099999999999998},"min":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":0.86299999999999999,"5min":0.86299999999999999,"15min":0.86299999999999999},"last":1.2829999999999999},{"interface":"front","average":{"1min":0.53300000000000003,"5min":0.54500000000000004,"15min":0.54700000000000004},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":0.80300000000000005,"5min":0.80300000000000005,"15min":0.80300000000000005},"last":0.64200000000000002}]}]},{"osd":1,"up_from":13,"seq":55834574910,"num_pgs":59,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":936552,"kb_used_data":639200,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":92808600,"statfs":{"total":95995035648,"available":95036006400,"internally_reserved":0,"allocated":654540800,"data_stored":654128440,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 19:11:42 2024","interfaces":[{"interface":"back","average":{"1min":0.38500000000000001,"5min":0.378,"15min":0.33800000000000002},"min":{"1min":0.21099999999999999,"5min":0.188,"15min":0.188},"max":{"1min":1.0629999999999999,"5min":2.7730000000000001,"15min":2.7730000000000001},"last":1.9810000000000001},{"interface":"front","average":{"1min":0.36699999999999999,"5min":0.38,"15min":0.34699999999999998},"min":{"1min":0.20699999999999999,"5min":0.20599999999999999,"15min":0.20599999999999999},"max":{"1min":0.53700000000000003,"5min":2.8380000000000001,"15min":2.8380000000000001},"last":3.8999999999999999}]},{"osd":2,"last update":"Tue Sep 10 19:12:04 2024","interfaces":[{"interface":"back","average":{"1min":0.35399999999999998,"5min":0.38200000000000001,"15min":0.372},"min":{"1min":0.23699999999999999,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":0.52000000000000002,"5min":1.1779999999999999,"15min":1.1779999999999999},"last":5.1529999999999996},{"interface":"front","average":{"1min":0.35999999999999999,"5min":0.38,"15min":0.36799999999999999},"min":{"1min":0.25,"5min":0.25,"15min":0.25},"max":{"1min":0.51000000000000001,"5min":1.103,"15min":1.103},"last":0.77100000000000002}]},{"osd":3,"last update":"Tue Sep 10 19:11:29 2024","interfaces":[{"interface":"back","average":{"1min":0.44700000000000001,"5min":0.45600000000000002,"15min":0.44900000000000001},"min":{"1min":0.27500000000000002,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":1.093,"5min":2.0670000000000002,"15min":2.0670000000000002},"last":0.80400000000000005},{"interface":"front","average":{"1min":0.40100000000000002,"5min":0.42299999999999999,"15min":0.42099999999999999},"min":{"1min":0.30299999999999999,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":0.54600000000000004,"5min":1.956,"15min":1.956},"last":4.0439999999999996}]},{"osd":4,"last update":"Tue Sep 10 19:11:49 2024","interfaces":[{"interface":"back","average":{"1min":0.55200000000000005,"5min":0.53300000000000003,"15min":0.51100000000000001},"min":{"1min":0.376,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.71899999999999997,"5min":2.665,"15min":2.665},"last":0.251},{"interface":"front","average":{"1min":0.57499999999999996,"5min":0.56899999999999995,"15min":0.54900000000000004},"min":{"1min":0.39000000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.76400000000000001,"5min":2.7330000000000001,"15min":2.7330000000000001},"last":0.33500000000000002}]},{"osd":5,"last update":"Tue Sep 10 19:12:13 2024","interfaces":[{"interface":"back","average":{"1min":0.55900000000000005,"5min":0.54400000000000004,"15min":0.53200000000000003},"min":{"1min":0.39800000000000002,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.81899999999999995,"5min":1.851,"15min":1.851},"last":3.7770000000000001},{"interface":"front","average":{"1min":0.54100000000000004,"5min":0.55300000000000005,"15min":0.53800000000000003},"min":{"1min":0.36599999999999999,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.72199999999999998,"5min":2.706,"15min":2.706},"last":0.28599999999999998}]},{"osd":6,"last update":"Tue Sep 10 19:11:35 2024","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.60199999999999998,"15min":0.61299999999999999},"min":{"1min":0.41599999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.68500000000000005,"5min":1.9950000000000001,"15min":1.9950000000000001},"last":3.8679999999999999},{"interface":"front","average":{"1min":0.54200000000000004,"5min":0.58599999999999997,"15min":0.59399999999999997},"min":{"1min":0.39300000000000002,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.90600000000000003,"5min":1.9399999999999999,"15min":1.9399999999999999},"last":0.31}]},{"osd":7,"last update":"Tue Sep 10 19:11:57 2024","interfaces":[{"interface":"back","average":{"1min":0.57499999999999996,"5min":0.64700000000000002,"15min":0.65900000000000003},"min":{"1min":0.42199999999999999,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.79000000000000004,"5min":3.4620000000000002,"15min":3.4620000000000002},"last":0.373},{"interface":"front","average":{"1min":0.56000000000000005,"5min":0.60799999999999998,"15min":0.61599999999999999},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.78300000000000003,"5min":1.514,"15min":1.514},"last":0.52700000000000002}]}]},{"osd":0,"up_from":8,"seq":34359738435,"num_pgs":55,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1182036,"kb_used_data":876364,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":92563116,"statfs":{"total":95995035648,"available":94784630784,"internally_reserved":0,"allocated":897396736,"data_stored":897005179,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,2,3],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Sep 10 19:11:42 2024","interfaces":[{"interface":"back","average":{"1min":0.318,"5min":0.34799999999999998,"15min":0.33100000000000002},"min":{"1min":0.23999999999999999,"5min":0.221,"15min":0.221},"max":{"1min":0.40799999999999997,"5min":0.63300000000000001,"15min":0.63300000000000001},"last":0.81999999999999995},{"interface":"front","average":{"1min":0.33600000000000002,"5min":0.35199999999999998,"15min":0.33500000000000002},"min":{"1min":0.23799999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":0.46899999999999997,"5min":0.61899999999999999,"15min":0.61899999999999999},"last":1.1739999999999999}]},{"osd":2,"last update":"Tue Sep 10 19:12:10 2024","interfaces":[{"interface":"back","average":{"1min":0.32000000000000001,"5min":0.375,"15min":0.379},"min":{"1min":0.21299999999999999,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":0.48399999999999999,"5min":0.57399999999999995,"15min":0.57399999999999995},"last":0.876},{"interface":"front","average":{"1min":0.313,"5min":0.36299999999999999,"15min":0.36899999999999999},"min":{"1min":0.187,"5min":0.187,"15min":0.187},"max":{"1min":0.46300000000000002,"5min":0.54100000000000004,"15min":0.54100000000000004},"last":0.96899999999999997}]},{"osd":3,"last update":"Tue Sep 10 19:11:30 2024","interfaces":[{"interface":"back","average":{"1min":0.35999999999999999,"5min":0.41499999999999998,"15min":0.42099999999999999},"min":{"1min":0.29499999999999998,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.45500000000000002,"5min":0.65000000000000002,"15min":0.65000000000000002},"last":1.0229999999999999},{"interface":"front","average":{"1min":0.34499999999999997,"5min":0.39900000000000002,"15min":0.40500000000000003},"min":{"1min":0.27800000000000002,"5min":0.246,"15min":0.246},"max":{"1min":0.45900000000000002,"5min":0.58799999999999997,"15min":0.58799999999999997},"last":1.411}]},{"osd":4,"last update":"Tue Sep 10 19:11:46 2024","interfaces":[{"interface":"back","average":{"1min":0.496,"5min":0.54800000000000004,"15min":0.56000000000000005},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.81499999999999995,"5min":0.81499999999999995,"15min":0.81499999999999995},"last":0.78300000000000003},{"interface":"front","average":{"1min":0.51200000000000001,"5min":0.55300000000000005,"15min":0.55900000000000005},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.71599999999999997,"5min":0.77000000000000002,"15min":0.77000000000000002},"last":1.0449999999999999}]},{"osd":5,"last update":"Tue Sep 10 19:12:10 2024","interfaces":[{"interface":"back","average":{"1min":0.46600000000000003,"5min":0.56100000000000005,"15min":0.57699999999999996},"min":{"1min":0.28199999999999997,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.68700000000000006,"5min":0.82599999999999996,"15min":0.82599999999999996},"last":1.0700000000000001},{"interface":"front","average":{"1min":0.49199999999999999,"5min":0.55700000000000005,"15min":0.56899999999999995},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.755,"5min":0.82899999999999996,"15min":0.82899999999999996},"last":1.431}]},{"osd":6,"last update":"Tue Sep 10 19:11:33 2024","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":0.55800000000000005,"15min":0.56399999999999995},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.78900000000000003,"5min":0.78900000000000003,"15min":0.78900000000000003},"last":0.84499999999999997},{"interface":"front","average":{"1min":0.54800000000000004,"5min":0.57899999999999996,"15min":0.58399999999999996},"min":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":0.70499999999999996,"5min":0.76100000000000001,"15min":0.76100000000000001},"last":1.0940000000000001}]},{"osd":7,"last update":"Tue Sep 10 19:11:59 2024","interfaces":[{"interface":"back","average":{"1min":0.52900000000000003,"5min":0.56799999999999995,"15min":0.57399999999999995},"min":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.71999999999999997,"5min":0.76900000000000002,"15min":0.76900000000000002},"last":1.117},{"interface":"front","average":{"1min":0.51000000000000001,"5min":0.55100000000000005,"15min":0.55800000000000005},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.80300000000000005,"5min":0.80300000000000005,"15min":0.80300000000000005},"last":1.381}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":966,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1236,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1236,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":362,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":2118,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1018,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":918880256,"data_stored":918880256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":676331520,"data_stored":676331520,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":684851200,"data_stored":684851200,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1028849664,"data_stored":1028849664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":575537152,"data_stored":575537152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":449511424,"data_stored":449511424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":564854784,"data_stored":564854784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":571015168,"data_stored":571015168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T19:12:35.461 INFO:tasks.cephadm.ceph_manager.ceph:would drop below min_pgs, proj 6, currently 16 2024-09-10T19:12:35.536 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:12:35.576 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:12:35.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:12:35.577 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:12:35.644 INFO:teuthology.orchestra.run.smithi183.stdout:{ 2024-09-10T19:12:35.644 INFO:teuthology.orchestra.run.smithi183.stdout: "size": 20, 2024-09-10T19:12:35.644 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 600, 2024-09-10T19:12:35.644 INFO:teuthology.orchestra.run.smithi183.stdout: "ops": [ 2024-09-10T19:12:35.645 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:12:35.645 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.14781.0:16986 7.4 7:2b0959cf:::benchmark_data_smithi183_84160_object16985:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e90)", 2024-09-10T19:12:35.645 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-10T19:12:24.656246+0000", 2024-09-10T19:12:35.645 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 10.152090619000001, 2024-09-10T19:12:35.645 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.106528848, 2024-09-10T19:12:35.645 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-10T19:12:35.645 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T19:12:35.645 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-10T19:12:35.645 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.14781", 2024-09-10T19:12:35.645 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.183:0/2025776420", 2024-09-10T19:12:35.645 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 16986 2024-09-10T19:12:35.645 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:12:35.645 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-10T19:12:35.646 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:12:35.646 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-10T19:12:35.646 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:24.656246+0000", 2024-09-10T19:12:35.646 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:12:35.646 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:12:35.646 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:12:35.646 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-10T19:12:35.646 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:24.656246+0000", 2024-09-10T19:12:35.646 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:12:35.646 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:12:35.646 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:12:35.646 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-10T19:12:35.646 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:24.656245+0000", 2024-09-10T19:12:35.647 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4294967295.999999 2024-09-10T19:12:35.647 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:12:35.647 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:12:35.647 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-10T19:12:35.647 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:24.656274+0000", 2024-09-10T19:12:35.647 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.8673000000000001e-05 2024-09-10T19:12:35.647 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:12:35.647 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:12:35.647 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-10T19:12:35.647 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:24.656430+0000", 2024-09-10T19:12:35.647 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00015557799999999999 2024-09-10T19:12:35.647 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:12:35.647 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:12:35.647 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-10T19:12:35.648 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:24.656435+0000", 2024-09-10T19:12:35.648 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.7030000000000003e-06 2024-09-10T19:12:35.648 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:12:35.648 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:12:35.648 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-10T19:12:35.648 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:24.660597+0000", 2024-09-10T19:12:35.648 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0041619700000000001 2024-09-10T19:12:35.648 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:12:35.648 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:12:35.648 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-10T19:12:35.648 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:24.660705+0000", 2024-09-10T19:12:35.648 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000107416 2024-09-10T19:12:35.648 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:12:35.649 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:12:35.649 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 1,4", 2024-09-10T19:12:35.649 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:24.660829+0000", 2024-09-10T19:12:35.649 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00012446399999999999 2024-09-10T19:12:35.649 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:12:35.649 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:12:35.649 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-10T19:12:35.649 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:24.761692+0000", 2024-09-10T19:12:35.649 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.100863017 2024-09-10T19:12:35.649 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:12:35.649 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:12:35.649 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:12:35.649 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:24.762668+0000", 2024-09-10T19:12:35.649 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00097569499999999995 2024-09-10T19:12:35.650 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:12:35.650 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:12:35.650 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:12:35.650 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:24.762706+0000", 2024-09-10T19:12:35.650 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.7991999999999998e-05 2024-09-10T19:12:35.650 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:12:35.652 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:12:35.652 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-10T19:12:35.652 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:24.762743+0000", 2024-09-10T19:12:35.652 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.6928999999999998e-05 2024-09-10T19:12:35.652 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:12:35.652 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:12:35.652 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-10T19:12:35.652 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:24.762775+0000", 2024-09-10T19:12:35.652 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.2129999999999999e-05 2024-09-10T19:12:35.652 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:12:35.652 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:12:35.653 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:12:35.653 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:12:35.653 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:12:35.653 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.14781.0:16995 7.f 7:f10a42e4:::benchmark_data_smithi183_84160_object16994:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e90)", 2024-09-10T19:12:35.653 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-10T19:12:24.661345+0000", 2024-09-10T19:12:35.653 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 10.146991888000001, 2024-09-10T19:12:35.653 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.105958914, 2024-09-10T19:12:35.653 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-10T19:12:35.653 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T19:12:35.653 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-10T19:12:35.653 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.14781", 2024-09-10T19:12:35.653 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.183:0/2025776420", 2024-09-10T19:12:35.653 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 16995 2024-09-10T19:12:35.654 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:12:35.654 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-10T19:12:35.654 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:12:35.654 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-10T19:12:35.654 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:24.661345+0000", 2024-09-10T19:12:35.654 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:12:35.654 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:12:35.654 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:12:35.654 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-10T19:12:35.654 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:24.661345+0000", 2024-09-10T19:12:35.654 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:12:35.654 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:12:35.654 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:12:35.654 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-10T19:12:35.654 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:24.661343+0000", 2024-09-10T19:12:35.655 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4294967295.9999986 2024-09-10T19:12:35.655 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:12:35.655 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:12:35.655 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-10T19:12:35.655 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:24.661365+0000", 2024-09-10T19:12:35.655 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.1920999999999999e-05 2024-09-10T19:12:35.655 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:12:35.655 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:12:35.655 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-10T19:12:35.655 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:24.661376+0000", 2024-09-10T19:12:35.655 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.0464e-05 2024-09-10T19:12:35.655 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:12:35.655 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:12:35.655 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-10T19:12:35.656 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:24.661381+0000", 2024-09-10T19:12:35.656 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.9230000000000001e-06 2024-09-10T19:12:35.656 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:12:35.656 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:12:35.656 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-10T19:12:35.656 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:24.661436+0000", 2024-09-10T19:12:35.656 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.5061e-05 2024-09-10T19:12:35.656 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:12:35.656 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:12:35.656 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-10T19:12:35.656 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:24.662103+0000", 2024-09-10T19:12:35.656 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00066722000000000005 2024-09-10T19:12:35.656 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:12:35.656 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:12:35.657 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 3,7", 2024-09-10T19:12:35.657 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:24.662228+0000", 2024-09-10T19:12:35.657 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00012464999999999999 2024-09-10T19:12:35.657 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:12:35.657 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:12:35.657 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:12:35.657 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:24.765067+0000", 2024-09-10T19:12:35.657 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.10283895 2024-09-10T19:12:35.657 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:12:35.657 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:12:35.657 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-10T19:12:35.658 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:24.765330+0000", 2024-09-10T19:12:35.658 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000263093 2024-09-10T19:12:35.658 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:12:35.659 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:12:35.659 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:12:35.659 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:24.767167+0000", 2024-09-10T19:12:35.659 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0018376029999999999 2024-09-10T19:12:35.659 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:12:35.659 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:12:35.659 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-10T19:12:35.659 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:24.767258+0000", 2024-09-10T19:12:35.659 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.1174999999999994e-05 2024-09-10T19:12:35.659 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:12:35.659 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:12:35.659 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-10T19:12:35.659 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:24.767304+0000", 2024-09-10T19:12:35.659 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.5269000000000003e-05 2024-09-10T19:12:35.660 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:12:35.660 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:12:35.660 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:12:35.660 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:12:35.660 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:12:35.660 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_repop(client.14781.0:16998 7.5 e90/86 7:a4079dc6:::benchmark_data_smithi183_84160_object16997:head v 90'1042, mlcod=90'1039)", 2024-09-10T19:12:35.660 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-10T19:12:24.661861+0000", 2024-09-10T19:12:35.660 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 10.14647576, 2024-09-10T19:12:35.660 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.101015572, 2024-09-10T19:12:35.660 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-10T19:12:35.660 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T19:12:35.660 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-10T19:12:35.660 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:12:35.660 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-10T19:12:35.661 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:24.661861+0000", 2024-09-10T19:12:35.661 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:12:35.661 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:12:35.661 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:12:35.661 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-10T19:12:35.661 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:24.661861+0000", 2024-09-10T19:12:35.661 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:12:35.661 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:12:35.661 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:12:35.661 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-10T19:12:35.661 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:24.661860+0000", 2024-09-10T19:12:35.661 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4294967295.9999995 2024-09-10T19:12:35.661 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:12:35.662 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:12:35.662 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-10T19:12:35.662 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:24.661992+0000", 2024-09-10T19:12:35.662 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00013175799999999999 2024-09-10T19:12:35.662 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:12:35.662 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:12:35.662 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-10T19:12:35.662 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:24.662001+0000", 2024-09-10T19:12:35.662 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.9889999999999998e-06 2024-09-10T19:12:35.662 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:12:35.662 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:12:35.662 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-10T19:12:35.662 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:24.662006+0000", 2024-09-10T19:12:35.662 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.5070000000000002e-06 2024-09-10T19:12:35.663 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:12:35.663 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:12:35.663 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-10T19:12:35.663 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:24.662115+0000", 2024-09-10T19:12:35.663 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000108991 2024-09-10T19:12:35.663 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:12:35.663 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:12:35.663 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-10T19:12:35.663 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:24.662132+0000", 2024-09-10T19:12:35.663 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.6948000000000001e-05 2024-09-10T19:12:35.663 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:12:35.663 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:12:35.663 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-10T19:12:35.663 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:24.762847+0000", 2024-09-10T19:12:35.664 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.100715716 2024-09-10T19:12:35.664 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:12:35.664 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:12:35.664 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-10T19:12:35.664 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:24.762877+0000", 2024-09-10T19:12:35.664 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.9212000000000001e-05 2024-09-10T19:12:35.664 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:12:35.664 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:12:35.664 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:12:35.664 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:12:35.664 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:12:35.664 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.14781.0:16999 7.6 7:65065125:::benchmark_data_smithi183_84160_object16998:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e90)", 2024-09-10T19:12:35.664 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-10T19:12:24.662022+0000", 2024-09-10T19:12:35.665 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 10.146314586000001, 2024-09-10T19:12:35.665 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.104144664, 2024-09-10T19:12:35.665 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-10T19:12:35.665 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T19:12:35.665 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-10T19:12:35.665 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.14781", 2024-09-10T19:12:35.665 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.183:0/2025776420", 2024-09-10T19:12:35.665 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 16999 2024-09-10T19:12:35.665 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:12:35.665 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-10T19:12:35.665 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:12:35.665 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-10T19:12:35.665 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:24.662022+0000", 2024-09-10T19:12:35.665 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:12:35.666 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:12:35.666 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:12:35.666 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-10T19:12:35.666 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:24.662022+0000", 2024-09-10T19:12:35.666 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:12:35.666 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:12:35.666 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:12:35.666 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-10T19:12:35.666 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:24.662021+0000", 2024-09-10T19:12:35.666 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4294967295.9999995 2024-09-10T19:12:35.666 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:12:35.666 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:12:35.666 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-10T19:12:35.666 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:24.662043+0000", 2024-09-10T19:12:35.666 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.1489e-05 2024-09-10T19:12:35.667 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:12:35.667 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:12:35.667 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-10T19:12:35.667 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:24.662051+0000", 2024-09-10T19:12:35.667 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.7619999999999995e-06 2024-09-10T19:12:35.667 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:12:35.667 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:12:35.667 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-10T19:12:35.667 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:24.662055+0000", 2024-09-10T19:12:35.667 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.1049999999999997e-06 2024-09-10T19:12:35.667 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:12:35.667 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:12:35.667 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-10T19:12:35.667 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:24.761584+0000", 2024-09-10T19:12:35.668 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.099528944999999994 2024-09-10T19:12:35.668 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:12:35.668 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:12:35.668 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-10T19:12:35.668 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:24.761705+0000", 2024-09-10T19:12:35.668 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000121357 2024-09-10T19:12:35.668 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:12:35.668 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:12:35.668 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 3,7", 2024-09-10T19:12:35.668 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:24.761828+0000", 2024-09-10T19:12:35.668 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00012249099999999999 2024-09-10T19:12:35.668 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:12:35.668 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:12:35.668 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:12:35.669 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:24.765383+0000", 2024-09-10T19:12:35.669 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0035550220000000001 2024-09-10T19:12:35.669 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:12:35.669 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:12:35.669 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-10T19:12:35.669 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:24.765543+0000", 2024-09-10T19:12:35.669 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000160027 2024-09-10T19:12:35.669 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:12:35.669 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:12:35.669 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:12:35.669 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:24.766077+0000", 2024-09-10T19:12:35.669 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00053481399999999999 2024-09-10T19:12:35.669 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:12:35.669 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:12:35.670 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-10T19:12:35.670 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:24.766128+0000", 2024-09-10T19:12:35.670 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.0719e-05 2024-09-10T19:12:35.670 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:12:35.670 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:12:35.670 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-10T19:12:35.670 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:24.766167+0000", 2024-09-10T19:12:35.670 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.8628999999999999e-05 2024-09-10T19:12:35.670 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:12:35.670 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:12:35.670 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:12:35.670 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:12:35.670 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:12:35.670 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_repop_reply(client.14781.0:16986 7.4 e90/86 ondisk, result = 0)", 2024-09-10T19:12:35.671 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-10T19:12:24.662265+0000", 2024-09-10T19:12:35.671 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 10.146071321999999, 2024-09-10T19:12:35.671 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.10040881, 2024-09-10T19:12:35.671 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-10T19:12:35.671 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "started", 2024-09-10T19:12:35.671 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-10T19:12:35.671 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:12:35.671 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-10T19:12:35.671 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:24.662265+0000", 2024-09-10T19:12:35.671 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:12:35.671 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:12:35.671 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:12:35.671 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-10T19:12:35.671 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:24.662265+0000", 2024-09-10T19:12:35.671 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:12:35.672 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:12:35.672 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:12:35.672 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-10T19:12:35.672 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:24.662265+0000", 2024-09-10T19:12:35.672 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4294967295.9999995 2024-09-10T19:12:35.672 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:12:35.672 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:12:35.672 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-10T19:12:35.672 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:24.662269+0000", 2024-09-10T19:12:35.672 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.2420000000000002e-06 2024-09-10T19:12:35.672 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:12:35.672 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:12:35.672 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-10T19:12:35.672 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:24.761418+0000", 2024-09-10T19:12:35.673 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.099148606 2024-09-10T19:12:35.673 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:12:35.673 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:12:35.673 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-10T19:12:35.673 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:24.761428+0000", 2024-09-10T19:12:35.673 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.0504e-05 2024-09-10T19:12:35.673 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:12:35.673 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:12:35.673 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-10T19:12:35.673 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:24.761727+0000", 2024-09-10T19:12:35.673 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00029919799999999998 2024-09-10T19:12:35.673 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:12:35.673 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:12:35.673 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-10T19:12:35.674 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:24.762662+0000", 2024-09-10T19:12:35.674 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00093480600000000005 2024-09-10T19:12:35.674 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:12:35.674 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:12:35.674 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-10T19:12:35.674 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:24.762674+0000", 2024-09-10T19:12:35.674 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.2051e-05 2024-09-10T19:12:35.674 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:12:35.674 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:12:35.674 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:12:35.674 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:12:35.674 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:12:35.674 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_repop(client.14781.0:16994 7.e e90/86 7:78896f87:::benchmark_data_smithi183_84160_object16993:head v 90'1070, mlcod=90'1067)", 2024-09-10T19:12:35.674 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-10T19:12:24.662799+0000", 2024-09-10T19:12:35.674 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 10.145537703, 2024-09-10T19:12:35.675 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.100980772, 2024-09-10T19:12:35.675 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-10T19:12:35.675 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T19:12:35.675 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-10T19:12:35.675 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:12:35.675 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-10T19:12:35.675 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:24.662799+0000", 2024-09-10T19:12:35.675 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:12:35.675 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:12:35.675 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:12:35.675 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-10T19:12:35.675 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:24.662799+0000", 2024-09-10T19:12:35.675 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:12:35.675 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:12:35.676 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:12:35.676 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-10T19:12:35.676 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:24.662798+0000", 2024-09-10T19:12:35.676 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4294967295.9999995 2024-09-10T19:12:35.676 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:12:35.676 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:12:35.676 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-10T19:12:35.676 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:24.662822+0000", 2024-09-10T19:12:35.676 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.3584999999999999e-05 2024-09-10T19:12:35.676 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:12:35.676 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:12:35.676 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-10T19:12:35.676 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:24.761636+0000", 2024-09-10T19:12:35.677 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.098814564999999993 2024-09-10T19:12:35.677 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:12:35.677 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:12:35.677 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-10T19:12:35.677 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:24.761647+0000", 2024-09-10T19:12:35.677 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.0372e-05 2024-09-10T19:12:35.677 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:12:35.677 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:12:35.677 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-10T19:12:35.677 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:24.761850+0000", 2024-09-10T19:12:35.677 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00020292199999999999 2024-09-10T19:12:35.677 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:12:35.677 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:12:35.677 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-10T19:12:35.678 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:24.761870+0000", 2024-09-10T19:12:35.678 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.9769999999999999e-05 2024-09-10T19:12:35.678 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:12:35.678 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:12:35.678 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-10T19:12:35.678 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:24.763731+0000", 2024-09-10T19:12:35.678 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0018613989999999999 2024-09-10T19:12:35.678 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:12:35.678 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:12:35.678 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-10T19:12:35.678 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:24.763780+0000", 2024-09-10T19:12:35.679 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.8832000000000003e-05 2024-09-10T19:12:35.679 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:12:35.679 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:12:35.679 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:12:35.679 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:12:35.679 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:12:35.679 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.14781.0:23958 7.6 7:6a266b98:::benchmark_data_smithi183_84160_object23957:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e91)", 2024-09-10T19:12:35.679 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-10T19:12:27.625727+0000", 2024-09-10T19:12:35.679 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 7.182610103, 2024-09-10T19:12:35.679 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.10080093599999999, 2024-09-10T19:12:35.679 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-10T19:12:35.679 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T19:12:35.679 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-10T19:12:35.680 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.14781", 2024-09-10T19:12:35.680 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.183:0/2025776420", 2024-09-10T19:12:35.680 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 23958 2024-09-10T19:12:35.680 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:12:35.680 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-10T19:12:35.680 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:12:35.680 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-10T19:12:35.680 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:27.625727+0000", 2024-09-10T19:12:35.680 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:12:35.680 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:12:35.680 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:12:35.680 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-10T19:12:35.680 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:27.625727+0000", 2024-09-10T19:12:35.680 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:12:35.681 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:12:35.681 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:12:35.681 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-10T19:12:35.681 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:27.625726+0000", 2024-09-10T19:12:35.681 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4294967295.999999 2024-09-10T19:12:35.681 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:12:35.681 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:12:35.681 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-10T19:12:35.681 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:27.625758+0000", 2024-09-10T19:12:35.681 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.2268999999999999e-05 2024-09-10T19:12:35.681 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:12:35.681 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:12:35.681 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-10T19:12:35.681 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:27.625769+0000", 2024-09-10T19:12:35.682 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.0540000000000001e-05 2024-09-10T19:12:35.682 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:12:35.682 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:12:35.682 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-10T19:12:35.682 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:27.625774+0000", 2024-09-10T19:12:35.682 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.1499999999999998e-06 2024-09-10T19:12:35.682 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:12:35.682 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:12:35.682 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-10T19:12:35.682 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:27.627709+0000", 2024-09-10T19:12:35.682 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.001935296 2024-09-10T19:12:35.682 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:12:35.682 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:12:35.683 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-10T19:12:35.683 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:27.628077+0000", 2024-09-10T19:12:35.683 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00036759299999999999 2024-09-10T19:12:35.683 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:12:35.683 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:12:35.683 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 3,7", 2024-09-10T19:12:35.683 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:27.628212+0000", 2024-09-10T19:12:35.683 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00013590400000000001 2024-09-10T19:12:35.683 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:12:35.683 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:12:35.683 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:12:35.683 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:27.630501+0000", 2024-09-10T19:12:35.683 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.002288853 2024-09-10T19:12:35.683 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:12:35.684 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:12:35.684 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-10T19:12:35.684 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:27.631701+0000", 2024-09-10T19:12:35.684 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0012000419999999999 2024-09-10T19:12:35.684 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:12:35.684 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:12:35.684 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:12:35.684 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:27.726444+0000", 2024-09-10T19:12:35.684 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.094742241000000005 2024-09-10T19:12:35.684 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:12:35.684 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:12:35.684 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-10T19:12:35.684 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:27.726490+0000", 2024-09-10T19:12:35.684 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.6366999999999998e-05 2024-09-10T19:12:35.685 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:12:35.685 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:12:35.685 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-10T19:12:35.685 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:27.726528+0000", 2024-09-10T19:12:35.685 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.7546999999999998e-05 2024-09-10T19:12:35.685 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:12:35.685 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:12:35.685 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:12:35.685 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:12:35.685 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:12:35.685 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.14781.0:23959 7.f 7:f79c078d:::benchmark_data_smithi183_84160_object23958:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e91)", 2024-09-10T19:12:35.685 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-10T19:12:27.627080+0000", 2024-09-10T19:12:35.685 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 7.1812562709999996, 2024-09-10T19:12:35.685 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.099870543000000006, 2024-09-10T19:12:35.686 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-10T19:12:35.686 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T19:12:35.686 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-10T19:12:35.686 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.14781", 2024-09-10T19:12:35.686 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.183:0/2025776420", 2024-09-10T19:12:35.686 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 23959 2024-09-10T19:12:35.686 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:12:35.686 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-10T19:12:35.686 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:12:35.686 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-10T19:12:35.686 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:27.627080+0000", 2024-09-10T19:12:35.686 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:12:35.686 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:12:35.686 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:12:35.687 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-10T19:12:35.687 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:27.627080+0000", 2024-09-10T19:12:35.687 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:12:35.687 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:12:35.687 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:12:35.687 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-10T19:12:35.687 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:27.627079+0000", 2024-09-10T19:12:35.687 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4294967295.9999986 2024-09-10T19:12:35.687 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:12:35.687 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:12:35.687 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-10T19:12:35.687 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:27.627101+0000", 2024-09-10T19:12:35.687 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.1610000000000001e-05 2024-09-10T19:12:35.687 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:12:35.687 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:12:35.688 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-10T19:12:35.688 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:27.627112+0000", 2024-09-10T19:12:35.688 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.1582e-05 2024-09-10T19:12:35.688 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:12:35.688 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:12:35.688 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-10T19:12:35.688 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:27.627118+0000", 2024-09-10T19:12:35.688 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.643e-06 2024-09-10T19:12:35.688 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:12:35.688 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:12:35.688 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-10T19:12:35.688 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:27.628408+0000", 2024-09-10T19:12:35.688 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0012900279999999999 2024-09-10T19:12:35.688 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:12:35.689 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:12:35.689 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-10T19:12:35.689 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:27.628509+0000", 2024-09-10T19:12:35.689 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000100589 2024-09-10T19:12:35.689 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:12:35.689 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:12:35.689 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 3,7", 2024-09-10T19:12:35.689 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:27.628632+0000", 2024-09-10T19:12:35.689 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00012363900000000001 2024-09-10T19:12:35.689 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:12:35.689 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:12:35.689 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:12:35.689 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:27.631227+0000", 2024-09-10T19:12:35.689 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.002594552 2024-09-10T19:12:35.690 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:12:35.690 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:12:35.690 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:12:35.690 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:27.631320+0000", 2024-09-10T19:12:35.690 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.2961999999999995e-05 2024-09-10T19:12:35.690 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:12:35.690 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:12:35.690 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-10T19:12:35.690 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:27.726857+0000", 2024-09-10T19:12:35.690 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.095537056999999995 2024-09-10T19:12:35.690 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:12:35.690 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:12:35.690 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-10T19:12:35.690 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:27.726895+0000", 2024-09-10T19:12:35.691 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.8624999999999997e-05 2024-09-10T19:12:35.691 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:12:35.691 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:12:35.691 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-10T19:12:35.691 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:27.726951+0000", 2024-09-10T19:12:35.691 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.5482000000000002e-05 2024-09-10T19:12:35.691 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:12:35.691 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:12:35.691 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:12:35.691 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:12:35.691 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:12:35.691 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.14781.0:23960 7.9 7:9ce42710:::benchmark_data_smithi183_84160_object23959:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e91)", 2024-09-10T19:12:35.691 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-10T19:12:27.627124+0000", 2024-09-10T19:12:35.691 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 7.1812125819999997, 2024-09-10T19:12:35.691 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.098735662000000002, 2024-09-10T19:12:35.692 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-10T19:12:35.692 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T19:12:35.692 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-10T19:12:35.692 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.14781", 2024-09-10T19:12:35.692 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.183:0/2025776420", 2024-09-10T19:12:35.692 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 23960 2024-09-10T19:12:35.692 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:12:35.692 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-10T19:12:35.692 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:12:35.692 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-10T19:12:35.692 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:27.627124+0000", 2024-09-10T19:12:35.692 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:12:35.692 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:12:35.692 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:12:35.693 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-10T19:12:35.693 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:27.627124+0000", 2024-09-10T19:12:35.693 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:12:35.693 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:12:35.693 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:12:35.693 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-10T19:12:35.693 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:27.627124+0000", 2024-09-10T19:12:35.693 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4294967295.9999995 2024-09-10T19:12:35.693 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:12:35.693 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:12:35.693 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-10T19:12:35.693 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:27.627140+0000", 2024-09-10T19:12:35.693 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.6578999999999998e-05 2024-09-10T19:12:35.693 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:12:35.694 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:12:35.694 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-10T19:12:35.694 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:27.627146+0000", 2024-09-10T19:12:35.694 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.0789999999999997e-06 2024-09-10T19:12:35.694 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:12:35.694 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:12:35.694 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-10T19:12:35.694 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:27.627150+0000", 2024-09-10T19:12:35.694 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.9280000000000003e-06 2024-09-10T19:12:35.694 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:12:35.694 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:12:35.694 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-10T19:12:35.694 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:27.628938+0000", 2024-09-10T19:12:35.694 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.001787343 2024-09-10T19:12:35.695 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:12:35.695 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:12:35.695 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-10T19:12:35.695 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:27.629041+0000", 2024-09-10T19:12:35.695 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000103582 2024-09-10T19:12:35.695 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:12:35.695 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:12:35.695 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 1,4", 2024-09-10T19:12:35.695 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:27.629167+0000", 2024-09-10T19:12:35.695 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000125594 2024-09-10T19:12:35.695 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:12:35.695 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:12:35.695 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:12:35.695 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:27.631734+0000", 2024-09-10T19:12:35.696 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0025669540000000002 2024-09-10T19:12:35.696 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:12:35.696 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:12:35.696 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-10T19:12:35.696 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:27.725715+0000", 2024-09-10T19:12:35.696 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.093981239999999994 2024-09-10T19:12:35.696 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:12:35.696 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:12:35.696 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:12:35.696 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:27.725772+0000", 2024-09-10T19:12:35.696 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.6606e-05 2024-09-10T19:12:35.696 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:12:35.696 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:12:35.696 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-10T19:12:35.697 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:27.725822+0000", 2024-09-10T19:12:35.697 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.0766999999999997e-05 2024-09-10T19:12:35.697 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:12:35.697 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:12:35.697 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-10T19:12:35.697 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:27.725860+0000", 2024-09-10T19:12:35.697 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.7475999999999998e-05 2024-09-10T19:12:35.697 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:12:35.697 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:12:35.697 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:12:35.697 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:12:35.697 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:12:35.697 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.14781.0:23961 7.f 7:f635cd58:::benchmark_data_smithi183_84160_object23960:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e91)", 2024-09-10T19:12:35.697 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-10T19:12:27.627175+0000", 2024-09-10T19:12:35.698 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 7.1811613139999997, 2024-09-10T19:12:35.698 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.099879406000000004, 2024-09-10T19:12:35.698 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-10T19:12:35.698 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T19:12:35.698 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-10T19:12:35.698 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.14781", 2024-09-10T19:12:35.698 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.183:0/2025776420", 2024-09-10T19:12:35.698 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 23961 2024-09-10T19:12:35.698 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:12:35.698 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-10T19:12:35.698 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:12:35.698 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-10T19:12:35.698 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:27.627175+0000", 2024-09-10T19:12:35.698 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:12:35.699 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:12:35.699 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:12:35.699 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-10T19:12:35.699 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:27.627175+0000", 2024-09-10T19:12:35.699 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:12:35.699 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:12:35.699 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:12:35.699 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-10T19:12:35.699 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:27.627175+0000", 2024-09-10T19:12:35.699 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4294967295.9999995 2024-09-10T19:12:35.699 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:12:35.699 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:12:35.699 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-10T19:12:35.699 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:27.627193+0000", 2024-09-10T19:12:35.699 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.8020999999999999e-05 2024-09-10T19:12:35.700 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:12:35.700 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:12:35.700 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-10T19:12:35.700 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:27.627199+0000", 2024-09-10T19:12:35.700 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.1330000000000004e-06 2024-09-10T19:12:35.700 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:12:35.700 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:12:35.700 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-10T19:12:35.700 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:27.627203+0000", 2024-09-10T19:12:35.700 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.6990000000000001e-06 2024-09-10T19:12:35.700 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:12:35.700 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:12:35.700 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-10T19:12:35.700 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:27.629439+0000", 2024-09-10T19:12:35.701 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0022363790000000001 2024-09-10T19:12:35.701 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:12:35.701 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:12:35.701 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-10T19:12:35.701 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:27.629532+0000", 2024-09-10T19:12:35.701 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.2315e-05 2024-09-10T19:12:35.701 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:12:35.701 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:12:35.701 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 3,7", 2024-09-10T19:12:35.701 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:27.629638+0000", 2024-09-10T19:12:35.701 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00010642 2024-09-10T19:12:35.701 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:12:35.701 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:12:35.701 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:12:35.701 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:27.726821+0000", 2024-09-10T19:12:35.702 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.097183118999999998 2024-09-10T19:12:35.702 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:12:35.702 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:12:35.702 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:12:35.702 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:27.726844+0000", 2024-09-10T19:12:35.702 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.2558e-05 2024-09-10T19:12:35.702 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:12:35.702 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:12:35.702 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-10T19:12:35.702 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:27.726994+0000", 2024-09-10T19:12:35.702 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00014986599999999999 2024-09-10T19:12:35.702 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:12:35.702 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:12:35.702 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-10T19:12:35.703 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:27.727027+0000", 2024-09-10T19:12:35.703 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.3547999999999998e-05 2024-09-10T19:12:35.703 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:12:35.703 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:12:35.703 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-10T19:12:35.703 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:27.727055+0000", 2024-09-10T19:12:35.703 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.7732000000000001e-05 2024-09-10T19:12:35.703 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:12:35.703 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:12:35.703 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:12:35.703 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:12:35.703 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:12:35.703 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.14781.0:23962 7.f 7:fe03eb64:::benchmark_data_smithi183_84160_object23961:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e91)", 2024-09-10T19:12:35.704 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-10T19:12:27.627207+0000", 2024-09-10T19:12:35.704 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 7.1811300789999999, 2024-09-10T19:12:35.704 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.099936922999999997, 2024-09-10T19:12:35.704 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-10T19:12:35.704 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T19:12:35.704 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-10T19:12:35.704 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.14781", 2024-09-10T19:12:35.704 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.183:0/2025776420", 2024-09-10T19:12:35.704 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 23962 2024-09-10T19:12:35.704 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:12:35.704 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-10T19:12:35.704 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:12:35.704 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-10T19:12:35.704 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:27.627207+0000", 2024-09-10T19:12:35.705 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:12:35.705 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:12:35.705 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:12:35.705 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-10T19:12:35.705 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:27.627207+0000", 2024-09-10T19:12:35.705 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:12:35.705 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:12:35.705 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:12:35.705 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-10T19:12:35.705 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:27.627206+0000", 2024-09-10T19:12:35.705 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4294967295.9999995 2024-09-10T19:12:35.705 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:12:35.705 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:12:35.706 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-10T19:12:35.706 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:27.627224+0000", 2024-09-10T19:12:35.706 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.7618999999999999e-05 2024-09-10T19:12:35.706 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:12:35.706 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:12:35.706 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-10T19:12:35.706 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:27.627229+0000", 2024-09-10T19:12:35.706 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.6359999999999999e-06 2024-09-10T19:12:35.706 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:12:35.706 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:12:35.706 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-10T19:12:35.706 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:27.627233+0000", 2024-09-10T19:12:35.706 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.213e-06 2024-09-10T19:12:35.706 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:12:35.707 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:12:35.707 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-10T19:12:35.707 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:27.630314+0000", 2024-09-10T19:12:35.707 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.003081075 2024-09-10T19:12:35.707 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:12:35.707 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:12:35.707 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-10T19:12:35.707 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:27.630419+0000", 2024-09-10T19:12:35.707 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000105053 2024-09-10T19:12:35.707 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:12:35.707 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:12:35.707 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 3,7", 2024-09-10T19:12:35.707 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:27.630545+0000", 2024-09-10T19:12:35.707 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00012658299999999999 2024-09-10T19:12:35.708 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:12:35.708 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:12:35.708 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:12:35.708 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:27.726988+0000", 2024-09-10T19:12:35.708 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.096442782000000005 2024-09-10T19:12:35.708 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:12:35.708 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:12:35.708 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-10T19:12:35.708 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:27.727061+0000", 2024-09-10T19:12:35.708 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.2616999999999999e-05 2024-09-10T19:12:35.708 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:12:35.708 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:12:35.708 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:12:35.709 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:27.727087+0000", 2024-09-10T19:12:35.709 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.6658000000000001e-05 2024-09-10T19:12:35.709 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:12:35.709 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:12:35.709 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-10T19:12:35.709 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:27.727118+0000", 2024-09-10T19:12:35.709 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.0071999999999999e-05 2024-09-10T19:12:35.709 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:12:35.709 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:12:35.709 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-10T19:12:35.709 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:27.727144+0000", 2024-09-10T19:12:35.709 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.6004999999999999e-05 2024-09-10T19:12:35.709 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:12:35.709 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:12:35.710 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:12:35.710 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:12:35.710 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:12:35.710 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.14781.0:23963 7.4 7:247df721:::benchmark_data_smithi183_84160_object23962:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e91)", 2024-09-10T19:12:35.710 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-10T19:12:27.628451+0000", 2024-09-10T19:12:35.710 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 7.179886164, 2024-09-10T19:12:35.710 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.097586700999999998, 2024-09-10T19:12:35.710 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-10T19:12:35.710 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T19:12:35.710 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-10T19:12:35.710 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.14781", 2024-09-10T19:12:35.710 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.183:0/2025776420", 2024-09-10T19:12:35.710 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 23963 2024-09-10T19:12:35.710 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:12:35.710 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-10T19:12:35.711 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:12:35.711 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-10T19:12:35.711 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:27.628451+0000", 2024-09-10T19:12:35.711 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:12:35.711 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:12:35.711 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:12:35.711 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-10T19:12:35.711 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:27.628451+0000", 2024-09-10T19:12:35.711 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:12:35.711 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:12:35.711 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:12:35.711 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-10T19:12:35.711 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:27.628449+0000", 2024-09-10T19:12:35.711 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4294967295.9999986 2024-09-10T19:12:35.712 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:12:35.712 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:12:35.712 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-10T19:12:35.712 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:27.628472+0000", 2024-09-10T19:12:35.712 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.2796e-05 2024-09-10T19:12:35.712 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:12:35.712 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:12:35.712 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-10T19:12:35.712 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:27.628483+0000", 2024-09-10T19:12:35.712 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.0738e-05 2024-09-10T19:12:35.712 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:12:35.712 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:12:35.712 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-10T19:12:35.712 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:27.628489+0000", 2024-09-10T19:12:35.712 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.7649999999999999e-06 2024-09-10T19:12:35.713 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:12:35.713 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:12:35.713 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-10T19:12:35.713 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:27.628856+0000", 2024-09-10T19:12:35.713 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00036741399999999999 2024-09-10T19:12:35.713 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:12:35.713 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:12:35.713 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-10T19:12:35.713 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:27.628972+0000", 2024-09-10T19:12:35.713 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000116161 2024-09-10T19:12:35.713 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:12:35.713 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:12:35.713 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 1,4", 2024-09-10T19:12:35.713 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:27.629098+0000", 2024-09-10T19:12:35.714 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00012609899999999999 2024-09-10T19:12:35.714 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:12:35.714 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:12:35.714 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:12:35.714 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:27.631505+0000", 2024-09-10T19:12:35.714 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.002407083 2024-09-10T19:12:35.714 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:12:35.714 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:12:35.714 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-10T19:12:35.714 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:27.631780+0000", 2024-09-10T19:12:35.714 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00027432500000000001 2024-09-10T19:12:35.714 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:12:35.714 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:12:35.714 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:12:35.714 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:27.725930+0000", 2024-09-10T19:12:35.715 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.094150238999999997 2024-09-10T19:12:35.715 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:12:35.715 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:12:35.715 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-10T19:12:35.715 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:27.726003+0000", 2024-09-10T19:12:35.715 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.3230000000000002e-05 2024-09-10T19:12:35.715 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:12:35.715 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:12:35.715 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-10T19:12:35.715 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:27.726037+0000", 2024-09-10T19:12:35.715 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.4106999999999997e-05 2024-09-10T19:12:35.715 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:12:35.715 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:12:35.716 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:12:35.716 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:12:35.716 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:12:35.716 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.14781.0:23965 7.6 7:672d2240:::benchmark_data_smithi183_84160_object23964:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e91)", 2024-09-10T19:12:35.716 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-10T19:12:27.628494+0000", 2024-09-10T19:12:35.716 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 7.179842378, 2024-09-10T19:12:35.716 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.099013499000000005, 2024-09-10T19:12:35.716 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-10T19:12:35.716 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T19:12:35.716 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-10T19:12:35.716 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.14781", 2024-09-10T19:12:35.716 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.183:0/2025776420", 2024-09-10T19:12:35.716 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 23965 2024-09-10T19:12:35.716 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:12:35.717 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-10T19:12:35.717 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:12:35.717 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-10T19:12:35.717 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:27.628494+0000", 2024-09-10T19:12:35.717 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:12:35.717 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:12:35.717 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:12:35.717 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-10T19:12:35.717 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:27.628494+0000", 2024-09-10T19:12:35.717 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:12:35.717 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:12:35.717 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:12:35.717 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-10T19:12:35.717 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:27.628494+0000", 2024-09-10T19:12:35.718 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4294967295.9999995 2024-09-10T19:12:35.718 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:12:35.718 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:12:35.718 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-10T19:12:35.718 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:27.628511+0000", 2024-09-10T19:12:35.718 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.734e-05 2024-09-10T19:12:35.718 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:12:35.718 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:12:35.718 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-10T19:12:35.718 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:27.628517+0000", 2024-09-10T19:12:35.718 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.1589999999999998e-06 2024-09-10T19:12:35.718 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:12:35.718 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:12:35.718 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-10T19:12:35.719 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:27.628521+0000", 2024-09-10T19:12:35.719 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.681e-06 2024-09-10T19:12:35.719 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:12:35.719 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:12:35.719 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-10T19:12:35.719 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:27.629309+0000", 2024-09-10T19:12:35.719 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00078791600000000005 2024-09-10T19:12:35.719 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:12:35.719 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:12:35.719 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-10T19:12:35.719 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:27.629403+0000", 2024-09-10T19:12:35.719 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.4236999999999999e-05 2024-09-10T19:12:35.719 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:12:35.719 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:12:35.720 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 3,7", 2024-09-10T19:12:35.720 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:27.629513+0000", 2024-09-10T19:12:35.720 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00011011800000000001 2024-09-10T19:12:35.720 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:12:35.720 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:12:35.720 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-10T19:12:35.720 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:27.631780+0000", 2024-09-10T19:12:35.720 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0022670339999999998 2024-09-10T19:12:35.720 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:12:35.720 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:12:35.720 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:12:35.720 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:27.726563+0000", 2024-09-10T19:12:35.720 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.094783038999999999 2024-09-10T19:12:35.720 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:12:35.721 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:12:35.721 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:12:35.721 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:27.727441+0000", 2024-09-10T19:12:35.721 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00087747599999999997 2024-09-10T19:12:35.721 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:12:35.721 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:12:35.721 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-10T19:12:35.721 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:27.727477+0000", 2024-09-10T19:12:35.721 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.5759000000000001e-05 2024-09-10T19:12:35.721 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:12:35.721 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:12:35.721 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-10T19:12:35.721 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:27.727508+0000", 2024-09-10T19:12:35.721 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.1214999999999998e-05 2024-09-10T19:12:35.722 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:12:35.722 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:12:35.722 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:12:35.722 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:12:35.722 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:12:35.722 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.14781.0:23968 7.f 7:f766b5fd:::benchmark_data_smithi183_84160_object23967:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e91)", 2024-09-10T19:12:35.722 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-10T19:12:27.631094+0000", 2024-09-10T19:12:35.722 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 7.1772428799999997, 2024-09-10T19:12:35.722 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.099319093999999997, 2024-09-10T19:12:35.722 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-10T19:12:35.722 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T19:12:35.722 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-10T19:12:35.722 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.14781", 2024-09-10T19:12:35.722 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.183:0/2025776420", 2024-09-10T19:12:35.723 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 23968 2024-09-10T19:12:35.723 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:12:35.723 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-10T19:12:35.723 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:12:35.723 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-10T19:12:35.723 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:27.631094+0000", 2024-09-10T19:12:35.723 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:12:35.723 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:12:35.723 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:12:35.723 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-10T19:12:35.723 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:27.631094+0000", 2024-09-10T19:12:35.723 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:12:35.723 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:12:35.723 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:12:35.724 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-10T19:12:35.724 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:27.631092+0000", 2024-09-10T19:12:35.724 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4294967295.9999986 2024-09-10T19:12:35.724 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:12:35.724 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:12:35.724 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-10T19:12:35.724 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:27.631118+0000", 2024-09-10T19:12:35.724 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.5761999999999998e-05 2024-09-10T19:12:35.724 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:12:35.724 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:12:35.724 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-10T19:12:35.724 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:27.631129+0000", 2024-09-10T19:12:35.724 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.0710000000000001e-05 2024-09-10T19:12:35.725 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:12:35.725 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:12:35.725 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-10T19:12:35.725 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:27.631135+0000", 2024-09-10T19:12:35.725 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.7760000000000003e-06 2024-09-10T19:12:35.725 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:12:35.725 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:12:35.725 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-10T19:12:35.725 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:27.631435+0000", 2024-09-10T19:12:35.725 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00030010600000000002 2024-09-10T19:12:35.725 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:12:35.725 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:12:35.725 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-10T19:12:35.725 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:27.631654+0000", 2024-09-10T19:12:35.725 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00021918600000000001 2024-09-10T19:12:35.726 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:12:35.726 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:12:35.726 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 3,7", 2024-09-10T19:12:35.726 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:27.631779+0000", 2024-09-10T19:12:35.726 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00012483500000000001 2024-09-10T19:12:35.726 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:12:35.726 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:12:35.726 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:12:35.726 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:27.727177+0000", 2024-09-10T19:12:35.726 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.095398005999999994 2024-09-10T19:12:35.726 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:12:35.726 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:12:35.726 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:12:35.726 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:27.728494+0000", 2024-09-10T19:12:35.727 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0013167039999999999 2024-09-10T19:12:35.727 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:12:35.727 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:12:35.727 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-10T19:12:35.727 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:27.730329+0000", 2024-09-10T19:12:35.727 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.001835868 2024-09-10T19:12:35.727 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:12:35.727 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:12:35.727 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-10T19:12:35.727 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:27.730376+0000", 2024-09-10T19:12:35.727 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.6702000000000001e-05 2024-09-10T19:12:35.727 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:12:35.727 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:12:35.727 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-10T19:12:35.727 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:27.730413+0000", 2024-09-10T19:12:35.728 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.6783999999999999e-05 2024-09-10T19:12:35.728 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:12:35.728 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:12:35.728 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:12:35.728 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:12:35.728 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:12:35.728 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_repop_reply(client.14781.0:23961 7.f e91/86 ondisk, result = 0)", 2024-09-10T19:12:35.728 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-10T19:12:27.632000+0000", 2024-09-10T19:12:35.728 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 7.1763370750000002, 2024-09-10T19:12:35.728 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.094823378, 2024-09-10T19:12:35.728 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-10T19:12:35.728 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "started", 2024-09-10T19:12:35.728 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-10T19:12:35.728 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:12:35.729 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-10T19:12:35.729 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:27.632000+0000", 2024-09-10T19:12:35.729 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:12:35.729 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:12:35.729 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:12:35.729 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-10T19:12:35.729 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:27.632000+0000", 2024-09-10T19:12:35.729 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:12:35.729 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:12:35.729 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:12:35.729 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-10T19:12:35.729 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:27.631999+0000", 2024-09-10T19:12:35.729 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4294967295.9999995 2024-09-10T19:12:35.729 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:12:35.730 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:12:35.730 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-10T19:12:35.730 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:27.632006+0000", 2024-09-10T19:12:35.730 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.5869999999999998e-06 2024-09-10T19:12:35.730 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:12:35.730 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:12:35.730 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-10T19:12:35.730 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:27.725716+0000", 2024-09-10T19:12:35.730 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.093710608000000001 2024-09-10T19:12:35.730 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:12:35.730 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:12:35.730 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-10T19:12:35.730 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:27.725726+0000", 2024-09-10T19:12:35.730 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.6590000000000002e-06 2024-09-10T19:12:35.731 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:12:35.731 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:12:35.731 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-10T19:12:35.731 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:27.726811+0000", 2024-09-10T19:12:35.731 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0010847770000000001 2024-09-10T19:12:35.731 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:12:35.731 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:12:35.731 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-10T19:12:35.731 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:27.726816+0000", 2024-09-10T19:12:35.731 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.6250000000000004e-06 2024-09-10T19:12:35.731 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:12:35.731 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:12:35.731 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-10T19:12:35.731 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:27.726823+0000", 2024-09-10T19:12:35.732 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.7440000000000001e-06 2024-09-10T19:12:35.732 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:12:35.732 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:12:35.732 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:12:35.732 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:12:35.732 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:12:35.732 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_repop_reply(client.14781.0:23965 7.6 e91/86 ondisk, result = 0)", 2024-09-10T19:12:35.732 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-10T19:12:27.632028+0000", 2024-09-10T19:12:35.732 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 7.1763085110000002, 2024-09-10T19:12:35.732 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.094536892999999997, 2024-09-10T19:12:35.732 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-10T19:12:35.732 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "started", 2024-09-10T19:12:35.732 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-10T19:12:35.732 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:12:35.733 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-10T19:12:35.733 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:27.632028+0000", 2024-09-10T19:12:35.733 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:12:35.733 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:12:35.733 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:12:35.733 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-10T19:12:35.733 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:27.632028+0000", 2024-09-10T19:12:35.733 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:12:35.733 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:12:35.733 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:12:35.733 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-10T19:12:35.733 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:27.632028+0000", 2024-09-10T19:12:35.733 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4294967295.9999995 2024-09-10T19:12:35.733 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:12:35.734 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:12:35.734 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-10T19:12:35.734 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:27.632032+0000", 2024-09-10T19:12:35.734 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.8759999999999998e-06 2024-09-10T19:12:35.734 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:12:35.734 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:12:35.734 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-10T19:12:35.734 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:27.725936+0000", 2024-09-10T19:12:35.734 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.093904545000000006 2024-09-10T19:12:35.734 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:12:35.734 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:12:35.734 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-10T19:12:35.734 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:27.725943+0000", 2024-09-10T19:12:35.734 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.6109999999999999e-06 2024-09-10T19:12:35.734 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:12:35.735 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:12:35.735 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-10T19:12:35.735 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:27.726553+0000", 2024-09-10T19:12:35.735 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00061012100000000004 2024-09-10T19:12:35.735 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:12:35.735 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:12:35.735 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-10T19:12:35.735 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:27.726558+0000", 2024-09-10T19:12:35.735 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.4990000000000002e-06 2024-09-10T19:12:35.735 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:12:35.735 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:12:35.735 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-10T19:12:35.735 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:27.726565+0000", 2024-09-10T19:12:35.736 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.7710000000000001e-06 2024-09-10T19:12:35.736 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:12:35.736 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:12:35.736 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:12:35.736 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:12:35.736 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:12:35.736 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.14781.0:26405 7.f 7:fc4f1962:::benchmark_data_smithi183_84160_object26404:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e91)", 2024-09-10T19:12:35.736 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-10T19:12:28.896885+0000", 2024-09-10T19:12:35.736 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 5.9114518990000002, 2024-09-10T19:12:35.736 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.093489379999999997, 2024-09-10T19:12:35.736 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-10T19:12:35.736 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T19:12:35.736 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-10T19:12:35.736 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.14781", 2024-09-10T19:12:35.737 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.183:0/2025776420", 2024-09-10T19:12:35.737 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 26405 2024-09-10T19:12:35.737 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:12:35.737 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-10T19:12:35.737 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:12:35.737 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-10T19:12:35.737 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:28.896885+0000", 2024-09-10T19:12:35.737 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:12:35.737 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:12:35.737 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:12:35.737 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-10T19:12:35.737 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:28.896885+0000", 2024-09-10T19:12:35.737 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:12:35.737 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:12:35.738 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:12:35.738 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-10T19:12:35.738 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:28.896884+0000", 2024-09-10T19:12:35.738 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4294967295.9999995 2024-09-10T19:12:35.738 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:12:35.738 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:12:35.738 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-10T19:12:35.738 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:28.896905+0000", 2024-09-10T19:12:35.738 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.0356999999999998e-05 2024-09-10T19:12:35.738 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:12:35.738 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:12:35.738 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-10T19:12:35.738 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:28.896912+0000", 2024-09-10T19:12:35.739 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.0700000000000001e-06 2024-09-10T19:12:35.739 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:12:35.739 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:12:35.739 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-10T19:12:35.739 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:28.896917+0000", 2024-09-10T19:12:35.739 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.361e-06 2024-09-10T19:12:35.739 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:12:35.739 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:12:35.739 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-10T19:12:35.739 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:28.898226+0000", 2024-09-10T19:12:35.739 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0013086930000000001 2024-09-10T19:12:35.739 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:12:35.739 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:12:35.739 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-10T19:12:35.740 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:28.898341+0000", 2024-09-10T19:12:35.740 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000115241 2024-09-10T19:12:35.740 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:12:35.740 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:12:35.740 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 3,7", 2024-09-10T19:12:35.740 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:28.898460+0000", 2024-09-10T19:12:35.740 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000118486 2024-09-10T19:12:35.740 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:12:35.740 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:12:35.740 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-10T19:12:35.740 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:28.988298+0000", 2024-09-10T19:12:35.740 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.089838658000000002 2024-09-10T19:12:35.740 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:12:35.740 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:12:35.741 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:12:35.741 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:28.988388+0000", 2024-09-10T19:12:35.741 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.9622000000000004e-05 2024-09-10T19:12:35.741 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:12:35.741 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:12:35.741 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:12:35.741 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:28.990297+0000", 2024-09-10T19:12:35.741 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.001909427 2024-09-10T19:12:35.741 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:12:35.741 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:12:35.741 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-10T19:12:35.741 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:28.990341+0000", 2024-09-10T19:12:35.741 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.3891999999999999e-05 2024-09-10T19:12:35.741 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:12:35.742 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:12:35.742 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-10T19:12:35.742 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:28.990374+0000", 2024-09-10T19:12:35.742 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.3079e-05 2024-09-10T19:12:35.742 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:12:35.742 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:12:35.742 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:12:35.742 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:12:35.742 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:12:35.742 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.14781.0:29353 7.6 7:68b5a119:::benchmark_data_smithi183_84160_object29352:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e91)", 2024-09-10T19:12:35.742 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-10T19:12:30.354816+0000", 2024-09-10T19:12:35.742 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 4.4535208050000001, 2024-09-10T19:12:35.743 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.095202593000000002, 2024-09-10T19:12:35.743 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-10T19:12:35.743 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T19:12:35.743 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-10T19:12:35.743 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.14781", 2024-09-10T19:12:35.743 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.183:0/2025776420", 2024-09-10T19:12:35.743 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 29353 2024-09-10T19:12:35.743 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:12:35.743 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-10T19:12:35.743 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:12:35.743 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-10T19:12:35.743 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:30.354816+0000", 2024-09-10T19:12:35.743 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:12:35.744 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:12:35.744 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:12:35.744 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-10T19:12:35.744 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:30.354816+0000", 2024-09-10T19:12:35.744 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:12:35.744 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:12:35.744 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:12:35.744 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-10T19:12:35.744 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:30.354815+0000", 2024-09-10T19:12:35.744 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4294967295.999999 2024-09-10T19:12:35.744 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:12:35.744 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:12:35.745 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-10T19:12:35.745 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:30.354838+0000", 2024-09-10T19:12:35.745 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.2864e-05 2024-09-10T19:12:35.745 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:12:35.745 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:12:35.745 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-10T19:12:35.745 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:30.354849+0000", 2024-09-10T19:12:35.745 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.1803e-05 2024-09-10T19:12:35.745 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:12:35.745 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:12:35.745 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-10T19:12:35.745 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:30.354855+0000", 2024-09-10T19:12:35.745 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.9649999999999996e-06 2024-09-10T19:12:35.746 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:12:35.746 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:12:35.746 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-10T19:12:35.746 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:30.355151+0000", 2024-09-10T19:12:35.746 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00029531099999999998 2024-09-10T19:12:35.746 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:12:35.746 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:12:35.746 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-10T19:12:35.746 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:30.355267+0000", 2024-09-10T19:12:35.746 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000116177 2024-09-10T19:12:35.746 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:12:35.746 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:12:35.746 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 3,7", 2024-09-10T19:12:35.746 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:30.355402+0000", 2024-09-10T19:12:35.747 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000135208 2024-09-10T19:12:35.747 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:12:35.747 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:12:35.747 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-10T19:12:35.747 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:30.357243+0000", 2024-09-10T19:12:35.747 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.001840649 2024-09-10T19:12:35.747 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:12:35.747 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:12:35.747 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:12:35.747 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:30.358029+0000", 2024-09-10T19:12:35.747 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00078624399999999996 2024-09-10T19:12:35.747 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:12:35.747 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:12:35.747 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:12:35.747 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:30.449934+0000", 2024-09-10T19:12:35.748 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.091905136999999998 2024-09-10T19:12:35.748 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:12:35.748 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:12:35.748 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-10T19:12:35.748 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:30.449976+0000", 2024-09-10T19:12:35.748 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.1504000000000001e-05 2024-09-10T19:12:35.748 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:12:35.748 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:12:35.748 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-10T19:12:35.748 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:30.450018+0000", 2024-09-10T19:12:35.748 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.2910000000000001e-05 2024-09-10T19:12:35.748 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:12:35.748 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:12:35.749 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:12:35.749 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:12:35.749 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:12:35.749 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.14781.0:29362 7.4 7:2b387afe:::benchmark_data_smithi183_84160_object29361:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e91)", 2024-09-10T19:12:35.749 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-10T19:12:30.357986+0000", 2024-09-10T19:12:35.749 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 4.4503502949999998, 2024-09-10T19:12:35.749 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.093925758999999998, 2024-09-10T19:12:35.749 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-10T19:12:35.749 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T19:12:35.749 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-10T19:12:35.749 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.14781", 2024-09-10T19:12:35.749 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.183:0/2025776420", 2024-09-10T19:12:35.749 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 29362 2024-09-10T19:12:35.750 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:12:35.750 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-10T19:12:35.750 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:12:35.750 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-10T19:12:35.750 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:30.357986+0000", 2024-09-10T19:12:35.750 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:12:35.750 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:12:35.750 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:12:35.750 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-10T19:12:35.750 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:30.357986+0000", 2024-09-10T19:12:35.750 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:12:35.750 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:12:35.750 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:12:35.750 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-10T19:12:35.751 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:30.357985+0000", 2024-09-10T19:12:35.751 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4294967295.9999986 2024-09-10T19:12:35.751 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:12:35.751 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:12:35.751 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-10T19:12:35.751 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:30.358007+0000", 2024-09-10T19:12:35.751 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.2208000000000002e-05 2024-09-10T19:12:35.751 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:12:35.751 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:12:35.751 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-10T19:12:35.751 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:30.358017+0000", 2024-09-10T19:12:35.751 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.4350000000000003e-06 2024-09-10T19:12:35.751 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:12:35.752 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:12:35.752 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-10T19:12:35.752 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:30.358021+0000", 2024-09-10T19:12:35.752 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.8910000000000003e-06 2024-09-10T19:12:35.752 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:12:35.752 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:12:35.752 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-10T19:12:35.752 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:30.358056+0000", 2024-09-10T19:12:35.752 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.4564000000000003e-05 2024-09-10T19:12:35.752 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:12:35.752 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:12:35.752 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-10T19:12:35.752 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:30.358167+0000", 2024-09-10T19:12:35.752 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000111397 2024-09-10T19:12:35.753 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:12:35.753 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:12:35.753 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 1,4", 2024-09-10T19:12:35.753 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:30.446342+0000", 2024-09-10T19:12:35.753 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.088174653000000006 2024-09-10T19:12:35.753 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:12:35.753 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:12:35.753 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:12:35.753 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:30.448224+0000", 2024-09-10T19:12:35.753 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.001882337 2024-09-10T19:12:35.753 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:12:35.753 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:12:35.753 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-10T19:12:35.753 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:30.449510+0000", 2024-09-10T19:12:35.754 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0012854400000000001 2024-09-10T19:12:35.754 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:12:35.754 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:12:35.754 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:12:35.754 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:30.451834+0000", 2024-09-10T19:12:35.754 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.002324376 2024-09-10T19:12:35.754 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:12:35.754 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:12:35.754 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-10T19:12:35.754 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:30.451880+0000", 2024-09-10T19:12:35.754 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.5914000000000001e-05 2024-09-10T19:12:35.754 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:12:35.754 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:12:35.754 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-10T19:12:35.755 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:30.451912+0000", 2024-09-10T19:12:35.755 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.2011000000000002e-05 2024-09-10T19:12:35.755 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:12:35.755 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:12:35.755 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:12:35.755 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:12:35.755 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:12:35.755 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.14781.0:29363 7.6 7:637ae986:::benchmark_data_smithi183_84160_object29362:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e91)", 2024-09-10T19:12:35.755 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-10T19:12:30.358033+0000", 2024-09-10T19:12:35.755 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 4.4503032850000004, 2024-09-10T19:12:35.755 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.093725952000000001, 2024-09-10T19:12:35.755 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-10T19:12:35.755 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T19:12:35.755 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-10T19:12:35.755 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.14781", 2024-09-10T19:12:35.756 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.183:0/2025776420", 2024-09-10T19:12:35.756 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 29363 2024-09-10T19:12:35.756 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:12:35.756 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-10T19:12:35.756 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:12:35.756 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-10T19:12:35.756 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:30.358033+0000", 2024-09-10T19:12:35.756 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:12:35.756 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:12:35.756 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:12:35.756 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-10T19:12:35.756 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:30.358033+0000", 2024-09-10T19:12:35.756 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:12:35.757 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:12:35.757 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:12:35.757 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-10T19:12:35.757 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:30.358033+0000", 2024-09-10T19:12:35.757 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4294967295.9999995 2024-09-10T19:12:35.757 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:12:35.757 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:12:35.757 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-10T19:12:35.757 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:30.358052+0000", 2024-09-10T19:12:35.757 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.9621000000000001e-05 2024-09-10T19:12:35.757 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:12:35.757 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:12:35.757 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-10T19:12:35.757 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:30.358060+0000", 2024-09-10T19:12:35.758 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.2189999999999998e-06 2024-09-10T19:12:35.758 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:12:35.758 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:12:35.758 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-10T19:12:35.758 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:30.358064+0000", 2024-09-10T19:12:35.758 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.0720000000000001e-06 2024-09-10T19:12:35.758 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:12:35.758 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:12:35.758 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-10T19:12:35.758 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:30.358098+0000", 2024-09-10T19:12:35.758 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.4069000000000002e-05 2024-09-10T19:12:35.758 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:12:35.758 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:12:35.758 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-10T19:12:35.759 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:30.358203+0000", 2024-09-10T19:12:35.759 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00010556300000000001 2024-09-10T19:12:35.759 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:12:35.759 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:12:35.759 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 3,7", 2024-09-10T19:12:35.759 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:30.446939+0000", 2024-09-10T19:12:35.759 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.088736051999999996 2024-09-10T19:12:35.759 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:12:35.759 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:12:35.759 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:12:35.759 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:30.450047+0000", 2024-09-10T19:12:35.759 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0031079469999999998 2024-09-10T19:12:35.759 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:12:35.759 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:12:35.760 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-10T19:12:35.760 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:30.451234+0000", 2024-09-10T19:12:35.760 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00118684 2024-09-10T19:12:35.760 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:12:35.760 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:12:35.760 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:12:35.760 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:30.451546+0000", 2024-09-10T19:12:35.760 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00031156000000000001 2024-09-10T19:12:35.760 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:12:35.760 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:12:35.760 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-10T19:12:35.760 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:30.451676+0000", 2024-09-10T19:12:35.760 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00013042300000000001 2024-09-10T19:12:35.760 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:12:35.761 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:12:35.761 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-10T19:12:35.761 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:30.451759+0000", 2024-09-10T19:12:35.761 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.3176999999999993e-05 2024-09-10T19:12:35.761 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:12:35.761 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:12:35.761 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:12:35.761 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:12:35.761 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:12:35.761 INFO:teuthology.orchestra.run.smithi183.stdout:} 2024-09-10T19:12:35.761 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:12:35.801 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:12:35.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:12:35.802 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:12:35.902 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: 17 16 39905 39889 146.621 120.188 0.00571307 0.0068153 2024-09-10T19:12:35.902 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:12:35.942 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:12:35.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:12:35.943 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:12:36.042 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:12:35 smithi183 ceph-mon[29736]: purged_snaps scrub starts 2024-09-10T19:12:36.042 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:12:35 smithi183 ceph-mon[29736]: purged_snaps scrub ok 2024-09-10T19:12:36.042 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:12:35 smithi183 ceph-mon[29736]: osdmap e94: 8 total, 8 up, 8 in 2024-09-10T19:12:36.042 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:12:35 smithi183 ceph-mon[41562]: purged_snaps scrub starts 2024-09-10T19:12:36.042 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:12:35 smithi183 ceph-mon[41562]: purged_snaps scrub ok 2024-09-10T19:12:36.042 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:12:35 smithi183 ceph-mon[41562]: osdmap e94: 8 total, 8 up, 8 in 2024-09-10T19:12:36.044 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:12:36.072 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:12:36.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:12:36.073 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:12:36.174 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:12:36.210 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:12:36.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:12:36.211 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:12:36.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:12:35 smithi191 ceph-mon[32884]: purged_snaps scrub starts 2024-09-10T19:12:36.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:12:35 smithi191 ceph-mon[32884]: purged_snaps scrub ok 2024-09-10T19:12:36.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:12:35 smithi191 ceph-mon[32884]: osdmap e94: 8 total, 8 up, 8 in 2024-09-10T19:12:36.312 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:12:36.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:12:36.342 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:12:36.343 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:12:36.444 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:12:36.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:12:36.484 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:12:36.485 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:12:36.511 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph daemon osd.1 dump_ops_in_flight 2024-09-10T19:12:36.516 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:12:36.516 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:12:36.585 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:12:36.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:12:36.642 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:12:36.642 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:12:36.743 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:12:36.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:12:36.800 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:12:36.800 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:12:36.901 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:12:36.902 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: 18 16 41675 41659 144.62 110.625 0.0130509 0.00690811 2024-09-10T19:12:36.925 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:12:36.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:12:36.926 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:12:37.027 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:12:37.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:12:37.071 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:12:37.071 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:12:37.137 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:12:36 smithi183 ceph-mon[29736]: pgmap v80: 148 pgs: 148 active+clean; 2.3 GiB data, 9.1 GiB used, 706 GiB / 715 GiB avail; 852 B/s rd, 101 MiB/s wr, 1.61k op/s 2024-09-10T19:12:37.138 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:12:36 smithi183 ceph-mon[29736]: purged_snaps scrub starts 2024-09-10T19:12:37.138 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:12:36 smithi183 ceph-mon[29736]: purged_snaps scrub ok 2024-09-10T19:12:37.138 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:12:36 smithi183 ceph-mon[29736]: osdmap e95: 8 total, 8 up, 8 in 2024-09-10T19:12:37.138 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:12:36 smithi183 ceph-mon[41562]: pgmap v80: 148 pgs: 148 active+clean; 2.3 GiB data, 9.1 GiB used, 706 GiB / 715 GiB avail; 852 B/s rd, 101 MiB/s wr, 1.61k op/s 2024-09-10T19:12:37.138 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:12:36 smithi183 ceph-mon[41562]: purged_snaps scrub starts 2024-09-10T19:12:37.138 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:12:36 smithi183 ceph-mon[41562]: purged_snaps scrub ok 2024-09-10T19:12:37.138 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:12:36 smithi183 ceph-mon[41562]: osdmap e95: 8 total, 8 up, 8 in 2024-09-10T19:12:37.173 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:12:37.200 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:12:37.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:12:37.201 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:12:37.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:12:36 smithi191 ceph-mon[32884]: pgmap v80: 148 pgs: 148 active+clean; 2.3 GiB data, 9.1 GiB used, 706 GiB / 715 GiB avail; 852 B/s rd, 101 MiB/s wr, 1.61k op/s 2024-09-10T19:12:37.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:12:36 smithi191 ceph-mon[32884]: purged_snaps scrub starts 2024-09-10T19:12:37.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:12:36 smithi191 ceph-mon[32884]: purged_snaps scrub ok 2024-09-10T19:12:37.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:12:36 smithi191 ceph-mon[32884]: osdmap e95: 8 total, 8 up, 8 in 2024-09-10T19:12:37.302 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:12:37.326 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:12:37.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:12:37.327 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:12:37.428 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:12:37.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:12:37.457 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:12:37.457 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:12:37.558 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:12:37.596 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:12:37.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:12:37.597 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:12:37.698 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:12:37.715 INFO:journalctl@ceph.mgr.x.smithi191.stdout:Sep 10 19:12:37 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-x[34967]: ::ffff:172.21.15.191 - - [10/Sep/2024:19:12:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T19:12:37.735 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:12:37.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:12:37.736 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:12:37.837 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:12:37.865 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:12:37.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:12:37.866 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:12:37.902 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: 19 16 43498 43482 143.005 113.938 0.00908405 0.00698772 2024-09-10T19:12:37.967 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:12:37.994 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:12:37.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:12:37.995 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:12:38.096 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:12:38.135 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:12:38.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:12:38.136 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:12:38.236 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:12:38.271 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:12:38.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:12:38.272 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:12:38.373 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:12:38.409 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:12:38.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:12:38.410 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:12:38.511 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:12:38.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:12:38.574 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:12:38.574 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:12:38.675 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:12:38.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:12:38.712 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:12:38.712 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:12:38.813 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:12:38.839 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:12:38.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:12:38.841 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:12:38.902 INFO:tasks.radosbench.radosbench.0.smithi183.stdout:2024-09-10T19:12:38.902257+0000 min lat: 0.00124697 max lat: 0.112346 avg lat: 0.00706613 2024-09-10T19:12:38.902 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-10T19:12:38.903 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: 20 15 45276 45261 141.414 111.188 0.00404018 0.00706613 2024-09-10T19:12:38.931 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:12:38 smithi183 ceph-mon[29736]: pgmap v82: 148 pgs: 148 active+clean; 2.3 GiB data, 9.1 GiB used, 706 GiB / 715 GiB avail; 865 B/s rd, 102 MiB/s wr, 1.64k op/s 2024-09-10T19:12:38.931 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:12:38 smithi183 ceph-mon[41562]: pgmap v82: 148 pgs: 148 active+clean; 2.3 GiB data, 9.1 GiB used, 706 GiB / 715 GiB avail; 865 B/s rd, 102 MiB/s wr, 1.64k op/s 2024-09-10T19:12:38.941 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:12:38.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:12:38.984 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:12:38.984 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:12:39.085 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:12:39.111 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:12:39.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:12:39.112 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:12:39.213 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:12:39.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:12:38 smithi191 ceph-mon[32884]: pgmap v82: 148 pgs: 148 active+clean; 2.3 GiB data, 9.1 GiB used, 706 GiB / 715 GiB avail; 865 B/s rd, 102 MiB/s wr, 1.64k op/s 2024-09-10T19:12:39.216 INFO:journalctl@ceph.iscsi.iscsi.a.smithi191.stdout:Sep 10 19:12:38 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-iscsi-iscsi-a[61816]: debug there is no tcmu-runner data available 2024-09-10T19:12:39.242 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:12:39.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:12:39.243 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:12:39.344 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:12:39.380 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:12:39.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:12:39.381 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:12:39.482 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:12:39.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:12:39.523 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:12:39.523 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:12:39.620 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:12:39.624 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:12:39.649 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:12:39.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:12:39.650 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:12:39.751 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:12:39.780 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:12:39.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:12:39.781 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:12:39.882 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:12:39.902 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: 21 15 46980 46965 139.751 106.5 0.00433687 0.00714979 2024-09-10T19:12:39.909 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:12:39.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:12:39.910 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:12:39.931 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:12:39 smithi183 ceph-mon[29736]: from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:12:39.931 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:12:39 smithi183 ceph-mon[29736]: osdmap e96: 8 total, 8 up, 8 in 2024-09-10T19:12:39.931 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:12:39 smithi183 ceph-mon[41562]: from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:12:39.931 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:12:39 smithi183 ceph-mon[41562]: osdmap e96: 8 total, 8 up, 8 in 2024-09-10T19:12:40.011 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:12:40.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:12:40.040 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:12:40.040 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:12:40.141 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:12:40.168 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:12:40.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:12:40.169 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:12:40.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:12:39 smithi191 ceph-mon[32884]: from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:12:40.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:12:39 smithi191 ceph-mon[32884]: osdmap e96: 8 total, 8 up, 8 in 2024-09-10T19:12:40.271 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:12:40.305 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:12:40.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:12:40.307 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:12:40.407 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:12:40.434 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:12:40.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:12:40.436 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:12:40.462 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-10T19:12:40.462 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-10T19:12:40.463 INFO:tasks.thrashosds.thrasher:Killing osd 1, live_osds are [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-10T19:12:40.463 INFO:tasks.cephadm.osd.1:Stopping osd.1... 2024-09-10T19:12:40.463 DEBUG:teuthology.orchestra.run.smithi183:> sudo systemctl stop ceph-320f6df8-6fa7-11ef-bcea-c7b262605968@osd.1 2024-09-10T19:12:40.536 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:12:40.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:12:40.566 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:12:40.566 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:12:40.667 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:12:40.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:12:40.722 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:12:40.722 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:12:40.774 INFO:journalctl@ceph.osd.1.smithi183.stdout:Sep 10 19:12:40 smithi183 systemd[1]: Stopping Ceph osd.1 for 320f6df8-6fa7-11ef-bcea-c7b262605968... 2024-09-10T19:12:40.823 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:12:40.852 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:12:40.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:12:40.853 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:12:40.905 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: 22 16 48545 48529 137.826 97.75 0.0201375 0.00724904 2024-09-10T19:12:40.953 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:12:41.012 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:12:41.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:12:41.014 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:12:41.114 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:12:41.129 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:12:40 smithi183 ceph-mon[29736]: pgmap v83: 148 pgs: 148 active+clean; 2.8 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 1.4 KiB/s rd, 169 MiB/s wr, 2.70k op/s 2024-09-10T19:12:41.130 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:12:40 smithi183 ceph-mon[41562]: pgmap v83: 148 pgs: 148 active+clean; 2.8 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 1.4 KiB/s rd, 169 MiB/s wr, 2.70k op/s 2024-09-10T19:12:41.141 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:12:41.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:12:41.142 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:12:41.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:12:40 smithi191 ceph-mon[32884]: pgmap v83: 148 pgs: 148 active+clean; 2.8 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 1.4 KiB/s rd, 169 MiB/s wr, 2.70k op/s 2024-09-10T19:12:41.243 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:12:41.271 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:12:41.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:12:41.272 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:12:41.373 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:12:41.400 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:12:41.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:12:41.401 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:12:41.431 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Sep 10 19:12:41 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-y[29989]: ::ffff:172.21.15.191 - - [10/Sep/2024:19:12:41] "GET /metrics HTTP/1.1" 200 223218 "" "Prometheus/2.43.0" 2024-09-10T19:12:41.502 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:12:41.531 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:12:41.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:12:41.532 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:12:41.574 INFO:teuthology.orchestra.run.smithi183.stdout:osd.0: {} 2024-09-10T19:12:41.574 INFO:teuthology.orchestra.run.smithi183.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-10T19:12:41.610 INFO:teuthology.orchestra.run.smithi183.stdout:osd.1: {} 2024-09-10T19:12:41.610 INFO:teuthology.orchestra.run.smithi183.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-10T19:12:41.633 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:12:41.643 INFO:teuthology.orchestra.run.smithi183.stdout:osd.2: {} 2024-09-10T19:12:41.643 INFO:teuthology.orchestra.run.smithi183.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-10T19:12:41.660 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:12:41.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:12:41.661 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:12:41.677 INFO:teuthology.orchestra.run.smithi183.stdout:osd.3: {} 2024-09-10T19:12:41.677 INFO:teuthology.orchestra.run.smithi183.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-10T19:12:41.708 INFO:teuthology.orchestra.run.smithi183.stdout:osd.4: {} 2024-09-10T19:12:41.709 INFO:teuthology.orchestra.run.smithi183.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-10T19:12:41.739 INFO:teuthology.orchestra.run.smithi183.stdout:osd.5: {} 2024-09-10T19:12:41.739 INFO:teuthology.orchestra.run.smithi183.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-10T19:12:41.762 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:12:41.771 INFO:teuthology.orchestra.run.smithi183.stdout:osd.6: {} 2024-09-10T19:12:41.771 INFO:teuthology.orchestra.run.smithi183.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-10T19:12:41.800 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:12:41.801 INFO:teuthology.orchestra.run.smithi183.stdout:osd.7: {} 2024-09-10T19:12:41.801 INFO:teuthology.orchestra.run.smithi183.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-10T19:12:41.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:12:41.802 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:12:41.903 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:12:41.905 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: 23 16 50133 50117 136.149 99.25 0.00647962 0.00733946 2024-09-10T19:12:41.931 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:12:41.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:12:41.933 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:12:42.034 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:12:42.058 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:12:42.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:12:42.059 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:12:42.158 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:12:41 smithi183 ceph-mon[41562]: from='client.? 172.21.15.183:0/1993154923' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T19:12:42.159 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:12:41 smithi183 ceph-mon[29736]: from='client.? 172.21.15.183:0/1993154923' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T19:12:42.160 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:12:42.187 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:12:42.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:12:42.188 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:12:42.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:12:41 smithi191 ceph-mon[32884]: from='client.? 172.21.15.183:0/1993154923' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T19:12:42.289 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:12:42.318 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:12:42.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:12:42.319 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:12:42.419 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:12:42.449 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:12:42.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:12:42.450 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:12:42.550 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:12:42.577 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:12:42.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:12:42.579 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:12:42.679 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:12:42.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:12:42.718 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:12:42.718 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:12:42.819 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:12:42.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:12:42.857 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:12:42.858 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:12:42.905 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: 24 16 51852 51836 134.952 107.438 0.012482 0.00740479 2024-09-10T19:12:42.931 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:12:42 smithi183 ceph-mon[29736]: pgmap v85: 148 pgs: 148 active+clean; 2.8 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 743 B/s rd, 82 MiB/s wr, 1.31k op/s 2024-09-10T19:12:42.931 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:12:42 smithi183 ceph-mon[41562]: pgmap v85: 148 pgs: 148 active+clean; 2.8 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 743 B/s rd, 82 MiB/s wr, 1.31k op/s 2024-09-10T19:12:42.959 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:12:42.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:12:42.997 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:12:42.997 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:12:43.098 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:12:43.122 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:12:43.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:12:43.123 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:12:43.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:12:42 smithi191 ceph-mon[32884]: pgmap v85: 148 pgs: 148 active+clean; 2.8 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 743 B/s rd, 82 MiB/s wr, 1.31k op/s 2024-09-10T19:12:43.225 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:12:43.247 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:12:43.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:12:43.248 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:12:43.349 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:12:43.379 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:12:43.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:12:43.380 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:12:43.481 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:12:43.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:12:43.522 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:12:43.522 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:12:43.623 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:12:43.649 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:12:43.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:12:43.650 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:12:43.751 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:12:43.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:12:43.801 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:12:43.802 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:12:43.815 INFO:teuthology.orchestra.run.smithi183.stderr:noscrub is unset 2024-09-10T19:12:43.902 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:12:43.905 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: 25 16 53279 53263 133.122 89.1875 0.00923094 0.00750682 2024-09-10T19:12:43.927 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:12:43.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:12:43.928 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:12:44.029 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:12:44.056 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:12:44.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:12:44.058 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:12:44.066 INFO:teuthology.orchestra.run.smithi183.stdout:ERROR: (22) Invalid argument 2024-09-10T19:12:44.066 INFO:teuthology.orchestra.run.smithi183.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-10T19:12:44.067 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:12:43 smithi183 ceph-mon[41562]: from='client.? 172.21.15.183:0/3938044224' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-10T19:12:44.067 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:12:43 smithi183 ceph-mon[41562]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:12:44.067 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:12:43 smithi183 ceph-mon[29736]: from='client.? 172.21.15.183:0/3938044224' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-10T19:12:44.067 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:12:43 smithi183 ceph-mon[29736]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:12:44.159 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:12:44.201 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:12:44.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:12:44.202 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:12:44.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:12:43 smithi191 ceph-mon[32884]: from='client.? 172.21.15.183:0/3938044224' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-10T19:12:44.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:12:43 smithi191 ceph-mon[32884]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:12:44.303 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:12:44.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:12:44.338 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:12:44.338 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:12:44.439 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:12:44.465 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:12:44.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:12:44.466 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:12:44.567 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:12:44.593 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:12:44.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:12:44.594 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:12:44.633 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-10T19:12:44.695 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:12:44.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:12:44.752 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:12:44.752 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:12:44.853 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:12:44.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:12:44.892 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:12:44.893 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:12:44.908 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: 26 16 54734 54718 131.499 90.9375 0.00896302 0.00759964 2024-09-10T19:12:44.993 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:12:45.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:12:45.039 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:12:45.039 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:12:45.060 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:12:44 smithi183 ceph-mon[29736]: pgmap v86: 148 pgs: 148 active+clean; 2.8 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 639 B/s rd, 70 MiB/s wr, 1.12k op/s 2024-09-10T19:12:45.060 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:12:44 smithi183 ceph-mon[29736]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T19:12:45.060 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:12:44 smithi183 ceph-mon[29736]: from='client.? 172.21.15.183:0/3938044224' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-10T19:12:45.060 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:12:44 smithi183 ceph-mon[29736]: osdmap e97: 8 total, 8 up, 8 in 2024-09-10T19:12:45.060 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:12:44 smithi183 ceph-mon[29736]: 5.7 scrub starts 2024-09-10T19:12:45.060 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:12:44 smithi183 ceph-mon[29736]: 5.7 scrub ok 2024-09-10T19:12:45.060 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:12:44 smithi183 ceph-mon[29736]: osdmap e98: 8 total, 8 up, 8 in 2024-09-10T19:12:45.061 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:12:44 smithi183 ceph-mon[41562]: pgmap v86: 148 pgs: 148 active+clean; 2.8 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 639 B/s rd, 70 MiB/s wr, 1.12k op/s 2024-09-10T19:12:45.061 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:12:44 smithi183 ceph-mon[41562]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T19:12:45.061 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:12:44 smithi183 ceph-mon[41562]: from='client.? 172.21.15.183:0/3938044224' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-10T19:12:45.061 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:12:44 smithi183 ceph-mon[41562]: osdmap e97: 8 total, 8 up, 8 in 2024-09-10T19:12:45.061 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:12:44 smithi183 ceph-mon[41562]: 5.7 scrub starts 2024-09-10T19:12:45.061 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:12:44 smithi183 ceph-mon[41562]: 5.7 scrub ok 2024-09-10T19:12:45.061 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:12:44 smithi183 ceph-mon[41562]: osdmap e98: 8 total, 8 up, 8 in 2024-09-10T19:12:45.140 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:12:45.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:12:45.202 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:12:45.202 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:12:45.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:12:44 smithi191 ceph-mon[32884]: pgmap v86: 148 pgs: 148 active+clean; 2.8 GiB data, 11 GiB used, 705 GiB / 715 GiB avail; 639 B/s rd, 70 MiB/s wr, 1.12k op/s 2024-09-10T19:12:45.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:12:44 smithi191 ceph-mon[32884]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T19:12:45.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:12:44 smithi191 ceph-mon[32884]: from='client.? 172.21.15.183:0/3938044224' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-10T19:12:45.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:12:44 smithi191 ceph-mon[32884]: osdmap e97: 8 total, 8 up, 8 in 2024-09-10T19:12:45.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:12:44 smithi191 ceph-mon[32884]: 5.7 scrub starts 2024-09-10T19:12:45.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:12:44 smithi191 ceph-mon[32884]: 5.7 scrub ok 2024-09-10T19:12:45.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:12:44 smithi191 ceph-mon[32884]: osdmap e98: 8 total, 8 up, 8 in 2024-09-10T19:12:45.303 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:12:45.339 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:12:45.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:12:45.340 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:12:45.431 INFO:journalctl@ceph.osd.1.smithi183.stdout:Sep 10 19:12:45 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd-1[52280]: 2024-09-10T19:12:45.171+0000 7f77b25c1640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-10T19:12:45.431 INFO:journalctl@ceph.osd.1.smithi183.stdout:Sep 10 19:12:45 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd-1[52280]: 2024-09-10T19:12:45.171+0000 7f77b25c1640 -1 osd.1 98 *** Got signal Terminated *** 2024-09-10T19:12:45.431 INFO:journalctl@ceph.osd.1.smithi183.stdout:Sep 10 19:12:45 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd-1[52280]: 2024-09-10T19:12:45.171+0000 7f77b25c1640 -1 osd.1 98 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-10T19:12:45.442 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:12:45.469 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:12:45.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:12:45.470 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:12:45.571 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:12:45.611 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:12:45.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:12:45.612 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:12:45.713 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:12:45.741 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:12:45.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:12:45.742 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:12:45.843 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:12:45.864 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:12:45.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:12:45.865 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:12:45.905 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: 27 16 56119 56103 129.835 86.5625 0.00238122 0.00767416 2024-09-10T19:12:45.931 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:12:45 smithi183 ceph-mon[41562]: 2.15 scrub starts 2024-09-10T19:12:45.931 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:12:45 smithi183 ceph-mon[41562]: 2.15 scrub ok 2024-09-10T19:12:45.931 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:12:45 smithi183 ceph-mon[41562]: 6.c scrub starts 2024-09-10T19:12:45.931 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:12:45 smithi183 ceph-mon[41562]: 6.c scrub ok 2024-09-10T19:12:45.931 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:12:45 smithi183 ceph-mon[41562]: osd.1 marked itself down and dead 2024-09-10T19:12:45.931 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:12:45 smithi183 ceph-mon[41562]: 2.3 scrub starts 2024-09-10T19:12:45.931 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:12:45 smithi183 ceph-mon[41562]: 2.3 scrub ok 2024-09-10T19:12:45.932 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:12:45 smithi183 ceph-mon[29736]: 2.15 scrub starts 2024-09-10T19:12:45.932 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:12:45 smithi183 ceph-mon[29736]: 2.15 scrub ok 2024-09-10T19:12:45.932 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:12:45 smithi183 ceph-mon[29736]: 6.c scrub starts 2024-09-10T19:12:45.932 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:12:45 smithi183 ceph-mon[29736]: 6.c scrub ok 2024-09-10T19:12:45.932 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:12:45 smithi183 ceph-mon[29736]: osd.1 marked itself down and dead 2024-09-10T19:12:45.932 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:12:45 smithi183 ceph-mon[29736]: 2.3 scrub starts 2024-09-10T19:12:45.932 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:12:45 smithi183 ceph-mon[29736]: 2.3 scrub ok 2024-09-10T19:12:45.966 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:12:45.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:12:45.988 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:12:45.988 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:12:46.089 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:12:46.113 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:12:46.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:12:46.114 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:12:46.214 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:12:46.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:12:45 smithi191 ceph-mon[32884]: 2.15 scrub starts 2024-09-10T19:12:46.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:12:45 smithi191 ceph-mon[32884]: 2.15 scrub ok 2024-09-10T19:12:46.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:12:45 smithi191 ceph-mon[32884]: 6.c scrub starts 2024-09-10T19:12:46.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:12:45 smithi191 ceph-mon[32884]: 6.c scrub ok 2024-09-10T19:12:46.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:12:45 smithi191 ceph-mon[32884]: osd.1 marked itself down and dead 2024-09-10T19:12:46.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:12:45 smithi191 ceph-mon[32884]: 2.3 scrub starts 2024-09-10T19:12:46.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:12:45 smithi191 ceph-mon[32884]: 2.3 scrub ok 2024-09-10T19:12:46.236 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:12:46.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:12:46.237 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:12:46.338 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:12:46.363 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:12:46.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:12:46.364 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:12:46.465 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:12:46.486 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:12:46.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:12:46.487 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:12:46.586 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph daemon osd.1 dump_blocked_ops 2024-09-10T19:12:46.588 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:12:46.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:12:46.610 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:12:46.610 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:12:46.711 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:12:46.732 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:12:46.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:12:46.733 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:12:46.825 INFO:journalctl@ceph.osd.1.smithi183.stdout:Sep 10 19:12:46 smithi183 podman[87497]: 2024-09-10 19:12:46.581394153 +0000 UTC m=+2.824566900 container died bd74997440b09d1e1f2a149b31787c72e3fffc8cd1adb142f562cd2b80fab856 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3bfeaf4417585c821cab9ca76e4d95f390b0e769807122a45463640820b5cd16, name=ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd-1, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.schema-version=1.0, ceph=True, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, RELEASE=wip-guits-quincy-2024-09-09-1810-quincy-5be5739, GIT_CLEAN=True, maintainer=Guillaume Abrioux ) 2024-09-10T19:12:46.826 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:12:46 smithi183 ceph-mon[29736]: 4.16 scrub starts 2024-09-10T19:12:46.827 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:12:46 smithi183 ceph-mon[29736]: 4.16 scrub ok 2024-09-10T19:12:46.827 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:12:46 smithi183 ceph-mon[29736]: 4.4 scrub starts 2024-09-10T19:12:46.827 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:12:46 smithi183 ceph-mon[29736]: 4.4 scrub ok 2024-09-10T19:12:46.827 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:12:46 smithi183 ceph-mon[29736]: 4.d scrub starts 2024-09-10T19:12:46.827 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:12:46 smithi183 ceph-mon[29736]: 4.d scrub ok 2024-09-10T19:12:46.827 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:12:46 smithi183 ceph-mon[29736]: 4.1c scrub starts 2024-09-10T19:12:46.827 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:12:46 smithi183 ceph-mon[29736]: 4.1c scrub ok 2024-09-10T19:12:46.833 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:12:46.862 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:12:46.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:12:46.863 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:12:46.906 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: 28 16 56248 56232 125.486 8.0625 0.00766384 0.00796452 2024-09-10T19:12:46.964 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:12:46.985 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:12:46.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:12:46.986 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:12:47.087 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:12:47.107 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-10T19:12:47.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:12:47.123 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:12:47.123 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:12:47.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:12:46 smithi183 ceph-mon[29736]: 6.12 scrub starts 2024-09-10T19:12:47.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:12:46 smithi183 ceph-mon[29736]: 6.12 scrub ok 2024-09-10T19:12:47.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:12:46 smithi183 ceph-mon[29736]: pgmap v89: 148 pgs: 148 active+clean; 3.3 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 852 B/s rd, 81 MiB/s wr, 1.30k op/s 2024-09-10T19:12:47.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:12:46 smithi183 ceph-mon[29736]: Health check failed: 1 osds down (OSD_DOWN) 2024-09-10T19:12:47.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:12:46 smithi183 ceph-mon[29736]: osdmap e99: 8 total, 7 up, 8 in 2024-09-10T19:12:47.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:12:46 smithi183 ceph-mon[29736]: 2.12 scrub starts 2024-09-10T19:12:47.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:12:46 smithi183 ceph-mon[29736]: 2.12 scrub ok 2024-09-10T19:12:47.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:12:46 smithi183 ceph-mon[29736]: osdmap e100: 8 total, 7 up, 8 in 2024-09-10T19:12:47.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:12:46 smithi183 ceph-mon[41562]: 4.16 scrub starts 2024-09-10T19:12:47.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:12:46 smithi183 ceph-mon[41562]: 4.16 scrub ok 2024-09-10T19:12:47.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:12:46 smithi183 ceph-mon[41562]: 4.4 scrub starts 2024-09-10T19:12:47.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:12:46 smithi183 ceph-mon[41562]: 4.4 scrub ok 2024-09-10T19:12:47.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:12:46 smithi183 ceph-mon[41562]: 4.d scrub starts 2024-09-10T19:12:47.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:12:46 smithi183 ceph-mon[41562]: 4.d scrub ok 2024-09-10T19:12:47.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:12:46 smithi183 ceph-mon[41562]: 4.1c scrub starts 2024-09-10T19:12:47.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:12:46 smithi183 ceph-mon[41562]: 4.1c scrub ok 2024-09-10T19:12:47.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:12:46 smithi183 ceph-mon[41562]: 6.12 scrub starts 2024-09-10T19:12:47.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:12:46 smithi183 ceph-mon[41562]: 6.12 scrub ok 2024-09-10T19:12:47.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:12:46 smithi183 ceph-mon[41562]: pgmap v89: 148 pgs: 148 active+clean; 3.3 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 852 B/s rd, 81 MiB/s wr, 1.30k op/s 2024-09-10T19:12:47.183 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:12:46 smithi183 ceph-mon[41562]: Health check failed: 1 osds down (OSD_DOWN) 2024-09-10T19:12:47.183 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:12:46 smithi183 ceph-mon[41562]: osdmap e99: 8 total, 7 up, 8 in 2024-09-10T19:12:47.183 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:12:46 smithi183 ceph-mon[41562]: 2.12 scrub starts 2024-09-10T19:12:47.183 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:12:46 smithi183 ceph-mon[41562]: 2.12 scrub ok 2024-09-10T19:12:47.183 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:12:46 smithi183 ceph-mon[41562]: osdmap e100: 8 total, 7 up, 8 in 2024-09-10T19:12:47.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:12:46 smithi191 ceph-mon[32884]: 4.16 scrub starts 2024-09-10T19:12:47.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:12:46 smithi191 ceph-mon[32884]: 4.16 scrub ok 2024-09-10T19:12:47.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:12:46 smithi191 ceph-mon[32884]: 4.4 scrub starts 2024-09-10T19:12:47.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:12:46 smithi191 ceph-mon[32884]: 4.4 scrub ok 2024-09-10T19:12:47.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:12:46 smithi191 ceph-mon[32884]: 4.d scrub starts 2024-09-10T19:12:47.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:12:46 smithi191 ceph-mon[32884]: 4.d scrub ok 2024-09-10T19:12:47.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:12:46 smithi191 ceph-mon[32884]: 4.1c scrub starts 2024-09-10T19:12:47.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:12:46 smithi191 ceph-mon[32884]: 4.1c scrub ok 2024-09-10T19:12:47.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:12:46 smithi191 ceph-mon[32884]: 6.12 scrub starts 2024-09-10T19:12:47.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:12:46 smithi191 ceph-mon[32884]: 6.12 scrub ok 2024-09-10T19:12:47.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:12:46 smithi191 ceph-mon[32884]: pgmap v89: 148 pgs: 148 active+clean; 3.3 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 852 B/s rd, 81 MiB/s wr, 1.30k op/s 2024-09-10T19:12:47.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:12:46 smithi191 ceph-mon[32884]: Health check failed: 1 osds down (OSD_DOWN) 2024-09-10T19:12:47.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:12:46 smithi191 ceph-mon[32884]: osdmap e99: 8 total, 7 up, 8 in 2024-09-10T19:12:47.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:12:46 smithi191 ceph-mon[32884]: 2.12 scrub starts 2024-09-10T19:12:47.217 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:12:46 smithi191 ceph-mon[32884]: 2.12 scrub ok 2024-09-10T19:12:47.217 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:12:46 smithi191 ceph-mon[32884]: osdmap e100: 8 total, 7 up, 8 in 2024-09-10T19:12:47.224 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:12:47.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:12:47.267 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:12:47.267 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:12:47.368 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:12:47.405 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:12:47.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:12:47.406 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:12:47.507 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:12:47.540 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:12:47.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:12:47.541 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:12:47.642 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:12:47.676 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:12:47.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:12:47.678 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:12:47.715 INFO:journalctl@ceph.mgr.x.smithi191.stdout:Sep 10 19:12:47 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-x[34967]: ::ffff:172.21.15.191 - - [10/Sep/2024:19:12:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T19:12:47.779 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:12:47.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:12:47.815 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:12:47.815 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:12:47.906 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: 29 16 57736 57720 124.365 93 0.00697982 0.00803526 2024-09-10T19:12:47.916 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:12:47.931 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:12:47 smithi183 ceph-mon[29736]: 2.b scrub starts 2024-09-10T19:12:47.932 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:12:47 smithi183 ceph-mon[29736]: 2.b scrub ok 2024-09-10T19:12:47.932 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:12:47 smithi183 ceph-mon[29736]: 6.6 scrub starts 2024-09-10T19:12:47.932 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:12:47 smithi183 ceph-mon[29736]: 6.6 scrub ok 2024-09-10T19:12:47.932 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:12:47 smithi183 ceph-mon[29736]: 5.2 scrub starts 2024-09-10T19:12:47.932 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:12:47 smithi183 ceph-mon[29736]: 5.2 scrub ok 2024-09-10T19:12:47.932 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:12:47 smithi183 ceph-mon[41562]: 2.b scrub starts 2024-09-10T19:12:47.932 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:12:47 smithi183 ceph-mon[41562]: 2.b scrub ok 2024-09-10T19:12:47.932 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:12:47 smithi183 ceph-mon[41562]: 6.6 scrub starts 2024-09-10T19:12:47.933 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:12:47 smithi183 ceph-mon[41562]: 6.6 scrub ok 2024-09-10T19:12:47.933 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:12:47 smithi183 ceph-mon[41562]: 5.2 scrub starts 2024-09-10T19:12:47.933 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:12:47 smithi183 ceph-mon[41562]: 5.2 scrub ok 2024-09-10T19:12:47.933 INFO:journalctl@ceph.osd.1.smithi183.stdout:Sep 10 19:12:47 smithi183 podman[87497]: 2024-09-10 19:12:47.805259264 +0000 UTC m=+4.048432012 container cleanup bd74997440b09d1e1f2a149b31787c72e3fffc8cd1adb142f562cd2b80fab856 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3bfeaf4417585c821cab9ca76e4d95f390b0e769807122a45463640820b5cd16, name=ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd-1, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-guits-quincy-2024-09-09-1810-quincy-5be5739, GIT_BRANCH=HEAD, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux ) 2024-09-10T19:12:47.933 INFO:journalctl@ceph.osd.1.smithi183.stdout:Sep 10 19:12:47 smithi183 bash[87497]: ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd-1 2024-09-10T19:12:47.968 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:12:47.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:12:47.969 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:12:48.070 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:12:48.106 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:12:48.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:12:48.107 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:12:48.208 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:12:48.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:12:47 smithi191 ceph-mon[32884]: 2.b scrub starts 2024-09-10T19:12:48.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:12:47 smithi191 ceph-mon[32884]: 2.b scrub ok 2024-09-10T19:12:48.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:12:47 smithi191 ceph-mon[32884]: 6.6 scrub starts 2024-09-10T19:12:48.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:12:47 smithi191 ceph-mon[32884]: 6.6 scrub ok 2024-09-10T19:12:48.217 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:12:47 smithi191 ceph-mon[32884]: 5.2 scrub starts 2024-09-10T19:12:48.217 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:12:47 smithi191 ceph-mon[32884]: 5.2 scrub ok 2024-09-10T19:12:48.231 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:12:48.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:12:48.232 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:12:48.333 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:12:48.363 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:12:48.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:12:48.364 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:12:48.465 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:12:48.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:12:48.497 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:12:48.498 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:12:48.598 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:12:48.630 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:12:48.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:12:48.631 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:12:48.732 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:12:48.768 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:12:48.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:12:48.770 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:12:48.871 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:12:48.901 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:12:48.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:12:48.902 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:12:48.906 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: 30 16 59324 59308 123.528 99.25 0.0080137 0.00809053 2024-09-10T19:12:49.005 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:12:49.039 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:12:49.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:12:49.040 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:12:49.141 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:12:49.177 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:12:49.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:12:49.178 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:12:49.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:12:48 smithi183 ceph-mon[29736]: 4.c scrub starts 2024-09-10T19:12:49.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:12:48 smithi183 ceph-mon[29736]: 4.c scrub ok 2024-09-10T19:12:49.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:12:48 smithi183 ceph-mon[29736]: pgmap v92: 148 pgs: 19 stale+active+clean, 129 active+clean; 3.3 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.2 KiB/s rd, 122 MiB/s wr, 1.95k op/s 2024-09-10T19:12:49.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:12:48 smithi183 ceph-mon[29736]: 2.1e scrub starts 2024-09-10T19:12:49.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:12:48 smithi183 ceph-mon[29736]: 2.1e scrub ok 2024-09-10T19:12:49.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:12:48 smithi183 ceph-mon[29736]: 2.e scrub starts 2024-09-10T19:12:49.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:12:48 smithi183 ceph-mon[29736]: 2.e scrub ok 2024-09-10T19:12:49.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:12:48 smithi183 ceph-mon[29736]: 2.16 scrub starts 2024-09-10T19:12:49.182 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:12:48 smithi183 ceph-mon[29736]: 2.16 scrub ok 2024-09-10T19:12:49.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:12:48 smithi183 ceph-mon[41562]: 4.c scrub starts 2024-09-10T19:12:49.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:12:48 smithi183 ceph-mon[41562]: 4.c scrub ok 2024-09-10T19:12:49.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:12:48 smithi183 ceph-mon[41562]: pgmap v92: 148 pgs: 19 stale+active+clean, 129 active+clean; 3.3 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.2 KiB/s rd, 122 MiB/s wr, 1.95k op/s 2024-09-10T19:12:49.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:12:48 smithi183 ceph-mon[41562]: 2.1e scrub starts 2024-09-10T19:12:49.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:12:48 smithi183 ceph-mon[41562]: 2.1e scrub ok 2024-09-10T19:12:49.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:12:48 smithi183 ceph-mon[41562]: 2.e scrub starts 2024-09-10T19:12:49.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:12:48 smithi183 ceph-mon[41562]: 2.e scrub ok 2024-09-10T19:12:49.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:12:48 smithi183 ceph-mon[41562]: 2.16 scrub starts 2024-09-10T19:12:49.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:12:48 smithi183 ceph-mon[41562]: 2.16 scrub ok 2024-09-10T19:12:49.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:12:48 smithi191 ceph-mon[32884]: 4.c scrub starts 2024-09-10T19:12:49.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:12:48 smithi191 ceph-mon[32884]: 4.c scrub ok 2024-09-10T19:12:49.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:12:48 smithi191 ceph-mon[32884]: pgmap v92: 148 pgs: 19 stale+active+clean, 129 active+clean; 3.3 GiB data, 12 GiB used, 703 GiB / 715 GiB avail; 1.2 KiB/s rd, 122 MiB/s wr, 1.95k op/s 2024-09-10T19:12:49.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:12:48 smithi191 ceph-mon[32884]: 2.1e scrub starts 2024-09-10T19:12:49.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:12:48 smithi191 ceph-mon[32884]: 2.1e scrub ok 2024-09-10T19:12:49.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:12:48 smithi191 ceph-mon[32884]: 2.e scrub starts 2024-09-10T19:12:49.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:12:48 smithi191 ceph-mon[32884]: 2.e scrub ok 2024-09-10T19:12:49.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:12:48 smithi191 ceph-mon[32884]: 2.16 scrub starts 2024-09-10T19:12:49.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:12:48 smithi191 ceph-mon[32884]: 2.16 scrub ok 2024-09-10T19:12:49.217 INFO:journalctl@ceph.iscsi.iscsi.a.smithi191.stdout:Sep 10 19:12:48 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-iscsi-iscsi-a[61816]: debug there is no tcmu-runner data available 2024-09-10T19:12:49.279 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:12:49.313 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:12:49.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:12:49.315 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:12:49.415 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:12:49.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:12:49.455 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:12:49.455 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:12:49.556 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:12:49.586 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:12:49.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:12:49.586 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:12:49.687 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:12:49.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:12:49.735 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:12:49.735 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:12:49.836 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:12:49.863 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:12:49.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:12:49.864 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:12:49.892 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:12:49.897 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:12:49.908 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: 31 16 61076 61060 123.067 109.5 0.0223103 0.00811897 2024-09-10T19:12:49.932 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:12:49 smithi183 ceph-mon[41562]: 4.1 scrub starts 2024-09-10T19:12:49.933 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:12:49 smithi183 ceph-mon[41562]: 4.1 scrub ok 2024-09-10T19:12:49.933 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:12:49 smithi183 ceph-mon[41562]: 5.1f scrub starts 2024-09-10T19:12:49.933 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:12:49 smithi183 ceph-mon[41562]: 5.1f scrub ok 2024-09-10T19:12:49.933 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:12:49 smithi183 ceph-mon[41562]: from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:12:49.933 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:12:49 smithi183 ceph-mon[41562]: 6.f scrub starts 2024-09-10T19:12:49.933 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:12:49 smithi183 ceph-mon[41562]: 6.f scrub ok 2024-09-10T19:12:49.933 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:12:49 smithi183 ceph-mon[41562]: osdmap e101: 8 total, 7 up, 8 in 2024-09-10T19:12:49.933 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:12:49 smithi183 ceph-mon[29736]: 4.1 scrub starts 2024-09-10T19:12:49.933 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:12:49 smithi183 ceph-mon[29736]: 4.1 scrub ok 2024-09-10T19:12:49.933 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:12:49 smithi183 ceph-mon[29736]: 5.1f scrub starts 2024-09-10T19:12:49.934 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:12:49 smithi183 ceph-mon[29736]: 5.1f scrub ok 2024-09-10T19:12:49.934 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:12:49 smithi183 ceph-mon[29736]: from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:12:49.934 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:12:49 smithi183 ceph-mon[29736]: 6.f scrub starts 2024-09-10T19:12:49.934 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:12:49 smithi183 ceph-mon[29736]: 6.f scrub ok 2024-09-10T19:12:49.934 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:12:49 smithi183 ceph-mon[29736]: osdmap e101: 8 total, 7 up, 8 in 2024-09-10T19:12:49.965 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:12:50.012 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:12:50.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:12:50.013 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:12:50.114 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:12:50.137 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:12:50.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:12:50.139 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:12:50.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:12:49 smithi191 ceph-mon[32884]: 4.1 scrub starts 2024-09-10T19:12:50.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:12:49 smithi191 ceph-mon[32884]: 4.1 scrub ok 2024-09-10T19:12:50.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:12:49 smithi191 ceph-mon[32884]: 5.1f scrub starts 2024-09-10T19:12:50.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:12:49 smithi191 ceph-mon[32884]: 5.1f scrub ok 2024-09-10T19:12:50.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:12:49 smithi191 ceph-mon[32884]: from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:12:50.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:12:49 smithi191 ceph-mon[32884]: 6.f scrub starts 2024-09-10T19:12:50.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:12:49 smithi191 ceph-mon[32884]: 6.f scrub ok 2024-09-10T19:12:50.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:12:49 smithi191 ceph-mon[32884]: osdmap e101: 8 total, 7 up, 8 in 2024-09-10T19:12:50.239 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:12:50.275 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:12:50.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:12:50.276 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:12:50.377 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:12:50.403 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:12:50.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:12:50.404 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:12:50.504 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:12:50.533 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:12:50.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:12:50.535 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:12:50.635 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:12:50.672 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:12:50.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:12:50.673 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:12:50.774 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:12:50.809 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:12:50.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:12:50.810 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:12:50.908 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: 32 16 62712 62696 122.416 102.25 0.00367136 0.00816408 2024-09-10T19:12:50.911 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:12:50.950 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:12:50.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:12:50.951 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:12:51.052 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:12:51.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:12:51.084 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:12:51.084 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:12:51.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:12:50 smithi183 ceph-mon[29736]: 2.1d scrub starts 2024-09-10T19:12:51.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:12:50 smithi183 ceph-mon[29736]: 2.1d scrub ok 2024-09-10T19:12:51.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:12:50 smithi183 ceph-mon[29736]: pgmap v93: 148 pgs: 25 active+undersized, 3 stale+active+clean, 15 active+undersized+degraded, 105 active+clean; 3.5 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 122 MiB/s wr, 1.95k op/s; 11307/174192 objects degraded (6.491%) 2024-09-10T19:12:51.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:12:50 smithi183 ceph-mon[29736]: 4.0 scrub starts 2024-09-10T19:12:51.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:12:50 smithi183 ceph-mon[29736]: 4.0 scrub ok 2024-09-10T19:12:51.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:12:50 smithi183 ceph-mon[29736]: Health check failed: Degraded data redundancy: 11307/174192 objects degraded (6.491%), 15 pgs degraded (PG_DEGRADED) 2024-09-10T19:12:51.182 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:12:50 smithi183 ceph-mon[29736]: 4.1e scrub starts 2024-09-10T19:12:51.182 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:12:50 smithi183 ceph-mon[29736]: 4.1e scrub ok 2024-09-10T19:12:51.182 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:12:50 smithi183 ceph-mon[29736]: 4.8 scrub starts 2024-09-10T19:12:51.182 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:12:50 smithi183 ceph-mon[29736]: 4.8 scrub ok 2024-09-10T19:12:51.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:12:50 smithi183 ceph-mon[41562]: 2.1d scrub starts 2024-09-10T19:12:51.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:12:50 smithi183 ceph-mon[41562]: 2.1d scrub ok 2024-09-10T19:12:51.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:12:50 smithi183 ceph-mon[41562]: pgmap v93: 148 pgs: 25 active+undersized, 3 stale+active+clean, 15 active+undersized+degraded, 105 active+clean; 3.5 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 122 MiB/s wr, 1.95k op/s; 11307/174192 objects degraded (6.491%) 2024-09-10T19:12:51.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:12:50 smithi183 ceph-mon[41562]: 4.0 scrub starts 2024-09-10T19:12:51.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:12:50 smithi183 ceph-mon[41562]: 4.0 scrub ok 2024-09-10T19:12:51.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:12:50 smithi183 ceph-mon[41562]: Health check failed: Degraded data redundancy: 11307/174192 objects degraded (6.491%), 15 pgs degraded (PG_DEGRADED) 2024-09-10T19:12:51.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:12:50 smithi183 ceph-mon[41562]: 4.1e scrub starts 2024-09-10T19:12:51.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:12:50 smithi183 ceph-mon[41562]: 4.1e scrub ok 2024-09-10T19:12:51.183 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:12:50 smithi183 ceph-mon[41562]: 4.8 scrub starts 2024-09-10T19:12:51.183 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:12:50 smithi183 ceph-mon[41562]: 4.8 scrub ok 2024-09-10T19:12:51.185 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:12:51.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:12:50 smithi191 ceph-mon[32884]: 2.1d scrub starts 2024-09-10T19:12:51.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:12:50 smithi191 ceph-mon[32884]: 2.1d scrub ok 2024-09-10T19:12:51.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:12:50 smithi191 ceph-mon[32884]: pgmap v93: 148 pgs: 25 active+undersized, 3 stale+active+clean, 15 active+undersized+degraded, 105 active+clean; 3.5 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 122 MiB/s wr, 1.95k op/s; 11307/174192 objects degraded (6.491%) 2024-09-10T19:12:51.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:12:50 smithi191 ceph-mon[32884]: 4.0 scrub starts 2024-09-10T19:12:51.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:12:50 smithi191 ceph-mon[32884]: 4.0 scrub ok 2024-09-10T19:12:51.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:12:50 smithi191 ceph-mon[32884]: Health check failed: Degraded data redundancy: 11307/174192 objects degraded (6.491%), 15 pgs degraded (PG_DEGRADED) 2024-09-10T19:12:51.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:12:50 smithi191 ceph-mon[32884]: 4.1e scrub starts 2024-09-10T19:12:51.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:12:50 smithi191 ceph-mon[32884]: 4.1e scrub ok 2024-09-10T19:12:51.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:12:50 smithi191 ceph-mon[32884]: 4.8 scrub starts 2024-09-10T19:12:51.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:12:50 smithi191 ceph-mon[32884]: 4.8 scrub ok 2024-09-10T19:12:51.218 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:12:51.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:12:51.219 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:12:51.320 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:12:51.349 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:12:51.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:12:51.350 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:12:51.451 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:12:51.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:12:51.490 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:12:51.491 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:12:51.591 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:12:51.620 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:12:51.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:12:51.622 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:12:51.681 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Sep 10 19:12:51 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-y[29989]: ::ffff:172.21.15.191 - - [10/Sep/2024:19:12:51] "GET /metrics HTTP/1.1" 200 223521 "" "Prometheus/2.43.0" 2024-09-10T19:12:51.723 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:12:51.753 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:12:51.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:12:51.755 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:12:51.855 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:12:51.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:12:51.902 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:12:51.902 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:12:51.909 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: 33 16 64447 64431 121.99 108.438 0.00470326 0.00819202 2024-09-10T19:12:52.004 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:12:52.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:12:52.033 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:12:52.033 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:12:52.106 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:12:52.134 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:12:52.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:12:52.161 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:12:52.161 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:12:52.183 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:12:51 smithi183 ceph-mon[29736]: 2.f scrub starts 2024-09-10T19:12:52.183 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:12:51 smithi183 ceph-mon[29736]: 2.f scrub ok 2024-09-10T19:12:52.183 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:12:51 smithi183 ceph-mon[29736]: 2.1b scrub starts 2024-09-10T19:12:52.183 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:12:51 smithi183 ceph-mon[29736]: 2.1b scrub ok 2024-09-10T19:12:52.183 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:12:51 smithi183 ceph-mon[29736]: 5.d scrub starts 2024-09-10T19:12:52.183 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:12:51 smithi183 ceph-mon[29736]: 5.d scrub ok 2024-09-10T19:12:52.183 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:12:51 smithi183 ceph-mon[29736]: 5.1b scrub starts 2024-09-10T19:12:52.183 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:12:51 smithi183 ceph-mon[29736]: 5.1b scrub ok 2024-09-10T19:12:52.184 INFO:journalctl@ceph.osd.1.smithi183.stdout:Sep 10 19:12:52 smithi183 podman[87840]: 2024-09-10 19:12:52.096993502 +0000 UTC m=+5.737108252 container remove bd74997440b09d1e1f2a149b31787c72e3fffc8cd1adb142f562cd2b80fab856 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3bfeaf4417585c821cab9ca76e4d95f390b0e769807122a45463640820b5cd16, name=ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd-1, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.schema-version=1.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, io.buildah.version=1.37.2, RELEASE=wip-guits-quincy-2024-09-09-1810-quincy-5be5739, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-10T19:12:52.184 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:12:51 smithi183 ceph-mon[41562]: 2.f scrub starts 2024-09-10T19:12:52.184 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:12:51 smithi183 ceph-mon[41562]: 2.f scrub ok 2024-09-10T19:12:52.184 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:12:51 smithi183 ceph-mon[41562]: 2.1b scrub starts 2024-09-10T19:12:52.184 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:12:51 smithi183 ceph-mon[41562]: 2.1b scrub ok 2024-09-10T19:12:52.184 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:12:51 smithi183 ceph-mon[41562]: 5.d scrub starts 2024-09-10T19:12:52.184 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:12:51 smithi183 ceph-mon[41562]: 5.d scrub ok 2024-09-10T19:12:52.184 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:12:51 smithi183 ceph-mon[41562]: 5.1b scrub starts 2024-09-10T19:12:52.184 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:12:51 smithi183 ceph-mon[41562]: 5.1b scrub ok 2024-09-10T19:12:52.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:12:51 smithi191 ceph-mon[32884]: 2.f scrub starts 2024-09-10T19:12:52.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:12:51 smithi191 ceph-mon[32884]: 2.f scrub ok 2024-09-10T19:12:52.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:12:51 smithi191 ceph-mon[32884]: 2.1b scrub starts 2024-09-10T19:12:52.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:12:51 smithi191 ceph-mon[32884]: 2.1b scrub ok 2024-09-10T19:12:52.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:12:51 smithi191 ceph-mon[32884]: 5.d scrub starts 2024-09-10T19:12:52.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:12:51 smithi191 ceph-mon[32884]: 5.d scrub ok 2024-09-10T19:12:52.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:12:51 smithi191 ceph-mon[32884]: 5.1b scrub starts 2024-09-10T19:12:52.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:12:51 smithi191 ceph-mon[32884]: 5.1b scrub ok 2024-09-10T19:12:52.262 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:12:52.290 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:12:52.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:12:52.291 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:12:52.392 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:12:52.439 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:12:52.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:12:52.440 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:12:52.540 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:12:52.571 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:12:52.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:12:52.572 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:12:52.673 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:12:52.701 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:12:52.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:12:52.703 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:12:52.803 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:12:52.869 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:12:52.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:12:52.871 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:12:52.909 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: 34 16 66098 66082 121.437 103.188 0.00813993 0.00822993 2024-09-10T19:12:52.972 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:12:53.002 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:12:53.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:12:53.003 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:12:53.104 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:12:53.124 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:12:52 smithi183 ceph-mon[29736]: 6.15 scrub starts 2024-09-10T19:12:53.124 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:12:52 smithi183 ceph-mon[29736]: 6.15 scrub ok 2024-09-10T19:12:53.124 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:12:52 smithi183 ceph-mon[29736]: pgmap v95: 148 pgs: 33 active+undersized, 26 active+undersized+degraded, 89 active+clean; 3.7 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 66 MiB/s wr, 1.05k op/s; 22644/182424 objects degraded (12.413%); 11 KiB/s, 0 objects/s recovering 2024-09-10T19:12:53.124 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:12:52 smithi183 ceph-mon[29736]: 6.1f scrub starts 2024-09-10T19:12:53.124 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:12:52 smithi183 ceph-mon[29736]: 6.1f scrub ok 2024-09-10T19:12:53.124 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:12:52 smithi183 ceph-mon[29736]: 4.1f scrub starts 2024-09-10T19:12:53.125 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:12:52 smithi183 ceph-mon[29736]: 4.1f scrub ok 2024-09-10T19:12:53.125 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:12:52 smithi183 ceph-mon[29736]: 4.15 scrub starts 2024-09-10T19:12:53.125 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:12:52 smithi183 ceph-mon[29736]: 4.15 scrub ok 2024-09-10T19:12:53.125 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:12:52 smithi183 ceph-mon[41562]: 6.15 scrub starts 2024-09-10T19:12:53.125 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:12:52 smithi183 ceph-mon[41562]: 6.15 scrub ok 2024-09-10T19:12:53.125 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:12:52 smithi183 ceph-mon[41562]: pgmap v95: 148 pgs: 33 active+undersized, 26 active+undersized+degraded, 89 active+clean; 3.7 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 66 MiB/s wr, 1.05k op/s; 22644/182424 objects degraded (12.413%); 11 KiB/s, 0 objects/s recovering 2024-09-10T19:12:53.125 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:12:52 smithi183 ceph-mon[41562]: 6.1f scrub starts 2024-09-10T19:12:53.125 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:12:52 smithi183 ceph-mon[41562]: 6.1f scrub ok 2024-09-10T19:12:53.125 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:12:52 smithi183 ceph-mon[41562]: 4.1f scrub starts 2024-09-10T19:12:53.125 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:12:52 smithi183 ceph-mon[41562]: 4.1f scrub ok 2024-09-10T19:12:53.125 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:12:52 smithi183 ceph-mon[41562]: 4.15 scrub starts 2024-09-10T19:12:53.126 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:12:52 smithi183 ceph-mon[41562]: 4.15 scrub ok 2024-09-10T19:12:53.138 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:12:53.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:12:53.140 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:12:53.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:12:52 smithi191 ceph-mon[32884]: 6.15 scrub starts 2024-09-10T19:12:53.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:12:52 smithi191 ceph-mon[32884]: 6.15 scrub ok 2024-09-10T19:12:53.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:12:52 smithi191 ceph-mon[32884]: pgmap v95: 148 pgs: 33 active+undersized, 26 active+undersized+degraded, 89 active+clean; 3.7 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 66 MiB/s wr, 1.05k op/s; 22644/182424 objects degraded (12.413%); 11 KiB/s, 0 objects/s recovering 2024-09-10T19:12:53.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:12:52 smithi191 ceph-mon[32884]: 6.1f scrub starts 2024-09-10T19:12:53.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:12:52 smithi191 ceph-mon[32884]: 6.1f scrub ok 2024-09-10T19:12:53.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:12:52 smithi191 ceph-mon[32884]: 4.1f scrub starts 2024-09-10T19:12:53.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:12:52 smithi191 ceph-mon[32884]: 4.1f scrub ok 2024-09-10T19:12:53.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:12:52 smithi191 ceph-mon[32884]: 4.15 scrub starts 2024-09-10T19:12:53.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:12:52 smithi191 ceph-mon[32884]: 4.15 scrub ok 2024-09-10T19:12:53.241 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:12:53.270 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:12:53.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:12:53.271 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:12:53.372 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:12:53.402 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:12:53.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:12:53.403 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:12:53.503 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:12:53.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:12:53.539 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:12:53.539 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:12:53.640 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:12:53.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:12:53.672 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:12:53.672 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:12:53.773 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:12:53.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:12:53.813 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:12:53.814 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:12:53.909 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: 35 16 67803 67787 121.012 106.562 0.00670759 0.00825965 2024-09-10T19:12:53.915 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:12:53.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:12:53.947 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:12:53.947 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:12:54.048 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:12:54.086 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:12:54.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:12:54.087 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:12:54.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:12:53 smithi183 ceph-mon[29736]: 2.7 scrub starts 2024-09-10T19:12:54.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:12:53 smithi183 ceph-mon[29736]: 2.7 scrub ok 2024-09-10T19:12:54.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:12:53 smithi183 ceph-mon[29736]: 2.1 scrub starts 2024-09-10T19:12:54.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:12:53 smithi183 ceph-mon[29736]: 2.1 scrub ok 2024-09-10T19:12:54.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:12:53 smithi183 ceph-mon[41562]: 2.7 scrub starts 2024-09-10T19:12:54.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:12:53 smithi183 ceph-mon[41562]: 2.7 scrub ok 2024-09-10T19:12:54.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:12:53 smithi183 ceph-mon[41562]: 2.1 scrub starts 2024-09-10T19:12:54.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:12:53 smithi183 ceph-mon[41562]: 2.1 scrub ok 2024-09-10T19:12:54.187 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:12:54.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:12:53 smithi191 ceph-mon[32884]: 2.7 scrub starts 2024-09-10T19:12:54.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:12:53 smithi191 ceph-mon[32884]: 2.7 scrub ok 2024-09-10T19:12:54.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:12:53 smithi191 ceph-mon[32884]: 2.1 scrub starts 2024-09-10T19:12:54.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:12:53 smithi191 ceph-mon[32884]: 2.1 scrub ok 2024-09-10T19:12:54.230 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:12:54.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:12:54.231 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:12:54.332 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:12:54.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:12:54.359 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:12:54.359 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:12:54.460 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:12:54.495 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:12:54.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:12:54.496 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:12:54.597 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:12:54.628 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:12:54.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:12:54.629 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:12:54.730 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:12:54.767 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:12:54.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:12:54.768 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:12:54.869 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:12:54.905 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:12:54.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:12:54.906 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:12:54.909 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: 36 16 69419 69403 120.456 101 0.00883754 0.00829721 2024-09-10T19:12:55.007 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:12:55.041 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:12:55.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:12:55.042 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:12:55.143 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:12:55.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:12:55.178 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:12:55.178 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:12:55.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:12:54 smithi183 ceph-mon[29736]: 2.5 scrub starts 2024-09-10T19:12:55.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:12:54 smithi183 ceph-mon[29736]: 2.5 scrub ok 2024-09-10T19:12:55.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:12:54 smithi183 ceph-mon[29736]: pgmap v96: 148 pgs: 33 active+undersized, 26 active+undersized+degraded, 89 active+clean; 3.7 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 50 MiB/s wr, 805 op/s; 22644/182424 objects degraded (12.413%); 8.2 KiB/s, 0 objects/s recovering 2024-09-10T19:12:55.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:12:54 smithi183 ceph-mon[29736]: 6.14 scrub starts 2024-09-10T19:12:55.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:12:54 smithi183 ceph-mon[29736]: 6.14 scrub ok 2024-09-10T19:12:55.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:12:54 smithi183 ceph-mon[41562]: 2.5 scrub starts 2024-09-10T19:12:55.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:12:54 smithi183 ceph-mon[41562]: 2.5 scrub ok 2024-09-10T19:12:55.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:12:54 smithi183 ceph-mon[41562]: pgmap v96: 148 pgs: 33 active+undersized, 26 active+undersized+degraded, 89 active+clean; 3.7 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 50 MiB/s wr, 805 op/s; 22644/182424 objects degraded (12.413%); 8.2 KiB/s, 0 objects/s recovering 2024-09-10T19:12:55.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:12:54 smithi183 ceph-mon[41562]: 6.14 scrub starts 2024-09-10T19:12:55.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:12:54 smithi183 ceph-mon[41562]: 6.14 scrub ok 2024-09-10T19:12:55.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:12:54 smithi191 ceph-mon[32884]: 2.5 scrub starts 2024-09-10T19:12:55.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:12:54 smithi191 ceph-mon[32884]: 2.5 scrub ok 2024-09-10T19:12:55.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:12:54 smithi191 ceph-mon[32884]: pgmap v96: 148 pgs: 33 active+undersized, 26 active+undersized+degraded, 89 active+clean; 3.7 GiB data, 13 GiB used, 702 GiB / 715 GiB avail; 50 MiB/s wr, 805 op/s; 22644/182424 objects degraded (12.413%); 8.2 KiB/s, 0 objects/s recovering 2024-09-10T19:12:55.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:12:54 smithi191 ceph-mon[32884]: 6.14 scrub starts 2024-09-10T19:12:55.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:12:54 smithi191 ceph-mon[32884]: 6.14 scrub ok 2024-09-10T19:12:55.279 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:12:55.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:12:55.310 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:12:55.310 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:12:55.411 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:12:55.444 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:12:55.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:12:55.445 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:12:55.545 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:12:55.566 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:12:55.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:12:55.567 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:12:55.668 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:12:55.702 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:12:55.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:12:55.703 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:12:55.803 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:12:55.832 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:12:55.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:12:55.833 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:12:55.910 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: 37 15 70922 70907 119.739 94 0.00638227 0.00834603 2024-09-10T19:12:55.934 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:12:55.964 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:12:55.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:12:55.965 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:12:56.066 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:12:56.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:12:56.106 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:12:56.106 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:12:56.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:12:55 smithi183 ceph-mon[29736]: 2.1c scrub starts 2024-09-10T19:12:56.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:12:55 smithi183 ceph-mon[29736]: 2.1c scrub ok 2024-09-10T19:12:56.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:12:55 smithi183 ceph-mon[29736]: 6.11 scrub starts 2024-09-10T19:12:56.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:12:55 smithi183 ceph-mon[29736]: 6.11 scrub ok 2024-09-10T19:12:56.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:12:55 smithi183 ceph-mon[29736]: 4.5 scrub starts 2024-09-10T19:12:56.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:12:55 smithi183 ceph-mon[29736]: 4.5 scrub ok 2024-09-10T19:12:56.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:12:55 smithi183 ceph-mon[29736]: 5.b scrub starts 2024-09-10T19:12:56.182 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:12:55 smithi183 ceph-mon[29736]: 5.b scrub ok 2024-09-10T19:12:56.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:12:55 smithi183 ceph-mon[41562]: 2.1c scrub starts 2024-09-10T19:12:56.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:12:55 smithi183 ceph-mon[41562]: 2.1c scrub ok 2024-09-10T19:12:56.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:12:55 smithi183 ceph-mon[41562]: 6.11 scrub starts 2024-09-10T19:12:56.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:12:55 smithi183 ceph-mon[41562]: 6.11 scrub ok 2024-09-10T19:12:56.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:12:55 smithi183 ceph-mon[41562]: 4.5 scrub starts 2024-09-10T19:12:56.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:12:55 smithi183 ceph-mon[41562]: 4.5 scrub ok 2024-09-10T19:12:56.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:12:55 smithi183 ceph-mon[41562]: 5.b scrub starts 2024-09-10T19:12:56.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:12:55 smithi183 ceph-mon[41562]: 5.b scrub ok 2024-09-10T19:12:56.207 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:12:56.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:12:55 smithi191 ceph-mon[32884]: 2.1c scrub starts 2024-09-10T19:12:56.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:12:55 smithi191 ceph-mon[32884]: 2.1c scrub ok 2024-09-10T19:12:56.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:12:55 smithi191 ceph-mon[32884]: 6.11 scrub starts 2024-09-10T19:12:56.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:12:55 smithi191 ceph-mon[32884]: 6.11 scrub ok 2024-09-10T19:12:56.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:12:55 smithi191 ceph-mon[32884]: 4.5 scrub starts 2024-09-10T19:12:56.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:12:55 smithi191 ceph-mon[32884]: 4.5 scrub ok 2024-09-10T19:12:56.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:12:55 smithi191 ceph-mon[32884]: 5.b scrub starts 2024-09-10T19:12:56.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:12:55 smithi191 ceph-mon[32884]: 5.b scrub ok 2024-09-10T19:12:56.232 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:12:56.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:12:56.233 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:12:56.334 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:12:56.377 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:12:56.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:12:56.378 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:12:56.479 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:12:56.524 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:12:56.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:12:56.525 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:12:56.626 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:12:56.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:12:56.662 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:12:56.662 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:12:56.763 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:12:56.800 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:12:56.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:12:56.801 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:12:56.902 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:12:56.910 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: 38 16 72521 72505 119.216 99.875 0.0179396 0.00838286 2024-09-10T19:12:56.932 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:12:56.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:12:56.933 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:12:57.033 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:12:57.077 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:12:57.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:12:57.078 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:12:57.179 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:12:57.181 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:12:56 smithi183 ceph-mon[41562]: 4.1b scrub starts 2024-09-10T19:12:57.181 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:12:56 smithi183 ceph-mon[41562]: 4.1b scrub ok 2024-09-10T19:12:57.181 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:12:56 smithi183 ceph-mon[41562]: pgmap v97: 148 pgs: 33 active+undersized, 26 active+undersized+degraded, 89 active+clean; 4.2 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 104 MiB/s wr, 1.66k op/s; 25595/207465 objects degraded (12.337%); 7.3 KiB/s, 0 objects/s recovering 2024-09-10T19:12:57.181 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:12:56 smithi183 ceph-mon[41562]: Health check update: Degraded data redundancy: 25595/207465 objects degraded (12.337%), 26 pgs degraded (PG_DEGRADED) 2024-09-10T19:12:57.181 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:12:56 smithi183 ceph-mon[41562]: 6.9 scrub starts 2024-09-10T19:12:57.181 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:12:56 smithi183 ceph-mon[41562]: 6.9 scrub ok 2024-09-10T19:12:57.181 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:12:56 smithi183 ceph-mon[41562]: 4.2 scrub starts 2024-09-10T19:12:57.181 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:12:56 smithi183 ceph-mon[41562]: 4.2 scrub ok 2024-09-10T19:12:57.182 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:12:56 smithi183 ceph-mon[29736]: 4.1b scrub starts 2024-09-10T19:12:57.182 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:12:56 smithi183 ceph-mon[29736]: 4.1b scrub ok 2024-09-10T19:12:57.182 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:12:56 smithi183 ceph-mon[29736]: pgmap v97: 148 pgs: 33 active+undersized, 26 active+undersized+degraded, 89 active+clean; 4.2 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 104 MiB/s wr, 1.66k op/s; 25595/207465 objects degraded (12.337%); 7.3 KiB/s, 0 objects/s recovering 2024-09-10T19:12:57.182 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:12:56 smithi183 ceph-mon[29736]: Health check update: Degraded data redundancy: 25595/207465 objects degraded (12.337%), 26 pgs degraded (PG_DEGRADED) 2024-09-10T19:12:57.182 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:12:56 smithi183 ceph-mon[29736]: 6.9 scrub starts 2024-09-10T19:12:57.182 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:12:56 smithi183 ceph-mon[29736]: 6.9 scrub ok 2024-09-10T19:12:57.182 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:12:56 smithi183 ceph-mon[29736]: 4.2 scrub starts 2024-09-10T19:12:57.183 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:12:56 smithi183 ceph-mon[29736]: 4.2 scrub ok 2024-09-10T19:12:57.183 INFO:journalctl@ceph.osd.1.smithi183.stdout:Sep 10 19:12:57 smithi183 podman[88733]: 2024-09-10 19:12:56.941844782 +0000 UTC m=+1.169239086 image pull 4c55dd4096115e48502f368cff1eab06769f71f030e1692e2611bbc6c6f637e9 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3bfeaf4417585c821cab9ca76e4d95f390b0e769807122a45463640820b5cd16 2024-09-10T19:12:57.210 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:12:57.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:12:57.211 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:12:57.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:12:56 smithi191 ceph-mon[32884]: 4.1b scrub starts 2024-09-10T19:12:57.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:12:56 smithi191 ceph-mon[32884]: 4.1b scrub ok 2024-09-10T19:12:57.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:12:56 smithi191 ceph-mon[32884]: pgmap v97: 148 pgs: 33 active+undersized, 26 active+undersized+degraded, 89 active+clean; 4.2 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 104 MiB/s wr, 1.66k op/s; 25595/207465 objects degraded (12.337%); 7.3 KiB/s, 0 objects/s recovering 2024-09-10T19:12:57.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:12:56 smithi191 ceph-mon[32884]: Health check update: Degraded data redundancy: 25595/207465 objects degraded (12.337%), 26 pgs degraded (PG_DEGRADED) 2024-09-10T19:12:57.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:12:56 smithi191 ceph-mon[32884]: 6.9 scrub starts 2024-09-10T19:12:57.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:12:56 smithi191 ceph-mon[32884]: 6.9 scrub ok 2024-09-10T19:12:57.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:12:56 smithi191 ceph-mon[32884]: 4.2 scrub starts 2024-09-10T19:12:57.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:12:56 smithi191 ceph-mon[32884]: 4.2 scrub ok 2024-09-10T19:12:57.312 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:12:57.352 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:12:57.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:12:57.353 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:12:57.454 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:12:57.484 INFO:journalctl@ceph.osd.1.smithi183.stdout:Sep 10 19:12:57 smithi183 podman[88733]: 2024-09-10 19:12:57.226312034 +0000 UTC m=+1.453706316 container create 7066fd678ec0fa710c99eb7bdc1f19f578f472c188bc75d306f5fc96c1012f11 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3bfeaf4417585c821cab9ca76e4d95f390b0e769807122a45463640820b5cd16, name=ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd-1-deactivate, io.buildah.version=1.37.2, ceph=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, RELEASE=wip-guits-quincy-2024-09-09-1810-quincy-5be5739, GIT_CLEAN=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0) 2024-09-10T19:12:57.495 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:12:57.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:12:57.497 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:12:57.597 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:12:57.626 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:12:57.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:12:57.627 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:12:57.715 INFO:journalctl@ceph.mgr.x.smithi191.stdout:Sep 10 19:12:57 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-x[34967]: ::ffff:172.21.15.191 - - [10/Sep/2024:19:12:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T19:12:57.728 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:12:57.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:12:57.771 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:12:57.771 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:12:57.782 INFO:journalctl@ceph.osd.1.smithi183.stdout:Sep 10 19:12:57 smithi183 podman[88733]: 2024-09-10 19:12:57.546565206 +0000 UTC m=+1.773959506 container init 7066fd678ec0fa710c99eb7bdc1f19f578f472c188bc75d306f5fc96c1012f11 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3bfeaf4417585c821cab9ca76e4d95f390b0e769807122a45463640820b5cd16, name=ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd-1-deactivate, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_CLEAN=True, ceph=True, CEPH_POINT_RELEASE=, RELEASE=wip-guits-quincy-2024-09-09-1810-quincy-5be5739, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731) 2024-09-10T19:12:57.783 INFO:journalctl@ceph.osd.1.smithi183.stdout:Sep 10 19:12:57 smithi183 podman[88733]: 2024-09-10 19:12:57.556445435 +0000 UTC m=+1.783839710 container start 7066fd678ec0fa710c99eb7bdc1f19f578f472c188bc75d306f5fc96c1012f11 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3bfeaf4417585c821cab9ca76e4d95f390b0e769807122a45463640820b5cd16, name=ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd-1-deactivate, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, RELEASE=wip-guits-quincy-2024-09-09-1810-quincy-5be5739, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, org.label-schema.build-date=20240731, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2) 2024-09-10T19:12:57.783 INFO:journalctl@ceph.osd.1.smithi183.stdout:Sep 10 19:12:57 smithi183 podman[88733]: 2024-09-10 19:12:57.654526796 +0000 UTC m=+1.881921077 container attach 7066fd678ec0fa710c99eb7bdc1f19f578f472c188bc75d306f5fc96c1012f11 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3bfeaf4417585c821cab9ca76e4d95f390b0e769807122a45463640820b5cd16, name=ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd-1-deactivate, ceph=True, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_BRANCH=HEAD, RELEASE=wip-guits-quincy-2024-09-09-1810-quincy-5be5739, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_CLEAN=True) 2024-09-10T19:12:57.872 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:12:57.910 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: 39 16 73986 73970 118.507 91.5625 0.0130163 0.00843381 2024-09-10T19:12:57.910 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:12:57.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:12:57.912 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:12:58.013 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:12:58.036 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:12:57 smithi183 ceph-mon[29736]: 6.a scrub starts 2024-09-10T19:12:58.036 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:12:57 smithi183 ceph-mon[29736]: 6.a scrub ok 2024-09-10T19:12:58.036 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:12:57 smithi183 ceph-mon[41562]: 6.a scrub starts 2024-09-10T19:12:58.036 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:12:57 smithi183 ceph-mon[41562]: 6.a scrub ok 2024-09-10T19:12:58.037 INFO:journalctl@ceph.osd.1.smithi183.stdout:Sep 10 19:12:58 smithi183 podman[88733]: 2024-09-10 19:12:58.02478342 +0000 UTC m=+2.252177703 container died 7066fd678ec0fa710c99eb7bdc1f19f578f472c188bc75d306f5fc96c1012f11 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3bfeaf4417585c821cab9ca76e4d95f390b0e769807122a45463640820b5cd16, name=ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd-1-deactivate, maintainer=Guillaume Abrioux , RELEASE=wip-guits-quincy-2024-09-09-1810-quincy-5be5739, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.schema-version=1.0, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2) 2024-09-10T19:12:58.040 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:12:58.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:12:58.041 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:12:58.142 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:12:58.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:12:58.174 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:12:58.174 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:12:58.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:12:57 smithi191 ceph-mon[32884]: 6.a scrub starts 2024-09-10T19:12:58.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:12:57 smithi191 ceph-mon[32884]: 6.a scrub ok 2024-09-10T19:12:58.275 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:12:58.305 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:12:58.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:12:58.306 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:12:58.407 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:12:58.450 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:12:58.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:12:58.451 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:12:58.552 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:12:58.579 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:12:58.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:12:58.580 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:12:58.681 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:12:58.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:12:58.718 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:12:58.718 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:12:58.819 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:12:58.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:12:58.858 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:12:58.859 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:12:58.910 INFO:tasks.radosbench.radosbench.0.smithi183.stdout:2024-09-10T19:12:58.910345+0000 min lat: 0.00124697 max lat: 1.04074 avg lat: 0.00848331 2024-09-10T19:12:58.911 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-10T19:12:58.911 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: 40 16 75440 75424 117.815 90.875 0.0126331 0.00848331 2024-09-10T19:12:58.959 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:12:58.989 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:12:58.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:12:58.990 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:12:59.091 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:12:59.097 INFO:teuthology.orchestra.run.smithi183.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-09-10T19:12:59.116 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:12:59.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:12:59.118 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:12:59.181 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:12:58 smithi183 ceph-mon[41562]: 6.2 scrub starts 2024-09-10T19:12:59.181 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:12:58 smithi183 ceph-mon[41562]: 6.2 scrub ok 2024-09-10T19:12:59.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:12:58 smithi183 ceph-mon[41562]: 6.1c scrub starts 2024-09-10T19:12:59.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:12:58 smithi183 ceph-mon[41562]: 6.1c scrub ok 2024-09-10T19:12:59.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:12:58 smithi183 ceph-mon[41562]: pgmap v98: 148 pgs: 33 active+undersized, 26 active+undersized+degraded, 89 active+clean; 4.2 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 91 MiB/s wr, 1.46k op/s; 25595/207465 objects degraded (12.337%); 6.4 KiB/s, 0 objects/s recovering 2024-09-10T19:12:59.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:12:58 smithi183 ceph-mon[41562]: from='mgr.14556 ' entity='mgr.y' 2024-09-10T19:12:59.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:12:58 smithi183 ceph-mon[41562]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:12:59.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:12:58 smithi183 ceph-mon[41562]: purged_snaps scrub starts 2024-09-10T19:12:59.183 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:12:58 smithi183 ceph-mon[41562]: purged_snaps scrub ok 2024-09-10T19:12:59.183 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:12:58 smithi183 ceph-mon[41562]: from='mgr.14556 ' entity='mgr.y' 2024-09-10T19:12:59.183 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:12:58 smithi183 ceph-mon[41562]: from='mgr.14556 ' entity='mgr.y' 2024-09-10T19:12:59.183 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:12:58 smithi183 ceph-mon[41562]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T19:12:59.183 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:12:58 smithi183 ceph-mon[41562]: from='client.? 172.21.15.183:0/1574703421' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-10T19:12:59.183 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:12:58 smithi183 ceph-mon[41562]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-10T19:12:59.183 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:12:58 smithi183 ceph-mon[41562]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T19:12:59.183 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:12:58 smithi183 ceph-mon[41562]: from='mgr.14556 ' entity='mgr.y' 2024-09-10T19:12:59.183 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:12:58 smithi183 ceph-mon[29736]: 6.2 scrub starts 2024-09-10T19:12:59.184 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:12:58 smithi183 ceph-mon[29736]: 6.2 scrub ok 2024-09-10T19:12:59.184 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:12:58 smithi183 ceph-mon[29736]: 6.1c scrub starts 2024-09-10T19:12:59.184 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:12:58 smithi183 ceph-mon[29736]: 6.1c scrub ok 2024-09-10T19:12:59.184 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:12:58 smithi183 ceph-mon[29736]: pgmap v98: 148 pgs: 33 active+undersized, 26 active+undersized+degraded, 89 active+clean; 4.2 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 91 MiB/s wr, 1.46k op/s; 25595/207465 objects degraded (12.337%); 6.4 KiB/s, 0 objects/s recovering 2024-09-10T19:12:59.184 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:12:58 smithi183 ceph-mon[29736]: from='mgr.14556 ' entity='mgr.y' 2024-09-10T19:12:59.184 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:12:58 smithi183 ceph-mon[29736]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:12:59.184 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:12:58 smithi183 ceph-mon[29736]: purged_snaps scrub starts 2024-09-10T19:12:59.184 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:12:58 smithi183 ceph-mon[29736]: purged_snaps scrub ok 2024-09-10T19:12:59.184 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:12:58 smithi183 ceph-mon[29736]: from='mgr.14556 ' entity='mgr.y' 2024-09-10T19:12:59.184 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:12:58 smithi183 ceph-mon[29736]: from='mgr.14556 ' entity='mgr.y' 2024-09-10T19:12:59.184 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:12:58 smithi183 ceph-mon[29736]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T19:12:59.184 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:12:58 smithi183 ceph-mon[29736]: from='client.? 172.21.15.183:0/1574703421' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-10T19:12:59.185 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:12:58 smithi183 ceph-mon[29736]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-10T19:12:59.185 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:12:58 smithi183 ceph-mon[29736]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T19:12:59.185 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:12:58 smithi183 ceph-mon[29736]: from='mgr.14556 ' entity='mgr.y' 2024-09-10T19:12:59.215 INFO:journalctl@ceph.iscsi.iscsi.a.smithi191.stdout:Sep 10 19:12:58 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-iscsi-iscsi-a[61816]: debug there is no tcmu-runner data available 2024-09-10T19:12:59.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:12:58 smithi191 ceph-mon[32884]: 6.2 scrub starts 2024-09-10T19:12:59.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:12:58 smithi191 ceph-mon[32884]: 6.2 scrub ok 2024-09-10T19:12:59.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:12:58 smithi191 ceph-mon[32884]: 6.1c scrub starts 2024-09-10T19:12:59.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:12:58 smithi191 ceph-mon[32884]: 6.1c scrub ok 2024-09-10T19:12:59.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:12:58 smithi191 ceph-mon[32884]: pgmap v98: 148 pgs: 33 active+undersized, 26 active+undersized+degraded, 89 active+clean; 4.2 GiB data, 14 GiB used, 701 GiB / 715 GiB avail; 91 MiB/s wr, 1.46k op/s; 25595/207465 objects degraded (12.337%); 6.4 KiB/s, 0 objects/s recovering 2024-09-10T19:12:59.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:12:58 smithi191 ceph-mon[32884]: from='mgr.14556 ' entity='mgr.y' 2024-09-10T19:12:59.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:12:58 smithi191 ceph-mon[32884]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:12:59.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:12:58 smithi191 ceph-mon[32884]: purged_snaps scrub starts 2024-09-10T19:12:59.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:12:58 smithi191 ceph-mon[32884]: purged_snaps scrub ok 2024-09-10T19:12:59.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:12:58 smithi191 ceph-mon[32884]: from='mgr.14556 ' entity='mgr.y' 2024-09-10T19:12:59.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:12:58 smithi191 ceph-mon[32884]: from='mgr.14556 ' entity='mgr.y' 2024-09-10T19:12:59.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:12:58 smithi191 ceph-mon[32884]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T19:12:59.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:12:58 smithi191 ceph-mon[32884]: from='client.? 172.21.15.183:0/1574703421' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-10T19:12:59.217 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:12:58 smithi191 ceph-mon[32884]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-10T19:12:59.217 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:12:58 smithi191 ceph-mon[32884]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T19:12:59.217 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:12:58 smithi191 ceph-mon[32884]: from='mgr.14556 ' entity='mgr.y' 2024-09-10T19:12:59.219 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:12:59.252 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:12:59.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:12:59.253 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:12:59.353 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:12:59.386 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:12:59.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:12:59.387 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:12:59.488 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:12:59.515 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:12:59.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:12:59.516 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:12:59.617 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:12:59.647 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:12:59.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:12:59.648 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:12:59.660 INFO:teuthology.orchestra.run.smithi183.stderr:nodeep-scrub is unset 2024-09-10T19:12:59.748 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:12:59.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:12:59.778 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:12:59.779 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:12:59.879 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:12:59.910 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: 41 16 76863 76847 117.111 88.9375 0.00900244 0.00853483 2024-09-10T19:12:59.922 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:12:59.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:12:59.923 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:13:00.023 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:13:00.056 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:13:00.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:13:00.057 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:13:00.158 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:13:00.175 INFO:teuthology.orchestra.run.smithi183.stdout:osd.0: {} 2024-09-10T19:13:00.176 INFO:teuthology.orchestra.run.smithi183.stdout:osd.2: {} 2024-09-10T19:13:00.176 INFO:teuthology.orchestra.run.smithi183.stdout:osd.3: {} 2024-09-10T19:13:00.176 INFO:teuthology.orchestra.run.smithi183.stdout:osd.4: {} 2024-09-10T19:13:00.176 INFO:teuthology.orchestra.run.smithi183.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-10T19:13:00.176 INFO:teuthology.orchestra.run.smithi183.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-09-10T19:13:00.176 INFO:teuthology.orchestra.run.smithi183.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-10T19:13:00.176 INFO:teuthology.orchestra.run.smithi183.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-10T19:13:00.176 INFO:teuthology.orchestra.run.smithi183.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-10T19:13:00.177 INFO:teuthology.orchestra.run.smithi183.stdout:osd.5: {} 2024-09-10T19:13:00.177 INFO:teuthology.orchestra.run.smithi183.stdout:osd.6: {} 2024-09-10T19:13:00.177 INFO:teuthology.orchestra.run.smithi183.stdout:osd.7: {} 2024-09-10T19:13:00.177 INFO:teuthology.orchestra.run.smithi183.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-10T19:13:00.177 INFO:teuthology.orchestra.run.smithi183.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-10T19:13:00.177 INFO:teuthology.orchestra.run.smithi183.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-10T19:13:00.178 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:12:59 smithi183 ceph-mon[29736]: 5.1 scrub starts 2024-09-10T19:13:00.178 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:12:59 smithi183 ceph-mon[29736]: 5.1 scrub ok 2024-09-10T19:13:00.178 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:12:59 smithi183 ceph-mon[29736]: from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:13:00.178 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:12:59 smithi183 ceph-mon[29736]: 5.1e scrub starts 2024-09-10T19:13:00.178 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:12:59 smithi183 ceph-mon[29736]: 5.1e scrub ok 2024-09-10T19:13:00.178 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:12:59 smithi183 ceph-mon[29736]: from='client.? 172.21.15.183:0/1206838630' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T19:13:00.178 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:12:59 smithi183 ceph-mon[29736]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-10T19:13:00.178 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:12:59 smithi183 ceph-mon[29736]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-10T19:13:00.178 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:12:59 smithi183 ceph-mon[29736]: osdmap e102: 8 total, 7 up, 8 in 2024-09-10T19:13:00.178 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:12:59 smithi183 ceph-mon[29736]: osdmap e103: 8 total, 7 up, 8 in 2024-09-10T19:13:00.178 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:12:59 smithi183 ceph-mon[41562]: 5.1 scrub starts 2024-09-10T19:13:00.179 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:12:59 smithi183 ceph-mon[41562]: 5.1 scrub ok 2024-09-10T19:13:00.179 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:12:59 smithi183 ceph-mon[41562]: from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:13:00.179 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:12:59 smithi183 ceph-mon[41562]: 5.1e scrub starts 2024-09-10T19:13:00.179 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:12:59 smithi183 ceph-mon[41562]: 5.1e scrub ok 2024-09-10T19:13:00.179 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:12:59 smithi183 ceph-mon[41562]: from='client.? 172.21.15.183:0/1206838630' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T19:13:00.179 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:12:59 smithi183 ceph-mon[41562]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-10T19:13:00.179 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:12:59 smithi183 ceph-mon[41562]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-10T19:13:00.179 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:12:59 smithi183 ceph-mon[41562]: osdmap e102: 8 total, 7 up, 8 in 2024-09-10T19:13:00.179 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:12:59 smithi183 ceph-mon[41562]: osdmap e103: 8 total, 7 up, 8 in 2024-09-10T19:13:00.187 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:13:00.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:13:00.188 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:13:00.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:12:59 smithi191 ceph-mon[32884]: 5.1 scrub starts 2024-09-10T19:13:00.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:12:59 smithi191 ceph-mon[32884]: 5.1 scrub ok 2024-09-10T19:13:00.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:12:59 smithi191 ceph-mon[32884]: from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:13:00.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:12:59 smithi191 ceph-mon[32884]: 5.1e scrub starts 2024-09-10T19:13:00.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:12:59 smithi191 ceph-mon[32884]: 5.1e scrub ok 2024-09-10T19:13:00.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:12:59 smithi191 ceph-mon[32884]: from='client.? 172.21.15.183:0/1206838630' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T19:13:00.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:12:59 smithi191 ceph-mon[32884]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-10T19:13:00.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:12:59 smithi191 ceph-mon[32884]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-10T19:13:00.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:12:59 smithi191 ceph-mon[32884]: osdmap e102: 8 total, 7 up, 8 in 2024-09-10T19:13:00.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:12:59 smithi191 ceph-mon[32884]: osdmap e103: 8 total, 7 up, 8 in 2024-09-10T19:13:00.289 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:13:00.318 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:13:00.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:13:00.319 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:13:00.420 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:13:00.460 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:13:00.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:13:00.461 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:13:00.540 DEBUG:teuthology.orchestra.run.smithi183:> sudo pkill -f 'journalctl -f -n 0 -u ceph-320f6df8-6fa7-11ef-bcea-c7b262605968@osd.1.service' 2024-09-10T19:13:00.563 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:13:00.623 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:13:00.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:13:00.624 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:13:00.646 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-10T19:13:00.646 INFO:tasks.cephadm.osd.1:Stopped osd.1 2024-09-10T19:13:00.646 INFO:tasks.thrashosds.thrasher:Testing ceph-objectstore-tool on down osd.1 2024-09-10T19:13:00.646 INFO:tasks.thrashosds.thrasher:remote for osd 1 is ubuntu@smithi183.front.sepia.ceph.com 2024-09-10T19:13:00.647 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell -n osd.1 --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-1 --journal-path /var/lib/ceph/osd/ceph-1/journal --op list-pgs 2024-09-10T19:13:00.725 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:13:00.754 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:13:00.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:13:00.755 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:13:00.856 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:13:00.884 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:13:00.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:13:00.885 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:13:00.910 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: 42 16 78391 78375 116.596 95.5 0.0108117 0.0085717 2024-09-10T19:13:00.986 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:13:01.015 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:13:01.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:13:01.017 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:13:01.099 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/osd.1/config 2024-09-10T19:13:01.117 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:13:01.127 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-10T19:13:01.127 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-10T19:13:01.146 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:13:01.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:13:01.147 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:13:01.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:13:00 smithi183 ceph-mon[29736]: pgmap v99: 148 pgs: 33 active+undersized, 26 active+undersized+degraded, 89 active+clean; 4.5 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 921 B/s rd, 96 MiB/s wr, 1.53k op/s; 27025/220194 objects degraded (12.273%); 6.4 KiB/s, 0 objects/s recovering 2024-09-10T19:13:01.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:13:00 smithi183 ceph-mon[29736]: 5.14 scrub starts 2024-09-10T19:13:01.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:13:00 smithi183 ceph-mon[29736]: 5.14 scrub ok 2024-09-10T19:13:01.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:13:00 smithi183 ceph-mon[29736]: 6.16 scrub starts 2024-09-10T19:13:01.182 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:13:00 smithi183 ceph-mon[29736]: 5.6 scrub starts 2024-09-10T19:13:01.182 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:13:00 smithi183 ceph-mon[29736]: 5.6 scrub ok 2024-09-10T19:13:01.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:13:00 smithi183 ceph-mon[41562]: pgmap v99: 148 pgs: 33 active+undersized, 26 active+undersized+degraded, 89 active+clean; 4.5 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 921 B/s rd, 96 MiB/s wr, 1.53k op/s; 27025/220194 objects degraded (12.273%); 6.4 KiB/s, 0 objects/s recovering 2024-09-10T19:13:01.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:13:00 smithi183 ceph-mon[41562]: 5.14 scrub starts 2024-09-10T19:13:01.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:13:00 smithi183 ceph-mon[41562]: 5.14 scrub ok 2024-09-10T19:13:01.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:13:00 smithi183 ceph-mon[41562]: 6.16 scrub starts 2024-09-10T19:13:01.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:13:00 smithi183 ceph-mon[41562]: 5.6 scrub starts 2024-09-10T19:13:01.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:13:00 smithi183 ceph-mon[41562]: 5.6 scrub ok 2024-09-10T19:13:01.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:13:00 smithi191 ceph-mon[32884]: pgmap v99: 148 pgs: 33 active+undersized, 26 active+undersized+degraded, 89 active+clean; 4.5 GiB data, 15 GiB used, 700 GiB / 715 GiB avail; 921 B/s rd, 96 MiB/s wr, 1.53k op/s; 27025/220194 objects degraded (12.273%); 6.4 KiB/s, 0 objects/s recovering 2024-09-10T19:13:01.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:13:00 smithi191 ceph-mon[32884]: 5.14 scrub starts 2024-09-10T19:13:01.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:13:00 smithi191 ceph-mon[32884]: 5.14 scrub ok 2024-09-10T19:13:01.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:13:00 smithi191 ceph-mon[32884]: 6.16 scrub starts 2024-09-10T19:13:01.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:13:00 smithi191 ceph-mon[32884]: 5.6 scrub starts 2024-09-10T19:13:01.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:13:00 smithi191 ceph-mon[32884]: 5.6 scrub ok 2024-09-10T19:13:01.247 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:13:01.285 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:13:01.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:13:01.286 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:13:01.387 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:13:01.416 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:13:01.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:13:01.417 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:13:01.518 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:13:01.549 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:13:01.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:13:01.550 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:13:01.562 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Sep 10 19:13:01 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-y[29989]: ::ffff:172.21.15.191 - - [10/Sep/2024:19:13:01] "GET /metrics HTTP/1.1" 200 223740 "" "Prometheus/2.43.0" 2024-09-10T19:13:01.651 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:13:01.695 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:13:01.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:13:01.696 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:13:01.797 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:13:01.828 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:13:01.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:13:01.829 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:13:01.911 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: 43 16 79866 79850 116.028 92.1875 0.00979486 0.00861305 2024-09-10T19:13:01.929 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:13:01.959 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:13:01.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:13:01.961 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:13:02.061 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:13:02.089 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:13:02.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:13:02.090 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:13:02.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:13:01 smithi183 ceph-mon[29736]: 6.16 scrub ok 2024-09-10T19:13:02.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:13:01 smithi183 ceph-mon[29736]: purged_snaps scrub starts 2024-09-10T19:13:02.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:13:01 smithi183 ceph-mon[29736]: purged_snaps scrub ok 2024-09-10T19:13:02.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:13:01 smithi183 ceph-mon[29736]: 6.1e scrub starts 2024-09-10T19:13:02.182 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:13:01 smithi183 ceph-mon[29736]: 6.1e scrub ok 2024-09-10T19:13:02.182 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:13:01 smithi183 ceph-mon[29736]: 3.2 scrub starts 2024-09-10T19:13:02.182 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:13:01 smithi183 ceph-mon[29736]: 3.2 scrub ok 2024-09-10T19:13:02.182 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:13:01 smithi183 ceph-mon[29736]: osdmap e104: 8 total, 7 up, 8 in 2024-09-10T19:13:02.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:13:01 smithi183 ceph-mon[41562]: 6.16 scrub ok 2024-09-10T19:13:02.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:13:01 smithi183 ceph-mon[41562]: purged_snaps scrub starts 2024-09-10T19:13:02.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:13:01 smithi183 ceph-mon[41562]: purged_snaps scrub ok 2024-09-10T19:13:02.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:13:01 smithi183 ceph-mon[41562]: 6.1e scrub starts 2024-09-10T19:13:02.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:13:01 smithi183 ceph-mon[41562]: 6.1e scrub ok 2024-09-10T19:13:02.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:13:01 smithi183 ceph-mon[41562]: 3.2 scrub starts 2024-09-10T19:13:02.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:13:01 smithi183 ceph-mon[41562]: 3.2 scrub ok 2024-09-10T19:13:02.183 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:13:01 smithi183 ceph-mon[41562]: osdmap e104: 8 total, 7 up, 8 in 2024-09-10T19:13:02.190 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:13:02.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:13:01 smithi191 ceph-mon[32884]: 6.16 scrub ok 2024-09-10T19:13:02.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:13:01 smithi191 ceph-mon[32884]: purged_snaps scrub starts 2024-09-10T19:13:02.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:13:01 smithi191 ceph-mon[32884]: purged_snaps scrub ok 2024-09-10T19:13:02.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:13:01 smithi191 ceph-mon[32884]: 6.1e scrub starts 2024-09-10T19:13:02.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:13:01 smithi191 ceph-mon[32884]: 6.1e scrub ok 2024-09-10T19:13:02.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:13:01 smithi191 ceph-mon[32884]: 3.2 scrub starts 2024-09-10T19:13:02.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:13:01 smithi191 ceph-mon[32884]: 3.2 scrub ok 2024-09-10T19:13:02.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:13:01 smithi191 ceph-mon[32884]: osdmap e104: 8 total, 7 up, 8 in 2024-09-10T19:13:02.221 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:13:02.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:13:02.222 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:13:02.323 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:13:02.351 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:13:02.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:13:02.352 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:13:02.453 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:13:02.480 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:13:02.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:13:02.481 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:13:02.582 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:13:02.609 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:13:02.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:13:02.610 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:13:02.711 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:13:02.731 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:13:02.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:13:02.732 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:13:02.833 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:13:02.870 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:13:02.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:13:02.871 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:13:02.911 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: 44 16 81246 81230 115.351 86.25 0.0130075 0.00866454 2024-09-10T19:13:02.973 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:13:03.008 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:13:03.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:13:03.009 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:13:03.109 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:13:03.129 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-10T19:13:03.145 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:13:03.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:13:03.147 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:13:03.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:13:02 smithi191 ceph-mon[32884]: 2.13 scrub starts 2024-09-10T19:13:03.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:13:02 smithi191 ceph-mon[32884]: 2.13 scrub ok 2024-09-10T19:13:03.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:13:02 smithi191 ceph-mon[32884]: pgmap v103: 148 pgs: 33 active+undersized, 26 active+undersized+degraded, 89 active+clean; 4.7 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 852 B/s rd, 78 MiB/s wr, 1.26k op/s; 28433/230085 objects degraded (12.358%) 2024-09-10T19:13:03.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:13:02 smithi191 ceph-mon[32884]: Health check update: Degraded data redundancy: 28433/230085 objects degraded (12.358%), 26 pgs degraded (PG_DEGRADED) 2024-09-10T19:13:03.248 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:13:03.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:13:03.285 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:13:03.286 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:13:03.386 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:13:03.414 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:13:03.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:13:03.415 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:13:03.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:13:02 smithi183 ceph-mon[29736]: 2.13 scrub starts 2024-09-10T19:13:03.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:13:02 smithi183 ceph-mon[29736]: 2.13 scrub ok 2024-09-10T19:13:03.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:13:02 smithi183 ceph-mon[29736]: pgmap v103: 148 pgs: 33 active+undersized, 26 active+undersized+degraded, 89 active+clean; 4.7 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 852 B/s rd, 78 MiB/s wr, 1.26k op/s; 28433/230085 objects degraded (12.358%) 2024-09-10T19:13:03.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:13:02 smithi183 ceph-mon[29736]: Health check update: Degraded data redundancy: 28433/230085 objects degraded (12.358%), 26 pgs degraded (PG_DEGRADED) 2024-09-10T19:13:03.431 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:13:02 smithi183 ceph-mon[41562]: 2.13 scrub starts 2024-09-10T19:13:03.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:13:02 smithi183 ceph-mon[41562]: 2.13 scrub ok 2024-09-10T19:13:03.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:13:02 smithi183 ceph-mon[41562]: pgmap v103: 148 pgs: 33 active+undersized, 26 active+undersized+degraded, 89 active+clean; 4.7 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 852 B/s rd, 78 MiB/s wr, 1.26k op/s; 28433/230085 objects degraded (12.358%) 2024-09-10T19:13:03.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:13:02 smithi183 ceph-mon[41562]: Health check update: Degraded data redundancy: 28433/230085 objects degraded (12.358%), 26 pgs degraded (PG_DEGRADED) 2024-09-10T19:13:03.516 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:13:03.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:13:03.547 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:13:03.548 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:13:03.606 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph osd set noscrub 2024-09-10T19:13:03.620 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-10T19:13:03.621 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph daemon osd.1 dump_historic_ops 2024-09-10T19:13:03.648 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:13:03.675 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:13:03.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:13:03.676 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:13:03.776 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:13:03.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:13:03.841 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:13:03.841 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:13:03.911 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: 45 16 82597 82581 114.664 84.4375 0.0186688 0.00871552 2024-09-10T19:13:03.942 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:13:03.990 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:13:03.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:13:03.992 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:13:04.093 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:13:04.146 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:13:04.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:13:04.148 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:13:04.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:13:03 smithi191 ceph-mon[32884]: 4.3 deep-scrub starts 2024-09-10T19:13:04.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:13:03 smithi191 ceph-mon[32884]: 4.3 deep-scrub ok 2024-09-10T19:13:04.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:13:03 smithi191 ceph-mon[32884]: 4.1a scrub starts 2024-09-10T19:13:04.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:13:03 smithi191 ceph-mon[32884]: 4.1a scrub ok 2024-09-10T19:13:04.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:13:03 smithi191 ceph-mon[32884]: 2.19 scrub starts 2024-09-10T19:13:04.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:13:03 smithi191 ceph-mon[32884]: 2.19 scrub ok 2024-09-10T19:13:04.229 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:13:04.249 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:13:04.263 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:13:04.316 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:13:04.317 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:13:03 smithi183 ceph-mon[29736]: 4.3 deep-scrub starts 2024-09-10T19:13:04.317 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:13:03 smithi183 ceph-mon[29736]: 4.3 deep-scrub ok 2024-09-10T19:13:04.317 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:13:03 smithi183 ceph-mon[29736]: 4.1a scrub starts 2024-09-10T19:13:04.317 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:13:03 smithi183 ceph-mon[29736]: 4.1a scrub ok 2024-09-10T19:13:04.317 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:13:03 smithi183 ceph-mon[29736]: 2.19 scrub starts 2024-09-10T19:13:04.317 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:13:03 smithi183 ceph-mon[29736]: 2.19 scrub ok 2024-09-10T19:13:04.317 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:13:03 smithi183 ceph-mon[41562]: 4.3 deep-scrub starts 2024-09-10T19:13:04.318 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:13:03 smithi183 ceph-mon[41562]: 4.3 deep-scrub ok 2024-09-10T19:13:04.318 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:13:03 smithi183 ceph-mon[41562]: 4.1a scrub starts 2024-09-10T19:13:04.318 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:13:03 smithi183 ceph-mon[41562]: 4.1a scrub ok 2024-09-10T19:13:04.318 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:13:03 smithi183 ceph-mon[41562]: 2.19 scrub starts 2024-09-10T19:13:04.318 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:13:03 smithi183 ceph-mon[41562]: 2.19 scrub ok 2024-09-10T19:13:04.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:13:04.319 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:13:04.420 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:13:04.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:13:04.459 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:13:04.460 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:13:04.517 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:13:04.560 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:13:04.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:13:04.615 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:13:04.615 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:13:04.716 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:13:04.745 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:13:04.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:13:04.746 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:13:04.847 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:13:04.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:13:04.884 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:13:04.884 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:13:04.911 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: 46 16 83900 83884 113.942 81.4375 0.0255251 0.0087718 2024-09-10T19:13:04.986 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:13:05.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:13:05.018 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:13:05.018 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:13:05.119 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:13:05.155 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:13:05.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:13:05.156 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:13:05.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:13:04 smithi191 ceph-mon[32884]: pgmap v104: 148 pgs: 33 active+undersized, 26 active+undersized+degraded, 89 active+clean; 4.7 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 853 B/s rd, 79 MiB/s wr, 1.26k op/s; 28433/230085 objects degraded (12.358%) 2024-09-10T19:13:05.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:13:04 smithi191 ceph-mon[32884]: osdmap e105: 8 total, 7 up, 8 in 2024-09-10T19:13:05.257 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:13:05.288 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:13:04 smithi183 ceph-mon[29736]: pgmap v104: 148 pgs: 33 active+undersized, 26 active+undersized+degraded, 89 active+clean; 4.7 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 853 B/s rd, 79 MiB/s wr, 1.26k op/s; 28433/230085 objects degraded (12.358%) 2024-09-10T19:13:05.288 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:13:04 smithi183 ceph-mon[29736]: osdmap e105: 8 total, 7 up, 8 in 2024-09-10T19:13:05.288 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:13:04 smithi183 ceph-mon[41562]: pgmap v104: 148 pgs: 33 active+undersized, 26 active+undersized+degraded, 89 active+clean; 4.7 GiB data, 16 GiB used, 699 GiB / 715 GiB avail; 853 B/s rd, 79 MiB/s wr, 1.26k op/s; 28433/230085 objects degraded (12.358%) 2024-09-10T19:13:05.288 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:13:04 smithi183 ceph-mon[41562]: osdmap e105: 8 total, 7 up, 8 in 2024-09-10T19:13:05.294 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:13:05.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:13:05.295 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:13:05.396 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:13:05.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:13:05.436 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:13:05.436 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:13:05.537 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:13:05.581 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:13:05.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:13:05.582 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:13:05.683 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:13:05.704 INFO:teuthology.orchestra.run.smithi183.stdout:4.e 2024-09-10T19:13:05.704 INFO:teuthology.orchestra.run.smithi183.stdout:2.4 2024-09-10T19:13:05.705 INFO:teuthology.orchestra.run.smithi183.stdout:4.b 2024-09-10T19:13:05.705 INFO:teuthology.orchestra.run.smithi183.stdout:4.7 2024-09-10T19:13:05.705 INFO:teuthology.orchestra.run.smithi183.stdout:2.8 2024-09-10T19:13:05.705 INFO:teuthology.orchestra.run.smithi183.stdout:4.4 2024-09-10T19:13:05.705 INFO:teuthology.orchestra.run.smithi183.stdout:4.f 2024-09-10T19:13:05.705 INFO:teuthology.orchestra.run.smithi183.stdout:2.10 2024-09-10T19:13:05.705 INFO:teuthology.orchestra.run.smithi183.stdout:4.d 2024-09-10T19:13:05.705 INFO:teuthology.orchestra.run.smithi183.stdout:4.1d 2024-09-10T19:13:05.705 INFO:teuthology.orchestra.run.smithi183.stdout:4.18 2024-09-10T19:13:05.705 INFO:teuthology.orchestra.run.smithi183.stdout:6.19 2024-09-10T19:13:05.705 INFO:teuthology.orchestra.run.smithi183.stdout:4.12 2024-09-10T19:13:05.705 INFO:teuthology.orchestra.run.smithi183.stdout:4.9 2024-09-10T19:13:05.705 INFO:teuthology.orchestra.run.smithi183.stdout:2.9 2024-09-10T19:13:05.706 INFO:teuthology.orchestra.run.smithi183.stdout:4.1c 2024-09-10T19:13:05.706 INFO:teuthology.orchestra.run.smithi183.stdout:2.15 2024-09-10T19:13:05.706 INFO:teuthology.orchestra.run.smithi183.stdout:7.b 2024-09-10T19:13:05.706 INFO:teuthology.orchestra.run.smithi183.stdout:4.6 2024-09-10T19:13:05.706 INFO:teuthology.orchestra.run.smithi183.stdout:4.17 2024-09-10T19:13:05.706 INFO:teuthology.orchestra.run.smithi183.stdout:5.0 2024-09-10T19:13:05.706 INFO:teuthology.orchestra.run.smithi183.stdout:4.a 2024-09-10T19:13:05.706 INFO:teuthology.orchestra.run.smithi183.stdout:4.13 2024-09-10T19:13:05.706 INFO:teuthology.orchestra.run.smithi183.stdout:4.14 2024-09-10T19:13:05.706 INFO:teuthology.orchestra.run.smithi183.stdout:2.0 2024-09-10T19:13:05.706 INFO:teuthology.orchestra.run.smithi183.stdout:5.13 2024-09-10T19:13:05.706 INFO:teuthology.orchestra.run.smithi183.stdout:2.6 2024-09-10T19:13:05.706 INFO:teuthology.orchestra.run.smithi183.stdout:2.a 2024-09-10T19:13:05.706 INFO:teuthology.orchestra.run.smithi183.stdout:5.15 2024-09-10T19:13:05.706 INFO:teuthology.orchestra.run.smithi183.stdout:5.1a 2024-09-10T19:13:05.707 INFO:teuthology.orchestra.run.smithi183.stdout:5.7 2024-09-10T19:13:05.707 INFO:teuthology.orchestra.run.smithi183.stdout:2.11 2024-09-10T19:13:05.707 INFO:teuthology.orchestra.run.smithi183.stdout:5.1d 2024-09-10T19:13:05.707 INFO:teuthology.orchestra.run.smithi183.stdout:6.1 2024-09-10T19:13:05.707 INFO:teuthology.orchestra.run.smithi183.stdout:2.d 2024-09-10T19:13:05.707 INFO:teuthology.orchestra.run.smithi183.stdout:7.9 2024-09-10T19:13:05.707 INFO:teuthology.orchestra.run.smithi183.stdout:3.0 2024-09-10T19:13:05.707 INFO:teuthology.orchestra.run.smithi183.stdout:5.19 2024-09-10T19:13:05.707 INFO:teuthology.orchestra.run.smithi183.stdout:5.5 2024-09-10T19:13:05.707 INFO:teuthology.orchestra.run.smithi183.stdout:5.12 2024-09-10T19:13:05.707 INFO:teuthology.orchestra.run.smithi183.stdout:4.10 2024-09-10T19:13:05.707 INFO:teuthology.orchestra.run.smithi183.stdout:6.e 2024-09-10T19:13:05.707 INFO:teuthology.orchestra.run.smithi183.stdout:5.16 2024-09-10T19:13:05.707 INFO:teuthology.orchestra.run.smithi183.stdout:5.17 2024-09-10T19:13:05.707 INFO:teuthology.orchestra.run.smithi183.stdout:5.18 2024-09-10T19:13:05.707 INFO:teuthology.orchestra.run.smithi183.stdout:5.8 2024-09-10T19:13:05.708 INFO:teuthology.orchestra.run.smithi183.stdout:5.c 2024-09-10T19:13:05.708 INFO:teuthology.orchestra.run.smithi183.stdout:7.2 2024-09-10T19:13:05.708 INFO:teuthology.orchestra.run.smithi183.stdout:6.10 2024-09-10T19:13:05.708 INFO:teuthology.orchestra.run.smithi183.stdout:2.2 2024-09-10T19:13:05.708 INFO:teuthology.orchestra.run.smithi183.stdout:6.18 2024-09-10T19:13:05.708 INFO:teuthology.orchestra.run.smithi183.stdout:7.0 2024-09-10T19:13:05.708 INFO:teuthology.orchestra.run.smithi183.stdout:6.1a 2024-09-10T19:13:05.708 INFO:teuthology.orchestra.run.smithi183.stdout:6.1d 2024-09-10T19:13:05.708 INFO:teuthology.orchestra.run.smithi183.stdout:6.4 2024-09-10T19:13:05.708 INFO:teuthology.orchestra.run.smithi183.stdout:6.b 2024-09-10T19:13:05.708 INFO:teuthology.orchestra.run.smithi183.stdout:6.d 2024-09-10T19:13:05.708 INFO:teuthology.orchestra.run.smithi183.stdout:7.4 2024-09-10T19:13:05.708 INFO:teuthology.orchestra.run.smithi183.stdout:7.d 2024-09-10T19:13:05.720 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:13:05.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:13:05.722 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:13:05.822 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:13:05.864 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:13:05.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:13:05.865 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:13:05.911 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: 47 16 85217 85201 113.269 82.3125 0.00745709 0.00882398 2024-09-10T19:13:05.966 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:13:05.998 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:13:06.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:13:06.000 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:13:06.101 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:13:06.133 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:13:06.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:13:06.134 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:13:06.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:13:05 smithi183 ceph-mon[29736]: 5.11 scrub starts 2024-09-10T19:13:06.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:13:05 smithi183 ceph-mon[29736]: 5.11 scrub ok 2024-09-10T19:13:06.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:13:05 smithi183 ceph-mon[29736]: 6.7 scrub starts 2024-09-10T19:13:06.182 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:13:05 smithi183 ceph-mon[29736]: 6.7 scrub ok 2024-09-10T19:13:06.182 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:13:05 smithi183 ceph-mon[29736]: 2.c scrub starts 2024-09-10T19:13:06.182 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:13:05 smithi183 ceph-mon[29736]: 2.c scrub ok 2024-09-10T19:13:06.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:13:05 smithi183 ceph-mon[41562]: 5.11 scrub starts 2024-09-10T19:13:06.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:13:05 smithi183 ceph-mon[41562]: 5.11 scrub ok 2024-09-10T19:13:06.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:13:05 smithi183 ceph-mon[41562]: 6.7 scrub starts 2024-09-10T19:13:06.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:13:05 smithi183 ceph-mon[41562]: 6.7 scrub ok 2024-09-10T19:13:06.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:13:05 smithi183 ceph-mon[41562]: 2.c scrub starts 2024-09-10T19:13:06.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:13:05 smithi183 ceph-mon[41562]: 2.c scrub ok 2024-09-10T19:13:06.235 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:13:06.271 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:13:06.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:13:06.272 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:13:06.372 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:13:06.403 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:13:06.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:13:06.404 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:13:06.465 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:13:05 smithi191 ceph-mon[32884]: 5.11 scrub starts 2024-09-10T19:13:06.465 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:13:05 smithi191 ceph-mon[32884]: 5.11 scrub ok 2024-09-10T19:13:06.465 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:13:05 smithi191 ceph-mon[32884]: 6.7 scrub starts 2024-09-10T19:13:06.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:13:05 smithi191 ceph-mon[32884]: 6.7 scrub ok 2024-09-10T19:13:06.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:13:05 smithi191 ceph-mon[32884]: 2.c scrub starts 2024-09-10T19:13:06.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:13:05 smithi191 ceph-mon[32884]: 2.c scrub ok 2024-09-10T19:13:06.504 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:13:06.542 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:13:06.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:13:06.543 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:13:06.643 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:13:06.678 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:13:06.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:13:06.679 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:13:06.779 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:13:06.813 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:13:06.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:13:06.814 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:13:06.911 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: 48 15 86685 86670 112.822 91.8125 0.00490866 0.00885907 2024-09-10T19:13:06.915 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:13:06.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:13:06.948 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:13:06.948 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:13:07.049 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:13:07.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:13:07.076 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:13:07.076 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:13:07.177 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:13:07.207 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:13:07.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:13:07.209 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:13:07.309 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:13:07.335 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:13:07.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:13:07.336 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:13:07.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:13:06 smithi183 ceph-mon[29736]: pgmap v106: 148 pgs: 33 active+undersized, 26 active+undersized+degraded, 89 active+clean; 5.1 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 856 B/s rd, 108 MiB/s wr, 1.73k op/s; 31070/251121 objects degraded (12.373%) 2024-09-10T19:13:07.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:13:06 smithi183 ceph-mon[29736]: 4.11 scrub starts 2024-09-10T19:13:07.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:13:06 smithi183 ceph-mon[29736]: 4.11 scrub ok 2024-09-10T19:13:07.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:13:06 smithi183 ceph-mon[29736]: 2.17 deep-scrub starts 2024-09-10T19:13:07.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:13:06 smithi183 ceph-mon[29736]: 2.17 deep-scrub ok 2024-09-10T19:13:07.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:13:06 smithi183 ceph-mon[29736]: 5.f scrub starts 2024-09-10T19:13:07.433 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:13:06 smithi183 ceph-mon[29736]: 5.f scrub ok 2024-09-10T19:13:07.433 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:13:06 smithi183 ceph-mon[41562]: pgmap v106: 148 pgs: 33 active+undersized, 26 active+undersized+degraded, 89 active+clean; 5.1 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 856 B/s rd, 108 MiB/s wr, 1.73k op/s; 31070/251121 objects degraded (12.373%) 2024-09-10T19:13:07.433 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:13:06 smithi183 ceph-mon[41562]: 4.11 scrub starts 2024-09-10T19:13:07.433 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:13:06 smithi183 ceph-mon[41562]: 4.11 scrub ok 2024-09-10T19:13:07.433 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:13:06 smithi183 ceph-mon[41562]: 2.17 deep-scrub starts 2024-09-10T19:13:07.433 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:13:06 smithi183 ceph-mon[41562]: 2.17 deep-scrub ok 2024-09-10T19:13:07.433 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:13:06 smithi183 ceph-mon[41562]: 5.f scrub starts 2024-09-10T19:13:07.433 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:13:06 smithi183 ceph-mon[41562]: 5.f scrub ok 2024-09-10T19:13:07.438 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:13:07.450 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:13:06 smithi191 ceph-mon[32884]: pgmap v106: 148 pgs: 33 active+undersized, 26 active+undersized+degraded, 89 active+clean; 5.1 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 856 B/s rd, 108 MiB/s wr, 1.73k op/s; 31070/251121 objects degraded (12.373%) 2024-09-10T19:13:07.451 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:13:06 smithi191 ceph-mon[32884]: 4.11 scrub starts 2024-09-10T19:13:07.451 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:13:06 smithi191 ceph-mon[32884]: 4.11 scrub ok 2024-09-10T19:13:07.451 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:13:06 smithi191 ceph-mon[32884]: 2.17 deep-scrub starts 2024-09-10T19:13:07.451 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:13:06 smithi191 ceph-mon[32884]: 2.17 deep-scrub ok 2024-09-10T19:13:07.451 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:13:06 smithi191 ceph-mon[32884]: 5.f scrub starts 2024-09-10T19:13:07.451 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:13:06 smithi191 ceph-mon[32884]: 5.f scrub ok 2024-09-10T19:13:07.469 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:13:07.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:13:07.470 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:13:07.570 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:13:07.605 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:13:07.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:13:07.606 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:13:07.707 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:13:07.715 INFO:journalctl@ceph.mgr.x.smithi191.stdout:Sep 10 19:13:07 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-x[34967]: ::ffff:172.21.15.191 - - [10/Sep/2024:19:13:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T19:13:07.746 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:13:07.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:13:07.747 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:13:07.848 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:13:07.875 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:13:07.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:13:07.877 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:13:07.911 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: 49 16 88141 88125 112.375 90.9375 0.00931494 0.00889411 2024-09-10T19:13:07.977 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:13:08.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:13:08.009 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:13:08.010 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:13:08.110 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:13:08.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:13:08.161 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:13:08.161 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:13:08.251 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:13:08 smithi183 ceph-mon[29736]: 3.1 scrub starts 2024-09-10T19:13:08.251 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:13:08 smithi183 ceph-mon[29736]: 3.1 scrub ok 2024-09-10T19:13:08.251 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:13:08 smithi183 ceph-mon[41562]: 3.1 scrub starts 2024-09-10T19:13:08.251 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:13:08 smithi183 ceph-mon[41562]: 3.1 scrub ok 2024-09-10T19:13:08.262 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:13:08.300 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:13:08.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:13:08.301 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:13:08.402 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:13:08.437 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:13:08.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:13:08.439 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:13:08.465 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:13:08 smithi191 ceph-mon[32884]: 3.1 scrub starts 2024-09-10T19:13:08.465 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:13:08 smithi191 ceph-mon[32884]: 3.1 scrub ok 2024-09-10T19:13:08.539 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:13:08.570 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:13:08.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:13:08.571 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:13:08.609 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell -n osd.1 --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-1 --journal-path /var/lib/ceph/osd/ceph-1/journal --op export --pgid 5.13 --file /var/log/ceph/exp.5.13.1 2024-09-10T19:13:08.672 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:13:08.697 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:13:08.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:13:08.698 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:13:08.799 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:13:08.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:13:08.845 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:13:08.845 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:13:08.914 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: 50 16 89540 89524 111.87 87.4375 0.00907515 0.00893399 2024-09-10T19:13:08.945 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:13:08.987 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:13:08.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:13:08.988 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:13:08.999 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/osd.1/config 2024-09-10T19:13:09.089 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:13:09.123 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:13:09.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:13:09.124 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:13:09.137 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:13:09 smithi183 ceph-mon[29736]: pgmap v107: 148 pgs: 33 active+undersized, 26 active+undersized+degraded, 89 active+clean; 5.1 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 643 B/s rd, 55 MiB/s wr, 882 op/s; 31070/251121 objects degraded (12.373%) 2024-09-10T19:13:09.137 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:13:09 smithi183 ceph-mon[29736]: 5.a scrub starts 2024-09-10T19:13:09.137 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:13:09 smithi183 ceph-mon[29736]: 5.a scrub ok 2024-09-10T19:13:09.137 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:13:09 smithi183 ceph-mon[41562]: pgmap v107: 148 pgs: 33 active+undersized, 26 active+undersized+degraded, 89 active+clean; 5.1 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 643 B/s rd, 55 MiB/s wr, 882 op/s; 31070/251121 objects degraded (12.373%) 2024-09-10T19:13:09.137 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:13:09 smithi183 ceph-mon[41562]: 5.a scrub starts 2024-09-10T19:13:09.137 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:13:09 smithi183 ceph-mon[41562]: 5.a scrub ok 2024-09-10T19:13:09.216 INFO:journalctl@ceph.iscsi.iscsi.a.smithi191.stdout:Sep 10 19:13:08 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-iscsi-iscsi-a[61816]: debug there is no tcmu-runner data available 2024-09-10T19:13:09.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:13:09 smithi191 ceph-mon[32884]: pgmap v107: 148 pgs: 33 active+undersized, 26 active+undersized+degraded, 89 active+clean; 5.1 GiB data, 17 GiB used, 698 GiB / 715 GiB avail; 643 B/s rd, 55 MiB/s wr, 882 op/s; 31070/251121 objects degraded (12.373%) 2024-09-10T19:13:09.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:13:09 smithi191 ceph-mon[32884]: 5.a scrub starts 2024-09-10T19:13:09.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:13:09 smithi191 ceph-mon[32884]: 5.a scrub ok 2024-09-10T19:13:09.225 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:13:09.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:13:09.268 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:13:09.268 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:13:09.369 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:13:09.398 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:13:09.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:13:09.399 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:13:09.440 INFO:teuthology.orchestra.run.smithi183.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-09-10T19:13:09.500 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:13:09.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:13:09.533 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:13:09.533 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:13:09.634 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:13:09.670 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:13:09.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:13:09.672 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:13:09.772 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:13:09.805 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:13:09.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:13:09.806 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:13:09.907 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:13:09.914 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: 51 16 90901 90885 111.345 85.0625 0.0203483 0.00897619 2024-09-10T19:13:09.944 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:13:09.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:13:09.945 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:13:10.045 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:13:10.076 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:13:10.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:13:10.077 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:13:10.178 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:13:10.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:13:10 smithi183 ceph-mon[29736]: 6.1b scrub starts 2024-09-10T19:13:10.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:13:10 smithi183 ceph-mon[29736]: 6.1b scrub ok 2024-09-10T19:13:10.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:13:10 smithi183 ceph-mon[29736]: 2.14 scrub starts 2024-09-10T19:13:10.182 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:13:10 smithi183 ceph-mon[29736]: 2.14 scrub ok 2024-09-10T19:13:10.182 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:13:10 smithi183 ceph-mon[29736]: from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:13:10.182 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:13:10 smithi183 ceph-mon[29736]: 6.17 scrub starts 2024-09-10T19:13:10.182 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:13:10 smithi183 ceph-mon[29736]: 6.17 scrub ok 2024-09-10T19:13:10.182 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:13:10 smithi183 ceph-mon[29736]: Health check update: Degraded data redundancy: 31070/251121 objects degraded (12.373%), 26 pgs degraded (PG_DEGRADED) 2024-09-10T19:13:10.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:13:10 smithi183 ceph-mon[41562]: 6.1b scrub starts 2024-09-10T19:13:10.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:13:10 smithi183 ceph-mon[41562]: 6.1b scrub ok 2024-09-10T19:13:10.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:13:10 smithi183 ceph-mon[41562]: 2.14 scrub starts 2024-09-10T19:13:10.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:13:10 smithi183 ceph-mon[41562]: 2.14 scrub ok 2024-09-10T19:13:10.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:13:10 smithi183 ceph-mon[41562]: from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:13:10.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:13:10 smithi183 ceph-mon[41562]: 6.17 scrub starts 2024-09-10T19:13:10.183 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:13:10 smithi183 ceph-mon[41562]: 6.17 scrub ok 2024-09-10T19:13:10.183 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:13:10 smithi183 ceph-mon[41562]: Health check update: Degraded data redundancy: 31070/251121 objects degraded (12.373%), 26 pgs degraded (PG_DEGRADED) 2024-09-10T19:13:10.208 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:13:10.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:13:10.209 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:13:10.310 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:13:10.345 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:13:10.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:13:10.346 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:13:10.447 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:13:10.458 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-10T19:13:10.458 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph daemon osd.3 dump_ops_in_flight 2024-09-10T19:13:10.465 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:13:10 smithi191 ceph-mon[32884]: 6.1b scrub starts 2024-09-10T19:13:10.465 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:13:10 smithi191 ceph-mon[32884]: 6.1b scrub ok 2024-09-10T19:13:10.465 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:13:10 smithi191 ceph-mon[32884]: 2.14 scrub starts 2024-09-10T19:13:10.465 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:13:10 smithi191 ceph-mon[32884]: 2.14 scrub ok 2024-09-10T19:13:10.465 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:13:10 smithi191 ceph-mon[32884]: from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:13:10.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:13:10 smithi191 ceph-mon[32884]: 6.17 scrub starts 2024-09-10T19:13:10.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:13:10 smithi191 ceph-mon[32884]: 6.17 scrub ok 2024-09-10T19:13:10.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:13:10 smithi191 ceph-mon[32884]: Health check update: Degraded data redundancy: 31070/251121 objects degraded (12.373%), 26 pgs degraded (PG_DEGRADED) 2024-09-10T19:13:10.474 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:13:10.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:13:10.475 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:13:10.576 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:13:10.629 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:13:10.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:13:10.630 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:13:10.731 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:13:10.758 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:13:10.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:13:10.759 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:13:10.860 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:13:10.899 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:13:10.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:13:10.900 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:13:10.914 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: 52 16 92236 92220 110.808 83.4375 0.0222027 0.00901986 2024-09-10T19:13:11.001 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:13:11.036 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:13:11.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:13:11.037 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:13:11.138 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:13:11.166 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:13:11.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:13:11.167 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:13:11.268 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:13:11.296 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:13:11.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:13:11.297 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:13:11.316 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:13:11 smithi183 ceph-mon[29736]: pgmap v108: 148 pgs: 33 active+undersized, 26 active+undersized+degraded, 89 active+clean; 5.3 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 1.2 KiB/s rd, 79 MiB/s wr, 1.26k op/s; 32401/262974 objects degraded (12.321%) 2024-09-10T19:13:11.316 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:13:11 smithi183 ceph-mon[41562]: pgmap v108: 148 pgs: 33 active+undersized, 26 active+undersized+degraded, 89 active+clean; 5.3 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 1.2 KiB/s rd, 79 MiB/s wr, 1.26k op/s; 32401/262974 objects degraded (12.321%) 2024-09-10T19:13:11.398 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:13:11.437 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:13:11.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:13:11.438 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:13:11.465 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:13:11 smithi191 ceph-mon[32884]: pgmap v108: 148 pgs: 33 active+undersized, 26 active+undersized+degraded, 89 active+clean; 5.3 GiB data, 18 GiB used, 698 GiB / 715 GiB avail; 1.2 KiB/s rd, 79 MiB/s wr, 1.26k op/s; 32401/262974 objects degraded (12.321%) 2024-09-10T19:13:11.539 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:13:11.578 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:13:11.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:13:11.579 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:13:11.680 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:13:11.681 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Sep 10 19:13:11 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-y[29989]: ::ffff:172.21.15.191 - - [10/Sep/2024:19:13:11] "GET /metrics HTTP/1.1" 200 223740 "" "Prometheus/2.43.0" 2024-09-10T19:13:11.719 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:13:11.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:13:11.720 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:13:11.821 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:13:11.847 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:13:11.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:13:11.849 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:13:11.914 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: 53 16 93591 93575 110.315 84.6875 0.00374789 0.00906069 2024-09-10T19:13:11.950 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:13:11.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:13:11.982 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:13:11.983 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:13:12.083 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:13:12.118 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:13:12.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:13:12.119 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:13:12.219 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:13:12.258 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:13:12.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:13:12.259 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:13:12.359 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:13:12.388 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:13:12.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:13:12.389 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:13:12.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:13:12 smithi183 ceph-mon[29736]: 5.1c scrub starts 2024-09-10T19:13:12.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:13:12 smithi183 ceph-mon[29736]: 5.1c scrub ok 2024-09-10T19:13:12.431 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:13:12 smithi183 ceph-mon[41562]: 5.1c scrub starts 2024-09-10T19:13:12.431 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:13:12 smithi183 ceph-mon[41562]: 5.1c scrub ok 2024-09-10T19:13:12.465 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:13:12 smithi191 ceph-mon[32884]: 5.1c scrub starts 2024-09-10T19:13:12.465 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:13:12 smithi191 ceph-mon[32884]: 5.1c scrub ok 2024-09-10T19:13:12.490 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:13:12.527 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:13:12.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:13:12.528 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:13:12.628 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:13:12.669 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:13:12.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:13:12.670 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:13:12.771 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:13:12.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:13:12.806 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:13:12.806 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:13:12.907 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:13:12.914 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: 54 16 95038 95022 109.947 90.4375 0.00683912 0.00909079 2024-09-10T19:13:12.938 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:13:12.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:13:12.939 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:13:13.040 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:13:13.073 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:13:13.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:13:13.075 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:13:13.175 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:13:13.199 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:13:13.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:13:13.200 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:13:13.301 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:13:13.315 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:13:13 smithi183 ceph-mon[29736]: pgmap v109: 148 pgs: 33 active+undersized, 26 active+undersized+degraded, 89 active+clean; 5.5 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1023 B/s rd, 88 MiB/s wr, 1.40k op/s; 33709/272121 objects degraded (12.388%) 2024-09-10T19:13:13.316 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:13:13 smithi183 ceph-mon[41562]: pgmap v109: 148 pgs: 33 active+undersized, 26 active+undersized+degraded, 89 active+clean; 5.5 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1023 B/s rd, 88 MiB/s wr, 1.40k op/s; 33709/272121 objects degraded (12.388%) 2024-09-10T19:13:13.340 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:13:13.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:13:13.341 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:13:13.442 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:13:13.465 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:13:13 smithi191 ceph-mon[32884]: pgmap v109: 148 pgs: 33 active+undersized, 26 active+undersized+degraded, 89 active+clean; 5.5 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1023 B/s rd, 88 MiB/s wr, 1.40k op/s; 33709/272121 objects degraded (12.388%) 2024-09-10T19:13:13.470 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:13:13.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:13:13.472 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:13:13.573 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:13:13.601 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:13:13.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:13:13.602 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:13:13.703 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:13:13.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:13:13.741 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:13:13.741 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:13:13.842 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:13:13.876 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:13:13.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:13:13.878 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:13:13.914 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: 55 16 96345 96329 109.433 81.6875 0.00983244 0.00913377 2024-09-10T19:13:13.979 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:13:14.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:13:14.017 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:13:14.017 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:13:14.118 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:13:14.154 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:13:14.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:13:14.155 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:13:14.256 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:13:14.286 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:13:14.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:13:14.287 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:13:14.388 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:13:14.416 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:13:14.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:13:14.417 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:13:14.517 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:13:14.549 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:13:14.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:13:14.550 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:13:14.651 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:13:14.680 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:13:14.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:13:14.681 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:13:14.782 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:13:14.812 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:13:14.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:13:14.813 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:13:14.914 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:13:14.915 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: 56 16 97693 97677 108.983 84.25 0.0230862 0.00917142 2024-09-10T19:13:14.931 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:13:14 smithi183 ceph-mon[41562]: pgmap v110: 148 pgs: 33 active+undersized, 26 active+undersized+degraded, 89 active+clean; 5.5 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1023 B/s rd, 88 MiB/s wr, 1.40k op/s; 33709/272121 objects degraded (12.388%) 2024-09-10T19:13:14.931 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:13:14 smithi183 ceph-mon[41562]: from='mgr.14556 ' entity='mgr.y' 2024-09-10T19:13:14.931 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:13:14 smithi183 ceph-mon[41562]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:13:14.932 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:13:14 smithi183 ceph-mon[29736]: pgmap v110: 148 pgs: 33 active+undersized, 26 active+undersized+degraded, 89 active+clean; 5.5 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1023 B/s rd, 88 MiB/s wr, 1.40k op/s; 33709/272121 objects degraded (12.388%) 2024-09-10T19:13:14.932 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:13:14 smithi183 ceph-mon[29736]: from='mgr.14556 ' entity='mgr.y' 2024-09-10T19:13:14.932 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:13:14 smithi183 ceph-mon[29736]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:13:14.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:13:14.952 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:13:14.952 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:13:14.965 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:13:14 smithi191 ceph-mon[32884]: pgmap v110: 148 pgs: 33 active+undersized, 26 active+undersized+degraded, 89 active+clean; 5.5 GiB data, 18 GiB used, 697 GiB / 715 GiB avail; 1023 B/s rd, 88 MiB/s wr, 1.40k op/s; 33709/272121 objects degraded (12.388%) 2024-09-10T19:13:14.965 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:13:14 smithi191 ceph-mon[32884]: from='mgr.14556 ' entity='mgr.y' 2024-09-10T19:13:14.965 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:13:14 smithi191 ceph-mon[32884]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:13:15.054 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:13:15.087 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:13:15.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:13:15.088 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:13:15.189 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:13:15.223 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:13:15.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:13:15.224 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:13:15.325 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:13:15.357 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:13:15.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:13:15.358 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:13:15.459 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:13:15.487 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:13:15.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:13:15.488 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:13:15.589 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:13:15.630 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:13:15.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:13:15.632 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:13:15.654 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:13:15 smithi183 ceph-mon[29736]: 6.8 scrub starts 2024-09-10T19:13:15.654 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:13:15 smithi183 ceph-mon[29736]: 6.8 scrub ok 2024-09-10T19:13:15.654 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:13:15 smithi183 ceph-mon[29736]: 2.18 scrub starts 2024-09-10T19:13:15.655 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:13:15 smithi183 ceph-mon[29736]: 2.18 scrub ok 2024-09-10T19:13:15.655 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:13:15 smithi183 ceph-mon[29736]: Health check update: Degraded data redundancy: 33709/272121 objects degraded (12.388%), 26 pgs degraded (PG_DEGRADED) 2024-09-10T19:13:15.732 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:13:15.775 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:13:15.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:13:15.776 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:13:15.877 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:13:15.901 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:13:15.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:13:15.902 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:13:15.914 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: 57 16 98985 98969 108.488 80.75 0.0131418 0.00921155 2024-09-10T19:13:15.931 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:13:15 smithi183 ceph-mon[41562]: 6.8 scrub starts 2024-09-10T19:13:15.931 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:13:15 smithi183 ceph-mon[41562]: 6.8 scrub ok 2024-09-10T19:13:15.931 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:13:15 smithi183 ceph-mon[41562]: 2.18 scrub starts 2024-09-10T19:13:15.931 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:13:15 smithi183 ceph-mon[41562]: 2.18 scrub ok 2024-09-10T19:13:15.931 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:13:15 smithi183 ceph-mon[41562]: Health check update: Degraded data redundancy: 33709/272121 objects degraded (12.388%), 26 pgs degraded (PG_DEGRADED) 2024-09-10T19:13:15.965 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:13:15 smithi191 ceph-mon[32884]: 6.8 scrub starts 2024-09-10T19:13:15.965 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:13:15 smithi191 ceph-mon[32884]: 6.8 scrub ok 2024-09-10T19:13:15.966 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:13:15 smithi191 ceph-mon[32884]: 2.18 scrub starts 2024-09-10T19:13:15.966 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:13:15 smithi191 ceph-mon[32884]: 2.18 scrub ok 2024-09-10T19:13:15.966 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:13:15 smithi191 ceph-mon[32884]: Health check update: Degraded data redundancy: 33709/272121 objects degraded (12.388%), 26 pgs degraded (PG_DEGRADED) 2024-09-10T19:13:16.003 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:13:16.053 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:13:16.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:13:16.054 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:13:16.155 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:13:16.200 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:13:16.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:13:16.201 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:13:16.285 INFO:teuthology.orchestra.run.smithi183.stdout:osd.0: {} 2024-09-10T19:13:16.285 INFO:teuthology.orchestra.run.smithi183.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-10T19:13:16.286 INFO:teuthology.orchestra.run.smithi183.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-09-10T19:13:16.303 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:13:16.326 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:13:16.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:13:16.327 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:13:16.329 INFO:teuthology.orchestra.run.smithi183.stdout:osd.2: {} 2024-09-10T19:13:16.330 INFO:teuthology.orchestra.run.smithi183.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-10T19:13:16.364 INFO:teuthology.orchestra.run.smithi183.stdout:osd.3: {} 2024-09-10T19:13:16.364 INFO:teuthology.orchestra.run.smithi183.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-10T19:13:16.398 INFO:teuthology.orchestra.run.smithi183.stdout:osd.4: {} 2024-09-10T19:13:16.398 INFO:teuthology.orchestra.run.smithi183.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-10T19:13:16.428 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:13:16.430 INFO:teuthology.orchestra.run.smithi183.stdout:osd.5: {} 2024-09-10T19:13:16.430 INFO:teuthology.orchestra.run.smithi183.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-10T19:13:16.454 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:13:16.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:13:16.455 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:13:16.461 INFO:teuthology.orchestra.run.smithi183.stdout:osd.6: {} 2024-09-10T19:13:16.461 INFO:teuthology.orchestra.run.smithi183.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-10T19:13:16.493 INFO:teuthology.orchestra.run.smithi183.stdout:osd.7: {} 2024-09-10T19:13:16.494 INFO:teuthology.orchestra.run.smithi183.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-10T19:13:16.556 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:13:16.586 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:13:16.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:13:16.587 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:13:16.687 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:13:16.698 INFO:teuthology.orchestra.run.smithi183.stderr:noscrub is set 2024-09-10T19:13:16.713 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:13:16.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:13:16.714 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:13:16.723 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:13:16 smithi183 ceph-mon[29736]: 6.3 scrub starts 2024-09-10T19:13:16.723 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:13:16 smithi183 ceph-mon[29736]: 6.3 scrub ok 2024-09-10T19:13:16.723 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:13:16 smithi183 ceph-mon[29736]: pgmap v111: 148 pgs: 33 active+undersized, 26 active+undersized+degraded, 89 active+clean; 5.9 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 935 B/s rd, 79 MiB/s wr, 1.26k op/s; 36308/292536 objects degraded (12.411%) 2024-09-10T19:13:16.723 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:13:16 smithi183 ceph-mon[29736]: 6.13 scrub starts 2024-09-10T19:13:16.724 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:13:16 smithi183 ceph-mon[29736]: 6.13 scrub ok 2024-09-10T19:13:16.724 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:13:16 smithi183 ceph-mon[29736]: from='client.? 172.21.15.183:0/3527136504' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-10T19:13:16.724 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:13:16 smithi183 ceph-mon[29736]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-10T19:13:16.724 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:13:16 smithi183 ceph-mon[29736]: 2.1f scrub starts 2024-09-10T19:13:16.724 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:13:16 smithi183 ceph-mon[29736]: from='client.? 172.21.15.183:0/407470305' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T19:13:16.724 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:13:16 smithi183 ceph-mon[41562]: 6.3 scrub starts 2024-09-10T19:13:16.724 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:13:16 smithi183 ceph-mon[41562]: 6.3 scrub ok 2024-09-10T19:13:16.724 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:13:16 smithi183 ceph-mon[41562]: pgmap v111: 148 pgs: 33 active+undersized, 26 active+undersized+degraded, 89 active+clean; 5.9 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 935 B/s rd, 79 MiB/s wr, 1.26k op/s; 36308/292536 objects degraded (12.411%) 2024-09-10T19:13:16.724 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:13:16 smithi183 ceph-mon[41562]: 6.13 scrub starts 2024-09-10T19:13:16.725 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:13:16 smithi183 ceph-mon[41562]: 6.13 scrub ok 2024-09-10T19:13:16.725 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:13:16 smithi183 ceph-mon[41562]: from='client.? 172.21.15.183:0/3527136504' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-10T19:13:16.725 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:13:16 smithi183 ceph-mon[41562]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-10T19:13:16.725 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:13:16 smithi183 ceph-mon[41562]: 2.1f scrub starts 2024-09-10T19:13:16.725 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:13:16 smithi183 ceph-mon[41562]: from='client.? 172.21.15.183:0/407470305' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T19:13:16.815 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:13:16.845 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:13:16.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:13:16.846 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:13:16.915 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: 58 16 100063 100047 107.779 67.375 0.0140094 0.00927352 2024-09-10T19:13:16.946 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:13:16.965 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:13:16 smithi191 ceph-mon[32884]: 6.3 scrub starts 2024-09-10T19:13:16.965 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:13:16 smithi191 ceph-mon[32884]: 6.3 scrub ok 2024-09-10T19:13:16.966 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:13:16 smithi191 ceph-mon[32884]: pgmap v111: 148 pgs: 33 active+undersized, 26 active+undersized+degraded, 89 active+clean; 5.9 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 935 B/s rd, 79 MiB/s wr, 1.26k op/s; 36308/292536 objects degraded (12.411%) 2024-09-10T19:13:16.966 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:13:16 smithi191 ceph-mon[32884]: 6.13 scrub starts 2024-09-10T19:13:16.966 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:13:16 smithi191 ceph-mon[32884]: 6.13 scrub ok 2024-09-10T19:13:16.966 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:13:16 smithi191 ceph-mon[32884]: from='client.? 172.21.15.183:0/3527136504' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-10T19:13:16.966 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:13:16 smithi191 ceph-mon[32884]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-10T19:13:16.966 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:13:16 smithi191 ceph-mon[32884]: 2.1f scrub starts 2024-09-10T19:13:16.966 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:13:16 smithi191 ceph-mon[32884]: from='client.? 172.21.15.183:0/407470305' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T19:13:16.980 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:13:16.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:13:16.981 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:13:17.082 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:13:17.125 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:13:17.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:13:17.126 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:13:17.227 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:13:17.255 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:13:17.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:13:17.256 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:13:17.356 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:13:17.381 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:13:17.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:13:17.382 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:13:17.483 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:13:17.510 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:13:17.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:13:17.511 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:13:17.612 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:13:17.648 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:13:17.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:13:17.649 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:13:17.708 INFO:journalctl@ceph.mgr.x.smithi191.stdout:Sep 10 19:13:17 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-x[34967]: ::ffff:172.21.15.191 - - [10/Sep/2024:19:13:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T19:13:17.738 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:13:17.749 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:13:17.774 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-10T19:13:17.775 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-10T19:13:17.802 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:13:17.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:13:17.803 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:13:17.904 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:13:17.915 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: 59 16 101300 101284 107.263 77.3125 0.0202777 0.00931866 2024-09-10T19:13:17.931 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:13:17 smithi183 ceph-mon[29736]: 2.1f scrub ok 2024-09-10T19:13:17.931 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:13:17 smithi183 ceph-mon[29736]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T19:13:17.931 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:13:17 smithi183 ceph-mon[29736]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-10T19:13:17.931 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:13:17 smithi183 ceph-mon[29736]: osdmap e106: 8 total, 7 up, 8 in 2024-09-10T19:13:17.931 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:13:17 smithi183 ceph-mon[41562]: 2.1f scrub ok 2024-09-10T19:13:17.931 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:13:17 smithi183 ceph-mon[41562]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T19:13:17.932 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:13:17 smithi183 ceph-mon[41562]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-10T19:13:17.932 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:13:17 smithi183 ceph-mon[41562]: osdmap e106: 8 total, 7 up, 8 in 2024-09-10T19:13:17.939 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:13:17.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:13:17.940 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:13:17.965 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:13:17 smithi191 ceph-mon[32884]: 2.1f scrub ok 2024-09-10T19:13:17.965 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:13:17 smithi191 ceph-mon[32884]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T19:13:17.965 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:13:17 smithi191 ceph-mon[32884]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-10T19:13:17.965 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:13:17 smithi191 ceph-mon[32884]: osdmap e106: 8 total, 7 up, 8 in 2024-09-10T19:13:18.041 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:13:18.074 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:13:18.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:13:18.075 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:13:18.176 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:13:18.213 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:13:18.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:13:18.214 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:13:18.315 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:13:18.343 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:13:18.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:13:18.344 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:13:18.444 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:13:18.487 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:13:18.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:13:18.489 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:13:18.589 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:13:18.620 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:13:18.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:13:18.621 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:13:18.722 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:13:18.756 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:13:18.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:13:18.757 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:13:18.858 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:13:18.894 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:13:18.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:13:18.895 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:13:18.915 INFO:tasks.radosbench.radosbench.0.smithi183.stdout:2024-09-10T19:13:18.914818+0000 min lat: 0.00124697 max lat: 1.04074 avg lat: 0.00935376 2024-09-10T19:13:18.915 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-10T19:13:18.915 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: 60 16 102629 102613 106.859 83.0625 0.00977183 0.00935376 2024-09-10T19:13:18.931 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:13:18 smithi183 ceph-mon[29736]: pgmap v113: 148 pgs: 33 active+undersized, 26 active+undersized+degraded, 89 active+clean; 5.9 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1023 B/s rd, 86 MiB/s wr, 1.38k op/s; 36308/292536 objects degraded (12.411%) 2024-09-10T19:13:18.931 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:13:18 smithi183 ceph-mon[41562]: pgmap v113: 148 pgs: 33 active+undersized, 26 active+undersized+degraded, 89 active+clean; 5.9 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1023 B/s rd, 86 MiB/s wr, 1.38k op/s; 36308/292536 objects degraded (12.411%) 2024-09-10T19:13:18.965 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:13:18 smithi191 ceph-mon[32884]: pgmap v113: 148 pgs: 33 active+undersized, 26 active+undersized+degraded, 89 active+clean; 5.9 GiB data, 19 GiB used, 696 GiB / 715 GiB avail; 1023 B/s rd, 86 MiB/s wr, 1.38k op/s; 36308/292536 objects degraded (12.411%) 2024-09-10T19:13:18.996 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:13:19.030 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:13:19.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:13:19.031 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:13:19.132 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:13:19.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:13:19.171 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:13:19.171 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:13:19.272 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:13:19.301 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:13:19.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:13:19.302 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:13:19.403 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:13:19.437 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:13:19.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:13:19.438 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:13:19.465 INFO:journalctl@ceph.iscsi.iscsi.a.smithi191.stdout:Sep 10 19:13:18 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-iscsi-iscsi-a[61816]: debug there is no tcmu-runner data available 2024-09-10T19:13:19.540 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:13:19.576 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:13:19.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:13:19.577 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:13:19.668 INFO:teuthology.orchestra.run.smithi183.stderr:Exporting 5.13 info 5.13( empty local-lis/les=62/63 n=0 ec=62/62 lis/c=62/62 les/c/f=63/63/0 sis=62) 2024-09-10T19:13:19.671 INFO:teuthology.orchestra.run.smithi183.stderr:Export successful 2024-09-10T19:13:19.678 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:13:19.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:13:19.714 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:13:19.715 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:13:19.775 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-10T19:13:19.815 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:13:19 smithi183 ceph-mon[29736]: from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:13:19.815 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:13:19 smithi183 ceph-mon[29736]: Health check update: Degraded data redundancy: 36308/292536 objects degraded (12.411%), 26 pgs degraded (PG_DEGRADED) 2024-09-10T19:13:19.815 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:13:19 smithi183 ceph-mon[29736]: osdmap e107: 8 total, 7 up, 8 in 2024-09-10T19:13:19.815 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:13:19.816 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:13:19 smithi183 ceph-mon[41562]: from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:13:19.816 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:13:19 smithi183 ceph-mon[41562]: Health check update: Degraded data redundancy: 36308/292536 objects degraded (12.411%), 26 pgs degraded (PG_DEGRADED) 2024-09-10T19:13:19.817 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:13:19 smithi183 ceph-mon[41562]: osdmap e107: 8 total, 7 up, 8 in 2024-09-10T19:13:19.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:13:19.859 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:13:19.859 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:13:19.915 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: 61 16 103969 103953 106.48 83.75 0.0151565 0.0093868 2024-09-10T19:13:19.960 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:13:19.985 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:13:19.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:13:19.986 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:13:20.087 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:13:20.113 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:13:20.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:13:20.115 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:13:20.215 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:13:20.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:13:19 smithi191 ceph-mon[32884]: from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:13:20.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:13:19 smithi191 ceph-mon[32884]: Health check update: Degraded data redundancy: 36308/292536 objects degraded (12.411%), 26 pgs degraded (PG_DEGRADED) 2024-09-10T19:13:20.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:13:19 smithi191 ceph-mon[32884]: osdmap e107: 8 total, 7 up, 8 in 2024-09-10T19:13:20.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:13:20.258 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:13:20.258 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:13:20.276 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-10T19:13:20.359 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:13:20.383 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:13:20.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:13:20.384 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:13:20.485 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:13:20.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:13:20.531 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:13:20.531 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:13:20.632 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:13:20.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:13:20.680 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:13:20.681 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:13:20.781 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:13:20.828 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:13:20.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:13:20.830 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:13:20.915 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: 62 16 105212 105196 106.016 77.6875 0.0144162 0.00942826 2024-09-10T19:13:20.930 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:13:20.931 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:13:20 smithi183 ceph-mon[41562]: pgmap v114: 148 pgs: 33 active+undersized, 26 active+undersized+degraded, 89 active+clean; 6.1 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 511 B/s rd, 79 MiB/s wr, 1.26k op/s; 37542/300939 objects degraded (12.475%) 2024-09-10T19:13:20.931 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:13:20 smithi183 ceph-mon[29736]: pgmap v114: 148 pgs: 33 active+undersized, 26 active+undersized+degraded, 89 active+clean; 6.1 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 511 B/s rd, 79 MiB/s wr, 1.26k op/s; 37542/300939 objects degraded (12.475%) 2024-09-10T19:13:20.968 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:13:20.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:13:20.969 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:13:21.069 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:13:21.117 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:13:21.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:13:21.119 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:13:21.157 INFO:teuthology.orchestra.run.smithi183.stdout:ERROR: (22) Invalid argument 2024-09-10T19:13:21.157 INFO:teuthology.orchestra.run.smithi183.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-10T19:13:21.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:13:20 smithi191 ceph-mon[32884]: pgmap v114: 148 pgs: 33 active+undersized, 26 active+undersized+degraded, 89 active+clean; 6.1 GiB data, 20 GiB used, 696 GiB / 715 GiB avail; 511 B/s rd, 79 MiB/s wr, 1.26k op/s; 37542/300939 objects degraded (12.475%) 2024-09-10T19:13:21.220 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:13:21.264 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:13:21.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:13:21.265 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:13:21.366 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:13:21.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:13:21.406 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:13:21.406 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:13:21.507 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:13:21.543 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:13:21.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:13:21.544 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:13:21.645 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:13:21.676 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:13:21.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:13:21.677 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:13:21.681 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Sep 10 19:13:21 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-y[29989]: ::ffff:172.21.15.191 - - [10/Sep/2024:19:13:21] "GET /metrics HTTP/1.1" 200 223789 "" "Prometheus/2.43.0" 2024-09-10T19:13:21.779 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:13:21.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:13:21.819 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:13:21.819 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:13:21.915 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: 63 16 106549 106533 105.659 83.5625 0.00538397 0.00945999 2024-09-10T19:13:21.919 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:13:21.959 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:13:21.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:13:21.963 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:13:21.963 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:13:22.002 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph daemon osd.3 dump_blocked_ops 2024-09-10T19:13:22.064 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:13:22.132 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:13:22.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:13:22.133 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:13:22.234 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:13:22.263 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:13:22.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:13:22.264 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:13:22.365 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:13:22.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:13:22.400 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:13:22.400 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:13:22.500 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:13:22.506 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell -n osd.1 --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-1 --journal-path /var/lib/ceph/osd/ceph-1/journal --force --op remove --pgid 5.13 2024-09-10T19:13:22.527 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:13:22.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:13:22.528 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:13:22.629 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:13:22.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:13:22.653 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:13:22.653 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:13:22.754 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:13:22.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:13:22.800 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:13:22.800 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:13:22.900 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:13:22.915 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: 64 16 107792 107776 105.222 77.6875 0.00620742 0.00949927 2024-09-10T19:13:22.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:13:22.945 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:13:22.946 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:13:23.046 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:13:23.081 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:13:23.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:13:23.082 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:13:23.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:13:22 smithi183 ceph-mon[29736]: pgmap v116: 148 pgs: 33 active+undersized, 26 active+undersized+degraded, 89 active+clean; 6.3 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.2 KiB/s rd, 102 MiB/s wr, 1.64k op/s; 38750/311478 objects degraded (12.441%) 2024-09-10T19:13:23.181 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:13:22 smithi183 ceph-mon[41562]: pgmap v116: 148 pgs: 33 active+undersized, 26 active+undersized+degraded, 89 active+clean; 6.3 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.2 KiB/s rd, 102 MiB/s wr, 1.64k op/s; 38750/311478 objects degraded (12.441%) 2024-09-10T19:13:23.183 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:13:23.209 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:13:23.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:13:23.211 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:13:23.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:13:22 smithi191 ceph-mon[32884]: pgmap v116: 148 pgs: 33 active+undersized, 26 active+undersized+degraded, 89 active+clean; 6.3 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 1.2 KiB/s rd, 102 MiB/s wr, 1.64k op/s; 38750/311478 objects degraded (12.441%) 2024-09-10T19:13:23.312 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:13:23.346 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:13:23.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:13:23.347 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:13:23.448 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:13:23.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:13:23.480 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:13:23.480 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:13:23.581 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:13:23.609 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:13:23.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:13:23.612 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:13:23.712 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:13:23.747 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:13:23.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:13:23.748 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:13:23.849 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:13:23.882 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:13:23.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:13:23.883 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:13:23.915 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: 65 16 109171 109155 104.929 86.1875 0.017557 0.00952583 2024-09-10T19:13:23.938 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:13:23.967 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/osd.1/config 2024-09-10T19:13:23.984 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:13:24.011 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:13:24.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:13:24.013 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:13:24.113 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:13:24.159 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:13:24.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:13:24.160 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:13:24.261 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:13:24.301 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:13:24.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:13:24.303 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:13:24.404 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:13:24.435 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:13:24.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:13:24.436 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:13:24.536 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:13:24.572 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:13:24.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:13:24.573 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:13:24.674 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:13:24.695 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:13:24.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:13:24.696 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:13:24.797 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:13:24.830 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:13:24.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:13:24.831 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:13:24.916 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: 66 16 110409 110393 104.512 77.375 0.00934987 0.00956414 2024-09-10T19:13:24.932 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:13:24.966 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:13:24.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:13:24.967 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:13:25.067 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:13:25.098 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:13:25.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:13:25.099 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:13:25.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:13:24 smithi183 ceph-mon[29736]: pgmap v117: 148 pgs: 33 active+undersized, 26 active+undersized+degraded, 89 active+clean; 6.3 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 639 B/s rd, 49 MiB/s wr, 789 op/s; 38750/311478 objects degraded (12.441%) 2024-09-10T19:13:25.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:13:24 smithi183 ceph-mon[29736]: Health check update: Degraded data redundancy: 38750/311478 objects degraded (12.441%), 26 pgs degraded (PG_DEGRADED) 2024-09-10T19:13:25.181 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:13:24 smithi183 ceph-mon[41562]: pgmap v117: 148 pgs: 33 active+undersized, 26 active+undersized+degraded, 89 active+clean; 6.3 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 639 B/s rd, 49 MiB/s wr, 789 op/s; 38750/311478 objects degraded (12.441%) 2024-09-10T19:13:25.181 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:13:24 smithi183 ceph-mon[41562]: Health check update: Degraded data redundancy: 38750/311478 objects degraded (12.441%), 26 pgs degraded (PG_DEGRADED) 2024-09-10T19:13:25.200 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:13:25.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:13:24 smithi191 ceph-mon[32884]: pgmap v117: 148 pgs: 33 active+undersized, 26 active+undersized+degraded, 89 active+clean; 6.3 GiB data, 20 GiB used, 695 GiB / 715 GiB avail; 639 B/s rd, 49 MiB/s wr, 789 op/s; 38750/311478 objects degraded (12.441%) 2024-09-10T19:13:25.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:13:24 smithi191 ceph-mon[32884]: Health check update: Degraded data redundancy: 38750/311478 objects degraded (12.441%), 26 pgs degraded (PG_DEGRADED) 2024-09-10T19:13:25.227 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:13:25.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:13:25.227 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:13:25.328 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:13:25.367 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:13:25.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:13:25.368 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:13:25.469 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:13:25.491 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:13:25.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:13:25.492 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:13:25.593 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:13:25.628 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:13:25.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:13:25.629 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:13:25.729 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:13:25.766 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:13:25.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:13:25.767 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:13:25.868 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:13:25.896 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:13:25.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:13:25.897 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:13:25.916 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: 67 15 111772 111757 104.224 85.25 0.00528903 0.00959005 2024-09-10T19:13:25.998 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:13:26.028 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:13:26.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:13:26.029 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:13:26.130 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:13:26.159 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:13:26.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:13:26.160 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:13:26.261 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:13:26.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:13:26.292 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:13:26.292 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:13:26.393 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:13:26.420 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:13:26.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:13:26.421 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:13:26.521 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:13:26.548 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:13:26.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:13:26.549 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:13:26.650 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:13:26.677 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:13:26.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:13:26.678 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:13:26.779 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:13:26.811 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:13:26.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:13:26.812 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:13:26.913 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:13:26.916 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: 68 16 113145 113129 103.953 85.75 0.0251589 0.00961551 2024-09-10T19:13:26.946 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:13:26.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:13:26.947 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:13:27.017 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:13:26 smithi191 ceph-mon[32884]: pgmap v118: 148 pgs: 33 active+undersized, 26 active+undersized+degraded, 89 active+clean; 6.7 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.1 KiB/s rd, 89 MiB/s wr, 1.43k op/s; 41166/330924 objects degraded (12.440%) 2024-09-10T19:13:27.048 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:13:27.090 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:13:27.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:13:27.091 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:13:27.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:13:26 smithi183 ceph-mon[29736]: pgmap v118: 148 pgs: 33 active+undersized, 26 active+undersized+degraded, 89 active+clean; 6.7 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.1 KiB/s rd, 89 MiB/s wr, 1.43k op/s; 41166/330924 objects degraded (12.440%) 2024-09-10T19:13:27.181 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:13:26 smithi183 ceph-mon[41562]: pgmap v118: 148 pgs: 33 active+undersized, 26 active+undersized+degraded, 89 active+clean; 6.7 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1.1 KiB/s rd, 89 MiB/s wr, 1.43k op/s; 41166/330924 objects degraded (12.440%) 2024-09-10T19:13:27.192 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:13:27.221 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:13:27.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:13:27.222 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:13:27.323 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:13:27.351 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:13:27.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:13:27.352 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:13:27.453 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:13:27.486 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:13:27.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:13:27.488 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:13:27.588 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:13:27.614 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:13:27.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:13:27.615 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:13:27.715 INFO:journalctl@ceph.mgr.x.smithi191.stdout:Sep 10 19:13:27 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-x[34967]: ::ffff:172.21.15.191 - - [10/Sep/2024:19:13:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T19:13:27.716 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:13:27.745 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:13:27.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:13:27.746 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:13:27.846 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:13:27.876 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:13:27.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:13:27.878 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:13:27.916 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: 69 16 114519 114503 103.691 85.875 0.00615054 0.00963955 2024-09-10T19:13:27.978 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:13:28.006 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:13:28.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:13:28.007 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:13:28.107 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:13:28.136 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:13:28.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:13:28.137 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:13:28.235 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:13:28.238 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:13:28.267 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:13:28.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:13:28.268 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:13:28.369 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:13:28.395 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:13:28.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:13:28.396 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:13:28.497 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:13:28.524 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:13:28.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:13:28.525 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:13:28.626 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:13:28.653 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:13:28.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:13:28.660 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:13:28.760 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:13:28.809 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:13:28.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:13:28.810 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:13:28.910 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:13:28.916 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: 70 16 115843 115827 103.391 82.75 0.00722537 0.0096679 2024-09-10T19:13:28.931 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:13:28 smithi183 ceph-mon[29736]: pgmap v119: 148 pgs: 33 active+undersized, 26 active+undersized+degraded, 89 active+clean; 6.7 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1023 B/s rd, 80 MiB/s wr, 1.28k op/s; 41166/330924 objects degraded (12.440%) 2024-09-10T19:13:28.931 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:13:28 smithi183 ceph-mon[29736]: from='mgr.14556 ' entity='mgr.y' 2024-09-10T19:13:28.931 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:13:28 smithi183 ceph-mon[29736]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:13:28.932 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:13:28 smithi183 ceph-mon[41562]: pgmap v119: 148 pgs: 33 active+undersized, 26 active+undersized+degraded, 89 active+clean; 6.7 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1023 B/s rd, 80 MiB/s wr, 1.28k op/s; 41166/330924 objects degraded (12.440%) 2024-09-10T19:13:28.932 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:13:28 smithi183 ceph-mon[41562]: from='mgr.14556 ' entity='mgr.y' 2024-09-10T19:13:28.932 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:13:28 smithi183 ceph-mon[41562]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:13:28.943 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:13:28.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:13:28.944 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:13:29.045 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:13:29.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:13:29.078 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:13:29.078 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:13:29.179 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:13:29.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:13:28 smithi191 ceph-mon[32884]: pgmap v119: 148 pgs: 33 active+undersized, 26 active+undersized+degraded, 89 active+clean; 6.7 GiB data, 21 GiB used, 694 GiB / 715 GiB avail; 1023 B/s rd, 80 MiB/s wr, 1.28k op/s; 41166/330924 objects degraded (12.440%) 2024-09-10T19:13:29.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:13:28 smithi191 ceph-mon[32884]: from='mgr.14556 ' entity='mgr.y' 2024-09-10T19:13:29.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:13:28 smithi191 ceph-mon[32884]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:13:29.216 INFO:journalctl@ceph.iscsi.iscsi.a.smithi191.stdout:Sep 10 19:13:28 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-iscsi-iscsi-a[61816]: debug there is no tcmu-runner data available 2024-09-10T19:13:29.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:13:29.219 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:13:29.219 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:13:29.320 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:13:29.349 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:13:29.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:13:29.350 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:13:29.451 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:13:29.477 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:13:29.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:13:29.478 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:13:29.579 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:13:29.605 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:13:29.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:13:29.606 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:13:29.707 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:13:29.742 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:13:29.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:13:29.743 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:13:29.768 INFO:teuthology.orchestra.run.smithi183.stderr:nodeep-scrub is set 2024-09-10T19:13:29.844 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:13:29.878 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:13:29.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:13:29.879 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:13:29.916 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: 71 16 117065 117049 103.011 76.375 0.0125283 0.00970335 2024-09-10T19:13:29.921 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:13:29 smithi183 ceph-mon[29736]: from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:13:29.922 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:13:29 smithi183 ceph-mon[29736]: from='client.? 172.21.15.183:0/2020680005' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-10T19:13:29.922 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:13:29 smithi183 ceph-mon[29736]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-10T19:13:29.922 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:13:29 smithi183 ceph-mon[29736]: from='client.? 172.21.15.183:0/1991559395' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T19:13:29.922 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:13:29 smithi183 ceph-mon[29736]: Health check update: Degraded data redundancy: 41166/330924 objects degraded (12.440%), 26 pgs degraded (PG_DEGRADED) 2024-09-10T19:13:29.922 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:13:29 smithi183 ceph-mon[41562]: from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:13:29.922 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:13:29 smithi183 ceph-mon[41562]: from='client.? 172.21.15.183:0/2020680005' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-10T19:13:29.922 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:13:29 smithi183 ceph-mon[41562]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-10T19:13:29.922 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:13:29 smithi183 ceph-mon[41562]: from='client.? 172.21.15.183:0/1991559395' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T19:13:29.922 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:13:29 smithi183 ceph-mon[41562]: Health check update: Degraded data redundancy: 41166/330924 objects degraded (12.440%), 26 pgs degraded (PG_DEGRADED) 2024-09-10T19:13:29.923 INFO:teuthology.orchestra.run.smithi183.stdout:osd.0: {} 2024-09-10T19:13:29.923 INFO:teuthology.orchestra.run.smithi183.stdout:osd.2: {} 2024-09-10T19:13:29.923 INFO:teuthology.orchestra.run.smithi183.stdout:osd.3: {} 2024-09-10T19:13:29.923 INFO:teuthology.orchestra.run.smithi183.stdout:osd.4: {} 2024-09-10T19:13:29.923 INFO:teuthology.orchestra.run.smithi183.stdout:osd.5: {} 2024-09-10T19:13:29.923 INFO:teuthology.orchestra.run.smithi183.stdout:osd.6: {} 2024-09-10T19:13:29.923 INFO:teuthology.orchestra.run.smithi183.stdout:osd.7: {} 2024-09-10T19:13:29.923 INFO:teuthology.orchestra.run.smithi183.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-10T19:13:29.923 INFO:teuthology.orchestra.run.smithi183.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-09-10T19:13:29.924 INFO:teuthology.orchestra.run.smithi183.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-10T19:13:29.924 INFO:teuthology.orchestra.run.smithi183.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-10T19:13:29.924 INFO:teuthology.orchestra.run.smithi183.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-10T19:13:29.924 INFO:teuthology.orchestra.run.smithi183.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-10T19:13:29.924 INFO:teuthology.orchestra.run.smithi183.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-10T19:13:29.924 INFO:teuthology.orchestra.run.smithi183.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-10T19:13:29.979 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:13:30.012 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:13:30.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:13:30.013 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:13:30.114 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:13:30.153 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:13:30.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:13:30.154 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:13:30.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:13:29 smithi191 ceph-mon[32884]: from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:13:30.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:13:29 smithi191 ceph-mon[32884]: from='client.? 172.21.15.183:0/2020680005' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-10T19:13:30.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:13:29 smithi191 ceph-mon[32884]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-10T19:13:30.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:13:29 smithi191 ceph-mon[32884]: from='client.? 172.21.15.183:0/1991559395' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T19:13:30.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:13:29 smithi191 ceph-mon[32884]: Health check update: Degraded data redundancy: 41166/330924 objects degraded (12.440%), 26 pgs degraded (PG_DEGRADED) 2024-09-10T19:13:30.255 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:13:30.293 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:13:30.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:13:30.294 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:13:30.395 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:13:30.430 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:13:30.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:13:30.431 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:13:30.531 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:13:30.562 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:13:30.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:13:30.563 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:13:30.663 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:13:30.689 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:13:30.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:13:30.690 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:13:30.791 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:13:30.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:13:30.837 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:13:30.837 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:13:30.916 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: 72 16 118457 118441 102.788 87 0.0101128 0.00972474 2024-09-10T19:13:30.930 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:13:30 smithi183 ceph-mon[29736]: pgmap v120: 148 pgs: 33 active+undersized, 26 active+undersized+degraded, 89 active+clean; 6.9 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 1023 B/s rd, 80 MiB/s wr, 1.29k op/s; 42394/339576 objects degraded (12.484%) 2024-09-10T19:13:30.930 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:13:30 smithi183 ceph-mon[29736]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T19:13:30.930 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:13:30 smithi183 ceph-mon[29736]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-10T19:13:30.930 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:13:30 smithi183 ceph-mon[29736]: osdmap e108: 8 total, 7 up, 8 in 2024-09-10T19:13:30.931 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:13:30 smithi183 ceph-mon[41562]: pgmap v120: 148 pgs: 33 active+undersized, 26 active+undersized+degraded, 89 active+clean; 6.9 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 1023 B/s rd, 80 MiB/s wr, 1.29k op/s; 42394/339576 objects degraded (12.484%) 2024-09-10T19:13:30.931 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:13:30 smithi183 ceph-mon[41562]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T19:13:30.931 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:13:30 smithi183 ceph-mon[41562]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-10T19:13:30.931 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:13:30 smithi183 ceph-mon[41562]: osdmap e108: 8 total, 7 up, 8 in 2024-09-10T19:13:30.938 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:13:30.972 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:13:30.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:13:30.973 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:13:31.074 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:13:31.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:13:31.102 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:13:31.102 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:13:31.203 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:13:31.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:13:30 smithi191 ceph-mon[32884]: pgmap v120: 148 pgs: 33 active+undersized, 26 active+undersized+degraded, 89 active+clean; 6.9 GiB data, 22 GiB used, 694 GiB / 715 GiB avail; 1023 B/s rd, 80 MiB/s wr, 1.29k op/s; 42394/339576 objects degraded (12.484%) 2024-09-10T19:13:31.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:13:30 smithi191 ceph-mon[32884]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T19:13:31.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:13:30 smithi191 ceph-mon[32884]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-10T19:13:31.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:13:30 smithi191 ceph-mon[32884]: osdmap e108: 8 total, 7 up, 8 in 2024-09-10T19:13:31.232 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:13:31.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:13:31.233 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:13:31.334 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:13:31.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:13:31.368 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:13:31.368 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:13:31.469 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:13:31.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:13:31.491 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:13:31.492 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:13:31.592 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:13:31.622 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:13:31.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:13:31.623 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:13:31.681 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Sep 10 19:13:31 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-y[29989]: ::ffff:172.21.15.191 - - [10/Sep/2024:19:13:31] "GET /metrics HTTP/1.1" 200 223813 "" "Prometheus/2.43.0" 2024-09-10T19:13:31.724 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:13:31.747 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:13:31.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:13:31.748 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:13:31.849 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:13:31.879 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:13:31.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:13:31.880 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:13:31.916 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: 73 16 119693 119677 102.438 77.25 0.00389751 0.00975551 2024-09-10T19:13:31.981 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:13:32.015 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:13:32.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:13:32.016 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:13:32.117 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:13:32.151 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:13:32.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:13:32.152 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:13:32.253 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:13:32.282 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:13:32.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:13:32.283 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:13:32.384 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:13:32.413 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:13:32.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:13:32.414 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:13:32.514 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:13:32.543 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:13:32.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:13:32.544 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:13:32.645 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:13:32.677 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:13:32.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:13:32.678 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:13:32.779 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:13:32.806 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:13:32.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:13:32.807 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:13:32.908 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:13:32.916 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: 74 16 121060 121044 102.209 85.4375 0.00583108 0.00977951 2024-09-10T19:13:32.938 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:13:32.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:13:32.939 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:13:33.036 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:13:32 smithi183 ceph-mon[29736]: pgmap v122: 148 pgs: 33 active+undersized, 26 active+undersized+degraded, 89 active+clean; 7.1 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1023 B/s rd, 82 MiB/s wr, 1.31k op/s; 43696/350943 objects degraded (12.451%) 2024-09-10T19:13:33.036 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:13:32 smithi183 ceph-mon[29736]: purged_snaps scrub starts 2024-09-10T19:13:33.036 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:13:32 smithi183 ceph-mon[29736]: purged_snaps scrub ok 2024-09-10T19:13:33.036 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:13:32 smithi183 ceph-mon[29736]: purged_snaps scrub starts 2024-09-10T19:13:33.036 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:13:32 smithi183 ceph-mon[29736]: purged_snaps scrub ok 2024-09-10T19:13:33.037 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:13:32 smithi183 ceph-mon[41562]: pgmap v122: 148 pgs: 33 active+undersized, 26 active+undersized+degraded, 89 active+clean; 7.1 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1023 B/s rd, 82 MiB/s wr, 1.31k op/s; 43696/350943 objects degraded (12.451%) 2024-09-10T19:13:33.037 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:13:32 smithi183 ceph-mon[41562]: purged_snaps scrub starts 2024-09-10T19:13:33.037 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:13:32 smithi183 ceph-mon[41562]: purged_snaps scrub ok 2024-09-10T19:13:33.037 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:13:32 smithi183 ceph-mon[41562]: purged_snaps scrub starts 2024-09-10T19:13:33.037 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:13:32 smithi183 ceph-mon[41562]: purged_snaps scrub ok 2024-09-10T19:13:33.040 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:13:33.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:13:33.086 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:13:33.086 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:13:33.086 INFO:teuthology.orchestra.run.smithi183.stdout:{ 2024-09-10T19:13:33.087 INFO:teuthology.orchestra.run.smithi183.stdout: "ops": [], 2024-09-10T19:13:33.087 INFO:teuthology.orchestra.run.smithi183.stdout: "complaint_time": 30, 2024-09-10T19:13:33.087 INFO:teuthology.orchestra.run.smithi183.stdout: "num_blocked_ops": 0 2024-09-10T19:13:33.087 INFO:teuthology.orchestra.run.smithi183.stdout:} 2024-09-10T19:13:33.187 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:13:33.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:13:32 smithi191 ceph-mon[32884]: pgmap v122: 148 pgs: 33 active+undersized, 26 active+undersized+degraded, 89 active+clean; 7.1 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 1023 B/s rd, 82 MiB/s wr, 1.31k op/s; 43696/350943 objects degraded (12.451%) 2024-09-10T19:13:33.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:13:32 smithi191 ceph-mon[32884]: purged_snaps scrub starts 2024-09-10T19:13:33.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:13:32 smithi191 ceph-mon[32884]: purged_snaps scrub ok 2024-09-10T19:13:33.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:13:32 smithi191 ceph-mon[32884]: purged_snaps scrub starts 2024-09-10T19:13:33.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:13:32 smithi191 ceph-mon[32884]: purged_snaps scrub ok 2024-09-10T19:13:33.217 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:13:33.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:13:33.220 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:13:33.278 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph osd unset noscrub 2024-09-10T19:13:33.321 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:13:33.348 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:13:33.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:13:33.349 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:13:33.450 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:13:33.477 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:13:33.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:13:33.478 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:13:33.578 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:13:33.609 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:13:33.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:13:33.610 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:13:33.711 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:13:33.724 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-10T19:13:33.725 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-10T19:13:33.737 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:13:33.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:13:33.738 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:13:33.839 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:13:33.879 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:13:33.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:13:33.880 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:13:33.916 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: 75 16 122362 122346 101.931 81.375 0.00404715 0.00980598 2024-09-10T19:13:33.931 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:13:33 smithi183 ceph-mon[29736]: osdmap e109: 8 total, 7 up, 8 in 2024-09-10T19:13:33.931 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:13:33 smithi183 ceph-mon[41562]: osdmap e109: 8 total, 7 up, 8 in 2024-09-10T19:13:33.982 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:13:34.008 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:13:34.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:13:34.009 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:13:34.110 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:13:34.147 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:13:34.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:13:34.149 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:13:34.195 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph daemon osd.3 dump_historic_ops 2024-09-10T19:13:34.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:13:33 smithi191 ceph-mon[32884]: osdmap e109: 8 total, 7 up, 8 in 2024-09-10T19:13:34.249 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:13:34.275 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:13:34.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:13:34.276 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:13:34.377 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:13:34.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:13:34.425 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:13:34.425 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:13:34.516 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:13:34.526 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:13:34.552 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:13:34.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:13:34.553 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:13:34.654 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:13:34.665 INFO:teuthology.orchestra.run.smithi183.stdout: marking collection for removal 2024-09-10T19:13:34.668 INFO:teuthology.orchestra.run.smithi183.stdout:setting '_remove' omap key 2024-09-10T19:13:34.668 INFO:teuthology.orchestra.run.smithi183.stdout:finish_remove_pgs 5.13_head removing 5.13 2024-09-10T19:13:34.672 INFO:teuthology.orchestra.run.smithi183.stdout:Remove successful 2024-09-10T19:13:34.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:13:34.700 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:13:34.701 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:13:34.801 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:13:34.853 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:13:34.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:13:34.855 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:13:34.916 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: 76 16 123616 123600 101.621 78.375 0.00746216 0.00983583 2024-09-10T19:13:34.956 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:13:34.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:13:34.994 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:13:34.994 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:13:35.049 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:13:34 smithi183 ceph-mon[29736]: pgmap v124: 148 pgs: 33 active+undersized, 26 active+undersized+degraded, 89 active+clean; 7.1 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 639 B/s rd, 52 MiB/s wr, 834 op/s; 43696/350943 objects degraded (12.451%) 2024-09-10T19:13:35.049 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:13:34 smithi183 ceph-mon[29736]: Health check update: Degraded data redundancy: 43696/350943 objects degraded (12.451%), 26 pgs degraded (PG_DEGRADED) 2024-09-10T19:13:35.049 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:13:34 smithi183 ceph-mon[41562]: pgmap v124: 148 pgs: 33 active+undersized, 26 active+undersized+degraded, 89 active+clean; 7.1 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 639 B/s rd, 52 MiB/s wr, 834 op/s; 43696/350943 objects degraded (12.451%) 2024-09-10T19:13:35.050 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:13:34 smithi183 ceph-mon[41562]: Health check update: Degraded data redundancy: 43696/350943 objects degraded (12.451%), 26 pgs degraded (PG_DEGRADED) 2024-09-10T19:13:35.095 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:13:35.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:13:35.132 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:13:35.132 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:13:35.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:13:34 smithi191 ceph-mon[32884]: pgmap v124: 148 pgs: 33 active+undersized, 26 active+undersized+degraded, 89 active+clean; 7.1 GiB data, 22 GiB used, 693 GiB / 715 GiB avail; 639 B/s rd, 52 MiB/s wr, 834 op/s; 43696/350943 objects degraded (12.451%) 2024-09-10T19:13:35.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:13:34 smithi191 ceph-mon[32884]: Health check update: Degraded data redundancy: 43696/350943 objects degraded (12.451%), 26 pgs degraded (PG_DEGRADED) 2024-09-10T19:13:35.234 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:13:35.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:13:35.270 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:13:35.270 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:13:35.371 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:13:35.403 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:13:35.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:13:35.404 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:13:35.505 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:13:35.538 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:13:35.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:13:35.539 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:13:35.640 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:13:35.674 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:13:35.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:13:35.675 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:13:35.725 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-10T19:13:35.776 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:13:35.827 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:13:35.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:13:35.828 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:13:35.917 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: 77 16 124968 124952 101.398 84.5 0.0205828 0.00985785 2024-09-10T19:13:35.929 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:13:35.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:13:35.966 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:13:35.967 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:13:36.039 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:13:35 smithi183 ceph-mon[41562]: purged_snaps scrub starts 2024-09-10T19:13:36.039 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:13:35 smithi183 ceph-mon[41562]: purged_snaps scrub ok 2024-09-10T19:13:36.039 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:13:35 smithi183 ceph-mon[29736]: purged_snaps scrub starts 2024-09-10T19:13:36.039 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:13:35 smithi183 ceph-mon[29736]: purged_snaps scrub ok 2024-09-10T19:13:36.056 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:13:36.067 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:13:36.092 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell -n osd.1 --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph-objectstore-tool --err-to-stderr --data-path /var/lib/ceph/osd/ceph-1 --journal-path /var/lib/ceph/osd/ceph-1/journal '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --op import --file /var/log/ceph/exp.5.13.1 2024-09-10T19:13:36.098 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:13:36.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:13:36.099 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:13:36.200 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:13:36.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:13:35 smithi191 ceph-mon[32884]: purged_snaps scrub starts 2024-09-10T19:13:36.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:13:35 smithi191 ceph-mon[32884]: purged_snaps scrub ok 2024-09-10T19:13:36.256 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:13:36.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:13:36.257 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:13:36.358 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:13:36.384 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:13:36.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:13:36.385 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:13:36.440 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/osd.1/config 2024-09-10T19:13:36.486 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:13:36.524 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:13:36.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:13:36.526 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:13:36.626 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:13:36.662 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:13:36.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:13:36.663 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:13:36.763 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:13:36.794 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:13:36.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:13:36.795 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:13:36.895 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:13:36.918 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: 78 16 126211 126195 101.093 77.6875 0.00593287 0.00988745 2024-09-10T19:13:36.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:13:36.937 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:13:36.938 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:13:37.038 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:13:37.070 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:13:36 smithi183 ceph-mon[29736]: pgmap v125: 148 pgs: 33 active+undersized, 26 active+undersized+degraded, 89 active+clean; 7.5 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 1.2 KiB/s rd, 103 MiB/s wr, 1.65k op/s; 46143/370497 objects degraded (12.454%) 2024-09-10T19:13:37.070 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:13:36 smithi183 ceph-mon[29736]: osdmap e110: 8 total, 7 up, 8 in 2024-09-10T19:13:37.071 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:13:36 smithi183 ceph-mon[41562]: pgmap v125: 148 pgs: 33 active+undersized, 26 active+undersized+degraded, 89 active+clean; 7.5 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 1.2 KiB/s rd, 103 MiB/s wr, 1.65k op/s; 46143/370497 objects degraded (12.454%) 2024-09-10T19:13:37.071 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:13:36 smithi183 ceph-mon[41562]: osdmap e110: 8 total, 7 up, 8 in 2024-09-10T19:13:37.076 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:13:37.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:13:37.077 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:13:37.178 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:13:37.206 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:13:37.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:13:37.207 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:13:37.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:13:36 smithi191 ceph-mon[32884]: pgmap v125: 148 pgs: 33 active+undersized, 26 active+undersized+degraded, 89 active+clean; 7.5 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 1.2 KiB/s rd, 103 MiB/s wr, 1.65k op/s; 46143/370497 objects degraded (12.454%) 2024-09-10T19:13:37.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:13:36 smithi191 ceph-mon[32884]: osdmap e110: 8 total, 7 up, 8 in 2024-09-10T19:13:37.308 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:13:37.334 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:13:37.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:13:37.335 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:13:37.435 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:13:37.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:13:37.471 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:13:37.472 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:13:37.574 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:13:37.603 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:13:37.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:13:37.605 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:13:37.666 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:13:37.706 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:13:37.715 INFO:journalctl@ceph.mgr.x.smithi191.stdout:Sep 10 19:13:37 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-x[34967]: ::ffff:172.21.15.191 - - [10/Sep/2024:19:13:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T19:13:37.737 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:13:37.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:13:37.738 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:13:37.839 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:13:37.869 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:13:37.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:13:37.870 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:13:37.918 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: 79 16 127554 127538 100.875 83.9375 0.0193189 0.00990852 2024-09-10T19:13:37.971 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:13:38.004 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:13:38.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:13:38.005 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:13:38.106 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:13:38.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:13:38.148 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:13:38.148 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:13:38.249 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:13:38.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:13:38.294 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:13:38.294 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:13:38.395 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:13:38.437 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:13:38.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:13:38.438 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:13:38.539 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:13:38.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:13:38.569 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:13:38.569 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:13:38.670 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:13:38.697 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:13:38.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:13:38.698 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:13:38.799 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:13:38.837 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:13:38.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:13:38.838 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:13:38.918 INFO:tasks.radosbench.radosbench.0.smithi183.stdout:2024-09-10T19:13:38.918112+0000 min lat: 0.00124697 max lat: 1.04074 avg lat: 0.00992791 2024-09-10T19:13:38.918 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-10T19:13:38.918 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: 80 16 128923 128907 100.684 85.5625 0.0123162 0.00992791 2024-09-10T19:13:38.940 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:13:38.967 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:13:38.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:13:38.968 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:13:39.069 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:13:39.096 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:13:39.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:13:39.097 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:13:39.103 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:13:38 smithi183 ceph-mon[41562]: pgmap v127: 148 pgs: 33 active+undersized, 26 active+undersized+degraded, 89 active+clean; 7.5 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 649 B/s rd, 52 MiB/s wr, 826 op/s; 46143/370497 objects degraded (12.454%) 2024-09-10T19:13:39.103 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:13:38 smithi183 ceph-mon[29736]: pgmap v127: 148 pgs: 33 active+undersized, 26 active+undersized+degraded, 89 active+clean; 7.5 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 649 B/s rd, 52 MiB/s wr, 826 op/s; 46143/370497 objects degraded (12.454%) 2024-09-10T19:13:39.198 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:13:39.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:13:38 smithi191 ceph-mon[32884]: pgmap v127: 148 pgs: 33 active+undersized, 26 active+undersized+degraded, 89 active+clean; 7.5 GiB data, 23 GiB used, 692 GiB / 715 GiB avail; 649 B/s rd, 52 MiB/s wr, 826 op/s; 46143/370497 objects degraded (12.454%) 2024-09-10T19:13:39.215 INFO:journalctl@ceph.iscsi.iscsi.a.smithi191.stdout:Sep 10 19:13:38 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-iscsi-iscsi-a[61816]: debug there is no tcmu-runner data available 2024-09-10T19:13:39.237 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:13:39.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:13:39.239 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:13:39.339 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:13:39.364 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:13:39.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:13:39.365 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:13:39.465 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:13:39.501 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:13:39.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:13:39.502 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:13:39.602 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:13:39.631 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:13:39.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:13:39.632 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:13:39.700 INFO:teuthology.orchestra.run.smithi183.stderr:noscrub is unset 2024-09-10T19:13:39.733 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:13:39.758 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:13:39.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:13:39.759 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:13:39.860 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:13:39.888 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:13:39.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:13:39.889 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:13:39.918 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: 81 15 130133 130118 100.375 75.6875 0.0136848 0.00995831 2024-09-10T19:13:39.990 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:13:40.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:13:40.027 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:13:40.027 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:13:40.128 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:13:40.162 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:13:40.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:13:40.163 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:13:40.181 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:13:39 smithi183 ceph-mon[41562]: from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:13:40.181 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:13:39 smithi183 ceph-mon[41562]: from='client.? 172.21.15.183:0/2923166480' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-10T19:13:40.181 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:13:39 smithi183 ceph-mon[41562]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-10T19:13:40.181 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:13:39 smithi183 ceph-mon[41562]: Health check update: Degraded data redundancy: 46143/370497 objects degraded (12.454%), 26 pgs degraded (PG_DEGRADED) 2024-09-10T19:13:40.181 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:13:39 smithi183 ceph-mon[41562]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T19:13:40.181 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:13:39 smithi183 ceph-mon[41562]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-10T19:13:40.181 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:13:39 smithi183 ceph-mon[41562]: osdmap e111: 8 total, 7 up, 8 in 2024-09-10T19:13:40.184 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:13:39 smithi183 ceph-mon[29736]: from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:13:40.184 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:13:39 smithi183 ceph-mon[29736]: from='client.? 172.21.15.183:0/2923166480' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-10T19:13:40.184 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:13:39 smithi183 ceph-mon[29736]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-10T19:13:40.184 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:13:39 smithi183 ceph-mon[29736]: Health check update: Degraded data redundancy: 46143/370497 objects degraded (12.454%), 26 pgs degraded (PG_DEGRADED) 2024-09-10T19:13:40.184 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:13:39 smithi183 ceph-mon[29736]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T19:13:40.184 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:13:39 smithi183 ceph-mon[29736]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-10T19:13:40.184 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:13:39 smithi183 ceph-mon[29736]: osdmap e111: 8 total, 7 up, 8 in 2024-09-10T19:13:40.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:13:39 smithi191 ceph-mon[32884]: from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:13:40.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:13:39 smithi191 ceph-mon[32884]: from='client.? 172.21.15.183:0/2923166480' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-10T19:13:40.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:13:39 smithi191 ceph-mon[32884]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-10T19:13:40.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:13:39 smithi191 ceph-mon[32884]: Health check update: Degraded data redundancy: 46143/370497 objects degraded (12.454%), 26 pgs degraded (PG_DEGRADED) 2024-09-10T19:13:40.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:13:39 smithi191 ceph-mon[32884]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T19:13:40.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:13:39 smithi191 ceph-mon[32884]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-10T19:13:40.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:13:39 smithi191 ceph-mon[32884]: osdmap e111: 8 total, 7 up, 8 in 2024-09-10T19:13:40.264 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:13:40.292 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:13:40.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:13:40.293 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:13:40.394 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:13:40.430 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:13:40.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:13:40.431 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:13:40.532 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:13:40.561 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:13:40.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:13:40.562 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:13:40.662 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:13:40.695 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:13:40.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:13:40.696 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:13:40.797 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:13:40.838 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:13:40.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:13:40.839 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:13:40.918 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: 82 16 131502 131486 100.194 85.5 0.0110501 0.00997654 2024-09-10T19:13:40.940 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:13:40.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:13:40.977 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:13:40.977 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:13:41.078 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:13:41.112 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:13:41.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:13:41.113 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:13:41.173 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:13:40 smithi183 ceph-mon[29736]: pgmap v128: 148 pgs: 33 active+undersized, 26 active+undersized+degraded, 89 active+clean; 7.7 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 639 B/s rd, 73 MiB/s wr, 1.17k op/s; 47352/378993 objects degraded (12.494%) 2024-09-10T19:13:41.173 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:13:40 smithi183 ceph-mon[29736]: 7.a scrub starts 2024-09-10T19:13:41.173 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:13:40 smithi183 ceph-mon[29736]: 6.0 scrub starts 2024-09-10T19:13:41.174 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:13:40 smithi183 ceph-mon[41562]: pgmap v128: 148 pgs: 33 active+undersized, 26 active+undersized+degraded, 89 active+clean; 7.7 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 639 B/s rd, 73 MiB/s wr, 1.17k op/s; 47352/378993 objects degraded (12.494%) 2024-09-10T19:13:41.174 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:13:40 smithi183 ceph-mon[41562]: 7.a scrub starts 2024-09-10T19:13:41.174 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:13:40 smithi183 ceph-mon[41562]: 6.0 scrub starts 2024-09-10T19:13:41.213 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:13:41.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:13:40 smithi191 ceph-mon[32884]: pgmap v128: 148 pgs: 33 active+undersized, 26 active+undersized+degraded, 89 active+clean; 7.7 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 639 B/s rd, 73 MiB/s wr, 1.17k op/s; 47352/378993 objects degraded (12.494%) 2024-09-10T19:13:41.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:13:40 smithi191 ceph-mon[32884]: 7.a scrub starts 2024-09-10T19:13:41.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:13:40 smithi191 ceph-mon[32884]: 6.0 scrub starts 2024-09-10T19:13:41.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:13:41.266 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:13:41.266 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:13:41.367 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:13:41.397 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:13:41.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:13:41.398 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:13:41.431 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Sep 10 19:13:41 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-y[29989]: ::ffff:172.21.15.191 - - [10/Sep/2024:19:13:41] "GET /metrics HTTP/1.1" 200 223813 "" "Prometheus/2.43.0" 2024-09-10T19:13:41.498 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:13:41.525 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:13:41.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:13:41.526 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:13:41.627 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:13:41.668 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:13:41.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:13:41.669 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:13:41.770 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:13:41.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:13:41.798 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:13:41.798 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:13:41.899 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:13:41.919 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: 83 15 132710 132695 99.897 75.5625 0.0245981 0.0100059 2024-09-10T19:13:41.921 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:13:41 smithi183 ceph-mon[29736]: 6.0 scrub ok 2024-09-10T19:13:41.921 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:13:41 smithi183 ceph-mon[29736]: 5.e scrub starts 2024-09-10T19:13:41.921 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:13:41 smithi183 ceph-mon[29736]: 5.e scrub ok 2024-09-10T19:13:41.922 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:13:41 smithi183 ceph-mon[29736]: purged_snaps scrub starts 2024-09-10T19:13:41.922 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:13:41 smithi183 ceph-mon[29736]: purged_snaps scrub ok 2024-09-10T19:13:41.922 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:13:41 smithi183 ceph-mon[29736]: osdmap e112: 8 total, 7 up, 8 in 2024-09-10T19:13:41.922 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:13:41 smithi183 ceph-mon[29736]: 7.3 scrub starts 2024-09-10T19:13:41.922 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:13:41 smithi183 ceph-mon[41562]: 6.0 scrub ok 2024-09-10T19:13:41.922 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:13:41 smithi183 ceph-mon[41562]: 5.e scrub starts 2024-09-10T19:13:41.922 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:13:41 smithi183 ceph-mon[41562]: 5.e scrub ok 2024-09-10T19:13:41.922 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:13:41 smithi183 ceph-mon[41562]: purged_snaps scrub starts 2024-09-10T19:13:41.922 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:13:41 smithi183 ceph-mon[41562]: purged_snaps scrub ok 2024-09-10T19:13:41.922 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:13:41 smithi183 ceph-mon[41562]: osdmap e112: 8 total, 7 up, 8 in 2024-09-10T19:13:41.922 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:13:41 smithi183 ceph-mon[41562]: 7.3 scrub starts 2024-09-10T19:13:41.925 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:13:41.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:13:41.926 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:13:42.027 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:13:42.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:13:42.072 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:13:42.072 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:13:42.173 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:13:42.200 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:13:42.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:13:42.201 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:13:42.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:13:41 smithi191 ceph-mon[32884]: 6.0 scrub ok 2024-09-10T19:13:42.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:13:41 smithi191 ceph-mon[32884]: 5.e scrub starts 2024-09-10T19:13:42.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:13:41 smithi191 ceph-mon[32884]: 5.e scrub ok 2024-09-10T19:13:42.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:13:41 smithi191 ceph-mon[32884]: purged_snaps scrub starts 2024-09-10T19:13:42.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:13:41 smithi191 ceph-mon[32884]: purged_snaps scrub ok 2024-09-10T19:13:42.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:13:41 smithi191 ceph-mon[32884]: osdmap e112: 8 total, 7 up, 8 in 2024-09-10T19:13:42.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:13:41 smithi191 ceph-mon[32884]: 7.3 scrub starts 2024-09-10T19:13:42.302 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:13:42.340 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:13:42.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:13:42.341 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:13:42.442 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:13:42.476 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:13:42.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:13:42.477 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:13:42.577 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:13:42.616 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:13:42.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:13:42.617 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:13:42.702 INFO:teuthology.orchestra.run.smithi183.stdout:get_pg_num_history pg_num_history pg_num_history(e98 pg_nums {1={20=1},2={58=32},3={58=3},4={60=32},5={62=32},6={64=32},7={86=16}} deleted_pools ) 2024-09-10T19:13:42.702 INFO:teuthology.orchestra.run.smithi183.stdout:Importing pgid 5.13 2024-09-10T19:13:42.702 INFO:teuthology.orchestra.run.smithi183.stdout:write_pg epoch 84 info 5.13( empty local-lis/les=62/63 n=0 ec=62/62 lis/c=62/62 les/c/f=63/63/0 sis=62) 2024-09-10T19:13:42.702 INFO:teuthology.orchestra.run.smithi183.stdout:Import successful 2024-09-10T19:13:42.719 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:13:42.761 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:13:42.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:13:42.762 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:13:42.863 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:13:42.901 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:13:42.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:13:42.902 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:13:42.918 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: 84 16 133853 133837 99.5574 71.375 0.00799128 0.0100398 2024-09-10T19:13:43.003 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:13:43.046 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:13:43.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:13:43.047 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:13:43.069 INFO:teuthology.orchestra.run.smithi183.stdout:{ 2024-09-10T19:13:43.069 INFO:teuthology.orchestra.run.smithi183.stdout: "size": 20, 2024-09-10T19:13:43.069 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 600, 2024-09-10T19:13:43.070 INFO:teuthology.orchestra.run.smithi183.stdout: "ops": [ 2024-09-10T19:13:43.070 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:13:43.070 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.14781.0:16981 7.2 7:4e90d583:::benchmark_data_smithi183_84160_object16980:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e90)", 2024-09-10T19:13:43.070 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-10T19:12:24.655246+0000", 2024-09-10T19:13:43.070 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 77.641771731000006, 2024-09-10T19:13:43.070 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.107343891, 2024-09-10T19:13:43.070 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-10T19:13:43.070 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T19:13:43.070 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-10T19:13:43.070 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.14781", 2024-09-10T19:13:43.070 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.183:0/2025776420", 2024-09-10T19:13:43.070 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 16981 2024-09-10T19:13:43.070 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:13:43.071 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-10T19:13:43.071 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:13:43.071 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-10T19:13:43.071 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:24.655246+0000", 2024-09-10T19:13:43.071 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:13:43.071 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:13:43.071 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:13:43.071 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-10T19:13:43.071 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:24.655246+0000", 2024-09-10T19:13:43.071 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:13:43.071 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:13:43.071 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:13:43.071 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-10T19:13:43.071 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:24.655245+0000", 2024-09-10T19:13:43.072 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4294967295.999999 2024-09-10T19:13:43.072 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:13:43.072 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:13:43.072 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-10T19:13:43.072 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:24.655267+0000", 2024-09-10T19:13:43.072 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.1708e-05 2024-09-10T19:13:43.072 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:13:43.072 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:13:43.072 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-10T19:13:43.072 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:24.655279+0000", 2024-09-10T19:13:43.072 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.2335e-05 2024-09-10T19:13:43.072 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:13:43.072 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:13:43.073 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-10T19:13:43.073 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:24.655285+0000", 2024-09-10T19:13:43.073 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.5289999999999999e-06 2024-09-10T19:13:43.073 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:13:43.073 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:13:43.073 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-10T19:13:43.073 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:24.656828+0000", 2024-09-10T19:13:43.073 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0015431710000000001 2024-09-10T19:13:43.073 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:13:43.073 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:13:43.073 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-10T19:13:43.073 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:24.656943+0000", 2024-09-10T19:13:43.074 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000115208 2024-09-10T19:13:43.074 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:13:43.074 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:13:43.074 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 1,2", 2024-09-10T19:13:43.074 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:24.657073+0000", 2024-09-10T19:13:43.074 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00013012200000000001 2024-09-10T19:13:43.074 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:13:43.074 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:13:43.074 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:13:43.074 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:24.660016+0000", 2024-09-10T19:13:43.074 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0029422960000000001 2024-09-10T19:13:43.074 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:13:43.074 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:13:43.074 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-10T19:13:43.075 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:24.660023+0000", 2024-09-10T19:13:43.075 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.5310000000000004e-06 2024-09-10T19:13:43.075 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:13:43.075 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:13:43.075 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:13:43.075 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:24.762025+0000", 2024-09-10T19:13:43.075 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.102002123 2024-09-10T19:13:43.075 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:13:43.075 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:13:43.075 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-10T19:13:43.075 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:24.762072+0000", 2024-09-10T19:13:43.075 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.6910000000000003e-05 2024-09-10T19:13:43.075 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:13:43.076 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:13:43.076 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-10T19:13:43.076 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:24.762590+0000", 2024-09-10T19:13:43.076 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00051799300000000001 2024-09-10T19:13:43.076 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:13:43.076 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:13:43.076 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:13:43.076 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:13:43.076 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:13:43.076 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.14781.0:16982 7.a 7:5f1609a5:::benchmark_data_smithi183_84160_object16981:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e90)", 2024-09-10T19:13:43.076 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-10T19:12:24.656290+0000", 2024-09-10T19:13:43.076 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 77.640728061000004, 2024-09-10T19:13:43.077 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.106368327, 2024-09-10T19:13:43.077 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-10T19:13:43.077 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T19:13:43.077 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-10T19:13:43.077 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.14781", 2024-09-10T19:13:43.077 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.183:0/2025776420", 2024-09-10T19:13:43.077 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 16982 2024-09-10T19:13:43.077 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:13:43.077 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-10T19:13:43.078 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:13:43.078 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-10T19:13:43.078 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:24.656290+0000", 2024-09-10T19:13:43.078 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:13:43.078 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:13:43.078 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:13:43.078 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-10T19:13:43.078 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:24.656290+0000", 2024-09-10T19:13:43.079 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:13:43.079 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:13:43.079 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:13:43.079 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-10T19:13:43.079 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:24.656288+0000", 2024-09-10T19:13:43.079 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4294967295.9999986 2024-09-10T19:13:43.079 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:13:43.079 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:13:43.080 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-10T19:13:43.080 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:24.656313+0000", 2024-09-10T19:13:43.080 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.4885e-05 2024-09-10T19:13:43.080 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:13:43.080 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:13:43.080 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-10T19:13:43.081 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:24.656326+0000", 2024-09-10T19:13:43.081 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.2561999999999999e-05 2024-09-10T19:13:43.081 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:13:43.081 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:13:43.081 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-10T19:13:43.081 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:24.656332+0000", 2024-09-10T19:13:43.081 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.4579999999999998e-06 2024-09-10T19:13:43.082 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:13:43.082 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:13:43.082 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-10T19:13:43.082 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:24.657828+0000", 2024-09-10T19:13:43.082 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0014952800000000001 2024-09-10T19:13:43.082 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:13:43.083 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:13:43.083 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-10T19:13:43.083 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:24.657920+0000", 2024-09-10T19:13:43.083 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.2528000000000006e-05 2024-09-10T19:13:43.083 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:13:43.083 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:13:43.084 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 2,5", 2024-09-10T19:13:43.084 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:24.658040+0000", 2024-09-10T19:13:43.084 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00011966799999999999 2024-09-10T19:13:43.084 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:13:43.084 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:13:43.084 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:13:43.084 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:24.661404+0000", 2024-09-10T19:13:43.085 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0033642709999999998 2024-09-10T19:13:43.085 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:13:43.085 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:13:43.085 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:13:43.085 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:24.661435+0000", 2024-09-10T19:13:43.085 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.1359000000000002e-05 2024-09-10T19:13:43.085 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:13:43.085 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:13:43.085 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-10T19:13:43.086 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:24.762462+0000", 2024-09-10T19:13:43.086 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.101026276 2024-09-10T19:13:43.086 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:13:43.086 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:13:43.086 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-10T19:13:43.086 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:24.762624+0000", 2024-09-10T19:13:43.086 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00016213100000000001 2024-09-10T19:13:43.086 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:13:43.087 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:13:43.087 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-10T19:13:43.087 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:24.762658+0000", 2024-09-10T19:13:43.087 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.4538e-05 2024-09-10T19:13:43.087 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:13:43.087 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:13:43.088 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:13:43.088 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:13:43.088 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:13:43.088 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.14781.0:16987 7.a 7:592c470f:::benchmark_data_smithi183_84160_object16986:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e90)", 2024-09-10T19:13:43.088 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-10T19:12:24.656339+0000", 2024-09-10T19:13:43.088 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 77.640679074000005, 2024-09-10T19:13:43.088 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.10738152199999999, 2024-09-10T19:13:43.088 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-10T19:13:43.089 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T19:13:43.089 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-10T19:13:43.089 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.14781", 2024-09-10T19:13:43.089 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.183:0/2025776420", 2024-09-10T19:13:43.089 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 16987 2024-09-10T19:13:43.089 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:13:43.089 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-10T19:13:43.089 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:13:43.090 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-10T19:13:43.090 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:24.656339+0000", 2024-09-10T19:13:43.090 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:13:43.090 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:13:43.090 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:13:43.090 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-10T19:13:43.090 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:24.656339+0000", 2024-09-10T19:13:43.090 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:13:43.090 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:13:43.091 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:13:43.091 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-10T19:13:43.091 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:24.656339+0000", 2024-09-10T19:13:43.091 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4294967295.9999995 2024-09-10T19:13:43.091 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:13:43.091 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:13:43.091 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-10T19:13:43.091 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:24.656359+0000", 2024-09-10T19:13:43.091 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.0400000000000001e-05 2024-09-10T19:13:43.091 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:13:43.091 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:13:43.091 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-10T19:13:43.091 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:24.656366+0000", 2024-09-10T19:13:43.092 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.1790000000000002e-06 2024-09-10T19:13:43.092 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:13:43.092 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:13:43.092 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-10T19:13:43.092 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:24.656370+0000", 2024-09-10T19:13:43.092 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.0419999999999996e-06 2024-09-10T19:13:43.092 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:13:43.092 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:13:43.092 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-10T19:13:43.092 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:24.660136+0000", 2024-09-10T19:13:43.092 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0037659410000000001 2024-09-10T19:13:43.092 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:13:43.093 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:13:43.093 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-10T19:13:43.093 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:24.660252+0000", 2024-09-10T19:13:43.093 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00011594700000000001 2024-09-10T19:13:43.093 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:13:43.093 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:13:43.093 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 2,5", 2024-09-10T19:13:43.093 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:24.660381+0000", 2024-09-10T19:13:43.093 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00012898 2024-09-10T19:13:43.093 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:13:43.093 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:13:43.094 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:13:43.094 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:24.762455+0000", 2024-09-10T19:13:43.094 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.102074283 2024-09-10T19:13:43.094 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:13:43.094 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:13:43.094 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-10T19:13:43.094 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:24.762666+0000", 2024-09-10T19:13:43.094 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00021037599999999999 2024-09-10T19:13:43.094 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:13:43.094 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:13:43.094 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:13:43.094 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:24.763646+0000", 2024-09-10T19:13:43.094 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00098002799999999998 2024-09-10T19:13:43.094 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:13:43.095 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:13:43.095 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-10T19:13:43.095 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:24.763688+0000", 2024-09-10T19:13:43.095 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.265e-05 2024-09-10T19:13:43.095 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:13:43.095 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:13:43.095 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-10T19:13:43.095 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:24.763720+0000", 2024-09-10T19:13:43.095 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.2153000000000002e-05 2024-09-10T19:13:43.095 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:13:43.095 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:13:43.095 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:13:43.095 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:13:43.096 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:13:43.096 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.14781.0:16988 7.a 7:5d66cb05:::benchmark_data_smithi183_84160_object16987:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e90)", 2024-09-10T19:13:43.096 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-10T19:12:24.658224+0000", 2024-09-10T19:13:43.096 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 77.638794008000005, 2024-09-10T19:13:43.096 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.108868989, 2024-09-10T19:13:43.096 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-10T19:13:43.096 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T19:13:43.096 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-10T19:13:43.096 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.14781", 2024-09-10T19:13:43.096 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.183:0/2025776420", 2024-09-10T19:13:43.096 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 16988 2024-09-10T19:13:43.096 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:13:43.096 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-10T19:13:43.097 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:13:43.097 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-10T19:13:43.097 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:24.658224+0000", 2024-09-10T19:13:43.097 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:13:43.097 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:13:43.097 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:13:43.097 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-10T19:13:43.097 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:24.658224+0000", 2024-09-10T19:13:43.097 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:13:43.097 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:13:43.097 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:13:43.097 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-10T19:13:43.097 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:24.658223+0000", 2024-09-10T19:13:43.098 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4294967295.999999 2024-09-10T19:13:43.098 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:13:43.098 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:13:43.098 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-10T19:13:43.098 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:24.658247+0000", 2024-09-10T19:13:43.098 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.4416000000000001e-05 2024-09-10T19:13:43.098 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:13:43.098 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:13:43.098 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-10T19:13:43.098 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:24.658257+0000", 2024-09-10T19:13:43.098 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.0159e-05 2024-09-10T19:13:43.098 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:13:43.098 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:13:43.098 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-10T19:13:43.099 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:24.658263+0000", 2024-09-10T19:13:43.099 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.3990000000000003e-06 2024-09-10T19:13:43.099 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:13:43.099 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:13:43.099 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-10T19:13:43.099 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:24.661527+0000", 2024-09-10T19:13:43.099 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0032639940000000001 2024-09-10T19:13:43.099 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:13:43.099 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:13:43.099 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-10T19:13:43.099 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:24.661636+0000", 2024-09-10T19:13:43.099 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00010902 2024-09-10T19:13:43.099 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:13:43.099 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:13:43.100 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 2,5", 2024-09-10T19:13:43.100 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:24.661763+0000", 2024-09-10T19:13:43.100 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000127324 2024-09-10T19:13:43.100 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:13:43.100 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:13:43.100 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-10T19:13:43.100 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:24.765633+0000", 2024-09-10T19:13:43.100 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.103869645 2024-09-10T19:13:43.100 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:13:43.100 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:13:43.100 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:13:43.100 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:24.766912+0000", 2024-09-10T19:13:43.100 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.001279206 2024-09-10T19:13:43.101 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:13:43.101 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:13:43.101 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:13:43.101 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:24.767026+0000", 2024-09-10T19:13:43.101 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00011370399999999999 2024-09-10T19:13:43.101 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:13:43.101 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:13:43.101 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-10T19:13:43.101 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:24.767059+0000", 2024-09-10T19:13:43.101 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.3556999999999997e-05 2024-09-10T19:13:43.101 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:13:43.101 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:13:43.101 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-10T19:13:43.102 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:24.767093+0000", 2024-09-10T19:13:43.102 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.3745000000000003e-05 2024-09-10T19:13:43.102 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:13:43.102 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:13:43.102 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:13:43.102 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:13:43.102 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:13:43.102 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_repop(client.14781.0:16990 7.0 e90/86 7:029aa942:::benchmark_data_smithi183_84160_object16989:head v 90'1090, mlcod=90'1087)", 2024-09-10T19:13:43.102 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-10T19:12:24.659720+0000", 2024-09-10T19:13:43.102 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 77.637298103000006, 2024-09-10T19:13:43.102 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.10474438899999999, 2024-09-10T19:13:43.102 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-10T19:13:43.102 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T19:13:43.102 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-10T19:13:43.103 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:13:43.103 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-10T19:13:43.103 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:24.659720+0000", 2024-09-10T19:13:43.103 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:13:43.103 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:13:43.103 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:13:43.103 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-10T19:13:43.103 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:24.659720+0000", 2024-09-10T19:13:43.103 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:13:43.103 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:13:43.103 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:13:43.103 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-10T19:13:43.103 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:24.659719+0000", 2024-09-10T19:13:43.104 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4294967295.999999 2024-09-10T19:13:43.104 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:13:43.104 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:13:43.104 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-10T19:13:43.104 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:24.659757+0000", 2024-09-10T19:13:43.104 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.7710000000000003e-05 2024-09-10T19:13:43.104 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:13:43.104 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:13:43.104 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-10T19:13:43.104 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:24.659768+0000", 2024-09-10T19:13:43.104 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.1637000000000001e-05 2024-09-10T19:13:43.104 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:13:43.105 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:13:43.105 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-10T19:13:43.105 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:24.659774+0000", 2024-09-10T19:13:43.105 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.4779999999999999e-06 2024-09-10T19:13:43.105 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:13:43.105 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:13:43.105 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-10T19:13:43.105 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:24.661725+0000", 2024-09-10T19:13:43.105 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.001950661 2024-09-10T19:13:43.105 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:13:43.105 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:13:43.105 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-10T19:13:43.105 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:24.661744+0000", 2024-09-10T19:13:43.106 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.9862999999999999e-05 2024-09-10T19:13:43.106 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:13:43.106 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:13:43.106 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-10T19:13:43.106 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:24.764435+0000", 2024-09-10T19:13:43.106 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.102690464 2024-09-10T19:13:43.106 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:13:43.106 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:13:43.106 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-10T19:13:43.106 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:24.764464+0000", 2024-09-10T19:13:43.106 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.9431999999999999e-05 2024-09-10T19:13:43.106 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:13:43.106 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:13:43.106 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:13:43.107 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:13:43.107 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:13:43.107 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.14781.0:16997 7.3 7:ce91a782:::benchmark_data_smithi183_84160_object16996:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e90)", 2024-09-10T19:13:43.107 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-10T19:12:24.661854+0000", 2024-09-10T19:13:43.107 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 77.635164326999998, 2024-09-10T19:13:43.107 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.105221789, 2024-09-10T19:13:43.107 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-10T19:13:43.107 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T19:13:43.107 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-10T19:13:43.107 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.14781", 2024-09-10T19:13:43.107 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.183:0/2025776420", 2024-09-10T19:13:43.107 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 16997 2024-09-10T19:13:43.107 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:13:43.107 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-10T19:13:43.107 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:13:43.108 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-10T19:13:43.108 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:24.661854+0000", 2024-09-10T19:13:43.108 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:13:43.108 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:13:43.108 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:13:43.108 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-10T19:13:43.108 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:24.661854+0000", 2024-09-10T19:13:43.108 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:13:43.108 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:13:43.108 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:13:43.108 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-10T19:13:43.109 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:24.661852+0000", 2024-09-10T19:13:43.109 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4294967295.9999986 2024-09-10T19:13:43.109 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:13:43.109 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:13:43.109 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-10T19:13:43.109 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:24.661878+0000", 2024-09-10T19:13:43.109 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.5771000000000001e-05 2024-09-10T19:13:43.109 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:13:43.110 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:13:43.110 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-10T19:13:43.110 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:24.761301+0000", 2024-09-10T19:13:43.110 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.099422930000000007 2024-09-10T19:13:43.110 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:13:43.110 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:13:43.110 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-10T19:13:43.110 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:24.761317+0000", 2024-09-10T19:13:43.110 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.5852e-05 2024-09-10T19:13:43.110 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:13:43.110 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:13:43.110 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-10T19:13:43.110 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:24.761371+0000", 2024-09-10T19:13:43.111 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.3890000000000001e-05 2024-09-10T19:13:43.111 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:13:43.111 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:13:43.111 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-10T19:13:43.111 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:24.761514+0000", 2024-09-10T19:13:43.111 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00014285400000000001 2024-09-10T19:13:43.111 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:13:43.111 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:13:43.111 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 2,4", 2024-09-10T19:13:43.111 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:24.761664+0000", 2024-09-10T19:13:43.111 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00015074000000000001 2024-09-10T19:13:43.111 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:13:43.111 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:13:43.111 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-10T19:13:43.112 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:24.765032+0000", 2024-09-10T19:13:43.112 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0033678499999999999 2024-09-10T19:13:43.112 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:13:43.112 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:13:43.112 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:13:43.112 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:24.765694+0000", 2024-09-10T19:13:43.112 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00066183300000000002 2024-09-10T19:13:43.112 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:13:43.112 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:13:43.112 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:13:43.112 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:24.766953+0000", 2024-09-10T19:13:43.112 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0012587130000000001 2024-09-10T19:13:43.112 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:13:43.112 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:13:43.113 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-10T19:13:43.113 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:24.767044+0000", 2024-09-10T19:13:43.113 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.1170999999999998e-05 2024-09-10T19:13:43.113 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:13:43.113 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:13:43.113 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-10T19:13:43.113 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:24.767076+0000", 2024-09-10T19:13:43.113 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.1535999999999998e-05 2024-09-10T19:13:43.113 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:13:43.113 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:13:43.113 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:13:43.113 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:13:43.113 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:13:43.113 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_repop_reply(client.14781.0:16987 7.a e90/86 ondisk, result = 0)", 2024-09-10T19:13:43.114 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-10T19:12:24.661985+0000", 2024-09-10T19:13:43.114 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 77.635032960999993, 2024-09-10T19:13:43.114 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.100681309, 2024-09-10T19:13:43.114 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-10T19:13:43.114 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "started", 2024-09-10T19:13:43.114 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-10T19:13:43.114 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:13:43.114 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-10T19:13:43.114 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:24.661985+0000", 2024-09-10T19:13:43.114 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:13:43.114 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:13:43.114 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:13:43.114 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-10T19:13:43.114 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:24.661985+0000", 2024-09-10T19:13:43.115 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:13:43.115 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:13:43.115 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:13:43.115 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-10T19:13:43.115 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:24.661984+0000", 2024-09-10T19:13:43.115 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4294967295.999999 2024-09-10T19:13:43.115 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:13:43.115 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:13:43.115 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-10T19:13:43.115 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:24.661989+0000", 2024-09-10T19:13:43.115 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.3780000000000003e-06 2024-09-10T19:13:43.115 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:13:43.115 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:13:43.115 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-10T19:13:43.115 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:24.762035+0000", 2024-09-10T19:13:43.116 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.10004658700000001 2024-09-10T19:13:43.116 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:13:43.116 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:13:43.116 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-10T19:13:43.116 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:24.762041+0000", 2024-09-10T19:13:43.116 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.1600000000000003e-06 2024-09-10T19:13:43.116 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:13:43.116 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:13:43.116 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-10T19:13:43.116 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:24.762444+0000", 2024-09-10T19:13:43.116 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00040231000000000002 2024-09-10T19:13:43.116 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:13:43.116 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:13:43.116 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-10T19:13:43.116 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:24.762450+0000", 2024-09-10T19:13:43.116 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.088e-06 2024-09-10T19:13:43.117 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:13:43.117 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:13:43.117 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-10T19:13:43.117 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:24.762666+0000", 2024-09-10T19:13:43.117 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00021668499999999999 2024-09-10T19:13:43.117 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:13:43.117 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:13:43.117 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:13:43.117 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:13:43.117 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:13:43.117 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_repop(client.14781.0:17001 7.7 e90/86 7:e2b8e459:::benchmark_data_smithi183_84160_object17000:head v 90'1113, mlcod=90'1111)", 2024-09-10T19:13:43.117 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-10T19:12:24.662853+0000", 2024-09-10T19:13:43.117 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 77.634164828999999, 2024-09-10T19:13:43.118 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.102108078, 2024-09-10T19:13:43.118 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-10T19:13:43.118 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T19:13:43.118 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-10T19:13:43.118 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:13:43.118 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-10T19:13:43.118 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:24.662853+0000", 2024-09-10T19:13:43.118 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:13:43.118 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:13:43.118 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:13:43.118 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-10T19:13:43.118 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:24.662853+0000", 2024-09-10T19:13:43.118 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:13:43.118 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:13:43.119 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:13:43.119 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-10T19:13:43.119 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:24.662852+0000", 2024-09-10T19:13:43.119 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4294967295.999999 2024-09-10T19:13:43.119 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:13:43.119 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:13:43.119 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-10T19:13:43.119 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:24.662919+0000", 2024-09-10T19:13:43.119 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.6282e-05 2024-09-10T19:13:43.119 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:13:43.119 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:13:43.119 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-10T19:13:43.119 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:24.761586+0000", 2024-09-10T19:13:43.119 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.098666832999999995 2024-09-10T19:13:43.120 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:13:43.120 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:13:43.120 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-10T19:13:43.120 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:24.761592+0000", 2024-09-10T19:13:43.120 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.4110000000000001e-06 2024-09-10T19:13:43.120 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:13:43.120 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:13:43.120 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-10T19:13:43.120 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:24.761716+0000", 2024-09-10T19:13:43.120 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000123609 2024-09-10T19:13:43.120 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:13:43.120 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:13:43.120 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-10T19:13:43.120 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:24.761737+0000", 2024-09-10T19:13:43.120 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.1903e-05 2024-09-10T19:13:43.121 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:13:43.121 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:13:43.121 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-10T19:13:43.121 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:24.764879+0000", 2024-09-10T19:13:43.121 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0031413869999999998 2024-09-10T19:13:43.121 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:13:43.121 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:13:43.121 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-10T19:13:43.121 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:24.764961+0000", 2024-09-10T19:13:43.121 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.2459000000000005e-05 2024-09-10T19:13:43.121 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:13:43.121 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:13:43.121 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:13:43.122 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:13:43.122 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:13:43.122 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.14781.0:29341 7.a 7:5ef64ee5:::benchmark_data_smithi183_84160_object29340:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e91)", 2024-09-10T19:13:43.122 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-10T19:12:30.349058+0000", 2024-09-10T19:13:43.122 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 71.947960222999995, 2024-09-10T19:13:43.122 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.095992693000000004, 2024-09-10T19:13:43.122 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-10T19:13:43.122 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T19:13:43.122 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-10T19:13:43.122 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.14781", 2024-09-10T19:13:43.122 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.183:0/2025776420", 2024-09-10T19:13:43.122 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 29341 2024-09-10T19:13:43.122 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:13:43.122 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-10T19:13:43.123 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:13:43.123 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-10T19:13:43.123 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:30.349058+0000", 2024-09-10T19:13:43.123 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:13:43.123 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:13:43.123 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:13:43.123 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-10T19:13:43.123 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:30.349058+0000", 2024-09-10T19:13:43.123 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:13:43.123 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:13:43.123 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:13:43.123 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-10T19:13:43.123 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:30.349057+0000", 2024-09-10T19:13:43.123 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4294967295.9999986 2024-09-10T19:13:43.124 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:13:43.124 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:13:43.124 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-10T19:13:43.124 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:30.349080+0000", 2024-09-10T19:13:43.124 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.3816999999999999e-05 2024-09-10T19:13:43.124 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:13:43.124 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:13:43.124 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-10T19:13:43.124 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:30.349096+0000", 2024-09-10T19:13:43.124 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.5849e-05 2024-09-10T19:13:43.124 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:13:43.124 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:13:43.124 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-10T19:13:43.124 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:30.349102+0000", 2024-09-10T19:13:43.125 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.3839999999999996e-06 2024-09-10T19:13:43.125 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:13:43.125 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:13:43.125 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-10T19:13:43.125 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:30.350835+0000", 2024-09-10T19:13:43.125 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.001733109 2024-09-10T19:13:43.125 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:13:43.125 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:13:43.125 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-10T19:13:43.125 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:30.350941+0000", 2024-09-10T19:13:43.125 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00010676800000000001 2024-09-10T19:13:43.125 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:13:43.125 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:13:43.125 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 2,5", 2024-09-10T19:13:43.126 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:30.351100+0000", 2024-09-10T19:13:43.126 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00015852999999999999 2024-09-10T19:13:43.126 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:13:43.126 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:13:43.126 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-10T19:13:43.126 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:30.352741+0000", 2024-09-10T19:13:43.126 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0016408270000000001 2024-09-10T19:13:43.126 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:13:43.126 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:13:43.126 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:13:43.126 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:30.354470+0000", 2024-09-10T19:13:43.126 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0017296449999999999 2024-09-10T19:13:43.126 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:13:43.127 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:13:43.127 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:13:43.127 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:30.357672+0000", 2024-09-10T19:13:43.127 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0032019940000000001 2024-09-10T19:13:43.127 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:13:43.127 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:13:43.127 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-10T19:13:43.127 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:30.357699+0000", 2024-09-10T19:13:43.127 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.6183e-05 2024-09-10T19:13:43.127 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:13:43.127 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:13:43.127 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-10T19:13:43.127 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:30.445051+0000", 2024-09-10T19:13:43.128 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.087351873999999996 2024-09-10T19:13:43.128 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:13:43.128 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:13:43.128 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:13:43.128 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:13:43.128 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:13:43.128 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.14781.0:29345 7.a 7:53b694cb:::benchmark_data_smithi183_84160_object29344:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e91)", 2024-09-10T19:13:43.128 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-10T19:12:30.352679+0000", 2024-09-10T19:13:43.128 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 71.944339055, 2024-09-10T19:13:43.128 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.096149120000000005, 2024-09-10T19:13:43.128 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-10T19:13:43.128 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T19:13:43.129 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-10T19:13:43.129 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.14781", 2024-09-10T19:13:43.129 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.183:0/2025776420", 2024-09-10T19:13:43.129 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 29345 2024-09-10T19:13:43.129 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:13:43.129 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-10T19:13:43.129 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:13:43.129 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-10T19:13:43.129 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:30.352679+0000", 2024-09-10T19:13:43.129 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:13:43.129 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:13:43.129 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:13:43.129 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-10T19:13:43.129 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:30.352679+0000", 2024-09-10T19:13:43.130 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:13:43.130 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:13:43.130 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:13:43.130 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-10T19:13:43.130 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:30.352678+0000", 2024-09-10T19:13:43.130 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4294967295.9999986 2024-09-10T19:13:43.130 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:13:43.130 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:13:43.130 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-10T19:13:43.130 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:30.352778+0000", 2024-09-10T19:13:43.130 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000100793 2024-09-10T19:13:43.130 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:13:43.130 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:13:43.131 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-10T19:13:43.131 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:30.352788+0000", 2024-09-10T19:13:43.131 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.0020000000000001e-05 2024-09-10T19:13:43.131 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:13:43.131 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:13:43.131 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-10T19:13:43.131 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:30.352793+0000", 2024-09-10T19:13:43.131 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.69e-06 2024-09-10T19:13:43.131 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:13:43.131 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:13:43.131 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-10T19:13:43.131 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:30.352828+0000", 2024-09-10T19:13:43.131 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.5308000000000001e-05 2024-09-10T19:13:43.131 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:13:43.132 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:13:43.132 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-10T19:13:43.132 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:30.352958+0000", 2024-09-10T19:13:43.132 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000129701 2024-09-10T19:13:43.132 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:13:43.132 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:13:43.132 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 2,5", 2024-09-10T19:13:43.132 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:30.353123+0000", 2024-09-10T19:13:43.132 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00016525099999999999 2024-09-10T19:13:43.132 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:13:43.132 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:13:43.132 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-10T19:13:43.132 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:30.354713+0000", 2024-09-10T19:13:43.132 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0015895379999999999 2024-09-10T19:13:43.133 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:13:43.133 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:13:43.133 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:13:43.133 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:30.357359+0000", 2024-09-10T19:13:43.133 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0026461620000000001 2024-09-10T19:13:43.133 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:13:43.133 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:13:43.133 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:13:43.133 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:30.448635+0000", 2024-09-10T19:13:43.133 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.091275633999999994 2024-09-10T19:13:43.133 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:13:43.133 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:13:43.133 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-10T19:13:43.134 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:30.448789+0000", 2024-09-10T19:13:43.134 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000154597 2024-09-10T19:13:43.134 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:13:43.134 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:13:43.134 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-10T19:13:43.134 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:30.448828+0000", 2024-09-10T19:13:43.134 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.8844000000000003e-05 2024-09-10T19:13:43.134 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:13:43.134 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:13:43.134 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:13:43.134 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:13:43.134 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:13:43.134 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.14781.0:29351 7.a 7:5c459497:::benchmark_data_smithi183_84160_object29350:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e91)", 2024-09-10T19:13:43.135 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-10T19:12:30.353812+0000", 2024-09-10T19:13:43.135 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 71.943206235000005, 2024-09-10T19:13:43.135 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.095303251000000005, 2024-09-10T19:13:43.135 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-10T19:13:43.135 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T19:13:43.135 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-10T19:13:43.135 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.14781", 2024-09-10T19:13:43.135 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.183:0/2025776420", 2024-09-10T19:13:43.135 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 29351 2024-09-10T19:13:43.135 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:13:43.135 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-10T19:13:43.135 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:13:43.135 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-10T19:13:43.135 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:30.353812+0000", 2024-09-10T19:13:43.136 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:13:43.136 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:13:43.136 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:13:43.136 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-10T19:13:43.136 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:30.353812+0000", 2024-09-10T19:13:43.136 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:13:43.136 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:13:43.136 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:13:43.136 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-10T19:13:43.136 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:30.353810+0000", 2024-09-10T19:13:43.136 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4294967295.9999986 2024-09-10T19:13:43.136 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:13:43.136 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:13:43.137 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-10T19:13:43.137 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:30.353833+0000", 2024-09-10T19:13:43.137 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.2779999999999999e-05 2024-09-10T19:13:43.137 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:13:43.137 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:13:43.137 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-10T19:13:43.137 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:30.353845+0000", 2024-09-10T19:13:43.137 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.2024e-05 2024-09-10T19:13:43.137 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:13:43.137 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:13:43.137 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-10T19:13:43.137 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:30.353851+0000", 2024-09-10T19:13:43.137 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.4879999999999998e-06 2024-09-10T19:13:43.138 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:13:43.138 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:13:43.138 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-10T19:13:43.138 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:30.354732+0000", 2024-09-10T19:13:43.138 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00088167599999999997 2024-09-10T19:13:43.138 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:13:43.138 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:13:43.138 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-10T19:13:43.138 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:30.354826+0000", 2024-09-10T19:13:43.138 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.3289999999999996e-05 2024-09-10T19:13:43.138 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:13:43.138 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:13:43.138 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 2,5", 2024-09-10T19:13:43.138 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:30.354951+0000", 2024-09-10T19:13:43.138 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000124845 2024-09-10T19:13:43.139 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:13:43.139 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:13:43.139 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-10T19:13:43.139 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:30.445066+0000", 2024-09-10T19:13:43.139 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.090115274999999995 2024-09-10T19:13:43.139 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:13:43.139 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:13:43.139 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:13:43.139 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:30.445106+0000", 2024-09-10T19:13:43.139 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.0266999999999999e-05 2024-09-10T19:13:43.139 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:13:43.139 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:13:43.139 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:13:43.139 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:30.448867+0000", 2024-09-10T19:13:43.139 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0037610249999999999 2024-09-10T19:13:43.140 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:13:43.140 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:13:43.140 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-10T19:13:43.140 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:30.449077+0000", 2024-09-10T19:13:43.140 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00020999500000000001 2024-09-10T19:13:43.140 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:13:43.140 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:13:43.140 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-10T19:13:43.140 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:30.449115+0000", 2024-09-10T19:13:43.140 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.7920000000000003e-05 2024-09-10T19:13:43.140 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:13:43.140 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:13:43.140 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:13:43.141 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:13:43.141 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:13:43.141 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.14781.0:29352 7.a 7:589f74a5:::benchmark_data_smithi183_84160_object29351:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e91)", 2024-09-10T19:13:43.141 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-10T19:12:30.354601+0000", 2024-09-10T19:13:43.141 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 71.942417023000004, 2024-09-10T19:13:43.141 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.099474824000000003, 2024-09-10T19:13:43.141 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-10T19:13:43.141 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T19:13:43.141 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-10T19:13:43.141 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.14781", 2024-09-10T19:13:43.141 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.183:0/2025776420", 2024-09-10T19:13:43.141 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 29352 2024-09-10T19:13:43.141 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:13:43.141 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-10T19:13:43.142 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:13:43.142 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-10T19:13:43.142 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:30.354601+0000", 2024-09-10T19:13:43.142 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:13:43.142 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:13:43.142 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:13:43.142 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-10T19:13:43.142 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:30.354601+0000", 2024-09-10T19:13:43.142 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:13:43.142 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:13:43.142 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:13:43.142 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-10T19:13:43.142 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:30.354600+0000", 2024-09-10T19:13:43.142 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4294967295.999999 2024-09-10T19:13:43.142 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:13:43.143 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:13:43.143 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-10T19:13:43.143 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:30.354623+0000", 2024-09-10T19:13:43.143 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.3119999999999999e-05 2024-09-10T19:13:43.143 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:13:43.143 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:13:43.143 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-10T19:13:43.143 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:30.354633+0000", 2024-09-10T19:13:43.143 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.3039999999999994e-06 2024-09-10T19:13:43.143 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:13:43.143 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:13:43.143 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-10T19:13:43.143 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:30.354637+0000", 2024-09-10T19:13:43.144 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.9080000000000003e-06 2024-09-10T19:13:43.144 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:13:43.144 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:13:43.144 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-10T19:13:43.144 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:30.445129+0000", 2024-09-10T19:13:43.144 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.090491826999999997 2024-09-10T19:13:43.144 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:13:43.144 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:13:43.144 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-10T19:13:43.144 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:30.445249+0000", 2024-09-10T19:13:43.144 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000119257 2024-09-10T19:13:43.144 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:13:43.144 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:13:43.144 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 2,5", 2024-09-10T19:13:43.144 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:30.445391+0000", 2024-09-10T19:13:43.145 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00014216399999999999 2024-09-10T19:13:43.145 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:13:43.145 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:13:43.145 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-10T19:13:43.145 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:30.448838+0000", 2024-09-10T19:13:43.145 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0034468789999999999 2024-09-10T19:13:43.145 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:13:43.145 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:13:43.145 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:13:43.145 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:30.449155+0000", 2024-09-10T19:13:43.145 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00031763100000000002 2024-09-10T19:13:43.145 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:13:43.145 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:13:43.145 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:13:43.146 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:30.453727+0000", 2024-09-10T19:13:43.146 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0045715720000000003 2024-09-10T19:13:43.146 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:13:43.146 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:13:43.146 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-10T19:13:43.146 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:30.454033+0000", 2024-09-10T19:13:43.146 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000305996 2024-09-10T19:13:43.146 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:13:43.146 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:13:43.146 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-10T19:13:43.146 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:30.454076+0000", 2024-09-10T19:13:43.146 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.3102000000000002e-05 2024-09-10T19:13:43.146 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:13:43.146 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:13:43.146 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:13:43.147 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:13:43.147 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:13:43.147 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_repop_reply(client.14781.0:29341 7.a e91/86 ondisk, result = 0)", 2024-09-10T19:13:43.147 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-10T19:12:30.355360+0000", 2024-09-10T19:13:43.147 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 71.941657833999997, 2024-09-10T19:13:43.147 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.089706780999999999, 2024-09-10T19:13:43.147 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-10T19:13:43.147 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "started", 2024-09-10T19:13:43.147 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-10T19:13:43.147 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:13:43.147 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-10T19:13:43.147 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:30.355360+0000", 2024-09-10T19:13:43.147 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:13:43.147 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:13:43.147 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:13:43.147 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-10T19:13:43.148 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:30.355360+0000", 2024-09-10T19:13:43.148 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:13:43.148 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:13:43.148 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:13:43.148 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-10T19:13:43.148 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:30.355360+0000", 2024-09-10T19:13:43.148 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4294967296 2024-09-10T19:13:43.148 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:13:43.148 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:13:43.148 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-10T19:13:43.148 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:30.355362+0000", 2024-09-10T19:13:43.148 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.3990000000000001e-06 2024-09-10T19:13:43.148 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:13:43.148 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:13:43.149 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-10T19:13:43.149 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:30.355366+0000", 2024-09-10T19:13:43.149 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.3470000000000001e-06 2024-09-10T19:13:43.149 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:13:43.149 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:13:43.149 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-10T19:13:43.149 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:30.355368+0000", 2024-09-10T19:13:43.149 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.441e-06 2024-09-10T19:13:43.149 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:13:43.149 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:13:43.149 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-10T19:13:43.149 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:30.357666+0000", 2024-09-10T19:13:43.149 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0022978690000000001 2024-09-10T19:13:43.149 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:13:43.150 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:13:43.150 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-10T19:13:43.150 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:30.357670+0000", 2024-09-10T19:13:43.150 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.473e-06 2024-09-10T19:13:43.150 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:13:43.150 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:13:43.150 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-10T19:13:43.150 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:30.445067+0000", 2024-09-10T19:13:43.150 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.087397363000000006 2024-09-10T19:13:43.150 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:13:43.150 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:13:43.150 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:13:43.151 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:13:43.151 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:13:43.151 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.14781.0:29356 7.3 7:ccfb99b9:::benchmark_data_smithi183_84160_object29355:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e91)", 2024-09-10T19:13:43.151 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-10T19:12:30.355520+0000", 2024-09-10T19:13:43.151 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 71.941498390000007, 2024-09-10T19:13:43.151 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.089485954000000006, 2024-09-10T19:13:43.151 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-10T19:13:43.151 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T19:13:43.151 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-10T19:13:43.151 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.14781", 2024-09-10T19:13:43.151 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.183:0/2025776420", 2024-09-10T19:13:43.151 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 29356 2024-09-10T19:13:43.151 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:13:43.151 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-10T19:13:43.151 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:13:43.152 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-10T19:13:43.152 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:30.355520+0000", 2024-09-10T19:13:43.152 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:13:43.152 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:13:43.152 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:13:43.152 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-10T19:13:43.152 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:30.355520+0000", 2024-09-10T19:13:43.152 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:13:43.152 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:13:43.152 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:13:43.152 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-10T19:13:43.152 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:30.355519+0000", 2024-09-10T19:13:43.152 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4294967295.999999 2024-09-10T19:13:43.152 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:13:43.152 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:13:43.153 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-10T19:13:43.153 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:30.355541+0000", 2024-09-10T19:13:43.153 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.1993000000000001e-05 2024-09-10T19:13:43.153 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:13:43.153 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:13:43.153 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-10T19:13:43.153 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:30.355551+0000", 2024-09-10T19:13:43.153 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.0682999999999999e-05 2024-09-10T19:13:43.153 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:13:43.153 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:13:43.153 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-10T19:13:43.153 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:30.355557+0000", 2024-09-10T19:13:43.153 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.2429999999999996e-06 2024-09-10T19:13:43.154 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:13:43.154 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:13:43.154 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-10T19:13:43.154 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:30.355593+0000", 2024-09-10T19:13:43.154 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.6174000000000003e-05 2024-09-10T19:13:43.154 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:13:43.154 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:13:43.154 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-10T19:13:43.154 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:30.355702+0000", 2024-09-10T19:13:43.154 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000109339 2024-09-10T19:13:43.154 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:13:43.154 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:13:43.154 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 2,4", 2024-09-10T19:13:43.154 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:30.355825+0000", 2024-09-10T19:13:43.154 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00012280099999999999 2024-09-10T19:13:43.154 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:13:43.155 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:13:43.155 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-10T19:13:43.155 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:30.357383+0000", 2024-09-10T19:13:43.155 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0015577939999999999 2024-09-10T19:13:43.155 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:13:43.155 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:13:43.155 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:13:43.155 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:30.357451+0000", 2024-09-10T19:13:43.155 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.8332999999999998e-05 2024-09-10T19:13:43.155 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:13:43.155 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:13:43.155 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:13:43.155 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:30.444873+0000", 2024-09-10T19:13:43.155 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.087421518000000004 2024-09-10T19:13:43.155 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:13:43.156 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:13:43.156 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-10T19:13:43.156 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:30.444968+0000", 2024-09-10T19:13:43.156 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.5118999999999995e-05 2024-09-10T19:13:43.156 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:13:43.156 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:13:43.156 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-10T19:13:43.156 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:30.445006+0000", 2024-09-10T19:13:43.156 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.786e-05 2024-09-10T19:13:43.156 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:13:43.156 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:13:43.156 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:13:43.156 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:13:43.157 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:13:43.157 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.14781.0:29357 7.a 7:5e9add9d:::benchmark_data_smithi183_84160_object29356:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e91)", 2024-09-10T19:13:43.157 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-10T19:12:30.355764+0000", 2024-09-10T19:13:43.157 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 71.941254153000003, 2024-09-10T19:13:43.157 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.102148164, 2024-09-10T19:13:43.157 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-10T19:13:43.157 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T19:13:43.157 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-10T19:13:43.157 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.14781", 2024-09-10T19:13:43.157 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.183:0/2025776420", 2024-09-10T19:13:43.157 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 29357 2024-09-10T19:13:43.157 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:13:43.157 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-10T19:13:43.158 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:13:43.158 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-10T19:13:43.158 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:30.355764+0000", 2024-09-10T19:13:43.158 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:13:43.158 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:13:43.158 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:13:43.158 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-10T19:13:43.158 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:30.355764+0000", 2024-09-10T19:13:43.158 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:13:43.158 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:13:43.158 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:13:43.158 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-10T19:13:43.158 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:30.355763+0000", 2024-09-10T19:13:43.158 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4294967295.999999 2024-09-10T19:13:43.159 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:13:43.159 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:13:43.159 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-10T19:13:43.159 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:30.355783+0000", 2024-09-10T19:13:43.159 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.0105000000000001e-05 2024-09-10T19:13:43.159 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:13:43.159 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:13:43.159 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-10T19:13:43.159 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:30.355793+0000", 2024-09-10T19:13:43.159 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.0108e-05 2024-09-10T19:13:43.159 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:13:43.159 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:13:43.159 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-10T19:13:43.159 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:30.355798+0000", 2024-09-10T19:13:43.159 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.8350000000000003e-06 2024-09-10T19:13:43.160 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:13:43.160 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:13:43.160 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-10T19:13:43.160 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:30.447036+0000", 2024-09-10T19:13:43.160 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.091237910000000005 2024-09-10T19:13:43.160 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:13:43.160 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:13:43.160 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-10T19:13:43.160 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:30.447155+0000", 2024-09-10T19:13:43.160 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000119117 2024-09-10T19:13:43.160 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:13:43.160 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:13:43.160 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 2,5", 2024-09-10T19:13:43.160 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:30.447483+0000", 2024-09-10T19:13:43.161 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00032781500000000002 2024-09-10T19:13:43.161 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:13:43.161 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:13:43.161 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:13:43.161 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:30.453700+0000", 2024-09-10T19:13:43.161 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0062171819999999999 2024-09-10T19:13:43.161 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:13:43.161 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:13:43.161 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:13:43.161 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:30.454117+0000", 2024-09-10T19:13:43.161 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00041701200000000003 2024-09-10T19:13:43.161 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:13:43.161 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:13:43.161 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-10T19:13:43.162 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:30.457832+0000", 2024-09-10T19:13:43.162 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0037143829999999999 2024-09-10T19:13:43.162 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:13:43.162 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:13:43.162 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-10T19:13:43.162 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:30.457874+0000", 2024-09-10T19:13:43.162 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.2738000000000003e-05 2024-09-10T19:13:43.162 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:13:43.162 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:13:43.162 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-10T19:13:43.162 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:30.457912+0000", 2024-09-10T19:13:43.162 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.7725000000000002e-05 2024-09-10T19:13:43.162 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:13:43.162 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:13:43.162 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:13:43.162 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:13:43.163 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:13:43.163 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_repop(client.14781.0:29355 7.c e91/86 7:3ae4b2e0:::benchmark_data_smithi183_84160_object29354:head v 91'1789, mlcod=91'1787)", 2024-09-10T19:13:43.163 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-10T19:12:30.356378+0000", 2024-09-10T19:13:43.163 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 71.940639881999999, 2024-09-10T19:13:43.163 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.088384734000000006, 2024-09-10T19:13:43.163 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-10T19:13:43.163 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T19:13:43.163 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-10T19:13:43.163 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:13:43.163 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-10T19:13:43.163 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:30.356378+0000", 2024-09-10T19:13:43.163 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:13:43.163 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:13:43.163 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:13:43.163 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-10T19:13:43.164 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:30.356378+0000", 2024-09-10T19:13:43.164 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:13:43.164 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:13:43.164 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:13:43.164 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-10T19:13:43.164 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:30.356377+0000", 2024-09-10T19:13:43.164 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4294967295.999999 2024-09-10T19:13:43.164 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:13:43.164 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:13:43.164 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-10T19:13:43.164 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:30.356410+0000", 2024-09-10T19:13:43.164 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.2996000000000001e-05 2024-09-10T19:13:43.164 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:13:43.164 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:13:43.164 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-10T19:13:43.165 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:30.356420+0000", 2024-09-10T19:13:43.165 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.0006e-05 2024-09-10T19:13:43.165 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:13:43.165 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:13:43.165 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-10T19:13:43.165 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:30.356425+0000", 2024-09-10T19:13:43.165 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.04e-06 2024-09-10T19:13:43.165 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:13:43.165 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:13:43.165 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-10T19:13:43.165 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:30.356459+0000", 2024-09-10T19:13:43.165 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.3844999999999999e-05 2024-09-10T19:13:43.165 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:13:43.165 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:13:43.166 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-10T19:13:43.166 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:30.356475+0000", 2024-09-10T19:13:43.166 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.6058000000000001e-05 2024-09-10T19:13:43.166 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:13:43.166 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:13:43.166 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-10T19:13:43.166 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:30.357671+0000", 2024-09-10T19:13:43.166 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0011955729999999999 2024-09-10T19:13:43.166 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:13:43.166 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:13:43.166 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-10T19:13:43.166 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:30.444763+0000", 2024-09-10T19:13:43.166 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.087091956999999998 2024-09-10T19:13:43.166 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:13:43.166 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:13:43.166 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:13:43.167 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:13:43.167 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:13:43.167 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_repop_reply(client.14781.0:29351 7.a e91/86 ondisk, result = 0)", 2024-09-10T19:13:43.167 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-10T19:12:30.357366+0000", 2024-09-10T19:13:43.167 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 71.939652281999997, 2024-09-10T19:13:43.167 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.087742205000000004, 2024-09-10T19:13:43.167 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-10T19:13:43.167 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "started", 2024-09-10T19:13:43.167 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-10T19:13:43.167 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:13:43.167 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-10T19:13:43.167 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:30.357366+0000", 2024-09-10T19:13:43.167 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:13:43.167 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:13:43.167 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:13:43.167 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-10T19:13:43.168 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:30.357366+0000", 2024-09-10T19:13:43.168 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:13:43.168 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:13:43.168 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:13:43.168 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-10T19:13:43.168 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:30.357365+0000", 2024-09-10T19:13:43.168 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4294967295.9999995 2024-09-10T19:13:43.168 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:13:43.168 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:13:43.168 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-10T19:13:43.168 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:30.357370+0000", 2024-09-10T19:13:43.168 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.4920000000000004e-06 2024-09-10T19:13:43.169 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:13:43.169 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:13:43.169 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-10T19:13:43.169 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:30.357379+0000", 2024-09-10T19:13:43.169 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.7070000000000004e-06 2024-09-10T19:13:43.169 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:13:43.169 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:13:43.169 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-10T19:13:43.169 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:30.357384+0000", 2024-09-10T19:13:43.169 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.9049999999999996e-06 2024-09-10T19:13:43.169 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:13:43.169 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:13:43.169 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-10T19:13:43.169 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:30.445095+0000", 2024-09-10T19:13:43.170 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.087711006999999994 2024-09-10T19:13:43.170 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:13:43.170 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:13:43.170 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-10T19:13:43.170 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:30.445101+0000", 2024-09-10T19:13:43.170 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.9309999999999996e-06 2024-09-10T19:13:43.170 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:13:43.170 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:13:43.170 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-10T19:13:43.170 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:30.445108+0000", 2024-09-10T19:13:43.170 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.7719999999999997e-06 2024-09-10T19:13:43.170 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:13:43.170 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:13:43.171 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:13:43.171 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:13:43.171 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:13:43.171 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_repop_reply(client.14781.0:29345 7.a e91/86 ondisk, result = 0)", 2024-09-10T19:13:43.171 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-10T19:12:30.357710+0000", 2024-09-10T19:13:43.171 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 71.939307608999997, 2024-09-10T19:13:43.171 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.091128073000000004, 2024-09-10T19:13:43.171 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-10T19:13:43.171 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "started", 2024-09-10T19:13:43.171 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-10T19:13:43.171 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:13:43.171 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-10T19:13:43.171 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:30.357710+0000", 2024-09-10T19:13:43.171 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:13:43.171 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:13:43.171 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:13:43.172 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-10T19:13:43.172 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:30.357710+0000", 2024-09-10T19:13:43.172 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:13:43.172 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:13:43.172 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:13:43.172 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-10T19:13:43.172 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:30.357710+0000", 2024-09-10T19:13:43.172 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4294967295.9999995 2024-09-10T19:13:43.172 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:13:43.172 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:13:43.172 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-10T19:13:43.172 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:30.357713+0000", 2024-09-10T19:13:43.172 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.9869999999999999e-06 2024-09-10T19:13:43.172 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:13:43.172 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:13:43.173 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-10T19:13:43.173 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:30.445506+0000", 2024-09-10T19:13:43.173 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.087792676 2024-09-10T19:13:43.173 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:13:43.173 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:13:43.173 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-10T19:13:43.173 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:30.445511+0000", 2024-09-10T19:13:43.173 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.3970000000000002e-06 2024-09-10T19:13:43.173 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:13:43.173 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:13:43.173 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-10T19:13:43.173 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:30.448624+0000", 2024-09-10T19:13:43.174 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0031130649999999999 2024-09-10T19:13:43.174 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:13:43.174 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:13:43.174 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-10T19:13:43.174 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:30.448630+0000", 2024-09-10T19:13:43.174 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.6250000000000004e-06 2024-09-10T19:13:43.174 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:13:43.174 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:13:43.174 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-10T19:13:43.174 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:30.448839+0000", 2024-09-10T19:13:43.174 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00020881900000000001 2024-09-10T19:13:43.174 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:13:43.174 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:13:43.175 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:13:43.175 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:13:43.175 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:13:43.175 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.14781.0:29359 7.a 7:5b67e475:::benchmark_data_smithi183_84160_object29358:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e91)", 2024-09-10T19:13:43.175 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-10T19:12:30.357782+0000", 2024-09-10T19:13:43.175 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 71.939235980999996, 2024-09-10T19:13:43.175 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.100472099, 2024-09-10T19:13:43.175 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-10T19:13:43.175 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T19:13:43.175 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-10T19:13:43.175 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.14781", 2024-09-10T19:13:43.175 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.183:0/2025776420", 2024-09-10T19:13:43.175 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 29359 2024-09-10T19:13:43.175 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:13:43.175 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-10T19:13:43.176 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:13:43.176 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-10T19:13:43.176 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:30.357782+0000", 2024-09-10T19:13:43.176 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:13:43.176 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:13:43.176 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:13:43.176 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-10T19:13:43.176 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:30.357782+0000", 2024-09-10T19:13:43.176 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:13:43.176 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:13:43.176 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:13:43.176 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-10T19:13:43.176 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:30.357781+0000", 2024-09-10T19:13:43.177 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4294967295.999999 2024-09-10T19:13:43.177 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:13:43.177 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:13:43.177 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-10T19:13:43.177 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:30.357804+0000", 2024-09-10T19:13:43.177 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.2770999999999999e-05 2024-09-10T19:13:43.177 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:13:43.177 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:13:43.177 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-10T19:13:43.177 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:30.444629+0000", 2024-09-10T19:13:43.177 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.086825663999999997 2024-09-10T19:13:43.177 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:13:43.177 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:13:43.178 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-10T19:13:43.178 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:30.444657+0000", 2024-09-10T19:13:43.178 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.7268999999999999e-05 2024-09-10T19:13:43.178 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:13:43.178 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:13:43.178 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-10T19:13:43.178 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:30.449493+0000", 2024-09-10T19:13:43.178 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0048359350000000004 2024-09-10T19:13:43.178 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:13:43.178 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:13:43.178 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-10T19:13:43.178 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:30.449637+0000", 2024-09-10T19:13:43.178 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00014478599999999999 2024-09-10T19:13:43.178 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:13:43.179 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:13:43.179 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 2,5", 2024-09-10T19:13:43.179 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:30.451528+0000", 2024-09-10T19:13:43.179 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0018907520000000001 2024-09-10T19:13:43.179 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:13:43.179 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:13:43.179 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:13:43.179 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:30.454151+0000", 2024-09-10T19:13:43.179 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0026227009999999999 2024-09-10T19:13:43.179 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:13:43.179 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:13:43.179 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:13:43.179 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:30.457723+0000", 2024-09-10T19:13:43.179 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0035718849999999999 2024-09-10T19:13:43.179 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:13:43.180 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:13:43.180 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-10T19:13:43.180 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:30.457919+0000", 2024-09-10T19:13:43.180 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00019599400000000001 2024-09-10T19:13:43.180 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:13:43.180 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:13:43.180 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-10T19:13:43.180 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:30.458218+0000", 2024-09-10T19:13:43.180 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00029970499999999999 2024-09-10T19:13:43.180 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:13:43.180 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:13:43.180 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-10T19:13:43.180 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:30.458254+0000", 2024-09-10T19:13:43.180 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.5769000000000003e-05 2024-09-10T19:13:43.180 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:13:43.181 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:13:43.181 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:13:43.181 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:13:43.181 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:13:43.181 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_repop(client.14781.0:29364 7.7 e91/86 7:ed10a9ac:::benchmark_data_smithi183_84160_object29363:head v 91'1886, mlcod=91'1884)", 2024-09-10T19:13:43.181 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-10T19:12:30.358863+0000", 2024-09-10T19:13:43.181 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 71.938154908000001, 2024-09-10T19:13:43.181 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.091335237, 2024-09-10T19:13:43.181 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-10T19:13:43.181 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T19:13:43.181 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-10T19:13:43.181 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:13:43.181 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-10T19:13:43.182 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:30.358863+0000", 2024-09-10T19:13:43.182 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:13:43.182 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:13:43.182 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:13:43.182 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-10T19:13:43.182 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:30.358863+0000", 2024-09-10T19:13:43.182 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:13:43.182 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:13:43.182 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:13:43.182 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-10T19:13:43.182 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:30.358863+0000", 2024-09-10T19:13:43.182 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4294967295.9999995 2024-09-10T19:13:43.182 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:13:43.182 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:13:43.183 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-10T19:13:43.183 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:30.358907+0000", 2024-09-10T19:13:43.183 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.4094999999999997e-05 2024-09-10T19:13:43.183 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:13:43.183 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:13:43.183 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-10T19:13:43.183 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:30.445440+0000", 2024-09-10T19:13:43.183 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.086532980999999995 2024-09-10T19:13:43.183 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:13:43.183 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:13:43.183 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-10T19:13:43.183 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:30.445447+0000", 2024-09-10T19:13:43.183 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.0910000000000004e-06 2024-09-10T19:13:43.183 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:13:43.183 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:13:43.183 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-10T19:13:43.184 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:30.447371+0000", 2024-09-10T19:13:43.184 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0019239700000000001 2024-09-10T19:13:43.184 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:13:43.184 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:13:43.184 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-10T19:13:43.184 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:30.447391+0000", 2024-09-10T19:13:43.184 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.054e-05 2024-09-10T19:13:43.184 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:13:43.184 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:13:43.184 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-10T19:13:43.184 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:30.450123+0000", 2024-09-10T19:13:43.184 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0027317880000000002 2024-09-10T19:13:43.184 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:13:43.185 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:13:43.185 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-10T19:13:43.185 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:30.450198+0000", 2024-09-10T19:13:43.185 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.5358999999999995e-05 2024-09-10T19:13:43.185 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:13:43.185 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:13:43.185 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:13:43.185 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:13:43.185 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:13:43.185 INFO:teuthology.orchestra.run.smithi183.stdout:} 2024-09-10T19:13:43.185 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:13:43.186 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:13:42 smithi183 ceph-mon[29736]: 5.10 scrub starts 2024-09-10T19:13:43.186 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:13:42 smithi183 ceph-mon[29736]: 5.10 scrub ok 2024-09-10T19:13:43.186 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:13:42 smithi183 ceph-mon[29736]: pgmap v131: 148 pgs: 33 active+undersized, 26 active+undersized+degraded, 89 active+clean; 7.9 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 852 B/s rd, 68 MiB/s wr, 1.09k op/s; 48605/390054 objects degraded (12.461%) 2024-09-10T19:13:43.187 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:13:42 smithi183 ceph-mon[41562]: 5.10 scrub starts 2024-09-10T19:13:43.187 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:13:42 smithi183 ceph-mon[41562]: 5.10 scrub ok 2024-09-10T19:13:43.187 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:13:42 smithi183 ceph-mon[41562]: pgmap v131: 148 pgs: 33 active+undersized, 26 active+undersized+degraded, 89 active+clean; 7.9 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 852 B/s rd, 68 MiB/s wr, 1.09k op/s; 48605/390054 objects degraded (12.461%) 2024-09-10T19:13:43.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:13:42 smithi191 ceph-mon[32884]: 5.10 scrub starts 2024-09-10T19:13:43.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:13:42 smithi191 ceph-mon[32884]: 5.10 scrub ok 2024-09-10T19:13:43.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:13:42 smithi191 ceph-mon[32884]: pgmap v131: 148 pgs: 33 active+undersized, 26 active+undersized+degraded, 89 active+clean; 7.9 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 852 B/s rd, 68 MiB/s wr, 1.09k op/s; 48605/390054 objects degraded (12.461%) 2024-09-10T19:13:43.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:13:43.230 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:13:43.230 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:13:43.330 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:13:43.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:13:43.369 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:13:43.369 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:13:43.470 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:13:43.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:13:43.500 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:13:43.500 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:13:43.601 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:13:43.641 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:13:43.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:13:43.643 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:13:43.743 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:13:43.781 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:13:43.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:13:43.782 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:13:43.882 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:13:43.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:13:43.919 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:13:43.919 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:13:43.919 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: 85 16 135017 135001 99.242 72.75 0.00487218 0.0100719 2024-09-10T19:13:44.020 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:13:44.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:13:44.049 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:13:44.049 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:13:44.140 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:13:43 smithi183 ceph-mon[29736]: 7.1 scrub starts 2024-09-10T19:13:44.140 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:13:43 smithi183 ceph-mon[29736]: 2.1a scrub starts 2024-09-10T19:13:44.140 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:13:43 smithi183 ceph-mon[29736]: 2.1a scrub ok 2024-09-10T19:13:44.140 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:13:43 smithi183 ceph-mon[29736]: 7.6 scrub starts 2024-09-10T19:13:44.140 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:13:43 smithi183 ceph-mon[29736]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:13:44.140 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:13:43 smithi183 ceph-mon[41562]: 7.1 scrub starts 2024-09-10T19:13:44.140 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:13:43 smithi183 ceph-mon[41562]: 2.1a scrub starts 2024-09-10T19:13:44.141 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:13:43 smithi183 ceph-mon[41562]: 2.1a scrub ok 2024-09-10T19:13:44.141 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:13:43 smithi183 ceph-mon[41562]: 7.6 scrub starts 2024-09-10T19:13:44.141 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:13:43 smithi183 ceph-mon[41562]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:13:44.150 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:13:44.186 DEBUG:teuthology.orchestra.run.smithi191:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph daemon osd.4 dump_ops_in_flight 2024-09-10T19:13:44.200 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:13:44.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:13:44.201 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:13:44.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:13:43 smithi191 ceph-mon[32884]: 7.1 scrub starts 2024-09-10T19:13:44.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:13:43 smithi191 ceph-mon[32884]: 2.1a scrub starts 2024-09-10T19:13:44.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:13:43 smithi191 ceph-mon[32884]: 2.1a scrub ok 2024-09-10T19:13:44.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:13:43 smithi191 ceph-mon[32884]: 7.6 scrub starts 2024-09-10T19:13:44.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:13:43 smithi191 ceph-mon[32884]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:13:44.302 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:13:44.337 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:13:44.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:13:44.339 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:13:44.440 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:13:44.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:13:44.476 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:13:44.477 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:13:44.577 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:13:44.618 INFO:teuthology.orchestra.run.smithi191.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.b/config 2024-09-10T19:13:44.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:13:44.623 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:13:44.624 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:13:44.724 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:13:44.764 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:13:44.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:13:44.765 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:13:44.866 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:13:44.887 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:13:44.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:13:44.888 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:13:44.919 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: 86 16 136213 136197 98.9572 74.75 0.00865897 0.010101 2024-09-10T19:13:44.989 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:13:45.023 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:13:45.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:13:45.024 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:13:45.069 DEBUG:teuthology.orchestra.run.smithi183:> sudo rm -f /var/log/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/exp.5.13.1 2024-09-10T19:13:45.119 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:13:44 smithi191 ceph-mon[32884]: 5.9 scrub starts 2024-09-10T19:13:45.120 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:13:44 smithi191 ceph-mon[32884]: 5.9 scrub ok 2024-09-10T19:13:45.120 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:13:44 smithi191 ceph-mon[32884]: pgmap v132: 148 pgs: 33 active+undersized, 26 active+undersized+degraded, 89 active+clean; 7.9 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 652 B/s rd, 52 MiB/s wr, 831 op/s; 48605/390054 objects degraded (12.461%) 2024-09-10T19:13:45.120 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:13:44 smithi191 ceph-mon[32884]: 7.a scrub ok 2024-09-10T19:13:45.120 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:13:44 smithi191 ceph-mon[32884]: 7.3 scrub ok 2024-09-10T19:13:45.120 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:13:44 smithi191 ceph-mon[32884]: Health check update: Degraded data redundancy: 48605/390054 objects degraded (12.461%), 26 pgs degraded (PG_DEGRADED) 2024-09-10T19:13:45.120 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:13:44 smithi191 ceph-mon[32884]: osdmap e113: 8 total, 7 up, 8 in 2024-09-10T19:13:45.124 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:13:45.159 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:13:45.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:13:45.160 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:13:45.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:13:44 smithi183 ceph-mon[29736]: 5.9 scrub starts 2024-09-10T19:13:45.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:13:44 smithi183 ceph-mon[29736]: 5.9 scrub ok 2024-09-10T19:13:45.182 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:13:44 smithi183 ceph-mon[29736]: pgmap v132: 148 pgs: 33 active+undersized, 26 active+undersized+degraded, 89 active+clean; 7.9 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 652 B/s rd, 52 MiB/s wr, 831 op/s; 48605/390054 objects degraded (12.461%) 2024-09-10T19:13:45.182 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:13:44 smithi183 ceph-mon[29736]: 7.a scrub ok 2024-09-10T19:13:45.182 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:13:44 smithi183 ceph-mon[29736]: 7.3 scrub ok 2024-09-10T19:13:45.182 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:13:44 smithi183 ceph-mon[29736]: Health check update: Degraded data redundancy: 48605/390054 objects degraded (12.461%), 26 pgs degraded (PG_DEGRADED) 2024-09-10T19:13:45.182 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:13:44 smithi183 ceph-mon[29736]: osdmap e113: 8 total, 7 up, 8 in 2024-09-10T19:13:45.183 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:13:44 smithi183 ceph-mon[41562]: 5.9 scrub starts 2024-09-10T19:13:45.183 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:13:44 smithi183 ceph-mon[41562]: 5.9 scrub ok 2024-09-10T19:13:45.183 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:13:44 smithi183 ceph-mon[41562]: pgmap v132: 148 pgs: 33 active+undersized, 26 active+undersized+degraded, 89 active+clean; 7.9 GiB data, 24 GiB used, 691 GiB / 715 GiB avail; 652 B/s rd, 52 MiB/s wr, 831 op/s; 48605/390054 objects degraded (12.461%) 2024-09-10T19:13:45.183 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:13:44 smithi183 ceph-mon[41562]: 7.a scrub ok 2024-09-10T19:13:45.183 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:13:44 smithi183 ceph-mon[41562]: 7.3 scrub ok 2024-09-10T19:13:45.183 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:13:44 smithi183 ceph-mon[41562]: Health check update: Degraded data redundancy: 48605/390054 objects degraded (12.461%), 26 pgs degraded (PG_DEGRADED) 2024-09-10T19:13:45.184 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:13:44 smithi183 ceph-mon[41562]: osdmap e113: 8 total, 7 up, 8 in 2024-09-10T19:13:45.261 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:13:45.306 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:13:45.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:13:45.307 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:13:45.408 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:13:45.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:13:45.444 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:13:45.444 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:13:45.545 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:13:45.576 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:13:45.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:13:45.577 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:13:45.678 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:13:45.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:13:45.710 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:13:45.710 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:13:45.811 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:13:45.840 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:13:45.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:13:45.841 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:13:45.919 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: 87 16 137343 137327 98.6315 70.625 0.00692048 0.0101345 2024-09-10T19:13:45.942 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:13:45.975 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:13:45.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:13:45.977 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:13:46.077 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:13:46.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:13:46.109 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:13:46.109 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:13:46.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:13:45 smithi183 ceph-mon[29736]: 7.f scrub starts 2024-09-10T19:13:46.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:13:45 smithi183 ceph-mon[29736]: 6.5 scrub starts 2024-09-10T19:13:46.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:13:45 smithi183 ceph-mon[29736]: 5.3 scrub starts 2024-09-10T19:13:46.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:13:45 smithi183 ceph-mon[29736]: 5.3 scrub ok 2024-09-10T19:13:46.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:13:45 smithi183 ceph-mon[41562]: 7.f scrub starts 2024-09-10T19:13:46.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:13:45 smithi183 ceph-mon[41562]: 6.5 scrub starts 2024-09-10T19:13:46.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:13:45 smithi183 ceph-mon[41562]: 5.3 scrub starts 2024-09-10T19:13:46.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:13:45 smithi183 ceph-mon[41562]: 5.3 scrub ok 2024-09-10T19:13:46.208 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:13:45 smithi191 ceph-mon[32884]: 7.f scrub starts 2024-09-10T19:13:46.208 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:13:45 smithi191 ceph-mon[32884]: 6.5 scrub starts 2024-09-10T19:13:46.208 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:13:45 smithi191 ceph-mon[32884]: 5.3 scrub starts 2024-09-10T19:13:46.208 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:13:45 smithi191 ceph-mon[32884]: 5.3 scrub ok 2024-09-10T19:13:46.209 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:13:46.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:13:46.242 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:13:46.242 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:13:46.343 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:13:46.387 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:13:46.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:13:46.388 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:13:46.489 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:13:46.516 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:13:46.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:13:46.517 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:13:46.618 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:13:46.656 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:13:46.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:13:46.657 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:13:46.689 INFO:teuthology.orchestra.run.smithi191.stdout:ERROR: (22) Invalid argument 2024-09-10T19:13:46.689 INFO:teuthology.orchestra.run.smithi191.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-10T19:13:46.757 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:13:46.779 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:13:46.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:13:46.779 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:13:46.880 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:13:46.909 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:13:46.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:13:46.910 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:13:46.919 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: 88 16 138314 138298 98.2003 60.6875 0.0109723 0.0101787 2024-09-10T19:13:46.931 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:13:46 smithi183 ceph-mon[29736]: 7.1 scrub ok 2024-09-10T19:13:46.931 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:13:46 smithi183 ceph-mon[29736]: pgmap v134: 148 pgs: 2 active+clean+scrubbing, 33 active+undersized, 26 active+undersized+degraded, 87 active+clean; 8.3 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 1.7 KiB/s rd, 102 MiB/s wr, 1.64k op/s; 50960/408444 objects degraded (12.477%) 2024-09-10T19:13:46.931 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:13:46 smithi183 ceph-mon[29736]: 7.8 scrub starts 2024-09-10T19:13:46.931 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:13:46 smithi183 ceph-mon[29736]: from='client.? 172.21.15.183:0/3881885901' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T19:13:46.931 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:13:46 smithi183 ceph-mon[41562]: 7.1 scrub ok 2024-09-10T19:13:46.931 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:13:46 smithi183 ceph-mon[41562]: pgmap v134: 148 pgs: 2 active+clean+scrubbing, 33 active+undersized, 26 active+undersized+degraded, 87 active+clean; 8.3 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 1.7 KiB/s rd, 102 MiB/s wr, 1.64k op/s; 50960/408444 objects degraded (12.477%) 2024-09-10T19:13:46.931 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:13:46 smithi183 ceph-mon[41562]: 7.8 scrub starts 2024-09-10T19:13:46.931 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:13:46 smithi183 ceph-mon[41562]: from='client.? 172.21.15.183:0/3881885901' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T19:13:47.011 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:13:47.040 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:13:47.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:13:47.041 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:13:47.142 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:13:47.185 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:13:47.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:13:47.186 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:13:47.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:13:46 smithi191 ceph-mon[32884]: 7.1 scrub ok 2024-09-10T19:13:47.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:13:46 smithi191 ceph-mon[32884]: pgmap v134: 148 pgs: 2 active+clean+scrubbing, 33 active+undersized, 26 active+undersized+degraded, 87 active+clean; 8.3 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 1.7 KiB/s rd, 102 MiB/s wr, 1.64k op/s; 50960/408444 objects degraded (12.477%) 2024-09-10T19:13:47.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:13:46 smithi191 ceph-mon[32884]: 7.8 scrub starts 2024-09-10T19:13:47.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:13:46 smithi191 ceph-mon[32884]: from='client.? 172.21.15.183:0/3881885901' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T19:13:47.287 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:13:47.324 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:13:47.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:13:47.325 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:13:47.426 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:13:47.460 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:13:47.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:13:47.461 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:13:47.562 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:13:47.597 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:13:47.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:13:47.597 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:13:47.601 DEBUG:teuthology.orchestra.run.smithi191:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph daemon osd.4 dump_blocked_ops 2024-09-10T19:13:47.697 INFO:teuthology.orchestra.run.smithi183.stdout:osd.0: {} 2024-09-10T19:13:47.697 INFO:teuthology.orchestra.run.smithi183.stdout:osd.2: {} 2024-09-10T19:13:47.697 INFO:teuthology.orchestra.run.smithi183.stdout:osd.3: {} 2024-09-10T19:13:47.697 INFO:teuthology.orchestra.run.smithi183.stdout:osd.4: {} 2024-09-10T19:13:47.697 INFO:teuthology.orchestra.run.smithi183.stdout:osd.5: {} 2024-09-10T19:13:47.697 INFO:teuthology.orchestra.run.smithi183.stdout:osd.6: {} 2024-09-10T19:13:47.697 INFO:teuthology.orchestra.run.smithi183.stdout:osd.7: {} 2024-09-10T19:13:47.698 INFO:teuthology.orchestra.run.smithi183.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-10T19:13:47.698 INFO:teuthology.orchestra.run.smithi183.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-09-10T19:13:47.698 INFO:teuthology.orchestra.run.smithi183.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-10T19:13:47.698 INFO:teuthology.orchestra.run.smithi183.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-10T19:13:47.698 INFO:teuthology.orchestra.run.smithi183.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-10T19:13:47.698 INFO:teuthology.orchestra.run.smithi183.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-10T19:13:47.698 INFO:teuthology.orchestra.run.smithi183.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-10T19:13:47.698 INFO:teuthology.orchestra.run.smithi183.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-10T19:13:47.698 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:13:47.715 INFO:journalctl@ceph.mgr.x.smithi191.stdout:Sep 10 19:13:47 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-x[34967]: ::ffff:172.21.15.191 - - [10/Sep/2024:19:13:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T19:13:47.743 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:13:47.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:13:47.744 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:13:47.845 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:13:47.865 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:13:47.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:13:47.866 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:13:47.919 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: 89 16 139325 139309 97.8069 63.1875 0.121976 0.0102106 2024-09-10T19:13:47.967 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:13:47.977 INFO:teuthology.orchestra.run.smithi191.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.b/config 2024-09-10T19:13:48.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:13:48.016 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:13:48.016 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:13:48.117 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:13:48.137 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:13:48.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:13:48.139 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:13:48.172 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:13:47 smithi191 ceph-mon[32884]: 1.0 scrub starts 2024-09-10T19:13:48.172 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:13:47 smithi191 ceph-mon[32884]: 1.0 scrub ok 2024-09-10T19:13:48.172 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:13:47 smithi191 ceph-mon[32884]: 4.19 scrub starts 2024-09-10T19:13:48.172 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:13:47 smithi191 ceph-mon[32884]: 4.19 scrub ok 2024-09-10T19:13:48.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:13:47 smithi183 ceph-mon[29736]: 1.0 scrub starts 2024-09-10T19:13:48.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:13:47 smithi183 ceph-mon[29736]: 1.0 scrub ok 2024-09-10T19:13:48.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:13:47 smithi183 ceph-mon[29736]: 4.19 scrub starts 2024-09-10T19:13:48.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:13:47 smithi183 ceph-mon[29736]: 4.19 scrub ok 2024-09-10T19:13:48.181 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:13:47 smithi183 ceph-mon[41562]: 1.0 scrub starts 2024-09-10T19:13:48.181 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:13:47 smithi183 ceph-mon[41562]: 1.0 scrub ok 2024-09-10T19:13:48.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:13:47 smithi183 ceph-mon[41562]: 4.19 scrub starts 2024-09-10T19:13:48.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:13:47 smithi183 ceph-mon[41562]: 4.19 scrub ok 2024-09-10T19:13:48.239 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:13:48.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:13:48.263 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:13:48.263 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:13:48.364 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:13:48.393 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:13:48.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:13:48.395 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:13:48.495 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:13:48.527 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:13:48.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:13:48.529 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:13:48.629 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:13:48.660 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:13:48.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:13:48.661 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:13:48.762 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:13:48.791 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:13:48.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:13:48.792 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:13:48.893 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:13:48.919 INFO:tasks.radosbench.radosbench.0.smithi183.stdout:Total time run: 90.0112 2024-09-10T19:13:48.920 INFO:tasks.radosbench.radosbench.0.smithi183.stdout:Total writes made: 140378 2024-09-10T19:13:48.920 INFO:tasks.radosbench.radosbench.0.smithi183.stdout:Write size: 65536 2024-09-10T19:13:48.920 INFO:tasks.radosbench.radosbench.0.smithi183.stdout:Object size: 65536 2024-09-10T19:13:48.920 INFO:tasks.radosbench.radosbench.0.smithi183.stdout:Bandwidth (MB/sec): 97.4726 2024-09-10T19:13:48.920 INFO:tasks.radosbench.radosbench.0.smithi183.stdout:Stddev Bandwidth: 30.3651 2024-09-10T19:13:48.920 INFO:tasks.radosbench.radosbench.0.smithi183.stdout:Max bandwidth (MB/sec): 206.938 2024-09-10T19:13:48.920 INFO:tasks.radosbench.radosbench.0.smithi183.stdout:Min bandwidth (MB/sec): 8.0625 2024-09-10T19:13:48.921 INFO:tasks.radosbench.radosbench.0.smithi183.stdout:Average IOPS: 1559 2024-09-10T19:13:48.921 INFO:tasks.radosbench.radosbench.0.smithi183.stdout:Stddev IOPS: 485.842 2024-09-10T19:13:48.921 INFO:tasks.radosbench.radosbench.0.smithi183.stdout:Max IOPS: 3311 2024-09-10T19:13:48.921 INFO:tasks.radosbench.radosbench.0.smithi183.stdout:Min IOPS: 129 2024-09-10T19:13:48.921 INFO:tasks.radosbench.radosbench.0.smithi183.stdout:Average Latency(s): 0.0102554 2024-09-10T19:13:48.921 INFO:tasks.radosbench.radosbench.0.smithi183.stdout:Stddev Latency(s): 0.0128672 2024-09-10T19:13:48.921 INFO:tasks.radosbench.radosbench.0.smithi183.stdout:Max latency(s): 1.04074 2024-09-10T19:13:48.921 INFO:tasks.radosbench.radosbench.0.smithi183.stdout:Min latency(s): 0.00124697 2024-09-10T19:13:48.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:13:48.923 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:13:48.924 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:13:48.924 INFO:tasks.radosbench.radosbench.0.smithi183.stdout:Cleaning up (deleting benchmark objects) 2024-09-10T19:13:49.024 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:13:49.057 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:13:49.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:13:49.058 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:13:49.159 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:13:49.190 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:13:48 smithi183 ceph-mon[29736]: 5.4 scrub starts 2024-09-10T19:13:49.190 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:13:48 smithi183 ceph-mon[29736]: 5.4 scrub ok 2024-09-10T19:13:49.190 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:13:48 smithi183 ceph-mon[29736]: pgmap v135: 148 pgs: 2 active+clean+scrubbing, 33 active+undersized, 26 active+undersized+degraded, 87 active+clean; 8.3 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 643 B/s rd, 71 MiB/s wr, 1.13k op/s; 50960/408444 objects degraded (12.477%) 2024-09-10T19:13:49.190 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:13:48 smithi183 ceph-mon[41562]: 5.4 scrub starts 2024-09-10T19:13:49.190 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:13:48 smithi183 ceph-mon[41562]: 5.4 scrub ok 2024-09-10T19:13:49.191 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:13:48 smithi183 ceph-mon[41562]: pgmap v135: 148 pgs: 2 active+clean+scrubbing, 33 active+undersized, 26 active+undersized+degraded, 87 active+clean; 8.3 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 643 B/s rd, 71 MiB/s wr, 1.13k op/s; 50960/408444 objects degraded (12.477%) 2024-09-10T19:13:49.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:13:48 smithi191 ceph-mon[32884]: 5.4 scrub starts 2024-09-10T19:13:49.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:13:48 smithi191 ceph-mon[32884]: 5.4 scrub ok 2024-09-10T19:13:49.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:13:48 smithi191 ceph-mon[32884]: pgmap v135: 148 pgs: 2 active+clean+scrubbing, 33 active+undersized, 26 active+undersized+degraded, 87 active+clean; 8.3 GiB data, 25 GiB used, 690 GiB / 715 GiB avail; 643 B/s rd, 71 MiB/s wr, 1.13k op/s; 50960/408444 objects degraded (12.477%) 2024-09-10T19:13:49.217 INFO:journalctl@ceph.iscsi.iscsi.a.smithi191.stdout:Sep 10 19:13:49 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-iscsi-iscsi-a[61816]: debug there is no tcmu-runner data available 2024-09-10T19:13:49.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:13:49.224 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:13:49.225 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:13:49.326 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:13:49.361 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:13:49.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:13:49.362 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:13:49.463 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:13:49.494 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:13:49.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:13:49.495 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:13:49.596 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:13:49.616 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:13:49.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:13:49.617 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:13:49.718 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:13:49.751 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:13:49.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:13:49.752 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:13:49.852 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:13:49.883 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:13:49.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:13:49.884 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:13:49.911 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-10T19:13:49.911 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-10T19:13:49.985 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:13:50.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:13:50.024 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:13:50.024 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:13:50.117 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [1] live_osds: [0, 2, 3, 4, 5, 6, 7] 2024-09-10T19:13:50.118 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-10T19:13:50.118 INFO:tasks.thrashosds.thrasher:Removing osd 1, in_osds are: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-10T19:13:50.119 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph osd out 1 2024-09-10T19:13:50.124 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:13:50.150 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:13:50.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:13:50.151 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:13:50.159 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:13:49 smithi183 ceph-mon[29736]: 7.e scrub starts 2024-09-10T19:13:50.159 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:13:49 smithi183 ceph-mon[29736]: from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:13:50.160 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:13:49 smithi183 ceph-mon[29736]: purged_snaps scrub starts 2024-09-10T19:13:50.160 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:13:49 smithi183 ceph-mon[29736]: purged_snaps scrub ok 2024-09-10T19:13:50.160 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:13:49 smithi183 ceph-mon[29736]: Health check update: Degraded data redundancy: 50960/408444 objects degraded (12.477%), 26 pgs degraded, 59 pgs undersized (PG_DEGRADED) 2024-09-10T19:13:50.160 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:13:49 smithi183 ceph-mon[41562]: 7.e scrub starts 2024-09-10T19:13:50.160 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:13:49 smithi183 ceph-mon[41562]: from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:13:50.160 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:13:49 smithi183 ceph-mon[41562]: purged_snaps scrub starts 2024-09-10T19:13:50.160 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:13:49 smithi183 ceph-mon[41562]: purged_snaps scrub ok 2024-09-10T19:13:50.160 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:13:49 smithi183 ceph-mon[41562]: Health check update: Degraded data redundancy: 50960/408444 objects degraded (12.477%), 26 pgs degraded, 59 pgs undersized (PG_DEGRADED) 2024-09-10T19:13:50.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:13:49 smithi191 ceph-mon[32884]: 7.e scrub starts 2024-09-10T19:13:50.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:13:49 smithi191 ceph-mon[32884]: from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:13:50.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:13:49 smithi191 ceph-mon[32884]: purged_snaps scrub starts 2024-09-10T19:13:50.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:13:49 smithi191 ceph-mon[32884]: purged_snaps scrub ok 2024-09-10T19:13:50.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:13:49 smithi191 ceph-mon[32884]: Health check update: Degraded data redundancy: 50960/408444 objects degraded (12.477%), 26 pgs degraded, 59 pgs undersized (PG_DEGRADED) 2024-09-10T19:13:50.252 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:13:50.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:13:50.293 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:13:50.294 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:13:50.395 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:13:50.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:13:50.422 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:13:50.422 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:13:50.489 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:13:50.523 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:13:50.551 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:13:50.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:13:50.552 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:13:50.653 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:13:50.683 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:13:50.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:13:50.684 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:13:50.785 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:13:50.816 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:13:50.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:13:50.817 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:13:50.917 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:13:50.951 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:13:50.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:13:50.952 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:13:51.053 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:13:51.088 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:13:51.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:13:51.089 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:13:51.178 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:13:50 smithi191 ceph-mon[32884]: pgmap v136: 148 pgs: 1 active+clean+scrubbing, 33 active+undersized, 26 active+undersized+degraded, 88 active+clean; 8.4 GiB data, 26 GiB used, 690 GiB / 715 GiB avail; 696 B/s rd, 57 MiB/s wr, 957 op/s; 51686/413409 objects degraded (12.502%) 2024-09-10T19:13:51.178 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:13:50 smithi191 ceph-mon[32884]: 7.8 scrub ok 2024-09-10T19:13:51.179 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:13:50 smithi191 ceph-mon[32884]: osdmap e114: 8 total, 7 up, 8 in 2024-09-10T19:13:51.179 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:13:50 smithi191 ceph-mon[32884]: 7.f scrub ok 2024-09-10T19:13:51.179 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:13:50 smithi191 ceph-mon[32884]: 4.16 scrub starts 2024-09-10T19:13:51.179 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:13:50 smithi191 ceph-mon[32884]: 4.16 scrub ok 2024-09-10T19:13:51.179 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:13:50 smithi191 ceph-mon[32884]: 2.3 scrub starts 2024-09-10T19:13:51.179 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:13:50 smithi191 ceph-mon[32884]: 2.3 scrub ok 2024-09-10T19:13:51.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:13:50 smithi183 ceph-mon[29736]: pgmap v136: 148 pgs: 1 active+clean+scrubbing, 33 active+undersized, 26 active+undersized+degraded, 88 active+clean; 8.4 GiB data, 26 GiB used, 690 GiB / 715 GiB avail; 696 B/s rd, 57 MiB/s wr, 957 op/s; 51686/413409 objects degraded (12.502%) 2024-09-10T19:13:51.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:13:50 smithi183 ceph-mon[29736]: 7.8 scrub ok 2024-09-10T19:13:51.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:13:50 smithi183 ceph-mon[29736]: osdmap e114: 8 total, 7 up, 8 in 2024-09-10T19:13:51.182 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:13:50 smithi183 ceph-mon[29736]: 7.f scrub ok 2024-09-10T19:13:51.182 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:13:50 smithi183 ceph-mon[29736]: 4.16 scrub starts 2024-09-10T19:13:51.182 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:13:50 smithi183 ceph-mon[29736]: 4.16 scrub ok 2024-09-10T19:13:51.182 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:13:50 smithi183 ceph-mon[29736]: 2.3 scrub starts 2024-09-10T19:13:51.182 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:13:50 smithi183 ceph-mon[29736]: 2.3 scrub ok 2024-09-10T19:13:51.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:13:50 smithi183 ceph-mon[41562]: pgmap v136: 148 pgs: 1 active+clean+scrubbing, 33 active+undersized, 26 active+undersized+degraded, 88 active+clean; 8.4 GiB data, 26 GiB used, 690 GiB / 715 GiB avail; 696 B/s rd, 57 MiB/s wr, 957 op/s; 51686/413409 objects degraded (12.502%) 2024-09-10T19:13:51.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:13:50 smithi183 ceph-mon[41562]: 7.8 scrub ok 2024-09-10T19:13:51.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:13:50 smithi183 ceph-mon[41562]: osdmap e114: 8 total, 7 up, 8 in 2024-09-10T19:13:51.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:13:50 smithi183 ceph-mon[41562]: 7.f scrub ok 2024-09-10T19:13:51.183 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:13:50 smithi183 ceph-mon[41562]: 4.16 scrub starts 2024-09-10T19:13:51.183 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:13:50 smithi183 ceph-mon[41562]: 4.16 scrub ok 2024-09-10T19:13:51.183 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:13:50 smithi183 ceph-mon[41562]: 2.3 scrub starts 2024-09-10T19:13:51.183 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:13:50 smithi183 ceph-mon[41562]: 2.3 scrub ok 2024-09-10T19:13:51.190 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:13:51.228 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:13:51.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:13:51.230 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:13:51.330 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:13:51.353 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:13:51.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:13:51.354 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:13:51.379 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-10T19:13:51.455 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:13:51.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:13:51.483 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:13:51.484 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:13:51.584 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:13:51.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:13:51.615 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:13:51.615 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:13:51.628 INFO:teuthology.orchestra.run.smithi191.stdout:ERROR: (22) Invalid argument 2024-09-10T19:13:51.628 INFO:teuthology.orchestra.run.smithi191.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-10T19:13:51.681 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Sep 10 19:13:51 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-y[29989]: ::ffff:172.21.15.191 - - [10/Sep/2024:19:13:51] "GET /metrics HTTP/1.1" 200 223824 "" "Prometheus/2.43.0" 2024-09-10T19:13:51.716 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:13:51.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:13:51.762 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:13:51.762 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:13:51.863 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:13:51.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:13:51.908 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:13:51.908 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:13:51.912 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-10T19:13:52.009 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:13:52.036 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:13:52.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:13:52.037 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:13:52.138 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:13:52.165 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:13:52.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:13:52.166 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:13:52.267 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:13:52.301 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:13:52.320 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:13:52.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:13:52.321 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:13:52.351 DEBUG:teuthology.orchestra.run.smithi191:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph daemon osd.4 dump_historic_ops 2024-09-10T19:13:52.422 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:13:52.461 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:13:52.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:13:52.462 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:13:52.563 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:13:52.611 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:13:52.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:13:52.612 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:13:52.712 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:13:52.739 INFO:teuthology.orchestra.run.smithi191.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.b/config 2024-09-10T19:13:52.751 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:13:52.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:13:52.752 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:13:52.853 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:13:52.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:13:52.898 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:13:52.898 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:13:52.999 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:13:53.034 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:13:53.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:13:53.035 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:13:53.097 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:13:52 smithi183 ceph-mon[29736]: 7.c scrub starts 2024-09-10T19:13:53.097 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:13:52 smithi183 ceph-mon[29736]: 4.c scrub starts 2024-09-10T19:13:53.097 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:13:52 smithi183 ceph-mon[29736]: 4.c scrub ok 2024-09-10T19:13:53.097 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:13:52 smithi183 ceph-mon[29736]: pgmap v138: 148 pgs: 3 active+clean+scrubbing, 33 active+undersized, 26 active+undersized+degraded, 86 active+clean; 8.5 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 1.4 KiB/s rd, 82 MiB/s wr, 1.43k op/s; 52318/419094 objects degraded (12.484%) 2024-09-10T19:13:53.097 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:13:52 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mon-a[29713]: 2024-09-10T19:13:52.920+0000 7f9f28e29640 -1 mon.a@0(leader).osd e114 definitely_dead 0 2024-09-10T19:13:53.097 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:13:52 smithi183 ceph-mon[41562]: 7.c scrub starts 2024-09-10T19:13:53.098 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:13:52 smithi183 ceph-mon[41562]: 4.c scrub starts 2024-09-10T19:13:53.098 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:13:52 smithi183 ceph-mon[41562]: 4.c scrub ok 2024-09-10T19:13:53.098 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:13:52 smithi183 ceph-mon[41562]: pgmap v138: 148 pgs: 3 active+clean+scrubbing, 33 active+undersized, 26 active+undersized+degraded, 86 active+clean; 8.5 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 1.4 KiB/s rd, 82 MiB/s wr, 1.43k op/s; 52318/419094 objects degraded (12.484%) 2024-09-10T19:13:53.136 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:13:53.176 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:13:53.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:13:53.177 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:13:53.214 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:13:52 smithi191 ceph-mon[32884]: 7.c scrub starts 2024-09-10T19:13:53.214 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:13:52 smithi191 ceph-mon[32884]: 4.c scrub starts 2024-09-10T19:13:53.214 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:13:52 smithi191 ceph-mon[32884]: 4.c scrub ok 2024-09-10T19:13:53.214 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:13:52 smithi191 ceph-mon[32884]: pgmap v138: 148 pgs: 3 active+clean+scrubbing, 33 active+undersized, 26 active+undersized+degraded, 86 active+clean; 8.5 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 1.4 KiB/s rd, 82 MiB/s wr, 1.43k op/s; 52318/419094 objects degraded (12.484%) 2024-09-10T19:13:53.279 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:13:53.319 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:13:53.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:13:53.320 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:13:53.420 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:13:53.457 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:13:53.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:13:53.458 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:13:53.537 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:13:53.559 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:13:53.621 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:13:53.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:13:53.622 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:13:53.722 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:13:53.752 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:13:53.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:13:53.753 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:13:53.854 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:13:53.884 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:13:53.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:13:53.885 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:13:53.934 INFO:teuthology.orchestra.run.smithi183.stderr:marked out osd.1. 2024-09-10T19:13:53.965 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:13:53 smithi191 ceph-mon[32884]: 7.e scrub ok 2024-09-10T19:13:53.965 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:13:53 smithi191 ceph-mon[32884]: from='client.? 172.21.15.183:0/2636122212' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["1"]}]: dispatch 2024-09-10T19:13:53.966 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:13:53 smithi191 ceph-mon[32884]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["1"]}]: dispatch 2024-09-10T19:13:53.966 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:13:53 smithi191 ceph-mon[32884]: Client client.admin marked osd.1 out, after it was down for 67 seconds 2024-09-10T19:13:53.986 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:13:54.011 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:13:54.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:13:54.012 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:13:54.112 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:13:54.137 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:13:54.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:13:54.138 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:13:54.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:13:53 smithi183 ceph-mon[29736]: 7.e scrub ok 2024-09-10T19:13:54.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:13:53 smithi183 ceph-mon[29736]: from='client.? 172.21.15.183:0/2636122212' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["1"]}]: dispatch 2024-09-10T19:13:54.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:13:53 smithi183 ceph-mon[29736]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["1"]}]: dispatch 2024-09-10T19:13:54.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:13:53 smithi183 ceph-mon[29736]: Client client.admin marked osd.1 out, after it was down for 67 seconds 2024-09-10T19:13:54.181 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:13:53 smithi183 ceph-mon[41562]: 7.e scrub ok 2024-09-10T19:13:54.181 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:13:53 smithi183 ceph-mon[41562]: from='client.? 172.21.15.183:0/2636122212' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["1"]}]: dispatch 2024-09-10T19:13:54.181 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:13:53 smithi183 ceph-mon[41562]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["1"]}]: dispatch 2024-09-10T19:13:54.181 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:13:53 smithi183 ceph-mon[41562]: Client client.admin marked osd.1 out, after it was down for 67 seconds 2024-09-10T19:13:54.239 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:13:54.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:13:54.262 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:13:54.263 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:13:54.363 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:13:54.384 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:13:54.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:13:54.385 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:13:54.486 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:13:54.508 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:13:54.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:13:54.509 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:13:54.610 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:13:54.634 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:13:54.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:13:54.635 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:13:54.736 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:13:54.767 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:13:54.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:13:54.768 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:13:54.853 INFO:teuthology.orchestra.run.smithi191.stdout:ERROR: (22) Invalid argument 2024-09-10T19:13:54.853 INFO:teuthology.orchestra.run.smithi191.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-10T19:13:54.869 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:13:54.892 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:13:54.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:13:54.893 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:13:54.994 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:13:55.017 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:13:55.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:13:55.019 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:13:55.119 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:13:55.142 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:13:55.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:13:55.143 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:13:55.173 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:13:54 smithi191 ceph-mon[32884]: pgmap v139: 148 pgs: 3 active+clean+scrubbing, 33 active+undersized, 26 active+undersized+degraded, 86 active+clean; 8.5 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 685 B/s rd, 49 MiB/s wr, 887 op/s; 52318/419094 objects degraded (12.484%) 2024-09-10T19:13:55.173 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:13:54 smithi191 ceph-mon[32884]: 7.c scrub ok 2024-09-10T19:13:55.174 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:13:54 smithi191 ceph-mon[32884]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-09-10T19:13:55.174 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:13:54 smithi191 ceph-mon[32884]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["1"]}]': finished 2024-09-10T19:13:55.174 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:13:54 smithi191 ceph-mon[32884]: osdmap e115: 8 total, 7 up, 7 in 2024-09-10T19:13:55.174 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:13:54 smithi191 ceph-mon[32884]: 7.7 scrub starts 2024-09-10T19:13:55.174 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:13:54 smithi191 ceph-mon[32884]: 6.c scrub starts 2024-09-10T19:13:55.174 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:13:54 smithi191 ceph-mon[32884]: 6.c scrub ok 2024-09-10T19:13:55.174 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:13:54 smithi191 ceph-mon[32884]: Health check update: Degraded data redundancy: 52318/419094 objects degraded (12.484%), 26 pgs degraded, 59 pgs undersized (PG_DEGRADED) 2024-09-10T19:13:55.174 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:13:54 smithi191 ceph-mon[32884]: osdmap e116: 8 total, 7 up, 7 in 2024-09-10T19:13:55.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:13:54 smithi183 ceph-mon[29736]: pgmap v139: 148 pgs: 3 active+clean+scrubbing, 33 active+undersized, 26 active+undersized+degraded, 86 active+clean; 8.5 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 685 B/s rd, 49 MiB/s wr, 887 op/s; 52318/419094 objects degraded (12.484%) 2024-09-10T19:13:55.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:13:54 smithi183 ceph-mon[29736]: 7.c scrub ok 2024-09-10T19:13:55.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:13:54 smithi183 ceph-mon[29736]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-09-10T19:13:55.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:13:54 smithi183 ceph-mon[29736]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["1"]}]': finished 2024-09-10T19:13:55.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:13:54 smithi183 ceph-mon[29736]: osdmap e115: 8 total, 7 up, 7 in 2024-09-10T19:13:55.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:13:54 smithi183 ceph-mon[29736]: 7.7 scrub starts 2024-09-10T19:13:55.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:13:54 smithi183 ceph-mon[29736]: 6.c scrub starts 2024-09-10T19:13:55.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:13:54 smithi183 ceph-mon[29736]: 6.c scrub ok 2024-09-10T19:13:55.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:13:54 smithi183 ceph-mon[29736]: Health check update: Degraded data redundancy: 52318/419094 objects degraded (12.484%), 26 pgs degraded, 59 pgs undersized (PG_DEGRADED) 2024-09-10T19:13:55.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:13:54 smithi183 ceph-mon[29736]: osdmap e116: 8 total, 7 up, 7 in 2024-09-10T19:13:55.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:13:54 smithi183 ceph-mon[41562]: pgmap v139: 148 pgs: 3 active+clean+scrubbing, 33 active+undersized, 26 active+undersized+degraded, 86 active+clean; 8.5 GiB data, 26 GiB used, 689 GiB / 715 GiB avail; 685 B/s rd, 49 MiB/s wr, 887 op/s; 52318/419094 objects degraded (12.484%) 2024-09-10T19:13:55.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:13:54 smithi183 ceph-mon[41562]: 7.c scrub ok 2024-09-10T19:13:55.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:13:54 smithi183 ceph-mon[41562]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-09-10T19:13:55.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:13:54 smithi183 ceph-mon[41562]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["1"]}]': finished 2024-09-10T19:13:55.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:13:54 smithi183 ceph-mon[41562]: osdmap e115: 8 total, 7 up, 7 in 2024-09-10T19:13:55.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:13:54 smithi183 ceph-mon[41562]: 7.7 scrub starts 2024-09-10T19:13:55.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:13:54 smithi183 ceph-mon[41562]: 6.c scrub starts 2024-09-10T19:13:55.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:13:54 smithi183 ceph-mon[41562]: 6.c scrub ok 2024-09-10T19:13:55.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:13:54 smithi183 ceph-mon[41562]: Health check update: Degraded data redundancy: 52318/419094 objects degraded (12.484%), 26 pgs degraded, 59 pgs undersized (PG_DEGRADED) 2024-09-10T19:13:55.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:13:54 smithi183 ceph-mon[41562]: osdmap e116: 8 total, 7 up, 7 in 2024-09-10T19:13:55.245 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:13:55.267 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:13:55.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:13:55.268 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:13:55.368 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:13:55.391 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:13:55.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:13:55.392 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:13:55.493 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:13:55.515 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:13:55.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:13:55.516 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:13:55.616 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:13:55.638 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:13:55.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:13:55.639 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:13:55.740 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:13:55.762 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:13:55.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:13:55.763 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:13:55.863 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:13:55.893 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:13:55.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:13:55.894 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:13:55.995 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:13:56.024 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:13:56.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:13:56.025 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:13:56.126 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:13:56.158 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:13:56.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:13:56.159 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:13:56.201 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:13:55 smithi183 ceph-mon[29736]: 7.5 scrub starts 2024-09-10T19:13:56.202 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:13:55 smithi183 ceph-mon[29736]: 7.7 scrub ok 2024-09-10T19:13:56.202 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:13:55 smithi183 ceph-mon[29736]: 4.17 scrub starts 2024-09-10T19:13:56.202 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:13:55 smithi183 ceph-mon[29736]: 4.17 scrub ok 2024-09-10T19:13:56.202 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:13:55 smithi183 ceph-mon[29736]: osdmap e117: 8 total, 7 up, 7 in 2024-09-10T19:13:56.203 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:13:55 smithi183 ceph-mon[41562]: 7.5 scrub starts 2024-09-10T19:13:56.203 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:13:55 smithi183 ceph-mon[41562]: 7.7 scrub ok 2024-09-10T19:13:56.203 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:13:55 smithi183 ceph-mon[41562]: 4.17 scrub starts 2024-09-10T19:13:56.203 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:13:55 smithi183 ceph-mon[41562]: 4.17 scrub ok 2024-09-10T19:13:56.203 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:13:55 smithi183 ceph-mon[41562]: osdmap e117: 8 total, 7 up, 7 in 2024-09-10T19:13:56.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:13:55 smithi191 ceph-mon[32884]: 7.5 scrub starts 2024-09-10T19:13:56.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:13:55 smithi191 ceph-mon[32884]: 7.7 scrub ok 2024-09-10T19:13:56.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:13:55 smithi191 ceph-mon[32884]: 4.17 scrub starts 2024-09-10T19:13:56.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:13:55 smithi191 ceph-mon[32884]: 4.17 scrub ok 2024-09-10T19:13:56.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:13:55 smithi191 ceph-mon[32884]: osdmap e117: 8 total, 7 up, 7 in 2024-09-10T19:13:56.260 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:13:56.296 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:13:56.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:13:56.297 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:13:56.398 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:13:56.414 DEBUG:teuthology.orchestra.run.smithi191:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph daemon osd.5 dump_ops_in_flight 2024-09-10T19:13:56.438 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:13:56.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:13:56.439 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:13:56.540 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:13:56.577 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:13:56.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:13:56.578 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:13:56.679 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:13:56.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:13:56.717 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:13:56.717 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:13:56.798 INFO:teuthology.orchestra.run.smithi191.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.b/config 2024-09-10T19:13:56.818 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:13:56.866 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:13:56.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:13:56.867 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:13:56.951 INFO:teuthology.orchestra.run.smithi183.stderr:nodeep-scrub is unset 2024-09-10T19:13:56.968 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:13:57.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:13:57.011 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:13:57.011 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:13:57.112 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:13:57.149 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:13:57.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:13:57.150 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:13:57.161 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:13:56 smithi183 ceph-mon[29736]: 7.5 scrub ok 2024-09-10T19:13:57.161 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:13:56 smithi183 ceph-mon[29736]: pgmap v142: 148 pgs: 3 remapped+peering, 1 active+recovering+degraded, 42 peering, 2 active+clean+scrubbing, 1 active+recovery_wait+degraded, 99 active+clean; 8.2 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 852 B/s rd, 25 MiB/s wr, 1.41k op/s; 10/402747 objects degraded (0.002%); 36 B/s, 2 objects/s recovering 2024-09-10T19:13:57.161 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:13:56 smithi183 ceph-mon[29736]: 7.2 starting backfill to osd.7 from (0'0,0'0] MAX to 114'9332 2024-09-10T19:13:57.161 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:13:56 smithi183 ceph-mon[29736]: 7.0 starting backfill to osd.2 from (0'0,0'0] MAX to 114'9142 2024-09-10T19:13:57.161 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:13:56 smithi183 ceph-mon[29736]: 7.b starting backfill to osd.2 from (0'0,0'0] MAX to 114'9141 2024-09-10T19:13:57.161 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:13:56 smithi183 ceph-mon[29736]: 7.d starting backfill to osd.3 from (0'0,0'0] MAX to 114'9174 2024-09-10T19:13:57.161 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:13:56 smithi183 ceph-mon[29736]: 7.4 starting backfill to osd.6 from (0'0,0'0] MAX to 114'9096 2024-09-10T19:13:57.161 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:13:56 smithi183 ceph-mon[29736]: 7.9 starting backfill to osd.7 from (0'0,0'0] MAX to 114'9072 2024-09-10T19:13:57.162 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:13:56 smithi183 ceph-mon[29736]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-09-10T19:13:57.162 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:13:56 smithi183 ceph-mon[29736]: from='client.? 172.21.15.183:0/791466231' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-10T19:13:57.162 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:13:56 smithi183 ceph-mon[41562]: 7.5 scrub ok 2024-09-10T19:13:57.162 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:13:56 smithi183 ceph-mon[41562]: pgmap v142: 148 pgs: 3 remapped+peering, 1 active+recovering+degraded, 42 peering, 2 active+clean+scrubbing, 1 active+recovery_wait+degraded, 99 active+clean; 8.2 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 852 B/s rd, 25 MiB/s wr, 1.41k op/s; 10/402747 objects degraded (0.002%); 36 B/s, 2 objects/s recovering 2024-09-10T19:13:57.162 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:13:56 smithi183 ceph-mon[41562]: 7.2 starting backfill to osd.7 from (0'0,0'0] MAX to 114'9332 2024-09-10T19:13:57.162 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:13:56 smithi183 ceph-mon[41562]: 7.0 starting backfill to osd.2 from (0'0,0'0] MAX to 114'9142 2024-09-10T19:13:57.162 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:13:56 smithi183 ceph-mon[41562]: 7.b starting backfill to osd.2 from (0'0,0'0] MAX to 114'9141 2024-09-10T19:13:57.162 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:13:56 smithi183 ceph-mon[41562]: 7.d starting backfill to osd.3 from (0'0,0'0] MAX to 114'9174 2024-09-10T19:13:57.162 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:13:56 smithi183 ceph-mon[41562]: 7.4 starting backfill to osd.6 from (0'0,0'0] MAX to 114'9096 2024-09-10T19:13:57.162 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:13:56 smithi183 ceph-mon[41562]: 7.9 starting backfill to osd.7 from (0'0,0'0] MAX to 114'9072 2024-09-10T19:13:57.162 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:13:56 smithi183 ceph-mon[41562]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-09-10T19:13:57.162 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:13:56 smithi183 ceph-mon[41562]: from='client.? 172.21.15.183:0/791466231' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-10T19:13:57.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:13:56 smithi191 ceph-mon[32884]: 7.5 scrub ok 2024-09-10T19:13:57.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:13:56 smithi191 ceph-mon[32884]: pgmap v142: 148 pgs: 3 remapped+peering, 1 active+recovering+degraded, 42 peering, 2 active+clean+scrubbing, 1 active+recovery_wait+degraded, 99 active+clean; 8.2 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 852 B/s rd, 25 MiB/s wr, 1.41k op/s; 10/402747 objects degraded (0.002%); 36 B/s, 2 objects/s recovering 2024-09-10T19:13:57.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:13:56 smithi191 ceph-mon[32884]: 7.2 starting backfill to osd.7 from (0'0,0'0] MAX to 114'9332 2024-09-10T19:13:57.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:13:56 smithi191 ceph-mon[32884]: 7.0 starting backfill to osd.2 from (0'0,0'0] MAX to 114'9142 2024-09-10T19:13:57.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:13:56 smithi191 ceph-mon[32884]: 7.b starting backfill to osd.2 from (0'0,0'0] MAX to 114'9141 2024-09-10T19:13:57.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:13:56 smithi191 ceph-mon[32884]: 7.d starting backfill to osd.3 from (0'0,0'0] MAX to 114'9174 2024-09-10T19:13:57.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:13:56 smithi191 ceph-mon[32884]: 7.4 starting backfill to osd.6 from (0'0,0'0] MAX to 114'9096 2024-09-10T19:13:57.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:13:56 smithi191 ceph-mon[32884]: 7.9 starting backfill to osd.7 from (0'0,0'0] MAX to 114'9072 2024-09-10T19:13:57.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:13:56 smithi191 ceph-mon[32884]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-09-10T19:13:57.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:13:56 smithi191 ceph-mon[32884]: from='client.? 172.21.15.183:0/791466231' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-10T19:13:57.250 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:13:57.287 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:13:57.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:13:57.288 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:13:57.389 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:13:57.423 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:13:57.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:13:57.424 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:13:57.525 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:13:57.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:13:57.559 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:13:57.559 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:13:57.659 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:13:57.701 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:13:57.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:13:57.702 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:13:57.715 INFO:journalctl@ceph.mgr.x.smithi191.stdout:Sep 10 19:13:57 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-x[34967]: ::ffff:172.21.15.191 - - [10/Sep/2024:19:13:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T19:13:57.803 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:13:57.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:13:57.838 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:13:57.838 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:13:57.939 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:13:57.968 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:13:57.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:13:57.969 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:13:58.070 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:13:58.110 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:13:58.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:13:58.111 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:13:58.203 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:13:57 smithi191 ceph-mon[32884]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-10T19:13:58.203 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:13:57 smithi191 ceph-mon[32884]: from='client.? 172.21.15.183:0/791466231' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-10T19:13:58.203 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:13:57 smithi191 ceph-mon[32884]: osdmap e118: 8 total, 7 up, 7 in 2024-09-10T19:13:58.212 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:13:58.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:13:58.254 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:13:58.254 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:13:58.308 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:13:57 smithi183 ceph-mon[29736]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-10T19:13:58.309 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:13:57 smithi183 ceph-mon[29736]: from='client.? 172.21.15.183:0/791466231' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-10T19:13:58.309 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:13:57 smithi183 ceph-mon[29736]: osdmap e118: 8 total, 7 up, 7 in 2024-09-10T19:13:58.309 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:13:57 smithi183 ceph-mon[41562]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-10T19:13:58.309 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:13:57 smithi183 ceph-mon[41562]: from='client.? 172.21.15.183:0/791466231' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-10T19:13:58.309 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:13:57 smithi183 ceph-mon[41562]: osdmap e118: 8 total, 7 up, 7 in 2024-09-10T19:13:58.355 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:13:58.393 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:13:58.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:13:58.394 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:13:58.495 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:13:58.525 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:13:58.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:13:58.526 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:13:58.627 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:13:58.662 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:13:58.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:13:58.663 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:13:58.764 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:13:58.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:13:58.802 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:13:58.803 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:13:58.887 INFO:teuthology.orchestra.run.smithi191.stdout:ERROR: (22) Invalid argument 2024-09-10T19:13:58.887 INFO:teuthology.orchestra.run.smithi191.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-10T19:13:58.903 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:13:58.931 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Sep 10 19:13:58 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-y[29989]: 2024-09-10T19:13:58.668+0000 7f9fad6cd640 -1 mgr get_metadata_python Requested missing service osd.1 2024-09-10T19:13:58.940 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:13:58.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:13:58.941 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:13:59.041 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:13:59.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:13:59.090 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:13:59.091 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:13:59.191 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:13:59.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:13:58 smithi191 ceph-mon[32884]: pgmap v145: 148 pgs: 3 remapped+peering, 1 active+recovering+degraded, 42 peering, 2 active+clean+scrubbing, 1 active+recovery_wait+degraded, 99 active+clean; 8.2 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 0 B/s wr, 1.36k op/s; 10/402747 objects degraded (0.002%); 54 B/s, 4 objects/s recovering 2024-09-10T19:13:59.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:13:58 smithi191 ceph-mon[32884]: from='mgr.14556 ' entity='mgr.y' 2024-09-10T19:13:59.217 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:13:58 smithi191 ceph-mon[32884]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:13:59.217 INFO:journalctl@ceph.iscsi.iscsi.a.smithi191.stdout:Sep 10 19:13:59 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-iscsi-iscsi-a[61816]: debug there is no tcmu-runner data available 2024-09-10T19:13:59.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:13:59.234 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:13:59.234 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:13:59.335 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:13:59.375 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:13:59.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:13:59.376 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:13:59.431 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:13:58 smithi183 ceph-mon[41562]: pgmap v145: 148 pgs: 3 remapped+peering, 1 active+recovering+degraded, 42 peering, 2 active+clean+scrubbing, 1 active+recovery_wait+degraded, 99 active+clean; 8.2 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 0 B/s wr, 1.36k op/s; 10/402747 objects degraded (0.002%); 54 B/s, 4 objects/s recovering 2024-09-10T19:13:59.431 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:13:58 smithi183 ceph-mon[41562]: from='mgr.14556 ' entity='mgr.y' 2024-09-10T19:13:59.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:13:58 smithi183 ceph-mon[41562]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:13:59.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:13:58 smithi183 ceph-mon[29736]: pgmap v145: 148 pgs: 3 remapped+peering, 1 active+recovering+degraded, 42 peering, 2 active+clean+scrubbing, 1 active+recovery_wait+degraded, 99 active+clean; 8.2 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 0 B/s wr, 1.36k op/s; 10/402747 objects degraded (0.002%); 54 B/s, 4 objects/s recovering 2024-09-10T19:13:59.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:13:58 smithi183 ceph-mon[29736]: from='mgr.14556 ' entity='mgr.y' 2024-09-10T19:13:59.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:13:58 smithi183 ceph-mon[29736]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:13:59.477 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:13:59.516 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:13:59.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:13:59.517 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:13:59.618 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:13:59.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:13:59.657 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:13:59.657 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:13:59.758 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:13:59.795 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:13:59.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:13:59.796 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:13:59.897 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:13:59.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:13:59.934 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:13:59.935 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:14:00.035 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:14:00.067 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:14:00.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:14:00.070 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:14:00.150 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:13:59 smithi191 ceph-mon[32884]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T19:14:00.150 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:13:59 smithi191 ceph-mon[32884]: from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:14:00.150 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:13:59 smithi191 ceph-mon[32884]: Health check update: Degraded data redundancy: 10/402747 objects degraded (0.002%), 2 pgs degraded (PG_DEGRADED) 2024-09-10T19:14:00.170 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:14:00.189 DEBUG:teuthology.orchestra.run.smithi191:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph daemon osd.5 dump_blocked_ops 2024-09-10T19:14:00.219 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:14:00.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:14:00.221 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:14:00.321 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:14:00.338 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:13:59 smithi183 ceph-mon[29736]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T19:14:00.339 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:13:59 smithi183 ceph-mon[29736]: from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:14:00.339 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:13:59 smithi183 ceph-mon[29736]: Health check update: Degraded data redundancy: 10/402747 objects degraded (0.002%), 2 pgs degraded (PG_DEGRADED) 2024-09-10T19:14:00.339 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:13:59 smithi183 ceph-mon[41562]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T19:14:00.339 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:13:59 smithi183 ceph-mon[41562]: from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:14:00.339 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:13:59 smithi183 ceph-mon[41562]: Health check update: Degraded data redundancy: 10/402747 objects degraded (0.002%), 2 pgs degraded (PG_DEGRADED) 2024-09-10T19:14:00.349 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph osd set noscrub 2024-09-10T19:14:00.383 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:14:00.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:14:00.385 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:14:00.485 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:14:00.521 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:14:00.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:14:00.522 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:14:00.623 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:14:00.670 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:14:00.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:14:00.672 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:14:00.684 INFO:teuthology.orchestra.run.smithi191.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.b/config 2024-09-10T19:14:00.773 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:14:00.807 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:14:00.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:14:00.811 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:14:00.848 INFO:teuthology.orchestra.run.smithi183.stdout:osd.0: {} 2024-09-10T19:14:00.849 INFO:teuthology.orchestra.run.smithi183.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-10T19:14:00.850 INFO:teuthology.orchestra.run.smithi183.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-09-10T19:14:00.885 INFO:teuthology.orchestra.run.smithi183.stdout:osd.2: {} 2024-09-10T19:14:00.885 INFO:teuthology.orchestra.run.smithi183.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-10T19:14:00.909 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:14:00.912 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:14:00.918 INFO:teuthology.orchestra.run.smithi183.stdout:osd.3: {} 2024-09-10T19:14:00.919 INFO:teuthology.orchestra.run.smithi183.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-10T19:14:00.938 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:14:00.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:14:00.939 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:14:00.964 INFO:teuthology.orchestra.run.smithi183.stdout:osd.4: {} 2024-09-10T19:14:00.965 INFO:teuthology.orchestra.run.smithi183.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-10T19:14:00.997 INFO:teuthology.orchestra.run.smithi183.stdout:osd.5: {} 2024-09-10T19:14:00.997 INFO:teuthology.orchestra.run.smithi183.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-10T19:14:01.022 INFO:teuthology.orchestra.run.smithi183.stdout:osd.6: {} 2024-09-10T19:14:01.022 INFO:teuthology.orchestra.run.smithi183.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-10T19:14:01.040 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:14:01.063 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:14:01.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:14:01.064 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:14:01.120 INFO:teuthology.orchestra.run.smithi183.stdout:osd.7: {} 2024-09-10T19:14:01.120 INFO:teuthology.orchestra.run.smithi183.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-10T19:14:01.165 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:14:01.207 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:14:01.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:14:01.208 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:14:01.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:14:00 smithi191 ceph-mon[32884]: pgmap v146: 148 pgs: 2 active+undersized+degraded+remapped+backfill_toofull, 3 remapped+peering, 1 active+recovering+degraded, 1 active+undersized+degraded+remapped+backfilling, 5 peering, 1 active+recovery_wait+degraded, 135 active+clean; 8.1 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 0 B/s wr, 1.32k op/s; 23638/396456 objects degraded (5.962%); 5.9 MiB/s, 107 objects/s recovering 2024-09-10T19:14:01.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:14:00 smithi191 ceph-mon[32884]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-09-10T19:14:01.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:14:00 smithi191 ceph-mon[32884]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-09-10T19:14:01.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:14:00 smithi191 ceph-mon[32884]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T19:14:01.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:14:00 smithi191 ceph-mon[32884]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T19:14:01.217 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:14:00 smithi191 ceph-mon[32884]: from='mgr.14556 ' entity='mgr.y' 2024-09-10T19:14:01.217 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:14:00 smithi191 ceph-mon[32884]: from='client.? 172.21.15.183:0/3479238357' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T19:14:01.219 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:14:00 smithi183 ceph-mon[29736]: pgmap v146: 148 pgs: 2 active+undersized+degraded+remapped+backfill_toofull, 3 remapped+peering, 1 active+recovering+degraded, 1 active+undersized+degraded+remapped+backfilling, 5 peering, 1 active+recovery_wait+degraded, 135 active+clean; 8.1 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 0 B/s wr, 1.32k op/s; 23638/396456 objects degraded (5.962%); 5.9 MiB/s, 107 objects/s recovering 2024-09-10T19:14:01.219 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:14:00 smithi183 ceph-mon[29736]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-09-10T19:14:01.219 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:14:00 smithi183 ceph-mon[29736]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-09-10T19:14:01.219 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:14:00 smithi183 ceph-mon[29736]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T19:14:01.219 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:14:00 smithi183 ceph-mon[29736]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T19:14:01.219 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:14:00 smithi183 ceph-mon[29736]: from='mgr.14556 ' entity='mgr.y' 2024-09-10T19:14:01.220 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:14:00 smithi183 ceph-mon[29736]: from='client.? 172.21.15.183:0/3479238357' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T19:14:01.220 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:14:00 smithi183 ceph-mon[41562]: pgmap v146: 148 pgs: 2 active+undersized+degraded+remapped+backfill_toofull, 3 remapped+peering, 1 active+recovering+degraded, 1 active+undersized+degraded+remapped+backfilling, 5 peering, 1 active+recovery_wait+degraded, 135 active+clean; 8.1 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 0 B/s wr, 1.32k op/s; 23638/396456 objects degraded (5.962%); 5.9 MiB/s, 107 objects/s recovering 2024-09-10T19:14:01.220 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:14:00 smithi183 ceph-mon[41562]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-09-10T19:14:01.220 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:14:00 smithi183 ceph-mon[41562]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-09-10T19:14:01.220 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:14:00 smithi183 ceph-mon[41562]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T19:14:01.220 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:14:00 smithi183 ceph-mon[41562]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T19:14:01.220 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:14:00 smithi183 ceph-mon[41562]: from='mgr.14556 ' entity='mgr.y' 2024-09-10T19:14:01.220 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:14:00 smithi183 ceph-mon[41562]: from='client.? 172.21.15.183:0/3479238357' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T19:14:01.309 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:14:01.339 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:14:01.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:14:01.340 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:14:01.440 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:14:01.469 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:14:01.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:14:01.470 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:14:01.515 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Sep 10 19:14:01 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-y[29989]: ::ffff:172.21.15.191 - - [10/Sep/2024:19:14:01] "GET /metrics HTTP/1.1" 200 208860 "" "Prometheus/2.43.0" 2024-09-10T19:14:01.571 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:14:01.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:14:01.614 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:14:01.614 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:14:01.715 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:14:01.754 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:14:01.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:14:01.756 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:14:01.856 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:14:01.889 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:14:01.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:14:01.890 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:14:01.991 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:14:01.996 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-10T19:14:01.997 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-10T19:14:02.021 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:14:02.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:14:02.022 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:14:02.123 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:14:02.160 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:14:02.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:14:02.161 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:14:02.205 INFO:tasks.thrashosds.thrasher:in_osds: [0, 2, 3, 4, 5, 6, 7] out_osds: [1] dead_osds: [1] live_osds: [0, 2, 3, 4, 5, 6, 7] 2024-09-10T19:14:02.206 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-10T19:14:02.206 INFO:tasks.thrashosds.thrasher:Shrinking pool .mgr 2024-09-10T19:14:02.206 INFO:tasks.cephadm.ceph_manager.ceph:contract_pool .mgr by 10 min 8 2024-09-10T19:14:02.207 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph pg dump --format=json 2024-09-10T19:14:02.241 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:14:01 smithi183 ceph-mon[29736]: 6.e scrub starts 2024-09-10T19:14:02.242 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:14:01 smithi183 ceph-mon[41562]: 6.e scrub starts 2024-09-10T19:14:02.262 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:14:02.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:14:02.319 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:14:02.319 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:14:02.419 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:14:02.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:14:02.455 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:14:02.455 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:14:02.465 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:14:01 smithi191 ceph-mon[32884]: 6.e scrub starts 2024-09-10T19:14:02.556 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:14:02.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:14:02.577 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:14:02.578 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:14:02.679 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:14:02.712 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:14:02.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:14:02.713 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:14:02.814 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:14:02.851 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:14:02.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:14:02.853 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:14:02.954 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:14:02.995 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:14:03.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:14:03.001 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:14:03.001 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:14:03.102 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:14:03.135 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:14:02 smithi183 ceph-mon[29736]: pgmap v147: 148 pgs: 3 active+undersized+degraded+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 142 active+clean; 7.9 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 735 B/s rd, 0 B/s wr, 1.05k op/s; 46864/387711 objects degraded (12.087%); 14 MiB/s, 233 objects/s recovering 2024-09-10T19:14:03.136 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:14:02 smithi183 ceph-mon[41562]: pgmap v147: 148 pgs: 3 active+undersized+degraded+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 142 active+clean; 7.9 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 735 B/s rd, 0 B/s wr, 1.05k op/s; 46864/387711 objects degraded (12.087%); 14 MiB/s, 233 objects/s recovering 2024-09-10T19:14:03.146 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:14:03.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:14:03.147 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:14:03.248 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:14:03.288 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:14:02 smithi191 ceph-mon[32884]: pgmap v147: 148 pgs: 3 active+undersized+degraded+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 142 active+clean; 7.9 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 735 B/s rd, 0 B/s wr, 1.05k op/s; 46864/387711 objects degraded (12.087%); 14 MiB/s, 233 objects/s recovering 2024-09-10T19:14:03.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:14:03.289 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:14:03.289 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:14:03.390 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:14:03.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:14:03.420 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:14:03.420 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:14:03.521 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:14:03.548 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:14:03.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:14:03.549 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:14:03.650 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:14:03.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:14:03.688 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:14:03.688 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:14:03.789 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:14:03.817 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:14:03.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:14:03.818 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:14:03.919 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:14:03.957 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:14:03.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:14:03.958 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:14:03.998 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-10T19:14:04.059 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:14:04.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:14:04.090 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:14:04.090 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:14:04.191 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:14:04.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:14:04.238 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:14:04.239 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:14:04.339 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:14:04.373 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:14:04.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:14:04.374 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:14:04.475 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:14:04.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:14:04.513 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:14:04.513 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:14:04.614 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:14:04.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:14:04.638 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:14:04.639 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:14:04.739 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:14:04.767 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:14:04.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:14:04.768 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:14:04.869 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:14:04.908 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:14:04.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:14:04.909 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:14:04.995 INFO:teuthology.orchestra.run.smithi183.stderr:noscrub is set 2024-09-10T19:14:05.009 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:14:05.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:14:05.050 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:14:05.050 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:14:05.151 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:14:05.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:14:05.192 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:14:05.192 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:14:05.217 INFO:teuthology.orchestra.run.smithi191.stdout:{ 2024-09-10T19:14:05.217 INFO:teuthology.orchestra.run.smithi191.stdout: "ops": [], 2024-09-10T19:14:05.218 INFO:teuthology.orchestra.run.smithi191.stdout: "complaint_time": 30, 2024-09-10T19:14:05.218 INFO:teuthology.orchestra.run.smithi191.stdout: "num_blocked_ops": 0 2024-09-10T19:14:05.218 INFO:teuthology.orchestra.run.smithi191.stdout:} 2024-09-10T19:14:05.293 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:14:05.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:14:05.314 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:14:05.315 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:14:05.415 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:14:05.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:14:04 smithi183 ceph-mon[29736]: pgmap v148: 148 pgs: 3 active+undersized+degraded+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 142 active+clean; 7.9 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 626 op/s; 46864/387711 objects degraded (12.087%); 12 MiB/s, 200 objects/s recovering 2024-09-10T19:14:05.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:14:04 smithi183 ceph-mon[29736]: from='client.? 172.21.15.183:0/2391269199' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-10T19:14:05.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:14:04 smithi183 ceph-mon[29736]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-10T19:14:05.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:14:04 smithi183 ceph-mon[29736]: Health check update: Degraded data redundancy: 46864/387711 objects degraded (12.087%), 6 pgs degraded (PG_DEGRADED) 2024-09-10T19:14:05.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:14:04 smithi183 ceph-mon[41562]: pgmap v148: 148 pgs: 3 active+undersized+degraded+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 142 active+clean; 7.9 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 626 op/s; 46864/387711 objects degraded (12.087%); 12 MiB/s, 200 objects/s recovering 2024-09-10T19:14:05.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:14:04 smithi183 ceph-mon[41562]: from='client.? 172.21.15.183:0/2391269199' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-10T19:14:05.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:14:04 smithi183 ceph-mon[41562]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-10T19:14:05.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:14:04 smithi183 ceph-mon[41562]: Health check update: Degraded data redundancy: 46864/387711 objects degraded (12.087%), 6 pgs degraded (PG_DEGRADED) 2024-09-10T19:14:05.446 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:14:05.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:14:05.447 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:14:05.465 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:14:04 smithi191 ceph-mon[32884]: pgmap v148: 148 pgs: 3 active+undersized+degraded+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 142 active+clean; 7.9 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 626 op/s; 46864/387711 objects degraded (12.087%); 12 MiB/s, 200 objects/s recovering 2024-09-10T19:14:05.465 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:14:04 smithi191 ceph-mon[32884]: from='client.? 172.21.15.183:0/2391269199' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-10T19:14:05.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:14:04 smithi191 ceph-mon[32884]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-10T19:14:05.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:14:04 smithi191 ceph-mon[32884]: Health check update: Degraded data redundancy: 46864/387711 objects degraded (12.087%), 6 pgs degraded (PG_DEGRADED) 2024-09-10T19:14:05.549 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:14:05.583 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:14:05.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:14:05.584 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:14:05.685 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:14:05.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:14:05.721 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:14:05.721 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:14:05.822 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:14:05.844 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:14:05.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:14:05.845 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:14:05.946 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:14:05.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:14:05.985 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:14:05.985 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:14:05.999 DEBUG:teuthology.orchestra.run.smithi191:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph daemon osd.5 dump_historic_ops 2024-09-10T19:14:06.086 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:14:06.123 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:14:06.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:14:06.124 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:14:06.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:14:06 smithi191 ceph-mon[32884]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T19:14:06.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:14:06 smithi191 ceph-mon[32884]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-10T19:14:06.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:14:06 smithi191 ceph-mon[32884]: osdmap e119: 8 total, 7 up, 7 in 2024-09-10T19:14:06.225 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:14:06.247 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:14:06.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:14:06.248 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:14:06.349 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:14:06.372 INFO:teuthology.orchestra.run.smithi191.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.b/config 2024-09-10T19:14:06.391 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:14:06.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:14:06.392 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:14:06.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:14:06 smithi183 ceph-mon[29736]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T19:14:06.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:14:06 smithi183 ceph-mon[29736]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-10T19:14:06.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:14:06 smithi183 ceph-mon[29736]: osdmap e119: 8 total, 7 up, 7 in 2024-09-10T19:14:06.431 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:14:06 smithi183 ceph-mon[41562]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T19:14:06.431 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:14:06 smithi183 ceph-mon[41562]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-10T19:14:06.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:14:06 smithi183 ceph-mon[41562]: osdmap e119: 8 total, 7 up, 7 in 2024-09-10T19:14:06.493 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:14:06.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:14:06.525 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:14:06.525 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:14:06.626 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:14:06.661 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:14:06.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:14:06.662 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:14:06.762 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:14:06.796 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:14:06.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:14:06.797 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:14:06.898 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:14:06.931 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:14:06.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:14:06.932 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:14:07.033 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:14:07.055 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:14:07.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:14:07.056 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:14:07.157 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:14:07.194 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:14:07.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:14:07.195 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:14:07.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:14:07 smithi191 ceph-mon[32884]: pgmap v150: 148 pgs: 3 active+undersized+degraded+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 142 active+clean; 7.5 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.28k op/s; 42686/369351 objects degraded (11.557%); 25 MiB/s, 413 objects/s recovering 2024-09-10T19:14:07.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:14:07 smithi191 ceph-mon[32884]: purged_snaps scrub starts 2024-09-10T19:14:07.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:14:07 smithi191 ceph-mon[32884]: purged_snaps scrub ok 2024-09-10T19:14:07.296 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:14:07.341 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:14:07.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:14:07.343 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:14:07.365 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:14:07 smithi183 ceph-mon[29736]: pgmap v150: 148 pgs: 3 active+undersized+degraded+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 142 active+clean; 7.5 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.28k op/s; 42686/369351 objects degraded (11.557%); 25 MiB/s, 413 objects/s recovering 2024-09-10T19:14:07.365 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:14:07 smithi183 ceph-mon[29736]: purged_snaps scrub starts 2024-09-10T19:14:07.365 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:14:07 smithi183 ceph-mon[29736]: purged_snaps scrub ok 2024-09-10T19:14:07.366 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:14:07 smithi183 ceph-mon[41562]: pgmap v150: 148 pgs: 3 active+undersized+degraded+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 142 active+clean; 7.5 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.28k op/s; 42686/369351 objects degraded (11.557%); 25 MiB/s, 413 objects/s recovering 2024-09-10T19:14:07.366 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:14:07 smithi183 ceph-mon[41562]: purged_snaps scrub starts 2024-09-10T19:14:07.366 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:14:07 smithi183 ceph-mon[41562]: purged_snaps scrub ok 2024-09-10T19:14:07.443 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:14:07.476 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:14:07.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:14:07.477 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:14:07.578 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:14:07.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:14:07.612 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:14:07.613 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:14:07.713 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:14:07.715 INFO:journalctl@ceph.mgr.x.smithi191.stdout:Sep 10 19:14:07 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-x[34967]: ::ffff:172.21.15.191 - - [10/Sep/2024:19:14:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T19:14:07.751 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:14:07.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:14:07.752 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:14:07.853 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:14:07.875 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:14:07.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:14:07.876 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:14:07.977 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:14:08.008 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:14:08.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:14:08.009 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:14:08.109 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:14:08.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:14:08.131 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:14:08.131 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:14:08.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:14:08 smithi191 ceph-mon[32884]: osdmap e120: 8 total, 7 up, 7 in 2024-09-10T19:14:08.232 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:14:08.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:14:08.263 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:14:08.263 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:14:08.364 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:14:08.404 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:14:08.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:14:08.405 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:14:08.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:14:08 smithi183 ceph-mon[29736]: osdmap e120: 8 total, 7 up, 7 in 2024-09-10T19:14:08.431 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:14:08 smithi183 ceph-mon[41562]: osdmap e120: 8 total, 7 up, 7 in 2024-09-10T19:14:08.506 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:14:08.535 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:14:08.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:14:08.536 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:14:08.637 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:14:08.675 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:14:08.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:14:08.676 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:14:08.777 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:14:08.811 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:14:08.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:14:08.813 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:14:08.913 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:14:08.949 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:14:08.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:14:08.950 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:14:09.003 INFO:teuthology.orchestra.run.smithi191.stdout:{ 2024-09-10T19:14:09.004 INFO:teuthology.orchestra.run.smithi191.stdout: "size": 20, 2024-09-10T19:14:09.004 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 600, 2024-09-10T19:14:09.004 INFO:teuthology.orchestra.run.smithi191.stdout: "ops": [ 2024-09-10T19:14:09.004 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:14:09.004 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.14781.0:16990 7.0 7:029aa942:::benchmark_data_smithi183_84160_object16989:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e90)", 2024-09-10T19:14:09.004 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-10T19:12:24.658426+0000", 2024-09-10T19:14:09.004 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 104.341165026, 2024-09-10T19:14:09.004 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.10757966200000001, 2024-09-10T19:14:09.004 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-10T19:14:09.004 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T19:14:09.004 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-10T19:14:09.004 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.14781", 2024-09-10T19:14:09.005 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.183:0/2025776420", 2024-09-10T19:14:09.005 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 16990 2024-09-10T19:14:09.005 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:14:09.005 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-10T19:14:09.005 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:14:09.005 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-10T19:14:09.005 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:12:24.658426+0000", 2024-09-10T19:14:09.005 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:14:09.005 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:14:09.005 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:14:09.005 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-10T19:14:09.005 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:12:24.658426+0000", 2024-09-10T19:14:09.006 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:14:09.006 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:14:09.006 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:14:09.006 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-10T19:14:09.006 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:12:24.658425+0000", 2024-09-10T19:14:09.006 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.999999 2024-09-10T19:14:09.006 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:14:09.006 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:14:09.006 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-10T19:14:09.006 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:12:24.658476+0000", 2024-09-10T19:14:09.006 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.0488000000000002e-05 2024-09-10T19:14:09.006 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:14:09.006 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:14:09.006 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-10T19:14:09.007 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:12:24.658487+0000", 2024-09-10T19:14:09.007 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.0648999999999999e-05 2024-09-10T19:14:09.007 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:14:09.007 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:14:09.007 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-10T19:14:09.007 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:12:24.658492+0000", 2024-09-10T19:14:09.007 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.451e-06 2024-09-10T19:14:09.007 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:14:09.007 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:14:09.007 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:14:09.007 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:12:24.658531+0000", 2024-09-10T19:14:09.007 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.9390000000000001e-05 2024-09-10T19:14:09.007 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:14:09.007 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:14:09.008 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-10T19:14:09.008 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:12:24.658639+0000", 2024-09-10T19:14:09.008 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000107923 2024-09-10T19:14:09.008 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:14:09.008 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:14:09.008 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 1,3", 2024-09-10T19:14:09.008 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:12:24.658737+0000", 2024-09-10T19:14:09.008 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.7850999999999995e-05 2024-09-10T19:14:09.008 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:14:09.008 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:14:09.008 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-09-10T19:14:09.008 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:12:24.659385+0000", 2024-09-10T19:14:09.008 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00064739799999999998 2024-09-10T19:14:09.009 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:14:09.009 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:14:09.009 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:14:09.009 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:12:24.661500+0000", 2024-09-10T19:14:09.009 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0021156080000000002 2024-09-10T19:14:09.009 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:14:09.009 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:14:09.009 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:14:09.009 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:12:24.765933+0000", 2024-09-10T19:14:09.009 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.104432953 2024-09-10T19:14:09.009 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:14:09.009 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:14:09.009 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-09-10T19:14:09.010 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:12:24.765964+0000", 2024-09-10T19:14:09.010 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.0654999999999997e-05 2024-09-10T19:14:09.010 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:14:09.010 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:14:09.010 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-10T19:14:09.010 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:12:24.766006+0000", 2024-09-10T19:14:09.010 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.2261000000000001e-05 2024-09-10T19:14:09.010 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:14:09.010 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-10T19:14:09.010 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:14:09.010 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:14:09.011 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:14:09.011 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.14781.0:17000 7.0 7:077d74d9:::benchmark_data_smithi183_84160_object16999:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e90)", 2024-09-10T19:14:09.011 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-10T19:12:24.661962+0000", 2024-09-10T19:14:09.011 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 104.337629034, 2024-09-10T19:14:09.011 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.104160778, 2024-09-10T19:14:09.011 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-10T19:14:09.011 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T19:14:09.011 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-10T19:14:09.011 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.14781", 2024-09-10T19:14:09.011 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.183:0/2025776420", 2024-09-10T19:14:09.011 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 17000 2024-09-10T19:14:09.011 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:14:09.011 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-10T19:14:09.012 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:14:09.012 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-10T19:14:09.012 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:12:24.661962+0000", 2024-09-10T19:14:09.012 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:14:09.012 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:14:09.012 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:14:09.012 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-10T19:14:09.012 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:12:24.661962+0000", 2024-09-10T19:14:09.012 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:14:09.012 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:14:09.012 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:14:09.012 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-10T19:14:09.012 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:12:24.661961+0000", 2024-09-10T19:14:09.012 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.999999 2024-09-10T19:14:09.013 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:14:09.013 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:14:09.013 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-10T19:14:09.013 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:12:24.662040+0000", 2024-09-10T19:14:09.013 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.8257999999999995e-05 2024-09-10T19:14:09.013 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:14:09.013 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:14:09.013 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-10T19:14:09.013 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:12:24.662056+0000", 2024-09-10T19:14:09.013 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.6093e-05 2024-09-10T19:14:09.013 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:14:09.013 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:14:09.013 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-10T19:14:09.013 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:12:24.662063+0000", 2024-09-10T19:14:09.014 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.8510000000000001e-06 2024-09-10T19:14:09.014 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:14:09.014 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:14:09.014 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:14:09.014 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:12:24.662114+0000", 2024-09-10T19:14:09.014 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.1026999999999998e-05 2024-09-10T19:14:09.014 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:14:09.014 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:14:09.014 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-10T19:14:09.014 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:12:24.662232+0000", 2024-09-10T19:14:09.014 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000117842 2024-09-10T19:14:09.014 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:14:09.015 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:14:09.015 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 1,3", 2024-09-10T19:14:09.015 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:12:24.662321+0000", 2024-09-10T19:14:09.015 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.8996e-05 2024-09-10T19:14:09.015 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:14:09.015 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:14:09.015 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-09-10T19:14:09.015 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:12:24.663029+0000", 2024-09-10T19:14:09.015 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00070813799999999997 2024-09-10T19:14:09.015 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:14:09.015 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:14:09.015 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:14:09.016 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:12:24.663353+0000", 2024-09-10T19:14:09.016 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00032453400000000002 2024-09-10T19:14:09.016 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:14:09.016 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:14:09.016 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:14:09.016 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:12:24.766043+0000", 2024-09-10T19:14:09.016 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.102690211 2024-09-10T19:14:09.016 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:14:09.016 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:14:09.016 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-09-10T19:14:09.016 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:12:24.766089+0000", 2024-09-10T19:14:09.016 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.5948999999999997e-05 2024-09-10T19:14:09.016 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:14:09.017 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:14:09.017 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-10T19:14:09.017 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:12:24.766123+0000", 2024-09-10T19:14:09.017 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.3880000000000001e-05 2024-09-10T19:14:09.017 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:14:09.017 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-10T19:14:09.017 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:14:09.017 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:14:09.017 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:14:09.017 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.14781.0:28872 7.0 7:0e81dbf5:::benchmark_data_smithi183_84160_object28871:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e91)", 2024-09-10T19:14:09.017 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-10T19:12:30.078416+0000", 2024-09-10T19:14:09.017 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 98.921175231000007, 2024-09-10T19:14:09.017 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.066826627999999999, 2024-09-10T19:14:09.018 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-10T19:14:09.018 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T19:14:09.018 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-10T19:14:09.018 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.14781", 2024-09-10T19:14:09.018 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.183:0/2025776420", 2024-09-10T19:14:09.019 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 28872 2024-09-10T19:14:09.019 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:14:09.019 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-10T19:14:09.019 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:14:09.019 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-10T19:14:09.019 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:12:30.078416+0000", 2024-09-10T19:14:09.019 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:14:09.019 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:14:09.019 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:14:09.019 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-10T19:14:09.019 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:12:30.078416+0000", 2024-09-10T19:14:09.019 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:14:09.020 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:14:09.020 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:14:09.020 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-10T19:14:09.020 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:12:30.078415+0000", 2024-09-10T19:14:09.020 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999986 2024-09-10T19:14:09.020 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:14:09.020 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:14:09.020 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-10T19:14:09.020 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:12:30.078492+0000", 2024-09-10T19:14:09.020 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.7010999999999998e-05 2024-09-10T19:14:09.020 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:14:09.020 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:14:09.020 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-10T19:14:09.020 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:12:30.078507+0000", 2024-09-10T19:14:09.021 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.5375000000000001e-05 2024-09-10T19:14:09.021 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:14:09.021 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:14:09.021 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-10T19:14:09.021 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:12:30.078514+0000", 2024-09-10T19:14:09.021 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.6139999999999997e-06 2024-09-10T19:14:09.021 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:14:09.021 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:14:09.021 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:14:09.021 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:12:30.078583+0000", 2024-09-10T19:14:09.021 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.8998000000000006e-05 2024-09-10T19:14:09.021 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:14:09.021 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:14:09.022 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-10T19:14:09.022 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:12:30.079289+0000", 2024-09-10T19:14:09.022 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00070605199999999998 2024-09-10T19:14:09.022 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:14:09.022 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:14:09.022 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 1,3", 2024-09-10T19:14:09.022 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:12:30.079450+0000", 2024-09-10T19:14:09.022 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00016090200000000001 2024-09-10T19:14:09.022 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:14:09.022 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:14:09.022 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-09-10T19:14:09.022 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:12:30.080679+0000", 2024-09-10T19:14:09.022 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0012291789999999999 2024-09-10T19:14:09.022 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:14:09.023 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:14:09.023 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:14:09.023 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:12:30.083419+0000", 2024-09-10T19:14:09.023 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0027396230000000001 2024-09-10T19:14:09.023 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:14:09.023 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:14:09.023 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:14:09.023 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:12:30.145154+0000", 2024-09-10T19:14:09.023 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.061734833000000003 2024-09-10T19:14:09.023 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:14:09.023 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:14:09.023 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-09-10T19:14:09.023 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:12:30.145209+0000", 2024-09-10T19:14:09.024 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.5309999999999997e-05 2024-09-10T19:14:09.024 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:14:09.024 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:14:09.024 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-10T19:14:09.024 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:12:30.145243+0000", 2024-09-10T19:14:09.024 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.3939999999999997e-05 2024-09-10T19:14:09.024 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:14:09.024 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-10T19:14:09.024 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:14:09.024 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:14:09.024 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:14:09.024 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.14781.0:28881 7.0 7:0026bf7d:::benchmark_data_smithi183_84160_object28880:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e91)", 2024-09-10T19:14:09.024 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-10T19:12:30.082160+0000", 2024-09-10T19:14:09.024 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 98.917431621000006, 2024-09-10T19:14:09.025 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.070512349000000002, 2024-09-10T19:14:09.025 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-10T19:14:09.025 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T19:14:09.025 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-10T19:14:09.025 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.14781", 2024-09-10T19:14:09.025 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.183:0/2025776420", 2024-09-10T19:14:09.025 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 28881 2024-09-10T19:14:09.025 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:14:09.025 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-10T19:14:09.025 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:14:09.025 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-10T19:14:09.026 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:12:30.082160+0000", 2024-09-10T19:14:09.026 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:14:09.026 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:14:09.026 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:14:09.026 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-10T19:14:09.026 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:12:30.082160+0000", 2024-09-10T19:14:09.026 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:14:09.026 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:14:09.026 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:14:09.026 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-10T19:14:09.026 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:12:30.082159+0000", 2024-09-10T19:14:09.026 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.999999 2024-09-10T19:14:09.026 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:14:09.026 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:14:09.027 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-10T19:14:09.027 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:12:30.082301+0000", 2024-09-10T19:14:09.027 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00014177999999999999 2024-09-10T19:14:09.027 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:14:09.027 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:14:09.027 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-10T19:14:09.027 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:12:30.082315+0000", 2024-09-10T19:14:09.027 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.4377e-05 2024-09-10T19:14:09.027 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:14:09.027 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:14:09.027 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-10T19:14:09.027 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:12:30.082321+0000", 2024-09-10T19:14:09.027 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.2920000000000001e-06 2024-09-10T19:14:09.028 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:14:09.028 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:14:09.028 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:14:09.028 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:12:30.082360+0000", 2024-09-10T19:14:09.028 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.9033999999999999e-05 2024-09-10T19:14:09.028 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:14:09.028 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:14:09.028 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-10T19:14:09.028 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:12:30.082439+0000", 2024-09-10T19:14:09.028 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.8147999999999998e-05 2024-09-10T19:14:09.028 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:14:09.028 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:14:09.029 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 1,3", 2024-09-10T19:14:09.029 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:12:30.082539+0000", 2024-09-10T19:14:09.029 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00010041500000000001 2024-09-10T19:14:09.029 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:14:09.029 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:14:09.029 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-09-10T19:14:09.029 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:12:30.084398+0000", 2024-09-10T19:14:09.029 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00185914 2024-09-10T19:14:09.029 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:14:09.029 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:14:09.029 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:14:09.029 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:12:30.084425+0000", 2024-09-10T19:14:09.030 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.7319999999999999e-05 2024-09-10T19:14:09.030 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:14:09.030 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:14:09.030 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:14:09.030 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:12:30.152567+0000", 2024-09-10T19:14:09.030 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.068141675999999998 2024-09-10T19:14:09.030 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:14:09.030 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:14:09.030 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-09-10T19:14:09.030 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:12:30.152624+0000", 2024-09-10T19:14:09.030 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.7376000000000001e-05 2024-09-10T19:14:09.030 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:14:09.030 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:14:09.031 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-10T19:14:09.031 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:12:30.152672+0000", 2024-09-10T19:14:09.031 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.7754999999999999e-05 2024-09-10T19:14:09.031 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:14:09.031 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-10T19:14:09.031 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:14:09.031 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:14:09.031 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:14:09.031 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.14781.0:28883 7.0 7:077e6950:::benchmark_data_smithi183_84160_object28882:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e91)", 2024-09-10T19:14:09.031 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-10T19:12:30.082623+0000", 2024-09-10T19:14:09.031 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 98.916968709000002, 2024-09-10T19:14:09.031 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.070187617999999993, 2024-09-10T19:14:09.031 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-10T19:14:09.031 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T19:14:09.032 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-10T19:14:09.032 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.14781", 2024-09-10T19:14:09.032 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.183:0/2025776420", 2024-09-10T19:14:09.032 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 28883 2024-09-10T19:14:09.032 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:14:09.032 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-10T19:14:09.032 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:14:09.032 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-10T19:14:09.032 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:12:30.082623+0000", 2024-09-10T19:14:09.032 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:14:09.032 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:14:09.032 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:14:09.032 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-10T19:14:09.032 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:12:30.082623+0000", 2024-09-10T19:14:09.033 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:14:09.033 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:14:09.033 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:14:09.033 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-10T19:14:09.033 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:12:30.082622+0000", 2024-09-10T19:14:09.033 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999995 2024-09-10T19:14:09.033 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:14:09.033 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:14:09.033 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-10T19:14:09.033 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:12:30.082695+0000", 2024-09-10T19:14:09.033 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.2685e-05 2024-09-10T19:14:09.033 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:14:09.034 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:14:09.034 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-10T19:14:09.034 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:12:30.082709+0000", 2024-09-10T19:14:09.034 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.4341999999999999e-05 2024-09-10T19:14:09.034 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:14:09.034 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:14:09.034 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-10T19:14:09.034 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:12:30.082717+0000", 2024-09-10T19:14:09.034 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.6890000000000004e-06 2024-09-10T19:14:09.034 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:14:09.034 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:14:09.034 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:14:09.034 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:12:30.083451+0000", 2024-09-10T19:14:09.034 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000734141 2024-09-10T19:14:09.035 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:14:09.035 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:14:09.035 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-10T19:14:09.035 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:12:30.083555+0000", 2024-09-10T19:14:09.035 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000104366 2024-09-10T19:14:09.035 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:14:09.035 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:14:09.035 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 1,3", 2024-09-10T19:14:09.035 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:12:30.083677+0000", 2024-09-10T19:14:09.035 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000121785 2024-09-10T19:14:09.035 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:14:09.035 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:14:09.035 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-09-10T19:14:09.035 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:12:30.084519+0000", 2024-09-10T19:14:09.036 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00084176900000000002 2024-09-10T19:14:09.036 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:14:09.036 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:14:09.036 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:14:09.036 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:12:30.084695+0000", 2024-09-10T19:14:09.036 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00017657799999999999 2024-09-10T19:14:09.036 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:14:09.036 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:14:09.036 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:14:09.037 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:12:30.152720+0000", 2024-09-10T19:14:09.037 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.068024799999999996 2024-09-10T19:14:09.037 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:14:09.037 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:14:09.037 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-09-10T19:14:09.037 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:12:30.152778+0000", 2024-09-10T19:14:09.037 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.8050000000000002e-05 2024-09-10T19:14:09.037 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:14:09.037 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:14:09.037 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-10T19:14:09.037 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:12:30.152810+0000", 2024-09-10T19:14:09.037 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.2041e-05 2024-09-10T19:14:09.037 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:14:09.037 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-10T19:14:09.038 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:14:09.038 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:14:09.038 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:14:09.038 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.14781.0:97065 7.0 7:06c19f36:::benchmark_data_smithi183_84160_object97064:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105)", 2024-09-10T19:14:09.038 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-10T19:13:14.418862+0000", 2024-09-10T19:14:09.038 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 54.580729906999998, 2024-09-10T19:14:09.038 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.062726241000000002, 2024-09-10T19:14:09.038 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-10T19:14:09.038 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T19:14:09.038 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-10T19:14:09.038 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.14781", 2024-09-10T19:14:09.038 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.183:0/2025776420", 2024-09-10T19:14:09.038 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 97065 2024-09-10T19:14:09.038 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:14:09.039 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-10T19:14:09.039 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:14:09.039 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-10T19:14:09.039 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:13:14.418862+0000", 2024-09-10T19:14:09.039 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:14:09.039 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:14:09.039 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:14:09.039 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-10T19:14:09.039 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:13:14.418862+0000", 2024-09-10T19:14:09.039 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:14:09.039 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:14:09.039 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:14:09.039 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-10T19:14:09.040 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:13:14.418860+0000", 2024-09-10T19:14:09.040 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999986 2024-09-10T19:14:09.040 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:14:09.040 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:14:09.040 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-10T19:14:09.040 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:13:14.418925+0000", 2024-09-10T19:14:09.040 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.4329e-05 2024-09-10T19:14:09.040 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:14:09.040 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:14:09.040 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-10T19:14:09.040 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:13:14.419020+0000", 2024-09-10T19:14:09.040 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.5592000000000002e-05 2024-09-10T19:14:09.040 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:14:09.041 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:14:09.041 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-10T19:14:09.041 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:13:14.419027+0000", 2024-09-10T19:14:09.041 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.4069999999999998e-06 2024-09-10T19:14:09.041 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:14:09.041 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:14:09.041 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:14:09.041 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:13:14.419144+0000", 2024-09-10T19:14:09.041 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000117268 2024-09-10T19:14:09.041 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:14:09.041 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:14:09.041 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-10T19:14:09.041 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:13:14.419256+0000", 2024-09-10T19:14:09.042 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00011212100000000001 2024-09-10T19:14:09.042 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:14:09.042 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:14:09.042 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 3", 2024-09-10T19:14:09.042 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:13:14.419399+0000", 2024-09-10T19:14:09.042 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00014259299999999999 2024-09-10T19:14:09.042 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:14:09.042 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:14:09.042 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-09-10T19:14:09.042 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:13:14.425559+0000", 2024-09-10T19:14:09.042 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0061598360000000001 2024-09-10T19:14:09.042 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:14:09.042 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:14:09.043 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:14:09.043 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:13:14.481476+0000", 2024-09-10T19:14:09.043 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.055917432000000003 2024-09-10T19:14:09.043 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:14:09.043 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:14:09.043 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-09-10T19:14:09.043 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:13:14.481533+0000", 2024-09-10T19:14:09.043 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.7234000000000001e-05 2024-09-10T19:14:09.043 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:14:09.043 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:14:09.043 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-10T19:14:09.043 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:13:14.481588+0000", 2024-09-10T19:14:09.043 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.4636999999999999e-05 2024-09-10T19:14:09.044 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:14:09.044 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-10T19:14:09.044 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:14:09.044 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:14:09.044 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:14:09.044 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.14781.0:97076 7.0 7:0a77f5e2:::benchmark_data_smithi183_84160_object97075:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105)", 2024-09-10T19:14:09.044 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-10T19:13:14.426486+0000", 2024-09-10T19:14:09.044 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 54.573105243999997, 2024-09-10T19:14:09.044 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.067507767999999996, 2024-09-10T19:14:09.044 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-10T19:14:09.044 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T19:14:09.044 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-10T19:14:09.045 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.14781", 2024-09-10T19:14:09.045 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.183:0/2025776420", 2024-09-10T19:14:09.045 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 97076 2024-09-10T19:14:09.045 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:14:09.045 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-10T19:14:09.045 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:14:09.045 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-10T19:14:09.045 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:13:14.426486+0000", 2024-09-10T19:14:09.045 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:14:09.045 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:14:09.045 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:14:09.045 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-10T19:14:09.045 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:13:14.426486+0000", 2024-09-10T19:14:09.046 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:14:09.046 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:14:09.046 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:14:09.046 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-10T19:14:09.046 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:13:14.426485+0000", 2024-09-10T19:14:09.046 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999981 2024-09-10T19:14:09.046 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:14:09.046 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:14:09.046 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-10T19:14:09.046 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:13:14.426526+0000", 2024-09-10T19:14:09.046 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.1522e-05 2024-09-10T19:14:09.046 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:14:09.046 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:14:09.047 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-10T19:14:09.047 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:13:14.426543+0000", 2024-09-10T19:14:09.047 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.6558e-05 2024-09-10T19:14:09.047 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:14:09.047 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:14:09.047 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-10T19:14:09.047 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:13:14.426549+0000", 2024-09-10T19:14:09.047 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.4849999999999997e-06 2024-09-10T19:14:09.047 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:14:09.047 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:14:09.047 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:14:09.047 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:13:14.426597+0000", 2024-09-10T19:14:09.047 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.7924999999999999e-05 2024-09-10T19:14:09.048 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:14:09.048 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:14:09.048 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-10T19:14:09.048 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:13:14.426721+0000", 2024-09-10T19:14:09.048 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00012440199999999999 2024-09-10T19:14:09.048 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:14:09.048 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:14:09.048 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 3", 2024-09-10T19:14:09.048 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:13:14.426883+0000", 2024-09-10T19:14:09.048 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000161256 2024-09-10T19:14:09.048 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:14:09.048 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:14:09.048 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-09-10T19:14:09.049 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:13:14.429491+0000", 2024-09-10T19:14:09.049 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0026083819999999998 2024-09-10T19:14:09.049 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:14:09.049 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:14:09.049 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:14:09.049 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:13:14.493885+0000", 2024-09-10T19:14:09.049 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.064393905000000001 2024-09-10T19:14:09.049 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:14:09.049 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:14:09.049 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-09-10T19:14:09.049 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:13:14.493930+0000", 2024-09-10T19:14:09.049 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.5099000000000003e-05 2024-09-10T19:14:09.049 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:14:09.050 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:14:09.050 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-10T19:14:09.050 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:13:14.493994+0000", 2024-09-10T19:14:09.050 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.3915e-05 2024-09-10T19:14:09.050 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:14:09.050 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-10T19:14:09.050 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:14:09.050 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:14:09.050 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:14:09.050 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.14781.0:119189 7.0 7:095613c9:::benchmark_data_smithi183_84160_object119188:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e108)", 2024-09-10T19:14:09.050 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-10T19:13:31.436599+0000", 2024-09-10T19:14:09.050 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 37.562992360999999, 2024-09-10T19:14:09.050 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.081299402000000007, 2024-09-10T19:14:09.051 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-10T19:14:09.051 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T19:14:09.051 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-10T19:14:09.051 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.14781", 2024-09-10T19:14:09.051 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.183:0/2025776420", 2024-09-10T19:14:09.051 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 119189 2024-09-10T19:14:09.051 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:14:09.051 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-10T19:14:09.051 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:14:09.051 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-10T19:14:09.051 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:13:31.436599+0000", 2024-09-10T19:14:09.051 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:14:09.051 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:14:09.051 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:14:09.052 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-10T19:14:09.052 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:13:31.436599+0000", 2024-09-10T19:14:09.052 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:14:09.052 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:14:09.052 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:14:09.052 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-10T19:14:09.052 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:13:31.436597+0000", 2024-09-10T19:14:09.052 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999981 2024-09-10T19:14:09.052 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:14:09.052 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:14:09.052 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-10T19:14:09.052 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:13:31.436642+0000", 2024-09-10T19:14:09.052 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.4502999999999997e-05 2024-09-10T19:14:09.053 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:14:09.053 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:14:09.053 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-10T19:14:09.053 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:13:31.436654+0000", 2024-09-10T19:14:09.053 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.2366e-05 2024-09-10T19:14:09.053 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:14:09.053 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:14:09.053 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-10T19:14:09.053 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:13:31.436661+0000", 2024-09-10T19:14:09.053 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.4649999999999999e-06 2024-09-10T19:14:09.053 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:14:09.053 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:14:09.053 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:14:09.054 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:13:31.440106+0000", 2024-09-10T19:14:09.054 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0034452789999999999 2024-09-10T19:14:09.054 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:14:09.054 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:14:09.054 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-10T19:14:09.054 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:13:31.440219+0000", 2024-09-10T19:14:09.054 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000112868 2024-09-10T19:14:09.054 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:14:09.054 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:14:09.054 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 3", 2024-09-10T19:14:09.054 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:13:31.440530+0000", 2024-09-10T19:14:09.054 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00031094500000000003 2024-09-10T19:14:09.054 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:14:09.055 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:14:09.055 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-09-10T19:14:09.055 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:13:31.444307+0000", 2024-09-10T19:14:09.055 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0037775949999999999 2024-09-10T19:14:09.055 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:14:09.055 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:14:09.055 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:14:09.055 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:13:31.517745+0000", 2024-09-10T19:14:09.055 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.073437737000000003 2024-09-10T19:14:09.055 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:14:09.055 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:14:09.055 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-09-10T19:14:09.056 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:13:31.517810+0000", 2024-09-10T19:14:09.056 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.4578000000000004e-05 2024-09-10T19:14:09.056 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:14:09.056 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:14:09.056 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-10T19:14:09.056 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:13:31.517899+0000", 2024-09-10T19:14:09.056 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.8888999999999995e-05 2024-09-10T19:14:09.056 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:14:09.056 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-10T19:14:09.056 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:14:09.056 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:14:09.056 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:14:09.056 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.14781.0:155169 7.0 7:019745de:::benchmark_data_smithi183_84160_object14788:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e118)", 2024-09-10T19:14:09.057 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-10T19:14:02.543677+0000", 2024-09-10T19:14:09.057 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 6.4559142329999997, 2024-09-10T19:14:09.057 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.075378938000000006, 2024-09-10T19:14:09.057 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-10T19:14:09.057 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T19:14:09.057 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-10T19:14:09.057 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.14781", 2024-09-10T19:14:09.057 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.183:0/2025776420", 2024-09-10T19:14:09.057 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 155169 2024-09-10T19:14:09.057 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:14:09.057 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-10T19:14:09.057 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:14:09.057 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-10T19:14:09.057 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:02.543677+0000", 2024-09-10T19:14:09.058 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:14:09.058 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:14:09.058 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:14:09.058 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-10T19:14:09.058 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:02.543677+0000", 2024-09-10T19:14:09.058 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:14:09.058 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:14:09.058 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:14:09.058 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-10T19:14:09.058 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:02.543676+0000", 2024-09-10T19:14:09.058 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999986 2024-09-10T19:14:09.058 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:14:09.058 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:14:09.058 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-10T19:14:09.059 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:02.543681+0000", 2024-09-10T19:14:09.059 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.6060000000000002e-06 2024-09-10T19:14:09.059 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:14:09.059 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:14:09.059 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-10T19:14:09.059 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:02.543691+0000", 2024-09-10T19:14:09.059 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.5840000000000001e-06 2024-09-10T19:14:09.059 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:14:09.059 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:14:09.059 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-10T19:14:09.059 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:02.543701+0000", 2024-09-10T19:14:09.059 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.4399999999999994e-06 2024-09-10T19:14:09.059 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:14:09.060 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:14:09.060 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:14:09.060 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:02.543739+0000", 2024-09-10T19:14:09.060 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.8525999999999997e-05 2024-09-10T19:14:09.060 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:14:09.060 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:14:09.060 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-10T19:14:09.060 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:02.543853+0000", 2024-09-10T19:14:09.060 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000113793 2024-09-10T19:14:09.060 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:14:09.060 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:14:09.060 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 2,3", 2024-09-10T19:14:09.060 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:02.543968+0000", 2024-09-10T19:14:09.061 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000115326 2024-09-10T19:14:09.061 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:14:09.061 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:14:09.061 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-09-10T19:14:09.061 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:02.615821+0000", 2024-09-10T19:14:09.061 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.071852999000000001 2024-09-10T19:14:09.061 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:14:09.061 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:14:09.061 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:14:09.061 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:02.618843+0000", 2024-09-10T19:14:09.061 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0030217569999999999 2024-09-10T19:14:09.061 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:14:09.061 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:14:09.062 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:14:09.062 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:02.618886+0000", 2024-09-10T19:14:09.062 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.2843999999999998e-05 2024-09-10T19:14:09.062 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:14:09.062 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:14:09.062 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:14:09.063 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-09-10T19:14:09.063 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:02.618937+0000", 2024-09-10T19:14:09.063 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.1189000000000001e-05 2024-09-10T19:14:09.063 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:14:09.064 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:14:09.064 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-10T19:14:09.064 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:02.619056+0000", 2024-09-10T19:14:09.064 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000119228 2024-09-10T19:14:09.064 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:14:09.064 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-10T19:14:09.064 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:14:09.064 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:14:09.064 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:14:09.064 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_repop(client.14781.0:155167 7.a e118/86 7:5b758192:::benchmark_data_smithi183_84160_object14786:head v 118'9783, mlcod=118'9780)", 2024-09-10T19:14:09.064 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-10T19:14:02.544168+0000", 2024-09-10T19:14:09.064 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 6.4554232909999998, 2024-09-10T19:14:09.064 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.077390608999999999, 2024-09-10T19:14:09.065 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-10T19:14:09.065 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T19:14:09.065 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-10T19:14:09.065 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:14:09.065 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-10T19:14:09.065 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:02.544168+0000", 2024-09-10T19:14:09.065 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:14:09.065 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:14:09.065 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:14:09.065 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-10T19:14:09.065 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:02.544168+0000", 2024-09-10T19:14:09.065 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:14:09.065 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:14:09.065 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:14:09.066 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-10T19:14:09.066 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:02.544167+0000", 2024-09-10T19:14:09.066 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.999999 2024-09-10T19:14:09.066 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:14:09.066 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:14:09.066 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-10T19:14:09.066 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:02.544173+0000", 2024-09-10T19:14:09.066 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.293e-06 2024-09-10T19:14:09.066 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:14:09.066 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:14:09.066 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-10T19:14:09.066 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:02.544181+0000", 2024-09-10T19:14:09.066 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.5750000000000001e-06 2024-09-10T19:14:09.067 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:14:09.067 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:14:09.067 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-10T19:14:09.067 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:02.544185+0000", 2024-09-10T19:14:09.067 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.9809999999999997e-06 2024-09-10T19:14:09.067 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:14:09.067 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:14:09.067 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:14:09.067 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:02.615771+0000", 2024-09-10T19:14:09.067 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.071585278000000002 2024-09-10T19:14:09.067 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:14:09.068 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:14:09.068 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-10T19:14:09.068 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:02.615792+0000", 2024-09-10T19:14:09.068 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.1463000000000001e-05 2024-09-10T19:14:09.068 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:14:09.068 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:14:09.068 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-09-10T19:14:09.068 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:02.621495+0000", 2024-09-10T19:14:09.068 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.005702581 2024-09-10T19:14:09.068 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:14:09.068 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:14:09.068 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-10T19:14:09.069 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:02.621559+0000", 2024-09-10T19:14:09.069 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.4239000000000006e-05 2024-09-10T19:14:09.069 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:14:09.069 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-10T19:14:09.069 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:14:09.069 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:14:09.069 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:14:09.069 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_repop_reply(client.14781.0:155169 7.0 e118/116 ondisk, result = 0)", 2024-09-10T19:14:09.069 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-10T19:14:02.547490+0000", 2024-09-10T19:14:09.069 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 6.4521014000000001, 2024-09-10T19:14:09.069 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.071358435999999997, 2024-09-10T19:14:09.069 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-10T19:14:09.069 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "started", 2024-09-10T19:14:09.070 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-10T19:14:09.070 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:14:09.070 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-10T19:14:09.070 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:02.547490+0000", 2024-09-10T19:14:09.070 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:14:09.070 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:14:09.070 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:14:09.070 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-10T19:14:09.070 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:02.547490+0000", 2024-09-10T19:14:09.070 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:14:09.070 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:14:09.070 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:14:09.070 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-10T19:14:09.070 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:02.547490+0000", 2024-09-10T19:14:09.071 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999995 2024-09-10T19:14:09.071 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:14:09.071 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:14:09.071 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-10T19:14:09.071 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:02.547493+0000", 2024-09-10T19:14:09.071 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.675e-06 2024-09-10T19:14:09.071 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:14:09.071 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:14:09.071 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-10T19:14:09.071 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:02.615760+0000", 2024-09-10T19:14:09.072 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.068266429000000003 2024-09-10T19:14:09.072 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:14:09.072 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:14:09.072 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-10T19:14:09.072 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:02.615769+0000", 2024-09-10T19:14:09.072 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.7850000000000003e-06 2024-09-10T19:14:09.072 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:14:09.072 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:14:09.072 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:14:09.072 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:02.618821+0000", 2024-09-10T19:14:09.072 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0030512650000000001 2024-09-10T19:14:09.072 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:14:09.072 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:14:09.073 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-10T19:14:09.073 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:02.618836+0000", 2024-09-10T19:14:09.073 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.4955e-05 2024-09-10T19:14:09.073 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:14:09.073 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:14:09.073 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-10T19:14:09.073 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:02.618849+0000", 2024-09-10T19:14:09.073 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.2857000000000001e-05 2024-09-10T19:14:09.073 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:14:09.073 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-10T19:14:09.073 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:14:09.073 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:14:09.074 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:14:09.074 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.14781.0:157918 7.0 7:0df4c65a:::benchmark_data_smithi183_84160_object17537:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e118)", 2024-09-10T19:14:09.074 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-10T19:14:04.751954+0000", 2024-09-10T19:14:09.074 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 4.2476373279999997, 2024-09-10T19:14:09.074 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.066766651999999996, 2024-09-10T19:14:09.074 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-10T19:14:09.074 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T19:14:09.074 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-10T19:14:09.074 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.14781", 2024-09-10T19:14:09.074 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.183:0/2025776420", 2024-09-10T19:14:09.074 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 157918 2024-09-10T19:14:09.074 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:14:09.074 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-10T19:14:09.075 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:14:09.075 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-10T19:14:09.075 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:04.751954+0000", 2024-09-10T19:14:09.075 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:14:09.075 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:14:09.075 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:14:09.075 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-10T19:14:09.075 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:04.751954+0000", 2024-09-10T19:14:09.075 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:14:09.075 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:14:09.075 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:14:09.075 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-10T19:14:09.075 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:04.751952+0000", 2024-09-10T19:14:09.076 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999981 2024-09-10T19:14:09.076 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:14:09.076 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:14:09.076 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-10T19:14:09.076 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:04.751960+0000", 2024-09-10T19:14:09.076 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.5229999999999998e-06 2024-09-10T19:14:09.076 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:14:09.076 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:14:09.076 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-10T19:14:09.076 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:04.751973+0000", 2024-09-10T19:14:09.076 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.3227999999999999e-05 2024-09-10T19:14:09.076 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:14:09.077 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:14:09.077 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-10T19:14:09.077 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:04.751978+0000", 2024-09-10T19:14:09.077 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.4840000000000003e-06 2024-09-10T19:14:09.077 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:14:09.077 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:14:09.077 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:14:09.077 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:04.752021+0000", 2024-09-10T19:14:09.077 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.2719000000000002e-05 2024-09-10T19:14:09.077 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:14:09.077 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:14:09.077 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-10T19:14:09.078 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:04.752154+0000", 2024-09-10T19:14:09.078 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00013276600000000001 2024-09-10T19:14:09.078 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:14:09.078 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:14:09.078 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 2,3", 2024-09-10T19:14:09.078 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:04.752267+0000", 2024-09-10T19:14:09.078 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00011303600000000001 2024-09-10T19:14:09.078 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:14:09.078 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:14:09.078 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-09-10T19:14:09.078 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:04.758506+0000", 2024-09-10T19:14:09.078 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0062390659999999997 2024-09-10T19:14:09.078 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:14:09.079 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:14:09.079 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:14:09.079 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:04.761065+0000", 2024-09-10T19:14:09.079 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0025590230000000001 2024-09-10T19:14:09.079 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:14:09.079 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:14:09.079 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:14:09.079 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:04.818566+0000", 2024-09-10T19:14:09.079 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.057501297 2024-09-10T19:14:09.079 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:14:09.079 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:14:09.079 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-09-10T19:14:09.079 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:04.818616+0000", 2024-09-10T19:14:09.080 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.9894e-05 2024-09-10T19:14:09.080 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:14:09.080 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:14:09.080 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-10T19:14:09.080 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:04.818721+0000", 2024-09-10T19:14:09.080 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000104666 2024-09-10T19:14:09.080 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:14:09.080 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-10T19:14:09.080 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:14:09.080 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:14:09.080 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:14:09.080 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.14781.0:157920 7.0 7:08ffa193:::benchmark_data_smithi183_84160_object17539:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e118)", 2024-09-10T19:14:09.080 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-10T19:14:04.753572+0000", 2024-09-10T19:14:09.081 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 4.2460193540000004, 2024-09-10T19:14:09.081 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.065320845000000002, 2024-09-10T19:14:09.081 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-10T19:14:09.081 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T19:14:09.081 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-10T19:14:09.081 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.14781", 2024-09-10T19:14:09.081 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.183:0/2025776420", 2024-09-10T19:14:09.081 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 157920 2024-09-10T19:14:09.081 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:14:09.081 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-10T19:14:09.081 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:14:09.081 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-10T19:14:09.081 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:04.753572+0000", 2024-09-10T19:14:09.082 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:14:09.082 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:14:09.082 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:14:09.082 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-10T19:14:09.082 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:04.753572+0000", 2024-09-10T19:14:09.082 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:14:09.082 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:14:09.082 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:14:09.082 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-10T19:14:09.082 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:04.753571+0000", 2024-09-10T19:14:09.082 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999986 2024-09-10T19:14:09.082 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:14:09.082 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:14:09.083 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-10T19:14:09.083 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:04.753580+0000", 2024-09-10T19:14:09.083 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.1749999999999994e-06 2024-09-10T19:14:09.083 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:14:09.083 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:14:09.083 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-10T19:14:09.083 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:04.753594+0000", 2024-09-10T19:14:09.083 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.3563999999999999e-05 2024-09-10T19:14:09.083 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:14:09.083 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:14:09.083 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-10T19:14:09.083 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:04.753600+0000", 2024-09-10T19:14:09.083 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.3110000000000002e-06 2024-09-10T19:14:09.084 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:14:09.084 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:14:09.084 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:14:09.084 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:04.755729+0000", 2024-09-10T19:14:09.084 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0021287189999999998 2024-09-10T19:14:09.084 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:14:09.084 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:14:09.084 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-10T19:14:09.084 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:04.755865+0000", 2024-09-10T19:14:09.084 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00013623199999999999 2024-09-10T19:14:09.084 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:14:09.084 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:14:09.084 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 2,3", 2024-09-10T19:14:09.085 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:04.755981+0000", 2024-09-10T19:14:09.085 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000116344 2024-09-10T19:14:09.085 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:14:09.085 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:14:09.085 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-09-10T19:14:09.085 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:04.758513+0000", 2024-09-10T19:14:09.085 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0025321649999999998 2024-09-10T19:14:09.085 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:14:09.085 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:14:09.085 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:14:09.085 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:04.762927+0000", 2024-09-10T19:14:09.085 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0044132900000000003 2024-09-10T19:14:09.085 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:14:09.086 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:14:09.086 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:14:09.086 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:04.818764+0000", 2024-09-10T19:14:09.086 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.055837422999999997 2024-09-10T19:14:09.086 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:14:09.086 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:14:09.086 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-09-10T19:14:09.086 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:04.818813+0000", 2024-09-10T19:14:09.086 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.9236000000000001e-05 2024-09-10T19:14:09.086 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:14:09.086 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:14:09.086 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-10T19:14:09.086 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:04.818893+0000", 2024-09-10T19:14:09.087 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.9682e-05 2024-09-10T19:14:09.087 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:14:09.087 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-10T19:14:09.087 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:14:09.087 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:14:09.087 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:14:09.087 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.14781.0:159143 7.0 7:0b11bd49:::benchmark_data_smithi183_84160_object18762:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119)", 2024-09-10T19:14:09.087 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-10T19:14:05.772272+0000", 2024-09-10T19:14:09.087 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 3.2273197229999999, 2024-09-10T19:14:09.087 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.138067563, 2024-09-10T19:14:09.087 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-10T19:14:09.087 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T19:14:09.087 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-10T19:14:09.087 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.14781", 2024-09-10T19:14:09.088 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.183:0/2025776420", 2024-09-10T19:14:09.088 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 159143 2024-09-10T19:14:09.088 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:14:09.088 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-10T19:14:09.088 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:14:09.088 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-10T19:14:09.088 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:05.772272+0000", 2024-09-10T19:14:09.088 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:14:09.088 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:14:09.089 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:14:09.089 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-10T19:14:09.089 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:05.772272+0000", 2024-09-10T19:14:09.089 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:14:09.089 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:14:09.089 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:14:09.089 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-10T19:14:09.089 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:05.772270+0000", 2024-09-10T19:14:09.089 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999986 2024-09-10T19:14:09.089 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:14:09.089 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:14:09.089 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-10T19:14:09.089 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:05.772277+0000", 2024-09-10T19:14:09.089 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.8210000000000004e-06 2024-09-10T19:14:09.090 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:14:09.090 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:14:09.090 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-10T19:14:09.090 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:05.772291+0000", 2024-09-10T19:14:09.090 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.3824e-05 2024-09-10T19:14:09.090 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:14:09.090 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:14:09.090 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-10T19:14:09.090 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:05.772297+0000", 2024-09-10T19:14:09.090 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.3350000000000003e-06 2024-09-10T19:14:09.090 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:14:09.090 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:14:09.090 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:14:09.091 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:05.772344+0000", 2024-09-10T19:14:09.091 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.6835000000000001e-05 2024-09-10T19:14:09.091 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:14:09.091 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:14:09.091 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-10T19:14:09.091 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:05.772479+0000", 2024-09-10T19:14:09.091 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000134723 2024-09-10T19:14:09.091 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:14:09.091 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:14:09.091 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 2,3", 2024-09-10T19:14:09.091 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:05.772629+0000", 2024-09-10T19:14:09.091 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00015009100000000001 2024-09-10T19:14:09.092 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:14:09.092 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:14:09.092 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-09-10T19:14:09.092 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:05.777906+0000", 2024-09-10T19:14:09.092 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0052770350000000002 2024-09-10T19:14:09.092 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:14:09.092 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:14:09.092 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:14:09.092 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:05.777939+0000", 2024-09-10T19:14:09.092 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.3529999999999999e-05 2024-09-10T19:14:09.092 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:14:09.092 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:14:09.092 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:14:09.093 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:05.910239+0000", 2024-09-10T19:14:09.093 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.132299735 2024-09-10T19:14:09.093 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:14:09.093 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:14:09.093 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-09-10T19:14:09.093 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:05.910278+0000", 2024-09-10T19:14:09.093 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.8813999999999998e-05 2024-09-10T19:14:09.093 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:14:09.093 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:14:09.093 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-10T19:14:09.093 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:05.910339+0000", 2024-09-10T19:14:09.093 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.1340999999999994e-05 2024-09-10T19:14:09.093 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:14:09.094 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-10T19:14:09.094 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:14:09.094 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:14:09.094 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:14:09.094 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.14781.0:159145 7.0 7:04417748:::benchmark_data_smithi183_84160_object18764:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119)", 2024-09-10T19:14:09.094 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-10T19:14:05.774278+0000", 2024-09-10T19:14:09.094 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 3.2253133630000002, 2024-09-10T19:14:09.094 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.13617543700000001, 2024-09-10T19:14:09.094 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-10T19:14:09.094 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T19:14:09.094 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-10T19:14:09.095 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.14781", 2024-09-10T19:14:09.095 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.183:0/2025776420", 2024-09-10T19:14:09.095 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 159145 2024-09-10T19:14:09.095 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:14:09.095 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-10T19:14:09.095 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:14:09.095 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-10T19:14:09.095 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:05.774278+0000", 2024-09-10T19:14:09.095 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:14:09.095 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:14:09.096 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:14:09.096 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-10T19:14:09.096 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:05.774278+0000", 2024-09-10T19:14:09.096 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:14:09.096 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:14:09.096 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:14:09.096 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-10T19:14:09.096 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:05.774277+0000", 2024-09-10T19:14:09.096 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.999999 2024-09-10T19:14:09.096 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:14:09.096 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:14:09.097 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-10T19:14:09.097 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:05.774282+0000", 2024-09-10T19:14:09.097 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.0309999999999998e-06 2024-09-10T19:14:09.097 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:14:09.097 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:14:09.097 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-10T19:14:09.097 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:05.774294+0000", 2024-09-10T19:14:09.097 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.2035000000000001e-05 2024-09-10T19:14:09.097 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:14:09.097 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:14:09.097 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-10T19:14:09.097 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:05.774300+0000", 2024-09-10T19:14:09.098 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.4330000000000003e-06 2024-09-10T19:14:09.098 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:14:09.098 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:14:09.098 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:14:09.098 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:05.775106+0000", 2024-09-10T19:14:09.098 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00080659199999999996 2024-09-10T19:14:09.098 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:14:09.098 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:14:09.098 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-10T19:14:09.098 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:05.775197+0000", 2024-09-10T19:14:09.098 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.0838e-05 2024-09-10T19:14:09.098 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:14:09.098 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:14:09.099 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 2,3", 2024-09-10T19:14:09.099 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:05.775512+0000", 2024-09-10T19:14:09.099 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00031528900000000001 2024-09-10T19:14:09.099 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:14:09.099 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:14:09.099 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-09-10T19:14:09.099 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:05.778246+0000", 2024-09-10T19:14:09.099 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0027339590000000002 2024-09-10T19:14:09.099 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:14:09.099 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:14:09.099 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:14:09.099 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:05.779965+0000", 2024-09-10T19:14:09.099 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.001718266 2024-09-10T19:14:09.099 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:14:09.100 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:14:09.100 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:14:09.100 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:05.910373+0000", 2024-09-10T19:14:09.100 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.13040834300000001 2024-09-10T19:14:09.100 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:14:09.100 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:14:09.100 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-09-10T19:14:09.100 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:05.910404+0000", 2024-09-10T19:14:09.100 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.1334000000000002e-05 2024-09-10T19:14:09.100 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:14:09.100 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:14:09.100 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-10T19:14:09.100 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:05.910454+0000", 2024-09-10T19:14:09.101 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.9230999999999997e-05 2024-09-10T19:14:09.101 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:14:09.101 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-10T19:14:09.101 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:14:09.101 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:14:09.101 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:14:09.101 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.14781.0:159513 7.0 7:0dd6399d:::benchmark_data_smithi183_84160_object19132:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119)", 2024-09-10T19:14:09.101 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-10T19:14:06.165818+0000", 2024-09-10T19:14:09.101 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 2.8337730379999999, 2024-09-10T19:14:09.101 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.089520774999999997, 2024-09-10T19:14:09.101 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-10T19:14:09.101 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T19:14:09.102 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-10T19:14:09.102 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.14781", 2024-09-10T19:14:09.102 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.183:0/2025776420", 2024-09-10T19:14:09.102 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 159513 2024-09-10T19:14:09.102 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:14:09.102 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-10T19:14:09.102 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:14:09.102 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-10T19:14:09.102 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:06.165818+0000", 2024-09-10T19:14:09.102 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:14:09.102 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:14:09.102 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:14:09.103 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-10T19:14:09.103 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:06.165818+0000", 2024-09-10T19:14:09.103 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:14:09.103 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:14:09.103 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:14:09.103 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-10T19:14:09.103 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:06.165817+0000", 2024-09-10T19:14:09.103 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999986 2024-09-10T19:14:09.103 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:14:09.103 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:14:09.103 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-10T19:14:09.103 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:06.165824+0000", 2024-09-10T19:14:09.103 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.4919999999999996e-06 2024-09-10T19:14:09.104 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:14:09.104 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:14:09.104 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-10T19:14:09.104 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:06.165840+0000", 2024-09-10T19:14:09.104 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.5092999999999999e-05 2024-09-10T19:14:09.104 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:14:09.104 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:14:09.104 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-10T19:14:09.104 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:06.165845+0000", 2024-09-10T19:14:09.104 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.6300000000000003e-06 2024-09-10T19:14:09.104 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:14:09.104 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:14:09.105 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:14:09.105 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:06.165910+0000", 2024-09-10T19:14:09.105 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.4429000000000003e-05 2024-09-10T19:14:09.105 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:14:09.105 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:14:09.105 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-10T19:14:09.105 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:06.166064+0000", 2024-09-10T19:14:09.105 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00015447099999999999 2024-09-10T19:14:09.105 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:14:09.105 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:14:09.105 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 2,3", 2024-09-10T19:14:09.105 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:06.166219+0000", 2024-09-10T19:14:09.106 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00015475499999999999 2024-09-10T19:14:09.106 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:14:09.106 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:14:09.106 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-09-10T19:14:09.106 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:06.168892+0000", 2024-09-10T19:14:09.106 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0026733569999999999 2024-09-10T19:14:09.106 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:14:09.106 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:14:09.106 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:14:09.106 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:06.169320+0000", 2024-09-10T19:14:09.106 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00042788599999999999 2024-09-10T19:14:09.106 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:14:09.107 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:14:09.107 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:14:09.107 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:06.255229+0000", 2024-09-10T19:14:09.107 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.085908957999999994 2024-09-10T19:14:09.107 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:14:09.107 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:14:09.107 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-09-10T19:14:09.107 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:06.255273+0000", 2024-09-10T19:14:09.107 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.4209999999999999e-05 2024-09-10T19:14:09.107 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:14:09.107 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:14:09.107 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-10T19:14:09.108 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:06.255339+0000", 2024-09-10T19:14:09.108 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.5953999999999999e-05 2024-09-10T19:14:09.108 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:14:09.108 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-10T19:14:09.108 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:14:09.108 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:14:09.108 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:14:09.108 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.14781.0:161681 7.0 7:0fcfee5d:::benchmark_data_smithi183_84160_object21300:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120)", 2024-09-10T19:14:09.108 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-10T19:14:08.094534+0000", 2024-09-10T19:14:09.108 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 0.90505743400000005, 2024-09-10T19:14:09.108 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.15055376300000001, 2024-09-10T19:14:09.108 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-10T19:14:09.108 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T19:14:09.108 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-10T19:14:09.109 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.14781", 2024-09-10T19:14:09.109 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.183:0/2025776420", 2024-09-10T19:14:09.109 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 161681 2024-09-10T19:14:09.109 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:14:09.109 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-10T19:14:09.109 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:14:09.110 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-10T19:14:09.110 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:08.094534+0000", 2024-09-10T19:14:09.110 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:14:09.110 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:14:09.110 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:14:09.110 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-10T19:14:09.110 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:08.094534+0000", 2024-09-10T19:14:09.110 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:14:09.110 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:14:09.110 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:14:09.110 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-10T19:14:09.111 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:08.094533+0000", 2024-09-10T19:14:09.111 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999986 2024-09-10T19:14:09.111 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:14:09.111 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:14:09.111 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-10T19:14:09.111 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:08.094541+0000", 2024-09-10T19:14:09.111 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.6740000000000003e-06 2024-09-10T19:14:09.111 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:14:09.111 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:14:09.111 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-10T19:14:09.111 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:08.094570+0000", 2024-09-10T19:14:09.111 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.8263999999999999e-05 2024-09-10T19:14:09.111 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:14:09.112 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:14:09.112 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-10T19:14:09.112 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:08.094577+0000", 2024-09-10T19:14:09.112 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.8620000000000003e-06 2024-09-10T19:14:09.112 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:14:09.112 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:14:09.112 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:14:09.112 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:08.094630+0000", 2024-09-10T19:14:09.112 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.2346999999999999e-05 2024-09-10T19:14:09.112 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:14:09.112 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:14:09.112 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-10T19:14:09.112 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:08.094784+0000", 2024-09-10T19:14:09.113 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00015472700000000001 2024-09-10T19:14:09.113 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:14:09.113 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:14:09.113 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 2,3", 2024-09-10T19:14:09.113 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:08.094939+0000", 2024-09-10T19:14:09.113 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00015455300000000001 2024-09-10T19:14:09.113 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:14:09.113 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:14:09.113 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-09-10T19:14:09.113 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:08.104366+0000", 2024-09-10T19:14:09.113 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0094271379999999998 2024-09-10T19:14:09.113 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:14:09.113 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:14:09.113 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:14:09.114 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:08.104401+0000", 2024-09-10T19:14:09.114 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.5089000000000002e-05 2024-09-10T19:14:09.114 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:14:09.114 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:14:09.114 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:14:09.114 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:08.244881+0000", 2024-09-10T19:14:09.114 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.14047984699999999 2024-09-10T19:14:09.114 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:14:09.114 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:14:09.114 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-09-10T19:14:09.114 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:08.244966+0000", 2024-09-10T19:14:09.114 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.4843000000000001e-05 2024-09-10T19:14:09.114 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:14:09.115 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:14:09.115 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-10T19:14:09.115 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:08.245088+0000", 2024-09-10T19:14:09.115 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000121877 2024-09-10T19:14:09.115 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:14:09.115 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-10T19:14:09.115 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:14:09.115 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:14:09.115 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:14:09.115 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.14781.0:161683 7.0 7:085c0f2f:::benchmark_data_smithi183_84160_object21302:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120)", 2024-09-10T19:14:09.115 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-10T19:14:08.095469+0000", 2024-09-10T19:14:09.115 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 0.90412282899999996, 2024-09-10T19:14:09.115 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.159833796, 2024-09-10T19:14:09.115 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-10T19:14:09.116 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T19:14:09.116 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-10T19:14:09.116 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.14781", 2024-09-10T19:14:09.116 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.183:0/2025776420", 2024-09-10T19:14:09.116 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 161683 2024-09-10T19:14:09.116 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:14:09.116 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-10T19:14:09.116 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:14:09.116 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-10T19:14:09.116 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:08.095469+0000", 2024-09-10T19:14:09.116 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:14:09.116 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:14:09.117 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:14:09.117 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-10T19:14:09.117 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:08.095469+0000", 2024-09-10T19:14:09.117 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:14:09.117 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:14:09.117 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:14:09.117 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-10T19:14:09.117 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:08.095468+0000", 2024-09-10T19:14:09.117 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.999999 2024-09-10T19:14:09.117 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:14:09.117 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:14:09.117 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-10T19:14:09.117 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:08.095473+0000", 2024-09-10T19:14:09.117 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.062e-06 2024-09-10T19:14:09.118 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:14:09.118 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:14:09.118 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-10T19:14:09.118 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:08.095483+0000", 2024-09-10T19:14:09.118 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.0701e-05 2024-09-10T19:14:09.118 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:14:09.118 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:14:09.118 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-10T19:14:09.118 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:08.095489+0000", 2024-09-10T19:14:09.118 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.4199999999999998e-06 2024-09-10T19:14:09.118 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:14:09.118 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:14:09.118 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:14:09.119 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:08.097769+0000", 2024-09-10T19:14:09.119 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0022801610000000002 2024-09-10T19:14:09.119 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:14:09.119 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:14:09.119 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-10T19:14:09.119 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:08.097884+0000", 2024-09-10T19:14:09.119 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000114925 2024-09-10T19:14:09.119 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:14:09.119 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:14:09.119 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 2,3", 2024-09-10T19:14:09.119 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:08.098123+0000", 2024-09-10T19:14:09.119 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00023907900000000001 2024-09-10T19:14:09.119 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:14:09.119 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:14:09.120 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:14:09.120 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:08.104426+0000", 2024-09-10T19:14:09.120 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0063033960000000002 2024-09-10T19:14:09.120 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:14:09.120 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:14:09.120 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-09-10T19:14:09.120 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:08.104430+0000", 2024-09-10T19:14:09.120 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.9149999999999998e-06 2024-09-10T19:14:09.120 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:14:09.120 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:14:09.120 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:14:09.120 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:08.255129+0000", 2024-09-10T19:14:09.120 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.15069867300000001 2024-09-10T19:14:09.121 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:14:09.121 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:14:09.121 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-09-10T19:14:09.121 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:08.255209+0000", 2024-09-10T19:14:09.121 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.9954000000000001e-05 2024-09-10T19:14:09.121 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:14:09.121 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:14:09.121 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-10T19:14:09.121 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:08.255302+0000", 2024-09-10T19:14:09.121 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.3441000000000002e-05 2024-09-10T19:14:09.121 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:14:09.121 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-10T19:14:09.121 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:14:09.121 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:14:09.122 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:14:09.122 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.14781.0:161685 7.0 7:07c220a0:::benchmark_data_smithi183_84160_object21304:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120)", 2024-09-10T19:14:09.122 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-10T19:14:08.097502+0000", 2024-09-10T19:14:09.122 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 0.90208957999999995, 2024-09-10T19:14:09.122 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.15798367699999999, 2024-09-10T19:14:09.122 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-10T19:14:09.122 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T19:14:09.122 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-10T19:14:09.122 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.14781", 2024-09-10T19:14:09.122 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.183:0/2025776420", 2024-09-10T19:14:09.122 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 161685 2024-09-10T19:14:09.122 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:14:09.122 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-10T19:14:09.123 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:14:09.123 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-10T19:14:09.123 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:08.097502+0000", 2024-09-10T19:14:09.123 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:14:09.123 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:14:09.123 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:14:09.123 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-10T19:14:09.123 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:08.097502+0000", 2024-09-10T19:14:09.123 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:14:09.123 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:14:09.123 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:14:09.123 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-10T19:14:09.123 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:08.097501+0000", 2024-09-10T19:14:09.123 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999986 2024-09-10T19:14:09.124 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:14:09.124 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:14:09.124 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-10T19:14:09.124 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:08.097507+0000", 2024-09-10T19:14:09.124 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.241e-06 2024-09-10T19:14:09.124 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:14:09.124 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:14:09.124 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-10T19:14:09.124 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:08.097522+0000", 2024-09-10T19:14:09.124 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.4732999999999999e-05 2024-09-10T19:14:09.124 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:14:09.124 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:14:09.125 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-10T19:14:09.125 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:08.097528+0000", 2024-09-10T19:14:09.125 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.2709999999999998e-06 2024-09-10T19:14:09.125 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:14:09.125 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:14:09.125 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:14:09.125 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:08.100705+0000", 2024-09-10T19:14:09.125 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0031766519999999999 2024-09-10T19:14:09.125 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:14:09.125 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:14:09.125 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-10T19:14:09.126 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:08.100839+0000", 2024-09-10T19:14:09.126 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000134031 2024-09-10T19:14:09.126 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:14:09.126 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:14:09.126 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 2,3", 2024-09-10T19:14:09.126 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:08.100969+0000", 2024-09-10T19:14:09.126 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00013078699999999999 2024-09-10T19:14:09.126 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:14:09.126 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:14:09.126 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-09-10T19:14:09.126 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:08.104434+0000", 2024-09-10T19:14:09.126 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0034643619999999999 2024-09-10T19:14:09.126 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:14:09.126 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:14:09.126 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:14:09.127 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:08.233896+0000", 2024-09-10T19:14:09.127 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.12946259600000001 2024-09-10T19:14:09.127 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:14:09.127 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:14:09.127 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:14:09.127 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:08.255335+0000", 2024-09-10T19:14:09.127 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.021438167000000001 2024-09-10T19:14:09.127 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:14:09.127 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:14:09.127 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-09-10T19:14:09.127 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:08.255401+0000", 2024-09-10T19:14:09.127 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.6044999999999995e-05 2024-09-10T19:14:09.127 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:14:09.128 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:14:09.128 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-10T19:14:09.128 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:08.255486+0000", 2024-09-10T19:14:09.128 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.5014999999999998e-05 2024-09-10T19:14:09.128 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:14:09.128 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-10T19:14:09.128 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:14:09.128 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:14:09.128 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:14:09.128 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.14781.0:161686 7.0 7:0d68007f:::benchmark_data_smithi183_84160_object21305:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120)", 2024-09-10T19:14:09.128 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-10T19:14:08.097541+0000", 2024-09-10T19:14:09.128 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 0.90205066899999997, 2024-09-10T19:14:09.128 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.15814488099999999, 2024-09-10T19:14:09.128 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-10T19:14:09.128 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T19:14:09.129 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-10T19:14:09.129 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.14781", 2024-09-10T19:14:09.129 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.183:0/2025776420", 2024-09-10T19:14:09.129 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 161686 2024-09-10T19:14:09.129 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:14:09.129 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-10T19:14:09.129 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:14:09.129 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-10T19:14:09.129 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:08.097541+0000", 2024-09-10T19:14:09.129 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:14:09.129 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:14:09.129 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:14:09.129 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-10T19:14:09.130 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:08.097541+0000", 2024-09-10T19:14:09.130 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:14:09.130 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:14:09.130 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:14:09.130 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-10T19:14:09.130 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:08.097540+0000", 2024-09-10T19:14:09.130 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999995 2024-09-10T19:14:09.130 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:14:09.130 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:14:09.130 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-10T19:14:09.130 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:08.097544+0000", 2024-09-10T19:14:09.130 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.225e-06 2024-09-10T19:14:09.130 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:14:09.130 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:14:09.131 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-10T19:14:09.131 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:08.097551+0000", 2024-09-10T19:14:09.131 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.1420000000000004e-06 2024-09-10T19:14:09.131 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:14:09.131 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:14:09.131 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-10T19:14:09.131 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:08.097555+0000", 2024-09-10T19:14:09.131 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.1590000000000003e-06 2024-09-10T19:14:09.131 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:14:09.131 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:14:09.131 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:14:09.131 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:08.103202+0000", 2024-09-10T19:14:09.131 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0056468480000000003 2024-09-10T19:14:09.131 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:14:09.131 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:14:09.132 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-10T19:14:09.132 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:08.103286+0000", 2024-09-10T19:14:09.132 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.4546e-05 2024-09-10T19:14:09.132 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:14:09.132 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:14:09.132 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 2,3", 2024-09-10T19:14:09.132 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:08.103365+0000", 2024-09-10T19:14:09.132 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.8419999999999998e-05 2024-09-10T19:14:09.132 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:14:09.132 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:14:09.132 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-09-10T19:14:09.132 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:08.104486+0000", 2024-09-10T19:14:09.132 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.001121389 2024-09-10T19:14:09.133 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:14:09.133 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:14:09.133 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:14:09.133 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:08.239595+0000", 2024-09-10T19:14:09.133 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.13510910500000001 2024-09-10T19:14:09.133 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:14:09.133 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:14:09.133 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:14:09.133 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:08.255532+0000", 2024-09-10T19:14:09.133 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.015937182000000001 2024-09-10T19:14:09.133 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:14:09.133 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:14:09.133 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-09-10T19:14:09.133 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:08.255597+0000", 2024-09-10T19:14:09.134 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.5009e-05 2024-09-10T19:14:09.134 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:14:09.134 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:14:09.134 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-10T19:14:09.134 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:08.255686+0000", 2024-09-10T19:14:09.134 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.8376999999999998e-05 2024-09-10T19:14:09.134 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:14:09.134 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-10T19:14:09.134 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:14:09.134 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:14:09.134 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-10T19:14:09.134 INFO:teuthology.orchestra.run.smithi191.stdout:} 2024-09-10T19:14:09.141 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:14:09.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:14:09.142 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:14:09.242 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:14:09.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:14:09.278 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:14:09.279 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:14:09.379 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:14:09.415 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:14:09.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:14:09.416 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:14:09.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:14:09 smithi183 ceph-mon[29736]: pgmap v152: 148 pgs: 3 active+undersized+degraded+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 142 active+clean; 7.5 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.13k op/s; 42686/369351 objects degraded (11.557%); 23 MiB/s, 376 objects/s recovering 2024-09-10T19:14:09.431 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:14:09 smithi183 ceph-mon[41562]: pgmap v152: 148 pgs: 3 active+undersized+degraded+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 142 active+clean; 7.5 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.13k op/s; 42686/369351 objects degraded (11.557%); 23 MiB/s, 376 objects/s recovering 2024-09-10T19:14:09.433 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:14:09 smithi191 ceph-mon[32884]: pgmap v152: 148 pgs: 3 active+undersized+degraded+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 142 active+clean; 7.5 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.13k op/s; 42686/369351 objects degraded (11.557%); 23 MiB/s, 376 objects/s recovering 2024-09-10T19:14:09.433 INFO:journalctl@ceph.iscsi.iscsi.a.smithi191.stdout:Sep 10 19:14:09 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-iscsi-iscsi-a[61816]: debug there is no tcmu-runner data available 2024-09-10T19:14:09.517 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:14:09.524 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:14:09.546 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:14:09.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:14:09.547 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:14:09.648 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:14:09.679 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:14:09.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:14:09.681 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:14:09.782 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:14:09.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:14:09.837 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:14:09.837 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:14:09.938 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:14:09.964 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:14:09.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:14:09.965 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:14:10.066 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:14:10.104 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:14:10.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:14:10.105 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:14:10.159 DEBUG:teuthology.orchestra.run.smithi191:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph daemon osd.6 dump_ops_in_flight 2024-09-10T19:14:10.206 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:14:10.240 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:14:10.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:14:10.241 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:14:10.341 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:14:10.380 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:14:10.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:14:10.381 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:14:10.398 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:14:10 smithi183 ceph-mon[29736]: purged_snaps scrub starts 2024-09-10T19:14:10.398 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:14:10 smithi183 ceph-mon[29736]: purged_snaps scrub ok 2024-09-10T19:14:10.399 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:14:10 smithi183 ceph-mon[29736]: from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:14:10.399 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:14:10 smithi183 ceph-mon[29736]: osdmap e121: 8 total, 7 up, 7 in 2024-09-10T19:14:10.399 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:14:10 smithi183 ceph-mon[29736]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 3 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-09-10T19:14:10.399 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:14:10 smithi183 ceph-mon[29736]: Health check update: Degraded data redundancy: 42686/369351 objects degraded (11.557%), 6 pgs degraded (PG_DEGRADED) 2024-09-10T19:14:10.400 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:14:10 smithi183 ceph-mon[41562]: purged_snaps scrub starts 2024-09-10T19:14:10.401 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:14:10 smithi183 ceph-mon[41562]: purged_snaps scrub ok 2024-09-10T19:14:10.401 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:14:10 smithi183 ceph-mon[41562]: from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:14:10.401 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:14:10 smithi183 ceph-mon[41562]: osdmap e121: 8 total, 7 up, 7 in 2024-09-10T19:14:10.401 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:14:10 smithi183 ceph-mon[41562]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 3 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-09-10T19:14:10.401 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:14:10 smithi183 ceph-mon[41562]: Health check update: Degraded data redundancy: 42686/369351 objects degraded (11.557%), 6 pgs degraded (PG_DEGRADED) 2024-09-10T19:14:10.465 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:14:10 smithi191 ceph-mon[32884]: purged_snaps scrub starts 2024-09-10T19:14:10.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:14:10 smithi191 ceph-mon[32884]: purged_snaps scrub ok 2024-09-10T19:14:10.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:14:10 smithi191 ceph-mon[32884]: from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:14:10.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:14:10 smithi191 ceph-mon[32884]: osdmap e121: 8 total, 7 up, 7 in 2024-09-10T19:14:10.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:14:10 smithi191 ceph-mon[32884]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 3 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-09-10T19:14:10.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:14:10 smithi191 ceph-mon[32884]: Health check update: Degraded data redundancy: 42686/369351 objects degraded (11.557%), 6 pgs degraded (PG_DEGRADED) 2024-09-10T19:14:10.482 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:14:10.524 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:14:10.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:14:10.525 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:14:10.555 INFO:teuthology.orchestra.run.smithi191.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.b/config 2024-09-10T19:14:10.626 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:14:10.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:14:10.667 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:14:10.667 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:14:10.768 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:14:10.800 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:14:10.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:14:10.801 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:14:10.892 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-09-10T19:14:10.895 INFO:teuthology.orchestra.run.smithi183.stderr:dumped all 2024-09-10T19:14:10.902 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:14:10.941 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:14:10.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:14:10.942 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:14:11.043 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:14:11.082 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:14:11.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:14:11.083 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:14:11.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:14:11 smithi183 ceph-mon[29736]: pgmap v154: 148 pgs: 3 active+undersized+degraded+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 142 active+clean; 7.4 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 1.38k op/s; 41014/362877 objects degraded (11.302%); 28 MiB/s, 446 objects/s recovering 2024-09-10T19:14:11.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:14:11 smithi183 ceph-mon[41562]: pgmap v154: 148 pgs: 3 active+undersized+degraded+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 142 active+clean; 7.4 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 1.38k op/s; 41014/362877 objects degraded (11.302%); 28 MiB/s, 446 objects/s recovering 2024-09-10T19:14:11.184 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:14:11.212 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:14:11.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:14:11.213 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:14:11.313 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:14:11.347 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:14:11.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:14:11.348 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:14:11.449 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:14:11.465 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:14:11 smithi191 ceph-mon[32884]: pgmap v154: 148 pgs: 3 active+undersized+degraded+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 142 active+clean; 7.4 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 1.38k op/s; 41014/362877 objects degraded (11.302%); 28 MiB/s, 446 objects/s recovering 2024-09-10T19:14:11.480 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:14:11.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:14:11.481 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:14:11.516 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-10T19:14:11.557 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Sep 10 19:14:11 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-y[29989]: ::ffff:172.21.15.191 - - [10/Sep/2024:19:14:11] "GET /metrics HTTP/1.1" 200 208860 "" "Prometheus/2.43.0" 2024-09-10T19:14:11.582 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:14:11.610 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:14:11.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:14:11.611 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:14:11.712 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:14:11.733 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:14:11.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:14:11.734 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:14:11.834 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:14:11.864 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:14:11.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:14:11.865 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:14:11.966 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:14:12.001 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:14:12.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:14:12.003 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:14:12.103 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:14:12.133 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:14:12.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:14:12.134 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:14:12.234 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:14:12.266 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:14:12.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:14:12.267 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:14:12.338 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:14:12 smithi183 ceph-mon[29736]: from='client.24767 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-10T19:14:12.340 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:14:12 smithi183 ceph-mon[41562]: from='client.24767 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-10T19:14:12.368 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:14:12.403 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:14:12.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:14:12.404 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:14:12.465 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:14:12 smithi191 ceph-mon[32884]: from='client.24767 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-10T19:14:12.505 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:14:12.534 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:14:12.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:14:12.535 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:14:12.636 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:14:12.661 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:14:12.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:14:12.662 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:14:12.763 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:14:12.799 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:14:12.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:14:12.800 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:14:12.901 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:14:12.930 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:14:12.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:14:12.931 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:14:13.013 INFO:teuthology.orchestra.run.smithi183.stdout:{"pg_ready":true,"pg_map":{"version":154,"stamp":"2024-09-10T19:14:09.661589+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":7914854644,"num_objects":120959,"num_object_clones":0,"num_object_copies":362877,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":41014,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":120959,"num_whiteouts":0,"num_read":929,"num_read_kb":786,"num_write":160538,"num_write_kb":8985864,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4314,"num_bytes_recovered":275516725,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":77085,"ondisk_log_size":77085,"up":444,"acting":438,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":444,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":656216064,"kb_used":22829672,"kb_used_data":20685792,"kb_used_omap":11,"kb_used_meta":2143796,"kb_avail":633386392,"statfs":{"total":671965249536,"available":648587665408,"internally_reserved":0,"allocated":21182251008,"data_stored":21179812389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":11928,"internal_metadata":2195247464},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,5,10,11,4,3],"upper_bound":64},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-542507008,"num_objects":-8278,"num_object_clones":0,"num_object_copies":-24834,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-5850,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-8278,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":8278,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2680,"num_bytes_recovered":175636480,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001495"},"pg_stats":[{"pgid":"6.1b","version":"0'0","reported_seq":82,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-09-10T19:13:55.710246+0000","last_change":"2024-09-10T19:13:08.710758+0000","last_active":"2024-09-10T19:13:55.710246+0000","last_peered":"2024-09-10T19:13:55.710246+0000","last_clean":"2024-09-10T19:13:55.710246+0000","last_became_active":"2024-09-10T19:10:14.056187+0000","last_became_peered":"2024-09-10T19:10:14.056187+0000","last_unstale":"2024-09-10T19:13:55.710246+0000","last_undegraded":"2024-09-10T19:13:55.710246+0000","last_fullsized":"2024-09-10T19:13:55.710246+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:13:08.710679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:13:08.710679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:14:14.543504+0000","scrub_duration":0.0027252499999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":98,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-09-10T19:13:55.712025+0000","last_change":"2024-09-10T19:13:16.219506+0000","last_active":"2024-09-10T19:13:55.712025+0000","last_peered":"2024-09-10T19:13:55.712025+0000","last_clean":"2024-09-10T19:13:55.712025+0000","last_became_active":"2024-09-10T19:11:44.748696+0000","last_became_peered":"2024-09-10T19:11:44.748696+0000","last_unstale":"2024-09-10T19:13:55.712025+0000","last_undegraded":"2024-09-10T19:13:55.712025+0000","last_fullsized":"2024-09-10T19:13:55.712025+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:13:16.219364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:13:16.219364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:14:17.144458+0000","scrub_duration":0.023972094999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.19","version":"66'15","reported_seq":111,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-09-10T19:13:55.710082+0000","last_change":"2024-09-10T19:13:47.165943+0000","last_active":"2024-09-10T19:13:55.710082+0000","last_peered":"2024-09-10T19:13:55.710082+0000","last_clean":"2024-09-10T19:13:55.710082+0000","last_became_active":"2024-09-10T19:10:09.688432+0000","last_became_peered":"2024-09-10T19:10:09.688432+0000","last_unstale":"2024-09-10T19:13:55.710082+0000","last_undegraded":"2024-09-10T19:13:55.710082+0000","last_fullsized":"2024-09-10T19:13:55.710082+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'15","last_scrub_stamp":"2024-09-10T19:13:47.165848+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:13:47.165848+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:15:16.492257+0000","scrub_duration":0.004982841,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":93,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-09-10T19:14:07.012278+0000","last_change":"2024-09-10T19:13:54.719103+0000","last_active":"2024-09-10T19:14:07.012278+0000","last_peered":"2024-09-10T19:14:07.012278+0000","last_clean":"2024-09-10T19:14:07.012278+0000","last_became_active":"2024-09-10T19:13:54.714824+0000","last_became_peered":"2024-09-10T19:13:54.714824+0000","last_unstale":"2024-09-10T19:14:07.012278+0000","last_undegraded":"2024-09-10T19:14:07.012278+0000","last_fullsized":"2024-09-10T19:14:07.012278+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:11:23.086191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:11:23.086191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0016087969999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"59'1","reported_seq":96,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-09-10T19:13:55.711228+0000","last_change":"2024-09-10T19:12:47.704757+0000","last_active":"2024-09-10T19:13:55.711228+0000","last_peered":"2024-09-10T19:13:55.711228+0000","last_clean":"2024-09-10T19:13:55.711228+0000","last_became_active":"2024-09-10T19:10:07.686143+0000","last_became_peered":"2024-09-10T19:10:07.686143+0000","last_unstale":"2024-09-10T19:13:55.711228+0000","last_undegraded":"2024-09-10T19:13:55.711228+0000","last_fullsized":"2024-09-10T19:13:55.711228+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'1","last_scrub_stamp":"2024-09-10T19:12:47.704508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:12:47.704508+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.010519827000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":874,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"66'9","reported_seq":121,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-09-10T19:14:07.012722+0000","last_change":"2024-09-10T19:13:54.747869+0000","last_active":"2024-09-10T19:14:07.012722+0000","last_peered":"2024-09-10T19:14:07.012722+0000","last_clean":"2024-09-10T19:14:07.012722+0000","last_became_active":"2024-09-10T19:13:54.716117+0000","last_became_peered":"2024-09-10T19:13:54.716117+0000","last_unstale":"2024-09-10T19:14:07.012722+0000","last_undegraded":"2024-09-10T19:14:07.012722+0000","last_fullsized":"2024-09-10T19:14:07.012722+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'9","last_scrub_stamp":"2024-09-10T19:12:04.079264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:12:04.079264+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0015358819999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,3],"acting":[4,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":41,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-09-10T19:14:07.058220+0000","last_change":"2024-09-10T19:13:54.714747+0000","last_active":"2024-09-10T19:14:07.058220+0000","last_peered":"2024-09-10T19:14:07.058220+0000","last_clean":"2024-09-10T19:14:07.058220+0000","last_became_active":"2024-09-10T19:13:54.714365+0000","last_became_peered":"2024-09-10T19:13:54.714365+0000","last_unstale":"2024-09-10T19:14:07.058220+0000","last_undegraded":"2024-09-10T19:14:07.058220+0000","last_fullsized":"2024-09-10T19:14:07.058220+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:11:56.186365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:11:56.186365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":91,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-09-10T19:14:07.012515+0000","last_change":"2024-09-10T19:13:54.713823+0000","last_active":"2024-09-10T19:14:07.012515+0000","last_peered":"2024-09-10T19:14:07.012515+0000","last_clean":"2024-09-10T19:14:07.012515+0000","last_became_active":"2024-09-10T19:13:54.713538+0000","last_became_peered":"2024-09-10T19:13:54.713538+0000","last_unstale":"2024-09-10T19:14:07.012515+0000","last_undegraded":"2024-09-10T19:14:07.012515+0000","last_fullsized":"2024-09-10T19:14:07.012515+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:11:32.056254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:11:32.056254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0013010680000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,7],"acting":[4,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":88,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-09-10T19:13:55.804106+0000","last_change":"2024-09-10T19:12:49.527255+0000","last_active":"2024-09-10T19:13:55.804106+0000","last_peered":"2024-09-10T19:13:55.804106+0000","last_clean":"2024-09-10T19:13:55.804106+0000","last_became_active":"2024-09-10T19:10:07.690728+0000","last_became_peered":"2024-09-10T19:10:07.690728+0000","last_unstale":"2024-09-10T19:13:55.804106+0000","last_undegraded":"2024-09-10T19:13:55.804106+0000","last_fullsized":"2024-09-10T19:13:55.804106+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:12:49.527167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:12:49.527167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0040597050000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1b","version":"66'5","reported_seq":114,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-09-10T19:14:07.012652+0000","last_change":"2024-09-10T19:12:55.359526+0000","last_active":"2024-09-10T19:14:07.012652+0000","last_peered":"2024-09-10T19:14:07.012652+0000","last_clean":"2024-09-10T19:14:07.012652+0000","last_became_active":"2024-09-10T19:11:44.748296+0000","last_became_peered":"2024-09-10T19:11:44.748296+0000","last_unstale":"2024-09-10T19:14:07.012652+0000","last_undegraded":"2024-09-10T19:14:07.012652+0000","last_fullsized":"2024-09-10T19:14:07.012652+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'5","last_scrub_stamp":"2024-09-10T19:12:55.359379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:12:55.359379+0000","objects_scrubbed":2,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:14:11.267316+0000","scrub_duration":0.0050448070000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":91,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-09-10T19:13:55.879453+0000","last_change":"2024-09-10T19:13:54.720135+0000","last_active":"2024-09-10T19:13:55.879453+0000","last_peered":"2024-09-10T19:13:55.879453+0000","last_clean":"2024-09-10T19:13:55.879453+0000","last_became_active":"2024-09-10T19:13:54.719063+0000","last_became_peered":"2024-09-10T19:13:54.719063+0000","last_unstale":"2024-09-10T19:13:55.879453+0000","last_undegraded":"2024-09-10T19:13:55.879453+0000","last_fullsized":"2024-09-10T19:13:55.879453+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:11:43.142734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:11:43.142734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0013612330000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":92,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-09-10T19:14:07.058252+0000","last_change":"2024-09-10T19:13:54.723892+0000","last_active":"2024-09-10T19:14:07.058252+0000","last_peered":"2024-09-10T19:14:07.058252+0000","last_clean":"2024-09-10T19:14:07.058252+0000","last_became_active":"2024-09-10T19:13:54.723590+0000","last_became_peered":"2024-09-10T19:13:54.723590+0000","last_unstale":"2024-09-10T19:14:07.058252+0000","last_undegraded":"2024-09-10T19:14:07.058252+0000","last_fullsized":"2024-09-10T19:14:07.058252+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:11:45.411321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:11:45.411321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001536576,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":88,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-09-10T19:13:55.801743+0000","last_change":"2024-09-10T19:12:54.477202+0000","last_active":"2024-09-10T19:13:55.801743+0000","last_peered":"2024-09-10T19:13:55.801743+0000","last_clean":"2024-09-10T19:13:55.801743+0000","last_became_active":"2024-09-10T19:10:07.690174+0000","last_became_peered":"2024-09-10T19:10:07.690174+0000","last_unstale":"2024-09-10T19:13:55.801743+0000","last_undegraded":"2024-09-10T19:13:55.801743+0000","last_fullsized":"2024-09-10T19:13:55.801743+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:12:54.477149+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:12:54.477149+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:14:14.837613+0000","scrub_duration":0.0039768319999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1a","version":"66'9","reported_seq":104,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-09-10T19:14:07.013025+0000","last_change":"2024-09-10T19:13:02.431528+0000","last_active":"2024-09-10T19:14:07.013025+0000","last_peered":"2024-09-10T19:14:07.013025+0000","last_clean":"2024-09-10T19:14:07.013025+0000","last_became_active":"2024-09-10T19:10:09.688987+0000","last_became_peered":"2024-09-10T19:10:09.688987+0000","last_unstale":"2024-09-10T19:14:07.013025+0000","last_undegraded":"2024-09-10T19:14:07.013025+0000","last_fullsized":"2024-09-10T19:14:07.013025+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'9","last_scrub_stamp":"2024-09-10T19:13:02.431443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:13:02.431443+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:14:27.616203+0000","scrub_duration":0.0084913290000000006,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":87,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-09-10T19:14:07.060916+0000","last_change":"2024-09-10T19:12:51.449179+0000","last_active":"2024-09-10T19:14:07.060916+0000","last_peered":"2024-09-10T19:14:07.060916+0000","last_clean":"2024-09-10T19:14:07.060916+0000","last_became_active":"2024-09-10T19:10:11.701799+0000","last_became_peered":"2024-09-10T19:10:11.701799+0000","last_unstale":"2024-09-10T19:14:07.060916+0000","last_undegraded":"2024-09-10T19:14:07.060916+0000","last_fullsized":"2024-09-10T19:14:07.060916+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:12:51.449046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:12:51.449046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.002348826,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":89,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-09-10T19:13:55.708711+0000","last_change":"2024-09-10T19:13:54.719488+0000","last_active":"2024-09-10T19:13:55.708711+0000","last_peered":"2024-09-10T19:13:55.708711+0000","last_clean":"2024-09-10T19:13:55.708711+0000","last_became_active":"2024-09-10T19:13:54.718860+0000","last_became_peered":"2024-09-10T19:13:54.718860+0000","last_unstale":"2024-09-10T19:13:55.708711+0000","last_undegraded":"2024-09-10T19:13:55.708711+0000","last_fullsized":"2024-09-10T19:13:55.708711+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:11:12.970173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:11:12.970173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001527219,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.e","version":"118'10035","reported_seq":10437,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-09-10T19:14:04.657467+0000","last_change":"2024-09-10T19:13:52.171265+0000","last_active":"2024-09-10T19:14:04.657467+0000","last_peered":"2024-09-10T19:14:04.657467+0000","last_clean":"2024-09-10T19:14:04.657467+0000","last_became_active":"2024-09-10T19:12:11.032901+0000","last_became_peered":"2024-09-10T19:12:11.032901+0000","last_unstale":"2024-09-10T19:14:04.657467+0000","last_undegraded":"2024-09-10T19:14:04.657467+0000","last_fullsized":"2024-09-10T19:14:04.657467+0000","mapping_epoch":86,"log_start":"101'4862","ondisk_log_start":"101'4862","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"114'9197","last_scrub_stamp":"2024-09-10T19:13:52.169437+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:12:10.007465+0000","last_clean_scrub_stamp":"2024-09-10T19:13:52.169437+0000","objects_scrubbed":8820,"log_size":5173,"log_dups_size":0,"ondisk_log_size":5173,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:15:07.775054+0000","scrub_duration":3.857076422,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":514916352,"num_objects":7857,"num_object_clones":0,"num_object_copies":23571,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7857,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10035,"num_write_kb":572544,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.d","version":"66'17","reported_seq":151,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-09-10T19:14:07.013151+0000","last_change":"2024-09-10T19:13:54.747117+0000","last_active":"2024-09-10T19:14:07.013151+0000","last_peered":"2024-09-10T19:14:07.013151+0000","last_clean":"2024-09-10T19:14:07.013151+0000","last_became_active":"2024-09-10T19:13:54.713411+0000","last_became_peered":"2024-09-10T19:13:54.713411+0000","last_unstale":"2024-09-10T19:14:07.013151+0000","last_undegraded":"2024-09-10T19:14:07.013151+0000","last_fullsized":"2024-09-10T19:14:07.013151+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'17","last_scrub_stamp":"2024-09-10T19:12:45.367252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:12:45.367252+0000","objects_scrubbed":8,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0031240310000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":88,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-09-10T19:13:55.801180+0000","last_change":"2024-09-10T19:12:46.519868+0000","last_active":"2024-09-10T19:13:55.801180+0000","last_peered":"2024-09-10T19:13:55.801180+0000","last_clean":"2024-09-10T19:13:55.801180+0000","last_became_active":"2024-09-10T19:10:07.691119+0000","last_became_peered":"2024-09-10T19:10:07.691119+0000","last_unstale":"2024-09-10T19:13:55.801180+0000","last_undegraded":"2024-09-10T19:13:55.801180+0000","last_fullsized":"2024-09-10T19:13:55.801180+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:12:46.519818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:11:22.280002+0000","last_clean_scrub_stamp":"2024-09-10T19:12:46.519818+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:14:07.544825+0000","scrub_duration":0.001139279,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":41,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-09-10T19:14:07.012827+0000","last_change":"2024-09-10T19:13:54.713806+0000","last_active":"2024-09-10T19:14:07.012827+0000","last_peered":"2024-09-10T19:14:07.012827+0000","last_clean":"2024-09-10T19:14:07.012827+0000","last_became_active":"2024-09-10T19:13:54.712883+0000","last_became_peered":"2024-09-10T19:13:54.712883+0000","last_unstale":"2024-09-10T19:14:07.012827+0000","last_undegraded":"2024-09-10T19:14:07.012827+0000","last_fullsized":"2024-09-10T19:14:07.012827+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:12:00.240841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:12:00.240841+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":82,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-09-10T19:13:55.706894+0000","last_change":"2024-09-10T19:12:49.342906+0000","last_active":"2024-09-10T19:13:55.706894+0000","last_peered":"2024-09-10T19:13:55.706894+0000","last_clean":"2024-09-10T19:13:55.706894+0000","last_became_active":"2024-09-10T19:10:13.722894+0000","last_became_peered":"2024-09-10T19:10:13.722894+0000","last_unstale":"2024-09-10T19:13:55.706894+0000","last_undegraded":"2024-09-10T19:13:55.706894+0000","last_fullsized":"2024-09-10T19:13:55.706894+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:12:49.342837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:12:49.342837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:14:01.157284+0000","scrub_duration":0.0092298559999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.f","version":"118'9837","reported_seq":10232,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-09-10T19:14:04.736906+0000","last_change":"2024-09-10T19:13:49.950750+0000","last_active":"2024-09-10T19:14:04.736906+0000","last_peered":"2024-09-10T19:14:04.736906+0000","last_clean":"2024-09-10T19:14:04.736906+0000","last_became_active":"2024-09-10T19:12:11.036181+0000","last_became_peered":"2024-09-10T19:12:11.036181+0000","last_unstale":"2024-09-10T19:14:04.736906+0000","last_undegraded":"2024-09-10T19:14:04.736906+0000","last_fullsized":"2024-09-10T19:14:04.736906+0000","mapping_epoch":86,"log_start":"104'5125","ondisk_log_start":"104'5125","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"114'8808","last_scrub_stamp":"2024-09-10T19:13:49.950668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:12:10.007465+0000","last_clean_scrub_stamp":"2024-09-10T19:13:49.950668+0000","objects_scrubbed":8620,"log_size":4712,"log_dups_size":0,"ondisk_log_size":4712,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:14:50.581709+0000","scrub_duration":5.7089046000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":500629504,"num_objects":7639,"num_object_clones":0,"num_object_copies":22917,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7639,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9837,"num_write_kb":559232,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.c","version":"66'10","reported_seq":113,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-09-10T19:14:07.012951+0000","last_change":"2024-09-10T19:13:51.416075+0000","last_active":"2024-09-10T19:14:07.012951+0000","last_peered":"2024-09-10T19:14:07.012951+0000","last_clean":"2024-09-10T19:14:07.012951+0000","last_became_active":"2024-09-10T19:10:09.688321+0000","last_became_peered":"2024-09-10T19:10:09.688321+0000","last_unstale":"2024-09-10T19:14:07.012951+0000","last_undegraded":"2024-09-10T19:14:07.012951+0000","last_fullsized":"2024-09-10T19:14:07.012951+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'10","last_scrub_stamp":"2024-09-10T19:13:51.415998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:13:51.415998+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:15:20.996413+0000","scrub_duration":0.005972799,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":38,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-09-10T19:13:55.720634+0000","last_change":"2024-09-10T19:13:54.731440+0000","last_active":"2024-09-10T19:13:55.720634+0000","last_peered":"2024-09-10T19:13:55.720634+0000","last_clean":"2024-09-10T19:13:55.720634+0000","last_became_active":"2024-09-10T19:13:54.731134+0000","last_became_peered":"2024-09-10T19:13:54.731134+0000","last_unstale":"2024-09-10T19:13:55.720634+0000","last_undegraded":"2024-09-10T19:13:55.720634+0000","last_fullsized":"2024-09-10T19:13:55.720634+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:11:26.110138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:11:26.110138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,5],"acting":[3,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"80'9","reported_seq":143,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-09-10T19:13:56.959137+0000","last_change":"2024-09-10T19:12:51.343854+0000","last_active":"2024-09-10T19:13:56.959137+0000","last_peered":"2024-09-10T19:13:56.959137+0000","last_clean":"2024-09-10T19:13:56.959137+0000","last_became_active":"2024-09-10T19:10:11.698547+0000","last_became_peered":"2024-09-10T19:10:11.698547+0000","last_unstale":"2024-09-10T19:13:56.959137+0000","last_undegraded":"2024-09-10T19:13:56.959137+0000","last_fullsized":"2024-09-10T19:13:56.959137+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"80'9","last_scrub_stamp":"2024-09-10T19:12:51.343776+0000","last_deep_scrub":"80'9","last_deep_scrub_stamp":"2024-09-10T19:11:25.156465+0000","last_clean_scrub_stamp":"2024-09-10T19:12:51.343776+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:14:12.772014+0000","scrub_duration":0.0026375819999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":94,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-09-10T19:14:07.012949+0000","last_change":"2024-09-10T19:14:00.299240+0000","last_active":"2024-09-10T19:14:07.012949+0000","last_peered":"2024-09-10T19:14:07.012949+0000","last_clean":"2024-09-10T19:14:07.012949+0000","last_became_active":"2024-09-10T19:13:54.719498+0000","last_became_peered":"2024-09-10T19:13:54.719498+0000","last_unstale":"2024-09-10T19:14:07.012949+0000","last_undegraded":"2024-09-10T19:14:07.012949+0000","last_fullsized":"2024-09-10T19:14:07.012949+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:11:26.096090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:11:26.096090+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0015599909999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.c","version":"121'10245","reported_seq":10639,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-09-10T19:14:09.616812+0000","last_change":"2024-09-10T19:13:53.867193+0000","last_active":"2024-09-10T19:14:09.616812+0000","last_peered":"2024-09-10T19:14:09.616812+0000","last_clean":"2024-09-10T19:14:09.616812+0000","last_became_active":"2024-09-10T19:12:11.029860+0000","last_became_peered":"2024-09-10T19:12:11.029860+0000","last_unstale":"2024-09-10T19:14:09.616812+0000","last_undegraded":"2024-09-10T19:14:09.616812+0000","last_fullsized":"2024-09-10T19:14:09.616812+0000","mapping_epoch":86,"log_start":"105'5282","ondisk_log_start":"105'5282","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"114'9191","last_scrub_stamp":"2024-09-10T19:13:53.867107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:12:10.007465+0000","last_clean_scrub_stamp":"2024-09-10T19:13:53.867107+0000","objects_scrubbed":8534,"log_size":4963,"log_dups_size":0,"ondisk_log_size":4963,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:15:11.622513+0000","scrub_duration":2.6024067190000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":482934784,"num_objects":7369,"num_object_clones":0,"num_object_copies":22107,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7369,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10245,"num_write_kb":563648,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.f","version":"66'15","reported_seq":91,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-09-10T19:13:55.711528+0000","last_change":"2024-09-10T19:13:54.742958+0000","last_active":"2024-09-10T19:13:55.711528+0000","last_peered":"2024-09-10T19:13:55.711528+0000","last_clean":"2024-09-10T19:13:55.711528+0000","last_became_active":"2024-09-10T19:13:54.711634+0000","last_became_peered":"2024-09-10T19:13:54.711634+0000","last_unstale":"2024-09-10T19:13:55.711528+0000","last_undegraded":"2024-09-10T19:13:55.711528+0000","last_fullsized":"2024-09-10T19:13:55.711528+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'15","last_scrub_stamp":"2024-09-10T19:11:59.232145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:11:59.232145+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":38,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-09-10T19:13:55.887724+0000","last_change":"2024-09-10T19:13:54.718740+0000","last_active":"2024-09-10T19:13:55.887724+0000","last_peered":"2024-09-10T19:13:55.887724+0000","last_clean":"2024-09-10T19:13:55.887724+0000","last_became_active":"2024-09-10T19:13:54.718452+0000","last_became_peered":"2024-09-10T19:13:54.718452+0000","last_unstale":"2024-09-10T19:13:55.887724+0000","last_undegraded":"2024-09-10T19:13:55.887724+0000","last_fullsized":"2024-09-10T19:13:55.887724+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:11:11.035954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:11:11.035954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.e","version":"66'8","reported_seq":142,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-09-10T19:14:07.013059+0000","last_change":"2024-09-10T19:13:40.475460+0000","last_active":"2024-09-10T19:14:07.013059+0000","last_peered":"2024-09-10T19:14:07.013059+0000","last_clean":"2024-09-10T19:14:07.013059+0000","last_became_active":"2024-09-10T19:10:11.699232+0000","last_became_peered":"2024-09-10T19:10:11.699232+0000","last_unstale":"2024-09-10T19:14:07.013059+0000","last_undegraded":"2024-09-10T19:14:07.013059+0000","last_fullsized":"2024-09-10T19:14:07.013059+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'8","last_scrub_stamp":"2024-09-10T19:13:40.475375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:13:40.475375+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:14:43.741970+0000","scrub_duration":0.035806152000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":92,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-09-10T19:14:07.058825+0000","last_change":"2024-09-10T19:13:54.723273+0000","last_active":"2024-09-10T19:14:07.058825+0000","last_peered":"2024-09-10T19:14:07.058825+0000","last_clean":"2024-09-10T19:14:07.058825+0000","last_became_active":"2024-09-10T19:13:54.722688+0000","last_became_peered":"2024-09-10T19:13:54.722688+0000","last_unstale":"2024-09-10T19:14:07.058825+0000","last_undegraded":"2024-09-10T19:14:07.058825+0000","last_fullsized":"2024-09-10T19:14:07.058825+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:11:18.208782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:11:18.208782+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001904481,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.d","version":"121'10190","reported_seq":10251,"reported_epoch":121,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2024-09-10T19:14:09.629076+0000","last_change":"2024-09-10T19:13:55.810177+0000","last_active":"2024-09-10T19:14:09.629076+0000","last_peered":"2024-09-10T19:14:09.629076+0000","last_clean":"2024-09-10T19:12:45.797387+0000","last_became_active":"2024-09-10T19:13:55.789208+0000","last_became_peered":"2024-09-10T19:13:55.789208+0000","last_unstale":"2024-09-10T19:14:09.629076+0000","last_undegraded":"2024-09-10T19:13:55.724690+0000","last_fullsized":"2024-09-10T19:13:55.711897+0000","mapping_epoch":116,"log_start":"105'5256","ondisk_log_start":"105'5256","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:12:10.007465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:12:10.007465+0000","last_clean_scrub_stamp":"2024-09-10T19:12:10.007465+0000","objects_scrubbed":0,"log_size":4934,"log_dups_size":0,"ondisk_log_size":4934,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":477560856,"num_objects":7288,"num_object_clones":0,"num_object_copies":21864,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7288,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7288,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10190,"num_write_kb":559233,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0],"avail_no_missing":["6","0"],"object_location_counts":[{"shards":"0,6","objects":7288}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.e","version":"84'16","reported_seq":131,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-09-10T19:14:07.013114+0000","last_change":"2024-09-10T19:13:54.741260+0000","last_active":"2024-09-10T19:14:07.013114+0000","last_peered":"2024-09-10T19:14:07.013114+0000","last_clean":"2024-09-10T19:14:07.013114+0000","last_became_active":"2024-09-10T19:13:54.717022+0000","last_became_peered":"2024-09-10T19:13:54.717022+0000","last_unstale":"2024-09-10T19:14:07.013114+0000","last_undegraded":"2024-09-10T19:14:07.013114+0000","last_fullsized":"2024-09-10T19:14:07.013114+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'16","last_scrub_stamp":"2024-09-10T19:11:56.157975+0000","last_deep_scrub":"84'16","last_deep_scrub_stamp":"2024-09-10T19:11:56.157975+0000","last_clean_scrub_stamp":"2024-09-10T19:11:56.157975+0000","objects_scrubbed":5,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0045462239999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,3],"acting":[4,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":95,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-09-10T19:13:55.801479+0000","last_change":"2024-09-10T19:13:54.721400+0000","last_active":"2024-09-10T19:13:55.801479+0000","last_peered":"2024-09-10T19:13:55.801479+0000","last_clean":"2024-09-10T19:13:55.801479+0000","last_became_active":"2024-09-10T19:13:54.720293+0000","last_became_peered":"2024-09-10T19:13:54.720293+0000","last_unstale":"2024-09-10T19:13:55.801479+0000","last_undegraded":"2024-09-10T19:13:55.801479+0000","last_fullsized":"2024-09-10T19:13:55.801479+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:11:53.066849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:11:53.066849+0000","last_clean_scrub_stamp":"2024-09-10T19:11:53.066849+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0024992679999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":87,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-09-10T19:14:07.060192+0000","last_change":"2024-09-10T19:13:06.358188+0000","last_active":"2024-09-10T19:14:07.060192+0000","last_peered":"2024-09-10T19:14:07.060192+0000","last_clean":"2024-09-10T19:14:07.060192+0000","last_became_active":"2024-09-10T19:10:11.692215+0000","last_became_peered":"2024-09-10T19:10:11.692215+0000","last_unstale":"2024-09-10T19:14:07.060192+0000","last_undegraded":"2024-09-10T19:14:07.060192+0000","last_fullsized":"2024-09-10T19:14:07.060192+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:13:06.358109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:13:06.358109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:14:34.922567+0000","scrub_duration":0.0026724549999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":82,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-09-10T19:13:55.710237+0000","last_change":"2024-09-10T19:13:54.304493+0000","last_active":"2024-09-10T19:13:55.710237+0000","last_peered":"2024-09-10T19:13:55.710237+0000","last_clean":"2024-09-10T19:13:55.710237+0000","last_became_active":"2024-09-10T19:10:14.053736+0000","last_became_peered":"2024-09-10T19:10:14.053736+0000","last_unstale":"2024-09-10T19:13:55.710237+0000","last_undegraded":"2024-09-10T19:13:55.710237+0000","last_fullsized":"2024-09-10T19:13:55.710237+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:13:54.304435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:13:54.304435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:15:07.353025+0000","scrub_duration":0.0011043820000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.2","version":"118'9992","reported_seq":12204,"reported_epoch":118,"state":"active+undersized+degraded+remapped+backfilling","last_fresh":"2024-09-10T19:14:04.751746+0000","last_change":"2024-09-10T19:13:55.940201+0000","last_active":"2024-09-10T19:14:04.751746+0000","last_peered":"2024-09-10T19:14:04.751746+0000","last_clean":"2024-09-10T19:12:45.804432+0000","last_became_active":"2024-09-10T19:13:55.881880+0000","last_became_peered":"2024-09-10T19:13:55.881880+0000","last_unstale":"2024-09-10T19:14:04.751746+0000","last_undegraded":"2024-09-10T19:13:55.719843+0000","last_fullsized":"2024-09-10T19:13:55.709265+0000","mapping_epoch":116,"log_start":"105'5707","ondisk_log_start":"105'5707","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:12:10.007465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:12:10.007465+0000","last_clean_scrub_stamp":"2024-09-10T19:12:10.007465+0000","objects_scrubbed":0,"log_size":4285,"log_dups_size":0,"ondisk_log_size":4285,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":514195456,"num_objects":7846,"num_object_clones":0,"num_object_copies":23538,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6813,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7846,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9992,"num_write_kb":570816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1077,"num_bytes_recovered":70582272,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7],"acting":[3,2],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":7846}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"66'14","reported_seq":109,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-09-10T19:14:07.012546+0000","last_change":"2024-09-10T19:12:48.462230+0000","last_active":"2024-09-10T19:14:07.012546+0000","last_peered":"2024-09-10T19:14:07.012546+0000","last_clean":"2024-09-10T19:14:07.012546+0000","last_became_active":"2024-09-10T19:10:09.686534+0000","last_became_peered":"2024-09-10T19:10:09.686534+0000","last_unstale":"2024-09-10T19:14:07.012546+0000","last_undegraded":"2024-09-10T19:14:07.012546+0000","last_fullsized":"2024-09-10T19:14:07.012546+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'14","last_scrub_stamp":"2024-09-10T19:12:48.462133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:12:48.462133+0000","objects_scrubbed":7,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.002699791,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":91,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-09-10T19:14:07.063219+0000","last_change":"2024-09-10T19:12:52.885643+0000","last_active":"2024-09-10T19:14:07.063219+0000","last_peered":"2024-09-10T19:14:07.063219+0000","last_clean":"2024-09-10T19:14:07.063219+0000","last_became_active":"2024-09-10T19:10:07.684864+0000","last_became_peered":"2024-09-10T19:10:07.684864+0000","last_unstale":"2024-09-10T19:14:07.063219+0000","last_undegraded":"2024-09-10T19:14:07.063219+0000","last_fullsized":"2024-09-10T19:14:07.063219+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:12:52.885563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:12:52.885563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:14:12.908306+0000","scrub_duration":0.0035531249999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.0","version":"117'10","reported_seq":151,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-09-10T19:13:56.959397+0000","last_change":"2024-09-10T19:13:54.735247+0000","last_active":"2024-09-10T19:13:56.959397+0000","last_peered":"2024-09-10T19:13:56.959397+0000","last_clean":"2024-09-10T19:13:56.959397+0000","last_became_active":"2024-09-10T19:13:54.709846+0000","last_became_peered":"2024-09-10T19:13:54.709846+0000","last_unstale":"2024-09-10T19:13:56.959397+0000","last_undegraded":"2024-09-10T19:13:56.959397+0000","last_fullsized":"2024-09-10T19:13:56.959397+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'8","last_scrub_stamp":"2024-09-10T19:11:29.884739+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:11:29.884739+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0014372650000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,0],"acting":[3,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":82,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-09-10T19:13:55.800945+0000","last_change":"2024-09-10T19:13:15.350815+0000","last_active":"2024-09-10T19:13:55.800945+0000","last_peered":"2024-09-10T19:13:55.800945+0000","last_clean":"2024-09-10T19:13:55.800945+0000","last_became_active":"2024-09-10T19:10:14.054456+0000","last_became_peered":"2024-09-10T19:10:14.054456+0000","last_unstale":"2024-09-10T19:13:55.800945+0000","last_undegraded":"2024-09-10T19:13:55.800945+0000","last_fullsized":"2024-09-10T19:13:55.800945+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:13:15.350720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:11:41.073833+0000","last_clean_scrub_stamp":"2024-09-10T19:13:15.350720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:14:37.565761+0000","scrub_duration":0.0014361000000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.3","version":"118'9854","reported_seq":10237,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-09-10T19:14:04.723899+0000","last_change":"2024-09-10T19:13:44.417102+0000","last_active":"2024-09-10T19:14:04.723899+0000","last_peered":"2024-09-10T19:14:04.723899+0000","last_clean":"2024-09-10T19:14:04.723899+0000","last_became_active":"2024-09-10T19:12:11.029906+0000","last_became_peered":"2024-09-10T19:12:11.029906+0000","last_unstale":"2024-09-10T19:14:04.723899+0000","last_undegraded":"2024-09-10T19:14:04.723899+0000","last_fullsized":"2024-09-10T19:14:04.723899+0000","mapping_epoch":86,"log_start":"105'5533","ondisk_log_start":"105'5533","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"112'8444","last_scrub_stamp":"2024-09-10T19:13:44.417011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:12:10.007465+0000","last_clean_scrub_stamp":"2024-09-10T19:13:44.417011+0000","objects_scrubbed":8333,"log_size":4321,"log_dups_size":0,"ondisk_log_size":4321,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:14:54.007266+0000","scrub_duration":3.2086845460000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":500432896,"num_objects":7636,"num_object_clones":0,"num_object_copies":22908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7636,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9854,"num_write_kb":559680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"66'18","reported_seq":118,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-09-10T19:13:55.709705+0000","last_change":"2024-09-10T19:12:49.699111+0000","last_active":"2024-09-10T19:13:55.709705+0000","last_peered":"2024-09-10T19:13:55.709705+0000","last_clean":"2024-09-10T19:13:55.709705+0000","last_became_active":"2024-09-10T19:10:09.821836+0000","last_became_peered":"2024-09-10T19:10:09.821836+0000","last_unstale":"2024-09-10T19:13:55.709705+0000","last_undegraded":"2024-09-10T19:13:55.709705+0000","last_fullsized":"2024-09-10T19:13:55.709705+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'18","last_scrub_stamp":"2024-09-10T19:12:49.699054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:12:49.699054+0000","objects_scrubbed":8,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:14:15.625471+0000","scrub_duration":0.0073070890000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":41,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-09-10T19:14:07.061198+0000","last_change":"2024-09-10T19:13:54.711273+0000","last_active":"2024-09-10T19:14:07.061198+0000","last_peered":"2024-09-10T19:14:07.061198+0000","last_clean":"2024-09-10T19:14:07.061198+0000","last_became_active":"2024-09-10T19:13:54.710968+0000","last_became_peered":"2024-09-10T19:13:54.710968+0000","last_unstale":"2024-09-10T19:14:07.061198+0000","last_undegraded":"2024-09-10T19:14:07.061198+0000","last_fullsized":"2024-09-10T19:14:07.061198+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:11:42.202689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:11:42.202689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,3],"acting":[6,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":86,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-09-10T19:14:07.012388+0000","last_change":"2024-09-10T19:12:58.400261+0000","last_active":"2024-09-10T19:14:07.012388+0000","last_peered":"2024-09-10T19:14:07.012388+0000","last_clean":"2024-09-10T19:14:07.012388+0000","last_became_active":"2024-09-10T19:10:11.698054+0000","last_became_peered":"2024-09-10T19:10:11.698054+0000","last_unstale":"2024-09-10T19:14:07.012388+0000","last_undegraded":"2024-09-10T19:14:07.012388+0000","last_fullsized":"2024-09-10T19:14:07.012388+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:12:58.400206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:12:58.400206+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0071235179999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":84,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-09-10T19:14:07.012928+0000","last_change":"2024-09-10T19:12:57.409831+0000","last_active":"2024-09-10T19:14:07.012928+0000","last_peered":"2024-09-10T19:14:07.012928+0000","last_clean":"2024-09-10T19:14:07.012928+0000","last_became_active":"2024-09-10T19:10:13.722784+0000","last_became_peered":"2024-09-10T19:10:13.722784+0000","last_unstale":"2024-09-10T19:14:07.012928+0000","last_undegraded":"2024-09-10T19:14:07.012928+0000","last_fullsized":"2024-09-10T19:14:07.012928+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:12:57.409749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:12:57.409749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:14:23.578390+0000","scrub_duration":0.0087033260000000008,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.0","version":"121'10217","reported_seq":10280,"reported_epoch":121,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2024-09-10T19:14:09.618450+0000","last_change":"2024-09-10T19:13:55.804438+0000","last_active":"2024-09-10T19:14:09.618450+0000","last_peered":"2024-09-10T19:14:09.618450+0000","last_clean":"2024-09-10T19:12:45.799830+0000","last_became_active":"2024-09-10T19:13:55.794303+0000","last_became_peered":"2024-09-10T19:13:55.794303+0000","last_unstale":"2024-09-10T19:14:09.618450+0000","last_undegraded":"2024-09-10T19:13:55.727127+0000","last_fullsized":"2024-09-10T19:13:55.710552+0000","mapping_epoch":116,"log_start":"104'5142","ondisk_log_start":"104'5142","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:12:10.007465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:12:10.007465+0000","last_clean_scrub_stamp":"2024-09-10T19:12:10.007465+0000","objects_scrubbed":0,"log_size":5075,"log_dups_size":0,"ondisk_log_size":5075,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":475987968,"num_objects":7263,"num_object_clones":0,"num_object_copies":21789,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7263,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7263,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10217,"num_write_kb":559360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3],"avail_no_missing":["5","3"],"object_location_counts":[{"shards":"3,5","objects":7263}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.3","version":"104'27","reported_seq":130,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-09-10T19:13:55.710731+0000","last_change":"2024-09-10T19:13:02.226033+0000","last_active":"2024-09-10T19:13:55.710731+0000","last_peered":"2024-09-10T19:13:55.710731+0000","last_clean":"2024-09-10T19:13:55.710731+0000","last_became_active":"2024-09-10T19:10:09.823003+0000","last_became_peered":"2024-09-10T19:10:09.823003+0000","last_unstale":"2024-09-10T19:13:55.710731+0000","last_undegraded":"2024-09-10T19:13:55.710731+0000","last_fullsized":"2024-09-10T19:13:55.710731+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"104'27","last_scrub_stamp":"2024-09-10T19:13:02.225944+0000","last_deep_scrub":"104'27","last_deep_scrub_stamp":"2024-09-10T19:13:02.225944+0000","last_clean_scrub_stamp":"2024-09-10T19:13:02.225944+0000","objects_scrubbed":8,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:14:15.557784+0000","scrub_duration":0.0099233020000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":88,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-09-10T19:13:55.804173+0000","last_change":"2024-09-10T19:12:53.513000+0000","last_active":"2024-09-10T19:13:55.804173+0000","last_peered":"2024-09-10T19:13:55.804173+0000","last_clean":"2024-09-10T19:13:55.804173+0000","last_became_active":"2024-09-10T19:10:07.690325+0000","last_became_peered":"2024-09-10T19:10:07.690325+0000","last_unstale":"2024-09-10T19:13:55.804173+0000","last_undegraded":"2024-09-10T19:13:55.804173+0000","last_fullsized":"2024-09-10T19:13:55.804173+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:12:53.512923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:12:53.512923+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:14:06.619886+0000","scrub_duration":0.0087393290000000005,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":87,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-09-10T19:14:07.058871+0000","last_change":"2024-09-10T19:12:46.963842+0000","last_active":"2024-09-10T19:14:07.058871+0000","last_peered":"2024-09-10T19:14:07.058871+0000","last_clean":"2024-09-10T19:14:07.058871+0000","last_became_active":"2024-09-10T19:10:11.700370+0000","last_became_peered":"2024-09-10T19:10:11.700370+0000","last_unstale":"2024-09-10T19:14:07.058871+0000","last_undegraded":"2024-09-10T19:14:07.058871+0000","last_fullsized":"2024-09-10T19:14:07.058871+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:12:46.963746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:12:46.963746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001460004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":41,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-09-10T19:14:07.059124+0000","last_change":"2024-09-10T19:13:54.718253+0000","last_active":"2024-09-10T19:14:07.059124+0000","last_peered":"2024-09-10T19:14:07.059124+0000","last_clean":"2024-09-10T19:14:07.059124+0000","last_became_active":"2024-09-10T19:13:54.717977+0000","last_became_peered":"2024-09-10T19:13:54.717977+0000","last_unstale":"2024-09-10T19:14:07.059124+0000","last_undegraded":"2024-09-10T19:14:07.059124+0000","last_fullsized":"2024-09-10T19:14:07.059124+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:11:51.187109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:11:51.187109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,7],"acting":[6,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.1","version":"121'10163","reported_seq":10552,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-09-10T19:14:09.631783+0000","last_change":"2024-09-10T19:13:45.105571+0000","last_active":"2024-09-10T19:14:09.631783+0000","last_peered":"2024-09-10T19:14:09.631783+0000","last_clean":"2024-09-10T19:14:09.631783+0000","last_became_active":"2024-09-10T19:12:11.030399+0000","last_became_peered":"2024-09-10T19:12:11.030399+0000","last_unstale":"2024-09-10T19:14:09.631783+0000","last_undegraded":"2024-09-10T19:14:09.631783+0000","last_fullsized":"2024-09-10T19:14:09.631783+0000","mapping_epoch":86,"log_start":"105'5178","ondisk_log_start":"105'5178","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"113'8485","last_scrub_stamp":"2024-09-10T19:13:45.105477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:12:10.007465+0000","last_clean_scrub_stamp":"2024-09-10T19:13:45.105477+0000","objects_scrubbed":8400,"log_size":4985,"log_dups_size":0,"ondisk_log_size":4985,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:14:48.386316+0000","scrub_duration":2.6533322940000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":474021888,"num_objects":7233,"num_object_clones":0,"num_object_copies":21699,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7233,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10163,"num_write_kb":556672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.2","version":"66'10","reported_seq":57,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-09-10T19:14:07.060510+0000","last_change":"2024-09-10T19:13:54.748881+0000","last_active":"2024-09-10T19:14:07.060510+0000","last_peered":"2024-09-10T19:14:07.060510+0000","last_clean":"2024-09-10T19:14:07.060510+0000","last_became_active":"2024-09-10T19:13:54.714542+0000","last_became_peered":"2024-09-10T19:13:54.714542+0000","last_unstale":"2024-09-10T19:14:07.060510+0000","last_undegraded":"2024-09-10T19:14:07.060510+0000","last_fullsized":"2024-09-10T19:14:07.060510+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'10","last_scrub_stamp":"2024-09-10T19:12:56.360409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:12:56.360409+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:14:21.848263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":14,"num_read_kb":9,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,0],"acting":[5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":38,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-09-10T19:13:55.710221+0000","last_change":"2024-09-10T19:13:54.718993+0000","last_active":"2024-09-10T19:13:55.710221+0000","last_peered":"2024-09-10T19:13:55.710221+0000","last_clean":"2024-09-10T19:13:55.710221+0000","last_became_active":"2024-09-10T19:13:54.718267+0000","last_became_peered":"2024-09-10T19:13:54.718267+0000","last_unstale":"2024-09-10T19:13:55.710221+0000","last_undegraded":"2024-09-10T19:13:55.710221+0000","last_fullsized":"2024-09-10T19:13:55.710221+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:11:55.227355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:11:55.227355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3","version":"84'9","reported_seq":140,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-09-10T19:13:56.951702+0000","last_change":"2024-09-10T19:13:45.236207+0000","last_active":"2024-09-10T19:13:56.951702+0000","last_peered":"2024-09-10T19:13:56.951702+0000","last_clean":"2024-09-10T19:13:56.951702+0000","last_became_active":"2024-09-10T19:10:11.699690+0000","last_became_peered":"2024-09-10T19:10:11.699690+0000","last_unstale":"2024-09-10T19:13:56.951702+0000","last_undegraded":"2024-09-10T19:13:56.951702+0000","last_fullsized":"2024-09-10T19:13:56.951702+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'9","last_scrub_stamp":"2024-09-10T19:13:45.236094+0000","last_deep_scrub":"84'9","last_deep_scrub_stamp":"2024-09-10T19:11:53.879272+0000","last_clean_scrub_stamp":"2024-09-10T19:13:45.236094+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:15:04.114110+0000","scrub_duration":0.036165602999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":82,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-09-10T19:13:55.708583+0000","last_change":"2024-09-10T19:13:40.234378+0000","last_active":"2024-09-10T19:13:55.708583+0000","last_peered":"2024-09-10T19:13:55.708583+0000","last_clean":"2024-09-10T19:13:55.708583+0000","last_became_active":"2024-09-10T19:10:13.720403+0000","last_became_peered":"2024-09-10T19:10:13.720403+0000","last_unstale":"2024-09-10T19:13:55.708583+0000","last_undegraded":"2024-09-10T19:13:55.708583+0000","last_fullsized":"2024-09-10T19:13:55.708583+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:13:40.234289+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:13:40.234289+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:14:58.963116+0000","scrub_duration":0.0086405590000000008,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.7","version":"121'10226","reported_seq":10612,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-09-10T19:14:09.643386+0000","last_change":"2024-09-10T19:13:55.214546+0000","last_active":"2024-09-10T19:14:09.643386+0000","last_peered":"2024-09-10T19:14:09.643386+0000","last_clean":"2024-09-10T19:14:09.643386+0000","last_became_active":"2024-09-10T19:12:11.032857+0000","last_became_peered":"2024-09-10T19:12:11.032857+0000","last_unstale":"2024-09-10T19:14:09.643386+0000","last_undegraded":"2024-09-10T19:14:09.643386+0000","last_fullsized":"2024-09-10T19:14:09.643386+0000","mapping_epoch":86,"log_start":"104'5225","ondisk_log_start":"104'5225","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"114'9143","last_scrub_stamp":"2024-09-10T19:13:55.214477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:12:10.007465+0000","last_clean_scrub_stamp":"2024-09-10T19:13:55.214477+0000","objects_scrubbed":8339,"log_size":5001,"log_dups_size":0,"ondisk_log_size":5001,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:15:20.270435+0000","scrub_duration":0.96843253399999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":475529216,"num_objects":7256,"num_object_clones":0,"num_object_copies":21768,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7256,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10226,"num_write_kb":559424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.4","version":"117'30","reported_seq":137,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-09-10T19:13:56.955809+0000","last_change":"2024-09-10T19:13:54.752131+0000","last_active":"2024-09-10T19:13:56.955809+0000","last_peered":"2024-09-10T19:13:56.955809+0000","last_clean":"2024-09-10T19:13:56.955809+0000","last_became_active":"2024-09-10T19:13:54.715642+0000","last_became_peered":"2024-09-10T19:13:54.715642+0000","last_unstale":"2024-09-10T19:13:56.955809+0000","last_undegraded":"2024-09-10T19:13:56.955809+0000","last_fullsized":"2024-09-10T19:13:56.955809+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'28","last_scrub_stamp":"2024-09-10T19:12:45.260189+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:12:45.260189+0000","objects_scrubbed":0,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:14:04.348670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":47,"num_read_kb":32,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":358,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":98,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-09-10T19:14:07.060806+0000","last_change":"2024-09-10T19:13:54.730667+0000","last_active":"2024-09-10T19:14:07.060806+0000","last_peered":"2024-09-10T19:14:07.060806+0000","last_clean":"2024-09-10T19:14:07.060806+0000","last_became_active":"2024-09-10T19:13:54.730096+0000","last_became_peered":"2024-09-10T19:13:54.730096+0000","last_unstale":"2024-09-10T19:14:07.060806+0000","last_undegraded":"2024-09-10T19:14:07.060806+0000","last_fullsized":"2024-09-10T19:14:07.060806+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:11:41.429402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:11:41.429402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0014226359999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":91,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-09-10T19:13:55.709295+0000","last_change":"2024-09-10T19:13:54.714602+0000","last_active":"2024-09-10T19:13:55.709295+0000","last_peered":"2024-09-10T19:13:55.709295+0000","last_clean":"2024-09-10T19:13:55.709295+0000","last_became_active":"2024-09-10T19:13:54.714017+0000","last_became_peered":"2024-09-10T19:13:54.714017+0000","last_unstale":"2024-09-10T19:13:55.709295+0000","last_undegraded":"2024-09-10T19:13:55.709295+0000","last_fullsized":"2024-09-10T19:13:55.709295+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:11:14.985249+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:11:14.985249+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001460161,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,6],"acting":[0,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":72,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-09-10T19:13:55.711022+0000","last_change":"2024-09-10T19:12:46.691448+0000","last_active":"2024-09-10T19:13:55.711022+0000","last_peered":"2024-09-10T19:13:55.711022+0000","last_clean":"2024-09-10T19:13:55.711022+0000","last_became_active":"2024-09-10T19:10:13.721744+0000","last_became_peered":"2024-09-10T19:10:13.721744+0000","last_unstale":"2024-09-10T19:13:55.711022+0000","last_undegraded":"2024-09-10T19:13:55.711022+0000","last_fullsized":"2024-09-10T19:13:55.711022+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:12:46.691353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:12:46.691353+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:13:58.190370+0000","scrub_duration":0.0014482360000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.4","version":"118'9790","reported_seq":12325,"reported_epoch":118,"state":"active+undersized+degraded+remapped+backfilling","last_fresh":"2024-09-10T19:14:04.769141+0000","last_change":"2024-09-10T19:13:55.827146+0000","last_active":"2024-09-10T19:14:04.769141+0000","last_peered":"2024-09-10T19:14:04.769141+0000","last_clean":"2024-09-10T19:12:45.806743+0000","last_became_active":"2024-09-10T19:13:55.781235+0000","last_became_peered":"2024-09-10T19:13:55.781235+0000","last_unstale":"2024-09-10T19:14:04.769141+0000","last_undegraded":"2024-09-10T19:13:55.814241+0000","last_fullsized":"2024-09-10T19:13:55.709078+0000","mapping_epoch":116,"log_start":"104'5009","ondisk_log_start":"104'5009","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:12:10.007465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:12:10.007465+0000","last_clean_scrub_stamp":"2024-09-10T19:12:10.007465+0000","objects_scrubbed":0,"log_size":4781,"log_dups_size":0,"ondisk_log_size":4781,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":498073600,"num_objects":7600,"num_object_clones":0,"num_object_copies":22800,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6416,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7600,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9790,"num_write_kb":556480,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1237,"num_bytes_recovered":80936960,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,6],"acting":[0,4],"avail_no_missing":["0","4"],"object_location_counts":[{"shards":"0,4","objects":7600}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.7","version":"82'18","reported_seq":114,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-09-10T19:14:07.060326+0000","last_change":"2024-09-10T19:13:54.737836+0000","last_active":"2024-09-10T19:14:07.060326+0000","last_peered":"2024-09-10T19:14:07.060326+0000","last_clean":"2024-09-10T19:14:07.060326+0000","last_became_active":"2024-09-10T19:13:54.712065+0000","last_became_peered":"2024-09-10T19:13:54.712065+0000","last_unstale":"2024-09-10T19:14:07.060326+0000","last_undegraded":"2024-09-10T19:14:07.060326+0000","last_fullsized":"2024-09-10T19:14:07.060326+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'18","last_scrub_stamp":"2024-09-10T19:11:49.207309+0000","last_deep_scrub":"82'18","last_deep_scrub_stamp":"2024-09-10T19:11:49.207309+0000","last_clean_scrub_stamp":"2024-09-10T19:11:49.207309+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":19,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.0","version":"67'5","reported_seq":147,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-09-10T19:14:02.730792+0000","last_change":"2024-09-10T19:13:54.749420+0000","last_active":"2024-09-10T19:14:02.730792+0000","last_peered":"2024-09-10T19:14:02.730792+0000","last_clean":"2024-09-10T19:14:02.730792+0000","last_became_active":"2024-09-10T19:13:54.715570+0000","last_became_peered":"2024-09-10T19:13:54.715570+0000","last_unstale":"2024-09-10T19:14:02.730792+0000","last_undegraded":"2024-09-10T19:14:02.730792+0000","last_fullsized":"2024-09-10T19:14:02.730792+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'5","last_scrub_stamp":"2024-09-10T19:11:16.007261+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:11:16.007261+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00034454499999999997,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":80,"num_read_kb":75,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":389,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,0],"acting":[2,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":88,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-09-10T19:13:55.706186+0000","last_change":"2024-09-10T19:12:53.318459+0000","last_active":"2024-09-10T19:13:55.706186+0000","last_peered":"2024-09-10T19:13:55.706186+0000","last_clean":"2024-09-10T19:13:55.706186+0000","last_became_active":"2024-09-10T19:10:07.681456+0000","last_became_peered":"2024-09-10T19:10:07.681456+0000","last_unstale":"2024-09-10T19:13:55.706186+0000","last_undegraded":"2024-09-10T19:13:55.706186+0000","last_fullsized":"2024-09-10T19:13:55.706186+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:12:53.318408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:12:53.318408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:14:19.519971+0000","scrub_duration":0.01029277,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":84,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-09-10T19:13:55.706826+0000","last_change":"2024-09-10T19:13:00.366695+0000","last_active":"2024-09-10T19:13:55.706826+0000","last_peered":"2024-09-10T19:13:55.706826+0000","last_clean":"2024-09-10T19:13:55.706826+0000","last_became_active":"2024-09-10T19:10:11.695205+0000","last_became_peered":"2024-09-10T19:10:11.695205+0000","last_unstale":"2024-09-10T19:13:55.706826+0000","last_undegraded":"2024-09-10T19:13:55.706826+0000","last_fullsized":"2024-09-10T19:13:55.706826+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:13:00.366610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:13:00.366610+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:14:22.508784+0000","scrub_duration":0.0023966700000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":75,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-09-10T19:13:55.803889+0000","last_change":"2024-09-10T19:13:44.394892+0000","last_active":"2024-09-10T19:13:55.803889+0000","last_peered":"2024-09-10T19:13:55.803889+0000","last_clean":"2024-09-10T19:13:55.803889+0000","last_became_active":"2024-09-10T19:10:14.053697+0000","last_became_peered":"2024-09-10T19:10:14.053697+0000","last_unstale":"2024-09-10T19:13:55.803889+0000","last_undegraded":"2024-09-10T19:13:55.803889+0000","last_fullsized":"2024-09-10T19:13:55.803889+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:12:06.039981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:12:06.039981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0013437499999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.5","version":"118'9900","reported_seq":10288,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-09-10T19:14:04.639602+0000","last_change":"2024-09-10T19:13:55.551583+0000","last_active":"2024-09-10T19:14:04.639602+0000","last_peered":"2024-09-10T19:14:04.639602+0000","last_clean":"2024-09-10T19:14:04.639602+0000","last_became_active":"2024-09-10T19:12:11.032880+0000","last_became_peered":"2024-09-10T19:12:11.032880+0000","last_unstale":"2024-09-10T19:14:04.639602+0000","last_undegraded":"2024-09-10T19:14:04.639602+0000","last_fullsized":"2024-09-10T19:14:04.639602+0000","mapping_epoch":86,"log_start":"105'5552","ondisk_log_start":"105'5552","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"115'9199","last_scrub_stamp":"2024-09-10T19:13:55.551514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:12:10.007465+0000","last_clean_scrub_stamp":"2024-09-10T19:13:55.551514+0000","objects_scrubbed":8461,"log_size":4348,"log_dups_size":0,"ondisk_log_size":4348,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:15:07.317469+0000","scrub_duration":1.028952525,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":508559360,"num_objects":7760,"num_object_clones":0,"num_object_copies":23280,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7760,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9900,"num_write_kb":565120,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"66'12","reported_seq":125,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-09-10T19:13:55.709967+0000","last_change":"2024-09-10T19:13:54.735877+0000","last_active":"2024-09-10T19:13:55.709967+0000","last_peered":"2024-09-10T19:13:55.709967+0000","last_clean":"2024-09-10T19:13:55.709967+0000","last_became_active":"2024-09-10T19:13:54.719200+0000","last_became_peered":"2024-09-10T19:13:54.719200+0000","last_unstale":"2024-09-10T19:13:55.709967+0000","last_undegraded":"2024-09-10T19:13:55.709967+0000","last_fullsized":"2024-09-10T19:13:55.709967+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'12","last_scrub_stamp":"2024-09-10T19:11:38.936857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:11:38.936857+0000","objects_scrubbed":6,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0016916909999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,7],"acting":[0,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"61'1","reported_seq":94,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-09-10T19:13:55.712018+0000","last_change":"2024-09-10T19:13:06.265721+0000","last_active":"2024-09-10T19:13:55.712018+0000","last_peered":"2024-09-10T19:13:55.712018+0000","last_clean":"2024-09-10T19:13:55.712018+0000","last_became_active":"2024-09-10T19:10:07.683259+0000","last_became_peered":"2024-09-10T19:10:07.683259+0000","last_unstale":"2024-09-10T19:13:55.712018+0000","last_undegraded":"2024-09-10T19:13:55.712018+0000","last_fullsized":"2024-09-10T19:13:55.712018+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'1","last_scrub_stamp":"2024-09-10T19:13:06.265646+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:13:06.265646+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:14:31.103103+0000","scrub_duration":0.012205384,"objects_trimmed":0,"snaptrim_duration":0.0014341130000000001,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":95,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-09-10T19:13:55.811221+0000","last_change":"2024-09-10T19:13:54.721892+0000","last_active":"2024-09-10T19:13:55.811221+0000","last_peered":"2024-09-10T19:13:55.811221+0000","last_clean":"2024-09-10T19:13:55.811221+0000","last_became_active":"2024-09-10T19:13:54.721486+0000","last_became_peered":"2024-09-10T19:13:54.721486+0000","last_unstale":"2024-09-10T19:13:55.811221+0000","last_undegraded":"2024-09-10T19:13:55.811221+0000","last_fullsized":"2024-09-10T19:13:55.811221+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:11:21.301058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:11:21.301058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0021074710000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":104,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-09-10T19:14:07.060256+0000","last_change":"2024-09-10T19:13:54.712428+0000","last_active":"2024-09-10T19:14:07.060256+0000","last_peered":"2024-09-10T19:14:07.060256+0000","last_clean":"2024-09-10T19:14:07.060256+0000","last_became_active":"2024-09-10T19:13:54.711685+0000","last_became_peered":"2024-09-10T19:13:54.711685+0000","last_unstale":"2024-09-10T19:14:07.060256+0000","last_undegraded":"2024-09-10T19:14:07.060256+0000","last_fullsized":"2024-09-10T19:14:07.060256+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:12:44.527561+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:12:44.527561+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.022638367999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,4],"acting":[5,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":41,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-09-10T19:14:07.058480+0000","last_change":"2024-09-10T19:13:54.723228+0000","last_active":"2024-09-10T19:14:07.058480+0000","last_peered":"2024-09-10T19:14:07.058480+0000","last_clean":"2024-09-10T19:14:07.058480+0000","last_became_active":"2024-09-10T19:13:54.722937+0000","last_became_peered":"2024-09-10T19:13:54.722937+0000","last_unstale":"2024-09-10T19:14:07.058480+0000","last_undegraded":"2024-09-10T19:14:07.058480+0000","last_fullsized":"2024-09-10T19:14:07.058480+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:12:05.165500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:12:05.165500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.6","version":"118'9737","reported_seq":9779,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-09-10T19:14:04.754979+0000","last_change":"2024-09-10T19:13:43.239851+0000","last_active":"2024-09-10T19:14:04.754979+0000","last_peered":"2024-09-10T19:14:04.754979+0000","last_clean":"2024-09-10T19:14:04.754979+0000","last_became_active":"2024-09-10T19:12:11.035290+0000","last_became_peered":"2024-09-10T19:12:11.035290+0000","last_unstale":"2024-09-10T19:14:04.754979+0000","last_undegraded":"2024-09-10T19:14:04.754979+0000","last_fullsized":"2024-09-10T19:14:04.754979+0000","mapping_epoch":86,"log_start":"104'5042","ondisk_log_start":"104'5042","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:12:10.007465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:12:10.007465+0000","last_clean_scrub_stamp":"2024-09-10T19:12:10.007465+0000","objects_scrubbed":0,"log_size":4695,"log_dups_size":0,"ondisk_log_size":4695,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":495910912,"num_objects":7567,"num_object_clones":0,"num_object_copies":22701,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7567,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9737,"num_write_kb":553728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.5","version":"80'21","reported_seq":118,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-09-10T19:14:07.060476+0000","last_change":"2024-09-10T19:12:54.860083+0000","last_active":"2024-09-10T19:14:07.060476+0000","last_peered":"2024-09-10T19:14:07.060476+0000","last_clean":"2024-09-10T19:14:07.060476+0000","last_became_active":"2024-09-10T19:10:09.822184+0000","last_became_peered":"2024-09-10T19:10:09.822184+0000","last_unstale":"2024-09-10T19:14:07.060476+0000","last_undegraded":"2024-09-10T19:14:07.060476+0000","last_fullsized":"2024-09-10T19:14:07.060476+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"80'21","last_scrub_stamp":"2024-09-10T19:12:54.860013+0000","last_deep_scrub":"80'21","last_deep_scrub_stamp":"2024-09-10T19:11:39.836144+0000","last_clean_scrub_stamp":"2024-09-10T19:12:54.860013+0000","objects_scrubbed":7,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0021624890000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"63'2","reported_seq":95,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-09-10T19:13:55.720705+0000","last_change":"2024-09-10T19:13:00.752806+0000","last_active":"2024-09-10T19:13:55.720705+0000","last_peered":"2024-09-10T19:13:55.720705+0000","last_clean":"2024-09-10T19:13:55.720705+0000","last_became_active":"2024-09-10T19:10:07.684784+0000","last_became_peered":"2024-09-10T19:10:07.684784+0000","last_unstale":"2024-09-10T19:13:55.720705+0000","last_undegraded":"2024-09-10T19:13:55.720705+0000","last_fullsized":"2024-09-10T19:13:55.720705+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'2","last_scrub_stamp":"2024-09-10T19:13:00.752715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:13:00.752715+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:14:06.113040+0000","scrub_duration":0.0083291979999999995,"objects_trimmed":0,"snaptrim_duration":0.00075522999999999996,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":101,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-09-10T19:14:07.061212+0000","last_change":"2024-09-10T19:13:50.411573+0000","last_active":"2024-09-10T19:14:07.061212+0000","last_peered":"2024-09-10T19:14:07.061212+0000","last_clean":"2024-09-10T19:14:07.061212+0000","last_became_active":"2024-09-10T19:10:07.687596+0000","last_became_peered":"2024-09-10T19:10:07.687596+0000","last_unstale":"2024-09-10T19:14:07.061212+0000","last_undegraded":"2024-09-10T19:14:07.061212+0000","last_fullsized":"2024-09-10T19:14:07.061212+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:13:50.411500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:13:50.411500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:15:20.319236+0000","scrub_duration":0.0032604869999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"74'79","reported_seq":133,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-09-10T19:13:55.811320+0000","last_change":"2024-09-10T19:13:46.419002+0000","last_active":"2024-09-10T19:13:55.811320+0000","last_peered":"2024-09-10T19:13:55.811320+0000","last_clean":"2024-09-10T19:13:55.811320+0000","last_became_active":"2024-09-10T19:09:56.634919+0000","last_became_peered":"2024-09-10T19:09:56.634919+0000","last_unstale":"2024-09-10T19:13:55.811320+0000","last_undegraded":"2024-09-10T19:13:55.811320+0000","last_fullsized":"2024-09-10T19:13:55.811320+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'79","last_scrub_stamp":"2024-09-10T19:13:46.418921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:08:00.699315+0000","last_clean_scrub_stamp":"2024-09-10T19:13:46.418921+0000","objects_scrubbed":2,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:14:48.812041+0000","scrub_duration":0.0029827999999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":84,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-09-10T19:13:55.801584+0000","last_change":"2024-09-10T19:13:47.453309+0000","last_active":"2024-09-10T19:13:55.801584+0000","last_peered":"2024-09-10T19:13:55.801584+0000","last_clean":"2024-09-10T19:13:55.801584+0000","last_became_active":"2024-09-10T19:10:11.694986+0000","last_became_peered":"2024-09-10T19:10:11.694986+0000","last_unstale":"2024-09-10T19:13:55.801584+0000","last_undegraded":"2024-09-10T19:13:55.801584+0000","last_fullsized":"2024-09-10T19:13:55.801584+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:13:47.453228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:13:47.453228+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:15:07.672581+0000","scrub_duration":0.010282463,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":85,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-09-10T19:14:07.060394+0000","last_change":"2024-09-10T19:13:05.390398+0000","last_active":"2024-09-10T19:14:07.060394+0000","last_peered":"2024-09-10T19:14:07.060394+0000","last_clean":"2024-09-10T19:14:07.060394+0000","last_became_active":"2024-09-10T19:10:13.720647+0000","last_became_peered":"2024-09-10T19:10:13.720647+0000","last_unstale":"2024-09-10T19:14:07.060394+0000","last_undegraded":"2024-09-10T19:14:07.060394+0000","last_fullsized":"2024-09-10T19:14:07.060394+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:13:05.390317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:13:05.390317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0058030640000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.9","version":"118'9802","reported_seq":9865,"reported_epoch":118,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2024-09-10T19:14:04.722369+0000","last_change":"2024-09-10T19:13:55.963920+0000","last_active":"2024-09-10T19:14:04.722369+0000","last_peered":"2024-09-10T19:14:04.722369+0000","last_clean":"2024-09-10T19:12:45.783536+0000","last_became_active":"2024-09-10T19:13:55.890181+0000","last_became_peered":"2024-09-10T19:13:55.890181+0000","last_unstale":"2024-09-10T19:14:04.722369+0000","last_undegraded":"2024-09-10T19:13:55.954251+0000","last_fullsized":"2024-09-10T19:13:55.708743+0000","mapping_epoch":116,"log_start":"104'5085","ondisk_log_start":"104'5085","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:12:10.007465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:12:10.007465+0000","last_clean_scrub_stamp":"2024-09-10T19:12:10.007465+0000","objects_scrubbed":0,"log_size":4717,"log_dups_size":0,"ondisk_log_size":4717,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":498860032,"num_objects":7612,"num_object_clones":0,"num_object_copies":22836,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7612,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7612,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9802,"num_write_kb":557248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4],"avail_no_missing":["0","4"],"object_location_counts":[{"shards":"0,4","objects":7612}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"66'19","reported_seq":147,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-09-10T19:14:07.059063+0000","last_change":"2024-09-10T19:13:54.755526+0000","last_active":"2024-09-10T19:14:07.059063+0000","last_peered":"2024-09-10T19:14:07.059063+0000","last_clean":"2024-09-10T19:14:07.059063+0000","last_became_active":"2024-09-10T19:13:54.731274+0000","last_became_peered":"2024-09-10T19:13:54.731274+0000","last_unstale":"2024-09-10T19:14:07.059063+0000","last_undegraded":"2024-09-10T19:14:07.059063+0000","last_fullsized":"2024-09-10T19:14:07.059063+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'19","last_scrub_stamp":"2024-09-10T19:11:42.941375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:11:42.941375+0000","objects_scrubbed":9,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0021647599999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,5],"acting":[6,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":88,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-09-10T19:13:55.706836+0000","last_change":"2024-09-10T19:13:05.404535+0000","last_active":"2024-09-10T19:13:55.706836+0000","last_peered":"2024-09-10T19:13:55.706836+0000","last_clean":"2024-09-10T19:13:55.706836+0000","last_became_active":"2024-09-10T19:10:07.686236+0000","last_became_peered":"2024-09-10T19:10:07.686236+0000","last_unstale":"2024-09-10T19:13:55.706836+0000","last_undegraded":"2024-09-10T19:13:55.706836+0000","last_fullsized":"2024-09-10T19:13:55.706836+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:13:05.404457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:13:05.404457+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:14:11.004757+0000","scrub_duration":0.0037223199999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":84,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-09-10T19:13:55.706437+0000","last_change":"2024-09-10T19:12:55.335929+0000","last_active":"2024-09-10T19:13:55.706437+0000","last_peered":"2024-09-10T19:13:55.706437+0000","last_clean":"2024-09-10T19:13:55.706437+0000","last_became_active":"2024-09-10T19:10:11.699745+0000","last_became_peered":"2024-09-10T19:10:11.699745+0000","last_unstale":"2024-09-10T19:13:55.706437+0000","last_undegraded":"2024-09-10T19:13:55.706437+0000","last_fullsized":"2024-09-10T19:13:55.706437+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:12:55.335834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:12:55.335834+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:14:15.218512+0000","scrub_duration":0.0060173240000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":82,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-09-10T19:13:55.811315+0000","last_change":"2024-09-10T19:13:14.311675+0000","last_active":"2024-09-10T19:13:55.811315+0000","last_peered":"2024-09-10T19:13:55.811315+0000","last_clean":"2024-09-10T19:13:55.811315+0000","last_became_active":"2024-09-10T19:10:13.719973+0000","last_became_peered":"2024-09-10T19:10:13.719973+0000","last_unstale":"2024-09-10T19:13:55.811315+0000","last_undegraded":"2024-09-10T19:13:55.811315+0000","last_fullsized":"2024-09-10T19:13:55.811315+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:13:14.311620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:13:14.311620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:14:23.452754+0000","scrub_duration":0.0054101059999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.8","version":"118'9940","reported_seq":10340,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-09-10T19:14:04.686824+0000","last_change":"2024-09-10T19:13:49.740030+0000","last_active":"2024-09-10T19:14:04.686824+0000","last_peered":"2024-09-10T19:14:04.686824+0000","last_clean":"2024-09-10T19:14:04.686824+0000","last_became_active":"2024-09-10T19:12:11.031441+0000","last_became_peered":"2024-09-10T19:12:11.031441+0000","last_unstale":"2024-09-10T19:14:04.686824+0000","last_undegraded":"2024-09-10T19:14:04.686824+0000","last_fullsized":"2024-09-10T19:14:04.686824+0000","mapping_epoch":86,"log_start":"101'4761","ondisk_log_start":"101'4761","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"113'8888","last_scrub_stamp":"2024-09-10T19:13:49.739947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:12:10.007465+0000","last_clean_scrub_stamp":"2024-09-10T19:13:49.739947+0000","objects_scrubbed":8760,"log_size":5179,"log_dups_size":0,"ondisk_log_size":5179,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:14:53.397517+0000","scrub_duration":3.4609629229999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":505675776,"num_objects":7716,"num_object_clones":0,"num_object_copies":23148,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7716,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9940,"num_write_kb":564992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.b","version":"66'9","reported_seq":119,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-09-10T19:13:55.710424+0000","last_change":"2024-09-10T19:13:54.736863+0000","last_active":"2024-09-10T19:13:55.710424+0000","last_peered":"2024-09-10T19:13:55.710424+0000","last_clean":"2024-09-10T19:13:55.710424+0000","last_became_active":"2024-09-10T19:13:54.710613+0000","last_became_peered":"2024-09-10T19:13:54.710613+0000","last_unstale":"2024-09-10T19:13:55.710424+0000","last_undegraded":"2024-09-10T19:13:55.710424+0000","last_fullsized":"2024-09-10T19:13:55.710424+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'9","last_scrub_stamp":"2024-09-10T19:11:22.870369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:11:22.870369+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0017468729999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":41,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-09-10T19:14:07.013255+0000","last_change":"2024-09-10T19:13:54.708605+0000","last_active":"2024-09-10T19:14:07.013255+0000","last_peered":"2024-09-10T19:14:07.013255+0000","last_clean":"2024-09-10T19:14:07.013255+0000","last_became_active":"2024-09-10T19:13:54.708351+0000","last_became_peered":"2024-09-10T19:13:54.708351+0000","last_unstale":"2024-09-10T19:14:07.013255+0000","last_undegraded":"2024-09-10T19:14:07.013255+0000","last_fullsized":"2024-09-10T19:14:07.013255+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:11:44.254513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:11:44.254513+0000","last_clean_scrub_stamp":"2024-09-10T19:11:44.254513+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":84,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-09-10T19:13:55.706406+0000","last_change":"2024-09-10T19:13:08.398177+0000","last_active":"2024-09-10T19:13:55.706406+0000","last_peered":"2024-09-10T19:13:55.706406+0000","last_clean":"2024-09-10T19:13:55.706406+0000","last_became_active":"2024-09-10T19:10:11.694405+0000","last_became_peered":"2024-09-10T19:10:11.694405+0000","last_unstale":"2024-09-10T19:13:55.706406+0000","last_undegraded":"2024-09-10T19:13:55.706406+0000","last_fullsized":"2024-09-10T19:13:55.706406+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:13:08.398036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:13:08.398036+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:14:34.880713+0000","scrub_duration":0.016845473999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":82,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-09-10T19:13:55.711930+0000","last_change":"2024-09-10T19:12:56.145358+0000","last_active":"2024-09-10T19:13:55.711930+0000","last_peered":"2024-09-10T19:13:55.711930+0000","last_clean":"2024-09-10T19:13:55.711930+0000","last_became_active":"2024-09-10T19:10:13.714952+0000","last_became_peered":"2024-09-10T19:10:13.714952+0000","last_unstale":"2024-09-10T19:13:55.711930+0000","last_undegraded":"2024-09-10T19:13:55.711930+0000","last_fullsized":"2024-09-10T19:13:55.711930+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:12:56.145218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:12:56.145218+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:14:21.830535+0000","scrub_duration":0.0058765429999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.b","version":"121'10103","reported_seq":13929,"reported_epoch":121,"state":"active+undersized+degraded+remapped+backfilling","last_fresh":"2024-09-10T19:14:09.647222+0000","last_change":"2024-09-10T19:13:55.833967+0000","last_active":"2024-09-10T19:14:09.647222+0000","last_peered":"2024-09-10T19:14:09.647222+0000","last_clean":"2024-09-10T19:12:45.807438+0000","last_became_active":"2024-09-10T19:13:55.804993+0000","last_became_peered":"2024-09-10T19:13:55.804993+0000","last_unstale":"2024-09-10T19:14:09.647222+0000","last_undegraded":"2024-09-10T19:13:55.727192+0000","last_fullsized":"2024-09-10T19:13:55.710998+0000","mapping_epoch":116,"log_start":"104'5041","ondisk_log_start":"104'5041","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:12:10.007465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:12:10.007465+0000","last_clean_scrub_stamp":"2024-09-10T19:12:10.007465+0000","objects_scrubbed":0,"log_size":5062,"log_dups_size":0,"ondisk_log_size":5062,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":484638720,"num_objects":7395,"num_object_clones":0,"num_object_copies":22185,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5622,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7395,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10103,"num_write_kb":559936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1883,"num_bytes_recovered":123404288,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6],"avail_no_missing":["5","6"],"object_location_counts":[{"shards":"5,6","objects":7395}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.8","version":"66'15","reported_seq":114,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-09-10T19:14:07.058471+0000","last_change":"2024-09-10T19:12:50.449684+0000","last_active":"2024-09-10T19:14:07.058471+0000","last_peered":"2024-09-10T19:14:07.058471+0000","last_clean":"2024-09-10T19:14:07.058471+0000","last_became_active":"2024-09-10T19:10:09.820950+0000","last_became_peered":"2024-09-10T19:10:09.820950+0000","last_unstale":"2024-09-10T19:14:07.058471+0000","last_undegraded":"2024-09-10T19:14:07.058471+0000","last_fullsized":"2024-09-10T19:14:07.058471+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'15","last_scrub_stamp":"2024-09-10T19:12:50.449596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:12:50.449596+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:14:18.973373+0000","scrub_duration":0.0055729239999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":88,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-09-10T19:13:55.707002+0000","last_change":"2024-09-10T19:12:48.346908+0000","last_active":"2024-09-10T19:13:55.707002+0000","last_peered":"2024-09-10T19:13:55.707002+0000","last_clean":"2024-09-10T19:13:55.707002+0000","last_became_active":"2024-09-10T19:10:07.682286+0000","last_became_peered":"2024-09-10T19:10:07.682286+0000","last_unstale":"2024-09-10T19:13:55.707002+0000","last_undegraded":"2024-09-10T19:13:55.707002+0000","last_fullsized":"2024-09-10T19:13:55.707002+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:12:48.346798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:12:48.346798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.00606165,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.9","version":"80'9","reported_seq":140,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-09-10T19:13:56.948941+0000","last_change":"2024-09-10T19:13:43.358984+0000","last_active":"2024-09-10T19:13:56.948941+0000","last_peered":"2024-09-10T19:13:56.948941+0000","last_clean":"2024-09-10T19:13:56.948941+0000","last_became_active":"2024-09-10T19:10:11.702878+0000","last_became_peered":"2024-09-10T19:10:11.702878+0000","last_unstale":"2024-09-10T19:13:56.948941+0000","last_undegraded":"2024-09-10T19:13:56.948941+0000","last_fullsized":"2024-09-10T19:13:56.948941+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"80'9","last_scrub_stamp":"2024-09-10T19:13:43.358901+0000","last_deep_scrub":"80'9","last_deep_scrub_stamp":"2024-09-10T19:11:35.174218+0000","last_clean_scrub_stamp":"2024-09-10T19:13:43.358901+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:15:04.578959+0000","scrub_duration":0.0066143260000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":85,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-09-10T19:14:07.060333+0000","last_change":"2024-09-10T19:12:57.392898+0000","last_active":"2024-09-10T19:14:07.060333+0000","last_peered":"2024-09-10T19:14:07.060333+0000","last_clean":"2024-09-10T19:14:07.060333+0000","last_became_active":"2024-09-10T19:10:14.053875+0000","last_became_peered":"2024-09-10T19:10:14.053875+0000","last_unstale":"2024-09-10T19:14:07.060333+0000","last_undegraded":"2024-09-10T19:14:07.060333+0000","last_fullsized":"2024-09-10T19:14:07.060333+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:12:57.392844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:12:57.392844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0046180839999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.a","version":"118'9964","reported_seq":10349,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-09-10T19:14:04.751661+0000","last_change":"2024-09-10T19:13:44.230819+0000","last_active":"2024-09-10T19:14:04.751661+0000","last_peered":"2024-09-10T19:14:04.751661+0000","last_clean":"2024-09-10T19:14:04.751661+0000","last_became_active":"2024-09-10T19:12:11.034838+0000","last_became_peered":"2024-09-10T19:12:11.034838+0000","last_unstale":"2024-09-10T19:14:04.751661+0000","last_undegraded":"2024-09-10T19:14:04.751661+0000","last_fullsized":"2024-09-10T19:14:04.751661+0000","mapping_epoch":86,"log_start":"105'5719","ondisk_log_start":"105'5719","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"112'8534","last_scrub_stamp":"2024-09-10T19:13:44.230739+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:12:10.007465+0000","last_clean_scrub_stamp":"2024-09-10T19:13:44.230739+0000","objects_scrubbed":8364,"log_size":4245,"log_dups_size":0,"ondisk_log_size":4245,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:15:06.184625+0000","scrub_duration":4.047209821,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":506331136,"num_objects":7726,"num_object_clones":0,"num_object_copies":23178,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7726,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9964,"num_write_kb":566080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.9","version":"66'12","reported_seq":130,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-09-10T19:14:07.012469+0000","last_change":"2024-09-10T19:13:54.760391+0000","last_active":"2024-09-10T19:14:07.012469+0000","last_peered":"2024-09-10T19:14:07.012469+0000","last_clean":"2024-09-10T19:14:07.012469+0000","last_became_active":"2024-09-10T19:13:54.718152+0000","last_became_peered":"2024-09-10T19:13:54.718152+0000","last_unstale":"2024-09-10T19:14:07.012469+0000","last_undegraded":"2024-09-10T19:14:07.012469+0000","last_fullsized":"2024-09-10T19:14:07.012469+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'12","last_scrub_stamp":"2024-09-10T19:11:45.012377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:11:45.012377+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0019105979999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.f","version":"59'2","reported_seq":106,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-09-10T19:14:07.012942+0000","last_change":"2024-09-10T19:12:50.437332+0000","last_active":"2024-09-10T19:14:07.012942+0000","last_peered":"2024-09-10T19:14:07.012942+0000","last_clean":"2024-09-10T19:14:07.012942+0000","last_became_active":"2024-09-10T19:10:07.681719+0000","last_became_peered":"2024-09-10T19:10:07.681719+0000","last_unstale":"2024-09-10T19:14:07.012942+0000","last_undegraded":"2024-09-10T19:14:07.012942+0000","last_fullsized":"2024-09-10T19:14:07.012942+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'2","last_scrub_stamp":"2024-09-10T19:12:50.437245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:12:50.437245+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0070075770000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":14,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":91,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-09-10T19:13:55.706070+0000","last_change":"2024-09-10T19:13:54.715893+0000","last_active":"2024-09-10T19:13:55.706070+0000","last_peered":"2024-09-10T19:13:55.706070+0000","last_clean":"2024-09-10T19:13:55.706070+0000","last_became_active":"2024-09-10T19:13:54.715528+0000","last_became_peered":"2024-09-10T19:13:54.715528+0000","last_unstale":"2024-09-10T19:13:55.706070+0000","last_undegraded":"2024-09-10T19:13:55.706070+0000","last_fullsized":"2024-09-10T19:13:55.706070+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:11:46.587846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:11:46.587846+0000","last_clean_scrub_stamp":"2024-09-10T19:11:46.587846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001267142,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":89,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-09-10T19:13:55.710312+0000","last_change":"2024-09-10T19:13:54.712104+0000","last_active":"2024-09-10T19:13:55.710312+0000","last_peered":"2024-09-10T19:13:55.710312+0000","last_clean":"2024-09-10T19:13:55.710312+0000","last_became_active":"2024-09-10T19:13:54.711804+0000","last_became_peered":"2024-09-10T19:13:54.711804+0000","last_unstale":"2024-09-10T19:13:55.710312+0000","last_undegraded":"2024-09-10T19:13:55.710312+0000","last_fullsized":"2024-09-10T19:13:55.710312+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:11:42.723437+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:11:42.723437+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001385631,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.16","version":"66'9","reported_seq":112,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-09-10T19:13:55.709537+0000","last_change":"2024-09-10T19:13:50.172901+0000","last_active":"2024-09-10T19:13:55.709537+0000","last_peered":"2024-09-10T19:13:55.709537+0000","last_clean":"2024-09-10T19:13:55.709537+0000","last_became_active":"2024-09-10T19:10:09.822994+0000","last_became_peered":"2024-09-10T19:10:09.822994+0000","last_unstale":"2024-09-10T19:13:55.709537+0000","last_undegraded":"2024-09-10T19:13:55.709537+0000","last_fullsized":"2024-09-10T19:13:55.709537+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'9","last_scrub_stamp":"2024-09-10T19:13:50.172825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:13:50.172825+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:14:55.898340+0000","scrub_duration":0.0094381619999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":95,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-09-10T19:13:55.706278+0000","last_change":"2024-09-10T19:13:54.716196+0000","last_active":"2024-09-10T19:13:55.706278+0000","last_peered":"2024-09-10T19:13:55.706278+0000","last_clean":"2024-09-10T19:13:55.706278+0000","last_became_active":"2024-09-10T19:13:54.715910+0000","last_became_peered":"2024-09-10T19:13:54.715910+0000","last_unstale":"2024-09-10T19:13:55.706278+0000","last_undegraded":"2024-09-10T19:13:55.706278+0000","last_fullsized":"2024-09-10T19:13:55.706278+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:11:41.458523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:11:41.458523+0000","last_clean_scrub_stamp":"2024-09-10T19:11:41.458523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0012499830000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,4],"acting":[2,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":91,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-09-10T19:13:55.711474+0000","last_change":"2024-09-10T19:13:54.712640+0000","last_active":"2024-09-10T19:13:55.711474+0000","last_peered":"2024-09-10T19:13:55.711474+0000","last_clean":"2024-09-10T19:13:55.711474+0000","last_became_active":"2024-09-10T19:13:54.712275+0000","last_became_peered":"2024-09-10T19:13:54.712275+0000","last_unstale":"2024-09-10T19:13:55.711474+0000","last_undegraded":"2024-09-10T19:13:55.711474+0000","last_fullsized":"2024-09-10T19:13:55.711474+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:12:11.619202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:12:11.619202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001950834,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"66'1","reported_seq":83,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-09-10T19:13:55.706543+0000","last_change":"2024-09-10T19:12:54.361402+0000","last_active":"2024-09-10T19:13:55.706543+0000","last_peered":"2024-09-10T19:13:55.706543+0000","last_clean":"2024-09-10T19:13:55.706543+0000","last_became_active":"2024-09-10T19:10:13.717001+0000","last_became_peered":"2024-09-10T19:10:13.717001+0000","last_unstale":"2024-09-10T19:13:55.706543+0000","last_undegraded":"2024-09-10T19:13:55.706543+0000","last_fullsized":"2024-09-10T19:13:55.706543+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'1","last_scrub_stamp":"2024-09-10T19:12:54.361312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:12:54.361312+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:14:07.286286+0000","scrub_duration":0.0039806390000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":98,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-09-10T19:14:07.060877+0000","last_change":"2024-09-10T19:13:54.724442+0000","last_active":"2024-09-10T19:14:07.060877+0000","last_peered":"2024-09-10T19:14:07.060877+0000","last_clean":"2024-09-10T19:14:07.060877+0000","last_became_active":"2024-09-10T19:13:54.724027+0000","last_became_peered":"2024-09-10T19:13:54.724027+0000","last_unstale":"2024-09-10T19:14:07.060877+0000","last_undegraded":"2024-09-10T19:14:07.060877+0000","last_fullsized":"2024-09-10T19:14:07.060877+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:11:32.859106+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:11:32.859106+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0014403980000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,5],"acting":[6,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"66'6","reported_seq":110,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-09-10T19:13:55.711603+0000","last_change":"2024-09-10T19:13:55.370749+0000","last_active":"2024-09-10T19:13:55.711603+0000","last_peered":"2024-09-10T19:13:55.711603+0000","last_clean":"2024-09-10T19:13:55.711603+0000","last_became_active":"2024-09-10T19:13:54.717949+0000","last_became_peered":"2024-09-10T19:13:54.717949+0000","last_unstale":"2024-09-10T19:13:55.711603+0000","last_undegraded":"2024-09-10T19:13:55.711603+0000","last_fullsized":"2024-09-10T19:13:55.711603+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'6","last_scrub_stamp":"2024-09-10T19:13:55.370650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:13:55.370650+0000","objects_scrubbed":3,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:15:17.175995+0000","scrub_duration":0.071693797000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":94,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-09-10T19:14:07.061808+0000","last_change":"2024-09-10T19:13:54.730473+0000","last_active":"2024-09-10T19:14:07.061808+0000","last_peered":"2024-09-10T19:14:07.061808+0000","last_clean":"2024-09-10T19:14:07.061808+0000","last_became_active":"2024-09-10T19:13:54.729917+0000","last_became_peered":"2024-09-10T19:13:54.729917+0000","last_unstale":"2024-09-10T19:14:07.061808+0000","last_undegraded":"2024-09-10T19:14:07.061808+0000","last_fullsized":"2024-09-10T19:14:07.061808+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:11:23.264793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:11:23.264793+0000","last_clean_scrub_stamp":"2024-09-10T19:11:23.264793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0015950649999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":82,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-09-10T19:13:55.803817+0000","last_change":"2024-09-10T19:12:51.508398+0000","last_active":"2024-09-10T19:13:55.803817+0000","last_peered":"2024-09-10T19:13:55.803817+0000","last_clean":"2024-09-10T19:13:55.803817+0000","last_became_active":"2024-09-10T19:10:14.054360+0000","last_became_peered":"2024-09-10T19:10:14.054360+0000","last_unstale":"2024-09-10T19:13:55.803817+0000","last_undegraded":"2024-09-10T19:13:55.803817+0000","last_fullsized":"2024-09-10T19:13:55.803817+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:12:51.508263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:12:51.508263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:14:18.164771+0000","scrub_duration":0.001748274,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"80'15","reported_seq":125,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-09-10T19:13:55.710846+0000","last_change":"2024-09-10T19:13:54.749388+0000","last_active":"2024-09-10T19:13:55.710846+0000","last_peered":"2024-09-10T19:13:55.710846+0000","last_clean":"2024-09-10T19:13:55.710846+0000","last_became_active":"2024-09-10T19:13:54.719574+0000","last_became_peered":"2024-09-10T19:13:54.719574+0000","last_unstale":"2024-09-10T19:13:55.710846+0000","last_undegraded":"2024-09-10T19:13:55.710846+0000","last_fullsized":"2024-09-10T19:13:55.710846+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"80'15","last_scrub_stamp":"2024-09-10T19:11:41.719310+0000","last_deep_scrub":"80'15","last_deep_scrub_stamp":"2024-09-10T19:11:41.719310+0000","last_clean_scrub_stamp":"2024-09-10T19:11:41.719310+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0046706430000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":91,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-09-10T19:14:07.060667+0000","last_change":"2024-09-10T19:12:46.506129+0000","last_active":"2024-09-10T19:14:07.060667+0000","last_peered":"2024-09-10T19:14:07.060667+0000","last_clean":"2024-09-10T19:14:07.060667+0000","last_became_active":"2024-09-10T19:10:07.685870+0000","last_became_peered":"2024-09-10T19:10:07.685870+0000","last_unstale":"2024-09-10T19:14:07.060667+0000","last_undegraded":"2024-09-10T19:14:07.060667+0000","last_fullsized":"2024-09-10T19:14:07.060667+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:12:46.506071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:12:46.506071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:14:15.420503+0000","scrub_duration":0.001257935,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"117'10","reported_seq":155,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-09-10T19:14:07.058897+0000","last_change":"2024-09-10T19:13:54.755445+0000","last_active":"2024-09-10T19:14:07.058897+0000","last_peered":"2024-09-10T19:14:07.058897+0000","last_clean":"2024-09-10T19:14:07.058897+0000","last_became_active":"2024-09-10T19:13:54.724780+0000","last_became_peered":"2024-09-10T19:13:54.724780+0000","last_unstale":"2024-09-10T19:14:07.058897+0000","last_undegraded":"2024-09-10T19:14:07.058897+0000","last_fullsized":"2024-09-10T19:14:07.058897+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'8","last_scrub_stamp":"2024-09-10T19:11:50.330303+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:11:50.330303+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0020032069999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3],"acting":[5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":82,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-09-10T19:13:55.709255+0000","last_change":"2024-09-10T19:13:00.212150+0000","last_active":"2024-09-10T19:13:55.709255+0000","last_peered":"2024-09-10T19:13:55.709255+0000","last_clean":"2024-09-10T19:13:55.709255+0000","last_became_active":"2024-09-10T19:10:13.721035+0000","last_became_peered":"2024-09-10T19:10:13.721035+0000","last_unstale":"2024-09-10T19:13:55.709255+0000","last_undegraded":"2024-09-10T19:13:55.709255+0000","last_fullsized":"2024-09-10T19:13:55.709255+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:13:00.212049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:13:00.212049+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:14:30.099769+0000","scrub_duration":0.015146817999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"66'9","reported_seq":105,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-09-10T19:14:07.058902+0000","last_change":"2024-09-10T19:12:52.416838+0000","last_active":"2024-09-10T19:14:07.058902+0000","last_peered":"2024-09-10T19:14:07.058902+0000","last_clean":"2024-09-10T19:14:07.058902+0000","last_became_active":"2024-09-10T19:10:09.823006+0000","last_became_peered":"2024-09-10T19:10:09.823006+0000","last_unstale":"2024-09-10T19:14:07.058902+0000","last_undegraded":"2024-09-10T19:14:07.058902+0000","last_fullsized":"2024-09-10T19:14:07.058902+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'9","last_scrub_stamp":"2024-09-10T19:12:52.416748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:12:52.416748+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:14:10.476379+0000","scrub_duration":0.0047220839999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":88,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-09-10T19:13:55.710568+0000","last_change":"2024-09-10T19:13:01.210932+0000","last_active":"2024-09-10T19:13:55.710568+0000","last_peered":"2024-09-10T19:13:55.710568+0000","last_clean":"2024-09-10T19:13:55.710568+0000","last_became_active":"2024-09-10T19:10:07.688499+0000","last_became_peered":"2024-09-10T19:10:07.688499+0000","last_unstale":"2024-09-10T19:13:55.710568+0000","last_undegraded":"2024-09-10T19:13:55.710568+0000","last_fullsized":"2024-09-10T19:13:55.710568+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:13:01.210848+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:13:01.210848+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:14:17.716737+0000","scrub_duration":0.0049627250000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"66'8","reported_seq":139,"reported_epoch":118,"state":"active+clean","last_fresh":"2024-09-10T19:13:56.959788+0000","last_change":"2024-09-10T19:12:59.723961+0000","last_active":"2024-09-10T19:13:56.959788+0000","last_peered":"2024-09-10T19:13:56.959788+0000","last_clean":"2024-09-10T19:13:56.959788+0000","last_became_active":"2024-09-10T19:10:11.692359+0000","last_became_peered":"2024-09-10T19:10:11.692359+0000","last_unstale":"2024-09-10T19:13:56.959788+0000","last_undegraded":"2024-09-10T19:13:56.959788+0000","last_fullsized":"2024-09-10T19:13:56.959788+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'8","last_scrub_stamp":"2024-09-10T19:12:59.723712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:12:59.723712+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:14:06.688043+0000","scrub_duration":0.0083838109999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":84,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-09-10T19:14:07.012775+0000","last_change":"2024-09-10T19:13:09.463096+0000","last_active":"2024-09-10T19:14:07.012775+0000","last_peered":"2024-09-10T19:14:07.012775+0000","last_clean":"2024-09-10T19:14:07.012775+0000","last_became_active":"2024-09-10T19:10:13.718285+0000","last_became_peered":"2024-09-10T19:10:13.718285+0000","last_unstale":"2024-09-10T19:14:07.012775+0000","last_undegraded":"2024-09-10T19:14:07.012775+0000","last_fullsized":"2024-09-10T19:14:07.012775+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:13:09.463001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:13:09.463001+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:14:23.918770+0000","scrub_duration":0.012199573,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.12","version":"79'13","reported_seq":98,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-09-10T19:14:07.058947+0000","last_change":"2024-09-10T19:13:54.758707+0000","last_active":"2024-09-10T19:14:07.058947+0000","last_peered":"2024-09-10T19:14:07.058947+0000","last_clean":"2024-09-10T19:14:07.058947+0000","last_became_active":"2024-09-10T19:13:54.721560+0000","last_became_peered":"2024-09-10T19:13:54.721560+0000","last_unstale":"2024-09-10T19:14:07.058947+0000","last_undegraded":"2024-09-10T19:14:07.058947+0000","last_fullsized":"2024-09-10T19:14:07.058947+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'13","last_scrub_stamp":"2024-09-10T19:11:22.072400+0000","last_deep_scrub":"79'13","last_deep_scrub_stamp":"2024-09-10T19:11:22.072400+0000","last_clean_scrub_stamp":"2024-09-10T19:11:22.072400+0000","objects_scrubbed":4,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,0],"acting":[6,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.14","version":"59'1","reported_seq":98,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-09-10T19:14:07.061116+0000","last_change":"2024-09-10T19:13:08.917822+0000","last_active":"2024-09-10T19:14:07.061116+0000","last_peered":"2024-09-10T19:14:07.061116+0000","last_clean":"2024-09-10T19:14:07.061116+0000","last_became_active":"2024-09-10T19:10:07.683263+0000","last_became_peered":"2024-09-10T19:10:07.683263+0000","last_unstale":"2024-09-10T19:14:07.061116+0000","last_undegraded":"2024-09-10T19:14:07.061116+0000","last_fullsized":"2024-09-10T19:14:07.061116+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'1","last_scrub_stamp":"2024-09-10T19:13:08.917719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:13:08.917719+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:14:36.333049+0000","scrub_duration":0.0033669569999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":362,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":91,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-09-10T19:13:55.710369+0000","last_change":"2024-09-10T19:13:54.730422+0000","last_active":"2024-09-10T19:13:55.710369+0000","last_peered":"2024-09-10T19:13:55.710369+0000","last_clean":"2024-09-10T19:13:55.710369+0000","last_became_active":"2024-09-10T19:13:54.730185+0000","last_became_peered":"2024-09-10T19:13:54.730185+0000","last_unstale":"2024-09-10T19:13:55.710369+0000","last_undegraded":"2024-09-10T19:13:55.710369+0000","last_fullsized":"2024-09-10T19:13:55.710369+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:12:00.660284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:12:00.660284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0012510729999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":89,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-09-10T19:13:55.708500+0000","last_change":"2024-09-10T19:13:54.715762+0000","last_active":"2024-09-10T19:13:55.708500+0000","last_peered":"2024-09-10T19:13:55.708500+0000","last_clean":"2024-09-10T19:13:55.708500+0000","last_became_active":"2024-09-10T19:13:54.714861+0000","last_became_peered":"2024-09-10T19:13:54.714861+0000","last_unstale":"2024-09-10T19:13:55.708500+0000","last_undegraded":"2024-09-10T19:13:55.708500+0000","last_fullsized":"2024-09-10T19:13:55.708500+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:11:51.920006+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:11:51.920006+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001865058,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,6],"acting":[0,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"66'11","reported_seq":126,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-09-10T19:14:07.012728+0000","last_change":"2024-09-10T19:13:54.758776+0000","last_active":"2024-09-10T19:14:07.012728+0000","last_peered":"2024-09-10T19:14:07.012728+0000","last_clean":"2024-09-10T19:14:07.012728+0000","last_became_active":"2024-09-10T19:13:54.720034+0000","last_became_peered":"2024-09-10T19:13:54.720034+0000","last_unstale":"2024-09-10T19:14:07.012728+0000","last_undegraded":"2024-09-10T19:14:07.012728+0000","last_fullsized":"2024-09-10T19:14:07.012728+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'11","last_scrub_stamp":"2024-09-10T19:11:44.025361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:11:44.025361+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.002036894,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":41,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-09-10T19:14:07.058968+0000","last_change":"2024-09-10T19:13:54.723475+0000","last_active":"2024-09-10T19:14:07.058968+0000","last_peered":"2024-09-10T19:14:07.058968+0000","last_clean":"2024-09-10T19:14:07.058968+0000","last_became_active":"2024-09-10T19:13:54.722947+0000","last_became_peered":"2024-09-10T19:13:54.722947+0000","last_unstale":"2024-09-10T19:14:07.058968+0000","last_undegraded":"2024-09-10T19:14:07.058968+0000","last_fullsized":"2024-09-10T19:14:07.058968+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:12:44.249765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:11:10.031245+0000","last_clean_scrub_stamp":"2024-09-10T19:12:44.249765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,6],"acting":[5,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":41,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-09-10T19:14:07.060438+0000","last_change":"2024-09-10T19:13:54.730370+0000","last_active":"2024-09-10T19:14:07.060438+0000","last_peered":"2024-09-10T19:14:07.060438+0000","last_clean":"2024-09-10T19:14:07.060438+0000","last_became_active":"2024-09-10T19:13:54.729703+0000","last_became_peered":"2024-09-10T19:13:54.729703+0000","last_unstale":"2024-09-10T19:14:07.060438+0000","last_undegraded":"2024-09-10T19:14:07.060438+0000","last_fullsized":"2024-09-10T19:14:07.060438+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:11:31.249343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:11:31.249343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":82,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-09-10T19:13:55.709892+0000","last_change":"2024-09-10T19:12:54.691886+0000","last_active":"2024-09-10T19:13:55.709892+0000","last_peered":"2024-09-10T19:13:55.709892+0000","last_clean":"2024-09-10T19:13:55.709892+0000","last_became_active":"2024-09-10T19:10:13.724201+0000","last_became_peered":"2024-09-10T19:10:13.724201+0000","last_unstale":"2024-09-10T19:13:55.709892+0000","last_undegraded":"2024-09-10T19:13:55.709892+0000","last_fullsized":"2024-09-10T19:13:55.709892+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:12:54.691810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:12:54.691810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:14:17.760354+0000","scrub_duration":0.006702927,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"66'4","reported_seq":105,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-09-10T19:13:55.709634+0000","last_change":"2024-09-10T19:13:54.732227+0000","last_active":"2024-09-10T19:13:55.709634+0000","last_peered":"2024-09-10T19:13:55.709634+0000","last_clean":"2024-09-10T19:13:55.709634+0000","last_became_active":"2024-09-10T19:13:54.711125+0000","last_became_peered":"2024-09-10T19:13:54.711125+0000","last_unstale":"2024-09-10T19:13:55.709634+0000","last_undegraded":"2024-09-10T19:13:55.709634+0000","last_fullsized":"2024-09-10T19:13:55.709634+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'4","last_scrub_stamp":"2024-09-10T19:11:14.816632+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:11:14.816632+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001902504,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,4],"acting":[3,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":91,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-09-10T19:14:07.061913+0000","last_change":"2024-09-10T19:12:48.517928+0000","last_active":"2024-09-10T19:14:07.061913+0000","last_peered":"2024-09-10T19:14:07.061913+0000","last_clean":"2024-09-10T19:14:07.061913+0000","last_became_active":"2024-09-10T19:10:07.687723+0000","last_became_peered":"2024-09-10T19:10:07.687723+0000","last_unstale":"2024-09-10T19:14:07.061913+0000","last_undegraded":"2024-09-10T19:14:07.061913+0000","last_fullsized":"2024-09-10T19:14:07.061913+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:12:48.517853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:12:48.517853+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001753459,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":87,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-09-10T19:14:07.059189+0000","last_change":"2024-09-10T19:13:04.855026+0000","last_active":"2024-09-10T19:14:07.059189+0000","last_peered":"2024-09-10T19:14:07.059189+0000","last_clean":"2024-09-10T19:14:07.059189+0000","last_became_active":"2024-09-10T19:10:11.698536+0000","last_became_peered":"2024-09-10T19:10:11.698536+0000","last_unstale":"2024-09-10T19:14:07.059189+0000","last_undegraded":"2024-09-10T19:14:07.059189+0000","last_fullsized":"2024-09-10T19:14:07.059189+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:13:04.854929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:11:31.837944+0000","last_clean_scrub_stamp":"2024-09-10T19:13:04.854929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:14:32.441023+0000","scrub_duration":0.0012499329999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":82,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-09-10T19:13:55.879365+0000","last_change":"2024-09-10T19:12:45.536856+0000","last_active":"2024-09-10T19:13:55.879365+0000","last_peered":"2024-09-10T19:13:55.879365+0000","last_clean":"2024-09-10T19:13:55.879365+0000","last_became_active":"2024-09-10T19:10:13.718271+0000","last_became_peered":"2024-09-10T19:10:13.718271+0000","last_unstale":"2024-09-10T19:13:55.879365+0000","last_undegraded":"2024-09-10T19:13:55.879365+0000","last_fullsized":"2024-09-10T19:13:55.879365+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:12:45.536799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:12:45.536799+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0036938909999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.11","version":"66'11","reported_seq":105,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-09-10T19:13:55.709977+0000","last_change":"2024-09-10T19:13:05.730096+0000","last_active":"2024-09-10T19:13:55.709977+0000","last_peered":"2024-09-10T19:13:55.709977+0000","last_clean":"2024-09-10T19:13:55.709977+0000","last_became_active":"2024-09-10T19:10:09.821357+0000","last_became_peered":"2024-09-10T19:10:09.821357+0000","last_unstale":"2024-09-10T19:13:55.709977+0000","last_undegraded":"2024-09-10T19:13:55.709977+0000","last_fullsized":"2024-09-10T19:13:55.709977+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'11","last_scrub_stamp":"2024-09-10T19:13:05.729854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:13:05.729854+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:14:32.355799+0000","scrub_duration":0.010667395,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":91,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-09-10T19:14:07.063179+0000","last_change":"2024-09-10T19:13:05.876708+0000","last_active":"2024-09-10T19:14:07.063179+0000","last_peered":"2024-09-10T19:14:07.063179+0000","last_clean":"2024-09-10T19:14:07.063179+0000","last_became_active":"2024-09-10T19:10:07.685223+0000","last_became_peered":"2024-09-10T19:10:07.685223+0000","last_unstale":"2024-09-10T19:14:07.063179+0000","last_undegraded":"2024-09-10T19:14:07.063179+0000","last_fullsized":"2024-09-10T19:14:07.063179+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:13:05.876655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:13:05.876655+0000","last_clean_scrub_stamp":"2024-09-10T19:13:05.876655+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:14:10.896708+0000","scrub_duration":0.0013242589999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":84,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-09-10T19:13:55.801191+0000","last_change":"2024-09-10T19:13:41.326028+0000","last_active":"2024-09-10T19:13:55.801191+0000","last_peered":"2024-09-10T19:13:55.801191+0000","last_clean":"2024-09-10T19:13:55.801191+0000","last_became_active":"2024-09-10T19:10:11.699013+0000","last_became_peered":"2024-09-10T19:10:11.699013+0000","last_unstale":"2024-09-10T19:13:55.801191+0000","last_undegraded":"2024-09-10T19:13:55.801191+0000","last_fullsized":"2024-09-10T19:13:55.801191+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:13:41.325941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:11:57.112092+0000","last_clean_scrub_stamp":"2024-09-10T19:13:41.325941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:15:05.192727+0000","scrub_duration":0.001598676,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":82,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-09-10T19:13:55.710292+0000","last_change":"2024-09-10T19:13:15.812326+0000","last_active":"2024-09-10T19:13:55.710292+0000","last_peered":"2024-09-10T19:13:55.710292+0000","last_clean":"2024-09-10T19:13:55.710292+0000","last_became_active":"2024-09-10T19:10:14.056172+0000","last_became_peered":"2024-09-10T19:10:14.056172+0000","last_unstale":"2024-09-10T19:13:55.710292+0000","last_undegraded":"2024-09-10T19:13:55.710292+0000","last_fullsized":"2024-09-10T19:13:55.710292+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:13:15.812244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:13:15.812244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:14:36.621092+0000","scrub_duration":0.0048044389999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"80'2","reported_seq":84,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-09-10T19:13:55.801523+0000","last_change":"2024-09-10T19:12:57.454088+0000","last_active":"2024-09-10T19:13:55.801523+0000","last_peered":"2024-09-10T19:13:55.801523+0000","last_clean":"2024-09-10T19:13:55.801523+0000","last_became_active":"2024-09-10T19:10:13.718875+0000","last_became_peered":"2024-09-10T19:10:13.718875+0000","last_unstale":"2024-09-10T19:13:55.801523+0000","last_undegraded":"2024-09-10T19:13:55.801523+0000","last_fullsized":"2024-09-10T19:13:55.801523+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"80'2","last_scrub_stamp":"2024-09-10T19:12:57.453993+0000","last_deep_scrub":"80'2","last_deep_scrub_stamp":"2024-09-10T19:11:30.272043+0000","last_clean_scrub_stamp":"2024-09-10T19:12:57.453993+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:14:13.596828+0000","scrub_duration":0.0018294990000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":369,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":91,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-09-10T19:14:07.058583+0000","last_change":"2024-09-10T19:13:14.449643+0000","last_active":"2024-09-10T19:14:07.058583+0000","last_peered":"2024-09-10T19:14:07.058583+0000","last_clean":"2024-09-10T19:14:07.058583+0000","last_became_active":"2024-09-10T19:10:07.689877+0000","last_became_peered":"2024-09-10T19:10:07.689877+0000","last_unstale":"2024-09-10T19:14:07.058583+0000","last_undegraded":"2024-09-10T19:14:07.058583+0000","last_fullsized":"2024-09-10T19:14:07.058583+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:13:14.449593+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:13:14.449593+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:14:27.688141+0000","scrub_duration":0.001195542,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1e","version":"80'15","reported_seq":106,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-09-10T19:13:55.709494+0000","last_change":"2024-09-10T19:12:50.148157+0000","last_active":"2024-09-10T19:13:55.709494+0000","last_peered":"2024-09-10T19:13:55.709494+0000","last_clean":"2024-09-10T19:13:55.709494+0000","last_became_active":"2024-09-10T19:10:09.823061+0000","last_became_peered":"2024-09-10T19:10:09.823061+0000","last_unstale":"2024-09-10T19:13:55.709494+0000","last_undegraded":"2024-09-10T19:13:55.709494+0000","last_fullsized":"2024-09-10T19:13:55.709494+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"80'15","last_scrub_stamp":"2024-09-10T19:12:50.147336+0000","last_deep_scrub":"80'15","last_deep_scrub_stamp":"2024-09-10T19:11:25.883919+0000","last_clean_scrub_stamp":"2024-09-10T19:12:50.147336+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:14:02.846636+0000","scrub_duration":0.0064585019999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1f","version":"66'8","reported_seq":146,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-09-10T19:14:07.063111+0000","last_change":"2024-09-10T19:12:48.889499+0000","last_active":"2024-09-10T19:14:07.063111+0000","last_peered":"2024-09-10T19:14:07.063111+0000","last_clean":"2024-09-10T19:14:07.063111+0000","last_became_active":"2024-09-10T19:10:11.695427+0000","last_became_peered":"2024-09-10T19:10:11.695427+0000","last_unstale":"2024-09-10T19:14:07.063111+0000","last_undegraded":"2024-09-10T19:14:07.063111+0000","last_fullsized":"2024-09-10T19:14:07.063111+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'8","last_scrub_stamp":"2024-09-10T19:12:48.889410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:12:48.889410+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:14:09.607466+0000","scrub_duration":0.001725311,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":41,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-09-10T19:14:07.058776+0000","last_change":"2024-09-10T19:13:54.715498+0000","last_active":"2024-09-10T19:14:07.058776+0000","last_peered":"2024-09-10T19:14:07.058776+0000","last_clean":"2024-09-10T19:14:07.058776+0000","last_became_active":"2024-09-10T19:13:54.715192+0000","last_became_peered":"2024-09-10T19:13:54.715192+0000","last_unstale":"2024-09-10T19:14:07.058776+0000","last_undegraded":"2024-09-10T19:14:07.058776+0000","last_fullsized":"2024-09-10T19:14:07.058776+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:11:30.210603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:11:30.210603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,2],"acting":[5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":88,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-09-10T19:13:55.710156+0000","last_change":"2024-09-10T19:13:02.737323+0000","last_active":"2024-09-10T19:13:55.710156+0000","last_peered":"2024-09-10T19:13:55.710156+0000","last_clean":"2024-09-10T19:13:55.710156+0000","last_became_active":"2024-09-10T19:10:07.682689+0000","last_became_peered":"2024-09-10T19:10:07.682689+0000","last_unstale":"2024-09-10T19:13:55.710156+0000","last_undegraded":"2024-09-10T19:13:55.710156+0000","last_fullsized":"2024-09-10T19:13:55.710156+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:13:02.737236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:13:02.737236+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:14:06.258995+0000","scrub_duration":0.0027813289999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"80'16","reported_seq":157,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-09-10T19:14:07.063153+0000","last_change":"2024-09-10T19:13:54.761453+0000","last_active":"2024-09-10T19:14:07.063153+0000","last_peered":"2024-09-10T19:14:07.063153+0000","last_clean":"2024-09-10T19:14:07.063153+0000","last_became_active":"2024-09-10T19:13:54.713563+0000","last_became_peered":"2024-09-10T19:13:54.713563+0000","last_unstale":"2024-09-10T19:14:07.063153+0000","last_undegraded":"2024-09-10T19:14:07.063153+0000","last_fullsized":"2024-09-10T19:14:07.063153+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"80'16","last_scrub_stamp":"2024-09-10T19:12:51.915777+0000","last_deep_scrub":"80'16","last_deep_scrub_stamp":"2024-09-10T19:11:40.853611+0000","last_clean_scrub_stamp":"2024-09-10T19:12:51.915777+0000","objects_scrubbed":5,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0019885430000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,0],"acting":[6,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":84,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-09-10T19:13:55.709458+0000","last_change":"2024-09-10T19:12:59.195000+0000","last_active":"2024-09-10T19:13:55.709458+0000","last_peered":"2024-09-10T19:13:55.709458+0000","last_clean":"2024-09-10T19:13:55.709458+0000","last_became_active":"2024-09-10T19:10:11.701735+0000","last_became_peered":"2024-09-10T19:10:11.701735+0000","last_unstale":"2024-09-10T19:13:55.709458+0000","last_undegraded":"2024-09-10T19:13:55.709458+0000","last_fullsized":"2024-09-10T19:13:55.709458+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:12:59.194914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:11:28.822327+0000","last_clean_scrub_stamp":"2024-09-10T19:12:59.194914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:14:24.283020+0000","scrub_duration":0.0022130729999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":84,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-09-10T19:14:07.013024+0000","last_change":"2024-09-10T19:13:00.388592+0000","last_active":"2024-09-10T19:14:07.013024+0000","last_peered":"2024-09-10T19:14:07.013024+0000","last_clean":"2024-09-10T19:14:07.013024+0000","last_became_active":"2024-09-10T19:10:14.053619+0000","last_became_peered":"2024-09-10T19:10:14.053619+0000","last_unstale":"2024-09-10T19:14:07.013024+0000","last_undegraded":"2024-09-10T19:14:07.013024+0000","last_fullsized":"2024-09-10T19:14:07.013024+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:13:00.388466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:13:00.388466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:14:17.117310+0000","scrub_duration":0.003120996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":91,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-09-10T19:14:07.059180+0000","last_change":"2024-09-10T19:13:43.205095+0000","last_active":"2024-09-10T19:14:07.059180+0000","last_peered":"2024-09-10T19:14:07.059180+0000","last_clean":"2024-09-10T19:14:07.059180+0000","last_became_active":"2024-09-10T19:10:07.685997+0000","last_became_peered":"2024-09-10T19:10:07.685997+0000","last_unstale":"2024-09-10T19:14:07.059180+0000","last_undegraded":"2024-09-10T19:14:07.059180+0000","last_fullsized":"2024-09-10T19:14:07.059180+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:13:43.205042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:13:43.205042+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:14:47.993594+0000","scrub_duration":0.0013181790000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1c","version":"78'22","reported_seq":151,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-09-10T19:13:55.706710+0000","last_change":"2024-09-10T19:13:54.751066+0000","last_active":"2024-09-10T19:13:55.706710+0000","last_peered":"2024-09-10T19:13:55.706710+0000","last_clean":"2024-09-10T19:13:55.706710+0000","last_became_active":"2024-09-10T19:13:54.715570+0000","last_became_peered":"2024-09-10T19:13:54.715570+0000","last_unstale":"2024-09-10T19:13:55.706710+0000","last_undegraded":"2024-09-10T19:13:55.706710+0000","last_fullsized":"2024-09-10T19:13:55.706710+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"78'22","last_scrub_stamp":"2024-09-10T19:12:45.399973+0000","last_deep_scrub":"78'22","last_deep_scrub_stamp":"2024-09-10T19:11:21.118937+0000","last_clean_scrub_stamp":"2024-09-10T19:12:45.399973+0000","objects_scrubbed":7,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:14:05.680595+0000","scrub_duration":0.0084828799999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":41,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-09-10T19:14:07.012575+0000","last_change":"2024-09-10T19:13:54.709308+0000","last_active":"2024-09-10T19:14:07.012575+0000","last_peered":"2024-09-10T19:14:07.012575+0000","last_clean":"2024-09-10T19:14:07.012575+0000","last_became_active":"2024-09-10T19:13:54.709055+0000","last_became_peered":"2024-09-10T19:13:54.709055+0000","last_unstale":"2024-09-10T19:14:07.012575+0000","last_undegraded":"2024-09-10T19:14:07.012575+0000","last_fullsized":"2024-09-10T19:14:07.012575+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:11:43.221323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:11:43.221323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,5],"acting":[4,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":82,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-09-10T19:13:55.711518+0000","last_change":"2024-09-10T19:12:51.767586+0000","last_active":"2024-09-10T19:13:55.711518+0000","last_peered":"2024-09-10T19:13:55.711518+0000","last_clean":"2024-09-10T19:13:55.711518+0000","last_became_active":"2024-09-10T19:10:14.053698+0000","last_became_peered":"2024-09-10T19:10:14.053698+0000","last_unstale":"2024-09-10T19:13:55.711518+0000","last_undegraded":"2024-09-10T19:13:55.711518+0000","last_fullsized":"2024-09-10T19:13:55.711518+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:12:51.767056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:12:51.767056+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:14:17.907092+0000","scrub_duration":0.006796519,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":88,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-09-10T19:13:55.710557+0000","last_change":"2024-09-10T19:12:50.719167+0000","last_active":"2024-09-10T19:13:55.710557+0000","last_peered":"2024-09-10T19:13:55.710557+0000","last_clean":"2024-09-10T19:13:55.710557+0000","last_became_active":"2024-09-10T19:10:07.682811+0000","last_became_peered":"2024-09-10T19:10:07.682811+0000","last_unstale":"2024-09-10T19:13:55.710557+0000","last_undegraded":"2024-09-10T19:13:55.710557+0000","last_fullsized":"2024-09-10T19:13:55.710557+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:12:50.718663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:12:50.718663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:14:18.173382+0000","scrub_duration":0.0042138399999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1d","version":"66'12","reported_seq":128,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-09-10T19:13:55.711034+0000","last_change":"2024-09-10T19:13:54.758541+0000","last_active":"2024-09-10T19:13:55.711034+0000","last_peered":"2024-09-10T19:13:55.711034+0000","last_clean":"2024-09-10T19:13:55.711034+0000","last_became_active":"2024-09-10T19:13:54.714629+0000","last_became_peered":"2024-09-10T19:13:54.714629+0000","last_unstale":"2024-09-10T19:13:55.711034+0000","last_undegraded":"2024-09-10T19:13:55.711034+0000","last_fullsized":"2024-09-10T19:13:55.711034+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'12","last_scrub_stamp":"2024-09-10T19:11:52.663657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:11:52.663657+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0019136909999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":86,"reported_epoch":120,"state":"active+clean","last_fresh":"2024-09-10T19:14:07.013002+0000","last_change":"2024-09-10T19:13:11.445426+0000","last_active":"2024-09-10T19:14:07.013002+0000","last_peered":"2024-09-10T19:14:07.013002+0000","last_clean":"2024-09-10T19:14:07.013002+0000","last_became_active":"2024-09-10T19:10:11.701155+0000","last_became_peered":"2024-09-10T19:10:11.701155+0000","last_unstale":"2024-09-10T19:14:07.013002+0000","last_undegraded":"2024-09-10T19:14:07.013002+0000","last_fullsized":"2024-09-10T19:14:07.013002+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:13:11.445238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:13:11.445238+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:14:41.165035+0000","scrub_duration":0.0057421110000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":16,"stat_sum":{"num_bytes":7914258456,"num_objects":120763,"num_object_clones":0,"num_object_copies":362289,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":41014,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":120763,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":159995,"num_write_kb":8984193,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4198,"num_bytes_recovered":274923520,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":22393462784,"data_stored":22393454640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76476,"ondisk_log_size":76476,"up":48,"acting":42,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":71,"ondisk_log_size":71,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":177,"num_object_clones":0,"num_object_copies":531,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":177,"num_whiteouts":0,"num_read":695,"num_read_kb":452,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":111,"num_bytes_recovered":2448,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":499712,"data_stored":13224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":444,"ondisk_log_size":444,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":80,"num_read_kb":75,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":389,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1613,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1328,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":27,"num_read_kb":27,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":3984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4,"ondisk_log_size":4,"up":96,"acting":96,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79,"ondisk_log_size":79,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":54,"seq":231928234036,"num_pgs":69,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2845744,"kb_used_data":2544232,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":90899408,"statfs":{"total":95995035648,"available":93080993792,"internally_reserved":0,"allocated":2605293568,"data_stored":2604945148,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 19:13:58 2024","interfaces":[{"interface":"back","average":{"1min":3.7909999999999999,"5min":1.619,"15min":0.89600000000000002},"min":{"1min":0.215,"5min":0.215,"15min":0.215},"max":{"1min":25.640999999999998,"5min":35.551000000000002,"15min":35.551000000000002},"last":2.052},{"interface":"front","average":{"1min":3.0299999999999998,"5min":1.4099999999999999,"15min":0.82799999999999996},"min":{"1min":0.224,"5min":0.223,"15min":0.223},"max":{"1min":27.870999999999999,"5min":35.503999999999998,"15min":35.503999999999998},"last":3.0259999999999998}]},{"osd":1,"last update":"Tue Sep 10 19:11:56 2024","interfaces":[{"interface":"back","average":{"1min":0.52400000000000002,"5min":0.52400000000000002,"15min":0.52400000000000002},"min":{"1min":0.36199999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.70199999999999996,"5min":0.79400000000000004,"15min":0.79400000000000004},"last":0.45600000000000002},{"interface":"front","average":{"1min":0.49199999999999999,"5min":0.51200000000000001,"15min":0.51500000000000001},"min":{"1min":0.28999999999999998,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.68500000000000005,"5min":0.71799999999999997,"15min":0.71799999999999997},"last":1.5}]},{"osd":2,"last update":"Tue Sep 10 19:13:58 2024","interfaces":[{"interface":"back","average":{"1min":2.6709999999999998,"5min":0.93799999999999994,"15min":0.66200000000000003},"min":{"1min":0.25600000000000001,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":31.940999999999999,"5min":31.940999999999999,"15min":31.940999999999999},"last":1.542},{"interface":"front","average":{"1min":2.6989999999999998,"5min":1.014,"15min":0.70399999999999996},"min":{"1min":0.24099999999999999,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":25.056000000000001,"5min":25.056000000000001,"15min":25.056000000000001},"last":0.249}]},{"osd":3,"last update":"Tue Sep 10 19:13:58 2024","interfaces":[{"interface":"back","average":{"1min":2.6509999999999998,"5min":1.5509999999999999,"15min":0.874},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":25.443999999999999,"5min":35.277999999999999,"15min":35.277999999999999},"last":0.26000000000000001},{"interface":"front","average":{"1min":2.8879999999999999,"5min":1.595,"15min":0.878},"min":{"1min":0.30299999999999999,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":25.530000000000001,"5min":36.512999999999998,"15min":36.512999999999998},"last":0.75800000000000001}]},{"osd":4,"last update":"Tue Sep 10 19:13:58 2024","interfaces":[{"interface":"back","average":{"1min":0.59699999999999998,"5min":0.47199999999999998,"15min":0.46000000000000002},"min":{"1min":0.23499999999999999,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":2.863,"5min":2.863,"15min":2.863},"last":0.45700000000000002},{"interface":"front","average":{"1min":0.57599999999999996,"5min":0.47699999999999998,"15min":0.46500000000000002},"min":{"1min":0.30399999999999999,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":1.1020000000000001,"5min":1.1020000000000001,"15min":1.1020000000000001},"last":0.23000000000000001}]},{"osd":5,"last update":"Tue Sep 10 19:13:58 2024","interfaces":[{"interface":"back","average":{"1min":1.085,"5min":0.59299999999999997,"15min":0.51700000000000002},"min":{"1min":0.27700000000000002,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":7.2240000000000002,"5min":7.2240000000000002,"15min":7.2240000000000002},"last":0.27100000000000002},{"interface":"front","average":{"1min":0.626,"5min":0.51000000000000001,"15min":0.48699999999999999},"min":{"1min":0.26600000000000001,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":2.8980000000000001,"5min":2.8980000000000001,"15min":2.8980000000000001},"last":0.48399999999999999}]},{"osd":6,"last update":"Tue Sep 10 19:13:58 2024","interfaces":[{"interface":"back","average":{"1min":0.54600000000000004,"5min":0.51900000000000002,"15min":0.50600000000000001},"min":{"1min":0.29099999999999998,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.91400000000000003,"5min":0.91600000000000004,"15min":0.91600000000000004},"last":0.54600000000000004},{"interface":"front","average":{"1min":0.48899999999999999,"5min":0.48599999999999999,"15min":0.48799999999999999},"min":{"1min":0.28000000000000003,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.83499999999999996,"5min":0.83499999999999996,"15min":0.83499999999999996},"last":0.36099999999999999}]}]},{"osd":6,"up_from":47,"seq":201863462970,"num_pgs":61,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2771172,"kb_used_data":2457176,"kb_used_omap":1,"kb_used_meta":313982,"kb_avail":90973980,"statfs":{"total":95995035648,"available":93157355520,"internally_reserved":0,"allocated":2516148224,"data_stored":2515775851,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":321517912},"hb_peers":[0,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,3],"upper_bound":64},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 19:13:42 2024","interfaces":[{"interface":"back","average":{"1min":0.90700000000000003,"5min":0.58499999999999996,"15min":0.54300000000000004},"min":{"1min":0.19800000000000001,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":2.8050000000000002,"5min":2.8050000000000002,"15min":2.8050000000000002},"last":112.988},{"interface":"front","average":{"1min":0.96599999999999997,"5min":0.62,"15min":0.56899999999999995},"min":{"1min":0.23300000000000001,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":3.4609999999999999,"5min":3.4609999999999999,"15min":3.4609999999999999},"last":113.18300000000001}]},{"osd":1,"last update":"Tue Sep 10 19:12:41 2024","interfaces":[{"interface":"back","average":{"1min":0.59999999999999998,"5min":0.53700000000000003,"15min":0.53100000000000003},"min":{"1min":0.33800000000000002,"5min":0.318,"15min":0.318},"max":{"1min":1.29,"5min":1.29,"15min":1.29},"last":0.26500000000000001},{"interface":"front","average":{"1min":0.69399999999999995,"5min":0.55400000000000005,"15min":0.53800000000000003},"min":{"1min":0.32400000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":2.6859999999999999,"5min":2.6859999999999999,"15min":2.6859999999999999},"last":0.38200000000000001}]},{"osd":2,"last update":"Tue Sep 10 19:13:42 2024","interfaces":[{"interface":"back","average":{"1min":0.84099999999999997,"5min":0.59499999999999997,"15min":0.56100000000000005},"min":{"1min":0.248,"5min":0.248,"15min":0.248},"max":{"1min":2.0049999999999999,"5min":2.0049999999999999,"15min":2.0049999999999999},"last":112.864},{"interface":"front","average":{"1min":0.97999999999999998,"5min":0.61099999999999999,"15min":0.54300000000000004},"min":{"1min":0.28699999999999998,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":3.5659999999999998,"5min":3.5659999999999998,"15min":3.5659999999999998},"last":112.822}]},{"osd":3,"last update":"Tue Sep 10 19:13:42 2024","interfaces":[{"interface":"back","average":{"1min":0.97199999999999998,"5min":0.628,"15min":0.55300000000000005},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":2.3119999999999998,"5min":2.3119999999999998,"15min":2.3119999999999998},"last":125.56399999999999},{"interface":"front","average":{"1min":1.0529999999999999,"5min":0.68300000000000005,"15min":0.60499999999999998},"min":{"1min":0.23200000000000001,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":3.5840000000000001,"5min":3.5840000000000001,"15min":3.5840000000000001},"last":125.47}]},{"osd":4,"last update":"Tue Sep 10 19:13:42 2024","interfaces":[{"interface":"back","average":{"1min":0.46200000000000002,"5min":0.39700000000000002,"15min":0.36399999999999999},"min":{"1min":0.20699999999999999,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":1.6879999999999999,"5min":1.6879999999999999,"15min":1.6879999999999999},"last":0.32900000000000001},{"interface":"front","average":{"1min":0.60099999999999998,"5min":0.45800000000000002,"15min":0.40200000000000002},"min":{"1min":0.25800000000000001,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":2.0710000000000002,"5min":2.0710000000000002,"15min":2.0710000000000002},"last":0.34899999999999998}]},{"osd":5,"last update":"Tue Sep 10 19:13:42 2024","interfaces":[{"interface":"back","average":{"1min":0.41999999999999998,"5min":0.41199999999999998,"15min":0.38700000000000001},"min":{"1min":0.248,"5min":0.248,"15min":0.248},"max":{"1min":1.571,"5min":1.571,"15min":1.571},"last":0.36899999999999999},{"interface":"front","average":{"1min":0.61299999999999999,"5min":0.48099999999999998,"15min":0.40899999999999997},"min":{"1min":0.27500000000000002,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":2.298,"5min":2.298,"15min":2.298},"last":0.39600000000000002}]},{"osd":7,"last update":"Tue Sep 10 19:14:00 2024","interfaces":[{"interface":"back","average":{"1min":0.47499999999999998,"5min":0.47099999999999997,"15min":0.45500000000000002},"min":{"1min":0.27500000000000002,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":1.6559999999999999,"5min":1.6559999999999999,"15min":1.6559999999999999},"last":0.379},{"interface":"front","average":{"1min":0.64300000000000002,"5min":0.53200000000000003,"15min":0.47399999999999998},"min":{"1min":0.29299999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":2.3260000000000001,"5min":2.3260000000000001,"15min":2.3260000000000001},"last":0.41799999999999998}]}]},{"osd":5,"up_from":39,"seq":167503724607,"num_pgs":60,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2210692,"kb_used_data":1896696,"kb_used_omap":1,"kb_used_meta":313982,"kb_avail":91534460,"statfs":{"total":95995035648,"available":93731287040,"internally_reserved":0,"allocated":1942216704,"data_stored":1941876653,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":321517912},"hb_peers":[0,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,3,3,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 19:13:11 2024","interfaces":[{"interface":"back","average":{"1min":0.65500000000000003,"5min":0.51900000000000002,"15min":0.48899999999999999},"min":{"1min":0.223,"5min":0.223,"15min":0.223},"max":{"1min":2.2959999999999998,"5min":2.2959999999999998,"15min":2.2959999999999998},"last":1.3520000000000001},{"interface":"front","average":{"1min":0.64100000000000001,"5min":0.52000000000000002,"15min":0.48699999999999999},"min":{"1min":0.21299999999999999,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":2.8700000000000001,"5min":2.8700000000000001,"15min":2.8700000000000001},"last":1.113}]},{"osd":1,"last update":"Tue Sep 10 19:12:09 2024","interfaces":[{"interface":"back","average":{"1min":0.48199999999999998,"5min":0.51500000000000001,"15min":0.50700000000000001},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.64600000000000002,"5min":0.72799999999999998,"15min":0.72799999999999998},"last":0.51000000000000001},{"interface":"front","average":{"1min":0.47299999999999998,"5min":0.47899999999999998,"15min":0.47399999999999998},"min":{"1min":0.34399999999999997,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.61599999999999999,"5min":0.68799999999999994,"15min":0.68799999999999994},"last":3.1960000000000002}]},{"osd":2,"last update":"Tue Sep 10 19:13:11 2024","interfaces":[{"interface":"back","average":{"1min":0.79800000000000004,"5min":0.55600000000000005,"15min":0.501},"min":{"1min":0.253,"5min":0.253,"15min":0.253},"max":{"1min":2.2679999999999998,"5min":2.2679999999999998,"15min":2.2679999999999998},"last":0.35899999999999999},{"interface":"front","average":{"1min":0.49099999999999999,"5min":0.505,"15min":0.49299999999999999},"min":{"1min":0.255,"5min":0.255,"15min":0.255},"max":{"1min":0.90700000000000003,"5min":0.90700000000000003,"15min":0.90700000000000003},"last":0.81399999999999995}]},{"osd":3,"last update":"Tue Sep 10 19:13:11 2024","interfaces":[{"interface":"back","average":{"1min":0.79700000000000004,"5min":0.57799999999999996,"15min":0.53300000000000003},"min":{"1min":0.28100000000000003,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":2.5449999999999999,"5min":2.5449999999999999,"15min":2.5449999999999999},"last":0.32600000000000001},{"interface":"front","average":{"1min":0.64900000000000002,"5min":0.56699999999999995,"15min":0.53700000000000003},"min":{"1min":0.248,"5min":0.248,"15min":0.248},"max":{"1min":2.0270000000000001,"5min":2.0270000000000001,"15min":2.0270000000000001},"last":0.26700000000000002}]},{"osd":4,"last update":"Tue Sep 10 19:13:11 2024","interfaces":[{"interface":"back","average":{"1min":0.40899999999999997,"5min":0.39200000000000002,"15min":0.36599999999999999},"min":{"1min":0.23699999999999999,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":0.93899999999999995,"5min":0.93899999999999995,"15min":0.93899999999999995},"last":1.0629999999999999},{"interface":"front","average":{"1min":0.63100000000000001,"5min":0.45000000000000001,"15min":0.39000000000000001},"min":{"1min":0.29199999999999998,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":2.5179999999999998,"5min":2.5179999999999998,"15min":2.5179999999999998},"last":0.66800000000000004}]},{"osd":6,"last update":"Tue Sep 10 19:13:37 2024","interfaces":[{"interface":"back","average":{"1min":0.61499999999999999,"5min":0.49299999999999999,"15min":0.438},"min":{"1min":0.27100000000000002,"5min":0.223,"15min":0.223},"max":{"1min":2.8540000000000001,"5min":2.8540000000000001,"15min":2.8540000000000001},"last":1.3799999999999999},{"interface":"front","average":{"1min":0.59699999999999998,"5min":0.47699999999999998,"15min":0.42899999999999999},"min":{"1min":0.32600000000000001,"5min":0.25,"15min":0.25},"max":{"1min":1.2010000000000001,"5min":1.2010000000000001,"15min":1.2010000000000001},"last":0.64500000000000002}]},{"osd":7,"last update":"Tue Sep 10 19:13:57 2024","interfaces":[{"interface":"back","average":{"1min":0.72299999999999998,"5min":0.54300000000000004,"15min":0.52100000000000002},"min":{"1min":0.316,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":1.9970000000000001,"5min":1.9970000000000001,"15min":1.9970000000000001},"last":0.86899999999999999},{"interface":"front","average":{"1min":0.60799999999999998,"5min":0.50900000000000001,"15min":0.48899999999999999},"min":{"1min":0.30499999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":2.0099999999999998,"5min":2.0099999999999998,"15min":2.0099999999999998},"last":0.313}]}]},{"osd":4,"up_from":32,"seq":137438953539,"num_pgs":61,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2640268,"kb_used_data":2338748,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":91104884,"statfs":{"total":95995035648,"available":93291401216,"internally_reserved":0,"allocated":2394877952,"data_stored":2394532550,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 19:13:55 2024","interfaces":[{"interface":"back","average":{"1min":0.61499999999999999,"5min":0.54600000000000004,"15min":0.54300000000000004},"min":{"1min":0.22900000000000001,"5min":0.22,"15min":0.22},"max":{"1min":2.157,"5min":2.157,"15min":2.157},"last":2.9199999999999999},{"interface":"front","average":{"1min":0.60399999999999998,"5min":0.54600000000000004,"15min":0.53600000000000003},"min":{"1min":0.20999999999999999,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":2.1070000000000002,"5min":2.5499999999999998,"15min":2.5499999999999998},"last":3.0579999999999998}]},{"osd":1,"last update":"Tue Sep 10 19:11:49 2024","interfaces":[{"interface":"back","average":{"1min":0.47999999999999998,"5min":0.503,"15min":0.50900000000000001},"min":{"1min":0.29999999999999999,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.72199999999999998,"5min":0.83499999999999996,"15min":0.83499999999999996},"last":0.997},{"interface":"front","average":{"1min":0.51000000000000001,"5min":0.51700000000000002,"15min":0.52200000000000002},"min":{"1min":0.27500000000000002,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.68999999999999995,"5min":0.77100000000000002,"15min":0.77100000000000002},"last":0.67400000000000004}]},{"osd":2,"last update":"Tue Sep 10 19:13:55 2024","interfaces":[{"interface":"back","average":{"1min":0.83099999999999996,"5min":0.56999999999999995,"15min":0.54900000000000004},"min":{"1min":0.27400000000000002,"5min":0.221,"15min":0.221},"max":{"1min":2.54,"5min":2.54,"15min":2.54},"last":0.63700000000000001},{"interface":"front","average":{"1min":0.753,"5min":0.56499999999999995,"15min":0.54700000000000004},"min":{"1min":0.27100000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":2.8679999999999999,"5min":2.8679999999999999,"15min":2.8679999999999999},"last":0.23400000000000001}]},{"osd":3,"last update":"Tue Sep 10 19:13:55 2024","interfaces":[{"interface":"back","average":{"1min":0.755,"5min":0.59399999999999997,"15min":0.54900000000000004},"min":{"1min":0.252,"5min":0.252,"15min":0.252},"max":{"1min":1.712,"5min":2.109,"15min":2.109},"last":0.39600000000000002},{"interface":"front","average":{"1min":0.95099999999999996,"5min":0.629,"15min":0.57699999999999996},"min":{"1min":0.26200000000000001,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":2.6850000000000001,"5min":2.6850000000000001,"15min":2.6850000000000001},"last":0.26300000000000001}]},{"osd":5,"last update":"Tue Sep 10 19:13:15 2024","interfaces":[{"interface":"back","average":{"1min":0.37,"5min":0.40000000000000002,"15min":0.38200000000000001},"min":{"1min":0.22900000000000001,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":0.62,"5min":0.62,"15min":0.62},"last":0.27900000000000003},{"interface":"front","average":{"1min":0.496,"5min":0.434,"15min":0.40699999999999997},"min":{"1min":0.28000000000000003,"5min":0.223,"15min":0.223},"max":{"1min":1.079,"5min":1.079,"15min":1.079},"last":0.248}]},{"osd":6,"last update":"Tue Sep 10 19:13:43 2024","interfaces":[{"interface":"back","average":{"1min":0.54600000000000004,"5min":0.47099999999999997,"15min":0.432},"min":{"1min":0.253,"5min":0.253,"15min":0.253},"max":{"1min":1.992,"5min":1.992,"15min":1.992},"last":0.28999999999999998},{"interface":"front","average":{"1min":0.48199999999999998,"5min":0.44700000000000001,"15min":0.42499999999999999},"min":{"1min":0.25,"5min":0.25,"15min":0.25},"max":{"1min":1.0469999999999999,"5min":1.0469999999999999,"15min":1.0469999999999999},"last":0.307}]},{"osd":7,"last update":"Tue Sep 10 19:14:03 2024","interfaces":[{"interface":"back","average":{"1min":2.149,"5min":0.82499999999999996,"15min":0.625},"min":{"1min":0.29299999999999998,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":23.081,"5min":23.081,"15min":23.081},"last":0.32400000000000001},{"interface":"front","average":{"1min":2.1280000000000001,"5min":0.82799999999999996,"15min":0.61099999999999999},"min":{"1min":0.311,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":22.666,"5min":22.666,"15min":22.666},"last":0.29699999999999999}]}]},{"osd":3,"up_from":26,"seq":111669149767,"num_pgs":71,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":4735040,"kb_used_data":4433528,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":89010112,"statfs":{"total":95995035648,"available":91146354688,"internally_reserved":0,"allocated":4539932672,"data_stored":4539568639,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,3,6],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 19:13:29 2024","interfaces":[{"interface":"back","average":{"1min":1.5009999999999999,"5min":0.62,"15min":0.47799999999999998},"min":{"1min":0.16800000000000001,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":4.452,"5min":4.452,"15min":4.452},"last":3.9369999999999998},{"interface":"front","average":{"1min":1.8919999999999999,"5min":0.71599999999999997,"15min":0.51800000000000002},"min":{"1min":0.23300000000000001,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":6.1429999999999998,"5min":6.1429999999999998,"15min":6.1429999999999998},"last":2.996}]},{"osd":1,"last update":"Tue Sep 10 19:12:27 2024","interfaces":[{"interface":"back","average":{"1min":0.52800000000000002,"5min":0.44400000000000001,"15min":0.45200000000000001},"min":{"1min":0.216,"5min":0.216,"15min":0.216},"max":{"1min":3.5150000000000001,"5min":3.5150000000000001,"15min":3.5150000000000001},"last":0.79700000000000004},{"interface":"front","average":{"1min":0.46600000000000003,"5min":0.40000000000000002,"15min":0.39800000000000002},"min":{"1min":0.187,"5min":0.187,"15min":0.187},"max":{"1min":1.3640000000000001,"5min":1.3640000000000001,"15min":1.3640000000000001},"last":0.59099999999999997}]},{"osd":2,"last update":"Tue Sep 10 19:13:29 2024","interfaces":[{"interface":"back","average":{"1min":1.0920000000000001,"5min":0.56599999999999995,"15min":0.48099999999999998},"min":{"1min":0.186,"5min":0.186,"15min":0.186},"max":{"1min":3.6459999999999999,"5min":3.6459999999999999,"15min":3.6459999999999999},"last":3.5070000000000001},{"interface":"front","average":{"1min":1.4490000000000001,"5min":0.626,"15min":0.48599999999999999},"min":{"1min":0.36399999999999999,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":4.2960000000000003,"5min":4.2960000000000003,"15min":4.2960000000000003},"last":2.843}]},{"osd":4,"last update":"Tue Sep 10 19:13:56 2024","interfaces":[{"interface":"back","average":{"1min":1.73,"5min":0.94199999999999995,"15min":0.67700000000000005},"min":{"1min":0.31900000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":4.54,"5min":11.964,"15min":11.964},"last":2.7429999999999999},{"interface":"front","average":{"1min":1.6799999999999999,"5min":1.026,"15min":0.70699999999999996},"min":{"1min":0.26200000000000001,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":6.0350000000000001,"5min":11.881,"15min":11.881},"last":2.9129999999999998}]},{"osd":5,"last update":"Tue Sep 10 19:13:14 2024","interfaces":[{"interface":"back","average":{"1min":1.2529999999999999,"5min":0.69899999999999995,"15min":0.60999999999999999},"min":{"1min":0.29099999999999998,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":4.3399999999999999,"5min":4.3399999999999999,"15min":4.3399999999999999},"last":2.8940000000000001},{"interface":"front","average":{"1min":1.8500000000000001,"5min":0.81699999999999995,"15min":0.64600000000000002},"min":{"1min":0.316,"5min":0.316,"15min":0.316},"max":{"1min":5.9950000000000001,"5min":5.9950000000000001,"15min":5.9950000000000001},"last":2.9550000000000001}]},{"osd":6,"last update":"Tue Sep 10 19:13:36 2024","interfaces":[{"interface":"back","average":{"1min":1.681,"5min":0.81200000000000006,"15min":0.65600000000000003},"min":{"1min":0.30399999999999999,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":4.5629999999999997,"5min":4.5629999999999997,"15min":4.5629999999999997},"last":2.8700000000000001},{"interface":"front","average":{"1min":0.89400000000000002,"5min":0.65300000000000002,"15min":0.58899999999999997},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":2.9249999999999998,"5min":2.9249999999999998,"15min":2.9249999999999998},"last":3.101}]},{"osd":7,"last update":"Tue Sep 10 19:13:07 2024","interfaces":[{"interface":"back","average":{"1min":1.3080000000000001,"5min":0.72099999999999997,"15min":0.627},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":13.538,"5min":13.538,"15min":13.538},"last":3.1360000000000001},{"interface":"front","average":{"1min":1.7310000000000001,"5min":0.81399999999999995,"15min":0.66000000000000003},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":13.522,"5min":13.522,"15min":13.522},"last":3.0259999999999998}]}]},{"osd":2,"up_from":18,"seq":77309411404,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3367056,"kb_used_data":3065544,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":90378096,"statfs":{"total":95995035648,"available":92547170304,"internally_reserved":0,"allocated":3139117056,"data_stored":3138800371,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,3,2,2,3],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 19:13:12 2024","interfaces":[{"interface":"back","average":{"1min":5.1200000000000001,"5min":1.2969999999999999,"15min":0.65900000000000003},"min":{"1min":0.28100000000000003,"5min":0.188,"15min":0.188},"max":{"1min":62.543999999999997,"5min":62.543999999999997,"15min":62.543999999999997},"last":1.1559999999999999},{"interface":"front","average":{"1min":4.3300000000000001,"5min":1.1619999999999999,"15min":0.63},"min":{"1min":0.22600000000000001,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":62.576999999999998,"5min":62.576999999999998,"15min":62.576999999999998},"last":2.4009999999999998}]},{"osd":1,"last update":"Tue Sep 10 19:12:08 2024","interfaces":[{"interface":"back","average":{"1min":0.317,"5min":0.378,"15min":0.375},"min":{"1min":0.23400000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.46800000000000003,"5min":0.69799999999999995,"15min":0.69799999999999995},"last":2.6230000000000002},{"interface":"front","average":{"1min":0.33300000000000002,"5min":0.38600000000000001,"15min":0.38800000000000001},"min":{"1min":0.255,"5min":0.255,"15min":0.255},"max":{"1min":0.499,"5min":0.63400000000000001,"15min":0.63400000000000001},"last":3.9940000000000002}]},{"osd":3,"last update":"Tue Sep 10 19:13:33 2024","interfaces":[{"interface":"back","average":{"1min":2.0190000000000001,"5min":1.409,"15min":0.77100000000000002},"min":{"1min":0.30599999999999999,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":4.6950000000000003,"5min":63.997,"15min":63.997},"last":2.012},{"interface":"front","average":{"1min":1.8560000000000001,"5min":1.298,"15min":0.73099999999999998},"min":{"1min":0.22800000000000001,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":4.1210000000000004,"5min":62.365000000000002,"15min":62.365000000000002},"last":0.33500000000000002}]},{"osd":4,"last update":"Tue Sep 10 19:13:57 2024","interfaces":[{"interface":"back","average":{"1min":1.127,"5min":0.75800000000000001,"15min":0.64000000000000001},"min":{"1min":0.254,"5min":0.254,"15min":0.254},"max":{"1min":4.3650000000000002,"5min":4.3650000000000002,"15min":4.3650000000000002},"last":0.627},{"interface":"front","average":{"1min":1.1619999999999999,"5min":0.752,"15min":0.63200000000000001},"min":{"1min":0.26000000000000001,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":3.0899999999999999,"5min":4.0750000000000002,"15min":4.0750000000000002},"last":0.35499999999999998}]},{"osd":5,"last update":"Tue Sep 10 19:13:12 2024","interfaces":[{"interface":"back","average":{"1min":1.177,"5min":0.65300000000000002,"15min":0.56699999999999995},"min":{"1min":0.28299999999999997,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":5.4820000000000002,"5min":5.4820000000000002,"15min":5.4820000000000002},"last":0.29099999999999998},{"interface":"front","average":{"1min":1.1579999999999999,"5min":0.68700000000000006,"15min":0.61499999999999999},"min":{"1min":0.309,"5min":0.309,"15min":0.309},"max":{"1min":4.6429999999999998,"5min":4.6429999999999998,"15min":4.6429999999999998},"last":0.68400000000000005}]},{"osd":6,"last update":"Tue Sep 10 19:13:38 2024","interfaces":[{"interface":"back","average":{"1min":1.3029999999999999,"5min":0.754,"15min":0.624},"min":{"1min":0.41199999999999998,"5min":0.37,"15min":0.37},"max":{"1min":4.6180000000000003,"5min":4.6180000000000003,"15min":4.6180000000000003},"last":0.66100000000000003},{"interface":"front","average":{"1min":1.464,"5min":0.75600000000000001,"15min":0.61899999999999999},"min":{"1min":0.28999999999999998,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":4.8049999999999997,"5min":4.8049999999999997,"15min":4.8049999999999997},"last":0.373}]},{"osd":7,"last update":"Tue Sep 10 19:14:03 2024","interfaces":[{"interface":"back","average":{"1min":1.589,"5min":0.84699999999999998,"15min":0.68600000000000005},"min":{"1min":0.41599999999999998,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":4.5640000000000001,"5min":4.5640000000000001,"15min":4.5640000000000001},"last":0.70299999999999996},{"interface":"front","average":{"1min":1.226,"5min":0.79200000000000004,"15min":0.63},"min":{"1min":0.312,"5min":0.312,"15min":0.312},"max":{"1min":3.1179999999999999,"5min":4.0970000000000004,"15min":4.0970000000000004},"last":0.39000000000000001}]}]},{"osd":1,"up_from":0,"seq":0,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":8,"seq":34359738454,"num_pgs":64,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":4259700,"kb_used_data":3949868,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":89485452,"statfs":{"total":95995035648,"available":91633102848,"internally_reserved":0,"allocated":4044664832,"data_stored":4044313177,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Sep 10 19:12:44 2024","interfaces":[{"interface":"back","average":{"1min":0.73199999999999998,"5min":0.42999999999999999,"15min":0.35799999999999998},"min":{"1min":0.17399999999999999,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":2.7320000000000002,"5min":2.7320000000000002,"15min":2.7320000000000002},"last":0.80700000000000005},{"interface":"front","average":{"1min":0.82299999999999995,"5min":0.45100000000000001,"15min":0.36799999999999999},"min":{"1min":0.21199999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":2.4609999999999999,"5min":2.4609999999999999,"15min":2.4609999999999999},"last":2.4609999999999999}]},{"osd":2,"last update":"Tue Sep 10 19:13:14 2024","interfaces":[{"interface":"back","average":{"1min":1.03,"5min":0.504,"15min":0.42199999999999999},"min":{"1min":0.316,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":3.581,"5min":3.581,"15min":3.581},"last":0.77200000000000002},{"interface":"front","average":{"1min":0.81799999999999995,"5min":0.45200000000000001,"15min":0.39800000000000002},"min":{"1min":0.193,"5min":0.187,"15min":0.187},"max":{"1min":2.2850000000000001,"5min":2.2850000000000001,"15min":2.2850000000000001},"last":0.57699999999999996}]},{"osd":3,"last update":"Tue Sep 10 19:13:36 2024","interfaces":[{"interface":"back","average":{"1min":1.3049999999999999,"5min":0.59499999999999997,"15min":0.48099999999999998},"min":{"1min":0.253,"5min":0.252,"15min":0.252},"max":{"1min":2.8039999999999998,"5min":2.8039999999999998,"15min":2.8039999999999998},"last":0.98599999999999999},{"interface":"front","average":{"1min":1.3859999999999999,"5min":0.60799999999999998,"15min":0.47499999999999998},"min":{"1min":0.372,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":4.9119999999999999,"5min":4.9119999999999999,"15min":4.9119999999999999},"last":0.66700000000000004}]},{"osd":4,"last update":"Tue Sep 10 19:13:49 2024","interfaces":[{"interface":"back","average":{"1min":1.3260000000000001,"5min":0.70699999999999996,"15min":0.61299999999999999},"min":{"1min":0.24399999999999999,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":5.8499999999999996,"5min":5.8499999999999996,"15min":5.8499999999999996},"last":0.60699999999999998},{"interface":"front","average":{"1min":0.998,"5min":0.70899999999999996,"15min":0.61099999999999999},"min":{"1min":0.27700000000000002,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":3.2799999999999998,"5min":3.2999999999999998,"15min":3.2999999999999998},"last":1.016}]},{"osd":5,"last update":"Tue Sep 10 19:13:14 2024","interfaces":[{"interface":"back","average":{"1min":0.94099999999999995,"5min":0.63200000000000001,"15min":0.59999999999999998},"min":{"1min":0.29499999999999998,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":3.3380000000000001,"5min":3.3380000000000001,"15min":3.3380000000000001},"last":0.72199999999999998},{"interface":"front","average":{"1min":1.0389999999999999,"5min":0.65000000000000002,"15min":0.59999999999999998},"min":{"1min":0.29799999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":4.9790000000000001,"5min":4.9790000000000001,"15min":4.9790000000000001},"last":0.65000000000000002}]},{"osd":6,"last update":"Tue Sep 10 19:13:36 2024","interfaces":[{"interface":"back","average":{"1min":1.1559999999999999,"5min":0.67800000000000005,"15min":0.60399999999999998},"min":{"1min":0.27200000000000002,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":3.536,"5min":3.536,"15min":3.536},"last":0.46000000000000002},{"interface":"front","average":{"1min":1.1930000000000001,"5min":0.70199999999999996,"15min":0.625},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":3.3660000000000001,"5min":3.3660000000000001,"15min":3.3660000000000001},"last":0.51800000000000002}]},{"osd":7,"last update":"Tue Sep 10 19:14:02 2024","interfaces":[{"interface":"back","average":{"1min":1.0629999999999999,"5min":0.76000000000000001,"15min":0.63800000000000001},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":3.3580000000000001,"5min":3.3919999999999999,"15min":3.3919999999999999},"last":0.745},{"interface":"front","average":{"1min":1.1279999999999999,"5min":0.69399999999999995,"15min":0.60599999999999998},"min":{"1min":0.39500000000000002,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":4.9470000000000001,"5min":4.9470000000000001,"15min":4.9470000000000001},"last":0.61899999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":966,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1236,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1236,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":362,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1650,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":2118,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":908,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":2228,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1942,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4013756416,"data_stored":4013752344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1325072384,"data_stored":1325072384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":3127771136,"data_stored":3127771136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4518510592,"data_stored":4518510592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":2381381632,"data_stored":2381381632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1935540224,"data_stored":1935540224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":2504724480,"data_stored":2504720408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":2586705920,"data_stored":2586705920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T19:14:13.021 INFO:tasks.cephadm.ceph_manager.ceph:would drop below min_pgs, proj -9, currently 1 2024-09-10T19:14:13.032 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:14:13.059 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:14:13.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:14:13.060 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:14:13.161 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:14:13.193 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:14:13.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:14:13.194 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:14:13.295 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:14:13.331 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:14:13.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:14:13.333 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:14:13.351 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:14:13 smithi183 ceph-mon[29736]: pgmap v155: 148 pgs: 3 active+undersized+degraded+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 142 active+clean; 7.1 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 767 B/s rd, 0 B/s wr, 870 op/s; 38828/351933 objects degraded (11.033%); 18 MiB/s, 280 objects/s recovering 2024-09-10T19:14:13.351 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:14:13 smithi183 ceph-mon[41562]: pgmap v155: 148 pgs: 3 active+undersized+degraded+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 142 active+clean; 7.1 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 767 B/s rd, 0 B/s wr, 870 op/s; 38828/351933 objects degraded (11.033%); 18 MiB/s, 280 objects/s recovering 2024-09-10T19:14:13.433 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:14:13.463 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:14:13.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:14:13.464 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:14:13.465 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:14:13 smithi191 ceph-mon[32884]: pgmap v155: 148 pgs: 3 active+undersized+degraded+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 142 active+clean; 7.1 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 767 B/s rd, 0 B/s wr, 870 op/s; 38828/351933 objects degraded (11.033%); 18 MiB/s, 280 objects/s recovering 2024-09-10T19:14:13.565 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:14:13.604 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:14:13.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:14:13.605 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:14:13.646 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:14:13.686 INFO:teuthology.orchestra.run.smithi183.stdout:osd.0: {} 2024-09-10T19:14:13.686 INFO:teuthology.orchestra.run.smithi183.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-10T19:14:13.686 INFO:teuthology.orchestra.run.smithi183.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-09-10T19:14:13.701 INFO:teuthology.orchestra.run.smithi183.stdout:osd.2: {} 2024-09-10T19:14:13.701 INFO:teuthology.orchestra.run.smithi183.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-10T19:14:13.706 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:14:13.716 INFO:teuthology.orchestra.run.smithi183.stdout:osd.3: {} 2024-09-10T19:14:13.716 INFO:teuthology.orchestra.run.smithi183.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-10T19:14:13.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:14:13.728 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:14:13.728 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:14:13.756 INFO:teuthology.orchestra.run.smithi183.stdout:osd.4: {} 2024-09-10T19:14:13.757 INFO:teuthology.orchestra.run.smithi183.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-10T19:14:13.788 INFO:teuthology.orchestra.run.smithi183.stdout:osd.5: {} 2024-09-10T19:14:13.788 INFO:teuthology.orchestra.run.smithi183.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-10T19:14:13.819 INFO:teuthology.orchestra.run.smithi183.stdout:osd.6: {} 2024-09-10T19:14:13.820 INFO:teuthology.orchestra.run.smithi183.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-10T19:14:13.829 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:14:13.853 INFO:teuthology.orchestra.run.smithi183.stdout:osd.7: {} 2024-09-10T19:14:13.853 INFO:teuthology.orchestra.run.smithi183.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-10T19:14:13.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:14:13.869 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:14:13.869 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:14:13.912 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Sep 10 19:14:13 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-y[29989]: 2024-09-10T19:14:13.658+0000 7f9fad6cd640 -1 mgr get_metadata_python Requested missing service osd.1 2024-09-10T19:14:13.970 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:14:14.006 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:14:14.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:14:14.007 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:14:14.107 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:14:14.136 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:14:14.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:14:14.137 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:14:14.237 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:14:14.279 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:14:14.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:14:14.280 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:14:14.381 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:14:14.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:14:14.419 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:14:14.420 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:14:14.520 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:14:14.555 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:14:14.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:14:14.556 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:14:14.657 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:14:14.686 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:14:14.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:14:14.689 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:14:14.790 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:14:14.847 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:14:14.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:14:14.848 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:14:14.931 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:14:14 smithi183 ceph-mon[29736]: from='mgr.14556 ' entity='mgr.y' 2024-09-10T19:14:14.931 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:14:14 smithi183 ceph-mon[29736]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:14:14.931 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:14:14 smithi183 ceph-mon[29736]: from='client.? 172.21.15.183:0/1658615269' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T19:14:14.931 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:14:14 smithi183 ceph-mon[29736]: pgmap v156: 148 pgs: 3 active+undersized+degraded+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 142 active+clean; 7.1 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 725 op/s; 38828/351933 objects degraded (11.033%); 15 MiB/s, 233 objects/s recovering 2024-09-10T19:14:14.931 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:14:14 smithi183 ceph-mon[41562]: from='mgr.14556 ' entity='mgr.y' 2024-09-10T19:14:14.932 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:14:14 smithi183 ceph-mon[41562]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:14:14.932 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:14:14 smithi183 ceph-mon[41562]: from='client.? 172.21.15.183:0/1658615269' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T19:14:14.932 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:14:14 smithi183 ceph-mon[41562]: pgmap v156: 148 pgs: 3 active+undersized+degraded+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 142 active+clean; 7.1 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 725 op/s; 38828/351933 objects degraded (11.033%); 15 MiB/s, 233 objects/s recovering 2024-09-10T19:14:14.949 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:14:14.966 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:14:14 smithi191 ceph-mon[32884]: from='mgr.14556 ' entity='mgr.y' 2024-09-10T19:14:14.966 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:14:14 smithi191 ceph-mon[32884]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:14:14.967 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:14:14 smithi191 ceph-mon[32884]: from='client.? 172.21.15.183:0/1658615269' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T19:14:14.967 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:14:14 smithi191 ceph-mon[32884]: pgmap v156: 148 pgs: 3 active+undersized+degraded+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 142 active+clean; 7.1 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 725 op/s; 38828/351933 objects degraded (11.033%); 15 MiB/s, 233 objects/s recovering 2024-09-10T19:14:14.981 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:14:14.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:14:14.984 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:14:15.028 INFO:teuthology.orchestra.run.smithi191.stdout:ERROR: (22) Invalid argument 2024-09-10T19:14:15.029 INFO:teuthology.orchestra.run.smithi191.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-10T19:14:15.085 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:14:15.118 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:14:15.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:14:15.119 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:14:15.219 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:14:15.254 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:14:15.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:14:15.255 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:14:15.356 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:14:15.389 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:14:15.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:14:15.393 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:14:15.493 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:14:15.524 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:14:15.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:14:15.525 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:14:15.626 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:14:15.659 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:14:15.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:14:15.660 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:14:15.713 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:14:15 smithi191 ceph-mon[32884]: Health check update: Degraded data redundancy: 38828/351933 objects degraded (11.033%), 6 pgs degraded (PG_DEGRADED) 2024-09-10T19:14:15.753 DEBUG:teuthology.orchestra.run.smithi191:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph daemon osd.6 dump_blocked_ops 2024-09-10T19:14:15.760 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:14:15.837 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:14:15.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:14:15.838 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:14:15.931 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:14:15 smithi183 ceph-mon[29736]: Health check update: Degraded data redundancy: 38828/351933 objects degraded (11.033%), 6 pgs degraded (PG_DEGRADED) 2024-09-10T19:14:15.931 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:14:15 smithi183 ceph-mon[41562]: Health check update: Degraded data redundancy: 38828/351933 objects degraded (11.033%), 6 pgs degraded (PG_DEGRADED) 2024-09-10T19:14:15.938 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:14:15.972 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:14:15.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:14:15.973 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:14:16.052 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-10T19:14:16.053 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-10T19:14:16.074 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:14:16.116 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:14:16.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:14:16.118 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:14:16.150 INFO:teuthology.orchestra.run.smithi191.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.b/config 2024-09-10T19:14:16.218 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:14:16.249 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:14:16.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:14:16.250 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:14:16.351 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:14:16.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:14:16.390 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:14:16.390 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:14:16.490 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:14:16.525 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:14:16.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:14:16.526 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:14:16.626 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:14:16.681 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:14:16.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:14:16.682 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:14:16.736 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:14:16 smithi183 ceph-mon[29736]: pgmap v157: 148 pgs: 3 active+undersized+degraded+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 142 active+clean; 6.8 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.39k op/s; 34886/333162 objects degraded (10.471%); 28 MiB/s, 441 objects/s recovering 2024-09-10T19:14:16.736 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:14:16 smithi183 ceph-mon[41562]: pgmap v157: 148 pgs: 3 active+undersized+degraded+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 142 active+clean; 6.8 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.39k op/s; 34886/333162 objects degraded (10.471%); 28 MiB/s, 441 objects/s recovering 2024-09-10T19:14:16.784 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:14:16.812 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:14:16.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:14:16.813 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:14:16.914 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:14:16.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:14:16.957 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:14:16.957 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:14:16.965 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:14:16 smithi191 ceph-mon[32884]: pgmap v157: 148 pgs: 3 active+undersized+degraded+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 142 active+clean; 6.8 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.39k op/s; 34886/333162 objects degraded (10.471%); 28 MiB/s, 441 objects/s recovering 2024-09-10T19:14:17.058 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:14:17.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:14:17.101 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:14:17.101 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:14:17.202 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:14:17.242 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:14:17.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:14:17.243 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:14:17.344 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:14:17.370 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:14:17.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:14:17.371 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:14:17.472 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:14:17.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:14:17.509 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:14:17.509 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:14:17.610 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:14:17.638 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:14:17.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:14:17.639 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:14:17.699 INFO:teuthology.orchestra.run.smithi183.stderr:nodeep-scrub is set 2024-09-10T19:14:17.704 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:14:17 smithi191 ceph-mon[32884]: from='client.? 172.21.15.183:0/1860147595' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-10T19:14:17.704 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:14:17 smithi191 ceph-mon[32884]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-10T19:14:17.704 INFO:journalctl@ceph.mgr.x.smithi191.stdout:Sep 10 19:14:17 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-x[34967]: ::ffff:172.21.15.191 - - [10/Sep/2024:19:14:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T19:14:17.740 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:14:17.771 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:14:17.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:14:17.772 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:14:17.873 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:14:17.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:14:17.904 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:14:17.905 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:14:17.931 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:14:17 smithi183 ceph-mon[29736]: from='client.? 172.21.15.183:0/1860147595' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-10T19:14:17.931 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:14:17 smithi183 ceph-mon[29736]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-10T19:14:17.931 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:14:17 smithi183 ceph-mon[41562]: from='client.? 172.21.15.183:0/1860147595' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-10T19:14:17.932 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:14:17 smithi183 ceph-mon[41562]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-10T19:14:18.005 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:14:18.022 INFO:tasks.thrashosds.thrasher:in_osds: [0, 2, 3, 4, 5, 6, 7] out_osds: [1] dead_osds: [1] live_osds: [0, 2, 3, 4, 5, 6, 7] 2024-09-10T19:14:18.022 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-10T19:14:18.022 INFO:tasks.thrashosds.thrasher:Shrinking pool unique_pool_0 2024-09-10T19:14:18.023 INFO:tasks.cephadm.ceph_manager.ceph:contract_pool unique_pool_0 by 10 min 8 2024-09-10T19:14:18.023 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph pg dump --format=json 2024-09-10T19:14:18.035 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:14:18.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:14:18.036 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:14:18.054 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-10T19:14:18.137 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:14:18.168 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:14:18.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:14:18.169 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:14:18.170 INFO:teuthology.orchestra.run.smithi191.stdout:ERROR: (22) Invalid argument 2024-09-10T19:14:18.170 INFO:teuthology.orchestra.run.smithi191.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-10T19:14:18.270 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:14:18.335 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:14:18.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:14:18.336 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:14:18.437 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:14:18.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:14:18.506 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:14:18.506 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:14:18.546 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:14:18.553 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:14:18.607 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:14:18.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:14:18.641 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:14:18.641 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:14:18.742 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:14:18.773 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:14:18.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:14:18.774 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:14:18.875 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:14:18.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:14:18.918 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:14:18.918 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:14:18.931 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:14:18 smithi183 ceph-mon[29736]: pgmap v158: 148 pgs: 3 active+undersized+degraded+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 142 active+clean; 6.8 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.21k op/s; 34886/333162 objects degraded (10.471%); 24 MiB/s, 382 objects/s recovering 2024-09-10T19:14:18.931 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:14:18 smithi183 ceph-mon[29736]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T19:14:18.931 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:14:18 smithi183 ceph-mon[29736]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-10T19:14:18.931 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:14:18 smithi183 ceph-mon[29736]: osdmap e122: 8 total, 7 up, 7 in 2024-09-10T19:14:18.931 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:14:18 smithi183 ceph-mon[41562]: pgmap v158: 148 pgs: 3 active+undersized+degraded+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 142 active+clean; 6.8 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.21k op/s; 34886/333162 objects degraded (10.471%); 24 MiB/s, 382 objects/s recovering 2024-09-10T19:14:18.931 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:14:18 smithi183 ceph-mon[41562]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T19:14:18.932 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:14:18 smithi183 ceph-mon[41562]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-10T19:14:18.932 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:14:18 smithi183 ceph-mon[41562]: osdmap e122: 8 total, 7 up, 7 in 2024-09-10T19:14:18.966 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:14:18 smithi191 ceph-mon[32884]: pgmap v158: 148 pgs: 3 active+undersized+degraded+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 142 active+clean; 6.8 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.21k op/s; 34886/333162 objects degraded (10.471%); 24 MiB/s, 382 objects/s recovering 2024-09-10T19:14:18.966 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:14:18 smithi191 ceph-mon[32884]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T19:14:18.966 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:14:18 smithi191 ceph-mon[32884]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-10T19:14:18.966 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:14:18 smithi191 ceph-mon[32884]: osdmap e122: 8 total, 7 up, 7 in 2024-09-10T19:14:19.019 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:14:19.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:14:19.058 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:14:19.058 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:14:19.159 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:14:19.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:14:19.192 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:14:19.192 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:14:19.293 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:14:19.324 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:14:19.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:14:19.325 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:14:19.426 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:14:19.438 INFO:journalctl@ceph.iscsi.iscsi.a.smithi191.stdout:Sep 10 19:14:19 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-iscsi-iscsi-a[61816]: debug there is no tcmu-runner data available 2024-09-10T19:14:19.471 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:14:19.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:14:19.472 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:14:19.481 DEBUG:teuthology.orchestra.run.smithi191:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph daemon osd.6 dump_historic_ops 2024-09-10T19:14:19.573 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:14:19.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:14:19.615 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:14:19.615 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:14:19.715 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:14:19 smithi191 ceph-mon[32884]: from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:14:19.717 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:14:19.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:14:19.754 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:14:19.754 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:14:19.855 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:14:19.909 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:14:19.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:14:19.910 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:14:19.910 INFO:teuthology.orchestra.run.smithi191.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.b/config 2024-09-10T19:14:19.931 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:14:19 smithi183 ceph-mon[29736]: from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:14:19.931 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:14:19 smithi183 ceph-mon[41562]: from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:14:20.011 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:14:20.046 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:14:20.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:14:20.047 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:14:20.148 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:14:20.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:14:20.176 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:14:20.176 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:14:20.277 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:14:20.307 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:14:20.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:14:20.308 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:14:20.409 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:14:20.437 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph osd unset noscrub 2024-09-10T19:14:20.443 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:14:20.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:14:20.445 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:14:20.546 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:14:20.577 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:14:20.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:14:20.578 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:14:20.678 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:14:20.708 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:14:20.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:14:20.709 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:14:20.810 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:14:20.842 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:14:20.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:14:20.843 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:14:20.944 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:14:20.966 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:14:20 smithi191 ceph-mon[32884]: pgmap v160: 148 pgs: 3 active+undersized+degraded+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 142 active+clean; 6.6 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.25k op/s; 33210/325251 objects degraded (10.211%); 24 MiB/s, 381 objects/s recovering 2024-09-10T19:14:20.966 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:14:20 smithi191 ceph-mon[32884]: Health check update: Degraded data redundancy: 33210/325251 objects degraded (10.211%), 6 pgs degraded (PG_DEGRADED) 2024-09-10T19:14:20.976 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:14:20.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:14:20.977 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:14:21.078 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:14:21.114 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:14:21.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:14:21.116 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:14:21.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:14:20 smithi183 ceph-mon[29736]: pgmap v160: 148 pgs: 3 active+undersized+degraded+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 142 active+clean; 6.6 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.25k op/s; 33210/325251 objects degraded (10.211%); 24 MiB/s, 381 objects/s recovering 2024-09-10T19:14:21.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:14:20 smithi183 ceph-mon[29736]: Health check update: Degraded data redundancy: 33210/325251 objects degraded (10.211%), 6 pgs degraded (PG_DEGRADED) 2024-09-10T19:14:21.181 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:14:20 smithi183 ceph-mon[41562]: pgmap v160: 148 pgs: 3 active+undersized+degraded+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 142 active+clean; 6.6 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.25k op/s; 33210/325251 objects degraded (10.211%); 24 MiB/s, 381 objects/s recovering 2024-09-10T19:14:21.181 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:14:20 smithi183 ceph-mon[41562]: Health check update: Degraded data redundancy: 33210/325251 objects degraded (10.211%), 6 pgs degraded (PG_DEGRADED) 2024-09-10T19:14:21.217 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:14:21.245 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:14:21.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:14:21.246 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:14:21.347 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:14:21.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:14:21.393 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:14:21.393 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:14:21.488 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Sep 10 19:14:21 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-y[29989]: ::ffff:172.21.15.191 - - [10/Sep/2024:19:14:21] "GET /metrics HTTP/1.1" 200 209075 "" "Prometheus/2.43.0" 2024-09-10T19:14:21.494 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:14:21.540 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:14:21.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:14:21.541 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:14:21.642 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:14:21.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:14:21.683 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:14:21.683 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:14:21.784 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:14:21.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:14:21.816 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:14:21.816 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:14:21.916 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:14:21.954 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:14:21.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:14:21.955 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:14:22.056 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:14:22.077 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-09-10T19:14:22.080 INFO:teuthology.orchestra.run.smithi183.stderr:dumped all 2024-09-10T19:14:22.097 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:14:22.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:14:22.099 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:14:22.164 INFO:teuthology.orchestra.run.smithi183.stdout:osd.0: {} 2024-09-10T19:14:22.164 INFO:teuthology.orchestra.run.smithi183.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-10T19:14:22.172 INFO:teuthology.orchestra.run.smithi183.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-09-10T19:14:22.174 INFO:teuthology.orchestra.run.smithi191.stdout:ERROR: (22) Invalid argument 2024-09-10T19:14:22.174 INFO:teuthology.orchestra.run.smithi191.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-10T19:14:22.200 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:14:22.210 INFO:teuthology.orchestra.run.smithi183.stdout:osd.2: {} 2024-09-10T19:14:22.210 INFO:teuthology.orchestra.run.smithi183.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-10T19:14:22.241 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:14:22.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:14:22.242 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:14:22.242 INFO:teuthology.orchestra.run.smithi183.stdout:osd.3: {} 2024-09-10T19:14:22.242 INFO:teuthology.orchestra.run.smithi183.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-10T19:14:22.269 INFO:teuthology.orchestra.run.smithi183.stdout:osd.4: {} 2024-09-10T19:14:22.270 INFO:teuthology.orchestra.run.smithi183.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-10T19:14:22.300 INFO:teuthology.orchestra.run.smithi183.stdout:osd.5: {} 2024-09-10T19:14:22.300 INFO:teuthology.orchestra.run.smithi183.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-10T19:14:22.331 INFO:teuthology.orchestra.run.smithi183.stdout:osd.6: {} 2024-09-10T19:14:22.331 INFO:teuthology.orchestra.run.smithi183.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-10T19:14:22.343 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:14:22.364 INFO:teuthology.orchestra.run.smithi183.stdout:osd.7: {} 2024-09-10T19:14:22.364 INFO:teuthology.orchestra.run.smithi183.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-10T19:14:22.380 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:14:22.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:14:22.381 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:14:22.482 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:14:22.520 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:14:22.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:14:22.521 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:14:22.621 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:14:22.623 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:14:22.650 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:14:22.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:14:22.651 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:14:22.752 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:14:22.782 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:14:22.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:14:22.783 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:14:22.883 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:14:22.914 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:14:22.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:14:22.915 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:14:22.946 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:14:22 smithi191 ceph-mon[32884]: pgmap v161: 148 pgs: 3 active+undersized+degraded+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 142 active+clean; 6.4 GiB data, 19 GiB used, 606 GiB / 626 GiB avail; 1022 B/s rd, 0 B/s wr, 1.23k op/s; 31107/314940 objects degraded (9.877%); 24 MiB/s, 386 objects/s recovering 2024-09-10T19:14:22.946 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:14:22 smithi191 ceph-mon[32884]: from='client.24782 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-10T19:14:22.946 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:14:22 smithi191 ceph-mon[32884]: from='client.? 172.21.15.183:0/211247895' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T19:14:22.985 DEBUG:teuthology.orchestra.run.smithi191:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph daemon osd.7 dump_ops_in_flight 2024-09-10T19:14:23.017 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:14:23.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:14:23.062 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:14:23.062 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:14:23.163 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:14:23.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:14:22 smithi183 ceph-mon[29736]: pgmap v161: 148 pgs: 3 active+undersized+degraded+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 142 active+clean; 6.4 GiB data, 19 GiB used, 606 GiB / 626 GiB avail; 1022 B/s rd, 0 B/s wr, 1.23k op/s; 31107/314940 objects degraded (9.877%); 24 MiB/s, 386 objects/s recovering 2024-09-10T19:14:23.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:14:22 smithi183 ceph-mon[29736]: from='client.24782 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-10T19:14:23.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:14:22 smithi183 ceph-mon[29736]: from='client.? 172.21.15.183:0/211247895' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T19:14:23.181 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:14:22 smithi183 ceph-mon[41562]: pgmap v161: 148 pgs: 3 active+undersized+degraded+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 142 active+clean; 6.4 GiB data, 19 GiB used, 606 GiB / 626 GiB avail; 1022 B/s rd, 0 B/s wr, 1.23k op/s; 31107/314940 objects degraded (9.877%); 24 MiB/s, 386 objects/s recovering 2024-09-10T19:14:23.181 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:14:22 smithi183 ceph-mon[41562]: from='client.24782 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-10T19:14:23.181 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:14:22 smithi183 ceph-mon[41562]: from='client.? 172.21.15.183:0/211247895' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T19:14:23.200 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:14:23.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:14:23.201 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:14:23.239 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-10T19:14:23.240 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-10T19:14:23.302 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:14:23.338 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:14:23.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:14:23.339 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:14:23.423 INFO:teuthology.orchestra.run.smithi191.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.b/config 2024-09-10T19:14:23.440 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:14:23.482 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:14:23.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:14:23.483 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:14:23.584 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:14:23.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:14:23.614 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:14:23.615 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:14:23.715 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:14:23.751 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:14:23.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:14:23.752 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:14:23.853 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:14:23.881 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:14:23.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:14:23.882 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:14:23.983 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:14:24.017 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:14:24.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:14:24.018 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:14:24.118 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:14:24.152 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:14:24.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:14:24.153 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:14:24.254 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:14:24.289 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:14:24.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:14:24.290 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:14:24.390 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:14:24.429 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:14:24.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:14:24.430 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:14:24.530 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:14:24.567 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:14:24.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:14:24.568 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:14:24.669 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:14:24.714 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:14:24.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:14:24.715 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:14:24.815 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:14:24.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:14:24.853 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:14:24.853 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:14:24.954 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:14:24.965 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:14:24 smithi191 ceph-mon[32884]: pgmap v162: 148 pgs: 3 active+undersized+degraded+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 142 active+clean; 6.4 GiB data, 19 GiB used, 606 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.23k op/s; 31107/314940 objects degraded (9.877%); 24 MiB/s, 386 objects/s recovering 2024-09-10T19:14:24.991 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:14:24.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:14:24.992 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:14:25.040 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:14:24 smithi183 ceph-mon[29736]: pgmap v162: 148 pgs: 3 active+undersized+degraded+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 142 active+clean; 6.4 GiB data, 19 GiB used, 606 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.23k op/s; 31107/314940 objects degraded (9.877%); 24 MiB/s, 386 objects/s recovering 2024-09-10T19:14:25.041 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:14:24 smithi183 ceph-mon[41562]: pgmap v162: 148 pgs: 3 active+undersized+degraded+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 142 active+clean; 6.4 GiB data, 19 GiB used, 606 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.23k op/s; 31107/314940 objects degraded (9.877%); 24 MiB/s, 386 objects/s recovering 2024-09-10T19:14:25.081 INFO:teuthology.orchestra.run.smithi183.stdout:{"pg_ready":true,"pg_map":{"version":161,"stamp":"2024-09-10T19:14:21.674064+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":6867654900,"num_objects":104980,"num_object_clones":0,"num_object_copies":314940,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31107,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":104980,"num_whiteouts":0,"num_read":944,"num_read_kb":801,"num_write":176517,"num_write_kb":8985864,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9375,"num_bytes_recovered":607194421,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":77064,"ondisk_log_size":77064,"up":444,"acting":438,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":444,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":656216064,"kb_used":20362132,"kb_used_data":18247108,"kb_used_omap":11,"kb_used_meta":2114676,"kb_avail":635853932,"statfs":{"total":671965249536,"available":651114426368,"internally_reserved":0,"allocated":18685038592,"data_stored":18682522264,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":11928,"internal_metadata":2165428584},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-808124416,"num_objects":-12331,"num_object_clones":0,"num_object_copies":-36993,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-7721,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-12331,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":12331,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3866,"num_bytes_recovered":253362176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.011014"},"pg_stats":[{"pgid":"6.1b","version":"0'0","reported_seq":86,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-09-10T19:14:09.035182+0000","last_change":"2024-09-10T19:13:08.710758+0000","last_active":"2024-09-10T19:14:09.035182+0000","last_peered":"2024-09-10T19:14:09.035182+0000","last_clean":"2024-09-10T19:14:09.035182+0000","last_became_active":"2024-09-10T19:10:14.056187+0000","last_became_peered":"2024-09-10T19:10:14.056187+0000","last_unstale":"2024-09-10T19:14:09.035182+0000","last_undegraded":"2024-09-10T19:14:09.035182+0000","last_fullsized":"2024-09-10T19:14:09.035182+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:13:08.710679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:13:08.710679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0027252499999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":102,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-09-10T19:14:09.040578+0000","last_change":"2024-09-10T19:13:16.219506+0000","last_active":"2024-09-10T19:14:09.040578+0000","last_peered":"2024-09-10T19:14:09.040578+0000","last_clean":"2024-09-10T19:14:09.040578+0000","last_became_active":"2024-09-10T19:11:44.748696+0000","last_became_peered":"2024-09-10T19:11:44.748696+0000","last_unstale":"2024-09-10T19:14:09.040578+0000","last_undegraded":"2024-09-10T19:14:09.040578+0000","last_fullsized":"2024-09-10T19:14:09.040578+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:13:16.219364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:13:16.219364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.023972094999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.19","version":"66'15","reported_seq":115,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-09-10T19:14:09.034908+0000","last_change":"2024-09-10T19:13:47.165943+0000","last_active":"2024-09-10T19:14:09.034908+0000","last_peered":"2024-09-10T19:14:09.034908+0000","last_clean":"2024-09-10T19:14:09.034908+0000","last_became_active":"2024-09-10T19:10:09.688432+0000","last_became_peered":"2024-09-10T19:10:09.688432+0000","last_unstale":"2024-09-10T19:14:09.034908+0000","last_undegraded":"2024-09-10T19:14:09.034908+0000","last_fullsized":"2024-09-10T19:14:09.034908+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'15","last_scrub_stamp":"2024-09-10T19:13:47.165848+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:13:47.165848+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:15:16.492257+0000","scrub_duration":0.004982841,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":94,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-09-10T19:14:09.033249+0000","last_change":"2024-09-10T19:13:54.719103+0000","last_active":"2024-09-10T19:14:09.033249+0000","last_peered":"2024-09-10T19:14:09.033249+0000","last_clean":"2024-09-10T19:14:09.033249+0000","last_became_active":"2024-09-10T19:13:54.714824+0000","last_became_peered":"2024-09-10T19:13:54.714824+0000","last_unstale":"2024-09-10T19:14:09.033249+0000","last_undegraded":"2024-09-10T19:14:09.033249+0000","last_fullsized":"2024-09-10T19:14:09.033249+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:11:23.086191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:11:23.086191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0016087969999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"59'1","reported_seq":100,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-09-10T19:14:09.036521+0000","last_change":"2024-09-10T19:12:47.704757+0000","last_active":"2024-09-10T19:14:09.036521+0000","last_peered":"2024-09-10T19:14:09.036521+0000","last_clean":"2024-09-10T19:14:09.036521+0000","last_became_active":"2024-09-10T19:10:07.686143+0000","last_became_peered":"2024-09-10T19:10:07.686143+0000","last_unstale":"2024-09-10T19:14:09.036521+0000","last_undegraded":"2024-09-10T19:14:09.036521+0000","last_fullsized":"2024-09-10T19:14:09.036521+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'1","last_scrub_stamp":"2024-09-10T19:12:47.704508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:12:47.704508+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.010519827000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":874,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"66'9","reported_seq":122,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-09-10T19:14:09.033349+0000","last_change":"2024-09-10T19:13:54.747869+0000","last_active":"2024-09-10T19:14:09.033349+0000","last_peered":"2024-09-10T19:14:09.033349+0000","last_clean":"2024-09-10T19:14:09.033349+0000","last_became_active":"2024-09-10T19:13:54.716117+0000","last_became_peered":"2024-09-10T19:13:54.716117+0000","last_unstale":"2024-09-10T19:14:09.033349+0000","last_undegraded":"2024-09-10T19:14:09.033349+0000","last_fullsized":"2024-09-10T19:14:09.033349+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'9","last_scrub_stamp":"2024-09-10T19:12:04.079264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:12:04.079264+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0015358819999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,3],"acting":[4,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":42,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-09-10T19:14:09.041705+0000","last_change":"2024-09-10T19:13:54.714747+0000","last_active":"2024-09-10T19:14:09.041705+0000","last_peered":"2024-09-10T19:14:09.041705+0000","last_clean":"2024-09-10T19:14:09.041705+0000","last_became_active":"2024-09-10T19:13:54.714365+0000","last_became_peered":"2024-09-10T19:13:54.714365+0000","last_unstale":"2024-09-10T19:14:09.041705+0000","last_undegraded":"2024-09-10T19:14:09.041705+0000","last_fullsized":"2024-09-10T19:14:09.041705+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:11:56.186365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:11:56.186365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":92,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-09-10T19:14:09.033921+0000","last_change":"2024-09-10T19:13:54.713823+0000","last_active":"2024-09-10T19:14:09.033921+0000","last_peered":"2024-09-10T19:14:09.033921+0000","last_clean":"2024-09-10T19:14:09.033921+0000","last_became_active":"2024-09-10T19:13:54.713538+0000","last_became_peered":"2024-09-10T19:13:54.713538+0000","last_unstale":"2024-09-10T19:14:09.033921+0000","last_undegraded":"2024-09-10T19:14:09.033921+0000","last_fullsized":"2024-09-10T19:14:09.033921+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:11:32.056254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:11:32.056254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0013010680000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,7],"acting":[4,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":92,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-09-10T19:14:09.042283+0000","last_change":"2024-09-10T19:12:49.527255+0000","last_active":"2024-09-10T19:14:09.042283+0000","last_peered":"2024-09-10T19:14:09.042283+0000","last_clean":"2024-09-10T19:14:09.042283+0000","last_became_active":"2024-09-10T19:10:07.690728+0000","last_became_peered":"2024-09-10T19:10:07.690728+0000","last_unstale":"2024-09-10T19:14:09.042283+0000","last_undegraded":"2024-09-10T19:14:09.042283+0000","last_fullsized":"2024-09-10T19:14:09.042283+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:12:49.527167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:12:49.527167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0040597050000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1b","version":"66'5","reported_seq":115,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-09-10T19:14:09.033501+0000","last_change":"2024-09-10T19:12:55.359526+0000","last_active":"2024-09-10T19:14:09.033501+0000","last_peered":"2024-09-10T19:14:09.033501+0000","last_clean":"2024-09-10T19:14:09.033501+0000","last_became_active":"2024-09-10T19:11:44.748296+0000","last_became_peered":"2024-09-10T19:11:44.748296+0000","last_unstale":"2024-09-10T19:14:09.033501+0000","last_undegraded":"2024-09-10T19:14:09.033501+0000","last_fullsized":"2024-09-10T19:14:09.033501+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'5","last_scrub_stamp":"2024-09-10T19:12:55.359379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:12:55.359379+0000","objects_scrubbed":2,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0050448070000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":95,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-09-10T19:14:09.042436+0000","last_change":"2024-09-10T19:13:54.720135+0000","last_active":"2024-09-10T19:14:09.042436+0000","last_peered":"2024-09-10T19:14:09.042436+0000","last_clean":"2024-09-10T19:14:09.042436+0000","last_became_active":"2024-09-10T19:13:54.719063+0000","last_became_peered":"2024-09-10T19:13:54.719063+0000","last_unstale":"2024-09-10T19:14:09.042436+0000","last_undegraded":"2024-09-10T19:14:09.042436+0000","last_fullsized":"2024-09-10T19:14:09.042436+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:11:43.142734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:11:43.142734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0013612330000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":93,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-09-10T19:14:09.041809+0000","last_change":"2024-09-10T19:13:54.723892+0000","last_active":"2024-09-10T19:14:09.041809+0000","last_peered":"2024-09-10T19:14:09.041809+0000","last_clean":"2024-09-10T19:14:09.041809+0000","last_became_active":"2024-09-10T19:13:54.723590+0000","last_became_peered":"2024-09-10T19:13:54.723590+0000","last_unstale":"2024-09-10T19:14:09.041809+0000","last_undegraded":"2024-09-10T19:14:09.041809+0000","last_fullsized":"2024-09-10T19:14:09.041809+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:11:45.411321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:11:45.411321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001536576,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":92,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-09-10T19:14:09.042352+0000","last_change":"2024-09-10T19:12:54.477202+0000","last_active":"2024-09-10T19:14:09.042352+0000","last_peered":"2024-09-10T19:14:09.042352+0000","last_clean":"2024-09-10T19:14:09.042352+0000","last_became_active":"2024-09-10T19:10:07.690174+0000","last_became_peered":"2024-09-10T19:10:07.690174+0000","last_unstale":"2024-09-10T19:14:09.042352+0000","last_undegraded":"2024-09-10T19:14:09.042352+0000","last_fullsized":"2024-09-10T19:14:09.042352+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:12:54.477149+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:12:54.477149+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0039768319999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1a","version":"66'9","reported_seq":105,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-09-10T19:14:09.034149+0000","last_change":"2024-09-10T19:13:02.431528+0000","last_active":"2024-09-10T19:14:09.034149+0000","last_peered":"2024-09-10T19:14:09.034149+0000","last_clean":"2024-09-10T19:14:09.034149+0000","last_became_active":"2024-09-10T19:10:09.688987+0000","last_became_peered":"2024-09-10T19:10:09.688987+0000","last_unstale":"2024-09-10T19:14:09.034149+0000","last_undegraded":"2024-09-10T19:14:09.034149+0000","last_fullsized":"2024-09-10T19:14:09.034149+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'9","last_scrub_stamp":"2024-09-10T19:13:02.431443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:13:02.431443+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:14:27.616203+0000","scrub_duration":0.0084913290000000006,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":88,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-09-10T19:14:09.042764+0000","last_change":"2024-09-10T19:12:51.449179+0000","last_active":"2024-09-10T19:14:09.042764+0000","last_peered":"2024-09-10T19:14:09.042764+0000","last_clean":"2024-09-10T19:14:09.042764+0000","last_became_active":"2024-09-10T19:10:11.701799+0000","last_became_peered":"2024-09-10T19:10:11.701799+0000","last_unstale":"2024-09-10T19:14:09.042764+0000","last_undegraded":"2024-09-10T19:14:09.042764+0000","last_fullsized":"2024-09-10T19:14:09.042764+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:12:51.449046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:12:51.449046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.002348826,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":93,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-09-10T19:14:09.038978+0000","last_change":"2024-09-10T19:13:54.719488+0000","last_active":"2024-09-10T19:14:09.038978+0000","last_peered":"2024-09-10T19:14:09.038978+0000","last_clean":"2024-09-10T19:14:09.038978+0000","last_became_active":"2024-09-10T19:13:54.718860+0000","last_became_peered":"2024-09-10T19:13:54.718860+0000","last_unstale":"2024-09-10T19:14:09.038978+0000","last_undegraded":"2024-09-10T19:14:09.038978+0000","last_fullsized":"2024-09-10T19:14:09.038978+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:11:12.970173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:11:12.970173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001527219,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.e","version":"122'11184","reported_seq":11590,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-09-10T19:14:19.653039+0000","last_change":"2024-09-10T19:13:52.171265+0000","last_active":"2024-09-10T19:14:19.653039+0000","last_peered":"2024-09-10T19:14:19.653039+0000","last_clean":"2024-09-10T19:14:19.653039+0000","last_became_active":"2024-09-10T19:12:11.032901+0000","last_became_peered":"2024-09-10T19:12:11.032901+0000","last_unstale":"2024-09-10T19:14:19.653039+0000","last_undegraded":"2024-09-10T19:14:19.653039+0000","last_fullsized":"2024-09-10T19:14:19.653039+0000","mapping_epoch":86,"log_start":"105'5962","ondisk_log_start":"105'5962","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"114'9197","last_scrub_stamp":"2024-09-10T19:13:52.169437+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:12:10.007465+0000","last_clean_scrub_stamp":"2024-09-10T19:13:52.169437+0000","objects_scrubbed":8820,"log_size":5222,"log_dups_size":0,"ondisk_log_size":5222,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:15:07.775054+0000","scrub_duration":3.857076422,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":439615488,"num_objects":6708,"num_object_clones":0,"num_object_copies":20124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6708,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11184,"num_write_kb":572544,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.d","version":"66'17","reported_seq":152,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-09-10T19:14:09.034567+0000","last_change":"2024-09-10T19:13:54.747117+0000","last_active":"2024-09-10T19:14:09.034567+0000","last_peered":"2024-09-10T19:14:09.034567+0000","last_clean":"2024-09-10T19:14:09.034567+0000","last_became_active":"2024-09-10T19:13:54.713411+0000","last_became_peered":"2024-09-10T19:13:54.713411+0000","last_unstale":"2024-09-10T19:14:09.034567+0000","last_undegraded":"2024-09-10T19:14:09.034567+0000","last_fullsized":"2024-09-10T19:14:09.034567+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'17","last_scrub_stamp":"2024-09-10T19:12:45.367252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:12:45.367252+0000","objects_scrubbed":8,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0031240310000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":92,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-09-10T19:14:09.040803+0000","last_change":"2024-09-10T19:12:46.519868+0000","last_active":"2024-09-10T19:14:09.040803+0000","last_peered":"2024-09-10T19:14:09.040803+0000","last_clean":"2024-09-10T19:14:09.040803+0000","last_became_active":"2024-09-10T19:10:07.691119+0000","last_became_peered":"2024-09-10T19:10:07.691119+0000","last_unstale":"2024-09-10T19:14:09.040803+0000","last_undegraded":"2024-09-10T19:14:09.040803+0000","last_fullsized":"2024-09-10T19:14:09.040803+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:12:46.519818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:11:22.280002+0000","last_clean_scrub_stamp":"2024-09-10T19:12:46.519818+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001139279,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":42,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-09-10T19:14:09.033649+0000","last_change":"2024-09-10T19:13:54.713806+0000","last_active":"2024-09-10T19:14:09.033649+0000","last_peered":"2024-09-10T19:14:09.033649+0000","last_clean":"2024-09-10T19:14:09.033649+0000","last_became_active":"2024-09-10T19:13:54.712883+0000","last_became_peered":"2024-09-10T19:13:54.712883+0000","last_unstale":"2024-09-10T19:14:09.033649+0000","last_undegraded":"2024-09-10T19:14:09.033649+0000","last_fullsized":"2024-09-10T19:14:09.033649+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:12:00.240841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:12:00.240841+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":86,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-09-10T19:14:09.038660+0000","last_change":"2024-09-10T19:12:49.342906+0000","last_active":"2024-09-10T19:14:09.038660+0000","last_peered":"2024-09-10T19:14:09.038660+0000","last_clean":"2024-09-10T19:14:09.038660+0000","last_became_active":"2024-09-10T19:10:13.722894+0000","last_became_peered":"2024-09-10T19:10:13.722894+0000","last_unstale":"2024-09-10T19:14:09.038660+0000","last_undegraded":"2024-09-10T19:14:09.038660+0000","last_fullsized":"2024-09-10T19:14:09.038660+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:12:49.342837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:12:49.342837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0092298559999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.f","version":"122'11008","reported_seq":11407,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-09-10T19:14:19.785808+0000","last_change":"2024-09-10T19:13:49.950750+0000","last_active":"2024-09-10T19:14:19.785808+0000","last_peered":"2024-09-10T19:14:19.785808+0000","last_clean":"2024-09-10T19:14:19.785808+0000","last_became_active":"2024-09-10T19:12:11.036181+0000","last_became_peered":"2024-09-10T19:12:11.036181+0000","last_unstale":"2024-09-10T19:14:19.785808+0000","last_undegraded":"2024-09-10T19:14:19.785808+0000","last_fullsized":"2024-09-10T19:14:19.785808+0000","mapping_epoch":86,"log_start":"105'6225","ondisk_log_start":"105'6225","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"114'8808","last_scrub_stamp":"2024-09-10T19:13:49.950668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:12:10.007465+0000","last_clean_scrub_stamp":"2024-09-10T19:13:49.950668+0000","objects_scrubbed":8620,"log_size":4783,"log_dups_size":0,"ondisk_log_size":4783,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:14:50.581709+0000","scrub_duration":5.7089046000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":423886848,"num_objects":6468,"num_object_clones":0,"num_object_copies":19404,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6468,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11008,"num_write_kb":559232,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.c","version":"66'10","reported_seq":114,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-09-10T19:14:09.033833+0000","last_change":"2024-09-10T19:13:51.416075+0000","last_active":"2024-09-10T19:14:09.033833+0000","last_peered":"2024-09-10T19:14:09.033833+0000","last_clean":"2024-09-10T19:14:09.033833+0000","last_became_active":"2024-09-10T19:10:09.688321+0000","last_became_peered":"2024-09-10T19:10:09.688321+0000","last_unstale":"2024-09-10T19:14:09.033833+0000","last_undegraded":"2024-09-10T19:14:09.033833+0000","last_fullsized":"2024-09-10T19:14:09.033833+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'10","last_scrub_stamp":"2024-09-10T19:13:51.415998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:13:51.415998+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:15:20.996413+0000","scrub_duration":0.005972799,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":42,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-09-10T19:14:09.035324+0000","last_change":"2024-09-10T19:13:54.731440+0000","last_active":"2024-09-10T19:14:09.035324+0000","last_peered":"2024-09-10T19:14:09.035324+0000","last_clean":"2024-09-10T19:14:09.035324+0000","last_became_active":"2024-09-10T19:13:54.731134+0000","last_became_peered":"2024-09-10T19:13:54.731134+0000","last_unstale":"2024-09-10T19:14:09.035324+0000","last_undegraded":"2024-09-10T19:14:09.035324+0000","last_fullsized":"2024-09-10T19:14:09.035324+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:11:26.110138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:11:26.110138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,5],"acting":[3,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"80'9","reported_seq":150,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-09-10T19:14:15.796561+0000","last_change":"2024-09-10T19:12:51.343854+0000","last_active":"2024-09-10T19:14:15.796561+0000","last_peered":"2024-09-10T19:14:15.796561+0000","last_clean":"2024-09-10T19:14:15.796561+0000","last_became_active":"2024-09-10T19:10:11.698547+0000","last_became_peered":"2024-09-10T19:10:11.698547+0000","last_unstale":"2024-09-10T19:14:15.796561+0000","last_undegraded":"2024-09-10T19:14:15.796561+0000","last_fullsized":"2024-09-10T19:14:15.796561+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"80'9","last_scrub_stamp":"2024-09-10T19:12:51.343776+0000","last_deep_scrub":"80'9","last_deep_scrub_stamp":"2024-09-10T19:11:25.156465+0000","last_clean_scrub_stamp":"2024-09-10T19:12:51.343776+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0026375819999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":95,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-09-10T19:14:09.033733+0000","last_change":"2024-09-10T19:14:00.299240+0000","last_active":"2024-09-10T19:14:09.033733+0000","last_peered":"2024-09-10T19:14:09.033733+0000","last_clean":"2024-09-10T19:14:09.033733+0000","last_became_active":"2024-09-10T19:13:54.719498+0000","last_became_peered":"2024-09-10T19:13:54.719498+0000","last_unstale":"2024-09-10T19:14:09.033733+0000","last_undegraded":"2024-09-10T19:14:09.033733+0000","last_fullsized":"2024-09-10T19:14:09.033733+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:11:26.096090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:11:26.096090+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0015599909999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.c","version":"122'10991","reported_seq":11386,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-09-10T19:14:19.639090+0000","last_change":"2024-09-10T19:13:53.867193+0000","last_active":"2024-09-10T19:14:19.639090+0000","last_peered":"2024-09-10T19:14:19.639090+0000","last_clean":"2024-09-10T19:14:19.639090+0000","last_became_active":"2024-09-10T19:12:11.029860+0000","last_became_peered":"2024-09-10T19:12:11.029860+0000","last_unstale":"2024-09-10T19:14:19.639090+0000","last_undegraded":"2024-09-10T19:14:19.639090+0000","last_fullsized":"2024-09-10T19:14:19.639090+0000","mapping_epoch":86,"log_start":"105'5982","ondisk_log_start":"105'5982","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"114'9191","last_scrub_stamp":"2024-09-10T19:13:53.867107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:12:10.007465+0000","last_clean_scrub_stamp":"2024-09-10T19:13:53.867107+0000","objects_scrubbed":8534,"log_size":5009,"log_dups_size":0,"ondisk_log_size":5009,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:15:11.622513+0000","scrub_duration":2.6024067190000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":434044928,"num_objects":6623,"num_object_clones":0,"num_object_copies":19869,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6623,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10991,"num_write_kb":563648,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.f","version":"66'15","reported_seq":95,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-09-10T19:14:09.041431+0000","last_change":"2024-09-10T19:13:54.742958+0000","last_active":"2024-09-10T19:14:09.041431+0000","last_peered":"2024-09-10T19:14:09.041431+0000","last_clean":"2024-09-10T19:14:09.041431+0000","last_became_active":"2024-09-10T19:13:54.711634+0000","last_became_peered":"2024-09-10T19:13:54.711634+0000","last_unstale":"2024-09-10T19:14:09.041431+0000","last_undegraded":"2024-09-10T19:14:09.041431+0000","last_fullsized":"2024-09-10T19:14:09.041431+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'15","last_scrub_stamp":"2024-09-10T19:11:59.232145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:11:59.232145+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":42,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-09-10T19:14:09.041828+0000","last_change":"2024-09-10T19:13:54.718740+0000","last_active":"2024-09-10T19:14:09.041828+0000","last_peered":"2024-09-10T19:14:09.041828+0000","last_clean":"2024-09-10T19:14:09.041828+0000","last_became_active":"2024-09-10T19:13:54.718452+0000","last_became_peered":"2024-09-10T19:13:54.718452+0000","last_unstale":"2024-09-10T19:14:09.041828+0000","last_undegraded":"2024-09-10T19:14:09.041828+0000","last_fullsized":"2024-09-10T19:14:09.041828+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:11:11.035954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:11:11.035954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.e","version":"66'8","reported_seq":145,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-09-10T19:14:15.795513+0000","last_change":"2024-09-10T19:13:40.475460+0000","last_active":"2024-09-10T19:14:15.795513+0000","last_peered":"2024-09-10T19:14:15.795513+0000","last_clean":"2024-09-10T19:14:15.795513+0000","last_became_active":"2024-09-10T19:10:11.699232+0000","last_became_peered":"2024-09-10T19:10:11.699232+0000","last_unstale":"2024-09-10T19:14:15.795513+0000","last_undegraded":"2024-09-10T19:14:15.795513+0000","last_fullsized":"2024-09-10T19:14:15.795513+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'8","last_scrub_stamp":"2024-09-10T19:13:40.475375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:13:40.475375+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:14:43.741970+0000","scrub_duration":0.035806152000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":93,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-09-10T19:14:09.042736+0000","last_change":"2024-09-10T19:13:54.723273+0000","last_active":"2024-09-10T19:14:09.042736+0000","last_peered":"2024-09-10T19:14:09.042736+0000","last_clean":"2024-09-10T19:14:09.042736+0000","last_became_active":"2024-09-10T19:13:54.722688+0000","last_became_peered":"2024-09-10T19:13:54.722688+0000","last_unstale":"2024-09-10T19:14:09.042736+0000","last_undegraded":"2024-09-10T19:14:09.042736+0000","last_fullsized":"2024-09-10T19:14:09.042736+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:11:18.208782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:11:18.208782+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001904481,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.d","version":"122'10983","reported_seq":11043,"reported_epoch":122,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2024-09-10T19:14:19.650977+0000","last_change":"2024-09-10T19:13:55.810177+0000","last_active":"2024-09-10T19:14:19.650977+0000","last_peered":"2024-09-10T19:14:19.650977+0000","last_clean":"2024-09-10T19:12:45.797387+0000","last_became_active":"2024-09-10T19:13:55.789208+0000","last_became_peered":"2024-09-10T19:13:55.789208+0000","last_unstale":"2024-09-10T19:14:19.650977+0000","last_undegraded":"2024-09-10T19:13:55.724690+0000","last_fullsized":"2024-09-10T19:13:55.711897+0000","mapping_epoch":116,"log_start":"105'6056","ondisk_log_start":"105'6056","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:12:10.007465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:12:10.007465+0000","last_clean_scrub_stamp":"2024-09-10T19:12:10.007465+0000","objects_scrubbed":0,"log_size":4927,"log_dups_size":0,"ondisk_log_size":4927,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":425590808,"num_objects":6495,"num_object_clones":0,"num_object_copies":19485,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6495,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6495,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10983,"num_write_kb":559233,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0],"avail_no_missing":["6","0"],"object_location_counts":[{"shards":"0,6","objects":6495}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.e","version":"84'16","reported_seq":132,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-09-10T19:14:09.033978+0000","last_change":"2024-09-10T19:13:54.741260+0000","last_active":"2024-09-10T19:14:09.033978+0000","last_peered":"2024-09-10T19:14:09.033978+0000","last_clean":"2024-09-10T19:14:09.033978+0000","last_became_active":"2024-09-10T19:13:54.717022+0000","last_became_peered":"2024-09-10T19:13:54.717022+0000","last_unstale":"2024-09-10T19:14:09.033978+0000","last_undegraded":"2024-09-10T19:14:09.033978+0000","last_fullsized":"2024-09-10T19:14:09.033978+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'16","last_scrub_stamp":"2024-09-10T19:11:56.157975+0000","last_deep_scrub":"84'16","last_deep_scrub_stamp":"2024-09-10T19:11:56.157975+0000","last_clean_scrub_stamp":"2024-09-10T19:11:56.157975+0000","objects_scrubbed":5,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0045462239999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,3],"acting":[4,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":99,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-09-10T19:14:09.043528+0000","last_change":"2024-09-10T19:13:54.721400+0000","last_active":"2024-09-10T19:14:09.043528+0000","last_peered":"2024-09-10T19:14:09.043528+0000","last_clean":"2024-09-10T19:14:09.043528+0000","last_became_active":"2024-09-10T19:13:54.720293+0000","last_became_peered":"2024-09-10T19:13:54.720293+0000","last_unstale":"2024-09-10T19:14:09.043528+0000","last_undegraded":"2024-09-10T19:14:09.043528+0000","last_fullsized":"2024-09-10T19:14:09.043528+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:11:53.066849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:11:53.066849+0000","last_clean_scrub_stamp":"2024-09-10T19:11:53.066849+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0024992679999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":88,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-09-10T19:14:09.045640+0000","last_change":"2024-09-10T19:13:06.358188+0000","last_active":"2024-09-10T19:14:09.045640+0000","last_peered":"2024-09-10T19:14:09.045640+0000","last_clean":"2024-09-10T19:14:09.045640+0000","last_became_active":"2024-09-10T19:10:11.692215+0000","last_became_peered":"2024-09-10T19:10:11.692215+0000","last_unstale":"2024-09-10T19:14:09.045640+0000","last_undegraded":"2024-09-10T19:14:09.045640+0000","last_fullsized":"2024-09-10T19:14:09.045640+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:13:06.358109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:13:06.358109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:14:34.922567+0000","scrub_duration":0.0026724549999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":86,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-09-10T19:14:09.036290+0000","last_change":"2024-09-10T19:13:54.304493+0000","last_active":"2024-09-10T19:14:09.036290+0000","last_peered":"2024-09-10T19:14:09.036290+0000","last_clean":"2024-09-10T19:14:09.036290+0000","last_became_active":"2024-09-10T19:10:14.053736+0000","last_became_peered":"2024-09-10T19:10:14.053736+0000","last_unstale":"2024-09-10T19:14:09.036290+0000","last_undegraded":"2024-09-10T19:14:09.036290+0000","last_fullsized":"2024-09-10T19:14:09.036290+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:13:54.304435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:13:54.304435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:15:07.353025+0000","scrub_duration":0.0011043820000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.2","version":"122'11156","reported_seq":17017,"reported_epoch":122,"state":"active+undersized+degraded+remapped+backfilling","last_fresh":"2024-09-10T19:14:19.769716+0000","last_change":"2024-09-10T19:13:55.940201+0000","last_active":"2024-09-10T19:14:19.769716+0000","last_peered":"2024-09-10T19:14:19.769716+0000","last_clean":"2024-09-10T19:12:45.804432+0000","last_became_active":"2024-09-10T19:13:55.881880+0000","last_became_peered":"2024-09-10T19:13:55.881880+0000","last_unstale":"2024-09-10T19:14:19.769716+0000","last_undegraded":"2024-09-10T19:13:55.719843+0000","last_fullsized":"2024-09-10T19:13:55.709265+0000","mapping_epoch":116,"log_start":"107'6907","ondisk_log_start":"107'6907","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:12:10.007465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:12:10.007465+0000","last_clean_scrub_stamp":"2024-09-10T19:12:10.007465+0000","objects_scrubbed":0,"log_size":4249,"log_dups_size":0,"ondisk_log_size":4249,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":437911552,"num_objects":6682,"num_object_clones":0,"num_object_copies":20046,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4131,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6682,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11156,"num_write_kb":570816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2899,"num_bytes_recovered":189988864,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7],"acting":[3,2],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":6682}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"66'14","reported_seq":110,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-09-10T19:14:09.033612+0000","last_change":"2024-09-10T19:12:48.462230+0000","last_active":"2024-09-10T19:14:09.033612+0000","last_peered":"2024-09-10T19:14:09.033612+0000","last_clean":"2024-09-10T19:14:09.033612+0000","last_became_active":"2024-09-10T19:10:09.686534+0000","last_became_peered":"2024-09-10T19:10:09.686534+0000","last_unstale":"2024-09-10T19:14:09.033612+0000","last_undegraded":"2024-09-10T19:14:09.033612+0000","last_fullsized":"2024-09-10T19:14:09.033612+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'14","last_scrub_stamp":"2024-09-10T19:12:48.462133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:12:48.462133+0000","objects_scrubbed":7,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.002699791,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":92,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-09-10T19:14:09.048084+0000","last_change":"2024-09-10T19:12:52.885643+0000","last_active":"2024-09-10T19:14:09.048084+0000","last_peered":"2024-09-10T19:14:09.048084+0000","last_clean":"2024-09-10T19:14:09.048084+0000","last_became_active":"2024-09-10T19:10:07.684864+0000","last_became_peered":"2024-09-10T19:10:07.684864+0000","last_unstale":"2024-09-10T19:14:09.048084+0000","last_undegraded":"2024-09-10T19:14:09.048084+0000","last_fullsized":"2024-09-10T19:14:09.048084+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:12:52.885563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:12:52.885563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0035531249999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.0","version":"117'10","reported_seq":158,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-09-10T19:14:15.798129+0000","last_change":"2024-09-10T19:13:54.735247+0000","last_active":"2024-09-10T19:14:15.798129+0000","last_peered":"2024-09-10T19:14:15.798129+0000","last_clean":"2024-09-10T19:14:15.798129+0000","last_became_active":"2024-09-10T19:13:54.709846+0000","last_became_peered":"2024-09-10T19:13:54.709846+0000","last_unstale":"2024-09-10T19:14:15.798129+0000","last_undegraded":"2024-09-10T19:14:15.798129+0000","last_fullsized":"2024-09-10T19:14:15.798129+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'8","last_scrub_stamp":"2024-09-10T19:11:29.884739+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:11:29.884739+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0014372650000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,0],"acting":[3,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":86,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-09-10T19:14:09.040487+0000","last_change":"2024-09-10T19:13:15.350815+0000","last_active":"2024-09-10T19:14:09.040487+0000","last_peered":"2024-09-10T19:14:09.040487+0000","last_clean":"2024-09-10T19:14:09.040487+0000","last_became_active":"2024-09-10T19:10:14.054456+0000","last_became_peered":"2024-09-10T19:10:14.054456+0000","last_unstale":"2024-09-10T19:14:09.040487+0000","last_undegraded":"2024-09-10T19:14:09.040487+0000","last_fullsized":"2024-09-10T19:14:09.040487+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:13:15.350720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:11:41.073833+0000","last_clean_scrub_stamp":"2024-09-10T19:13:15.350720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:14:37.565761+0000","scrub_duration":0.0014361000000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.3","version":"122'10978","reported_seq":11365,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-09-10T19:14:19.749609+0000","last_change":"2024-09-10T19:13:44.417102+0000","last_active":"2024-09-10T19:14:19.749609+0000","last_peered":"2024-09-10T19:14:19.749609+0000","last_clean":"2024-09-10T19:14:19.749609+0000","last_became_active":"2024-09-10T19:12:11.029906+0000","last_became_peered":"2024-09-10T19:12:11.029906+0000","last_unstale":"2024-09-10T19:14:19.749609+0000","last_undegraded":"2024-09-10T19:14:19.749609+0000","last_fullsized":"2024-09-10T19:14:19.749609+0000","mapping_epoch":86,"log_start":"107'6733","ondisk_log_start":"107'6733","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"112'8444","last_scrub_stamp":"2024-09-10T19:13:44.417011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:12:10.007465+0000","last_clean_scrub_stamp":"2024-09-10T19:13:44.417011+0000","objects_scrubbed":8333,"log_size":4245,"log_dups_size":0,"ondisk_log_size":4245,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:14:54.007266+0000","scrub_duration":3.2086845460000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":426770432,"num_objects":6512,"num_object_clones":0,"num_object_copies":19536,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6512,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10978,"num_write_kb":559680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"66'18","reported_seq":122,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-09-10T19:14:09.036184+0000","last_change":"2024-09-10T19:12:49.699111+0000","last_active":"2024-09-10T19:14:09.036184+0000","last_peered":"2024-09-10T19:14:09.036184+0000","last_clean":"2024-09-10T19:14:09.036184+0000","last_became_active":"2024-09-10T19:10:09.821836+0000","last_became_peered":"2024-09-10T19:10:09.821836+0000","last_unstale":"2024-09-10T19:14:09.036184+0000","last_undegraded":"2024-09-10T19:14:09.036184+0000","last_fullsized":"2024-09-10T19:14:09.036184+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'18","last_scrub_stamp":"2024-09-10T19:12:49.699054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:12:49.699054+0000","objects_scrubbed":8,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0073070890000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":42,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-09-10T19:14:09.047314+0000","last_change":"2024-09-10T19:13:54.711273+0000","last_active":"2024-09-10T19:14:09.047314+0000","last_peered":"2024-09-10T19:14:09.047314+0000","last_clean":"2024-09-10T19:14:09.047314+0000","last_became_active":"2024-09-10T19:13:54.710968+0000","last_became_peered":"2024-09-10T19:13:54.710968+0000","last_unstale":"2024-09-10T19:14:09.047314+0000","last_undegraded":"2024-09-10T19:14:09.047314+0000","last_fullsized":"2024-09-10T19:14:09.047314+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:11:42.202689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:11:42.202689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,3],"acting":[6,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":87,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-09-10T19:14:09.033357+0000","last_change":"2024-09-10T19:12:58.400261+0000","last_active":"2024-09-10T19:14:09.033357+0000","last_peered":"2024-09-10T19:14:09.033357+0000","last_clean":"2024-09-10T19:14:09.033357+0000","last_became_active":"2024-09-10T19:10:11.698054+0000","last_became_peered":"2024-09-10T19:10:11.698054+0000","last_unstale":"2024-09-10T19:14:09.033357+0000","last_undegraded":"2024-09-10T19:14:09.033357+0000","last_fullsized":"2024-09-10T19:14:09.033357+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:12:58.400206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:12:58.400206+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0071235179999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":85,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-09-10T19:14:09.034079+0000","last_change":"2024-09-10T19:12:57.409831+0000","last_active":"2024-09-10T19:14:09.034079+0000","last_peered":"2024-09-10T19:14:09.034079+0000","last_clean":"2024-09-10T19:14:09.034079+0000","last_became_active":"2024-09-10T19:10:13.722784+0000","last_became_peered":"2024-09-10T19:10:13.722784+0000","last_unstale":"2024-09-10T19:14:09.034079+0000","last_undegraded":"2024-09-10T19:14:09.034079+0000","last_fullsized":"2024-09-10T19:14:09.034079+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:12:57.409749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:12:57.409749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:14:23.578390+0000","scrub_duration":0.0087033260000000008,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.0","version":"122'10991","reported_seq":11055,"reported_epoch":122,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2024-09-10T19:14:19.615182+0000","last_change":"2024-09-10T19:13:55.804438+0000","last_active":"2024-09-10T19:14:19.615182+0000","last_peered":"2024-09-10T19:14:19.615182+0000","last_clean":"2024-09-10T19:12:45.799830+0000","last_became_active":"2024-09-10T19:13:55.794303+0000","last_became_peered":"2024-09-10T19:13:55.794303+0000","last_unstale":"2024-09-10T19:14:19.615182+0000","last_undegraded":"2024-09-10T19:13:55.727127+0000","last_fullsized":"2024-09-10T19:13:55.710552+0000","mapping_epoch":116,"log_start":"105'5942","ondisk_log_start":"105'5942","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:12:10.007465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:12:10.007465+0000","last_clean_scrub_stamp":"2024-09-10T19:12:10.007465+0000","objects_scrubbed":0,"log_size":5049,"log_dups_size":0,"ondisk_log_size":5049,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":425263104,"num_objects":6489,"num_object_clones":0,"num_object_copies":19467,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6489,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6489,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10991,"num_write_kb":559360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3],"avail_no_missing":["5","3"],"object_location_counts":[{"shards":"3,5","objects":6489}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.3","version":"104'27","reported_seq":134,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-09-10T19:14:09.039843+0000","last_change":"2024-09-10T19:13:02.226033+0000","last_active":"2024-09-10T19:14:09.039843+0000","last_peered":"2024-09-10T19:14:09.039843+0000","last_clean":"2024-09-10T19:14:09.039843+0000","last_became_active":"2024-09-10T19:10:09.823003+0000","last_became_peered":"2024-09-10T19:10:09.823003+0000","last_unstale":"2024-09-10T19:14:09.039843+0000","last_undegraded":"2024-09-10T19:14:09.039843+0000","last_fullsized":"2024-09-10T19:14:09.039843+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"104'27","last_scrub_stamp":"2024-09-10T19:13:02.225944+0000","last_deep_scrub":"104'27","last_deep_scrub_stamp":"2024-09-10T19:13:02.225944+0000","last_clean_scrub_stamp":"2024-09-10T19:13:02.225944+0000","objects_scrubbed":8,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0099233020000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":92,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-09-10T19:14:09.042353+0000","last_change":"2024-09-10T19:12:53.513000+0000","last_active":"2024-09-10T19:14:09.042353+0000","last_peered":"2024-09-10T19:14:09.042353+0000","last_clean":"2024-09-10T19:14:09.042353+0000","last_became_active":"2024-09-10T19:10:07.690325+0000","last_became_peered":"2024-09-10T19:10:07.690325+0000","last_unstale":"2024-09-10T19:14:09.042353+0000","last_undegraded":"2024-09-10T19:14:09.042353+0000","last_fullsized":"2024-09-10T19:14:09.042353+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:12:53.512923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:12:53.512923+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0087393290000000005,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":88,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-09-10T19:14:09.046828+0000","last_change":"2024-09-10T19:12:46.963842+0000","last_active":"2024-09-10T19:14:09.046828+0000","last_peered":"2024-09-10T19:14:09.046828+0000","last_clean":"2024-09-10T19:14:09.046828+0000","last_became_active":"2024-09-10T19:10:11.700370+0000","last_became_peered":"2024-09-10T19:10:11.700370+0000","last_unstale":"2024-09-10T19:14:09.046828+0000","last_undegraded":"2024-09-10T19:14:09.046828+0000","last_fullsized":"2024-09-10T19:14:09.046828+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:12:46.963746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:12:46.963746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001460004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":42,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-09-10T19:14:09.047877+0000","last_change":"2024-09-10T19:13:54.718253+0000","last_active":"2024-09-10T19:14:09.047877+0000","last_peered":"2024-09-10T19:14:09.047877+0000","last_clean":"2024-09-10T19:14:09.047877+0000","last_became_active":"2024-09-10T19:13:54.717977+0000","last_became_peered":"2024-09-10T19:13:54.717977+0000","last_unstale":"2024-09-10T19:14:09.047877+0000","last_undegraded":"2024-09-10T19:14:09.047877+0000","last_fullsized":"2024-09-10T19:14:09.047877+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:11:51.187109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:11:51.187109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,7],"acting":[6,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.1","version":"122'10916","reported_seq":11306,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-09-10T19:14:19.631263+0000","last_change":"2024-09-10T19:13:45.105571+0000","last_active":"2024-09-10T19:14:19.631263+0000","last_peered":"2024-09-10T19:14:19.631263+0000","last_clean":"2024-09-10T19:14:19.631263+0000","last_became_active":"2024-09-10T19:12:11.030399+0000","last_became_peered":"2024-09-10T19:12:11.030399+0000","last_unstale":"2024-09-10T19:14:19.631263+0000","last_undegraded":"2024-09-10T19:14:19.631263+0000","last_fullsized":"2024-09-10T19:14:19.631263+0000","mapping_epoch":86,"log_start":"105'5978","ondisk_log_start":"105'5978","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"113'8485","last_scrub_stamp":"2024-09-10T19:13:45.105477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:12:10.007465+0000","last_clean_scrub_stamp":"2024-09-10T19:13:45.105477+0000","objects_scrubbed":8400,"log_size":4938,"log_dups_size":0,"ondisk_log_size":4938,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:14:48.386316+0000","scrub_duration":2.6533322940000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":424673280,"num_objects":6480,"num_object_clones":0,"num_object_copies":19440,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6480,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10916,"num_write_kb":556672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.2","version":"66'10","reported_seq":58,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-09-10T19:14:09.041945+0000","last_change":"2024-09-10T19:13:54.748881+0000","last_active":"2024-09-10T19:14:09.041945+0000","last_peered":"2024-09-10T19:14:09.041945+0000","last_clean":"2024-09-10T19:14:09.041945+0000","last_became_active":"2024-09-10T19:13:54.714542+0000","last_became_peered":"2024-09-10T19:13:54.714542+0000","last_unstale":"2024-09-10T19:14:09.041945+0000","last_undegraded":"2024-09-10T19:14:09.041945+0000","last_fullsized":"2024-09-10T19:14:09.041945+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'10","last_scrub_stamp":"2024-09-10T19:12:56.360409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:12:56.360409+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:14:21.848263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":14,"num_read_kb":9,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,0],"acting":[5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":42,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-09-10T19:14:09.041608+0000","last_change":"2024-09-10T19:13:54.718993+0000","last_active":"2024-09-10T19:14:09.041608+0000","last_peered":"2024-09-10T19:14:09.041608+0000","last_clean":"2024-09-10T19:14:09.041608+0000","last_became_active":"2024-09-10T19:13:54.718267+0000","last_became_peered":"2024-09-10T19:13:54.718267+0000","last_unstale":"2024-09-10T19:14:09.041608+0000","last_undegraded":"2024-09-10T19:14:09.041608+0000","last_fullsized":"2024-09-10T19:14:09.041608+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:11:55.227355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:11:55.227355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3","version":"84'9","reported_seq":147,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-09-10T19:14:15.795685+0000","last_change":"2024-09-10T19:13:45.236207+0000","last_active":"2024-09-10T19:14:15.795685+0000","last_peered":"2024-09-10T19:14:15.795685+0000","last_clean":"2024-09-10T19:14:15.795685+0000","last_became_active":"2024-09-10T19:10:11.699690+0000","last_became_peered":"2024-09-10T19:10:11.699690+0000","last_unstale":"2024-09-10T19:14:15.795685+0000","last_undegraded":"2024-09-10T19:14:15.795685+0000","last_fullsized":"2024-09-10T19:14:15.795685+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'9","last_scrub_stamp":"2024-09-10T19:13:45.236094+0000","last_deep_scrub":"84'9","last_deep_scrub_stamp":"2024-09-10T19:11:53.879272+0000","last_clean_scrub_stamp":"2024-09-10T19:13:45.236094+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:15:04.114110+0000","scrub_duration":0.036165602999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":86,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-09-10T19:14:09.038851+0000","last_change":"2024-09-10T19:13:40.234378+0000","last_active":"2024-09-10T19:14:09.038851+0000","last_peered":"2024-09-10T19:14:09.038851+0000","last_clean":"2024-09-10T19:14:09.038851+0000","last_became_active":"2024-09-10T19:10:13.720403+0000","last_became_peered":"2024-09-10T19:10:13.720403+0000","last_unstale":"2024-09-10T19:14:09.038851+0000","last_undegraded":"2024-09-10T19:14:09.038851+0000","last_fullsized":"2024-09-10T19:14:09.038851+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:13:40.234289+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:13:40.234289+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:14:58.963116+0000","scrub_duration":0.0086405590000000008,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.7","version":"122'10991","reported_seq":11378,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-09-10T19:14:19.618610+0000","last_change":"2024-09-10T19:13:55.214546+0000","last_active":"2024-09-10T19:14:19.618610+0000","last_peered":"2024-09-10T19:14:19.618610+0000","last_clean":"2024-09-10T19:14:19.618610+0000","last_became_active":"2024-09-10T19:12:11.032857+0000","last_became_peered":"2024-09-10T19:12:11.032857+0000","last_unstale":"2024-09-10T19:14:19.618610+0000","last_undegraded":"2024-09-10T19:14:19.618610+0000","last_fullsized":"2024-09-10T19:14:19.618610+0000","mapping_epoch":86,"log_start":"105'6025","ondisk_log_start":"105'6025","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"114'9143","last_scrub_stamp":"2024-09-10T19:13:55.214477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:12:10.007465+0000","last_clean_scrub_stamp":"2024-09-10T19:13:55.214477+0000","objects_scrubbed":8339,"log_size":4966,"log_dups_size":0,"ondisk_log_size":4966,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:15:20.270435+0000","scrub_duration":0.96843253399999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":425394176,"num_objects":6491,"num_object_clones":0,"num_object_copies":19473,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6491,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10991,"num_write_kb":559424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.4","version":"117'30","reported_seq":144,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-09-10T19:14:15.795639+0000","last_change":"2024-09-10T19:13:54.752131+0000","last_active":"2024-09-10T19:14:15.795639+0000","last_peered":"2024-09-10T19:14:15.795639+0000","last_clean":"2024-09-10T19:14:15.795639+0000","last_became_active":"2024-09-10T19:13:54.715642+0000","last_became_peered":"2024-09-10T19:13:54.715642+0000","last_unstale":"2024-09-10T19:14:15.795639+0000","last_undegraded":"2024-09-10T19:14:15.795639+0000","last_fullsized":"2024-09-10T19:14:15.795639+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'28","last_scrub_stamp":"2024-09-10T19:12:45.260189+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:12:45.260189+0000","objects_scrubbed":0,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":47,"num_read_kb":32,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":358,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":99,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-09-10T19:14:09.042190+0000","last_change":"2024-09-10T19:13:54.730667+0000","last_active":"2024-09-10T19:14:09.042190+0000","last_peered":"2024-09-10T19:14:09.042190+0000","last_clean":"2024-09-10T19:14:09.042190+0000","last_became_active":"2024-09-10T19:13:54.730096+0000","last_became_peered":"2024-09-10T19:13:54.730096+0000","last_unstale":"2024-09-10T19:14:09.042190+0000","last_undegraded":"2024-09-10T19:14:09.042190+0000","last_fullsized":"2024-09-10T19:14:09.042190+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:11:41.429402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:11:41.429402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0014226359999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":95,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-09-10T19:14:09.040746+0000","last_change":"2024-09-10T19:13:54.714602+0000","last_active":"2024-09-10T19:14:09.040746+0000","last_peered":"2024-09-10T19:14:09.040746+0000","last_clean":"2024-09-10T19:14:09.040746+0000","last_became_active":"2024-09-10T19:13:54.714017+0000","last_became_peered":"2024-09-10T19:13:54.714017+0000","last_unstale":"2024-09-10T19:14:09.040746+0000","last_undegraded":"2024-09-10T19:14:09.040746+0000","last_fullsized":"2024-09-10T19:14:09.040746+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:11:14.985249+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:11:14.985249+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001460161,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,6],"acting":[0,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":76,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-09-10T19:14:09.036312+0000","last_change":"2024-09-10T19:12:46.691448+0000","last_active":"2024-09-10T19:14:09.036312+0000","last_peered":"2024-09-10T19:14:09.036312+0000","last_clean":"2024-09-10T19:14:09.036312+0000","last_became_active":"2024-09-10T19:10:13.721744+0000","last_became_peered":"2024-09-10T19:10:13.721744+0000","last_unstale":"2024-09-10T19:14:09.036312+0000","last_undegraded":"2024-09-10T19:14:09.036312+0000","last_fullsized":"2024-09-10T19:14:09.036312+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:12:46.691353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:12:46.691353+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0014482360000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.4","version":"122'10997","reported_seq":17331,"reported_epoch":122,"state":"active+undersized+degraded+remapped+backfilling","last_fresh":"2024-09-10T19:14:19.812039+0000","last_change":"2024-09-10T19:13:55.827146+0000","last_active":"2024-09-10T19:14:19.812039+0000","last_peered":"2024-09-10T19:14:19.812039+0000","last_clean":"2024-09-10T19:12:45.806743+0000","last_became_active":"2024-09-10T19:13:55.781235+0000","last_became_peered":"2024-09-10T19:13:55.781235+0000","last_unstale":"2024-09-10T19:14:19.812039+0000","last_undegraded":"2024-09-10T19:13:55.814241+0000","last_fullsized":"2024-09-10T19:13:55.709078+0000","mapping_epoch":116,"log_start":"106'6309","ondisk_log_start":"106'6309","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:12:10.007465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:12:10.007465+0000","last_clean_scrub_stamp":"2024-09-10T19:12:10.007465+0000","objects_scrubbed":0,"log_size":4688,"log_dups_size":0,"ondisk_log_size":4688,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":418971648,"num_objects":6393,"num_object_clones":0,"num_object_copies":19179,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3669,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6393,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10997,"num_write_kb":556480,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3135,"num_bytes_recovered":205324288,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,6],"acting":[0,4],"avail_no_missing":["0","4"],"object_location_counts":[{"shards":"0,4","objects":6393}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.7","version":"82'18","reported_seq":115,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-09-10T19:14:09.043691+0000","last_change":"2024-09-10T19:13:54.737836+0000","last_active":"2024-09-10T19:14:09.043691+0000","last_peered":"2024-09-10T19:14:09.043691+0000","last_clean":"2024-09-10T19:14:09.043691+0000","last_became_active":"2024-09-10T19:13:54.712065+0000","last_became_peered":"2024-09-10T19:13:54.712065+0000","last_unstale":"2024-09-10T19:14:09.043691+0000","last_undegraded":"2024-09-10T19:14:09.043691+0000","last_fullsized":"2024-09-10T19:14:09.043691+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'18","last_scrub_stamp":"2024-09-10T19:11:49.207309+0000","last_deep_scrub":"82'18","last_deep_scrub_stamp":"2024-09-10T19:11:49.207309+0000","last_clean_scrub_stamp":"2024-09-10T19:11:49.207309+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":19,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.0","version":"67'5","reported_seq":166,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-09-10T19:14:17.923108+0000","last_change":"2024-09-10T19:13:54.749420+0000","last_active":"2024-09-10T19:14:17.923108+0000","last_peered":"2024-09-10T19:14:17.923108+0000","last_clean":"2024-09-10T19:14:17.923108+0000","last_became_active":"2024-09-10T19:13:54.715570+0000","last_became_peered":"2024-09-10T19:13:54.715570+0000","last_unstale":"2024-09-10T19:14:17.923108+0000","last_undegraded":"2024-09-10T19:14:17.923108+0000","last_fullsized":"2024-09-10T19:14:17.923108+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'5","last_scrub_stamp":"2024-09-10T19:11:16.007261+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:11:16.007261+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00034454499999999997,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":95,"num_read_kb":90,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":389,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,0],"acting":[2,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":92,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-09-10T19:14:09.037593+0000","last_change":"2024-09-10T19:12:53.318459+0000","last_active":"2024-09-10T19:14:09.037593+0000","last_peered":"2024-09-10T19:14:09.037593+0000","last_clean":"2024-09-10T19:14:09.037593+0000","last_became_active":"2024-09-10T19:10:07.681456+0000","last_became_peered":"2024-09-10T19:10:07.681456+0000","last_unstale":"2024-09-10T19:14:09.037593+0000","last_undegraded":"2024-09-10T19:14:09.037593+0000","last_fullsized":"2024-09-10T19:14:09.037593+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:12:53.318408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:12:53.318408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:14:19.519971+0000","scrub_duration":0.01029277,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":88,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-09-10T19:14:09.039166+0000","last_change":"2024-09-10T19:13:00.366695+0000","last_active":"2024-09-10T19:14:09.039166+0000","last_peered":"2024-09-10T19:14:09.039166+0000","last_clean":"2024-09-10T19:14:09.039166+0000","last_became_active":"2024-09-10T19:10:11.695205+0000","last_became_peered":"2024-09-10T19:10:11.695205+0000","last_unstale":"2024-09-10T19:14:09.039166+0000","last_undegraded":"2024-09-10T19:14:09.039166+0000","last_fullsized":"2024-09-10T19:14:09.039166+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:13:00.366610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:13:00.366610+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:14:22.508784+0000","scrub_duration":0.0023966700000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":79,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-09-10T19:14:09.041329+0000","last_change":"2024-09-10T19:13:44.394892+0000","last_active":"2024-09-10T19:14:09.041329+0000","last_peered":"2024-09-10T19:14:09.041329+0000","last_clean":"2024-09-10T19:14:09.041329+0000","last_became_active":"2024-09-10T19:10:14.053697+0000","last_became_peered":"2024-09-10T19:10:14.053697+0000","last_unstale":"2024-09-10T19:14:09.041329+0000","last_undegraded":"2024-09-10T19:14:09.041329+0000","last_fullsized":"2024-09-10T19:14:09.041329+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:12:06.039981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:12:06.039981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0013437499999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.5","version":"122'10998","reported_seq":11390,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-09-10T19:14:19.581099+0000","last_change":"2024-09-10T19:13:55.551583+0000","last_active":"2024-09-10T19:14:19.581099+0000","last_peered":"2024-09-10T19:14:19.581099+0000","last_clean":"2024-09-10T19:14:19.581099+0000","last_became_active":"2024-09-10T19:12:11.032880+0000","last_became_peered":"2024-09-10T19:12:11.032880+0000","last_unstale":"2024-09-10T19:14:19.581099+0000","last_undegraded":"2024-09-10T19:14:19.581099+0000","last_fullsized":"2024-09-10T19:14:19.581099+0000","mapping_epoch":86,"log_start":"107'6552","ondisk_log_start":"107'6552","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"115'9199","last_scrub_stamp":"2024-09-10T19:13:55.551514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:12:10.007465+0000","last_clean_scrub_stamp":"2024-09-10T19:13:55.551514+0000","objects_scrubbed":8461,"log_size":4446,"log_dups_size":0,"ondisk_log_size":4446,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:15:07.317469+0000","scrub_duration":1.028952525,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436600832,"num_objects":6662,"num_object_clones":0,"num_object_copies":19986,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6662,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10998,"num_write_kb":565120,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"66'12","reported_seq":129,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-09-10T19:14:09.040310+0000","last_change":"2024-09-10T19:13:54.735877+0000","last_active":"2024-09-10T19:14:09.040310+0000","last_peered":"2024-09-10T19:14:09.040310+0000","last_clean":"2024-09-10T19:14:09.040310+0000","last_became_active":"2024-09-10T19:13:54.719200+0000","last_became_peered":"2024-09-10T19:13:54.719200+0000","last_unstale":"2024-09-10T19:14:09.040310+0000","last_undegraded":"2024-09-10T19:14:09.040310+0000","last_fullsized":"2024-09-10T19:14:09.040310+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'12","last_scrub_stamp":"2024-09-10T19:11:38.936857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:11:38.936857+0000","objects_scrubbed":6,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0016916909999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,7],"acting":[0,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"61'1","reported_seq":98,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-09-10T19:14:09.046737+0000","last_change":"2024-09-10T19:13:06.265721+0000","last_active":"2024-09-10T19:14:09.046737+0000","last_peered":"2024-09-10T19:14:09.046737+0000","last_clean":"2024-09-10T19:14:09.046737+0000","last_became_active":"2024-09-10T19:10:07.683259+0000","last_became_peered":"2024-09-10T19:10:07.683259+0000","last_unstale":"2024-09-10T19:14:09.046737+0000","last_undegraded":"2024-09-10T19:14:09.046737+0000","last_fullsized":"2024-09-10T19:14:09.046737+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'1","last_scrub_stamp":"2024-09-10T19:13:06.265646+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:13:06.265646+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:14:31.103103+0000","scrub_duration":0.012205384,"objects_trimmed":0,"snaptrim_duration":0.0014341130000000001,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":99,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-09-10T19:14:09.043637+0000","last_change":"2024-09-10T19:13:54.721892+0000","last_active":"2024-09-10T19:14:09.043637+0000","last_peered":"2024-09-10T19:14:09.043637+0000","last_clean":"2024-09-10T19:14:09.043637+0000","last_became_active":"2024-09-10T19:13:54.721486+0000","last_became_peered":"2024-09-10T19:13:54.721486+0000","last_unstale":"2024-09-10T19:14:09.043637+0000","last_undegraded":"2024-09-10T19:14:09.043637+0000","last_fullsized":"2024-09-10T19:14:09.043637+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:11:21.301058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:11:21.301058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0021074710000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":105,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-09-10T19:14:09.043614+0000","last_change":"2024-09-10T19:13:54.712428+0000","last_active":"2024-09-10T19:14:09.043614+0000","last_peered":"2024-09-10T19:14:09.043614+0000","last_clean":"2024-09-10T19:14:09.043614+0000","last_became_active":"2024-09-10T19:13:54.711685+0000","last_became_peered":"2024-09-10T19:13:54.711685+0000","last_unstale":"2024-09-10T19:14:09.043614+0000","last_undegraded":"2024-09-10T19:14:09.043614+0000","last_fullsized":"2024-09-10T19:14:09.043614+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:12:44.527561+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:12:44.527561+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.022638367999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,4],"acting":[5,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":42,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-09-10T19:14:09.042002+0000","last_change":"2024-09-10T19:13:54.723228+0000","last_active":"2024-09-10T19:14:09.042002+0000","last_peered":"2024-09-10T19:14:09.042002+0000","last_clean":"2024-09-10T19:14:09.042002+0000","last_became_active":"2024-09-10T19:13:54.722937+0000","last_became_peered":"2024-09-10T19:13:54.722937+0000","last_unstale":"2024-09-10T19:14:09.042002+0000","last_undegraded":"2024-09-10T19:14:09.042002+0000","last_fullsized":"2024-09-10T19:14:09.042002+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:12:05.165500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:12:05.165500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.6","version":"122'10842","reported_seq":10889,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-09-10T19:14:19.794319+0000","last_change":"2024-09-10T19:13:43.239851+0000","last_active":"2024-09-10T19:14:19.794319+0000","last_peered":"2024-09-10T19:14:19.794319+0000","last_clean":"2024-09-10T19:14:19.794319+0000","last_became_active":"2024-09-10T19:12:11.035290+0000","last_became_peered":"2024-09-10T19:12:11.035290+0000","last_unstale":"2024-09-10T19:14:19.794319+0000","last_undegraded":"2024-09-10T19:14:19.794319+0000","last_fullsized":"2024-09-10T19:14:19.794319+0000","mapping_epoch":86,"log_start":"105'6142","ondisk_log_start":"105'6142","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:12:10.007465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:12:10.007465+0000","last_clean_scrub_stamp":"2024-09-10T19:12:10.007465+0000","objects_scrubbed":0,"log_size":4700,"log_dups_size":0,"ondisk_log_size":4700,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":423493632,"num_objects":6462,"num_object_clones":0,"num_object_copies":19386,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6462,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10842,"num_write_kb":553728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.5","version":"80'21","reported_seq":119,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-09-10T19:14:09.049091+0000","last_change":"2024-09-10T19:12:54.860083+0000","last_active":"2024-09-10T19:14:09.049091+0000","last_peered":"2024-09-10T19:14:09.049091+0000","last_clean":"2024-09-10T19:14:09.049091+0000","last_became_active":"2024-09-10T19:10:09.822184+0000","last_became_peered":"2024-09-10T19:10:09.822184+0000","last_unstale":"2024-09-10T19:14:09.049091+0000","last_undegraded":"2024-09-10T19:14:09.049091+0000","last_fullsized":"2024-09-10T19:14:09.049091+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"80'21","last_scrub_stamp":"2024-09-10T19:12:54.860013+0000","last_deep_scrub":"80'21","last_deep_scrub_stamp":"2024-09-10T19:11:39.836144+0000","last_clean_scrub_stamp":"2024-09-10T19:12:54.860013+0000","objects_scrubbed":7,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0021624890000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"63'2","reported_seq":99,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-09-10T19:14:09.035391+0000","last_change":"2024-09-10T19:13:00.752806+0000","last_active":"2024-09-10T19:14:09.035391+0000","last_peered":"2024-09-10T19:14:09.035391+0000","last_clean":"2024-09-10T19:14:09.035391+0000","last_became_active":"2024-09-10T19:10:07.684784+0000","last_became_peered":"2024-09-10T19:10:07.684784+0000","last_unstale":"2024-09-10T19:14:09.035391+0000","last_undegraded":"2024-09-10T19:14:09.035391+0000","last_fullsized":"2024-09-10T19:14:09.035391+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'2","last_scrub_stamp":"2024-09-10T19:13:00.752715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:13:00.752715+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0083291979999999995,"objects_trimmed":0,"snaptrim_duration":0.00075522999999999996,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":102,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-09-10T19:14:09.045496+0000","last_change":"2024-09-10T19:13:50.411573+0000","last_active":"2024-09-10T19:14:09.045496+0000","last_peered":"2024-09-10T19:14:09.045496+0000","last_clean":"2024-09-10T19:14:09.045496+0000","last_became_active":"2024-09-10T19:10:07.687596+0000","last_became_peered":"2024-09-10T19:10:07.687596+0000","last_unstale":"2024-09-10T19:14:09.045496+0000","last_undegraded":"2024-09-10T19:14:09.045496+0000","last_fullsized":"2024-09-10T19:14:09.045496+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:13:50.411500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:13:50.411500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:15:20.319236+0000","scrub_duration":0.0032604869999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"74'79","reported_seq":137,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-09-10T19:14:09.043778+0000","last_change":"2024-09-10T19:13:46.419002+0000","last_active":"2024-09-10T19:14:09.043778+0000","last_peered":"2024-09-10T19:14:09.043778+0000","last_clean":"2024-09-10T19:14:09.043778+0000","last_became_active":"2024-09-10T19:09:56.634919+0000","last_became_peered":"2024-09-10T19:09:56.634919+0000","last_unstale":"2024-09-10T19:14:09.043778+0000","last_undegraded":"2024-09-10T19:14:09.043778+0000","last_fullsized":"2024-09-10T19:14:09.043778+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'79","last_scrub_stamp":"2024-09-10T19:13:46.418921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:08:00.699315+0000","last_clean_scrub_stamp":"2024-09-10T19:13:46.418921+0000","objects_scrubbed":2,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:14:48.812041+0000","scrub_duration":0.0029827999999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":88,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-09-10T19:14:09.041940+0000","last_change":"2024-09-10T19:13:47.453309+0000","last_active":"2024-09-10T19:14:09.041940+0000","last_peered":"2024-09-10T19:14:09.041940+0000","last_clean":"2024-09-10T19:14:09.041940+0000","last_became_active":"2024-09-10T19:10:11.694986+0000","last_became_peered":"2024-09-10T19:10:11.694986+0000","last_unstale":"2024-09-10T19:14:09.041940+0000","last_undegraded":"2024-09-10T19:14:09.041940+0000","last_fullsized":"2024-09-10T19:14:09.041940+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:13:47.453228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:13:47.453228+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:15:07.672581+0000","scrub_duration":0.010282463,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":86,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-09-10T19:14:09.043772+0000","last_change":"2024-09-10T19:13:05.390398+0000","last_active":"2024-09-10T19:14:09.043772+0000","last_peered":"2024-09-10T19:14:09.043772+0000","last_clean":"2024-09-10T19:14:09.043772+0000","last_became_active":"2024-09-10T19:10:13.720647+0000","last_became_peered":"2024-09-10T19:10:13.720647+0000","last_unstale":"2024-09-10T19:14:09.043772+0000","last_undegraded":"2024-09-10T19:14:09.043772+0000","last_fullsized":"2024-09-10T19:14:09.043772+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:13:05.390317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:13:05.390317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0058030640000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.9","version":"122'10896","reported_seq":10963,"reported_epoch":122,"state":"active+undersized+degraded+remapped+backfill_toofull","last_fresh":"2024-09-10T19:14:19.747157+0000","last_change":"2024-09-10T19:13:55.963920+0000","last_active":"2024-09-10T19:14:19.747157+0000","last_peered":"2024-09-10T19:14:19.747157+0000","last_clean":"2024-09-10T19:12:45.783536+0000","last_became_active":"2024-09-10T19:13:55.890181+0000","last_became_peered":"2024-09-10T19:13:55.890181+0000","last_unstale":"2024-09-10T19:14:19.747157+0000","last_undegraded":"2024-09-10T19:13:55.954251+0000","last_fullsized":"2024-09-10T19:13:55.708743+0000","mapping_epoch":116,"log_start":"106'6185","ondisk_log_start":"106'6185","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:12:10.007465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:12:10.007465+0000","last_clean_scrub_stamp":"2024-09-10T19:12:10.007465+0000","objects_scrubbed":0,"log_size":4711,"log_dups_size":0,"ondisk_log_size":4711,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":427163648,"num_objects":6518,"num_object_clones":0,"num_object_copies":19554,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6518,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6518,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10896,"num_write_kb":557248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4],"avail_no_missing":["0","4"],"object_location_counts":[{"shards":"0,4","objects":6518}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"66'19","reported_seq":148,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-09-10T19:14:09.047412+0000","last_change":"2024-09-10T19:13:54.755526+0000","last_active":"2024-09-10T19:14:09.047412+0000","last_peered":"2024-09-10T19:14:09.047412+0000","last_clean":"2024-09-10T19:14:09.047412+0000","last_became_active":"2024-09-10T19:13:54.731274+0000","last_became_peered":"2024-09-10T19:13:54.731274+0000","last_unstale":"2024-09-10T19:14:09.047412+0000","last_undegraded":"2024-09-10T19:14:09.047412+0000","last_fullsized":"2024-09-10T19:14:09.047412+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'19","last_scrub_stamp":"2024-09-10T19:11:42.941375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:11:42.941375+0000","objects_scrubbed":9,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0021647599999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,5],"acting":[6,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":92,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-09-10T19:14:09.038218+0000","last_change":"2024-09-10T19:13:05.404535+0000","last_active":"2024-09-10T19:14:09.038218+0000","last_peered":"2024-09-10T19:14:09.038218+0000","last_clean":"2024-09-10T19:14:09.038218+0000","last_became_active":"2024-09-10T19:10:07.686236+0000","last_became_peered":"2024-09-10T19:10:07.686236+0000","last_unstale":"2024-09-10T19:14:09.038218+0000","last_undegraded":"2024-09-10T19:14:09.038218+0000","last_fullsized":"2024-09-10T19:14:09.038218+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:13:05.404457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:13:05.404457+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0037223199999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":88,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-09-10T19:14:09.038716+0000","last_change":"2024-09-10T19:12:55.335929+0000","last_active":"2024-09-10T19:14:09.038716+0000","last_peered":"2024-09-10T19:14:09.038716+0000","last_clean":"2024-09-10T19:14:09.038716+0000","last_became_active":"2024-09-10T19:10:11.699745+0000","last_became_peered":"2024-09-10T19:10:11.699745+0000","last_unstale":"2024-09-10T19:14:09.038716+0000","last_undegraded":"2024-09-10T19:14:09.038716+0000","last_fullsized":"2024-09-10T19:14:09.038716+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:12:55.335834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:12:55.335834+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0060173240000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":86,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-09-10T19:14:09.043703+0000","last_change":"2024-09-10T19:13:14.311675+0000","last_active":"2024-09-10T19:14:09.043703+0000","last_peered":"2024-09-10T19:14:09.043703+0000","last_clean":"2024-09-10T19:14:09.043703+0000","last_became_active":"2024-09-10T19:10:13.719973+0000","last_became_peered":"2024-09-10T19:10:13.719973+0000","last_unstale":"2024-09-10T19:14:09.043703+0000","last_undegraded":"2024-09-10T19:14:09.043703+0000","last_fullsized":"2024-09-10T19:14:09.043703+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:13:14.311620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:13:14.311620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:14:23.452754+0000","scrub_duration":0.0054101059999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.8","version":"122'11094","reported_seq":11498,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-09-10T19:14:19.692997+0000","last_change":"2024-09-10T19:13:49.740030+0000","last_active":"2024-09-10T19:14:19.692997+0000","last_peered":"2024-09-10T19:14:19.692997+0000","last_clean":"2024-09-10T19:14:19.692997+0000","last_became_active":"2024-09-10T19:12:11.031441+0000","last_became_peered":"2024-09-10T19:12:11.031441+0000","last_unstale":"2024-09-10T19:14:19.692997+0000","last_undegraded":"2024-09-10T19:14:19.692997+0000","last_fullsized":"2024-09-10T19:14:19.692997+0000","mapping_epoch":86,"log_start":"105'5861","ondisk_log_start":"105'5861","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"113'8888","last_scrub_stamp":"2024-09-10T19:13:49.739947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:12:10.007465+0000","last_clean_scrub_stamp":"2024-09-10T19:13:49.739947+0000","objects_scrubbed":8760,"log_size":5233,"log_dups_size":0,"ondisk_log_size":5233,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:14:53.397517+0000","scrub_duration":3.4609629229999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":430047232,"num_objects":6562,"num_object_clones":0,"num_object_copies":19686,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6562,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11094,"num_write_kb":564992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.b","version":"66'9","reported_seq":123,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-09-10T19:14:09.039814+0000","last_change":"2024-09-10T19:13:54.736863+0000","last_active":"2024-09-10T19:14:09.039814+0000","last_peered":"2024-09-10T19:14:09.039814+0000","last_clean":"2024-09-10T19:14:09.039814+0000","last_became_active":"2024-09-10T19:13:54.710613+0000","last_became_peered":"2024-09-10T19:13:54.710613+0000","last_unstale":"2024-09-10T19:14:09.039814+0000","last_undegraded":"2024-09-10T19:14:09.039814+0000","last_fullsized":"2024-09-10T19:14:09.039814+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'9","last_scrub_stamp":"2024-09-10T19:11:22.870369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:11:22.870369+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0017468729999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":42,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-09-10T19:14:09.034646+0000","last_change":"2024-09-10T19:13:54.708605+0000","last_active":"2024-09-10T19:14:09.034646+0000","last_peered":"2024-09-10T19:14:09.034646+0000","last_clean":"2024-09-10T19:14:09.034646+0000","last_became_active":"2024-09-10T19:13:54.708351+0000","last_became_peered":"2024-09-10T19:13:54.708351+0000","last_unstale":"2024-09-10T19:14:09.034646+0000","last_undegraded":"2024-09-10T19:14:09.034646+0000","last_fullsized":"2024-09-10T19:14:09.034646+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:11:44.254513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:11:44.254513+0000","last_clean_scrub_stamp":"2024-09-10T19:11:44.254513+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":88,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-09-10T19:14:09.038084+0000","last_change":"2024-09-10T19:13:08.398177+0000","last_active":"2024-09-10T19:14:09.038084+0000","last_peered":"2024-09-10T19:14:09.038084+0000","last_clean":"2024-09-10T19:14:09.038084+0000","last_became_active":"2024-09-10T19:10:11.694405+0000","last_became_peered":"2024-09-10T19:10:11.694405+0000","last_unstale":"2024-09-10T19:14:09.038084+0000","last_undegraded":"2024-09-10T19:14:09.038084+0000","last_fullsized":"2024-09-10T19:14:09.038084+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:13:08.398036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:13:08.398036+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:14:34.880713+0000","scrub_duration":0.016845473999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":86,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-09-10T19:14:09.046665+0000","last_change":"2024-09-10T19:12:56.145358+0000","last_active":"2024-09-10T19:14:09.046665+0000","last_peered":"2024-09-10T19:14:09.046665+0000","last_clean":"2024-09-10T19:14:09.046665+0000","last_became_active":"2024-09-10T19:10:13.714952+0000","last_became_peered":"2024-09-10T19:10:13.714952+0000","last_unstale":"2024-09-10T19:14:09.046665+0000","last_undegraded":"2024-09-10T19:14:09.046665+0000","last_fullsized":"2024-09-10T19:14:09.046665+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:12:56.145218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:12:56.145218+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:14:21.830535+0000","scrub_duration":0.0058765429999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.b","version":"122'10851","reported_seq":17360,"reported_epoch":122,"state":"active+undersized+degraded+remapped+backfilling","last_fresh":"2024-09-10T19:14:19.644024+0000","last_change":"2024-09-10T19:13:55.833967+0000","last_active":"2024-09-10T19:14:19.644024+0000","last_peered":"2024-09-10T19:14:19.644024+0000","last_clean":"2024-09-10T19:12:45.807438+0000","last_became_active":"2024-09-10T19:13:55.804993+0000","last_became_peered":"2024-09-10T19:13:55.804993+0000","last_unstale":"2024-09-10T19:14:19.644024+0000","last_undegraded":"2024-09-10T19:13:55.727192+0000","last_fullsized":"2024-09-10T19:13:55.710998+0000","mapping_epoch":116,"log_start":"105'5841","ondisk_log_start":"105'5841","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:12:10.007465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:12:10.007465+0000","last_clean_scrub_stamp":"2024-09-10T19:12:10.007465+0000","objects_scrubbed":0,"log_size":5010,"log_dups_size":0,"ondisk_log_size":5010,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":435617792,"num_objects":6647,"num_object_clones":0,"num_object_copies":19941,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3805,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6647,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10851,"num_write_kb":559936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3224,"num_bytes_recovered":211288064,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6],"avail_no_missing":["5","6"],"object_location_counts":[{"shards":"5,6","objects":6647}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.8","version":"66'15","reported_seq":115,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-09-10T19:14:09.043008+0000","last_change":"2024-09-10T19:12:50.449684+0000","last_active":"2024-09-10T19:14:09.043008+0000","last_peered":"2024-09-10T19:14:09.043008+0000","last_clean":"2024-09-10T19:14:09.043008+0000","last_became_active":"2024-09-10T19:10:09.820950+0000","last_became_peered":"2024-09-10T19:10:09.820950+0000","last_unstale":"2024-09-10T19:14:09.043008+0000","last_undegraded":"2024-09-10T19:14:09.043008+0000","last_fullsized":"2024-09-10T19:14:09.043008+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'15","last_scrub_stamp":"2024-09-10T19:12:50.449596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:12:50.449596+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:14:18.973373+0000","scrub_duration":0.0055729239999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":92,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-09-10T19:14:09.039306+0000","last_change":"2024-09-10T19:12:48.346908+0000","last_active":"2024-09-10T19:14:09.039306+0000","last_peered":"2024-09-10T19:14:09.039306+0000","last_clean":"2024-09-10T19:14:09.039306+0000","last_became_active":"2024-09-10T19:10:07.682286+0000","last_became_peered":"2024-09-10T19:10:07.682286+0000","last_unstale":"2024-09-10T19:14:09.039306+0000","last_undegraded":"2024-09-10T19:14:09.039306+0000","last_fullsized":"2024-09-10T19:14:09.039306+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:12:48.346798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:12:48.346798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.00606165,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.9","version":"80'9","reported_seq":147,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-09-10T19:14:15.795828+0000","last_change":"2024-09-10T19:13:43.358984+0000","last_active":"2024-09-10T19:14:15.795828+0000","last_peered":"2024-09-10T19:14:15.795828+0000","last_clean":"2024-09-10T19:14:15.795828+0000","last_became_active":"2024-09-10T19:10:11.702878+0000","last_became_peered":"2024-09-10T19:10:11.702878+0000","last_unstale":"2024-09-10T19:14:15.795828+0000","last_undegraded":"2024-09-10T19:14:15.795828+0000","last_fullsized":"2024-09-10T19:14:15.795828+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"80'9","last_scrub_stamp":"2024-09-10T19:13:43.358901+0000","last_deep_scrub":"80'9","last_deep_scrub_stamp":"2024-09-10T19:11:35.174218+0000","last_clean_scrub_stamp":"2024-09-10T19:13:43.358901+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:15:04.578959+0000","scrub_duration":0.0066143260000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":86,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-09-10T19:14:09.041722+0000","last_change":"2024-09-10T19:12:57.392898+0000","last_active":"2024-09-10T19:14:09.041722+0000","last_peered":"2024-09-10T19:14:09.041722+0000","last_clean":"2024-09-10T19:14:09.041722+0000","last_became_active":"2024-09-10T19:10:14.053875+0000","last_became_peered":"2024-09-10T19:10:14.053875+0000","last_unstale":"2024-09-10T19:14:09.041722+0000","last_undegraded":"2024-09-10T19:14:09.041722+0000","last_fullsized":"2024-09-10T19:14:09.041722+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:12:57.392844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:12:57.392844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0046180839999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.a","version":"122'11098","reported_seq":11487,"reported_epoch":122,"state":"active+clean","last_fresh":"2024-09-10T19:14:19.763342+0000","last_change":"2024-09-10T19:13:44.230819+0000","last_active":"2024-09-10T19:14:19.763342+0000","last_peered":"2024-09-10T19:14:19.763342+0000","last_clean":"2024-09-10T19:14:19.763342+0000","last_became_active":"2024-09-10T19:12:11.034838+0000","last_became_peered":"2024-09-10T19:12:11.034838+0000","last_unstale":"2024-09-10T19:14:19.763342+0000","last_undegraded":"2024-09-10T19:14:19.763342+0000","last_fullsized":"2024-09-10T19:14:19.763342+0000","mapping_epoch":86,"log_start":"107'6819","ondisk_log_start":"107'6819","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"112'8534","last_scrub_stamp":"2024-09-10T19:13:44.230739+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:12:10.007465+0000","last_clean_scrub_stamp":"2024-09-10T19:13:44.230739+0000","objects_scrubbed":8364,"log_size":4279,"log_dups_size":0,"ondisk_log_size":4279,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:15:06.184625+0000","scrub_duration":4.047209821,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":432013312,"num_objects":6592,"num_object_clones":0,"num_object_copies":19776,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6592,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11098,"num_write_kb":566080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.9","version":"66'12","reported_seq":131,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-09-10T19:14:09.033438+0000","last_change":"2024-09-10T19:13:54.760391+0000","last_active":"2024-09-10T19:14:09.033438+0000","last_peered":"2024-09-10T19:14:09.033438+0000","last_clean":"2024-09-10T19:14:09.033438+0000","last_became_active":"2024-09-10T19:13:54.718152+0000","last_became_peered":"2024-09-10T19:13:54.718152+0000","last_unstale":"2024-09-10T19:14:09.033438+0000","last_undegraded":"2024-09-10T19:14:09.033438+0000","last_fullsized":"2024-09-10T19:14:09.033438+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'12","last_scrub_stamp":"2024-09-10T19:11:45.012377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:11:45.012377+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0019105979999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.f","version":"59'2","reported_seq":107,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-09-10T19:14:09.034503+0000","last_change":"2024-09-10T19:12:50.437332+0000","last_active":"2024-09-10T19:14:09.034503+0000","last_peered":"2024-09-10T19:14:09.034503+0000","last_clean":"2024-09-10T19:14:09.034503+0000","last_became_active":"2024-09-10T19:10:07.681719+0000","last_became_peered":"2024-09-10T19:10:07.681719+0000","last_unstale":"2024-09-10T19:14:09.034503+0000","last_undegraded":"2024-09-10T19:14:09.034503+0000","last_fullsized":"2024-09-10T19:14:09.034503+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'2","last_scrub_stamp":"2024-09-10T19:12:50.437245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:12:50.437245+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0070075770000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":14,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":95,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-09-10T19:14:09.037604+0000","last_change":"2024-09-10T19:13:54.715893+0000","last_active":"2024-09-10T19:14:09.037604+0000","last_peered":"2024-09-10T19:14:09.037604+0000","last_clean":"2024-09-10T19:14:09.037604+0000","last_became_active":"2024-09-10T19:13:54.715528+0000","last_became_peered":"2024-09-10T19:13:54.715528+0000","last_unstale":"2024-09-10T19:14:09.037604+0000","last_undegraded":"2024-09-10T19:14:09.037604+0000","last_fullsized":"2024-09-10T19:14:09.037604+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:11:46.587846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:11:46.587846+0000","last_clean_scrub_stamp":"2024-09-10T19:11:46.587846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001267142,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":93,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-09-10T19:14:09.035336+0000","last_change":"2024-09-10T19:13:54.712104+0000","last_active":"2024-09-10T19:14:09.035336+0000","last_peered":"2024-09-10T19:14:09.035336+0000","last_clean":"2024-09-10T19:14:09.035336+0000","last_became_active":"2024-09-10T19:13:54.711804+0000","last_became_peered":"2024-09-10T19:13:54.711804+0000","last_unstale":"2024-09-10T19:14:09.035336+0000","last_undegraded":"2024-09-10T19:14:09.035336+0000","last_fullsized":"2024-09-10T19:14:09.035336+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:11:42.723437+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:11:42.723437+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001385631,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.16","version":"66'9","reported_seq":116,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-09-10T19:14:09.040293+0000","last_change":"2024-09-10T19:13:50.172901+0000","last_active":"2024-09-10T19:14:09.040293+0000","last_peered":"2024-09-10T19:14:09.040293+0000","last_clean":"2024-09-10T19:14:09.040293+0000","last_became_active":"2024-09-10T19:10:09.822994+0000","last_became_peered":"2024-09-10T19:10:09.822994+0000","last_unstale":"2024-09-10T19:14:09.040293+0000","last_undegraded":"2024-09-10T19:14:09.040293+0000","last_fullsized":"2024-09-10T19:14:09.040293+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'9","last_scrub_stamp":"2024-09-10T19:13:50.172825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:13:50.172825+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:14:55.898340+0000","scrub_duration":0.0094381619999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":99,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-09-10T19:14:09.037804+0000","last_change":"2024-09-10T19:13:54.716196+0000","last_active":"2024-09-10T19:14:09.037804+0000","last_peered":"2024-09-10T19:14:09.037804+0000","last_clean":"2024-09-10T19:14:09.037804+0000","last_became_active":"2024-09-10T19:13:54.715910+0000","last_became_peered":"2024-09-10T19:13:54.715910+0000","last_unstale":"2024-09-10T19:14:09.037804+0000","last_undegraded":"2024-09-10T19:14:09.037804+0000","last_fullsized":"2024-09-10T19:14:09.037804+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:11:41.458523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:11:41.458523+0000","last_clean_scrub_stamp":"2024-09-10T19:11:41.458523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0012499830000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,4],"acting":[2,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":95,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-09-10T19:14:09.041353+0000","last_change":"2024-09-10T19:13:54.712640+0000","last_active":"2024-09-10T19:14:09.041353+0000","last_peered":"2024-09-10T19:14:09.041353+0000","last_clean":"2024-09-10T19:14:09.041353+0000","last_became_active":"2024-09-10T19:13:54.712275+0000","last_became_peered":"2024-09-10T19:13:54.712275+0000","last_unstale":"2024-09-10T19:14:09.041353+0000","last_undegraded":"2024-09-10T19:14:09.041353+0000","last_fullsized":"2024-09-10T19:14:09.041353+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:12:11.619202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:12:11.619202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001950834,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"66'1","reported_seq":87,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-09-10T19:14:09.037775+0000","last_change":"2024-09-10T19:12:54.361402+0000","last_active":"2024-09-10T19:14:09.037775+0000","last_peered":"2024-09-10T19:14:09.037775+0000","last_clean":"2024-09-10T19:14:09.037775+0000","last_became_active":"2024-09-10T19:10:13.717001+0000","last_became_peered":"2024-09-10T19:10:13.717001+0000","last_unstale":"2024-09-10T19:14:09.037775+0000","last_undegraded":"2024-09-10T19:14:09.037775+0000","last_fullsized":"2024-09-10T19:14:09.037775+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'1","last_scrub_stamp":"2024-09-10T19:12:54.361312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:12:54.361312+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0039806390000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":99,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-09-10T19:14:09.048066+0000","last_change":"2024-09-10T19:13:54.724442+0000","last_active":"2024-09-10T19:14:09.048066+0000","last_peered":"2024-09-10T19:14:09.048066+0000","last_clean":"2024-09-10T19:14:09.048066+0000","last_became_active":"2024-09-10T19:13:54.724027+0000","last_became_peered":"2024-09-10T19:13:54.724027+0000","last_unstale":"2024-09-10T19:14:09.048066+0000","last_undegraded":"2024-09-10T19:14:09.048066+0000","last_fullsized":"2024-09-10T19:14:09.048066+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:11:32.859106+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:11:32.859106+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0014403980000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,5],"acting":[6,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"66'6","reported_seq":114,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-09-10T19:14:09.041433+0000","last_change":"2024-09-10T19:13:55.370749+0000","last_active":"2024-09-10T19:14:09.041433+0000","last_peered":"2024-09-10T19:14:09.041433+0000","last_clean":"2024-09-10T19:14:09.041433+0000","last_became_active":"2024-09-10T19:13:54.717949+0000","last_became_peered":"2024-09-10T19:13:54.717949+0000","last_unstale":"2024-09-10T19:14:09.041433+0000","last_undegraded":"2024-09-10T19:14:09.041433+0000","last_fullsized":"2024-09-10T19:14:09.041433+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'6","last_scrub_stamp":"2024-09-10T19:13:55.370650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:13:55.370650+0000","objects_scrubbed":3,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:15:17.175995+0000","scrub_duration":0.071693797000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":95,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-09-10T19:14:09.045966+0000","last_change":"2024-09-10T19:13:54.730473+0000","last_active":"2024-09-10T19:14:09.045966+0000","last_peered":"2024-09-10T19:14:09.045966+0000","last_clean":"2024-09-10T19:14:09.045966+0000","last_became_active":"2024-09-10T19:13:54.729917+0000","last_became_peered":"2024-09-10T19:13:54.729917+0000","last_unstale":"2024-09-10T19:14:09.045966+0000","last_undegraded":"2024-09-10T19:14:09.045966+0000","last_fullsized":"2024-09-10T19:14:09.045966+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:11:23.264793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:11:23.264793+0000","last_clean_scrub_stamp":"2024-09-10T19:11:23.264793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0015950649999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":86,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-09-10T19:14:09.041195+0000","last_change":"2024-09-10T19:12:51.508398+0000","last_active":"2024-09-10T19:14:09.041195+0000","last_peered":"2024-09-10T19:14:09.041195+0000","last_clean":"2024-09-10T19:14:09.041195+0000","last_became_active":"2024-09-10T19:10:14.054360+0000","last_became_peered":"2024-09-10T19:10:14.054360+0000","last_unstale":"2024-09-10T19:14:09.041195+0000","last_undegraded":"2024-09-10T19:14:09.041195+0000","last_fullsized":"2024-09-10T19:14:09.041195+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:12:51.508263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:12:51.508263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:14:18.164771+0000","scrub_duration":0.001748274,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"80'15","reported_seq":129,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-09-10T19:14:09.040346+0000","last_change":"2024-09-10T19:13:54.749388+0000","last_active":"2024-09-10T19:14:09.040346+0000","last_peered":"2024-09-10T19:14:09.040346+0000","last_clean":"2024-09-10T19:14:09.040346+0000","last_became_active":"2024-09-10T19:13:54.719574+0000","last_became_peered":"2024-09-10T19:13:54.719574+0000","last_unstale":"2024-09-10T19:14:09.040346+0000","last_undegraded":"2024-09-10T19:14:09.040346+0000","last_fullsized":"2024-09-10T19:14:09.040346+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"80'15","last_scrub_stamp":"2024-09-10T19:11:41.719310+0000","last_deep_scrub":"80'15","last_deep_scrub_stamp":"2024-09-10T19:11:41.719310+0000","last_clean_scrub_stamp":"2024-09-10T19:11:41.719310+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0046706430000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":92,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-09-10T19:14:09.042141+0000","last_change":"2024-09-10T19:12:46.506129+0000","last_active":"2024-09-10T19:14:09.042141+0000","last_peered":"2024-09-10T19:14:09.042141+0000","last_clean":"2024-09-10T19:14:09.042141+0000","last_became_active":"2024-09-10T19:10:07.685870+0000","last_became_peered":"2024-09-10T19:10:07.685870+0000","last_unstale":"2024-09-10T19:14:09.042141+0000","last_undegraded":"2024-09-10T19:14:09.042141+0000","last_fullsized":"2024-09-10T19:14:09.042141+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:12:46.506071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:12:46.506071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001257935,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"117'10","reported_seq":158,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-09-10T19:14:15.795618+0000","last_change":"2024-09-10T19:13:54.755445+0000","last_active":"2024-09-10T19:14:15.795618+0000","last_peered":"2024-09-10T19:14:15.795618+0000","last_clean":"2024-09-10T19:14:15.795618+0000","last_became_active":"2024-09-10T19:13:54.724780+0000","last_became_peered":"2024-09-10T19:13:54.724780+0000","last_unstale":"2024-09-10T19:14:15.795618+0000","last_undegraded":"2024-09-10T19:14:15.795618+0000","last_fullsized":"2024-09-10T19:14:15.795618+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'8","last_scrub_stamp":"2024-09-10T19:11:50.330303+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:11:50.330303+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0020032069999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3],"acting":[5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":86,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-09-10T19:14:09.039784+0000","last_change":"2024-09-10T19:13:00.212150+0000","last_active":"2024-09-10T19:14:09.039784+0000","last_peered":"2024-09-10T19:14:09.039784+0000","last_clean":"2024-09-10T19:14:09.039784+0000","last_became_active":"2024-09-10T19:10:13.721035+0000","last_became_peered":"2024-09-10T19:10:13.721035+0000","last_unstale":"2024-09-10T19:14:09.039784+0000","last_undegraded":"2024-09-10T19:14:09.039784+0000","last_fullsized":"2024-09-10T19:14:09.039784+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:13:00.212049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:13:00.212049+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:14:30.099769+0000","scrub_duration":0.015146817999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"66'9","reported_seq":106,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-09-10T19:14:09.043004+0000","last_change":"2024-09-10T19:12:52.416838+0000","last_active":"2024-09-10T19:14:09.043004+0000","last_peered":"2024-09-10T19:14:09.043004+0000","last_clean":"2024-09-10T19:14:09.043004+0000","last_became_active":"2024-09-10T19:10:09.823006+0000","last_became_peered":"2024-09-10T19:10:09.823006+0000","last_unstale":"2024-09-10T19:14:09.043004+0000","last_undegraded":"2024-09-10T19:14:09.043004+0000","last_fullsized":"2024-09-10T19:14:09.043004+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'9","last_scrub_stamp":"2024-09-10T19:12:52.416748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:12:52.416748+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0047220839999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":92,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-09-10T19:14:09.039908+0000","last_change":"2024-09-10T19:13:01.210932+0000","last_active":"2024-09-10T19:14:09.039908+0000","last_peered":"2024-09-10T19:14:09.039908+0000","last_clean":"2024-09-10T19:14:09.039908+0000","last_became_active":"2024-09-10T19:10:07.688499+0000","last_became_peered":"2024-09-10T19:10:07.688499+0000","last_unstale":"2024-09-10T19:14:09.039908+0000","last_undegraded":"2024-09-10T19:14:09.039908+0000","last_fullsized":"2024-09-10T19:14:09.039908+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:13:01.210848+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:13:01.210848+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:14:17.716737+0000","scrub_duration":0.0049627250000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"66'8","reported_seq":146,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-09-10T19:14:15.795759+0000","last_change":"2024-09-10T19:12:59.723961+0000","last_active":"2024-09-10T19:14:15.795759+0000","last_peered":"2024-09-10T19:14:15.795759+0000","last_clean":"2024-09-10T19:14:15.795759+0000","last_became_active":"2024-09-10T19:10:11.692359+0000","last_became_peered":"2024-09-10T19:10:11.692359+0000","last_unstale":"2024-09-10T19:14:15.795759+0000","last_undegraded":"2024-09-10T19:14:15.795759+0000","last_fullsized":"2024-09-10T19:14:15.795759+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'8","last_scrub_stamp":"2024-09-10T19:12:59.723712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:12:59.723712+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0083838109999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":85,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-09-10T19:14:09.034238+0000","last_change":"2024-09-10T19:13:09.463096+0000","last_active":"2024-09-10T19:14:09.034238+0000","last_peered":"2024-09-10T19:14:09.034238+0000","last_clean":"2024-09-10T19:14:09.034238+0000","last_became_active":"2024-09-10T19:10:13.718285+0000","last_became_peered":"2024-09-10T19:10:13.718285+0000","last_unstale":"2024-09-10T19:14:09.034238+0000","last_undegraded":"2024-09-10T19:14:09.034238+0000","last_fullsized":"2024-09-10T19:14:09.034238+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:13:09.463001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:13:09.463001+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:14:23.918770+0000","scrub_duration":0.012199573,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.12","version":"79'13","reported_seq":99,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-09-10T19:14:09.046898+0000","last_change":"2024-09-10T19:13:54.758707+0000","last_active":"2024-09-10T19:14:09.046898+0000","last_peered":"2024-09-10T19:14:09.046898+0000","last_clean":"2024-09-10T19:14:09.046898+0000","last_became_active":"2024-09-10T19:13:54.721560+0000","last_became_peered":"2024-09-10T19:13:54.721560+0000","last_unstale":"2024-09-10T19:14:09.046898+0000","last_undegraded":"2024-09-10T19:14:09.046898+0000","last_fullsized":"2024-09-10T19:14:09.046898+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'13","last_scrub_stamp":"2024-09-10T19:11:22.072400+0000","last_deep_scrub":"79'13","last_deep_scrub_stamp":"2024-09-10T19:11:22.072400+0000","last_clean_scrub_stamp":"2024-09-10T19:11:22.072400+0000","objects_scrubbed":4,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,0],"acting":[6,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.14","version":"59'1","reported_seq":99,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-09-10T19:14:09.047584+0000","last_change":"2024-09-10T19:13:08.917822+0000","last_active":"2024-09-10T19:14:09.047584+0000","last_peered":"2024-09-10T19:14:09.047584+0000","last_clean":"2024-09-10T19:14:09.047584+0000","last_became_active":"2024-09-10T19:10:07.683263+0000","last_became_peered":"2024-09-10T19:10:07.683263+0000","last_unstale":"2024-09-10T19:14:09.047584+0000","last_undegraded":"2024-09-10T19:14:09.047584+0000","last_fullsized":"2024-09-10T19:14:09.047584+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'1","last_scrub_stamp":"2024-09-10T19:13:08.917719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:13:08.917719+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:14:36.333049+0000","scrub_duration":0.0033669569999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":362,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":95,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-09-10T19:14:09.035427+0000","last_change":"2024-09-10T19:13:54.730422+0000","last_active":"2024-09-10T19:14:09.035427+0000","last_peered":"2024-09-10T19:14:09.035427+0000","last_clean":"2024-09-10T19:14:09.035427+0000","last_became_active":"2024-09-10T19:13:54.730185+0000","last_became_peered":"2024-09-10T19:13:54.730185+0000","last_unstale":"2024-09-10T19:14:09.035427+0000","last_undegraded":"2024-09-10T19:14:09.035427+0000","last_fullsized":"2024-09-10T19:14:09.035427+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:12:00.660284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:12:00.660284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0012510729999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":93,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-09-10T19:14:09.038787+0000","last_change":"2024-09-10T19:13:54.715762+0000","last_active":"2024-09-10T19:14:09.038787+0000","last_peered":"2024-09-10T19:14:09.038787+0000","last_clean":"2024-09-10T19:14:09.038787+0000","last_became_active":"2024-09-10T19:13:54.714861+0000","last_became_peered":"2024-09-10T19:13:54.714861+0000","last_unstale":"2024-09-10T19:14:09.038787+0000","last_undegraded":"2024-09-10T19:14:09.038787+0000","last_fullsized":"2024-09-10T19:14:09.038787+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:11:51.920006+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:11:51.920006+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001865058,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,6],"acting":[0,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"66'11","reported_seq":127,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-09-10T19:14:09.033588+0000","last_change":"2024-09-10T19:13:54.758776+0000","last_active":"2024-09-10T19:14:09.033588+0000","last_peered":"2024-09-10T19:14:09.033588+0000","last_clean":"2024-09-10T19:14:09.033588+0000","last_became_active":"2024-09-10T19:13:54.720034+0000","last_became_peered":"2024-09-10T19:13:54.720034+0000","last_unstale":"2024-09-10T19:14:09.033588+0000","last_undegraded":"2024-09-10T19:14:09.033588+0000","last_fullsized":"2024-09-10T19:14:09.033588+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'11","last_scrub_stamp":"2024-09-10T19:11:44.025361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:11:44.025361+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.002036894,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":42,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-09-10T19:14:09.043584+0000","last_change":"2024-09-10T19:13:54.723475+0000","last_active":"2024-09-10T19:14:09.043584+0000","last_peered":"2024-09-10T19:14:09.043584+0000","last_clean":"2024-09-10T19:14:09.043584+0000","last_became_active":"2024-09-10T19:13:54.722947+0000","last_became_peered":"2024-09-10T19:13:54.722947+0000","last_unstale":"2024-09-10T19:14:09.043584+0000","last_undegraded":"2024-09-10T19:14:09.043584+0000","last_fullsized":"2024-09-10T19:14:09.043584+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:12:44.249765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:11:10.031245+0000","last_clean_scrub_stamp":"2024-09-10T19:12:44.249765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,6],"acting":[5,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":42,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-09-10T19:14:09.041867+0000","last_change":"2024-09-10T19:13:54.730370+0000","last_active":"2024-09-10T19:14:09.041867+0000","last_peered":"2024-09-10T19:14:09.041867+0000","last_clean":"2024-09-10T19:14:09.041867+0000","last_became_active":"2024-09-10T19:13:54.729703+0000","last_became_peered":"2024-09-10T19:13:54.729703+0000","last_unstale":"2024-09-10T19:14:09.041867+0000","last_undegraded":"2024-09-10T19:14:09.041867+0000","last_fullsized":"2024-09-10T19:14:09.041867+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:11:31.249343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:11:31.249343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":86,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-09-10T19:14:09.034535+0000","last_change":"2024-09-10T19:12:54.691886+0000","last_active":"2024-09-10T19:14:09.034535+0000","last_peered":"2024-09-10T19:14:09.034535+0000","last_clean":"2024-09-10T19:14:09.034535+0000","last_became_active":"2024-09-10T19:10:13.724201+0000","last_became_peered":"2024-09-10T19:10:13.724201+0000","last_unstale":"2024-09-10T19:14:09.034535+0000","last_undegraded":"2024-09-10T19:14:09.034535+0000","last_fullsized":"2024-09-10T19:14:09.034535+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:12:54.691810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:12:54.691810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:14:17.760354+0000","scrub_duration":0.006702927,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"66'4","reported_seq":109,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-09-10T19:14:09.036108+0000","last_change":"2024-09-10T19:13:54.732227+0000","last_active":"2024-09-10T19:14:09.036108+0000","last_peered":"2024-09-10T19:14:09.036108+0000","last_clean":"2024-09-10T19:14:09.036108+0000","last_became_active":"2024-09-10T19:13:54.711125+0000","last_became_peered":"2024-09-10T19:13:54.711125+0000","last_unstale":"2024-09-10T19:14:09.036108+0000","last_undegraded":"2024-09-10T19:14:09.036108+0000","last_fullsized":"2024-09-10T19:14:09.036108+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'4","last_scrub_stamp":"2024-09-10T19:11:14.816632+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:11:14.816632+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001902504,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,4],"acting":[3,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":92,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-09-10T19:14:09.046069+0000","last_change":"2024-09-10T19:12:48.517928+0000","last_active":"2024-09-10T19:14:09.046069+0000","last_peered":"2024-09-10T19:14:09.046069+0000","last_clean":"2024-09-10T19:14:09.046069+0000","last_became_active":"2024-09-10T19:10:07.687723+0000","last_became_peered":"2024-09-10T19:10:07.687723+0000","last_unstale":"2024-09-10T19:14:09.046069+0000","last_undegraded":"2024-09-10T19:14:09.046069+0000","last_fullsized":"2024-09-10T19:14:09.046069+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:12:48.517853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:12:48.517853+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001753459,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":88,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-09-10T19:14:09.047882+0000","last_change":"2024-09-10T19:13:04.855026+0000","last_active":"2024-09-10T19:14:09.047882+0000","last_peered":"2024-09-10T19:14:09.047882+0000","last_clean":"2024-09-10T19:14:09.047882+0000","last_became_active":"2024-09-10T19:10:11.698536+0000","last_became_peered":"2024-09-10T19:10:11.698536+0000","last_unstale":"2024-09-10T19:14:09.047882+0000","last_undegraded":"2024-09-10T19:14:09.047882+0000","last_fullsized":"2024-09-10T19:14:09.047882+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:13:04.854929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:11:31.837944+0000","last_clean_scrub_stamp":"2024-09-10T19:13:04.854929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:14:32.441023+0000","scrub_duration":0.0012499329999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":86,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-09-10T19:14:09.042160+0000","last_change":"2024-09-10T19:12:45.536856+0000","last_active":"2024-09-10T19:14:09.042160+0000","last_peered":"2024-09-10T19:14:09.042160+0000","last_clean":"2024-09-10T19:14:09.042160+0000","last_became_active":"2024-09-10T19:10:13.718271+0000","last_became_peered":"2024-09-10T19:10:13.718271+0000","last_unstale":"2024-09-10T19:14:09.042160+0000","last_undegraded":"2024-09-10T19:14:09.042160+0000","last_fullsized":"2024-09-10T19:14:09.042160+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:12:45.536799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:12:45.536799+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0036938909999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.11","version":"66'11","reported_seq":109,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-09-10T19:14:09.034726+0000","last_change":"2024-09-10T19:13:05.730096+0000","last_active":"2024-09-10T19:14:09.034726+0000","last_peered":"2024-09-10T19:14:09.034726+0000","last_clean":"2024-09-10T19:14:09.034726+0000","last_became_active":"2024-09-10T19:10:09.821357+0000","last_became_peered":"2024-09-10T19:10:09.821357+0000","last_unstale":"2024-09-10T19:14:09.034726+0000","last_undegraded":"2024-09-10T19:14:09.034726+0000","last_fullsized":"2024-09-10T19:14:09.034726+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'11","last_scrub_stamp":"2024-09-10T19:13:05.729854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:13:05.729854+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:14:32.355799+0000","scrub_duration":0.010667395,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":92,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-09-10T19:14:09.048022+0000","last_change":"2024-09-10T19:13:05.876708+0000","last_active":"2024-09-10T19:14:09.048022+0000","last_peered":"2024-09-10T19:14:09.048022+0000","last_clean":"2024-09-10T19:14:09.048022+0000","last_became_active":"2024-09-10T19:10:07.685223+0000","last_became_peered":"2024-09-10T19:10:07.685223+0000","last_unstale":"2024-09-10T19:14:09.048022+0000","last_undegraded":"2024-09-10T19:14:09.048022+0000","last_fullsized":"2024-09-10T19:14:09.048022+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:13:05.876655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:13:05.876655+0000","last_clean_scrub_stamp":"2024-09-10T19:13:05.876655+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0013242589999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":88,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-09-10T19:14:09.041303+0000","last_change":"2024-09-10T19:13:41.326028+0000","last_active":"2024-09-10T19:14:09.041303+0000","last_peered":"2024-09-10T19:14:09.041303+0000","last_clean":"2024-09-10T19:14:09.041303+0000","last_became_active":"2024-09-10T19:10:11.699013+0000","last_became_peered":"2024-09-10T19:10:11.699013+0000","last_unstale":"2024-09-10T19:14:09.041303+0000","last_undegraded":"2024-09-10T19:14:09.041303+0000","last_fullsized":"2024-09-10T19:14:09.041303+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:13:41.325941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:11:57.112092+0000","last_clean_scrub_stamp":"2024-09-10T19:13:41.325941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:15:05.192727+0000","scrub_duration":0.001598676,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":86,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-09-10T19:14:09.035260+0000","last_change":"2024-09-10T19:13:15.812326+0000","last_active":"2024-09-10T19:14:09.035260+0000","last_peered":"2024-09-10T19:14:09.035260+0000","last_clean":"2024-09-10T19:14:09.035260+0000","last_became_active":"2024-09-10T19:10:14.056172+0000","last_became_peered":"2024-09-10T19:10:14.056172+0000","last_unstale":"2024-09-10T19:14:09.035260+0000","last_undegraded":"2024-09-10T19:14:09.035260+0000","last_fullsized":"2024-09-10T19:14:09.035260+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:13:15.812244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:13:15.812244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:14:36.621092+0000","scrub_duration":0.0048044389999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"80'2","reported_seq":88,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-09-10T19:14:09.042221+0000","last_change":"2024-09-10T19:12:57.454088+0000","last_active":"2024-09-10T19:14:09.042221+0000","last_peered":"2024-09-10T19:14:09.042221+0000","last_clean":"2024-09-10T19:14:09.042221+0000","last_became_active":"2024-09-10T19:10:13.718875+0000","last_became_peered":"2024-09-10T19:10:13.718875+0000","last_unstale":"2024-09-10T19:14:09.042221+0000","last_undegraded":"2024-09-10T19:14:09.042221+0000","last_fullsized":"2024-09-10T19:14:09.042221+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"80'2","last_scrub_stamp":"2024-09-10T19:12:57.453993+0000","last_deep_scrub":"80'2","last_deep_scrub_stamp":"2024-09-10T19:11:30.272043+0000","last_clean_scrub_stamp":"2024-09-10T19:12:57.453993+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0018294990000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":369,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":92,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-09-10T19:14:09.043998+0000","last_change":"2024-09-10T19:13:14.449643+0000","last_active":"2024-09-10T19:14:09.043998+0000","last_peered":"2024-09-10T19:14:09.043998+0000","last_clean":"2024-09-10T19:14:09.043998+0000","last_became_active":"2024-09-10T19:10:07.689877+0000","last_became_peered":"2024-09-10T19:10:07.689877+0000","last_unstale":"2024-09-10T19:14:09.043998+0000","last_undegraded":"2024-09-10T19:14:09.043998+0000","last_fullsized":"2024-09-10T19:14:09.043998+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:13:14.449593+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:13:14.449593+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:14:27.688141+0000","scrub_duration":0.001195542,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1e","version":"80'15","reported_seq":110,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-09-10T19:14:09.040217+0000","last_change":"2024-09-10T19:12:50.148157+0000","last_active":"2024-09-10T19:14:09.040217+0000","last_peered":"2024-09-10T19:14:09.040217+0000","last_clean":"2024-09-10T19:14:09.040217+0000","last_became_active":"2024-09-10T19:10:09.823061+0000","last_became_peered":"2024-09-10T19:10:09.823061+0000","last_unstale":"2024-09-10T19:14:09.040217+0000","last_undegraded":"2024-09-10T19:14:09.040217+0000","last_fullsized":"2024-09-10T19:14:09.040217+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"80'15","last_scrub_stamp":"2024-09-10T19:12:50.147336+0000","last_deep_scrub":"80'15","last_deep_scrub_stamp":"2024-09-10T19:11:25.883919+0000","last_clean_scrub_stamp":"2024-09-10T19:12:50.147336+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0064585019999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1f","version":"66'8","reported_seq":149,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-09-10T19:14:15.795539+0000","last_change":"2024-09-10T19:12:48.889499+0000","last_active":"2024-09-10T19:14:15.795539+0000","last_peered":"2024-09-10T19:14:15.795539+0000","last_clean":"2024-09-10T19:14:15.795539+0000","last_became_active":"2024-09-10T19:10:11.695427+0000","last_became_peered":"2024-09-10T19:10:11.695427+0000","last_unstale":"2024-09-10T19:14:15.795539+0000","last_undegraded":"2024-09-10T19:14:15.795539+0000","last_fullsized":"2024-09-10T19:14:15.795539+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'8","last_scrub_stamp":"2024-09-10T19:12:48.889410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:12:48.889410+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001725311,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":42,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-09-10T19:14:09.041634+0000","last_change":"2024-09-10T19:13:54.715498+0000","last_active":"2024-09-10T19:14:09.041634+0000","last_peered":"2024-09-10T19:14:09.041634+0000","last_clean":"2024-09-10T19:14:09.041634+0000","last_became_active":"2024-09-10T19:13:54.715192+0000","last_became_peered":"2024-09-10T19:13:54.715192+0000","last_unstale":"2024-09-10T19:14:09.041634+0000","last_undegraded":"2024-09-10T19:14:09.041634+0000","last_fullsized":"2024-09-10T19:14:09.041634+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:11:30.210603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:11:30.210603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,2],"acting":[5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":92,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-09-10T19:14:09.035035+0000","last_change":"2024-09-10T19:13:02.737323+0000","last_active":"2024-09-10T19:14:09.035035+0000","last_peered":"2024-09-10T19:14:09.035035+0000","last_clean":"2024-09-10T19:14:09.035035+0000","last_became_active":"2024-09-10T19:10:07.682689+0000","last_became_peered":"2024-09-10T19:10:07.682689+0000","last_unstale":"2024-09-10T19:14:09.035035+0000","last_undegraded":"2024-09-10T19:14:09.035035+0000","last_fullsized":"2024-09-10T19:14:09.035035+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:13:02.737236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:13:02.737236+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0027813289999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"80'16","reported_seq":158,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-09-10T19:14:09.048571+0000","last_change":"2024-09-10T19:13:54.761453+0000","last_active":"2024-09-10T19:14:09.048571+0000","last_peered":"2024-09-10T19:14:09.048571+0000","last_clean":"2024-09-10T19:14:09.048571+0000","last_became_active":"2024-09-10T19:13:54.713563+0000","last_became_peered":"2024-09-10T19:13:54.713563+0000","last_unstale":"2024-09-10T19:14:09.048571+0000","last_undegraded":"2024-09-10T19:14:09.048571+0000","last_fullsized":"2024-09-10T19:14:09.048571+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"80'16","last_scrub_stamp":"2024-09-10T19:12:51.915777+0000","last_deep_scrub":"80'16","last_deep_scrub_stamp":"2024-09-10T19:11:40.853611+0000","last_clean_scrub_stamp":"2024-09-10T19:12:51.915777+0000","objects_scrubbed":5,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0019885430000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,0],"acting":[6,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":88,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-09-10T19:14:09.040078+0000","last_change":"2024-09-10T19:12:59.195000+0000","last_active":"2024-09-10T19:14:09.040078+0000","last_peered":"2024-09-10T19:14:09.040078+0000","last_clean":"2024-09-10T19:14:09.040078+0000","last_became_active":"2024-09-10T19:10:11.701735+0000","last_became_peered":"2024-09-10T19:10:11.701735+0000","last_unstale":"2024-09-10T19:14:09.040078+0000","last_undegraded":"2024-09-10T19:14:09.040078+0000","last_fullsized":"2024-09-10T19:14:09.040078+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:12:59.194914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:11:28.822327+0000","last_clean_scrub_stamp":"2024-09-10T19:12:59.194914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:14:24.283020+0000","scrub_duration":0.0022130729999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":85,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-09-10T19:14:09.033818+0000","last_change":"2024-09-10T19:13:00.388592+0000","last_active":"2024-09-10T19:14:09.033818+0000","last_peered":"2024-09-10T19:14:09.033818+0000","last_clean":"2024-09-10T19:14:09.033818+0000","last_became_active":"2024-09-10T19:10:14.053619+0000","last_became_peered":"2024-09-10T19:10:14.053619+0000","last_unstale":"2024-09-10T19:14:09.033818+0000","last_undegraded":"2024-09-10T19:14:09.033818+0000","last_fullsized":"2024-09-10T19:14:09.033818+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:13:00.388466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:13:00.388466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.003120996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":92,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-09-10T19:14:09.047610+0000","last_change":"2024-09-10T19:13:43.205095+0000","last_active":"2024-09-10T19:14:09.047610+0000","last_peered":"2024-09-10T19:14:09.047610+0000","last_clean":"2024-09-10T19:14:09.047610+0000","last_became_active":"2024-09-10T19:10:07.685997+0000","last_became_peered":"2024-09-10T19:10:07.685997+0000","last_unstale":"2024-09-10T19:14:09.047610+0000","last_undegraded":"2024-09-10T19:14:09.047610+0000","last_fullsized":"2024-09-10T19:14:09.047610+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:13:43.205042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:13:43.205042+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:14:47.993594+0000","scrub_duration":0.0013181790000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1c","version":"78'22","reported_seq":155,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-09-10T19:14:09.038029+0000","last_change":"2024-09-10T19:13:54.751066+0000","last_active":"2024-09-10T19:14:09.038029+0000","last_peered":"2024-09-10T19:14:09.038029+0000","last_clean":"2024-09-10T19:14:09.038029+0000","last_became_active":"2024-09-10T19:13:54.715570+0000","last_became_peered":"2024-09-10T19:13:54.715570+0000","last_unstale":"2024-09-10T19:14:09.038029+0000","last_undegraded":"2024-09-10T19:14:09.038029+0000","last_fullsized":"2024-09-10T19:14:09.038029+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"78'22","last_scrub_stamp":"2024-09-10T19:12:45.399973+0000","last_deep_scrub":"78'22","last_deep_scrub_stamp":"2024-09-10T19:11:21.118937+0000","last_clean_scrub_stamp":"2024-09-10T19:12:45.399973+0000","objects_scrubbed":7,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0084828799999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":42,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-09-10T19:14:09.034347+0000","last_change":"2024-09-10T19:13:54.709308+0000","last_active":"2024-09-10T19:14:09.034347+0000","last_peered":"2024-09-10T19:14:09.034347+0000","last_clean":"2024-09-10T19:14:09.034347+0000","last_became_active":"2024-09-10T19:13:54.709055+0000","last_became_peered":"2024-09-10T19:13:54.709055+0000","last_unstale":"2024-09-10T19:14:09.034347+0000","last_undegraded":"2024-09-10T19:14:09.034347+0000","last_fullsized":"2024-09-10T19:14:09.034347+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:11:43.221323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:11:43.221323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,5],"acting":[4,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":86,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-09-10T19:14:09.041291+0000","last_change":"2024-09-10T19:12:51.767586+0000","last_active":"2024-09-10T19:14:09.041291+0000","last_peered":"2024-09-10T19:14:09.041291+0000","last_clean":"2024-09-10T19:14:09.041291+0000","last_became_active":"2024-09-10T19:10:14.053698+0000","last_became_peered":"2024-09-10T19:10:14.053698+0000","last_unstale":"2024-09-10T19:14:09.041291+0000","last_undegraded":"2024-09-10T19:14:09.041291+0000","last_fullsized":"2024-09-10T19:14:09.041291+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:12:51.767056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:12:51.767056+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:14:17.907092+0000","scrub_duration":0.006796519,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":92,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-09-10T19:14:09.035684+0000","last_change":"2024-09-10T19:12:50.719167+0000","last_active":"2024-09-10T19:14:09.035684+0000","last_peered":"2024-09-10T19:14:09.035684+0000","last_clean":"2024-09-10T19:14:09.035684+0000","last_became_active":"2024-09-10T19:10:07.682811+0000","last_became_peered":"2024-09-10T19:10:07.682811+0000","last_unstale":"2024-09-10T19:14:09.035684+0000","last_undegraded":"2024-09-10T19:14:09.035684+0000","last_fullsized":"2024-09-10T19:14:09.035684+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:12:50.718663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:12:50.718663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:14:18.173382+0000","scrub_duration":0.0042138399999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1d","version":"66'12","reported_seq":132,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-09-10T19:14:09.036020+0000","last_change":"2024-09-10T19:13:54.758541+0000","last_active":"2024-09-10T19:14:09.036020+0000","last_peered":"2024-09-10T19:14:09.036020+0000","last_clean":"2024-09-10T19:14:09.036020+0000","last_became_active":"2024-09-10T19:13:54.714629+0000","last_became_peered":"2024-09-10T19:13:54.714629+0000","last_unstale":"2024-09-10T19:14:09.036020+0000","last_undegraded":"2024-09-10T19:14:09.036020+0000","last_fullsized":"2024-09-10T19:14:09.036020+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'12","last_scrub_stamp":"2024-09-10T19:11:52.663657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:11:52.663657+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0019136909999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":87,"reported_epoch":121,"state":"active+clean","last_fresh":"2024-09-10T19:14:09.033723+0000","last_change":"2024-09-10T19:13:11.445426+0000","last_active":"2024-09-10T19:14:09.033723+0000","last_peered":"2024-09-10T19:14:09.033723+0000","last_clean":"2024-09-10T19:14:09.033723+0000","last_became_active":"2024-09-10T19:10:11.701155+0000","last_became_peered":"2024-09-10T19:10:11.701155+0000","last_unstale":"2024-09-10T19:14:09.033723+0000","last_undegraded":"2024-09-10T19:14:09.033723+0000","last_fullsized":"2024-09-10T19:14:09.033723+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:13:11.445238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:13:11.445238+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:14:41.165035+0000","scrub_duration":0.0057421110000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":16,"stat_sum":{"num_bytes":6867058712,"num_objects":104784,"num_object_clones":0,"num_object_copies":314352,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":31107,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":104784,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":175974,"num_write_kb":8984193,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9259,"num_bytes_recovered":606601216,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":19885400064,"data_stored":19885391920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76455,"ondisk_log_size":76455,"up":48,"acting":42,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":71,"ondisk_log_size":71,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":177,"num_object_clones":0,"num_object_copies":531,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":177,"num_whiteouts":0,"num_read":695,"num_read_kb":452,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":111,"num_bytes_recovered":2448,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":499712,"data_stored":13224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":444,"ondisk_log_size":444,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":95,"num_read_kb":90,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":389,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1613,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1328,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":27,"num_read_kb":27,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":3984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4,"ondisk_log_size":4,"up":96,"acting":96,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79,"ondisk_log_size":79,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":54,"seq":231928234039,"num_pgs":69,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2574820,"kb_used_data":2277464,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":91170332,"statfs":{"total":95995035648,"available":93358419968,"internally_reserved":0,"allocated":2332123136,"data_stored":2331758372,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 19:13:58 2024","interfaces":[{"interface":"back","average":{"1min":3.7909999999999999,"5min":1.619,"15min":0.89600000000000002},"min":{"1min":0.215,"5min":0.215,"15min":0.215},"max":{"1min":25.640999999999998,"5min":35.551000000000002,"15min":35.551000000000002},"last":0.83199999999999996},{"interface":"front","average":{"1min":3.0299999999999998,"5min":1.4099999999999999,"15min":0.82799999999999996},"min":{"1min":0.224,"5min":0.223,"15min":0.223},"max":{"1min":27.870999999999999,"5min":35.503999999999998,"15min":35.503999999999998},"last":2.464}]},{"osd":1,"last update":"Tue Sep 10 19:11:56 2024","interfaces":[{"interface":"back","average":{"1min":0.52400000000000002,"5min":0.52400000000000002,"15min":0.52400000000000002},"min":{"1min":0.36199999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.70199999999999996,"5min":0.79400000000000004,"15min":0.79400000000000004},"last":0.45600000000000002},{"interface":"front","average":{"1min":0.49199999999999999,"5min":0.51200000000000001,"15min":0.51500000000000001},"min":{"1min":0.28999999999999998,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.68500000000000005,"5min":0.71799999999999997,"15min":0.71799999999999997},"last":1.5}]},{"osd":2,"last update":"Tue Sep 10 19:13:58 2024","interfaces":[{"interface":"back","average":{"1min":2.6709999999999998,"5min":0.93799999999999994,"15min":0.66200000000000003},"min":{"1min":0.25600000000000001,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":31.940999999999999,"5min":31.940999999999999,"15min":31.940999999999999},"last":0.64800000000000002},{"interface":"front","average":{"1min":2.6989999999999998,"5min":1.014,"15min":0.70399999999999996},"min":{"1min":0.24099999999999999,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":25.056000000000001,"5min":25.056000000000001,"15min":25.056000000000001},"last":0.36899999999999999}]},{"osd":3,"last update":"Tue Sep 10 19:13:58 2024","interfaces":[{"interface":"back","average":{"1min":2.6509999999999998,"5min":1.5509999999999999,"15min":0.874},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":25.443999999999999,"5min":35.277999999999999,"15min":35.277999999999999},"last":0.377},{"interface":"front","average":{"1min":2.8879999999999999,"5min":1.595,"15min":0.878},"min":{"1min":0.30299999999999999,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":25.530000000000001,"5min":36.512999999999998,"15min":36.512999999999998},"last":0.34200000000000003}]},{"osd":4,"last update":"Tue Sep 10 19:13:58 2024","interfaces":[{"interface":"back","average":{"1min":0.59699999999999998,"5min":0.47199999999999998,"15min":0.46000000000000002},"min":{"1min":0.23499999999999999,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":2.863,"5min":2.863,"15min":2.863},"last":2.5249999999999999},{"interface":"front","average":{"1min":0.57599999999999996,"5min":0.47699999999999998,"15min":0.46500000000000002},"min":{"1min":0.30399999999999999,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":1.1020000000000001,"5min":1.1020000000000001,"15min":1.1020000000000001},"last":0.73999999999999999}]},{"osd":5,"last update":"Tue Sep 10 19:13:58 2024","interfaces":[{"interface":"back","average":{"1min":1.085,"5min":0.59299999999999997,"15min":0.51700000000000002},"min":{"1min":0.27700000000000002,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":7.2240000000000002,"5min":7.2240000000000002,"15min":7.2240000000000002},"last":0.30099999999999999},{"interface":"front","average":{"1min":0.626,"5min":0.51000000000000001,"15min":0.48699999999999999},"min":{"1min":0.26600000000000001,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":2.8980000000000001,"5min":2.8980000000000001,"15min":2.8980000000000001},"last":2.4980000000000002}]},{"osd":6,"last update":"Tue Sep 10 19:13:58 2024","interfaces":[{"interface":"back","average":{"1min":0.54600000000000004,"5min":0.51900000000000002,"15min":0.50600000000000001},"min":{"1min":0.29099999999999998,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.91400000000000003,"5min":0.91600000000000004,"15min":0.91600000000000004},"last":2.5550000000000002},{"interface":"front","average":{"1min":0.48899999999999999,"5min":0.48599999999999999,"15min":0.48799999999999999},"min":{"1min":0.28000000000000003,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.83499999999999996,"5min":0.83499999999999996,"15min":0.83499999999999996},"last":1.3400000000000001}]}]},{"osd":6,"up_from":47,"seq":201863462972,"num_pgs":61,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2581744,"kb_used_data":2276068,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":91163408,"statfs":{"total":95995035648,"available":93351329792,"internally_reserved":0,"allocated":2330693632,"data_stored":2330317148,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 19:13:42 2024","interfaces":[{"interface":"back","average":{"1min":0.90700000000000003,"5min":0.58499999999999996,"15min":0.54300000000000004},"min":{"1min":0.19800000000000001,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":2.8050000000000002,"5min":2.8050000000000002,"15min":2.8050000000000002},"last":0.40999999999999998},{"interface":"front","average":{"1min":0.96599999999999997,"5min":0.62,"15min":0.56899999999999995},"min":{"1min":0.23300000000000001,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":3.4609999999999999,"5min":3.4609999999999999,"15min":3.4609999999999999},"last":0.22900000000000001}]},{"osd":1,"last update":"Tue Sep 10 19:12:41 2024","interfaces":[{"interface":"back","average":{"1min":0.59999999999999998,"5min":0.53700000000000003,"15min":0.53100000000000003},"min":{"1min":0.33800000000000002,"5min":0.318,"15min":0.318},"max":{"1min":1.29,"5min":1.29,"15min":1.29},"last":0.26500000000000001},{"interface":"front","average":{"1min":0.69399999999999995,"5min":0.55400000000000005,"15min":0.53800000000000003},"min":{"1min":0.32400000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":2.6859999999999999,"5min":2.6859999999999999,"15min":2.6859999999999999},"last":0.38200000000000001}]},{"osd":2,"last update":"Tue Sep 10 19:13:42 2024","interfaces":[{"interface":"back","average":{"1min":0.84099999999999997,"5min":0.59499999999999997,"15min":0.56100000000000005},"min":{"1min":0.248,"5min":0.248,"15min":0.248},"max":{"1min":2.0049999999999999,"5min":2.0049999999999999,"15min":2.0049999999999999},"last":0.35499999999999998},{"interface":"front","average":{"1min":0.97999999999999998,"5min":0.61099999999999999,"15min":0.54300000000000004},"min":{"1min":0.28699999999999998,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":3.5659999999999998,"5min":3.5659999999999998,"15min":3.5659999999999998},"last":0.307}]},{"osd":3,"last update":"Tue Sep 10 19:13:42 2024","interfaces":[{"interface":"back","average":{"1min":0.97199999999999998,"5min":0.628,"15min":0.55300000000000005},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":2.3119999999999998,"5min":2.3119999999999998,"15min":2.3119999999999998},"last":0.53700000000000003},{"interface":"front","average":{"1min":1.0529999999999999,"5min":0.68300000000000005,"15min":0.60499999999999998},"min":{"1min":0.23200000000000001,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":3.5840000000000001,"5min":3.5840000000000001,"15min":3.5840000000000001},"last":0.81999999999999995}]},{"osd":4,"last update":"Tue Sep 10 19:13:42 2024","interfaces":[{"interface":"back","average":{"1min":0.46200000000000002,"5min":0.39700000000000002,"15min":0.36399999999999999},"min":{"1min":0.20699999999999999,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":1.6879999999999999,"5min":1.6879999999999999,"15min":1.6879999999999999},"last":0.377},{"interface":"front","average":{"1min":0.60099999999999998,"5min":0.45800000000000002,"15min":0.40200000000000002},"min":{"1min":0.25800000000000001,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":2.0710000000000002,"5min":2.0710000000000002,"15min":2.0710000000000002},"last":0.51200000000000001}]},{"osd":5,"last update":"Tue Sep 10 19:13:42 2024","interfaces":[{"interface":"back","average":{"1min":0.41999999999999998,"5min":0.41199999999999998,"15min":0.38700000000000001},"min":{"1min":0.248,"5min":0.248,"15min":0.248},"max":{"1min":1.571,"5min":1.571,"15min":1.571},"last":0.56899999999999995},{"interface":"front","average":{"1min":0.61299999999999999,"5min":0.48099999999999998,"15min":0.40899999999999997},"min":{"1min":0.27500000000000002,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":2.298,"5min":2.298,"15min":2.298},"last":1.0549999999999999}]},{"osd":7,"last update":"Tue Sep 10 19:14:00 2024","interfaces":[{"interface":"back","average":{"1min":0.47499999999999998,"5min":0.47099999999999997,"15min":0.45500000000000002},"min":{"1min":0.27500000000000002,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":1.6559999999999999,"5min":1.6559999999999999,"15min":1.6559999999999999},"last":0.38900000000000001},{"interface":"front","average":{"1min":0.64300000000000002,"5min":0.53200000000000003,"15min":0.47399999999999998},"min":{"1min":0.29299999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":2.3260000000000001,"5min":2.3260000000000001,"15min":2.3260000000000001},"last":0.65600000000000003}]}]},{"osd":5,"up_from":39,"seq":167503724609,"num_pgs":60,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2008656,"kb_used_data":1702980,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":91736496,"statfs":{"total":95995035648,"available":93938171904,"internally_reserved":0,"allocated":1743851520,"data_stored":1743507358,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 19:14:13 2024","interfaces":[{"interface":"back","average":{"1min":1.0089999999999999,"5min":0.626,"15min":0.52500000000000002},"min":{"1min":0.24099999999999999,"5min":0.223,"15min":0.223},"max":{"1min":5.851,"5min":5.851,"15min":5.851},"last":2.3340000000000001},{"interface":"front","average":{"1min":0.98199999999999998,"5min":0.622,"15min":0.52100000000000002},"min":{"1min":0.224,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":5.6500000000000004,"5min":5.6500000000000004,"15min":5.6500000000000004},"last":0.52200000000000002}]},{"osd":1,"last update":"Tue Sep 10 19:12:09 2024","interfaces":[{"interface":"back","average":{"1min":0.48199999999999998,"5min":0.51500000000000001,"15min":0.50700000000000001},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.64600000000000002,"5min":0.72799999999999998,"15min":0.72799999999999998},"last":0.51000000000000001},{"interface":"front","average":{"1min":0.47299999999999998,"5min":0.47899999999999998,"15min":0.47399999999999998},"min":{"1min":0.34399999999999997,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.61599999999999999,"5min":0.68799999999999994,"15min":0.68799999999999994},"last":3.1960000000000002}]},{"osd":2,"last update":"Tue Sep 10 19:14:13 2024","interfaces":[{"interface":"back","average":{"1min":0.69299999999999995,"5min":0.59999999999999998,"15min":0.51600000000000001},"min":{"1min":0.27000000000000002,"5min":0.253,"15min":0.253},"max":{"1min":1.7270000000000001,"5min":2.2679999999999998,"15min":2.2679999999999998},"last":0.33100000000000002},{"interface":"front","average":{"1min":1.645,"5min":0.73599999999999999,"15min":0.56999999999999995},"min":{"1min":0.224,"5min":0.224,"15min":0.224},"max":{"1min":6.3789999999999996,"5min":6.3789999999999996,"15min":6.3789999999999996},"last":0.28999999999999998}]},{"osd":3,"last update":"Tue Sep 10 19:14:13 2024","interfaces":[{"interface":"back","average":{"1min":0.64400000000000002,"5min":0.60499999999999998,"15min":0.54200000000000004},"min":{"1min":0.26700000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":3.351,"5min":3.351,"15min":3.351},"last":0.372},{"interface":"front","average":{"1min":0.76200000000000001,"5min":0.61499999999999999,"15min":0.55300000000000005},"min":{"1min":0.251,"5min":0.248,"15min":0.248},"max":{"1min":3.403,"5min":3.403,"15min":3.403},"last":0.48099999999999998}]},{"osd":4,"last update":"Tue Sep 10 19:14:13 2024","interfaces":[{"interface":"back","average":{"1min":0.77500000000000002,"5min":0.47599999999999998,"15min":0.39400000000000002},"min":{"1min":0.26000000000000001,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":2.8090000000000002,"5min":2.8090000000000002,"15min":2.8090000000000002},"last":1.0629999999999999},{"interface":"front","average":{"1min":0.96199999999999997,"5min":0.56999999999999995,"15min":0.42999999999999999},"min":{"1min":0.311,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":5.8899999999999997,"5min":5.8899999999999997,"15min":5.8899999999999997},"last":0.66600000000000004}]},{"osd":6,"last update":"Tue Sep 10 19:13:37 2024","interfaces":[{"interface":"back","average":{"1min":0.61499999999999999,"5min":0.49299999999999999,"15min":0.438},"min":{"1min":0.27100000000000002,"5min":0.223,"15min":0.223},"max":{"1min":2.8540000000000001,"5min":2.8540000000000001,"15min":2.8540000000000001},"last":0.59899999999999998},{"interface":"front","average":{"1min":0.59699999999999998,"5min":0.47699999999999998,"15min":0.42899999999999999},"min":{"1min":0.32600000000000001,"5min":0.25,"15min":0.25},"max":{"1min":1.2010000000000001,"5min":1.2010000000000001,"15min":1.2010000000000001},"last":0.876}]},{"osd":7,"last update":"Tue Sep 10 19:13:57 2024","interfaces":[{"interface":"back","average":{"1min":0.72299999999999998,"5min":0.54300000000000004,"15min":0.52100000000000002},"min":{"1min":0.316,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":1.9970000000000001,"5min":1.9970000000000001,"15min":1.9970000000000001},"last":0.57099999999999995},{"interface":"front","average":{"1min":0.60799999999999998,"5min":0.50900000000000001,"15min":0.48899999999999999},"min":{"1min":0.30499999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":2.0099999999999998,"5min":2.0099999999999998,"15min":2.0099999999999998},"last":1.0329999999999999}]}]},{"osd":4,"up_from":32,"seq":137438953541,"num_pgs":61,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2394384,"kb_used_data":2096968,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":91350768,"statfs":{"total":95995035648,"available":93543186432,"internally_reserved":0,"allocated":2147295232,"data_stored":2146945719,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 19:13:55 2024","interfaces":[{"interface":"back","average":{"1min":0.61499999999999999,"5min":0.54600000000000004,"15min":0.54300000000000004},"min":{"1min":0.22900000000000001,"5min":0.22,"15min":0.22},"max":{"1min":2.157,"5min":2.157,"15min":2.157},"last":0.79400000000000004},{"interface":"front","average":{"1min":0.60399999999999998,"5min":0.54600000000000004,"15min":0.53600000000000003},"min":{"1min":0.20999999999999999,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":2.1070000000000002,"5min":2.5499999999999998,"15min":2.5499999999999998},"last":0.39800000000000002}]},{"osd":1,"last update":"Tue Sep 10 19:11:49 2024","interfaces":[{"interface":"back","average":{"1min":0.47999999999999998,"5min":0.503,"15min":0.50900000000000001},"min":{"1min":0.29999999999999999,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.72199999999999998,"5min":0.83499999999999996,"15min":0.83499999999999996},"last":0.997},{"interface":"front","average":{"1min":0.51000000000000001,"5min":0.51700000000000002,"15min":0.52200000000000002},"min":{"1min":0.27500000000000002,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.68999999999999995,"5min":0.77100000000000002,"15min":0.77100000000000002},"last":0.67400000000000004}]},{"osd":2,"last update":"Tue Sep 10 19:13:55 2024","interfaces":[{"interface":"back","average":{"1min":0.83099999999999996,"5min":0.56999999999999995,"15min":0.54900000000000004},"min":{"1min":0.27400000000000002,"5min":0.221,"15min":0.221},"max":{"1min":2.54,"5min":2.54,"15min":2.54},"last":0.82399999999999995},{"interface":"front","average":{"1min":0.753,"5min":0.56499999999999995,"15min":0.54700000000000004},"min":{"1min":0.27100000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":2.8679999999999999,"5min":2.8679999999999999,"15min":2.8679999999999999},"last":3.73}]},{"osd":3,"last update":"Tue Sep 10 19:13:55 2024","interfaces":[{"interface":"back","average":{"1min":0.755,"5min":0.59399999999999997,"15min":0.54900000000000004},"min":{"1min":0.252,"5min":0.252,"15min":0.252},"max":{"1min":1.712,"5min":2.109,"15min":2.109},"last":0.371},{"interface":"front","average":{"1min":0.95099999999999996,"5min":0.629,"15min":0.57699999999999996},"min":{"1min":0.26200000000000001,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":2.6850000000000001,"5min":2.6850000000000001,"15min":2.6850000000000001},"last":3.9809999999999999}]},{"osd":5,"last update":"Tue Sep 10 19:14:18 2024","interfaces":[{"interface":"back","average":{"1min":0.61299999999999999,"5min":0.44800000000000001,"15min":0.39800000000000002},"min":{"1min":0.22900000000000001,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":3.629,"5min":3.629,"15min":3.629},"last":0.30599999999999999},{"interface":"front","average":{"1min":0.62,"5min":0.47899999999999998,"15min":0.42199999999999999},"min":{"1min":0.248,"5min":0.223,"15min":0.223},"max":{"1min":2.1880000000000002,"5min":2.1880000000000002,"15min":2.1880000000000002},"last":2.1880000000000002}]},{"osd":6,"last update":"Tue Sep 10 19:13:43 2024","interfaces":[{"interface":"back","average":{"1min":0.54600000000000004,"5min":0.47099999999999997,"15min":0.432},"min":{"1min":0.253,"5min":0.253,"15min":0.253},"max":{"1min":1.992,"5min":1.992,"15min":1.992},"last":2.4529999999999998},{"interface":"front","average":{"1min":0.48199999999999998,"5min":0.44700000000000001,"15min":0.42499999999999999},"min":{"1min":0.25,"5min":0.25,"15min":0.25},"max":{"1min":1.0469999999999999,"5min":1.0469999999999999,"15min":1.0469999999999999},"last":0.58399999999999996}]},{"osd":7,"last update":"Tue Sep 10 19:14:03 2024","interfaces":[{"interface":"back","average":{"1min":2.149,"5min":0.82499999999999996,"15min":0.625},"min":{"1min":0.29299999999999998,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":23.081,"5min":23.081,"15min":23.081},"last":0.77700000000000002},{"interface":"front","average":{"1min":2.1280000000000001,"5min":0.82799999999999996,"15min":0.61099999999999999},"min":{"1min":0.311,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":22.666,"5min":22.666,"15min":22.666},"last":0.41799999999999998}]}]},{"osd":3,"up_from":26,"seq":111669149770,"num_pgs":71,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":4096180,"kb_used_data":3790504,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":89648972,"statfs":{"total":95995035648,"available":91800547328,"internally_reserved":0,"allocated":3881476096,"data_stored":3881095719,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 19:13:29 2024","interfaces":[{"interface":"back","average":{"1min":1.5009999999999999,"5min":0.62,"15min":0.47799999999999998},"min":{"1min":0.16800000000000001,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":4.452,"5min":4.452,"15min":4.452},"last":2.6429999999999998},{"interface":"front","average":{"1min":1.8919999999999999,"5min":0.71599999999999997,"15min":0.51800000000000002},"min":{"1min":0.23300000000000001,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":6.1429999999999998,"5min":6.1429999999999998,"15min":6.1429999999999998},"last":2.52}]},{"osd":1,"last update":"Tue Sep 10 19:12:27 2024","interfaces":[{"interface":"back","average":{"1min":0.52800000000000002,"5min":0.44400000000000001,"15min":0.45200000000000001},"min":{"1min":0.216,"5min":0.216,"15min":0.216},"max":{"1min":3.5150000000000001,"5min":3.5150000000000001,"15min":3.5150000000000001},"last":0.79700000000000004},{"interface":"front","average":{"1min":0.46600000000000003,"5min":0.40000000000000002,"15min":0.39800000000000002},"min":{"1min":0.187,"5min":0.187,"15min":0.187},"max":{"1min":1.3640000000000001,"5min":1.3640000000000001,"15min":1.3640000000000001},"last":0.59099999999999997}]},{"osd":2,"last update":"Tue Sep 10 19:13:29 2024","interfaces":[{"interface":"back","average":{"1min":1.0920000000000001,"5min":0.56599999999999995,"15min":0.48099999999999998},"min":{"1min":0.186,"5min":0.186,"15min":0.186},"max":{"1min":3.6459999999999999,"5min":3.6459999999999999,"15min":3.6459999999999999},"last":0.378},{"interface":"front","average":{"1min":1.4490000000000001,"5min":0.626,"15min":0.48599999999999999},"min":{"1min":0.36399999999999999,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":4.2960000000000003,"5min":4.2960000000000003,"15min":4.2960000000000003},"last":2.2850000000000001}]},{"osd":4,"last update":"Tue Sep 10 19:13:56 2024","interfaces":[{"interface":"back","average":{"1min":1.73,"5min":0.94199999999999995,"15min":0.67700000000000005},"min":{"1min":0.31900000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":4.54,"5min":11.964,"15min":11.964},"last":2.5750000000000002},{"interface":"front","average":{"1min":1.6799999999999999,"5min":1.026,"15min":0.70699999999999996},"min":{"1min":0.26200000000000001,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":6.0350000000000001,"5min":11.881,"15min":11.881},"last":2.9350000000000001}]},{"osd":5,"last update":"Tue Sep 10 19:14:16 2024","interfaces":[{"interface":"back","average":{"1min":1.456,"5min":0.877,"15min":0.66900000000000004},"min":{"1min":0.32500000000000001,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":4.6070000000000002,"5min":4.6070000000000002,"15min":4.6070000000000002},"last":2.5539999999999998},{"interface":"front","average":{"1min":1.643,"5min":1.034,"15min":0.71799999999999997},"min":{"1min":0.28599999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":4.2770000000000001,"5min":5.9950000000000001,"15min":5.9950000000000001},"last":2.907}]},{"osd":6,"last update":"Tue Sep 10 19:13:36 2024","interfaces":[{"interface":"back","average":{"1min":1.681,"5min":0.81200000000000006,"15min":0.65600000000000003},"min":{"1min":0.30399999999999999,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":4.5629999999999997,"5min":4.5629999999999997,"15min":4.5629999999999997},"last":2.621},{"interface":"front","average":{"1min":0.89400000000000002,"5min":0.65300000000000002,"15min":0.58899999999999997},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":2.9249999999999998,"5min":2.9249999999999998,"15min":2.9249999999999998},"last":0.41899999999999998}]},{"osd":7,"last update":"Tue Sep 10 19:14:07 2024","interfaces":[{"interface":"back","average":{"1min":1.3009999999999999,"5min":0.86499999999999999,"15min":0.67500000000000004},"min":{"1min":0.434,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":3.3029999999999999,"5min":13.538,"15min":13.538},"last":0.61099999999999999},{"interface":"front","average":{"1min":1.595,"5min":1.016,"15min":0.72799999999999998},"min":{"1min":0.35499999999999998,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":4.6280000000000001,"5min":13.522,"15min":13.522},"last":2.3660000000000001}]}]},{"osd":2,"up_from":18,"seq":77309411407,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3028484,"kb_used_data":2731064,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":90716668,"statfs":{"total":95995035648,"available":92893868032,"internally_reserved":0,"allocated":2796609536,"data_stored":2796276507,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 19:14:16 2024","interfaces":[{"interface":"back","average":{"1min":1.3799999999999999,"5min":1.5049999999999999,"15min":0.72799999999999998},"min":{"1min":0.20000000000000001,"5min":0.188,"15min":0.188},"max":{"1min":4.4989999999999997,"5min":62.543999999999997,"15min":62.543999999999997},"last":0.70799999999999996},{"interface":"front","average":{"1min":1.548,"5min":1.3979999999999999,"15min":0.70899999999999996},"min":{"1min":0.28499999999999998,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":3.0979999999999999,"5min":62.576999999999998,"15min":62.576999999999998},"last":0.77500000000000002}]},{"osd":1,"last update":"Tue Sep 10 19:12:08 2024","interfaces":[{"interface":"back","average":{"1min":0.317,"5min":0.378,"15min":0.375},"min":{"1min":0.23400000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.46800000000000003,"5min":0.69799999999999995,"15min":0.69799999999999995},"last":2.6230000000000002},{"interface":"front","average":{"1min":0.33300000000000002,"5min":0.38600000000000001,"15min":0.38800000000000001},"min":{"1min":0.255,"5min":0.255,"15min":0.255},"max":{"1min":0.499,"5min":0.63400000000000001,"15min":0.63400000000000001},"last":3.9940000000000002}]},{"osd":3,"last update":"Tue Sep 10 19:13:33 2024","interfaces":[{"interface":"back","average":{"1min":2.0190000000000001,"5min":1.409,"15min":0.77100000000000002},"min":{"1min":0.30599999999999999,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":4.6950000000000003,"5min":63.997,"15min":63.997},"last":0.45300000000000001},{"interface":"front","average":{"1min":1.8560000000000001,"5min":1.298,"15min":0.73099999999999998},"min":{"1min":0.22800000000000001,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":4.1210000000000004,"5min":62.365000000000002,"15min":62.365000000000002},"last":0.36499999999999999}]},{"osd":4,"last update":"Tue Sep 10 19:13:57 2024","interfaces":[{"interface":"back","average":{"1min":1.127,"5min":0.75800000000000001,"15min":0.64000000000000001},"min":{"1min":0.254,"5min":0.254,"15min":0.254},"max":{"1min":4.3650000000000002,"5min":4.3650000000000002,"15min":4.3650000000000002},"last":0.39800000000000002},{"interface":"front","average":{"1min":1.1619999999999999,"5min":0.752,"15min":0.63200000000000001},"min":{"1min":0.26000000000000001,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":3.0899999999999999,"5min":4.0750000000000002,"15min":4.0750000000000002},"last":0.317}]},{"osd":5,"last update":"Tue Sep 10 19:14:16 2024","interfaces":[{"interface":"back","average":{"1min":0.65700000000000003,"5min":0.68000000000000005,"15min":0.57599999999999996},"min":{"1min":0.255,"5min":0.255,"15min":0.255},"max":{"1min":3.234,"5min":5.4820000000000002,"15min":5.4820000000000002},"last":0.95999999999999996},{"interface":"front","average":{"1min":1.2490000000000001,"5min":0.82099999999999995,"15min":0.65900000000000003},"min":{"1min":0.35299999999999998,"5min":0.309,"15min":0.309},"max":{"1min":4.4729999999999999,"5min":4.6429999999999998,"15min":4.6429999999999998},"last":0.42799999999999999}]},{"osd":6,"last update":"Tue Sep 10 19:13:38 2024","interfaces":[{"interface":"back","average":{"1min":1.3029999999999999,"5min":0.754,"15min":0.624},"min":{"1min":0.41199999999999998,"5min":0.37,"15min":0.37},"max":{"1min":4.6180000000000003,"5min":4.6180000000000003,"15min":4.6180000000000003},"last":0.73799999999999999},{"interface":"front","average":{"1min":1.464,"5min":0.75600000000000001,"15min":0.61899999999999999},"min":{"1min":0.28999999999999998,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":4.8049999999999997,"5min":4.8049999999999997,"15min":4.8049999999999997},"last":1}]},{"osd":7,"last update":"Tue Sep 10 19:14:03 2024","interfaces":[{"interface":"back","average":{"1min":1.589,"5min":0.84699999999999998,"15min":0.68600000000000005},"min":{"1min":0.41599999999999998,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":4.5640000000000001,"5min":4.5640000000000001,"15min":4.5640000000000001},"last":1.3819999999999999},{"interface":"front","average":{"1min":1.226,"5min":0.79200000000000004,"15min":0.63},"min":{"1min":0.312,"5min":0.312,"15min":0.312},"max":{"1min":3.1179999999999999,"5min":4.0970000000000004,"15min":4.0970000000000004},"last":0.91800000000000004}]}]},{"osd":1,"up_from":0,"seq":0,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":8,"seq":34359738457,"num_pgs":64,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3677864,"kb_used_data":3372060,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":90067288,"statfs":{"total":95995035648,"available":92228902912,"internally_reserved":0,"allocated":3452989440,"data_stored":3452621441,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Sep 10 19:12:44 2024","interfaces":[{"interface":"back","average":{"1min":0.73199999999999998,"5min":0.42999999999999999,"15min":0.35799999999999998},"min":{"1min":0.17399999999999999,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":2.7320000000000002,"5min":2.7320000000000002,"15min":2.7320000000000002},"last":0.80700000000000005},{"interface":"front","average":{"1min":0.82299999999999995,"5min":0.45100000000000001,"15min":0.36799999999999999},"min":{"1min":0.21199999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":2.4609999999999999,"5min":2.4609999999999999,"15min":2.4609999999999999},"last":2.4609999999999999}]},{"osd":2,"last update":"Tue Sep 10 19:14:16 2024","interfaces":[{"interface":"back","average":{"1min":1.252,"5min":0.67800000000000005,"15min":0.47999999999999998},"min":{"1min":0.17199999999999999,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":5.8140000000000001,"5min":5.8140000000000001,"15min":5.8140000000000001},"last":3.2909999999999999},{"interface":"front","average":{"1min":1.1180000000000001,"5min":0.60099999999999998,"15min":0.44800000000000001},"min":{"1min":0.216,"5min":0.187,"15min":0.187},"max":{"1min":3.399,"5min":3.399,"15min":3.399},"last":0.83299999999999996}]},{"osd":3,"last update":"Tue Sep 10 19:13:36 2024","interfaces":[{"interface":"back","average":{"1min":1.3049999999999999,"5min":0.59499999999999997,"15min":0.48099999999999998},"min":{"1min":0.253,"5min":0.252,"15min":0.252},"max":{"1min":2.8039999999999998,"5min":2.8039999999999998,"15min":2.8039999999999998},"last":0.38300000000000001},{"interface":"front","average":{"1min":1.3859999999999999,"5min":0.60799999999999998,"15min":0.47499999999999998},"min":{"1min":0.372,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":4.9119999999999999,"5min":4.9119999999999999,"15min":4.9119999999999999},"last":6.931}]},{"osd":4,"last update":"Tue Sep 10 19:13:49 2024","interfaces":[{"interface":"back","average":{"1min":1.3260000000000001,"5min":0.70699999999999996,"15min":0.61299999999999999},"min":{"1min":0.24399999999999999,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":5.8499999999999996,"5min":5.8499999999999996,"15min":5.8499999999999996},"last":0.32800000000000001},{"interface":"front","average":{"1min":0.998,"5min":0.70899999999999996,"15min":0.61099999999999999},"min":{"1min":0.27700000000000002,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":3.2799999999999998,"5min":3.2999999999999998,"15min":3.2999999999999998},"last":0.41499999999999998}]},{"osd":5,"last update":"Tue Sep 10 19:14:16 2024","interfaces":[{"interface":"back","average":{"1min":1.0620000000000001,"5min":0.72699999999999998,"15min":0.63200000000000001},"min":{"1min":0.29499999999999998,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":3.6190000000000002,"5min":3.6190000000000002,"15min":3.6190000000000002},"last":0.44700000000000001},{"interface":"front","average":{"1min":1.0640000000000001,"5min":0.748,"15min":0.63300000000000001},"min":{"1min":0.253,"5min":0.253,"15min":0.253},"max":{"1min":3.5720000000000001,"5min":4.9790000000000001,"15min":4.9790000000000001},"last":0.499}]},{"osd":6,"last update":"Tue Sep 10 19:13:36 2024","interfaces":[{"interface":"back","average":{"1min":1.1559999999999999,"5min":0.67800000000000005,"15min":0.60399999999999998},"min":{"1min":0.27200000000000002,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":3.536,"5min":3.536,"15min":3.536},"last":0.35299999999999998},{"interface":"front","average":{"1min":1.1930000000000001,"5min":0.70199999999999996,"15min":0.625},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":3.3660000000000001,"5min":3.3660000000000001,"15min":3.3660000000000001},"last":0.48599999999999999}]},{"osd":7,"last update":"Tue Sep 10 19:14:02 2024","interfaces":[{"interface":"back","average":{"1min":1.0629999999999999,"5min":0.76000000000000001,"15min":0.63800000000000001},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":3.3580000000000001,"5min":3.3919999999999999,"15min":3.3919999999999999},"last":0.46899999999999997},{"interface":"front","average":{"1min":1.1279999999999999,"5min":0.69399999999999995,"15min":0.60599999999999998},"min":{"1min":0.39500000000000002,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":4.9470000000000001,"5min":4.9470000000000001,"15min":4.9470000000000001},"last":0.52600000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":966,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1236,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1236,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":362,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1650,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":2118,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":908,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":2228,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1942,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":3417509888,"data_stored":3417505816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1325072384,"data_stored":1325072384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":2789670912,"data_stored":2789670912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":3852075008,"data_stored":3852075008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":2129330176,"data_stored":2129330176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1733099520,"data_stored":1733099520,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":2322665472,"data_stored":2322661400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":2315976704,"data_stored":2315976704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T19:14:25.088 INFO:tasks.cephadm.ceph_manager.ceph:would drop below min_pgs, proj 6, currently 16 2024-09-10T19:14:25.092 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:14:25.124 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:14:25.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:14:25.125 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:14:25.226 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:14:25.241 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-10T19:14:25.262 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:14:25.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:14:25.263 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:14:25.364 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:14:25.395 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:14:25.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:14:25.396 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:14:25.497 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:14:25.528 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:14:25.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:14:25.529 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:14:25.630 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:14:25.663 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:14:25.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:14:25.664 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:14:25.765 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:14:25.796 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:14:25.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:14:25.797 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:14:25.898 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:14:25.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:14:25.929 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:14:25.929 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:14:25.931 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:14:25 smithi183 ceph-mon[29736]: Health check update: Degraded data redundancy: 31107/314940 objects degraded (9.877%), 6 pgs degraded (PG_DEGRADED) 2024-09-10T19:14:25.931 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:14:25 smithi183 ceph-mon[41562]: Health check update: Degraded data redundancy: 31107/314940 objects degraded (9.877%), 6 pgs degraded (PG_DEGRADED) 2024-09-10T19:14:25.965 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:14:25 smithi191 ceph-mon[32884]: Health check update: Degraded data redundancy: 31107/314940 objects degraded (9.877%), 6 pgs degraded (PG_DEGRADED) 2024-09-10T19:14:26.030 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:14:26.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:14:26.062 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:14:26.062 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:14:26.162 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:14:26.195 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:14:26.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:14:26.197 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:14:26.297 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:14:26.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:14:26.344 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:14:26.344 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:14:26.445 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:14:26.487 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:14:26.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:14:26.488 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:14:26.589 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:14:26.613 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:14:26.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:14:26.614 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:14:26.624 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:14:26.714 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:14:26.761 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:14:26.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:14:26.762 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:14:26.863 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:14:26.891 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:14:26.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:14:26.892 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:14:26.926 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:14:26 smithi183 ceph-mon[29736]: pgmap v163: 148 pgs: 3 active+undersized+degraded+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 142 active+clean; 6.0 GiB data, 18 GiB used, 607 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.26k op/s; 27213/295281 objects degraded (9.216%); 24 MiB/s, 389 objects/s recovering 2024-09-10T19:14:26.927 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:14:26 smithi183 ceph-mon[41562]: pgmap v163: 148 pgs: 3 active+undersized+degraded+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 142 active+clean; 6.0 GiB data, 18 GiB used, 607 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.26k op/s; 27213/295281 objects degraded (9.216%); 24 MiB/s, 389 objects/s recovering 2024-09-10T19:14:26.965 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:14:26 smithi191 ceph-mon[32884]: pgmap v163: 148 pgs: 3 active+undersized+degraded+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 142 active+clean; 6.0 GiB data, 18 GiB used, 607 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.26k op/s; 27213/295281 objects degraded (9.216%); 24 MiB/s, 389 objects/s recovering 2024-09-10T19:14:26.993 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:14:27.033 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:14:27.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:14:27.034 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:14:27.134 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:14:27.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:14:27.173 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:14:27.173 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:14:27.274 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:14:27.311 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:14:27.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:14:27.312 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:14:27.413 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:14:27.457 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:14:27.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:14:27.458 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:14:27.559 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:14:27.599 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:14:27.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:14:27.600 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:14:27.701 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:14:27.712 INFO:journalctl@ceph.mgr.x.smithi191.stdout:Sep 10 19:14:27 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-x[34967]: ::ffff:172.21.15.191 - - [10/Sep/2024:19:14:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T19:14:27.728 INFO:teuthology.orchestra.run.smithi183.stderr:noscrub is unset 2024-09-10T19:14:27.744 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:14:27.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:14:27.745 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:14:27.765 INFO:teuthology.orchestra.run.smithi191.stdout:{ 2024-09-10T19:14:27.765 INFO:teuthology.orchestra.run.smithi191.stdout: "ops": [ 2024-09-10T19:14:27.766 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:14:27.766 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_repop(client.14781.0:186085 7.8 e123/86 7:1c1dedc5:::benchmark_data_smithi183_84160_object45704:head v 123'11691, mlcod=122'11687)", 2024-09-10T19:14:27.766 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-10T19:14:27.745104+0000", 2024-09-10T19:14:27.766 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 0.016301814000000001, 2024-09-10T19:14:27.766 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.016325763, 2024-09-10T19:14:27.766 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-10T19:14:27.766 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "started", 2024-09-10T19:14:27.766 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-10T19:14:27.766 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:14:27.766 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-10T19:14:27.766 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:27.745104+0000", 2024-09-10T19:14:27.766 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:14:27.766 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:14:27.767 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:14:27.767 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-10T19:14:27.767 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:27.745104+0000", 2024-09-10T19:14:27.767 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:14:27.767 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:14:27.767 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:14:27.767 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-10T19:14:27.767 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:27.745103+0000", 2024-09-10T19:14:27.767 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999995 2024-09-10T19:14:27.767 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:14:27.767 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:14:27.767 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-10T19:14:27.768 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:27.745107+0000", 2024-09-10T19:14:27.768 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.6909999999999999e-06 2024-09-10T19:14:27.768 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:14:27.768 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:14:27.768 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-10T19:14:27.768 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:27.745113+0000", 2024-09-10T19:14:27.768 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.1870000000000002e-06 2024-09-10T19:14:27.768 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:14:27.768 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:14:27.768 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-10T19:14:27.768 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:27.745117+0000", 2024-09-10T19:14:27.768 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.6119999999999999e-06 2024-09-10T19:14:27.768 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:14:27.768 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:14:27.769 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:14:27.769 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:27.749882+0000", 2024-09-10T19:14:27.769 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0047651969999999997 2024-09-10T19:14:27.769 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:14:27.769 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:14:27.769 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-10T19:14:27.769 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:27.749901+0000", 2024-09-10T19:14:27.769 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.9062000000000001e-05 2024-09-10T19:14:27.769 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:14:27.769 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-10T19:14:27.769 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:14:27.769 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:14:27.769 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:14:27.770 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_repop(client.14781.0:186080 7.c e123/86 7:36b18485:::benchmark_data_smithi183_84160_object45699:head v 123'11635, mlcod=122'11629)", 2024-09-10T19:14:27.770 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-10T19:14:27.749090+0000", 2024-09-10T19:14:27.770 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 0.012315144, 2024-09-10T19:14:27.770 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.012405816, 2024-09-10T19:14:27.770 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-10T19:14:27.770 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "started", 2024-09-10T19:14:27.770 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-10T19:14:27.770 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:14:27.770 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-10T19:14:27.770 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:27.749090+0000", 2024-09-10T19:14:27.770 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:14:27.770 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:14:27.770 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:14:27.771 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-10T19:14:27.771 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:27.749090+0000", 2024-09-10T19:14:27.771 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:14:27.771 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:14:27.771 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:14:27.771 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-10T19:14:27.771 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:27.749090+0000", 2024-09-10T19:14:27.771 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.999999 2024-09-10T19:14:27.771 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:14:27.771 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:14:27.771 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-10T19:14:27.771 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:27.749098+0000", 2024-09-10T19:14:27.771 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.422e-06 2024-09-10T19:14:27.772 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:14:27.772 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:14:27.772 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-10T19:14:27.772 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:27.749115+0000", 2024-09-10T19:14:27.772 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.6911000000000001e-05 2024-09-10T19:14:27.772 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:14:27.772 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:14:27.772 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-10T19:14:27.772 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:27.749121+0000", 2024-09-10T19:14:27.772 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.3369999999999996e-06 2024-09-10T19:14:27.772 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:14:27.772 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:14:27.772 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:14:27.772 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:27.752207+0000", 2024-09-10T19:14:27.773 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0030853450000000002 2024-09-10T19:14:27.773 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:14:27.773 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:14:27.773 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-10T19:14:27.773 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:27.752226+0000", 2024-09-10T19:14:27.773 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.9403999999999999e-05 2024-09-10T19:14:27.773 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:14:27.773 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-10T19:14:27.773 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:14:27.773 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:14:27.773 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:14:27.773 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_repop(client.14781.0:186087 7.2 e123/116 7:4e39756a:::benchmark_data_smithi183_84160_object45706:head v 123'11780, mlcod=122'11777)", 2024-09-10T19:14:27.773 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-10T19:14:27.752928+0000", 2024-09-10T19:14:27.773 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 0.0084777350000000001, 2024-09-10T19:14:27.774 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0086298220000000005, 2024-09-10T19:14:27.774 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-10T19:14:27.774 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "started", 2024-09-10T19:14:27.774 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-10T19:14:27.774 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:14:27.774 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-10T19:14:27.774 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:27.752928+0000", 2024-09-10T19:14:27.774 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:14:27.774 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:14:27.774 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:14:27.774 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-10T19:14:27.774 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:27.752928+0000", 2024-09-10T19:14:27.774 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:14:27.775 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:14:27.775 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:14:27.775 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-10T19:14:27.775 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:27.752927+0000", 2024-09-10T19:14:27.775 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.999999 2024-09-10T19:14:27.775 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:14:27.775 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:14:27.775 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-10T19:14:27.775 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:27.752935+0000", 2024-09-10T19:14:27.775 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.2050000000000002e-06 2024-09-10T19:14:27.775 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:14:27.775 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:14:27.775 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-10T19:14:27.775 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:27.752946+0000", 2024-09-10T19:14:27.776 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.1219e-05 2024-09-10T19:14:27.776 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:14:27.776 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:14:27.776 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-10T19:14:27.776 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:27.752952+0000", 2024-09-10T19:14:27.776 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.6590000000000004e-06 2024-09-10T19:14:27.776 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:14:27.776 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:14:27.776 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:14:27.776 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:27.757008+0000", 2024-09-10T19:14:27.776 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0040560259999999999 2024-09-10T19:14:27.776 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:14:27.776 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:14:27.777 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-10T19:14:27.777 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:27.757028+0000", 2024-09-10T19:14:27.777 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.0567000000000001e-05 2024-09-10T19:14:27.777 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:14:27.777 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-10T19:14:27.777 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:14:27.777 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:14:27.777 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:14:27.777 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_repop(client.14781.0:186090 7.f e123/86 7:fefd1109:::benchmark_data_smithi183_84160_object45709:head v 123'11637, mlcod=122'11635)", 2024-09-10T19:14:27.777 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-10T19:14:27.752947+0000", 2024-09-10T19:14:27.777 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 0.0084582149999999998, 2024-09-10T19:14:27.777 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0086740159999999997, 2024-09-10T19:14:27.777 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-10T19:14:27.778 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "started", 2024-09-10T19:14:27.778 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-10T19:14:27.778 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:14:27.778 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-10T19:14:27.778 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:27.752947+0000", 2024-09-10T19:14:27.778 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:14:27.778 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:14:27.778 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:14:27.778 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-10T19:14:27.778 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:27.752947+0000", 2024-09-10T19:14:27.778 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:14:27.778 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:14:27.778 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:14:27.778 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-10T19:14:27.779 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:27.752947+0000", 2024-09-10T19:14:27.779 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.999999 2024-09-10T19:14:27.779 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:14:27.779 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:14:27.779 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-10T19:14:27.779 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:27.752956+0000", 2024-09-10T19:14:27.779 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.8840000000000006e-06 2024-09-10T19:14:27.779 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:14:27.779 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:14:27.779 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-10T19:14:27.779 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:27.752966+0000", 2024-09-10T19:14:27.779 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.5880000000000004e-06 2024-09-10T19:14:27.779 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:14:27.779 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:14:27.780 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-10T19:14:27.780 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:27.752971+0000", 2024-09-10T19:14:27.780 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.8389999999999997e-06 2024-09-10T19:14:27.780 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:14:27.780 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:14:27.780 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:14:27.780 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:27.753482+0000", 2024-09-10T19:14:27.780 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00051086999999999997 2024-09-10T19:14:27.781 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:14:27.781 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:14:27.781 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-10T19:14:27.781 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:27.753499+0000", 2024-09-10T19:14:27.781 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.7192000000000001e-05 2024-09-10T19:14:27.781 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:14:27.781 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-10T19:14:27.781 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:14:27.781 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:14:27.781 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:14:27.781 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_repop(client.14781.0:186093 7.9 e123/116 7:99845182:::benchmark_data_smithi183_84160_object45712:head v 123'11504, mlcod=122'11502)", 2024-09-10T19:14:27.781 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-10T19:14:27.757898+0000", 2024-09-10T19:14:27.781 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 0.0035078119999999999, 2024-09-10T19:14:27.782 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.003798161, 2024-09-10T19:14:27.782 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-10T19:14:27.782 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "started", 2024-09-10T19:14:27.782 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-10T19:14:27.782 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:14:27.782 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-10T19:14:27.782 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:27.757898+0000", 2024-09-10T19:14:27.782 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:14:27.782 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:14:27.782 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:14:27.782 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-10T19:14:27.782 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:27.757898+0000", 2024-09-10T19:14:27.782 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:14:27.782 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:14:27.783 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:14:27.783 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-10T19:14:27.783 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:27.757896+0000", 2024-09-10T19:14:27.783 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999986 2024-09-10T19:14:27.783 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:14:27.783 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:14:27.783 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-10T19:14:27.783 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:27.757906+0000", 2024-09-10T19:14:27.783 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.2939999999999994e-06 2024-09-10T19:14:27.783 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:14:27.783 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:14:27.783 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-10T19:14:27.783 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:27.757916+0000", 2024-09-10T19:14:27.784 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.0728e-05 2024-09-10T19:14:27.784 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:14:27.784 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:14:27.784 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-10T19:14:27.784 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:27.757923+0000", 2024-09-10T19:14:27.784 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.7499999999999997e-06 2024-09-10T19:14:27.784 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:14:27.784 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:14:27.784 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:14:27.784 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:27.757976+0000", 2024-09-10T19:14:27.784 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.2621000000000003e-05 2024-09-10T19:14:27.784 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:14:27.784 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:14:27.785 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-10T19:14:27.785 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:27.757993+0000", 2024-09-10T19:14:27.785 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.7005999999999999e-05 2024-09-10T19:14:27.785 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:14:27.785 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-10T19:14:27.785 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:14:27.785 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:14:27.785 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:14:27.785 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_repop(client.14781.0:186094 7.2 e123/116 7:45b59944:::benchmark_data_smithi183_84160_object45713:head v 123'11781, mlcod=122'11778)", 2024-09-10T19:14:27.785 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-10T19:14:27.758749+0000", 2024-09-10T19:14:27.785 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 0.002656144, 2024-09-10T19:14:27.785 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0030113029999999999, 2024-09-10T19:14:27.786 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-10T19:14:27.786 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "started", 2024-09-10T19:14:27.786 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-10T19:14:27.786 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:14:27.786 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-10T19:14:27.786 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:27.758749+0000", 2024-09-10T19:14:27.786 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:14:27.786 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:14:27.786 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:14:27.786 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-10T19:14:27.786 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:27.758749+0000", 2024-09-10T19:14:27.786 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:14:27.787 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:14:27.787 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:14:27.787 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-10T19:14:27.787 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:27.758749+0000", 2024-09-10T19:14:27.787 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999995 2024-09-10T19:14:27.787 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:14:27.787 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:14:27.787 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-10T19:14:27.787 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:27.758761+0000", 2024-09-10T19:14:27.787 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.1949e-05 2024-09-10T19:14:27.787 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:14:27.787 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:14:27.787 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-10T19:14:27.787 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:27.758768+0000", 2024-09-10T19:14:27.788 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.5000000000000002e-06 2024-09-10T19:14:27.788 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:14:27.788 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:14:27.788 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-10T19:14:27.788 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:27.758773+0000", 2024-09-10T19:14:27.788 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.1280000000000001e-06 2024-09-10T19:14:27.788 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:14:27.788 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:14:27.788 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:14:27.788 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:27.758925+0000", 2024-09-10T19:14:27.788 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00015207799999999999 2024-09-10T19:14:27.788 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:14:27.789 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:14:27.789 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-10T19:14:27.789 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:27.758943+0000", 2024-09-10T19:14:27.789 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.8536e-05 2024-09-10T19:14:27.789 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:14:27.789 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-10T19:14:27.789 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:14:27.789 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:14:27.789 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:14:27.789 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDPGPush(7.9 123/116 [PushOp(7:908fcd15:::benchmark_data_smithi183_84160_object85693:head, version: 105'5233, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(7:908fcd15:::benchmark_data_smithi183_84160_object85693:head@105'5233, size: 65536, copy_subset: [0~65536], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:65536, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-10T19:14:27.789 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-10T19:14:27.761588+0000", 2024-09-10T19:14:27.789 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 4294967295.9998174, 2024-09-10T19:14:27.789 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00026920200000000002, 2024-09-10T19:14:27.790 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-10T19:14:27.790 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "queued for pg", 2024-09-10T19:14:27.790 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-10T19:14:27.790 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:14:27.790 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-10T19:14:27.790 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:27.761588+0000", 2024-09-10T19:14:27.790 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:14:27.790 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:14:27.790 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:14:27.790 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-10T19:14:27.790 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:27.761588+0000", 2024-09-10T19:14:27.790 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:14:27.790 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:14:27.790 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:14:27.790 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-10T19:14:27.791 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:27.761588+0000", 2024-09-10T19:14:27.791 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999995 2024-09-10T19:14:27.791 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:14:27.791 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:14:27.791 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-10T19:14:27.791 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:27.761720+0000", 2024-09-10T19:14:27.791 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00013238799999999999 2024-09-10T19:14:27.791 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:14:27.791 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:14:27.791 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-10T19:14:27.791 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:27.761740+0000", 2024-09-10T19:14:27.791 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.9587000000000001e-05 2024-09-10T19:14:27.791 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:14:27.792 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:14:27.792 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-10T19:14:27.792 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:27.761749+0000", 2024-09-10T19:14:27.792 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.7000000000000003e-06 2024-09-10T19:14:27.792 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:14:27.792 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-10T19:14:27.792 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:14:27.792 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:14:27.792 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:14:27.792 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "pg_backfill(progress 7.9 e 123/123 lb 7:908f47bc:::benchmark_data_smithi183_84160_object46815:head)", 2024-09-10T19:14:27.792 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-10T19:14:27.761799+0000", 2024-09-10T19:14:27.792 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 4294967295.9996066, 2024-09-10T19:14:27.792 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00010736200000000001, 2024-09-10T19:14:27.792 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-10T19:14:27.793 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "queued for pg", 2024-09-10T19:14:27.793 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-10T19:14:27.793 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:14:27.793 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-10T19:14:27.793 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:27.761799+0000", 2024-09-10T19:14:27.793 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:14:27.793 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:14:27.793 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:14:27.793 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-10T19:14:27.793 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:27.761799+0000", 2024-09-10T19:14:27.793 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:14:27.793 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:14:27.793 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:14:27.793 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-10T19:14:27.794 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:27.761799+0000", 2024-09-10T19:14:27.794 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999995 2024-09-10T19:14:27.794 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:14:27.794 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:14:27.794 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-10T19:14:27.794 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:27.761808+0000", 2024-09-10T19:14:27.794 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.747e-06 2024-09-10T19:14:27.794 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:14:27.794 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:14:27.794 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-10T19:14:27.794 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:27.761817+0000", 2024-09-10T19:14:27.794 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.5339999999999999e-06 2024-09-10T19:14:27.794 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:14:27.795 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:14:27.795 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-10T19:14:27.795 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:27.761822+0000", 2024-09-10T19:14:27.795 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.1279999999999999e-06 2024-09-10T19:14:27.795 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:14:27.795 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-10T19:14:27.795 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:14:27.795 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:14:27.795 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:14:27.795 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.14781.0:186084 7.5 7:a2813dba:::benchmark_data_smithi183_84160_object45703:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e123)", 2024-09-10T19:14:27.795 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-10T19:14:27.744854+0000", 2024-09-10T19:14:27.795 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 0.016551461, 2024-09-10T19:14:27.795 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.017110700999999999, 2024-09-10T19:14:27.796 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-10T19:14:27.796 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "waiting for sub ops", 2024-09-10T19:14:27.796 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-10T19:14:27.796 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.14781", 2024-09-10T19:14:27.796 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.183:0/2025776420", 2024-09-10T19:14:27.796 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 186084 2024-09-10T19:14:27.796 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:14:27.796 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-10T19:14:27.796 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:14:27.797 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-10T19:14:27.797 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:27.744854+0000", 2024-09-10T19:14:27.797 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:14:27.797 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:14:27.797 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:14:27.797 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-10T19:14:27.797 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:27.744854+0000", 2024-09-10T19:14:27.797 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:14:27.797 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:14:27.797 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:14:27.797 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-10T19:14:27.797 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:27.744853+0000", 2024-09-10T19:14:27.797 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999986 2024-09-10T19:14:27.798 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:14:27.798 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:14:27.798 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-10T19:14:27.798 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:27.744860+0000", 2024-09-10T19:14:27.798 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.2280000000000001e-06 2024-09-10T19:14:27.798 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:14:27.798 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:14:27.798 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-10T19:14:27.798 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:27.744872+0000", 2024-09-10T19:14:27.798 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.2199e-05 2024-09-10T19:14:27.798 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:14:27.798 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:14:27.798 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-10T19:14:27.798 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:27.744886+0000", 2024-09-10T19:14:27.799 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.4301000000000001e-05 2024-09-10T19:14:27.799 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:14:27.799 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:14:27.799 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:14:27.799 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:27.745013+0000", 2024-09-10T19:14:27.799 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00012660699999999999 2024-09-10T19:14:27.799 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:14:27.799 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:14:27.799 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op must wait for map", 2024-09-10T19:14:27.799 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:27.745019+0000", 2024-09-10T19:14:27.799 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.1449999999999996e-06 2024-09-10T19:14:27.799 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:14:27.799 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:14:27.800 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:14:27.800 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:27.753123+0000", 2024-09-10T19:14:27.800 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0081035650000000001 2024-09-10T19:14:27.800 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:14:27.800 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:14:27.800 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-10T19:14:27.800 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:27.753249+0000", 2024-09-10T19:14:27.800 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00012655899999999999 2024-09-10T19:14:27.800 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:14:27.800 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:14:27.800 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0,2", 2024-09-10T19:14:27.800 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:27.754481+0000", 2024-09-10T19:14:27.800 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0012321249999999999 2024-09-10T19:14:27.801 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:14:27.801 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:14:27.801 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:14:27.801 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:27.759483+0000", 2024-09-10T19:14:27.801 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0050017910000000002 2024-09-10T19:14:27.801 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:14:27.801 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:14:27.801 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:14:27.801 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:27.760651+0000", 2024-09-10T19:14:27.801 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0011675279999999999 2024-09-10T19:14:27.801 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:14:27.801 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-10T19:14:27.801 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:14:27.801 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:14:27.802 INFO:teuthology.orchestra.run.smithi191.stdout: ], 2024-09-10T19:14:27.802 INFO:teuthology.orchestra.run.smithi191.stdout: "num_ops": 9 2024-09-10T19:14:27.802 INFO:teuthology.orchestra.run.smithi191.stdout:} 2024-09-10T19:14:27.846 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:14:27.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:14:27.876 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:14:27.876 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:14:27.931 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:14:27 smithi183 ceph-mon[29736]: from='client.? 172.21.15.183:0/2934802828' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-10T19:14:27.931 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:14:27 smithi183 ceph-mon[29736]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-10T19:14:27.931 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:14:27 smithi183 ceph-mon[41562]: from='client.? 172.21.15.183:0/2934802828' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-10T19:14:27.931 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:14:27 smithi183 ceph-mon[41562]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-10T19:14:27.965 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:14:27 smithi191 ceph-mon[32884]: from='client.? 172.21.15.183:0/2934802828' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-10T19:14:27.965 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:14:27 smithi191 ceph-mon[32884]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-10T19:14:27.977 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:14:28.017 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:14:28.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:14:28.018 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:14:28.119 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:14:28.143 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:14:28.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:14:28.144 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:14:28.245 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:14:28.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:14:28.280 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:14:28.280 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:14:28.381 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:14:28.418 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:14:28.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:14:28.420 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:14:28.520 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:14:28.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:14:28.559 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:14:28.560 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:14:28.660 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:14:28.699 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:14:28.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:14:28.701 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:14:28.802 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:14:28.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:14:28.844 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:14:28.844 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:14:28.913 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:14:28 smithi191 ceph-mon[32884]: pgmap v164: 148 pgs: 3 active+undersized+degraded+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 142 active+clean; 6.0 GiB data, 18 GiB used, 607 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.26k op/s; 27213/295281 objects degraded (9.216%); 24 MiB/s, 389 objects/s recovering 2024-09-10T19:14:28.913 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:14:28 smithi191 ceph-mon[32884]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T19:14:28.914 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:14:28 smithi191 ceph-mon[32884]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-10T19:14:28.914 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:14:28 smithi191 ceph-mon[32884]: osdmap e123: 8 total, 7 up, 7 in 2024-09-10T19:14:28.914 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:14:28 smithi191 ceph-mon[32884]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:14:28.931 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:14:28 smithi183 ceph-mon[29736]: pgmap v164: 148 pgs: 3 active+undersized+degraded+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 142 active+clean; 6.0 GiB data, 18 GiB used, 607 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.26k op/s; 27213/295281 objects degraded (9.216%); 24 MiB/s, 389 objects/s recovering 2024-09-10T19:14:28.931 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:14:28 smithi183 ceph-mon[29736]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T19:14:28.931 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:14:28 smithi183 ceph-mon[29736]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-10T19:14:28.931 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:14:28 smithi183 ceph-mon[29736]: osdmap e123: 8 total, 7 up, 7 in 2024-09-10T19:14:28.932 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:14:28 smithi183 ceph-mon[29736]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:14:28.932 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:14:28 smithi183 ceph-mon[41562]: pgmap v164: 148 pgs: 3 active+undersized+degraded+remapped+backfill_toofull, 3 active+undersized+degraded+remapped+backfilling, 142 active+clean; 6.0 GiB data, 18 GiB used, 607 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.26k op/s; 27213/295281 objects degraded (9.216%); 24 MiB/s, 389 objects/s recovering 2024-09-10T19:14:28.932 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:14:28 smithi183 ceph-mon[41562]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T19:14:28.932 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:14:28 smithi183 ceph-mon[41562]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-10T19:14:28.932 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:14:28 smithi183 ceph-mon[41562]: osdmap e123: 8 total, 7 up, 7 in 2024-09-10T19:14:28.932 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:14:28 smithi183 ceph-mon[41562]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:14:28.932 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Sep 10 19:14:28 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-y[29989]: 2024-09-10T19:14:28.655+0000 7f9fad6cd640 -1 mgr get_metadata_python Requested missing service osd.1 2024-09-10T19:14:28.945 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:14:28.954 DEBUG:teuthology.orchestra.run.smithi191:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph daemon osd.7 dump_blocked_ops 2024-09-10T19:14:28.986 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:14:28.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:14:28.987 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:14:29.088 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:14:29.140 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:14:29.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:14:29.141 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:14:29.215 INFO:journalctl@ceph.iscsi.iscsi.a.smithi191.stdout:Sep 10 19:14:29 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-iscsi-iscsi-a[61816]: debug there is no tcmu-runner data available 2024-09-10T19:14:29.242 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:14:29.296 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:14:29.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:14:29.297 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:14:29.398 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:14:29.404 INFO:teuthology.orchestra.run.smithi191.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.b/config 2024-09-10T19:14:29.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:14:29.454 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:14:29.454 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:14:29.555 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:14:29.598 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:14:29.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:14:29.599 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:14:29.700 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:14:29.727 INFO:teuthology.orchestra.run.smithi183.stdout:osd.0: {} 2024-09-10T19:14:29.728 INFO:teuthology.orchestra.run.smithi183.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-10T19:14:29.735 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:14:29.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:14:29.737 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:14:29.741 INFO:teuthology.orchestra.run.smithi183.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-09-10T19:14:29.780 INFO:teuthology.orchestra.run.smithi183.stdout:osd.2: {} 2024-09-10T19:14:29.781 INFO:teuthology.orchestra.run.smithi183.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-10T19:14:29.826 INFO:teuthology.orchestra.run.smithi183.stdout:osd.3: {} 2024-09-10T19:14:29.827 INFO:teuthology.orchestra.run.smithi183.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-10T19:14:29.837 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:14:29.860 INFO:teuthology.orchestra.run.smithi183.stdout:osd.4: {} 2024-09-10T19:14:29.860 INFO:teuthology.orchestra.run.smithi183.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-10T19:14:29.868 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:14:29.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:14:29.869 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:14:29.894 INFO:teuthology.orchestra.run.smithi183.stdout:osd.5: {} 2024-09-10T19:14:29.894 INFO:teuthology.orchestra.run.smithi183.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-10T19:14:29.927 INFO:teuthology.orchestra.run.smithi183.stdout:osd.6: {} 2024-09-10T19:14:29.928 INFO:teuthology.orchestra.run.smithi183.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-10T19:14:29.928 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:14:29 smithi183 ceph-mon[29736]: from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:14:29.928 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:14:29 smithi183 ceph-mon[29736]: from='client.? 172.21.15.183:0/4204778076' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T19:14:29.928 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:14:29 smithi183 ceph-mon[29736]: Health check update: Degraded data redundancy: 27213/295281 objects degraded (9.216%), 6 pgs degraded (PG_DEGRADED) 2024-09-10T19:14:29.928 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:14:29 smithi183 ceph-mon[41562]: from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:14:29.928 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:14:29 smithi183 ceph-mon[41562]: from='client.? 172.21.15.183:0/4204778076' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T19:14:29.928 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:14:29 smithi183 ceph-mon[41562]: Health check update: Degraded data redundancy: 27213/295281 objects degraded (9.216%), 6 pgs degraded (PG_DEGRADED) 2024-09-10T19:14:29.957 INFO:teuthology.orchestra.run.smithi183.stdout:osd.7: {} 2024-09-10T19:14:29.957 INFO:teuthology.orchestra.run.smithi183.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-10T19:14:29.969 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:14:30.006 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:14:30.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:14:30.007 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:14:30.089 INFO:tasks.thrashosds.thrasher:in_osds: [0, 2, 3, 4, 5, 6, 7] out_osds: [1] dead_osds: [1] live_osds: [0, 2, 3, 4, 5, 6, 7] 2024-09-10T19:14:30.089 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-10T19:14:30.089 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap_items 2024-09-10T19:14:30.089 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph osd dump -f json-pretty 2024-09-10T19:14:30.090 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:14:29 smithi191 ceph-mon[32884]: from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:14:30.091 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:14:29 smithi191 ceph-mon[32884]: from='client.? 172.21.15.183:0/4204778076' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T19:14:30.091 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:14:29 smithi191 ceph-mon[32884]: Health check update: Degraded data redundancy: 27213/295281 objects degraded (9.216%), 6 pgs degraded (PG_DEGRADED) 2024-09-10T19:14:30.108 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:14:30.146 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:14:30.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:14:30.147 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:14:30.248 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:14:30.296 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:14:30.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:14:30.297 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:14:30.398 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:14:30.429 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:14:30.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:14:30.430 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:14:30.531 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:14:30.570 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:14:30.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:14:30.571 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:14:30.653 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-10T19:14:30.654 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-10T19:14:30.671 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:14:30.718 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:14:30.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:14:30.719 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:14:30.764 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:14:30.821 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:14:30.851 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:14:30.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:14:30.853 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:14:30.899 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:14:30 smithi183 ceph-mon[29736]: pgmap v166: 148 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 5 active+undersized+degraded+remapped+backfilling, 142 active+clean; 5.8 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.26k op/s; 24745/287451 objects degraded (8.608%); 30 MiB/s, 483 objects/s recovering 2024-09-10T19:14:30.900 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:14:30 smithi183 ceph-mon[29736]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-09-10T19:14:30.900 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:14:30 smithi183 ceph-mon[41562]: pgmap v166: 148 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 5 active+undersized+degraded+remapped+backfilling, 142 active+clean; 5.8 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.26k op/s; 24745/287451 objects degraded (8.608%); 30 MiB/s, 483 objects/s recovering 2024-09-10T19:14:30.900 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:14:30 smithi183 ceph-mon[41562]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-09-10T19:14:30.953 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:14:30.989 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:14:30.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:14:30.990 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:14:31.091 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:14:31.100 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-10T19:14:31.124 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:14:31.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:14:31.125 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:14:31.172 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:14:30 smithi191 ceph-mon[32884]: pgmap v166: 148 pgs: 1 active+undersized+degraded+remapped+backfill_toofull, 5 active+undersized+degraded+remapped+backfilling, 142 active+clean; 5.8 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.26k op/s; 24745/287451 objects degraded (8.608%); 30 MiB/s, 483 objects/s recovering 2024-09-10T19:14:31.173 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:14:30 smithi191 ceph-mon[32884]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-09-10T19:14:31.226 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:14:31.263 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:14:31.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:14:31.265 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:14:31.366 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:14:31.423 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Sep 10 19:14:31 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-y[29989]: ::ffff:172.21.15.191 - - [10/Sep/2024:19:14:31] "GET /metrics HTTP/1.1" 200 209087 "" "Prometheus/2.43.0" 2024-09-10T19:14:31.439 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:14:31.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:14:31.440 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:14:31.541 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:14:31.588 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:14:31.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:14:31.589 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:14:31.690 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:14:31.731 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:14:31.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:14:31.732 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:14:31.812 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:14:31 smithi191 ceph-mon[32884]: 6.e scrub starts 2024-09-10T19:14:31.833 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:14:31.862 INFO:teuthology.orchestra.run.smithi191.stdout:ERROR: (22) Invalid argument 2024-09-10T19:14:31.862 INFO:teuthology.orchestra.run.smithi191.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-10T19:14:31.867 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:14:31.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:14:31.869 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:14:31.969 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:14:31.999 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:14:32.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:14:32.000 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:14:32.100 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:14:31 smithi183 ceph-mon[29736]: 6.e scrub starts 2024-09-10T19:14:32.100 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:14:31 smithi183 ceph-mon[41562]: 6.e scrub starts 2024-09-10T19:14:32.101 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:14:32.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:14:32.135 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:14:32.135 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:14:32.236 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:14:32.267 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:14:32.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:14:32.268 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:14:32.369 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:14:32.400 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:14:32.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:14:32.401 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:14:32.457 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:14:32.502 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:14:32.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:14:32.532 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:14:32.532 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:14:32.633 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:14:32.655 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-10T19:14:32.667 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:14:32.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:14:32.668 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:14:32.691 DEBUG:teuthology.orchestra.run.smithi191:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph daemon osd.7 dump_historic_ops 2024-09-10T19:14:32.769 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:14:32.804 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:14:32.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:14:32.805 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:14:32.906 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:14:32.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:14:32.962 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:14:32.962 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:14:32.965 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:14:32 smithi191 ceph-mon[32884]: pgmap v167: 148 pgs: 6 active+undersized+degraded+remapped+backfilling, 142 active+clean; 5.6 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.24k op/s; 22387/277617 objects degraded (8.064%); 33 MiB/s, 531 objects/s recovering 2024-09-10T19:14:32.965 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:14:32 smithi191 ceph-mon[32884]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull) 2024-09-10T19:14:33.021 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:14:32 smithi183 ceph-mon[29736]: pgmap v167: 148 pgs: 6 active+undersized+degraded+remapped+backfilling, 142 active+clean; 5.6 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.24k op/s; 22387/277617 objects degraded (8.064%); 33 MiB/s, 531 objects/s recovering 2024-09-10T19:14:33.021 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:14:32 smithi183 ceph-mon[29736]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull) 2024-09-10T19:14:33.021 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:14:32 smithi183 ceph-mon[41562]: pgmap v167: 148 pgs: 6 active+undersized+degraded+remapped+backfilling, 142 active+clean; 5.6 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.24k op/s; 22387/277617 objects degraded (8.064%); 33 MiB/s, 531 objects/s recovering 2024-09-10T19:14:33.022 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:14:32 smithi183 ceph-mon[41562]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull) 2024-09-10T19:14:33.063 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:14:33.112 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:14:33.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:14:33.116 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:14:33.128 INFO:teuthology.orchestra.run.smithi191.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.b/config 2024-09-10T19:14:33.218 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:14:33.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:14:33.250 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:14:33.251 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:14:33.352 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:14:33.396 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:14:33.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:14:33.397 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:14:33.498 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:14:33.520 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:14:33.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:14:33.521 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:14:33.622 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:14:33.652 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:14:33.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:14:33.654 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:14:33.754 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:14:33.786 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:14:33.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:14:33.787 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:14:33.887 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:14:33.929 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:14:33.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:14:33.931 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:14:34.031 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:14:34.057 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:14:34.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:14:34.058 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:14:34.159 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:14:34.197 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:14:34.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:14:34.198 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:14:34.299 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:14:34.329 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:14:34.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:14:34.330 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:14:34.431 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:14:34.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:14:34.456 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:14:34.456 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:14:34.558 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:14:34.595 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:14:34.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:14:34.596 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:14:34.697 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:14:34.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:14:34.731 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:14:34.732 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:14:34.833 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:14:34.873 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:14:34.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:14:34.874 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:14:34.965 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:14:34 smithi191 ceph-mon[32884]: pgmap v168: 148 pgs: 6 active+undersized+degraded+remapped+backfilling, 142 active+clean; 5.6 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.24k op/s; 22387/277617 objects degraded (8.064%); 33 MiB/s, 531 objects/s recovering 2024-09-10T19:14:34.965 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:14:34 smithi191 ceph-mon[32884]: Health check update: Degraded data redundancy: 22387/277617 objects degraded (8.064%), 6 pgs degraded (PG_DEGRADED) 2024-09-10T19:14:34.974 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:14:35.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:14:35.015 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:14:35.015 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:14:35.116 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:14:35.134 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:14:34 smithi183 ceph-mon[29736]: pgmap v168: 148 pgs: 6 active+undersized+degraded+remapped+backfilling, 142 active+clean; 5.6 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.24k op/s; 22387/277617 objects degraded (8.064%); 33 MiB/s, 531 objects/s recovering 2024-09-10T19:14:35.135 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:14:34 smithi183 ceph-mon[29736]: Health check update: Degraded data redundancy: 22387/277617 objects degraded (8.064%), 6 pgs degraded (PG_DEGRADED) 2024-09-10T19:14:35.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:14:34 smithi183 ceph-mon[41562]: pgmap v168: 148 pgs: 6 active+undersized+degraded+remapped+backfilling, 142 active+clean; 5.6 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.24k op/s; 22387/277617 objects degraded (8.064%); 33 MiB/s, 531 objects/s recovering 2024-09-10T19:14:35.135 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:14:34 smithi183 ceph-mon[41562]: Health check update: Degraded data redundancy: 22387/277617 objects degraded (8.064%), 6 pgs degraded (PG_DEGRADED) 2024-09-10T19:14:35.155 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:14:35.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:14:35.156 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:14:35.257 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:14:35.331 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:14:35.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:14:35.332 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:14:35.433 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:14:35.471 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:14:35.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:14:35.472 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:14:35.573 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:14:35.609 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:14:35.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:14:35.610 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:14:35.711 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:14:35.744 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:14:35.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:14:35.745 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:14:35.846 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:14:35.878 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:14:35.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:14:35.879 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:14:35.980 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:14:36.017 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:14:36.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:14:36.018 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:14:36.119 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:14:36.159 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:14:36.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:14:36.160 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:14:36.261 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:14:36.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:14:36.297 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:14:36.297 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:14:36.398 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:14:36.421 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:14:36.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:14:36.422 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:14:36.523 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:14:36.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:14:36.578 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:14:36.578 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:14:36.679 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:14:36.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:14:36.706 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:14:36.706 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:14:36.807 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:14:36.834 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:14:36.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:14:36.835 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:14:36.936 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:14:36.967 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:14:36.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:14:36.969 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:14:37.069 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:14:37.103 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:14:37.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:14:37.104 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:14:37.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:14:36 smithi183 ceph-mon[29736]: 5.18 scrub starts 2024-09-10T19:14:37.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:14:36 smithi183 ceph-mon[29736]: pgmap v169: 148 pgs: 6 active+undersized+degraded+remapped+backfilling, 142 active+clean; 5.3 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.16k op/s; 17230/260613 objects degraded (6.611%); 45 MiB/s, 725 objects/s recovering 2024-09-10T19:14:37.181 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:14:36 smithi183 ceph-mon[41562]: 5.18 scrub starts 2024-09-10T19:14:37.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:14:36 smithi183 ceph-mon[41562]: pgmap v169: 148 pgs: 6 active+undersized+degraded+remapped+backfilling, 142 active+clean; 5.3 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.16k op/s; 17230/260613 objects degraded (6.611%); 45 MiB/s, 725 objects/s recovering 2024-09-10T19:14:37.205 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:14:37.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:14:36 smithi191 ceph-mon[32884]: 5.18 scrub starts 2024-09-10T19:14:37.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:14:36 smithi191 ceph-mon[32884]: pgmap v169: 148 pgs: 6 active+undersized+degraded+remapped+backfilling, 142 active+clean; 5.3 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.16k op/s; 17230/260613 objects degraded (6.611%); 45 MiB/s, 725 objects/s recovering 2024-09-10T19:14:37.248 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:14:37.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:14:37.249 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:14:37.350 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:14:37.374 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:14:37.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:14:37.375 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:14:37.476 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:14:37.506 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:14:37.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:14:37.507 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:14:37.608 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:14:37.629 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:14:37.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:14:37.630 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:14:37.715 INFO:journalctl@ceph.mgr.x.smithi191.stdout:Sep 10 19:14:37 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-x[34967]: ::ffff:172.21.15.191 - - [10/Sep/2024:19:14:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T19:14:37.732 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:14:37.761 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:14:37.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:14:37.762 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:14:37.863 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:14:37.894 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:14:37.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:14:37.895 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:14:37.996 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:14:38.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:14:38.019 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:14:38.019 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:14:38.120 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:14:38.153 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:14:38.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:14:38.154 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:14:38.255 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:14:38.286 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:14:38.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:14:38.287 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:14:38.388 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:14:38.410 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:14:38.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:14:38.410 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:14:38.511 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:14:38.544 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:14:38.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:14:38.545 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:14:38.647 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-09-10T19:14:38.647 INFO:teuthology.orchestra.run.smithi183.stdout:{ 2024-09-10T19:14:38.647 INFO:teuthology.orchestra.run.smithi183.stdout: "epoch": 123, 2024-09-10T19:14:38.647 INFO:teuthology.orchestra.run.smithi183.stdout: "fsid": "320f6df8-6fa7-11ef-bcea-c7b262605968", 2024-09-10T19:14:38.647 INFO:teuthology.orchestra.run.smithi183.stdout: "created": "2024-09-10T19:05:03.688531+0000", 2024-09-10T19:14:38.647 INFO:teuthology.orchestra.run.smithi183.stdout: "modified": "2024-09-10T19:14:27.708730+0000", 2024-09-10T19:14:38.647 INFO:teuthology.orchestra.run.smithi183.stdout: "last_up_change": "2024-09-10T19:12:45.801705+0000", 2024-09-10T19:14:38.647 INFO:teuthology.orchestra.run.smithi183.stdout: "last_in_change": "2024-09-10T19:13:53.915409+0000", 2024-09-10T19:14:38.647 INFO:teuthology.orchestra.run.smithi183.stdout: "flags": "nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-09-10T19:14:38.647 INFO:teuthology.orchestra.run.smithi183.stdout: "flags_num": 5804032, 2024-09-10T19:14:38.647 INFO:teuthology.orchestra.run.smithi183.stdout: "flags_set": [ 2024-09-10T19:14:38.648 INFO:teuthology.orchestra.run.smithi183.stdout: "nodeep-scrub", 2024-09-10T19:14:38.648 INFO:teuthology.orchestra.run.smithi183.stdout: "pglog_hardlimit", 2024-09-10T19:14:38.648 INFO:teuthology.orchestra.run.smithi183.stdout: "purged_snapdirs", 2024-09-10T19:14:38.648 INFO:teuthology.orchestra.run.smithi183.stdout: "recovery_deletes", 2024-09-10T19:14:38.648 INFO:teuthology.orchestra.run.smithi183.stdout: "sortbitwise" 2024-09-10T19:14:38.648 INFO:teuthology.orchestra.run.smithi183.stdout: ], 2024-09-10T19:14:38.648 INFO:teuthology.orchestra.run.smithi183.stdout: "crush_version": 18, 2024-09-10T19:14:38.648 INFO:teuthology.orchestra.run.smithi183.stdout: "full_ratio": 0.94999998807907104, 2024-09-10T19:14:38.648 INFO:teuthology.orchestra.run.smithi183.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-09-10T19:14:38.648 INFO:teuthology.orchestra.run.smithi183.stdout: "nearfull_ratio": 0.85000002384185791, 2024-09-10T19:14:38.648 INFO:teuthology.orchestra.run.smithi183.stdout: "cluster_snapshot": "", 2024-09-10T19:14:38.648 INFO:teuthology.orchestra.run.smithi183.stdout: "pool_max": 7, 2024-09-10T19:14:38.648 INFO:teuthology.orchestra.run.smithi183.stdout: "max_osd": 8, 2024-09-10T19:14:38.648 INFO:teuthology.orchestra.run.smithi183.stdout: "require_min_compat_client": "luminous", 2024-09-10T19:14:38.649 INFO:teuthology.orchestra.run.smithi183.stdout: "min_compat_client": "luminous", 2024-09-10T19:14:38.649 INFO:teuthology.orchestra.run.smithi183.stdout: "require_osd_release": "quincy", 2024-09-10T19:14:38.649 INFO:teuthology.orchestra.run.smithi183.stdout: "pools": [ 2024-09-10T19:14:38.649 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:14:38.649 INFO:teuthology.orchestra.run.smithi183.stdout: "pool": 1, 2024-09-10T19:14:38.649 INFO:teuthology.orchestra.run.smithi183.stdout: "pool_name": ".mgr", 2024-09-10T19:14:38.649 INFO:teuthology.orchestra.run.smithi183.stdout: "create_time": "2024-09-10T19:08:00.456786+0000", 2024-09-10T19:14:38.649 INFO:teuthology.orchestra.run.smithi183.stdout: "flags": 1, 2024-09-10T19:14:38.649 INFO:teuthology.orchestra.run.smithi183.stdout: "flags_names": "hashpspool", 2024-09-10T19:14:38.649 INFO:teuthology.orchestra.run.smithi183.stdout: "type": 1, 2024-09-10T19:14:38.649 INFO:teuthology.orchestra.run.smithi183.stdout: "size": 3, 2024-09-10T19:14:38.649 INFO:teuthology.orchestra.run.smithi183.stdout: "min_size": 2, 2024-09-10T19:14:38.649 INFO:teuthology.orchestra.run.smithi183.stdout: "crush_rule": 0, 2024-09-10T19:14:38.649 INFO:teuthology.orchestra.run.smithi183.stdout: "peering_crush_bucket_count": 0, 2024-09-10T19:14:38.650 INFO:teuthology.orchestra.run.smithi183.stdout: "peering_crush_bucket_target": 0, 2024-09-10T19:14:38.650 INFO:teuthology.orchestra.run.smithi183.stdout: "peering_crush_bucket_barrier": 0, 2024-09-10T19:14:38.650 INFO:teuthology.orchestra.run.smithi183.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-10T19:14:38.650 INFO:teuthology.orchestra.run.smithi183.stdout: "object_hash": 2, 2024-09-10T19:14:38.650 INFO:teuthology.orchestra.run.smithi183.stdout: "pg_autoscale_mode": "off", 2024-09-10T19:14:38.650 INFO:teuthology.orchestra.run.smithi183.stdout: "pg_num": 1, 2024-09-10T19:14:38.650 INFO:teuthology.orchestra.run.smithi183.stdout: "pg_placement_num": 1, 2024-09-10T19:14:38.650 INFO:teuthology.orchestra.run.smithi183.stdout: "pg_placement_num_target": 1, 2024-09-10T19:14:38.650 INFO:teuthology.orchestra.run.smithi183.stdout: "pg_num_target": 1, 2024-09-10T19:14:38.650 INFO:teuthology.orchestra.run.smithi183.stdout: "pg_num_pending": 1, 2024-09-10T19:14:38.650 INFO:teuthology.orchestra.run.smithi183.stdout: "last_pg_merge_meta": { 2024-09-10T19:14:38.650 INFO:teuthology.orchestra.run.smithi183.stdout: "source_pgid": "0.0", 2024-09-10T19:14:38.650 INFO:teuthology.orchestra.run.smithi183.stdout: "ready_epoch": 0, 2024-09-10T19:14:38.650 INFO:teuthology.orchestra.run.smithi183.stdout: "last_epoch_started": 0, 2024-09-10T19:14:38.650 INFO:teuthology.orchestra.run.smithi183.stdout: "last_epoch_clean": 0, 2024-09-10T19:14:38.651 INFO:teuthology.orchestra.run.smithi183.stdout: "source_version": "0'0", 2024-09-10T19:14:38.651 INFO:teuthology.orchestra.run.smithi183.stdout: "target_version": "0'0" 2024-09-10T19:14:38.651 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:14:38.651 INFO:teuthology.orchestra.run.smithi183.stdout: "last_change": "22", 2024-09-10T19:14:38.651 INFO:teuthology.orchestra.run.smithi183.stdout: "last_force_op_resend": "0", 2024-09-10T19:14:38.651 INFO:teuthology.orchestra.run.smithi183.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-10T19:14:38.651 INFO:teuthology.orchestra.run.smithi183.stdout: "last_force_op_resend_preluminous": "0", 2024-09-10T19:14:38.651 INFO:teuthology.orchestra.run.smithi183.stdout: "auid": 0, 2024-09-10T19:14:38.651 INFO:teuthology.orchestra.run.smithi183.stdout: "snap_mode": "selfmanaged", 2024-09-10T19:14:38.651 INFO:teuthology.orchestra.run.smithi183.stdout: "snap_seq": 0, 2024-09-10T19:14:38.651 INFO:teuthology.orchestra.run.smithi183.stdout: "snap_epoch": 0, 2024-09-10T19:14:38.651 INFO:teuthology.orchestra.run.smithi183.stdout: "pool_snaps": [], 2024-09-10T19:14:38.651 INFO:teuthology.orchestra.run.smithi183.stdout: "removed_snaps": "[]", 2024-09-10T19:14:38.652 INFO:teuthology.orchestra.run.smithi183.stdout: "quota_max_bytes": 0, 2024-09-10T19:14:38.652 INFO:teuthology.orchestra.run.smithi183.stdout: "quota_max_objects": 0, 2024-09-10T19:14:38.652 INFO:teuthology.orchestra.run.smithi183.stdout: "tiers": [], 2024-09-10T19:14:38.652 INFO:teuthology.orchestra.run.smithi183.stdout: "tier_of": -1, 2024-09-10T19:14:38.652 INFO:teuthology.orchestra.run.smithi183.stdout: "read_tier": -1, 2024-09-10T19:14:38.652 INFO:teuthology.orchestra.run.smithi183.stdout: "write_tier": -1, 2024-09-10T19:14:38.652 INFO:teuthology.orchestra.run.smithi183.stdout: "cache_mode": "none", 2024-09-10T19:14:38.652 INFO:teuthology.orchestra.run.smithi183.stdout: "target_max_bytes": 0, 2024-09-10T19:14:38.652 INFO:teuthology.orchestra.run.smithi183.stdout: "target_max_objects": 0, 2024-09-10T19:14:38.652 INFO:teuthology.orchestra.run.smithi183.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-10T19:14:38.652 INFO:teuthology.orchestra.run.smithi183.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-10T19:14:38.652 INFO:teuthology.orchestra.run.smithi183.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-10T19:14:38.652 INFO:teuthology.orchestra.run.smithi183.stdout: "cache_min_flush_age": 0, 2024-09-10T19:14:38.652 INFO:teuthology.orchestra.run.smithi183.stdout: "cache_min_evict_age": 0, 2024-09-10T19:14:38.653 INFO:teuthology.orchestra.run.smithi183.stdout: "erasure_code_profile": "", 2024-09-10T19:14:38.653 INFO:teuthology.orchestra.run.smithi183.stdout: "hit_set_params": { 2024-09-10T19:14:38.653 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "none" 2024-09-10T19:14:38.653 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:14:38.653 INFO:teuthology.orchestra.run.smithi183.stdout: "hit_set_period": 0, 2024-09-10T19:14:38.653 INFO:teuthology.orchestra.run.smithi183.stdout: "hit_set_count": 0, 2024-09-10T19:14:38.653 INFO:teuthology.orchestra.run.smithi183.stdout: "use_gmt_hitset": true, 2024-09-10T19:14:38.653 INFO:teuthology.orchestra.run.smithi183.stdout: "min_read_recency_for_promote": 0, 2024-09-10T19:14:38.653 INFO:teuthology.orchestra.run.smithi183.stdout: "min_write_recency_for_promote": 0, 2024-09-10T19:14:38.653 INFO:teuthology.orchestra.run.smithi183.stdout: "hit_set_grade_decay_rate": 0, 2024-09-10T19:14:38.653 INFO:teuthology.orchestra.run.smithi183.stdout: "hit_set_search_last_n": 0, 2024-09-10T19:14:38.653 INFO:teuthology.orchestra.run.smithi183.stdout: "grade_table": [], 2024-09-10T19:14:38.653 INFO:teuthology.orchestra.run.smithi183.stdout: "stripe_width": 0, 2024-09-10T19:14:38.654 INFO:teuthology.orchestra.run.smithi183.stdout: "expected_num_objects": 0, 2024-09-10T19:14:38.654 INFO:teuthology.orchestra.run.smithi183.stdout: "fast_read": false, 2024-09-10T19:14:38.654 INFO:teuthology.orchestra.run.smithi183.stdout: "options": { 2024-09-10T19:14:38.654 INFO:teuthology.orchestra.run.smithi183.stdout: "pg_num_max": 32, 2024-09-10T19:14:38.654 INFO:teuthology.orchestra.run.smithi183.stdout: "pg_num_min": 1 2024-09-10T19:14:38.654 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:14:38.654 INFO:teuthology.orchestra.run.smithi183.stdout: "application_metadata": { 2024-09-10T19:14:38.654 INFO:teuthology.orchestra.run.smithi183.stdout: "mgr": {} 2024-09-10T19:14:38.654 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:14:38.654 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:14:38.654 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:14:38.654 INFO:teuthology.orchestra.run.smithi183.stdout: "pool": 2, 2024-09-10T19:14:38.654 INFO:teuthology.orchestra.run.smithi183.stdout: "pool_name": ".rgw.root", 2024-09-10T19:14:38.654 INFO:teuthology.orchestra.run.smithi183.stdout: "create_time": "2024-09-10T19:10:05.993763+0000", 2024-09-10T19:14:38.654 INFO:teuthology.orchestra.run.smithi183.stdout: "flags": 1, 2024-09-10T19:14:38.655 INFO:teuthology.orchestra.run.smithi183.stdout: "flags_names": "hashpspool", 2024-09-10T19:14:38.655 INFO:teuthology.orchestra.run.smithi183.stdout: "type": 1, 2024-09-10T19:14:38.655 INFO:teuthology.orchestra.run.smithi183.stdout: "size": 3, 2024-09-10T19:14:38.655 INFO:teuthology.orchestra.run.smithi183.stdout: "min_size": 2, 2024-09-10T19:14:38.655 INFO:teuthology.orchestra.run.smithi183.stdout: "crush_rule": 0, 2024-09-10T19:14:38.655 INFO:teuthology.orchestra.run.smithi183.stdout: "peering_crush_bucket_count": 0, 2024-09-10T19:14:38.655 INFO:teuthology.orchestra.run.smithi183.stdout: "peering_crush_bucket_target": 0, 2024-09-10T19:14:38.655 INFO:teuthology.orchestra.run.smithi183.stdout: "peering_crush_bucket_barrier": 0, 2024-09-10T19:14:38.655 INFO:teuthology.orchestra.run.smithi183.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-10T19:14:38.655 INFO:teuthology.orchestra.run.smithi183.stdout: "object_hash": 2, 2024-09-10T19:14:38.655 INFO:teuthology.orchestra.run.smithi183.stdout: "pg_autoscale_mode": "off", 2024-09-10T19:14:38.655 INFO:teuthology.orchestra.run.smithi183.stdout: "pg_num": 32, 2024-09-10T19:14:38.655 INFO:teuthology.orchestra.run.smithi183.stdout: "pg_placement_num": 32, 2024-09-10T19:14:38.655 INFO:teuthology.orchestra.run.smithi183.stdout: "pg_placement_num_target": 32, 2024-09-10T19:14:38.656 INFO:teuthology.orchestra.run.smithi183.stdout: "pg_num_target": 32, 2024-09-10T19:14:38.656 INFO:teuthology.orchestra.run.smithi183.stdout: "pg_num_pending": 32, 2024-09-10T19:14:38.656 INFO:teuthology.orchestra.run.smithi183.stdout: "last_pg_merge_meta": { 2024-09-10T19:14:38.656 INFO:teuthology.orchestra.run.smithi183.stdout: "source_pgid": "0.0", 2024-09-10T19:14:38.656 INFO:teuthology.orchestra.run.smithi183.stdout: "ready_epoch": 0, 2024-09-10T19:14:38.656 INFO:teuthology.orchestra.run.smithi183.stdout: "last_epoch_started": 0, 2024-09-10T19:14:38.656 INFO:teuthology.orchestra.run.smithi183.stdout: "last_epoch_clean": 0, 2024-09-10T19:14:38.656 INFO:teuthology.orchestra.run.smithi183.stdout: "source_version": "0'0", 2024-09-10T19:14:38.656 INFO:teuthology.orchestra.run.smithi183.stdout: "target_version": "0'0" 2024-09-10T19:14:38.656 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:14:38.656 INFO:teuthology.orchestra.run.smithi183.stdout: "last_change": "60", 2024-09-10T19:14:38.656 INFO:teuthology.orchestra.run.smithi183.stdout: "last_force_op_resend": "0", 2024-09-10T19:14:38.656 INFO:teuthology.orchestra.run.smithi183.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-10T19:14:38.657 INFO:teuthology.orchestra.run.smithi183.stdout: "last_force_op_resend_preluminous": "0", 2024-09-10T19:14:38.657 INFO:teuthology.orchestra.run.smithi183.stdout: "auid": 0, 2024-09-10T19:14:38.657 INFO:teuthology.orchestra.run.smithi183.stdout: "snap_mode": "selfmanaged", 2024-09-10T19:14:38.657 INFO:teuthology.orchestra.run.smithi183.stdout: "snap_seq": 0, 2024-09-10T19:14:38.657 INFO:teuthology.orchestra.run.smithi183.stdout: "snap_epoch": 0, 2024-09-10T19:14:38.657 INFO:teuthology.orchestra.run.smithi183.stdout: "pool_snaps": [], 2024-09-10T19:14:38.657 INFO:teuthology.orchestra.run.smithi183.stdout: "removed_snaps": "[]", 2024-09-10T19:14:38.657 INFO:teuthology.orchestra.run.smithi183.stdout: "quota_max_bytes": 0, 2024-09-10T19:14:38.657 INFO:teuthology.orchestra.run.smithi183.stdout: "quota_max_objects": 0, 2024-09-10T19:14:38.657 INFO:teuthology.orchestra.run.smithi183.stdout: "tiers": [], 2024-09-10T19:14:38.657 INFO:teuthology.orchestra.run.smithi183.stdout: "tier_of": -1, 2024-09-10T19:14:38.657 INFO:teuthology.orchestra.run.smithi183.stdout: "read_tier": -1, 2024-09-10T19:14:38.657 INFO:teuthology.orchestra.run.smithi183.stdout: "write_tier": -1, 2024-09-10T19:14:38.657 INFO:teuthology.orchestra.run.smithi183.stdout: "cache_mode": "none", 2024-09-10T19:14:38.657 INFO:teuthology.orchestra.run.smithi183.stdout: "target_max_bytes": 0, 2024-09-10T19:14:38.658 INFO:teuthology.orchestra.run.smithi183.stdout: "target_max_objects": 0, 2024-09-10T19:14:38.658 INFO:teuthology.orchestra.run.smithi183.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-10T19:14:38.658 INFO:teuthology.orchestra.run.smithi183.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-10T19:14:38.658 INFO:teuthology.orchestra.run.smithi183.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-10T19:14:38.658 INFO:teuthology.orchestra.run.smithi183.stdout: "cache_min_flush_age": 0, 2024-09-10T19:14:38.658 INFO:teuthology.orchestra.run.smithi183.stdout: "cache_min_evict_age": 0, 2024-09-10T19:14:38.658 INFO:teuthology.orchestra.run.smithi183.stdout: "erasure_code_profile": "", 2024-09-10T19:14:38.658 INFO:teuthology.orchestra.run.smithi183.stdout: "hit_set_params": { 2024-09-10T19:14:38.658 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "none" 2024-09-10T19:14:38.658 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:14:38.658 INFO:teuthology.orchestra.run.smithi183.stdout: "hit_set_period": 0, 2024-09-10T19:14:38.658 INFO:teuthology.orchestra.run.smithi183.stdout: "hit_set_count": 0, 2024-09-10T19:14:38.658 INFO:teuthology.orchestra.run.smithi183.stdout: "use_gmt_hitset": true, 2024-09-10T19:14:38.659 INFO:teuthology.orchestra.run.smithi183.stdout: "min_read_recency_for_promote": 0, 2024-09-10T19:14:38.659 INFO:teuthology.orchestra.run.smithi183.stdout: "min_write_recency_for_promote": 0, 2024-09-10T19:14:38.659 INFO:teuthology.orchestra.run.smithi183.stdout: "hit_set_grade_decay_rate": 0, 2024-09-10T19:14:38.659 INFO:teuthology.orchestra.run.smithi183.stdout: "hit_set_search_last_n": 0, 2024-09-10T19:14:38.659 INFO:teuthology.orchestra.run.smithi183.stdout: "grade_table": [], 2024-09-10T19:14:38.659 INFO:teuthology.orchestra.run.smithi183.stdout: "stripe_width": 0, 2024-09-10T19:14:38.659 INFO:teuthology.orchestra.run.smithi183.stdout: "expected_num_objects": 0, 2024-09-10T19:14:38.659 INFO:teuthology.orchestra.run.smithi183.stdout: "fast_read": false, 2024-09-10T19:14:38.659 INFO:teuthology.orchestra.run.smithi183.stdout: "options": {}, 2024-09-10T19:14:38.659 INFO:teuthology.orchestra.run.smithi183.stdout: "application_metadata": { 2024-09-10T19:14:38.659 INFO:teuthology.orchestra.run.smithi183.stdout: "rgw": {} 2024-09-10T19:14:38.659 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:14:38.659 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:14:38.659 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:14:38.659 INFO:teuthology.orchestra.run.smithi183.stdout: "pool": 3, 2024-09-10T19:14:38.659 INFO:teuthology.orchestra.run.smithi183.stdout: "pool_name": "datapool", 2024-09-10T19:14:38.660 INFO:teuthology.orchestra.run.smithi183.stdout: "create_time": "2024-09-10T19:10:06.224267+0000", 2024-09-10T19:14:38.660 INFO:teuthology.orchestra.run.smithi183.stdout: "flags": 8193, 2024-09-10T19:14:38.660 INFO:teuthology.orchestra.run.smithi183.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-09-10T19:14:38.660 INFO:teuthology.orchestra.run.smithi183.stdout: "type": 1, 2024-09-10T19:14:38.660 INFO:teuthology.orchestra.run.smithi183.stdout: "size": 3, 2024-09-10T19:14:38.660 INFO:teuthology.orchestra.run.smithi183.stdout: "min_size": 2, 2024-09-10T19:14:38.660 INFO:teuthology.orchestra.run.smithi183.stdout: "crush_rule": 0, 2024-09-10T19:14:38.660 INFO:teuthology.orchestra.run.smithi183.stdout: "peering_crush_bucket_count": 0, 2024-09-10T19:14:38.660 INFO:teuthology.orchestra.run.smithi183.stdout: "peering_crush_bucket_target": 0, 2024-09-10T19:14:38.660 INFO:teuthology.orchestra.run.smithi183.stdout: "peering_crush_bucket_barrier": 0, 2024-09-10T19:14:38.660 INFO:teuthology.orchestra.run.smithi183.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-10T19:14:38.660 INFO:teuthology.orchestra.run.smithi183.stdout: "object_hash": 2, 2024-09-10T19:14:38.660 INFO:teuthology.orchestra.run.smithi183.stdout: "pg_autoscale_mode": "off", 2024-09-10T19:14:38.660 INFO:teuthology.orchestra.run.smithi183.stdout: "pg_num": 3, 2024-09-10T19:14:38.661 INFO:teuthology.orchestra.run.smithi183.stdout: "pg_placement_num": 3, 2024-09-10T19:14:38.661 INFO:teuthology.orchestra.run.smithi183.stdout: "pg_placement_num_target": 3, 2024-09-10T19:14:38.661 INFO:teuthology.orchestra.run.smithi183.stdout: "pg_num_target": 3, 2024-09-10T19:14:38.661 INFO:teuthology.orchestra.run.smithi183.stdout: "pg_num_pending": 3, 2024-09-10T19:14:38.661 INFO:teuthology.orchestra.run.smithi183.stdout: "last_pg_merge_meta": { 2024-09-10T19:14:38.661 INFO:teuthology.orchestra.run.smithi183.stdout: "source_pgid": "0.0", 2024-09-10T19:14:38.661 INFO:teuthology.orchestra.run.smithi183.stdout: "ready_epoch": 0, 2024-09-10T19:14:38.661 INFO:teuthology.orchestra.run.smithi183.stdout: "last_epoch_started": 0, 2024-09-10T19:14:38.661 INFO:teuthology.orchestra.run.smithi183.stdout: "last_epoch_clean": 0, 2024-09-10T19:14:38.661 INFO:teuthology.orchestra.run.smithi183.stdout: "source_version": "0'0", 2024-09-10T19:14:38.661 INFO:teuthology.orchestra.run.smithi183.stdout: "target_version": "0'0" 2024-09-10T19:14:38.661 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:14:38.661 INFO:teuthology.orchestra.run.smithi183.stdout: "last_change": "63", 2024-09-10T19:14:38.661 INFO:teuthology.orchestra.run.smithi183.stdout: "last_force_op_resend": "0", 2024-09-10T19:14:38.661 INFO:teuthology.orchestra.run.smithi183.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-10T19:14:38.662 INFO:teuthology.orchestra.run.smithi183.stdout: "last_force_op_resend_preluminous": "0", 2024-09-10T19:14:38.662 INFO:teuthology.orchestra.run.smithi183.stdout: "auid": 0, 2024-09-10T19:14:38.662 INFO:teuthology.orchestra.run.smithi183.stdout: "snap_mode": "selfmanaged", 2024-09-10T19:14:38.662 INFO:teuthology.orchestra.run.smithi183.stdout: "snap_seq": 3, 2024-09-10T19:14:38.662 INFO:teuthology.orchestra.run.smithi183.stdout: "snap_epoch": 63, 2024-09-10T19:14:38.662 INFO:teuthology.orchestra.run.smithi183.stdout: "pool_snaps": [], 2024-09-10T19:14:38.662 INFO:teuthology.orchestra.run.smithi183.stdout: "removed_snaps": "[]", 2024-09-10T19:14:38.662 INFO:teuthology.orchestra.run.smithi183.stdout: "quota_max_bytes": 0, 2024-09-10T19:14:38.662 INFO:teuthology.orchestra.run.smithi183.stdout: "quota_max_objects": 0, 2024-09-10T19:14:38.662 INFO:teuthology.orchestra.run.smithi183.stdout: "tiers": [], 2024-09-10T19:14:38.662 INFO:teuthology.orchestra.run.smithi183.stdout: "tier_of": -1, 2024-09-10T19:14:38.662 INFO:teuthology.orchestra.run.smithi183.stdout: "read_tier": -1, 2024-09-10T19:14:38.662 INFO:teuthology.orchestra.run.smithi183.stdout: "write_tier": -1, 2024-09-10T19:14:38.662 INFO:teuthology.orchestra.run.smithi183.stdout: "cache_mode": "none", 2024-09-10T19:14:38.663 INFO:teuthology.orchestra.run.smithi183.stdout: "target_max_bytes": 0, 2024-09-10T19:14:38.663 INFO:teuthology.orchestra.run.smithi183.stdout: "target_max_objects": 0, 2024-09-10T19:14:38.663 INFO:teuthology.orchestra.run.smithi183.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-10T19:14:38.663 INFO:teuthology.orchestra.run.smithi183.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-10T19:14:38.663 INFO:teuthology.orchestra.run.smithi183.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-10T19:14:38.663 INFO:teuthology.orchestra.run.smithi183.stdout: "cache_min_flush_age": 0, 2024-09-10T19:14:38.663 INFO:teuthology.orchestra.run.smithi183.stdout: "cache_min_evict_age": 0, 2024-09-10T19:14:38.663 INFO:teuthology.orchestra.run.smithi183.stdout: "erasure_code_profile": "", 2024-09-10T19:14:38.663 INFO:teuthology.orchestra.run.smithi183.stdout: "hit_set_params": { 2024-09-10T19:14:38.663 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "none" 2024-09-10T19:14:38.663 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:14:38.663 INFO:teuthology.orchestra.run.smithi183.stdout: "hit_set_period": 0, 2024-09-10T19:14:38.663 INFO:teuthology.orchestra.run.smithi183.stdout: "hit_set_count": 0, 2024-09-10T19:14:38.663 INFO:teuthology.orchestra.run.smithi183.stdout: "use_gmt_hitset": true, 2024-09-10T19:14:38.664 INFO:teuthology.orchestra.run.smithi183.stdout: "min_read_recency_for_promote": 0, 2024-09-10T19:14:38.664 INFO:teuthology.orchestra.run.smithi183.stdout: "min_write_recency_for_promote": 0, 2024-09-10T19:14:38.664 INFO:teuthology.orchestra.run.smithi183.stdout: "hit_set_grade_decay_rate": 0, 2024-09-10T19:14:38.664 INFO:teuthology.orchestra.run.smithi183.stdout: "hit_set_search_last_n": 0, 2024-09-10T19:14:38.664 INFO:teuthology.orchestra.run.smithi183.stdout: "grade_table": [], 2024-09-10T19:14:38.664 INFO:teuthology.orchestra.run.smithi183.stdout: "stripe_width": 0, 2024-09-10T19:14:38.664 INFO:teuthology.orchestra.run.smithi183.stdout: "expected_num_objects": 0, 2024-09-10T19:14:38.664 INFO:teuthology.orchestra.run.smithi183.stdout: "fast_read": false, 2024-09-10T19:14:38.664 INFO:teuthology.orchestra.run.smithi183.stdout: "options": {}, 2024-09-10T19:14:38.664 INFO:teuthology.orchestra.run.smithi183.stdout: "application_metadata": { 2024-09-10T19:14:38.664 INFO:teuthology.orchestra.run.smithi183.stdout: "rbd": {} 2024-09-10T19:14:38.664 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:14:38.664 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:14:38.664 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:14:38.665 INFO:teuthology.orchestra.run.smithi183.stdout: "pool": 4, 2024-09-10T19:14:38.665 INFO:teuthology.orchestra.run.smithi183.stdout: "pool_name": "default.rgw.log", 2024-09-10T19:14:38.665 INFO:teuthology.orchestra.run.smithi183.stdout: "create_time": "2024-09-10T19:10:07.958284+0000", 2024-09-10T19:14:38.665 INFO:teuthology.orchestra.run.smithi183.stdout: "flags": 1, 2024-09-10T19:14:38.665 INFO:teuthology.orchestra.run.smithi183.stdout: "flags_names": "hashpspool", 2024-09-10T19:14:38.665 INFO:teuthology.orchestra.run.smithi183.stdout: "type": 1, 2024-09-10T19:14:38.665 INFO:teuthology.orchestra.run.smithi183.stdout: "size": 3, 2024-09-10T19:14:38.665 INFO:teuthology.orchestra.run.smithi183.stdout: "min_size": 2, 2024-09-10T19:14:38.665 INFO:teuthology.orchestra.run.smithi183.stdout: "crush_rule": 0, 2024-09-10T19:14:38.665 INFO:teuthology.orchestra.run.smithi183.stdout: "peering_crush_bucket_count": 0, 2024-09-10T19:14:38.665 INFO:teuthology.orchestra.run.smithi183.stdout: "peering_crush_bucket_target": 0, 2024-09-10T19:14:38.665 INFO:teuthology.orchestra.run.smithi183.stdout: "peering_crush_bucket_barrier": 0, 2024-09-10T19:14:38.665 INFO:teuthology.orchestra.run.smithi183.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-10T19:14:38.665 INFO:teuthology.orchestra.run.smithi183.stdout: "object_hash": 2, 2024-09-10T19:14:38.665 INFO:teuthology.orchestra.run.smithi183.stdout: "pg_autoscale_mode": "off", 2024-09-10T19:14:38.666 INFO:teuthology.orchestra.run.smithi183.stdout: "pg_num": 32, 2024-09-10T19:14:38.666 INFO:teuthology.orchestra.run.smithi183.stdout: "pg_placement_num": 32, 2024-09-10T19:14:38.666 INFO:teuthology.orchestra.run.smithi183.stdout: "pg_placement_num_target": 32, 2024-09-10T19:14:38.666 INFO:teuthology.orchestra.run.smithi183.stdout: "pg_num_target": 32, 2024-09-10T19:14:38.666 INFO:teuthology.orchestra.run.smithi183.stdout: "pg_num_pending": 32, 2024-09-10T19:14:38.666 INFO:teuthology.orchestra.run.smithi183.stdout: "last_pg_merge_meta": { 2024-09-10T19:14:38.666 INFO:teuthology.orchestra.run.smithi183.stdout: "source_pgid": "0.0", 2024-09-10T19:14:38.666 INFO:teuthology.orchestra.run.smithi183.stdout: "ready_epoch": 0, 2024-09-10T19:14:38.666 INFO:teuthology.orchestra.run.smithi183.stdout: "last_epoch_started": 0, 2024-09-10T19:14:38.666 INFO:teuthology.orchestra.run.smithi183.stdout: "last_epoch_clean": 0, 2024-09-10T19:14:38.666 INFO:teuthology.orchestra.run.smithi183.stdout: "source_version": "0'0", 2024-09-10T19:14:38.666 INFO:teuthology.orchestra.run.smithi183.stdout: "target_version": "0'0" 2024-09-10T19:14:38.666 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:14:38.667 INFO:teuthology.orchestra.run.smithi183.stdout: "last_change": "62", 2024-09-10T19:14:38.667 INFO:teuthology.orchestra.run.smithi183.stdout: "last_force_op_resend": "0", 2024-09-10T19:14:38.667 INFO:teuthology.orchestra.run.smithi183.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-10T19:14:38.667 INFO:teuthology.orchestra.run.smithi183.stdout: "last_force_op_resend_preluminous": "0", 2024-09-10T19:14:38.667 INFO:teuthology.orchestra.run.smithi183.stdout: "auid": 0, 2024-09-10T19:14:38.667 INFO:teuthology.orchestra.run.smithi183.stdout: "snap_mode": "selfmanaged", 2024-09-10T19:14:38.667 INFO:teuthology.orchestra.run.smithi183.stdout: "snap_seq": 0, 2024-09-10T19:14:38.667 INFO:teuthology.orchestra.run.smithi183.stdout: "snap_epoch": 0, 2024-09-10T19:14:38.667 INFO:teuthology.orchestra.run.smithi183.stdout: "pool_snaps": [], 2024-09-10T19:14:38.667 INFO:teuthology.orchestra.run.smithi183.stdout: "removed_snaps": "[]", 2024-09-10T19:14:38.667 INFO:teuthology.orchestra.run.smithi183.stdout: "quota_max_bytes": 0, 2024-09-10T19:14:38.667 INFO:teuthology.orchestra.run.smithi183.stdout: "quota_max_objects": 0, 2024-09-10T19:14:38.667 INFO:teuthology.orchestra.run.smithi183.stdout: "tiers": [], 2024-09-10T19:14:38.667 INFO:teuthology.orchestra.run.smithi183.stdout: "tier_of": -1, 2024-09-10T19:14:38.668 INFO:teuthology.orchestra.run.smithi183.stdout: "read_tier": -1, 2024-09-10T19:14:38.668 INFO:teuthology.orchestra.run.smithi183.stdout: "write_tier": -1, 2024-09-10T19:14:38.668 INFO:teuthology.orchestra.run.smithi183.stdout: "cache_mode": "none", 2024-09-10T19:14:38.668 INFO:teuthology.orchestra.run.smithi183.stdout: "target_max_bytes": 0, 2024-09-10T19:14:38.668 INFO:teuthology.orchestra.run.smithi183.stdout: "target_max_objects": 0, 2024-09-10T19:14:38.668 INFO:teuthology.orchestra.run.smithi183.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-10T19:14:38.668 INFO:teuthology.orchestra.run.smithi183.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-10T19:14:38.668 INFO:teuthology.orchestra.run.smithi183.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-10T19:14:38.668 INFO:teuthology.orchestra.run.smithi183.stdout: "cache_min_flush_age": 0, 2024-09-10T19:14:38.668 INFO:teuthology.orchestra.run.smithi183.stdout: "cache_min_evict_age": 0, 2024-09-10T19:14:38.668 INFO:teuthology.orchestra.run.smithi183.stdout: "erasure_code_profile": "", 2024-09-10T19:14:38.668 INFO:teuthology.orchestra.run.smithi183.stdout: "hit_set_params": { 2024-09-10T19:14:38.668 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "none" 2024-09-10T19:14:38.668 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:14:38.669 INFO:teuthology.orchestra.run.smithi183.stdout: "hit_set_period": 0, 2024-09-10T19:14:38.669 INFO:teuthology.orchestra.run.smithi183.stdout: "hit_set_count": 0, 2024-09-10T19:14:38.669 INFO:teuthology.orchestra.run.smithi183.stdout: "use_gmt_hitset": true, 2024-09-10T19:14:38.669 INFO:teuthology.orchestra.run.smithi183.stdout: "min_read_recency_for_promote": 0, 2024-09-10T19:14:38.669 INFO:teuthology.orchestra.run.smithi183.stdout: "min_write_recency_for_promote": 0, 2024-09-10T19:14:38.669 INFO:teuthology.orchestra.run.smithi183.stdout: "hit_set_grade_decay_rate": 0, 2024-09-10T19:14:38.669 INFO:teuthology.orchestra.run.smithi183.stdout: "hit_set_search_last_n": 0, 2024-09-10T19:14:38.669 INFO:teuthology.orchestra.run.smithi183.stdout: "grade_table": [], 2024-09-10T19:14:38.669 INFO:teuthology.orchestra.run.smithi183.stdout: "stripe_width": 0, 2024-09-10T19:14:38.669 INFO:teuthology.orchestra.run.smithi183.stdout: "expected_num_objects": 0, 2024-09-10T19:14:38.669 INFO:teuthology.orchestra.run.smithi183.stdout: "fast_read": false, 2024-09-10T19:14:38.669 INFO:teuthology.orchestra.run.smithi183.stdout: "options": {}, 2024-09-10T19:14:38.669 INFO:teuthology.orchestra.run.smithi183.stdout: "application_metadata": { 2024-09-10T19:14:38.669 INFO:teuthology.orchestra.run.smithi183.stdout: "rgw": {} 2024-09-10T19:14:38.670 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:14:38.670 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:14:38.670 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:14:38.670 INFO:teuthology.orchestra.run.smithi183.stdout: "pool": 5, 2024-09-10T19:14:38.670 INFO:teuthology.orchestra.run.smithi183.stdout: "pool_name": "default.rgw.control", 2024-09-10T19:14:38.670 INFO:teuthology.orchestra.run.smithi183.stdout: "create_time": "2024-09-10T19:10:09.933596+0000", 2024-09-10T19:14:38.670 INFO:teuthology.orchestra.run.smithi183.stdout: "flags": 1, 2024-09-10T19:14:38.670 INFO:teuthology.orchestra.run.smithi183.stdout: "flags_names": "hashpspool", 2024-09-10T19:14:38.670 INFO:teuthology.orchestra.run.smithi183.stdout: "type": 1, 2024-09-10T19:14:38.670 INFO:teuthology.orchestra.run.smithi183.stdout: "size": 3, 2024-09-10T19:14:38.670 INFO:teuthology.orchestra.run.smithi183.stdout: "min_size": 2, 2024-09-10T19:14:38.670 INFO:teuthology.orchestra.run.smithi183.stdout: "crush_rule": 0, 2024-09-10T19:14:38.670 INFO:teuthology.orchestra.run.smithi183.stdout: "peering_crush_bucket_count": 0, 2024-09-10T19:14:38.670 INFO:teuthology.orchestra.run.smithi183.stdout: "peering_crush_bucket_target": 0, 2024-09-10T19:14:38.671 INFO:teuthology.orchestra.run.smithi183.stdout: "peering_crush_bucket_barrier": 0, 2024-09-10T19:14:38.671 INFO:teuthology.orchestra.run.smithi183.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-10T19:14:38.671 INFO:teuthology.orchestra.run.smithi183.stdout: "object_hash": 2, 2024-09-10T19:14:38.671 INFO:teuthology.orchestra.run.smithi183.stdout: "pg_autoscale_mode": "off", 2024-09-10T19:14:38.671 INFO:teuthology.orchestra.run.smithi183.stdout: "pg_num": 32, 2024-09-10T19:14:38.671 INFO:teuthology.orchestra.run.smithi183.stdout: "pg_placement_num": 32, 2024-09-10T19:14:38.671 INFO:teuthology.orchestra.run.smithi183.stdout: "pg_placement_num_target": 32, 2024-09-10T19:14:38.671 INFO:teuthology.orchestra.run.smithi183.stdout: "pg_num_target": 32, 2024-09-10T19:14:38.671 INFO:teuthology.orchestra.run.smithi183.stdout: "pg_num_pending": 32, 2024-09-10T19:14:38.671 INFO:teuthology.orchestra.run.smithi183.stdout: "last_pg_merge_meta": { 2024-09-10T19:14:38.671 INFO:teuthology.orchestra.run.smithi183.stdout: "source_pgid": "0.0", 2024-09-10T19:14:38.671 INFO:teuthology.orchestra.run.smithi183.stdout: "ready_epoch": 0, 2024-09-10T19:14:38.671 INFO:teuthology.orchestra.run.smithi183.stdout: "last_epoch_started": 0, 2024-09-10T19:14:38.672 INFO:teuthology.orchestra.run.smithi183.stdout: "last_epoch_clean": 0, 2024-09-10T19:14:38.672 INFO:teuthology.orchestra.run.smithi183.stdout: "source_version": "0'0", 2024-09-10T19:14:38.672 INFO:teuthology.orchestra.run.smithi183.stdout: "target_version": "0'0" 2024-09-10T19:14:38.672 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:14:38.672 INFO:teuthology.orchestra.run.smithi183.stdout: "last_change": "64", 2024-09-10T19:14:38.672 INFO:teuthology.orchestra.run.smithi183.stdout: "last_force_op_resend": "0", 2024-09-10T19:14:38.672 INFO:teuthology.orchestra.run.smithi183.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-10T19:14:38.672 INFO:teuthology.orchestra.run.smithi183.stdout: "last_force_op_resend_preluminous": "0", 2024-09-10T19:14:38.672 INFO:teuthology.orchestra.run.smithi183.stdout: "auid": 0, 2024-09-10T19:14:38.672 INFO:teuthology.orchestra.run.smithi183.stdout: "snap_mode": "selfmanaged", 2024-09-10T19:14:38.672 INFO:teuthology.orchestra.run.smithi183.stdout: "snap_seq": 0, 2024-09-10T19:14:38.672 INFO:teuthology.orchestra.run.smithi183.stdout: "snap_epoch": 0, 2024-09-10T19:14:38.672 INFO:teuthology.orchestra.run.smithi183.stdout: "pool_snaps": [], 2024-09-10T19:14:38.673 INFO:teuthology.orchestra.run.smithi183.stdout: "removed_snaps": "[]", 2024-09-10T19:14:38.673 INFO:teuthology.orchestra.run.smithi183.stdout: "quota_max_bytes": 0, 2024-09-10T19:14:38.673 INFO:teuthology.orchestra.run.smithi183.stdout: "quota_max_objects": 0, 2024-09-10T19:14:38.673 INFO:teuthology.orchestra.run.smithi183.stdout: "tiers": [], 2024-09-10T19:14:38.673 INFO:teuthology.orchestra.run.smithi183.stdout: "tier_of": -1, 2024-09-10T19:14:38.673 INFO:teuthology.orchestra.run.smithi183.stdout: "read_tier": -1, 2024-09-10T19:14:38.673 INFO:teuthology.orchestra.run.smithi183.stdout: "write_tier": -1, 2024-09-10T19:14:38.673 INFO:teuthology.orchestra.run.smithi183.stdout: "cache_mode": "none", 2024-09-10T19:14:38.673 INFO:teuthology.orchestra.run.smithi183.stdout: "target_max_bytes": 0, 2024-09-10T19:14:38.673 INFO:teuthology.orchestra.run.smithi183.stdout: "target_max_objects": 0, 2024-09-10T19:14:38.673 INFO:teuthology.orchestra.run.smithi183.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-10T19:14:38.673 INFO:teuthology.orchestra.run.smithi183.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-10T19:14:38.674 INFO:teuthology.orchestra.run.smithi183.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-10T19:14:38.674 INFO:teuthology.orchestra.run.smithi183.stdout: "cache_min_flush_age": 0, 2024-09-10T19:14:38.674 INFO:teuthology.orchestra.run.smithi183.stdout: "cache_min_evict_age": 0, 2024-09-10T19:14:38.674 INFO:teuthology.orchestra.run.smithi183.stdout: "erasure_code_profile": "", 2024-09-10T19:14:38.674 INFO:teuthology.orchestra.run.smithi183.stdout: "hit_set_params": { 2024-09-10T19:14:38.674 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "none" 2024-09-10T19:14:38.674 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:14:38.674 INFO:teuthology.orchestra.run.smithi183.stdout: "hit_set_period": 0, 2024-09-10T19:14:38.674 INFO:teuthology.orchestra.run.smithi183.stdout: "hit_set_count": 0, 2024-09-10T19:14:38.674 INFO:teuthology.orchestra.run.smithi183.stdout: "use_gmt_hitset": true, 2024-09-10T19:14:38.674 INFO:teuthology.orchestra.run.smithi183.stdout: "min_read_recency_for_promote": 0, 2024-09-10T19:14:38.674 INFO:teuthology.orchestra.run.smithi183.stdout: "min_write_recency_for_promote": 0, 2024-09-10T19:14:38.674 INFO:teuthology.orchestra.run.smithi183.stdout: "hit_set_grade_decay_rate": 0, 2024-09-10T19:14:38.675 INFO:teuthology.orchestra.run.smithi183.stdout: "hit_set_search_last_n": 0, 2024-09-10T19:14:38.675 INFO:teuthology.orchestra.run.smithi183.stdout: "grade_table": [], 2024-09-10T19:14:38.675 INFO:teuthology.orchestra.run.smithi183.stdout: "stripe_width": 0, 2024-09-10T19:14:38.675 INFO:teuthology.orchestra.run.smithi183.stdout: "expected_num_objects": 0, 2024-09-10T19:14:38.675 INFO:teuthology.orchestra.run.smithi183.stdout: "fast_read": false, 2024-09-10T19:14:38.675 INFO:teuthology.orchestra.run.smithi183.stdout: "options": {}, 2024-09-10T19:14:38.675 INFO:teuthology.orchestra.run.smithi183.stdout: "application_metadata": { 2024-09-10T19:14:38.675 INFO:teuthology.orchestra.run.smithi183.stdout: "rgw": {} 2024-09-10T19:14:38.675 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:14:38.675 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:14:38.675 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:14:38.675 INFO:teuthology.orchestra.run.smithi183.stdout: "pool": 6, 2024-09-10T19:14:38.675 INFO:teuthology.orchestra.run.smithi183.stdout: "pool_name": "default.rgw.meta", 2024-09-10T19:14:38.676 INFO:teuthology.orchestra.run.smithi183.stdout: "create_time": "2024-09-10T19:10:11.993712+0000", 2024-09-10T19:14:38.676 INFO:teuthology.orchestra.run.smithi183.stdout: "flags": 1, 2024-09-10T19:14:38.676 INFO:teuthology.orchestra.run.smithi183.stdout: "flags_names": "hashpspool", 2024-09-10T19:14:38.676 INFO:teuthology.orchestra.run.smithi183.stdout: "type": 1, 2024-09-10T19:14:38.676 INFO:teuthology.orchestra.run.smithi183.stdout: "size": 3, 2024-09-10T19:14:38.676 INFO:teuthology.orchestra.run.smithi183.stdout: "min_size": 2, 2024-09-10T19:14:38.676 INFO:teuthology.orchestra.run.smithi183.stdout: "crush_rule": 0, 2024-09-10T19:14:38.676 INFO:teuthology.orchestra.run.smithi183.stdout: "peering_crush_bucket_count": 0, 2024-09-10T19:14:38.676 INFO:teuthology.orchestra.run.smithi183.stdout: "peering_crush_bucket_target": 0, 2024-09-10T19:14:38.676 INFO:teuthology.orchestra.run.smithi183.stdout: "peering_crush_bucket_barrier": 0, 2024-09-10T19:14:38.676 INFO:teuthology.orchestra.run.smithi183.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-10T19:14:38.676 INFO:teuthology.orchestra.run.smithi183.stdout: "object_hash": 2, 2024-09-10T19:14:38.676 INFO:teuthology.orchestra.run.smithi183.stdout: "pg_autoscale_mode": "off", 2024-09-10T19:14:38.676 INFO:teuthology.orchestra.run.smithi183.stdout: "pg_num": 32, 2024-09-10T19:14:38.677 INFO:teuthology.orchestra.run.smithi183.stdout: "pg_placement_num": 32, 2024-09-10T19:14:38.677 INFO:teuthology.orchestra.run.smithi183.stdout: "pg_placement_num_target": 32, 2024-09-10T19:14:38.677 INFO:teuthology.orchestra.run.smithi183.stdout: "pg_num_target": 32, 2024-09-10T19:14:38.677 INFO:teuthology.orchestra.run.smithi183.stdout: "pg_num_pending": 32, 2024-09-10T19:14:38.677 INFO:teuthology.orchestra.run.smithi183.stdout: "last_pg_merge_meta": { 2024-09-10T19:14:38.677 INFO:teuthology.orchestra.run.smithi183.stdout: "source_pgid": "0.0", 2024-09-10T19:14:38.677 INFO:teuthology.orchestra.run.smithi183.stdout: "ready_epoch": 0, 2024-09-10T19:14:38.677 INFO:teuthology.orchestra.run.smithi183.stdout: "last_epoch_started": 0, 2024-09-10T19:14:38.677 INFO:teuthology.orchestra.run.smithi183.stdout: "last_epoch_clean": 0, 2024-09-10T19:14:38.677 INFO:teuthology.orchestra.run.smithi183.stdout: "source_version": "0'0", 2024-09-10T19:14:38.677 INFO:teuthology.orchestra.run.smithi183.stdout: "target_version": "0'0" 2024-09-10T19:14:38.677 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:14:38.677 INFO:teuthology.orchestra.run.smithi183.stdout: "last_change": "66", 2024-09-10T19:14:38.678 INFO:teuthology.orchestra.run.smithi183.stdout: "last_force_op_resend": "0", 2024-09-10T19:14:38.678 INFO:teuthology.orchestra.run.smithi183.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-10T19:14:38.678 INFO:teuthology.orchestra.run.smithi183.stdout: "last_force_op_resend_preluminous": "0", 2024-09-10T19:14:38.678 INFO:teuthology.orchestra.run.smithi183.stdout: "auid": 0, 2024-09-10T19:14:38.678 INFO:teuthology.orchestra.run.smithi183.stdout: "snap_mode": "selfmanaged", 2024-09-10T19:14:38.678 INFO:teuthology.orchestra.run.smithi183.stdout: "snap_seq": 0, 2024-09-10T19:14:38.678 INFO:teuthology.orchestra.run.smithi183.stdout: "snap_epoch": 0, 2024-09-10T19:14:38.678 INFO:teuthology.orchestra.run.smithi183.stdout: "pool_snaps": [], 2024-09-10T19:14:38.678 INFO:teuthology.orchestra.run.smithi183.stdout: "removed_snaps": "[]", 2024-09-10T19:14:38.678 INFO:teuthology.orchestra.run.smithi183.stdout: "quota_max_bytes": 0, 2024-09-10T19:14:38.678 INFO:teuthology.orchestra.run.smithi183.stdout: "quota_max_objects": 0, 2024-09-10T19:14:38.678 INFO:teuthology.orchestra.run.smithi183.stdout: "tiers": [], 2024-09-10T19:14:38.679 INFO:teuthology.orchestra.run.smithi183.stdout: "tier_of": -1, 2024-09-10T19:14:38.679 INFO:teuthology.orchestra.run.smithi183.stdout: "read_tier": -1, 2024-09-10T19:14:38.679 INFO:teuthology.orchestra.run.smithi183.stdout: "write_tier": -1, 2024-09-10T19:14:38.679 INFO:teuthology.orchestra.run.smithi183.stdout: "cache_mode": "none", 2024-09-10T19:14:38.679 INFO:teuthology.orchestra.run.smithi183.stdout: "target_max_bytes": 0, 2024-09-10T19:14:38.679 INFO:teuthology.orchestra.run.smithi183.stdout: "target_max_objects": 0, 2024-09-10T19:14:38.679 INFO:teuthology.orchestra.run.smithi183.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-10T19:14:38.679 INFO:teuthology.orchestra.run.smithi183.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-10T19:14:38.679 INFO:teuthology.orchestra.run.smithi183.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-10T19:14:38.679 INFO:teuthology.orchestra.run.smithi183.stdout: "cache_min_flush_age": 0, 2024-09-10T19:14:38.679 INFO:teuthology.orchestra.run.smithi183.stdout: "cache_min_evict_age": 0, 2024-09-10T19:14:38.679 INFO:teuthology.orchestra.run.smithi183.stdout: "erasure_code_profile": "", 2024-09-10T19:14:38.680 INFO:teuthology.orchestra.run.smithi183.stdout: "hit_set_params": { 2024-09-10T19:14:38.680 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "none" 2024-09-10T19:14:38.680 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:14:38.680 INFO:teuthology.orchestra.run.smithi183.stdout: "hit_set_period": 0, 2024-09-10T19:14:38.680 INFO:teuthology.orchestra.run.smithi183.stdout: "hit_set_count": 0, 2024-09-10T19:14:38.680 INFO:teuthology.orchestra.run.smithi183.stdout: "use_gmt_hitset": true, 2024-09-10T19:14:38.680 INFO:teuthology.orchestra.run.smithi183.stdout: "min_read_recency_for_promote": 0, 2024-09-10T19:14:38.680 INFO:teuthology.orchestra.run.smithi183.stdout: "min_write_recency_for_promote": 0, 2024-09-10T19:14:38.680 INFO:teuthology.orchestra.run.smithi183.stdout: "hit_set_grade_decay_rate": 0, 2024-09-10T19:14:38.680 INFO:teuthology.orchestra.run.smithi183.stdout: "hit_set_search_last_n": 0, 2024-09-10T19:14:38.680 INFO:teuthology.orchestra.run.smithi183.stdout: "grade_table": [], 2024-09-10T19:14:38.680 INFO:teuthology.orchestra.run.smithi183.stdout: "stripe_width": 0, 2024-09-10T19:14:38.680 INFO:teuthology.orchestra.run.smithi183.stdout: "expected_num_objects": 0, 2024-09-10T19:14:38.681 INFO:teuthology.orchestra.run.smithi183.stdout: "fast_read": false, 2024-09-10T19:14:38.681 INFO:teuthology.orchestra.run.smithi183.stdout: "options": { 2024-09-10T19:14:38.681 INFO:teuthology.orchestra.run.smithi183.stdout: "pg_autoscale_bias": 4 2024-09-10T19:14:38.681 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:14:38.681 INFO:teuthology.orchestra.run.smithi183.stdout: "application_metadata": { 2024-09-10T19:14:38.681 INFO:teuthology.orchestra.run.smithi183.stdout: "rgw": {} 2024-09-10T19:14:38.681 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:14:38.681 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:14:38.681 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:14:38.681 INFO:teuthology.orchestra.run.smithi183.stdout: "pool": 7, 2024-09-10T19:14:38.681 INFO:teuthology.orchestra.run.smithi183.stdout: "pool_name": "unique_pool_0", 2024-09-10T19:14:38.681 INFO:teuthology.orchestra.run.smithi183.stdout: "create_time": "2024-09-10T19:12:10.002625+0000", 2024-09-10T19:14:38.681 INFO:teuthology.orchestra.run.smithi183.stdout: "flags": 1, 2024-09-10T19:14:38.682 INFO:teuthology.orchestra.run.smithi183.stdout: "flags_names": "hashpspool", 2024-09-10T19:14:38.682 INFO:teuthology.orchestra.run.smithi183.stdout: "type": 1, 2024-09-10T19:14:38.682 INFO:teuthology.orchestra.run.smithi183.stdout: "size": 3, 2024-09-10T19:14:38.682 INFO:teuthology.orchestra.run.smithi183.stdout: "min_size": 2, 2024-09-10T19:14:38.682 INFO:teuthology.orchestra.run.smithi183.stdout: "crush_rule": 0, 2024-09-10T19:14:38.682 INFO:teuthology.orchestra.run.smithi183.stdout: "peering_crush_bucket_count": 0, 2024-09-10T19:14:38.682 INFO:teuthology.orchestra.run.smithi183.stdout: "peering_crush_bucket_target": 0, 2024-09-10T19:14:38.682 INFO:teuthology.orchestra.run.smithi183.stdout: "peering_crush_bucket_barrier": 0, 2024-09-10T19:14:38.682 INFO:teuthology.orchestra.run.smithi183.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-10T19:14:38.682 INFO:teuthology.orchestra.run.smithi183.stdout: "object_hash": 2, 2024-09-10T19:14:38.682 INFO:teuthology.orchestra.run.smithi183.stdout: "pg_autoscale_mode": "off", 2024-09-10T19:14:38.682 INFO:teuthology.orchestra.run.smithi183.stdout: "pg_num": 16, 2024-09-10T19:14:38.682 INFO:teuthology.orchestra.run.smithi183.stdout: "pg_placement_num": 16, 2024-09-10T19:14:38.682 INFO:teuthology.orchestra.run.smithi183.stdout: "pg_placement_num_target": 16, 2024-09-10T19:14:38.683 INFO:teuthology.orchestra.run.smithi183.stdout: "pg_num_target": 16, 2024-09-10T19:14:38.683 INFO:teuthology.orchestra.run.smithi183.stdout: "pg_num_pending": 16, 2024-09-10T19:14:38.683 INFO:teuthology.orchestra.run.smithi183.stdout: "last_pg_merge_meta": { 2024-09-10T19:14:38.683 INFO:teuthology.orchestra.run.smithi183.stdout: "source_pgid": "0.0", 2024-09-10T19:14:38.683 INFO:teuthology.orchestra.run.smithi183.stdout: "ready_epoch": 0, 2024-09-10T19:14:38.683 INFO:teuthology.orchestra.run.smithi183.stdout: "last_epoch_started": 0, 2024-09-10T19:14:38.683 INFO:teuthology.orchestra.run.smithi183.stdout: "last_epoch_clean": 0, 2024-09-10T19:14:38.683 INFO:teuthology.orchestra.run.smithi183.stdout: "source_version": "0'0", 2024-09-10T19:14:38.683 INFO:teuthology.orchestra.run.smithi183.stdout: "target_version": "0'0" 2024-09-10T19:14:38.683 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:14:38.683 INFO:teuthology.orchestra.run.smithi183.stdout: "last_change": "90", 2024-09-10T19:14:38.683 INFO:teuthology.orchestra.run.smithi183.stdout: "last_force_op_resend": "0", 2024-09-10T19:14:38.684 INFO:teuthology.orchestra.run.smithi183.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-10T19:14:38.684 INFO:teuthology.orchestra.run.smithi183.stdout: "last_force_op_resend_preluminous": "0", 2024-09-10T19:14:38.684 INFO:teuthology.orchestra.run.smithi183.stdout: "auid": 0, 2024-09-10T19:14:38.684 INFO:teuthology.orchestra.run.smithi183.stdout: "snap_mode": "selfmanaged", 2024-09-10T19:14:38.684 INFO:teuthology.orchestra.run.smithi183.stdout: "snap_seq": 0, 2024-09-10T19:14:38.684 INFO:teuthology.orchestra.run.smithi183.stdout: "snap_epoch": 0, 2024-09-10T19:14:38.684 INFO:teuthology.orchestra.run.smithi183.stdout: "pool_snaps": [], 2024-09-10T19:14:38.684 INFO:teuthology.orchestra.run.smithi183.stdout: "removed_snaps": "[]", 2024-09-10T19:14:38.684 INFO:teuthology.orchestra.run.smithi183.stdout: "quota_max_bytes": 0, 2024-09-10T19:14:38.684 INFO:teuthology.orchestra.run.smithi183.stdout: "quota_max_objects": 0, 2024-09-10T19:14:38.684 INFO:teuthology.orchestra.run.smithi183.stdout: "tiers": [], 2024-09-10T19:14:38.684 INFO:teuthology.orchestra.run.smithi183.stdout: "tier_of": -1, 2024-09-10T19:14:38.684 INFO:teuthology.orchestra.run.smithi183.stdout: "read_tier": -1, 2024-09-10T19:14:38.684 INFO:teuthology.orchestra.run.smithi183.stdout: "write_tier": -1, 2024-09-10T19:14:38.685 INFO:teuthology.orchestra.run.smithi183.stdout: "cache_mode": "none", 2024-09-10T19:14:38.685 INFO:teuthology.orchestra.run.smithi183.stdout: "target_max_bytes": 0, 2024-09-10T19:14:38.685 INFO:teuthology.orchestra.run.smithi183.stdout: "target_max_objects": 0, 2024-09-10T19:14:38.685 INFO:teuthology.orchestra.run.smithi183.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-10T19:14:38.685 INFO:teuthology.orchestra.run.smithi183.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-10T19:14:38.685 INFO:teuthology.orchestra.run.smithi183.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-10T19:14:38.685 INFO:teuthology.orchestra.run.smithi183.stdout: "cache_min_flush_age": 0, 2024-09-10T19:14:38.685 INFO:teuthology.orchestra.run.smithi183.stdout: "cache_min_evict_age": 0, 2024-09-10T19:14:38.685 INFO:teuthology.orchestra.run.smithi183.stdout: "erasure_code_profile": "", 2024-09-10T19:14:38.685 INFO:teuthology.orchestra.run.smithi183.stdout: "hit_set_params": { 2024-09-10T19:14:38.686 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "none" 2024-09-10T19:14:38.686 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:14:38.686 INFO:teuthology.orchestra.run.smithi183.stdout: "hit_set_period": 0, 2024-09-10T19:14:38.686 INFO:teuthology.orchestra.run.smithi183.stdout: "hit_set_count": 0, 2024-09-10T19:14:38.686 INFO:teuthology.orchestra.run.smithi183.stdout: "use_gmt_hitset": true, 2024-09-10T19:14:38.686 INFO:teuthology.orchestra.run.smithi183.stdout: "min_read_recency_for_promote": 0, 2024-09-10T19:14:38.686 INFO:teuthology.orchestra.run.smithi183.stdout: "min_write_recency_for_promote": 0, 2024-09-10T19:14:38.686 INFO:teuthology.orchestra.run.smithi183.stdout: "hit_set_grade_decay_rate": 0, 2024-09-10T19:14:38.686 INFO:teuthology.orchestra.run.smithi183.stdout: "hit_set_search_last_n": 0, 2024-09-10T19:14:38.686 INFO:teuthology.orchestra.run.smithi183.stdout: "grade_table": [], 2024-09-10T19:14:38.686 INFO:teuthology.orchestra.run.smithi183.stdout: "stripe_width": 0, 2024-09-10T19:14:38.686 INFO:teuthology.orchestra.run.smithi183.stdout: "expected_num_objects": 0, 2024-09-10T19:14:38.686 INFO:teuthology.orchestra.run.smithi183.stdout: "fast_read": false, 2024-09-10T19:14:38.687 INFO:teuthology.orchestra.run.smithi183.stdout: "options": {}, 2024-09-10T19:14:38.687 INFO:teuthology.orchestra.run.smithi183.stdout: "application_metadata": { 2024-09-10T19:14:38.687 INFO:teuthology.orchestra.run.smithi183.stdout: "rados": {} 2024-09-10T19:14:38.687 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:14:38.687 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:14:38.687 INFO:teuthology.orchestra.run.smithi183.stdout: ], 2024-09-10T19:14:38.687 INFO:teuthology.orchestra.run.smithi183.stdout: "osds": [ 2024-09-10T19:14:38.687 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:14:38.687 INFO:teuthology.orchestra.run.smithi183.stdout: "osd": 0, 2024-09-10T19:14:38.687 INFO:teuthology.orchestra.run.smithi183.stdout: "uuid": "9d323010-394a-49aa-ba9b-244d87faf36e", 2024-09-10T19:14:38.687 INFO:teuthology.orchestra.run.smithi183.stdout: "up": 1, 2024-09-10T19:14:38.687 INFO:teuthology.orchestra.run.smithi183.stdout: "in": 1, 2024-09-10T19:14:38.687 INFO:teuthology.orchestra.run.smithi183.stdout: "weight": 1, 2024-09-10T19:14:38.688 INFO:teuthology.orchestra.run.smithi183.stdout: "primary_affinity": 1, 2024-09-10T19:14:38.688 INFO:teuthology.orchestra.run.smithi183.stdout: "last_clean_begin": 0, 2024-09-10T19:14:38.688 INFO:teuthology.orchestra.run.smithi183.stdout: "last_clean_end": 0, 2024-09-10T19:14:38.688 INFO:teuthology.orchestra.run.smithi183.stdout: "up_from": 8, 2024-09-10T19:14:38.688 INFO:teuthology.orchestra.run.smithi183.stdout: "up_thru": 116, 2024-09-10T19:14:38.688 INFO:teuthology.orchestra.run.smithi183.stdout: "down_at": 0, 2024-09-10T19:14:38.688 INFO:teuthology.orchestra.run.smithi183.stdout: "lost_at": 0, 2024-09-10T19:14:38.688 INFO:teuthology.orchestra.run.smithi183.stdout: "public_addrs": { 2024-09-10T19:14:38.688 INFO:teuthology.orchestra.run.smithi183.stdout: "addrvec": [ 2024-09-10T19:14:38.688 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:14:38.688 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v2", 2024-09-10T19:14:38.688 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.183:6802", 2024-09-10T19:14:38.688 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 2813074952 2024-09-10T19:14:38.689 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:14:38.689 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:14:38.689 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v1", 2024-09-10T19:14:38.689 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.183:6803", 2024-09-10T19:14:38.689 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 2813074952 2024-09-10T19:14:38.689 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:14:38.689 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:14:38.689 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:14:38.690 INFO:teuthology.orchestra.run.smithi183.stdout: "cluster_addrs": { 2024-09-10T19:14:38.690 INFO:teuthology.orchestra.run.smithi183.stdout: "addrvec": [ 2024-09-10T19:14:38.690 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:14:38.690 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v2", 2024-09-10T19:14:38.690 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.183:6804", 2024-09-10T19:14:38.690 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 2813074952 2024-09-10T19:14:38.690 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:14:38.690 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:14:38.690 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v1", 2024-09-10T19:14:38.690 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.183:6805", 2024-09-10T19:14:38.690 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 2813074952 2024-09-10T19:14:38.690 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:14:38.691 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:14:38.691 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:14:38.691 INFO:teuthology.orchestra.run.smithi183.stdout: "heartbeat_back_addrs": { 2024-09-10T19:14:38.691 INFO:teuthology.orchestra.run.smithi183.stdout: "addrvec": [ 2024-09-10T19:14:38.691 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:14:38.691 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v2", 2024-09-10T19:14:38.691 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.183:6808", 2024-09-10T19:14:38.691 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 2813074952 2024-09-10T19:14:38.691 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:14:38.691 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:14:38.691 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v1", 2024-09-10T19:14:38.691 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.183:6809", 2024-09-10T19:14:38.691 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 2813074952 2024-09-10T19:14:38.691 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:14:38.692 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:14:38.692 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:14:38.692 INFO:teuthology.orchestra.run.smithi183.stdout: "heartbeat_front_addrs": { 2024-09-10T19:14:38.692 INFO:teuthology.orchestra.run.smithi183.stdout: "addrvec": [ 2024-09-10T19:14:38.692 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:14:38.692 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v2", 2024-09-10T19:14:38.692 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.183:6806", 2024-09-10T19:14:38.692 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 2813074952 2024-09-10T19:14:38.692 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:14:38.692 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:14:38.692 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v1", 2024-09-10T19:14:38.692 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.183:6807", 2024-09-10T19:14:38.693 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 2813074952 2024-09-10T19:14:38.693 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:14:38.693 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:14:38.693 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:14:38.693 INFO:teuthology.orchestra.run.smithi183.stdout: "public_addr": "172.21.15.183:6803/2813074952", 2024-09-10T19:14:38.693 INFO:teuthology.orchestra.run.smithi183.stdout: "cluster_addr": "172.21.15.183:6805/2813074952", 2024-09-10T19:14:38.693 INFO:teuthology.orchestra.run.smithi183.stdout: "heartbeat_back_addr": "172.21.15.183:6809/2813074952", 2024-09-10T19:14:38.693 INFO:teuthology.orchestra.run.smithi183.stdout: "heartbeat_front_addr": "172.21.15.183:6807/2813074952", 2024-09-10T19:14:38.693 INFO:teuthology.orchestra.run.smithi183.stdout: "state": [ 2024-09-10T19:14:38.693 INFO:teuthology.orchestra.run.smithi183.stdout: "exists", 2024-09-10T19:14:38.693 INFO:teuthology.orchestra.run.smithi183.stdout: "up" 2024-09-10T19:14:38.693 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:14:38.693 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:14:38.694 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:14:38.694 INFO:teuthology.orchestra.run.smithi183.stdout: "osd": 1, 2024-09-10T19:14:38.694 INFO:teuthology.orchestra.run.smithi183.stdout: "uuid": "71a6c5dc-5109-4284-8d20-187d8e7afc9e", 2024-09-10T19:14:38.694 INFO:teuthology.orchestra.run.smithi183.stdout: "up": 0, 2024-09-10T19:14:38.694 INFO:teuthology.orchestra.run.smithi183.stdout: "in": 0, 2024-09-10T19:14:38.694 INFO:teuthology.orchestra.run.smithi183.stdout: "weight": 0, 2024-09-10T19:14:38.694 INFO:teuthology.orchestra.run.smithi183.stdout: "primary_affinity": 1, 2024-09-10T19:14:38.694 INFO:teuthology.orchestra.run.smithi183.stdout: "last_clean_begin": 0, 2024-09-10T19:14:38.694 INFO:teuthology.orchestra.run.smithi183.stdout: "last_clean_end": 0, 2024-09-10T19:14:38.694 INFO:teuthology.orchestra.run.smithi183.stdout: "up_from": 13, 2024-09-10T19:14:38.694 INFO:teuthology.orchestra.run.smithi183.stdout: "up_thru": 86, 2024-09-10T19:14:38.694 INFO:teuthology.orchestra.run.smithi183.stdout: "down_at": 99, 2024-09-10T19:14:38.694 INFO:teuthology.orchestra.run.smithi183.stdout: "lost_at": 0, 2024-09-10T19:14:38.695 INFO:teuthology.orchestra.run.smithi183.stdout: "public_addrs": { 2024-09-10T19:14:38.695 INFO:teuthology.orchestra.run.smithi183.stdout: "addrvec": [ 2024-09-10T19:14:38.695 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:14:38.695 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v2", 2024-09-10T19:14:38.695 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.183:6810", 2024-09-10T19:14:38.695 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 3880626114 2024-09-10T19:14:38.695 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:14:38.695 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:14:38.695 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v1", 2024-09-10T19:14:38.695 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.183:6811", 2024-09-10T19:14:38.695 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 3880626114 2024-09-10T19:14:38.695 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:14:38.695 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:14:38.695 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:14:38.696 INFO:teuthology.orchestra.run.smithi183.stdout: "cluster_addrs": { 2024-09-10T19:14:38.696 INFO:teuthology.orchestra.run.smithi183.stdout: "addrvec": [ 2024-09-10T19:14:38.696 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:14:38.696 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v2", 2024-09-10T19:14:38.696 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.183:6812", 2024-09-10T19:14:38.696 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 3880626114 2024-09-10T19:14:38.696 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:14:38.696 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:14:38.696 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v1", 2024-09-10T19:14:38.696 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.183:6813", 2024-09-10T19:14:38.696 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 3880626114 2024-09-10T19:14:38.696 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:14:38.696 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:14:38.696 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:14:38.697 INFO:teuthology.orchestra.run.smithi183.stdout: "heartbeat_back_addrs": { 2024-09-10T19:14:38.697 INFO:teuthology.orchestra.run.smithi183.stdout: "addrvec": [ 2024-09-10T19:14:38.697 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:14:38.697 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v2", 2024-09-10T19:14:38.697 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.183:6816", 2024-09-10T19:14:38.697 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 3880626114 2024-09-10T19:14:38.697 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:14:38.697 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:14:38.697 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v1", 2024-09-10T19:14:38.697 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.183:6817", 2024-09-10T19:14:38.697 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 3880626114 2024-09-10T19:14:38.697 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:14:38.697 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:14:38.698 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:14:38.698 INFO:teuthology.orchestra.run.smithi183.stdout: "heartbeat_front_addrs": { 2024-09-10T19:14:38.698 INFO:teuthology.orchestra.run.smithi183.stdout: "addrvec": [ 2024-09-10T19:14:38.698 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:14:38.698 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v2", 2024-09-10T19:14:38.698 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.183:6814", 2024-09-10T19:14:38.698 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 3880626114 2024-09-10T19:14:38.698 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:14:38.698 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:14:38.698 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v1", 2024-09-10T19:14:38.698 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.183:6815", 2024-09-10T19:14:38.698 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 3880626114 2024-09-10T19:14:38.698 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:14:38.698 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:14:38.698 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:14:38.698 INFO:teuthology.orchestra.run.smithi183.stdout: "public_addr": "172.21.15.183:6811/3880626114", 2024-09-10T19:14:38.699 INFO:teuthology.orchestra.run.smithi183.stdout: "cluster_addr": "172.21.15.183:6813/3880626114", 2024-09-10T19:14:38.699 INFO:teuthology.orchestra.run.smithi183.stdout: "heartbeat_back_addr": "172.21.15.183:6817/3880626114", 2024-09-10T19:14:38.699 INFO:teuthology.orchestra.run.smithi183.stdout: "heartbeat_front_addr": "172.21.15.183:6815/3880626114", 2024-09-10T19:14:38.699 INFO:teuthology.orchestra.run.smithi183.stdout: "state": [ 2024-09-10T19:14:38.699 INFO:teuthology.orchestra.run.smithi183.stdout: "exists" 2024-09-10T19:14:38.699 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:14:38.699 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:14:38.699 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:14:38.699 INFO:teuthology.orchestra.run.smithi183.stdout: "osd": 2, 2024-09-10T19:14:38.699 INFO:teuthology.orchestra.run.smithi183.stdout: "uuid": "fc310d5b-b89b-4655-9989-7e10ba356509", 2024-09-10T19:14:38.699 INFO:teuthology.orchestra.run.smithi183.stdout: "up": 1, 2024-09-10T19:14:38.699 INFO:teuthology.orchestra.run.smithi183.stdout: "in": 1, 2024-09-10T19:14:38.700 INFO:teuthology.orchestra.run.smithi183.stdout: "weight": 1, 2024-09-10T19:14:38.700 INFO:teuthology.orchestra.run.smithi183.stdout: "primary_affinity": 1, 2024-09-10T19:14:38.700 INFO:teuthology.orchestra.run.smithi183.stdout: "last_clean_begin": 0, 2024-09-10T19:14:38.700 INFO:teuthology.orchestra.run.smithi183.stdout: "last_clean_end": 0, 2024-09-10T19:14:38.700 INFO:teuthology.orchestra.run.smithi183.stdout: "up_from": 18, 2024-09-10T19:14:38.700 INFO:teuthology.orchestra.run.smithi183.stdout: "up_thru": 115, 2024-09-10T19:14:38.700 INFO:teuthology.orchestra.run.smithi183.stdout: "down_at": 0, 2024-09-10T19:14:38.700 INFO:teuthology.orchestra.run.smithi183.stdout: "lost_at": 0, 2024-09-10T19:14:38.700 INFO:teuthology.orchestra.run.smithi183.stdout: "public_addrs": { 2024-09-10T19:14:38.700 INFO:teuthology.orchestra.run.smithi183.stdout: "addrvec": [ 2024-09-10T19:14:38.700 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:14:38.700 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v2", 2024-09-10T19:14:38.700 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.183:6818", 2024-09-10T19:14:38.700 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 3730013786 2024-09-10T19:14:38.700 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:14:38.701 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:14:38.701 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v1", 2024-09-10T19:14:38.701 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.183:6819", 2024-09-10T19:14:38.701 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 3730013786 2024-09-10T19:14:38.701 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:14:38.701 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:14:38.701 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:14:38.701 INFO:teuthology.orchestra.run.smithi183.stdout: "cluster_addrs": { 2024-09-10T19:14:38.701 INFO:teuthology.orchestra.run.smithi183.stdout: "addrvec": [ 2024-09-10T19:14:38.701 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:14:38.701 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v2", 2024-09-10T19:14:38.701 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.183:6820", 2024-09-10T19:14:38.702 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 3730013786 2024-09-10T19:14:38.702 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:14:38.702 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:14:38.702 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v1", 2024-09-10T19:14:38.702 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.183:6821", 2024-09-10T19:14:38.702 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 3730013786 2024-09-10T19:14:38.702 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:14:38.702 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:14:38.702 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:14:38.702 INFO:teuthology.orchestra.run.smithi183.stdout: "heartbeat_back_addrs": { 2024-09-10T19:14:38.702 INFO:teuthology.orchestra.run.smithi183.stdout: "addrvec": [ 2024-09-10T19:14:38.702 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:14:38.702 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v2", 2024-09-10T19:14:38.702 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.183:6824", 2024-09-10T19:14:38.702 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 3730013786 2024-09-10T19:14:38.702 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:14:38.703 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:14:38.703 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v1", 2024-09-10T19:14:38.703 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.183:6825", 2024-09-10T19:14:38.703 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 3730013786 2024-09-10T19:14:38.703 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:14:38.703 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:14:38.703 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:14:38.703 INFO:teuthology.orchestra.run.smithi183.stdout: "heartbeat_front_addrs": { 2024-09-10T19:14:38.703 INFO:teuthology.orchestra.run.smithi183.stdout: "addrvec": [ 2024-09-10T19:14:38.703 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:14:38.703 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v2", 2024-09-10T19:14:38.703 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.183:6822", 2024-09-10T19:14:38.703 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 3730013786 2024-09-10T19:14:38.704 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:14:38.704 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:14:38.704 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v1", 2024-09-10T19:14:38.704 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.183:6823", 2024-09-10T19:14:38.704 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 3730013786 2024-09-10T19:14:38.704 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:14:38.704 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:14:38.704 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:14:38.704 INFO:teuthology.orchestra.run.smithi183.stdout: "public_addr": "172.21.15.183:6819/3730013786", 2024-09-10T19:14:38.704 INFO:teuthology.orchestra.run.smithi183.stdout: "cluster_addr": "172.21.15.183:6821/3730013786", 2024-09-10T19:14:38.704 INFO:teuthology.orchestra.run.smithi183.stdout: "heartbeat_back_addr": "172.21.15.183:6825/3730013786", 2024-09-10T19:14:38.704 INFO:teuthology.orchestra.run.smithi183.stdout: "heartbeat_front_addr": "172.21.15.183:6823/3730013786", 2024-09-10T19:14:38.704 INFO:teuthology.orchestra.run.smithi183.stdout: "state": [ 2024-09-10T19:14:38.705 INFO:teuthology.orchestra.run.smithi183.stdout: "exists", 2024-09-10T19:14:38.705 INFO:teuthology.orchestra.run.smithi183.stdout: "up" 2024-09-10T19:14:38.705 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:14:38.705 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:14:38.705 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:14:38.705 INFO:teuthology.orchestra.run.smithi183.stdout: "osd": 3, 2024-09-10T19:14:38.705 INFO:teuthology.orchestra.run.smithi183.stdout: "uuid": "7fd07a95-ad22-43f5-a968-539552283e89", 2024-09-10T19:14:38.705 INFO:teuthology.orchestra.run.smithi183.stdout: "up": 1, 2024-09-10T19:14:38.705 INFO:teuthology.orchestra.run.smithi183.stdout: "in": 1, 2024-09-10T19:14:38.705 INFO:teuthology.orchestra.run.smithi183.stdout: "weight": 1, 2024-09-10T19:14:38.705 INFO:teuthology.orchestra.run.smithi183.stdout: "primary_affinity": 1, 2024-09-10T19:14:38.705 INFO:teuthology.orchestra.run.smithi183.stdout: "last_clean_begin": 0, 2024-09-10T19:14:38.705 INFO:teuthology.orchestra.run.smithi183.stdout: "last_clean_end": 0, 2024-09-10T19:14:38.706 INFO:teuthology.orchestra.run.smithi183.stdout: "up_from": 26, 2024-09-10T19:14:38.706 INFO:teuthology.orchestra.run.smithi183.stdout: "up_thru": 116, 2024-09-10T19:14:38.706 INFO:teuthology.orchestra.run.smithi183.stdout: "down_at": 0, 2024-09-10T19:14:38.706 INFO:teuthology.orchestra.run.smithi183.stdout: "lost_at": 0, 2024-09-10T19:14:38.706 INFO:teuthology.orchestra.run.smithi183.stdout: "public_addrs": { 2024-09-10T19:14:38.706 INFO:teuthology.orchestra.run.smithi183.stdout: "addrvec": [ 2024-09-10T19:14:38.706 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:14:38.706 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v2", 2024-09-10T19:14:38.706 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.183:6826", 2024-09-10T19:14:38.706 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 3522836984 2024-09-10T19:14:38.706 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:14:38.706 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:14:38.706 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v1", 2024-09-10T19:14:38.707 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.183:6827", 2024-09-10T19:14:38.707 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 3522836984 2024-09-10T19:14:38.707 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:14:38.707 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:14:38.707 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:14:38.707 INFO:teuthology.orchestra.run.smithi183.stdout: "cluster_addrs": { 2024-09-10T19:14:38.707 INFO:teuthology.orchestra.run.smithi183.stdout: "addrvec": [ 2024-09-10T19:14:38.707 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:14:38.707 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v2", 2024-09-10T19:14:38.707 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.183:6828", 2024-09-10T19:14:38.707 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 3522836984 2024-09-10T19:14:38.707 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:14:38.707 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:14:38.708 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v1", 2024-09-10T19:14:38.708 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.183:6829", 2024-09-10T19:14:38.708 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 3522836984 2024-09-10T19:14:38.708 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:14:38.708 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:14:38.708 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:14:38.708 INFO:teuthology.orchestra.run.smithi183.stdout: "heartbeat_back_addrs": { 2024-09-10T19:14:38.708 INFO:teuthology.orchestra.run.smithi183.stdout: "addrvec": [ 2024-09-10T19:14:38.708 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:14:38.708 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v2", 2024-09-10T19:14:38.708 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.183:6832", 2024-09-10T19:14:38.708 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 3522836984 2024-09-10T19:14:38.708 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:14:38.708 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:14:38.708 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v1", 2024-09-10T19:14:38.709 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.183:6833", 2024-09-10T19:14:38.709 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 3522836984 2024-09-10T19:14:38.709 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:14:38.709 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:14:38.709 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:14:38.709 INFO:teuthology.orchestra.run.smithi183.stdout: "heartbeat_front_addrs": { 2024-09-10T19:14:38.709 INFO:teuthology.orchestra.run.smithi183.stdout: "addrvec": [ 2024-09-10T19:14:38.709 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:14:38.709 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v2", 2024-09-10T19:14:38.709 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.183:6830", 2024-09-10T19:14:38.709 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 3522836984 2024-09-10T19:14:38.709 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:14:38.709 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:14:38.709 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v1", 2024-09-10T19:14:38.710 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.183:6831", 2024-09-10T19:14:38.710 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 3522836984 2024-09-10T19:14:38.710 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:14:38.710 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:14:38.710 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:14:38.710 INFO:teuthology.orchestra.run.smithi183.stdout: "public_addr": "172.21.15.183:6827/3522836984", 2024-09-10T19:14:38.710 INFO:teuthology.orchestra.run.smithi183.stdout: "cluster_addr": "172.21.15.183:6829/3522836984", 2024-09-10T19:14:38.710 INFO:teuthology.orchestra.run.smithi183.stdout: "heartbeat_back_addr": "172.21.15.183:6833/3522836984", 2024-09-10T19:14:38.710 INFO:teuthology.orchestra.run.smithi183.stdout: "heartbeat_front_addr": "172.21.15.183:6831/3522836984", 2024-09-10T19:14:38.710 INFO:teuthology.orchestra.run.smithi183.stdout: "state": [ 2024-09-10T19:14:38.710 INFO:teuthology.orchestra.run.smithi183.stdout: "exists", 2024-09-10T19:14:38.710 INFO:teuthology.orchestra.run.smithi183.stdout: "up" 2024-09-10T19:14:38.710 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:14:38.710 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:14:38.710 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:14:38.711 INFO:teuthology.orchestra.run.smithi183.stdout: "osd": 4, 2024-09-10T19:14:38.711 INFO:teuthology.orchestra.run.smithi183.stdout: "uuid": "631083fb-e503-44da-9b6b-154e88539ee7", 2024-09-10T19:14:38.711 INFO:teuthology.orchestra.run.smithi183.stdout: "up": 1, 2024-09-10T19:14:38.711 INFO:teuthology.orchestra.run.smithi183.stdout: "in": 1, 2024-09-10T19:14:38.711 INFO:teuthology.orchestra.run.smithi183.stdout: "weight": 1, 2024-09-10T19:14:38.711 INFO:teuthology.orchestra.run.smithi183.stdout: "primary_affinity": 1, 2024-09-10T19:14:38.711 INFO:teuthology.orchestra.run.smithi183.stdout: "last_clean_begin": 0, 2024-09-10T19:14:38.711 INFO:teuthology.orchestra.run.smithi183.stdout: "last_clean_end": 0, 2024-09-10T19:14:38.711 INFO:teuthology.orchestra.run.smithi183.stdout: "up_from": 32, 2024-09-10T19:14:38.711 INFO:teuthology.orchestra.run.smithi183.stdout: "up_thru": 115, 2024-09-10T19:14:38.711 INFO:teuthology.orchestra.run.smithi183.stdout: "down_at": 0, 2024-09-10T19:14:38.711 INFO:teuthology.orchestra.run.smithi183.stdout: "lost_at": 0, 2024-09-10T19:14:38.711 INFO:teuthology.orchestra.run.smithi183.stdout: "public_addrs": { 2024-09-10T19:14:38.711 INFO:teuthology.orchestra.run.smithi183.stdout: "addrvec": [ 2024-09-10T19:14:38.711 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:14:38.712 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v2", 2024-09-10T19:14:38.712 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.191:6800", 2024-09-10T19:14:38.712 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 2674828539 2024-09-10T19:14:38.712 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:14:38.712 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:14:38.712 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v1", 2024-09-10T19:14:38.712 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.191:6801", 2024-09-10T19:14:38.712 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 2674828539 2024-09-10T19:14:38.712 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:14:38.712 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:14:38.712 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:14:38.712 INFO:teuthology.orchestra.run.smithi183.stdout: "cluster_addrs": { 2024-09-10T19:14:38.712 INFO:teuthology.orchestra.run.smithi183.stdout: "addrvec": [ 2024-09-10T19:14:38.712 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:14:38.713 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v2", 2024-09-10T19:14:38.713 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.191:6802", 2024-09-10T19:14:38.713 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 2674828539 2024-09-10T19:14:38.713 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:14:38.713 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:14:38.713 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v1", 2024-09-10T19:14:38.713 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.191:6803", 2024-09-10T19:14:38.713 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 2674828539 2024-09-10T19:14:38.713 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:14:38.713 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:14:38.713 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:14:38.713 INFO:teuthology.orchestra.run.smithi183.stdout: "heartbeat_back_addrs": { 2024-09-10T19:14:38.713 INFO:teuthology.orchestra.run.smithi183.stdout: "addrvec": [ 2024-09-10T19:14:38.713 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:14:38.714 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v2", 2024-09-10T19:14:38.714 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.191:6806", 2024-09-10T19:14:38.714 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 2674828539 2024-09-10T19:14:38.714 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:14:38.714 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:14:38.714 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v1", 2024-09-10T19:14:38.714 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.191:6807", 2024-09-10T19:14:38.714 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 2674828539 2024-09-10T19:14:38.714 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:14:38.714 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:14:38.714 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:14:38.714 INFO:teuthology.orchestra.run.smithi183.stdout: "heartbeat_front_addrs": { 2024-09-10T19:14:38.714 INFO:teuthology.orchestra.run.smithi183.stdout: "addrvec": [ 2024-09-10T19:14:38.715 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:14:38.715 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v2", 2024-09-10T19:14:38.715 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.191:6804", 2024-09-10T19:14:38.715 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 2674828539 2024-09-10T19:14:38.715 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:14:38.715 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:14:38.715 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v1", 2024-09-10T19:14:38.715 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.191:6805", 2024-09-10T19:14:38.715 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 2674828539 2024-09-10T19:14:38.715 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:14:38.715 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:14:38.715 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:14:38.715 INFO:teuthology.orchestra.run.smithi183.stdout: "public_addr": "172.21.15.191:6801/2674828539", 2024-09-10T19:14:38.716 INFO:teuthology.orchestra.run.smithi183.stdout: "cluster_addr": "172.21.15.191:6803/2674828539", 2024-09-10T19:14:38.716 INFO:teuthology.orchestra.run.smithi183.stdout: "heartbeat_back_addr": "172.21.15.191:6807/2674828539", 2024-09-10T19:14:38.716 INFO:teuthology.orchestra.run.smithi183.stdout: "heartbeat_front_addr": "172.21.15.191:6805/2674828539", 2024-09-10T19:14:38.716 INFO:teuthology.orchestra.run.smithi183.stdout: "state": [ 2024-09-10T19:14:38.716 INFO:teuthology.orchestra.run.smithi183.stdout: "exists", 2024-09-10T19:14:38.716 INFO:teuthology.orchestra.run.smithi183.stdout: "up" 2024-09-10T19:14:38.716 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:14:38.716 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:14:38.716 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:14:38.716 INFO:teuthology.orchestra.run.smithi183.stdout: "osd": 5, 2024-09-10T19:14:38.716 INFO:teuthology.orchestra.run.smithi183.stdout: "uuid": "6b963695-3eca-4e6e-9207-05b8010ebff9", 2024-09-10T19:14:38.716 INFO:teuthology.orchestra.run.smithi183.stdout: "up": 1, 2024-09-10T19:14:38.716 INFO:teuthology.orchestra.run.smithi183.stdout: "in": 1, 2024-09-10T19:14:38.717 INFO:teuthology.orchestra.run.smithi183.stdout: "weight": 1, 2024-09-10T19:14:38.717 INFO:teuthology.orchestra.run.smithi183.stdout: "primary_affinity": 1, 2024-09-10T19:14:38.717 INFO:teuthology.orchestra.run.smithi183.stdout: "last_clean_begin": 0, 2024-09-10T19:14:38.717 INFO:teuthology.orchestra.run.smithi183.stdout: "last_clean_end": 0, 2024-09-10T19:14:38.717 INFO:teuthology.orchestra.run.smithi183.stdout: "up_from": 39, 2024-09-10T19:14:38.717 INFO:teuthology.orchestra.run.smithi183.stdout: "up_thru": 116, 2024-09-10T19:14:38.717 INFO:teuthology.orchestra.run.smithi183.stdout: "down_at": 0, 2024-09-10T19:14:38.717 INFO:teuthology.orchestra.run.smithi183.stdout: "lost_at": 0, 2024-09-10T19:14:38.717 INFO:teuthology.orchestra.run.smithi183.stdout: "public_addrs": { 2024-09-10T19:14:38.717 INFO:teuthology.orchestra.run.smithi183.stdout: "addrvec": [ 2024-09-10T19:14:38.717 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:14:38.717 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v2", 2024-09-10T19:14:38.717 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.191:6808", 2024-09-10T19:14:38.718 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 681733378 2024-09-10T19:14:38.718 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:14:38.718 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:14:38.718 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v1", 2024-09-10T19:14:38.718 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.191:6809", 2024-09-10T19:14:38.718 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 681733378 2024-09-10T19:14:38.718 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:14:38.718 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:14:38.718 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:14:38.718 INFO:teuthology.orchestra.run.smithi183.stdout: "cluster_addrs": { 2024-09-10T19:14:38.718 INFO:teuthology.orchestra.run.smithi183.stdout: "addrvec": [ 2024-09-10T19:14:38.718 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:14:38.718 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v2", 2024-09-10T19:14:38.718 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.191:6810", 2024-09-10T19:14:38.719 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 681733378 2024-09-10T19:14:38.719 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:14:38.719 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:14:38.719 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v1", 2024-09-10T19:14:38.719 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.191:6811", 2024-09-10T19:14:38.719 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 681733378 2024-09-10T19:14:38.719 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:14:38.719 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:14:38.719 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:14:38.719 INFO:teuthology.orchestra.run.smithi183.stdout: "heartbeat_back_addrs": { 2024-09-10T19:14:38.719 INFO:teuthology.orchestra.run.smithi183.stdout: "addrvec": [ 2024-09-10T19:14:38.719 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:14:38.719 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v2", 2024-09-10T19:14:38.720 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.191:6814", 2024-09-10T19:14:38.720 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 681733378 2024-09-10T19:14:38.720 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:14:38.720 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:14:38.720 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v1", 2024-09-10T19:14:38.720 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.191:6815", 2024-09-10T19:14:38.720 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 681733378 2024-09-10T19:14:38.720 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:14:38.720 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:14:38.720 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:14:38.720 INFO:teuthology.orchestra.run.smithi183.stdout: "heartbeat_front_addrs": { 2024-09-10T19:14:38.720 INFO:teuthology.orchestra.run.smithi183.stdout: "addrvec": [ 2024-09-10T19:14:38.720 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:14:38.721 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v2", 2024-09-10T19:14:38.721 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.191:6812", 2024-09-10T19:14:38.721 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 681733378 2024-09-10T19:14:38.721 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:14:38.721 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:14:38.721 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v1", 2024-09-10T19:14:38.721 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.191:6813", 2024-09-10T19:14:38.721 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 681733378 2024-09-10T19:14:38.721 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:14:38.721 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:14:38.721 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:14:38.721 INFO:teuthology.orchestra.run.smithi183.stdout: "public_addr": "172.21.15.191:6809/681733378", 2024-09-10T19:14:38.721 INFO:teuthology.orchestra.run.smithi183.stdout: "cluster_addr": "172.21.15.191:6811/681733378", 2024-09-10T19:14:38.721 INFO:teuthology.orchestra.run.smithi183.stdout: "heartbeat_back_addr": "172.21.15.191:6815/681733378", 2024-09-10T19:14:38.722 INFO:teuthology.orchestra.run.smithi183.stdout: "heartbeat_front_addr": "172.21.15.191:6813/681733378", 2024-09-10T19:14:38.722 INFO:teuthology.orchestra.run.smithi183.stdout: "state": [ 2024-09-10T19:14:38.722 INFO:teuthology.orchestra.run.smithi183.stdout: "exists", 2024-09-10T19:14:38.722 INFO:teuthology.orchestra.run.smithi183.stdout: "up" 2024-09-10T19:14:38.722 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:14:38.722 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:14:38.722 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:14:38.722 INFO:teuthology.orchestra.run.smithi183.stdout: "osd": 6, 2024-09-10T19:14:38.722 INFO:teuthology.orchestra.run.smithi183.stdout: "uuid": "2dc9754f-6aef-4e19-8605-2f1591921811", 2024-09-10T19:14:38.722 INFO:teuthology.orchestra.run.smithi183.stdout: "up": 1, 2024-09-10T19:14:38.722 INFO:teuthology.orchestra.run.smithi183.stdout: "in": 1, 2024-09-10T19:14:38.722 INFO:teuthology.orchestra.run.smithi183.stdout: "weight": 1, 2024-09-10T19:14:38.722 INFO:teuthology.orchestra.run.smithi183.stdout: "primary_affinity": 1, 2024-09-10T19:14:38.723 INFO:teuthology.orchestra.run.smithi183.stdout: "last_clean_begin": 0, 2024-09-10T19:14:38.723 INFO:teuthology.orchestra.run.smithi183.stdout: "last_clean_end": 0, 2024-09-10T19:14:38.723 INFO:teuthology.orchestra.run.smithi183.stdout: "up_from": 47, 2024-09-10T19:14:38.723 INFO:teuthology.orchestra.run.smithi183.stdout: "up_thru": 116, 2024-09-10T19:14:38.723 INFO:teuthology.orchestra.run.smithi183.stdout: "down_at": 0, 2024-09-10T19:14:38.723 INFO:teuthology.orchestra.run.smithi183.stdout: "lost_at": 0, 2024-09-10T19:14:38.723 INFO:teuthology.orchestra.run.smithi183.stdout: "public_addrs": { 2024-09-10T19:14:38.723 INFO:teuthology.orchestra.run.smithi183.stdout: "addrvec": [ 2024-09-10T19:14:38.723 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:14:38.723 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v2", 2024-09-10T19:14:38.723 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.191:6816", 2024-09-10T19:14:38.723 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 2825813477 2024-09-10T19:14:38.724 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:14:38.724 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:14:38.724 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v1", 2024-09-10T19:14:38.724 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.191:6817", 2024-09-10T19:14:38.724 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 2825813477 2024-09-10T19:14:38.724 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:14:38.724 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:14:38.724 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:14:38.724 INFO:teuthology.orchestra.run.smithi183.stdout: "cluster_addrs": { 2024-09-10T19:14:38.724 INFO:teuthology.orchestra.run.smithi183.stdout: "addrvec": [ 2024-09-10T19:14:38.724 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:14:38.724 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v2", 2024-09-10T19:14:38.724 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.191:6818", 2024-09-10T19:14:38.725 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 2825813477 2024-09-10T19:14:38.725 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:14:38.725 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:14:38.725 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v1", 2024-09-10T19:14:38.725 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.191:6819", 2024-09-10T19:14:38.725 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 2825813477 2024-09-10T19:14:38.725 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:14:38.725 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:14:38.725 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:14:38.725 INFO:teuthology.orchestra.run.smithi183.stdout: "heartbeat_back_addrs": { 2024-09-10T19:14:38.725 INFO:teuthology.orchestra.run.smithi183.stdout: "addrvec": [ 2024-09-10T19:14:38.725 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:14:38.725 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v2", 2024-09-10T19:14:38.726 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.191:6822", 2024-09-10T19:14:38.726 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 2825813477 2024-09-10T19:14:38.726 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:14:38.726 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:14:38.726 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v1", 2024-09-10T19:14:38.726 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.191:6823", 2024-09-10T19:14:38.726 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 2825813477 2024-09-10T19:14:38.726 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:14:38.726 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:14:38.726 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:14:38.726 INFO:teuthology.orchestra.run.smithi183.stdout: "heartbeat_front_addrs": { 2024-09-10T19:14:38.726 INFO:teuthology.orchestra.run.smithi183.stdout: "addrvec": [ 2024-09-10T19:14:38.726 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:14:38.727 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v2", 2024-09-10T19:14:38.727 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.191:6820", 2024-09-10T19:14:38.727 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 2825813477 2024-09-10T19:14:38.727 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:14:38.727 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:14:38.727 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v1", 2024-09-10T19:14:38.727 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.191:6821", 2024-09-10T19:14:38.727 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 2825813477 2024-09-10T19:14:38.727 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:14:38.727 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:14:38.727 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:14:38.727 INFO:teuthology.orchestra.run.smithi183.stdout: "public_addr": "172.21.15.191:6817/2825813477", 2024-09-10T19:14:38.727 INFO:teuthology.orchestra.run.smithi183.stdout: "cluster_addr": "172.21.15.191:6819/2825813477", 2024-09-10T19:14:38.728 INFO:teuthology.orchestra.run.smithi183.stdout: "heartbeat_back_addr": "172.21.15.191:6823/2825813477", 2024-09-10T19:14:38.728 INFO:teuthology.orchestra.run.smithi183.stdout: "heartbeat_front_addr": "172.21.15.191:6821/2825813477", 2024-09-10T19:14:38.728 INFO:teuthology.orchestra.run.smithi183.stdout: "state": [ 2024-09-10T19:14:38.728 INFO:teuthology.orchestra.run.smithi183.stdout: "exists", 2024-09-10T19:14:38.728 INFO:teuthology.orchestra.run.smithi183.stdout: "up" 2024-09-10T19:14:38.728 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:14:38.728 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:14:38.728 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:14:38.728 INFO:teuthology.orchestra.run.smithi183.stdout: "osd": 7, 2024-09-10T19:14:38.728 INFO:teuthology.orchestra.run.smithi183.stdout: "uuid": "1a9a299e-d5ba-4b9a-85d5-7f6a5bc754e1", 2024-09-10T19:14:38.728 INFO:teuthology.orchestra.run.smithi183.stdout: "up": 1, 2024-09-10T19:14:38.728 INFO:teuthology.orchestra.run.smithi183.stdout: "in": 1, 2024-09-10T19:14:38.728 INFO:teuthology.orchestra.run.smithi183.stdout: "weight": 1, 2024-09-10T19:14:38.729 INFO:teuthology.orchestra.run.smithi183.stdout: "primary_affinity": 1, 2024-09-10T19:14:38.729 INFO:teuthology.orchestra.run.smithi183.stdout: "last_clean_begin": 0, 2024-09-10T19:14:38.729 INFO:teuthology.orchestra.run.smithi183.stdout: "last_clean_end": 0, 2024-09-10T19:14:38.729 INFO:teuthology.orchestra.run.smithi183.stdout: "up_from": 54, 2024-09-10T19:14:38.729 INFO:teuthology.orchestra.run.smithi183.stdout: "up_thru": 115, 2024-09-10T19:14:38.729 INFO:teuthology.orchestra.run.smithi183.stdout: "down_at": 0, 2024-09-10T19:14:38.729 INFO:teuthology.orchestra.run.smithi183.stdout: "lost_at": 0, 2024-09-10T19:14:38.729 INFO:teuthology.orchestra.run.smithi183.stdout: "public_addrs": { 2024-09-10T19:14:38.729 INFO:teuthology.orchestra.run.smithi183.stdout: "addrvec": [ 2024-09-10T19:14:38.729 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:14:38.729 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v2", 2024-09-10T19:14:38.729 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.191:6824", 2024-09-10T19:14:38.729 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 1422610021 2024-09-10T19:14:38.729 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:14:38.730 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:14:38.730 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v1", 2024-09-10T19:14:38.730 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.191:6825", 2024-09-10T19:14:38.730 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 1422610021 2024-09-10T19:14:38.730 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:14:38.730 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:14:38.730 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:14:38.730 INFO:teuthology.orchestra.run.smithi183.stdout: "cluster_addrs": { 2024-09-10T19:14:38.730 INFO:teuthology.orchestra.run.smithi183.stdout: "addrvec": [ 2024-09-10T19:14:38.730 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:14:38.730 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v2", 2024-09-10T19:14:38.730 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.191:6826", 2024-09-10T19:14:38.730 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 1422610021 2024-09-10T19:14:38.731 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:14:38.731 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:14:38.731 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v1", 2024-09-10T19:14:38.731 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.191:6827", 2024-09-10T19:14:38.731 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 1422610021 2024-09-10T19:14:38.731 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:14:38.731 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:14:38.731 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:14:38.731 INFO:teuthology.orchestra.run.smithi183.stdout: "heartbeat_back_addrs": { 2024-09-10T19:14:38.731 INFO:teuthology.orchestra.run.smithi183.stdout: "addrvec": [ 2024-09-10T19:14:38.731 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:14:38.731 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v2", 2024-09-10T19:14:38.731 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.191:6830", 2024-09-10T19:14:38.731 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 1422610021 2024-09-10T19:14:38.732 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:14:38.732 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:14:38.732 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v1", 2024-09-10T19:14:38.732 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.191:6831", 2024-09-10T19:14:38.732 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 1422610021 2024-09-10T19:14:38.732 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:14:38.732 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:14:38.732 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:14:38.732 INFO:teuthology.orchestra.run.smithi183.stdout: "heartbeat_front_addrs": { 2024-09-10T19:14:38.732 INFO:teuthology.orchestra.run.smithi183.stdout: "addrvec": [ 2024-09-10T19:14:38.732 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:14:38.732 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v2", 2024-09-10T19:14:38.732 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.191:6828", 2024-09-10T19:14:38.732 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 1422610021 2024-09-10T19:14:38.733 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:14:38.733 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:14:38.733 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v1", 2024-09-10T19:14:38.733 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.191:6829", 2024-09-10T19:14:38.733 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 1422610021 2024-09-10T19:14:38.733 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:14:38.733 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:14:38.733 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:14:38.733 INFO:teuthology.orchestra.run.smithi183.stdout: "public_addr": "172.21.15.191:6825/1422610021", 2024-09-10T19:14:38.733 INFO:teuthology.orchestra.run.smithi183.stdout: "cluster_addr": "172.21.15.191:6827/1422610021", 2024-09-10T19:14:38.733 INFO:teuthology.orchestra.run.smithi183.stdout: "heartbeat_back_addr": "172.21.15.191:6831/1422610021", 2024-09-10T19:14:38.733 INFO:teuthology.orchestra.run.smithi183.stdout: "heartbeat_front_addr": "172.21.15.191:6829/1422610021", 2024-09-10T19:14:38.734 INFO:teuthology.orchestra.run.smithi183.stdout: "state": [ 2024-09-10T19:14:38.734 INFO:teuthology.orchestra.run.smithi183.stdout: "exists", 2024-09-10T19:14:38.734 INFO:teuthology.orchestra.run.smithi183.stdout: "up" 2024-09-10T19:14:38.734 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:14:38.734 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:14:38.734 INFO:teuthology.orchestra.run.smithi183.stdout: ], 2024-09-10T19:14:38.734 INFO:teuthology.orchestra.run.smithi183.stdout: "osd_xinfo": [ 2024-09-10T19:14:38.734 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:14:38.734 INFO:teuthology.orchestra.run.smithi183.stdout: "osd": 0, 2024-09-10T19:14:38.734 INFO:teuthology.orchestra.run.smithi183.stdout: "down_stamp": "0.000000", 2024-09-10T19:14:38.734 INFO:teuthology.orchestra.run.smithi183.stdout: "laggy_probability": 0, 2024-09-10T19:14:38.734 INFO:teuthology.orchestra.run.smithi183.stdout: "laggy_interval": 0, 2024-09-10T19:14:38.734 INFO:teuthology.orchestra.run.smithi183.stdout: "features": 4540138320759226367, 2024-09-10T19:14:38.735 INFO:teuthology.orchestra.run.smithi183.stdout: "old_weight": 0, 2024-09-10T19:14:38.735 INFO:teuthology.orchestra.run.smithi183.stdout: "last_purged_snaps_scrub": "2024-09-10T19:14:08.336733+0000", 2024-09-10T19:14:38.735 INFO:teuthology.orchestra.run.smithi183.stdout: "dead_epoch": 0 2024-09-10T19:14:38.735 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:14:38.735 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:14:38.735 INFO:teuthology.orchestra.run.smithi183.stdout: "osd": 1, 2024-09-10T19:14:38.735 INFO:teuthology.orchestra.run.smithi183.stdout: "down_stamp": "0.000000", 2024-09-10T19:14:38.735 INFO:teuthology.orchestra.run.smithi183.stdout: "laggy_probability": 0, 2024-09-10T19:14:38.735 INFO:teuthology.orchestra.run.smithi183.stdout: "laggy_interval": 0, 2024-09-10T19:14:38.735 INFO:teuthology.orchestra.run.smithi183.stdout: "features": 4540138320759226367, 2024-09-10T19:14:38.735 INFO:teuthology.orchestra.run.smithi183.stdout: "old_weight": 65536, 2024-09-10T19:14:38.735 INFO:teuthology.orchestra.run.smithi183.stdout: "last_purged_snaps_scrub": "2024-09-10T19:12:08.593593+0000", 2024-09-10T19:14:38.735 INFO:teuthology.orchestra.run.smithi183.stdout: "dead_epoch": 98 2024-09-10T19:14:38.735 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:14:38.736 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:14:38.736 INFO:teuthology.orchestra.run.smithi183.stdout: "osd": 2, 2024-09-10T19:14:38.736 INFO:teuthology.orchestra.run.smithi183.stdout: "down_stamp": "0.000000", 2024-09-10T19:14:38.736 INFO:teuthology.orchestra.run.smithi183.stdout: "laggy_probability": 0, 2024-09-10T19:14:38.736 INFO:teuthology.orchestra.run.smithi183.stdout: "laggy_interval": 0, 2024-09-10T19:14:38.736 INFO:teuthology.orchestra.run.smithi183.stdout: "features": 4540138320759226367, 2024-09-10T19:14:38.736 INFO:teuthology.orchestra.run.smithi183.stdout: "old_weight": 0, 2024-09-10T19:14:38.736 INFO:teuthology.orchestra.run.smithi183.stdout: "last_purged_snaps_scrub": "2024-09-10T19:14:06.494190+0000", 2024-09-10T19:14:38.736 INFO:teuthology.orchestra.run.smithi183.stdout: "dead_epoch": 0 2024-09-10T19:14:38.736 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:14:38.736 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:14:38.736 INFO:teuthology.orchestra.run.smithi183.stdout: "osd": 3, 2024-09-10T19:14:38.736 INFO:teuthology.orchestra.run.smithi183.stdout: "down_stamp": "0.000000", 2024-09-10T19:14:38.736 INFO:teuthology.orchestra.run.smithi183.stdout: "laggy_probability": 0, 2024-09-10T19:14:38.737 INFO:teuthology.orchestra.run.smithi183.stdout: "laggy_interval": 0, 2024-09-10T19:14:38.737 INFO:teuthology.orchestra.run.smithi183.stdout: "features": 4540138320759226367, 2024-09-10T19:14:38.737 INFO:teuthology.orchestra.run.smithi183.stdout: "old_weight": 0, 2024-09-10T19:14:38.737 INFO:teuthology.orchestra.run.smithi183.stdout: "last_purged_snaps_scrub": "2024-09-10T19:13:32.119763+0000", 2024-09-10T19:14:38.737 INFO:teuthology.orchestra.run.smithi183.stdout: "dead_epoch": 0 2024-09-10T19:14:38.737 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:14:38.737 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:14:38.737 INFO:teuthology.orchestra.run.smithi183.stdout: "osd": 4, 2024-09-10T19:14:38.737 INFO:teuthology.orchestra.run.smithi183.stdout: "down_stamp": "0.000000", 2024-09-10T19:14:38.737 INFO:teuthology.orchestra.run.smithi183.stdout: "laggy_probability": 0, 2024-09-10T19:14:38.737 INFO:teuthology.orchestra.run.smithi183.stdout: "laggy_interval": 0, 2024-09-10T19:14:38.737 INFO:teuthology.orchestra.run.smithi183.stdout: "features": 4540138320759226367, 2024-09-10T19:14:38.737 INFO:teuthology.orchestra.run.smithi183.stdout: "old_weight": 0, 2024-09-10T19:14:38.737 INFO:teuthology.orchestra.run.smithi183.stdout: "last_purged_snaps_scrub": "2024-09-10T19:13:34.944494+0000", 2024-09-10T19:14:38.738 INFO:teuthology.orchestra.run.smithi183.stdout: "dead_epoch": 0 2024-09-10T19:14:38.738 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:14:38.738 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:14:38.738 INFO:teuthology.orchestra.run.smithi183.stdout: "osd": 5, 2024-09-10T19:14:38.738 INFO:teuthology.orchestra.run.smithi183.stdout: "down_stamp": "0.000000", 2024-09-10T19:14:38.738 INFO:teuthology.orchestra.run.smithi183.stdout: "laggy_probability": 0, 2024-09-10T19:14:38.738 INFO:teuthology.orchestra.run.smithi183.stdout: "laggy_interval": 0, 2024-09-10T19:14:38.738 INFO:teuthology.orchestra.run.smithi183.stdout: "features": 4540138320759226367, 2024-09-10T19:14:38.738 INFO:teuthology.orchestra.run.smithi183.stdout: "old_weight": 0, 2024-09-10T19:14:38.738 INFO:teuthology.orchestra.run.smithi183.stdout: "last_purged_snaps_scrub": "2024-09-10T19:13:49.264552+0000", 2024-09-10T19:14:38.738 INFO:teuthology.orchestra.run.smithi183.stdout: "dead_epoch": 0 2024-09-10T19:14:38.738 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:14:38.738 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:14:38.738 INFO:teuthology.orchestra.run.smithi183.stdout: "osd": 6, 2024-09-10T19:14:38.739 INFO:teuthology.orchestra.run.smithi183.stdout: "down_stamp": "0.000000", 2024-09-10T19:14:38.739 INFO:teuthology.orchestra.run.smithi183.stdout: "laggy_probability": 0, 2024-09-10T19:14:38.739 INFO:teuthology.orchestra.run.smithi183.stdout: "laggy_interval": 0, 2024-09-10T19:14:38.739 INFO:teuthology.orchestra.run.smithi183.stdout: "features": 4540138320759226367, 2024-09-10T19:14:38.739 INFO:teuthology.orchestra.run.smithi183.stdout: "old_weight": 0, 2024-09-10T19:14:38.739 INFO:teuthology.orchestra.run.smithi183.stdout: "last_purged_snaps_scrub": "2024-09-10T19:13:40.759766+0000", 2024-09-10T19:14:38.739 INFO:teuthology.orchestra.run.smithi183.stdout: "dead_epoch": 0 2024-09-10T19:14:38.739 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:14:38.739 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:14:38.739 INFO:teuthology.orchestra.run.smithi183.stdout: "osd": 7, 2024-09-10T19:14:38.739 INFO:teuthology.orchestra.run.smithi183.stdout: "down_stamp": "0.000000", 2024-09-10T19:14:38.739 INFO:teuthology.orchestra.run.smithi183.stdout: "laggy_probability": 0, 2024-09-10T19:14:38.739 INFO:teuthology.orchestra.run.smithi183.stdout: "laggy_interval": 0, 2024-09-10T19:14:38.739 INFO:teuthology.orchestra.run.smithi183.stdout: "features": 4540138320759226367, 2024-09-10T19:14:38.740 INFO:teuthology.orchestra.run.smithi183.stdout: "old_weight": 0, 2024-09-10T19:14:38.740 INFO:teuthology.orchestra.run.smithi183.stdout: "last_purged_snaps_scrub": "2024-09-10T19:13:31.920321+0000", 2024-09-10T19:14:38.740 INFO:teuthology.orchestra.run.smithi183.stdout: "dead_epoch": 0 2024-09-10T19:14:38.740 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:14:38.740 INFO:teuthology.orchestra.run.smithi183.stdout: ], 2024-09-10T19:14:38.740 INFO:teuthology.orchestra.run.smithi183.stdout: "pg_upmap": [], 2024-09-10T19:14:38.740 INFO:teuthology.orchestra.run.smithi183.stdout: "pg_upmap_items": [ 2024-09-10T19:14:38.740 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:14:38.740 INFO:teuthology.orchestra.run.smithi183.stdout: "pgid": "2.1f", 2024-09-10T19:14:38.740 INFO:teuthology.orchestra.run.smithi183.stdout: "mappings": [ 2024-09-10T19:14:38.740 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:14:38.740 INFO:teuthology.orchestra.run.smithi183.stdout: "from": 7, 2024-09-10T19:14:38.740 INFO:teuthology.orchestra.run.smithi183.stdout: "to": 2 2024-09-10T19:14:38.741 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:14:38.741 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:14:38.741 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:14:38.741 INFO:teuthology.orchestra.run.smithi183.stdout: ], 2024-09-10T19:14:38.741 INFO:teuthology.orchestra.run.smithi183.stdout: "pg_temp": [ 2024-09-10T19:14:38.741 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:14:38.741 INFO:teuthology.orchestra.run.smithi183.stdout: "pgid": "7.0", 2024-09-10T19:14:38.741 INFO:teuthology.orchestra.run.smithi183.stdout: "osds": [ 2024-09-10T19:14:38.741 INFO:teuthology.orchestra.run.smithi183.stdout: 5, 2024-09-10T19:14:38.741 INFO:teuthology.orchestra.run.smithi183.stdout: 3 2024-09-10T19:14:38.741 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:14:38.741 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:14:38.741 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:14:38.741 INFO:teuthology.orchestra.run.smithi183.stdout: "pgid": "7.2", 2024-09-10T19:14:38.742 INFO:teuthology.orchestra.run.smithi183.stdout: "osds": [ 2024-09-10T19:14:38.742 INFO:teuthology.orchestra.run.smithi183.stdout: 3, 2024-09-10T19:14:38.742 INFO:teuthology.orchestra.run.smithi183.stdout: 2 2024-09-10T19:14:38.742 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:14:38.742 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:14:38.742 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:14:38.742 INFO:teuthology.orchestra.run.smithi183.stdout: "pgid": "7.4", 2024-09-10T19:14:38.742 INFO:teuthology.orchestra.run.smithi183.stdout: "osds": [ 2024-09-10T19:14:38.742 INFO:teuthology.orchestra.run.smithi183.stdout: 0, 2024-09-10T19:14:38.742 INFO:teuthology.orchestra.run.smithi183.stdout: 4 2024-09-10T19:14:38.742 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:14:38.742 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:14:38.742 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:14:38.742 INFO:teuthology.orchestra.run.smithi183.stdout: "pgid": "7.9", 2024-09-10T19:14:38.743 INFO:teuthology.orchestra.run.smithi183.stdout: "osds": [ 2024-09-10T19:14:38.743 INFO:teuthology.orchestra.run.smithi183.stdout: 0, 2024-09-10T19:14:38.743 INFO:teuthology.orchestra.run.smithi183.stdout: 4 2024-09-10T19:14:38.743 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:14:38.743 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:14:38.743 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:14:38.743 INFO:teuthology.orchestra.run.smithi183.stdout: "pgid": "7.b", 2024-09-10T19:14:38.743 INFO:teuthology.orchestra.run.smithi183.stdout: "osds": [ 2024-09-10T19:14:38.743 INFO:teuthology.orchestra.run.smithi183.stdout: 5, 2024-09-10T19:14:38.743 INFO:teuthology.orchestra.run.smithi183.stdout: 6 2024-09-10T19:14:38.743 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:14:38.743 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:14:38.743 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:14:38.744 INFO:teuthology.orchestra.run.smithi183.stdout: "pgid": "7.d", 2024-09-10T19:14:38.744 INFO:teuthology.orchestra.run.smithi183.stdout: "osds": [ 2024-09-10T19:14:38.744 INFO:teuthology.orchestra.run.smithi183.stdout: 6, 2024-09-10T19:14:38.744 INFO:teuthology.orchestra.run.smithi183.stdout: 0 2024-09-10T19:14:38.744 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:14:38.744 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:14:38.744 INFO:teuthology.orchestra.run.smithi183.stdout: ], 2024-09-10T19:14:38.744 INFO:teuthology.orchestra.run.smithi183.stdout: "primary_temp": [], 2024-09-10T19:14:38.744 INFO:teuthology.orchestra.run.smithi183.stdout: "blocklist": { 2024-09-10T19:14:38.744 INFO:teuthology.orchestra.run.smithi183.stdout: "172.21.15.183:6801/3311783362": "2024-09-11T19:10:43.576943+0000", 2024-09-10T19:14:38.744 INFO:teuthology.orchestra.run.smithi183.stdout: "172.21.15.183:0/4088553922": "2024-09-11T19:10:43.576943+0000", 2024-09-10T19:14:38.744 INFO:teuthology.orchestra.run.smithi183.stdout: "172.21.15.183:0/3528769414": "2024-09-11T19:05:48.153879+0000", 2024-09-10T19:14:38.744 INFO:teuthology.orchestra.run.smithi183.stdout: "172.21.15.183:0/1766036658": "2024-09-11T19:05:48.153879+0000", 2024-09-10T19:14:38.744 INFO:teuthology.orchestra.run.smithi183.stdout: "172.21.15.183:0/3044170088": "2024-09-11T19:05:25.702354+0000", 2024-09-10T19:14:38.744 INFO:teuthology.orchestra.run.smithi183.stdout: "172.21.15.183:0/3037333778": "2024-09-11T19:05:25.702354+0000", 2024-09-10T19:14:38.745 INFO:teuthology.orchestra.run.smithi183.stdout: "172.21.15.183:6801/3747320570": "2024-09-11T19:05:25.702354+0000", 2024-09-10T19:14:38.745 INFO:teuthology.orchestra.run.smithi183.stdout: "172.21.15.183:0/3982386906": "2024-09-11T19:10:43.576943+0000", 2024-09-10T19:14:38.745 INFO:teuthology.orchestra.run.smithi183.stdout: "172.21.15.183:0/2436014455": "2024-09-11T19:05:25.702354+0000", 2024-09-10T19:14:38.745 INFO:teuthology.orchestra.run.smithi183.stdout: "172.21.15.183:6801/659745765": "2024-09-11T19:05:48.153879+0000", 2024-09-10T19:14:38.745 INFO:teuthology.orchestra.run.smithi183.stdout: "172.21.15.183:6800/3747320570": "2024-09-11T19:05:25.702354+0000", 2024-09-10T19:14:38.745 INFO:teuthology.orchestra.run.smithi183.stdout: "172.21.15.183:0/1652576282": "2024-09-11T19:10:43.576943+0000", 2024-09-10T19:14:38.745 INFO:teuthology.orchestra.run.smithi183.stdout: "172.21.15.183:6800/3311783362": "2024-09-11T19:10:43.576943+0000", 2024-09-10T19:14:38.745 INFO:teuthology.orchestra.run.smithi183.stdout: "172.21.15.183:0/1012899549": "2024-09-11T19:05:48.153879+0000", 2024-09-10T19:14:38.745 INFO:teuthology.orchestra.run.smithi183.stdout: "172.21.15.183:6800/659745765": "2024-09-11T19:05:48.153879+0000", 2024-09-10T19:14:38.745 INFO:teuthology.orchestra.run.smithi183.stdout: "172.21.15.183:0/412044400": "2024-09-11T19:10:43.576943+0000", 2024-09-10T19:14:38.745 INFO:teuthology.orchestra.run.smithi183.stdout: "172.21.15.183:0/3636171401": "2024-09-11T19:10:43.576943+0000" 2024-09-10T19:14:38.745 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:14:38.745 INFO:teuthology.orchestra.run.smithi183.stdout: "range_blocklist": {}, 2024-09-10T19:14:38.745 INFO:teuthology.orchestra.run.smithi183.stdout: "erasure_code_profiles": { 2024-09-10T19:14:38.746 INFO:teuthology.orchestra.run.smithi183.stdout: "default": { 2024-09-10T19:14:38.746 INFO:teuthology.orchestra.run.smithi183.stdout: "crush-failure-domain": "osd", 2024-09-10T19:14:38.746 INFO:teuthology.orchestra.run.smithi183.stdout: "k": "2", 2024-09-10T19:14:38.746 INFO:teuthology.orchestra.run.smithi183.stdout: "m": "1", 2024-09-10T19:14:38.746 INFO:teuthology.orchestra.run.smithi183.stdout: "plugin": "jerasure", 2024-09-10T19:14:38.746 INFO:teuthology.orchestra.run.smithi183.stdout: "technique": "reed_sol_van" 2024-09-10T19:14:38.746 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:14:38.746 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:14:38.746 INFO:teuthology.orchestra.run.smithi183.stdout: "removed_snaps_queue": [], 2024-09-10T19:14:38.746 INFO:teuthology.orchestra.run.smithi183.stdout: "new_removed_snaps": [], 2024-09-10T19:14:38.746 INFO:teuthology.orchestra.run.smithi183.stdout: "new_purged_snaps": [], 2024-09-10T19:14:38.746 INFO:teuthology.orchestra.run.smithi183.stdout: "crush_node_flags": {}, 2024-09-10T19:14:38.746 INFO:teuthology.orchestra.run.smithi183.stdout: "device_class_flags": {}, 2024-09-10T19:14:38.746 INFO:teuthology.orchestra.run.smithi183.stdout: "stretch_mode": { 2024-09-10T19:14:38.747 INFO:teuthology.orchestra.run.smithi183.stdout: "stretch_mode_enabled": false, 2024-09-10T19:14:38.747 INFO:teuthology.orchestra.run.smithi183.stdout: "stretch_bucket_count": 0, 2024-09-10T19:14:38.747 INFO:teuthology.orchestra.run.smithi183.stdout: "degraded_stretch_mode": 0, 2024-09-10T19:14:38.747 INFO:teuthology.orchestra.run.smithi183.stdout: "recovering_stretch_mode": 0, 2024-09-10T19:14:38.747 INFO:teuthology.orchestra.run.smithi183.stdout: "stretch_mode_bucket": 0 2024-09-10T19:14:38.747 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:14:38.747 INFO:teuthology.orchestra.run.smithi183.stdout:} 2024-09-10T19:14:38.747 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:14:38.776 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:14:38.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:14:38.777 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:14:38.878 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:14:38.905 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:14:38.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:14:38.906 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:14:38.931 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:14:38 smithi183 ceph-mon[29736]: 6.1a scrub starts 2024-09-10T19:14:38.931 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:14:38 smithi183 ceph-mon[29736]: pgmap v170: 148 pgs: 6 active+undersized+degraded+remapped+backfilling, 142 active+clean; 5.3 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.16k op/s; 17230/260613 objects degraded (6.611%); 45 MiB/s, 725 objects/s recovering 2024-09-10T19:14:38.931 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:14:38 smithi183 ceph-mon[29736]: from='client.? 172.21.15.183:0/2274115313' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-09-10T19:14:38.931 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:14:38 smithi183 ceph-mon[41562]: 6.1a scrub starts 2024-09-10T19:14:38.931 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:14:38 smithi183 ceph-mon[41562]: pgmap v170: 148 pgs: 6 active+undersized+degraded+remapped+backfilling, 142 active+clean; 5.3 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.16k op/s; 17230/260613 objects degraded (6.611%); 45 MiB/s, 725 objects/s recovering 2024-09-10T19:14:38.931 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:14:38 smithi183 ceph-mon[41562]: from='client.? 172.21.15.183:0/2274115313' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-09-10T19:14:38.965 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:14:38 smithi191 ceph-mon[32884]: 6.1a scrub starts 2024-09-10T19:14:38.965 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:14:38 smithi191 ceph-mon[32884]: pgmap v170: 148 pgs: 6 active+undersized+degraded+remapped+backfilling, 142 active+clean; 5.3 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.16k op/s; 17230/260613 objects degraded (6.611%); 45 MiB/s, 725 objects/s recovering 2024-09-10T19:14:38.965 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:14:38 smithi191 ceph-mon[32884]: from='client.? 172.21.15.183:0/2274115313' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-09-10T19:14:39.007 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:14:39.034 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:14:39.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:14:39.036 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:14:39.136 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:14:39.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:14:39.174 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:14:39.174 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:14:39.275 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:14:39.305 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:14:39.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:14:39.306 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:14:39.407 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:14:39.438 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:14:39.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:14:39.440 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:14:39.465 INFO:journalctl@ceph.iscsi.iscsi.a.smithi191.stdout:Sep 10 19:14:39 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-iscsi-iscsi-a[61816]: debug there is no tcmu-runner data available 2024-09-10T19:14:39.540 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:14:39.571 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:14:39.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:14:39.572 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:14:39.673 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:14:39.708 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:14:39.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:14:39.709 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:14:39.810 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:14:39.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:14:39.856 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:14:39.856 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:14:39.957 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:14:39.965 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:14:39 smithi191 ceph-mon[32884]: from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:14:39.965 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:14:39 smithi191 ceph-mon[32884]: Health check update: Degraded data redundancy: 17230/260613 objects degraded (6.611%), 6 pgs degraded (PG_DEGRADED) 2024-09-10T19:14:39.977 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:14:39.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:14:39.978 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:14:40.079 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:14:40.107 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:14:40.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:14:40.109 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:14:40.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:14:39 smithi183 ceph-mon[29736]: from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:14:40.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:14:39 smithi183 ceph-mon[29736]: Health check update: Degraded data redundancy: 17230/260613 objects degraded (6.611%), 6 pgs degraded (PG_DEGRADED) 2024-09-10T19:14:40.181 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:14:39 smithi183 ceph-mon[41562]: from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:14:40.181 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:14:39 smithi183 ceph-mon[41562]: Health check update: Degraded data redundancy: 17230/260613 objects degraded (6.611%), 6 pgs degraded (PG_DEGRADED) 2024-09-10T19:14:40.209 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:14:40.243 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:14:40.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:14:40.245 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:14:40.345 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:14:40.381 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:14:40.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:14:40.383 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:14:40.483 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:14:40.515 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:14:40.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:14:40.516 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:14:40.617 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:14:40.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:14:40.653 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:14:40.653 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:14:40.754 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:14:40.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:14:40.785 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:14:40.785 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:14:40.882 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:14:40 smithi191 ceph-mon[32884]: pgmap v171: 148 pgs: 6 active+undersized+degraded+remapped+backfilling, 142 active+clean; 5.2 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 856 B/s rd, 0 B/s wr, 1.13k op/s; 14940/254736 objects degraded (5.865%); 47 MiB/s, 753 objects/s recovering 2024-09-10T19:14:40.886 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:14:40.926 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:14:40.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:14:40.927 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:14:41.028 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:14:41.063 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:14:41.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:14:41.064 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:14:41.160 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:14:40 smithi183 ceph-mon[29736]: pgmap v171: 148 pgs: 6 active+undersized+degraded+remapped+backfilling, 142 active+clean; 5.2 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 856 B/s rd, 0 B/s wr, 1.13k op/s; 14940/254736 objects degraded (5.865%); 47 MiB/s, 753 objects/s recovering 2024-09-10T19:14:41.160 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:14:40 smithi183 ceph-mon[41562]: pgmap v171: 148 pgs: 6 active+undersized+degraded+remapped+backfilling, 142 active+clean; 5.2 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 856 B/s rd, 0 B/s wr, 1.13k op/s; 14940/254736 objects degraded (5.865%); 47 MiB/s, 753 objects/s recovering 2024-09-10T19:14:41.165 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:14:41.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:14:41.201 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:14:41.201 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:14:41.302 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:14:41.337 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:14:41.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:14:41.338 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:14:41.348 INFO:teuthology.orchestra.run.smithi191.stdout:ERROR: (22) Invalid argument 2024-09-10T19:14:41.348 INFO:teuthology.orchestra.run.smithi191.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-10T19:14:41.431 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Sep 10 19:14:41 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-y[29989]: ::ffff:172.21.15.191 - - [10/Sep/2024:19:14:41] "GET /metrics HTTP/1.1" 200 209087 "" "Prometheus/2.43.0" 2024-09-10T19:14:41.439 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:14:41.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:14:41.476 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:14:41.476 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:14:41.577 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:14:41.611 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:14:41.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:14:41.612 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:14:41.713 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:14:41.752 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:14:41.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:14:41.753 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:14:41.813 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:14:41.850 INFO:tasks.thrashosds.thrasher:j is {'epoch': 123, 'fsid': '320f6df8-6fa7-11ef-bcea-c7b262605968', 'created': '2024-09-10T19:05:03.688531+0000', 'modified': '2024-09-10T19:14:27.708730+0000', 'last_up_change': '2024-09-10T19:12:45.801705+0000', 'last_in_change': '2024-09-10T19:13:53.915409+0000', 'flags': 'nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5804032, 'flags_set': ['nodeep-scrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 18, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 7, 'max_osd': 8, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'quincy', 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-10T19:08:00.456786+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}, {'pool': 2, 'pool_name': '.rgw.root', 'create_time': '2024-09-10T19:10:05.993763+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '60', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}}, {'pool': 3, 'pool_name': 'datapool', 'create_time': '2024-09-10T19:10:06.224267+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 3, 'pg_placement_num': 3, 'pg_placement_num_target': 3, 'pg_num_target': 3, 'pg_num_pending': 3, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '63', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 3, 'snap_epoch': 63, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 4, 'pool_name': 'default.rgw.log', 'create_time': '2024-09-10T19:10:07.958284+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '62', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}}, {'pool': 5, 'pool_name': 'default.rgw.control', 'create_time': '2024-09-10T19:10:09.933596+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '64', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}}, {'pool': 6, 'pool_name': 'default.rgw.meta', 'create_time': '2024-09-10T19:10:11.993712+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '66', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4}, 'application_metadata': {'rgw': {}}}, {'pool': 7, 'pool_name': 'unique_pool_0', 'create_time': '2024-09-10T19:12:10.002625+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 16, 'pg_placement_num': 16, 'pg_placement_num_target': 16, 'pg_num_target': 16, 'pg_num_pending': 16, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '90', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}}], 'osds': [{'osd': 0, 'uuid': '9d323010-394a-49aa-ba9b-244d87faf36e', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 8, 'up_thru': 116, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.183:6802', 'nonce': 2813074952}, {'type': 'v1', 'addr': '172.21.15.183:6803', 'nonce': 2813074952}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.183:6804', 'nonce': 2813074952}, {'type': 'v1', 'addr': '172.21.15.183:6805', 'nonce': 2813074952}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.183:6808', 'nonce': 2813074952}, {'type': 'v1', 'addr': '172.21.15.183:6809', 'nonce': 2813074952}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.183:6806', 'nonce': 2813074952}, {'type': 'v1', 'addr': '172.21.15.183:6807', 'nonce': 2813074952}]}, 'public_addr': '172.21.15.183:6803/2813074952', 'cluster_addr': '172.21.15.183:6805/2813074952', 'heartbeat_back_addr': '172.21.15.183:6809/2813074952', 'heartbeat_front_addr': '172.21.15.183:6807/2813074952', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': '71a6c5dc-5109-4284-8d20-187d8e7afc9e', 'up': 0, 'in': 0, 'weight': 0, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 86, 'down_at': 99, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.183:6810', 'nonce': 3880626114}, {'type': 'v1', 'addr': '172.21.15.183:6811', 'nonce': 3880626114}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.183:6812', 'nonce': 3880626114}, {'type': 'v1', 'addr': '172.21.15.183:6813', 'nonce': 3880626114}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.183:6816', 'nonce': 3880626114}, {'type': 'v1', 'addr': '172.21.15.183:6817', 'nonce': 3880626114}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.183:6814', 'nonce': 3880626114}, {'type': 'v1', 'addr': '172.21.15.183:6815', 'nonce': 3880626114}]}, 'public_addr': '172.21.15.183:6811/3880626114', 'cluster_addr': '172.21.15.183:6813/3880626114', 'heartbeat_back_addr': '172.21.15.183:6817/3880626114', 'heartbeat_front_addr': '172.21.15.183:6815/3880626114', 'state': ['exists']}, {'osd': 2, 'uuid': 'fc310d5b-b89b-4655-9989-7e10ba356509', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 18, 'up_thru': 115, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.183:6818', 'nonce': 3730013786}, {'type': 'v1', 'addr': '172.21.15.183:6819', 'nonce': 3730013786}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.183:6820', 'nonce': 3730013786}, {'type': 'v1', 'addr': '172.21.15.183:6821', 'nonce': 3730013786}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.183:6824', 'nonce': 3730013786}, {'type': 'v1', 'addr': '172.21.15.183:6825', 'nonce': 3730013786}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.183:6822', 'nonce': 3730013786}, {'type': 'v1', 'addr': '172.21.15.183:6823', 'nonce': 3730013786}]}, 'public_addr': '172.21.15.183:6819/3730013786', 'cluster_addr': '172.21.15.183:6821/3730013786', 'heartbeat_back_addr': '172.21.15.183:6825/3730013786', 'heartbeat_front_addr': '172.21.15.183:6823/3730013786', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': '7fd07a95-ad22-43f5-a968-539552283e89', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 26, 'up_thru': 116, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.183:6826', 'nonce': 3522836984}, {'type': 'v1', 'addr': '172.21.15.183:6827', 'nonce': 3522836984}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.183:6828', 'nonce': 3522836984}, {'type': 'v1', 'addr': '172.21.15.183:6829', 'nonce': 3522836984}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.183:6832', 'nonce': 3522836984}, {'type': 'v1', 'addr': '172.21.15.183:6833', 'nonce': 3522836984}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.183:6830', 'nonce': 3522836984}, {'type': 'v1', 'addr': '172.21.15.183:6831', 'nonce': 3522836984}]}, 'public_addr': '172.21.15.183:6827/3522836984', 'cluster_addr': '172.21.15.183:6829/3522836984', 'heartbeat_back_addr': '172.21.15.183:6833/3522836984', 'heartbeat_front_addr': '172.21.15.183:6831/3522836984', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': '631083fb-e503-44da-9b6b-154e88539ee7', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 32, 'up_thru': 115, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.191:6800', 'nonce': 2674828539}, {'type': 'v1', 'addr': '172.21.15.191:6801', 'nonce': 2674828539}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.191:6802', 'nonce': 2674828539}, {'type': 'v1', 'addr': '172.21.15.191:6803', 'nonce': 2674828539}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.191:6806', 'nonce': 2674828539}, {'type': 'v1', 'addr': '172.21.15.191:6807', 'nonce': 2674828539}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.191:6804', 'nonce': 2674828539}, {'type': 'v1', 'addr': '172.21.15.191:6805', 'nonce': 2674828539}]}, 'public_addr': '172.21.15.191:6801/2674828539', 'cluster_addr': '172.21.15.191:6803/2674828539', 'heartbeat_back_addr': '172.21.15.191:6807/2674828539', 'heartbeat_front_addr': '172.21.15.191:6805/2674828539', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': '6b963695-3eca-4e6e-9207-05b8010ebff9', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 39, 'up_thru': 116, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.191:6808', 'nonce': 681733378}, {'type': 'v1', 'addr': '172.21.15.191:6809', 'nonce': 681733378}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.191:6810', 'nonce': 681733378}, {'type': 'v1', 'addr': '172.21.15.191:6811', 'nonce': 681733378}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.191:6814', 'nonce': 681733378}, {'type': 'v1', 'addr': '172.21.15.191:6815', 'nonce': 681733378}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.191:6812', 'nonce': 681733378}, {'type': 'v1', 'addr': '172.21.15.191:6813', 'nonce': 681733378}]}, 'public_addr': '172.21.15.191:6809/681733378', 'cluster_addr': '172.21.15.191:6811/681733378', 'heartbeat_back_addr': '172.21.15.191:6815/681733378', 'heartbeat_front_addr': '172.21.15.191:6813/681733378', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': '2dc9754f-6aef-4e19-8605-2f1591921811', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 47, 'up_thru': 116, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.191:6816', 'nonce': 2825813477}, {'type': 'v1', 'addr': '172.21.15.191:6817', 'nonce': 2825813477}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.191:6818', 'nonce': 2825813477}, {'type': 'v1', 'addr': '172.21.15.191:6819', 'nonce': 2825813477}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.191:6822', 'nonce': 2825813477}, {'type': 'v1', 'addr': '172.21.15.191:6823', 'nonce': 2825813477}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.191:6820', 'nonce': 2825813477}, {'type': 'v1', 'addr': '172.21.15.191:6821', 'nonce': 2825813477}]}, 'public_addr': '172.21.15.191:6817/2825813477', 'cluster_addr': '172.21.15.191:6819/2825813477', 'heartbeat_back_addr': '172.21.15.191:6823/2825813477', 'heartbeat_front_addr': '172.21.15.191:6821/2825813477', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': '1a9a299e-d5ba-4b9a-85d5-7f6a5bc754e1', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 54, 'up_thru': 115, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.191:6824', 'nonce': 1422610021}, {'type': 'v1', 'addr': '172.21.15.191:6825', 'nonce': 1422610021}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.191:6826', 'nonce': 1422610021}, {'type': 'v1', 'addr': '172.21.15.191:6827', 'nonce': 1422610021}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.191:6830', 'nonce': 1422610021}, {'type': 'v1', 'addr': '172.21.15.191:6831', 'nonce': 1422610021}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.191:6828', 'nonce': 1422610021}, {'type': 'v1', 'addr': '172.21.15.191:6829', 'nonce': 1422610021}]}, 'public_addr': '172.21.15.191:6825/1422610021', 'cluster_addr': '172.21.15.191:6827/1422610021', 'heartbeat_back_addr': '172.21.15.191:6831/1422610021', 'heartbeat_front_addr': '172.21.15.191:6829/1422610021', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138320759226367, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-10T19:14:08.336733+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138320759226367, 'old_weight': 65536, 'last_purged_snaps_scrub': '2024-09-10T19:12:08.593593+0000', 'dead_epoch': 98}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138320759226367, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-10T19:14:06.494190+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138320759226367, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-10T19:13:32.119763+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138320759226367, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-10T19:13:34.944494+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138320759226367, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-10T19:13:49.264552+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138320759226367, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-10T19:13:40.759766+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138320759226367, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-10T19:13:31.920321+0000', 'dead_epoch': 0}], 'pg_upmap': [], 'pg_upmap_items': [{'pgid': '2.1f', 'mappings': [{'from': 7, 'to': 2}]}], 'pg_temp': [{'pgid': '7.0', 'osds': [5, 3]}, {'pgid': '7.2', 'osds': [3, 2]}, {'pgid': '7.4', 'osds': [0, 4]}, {'pgid': '7.9', 'osds': [0, 4]}, {'pgid': '7.b', 'osds': [5, 6]}, {'pgid': '7.d', 'osds': [6, 0]}], 'primary_temp': [], 'blocklist': {'172.21.15.183:6801/3311783362': '2024-09-11T19:10:43.576943+0000', '172.21.15.183:0/4088553922': '2024-09-11T19:10:43.576943+0000', '172.21.15.183:0/3528769414': '2024-09-11T19:05:48.153879+0000', '172.21.15.183:0/1766036658': '2024-09-11T19:05:48.153879+0000', '172.21.15.183:0/3044170088': '2024-09-11T19:05:25.702354+0000', '172.21.15.183:0/3037333778': '2024-09-11T19:05:25.702354+0000', '172.21.15.183:6801/3747320570': '2024-09-11T19:05:25.702354+0000', '172.21.15.183:0/3982386906': '2024-09-11T19:10:43.576943+0000', '172.21.15.183:0/2436014455': '2024-09-11T19:05:25.702354+0000', '172.21.15.183:6801/659745765': '2024-09-11T19:05:48.153879+0000', '172.21.15.183:6800/3747320570': '2024-09-11T19:05:25.702354+0000', '172.21.15.183:0/1652576282': '2024-09-11T19:10:43.576943+0000', '172.21.15.183:6800/3311783362': '2024-09-11T19:10:43.576943+0000', '172.21.15.183:0/1012899549': '2024-09-11T19:05:48.153879+0000', '172.21.15.183:6800/659745765': '2024-09-11T19:05:48.153879+0000', '172.21.15.183:0/412044400': '2024-09-11T19:10:43.576943+0000', '172.21.15.183:0/3636171401': '2024-09-11T19:10:43.576943+0000'}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-09-10T19:14:41.850 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph pg dump --format=json 2024-09-10T19:14:41.854 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:14:41.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:14:41.945 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:14:41.945 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:14:41.965 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:14:41 smithi191 ceph-mon[32884]: 4.13 scrub starts 2024-09-10T19:14:42.046 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:14:42.068 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:14:41 smithi183 ceph-mon[41562]: 4.13 scrub starts 2024-09-10T19:14:42.069 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:14:41 smithi183 ceph-mon[29736]: 4.13 scrub starts 2024-09-10T19:14:42.075 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:14:42.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:14:42.075 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:14:42.164 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph daemon osd.0 dump_ops_in_flight 2024-09-10T19:14:42.177 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:14:42.205 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:14:42.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:14:42.206 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:14:42.307 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:14:42.336 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:14:42.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:14:42.337 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:14:42.438 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:14:42.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:14:42.465 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:14:42.465 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:14:42.566 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:14:42.601 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:14:42.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:14:42.602 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:14:42.703 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:14:42.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:14:42.756 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:14:42.756 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:14:42.798 INFO:teuthology.orchestra.run.smithi183.stderr:nodeep-scrub is unset 2024-09-10T19:14:42.858 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:14:42.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:14:42.893 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:14:42.893 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:14:42.994 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:14:43.028 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:14:43.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:14:43.029 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:14:43.130 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:14:43.160 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:14:43.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:14:43.162 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:14:43.182 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:14:42 smithi183 ceph-mon[29736]: 5.1d scrub starts 2024-09-10T19:14:43.182 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:14:42 smithi183 ceph-mon[29736]: pgmap v172: 148 pgs: 6 active+undersized+degraded+remapped+backfilling, 142 active+clean; 5.0 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.19k op/s; 12627/244644 objects degraded (5.161%); 49 MiB/s, 778 objects/s recovering 2024-09-10T19:14:43.182 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:14:42 smithi183 ceph-mon[29736]: from='client.? 172.21.15.183:0/1875931414' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-10T19:14:43.182 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:14:42 smithi183 ceph-mon[29736]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-10T19:14:43.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:14:42 smithi183 ceph-mon[41562]: 5.1d scrub starts 2024-09-10T19:14:43.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:14:42 smithi183 ceph-mon[41562]: pgmap v172: 148 pgs: 6 active+undersized+degraded+remapped+backfilling, 142 active+clean; 5.0 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.19k op/s; 12627/244644 objects degraded (5.161%); 49 MiB/s, 778 objects/s recovering 2024-09-10T19:14:43.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:14:42 smithi183 ceph-mon[41562]: from='client.? 172.21.15.183:0/1875931414' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-10T19:14:43.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:14:42 smithi183 ceph-mon[41562]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-10T19:14:43.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:14:42 smithi191 ceph-mon[32884]: 5.1d scrub starts 2024-09-10T19:14:43.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:14:42 smithi191 ceph-mon[32884]: pgmap v172: 148 pgs: 6 active+undersized+degraded+remapped+backfilling, 142 active+clean; 5.0 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.19k op/s; 12627/244644 objects degraded (5.161%); 49 MiB/s, 778 objects/s recovering 2024-09-10T19:14:43.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:14:42 smithi191 ceph-mon[32884]: from='client.? 172.21.15.183:0/1875931414' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-10T19:14:43.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:14:42 smithi191 ceph-mon[32884]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-10T19:14:43.262 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:14:43.303 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:14:43.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:14:43.304 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:14:43.405 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:14:43.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:14:43.444 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:14:43.445 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:14:43.546 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:14:43.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:14:43.578 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:14:43.578 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:14:43.635 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:14:43.679 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:14:43.707 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:14:43.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:14:43.708 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:14:43.809 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:14:43.845 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:14:43.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:14:43.846 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:14:43.931 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Sep 10 19:14:43 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-y[29989]: 2024-09-10T19:14:43.673+0000 7f9fad6cd640 -1 mgr get_metadata_python Requested missing service osd.1 2024-09-10T19:14:43.931 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:14:43 smithi183 ceph-mon[29736]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-10T19:14:43.931 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:14:43 smithi183 ceph-mon[29736]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-10T19:14:43.931 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:14:43 smithi183 ceph-mon[29736]: osdmap e124: 8 total, 7 up, 7 in 2024-09-10T19:14:43.932 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:14:43 smithi183 ceph-mon[29736]: from='mgr.14556 ' entity='mgr.y' 2024-09-10T19:14:43.932 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:14:43 smithi183 ceph-mon[29736]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:14:43.932 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:14:43 smithi183 ceph-mon[41562]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-10T19:14:43.932 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:14:43 smithi183 ceph-mon[41562]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-10T19:14:43.932 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:14:43 smithi183 ceph-mon[41562]: osdmap e124: 8 total, 7 up, 7 in 2024-09-10T19:14:43.932 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:14:43 smithi183 ceph-mon[41562]: from='mgr.14556 ' entity='mgr.y' 2024-09-10T19:14:43.932 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:14:43 smithi183 ceph-mon[41562]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:14:43.947 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:14:43.977 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:14:43.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:14:43.978 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:14:44.079 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:14:44.114 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:14:44.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:14:44.115 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:14:44.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:14:43 smithi191 ceph-mon[32884]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-10T19:14:44.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:14:43 smithi191 ceph-mon[32884]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-10T19:14:44.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:14:43 smithi191 ceph-mon[32884]: osdmap e124: 8 total, 7 up, 7 in 2024-09-10T19:14:44.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:14:43 smithi191 ceph-mon[32884]: from='mgr.14556 ' entity='mgr.y' 2024-09-10T19:14:44.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:14:43 smithi191 ceph-mon[32884]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:14:44.216 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:14:44.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:14:44.244 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:14:44.245 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:14:44.345 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:14:44.377 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:14:44.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:14:44.379 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:14:44.480 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:14:44.531 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:14:44.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:14:44.532 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:14:44.632 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:14:44.678 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:14:44.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:14:44.679 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:14:44.780 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:14:44.807 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:14:44.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:14:44.808 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:14:44.908 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:14:44.937 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:14:44.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:14:44.938 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:14:45.039 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:14:45.064 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:14:45.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:14:45.066 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:14:45.166 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:14:45.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:14:44 smithi183 ceph-mon[29736]: 2.d scrub starts 2024-09-10T19:14:45.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:14:44 smithi183 ceph-mon[29736]: pgmap v174: 148 pgs: 6 active+undersized+degraded+remapped+backfilling, 142 active+clean; 5.0 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.10k op/s; 12627/244644 objects degraded (5.161%); 48 MiB/s, 762 objects/s recovering 2024-09-10T19:14:45.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:14:44 smithi183 ceph-mon[29736]: Health check update: Degraded data redundancy: 12627/244644 objects degraded (5.161%), 6 pgs degraded (PG_DEGRADED) 2024-09-10T19:14:45.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:14:44 smithi183 ceph-mon[29736]: osdmap e125: 8 total, 7 up, 7 in 2024-09-10T19:14:45.181 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:14:44 smithi183 ceph-mon[41562]: 2.d scrub starts 2024-09-10T19:14:45.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:14:44 smithi183 ceph-mon[41562]: pgmap v174: 148 pgs: 6 active+undersized+degraded+remapped+backfilling, 142 active+clean; 5.0 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.10k op/s; 12627/244644 objects degraded (5.161%); 48 MiB/s, 762 objects/s recovering 2024-09-10T19:14:45.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:14:44 smithi183 ceph-mon[41562]: Health check update: Degraded data redundancy: 12627/244644 objects degraded (5.161%), 6 pgs degraded (PG_DEGRADED) 2024-09-10T19:14:45.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:14:44 smithi183 ceph-mon[41562]: osdmap e125: 8 total, 7 up, 7 in 2024-09-10T19:14:45.182 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:14:45.191 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:14:45.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:14:45.193 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:14:45.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:14:44 smithi191 ceph-mon[32884]: 2.d scrub starts 2024-09-10T19:14:45.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:14:44 smithi191 ceph-mon[32884]: pgmap v174: 148 pgs: 6 active+undersized+degraded+remapped+backfilling, 142 active+clean; 5.0 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.10k op/s; 12627/244644 objects degraded (5.161%); 48 MiB/s, 762 objects/s recovering 2024-09-10T19:14:45.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:14:44 smithi191 ceph-mon[32884]: Health check update: Degraded data redundancy: 12627/244644 objects degraded (5.161%), 6 pgs degraded (PG_DEGRADED) 2024-09-10T19:14:45.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:14:44 smithi191 ceph-mon[32884]: osdmap e125: 8 total, 7 up, 7 in 2024-09-10T19:14:45.296 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:14:45.321 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:14:45.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:14:45.323 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:14:45.424 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:14:45.451 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:14:45.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:14:45.452 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:14:45.553 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:14:45.578 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:14:45.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:14:45.579 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:14:45.594 INFO:teuthology.orchestra.run.smithi183.stdout:osd.0: {} 2024-09-10T19:14:45.595 INFO:teuthology.orchestra.run.smithi183.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-10T19:14:45.595 INFO:teuthology.orchestra.run.smithi183.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-09-10T19:14:45.615 INFO:teuthology.orchestra.run.smithi183.stdout:osd.2: {} 2024-09-10T19:14:45.616 INFO:teuthology.orchestra.run.smithi183.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-10T19:14:45.639 INFO:teuthology.orchestra.run.smithi183.stdout:osd.3: {} 2024-09-10T19:14:45.639 INFO:teuthology.orchestra.run.smithi183.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-10T19:14:45.656 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph osd set noscrub 2024-09-10T19:14:45.657 INFO:teuthology.orchestra.run.smithi183.stdout:osd.4: {} 2024-09-10T19:14:45.658 INFO:teuthology.orchestra.run.smithi183.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-10T19:14:45.677 INFO:teuthology.orchestra.run.smithi183.stdout:osd.5: {} 2024-09-10T19:14:45.678 INFO:teuthology.orchestra.run.smithi183.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-10T19:14:45.680 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:14:45.699 INFO:teuthology.orchestra.run.smithi183.stdout:osd.6: {} 2024-09-10T19:14:45.699 INFO:teuthology.orchestra.run.smithi183.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-10T19:14:45.704 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:14:45.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:14:45.709 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:14:45.721 INFO:teuthology.orchestra.run.smithi183.stdout:osd.7: {} 2024-09-10T19:14:45.722 INFO:teuthology.orchestra.run.smithi183.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-10T19:14:45.809 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:14:45.845 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:14:45.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:14:45.846 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:14:45.947 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:14:45.973 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:14:45.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:14:45.974 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:14:46.074 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:14:45 smithi183 ceph-mon[29736]: 4.9 scrub starts 2024-09-10T19:14:46.074 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:14:45 smithi183 ceph-mon[29736]: from='client.? 172.21.15.183:0/1568802768' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T19:14:46.074 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:14:45 smithi183 ceph-mon[29736]: osdmap e126: 8 total, 7 up, 7 in 2024-09-10T19:14:46.074 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:14:45 smithi183 ceph-mon[41562]: 4.9 scrub starts 2024-09-10T19:14:46.075 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:14:45 smithi183 ceph-mon[41562]: from='client.? 172.21.15.183:0/1568802768' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T19:14:46.075 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:14:45 smithi183 ceph-mon[41562]: osdmap e126: 8 total, 7 up, 7 in 2024-09-10T19:14:46.075 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:14:46.104 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:14:46.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:14:46.105 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:14:46.206 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:14:46.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:14:45 smithi191 ceph-mon[32884]: 4.9 scrub starts 2024-09-10T19:14:46.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:14:45 smithi191 ceph-mon[32884]: from='client.? 172.21.15.183:0/1568802768' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T19:14:46.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:14:45 smithi191 ceph-mon[32884]: osdmap e126: 8 total, 7 up, 7 in 2024-09-10T19:14:46.232 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:14:46.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:14:46.233 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:14:46.334 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:14:46.359 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:14:46.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:14:46.361 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:14:46.461 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:14:46.484 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:14:46.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:14:46.485 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:14:46.586 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:14:46.610 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:14:46.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:14:46.611 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:14:46.712 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:14:46.738 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:14:46.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:14:46.739 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:14:46.840 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:14:46.871 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:14:46.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:14:46.872 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:14:46.973 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:14:47.004 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:14:47.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:14:47.005 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:14:47.107 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:14:47.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:14:47.138 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:14:47.138 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:14:47.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:14:46 smithi183 ceph-mon[29736]: pgmap v176: 148 pgs: 1 peering, 5 active+undersized+degraded+remapped+backfilling, 142 active+clean; 4.6 GiB data, 15 GiB used, 610 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.40k op/s; 7919/227046 objects degraded (3.488%); 60 MiB/s, 961 objects/s recovering 2024-09-10T19:14:47.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:14:46 smithi183 ceph-mon[29736]: osdmap e127: 8 total, 7 up, 7 in 2024-09-10T19:14:47.181 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:14:46 smithi183 ceph-mon[41562]: pgmap v176: 148 pgs: 1 peering, 5 active+undersized+degraded+remapped+backfilling, 142 active+clean; 4.6 GiB data, 15 GiB used, 610 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.40k op/s; 7919/227046 objects degraded (3.488%); 60 MiB/s, 961 objects/s recovering 2024-09-10T19:14:47.181 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:14:46 smithi183 ceph-mon[41562]: osdmap e127: 8 total, 7 up, 7 in 2024-09-10T19:14:47.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:14:46 smithi191 ceph-mon[32884]: pgmap v176: 148 pgs: 1 peering, 5 active+undersized+degraded+remapped+backfilling, 142 active+clean; 4.6 GiB data, 15 GiB used, 610 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.40k op/s; 7919/227046 objects degraded (3.488%); 60 MiB/s, 961 objects/s recovering 2024-09-10T19:14:47.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:14:46 smithi191 ceph-mon[32884]: osdmap e127: 8 total, 7 up, 7 in 2024-09-10T19:14:47.239 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:14:47.275 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:14:47.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:14:47.276 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:14:47.377 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:14:47.411 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:14:47.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:14:47.412 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:14:47.513 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:14:47.534 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:14:47.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:14:47.535 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:14:47.636 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:14:47.667 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:14:47.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:14:47.668 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:14:47.715 INFO:journalctl@ceph.mgr.x.smithi191.stdout:Sep 10 19:14:47 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-x[34967]: ::ffff:172.21.15.191 - - [10/Sep/2024:19:14:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T19:14:47.769 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:14:47.799 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:14:47.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:14:47.801 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:14:47.901 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:14:47.935 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:14:47.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:14:47.936 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:14:48.037 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:14:48.070 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:14:48.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:14:48.070 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:14:48.171 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:14:48.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:14:47 smithi183 ceph-mon[29736]: 4.e scrub starts 2024-09-10T19:14:48.181 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:14:47 smithi183 ceph-mon[41562]: 4.e scrub starts 2024-09-10T19:14:48.210 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:14:48.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:14:48.211 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:14:48.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:14:47 smithi191 ceph-mon[32884]: 4.e scrub starts 2024-09-10T19:14:48.312 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:14:48.350 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:14:48.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:14:48.351 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:14:48.452 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:14:48.484 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:14:48.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:14:48.485 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:14:48.586 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:14:48.619 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:14:48.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:14:48.620 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:14:48.720 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:14:48.752 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:14:48.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:14:48.753 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:14:48.854 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:14:48.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:14:48.889 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:14:48.889 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:14:48.931 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:14:48 smithi183 ceph-mon[29736]: pgmap v179: 148 pgs: 1 peering, 5 active+undersized+degraded+remapped+backfilling, 142 active+clean; 4.6 GiB data, 15 GiB used, 610 GiB / 626 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1.20k op/s; 7919/227046 objects degraded (3.488%); 51 MiB/s, 816 objects/s recovering 2024-09-10T19:14:48.931 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:14:48 smithi183 ceph-mon[41562]: pgmap v179: 148 pgs: 1 peering, 5 active+undersized+degraded+remapped+backfilling, 142 active+clean; 4.6 GiB data, 15 GiB used, 610 GiB / 626 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1.20k op/s; 7919/227046 objects degraded (3.488%); 51 MiB/s, 816 objects/s recovering 2024-09-10T19:14:48.990 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:14:49.019 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:14:49.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:14:49.020 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:14:49.074 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:14:48 smithi191 ceph-mon[32884]: pgmap v179: 148 pgs: 1 peering, 5 active+undersized+degraded+remapped+backfilling, 142 active+clean; 4.6 GiB data, 15 GiB used, 610 GiB / 626 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1.20k op/s; 7919/227046 objects degraded (3.488%); 51 MiB/s, 816 objects/s recovering 2024-09-10T19:14:49.121 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:14:49.160 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:14:49.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:14:49.161 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:14:49.262 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:14:49.291 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:14:49.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:14:49.292 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:14:49.393 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:14:49.421 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:14:49.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:14:49.422 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:14:49.465 INFO:journalctl@ceph.iscsi.iscsi.a.smithi191.stdout:Sep 10 19:14:49 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-iscsi-iscsi-a[61816]: debug there is no tcmu-runner data available 2024-09-10T19:14:49.523 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:14:49.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:14:49.557 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:14:49.557 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:14:49.658 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:14:49.688 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:14:49.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:14:49.689 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:14:49.790 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:14:49.819 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:14:49.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:14:49.820 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:14:49.920 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:14:49.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:14:49.962 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:14:49.962 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:14:49.977 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:14:49 smithi183 ceph-mon[29736]: from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:14:49.977 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:14:49 smithi183 ceph-mon[29736]: Health check update: Degraded data redundancy: 7919/227046 objects degraded (3.488%), 5 pgs degraded (PG_DEGRADED) 2024-09-10T19:14:49.977 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:14:49 smithi183 ceph-mon[29736]: osdmap e128: 8 total, 7 up, 7 in 2024-09-10T19:14:49.977 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:14:49 smithi183 ceph-mon[41562]: from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:14:49.977 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:14:49 smithi183 ceph-mon[41562]: Health check update: Degraded data redundancy: 7919/227046 objects degraded (3.488%), 5 pgs degraded (PG_DEGRADED) 2024-09-10T19:14:49.978 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:14:49 smithi183 ceph-mon[41562]: osdmap e128: 8 total, 7 up, 7 in 2024-09-10T19:14:49.979 INFO:teuthology.orchestra.run.smithi183.stdout:{ 2024-09-10T19:14:49.979 INFO:teuthology.orchestra.run.smithi183.stdout: "ops": [ 2024-09-10T19:14:49.979 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:14:49.979 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.14781.0:209685 7.f 7.9df7c15f (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e128)", 2024-09-10T19:14:49.979 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-10T19:14:49.960355+0000", 2024-09-10T19:14:49.979 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 0.014769852, 2024-09-10T19:14:49.979 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.014794761999999999, 2024-09-10T19:14:49.980 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-10T19:14:49.980 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "queued for pg", 2024-09-10T19:14:49.980 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-10T19:14:49.980 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.14781", 2024-09-10T19:14:49.980 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.183:0/2025776420", 2024-09-10T19:14:49.980 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 209685 2024-09-10T19:14:49.980 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:14:49.980 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-10T19:14:49.980 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:14:49.980 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-10T19:14:49.981 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:49.960355+0000", 2024-09-10T19:14:49.981 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:14:49.981 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:14:49.981 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:14:49.981 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-10T19:14:49.981 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:49.960355+0000", 2024-09-10T19:14:49.981 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:14:49.981 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:14:49.981 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:14:49.982 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-10T19:14:49.982 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:49.960354+0000", 2024-09-10T19:14:49.982 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4294967295.9999986 2024-09-10T19:14:49.982 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:14:49.982 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:14:49.982 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-10T19:14:49.982 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:49.960362+0000", 2024-09-10T19:14:49.982 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.1729999999999995e-06 2024-09-10T19:14:49.983 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:14:49.983 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:14:49.983 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-10T19:14:49.983 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:49.960375+0000", 2024-09-10T19:14:49.983 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.3379e-05 2024-09-10T19:14:49.983 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:14:49.983 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:14:49.983 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-10T19:14:49.984 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:49.960382+0000", 2024-09-10T19:14:49.984 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.8510000000000001e-06 2024-09-10T19:14:49.984 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:14:49.984 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:14:49.984 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:14:49.984 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:14:49.984 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:14:49.984 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.14781.0:209692 7.6 7:69712ac9:::benchmark_data_smithi183_84160_object69311:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e128)", 2024-09-10T19:14:49.985 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-10T19:14:49.964303+0000", 2024-09-10T19:14:49.985 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 0.010822076999999999, 2024-09-10T19:14:49.985 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.010916778, 2024-09-10T19:14:49.985 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-10T19:14:49.985 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "waiting for sub ops", 2024-09-10T19:14:49.985 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-10T19:14:49.985 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.14781", 2024-09-10T19:14:49.986 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.183:0/2025776420", 2024-09-10T19:14:49.986 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 209692 2024-09-10T19:14:49.986 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:14:49.986 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-10T19:14:49.986 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:14:49.986 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-10T19:14:49.986 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:49.964303+0000", 2024-09-10T19:14:49.987 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:14:49.987 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:14:49.987 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:14:49.987 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-10T19:14:49.987 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:49.964303+0000", 2024-09-10T19:14:49.987 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:14:49.987 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:14:49.987 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:14:49.988 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-10T19:14:49.988 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:49.964302+0000", 2024-09-10T19:14:49.988 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4294967295.999999 2024-09-10T19:14:49.988 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:14:49.988 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:14:49.988 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-10T19:14:49.988 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:49.964307+0000", 2024-09-10T19:14:49.988 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.536e-06 2024-09-10T19:14:49.989 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:14:49.989 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:14:49.989 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-10T19:14:49.989 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:49.964319+0000", 2024-09-10T19:14:49.989 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.1426999999999999e-05 2024-09-10T19:14:49.989 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:14:49.989 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:14:49.990 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-10T19:14:49.990 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:49.964324+0000", 2024-09-10T19:14:49.990 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.7010000000000001e-06 2024-09-10T19:14:49.990 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:14:49.990 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:14:49.990 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-10T19:14:49.990 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:49.964374+0000", 2024-09-10T19:14:49.991 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.9780000000000001e-05 2024-09-10T19:14:49.991 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:14:49.991 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:14:49.991 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-10T19:14:49.991 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:49.964500+0000", 2024-09-10T19:14:49.991 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000126026 2024-09-10T19:14:49.991 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:14:49.991 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:14:49.991 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 3,7", 2024-09-10T19:14:49.992 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:49.964620+0000", 2024-09-10T19:14:49.992 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000120066 2024-09-10T19:14:49.992 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:14:49.992 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:14:49.992 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:14:49.992 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:49.971916+0000", 2024-09-10T19:14:49.992 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0072963840000000004 2024-09-10T19:14:49.992 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:14:49.992 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:14:49.993 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-10T19:14:49.993 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:49.972323+0000", 2024-09-10T19:14:49.993 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00040605200000000001 2024-09-10T19:14:49.993 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:14:49.993 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:14:49.993 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:14:49.993 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:14:49.994 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:14:49.994 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.14781.0:209694 7.6 7:6df079b4:::benchmark_data_smithi183_84160_object69313:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e128)", 2024-09-10T19:14:49.994 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-10T19:14:49.965016+0000", 2024-09-10T19:14:49.994 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 0.010108903000000001, 2024-09-10T19:14:49.994 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.010291959, 2024-09-10T19:14:49.994 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-10T19:14:49.994 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "waiting for sub ops", 2024-09-10T19:14:49.994 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-10T19:14:49.995 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.14781", 2024-09-10T19:14:49.995 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.183:0/2025776420", 2024-09-10T19:14:49.995 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 209694 2024-09-10T19:14:49.995 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:14:49.995 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-10T19:14:49.995 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:14:49.995 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-10T19:14:49.995 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:49.965016+0000", 2024-09-10T19:14:49.995 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:14:49.996 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:14:49.996 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:14:49.996 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-10T19:14:49.996 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:49.965016+0000", 2024-09-10T19:14:49.996 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:14:49.996 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:14:49.996 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:14:49.996 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-10T19:14:49.996 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:49.965015+0000", 2024-09-10T19:14:49.996 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4294967295.999999 2024-09-10T19:14:49.996 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:14:49.997 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:14:49.997 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-10T19:14:49.997 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:49.965019+0000", 2024-09-10T19:14:49.997 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.4039999999999997e-06 2024-09-10T19:14:49.997 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:14:49.997 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:14:49.997 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-10T19:14:49.997 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:49.965033+0000", 2024-09-10T19:14:49.997 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.38e-05 2024-09-10T19:14:49.997 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:14:49.997 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:14:49.997 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-10T19:14:49.997 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:49.965039+0000", 2024-09-10T19:14:49.998 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.6509999999999998e-06 2024-09-10T19:14:49.998 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:14:49.998 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:14:49.998 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-10T19:14:49.998 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:49.968393+0000", 2024-09-10T19:14:49.998 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0033535829999999998 2024-09-10T19:14:49.998 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:14:49.998 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:14:49.998 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-10T19:14:49.998 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:49.968523+0000", 2024-09-10T19:14:49.998 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00013081900000000001 2024-09-10T19:14:49.998 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:14:49.998 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:14:49.999 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 3,7", 2024-09-10T19:14:49.999 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:49.968620+0000", 2024-09-10T19:14:49.999 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.6947999999999994e-05 2024-09-10T19:14:49.999 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:14:49.999 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:14:49.999 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:14:49.999 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:49.972008+0000", 2024-09-10T19:14:49.999 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.003387714 2024-09-10T19:14:49.999 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:14:49.999 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:14:49.999 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-10T19:14:49.999 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:49.972327+0000", 2024-09-10T19:14:49.999 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00031940499999999998 2024-09-10T19:14:49.999 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:14:50.000 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:14:50.000 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:14:50.000 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:14:50.000 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:14:50.000 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_repop(client.14781.0:209693 7.7 e128/86 7:eea409d0:::benchmark_data_smithi183_84160_object69312:head v 128'13140, mlcod=128'13136)", 2024-09-10T19:14:50.000 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-10T19:14:49.965105+0000", 2024-09-10T19:14:50.000 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 0.010019333, 2024-09-10T19:14:50.000 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.010288125, 2024-09-10T19:14:50.000 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-10T19:14:50.000 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "started", 2024-09-10T19:14:50.000 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-10T19:14:50.000 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:14:50.000 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-10T19:14:50.001 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:49.965105+0000", 2024-09-10T19:14:50.001 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:14:50.001 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:14:50.001 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:14:50.001 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-10T19:14:50.001 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:49.965105+0000", 2024-09-10T19:14:50.001 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:14:50.001 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:14:50.001 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:14:50.001 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-10T19:14:50.001 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:49.965105+0000", 2024-09-10T19:14:50.002 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4294967295.9999995 2024-09-10T19:14:50.002 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:14:50.002 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:14:50.002 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-10T19:14:50.002 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:49.965110+0000", 2024-09-10T19:14:50.002 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.57e-06 2024-09-10T19:14:50.002 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:14:50.002 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:14:50.002 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-10T19:14:50.002 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:49.965121+0000", 2024-09-10T19:14:50.002 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.0267e-05 2024-09-10T19:14:50.002 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:14:50.003 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:14:50.003 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-10T19:14:50.003 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:49.965126+0000", 2024-09-10T19:14:50.003 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.8029999999999996e-06 2024-09-10T19:14:50.003 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:14:50.003 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:14:50.003 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-10T19:14:50.003 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:49.971211+0000", 2024-09-10T19:14:50.003 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0060854439999999997 2024-09-10T19:14:50.003 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:14:50.003 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:14:50.003 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-10T19:14:50.003 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:49.971227+0000", 2024-09-10T19:14:50.004 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.6101999999999999e-05 2024-09-10T19:14:50.004 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:14:50.004 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:14:50.004 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:14:50.004 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:14:50.004 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:14:50.004 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.14781.0:209697 7.9 7:96f1b8e0:::benchmark_data_smithi183_84160_object69316:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e128)", 2024-09-10T19:14:50.004 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-10T19:14:49.967460+0000", 2024-09-10T19:14:50.004 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 0.0076651540000000004, 2024-09-10T19:14:50.004 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0079940159999999996, 2024-09-10T19:14:50.004 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-10T19:14:50.004 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "waiting for sub ops", 2024-09-10T19:14:50.004 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-10T19:14:50.005 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.14781", 2024-09-10T19:14:50.005 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.183:0/2025776420", 2024-09-10T19:14:50.005 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 209697 2024-09-10T19:14:50.005 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:14:50.005 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-10T19:14:50.005 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:14:50.005 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-10T19:14:50.005 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:49.967460+0000", 2024-09-10T19:14:50.005 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:14:50.005 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:14:50.005 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:14:50.005 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-10T19:14:50.005 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:49.967460+0000", 2024-09-10T19:14:50.006 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:14:50.006 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:14:50.006 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:14:50.006 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-10T19:14:50.006 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:49.967458+0000", 2024-09-10T19:14:50.006 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4294967295.999999 2024-09-10T19:14:50.006 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:14:50.006 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:14:50.006 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-10T19:14:50.006 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:49.967465+0000", 2024-09-10T19:14:50.006 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.0329999999999997e-06 2024-09-10T19:14:50.006 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:14:50.006 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:14:50.006 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-10T19:14:50.007 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:49.967477+0000", 2024-09-10T19:14:50.007 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.2813000000000001e-05 2024-09-10T19:14:50.007 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:14:50.007 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:14:50.007 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-10T19:14:50.007 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:49.967484+0000", 2024-09-10T19:14:50.007 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.4949999999999996e-06 2024-09-10T19:14:50.007 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:14:50.007 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:14:50.007 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-10T19:14:50.007 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:49.967613+0000", 2024-09-10T19:14:50.007 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00012896699999999999 2024-09-10T19:14:50.007 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:14:50.007 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:14:50.008 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-10T19:14:50.008 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:49.967719+0000", 2024-09-10T19:14:50.008 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00010590399999999999 2024-09-10T19:14:50.008 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:14:50.008 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:14:50.008 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 4,7", 2024-09-10T19:14:50.008 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:49.967840+0000", 2024-09-10T19:14:50.008 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000120868 2024-09-10T19:14:50.008 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:14:50.008 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:14:50.008 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:14:50.008 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:14:50.008 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:14:50.009 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.14781.0:209675 7.f 7:f8d722ff:::benchmark_data_smithi183_84160_object69294:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e128)", 2024-09-10T19:14:50.009 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-10T19:14:49.943366+0000", 2024-09-10T19:14:50.009 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 0.031759203999999999, 2024-09-10T19:14:50.009 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.032160572999999998, 2024-09-10T19:14:50.009 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-10T19:14:50.009 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "waiting for sub ops", 2024-09-10T19:14:50.009 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-10T19:14:50.009 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.14781", 2024-09-10T19:14:50.009 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.183:0/2025776420", 2024-09-10T19:14:50.009 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 209675 2024-09-10T19:14:50.009 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:14:50.009 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-10T19:14:50.009 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:14:50.009 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-10T19:14:50.010 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:49.943366+0000", 2024-09-10T19:14:50.010 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:14:50.010 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:14:50.010 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:14:50.010 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-10T19:14:50.010 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:49.943366+0000", 2024-09-10T19:14:50.010 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:14:50.010 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:14:50.010 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:14:50.010 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-10T19:14:50.010 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:49.943364+0000", 2024-09-10T19:14:50.010 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4294967295.999999 2024-09-10T19:14:50.010 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:14:50.011 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:14:50.011 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-10T19:14:50.011 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:49.943370+0000", 2024-09-10T19:14:50.011 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.7749999999999998e-06 2024-09-10T19:14:50.011 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:14:50.011 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:14:50.011 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-10T19:14:50.011 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:49.943380+0000", 2024-09-10T19:14:50.011 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.5619999999999993e-06 2024-09-10T19:14:50.011 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:14:50.011 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:14:50.011 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-10T19:14:50.011 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:49.943385+0000", 2024-09-10T19:14:50.011 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.2259999999999996e-06 2024-09-10T19:14:50.011 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:14:50.012 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:14:50.012 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-10T19:14:50.012 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:49.971939+0000", 2024-09-10T19:14:50.012 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.028554202000000001 2024-09-10T19:14:50.012 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:14:50.012 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:14:50.012 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-10T19:14:50.012 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:49.972149+0000", 2024-09-10T19:14:50.012 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000209452 2024-09-10T19:14:50.012 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:14:50.013 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:14:50.013 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 3,7", 2024-09-10T19:14:50.013 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:49.972858+0000", 2024-09-10T19:14:50.013 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00070927499999999999 2024-09-10T19:14:50.013 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:14:50.014 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:14:50.014 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:14:50.014 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:14:50.014 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:14:50.021 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.14781.0:209699 7.f 7.cc26215f (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e128)", 2024-09-10T19:14:50.021 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-10T19:14:49.969055+0000", 2024-09-10T19:14:50.021 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 0.0060696190000000001, 2024-09-10T19:14:50.021 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0065382499999999998, 2024-09-10T19:14:50.021 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-10T19:14:50.021 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "queued for pg", 2024-09-10T19:14:50.021 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-10T19:14:50.022 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.14781", 2024-09-10T19:14:50.022 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.183:0/2025776420", 2024-09-10T19:14:50.022 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 209699 2024-09-10T19:14:50.022 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:14:50.022 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-10T19:14:50.022 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:14:50.022 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-10T19:14:50.022 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:49.969055+0000", 2024-09-10T19:14:50.023 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:14:50.023 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:14:50.023 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:14:50.023 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-10T19:14:50.023 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:49.969055+0000", 2024-09-10T19:14:50.023 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:14:50.024 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:14:50.024 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:14:50.025 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-10T19:14:50.025 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:49.969054+0000", 2024-09-10T19:14:50.025 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4294967295.999999 2024-09-10T19:14:50.025 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:14:50.025 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:14:50.026 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-10T19:14:50.026 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:49.969059+0000", 2024-09-10T19:14:50.026 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.8420000000000004e-06 2024-09-10T19:14:50.026 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:14:50.026 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:14:50.026 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-10T19:14:50.026 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:49.969069+0000", 2024-09-10T19:14:50.026 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.4420000000000004e-06 2024-09-10T19:14:50.027 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:14:50.027 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:14:50.027 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-10T19:14:50.027 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:49.969074+0000", 2024-09-10T19:14:50.027 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.2850000000000002e-06 2024-09-10T19:14:50.027 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:14:50.027 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:14:50.027 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:14:50.027 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:14:50.027 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:14:50.027 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.14781.0:209701 7.6 7:6f5980f2:::benchmark_data_smithi183_84160_object69320:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e128)", 2024-09-10T19:14:50.027 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-10T19:14:49.971922+0000", 2024-09-10T19:14:50.028 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 0.0032031719999999998, 2024-09-10T19:14:50.028 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0037305279999999999, 2024-09-10T19:14:50.028 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-10T19:14:50.028 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "waiting for sub ops", 2024-09-10T19:14:50.028 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-10T19:14:50.028 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.14781", 2024-09-10T19:14:50.028 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.183:0/2025776420", 2024-09-10T19:14:50.028 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 209701 2024-09-10T19:14:50.028 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:14:50.028 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-10T19:14:50.028 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:14:50.028 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-10T19:14:50.029 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:49.971922+0000", 2024-09-10T19:14:50.029 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:14:50.029 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:14:50.029 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:14:50.029 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-10T19:14:50.029 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:49.971922+0000", 2024-09-10T19:14:50.029 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:14:50.030 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:14:50.030 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:14:50.030 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-10T19:14:50.030 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:49.971920+0000", 2024-09-10T19:14:50.030 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4294967295.999999 2024-09-10T19:14:50.030 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:14:50.030 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:14:50.030 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-10T19:14:50.030 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:49.971932+0000", 2024-09-10T19:14:50.030 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.1328999999999999e-05 2024-09-10T19:14:50.030 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:14:50.031 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:14:50.031 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-10T19:14:50.031 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:49.971945+0000", 2024-09-10T19:14:50.031 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.2799000000000001e-05 2024-09-10T19:14:50.031 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:14:50.031 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:14:50.031 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-10T19:14:50.031 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:49.971951+0000", 2024-09-10T19:14:50.031 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.2299999999999996e-06 2024-09-10T19:14:50.031 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:14:50.031 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:14:50.031 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-10T19:14:50.031 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:49.972861+0000", 2024-09-10T19:14:50.032 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00090996299999999996 2024-09-10T19:14:50.032 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:14:50.032 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:14:50.032 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-10T19:14:50.032 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:49.973021+0000", 2024-09-10T19:14:50.032 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00016063000000000001 2024-09-10T19:14:50.032 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:14:50.032 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:14:50.032 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 3,7", 2024-09-10T19:14:50.032 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:49.975556+0000", 2024-09-10T19:14:50.032 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0025350120000000001 2024-09-10T19:14:50.032 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:14:50.032 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:14:50.033 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:14:50.033 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:14:50.033 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:14:50.033 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_repop(client.14781.0:209700 7.5 e128/86 7:a4403f47:::benchmark_data_smithi183_84160_object69319:head v 128'13194, mlcod=128'13192)", 2024-09-10T19:14:50.033 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-10T19:14:49.972428+0000", 2024-09-10T19:14:50.033 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 0.0026963590000000002, 2024-09-10T19:14:50.033 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0032906110000000001, 2024-09-10T19:14:50.033 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-10T19:14:50.033 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "started", 2024-09-10T19:14:50.033 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-10T19:14:50.033 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:14:50.033 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-10T19:14:50.033 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:49.972428+0000", 2024-09-10T19:14:50.034 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:14:50.034 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:14:50.034 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:14:50.034 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-10T19:14:50.034 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:49.972428+0000", 2024-09-10T19:14:50.034 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:14:50.034 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:14:50.034 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:14:50.034 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-10T19:14:50.034 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:49.972428+0000", 2024-09-10T19:14:50.034 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4294967295.9999995 2024-09-10T19:14:50.034 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:14:50.034 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:14:50.034 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-10T19:14:50.035 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:49.972434+0000", 2024-09-10T19:14:50.035 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.614e-06 2024-09-10T19:14:50.035 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:14:50.035 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:14:50.035 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-10T19:14:50.035 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:49.972445+0000", 2024-09-10T19:14:50.035 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.1319e-05 2024-09-10T19:14:50.035 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:14:50.035 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:14:50.035 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-10T19:14:50.035 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:49.972451+0000", 2024-09-10T19:14:50.035 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.6910000000000002e-06 2024-09-10T19:14:50.035 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:14:50.036 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:14:50.036 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-10T19:14:50.036 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:49.972512+0000", 2024-09-10T19:14:50.036 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.1367999999999999e-05 2024-09-10T19:14:50.036 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:14:50.036 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:14:50.036 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-10T19:14:50.036 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:49.972531+0000", 2024-09-10T19:14:50.036 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.9561000000000001e-05 2024-09-10T19:14:50.036 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:14:50.036 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:14:50.036 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:14:50.036 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:14:50.036 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:14:50.037 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.14781.0:209684 7.f 7.a73a6f (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e128)", 2024-09-10T19:14:50.037 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-10T19:14:49.955736+0000", 2024-09-10T19:14:50.037 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 0.019388659999999999, 2024-09-10T19:14:50.037 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.020035216000000002, 2024-09-10T19:14:50.037 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-10T19:14:50.037 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "queued for pg", 2024-09-10T19:14:50.037 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-10T19:14:50.037 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.14781", 2024-09-10T19:14:50.037 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.183:0/2025776420", 2024-09-10T19:14:50.037 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 209684 2024-09-10T19:14:50.037 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:14:50.037 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-10T19:14:50.037 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:14:50.038 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-10T19:14:50.038 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:49.955736+0000", 2024-09-10T19:14:50.038 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:14:50.038 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:14:50.038 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:14:50.038 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-10T19:14:50.038 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:49.955736+0000", 2024-09-10T19:14:50.038 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:14:50.038 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:14:50.038 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:14:50.038 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-10T19:14:50.038 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:49.955734+0000", 2024-09-10T19:14:50.038 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4294967295.9999981 2024-09-10T19:14:50.039 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:14:50.039 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:14:50.039 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-10T19:14:50.039 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:49.955743+0000", 2024-09-10T19:14:50.039 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.6540000000000005e-06 2024-09-10T19:14:50.039 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:14:50.039 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:14:50.039 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-10T19:14:50.039 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:49.955759+0000", 2024-09-10T19:14:50.039 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.6422e-05 2024-09-10T19:14:50.039 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:14:50.039 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:14:50.039 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-10T19:14:50.039 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:49.955766+0000", 2024-09-10T19:14:50.040 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.4219999999999997e-06 2024-09-10T19:14:50.040 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:14:50.040 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:14:50.040 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:14:50.040 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:14:50.040 INFO:teuthology.orchestra.run.smithi183.stdout: ], 2024-09-10T19:14:50.040 INFO:teuthology.orchestra.run.smithi183.stdout: "num_ops": 10 2024-09-10T19:14:50.040 INFO:teuthology.orchestra.run.smithi183.stdout:} 2024-09-10T19:14:50.062 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:14:50.089 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:14:50.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:14:50.090 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:14:50.188 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-09-10T19:14:50.191 INFO:teuthology.orchestra.run.smithi183.stderr:dumped all 2024-09-10T19:14:50.192 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:14:50.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:14:49 smithi191 ceph-mon[32884]: from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:14:50.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:14:49 smithi191 ceph-mon[32884]: Health check update: Degraded data redundancy: 7919/227046 objects degraded (3.488%), 5 pgs degraded (PG_DEGRADED) 2024-09-10T19:14:50.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:14:49 smithi191 ceph-mon[32884]: osdmap e128: 8 total, 7 up, 7 in 2024-09-10T19:14:50.270 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:14:50.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:14:50.272 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:14:50.373 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:14:50.402 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:14:50.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:14:50.403 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:14:50.504 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:14:50.536 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:14:50.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:14:50.538 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:14:50.638 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:14:50.685 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:14:50.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:14:50.686 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:14:50.786 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:14:50.821 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:14:50.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:14:50.822 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:14:50.922 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:14:50.957 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:14:50.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:14:50.958 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:14:51.059 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:14:51.072 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:14:50 smithi183 ceph-mon[29736]: pgmap v180: 148 pgs: 1 peering, 4 active+undersized+degraded+remapped+backfilling, 143 active+clean; 4.5 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 852 B/s rd, 0 B/s wr, 1.23k op/s; 4890/222540 objects degraded (2.197%); 72 MiB/s, 1.15k objects/s recovering 2024-09-10T19:14:51.072 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:14:50 smithi183 ceph-mon[29736]: from='client.24818 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-10T19:14:51.072 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:14:50 smithi183 ceph-mon[41562]: pgmap v180: 148 pgs: 1 peering, 4 active+undersized+degraded+remapped+backfilling, 143 active+clean; 4.5 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 852 B/s rd, 0 B/s wr, 1.23k op/s; 4890/222540 objects degraded (2.197%); 72 MiB/s, 1.15k objects/s recovering 2024-09-10T19:14:51.073 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:14:50 smithi183 ceph-mon[41562]: from='client.24818 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-10T19:14:51.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:14:51.109 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:14:51.110 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:14:51.115 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-10T19:14:51.115 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-10T19:14:51.211 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:14:51.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:14:50 smithi191 ceph-mon[32884]: pgmap v180: 148 pgs: 1 peering, 4 active+undersized+degraded+remapped+backfilling, 143 active+clean; 4.5 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 852 B/s rd, 0 B/s wr, 1.23k op/s; 4890/222540 objects degraded (2.197%); 72 MiB/s, 1.15k objects/s recovering 2024-09-10T19:14:51.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:14:50 smithi191 ceph-mon[32884]: from='client.24818 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-10T19:14:51.240 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:14:51.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:14:51.241 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:14:51.331 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Sep 10 19:14:51 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-y[29989]: ::ffff:172.21.15.191 - - [10/Sep/2024:19:14:51] "GET /metrics HTTP/1.1" 200 209145 "" "Prometheus/2.43.0" 2024-09-10T19:14:51.334 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph daemon osd.0 dump_blocked_ops 2024-09-10T19:14:51.341 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:14:51.366 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:14:51.369 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:14:51.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:14:51.370 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:14:51.470 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:14:51.498 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:14:51.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:14:51.499 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:14:51.600 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:14:51.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:14:51.637 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:14:51.637 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:14:51.738 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:14:51.771 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:14:51.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:14:51.772 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:14:51.842 INFO:teuthology.orchestra.run.smithi183.stdout:{"pg_ready":true,"pg_map":{"version":181,"stamp":"2024-09-10T19:14:49.718522+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":4830009588,"num_objects":73888,"num_object_clones":0,"num_object_copies":221664,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4890,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":73888,"num_whiteouts":0,"num_read":969,"num_read_kb":826,"num_write":207609,"num_write_kb":8985864,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":29206,"num_bytes_recovered":1906773325,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":77156,"ondisk_log_size":77156,"up":444,"acting":440,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":444,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":656216064,"kb_used":16011448,"kb_used_data":13867516,"kb_used_omap":11,"kb_used_meta":2143796,"kb_avail":640204616,"statfs":{"total":671965249536,"available":655569526784,"internally_reserved":0,"allocated":14200336384,"data_stored":14197712802,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":11928,"internal_metadata":2195247464},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,3,3,2,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-339148800,"num_objects":-5175,"num_object_clones":0,"num_object_copies":-15525,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-5319,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-5175,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":5175,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4890,"num_bytes_recovered":320405528,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.002410"},"pg_stats":[{"pgid":"6.1b","version":"0'0","reported_seq":89,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-09-10T19:14:42.812180+0000","last_change":"2024-09-10T19:13:08.710758+0000","last_active":"2024-09-10T19:14:42.812180+0000","last_peered":"2024-09-10T19:14:42.812180+0000","last_clean":"2024-09-10T19:14:42.812180+0000","last_became_active":"2024-09-10T19:10:14.056187+0000","last_became_peered":"2024-09-10T19:10:14.056187+0000","last_unstale":"2024-09-10T19:14:42.812180+0000","last_undegraded":"2024-09-10T19:14:42.812180+0000","last_fullsized":"2024-09-10T19:14:42.812180+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:13:08.710679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:13:08.710679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0027252499999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":105,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-09-10T19:14:42.806119+0000","last_change":"2024-09-10T19:13:16.219506+0000","last_active":"2024-09-10T19:14:42.806119+0000","last_peered":"2024-09-10T19:14:42.806119+0000","last_clean":"2024-09-10T19:14:42.806119+0000","last_became_active":"2024-09-10T19:11:44.748696+0000","last_became_peered":"2024-09-10T19:11:44.748696+0000","last_unstale":"2024-09-10T19:14:42.806119+0000","last_undegraded":"2024-09-10T19:14:42.806119+0000","last_fullsized":"2024-09-10T19:14:42.806119+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:13:16.219364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:13:16.219364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.023972094999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.19","version":"66'15","reported_seq":118,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-09-10T19:14:42.809262+0000","last_change":"2024-09-10T19:13:47.165943+0000","last_active":"2024-09-10T19:14:42.809262+0000","last_peered":"2024-09-10T19:14:42.809262+0000","last_clean":"2024-09-10T19:14:42.809262+0000","last_became_active":"2024-09-10T19:10:09.688432+0000","last_became_peered":"2024-09-10T19:10:09.688432+0000","last_unstale":"2024-09-10T19:14:42.809262+0000","last_undegraded":"2024-09-10T19:14:42.809262+0000","last_fullsized":"2024-09-10T19:14:42.809262+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'15","last_scrub_stamp":"2024-09-10T19:13:47.165848+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:13:47.165848+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:15:16.492257+0000","scrub_duration":0.004982841,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":102,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-09-10T19:14:45.717346+0000","last_change":"2024-09-10T19:14:35.081003+0000","last_active":"2024-09-10T19:14:45.717346+0000","last_peered":"2024-09-10T19:14:45.717346+0000","last_clean":"2024-09-10T19:14:45.717346+0000","last_became_active":"2024-09-10T19:13:54.714824+0000","last_became_peered":"2024-09-10T19:13:54.714824+0000","last_unstale":"2024-09-10T19:14:45.717346+0000","last_undegraded":"2024-09-10T19:14:45.717346+0000","last_fullsized":"2024-09-10T19:14:45.717346+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:11:23.086191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:11:23.086191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0016087969999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"59'1","reported_seq":103,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-09-10T19:14:42.815560+0000","last_change":"2024-09-10T19:12:47.704757+0000","last_active":"2024-09-10T19:14:42.815560+0000","last_peered":"2024-09-10T19:14:42.815560+0000","last_clean":"2024-09-10T19:14:42.815560+0000","last_became_active":"2024-09-10T19:10:07.686143+0000","last_became_peered":"2024-09-10T19:10:07.686143+0000","last_unstale":"2024-09-10T19:14:42.815560+0000","last_undegraded":"2024-09-10T19:14:42.815560+0000","last_fullsized":"2024-09-10T19:14:42.815560+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'1","last_scrub_stamp":"2024-09-10T19:12:47.704508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:12:47.704508+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.010519827000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":874,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"66'9","reported_seq":127,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-09-10T19:14:45.717476+0000","last_change":"2024-09-10T19:13:54.747869+0000","last_active":"2024-09-10T19:14:45.717476+0000","last_peered":"2024-09-10T19:14:45.717476+0000","last_clean":"2024-09-10T19:14:45.717476+0000","last_became_active":"2024-09-10T19:13:54.716117+0000","last_became_peered":"2024-09-10T19:13:54.716117+0000","last_unstale":"2024-09-10T19:14:45.717476+0000","last_undegraded":"2024-09-10T19:14:45.717476+0000","last_fullsized":"2024-09-10T19:14:45.717476+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'9","last_scrub_stamp":"2024-09-10T19:12:04.079264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:12:04.079264+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0015358819999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,3],"acting":[4,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":47,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-09-10T19:14:45.716162+0000","last_change":"2024-09-10T19:13:54.714747+0000","last_active":"2024-09-10T19:14:45.716162+0000","last_peered":"2024-09-10T19:14:45.716162+0000","last_clean":"2024-09-10T19:14:45.716162+0000","last_became_active":"2024-09-10T19:13:54.714365+0000","last_became_peered":"2024-09-10T19:13:54.714365+0000","last_unstale":"2024-09-10T19:14:45.716162+0000","last_undegraded":"2024-09-10T19:14:45.716162+0000","last_fullsized":"2024-09-10T19:14:45.716162+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:11:56.186365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:11:56.186365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":100,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-09-10T19:14:45.716722+0000","last_change":"2024-09-10T19:14:37.081057+0000","last_active":"2024-09-10T19:14:45.716722+0000","last_peered":"2024-09-10T19:14:45.716722+0000","last_clean":"2024-09-10T19:14:45.716722+0000","last_became_active":"2024-09-10T19:13:54.713538+0000","last_became_peered":"2024-09-10T19:13:54.713538+0000","last_unstale":"2024-09-10T19:14:45.716722+0000","last_undegraded":"2024-09-10T19:14:45.716722+0000","last_fullsized":"2024-09-10T19:14:45.716722+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:11:32.056254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:11:32.056254+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0013010680000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,7],"acting":[4,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":97,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-09-10T19:14:45.722051+0000","last_change":"2024-09-10T19:12:49.527255+0000","last_active":"2024-09-10T19:14:45.722051+0000","last_peered":"2024-09-10T19:14:45.722051+0000","last_clean":"2024-09-10T19:14:45.722051+0000","last_became_active":"2024-09-10T19:10:07.690728+0000","last_became_peered":"2024-09-10T19:10:07.690728+0000","last_unstale":"2024-09-10T19:14:45.722051+0000","last_undegraded":"2024-09-10T19:14:45.722051+0000","last_fullsized":"2024-09-10T19:14:45.722051+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:12:49.527167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:12:49.527167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0040597050000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1b","version":"66'5","reported_seq":120,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-09-10T19:14:45.716766+0000","last_change":"2024-09-10T19:12:55.359526+0000","last_active":"2024-09-10T19:14:45.716766+0000","last_peered":"2024-09-10T19:14:45.716766+0000","last_clean":"2024-09-10T19:14:45.716766+0000","last_became_active":"2024-09-10T19:11:44.748296+0000","last_became_peered":"2024-09-10T19:11:44.748296+0000","last_unstale":"2024-09-10T19:14:45.716766+0000","last_undegraded":"2024-09-10T19:14:45.716766+0000","last_fullsized":"2024-09-10T19:14:45.716766+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'5","last_scrub_stamp":"2024-09-10T19:12:55.359379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:12:55.359379+0000","objects_scrubbed":2,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0050448070000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":100,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-09-10T19:14:45.722067+0000","last_change":"2024-09-10T19:13:54.720135+0000","last_active":"2024-09-10T19:14:45.722067+0000","last_peered":"2024-09-10T19:14:45.722067+0000","last_clean":"2024-09-10T19:14:45.722067+0000","last_became_active":"2024-09-10T19:13:54.719063+0000","last_became_peered":"2024-09-10T19:13:54.719063+0000","last_unstale":"2024-09-10T19:14:45.722067+0000","last_undegraded":"2024-09-10T19:14:45.722067+0000","last_fullsized":"2024-09-10T19:14:45.722067+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:11:43.142734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:11:43.142734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0013612330000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":98,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-09-10T19:14:45.716182+0000","last_change":"2024-09-10T19:13:54.723892+0000","last_active":"2024-09-10T19:14:45.716182+0000","last_peered":"2024-09-10T19:14:45.716182+0000","last_clean":"2024-09-10T19:14:45.716182+0000","last_became_active":"2024-09-10T19:13:54.723590+0000","last_became_peered":"2024-09-10T19:13:54.723590+0000","last_unstale":"2024-09-10T19:14:45.716182+0000","last_undegraded":"2024-09-10T19:14:45.716182+0000","last_fullsized":"2024-09-10T19:14:45.716182+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:11:45.411321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:11:45.411321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001536576,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":97,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-09-10T19:14:45.723176+0000","last_change":"2024-09-10T19:12:54.477202+0000","last_active":"2024-09-10T19:14:45.723176+0000","last_peered":"2024-09-10T19:14:45.723176+0000","last_clean":"2024-09-10T19:14:45.723176+0000","last_became_active":"2024-09-10T19:10:07.690174+0000","last_became_peered":"2024-09-10T19:10:07.690174+0000","last_unstale":"2024-09-10T19:14:45.723176+0000","last_undegraded":"2024-09-10T19:14:45.723176+0000","last_fullsized":"2024-09-10T19:14:45.723176+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:12:54.477149+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:12:54.477149+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0039768319999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1a","version":"66'9","reported_seq":110,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-09-10T19:14:45.716977+0000","last_change":"2024-09-10T19:13:02.431528+0000","last_active":"2024-09-10T19:14:45.716977+0000","last_peered":"2024-09-10T19:14:45.716977+0000","last_clean":"2024-09-10T19:14:45.716977+0000","last_became_active":"2024-09-10T19:10:09.688987+0000","last_became_peered":"2024-09-10T19:10:09.688987+0000","last_unstale":"2024-09-10T19:14:45.716977+0000","last_undegraded":"2024-09-10T19:14:45.716977+0000","last_fullsized":"2024-09-10T19:14:45.716977+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'9","last_scrub_stamp":"2024-09-10T19:13:02.431443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:13:02.431443+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0084913290000000006,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":93,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-09-10T19:14:45.719577+0000","last_change":"2024-09-10T19:12:51.449179+0000","last_active":"2024-09-10T19:14:45.719577+0000","last_peered":"2024-09-10T19:14:45.719577+0000","last_clean":"2024-09-10T19:14:45.719577+0000","last_became_active":"2024-09-10T19:10:11.701799+0000","last_became_peered":"2024-09-10T19:10:11.701799+0000","last_unstale":"2024-09-10T19:14:45.719577+0000","last_undegraded":"2024-09-10T19:14:45.719577+0000","last_fullsized":"2024-09-10T19:14:45.719577+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:12:51.449046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:12:51.449046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.002348826,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":96,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-09-10T19:14:42.801943+0000","last_change":"2024-09-10T19:13:54.719488+0000","last_active":"2024-09-10T19:14:42.801943+0000","last_peered":"2024-09-10T19:14:42.801943+0000","last_clean":"2024-09-10T19:14:42.801943+0000","last_became_active":"2024-09-10T19:13:54.718860+0000","last_became_peered":"2024-09-10T19:13:54.718860+0000","last_unstale":"2024-09-10T19:14:42.801943+0000","last_undegraded":"2024-09-10T19:14:42.801943+0000","last_fullsized":"2024-09-10T19:14:42.801943+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:11:12.970173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:11:12.970173+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001527219,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.e","version":"125'13076","reported_seq":13485,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-10T19:14:44.735302+0000","last_change":"2024-09-10T19:13:52.171265+0000","last_active":"2024-09-10T19:14:44.735302+0000","last_peered":"2024-09-10T19:14:44.735302+0000","last_clean":"2024-09-10T19:14:44.735302+0000","last_became_active":"2024-09-10T19:12:11.032901+0000","last_became_peered":"2024-09-10T19:12:11.032901+0000","last_unstale":"2024-09-10T19:14:44.735302+0000","last_undegraded":"2024-09-10T19:14:44.735302+0000","last_fullsized":"2024-09-10T19:14:44.735302+0000","mapping_epoch":86,"log_start":"109'7862","ondisk_log_start":"109'7862","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"114'9197","last_scrub_stamp":"2024-09-10T19:13:52.169437+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:12:10.007465+0000","last_clean_scrub_stamp":"2024-09-10T19:13:52.169437+0000","objects_scrubbed":8820,"log_size":5214,"log_dups_size":0,"ondisk_log_size":5214,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:15:07.775054+0000","scrub_duration":3.857076422,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":315621376,"num_objects":4816,"num_object_clones":0,"num_object_copies":14448,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4816,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13076,"num_write_kb":572544,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.d","version":"66'17","reported_seq":157,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-09-10T19:14:45.718441+0000","last_change":"2024-09-10T19:13:54.747117+0000","last_active":"2024-09-10T19:14:45.718441+0000","last_peered":"2024-09-10T19:14:45.718441+0000","last_clean":"2024-09-10T19:14:45.718441+0000","last_became_active":"2024-09-10T19:13:54.713411+0000","last_became_peered":"2024-09-10T19:13:54.713411+0000","last_unstale":"2024-09-10T19:14:45.718441+0000","last_undegraded":"2024-09-10T19:14:45.718441+0000","last_fullsized":"2024-09-10T19:14:45.718441+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'17","last_scrub_stamp":"2024-09-10T19:12:45.367252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:12:45.367252+0000","objects_scrubbed":8,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0031240310000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":97,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-09-10T19:14:45.722484+0000","last_change":"2024-09-10T19:12:46.519868+0000","last_active":"2024-09-10T19:14:45.722484+0000","last_peered":"2024-09-10T19:14:45.722484+0000","last_clean":"2024-09-10T19:14:45.722484+0000","last_became_active":"2024-09-10T19:10:07.691119+0000","last_became_peered":"2024-09-10T19:10:07.691119+0000","last_unstale":"2024-09-10T19:14:45.722484+0000","last_undegraded":"2024-09-10T19:14:45.722484+0000","last_fullsized":"2024-09-10T19:14:45.722484+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:12:46.519818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:11:22.280002+0000","last_clean_scrub_stamp":"2024-09-10T19:12:46.519818+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001139279,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":47,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-09-10T19:14:45.717900+0000","last_change":"2024-09-10T19:13:54.713806+0000","last_active":"2024-09-10T19:14:45.717900+0000","last_peered":"2024-09-10T19:14:45.717900+0000","last_clean":"2024-09-10T19:14:45.717900+0000","last_became_active":"2024-09-10T19:13:54.712883+0000","last_became_peered":"2024-09-10T19:13:54.712883+0000","last_unstale":"2024-09-10T19:14:45.717900+0000","last_undegraded":"2024-09-10T19:14:45.717900+0000","last_fullsized":"2024-09-10T19:14:45.717900+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:12:00.240841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:12:00.240841+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":89,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-09-10T19:14:42.807284+0000","last_change":"2024-09-10T19:12:49.342906+0000","last_active":"2024-09-10T19:14:42.807284+0000","last_peered":"2024-09-10T19:14:42.807284+0000","last_clean":"2024-09-10T19:14:42.807284+0000","last_became_active":"2024-09-10T19:10:13.722894+0000","last_became_peered":"2024-09-10T19:10:13.722894+0000","last_unstale":"2024-09-10T19:14:42.807284+0000","last_undegraded":"2024-09-10T19:14:42.807284+0000","last_fullsized":"2024-09-10T19:14:42.807284+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:12:49.342837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:12:49.342837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0092298559999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.f","version":"125'12843","reported_seq":13245,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-10T19:14:44.860638+0000","last_change":"2024-09-10T19:13:49.950750+0000","last_active":"2024-09-10T19:14:44.860638+0000","last_peered":"2024-09-10T19:14:44.860638+0000","last_clean":"2024-09-10T19:14:44.860638+0000","last_became_active":"2024-09-10T19:12:11.036181+0000","last_became_peered":"2024-09-10T19:12:11.036181+0000","last_unstale":"2024-09-10T19:14:44.860638+0000","last_undegraded":"2024-09-10T19:14:44.860638+0000","last_fullsized":"2024-09-10T19:14:44.860638+0000","mapping_epoch":86,"log_start":"111'8125","ondisk_log_start":"111'8125","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"114'8808","last_scrub_stamp":"2024-09-10T19:13:49.950668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:12:10.007465+0000","last_clean_scrub_stamp":"2024-09-10T19:13:49.950668+0000","objects_scrubbed":8620,"log_size":4718,"log_dups_size":0,"ondisk_log_size":4718,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:14:50.581709+0000","scrub_duration":5.7089046000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":303628288,"num_objects":4633,"num_object_clones":0,"num_object_copies":13899,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4633,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12843,"num_write_kb":559232,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.c","version":"66'10","reported_seq":119,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-09-10T19:14:45.718017+0000","last_change":"2024-09-10T19:13:51.416075+0000","last_active":"2024-09-10T19:14:45.718017+0000","last_peered":"2024-09-10T19:14:45.718017+0000","last_clean":"2024-09-10T19:14:45.718017+0000","last_became_active":"2024-09-10T19:10:09.688321+0000","last_became_peered":"2024-09-10T19:10:09.688321+0000","last_unstale":"2024-09-10T19:14:45.718017+0000","last_undegraded":"2024-09-10T19:14:45.718017+0000","last_fullsized":"2024-09-10T19:14:45.718017+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'10","last_scrub_stamp":"2024-09-10T19:13:51.415998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:13:51.415998+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:15:20.996413+0000","scrub_duration":0.005972799,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":45,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-09-10T19:14:42.809734+0000","last_change":"2024-09-10T19:13:54.731440+0000","last_active":"2024-09-10T19:14:42.809734+0000","last_peered":"2024-09-10T19:14:42.809734+0000","last_clean":"2024-09-10T19:14:42.809734+0000","last_became_active":"2024-09-10T19:13:54.731134+0000","last_became_peered":"2024-09-10T19:13:54.731134+0000","last_unstale":"2024-09-10T19:14:42.809734+0000","last_undegraded":"2024-09-10T19:14:42.809734+0000","last_fullsized":"2024-09-10T19:14:42.809734+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:11:26.110138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:11:26.110138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,5],"acting":[3,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"80'9","reported_seq":158,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-09-10T19:14:42.804032+0000","last_change":"2024-09-10T19:12:51.343854+0000","last_active":"2024-09-10T19:14:42.804032+0000","last_peered":"2024-09-10T19:14:42.804032+0000","last_clean":"2024-09-10T19:14:42.804032+0000","last_became_active":"2024-09-10T19:10:11.698547+0000","last_became_peered":"2024-09-10T19:10:11.698547+0000","last_unstale":"2024-09-10T19:14:42.804032+0000","last_undegraded":"2024-09-10T19:14:42.804032+0000","last_fullsized":"2024-09-10T19:14:42.804032+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"80'9","last_scrub_stamp":"2024-09-10T19:12:51.343776+0000","last_deep_scrub":"80'9","last_deep_scrub_stamp":"2024-09-10T19:11:25.156465+0000","last_clean_scrub_stamp":"2024-09-10T19:12:51.343776+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0026375819999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":103,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-09-10T19:14:45.718061+0000","last_change":"2024-09-10T19:14:30.159680+0000","last_active":"2024-09-10T19:14:45.718061+0000","last_peered":"2024-09-10T19:14:45.718061+0000","last_clean":"2024-09-10T19:14:45.718061+0000","last_became_active":"2024-09-10T19:13:54.719498+0000","last_became_peered":"2024-09-10T19:13:54.719498+0000","last_unstale":"2024-09-10T19:14:45.718061+0000","last_undegraded":"2024-09-10T19:14:45.718061+0000","last_fullsized":"2024-09-10T19:14:45.718061+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:11:26.096090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:11:26.096090+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0015599909999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.c","version":"127'13115","reported_seq":13515,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-10T19:14:49.655343+0000","last_change":"2024-09-10T19:13:53.867193+0000","last_active":"2024-09-10T19:14:49.655343+0000","last_peered":"2024-09-10T19:14:49.655343+0000","last_clean":"2024-09-10T19:14:49.655343+0000","last_became_active":"2024-09-10T19:12:11.029860+0000","last_became_peered":"2024-09-10T19:12:11.029860+0000","last_unstale":"2024-09-10T19:14:49.655343+0000","last_undegraded":"2024-09-10T19:14:49.655343+0000","last_fullsized":"2024-09-10T19:14:49.655343+0000","mapping_epoch":86,"log_start":"111'8182","ondisk_log_start":"111'8182","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"114'9191","last_scrub_stamp":"2024-09-10T19:13:53.867107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:12:10.007465+0000","last_clean_scrub_stamp":"2024-09-10T19:13:53.867107+0000","objects_scrubbed":8534,"log_size":4933,"log_dups_size":0,"ondisk_log_size":4933,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:15:11.622513+0000","scrub_duration":2.6024067190000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":294846464,"num_objects":4499,"num_object_clones":0,"num_object_copies":13497,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4499,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13115,"num_write_kb":563648,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.f","version":"66'15","reported_seq":98,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-09-10T19:14:42.814293+0000","last_change":"2024-09-10T19:13:54.742958+0000","last_active":"2024-09-10T19:14:42.814293+0000","last_peered":"2024-09-10T19:14:42.814293+0000","last_clean":"2024-09-10T19:14:42.814293+0000","last_became_active":"2024-09-10T19:13:54.711634+0000","last_became_peered":"2024-09-10T19:13:54.711634+0000","last_unstale":"2024-09-10T19:14:42.814293+0000","last_undegraded":"2024-09-10T19:14:42.814293+0000","last_fullsized":"2024-09-10T19:14:42.814293+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'15","last_scrub_stamp":"2024-09-10T19:11:59.232145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:11:59.232145+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":47,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-09-10T19:14:45.722976+0000","last_change":"2024-09-10T19:13:54.718740+0000","last_active":"2024-09-10T19:14:45.722976+0000","last_peered":"2024-09-10T19:14:45.722976+0000","last_clean":"2024-09-10T19:14:45.722976+0000","last_became_active":"2024-09-10T19:13:54.718452+0000","last_became_peered":"2024-09-10T19:13:54.718452+0000","last_unstale":"2024-09-10T19:14:45.722976+0000","last_undegraded":"2024-09-10T19:14:45.722976+0000","last_fullsized":"2024-09-10T19:14:45.722976+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:11:11.035954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:11:11.035954+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.e","version":"66'8","reported_seq":156,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-09-10T19:14:45.803677+0000","last_change":"2024-09-10T19:13:40.475460+0000","last_active":"2024-09-10T19:14:45.803677+0000","last_peered":"2024-09-10T19:14:45.803677+0000","last_clean":"2024-09-10T19:14:45.803677+0000","last_became_active":"2024-09-10T19:10:11.699232+0000","last_became_peered":"2024-09-10T19:10:11.699232+0000","last_unstale":"2024-09-10T19:14:45.803677+0000","last_undegraded":"2024-09-10T19:14:45.803677+0000","last_fullsized":"2024-09-10T19:14:45.803677+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'8","last_scrub_stamp":"2024-09-10T19:13:40.475375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:13:40.475375+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.035806152000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":98,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-09-10T19:14:45.717352+0000","last_change":"2024-09-10T19:13:54.723273+0000","last_active":"2024-09-10T19:14:45.717352+0000","last_peered":"2024-09-10T19:14:45.717352+0000","last_clean":"2024-09-10T19:14:45.717352+0000","last_became_active":"2024-09-10T19:13:54.722688+0000","last_became_peered":"2024-09-10T19:13:54.722688+0000","last_unstale":"2024-09-10T19:14:45.717352+0000","last_undegraded":"2024-09-10T19:14:45.717352+0000","last_fullsized":"2024-09-10T19:14:45.717352+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:11:18.208782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:11:18.208782+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001904481,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.d","version":"127'13004","reported_seq":21262,"reported_epoch":127,"state":"active+undersized+degraded+remapped+backfilling","last_fresh":"2024-09-10T19:14:49.677590+0000","last_change":"2024-09-10T19:14:25.816207+0000","last_active":"2024-09-10T19:14:49.677590+0000","last_peered":"2024-09-10T19:14:49.677590+0000","last_clean":"2024-09-10T19:12:45.797387+0000","last_became_active":"2024-09-10T19:13:55.789208+0000","last_became_peered":"2024-09-10T19:13:55.789208+0000","last_unstale":"2024-09-10T19:14:49.677590+0000","last_undegraded":"2024-09-10T19:13:55.724690+0000","last_fullsized":"2024-09-10T19:13:55.711897+0000","mapping_epoch":116,"log_start":"111'8056","ondisk_log_start":"111'8056","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:12:10.007465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:12:10.007465+0000","last_clean_scrub_stamp":"2024-09-10T19:12:10.007465+0000","objects_scrubbed":0,"log_size":4948,"log_dups_size":0,"ondisk_log_size":4948,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":293142552,"num_objects":4474,"num_object_clones":0,"num_object_copies":13422,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":857,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4474,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":13004,"num_write_kb":559233,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4094,"num_bytes_recovered":268238872,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0],"avail_no_missing":["6","0"],"object_location_counts":[{"shards":"0,6","objects":4474}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.e","version":"84'16","reported_seq":140,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-09-10T19:14:46.008724+0000","last_change":"2024-09-10T19:14:46.008706+0000","last_active":"2024-09-10T19:14:46.008724+0000","last_peered":"2024-09-10T19:14:46.008724+0000","last_clean":"2024-09-10T19:14:46.008724+0000","last_became_active":"2024-09-10T19:13:54.717022+0000","last_became_peered":"2024-09-10T19:13:54.717022+0000","last_unstale":"2024-09-10T19:14:46.008724+0000","last_undegraded":"2024-09-10T19:14:46.008724+0000","last_fullsized":"2024-09-10T19:14:46.008724+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'16","last_scrub_stamp":"2024-09-10T19:11:56.157975+0000","last_deep_scrub":"84'16","last_deep_scrub_stamp":"2024-09-10T19:11:56.157975+0000","last_clean_scrub_stamp":"2024-09-10T19:11:56.157975+0000","objects_scrubbed":5,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0045462239999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,3],"acting":[4,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":104,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-09-10T19:14:45.721590+0000","last_change":"2024-09-10T19:13:54.721400+0000","last_active":"2024-09-10T19:14:45.721590+0000","last_peered":"2024-09-10T19:14:45.721590+0000","last_clean":"2024-09-10T19:14:45.721590+0000","last_became_active":"2024-09-10T19:13:54.720293+0000","last_became_peered":"2024-09-10T19:13:54.720293+0000","last_unstale":"2024-09-10T19:14:45.721590+0000","last_undegraded":"2024-09-10T19:14:45.721590+0000","last_fullsized":"2024-09-10T19:14:45.721590+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:11:53.066849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:11:53.066849+0000","last_clean_scrub_stamp":"2024-09-10T19:11:53.066849+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0024992679999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":93,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-09-10T19:14:45.718204+0000","last_change":"2024-09-10T19:13:06.358188+0000","last_active":"2024-09-10T19:14:45.718204+0000","last_peered":"2024-09-10T19:14:45.718204+0000","last_clean":"2024-09-10T19:14:45.718204+0000","last_became_active":"2024-09-10T19:10:11.692215+0000","last_became_peered":"2024-09-10T19:10:11.692215+0000","last_unstale":"2024-09-10T19:14:45.718204+0000","last_undegraded":"2024-09-10T19:14:45.718204+0000","last_fullsized":"2024-09-10T19:14:45.718204+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:13:06.358109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:13:06.358109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0026724549999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":89,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-09-10T19:14:42.813093+0000","last_change":"2024-09-10T19:13:54.304493+0000","last_active":"2024-09-10T19:14:42.813093+0000","last_peered":"2024-09-10T19:14:42.813093+0000","last_clean":"2024-09-10T19:14:42.813093+0000","last_became_active":"2024-09-10T19:10:14.053736+0000","last_became_peered":"2024-09-10T19:10:14.053736+0000","last_unstale":"2024-09-10T19:14:42.813093+0000","last_undegraded":"2024-09-10T19:14:42.813093+0000","last_fullsized":"2024-09-10T19:14:42.813093+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:13:54.304435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:13:54.304435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:15:07.353025+0000","scrub_duration":0.0011043820000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.2","version":"125'13009","reported_seq":24957,"reported_epoch":125,"state":"active+undersized+degraded+remapped+backfilling","last_fresh":"2024-09-10T19:14:44.817590+0000","last_change":"2024-09-10T19:13:55.940201+0000","last_active":"2024-09-10T19:14:44.817590+0000","last_peered":"2024-09-10T19:14:44.817590+0000","last_clean":"2024-09-10T19:12:45.804432+0000","last_became_active":"2024-09-10T19:13:55.881880+0000","last_became_peered":"2024-09-10T19:13:55.881880+0000","last_unstale":"2024-09-10T19:14:44.817590+0000","last_undegraded":"2024-09-10T19:13:55.719843+0000","last_fullsized":"2024-09-10T19:13:55.709265+0000","mapping_epoch":116,"log_start":"113'8707","ondisk_log_start":"113'8707","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:12:10.007465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:12:10.007465+0000","last_clean_scrub_stamp":"2024-09-10T19:12:10.007465+0000","objects_scrubbed":0,"log_size":4302,"log_dups_size":0,"ondisk_log_size":4302,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":316473344,"num_objects":4829,"num_object_clones":0,"num_object_copies":14487,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":422,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4829,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13009,"num_write_kb":570816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5941,"num_bytes_recovered":389349376,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7],"acting":[3,2],"avail_no_missing":["3","2"],"object_location_counts":[{"shards":"2,3","objects":4829}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"66'14","reported_seq":115,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-09-10T19:14:45.716806+0000","last_change":"2024-09-10T19:12:48.462230+0000","last_active":"2024-09-10T19:14:45.716806+0000","last_peered":"2024-09-10T19:14:45.716806+0000","last_clean":"2024-09-10T19:14:45.716806+0000","last_became_active":"2024-09-10T19:10:09.686534+0000","last_became_peered":"2024-09-10T19:10:09.686534+0000","last_unstale":"2024-09-10T19:14:45.716806+0000","last_undegraded":"2024-09-10T19:14:45.716806+0000","last_fullsized":"2024-09-10T19:14:45.716806+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'14","last_scrub_stamp":"2024-09-10T19:12:48.462133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:12:48.462133+0000","objects_scrubbed":7,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.002699791,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":97,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-09-10T19:14:45.721559+0000","last_change":"2024-09-10T19:12:52.885643+0000","last_active":"2024-09-10T19:14:45.721559+0000","last_peered":"2024-09-10T19:14:45.721559+0000","last_clean":"2024-09-10T19:14:45.721559+0000","last_became_active":"2024-09-10T19:10:07.684864+0000","last_became_peered":"2024-09-10T19:10:07.684864+0000","last_unstale":"2024-09-10T19:14:45.721559+0000","last_undegraded":"2024-09-10T19:14:45.721559+0000","last_fullsized":"2024-09-10T19:14:45.721559+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:12:52.885563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:12:52.885563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0035531249999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.0","version":"117'10","reported_seq":166,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-09-10T19:14:42.799984+0000","last_change":"2024-09-10T19:13:54.735247+0000","last_active":"2024-09-10T19:14:42.799984+0000","last_peered":"2024-09-10T19:14:42.799984+0000","last_clean":"2024-09-10T19:14:42.799984+0000","last_became_active":"2024-09-10T19:13:54.709846+0000","last_became_peered":"2024-09-10T19:13:54.709846+0000","last_unstale":"2024-09-10T19:14:42.799984+0000","last_undegraded":"2024-09-10T19:14:42.799984+0000","last_fullsized":"2024-09-10T19:14:42.799984+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'8","last_scrub_stamp":"2024-09-10T19:11:29.884739+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:11:29.884739+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0014372650000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,0],"acting":[3,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":91,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-09-10T19:14:45.721900+0000","last_change":"2024-09-10T19:13:15.350815+0000","last_active":"2024-09-10T19:14:45.721900+0000","last_peered":"2024-09-10T19:14:45.721900+0000","last_clean":"2024-09-10T19:14:45.721900+0000","last_became_active":"2024-09-10T19:10:14.054456+0000","last_became_peered":"2024-09-10T19:10:14.054456+0000","last_unstale":"2024-09-10T19:14:45.721900+0000","last_undegraded":"2024-09-10T19:14:45.721900+0000","last_fullsized":"2024-09-10T19:14:45.721900+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:13:15.350720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:11:41.073833+0000","last_clean_scrub_stamp":"2024-09-10T19:13:15.350720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0014361000000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.3","version":"125'12832","reported_seq":13222,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-10T19:14:44.810121+0000","last_change":"2024-09-10T19:13:44.417102+0000","last_active":"2024-09-10T19:14:44.810121+0000","last_peered":"2024-09-10T19:14:44.810121+0000","last_clean":"2024-09-10T19:14:44.810121+0000","last_became_active":"2024-09-10T19:12:11.029906+0000","last_became_peered":"2024-09-10T19:12:11.029906+0000","last_unstale":"2024-09-10T19:14:44.810121+0000","last_undegraded":"2024-09-10T19:14:44.810121+0000","last_fullsized":"2024-09-10T19:14:44.810121+0000","mapping_epoch":86,"log_start":"113'8533","ondisk_log_start":"113'8533","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"112'8444","last_scrub_stamp":"2024-09-10T19:13:44.417011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:12:10.007465+0000","last_clean_scrub_stamp":"2024-09-10T19:13:44.417011+0000","objects_scrubbed":8333,"log_size":4299,"log_dups_size":0,"ondisk_log_size":4299,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:14:54.007266+0000","scrub_duration":3.2086845460000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":305266688,"num_objects":4658,"num_object_clones":0,"num_object_copies":13974,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4658,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12832,"num_write_kb":559680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"66'18","reported_seq":125,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-09-10T19:14:42.800190+0000","last_change":"2024-09-10T19:12:49.699111+0000","last_active":"2024-09-10T19:14:42.800190+0000","last_peered":"2024-09-10T19:14:42.800190+0000","last_clean":"2024-09-10T19:14:42.800190+0000","last_became_active":"2024-09-10T19:10:09.821836+0000","last_became_peered":"2024-09-10T19:10:09.821836+0000","last_unstale":"2024-09-10T19:14:42.800190+0000","last_undegraded":"2024-09-10T19:14:42.800190+0000","last_fullsized":"2024-09-10T19:14:42.800190+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'18","last_scrub_stamp":"2024-09-10T19:12:49.699054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:12:49.699054+0000","objects_scrubbed":8,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0073070890000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":47,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-09-10T19:14:45.721557+0000","last_change":"2024-09-10T19:13:54.711273+0000","last_active":"2024-09-10T19:14:45.721557+0000","last_peered":"2024-09-10T19:14:45.721557+0000","last_clean":"2024-09-10T19:14:45.721557+0000","last_became_active":"2024-09-10T19:13:54.710968+0000","last_became_peered":"2024-09-10T19:13:54.710968+0000","last_unstale":"2024-09-10T19:14:45.721557+0000","last_undegraded":"2024-09-10T19:14:45.721557+0000","last_fullsized":"2024-09-10T19:14:45.721557+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:11:42.202689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:11:42.202689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,3],"acting":[6,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":92,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-09-10T19:14:45.716644+0000","last_change":"2024-09-10T19:12:58.400261+0000","last_active":"2024-09-10T19:14:45.716644+0000","last_peered":"2024-09-10T19:14:45.716644+0000","last_clean":"2024-09-10T19:14:45.716644+0000","last_became_active":"2024-09-10T19:10:11.698054+0000","last_became_peered":"2024-09-10T19:10:11.698054+0000","last_unstale":"2024-09-10T19:14:45.716644+0000","last_undegraded":"2024-09-10T19:14:45.716644+0000","last_fullsized":"2024-09-10T19:14:45.716644+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:12:58.400206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:12:58.400206+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0071235179999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":90,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-09-10T19:14:45.716899+0000","last_change":"2024-09-10T19:12:57.409831+0000","last_active":"2024-09-10T19:14:45.716899+0000","last_peered":"2024-09-10T19:14:45.716899+0000","last_clean":"2024-09-10T19:14:45.716899+0000","last_became_active":"2024-09-10T19:10:13.722784+0000","last_became_peered":"2024-09-10T19:10:13.722784+0000","last_unstale":"2024-09-10T19:14:45.716899+0000","last_undegraded":"2024-09-10T19:14:45.716899+0000","last_fullsized":"2024-09-10T19:14:45.716899+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:12:57.409749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:12:57.409749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0087033260000000008,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.0","version":"127'12996","reported_seq":20136,"reported_epoch":127,"state":"active+undersized+degraded+remapped+backfilling","last_fresh":"2024-09-10T19:14:49.662117+0000","last_change":"2024-09-10T19:14:25.809770+0000","last_active":"2024-09-10T19:14:49.662117+0000","last_peered":"2024-09-10T19:14:49.662117+0000","last_clean":"2024-09-10T19:12:45.799830+0000","last_became_active":"2024-09-10T19:13:55.794303+0000","last_became_peered":"2024-09-10T19:13:55.794303+0000","last_unstale":"2024-09-10T19:14:49.662117+0000","last_undegraded":"2024-09-10T19:13:55.727127+0000","last_fullsized":"2024-09-10T19:13:55.710552+0000","mapping_epoch":116,"log_start":"110'7942","ondisk_log_start":"110'7942","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:12:10.007465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:12:10.007465+0000","last_clean_scrub_stamp":"2024-09-10T19:12:10.007465+0000","objects_scrubbed":0,"log_size":5054,"log_dups_size":0,"ondisk_log_size":5054,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":293863424,"num_objects":4484,"num_object_clones":0,"num_object_copies":13452,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1349,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4484,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12996,"num_write_kb":559360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3534,"num_bytes_recovered":231604224,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3],"avail_no_missing":["5","3"],"object_location_counts":[{"shards":"3,5","objects":4484}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.3","version":"104'27","reported_seq":137,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-09-10T19:14:42.806299+0000","last_change":"2024-09-10T19:13:02.226033+0000","last_active":"2024-09-10T19:14:42.806299+0000","last_peered":"2024-09-10T19:14:42.806299+0000","last_clean":"2024-09-10T19:14:42.806299+0000","last_became_active":"2024-09-10T19:10:09.823003+0000","last_became_peered":"2024-09-10T19:10:09.823003+0000","last_unstale":"2024-09-10T19:14:42.806299+0000","last_undegraded":"2024-09-10T19:14:42.806299+0000","last_fullsized":"2024-09-10T19:14:42.806299+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"104'27","last_scrub_stamp":"2024-09-10T19:13:02.225944+0000","last_deep_scrub":"104'27","last_deep_scrub_stamp":"2024-09-10T19:13:02.225944+0000","last_clean_scrub_stamp":"2024-09-10T19:13:02.225944+0000","objects_scrubbed":8,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0099233020000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":97,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-09-10T19:14:45.722133+0000","last_change":"2024-09-10T19:12:53.513000+0000","last_active":"2024-09-10T19:14:45.722133+0000","last_peered":"2024-09-10T19:14:45.722133+0000","last_clean":"2024-09-10T19:14:45.722133+0000","last_became_active":"2024-09-10T19:10:07.690325+0000","last_became_peered":"2024-09-10T19:10:07.690325+0000","last_unstale":"2024-09-10T19:14:45.722133+0000","last_undegraded":"2024-09-10T19:14:45.722133+0000","last_fullsized":"2024-09-10T19:14:45.722133+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:12:53.512923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:12:53.512923+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0087393290000000005,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":93,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-09-10T19:14:45.720875+0000","last_change":"2024-09-10T19:12:46.963842+0000","last_active":"2024-09-10T19:14:45.720875+0000","last_peered":"2024-09-10T19:14:45.720875+0000","last_clean":"2024-09-10T19:14:45.720875+0000","last_became_active":"2024-09-10T19:10:11.700370+0000","last_became_peered":"2024-09-10T19:10:11.700370+0000","last_unstale":"2024-09-10T19:14:45.720875+0000","last_undegraded":"2024-09-10T19:14:45.720875+0000","last_fullsized":"2024-09-10T19:14:45.720875+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:12:46.963746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:12:46.963746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001460004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":47,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-09-10T19:14:45.720784+0000","last_change":"2024-09-10T19:13:54.718253+0000","last_active":"2024-09-10T19:14:45.720784+0000","last_peered":"2024-09-10T19:14:45.720784+0000","last_clean":"2024-09-10T19:14:45.720784+0000","last_became_active":"2024-09-10T19:13:54.717977+0000","last_became_peered":"2024-09-10T19:13:54.717977+0000","last_unstale":"2024-09-10T19:14:45.720784+0000","last_undegraded":"2024-09-10T19:14:45.720784+0000","last_fullsized":"2024-09-10T19:14:45.720784+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:11:51.187109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:11:51.187109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,7],"acting":[6,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.1","version":"127'12958","reported_seq":13353,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-10T19:14:49.602408+0000","last_change":"2024-09-10T19:13:45.105571+0000","last_active":"2024-09-10T19:14:49.602408+0000","last_peered":"2024-09-10T19:14:49.602408+0000","last_clean":"2024-09-10T19:14:49.602408+0000","last_became_active":"2024-09-10T19:12:11.030399+0000","last_became_peered":"2024-09-10T19:12:11.030399+0000","last_unstale":"2024-09-10T19:14:49.602408+0000","last_undegraded":"2024-09-10T19:14:49.602408+0000","last_fullsized":"2024-09-10T19:14:49.602408+0000","mapping_epoch":86,"log_start":"110'7978","ondisk_log_start":"110'7978","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"113'8485","last_scrub_stamp":"2024-09-10T19:13:45.105477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:12:10.007465+0000","last_clean_scrub_stamp":"2024-09-10T19:13:45.105477+0000","objects_scrubbed":8400,"log_size":4980,"log_dups_size":0,"ondisk_log_size":4980,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"queued for scrub","scrub_duration":2.6533322940000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":290848768,"num_objects":4438,"num_object_clones":0,"num_object_copies":13314,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4438,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12958,"num_write_kb":556672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.2","version":"66'10","reported_seq":63,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-09-10T19:14:45.716438+0000","last_change":"2024-09-10T19:13:54.748881+0000","last_active":"2024-09-10T19:14:45.716438+0000","last_peered":"2024-09-10T19:14:45.716438+0000","last_clean":"2024-09-10T19:14:45.716438+0000","last_became_active":"2024-09-10T19:13:54.714542+0000","last_became_peered":"2024-09-10T19:13:54.714542+0000","last_unstale":"2024-09-10T19:14:45.716438+0000","last_undegraded":"2024-09-10T19:14:45.716438+0000","last_fullsized":"2024-09-10T19:14:45.716438+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'10","last_scrub_stamp":"2024-09-10T19:12:56.360409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:12:56.360409+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":14,"num_read_kb":9,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,0],"acting":[5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":45,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-09-10T19:14:42.805454+0000","last_change":"2024-09-10T19:13:54.718993+0000","last_active":"2024-09-10T19:14:42.805454+0000","last_peered":"2024-09-10T19:14:42.805454+0000","last_clean":"2024-09-10T19:14:42.805454+0000","last_became_active":"2024-09-10T19:13:54.718267+0000","last_became_peered":"2024-09-10T19:13:54.718267+0000","last_unstale":"2024-09-10T19:14:42.805454+0000","last_undegraded":"2024-09-10T19:14:42.805454+0000","last_fullsized":"2024-09-10T19:14:42.805454+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:11:55.227355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:11:55.227355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3","version":"84'9","reported_seq":155,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-09-10T19:14:42.806134+0000","last_change":"2024-09-10T19:13:45.236207+0000","last_active":"2024-09-10T19:14:42.806134+0000","last_peered":"2024-09-10T19:14:42.806134+0000","last_clean":"2024-09-10T19:14:42.806134+0000","last_became_active":"2024-09-10T19:10:11.699690+0000","last_became_peered":"2024-09-10T19:10:11.699690+0000","last_unstale":"2024-09-10T19:14:42.806134+0000","last_undegraded":"2024-09-10T19:14:42.806134+0000","last_fullsized":"2024-09-10T19:14:42.806134+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'9","last_scrub_stamp":"2024-09-10T19:13:45.236094+0000","last_deep_scrub":"84'9","last_deep_scrub_stamp":"2024-09-10T19:11:53.879272+0000","last_clean_scrub_stamp":"2024-09-10T19:13:45.236094+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:15:04.114110+0000","scrub_duration":0.036165602999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":89,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-09-10T19:14:42.801816+0000","last_change":"2024-09-10T19:13:40.234378+0000","last_active":"2024-09-10T19:14:42.801816+0000","last_peered":"2024-09-10T19:14:42.801816+0000","last_clean":"2024-09-10T19:14:42.801816+0000","last_became_active":"2024-09-10T19:10:13.720403+0000","last_became_peered":"2024-09-10T19:10:13.720403+0000","last_unstale":"2024-09-10T19:14:42.801816+0000","last_undegraded":"2024-09-10T19:14:42.801816+0000","last_fullsized":"2024-09-10T19:14:42.801816+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:13:40.234289+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:13:40.234289+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:14:58.963116+0000","scrub_duration":0.0086405590000000008,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.7","version":"127'13119","reported_seq":13511,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-10T19:14:49.642257+0000","last_change":"2024-09-10T19:13:55.214546+0000","last_active":"2024-09-10T19:14:49.642257+0000","last_peered":"2024-09-10T19:14:49.642257+0000","last_clean":"2024-09-10T19:14:49.642257+0000","last_became_active":"2024-09-10T19:12:11.032857+0000","last_became_peered":"2024-09-10T19:12:11.032857+0000","last_unstale":"2024-09-10T19:14:49.642257+0000","last_undegraded":"2024-09-10T19:14:49.642257+0000","last_fullsized":"2024-09-10T19:14:49.642257+0000","mapping_epoch":86,"log_start":"111'8125","ondisk_log_start":"111'8125","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"114'9143","last_scrub_stamp":"2024-09-10T19:13:55.214477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:12:10.007465+0000","last_clean_scrub_stamp":"2024-09-10T19:13:55.214477+0000","objects_scrubbed":8339,"log_size":4994,"log_dups_size":0,"ondisk_log_size":4994,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:15:20.270435+0000","scrub_duration":0.96843253399999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":285933568,"num_objects":4363,"num_object_clones":0,"num_object_copies":13089,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4363,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13119,"num_write_kb":559424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.4","version":"117'30","reported_seq":152,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-09-10T19:14:42.803387+0000","last_change":"2024-09-10T19:13:54.752131+0000","last_active":"2024-09-10T19:14:42.803387+0000","last_peered":"2024-09-10T19:14:42.803387+0000","last_clean":"2024-09-10T19:14:42.803387+0000","last_became_active":"2024-09-10T19:13:54.715642+0000","last_became_peered":"2024-09-10T19:13:54.715642+0000","last_unstale":"2024-09-10T19:14:42.803387+0000","last_undegraded":"2024-09-10T19:14:42.803387+0000","last_fullsized":"2024-09-10T19:14:42.803387+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'28","last_scrub_stamp":"2024-09-10T19:12:45.260189+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:12:45.260189+0000","objects_scrubbed":0,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":47,"num_read_kb":32,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":358,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":104,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-09-10T19:14:45.717723+0000","last_change":"2024-09-10T19:13:54.730667+0000","last_active":"2024-09-10T19:14:45.717723+0000","last_peered":"2024-09-10T19:14:45.717723+0000","last_clean":"2024-09-10T19:14:45.717723+0000","last_became_active":"2024-09-10T19:13:54.730096+0000","last_became_peered":"2024-09-10T19:13:54.730096+0000","last_unstale":"2024-09-10T19:14:45.717723+0000","last_undegraded":"2024-09-10T19:14:45.717723+0000","last_fullsized":"2024-09-10T19:14:45.717723+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:11:41.429402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:11:41.429402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0014226359999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":98,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-09-10T19:14:42.803976+0000","last_change":"2024-09-10T19:13:54.714602+0000","last_active":"2024-09-10T19:14:42.803976+0000","last_peered":"2024-09-10T19:14:42.803976+0000","last_clean":"2024-09-10T19:14:42.803976+0000","last_became_active":"2024-09-10T19:13:54.714017+0000","last_became_peered":"2024-09-10T19:13:54.714017+0000","last_unstale":"2024-09-10T19:14:42.803976+0000","last_undegraded":"2024-09-10T19:14:42.803976+0000","last_fullsized":"2024-09-10T19:14:42.803976+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:11:14.985249+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:11:14.985249+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001460161,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,6],"acting":[0,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":79,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-09-10T19:14:42.813738+0000","last_change":"2024-09-10T19:12:46.691448+0000","last_active":"2024-09-10T19:14:42.813738+0000","last_peered":"2024-09-10T19:14:42.813738+0000","last_clean":"2024-09-10T19:14:42.813738+0000","last_became_active":"2024-09-10T19:10:13.721744+0000","last_became_peered":"2024-09-10T19:10:13.721744+0000","last_unstale":"2024-09-10T19:14:42.813738+0000","last_undegraded":"2024-09-10T19:14:42.813738+0000","last_fullsized":"2024-09-10T19:14:42.813738+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:12:46.691353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:12:46.691353+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0014482360000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.4","version":"124'12735","reported_seq":25397,"reported_epoch":125,"state":"peering","last_fresh":"2024-09-10T19:14:44.724709+0000","last_change":"2024-09-10T19:14:44.726873+0000","last_active":"2024-09-10T19:14:44.724675+0000","last_peered":"2024-09-10T19:14:44.716106+0000","last_clean":"2024-09-10T19:12:45.806743+0000","last_became_active":"2024-09-10T19:13:55.781235+0000","last_became_peered":"2024-09-10T19:13:55.781235+0000","last_unstale":"2024-09-10T19:14:44.724709+0000","last_undegraded":"2024-09-10T19:14:44.724709+0000","last_fullsized":"2024-09-10T19:14:44.724709+0000","mapping_epoch":125,"log_start":"110'8009","ondisk_log_start":"110'8009","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:12:10.007465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:12:10.007465+0000","last_clean_scrub_stamp":"2024-09-10T19:12:10.007465+0000","objects_scrubbed":0,"log_size":4726,"log_dups_size":0,"ondisk_log_size":4726,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":305070080,"num_objects":4655,"num_object_clones":0,"num_object_copies":13965,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4655,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12735,"num_write_kb":556480,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6296,"num_bytes_recovered":412483584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,6],"acting":[0,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[4,6],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.7","version":"82'18","reported_seq":120,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-09-10T19:14:45.718347+0000","last_change":"2024-09-10T19:13:54.737836+0000","last_active":"2024-09-10T19:14:45.718347+0000","last_peered":"2024-09-10T19:14:45.718347+0000","last_clean":"2024-09-10T19:14:45.718347+0000","last_became_active":"2024-09-10T19:13:54.712065+0000","last_became_peered":"2024-09-10T19:13:54.712065+0000","last_unstale":"2024-09-10T19:14:45.718347+0000","last_undegraded":"2024-09-10T19:14:45.718347+0000","last_fullsized":"2024-09-10T19:14:45.718347+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'18","last_scrub_stamp":"2024-09-10T19:11:49.207309+0000","last_deep_scrub":"82'18","last_deep_scrub_stamp":"2024-09-10T19:11:49.207309+0000","last_clean_scrub_stamp":"2024-09-10T19:11:49.207309+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":19,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.0","version":"67'5","reported_seq":194,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-09-10T19:14:44.009481+0000","last_change":"2024-09-10T19:13:54.749420+0000","last_active":"2024-09-10T19:14:44.009481+0000","last_peered":"2024-09-10T19:14:44.009481+0000","last_clean":"2024-09-10T19:14:44.009481+0000","last_became_active":"2024-09-10T19:13:54.715570+0000","last_became_peered":"2024-09-10T19:13:54.715570+0000","last_unstale":"2024-09-10T19:14:44.009481+0000","last_undegraded":"2024-09-10T19:14:44.009481+0000","last_fullsized":"2024-09-10T19:14:44.009481+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'5","last_scrub_stamp":"2024-09-10T19:11:16.007261+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:11:16.007261+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00034454499999999997,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":120,"num_read_kb":115,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":389,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,0],"acting":[2,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":95,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-09-10T19:14:42.803771+0000","last_change":"2024-09-10T19:12:53.318459+0000","last_active":"2024-09-10T19:14:42.803771+0000","last_peered":"2024-09-10T19:14:42.803771+0000","last_clean":"2024-09-10T19:14:42.803771+0000","last_became_active":"2024-09-10T19:10:07.681456+0000","last_became_peered":"2024-09-10T19:10:07.681456+0000","last_unstale":"2024-09-10T19:14:42.803771+0000","last_undegraded":"2024-09-10T19:14:42.803771+0000","last_fullsized":"2024-09-10T19:14:42.803771+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:12:53.318408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:12:53.318408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.01029277,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":91,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-09-10T19:14:42.804269+0000","last_change":"2024-09-10T19:13:00.366695+0000","last_active":"2024-09-10T19:14:42.804269+0000","last_peered":"2024-09-10T19:14:42.804269+0000","last_clean":"2024-09-10T19:14:42.804269+0000","last_became_active":"2024-09-10T19:10:11.695205+0000","last_became_peered":"2024-09-10T19:10:11.695205+0000","last_unstale":"2024-09-10T19:14:42.804269+0000","last_undegraded":"2024-09-10T19:14:42.804269+0000","last_fullsized":"2024-09-10T19:14:42.804269+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:13:00.366610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:13:00.366610+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0023966700000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":84,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-09-10T19:14:45.721908+0000","last_change":"2024-09-10T19:13:44.394892+0000","last_active":"2024-09-10T19:14:45.721908+0000","last_peered":"2024-09-10T19:14:45.721908+0000","last_clean":"2024-09-10T19:14:45.721908+0000","last_became_active":"2024-09-10T19:10:14.053697+0000","last_became_peered":"2024-09-10T19:10:14.053697+0000","last_unstale":"2024-09-10T19:14:45.721908+0000","last_undegraded":"2024-09-10T19:14:45.721908+0000","last_fullsized":"2024-09-10T19:14:45.721908+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:12:06.039981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:12:06.039981+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0013437499999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.5","version":"127'13173","reported_seq":13570,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-10T19:14:49.658861+0000","last_change":"2024-09-10T19:13:55.551583+0000","last_active":"2024-09-10T19:14:49.658861+0000","last_peered":"2024-09-10T19:14:49.658861+0000","last_clean":"2024-09-10T19:14:49.658861+0000","last_became_active":"2024-09-10T19:12:11.032880+0000","last_became_peered":"2024-09-10T19:12:11.032880+0000","last_unstale":"2024-09-10T19:14:49.658861+0000","last_undegraded":"2024-09-10T19:14:49.658861+0000","last_fullsized":"2024-09-10T19:14:49.658861+0000","mapping_epoch":86,"log_start":"113'8752","ondisk_log_start":"113'8752","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"115'9199","last_scrub_stamp":"2024-09-10T19:13:55.551514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:12:10.007465+0000","last_clean_scrub_stamp":"2024-09-10T19:13:55.551514+0000","objects_scrubbed":8461,"log_size":4421,"log_dups_size":0,"ondisk_log_size":4421,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:15:07.317469+0000","scrub_duration":1.028952525,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":294060032,"num_objects":4487,"num_object_clones":0,"num_object_copies":13461,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4487,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13173,"num_write_kb":565120,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"66'12","reported_seq":132,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-09-10T19:14:42.807042+0000","last_change":"2024-09-10T19:13:54.735877+0000","last_active":"2024-09-10T19:14:42.807042+0000","last_peered":"2024-09-10T19:14:42.807042+0000","last_clean":"2024-09-10T19:14:42.807042+0000","last_became_active":"2024-09-10T19:13:54.719200+0000","last_became_peered":"2024-09-10T19:13:54.719200+0000","last_unstale":"2024-09-10T19:14:42.807042+0000","last_undegraded":"2024-09-10T19:14:42.807042+0000","last_fullsized":"2024-09-10T19:14:42.807042+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'12","last_scrub_stamp":"2024-09-10T19:11:38.936857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:11:38.936857+0000","objects_scrubbed":6,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0016916909999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,7],"acting":[0,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"61'1","reported_seq":101,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-09-10T19:14:42.802516+0000","last_change":"2024-09-10T19:13:06.265721+0000","last_active":"2024-09-10T19:14:42.802516+0000","last_peered":"2024-09-10T19:14:42.802516+0000","last_clean":"2024-09-10T19:14:42.802516+0000","last_became_active":"2024-09-10T19:10:07.683259+0000","last_became_peered":"2024-09-10T19:10:07.683259+0000","last_unstale":"2024-09-10T19:14:42.802516+0000","last_undegraded":"2024-09-10T19:14:42.802516+0000","last_fullsized":"2024-09-10T19:14:42.802516+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'1","last_scrub_stamp":"2024-09-10T19:13:06.265646+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:13:06.265646+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.012205384,"objects_trimmed":0,"snaptrim_duration":0.0014341130000000001,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":104,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-09-10T19:14:45.721668+0000","last_change":"2024-09-10T19:13:54.721892+0000","last_active":"2024-09-10T19:14:45.721668+0000","last_peered":"2024-09-10T19:14:45.721668+0000","last_clean":"2024-09-10T19:14:45.721668+0000","last_became_active":"2024-09-10T19:13:54.721486+0000","last_became_peered":"2024-09-10T19:13:54.721486+0000","last_unstale":"2024-09-10T19:14:45.721668+0000","last_undegraded":"2024-09-10T19:14:45.721668+0000","last_fullsized":"2024-09-10T19:14:45.721668+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:11:21.301058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:11:21.301058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0021074710000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":110,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-09-10T19:14:45.718273+0000","last_change":"2024-09-10T19:13:54.712428+0000","last_active":"2024-09-10T19:14:45.718273+0000","last_peered":"2024-09-10T19:14:45.718273+0000","last_clean":"2024-09-10T19:14:45.718273+0000","last_became_active":"2024-09-10T19:13:54.711685+0000","last_became_peered":"2024-09-10T19:13:54.711685+0000","last_unstale":"2024-09-10T19:14:45.718273+0000","last_undegraded":"2024-09-10T19:14:45.718273+0000","last_fullsized":"2024-09-10T19:14:45.718273+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:12:44.527561+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:12:44.527561+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.022638367999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,4],"acting":[5,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":47,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-09-10T19:14:45.716984+0000","last_change":"2024-09-10T19:13:54.723228+0000","last_active":"2024-09-10T19:14:45.716984+0000","last_peered":"2024-09-10T19:14:45.716984+0000","last_clean":"2024-09-10T19:14:45.716984+0000","last_became_active":"2024-09-10T19:13:54.722937+0000","last_became_peered":"2024-09-10T19:13:54.722937+0000","last_unstale":"2024-09-10T19:14:45.716984+0000","last_undegraded":"2024-09-10T19:14:45.716984+0000","last_fullsized":"2024-09-10T19:14:45.716984+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:12:05.165500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:12:05.165500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.6","version":"125'12670","reported_seq":12720,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-10T19:14:44.835577+0000","last_change":"2024-09-10T19:13:43.239851+0000","last_active":"2024-09-10T19:14:44.835577+0000","last_peered":"2024-09-10T19:14:44.835577+0000","last_clean":"2024-09-10T19:14:44.835577+0000","last_became_active":"2024-09-10T19:12:11.035290+0000","last_became_peered":"2024-09-10T19:12:11.035290+0000","last_unstale":"2024-09-10T19:14:44.835577+0000","last_undegraded":"2024-09-10T19:14:44.835577+0000","last_fullsized":"2024-09-10T19:14:44.835577+0000","mapping_epoch":86,"log_start":"110'7942","ondisk_log_start":"110'7942","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:12:10.007465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:12:10.007465+0000","last_clean_scrub_stamp":"2024-09-10T19:12:10.007465+0000","objects_scrubbed":0,"log_size":4728,"log_dups_size":0,"ondisk_log_size":4728,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":303693824,"num_objects":4634,"num_object_clones":0,"num_object_copies":13902,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4634,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12670,"num_write_kb":553728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.5","version":"80'21","reported_seq":124,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-09-10T19:14:45.722395+0000","last_change":"2024-09-10T19:12:54.860083+0000","last_active":"2024-09-10T19:14:45.722395+0000","last_peered":"2024-09-10T19:14:45.722395+0000","last_clean":"2024-09-10T19:14:45.722395+0000","last_became_active":"2024-09-10T19:10:09.822184+0000","last_became_peered":"2024-09-10T19:10:09.822184+0000","last_unstale":"2024-09-10T19:14:45.722395+0000","last_undegraded":"2024-09-10T19:14:45.722395+0000","last_fullsized":"2024-09-10T19:14:45.722395+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"80'21","last_scrub_stamp":"2024-09-10T19:12:54.860013+0000","last_deep_scrub":"80'21","last_deep_scrub_stamp":"2024-09-10T19:11:39.836144+0000","last_clean_scrub_stamp":"2024-09-10T19:12:54.860013+0000","objects_scrubbed":7,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0021624890000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"63'2","reported_seq":102,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-09-10T19:14:42.809807+0000","last_change":"2024-09-10T19:13:00.752806+0000","last_active":"2024-09-10T19:14:42.809807+0000","last_peered":"2024-09-10T19:14:42.809807+0000","last_clean":"2024-09-10T19:14:42.809807+0000","last_became_active":"2024-09-10T19:10:07.684784+0000","last_became_peered":"2024-09-10T19:10:07.684784+0000","last_unstale":"2024-09-10T19:14:42.809807+0000","last_undegraded":"2024-09-10T19:14:42.809807+0000","last_fullsized":"2024-09-10T19:14:42.809807+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'2","last_scrub_stamp":"2024-09-10T19:13:00.752715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:13:00.752715+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0083291979999999995,"objects_trimmed":0,"snaptrim_duration":0.00075522999999999996,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":107,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-09-10T19:14:45.719759+0000","last_change":"2024-09-10T19:13:50.411573+0000","last_active":"2024-09-10T19:14:45.719759+0000","last_peered":"2024-09-10T19:14:45.719759+0000","last_clean":"2024-09-10T19:14:45.719759+0000","last_became_active":"2024-09-10T19:10:07.687596+0000","last_became_peered":"2024-09-10T19:10:07.687596+0000","last_unstale":"2024-09-10T19:14:45.719759+0000","last_undegraded":"2024-09-10T19:14:45.719759+0000","last_fullsized":"2024-09-10T19:14:45.719759+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:13:50.411500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:13:50.411500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:15:20.319236+0000","scrub_duration":0.0032604869999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"74'79","reported_seq":142,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-09-10T19:14:45.721770+0000","last_change":"2024-09-10T19:13:46.419002+0000","last_active":"2024-09-10T19:14:45.721770+0000","last_peered":"2024-09-10T19:14:45.721770+0000","last_clean":"2024-09-10T19:14:45.721770+0000","last_became_active":"2024-09-10T19:09:56.634919+0000","last_became_peered":"2024-09-10T19:09:56.634919+0000","last_unstale":"2024-09-10T19:14:45.721770+0000","last_undegraded":"2024-09-10T19:14:45.721770+0000","last_fullsized":"2024-09-10T19:14:45.721770+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'79","last_scrub_stamp":"2024-09-10T19:13:46.418921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:08:00.699315+0000","last_clean_scrub_stamp":"2024-09-10T19:13:46.418921+0000","objects_scrubbed":2,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:14:48.812041+0000","scrub_duration":0.0029827999999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":93,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-09-10T19:14:45.721998+0000","last_change":"2024-09-10T19:13:47.453309+0000","last_active":"2024-09-10T19:14:45.721998+0000","last_peered":"2024-09-10T19:14:45.721998+0000","last_clean":"2024-09-10T19:14:45.721998+0000","last_became_active":"2024-09-10T19:10:11.694986+0000","last_became_peered":"2024-09-10T19:10:11.694986+0000","last_unstale":"2024-09-10T19:14:45.721998+0000","last_undegraded":"2024-09-10T19:14:45.721998+0000","last_fullsized":"2024-09-10T19:14:45.721998+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:13:47.453228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:13:47.453228+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:15:07.672581+0000","scrub_duration":0.010282463,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":91,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-09-10T19:14:45.718470+0000","last_change":"2024-09-10T19:13:05.390398+0000","last_active":"2024-09-10T19:14:45.718470+0000","last_peered":"2024-09-10T19:14:45.718470+0000","last_clean":"2024-09-10T19:14:45.718470+0000","last_became_active":"2024-09-10T19:10:13.720647+0000","last_became_peered":"2024-09-10T19:10:13.720647+0000","last_unstale":"2024-09-10T19:14:45.718470+0000","last_undegraded":"2024-09-10T19:14:45.718470+0000","last_fullsized":"2024-09-10T19:14:45.718470+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:13:05.390317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:13:05.390317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0058030640000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.9","version":"125'12677","reported_seq":18292,"reported_epoch":125,"state":"active+undersized+degraded+remapped+backfilling","last_fresh":"2024-09-10T19:14:44.857725+0000","last_change":"2024-09-10T19:14:25.972695+0000","last_active":"2024-09-10T19:14:44.857725+0000","last_peered":"2024-09-10T19:14:44.857725+0000","last_clean":"2024-09-10T19:12:45.783536+0000","last_became_active":"2024-09-10T19:13:55.890181+0000","last_became_peered":"2024-09-10T19:13:55.890181+0000","last_unstale":"2024-09-10T19:14:44.857725+0000","last_undegraded":"2024-09-10T19:13:55.954251+0000","last_fullsized":"2024-09-10T19:13:55.708743+0000","mapping_epoch":116,"log_start":"110'7985","ondisk_log_start":"110'7985","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:12:10.007465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:12:10.007465+0000","last_clean_scrub_stamp":"2024-09-10T19:12:10.007465+0000","objects_scrubbed":0,"log_size":4692,"log_dups_size":0,"ondisk_log_size":4692,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":310444032,"num_objects":4737,"num_object_clones":0,"num_object_copies":14211,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2262,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4737,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12677,"num_write_kb":557248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2772,"num_bytes_recovered":181600256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4],"avail_no_missing":["0","4"],"object_location_counts":[{"shards":"0,4","objects":4737}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"66'19","reported_seq":153,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-09-10T19:14:45.720998+0000","last_change":"2024-09-10T19:13:54.755526+0000","last_active":"2024-09-10T19:14:45.720998+0000","last_peered":"2024-09-10T19:14:45.720998+0000","last_clean":"2024-09-10T19:14:45.720998+0000","last_became_active":"2024-09-10T19:13:54.731274+0000","last_became_peered":"2024-09-10T19:13:54.731274+0000","last_unstale":"2024-09-10T19:14:45.720998+0000","last_undegraded":"2024-09-10T19:14:45.720998+0000","last_fullsized":"2024-09-10T19:14:45.720998+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'19","last_scrub_stamp":"2024-09-10T19:11:42.941375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:11:42.941375+0000","objects_scrubbed":9,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0021647599999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,5],"acting":[6,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":95,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-09-10T19:14:42.803676+0000","last_change":"2024-09-10T19:13:05.404535+0000","last_active":"2024-09-10T19:14:42.803676+0000","last_peered":"2024-09-10T19:14:42.803676+0000","last_clean":"2024-09-10T19:14:42.803676+0000","last_became_active":"2024-09-10T19:10:07.686236+0000","last_became_peered":"2024-09-10T19:10:07.686236+0000","last_unstale":"2024-09-10T19:14:42.803676+0000","last_undegraded":"2024-09-10T19:14:42.803676+0000","last_fullsized":"2024-09-10T19:14:42.803676+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:13:05.404457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:13:05.404457+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0037223199999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":91,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-09-10T19:14:42.805717+0000","last_change":"2024-09-10T19:12:55.335929+0000","last_active":"2024-09-10T19:14:42.805717+0000","last_peered":"2024-09-10T19:14:42.805717+0000","last_clean":"2024-09-10T19:14:42.805717+0000","last_became_active":"2024-09-10T19:10:11.699745+0000","last_became_peered":"2024-09-10T19:10:11.699745+0000","last_unstale":"2024-09-10T19:14:42.805717+0000","last_undegraded":"2024-09-10T19:14:42.805717+0000","last_fullsized":"2024-09-10T19:14:42.805717+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:12:55.335834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:12:55.335834+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0060173240000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":91,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-09-10T19:14:45.721682+0000","last_change":"2024-09-10T19:13:14.311675+0000","last_active":"2024-09-10T19:14:45.721682+0000","last_peered":"2024-09-10T19:14:45.721682+0000","last_clean":"2024-09-10T19:14:45.721682+0000","last_became_active":"2024-09-10T19:10:13.719973+0000","last_became_peered":"2024-09-10T19:10:13.719973+0000","last_unstale":"2024-09-10T19:14:45.721682+0000","last_undegraded":"2024-09-10T19:14:45.721682+0000","last_fullsized":"2024-09-10T19:14:45.721682+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:13:14.311620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:13:14.311620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0054101059999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.8","version":"125'12959","reported_seq":13366,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-10T19:14:44.744536+0000","last_change":"2024-09-10T19:13:49.740030+0000","last_active":"2024-09-10T19:14:44.744536+0000","last_peered":"2024-09-10T19:14:44.744536+0000","last_clean":"2024-09-10T19:14:44.744536+0000","last_became_active":"2024-09-10T19:12:11.031441+0000","last_became_peered":"2024-09-10T19:12:11.031441+0000","last_unstale":"2024-09-10T19:14:44.744536+0000","last_undegraded":"2024-09-10T19:14:44.744536+0000","last_fullsized":"2024-09-10T19:14:44.744536+0000","mapping_epoch":86,"log_start":"109'7761","ondisk_log_start":"109'7761","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"113'8888","last_scrub_stamp":"2024-09-10T19:13:49.739947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:12:10.007465+0000","last_clean_scrub_stamp":"2024-09-10T19:13:49.739947+0000","objects_scrubbed":8760,"log_size":5198,"log_dups_size":0,"ondisk_log_size":5198,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:14:53.397517+0000","scrub_duration":3.4609629229999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":307822592,"num_objects":4697,"num_object_clones":0,"num_object_copies":14091,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4697,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12959,"num_write_kb":564992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.b","version":"66'9","reported_seq":126,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-09-10T19:14:42.806221+0000","last_change":"2024-09-10T19:13:54.736863+0000","last_active":"2024-09-10T19:14:42.806221+0000","last_peered":"2024-09-10T19:14:42.806221+0000","last_clean":"2024-09-10T19:14:42.806221+0000","last_became_active":"2024-09-10T19:13:54.710613+0000","last_became_peered":"2024-09-10T19:13:54.710613+0000","last_unstale":"2024-09-10T19:14:42.806221+0000","last_undegraded":"2024-09-10T19:14:42.806221+0000","last_fullsized":"2024-09-10T19:14:42.806221+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'9","last_scrub_stamp":"2024-09-10T19:11:22.870369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:11:22.870369+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0017468729999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":50,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-09-10T19:14:45.718080+0000","last_change":"2024-09-10T19:14:43.036928+0000","last_active":"2024-09-10T19:14:45.718080+0000","last_peered":"2024-09-10T19:14:45.718080+0000","last_clean":"2024-09-10T19:14:45.718080+0000","last_became_active":"2024-09-10T19:13:54.708351+0000","last_became_peered":"2024-09-10T19:13:54.708351+0000","last_unstale":"2024-09-10T19:14:45.718080+0000","last_undegraded":"2024-09-10T19:14:45.718080+0000","last_fullsized":"2024-09-10T19:14:45.718080+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:11:44.254513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:11:44.254513+0000","last_clean_scrub_stamp":"2024-09-10T19:11:44.254513+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":91,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-09-10T19:14:42.803412+0000","last_change":"2024-09-10T19:13:08.398177+0000","last_active":"2024-09-10T19:14:42.803412+0000","last_peered":"2024-09-10T19:14:42.803412+0000","last_clean":"2024-09-10T19:14:42.803412+0000","last_became_active":"2024-09-10T19:10:11.694405+0000","last_became_peered":"2024-09-10T19:10:11.694405+0000","last_unstale":"2024-09-10T19:14:42.803412+0000","last_undegraded":"2024-09-10T19:14:42.803412+0000","last_fullsized":"2024-09-10T19:14:42.803412+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:13:08.398036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:13:08.398036+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.016845473999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":89,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-09-10T19:14:42.802002+0000","last_change":"2024-09-10T19:12:56.145358+0000","last_active":"2024-09-10T19:14:42.802002+0000","last_peered":"2024-09-10T19:14:42.802002+0000","last_clean":"2024-09-10T19:14:42.802002+0000","last_became_active":"2024-09-10T19:10:13.714952+0000","last_became_peered":"2024-09-10T19:10:13.714952+0000","last_unstale":"2024-09-10T19:14:42.802002+0000","last_undegraded":"2024-09-10T19:14:42.802002+0000","last_fullsized":"2024-09-10T19:14:42.802002+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:12:56.145218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:12:56.145218+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0058765429999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.b","version":"127'12936","reported_seq":25920,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-10T19:14:49.651424+0000","last_change":"2024-09-10T19:14:46.754630+0000","last_active":"2024-09-10T19:14:49.651424+0000","last_peered":"2024-09-10T19:14:49.651424+0000","last_clean":"2024-09-10T19:14:49.651424+0000","last_became_active":"2024-09-10T19:14:46.725565+0000","last_became_peered":"2024-09-10T19:14:46.725565+0000","last_unstale":"2024-09-10T19:14:49.651424+0000","last_undegraded":"2024-09-10T19:14:49.651424+0000","last_fullsized":"2024-09-10T19:14:49.651424+0000","mapping_epoch":126,"log_start":"110'7841","ondisk_log_start":"110'7841","created":86,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:12:10.007465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:12:10.007465+0000","last_clean_scrub_stamp":"2024-09-10T19:12:10.007465+0000","objects_scrubbed":0,"log_size":5095,"log_dups_size":0,"ondisk_log_size":5095,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":298975232,"num_objects":4562,"num_object_clones":0,"num_object_copies":13686,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4562,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12936,"num_write_kb":559936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6453,"num_bytes_recovered":422903808,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.8","version":"66'15","reported_seq":120,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-09-10T19:14:45.716203+0000","last_change":"2024-09-10T19:12:50.449684+0000","last_active":"2024-09-10T19:14:45.716203+0000","last_peered":"2024-09-10T19:14:45.716203+0000","last_clean":"2024-09-10T19:14:45.716203+0000","last_became_active":"2024-09-10T19:10:09.820950+0000","last_became_peered":"2024-09-10T19:10:09.820950+0000","last_unstale":"2024-09-10T19:14:45.716203+0000","last_undegraded":"2024-09-10T19:14:45.716203+0000","last_fullsized":"2024-09-10T19:14:45.716203+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'15","last_scrub_stamp":"2024-09-10T19:12:50.449596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:12:50.449596+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0055729239999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":95,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-09-10T19:14:42.804442+0000","last_change":"2024-09-10T19:12:48.346908+0000","last_active":"2024-09-10T19:14:42.804442+0000","last_peered":"2024-09-10T19:14:42.804442+0000","last_clean":"2024-09-10T19:14:42.804442+0000","last_became_active":"2024-09-10T19:10:07.682286+0000","last_became_peered":"2024-09-10T19:10:07.682286+0000","last_unstale":"2024-09-10T19:14:42.804442+0000","last_undegraded":"2024-09-10T19:14:42.804442+0000","last_fullsized":"2024-09-10T19:14:42.804442+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:12:48.346798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:12:48.346798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.00606165,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.9","version":"80'9","reported_seq":158,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-09-10T19:14:45.803365+0000","last_change":"2024-09-10T19:13:43.358984+0000","last_active":"2024-09-10T19:14:45.803365+0000","last_peered":"2024-09-10T19:14:45.803365+0000","last_clean":"2024-09-10T19:14:45.803365+0000","last_became_active":"2024-09-10T19:10:11.702878+0000","last_became_peered":"2024-09-10T19:10:11.702878+0000","last_unstale":"2024-09-10T19:14:45.803365+0000","last_undegraded":"2024-09-10T19:14:45.803365+0000","last_fullsized":"2024-09-10T19:14:45.803365+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"80'9","last_scrub_stamp":"2024-09-10T19:13:43.358901+0000","last_deep_scrub":"80'9","last_deep_scrub_stamp":"2024-09-10T19:11:35.174218+0000","last_clean_scrub_stamp":"2024-09-10T19:13:43.358901+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:15:04.578959+0000","scrub_duration":0.0066143260000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":91,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-09-10T19:14:45.716239+0000","last_change":"2024-09-10T19:12:57.392898+0000","last_active":"2024-09-10T19:14:45.716239+0000","last_peered":"2024-09-10T19:14:45.716239+0000","last_clean":"2024-09-10T19:14:45.716239+0000","last_became_active":"2024-09-10T19:10:14.053875+0000","last_became_peered":"2024-09-10T19:10:14.053875+0000","last_unstale":"2024-09-10T19:14:45.716239+0000","last_undegraded":"2024-09-10T19:14:45.716239+0000","last_fullsized":"2024-09-10T19:14:45.716239+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:12:57.392844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:12:57.392844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0046180839999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.a","version":"125'12964","reported_seq":13356,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-10T19:14:44.815544+0000","last_change":"2024-09-10T19:13:44.230819+0000","last_active":"2024-09-10T19:14:44.815544+0000","last_peered":"2024-09-10T19:14:44.815544+0000","last_clean":"2024-09-10T19:14:44.815544+0000","last_became_active":"2024-09-10T19:12:11.034838+0000","last_became_peered":"2024-09-10T19:12:11.034838+0000","last_unstale":"2024-09-10T19:14:44.815544+0000","last_undegraded":"2024-09-10T19:14:44.815544+0000","last_fullsized":"2024-09-10T19:14:44.815544+0000","mapping_epoch":86,"log_start":"113'8719","ondisk_log_start":"113'8719","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"112'8534","last_scrub_stamp":"2024-09-10T19:13:44.230739+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:12:10.007465+0000","last_clean_scrub_stamp":"2024-09-10T19:13:44.230739+0000","objects_scrubbed":8364,"log_size":4245,"log_dups_size":0,"ondisk_log_size":4245,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:15:06.184625+0000","scrub_duration":4.047209821,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":309723136,"num_objects":4726,"num_object_clones":0,"num_object_copies":14178,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4726,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12964,"num_write_kb":566080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.9","version":"66'12","reported_seq":139,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-09-10T19:14:45.716722+0000","last_change":"2024-09-10T19:14:44.972452+0000","last_active":"2024-09-10T19:14:45.716722+0000","last_peered":"2024-09-10T19:14:45.716722+0000","last_clean":"2024-09-10T19:14:45.716722+0000","last_became_active":"2024-09-10T19:13:54.718152+0000","last_became_peered":"2024-09-10T19:13:54.718152+0000","last_unstale":"2024-09-10T19:14:45.716722+0000","last_undegraded":"2024-09-10T19:14:45.716722+0000","last_fullsized":"2024-09-10T19:14:45.716722+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'12","last_scrub_stamp":"2024-09-10T19:11:45.012377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:11:45.012377+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0019105979999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.f","version":"59'2","reported_seq":112,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-09-10T19:14:45.718643+0000","last_change":"2024-09-10T19:12:50.437332+0000","last_active":"2024-09-10T19:14:45.718643+0000","last_peered":"2024-09-10T19:14:45.718643+0000","last_clean":"2024-09-10T19:14:45.718643+0000","last_became_active":"2024-09-10T19:10:07.681719+0000","last_became_peered":"2024-09-10T19:10:07.681719+0000","last_unstale":"2024-09-10T19:14:45.718643+0000","last_undegraded":"2024-09-10T19:14:45.718643+0000","last_fullsized":"2024-09-10T19:14:45.718643+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'2","last_scrub_stamp":"2024-09-10T19:12:50.437245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:12:50.437245+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0070075770000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":14,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":98,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-09-10T19:14:42.796870+0000","last_change":"2024-09-10T19:13:54.715893+0000","last_active":"2024-09-10T19:14:42.796870+0000","last_peered":"2024-09-10T19:14:42.796870+0000","last_clean":"2024-09-10T19:14:42.796870+0000","last_became_active":"2024-09-10T19:13:54.715528+0000","last_became_peered":"2024-09-10T19:13:54.715528+0000","last_unstale":"2024-09-10T19:14:42.796870+0000","last_undegraded":"2024-09-10T19:14:42.796870+0000","last_fullsized":"2024-09-10T19:14:42.796870+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:11:46.587846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:11:46.587846+0000","last_clean_scrub_stamp":"2024-09-10T19:11:46.587846+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001267142,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":96,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-09-10T19:14:42.812295+0000","last_change":"2024-09-10T19:13:54.712104+0000","last_active":"2024-09-10T19:14:42.812295+0000","last_peered":"2024-09-10T19:14:42.812295+0000","last_clean":"2024-09-10T19:14:42.812295+0000","last_became_active":"2024-09-10T19:13:54.711804+0000","last_became_peered":"2024-09-10T19:13:54.711804+0000","last_unstale":"2024-09-10T19:14:42.812295+0000","last_undegraded":"2024-09-10T19:14:42.812295+0000","last_fullsized":"2024-09-10T19:14:42.812295+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:11:42.723437+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:11:42.723437+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001385631,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.16","version":"66'9","reported_seq":119,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-09-10T19:14:42.806965+0000","last_change":"2024-09-10T19:13:50.172901+0000","last_active":"2024-09-10T19:14:42.806965+0000","last_peered":"2024-09-10T19:14:42.806965+0000","last_clean":"2024-09-10T19:14:42.806965+0000","last_became_active":"2024-09-10T19:10:09.822994+0000","last_became_peered":"2024-09-10T19:10:09.822994+0000","last_unstale":"2024-09-10T19:14:42.806965+0000","last_undegraded":"2024-09-10T19:14:42.806965+0000","last_fullsized":"2024-09-10T19:14:42.806965+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'9","last_scrub_stamp":"2024-09-10T19:13:50.172825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:13:50.172825+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:14:55.898340+0000","scrub_duration":0.0094381619999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":102,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-09-10T19:14:42.797056+0000","last_change":"2024-09-10T19:13:54.716196+0000","last_active":"2024-09-10T19:14:42.797056+0000","last_peered":"2024-09-10T19:14:42.797056+0000","last_clean":"2024-09-10T19:14:42.797056+0000","last_became_active":"2024-09-10T19:13:54.715910+0000","last_became_peered":"2024-09-10T19:13:54.715910+0000","last_unstale":"2024-09-10T19:14:42.797056+0000","last_undegraded":"2024-09-10T19:14:42.797056+0000","last_fullsized":"2024-09-10T19:14:42.797056+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:11:41.458523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:11:41.458523+0000","last_clean_scrub_stamp":"2024-09-10T19:11:41.458523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0012499830000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,4],"acting":[2,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":98,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-09-10T19:14:42.814219+0000","last_change":"2024-09-10T19:13:54.712640+0000","last_active":"2024-09-10T19:14:42.814219+0000","last_peered":"2024-09-10T19:14:42.814219+0000","last_clean":"2024-09-10T19:14:42.814219+0000","last_became_active":"2024-09-10T19:13:54.712275+0000","last_became_peered":"2024-09-10T19:13:54.712275+0000","last_unstale":"2024-09-10T19:14:42.814219+0000","last_undegraded":"2024-09-10T19:14:42.814219+0000","last_fullsized":"2024-09-10T19:14:42.814219+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:12:11.619202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:12:11.619202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001950834,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"66'1","reported_seq":90,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-09-10T19:14:42.803197+0000","last_change":"2024-09-10T19:12:54.361402+0000","last_active":"2024-09-10T19:14:42.803197+0000","last_peered":"2024-09-10T19:14:42.803197+0000","last_clean":"2024-09-10T19:14:42.803197+0000","last_became_active":"2024-09-10T19:10:13.717001+0000","last_became_peered":"2024-09-10T19:10:13.717001+0000","last_unstale":"2024-09-10T19:14:42.803197+0000","last_undegraded":"2024-09-10T19:14:42.803197+0000","last_fullsized":"2024-09-10T19:14:42.803197+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'1","last_scrub_stamp":"2024-09-10T19:12:54.361312+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:12:54.361312+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0039806390000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":104,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-09-10T19:14:45.721126+0000","last_change":"2024-09-10T19:13:54.724442+0000","last_active":"2024-09-10T19:14:45.721126+0000","last_peered":"2024-09-10T19:14:45.721126+0000","last_clean":"2024-09-10T19:14:45.721126+0000","last_became_active":"2024-09-10T19:13:54.724027+0000","last_became_peered":"2024-09-10T19:13:54.724027+0000","last_unstale":"2024-09-10T19:14:45.721126+0000","last_undegraded":"2024-09-10T19:14:45.721126+0000","last_fullsized":"2024-09-10T19:14:45.721126+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:11:32.859106+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:11:32.859106+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0014403980000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,5],"acting":[6,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"66'6","reported_seq":117,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-09-10T19:14:42.814373+0000","last_change":"2024-09-10T19:13:55.370749+0000","last_active":"2024-09-10T19:14:42.814373+0000","last_peered":"2024-09-10T19:14:42.814373+0000","last_clean":"2024-09-10T19:14:42.814373+0000","last_became_active":"2024-09-10T19:13:54.717949+0000","last_became_peered":"2024-09-10T19:13:54.717949+0000","last_unstale":"2024-09-10T19:14:42.814373+0000","last_undegraded":"2024-09-10T19:14:42.814373+0000","last_fullsized":"2024-09-10T19:14:42.814373+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'6","last_scrub_stamp":"2024-09-10T19:13:55.370650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:13:55.370650+0000","objects_scrubbed":3,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:15:17.175995+0000","scrub_duration":0.071693797000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":100,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-09-10T19:14:45.717989+0000","last_change":"2024-09-10T19:13:54.730473+0000","last_active":"2024-09-10T19:14:45.717989+0000","last_peered":"2024-09-10T19:14:45.717989+0000","last_clean":"2024-09-10T19:14:45.717989+0000","last_became_active":"2024-09-10T19:13:54.729917+0000","last_became_peered":"2024-09-10T19:13:54.729917+0000","last_unstale":"2024-09-10T19:14:45.717989+0000","last_undegraded":"2024-09-10T19:14:45.717989+0000","last_fullsized":"2024-09-10T19:14:45.717989+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:11:23.264793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:11:23.264793+0000","last_clean_scrub_stamp":"2024-09-10T19:11:23.264793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0015950649999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":91,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-09-10T19:14:45.721866+0000","last_change":"2024-09-10T19:12:51.508398+0000","last_active":"2024-09-10T19:14:45.721866+0000","last_peered":"2024-09-10T19:14:45.721866+0000","last_clean":"2024-09-10T19:14:45.721866+0000","last_became_active":"2024-09-10T19:10:14.054360+0000","last_became_peered":"2024-09-10T19:10:14.054360+0000","last_unstale":"2024-09-10T19:14:45.721866+0000","last_undegraded":"2024-09-10T19:14:45.721866+0000","last_fullsized":"2024-09-10T19:14:45.721866+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:12:51.508263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:12:51.508263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001748274,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"80'15","reported_seq":132,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-09-10T19:14:42.813658+0000","last_change":"2024-09-10T19:13:54.749388+0000","last_active":"2024-09-10T19:14:42.813658+0000","last_peered":"2024-09-10T19:14:42.813658+0000","last_clean":"2024-09-10T19:14:42.813658+0000","last_became_active":"2024-09-10T19:13:54.719574+0000","last_became_peered":"2024-09-10T19:13:54.719574+0000","last_unstale":"2024-09-10T19:14:42.813658+0000","last_undegraded":"2024-09-10T19:14:42.813658+0000","last_fullsized":"2024-09-10T19:14:42.813658+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"80'15","last_scrub_stamp":"2024-09-10T19:11:41.719310+0000","last_deep_scrub":"80'15","last_deep_scrub_stamp":"2024-09-10T19:11:41.719310+0000","last_clean_scrub_stamp":"2024-09-10T19:11:41.719310+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0046706430000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":97,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-09-10T19:14:45.716602+0000","last_change":"2024-09-10T19:12:46.506129+0000","last_active":"2024-09-10T19:14:45.716602+0000","last_peered":"2024-09-10T19:14:45.716602+0000","last_clean":"2024-09-10T19:14:45.716602+0000","last_became_active":"2024-09-10T19:10:07.685870+0000","last_became_peered":"2024-09-10T19:10:07.685870+0000","last_unstale":"2024-09-10T19:14:45.716602+0000","last_undegraded":"2024-09-10T19:14:45.716602+0000","last_fullsized":"2024-09-10T19:14:45.716602+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:12:46.506071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:12:46.506071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001257935,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"117'10","reported_seq":169,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-09-10T19:14:45.803408+0000","last_change":"2024-09-10T19:13:54.755445+0000","last_active":"2024-09-10T19:14:45.803408+0000","last_peered":"2024-09-10T19:14:45.803408+0000","last_clean":"2024-09-10T19:14:45.803408+0000","last_became_active":"2024-09-10T19:13:54.724780+0000","last_became_peered":"2024-09-10T19:13:54.724780+0000","last_unstale":"2024-09-10T19:14:45.803408+0000","last_undegraded":"2024-09-10T19:14:45.803408+0000","last_fullsized":"2024-09-10T19:14:45.803408+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'8","last_scrub_stamp":"2024-09-10T19:11:50.330303+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:11:50.330303+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0020032069999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3],"acting":[5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":89,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-09-10T19:14:42.803807+0000","last_change":"2024-09-10T19:13:00.212150+0000","last_active":"2024-09-10T19:14:42.803807+0000","last_peered":"2024-09-10T19:14:42.803807+0000","last_clean":"2024-09-10T19:14:42.803807+0000","last_became_active":"2024-09-10T19:10:13.721035+0000","last_became_peered":"2024-09-10T19:10:13.721035+0000","last_unstale":"2024-09-10T19:14:42.803807+0000","last_undegraded":"2024-09-10T19:14:42.803807+0000","last_fullsized":"2024-09-10T19:14:42.803807+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:13:00.212049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:13:00.212049+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.015146817999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"66'9","reported_seq":111,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-09-10T19:14:45.717533+0000","last_change":"2024-09-10T19:12:52.416838+0000","last_active":"2024-09-10T19:14:45.717533+0000","last_peered":"2024-09-10T19:14:45.717533+0000","last_clean":"2024-09-10T19:14:45.717533+0000","last_became_active":"2024-09-10T19:10:09.823006+0000","last_became_peered":"2024-09-10T19:10:09.823006+0000","last_unstale":"2024-09-10T19:14:45.717533+0000","last_undegraded":"2024-09-10T19:14:45.717533+0000","last_fullsized":"2024-09-10T19:14:45.717533+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'9","last_scrub_stamp":"2024-09-10T19:12:52.416748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:12:52.416748+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0047220839999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":95,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-09-10T19:14:42.806602+0000","last_change":"2024-09-10T19:13:01.210932+0000","last_active":"2024-09-10T19:14:42.806602+0000","last_peered":"2024-09-10T19:14:42.806602+0000","last_clean":"2024-09-10T19:14:42.806602+0000","last_became_active":"2024-09-10T19:10:07.688499+0000","last_became_peered":"2024-09-10T19:10:07.688499+0000","last_unstale":"2024-09-10T19:14:42.806602+0000","last_undegraded":"2024-09-10T19:14:42.806602+0000","last_fullsized":"2024-09-10T19:14:42.806602+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:13:01.210848+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:13:01.210848+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0049627250000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"66'8","reported_seq":154,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-09-10T19:14:42.813214+0000","last_change":"2024-09-10T19:12:59.723961+0000","last_active":"2024-09-10T19:14:42.813214+0000","last_peered":"2024-09-10T19:14:42.813214+0000","last_clean":"2024-09-10T19:14:42.813214+0000","last_became_active":"2024-09-10T19:10:11.692359+0000","last_became_peered":"2024-09-10T19:10:11.692359+0000","last_unstale":"2024-09-10T19:14:42.813214+0000","last_undegraded":"2024-09-10T19:14:42.813214+0000","last_fullsized":"2024-09-10T19:14:42.813214+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'8","last_scrub_stamp":"2024-09-10T19:12:59.723712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:12:59.723712+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0083838109999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":90,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-09-10T19:14:45.718436+0000","last_change":"2024-09-10T19:13:09.463096+0000","last_active":"2024-09-10T19:14:45.718436+0000","last_peered":"2024-09-10T19:14:45.718436+0000","last_clean":"2024-09-10T19:14:45.718436+0000","last_became_active":"2024-09-10T19:10:13.718285+0000","last_became_peered":"2024-09-10T19:10:13.718285+0000","last_unstale":"2024-09-10T19:14:45.718436+0000","last_undegraded":"2024-09-10T19:14:45.718436+0000","last_fullsized":"2024-09-10T19:14:45.718436+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:13:09.463001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:13:09.463001+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.012199573,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.12","version":"79'13","reported_seq":104,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-09-10T19:14:45.720853+0000","last_change":"2024-09-10T19:13:54.758707+0000","last_active":"2024-09-10T19:14:45.720853+0000","last_peered":"2024-09-10T19:14:45.720853+0000","last_clean":"2024-09-10T19:14:45.720853+0000","last_became_active":"2024-09-10T19:13:54.721560+0000","last_became_peered":"2024-09-10T19:13:54.721560+0000","last_unstale":"2024-09-10T19:14:45.720853+0000","last_undegraded":"2024-09-10T19:14:45.720853+0000","last_fullsized":"2024-09-10T19:14:45.720853+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'13","last_scrub_stamp":"2024-09-10T19:11:22.072400+0000","last_deep_scrub":"79'13","last_deep_scrub_stamp":"2024-09-10T19:11:22.072400+0000","last_clean_scrub_stamp":"2024-09-10T19:11:22.072400+0000","objects_scrubbed":4,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,0],"acting":[6,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.14","version":"59'1","reported_seq":104,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-09-10T19:14:45.721891+0000","last_change":"2024-09-10T19:13:08.917822+0000","last_active":"2024-09-10T19:14:45.721891+0000","last_peered":"2024-09-10T19:14:45.721891+0000","last_clean":"2024-09-10T19:14:45.721891+0000","last_became_active":"2024-09-10T19:10:07.683263+0000","last_became_peered":"2024-09-10T19:10:07.683263+0000","last_unstale":"2024-09-10T19:14:45.721891+0000","last_undegraded":"2024-09-10T19:14:45.721891+0000","last_fullsized":"2024-09-10T19:14:45.721891+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'1","last_scrub_stamp":"2024-09-10T19:13:08.917719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:13:08.917719+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0033669569999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":362,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":98,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-09-10T19:14:42.812336+0000","last_change":"2024-09-10T19:13:54.730422+0000","last_active":"2024-09-10T19:14:42.812336+0000","last_peered":"2024-09-10T19:14:42.812336+0000","last_clean":"2024-09-10T19:14:42.812336+0000","last_became_active":"2024-09-10T19:13:54.730185+0000","last_became_peered":"2024-09-10T19:13:54.730185+0000","last_unstale":"2024-09-10T19:14:42.812336+0000","last_undegraded":"2024-09-10T19:14:42.812336+0000","last_fullsized":"2024-09-10T19:14:42.812336+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:12:00.660284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:12:00.660284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0012510729999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":96,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-09-10T19:14:42.801737+0000","last_change":"2024-09-10T19:13:54.715762+0000","last_active":"2024-09-10T19:14:42.801737+0000","last_peered":"2024-09-10T19:14:42.801737+0000","last_clean":"2024-09-10T19:14:42.801737+0000","last_became_active":"2024-09-10T19:13:54.714861+0000","last_became_peered":"2024-09-10T19:13:54.714861+0000","last_unstale":"2024-09-10T19:14:42.801737+0000","last_undegraded":"2024-09-10T19:14:42.801737+0000","last_fullsized":"2024-09-10T19:14:42.801737+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:11:51.920006+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:11:51.920006+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001865058,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,6],"acting":[0,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"66'11","reported_seq":135,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-09-10T19:14:45.717132+0000","last_change":"2024-09-10T19:14:40.034928+0000","last_active":"2024-09-10T19:14:45.717132+0000","last_peered":"2024-09-10T19:14:45.717132+0000","last_clean":"2024-09-10T19:14:45.717132+0000","last_became_active":"2024-09-10T19:13:54.720034+0000","last_became_peered":"2024-09-10T19:13:54.720034+0000","last_unstale":"2024-09-10T19:14:45.717132+0000","last_undegraded":"2024-09-10T19:14:45.717132+0000","last_fullsized":"2024-09-10T19:14:45.717132+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'11","last_scrub_stamp":"2024-09-10T19:11:44.025361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:11:44.025361+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.002036894,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":47,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-09-10T19:14:45.717600+0000","last_change":"2024-09-10T19:13:54.723475+0000","last_active":"2024-09-10T19:14:45.717600+0000","last_peered":"2024-09-10T19:14:45.717600+0000","last_clean":"2024-09-10T19:14:45.717600+0000","last_became_active":"2024-09-10T19:13:54.722947+0000","last_became_peered":"2024-09-10T19:13:54.722947+0000","last_unstale":"2024-09-10T19:14:45.717600+0000","last_undegraded":"2024-09-10T19:14:45.717600+0000","last_fullsized":"2024-09-10T19:14:45.717600+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:12:44.249765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:11:10.031245+0000","last_clean_scrub_stamp":"2024-09-10T19:12:44.249765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,6],"acting":[5,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":47,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-09-10T19:14:45.716365+0000","last_change":"2024-09-10T19:13:54.730370+0000","last_active":"2024-09-10T19:14:45.716365+0000","last_peered":"2024-09-10T19:14:45.716365+0000","last_clean":"2024-09-10T19:14:45.716365+0000","last_became_active":"2024-09-10T19:13:54.729703+0000","last_became_peered":"2024-09-10T19:13:54.729703+0000","last_unstale":"2024-09-10T19:14:45.716365+0000","last_undegraded":"2024-09-10T19:14:45.716365+0000","last_fullsized":"2024-09-10T19:14:45.716365+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:11:31.249343+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:11:31.249343+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":89,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-09-10T19:14:42.809053+0000","last_change":"2024-09-10T19:12:54.691886+0000","last_active":"2024-09-10T19:14:42.809053+0000","last_peered":"2024-09-10T19:14:42.809053+0000","last_clean":"2024-09-10T19:14:42.809053+0000","last_became_active":"2024-09-10T19:10:13.724201+0000","last_became_peered":"2024-09-10T19:10:13.724201+0000","last_unstale":"2024-09-10T19:14:42.809053+0000","last_undegraded":"2024-09-10T19:14:42.809053+0000","last_fullsized":"2024-09-10T19:14:42.809053+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:12:54.691810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:12:54.691810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.006702927,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"66'4","reported_seq":112,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-09-10T19:14:42.800124+0000","last_change":"2024-09-10T19:13:54.732227+0000","last_active":"2024-09-10T19:14:42.800124+0000","last_peered":"2024-09-10T19:14:42.800124+0000","last_clean":"2024-09-10T19:14:42.800124+0000","last_became_active":"2024-09-10T19:13:54.711125+0000","last_became_peered":"2024-09-10T19:13:54.711125+0000","last_unstale":"2024-09-10T19:14:42.800124+0000","last_undegraded":"2024-09-10T19:14:42.800124+0000","last_fullsized":"2024-09-10T19:14:42.800124+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'4","last_scrub_stamp":"2024-09-10T19:11:14.816632+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:11:14.816632+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001902504,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,4],"acting":[3,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":97,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-09-10T19:14:45.718106+0000","last_change":"2024-09-10T19:12:48.517928+0000","last_active":"2024-09-10T19:14:45.718106+0000","last_peered":"2024-09-10T19:14:45.718106+0000","last_clean":"2024-09-10T19:14:45.718106+0000","last_became_active":"2024-09-10T19:10:07.687723+0000","last_became_peered":"2024-09-10T19:10:07.687723+0000","last_unstale":"2024-09-10T19:14:45.718106+0000","last_undegraded":"2024-09-10T19:14:45.718106+0000","last_fullsized":"2024-09-10T19:14:45.718106+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:12:48.517853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:12:48.517853+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001753459,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":93,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-09-10T19:14:45.720856+0000","last_change":"2024-09-10T19:13:04.855026+0000","last_active":"2024-09-10T19:14:45.720856+0000","last_peered":"2024-09-10T19:14:45.720856+0000","last_clean":"2024-09-10T19:14:45.720856+0000","last_became_active":"2024-09-10T19:10:11.698536+0000","last_became_peered":"2024-09-10T19:10:11.698536+0000","last_unstale":"2024-09-10T19:14:45.720856+0000","last_undegraded":"2024-09-10T19:14:45.720856+0000","last_fullsized":"2024-09-10T19:14:45.720856+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:13:04.854929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:11:31.837944+0000","last_clean_scrub_stamp":"2024-09-10T19:13:04.854929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0012499329999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":91,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-09-10T19:14:45.721681+0000","last_change":"2024-09-10T19:12:45.536856+0000","last_active":"2024-09-10T19:14:45.721681+0000","last_peered":"2024-09-10T19:14:45.721681+0000","last_clean":"2024-09-10T19:14:45.721681+0000","last_became_active":"2024-09-10T19:10:13.718271+0000","last_became_peered":"2024-09-10T19:10:13.718271+0000","last_unstale":"2024-09-10T19:14:45.721681+0000","last_undegraded":"2024-09-10T19:14:45.721681+0000","last_fullsized":"2024-09-10T19:14:45.721681+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:12:45.536799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:12:45.536799+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0036938909999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.11","version":"66'11","reported_seq":112,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-09-10T19:14:42.809173+0000","last_change":"2024-09-10T19:13:05.730096+0000","last_active":"2024-09-10T19:14:42.809173+0000","last_peered":"2024-09-10T19:14:42.809173+0000","last_clean":"2024-09-10T19:14:42.809173+0000","last_became_active":"2024-09-10T19:10:09.821357+0000","last_became_peered":"2024-09-10T19:10:09.821357+0000","last_unstale":"2024-09-10T19:14:42.809173+0000","last_undegraded":"2024-09-10T19:14:42.809173+0000","last_fullsized":"2024-09-10T19:14:42.809173+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'11","last_scrub_stamp":"2024-09-10T19:13:05.729854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:13:05.729854+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.010667395,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":97,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-09-10T19:14:45.721503+0000","last_change":"2024-09-10T19:13:05.876708+0000","last_active":"2024-09-10T19:14:45.721503+0000","last_peered":"2024-09-10T19:14:45.721503+0000","last_clean":"2024-09-10T19:14:45.721503+0000","last_became_active":"2024-09-10T19:10:07.685223+0000","last_became_peered":"2024-09-10T19:10:07.685223+0000","last_unstale":"2024-09-10T19:14:45.721503+0000","last_undegraded":"2024-09-10T19:14:45.721503+0000","last_fullsized":"2024-09-10T19:14:45.721503+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:13:05.876655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:13:05.876655+0000","last_clean_scrub_stamp":"2024-09-10T19:13:05.876655+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0013242589999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":93,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-09-10T19:14:45.721414+0000","last_change":"2024-09-10T19:13:41.326028+0000","last_active":"2024-09-10T19:14:45.721414+0000","last_peered":"2024-09-10T19:14:45.721414+0000","last_clean":"2024-09-10T19:14:45.721414+0000","last_became_active":"2024-09-10T19:10:11.699013+0000","last_became_peered":"2024-09-10T19:10:11.699013+0000","last_unstale":"2024-09-10T19:14:45.721414+0000","last_undegraded":"2024-09-10T19:14:45.721414+0000","last_fullsized":"2024-09-10T19:14:45.721414+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:13:41.325941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:11:57.112092+0000","last_clean_scrub_stamp":"2024-09-10T19:13:41.325941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:15:05.192727+0000","scrub_duration":0.001598676,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":89,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-09-10T19:14:42.812263+0000","last_change":"2024-09-10T19:13:15.812326+0000","last_active":"2024-09-10T19:14:42.812263+0000","last_peered":"2024-09-10T19:14:42.812263+0000","last_clean":"2024-09-10T19:14:42.812263+0000","last_became_active":"2024-09-10T19:10:14.056172+0000","last_became_peered":"2024-09-10T19:10:14.056172+0000","last_unstale":"2024-09-10T19:14:42.812263+0000","last_undegraded":"2024-09-10T19:14:42.812263+0000","last_fullsized":"2024-09-10T19:14:42.812263+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:13:15.812244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:13:15.812244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0048044389999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"80'2","reported_seq":93,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-09-10T19:14:45.721918+0000","last_change":"2024-09-10T19:12:57.454088+0000","last_active":"2024-09-10T19:14:45.721918+0000","last_peered":"2024-09-10T19:14:45.721918+0000","last_clean":"2024-09-10T19:14:45.721918+0000","last_became_active":"2024-09-10T19:10:13.718875+0000","last_became_peered":"2024-09-10T19:10:13.718875+0000","last_unstale":"2024-09-10T19:14:45.721918+0000","last_undegraded":"2024-09-10T19:14:45.721918+0000","last_fullsized":"2024-09-10T19:14:45.721918+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"80'2","last_scrub_stamp":"2024-09-10T19:12:57.453993+0000","last_deep_scrub":"80'2","last_deep_scrub_stamp":"2024-09-10T19:11:30.272043+0000","last_clean_scrub_stamp":"2024-09-10T19:12:57.453993+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0018294990000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":369,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":97,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-09-10T19:14:45.716287+0000","last_change":"2024-09-10T19:13:14.449643+0000","last_active":"2024-09-10T19:14:45.716287+0000","last_peered":"2024-09-10T19:14:45.716287+0000","last_clean":"2024-09-10T19:14:45.716287+0000","last_became_active":"2024-09-10T19:10:07.689877+0000","last_became_peered":"2024-09-10T19:10:07.689877+0000","last_unstale":"2024-09-10T19:14:45.716287+0000","last_undegraded":"2024-09-10T19:14:45.716287+0000","last_fullsized":"2024-09-10T19:14:45.716287+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:13:14.449593+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:13:14.449593+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001195542,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1e","version":"80'15","reported_seq":113,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-09-10T19:14:42.806874+0000","last_change":"2024-09-10T19:12:50.148157+0000","last_active":"2024-09-10T19:14:42.806874+0000","last_peered":"2024-09-10T19:14:42.806874+0000","last_clean":"2024-09-10T19:14:42.806874+0000","last_became_active":"2024-09-10T19:10:09.823061+0000","last_became_peered":"2024-09-10T19:10:09.823061+0000","last_unstale":"2024-09-10T19:14:42.806874+0000","last_undegraded":"2024-09-10T19:14:42.806874+0000","last_fullsized":"2024-09-10T19:14:42.806874+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"80'15","last_scrub_stamp":"2024-09-10T19:12:50.147336+0000","last_deep_scrub":"80'15","last_deep_scrub_stamp":"2024-09-10T19:11:25.883919+0000","last_clean_scrub_stamp":"2024-09-10T19:12:50.147336+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0064585019999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1f","version":"66'8","reported_seq":160,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-09-10T19:14:45.803937+0000","last_change":"2024-09-10T19:12:48.889499+0000","last_active":"2024-09-10T19:14:45.803937+0000","last_peered":"2024-09-10T19:14:45.803937+0000","last_clean":"2024-09-10T19:14:45.803937+0000","last_became_active":"2024-09-10T19:10:11.695427+0000","last_became_peered":"2024-09-10T19:10:11.695427+0000","last_unstale":"2024-09-10T19:14:45.803937+0000","last_undegraded":"2024-09-10T19:14:45.803937+0000","last_fullsized":"2024-09-10T19:14:45.803937+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'8","last_scrub_stamp":"2024-09-10T19:12:48.889410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:12:48.889410+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001725311,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":47,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-09-10T19:14:45.717241+0000","last_change":"2024-09-10T19:13:54.715498+0000","last_active":"2024-09-10T19:14:45.717241+0000","last_peered":"2024-09-10T19:14:45.717241+0000","last_clean":"2024-09-10T19:14:45.717241+0000","last_became_active":"2024-09-10T19:13:54.715192+0000","last_became_peered":"2024-09-10T19:13:54.715192+0000","last_unstale":"2024-09-10T19:14:45.717241+0000","last_undegraded":"2024-09-10T19:14:45.717241+0000","last_fullsized":"2024-09-10T19:14:45.717241+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:11:30.210603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:11:30.210603+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,2],"acting":[5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":95,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-09-10T19:14:42.809466+0000","last_change":"2024-09-10T19:13:02.737323+0000","last_active":"2024-09-10T19:14:42.809466+0000","last_peered":"2024-09-10T19:14:42.809466+0000","last_clean":"2024-09-10T19:14:42.809466+0000","last_became_active":"2024-09-10T19:10:07.682689+0000","last_became_peered":"2024-09-10T19:10:07.682689+0000","last_unstale":"2024-09-10T19:14:42.809466+0000","last_undegraded":"2024-09-10T19:14:42.809466+0000","last_fullsized":"2024-09-10T19:14:42.809466+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:13:02.737236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:13:02.737236+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0027813289999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"80'16","reported_seq":163,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-09-10T19:14:45.721491+0000","last_change":"2024-09-10T19:13:54.761453+0000","last_active":"2024-09-10T19:14:45.721491+0000","last_peered":"2024-09-10T19:14:45.721491+0000","last_clean":"2024-09-10T19:14:45.721491+0000","last_became_active":"2024-09-10T19:13:54.713563+0000","last_became_peered":"2024-09-10T19:13:54.713563+0000","last_unstale":"2024-09-10T19:14:45.721491+0000","last_undegraded":"2024-09-10T19:14:45.721491+0000","last_fullsized":"2024-09-10T19:14:45.721491+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"80'16","last_scrub_stamp":"2024-09-10T19:12:51.915777+0000","last_deep_scrub":"80'16","last_deep_scrub_stamp":"2024-09-10T19:11:40.853611+0000","last_clean_scrub_stamp":"2024-09-10T19:12:51.915777+0000","objects_scrubbed":5,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0019885430000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,0],"acting":[6,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":91,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-09-10T19:14:42.806482+0000","last_change":"2024-09-10T19:12:59.195000+0000","last_active":"2024-09-10T19:14:42.806482+0000","last_peered":"2024-09-10T19:14:42.806482+0000","last_clean":"2024-09-10T19:14:42.806482+0000","last_became_active":"2024-09-10T19:10:11.701735+0000","last_became_peered":"2024-09-10T19:10:11.701735+0000","last_unstale":"2024-09-10T19:14:42.806482+0000","last_undegraded":"2024-09-10T19:14:42.806482+0000","last_fullsized":"2024-09-10T19:14:42.806482+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:12:59.194914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:11:28.822327+0000","last_clean_scrub_stamp":"2024-09-10T19:12:59.194914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0022130729999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":90,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-09-10T19:14:45.718128+0000","last_change":"2024-09-10T19:13:00.388592+0000","last_active":"2024-09-10T19:14:45.718128+0000","last_peered":"2024-09-10T19:14:45.718128+0000","last_clean":"2024-09-10T19:14:45.718128+0000","last_became_active":"2024-09-10T19:10:14.053619+0000","last_became_peered":"2024-09-10T19:10:14.053619+0000","last_unstale":"2024-09-10T19:14:45.718128+0000","last_undegraded":"2024-09-10T19:14:45.718128+0000","last_fullsized":"2024-09-10T19:14:45.718128+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:13:00.388466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:13:00.388466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.003120996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":97,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-09-10T19:14:45.721051+0000","last_change":"2024-09-10T19:13:43.205095+0000","last_active":"2024-09-10T19:14:45.721051+0000","last_peered":"2024-09-10T19:14:45.721051+0000","last_clean":"2024-09-10T19:14:45.721051+0000","last_became_active":"2024-09-10T19:10:07.685997+0000","last_became_peered":"2024-09-10T19:10:07.685997+0000","last_unstale":"2024-09-10T19:14:45.721051+0000","last_undegraded":"2024-09-10T19:14:45.721051+0000","last_fullsized":"2024-09-10T19:14:45.721051+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:13:43.205042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:13:43.205042+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:14:47.993594+0000","scrub_duration":0.0013181790000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1c","version":"78'22","reported_seq":158,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-09-10T19:14:42.803466+0000","last_change":"2024-09-10T19:13:54.751066+0000","last_active":"2024-09-10T19:14:42.803466+0000","last_peered":"2024-09-10T19:14:42.803466+0000","last_clean":"2024-09-10T19:14:42.803466+0000","last_became_active":"2024-09-10T19:13:54.715570+0000","last_became_peered":"2024-09-10T19:13:54.715570+0000","last_unstale":"2024-09-10T19:14:42.803466+0000","last_undegraded":"2024-09-10T19:14:42.803466+0000","last_fullsized":"2024-09-10T19:14:42.803466+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"78'22","last_scrub_stamp":"2024-09-10T19:12:45.399973+0000","last_deep_scrub":"78'22","last_deep_scrub_stamp":"2024-09-10T19:11:21.118937+0000","last_clean_scrub_stamp":"2024-09-10T19:12:45.399973+0000","objects_scrubbed":7,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0084828799999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":50,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-09-10T19:14:45.717734+0000","last_change":"2024-09-10T19:14:41.058827+0000","last_active":"2024-09-10T19:14:45.717734+0000","last_peered":"2024-09-10T19:14:45.717734+0000","last_clean":"2024-09-10T19:14:45.717734+0000","last_became_active":"2024-09-10T19:13:54.709055+0000","last_became_peered":"2024-09-10T19:13:54.709055+0000","last_unstale":"2024-09-10T19:14:45.717734+0000","last_undegraded":"2024-09-10T19:14:45.717734+0000","last_fullsized":"2024-09-10T19:14:45.717734+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:11:43.221323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:11:43.221323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,5],"acting":[4,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":89,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-09-10T19:14:42.814150+0000","last_change":"2024-09-10T19:12:51.767586+0000","last_active":"2024-09-10T19:14:42.814150+0000","last_peered":"2024-09-10T19:14:42.814150+0000","last_clean":"2024-09-10T19:14:42.814150+0000","last_became_active":"2024-09-10T19:10:14.053698+0000","last_became_peered":"2024-09-10T19:10:14.053698+0000","last_unstale":"2024-09-10T19:14:42.814150+0000","last_undegraded":"2024-09-10T19:14:42.814150+0000","last_fullsized":"2024-09-10T19:14:42.814150+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:12:51.767056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:12:51.767056+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.006796519,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":95,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-09-10T19:14:42.812558+0000","last_change":"2024-09-10T19:12:50.719167+0000","last_active":"2024-09-10T19:14:42.812558+0000","last_peered":"2024-09-10T19:14:42.812558+0000","last_clean":"2024-09-10T19:14:42.812558+0000","last_became_active":"2024-09-10T19:10:07.682811+0000","last_became_peered":"2024-09-10T19:10:07.682811+0000","last_unstale":"2024-09-10T19:14:42.812558+0000","last_undegraded":"2024-09-10T19:14:42.812558+0000","last_fullsized":"2024-09-10T19:14:42.812558+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:12:50.718663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:12:50.718663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0042138399999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1d","version":"66'12","reported_seq":135,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-09-10T19:14:42.812975+0000","last_change":"2024-09-10T19:13:54.758541+0000","last_active":"2024-09-10T19:14:42.812975+0000","last_peered":"2024-09-10T19:14:42.812975+0000","last_clean":"2024-09-10T19:14:42.812975+0000","last_became_active":"2024-09-10T19:13:54.714629+0000","last_became_peered":"2024-09-10T19:13:54.714629+0000","last_unstale":"2024-09-10T19:14:42.812975+0000","last_undegraded":"2024-09-10T19:14:42.812975+0000","last_fullsized":"2024-09-10T19:14:42.812975+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'12","last_scrub_stamp":"2024-09-10T19:11:52.663657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:11:52.663657+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0019136909999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":92,"reported_epoch":126,"state":"active+clean","last_fresh":"2024-09-10T19:14:45.717966+0000","last_change":"2024-09-10T19:13:11.445426+0000","last_active":"2024-09-10T19:14:45.717966+0000","last_peered":"2024-09-10T19:14:45.717966+0000","last_clean":"2024-09-10T19:14:45.717966+0000","last_became_active":"2024-09-10T19:10:11.701155+0000","last_became_peered":"2024-09-10T19:10:11.701155+0000","last_unstale":"2024-09-10T19:14:45.717966+0000","last_undegraded":"2024-09-10T19:14:45.717966+0000","last_fullsized":"2024-09-10T19:14:45.717966+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:13:11.445238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:13:11.445238+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0057421110000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":16,"stat_sum":{"num_bytes":4829413400,"num_objects":73692,"num_object_clones":0,"num_object_copies":221076,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4890,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":73692,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":207066,"num_write_kb":8984193,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":29090,"num_bytes_recovered":1906180120,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":15394349056,"data_stored":15394340912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76547,"ondisk_log_size":76547,"up":48,"acting":44,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":71,"ondisk_log_size":71,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":177,"num_object_clones":0,"num_object_copies":531,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":177,"num_whiteouts":0,"num_read":695,"num_read_kb":452,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":111,"num_bytes_recovered":2448,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":499712,"data_stored":13224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":444,"ondisk_log_size":444,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":120,"num_read_kb":115,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":389,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1613,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1328,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":27,"num_read_kb":27,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":3984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4,"ondisk_log_size":4,"up":96,"acting":96,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79,"ondisk_log_size":79,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":54,"seq":231928234045,"num_pgs":69,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2259804,"kb_used_data":1958292,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":91485348,"statfs":{"total":95995035648,"available":93680996352,"internally_reserved":0,"allocated":2005291008,"data_stored":2004905597,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,2],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 19:13:58 2024","interfaces":[{"interface":"back","average":{"1min":3.7909999999999999,"5min":1.619,"15min":0.89600000000000002},"min":{"1min":0.215,"5min":0.215,"15min":0.215},"max":{"1min":25.640999999999998,"5min":35.551000000000002,"15min":35.551000000000002},"last":2.4580000000000002},{"interface":"front","average":{"1min":3.0299999999999998,"5min":1.4099999999999999,"15min":0.82799999999999996},"min":{"1min":0.224,"5min":0.223,"15min":0.223},"max":{"1min":27.870999999999999,"5min":35.503999999999998,"15min":35.503999999999998},"last":0.27500000000000002}]},{"osd":1,"last update":"Tue Sep 10 19:11:56 2024","interfaces":[{"interface":"back","average":{"1min":0.52400000000000002,"5min":0.52400000000000002,"15min":0.52400000000000002},"min":{"1min":0.36199999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.70199999999999996,"5min":0.79400000000000004,"15min":0.79400000000000004},"last":0.45600000000000002},{"interface":"front","average":{"1min":0.49199999999999999,"5min":0.51200000000000001,"15min":0.51500000000000001},"min":{"1min":0.28999999999999998,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.68500000000000005,"5min":0.71799999999999997,"15min":0.71799999999999997},"last":1.5}]},{"osd":2,"last update":"Tue Sep 10 19:13:58 2024","interfaces":[{"interface":"back","average":{"1min":2.6709999999999998,"5min":0.93799999999999994,"15min":0.66200000000000003},"min":{"1min":0.25600000000000001,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":31.940999999999999,"5min":31.940999999999999,"15min":31.940999999999999},"last":0.86099999999999999},{"interface":"front","average":{"1min":2.6989999999999998,"5min":1.014,"15min":0.70399999999999996},"min":{"1min":0.24099999999999999,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":25.056000000000001,"5min":25.056000000000001,"15min":25.056000000000001},"last":2.1669999999999998}]},{"osd":3,"last update":"Tue Sep 10 19:13:58 2024","interfaces":[{"interface":"back","average":{"1min":2.6509999999999998,"5min":1.5509999999999999,"15min":0.874},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":25.443999999999999,"5min":35.277999999999999,"15min":35.277999999999999},"last":1.879},{"interface":"front","average":{"1min":2.8879999999999999,"5min":1.595,"15min":0.878},"min":{"1min":0.30299999999999999,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":25.530000000000001,"5min":36.512999999999998,"15min":36.512999999999998},"last":2.355}]},{"osd":4,"last update":"Tue Sep 10 19:13:58 2024","interfaces":[{"interface":"back","average":{"1min":0.59699999999999998,"5min":0.47199999999999998,"15min":0.46000000000000002},"min":{"1min":0.23499999999999999,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":2.863,"5min":2.863,"15min":2.863},"last":0.34499999999999997},{"interface":"front","average":{"1min":0.57599999999999996,"5min":0.47699999999999998,"15min":0.46500000000000002},"min":{"1min":0.30399999999999999,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":1.1020000000000001,"5min":1.1020000000000001,"15min":1.1020000000000001},"last":0.90000000000000002}]},{"osd":5,"last update":"Tue Sep 10 19:13:58 2024","interfaces":[{"interface":"back","average":{"1min":1.085,"5min":0.59299999999999997,"15min":0.51700000000000002},"min":{"1min":0.27700000000000002,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":7.2240000000000002,"5min":7.2240000000000002,"15min":7.2240000000000002},"last":0.38900000000000001},{"interface":"front","average":{"1min":0.626,"5min":0.51000000000000001,"15min":0.48699999999999999},"min":{"1min":0.26600000000000001,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":2.8980000000000001,"5min":2.8980000000000001,"15min":2.8980000000000001},"last":1.2370000000000001}]},{"osd":6,"last update":"Tue Sep 10 19:13:58 2024","interfaces":[{"interface":"back","average":{"1min":0.54600000000000004,"5min":0.51900000000000002,"15min":0.50600000000000001},"min":{"1min":0.29099999999999998,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.91400000000000003,"5min":0.91600000000000004,"15min":0.91600000000000004},"last":0.50600000000000001},{"interface":"front","average":{"1min":0.48899999999999999,"5min":0.48599999999999999,"15min":0.48799999999999999},"min":{"1min":0.28000000000000003,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.83499999999999996,"5min":0.83499999999999996,"15min":0.83499999999999996},"last":1.0589999999999999}]}]},{"osd":6,"up_from":47,"seq":201863462978,"num_pgs":61,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2029672,"kb_used_data":1719840,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":91715480,"statfs":{"total":95995035648,"available":93916651520,"internally_reserved":0,"allocated":1761116160,"data_stored":1760719029,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[0,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 19:14:42 2024","interfaces":[{"interface":"back","average":{"1min":7.5679999999999996,"5min":1.994,"15min":1.012},"min":{"1min":0.219,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":112.988,"5min":112.988,"15min":112.988},"last":0.45300000000000001},{"interface":"front","average":{"1min":7.3399999999999999,"5min":1.9790000000000001,"15min":1.022},"min":{"1min":0.22900000000000001,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":113.18300000000001,"5min":113.18300000000001,"15min":113.18300000000001},"last":0.38200000000000001}]},{"osd":1,"last update":"Tue Sep 10 19:12:41 2024","interfaces":[{"interface":"back","average":{"1min":0.59999999999999998,"5min":0.53700000000000003,"15min":0.53100000000000003},"min":{"1min":0.33800000000000002,"5min":0.318,"15min":0.318},"max":{"1min":1.29,"5min":1.29,"15min":1.29},"last":0.26500000000000001},{"interface":"front","average":{"1min":0.69399999999999995,"5min":0.55400000000000005,"15min":0.53800000000000003},"min":{"1min":0.32400000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":2.6859999999999999,"5min":2.6859999999999999,"15min":2.6859999999999999},"last":0.38200000000000001}]},{"osd":2,"last update":"Tue Sep 10 19:14:42 2024","interfaces":[{"interface":"back","average":{"1min":7.0099999999999998,"5min":1.8879999999999999,"15min":0.99199999999999999},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":112.864,"5min":112.864,"15min":112.864},"last":0.52500000000000002},{"interface":"front","average":{"1min":7.2869999999999999,"5min":1.966,"15min":0.995},"min":{"1min":0.28299999999999997,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":112.822,"5min":112.822,"15min":112.822},"last":0.43099999999999999}]},{"osd":3,"last update":"Tue Sep 10 19:14:42 2024","interfaces":[{"interface":"back","average":{"1min":7.9199999999999999,"5min":2.109,"15min":1.0469999999999999},"min":{"1min":0.26000000000000001,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":125.56399999999999,"5min":125.56399999999999,"15min":125.56399999999999},"last":0.39800000000000002},{"interface":"front","average":{"1min":8.2170000000000005,"5min":2.2130000000000001,"15min":1.115},"min":{"1min":0.26500000000000001,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":125.47,"5min":125.47,"15min":125.47},"last":0.48199999999999998}]},{"osd":4,"last update":"Tue Sep 10 19:14:42 2024","interfaces":[{"interface":"back","average":{"1min":0.623,"5min":0.45200000000000001,"15min":0.38200000000000001},"min":{"1min":0.27800000000000002,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":3.3410000000000002,"5min":3.3410000000000002,"15min":3.3410000000000002},"last":0.40799999999999997},{"interface":"front","average":{"1min":0.52600000000000002,"5min":0.48799999999999999,"15min":0.41199999999999998},"min":{"1min":0.28799999999999998,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":1.889,"5min":2.0710000000000002,"15min":2.0710000000000002},"last":0.32400000000000001}]},{"osd":5,"last update":"Tue Sep 10 19:14:42 2024","interfaces":[{"interface":"back","average":{"1min":3.0190000000000001,"5min":0.94099999999999995,"15min":0.56299999999999994},"min":{"1min":0.312,"5min":0.248,"15min":0.248},"max":{"1min":44.765000000000001,"5min":44.765000000000001,"15min":44.765000000000001},"last":0.83299999999999996},{"interface":"front","average":{"1min":2.9990000000000001,"5min":1.006,"15min":0.58399999999999996},"min":{"1min":0.318,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":43.036000000000001,"5min":43.036000000000001,"15min":43.036000000000001},"last":0.86599999999999999}]},{"osd":7,"last update":"Tue Sep 10 19:14:00 2024","interfaces":[{"interface":"back","average":{"1min":0.47499999999999998,"5min":0.47099999999999997,"15min":0.45500000000000002},"min":{"1min":0.27500000000000002,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":1.6559999999999999,"5min":1.6559999999999999,"15min":1.6559999999999999},"last":0.503},{"interface":"front","average":{"1min":0.64300000000000002,"5min":0.53200000000000003,"15min":0.47399999999999998},"min":{"1min":0.29299999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":2.3260000000000001,"5min":2.3260000000000001,"15min":2.3260000000000001},"last":0.41899999999999998}]}]},{"osd":5,"up_from":39,"seq":167503724615,"num_pgs":60,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1470472,"kb_used_data":1160640,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":92274680,"statfs":{"total":95995035648,"available":94489272320,"internally_reserved":0,"allocated":1188495360,"data_stored":1188130551,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[0,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 19:14:13 2024","interfaces":[{"interface":"back","average":{"1min":1.0089999999999999,"5min":0.626,"15min":0.52500000000000002},"min":{"1min":0.24099999999999999,"5min":0.223,"15min":0.223},"max":{"1min":5.851,"5min":5.851,"15min":5.851},"last":1.161},{"interface":"front","average":{"1min":0.98199999999999998,"5min":0.622,"15min":0.52100000000000002},"min":{"1min":0.224,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":5.6500000000000004,"5min":5.6500000000000004,"15min":5.6500000000000004},"last":1.1040000000000001}]},{"osd":1,"last update":"Tue Sep 10 19:12:09 2024","interfaces":[{"interface":"back","average":{"1min":0.48199999999999998,"5min":0.51500000000000001,"15min":0.50700000000000001},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.64600000000000002,"5min":0.72799999999999998,"15min":0.72799999999999998},"last":0.51000000000000001},{"interface":"front","average":{"1min":0.47299999999999998,"5min":0.47899999999999998,"15min":0.47399999999999998},"min":{"1min":0.34399999999999997,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.61599999999999999,"5min":0.68799999999999994,"15min":0.68799999999999994},"last":3.1960000000000002}]},{"osd":2,"last update":"Tue Sep 10 19:14:13 2024","interfaces":[{"interface":"back","average":{"1min":0.69299999999999995,"5min":0.59999999999999998,"15min":0.51600000000000001},"min":{"1min":0.27000000000000002,"5min":0.253,"15min":0.253},"max":{"1min":1.7270000000000001,"5min":2.2679999999999998,"15min":2.2679999999999998},"last":0.77700000000000002},{"interface":"front","average":{"1min":1.645,"5min":0.73599999999999999,"15min":0.56999999999999995},"min":{"1min":0.224,"5min":0.224,"15min":0.224},"max":{"1min":6.3789999999999996,"5min":6.3789999999999996,"15min":6.3789999999999996},"last":0.48499999999999999}]},{"osd":3,"last update":"Tue Sep 10 19:14:13 2024","interfaces":[{"interface":"back","average":{"1min":0.64400000000000002,"5min":0.60499999999999998,"15min":0.54200000000000004},"min":{"1min":0.26700000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":3.351,"5min":3.351,"15min":3.351},"last":0.32200000000000001},{"interface":"front","average":{"1min":0.76200000000000001,"5min":0.61499999999999999,"15min":0.55300000000000005},"min":{"1min":0.251,"5min":0.248,"15min":0.248},"max":{"1min":3.403,"5min":3.403,"15min":3.403},"last":0.438}]},{"osd":4,"last update":"Tue Sep 10 19:14:13 2024","interfaces":[{"interface":"back","average":{"1min":0.77500000000000002,"5min":0.47599999999999998,"15min":0.39400000000000002},"min":{"1min":0.26000000000000001,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":2.8090000000000002,"5min":2.8090000000000002,"15min":2.8090000000000002},"last":0.42499999999999999},{"interface":"front","average":{"1min":0.96199999999999997,"5min":0.56999999999999995,"15min":0.42999999999999999},"min":{"1min":0.311,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":5.8899999999999997,"5min":5.8899999999999997,"15min":5.8899999999999997},"last":0.71599999999999997}]},{"osd":6,"last update":"Tue Sep 10 19:14:42 2024","interfaces":[{"interface":"back","average":{"1min":1.3979999999999999,"5min":0.69099999999999995,"15min":0.504},"min":{"1min":0.309,"5min":0.223,"15min":0.223},"max":{"1min":6.617,"5min":6.617,"15min":6.617},"last":0.29099999999999998},{"interface":"front","average":{"1min":1.2849999999999999,"5min":0.65200000000000002,"15min":0.48799999999999999},"min":{"1min":0.26900000000000002,"5min":0.25,"15min":0.25},"max":{"1min":5.9349999999999996,"5min":5.9349999999999996,"15min":5.9349999999999996},"last":0.627}]},{"osd":7,"last update":"Tue Sep 10 19:13:57 2024","interfaces":[{"interface":"back","average":{"1min":0.72299999999999998,"5min":0.54300000000000004,"15min":0.52100000000000002},"min":{"1min":0.316,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":1.9970000000000001,"5min":1.9970000000000001,"15min":1.9970000000000001},"last":0.51300000000000001},{"interface":"front","average":{"1min":0.60799999999999998,"5min":0.50900000000000001,"15min":0.48899999999999999},"min":{"1min":0.30499999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":2.0099999999999998,"5min":2.0099999999999998,"15min":2.0099999999999998},"last":0.45700000000000002}]}]},{"osd":4,"up_from":32,"seq":137438953547,"num_pgs":61,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1738828,"kb_used_data":1433092,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":92006324,"statfs":{"total":95995035648,"available":94214475776,"internally_reserved":0,"allocated":1467486208,"data_stored":1467116048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,2,2,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 19:13:55 2024","interfaces":[{"interface":"back","average":{"1min":0.61499999999999999,"5min":0.54600000000000004,"15min":0.54300000000000004},"min":{"1min":0.22900000000000001,"5min":0.22,"15min":0.22},"max":{"1min":2.157,"5min":2.157,"15min":2.157},"last":1.329},{"interface":"front","average":{"1min":0.60399999999999998,"5min":0.54600000000000004,"15min":0.53600000000000003},"min":{"1min":0.20999999999999999,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":2.1070000000000002,"5min":2.5499999999999998,"15min":2.5499999999999998},"last":2.4569999999999999}]},{"osd":1,"last update":"Tue Sep 10 19:11:49 2024","interfaces":[{"interface":"back","average":{"1min":0.47999999999999998,"5min":0.503,"15min":0.50900000000000001},"min":{"1min":0.29999999999999999,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.72199999999999998,"5min":0.83499999999999996,"15min":0.83499999999999996},"last":0.997},{"interface":"front","average":{"1min":0.51000000000000001,"5min":0.51700000000000002,"15min":0.52200000000000002},"min":{"1min":0.27500000000000002,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.68999999999999995,"5min":0.77100000000000002,"15min":0.77100000000000002},"last":0.67400000000000004}]},{"osd":2,"last update":"Tue Sep 10 19:13:55 2024","interfaces":[{"interface":"back","average":{"1min":0.83099999999999996,"5min":0.56999999999999995,"15min":0.54900000000000004},"min":{"1min":0.27400000000000002,"5min":0.221,"15min":0.221},"max":{"1min":2.54,"5min":2.54,"15min":2.54},"last":1.3069999999999999},{"interface":"front","average":{"1min":0.753,"5min":0.56499999999999995,"15min":0.54700000000000004},"min":{"1min":0.27100000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":2.8679999999999999,"5min":2.8679999999999999,"15min":2.8679999999999999},"last":1.5049999999999999}]},{"osd":3,"last update":"Tue Sep 10 19:13:55 2024","interfaces":[{"interface":"back","average":{"1min":0.755,"5min":0.59399999999999997,"15min":0.54900000000000004},"min":{"1min":0.252,"5min":0.252,"15min":0.252},"max":{"1min":1.712,"5min":2.109,"15min":2.109},"last":1.554},{"interface":"front","average":{"1min":0.95099999999999996,"5min":0.629,"15min":0.57699999999999996},"min":{"1min":0.26200000000000001,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":2.6850000000000001,"5min":2.6850000000000001,"15min":2.6850000000000001},"last":0.67200000000000004}]},{"osd":5,"last update":"Tue Sep 10 19:14:18 2024","interfaces":[{"interface":"back","average":{"1min":0.61299999999999999,"5min":0.44800000000000001,"15min":0.39800000000000002},"min":{"1min":0.22900000000000001,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":3.629,"5min":3.629,"15min":3.629},"last":0.39200000000000002},{"interface":"front","average":{"1min":0.62,"5min":0.47899999999999998,"15min":0.42199999999999999},"min":{"1min":0.248,"5min":0.223,"15min":0.223},"max":{"1min":2.1880000000000002,"5min":2.1880000000000002,"15min":2.1880000000000002},"last":0.46999999999999997}]},{"osd":6,"last update":"Tue Sep 10 19:14:44 2024","interfaces":[{"interface":"back","average":{"1min":1.0569999999999999,"5min":0.59999999999999998,"15min":0.47499999999999998},"min":{"1min":0.28999999999999998,"5min":0.253,"15min":0.253},"max":{"1min":3.964,"5min":3.964,"15min":3.964},"last":1.2769999999999999},{"interface":"front","average":{"1min":0.91800000000000004,"5min":0.54800000000000004,"15min":0.45800000000000002},"min":{"1min":0.27700000000000002,"5min":0.25,"15min":0.25},"max":{"1min":3.0129999999999999,"5min":3.0129999999999999,"15min":3.0129999999999999},"last":0.42399999999999999}]},{"osd":7,"last update":"Tue Sep 10 19:14:03 2024","interfaces":[{"interface":"back","average":{"1min":2.149,"5min":0.82499999999999996,"15min":0.625},"min":{"1min":0.29299999999999998,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":23.081,"5min":23.081,"15min":23.081},"last":1.147},{"interface":"front","average":{"1min":2.1280000000000001,"5min":0.82799999999999996,"15min":0.61099999999999999},"min":{"1min":0.311,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":22.666,"5min":22.666,"15min":22.666},"last":1.5329999999999999}]}]},{"osd":3,"up_from":26,"seq":111669149775,"num_pgs":71,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3170952,"kb_used_data":2873600,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":90574200,"statfs":{"total":95995035648,"available":92747980800,"internally_reserved":0,"allocated":2942566400,"data_stored":2942177801,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 19:14:30 2024","interfaces":[{"interface":"back","average":{"1min":1.8220000000000001,"5min":0.90300000000000002,"15min":0.57199999999999995},"min":{"1min":0.16800000000000001,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":4.5860000000000003,"5min":4.5860000000000003,"15min":4.5860000000000003},"last":4.9539999999999997},{"interface":"front","average":{"1min":2.3769999999999998,"5min":1.107,"15min":0.64900000000000002},"min":{"1min":0.17899999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":7.7400000000000002,"5min":7.7400000000000002,"15min":7.7400000000000002},"last":3.5710000000000002}]},{"osd":1,"last update":"Tue Sep 10 19:12:27 2024","interfaces":[{"interface":"back","average":{"1min":0.52800000000000002,"5min":0.44400000000000001,"15min":0.45200000000000001},"min":{"1min":0.216,"5min":0.216,"15min":0.216},"max":{"1min":3.5150000000000001,"5min":3.5150000000000001,"15min":3.5150000000000001},"last":0.79700000000000004},{"interface":"front","average":{"1min":0.46600000000000003,"5min":0.40000000000000002,"15min":0.39800000000000002},"min":{"1min":0.187,"5min":0.187,"15min":0.187},"max":{"1min":1.3640000000000001,"5min":1.3640000000000001,"15min":1.3640000000000001},"last":0.59099999999999997}]},{"osd":2,"last update":"Tue Sep 10 19:14:30 2024","interfaces":[{"interface":"back","average":{"1min":1.331,"5min":0.745,"15min":0.54100000000000004},"min":{"1min":0.20599999999999999,"5min":0.186,"15min":0.186},"max":{"1min":3.7909999999999999,"5min":3.7909999999999999,"15min":3.7909999999999999},"last":1.1890000000000001},{"interface":"front","average":{"1min":1.6339999999999999,"5min":0.86899999999999999,"15min":0.56699999999999995},"min":{"1min":0.23300000000000001,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":4.5,"5min":4.5,"15min":4.5},"last":2.4239999999999999}]},{"osd":4,"last update":"Tue Sep 10 19:13:56 2024","interfaces":[{"interface":"back","average":{"1min":1.73,"5min":0.94199999999999995,"15min":0.67700000000000005},"min":{"1min":0.31900000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":4.54,"5min":11.964,"15min":11.964},"last":0.499},{"interface":"front","average":{"1min":1.6799999999999999,"5min":1.026,"15min":0.70699999999999996},"min":{"1min":0.26200000000000001,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":6.0350000000000001,"5min":11.881,"15min":11.881},"last":3.5009999999999999}]},{"osd":5,"last update":"Tue Sep 10 19:14:16 2024","interfaces":[{"interface":"back","average":{"1min":1.456,"5min":0.877,"15min":0.66900000000000004},"min":{"1min":0.32500000000000001,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":4.6070000000000002,"5min":4.6070000000000002,"15min":4.6070000000000002},"last":0.56499999999999995},{"interface":"front","average":{"1min":1.643,"5min":1.034,"15min":0.71799999999999997},"min":{"1min":0.28599999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":4.2770000000000001,"5min":5.9950000000000001,"15min":5.9950000000000001},"last":3.54}]},{"osd":6,"last update":"Tue Sep 10 19:14:38 2024","interfaces":[{"interface":"back","average":{"1min":1.675,"5min":1.032,"15min":0.72999999999999998},"min":{"1min":0.36899999999999999,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":3.9329999999999998,"5min":4.5629999999999997,"15min":4.5629999999999997},"last":1.236},{"interface":"front","average":{"1min":1.1459999999999999,"5min":0.77100000000000002,"15min":0.628},"min":{"1min":0.30499999999999999,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":3.4199999999999999,"5min":3.4199999999999999,"15min":3.4199999999999999},"last":1.1679999999999999}]},{"osd":7,"last update":"Tue Sep 10 19:14:07 2024","interfaces":[{"interface":"back","average":{"1min":1.3009999999999999,"5min":0.86499999999999999,"15min":0.67500000000000004},"min":{"1min":0.434,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":3.3029999999999999,"5min":13.538,"15min":13.538},"last":0.66700000000000004},{"interface":"front","average":{"1min":1.595,"5min":1.016,"15min":0.72799999999999998},"min":{"1min":0.35499999999999998,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":4.6280000000000001,"5min":13.522,"15min":13.522},"last":0.53100000000000003}]}]},{"osd":2,"up_from":18,"seq":77309411412,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2587164,"kb_used_data":2277328,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":91157988,"statfs":{"total":95995035648,"available":93345779712,"internally_reserved":0,"allocated":2331983872,"data_stored":2331642621,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[0,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 19:14:16 2024","interfaces":[{"interface":"back","average":{"1min":1.3799999999999999,"5min":1.5049999999999999,"15min":0.72799999999999998},"min":{"1min":0.20000000000000001,"5min":0.188,"15min":0.188},"max":{"1min":4.4989999999999997,"5min":62.543999999999997,"15min":62.543999999999997},"last":0.47199999999999998},{"interface":"front","average":{"1min":1.548,"5min":1.3979999999999999,"15min":0.70899999999999996},"min":{"1min":0.28499999999999998,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":3.0979999999999999,"5min":62.576999999999998,"15min":62.576999999999998},"last":1.9790000000000001}]},{"osd":1,"last update":"Tue Sep 10 19:12:08 2024","interfaces":[{"interface":"back","average":{"1min":0.317,"5min":0.378,"15min":0.375},"min":{"1min":0.23400000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.46800000000000003,"5min":0.69799999999999995,"15min":0.69799999999999995},"last":2.6230000000000002},{"interface":"front","average":{"1min":0.33300000000000002,"5min":0.38600000000000001,"15min":0.38800000000000001},"min":{"1min":0.255,"5min":0.255,"15min":0.255},"max":{"1min":0.499,"5min":0.63400000000000001,"15min":0.63400000000000001},"last":3.9940000000000002}]},{"osd":3,"last update":"Tue Sep 10 19:14:37 2024","interfaces":[{"interface":"back","average":{"1min":2.0630000000000002,"5min":1.7310000000000001,"15min":0.878},"min":{"1min":0.20899999999999999,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":5.9889999999999999,"5min":63.997,"15min":63.997},"last":1.4399999999999999},{"interface":"front","average":{"1min":1.45,"5min":1.498,"15min":0.79800000000000004},"min":{"1min":0.25700000000000001,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":7.1289999999999996,"5min":62.365000000000002,"15min":62.365000000000002},"last":0.69899999999999995}]},{"osd":4,"last update":"Tue Sep 10 19:13:57 2024","interfaces":[{"interface":"back","average":{"1min":1.127,"5min":0.75800000000000001,"15min":0.64000000000000001},"min":{"1min":0.254,"5min":0.254,"15min":0.254},"max":{"1min":4.3650000000000002,"5min":4.3650000000000002,"15min":4.3650000000000002},"last":1.4710000000000001},{"interface":"front","average":{"1min":1.1619999999999999,"5min":0.752,"15min":0.63200000000000001},"min":{"1min":0.26000000000000001,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":3.0899999999999999,"5min":4.0750000000000002,"15min":4.0750000000000002},"last":2.0630000000000002}]},{"osd":5,"last update":"Tue Sep 10 19:14:16 2024","interfaces":[{"interface":"back","average":{"1min":0.65700000000000003,"5min":0.68000000000000005,"15min":0.57599999999999996},"min":{"1min":0.255,"5min":0.255,"15min":0.255},"max":{"1min":3.234,"5min":5.4820000000000002,"15min":5.4820000000000002},"last":0.627},{"interface":"front","average":{"1min":1.2490000000000001,"5min":0.82099999999999995,"15min":0.65900000000000003},"min":{"1min":0.35299999999999998,"5min":0.309,"15min":0.309},"max":{"1min":4.4729999999999999,"5min":4.6429999999999998,"15min":4.6429999999999998},"last":0.44900000000000001}]},{"osd":6,"last update":"Tue Sep 10 19:14:40 2024","interfaces":[{"interface":"back","average":{"1min":2.2309999999999999,"5min":1.089,"15min":0.73499999999999999},"min":{"1min":0.29499999999999998,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":12.611000000000001,"5min":12.611000000000001,"15min":12.611000000000001},"last":1.9670000000000001},{"interface":"front","average":{"1min":2.1389999999999998,"5min":1.073,"15min":0.72499999999999998},"min":{"1min":0.31,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":14.459,"5min":14.459,"15min":14.459},"last":2.0910000000000002}]},{"osd":7,"last update":"Tue Sep 10 19:14:03 2024","interfaces":[{"interface":"back","average":{"1min":1.589,"5min":0.84699999999999998,"15min":0.68600000000000005},"min":{"1min":0.41599999999999998,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":4.5640000000000001,"5min":4.5640000000000001,"15min":4.5640000000000001},"last":0.41399999999999998},{"interface":"front","average":{"1min":1.226,"5min":0.79200000000000004,"15min":0.63},"min":{"1min":0.312,"5min":0.312,"15min":0.312},"max":{"1min":3.1179999999999999,"5min":4.0970000000000004,"15min":4.0970000000000004},"last":0.67000000000000004}]}]},{"osd":1,"up_from":0,"seq":0,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":8,"seq":34359738462,"num_pgs":64,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2754556,"kb_used_data":2444724,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":90990596,"statfs":{"total":95995035648,"available":93174370304,"internally_reserved":0,"allocated":2503397376,"data_stored":2503021155,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Sep 10 19:12:44 2024","interfaces":[{"interface":"back","average":{"1min":0.73199999999999998,"5min":0.42999999999999999,"15min":0.35799999999999998},"min":{"1min":0.17399999999999999,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":2.7320000000000002,"5min":2.7320000000000002,"15min":2.7320000000000002},"last":0.80700000000000005},{"interface":"front","average":{"1min":0.82299999999999995,"5min":0.45100000000000001,"15min":0.36799999999999999},"min":{"1min":0.21199999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":2.4609999999999999,"5min":2.4609999999999999,"15min":2.4609999999999999},"last":2.4609999999999999}]},{"osd":2,"last update":"Tue Sep 10 19:14:16 2024","interfaces":[{"interface":"back","average":{"1min":1.252,"5min":0.67800000000000005,"15min":0.47999999999999998},"min":{"1min":0.17199999999999999,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":5.8140000000000001,"5min":5.8140000000000001,"15min":5.8140000000000001},"last":0.79100000000000004},{"interface":"front","average":{"1min":1.1180000000000001,"5min":0.60099999999999998,"15min":0.44800000000000001},"min":{"1min":0.216,"5min":0.187,"15min":0.187},"max":{"1min":3.399,"5min":3.399,"15min":3.399},"last":9.9399999999999995}]},{"osd":3,"last update":"Tue Sep 10 19:14:37 2024","interfaces":[{"interface":"back","average":{"1min":0.95099999999999996,"5min":0.70099999999999996,"15min":0.51600000000000001},"min":{"1min":0.23300000000000001,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":3.331,"5min":3.331,"15min":3.331},"last":4.9329999999999998},{"interface":"front","average":{"1min":1.516,"5min":0.82899999999999996,"15min":0.54900000000000004},"min":{"1min":0.27700000000000002,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":6.931,"5min":6.931,"15min":6.931},"last":6.3479999999999999}]},{"osd":4,"last update":"Tue Sep 10 19:13:49 2024","interfaces":[{"interface":"back","average":{"1min":1.3260000000000001,"5min":0.70699999999999996,"15min":0.61299999999999999},"min":{"1min":0.24399999999999999,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":5.8499999999999996,"5min":5.8499999999999996,"15min":5.8499999999999996},"last":0.36099999999999999},{"interface":"front","average":{"1min":0.998,"5min":0.70899999999999996,"15min":0.61099999999999999},"min":{"1min":0.27700000000000002,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":3.2799999999999998,"5min":3.2999999999999998,"15min":3.2999999999999998},"last":0.42099999999999999}]},{"osd":5,"last update":"Tue Sep 10 19:14:16 2024","interfaces":[{"interface":"back","average":{"1min":1.0620000000000001,"5min":0.72699999999999998,"15min":0.63200000000000001},"min":{"1min":0.29499999999999998,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":3.6190000000000002,"5min":3.6190000000000002,"15min":3.6190000000000002},"last":0.38800000000000001},{"interface":"front","average":{"1min":1.0640000000000001,"5min":0.748,"15min":0.63300000000000001},"min":{"1min":0.253,"5min":0.253,"15min":0.253},"max":{"1min":3.5720000000000001,"5min":4.9790000000000001,"15min":4.9790000000000001},"last":0.48799999999999999}]},{"osd":6,"last update":"Tue Sep 10 19:14:37 2024","interfaces":[{"interface":"back","average":{"1min":1.6040000000000001,"5min":0.88500000000000001,"15min":0.67300000000000004},"min":{"1min":0.312,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":5.8730000000000002,"5min":5.8730000000000002,"15min":5.8730000000000002},"last":0.439},{"interface":"front","average":{"1min":1.5489999999999999,"5min":0.89500000000000002,"15min":0.68899999999999995},"min":{"1min":0.34399999999999997,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":5.2850000000000001,"5min":5.2850000000000001,"15min":5.2850000000000001},"last":0.51000000000000001}]},{"osd":7,"last update":"Tue Sep 10 19:14:02 2024","interfaces":[{"interface":"back","average":{"1min":1.0629999999999999,"5min":0.76000000000000001,"15min":0.63800000000000001},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":3.3580000000000001,"5min":3.3919999999999999,"15min":3.3919999999999999},"last":0.47699999999999998},{"interface":"front","average":{"1min":1.1279999999999999,"5min":0.69399999999999995,"15min":0.60599999999999998},"min":{"1min":0.39500000000000002,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":4.9470000000000001,"5min":4.9470000000000001,"15min":4.9470000000000001},"last":0.45200000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":966,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1236,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1236,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":362,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1650,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":2118,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":908,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":2228,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1942,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":2461077504,"data_stored":2461073432,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1325072384,"data_stored":1325072384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":2307129344,"data_stored":2307129344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":2921725952,"data_stored":2921725952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1452277760,"data_stored":1452277760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1181745152,"data_stored":1181745152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1753026560,"data_stored":1753022488,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1992294400,"data_stored":1992294400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T19:14:51.850 INFO:tasks.thrashosds.thrasher:Setting 4.1 to [1, 0, 4, 2, 6, 7] 2024-09-10T19:14:51.850 INFO:tasks.thrashosds.thrasher:cmd ['osd', 'pg-upmap-items', '4.1', '1', '0', '4', '2', '6', '7'] 2024-09-10T19:14:51.850 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph osd pg-upmap-items 4.1 1 0 4 2 6 7 2024-09-10T19:14:51.872 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:14:51.901 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:14:51.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:14:51.902 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:14:52.002 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:14:52.034 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:14:52.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:14:52.035 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:14:52.135 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:14:52.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:14:52.179 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:14:52.179 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:14:52.280 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:14:52.311 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:14:52.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:14:52.312 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:14:52.413 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:14:52.460 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:14:52.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:14:52.461 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:14:52.562 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:14:52.590 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:14:52.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:14:52.592 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:14:52.692 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:14:52.725 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:14:52.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:14:52.726 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:14:52.826 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:14:52.843 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:14:52.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:14:52.878 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:14:52.878 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:14:52.925 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:14:52.979 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:14:53.013 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:14:53.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:14:53.014 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:14:53.115 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:14:53.115 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-10T19:14:53.159 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:14:53.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:14:53.160 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:14:53.166 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:14:52 smithi183 ceph-mon[29736]: 5.c scrub starts 2024-09-10T19:14:53.166 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:14:52 smithi183 ceph-mon[29736]: pgmap v182: 148 pgs: 3 active+undersized+degraded+remapped+backfilling, 145 active+clean; 4.4 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 852 B/s rd, 0 B/s wr, 700 op/s; 3038/214452 objects degraded (1.417%); 49 MiB/s, 779 objects/s recovering 2024-09-10T19:14:53.166 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:14:52 smithi183 ceph-mon[41562]: 5.c scrub starts 2024-09-10T19:14:53.166 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:14:52 smithi183 ceph-mon[41562]: pgmap v182: 148 pgs: 3 active+undersized+degraded+remapped+backfilling, 145 active+clean; 4.4 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 852 B/s rd, 0 B/s wr, 700 op/s; 3038/214452 objects degraded (1.417%); 49 MiB/s, 779 objects/s recovering 2024-09-10T19:14:53.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:14:52 smithi191 ceph-mon[32884]: 5.c scrub starts 2024-09-10T19:14:53.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:14:52 smithi191 ceph-mon[32884]: pgmap v182: 148 pgs: 3 active+undersized+degraded+remapped+backfilling, 145 active+clean; 4.4 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 852 B/s rd, 0 B/s wr, 700 op/s; 3038/214452 objects degraded (1.417%); 49 MiB/s, 779 objects/s recovering 2024-09-10T19:14:53.261 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:14:53.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:14:53.306 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:14:53.306 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:14:53.407 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:14:53.444 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:14:53.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:14:53.445 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:14:53.546 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:14:53.583 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:14:53.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:14:53.584 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:14:53.685 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:14:53.715 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:14:53.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:14:53.716 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:14:53.816 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:14:53.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:14:53.867 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:14:53.867 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:14:53.968 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:14:54.006 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:14:54.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:14:54.007 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:14:54.107 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:14:54.147 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:14:54.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:14:54.149 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:14:54.250 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:14:54.279 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:14:54.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:14:54.280 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:14:54.381 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:14:54.410 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:14:54.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:14:54.411 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:14:54.512 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:14:54.541 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:14:54.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:14:54.542 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:14:54.643 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:14:54.675 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:14:54.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:14:54.676 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:14:54.776 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:14:54.806 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:14:54.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:14:54.807 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:14:54.908 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:14:54.931 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:14:54 smithi183 ceph-mon[29736]: pgmap v183: 148 pgs: 3 active+undersized+degraded+remapped+backfilling, 145 active+clean; 4.4 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 641 B/s rd, 0 B/s wr, 526 op/s; 3038/214452 objects degraded (1.417%); 37 MiB/s, 586 objects/s recovering 2024-09-10T19:14:54.931 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:14:54 smithi183 ceph-mon[29736]: Health check update: Degraded data redundancy: 3038/214452 objects degraded (1.417%), 3 pgs degraded (PG_DEGRADED) 2024-09-10T19:14:54.931 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:14:54 smithi183 ceph-mon[29736]: osdmap e129: 8 total, 7 up, 7 in 2024-09-10T19:14:54.931 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:14:54 smithi183 ceph-mon[41562]: pgmap v183: 148 pgs: 3 active+undersized+degraded+remapped+backfilling, 145 active+clean; 4.4 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 641 B/s rd, 0 B/s wr, 526 op/s; 3038/214452 objects degraded (1.417%); 37 MiB/s, 586 objects/s recovering 2024-09-10T19:14:54.932 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:14:54 smithi183 ceph-mon[41562]: Health check update: Degraded data redundancy: 3038/214452 objects degraded (1.417%), 3 pgs degraded (PG_DEGRADED) 2024-09-10T19:14:54.932 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:14:54 smithi183 ceph-mon[41562]: osdmap e129: 8 total, 7 up, 7 in 2024-09-10T19:14:54.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:14:54.945 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:14:54.945 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:14:55.046 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:14:55.076 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:14:55.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:14:55.078 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:14:55.178 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:14:55.215 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:14:55.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:14:54 smithi191 ceph-mon[32884]: pgmap v183: 148 pgs: 3 active+undersized+degraded+remapped+backfilling, 145 active+clean; 4.4 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 641 B/s rd, 0 B/s wr, 526 op/s; 3038/214452 objects degraded (1.417%); 37 MiB/s, 586 objects/s recovering 2024-09-10T19:14:55.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:14:54 smithi191 ceph-mon[32884]: Health check update: Degraded data redundancy: 3038/214452 objects degraded (1.417%), 3 pgs degraded (PG_DEGRADED) 2024-09-10T19:14:55.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:14:54 smithi191 ceph-mon[32884]: osdmap e129: 8 total, 7 up, 7 in 2024-09-10T19:14:55.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:14:55.216 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:14:55.317 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:14:55.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:14:55.346 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:14:55.347 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:14:55.448 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:14:55.478 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:14:55.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:14:55.479 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:14:55.580 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:14:55.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:14:55.610 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:14:55.610 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:14:55.713 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:14:55.797 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:14:55.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:14:55.798 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:14:55.898 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:14:55.927 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:14:55.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:14:55.928 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:14:55.982 INFO:teuthology.orchestra.run.smithi183.stderr:noscrub is set 2024-09-10T19:14:56.029 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:14:56.051 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:14:56.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:14:56.052 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:14:56.152 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:14:56.174 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:14:56.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:14:56.175 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:14:56.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:14:55 smithi183 ceph-mon[29736]: purged_snaps scrub starts 2024-09-10T19:14:56.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:14:55 smithi183 ceph-mon[29736]: purged_snaps scrub ok 2024-09-10T19:14:56.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:14:55 smithi183 ceph-mon[29736]: purged_snaps scrub starts 2024-09-10T19:14:56.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:14:55 smithi183 ceph-mon[29736]: purged_snaps scrub ok 2024-09-10T19:14:56.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:14:55 smithi183 ceph-mon[29736]: from='client.? 172.21.15.183:0/1095435823' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-10T19:14:56.181 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:14:55 smithi183 ceph-mon[41562]: purged_snaps scrub starts 2024-09-10T19:14:56.181 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:14:55 smithi183 ceph-mon[41562]: purged_snaps scrub ok 2024-09-10T19:14:56.181 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:14:55 smithi183 ceph-mon[41562]: purged_snaps scrub starts 2024-09-10T19:14:56.181 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:14:55 smithi183 ceph-mon[41562]: purged_snaps scrub ok 2024-09-10T19:14:56.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:14:55 smithi183 ceph-mon[41562]: from='client.? 172.21.15.183:0/1095435823' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-10T19:14:56.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:14:55 smithi191 ceph-mon[32884]: purged_snaps scrub starts 2024-09-10T19:14:56.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:14:55 smithi191 ceph-mon[32884]: purged_snaps scrub ok 2024-09-10T19:14:56.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:14:55 smithi191 ceph-mon[32884]: purged_snaps scrub starts 2024-09-10T19:14:56.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:14:55 smithi191 ceph-mon[32884]: purged_snaps scrub ok 2024-09-10T19:14:56.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:14:55 smithi191 ceph-mon[32884]: from='client.? 172.21.15.183:0/1095435823' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-10T19:14:56.276 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:14:56.298 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:14:56.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:14:56.299 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:14:56.400 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:14:56.421 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:14:56.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:14:56.423 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:14:56.523 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:14:56.545 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:14:56.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:14:56.546 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:14:56.647 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:14:56.668 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:14:56.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:14:56.669 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:14:56.770 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:14:56.793 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:14:56.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:14:56.794 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:14:56.894 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:14:56.929 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:14:56.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:14:56.930 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:14:57.031 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:14:57.064 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:14:57.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:14:57.065 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:14:57.117 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:14:56 smithi191 ceph-mon[32884]: 4.18 scrub starts 2024-09-10T19:14:57.117 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:14:56 smithi191 ceph-mon[32884]: 4.18 scrub ok 2024-09-10T19:14:57.117 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:14:56 smithi191 ceph-mon[32884]: 2.11 deep-scrub starts 2024-09-10T19:14:57.117 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:14:56 smithi191 ceph-mon[32884]: pgmap v185: 148 pgs: 3 active+undersized+degraded+remapped+backfilling, 145 active+clean; 4.0 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.32k op/s; 742/195432 objects degraded (0.380%); 53 MiB/s, 848 objects/s recovering 2024-09-10T19:14:57.117 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:14:56 smithi191 ceph-mon[32884]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T19:14:57.117 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:14:56 smithi191 ceph-mon[32884]: from='client.? 172.21.15.183:0/1095435823' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-10T19:14:57.117 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:14:56 smithi191 ceph-mon[32884]: osdmap e130: 8 total, 7 up, 7 in 2024-09-10T19:14:57.117 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:14:56 smithi191 ceph-mon[32884]: osdmap e131: 8 total, 7 up, 7 in 2024-09-10T19:14:57.128 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:14:56 smithi183 ceph-mon[29736]: 4.18 scrub starts 2024-09-10T19:14:57.128 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:14:56 smithi183 ceph-mon[29736]: 4.18 scrub ok 2024-09-10T19:14:57.128 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:14:56 smithi183 ceph-mon[29736]: 2.11 deep-scrub starts 2024-09-10T19:14:57.128 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:14:56 smithi183 ceph-mon[29736]: pgmap v185: 148 pgs: 3 active+undersized+degraded+remapped+backfilling, 145 active+clean; 4.0 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.32k op/s; 742/195432 objects degraded (0.380%); 53 MiB/s, 848 objects/s recovering 2024-09-10T19:14:57.128 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:14:56 smithi183 ceph-mon[29736]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T19:14:57.129 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:14:56 smithi183 ceph-mon[29736]: from='client.? 172.21.15.183:0/1095435823' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-10T19:14:57.129 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:14:56 smithi183 ceph-mon[29736]: osdmap e130: 8 total, 7 up, 7 in 2024-09-10T19:14:57.129 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:14:56 smithi183 ceph-mon[29736]: osdmap e131: 8 total, 7 up, 7 in 2024-09-10T19:14:57.129 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:14:56 smithi183 ceph-mon[41562]: 4.18 scrub starts 2024-09-10T19:14:57.129 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:14:56 smithi183 ceph-mon[41562]: 4.18 scrub ok 2024-09-10T19:14:57.129 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:14:56 smithi183 ceph-mon[41562]: 2.11 deep-scrub starts 2024-09-10T19:14:57.129 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:14:56 smithi183 ceph-mon[41562]: pgmap v185: 148 pgs: 3 active+undersized+degraded+remapped+backfilling, 145 active+clean; 4.0 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.32k op/s; 742/195432 objects degraded (0.380%); 53 MiB/s, 848 objects/s recovering 2024-09-10T19:14:57.129 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:14:56 smithi183 ceph-mon[41562]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T19:14:57.129 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:14:56 smithi183 ceph-mon[41562]: from='client.? 172.21.15.183:0/1095435823' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-10T19:14:57.129 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:14:56 smithi183 ceph-mon[41562]: osdmap e130: 8 total, 7 up, 7 in 2024-09-10T19:14:57.129 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:14:56 smithi183 ceph-mon[41562]: osdmap e131: 8 total, 7 up, 7 in 2024-09-10T19:14:57.166 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:14:57.187 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:14:57.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:14:57.188 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:14:57.288 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:14:57.310 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:14:57.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:14:57.311 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:14:57.411 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:14:57.433 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:14:57.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:14:57.434 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:14:57.535 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:14:57.556 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:14:57.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:14:57.557 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:14:57.658 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:14:57.680 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:14:57.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:14:57.681 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:14:57.715 INFO:journalctl@ceph.mgr.x.smithi191.stdout:Sep 10 19:14:57 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-x[34967]: ::ffff:172.21.15.191 - - [10/Sep/2024:19:14:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T19:14:57.782 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:14:57.803 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:14:57.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:14:57.804 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:14:57.904 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:14:57.939 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:14:57.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:14:57.940 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:14:58.040 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:14:58.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:14:58.082 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:14:58.082 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:14:58.183 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:14:58.204 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:14:58.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:14:58.205 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:14:58.306 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:14:58.342 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:14:58.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:14:58.343 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:14:58.444 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:14:58.470 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:14:58.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:14:58.471 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:14:58.572 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:14:58.605 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:14:58.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:14:58.606 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:14:58.707 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:14:58.728 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:14:58.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:14:58.729 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:14:58.830 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:14:58.861 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:14:58.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:14:58.863 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:14:58.931 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Sep 10 19:14:58 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-y[29989]: 2024-09-10T19:14:58.671+0000 7f9fad6cd640 -1 mgr get_metadata_python Requested missing service osd.1 2024-09-10T19:14:58.931 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:14:58 smithi183 ceph-mon[29736]: purged_snaps scrub starts 2024-09-10T19:14:58.931 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:14:58 smithi183 ceph-mon[29736]: purged_snaps scrub ok 2024-09-10T19:14:58.931 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:14:58 smithi183 ceph-mon[29736]: pgmap v188: 148 pgs: 3 active+undersized+degraded+remapped+backfilling, 145 active+clean; 4.0 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 852 B/s rd, 0 B/s wr, 1.06k op/s; 742/195432 objects degraded (0.380%); 22 MiB/s, 351 objects/s recovering 2024-09-10T19:14:58.932 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:14:58 smithi183 ceph-mon[29736]: osdmap e132: 8 total, 7 up, 7 in 2024-09-10T19:14:58.932 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:14:58 smithi183 ceph-mon[29736]: from='mgr.14556 ' entity='mgr.y' 2024-09-10T19:14:58.932 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:14:58 smithi183 ceph-mon[29736]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:14:58.932 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:14:58 smithi183 ceph-mon[29736]: from='client.? 172.21.15.183:0/600542485' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "4.1", "id": [1, 0, 4, 2, 6, 7]}]: dispatch 2024-09-10T19:14:58.932 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:14:58 smithi183 ceph-mon[41562]: purged_snaps scrub starts 2024-09-10T19:14:58.932 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:14:58 smithi183 ceph-mon[41562]: purged_snaps scrub ok 2024-09-10T19:14:58.932 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:14:58 smithi183 ceph-mon[41562]: pgmap v188: 148 pgs: 3 active+undersized+degraded+remapped+backfilling, 145 active+clean; 4.0 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 852 B/s rd, 0 B/s wr, 1.06k op/s; 742/195432 objects degraded (0.380%); 22 MiB/s, 351 objects/s recovering 2024-09-10T19:14:58.932 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:14:58 smithi183 ceph-mon[41562]: osdmap e132: 8 total, 7 up, 7 in 2024-09-10T19:14:58.932 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:14:58 smithi183 ceph-mon[41562]: from='mgr.14556 ' entity='mgr.y' 2024-09-10T19:14:58.933 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:14:58 smithi183 ceph-mon[41562]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:14:58.933 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:14:58 smithi183 ceph-mon[41562]: from='client.? 172.21.15.183:0/600542485' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "4.1", "id": [1, 0, 4, 2, 6, 7]}]: dispatch 2024-09-10T19:14:58.963 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:14:59.002 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:14:59.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:14:59.003 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:14:59.074 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-10T19:14:59.081 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:14:58 smithi191 ceph-mon[32884]: purged_snaps scrub starts 2024-09-10T19:14:59.081 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:14:58 smithi191 ceph-mon[32884]: purged_snaps scrub ok 2024-09-10T19:14:59.081 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:14:58 smithi191 ceph-mon[32884]: pgmap v188: 148 pgs: 3 active+undersized+degraded+remapped+backfilling, 145 active+clean; 4.0 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 852 B/s rd, 0 B/s wr, 1.06k op/s; 742/195432 objects degraded (0.380%); 22 MiB/s, 351 objects/s recovering 2024-09-10T19:14:59.081 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:14:58 smithi191 ceph-mon[32884]: osdmap e132: 8 total, 7 up, 7 in 2024-09-10T19:14:59.081 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:14:58 smithi191 ceph-mon[32884]: from='mgr.14556 ' entity='mgr.y' 2024-09-10T19:14:59.082 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:14:58 smithi191 ceph-mon[32884]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:14:59.082 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:14:58 smithi191 ceph-mon[32884]: from='client.? 172.21.15.183:0/600542485' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "4.1", "id": [1, 0, 4, 2, 6, 7]}]: dispatch 2024-09-10T19:14:59.104 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:14:59.133 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:14:59.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:14:59.134 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:14:59.235 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:14:59.267 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:14:59.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:14:59.268 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:14:59.369 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:14:59.391 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:14:59.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:14:59.393 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:14:59.465 INFO:journalctl@ceph.iscsi.iscsi.a.smithi191.stdout:Sep 10 19:14:59 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-iscsi-iscsi-a[61816]: debug there is no tcmu-runner data available 2024-09-10T19:14:59.493 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:14:59.520 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:14:59.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:14:59.522 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:14:59.622 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:14:59.669 INFO:teuthology.orchestra.run.smithi183.stderr:set 4.1 pg_upmap_items mapping to [1->0,4->2,6->7] 2024-09-10T19:14:59.670 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:14:59.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:14:59.672 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:14:59.772 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:14:59.802 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:14:59.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:14:59.806 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:14:59.907 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:14:59.932 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:14:59.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:14:59.933 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:15:00.034 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:15:00.065 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:15:00.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:15:00.066 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:15:00.167 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:15:00.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:14:59 smithi183 ceph-mon[29736]: from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:15:00.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:14:59 smithi183 ceph-mon[29736]: from='client.? 172.21.15.183:0/600542485' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "4.1", "id": [1, 0, 4, 2, 6, 7]}]': finished 2024-09-10T19:15:00.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:14:59 smithi183 ceph-mon[29736]: osdmap e133: 8 total, 7 up, 7 in 2024-09-10T19:15:00.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:14:59 smithi183 ceph-mon[29736]: Health check update: Degraded data redundancy: 742/195432 objects degraded (0.380%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-10T19:15:00.181 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:14:59 smithi183 ceph-mon[41562]: from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:15:00.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:14:59 smithi183 ceph-mon[41562]: from='client.? 172.21.15.183:0/600542485' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "4.1", "id": [1, 0, 4, 2, 6, 7]}]': finished 2024-09-10T19:15:00.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:14:59 smithi183 ceph-mon[41562]: osdmap e133: 8 total, 7 up, 7 in 2024-09-10T19:15:00.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:14:59 smithi183 ceph-mon[41562]: Health check update: Degraded data redundancy: 742/195432 objects degraded (0.380%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-10T19:15:00.194 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:15:00.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:15:00.195 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:15:00.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:14:59 smithi191 ceph-mon[32884]: from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:15:00.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:14:59 smithi191 ceph-mon[32884]: from='client.? 172.21.15.183:0/600542485' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "4.1", "id": [1, 0, 4, 2, 6, 7]}]': finished 2024-09-10T19:15:00.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:14:59 smithi191 ceph-mon[32884]: osdmap e133: 8 total, 7 up, 7 in 2024-09-10T19:15:00.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:14:59 smithi191 ceph-mon[32884]: Health check update: Degraded data redundancy: 742/195432 objects degraded (0.380%), 3 pgs degraded, 3 pgs undersized (PG_DEGRADED) 2024-09-10T19:15:00.296 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:15:00.328 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:15:00.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:15:00.329 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:15:00.429 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:15:00.470 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:15:00.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:15:00.471 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:15:00.572 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:15:00.603 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:15:00.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:15:00.604 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:15:00.705 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:15:00.740 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:15:00.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:15:00.741 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:15:00.842 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:15:00.871 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:15:00.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:15:00.872 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:15:00.973 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:15:01.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:15:01.021 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:15:01.021 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:15:01.122 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:15:01.133 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:15:00 smithi183 ceph-mon[41562]: pgmap v191: 148 pgs: 1 active+undersized+degraded+remapped+backfilling, 147 active+clean; 3.9 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 0 B/s wr, 451 op/s; 27/190008 objects degraded (0.014%); 11 MiB/s, 173 objects/s recovering 2024-09-10T19:15:01.133 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:15:00 smithi183 ceph-mon[41562]: purged_snaps scrub starts 2024-09-10T19:15:01.133 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:15:00 smithi183 ceph-mon[41562]: purged_snaps scrub ok 2024-09-10T19:15:01.133 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:15:00 smithi183 ceph-mon[41562]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T19:15:01.133 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:15:00 smithi183 ceph-mon[41562]: osdmap e134: 8 total, 7 up, 7 in 2024-09-10T19:15:01.133 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:15:00 smithi183 ceph-mon[29736]: pgmap v191: 148 pgs: 1 active+undersized+degraded+remapped+backfilling, 147 active+clean; 3.9 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 0 B/s wr, 451 op/s; 27/190008 objects degraded (0.014%); 11 MiB/s, 173 objects/s recovering 2024-09-10T19:15:01.133 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:15:00 smithi183 ceph-mon[29736]: purged_snaps scrub starts 2024-09-10T19:15:01.134 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:15:00 smithi183 ceph-mon[29736]: purged_snaps scrub ok 2024-09-10T19:15:01.134 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:15:00 smithi183 ceph-mon[29736]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T19:15:01.134 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:15:00 smithi183 ceph-mon[29736]: osdmap e134: 8 total, 7 up, 7 in 2024-09-10T19:15:01.194 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:15:01.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:15:01.200 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:15:01.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:15:00 smithi191 ceph-mon[32884]: pgmap v191: 148 pgs: 1 active+undersized+degraded+remapped+backfilling, 147 active+clean; 3.9 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 0 B/s wr, 451 op/s; 27/190008 objects degraded (0.014%); 11 MiB/s, 173 objects/s recovering 2024-09-10T19:15:01.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:15:00 smithi191 ceph-mon[32884]: purged_snaps scrub starts 2024-09-10T19:15:01.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:15:00 smithi191 ceph-mon[32884]: purged_snaps scrub ok 2024-09-10T19:15:01.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:15:00 smithi191 ceph-mon[32884]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T19:15:01.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:15:00 smithi191 ceph-mon[32884]: osdmap e134: 8 total, 7 up, 7 in 2024-09-10T19:15:01.300 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:15:01.330 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:15:01.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:15:01.331 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:15:01.431 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Sep 10 19:15:01 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-y[29989]: ::ffff:172.21.15.191 - - [10/Sep/2024:19:15:01] "GET /metrics HTTP/1.1" 200 209208 "" "Prometheus/2.43.0" 2024-09-10T19:15:01.431 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:15:01.446 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:15:01.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:15:01.498 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:15:01.498 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:15:01.599 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:15:01.628 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:15:01.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:15:01.629 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:15:01.730 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:15:01.765 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:15:01.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:15:01.766 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:15:01.867 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:15:01.902 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:15:01.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:15:01.904 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:15:02.004 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:15:02.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:15:02.036 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:15:02.036 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:15:02.137 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:15:02.172 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:15:02.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:15:02.173 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:15:02.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:15:01 smithi183 ceph-mon[29736]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T19:15:02.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:15:01 smithi183 ceph-mon[29736]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T19:15:02.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:15:01 smithi183 ceph-mon[29736]: from='mgr.14556 ' entity='mgr.y' 2024-09-10T19:15:02.181 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:15:01 smithi183 ceph-mon[41562]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T19:15:02.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:15:01 smithi183 ceph-mon[41562]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T19:15:02.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:15:01 smithi183 ceph-mon[41562]: from='mgr.14556 ' entity='mgr.y' 2024-09-10T19:15:02.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:15:01 smithi191 ceph-mon[32884]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T19:15:02.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:15:01 smithi191 ceph-mon[32884]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T19:15:02.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:15:01 smithi191 ceph-mon[32884]: from='mgr.14556 ' entity='mgr.y' 2024-09-10T19:15:02.274 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:15:02.312 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:15:02.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:15:02.313 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:15:02.414 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:15:02.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:15:02.456 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:15:02.457 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:15:02.557 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:15:02.586 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:15:02.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:15:02.587 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:15:02.688 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:15:02.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:15:02.718 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:15:02.718 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:15:02.819 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:15:02.852 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:15:02.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:15:02.853 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:15:02.954 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:15:02.989 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:15:02.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:15:02.990 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:15:03.090 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:15:03.113 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:15:02 smithi183 ceph-mon[29736]: pgmap v193: 148 pgs: 1 peering, 147 active+clean; 3.7 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 0 B/s wr, 858 op/s; 9.2 MiB/s, 147 objects/s recovering 2024-09-10T19:15:03.113 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:15:02 smithi183 ceph-mon[29736]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 27/190008 objects degraded (0.014%), 1 pg degraded, 1 pg undersized) 2024-09-10T19:15:03.113 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:15:02 smithi183 ceph-mon[41562]: pgmap v193: 148 pgs: 1 peering, 147 active+clean; 3.7 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 0 B/s wr, 858 op/s; 9.2 MiB/s, 147 objects/s recovering 2024-09-10T19:15:03.113 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:15:02 smithi183 ceph-mon[41562]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 27/190008 objects degraded (0.014%), 1 pg degraded, 1 pg undersized) 2024-09-10T19:15:03.131 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:15:03.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:15:03.132 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:15:03.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:15:02 smithi191 ceph-mon[32884]: pgmap v193: 148 pgs: 1 peering, 147 active+clean; 3.7 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 0 B/s wr, 858 op/s; 9.2 MiB/s, 147 objects/s recovering 2024-09-10T19:15:03.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:15:02 smithi191 ceph-mon[32884]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 27/190008 objects degraded (0.014%), 1 pg degraded, 1 pg undersized) 2024-09-10T19:15:03.233 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:15:03.270 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:15:03.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:15:03.271 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:15:03.372 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:15:03.405 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:15:03.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:15:03.407 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:15:03.507 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:15:03.540 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:15:03.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:15:03.541 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:15:03.642 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:15:03.671 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:15:03.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:15:03.672 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:15:03.773 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:15:03.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:15:03.806 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:15:03.806 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:15:03.907 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:15:03.941 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:15:03.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:15:03.942 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:15:04.043 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:15:04.074 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:15:04.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:15:04.075 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:15:04.176 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:15:04.211 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:15:04.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:15:04.213 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:15:04.314 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:15:04.350 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:15:04.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:15:04.351 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:15:04.452 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:15:04.485 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:15:04.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:15:04.486 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:15:04.587 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:15:04.619 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:15:04.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:15:04.620 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:15:04.720 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:15:04.757 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:15:04.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:15:04.758 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:15:04.859 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:15:04.905 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:15:04.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:15:04.906 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:15:05.006 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:15:05.034 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:15:05.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:15:05.035 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:15:05.046 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:15:04 smithi183 ceph-mon[29736]: purged_snaps scrub starts 2024-09-10T19:15:05.046 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:15:04 smithi183 ceph-mon[29736]: purged_snaps scrub ok 2024-09-10T19:15:05.047 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:15:04 smithi183 ceph-mon[29736]: pgmap v194: 148 pgs: 1 peering, 147 active+clean; 3.7 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 0 B/s wr, 695 op/s; 7.5 MiB/s, 119 objects/s recovering 2024-09-10T19:15:05.047 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:15:04 smithi183 ceph-mon[29736]: osdmap e135: 8 total, 7 up, 7 in 2024-09-10T19:15:05.047 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:15:04 smithi183 ceph-mon[41562]: purged_snaps scrub starts 2024-09-10T19:15:05.047 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:15:04 smithi183 ceph-mon[41562]: purged_snaps scrub ok 2024-09-10T19:15:05.047 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:15:04 smithi183 ceph-mon[41562]: pgmap v194: 148 pgs: 1 peering, 147 active+clean; 3.7 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 0 B/s wr, 695 op/s; 7.5 MiB/s, 119 objects/s recovering 2024-09-10T19:15:05.047 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:15:04 smithi183 ceph-mon[41562]: osdmap e135: 8 total, 7 up, 7 in 2024-09-10T19:15:05.047 INFO:teuthology.orchestra.run.smithi183.stdout:{ 2024-09-10T19:15:05.048 INFO:teuthology.orchestra.run.smithi183.stdout: "ops": [], 2024-09-10T19:15:05.048 INFO:teuthology.orchestra.run.smithi183.stdout: "complaint_time": 30, 2024-09-10T19:15:05.048 INFO:teuthology.orchestra.run.smithi183.stdout: "num_blocked_ops": 0 2024-09-10T19:15:05.048 INFO:teuthology.orchestra.run.smithi183.stdout:} 2024-09-10T19:15:05.136 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:15:05.172 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:15:05.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:15:05.173 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:15:05.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:15:04 smithi191 ceph-mon[32884]: purged_snaps scrub starts 2024-09-10T19:15:05.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:15:04 smithi191 ceph-mon[32884]: purged_snaps scrub ok 2024-09-10T19:15:05.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:15:04 smithi191 ceph-mon[32884]: pgmap v194: 148 pgs: 1 peering, 147 active+clean; 3.7 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 0 B/s wr, 695 op/s; 7.5 MiB/s, 119 objects/s recovering 2024-09-10T19:15:05.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:15:04 smithi191 ceph-mon[32884]: osdmap e135: 8 total, 7 up, 7 in 2024-09-10T19:15:05.274 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:15:05.311 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:15:05.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:15:05.312 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:15:05.413 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:15:05.441 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:15:05.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:15:05.442 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:15:05.543 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:15:05.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:15:05.577 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:15:05.578 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:15:05.678 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:15:05.710 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:15:05.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:15:05.711 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:15:05.812 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:15:05.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:15:05.845 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:15:05.846 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:15:05.946 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:15:05.979 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:15:05.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:15:05.980 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:15:06.081 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:15:06.119 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:15:06.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:15:06.120 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:15:06.221 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:15:06.251 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:15:06.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:15:06.252 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:15:06.353 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:15:06.389 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:15:06.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:15:06.390 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:15:06.490 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:15:06.521 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:15:06.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:15:06.522 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:15:06.623 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:15:06.637 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph daemon osd.0 dump_historic_ops 2024-09-10T19:15:06.651 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:15:06.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:15:06.652 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:15:06.678 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:15:06.752 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:15:06.814 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:15:06.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:15:06.815 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:15:06.916 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:15:06.942 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:15:06.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:15:06.943 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:15:06.946 INFO:tasks.thrashosds.thrasher:in_osds: [0, 2, 3, 4, 5, 6, 7] out_osds: [1] dead_osds: [1] live_osds: [0, 2, 3, 4, 5, 6, 7] 2024-09-10T19:15:06.946 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-10T19:15:06.946 INFO:tasks.thrashosds.thrasher:Reviving osd 1 2024-09-10T19:15:06.946 INFO:tasks.cephadm.osd.1:Restarting osd.1 (starting--it wasn't running)... 2024-09-10T19:15:06.947 DEBUG:teuthology.orchestra.run.smithi183:osd.1> sudo journalctl -f -n 0 -u ceph-320f6df8-6fa7-11ef-bcea-c7b262605968@osd.1.service 2024-09-10T19:15:06.952 DEBUG:teuthology.orchestra.run.smithi183:> sudo systemctl start ceph-320f6df8-6fa7-11ef-bcea-c7b262605968@osd.1 2024-09-10T19:15:07.043 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:15:07.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:15:07.087 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:15:07.087 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:15:07.176 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:15:06 smithi183 ceph-mon[29736]: pgmap v196: 148 pgs: 148 active+clean; 3.3 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.79k op/s; 7.4 MiB/s, 120 objects/s recovering 2024-09-10T19:15:07.177 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:15:06 smithi183 ceph-mon[41562]: pgmap v196: 148 pgs: 148 active+clean; 3.3 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.79k op/s; 7.4 MiB/s, 120 objects/s recovering 2024-09-10T19:15:07.177 INFO:journalctl@ceph.osd.1.smithi183.stdout:Sep 10 19:15:07 smithi183 systemd[1]: Starting Ceph osd.1 for 320f6df8-6fa7-11ef-bcea-c7b262605968... 2024-09-10T19:15:07.189 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:15:07.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:15:06 smithi191 ceph-mon[32884]: pgmap v196: 148 pgs: 148 active+clean; 3.3 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.79k op/s; 7.4 MiB/s, 120 objects/s recovering 2024-09-10T19:15:07.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:15:07.249 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:15:07.249 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:15:07.350 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:15:07.390 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:15:07.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:15:07.391 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:15:07.492 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:15:07.535 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:15:07.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:15:07.536 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:15:07.637 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:15:07.675 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:15:07.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:15:07.676 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:15:07.715 INFO:journalctl@ceph.mgr.x.smithi191.stdout:Sep 10 19:15:07 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-x[34967]: ::ffff:172.21.15.191 - - [10/Sep/2024:19:15:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T19:15:07.777 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:15:07.808 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:15:07.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:15:07.809 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:15:07.910 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:15:07.956 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:15:07.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:15:07.957 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:15:08.058 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:15:08.090 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:15:08.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:15:08.091 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:15:08.192 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:15:08.222 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:15:08.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:15:08.224 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:15:08.251 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:15:08.324 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:15:08.364 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:15:08.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:15:08.365 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:15:08.466 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:15:08.493 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:15:08.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:15:08.495 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:15:08.595 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:15:08.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:15:08.626 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:15:08.626 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:15:08.643 INFO:journalctl@ceph.osd.1.smithi183.stdout:Sep 10 19:15:08 smithi183 podman[104166]: 2024-09-10 19:15:08.451689866 +0000 UTC m=+0.181383010 image pull 4c55dd4096115e48502f368cff1eab06769f71f030e1692e2611bbc6c6f637e9 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3bfeaf4417585c821cab9ca76e4d95f390b0e769807122a45463640820b5cd16 2024-09-10T19:15:08.727 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:15:08.767 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:15:08.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:15:08.768 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:15:08.868 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:15:08.899 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:15:08 smithi183 ceph-mon[29736]: pgmap v197: 148 pgs: 148 active+clean; 3.3 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.12k op/s; 208 KiB/s, 4 objects/s recovering 2024-09-10T19:15:08.899 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:15:08 smithi183 ceph-mon[29736]: from='client.? 172.21.15.183:0/2743464095' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T19:15:08.901 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:15:08 smithi183 ceph-mon[41562]: pgmap v197: 148 pgs: 148 active+clean; 3.3 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.12k op/s; 208 KiB/s, 4 objects/s recovering 2024-09-10T19:15:08.901 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:15:08 smithi183 ceph-mon[41562]: from='client.? 172.21.15.183:0/2743464095' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T19:15:08.901 INFO:journalctl@ceph.osd.1.smithi183.stdout:Sep 10 19:15:08 smithi183 podman[104166]: 2024-09-10 19:15:08.790816906 +0000 UTC m=+0.520510042 container create 5ead9e780297f79e907350d52a332fdcd2b8e4cc8d7948a408c43a5b7fb381c2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3bfeaf4417585c821cab9ca76e4d95f390b0e769807122a45463640820b5cd16, name=ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd-1-activate, ceph=True, RELEASE=wip-guits-quincy-2024-09-09-1810-quincy-5be5739, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731) 2024-09-10T19:15:08.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:15:08.902 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:15:08.902 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:15:09.003 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:15:09.033 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:15:09.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:15:09.034 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:15:09.135 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:15:09.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:15:09.171 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:15:09.171 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:15:09.215 INFO:journalctl@ceph.iscsi.iscsi.a.smithi191.stdout:Sep 10 19:15:09 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-iscsi-iscsi-a[61816]: debug there is no tcmu-runner data available 2024-09-10T19:15:09.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:15:08 smithi191 ceph-mon[32884]: pgmap v197: 148 pgs: 148 active+clean; 3.3 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.12k op/s; 208 KiB/s, 4 objects/s recovering 2024-09-10T19:15:09.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:15:08 smithi191 ceph-mon[32884]: from='client.? 172.21.15.183:0/2743464095' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T19:15:09.271 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:15:09.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:15:09.305 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:15:09.305 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:15:09.406 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:15:09.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:15:09.438 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:15:09.438 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:15:09.539 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:15:09.569 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:15:09.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:15:09.570 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:15:09.612 INFO:teuthology.orchestra.run.smithi183.stdout:osd.0: {} 2024-09-10T19:15:09.612 INFO:teuthology.orchestra.run.smithi183.stdout:osd.2: {} 2024-09-10T19:15:09.613 INFO:teuthology.orchestra.run.smithi183.stdout:osd.3: {} 2024-09-10T19:15:09.613 INFO:teuthology.orchestra.run.smithi183.stdout:osd.4: {} 2024-09-10T19:15:09.613 INFO:teuthology.orchestra.run.smithi183.stdout:osd.5: {} 2024-09-10T19:15:09.613 INFO:teuthology.orchestra.run.smithi183.stdout:osd.6: {} 2024-09-10T19:15:09.613 INFO:teuthology.orchestra.run.smithi183.stdout:osd.7: {} 2024-09-10T19:15:09.613 INFO:teuthology.orchestra.run.smithi183.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-10T19:15:09.613 INFO:teuthology.orchestra.run.smithi183.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-09-10T19:15:09.613 INFO:teuthology.orchestra.run.smithi183.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-10T19:15:09.613 INFO:teuthology.orchestra.run.smithi183.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-10T19:15:09.613 INFO:teuthology.orchestra.run.smithi183.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-10T19:15:09.613 INFO:teuthology.orchestra.run.smithi183.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-10T19:15:09.614 INFO:teuthology.orchestra.run.smithi183.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-10T19:15:09.614 INFO:teuthology.orchestra.run.smithi183.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-10T19:15:09.671 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:15:09.703 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:15:09.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:15:09.704 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:15:09.805 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:15:09.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:15:09.834 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:15:09.834 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:15:09.935 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:15:09.970 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:15:09.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:15:09.971 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:15:10.072 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:15:10.099 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:15:10.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:15:10.100 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:15:10.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:15:09 smithi183 ceph-mon[29736]: from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:15:10.181 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:15:09 smithi183 ceph-mon[41562]: from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:15:10.182 INFO:journalctl@ceph.osd.1.smithi183.stdout:Sep 10 19:15:09 smithi183 podman[104166]: 2024-09-10 19:15:09.989657402 +0000 UTC m=+1.719350539 container init 5ead9e780297f79e907350d52a332fdcd2b8e4cc8d7948a408c43a5b7fb381c2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3bfeaf4417585c821cab9ca76e4d95f390b0e769807122a45463640820b5cd16, name=ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd-1-activate, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, RELEASE=wip-guits-quincy-2024-09-09-1810-quincy-5be5739, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, maintainer=Guillaume Abrioux ) 2024-09-10T19:15:10.182 INFO:journalctl@ceph.osd.1.smithi183.stdout:Sep 10 19:15:09 smithi183 podman[104166]: 2024-09-10 19:15:09.99929007 +0000 UTC m=+1.728983211 container start 5ead9e780297f79e907350d52a332fdcd2b8e4cc8d7948a408c43a5b7fb381c2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3bfeaf4417585c821cab9ca76e4d95f390b0e769807122a45463640820b5cd16, name=ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd-1-activate, RELEASE=wip-guits-quincy-2024-09-09-1810-quincy-5be5739, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, ceph=True, io.buildah.version=1.37.2, org.label-schema.license=GPLv2) 2024-09-10T19:15:10.182 INFO:journalctl@ceph.osd.1.smithi183.stdout:Sep 10 19:15:10 smithi183 podman[104166]: 2024-09-10 19:15:10.09786115 +0000 UTC m=+1.827554283 container attach 5ead9e780297f79e907350d52a332fdcd2b8e4cc8d7948a408c43a5b7fb381c2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3bfeaf4417585c821cab9ca76e4d95f390b0e769807122a45463640820b5cd16, name=ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd-1-activate, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-guits-quincy-2024-09-09-1810-quincy-5be5739, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD) 2024-09-10T19:15:10.201 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:15:10.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:15:09 smithi191 ceph-mon[32884]: from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:15:10.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:15:10.233 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:15:10.233 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:15:10.334 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:15:10.360 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:15:10.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:15:10.361 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:15:10.462 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:15:10.490 INFO:journalctl@ceph.osd.1.smithi183.stdout:Sep 10 19:15:10 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd-1-activate[104330]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-09-10T19:15:10.491 INFO:journalctl@ceph.osd.1.smithi183.stdout:Sep 10 19:15:10 smithi183 bash[104166]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-09-10T19:15:10.491 INFO:journalctl@ceph.osd.1.smithi183.stdout:Sep 10 19:15:10 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd-1-activate[104330]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-1 --no-mon-config --dev /dev/mapper/vg_nvme-lv_3 2024-09-10T19:15:10.491 INFO:journalctl@ceph.osd.1.smithi183.stdout:Sep 10 19:15:10 smithi183 bash[104166]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-1 --no-mon-config --dev /dev/mapper/vg_nvme-lv_3 2024-09-10T19:15:10.491 INFO:journalctl@ceph.osd.1.smithi183.stdout:Sep 10 19:15:10 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd-1-activate[104330]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_3 2024-09-10T19:15:10.491 INFO:journalctl@ceph.osd.1.smithi183.stdout:Sep 10 19:15:10 smithi183 bash[104166]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_3 2024-09-10T19:15:10.491 INFO:journalctl@ceph.osd.1.smithi183.stdout:Sep 10 19:15:10 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd-1-activate[104330]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-2 2024-09-10T19:15:10.491 INFO:journalctl@ceph.osd.1.smithi183.stdout:Sep 10 19:15:10 smithi183 bash[104166]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-2 2024-09-10T19:15:10.491 INFO:journalctl@ceph.osd.1.smithi183.stdout:Sep 10 19:15:10 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd-1-activate[104330]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-1/block 2024-09-10T19:15:10.491 INFO:journalctl@ceph.osd.1.smithi183.stdout:Sep 10 19:15:10 smithi183 bash[104166]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-1/block 2024-09-10T19:15:10.491 INFO:journalctl@ceph.osd.1.smithi183.stdout:Sep 10 19:15:10 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd-1-activate[104330]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-09-10T19:15:10.491 INFO:journalctl@ceph.osd.1.smithi183.stdout:Sep 10 19:15:10 smithi183 bash[104166]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-09-10T19:15:10.491 INFO:journalctl@ceph.osd.1.smithi183.stdout:Sep 10 19:15:10 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd-1-activate[104330]: --> ceph-volume raw activate successful for osd ID: 1 2024-09-10T19:15:10.492 INFO:journalctl@ceph.osd.1.smithi183.stdout:Sep 10 19:15:10 smithi183 bash[104166]: --> ceph-volume raw activate successful for osd ID: 1 2024-09-10T19:15:10.492 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:15:10.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:15:10.493 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:15:10.594 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:15:10.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:15:10.629 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:15:10.629 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:15:10.730 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:15:10.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:15:10.763 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:15:10.763 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:15:10.864 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:15:10.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:15:10.897 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:15:10.897 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:15:10.998 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:15:11.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:15:11.031 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:15:11.031 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:15:11.132 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:15:11.172 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:15:11.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:15:11.173 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:15:11.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:15:10 smithi183 ceph-mon[29736]: pgmap v198: 148 pgs: 148 active+clean; 3.1 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 2.1 KiB/s rd, 0 B/s wr, 1.04k op/s; 0 B/s, 0 objects/s recovering 2024-09-10T19:15:11.181 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:15:10 smithi183 ceph-mon[41562]: pgmap v198: 148 pgs: 148 active+clean; 3.1 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 2.1 KiB/s rd, 0 B/s wr, 1.04k op/s; 0 B/s, 0 objects/s recovering 2024-09-10T19:15:11.181 INFO:journalctl@ceph.osd.1.smithi183.stdout:Sep 10 19:15:10 smithi183 podman[104166]: 2024-09-10 19:15:10.852786813 +0000 UTC m=+2.582479952 container died 5ead9e780297f79e907350d52a332fdcd2b8e4cc8d7948a408c43a5b7fb381c2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3bfeaf4417585c821cab9ca76e4d95f390b0e769807122a45463640820b5cd16, name=ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd-1-activate, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, RELEASE=wip-guits-quincy-2024-09-09-1810-quincy-5be5739) 2024-09-10T19:15:11.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:15:10 smithi191 ceph-mon[32884]: pgmap v198: 148 pgs: 148 active+clean; 3.1 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 2.1 KiB/s rd, 0 B/s wr, 1.04k op/s; 0 B/s, 0 objects/s recovering 2024-09-10T19:15:11.275 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:15:11.311 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:15:11.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:15:11.312 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:15:11.413 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:15:11.450 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:15:11.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:15:11.451 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:15:11.543 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Sep 10 19:15:11 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-y[29989]: ::ffff:172.21.15.191 - - [10/Sep/2024:19:15:11] "GET /metrics HTTP/1.1" 200 209208 "" "Prometheus/2.43.0" 2024-09-10T19:15:11.552 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:15:11.589 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:15:11.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:15:11.590 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:15:11.691 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:15:11.736 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:15:11.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:15:11.737 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:15:11.838 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:15:11.872 INFO:journalctl@ceph.osd.1.smithi183.stdout:Sep 10 19:15:11 smithi183 podman[104494]: 2024-09-10 19:15:11.542410937 +0000 UTC m=+1.015103791 container remove 5ead9e780297f79e907350d52a332fdcd2b8e4cc8d7948a408c43a5b7fb381c2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3bfeaf4417585c821cab9ca76e4d95f390b0e769807122a45463640820b5cd16, name=ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd-1-activate, ceph=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.license=GPLv2, RELEASE=wip-guits-quincy-2024-09-09-1810-quincy-5be5739, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-10T19:15:11.874 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:15:11.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:15:11.875 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:15:11.916 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-10T19:15:11.916 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-10T19:15:11.976 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:15:12.011 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:15:12.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:15:12.012 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:15:12.113 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:15:12.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:15:12.143 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:15:12.144 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:15:12.244 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:15:12.287 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:15:12.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:15:12.288 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:15:12.389 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:15:12.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:15:12.420 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:15:12.420 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:15:12.521 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:15:12.550 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:15:12.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:15:12.551 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:15:12.652 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:15:12.682 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:15:12.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:15:12.683 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:15:12.784 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:15:12.839 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:15:12.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:15:12.840 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:15:12.941 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:15:12.985 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:15:12.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:15:12.986 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:15:13.062 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:15:12 smithi183 ceph-mon[29736]: pgmap v199: 148 pgs: 148 active+clean; 2.9 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 2.4 KiB/s rd, 0 B/s wr, 1.36k op/s; 0 B/s, 0 objects/s recovering 2024-09-10T19:15:13.062 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:15:12 smithi183 ceph-mon[41562]: pgmap v199: 148 pgs: 148 active+clean; 2.9 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 2.4 KiB/s rd, 0 B/s wr, 1.36k op/s; 0 B/s, 0 objects/s recovering 2024-09-10T19:15:13.087 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:15:13.133 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:15:13.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:15:13.135 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:15:13.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:15:12 smithi191 ceph-mon[32884]: pgmap v199: 148 pgs: 148 active+clean; 2.9 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 2.4 KiB/s rd, 0 B/s wr, 1.36k op/s; 0 B/s, 0 objects/s recovering 2024-09-10T19:15:13.236 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:15:13.265 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:15:13.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:15:13.266 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:15:13.367 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:15:13.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:15:13.407 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:15:13.408 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:15:13.408 INFO:teuthology.orchestra.run.smithi183.stdout:ERROR: (22) Invalid argument 2024-09-10T19:15:13.408 INFO:teuthology.orchestra.run.smithi183.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-10T19:15:13.509 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:15:13.540 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:15:13.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:15:13.541 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:15:13.642 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:15:13.675 INFO:journalctl@ceph.osd.1.smithi183.stdout:Sep 10 19:15:13 smithi183 podman[104817]: 2024-09-10 19:15:13.427953314 +0000 UTC m=+0.043735106 image pull 4c55dd4096115e48502f368cff1eab06769f71f030e1692e2611bbc6c6f637e9 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3bfeaf4417585c821cab9ca76e4d95f390b0e769807122a45463640820b5cd16 2024-09-10T19:15:13.675 INFO:journalctl@ceph.osd.1.smithi183.stdout:Sep 10 19:15:13 smithi183 podman[104817]: 2024-09-10 19:15:13.549950086 +0000 UTC m=+0.165731864 container create 01ff17cbf9a0a513ce111980a9e1f53d36e11077aaa39457aaeaeba3cb5b570a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3bfeaf4417585c821cab9ca76e4d95f390b0e769807122a45463640820b5cd16, name=ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd-1, RELEASE=wip-guits-quincy-2024-09-09-1810-quincy-5be5739, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, ceph=True, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-10T19:15:13.679 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:15:13.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:15:13.680 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:15:13.781 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:15:13.811 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:15:13.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:15:13.812 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:15:13.913 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:15:13.918 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-10T19:15:13.928 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Sep 10 19:15:13 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-y[29989]: 2024-09-10T19:15:13.672+0000 7f9fad6cd640 -1 mgr get_metadata_python Requested missing service osd.1 2024-09-10T19:15:13.928 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:15:13 smithi183 ceph-mon[41562]: from='client.? 172.21.15.183:0/1069718127' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-10T19:15:13.928 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:15:13 smithi183 ceph-mon[41562]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-10T19:15:13.929 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:15:13 smithi183 ceph-mon[41562]: from='mgr.14556 ' entity='mgr.y' 2024-09-10T19:15:13.929 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:15:13 smithi183 ceph-mon[41562]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:15:13.929 INFO:teuthology.orchestra.run.smithi183.stderr:nodeep-scrub is set 2024-09-10T19:15:13.939 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:15:13 smithi183 ceph-mon[29736]: from='client.? 172.21.15.183:0/1069718127' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-10T19:15:13.939 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:15:13 smithi183 ceph-mon[29736]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-10T19:15:13.939 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:15:13 smithi183 ceph-mon[29736]: from='mgr.14556 ' entity='mgr.y' 2024-09-10T19:15:13.939 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:15:13 smithi183 ceph-mon[29736]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:15:13.951 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:15:13.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:15:13.952 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:15:14.053 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:15:14.082 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:15:14.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:15:14.084 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:15:14.185 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:15:14.215 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:15:14.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:15:13 smithi191 ceph-mon[32884]: from='client.? 172.21.15.183:0/1069718127' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-10T19:15:14.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:15:13 smithi191 ceph-mon[32884]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-10T19:15:14.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:15:13 smithi191 ceph-mon[32884]: from='mgr.14556 ' entity='mgr.y' 2024-09-10T19:15:14.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:15:13 smithi191 ceph-mon[32884]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:15:14.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:15:14.217 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:15:14.318 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:15:14.351 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:15:14.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:15:14.352 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:15:14.407 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph daemon osd.1 dump_ops_in_flight 2024-09-10T19:15:14.453 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:15:14.479 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:15:14.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:15:14.480 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:15:14.562 INFO:journalctl@ceph.osd.1.smithi183.stdout:Sep 10 19:15:14 smithi183 podman[104817]: 2024-09-10 19:15:14.273525464 +0000 UTC m=+0.889307260 container init 01ff17cbf9a0a513ce111980a9e1f53d36e11077aaa39457aaeaeba3cb5b570a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3bfeaf4417585c821cab9ca76e4d95f390b0e769807122a45463640820b5cd16, name=ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd-1, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, RELEASE=wip-guits-quincy-2024-09-09-1810-quincy-5be5739, GIT_CLEAN=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, ceph=True) 2024-09-10T19:15:14.562 INFO:journalctl@ceph.osd.1.smithi183.stdout:Sep 10 19:15:14 smithi183 podman[104817]: 2024-09-10 19:15:14.281937394 +0000 UTC m=+0.897719175 container start 01ff17cbf9a0a513ce111980a9e1f53d36e11077aaa39457aaeaeba3cb5b570a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3bfeaf4417585c821cab9ca76e4d95f390b0e769807122a45463640820b5cd16, name=ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd-1, CEPH_POINT_RELEASE=, RELEASE=wip-guits-quincy-2024-09-09-1810-quincy-5be5739, io.buildah.version=1.37.2, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7) 2024-09-10T19:15:14.562 INFO:journalctl@ceph.osd.1.smithi183.stdout:Sep 10 19:15:14 smithi183 bash[104817]: 01ff17cbf9a0a513ce111980a9e1f53d36e11077aaa39457aaeaeba3cb5b570a 2024-09-10T19:15:14.562 INFO:journalctl@ceph.osd.1.smithi183.stdout:Sep 10 19:15:14 smithi183 systemd[1]: Started Ceph osd.1 for 320f6df8-6fa7-11ef-bcea-c7b262605968. 2024-09-10T19:15:14.581 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:15:14.608 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:15:14.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:15:14.609 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:15:14.710 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:15:14.751 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:15:14.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:15:14.753 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:15:14.854 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:15:14.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:15:14.891 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:15:14.891 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:15:14.901 INFO:journalctl@ceph.osd.1.smithi183.stdout:Sep 10 19:15:14 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd-1[104866]: 2024-09-10T19:15:14.586+0000 7f3b7fa54440 -1 Falling back to public interface 2024-09-10T19:15:14.993 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:15:15.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:15:15.050 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:15:15.050 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:15:15.151 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:15:15.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:15:14 smithi183 ceph-mon[29736]: pgmap v200: 148 pgs: 148 active+clean; 2.9 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 2.4 KiB/s rd, 0 B/s wr, 1.36k op/s; 0 B/s, 0 objects/s recovering 2024-09-10T19:15:15.182 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:15:14 smithi183 ceph-mon[29736]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T19:15:15.182 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:15:14 smithi183 ceph-mon[29736]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-10T19:15:15.182 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:15:14 smithi183 ceph-mon[29736]: osdmap e136: 8 total, 7 up, 7 in 2024-09-10T19:15:15.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:15:14 smithi183 ceph-mon[41562]: pgmap v200: 148 pgs: 148 active+clean; 2.9 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 2.4 KiB/s rd, 0 B/s wr, 1.36k op/s; 0 B/s, 0 objects/s recovering 2024-09-10T19:15:15.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:15:14 smithi183 ceph-mon[41562]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T19:15:15.183 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:15:14 smithi183 ceph-mon[41562]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-10T19:15:15.183 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:15:14 smithi183 ceph-mon[41562]: osdmap e136: 8 total, 7 up, 7 in 2024-09-10T19:15:15.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:15:15.204 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:15:15.205 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:15:15.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:15:14 smithi191 ceph-mon[32884]: pgmap v200: 148 pgs: 148 active+clean; 2.9 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 2.4 KiB/s rd, 0 B/s wr, 1.36k op/s; 0 B/s, 0 objects/s recovering 2024-09-10T19:15:15.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:15:14 smithi191 ceph-mon[32884]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T19:15:15.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:15:14 smithi191 ceph-mon[32884]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-10T19:15:15.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:15:14 smithi191 ceph-mon[32884]: osdmap e136: 8 total, 7 up, 7 in 2024-09-10T19:15:15.305 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:15:15.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:15:15.354 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:15:15.354 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:15:15.455 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:15:15.494 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:15:15.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:15:15.495 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:15:15.596 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:15:15.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:15:15.622 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:15:15.622 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:15:15.723 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:15:15.752 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:15:15.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:15:15.753 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:15:15.854 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:15:15.879 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:15:15.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:15:15.880 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:15:15.981 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:15:16.006 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:15:16.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:15:16.007 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:15:16.108 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:15:16.135 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:15:16.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:15:16.136 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:15:16.237 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:15:16.264 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:15:16.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:15:16.265 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:15:16.365 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:15:16.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:15:16.408 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:15:16.408 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:15:16.509 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:15:16.557 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:15:16.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:15:16.559 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:15:16.573 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph daemon osd.2 dump_ops_in_flight 2024-09-10T19:15:16.589 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:15:16.590 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:15:16.660 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:15:16.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:15:16.712 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:15:16.712 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:15:16.813 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:15:16.862 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:15:16.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:15:16.863 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:15:16.964 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:15:17.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:15:17.011 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:15:17.011 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:15:17.014 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph osd unset noscrub 2024-09-10T19:15:17.066 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:15:16 smithi183 ceph-mon[29736]: pgmap v202: 148 pgs: 148 active+clean; 2.5 GiB data, 9.6 GiB used, 616 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.37k op/s 2024-09-10T19:15:17.066 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:15:16 smithi183 ceph-mon[41562]: pgmap v202: 148 pgs: 148 active+clean; 2.5 GiB data, 9.6 GiB used, 616 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.37k op/s 2024-09-10T19:15:17.112 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:15:17.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:15:17.160 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:15:17.160 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:15:17.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:15:16 smithi191 ceph-mon[32884]: pgmap v202: 148 pgs: 148 active+clean; 2.5 GiB data, 9.6 GiB used, 616 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.37k op/s 2024-09-10T19:15:17.261 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:15:17.286 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:15:17.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:15:17.287 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:15:17.388 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:15:17.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:15:17.419 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:15:17.419 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:15:17.519 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:15:17.562 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:15:17.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:15:17.563 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:15:17.663 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:15:17.693 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:15:17.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:15:17.694 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:15:17.715 INFO:journalctl@ceph.mgr.x.smithi191.stdout:Sep 10 19:15:17 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-x[34967]: ::ffff:172.21.15.191 - - [10/Sep/2024:19:15:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T19:15:17.795 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:15:17.827 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:15:17.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:15:17.828 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:15:17.929 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:15:17.966 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:15:17.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:15:17.967 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:15:18.068 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:15:18.095 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:15:18.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:15:18.096 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:15:18.197 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:15:18.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:15:18.231 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:15:18.231 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:15:18.332 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:15:18.367 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:15:18.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:15:18.368 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:15:18.431 INFO:journalctl@ceph.osd.1.smithi183.stdout:Sep 10 19:15:18 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd-1[104866]: 2024-09-10T19:15:18.194+0000 7f3b7fa54440 -1 osd.1 98 log_to_monitors true 2024-09-10T19:15:18.469 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:15:18.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:15:18.498 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:15:18.498 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:15:18.599 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:15:18.624 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:15:18.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:15:18.625 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:15:18.726 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:15:18.753 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:15:18.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:15:18.754 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:15:18.854 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:15:18.883 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:15:18.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:15:18.884 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:15:18.925 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:15:18 smithi183 ceph-mon[29736]: pgmap v203: 148 pgs: 148 active+clean; 2.5 GiB data, 9.6 GiB used, 616 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.37k op/s 2024-09-10T19:15:18.925 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:15:18 smithi183 ceph-mon[29736]: from='osd.1 [v2:172.21.15.183:6810/219854798,v1:172.21.15.183:6811/219854798]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-09-10T19:15:18.985 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:15:19.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:15:19.017 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:15:19.017 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:15:19.118 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:15:19.122 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:15:19.144 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:15:19.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:15:19.145 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:15:19.164 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:15:19.181 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:15:18 smithi183 ceph-mon[41562]: pgmap v203: 148 pgs: 148 active+clean; 2.5 GiB data, 9.6 GiB used, 616 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.37k op/s 2024-09-10T19:15:19.181 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:15:18 smithi183 ceph-mon[41562]: from='osd.1 [v2:172.21.15.183:6810/219854798,v1:172.21.15.183:6811/219854798]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-09-10T19:15:19.181 INFO:journalctl@ceph.osd.1.smithi183.stdout:Sep 10 19:15:18 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd-1[104866]: 2024-09-10T19:15:18.954+0000 7f3b77800640 -1 osd.1 98 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-10T19:15:19.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:15:18 smithi191 ceph-mon[32884]: pgmap v203: 148 pgs: 148 active+clean; 2.5 GiB data, 9.6 GiB used, 616 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.37k op/s 2024-09-10T19:15:19.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:15:18 smithi191 ceph-mon[32884]: from='osd.1 [v2:172.21.15.183:6810/219854798,v1:172.21.15.183:6811/219854798]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-09-10T19:15:19.216 INFO:journalctl@ceph.iscsi.iscsi.a.smithi191.stdout:Sep 10 19:15:19 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-iscsi-iscsi-a[61816]: debug there is no tcmu-runner data available 2024-09-10T19:15:19.246 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:15:19.306 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:15:19.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:15:19.308 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:15:19.408 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:15:19.437 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:15:19.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:15:19.438 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:15:19.539 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:15:19.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:15:19.574 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:15:19.574 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:15:19.674 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:15:19.709 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:15:19.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:15:19.710 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:15:19.811 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:15:19.841 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:15:19.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:15:19.842 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:15:19.943 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:15:19.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:15:19.994 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:15:19.994 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:15:20.095 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:15:20.105 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:15:19 smithi183 ceph-mon[29736]: from='osd.1 [v2:172.21.15.183:6810/219854798,v1:172.21.15.183:6811/219854798]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-09-10T19:15:20.105 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:15:19 smithi183 ceph-mon[29736]: osdmap e137: 8 total, 7 up, 7 in 2024-09-10T19:15:20.106 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:15:19 smithi183 ceph-mon[29736]: from='osd.1 [v2:172.21.15.183:6810/219854798,v1:172.21.15.183:6811/219854798]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi183", "root=default"]}]: dispatch 2024-09-10T19:15:20.106 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:15:19 smithi183 ceph-mon[29736]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-10T19:15:20.106 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:15:19 smithi183 ceph-mon[29736]: from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:15:20.106 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:15:19 smithi183 ceph-mon[41562]: from='osd.1 [v2:172.21.15.183:6810/219854798,v1:172.21.15.183:6811/219854798]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-09-10T19:15:20.106 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:15:19 smithi183 ceph-mon[41562]: osdmap e137: 8 total, 7 up, 7 in 2024-09-10T19:15:20.106 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:15:19 smithi183 ceph-mon[41562]: from='osd.1 [v2:172.21.15.183:6810/219854798,v1:172.21.15.183:6811/219854798]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi183", "root=default"]}]: dispatch 2024-09-10T19:15:20.106 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:15:19 smithi183 ceph-mon[41562]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-10T19:15:20.106 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:15:19 smithi183 ceph-mon[41562]: from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:15:20.125 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:15:20.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:15:20.127 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:15:20.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:15:19 smithi191 ceph-mon[32884]: from='osd.1 [v2:172.21.15.183:6810/219854798,v1:172.21.15.183:6811/219854798]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-09-10T19:15:20.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:15:19 smithi191 ceph-mon[32884]: osdmap e137: 8 total, 7 up, 7 in 2024-09-10T19:15:20.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:15:19 smithi191 ceph-mon[32884]: from='osd.1 [v2:172.21.15.183:6810/219854798,v1:172.21.15.183:6811/219854798]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi183", "root=default"]}]: dispatch 2024-09-10T19:15:20.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:15:19 smithi191 ceph-mon[32884]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-10T19:15:20.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:15:19 smithi191 ceph-mon[32884]: from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:15:20.227 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:15:20.252 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:15:20.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:15:20.253 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:15:20.354 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:15:20.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:15:20.382 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:15:20.383 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:15:20.484 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:15:20.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:15:20.520 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:15:20.520 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:15:20.621 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:15:20.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:15:20.655 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:15:20.655 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:15:20.756 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:15:20.788 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:15:20.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:15:20.790 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:15:20.890 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:15:20.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:15:20.921 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:15:20.921 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:15:21.022 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:15:21.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:15:21.051 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:15:21.051 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:15:21.152 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:15:21.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:15:20 smithi183 ceph-mon[29736]: pgmap v205: 148 pgs: 148 active+clean; 2.4 GiB data, 9.4 GiB used, 616 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 989 op/s 2024-09-10T19:15:21.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:15:20 smithi183 ceph-mon[29736]: osd.1 [v2:172.21.15.183:6810/219854798,v1:172.21.15.183:6811/219854798] boot 2024-09-10T19:15:21.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:15:20 smithi183 ceph-mon[29736]: osdmap e138: 8 total, 8 up, 7 in 2024-09-10T19:15:21.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:15:20 smithi183 ceph-mon[29736]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-10T19:15:21.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:15:20 smithi183 ceph-mon[41562]: pgmap v205: 148 pgs: 148 active+clean; 2.4 GiB data, 9.4 GiB used, 616 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 989 op/s 2024-09-10T19:15:21.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:15:20 smithi183 ceph-mon[41562]: osd.1 [v2:172.21.15.183:6810/219854798,v1:172.21.15.183:6811/219854798] boot 2024-09-10T19:15:21.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:15:20 smithi183 ceph-mon[41562]: osdmap e138: 8 total, 8 up, 7 in 2024-09-10T19:15:21.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:15:20 smithi183 ceph-mon[41562]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-10T19:15:21.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:15:21.183 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:15:21.183 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:15:21.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:15:20 smithi191 ceph-mon[32884]: pgmap v205: 148 pgs: 148 active+clean; 2.4 GiB data, 9.4 GiB used, 616 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 989 op/s 2024-09-10T19:15:21.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:15:20 smithi191 ceph-mon[32884]: osd.1 [v2:172.21.15.183:6810/219854798,v1:172.21.15.183:6811/219854798] boot 2024-09-10T19:15:21.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:15:20 smithi191 ceph-mon[32884]: osdmap e138: 8 total, 8 up, 7 in 2024-09-10T19:15:21.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:15:20 smithi191 ceph-mon[32884]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-10T19:15:21.284 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:15:21.305 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:15:21.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:15:21.306 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:15:21.407 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:15:21.441 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:15:21.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:15:21.442 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:15:21.542 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:15:21.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:15:21.581 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:15:21.582 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:15:21.681 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Sep 10 19:15:21 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-y[29989]: ::ffff:172.21.15.191 - - [10/Sep/2024:19:15:21] "GET /metrics HTTP/1.1" 200 209193 "" "Prometheus/2.43.0" 2024-09-10T19:15:21.682 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:15:21.713 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:15:21.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:15:21.714 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:15:21.815 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:15:21.853 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:15:21.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:15:21.855 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:15:21.955 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:15:21.983 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:15:21.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:15:21.984 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:15:22.085 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:15:22.113 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:15:22.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:15:22.113 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:15:22.214 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:15:22.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:15:21 smithi191 ceph-mon[32884]: purged_snaps scrub starts 2024-09-10T19:15:22.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:15:21 smithi191 ceph-mon[32884]: purged_snaps scrub ok 2024-09-10T19:15:22.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:15:21 smithi191 ceph-mon[32884]: osdmap e139: 8 total, 8 up, 7 in 2024-09-10T19:15:22.246 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:15:22.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:15:22.248 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:15:22.348 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:15:22.377 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:15:22.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:15:22.378 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:15:22.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:15:21 smithi183 ceph-mon[29736]: purged_snaps scrub starts 2024-09-10T19:15:22.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:15:21 smithi183 ceph-mon[29736]: purged_snaps scrub ok 2024-09-10T19:15:22.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:15:21 smithi183 ceph-mon[29736]: osdmap e139: 8 total, 8 up, 7 in 2024-09-10T19:15:22.431 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:15:21 smithi183 ceph-mon[41562]: purged_snaps scrub starts 2024-09-10T19:15:22.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:15:21 smithi183 ceph-mon[41562]: purged_snaps scrub ok 2024-09-10T19:15:22.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:15:21 smithi183 ceph-mon[41562]: osdmap e139: 8 total, 8 up, 7 in 2024-09-10T19:15:22.479 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:15:22.513 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:15:22.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:15:22.514 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:15:22.614 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:15:22.652 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:15:22.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:15:22.653 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:15:22.753 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:15:22.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:15:22.786 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:15:22.787 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:15:22.887 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:15:22.924 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:15:22.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:15:22.925 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:15:23.026 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:15:23.054 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:15:23.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:15:23.055 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:15:23.156 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:15:23.178 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:15:23.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:15:23.179 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:15:23.212 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:15:22 smithi183 ceph-mon[29736]: pgmap v208: 148 pgs: 148 active+clean; 2.1 GiB data, 8.4 GiB used, 617 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 1.02k op/s 2024-09-10T19:15:23.212 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:15:22 smithi183 ceph-mon[41562]: pgmap v208: 148 pgs: 148 active+clean; 2.1 GiB data, 8.4 GiB used, 617 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 1.02k op/s 2024-09-10T19:15:23.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:15:22 smithi191 ceph-mon[32884]: pgmap v208: 148 pgs: 148 active+clean; 2.1 GiB data, 8.4 GiB used, 617 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 1.02k op/s 2024-09-10T19:15:23.279 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:15:23.306 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:15:23.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:15:23.308 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:15:23.408 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:15:23.434 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:15:23.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:15:23.435 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:15:23.536 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:15:23.574 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:15:23.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:15:23.575 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:15:23.676 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:15:23.706 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:15:23.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:15:23.707 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:15:23.748 INFO:teuthology.orchestra.run.smithi183.stdout:osd.0: {} 2024-09-10T19:15:23.748 INFO:teuthology.orchestra.run.smithi183.stdout:osd.1: {} 2024-09-10T19:15:23.748 INFO:teuthology.orchestra.run.smithi183.stdout:osd.2: {} 2024-09-10T19:15:23.748 INFO:teuthology.orchestra.run.smithi183.stdout:osd.3: {} 2024-09-10T19:15:23.749 INFO:teuthology.orchestra.run.smithi183.stdout:osd.4: {} 2024-09-10T19:15:23.749 INFO:teuthology.orchestra.run.smithi183.stdout:osd.5: {} 2024-09-10T19:15:23.749 INFO:teuthology.orchestra.run.smithi183.stdout:osd.6: {} 2024-09-10T19:15:23.749 INFO:teuthology.orchestra.run.smithi183.stdout:osd.7: {} 2024-09-10T19:15:23.749 INFO:teuthology.orchestra.run.smithi183.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-10T19:15:23.749 INFO:teuthology.orchestra.run.smithi183.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-10T19:15:23.749 INFO:teuthology.orchestra.run.smithi183.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-10T19:15:23.749 INFO:teuthology.orchestra.run.smithi183.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-10T19:15:23.749 INFO:teuthology.orchestra.run.smithi183.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-10T19:15:23.749 INFO:teuthology.orchestra.run.smithi183.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-10T19:15:23.749 INFO:teuthology.orchestra.run.smithi183.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-10T19:15:23.749 INFO:teuthology.orchestra.run.smithi183.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-10T19:15:23.808 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:15:23.845 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:15:23.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:15:23.846 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:15:23.946 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:15:23.981 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:15:23.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:15:23.982 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:15:24.067 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:15:23 smithi183 ceph-mon[29736]: from='client.? 172.21.15.183:0/863037894' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T19:15:24.067 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:15:23 smithi183 ceph-mon[41562]: from='client.? 172.21.15.183:0/863037894' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T19:15:24.083 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:15:24.115 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:15:24.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:15:24.116 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:15:24.168 INFO:teuthology.orchestra.run.smithi183.stdout:{ 2024-09-10T19:15:24.168 INFO:teuthology.orchestra.run.smithi183.stdout: "ops": [], 2024-09-10T19:15:24.168 INFO:teuthology.orchestra.run.smithi183.stdout: "num_ops": 0 2024-09-10T19:15:24.168 INFO:teuthology.orchestra.run.smithi183.stdout:} 2024-09-10T19:15:24.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:15:23 smithi191 ceph-mon[32884]: from='client.? 172.21.15.183:0/863037894' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T19:15:24.217 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:15:24.257 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:15:24.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:15:24.258 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:15:24.359 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:15:24.394 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:15:24.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:15:24.395 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:15:24.495 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:15:24.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:15:24.525 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:15:24.525 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:15:24.626 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:15:24.664 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:15:24.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:15:24.665 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:15:24.766 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:15:24.796 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:15:24.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:15:24.798 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:15:24.898 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:15:24.928 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:15:24.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:15:24.929 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:15:25.030 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:15:25.059 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:15:25.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:15:25.060 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:15:25.161 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:15:25.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:15:24 smithi183 ceph-mon[29736]: pgmap v209: 148 pgs: 148 active+clean; 2.1 GiB data, 8.4 GiB used, 617 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 1.02k op/s 2024-09-10T19:15:25.181 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:15:24 smithi183 ceph-mon[41562]: pgmap v209: 148 pgs: 148 active+clean; 2.1 GiB data, 8.4 GiB used, 617 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 1.02k op/s 2024-09-10T19:15:25.197 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:15:25.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:15:25.199 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:15:25.300 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:15:25.333 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:15:25.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:15:25.334 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:15:25.435 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:15:25.465 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:15:25.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:15:24 smithi191 ceph-mon[32884]: pgmap v209: 148 pgs: 148 active+clean; 2.1 GiB data, 8.4 GiB used, 617 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 1.02k op/s 2024-09-10T19:15:25.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:15:25.466 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:15:25.567 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:15:25.594 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:15:25.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:15:25.595 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:15:25.696 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:15:25.733 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:15:25.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:15:25.734 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:15:25.835 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:15:25.870 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:15:25.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:15:25.871 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:15:25.972 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:15:26.006 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:15:26.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:15:26.008 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:15:26.108 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:15:26.135 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:15:26.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:15:26.136 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:15:26.237 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:15:26.268 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:15:26.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:15:26.269 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:15:26.370 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:15:26.409 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:15:26.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:15:26.448 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:15:26.548 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:15:26.586 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:15:26.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:15:26.587 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:15:26.688 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:15:26.722 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:15:26.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:15:26.723 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:15:26.824 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:15:26.866 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:15:26.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:15:26.868 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:15:26.968 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:15:27.010 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:15:27.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:15:27.011 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:15:27.111 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:15:27.142 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:15:27.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:15:27.143 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:15:27.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:15:26 smithi183 ceph-mon[29736]: pgmap v210: 148 pgs: 148 active+clean; 1.7 GiB data, 7.3 GiB used, 619 GiB / 626 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 1.88k op/s 2024-09-10T19:15:27.181 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:15:26 smithi183 ceph-mon[41562]: pgmap v210: 148 pgs: 148 active+clean; 1.7 GiB data, 7.3 GiB used, 619 GiB / 626 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 1.88k op/s 2024-09-10T19:15:27.244 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:15:27.269 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:15:26 smithi191 ceph-mon[32884]: pgmap v210: 148 pgs: 148 active+clean; 1.7 GiB data, 7.3 GiB used, 619 GiB / 626 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 1.88k op/s 2024-09-10T19:15:27.281 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:15:27.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:15:27.282 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:15:27.383 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:15:27.420 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:15:27.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:15:27.421 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:15:27.522 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:15:27.551 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:15:27.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:15:27.552 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:15:27.652 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:15:27.683 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:15:27.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:15:27.684 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:15:27.717 INFO:journalctl@ceph.mgr.x.smithi191.stdout:Sep 10 19:15:27 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-x[34967]: ::ffff:172.21.15.191 - - [10/Sep/2024:19:15:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T19:15:27.785 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:15:27.827 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:15:27.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:15:27.828 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:15:27.929 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:15:27.965 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:15:27.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:15:27.966 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:15:28.067 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:15:28.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:15:28.103 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:15:28.103 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:15:28.204 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:15:28.231 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:15:28.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:15:28.232 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:15:28.333 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:15:28.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:15:28.374 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:15:28.374 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:15:28.475 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:15:28.512 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:15:28.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:15:28.513 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:15:28.604 INFO:teuthology.orchestra.run.smithi183.stdout:{ 2024-09-10T19:15:28.604 INFO:teuthology.orchestra.run.smithi183.stdout: "ops": [ 2024-09-10T19:15:28.604 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:15:28.604 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_repop(client.14781.0:258301 7.b e139/126 7:d656d80e:::benchmark_data_smithi183_84160_object117920:head v 139'16095, mlcod=139'16092)", 2024-09-10T19:15:28.604 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-10T19:15:28.596337+0000", 2024-09-10T19:15:28.604 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 0.0051597379999999997, 2024-09-10T19:15:28.604 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0051886579999999996, 2024-09-10T19:15:28.604 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-10T19:15:28.604 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "started", 2024-09-10T19:15:28.604 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-10T19:15:28.605 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:15:28.605 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-10T19:15:28.605 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:15:28.596337+0000", 2024-09-10T19:15:28.605 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:15:28.605 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:15:28.605 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:15:28.605 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-10T19:15:28.605 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:15:28.596337+0000", 2024-09-10T19:15:28.605 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:15:28.605 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:15:28.605 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:15:28.605 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-10T19:15:28.606 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:15:28.596336+0000", 2024-09-10T19:15:28.606 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4294967295.999999 2024-09-10T19:15:28.606 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:15:28.606 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:15:28.606 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-10T19:15:28.606 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:15:28.596342+0000", 2024-09-10T19:15:28.606 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.8869999999999997e-06 2024-09-10T19:15:28.606 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:15:28.606 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:15:28.606 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-10T19:15:28.606 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:15:28.596353+0000", 2024-09-10T19:15:28.606 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.0719e-05 2024-09-10T19:15:28.606 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:15:28.606 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:15:28.607 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-10T19:15:28.607 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:15:28.596358+0000", 2024-09-10T19:15:28.607 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.288e-06 2024-09-10T19:15:28.607 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:15:28.607 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:15:28.607 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-10T19:15:28.607 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:15:28.597609+0000", 2024-09-10T19:15:28.607 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.001251217 2024-09-10T19:15:28.607 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:15:28.607 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:15:28.607 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-10T19:15:28.607 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:15:28.597652+0000", 2024-09-10T19:15:28.607 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.2320000000000001e-05 2024-09-10T19:15:28.608 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:15:28.608 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:15:28.608 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:15:28.608 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:15:28.608 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:15:28.608 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_repop(client.14781.0:258300 7.2 e139/126 7:47f008b0:::benchmark_data_smithi183_84160_object117919:head v 139'16383, mlcod=139'16380)", 2024-09-10T19:15:28.608 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-10T19:15:28.598873+0000", 2024-09-10T19:15:28.608 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 0.0026230770000000001, 2024-09-10T19:15:28.608 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.002713872, 2024-09-10T19:15:28.608 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-10T19:15:28.608 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "started", 2024-09-10T19:15:28.608 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-10T19:15:28.608 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:15:28.609 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-10T19:15:28.609 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:15:28.598873+0000", 2024-09-10T19:15:28.609 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:15:28.609 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:15:28.609 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:15:28.609 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-10T19:15:28.609 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:15:28.598873+0000", 2024-09-10T19:15:28.609 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:15:28.609 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:15:28.609 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:15:28.609 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-10T19:15:28.609 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:15:28.598873+0000", 2024-09-10T19:15:28.609 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4294967295.999999 2024-09-10T19:15:28.610 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:15:28.610 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:15:28.610 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-10T19:15:28.610 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:15:28.598880+0000", 2024-09-10T19:15:28.610 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.9100000000000005e-06 2024-09-10T19:15:28.610 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:15:28.610 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:15:28.610 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-10T19:15:28.610 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:15:28.598896+0000", 2024-09-10T19:15:28.610 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.5911999999999999e-05 2024-09-10T19:15:28.610 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:15:28.610 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:15:28.610 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-10T19:15:28.611 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:15:28.598903+0000", 2024-09-10T19:15:28.611 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.5889999999999999e-06 2024-09-10T19:15:28.611 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:15:28.611 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:15:28.611 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-10T19:15:28.611 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:15:28.598974+0000", 2024-09-10T19:15:28.611 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.1408999999999993e-05 2024-09-10T19:15:28.611 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:15:28.611 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:15:28.611 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-10T19:15:28.611 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:15:28.599049+0000", 2024-09-10T19:15:28.611 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.4324000000000002e-05 2024-09-10T19:15:28.611 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:15:28.611 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:15:28.611 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:15:28.612 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:15:28.612 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:15:28.612 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.14781.0:258307 7.e 7:75ec973d:::benchmark_data_smithi183_84160_object117926:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e139)", 2024-09-10T19:15:28.612 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-10T19:15:28.599536+0000", 2024-09-10T19:15:28.612 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 0.0019607270000000002, 2024-09-10T19:15:28.612 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0021083999999999999, 2024-09-10T19:15:28.612 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-10T19:15:28.612 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "waiting for sub ops", 2024-09-10T19:15:28.612 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-10T19:15:28.612 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.14781", 2024-09-10T19:15:28.612 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.183:0/2025776420", 2024-09-10T19:15:28.612 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 258307 2024-09-10T19:15:28.612 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:15:28.612 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-10T19:15:28.613 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:15:28.613 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-10T19:15:28.613 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:15:28.599536+0000", 2024-09-10T19:15:28.613 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:15:28.613 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:15:28.613 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:15:28.613 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-10T19:15:28.613 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:15:28.599536+0000", 2024-09-10T19:15:28.613 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:15:28.613 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:15:28.613 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:15:28.613 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-10T19:15:28.613 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:15:28.599534+0000", 2024-09-10T19:15:28.613 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4294967295.9999981 2024-09-10T19:15:28.614 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:15:28.614 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:15:28.614 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-10T19:15:28.614 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:15:28.599542+0000", 2024-09-10T19:15:28.614 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.6930000000000007e-06 2024-09-10T19:15:28.614 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:15:28.614 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:15:28.614 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-10T19:15:28.614 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:15:28.599560+0000", 2024-09-10T19:15:28.614 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.8108e-05 2024-09-10T19:15:28.614 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:15:28.614 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:15:28.614 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-10T19:15:28.614 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:15:28.599567+0000", 2024-09-10T19:15:28.615 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.4510000000000003e-06 2024-09-10T19:15:28.615 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:15:28.615 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:15:28.615 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-10T19:15:28.615 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:15:28.599624+0000", 2024-09-10T19:15:28.615 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.6218999999999998e-05 2024-09-10T19:15:28.615 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:15:28.615 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:15:28.615 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-10T19:15:28.615 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:15:28.600695+0000", 2024-09-10T19:15:28.615 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0010709700000000001 2024-09-10T19:15:28.615 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:15:28.615 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:15:28.615 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 0,5", 2024-09-10T19:15:28.616 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:15:28.600839+0000", 2024-09-10T19:15:28.616 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00014475499999999999 2024-09-10T19:15:28.616 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:15:28.616 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:15:28.616 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:15:28.616 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:15:28.616 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:15:28.616 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_repop(client.14781.0:258304 7.2 e139/126)", 2024-09-10T19:15:28.616 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-10T19:15:28.600050+0000", 2024-09-10T19:15:28.616 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 0.0014461820000000001, 2024-09-10T19:15:28.616 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0016557, 2024-09-10T19:15:28.616 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-10T19:15:28.616 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "queued for pg", 2024-09-10T19:15:28.616 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-10T19:15:28.617 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:15:28.617 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-10T19:15:28.617 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:15:28.600050+0000", 2024-09-10T19:15:28.617 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:15:28.617 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:15:28.617 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:15:28.617 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-10T19:15:28.617 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:15:28.600050+0000", 2024-09-10T19:15:28.617 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:15:28.617 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:15:28.617 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:15:28.617 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-10T19:15:28.618 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:15:28.600050+0000", 2024-09-10T19:15:28.618 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4294967295.9999995 2024-09-10T19:15:28.618 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:15:28.618 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:15:28.618 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-10T19:15:28.618 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:15:28.600056+0000", 2024-09-10T19:15:28.618 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.9120000000000003e-06 2024-09-10T19:15:28.618 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:15:28.618 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:15:28.618 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-10T19:15:28.618 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:15:28.600066+0000", 2024-09-10T19:15:28.618 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.0675e-05 2024-09-10T19:15:28.618 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:15:28.618 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:15:28.619 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-10T19:15:28.619 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:15:28.600072+0000", 2024-09-10T19:15:28.619 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.4090000000000002e-06 2024-09-10T19:15:28.619 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:15:28.619 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:15:28.619 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:15:28.619 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:15:28.619 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:15:28.619 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_repop(client.14781.0:258309 7.0 e139/131)", 2024-09-10T19:15:28.619 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-10T19:15:28.600950+0000", 2024-09-10T19:15:28.619 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 0.00054682299999999999, 2024-09-10T19:15:28.619 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00079717899999999996, 2024-09-10T19:15:28.619 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-10T19:15:28.619 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "queued for pg", 2024-09-10T19:15:28.620 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-10T19:15:28.620 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:15:28.620 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-10T19:15:28.620 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:15:28.600950+0000", 2024-09-10T19:15:28.620 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:15:28.620 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:15:28.620 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:15:28.620 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-10T19:15:28.620 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:15:28.600950+0000", 2024-09-10T19:15:28.621 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:15:28.621 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:15:28.621 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:15:28.621 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-10T19:15:28.621 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:15:28.600949+0000", 2024-09-10T19:15:28.621 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4294967295.9999995 2024-09-10T19:15:28.621 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:15:28.621 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:15:28.621 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-10T19:15:28.621 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:15:28.600954+0000", 2024-09-10T19:15:28.621 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.3650000000000003e-06 2024-09-10T19:15:28.621 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:15:28.621 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:15:28.621 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-10T19:15:28.622 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:15:28.600974+0000", 2024-09-10T19:15:28.622 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.0000000000000002e-05 2024-09-10T19:15:28.622 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:15:28.622 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:15:28.622 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-10T19:15:28.622 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:15:28.600980+0000", 2024-09-10T19:15:28.622 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.7529999999999998e-06 2024-09-10T19:15:28.622 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:15:28.622 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:15:28.622 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:15:28.622 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:15:28.622 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:15:28.622 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.14781.0:258310 7.e 7.7acdfbde (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e139)", 2024-09-10T19:15:28.622 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-10T19:15:28.601109+0000", 2024-09-10T19:15:28.623 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 0.00038740699999999999, 2024-09-10T19:15:28.623 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00067900700000000003, 2024-09-10T19:15:28.623 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-10T19:15:28.623 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "queued for pg", 2024-09-10T19:15:28.623 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-10T19:15:28.623 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.14781", 2024-09-10T19:15:28.623 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.183:0/2025776420", 2024-09-10T19:15:28.623 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 258310 2024-09-10T19:15:28.623 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:15:28.623 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-10T19:15:28.623 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:15:28.623 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-10T19:15:28.623 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:15:28.601109+0000", 2024-09-10T19:15:28.623 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:15:28.624 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:15:28.624 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:15:28.624 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-10T19:15:28.624 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:15:28.601109+0000", 2024-09-10T19:15:28.624 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:15:28.624 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:15:28.624 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:15:28.624 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-10T19:15:28.624 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:15:28.601108+0000", 2024-09-10T19:15:28.624 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4294967295.9999986 2024-09-10T19:15:28.624 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:15:28.624 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:15:28.624 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-10T19:15:28.624 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:15:28.601115+0000", 2024-09-10T19:15:28.625 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.6390000000000003e-06 2024-09-10T19:15:28.625 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:15:28.625 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:15:28.625 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-10T19:15:28.625 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:15:28.601131+0000", 2024-09-10T19:15:28.625 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.6694999999999999e-05 2024-09-10T19:15:28.625 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:15:28.625 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:15:28.625 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-10T19:15:28.625 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:15:28.601138+0000", 2024-09-10T19:15:28.625 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.5860000000000001e-06 2024-09-10T19:15:28.625 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:15:28.625 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:15:28.625 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:15:28.625 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:15:28.626 INFO:teuthology.orchestra.run.smithi183.stdout: ], 2024-09-10T19:15:28.626 INFO:teuthology.orchestra.run.smithi183.stdout: "num_ops": 6 2024-09-10T19:15:28.626 INFO:teuthology.orchestra.run.smithi183.stdout:} 2024-09-10T19:15:28.626 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:15:28.666 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:15:28.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:15:28.667 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:15:28.768 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:15:28.777 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-10T19:15:28.808 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:15:28.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:15:28.809 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:15:28.910 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:15:28.956 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:15:28.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:15:28.957 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:15:28.991 INFO:teuthology.orchestra.run.smithi183.stderr:noscrub is unset 2024-09-10T19:15:29.058 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:15:29.085 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:15:29.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:15:29.086 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:15:29.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:15:28 smithi183 ceph-mon[29736]: pgmap v211: 148 pgs: 148 active+clean; 1.7 GiB data, 7.3 GiB used, 619 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.45k op/s 2024-09-10T19:15:29.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:15:28 smithi183 ceph-mon[29736]: from='client.? 172.21.15.183:0/3580438709' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-10T19:15:29.182 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:15:28 smithi183 ceph-mon[29736]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-10T19:15:29.182 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:15:28 smithi183 ceph-mon[29736]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:15:29.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:15:28 smithi183 ceph-mon[41562]: pgmap v211: 148 pgs: 148 active+clean; 1.7 GiB data, 7.3 GiB used, 619 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.45k op/s 2024-09-10T19:15:29.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:15:28 smithi183 ceph-mon[41562]: from='client.? 172.21.15.183:0/3580438709' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-10T19:15:29.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:15:28 smithi183 ceph-mon[41562]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-10T19:15:29.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:15:28 smithi183 ceph-mon[41562]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:15:29.186 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:15:29.230 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:15:29.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:15:29.231 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:15:29.332 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:15:29.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:15:29.362 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:15:29.362 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:15:29.462 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:15:29.466 INFO:journalctl@ceph.iscsi.iscsi.a.smithi191.stdout:Sep 10 19:15:29 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-iscsi-iscsi-a[61816]: debug there is no tcmu-runner data available 2024-09-10T19:15:29.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:15:28 smithi191 ceph-mon[32884]: pgmap v211: 148 pgs: 148 active+clean; 1.7 GiB data, 7.3 GiB used, 619 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.45k op/s 2024-09-10T19:15:29.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:15:28 smithi191 ceph-mon[32884]: from='client.? 172.21.15.183:0/3580438709' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-10T19:15:29.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:15:28 smithi191 ceph-mon[32884]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-10T19:15:29.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:15:28 smithi191 ceph-mon[32884]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:15:29.502 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:15:29.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:15:29.503 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:15:29.604 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:15:29.633 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:15:29.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:15:29.633 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:15:29.661 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph daemon osd.2 dump_blocked_ops 2024-09-10T19:15:29.734 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:15:29.779 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:15:29.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:15:29.780 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:15:29.881 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:15:29.911 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:15:29.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:15:29.912 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:15:30.012 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:15:30.043 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:15:30.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:15:30.044 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:15:30.145 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:15:30.175 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:15:30.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:15:30.176 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:15:30.253 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:15:30.255 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:15:30.268 INFO:tasks.thrashosds.thrasher:in_osds: [0, 2, 3, 4, 5, 6, 7] out_osds: [1] dead_osds: [] live_osds: [0, 2, 3, 4, 5, 6, 7, 1] 2024-09-10T19:15:30.268 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-10T19:15:30.268 INFO:tasks.thrashosds.thrasher:inject_pause on osd.2 2024-09-10T19:15:30.269 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-09-10T19:15:30.269 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-09-10T19:15:30.269 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph daemon osd.2 config set filestore_inject_stall 3 2024-09-10T19:15:30.277 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:15:30.329 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:15:30 smithi183 ceph-mon[41562]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T19:15:30.329 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:15:30 smithi183 ceph-mon[41562]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-10T19:15:30.329 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:15:30 smithi183 ceph-mon[41562]: osdmap e140: 8 total, 8 up, 7 in 2024-09-10T19:15:30.329 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:15:30 smithi183 ceph-mon[41562]: from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:15:30.329 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:15:30 smithi183 ceph-mon[41562]: 6.1d scrub starts 2024-09-10T19:15:30.329 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:15:30 smithi183 ceph-mon[41562]: 6.1d scrub ok 2024-09-10T19:15:30.329 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:15:30 smithi183 ceph-mon[29736]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T19:15:30.329 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:15:30 smithi183 ceph-mon[29736]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-10T19:15:30.330 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:15:30 smithi183 ceph-mon[29736]: osdmap e140: 8 total, 8 up, 7 in 2024-09-10T19:15:30.330 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:15:30 smithi183 ceph-mon[29736]: from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:15:30.330 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:15:30 smithi183 ceph-mon[29736]: 6.1d scrub starts 2024-09-10T19:15:30.330 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:15:30 smithi183 ceph-mon[29736]: 6.1d scrub ok 2024-09-10T19:15:30.371 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:15:30.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:15:30.373 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:15:30.465 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:15:30 smithi191 ceph-mon[32884]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T19:15:30.465 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:15:30 smithi191 ceph-mon[32884]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-10T19:15:30.465 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:15:30 smithi191 ceph-mon[32884]: osdmap e140: 8 total, 8 up, 7 in 2024-09-10T19:15:30.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:15:30 smithi191 ceph-mon[32884]: from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:15:30.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:15:30 smithi191 ceph-mon[32884]: 6.1d scrub starts 2024-09-10T19:15:30.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:15:30 smithi191 ceph-mon[32884]: 6.1d scrub ok 2024-09-10T19:15:30.473 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:15:30.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:15:30.520 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:15:30.520 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:15:30.621 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:15:30.659 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:15:30.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:15:30.660 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:15:30.761 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:15:30.789 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:15:30.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:15:30.790 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:15:30.890 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:15:30.932 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:15:30.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:15:30.936 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:15:31.036 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:15:31.077 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:15:31.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:15:31.078 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:15:31.179 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:15:31.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:15:31 smithi183 ceph-mon[29736]: 2.9 scrub starts 2024-09-10T19:15:31.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:15:31 smithi183 ceph-mon[29736]: 2.9 scrub ok 2024-09-10T19:15:31.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:15:31 smithi183 ceph-mon[29736]: pgmap v213: 148 pgs: 148 active+clean; 1.6 GiB data, 7.0 GiB used, 619 GiB / 626 GiB avail; 584 B/s rd, 0 B/s wr, 912 op/s 2024-09-10T19:15:31.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:15:31 smithi183 ceph-mon[29736]: 4.12 scrub starts 2024-09-10T19:15:31.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:15:31 smithi183 ceph-mon[29736]: 4.12 scrub ok 2024-09-10T19:15:31.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:15:31 smithi183 ceph-mon[29736]: 6.e scrub starts 2024-09-10T19:15:31.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:15:31 smithi183 ceph-mon[29736]: 6.e scrub ok 2024-09-10T19:15:31.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:15:31 smithi183 ceph-mon[29736]: 5.16 scrub starts 2024-09-10T19:15:31.182 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:15:31 smithi183 ceph-mon[29736]: 5.16 scrub ok 2024-09-10T19:15:31.182 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:15:31 smithi183 ceph-mon[29736]: 3.0 scrub starts 2024-09-10T19:15:31.182 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:15:31 smithi183 ceph-mon[29736]: 3.0 scrub ok 2024-09-10T19:15:31.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:15:31 smithi183 ceph-mon[41562]: 2.9 scrub starts 2024-09-10T19:15:31.183 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:15:31 smithi183 ceph-mon[41562]: 2.9 scrub ok 2024-09-10T19:15:31.183 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:15:31 smithi183 ceph-mon[41562]: pgmap v213: 148 pgs: 148 active+clean; 1.6 GiB data, 7.0 GiB used, 619 GiB / 626 GiB avail; 584 B/s rd, 0 B/s wr, 912 op/s 2024-09-10T19:15:31.183 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:15:31 smithi183 ceph-mon[41562]: 4.12 scrub starts 2024-09-10T19:15:31.183 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:15:31 smithi183 ceph-mon[41562]: 4.12 scrub ok 2024-09-10T19:15:31.183 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:15:31 smithi183 ceph-mon[41562]: 6.e scrub starts 2024-09-10T19:15:31.183 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:15:31 smithi183 ceph-mon[41562]: 6.e scrub ok 2024-09-10T19:15:31.183 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:15:31 smithi183 ceph-mon[41562]: 5.16 scrub starts 2024-09-10T19:15:31.183 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:15:31 smithi183 ceph-mon[41562]: 5.16 scrub ok 2024-09-10T19:15:31.183 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:15:31 smithi183 ceph-mon[41562]: 3.0 scrub starts 2024-09-10T19:15:31.184 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:15:31 smithi183 ceph-mon[41562]: 3.0 scrub ok 2024-09-10T19:15:31.206 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:15:31.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:15:31.207 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:15:31.308 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:15:31.340 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:15:31.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:15:31.341 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:15:31.442 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:15:31.465 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:15:31 smithi191 ceph-mon[32884]: 2.9 scrub starts 2024-09-10T19:15:31.465 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:15:31 smithi191 ceph-mon[32884]: 2.9 scrub ok 2024-09-10T19:15:31.465 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:15:31 smithi191 ceph-mon[32884]: pgmap v213: 148 pgs: 148 active+clean; 1.6 GiB data, 7.0 GiB used, 619 GiB / 626 GiB avail; 584 B/s rd, 0 B/s wr, 912 op/s 2024-09-10T19:15:31.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:15:31 smithi191 ceph-mon[32884]: 4.12 scrub starts 2024-09-10T19:15:31.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:15:31 smithi191 ceph-mon[32884]: 4.12 scrub ok 2024-09-10T19:15:31.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:15:31 smithi191 ceph-mon[32884]: 6.e scrub starts 2024-09-10T19:15:31.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:15:31 smithi191 ceph-mon[32884]: 6.e scrub ok 2024-09-10T19:15:31.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:15:31 smithi191 ceph-mon[32884]: 5.16 scrub starts 2024-09-10T19:15:31.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:15:31 smithi191 ceph-mon[32884]: 5.16 scrub ok 2024-09-10T19:15:31.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:15:31 smithi191 ceph-mon[32884]: 3.0 scrub starts 2024-09-10T19:15:31.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:15:31 smithi191 ceph-mon[32884]: 3.0 scrub ok 2024-09-10T19:15:31.482 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:15:31.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:15:31.483 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:15:31.569 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Sep 10 19:15:31 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-y[29989]: ::ffff:172.21.15.191 - - [10/Sep/2024:19:15:31] "GET /metrics HTTP/1.1" 200 224058 "" "Prometheus/2.43.0" 2024-09-10T19:15:31.584 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:15:31.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:15:31.621 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:15:31.621 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:15:31.722 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:15:31.755 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:15:31.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:15:31.756 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:15:31.856 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:15:31.890 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:15:31.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:15:31.891 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:15:31.992 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:15:32.019 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:15:32.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:15:32.020 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:15:32.121 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:15:32.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:15:32.157 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:15:32.157 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:15:32.230 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-10T19:15:32.257 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:15:32.282 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:15:32.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:15:32.283 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:15:32.384 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:15:32.411 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:15:32.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:15:32.412 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:15:32.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:15:32 smithi183 ceph-mon[29736]: 2.0 scrub starts 2024-09-10T19:15:32.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:15:32 smithi183 ceph-mon[29736]: 2.0 scrub ok 2024-09-10T19:15:32.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:15:32 smithi183 ceph-mon[29736]: 5.8 scrub starts 2024-09-10T19:15:32.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:15:32 smithi183 ceph-mon[29736]: 5.8 scrub ok 2024-09-10T19:15:32.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:15:32 smithi183 ceph-mon[41562]: 2.0 scrub starts 2024-09-10T19:15:32.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:15:32 smithi183 ceph-mon[41562]: 2.0 scrub ok 2024-09-10T19:15:32.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:15:32 smithi183 ceph-mon[41562]: 5.8 scrub starts 2024-09-10T19:15:32.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:15:32 smithi183 ceph-mon[41562]: 5.8 scrub ok 2024-09-10T19:15:32.465 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:15:32 smithi191 ceph-mon[32884]: 2.0 scrub starts 2024-09-10T19:15:32.465 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:15:32 smithi191 ceph-mon[32884]: 2.0 scrub ok 2024-09-10T19:15:32.465 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:15:32 smithi191 ceph-mon[32884]: 5.8 scrub starts 2024-09-10T19:15:32.465 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:15:32 smithi191 ceph-mon[32884]: 5.8 scrub ok 2024-09-10T19:15:32.513 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:15:32.536 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:15:32.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:15:32.537 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:15:32.586 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:15:32.638 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:15:32.668 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:15:32.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:15:32.670 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:15:32.770 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:15:32.806 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:15:32.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:15:32.807 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:15:32.907 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:15:32.948 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:15:32.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:15:32.949 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:15:33.050 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:15:33.077 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:15:33.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:15:33.078 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:15:33.178 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:15:33.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:15:33.211 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:15:33.211 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:15:33.312 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:15:33.340 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:15:33.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:15:33.341 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:15:33.348 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:15:33 smithi183 ceph-mon[29736]: pgmap v214: 148 pgs: 148 active+clean; 1.3 GiB data, 6.0 GiB used, 620 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.35k op/s 2024-09-10T19:15:33.348 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:15:33 smithi183 ceph-mon[41562]: pgmap v214: 148 pgs: 148 active+clean; 1.3 GiB data, 6.0 GiB used, 620 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.35k op/s 2024-09-10T19:15:33.442 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:15:33.465 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:15:33 smithi191 ceph-mon[32884]: pgmap v214: 148 pgs: 148 active+clean; 1.3 GiB data, 6.0 GiB used, 620 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.35k op/s 2024-09-10T19:15:33.475 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:15:33.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:15:33.476 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:15:33.576 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:15:33.608 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:15:33.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:15:33.609 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:15:33.709 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:15:33.750 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:15:33.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:15:33.751 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:15:33.852 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:15:33.887 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:15:33.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:15:33.888 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:15:33.989 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:15:34.016 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:15:34.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:15:34.017 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:15:34.026 INFO:teuthology.orchestra.run.smithi183.stdout:{ 2024-09-10T19:15:34.026 INFO:teuthology.orchestra.run.smithi183.stdout: "ops": [], 2024-09-10T19:15:34.027 INFO:teuthology.orchestra.run.smithi183.stdout: "complaint_time": 30, 2024-09-10T19:15:34.027 INFO:teuthology.orchestra.run.smithi183.stdout: "num_blocked_ops": 0 2024-09-10T19:15:34.027 INFO:teuthology.orchestra.run.smithi183.stdout:} 2024-09-10T19:15:34.117 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:15:34.147 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:15:34.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:15:34.148 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:15:34.248 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:15:34.273 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:15:34 smithi183 ceph-mon[29736]: 5.1d scrub starts 2024-09-10T19:15:34.273 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:15:34 smithi183 ceph-mon[29736]: 5.1d scrub ok 2024-09-10T19:15:34.273 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:15:34 smithi183 ceph-mon[29736]: 5.18 scrub starts 2024-09-10T19:15:34.273 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:15:34 smithi183 ceph-mon[29736]: 5.18 scrub ok 2024-09-10T19:15:34.273 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:15:34 smithi183 ceph-mon[41562]: 5.1d scrub starts 2024-09-10T19:15:34.273 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:15:34 smithi183 ceph-mon[41562]: 5.1d scrub ok 2024-09-10T19:15:34.273 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:15:34 smithi183 ceph-mon[41562]: 5.18 scrub starts 2024-09-10T19:15:34.274 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:15:34 smithi183 ceph-mon[41562]: 5.18 scrub ok 2024-09-10T19:15:34.287 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:15:34.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:15:34.289 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:15:34.389 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:15:34.418 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:15:34.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:15:34.419 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:15:34.465 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:15:34 smithi191 ceph-mon[32884]: 5.1d scrub starts 2024-09-10T19:15:34.465 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:15:34 smithi191 ceph-mon[32884]: 5.1d scrub ok 2024-09-10T19:15:34.465 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:15:34 smithi191 ceph-mon[32884]: 5.18 scrub starts 2024-09-10T19:15:34.465 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:15:34 smithi191 ceph-mon[32884]: 5.18 scrub ok 2024-09-10T19:15:34.520 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:15:34.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:15:34.557 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:15:34.557 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:15:34.658 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:15:34.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:15:34.698 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:15:34.698 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:15:34.799 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:15:34.838 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:15:34.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:15:34.839 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:15:34.941 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:15:34.969 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:15:34.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:15:34.971 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:15:35.071 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:15:35.112 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:15:35.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:15:35.113 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:15:35.214 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:15:35.242 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:15:35.252 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:15:35.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:15:35.253 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:15:35.354 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:15:35.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:15:35.384 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:15:35.384 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:15:35.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:15:35 smithi183 ceph-mon[29736]: 5.1a scrub starts 2024-09-10T19:15:35.433 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:15:35 smithi183 ceph-mon[29736]: 5.1a scrub ok 2024-09-10T19:15:35.433 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:15:35 smithi183 ceph-mon[29736]: pgmap v215: 148 pgs: 148 active+clean; 1.3 GiB data, 6.0 GiB used, 620 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.35k op/s 2024-09-10T19:15:35.433 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:15:35 smithi183 ceph-mon[29736]: 6.18 scrub starts 2024-09-10T19:15:35.433 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:15:35 smithi183 ceph-mon[29736]: 6.18 scrub ok 2024-09-10T19:15:35.433 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:15:35 smithi183 ceph-mon[29736]: 4.10 scrub starts 2024-09-10T19:15:35.433 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:15:35 smithi183 ceph-mon[29736]: 4.10 scrub ok 2024-09-10T19:15:35.433 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:15:35 smithi183 ceph-mon[29736]: from='client.? 172.21.15.183:0/1391532094' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T19:15:35.433 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:15:35 smithi183 ceph-mon[41562]: 5.1a scrub starts 2024-09-10T19:15:35.433 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:15:35 smithi183 ceph-mon[41562]: 5.1a scrub ok 2024-09-10T19:15:35.433 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:15:35 smithi183 ceph-mon[41562]: pgmap v215: 148 pgs: 148 active+clean; 1.3 GiB data, 6.0 GiB used, 620 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.35k op/s 2024-09-10T19:15:35.434 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:15:35 smithi183 ceph-mon[41562]: 6.18 scrub starts 2024-09-10T19:15:35.434 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:15:35 smithi183 ceph-mon[41562]: 6.18 scrub ok 2024-09-10T19:15:35.434 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:15:35 smithi183 ceph-mon[41562]: 4.10 scrub starts 2024-09-10T19:15:35.434 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:15:35 smithi183 ceph-mon[41562]: 4.10 scrub ok 2024-09-10T19:15:35.434 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:15:35 smithi183 ceph-mon[41562]: from='client.? 172.21.15.183:0/1391532094' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T19:15:35.465 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:15:35 smithi191 ceph-mon[32884]: 5.1a scrub starts 2024-09-10T19:15:35.465 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:15:35 smithi191 ceph-mon[32884]: 5.1a scrub ok 2024-09-10T19:15:35.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:15:35 smithi191 ceph-mon[32884]: pgmap v215: 148 pgs: 148 active+clean; 1.3 GiB data, 6.0 GiB used, 620 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.35k op/s 2024-09-10T19:15:35.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:15:35 smithi191 ceph-mon[32884]: 6.18 scrub starts 2024-09-10T19:15:35.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:15:35 smithi191 ceph-mon[32884]: 6.18 scrub ok 2024-09-10T19:15:35.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:15:35 smithi191 ceph-mon[32884]: 4.10 scrub starts 2024-09-10T19:15:35.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:15:35 smithi191 ceph-mon[32884]: 4.10 scrub ok 2024-09-10T19:15:35.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:15:35 smithi191 ceph-mon[32884]: from='client.? 172.21.15.183:0/1391532094' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T19:15:35.485 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:15:35.530 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:15:35.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:15:35.531 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:15:35.632 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:15:35.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:15:35.665 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:15:35.665 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:15:35.766 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:15:35.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:15:35.796 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:15:35.796 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:15:35.897 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:15:35.927 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:15:35.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:15:35.928 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:15:35.960 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph daemon osd.2 dump_historic_ops 2024-09-10T19:15:36.028 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:15:36.060 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:15:36.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:15:36.061 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:15:36.162 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:15:36.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:15:36 smithi183 ceph-mon[29736]: 5.5 scrub starts 2024-09-10T19:15:36.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:15:36 smithi183 ceph-mon[29736]: 5.5 scrub ok 2024-09-10T19:15:36.181 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:15:36 smithi183 ceph-mon[41562]: 5.5 scrub starts 2024-09-10T19:15:36.181 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:15:36 smithi183 ceph-mon[41562]: 5.5 scrub ok 2024-09-10T19:15:36.187 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:15:36.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:15:36.188 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:15:36.288 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:15:36.305 INFO:teuthology.orchestra.run.smithi183.stdout:osd.0: {} 2024-09-10T19:15:36.305 INFO:teuthology.orchestra.run.smithi183.stdout:osd.1: {} 2024-09-10T19:15:36.305 INFO:teuthology.orchestra.run.smithi183.stdout:osd.2: {} 2024-09-10T19:15:36.305 INFO:teuthology.orchestra.run.smithi183.stdout:osd.3: {} 2024-09-10T19:15:36.306 INFO:teuthology.orchestra.run.smithi183.stdout:osd.4: {} 2024-09-10T19:15:36.306 INFO:teuthology.orchestra.run.smithi183.stdout:osd.5: {} 2024-09-10T19:15:36.306 INFO:teuthology.orchestra.run.smithi183.stdout:osd.6: {} 2024-09-10T19:15:36.306 INFO:teuthology.orchestra.run.smithi183.stdout:osd.7: {} 2024-09-10T19:15:36.306 INFO:teuthology.orchestra.run.smithi183.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-10T19:15:36.306 INFO:teuthology.orchestra.run.smithi183.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-10T19:15:36.306 INFO:teuthology.orchestra.run.smithi183.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-10T19:15:36.306 INFO:teuthology.orchestra.run.smithi183.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-10T19:15:36.306 INFO:teuthology.orchestra.run.smithi183.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-10T19:15:36.306 INFO:teuthology.orchestra.run.smithi183.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-10T19:15:36.306 INFO:teuthology.orchestra.run.smithi183.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-10T19:15:36.307 INFO:teuthology.orchestra.run.smithi183.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-10T19:15:36.326 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:15:36.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:15:36.327 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:15:36.428 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:15:36.465 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:15:36 smithi191 ceph-mon[32884]: 5.5 scrub starts 2024-09-10T19:15:36.465 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:15:36 smithi191 ceph-mon[32884]: 5.5 scrub ok 2024-09-10T19:15:36.467 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:15:36.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:15:36.468 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:15:36.569 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:15:36.616 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:15:36.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:15:36.617 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:15:36.718 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:15:36.749 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:15:36.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:15:36.750 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:15:36.851 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:15:36.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:15:36.886 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:15:36.886 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:15:36.987 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:15:37.034 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:15:37.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:15:37.035 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:15:37.136 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:15:37.173 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:15:37.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:15:37.175 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:15:37.245 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:15:37 smithi183 ceph-mon[29736]: pgmap v216: 148 pgs: 148 active+clean; 884 MiB data, 4.8 GiB used, 621 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.36k op/s 2024-09-10T19:15:37.245 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:15:37 smithi183 ceph-mon[29736]: 5.0 scrub starts 2024-09-10T19:15:37.245 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:15:37 smithi183 ceph-mon[29736]: 5.0 scrub ok 2024-09-10T19:15:37.245 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:15:37 smithi183 ceph-mon[29736]: 2.10 scrub starts 2024-09-10T19:15:37.245 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:15:37 smithi183 ceph-mon[29736]: 2.10 scrub ok 2024-09-10T19:15:37.245 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:15:37 smithi183 ceph-mon[29736]: 5.12 scrub starts 2024-09-10T19:15:37.245 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:15:37 smithi183 ceph-mon[29736]: 5.12 scrub ok 2024-09-10T19:15:37.246 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:15:37 smithi183 ceph-mon[41562]: pgmap v216: 148 pgs: 148 active+clean; 884 MiB data, 4.8 GiB used, 621 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.36k op/s 2024-09-10T19:15:37.246 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:15:37 smithi183 ceph-mon[41562]: 5.0 scrub starts 2024-09-10T19:15:37.246 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:15:37 smithi183 ceph-mon[41562]: 5.0 scrub ok 2024-09-10T19:15:37.246 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:15:37 smithi183 ceph-mon[41562]: 2.10 scrub starts 2024-09-10T19:15:37.246 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:15:37 smithi183 ceph-mon[41562]: 2.10 scrub ok 2024-09-10T19:15:37.246 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:15:37 smithi183 ceph-mon[41562]: 5.12 scrub starts 2024-09-10T19:15:37.246 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:15:37 smithi183 ceph-mon[41562]: 5.12 scrub ok 2024-09-10T19:15:37.275 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:15:37.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:15:37.304 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:15:37.304 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:15:37.405 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:15:37.442 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:15:37.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:15:37.443 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:15:37.449 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:15:37 smithi191 ceph-mon[32884]: pgmap v216: 148 pgs: 148 active+clean; 884 MiB data, 4.8 GiB used, 621 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.36k op/s 2024-09-10T19:15:37.449 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:15:37 smithi191 ceph-mon[32884]: 5.0 scrub starts 2024-09-10T19:15:37.450 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:15:37 smithi191 ceph-mon[32884]: 5.0 scrub ok 2024-09-10T19:15:37.450 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:15:37 smithi191 ceph-mon[32884]: 2.10 scrub starts 2024-09-10T19:15:37.450 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:15:37 smithi191 ceph-mon[32884]: 2.10 scrub ok 2024-09-10T19:15:37.450 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:15:37 smithi191 ceph-mon[32884]: 5.12 scrub starts 2024-09-10T19:15:37.450 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:15:37 smithi191 ceph-mon[32884]: 5.12 scrub ok 2024-09-10T19:15:37.543 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:15:37.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:15:37.575 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:15:37.576 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:15:37.676 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:15:37.702 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:15:37.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:15:37.703 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:15:37.715 INFO:journalctl@ceph.mgr.x.smithi191.stdout:Sep 10 19:15:37 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-x[34967]: ::ffff:172.21.15.191 - - [10/Sep/2024:19:15:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T19:15:37.804 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:15:37.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:15:37.859 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:15:37.859 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:15:37.960 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:15:38.004 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:15:38.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:15:38.005 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:15:38.106 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:15:38.145 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:15:38.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:15:38.146 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:15:38.247 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:15:38.277 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:15:38.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:15:38.278 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:15:38.378 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:15:38.417 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:15:38.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:15:38.418 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:15:38.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:15:38 smithi183 ceph-mon[29736]: 2.11 scrub starts 2024-09-10T19:15:38.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:15:38 smithi183 ceph-mon[29736]: 2.11 scrub ok 2024-09-10T19:15:38.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:15:38 smithi183 ceph-mon[29736]: purged_snaps scrub starts 2024-09-10T19:15:38.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:15:38 smithi183 ceph-mon[29736]: purged_snaps scrub ok 2024-09-10T19:15:38.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:15:38 smithi183 ceph-mon[29736]: 4.1c scrub starts 2024-09-10T19:15:38.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:15:38 smithi183 ceph-mon[29736]: 4.1c scrub ok 2024-09-10T19:15:38.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:15:38 smithi183 ceph-mon[29736]: 4.9 scrub starts 2024-09-10T19:15:38.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:15:38 smithi183 ceph-mon[29736]: 4.9 scrub ok 2024-09-10T19:15:38.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:15:38 smithi183 ceph-mon[41562]: 2.11 scrub starts 2024-09-10T19:15:38.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:15:38 smithi183 ceph-mon[41562]: 2.11 scrub ok 2024-09-10T19:15:38.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:15:38 smithi183 ceph-mon[41562]: purged_snaps scrub starts 2024-09-10T19:15:38.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:15:38 smithi183 ceph-mon[41562]: purged_snaps scrub ok 2024-09-10T19:15:38.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:15:38 smithi183 ceph-mon[41562]: 4.1c scrub starts 2024-09-10T19:15:38.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:15:38 smithi183 ceph-mon[41562]: 4.1c scrub ok 2024-09-10T19:15:38.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:15:38 smithi183 ceph-mon[41562]: 4.9 scrub starts 2024-09-10T19:15:38.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:15:38 smithi183 ceph-mon[41562]: 4.9 scrub ok 2024-09-10T19:15:38.465 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:15:38 smithi191 ceph-mon[32884]: 2.11 scrub starts 2024-09-10T19:15:38.465 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:15:38 smithi191 ceph-mon[32884]: 2.11 scrub ok 2024-09-10T19:15:38.465 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:15:38 smithi191 ceph-mon[32884]: purged_snaps scrub starts 2024-09-10T19:15:38.465 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:15:38 smithi191 ceph-mon[32884]: purged_snaps scrub ok 2024-09-10T19:15:38.465 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:15:38 smithi191 ceph-mon[32884]: 4.1c scrub starts 2024-09-10T19:15:38.465 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:15:38 smithi191 ceph-mon[32884]: 4.1c scrub ok 2024-09-10T19:15:38.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:15:38 smithi191 ceph-mon[32884]: 4.9 scrub starts 2024-09-10T19:15:38.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:15:38 smithi191 ceph-mon[32884]: 4.9 scrub ok 2024-09-10T19:15:38.519 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:15:38.555 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:15:38.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:15:38.556 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:15:38.657 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:15:38.693 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:15:38.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:15:38.694 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:15:38.794 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:15:38.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:15:38.840 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:15:38.840 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:15:38.941 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:15:38.980 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:15:38.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:15:38.982 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:15:38.992 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-10T19:15:39.083 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:15:39.110 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:15:39.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:15:39.111 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:15:39.212 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:15:39.240 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:15:39.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:15:39.241 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:15:39.341 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:15:39.376 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:15:39 smithi183 ceph-mon[41562]: pgmap v217: 148 pgs: 148 active+clean; 884 MiB data, 4.8 GiB used, 621 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.36k op/s 2024-09-10T19:15:39.376 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:15:39 smithi183 ceph-mon[41562]: 4.b scrub starts 2024-09-10T19:15:39.377 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:15:39 smithi183 ceph-mon[41562]: 4.b scrub ok 2024-09-10T19:15:39.377 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:15:39 smithi183 ceph-mon[41562]: osdmap e141: 8 total, 8 up, 7 in 2024-09-10T19:15:39.377 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:15:39 smithi183 ceph-mon[41562]: purged_snaps scrub starts 2024-09-10T19:15:39.377 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:15:39 smithi183 ceph-mon[41562]: purged_snaps scrub ok 2024-09-10T19:15:39.377 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:15:39 smithi183 ceph-mon[41562]: 2.e scrub starts 2024-09-10T19:15:39.377 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:15:39 smithi183 ceph-mon[41562]: 2.e scrub ok 2024-09-10T19:15:39.377 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:15:39 smithi183 ceph-mon[29736]: pgmap v217: 148 pgs: 148 active+clean; 884 MiB data, 4.8 GiB used, 621 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.36k op/s 2024-09-10T19:15:39.377 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:15:39 smithi183 ceph-mon[29736]: 4.b scrub starts 2024-09-10T19:15:39.377 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:15:39 smithi183 ceph-mon[29736]: 4.b scrub ok 2024-09-10T19:15:39.378 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:15:39 smithi183 ceph-mon[29736]: osdmap e141: 8 total, 8 up, 7 in 2024-09-10T19:15:39.378 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:15:39 smithi183 ceph-mon[29736]: purged_snaps scrub starts 2024-09-10T19:15:39.378 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:15:39 smithi183 ceph-mon[29736]: purged_snaps scrub ok 2024-09-10T19:15:39.378 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:15:39 smithi183 ceph-mon[29736]: 2.e scrub starts 2024-09-10T19:15:39.378 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:15:39 smithi183 ceph-mon[29736]: 2.e scrub ok 2024-09-10T19:15:39.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:15:39.381 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:15:39.381 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:15:39.465 INFO:journalctl@ceph.iscsi.iscsi.a.smithi191.stdout:Sep 10 19:15:39 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-iscsi-iscsi-a[61816]: debug there is no tcmu-runner data available 2024-09-10T19:15:39.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:15:39 smithi191 ceph-mon[32884]: pgmap v217: 148 pgs: 148 active+clean; 884 MiB data, 4.8 GiB used, 621 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.36k op/s 2024-09-10T19:15:39.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:15:39 smithi191 ceph-mon[32884]: 4.b scrub starts 2024-09-10T19:15:39.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:15:39 smithi191 ceph-mon[32884]: 4.b scrub ok 2024-09-10T19:15:39.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:15:39 smithi191 ceph-mon[32884]: osdmap e141: 8 total, 8 up, 7 in 2024-09-10T19:15:39.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:15:39 smithi191 ceph-mon[32884]: purged_snaps scrub starts 2024-09-10T19:15:39.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:15:39 smithi191 ceph-mon[32884]: purged_snaps scrub ok 2024-09-10T19:15:39.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:15:39 smithi191 ceph-mon[32884]: 2.e scrub starts 2024-09-10T19:15:39.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:15:39 smithi191 ceph-mon[32884]: 2.e scrub ok 2024-09-10T19:15:39.482 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:15:39.504 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:15:39.541 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:15:39.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:15:39.545 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:15:39.646 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:15:39.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:15:39.677 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:15:39.677 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:15:39.778 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:15:39.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:15:39.825 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:15:39.825 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:15:39.860 INFO:teuthology.orchestra.run.smithi183.stdout:{ 2024-09-10T19:15:39.860 INFO:teuthology.orchestra.run.smithi183.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-09-10T19:15:39.860 INFO:teuthology.orchestra.run.smithi183.stdout:} 2024-09-10T19:15:39.926 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:15:39.957 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:15:39.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:15:39.958 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:15:40.059 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:15:40.096 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:15:40.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:15:40.097 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:15:40.198 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:15:40.234 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:15:40.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:15:40.236 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:15:40.247 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:15:40.336 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:15:40.366 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:15:40.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:15:40.367 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:15:40.371 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:15:40 smithi183 ceph-mon[29736]: 6.5 scrub starts 2024-09-10T19:15:40.371 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:15:40 smithi183 ceph-mon[29736]: 6.5 scrub ok 2024-09-10T19:15:40.371 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:15:40 smithi183 ceph-mon[29736]: 4.a scrub starts 2024-09-10T19:15:40.371 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:15:40 smithi183 ceph-mon[29736]: 4.a scrub ok 2024-09-10T19:15:40.371 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:15:40 smithi183 ceph-mon[29736]: osdmap e142: 8 total, 8 up, 7 in 2024-09-10T19:15:40.371 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:15:40 smithi183 ceph-mon[29736]: from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:15:40.371 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:15:40 smithi183 ceph-mon[29736]: 6.1a scrub starts 2024-09-10T19:15:40.371 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:15:40 smithi183 ceph-mon[29736]: 6.1a scrub ok 2024-09-10T19:15:40.372 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:15:40 smithi183 ceph-mon[41562]: 6.5 scrub starts 2024-09-10T19:15:40.372 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:15:40 smithi183 ceph-mon[41562]: 6.5 scrub ok 2024-09-10T19:15:40.372 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:15:40 smithi183 ceph-mon[41562]: 4.a scrub starts 2024-09-10T19:15:40.372 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:15:40 smithi183 ceph-mon[41562]: 4.a scrub ok 2024-09-10T19:15:40.372 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:15:40 smithi183 ceph-mon[41562]: osdmap e142: 8 total, 8 up, 7 in 2024-09-10T19:15:40.372 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:15:40 smithi183 ceph-mon[41562]: from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:15:40.372 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:15:40 smithi183 ceph-mon[41562]: 6.1a scrub starts 2024-09-10T19:15:40.372 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:15:40 smithi183 ceph-mon[41562]: 6.1a scrub ok 2024-09-10T19:15:40.465 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:15:40 smithi191 ceph-mon[32884]: 6.5 scrub starts 2024-09-10T19:15:40.465 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:15:40 smithi191 ceph-mon[32884]: 6.5 scrub ok 2024-09-10T19:15:40.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:15:40 smithi191 ceph-mon[32884]: 4.a scrub starts 2024-09-10T19:15:40.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:15:40 smithi191 ceph-mon[32884]: 4.a scrub ok 2024-09-10T19:15:40.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:15:40 smithi191 ceph-mon[32884]: osdmap e142: 8 total, 8 up, 7 in 2024-09-10T19:15:40.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:15:40 smithi191 ceph-mon[32884]: from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:15:40.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:15:40 smithi191 ceph-mon[32884]: 6.1a scrub starts 2024-09-10T19:15:40.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:15:40 smithi191 ceph-mon[32884]: 6.1a scrub ok 2024-09-10T19:15:40.468 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:15:40.500 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:15:40.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:15:40.501 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:15:40.601 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:15:40.637 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:15:40.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:15:40.638 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:15:40.738 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:15:40.769 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:15:40.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:15:40.770 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:15:40.871 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:15:40.905 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:15:40.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:15:40.906 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:15:41.007 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:15:41.040 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:15:41.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:15:41.042 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:15:41.142 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:15:41.149 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:15:41 smithi183 ceph-mon[29736]: pgmap v220: 148 pgs: 148 active+clean; 805 MiB data, 4.5 GiB used, 621 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 999 op/s 2024-09-10T19:15:41.149 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:15:41 smithi183 ceph-mon[41562]: pgmap v220: 148 pgs: 148 active+clean; 805 MiB data, 4.5 GiB used, 621 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 999 op/s 2024-09-10T19:15:41.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:15:41.173 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:15:41.173 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:15:41.274 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:15:41.303 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:15:41.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:15:41.304 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:15:41.404 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:15:41.432 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Sep 10 19:15:41 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-y[29989]: ::ffff:172.21.15.191 - - [10/Sep/2024:19:15:41] "GET /metrics HTTP/1.1" 200 224058 "" "Prometheus/2.43.0" 2024-09-10T19:15:41.442 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:15:41.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:15:41.443 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:15:41.465 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:15:41 smithi191 ceph-mon[32884]: pgmap v220: 148 pgs: 148 active+clean; 805 MiB data, 4.5 GiB used, 621 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 999 op/s 2024-09-10T19:15:41.544 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:15:41.575 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:15:41.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:15:41.576 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:15:41.677 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:15:41.712 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:15:41.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:15:41.713 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:15:41.814 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:15:41.853 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:15:41.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:15:41.854 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:15:41.955 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:15:41.979 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:15:41.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:15:41.980 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:15:42.081 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:15:42.120 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:15:42.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:15:42.121 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:15:42.222 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:15:42.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:15:42.261 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:15:42.261 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:15:42.362 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:15:42.396 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:15:42.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:15:42.397 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:15:42.424 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:15:42 smithi183 ceph-mon[29736]: 4.6 scrub starts 2024-09-10T19:15:42.424 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:15:42 smithi183 ceph-mon[29736]: 4.6 scrub ok 2024-09-10T19:15:42.424 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:15:42 smithi183 ceph-mon[29736]: 6.b scrub starts 2024-09-10T19:15:42.424 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:15:42 smithi183 ceph-mon[29736]: 6.b scrub ok 2024-09-10T19:15:42.425 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:15:42 smithi183 ceph-mon[29736]: 4.1 scrub starts 2024-09-10T19:15:42.425 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:15:42 smithi183 ceph-mon[29736]: 4.1 scrub ok 2024-09-10T19:15:42.425 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:15:42 smithi183 ceph-mon[41562]: 4.6 scrub starts 2024-09-10T19:15:42.425 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:15:42 smithi183 ceph-mon[41562]: 4.6 scrub ok 2024-09-10T19:15:42.425 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:15:42 smithi183 ceph-mon[41562]: 6.b scrub starts 2024-09-10T19:15:42.425 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:15:42 smithi183 ceph-mon[41562]: 6.b scrub ok 2024-09-10T19:15:42.425 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:15:42 smithi183 ceph-mon[41562]: 4.1 scrub starts 2024-09-10T19:15:42.425 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:15:42 smithi183 ceph-mon[41562]: 4.1 scrub ok 2024-09-10T19:15:42.465 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:15:42 smithi191 ceph-mon[32884]: 4.6 scrub starts 2024-09-10T19:15:42.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:15:42 smithi191 ceph-mon[32884]: 4.6 scrub ok 2024-09-10T19:15:42.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:15:42 smithi191 ceph-mon[32884]: 6.b scrub starts 2024-09-10T19:15:42.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:15:42 smithi191 ceph-mon[32884]: 6.b scrub ok 2024-09-10T19:15:42.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:15:42 smithi191 ceph-mon[32884]: 4.1 scrub starts 2024-09-10T19:15:42.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:15:42 smithi191 ceph-mon[32884]: 4.1 scrub ok 2024-09-10T19:15:42.498 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:15:42.529 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:15:42.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:15:42.530 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:15:42.631 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:15:42.668 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:15:42.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:15:42.668 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:15:42.769 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:15:42.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:15:42.809 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:15:42.810 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:15:42.910 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:15:42.945 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:15:42.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:15:42.946 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:15:43.047 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:15:43.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:15:43.069 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:15:43.069 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:15:43.170 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:15:43.197 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:15:43.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:15:43.198 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:15:43.299 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:15:43.327 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:15:43.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:15:43.329 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:15:43.429 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:15:43.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:15:43 smithi183 ceph-mon[29736]: pgmap v221: 148 pgs: 148 active+clean; 457 MiB data, 3.5 GiB used, 622 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.70k op/s 2024-09-10T19:15:43.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:15:43 smithi183 ceph-mon[29736]: 6.1 scrub starts 2024-09-10T19:15:43.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:15:43 smithi183 ceph-mon[29736]: 6.1 scrub ok 2024-09-10T19:15:43.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:15:43 smithi183 ceph-mon[29736]: 2.4 scrub starts 2024-09-10T19:15:43.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:15:43 smithi183 ceph-mon[29736]: 2.4 scrub ok 2024-09-10T19:15:43.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:15:43 smithi183 ceph-mon[29736]: 2.a scrub starts 2024-09-10T19:15:43.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:15:43 smithi183 ceph-mon[29736]: 2.a scrub ok 2024-09-10T19:15:43.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:15:43 smithi183 ceph-mon[41562]: pgmap v221: 148 pgs: 148 active+clean; 457 MiB data, 3.5 GiB used, 622 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.70k op/s 2024-09-10T19:15:43.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:15:43 smithi183 ceph-mon[41562]: 6.1 scrub starts 2024-09-10T19:15:43.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:15:43 smithi183 ceph-mon[41562]: 6.1 scrub ok 2024-09-10T19:15:43.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:15:43 smithi183 ceph-mon[41562]: 2.4 scrub starts 2024-09-10T19:15:43.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:15:43 smithi183 ceph-mon[41562]: 2.4 scrub ok 2024-09-10T19:15:43.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:15:43 smithi183 ceph-mon[41562]: 2.a scrub starts 2024-09-10T19:15:43.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:15:43 smithi183 ceph-mon[41562]: 2.a scrub ok 2024-09-10T19:15:43.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:15:43.464 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:15:43.464 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:15:43.465 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:15:43 smithi191 ceph-mon[32884]: pgmap v221: 148 pgs: 148 active+clean; 457 MiB data, 3.5 GiB used, 622 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.70k op/s 2024-09-10T19:15:43.465 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:15:43 smithi191 ceph-mon[32884]: 6.1 scrub starts 2024-09-10T19:15:43.465 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:15:43 smithi191 ceph-mon[32884]: 6.1 scrub ok 2024-09-10T19:15:43.465 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:15:43 smithi191 ceph-mon[32884]: 2.4 scrub starts 2024-09-10T19:15:43.465 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:15:43 smithi191 ceph-mon[32884]: 2.4 scrub ok 2024-09-10T19:15:43.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:15:43 smithi191 ceph-mon[32884]: 2.a scrub starts 2024-09-10T19:15:43.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:15:43 smithi191 ceph-mon[32884]: 2.a scrub ok 2024-09-10T19:15:43.565 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:15:43.594 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:15:43.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:15:43.595 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:15:43.696 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:15:43.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:15:43.736 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:15:43.736 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:15:43.837 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:15:43.872 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:15:43.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:15:43.873 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:15:43.974 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:15:44.011 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:15:44.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:15:44.012 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:15:44.113 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:15:44.150 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:15:44.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:15:44.151 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:15:44.252 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:15:44.279 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:15:44.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:15:44.280 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:15:44.381 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:15:44.409 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:15:44.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:15:44.410 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:15:44.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:15:44 smithi183 ceph-mon[29736]: 2.8 scrub starts 2024-09-10T19:15:44.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:15:44 smithi183 ceph-mon[29736]: 2.8 scrub ok 2024-09-10T19:15:44.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:15:44 smithi183 ceph-mon[29736]: 2.6 scrub starts 2024-09-10T19:15:44.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:15:44 smithi183 ceph-mon[29736]: 2.6 scrub ok 2024-09-10T19:15:44.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:15:44 smithi183 ceph-mon[29736]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:15:44.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:15:44 smithi183 ceph-mon[29736]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.11", "id": [3, 5]}]: dispatch 2024-09-10T19:15:44.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:15:44 smithi183 ceph-mon[29736]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.15", "id": [3, 2]}]: dispatch 2024-09-10T19:15:44.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:15:44 smithi183 ceph-mon[29736]: from='mgr.14556 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.11", "id": [3, 5]}]: dispatch 2024-09-10T19:15:44.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:15:44 smithi183 ceph-mon[29736]: from='mgr.14556 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.15", "id": [3, 2]}]: dispatch 2024-09-10T19:15:44.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:15:44 smithi183 ceph-mon[41562]: 2.8 scrub starts 2024-09-10T19:15:44.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:15:44 smithi183 ceph-mon[41562]: 2.8 scrub ok 2024-09-10T19:15:44.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:15:44 smithi183 ceph-mon[41562]: 2.6 scrub starts 2024-09-10T19:15:44.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:15:44 smithi183 ceph-mon[41562]: 2.6 scrub ok 2024-09-10T19:15:44.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:15:44 smithi183 ceph-mon[41562]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:15:44.433 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:15:44 smithi183 ceph-mon[41562]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.11", "id": [3, 5]}]: dispatch 2024-09-10T19:15:44.433 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:15:44 smithi183 ceph-mon[41562]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.15", "id": [3, 2]}]: dispatch 2024-09-10T19:15:44.433 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:15:44 smithi183 ceph-mon[41562]: from='mgr.14556 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.11", "id": [3, 5]}]: dispatch 2024-09-10T19:15:44.433 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:15:44 smithi183 ceph-mon[41562]: from='mgr.14556 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.15", "id": [3, 2]}]: dispatch 2024-09-10T19:15:44.465 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:15:44 smithi191 ceph-mon[32884]: 2.8 scrub starts 2024-09-10T19:15:44.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:15:44 smithi191 ceph-mon[32884]: 2.8 scrub ok 2024-09-10T19:15:44.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:15:44 smithi191 ceph-mon[32884]: 2.6 scrub starts 2024-09-10T19:15:44.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:15:44 smithi191 ceph-mon[32884]: 2.6 scrub ok 2024-09-10T19:15:44.468 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:15:44 smithi191 ceph-mon[32884]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:15:44.468 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:15:44 smithi191 ceph-mon[32884]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.11", "id": [3, 5]}]: dispatch 2024-09-10T19:15:44.468 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:15:44 smithi191 ceph-mon[32884]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.15", "id": [3, 2]}]: dispatch 2024-09-10T19:15:44.468 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:15:44 smithi191 ceph-mon[32884]: from='mgr.14556 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.11", "id": [3, 5]}]: dispatch 2024-09-10T19:15:44.468 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:15:44 smithi191 ceph-mon[32884]: from='mgr.14556 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.15", "id": [3, 2]}]: dispatch 2024-09-10T19:15:44.511 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:15:44.548 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:15:44.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:15:44.549 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:15:44.650 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:15:44.685 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:15:44.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:15:44.686 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:15:44.787 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:15:44.821 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:15:44.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:15:44.822 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:15:44.923 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:15:44.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:15:44.948 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:15:44.948 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:15:45.049 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:15:45.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:15:45.077 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:15:45.077 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:15:45.178 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:15:45.199 INFO:teuthology.orchestra.run.smithi183.stderr:nodeep-scrub is unset 2024-09-10T19:15:45.206 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:15:45.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:15:45.208 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:15:45.309 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:15:45.319 INFO:tasks.radosbench.radosbench.0.smithi183.stdout:Removed 140378 objects 2024-09-10T19:15:45.319 INFO:tasks.radosbench.radosbench.0.smithi183.stdout:Clean up completed and total clean up time :116.394 2024-09-10T19:15:45.331 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:15:45.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:15:45.332 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:15:45.416 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:15:45 smithi183 ceph-mon[29736]: pgmap v222: 148 pgs: 148 active+clean; 457 MiB data, 3.5 GiB used, 622 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 854 op/s 2024-09-10T19:15:45.416 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:15:45 smithi183 ceph-mon[29736]: 7.d scrub starts 2024-09-10T19:15:45.416 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:15:45 smithi183 ceph-mon[29736]: 7.d scrub ok 2024-09-10T19:15:45.416 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:15:45 smithi183 ceph-mon[29736]: from='mgr.14556 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.11", "id": [3, 5]}]': finished 2024-09-10T19:15:45.416 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:15:45 smithi183 ceph-mon[29736]: from='mgr.14556 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.15", "id": [3, 2]}]': finished 2024-09-10T19:15:45.416 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:15:45 smithi183 ceph-mon[29736]: osdmap e143: 8 total, 8 up, 7 in 2024-09-10T19:15:45.417 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:15:45 smithi183 ceph-mon[29736]: 6.d scrub starts 2024-09-10T19:15:45.417 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:15:45 smithi183 ceph-mon[29736]: 6.d scrub ok 2024-09-10T19:15:45.417 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:15:45 smithi183 ceph-mon[29736]: from='client.? 172.21.15.183:0/3200058441' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-10T19:15:45.417 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:15:45 smithi183 ceph-mon[29736]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-10T19:15:45.417 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:15:45 smithi183 ceph-mon[41562]: pgmap v222: 148 pgs: 148 active+clean; 457 MiB data, 3.5 GiB used, 622 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 854 op/s 2024-09-10T19:15:45.418 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:15:45 smithi183 ceph-mon[41562]: 7.d scrub starts 2024-09-10T19:15:45.418 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:15:45 smithi183 ceph-mon[41562]: 7.d scrub ok 2024-09-10T19:15:45.418 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:15:45 smithi183 ceph-mon[41562]: from='mgr.14556 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.11", "id": [3, 5]}]': finished 2024-09-10T19:15:45.418 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:15:45 smithi183 ceph-mon[41562]: from='mgr.14556 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.15", "id": [3, 2]}]': finished 2024-09-10T19:15:45.418 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:15:45 smithi183 ceph-mon[41562]: osdmap e143: 8 total, 8 up, 7 in 2024-09-10T19:15:45.418 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:15:45 smithi183 ceph-mon[41562]: 6.d scrub starts 2024-09-10T19:15:45.418 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:15:45 smithi183 ceph-mon[41562]: 6.d scrub ok 2024-09-10T19:15:45.418 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:15:45 smithi183 ceph-mon[41562]: from='client.? 172.21.15.183:0/3200058441' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-10T19:15:45.418 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:15:45 smithi183 ceph-mon[41562]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-10T19:15:45.433 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:15:45.454 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:15:45.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:15:45.455 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:15:45.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:15:45 smithi191 ceph-mon[32884]: pgmap v222: 148 pgs: 148 active+clean; 457 MiB data, 3.5 GiB used, 622 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 854 op/s 2024-09-10T19:15:45.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:15:45 smithi191 ceph-mon[32884]: 7.d scrub starts 2024-09-10T19:15:45.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:15:45 smithi191 ceph-mon[32884]: 7.d scrub ok 2024-09-10T19:15:45.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:15:45 smithi191 ceph-mon[32884]: from='mgr.14556 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.11", "id": [3, 5]}]': finished 2024-09-10T19:15:45.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:15:45 smithi191 ceph-mon[32884]: from='mgr.14556 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.15", "id": [3, 2]}]': finished 2024-09-10T19:15:45.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:15:45 smithi191 ceph-mon[32884]: osdmap e143: 8 total, 8 up, 7 in 2024-09-10T19:15:45.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:15:45 smithi191 ceph-mon[32884]: 6.d scrub starts 2024-09-10T19:15:45.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:15:45 smithi191 ceph-mon[32884]: 6.d scrub ok 2024-09-10T19:15:45.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:15:45 smithi191 ceph-mon[32884]: from='client.? 172.21.15.183:0/3200058441' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-10T19:15:45.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:15:45 smithi191 ceph-mon[32884]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-10T19:15:45.556 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:15:45.577 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:15:45.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:15:45.578 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:15:45.679 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:15:45.702 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:15:45.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:15:45.703 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:15:45.804 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:15:45.828 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:15:45.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:15:45.829 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:15:45.899 INFO:tasks.thrashosds.thrasher:in_osds: [0, 2, 3, 4, 5, 6, 7] out_osds: [1] dead_osds: [] live_osds: [0, 2, 3, 4, 5, 6, 7, 1] 2024-09-10T19:15:45.899 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-10T19:15:45.899 INFO:tasks.thrashosds.thrasher:Shrinking pool .mgr 2024-09-10T19:15:45.900 INFO:tasks.cephadm.ceph_manager.ceph:contract_pool .mgr by 10 min 8 2024-09-10T19:15:45.900 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph pg dump --format=json 2024-09-10T19:15:45.930 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:15:45.966 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:15:45.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:15:45.967 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:15:46.067 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:15:46.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:15:46.091 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:15:46.091 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:15:46.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:15:46 smithi183 ceph-mon[29736]: 6.12 scrub starts 2024-09-10T19:15:46.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:15:46 smithi183 ceph-mon[29736]: 6.12 scrub ok 2024-09-10T19:15:46.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:15:46 smithi183 ceph-mon[29736]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-10T19:15:46.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:15:46 smithi183 ceph-mon[29736]: Cluster is now healthy 2024-09-10T19:15:46.182 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:15:46 smithi183 ceph-mon[29736]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-10T19:15:46.182 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:15:46 smithi183 ceph-mon[29736]: osdmap e144: 8 total, 8 up, 7 in 2024-09-10T19:15:46.182 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:15:46 smithi183 ceph-mon[29736]: 6.19 scrub starts 2024-09-10T19:15:46.182 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:15:46 smithi183 ceph-mon[29736]: 6.19 scrub ok 2024-09-10T19:15:46.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:15:46 smithi183 ceph-mon[41562]: 6.12 scrub starts 2024-09-10T19:15:46.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:15:46 smithi183 ceph-mon[41562]: 6.12 scrub ok 2024-09-10T19:15:46.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:15:46 smithi183 ceph-mon[41562]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-10T19:15:46.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:15:46 smithi183 ceph-mon[41562]: Cluster is now healthy 2024-09-10T19:15:46.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:15:46 smithi183 ceph-mon[41562]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-10T19:15:46.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:15:46 smithi183 ceph-mon[41562]: osdmap e144: 8 total, 8 up, 7 in 2024-09-10T19:15:46.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:15:46 smithi183 ceph-mon[41562]: 6.19 scrub starts 2024-09-10T19:15:46.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:15:46 smithi183 ceph-mon[41562]: 6.19 scrub ok 2024-09-10T19:15:46.192 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:15:46.215 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:15:46.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:15:46.216 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:15:46.317 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:15:46.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:15:46.339 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:15:46.339 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:15:46.440 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:15:46.465 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:15:46 smithi191 ceph-mon[32884]: 6.12 scrub starts 2024-09-10T19:15:46.465 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:15:46 smithi191 ceph-mon[32884]: 6.12 scrub ok 2024-09-10T19:15:46.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:15:46 smithi191 ceph-mon[32884]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-10T19:15:46.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:15:46 smithi191 ceph-mon[32884]: Cluster is now healthy 2024-09-10T19:15:46.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:15:46 smithi191 ceph-mon[32884]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-10T19:15:46.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:15:46 smithi191 ceph-mon[32884]: osdmap e144: 8 total, 8 up, 7 in 2024-09-10T19:15:46.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:15:46 smithi191 ceph-mon[32884]: 6.19 scrub starts 2024-09-10T19:15:46.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:15:46 smithi191 ceph-mon[32884]: 6.19 scrub ok 2024-09-10T19:15:46.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:15:46.467 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:15:46.467 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:15:46.568 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:15:46.591 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:15:46.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:15:46.592 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:15:46.693 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:15:46.719 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:15:46.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:15:46.720 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:15:46.821 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:15:46.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:15:46.843 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:15:46.844 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:15:46.944 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:15:46.970 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:15:46.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:15:46.971 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:15:47.072 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:15:47.098 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:15:47.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:15:47.099 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:15:47.200 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:15:47.226 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:15:47.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:15:47.227 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:15:47.327 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:15:47.349 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:15:47.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:15:47.350 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:15:47.408 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:15:47 smithi183 ceph-mon[29736]: pgmap v225: 148 pgs: 2 peering, 146 active+clean; 34 MiB data, 2.2 GiB used, 624 GiB / 626 GiB avail; 0 B/s wr, 2.05k op/s 2024-09-10T19:15:47.408 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:15:47 smithi183 ceph-mon[29736]: 4.14 scrub starts 2024-09-10T19:15:47.408 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:15:47 smithi183 ceph-mon[29736]: 4.14 scrub ok 2024-09-10T19:15:47.409 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:15:47 smithi183 ceph-mon[41562]: pgmap v225: 148 pgs: 2 peering, 146 active+clean; 34 MiB data, 2.2 GiB used, 624 GiB / 626 GiB avail; 0 B/s wr, 2.05k op/s 2024-09-10T19:15:47.409 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:15:47 smithi183 ceph-mon[41562]: 4.14 scrub starts 2024-09-10T19:15:47.409 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:15:47 smithi183 ceph-mon[41562]: 4.14 scrub ok 2024-09-10T19:15:47.450 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:15:47 smithi191 ceph-mon[32884]: pgmap v225: 148 pgs: 2 peering, 146 active+clean; 34 MiB data, 2.2 GiB used, 624 GiB / 626 GiB avail; 0 B/s wr, 2.05k op/s 2024-09-10T19:15:47.450 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:15:47 smithi191 ceph-mon[32884]: 4.14 scrub starts 2024-09-10T19:15:47.450 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:15:47 smithi191 ceph-mon[32884]: 4.14 scrub ok 2024-09-10T19:15:47.451 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:15:47.476 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:15:47.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:15:47.477 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:15:47.578 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:15:47.599 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:15:47.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:15:47.600 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:15:47.702 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:15:47.715 INFO:journalctl@ceph.mgr.x.smithi191.stdout:Sep 10 19:15:47 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-x[34967]: ::ffff:172.21.15.191 - - [10/Sep/2024:19:15:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T19:15:47.724 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:15:47.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:15:47.725 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:15:47.765 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:15:47.826 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:15:47.851 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:15:47.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:15:47.852 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:15:47.953 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:15:47.975 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:15:47.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:15:47.976 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:15:48.076 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:15:48.102 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:15:48.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:15:48.103 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:15:48.204 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:15:48.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:15:48.226 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:15:48.226 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:15:48.318 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:15:48 smithi183 ceph-mon[29736]: 6.f scrub starts 2024-09-10T19:15:48.318 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:15:48 smithi183 ceph-mon[29736]: 6.f scrub ok 2024-09-10T19:15:48.318 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:15:48 smithi183 ceph-mon[29736]: 4.e scrub starts 2024-09-10T19:15:48.318 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:15:48 smithi183 ceph-mon[29736]: 4.e scrub ok 2024-09-10T19:15:48.318 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:15:48 smithi183 ceph-mon[41562]: 6.f scrub starts 2024-09-10T19:15:48.318 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:15:48 smithi183 ceph-mon[41562]: 6.f scrub ok 2024-09-10T19:15:48.318 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:15:48 smithi183 ceph-mon[41562]: 4.e scrub starts 2024-09-10T19:15:48.318 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:15:48 smithi183 ceph-mon[41562]: 4.e scrub ok 2024-09-10T19:15:48.327 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:15:48.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:15:48.349 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:15:48.349 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:15:48.450 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:15:48.465 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:15:48 smithi191 ceph-mon[32884]: 6.f scrub starts 2024-09-10T19:15:48.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:15:48 smithi191 ceph-mon[32884]: 6.f scrub ok 2024-09-10T19:15:48.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:15:48 smithi191 ceph-mon[32884]: 4.e scrub starts 2024-09-10T19:15:48.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:15:48 smithi191 ceph-mon[32884]: 4.e scrub ok 2024-09-10T19:15:48.472 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:15:48.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:15:48.473 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:15:48.574 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:15:48.600 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:15:48.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:15:48.601 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:15:48.702 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:15:48.724 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:15:48.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:15:48.725 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:15:48.826 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:15:48.849 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:15:48.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:15:48.850 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:15:48.951 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:15:48.955 INFO:teuthology.orchestra.run.smithi183.stdout:{ 2024-09-10T19:15:48.955 INFO:teuthology.orchestra.run.smithi183.stdout: "size": 20, 2024-09-10T19:15:48.955 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 600, 2024-09-10T19:15:48.955 INFO:teuthology.orchestra.run.smithi183.stdout: "ops": [ 2024-09-10T19:15:48.956 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:15:48.956 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.14781.0:16994 7.e 7:78896f87:::benchmark_data_smithi183_84160_object16993:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e90)", 2024-09-10T19:15:48.956 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-10T19:12:24.659701+0000", 2024-09-10T19:15:48.956 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 204.27831999099999, 2024-09-10T19:15:48.956 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.104560112, 2024-09-10T19:15:48.956 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-10T19:15:48.956 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T19:15:48.956 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-10T19:15:48.956 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.14781", 2024-09-10T19:15:48.956 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.183:0/2025776420", 2024-09-10T19:15:48.956 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 16994 2024-09-10T19:15:48.956 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:15:48.957 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-10T19:15:48.957 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:15:48.957 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-10T19:15:48.957 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:24.659701+0000", 2024-09-10T19:15:48.957 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:15:48.957 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:15:48.957 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:15:48.957 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-10T19:15:48.957 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:24.659701+0000", 2024-09-10T19:15:48.957 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:15:48.957 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:15:48.957 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:15:48.957 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-10T19:15:48.958 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:24.659700+0000", 2024-09-10T19:15:48.958 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4294967295.999999 2024-09-10T19:15:48.958 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:15:48.958 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:15:48.958 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-10T19:15:48.958 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:24.659737+0000", 2024-09-10T19:15:48.958 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.7348999999999998e-05 2024-09-10T19:15:48.958 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:15:48.958 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:15:48.958 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-10T19:15:48.958 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:24.659748+0000", 2024-09-10T19:15:48.958 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.1077000000000001e-05 2024-09-10T19:15:48.958 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:15:48.959 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:15:48.959 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-10T19:15:48.959 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:24.659755+0000", 2024-09-10T19:15:48.959 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.6980000000000001e-06 2024-09-10T19:15:48.959 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:15:48.959 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:15:48.959 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-10T19:15:48.959 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:24.661704+0000", 2024-09-10T19:15:48.959 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0019488050000000001 2024-09-10T19:15:48.959 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:15:48.959 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:15:48.959 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-10T19:15:48.959 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:24.661898+0000", 2024-09-10T19:15:48.959 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00019394799999999999 2024-09-10T19:15:48.960 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:15:48.960 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:15:48.960 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 0,5", 2024-09-10T19:15:48.960 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:24.662695+0000", 2024-09-10T19:15:48.960 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000796789 2024-09-10T19:15:48.960 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:15:48.960 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:15:48.960 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:15:48.960 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:24.663749+0000", 2024-09-10T19:15:48.960 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0010547950000000001 2024-09-10T19:15:48.960 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:15:48.960 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:15:48.961 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-10T19:15:48.961 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:24.663765+0000", 2024-09-10T19:15:48.961 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.5290000000000001e-05 2024-09-10T19:15:48.961 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:15:48.961 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:15:48.961 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:15:48.961 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:24.763895+0000", 2024-09-10T19:15:48.961 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.100130362 2024-09-10T19:15:48.961 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:15:48.961 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:15:48.961 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-10T19:15:48.961 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:24.764223+0000", 2024-09-10T19:15:48.961 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00032781500000000002 2024-09-10T19:15:48.962 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:15:48.962 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:15:48.962 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-10T19:15:48.962 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:24.764261+0000", 2024-09-10T19:15:48.962 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.8194000000000001e-05 2024-09-10T19:15:48.962 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:15:48.962 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:15:48.962 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:15:48.962 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:15:48.962 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:15:48.962 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.14781.0:23975 7.e 7:70e2a0a7:::benchmark_data_smithi183_84160_object23974:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e91)", 2024-09-10T19:15:48.962 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-10T19:12:27.633672+0000", 2024-09-10T19:15:48.962 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 201.30434937300001, 2024-09-10T19:15:48.962 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.097315150000000003, 2024-09-10T19:15:48.963 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-10T19:15:48.963 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T19:15:48.963 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-10T19:15:48.963 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.14781", 2024-09-10T19:15:48.963 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.183:0/2025776420", 2024-09-10T19:15:48.963 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 23975 2024-09-10T19:15:48.963 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:15:48.963 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-10T19:15:48.963 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:15:48.963 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-10T19:15:48.963 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:27.633672+0000", 2024-09-10T19:15:48.963 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:15:48.963 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:15:48.964 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:15:48.964 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-10T19:15:48.964 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:27.633672+0000", 2024-09-10T19:15:48.964 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:15:48.964 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:15:48.964 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:15:48.964 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-10T19:15:48.964 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:27.633670+0000", 2024-09-10T19:15:48.964 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4294967295.999999 2024-09-10T19:15:48.964 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:15:48.964 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:15:48.964 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-10T19:15:48.964 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:27.633692+0000", 2024-09-10T19:15:48.964 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.2078000000000001e-05 2024-09-10T19:15:48.965 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:15:48.965 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:15:48.965 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-10T19:15:48.965 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:27.633704+0000", 2024-09-10T19:15:48.965 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.1752e-05 2024-09-10T19:15:48.965 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:15:48.965 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:15:48.965 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-10T19:15:48.965 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:27.633710+0000", 2024-09-10T19:15:48.965 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.818e-06 2024-09-10T19:15:48.965 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:15:48.965 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:15:48.966 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-10T19:15:48.966 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:27.633749+0000", 2024-09-10T19:15:48.966 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.9110999999999999e-05 2024-09-10T19:15:48.966 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:15:48.966 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:15:48.966 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-10T19:15:48.966 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:27.633835+0000", 2024-09-10T19:15:48.966 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.6102999999999999e-05 2024-09-10T19:15:48.966 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:15:48.966 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:15:48.966 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 0,5", 2024-09-10T19:15:48.966 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:27.633958+0000", 2024-09-10T19:15:48.966 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00012266499999999999 2024-09-10T19:15:48.967 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:15:48.967 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:15:48.967 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-10T19:15:48.967 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:27.634762+0000", 2024-09-10T19:15:48.967 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00080392400000000002 2024-09-10T19:15:48.967 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:15:48.967 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:15:48.967 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:15:48.967 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:27.635184+0000", 2024-09-10T19:15:48.967 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00042250399999999998 2024-09-10T19:15:48.967 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:15:48.967 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:15:48.967 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:15:48.967 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:27.730851+0000", 2024-09-10T19:15:48.968 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.095666553000000001 2024-09-10T19:15:48.968 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:15:48.968 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:15:48.968 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-10T19:15:48.968 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:27.730948+0000", 2024-09-10T19:15:48.968 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.7418999999999997e-05 2024-09-10T19:15:48.968 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:15:48.968 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:15:48.968 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-10T19:15:48.968 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:27.730987+0000", 2024-09-10T19:15:48.968 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.837e-05 2024-09-10T19:15:48.968 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:15:48.968 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:15:48.968 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:15:48.969 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:15:48.969 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:15:48.969 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.14781.0:153221 7.8 7:172abbc3:::benchmark_data_smithi183_84160_object12840:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e118)", 2024-09-10T19:15:48.969 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-10T19:14:00.995085+0000", 2024-09-10T19:15:48.969 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 107.94293608300001, 2024-09-10T19:15:48.969 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.099271308000000003, 2024-09-10T19:15:48.969 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-10T19:15:48.969 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T19:15:48.969 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-10T19:15:48.969 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.14781", 2024-09-10T19:15:48.970 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.183:0/2025776420", 2024-09-10T19:15:48.970 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 153221 2024-09-10T19:15:48.970 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:15:48.970 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-10T19:15:48.970 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:15:48.970 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-10T19:15:48.970 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:00.995085+0000", 2024-09-10T19:15:48.971 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:15:48.971 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:15:48.971 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:15:48.971 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-10T19:15:48.971 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:00.995085+0000", 2024-09-10T19:15:48.971 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:15:48.971 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:15:48.971 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:15:48.971 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-10T19:15:48.971 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:00.995083+0000", 2024-09-10T19:15:48.971 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4294967295.9999986 2024-09-10T19:15:48.971 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:15:48.971 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:15:48.971 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-10T19:15:48.972 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:00.995090+0000", 2024-09-10T19:15:48.972 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.0979999999999996e-06 2024-09-10T19:15:48.972 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:15:48.972 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:15:48.972 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-10T19:15:48.972 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:00.995107+0000", 2024-09-10T19:15:48.972 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.7221999999999999e-05 2024-09-10T19:15:48.972 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:15:48.972 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:15:48.972 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-10T19:15:48.972 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:00.995113+0000", 2024-09-10T19:15:48.972 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.8289999999999996e-06 2024-09-10T19:15:48.972 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:15:48.973 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:15:48.973 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-10T19:15:48.973 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:00.995155+0000", 2024-09-10T19:15:48.973 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.2175000000000002e-05 2024-09-10T19:15:48.973 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:15:48.973 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:15:48.973 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-10T19:15:48.973 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:00.995264+0000", 2024-09-10T19:15:48.973 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000108748 2024-09-10T19:15:48.973 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:15:48.973 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:15:48.973 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 4,7", 2024-09-10T19:15:48.973 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:00.995372+0000", 2024-09-10T19:15:48.974 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000107908 2024-09-10T19:15:48.974 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:15:48.974 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:15:48.974 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-10T19:15:48.974 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:00.999702+0000", 2024-09-10T19:15:48.974 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0043296580000000001 2024-09-10T19:15:48.974 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:15:48.974 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:15:48.974 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:15:48.974 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:01.003291+0000", 2024-09-10T19:15:48.974 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0035892910000000001 2024-09-10T19:15:48.974 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:15:48.974 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:15:48.974 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:15:48.974 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:01.094272+0000", 2024-09-10T19:15:48.974 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.090981399000000004 2024-09-10T19:15:48.975 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:15:48.975 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:15:48.975 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-10T19:15:48.975 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:01.094320+0000", 2024-09-10T19:15:48.975 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.7091e-05 2024-09-10T19:15:48.975 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:15:48.975 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:15:48.975 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-10T19:15:48.975 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:01.094356+0000", 2024-09-10T19:15:48.975 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.6526000000000002e-05 2024-09-10T19:15:48.975 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:15:48.975 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:15:48.975 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:15:48.975 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:15:48.976 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:15:48.976 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.14781.0:161349 7.e 7:7d3e032d:::benchmark_data_smithi183_84160_object20968:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120)", 2024-09-10T19:15:48.976 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-10T19:14:07.731291+0000", 2024-09-10T19:15:48.976 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 101.20672965, 2024-09-10T19:15:48.976 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.13911137500000001, 2024-09-10T19:15:48.976 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-10T19:15:48.976 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T19:15:48.976 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-10T19:15:48.976 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.14781", 2024-09-10T19:15:48.976 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.183:0/2025776420", 2024-09-10T19:15:48.976 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 161349 2024-09-10T19:15:48.976 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:15:48.976 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-10T19:15:48.976 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:15:48.977 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-10T19:15:48.977 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:07.731291+0000", 2024-09-10T19:15:48.977 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:15:48.977 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:15:48.977 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:15:48.977 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-10T19:15:48.977 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:07.731291+0000", 2024-09-10T19:15:48.977 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:15:48.977 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:15:48.977 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:15:48.977 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-10T19:15:48.977 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:07.731290+0000", 2024-09-10T19:15:48.977 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4294967295.9999986 2024-09-10T19:15:48.977 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:15:48.978 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:15:48.978 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-10T19:15:48.978 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:07.731303+0000", 2024-09-10T19:15:48.978 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.2734e-05 2024-09-10T19:15:48.978 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:15:48.978 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:15:48.978 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-10T19:15:48.978 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:07.731315+0000", 2024-09-10T19:15:48.978 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.2208999999999999e-05 2024-09-10T19:15:48.978 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:15:48.978 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:15:48.978 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-10T19:15:48.978 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:07.731321+0000", 2024-09-10T19:15:48.979 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.0580000000000003e-06 2024-09-10T19:15:48.979 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:15:48.979 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:15:48.979 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-10T19:15:48.979 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:07.732577+0000", 2024-09-10T19:15:48.979 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0012556690000000001 2024-09-10T19:15:48.979 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:15:48.979 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:15:48.979 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-10T19:15:48.979 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:07.732686+0000", 2024-09-10T19:15:48.979 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00010967300000000001 2024-09-10T19:15:48.979 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:15:48.979 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:15:48.979 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 0,5", 2024-09-10T19:15:48.980 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:07.732792+0000", 2024-09-10T19:15:48.980 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000106204 2024-09-10T19:15:48.980 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:15:48.980 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:15:48.980 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:15:48.980 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:07.739790+0000", 2024-09-10T19:15:48.980 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0069971460000000001 2024-09-10T19:15:48.980 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:15:48.980 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:15:48.980 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-10T19:15:48.980 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:07.739913+0000", 2024-09-10T19:15:48.980 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00012379599999999999 2024-09-10T19:15:48.980 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:15:48.980 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:15:48.981 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:15:48.981 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:07.870322+0000", 2024-09-10T19:15:48.981 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.13040858499999999 2024-09-10T19:15:48.981 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:15:48.981 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:15:48.981 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-10T19:15:48.981 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:07.870367+0000", 2024-09-10T19:15:48.981 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.4666999999999998e-05 2024-09-10T19:15:48.981 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:15:48.981 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:15:48.981 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-10T19:15:48.981 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:07.870403+0000", 2024-09-10T19:15:48.981 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.5917999999999997e-05 2024-09-10T19:15:48.981 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:15:48.982 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:15:48.982 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:15:48.982 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:15:48.982 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:15:48.982 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.14781.0:161353 7.e 7:7f1eb1cd:::benchmark_data_smithi183_84160_object20972:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120)", 2024-09-10T19:15:48.982 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-10T19:14:07.736149+0000", 2024-09-10T19:15:48.982 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 101.201872079, 2024-09-10T19:15:48.982 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.134348467, 2024-09-10T19:15:48.982 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-10T19:15:48.982 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T19:15:48.982 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-10T19:15:48.982 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.14781", 2024-09-10T19:15:48.982 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.183:0/2025776420", 2024-09-10T19:15:48.982 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 161353 2024-09-10T19:15:48.982 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:15:48.983 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-10T19:15:48.983 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:15:48.983 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-10T19:15:48.983 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:07.736149+0000", 2024-09-10T19:15:48.983 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:15:48.983 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:15:48.983 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:15:48.983 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-10T19:15:48.983 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:07.736149+0000", 2024-09-10T19:15:48.983 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:15:48.983 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:15:48.983 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:15:48.983 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-10T19:15:48.984 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:07.736147+0000", 2024-09-10T19:15:48.984 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4294967295.9999981 2024-09-10T19:15:48.984 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:15:48.984 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:15:48.984 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-10T19:15:48.984 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:07.736154+0000", 2024-09-10T19:15:48.984 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.9949999999999999e-06 2024-09-10T19:15:48.984 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:15:48.984 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:15:48.984 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-10T19:15:48.984 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:07.736170+0000", 2024-09-10T19:15:48.984 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.6535999999999999e-05 2024-09-10T19:15:48.984 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:15:48.985 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:15:48.985 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-10T19:15:48.985 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:07.736177+0000", 2024-09-10T19:15:48.985 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.3860000000000004e-06 2024-09-10T19:15:48.985 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:15:48.985 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:15:48.985 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-10T19:15:48.985 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:07.736348+0000", 2024-09-10T19:15:48.985 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000171053 2024-09-10T19:15:48.985 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:15:48.985 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:15:48.986 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-10T19:15:48.986 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:07.736451+0000", 2024-09-10T19:15:48.986 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000102731 2024-09-10T19:15:48.986 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:15:48.986 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:15:48.986 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 0,5", 2024-09-10T19:15:48.986 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:07.736555+0000", 2024-09-10T19:15:48.986 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000104147 2024-09-10T19:15:48.986 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:15:48.986 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:15:48.986 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:15:48.987 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:07.739816+0000", 2024-09-10T19:15:48.987 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.003260752 2024-09-10T19:15:48.987 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:15:48.987 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:15:48.987 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-10T19:15:48.987 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:07.740221+0000", 2024-09-10T19:15:48.987 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00040548299999999999 2024-09-10T19:15:48.987 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:15:48.987 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:15:48.987 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:15:48.987 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:07.870433+0000", 2024-09-10T19:15:48.988 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.13021206799999999 2024-09-10T19:15:48.988 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:15:48.988 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:15:48.988 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-10T19:15:48.988 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:07.870467+0000", 2024-09-10T19:15:48.988 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.4207e-05 2024-09-10T19:15:48.988 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:15:48.988 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:15:48.988 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-10T19:15:48.988 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:07.870497+0000", 2024-09-10T19:15:48.988 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.9977000000000001e-05 2024-09-10T19:15:48.988 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:15:48.988 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:15:48.988 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:15:48.988 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:15:48.989 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:15:48.989 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.14781.0:161359 7.e 7:7346129c:::benchmark_data_smithi183_84160_object20978:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120)", 2024-09-10T19:15:48.989 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-10T19:14:07.740514+0000", 2024-09-10T19:15:48.989 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 101.197506541, 2024-09-10T19:15:48.989 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.136079479, 2024-09-10T19:15:48.989 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-10T19:15:48.989 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T19:15:48.989 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-10T19:15:48.989 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.14781", 2024-09-10T19:15:48.989 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.183:0/2025776420", 2024-09-10T19:15:48.989 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 161359 2024-09-10T19:15:48.989 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:15:48.989 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-10T19:15:48.990 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:15:48.990 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-10T19:15:48.990 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:07.740514+0000", 2024-09-10T19:15:48.990 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:15:48.990 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:15:48.990 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:15:48.990 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-10T19:15:48.990 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:07.740514+0000", 2024-09-10T19:15:48.990 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:15:48.990 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:15:48.990 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:15:48.990 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-10T19:15:48.990 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:07.740513+0000", 2024-09-10T19:15:48.990 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4294967295.9999986 2024-09-10T19:15:48.990 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:15:48.991 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:15:48.991 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-10T19:15:48.991 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:07.740519+0000", 2024-09-10T19:15:48.991 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.0610000000000001e-06 2024-09-10T19:15:48.991 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:15:48.991 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:15:48.991 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-10T19:15:48.991 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:07.740530+0000", 2024-09-10T19:15:48.991 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.0874e-05 2024-09-10T19:15:48.991 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:15:48.991 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:15:48.991 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-10T19:15:48.991 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:07.740535+0000", 2024-09-10T19:15:48.991 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.4439999999999999e-06 2024-09-10T19:15:48.991 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:15:48.992 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:15:48.992 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-10T19:15:48.992 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:07.740673+0000", 2024-09-10T19:15:48.992 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00013725900000000001 2024-09-10T19:15:48.992 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:15:48.992 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:15:48.992 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-10T19:15:48.992 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:07.740773+0000", 2024-09-10T19:15:48.992 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000100739 2024-09-10T19:15:48.992 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:15:48.992 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:15:48.992 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 0,5", 2024-09-10T19:15:48.992 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:07.740878+0000", 2024-09-10T19:15:48.992 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00010482800000000001 2024-09-10T19:15:48.992 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:15:48.993 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:15:48.993 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:15:48.993 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:07.745774+0000", 2024-09-10T19:15:48.993 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0048961550000000001 2024-09-10T19:15:48.993 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:15:48.993 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:15:48.993 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-10T19:15:48.993 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:07.745798+0000", 2024-09-10T19:15:48.993 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.3125e-05 2024-09-10T19:15:48.993 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:15:48.993 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:15:48.993 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:15:48.993 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:07.876512+0000", 2024-09-10T19:15:48.993 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.130714624 2024-09-10T19:15:48.994 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:15:48.994 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:15:48.994 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-10T19:15:48.994 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:07.876558+0000", 2024-09-10T19:15:48.994 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.6252999999999999e-05 2024-09-10T19:15:48.994 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:15:48.994 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:15:48.994 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-10T19:15:48.994 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:07.876594+0000", 2024-09-10T19:15:48.994 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.5422000000000001e-05 2024-09-10T19:15:48.994 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:15:48.994 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:15:48.995 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:15:48.995 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:15:48.995 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:15:48.995 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.14781.0:161362 7.e 7:736c80f1:::benchmark_data_smithi183_84160_object20981:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120)", 2024-09-10T19:15:48.995 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-10T19:14:07.743069+0000", 2024-09-10T19:15:48.995 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 101.194951612, 2024-09-10T19:15:48.995 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.13626217800000001, 2024-09-10T19:15:48.995 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-10T19:15:48.995 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T19:15:48.995 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-10T19:15:48.995 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.14781", 2024-09-10T19:15:48.995 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.183:0/2025776420", 2024-09-10T19:15:48.995 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 161362 2024-09-10T19:15:48.995 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:15:48.996 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-10T19:15:48.996 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:15:48.996 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-10T19:15:48.996 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:07.743069+0000", 2024-09-10T19:15:48.996 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:15:48.996 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:15:48.996 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:15:48.996 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-10T19:15:48.996 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:07.743069+0000", 2024-09-10T19:15:48.997 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:15:48.997 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:15:48.997 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:15:48.997 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-10T19:15:48.997 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:07.743068+0000", 2024-09-10T19:15:48.997 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4294967295.9999986 2024-09-10T19:15:48.997 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:15:48.997 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:15:48.997 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-10T19:15:48.997 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:07.743075+0000", 2024-09-10T19:15:48.997 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.9500000000000004e-06 2024-09-10T19:15:48.997 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:15:48.997 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:15:48.997 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-10T19:15:48.998 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:07.743088+0000", 2024-09-10T19:15:48.998 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.3112000000000001e-05 2024-09-10T19:15:48.998 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:15:48.998 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:15:48.998 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-10T19:15:48.998 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:07.743095+0000", 2024-09-10T19:15:48.998 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.703e-06 2024-09-10T19:15:48.998 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:15:48.998 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:15:48.998 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-10T19:15:48.998 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:07.743684+0000", 2024-09-10T19:15:48.998 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00058946400000000005 2024-09-10T19:15:48.998 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:15:48.998 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:15:48.998 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-10T19:15:48.999 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:07.743755+0000", 2024-09-10T19:15:48.999 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.1375e-05 2024-09-10T19:15:48.999 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:15:48.999 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:15:48.999 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 0,5", 2024-09-10T19:15:48.999 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:07.743822+0000", 2024-09-10T19:15:48.999 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.6696e-05 2024-09-10T19:15:48.999 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:15:48.999 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:15:48.999 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:15:48.999 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:07.745791+0000", 2024-09-10T19:15:48.999 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0019687049999999998 2024-09-10T19:15:48.999 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:15:49.000 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:15:49.000 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-10T19:15:49.000 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:07.745800+0000", 2024-09-10T19:15:49.000 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.6129999999999993e-06 2024-09-10T19:15:49.000 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:15:49.000 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:15:49.000 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:15:49.000 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:07.879266+0000", 2024-09-10T19:15:49.000 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.133465363 2024-09-10T19:15:49.000 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:15:49.000 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:15:49.000 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-10T19:15:49.000 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:07.879302+0000", 2024-09-10T19:15:49.001 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.5754999999999999e-05 2024-09-10T19:15:49.001 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:15:49.001 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:15:49.001 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-10T19:15:49.001 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:07.879331+0000", 2024-09-10T19:15:49.001 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.9904e-05 2024-09-10T19:15:49.001 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:15:49.001 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:15:49.001 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:15:49.001 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:15:49.001 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:15:49.001 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "MOSDPGPush(7.b 120/116 [PushOp(7:d3313432:::benchmark_data_smithi183_84160_object50232:head, version: 96'3036, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(7:d3313432:::benchmark_data_smithi183_84160_object50232:head@96'3036, size: 65536, copy_subset: [0~65536], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:65536, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-10T19:15:49.001 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-10T19:14:07.749410+0000", 2024-09-10T19:15:49.001 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 101.188611122, 2024-09-10T19:15:49.002 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.120505768, 2024-09-10T19:15:49.002 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-10T19:15:49.002 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "started", 2024-09-10T19:15:49.002 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-10T19:15:49.002 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:15:49.002 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-10T19:15:49.002 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:07.749410+0000", 2024-09-10T19:15:49.002 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:15:49.002 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:15:49.002 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:15:49.002 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-10T19:15:49.002 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:07.749410+0000", 2024-09-10T19:15:49.002 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:15:49.003 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:15:49.003 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:15:49.003 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-10T19:15:49.003 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:07.749409+0000", 2024-09-10T19:15:49.003 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4294967295.9999995 2024-09-10T19:15:49.003 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:15:49.003 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:15:49.003 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-10T19:15:49.003 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:07.749556+0000", 2024-09-10T19:15:49.003 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00014702399999999999 2024-09-10T19:15:49.003 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:15:49.003 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:15:49.003 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-10T19:15:49.003 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:07.749567+0000", 2024-09-10T19:15:49.004 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.1066999999999999e-05 2024-09-10T19:15:49.004 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:15:49.004 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:15:49.004 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-10T19:15:49.004 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:07.749573+0000", 2024-09-10T19:15:49.004 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.5849999999999999e-06 2024-09-10T19:15:49.004 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:15:49.004 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:15:49.004 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-10T19:15:49.004 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:07.869617+0000", 2024-09-10T19:15:49.004 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.120043623 2024-09-10T19:15:49.004 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:15:49.004 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:15:49.005 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-10T19:15:49.005 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:07.869626+0000", 2024-09-10T19:15:49.005 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.6889999999999999e-06 2024-09-10T19:15:49.005 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:15:49.005 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:15:49.005 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-10T19:15:49.005 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:07.869916+0000", 2024-09-10T19:15:49.005 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00028926400000000001 2024-09-10T19:15:49.005 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:15:49.005 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:15:49.005 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:15:49.005 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:15:49.005 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:15:49.005 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "pg_backfill(progress 7.b e 120/120 lb 7:d330ac6a:::benchmark_data_smithi183_84160_object135588:head)", 2024-09-10T19:15:49.006 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-10T19:14:07.749584+0000", 2024-09-10T19:15:49.006 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 101.188436624, 2024-09-10T19:15:49.006 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.12054490599999999, 2024-09-10T19:15:49.006 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-10T19:15:49.006 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "started", 2024-09-10T19:15:49.006 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-10T19:15:49.006 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:15:49.006 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-10T19:15:49.006 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:07.749584+0000", 2024-09-10T19:15:49.006 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:15:49.006 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:15:49.006 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:15:49.006 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-10T19:15:49.007 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:07.749584+0000", 2024-09-10T19:15:49.007 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:15:49.007 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:15:49.007 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:15:49.007 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-10T19:15:49.007 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:07.749584+0000", 2024-09-10T19:15:49.007 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4294967295.9999995 2024-09-10T19:15:49.007 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:15:49.007 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:15:49.007 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-10T19:15:49.007 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:07.749589+0000", 2024-09-10T19:15:49.007 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.6879999999999998e-06 2024-09-10T19:15:49.007 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:15:49.007 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:15:49.008 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-10T19:15:49.008 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:07.749593+0000", 2024-09-10T19:15:49.008 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.8630000000000001e-06 2024-09-10T19:15:49.008 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:15:49.008 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:15:49.008 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-10T19:15:49.008 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:07.869569+0000", 2024-09-10T19:15:49.008 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.11997672299999999 2024-09-10T19:15:49.008 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:15:49.008 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:15:49.008 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-10T19:15:49.008 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:07.870077+0000", 2024-09-10T19:15:49.008 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00050739600000000002 2024-09-10T19:15:49.008 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:15:49.009 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:15:49.009 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-10T19:15:49.009 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:07.870093+0000", 2024-09-10T19:15:49.009 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.5860000000000001e-05 2024-09-10T19:15:49.009 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:15:49.009 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:15:49.009 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-10T19:15:49.009 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:07.870129+0000", 2024-09-10T19:15:49.009 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.6636e-05 2024-09-10T19:15:49.009 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:15:49.009 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:15:49.009 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:15:49.009 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:15:49.009 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:15:49.010 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.14781.0:161675 7.8 7:1d0bcc03:::benchmark_data_smithi183_84160_object21294:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120)", 2024-09-10T19:15:49.010 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-10T19:14:08.092546+0000", 2024-09-10T19:15:49.010 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 100.84547528500001, 2024-09-10T19:15:49.010 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.143957107, 2024-09-10T19:15:49.010 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-10T19:15:49.010 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T19:15:49.010 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-10T19:15:49.010 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.14781", 2024-09-10T19:15:49.010 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.183:0/2025776420", 2024-09-10T19:15:49.010 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 161675 2024-09-10T19:15:49.010 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:15:49.010 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-10T19:15:49.010 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:15:49.010 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-10T19:15:49.010 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:08.092546+0000", 2024-09-10T19:15:49.011 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:15:49.011 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:15:49.011 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:15:49.011 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-10T19:15:49.011 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:08.092546+0000", 2024-09-10T19:15:49.011 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:15:49.011 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:15:49.011 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:15:49.011 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-10T19:15:49.011 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:08.092544+0000", 2024-09-10T19:15:49.011 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4294967295.9999986 2024-09-10T19:15:49.011 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:15:49.011 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:15:49.011 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-10T19:15:49.011 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:08.092551+0000", 2024-09-10T19:15:49.012 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.4289999999999998e-06 2024-09-10T19:15:49.012 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:15:49.012 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:15:49.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:15:49.013 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-10T19:15:49.014 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:08.092563+0000", 2024-09-10T19:15:49.014 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.22e-05 2024-09-10T19:15:49.014 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:15:49.014 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:15:49.014 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-10T19:15:49.014 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:08.092569+0000", 2024-09-10T19:15:49.014 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.1700000000000002e-06 2024-09-10T19:15:49.014 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:15:49.014 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:15:49.014 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-10T19:15:49.014 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:08.094292+0000", 2024-09-10T19:15:49.014 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.001722433 2024-09-10T19:15:49.014 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:15:49.015 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:15:49.015 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-10T19:15:49.015 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:08.094446+0000", 2024-09-10T19:15:49.015 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00015463900000000001 2024-09-10T19:15:49.015 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:15:49.015 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:15:49.015 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 4,7", 2024-09-10T19:15:49.015 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:08.094552+0000", 2024-09-10T19:15:49.015 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00010600899999999999 2024-09-10T19:15:49.015 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:15:49.015 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:15:49.015 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:15:49.015 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:08.233696+0000", 2024-09-10T19:15:49.015 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.139144136 2024-09-10T19:15:49.015 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:15:49.016 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:15:49.016 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:15:49.016 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:08.233769+0000", 2024-09-10T19:15:49.016 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.2881999999999998e-05 2024-09-10T19:15:49.016 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:15:49.016 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:15:49.016 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-10T19:15:49.016 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:08.236418+0000", 2024-09-10T19:15:49.016 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.002648994 2024-09-10T19:15:49.016 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:15:49.016 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:15:49.016 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-10T19:15:49.016 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:08.236467+0000", 2024-09-10T19:15:49.016 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.8477999999999999e-05 2024-09-10T19:15:49.016 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:15:49.017 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:15:49.017 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-10T19:15:49.017 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:08.236503+0000", 2024-09-10T19:15:49.017 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.6031000000000001e-05 2024-09-10T19:15:49.017 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:15:49.017 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:15:49.017 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:15:49.017 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:15:49.017 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:15:49.017 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.14781.0:161676 7.8 7:1830f4cb:::benchmark_data_smithi183_84160_object21295:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120)", 2024-09-10T19:15:49.017 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-10T19:14:08.092580+0000", 2024-09-10T19:15:49.017 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 100.845440501, 2024-09-10T19:15:49.017 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.146075545, 2024-09-10T19:15:49.018 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-10T19:15:49.018 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T19:15:49.018 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-10T19:15:49.018 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.14781", 2024-09-10T19:15:49.018 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.183:0/2025776420", 2024-09-10T19:15:49.018 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 161676 2024-09-10T19:15:49.018 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:15:49.018 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-10T19:15:49.018 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:15:49.018 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-10T19:15:49.018 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:08.092580+0000", 2024-09-10T19:15:49.018 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:15:49.018 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:15:49.018 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:15:49.018 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-10T19:15:49.019 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:08.092580+0000", 2024-09-10T19:15:49.019 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:15:49.019 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:15:49.019 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:15:49.019 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-10T19:15:49.019 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:08.092580+0000", 2024-09-10T19:15:49.019 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4294967295.9999995 2024-09-10T19:15:49.019 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:15:49.019 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:15:49.019 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-10T19:15:49.019 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:08.092583+0000", 2024-09-10T19:15:49.019 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.2669999999999998e-06 2024-09-10T19:15:49.019 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:15:49.019 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:15:49.019 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-10T19:15:49.020 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:08.092590+0000", 2024-09-10T19:15:49.020 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.6630000000000004e-06 2024-09-10T19:15:49.020 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:15:49.020 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:15:49.020 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-10T19:15:49.020 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:08.092594+0000", 2024-09-10T19:15:49.020 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.3279999999999999e-06 2024-09-10T19:15:49.020 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:15:49.020 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:15:49.020 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-10T19:15:49.020 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:08.097367+0000", 2024-09-10T19:15:49.020 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0047731190000000001 2024-09-10T19:15:49.020 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:15:49.020 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:15:49.021 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-10T19:15:49.021 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:08.097460+0000", 2024-09-10T19:15:49.021 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.2694000000000004e-05 2024-09-10T19:15:49.021 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:15:49.021 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:15:49.021 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 4,7", 2024-09-10T19:15:49.021 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:08.097635+0000", 2024-09-10T19:15:49.021 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00017461899999999999 2024-09-10T19:15:49.021 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:15:49.021 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:15:49.021 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:15:49.021 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:08.233732+0000", 2024-09-10T19:15:49.021 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.13609732299999999 2024-09-10T19:15:49.021 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:15:49.021 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:15:49.022 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:15:49.022 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:08.233815+0000", 2024-09-10T19:15:49.022 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.2949999999999997e-05 2024-09-10T19:15:49.022 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:15:49.022 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:15:49.022 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-10T19:15:49.022 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:08.238577+0000", 2024-09-10T19:15:49.022 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0047622319999999999 2024-09-10T19:15:49.022 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:15:49.022 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:15:49.022 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-10T19:15:49.022 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:08.238623+0000", 2024-09-10T19:15:49.022 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.5930999999999998e-05 2024-09-10T19:15:49.022 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:15:49.023 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:15:49.023 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-10T19:15:49.023 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:08.238656+0000", 2024-09-10T19:15:49.023 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.2926999999999998e-05 2024-09-10T19:15:49.023 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:15:49.023 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:15:49.023 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:15:49.023 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:15:49.023 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:15:49.023 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.14781.0:161680 7.8 7:13ecf742:::benchmark_data_smithi183_84160_object21299:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120)", 2024-09-10T19:15:49.023 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-10T19:14:08.092602+0000", 2024-09-10T19:15:49.023 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 100.84541853899999, 2024-09-10T19:15:49.023 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.16144267000000001, 2024-09-10T19:15:49.023 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-10T19:15:49.024 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T19:15:49.024 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-10T19:15:49.024 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.14781", 2024-09-10T19:15:49.024 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.183:0/2025776420", 2024-09-10T19:15:49.024 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 161680 2024-09-10T19:15:49.024 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:15:49.024 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-10T19:15:49.024 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:15:49.024 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-10T19:15:49.024 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:08.092602+0000", 2024-09-10T19:15:49.024 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:15:49.024 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:15:49.024 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:15:49.025 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-10T19:15:49.025 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:08.092602+0000", 2024-09-10T19:15:49.025 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:15:49.025 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:15:49.025 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:15:49.025 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-10T19:15:49.025 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:08.092602+0000", 2024-09-10T19:15:49.025 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4294967295.9999995 2024-09-10T19:15:49.025 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:15:49.025 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:15:49.025 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-10T19:15:49.025 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:08.092605+0000", 2024-09-10T19:15:49.025 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.7690000000000001e-06 2024-09-10T19:15:49.026 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:15:49.026 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:15:49.026 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-10T19:15:49.026 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:08.092611+0000", 2024-09-10T19:15:49.026 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.1689999999999997e-06 2024-09-10T19:15:49.026 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:15:49.026 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:15:49.026 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-10T19:15:49.026 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:08.092615+0000", 2024-09-10T19:15:49.026 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.6990000000000001e-06 2024-09-10T19:15:49.026 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:15:49.026 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:15:49.026 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-10T19:15:49.026 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:08.244275+0000", 2024-09-10T19:15:49.026 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.15166089299999999 2024-09-10T19:15:49.026 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:15:49.027 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:15:49.027 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-10T19:15:49.027 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:08.244378+0000", 2024-09-10T19:15:49.027 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000102601 2024-09-10T19:15:49.027 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:15:49.027 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:15:49.027 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 4,7", 2024-09-10T19:15:49.027 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:08.244480+0000", 2024-09-10T19:15:49.027 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00010226 2024-09-10T19:15:49.027 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:15:49.027 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:15:49.027 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:15:49.027 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:08.253249+0000", 2024-09-10T19:15:49.027 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0087684270000000005 2024-09-10T19:15:49.027 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:15:49.028 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:15:49.028 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:15:49.028 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:08.253282+0000", 2024-09-10T19:15:49.028 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.2950999999999997e-05 2024-09-10T19:15:49.028 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:15:49.028 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:15:49.028 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-10T19:15:49.028 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:08.253963+0000", 2024-09-10T19:15:49.028 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00068178100000000001 2024-09-10T19:15:49.028 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:15:49.028 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:15:49.028 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-10T19:15:49.028 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:08.254009+0000", 2024-09-10T19:15:49.028 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.5176000000000003e-05 2024-09-10T19:15:49.029 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:15:49.029 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:15:49.029 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-10T19:15:49.029 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:08.254045+0000", 2024-09-10T19:15:49.029 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.6414000000000001e-05 2024-09-10T19:15:49.029 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:15:49.029 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:15:49.029 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:15:49.029 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:15:49.029 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:15:49.029 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_repop(client.14781.0:161681 7.0 e120/116 7:0fcfee5d:::benchmark_data_smithi183_84160_object21300:head v 120'10095, mlcod=120'10093)", 2024-09-10T19:15:49.029 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-10T19:14:08.095281+0000", 2024-09-10T19:15:49.029 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 100.84273983, 2024-09-10T19:15:49.029 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.14947492600000001, 2024-09-10T19:15:49.030 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-10T19:15:49.030 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T19:15:49.030 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-10T19:15:49.030 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:15:49.030 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-10T19:15:49.030 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:08.095281+0000", 2024-09-10T19:15:49.030 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:15:49.030 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:15:49.030 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:15:49.030 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-10T19:15:49.030 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:08.095281+0000", 2024-09-10T19:15:49.030 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:15:49.030 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:15:49.030 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:15:49.030 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-10T19:15:49.031 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:08.095280+0000", 2024-09-10T19:15:49.031 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4294967295.9999995 2024-09-10T19:15:49.031 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:15:49.031 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:15:49.031 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-10T19:15:49.031 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:08.095286+0000", 2024-09-10T19:15:49.031 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.1619999999999996e-06 2024-09-10T19:15:49.031 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:15:49.031 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:15:49.031 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-10T19:15:49.031 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:08.095297+0000", 2024-09-10T19:15:49.031 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.0651e-05 2024-09-10T19:15:49.031 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:15:49.031 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:15:49.031 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-10T19:15:49.032 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:08.095302+0000", 2024-09-10T19:15:49.032 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.057e-06 2024-09-10T19:15:49.032 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:15:49.032 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:15:49.032 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-10T19:15:49.032 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:08.097238+0000", 2024-09-10T19:15:49.032 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.001935888 2024-09-10T19:15:49.032 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:15:49.032 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:15:49.032 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-10T19:15:49.032 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:08.097255+0000", 2024-09-10T19:15:49.032 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.7133999999999999e-05 2024-09-10T19:15:49.032 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:15:49.032 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:15:49.032 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-10T19:15:49.033 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:08.244719+0000", 2024-09-10T19:15:49.033 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.14746366 2024-09-10T19:15:49.033 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:15:49.033 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:15:49.033 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-10T19:15:49.033 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:08.244756+0000", 2024-09-10T19:15:49.033 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.7085999999999997e-05 2024-09-10T19:15:49.033 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:15:49.033 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:15:49.033 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:15:49.033 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:15:49.033 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:15:49.033 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.14781.0:161688 7.e 7:76198a01:::benchmark_data_smithi183_84160_object21307:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120)", 2024-09-10T19:15:49.033 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-10T19:14:08.097940+0000", 2024-09-10T19:15:49.034 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 100.840081048, 2024-09-10T19:15:49.034 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.144387614, 2024-09-10T19:15:49.034 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-10T19:15:49.034 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T19:15:49.034 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-10T19:15:49.034 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.14781", 2024-09-10T19:15:49.034 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.183:0/2025776420", 2024-09-10T19:15:49.034 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 161688 2024-09-10T19:15:49.034 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:15:49.034 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-10T19:15:49.034 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:15:49.034 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-10T19:15:49.034 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:08.097940+0000", 2024-09-10T19:15:49.034 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:15:49.035 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:15:49.035 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:15:49.035 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-10T19:15:49.035 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:08.097940+0000", 2024-09-10T19:15:49.035 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:15:49.035 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:15:49.035 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:15:49.035 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-10T19:15:49.035 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:08.097939+0000", 2024-09-10T19:15:49.035 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4294967295.9999986 2024-09-10T19:15:49.035 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:15:49.035 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:15:49.035 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-10T19:15:49.035 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:08.097945+0000", 2024-09-10T19:15:49.035 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.6259999999999997e-06 2024-09-10T19:15:49.036 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:15:49.036 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:15:49.036 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-10T19:15:49.036 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:08.097956+0000", 2024-09-10T19:15:49.036 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.0625000000000001e-05 2024-09-10T19:15:49.036 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:15:49.036 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:15:49.036 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-10T19:15:49.036 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:08.097961+0000", 2024-09-10T19:15:49.036 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.5960000000000003e-06 2024-09-10T19:15:49.036 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:15:49.036 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:15:49.036 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-10T19:15:49.037 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:08.098002+0000", 2024-09-10T19:15:49.037 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.0593999999999998e-05 2024-09-10T19:15:49.037 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:15:49.037 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:15:49.037 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-10T19:15:49.037 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:08.098138+0000", 2024-09-10T19:15:49.037 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00013619500000000001 2024-09-10T19:15:49.037 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:15:49.037 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:15:49.037 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 0,5", 2024-09-10T19:15:49.037 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:08.098362+0000", 2024-09-10T19:15:49.037 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00022352299999999999 2024-09-10T19:15:49.037 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:15:49.037 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:15:49.038 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:15:49.038 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:08.233903+0000", 2024-09-10T19:15:49.038 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.13554107600000001 2024-09-10T19:15:49.038 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:15:49.038 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:15:49.038 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-10T19:15:49.038 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:08.233907+0000", 2024-09-10T19:15:49.038 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.2150000000000003e-06 2024-09-10T19:15:49.038 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:15:49.038 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:15:49.038 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:15:49.038 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:08.242091+0000", 2024-09-10T19:15:49.038 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0081839630000000007 2024-09-10T19:15:49.038 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:15:49.039 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:15:49.039 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-10T19:15:49.039 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:08.242286+0000", 2024-09-10T19:15:49.039 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00019465199999999999 2024-09-10T19:15:49.039 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:15:49.039 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:15:49.039 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-10T19:15:49.039 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:08.242327+0000", 2024-09-10T19:15:49.039 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.1820999999999998e-05 2024-09-10T19:15:49.039 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:15:49.039 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:15:49.039 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:15:49.039 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:15:49.040 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:15:49.040 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_repop(client.14781.0:161687 7.5 e120/86 7:a710c747:::benchmark_data_smithi183_84160_object21306:head v 120'10122, mlcod=120'10120)", 2024-09-10T19:15:49.040 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-10T19:14:08.098183+0000", 2024-09-10T19:15:49.040 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 100.83983791599999, 2024-09-10T19:15:49.040 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.13543248199999999, 2024-09-10T19:15:49.040 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-10T19:15:49.040 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T19:15:49.040 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-10T19:15:49.040 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:15:49.040 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-10T19:15:49.040 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:08.098183+0000", 2024-09-10T19:15:49.040 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:15:49.040 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:15:49.041 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:15:49.041 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-10T19:15:49.041 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:08.098183+0000", 2024-09-10T19:15:49.041 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:15:49.041 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:15:49.041 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:15:49.041 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-10T19:15:49.041 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:08.098182+0000", 2024-09-10T19:15:49.041 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4294967295.999999 2024-09-10T19:15:49.041 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:15:49.041 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:15:49.041 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-10T19:15:49.041 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:08.098190+0000", 2024-09-10T19:15:49.041 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.7239999999999992e-06 2024-09-10T19:15:49.042 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:15:49.042 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:15:49.042 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-10T19:15:49.042 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:08.098199+0000", 2024-09-10T19:15:49.042 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.7080000000000003e-06 2024-09-10T19:15:49.042 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:15:49.042 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:15:49.042 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-10T19:15:49.042 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:08.098203+0000", 2024-09-10T19:15:49.042 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.6979999999999997e-06 2024-09-10T19:15:49.042 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:15:49.042 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:15:49.042 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-10T19:15:49.042 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:08.098241+0000", 2024-09-10T19:15:49.043 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.7696e-05 2024-09-10T19:15:49.043 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:15:49.043 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:15:49.043 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-10T19:15:49.043 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:08.098258+0000", 2024-09-10T19:15:49.043 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.7184e-05 2024-09-10T19:15:49.043 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:15:49.043 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:15:49.043 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-10T19:15:49.043 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:08.100539+0000", 2024-09-10T19:15:49.043 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0022809810000000001 2024-09-10T19:15:49.043 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:15:49.043 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:15:49.043 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-10T19:15:49.043 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:08.233615+0000", 2024-09-10T19:15:49.044 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.133076212 2024-09-10T19:15:49.044 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:15:49.044 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:15:49.044 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:15:49.044 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:15:49.044 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:15:49.044 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_repop(client.14781.0:161683 7.0 e120/116 7:085c0f2f:::benchmark_data_smithi183_84160_object21302:head v 120'10096, mlcod=120'10093)", 2024-09-10T19:15:49.044 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-10T19:14:08.098314+0000", 2024-09-10T19:15:49.044 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 100.839706803, 2024-09-10T19:15:49.044 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.15554790700000001, 2024-09-10T19:15:49.044 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-10T19:15:49.044 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T19:15:49.044 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-10T19:15:49.044 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:15:49.045 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-10T19:15:49.045 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:08.098314+0000", 2024-09-10T19:15:49.045 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:15:49.045 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:15:49.045 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:15:49.045 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-10T19:15:49.045 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:08.098314+0000", 2024-09-10T19:15:49.045 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:15:49.045 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:15:49.045 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:15:49.045 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-10T19:15:49.045 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:08.098314+0000", 2024-09-10T19:15:49.045 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4294967295.9999995 2024-09-10T19:15:49.046 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:15:49.046 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:15:49.046 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-10T19:15:49.046 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:08.098318+0000", 2024-09-10T19:15:49.046 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.4229999999999998e-06 2024-09-10T19:15:49.046 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:15:49.046 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:15:49.046 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-10T19:15:49.046 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:08.098327+0000", 2024-09-10T19:15:49.046 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.2199999999999998e-06 2024-09-10T19:15:49.046 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:15:49.046 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:15:49.046 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-10T19:15:49.047 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:08.098331+0000", 2024-09-10T19:15:49.047 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.9299999999999996e-06 2024-09-10T19:15:49.047 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:15:49.047 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:15:49.047 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-10T19:15:49.047 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:08.233758+0000", 2024-09-10T19:15:49.047 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.135427031 2024-09-10T19:15:49.047 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:15:49.047 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:15:49.047 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-10T19:15:49.047 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:08.233780+0000", 2024-09-10T19:15:49.047 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.1761999999999999e-05 2024-09-10T19:15:49.047 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:15:49.048 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:15:49.048 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-10T19:15:49.048 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:08.253191+0000", 2024-09-10T19:15:49.048 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.019411069 2024-09-10T19:15:49.048 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:15:49.048 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:15:49.048 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-10T19:15:49.048 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:08.253862+0000", 2024-09-10T19:15:49.048 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00067096099999999997 2024-09-10T19:15:49.048 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:15:49.048 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:15:49.048 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:15:49.048 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:15:49.048 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:15:49.048 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_repop_reply(client.14781.0:161675 7.8 e120/86 ondisk, result = 0)", 2024-09-10T19:15:49.049 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-10T19:14:08.100726+0000", 2024-09-10T19:15:49.049 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 100.837295111, 2024-09-10T19:15:49.049 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.13297299300000001, 2024-09-10T19:15:49.049 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-10T19:15:49.049 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "started", 2024-09-10T19:15:49.049 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-10T19:15:49.049 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:15:49.049 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-10T19:15:49.049 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:08.100726+0000", 2024-09-10T19:15:49.049 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:15:49.049 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:15:49.049 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:15:49.049 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-10T19:15:49.050 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:08.100726+0000", 2024-09-10T19:15:49.050 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:15:49.050 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:15:49.050 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:15:49.050 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-10T19:15:49.050 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:08.100725+0000", 2024-09-10T19:15:49.050 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4294967295.999999 2024-09-10T19:15:49.050 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:15:49.050 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:15:49.050 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-10T19:15:49.050 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:08.100733+0000", 2024-09-10T19:15:49.050 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.3739999999999998e-06 2024-09-10T19:15:49.050 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:15:49.050 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:15:49.050 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-10T19:15:49.050 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:08.233540+0000", 2024-09-10T19:15:49.051 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.132806437 2024-09-10T19:15:49.051 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:15:49.051 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:15:49.051 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-10T19:15:49.051 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:08.233556+0000", 2024-09-10T19:15:49.051 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.5962000000000001e-05 2024-09-10T19:15:49.051 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:15:49.051 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:15:49.051 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-10T19:15:49.051 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:08.233678+0000", 2024-09-10T19:15:49.051 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00012219999999999999 2024-09-10T19:15:49.051 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:15:49.051 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:15:49.051 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-10T19:15:49.052 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:08.233688+0000", 2024-09-10T19:15:49.052 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.0138e-05 2024-09-10T19:15:49.052 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:15:49.052 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:15:49.052 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-10T19:15:49.052 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:08.233699+0000", 2024-09-10T19:15:49.052 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.0617e-05 2024-09-10T19:15:49.052 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:15:49.052 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:15:49.052 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:15:49.052 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:15:49.052 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:15:49.052 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.14781.0:184800 7.8 7:1e0ea857:::benchmark_data_smithi183_84160_object44419:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122)", 2024-09-10T19:15:49.053 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-10T19:14:26.531942+0000", 2024-09-10T19:15:49.053 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 82.406078586000007, 2024-09-10T19:15:49.053 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.099247427999999999, 2024-09-10T19:15:49.053 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-10T19:15:49.053 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T19:15:49.053 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-10T19:15:49.053 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.14781", 2024-09-10T19:15:49.053 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.183:0/2025776420", 2024-09-10T19:15:49.053 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 184800 2024-09-10T19:15:49.053 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:15:49.053 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-10T19:15:49.053 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:15:49.053 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-10T19:15:49.053 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:26.531942+0000", 2024-09-10T19:15:49.053 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:15:49.054 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:15:49.054 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:15:49.054 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-10T19:15:49.054 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:26.531942+0000", 2024-09-10T19:15:49.054 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:15:49.054 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:15:49.054 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:15:49.054 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-10T19:15:49.054 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:26.531942+0000", 2024-09-10T19:15:49.054 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4294967295.9999995 2024-09-10T19:15:49.054 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:15:49.054 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:15:49.054 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-10T19:15:49.054 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:26.531946+0000", 2024-09-10T19:15:49.054 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.743e-06 2024-09-10T19:15:49.055 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:15:49.055 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:15:49.055 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-10T19:15:49.055 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:26.531953+0000", 2024-09-10T19:15:49.055 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.2060000000000001e-06 2024-09-10T19:15:49.055 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:15:49.055 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:15:49.055 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-10T19:15:49.055 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:26.531957+0000", 2024-09-10T19:15:49.055 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.788e-06 2024-09-10T19:15:49.055 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:15:49.055 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:15:49.055 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-10T19:15:49.055 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:26.535574+0000", 2024-09-10T19:15:49.056 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.003617309 2024-09-10T19:15:49.056 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:15:49.056 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:15:49.056 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-10T19:15:49.056 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:26.535674+0000", 2024-09-10T19:15:49.056 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000100146 2024-09-10T19:15:49.056 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:15:49.056 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:15:49.056 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 4,7", 2024-09-10T19:15:49.056 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:26.535795+0000", 2024-09-10T19:15:49.056 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000121418 2024-09-10T19:15:49.056 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:15:49.056 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:15:49.057 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-10T19:15:49.057 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:26.538632+0000", 2024-09-10T19:15:49.057 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0028368500000000001 2024-09-10T19:15:49.057 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:15:49.057 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:15:49.057 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:15:49.057 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:26.549860+0000", 2024-09-10T19:15:49.057 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.01122799 2024-09-10T19:15:49.057 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:15:49.057 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:15:49.057 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:15:49.057 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:26.631100+0000", 2024-09-10T19:15:49.057 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.081239459 2024-09-10T19:15:49.057 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:15:49.058 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:15:49.058 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-10T19:15:49.058 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:26.631153+0000", 2024-09-10T19:15:49.058 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.3025999999999997e-05 2024-09-10T19:15:49.058 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:15:49.058 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:15:49.058 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-10T19:15:49.058 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:26.631190+0000", 2024-09-10T19:15:49.058 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.7020999999999997e-05 2024-09-10T19:15:49.058 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:15:49.058 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:15:49.058 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:15:49.058 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:15:49.058 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:15:49.059 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.14781.0:184815 7.8 7:1248c7ea:::benchmark_data_smithi183_84160_object44434:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122)", 2024-09-10T19:15:49.059 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-10T19:14:26.540210+0000", 2024-09-10T19:15:49.059 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 82.397810755999998, 2024-09-10T19:15:49.059 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.094977161000000004, 2024-09-10T19:15:49.059 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-10T19:15:49.059 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T19:15:49.059 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-10T19:15:49.059 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.14781", 2024-09-10T19:15:49.059 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.183:0/2025776420", 2024-09-10T19:15:49.059 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 184815 2024-09-10T19:15:49.059 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:15:49.059 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-10T19:15:49.059 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:15:49.059 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-10T19:15:49.059 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:26.540210+0000", 2024-09-10T19:15:49.060 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:15:49.060 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:15:49.060 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:15:49.060 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-10T19:15:49.060 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:26.540210+0000", 2024-09-10T19:15:49.060 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:15:49.060 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:15:49.060 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:15:49.060 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-10T19:15:49.060 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:26.540209+0000", 2024-09-10T19:15:49.060 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4294967295.999999 2024-09-10T19:15:49.060 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:15:49.060 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:15:49.060 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-10T19:15:49.060 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:26.540215+0000", 2024-09-10T19:15:49.061 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.7060000000000001e-06 2024-09-10T19:15:49.061 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:15:49.061 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:15:49.061 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-10T19:15:49.061 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:26.540226+0000", 2024-09-10T19:15:49.061 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.1377e-05 2024-09-10T19:15:49.061 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:15:49.061 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:15:49.061 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-10T19:15:49.061 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:26.540232+0000", 2024-09-10T19:15:49.061 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.913e-06 2024-09-10T19:15:49.061 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:15:49.061 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:15:49.061 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-10T19:15:49.062 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:26.544071+0000", 2024-09-10T19:15:49.062 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.003839103 2024-09-10T19:15:49.062 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:15:49.062 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:15:49.062 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-10T19:15:49.062 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:26.544192+0000", 2024-09-10T19:15:49.062 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000120612 2024-09-10T19:15:49.062 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:15:49.062 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:15:49.062 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 4,7", 2024-09-10T19:15:49.062 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:26.544332+0000", 2024-09-10T19:15:49.062 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000140436 2024-09-10T19:15:49.062 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:15:49.062 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:15:49.063 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-10T19:15:49.063 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:26.547364+0000", 2024-09-10T19:15:49.063 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0030315899999999998 2024-09-10T19:15:49.063 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:15:49.063 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:15:49.063 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:15:49.063 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:26.549943+0000", 2024-09-10T19:15:49.063 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.002579507 2024-09-10T19:15:49.063 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:15:49.063 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:15:49.063 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:15:49.063 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:26.635107+0000", 2024-09-10T19:15:49.063 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.085163191999999999 2024-09-10T19:15:49.063 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:15:49.064 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:15:49.064 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-10T19:15:49.064 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:26.635152+0000", 2024-09-10T19:15:49.064 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.5419e-05 2024-09-10T19:15:49.064 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:15:49.064 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:15:49.064 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-10T19:15:49.064 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:26.635187+0000", 2024-09-10T19:15:49.064 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.5364999999999998e-05 2024-09-10T19:15:49.064 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:15:49.064 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:15:49.064 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:15:49.064 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:15:49.064 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:15:49.065 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.14781.0:218745 7.e 7:76dfdc5d:::benchmark_data_smithi183_84160_object78364:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e132)", 2024-09-10T19:15:49.065 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-10T19:14:58.676347+0000", 2024-09-10T19:15:49.065 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 50.261674360000001, 2024-09-10T19:15:49.065 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.10844561699999999, 2024-09-10T19:15:49.065 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-10T19:15:49.065 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T19:15:49.065 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-10T19:15:49.065 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.14781", 2024-09-10T19:15:49.065 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.183:0/2025776420", 2024-09-10T19:15:49.065 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 218745 2024-09-10T19:15:49.065 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:15:49.065 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-10T19:15:49.065 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:15:49.066 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-10T19:15:49.066 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:58.676347+0000", 2024-09-10T19:15:49.066 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:15:49.066 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:15:49.066 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:15:49.066 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-10T19:15:49.066 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:58.676347+0000", 2024-09-10T19:15:49.066 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:15:49.066 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:15:49.066 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:15:49.066 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-10T19:15:49.066 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:58.676344+0000", 2024-09-10T19:15:49.066 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4294967295.9999976 2024-09-10T19:15:49.066 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:15:49.067 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:15:49.067 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-10T19:15:49.067 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:58.676352+0000", 2024-09-10T19:15:49.067 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.1280000000000008e-06 2024-09-10T19:15:49.067 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:15:49.067 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:15:49.067 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-10T19:15:49.067 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:58.676366+0000", 2024-09-10T19:15:49.067 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.3864e-05 2024-09-10T19:15:49.067 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:15:49.067 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:15:49.067 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-10T19:15:49.067 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:58.676372+0000", 2024-09-10T19:15:49.067 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.1009999999999997e-06 2024-09-10T19:15:49.068 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:15:49.068 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:15:49.068 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-10T19:15:49.068 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:58.681308+0000", 2024-09-10T19:15:49.068 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0049355149999999997 2024-09-10T19:15:49.068 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:15:49.068 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:15:49.068 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-10T19:15:49.068 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:58.681423+0000", 2024-09-10T19:15:49.068 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00011496 2024-09-10T19:15:49.068 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:15:49.068 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:15:49.068 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 0,5", 2024-09-10T19:15:49.069 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:58.681532+0000", 2024-09-10T19:15:49.069 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00010943899999999999 2024-09-10T19:15:49.069 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:15:49.069 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:15:49.069 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:15:49.069 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:58.686635+0000", 2024-09-10T19:15:49.069 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.005102667 2024-09-10T19:15:49.069 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:15:49.069 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:15:49.069 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-10T19:15:49.069 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:58.688709+0000", 2024-09-10T19:15:49.069 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0020736449999999998 2024-09-10T19:15:49.069 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:15:49.069 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:15:49.070 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:15:49.070 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:58.784711+0000", 2024-09-10T19:15:49.070 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.096002303999999997 2024-09-10T19:15:49.070 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:15:49.070 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:15:49.070 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-10T19:15:49.070 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:58.784753+0000", 2024-09-10T19:15:49.070 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.2373000000000002e-05 2024-09-10T19:15:49.070 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:15:49.070 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:15:49.070 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-10T19:15:49.070 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:58.784792+0000", 2024-09-10T19:15:49.070 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.8924999999999997e-05 2024-09-10T19:15:49.071 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:15:49.071 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:15:49.071 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:15:49.071 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:15:49.071 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:15:49.071 INFO:teuthology.orchestra.run.smithi183.stdout:} 2024-09-10T19:15:49.071 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:15:49.071 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:15:49.172 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:15:49.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:15:49 smithi183 ceph-mon[29736]: 4.1f scrub starts 2024-09-10T19:15:49.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:15:49 smithi183 ceph-mon[29736]: 4.1f scrub ok 2024-09-10T19:15:49.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:15:49 smithi183 ceph-mon[29736]: pgmap v226: 148 pgs: 2 peering, 146 active+clean; 34 MiB data, 2.2 GiB used, 624 GiB / 626 GiB avail; 0 B/s wr, 1.54k op/s 2024-09-10T19:15:49.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:15:49 smithi183 ceph-mon[29736]: 6.14 scrub starts 2024-09-10T19:15:49.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:15:49 smithi183 ceph-mon[29736]: 6.14 scrub ok 2024-09-10T19:15:49.182 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:15:49 smithi183 ceph-mon[29736]: from='client.? 172.21.15.183:0/822192628' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T19:15:49.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:15:49 smithi183 ceph-mon[41562]: 4.1f scrub starts 2024-09-10T19:15:49.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:15:49 smithi183 ceph-mon[41562]: 4.1f scrub ok 2024-09-10T19:15:49.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:15:49 smithi183 ceph-mon[41562]: pgmap v226: 148 pgs: 2 peering, 146 active+clean; 34 MiB data, 2.2 GiB used, 624 GiB / 626 GiB avail; 0 B/s wr, 1.54k op/s 2024-09-10T19:15:49.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:15:49 smithi183 ceph-mon[41562]: 6.14 scrub starts 2024-09-10T19:15:49.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:15:49 smithi183 ceph-mon[41562]: 6.14 scrub ok 2024-09-10T19:15:49.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:15:49 smithi183 ceph-mon[41562]: from='client.? 172.21.15.183:0/822192628' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T19:15:49.200 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:15:49.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:15:49.201 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:15:49.271 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph osd set noscrub 2024-09-10T19:15:49.302 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:15:49.336 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:15:49.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:15:49.337 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:15:49.438 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:15:49.464 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:15:49.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:15:49.465 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:15:49.465 INFO:journalctl@ceph.iscsi.iscsi.a.smithi191.stdout:Sep 10 19:15:49 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-iscsi-iscsi-a[61816]: debug there is no tcmu-runner data available 2024-09-10T19:15:49.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:15:49 smithi191 ceph-mon[32884]: 4.1f scrub starts 2024-09-10T19:15:49.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:15:49 smithi191 ceph-mon[32884]: 4.1f scrub ok 2024-09-10T19:15:49.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:15:49 smithi191 ceph-mon[32884]: pgmap v226: 148 pgs: 2 peering, 146 active+clean; 34 MiB data, 2.2 GiB used, 624 GiB / 626 GiB avail; 0 B/s wr, 1.54k op/s 2024-09-10T19:15:49.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:15:49 smithi191 ceph-mon[32884]: 6.14 scrub starts 2024-09-10T19:15:49.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:15:49 smithi191 ceph-mon[32884]: 6.14 scrub ok 2024-09-10T19:15:49.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:15:49 smithi191 ceph-mon[32884]: from='client.? 172.21.15.183:0/822192628' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T19:15:49.566 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:15:49.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:15:49.587 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:15:49.588 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:15:49.688 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:15:49.710 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:15:49.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:15:49.711 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:15:49.812 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:15:49.838 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:15:49.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:15:49.839 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:15:49.857 INFO:teuthology.orchestra.run.smithi183.stdout:osd.0: {} 2024-09-10T19:15:49.857 INFO:teuthology.orchestra.run.smithi183.stdout:osd.1: {} 2024-09-10T19:15:49.857 INFO:teuthology.orchestra.run.smithi183.stdout:osd.2: {} 2024-09-10T19:15:49.857 INFO:teuthology.orchestra.run.smithi183.stdout:osd.3: {} 2024-09-10T19:15:49.857 INFO:teuthology.orchestra.run.smithi183.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-10T19:15:49.858 INFO:teuthology.orchestra.run.smithi183.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-10T19:15:49.858 INFO:teuthology.orchestra.run.smithi183.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-10T19:15:49.858 INFO:teuthology.orchestra.run.smithi183.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-10T19:15:49.858 INFO:teuthology.orchestra.run.smithi183.stdout:osd.4: {} 2024-09-10T19:15:49.858 INFO:teuthology.orchestra.run.smithi183.stdout:osd.5: {} 2024-09-10T19:15:49.858 INFO:teuthology.orchestra.run.smithi183.stdout:osd.6: {} 2024-09-10T19:15:49.858 INFO:teuthology.orchestra.run.smithi183.stdout:osd.7: {} 2024-09-10T19:15:49.858 INFO:teuthology.orchestra.run.smithi183.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-10T19:15:49.858 INFO:teuthology.orchestra.run.smithi183.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-10T19:15:49.859 INFO:teuthology.orchestra.run.smithi183.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-10T19:15:49.859 INFO:teuthology.orchestra.run.smithi183.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-10T19:15:49.939 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:15:49.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:15:49.961 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:15:49.961 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:15:50.062 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:15:50.088 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:15:50.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:15:50.089 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:15:50.192 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:15:50.218 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:15:50.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:15:50.219 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:15:50.320 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:15:50.345 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:15:50.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:15:50.346 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:15:50.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:15:50 smithi183 ceph-mon[29736]: from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:15:50.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:15:50 smithi183 ceph-mon[29736]: 4.4 scrub starts 2024-09-10T19:15:50.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:15:50 smithi183 ceph-mon[29736]: 4.4 scrub ok 2024-09-10T19:15:50.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:15:50 smithi183 ceph-mon[41562]: from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:15:50.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:15:50 smithi183 ceph-mon[41562]: 4.4 scrub starts 2024-09-10T19:15:50.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:15:50 smithi183 ceph-mon[41562]: 4.4 scrub ok 2024-09-10T19:15:50.447 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:15:50.465 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:15:50 smithi191 ceph-mon[32884]: from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:15:50.465 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:15:50 smithi191 ceph-mon[32884]: 4.4 scrub starts 2024-09-10T19:15:50.465 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:15:50 smithi191 ceph-mon[32884]: 4.4 scrub ok 2024-09-10T19:15:50.472 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:15:50.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:15:50.473 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:15:50.574 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:15:50.596 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:15:50.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:15:50.597 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:15:50.697 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:15:50.723 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:15:50.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:15:50.724 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:15:50.825 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:15:50.850 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:15:50.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:15:50.851 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:15:50.951 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:15:50.977 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:15:50.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:15:50.978 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:15:51.079 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:15:51.104 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:15:51.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:15:51.105 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:15:51.206 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:15:51.231 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:15:51.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:15:51.232 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:15:51.316 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:15:51 smithi183 ceph-mon[29736]: 4.5 scrub starts 2024-09-10T19:15:51.316 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:15:51 smithi183 ceph-mon[29736]: 4.5 scrub ok 2024-09-10T19:15:51.316 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:15:51 smithi183 ceph-mon[29736]: pgmap v227: 148 pgs: 148 active+clean; 25 MiB data, 2.2 GiB used, 624 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 864 op/s; 27 B/s, 1 objects/s recovering 2024-09-10T19:15:51.316 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:15:51 smithi183 ceph-mon[29736]: 2.d scrub starts 2024-09-10T19:15:51.316 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:15:51 smithi183 ceph-mon[29736]: 2.d scrub ok 2024-09-10T19:15:51.316 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:15:51 smithi183 ceph-mon[41562]: 4.5 scrub starts 2024-09-10T19:15:51.316 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:15:51 smithi183 ceph-mon[41562]: 4.5 scrub ok 2024-09-10T19:15:51.316 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:15:51 smithi183 ceph-mon[41562]: pgmap v227: 148 pgs: 148 active+clean; 25 MiB data, 2.2 GiB used, 624 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 864 op/s; 27 B/s, 1 objects/s recovering 2024-09-10T19:15:51.317 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:15:51 smithi183 ceph-mon[41562]: 2.d scrub starts 2024-09-10T19:15:51.317 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:15:51 smithi183 ceph-mon[41562]: 2.d scrub ok 2024-09-10T19:15:51.332 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:15:51.354 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:15:51.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:15:51.355 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:15:51.456 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:15:51.465 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:15:51 smithi191 ceph-mon[32884]: 4.5 scrub starts 2024-09-10T19:15:51.465 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:15:51 smithi191 ceph-mon[32884]: 4.5 scrub ok 2024-09-10T19:15:51.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:15:51 smithi191 ceph-mon[32884]: pgmap v227: 148 pgs: 148 active+clean; 25 MiB data, 2.2 GiB used, 624 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 864 op/s; 27 B/s, 1 objects/s recovering 2024-09-10T19:15:51.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:15:51 smithi191 ceph-mon[32884]: 2.d scrub starts 2024-09-10T19:15:51.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:15:51 smithi191 ceph-mon[32884]: 2.d scrub ok 2024-09-10T19:15:51.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:15:51.479 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:15:51.479 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:15:51.491 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph daemon osd.3 dump_ops_in_flight 2024-09-10T19:15:51.580 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:15:51.607 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:15:51.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:15:51.608 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:15:51.636 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Sep 10 19:15:51 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-y[29989]: ::ffff:172.21.15.191 - - [10/Sep/2024:19:15:51] "GET /metrics HTTP/1.1" 200 224047 "" "Prometheus/2.43.0" 2024-09-10T19:15:51.709 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:15:51.732 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:15:51.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:15:51.733 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:15:51.765 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:15:51.833 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:15:51.856 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:15:51.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:15:51.857 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:15:51.906 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-09-10T19:15:51.909 INFO:teuthology.orchestra.run.smithi183.stderr:dumped all 2024-09-10T19:15:51.957 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:15:51.978 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:15:51.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:15:51.979 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:15:52.080 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:15:52.102 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:15:52.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:15:52.103 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:15:52.203 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:15:52.226 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:15:52.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:15:52.227 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:15:52.328 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:15:52.349 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:15:52.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:15:52.351 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:15:52.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:15:52 smithi183 ceph-mon[29736]: 5.2 scrub starts 2024-09-10T19:15:52.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:15:52 smithi183 ceph-mon[29736]: 5.2 scrub ok 2024-09-10T19:15:52.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:15:52 smithi183 ceph-mon[29736]: 7.6 scrub starts 2024-09-10T19:15:52.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:15:52 smithi183 ceph-mon[29736]: 7.6 scrub ok 2024-09-10T19:15:52.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:15:52 smithi183 ceph-mon[41562]: 5.2 scrub starts 2024-09-10T19:15:52.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:15:52 smithi183 ceph-mon[41562]: 5.2 scrub ok 2024-09-10T19:15:52.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:15:52 smithi183 ceph-mon[41562]: 7.6 scrub starts 2024-09-10T19:15:52.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:15:52 smithi183 ceph-mon[41562]: 7.6 scrub ok 2024-09-10T19:15:52.452 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:15:52.465 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:15:52 smithi191 ceph-mon[32884]: 5.2 scrub starts 2024-09-10T19:15:52.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:15:52 smithi191 ceph-mon[32884]: 5.2 scrub ok 2024-09-10T19:15:52.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:15:52 smithi191 ceph-mon[32884]: 7.6 scrub starts 2024-09-10T19:15:52.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:15:52 smithi191 ceph-mon[32884]: 7.6 scrub ok 2024-09-10T19:15:52.478 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:15:52.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:15:52.479 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:15:52.579 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:15:52.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:15:52.602 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:15:52.602 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:15:52.703 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:15:52.729 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:15:52.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:15:52.730 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:15:52.831 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:15:52.856 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:15:52.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:15:52.857 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:15:52.958 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:15:52.979 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:15:52.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:15:52.980 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:15:53.081 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:15:53.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:15:53.103 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:15:53.103 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:15:53.204 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:15:53.230 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:15:53.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:15:53.231 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:15:53.332 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:15:53.354 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:15:53.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:15:53.355 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:15:53.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:15:53 smithi183 ceph-mon[29736]: pgmap v228: 148 pgs: 148 active+clean; 582 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 913 op/s; 27 B/s, 1 objects/s recovering 2024-09-10T19:15:53.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:15:53 smithi183 ceph-mon[29736]: from='client.14982 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-10T19:15:53.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:15:53 smithi183 ceph-mon[41562]: pgmap v228: 148 pgs: 148 active+clean; 582 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 913 op/s; 27 B/s, 1 objects/s recovering 2024-09-10T19:15:53.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:15:53 smithi183 ceph-mon[41562]: from='client.14982 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-10T19:15:53.455 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:15:53.465 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:15:53 smithi191 ceph-mon[32884]: pgmap v228: 148 pgs: 148 active+clean; 582 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 913 op/s; 27 B/s, 1 objects/s recovering 2024-09-10T19:15:53.465 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:15:53 smithi191 ceph-mon[32884]: from='client.14982 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-10T19:15:53.481 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:15:53.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:15:53.482 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:15:53.582 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:15:53.607 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:15:53.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:15:53.608 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:15:53.627 INFO:teuthology.orchestra.run.smithi183.stdout:{"pg_ready":true,"pg_map":{"version":228,"stamp":"2024-09-10T19:15:51.713448+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596188,"num_objects":196,"num_object_clones":0,"num_object_copies":588,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":196,"num_whiteouts":0,"num_read":1034,"num_read_kb":891,"num_write":281301,"num_write_kb":8985864,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33842,"num_bytes_recovered":2209549865,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76866,"ondisk_log_size":76866,"up":444,"acting":444,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":444,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":656216064,"kb_used":2134752,"kb_used_data":7508,"kb_used_omap":11,"kb_used_meta":2127156,"kb_avail":654081312,"statfs":{"total":671965249536,"available":669779263488,"internally_reserved":0,"allocated":7688192,"data_stored":5337067,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":11928,"internal_metadata":2178208104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-478412824,"num_objects":-7301,"num_object_clones":0,"num_object_copies":-21903,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-7301,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":7301,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002022"},"pg_stats":[{"pgid":"6.1b","version":"0'0","reported_seq":108,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-09-10T19:15:44.143620+0000","last_change":"2024-09-10T19:13:08.710758+0000","last_active":"2024-09-10T19:15:44.143620+0000","last_peered":"2024-09-10T19:15:44.143620+0000","last_clean":"2024-09-10T19:15:44.143620+0000","last_became_active":"2024-09-10T19:10:14.056187+0000","last_became_peered":"2024-09-10T19:10:14.056187+0000","last_unstale":"2024-09-10T19:15:44.143620+0000","last_undegraded":"2024-09-10T19:15:44.143620+0000","last_fullsized":"2024-09-10T19:15:44.143620+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:13:08.710679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:13:08.710679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0027252499999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":124,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-09-10T19:15:44.140180+0000","last_change":"2024-09-10T19:13:16.219506+0000","last_active":"2024-09-10T19:15:44.140180+0000","last_peered":"2024-09-10T19:15:44.140180+0000","last_clean":"2024-09-10T19:15:44.140180+0000","last_became_active":"2024-09-10T19:11:44.748696+0000","last_became_peered":"2024-09-10T19:11:44.748696+0000","last_unstale":"2024-09-10T19:15:44.140180+0000","last_undegraded":"2024-09-10T19:15:44.140180+0000","last_fullsized":"2024-09-10T19:15:44.140180+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:13:16.219364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:13:16.219364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.023972094999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.19","version":"66'15","reported_seq":137,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-09-10T19:15:44.144374+0000","last_change":"2024-09-10T19:13:47.165943+0000","last_active":"2024-09-10T19:15:44.144374+0000","last_peered":"2024-09-10T19:15:44.144374+0000","last_clean":"2024-09-10T19:15:44.144374+0000","last_became_active":"2024-09-10T19:10:09.688432+0000","last_became_peered":"2024-09-10T19:10:09.688432+0000","last_unstale":"2024-09-10T19:15:44.144374+0000","last_undegraded":"2024-09-10T19:15:44.144374+0000","last_fullsized":"2024-09-10T19:15:44.144374+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'15","last_scrub_stamp":"2024-09-10T19:13:47.165848+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:13:47.165848+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.004982841,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":129,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-09-10T19:15:44.142082+0000","last_change":"2024-09-10T19:15:33.976761+0000","last_active":"2024-09-10T19:15:44.142082+0000","last_peered":"2024-09-10T19:15:44.142082+0000","last_clean":"2024-09-10T19:15:44.142082+0000","last_became_active":"2024-09-10T19:13:54.714824+0000","last_became_peered":"2024-09-10T19:13:54.714824+0000","last_unstale":"2024-09-10T19:15:44.142082+0000","last_undegraded":"2024-09-10T19:15:44.142082+0000","last_fullsized":"2024-09-10T19:15:44.142082+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:15:33.976696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:15:33.976696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:16:59.329419+0000","scrub_duration":0.0015310580000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"59'1","reported_seq":122,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-09-10T19:15:44.143461+0000","last_change":"2024-09-10T19:12:47.704757+0000","last_active":"2024-09-10T19:15:44.143461+0000","last_peered":"2024-09-10T19:15:44.143461+0000","last_clean":"2024-09-10T19:15:44.143461+0000","last_became_active":"2024-09-10T19:10:07.686143+0000","last_became_peered":"2024-09-10T19:10:07.686143+0000","last_unstale":"2024-09-10T19:15:44.143461+0000","last_undegraded":"2024-09-10T19:15:44.143461+0000","last_fullsized":"2024-09-10T19:15:44.143461+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'1","last_scrub_stamp":"2024-09-10T19:12:47.704508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:12:47.704508+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.010519827000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":874,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"66'9","reported_seq":154,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-09-10T19:15:44.142166+0000","last_change":"2024-09-10T19:14:55.117473+0000","last_active":"2024-09-10T19:15:44.142166+0000","last_peered":"2024-09-10T19:15:44.142166+0000","last_clean":"2024-09-10T19:15:44.142166+0000","last_became_active":"2024-09-10T19:13:54.716117+0000","last_became_peered":"2024-09-10T19:13:54.716117+0000","last_unstale":"2024-09-10T19:15:44.142166+0000","last_undegraded":"2024-09-10T19:15:44.142166+0000","last_fullsized":"2024-09-10T19:15:44.142166+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'9","last_scrub_stamp":"2024-09-10T19:14:55.117387+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:14:55.117387+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:15:55.884824+0000","scrub_duration":0.0074980940000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,3],"acting":[4,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":64,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-09-10T19:15:44.134069+0000","last_change":"2024-09-10T19:13:54.714747+0000","last_active":"2024-09-10T19:15:44.134069+0000","last_peered":"2024-09-10T19:15:44.134069+0000","last_clean":"2024-09-10T19:15:44.134069+0000","last_became_active":"2024-09-10T19:13:54.714365+0000","last_became_peered":"2024-09-10T19:13:54.714365+0000","last_unstale":"2024-09-10T19:15:44.134069+0000","last_undegraded":"2024-09-10T19:15:44.134069+0000","last_fullsized":"2024-09-10T19:15:44.134069+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:11:56.186365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:11:56.186365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":127,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-09-10T19:15:44.142264+0000","last_change":"2024-09-10T19:15:39.910169+0000","last_active":"2024-09-10T19:15:44.142264+0000","last_peered":"2024-09-10T19:15:44.142264+0000","last_clean":"2024-09-10T19:15:44.142264+0000","last_became_active":"2024-09-10T19:13:54.713538+0000","last_became_peered":"2024-09-10T19:13:54.713538+0000","last_unstale":"2024-09-10T19:15:44.142264+0000","last_undegraded":"2024-09-10T19:15:44.142264+0000","last_fullsized":"2024-09-10T19:15:44.142264+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:15:39.910118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:15:39.910118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:16:46.983414+0000","scrub_duration":0.0013551839999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,7],"acting":[4,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":114,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-09-10T19:15:44.137419+0000","last_change":"2024-09-10T19:12:49.527255+0000","last_active":"2024-09-10T19:15:44.137419+0000","last_peered":"2024-09-10T19:15:44.137419+0000","last_clean":"2024-09-10T19:15:44.137419+0000","last_became_active":"2024-09-10T19:10:07.690728+0000","last_became_peered":"2024-09-10T19:10:07.690728+0000","last_unstale":"2024-09-10T19:15:44.137419+0000","last_undegraded":"2024-09-10T19:15:44.137419+0000","last_fullsized":"2024-09-10T19:15:44.137419+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:12:49.527167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:12:49.527167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0040597050000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1b","version":"66'5","reported_seq":137,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-09-10T19:15:44.142361+0000","last_change":"2024-09-10T19:12:55.359526+0000","last_active":"2024-09-10T19:15:44.142361+0000","last_peered":"2024-09-10T19:15:44.142361+0000","last_clean":"2024-09-10T19:15:44.142361+0000","last_became_active":"2024-09-10T19:11:44.748296+0000","last_became_peered":"2024-09-10T19:11:44.748296+0000","last_unstale":"2024-09-10T19:15:44.142361+0000","last_undegraded":"2024-09-10T19:15:44.142361+0000","last_fullsized":"2024-09-10T19:15:44.142361+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'5","last_scrub_stamp":"2024-09-10T19:12:55.359379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:12:55.359379+0000","objects_scrubbed":2,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0050448070000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":127,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-09-10T19:15:44.137739+0000","last_change":"2024-09-10T19:15:33.518718+0000","last_active":"2024-09-10T19:15:44.137739+0000","last_peered":"2024-09-10T19:15:44.137739+0000","last_clean":"2024-09-10T19:15:44.137739+0000","last_became_active":"2024-09-10T19:13:54.719063+0000","last_became_peered":"2024-09-10T19:13:54.719063+0000","last_unstale":"2024-09-10T19:15:44.137739+0000","last_undegraded":"2024-09-10T19:15:44.137739+0000","last_fullsized":"2024-09-10T19:15:44.137739+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:15:33.518601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:15:33.518601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:16:52.587150+0000","scrub_duration":0.0013951650000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":125,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-09-10T19:15:45.616266+0000","last_change":"2024-09-10T19:15:45.616238+0000","last_active":"2024-09-10T19:15:45.616266+0000","last_peered":"2024-09-10T19:15:45.616266+0000","last_clean":"2024-09-10T19:15:45.616266+0000","last_became_active":"2024-09-10T19:13:54.723590+0000","last_became_peered":"2024-09-10T19:13:54.723590+0000","last_unstale":"2024-09-10T19:15:45.616266+0000","last_undegraded":"2024-09-10T19:15:45.616266+0000","last_fullsized":"2024-09-10T19:15:45.616266+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:15:45.616184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:15:45.616184+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:17:14.482505+0000","scrub_duration":0.0013989510000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":114,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-09-10T19:15:44.136936+0000","last_change":"2024-09-10T19:12:54.477202+0000","last_active":"2024-09-10T19:15:44.136936+0000","last_peered":"2024-09-10T19:15:44.136936+0000","last_clean":"2024-09-10T19:15:44.136936+0000","last_became_active":"2024-09-10T19:10:07.690174+0000","last_became_peered":"2024-09-10T19:10:07.690174+0000","last_unstale":"2024-09-10T19:15:44.136936+0000","last_undegraded":"2024-09-10T19:15:44.136936+0000","last_fullsized":"2024-09-10T19:15:44.136936+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:12:54.477149+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:12:54.477149+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0039768319999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1a","version":"66'9","reported_seq":127,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-09-10T19:15:44.142425+0000","last_change":"2024-09-10T19:13:02.431528+0000","last_active":"2024-09-10T19:15:44.142425+0000","last_peered":"2024-09-10T19:15:44.142425+0000","last_clean":"2024-09-10T19:15:44.142425+0000","last_became_active":"2024-09-10T19:10:09.688987+0000","last_became_peered":"2024-09-10T19:10:09.688987+0000","last_unstale":"2024-09-10T19:15:44.142425+0000","last_undegraded":"2024-09-10T19:15:44.142425+0000","last_fullsized":"2024-09-10T19:15:44.142425+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'9","last_scrub_stamp":"2024-09-10T19:13:02.431443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:13:02.431443+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0084913290000000006,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":110,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-09-10T19:15:44.134163+0000","last_change":"2024-09-10T19:12:51.449179+0000","last_active":"2024-09-10T19:15:44.134163+0000","last_peered":"2024-09-10T19:15:44.134163+0000","last_clean":"2024-09-10T19:15:44.134163+0000","last_became_active":"2024-09-10T19:10:11.701799+0000","last_became_peered":"2024-09-10T19:10:11.701799+0000","last_unstale":"2024-09-10T19:15:44.134163+0000","last_undegraded":"2024-09-10T19:15:44.134163+0000","last_fullsized":"2024-09-10T19:15:44.134163+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:12:51.449046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:12:51.449046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.002348826,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":125,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-09-10T19:15:44.139205+0000","last_change":"2024-09-10T19:15:33.839329+0000","last_active":"2024-09-10T19:15:44.139205+0000","last_peered":"2024-09-10T19:15:44.139205+0000","last_clean":"2024-09-10T19:15:44.139205+0000","last_became_active":"2024-09-10T19:13:54.718860+0000","last_became_peered":"2024-09-10T19:13:54.718860+0000","last_unstale":"2024-09-10T19:15:44.139205+0000","last_undegraded":"2024-09-10T19:15:44.139205+0000","last_fullsized":"2024-09-10T19:15:44.139205+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:15:33.839248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:15:33.839248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:17:01.970863+0000","scrub_duration":0.0032019980000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.e","version":"144'17892","reported_seq":18320,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-09-10T19:15:45.307277+0000","last_change":"2024-09-10T19:13:52.171265+0000","last_active":"2024-09-10T19:15:45.307277+0000","last_peered":"2024-09-10T19:15:45.307277+0000","last_clean":"2024-09-10T19:15:45.307277+0000","last_became_active":"2024-09-10T19:12:11.032901+0000","last_became_peered":"2024-09-10T19:12:11.032901+0000","last_unstale":"2024-09-10T19:15:45.307277+0000","last_undegraded":"2024-09-10T19:15:45.307277+0000","last_fullsized":"2024-09-10T19:15:45.307277+0000","mapping_epoch":86,"log_start":"123'12880","ondisk_log_start":"123'12880","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"114'9197","last_scrub_stamp":"2024-09-10T19:13:52.169437+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:12:10.007465+0000","last_clean_scrub_stamp":"2024-09-10T19:13:52.169437+0000","objects_scrubbed":8820,"log_size":5012,"log_dups_size":0,"ondisk_log_size":5012,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"queued for scrub","scrub_duration":3.857076422,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17892,"num_write_kb":572544,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.d","version":"66'17","reported_seq":174,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-09-10T19:15:44.142749+0000","last_change":"2024-09-10T19:13:54.747117+0000","last_active":"2024-09-10T19:15:44.142749+0000","last_peered":"2024-09-10T19:15:44.142749+0000","last_clean":"2024-09-10T19:15:44.142749+0000","last_became_active":"2024-09-10T19:13:54.713411+0000","last_became_peered":"2024-09-10T19:13:54.713411+0000","last_unstale":"2024-09-10T19:15:44.142749+0000","last_undegraded":"2024-09-10T19:15:44.142749+0000","last_fullsized":"2024-09-10T19:15:44.142749+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'17","last_scrub_stamp":"2024-09-10T19:12:45.367252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:12:45.367252+0000","objects_scrubbed":8,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0031240310000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":114,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-09-10T19:15:44.137157+0000","last_change":"2024-09-10T19:12:46.519868+0000","last_active":"2024-09-10T19:15:44.137157+0000","last_peered":"2024-09-10T19:15:44.137157+0000","last_clean":"2024-09-10T19:15:44.137157+0000","last_became_active":"2024-09-10T19:10:07.691119+0000","last_became_peered":"2024-09-10T19:10:07.691119+0000","last_unstale":"2024-09-10T19:15:44.137157+0000","last_undegraded":"2024-09-10T19:15:44.137157+0000","last_fullsized":"2024-09-10T19:15:44.137157+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:12:46.519818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:11:22.280002+0000","last_clean_scrub_stamp":"2024-09-10T19:12:46.519818+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001139279,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":67,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-09-10T19:15:44.143255+0000","last_change":"2024-09-10T19:14:51.085977+0000","last_active":"2024-09-10T19:15:44.143255+0000","last_peered":"2024-09-10T19:15:44.143255+0000","last_clean":"2024-09-10T19:15:44.143255+0000","last_became_active":"2024-09-10T19:13:54.712883+0000","last_became_peered":"2024-09-10T19:13:54.712883+0000","last_unstale":"2024-09-10T19:15:44.143255+0000","last_undegraded":"2024-09-10T19:15:44.143255+0000","last_fullsized":"2024-09-10T19:15:44.143255+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:12:00.240841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:12:00.240841+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":118,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-09-10T19:15:47.818466+0000","last_change":"2024-09-10T19:15:47.818436+0000","last_active":"2024-09-10T19:15:47.818466+0000","last_peered":"2024-09-10T19:15:47.818466+0000","last_clean":"2024-09-10T19:15:47.818466+0000","last_became_active":"2024-09-10T19:10:13.722894+0000","last_became_peered":"2024-09-10T19:10:13.722894+0000","last_unstale":"2024-09-10T19:15:47.818466+0000","last_undegraded":"2024-09-10T19:15:47.818466+0000","last_fullsized":"2024-09-10T19:15:47.818466+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:15:47.818381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:15:47.818381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:16:50.810900+0000","scrub_duration":0.0013815990000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.f","version":"144'17476","reported_seq":17897,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-09-10T19:15:45.306643+0000","last_change":"2024-09-10T19:13:49.950750+0000","last_active":"2024-09-10T19:15:45.306643+0000","last_peered":"2024-09-10T19:15:45.306643+0000","last_clean":"2024-09-10T19:15:45.306643+0000","last_became_active":"2024-09-10T19:12:11.036181+0000","last_became_peered":"2024-09-10T19:12:11.036181+0000","last_unstale":"2024-09-10T19:15:45.306643+0000","last_undegraded":"2024-09-10T19:15:45.306643+0000","last_fullsized":"2024-09-10T19:15:45.306643+0000","mapping_epoch":86,"log_start":"124'12725","ondisk_log_start":"124'12725","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"114'8808","last_scrub_stamp":"2024-09-10T19:13:49.950668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:12:10.007465+0000","last_clean_scrub_stamp":"2024-09-10T19:13:49.950668+0000","objects_scrubbed":8620,"log_size":4751,"log_dups_size":0,"ondisk_log_size":4751,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":6,"scrub_schedule":"queued for scrub","scrub_duration":5.7089046000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17476,"num_write_kb":559232,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.c","version":"66'10","reported_seq":136,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-09-10T19:15:44.143301+0000","last_change":"2024-09-10T19:13:51.416075+0000","last_active":"2024-09-10T19:15:44.143301+0000","last_peered":"2024-09-10T19:15:44.143301+0000","last_clean":"2024-09-10T19:15:44.143301+0000","last_became_active":"2024-09-10T19:10:09.688321+0000","last_became_peered":"2024-09-10T19:10:09.688321+0000","last_unstale":"2024-09-10T19:15:44.143301+0000","last_undegraded":"2024-09-10T19:15:44.143301+0000","last_fullsized":"2024-09-10T19:15:44.143301+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'10","last_scrub_stamp":"2024-09-10T19:13:51.415998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:13:51.415998+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.005972799,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":74,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-09-10T19:15:44.142837+0000","last_change":"2024-09-10T19:15:42.119912+0000","last_active":"2024-09-10T19:15:44.142837+0000","last_peered":"2024-09-10T19:15:44.142837+0000","last_clean":"2024-09-10T19:15:44.142837+0000","last_became_active":"2024-09-10T19:13:54.731134+0000","last_became_peered":"2024-09-10T19:13:54.731134+0000","last_unstale":"2024-09-10T19:15:44.142837+0000","last_undegraded":"2024-09-10T19:15:44.142837+0000","last_fullsized":"2024-09-10T19:15:44.142837+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:15:42.119590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:15:42.119590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:16:46.442054+0000","scrub_duration":0.018001347000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,5],"acting":[3,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"80'9","reported_seq":190,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-09-10T19:15:45.144624+0000","last_change":"2024-09-10T19:12:51.343854+0000","last_active":"2024-09-10T19:15:45.144624+0000","last_peered":"2024-09-10T19:15:45.144624+0000","last_clean":"2024-09-10T19:15:45.144624+0000","last_became_active":"2024-09-10T19:10:11.698547+0000","last_became_peered":"2024-09-10T19:10:11.698547+0000","last_unstale":"2024-09-10T19:15:45.144624+0000","last_undegraded":"2024-09-10T19:15:45.144624+0000","last_fullsized":"2024-09-10T19:15:45.144624+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"80'9","last_scrub_stamp":"2024-09-10T19:12:51.343776+0000","last_deep_scrub":"80'9","last_deep_scrub_stamp":"2024-09-10T19:11:25.156465+0000","last_clean_scrub_stamp":"2024-09-10T19:12:51.343776+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0026375819999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":130,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-09-10T19:15:44.142522+0000","last_change":"2024-09-10T19:15:29.989058+0000","last_active":"2024-09-10T19:15:44.142522+0000","last_peered":"2024-09-10T19:15:44.142522+0000","last_clean":"2024-09-10T19:15:44.142522+0000","last_became_active":"2024-09-10T19:13:54.719498+0000","last_became_peered":"2024-09-10T19:13:54.719498+0000","last_unstale":"2024-09-10T19:15:44.142522+0000","last_undegraded":"2024-09-10T19:15:44.142522+0000","last_fullsized":"2024-09-10T19:15:44.142522+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:15:29.988976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:15:29.988976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:16:41.546952+0000","scrub_duration":0.0053787150000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.c","version":"144'17614","reported_seq":18031,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-09-10T19:15:45.291138+0000","last_change":"2024-09-10T19:13:53.867193+0000","last_active":"2024-09-10T19:15:45.291138+0000","last_peered":"2024-09-10T19:15:45.291138+0000","last_clean":"2024-09-10T19:15:45.291138+0000","last_became_active":"2024-09-10T19:12:11.029860+0000","last_became_peered":"2024-09-10T19:12:11.029860+0000","last_unstale":"2024-09-10T19:15:45.291138+0000","last_undegraded":"2024-09-10T19:15:45.291138+0000","last_fullsized":"2024-09-10T19:15:45.291138+0000","mapping_epoch":86,"log_start":"123'12582","ondisk_log_start":"123'12582","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"114'9191","last_scrub_stamp":"2024-09-10T19:13:53.867107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:12:10.007465+0000","last_clean_scrub_stamp":"2024-09-10T19:13:53.867107+0000","objects_scrubbed":8534,"log_size":5032,"log_dups_size":0,"ondisk_log_size":5032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"queued for scrub","scrub_duration":2.6024067190000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17614,"num_write_kb":563648,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.f","version":"66'15","reported_seq":117,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-09-10T19:15:44.143767+0000","last_change":"2024-09-10T19:13:54.742958+0000","last_active":"2024-09-10T19:15:44.143767+0000","last_peered":"2024-09-10T19:15:44.143767+0000","last_clean":"2024-09-10T19:15:44.143767+0000","last_became_active":"2024-09-10T19:13:54.711634+0000","last_became_peered":"2024-09-10T19:13:54.711634+0000","last_unstale":"2024-09-10T19:15:44.143767+0000","last_undegraded":"2024-09-10T19:15:44.143767+0000","last_fullsized":"2024-09-10T19:15:44.143767+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'15","last_scrub_stamp":"2024-09-10T19:11:59.232145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:11:59.232145+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":74,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-09-10T19:15:44.136569+0000","last_change":"2024-09-10T19:15:29.553530+0000","last_active":"2024-09-10T19:15:44.136569+0000","last_peered":"2024-09-10T19:15:44.136569+0000","last_clean":"2024-09-10T19:15:44.136569+0000","last_became_active":"2024-09-10T19:13:54.718452+0000","last_became_peered":"2024-09-10T19:13:54.718452+0000","last_unstale":"2024-09-10T19:15:44.136569+0000","last_undegraded":"2024-09-10T19:15:44.136569+0000","last_fullsized":"2024-09-10T19:15:44.136569+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:15:29.553448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:15:29.553448+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:16:49.232851+0000","scrub_duration":0.0062080679999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.e","version":"66'8","reported_seq":185,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-09-10T19:15:45.136346+0000","last_change":"2024-09-10T19:13:40.475460+0000","last_active":"2024-09-10T19:15:45.136346+0000","last_peered":"2024-09-10T19:15:45.136346+0000","last_clean":"2024-09-10T19:15:45.136346+0000","last_became_active":"2024-09-10T19:10:11.699232+0000","last_became_peered":"2024-09-10T19:10:11.699232+0000","last_unstale":"2024-09-10T19:15:45.136346+0000","last_undegraded":"2024-09-10T19:15:45.136346+0000","last_fullsized":"2024-09-10T19:15:45.136346+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'8","last_scrub_stamp":"2024-09-10T19:13:40.475375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:13:40.475375+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.035806152000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":125,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-09-10T19:15:44.633180+0000","last_change":"2024-09-10T19:15:44.633122+0000","last_active":"2024-09-10T19:15:44.633180+0000","last_peered":"2024-09-10T19:15:44.633180+0000","last_clean":"2024-09-10T19:15:44.633180+0000","last_became_active":"2024-09-10T19:13:54.722688+0000","last_became_peered":"2024-09-10T19:13:54.722688+0000","last_unstale":"2024-09-10T19:15:44.633180+0000","last_undegraded":"2024-09-10T19:15:44.633180+0000","last_fullsized":"2024-09-10T19:15:44.633180+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:15:44.633043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:15:44.633043+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:17:13.623295+0000","scrub_duration":0.00368805,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.d","version":"144'17478","reported_seq":27402,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-09-10T19:15:45.315432+0000","last_change":"2024-09-10T19:15:43.770792+0000","last_active":"2024-09-10T19:15:45.315432+0000","last_peered":"2024-09-10T19:15:45.315432+0000","last_clean":"2024-09-10T19:15:45.315432+0000","last_became_active":"2024-09-10T19:14:56.840803+0000","last_became_peered":"2024-09-10T19:14:56.840803+0000","last_unstale":"2024-09-10T19:15:45.315432+0000","last_undegraded":"2024-09-10T19:15:45.315432+0000","last_fullsized":"2024-09-10T19:15:45.315432+0000","mapping_epoch":130,"log_start":"123'12456","ondisk_log_start":"123'12456","created":86,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"142'17373","last_scrub_stamp":"2024-09-10T19:15:43.770696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:12:10.007465+0000","last_clean_scrub_stamp":"2024-09-10T19:15:43.770696+0000","objects_scrubbed":105,"log_size":5022,"log_dups_size":0,"ondisk_log_size":5022,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:17:03.799827+0000","scrub_duration":0.052616865999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":17478,"num_write_kb":559233,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4906,"num_bytes_recovered":321454104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.e","version":"84'16","reported_seq":167,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-09-10T19:15:47.914521+0000","last_change":"2024-09-10T19:15:47.914487+0000","last_active":"2024-09-10T19:15:47.914521+0000","last_peered":"2024-09-10T19:15:47.914521+0000","last_clean":"2024-09-10T19:15:47.914521+0000","last_became_active":"2024-09-10T19:13:54.717022+0000","last_became_peered":"2024-09-10T19:13:54.717022+0000","last_unstale":"2024-09-10T19:15:47.914521+0000","last_undegraded":"2024-09-10T19:15:47.914521+0000","last_fullsized":"2024-09-10T19:15:47.914521+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'16","last_scrub_stamp":"2024-09-10T19:15:47.914420+0000","last_deep_scrub":"84'16","last_deep_scrub_stamp":"2024-09-10T19:11:56.157975+0000","last_clean_scrub_stamp":"2024-09-10T19:15:47.914420+0000","objects_scrubbed":5,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:16:48.311799+0000","scrub_duration":0.0019153390000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,3],"acting":[4,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":131,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-09-10T19:15:44.137020+0000","last_change":"2024-09-10T19:15:42.450149+0000","last_active":"2024-09-10T19:15:44.137020+0000","last_peered":"2024-09-10T19:15:44.137020+0000","last_clean":"2024-09-10T19:15:44.137020+0000","last_became_active":"2024-09-10T19:13:54.720293+0000","last_became_peered":"2024-09-10T19:13:54.720293+0000","last_unstale":"2024-09-10T19:15:44.137020+0000","last_undegraded":"2024-09-10T19:15:44.137020+0000","last_fullsized":"2024-09-10T19:15:44.137020+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:15:42.450068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:11:53.066849+0000","last_clean_scrub_stamp":"2024-09-10T19:15:42.450068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:16:57.302037+0000","scrub_duration":0.0037708360000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":110,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-09-10T19:15:44.134584+0000","last_change":"2024-09-10T19:13:06.358188+0000","last_active":"2024-09-10T19:15:44.134584+0000","last_peered":"2024-09-10T19:15:44.134584+0000","last_clean":"2024-09-10T19:15:44.134584+0000","last_became_active":"2024-09-10T19:10:11.692215+0000","last_became_peered":"2024-09-10T19:10:11.692215+0000","last_unstale":"2024-09-10T19:15:44.134584+0000","last_undegraded":"2024-09-10T19:15:44.134584+0000","last_fullsized":"2024-09-10T19:15:44.134584+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:13:06.358109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:13:06.358109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0026724549999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":108,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-09-10T19:15:44.142484+0000","last_change":"2024-09-10T19:13:54.304493+0000","last_active":"2024-09-10T19:15:44.142484+0000","last_peered":"2024-09-10T19:15:44.142484+0000","last_clean":"2024-09-10T19:15:44.142484+0000","last_became_active":"2024-09-10T19:10:14.053736+0000","last_became_peered":"2024-09-10T19:10:14.053736+0000","last_unstale":"2024-09-10T19:15:44.142484+0000","last_undegraded":"2024-09-10T19:15:44.142484+0000","last_fullsized":"2024-09-10T19:15:44.142484+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:13:54.304435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:13:54.304435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0011043820000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.2","version":"144'17838","reported_seq":30657,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-09-10T19:15:45.308861+0000","last_change":"2024-09-10T19:14:46.766991+0000","last_active":"2024-09-10T19:15:45.308861+0000","last_peered":"2024-09-10T19:15:45.308861+0000","last_clean":"2024-09-10T19:15:45.308861+0000","last_became_active":"2024-09-10T19:14:46.726836+0000","last_became_peered":"2024-09-10T19:14:46.726836+0000","last_unstale":"2024-09-10T19:15:45.308861+0000","last_undegraded":"2024-09-10T19:15:45.308861+0000","last_fullsized":"2024-09-10T19:15:45.308861+0000","mapping_epoch":126,"log_start":"128'13607","ondisk_log_start":"128'13607","created":86,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:12:10.007465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:12:10.007465+0000","last_clean_scrub_stamp":"2024-09-10T19:12:10.007465+0000","objects_scrubbed":0,"log_size":4231,"log_dups_size":0,"ondisk_log_size":4231,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17838,"num_write_kb":570816,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6361,"num_bytes_recovered":416874496,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7],"acting":[3,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"66'14","reported_seq":53,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-09-10T19:15:44.135582+0000","last_change":"2024-09-10T19:15:41.748183+0000","last_active":"2024-09-10T19:15:44.135582+0000","last_peered":"2024-09-10T19:15:44.135582+0000","last_clean":"2024-09-10T19:15:44.135582+0000","last_became_active":"2024-09-10T19:15:00.742474+0000","last_became_peered":"2024-09-10T19:15:00.742474+0000","last_unstale":"2024-09-10T19:15:44.135582+0000","last_undegraded":"2024-09-10T19:15:44.135582+0000","last_fullsized":"2024-09-10T19:15:44.135582+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'14","last_scrub_stamp":"2024-09-10T19:15:41.748101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:15:41.748101+0000","objects_scrubbed":7,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:16:54.266006+0000","scrub_duration":0.0055403099999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":114,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-09-10T19:15:44.134617+0000","last_change":"2024-09-10T19:12:52.885643+0000","last_active":"2024-09-10T19:15:44.134617+0000","last_peered":"2024-09-10T19:15:44.134617+0000","last_clean":"2024-09-10T19:15:44.134617+0000","last_became_active":"2024-09-10T19:10:07.684864+0000","last_became_peered":"2024-09-10T19:10:07.684864+0000","last_unstale":"2024-09-10T19:15:44.134617+0000","last_undegraded":"2024-09-10T19:15:44.134617+0000","last_fullsized":"2024-09-10T19:15:44.134617+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:12:52.885563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:12:52.885563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0035531249999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.0","version":"117'10","reported_seq":208,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-09-10T19:15:45.137954+0000","last_change":"2024-09-10T19:15:35.973188+0000","last_active":"2024-09-10T19:15:45.137954+0000","last_peered":"2024-09-10T19:15:45.137954+0000","last_clean":"2024-09-10T19:15:45.137954+0000","last_became_active":"2024-09-10T19:13:54.709846+0000","last_became_peered":"2024-09-10T19:13:54.709846+0000","last_unstale":"2024-09-10T19:15:45.137954+0000","last_undegraded":"2024-09-10T19:15:45.137954+0000","last_fullsized":"2024-09-10T19:15:45.137954+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"117'10","last_scrub_stamp":"2024-09-10T19:15:35.973095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:15:35.973095+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:16:55.086873+0000","scrub_duration":0.0076338860000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,0],"acting":[3,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":108,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-09-10T19:15:44.137000+0000","last_change":"2024-09-10T19:13:15.350815+0000","last_active":"2024-09-10T19:15:44.137000+0000","last_peered":"2024-09-10T19:15:44.137000+0000","last_clean":"2024-09-10T19:15:44.137000+0000","last_became_active":"2024-09-10T19:10:14.054456+0000","last_became_peered":"2024-09-10T19:10:14.054456+0000","last_unstale":"2024-09-10T19:15:44.137000+0000","last_undegraded":"2024-09-10T19:15:44.137000+0000","last_fullsized":"2024-09-10T19:15:44.137000+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:13:15.350720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:11:41.073833+0000","last_clean_scrub_stamp":"2024-09-10T19:13:15.350720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0014361000000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.3","version":"144'17490","reported_seq":17899,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-09-10T19:15:45.287518+0000","last_change":"2024-09-10T19:13:44.417102+0000","last_active":"2024-09-10T19:15:45.287518+0000","last_peered":"2024-09-10T19:15:45.287518+0000","last_clean":"2024-09-10T19:15:45.287518+0000","last_became_active":"2024-09-10T19:12:11.029906+0000","last_became_peered":"2024-09-10T19:12:11.029906+0000","last_unstale":"2024-09-10T19:15:45.287518+0000","last_undegraded":"2024-09-10T19:15:45.287518+0000","last_fullsized":"2024-09-10T19:15:45.287518+0000","mapping_epoch":86,"log_start":"128'13233","ondisk_log_start":"128'13233","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"112'8444","last_scrub_stamp":"2024-09-10T19:13:44.417011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:12:10.007465+0000","last_clean_scrub_stamp":"2024-09-10T19:13:44.417011+0000","objects_scrubbed":8333,"log_size":4257,"log_dups_size":0,"ondisk_log_size":4257,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"queued for scrub","scrub_duration":3.2086845460000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17490,"num_write_kb":559680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"66'18","reported_seq":144,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-09-10T19:15:44.142463+0000","last_change":"2024-09-10T19:12:49.699111+0000","last_active":"2024-09-10T19:15:44.142463+0000","last_peered":"2024-09-10T19:15:44.142463+0000","last_clean":"2024-09-10T19:15:44.142463+0000","last_became_active":"2024-09-10T19:10:09.821836+0000","last_became_peered":"2024-09-10T19:10:09.821836+0000","last_unstale":"2024-09-10T19:15:44.142463+0000","last_undegraded":"2024-09-10T19:15:44.142463+0000","last_fullsized":"2024-09-10T19:15:44.142463+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'18","last_scrub_stamp":"2024-09-10T19:12:49.699054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:12:49.699054+0000","objects_scrubbed":8,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0073070890000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":74,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-09-10T19:15:44.130732+0000","last_change":"2024-09-10T19:15:42.739879+0000","last_active":"2024-09-10T19:15:44.130732+0000","last_peered":"2024-09-10T19:15:44.130732+0000","last_clean":"2024-09-10T19:15:44.130732+0000","last_became_active":"2024-09-10T19:13:54.710968+0000","last_became_peered":"2024-09-10T19:13:54.710968+0000","last_unstale":"2024-09-10T19:15:44.130732+0000","last_undegraded":"2024-09-10T19:15:44.130732+0000","last_fullsized":"2024-09-10T19:15:44.130732+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:15:42.739801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:15:42.739801+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:17:05.377168+0000","scrub_duration":0.008487784,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,3],"acting":[6,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":109,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-09-10T19:15:44.142186+0000","last_change":"2024-09-10T19:12:58.400261+0000","last_active":"2024-09-10T19:15:44.142186+0000","last_peered":"2024-09-10T19:15:44.142186+0000","last_clean":"2024-09-10T19:15:44.142186+0000","last_became_active":"2024-09-10T19:10:11.698054+0000","last_became_peered":"2024-09-10T19:10:11.698054+0000","last_unstale":"2024-09-10T19:15:44.142186+0000","last_undegraded":"2024-09-10T19:15:44.142186+0000","last_fullsized":"2024-09-10T19:15:44.142186+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:12:58.400206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:12:58.400206+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0071235179999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":107,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-09-10T19:15:44.142353+0000","last_change":"2024-09-10T19:12:57.409831+0000","last_active":"2024-09-10T19:15:44.142353+0000","last_peered":"2024-09-10T19:15:44.142353+0000","last_clean":"2024-09-10T19:15:44.142353+0000","last_became_active":"2024-09-10T19:10:13.722784+0000","last_became_peered":"2024-09-10T19:10:13.722784+0000","last_unstale":"2024-09-10T19:15:44.142353+0000","last_undegraded":"2024-09-10T19:15:44.142353+0000","last_fullsized":"2024-09-10T19:15:44.142353+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:12:57.409749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:12:57.409749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0087033260000000008,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.0","version":"144'17480","reported_seq":27109,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-09-10T19:15:45.300852+0000","last_change":"2024-09-10T19:14:57.868575+0000","last_active":"2024-09-10T19:15:45.300852+0000","last_peered":"2024-09-10T19:15:45.300852+0000","last_clean":"2024-09-10T19:15:45.300852+0000","last_became_active":"2024-09-10T19:14:57.844102+0000","last_became_peered":"2024-09-10T19:14:57.844102+0000","last_unstale":"2024-09-10T19:15:45.300852+0000","last_undegraded":"2024-09-10T19:15:45.300852+0000","last_fullsized":"2024-09-10T19:15:45.300852+0000","mapping_epoch":131,"log_start":"123'12473","ondisk_log_start":"123'12473","created":86,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:12:10.007465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:12:10.007465+0000","last_clean_scrub_stamp":"2024-09-10T19:12:10.007465+0000","objects_scrubbed":0,"log_size":5007,"log_dups_size":0,"ondisk_log_size":5007,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17480,"num_write_kb":559360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4764,"num_bytes_recovered":312213504,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.3","version":"104'27","reported_seq":156,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-09-10T19:15:44.138574+0000","last_change":"2024-09-10T19:13:02.226033+0000","last_active":"2024-09-10T19:15:44.138574+0000","last_peered":"2024-09-10T19:15:44.138574+0000","last_clean":"2024-09-10T19:15:44.138574+0000","last_became_active":"2024-09-10T19:10:09.823003+0000","last_became_peered":"2024-09-10T19:10:09.823003+0000","last_unstale":"2024-09-10T19:15:44.138574+0000","last_undegraded":"2024-09-10T19:15:44.138574+0000","last_fullsized":"2024-09-10T19:15:44.138574+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"104'27","last_scrub_stamp":"2024-09-10T19:13:02.225944+0000","last_deep_scrub":"104'27","last_deep_scrub_stamp":"2024-09-10T19:13:02.225944+0000","last_clean_scrub_stamp":"2024-09-10T19:13:02.225944+0000","objects_scrubbed":8,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0099233020000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":114,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-09-10T19:15:44.137411+0000","last_change":"2024-09-10T19:12:53.513000+0000","last_active":"2024-09-10T19:15:44.137411+0000","last_peered":"2024-09-10T19:15:44.137411+0000","last_clean":"2024-09-10T19:15:44.137411+0000","last_became_active":"2024-09-10T19:10:07.690325+0000","last_became_peered":"2024-09-10T19:10:07.690325+0000","last_unstale":"2024-09-10T19:15:44.137411+0000","last_undegraded":"2024-09-10T19:15:44.137411+0000","last_fullsized":"2024-09-10T19:15:44.137411+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:12:53.512923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:12:53.512923+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0087393290000000005,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":110,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-09-10T19:15:44.130264+0000","last_change":"2024-09-10T19:12:46.963842+0000","last_active":"2024-09-10T19:15:44.130264+0000","last_peered":"2024-09-10T19:15:44.130264+0000","last_clean":"2024-09-10T19:15:44.130264+0000","last_became_active":"2024-09-10T19:10:11.700370+0000","last_became_peered":"2024-09-10T19:10:11.700370+0000","last_unstale":"2024-09-10T19:15:44.130264+0000","last_undegraded":"2024-09-10T19:15:44.130264+0000","last_fullsized":"2024-09-10T19:15:44.130264+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:12:46.963746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:12:46.963746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001460004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":74,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-09-10T19:15:44.130447+0000","last_change":"2024-09-10T19:15:41.749437+0000","last_active":"2024-09-10T19:15:44.130447+0000","last_peered":"2024-09-10T19:15:44.130447+0000","last_clean":"2024-09-10T19:15:44.130447+0000","last_became_active":"2024-09-10T19:13:54.717977+0000","last_became_peered":"2024-09-10T19:13:54.717977+0000","last_unstale":"2024-09-10T19:15:44.130447+0000","last_undegraded":"2024-09-10T19:15:44.130447+0000","last_fullsized":"2024-09-10T19:15:44.130447+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:15:41.749338+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:15:41.749338+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:16:43.216724+0000","scrub_duration":0.0018841260000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,7],"acting":[6,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.1","version":"144'17396","reported_seq":17808,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-09-10T19:15:45.315809+0000","last_change":"2024-09-10T19:13:45.105571+0000","last_active":"2024-09-10T19:15:45.315809+0000","last_peered":"2024-09-10T19:15:45.315809+0000","last_clean":"2024-09-10T19:15:45.315809+0000","last_became_active":"2024-09-10T19:12:11.030399+0000","last_became_peered":"2024-09-10T19:12:11.030399+0000","last_unstale":"2024-09-10T19:15:45.315809+0000","last_undegraded":"2024-09-10T19:15:45.315809+0000","last_fullsized":"2024-09-10T19:15:45.315809+0000","mapping_epoch":86,"log_start":"123'12378","ondisk_log_start":"123'12378","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"113'8485","last_scrub_stamp":"2024-09-10T19:13:45.105477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:12:10.007465+0000","last_clean_scrub_stamp":"2024-09-10T19:13:45.105477+0000","objects_scrubbed":8400,"log_size":5018,"log_dups_size":0,"ondisk_log_size":5018,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"queued for scrub","scrub_duration":2.6533322940000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17396,"num_write_kb":556672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.2","version":"66'10","reported_seq":80,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-09-10T19:15:44.134436+0000","last_change":"2024-09-10T19:13:54.748881+0000","last_active":"2024-09-10T19:15:44.134436+0000","last_peered":"2024-09-10T19:15:44.134436+0000","last_clean":"2024-09-10T19:15:44.134436+0000","last_became_active":"2024-09-10T19:13:54.714542+0000","last_became_peered":"2024-09-10T19:13:54.714542+0000","last_unstale":"2024-09-10T19:15:44.134436+0000","last_undegraded":"2024-09-10T19:15:44.134436+0000","last_fullsized":"2024-09-10T19:15:44.134436+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'10","last_scrub_stamp":"2024-09-10T19:12:56.360409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:12:56.360409+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":14,"num_read_kb":9,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,0],"acting":[5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":74,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-09-10T19:15:44.138714+0000","last_change":"2024-09-10T19:15:41.870492+0000","last_active":"2024-09-10T19:15:44.138714+0000","last_peered":"2024-09-10T19:15:44.138714+0000","last_clean":"2024-09-10T19:15:44.138714+0000","last_became_active":"2024-09-10T19:13:54.718267+0000","last_became_peered":"2024-09-10T19:13:54.718267+0000","last_unstale":"2024-09-10T19:15:44.138714+0000","last_undegraded":"2024-09-10T19:15:44.138714+0000","last_fullsized":"2024-09-10T19:15:44.138714+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:15:41.870422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:15:41.870422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:16:51.942406+0000","scrub_duration":0.0061607199999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3","version":"84'9","reported_seq":187,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-09-10T19:15:45.142144+0000","last_change":"2024-09-10T19:13:45.236207+0000","last_active":"2024-09-10T19:15:45.142144+0000","last_peered":"2024-09-10T19:15:45.142144+0000","last_clean":"2024-09-10T19:15:45.142144+0000","last_became_active":"2024-09-10T19:10:11.699690+0000","last_became_peered":"2024-09-10T19:10:11.699690+0000","last_unstale":"2024-09-10T19:15:45.142144+0000","last_undegraded":"2024-09-10T19:15:45.142144+0000","last_fullsized":"2024-09-10T19:15:45.142144+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'9","last_scrub_stamp":"2024-09-10T19:13:45.236094+0000","last_deep_scrub":"84'9","last_deep_scrub_stamp":"2024-09-10T19:11:53.879272+0000","last_clean_scrub_stamp":"2024-09-10T19:13:45.236094+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.036165602999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":108,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-09-10T19:15:44.139096+0000","last_change":"2024-09-10T19:13:40.234378+0000","last_active":"2024-09-10T19:15:44.139096+0000","last_peered":"2024-09-10T19:15:44.139096+0000","last_clean":"2024-09-10T19:15:44.139096+0000","last_became_active":"2024-09-10T19:10:13.720403+0000","last_became_peered":"2024-09-10T19:10:13.720403+0000","last_unstale":"2024-09-10T19:15:44.139096+0000","last_undegraded":"2024-09-10T19:15:44.139096+0000","last_fullsized":"2024-09-10T19:15:44.139096+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:13:40.234289+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:13:40.234289+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0086405590000000008,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.7","version":"144'17482","reported_seq":17891,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-09-10T19:15:45.318326+0000","last_change":"2024-09-10T19:13:55.214546+0000","last_active":"2024-09-10T19:15:45.318326+0000","last_peered":"2024-09-10T19:15:45.318326+0000","last_clean":"2024-09-10T19:15:45.318326+0000","last_became_active":"2024-09-10T19:12:11.032857+0000","last_became_peered":"2024-09-10T19:12:11.032857+0000","last_unstale":"2024-09-10T19:15:45.318326+0000","last_undegraded":"2024-09-10T19:15:45.318326+0000","last_fullsized":"2024-09-10T19:15:45.318326+0000","mapping_epoch":86,"log_start":"123'12425","ondisk_log_start":"123'12425","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"114'9143","last_scrub_stamp":"2024-09-10T19:13:55.214477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:12:10.007465+0000","last_clean_scrub_stamp":"2024-09-10T19:13:55.214477+0000","objects_scrubbed":8339,"log_size":5057,"log_dups_size":0,"ondisk_log_size":5057,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.96843253399999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17482,"num_write_kb":559424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.4","version":"117'30","reported_seq":184,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-09-10T19:15:45.143956+0000","last_change":"2024-09-10T19:13:54.752131+0000","last_active":"2024-09-10T19:15:45.143956+0000","last_peered":"2024-09-10T19:15:45.143956+0000","last_clean":"2024-09-10T19:15:45.143956+0000","last_became_active":"2024-09-10T19:13:54.715642+0000","last_became_peered":"2024-09-10T19:13:54.715642+0000","last_unstale":"2024-09-10T19:15:45.143956+0000","last_undegraded":"2024-09-10T19:15:45.143956+0000","last_fullsized":"2024-09-10T19:15:45.143956+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'28","last_scrub_stamp":"2024-09-10T19:12:45.260189+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:12:45.260189+0000","objects_scrubbed":0,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":47,"num_read_kb":32,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":358,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":121,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-09-10T19:15:44.135192+0000","last_change":"2024-09-10T19:13:54.730667+0000","last_active":"2024-09-10T19:15:44.135192+0000","last_peered":"2024-09-10T19:15:44.135192+0000","last_clean":"2024-09-10T19:15:44.135192+0000","last_became_active":"2024-09-10T19:13:54.730096+0000","last_became_peered":"2024-09-10T19:13:54.730096+0000","last_unstale":"2024-09-10T19:15:44.135192+0000","last_undegraded":"2024-09-10T19:15:44.135192+0000","last_fullsized":"2024-09-10T19:15:44.135192+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:11:41.429402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:11:41.429402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0014226359999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":127,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-09-10T19:15:44.138465+0000","last_change":"2024-09-10T19:15:34.802331+0000","last_active":"2024-09-10T19:15:44.138465+0000","last_peered":"2024-09-10T19:15:44.138465+0000","last_clean":"2024-09-10T19:15:44.138465+0000","last_became_active":"2024-09-10T19:13:54.714017+0000","last_became_peered":"2024-09-10T19:13:54.714017+0000","last_unstale":"2024-09-10T19:15:44.138465+0000","last_undegraded":"2024-09-10T19:15:44.138465+0000","last_fullsized":"2024-09-10T19:15:44.138465+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:15:34.802121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:15:34.802121+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:17:01.582331+0000","scrub_duration":0.0042778080000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,6],"acting":[0,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":98,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-09-10T19:15:44.143328+0000","last_change":"2024-09-10T19:12:46.691448+0000","last_active":"2024-09-10T19:15:44.143328+0000","last_peered":"2024-09-10T19:15:44.143328+0000","last_clean":"2024-09-10T19:15:44.143328+0000","last_became_active":"2024-09-10T19:10:13.721744+0000","last_became_peered":"2024-09-10T19:10:13.721744+0000","last_unstale":"2024-09-10T19:15:44.143328+0000","last_undegraded":"2024-09-10T19:15:44.143328+0000","last_fullsized":"2024-09-10T19:15:44.143328+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:12:46.691353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:12:46.691353+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0014482360000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.4","version":"144'17390","reported_seq":30079,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-09-10T19:15:45.290859+0000","last_change":"2024-09-10T19:14:45.751387+0000","last_active":"2024-09-10T19:15:45.290859+0000","last_peered":"2024-09-10T19:15:45.290859+0000","last_clean":"2024-09-10T19:15:45.290859+0000","last_became_active":"2024-09-10T19:14:45.722746+0000","last_became_peered":"2024-09-10T19:14:45.722746+0000","last_unstale":"2024-09-10T19:15:45.290859+0000","last_undegraded":"2024-09-10T19:15:45.290859+0000","last_fullsized":"2024-09-10T19:15:45.290859+0000","mapping_epoch":125,"log_start":"124'12609","ondisk_log_start":"124'12609","created":86,"last_epoch_clean":126,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:12:10.007465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:12:10.007465+0000","last_clean_scrub_stamp":"2024-09-10T19:12:10.007465+0000","objects_scrubbed":0,"log_size":4781,"log_dups_size":0,"ondisk_log_size":4781,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17390,"num_write_kb":556480,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6296,"num_bytes_recovered":412483584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,6],"acting":[0,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.7","version":"82'18","reported_seq":137,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-09-10T19:15:44.134870+0000","last_change":"2024-09-10T19:13:54.737836+0000","last_active":"2024-09-10T19:15:44.134870+0000","last_peered":"2024-09-10T19:15:44.134870+0000","last_clean":"2024-09-10T19:15:44.134870+0000","last_became_active":"2024-09-10T19:13:54.712065+0000","last_became_peered":"2024-09-10T19:13:54.712065+0000","last_unstale":"2024-09-10T19:15:44.134870+0000","last_undegraded":"2024-09-10T19:15:44.134870+0000","last_fullsized":"2024-09-10T19:15:44.134870+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'18","last_scrub_stamp":"2024-09-10T19:11:49.207309+0000","last_deep_scrub":"82'18","last_deep_scrub_stamp":"2024-09-10T19:11:49.207309+0000","last_clean_scrub_stamp":"2024-09-10T19:11:49.207309+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":19,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.0","version":"67'5","reported_seq":288,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-09-10T19:15:48.262020+0000","last_change":"2024-09-10T19:15:30.760345+0000","last_active":"2024-09-10T19:15:48.262020+0000","last_peered":"2024-09-10T19:15:48.262020+0000","last_clean":"2024-09-10T19:15:48.262020+0000","last_became_active":"2024-09-10T19:13:54.715570+0000","last_became_peered":"2024-09-10T19:13:54.715570+0000","last_unstale":"2024-09-10T19:15:48.262020+0000","last_undegraded":"2024-09-10T19:15:48.262020+0000","last_fullsized":"2024-09-10T19:15:48.262020+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'5","last_scrub_stamp":"2024-09-10T19:15:30.759591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:15:30.759591+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:16:38.105109+0000","scrub_duration":0.010111248999999999,"objects_trimmed":0,"snaptrim_duration":0.00034454499999999997,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":185,"num_read_kb":180,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":389,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,0],"acting":[2,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":114,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-09-10T19:15:44.135886+0000","last_change":"2024-09-10T19:12:53.318459+0000","last_active":"2024-09-10T19:15:44.135886+0000","last_peered":"2024-09-10T19:15:44.135886+0000","last_clean":"2024-09-10T19:15:44.135886+0000","last_became_active":"2024-09-10T19:10:07.681456+0000","last_became_peered":"2024-09-10T19:10:07.681456+0000","last_unstale":"2024-09-10T19:15:44.135886+0000","last_undegraded":"2024-09-10T19:15:44.135886+0000","last_fullsized":"2024-09-10T19:15:44.135886+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:12:53.318408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:12:53.318408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.01029277,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":110,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-09-10T19:15:44.134284+0000","last_change":"2024-09-10T19:13:00.366695+0000","last_active":"2024-09-10T19:15:44.134284+0000","last_peered":"2024-09-10T19:15:44.134284+0000","last_clean":"2024-09-10T19:15:44.134284+0000","last_became_active":"2024-09-10T19:10:11.695205+0000","last_became_peered":"2024-09-10T19:10:11.695205+0000","last_unstale":"2024-09-10T19:15:44.134284+0000","last_undegraded":"2024-09-10T19:15:44.134284+0000","last_fullsized":"2024-09-10T19:15:44.134284+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:13:00.366610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:13:00.366610+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0023966700000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":111,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-09-10T19:15:44.137273+0000","last_change":"2024-09-10T19:15:38.493179+0000","last_active":"2024-09-10T19:15:44.137273+0000","last_peered":"2024-09-10T19:15:44.137273+0000","last_clean":"2024-09-10T19:15:44.137273+0000","last_became_active":"2024-09-10T19:10:14.053697+0000","last_became_peered":"2024-09-10T19:10:14.053697+0000","last_unstale":"2024-09-10T19:15:44.137273+0000","last_undegraded":"2024-09-10T19:15:44.137273+0000","last_fullsized":"2024-09-10T19:15:44.137273+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:15:38.493102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:15:38.493102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:16:48.733504+0000","scrub_duration":0.0015544280000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.5","version":"144'17660","reported_seq":18074,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-09-10T19:15:45.308511+0000","last_change":"2024-09-10T19:13:55.551583+0000","last_active":"2024-09-10T19:15:45.308511+0000","last_peered":"2024-09-10T19:15:45.308511+0000","last_clean":"2024-09-10T19:15:45.308511+0000","last_became_active":"2024-09-10T19:12:11.032880+0000","last_became_peered":"2024-09-10T19:12:11.032880+0000","last_unstale":"2024-09-10T19:15:45.308511+0000","last_undegraded":"2024-09-10T19:15:45.308511+0000","last_fullsized":"2024-09-10T19:15:45.308511+0000","mapping_epoch":86,"log_start":"128'13355","ondisk_log_start":"128'13355","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"115'9199","last_scrub_stamp":"2024-09-10T19:13:55.551514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:12:10.007465+0000","last_clean_scrub_stamp":"2024-09-10T19:13:55.551514+0000","objects_scrubbed":8461,"log_size":4305,"log_dups_size":0,"ondisk_log_size":4305,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"queued for scrub","scrub_duration":1.028952525,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17660,"num_write_kb":565120,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"66'12","reported_seq":161,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-09-10T19:15:44.139609+0000","last_change":"2024-09-10T19:15:40.903370+0000","last_active":"2024-09-10T19:15:44.139609+0000","last_peered":"2024-09-10T19:15:44.139609+0000","last_clean":"2024-09-10T19:15:44.139609+0000","last_became_active":"2024-09-10T19:13:54.719200+0000","last_became_peered":"2024-09-10T19:13:54.719200+0000","last_unstale":"2024-09-10T19:15:44.139609+0000","last_undegraded":"2024-09-10T19:15:44.139609+0000","last_fullsized":"2024-09-10T19:15:44.139609+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'12","last_scrub_stamp":"2024-09-10T19:15:40.903279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:15:40.903279+0000","objects_scrubbed":6,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:17:02.013122+0000","scrub_duration":0.0271238,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,7],"acting":[0,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"61'1","reported_seq":120,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-09-10T19:15:44.138715+0000","last_change":"2024-09-10T19:13:06.265721+0000","last_active":"2024-09-10T19:15:44.138715+0000","last_peered":"2024-09-10T19:15:44.138715+0000","last_clean":"2024-09-10T19:15:44.138715+0000","last_became_active":"2024-09-10T19:10:07.683259+0000","last_became_peered":"2024-09-10T19:10:07.683259+0000","last_unstale":"2024-09-10T19:15:44.138715+0000","last_undegraded":"2024-09-10T19:15:44.138715+0000","last_fullsized":"2024-09-10T19:15:44.138715+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'1","last_scrub_stamp":"2024-09-10T19:13:06.265646+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:13:06.265646+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.012205384,"objects_trimmed":0,"snaptrim_duration":0.0014341130000000001,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":131,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-09-10T19:15:44.136762+0000","last_change":"2024-09-10T19:15:30.542477+0000","last_active":"2024-09-10T19:15:44.136762+0000","last_peered":"2024-09-10T19:15:44.136762+0000","last_clean":"2024-09-10T19:15:44.136762+0000","last_became_active":"2024-09-10T19:13:54.721486+0000","last_became_peered":"2024-09-10T19:13:54.721486+0000","last_unstale":"2024-09-10T19:15:44.136762+0000","last_undegraded":"2024-09-10T19:15:44.136762+0000","last_fullsized":"2024-09-10T19:15:44.136762+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:15:30.542399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:15:30.542399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:17:00.089482+0000","scrub_duration":0.0025142609999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":127,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-09-10T19:15:44.134654+0000","last_change":"2024-09-10T19:13:54.712428+0000","last_active":"2024-09-10T19:15:44.134654+0000","last_peered":"2024-09-10T19:15:44.134654+0000","last_clean":"2024-09-10T19:15:44.134654+0000","last_became_active":"2024-09-10T19:13:54.711685+0000","last_became_peered":"2024-09-10T19:13:54.711685+0000","last_unstale":"2024-09-10T19:15:44.134654+0000","last_undegraded":"2024-09-10T19:15:44.134654+0000","last_fullsized":"2024-09-10T19:15:44.134654+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:12:44.527561+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:12:44.527561+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.022638367999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,4],"acting":[5,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":64,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-09-10T19:15:44.134399+0000","last_change":"2024-09-10T19:13:54.723228+0000","last_active":"2024-09-10T19:15:44.134399+0000","last_peered":"2024-09-10T19:15:44.134399+0000","last_clean":"2024-09-10T19:15:44.134399+0000","last_became_active":"2024-09-10T19:13:54.722937+0000","last_became_peered":"2024-09-10T19:13:54.722937+0000","last_unstale":"2024-09-10T19:15:44.134399+0000","last_undegraded":"2024-09-10T19:15:44.134399+0000","last_fullsized":"2024-09-10T19:15:44.134399+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:12:05.165500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:12:05.165500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.6","version":"144'17304","reported_seq":17373,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-09-10T19:15:45.281184+0000","last_change":"2024-09-10T19:13:43.239851+0000","last_active":"2024-09-10T19:15:45.281184+0000","last_peered":"2024-09-10T19:15:45.281184+0000","last_clean":"2024-09-10T19:15:45.281184+0000","last_became_active":"2024-09-10T19:12:11.035290+0000","last_became_peered":"2024-09-10T19:12:11.035290+0000","last_unstale":"2024-09-10T19:15:45.281184+0000","last_undegraded":"2024-09-10T19:15:45.281184+0000","last_fullsized":"2024-09-10T19:15:45.281184+0000","mapping_epoch":86,"log_start":"124'12542","ondisk_log_start":"124'12542","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:12:10.007465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:12:10.007465+0000","last_clean_scrub_stamp":"2024-09-10T19:12:10.007465+0000","objects_scrubbed":0,"log_size":4762,"log_dups_size":0,"ondisk_log_size":4762,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17304,"num_write_kb":553728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.5","version":"80'21","reported_seq":151,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-09-10T19:15:49.682199+0000","last_change":"2024-09-10T19:15:49.682164+0000","last_active":"2024-09-10T19:15:49.682199+0000","last_peered":"2024-09-10T19:15:49.682199+0000","last_clean":"2024-09-10T19:15:49.682199+0000","last_became_active":"2024-09-10T19:10:09.822184+0000","last_became_peered":"2024-09-10T19:10:09.822184+0000","last_unstale":"2024-09-10T19:15:49.682199+0000","last_undegraded":"2024-09-10T19:15:49.682199+0000","last_fullsized":"2024-09-10T19:15:49.682199+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"80'21","last_scrub_stamp":"2024-09-10T19:15:49.682092+0000","last_deep_scrub":"80'21","last_deep_scrub_stamp":"2024-09-10T19:11:39.836144+0000","last_clean_scrub_stamp":"2024-09-10T19:15:49.682092+0000","objects_scrubbed":7,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:17:05.534997+0000","scrub_duration":0.0022142529999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"63'2","reported_seq":121,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-09-10T19:15:44.142906+0000","last_change":"2024-09-10T19:13:00.752806+0000","last_active":"2024-09-10T19:15:44.142906+0000","last_peered":"2024-09-10T19:15:44.142906+0000","last_clean":"2024-09-10T19:15:44.142906+0000","last_became_active":"2024-09-10T19:10:07.684784+0000","last_became_peered":"2024-09-10T19:10:07.684784+0000","last_unstale":"2024-09-10T19:15:44.142906+0000","last_undegraded":"2024-09-10T19:15:44.142906+0000","last_fullsized":"2024-09-10T19:15:44.142906+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'2","last_scrub_stamp":"2024-09-10T19:13:00.752715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:13:00.752715+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0083291979999999995,"objects_trimmed":0,"snaptrim_duration":0.00075522999999999996,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":124,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-09-10T19:15:44.134443+0000","last_change":"2024-09-10T19:13:50.411573+0000","last_active":"2024-09-10T19:15:44.134443+0000","last_peered":"2024-09-10T19:15:44.134443+0000","last_clean":"2024-09-10T19:15:44.134443+0000","last_became_active":"2024-09-10T19:10:07.687596+0000","last_became_peered":"2024-09-10T19:10:07.687596+0000","last_unstale":"2024-09-10T19:15:44.134443+0000","last_undegraded":"2024-09-10T19:15:44.134443+0000","last_fullsized":"2024-09-10T19:15:44.134443+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:13:50.411500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:13:50.411500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0032604869999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"74'79","reported_seq":159,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-09-10T19:15:44.136907+0000","last_change":"2024-09-10T19:13:46.419002+0000","last_active":"2024-09-10T19:15:44.136907+0000","last_peered":"2024-09-10T19:15:44.136907+0000","last_clean":"2024-09-10T19:15:44.136907+0000","last_became_active":"2024-09-10T19:09:56.634919+0000","last_became_peered":"2024-09-10T19:09:56.634919+0000","last_unstale":"2024-09-10T19:15:44.136907+0000","last_undegraded":"2024-09-10T19:15:44.136907+0000","last_fullsized":"2024-09-10T19:15:44.136907+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'79","last_scrub_stamp":"2024-09-10T19:13:46.418921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:08:00.699315+0000","last_clean_scrub_stamp":"2024-09-10T19:13:46.418921+0000","objects_scrubbed":2,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0029827999999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":110,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-09-10T19:15:44.136707+0000","last_change":"2024-09-10T19:13:47.453309+0000","last_active":"2024-09-10T19:15:44.136707+0000","last_peered":"2024-09-10T19:15:44.136707+0000","last_clean":"2024-09-10T19:15:44.136707+0000","last_became_active":"2024-09-10T19:10:11.694986+0000","last_became_peered":"2024-09-10T19:10:11.694986+0000","last_unstale":"2024-09-10T19:15:44.136707+0000","last_undegraded":"2024-09-10T19:15:44.136707+0000","last_fullsized":"2024-09-10T19:15:44.136707+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:13:47.453228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:13:47.453228+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.010282463,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":108,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-09-10T19:15:44.134949+0000","last_change":"2024-09-10T19:13:05.390398+0000","last_active":"2024-09-10T19:15:44.134949+0000","last_peered":"2024-09-10T19:15:44.134949+0000","last_clean":"2024-09-10T19:15:44.134949+0000","last_became_active":"2024-09-10T19:10:13.720647+0000","last_became_peered":"2024-09-10T19:10:13.720647+0000","last_unstale":"2024-09-10T19:15:44.134949+0000","last_undegraded":"2024-09-10T19:15:44.134949+0000","last_fullsized":"2024-09-10T19:15:44.134949+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:13:05.390317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:13:05.390317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0058030640000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.9","version":"144'17414","reported_seq":27376,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-09-10T19:15:45.297512+0000","last_change":"2024-09-10T19:14:56.867726+0000","last_active":"2024-09-10T19:15:45.297512+0000","last_peered":"2024-09-10T19:15:45.297512+0000","last_clean":"2024-09-10T19:15:45.297512+0000","last_became_active":"2024-09-10T19:14:56.842529+0000","last_became_peered":"2024-09-10T19:14:56.842529+0000","last_unstale":"2024-09-10T19:15:45.297512+0000","last_undegraded":"2024-09-10T19:15:45.297512+0000","last_fullsized":"2024-09-10T19:15:45.297512+0000","mapping_epoch":130,"log_start":"125'12685","ondisk_log_start":"125'12685","created":86,"last_epoch_clean":131,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:12:10.007465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:12:10.007465+0000","last_clean_scrub_stamp":"2024-09-10T19:12:10.007465+0000","objects_scrubbed":0,"log_size":4729,"log_dups_size":0,"ondisk_log_size":4729,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17414,"num_write_kb":557248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4930,"num_bytes_recovered":323026944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"66'19","reported_seq":180,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-09-10T19:15:44.130402+0000","last_change":"2024-09-10T19:15:38.722679+0000","last_active":"2024-09-10T19:15:44.130402+0000","last_peered":"2024-09-10T19:15:44.130402+0000","last_clean":"2024-09-10T19:15:44.130402+0000","last_became_active":"2024-09-10T19:13:54.731274+0000","last_became_peered":"2024-09-10T19:13:54.731274+0000","last_unstale":"2024-09-10T19:15:44.130402+0000","last_undegraded":"2024-09-10T19:15:44.130402+0000","last_fullsized":"2024-09-10T19:15:44.130402+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'19","last_scrub_stamp":"2024-09-10T19:15:38.722581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:15:38.722581+0000","objects_scrubbed":9,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:16:45.004511+0000","scrub_duration":0.0054904940000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,5],"acting":[6,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":114,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-09-10T19:15:44.134325+0000","last_change":"2024-09-10T19:13:05.404535+0000","last_active":"2024-09-10T19:15:44.134325+0000","last_peered":"2024-09-10T19:15:44.134325+0000","last_clean":"2024-09-10T19:15:44.134325+0000","last_became_active":"2024-09-10T19:10:07.686236+0000","last_became_peered":"2024-09-10T19:10:07.686236+0000","last_unstale":"2024-09-10T19:15:44.134325+0000","last_undegraded":"2024-09-10T19:15:44.134325+0000","last_fullsized":"2024-09-10T19:15:44.134325+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:13:05.404457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:13:05.404457+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0037223199999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":110,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-09-10T19:15:44.135569+0000","last_change":"2024-09-10T19:12:55.335929+0000","last_active":"2024-09-10T19:15:44.135569+0000","last_peered":"2024-09-10T19:15:44.135569+0000","last_clean":"2024-09-10T19:15:44.135569+0000","last_became_active":"2024-09-10T19:10:11.699745+0000","last_became_peered":"2024-09-10T19:10:11.699745+0000","last_unstale":"2024-09-10T19:15:44.135569+0000","last_undegraded":"2024-09-10T19:15:44.135569+0000","last_fullsized":"2024-09-10T19:15:44.135569+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:12:55.335834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:12:55.335834+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0060173240000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":108,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-09-10T19:15:44.136831+0000","last_change":"2024-09-10T19:13:14.311675+0000","last_active":"2024-09-10T19:15:44.136831+0000","last_peered":"2024-09-10T19:15:44.136831+0000","last_clean":"2024-09-10T19:15:44.136831+0000","last_became_active":"2024-09-10T19:10:13.719973+0000","last_became_peered":"2024-09-10T19:10:13.719973+0000","last_unstale":"2024-09-10T19:15:44.136831+0000","last_undegraded":"2024-09-10T19:15:44.136831+0000","last_fullsized":"2024-09-10T19:15:44.136831+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:13:14.311620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:13:14.311620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0054101059999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.8","version":"144'17656","reported_seq":18082,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-09-10T19:15:45.312455+0000","last_change":"2024-09-10T19:13:49.740030+0000","last_active":"2024-09-10T19:15:45.312455+0000","last_peered":"2024-09-10T19:15:45.312455+0000","last_clean":"2024-09-10T19:15:45.312455+0000","last_became_active":"2024-09-10T19:12:11.031441+0000","last_became_peered":"2024-09-10T19:12:11.031441+0000","last_unstale":"2024-09-10T19:15:45.312455+0000","last_undegraded":"2024-09-10T19:15:45.312455+0000","last_fullsized":"2024-09-10T19:15:45.312455+0000","mapping_epoch":86,"log_start":"123'12638","ondisk_log_start":"123'12638","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"113'8888","last_scrub_stamp":"2024-09-10T19:13:49.739947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:12:10.007465+0000","last_clean_scrub_stamp":"2024-09-10T19:13:49.739947+0000","objects_scrubbed":8760,"log_size":5018,"log_dups_size":0,"ondisk_log_size":5018,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"queued for scrub","scrub_duration":3.4609629229999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17656,"num_write_kb":564992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.b","version":"66'9","reported_seq":155,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-09-10T19:15:44.138497+0000","last_change":"2024-09-10T19:15:37.846895+0000","last_active":"2024-09-10T19:15:44.138497+0000","last_peered":"2024-09-10T19:15:44.138497+0000","last_clean":"2024-09-10T19:15:44.138497+0000","last_became_active":"2024-09-10T19:13:54.710613+0000","last_became_peered":"2024-09-10T19:13:54.710613+0000","last_unstale":"2024-09-10T19:15:44.138497+0000","last_undegraded":"2024-09-10T19:15:44.138497+0000","last_fullsized":"2024-09-10T19:15:44.138497+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'9","last_scrub_stamp":"2024-09-10T19:15:37.846807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:15:37.846807+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:16:45.860729+0000","scrub_duration":0.012189228,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":67,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-09-10T19:15:44.142862+0000","last_change":"2024-09-10T19:14:43.036928+0000","last_active":"2024-09-10T19:15:44.142862+0000","last_peered":"2024-09-10T19:15:44.142862+0000","last_clean":"2024-09-10T19:15:44.142862+0000","last_became_active":"2024-09-10T19:13:54.708351+0000","last_became_peered":"2024-09-10T19:13:54.708351+0000","last_unstale":"2024-09-10T19:15:44.142862+0000","last_undegraded":"2024-09-10T19:15:44.142862+0000","last_fullsized":"2024-09-10T19:15:44.142862+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:11:44.254513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:11:44.254513+0000","last_clean_scrub_stamp":"2024-09-10T19:11:44.254513+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":110,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-09-10T19:15:44.140957+0000","last_change":"2024-09-10T19:13:08.398177+0000","last_active":"2024-09-10T19:15:44.140957+0000","last_peered":"2024-09-10T19:15:44.140957+0000","last_clean":"2024-09-10T19:15:44.140957+0000","last_became_active":"2024-09-10T19:10:11.694405+0000","last_became_peered":"2024-09-10T19:10:11.694405+0000","last_unstale":"2024-09-10T19:15:44.140957+0000","last_undegraded":"2024-09-10T19:15:44.140957+0000","last_fullsized":"2024-09-10T19:15:44.140957+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:13:08.398036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:13:08.398036+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.016845473999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":108,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-09-10T19:15:44.138681+0000","last_change":"2024-09-10T19:12:56.145358+0000","last_active":"2024-09-10T19:15:44.138681+0000","last_peered":"2024-09-10T19:15:44.138681+0000","last_clean":"2024-09-10T19:15:44.138681+0000","last_became_active":"2024-09-10T19:10:13.714952+0000","last_became_peered":"2024-09-10T19:10:13.714952+0000","last_unstale":"2024-09-10T19:15:44.138681+0000","last_undegraded":"2024-09-10T19:15:44.138681+0000","last_fullsized":"2024-09-10T19:15:44.138681+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:12:56.145218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:12:56.145218+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0058765429999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.b","version":"144'17498","reported_seq":30499,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-09-10T19:15:45.312929+0000","last_change":"2024-09-10T19:14:46.754630+0000","last_active":"2024-09-10T19:15:45.312929+0000","last_peered":"2024-09-10T19:15:45.312929+0000","last_clean":"2024-09-10T19:15:45.312929+0000","last_became_active":"2024-09-10T19:14:46.725565+0000","last_became_peered":"2024-09-10T19:14:46.725565+0000","last_unstale":"2024-09-10T19:15:45.312929+0000","last_undegraded":"2024-09-10T19:15:45.312929+0000","last_fullsized":"2024-09-10T19:15:45.312929+0000","mapping_epoch":126,"log_start":"123'12494","ondisk_log_start":"123'12494","created":86,"last_epoch_clean":127,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:12:10.007465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:12:10.007465+0000","last_clean_scrub_stamp":"2024-09-10T19:12:10.007465+0000","objects_scrubbed":0,"log_size":5004,"log_dups_size":0,"ondisk_log_size":5004,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17498,"num_write_kb":559936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6453,"num_bytes_recovered":422903808,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.8","version":"66'15","reported_seq":137,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-09-10T19:15:44.134767+0000","last_change":"2024-09-10T19:12:50.449684+0000","last_active":"2024-09-10T19:15:44.134767+0000","last_peered":"2024-09-10T19:15:44.134767+0000","last_clean":"2024-09-10T19:15:44.134767+0000","last_became_active":"2024-09-10T19:10:09.820950+0000","last_became_peered":"2024-09-10T19:10:09.820950+0000","last_unstale":"2024-09-10T19:15:44.134767+0000","last_undegraded":"2024-09-10T19:15:44.134767+0000","last_fullsized":"2024-09-10T19:15:44.134767+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'15","last_scrub_stamp":"2024-09-10T19:12:50.449596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:12:50.449596+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0055729239999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":124,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-09-10T19:15:44.135491+0000","last_change":"2024-09-10T19:15:38.678030+0000","last_active":"2024-09-10T19:15:44.135491+0000","last_peered":"2024-09-10T19:15:44.135491+0000","last_clean":"2024-09-10T19:15:44.135491+0000","last_became_active":"2024-09-10T19:10:07.682286+0000","last_became_peered":"2024-09-10T19:10:07.682286+0000","last_unstale":"2024-09-10T19:15:44.135491+0000","last_undegraded":"2024-09-10T19:15:44.135491+0000","last_fullsized":"2024-09-10T19:15:44.135491+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:15:38.677960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:15:38.677960+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:16:51.670221+0000","scrub_duration":0.00460904,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.9","version":"80'9","reported_seq":187,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-09-10T19:15:45.148727+0000","last_change":"2024-09-10T19:13:43.358984+0000","last_active":"2024-09-10T19:15:45.148727+0000","last_peered":"2024-09-10T19:15:45.148727+0000","last_clean":"2024-09-10T19:15:45.148727+0000","last_became_active":"2024-09-10T19:10:11.702878+0000","last_became_peered":"2024-09-10T19:10:11.702878+0000","last_unstale":"2024-09-10T19:15:45.148727+0000","last_undegraded":"2024-09-10T19:15:45.148727+0000","last_fullsized":"2024-09-10T19:15:45.148727+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"80'9","last_scrub_stamp":"2024-09-10T19:13:43.358901+0000","last_deep_scrub":"80'9","last_deep_scrub_stamp":"2024-09-10T19:11:35.174218+0000","last_clean_scrub_stamp":"2024-09-10T19:13:43.358901+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0066143260000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":108,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-09-10T19:15:44.134219+0000","last_change":"2024-09-10T19:12:57.392898+0000","last_active":"2024-09-10T19:15:44.134219+0000","last_peered":"2024-09-10T19:15:44.134219+0000","last_clean":"2024-09-10T19:15:44.134219+0000","last_became_active":"2024-09-10T19:10:14.053875+0000","last_became_peered":"2024-09-10T19:10:14.053875+0000","last_unstale":"2024-09-10T19:15:44.134219+0000","last_undegraded":"2024-09-10T19:15:44.134219+0000","last_fullsized":"2024-09-10T19:15:44.134219+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:12:57.392844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:12:57.392844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0046180839999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.a","version":"144'17690","reported_seq":18101,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-09-10T19:15:45.305331+0000","last_change":"2024-09-10T19:13:44.230819+0000","last_active":"2024-09-10T19:15:45.305331+0000","last_peered":"2024-09-10T19:15:45.305331+0000","last_clean":"2024-09-10T19:15:45.305331+0000","last_became_active":"2024-09-10T19:12:11.034838+0000","last_became_peered":"2024-09-10T19:12:11.034838+0000","last_unstale":"2024-09-10T19:15:45.305331+0000","last_undegraded":"2024-09-10T19:15:45.305331+0000","last_fullsized":"2024-09-10T19:15:45.305331+0000","mapping_epoch":86,"log_start":"128'13419","ondisk_log_start":"128'13419","created":86,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"112'8534","last_scrub_stamp":"2024-09-10T19:13:44.230739+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:12:10.007465+0000","last_clean_scrub_stamp":"2024-09-10T19:13:44.230739+0000","objects_scrubbed":8364,"log_size":4271,"log_dups_size":0,"ondisk_log_size":4271,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":5,"scrub_schedule":"queued for scrub","scrub_duration":4.047209821,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17690,"num_write_kb":566080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.9","version":"66'12","reported_seq":166,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-09-10T19:15:44.142187+0000","last_change":"2024-09-10T19:15:37.869593+0000","last_active":"2024-09-10T19:15:44.142187+0000","last_peered":"2024-09-10T19:15:44.142187+0000","last_clean":"2024-09-10T19:15:44.142187+0000","last_became_active":"2024-09-10T19:13:54.718152+0000","last_became_peered":"2024-09-10T19:13:54.718152+0000","last_unstale":"2024-09-10T19:15:44.142187+0000","last_undegraded":"2024-09-10T19:15:44.142187+0000","last_fullsized":"2024-09-10T19:15:44.142187+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'12","last_scrub_stamp":"2024-09-10T19:15:37.869451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:15:37.869451+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:16:59.444694+0000","scrub_duration":0.0025441249999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.f","version":"59'2","reported_seq":129,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-09-10T19:15:44.143106+0000","last_change":"2024-09-10T19:12:50.437332+0000","last_active":"2024-09-10T19:15:44.143106+0000","last_peered":"2024-09-10T19:15:44.143106+0000","last_clean":"2024-09-10T19:15:44.143106+0000","last_became_active":"2024-09-10T19:10:07.681719+0000","last_became_peered":"2024-09-10T19:10:07.681719+0000","last_unstale":"2024-09-10T19:15:44.143106+0000","last_undegraded":"2024-09-10T19:15:44.143106+0000","last_fullsized":"2024-09-10T19:15:44.143106+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'2","last_scrub_stamp":"2024-09-10T19:12:50.437245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:12:50.437245+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0070075770000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":14,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":127,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-09-10T19:15:44.136634+0000","last_change":"2024-09-10T19:15:31.776584+0000","last_active":"2024-09-10T19:15:44.136634+0000","last_peered":"2024-09-10T19:15:44.136634+0000","last_clean":"2024-09-10T19:15:44.136634+0000","last_became_active":"2024-09-10T19:13:54.715528+0000","last_became_peered":"2024-09-10T19:13:54.715528+0000","last_unstale":"2024-09-10T19:15:44.136634+0000","last_undegraded":"2024-09-10T19:15:44.136634+0000","last_fullsized":"2024-09-10T19:15:44.136634+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:15:31.776481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:11:46.587846+0000","last_clean_scrub_stamp":"2024-09-10T19:15:31.776481+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:16:33.809672+0000","scrub_duration":0.011259251,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":125,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-09-10T19:15:44.143694+0000","last_change":"2024-09-10T19:15:41.093923+0000","last_active":"2024-09-10T19:15:44.143694+0000","last_peered":"2024-09-10T19:15:44.143694+0000","last_clean":"2024-09-10T19:15:44.143694+0000","last_became_active":"2024-09-10T19:13:54.711804+0000","last_became_peered":"2024-09-10T19:13:54.711804+0000","last_unstale":"2024-09-10T19:15:44.143694+0000","last_undegraded":"2024-09-10T19:15:44.143694+0000","last_fullsized":"2024-09-10T19:15:44.143694+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:15:41.093783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:15:41.093783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:16:51.502004+0000","scrub_duration":0.0023833050000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.16","version":"66'9","reported_seq":138,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-09-10T19:15:44.139531+0000","last_change":"2024-09-10T19:13:50.172901+0000","last_active":"2024-09-10T19:15:44.139531+0000","last_peered":"2024-09-10T19:15:44.139531+0000","last_clean":"2024-09-10T19:15:44.139531+0000","last_became_active":"2024-09-10T19:10:09.822994+0000","last_became_peered":"2024-09-10T19:10:09.822994+0000","last_unstale":"2024-09-10T19:15:44.139531+0000","last_undegraded":"2024-09-10T19:15:44.139531+0000","last_fullsized":"2024-09-10T19:15:44.139531+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'9","last_scrub_stamp":"2024-09-10T19:13:50.172825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:13:50.172825+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0094381619999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":131,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-09-10T19:15:44.136820+0000","last_change":"2024-09-10T19:15:36.662916+0000","last_active":"2024-09-10T19:15:44.136820+0000","last_peered":"2024-09-10T19:15:44.136820+0000","last_clean":"2024-09-10T19:15:44.136820+0000","last_became_active":"2024-09-10T19:13:54.715910+0000","last_became_peered":"2024-09-10T19:13:54.715910+0000","last_unstale":"2024-09-10T19:15:44.136820+0000","last_undegraded":"2024-09-10T19:15:44.136820+0000","last_fullsized":"2024-09-10T19:15:44.136820+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:15:36.662596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:11:41.458523+0000","last_clean_scrub_stamp":"2024-09-10T19:15:36.662596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:16:51.033034+0000","scrub_duration":0.0024511910000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,4],"acting":[2,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":117,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-09-10T19:15:44.143813+0000","last_change":"2024-09-10T19:13:54.712640+0000","last_active":"2024-09-10T19:15:44.143813+0000","last_peered":"2024-09-10T19:15:44.143813+0000","last_clean":"2024-09-10T19:15:44.143813+0000","last_became_active":"2024-09-10T19:13:54.712275+0000","last_became_peered":"2024-09-10T19:13:54.712275+0000","last_unstale":"2024-09-10T19:15:44.143813+0000","last_undegraded":"2024-09-10T19:15:44.143813+0000","last_fullsized":"2024-09-10T19:15:44.143813+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:12:11.619202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:12:11.619202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001950834,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"66'1","reported_seq":119,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-09-10T19:15:48.839669+0000","last_change":"2024-09-10T19:15:48.839638+0000","last_active":"2024-09-10T19:15:48.839669+0000","last_peered":"2024-09-10T19:15:48.839669+0000","last_clean":"2024-09-10T19:15:48.839669+0000","last_became_active":"2024-09-10T19:10:13.717001+0000","last_became_peered":"2024-09-10T19:10:13.717001+0000","last_unstale":"2024-09-10T19:15:48.839669+0000","last_undegraded":"2024-09-10T19:15:48.839669+0000","last_fullsized":"2024-09-10T19:15:48.839669+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'1","last_scrub_stamp":"2024-09-10T19:15:48.839582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:15:48.839582+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:16:48.848204+0000","scrub_duration":0.001851239,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":134,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-09-10T19:15:44.131270+0000","last_change":"2024-09-10T19:15:36.714392+0000","last_active":"2024-09-10T19:15:44.131270+0000","last_peered":"2024-09-10T19:15:44.131270+0000","last_clean":"2024-09-10T19:15:44.131270+0000","last_became_active":"2024-09-10T19:13:54.724027+0000","last_became_peered":"2024-09-10T19:13:54.724027+0000","last_unstale":"2024-09-10T19:15:44.131270+0000","last_undegraded":"2024-09-10T19:15:44.131270+0000","last_fullsized":"2024-09-10T19:15:44.131270+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:15:36.714327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:15:36.714327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:16:55.977022+0000","scrub_duration":0.001831287,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,5],"acting":[6,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"66'6","reported_seq":136,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-09-10T19:15:44.143990+0000","last_change":"2024-09-10T19:13:55.370749+0000","last_active":"2024-09-10T19:15:44.143990+0000","last_peered":"2024-09-10T19:15:44.143990+0000","last_clean":"2024-09-10T19:15:44.143990+0000","last_became_active":"2024-09-10T19:13:54.717949+0000","last_became_peered":"2024-09-10T19:13:54.717949+0000","last_unstale":"2024-09-10T19:15:44.143990+0000","last_undegraded":"2024-09-10T19:15:44.143990+0000","last_fullsized":"2024-09-10T19:15:44.143990+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'6","last_scrub_stamp":"2024-09-10T19:13:55.370650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:13:55.370650+0000","objects_scrubbed":3,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.071693797000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":127,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-09-10T19:15:44.135226+0000","last_change":"2024-09-10T19:15:30.583958+0000","last_active":"2024-09-10T19:15:44.135226+0000","last_peered":"2024-09-10T19:15:44.135226+0000","last_clean":"2024-09-10T19:15:44.135226+0000","last_became_active":"2024-09-10T19:13:54.729917+0000","last_became_peered":"2024-09-10T19:13:54.729917+0000","last_unstale":"2024-09-10T19:15:44.135226+0000","last_undegraded":"2024-09-10T19:15:44.135226+0000","last_fullsized":"2024-09-10T19:15:44.135226+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:15:30.583878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:11:23.264793+0000","last_clean_scrub_stamp":"2024-09-10T19:15:30.583878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:16:56.396822+0000","scrub_duration":0.0015146960000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":108,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-09-10T19:15:44.137194+0000","last_change":"2024-09-10T19:12:51.508398+0000","last_active":"2024-09-10T19:15:44.137194+0000","last_peered":"2024-09-10T19:15:44.137194+0000","last_clean":"2024-09-10T19:15:44.137194+0000","last_became_active":"2024-09-10T19:10:14.054360+0000","last_became_peered":"2024-09-10T19:10:14.054360+0000","last_unstale":"2024-09-10T19:15:44.137194+0000","last_undegraded":"2024-09-10T19:15:44.137194+0000","last_fullsized":"2024-09-10T19:15:44.137194+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:12:51.508263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:12:51.508263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001748274,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"80'15","reported_seq":161,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-09-10T19:15:46.020108+0000","last_change":"2024-09-10T19:15:46.020057+0000","last_active":"2024-09-10T19:15:46.020108+0000","last_peered":"2024-09-10T19:15:46.020108+0000","last_clean":"2024-09-10T19:15:46.020108+0000","last_became_active":"2024-09-10T19:13:54.719574+0000","last_became_peered":"2024-09-10T19:13:54.719574+0000","last_unstale":"2024-09-10T19:15:46.020108+0000","last_undegraded":"2024-09-10T19:15:46.020108+0000","last_fullsized":"2024-09-10T19:15:46.020108+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"80'15","last_scrub_stamp":"2024-09-10T19:15:46.019949+0000","last_deep_scrub":"80'15","last_deep_scrub_stamp":"2024-09-10T19:11:41.719310+0000","last_clean_scrub_stamp":"2024-09-10T19:15:46.019949+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:17:12.728277+0000","scrub_duration":0.0021039190000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":114,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-09-10T19:15:44.134626+0000","last_change":"2024-09-10T19:12:46.506129+0000","last_active":"2024-09-10T19:15:44.134626+0000","last_peered":"2024-09-10T19:15:44.134626+0000","last_clean":"2024-09-10T19:15:44.134626+0000","last_became_active":"2024-09-10T19:10:07.685870+0000","last_became_peered":"2024-09-10T19:10:07.685870+0000","last_unstale":"2024-09-10T19:15:44.134626+0000","last_undegraded":"2024-09-10T19:15:44.134626+0000","last_fullsized":"2024-09-10T19:15:44.134626+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:12:46.506071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:12:46.506071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001257935,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"117'10","reported_seq":198,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-09-10T19:15:45.138952+0000","last_change":"2024-09-10T19:13:54.755445+0000","last_active":"2024-09-10T19:15:45.138952+0000","last_peered":"2024-09-10T19:15:45.138952+0000","last_clean":"2024-09-10T19:15:45.138952+0000","last_became_active":"2024-09-10T19:13:54.724780+0000","last_became_peered":"2024-09-10T19:13:54.724780+0000","last_unstale":"2024-09-10T19:15:45.138952+0000","last_undegraded":"2024-09-10T19:15:45.138952+0000","last_fullsized":"2024-09-10T19:15:45.138952+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'8","last_scrub_stamp":"2024-09-10T19:11:50.330303+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:11:50.330303+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0020032069999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3],"acting":[5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":108,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-09-10T19:15:44.138881+0000","last_change":"2024-09-10T19:13:00.212150+0000","last_active":"2024-09-10T19:15:44.138881+0000","last_peered":"2024-09-10T19:15:44.138881+0000","last_clean":"2024-09-10T19:15:44.138881+0000","last_became_active":"2024-09-10T19:10:13.721035+0000","last_became_peered":"2024-09-10T19:10:13.721035+0000","last_unstale":"2024-09-10T19:15:44.138881+0000","last_undegraded":"2024-09-10T19:15:44.138881+0000","last_fullsized":"2024-09-10T19:15:44.138881+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:13:00.212049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:13:00.212049+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.015146817999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"66'9","reported_seq":148,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-09-10T19:15:45.182861+0000","last_change":"2024-09-10T19:15:45.182861+0000","last_active":"2024-09-10T19:15:45.182861+0000","last_peered":"2024-09-10T19:15:45.182861+0000","last_clean":"2024-09-10T19:15:45.182861+0000","last_became_active":"2024-09-10T19:15:45.149000+0000","last_became_peered":"2024-09-10T19:15:45.149000+0000","last_unstale":"2024-09-10T19:15:45.182861+0000","last_undegraded":"2024-09-10T19:15:45.182861+0000","last_fullsized":"2024-09-10T19:15:45.182861+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'9","last_scrub_stamp":"2024-09-10T19:12:52.416748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:12:52.416748+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0047220839999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":114,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-09-10T19:15:44.138726+0000","last_change":"2024-09-10T19:13:01.210932+0000","last_active":"2024-09-10T19:15:44.138726+0000","last_peered":"2024-09-10T19:15:44.138726+0000","last_clean":"2024-09-10T19:15:44.138726+0000","last_became_active":"2024-09-10T19:10:07.688499+0000","last_became_peered":"2024-09-10T19:10:07.688499+0000","last_unstale":"2024-09-10T19:15:44.138726+0000","last_undegraded":"2024-09-10T19:15:44.138726+0000","last_fullsized":"2024-09-10T19:15:44.138726+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:13:01.210848+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:13:01.210848+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0049627250000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"66'8","reported_seq":186,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-09-10T19:15:45.136701+0000","last_change":"2024-09-10T19:12:59.723961+0000","last_active":"2024-09-10T19:15:45.136701+0000","last_peered":"2024-09-10T19:15:45.136701+0000","last_clean":"2024-09-10T19:15:45.136701+0000","last_became_active":"2024-09-10T19:10:11.692359+0000","last_became_peered":"2024-09-10T19:10:11.692359+0000","last_unstale":"2024-09-10T19:15:45.136701+0000","last_undegraded":"2024-09-10T19:15:45.136701+0000","last_fullsized":"2024-09-10T19:15:45.136701+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'8","last_scrub_stamp":"2024-09-10T19:12:59.723712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:12:59.723712+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0083838109999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":107,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-09-10T19:15:44.142963+0000","last_change":"2024-09-10T19:13:09.463096+0000","last_active":"2024-09-10T19:15:44.142963+0000","last_peered":"2024-09-10T19:15:44.142963+0000","last_clean":"2024-09-10T19:15:44.142963+0000","last_became_active":"2024-09-10T19:10:13.718285+0000","last_became_peered":"2024-09-10T19:10:13.718285+0000","last_unstale":"2024-09-10T19:15:44.142963+0000","last_undegraded":"2024-09-10T19:15:44.142963+0000","last_fullsized":"2024-09-10T19:15:44.142963+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:13:09.463001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:13:09.463001+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.012199573,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.12","version":"79'13","reported_seq":131,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-09-10T19:15:44.130346+0000","last_change":"2024-09-10T19:15:29.854207+0000","last_active":"2024-09-10T19:15:44.130346+0000","last_peered":"2024-09-10T19:15:44.130346+0000","last_clean":"2024-09-10T19:15:44.130346+0000","last_became_active":"2024-09-10T19:13:54.721560+0000","last_became_peered":"2024-09-10T19:13:54.721560+0000","last_unstale":"2024-09-10T19:15:44.130346+0000","last_undegraded":"2024-09-10T19:15:44.130346+0000","last_fullsized":"2024-09-10T19:15:44.130346+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'13","last_scrub_stamp":"2024-09-10T19:15:29.854117+0000","last_deep_scrub":"79'13","last_deep_scrub_stamp":"2024-09-10T19:11:22.072400+0000","last_clean_scrub_stamp":"2024-09-10T19:15:29.854117+0000","objects_scrubbed":4,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:16:56.161162+0000","scrub_duration":0.0066627300000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,0],"acting":[6,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.14","version":"59'1","reported_seq":121,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-09-10T19:15:44.131493+0000","last_change":"2024-09-10T19:13:08.917822+0000","last_active":"2024-09-10T19:15:44.131493+0000","last_peered":"2024-09-10T19:15:44.131493+0000","last_clean":"2024-09-10T19:15:44.131493+0000","last_became_active":"2024-09-10T19:10:07.683263+0000","last_became_peered":"2024-09-10T19:10:07.683263+0000","last_unstale":"2024-09-10T19:15:44.131493+0000","last_undegraded":"2024-09-10T19:15:44.131493+0000","last_fullsized":"2024-09-10T19:15:44.131493+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'1","last_scrub_stamp":"2024-09-10T19:13:08.917719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:13:08.917719+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0033669569999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":362,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":117,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-09-10T19:15:44.143767+0000","last_change":"2024-09-10T19:13:54.730422+0000","last_active":"2024-09-10T19:15:44.143767+0000","last_peered":"2024-09-10T19:15:44.143767+0000","last_clean":"2024-09-10T19:15:44.143767+0000","last_became_active":"2024-09-10T19:13:54.730185+0000","last_became_peered":"2024-09-10T19:13:54.730185+0000","last_unstale":"2024-09-10T19:15:44.143767+0000","last_undegraded":"2024-09-10T19:15:44.143767+0000","last_fullsized":"2024-09-10T19:15:44.143767+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:12:00.660284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:12:00.660284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0012510729999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":115,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-09-10T19:15:44.139018+0000","last_change":"2024-09-10T19:13:54.715762+0000","last_active":"2024-09-10T19:15:44.139018+0000","last_peered":"2024-09-10T19:15:44.139018+0000","last_clean":"2024-09-10T19:15:44.139018+0000","last_became_active":"2024-09-10T19:13:54.714861+0000","last_became_peered":"2024-09-10T19:13:54.714861+0000","last_unstale":"2024-09-10T19:15:44.139018+0000","last_undegraded":"2024-09-10T19:15:44.139018+0000","last_fullsized":"2024-09-10T19:15:44.139018+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:11:51.920006+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:11:51.920006+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001865058,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,6],"acting":[0,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"66'11","reported_seq":152,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-09-10T19:15:44.142371+0000","last_change":"2024-09-10T19:14:40.034928+0000","last_active":"2024-09-10T19:15:44.142371+0000","last_peered":"2024-09-10T19:15:44.142371+0000","last_clean":"2024-09-10T19:15:44.142371+0000","last_became_active":"2024-09-10T19:13:54.720034+0000","last_became_peered":"2024-09-10T19:13:54.720034+0000","last_unstale":"2024-09-10T19:15:44.142371+0000","last_undegraded":"2024-09-10T19:15:44.142371+0000","last_fullsized":"2024-09-10T19:15:44.142371+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'11","last_scrub_stamp":"2024-09-10T19:11:44.025361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:11:44.025361+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.002036894,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":64,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-09-10T19:15:44.135327+0000","last_change":"2024-09-10T19:13:54.723475+0000","last_active":"2024-09-10T19:15:44.135327+0000","last_peered":"2024-09-10T19:15:44.135327+0000","last_clean":"2024-09-10T19:15:44.135327+0000","last_became_active":"2024-09-10T19:13:54.722947+0000","last_became_peered":"2024-09-10T19:13:54.722947+0000","last_unstale":"2024-09-10T19:15:44.135327+0000","last_undegraded":"2024-09-10T19:15:44.135327+0000","last_fullsized":"2024-09-10T19:15:44.135327+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:12:44.249765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:11:10.031245+0000","last_clean_scrub_stamp":"2024-09-10T19:12:44.249765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,6],"acting":[5,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":74,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-09-10T19:15:44.134348+0000","last_change":"2024-09-10T19:15:36.666310+0000","last_active":"2024-09-10T19:15:44.134348+0000","last_peered":"2024-09-10T19:15:44.134348+0000","last_clean":"2024-09-10T19:15:44.134348+0000","last_became_active":"2024-09-10T19:13:54.729703+0000","last_became_peered":"2024-09-10T19:13:54.729703+0000","last_unstale":"2024-09-10T19:15:44.134348+0000","last_undegraded":"2024-09-10T19:15:44.134348+0000","last_fullsized":"2024-09-10T19:15:44.134348+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:15:36.666216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:15:36.666216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:17:03.635276+0000","scrub_duration":0.003489884,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":108,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-09-10T19:15:44.143502+0000","last_change":"2024-09-10T19:12:54.691886+0000","last_active":"2024-09-10T19:15:44.143502+0000","last_peered":"2024-09-10T19:15:44.143502+0000","last_clean":"2024-09-10T19:15:44.143502+0000","last_became_active":"2024-09-10T19:10:13.724201+0000","last_became_peered":"2024-09-10T19:10:13.724201+0000","last_unstale":"2024-09-10T19:15:44.143502+0000","last_undegraded":"2024-09-10T19:15:44.143502+0000","last_fullsized":"2024-09-10T19:15:44.143502+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:12:54.691810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:12:54.691810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.006702927,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"66'4","reported_seq":141,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-09-10T19:15:44.142381+0000","last_change":"2024-09-10T19:15:33.995417+0000","last_active":"2024-09-10T19:15:44.142381+0000","last_peered":"2024-09-10T19:15:44.142381+0000","last_clean":"2024-09-10T19:15:44.142381+0000","last_became_active":"2024-09-10T19:13:54.711125+0000","last_became_peered":"2024-09-10T19:13:54.711125+0000","last_unstale":"2024-09-10T19:15:44.142381+0000","last_undegraded":"2024-09-10T19:15:44.142381+0000","last_fullsized":"2024-09-10T19:15:44.142381+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'4","last_scrub_stamp":"2024-09-10T19:15:33.995274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:15:33.995274+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:16:44.338208+0000","scrub_duration":0.011108425,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,4],"acting":[3,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":114,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-09-10T19:15:44.135326+0000","last_change":"2024-09-10T19:12:48.517928+0000","last_active":"2024-09-10T19:15:44.135326+0000","last_peered":"2024-09-10T19:15:44.135326+0000","last_clean":"2024-09-10T19:15:44.135326+0000","last_became_active":"2024-09-10T19:10:07.687723+0000","last_became_peered":"2024-09-10T19:10:07.687723+0000","last_unstale":"2024-09-10T19:15:44.135326+0000","last_undegraded":"2024-09-10T19:15:44.135326+0000","last_fullsized":"2024-09-10T19:15:44.135326+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:12:48.517853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:12:48.517853+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001753459,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":110,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-09-10T19:15:44.130520+0000","last_change":"2024-09-10T19:13:04.855026+0000","last_active":"2024-09-10T19:15:44.130520+0000","last_peered":"2024-09-10T19:15:44.130520+0000","last_clean":"2024-09-10T19:15:44.130520+0000","last_became_active":"2024-09-10T19:10:11.698536+0000","last_became_peered":"2024-09-10T19:10:11.698536+0000","last_unstale":"2024-09-10T19:15:44.130520+0000","last_undegraded":"2024-09-10T19:15:44.130520+0000","last_fullsized":"2024-09-10T19:15:44.130520+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:13:04.854929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:11:31.837944+0000","last_clean_scrub_stamp":"2024-09-10T19:13:04.854929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0012499329999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":118,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-09-10T19:15:44.523653+0000","last_change":"2024-09-10T19:15:44.523584+0000","last_active":"2024-09-10T19:15:44.523653+0000","last_peered":"2024-09-10T19:15:44.523653+0000","last_clean":"2024-09-10T19:15:44.523653+0000","last_became_active":"2024-09-10T19:10:13.718271+0000","last_became_peered":"2024-09-10T19:10:13.718271+0000","last_unstale":"2024-09-10T19:15:44.523653+0000","last_undegraded":"2024-09-10T19:15:44.523653+0000","last_fullsized":"2024-09-10T19:15:44.523653+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:15:44.523530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:15:44.523530+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:17:02.565497+0000","scrub_duration":0.027001213999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.11","version":"66'11","reported_seq":22,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-09-10T19:15:45.158623+0000","last_change":"2024-09-10T19:15:45.158623+0000","last_active":"2024-09-10T19:15:45.158623+0000","last_peered":"2024-09-10T19:15:45.158623+0000","last_clean":"2024-09-10T19:15:45.158623+0000","last_became_active":"2024-09-10T19:15:45.149864+0000","last_became_peered":"2024-09-10T19:15:45.149864+0000","last_unstale":"2024-09-10T19:15:45.158623+0000","last_undegraded":"2024-09-10T19:15:45.158623+0000","last_fullsized":"2024-09-10T19:15:45.158623+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'11","last_scrub_stamp":"2024-09-10T19:13:05.729854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:13:05.729854+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.010667395,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":114,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-09-10T19:15:44.134586+0000","last_change":"2024-09-10T19:13:05.876708+0000","last_active":"2024-09-10T19:15:44.134586+0000","last_peered":"2024-09-10T19:15:44.134586+0000","last_clean":"2024-09-10T19:15:44.134586+0000","last_became_active":"2024-09-10T19:10:07.685223+0000","last_became_peered":"2024-09-10T19:10:07.685223+0000","last_unstale":"2024-09-10T19:15:44.134586+0000","last_undegraded":"2024-09-10T19:15:44.134586+0000","last_fullsized":"2024-09-10T19:15:44.134586+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:13:05.876655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:13:05.876655+0000","last_clean_scrub_stamp":"2024-09-10T19:13:05.876655+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0013242589999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":110,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-09-10T19:15:44.136168+0000","last_change":"2024-09-10T19:13:41.326028+0000","last_active":"2024-09-10T19:15:44.136168+0000","last_peered":"2024-09-10T19:15:44.136168+0000","last_clean":"2024-09-10T19:15:44.136168+0000","last_became_active":"2024-09-10T19:10:11.699013+0000","last_became_peered":"2024-09-10T19:10:11.699013+0000","last_unstale":"2024-09-10T19:15:44.136168+0000","last_undegraded":"2024-09-10T19:15:44.136168+0000","last_fullsized":"2024-09-10T19:15:44.136168+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:13:41.325941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:11:57.112092+0000","last_clean_scrub_stamp":"2024-09-10T19:13:41.325941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001598676,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":108,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-09-10T19:15:44.143646+0000","last_change":"2024-09-10T19:13:15.812326+0000","last_active":"2024-09-10T19:15:44.143646+0000","last_peered":"2024-09-10T19:15:44.143646+0000","last_clean":"2024-09-10T19:15:44.143646+0000","last_became_active":"2024-09-10T19:10:14.056172+0000","last_became_peered":"2024-09-10T19:10:14.056172+0000","last_unstale":"2024-09-10T19:15:44.143646+0000","last_undegraded":"2024-09-10T19:15:44.143646+0000","last_fullsized":"2024-09-10T19:15:44.143646+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:13:15.812244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:13:15.812244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0048044389999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"80'2","reported_seq":110,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-09-10T19:15:44.136634+0000","last_change":"2024-09-10T19:12:57.454088+0000","last_active":"2024-09-10T19:15:44.136634+0000","last_peered":"2024-09-10T19:15:44.136634+0000","last_clean":"2024-09-10T19:15:44.136634+0000","last_became_active":"2024-09-10T19:10:13.718875+0000","last_became_peered":"2024-09-10T19:10:13.718875+0000","last_unstale":"2024-09-10T19:15:44.136634+0000","last_undegraded":"2024-09-10T19:15:44.136634+0000","last_fullsized":"2024-09-10T19:15:44.136634+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"80'2","last_scrub_stamp":"2024-09-10T19:12:57.453993+0000","last_deep_scrub":"80'2","last_deep_scrub_stamp":"2024-09-10T19:11:30.272043+0000","last_clean_scrub_stamp":"2024-09-10T19:12:57.453993+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0018294990000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":369,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":114,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-09-10T19:15:44.134259+0000","last_change":"2024-09-10T19:13:14.449643+0000","last_active":"2024-09-10T19:15:44.134259+0000","last_peered":"2024-09-10T19:15:44.134259+0000","last_clean":"2024-09-10T19:15:44.134259+0000","last_became_active":"2024-09-10T19:10:07.689877+0000","last_became_peered":"2024-09-10T19:10:07.689877+0000","last_unstale":"2024-09-10T19:15:44.134259+0000","last_undegraded":"2024-09-10T19:15:44.134259+0000","last_fullsized":"2024-09-10T19:15:44.134259+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:13:14.449593+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:13:14.449593+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001195542,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1e","version":"80'15","reported_seq":132,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-09-10T19:15:44.139457+0000","last_change":"2024-09-10T19:12:50.148157+0000","last_active":"2024-09-10T19:15:44.139457+0000","last_peered":"2024-09-10T19:15:44.139457+0000","last_clean":"2024-09-10T19:15:44.139457+0000","last_became_active":"2024-09-10T19:10:09.823061+0000","last_became_peered":"2024-09-10T19:10:09.823061+0000","last_unstale":"2024-09-10T19:15:44.139457+0000","last_undegraded":"2024-09-10T19:15:44.139457+0000","last_fullsized":"2024-09-10T19:15:44.139457+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"80'15","last_scrub_stamp":"2024-09-10T19:12:50.147336+0000","last_deep_scrub":"80'15","last_deep_scrub_stamp":"2024-09-10T19:11:25.883919+0000","last_clean_scrub_stamp":"2024-09-10T19:12:50.147336+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0064585019999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1f","version":"66'8","reported_seq":189,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-09-10T19:15:45.139002+0000","last_change":"2024-09-10T19:12:48.889499+0000","last_active":"2024-09-10T19:15:45.139002+0000","last_peered":"2024-09-10T19:15:45.139002+0000","last_clean":"2024-09-10T19:15:45.139002+0000","last_became_active":"2024-09-10T19:10:11.695427+0000","last_became_peered":"2024-09-10T19:10:11.695427+0000","last_unstale":"2024-09-10T19:15:45.139002+0000","last_undegraded":"2024-09-10T19:15:45.139002+0000","last_fullsized":"2024-09-10T19:15:45.139002+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'8","last_scrub_stamp":"2024-09-10T19:12:48.889410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:12:48.889410+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001725311,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":74,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-09-10T19:15:44.134070+0000","last_change":"2024-09-10T19:15:29.580679+0000","last_active":"2024-09-10T19:15:44.134070+0000","last_peered":"2024-09-10T19:15:44.134070+0000","last_clean":"2024-09-10T19:15:44.134070+0000","last_became_active":"2024-09-10T19:13:54.715192+0000","last_became_peered":"2024-09-10T19:13:54.715192+0000","last_unstale":"2024-09-10T19:15:44.134070+0000","last_undegraded":"2024-09-10T19:15:44.134070+0000","last_fullsized":"2024-09-10T19:15:44.134070+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:15:29.580605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:15:29.580605+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:16:52.844009+0000","scrub_duration":0.0047886710000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,2],"acting":[5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":114,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-09-10T19:15:44.143974+0000","last_change":"2024-09-10T19:13:02.737323+0000","last_active":"2024-09-10T19:15:44.143974+0000","last_peered":"2024-09-10T19:15:44.143974+0000","last_clean":"2024-09-10T19:15:44.143974+0000","last_became_active":"2024-09-10T19:10:07.682689+0000","last_became_peered":"2024-09-10T19:10:07.682689+0000","last_unstale":"2024-09-10T19:15:44.143974+0000","last_undegraded":"2024-09-10T19:15:44.143974+0000","last_fullsized":"2024-09-10T19:15:44.143974+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:13:02.737236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:13:02.737236+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0027813289999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"80'16","reported_seq":190,"reported_epoch":144,"state":"active+clean","last_fresh":"2024-09-10T19:15:47.700632+0000","last_change":"2024-09-10T19:15:47.700592+0000","last_active":"2024-09-10T19:15:47.700632+0000","last_peered":"2024-09-10T19:15:47.700632+0000","last_clean":"2024-09-10T19:15:47.700632+0000","last_became_active":"2024-09-10T19:13:54.713563+0000","last_became_peered":"2024-09-10T19:13:54.713563+0000","last_unstale":"2024-09-10T19:15:47.700632+0000","last_undegraded":"2024-09-10T19:15:47.700632+0000","last_fullsized":"2024-09-10T19:15:47.700632+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"80'16","last_scrub_stamp":"2024-09-10T19:15:47.700523+0000","last_deep_scrub":"80'16","last_deep_scrub_stamp":"2024-09-10T19:11:40.853611+0000","last_clean_scrub_stamp":"2024-09-10T19:15:47.700523+0000","objects_scrubbed":5,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:16:55.889805+0000","scrub_duration":0.0019149010000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,0],"acting":[6,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":110,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-09-10T19:15:44.139381+0000","last_change":"2024-09-10T19:12:59.195000+0000","last_active":"2024-09-10T19:15:44.139381+0000","last_peered":"2024-09-10T19:15:44.139381+0000","last_clean":"2024-09-10T19:15:44.139381+0000","last_became_active":"2024-09-10T19:10:11.701735+0000","last_became_peered":"2024-09-10T19:10:11.701735+0000","last_unstale":"2024-09-10T19:15:44.139381+0000","last_undegraded":"2024-09-10T19:15:44.139381+0000","last_fullsized":"2024-09-10T19:15:44.139381+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:12:59.194914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:11:28.822327+0000","last_clean_scrub_stamp":"2024-09-10T19:12:59.194914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0022130729999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":107,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-09-10T19:15:44.142589+0000","last_change":"2024-09-10T19:13:00.388592+0000","last_active":"2024-09-10T19:15:44.142589+0000","last_peered":"2024-09-10T19:15:44.142589+0000","last_clean":"2024-09-10T19:15:44.142589+0000","last_became_active":"2024-09-10T19:10:14.053619+0000","last_became_peered":"2024-09-10T19:10:14.053619+0000","last_unstale":"2024-09-10T19:15:44.142589+0000","last_undegraded":"2024-09-10T19:15:44.142589+0000","last_fullsized":"2024-09-10T19:15:44.142589+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:13:00.388466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:13:00.388466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.003120996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":114,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-09-10T19:15:44.130485+0000","last_change":"2024-09-10T19:13:43.205095+0000","last_active":"2024-09-10T19:15:44.130485+0000","last_peered":"2024-09-10T19:15:44.130485+0000","last_clean":"2024-09-10T19:15:44.130485+0000","last_became_active":"2024-09-10T19:10:07.685997+0000","last_became_peered":"2024-09-10T19:10:07.685997+0000","last_unstale":"2024-09-10T19:15:44.130485+0000","last_undegraded":"2024-09-10T19:15:44.130485+0000","last_fullsized":"2024-09-10T19:15:44.130485+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:13:43.205042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:13:43.205042+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0013181790000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1c","version":"78'22","reported_seq":187,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-09-10T19:15:44.134130+0000","last_change":"2024-09-10T19:15:37.702706+0000","last_active":"2024-09-10T19:15:44.134130+0000","last_peered":"2024-09-10T19:15:44.134130+0000","last_clean":"2024-09-10T19:15:44.134130+0000","last_became_active":"2024-09-10T19:13:54.715570+0000","last_became_peered":"2024-09-10T19:13:54.715570+0000","last_unstale":"2024-09-10T19:15:44.134130+0000","last_undegraded":"2024-09-10T19:15:44.134130+0000","last_fullsized":"2024-09-10T19:15:44.134130+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"78'22","last_scrub_stamp":"2024-09-10T19:15:37.702624+0000","last_deep_scrub":"78'22","last_deep_scrub_stamp":"2024-09-10T19:11:21.118937+0000","last_clean_scrub_stamp":"2024-09-10T19:15:37.702624+0000","objects_scrubbed":7,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:17:04.140997+0000","scrub_duration":0.033348765000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":77,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-09-10T19:15:44.142521+0000","last_change":"2024-09-10T19:15:33.008615+0000","last_active":"2024-09-10T19:15:44.142521+0000","last_peered":"2024-09-10T19:15:44.142521+0000","last_clean":"2024-09-10T19:15:44.142521+0000","last_became_active":"2024-09-10T19:13:54.709055+0000","last_became_peered":"2024-09-10T19:13:54.709055+0000","last_unstale":"2024-09-10T19:15:44.142521+0000","last_undegraded":"2024-09-10T19:15:44.142521+0000","last_fullsized":"2024-09-10T19:15:44.142521+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:15:33.008542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:15:33.008542+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:16:51.091294+0000","scrub_duration":0.004557363,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,5],"acting":[4,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":108,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-09-10T19:15:44.143602+0000","last_change":"2024-09-10T19:12:51.767586+0000","last_active":"2024-09-10T19:15:44.143602+0000","last_peered":"2024-09-10T19:15:44.143602+0000","last_clean":"2024-09-10T19:15:44.143602+0000","last_became_active":"2024-09-10T19:10:14.053698+0000","last_became_peered":"2024-09-10T19:10:14.053698+0000","last_unstale":"2024-09-10T19:15:44.143602+0000","last_undegraded":"2024-09-10T19:15:44.143602+0000","last_fullsized":"2024-09-10T19:15:44.143602+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:12:51.767056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:12:51.767056+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.006796519,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":114,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-09-10T19:15:44.144078+0000","last_change":"2024-09-10T19:12:50.719167+0000","last_active":"2024-09-10T19:15:44.144078+0000","last_peered":"2024-09-10T19:15:44.144078+0000","last_clean":"2024-09-10T19:15:44.144078+0000","last_became_active":"2024-09-10T19:10:07.682811+0000","last_became_peered":"2024-09-10T19:10:07.682811+0000","last_unstale":"2024-09-10T19:15:44.144078+0000","last_undegraded":"2024-09-10T19:15:44.144078+0000","last_fullsized":"2024-09-10T19:15:44.144078+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:12:50.718663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:12:50.718663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0042138399999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1d","version":"66'12","reported_seq":154,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-09-10T19:15:44.144659+0000","last_change":"2024-09-10T19:13:54.758541+0000","last_active":"2024-09-10T19:15:44.144659+0000","last_peered":"2024-09-10T19:15:44.144659+0000","last_clean":"2024-09-10T19:15:44.144659+0000","last_became_active":"2024-09-10T19:13:54.714629+0000","last_became_peered":"2024-09-10T19:13:54.714629+0000","last_unstale":"2024-09-10T19:15:44.144659+0000","last_undegraded":"2024-09-10T19:15:44.144659+0000","last_fullsized":"2024-09-10T19:15:44.144659+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'12","last_scrub_stamp":"2024-09-10T19:11:52.663657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:11:52.663657+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0019136909999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":109,"reported_epoch":143,"state":"active+clean","last_fresh":"2024-09-10T19:15:44.143316+0000","last_change":"2024-09-10T19:13:11.445426+0000","last_active":"2024-09-10T19:15:44.143316+0000","last_peered":"2024-09-10T19:15:44.143316+0000","last_clean":"2024-09-10T19:15:44.143316+0000","last_became_active":"2024-09-10T19:10:11.701155+0000","last_became_peered":"2024-09-10T19:10:11.701155+0000","last_unstale":"2024-09-10T19:15:44.143316+0000","last_undegraded":"2024-09-10T19:15:44.143316+0000","last_fullsized":"2024-09-10T19:15:44.143316+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:13:11.445238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:13:11.445238+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0057421110000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":16,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":280758,"num_write_kb":8984193,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33710,"num_bytes_recovered":2208956440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1018036224,"data_stored":1018036224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76257,"ondisk_log_size":76257,"up":48,"acting":48,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":71,"ondisk_log_size":71,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":177,"num_object_clones":0,"num_object_copies":531,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":177,"num_whiteouts":0,"num_read":695,"num_read_kb":452,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":127,"num_bytes_recovered":2668,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":499712,"data_stored":13224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":444,"ondisk_log_size":444,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":185,"num_read_kb":180,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":389,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1613,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1328,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":27,"num_read_kb":27,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":3984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4,"ondisk_log_size":4,"up":96,"acting":96,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79,"ondisk_log_size":79,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":54,"seq":231928234057,"num_pgs":70,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":302924,"kb_used_data":1408,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":93442228,"statfs":{"total":95995035648,"available":95684841472,"internally_reserved":0,"allocated":1441792,"data_stored":1099465,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 19:14:58 2024","interfaces":[{"interface":"back","average":{"1min":1.0620000000000001,"5min":1.724,"15min":0.93100000000000005},"min":{"1min":0.24299999999999999,"5min":0.215,"15min":0.215},"max":{"1min":2.887,"5min":35.551000000000002,"15min":35.551000000000002},"last":0.43099999999999999},{"interface":"front","average":{"1min":1.2150000000000001,"5min":1.5449999999999999,"15min":0.873},"min":{"1min":0.20699999999999999,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":3.2829999999999999,"5min":35.503999999999998,"15min":35.503999999999998},"last":0.44700000000000001}]},{"osd":1,"last update":"Tue Sep 10 19:11:56 2024","interfaces":[{"interface":"back","average":{"1min":0.52400000000000002,"5min":0.52400000000000002,"15min":0.52400000000000002},"min":{"1min":0.36199999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.70199999999999996,"5min":0.79400000000000004,"15min":0.79400000000000004},"last":0.38700000000000001},{"interface":"front","average":{"1min":0.49199999999999999,"5min":0.51200000000000001,"15min":0.51500000000000001},"min":{"1min":0.28999999999999998,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.68500000000000005,"5min":0.71799999999999997,"15min":0.71799999999999997},"last":0.46899999999999997}]},{"osd":2,"last update":"Tue Sep 10 19:14:58 2024","interfaces":[{"interface":"back","average":{"1min":1.0700000000000001,"5min":1.0469999999999999,"15min":0.69899999999999995},"min":{"1min":0.30199999999999999,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":4.21,"5min":31.940999999999999,"15min":31.940999999999999},"last":0.40300000000000002},{"interface":"front","average":{"1min":0.86599999999999999,"5min":1.077,"15min":0.72499999999999998},"min":{"1min":0.249,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":2.4119999999999999,"5min":25.056000000000001,"15min":25.056000000000001},"last":0.55300000000000005}]},{"osd":3,"last update":"Tue Sep 10 19:14:58 2024","interfaces":[{"interface":"back","average":{"1min":1.163,"5min":1.6759999999999999,"15min":0.91600000000000004},"min":{"1min":0.26000000000000001,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":2.8519999999999999,"5min":35.277999999999999,"15min":35.277999999999999},"last":0.48699999999999999},{"interface":"front","average":{"1min":0.82499999999999996,"5min":1.6559999999999999,"15min":0.89800000000000002},"min":{"1min":0.34200000000000003,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":2.355,"5min":36.512999999999998,"15min":36.512999999999998},"last":0.50800000000000001}]},{"osd":4,"last update":"Tue Sep 10 19:14:58 2024","interfaces":[{"interface":"back","average":{"1min":0.64000000000000001,"5min":0.50900000000000001,"15min":0.47299999999999998},"min":{"1min":0.26100000000000001,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":2.5249999999999999,"5min":2.863,"15min":2.863},"last":0.60299999999999998},{"interface":"front","average":{"1min":0.72399999999999998,"5min":0.52900000000000003,"15min":0.48299999999999998},"min":{"1min":0.23000000000000001,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":1.5049999999999999,"5min":1.5049999999999999,"15min":1.5049999999999999},"last":0.53900000000000003}]},{"osd":5,"last update":"Tue Sep 10 19:14:58 2024","interfaces":[{"interface":"back","average":{"1min":1.2849999999999999,"5min":0.754,"15min":0.57099999999999995},"min":{"1min":0.27100000000000002,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":11.621,"5min":11.621,"15min":11.621},"last":0.58999999999999997},{"interface":"front","average":{"1min":0.86399999999999999,"5min":0.58799999999999997,"15min":0.51300000000000001},"min":{"1min":0.26000000000000001,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":2.875,"5min":2.8980000000000001,"15min":2.8980000000000001},"last":0.52500000000000002}]},{"osd":6,"last update":"Tue Sep 10 19:14:58 2024","interfaces":[{"interface":"back","average":{"1min":1.0029999999999999,"5min":0.62,"15min":0.54000000000000004},"min":{"1min":0.317,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":2.9089999999999998,"5min":2.9089999999999998,"15min":2.9089999999999998},"last":0.57399999999999995},{"interface":"front","average":{"1min":0.86299999999999999,"5min":0.56100000000000005,"15min":0.51300000000000001},"min":{"1min":0.29499999999999998,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":1.5760000000000001,"5min":1.5760000000000001,"15min":1.5760000000000001},"last":0.56499999999999995}]}]},{"osd":6,"up_from":47,"seq":201863462990,"num_pgs":60,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":302928,"kb_used_data":1416,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":93442224,"statfs":{"total":95995035648,"available":95684837376,"internally_reserved":0,"allocated":1449984,"data_stored":1100009,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 19:15:45 2024","interfaces":[{"interface":"back","average":{"1min":1.3700000000000001,"5min":2.1629999999999998,"15min":1.069},"min":{"1min":0.20100000000000001,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":12.334,"5min":112.988,"15min":112.988},"last":0.30599999999999999},{"interface":"front","average":{"1min":1.3839999999999999,"5min":2.1469999999999998,"15min":1.0780000000000001},"min":{"1min":0.249,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":12.319000000000001,"5min":113.18300000000001,"15min":113.18300000000001},"last":0.39400000000000002}]},{"osd":1,"last update":"Tue Sep 10 19:12:41 2024","interfaces":[{"interface":"back","average":{"1min":0.59999999999999998,"5min":0.53700000000000003,"15min":0.53100000000000003},"min":{"1min":0.33800000000000002,"5min":0.318,"15min":0.318},"max":{"1min":1.29,"5min":1.29,"15min":1.29},"last":0.377},{"interface":"front","average":{"1min":0.69399999999999995,"5min":0.55400000000000005,"15min":0.53800000000000003},"min":{"1min":0.32400000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":2.6859999999999999,"5min":2.6859999999999999,"15min":2.6859999999999999},"last":0.36399999999999999}]},{"osd":2,"last update":"Tue Sep 10 19:15:45 2024","interfaces":[{"interface":"back","average":{"1min":0.88400000000000001,"5min":1.956,"15min":1.0149999999999999},"min":{"1min":0.25,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":2.7709999999999999,"5min":112.864,"15min":112.864},"last":0.40600000000000003},{"interface":"front","average":{"1min":1.026,"5min":2.069,"15min":1.0289999999999999},"min":{"1min":0.28399999999999997,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":3.456,"5min":112.822,"15min":112.822},"last":0.46500000000000002}]},{"osd":3,"last update":"Tue Sep 10 19:15:45 2024","interfaces":[{"interface":"back","average":{"1min":1.9870000000000001,"5min":2.403,"15min":1.145},"min":{"1min":0.32100000000000001,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":12.539,"5min":125.56399999999999,"15min":125.56399999999999},"last":0.32100000000000001},{"interface":"front","average":{"1min":1.948,"5min":2.4900000000000002,"15min":1.2070000000000001},"min":{"1min":0.438,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":12.513,"5min":125.47,"15min":125.47},"last":0.438}]},{"osd":4,"last update":"Tue Sep 10 19:15:45 2024","interfaces":[{"interface":"back","average":{"1min":0.54300000000000004,"5min":0.49099999999999999,"15min":0.39500000000000002},"min":{"1min":0.27600000000000002,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":1.3220000000000001,"5min":3.3410000000000002,"15min":3.3410000000000002},"last":0.33800000000000002},{"interface":"front","average":{"1min":0.59899999999999998,"5min":0.53300000000000003,"15min":0.42699999999999999},"min":{"1min":0.22500000000000001,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":1.831,"5min":2.0710000000000002,"15min":2.0710000000000002},"last":0.45100000000000001}]},{"osd":5,"last update":"Tue Sep 10 19:15:45 2024","interfaces":[{"interface":"back","average":{"1min":0.60499999999999998,"5min":0.98699999999999999,"15min":0.57899999999999996},"min":{"1min":0.32600000000000001,"5min":0.248,"15min":0.248},"max":{"1min":1.3540000000000001,"5min":44.765000000000001,"15min":44.765000000000001},"last":0.42699999999999999},{"interface":"front","average":{"1min":0.92300000000000004,"5min":1.1160000000000001,"15min":0.621},"min":{"1min":0.28899999999999998,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":4.0430000000000001,"5min":43.036000000000001,"15min":43.036000000000001},"last":0.47499999999999998}]},{"osd":7,"last update":"Tue Sep 10 19:15:03 2024","interfaces":[{"interface":"back","average":{"1min":0.69899999999999995,"5min":0.52200000000000002,"15min":0.47199999999999998},"min":{"1min":0.30599999999999999,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":1.7190000000000001,"5min":1.7190000000000001,"15min":1.7190000000000001},"last":0.48599999999999999},{"interface":"front","average":{"1min":0.77500000000000002,"5min":0.59799999999999998,"15min":0.496},"min":{"1min":0.31900000000000001,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":1.9139999999999999,"5min":2.3260000000000001,"15min":2.3260000000000001},"last":0.41699999999999998}]}]},{"osd":5,"up_from":39,"seq":167503724627,"num_pgs":61,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298172,"kb_used_data":816,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93446980,"statfs":{"total":95995035648,"available":95689707520,"internally_reserved":0,"allocated":835584,"data_stored":509873,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 19:15:15 2024","interfaces":[{"interface":"back","average":{"1min":1.2949999999999999,"5min":0.79000000000000004,"15min":0.57999999999999996},"min":{"1min":0.20899999999999999,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":3.403,"5min":5.851,"15min":5.851},"last":0.47599999999999998},{"interface":"front","average":{"1min":0.72599999999999998,"5min":0.67300000000000004,"15min":0.53800000000000003},"min":{"1min":0.251,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":2.883,"5min":5.6500000000000004,"15min":5.6500000000000004},"last":0.54900000000000004}]},{"osd":1,"last update":"Tue Sep 10 19:12:09 2024","interfaces":[{"interface":"back","average":{"1min":0.48199999999999998,"5min":0.51500000000000001,"15min":0.50700000000000001},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.64600000000000002,"5min":0.72799999999999998,"15min":0.72799999999999998},"last":0.45700000000000002},{"interface":"front","average":{"1min":0.47299999999999998,"5min":0.47899999999999998,"15min":0.47399999999999998},"min":{"1min":0.34399999999999997,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.61599999999999999,"5min":0.68799999999999994,"15min":0.68799999999999994},"last":0.499}]},{"osd":2,"last update":"Tue Sep 10 19:15:15 2024","interfaces":[{"interface":"back","average":{"1min":0.83699999999999997,"5min":0.67300000000000004,"15min":0.54000000000000004},"min":{"1min":0.29499999999999998,"5min":0.253,"15min":0.253},"max":{"1min":2.9100000000000001,"5min":2.9100000000000001,"15min":2.9100000000000001},"last":0.443},{"interface":"front","average":{"1min":0.75800000000000001,"5min":0.79000000000000004,"15min":0.58799999999999997},"min":{"1min":0.24399999999999999,"5min":0.224,"15min":0.224},"max":{"1min":2.4969999999999999,"5min":6.3789999999999996,"15min":6.3789999999999996},"last":0.51900000000000002}]},{"osd":3,"last update":"Tue Sep 10 19:15:15 2024","interfaces":[{"interface":"back","average":{"1min":0.85299999999999998,"5min":0.67300000000000004,"15min":0.56499999999999995},"min":{"1min":0.224,"5min":0.224,"15min":0.224},"max":{"1min":3.2709999999999999,"5min":3.351,"15min":3.351},"last":0.56399999999999995},{"interface":"front","average":{"1min":0.97599999999999998,"5min":0.70499999999999996,"15min":0.58299999999999996},"min":{"1min":0.27800000000000002,"5min":0.248,"15min":0.248},"max":{"1min":5.3869999999999996,"5min":5.3869999999999996,"15min":5.3869999999999996},"last":0.58099999999999996}]},{"osd":4,"last update":"Tue Sep 10 19:15:15 2024","interfaces":[{"interface":"back","average":{"1min":0.876,"5min":0.58099999999999996,"15min":0.42899999999999999},"min":{"1min":0.26600000000000001,"5min":0.23699999999999999,"15min":0.20699999999999999},"max":{"1min":4.3280000000000003,"5min":4.3280000000000003,"15min":4.3280000000000003},"last":0.53800000000000003},{"interface":"front","average":{"1min":0.72699999999999998,"5min":0.64300000000000002,"15min":0.45500000000000002},"min":{"1min":0.26000000000000001,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":3.0409999999999999,"5min":5.8899999999999997,"15min":5.8899999999999997},"last":0.59699999999999998}]},{"osd":6,"last update":"Tue Sep 10 19:15:44 2024","interfaces":[{"interface":"back","average":{"1min":0.81599999999999995,"5min":0.77200000000000002,"15min":0.53100000000000003},"min":{"1min":0.29099999999999998,"5min":0.27100000000000002,"15min":0.223},"max":{"1min":2.9399999999999999,"5min":6.617,"15min":6.617},"last":0.628},{"interface":"front","average":{"1min":0.80600000000000005,"5min":0.73199999999999998,"15min":0.51400000000000001},"min":{"1min":0.33800000000000002,"5min":0.26800000000000002,"15min":0.25},"max":{"1min":2.794,"5min":5.9349999999999996,"15min":5.9349999999999996},"last":0.61099999999999999}]},{"osd":7,"last update":"Tue Sep 10 19:15:00 2024","interfaces":[{"interface":"back","average":{"1min":1.1200000000000001,"5min":0.66500000000000004,"15min":0.56100000000000005},"min":{"1min":0.36699999999999999,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":5.9660000000000002,"5min":5.9660000000000002,"15min":5.9660000000000002},"last":0.63600000000000001},{"interface":"front","average":{"1min":0.84899999999999998,"5min":0.58299999999999996,"15min":0.51300000000000001},"min":{"1min":0.29499999999999998,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":2.415,"5min":2.415,"15min":2.415},"last":0.64800000000000002}]}]},{"osd":4,"up_from":32,"seq":137438953559,"num_pgs":60,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":310648,"kb_used_data":816,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":93434504,"statfs":{"total":95995035648,"available":95676932096,"internally_reserved":0,"allocated":835584,"data_stored":508508,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 19:14:59 2024","interfaces":[{"interface":"back","average":{"1min":0.83599999999999997,"5min":0.60499999999999998,"15min":0.56299999999999994},"min":{"1min":0.20899999999999999,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":3.0409999999999999,"5min":3.0409999999999999,"15min":3.0409999999999999},"last":0.45900000000000002},{"interface":"front","average":{"1min":0.85099999999999998,"5min":0.60999999999999999,"15min":0.55700000000000005},"min":{"1min":0.249,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":3.0579999999999998,"5min":3.0579999999999998,"15min":3.0579999999999998},"last":0.57999999999999996}]},{"osd":1,"last update":"Tue Sep 10 19:11:49 2024","interfaces":[{"interface":"back","average":{"1min":0.47999999999999998,"5min":0.503,"15min":0.50900000000000001},"min":{"1min":0.29999999999999999,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.72199999999999998,"5min":0.83499999999999996,"15min":0.83499999999999996},"last":0.55600000000000005},{"interface":"front","average":{"1min":0.51000000000000001,"5min":0.51700000000000002,"15min":0.52200000000000002},"min":{"1min":0.27500000000000002,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.68999999999999995,"5min":0.77100000000000002,"15min":0.77100000000000002},"last":0.54400000000000004}]},{"osd":2,"last update":"Tue Sep 10 19:14:59 2024","interfaces":[{"interface":"back","average":{"1min":0.74399999999999999,"5min":0.61099999999999999,"15min":0.56299999999999994},"min":{"1min":0.214,"5min":0.214,"15min":0.214},"max":{"1min":2.9609999999999999,"5min":2.9609999999999999,"15min":2.9609999999999999},"last":0.56599999999999995},{"interface":"front","average":{"1min":0.85999999999999999,"5min":0.629,"15min":0.56799999999999995},"min":{"1min":0.23400000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":3.73,"5min":3.73,"15min":3.73},"last":0.48699999999999999}]},{"osd":3,"last update":"Tue Sep 10 19:14:59 2024","interfaces":[{"interface":"back","average":{"1min":0.92000000000000004,"5min":0.67200000000000004,"15min":0.57499999999999996},"min":{"1min":0.28999999999999998,"5min":0.252,"15min":0.252},"max":{"1min":3.3839999999999999,"5min":3.3839999999999999,"15min":3.3839999999999999},"last":0.53100000000000003},{"interface":"front","average":{"1min":1.2310000000000001,"5min":0.76500000000000001,"15min":0.623},"min":{"1min":0.26300000000000001,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":4.6980000000000004,"5min":4.6980000000000004,"15min":4.6980000000000004},"last":0.39900000000000002}]},{"osd":5,"last update":"Tue Sep 10 19:15:22 2024","interfaces":[{"interface":"back","average":{"1min":0.66700000000000004,"5min":0.50600000000000001,"15min":0.41799999999999998},"min":{"1min":0.25700000000000001,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":2.9470000000000001,"5min":3.629,"15min":3.629},"last":0.46899999999999997},{"interface":"front","average":{"1min":0.47099999999999997,"5min":0.495,"15min":0.42699999999999999},"min":{"1min":0.249,"5min":0.248,"15min":0.223},"max":{"1min":1.6180000000000001,"5min":2.1880000000000002,"15min":2.1880000000000002},"last":0.41799999999999998}]},{"osd":6,"last update":"Tue Sep 10 19:15:48 2024","interfaces":[{"interface":"back","average":{"1min":0.61399999999999999,"5min":0.64000000000000001,"15min":0.48799999999999999},"min":{"1min":0.30599999999999999,"5min":0.253,"15min":0.253},"max":{"1min":1.643,"5min":3.964,"15min":3.964},"last":0.501},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.59499999999999997,"15min":0.47399999999999998},"min":{"1min":0.31900000000000001,"5min":0.25,"15min":0.25},"max":{"1min":1.4319999999999999,"5min":3.0129999999999999,"15min":3.0129999999999999},"last":0.59399999999999997}]},{"osd":7,"last update":"Tue Sep 10 19:15:03 2024","interfaces":[{"interface":"back","average":{"1min":0.95899999999999996,"5min":0.91200000000000003,"15min":0.65400000000000003},"min":{"1min":0.32400000000000001,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":3.5939999999999999,"5min":23.081,"15min":23.081},"last":0.51800000000000002},{"interface":"front","average":{"1min":0.64400000000000002,"5min":0.85599999999999998,"15min":0.62},"min":{"1min":0.27500000000000002,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":1.901,"5min":22.666,"15min":22.666},"last":0.441}]}]},{"osd":3,"up_from":26,"seq":111669149788,"num_pgs":69,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":306516,"kb_used_data":840,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":93438636,"statfs":{"total":95995035648,"available":95681163264,"internally_reserved":0,"allocated":860160,"data_stored":510192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 19:15:34 2024","interfaces":[{"interface":"back","average":{"1min":2.2490000000000001,"5min":1.2729999999999999,"15min":0.69499999999999995},"min":{"1min":0.20100000000000001,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":17.518000000000001,"5min":17.518000000000001,"15min":17.518000000000001},"last":0.25800000000000001},{"interface":"front","average":{"1min":2.306,"5min":1.488,"15min":0.77400000000000002},"min":{"1min":0.17399999999999999,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":17.609000000000002,"5min":17.609000000000002,"15min":17.609000000000002},"last":0.33600000000000002}]},{"osd":1,"last update":"Tue Sep 10 19:12:27 2024","interfaces":[{"interface":"back","average":{"1min":0.52800000000000002,"5min":0.44400000000000001,"15min":0.45200000000000001},"min":{"1min":0.216,"5min":0.216,"15min":0.216},"max":{"1min":3.5150000000000001,"5min":3.5150000000000001,"15min":3.5150000000000001},"last":0.318},{"interface":"front","average":{"1min":0.46600000000000003,"5min":0.40000000000000002,"15min":0.39800000000000002},"min":{"1min":0.187,"5min":0.187,"15min":0.187},"max":{"1min":1.3640000000000001,"5min":1.3640000000000001,"15min":1.3640000000000001},"last":0.30299999999999999}]},{"osd":2,"last update":"Tue Sep 10 19:15:34 2024","interfaces":[{"interface":"back","average":{"1min":1.3049999999999999,"5min":0.92000000000000004,"15min":0.59799999999999998},"min":{"1min":0.218,"5min":0.186,"15min":0.186},"max":{"1min":4.2290000000000001,"5min":4.2290000000000001,"15min":4.2290000000000001},"last":0.35799999999999998},{"interface":"front","average":{"1min":2.379,"5min":1.264,"15min":0.69799999999999995},"min":{"1min":0.314,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":17.806000000000001,"5min":17.806000000000001,"15min":17.806000000000001},"last":0.29099999999999998}]},{"osd":4,"last update":"Tue Sep 10 19:15:01 2024","interfaces":[{"interface":"back","average":{"1min":1.1870000000000001,"5min":1.071,"15min":0.71999999999999997},"min":{"1min":0.23599999999999999,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":3.7839999999999998,"5min":11.964,"15min":11.964},"last":0.375},{"interface":"front","average":{"1min":2.1120000000000001,"5min":1.339,"15min":0.81100000000000005},"min":{"1min":0.27700000000000002,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":4.0359999999999996,"5min":11.881,"15min":11.881},"last":0.41199999999999998}]},{"osd":5,"last update":"Tue Sep 10 19:15:19 2024","interfaces":[{"interface":"back","average":{"1min":0.82699999999999996,"5min":0.92900000000000005,"15min":0.68700000000000006},"min":{"1min":0.32200000000000001,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":1.9730000000000001,"5min":4.6070000000000002,"15min":4.6070000000000002},"last":0.40100000000000002},{"interface":"front","average":{"1min":1.5669999999999999,"5min":1.2350000000000001,"15min":0.78500000000000003},"min":{"1min":0.22,"5min":0.22,"15min":0.22},"max":{"1min":3.54,"5min":5.9950000000000001,"15min":5.9950000000000001},"last":0.49299999999999999}]},{"osd":6,"last update":"Tue Sep 10 19:15:38 2024","interfaces":[{"interface":"back","average":{"1min":2.0449999999999999,"5min":1.325,"15min":0.82699999999999996},"min":{"1min":0.38900000000000001,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":17.861000000000001,"5min":17.861000000000001,"15min":17.861000000000001},"last":0.46700000000000003},{"interface":"front","average":{"1min":2.0750000000000002,"5min":1.075,"15min":0.72899999999999998},"min":{"1min":0.311,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":17.687000000000001,"5min":17.687000000000001,"15min":17.687000000000001},"last":0.437}]},{"osd":7,"last update":"Tue Sep 10 19:15:10 2024","interfaces":[{"interface":"back","average":{"1min":1.226,"5min":0.99399999999999999,"15min":0.71799999999999997},"min":{"1min":0.31,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":2.859,"5min":13.538,"15min":13.538},"last":0.38300000000000001},{"interface":"front","average":{"1min":1.1539999999999999,"5min":1.1299999999999999,"15min":0.76600000000000001},"min":{"1min":0.41299999999999998,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":3.8130000000000002,"5min":13.522,"15min":13.522},"last":0.42599999999999999}]}]},{"osd":2,"up_from":18,"seq":77309411425,"num_pgs":60,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":302320,"kb_used_data":804,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":93442832,"statfs":{"total":95995035648,"available":95685459968,"internally_reserved":0,"allocated":823296,"data_stored":508718,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 19:15:21 2024","interfaces":[{"interface":"back","average":{"1min":1.909,"5min":1.8089999999999999,"15min":0.83199999999999996},"min":{"1min":0.20100000000000001,"5min":0.188,"15min":0.188},"max":{"1min":6.375,"5min":62.543999999999997,"15min":62.543999999999997},"last":0.27600000000000002},{"interface":"front","average":{"1min":1.5229999999999999,"5min":1.6140000000000001,"15min":0.78600000000000003},"min":{"1min":0.186,"5min":0.186,"15min":0.186},"max":{"1min":5.2839999999999998,"5min":62.576999999999998,"15min":62.576999999999998},"last":0.48799999999999999}]},{"osd":1,"last update":"Tue Sep 10 19:12:08 2024","interfaces":[{"interface":"back","average":{"1min":0.317,"5min":0.378,"15min":0.375},"min":{"1min":0.23400000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.46800000000000003,"5min":0.69799999999999995,"15min":0.69799999999999995},"last":0.308},{"interface":"front","average":{"1min":0.33300000000000002,"5min":0.38600000000000001,"15min":0.38800000000000001},"min":{"1min":0.255,"5min":0.255,"15min":0.255},"max":{"1min":0.499,"5min":0.63400000000000001,"15min":0.63400000000000001},"last":0.438}]},{"osd":3,"last update":"Tue Sep 10 19:15:42 2024","interfaces":[{"interface":"back","average":{"1min":1.3620000000000001,"5min":1.911,"15min":0.93899999999999995},"min":{"1min":0.20799999999999999,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":4.4649999999999999,"5min":63.997,"15min":63.997},"last":0.36199999999999999},{"interface":"front","average":{"1min":2.827,"5min":1.9730000000000001,"15min":0.95599999999999996},"min":{"1min":0.27400000000000002,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":8.9800000000000004,"5min":62.365000000000002,"15min":62.365000000000002},"last":0.52800000000000002}]},{"osd":4,"last update":"Tue Sep 10 19:14:58 2024","interfaces":[{"interface":"back","average":{"1min":1.2450000000000001,"5min":0.89100000000000001,"15min":0.68400000000000005},"min":{"1min":0.25800000000000001,"5min":0.254,"15min":0.254},"max":{"1min":6.0300000000000002,"5min":6.0300000000000002,"15min":6.0300000000000002},"last":0.54100000000000004},{"interface":"front","average":{"1min":0.64100000000000001,"5min":0.76600000000000001,"15min":0.63600000000000001},"min":{"1min":0.24399999999999999,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":2.0630000000000002,"5min":4.0750000000000002,"15min":4.0750000000000002},"last":0.626}]},{"osd":5,"last update":"Tue Sep 10 19:15:21 2024","interfaces":[{"interface":"back","average":{"1min":2.0779999999999998,"5min":0.98999999999999999,"15min":0.68000000000000005},"min":{"1min":0.33600000000000002,"5min":0.255,"15min":0.255},"max":{"1min":13.17,"5min":13.17,"15min":13.17},"last":0.61199999999999999},{"interface":"front","average":{"1min":2.1560000000000001,"5min":1.137,"15min":0.76500000000000001},"min":{"1min":0.29599999999999999,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":13.744999999999999,"5min":13.744999999999999,"15min":13.744999999999999},"last":0.57899999999999996}]},{"osd":6,"last update":"Tue Sep 10 19:15:42 2024","interfaces":[{"interface":"back","average":{"1min":1.1479999999999999,"5min":1.206,"15min":0.77400000000000002},"min":{"1min":0.35699999999999998,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":3.2799999999999998,"5min":12.611000000000001,"15min":12.611000000000001},"last":0.64000000000000001},{"interface":"front","average":{"1min":1.7929999999999999,"5min":1.3220000000000001,"15min":0.80800000000000005},"min":{"1min":0.28199999999999997,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":5.8620000000000001,"5min":14.459,"15min":14.459},"last":0.72599999999999998}]},{"osd":7,"last update":"Tue Sep 10 19:15:05 2024","interfaces":[{"interface":"back","average":{"1min":2.012,"5min":1.1279999999999999,"15min":0.78000000000000003},"min":{"1min":0.32500000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":13.779999999999999,"5min":13.779999999999999,"15min":13.779999999999999},"last":0.66600000000000004},{"interface":"front","average":{"1min":1.617,"5min":1.006,"15min":0.70099999999999996},"min":{"1min":0.36599999999999999,"5min":0.312,"15min":0.312},"max":{"1min":11.714,"5min":11.714,"15min":11.714},"last":0.70199999999999996}]}]},{"osd":1,"up_from":0,"seq":592705486855,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":8,"seq":34359738475,"num_pgs":64,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":311244,"kb_used_data":1408,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":93433908,"statfs":{"total":95995035648,"available":95676321792,"internally_reserved":0,"allocated":1441792,"data_stored":1100302,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Sep 10 19:12:44 2024","interfaces":[{"interface":"back","average":{"1min":0.73199999999999998,"5min":0.42999999999999999,"15min":0.35799999999999998},"min":{"1min":0.17399999999999999,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":2.7320000000000002,"5min":2.7320000000000002,"15min":2.7320000000000002},"last":0.25800000000000001},{"interface":"front","average":{"1min":0.82299999999999995,"5min":0.45100000000000001,"15min":0.36799999999999999},"min":{"1min":0.21199999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":2.4609999999999999,"5min":2.4609999999999999,"15min":2.4609999999999999},"last":0.34799999999999998}]},{"osd":2,"last update":"Tue Sep 10 19:15:18 2024","interfaces":[{"interface":"back","average":{"1min":1.4630000000000001,"5min":0.88800000000000001,"15min":0.55300000000000005},"min":{"1min":0.23499999999999999,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":6.4640000000000004,"5min":6.4640000000000004,"15min":6.4640000000000004},"last":0.29799999999999999},{"interface":"front","average":{"1min":2.3050000000000002,"5min":0.98299999999999998,"15min":0.57699999999999996},"min":{"1min":0.28100000000000003,"5min":0.187,"15min":0.187},"max":{"1min":9.9399999999999995,"5min":9.9399999999999995,"15min":9.9399999999999995},"last":0.30399999999999999}]},{"osd":3,"last update":"Tue Sep 10 19:15:38 2024","interfaces":[{"interface":"back","average":{"1min":2.0800000000000001,"5min":1.028,"15min":0.626},"min":{"1min":0.32300000000000001,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":6.5590000000000002,"5min":6.5590000000000002,"15min":6.5590000000000002},"last":0.33200000000000002},{"interface":"front","average":{"1min":2.581,"5min":1.26,"15min":0.69299999999999995},"min":{"1min":0.28100000000000003,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":7.2140000000000004,"5min":7.2140000000000004,"15min":7.2140000000000004},"last":0.31900000000000001}]},{"osd":4,"last update":"Tue Sep 10 19:14:54 2024","interfaces":[{"interface":"back","average":{"1min":0.87,"5min":0.76800000000000002,"15min":0.63300000000000001},"min":{"1min":0.28299999999999997,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":4.0049999999999999,"5min":5.8499999999999996,"15min":5.8499999999999996},"last":0.56299999999999994},{"interface":"front","average":{"1min":0.80900000000000005,"5min":0.75800000000000001,"15min":0.627},"min":{"1min":0.27100000000000002,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":2.556,"5min":3.2999999999999998,"15min":3.2999999999999998},"last":0.61699999999999999}]},{"osd":5,"last update":"Tue Sep 10 19:15:18 2024","interfaces":[{"interface":"back","average":{"1min":1.661,"5min":0.94199999999999995,"15min":0.70399999999999996},"min":{"1min":0.38800000000000001,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":5.3860000000000001,"5min":5.3860000000000001,"15min":5.3860000000000001},"last":0.59199999999999997},{"interface":"front","average":{"1min":1.2529999999999999,"5min":0.88300000000000001,"15min":0.67800000000000005},"min":{"1min":0.26000000000000001,"5min":0.253,"15min":0.253},"max":{"1min":5.9749999999999996,"5min":5.9749999999999996,"15min":5.9749999999999996},"last":0.55100000000000005}]},{"osd":6,"last update":"Tue Sep 10 19:15:38 2024","interfaces":[{"interface":"back","average":{"1min":0.81699999999999995,"5min":0.93500000000000005,"15min":0.68899999999999995},"min":{"1min":0.371,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":1.806,"5min":5.8730000000000002,"15min":5.8730000000000002},"last":0.60399999999999998},{"interface":"front","average":{"1min":0.85699999999999998,"5min":0.94899999999999995,"15min":0.70699999999999996},"min":{"1min":0.35799999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":3.8620000000000001,"5min":5.2850000000000001,"15min":5.2850000000000001},"last":0.66300000000000003}]},{"osd":7,"last update":"Tue Sep 10 19:15:06 2024","interfaces":[{"interface":"back","average":{"1min":1.3109999999999999,"5min":0.90700000000000003,"15min":0.68700000000000006},"min":{"1min":0.318,"5min":0.318,"15min":0.318},"max":{"1min":5.1219999999999999,"5min":5.1219999999999999,"15min":5.1219999999999999},"last":0.63300000000000001},{"interface":"front","average":{"1min":1.379,"5min":0.85699999999999998,"15min":0.66000000000000003},"min":{"1min":0.30499999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":6.0469999999999997,"5min":6.0469999999999997,"15min":6.0469999999999997},"last":0.56899999999999995}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":966,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1236,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1236,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":362,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1650,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":2118,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1018,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":2008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1942,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1018036224,"data_stored":1018036224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T19:15:53.634 INFO:tasks.cephadm.ceph_manager.ceph:would drop below min_pgs, proj -9, currently 1 2024-09-10T19:15:53.634 INFO:tasks.cephadm.ceph_manager.ceph:removing pool_name unique_pool_0 2024-09-10T19:15:53.634 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph osd pool rm unique_pool_0 unique_pool_0 --yes-i-really-really-mean-it 2024-09-10T19:15:53.697 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:15:53.709 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:15:53.731 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:15:53.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:15:53.732 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:15:53.833 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:15:53.853 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:15:53.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:15:53.854 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:15:53.955 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:15:53.977 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:15:53.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:15:53.978 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:15:54.079 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:15:54.101 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:15:54.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:15:54.102 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:15:54.202 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:15:54.212 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-10T19:15:54.227 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:15:54.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:15:54.228 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:15:54.328 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:15:54.352 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:15:54.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:15:54.353 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:15:54.402 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:15:54 smithi183 ceph-mon[29736]: 4.1d scrub starts 2024-09-10T19:15:54.402 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:15:54 smithi183 ceph-mon[29736]: 4.1d scrub ok 2024-09-10T19:15:54.402 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:15:54 smithi183 ceph-mon[29736]: 2.c scrub starts 2024-09-10T19:15:54.402 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:15:54 smithi183 ceph-mon[29736]: 2.c scrub ok 2024-09-10T19:15:54.402 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:15:54 smithi183 ceph-mon[29736]: 4.13 scrub starts 2024-09-10T19:15:54.402 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:15:54 smithi183 ceph-mon[29736]: 4.13 scrub ok 2024-09-10T19:15:54.402 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:15:54 smithi183 ceph-mon[41562]: 4.1d scrub starts 2024-09-10T19:15:54.402 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:15:54 smithi183 ceph-mon[41562]: 4.1d scrub ok 2024-09-10T19:15:54.402 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:15:54 smithi183 ceph-mon[41562]: 2.c scrub starts 2024-09-10T19:15:54.402 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:15:54 smithi183 ceph-mon[41562]: 2.c scrub ok 2024-09-10T19:15:54.403 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:15:54 smithi183 ceph-mon[41562]: 4.13 scrub starts 2024-09-10T19:15:54.403 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:15:54 smithi183 ceph-mon[41562]: 4.13 scrub ok 2024-09-10T19:15:54.454 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:15:54.465 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:15:54 smithi191 ceph-mon[32884]: 4.1d scrub starts 2024-09-10T19:15:54.465 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:15:54 smithi191 ceph-mon[32884]: 4.1d scrub ok 2024-09-10T19:15:54.465 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:15:54 smithi191 ceph-mon[32884]: 2.c scrub starts 2024-09-10T19:15:54.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:15:54 smithi191 ceph-mon[32884]: 2.c scrub ok 2024-09-10T19:15:54.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:15:54 smithi191 ceph-mon[32884]: 4.13 scrub starts 2024-09-10T19:15:54.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:15:54 smithi191 ceph-mon[32884]: 4.13 scrub ok 2024-09-10T19:15:54.477 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:15:54.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:15:54.478 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:15:54.578 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:15:54.600 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:15:54.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:15:54.601 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:15:54.702 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:15:54.727 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:15:54.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:15:54.728 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:15:54.829 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:15:54.850 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:15:54.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:15:54.851 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:15:54.951 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:15:54.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:15:54.973 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:15:54.974 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:15:55.074 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:15:55.097 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:15:55.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:15:55.098 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:15:55.198 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:15:55.219 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:15:55.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:15:55.220 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:15:55.321 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:15:55.345 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:15:55.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:15:55.346 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:15:55.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:15:55 smithi183 ceph-mon[29736]: 5.1f scrub starts 2024-09-10T19:15:55.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:15:55 smithi183 ceph-mon[29736]: 5.1f scrub ok 2024-09-10T19:15:55.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:15:55 smithi183 ceph-mon[29736]: pgmap v229: 148 pgs: 148 active+clean; 582 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 762 op/s; 22 B/s, 0 objects/s recovering 2024-09-10T19:15:55.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:15:55 smithi183 ceph-mon[29736]: 2.2 scrub starts 2024-09-10T19:15:55.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:15:55 smithi183 ceph-mon[29736]: 2.2 scrub ok 2024-09-10T19:15:55.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:15:55 smithi183 ceph-mon[41562]: 5.1f scrub starts 2024-09-10T19:15:55.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:15:55 smithi183 ceph-mon[41562]: 5.1f scrub ok 2024-09-10T19:15:55.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:15:55 smithi183 ceph-mon[41562]: pgmap v229: 148 pgs: 148 active+clean; 582 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 762 op/s; 22 B/s, 0 objects/s recovering 2024-09-10T19:15:55.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:15:55 smithi183 ceph-mon[41562]: 2.2 scrub starts 2024-09-10T19:15:55.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:15:55 smithi183 ceph-mon[41562]: 2.2 scrub ok 2024-09-10T19:15:55.446 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:15:55.465 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:15:55 smithi191 ceph-mon[32884]: 5.1f scrub starts 2024-09-10T19:15:55.465 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:15:55 smithi191 ceph-mon[32884]: 5.1f scrub ok 2024-09-10T19:15:55.465 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:15:55 smithi191 ceph-mon[32884]: pgmap v229: 148 pgs: 148 active+clean; 582 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 762 op/s; 22 B/s, 0 objects/s recovering 2024-09-10T19:15:55.465 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:15:55 smithi191 ceph-mon[32884]: 2.2 scrub starts 2024-09-10T19:15:55.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:15:55 smithi191 ceph-mon[32884]: 2.2 scrub ok 2024-09-10T19:15:55.469 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:15:55.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:15:55.470 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:15:55.570 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:15:55.593 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:15:55.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:15:55.594 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:15:55.694 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:15:55.716 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:15:55.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:15:55.717 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:15:55.742 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:15:55.818 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:15:55.839 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:15:55.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:15:55.840 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:15:55.941 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:15:55.967 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:15:55.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:15:55.968 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:15:56.069 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:15:56.091 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:15:56.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:15:56.092 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:15:56.188 INFO:teuthology.orchestra.run.smithi183.stderr:noscrub is set 2024-09-10T19:15:56.193 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:15:56.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:15:56.217 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:15:56.217 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:15:56.272 INFO:teuthology.orchestra.run.smithi183.stdout:{ 2024-09-10T19:15:56.272 INFO:teuthology.orchestra.run.smithi183.stdout: "ops": [], 2024-09-10T19:15:56.273 INFO:teuthology.orchestra.run.smithi183.stdout: "num_ops": 0 2024-09-10T19:15:56.273 INFO:teuthology.orchestra.run.smithi183.stdout:} 2024-09-10T19:15:56.318 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:15:56.343 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:15:56.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:15:56.345 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:15:56.431 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:15:56 smithi183 ceph-mon[41562]: from='client.? 172.21.15.183:0/2159202716' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-10T19:15:56.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:15:56 smithi183 ceph-mon[29736]: from='client.? 172.21.15.183:0/2159202716' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-10T19:15:56.445 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:15:56.465 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:15:56 smithi191 ceph-mon[32884]: from='client.? 172.21.15.183:0/2159202716' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-10T19:15:56.470 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:15:56.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:15:56.471 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:15:56.572 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:15:56.597 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:15:56.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:15:56.598 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:15:56.699 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:15:56.721 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:15:56.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:15:56.722 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:15:56.823 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:15:56.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:15:56.849 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:15:56.849 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:15:56.866 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph daemon osd.3 dump_blocked_ops 2024-09-10T19:15:56.950 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:15:56.976 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:15:56.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:15:56.977 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:15:57.078 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:15:57.102 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:15:57.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:15:57.103 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:15:57.204 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:15:57.229 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:15:57.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:15:57.230 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:15:57.331 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:15:57.353 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:15:57.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:15:57.354 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:15:57.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:15:57 smithi183 ceph-mon[29736]: pgmap v230: 148 pgs: 148 active+clean; 582 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 2.2 KiB/s rd, 96 B/s wr, 54 op/s; 20 B/s, 0 objects/s recovering 2024-09-10T19:15:57.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:15:57 smithi183 ceph-mon[29736]: 4.7 deep-scrub starts 2024-09-10T19:15:57.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:15:57 smithi183 ceph-mon[29736]: 4.7 deep-scrub ok 2024-09-10T19:15:57.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:15:57 smithi183 ceph-mon[29736]: 5.13 scrub starts 2024-09-10T19:15:57.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:15:57 smithi183 ceph-mon[29736]: 5.13 scrub ok 2024-09-10T19:15:57.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:15:57 smithi183 ceph-mon[29736]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T19:15:57.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:15:57 smithi183 ceph-mon[29736]: from='client.? 172.21.15.183:0/2159202716' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-10T19:15:57.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:15:57 smithi183 ceph-mon[29736]: osdmap e145: 8 total, 8 up, 7 in 2024-09-10T19:15:57.433 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:15:57 smithi183 ceph-mon[41562]: pgmap v230: 148 pgs: 148 active+clean; 582 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 2.2 KiB/s rd, 96 B/s wr, 54 op/s; 20 B/s, 0 objects/s recovering 2024-09-10T19:15:57.433 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:15:57 smithi183 ceph-mon[41562]: 4.7 deep-scrub starts 2024-09-10T19:15:57.433 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:15:57 smithi183 ceph-mon[41562]: 4.7 deep-scrub ok 2024-09-10T19:15:57.433 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:15:57 smithi183 ceph-mon[41562]: 5.13 scrub starts 2024-09-10T19:15:57.433 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:15:57 smithi183 ceph-mon[41562]: 5.13 scrub ok 2024-09-10T19:15:57.433 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:15:57 smithi183 ceph-mon[41562]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T19:15:57.433 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:15:57 smithi183 ceph-mon[41562]: from='client.? 172.21.15.183:0/2159202716' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-10T19:15:57.433 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:15:57 smithi183 ceph-mon[41562]: osdmap e145: 8 total, 8 up, 7 in 2024-09-10T19:15:57.450 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:15:57 smithi191 ceph-mon[32884]: pgmap v230: 148 pgs: 148 active+clean; 582 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 2.2 KiB/s rd, 96 B/s wr, 54 op/s; 20 B/s, 0 objects/s recovering 2024-09-10T19:15:57.450 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:15:57 smithi191 ceph-mon[32884]: 4.7 deep-scrub starts 2024-09-10T19:15:57.450 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:15:57 smithi191 ceph-mon[32884]: 4.7 deep-scrub ok 2024-09-10T19:15:57.450 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:15:57 smithi191 ceph-mon[32884]: 5.13 scrub starts 2024-09-10T19:15:57.451 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:15:57 smithi191 ceph-mon[32884]: 5.13 scrub ok 2024-09-10T19:15:57.451 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:15:57 smithi191 ceph-mon[32884]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T19:15:57.451 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:15:57 smithi191 ceph-mon[32884]: from='client.? 172.21.15.183:0/2159202716' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-10T19:15:57.451 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:15:57 smithi191 ceph-mon[32884]: osdmap e145: 8 total, 8 up, 7 in 2024-09-10T19:15:57.454 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:15:57.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:15:57.480 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:15:57.481 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:15:57.581 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:15:57.608 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:15:57.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:15:57.609 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:15:57.710 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:15:57.715 INFO:journalctl@ceph.mgr.x.smithi191.stdout:Sep 10 19:15:57 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-x[34967]: ::ffff:172.21.15.191 - - [10/Sep/2024:19:15:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T19:15:57.735 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:15:57.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:15:57.737 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:15:57.815 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:15:57.838 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:15:57.863 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:15:57.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:15:57.864 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:15:57.965 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:15:57.990 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:15:57.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:15:57.992 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:15:58.092 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:15:58.113 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:15:58.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:15:58.114 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:15:58.215 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:15:58.240 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:15:58.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:15:58.241 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:15:58.342 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:15:58.366 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:15:58.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:15:58.367 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:15:58.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:15:58 smithi183 ceph-mon[29736]: purged_snaps scrub starts 2024-09-10T19:15:58.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:15:58 smithi183 ceph-mon[29736]: purged_snaps scrub ok 2024-09-10T19:15:58.431 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:15:58 smithi183 ceph-mon[41562]: purged_snaps scrub starts 2024-09-10T19:15:58.431 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:15:58 smithi183 ceph-mon[41562]: purged_snaps scrub ok 2024-09-10T19:15:58.465 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:15:58 smithi191 ceph-mon[32884]: purged_snaps scrub starts 2024-09-10T19:15:58.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:15:58 smithi191 ceph-mon[32884]: purged_snaps scrub ok 2024-09-10T19:15:58.468 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:15:58.492 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:15:58.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:15:58.493 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:15:58.596 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:15:58.621 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:15:58.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:15:58.622 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:15:58.635 INFO:tasks.thrashosds.thrasher:in_osds: [0, 2, 3, 4, 5, 6, 7] out_osds: [1] dead_osds: [] live_osds: [0, 2, 3, 4, 5, 6, 7, 1] 2024-09-10T19:15:58.635 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-10T19:15:58.636 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph pg dump --format=json 2024-09-10T19:15:58.723 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:15:58.747 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:15:58.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:15:58.748 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:15:58.848 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:15:58.871 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:15:58.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:15:58.872 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:15:58.973 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:15:58.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:15:58.995 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:15:58.995 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:15:59.096 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:15:59.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:15:59.119 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:15:59.119 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:15:59.220 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:15:59.246 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:15:59.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:15:59.247 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:15:59.325 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:15:59 smithi183 ceph-mon[29736]: pgmap v232: 148 pgs: 148 active+clean; 582 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 2.3 KiB/s rd, 102 B/s wr, 57 op/s; 21 B/s, 0 objects/s recovering 2024-09-10T19:15:59.325 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:15:59 smithi183 ceph-mon[29736]: osdmap e146: 8 total, 8 up, 7 in 2024-09-10T19:15:59.325 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:15:59 smithi183 ceph-mon[29736]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:15:59.325 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:15:59 smithi183 ceph-mon[41562]: pgmap v232: 148 pgs: 148 active+clean; 582 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 2.3 KiB/s rd, 102 B/s wr, 57 op/s; 21 B/s, 0 objects/s recovering 2024-09-10T19:15:59.325 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:15:59 smithi183 ceph-mon[41562]: osdmap e146: 8 total, 8 up, 7 in 2024-09-10T19:15:59.326 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:15:59 smithi183 ceph-mon[41562]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:15:59.348 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:15:59.369 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:15:59.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:15:59.370 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:15:59.465 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:15:59 smithi191 ceph-mon[32884]: pgmap v232: 148 pgs: 148 active+clean; 582 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 2.3 KiB/s rd, 102 B/s wr, 57 op/s; 21 B/s, 0 objects/s recovering 2024-09-10T19:15:59.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:15:59 smithi191 ceph-mon[32884]: osdmap e146: 8 total, 8 up, 7 in 2024-09-10T19:15:59.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:15:59 smithi191 ceph-mon[32884]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:15:59.466 INFO:journalctl@ceph.iscsi.iscsi.a.smithi191.stdout:Sep 10 19:15:59 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-iscsi-iscsi-a[61816]: debug there is no tcmu-runner data available 2024-09-10T19:15:59.471 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:15:59.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:15:59.497 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:15:59.497 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:15:59.567 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:15:59.568 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:15:59.598 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:15:59.624 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:15:59.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:15:59.625 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:15:59.726 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:15:59.751 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:15:59.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:15:59.752 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:15:59.853 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:15:59.876 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:15:59.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:15:59.877 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:15:59.978 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:15:59.999 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:16:00.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:16:00.000 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:16:00.101 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:16:00.122 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:16:00.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:16:00.123 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:16:00.198 INFO:teuthology.orchestra.run.smithi183.stderr:pool 'unique_pool_0' removed 2024-09-10T19:16:00.199 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:16:00 smithi183 ceph-mon[29736]: from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:16:00.199 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:16:00 smithi183 ceph-mon[29736]: from='client.? 172.21.15.183:0/4023001106' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true}]: dispatch 2024-09-10T19:16:00.199 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:16:00 smithi183 ceph-mon[41562]: from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:16:00.199 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:16:00 smithi183 ceph-mon[41562]: from='client.? 172.21.15.183:0/4023001106' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true}]: dispatch 2024-09-10T19:16:00.224 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:16:00.284 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:16:00.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:16:00.285 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:16:00.333 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-10T19:16:00.386 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:16:00.408 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:16:00.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:16:00.409 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:16:00.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:16:00 smithi191 ceph-mon[32884]: from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:16:00.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:16:00 smithi191 ceph-mon[32884]: from='client.? 172.21.15.183:0/4023001106' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true}]: dispatch 2024-09-10T19:16:00.509 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:16:00.532 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:16:00.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:16:00.533 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:16:00.636 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:16:00.657 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:16:00.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:16:00.658 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:16:00.759 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:16:00.781 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:16:00.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:16:00.782 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:16:00.883 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:16:00.905 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:16:00.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:16:00.906 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:16:01.007 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:16:01.028 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:16:01.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:16:01.029 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:16:01.130 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:16:01.152 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:16:01.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:16:01.153 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:16:01.254 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:16:01.275 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:16:01.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:16:01.276 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:16:01.377 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:16:01.399 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:16:01.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:16:01.401 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:16:01.465 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:16:01 smithi191 ceph-mon[32884]: pgmap v234: 148 pgs: 148 active+clean; 582 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 639 B/s rd, 0 op/s 2024-09-10T19:16:01.465 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:16:01 smithi191 ceph-mon[32884]: purged_snaps scrub starts 2024-09-10T19:16:01.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:16:01 smithi191 ceph-mon[32884]: purged_snaps scrub ok 2024-09-10T19:16:01.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:16:01 smithi191 ceph-mon[32884]: from='client.? 172.21.15.183:0/4023001106' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true}]': finished 2024-09-10T19:16:01.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:16:01 smithi191 ceph-mon[32884]: osdmap e147: 8 total, 8 up, 7 in 2024-09-10T19:16:01.502 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:16:01.524 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:16:01.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:16:01.525 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:16:01.626 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:16:01.643 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Sep 10 19:16:01 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-y[29989]: ::ffff:172.21.15.191 - - [10/Sep/2024:19:16:01] "GET /metrics HTTP/1.1" 200 223966 "" "Prometheus/2.43.0" 2024-09-10T19:16:01.643 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:16:01 smithi183 ceph-mon[29736]: pgmap v234: 148 pgs: 148 active+clean; 582 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 639 B/s rd, 0 op/s 2024-09-10T19:16:01.644 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:16:01 smithi183 ceph-mon[29736]: purged_snaps scrub starts 2024-09-10T19:16:01.644 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:16:01 smithi183 ceph-mon[29736]: purged_snaps scrub ok 2024-09-10T19:16:01.644 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:16:01 smithi183 ceph-mon[29736]: from='client.? 172.21.15.183:0/4023001106' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true}]': finished 2024-09-10T19:16:01.644 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:16:01 smithi183 ceph-mon[29736]: osdmap e147: 8 total, 8 up, 7 in 2024-09-10T19:16:01.644 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:16:01 smithi183 ceph-mon[41562]: pgmap v234: 148 pgs: 148 active+clean; 582 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 639 B/s rd, 0 op/s 2024-09-10T19:16:01.644 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:16:01 smithi183 ceph-mon[41562]: purged_snaps scrub starts 2024-09-10T19:16:01.644 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:16:01 smithi183 ceph-mon[41562]: purged_snaps scrub ok 2024-09-10T19:16:01.644 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:16:01 smithi183 ceph-mon[41562]: from='client.? 172.21.15.183:0/4023001106' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true}]': finished 2024-09-10T19:16:01.644 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:16:01 smithi183 ceph-mon[41562]: osdmap e147: 8 total, 8 up, 7 in 2024-09-10T19:16:01.648 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:16:01.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:16:01.649 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:16:01.676 INFO:teuthology.task.full_sequential:In full_sequential, running task radosbench... 2024-09-10T19:16:01.677 INFO:tasks.radosbench:Beginning radosbench... 2024-09-10T19:16:01.678 INFO:tasks.cephadm.ceph_manager.ceph:creating pool_name unique_pool_1 2024-09-10T19:16:01.678 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph osd pool create unique_pool_1 16 2024-09-10T19:16:01.750 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:16:01.780 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:16:01.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:16:01.781 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:16:01.882 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:16:01.904 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:16:01.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:16:01.905 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:16:02.006 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:16:02.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:16:02.028 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:16:02.028 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:16:02.129 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:16:02.152 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:16:02.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:16:02.153 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:16:02.254 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:16:02.278 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:16:02.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:16:02.279 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:16:02.380 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:16:02.402 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:16:02.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:16:02.403 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:16:02.465 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:16:02 smithi191 ceph-mon[32884]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T19:16:02.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:16:02 smithi191 ceph-mon[32884]: from='mgr.14556 ' entity='mgr.y' 2024-09-10T19:16:02.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:16:02 smithi191 ceph-mon[32884]: from='mgr.14556 ' entity='mgr.y' 2024-09-10T19:16:02.504 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:16:02.527 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:16:02.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:16:02.528 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:16:02.629 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:16:02.651 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:16:02.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:16:02.652 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:16:02.657 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:16:02 smithi183 ceph-mon[41562]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T19:16:02.657 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:16:02 smithi183 ceph-mon[41562]: from='mgr.14556 ' entity='mgr.y' 2024-09-10T19:16:02.657 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:16:02 smithi183 ceph-mon[41562]: from='mgr.14556 ' entity='mgr.y' 2024-09-10T19:16:02.658 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:16:02 smithi183 ceph-mon[29736]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T19:16:02.658 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:16:02 smithi183 ceph-mon[29736]: from='mgr.14556 ' entity='mgr.y' 2024-09-10T19:16:02.658 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:16:02 smithi183 ceph-mon[29736]: from='mgr.14556 ' entity='mgr.y' 2024-09-10T19:16:02.753 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:16:02.775 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:16:02.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:16:02.776 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:16:02.877 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:16:02.899 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:16:02.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:16:02.900 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:16:03.001 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:16:03.024 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:16:03.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:16:03.025 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:16:03.126 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:16:03.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:16:03.148 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:16:03.149 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:16:03.249 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:16:03.271 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:16:03.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:16:03.272 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:16:03.373 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:16:03.397 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:16:03.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:16:03.398 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:16:03.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:16:03 smithi191 ceph-mon[32884]: pgmap v236: 132 pgs: 132 active+clean; 582 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 682 B/s rd, 0 op/s 2024-09-10T19:16:03.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:16:03 smithi191 ceph-mon[32884]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T19:16:03.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:16:03 smithi191 ceph-mon[32884]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T19:16:03.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:16:03 smithi191 ceph-mon[32884]: from='mgr.14556 ' entity='mgr.y' 2024-09-10T19:16:03.499 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:16:03.526 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:16:03.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:16:03.527 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:16:03.542 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:16:03 smithi183 ceph-mon[29736]: pgmap v236: 132 pgs: 132 active+clean; 582 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 682 B/s rd, 0 op/s 2024-09-10T19:16:03.542 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:16:03 smithi183 ceph-mon[29736]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T19:16:03.542 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:16:03 smithi183 ceph-mon[29736]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T19:16:03.542 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:16:03 smithi183 ceph-mon[29736]: from='mgr.14556 ' entity='mgr.y' 2024-09-10T19:16:03.543 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:16:03 smithi183 ceph-mon[41562]: pgmap v236: 132 pgs: 132 active+clean; 582 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 682 B/s rd, 0 op/s 2024-09-10T19:16:03.543 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:16:03 smithi183 ceph-mon[41562]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T19:16:03.543 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:16:03 smithi183 ceph-mon[41562]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T19:16:03.543 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:16:03 smithi183 ceph-mon[41562]: from='mgr.14556 ' entity='mgr.y' 2024-09-10T19:16:03.628 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:16:03.650 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:16:03.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:16:03.651 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:16:03.751 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:16:03.773 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:16:03.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:16:03.774 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:16:03.875 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:16:03.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:16:03.899 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:16:03.899 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:16:03.917 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-09-10T19:16:03.919 INFO:teuthology.orchestra.run.smithi183.stderr:dumped all 2024-09-10T19:16:03.983 INFO:teuthology.orchestra.run.smithi183.stdout:osd.0: {} 2024-09-10T19:16:03.983 INFO:teuthology.orchestra.run.smithi183.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-10T19:16:03.999 INFO:teuthology.orchestra.run.smithi183.stdout:osd.1: {} 2024-09-10T19:16:03.999 INFO:teuthology.orchestra.run.smithi183.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-10T19:16:04.000 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:16:04.013 INFO:teuthology.orchestra.run.smithi183.stdout:osd.2: {} 2024-09-10T19:16:04.014 INFO:teuthology.orchestra.run.smithi183.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-10T19:16:04.025 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:16:04.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:16:04.027 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:16:04.028 INFO:teuthology.orchestra.run.smithi183.stdout:osd.3: {} 2024-09-10T19:16:04.028 INFO:teuthology.orchestra.run.smithi183.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-10T19:16:04.044 INFO:teuthology.orchestra.run.smithi183.stdout:osd.4: {} 2024-09-10T19:16:04.044 INFO:teuthology.orchestra.run.smithi183.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-10T19:16:04.060 INFO:teuthology.orchestra.run.smithi183.stdout:osd.5: {} 2024-09-10T19:16:04.060 INFO:teuthology.orchestra.run.smithi183.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-10T19:16:04.076 INFO:teuthology.orchestra.run.smithi183.stdout:osd.6: {} 2024-09-10T19:16:04.076 INFO:teuthology.orchestra.run.smithi183.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-10T19:16:04.092 INFO:teuthology.orchestra.run.smithi183.stdout:osd.7: {} 2024-09-10T19:16:04.092 INFO:teuthology.orchestra.run.smithi183.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-10T19:16:04.128 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:16:04.152 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:16:04.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:16:04.154 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:16:04.254 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:16:04.276 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:16:04.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:16:04.277 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:16:04.378 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:16:04.403 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:16:04.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:16:04.404 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:16:04.455 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:16:04 smithi183 ceph-mon[29736]: from='client.? 172.21.15.183:0/3532561785' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T19:16:04.456 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:16:04 smithi183 ceph-mon[41562]: from='client.? 172.21.15.183:0/3532561785' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T19:16:04.465 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:16:04 smithi191 ceph-mon[32884]: from='client.? 172.21.15.183:0/3532561785' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T19:16:04.505 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:16:04.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:16:04.527 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:16:04.527 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:16:04.628 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:16:04.650 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:16:04.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:16:04.651 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:16:04.751 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:16:04.773 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:16:04.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:16:04.774 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:16:04.875 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:16:04.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:16:04.898 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:16:04.898 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:16:04.999 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:16:05.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:16:05.022 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:16:05.022 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:16:05.123 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:16:05.144 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:16:05.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:16:05.145 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:16:05.213 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:16:05.246 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:16:05.272 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:16:05.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:16:05.273 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:16:05.374 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:16:05.399 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:16:05.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:16:05.400 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:16:05.465 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:16:05 smithi191 ceph-mon[32884]: pgmap v237: 132 pgs: 132 active+clean; 582 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 543 B/s rd, 0 op/s 2024-09-10T19:16:05.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:16:05 smithi191 ceph-mon[32884]: from='client.24893 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-10T19:16:05.501 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:16:05.527 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:16:05.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:16:05.528 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:16:05.629 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:16:05.651 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:16:05.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:16:05.652 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:16:05.681 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:16:05 smithi183 ceph-mon[41562]: pgmap v237: 132 pgs: 132 active+clean; 582 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 543 B/s rd, 0 op/s 2024-09-10T19:16:05.681 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:16:05 smithi183 ceph-mon[41562]: from='client.24893 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-10T19:16:05.681 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:16:05 smithi183 ceph-mon[29736]: pgmap v237: 132 pgs: 132 active+clean; 582 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 543 B/s rd, 0 op/s 2024-09-10T19:16:05.682 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:16:05 smithi183 ceph-mon[29736]: from='client.24893 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-10T19:16:05.738 INFO:teuthology.orchestra.run.smithi183.stdout:{"pg_ready":true,"pg_map":{"version":237,"stamp":"2024-09-10T19:16:03.716252+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596188,"num_objects":196,"num_object_clones":0,"num_object_copies":588,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":196,"num_whiteouts":0,"num_read":1043,"num_read_kb":900,"num_write":543,"num_write_kb":1671,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":132,"num_bytes_recovered":593425,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":609,"ondisk_log_size":609,"up":396,"acting":396,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":444,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":656216064,"kb_used":2134928,"kb_used_data":7676,"kb_used_omap":11,"kb_used_meta":2127156,"kb_avail":654081136,"statfs":{"total":671965249536,"available":669779083264,"internally_reserved":0,"allocated":7860224,"data_stored":5450789,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":11928,"internal_metadata":2178208104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":-280758,"num_write_kb":-8984193,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-33710,"num_bytes_recovered":-2208956440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.533072"},"pg_stats":[{"pgid":"6.1b","version":"0'0","reported_seq":110,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-09-10T19:15:56.184875+0000","last_change":"2024-09-10T19:13:08.710758+0000","last_active":"2024-09-10T19:15:56.184875+0000","last_peered":"2024-09-10T19:15:56.184875+0000","last_clean":"2024-09-10T19:15:56.184875+0000","last_became_active":"2024-09-10T19:10:14.056187+0000","last_became_peered":"2024-09-10T19:10:14.056187+0000","last_unstale":"2024-09-10T19:15:56.184875+0000","last_undegraded":"2024-09-10T19:15:56.184875+0000","last_fullsized":"2024-09-10T19:15:56.184875+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:13:08.710679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:13:08.710679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0027252499999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":126,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-09-10T19:15:56.318347+0000","last_change":"2024-09-10T19:13:16.219506+0000","last_active":"2024-09-10T19:15:56.318347+0000","last_peered":"2024-09-10T19:15:56.318347+0000","last_clean":"2024-09-10T19:15:56.318347+0000","last_became_active":"2024-09-10T19:11:44.748696+0000","last_became_peered":"2024-09-10T19:11:44.748696+0000","last_unstale":"2024-09-10T19:15:56.318347+0000","last_undegraded":"2024-09-10T19:15:56.318347+0000","last_fullsized":"2024-09-10T19:15:56.318347+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:13:16.219364+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:13:16.219364+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.023972094999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.19","version":"66'15","reported_seq":139,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-09-10T19:15:56.184849+0000","last_change":"2024-09-10T19:13:47.165943+0000","last_active":"2024-09-10T19:15:56.184849+0000","last_peered":"2024-09-10T19:15:56.184849+0000","last_clean":"2024-09-10T19:15:56.184849+0000","last_became_active":"2024-09-10T19:10:09.688432+0000","last_became_peered":"2024-09-10T19:10:09.688432+0000","last_unstale":"2024-09-10T19:15:56.184849+0000","last_undegraded":"2024-09-10T19:15:56.184849+0000","last_fullsized":"2024-09-10T19:15:56.184849+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'15","last_scrub_stamp":"2024-09-10T19:13:47.165848+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:13:47.165848+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.004982841,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":131,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-09-10T19:15:56.186453+0000","last_change":"2024-09-10T19:15:33.976761+0000","last_active":"2024-09-10T19:15:56.186453+0000","last_peered":"2024-09-10T19:15:56.186453+0000","last_clean":"2024-09-10T19:15:56.186453+0000","last_became_active":"2024-09-10T19:13:54.714824+0000","last_became_peered":"2024-09-10T19:13:54.714824+0000","last_unstale":"2024-09-10T19:15:56.186453+0000","last_undegraded":"2024-09-10T19:15:56.186453+0000","last_fullsized":"2024-09-10T19:15:56.186453+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:15:33.976696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:15:33.976696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:16:59.329419+0000","scrub_duration":0.0015310580000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"59'1","reported_seq":124,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-09-10T19:15:56.185687+0000","last_change":"2024-09-10T19:12:47.704757+0000","last_active":"2024-09-10T19:15:56.185687+0000","last_peered":"2024-09-10T19:15:56.185687+0000","last_clean":"2024-09-10T19:15:56.185687+0000","last_became_active":"2024-09-10T19:10:07.686143+0000","last_became_peered":"2024-09-10T19:10:07.686143+0000","last_unstale":"2024-09-10T19:15:56.185687+0000","last_undegraded":"2024-09-10T19:15:56.185687+0000","last_fullsized":"2024-09-10T19:15:56.185687+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'1","last_scrub_stamp":"2024-09-10T19:12:47.704508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:12:47.704508+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.010519827000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":874,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"66'9","reported_seq":156,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-09-10T19:15:56.186573+0000","last_change":"2024-09-10T19:14:55.117473+0000","last_active":"2024-09-10T19:15:56.186573+0000","last_peered":"2024-09-10T19:15:56.186573+0000","last_clean":"2024-09-10T19:15:56.186573+0000","last_became_active":"2024-09-10T19:13:54.716117+0000","last_became_peered":"2024-09-10T19:13:54.716117+0000","last_unstale":"2024-09-10T19:15:56.186573+0000","last_undegraded":"2024-09-10T19:15:56.186573+0000","last_fullsized":"2024-09-10T19:15:56.186573+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'9","last_scrub_stamp":"2024-09-10T19:14:55.117387+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:14:55.117387+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0074980940000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,3],"acting":[4,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":66,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-09-10T19:15:56.258903+0000","last_change":"2024-09-10T19:13:54.714747+0000","last_active":"2024-09-10T19:15:56.258903+0000","last_peered":"2024-09-10T19:15:56.258903+0000","last_clean":"2024-09-10T19:15:56.258903+0000","last_became_active":"2024-09-10T19:13:54.714365+0000","last_became_peered":"2024-09-10T19:13:54.714365+0000","last_unstale":"2024-09-10T19:15:56.258903+0000","last_undegraded":"2024-09-10T19:15:56.258903+0000","last_fullsized":"2024-09-10T19:15:56.258903+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:11:56.186365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:11:56.186365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":129,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-09-10T19:15:56.186600+0000","last_change":"2024-09-10T19:15:39.910169+0000","last_active":"2024-09-10T19:15:56.186600+0000","last_peered":"2024-09-10T19:15:56.186600+0000","last_clean":"2024-09-10T19:15:56.186600+0000","last_became_active":"2024-09-10T19:13:54.713538+0000","last_became_peered":"2024-09-10T19:13:54.713538+0000","last_unstale":"2024-09-10T19:15:56.186600+0000","last_undegraded":"2024-09-10T19:15:56.186600+0000","last_fullsized":"2024-09-10T19:15:56.186600+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:15:39.910118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:15:39.910118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:16:46.983414+0000","scrub_duration":0.0013551839999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,7],"acting":[4,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":116,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-09-10T19:15:56.277054+0000","last_change":"2024-09-10T19:12:49.527255+0000","last_active":"2024-09-10T19:15:56.277054+0000","last_peered":"2024-09-10T19:15:56.277054+0000","last_clean":"2024-09-10T19:15:56.277054+0000","last_became_active":"2024-09-10T19:10:07.690728+0000","last_became_peered":"2024-09-10T19:10:07.690728+0000","last_unstale":"2024-09-10T19:15:56.277054+0000","last_undegraded":"2024-09-10T19:15:56.277054+0000","last_fullsized":"2024-09-10T19:15:56.277054+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:12:49.527167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:12:49.527167+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0040597050000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1b","version":"66'5","reported_seq":139,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-09-10T19:15:56.186896+0000","last_change":"2024-09-10T19:12:55.359526+0000","last_active":"2024-09-10T19:15:56.186896+0000","last_peered":"2024-09-10T19:15:56.186896+0000","last_clean":"2024-09-10T19:15:56.186896+0000","last_became_active":"2024-09-10T19:11:44.748296+0000","last_became_peered":"2024-09-10T19:11:44.748296+0000","last_unstale":"2024-09-10T19:15:56.186896+0000","last_undegraded":"2024-09-10T19:15:56.186896+0000","last_fullsized":"2024-09-10T19:15:56.186896+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'5","last_scrub_stamp":"2024-09-10T19:12:55.359379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:12:55.359379+0000","objects_scrubbed":2,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0050448070000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":129,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-09-10T19:15:56.276265+0000","last_change":"2024-09-10T19:15:33.518718+0000","last_active":"2024-09-10T19:15:56.276265+0000","last_peered":"2024-09-10T19:15:56.276265+0000","last_clean":"2024-09-10T19:15:56.276265+0000","last_became_active":"2024-09-10T19:13:54.719063+0000","last_became_peered":"2024-09-10T19:13:54.719063+0000","last_unstale":"2024-09-10T19:15:56.276265+0000","last_undegraded":"2024-09-10T19:15:56.276265+0000","last_fullsized":"2024-09-10T19:15:56.276265+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:15:33.518601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:15:33.518601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:16:52.587150+0000","scrub_duration":0.0013951650000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":127,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-09-10T19:15:56.258975+0000","last_change":"2024-09-10T19:15:45.616238+0000","last_active":"2024-09-10T19:15:56.258975+0000","last_peered":"2024-09-10T19:15:56.258975+0000","last_clean":"2024-09-10T19:15:56.258975+0000","last_became_active":"2024-09-10T19:13:54.723590+0000","last_became_peered":"2024-09-10T19:13:54.723590+0000","last_unstale":"2024-09-10T19:15:56.258975+0000","last_undegraded":"2024-09-10T19:15:56.258975+0000","last_fullsized":"2024-09-10T19:15:56.258975+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:15:45.616184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:15:45.616184+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:17:14.482505+0000","scrub_duration":0.0013989510000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":116,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-09-10T19:15:56.276955+0000","last_change":"2024-09-10T19:12:54.477202+0000","last_active":"2024-09-10T19:15:56.276955+0000","last_peered":"2024-09-10T19:15:56.276955+0000","last_clean":"2024-09-10T19:15:56.276955+0000","last_became_active":"2024-09-10T19:10:07.690174+0000","last_became_peered":"2024-09-10T19:10:07.690174+0000","last_unstale":"2024-09-10T19:15:56.276955+0000","last_undegraded":"2024-09-10T19:15:56.276955+0000","last_fullsized":"2024-09-10T19:15:56.276955+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:12:54.477149+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:12:54.477149+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0039768319999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1a","version":"66'9","reported_seq":129,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-09-10T19:15:56.186751+0000","last_change":"2024-09-10T19:13:02.431528+0000","last_active":"2024-09-10T19:15:56.186751+0000","last_peered":"2024-09-10T19:15:56.186751+0000","last_clean":"2024-09-10T19:15:56.186751+0000","last_became_active":"2024-09-10T19:10:09.688987+0000","last_became_peered":"2024-09-10T19:10:09.688987+0000","last_unstale":"2024-09-10T19:15:56.186751+0000","last_undegraded":"2024-09-10T19:15:56.186751+0000","last_fullsized":"2024-09-10T19:15:56.186751+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'9","last_scrub_stamp":"2024-09-10T19:13:02.431443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:13:02.431443+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0084913290000000006,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":112,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-09-10T19:15:56.259275+0000","last_change":"2024-09-10T19:12:51.449179+0000","last_active":"2024-09-10T19:15:56.259275+0000","last_peered":"2024-09-10T19:15:56.259275+0000","last_clean":"2024-09-10T19:15:56.259275+0000","last_became_active":"2024-09-10T19:10:11.701799+0000","last_became_peered":"2024-09-10T19:10:11.701799+0000","last_unstale":"2024-09-10T19:15:56.259275+0000","last_undegraded":"2024-09-10T19:15:56.259275+0000","last_fullsized":"2024-09-10T19:15:56.259275+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:12:51.449046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:12:51.449046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.002348826,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":127,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-09-10T19:15:56.316877+0000","last_change":"2024-09-10T19:15:33.839329+0000","last_active":"2024-09-10T19:15:56.316877+0000","last_peered":"2024-09-10T19:15:56.316877+0000","last_clean":"2024-09-10T19:15:56.316877+0000","last_became_active":"2024-09-10T19:13:54.718860+0000","last_became_peered":"2024-09-10T19:13:54.718860+0000","last_unstale":"2024-09-10T19:15:56.316877+0000","last_undegraded":"2024-09-10T19:15:56.316877+0000","last_fullsized":"2024-09-10T19:15:56.316877+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:15:33.839248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:15:33.839248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:17:01.970863+0000","scrub_duration":0.0032019980000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.d","version":"66'17","reported_seq":176,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-09-10T19:15:56.187322+0000","last_change":"2024-09-10T19:13:54.747117+0000","last_active":"2024-09-10T19:15:56.187322+0000","last_peered":"2024-09-10T19:15:56.187322+0000","last_clean":"2024-09-10T19:15:56.187322+0000","last_became_active":"2024-09-10T19:13:54.713411+0000","last_became_peered":"2024-09-10T19:13:54.713411+0000","last_unstale":"2024-09-10T19:15:56.187322+0000","last_undegraded":"2024-09-10T19:15:56.187322+0000","last_fullsized":"2024-09-10T19:15:56.187322+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'17","last_scrub_stamp":"2024-09-10T19:12:45.367252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:12:45.367252+0000","objects_scrubbed":8,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0031240310000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":116,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-09-10T19:15:56.276589+0000","last_change":"2024-09-10T19:12:46.519868+0000","last_active":"2024-09-10T19:15:56.276589+0000","last_peered":"2024-09-10T19:15:56.276589+0000","last_clean":"2024-09-10T19:15:56.276589+0000","last_became_active":"2024-09-10T19:10:07.691119+0000","last_became_peered":"2024-09-10T19:10:07.691119+0000","last_unstale":"2024-09-10T19:15:56.276589+0000","last_undegraded":"2024-09-10T19:15:56.276589+0000","last_fullsized":"2024-09-10T19:15:56.276589+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:12:46.519818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:11:22.280002+0000","last_clean_scrub_stamp":"2024-09-10T19:12:46.519818+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001139279,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":69,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-09-10T19:15:56.187040+0000","last_change":"2024-09-10T19:14:51.085977+0000","last_active":"2024-09-10T19:15:56.187040+0000","last_peered":"2024-09-10T19:15:56.187040+0000","last_clean":"2024-09-10T19:15:56.187040+0000","last_became_active":"2024-09-10T19:13:54.712883+0000","last_became_peered":"2024-09-10T19:13:54.712883+0000","last_unstale":"2024-09-10T19:15:56.187040+0000","last_undegraded":"2024-09-10T19:15:56.187040+0000","last_fullsized":"2024-09-10T19:15:56.187040+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:12:00.240841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:12:00.240841+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":120,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-09-10T19:15:56.317661+0000","last_change":"2024-09-10T19:15:47.818436+0000","last_active":"2024-09-10T19:15:56.317661+0000","last_peered":"2024-09-10T19:15:56.317661+0000","last_clean":"2024-09-10T19:15:56.317661+0000","last_became_active":"2024-09-10T19:10:13.722894+0000","last_became_peered":"2024-09-10T19:10:13.722894+0000","last_unstale":"2024-09-10T19:15:56.317661+0000","last_undegraded":"2024-09-10T19:15:56.317661+0000","last_fullsized":"2024-09-10T19:15:56.317661+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:15:47.818381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:15:47.818381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:16:50.810900+0000","scrub_duration":0.0013815990000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.c","version":"66'10","reported_seq":138,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-09-10T19:15:56.187162+0000","last_change":"2024-09-10T19:13:51.416075+0000","last_active":"2024-09-10T19:15:56.187162+0000","last_peered":"2024-09-10T19:15:56.187162+0000","last_clean":"2024-09-10T19:15:56.187162+0000","last_became_active":"2024-09-10T19:10:09.688321+0000","last_became_peered":"2024-09-10T19:10:09.688321+0000","last_unstale":"2024-09-10T19:15:56.187162+0000","last_undegraded":"2024-09-10T19:15:56.187162+0000","last_fullsized":"2024-09-10T19:15:56.187162+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'10","last_scrub_stamp":"2024-09-10T19:13:51.415998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:13:51.415998+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.005972799,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":76,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-09-10T19:15:56.185366+0000","last_change":"2024-09-10T19:15:42.119912+0000","last_active":"2024-09-10T19:15:56.185366+0000","last_peered":"2024-09-10T19:15:56.185366+0000","last_clean":"2024-09-10T19:15:56.185366+0000","last_became_active":"2024-09-10T19:13:54.731134+0000","last_became_peered":"2024-09-10T19:13:54.731134+0000","last_unstale":"2024-09-10T19:15:56.185366+0000","last_undegraded":"2024-09-10T19:15:56.185366+0000","last_fullsized":"2024-09-10T19:15:56.185366+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:15:42.119590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:15:42.119590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:16:46.442054+0000","scrub_duration":0.018001347000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,5],"acting":[3,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"80'9","reported_seq":194,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-09-10T19:15:56.317910+0000","last_change":"2024-09-10T19:12:51.343854+0000","last_active":"2024-09-10T19:15:56.317910+0000","last_peered":"2024-09-10T19:15:56.317910+0000","last_clean":"2024-09-10T19:15:56.317910+0000","last_became_active":"2024-09-10T19:10:11.698547+0000","last_became_peered":"2024-09-10T19:10:11.698547+0000","last_unstale":"2024-09-10T19:15:56.317910+0000","last_undegraded":"2024-09-10T19:15:56.317910+0000","last_fullsized":"2024-09-10T19:15:56.317910+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"80'9","last_scrub_stamp":"2024-09-10T19:12:51.343776+0000","last_deep_scrub":"80'9","last_deep_scrub_stamp":"2024-09-10T19:11:25.156465+0000","last_clean_scrub_stamp":"2024-09-10T19:12:51.343776+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0026375819999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":132,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-09-10T19:15:56.187298+0000","last_change":"2024-09-10T19:15:29.989058+0000","last_active":"2024-09-10T19:15:56.187298+0000","last_peered":"2024-09-10T19:15:56.187298+0000","last_clean":"2024-09-10T19:15:56.187298+0000","last_became_active":"2024-09-10T19:13:54.719498+0000","last_became_peered":"2024-09-10T19:13:54.719498+0000","last_unstale":"2024-09-10T19:15:56.187298+0000","last_undegraded":"2024-09-10T19:15:56.187298+0000","last_fullsized":"2024-09-10T19:15:56.187298+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:15:29.988976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:15:29.988976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:16:41.546952+0000","scrub_duration":0.0053787150000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.f","version":"66'15","reported_seq":119,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-09-10T19:15:56.185784+0000","last_change":"2024-09-10T19:13:54.742958+0000","last_active":"2024-09-10T19:15:56.185784+0000","last_peered":"2024-09-10T19:15:56.185784+0000","last_clean":"2024-09-10T19:15:56.185784+0000","last_became_active":"2024-09-10T19:13:54.711634+0000","last_became_peered":"2024-09-10T19:13:54.711634+0000","last_unstale":"2024-09-10T19:15:56.185784+0000","last_undegraded":"2024-09-10T19:15:56.185784+0000","last_fullsized":"2024-09-10T19:15:56.185784+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'15","last_scrub_stamp":"2024-09-10T19:11:59.232145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:11:59.232145+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":76,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-09-10T19:15:56.276644+0000","last_change":"2024-09-10T19:15:29.553530+0000","last_active":"2024-09-10T19:15:56.276644+0000","last_peered":"2024-09-10T19:15:56.276644+0000","last_clean":"2024-09-10T19:15:56.276644+0000","last_became_active":"2024-09-10T19:13:54.718452+0000","last_became_peered":"2024-09-10T19:13:54.718452+0000","last_unstale":"2024-09-10T19:15:56.276644+0000","last_undegraded":"2024-09-10T19:15:56.276644+0000","last_fullsized":"2024-09-10T19:15:56.276644+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:15:29.553448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:15:29.553448+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:16:49.232851+0000","scrub_duration":0.0062080679999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.e","version":"66'8","reported_seq":189,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-09-10T19:15:56.187399+0000","last_change":"2024-09-10T19:13:40.475460+0000","last_active":"2024-09-10T19:15:56.187399+0000","last_peered":"2024-09-10T19:15:56.187399+0000","last_clean":"2024-09-10T19:15:56.187399+0000","last_became_active":"2024-09-10T19:10:11.699232+0000","last_became_peered":"2024-09-10T19:10:11.699232+0000","last_unstale":"2024-09-10T19:15:56.187399+0000","last_undegraded":"2024-09-10T19:15:56.187399+0000","last_fullsized":"2024-09-10T19:15:56.187399+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'8","last_scrub_stamp":"2024-09-10T19:13:40.475375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:13:40.475375+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.035806152000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":127,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-09-10T19:15:56.259273+0000","last_change":"2024-09-10T19:15:44.633122+0000","last_active":"2024-09-10T19:15:56.259273+0000","last_peered":"2024-09-10T19:15:56.259273+0000","last_clean":"2024-09-10T19:15:56.259273+0000","last_became_active":"2024-09-10T19:13:54.722688+0000","last_became_peered":"2024-09-10T19:13:54.722688+0000","last_unstale":"2024-09-10T19:15:56.259273+0000","last_undegraded":"2024-09-10T19:15:56.259273+0000","last_fullsized":"2024-09-10T19:15:56.259273+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:15:44.633043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:15:44.633043+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:17:13.623295+0000","scrub_duration":0.00368805,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.e","version":"84'16","reported_seq":169,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-09-10T19:15:56.187458+0000","last_change":"2024-09-10T19:15:47.914487+0000","last_active":"2024-09-10T19:15:56.187458+0000","last_peered":"2024-09-10T19:15:56.187458+0000","last_clean":"2024-09-10T19:15:56.187458+0000","last_became_active":"2024-09-10T19:13:54.717022+0000","last_became_peered":"2024-09-10T19:13:54.717022+0000","last_unstale":"2024-09-10T19:15:56.187458+0000","last_undegraded":"2024-09-10T19:15:56.187458+0000","last_fullsized":"2024-09-10T19:15:56.187458+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'16","last_scrub_stamp":"2024-09-10T19:15:47.914420+0000","last_deep_scrub":"84'16","last_deep_scrub_stamp":"2024-09-10T19:11:56.157975+0000","last_clean_scrub_stamp":"2024-09-10T19:15:47.914420+0000","objects_scrubbed":5,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:16:48.311799+0000","scrub_duration":0.0019153390000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,3],"acting":[4,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":133,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-09-10T19:15:56.276109+0000","last_change":"2024-09-10T19:15:42.450149+0000","last_active":"2024-09-10T19:15:56.276109+0000","last_peered":"2024-09-10T19:15:56.276109+0000","last_clean":"2024-09-10T19:15:56.276109+0000","last_became_active":"2024-09-10T19:13:54.720293+0000","last_became_peered":"2024-09-10T19:13:54.720293+0000","last_unstale":"2024-09-10T19:15:56.276109+0000","last_undegraded":"2024-09-10T19:15:56.276109+0000","last_fullsized":"2024-09-10T19:15:56.276109+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:15:42.450068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:11:53.066849+0000","last_clean_scrub_stamp":"2024-09-10T19:15:42.450068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:16:57.302037+0000","scrub_duration":0.0037708360000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":112,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-09-10T19:15:56.259603+0000","last_change":"2024-09-10T19:13:06.358188+0000","last_active":"2024-09-10T19:15:56.259603+0000","last_peered":"2024-09-10T19:15:56.259603+0000","last_clean":"2024-09-10T19:15:56.259603+0000","last_became_active":"2024-09-10T19:10:11.692215+0000","last_became_peered":"2024-09-10T19:10:11.692215+0000","last_unstale":"2024-09-10T19:15:56.259603+0000","last_undegraded":"2024-09-10T19:15:56.259603+0000","last_fullsized":"2024-09-10T19:15:56.259603+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:13:06.358109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:13:06.358109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0026724549999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":110,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-09-10T19:15:56.184587+0000","last_change":"2024-09-10T19:13:54.304493+0000","last_active":"2024-09-10T19:15:56.184587+0000","last_peered":"2024-09-10T19:15:56.184587+0000","last_clean":"2024-09-10T19:15:56.184587+0000","last_became_active":"2024-09-10T19:10:14.053736+0000","last_became_peered":"2024-09-10T19:10:14.053736+0000","last_unstale":"2024-09-10T19:15:56.184587+0000","last_undegraded":"2024-09-10T19:15:56.184587+0000","last_fullsized":"2024-09-10T19:15:56.184587+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:13:54.304435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:13:54.304435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0011043820000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"66'14","reported_seq":55,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-09-10T19:15:56.317108+0000","last_change":"2024-09-10T19:15:41.748183+0000","last_active":"2024-09-10T19:15:56.317108+0000","last_peered":"2024-09-10T19:15:56.317108+0000","last_clean":"2024-09-10T19:15:56.317108+0000","last_became_active":"2024-09-10T19:15:00.742474+0000","last_became_peered":"2024-09-10T19:15:00.742474+0000","last_unstale":"2024-09-10T19:15:56.317108+0000","last_undegraded":"2024-09-10T19:15:56.317108+0000","last_fullsized":"2024-09-10T19:15:56.317108+0000","mapping_epoch":133,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":134,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'14","last_scrub_stamp":"2024-09-10T19:15:41.748101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:15:41.748101+0000","objects_scrubbed":7,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:16:54.266006+0000","scrub_duration":0.0055403099999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":116,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-09-10T19:15:56.318259+0000","last_change":"2024-09-10T19:12:52.885643+0000","last_active":"2024-09-10T19:15:56.318259+0000","last_peered":"2024-09-10T19:15:56.318259+0000","last_clean":"2024-09-10T19:15:56.318259+0000","last_became_active":"2024-09-10T19:10:07.684864+0000","last_became_peered":"2024-09-10T19:10:07.684864+0000","last_unstale":"2024-09-10T19:15:56.318259+0000","last_undegraded":"2024-09-10T19:15:56.318259+0000","last_fullsized":"2024-09-10T19:15:56.318259+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:12:52.885563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:12:52.885563+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0035531249999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.0","version":"117'10","reported_seq":212,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-09-10T19:15:56.184435+0000","last_change":"2024-09-10T19:15:35.973188+0000","last_active":"2024-09-10T19:15:56.184435+0000","last_peered":"2024-09-10T19:15:56.184435+0000","last_clean":"2024-09-10T19:15:56.184435+0000","last_became_active":"2024-09-10T19:13:54.709846+0000","last_became_peered":"2024-09-10T19:13:54.709846+0000","last_unstale":"2024-09-10T19:15:56.184435+0000","last_undegraded":"2024-09-10T19:15:56.184435+0000","last_fullsized":"2024-09-10T19:15:56.184435+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"117'10","last_scrub_stamp":"2024-09-10T19:15:35.973095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:15:35.973095+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:16:55.086873+0000","scrub_duration":0.0076338860000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,0],"acting":[3,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":110,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-09-10T19:15:56.276294+0000","last_change":"2024-09-10T19:13:15.350815+0000","last_active":"2024-09-10T19:15:56.276294+0000","last_peered":"2024-09-10T19:15:56.276294+0000","last_clean":"2024-09-10T19:15:56.276294+0000","last_became_active":"2024-09-10T19:10:14.054456+0000","last_became_peered":"2024-09-10T19:10:14.054456+0000","last_unstale":"2024-09-10T19:15:56.276294+0000","last_undegraded":"2024-09-10T19:15:56.276294+0000","last_fullsized":"2024-09-10T19:15:56.276294+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:13:15.350720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:11:41.073833+0000","last_clean_scrub_stamp":"2024-09-10T19:13:15.350720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0014361000000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.0","version":"66'18","reported_seq":146,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-09-10T19:15:56.184556+0000","last_change":"2024-09-10T19:12:49.699111+0000","last_active":"2024-09-10T19:15:56.184556+0000","last_peered":"2024-09-10T19:15:56.184556+0000","last_clean":"2024-09-10T19:15:56.184556+0000","last_became_active":"2024-09-10T19:10:09.821836+0000","last_became_peered":"2024-09-10T19:10:09.821836+0000","last_unstale":"2024-09-10T19:15:56.184556+0000","last_undegraded":"2024-09-10T19:15:56.184556+0000","last_fullsized":"2024-09-10T19:15:56.184556+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'18","last_scrub_stamp":"2024-09-10T19:12:49.699054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:12:49.699054+0000","objects_scrubbed":8,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0073070890000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":76,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-09-10T19:15:56.317998+0000","last_change":"2024-09-10T19:15:42.739879+0000","last_active":"2024-09-10T19:15:56.317998+0000","last_peered":"2024-09-10T19:15:56.317998+0000","last_clean":"2024-09-10T19:15:56.317998+0000","last_became_active":"2024-09-10T19:13:54.710968+0000","last_became_peered":"2024-09-10T19:13:54.710968+0000","last_unstale":"2024-09-10T19:15:56.317998+0000","last_undegraded":"2024-09-10T19:15:56.317998+0000","last_fullsized":"2024-09-10T19:15:56.317998+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:15:42.739801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:15:42.739801+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:17:05.377168+0000","scrub_duration":0.008487784,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,3],"acting":[6,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":111,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-09-10T19:15:56.186654+0000","last_change":"2024-09-10T19:12:58.400261+0000","last_active":"2024-09-10T19:15:56.186654+0000","last_peered":"2024-09-10T19:15:56.186654+0000","last_clean":"2024-09-10T19:15:56.186654+0000","last_became_active":"2024-09-10T19:10:11.698054+0000","last_became_peered":"2024-09-10T19:10:11.698054+0000","last_unstale":"2024-09-10T19:15:56.186654+0000","last_undegraded":"2024-09-10T19:15:56.186654+0000","last_fullsized":"2024-09-10T19:15:56.186654+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:12:58.400206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:12:58.400206+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0071235179999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":109,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-09-10T19:15:56.186711+0000","last_change":"2024-09-10T19:12:57.409831+0000","last_active":"2024-09-10T19:15:56.186711+0000","last_peered":"2024-09-10T19:15:56.186711+0000","last_clean":"2024-09-10T19:15:56.186711+0000","last_became_active":"2024-09-10T19:10:13.722784+0000","last_became_peered":"2024-09-10T19:10:13.722784+0000","last_unstale":"2024-09-10T19:15:56.186711+0000","last_undegraded":"2024-09-10T19:15:56.186711+0000","last_fullsized":"2024-09-10T19:15:56.186711+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:12:57.409749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:12:57.409749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0087033260000000008,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.3","version":"104'27","reported_seq":158,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-09-10T19:15:56.317610+0000","last_change":"2024-09-10T19:13:02.226033+0000","last_active":"2024-09-10T19:15:56.317610+0000","last_peered":"2024-09-10T19:15:56.317610+0000","last_clean":"2024-09-10T19:15:56.317610+0000","last_became_active":"2024-09-10T19:10:09.823003+0000","last_became_peered":"2024-09-10T19:10:09.823003+0000","last_unstale":"2024-09-10T19:15:56.317610+0000","last_undegraded":"2024-09-10T19:15:56.317610+0000","last_fullsized":"2024-09-10T19:15:56.317610+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"104'27","last_scrub_stamp":"2024-09-10T19:13:02.225944+0000","last_deep_scrub":"104'27","last_deep_scrub_stamp":"2024-09-10T19:13:02.225944+0000","last_clean_scrub_stamp":"2024-09-10T19:13:02.225944+0000","objects_scrubbed":8,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0099233020000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":116,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-09-10T19:15:56.277082+0000","last_change":"2024-09-10T19:12:53.513000+0000","last_active":"2024-09-10T19:15:56.277082+0000","last_peered":"2024-09-10T19:15:56.277082+0000","last_clean":"2024-09-10T19:15:56.277082+0000","last_became_active":"2024-09-10T19:10:07.690325+0000","last_became_peered":"2024-09-10T19:10:07.690325+0000","last_unstale":"2024-09-10T19:15:56.277082+0000","last_undegraded":"2024-09-10T19:15:56.277082+0000","last_fullsized":"2024-09-10T19:15:56.277082+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:12:53.512923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:12:53.512923+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0087393290000000005,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":122,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-09-10T19:15:56.317264+0000","last_change":"2024-09-10T19:15:50.669894+0000","last_active":"2024-09-10T19:15:56.317264+0000","last_peered":"2024-09-10T19:15:56.317264+0000","last_clean":"2024-09-10T19:15:56.317264+0000","last_became_active":"2024-09-10T19:10:11.700370+0000","last_became_peered":"2024-09-10T19:10:11.700370+0000","last_unstale":"2024-09-10T19:15:56.317264+0000","last_undegraded":"2024-09-10T19:15:56.317264+0000","last_fullsized":"2024-09-10T19:15:56.317264+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:15:50.669821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:15:50.669821+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:17:00.492456+0000","scrub_duration":0.0015765,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":76,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-09-10T19:15:56.317257+0000","last_change":"2024-09-10T19:15:41.749437+0000","last_active":"2024-09-10T19:15:56.317257+0000","last_peered":"2024-09-10T19:15:56.317257+0000","last_clean":"2024-09-10T19:15:56.317257+0000","last_became_active":"2024-09-10T19:13:54.717977+0000","last_became_peered":"2024-09-10T19:13:54.717977+0000","last_unstale":"2024-09-10T19:15:56.317257+0000","last_undegraded":"2024-09-10T19:15:56.317257+0000","last_fullsized":"2024-09-10T19:15:56.317257+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:15:41.749338+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:15:41.749338+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:16:43.216724+0000","scrub_duration":0.0018841260000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,7],"acting":[6,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.2","version":"66'10","reported_seq":82,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-09-10T19:15:56.259106+0000","last_change":"2024-09-10T19:13:54.748881+0000","last_active":"2024-09-10T19:15:56.259106+0000","last_peered":"2024-09-10T19:15:56.259106+0000","last_clean":"2024-09-10T19:15:56.259106+0000","last_became_active":"2024-09-10T19:13:54.714542+0000","last_became_peered":"2024-09-10T19:13:54.714542+0000","last_unstale":"2024-09-10T19:15:56.259106+0000","last_undegraded":"2024-09-10T19:15:56.259106+0000","last_fullsized":"2024-09-10T19:15:56.259106+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'10","last_scrub_stamp":"2024-09-10T19:12:56.360409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:12:56.360409+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":14,"num_read_kb":9,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,0],"acting":[5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":76,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-09-10T19:15:56.317896+0000","last_change":"2024-09-10T19:15:41.870492+0000","last_active":"2024-09-10T19:15:56.317896+0000","last_peered":"2024-09-10T19:15:56.317896+0000","last_clean":"2024-09-10T19:15:56.317896+0000","last_became_active":"2024-09-10T19:13:54.718267+0000","last_became_peered":"2024-09-10T19:13:54.718267+0000","last_unstale":"2024-09-10T19:15:56.317896+0000","last_undegraded":"2024-09-10T19:15:56.317896+0000","last_fullsized":"2024-09-10T19:15:56.317896+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:15:41.870422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:15:41.870422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:16:51.942406+0000","scrub_duration":0.0061607199999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3","version":"84'9","reported_seq":191,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-09-10T19:15:56.317382+0000","last_change":"2024-09-10T19:13:45.236207+0000","last_active":"2024-09-10T19:15:56.317382+0000","last_peered":"2024-09-10T19:15:56.317382+0000","last_clean":"2024-09-10T19:15:56.317382+0000","last_became_active":"2024-09-10T19:10:11.699690+0000","last_became_peered":"2024-09-10T19:10:11.699690+0000","last_unstale":"2024-09-10T19:15:56.317382+0000","last_undegraded":"2024-09-10T19:15:56.317382+0000","last_fullsized":"2024-09-10T19:15:56.317382+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'9","last_scrub_stamp":"2024-09-10T19:13:45.236094+0000","last_deep_scrub":"84'9","last_deep_scrub_stamp":"2024-09-10T19:11:53.879272+0000","last_clean_scrub_stamp":"2024-09-10T19:13:45.236094+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.036165602999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":110,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-09-10T19:15:56.316791+0000","last_change":"2024-09-10T19:13:40.234378+0000","last_active":"2024-09-10T19:15:56.316791+0000","last_peered":"2024-09-10T19:15:56.316791+0000","last_clean":"2024-09-10T19:15:56.316791+0000","last_became_active":"2024-09-10T19:10:13.720403+0000","last_became_peered":"2024-09-10T19:10:13.720403+0000","last_unstale":"2024-09-10T19:15:56.316791+0000","last_undegraded":"2024-09-10T19:15:56.316791+0000","last_fullsized":"2024-09-10T19:15:56.316791+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:13:40.234289+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:13:40.234289+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0086405590000000008,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"117'30","reported_seq":198,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-09-10T19:15:56.317765+0000","last_change":"2024-09-10T19:15:49.837906+0000","last_active":"2024-09-10T19:15:56.317765+0000","last_peered":"2024-09-10T19:15:56.317765+0000","last_clean":"2024-09-10T19:15:56.317765+0000","last_became_active":"2024-09-10T19:13:54.715642+0000","last_became_peered":"2024-09-10T19:13:54.715642+0000","last_unstale":"2024-09-10T19:15:56.317765+0000","last_undegraded":"2024-09-10T19:15:56.317765+0000","last_fullsized":"2024-09-10T19:15:56.317765+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"117'30","last_scrub_stamp":"2024-09-10T19:15:49.837844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:15:49.837844+0000","objects_scrubbed":10,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:17:01.943186+0000","scrub_duration":0.0024403329999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":47,"num_read_kb":32,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":358,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":133,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-09-10T19:15:56.259400+0000","last_change":"2024-09-10T19:15:54.746979+0000","last_active":"2024-09-10T19:15:56.259400+0000","last_peered":"2024-09-10T19:15:56.259400+0000","last_clean":"2024-09-10T19:15:56.259400+0000","last_became_active":"2024-09-10T19:13:54.730096+0000","last_became_peered":"2024-09-10T19:13:54.730096+0000","last_unstale":"2024-09-10T19:15:56.259400+0000","last_undegraded":"2024-09-10T19:15:56.259400+0000","last_fullsized":"2024-09-10T19:15:56.259400+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:15:54.746920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:15:54.746920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:17:00.357419+0000","scrub_duration":0.001300037,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":129,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-09-10T19:15:56.318083+0000","last_change":"2024-09-10T19:15:34.802331+0000","last_active":"2024-09-10T19:15:56.318083+0000","last_peered":"2024-09-10T19:15:56.318083+0000","last_clean":"2024-09-10T19:15:56.318083+0000","last_became_active":"2024-09-10T19:13:54.714017+0000","last_became_peered":"2024-09-10T19:13:54.714017+0000","last_unstale":"2024-09-10T19:15:56.318083+0000","last_undegraded":"2024-09-10T19:15:56.318083+0000","last_fullsized":"2024-09-10T19:15:56.318083+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:15:34.802121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:15:34.802121+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:17:01.582331+0000","scrub_duration":0.0042778080000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,6],"acting":[0,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":100,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-09-10T19:15:56.185490+0000","last_change":"2024-09-10T19:12:46.691448+0000","last_active":"2024-09-10T19:15:56.185490+0000","last_peered":"2024-09-10T19:15:56.185490+0000","last_clean":"2024-09-10T19:15:56.185490+0000","last_became_active":"2024-09-10T19:10:13.721744+0000","last_became_peered":"2024-09-10T19:10:13.721744+0000","last_unstale":"2024-09-10T19:15:56.185490+0000","last_undegraded":"2024-09-10T19:15:56.185490+0000","last_fullsized":"2024-09-10T19:15:56.185490+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:12:46.691353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:12:46.691353+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0014482360000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.7","version":"82'18","reported_seq":149,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-09-10T19:15:56.259728+0000","last_change":"2024-09-10T19:15:55.777428+0000","last_active":"2024-09-10T19:15:56.259728+0000","last_peered":"2024-09-10T19:15:56.259728+0000","last_clean":"2024-09-10T19:15:56.259728+0000","last_became_active":"2024-09-10T19:13:54.712065+0000","last_became_peered":"2024-09-10T19:13:54.712065+0000","last_unstale":"2024-09-10T19:15:56.259728+0000","last_undegraded":"2024-09-10T19:15:56.259728+0000","last_fullsized":"2024-09-10T19:15:56.259728+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'18","last_scrub_stamp":"2024-09-10T19:15:55.777361+0000","last_deep_scrub":"82'18","last_deep_scrub_stamp":"2024-09-10T19:15:55.777361+0000","last_clean_scrub_stamp":"2024-09-10T19:15:55.777361+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:17:17.118895+0000","scrub_duration":0.0023560040000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":19,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.0","version":"67'5","reported_seq":300,"reported_epoch":146,"state":"active+clean","last_fresh":"2024-09-10T19:15:58.277862+0000","last_change":"2024-09-10T19:15:30.760345+0000","last_active":"2024-09-10T19:15:58.277862+0000","last_peered":"2024-09-10T19:15:58.277862+0000","last_clean":"2024-09-10T19:15:58.277862+0000","last_became_active":"2024-09-10T19:13:54.715570+0000","last_became_peered":"2024-09-10T19:13:54.715570+0000","last_unstale":"2024-09-10T19:15:58.277862+0000","last_undegraded":"2024-09-10T19:15:58.277862+0000","last_fullsized":"2024-09-10T19:15:58.277862+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"67'5","last_scrub_stamp":"2024-09-10T19:15:30.759591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:15:30.759591+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:16:38.105109+0000","scrub_duration":0.010111248999999999,"objects_trimmed":0,"snaptrim_duration":0.00034454499999999997,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":195,"num_read_kb":190,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":389,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,0],"acting":[2,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":116,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-09-10T19:15:56.317406+0000","last_change":"2024-09-10T19:12:53.318459+0000","last_active":"2024-09-10T19:15:56.317406+0000","last_peered":"2024-09-10T19:15:56.317406+0000","last_clean":"2024-09-10T19:15:56.317406+0000","last_became_active":"2024-09-10T19:10:07.681456+0000","last_became_peered":"2024-09-10T19:10:07.681456+0000","last_unstale":"2024-09-10T19:15:56.317406+0000","last_undegraded":"2024-09-10T19:15:56.317406+0000","last_fullsized":"2024-09-10T19:15:56.317406+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:12:53.318408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:12:53.318408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.01029277,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":112,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-09-10T19:15:56.318084+0000","last_change":"2024-09-10T19:13:00.366695+0000","last_active":"2024-09-10T19:15:56.318084+0000","last_peered":"2024-09-10T19:15:56.318084+0000","last_clean":"2024-09-10T19:15:56.318084+0000","last_became_active":"2024-09-10T19:10:11.695205+0000","last_became_peered":"2024-09-10T19:10:11.695205+0000","last_unstale":"2024-09-10T19:15:56.318084+0000","last_undegraded":"2024-09-10T19:15:56.318084+0000","last_fullsized":"2024-09-10T19:15:56.318084+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:13:00.366610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:13:00.366610+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0023966700000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":113,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-09-10T19:15:56.276807+0000","last_change":"2024-09-10T19:15:38.493179+0000","last_active":"2024-09-10T19:15:56.276807+0000","last_peered":"2024-09-10T19:15:56.276807+0000","last_clean":"2024-09-10T19:15:56.276807+0000","last_became_active":"2024-09-10T19:10:14.053697+0000","last_became_peered":"2024-09-10T19:10:14.053697+0000","last_unstale":"2024-09-10T19:15:56.276807+0000","last_undegraded":"2024-09-10T19:15:56.276807+0000","last_fullsized":"2024-09-10T19:15:56.276807+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:15:38.493102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:15:38.493102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:16:48.733504+0000","scrub_duration":0.0015544280000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"66'12","reported_seq":163,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-09-10T19:15:56.318419+0000","last_change":"2024-09-10T19:15:40.903370+0000","last_active":"2024-09-10T19:15:56.318419+0000","last_peered":"2024-09-10T19:15:56.318419+0000","last_clean":"2024-09-10T19:15:56.318419+0000","last_became_active":"2024-09-10T19:13:54.719200+0000","last_became_peered":"2024-09-10T19:13:54.719200+0000","last_unstale":"2024-09-10T19:15:56.318419+0000","last_undegraded":"2024-09-10T19:15:56.318419+0000","last_fullsized":"2024-09-10T19:15:56.318419+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'12","last_scrub_stamp":"2024-09-10T19:15:40.903279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:15:40.903279+0000","objects_scrubbed":6,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:17:02.013122+0000","scrub_duration":0.0271238,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,7],"acting":[0,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"61'1","reported_seq":122,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-09-10T19:15:56.317083+0000","last_change":"2024-09-10T19:13:06.265721+0000","last_active":"2024-09-10T19:15:56.317083+0000","last_peered":"2024-09-10T19:15:56.317083+0000","last_clean":"2024-09-10T19:15:56.317083+0000","last_became_active":"2024-09-10T19:10:07.683259+0000","last_became_peered":"2024-09-10T19:10:07.683259+0000","last_unstale":"2024-09-10T19:15:56.317083+0000","last_undegraded":"2024-09-10T19:15:56.317083+0000","last_fullsized":"2024-09-10T19:15:56.317083+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'1","last_scrub_stamp":"2024-09-10T19:13:06.265646+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:13:06.265646+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.012205384,"objects_trimmed":0,"snaptrim_duration":0.0014341130000000001,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":133,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-09-10T19:15:56.276372+0000","last_change":"2024-09-10T19:15:30.542477+0000","last_active":"2024-09-10T19:15:56.276372+0000","last_peered":"2024-09-10T19:15:56.276372+0000","last_clean":"2024-09-10T19:15:56.276372+0000","last_became_active":"2024-09-10T19:13:54.721486+0000","last_became_peered":"2024-09-10T19:13:54.721486+0000","last_unstale":"2024-09-10T19:15:56.276372+0000","last_undegraded":"2024-09-10T19:15:56.276372+0000","last_fullsized":"2024-09-10T19:15:56.276372+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:15:30.542399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:15:30.542399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:17:00.089482+0000","scrub_duration":0.0025142609999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":129,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-09-10T19:15:56.259718+0000","last_change":"2024-09-10T19:13:54.712428+0000","last_active":"2024-09-10T19:15:56.259718+0000","last_peered":"2024-09-10T19:15:56.259718+0000","last_clean":"2024-09-10T19:15:56.259718+0000","last_became_active":"2024-09-10T19:13:54.711685+0000","last_became_peered":"2024-09-10T19:13:54.711685+0000","last_unstale":"2024-09-10T19:15:56.259718+0000","last_undegraded":"2024-09-10T19:15:56.259718+0000","last_fullsized":"2024-09-10T19:15:56.259718+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:12:44.527561+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:12:44.527561+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.022638367999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,4],"acting":[5,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":66,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-09-10T19:15:56.259201+0000","last_change":"2024-09-10T19:13:54.723228+0000","last_active":"2024-09-10T19:15:56.259201+0000","last_peered":"2024-09-10T19:15:56.259201+0000","last_clean":"2024-09-10T19:15:56.259201+0000","last_became_active":"2024-09-10T19:13:54.722937+0000","last_became_peered":"2024-09-10T19:13:54.722937+0000","last_unstale":"2024-09-10T19:15:56.259201+0000","last_undegraded":"2024-09-10T19:15:56.259201+0000","last_fullsized":"2024-09-10T19:15:56.259201+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:12:05.165500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:12:05.165500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.5","version":"80'21","reported_seq":153,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-09-10T19:15:56.318160+0000","last_change":"2024-09-10T19:15:49.682164+0000","last_active":"2024-09-10T19:15:56.318160+0000","last_peered":"2024-09-10T19:15:56.318160+0000","last_clean":"2024-09-10T19:15:56.318160+0000","last_became_active":"2024-09-10T19:10:09.822184+0000","last_became_peered":"2024-09-10T19:10:09.822184+0000","last_unstale":"2024-09-10T19:15:56.318160+0000","last_undegraded":"2024-09-10T19:15:56.318160+0000","last_fullsized":"2024-09-10T19:15:56.318160+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"80'21","last_scrub_stamp":"2024-09-10T19:15:49.682092+0000","last_deep_scrub":"80'21","last_deep_scrub_stamp":"2024-09-10T19:11:39.836144+0000","last_clean_scrub_stamp":"2024-09-10T19:15:49.682092+0000","objects_scrubbed":7,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:17:05.534997+0000","scrub_duration":0.0022142529999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"63'2","reported_seq":123,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-09-10T19:15:56.185392+0000","last_change":"2024-09-10T19:13:00.752806+0000","last_active":"2024-09-10T19:15:56.185392+0000","last_peered":"2024-09-10T19:15:56.185392+0000","last_clean":"2024-09-10T19:15:56.185392+0000","last_became_active":"2024-09-10T19:10:07.684784+0000","last_became_peered":"2024-09-10T19:10:07.684784+0000","last_unstale":"2024-09-10T19:15:56.185392+0000","last_undegraded":"2024-09-10T19:15:56.185392+0000","last_fullsized":"2024-09-10T19:15:56.185392+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'2","last_scrub_stamp":"2024-09-10T19:13:00.752715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:13:00.752715+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0083291979999999995,"objects_trimmed":0,"snaptrim_duration":0.00075522999999999996,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":126,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-09-10T19:15:56.259573+0000","last_change":"2024-09-10T19:13:50.411573+0000","last_active":"2024-09-10T19:15:56.259573+0000","last_peered":"2024-09-10T19:15:56.259573+0000","last_clean":"2024-09-10T19:15:56.259573+0000","last_became_active":"2024-09-10T19:10:07.687596+0000","last_became_peered":"2024-09-10T19:10:07.687596+0000","last_unstale":"2024-09-10T19:15:56.259573+0000","last_undegraded":"2024-09-10T19:15:56.259573+0000","last_fullsized":"2024-09-10T19:15:56.259573+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:13:50.411500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:13:50.411500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0032604869999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"74'79","reported_seq":161,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-09-10T19:15:56.276410+0000","last_change":"2024-09-10T19:13:46.419002+0000","last_active":"2024-09-10T19:15:56.276410+0000","last_peered":"2024-09-10T19:15:56.276410+0000","last_clean":"2024-09-10T19:15:56.276410+0000","last_became_active":"2024-09-10T19:09:56.634919+0000","last_became_peered":"2024-09-10T19:09:56.634919+0000","last_unstale":"2024-09-10T19:15:56.276410+0000","last_undegraded":"2024-09-10T19:15:56.276410+0000","last_fullsized":"2024-09-10T19:15:56.276410+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'79","last_scrub_stamp":"2024-09-10T19:13:46.418921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:08:00.699315+0000","last_clean_scrub_stamp":"2024-09-10T19:13:46.418921+0000","objects_scrubbed":2,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0029827999999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":112,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-09-10T19:15:56.276672+0000","last_change":"2024-09-10T19:13:47.453309+0000","last_active":"2024-09-10T19:15:56.276672+0000","last_peered":"2024-09-10T19:15:56.276672+0000","last_clean":"2024-09-10T19:15:56.276672+0000","last_became_active":"2024-09-10T19:10:11.694986+0000","last_became_peered":"2024-09-10T19:10:11.694986+0000","last_unstale":"2024-09-10T19:15:56.276672+0000","last_undegraded":"2024-09-10T19:15:56.276672+0000","last_fullsized":"2024-09-10T19:15:56.276672+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:13:47.453228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:13:47.453228+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.010282463,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":110,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-09-10T19:15:56.259832+0000","last_change":"2024-09-10T19:13:05.390398+0000","last_active":"2024-09-10T19:15:56.259832+0000","last_peered":"2024-09-10T19:15:56.259832+0000","last_clean":"2024-09-10T19:15:56.259832+0000","last_became_active":"2024-09-10T19:10:13.720647+0000","last_became_peered":"2024-09-10T19:10:13.720647+0000","last_unstale":"2024-09-10T19:15:56.259832+0000","last_undegraded":"2024-09-10T19:15:56.259832+0000","last_fullsized":"2024-09-10T19:15:56.259832+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:13:05.390317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:13:05.390317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0058030640000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.a","version":"66'19","reported_seq":182,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-09-10T19:15:56.317560+0000","last_change":"2024-09-10T19:15:38.722679+0000","last_active":"2024-09-10T19:15:56.317560+0000","last_peered":"2024-09-10T19:15:56.317560+0000","last_clean":"2024-09-10T19:15:56.317560+0000","last_became_active":"2024-09-10T19:13:54.731274+0000","last_became_peered":"2024-09-10T19:13:54.731274+0000","last_unstale":"2024-09-10T19:15:56.317560+0000","last_undegraded":"2024-09-10T19:15:56.317560+0000","last_fullsized":"2024-09-10T19:15:56.317560+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'19","last_scrub_stamp":"2024-09-10T19:15:38.722581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:15:38.722581+0000","objects_scrubbed":9,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:16:45.004511+0000","scrub_duration":0.0054904940000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,5],"acting":[6,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":126,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-09-10T19:15:56.318070+0000","last_change":"2024-09-10T19:15:53.803110+0000","last_active":"2024-09-10T19:15:56.318070+0000","last_peered":"2024-09-10T19:15:56.318070+0000","last_clean":"2024-09-10T19:15:56.318070+0000","last_became_active":"2024-09-10T19:10:07.686236+0000","last_became_peered":"2024-09-10T19:10:07.686236+0000","last_unstale":"2024-09-10T19:15:56.318070+0000","last_undegraded":"2024-09-10T19:15:56.318070+0000","last_fullsized":"2024-09-10T19:15:56.318070+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:15:53.803057+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:15:53.803057+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:17:01.962402+0000","scrub_duration":0.001605563,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":112,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-09-10T19:15:56.317468+0000","last_change":"2024-09-10T19:12:55.335929+0000","last_active":"2024-09-10T19:15:56.317468+0000","last_peered":"2024-09-10T19:15:56.317468+0000","last_clean":"2024-09-10T19:15:56.317468+0000","last_became_active":"2024-09-10T19:10:11.699745+0000","last_became_peered":"2024-09-10T19:10:11.699745+0000","last_unstale":"2024-09-10T19:15:56.317468+0000","last_undegraded":"2024-09-10T19:15:56.317468+0000","last_fullsized":"2024-09-10T19:15:56.317468+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:12:55.335834+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:12:55.335834+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0060173240000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":110,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-09-10T19:15:56.276294+0000","last_change":"2024-09-10T19:13:14.311675+0000","last_active":"2024-09-10T19:15:56.276294+0000","last_peered":"2024-09-10T19:15:56.276294+0000","last_clean":"2024-09-10T19:15:56.276294+0000","last_became_active":"2024-09-10T19:10:13.719973+0000","last_became_peered":"2024-09-10T19:10:13.719973+0000","last_unstale":"2024-09-10T19:15:56.276294+0000","last_undegraded":"2024-09-10T19:15:56.276294+0000","last_fullsized":"2024-09-10T19:15:56.276294+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:13:14.311620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:13:14.311620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0054101059999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"66'9","reported_seq":157,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-09-10T19:15:56.317599+0000","last_change":"2024-09-10T19:15:37.846895+0000","last_active":"2024-09-10T19:15:56.317599+0000","last_peered":"2024-09-10T19:15:56.317599+0000","last_clean":"2024-09-10T19:15:56.317599+0000","last_became_active":"2024-09-10T19:13:54.710613+0000","last_became_peered":"2024-09-10T19:13:54.710613+0000","last_unstale":"2024-09-10T19:15:56.317599+0000","last_undegraded":"2024-09-10T19:15:56.317599+0000","last_fullsized":"2024-09-10T19:15:56.317599+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'9","last_scrub_stamp":"2024-09-10T19:15:37.846807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:15:37.846807+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:16:45.860729+0000","scrub_duration":0.012189228,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":79,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-09-10T19:15:56.187453+0000","last_change":"2024-09-10T19:15:50.850641+0000","last_active":"2024-09-10T19:15:56.187453+0000","last_peered":"2024-09-10T19:15:56.187453+0000","last_clean":"2024-09-10T19:15:56.187453+0000","last_became_active":"2024-09-10T19:13:54.708351+0000","last_became_peered":"2024-09-10T19:13:54.708351+0000","last_unstale":"2024-09-10T19:15:56.187453+0000","last_undegraded":"2024-09-10T19:15:56.187453+0000","last_fullsized":"2024-09-10T19:15:56.187453+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:15:50.850576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:11:44.254513+0000","last_clean_scrub_stamp":"2024-09-10T19:15:50.850576+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:17:13.316946+0000","scrub_duration":0.001815419,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":112,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-09-10T19:15:56.317347+0000","last_change":"2024-09-10T19:13:08.398177+0000","last_active":"2024-09-10T19:15:56.317347+0000","last_peered":"2024-09-10T19:15:56.317347+0000","last_clean":"2024-09-10T19:15:56.317347+0000","last_became_active":"2024-09-10T19:10:11.694405+0000","last_became_peered":"2024-09-10T19:10:11.694405+0000","last_unstale":"2024-09-10T19:15:56.317347+0000","last_undegraded":"2024-09-10T19:15:56.317347+0000","last_fullsized":"2024-09-10T19:15:56.317347+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:13:08.398036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:13:08.398036+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.016845473999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":110,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-09-10T19:15:56.317011+0000","last_change":"2024-09-10T19:12:56.145358+0000","last_active":"2024-09-10T19:15:56.317011+0000","last_peered":"2024-09-10T19:15:56.317011+0000","last_clean":"2024-09-10T19:15:56.317011+0000","last_became_active":"2024-09-10T19:10:13.714952+0000","last_became_peered":"2024-09-10T19:10:13.714952+0000","last_unstale":"2024-09-10T19:15:56.317011+0000","last_undegraded":"2024-09-10T19:15:56.317011+0000","last_fullsized":"2024-09-10T19:15:56.317011+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:12:56.145218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:12:56.145218+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0058765429999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.8","version":"66'15","reported_seq":139,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-09-10T19:15:56.258748+0000","last_change":"2024-09-10T19:12:50.449684+0000","last_active":"2024-09-10T19:15:56.258748+0000","last_peered":"2024-09-10T19:15:56.258748+0000","last_clean":"2024-09-10T19:15:56.258748+0000","last_became_active":"2024-09-10T19:10:09.820950+0000","last_became_peered":"2024-09-10T19:10:09.820950+0000","last_unstale":"2024-09-10T19:15:56.258748+0000","last_undegraded":"2024-09-10T19:15:56.258748+0000","last_fullsized":"2024-09-10T19:15:56.258748+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'15","last_scrub_stamp":"2024-09-10T19:12:50.449596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:12:50.449596+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0055729239999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":126,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-09-10T19:15:56.318157+0000","last_change":"2024-09-10T19:15:38.678030+0000","last_active":"2024-09-10T19:15:56.318157+0000","last_peered":"2024-09-10T19:15:56.318157+0000","last_clean":"2024-09-10T19:15:56.318157+0000","last_became_active":"2024-09-10T19:10:07.682286+0000","last_became_peered":"2024-09-10T19:10:07.682286+0000","last_unstale":"2024-09-10T19:15:56.318157+0000","last_undegraded":"2024-09-10T19:15:56.318157+0000","last_fullsized":"2024-09-10T19:15:56.318157+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:15:38.677960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:15:38.677960+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:16:51.670221+0000","scrub_duration":0.00460904,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.9","version":"80'9","reported_seq":191,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-09-10T19:15:56.276635+0000","last_change":"2024-09-10T19:13:43.358984+0000","last_active":"2024-09-10T19:15:56.276635+0000","last_peered":"2024-09-10T19:15:56.276635+0000","last_clean":"2024-09-10T19:15:56.276635+0000","last_became_active":"2024-09-10T19:10:11.702878+0000","last_became_peered":"2024-09-10T19:10:11.702878+0000","last_unstale":"2024-09-10T19:15:56.276635+0000","last_undegraded":"2024-09-10T19:15:56.276635+0000","last_fullsized":"2024-09-10T19:15:56.276635+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"80'9","last_scrub_stamp":"2024-09-10T19:13:43.358901+0000","last_deep_scrub":"80'9","last_deep_scrub_stamp":"2024-09-10T19:11:35.174218+0000","last_clean_scrub_stamp":"2024-09-10T19:13:43.358901+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0066143260000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":110,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-09-10T19:15:56.258893+0000","last_change":"2024-09-10T19:12:57.392898+0000","last_active":"2024-09-10T19:15:56.258893+0000","last_peered":"2024-09-10T19:15:56.258893+0000","last_clean":"2024-09-10T19:15:56.258893+0000","last_became_active":"2024-09-10T19:10:14.053875+0000","last_became_peered":"2024-09-10T19:10:14.053875+0000","last_unstale":"2024-09-10T19:15:56.258893+0000","last_undegraded":"2024-09-10T19:15:56.258893+0000","last_fullsized":"2024-09-10T19:15:56.258893+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:12:57.392844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:12:57.392844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0046180839999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"66'12","reported_seq":168,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-09-10T19:15:56.186704+0000","last_change":"2024-09-10T19:15:37.869593+0000","last_active":"2024-09-10T19:15:56.186704+0000","last_peered":"2024-09-10T19:15:56.186704+0000","last_clean":"2024-09-10T19:15:56.186704+0000","last_became_active":"2024-09-10T19:13:54.718152+0000","last_became_peered":"2024-09-10T19:13:54.718152+0000","last_unstale":"2024-09-10T19:15:56.186704+0000","last_undegraded":"2024-09-10T19:15:56.186704+0000","last_fullsized":"2024-09-10T19:15:56.186704+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'12","last_scrub_stamp":"2024-09-10T19:15:37.869451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:15:37.869451+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:16:59.444694+0000","scrub_duration":0.0025441249999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.f","version":"59'2","reported_seq":131,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-09-10T19:15:56.187620+0000","last_change":"2024-09-10T19:12:50.437332+0000","last_active":"2024-09-10T19:15:56.187620+0000","last_peered":"2024-09-10T19:15:56.187620+0000","last_clean":"2024-09-10T19:15:56.187620+0000","last_became_active":"2024-09-10T19:10:07.681719+0000","last_became_peered":"2024-09-10T19:10:07.681719+0000","last_unstale":"2024-09-10T19:15:56.187620+0000","last_undegraded":"2024-09-10T19:15:56.187620+0000","last_fullsized":"2024-09-10T19:15:56.187620+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'2","last_scrub_stamp":"2024-09-10T19:12:50.437245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:12:50.437245+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0070075770000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":14,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":129,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-09-10T19:15:56.316999+0000","last_change":"2024-09-10T19:15:31.776584+0000","last_active":"2024-09-10T19:15:56.316999+0000","last_peered":"2024-09-10T19:15:56.316999+0000","last_clean":"2024-09-10T19:15:56.316999+0000","last_became_active":"2024-09-10T19:13:54.715528+0000","last_became_peered":"2024-09-10T19:13:54.715528+0000","last_unstale":"2024-09-10T19:15:56.316999+0000","last_undegraded":"2024-09-10T19:15:56.316999+0000","last_fullsized":"2024-09-10T19:15:56.316999+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:15:31.776481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:11:46.587846+0000","last_clean_scrub_stamp":"2024-09-10T19:15:31.776481+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:16:33.809672+0000","scrub_duration":0.011259251,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":127,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-09-10T19:15:56.184955+0000","last_change":"2024-09-10T19:15:41.093923+0000","last_active":"2024-09-10T19:15:56.184955+0000","last_peered":"2024-09-10T19:15:56.184955+0000","last_clean":"2024-09-10T19:15:56.184955+0000","last_became_active":"2024-09-10T19:13:54.711804+0000","last_became_peered":"2024-09-10T19:13:54.711804+0000","last_unstale":"2024-09-10T19:15:56.184955+0000","last_undegraded":"2024-09-10T19:15:56.184955+0000","last_fullsized":"2024-09-10T19:15:56.184955+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:15:41.093783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:15:41.093783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:16:51.502004+0000","scrub_duration":0.0023833050000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.16","version":"66'9","reported_seq":140,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-09-10T19:15:56.318402+0000","last_change":"2024-09-10T19:13:50.172901+0000","last_active":"2024-09-10T19:15:56.318402+0000","last_peered":"2024-09-10T19:15:56.318402+0000","last_clean":"2024-09-10T19:15:56.318402+0000","last_became_active":"2024-09-10T19:10:09.822994+0000","last_became_peered":"2024-09-10T19:10:09.822994+0000","last_unstale":"2024-09-10T19:15:56.318402+0000","last_undegraded":"2024-09-10T19:15:56.318402+0000","last_fullsized":"2024-09-10T19:15:56.318402+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'9","last_scrub_stamp":"2024-09-10T19:13:50.172825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:13:50.172825+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0094381619999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":133,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-09-10T19:15:56.318188+0000","last_change":"2024-09-10T19:15:36.662916+0000","last_active":"2024-09-10T19:15:56.318188+0000","last_peered":"2024-09-10T19:15:56.318188+0000","last_clean":"2024-09-10T19:15:56.318188+0000","last_became_active":"2024-09-10T19:13:54.715910+0000","last_became_peered":"2024-09-10T19:13:54.715910+0000","last_unstale":"2024-09-10T19:15:56.318188+0000","last_undegraded":"2024-09-10T19:15:56.318188+0000","last_fullsized":"2024-09-10T19:15:56.318188+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:15:36.662596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:11:41.458523+0000","last_clean_scrub_stamp":"2024-09-10T19:15:36.662596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:16:51.033034+0000","scrub_duration":0.0024511910000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,4],"acting":[2,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":119,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-09-10T19:15:56.185710+0000","last_change":"2024-09-10T19:13:54.712640+0000","last_active":"2024-09-10T19:15:56.185710+0000","last_peered":"2024-09-10T19:15:56.185710+0000","last_clean":"2024-09-10T19:15:56.185710+0000","last_became_active":"2024-09-10T19:13:54.712275+0000","last_became_peered":"2024-09-10T19:13:54.712275+0000","last_unstale":"2024-09-10T19:15:56.185710+0000","last_undegraded":"2024-09-10T19:15:56.185710+0000","last_fullsized":"2024-09-10T19:15:56.185710+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:12:11.619202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:12:11.619202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001950834,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"66'1","reported_seq":121,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-09-10T19:15:56.318163+0000","last_change":"2024-09-10T19:15:48.839638+0000","last_active":"2024-09-10T19:15:56.318163+0000","last_peered":"2024-09-10T19:15:56.318163+0000","last_clean":"2024-09-10T19:15:56.318163+0000","last_became_active":"2024-09-10T19:10:13.717001+0000","last_became_peered":"2024-09-10T19:10:13.717001+0000","last_unstale":"2024-09-10T19:15:56.318163+0000","last_undegraded":"2024-09-10T19:15:56.318163+0000","last_fullsized":"2024-09-10T19:15:56.318163+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'1","last_scrub_stamp":"2024-09-10T19:15:48.839582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:15:48.839582+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:16:48.848204+0000","scrub_duration":0.001851239,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":136,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-09-10T19:15:56.317591+0000","last_change":"2024-09-10T19:15:36.714392+0000","last_active":"2024-09-10T19:15:56.317591+0000","last_peered":"2024-09-10T19:15:56.317591+0000","last_clean":"2024-09-10T19:15:56.317591+0000","last_became_active":"2024-09-10T19:13:54.724027+0000","last_became_peered":"2024-09-10T19:13:54.724027+0000","last_unstale":"2024-09-10T19:15:56.317591+0000","last_undegraded":"2024-09-10T19:15:56.317591+0000","last_fullsized":"2024-09-10T19:15:56.317591+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:15:36.714327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:15:36.714327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:16:55.977022+0000","scrub_duration":0.001831287,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,5],"acting":[6,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"66'6","reported_seq":138,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-09-10T19:15:56.185799+0000","last_change":"2024-09-10T19:13:55.370749+0000","last_active":"2024-09-10T19:15:56.185799+0000","last_peered":"2024-09-10T19:15:56.185799+0000","last_clean":"2024-09-10T19:15:56.185799+0000","last_became_active":"2024-09-10T19:13:54.717949+0000","last_became_peered":"2024-09-10T19:13:54.717949+0000","last_unstale":"2024-09-10T19:15:56.185799+0000","last_undegraded":"2024-09-10T19:15:56.185799+0000","last_fullsized":"2024-09-10T19:15:56.185799+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'6","last_scrub_stamp":"2024-09-10T19:13:55.370650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:13:55.370650+0000","objects_scrubbed":3,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.071693797000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":129,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-09-10T19:15:56.259768+0000","last_change":"2024-09-10T19:15:30.583958+0000","last_active":"2024-09-10T19:15:56.259768+0000","last_peered":"2024-09-10T19:15:56.259768+0000","last_clean":"2024-09-10T19:15:56.259768+0000","last_became_active":"2024-09-10T19:13:54.729917+0000","last_became_peered":"2024-09-10T19:13:54.729917+0000","last_unstale":"2024-09-10T19:15:56.259768+0000","last_undegraded":"2024-09-10T19:15:56.259768+0000","last_fullsized":"2024-09-10T19:15:56.259768+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:15:30.583878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:11:23.264793+0000","last_clean_scrub_stamp":"2024-09-10T19:15:30.583878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:16:56.396822+0000","scrub_duration":0.0015146960000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":110,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-09-10T19:15:56.276696+0000","last_change":"2024-09-10T19:12:51.508398+0000","last_active":"2024-09-10T19:15:56.276696+0000","last_peered":"2024-09-10T19:15:56.276696+0000","last_clean":"2024-09-10T19:15:56.276696+0000","last_became_active":"2024-09-10T19:10:14.054360+0000","last_became_peered":"2024-09-10T19:10:14.054360+0000","last_unstale":"2024-09-10T19:15:56.276696+0000","last_undegraded":"2024-09-10T19:15:56.276696+0000","last_fullsized":"2024-09-10T19:15:56.276696+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:12:51.508263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:12:51.508263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001748274,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"80'15","reported_seq":163,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-09-10T19:15:56.185006+0000","last_change":"2024-09-10T19:15:46.020057+0000","last_active":"2024-09-10T19:15:56.185006+0000","last_peered":"2024-09-10T19:15:56.185006+0000","last_clean":"2024-09-10T19:15:56.185006+0000","last_became_active":"2024-09-10T19:13:54.719574+0000","last_became_peered":"2024-09-10T19:13:54.719574+0000","last_unstale":"2024-09-10T19:15:56.185006+0000","last_undegraded":"2024-09-10T19:15:56.185006+0000","last_fullsized":"2024-09-10T19:15:56.185006+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"80'15","last_scrub_stamp":"2024-09-10T19:15:46.019949+0000","last_deep_scrub":"80'15","last_deep_scrub_stamp":"2024-09-10T19:11:41.719310+0000","last_clean_scrub_stamp":"2024-09-10T19:15:46.019949+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:17:12.728277+0000","scrub_duration":0.0021039190000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":116,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-09-10T19:15:56.259228+0000","last_change":"2024-09-10T19:12:46.506129+0000","last_active":"2024-09-10T19:15:56.259228+0000","last_peered":"2024-09-10T19:15:56.259228+0000","last_clean":"2024-09-10T19:15:56.259228+0000","last_became_active":"2024-09-10T19:10:07.685870+0000","last_became_peered":"2024-09-10T19:10:07.685870+0000","last_unstale":"2024-09-10T19:15:56.259228+0000","last_undegraded":"2024-09-10T19:15:56.259228+0000","last_fullsized":"2024-09-10T19:15:56.259228+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:12:46.506071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:12:46.506071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001257935,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"117'10","reported_seq":202,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-09-10T19:15:56.259587+0000","last_change":"2024-09-10T19:13:54.755445+0000","last_active":"2024-09-10T19:15:56.259587+0000","last_peered":"2024-09-10T19:15:56.259587+0000","last_clean":"2024-09-10T19:15:56.259587+0000","last_became_active":"2024-09-10T19:13:54.724780+0000","last_became_peered":"2024-09-10T19:13:54.724780+0000","last_unstale":"2024-09-10T19:15:56.259587+0000","last_undegraded":"2024-09-10T19:15:56.259587+0000","last_fullsized":"2024-09-10T19:15:56.259587+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'8","last_scrub_stamp":"2024-09-10T19:11:50.330303+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:11:50.330303+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0020032069999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3],"acting":[5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":110,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-09-10T19:15:56.318268+0000","last_change":"2024-09-10T19:13:00.212150+0000","last_active":"2024-09-10T19:15:56.318268+0000","last_peered":"2024-09-10T19:15:56.318268+0000","last_clean":"2024-09-10T19:15:56.318268+0000","last_became_active":"2024-09-10T19:10:13.721035+0000","last_became_peered":"2024-09-10T19:10:13.721035+0000","last_unstale":"2024-09-10T19:15:56.318268+0000","last_undegraded":"2024-09-10T19:15:56.318268+0000","last_fullsized":"2024-09-10T19:15:56.318268+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:13:00.212049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:13:00.212049+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.015146817999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"66'9","reported_seq":150,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-09-10T19:15:56.259618+0000","last_change":"2024-09-10T19:15:45.182861+0000","last_active":"2024-09-10T19:15:56.259618+0000","last_peered":"2024-09-10T19:15:56.259618+0000","last_clean":"2024-09-10T19:15:56.259618+0000","last_became_active":"2024-09-10T19:15:45.149000+0000","last_became_peered":"2024-09-10T19:15:45.149000+0000","last_unstale":"2024-09-10T19:15:56.259618+0000","last_undegraded":"2024-09-10T19:15:56.259618+0000","last_fullsized":"2024-09-10T19:15:56.259618+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'9","last_scrub_stamp":"2024-09-10T19:12:52.416748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:12:52.416748+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0047220839999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":116,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-09-10T19:15:56.317710+0000","last_change":"2024-09-10T19:13:01.210932+0000","last_active":"2024-09-10T19:15:56.317710+0000","last_peered":"2024-09-10T19:15:56.317710+0000","last_clean":"2024-09-10T19:15:56.317710+0000","last_became_active":"2024-09-10T19:10:07.688499+0000","last_became_peered":"2024-09-10T19:10:07.688499+0000","last_unstale":"2024-09-10T19:15:56.317710+0000","last_undegraded":"2024-09-10T19:15:56.317710+0000","last_fullsized":"2024-09-10T19:15:56.317710+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:13:01.210848+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:13:01.210848+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0049627250000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"66'8","reported_seq":190,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-09-10T19:15:56.184744+0000","last_change":"2024-09-10T19:12:59.723961+0000","last_active":"2024-09-10T19:15:56.184744+0000","last_peered":"2024-09-10T19:15:56.184744+0000","last_clean":"2024-09-10T19:15:56.184744+0000","last_became_active":"2024-09-10T19:10:11.692359+0000","last_became_peered":"2024-09-10T19:10:11.692359+0000","last_unstale":"2024-09-10T19:15:56.184744+0000","last_undegraded":"2024-09-10T19:15:56.184744+0000","last_fullsized":"2024-09-10T19:15:56.184744+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'8","last_scrub_stamp":"2024-09-10T19:12:59.723712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:12:59.723712+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0083838109999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":109,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-09-10T19:15:56.187492+0000","last_change":"2024-09-10T19:13:09.463096+0000","last_active":"2024-09-10T19:15:56.187492+0000","last_peered":"2024-09-10T19:15:56.187492+0000","last_clean":"2024-09-10T19:15:56.187492+0000","last_became_active":"2024-09-10T19:10:13.718285+0000","last_became_peered":"2024-09-10T19:10:13.718285+0000","last_unstale":"2024-09-10T19:15:56.187492+0000","last_undegraded":"2024-09-10T19:15:56.187492+0000","last_fullsized":"2024-09-10T19:15:56.187492+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:13:09.463001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:13:09.463001+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.012199573,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.12","version":"79'13","reported_seq":133,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-09-10T19:15:56.317484+0000","last_change":"2024-09-10T19:15:29.854207+0000","last_active":"2024-09-10T19:15:56.317484+0000","last_peered":"2024-09-10T19:15:56.317484+0000","last_clean":"2024-09-10T19:15:56.317484+0000","last_became_active":"2024-09-10T19:13:54.721560+0000","last_became_peered":"2024-09-10T19:13:54.721560+0000","last_unstale":"2024-09-10T19:15:56.317484+0000","last_undegraded":"2024-09-10T19:15:56.317484+0000","last_fullsized":"2024-09-10T19:15:56.317484+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'13","last_scrub_stamp":"2024-09-10T19:15:29.854117+0000","last_deep_scrub":"79'13","last_deep_scrub_stamp":"2024-09-10T19:11:22.072400+0000","last_clean_scrub_stamp":"2024-09-10T19:15:29.854117+0000","objects_scrubbed":4,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:16:56.161162+0000","scrub_duration":0.0066627300000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,0],"acting":[6,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.14","version":"59'1","reported_seq":123,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-09-10T19:15:56.318199+0000","last_change":"2024-09-10T19:13:08.917822+0000","last_active":"2024-09-10T19:15:56.318199+0000","last_peered":"2024-09-10T19:15:56.318199+0000","last_clean":"2024-09-10T19:15:56.318199+0000","last_became_active":"2024-09-10T19:10:07.683263+0000","last_became_peered":"2024-09-10T19:10:07.683263+0000","last_unstale":"2024-09-10T19:15:56.318199+0000","last_undegraded":"2024-09-10T19:15:56.318199+0000","last_fullsized":"2024-09-10T19:15:56.318199+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'1","last_scrub_stamp":"2024-09-10T19:13:08.917719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:13:08.917719+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0033669569999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":362,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":129,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-09-10T19:15:56.185018+0000","last_change":"2024-09-10T19:15:55.925313+0000","last_active":"2024-09-10T19:15:56.185018+0000","last_peered":"2024-09-10T19:15:56.185018+0000","last_clean":"2024-09-10T19:15:56.185018+0000","last_became_active":"2024-09-10T19:13:54.730185+0000","last_became_peered":"2024-09-10T19:13:54.730185+0000","last_unstale":"2024-09-10T19:15:56.185018+0000","last_undegraded":"2024-09-10T19:15:56.185018+0000","last_fullsized":"2024-09-10T19:15:56.185018+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:15:55.925228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:15:55.925228+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:17:11.471760+0000","scrub_duration":0.0017202299999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":117,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-09-10T19:15:56.316732+0000","last_change":"2024-09-10T19:13:54.715762+0000","last_active":"2024-09-10T19:15:56.316732+0000","last_peered":"2024-09-10T19:15:56.316732+0000","last_clean":"2024-09-10T19:15:56.316732+0000","last_became_active":"2024-09-10T19:13:54.714861+0000","last_became_peered":"2024-09-10T19:13:54.714861+0000","last_unstale":"2024-09-10T19:15:56.316732+0000","last_undegraded":"2024-09-10T19:15:56.316732+0000","last_fullsized":"2024-09-10T19:15:56.316732+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:11:51.920006+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:11:51.920006+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001865058,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,6],"acting":[0,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"66'11","reported_seq":164,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-09-10T19:15:56.186871+0000","last_change":"2024-09-10T19:15:53.883613+0000","last_active":"2024-09-10T19:15:56.186871+0000","last_peered":"2024-09-10T19:15:56.186871+0000","last_clean":"2024-09-10T19:15:56.186871+0000","last_became_active":"2024-09-10T19:13:54.720034+0000","last_became_peered":"2024-09-10T19:13:54.720034+0000","last_unstale":"2024-09-10T19:15:56.186871+0000","last_undegraded":"2024-09-10T19:15:56.186871+0000","last_fullsized":"2024-09-10T19:15:56.186871+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'11","last_scrub_stamp":"2024-09-10T19:15:53.883548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:15:53.883548+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:17:04.699778+0000","scrub_duration":0.001858326,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":66,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-09-10T19:15:56.259680+0000","last_change":"2024-09-10T19:13:54.723475+0000","last_active":"2024-09-10T19:15:56.259680+0000","last_peered":"2024-09-10T19:15:56.259680+0000","last_clean":"2024-09-10T19:15:56.259680+0000","last_became_active":"2024-09-10T19:13:54.722947+0000","last_became_peered":"2024-09-10T19:13:54.722947+0000","last_unstale":"2024-09-10T19:15:56.259680+0000","last_undegraded":"2024-09-10T19:15:56.259680+0000","last_fullsized":"2024-09-10T19:15:56.259680+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:12:44.249765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:11:10.031245+0000","last_clean_scrub_stamp":"2024-09-10T19:12:44.249765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,6],"acting":[5,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":76,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-09-10T19:15:56.259023+0000","last_change":"2024-09-10T19:15:36.666310+0000","last_active":"2024-09-10T19:15:56.259023+0000","last_peered":"2024-09-10T19:15:56.259023+0000","last_clean":"2024-09-10T19:15:56.259023+0000","last_became_active":"2024-09-10T19:13:54.729703+0000","last_became_peered":"2024-09-10T19:13:54.729703+0000","last_unstale":"2024-09-10T19:15:56.259023+0000","last_undegraded":"2024-09-10T19:15:56.259023+0000","last_fullsized":"2024-09-10T19:15:56.259023+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:15:36.666216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:15:36.666216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:17:03.635276+0000","scrub_duration":0.003489884,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":110,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-09-10T19:15:56.184482+0000","last_change":"2024-09-10T19:12:54.691886+0000","last_active":"2024-09-10T19:15:56.184482+0000","last_peered":"2024-09-10T19:15:56.184482+0000","last_clean":"2024-09-10T19:15:56.184482+0000","last_became_active":"2024-09-10T19:10:13.724201+0000","last_became_peered":"2024-09-10T19:10:13.724201+0000","last_unstale":"2024-09-10T19:15:56.184482+0000","last_undegraded":"2024-09-10T19:15:56.184482+0000","last_fullsized":"2024-09-10T19:15:56.184482+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:12:54.691810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:12:54.691810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.006702927,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"66'4","reported_seq":143,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-09-10T19:15:56.184518+0000","last_change":"2024-09-10T19:15:33.995417+0000","last_active":"2024-09-10T19:15:56.184518+0000","last_peered":"2024-09-10T19:15:56.184518+0000","last_clean":"2024-09-10T19:15:56.184518+0000","last_became_active":"2024-09-10T19:13:54.711125+0000","last_became_peered":"2024-09-10T19:13:54.711125+0000","last_unstale":"2024-09-10T19:15:56.184518+0000","last_undegraded":"2024-09-10T19:15:56.184518+0000","last_fullsized":"2024-09-10T19:15:56.184518+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'4","last_scrub_stamp":"2024-09-10T19:15:33.995274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:15:33.995274+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:16:44.338208+0000","scrub_duration":0.011108425,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,4],"acting":[3,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":116,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-09-10T19:15:56.259830+0000","last_change":"2024-09-10T19:12:48.517928+0000","last_active":"2024-09-10T19:15:56.259830+0000","last_peered":"2024-09-10T19:15:56.259830+0000","last_clean":"2024-09-10T19:15:56.259830+0000","last_became_active":"2024-09-10T19:10:07.687723+0000","last_became_peered":"2024-09-10T19:10:07.687723+0000","last_unstale":"2024-09-10T19:15:56.259830+0000","last_undegraded":"2024-09-10T19:15:56.259830+0000","last_fullsized":"2024-09-10T19:15:56.259830+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:12:48.517853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:12:48.517853+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001753459,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":112,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-09-10T19:15:56.317255+0000","last_change":"2024-09-10T19:13:04.855026+0000","last_active":"2024-09-10T19:15:56.317255+0000","last_peered":"2024-09-10T19:15:56.317255+0000","last_clean":"2024-09-10T19:15:56.317255+0000","last_became_active":"2024-09-10T19:10:11.698536+0000","last_became_peered":"2024-09-10T19:10:11.698536+0000","last_unstale":"2024-09-10T19:15:56.317255+0000","last_undegraded":"2024-09-10T19:15:56.317255+0000","last_fullsized":"2024-09-10T19:15:56.317255+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:13:04.854929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:11:31.837944+0000","last_clean_scrub_stamp":"2024-09-10T19:13:04.854929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0012499329999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":120,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-09-10T19:15:56.276238+0000","last_change":"2024-09-10T19:15:44.523584+0000","last_active":"2024-09-10T19:15:56.276238+0000","last_peered":"2024-09-10T19:15:56.276238+0000","last_clean":"2024-09-10T19:15:56.276238+0000","last_became_active":"2024-09-10T19:10:13.718271+0000","last_became_peered":"2024-09-10T19:10:13.718271+0000","last_unstale":"2024-09-10T19:15:56.276238+0000","last_undegraded":"2024-09-10T19:15:56.276238+0000","last_fullsized":"2024-09-10T19:15:56.276238+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:15:44.523530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:15:44.523530+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:17:02.565497+0000","scrub_duration":0.027001213999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.11","version":"66'11","reported_seq":24,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-09-10T19:15:56.258773+0000","last_change":"2024-09-10T19:15:45.158623+0000","last_active":"2024-09-10T19:15:56.258773+0000","last_peered":"2024-09-10T19:15:56.258773+0000","last_clean":"2024-09-10T19:15:56.258773+0000","last_became_active":"2024-09-10T19:15:45.149864+0000","last_became_peered":"2024-09-10T19:15:45.149864+0000","last_unstale":"2024-09-10T19:15:56.258773+0000","last_undegraded":"2024-09-10T19:15:56.258773+0000","last_fullsized":"2024-09-10T19:15:56.258773+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'11","last_scrub_stamp":"2024-09-10T19:13:05.729854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:13:05.729854+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.010667395,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":116,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-09-10T19:15:56.318223+0000","last_change":"2024-09-10T19:13:05.876708+0000","last_active":"2024-09-10T19:15:56.318223+0000","last_peered":"2024-09-10T19:15:56.318223+0000","last_clean":"2024-09-10T19:15:56.318223+0000","last_became_active":"2024-09-10T19:10:07.685223+0000","last_became_peered":"2024-09-10T19:10:07.685223+0000","last_unstale":"2024-09-10T19:15:56.318223+0000","last_undegraded":"2024-09-10T19:15:56.318223+0000","last_fullsized":"2024-09-10T19:15:56.318223+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:13:05.876655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:13:05.876655+0000","last_clean_scrub_stamp":"2024-09-10T19:13:05.876655+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0013242589999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":112,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-09-10T19:15:56.275895+0000","last_change":"2024-09-10T19:13:41.326028+0000","last_active":"2024-09-10T19:15:56.275895+0000","last_peered":"2024-09-10T19:15:56.275895+0000","last_clean":"2024-09-10T19:15:56.275895+0000","last_became_active":"2024-09-10T19:10:11.699013+0000","last_became_peered":"2024-09-10T19:10:11.699013+0000","last_unstale":"2024-09-10T19:15:56.275895+0000","last_undegraded":"2024-09-10T19:15:56.275895+0000","last_fullsized":"2024-09-10T19:15:56.275895+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:13:41.325941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:11:57.112092+0000","last_clean_scrub_stamp":"2024-09-10T19:13:41.325941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001598676,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":110,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-09-10T19:15:56.184949+0000","last_change":"2024-09-10T19:13:15.812326+0000","last_active":"2024-09-10T19:15:56.184949+0000","last_peered":"2024-09-10T19:15:56.184949+0000","last_clean":"2024-09-10T19:15:56.184949+0000","last_became_active":"2024-09-10T19:10:14.056172+0000","last_became_peered":"2024-09-10T19:10:14.056172+0000","last_unstale":"2024-09-10T19:15:56.184949+0000","last_undegraded":"2024-09-10T19:15:56.184949+0000","last_fullsized":"2024-09-10T19:15:56.184949+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:13:15.812244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:13:15.812244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0048044389999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"80'2","reported_seq":112,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-09-10T19:15:56.276620+0000","last_change":"2024-09-10T19:12:57.454088+0000","last_active":"2024-09-10T19:15:56.276620+0000","last_peered":"2024-09-10T19:15:56.276620+0000","last_clean":"2024-09-10T19:15:56.276620+0000","last_became_active":"2024-09-10T19:10:13.718875+0000","last_became_peered":"2024-09-10T19:10:13.718875+0000","last_unstale":"2024-09-10T19:15:56.276620+0000","last_undegraded":"2024-09-10T19:15:56.276620+0000","last_fullsized":"2024-09-10T19:15:56.276620+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"80'2","last_scrub_stamp":"2024-09-10T19:12:57.453993+0000","last_deep_scrub":"80'2","last_deep_scrub_stamp":"2024-09-10T19:11:30.272043+0000","last_clean_scrub_stamp":"2024-09-10T19:12:57.453993+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0018294990000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":369,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":116,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-09-10T19:15:56.258865+0000","last_change":"2024-09-10T19:13:14.449643+0000","last_active":"2024-09-10T19:15:56.258865+0000","last_peered":"2024-09-10T19:15:56.258865+0000","last_clean":"2024-09-10T19:15:56.258865+0000","last_became_active":"2024-09-10T19:10:07.689877+0000","last_became_peered":"2024-09-10T19:10:07.689877+0000","last_unstale":"2024-09-10T19:15:56.258865+0000","last_undegraded":"2024-09-10T19:15:56.258865+0000","last_fullsized":"2024-09-10T19:15:56.258865+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:13:14.449593+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:13:14.449593+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001195542,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1e","version":"80'15","reported_seq":134,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-09-10T19:15:56.318371+0000","last_change":"2024-09-10T19:12:50.148157+0000","last_active":"2024-09-10T19:15:56.318371+0000","last_peered":"2024-09-10T19:15:56.318371+0000","last_clean":"2024-09-10T19:15:56.318371+0000","last_became_active":"2024-09-10T19:10:09.823061+0000","last_became_peered":"2024-09-10T19:10:09.823061+0000","last_unstale":"2024-09-10T19:15:56.318371+0000","last_undegraded":"2024-09-10T19:15:56.318371+0000","last_fullsized":"2024-09-10T19:15:56.318371+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"80'15","last_scrub_stamp":"2024-09-10T19:12:50.147336+0000","last_deep_scrub":"80'15","last_deep_scrub_stamp":"2024-09-10T19:11:25.883919+0000","last_clean_scrub_stamp":"2024-09-10T19:12:50.147336+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0064585019999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1f","version":"66'8","reported_seq":203,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-09-10T19:15:56.318122+0000","last_change":"2024-09-10T19:15:53.664303+0000","last_active":"2024-09-10T19:15:56.318122+0000","last_peered":"2024-09-10T19:15:56.318122+0000","last_clean":"2024-09-10T19:15:56.318122+0000","last_became_active":"2024-09-10T19:10:11.695427+0000","last_became_peered":"2024-09-10T19:10:11.695427+0000","last_unstale":"2024-09-10T19:15:56.318122+0000","last_undegraded":"2024-09-10T19:15:56.318122+0000","last_fullsized":"2024-09-10T19:15:56.318122+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'8","last_scrub_stamp":"2024-09-10T19:15:53.664208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:15:53.664208+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:17:15.336245+0000","scrub_duration":0.001849699,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":76,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-09-10T19:15:56.259253+0000","last_change":"2024-09-10T19:15:29.580679+0000","last_active":"2024-09-10T19:15:56.259253+0000","last_peered":"2024-09-10T19:15:56.259253+0000","last_clean":"2024-09-10T19:15:56.259253+0000","last_became_active":"2024-09-10T19:13:54.715192+0000","last_became_peered":"2024-09-10T19:13:54.715192+0000","last_unstale":"2024-09-10T19:15:56.259253+0000","last_undegraded":"2024-09-10T19:15:56.259253+0000","last_fullsized":"2024-09-10T19:15:56.259253+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:15:29.580605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:15:29.580605+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:16:52.844009+0000","scrub_duration":0.0047886710000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,2],"acting":[5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":116,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-09-10T19:15:56.184982+0000","last_change":"2024-09-10T19:13:02.737323+0000","last_active":"2024-09-10T19:15:56.184982+0000","last_peered":"2024-09-10T19:15:56.184982+0000","last_clean":"2024-09-10T19:15:56.184982+0000","last_became_active":"2024-09-10T19:10:07.682689+0000","last_became_peered":"2024-09-10T19:10:07.682689+0000","last_unstale":"2024-09-10T19:15:56.184982+0000","last_undegraded":"2024-09-10T19:15:56.184982+0000","last_fullsized":"2024-09-10T19:15:56.184982+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:13:02.737236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:13:02.737236+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0027813289999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"80'16","reported_seq":192,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-09-10T19:15:56.318189+0000","last_change":"2024-09-10T19:15:47.700592+0000","last_active":"2024-09-10T19:15:56.318189+0000","last_peered":"2024-09-10T19:15:56.318189+0000","last_clean":"2024-09-10T19:15:56.318189+0000","last_became_active":"2024-09-10T19:13:54.713563+0000","last_became_peered":"2024-09-10T19:13:54.713563+0000","last_unstale":"2024-09-10T19:15:56.318189+0000","last_undegraded":"2024-09-10T19:15:56.318189+0000","last_fullsized":"2024-09-10T19:15:56.318189+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"80'16","last_scrub_stamp":"2024-09-10T19:15:47.700523+0000","last_deep_scrub":"80'16","last_deep_scrub_stamp":"2024-09-10T19:11:40.853611+0000","last_clean_scrub_stamp":"2024-09-10T19:15:47.700523+0000","objects_scrubbed":5,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:16:55.889805+0000","scrub_duration":0.0019149010000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,0],"acting":[6,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":112,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-09-10T19:15:56.318356+0000","last_change":"2024-09-10T19:12:59.195000+0000","last_active":"2024-09-10T19:15:56.318356+0000","last_peered":"2024-09-10T19:15:56.318356+0000","last_clean":"2024-09-10T19:15:56.318356+0000","last_became_active":"2024-09-10T19:10:11.701735+0000","last_became_peered":"2024-09-10T19:10:11.701735+0000","last_unstale":"2024-09-10T19:15:56.318356+0000","last_undegraded":"2024-09-10T19:15:56.318356+0000","last_fullsized":"2024-09-10T19:15:56.318356+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:12:59.194914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:11:28.822327+0000","last_clean_scrub_stamp":"2024-09-10T19:12:59.194914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0022130729999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":109,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-09-10T19:15:56.187361+0000","last_change":"2024-09-10T19:13:00.388592+0000","last_active":"2024-09-10T19:15:56.187361+0000","last_peered":"2024-09-10T19:15:56.187361+0000","last_clean":"2024-09-10T19:15:56.187361+0000","last_became_active":"2024-09-10T19:10:14.053619+0000","last_became_peered":"2024-09-10T19:10:14.053619+0000","last_unstale":"2024-09-10T19:15:56.187361+0000","last_undegraded":"2024-09-10T19:15:56.187361+0000","last_fullsized":"2024-09-10T19:15:56.187361+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:13:00.388466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:13:00.388466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.003120996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":116,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-09-10T19:15:56.317698+0000","last_change":"2024-09-10T19:13:43.205095+0000","last_active":"2024-09-10T19:15:56.317698+0000","last_peered":"2024-09-10T19:15:56.317698+0000","last_clean":"2024-09-10T19:15:56.317698+0000","last_became_active":"2024-09-10T19:10:07.685997+0000","last_became_peered":"2024-09-10T19:10:07.685997+0000","last_unstale":"2024-09-10T19:15:56.317698+0000","last_undegraded":"2024-09-10T19:15:56.317698+0000","last_fullsized":"2024-09-10T19:15:56.317698+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:13:43.205042+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:13:43.205042+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0013181790000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1c","version":"78'22","reported_seq":189,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-09-10T19:15:56.317840+0000","last_change":"2024-09-10T19:15:37.702706+0000","last_active":"2024-09-10T19:15:56.317840+0000","last_peered":"2024-09-10T19:15:56.317840+0000","last_clean":"2024-09-10T19:15:56.317840+0000","last_became_active":"2024-09-10T19:13:54.715570+0000","last_became_peered":"2024-09-10T19:13:54.715570+0000","last_unstale":"2024-09-10T19:15:56.317840+0000","last_undegraded":"2024-09-10T19:15:56.317840+0000","last_fullsized":"2024-09-10T19:15:56.317840+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"78'22","last_scrub_stamp":"2024-09-10T19:15:37.702624+0000","last_deep_scrub":"78'22","last_deep_scrub_stamp":"2024-09-10T19:11:21.118937+0000","last_clean_scrub_stamp":"2024-09-10T19:15:37.702624+0000","objects_scrubbed":7,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:17:04.140997+0000","scrub_duration":0.033348765000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":79,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-09-10T19:15:56.187152+0000","last_change":"2024-09-10T19:15:33.008615+0000","last_active":"2024-09-10T19:15:56.187152+0000","last_peered":"2024-09-10T19:15:56.187152+0000","last_clean":"2024-09-10T19:15:56.187152+0000","last_became_active":"2024-09-10T19:13:54.709055+0000","last_became_peered":"2024-09-10T19:13:54.709055+0000","last_unstale":"2024-09-10T19:15:56.187152+0000","last_undegraded":"2024-09-10T19:15:56.187152+0000","last_fullsized":"2024-09-10T19:15:56.187152+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:15:33.008542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:15:33.008542+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:16:51.091294+0000","scrub_duration":0.004557363,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,5],"acting":[4,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":110,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-09-10T19:15:56.185641+0000","last_change":"2024-09-10T19:12:51.767586+0000","last_active":"2024-09-10T19:15:56.185641+0000","last_peered":"2024-09-10T19:15:56.185641+0000","last_clean":"2024-09-10T19:15:56.185641+0000","last_became_active":"2024-09-10T19:10:14.053698+0000","last_became_peered":"2024-09-10T19:10:14.053698+0000","last_unstale":"2024-09-10T19:15:56.185641+0000","last_undegraded":"2024-09-10T19:15:56.185641+0000","last_fullsized":"2024-09-10T19:15:56.185641+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:12:51.767056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:12:51.767056+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.006796519,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":116,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-09-10T19:15:56.185162+0000","last_change":"2024-09-10T19:12:50.719167+0000","last_active":"2024-09-10T19:15:56.185162+0000","last_peered":"2024-09-10T19:15:56.185162+0000","last_clean":"2024-09-10T19:15:56.185162+0000","last_became_active":"2024-09-10T19:10:07.682811+0000","last_became_peered":"2024-09-10T19:10:07.682811+0000","last_unstale":"2024-09-10T19:15:56.185162+0000","last_undegraded":"2024-09-10T19:15:56.185162+0000","last_fullsized":"2024-09-10T19:15:56.185162+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:12:50.718663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:12:50.718663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0042138399999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1d","version":"66'12","reported_seq":166,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-09-10T19:15:56.184854+0000","last_change":"2024-09-10T19:15:52.923561+0000","last_active":"2024-09-10T19:15:56.184854+0000","last_peered":"2024-09-10T19:15:56.184854+0000","last_clean":"2024-09-10T19:15:56.184854+0000","last_became_active":"2024-09-10T19:13:54.714629+0000","last_became_peered":"2024-09-10T19:13:54.714629+0000","last_unstale":"2024-09-10T19:15:56.184854+0000","last_undegraded":"2024-09-10T19:15:56.184854+0000","last_fullsized":"2024-09-10T19:15:56.184854+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'12","last_scrub_stamp":"2024-09-10T19:15:52.923498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:15:52.923498+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:17:20.423846+0000","scrub_duration":0.0019450450000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":111,"reported_epoch":145,"state":"active+clean","last_fresh":"2024-09-10T19:15:56.187038+0000","last_change":"2024-09-10T19:13:11.445426+0000","last_active":"2024-09-10T19:15:56.187038+0000","last_peered":"2024-09-10T19:15:56.187038+0000","last_clean":"2024-09-10T19:15:56.187038+0000","last_became_active":"2024-09-10T19:10:11.701155+0000","last_became_peered":"2024-09-10T19:10:11.701155+0000","last_unstale":"2024-09-10T19:15:56.187038+0000","last_undegraded":"2024-09-10T19:15:56.187038+0000","last_fullsized":"2024-09-10T19:15:56.187038+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:13:11.445238+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:13:11.445238+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0057421110000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":71,"ondisk_log_size":71,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":177,"num_object_clones":0,"num_object_copies":531,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":177,"num_whiteouts":0,"num_read":695,"num_read_kb":452,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":127,"num_bytes_recovered":2668,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":499712,"data_stored":13224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":444,"ondisk_log_size":444,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":195,"num_read_kb":190,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":389,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1613,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1328,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":27,"num_read_kb":27,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":3984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4,"ondisk_log_size":4,"up":96,"acting":96,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79,"ondisk_log_size":79,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":54,"seq":231928234059,"num_pgs":70,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":302948,"kb_used_data":1432,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":93442204,"statfs":{"total":95995035648,"available":95684816896,"internally_reserved":0,"allocated":1466368,"data_stored":1115711,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 19:14:58 2024","interfaces":[{"interface":"back","average":{"1min":1.0620000000000001,"5min":1.724,"15min":0.93100000000000005},"min":{"1min":0.24299999999999999,"5min":0.215,"15min":0.215},"max":{"1min":2.887,"5min":35.551000000000002,"15min":35.551000000000002},"last":0.41499999999999998},{"interface":"front","average":{"1min":1.2150000000000001,"5min":1.5449999999999999,"15min":0.873},"min":{"1min":0.20699999999999999,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":3.2829999999999999,"5min":35.503999999999998,"15min":35.503999999999998},"last":0.52300000000000002}]},{"osd":1,"last update":"Tue Sep 10 19:11:56 2024","interfaces":[{"interface":"back","average":{"1min":0.52400000000000002,"5min":0.52400000000000002,"15min":0.52400000000000002},"min":{"1min":0.36199999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.70199999999999996,"5min":0.79400000000000004,"15min":0.79400000000000004},"last":0.503},{"interface":"front","average":{"1min":0.49199999999999999,"5min":0.51200000000000001,"15min":0.51500000000000001},"min":{"1min":0.28999999999999998,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.68500000000000005,"5min":0.71799999999999997,"15min":0.71799999999999997},"last":0.45500000000000002}]},{"osd":2,"last update":"Tue Sep 10 19:14:58 2024","interfaces":[{"interface":"back","average":{"1min":1.0700000000000001,"5min":1.0469999999999999,"15min":0.69899999999999995},"min":{"1min":0.30199999999999999,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":4.21,"5min":31.940999999999999,"15min":31.940999999999999},"last":0.38900000000000001},{"interface":"front","average":{"1min":0.86599999999999999,"5min":1.077,"15min":0.72499999999999998},"min":{"1min":0.249,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":2.4119999999999999,"5min":25.056000000000001,"15min":25.056000000000001},"last":0.36099999999999999}]},{"osd":3,"last update":"Tue Sep 10 19:14:58 2024","interfaces":[{"interface":"back","average":{"1min":1.163,"5min":1.6759999999999999,"15min":0.91600000000000004},"min":{"1min":0.26000000000000001,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":2.8519999999999999,"5min":35.277999999999999,"15min":35.277999999999999},"last":0.48099999999999998},{"interface":"front","average":{"1min":0.82499999999999996,"5min":1.6559999999999999,"15min":0.89800000000000002},"min":{"1min":0.34200000000000003,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":2.355,"5min":36.512999999999998,"15min":36.512999999999998},"last":0.625}]},{"osd":4,"last update":"Tue Sep 10 19:14:58 2024","interfaces":[{"interface":"back","average":{"1min":0.64000000000000001,"5min":0.50900000000000001,"15min":0.47299999999999998},"min":{"1min":0.26100000000000001,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":2.5249999999999999,"5min":2.863,"15min":2.863},"last":0.58699999999999997},{"interface":"front","average":{"1min":0.72399999999999998,"5min":0.52900000000000003,"15min":0.48299999999999998},"min":{"1min":0.23000000000000001,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":1.5049999999999999,"5min":1.5049999999999999,"15min":1.5049999999999999},"last":0.60699999999999998}]},{"osd":5,"last update":"Tue Sep 10 19:14:58 2024","interfaces":[{"interface":"back","average":{"1min":1.2849999999999999,"5min":0.754,"15min":0.57099999999999995},"min":{"1min":0.27100000000000002,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":11.621,"5min":11.621,"15min":11.621},"last":0.55800000000000005},{"interface":"front","average":{"1min":0.86399999999999999,"5min":0.58799999999999997,"15min":0.51300000000000001},"min":{"1min":0.26000000000000001,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":2.875,"5min":2.8980000000000001,"15min":2.8980000000000001},"last":0.72999999999999998}]},{"osd":6,"last update":"Tue Sep 10 19:14:58 2024","interfaces":[{"interface":"back","average":{"1min":1.0029999999999999,"5min":0.62,"15min":0.54000000000000004},"min":{"1min":0.317,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":2.9089999999999998,"5min":2.9089999999999998,"15min":2.9089999999999998},"last":0.65100000000000002},{"interface":"front","average":{"1min":0.86299999999999999,"5min":0.56100000000000005,"15min":0.51300000000000001},"min":{"1min":0.29499999999999998,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":1.5760000000000001,"5min":1.5760000000000001,"15min":1.5760000000000001},"last":0.54300000000000004}]}]},{"osd":6,"up_from":47,"seq":201863462992,"num_pgs":60,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":302956,"kb_used_data":1440,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":93442196,"statfs":{"total":95995035648,"available":95684808704,"internally_reserved":0,"allocated":1474560,"data_stored":1116255,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 19:15:45 2024","interfaces":[{"interface":"back","average":{"1min":1.3700000000000001,"5min":2.1629999999999998,"15min":1.069},"min":{"1min":0.20100000000000001,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":12.334,"5min":112.988,"15min":112.988},"last":0.55000000000000004},{"interface":"front","average":{"1min":1.3839999999999999,"5min":2.1469999999999998,"15min":1.0780000000000001},"min":{"1min":0.249,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":12.319000000000001,"5min":113.18300000000001,"15min":113.18300000000001},"last":0.65400000000000003}]},{"osd":1,"last update":"Tue Sep 10 19:12:41 2024","interfaces":[{"interface":"back","average":{"1min":0.59999999999999998,"5min":0.53700000000000003,"15min":0.53100000000000003},"min":{"1min":0.33800000000000002,"5min":0.318,"15min":0.318},"max":{"1min":1.29,"5min":1.29,"15min":1.29},"last":0.64100000000000001},{"interface":"front","average":{"1min":0.69399999999999995,"5min":0.55400000000000005,"15min":0.53800000000000003},"min":{"1min":0.32400000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":2.6859999999999999,"5min":2.6859999999999999,"15min":2.6859999999999999},"last":0.47799999999999998}]},{"osd":2,"last update":"Tue Sep 10 19:15:45 2024","interfaces":[{"interface":"back","average":{"1min":0.88400000000000001,"5min":1.956,"15min":1.0149999999999999},"min":{"1min":0.25,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":2.7709999999999999,"5min":112.864,"15min":112.864},"last":0.45700000000000002},{"interface":"front","average":{"1min":1.026,"5min":2.069,"15min":1.0289999999999999},"min":{"1min":0.28399999999999997,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":3.456,"5min":112.822,"15min":112.822},"last":0.496}]},{"osd":3,"last update":"Tue Sep 10 19:15:45 2024","interfaces":[{"interface":"back","average":{"1min":1.9870000000000001,"5min":2.403,"15min":1.145},"min":{"1min":0.32100000000000001,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":12.539,"5min":125.56399999999999,"15min":125.56399999999999},"last":0.58299999999999996},{"interface":"front","average":{"1min":1.948,"5min":2.4900000000000002,"15min":1.2070000000000001},"min":{"1min":0.438,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":12.513,"5min":125.47,"15min":125.47},"last":0.51800000000000002}]},{"osd":4,"last update":"Tue Sep 10 19:15:45 2024","interfaces":[{"interface":"back","average":{"1min":0.54300000000000004,"5min":0.49099999999999999,"15min":0.39500000000000002},"min":{"1min":0.27600000000000002,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":1.3220000000000001,"5min":3.3410000000000002,"15min":3.3410000000000002},"last":0.61399999999999999},{"interface":"front","average":{"1min":0.59899999999999998,"5min":0.53300000000000003,"15min":0.42699999999999999},"min":{"1min":0.22500000000000001,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":1.831,"5min":2.0710000000000002,"15min":2.0710000000000002},"last":0.53300000000000003}]},{"osd":5,"last update":"Tue Sep 10 19:15:45 2024","interfaces":[{"interface":"back","average":{"1min":0.60499999999999998,"5min":0.98699999999999999,"15min":0.57899999999999996},"min":{"1min":0.32600000000000001,"5min":0.248,"15min":0.248},"max":{"1min":1.3540000000000001,"5min":44.765000000000001,"15min":44.765000000000001},"last":0.56299999999999994},{"interface":"front","average":{"1min":0.92300000000000004,"5min":1.1160000000000001,"15min":0.621},"min":{"1min":0.28899999999999998,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":4.0430000000000001,"5min":43.036000000000001,"15min":43.036000000000001},"last":0.629}]},{"osd":7,"last update":"Tue Sep 10 19:15:03 2024","interfaces":[{"interface":"back","average":{"1min":0.69899999999999995,"5min":0.52200000000000002,"15min":0.47199999999999998},"min":{"1min":0.30599999999999999,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":1.7190000000000001,"5min":1.7190000000000001,"15min":1.7190000000000001},"last":0.59999999999999998},{"interface":"front","average":{"1min":0.77500000000000002,"5min":0.59799999999999998,"15min":0.496},"min":{"1min":0.31900000000000001,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":1.9139999999999999,"5min":2.3260000000000001,"15min":2.3260000000000001},"last":0.66800000000000004}]}]},{"osd":5,"up_from":39,"seq":167503724629,"num_pgs":61,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298196,"kb_used_data":840,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93446956,"statfs":{"total":95995035648,"available":95689682944,"internally_reserved":0,"allocated":860160,"data_stored":526119,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 19:15:15 2024","interfaces":[{"interface":"back","average":{"1min":1.2949999999999999,"5min":0.79000000000000004,"15min":0.57999999999999996},"min":{"1min":0.20899999999999999,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":3.403,"5min":5.851,"15min":5.851},"last":0.50600000000000001},{"interface":"front","average":{"1min":0.72599999999999998,"5min":0.67300000000000004,"15min":0.53800000000000003},"min":{"1min":0.251,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":2.883,"5min":5.6500000000000004,"15min":5.6500000000000004},"last":0.56899999999999995}]},{"osd":1,"last update":"Tue Sep 10 19:12:09 2024","interfaces":[{"interface":"back","average":{"1min":0.48199999999999998,"5min":0.51500000000000001,"15min":0.50700000000000001},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.64600000000000002,"5min":0.72799999999999998,"15min":0.72799999999999998},"last":0.48699999999999999},{"interface":"front","average":{"1min":0.47299999999999998,"5min":0.47899999999999998,"15min":0.47399999999999998},"min":{"1min":0.34399999999999997,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.61599999999999999,"5min":0.68799999999999994,"15min":0.68799999999999994},"last":0.52300000000000002}]},{"osd":2,"last update":"Tue Sep 10 19:15:15 2024","interfaces":[{"interface":"back","average":{"1min":0.83699999999999997,"5min":0.67300000000000004,"15min":0.54000000000000004},"min":{"1min":0.29499999999999998,"5min":0.253,"15min":0.253},"max":{"1min":2.9100000000000001,"5min":2.9100000000000001,"15min":2.9100000000000001},"last":0.43099999999999999},{"interface":"front","average":{"1min":0.75800000000000001,"5min":0.79000000000000004,"15min":0.58799999999999997},"min":{"1min":0.24399999999999999,"5min":0.224,"15min":0.224},"max":{"1min":2.4969999999999999,"5min":6.3789999999999996,"15min":6.3789999999999996},"last":0.39200000000000002}]},{"osd":3,"last update":"Tue Sep 10 19:15:15 2024","interfaces":[{"interface":"back","average":{"1min":0.85299999999999998,"5min":0.67300000000000004,"15min":0.56499999999999995},"min":{"1min":0.224,"5min":0.224,"15min":0.224},"max":{"1min":3.2709999999999999,"5min":3.351,"15min":3.351},"last":0.59299999999999997},{"interface":"front","average":{"1min":0.97599999999999998,"5min":0.70499999999999996,"15min":0.58299999999999996},"min":{"1min":0.27800000000000002,"5min":0.248,"15min":0.248},"max":{"1min":5.3869999999999996,"5min":5.3869999999999996,"15min":5.3869999999999996},"last":0.64600000000000002}]},{"osd":4,"last update":"Tue Sep 10 19:15:15 2024","interfaces":[{"interface":"back","average":{"1min":0.876,"5min":0.58099999999999996,"15min":0.42899999999999999},"min":{"1min":0.26600000000000001,"5min":0.23699999999999999,"15min":0.20699999999999999},"max":{"1min":4.3280000000000003,"5min":4.3280000000000003,"15min":4.3280000000000003},"last":0.55800000000000005},{"interface":"front","average":{"1min":0.72699999999999998,"5min":0.64300000000000002,"15min":0.45500000000000002},"min":{"1min":0.26000000000000001,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":3.0409999999999999,"5min":5.8899999999999997,"15min":5.8899999999999997},"last":0.54600000000000004}]},{"osd":6,"last update":"Tue Sep 10 19:15:44 2024","interfaces":[{"interface":"back","average":{"1min":0.81599999999999995,"5min":0.77200000000000002,"15min":0.53100000000000003},"min":{"1min":0.29099999999999998,"5min":0.27100000000000002,"15min":0.223},"max":{"1min":2.9399999999999999,"5min":6.617,"15min":6.617},"last":0.63500000000000001},{"interface":"front","average":{"1min":0.80600000000000005,"5min":0.73199999999999998,"15min":0.51400000000000001},"min":{"1min":0.33800000000000002,"5min":0.26800000000000002,"15min":0.25},"max":{"1min":2.794,"5min":5.9349999999999996,"15min":5.9349999999999996},"last":0.69699999999999995}]},{"osd":7,"last update":"Tue Sep 10 19:15:00 2024","interfaces":[{"interface":"back","average":{"1min":1.1200000000000001,"5min":0.66500000000000004,"15min":0.56100000000000005},"min":{"1min":0.36699999999999999,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":5.9660000000000002,"5min":5.9660000000000002,"15min":5.9660000000000002},"last":0.61899999999999999},{"interface":"front","average":{"1min":0.84899999999999998,"5min":0.58299999999999996,"15min":0.51300000000000001},"min":{"1min":0.29499999999999998,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":2.415,"5min":2.415,"15min":2.415},"last":0.46200000000000002}]}]},{"osd":4,"up_from":32,"seq":137438953561,"num_pgs":60,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":310676,"kb_used_data":840,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":93434476,"statfs":{"total":95995035648,"available":95676903424,"internally_reserved":0,"allocated":860160,"data_stored":524754,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 19:14:59 2024","interfaces":[{"interface":"back","average":{"1min":0.83599999999999997,"5min":0.60499999999999998,"15min":0.56299999999999994},"min":{"1min":0.20899999999999999,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":3.0409999999999999,"5min":3.0409999999999999,"15min":3.0409999999999999},"last":0.52500000000000002},{"interface":"front","average":{"1min":0.85099999999999998,"5min":0.60999999999999999,"15min":0.55700000000000005},"min":{"1min":0.249,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":3.0579999999999998,"5min":3.0579999999999998,"15min":3.0579999999999998},"last":0.54900000000000004}]},{"osd":1,"last update":"Tue Sep 10 19:11:49 2024","interfaces":[{"interface":"back","average":{"1min":0.47999999999999998,"5min":0.503,"15min":0.50900000000000001},"min":{"1min":0.29999999999999999,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.72199999999999998,"5min":0.83499999999999996,"15min":0.83499999999999996},"last":0.49399999999999999},{"interface":"front","average":{"1min":0.51000000000000001,"5min":0.51700000000000002,"15min":0.52200000000000002},"min":{"1min":0.27500000000000002,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.68999999999999995,"5min":0.77100000000000002,"15min":0.77100000000000002},"last":0.60199999999999998}]},{"osd":2,"last update":"Tue Sep 10 19:14:59 2024","interfaces":[{"interface":"back","average":{"1min":0.74399999999999999,"5min":0.61099999999999999,"15min":0.56299999999999994},"min":{"1min":0.214,"5min":0.214,"15min":0.214},"max":{"1min":2.9609999999999999,"5min":2.9609999999999999,"15min":2.9609999999999999},"last":0.56599999999999995},{"interface":"front","average":{"1min":0.85999999999999999,"5min":0.629,"15min":0.56799999999999995},"min":{"1min":0.23400000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":3.73,"5min":3.73,"15min":3.73},"last":0.58799999999999997}]},{"osd":3,"last update":"Tue Sep 10 19:14:59 2024","interfaces":[{"interface":"back","average":{"1min":0.92000000000000004,"5min":0.67200000000000004,"15min":0.57499999999999996},"min":{"1min":0.28999999999999998,"5min":0.252,"15min":0.252},"max":{"1min":3.3839999999999999,"5min":3.3839999999999999,"15min":3.3839999999999999},"last":0.504},{"interface":"front","average":{"1min":1.2310000000000001,"5min":0.76500000000000001,"15min":0.623},"min":{"1min":0.26300000000000001,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":4.6980000000000004,"5min":4.6980000000000004,"15min":4.6980000000000004},"last":0.51500000000000001}]},{"osd":5,"last update":"Tue Sep 10 19:15:22 2024","interfaces":[{"interface":"back","average":{"1min":0.66700000000000004,"5min":0.50600000000000001,"15min":0.41799999999999998},"min":{"1min":0.25700000000000001,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":2.9470000000000001,"5min":3.629,"15min":3.629},"last":0.41999999999999998},{"interface":"front","average":{"1min":0.47099999999999997,"5min":0.495,"15min":0.42699999999999999},"min":{"1min":0.249,"5min":0.248,"15min":0.223},"max":{"1min":1.6180000000000001,"5min":2.1880000000000002,"15min":2.1880000000000002},"last":0.36299999999999999}]},{"osd":6,"last update":"Tue Sep 10 19:15:48 2024","interfaces":[{"interface":"back","average":{"1min":0.61399999999999999,"5min":0.64000000000000001,"15min":0.48799999999999999},"min":{"1min":0.30599999999999999,"5min":0.253,"15min":0.253},"max":{"1min":1.643,"5min":3.964,"15min":3.964},"last":0.46800000000000003},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.59499999999999997,"15min":0.47399999999999998},"min":{"1min":0.31900000000000001,"5min":0.25,"15min":0.25},"max":{"1min":1.4319999999999999,"5min":3.0129999999999999,"15min":3.0129999999999999},"last":0.45900000000000002}]},{"osd":7,"last update":"Tue Sep 10 19:15:03 2024","interfaces":[{"interface":"back","average":{"1min":0.95899999999999996,"5min":0.91200000000000003,"15min":0.65400000000000003},"min":{"1min":0.32400000000000001,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":3.5939999999999999,"5min":23.081,"15min":23.081},"last":0.53600000000000003},{"interface":"front","average":{"1min":0.64400000000000002,"5min":0.85599999999999998,"15min":0.62},"min":{"1min":0.27500000000000002,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":1.901,"5min":22.666,"15min":22.666},"last":0.443}]}]},{"osd":3,"up_from":26,"seq":111669149790,"num_pgs":69,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":306540,"kb_used_data":864,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":93438612,"statfs":{"total":95995035648,"available":95681138688,"internally_reserved":0,"allocated":884736,"data_stored":526438,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 19:15:34 2024","interfaces":[{"interface":"back","average":{"1min":2.2490000000000001,"5min":1.2729999999999999,"15min":0.69499999999999995},"min":{"1min":0.20100000000000001,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":17.518000000000001,"5min":17.518000000000001,"15min":17.518000000000001},"last":0.42099999999999999},{"interface":"front","average":{"1min":2.306,"5min":1.488,"15min":0.77400000000000002},"min":{"1min":0.17399999999999999,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":17.609000000000002,"5min":17.609000000000002,"15min":17.609000000000002},"last":0.35799999999999998}]},{"osd":1,"last update":"Tue Sep 10 19:12:27 2024","interfaces":[{"interface":"back","average":{"1min":0.52800000000000002,"5min":0.44400000000000001,"15min":0.45200000000000001},"min":{"1min":0.216,"5min":0.216,"15min":0.216},"max":{"1min":3.5150000000000001,"5min":3.5150000000000001,"15min":3.5150000000000001},"last":0.38400000000000001},{"interface":"front","average":{"1min":0.46600000000000003,"5min":0.40000000000000002,"15min":0.39800000000000002},"min":{"1min":0.187,"5min":0.187,"15min":0.187},"max":{"1min":1.3640000000000001,"5min":1.3640000000000001,"15min":1.3640000000000001},"last":0.40500000000000003}]},{"osd":2,"last update":"Tue Sep 10 19:15:34 2024","interfaces":[{"interface":"back","average":{"1min":1.3049999999999999,"5min":0.92000000000000004,"15min":0.59799999999999998},"min":{"1min":0.218,"5min":0.186,"15min":0.186},"max":{"1min":4.2290000000000001,"5min":4.2290000000000001,"15min":4.2290000000000001},"last":0.436},{"interface":"front","average":{"1min":2.379,"5min":1.264,"15min":0.69799999999999995},"min":{"1min":0.314,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":17.806000000000001,"5min":17.806000000000001,"15min":17.806000000000001},"last":0.44900000000000001}]},{"osd":4,"last update":"Tue Sep 10 19:15:01 2024","interfaces":[{"interface":"back","average":{"1min":1.1870000000000001,"5min":1.071,"15min":0.71999999999999997},"min":{"1min":0.23599999999999999,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":3.7839999999999998,"5min":11.964,"15min":11.964},"last":0.51500000000000001},{"interface":"front","average":{"1min":2.1120000000000001,"5min":1.339,"15min":0.81100000000000005},"min":{"1min":0.27700000000000002,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":4.0359999999999996,"5min":11.881,"15min":11.881},"last":0.60599999999999998}]},{"osd":5,"last update":"Tue Sep 10 19:15:19 2024","interfaces":[{"interface":"back","average":{"1min":0.82699999999999996,"5min":0.92900000000000005,"15min":0.68700000000000006},"min":{"1min":0.32200000000000001,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":1.9730000000000001,"5min":4.6070000000000002,"15min":4.6070000000000002},"last":0.54600000000000004},{"interface":"front","average":{"1min":1.5669999999999999,"5min":1.2350000000000001,"15min":0.78500000000000003},"min":{"1min":0.22,"5min":0.22,"15min":0.22},"max":{"1min":3.54,"5min":5.9950000000000001,"15min":5.9950000000000001},"last":0.53400000000000003}]},{"osd":6,"last update":"Tue Sep 10 19:15:38 2024","interfaces":[{"interface":"back","average":{"1min":2.0449999999999999,"5min":1.325,"15min":0.82699999999999996},"min":{"1min":0.38900000000000001,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":17.861000000000001,"5min":17.861000000000001,"15min":17.861000000000001},"last":0.56899999999999995},{"interface":"front","average":{"1min":2.0750000000000002,"5min":1.075,"15min":0.72899999999999998},"min":{"1min":0.311,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":17.687000000000001,"5min":17.687000000000001,"15min":17.687000000000001},"last":0.48499999999999999}]},{"osd":7,"last update":"Tue Sep 10 19:15:10 2024","interfaces":[{"interface":"back","average":{"1min":1.226,"5min":0.99399999999999999,"15min":0.71799999999999997},"min":{"1min":0.31,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":2.859,"5min":13.538,"15min":13.538},"last":0.52400000000000002},{"interface":"front","average":{"1min":1.1539999999999999,"5min":1.1299999999999999,"15min":0.76600000000000001},"min":{"1min":0.41299999999999998,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":3.8130000000000002,"5min":13.522,"15min":13.522},"last":0.49399999999999999}]}]},{"osd":2,"up_from":18,"seq":77309411427,"num_pgs":60,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":302344,"kb_used_data":828,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":93442808,"statfs":{"total":95995035648,"available":95685435392,"internally_reserved":0,"allocated":847872,"data_stored":524964,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 19:15:21 2024","interfaces":[{"interface":"back","average":{"1min":1.909,"5min":1.8089999999999999,"15min":0.83199999999999996},"min":{"1min":0.20100000000000001,"5min":0.188,"15min":0.188},"max":{"1min":6.375,"5min":62.543999999999997,"15min":62.543999999999997},"last":0.219},{"interface":"front","average":{"1min":1.5229999999999999,"5min":1.6140000000000001,"15min":0.78600000000000003},"min":{"1min":0.186,"5min":0.186,"15min":0.186},"max":{"1min":5.2839999999999998,"5min":62.576999999999998,"15min":62.576999999999998},"last":0.36299999999999999}]},{"osd":1,"last update":"Tue Sep 10 19:12:08 2024","interfaces":[{"interface":"back","average":{"1min":0.317,"5min":0.378,"15min":0.375},"min":{"1min":0.23400000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.46800000000000003,"5min":0.69799999999999995,"15min":0.69799999999999995},"last":0.253},{"interface":"front","average":{"1min":0.33300000000000002,"5min":0.38600000000000001,"15min":0.38800000000000001},"min":{"1min":0.255,"5min":0.255,"15min":0.255},"max":{"1min":0.499,"5min":0.63400000000000001,"15min":0.63400000000000001},"last":0.33600000000000002}]},{"osd":3,"last update":"Tue Sep 10 19:15:42 2024","interfaces":[{"interface":"back","average":{"1min":1.3620000000000001,"5min":1.911,"15min":0.93899999999999995},"min":{"1min":0.20799999999999999,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":4.4649999999999999,"5min":63.997,"15min":63.997},"last":0.28000000000000003},{"interface":"front","average":{"1min":2.827,"5min":1.9730000000000001,"15min":0.95599999999999996},"min":{"1min":0.27400000000000002,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":8.9800000000000004,"5min":62.365000000000002,"15min":62.365000000000002},"last":0.40600000000000003}]},{"osd":4,"last update":"Tue Sep 10 19:15:58 2024","interfaces":[{"interface":"back","average":{"1min":0.64400000000000002,"5min":0.90200000000000002,"15min":0.68799999999999994},"min":{"1min":0.35999999999999999,"5min":0.254,"15min":0.254},"max":{"1min":1.4470000000000001,"5min":6.0300000000000002,"15min":6.0300000000000002},"last":0.498},{"interface":"front","average":{"1min":1.714,"5min":0.998,"15min":0.71299999999999997},"min":{"1min":0.28699999999999998,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":5.8380000000000001,"5min":5.8380000000000001,"15min":5.8380000000000001},"last":0.68799999999999994}]},{"osd":5,"last update":"Tue Sep 10 19:15:21 2024","interfaces":[{"interface":"back","average":{"1min":2.0779999999999998,"5min":0.98999999999999999,"15min":0.68000000000000005},"min":{"1min":0.33600000000000002,"5min":0.255,"15min":0.255},"max":{"1min":13.17,"5min":13.17,"15min":13.17},"last":0.80800000000000005},{"interface":"front","average":{"1min":2.1560000000000001,"5min":1.137,"15min":0.76500000000000001},"min":{"1min":0.29599999999999999,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":13.744999999999999,"5min":13.744999999999999,"15min":13.744999999999999},"last":0.56799999999999995}]},{"osd":6,"last update":"Tue Sep 10 19:15:42 2024","interfaces":[{"interface":"back","average":{"1min":1.1479999999999999,"5min":1.206,"15min":0.77400000000000002},"min":{"1min":0.35699999999999998,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":3.2799999999999998,"5min":12.611000000000001,"15min":12.611000000000001},"last":0.52100000000000002},{"interface":"front","average":{"1min":1.7929999999999999,"5min":1.3220000000000001,"15min":0.80800000000000005},"min":{"1min":0.28199999999999997,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":5.8620000000000001,"5min":14.459,"15min":14.459},"last":0.72599999999999998}]},{"osd":7,"last update":"Tue Sep 10 19:15:05 2024","interfaces":[{"interface":"back","average":{"1min":2.012,"5min":1.1279999999999999,"15min":0.78000000000000003},"min":{"1min":0.32500000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":13.779999999999999,"5min":13.779999999999999,"15min":13.779999999999999},"last":0.54600000000000004},{"interface":"front","average":{"1min":1.617,"5min":1.006,"15min":0.70099999999999996},"min":{"1min":0.36599999999999999,"5min":0.312,"15min":0.312},"max":{"1min":11.714,"5min":11.714,"15min":11.714},"last":0.71199999999999997}]}]},{"osd":1,"up_from":0,"seq":592705486857,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":8,"seq":34359738477,"num_pgs":64,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":311268,"kb_used_data":1432,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":93433884,"statfs":{"total":95995035648,"available":95676297216,"internally_reserved":0,"allocated":1466368,"data_stored":1116548,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Sep 10 19:12:44 2024","interfaces":[{"interface":"back","average":{"1min":0.73199999999999998,"5min":0.42999999999999999,"15min":0.35799999999999998},"min":{"1min":0.17399999999999999,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":2.7320000000000002,"5min":2.7320000000000002,"15min":2.7320000000000002},"last":0.32400000000000001},{"interface":"front","average":{"1min":0.82299999999999995,"5min":0.45100000000000001,"15min":0.36799999999999999},"min":{"1min":0.21199999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":2.4609999999999999,"5min":2.4609999999999999,"15min":2.4609999999999999},"last":0.42199999999999999}]},{"osd":2,"last update":"Tue Sep 10 19:15:18 2024","interfaces":[{"interface":"back","average":{"1min":1.4630000000000001,"5min":0.88800000000000001,"15min":0.55300000000000005},"min":{"1min":0.23499999999999999,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":6.4640000000000004,"5min":6.4640000000000004,"15min":6.4640000000000004},"last":0.374},{"interface":"front","average":{"1min":2.3050000000000002,"5min":0.98299999999999998,"15min":0.57699999999999996},"min":{"1min":0.28100000000000003,"5min":0.187,"15min":0.187},"max":{"1min":9.9399999999999995,"5min":9.9399999999999995,"15min":9.9399999999999995},"last":0.25900000000000001}]},{"osd":3,"last update":"Tue Sep 10 19:15:38 2024","interfaces":[{"interface":"back","average":{"1min":2.0800000000000001,"5min":1.028,"15min":0.626},"min":{"1min":0.32300000000000001,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":6.5590000000000002,"5min":6.5590000000000002,"15min":6.5590000000000002},"last":0.34300000000000003},{"interface":"front","average":{"1min":2.581,"5min":1.26,"15min":0.69299999999999995},"min":{"1min":0.28100000000000003,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":7.2140000000000004,"5min":7.2140000000000004,"15min":7.2140000000000004},"last":0.35499999999999998}]},{"osd":4,"last update":"Tue Sep 10 19:15:57 2024","interfaces":[{"interface":"back","average":{"1min":0.622,"5min":0.78300000000000003,"15min":0.63700000000000001},"min":{"1min":0.35599999999999998,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":1.72,"5min":5.8499999999999996,"15min":5.8499999999999996},"last":0.63400000000000001},{"interface":"front","average":{"1min":0.83099999999999996,"5min":0.81100000000000005,"15min":0.64500000000000002},"min":{"1min":0.27900000000000003,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":3.8180000000000001,"5min":3.8180000000000001,"15min":3.8180000000000001},"last":0.75900000000000001}]},{"osd":5,"last update":"Tue Sep 10 19:15:18 2024","interfaces":[{"interface":"back","average":{"1min":1.661,"5min":0.94199999999999995,"15min":0.70399999999999996},"min":{"1min":0.38800000000000001,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":5.3860000000000001,"5min":5.3860000000000001,"15min":5.3860000000000001},"last":0.622},{"interface":"front","average":{"1min":1.2529999999999999,"5min":0.88300000000000001,"15min":0.67800000000000005},"min":{"1min":0.26000000000000001,"5min":0.253,"15min":0.253},"max":{"1min":5.9749999999999996,"5min":5.9749999999999996,"15min":5.9749999999999996},"last":0.68000000000000005}]},{"osd":6,"last update":"Tue Sep 10 19:15:38 2024","interfaces":[{"interface":"back","average":{"1min":0.81699999999999995,"5min":0.93500000000000005,"15min":0.68899999999999995},"min":{"1min":0.371,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":1.806,"5min":5.8730000000000002,"15min":5.8730000000000002},"last":0.78100000000000003},{"interface":"front","average":{"1min":0.85699999999999998,"5min":0.94899999999999995,"15min":0.70699999999999996},"min":{"1min":0.35799999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":3.8620000000000001,"5min":5.2850000000000001,"15min":5.2850000000000001},"last":0.65100000000000002}]},{"osd":7,"last update":"Tue Sep 10 19:15:06 2024","interfaces":[{"interface":"back","average":{"1min":1.3109999999999999,"5min":0.90700000000000003,"15min":0.68700000000000006},"min":{"1min":0.318,"5min":0.318,"15min":0.318},"max":{"1min":5.1219999999999999,"5min":5.1219999999999999,"15min":5.1219999999999999},"last":0.69099999999999995},{"interface":"front","average":{"1min":1.379,"5min":0.85699999999999998,"15min":0.66000000000000003},"min":{"1min":0.30499999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":6.0469999999999997,"5min":6.0469999999999997,"15min":6.0469999999999997},"last":0.72299999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":966,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1236,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1236,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":362,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1650,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":2118,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1018,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":2008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1942,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T19:16:05.753 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:16:05.778 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:16:05.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:16:05.779 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:16:05.880 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:16:05.905 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:16:05.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:16:05.907 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:16:06.007 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:16:06.032 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:16:06.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:16:06.033 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:16:06.133 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:16:06.157 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:16:06.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:16:06.158 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:16:06.259 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:16:06.282 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:16:06.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:16:06.283 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:16:06.383 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:16:06.406 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:16:06.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:16:06.407 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:16:06.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:16:06 smithi183 ceph-mon[29736]: pgmap v238: 132 pgs: 132 active+clean; 582 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-10T19:16:06.431 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:16:06 smithi183 ceph-mon[41562]: pgmap v238: 132 pgs: 132 active+clean; 582 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-10T19:16:06.465 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:16:06 smithi191 ceph-mon[32884]: pgmap v238: 132 pgs: 132 active+clean; 582 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-10T19:16:06.508 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:16:06.531 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:16:06.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:16:06.532 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:16:06.633 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:16:06.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:16:06.657 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:16:06.657 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:16:06.758 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:16:06.780 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:16:06.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:16:06.781 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:16:06.882 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:16:06.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:16:06.908 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:16:06.908 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:16:06.994 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:16:07.009 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:16:07.032 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:16:07.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:16:07.033 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:16:07.134 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:16:07.156 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:16:07.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:16:07.157 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:16:07.205 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-10T19:16:07.258 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:16:07.280 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:16:07.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:16:07.282 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:16:07.382 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:16:07.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:16:07.408 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:16:07.409 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:16:07.509 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:16:07.530 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:16:07.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:16:07.531 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:16:07.632 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:16:07.658 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:16:07.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:16:07.659 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:16:07.697 INFO:teuthology.orchestra.run.smithi183.stdout:ERROR: (22) Invalid argument 2024-09-10T19:16:07.697 INFO:teuthology.orchestra.run.smithi183.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-10T19:16:07.715 INFO:journalctl@ceph.mgr.x.smithi191.stdout:Sep 10 19:16:07 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-x[34967]: ::ffff:172.21.15.191 - - [10/Sep/2024:19:16:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T19:16:07.760 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:16:07.784 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:16:07.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:16:07.785 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:16:07.886 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:16:07.909 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:16:07.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:16:07.910 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:16:08.011 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:16:08.033 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:16:08.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:16:08.034 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:16:08.134 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:16:08.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:16:08.157 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:16:08.158 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:16:08.258 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:16:08.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:16:08.281 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:16:08.281 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:16:08.334 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph daemon osd.3 dump_historic_ops 2024-09-10T19:16:08.381 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:16:08.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:16:08.408 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:16:08.408 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:16:08.509 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:16:08.530 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:16:08.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:16:08.531 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:16:08.632 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:16:08.657 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:16:08.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:16:08.658 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:16:08.759 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:16:08.785 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:16:08.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:16:08.786 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:16:08.887 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:16:08.907 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:16:08 smithi183 ceph-mon[29736]: pgmap v239: 132 pgs: 132 active+clean; 582 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 966 B/s rd, 0 op/s 2024-09-10T19:16:08.907 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:16:08 smithi183 ceph-mon[41562]: pgmap v239: 132 pgs: 132 active+clean; 582 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 966 B/s rd, 0 op/s 2024-09-10T19:16:08.912 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:16:08.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:16:08.914 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:16:09.014 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:16:09.041 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:16:09.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:16:09.042 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:16:09.142 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:16:09.160 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:16:08 smithi191 ceph-mon[32884]: pgmap v239: 132 pgs: 132 active+clean; 582 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 966 B/s rd, 0 op/s 2024-09-10T19:16:09.166 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:16:09.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:16:09.167 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:16:09.268 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:16:09.293 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:16:09.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:16:09.294 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:16:09.309 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:16:09.339 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:16:09.394 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:16:09.416 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:16:09.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:16:09.417 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:16:09.465 INFO:journalctl@ceph.iscsi.iscsi.a.smithi191.stdout:Sep 10 19:16:09 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-iscsi-iscsi-a[61816]: debug there is no tcmu-runner data available 2024-09-10T19:16:09.518 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:16:09.540 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:16:09.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:16:09.541 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:16:09.641 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:16:09.667 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:16:09.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:16:09.668 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:16:09.769 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:16:09.792 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:16:09.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:16:09.793 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:16:09.894 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:16:09.915 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:16:09.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:16:09.916 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:16:10.016 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:16:10.039 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:16:10.039 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:16:09 smithi183 ceph-mon[29736]: from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:16:10.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:16:10.040 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:16:10.040 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:16:09 smithi183 ceph-mon[41562]: from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:16:10.141 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:16:10.162 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:16:10.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:16:10.162 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:16:10.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:16:09 smithi191 ceph-mon[32884]: from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:16:10.263 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:16:10.289 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:16:10.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:16:10.290 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:16:10.391 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:16:10.416 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:16:10.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:16:10.417 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:16:10.518 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:16:10.540 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:16:10.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:16:10.541 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:16:10.641 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:16:10.663 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:16:10.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:16:10.664 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:16:10.747 INFO:tasks.thrashosds.thrasher:in_osds: [0, 2, 3, 4, 5, 6, 7] out_osds: [1] dead_osds: [] live_osds: [0, 2, 3, 4, 5, 6, 7, 1] 2024-09-10T19:16:10.747 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-10T19:16:10.748 INFO:tasks.thrashosds.thrasher:Removing osd 2, in_osds are: [0, 2, 3, 4, 5, 6, 7] 2024-09-10T19:16:10.748 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph osd out 2 2024-09-10T19:16:10.764 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:16:10.812 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:16:10.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:16:10.813 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:16:10.913 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:16:10.938 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:16:10.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:16:10.939 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:16:11.040 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:16:11.065 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:16:11.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:16:11.065 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:16:11.086 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:16:10 smithi183 ceph-mon[29736]: pgmap v240: 132 pgs: 132 active+clean; 582 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 921 B/s rd, 0 op/s 2024-09-10T19:16:11.086 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:16:10 smithi183 ceph-mon[41562]: pgmap v240: 132 pgs: 132 active+clean; 582 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 921 B/s rd, 0 op/s 2024-09-10T19:16:11.167 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:16:11.192 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:16:11.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:16:11.193 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:16:11.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:16:10 smithi191 ceph-mon[32884]: pgmap v240: 132 pgs: 132 active+clean; 582 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 921 B/s rd, 0 op/s 2024-09-10T19:16:11.294 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:16:11.316 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:16:11.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:16:11.317 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:16:11.412 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Sep 10 19:16:11 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-y[29989]: ::ffff:172.21.15.191 - - [10/Sep/2024:19:16:11] "GET /metrics HTTP/1.1" 200 223966 "" "Prometheus/2.43.0" 2024-09-10T19:16:11.418 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:16:11.444 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:16:11.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:16:11.445 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:16:11.545 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:16:11.567 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:16:11.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:16:11.568 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:16:11.668 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:16:11.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:16:11.691 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:16:11.691 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:16:11.789 INFO:teuthology.orchestra.run.smithi183.stderr:pool 'unique_pool_1' created 2024-09-10T19:16:11.790 INFO:teuthology.orchestra.run.smithi183.stderr:nodeep-scrub is set 2024-09-10T19:16:11.792 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:16:11.816 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:16:11.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:16:11.817 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:16:11.918 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:16:11.942 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:16:11.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:16:11.943 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:16:12.025 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:16:11 smithi183 ceph-mon[29736]: purged_snaps scrub starts 2024-09-10T19:16:12.025 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:16:11 smithi183 ceph-mon[29736]: purged_snaps scrub ok 2024-09-10T19:16:12.025 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:16:11 smithi183 ceph-mon[29736]: from='client.? 172.21.15.183:0/535788597' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16}]: dispatch 2024-09-10T19:16:12.025 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:16:11 smithi183 ceph-mon[29736]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16}]: dispatch 2024-09-10T19:16:12.025 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:16:11 smithi183 ceph-mon[29736]: from='client.? 172.21.15.183:0/1623843690' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-10T19:16:12.025 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:16:11 smithi183 ceph-mon[29736]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-10T19:16:12.026 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:16:11 smithi183 ceph-mon[41562]: purged_snaps scrub starts 2024-09-10T19:16:12.026 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:16:11 smithi183 ceph-mon[41562]: purged_snaps scrub ok 2024-09-10T19:16:12.026 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:16:11 smithi183 ceph-mon[41562]: from='client.? 172.21.15.183:0/535788597' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16}]: dispatch 2024-09-10T19:16:12.026 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:16:11 smithi183 ceph-mon[41562]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16}]: dispatch 2024-09-10T19:16:12.026 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:16:11 smithi183 ceph-mon[41562]: from='client.? 172.21.15.183:0/1623843690' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-10T19:16:12.026 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:16:11 smithi183 ceph-mon[41562]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-10T19:16:12.044 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:16:12.068 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:16:12.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:16:12.069 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:16:12.170 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:16:12.194 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:16:12.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:16:12.195 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:16:12.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:16:11 smithi191 ceph-mon[32884]: purged_snaps scrub starts 2024-09-10T19:16:12.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:16:11 smithi191 ceph-mon[32884]: purged_snaps scrub ok 2024-09-10T19:16:12.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:16:11 smithi191 ceph-mon[32884]: from='client.? 172.21.15.183:0/535788597' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16}]: dispatch 2024-09-10T19:16:12.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:16:11 smithi191 ceph-mon[32884]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16}]: dispatch 2024-09-10T19:16:12.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:16:11 smithi191 ceph-mon[32884]: from='client.? 172.21.15.183:0/1623843690' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-10T19:16:12.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:16:11 smithi191 ceph-mon[32884]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-10T19:16:12.295 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:16:12.320 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:16:12.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:16:12.321 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:16:12.422 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:16:12.445 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:16:12.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:16:12.446 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:16:12.547 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:16:12.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:16:12.569 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:16:12.569 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:16:12.670 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:16:12.695 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:16:12.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:16:12.696 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:16:12.797 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:16:12.823 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:16:12.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:16:12.823 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:16:12.924 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:16:12.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:16:12.949 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:16:12.949 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:16:13.050 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:16:13.071 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:16:13.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:16:13.072 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:16:13.173 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:16:13.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:16:12 smithi183 ceph-mon[29736]: pgmap v241: 132 pgs: 132 active+clean; 582 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 888 B/s rd, 0 op/s 2024-09-10T19:16:13.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:16:12 smithi183 ceph-mon[29736]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T19:16:13.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:16:12 smithi183 ceph-mon[29736]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16}]': finished 2024-09-10T19:16:13.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:16:12 smithi183 ceph-mon[29736]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-10T19:16:13.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:16:12 smithi183 ceph-mon[29736]: osdmap e148: 8 total, 8 up, 7 in 2024-09-10T19:16:13.181 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:16:12 smithi183 ceph-mon[41562]: pgmap v241: 132 pgs: 132 active+clean; 582 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 888 B/s rd, 0 op/s 2024-09-10T19:16:13.181 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:16:12 smithi183 ceph-mon[41562]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T19:16:13.181 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:16:12 smithi183 ceph-mon[41562]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16}]': finished 2024-09-10T19:16:13.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:16:12 smithi183 ceph-mon[41562]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-10T19:16:13.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:16:12 smithi183 ceph-mon[41562]: osdmap e148: 8 total, 8 up, 7 in 2024-09-10T19:16:13.197 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:16:13.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:16:13.198 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:16:13.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:16:12 smithi191 ceph-mon[32884]: pgmap v241: 132 pgs: 132 active+clean; 582 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 888 B/s rd, 0 op/s 2024-09-10T19:16:13.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:16:12 smithi191 ceph-mon[32884]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T19:16:13.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:16:12 smithi191 ceph-mon[32884]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_1", "pg_num": 16}]': finished 2024-09-10T19:16:13.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:16:12 smithi191 ceph-mon[32884]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-10T19:16:13.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:16:12 smithi191 ceph-mon[32884]: osdmap e148: 8 total, 8 up, 7 in 2024-09-10T19:16:13.299 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:16:13.323 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:16:13.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:16:13.324 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:16:13.339 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph osd pool application enable unique_pool_1 rados --yes-i-really-mean-it || true 2024-09-10T19:16:13.425 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:16:13.449 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:16:13.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:16:13.450 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:16:13.551 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:16:13.575 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:16:13.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:16:13.576 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:16:13.677 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:16:13.701 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:16:13.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:16:13.702 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:16:13.803 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:16:13.827 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:16:13.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:16:13.828 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:16:13.929 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:16:13.951 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:16:13.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:16:13.952 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:16:14.053 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:16:14.074 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:16:14.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:16:14.075 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:16:14.087 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:16:13 smithi183 ceph-mon[29736]: osdmap e149: 8 total, 8 up, 7 in 2024-09-10T19:16:14.087 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:16:13 smithi183 ceph-mon[29736]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:16:14.087 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:16:13 smithi183 ceph-mon[41562]: osdmap e149: 8 total, 8 up, 7 in 2024-09-10T19:16:14.087 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:16:13 smithi183 ceph-mon[41562]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:16:14.176 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:16:14.202 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:16:14.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:16:14.203 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:16:14.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:16:13 smithi191 ceph-mon[32884]: osdmap e149: 8 total, 8 up, 7 in 2024-09-10T19:16:14.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:16:13 smithi191 ceph-mon[32884]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:16:14.304 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:16:14.325 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:16:14.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:16:14.326 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:16:14.427 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:16:14.448 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:16:14.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:16:14.449 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:16:14.550 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:16:14.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:16:14.571 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:16:14.571 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:16:14.672 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:16:14.699 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:16:14.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:16:14.700 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:16:14.801 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:16:14.824 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:16:14.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:16:14.825 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:16:14.926 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:16:14.949 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:16:14.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:16:14.950 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:16:14.985 INFO:teuthology.orchestra.run.smithi183.stdout:ERROR: (22) Invalid argument 2024-09-10T19:16:14.985 INFO:teuthology.orchestra.run.smithi183.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-10T19:16:15.053 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:16:15.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:16:15.077 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:16:15.077 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:16:15.178 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:16:15.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:16:14 smithi183 ceph-mon[29736]: pgmap v244: 148 pgs: 16 unknown, 132 active+clean; 582 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 639 B/s rd, 0 op/s 2024-09-10T19:16:15.182 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:16:14 smithi183 ceph-mon[29736]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T19:16:15.182 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:16:14 smithi183 ceph-mon[29736]: osdmap e150: 8 total, 8 up, 7 in 2024-09-10T19:16:15.182 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:16:14 smithi183 ceph-mon[29736]: osdmap e151: 8 total, 8 up, 7 in 2024-09-10T19:16:15.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:16:14 smithi183 ceph-mon[41562]: pgmap v244: 148 pgs: 16 unknown, 132 active+clean; 582 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 639 B/s rd, 0 op/s 2024-09-10T19:16:15.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:16:14 smithi183 ceph-mon[41562]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T19:16:15.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:16:14 smithi183 ceph-mon[41562]: osdmap e150: 8 total, 8 up, 7 in 2024-09-10T19:16:15.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:16:14 smithi183 ceph-mon[41562]: osdmap e151: 8 total, 8 up, 7 in 2024-09-10T19:16:15.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:16:15.202 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:16:15.202 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:16:15.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:16:14 smithi191 ceph-mon[32884]: pgmap v244: 148 pgs: 16 unknown, 132 active+clean; 582 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 639 B/s rd, 0 op/s 2024-09-10T19:16:15.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:16:14 smithi191 ceph-mon[32884]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T19:16:15.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:16:14 smithi191 ceph-mon[32884]: osdmap e150: 8 total, 8 up, 7 in 2024-09-10T19:16:15.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:16:14 smithi191 ceph-mon[32884]: osdmap e151: 8 total, 8 up, 7 in 2024-09-10T19:16:15.303 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:16:15.327 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:16:15.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:16:15.328 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:16:15.429 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:16:15.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:16:15.451 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:16:15.452 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:16:15.552 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:16:15.577 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:16:15.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:16:15.578 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:16:15.646 DEBUG:teuthology.orchestra.run.smithi191:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph daemon osd.4 dump_ops_in_flight 2024-09-10T19:16:15.679 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:16:15.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:16:15.701 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:16:15.702 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:16:15.802 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:16:15.824 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:16:15.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:16:15.825 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:16:15.897 INFO:teuthology.orchestra.run.smithi191.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.b/config 2024-09-10T19:16:15.926 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:16:15.946 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:16:15.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:16:15.947 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:16:16.028 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:16:16.048 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:16:16.069 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:16:16.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:16:16.070 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:16:16.170 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:16:16.193 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:16:16.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:16:16.194 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:16:16.276 INFO:teuthology.orchestra.run.smithi183.stdout:osd.0: {} 2024-09-10T19:16:16.276 INFO:teuthology.orchestra.run.smithi183.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-10T19:16:16.292 INFO:teuthology.orchestra.run.smithi183.stdout:osd.1: {} 2024-09-10T19:16:16.293 INFO:teuthology.orchestra.run.smithi183.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-10T19:16:16.295 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:16:16.311 INFO:teuthology.orchestra.run.smithi183.stdout:osd.2: {} 2024-09-10T19:16:16.312 INFO:teuthology.orchestra.run.smithi183.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-10T19:16:16.317 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:16:16.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:16:16.318 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:16:16.327 INFO:teuthology.orchestra.run.smithi183.stdout:osd.3: {} 2024-09-10T19:16:16.327 INFO:teuthology.orchestra.run.smithi183.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-10T19:16:16.342 INFO:teuthology.orchestra.run.smithi183.stdout:osd.4: {} 2024-09-10T19:16:16.343 INFO:teuthology.orchestra.run.smithi183.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-10T19:16:16.358 INFO:teuthology.orchestra.run.smithi183.stdout:osd.5: {} 2024-09-10T19:16:16.358 INFO:teuthology.orchestra.run.smithi183.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-10T19:16:16.373 INFO:teuthology.orchestra.run.smithi183.stdout:osd.6: {} 2024-09-10T19:16:16.373 INFO:teuthology.orchestra.run.smithi183.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-10T19:16:16.388 INFO:teuthology.orchestra.run.smithi183.stdout:osd.7: {} 2024-09-10T19:16:16.388 INFO:teuthology.orchestra.run.smithi183.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-10T19:16:16.419 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:16:16.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:16:16.441 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:16:16.442 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:16:16.511 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph osd unset noscrub 2024-09-10T19:16:16.542 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:16:16.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:16:16.566 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:16:16.566 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:16:16.667 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:16:16.689 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:16:16.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:16:16.690 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:16:16.791 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:16:16.815 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:16:16.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:16:16.816 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:16:16.916 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:16:16.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:16:16.940 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:16:16.940 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:16:17.041 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:16:17.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:16:17.063 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:16:17.063 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:16:17.064 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:16:16 smithi183 ceph-mon[41562]: pgmap v247: 148 pgs: 148 active+clean; 582 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:16:17.064 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:16:16 smithi183 ceph-mon[41562]: from='client.? 172.21.15.183:0/121807611' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T19:16:17.064 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:16:16 smithi183 ceph-mon[29736]: pgmap v247: 148 pgs: 148 active+clean; 582 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:16:17.064 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:16:16 smithi183 ceph-mon[29736]: from='client.? 172.21.15.183:0/121807611' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T19:16:17.164 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:16:17.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:16:17.188 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:16:17.188 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:16:17.195 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:16:16 smithi191 ceph-mon[32884]: pgmap v247: 148 pgs: 148 active+clean; 582 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:16:17.196 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:16:16 smithi191 ceph-mon[32884]: from='client.? 172.21.15.183:0/121807611' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T19:16:17.289 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:16:17.311 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:16:17.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:16:17.312 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:16:17.397 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:16:17.413 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:16:17.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:16:17.436 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:16:17.436 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:16:17.537 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:16:17.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:16:17.560 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:16:17.561 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:16:17.662 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:16:17.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:16:17.686 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:16:17.686 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:16:17.715 INFO:journalctl@ceph.mgr.x.smithi191.stdout:Sep 10 19:16:17 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-x[34967]: ::ffff:172.21.15.191 - - [10/Sep/2024:19:16:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T19:16:17.787 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:16:17.799 INFO:teuthology.orchestra.run.smithi191.stdout:{ 2024-09-10T19:16:17.799 INFO:teuthology.orchestra.run.smithi191.stdout: "ops": [], 2024-09-10T19:16:17.799 INFO:teuthology.orchestra.run.smithi191.stdout: "num_ops": 0 2024-09-10T19:16:17.799 INFO:teuthology.orchestra.run.smithi191.stdout:} 2024-09-10T19:16:17.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:16:17.810 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:16:17.810 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:16:17.912 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:16:17.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:16:17.934 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:16:17.934 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:16:18.035 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:16:18.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:16:18.057 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:16:18.058 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:16:18.158 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:16:18.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:16:18.181 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:16:18.181 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:16:18.282 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:16:18.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:16:18.305 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:16:18.305 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:16:18.406 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:16:18.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:16:18.430 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:16:18.430 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:16:18.461 DEBUG:teuthology.orchestra.run.smithi191:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph daemon osd.4 dump_blocked_ops 2024-09-10T19:16:18.531 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:16:18.566 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:16:18.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:16:18.567 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:16:18.668 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:16:18.691 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:16:18.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:16:18.692 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:16:18.751 INFO:teuthology.orchestra.run.smithi191.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.b/config 2024-09-10T19:16:18.793 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:16:18.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:16:18.815 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:16:18.815 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:16:18.916 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:16:18.931 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:16:18 smithi183 ceph-mon[29736]: pgmap v248: 148 pgs: 148 active+clean; 582 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 862 B/s rd, 0 op/s 2024-09-10T19:16:18.931 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:16:18 smithi183 ceph-mon[41562]: pgmap v248: 148 pgs: 148 active+clean; 582 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 862 B/s rd, 0 op/s 2024-09-10T19:16:18.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:16:18.939 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:16:18.939 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:16:19.040 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:16:19.061 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:16:19.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:16:19.062 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:16:19.114 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:16:19.163 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:16:19.166 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:16:18 smithi191 ceph-mon[32884]: pgmap v248: 148 pgs: 148 active+clean; 582 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 862 B/s rd, 0 op/s 2024-09-10T19:16:19.184 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:16:19.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:16:19.185 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:16:19.286 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:16:19.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:16:19.308 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:16:19.309 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:16:19.410 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:16:19.431 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:16:19.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:16:19.432 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:16:19.466 INFO:journalctl@ceph.iscsi.iscsi.a.smithi191.stdout:Sep 10 19:16:19 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-iscsi-iscsi-a[61816]: debug there is no tcmu-runner data available 2024-09-10T19:16:19.533 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:16:19.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:16:19.557 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:16:19.558 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:16:19.658 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:16:19.680 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:16:19.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:16:19.681 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:16:19.782 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:16:19.805 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:16:19.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:16:19.806 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:16:19.907 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:16:19.930 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:16:19.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:16:19.931 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:16:19.953 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:16:19 smithi191 ceph-mon[32884]: from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:16:19.954 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:16:19 smithi191 ceph-mon[32884]: osdmap e152: 8 total, 8 up, 7 in 2024-09-10T19:16:20.031 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:16:20.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:16:20.053 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:16:20.054 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:16:20.092 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:16:19 smithi183 ceph-mon[29736]: from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:16:20.092 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:16:19 smithi183 ceph-mon[29736]: osdmap e152: 8 total, 8 up, 7 in 2024-09-10T19:16:20.092 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:16:19 smithi183 ceph-mon[41562]: from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:16:20.092 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:16:19 smithi183 ceph-mon[41562]: osdmap e152: 8 total, 8 up, 7 in 2024-09-10T19:16:20.155 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:16:20.176 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:16:20.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:16:20.178 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:16:20.278 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:16:20.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:16:20.302 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:16:20.302 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:16:20.403 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:16:20.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:16:20.427 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:16:20.427 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:16:20.528 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:16:20.535 INFO:teuthology.orchestra.run.smithi191.stdout:{ 2024-09-10T19:16:20.536 INFO:teuthology.orchestra.run.smithi191.stdout: "ops": [], 2024-09-10T19:16:20.536 INFO:teuthology.orchestra.run.smithi191.stdout: "complaint_time": 30, 2024-09-10T19:16:20.536 INFO:teuthology.orchestra.run.smithi191.stdout: "num_blocked_ops": 0 2024-09-10T19:16:20.536 INFO:teuthology.orchestra.run.smithi191.stdout:} 2024-09-10T19:16:20.549 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:16:20.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:16:20.550 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:16:20.651 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:16:20.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:16:20.673 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:16:20.673 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:16:20.694 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-10T19:16:20.774 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:16:20.796 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:16:20.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:16:20.797 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:16:20.897 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:16:20.918 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:16:20.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:16:20.919 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:16:20.931 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:16:20 smithi183 ceph-mon[29736]: pgmap v249: 148 pgs: 148 active+clean; 582 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 738 B/s rd, 0 op/s 2024-09-10T19:16:20.931 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:16:20 smithi183 ceph-mon[41562]: pgmap v249: 148 pgs: 148 active+clean; 582 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 738 B/s rd, 0 op/s 2024-09-10T19:16:21.020 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:16:21.043 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:16:21.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:16:21.044 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:16:21.145 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:16:21.168 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:16:21.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:16:21.169 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:16:21.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:16:20 smithi191 ceph-mon[32884]: pgmap v249: 148 pgs: 148 active+clean; 582 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 738 B/s rd, 0 op/s 2024-09-10T19:16:21.261 DEBUG:teuthology.orchestra.run.smithi191:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph daemon osd.4 dump_historic_ops 2024-09-10T19:16:21.270 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:16:21.326 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:16:21.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:16:21.327 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:16:21.428 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:16:21.450 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:16:21.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:16:21.451 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:16:21.507 INFO:teuthology.orchestra.run.smithi191.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.b/config 2024-09-10T19:16:21.508 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Sep 10 19:16:21 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-y[29989]: ::ffff:172.21.15.191 - - [10/Sep/2024:19:16:21] "GET /metrics HTTP/1.1" 200 223836 "" "Prometheus/2.43.0" 2024-09-10T19:16:21.552 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:16:21.574 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:16:21.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:16:21.575 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:16:21.676 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:16:21.698 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:16:21.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:16:21.699 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:16:21.800 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:16:21.822 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:16:21.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:16:21.823 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:16:21.924 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:16:21.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:16:21.946 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:16:21.947 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:16:21.959 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:16:22.048 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:16:22.069 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:16:22.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:16:22.070 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:16:22.170 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:16:22.191 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:16:22.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:16:22.192 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:16:22.293 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:16:22.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:16:22.317 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:16:22.317 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:16:22.418 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:16:22.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:16:22.440 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:16:22.440 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:16:22.541 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:16:22.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:16:22.563 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:16:22.563 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:16:22.664 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:16:22.686 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:16:22.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:16:22.687 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:16:22.788 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:16:22.812 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:16:22.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:16:22.813 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:16:22.820 INFO:teuthology.orchestra.run.smithi183.stderr:noscrub is unset 2024-09-10T19:16:22.821 INFO:teuthology.orchestra.run.smithi183.stderr:enabled application 'rados' on pool 'unique_pool_1' 2024-09-10T19:16:22.913 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:16:22 smithi191 ceph-mon[32884]: pgmap v251: 148 pgs: 148 active+clean; 582 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-09-10T19:16:22.913 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:16:22 smithi191 ceph-mon[32884]: from='client.? 172.21.15.183:0/1169023023' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_1", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:16:22.913 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:16:22 smithi191 ceph-mon[32884]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_1", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:16:22.913 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:16:22 smithi191 ceph-mon[32884]: from='client.? 172.21.15.183:0/934239314' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-10T19:16:22.913 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:16:22 smithi191 ceph-mon[32884]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-10T19:16:22.913 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:16:22.934 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:16:22.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:16:22.935 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:16:23.036 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:16:23.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:16:23.059 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:16:23.059 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:16:23.160 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:16:23.164 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:16:22 smithi183 ceph-mon[41562]: pgmap v251: 148 pgs: 148 active+clean; 582 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-09-10T19:16:23.164 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:16:22 smithi183 ceph-mon[41562]: from='client.? 172.21.15.183:0/1169023023' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_1", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:16:23.164 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:16:22 smithi183 ceph-mon[41562]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_1", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:16:23.164 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:16:22 smithi183 ceph-mon[41562]: from='client.? 172.21.15.183:0/934239314' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-10T19:16:23.164 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:16:22 smithi183 ceph-mon[41562]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-10T19:16:23.164 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:16:22 smithi183 ceph-mon[29736]: pgmap v251: 148 pgs: 148 active+clean; 582 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-09-10T19:16:23.164 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:16:22 smithi183 ceph-mon[29736]: from='client.? 172.21.15.183:0/1169023023' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_1", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:16:23.164 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:16:22 smithi183 ceph-mon[29736]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_1", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:16:23.165 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:16:22 smithi183 ceph-mon[29736]: from='client.? 172.21.15.183:0/934239314' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-10T19:16:23.165 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:16:22 smithi183 ceph-mon[29736]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-10T19:16:23.165 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:16:22 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mon-a[29713]: 2024-09-10T19:16:22.857+0000 7f9f28e29640 -1 mon.a@0(leader).osd e153 definitely_dead 0 2024-09-10T19:16:23.175 INFO:teuthology.orchestra.run.smithi191.stdout:{ 2024-09-10T19:16:23.175 INFO:teuthology.orchestra.run.smithi191.stdout: "size": 20, 2024-09-10T19:16:23.175 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 600, 2024-09-10T19:16:23.175 INFO:teuthology.orchestra.run.smithi191.stdout: "ops": [ 2024-09-10T19:16:23.175 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:23.176 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.14781.0:10576 7.1 7:8a3bdf89:::benchmark_data_smithi183_84160_object10575:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e90)", 2024-09-10T19:16:23.176 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-10T19:12:22.248429+0000", 2024-09-10T19:16:23.176 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 240.89129436900001, 2024-09-10T19:16:23.176 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.062434195999999997, 2024-09-10T19:16:23.176 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-10T19:16:23.176 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T19:16:23.176 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-10T19:16:23.176 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.14781", 2024-09-10T19:16:23.176 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.183:0/2025776420", 2024-09-10T19:16:23.176 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 10576 2024-09-10T19:16:23.176 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:23.176 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-10T19:16:23.176 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:23.177 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-10T19:16:23.177 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:12:22.248429+0000", 2024-09-10T19:16:23.177 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:16:23.177 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:23.177 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:23.177 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-10T19:16:23.177 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:12:22.248429+0000", 2024-09-10T19:16:23.177 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:16:23.177 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:23.177 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:23.177 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-10T19:16:23.177 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:12:22.248428+0000", 2024-09-10T19:16:23.177 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999986 2024-09-10T19:16:23.178 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:23.178 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:23.178 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-10T19:16:23.178 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:12:22.248468+0000", 2024-09-10T19:16:23.178 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.0116e-05 2024-09-10T19:16:23.178 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:23.178 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:23.178 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-10T19:16:23.178 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:12:22.248478+0000", 2024-09-10T19:16:23.178 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.7399999999999999e-06 2024-09-10T19:16:23.178 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:23.178 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:23.178 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-10T19:16:23.178 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:12:22.248482+0000", 2024-09-10T19:16:23.178 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.1840000000000001e-06 2024-09-10T19:16:23.179 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:23.179 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:23.179 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:16:23.179 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:12:22.248520+0000", 2024-09-10T19:16:23.179 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.8549e-05 2024-09-10T19:16:23.179 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:23.179 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:23.179 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-10T19:16:23.179 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:12:22.248616+0000", 2024-09-10T19:16:23.179 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.5959999999999996e-05 2024-09-10T19:16:23.179 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:23.179 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:23.179 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 3,6", 2024-09-10T19:16:23.180 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:12:22.248765+0000", 2024-09-10T19:16:23.180 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000148644 2024-09-10T19:16:23.180 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:23.180 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:23.180 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-09-10T19:16:23.180 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:12:22.249487+0000", 2024-09-10T19:16:23.180 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00072205399999999997 2024-09-10T19:16:23.180 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:23.180 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:23.180 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:16:23.180 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:12:22.310245+0000", 2024-09-10T19:16:23.180 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.060758088000000002 2024-09-10T19:16:23.180 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:23.180 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:23.181 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:16:23.181 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:12:22.310781+0000", 2024-09-10T19:16:23.181 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00053565399999999999 2024-09-10T19:16:23.181 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:23.181 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:23.181 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-09-10T19:16:23.181 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:12:22.310828+0000", 2024-09-10T19:16:23.181 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.7586000000000001e-05 2024-09-10T19:16:23.181 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:23.181 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:23.181 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-10T19:16:23.181 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:12:22.310863+0000", 2024-09-10T19:16:23.182 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.4913000000000003e-05 2024-09-10T19:16:23.182 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:16:23.182 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-10T19:16:23.182 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:16:23.182 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:23.182 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:23.182 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_repop(client.14781.0:10578 7.9 e90/86 7:9e361324:::benchmark_data_smithi183_84160_object10577:head v 90'639, mlcod=90'637)", 2024-09-10T19:16:23.182 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-10T19:12:22.249794+0000", 2024-09-10T19:16:23.182 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 240.88992918599999, 2024-09-10T19:16:23.182 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.062014737, 2024-09-10T19:16:23.182 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-10T19:16:23.182 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T19:16:23.183 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-10T19:16:23.183 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:23.183 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-10T19:16:23.183 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:12:22.249794+0000", 2024-09-10T19:16:23.183 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:16:23.183 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:23.183 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:23.183 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-10T19:16:23.183 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:12:22.249794+0000", 2024-09-10T19:16:23.183 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:16:23.183 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:23.183 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:23.183 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-10T19:16:23.183 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:12:22.249794+0000", 2024-09-10T19:16:23.184 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999995 2024-09-10T19:16:23.184 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:23.184 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:23.184 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-10T19:16:23.184 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:12:22.249853+0000", 2024-09-10T19:16:23.184 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.9342999999999997e-05 2024-09-10T19:16:23.184 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:23.184 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:23.184 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-10T19:16:23.184 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:12:22.249868+0000", 2024-09-10T19:16:23.184 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.4983e-05 2024-09-10T19:16:23.184 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:23.184 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:23.185 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-10T19:16:23.185 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:12:22.249873+0000", 2024-09-10T19:16:23.185 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.31e-06 2024-09-10T19:16:23.185 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:23.185 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:23.185 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:16:23.185 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:12:22.249912+0000", 2024-09-10T19:16:23.185 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.8442999999999998e-05 2024-09-10T19:16:23.185 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:23.185 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:23.185 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-10T19:16:23.185 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:12:22.249931+0000", 2024-09-10T19:16:23.185 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.9210000000000001e-05 2024-09-10T19:16:23.186 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:23.186 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:23.186 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-09-10T19:16:23.186 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:12:22.311776+0000", 2024-09-10T19:16:23.186 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.061844611000000001 2024-09-10T19:16:23.186 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:23.186 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:23.186 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-10T19:16:23.186 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:12:22.311809+0000", 2024-09-10T19:16:23.186 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.3507000000000003e-05 2024-09-10T19:16:23.186 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:16:23.186 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-10T19:16:23.186 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:16:23.186 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:23.187 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:23.187 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.14781.0:17003 7.1 7:8e2df5b7:::benchmark_data_smithi183_84160_object17002:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e90)", 2024-09-10T19:16:23.187 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-10T19:12:24.663404+0000", 2024-09-10T19:16:23.187 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 238.47631949699999, 2024-09-10T19:16:23.187 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.102732874, 2024-09-10T19:16:23.187 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-10T19:16:23.187 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T19:16:23.187 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-10T19:16:23.187 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.14781", 2024-09-10T19:16:23.187 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.183:0/2025776420", 2024-09-10T19:16:23.187 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 17003 2024-09-10T19:16:23.187 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:23.187 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-10T19:16:23.187 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:23.188 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-10T19:16:23.188 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:12:24.663404+0000", 2024-09-10T19:16:23.188 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:16:23.188 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:23.188 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:23.188 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-10T19:16:23.188 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:12:24.663404+0000", 2024-09-10T19:16:23.188 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:16:23.188 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:23.188 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:23.188 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-10T19:16:23.188 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:12:24.663402+0000", 2024-09-10T19:16:23.188 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999981 2024-09-10T19:16:23.188 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:23.189 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:23.189 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-10T19:16:23.189 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:12:24.663466+0000", 2024-09-10T19:16:23.189 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.3664999999999994e-05 2024-09-10T19:16:23.189 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:23.189 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:23.189 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-10T19:16:23.189 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:12:24.663486+0000", 2024-09-10T19:16:23.189 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.9901000000000001e-05 2024-09-10T19:16:23.189 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:23.189 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:23.189 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-10T19:16:23.189 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:12:24.663493+0000", 2024-09-10T19:16:23.190 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.3429999999999998e-06 2024-09-10T19:16:23.190 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:23.190 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:23.190 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:16:23.190 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:12:24.663534+0000", 2024-09-10T19:16:23.190 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.1405000000000001e-05 2024-09-10T19:16:23.190 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:23.190 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:23.190 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-10T19:16:23.190 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:12:24.663654+0000", 2024-09-10T19:16:23.190 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000119912 2024-09-10T19:16:23.190 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:23.190 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:23.191 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 3,6", 2024-09-10T19:16:23.191 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:12:24.663745+0000", 2024-09-10T19:16:23.191 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.0255999999999998e-05 2024-09-10T19:16:23.191 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:23.191 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:23.191 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-09-10T19:16:23.191 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:12:24.664497+0000", 2024-09-10T19:16:23.191 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00075260699999999997 2024-09-10T19:16:23.191 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:23.191 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:23.191 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:16:23.191 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:12:24.664688+0000", 2024-09-10T19:16:23.191 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00019100700000000001 2024-09-10T19:16:23.191 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:23.191 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:23.192 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:16:23.192 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:12:24.766051+0000", 2024-09-10T19:16:23.192 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.101362305 2024-09-10T19:16:23.192 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:23.192 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:23.192 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-09-10T19:16:23.192 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:12:24.766099+0000", 2024-09-10T19:16:23.192 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.8535999999999997e-05 2024-09-10T19:16:23.192 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:23.192 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:23.192 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-10T19:16:23.192 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:12:24.766137+0000", 2024-09-10T19:16:23.192 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.7753999999999998e-05 2024-09-10T19:16:23.193 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:16:23.193 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-10T19:16:23.193 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:16:23.193 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:23.193 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:23.193 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.14781.0:28873 7.1 7:8f02cc65:::benchmark_data_smithi183_84160_object28872:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e91)", 2024-09-10T19:16:23.193 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-10T19:12:30.078511+0000", 2024-09-10T19:16:23.193 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 233.061212211, 2024-09-10T19:16:23.193 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.066771780000000003, 2024-09-10T19:16:23.193 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-10T19:16:23.193 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T19:16:23.193 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-10T19:16:23.193 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.14781", 2024-09-10T19:16:23.194 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.183:0/2025776420", 2024-09-10T19:16:23.194 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 28873 2024-09-10T19:16:23.194 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:23.194 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-10T19:16:23.194 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:23.194 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-10T19:16:23.194 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:12:30.078511+0000", 2024-09-10T19:16:23.194 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:16:23.194 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:23.194 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:23.194 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-10T19:16:23.194 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:12:30.078511+0000", 2024-09-10T19:16:23.194 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:16:23.194 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:23.195 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:23.195 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-10T19:16:23.195 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:12:30.078510+0000", 2024-09-10T19:16:23.195 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999986 2024-09-10T19:16:23.195 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:23.195 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:23.195 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-10T19:16:23.195 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:12:30.078631+0000", 2024-09-10T19:16:23.195 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000121511 2024-09-10T19:16:23.195 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:23.195 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:23.195 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-10T19:16:23.195 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:12:30.078646+0000", 2024-09-10T19:16:23.195 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.4486e-05 2024-09-10T19:16:23.196 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:23.196 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:23.196 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-10T19:16:23.196 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:12:30.078652+0000", 2024-09-10T19:16:23.196 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.7200000000000003e-06 2024-09-10T19:16:23.196 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:23.196 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:23.196 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:16:23.196 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:12:30.078694+0000", 2024-09-10T19:16:23.196 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.2070000000000002e-05 2024-09-10T19:16:23.196 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:23.196 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:23.196 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-10T19:16:23.196 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:12:30.078813+0000", 2024-09-10T19:16:23.197 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000119015 2024-09-10T19:16:23.197 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:23.197 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:23.197 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 3,6", 2024-09-10T19:16:23.197 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:12:30.078927+0000", 2024-09-10T19:16:23.197 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00011421099999999999 2024-09-10T19:16:23.197 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:23.197 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:23.197 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-09-10T19:16:23.197 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:12:30.081144+0000", 2024-09-10T19:16:23.197 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0022165409999999998 2024-09-10T19:16:23.197 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:23.197 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:23.197 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:16:23.198 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:12:30.081556+0000", 2024-09-10T19:16:23.198 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000412886 2024-09-10T19:16:23.198 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:23.198 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:23.198 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:16:23.198 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:12:30.145190+0000", 2024-09-10T19:16:23.198 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.063633217000000006 2024-09-10T19:16:23.198 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:23.198 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:23.198 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-09-10T19:16:23.198 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:12:30.145241+0000", 2024-09-10T19:16:23.198 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.1214000000000001e-05 2024-09-10T19:16:23.198 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:23.198 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:23.198 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-10T19:16:23.199 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:12:30.145283+0000", 2024-09-10T19:16:23.199 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.2197999999999998e-05 2024-09-10T19:16:23.199 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:16:23.199 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-10T19:16:23.199 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:16:23.199 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:23.199 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:23.199 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.14781.0:28875 7.1 7:80ca89d1:::benchmark_data_smithi183_84160_object28874:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e91)", 2024-09-10T19:16:23.199 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-10T19:12:30.079566+0000", 2024-09-10T19:16:23.199 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 233.06015793500001, 2024-09-10T19:16:23.199 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.065839871999999994, 2024-09-10T19:16:23.199 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-10T19:16:23.199 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T19:16:23.199 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-10T19:16:23.200 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.14781", 2024-09-10T19:16:23.200 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.183:0/2025776420", 2024-09-10T19:16:23.200 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 28875 2024-09-10T19:16:23.200 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:23.200 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-10T19:16:23.200 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:23.200 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-10T19:16:23.200 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:12:30.079566+0000", 2024-09-10T19:16:23.200 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:16:23.200 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:23.200 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:23.200 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-10T19:16:23.200 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:12:30.079566+0000", 2024-09-10T19:16:23.200 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:16:23.200 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:23.201 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:23.201 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-10T19:16:23.201 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:12:30.079565+0000", 2024-09-10T19:16:23.201 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.999999 2024-09-10T19:16:23.201 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:23.201 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:23.201 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-10T19:16:23.201 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:12:30.079595+0000", 2024-09-10T19:16:23.201 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.9901000000000001e-05 2024-09-10T19:16:23.201 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:23.201 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:23.201 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-10T19:16:23.201 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:12:30.079603+0000", 2024-09-10T19:16:23.201 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.8419999999999994e-06 2024-09-10T19:16:23.202 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:23.202 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:23.202 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-10T19:16:23.202 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:12:30.079608+0000", 2024-09-10T19:16:23.202 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.9840000000000001e-06 2024-09-10T19:16:23.202 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:23.202 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:23.202 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:16:23.202 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:12:30.079952+0000", 2024-09-10T19:16:23.202 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000343727 2024-09-10T19:16:23.202 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:23.202 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:23.202 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-10T19:16:23.203 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:12:30.080074+0000", 2024-09-10T19:16:23.203 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00012138600000000001 2024-09-10T19:16:23.203 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:23.203 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:23.203 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 3,6", 2024-09-10T19:16:23.203 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:12:30.080210+0000", 2024-09-10T19:16:23.203 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00013626600000000001 2024-09-10T19:16:23.203 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:23.203 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:23.203 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-09-10T19:16:23.203 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:12:30.081264+0000", 2024-09-10T19:16:23.203 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0010537039999999999 2024-09-10T19:16:23.203 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:23.204 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:23.204 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:16:23.204 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:12:30.081652+0000", 2024-09-10T19:16:23.204 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00038885500000000002 2024-09-10T19:16:23.204 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:23.204 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:23.204 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:16:23.204 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:12:30.145326+0000", 2024-09-10T19:16:23.204 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.063673599999999997 2024-09-10T19:16:23.204 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:23.204 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:23.204 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-09-10T19:16:23.204 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:12:30.145371+0000", 2024-09-10T19:16:23.204 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.5358999999999997e-05 2024-09-10T19:16:23.205 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:23.205 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:23.205 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-10T19:16:23.205 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:12:30.145405+0000", 2024-09-10T19:16:23.205 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.4050000000000001e-05 2024-09-10T19:16:23.205 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:16:23.205 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-10T19:16:23.205 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:16:23.205 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:23.205 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:23.205 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.14781.0:80715 7.1 7:87e1cb44:::benchmark_data_smithi183_84160_object80714:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e104)", 2024-09-10T19:16:23.205 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-10T19:13:02.461602+0000", 2024-09-10T19:16:23.205 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 200.67812115000001, 2024-09-10T19:16:23.206 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.062135136000000001, 2024-09-10T19:16:23.206 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-10T19:16:23.206 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T19:16:23.206 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-10T19:16:23.206 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.14781", 2024-09-10T19:16:23.206 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.183:0/2025776420", 2024-09-10T19:16:23.206 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 80715 2024-09-10T19:16:23.206 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:23.206 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-10T19:16:23.206 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:23.206 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-10T19:16:23.206 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:13:02.461602+0000", 2024-09-10T19:16:23.206 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:16:23.206 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:23.207 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:23.207 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-10T19:16:23.207 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:13:02.461602+0000", 2024-09-10T19:16:23.207 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:16:23.207 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:23.207 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:23.207 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-10T19:16:23.207 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:13:02.461593+0000", 2024-09-10T19:16:23.207 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999909 2024-09-10T19:16:23.207 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:23.207 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:23.207 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-10T19:16:23.207 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:13:02.461635+0000", 2024-09-10T19:16:23.207 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.2447000000000002e-05 2024-09-10T19:16:23.208 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:23.208 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:23.208 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-10T19:16:23.208 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:13:02.461650+0000", 2024-09-10T19:16:23.208 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.4473e-05 2024-09-10T19:16:23.208 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:23.208 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:23.208 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-10T19:16:23.208 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:13:02.461656+0000", 2024-09-10T19:16:23.208 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.4799999999999998e-06 2024-09-10T19:16:23.208 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:23.208 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:23.208 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:16:23.209 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:13:02.461706+0000", 2024-09-10T19:16:23.209 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.9681000000000001e-05 2024-09-10T19:16:23.209 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:23.209 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:23.209 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-10T19:16:23.209 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:13:02.461847+0000", 2024-09-10T19:16:23.209 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000140798 2024-09-10T19:16:23.209 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:23.209 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:23.209 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 3,6", 2024-09-10T19:16:23.209 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:13:02.461984+0000", 2024-09-10T19:16:23.209 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000137508 2024-09-10T19:16:23.209 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:23.209 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:23.210 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-09-10T19:16:23.210 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:13:02.466371+0000", 2024-09-10T19:16:23.210 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0043867760000000002 2024-09-10T19:16:23.210 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:23.210 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:23.210 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:16:23.210 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:13:02.468835+0000", 2024-09-10T19:16:23.210 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0024639990000000001 2024-09-10T19:16:23.210 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:23.210 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:23.210 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:16:23.210 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:13:02.523643+0000", 2024-09-10T19:16:23.210 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.054807907000000003 2024-09-10T19:16:23.210 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:23.210 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:23.211 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-09-10T19:16:23.211 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:13:02.523697+0000", 2024-09-10T19:16:23.211 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.3776000000000001e-05 2024-09-10T19:16:23.211 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:23.211 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:23.211 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-10T19:16:23.211 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:13:02.523737+0000", 2024-09-10T19:16:23.211 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.0581999999999999e-05 2024-09-10T19:16:23.211 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:16:23.211 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-10T19:16:23.211 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:16:23.211 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:23.212 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:23.212 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.14781.0:97069 7.1 7:8807335e:::benchmark_data_smithi183_84160_object97068:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105)", 2024-09-10T19:16:23.212 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-10T19:13:14.419582+0000", 2024-09-10T19:16:23.212 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 188.72014121500001, 2024-09-10T19:16:23.212 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.070188834000000005, 2024-09-10T19:16:23.212 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-10T19:16:23.212 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T19:16:23.212 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-10T19:16:23.212 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.14781", 2024-09-10T19:16:23.212 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.183:0/2025776420", 2024-09-10T19:16:23.213 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 97069 2024-09-10T19:16:23.213 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:23.213 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-10T19:16:23.213 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:23.213 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-10T19:16:23.213 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:13:14.419582+0000", 2024-09-10T19:16:23.213 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:16:23.213 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:23.213 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:23.213 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-10T19:16:23.213 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:13:14.419582+0000", 2024-09-10T19:16:23.213 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:16:23.213 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:23.213 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:23.214 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-10T19:16:23.214 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:13:14.419581+0000", 2024-09-10T19:16:23.214 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999986 2024-09-10T19:16:23.214 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:23.214 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:23.214 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-10T19:16:23.214 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:13:14.419614+0000", 2024-09-10T19:16:23.214 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.2987000000000001e-05 2024-09-10T19:16:23.214 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:23.214 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:23.214 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-10T19:16:23.214 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:13:14.419627+0000", 2024-09-10T19:16:23.214 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.3012e-05 2024-09-10T19:16:23.215 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:23.215 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:23.215 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-10T19:16:23.215 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:13:14.419633+0000", 2024-09-10T19:16:23.215 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.5740000000000003e-06 2024-09-10T19:16:23.215 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:23.215 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:23.215 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:16:23.215 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:13:14.423132+0000", 2024-09-10T19:16:23.215 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0034997549999999998 2024-09-10T19:16:23.215 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:23.215 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:23.215 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-10T19:16:23.215 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:13:14.423253+0000", 2024-09-10T19:16:23.216 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000121107 2024-09-10T19:16:23.216 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:23.216 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:23.216 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 3,6", 2024-09-10T19:16:23.216 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:13:14.423416+0000", 2024-09-10T19:16:23.216 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00016296999999999999 2024-09-10T19:16:23.216 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:23.216 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:23.216 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:16:23.216 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:13:14.426347+0000", 2024-09-10T19:16:23.216 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0029302379999999999 2024-09-10T19:16:23.216 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:23.216 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:23.217 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-09-10T19:16:23.217 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:13:14.428347+0000", 2024-09-10T19:16:23.217 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0020002000000000002 2024-09-10T19:16:23.217 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:23.217 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:23.217 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:16:23.217 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:13:14.489697+0000", 2024-09-10T19:16:23.217 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.061350276000000002 2024-09-10T19:16:23.217 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:23.217 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:23.217 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-09-10T19:16:23.217 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:13:14.489735+0000", 2024-09-10T19:16:23.217 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.8306999999999998e-05 2024-09-10T19:16:23.218 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:23.218 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:23.218 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-10T19:16:23.218 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:13:14.489771+0000", 2024-09-10T19:16:23.218 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.5645000000000002e-05 2024-09-10T19:16:23.218 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:16:23.218 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-10T19:16:23.218 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:16:23.218 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:23.218 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:23.218 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.14781.0:134758 7.1 7:85322e5e:::benchmark_data_smithi183_84160_object134757:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e112)", 2024-09-10T19:16:23.218 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-10T19:13:43.674481+0000", 2024-09-10T19:16:23.218 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 159.465241996, 2024-09-10T19:16:23.218 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.072535856999999995, 2024-09-10T19:16:23.219 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-10T19:16:23.219 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T19:16:23.219 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-10T19:16:23.219 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.14781", 2024-09-10T19:16:23.219 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.183:0/2025776420", 2024-09-10T19:16:23.219 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 134758 2024-09-10T19:16:23.219 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:23.219 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-10T19:16:23.219 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:23.219 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-10T19:16:23.219 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:13:43.674481+0000", 2024-09-10T19:16:23.219 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:16:23.219 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:23.219 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:23.220 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-10T19:16:23.220 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:13:43.674481+0000", 2024-09-10T19:16:23.220 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:16:23.220 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:23.220 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:23.220 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-10T19:16:23.220 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:13:43.674480+0000", 2024-09-10T19:16:23.220 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999986 2024-09-10T19:16:23.220 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:23.220 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:23.220 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-10T19:16:23.220 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:13:43.674535+0000", 2024-09-10T19:16:23.220 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.5655999999999997e-05 2024-09-10T19:16:23.221 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:23.221 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:23.221 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-10T19:16:23.221 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:13:43.719209+0000", 2024-09-10T19:16:23.221 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.044673894999999998 2024-09-10T19:16:23.221 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:23.221 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:23.221 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-10T19:16:23.221 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:13:43.719226+0000", 2024-09-10T19:16:23.221 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.6667000000000002e-05 2024-09-10T19:16:23.221 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:23.221 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:23.221 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:16:23.222 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:13:43.726657+0000", 2024-09-10T19:16:23.222 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0074307660000000001 2024-09-10T19:16:23.222 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:23.222 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:23.222 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-10T19:16:23.222 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:13:43.726778+0000", 2024-09-10T19:16:23.222 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000121317 2024-09-10T19:16:23.222 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:23.222 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:23.222 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 3,6", 2024-09-10T19:16:23.222 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:13:43.726954+0000", 2024-09-10T19:16:23.222 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00017615400000000001 2024-09-10T19:16:23.223 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:23.223 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:23.223 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:16:23.223 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:13:43.738987+0000", 2024-09-10T19:16:23.223 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.012032315 2024-09-10T19:16:23.223 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:23.223 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:23.223 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-09-10T19:16:23.223 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:13:43.738993+0000", 2024-09-10T19:16:23.223 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.7399999999999998e-06 2024-09-10T19:16:23.223 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:23.223 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:23.223 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:16:23.223 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:13:43.746962+0000", 2024-09-10T19:16:23.224 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0079682099999999999 2024-09-10T19:16:23.224 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:23.224 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:23.224 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-09-10T19:16:23.224 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:13:43.746994+0000", 2024-09-10T19:16:23.224 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.2738999999999999e-05 2024-09-10T19:16:23.224 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:23.224 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:23.224 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-10T19:16:23.224 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:13:43.747017+0000", 2024-09-10T19:16:23.224 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.3050000000000001e-05 2024-09-10T19:16:23.224 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:16:23.224 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-10T19:16:23.224 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:16:23.225 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:23.225 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:23.225 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.14781.0:158967 7.1 7:843591f8:::benchmark_data_smithi183_84160_object18586:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119)", 2024-09-10T19:16:23.225 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-10T19:14:05.593183+0000", 2024-09-10T19:16:23.225 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 137.54654034199999, 2024-09-10T19:16:23.225 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.062059778000000003, 2024-09-10T19:16:23.225 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-10T19:16:23.225 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T19:16:23.225 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-10T19:16:23.225 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.14781", 2024-09-10T19:16:23.225 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.183:0/2025776420", 2024-09-10T19:16:23.225 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 158967 2024-09-10T19:16:23.225 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:23.225 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-10T19:16:23.225 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:23.226 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-10T19:16:23.226 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:05.593183+0000", 2024-09-10T19:16:23.226 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:16:23.226 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:23.226 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:23.226 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-10T19:16:23.226 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:05.593183+0000", 2024-09-10T19:16:23.226 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:16:23.226 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:23.226 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:23.226 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-10T19:16:23.226 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:05.593183+0000", 2024-09-10T19:16:23.226 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999995 2024-09-10T19:16:23.226 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:23.226 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:23.227 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-10T19:16:23.227 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:05.593186+0000", 2024-09-10T19:16:23.227 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.6370000000000001e-06 2024-09-10T19:16:23.227 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:23.227 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:23.227 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-10T19:16:23.227 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:05.593193+0000", 2024-09-10T19:16:23.227 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.0539999999999997e-06 2024-09-10T19:16:23.227 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:23.227 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:23.227 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-10T19:16:23.227 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:05.593198+0000", 2024-09-10T19:16:23.227 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.4599999999999996e-06 2024-09-10T19:16:23.227 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:23.228 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:23.228 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:16:23.228 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:05.595623+0000", 2024-09-10T19:16:23.228 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0024256719999999998 2024-09-10T19:16:23.228 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:23.228 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:23.228 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-10T19:16:23.228 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:05.595695+0000", 2024-09-10T19:16:23.228 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.1674000000000005e-05 2024-09-10T19:16:23.228 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:23.228 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:23.228 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 3,6", 2024-09-10T19:16:23.228 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:05.595786+0000", 2024-09-10T19:16:23.229 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.1378000000000005e-05 2024-09-10T19:16:23.229 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:23.229 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:23.229 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-09-10T19:16:23.229 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:05.597641+0000", 2024-09-10T19:16:23.229 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0018544029999999999 2024-09-10T19:16:23.229 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:23.229 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:23.229 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:16:23.229 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:05.597730+0000", 2024-09-10T19:16:23.229 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.9190000000000005e-05 2024-09-10T19:16:23.229 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:23.229 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:23.229 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:16:23.230 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:05.655168+0000", 2024-09-10T19:16:23.230 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.057438305000000002 2024-09-10T19:16:23.230 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:23.230 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:23.230 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-09-10T19:16:23.230 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:05.655209+0000", 2024-09-10T19:16:23.230 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.0210000000000003e-05 2024-09-10T19:16:23.230 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:23.230 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:23.230 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-10T19:16:23.230 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:05.655243+0000", 2024-09-10T19:16:23.230 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.4384000000000001e-05 2024-09-10T19:16:23.230 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:16:23.230 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-10T19:16:23.231 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:16:23.231 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:23.231 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:23.231 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.14781.0:159152 7.1 7:83691580:::benchmark_data_smithi183_84160_object18771:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119)", 2024-09-10T19:16:23.231 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-10T19:14:05.777489+0000", 2024-09-10T19:16:23.231 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 137.36223465099999, 2024-09-10T19:16:23.231 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.139687582, 2024-09-10T19:16:23.231 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-10T19:16:23.231 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T19:16:23.231 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-10T19:16:23.231 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.14781", 2024-09-10T19:16:23.231 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.183:0/2025776420", 2024-09-10T19:16:23.231 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 159152 2024-09-10T19:16:23.232 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:23.232 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-10T19:16:23.232 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:23.232 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-10T19:16:23.232 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:05.777489+0000", 2024-09-10T19:16:23.232 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:16:23.232 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:23.232 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:23.232 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-10T19:16:23.232 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:05.777489+0000", 2024-09-10T19:16:23.232 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:16:23.232 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:23.232 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:23.233 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-10T19:16:23.233 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:05.777483+0000", 2024-09-10T19:16:23.233 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999943 2024-09-10T19:16:23.233 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:23.233 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:23.233 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-10T19:16:23.233 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:05.777494+0000", 2024-09-10T19:16:23.233 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.0413e-05 2024-09-10T19:16:23.233 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:23.233 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:23.233 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-10T19:16:23.234 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:05.777509+0000", 2024-09-10T19:16:23.234 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.5716999999999999e-05 2024-09-10T19:16:23.234 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:23.234 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:23.234 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-10T19:16:23.234 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:05.777516+0000", 2024-09-10T19:16:23.234 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.1700000000000002e-06 2024-09-10T19:16:23.234 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:23.234 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:23.234 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:16:23.234 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:05.777578+0000", 2024-09-10T19:16:23.234 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.2279000000000005e-05 2024-09-10T19:16:23.234 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:23.234 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:23.235 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-10T19:16:23.235 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:05.777676+0000", 2024-09-10T19:16:23.235 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.8061999999999997e-05 2024-09-10T19:16:23.235 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:23.235 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:23.235 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 3,6", 2024-09-10T19:16:23.235 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:05.777740+0000", 2024-09-10T19:16:23.235 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.4299000000000002e-05 2024-09-10T19:16:23.235 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:23.235 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:23.235 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:16:23.235 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:05.780719+0000", 2024-09-10T19:16:23.235 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.002978957 2024-09-10T19:16:23.236 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:23.236 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:23.236 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-09-10T19:16:23.236 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:05.783009+0000", 2024-09-10T19:16:23.236 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0022895150000000002 2024-09-10T19:16:23.236 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:23.236 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:23.236 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:16:23.236 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:05.917095+0000", 2024-09-10T19:16:23.236 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.134086283 2024-09-10T19:16:23.236 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:23.236 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:23.236 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-09-10T19:16:23.236 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:05.917140+0000", 2024-09-10T19:16:23.236 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.5303999999999999e-05 2024-09-10T19:16:23.237 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:23.237 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:23.237 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-10T19:16:23.237 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:05.917176+0000", 2024-09-10T19:16:23.237 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.6087999999999997e-05 2024-09-10T19:16:23.237 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:16:23.237 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-10T19:16:23.237 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:16:23.237 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:23.237 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:23.237 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.14781.0:159153 7.1 7:869d918e:::benchmark_data_smithi183_84160_object18772:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119)", 2024-09-10T19:16:23.237 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-10T19:14:05.780200+0000", 2024-09-10T19:16:23.237 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 137.35952367100001, 2024-09-10T19:16:23.237 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.13710039299999999, 2024-09-10T19:16:23.237 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-10T19:16:23.238 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T19:16:23.238 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-10T19:16:23.238 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.14781", 2024-09-10T19:16:23.238 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.183:0/2025776420", 2024-09-10T19:16:23.238 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 159153 2024-09-10T19:16:23.238 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:23.238 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-10T19:16:23.238 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:23.238 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-10T19:16:23.238 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:05.780200+0000", 2024-09-10T19:16:23.238 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:16:23.238 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:23.238 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:23.238 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-10T19:16:23.238 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:05.780200+0000", 2024-09-10T19:16:23.239 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:16:23.239 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:23.239 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:23.239 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-10T19:16:23.239 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:05.780199+0000", 2024-09-10T19:16:23.239 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999986 2024-09-10T19:16:23.239 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:23.239 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:23.239 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-10T19:16:23.239 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:05.780204+0000", 2024-09-10T19:16:23.239 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.8429999999999998e-06 2024-09-10T19:16:23.239 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:23.239 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:23.239 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-10T19:16:23.239 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:05.780216+0000", 2024-09-10T19:16:23.240 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.1992e-05 2024-09-10T19:16:23.240 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:23.240 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:23.240 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-10T19:16:23.240 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:05.780222+0000", 2024-09-10T19:16:23.240 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.7270000000000004e-06 2024-09-10T19:16:23.240 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:23.240 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:23.240 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:16:23.240 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:05.780740+0000", 2024-09-10T19:16:23.240 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00051840299999999996 2024-09-10T19:16:23.240 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:23.240 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:23.240 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-10T19:16:23.240 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:05.780856+0000", 2024-09-10T19:16:23.240 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000115924 2024-09-10T19:16:23.241 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:23.241 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:23.241 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 3,6", 2024-09-10T19:16:23.241 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:05.780935+0000", 2024-09-10T19:16:23.241 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.8712000000000001e-05 2024-09-10T19:16:23.241 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:23.241 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:23.241 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-09-10T19:16:23.241 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:05.783108+0000", 2024-09-10T19:16:23.241 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.002172509 2024-09-10T19:16:23.241 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:23.241 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:23.241 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:16:23.241 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:05.783314+0000", 2024-09-10T19:16:23.241 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000206057 2024-09-10T19:16:23.242 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:23.242 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:23.242 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:16:23.242 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:05.917215+0000", 2024-09-10T19:16:23.242 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.13390153599999999 2024-09-10T19:16:23.242 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:23.242 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:23.242 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-09-10T19:16:23.242 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:05.917264+0000", 2024-09-10T19:16:23.242 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.9257e-05 2024-09-10T19:16:23.242 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:23.242 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:23.242 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-10T19:16:23.242 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:05.917300+0000", 2024-09-10T19:16:23.242 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.5692000000000003e-05 2024-09-10T19:16:23.243 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:16:23.243 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-10T19:16:23.243 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:16:23.243 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:23.243 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:23.243 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.14781.0:159506 7.1 7:88a63a99:::benchmark_data_smithi183_84160_object19125:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119)", 2024-09-10T19:16:23.243 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-10T19:14:06.161820+0000", 2024-09-10T19:16:23.243 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 136.977903424, 2024-09-10T19:16:23.243 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.10357348700000001, 2024-09-10T19:16:23.243 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-10T19:16:23.243 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T19:16:23.243 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-10T19:16:23.243 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.14781", 2024-09-10T19:16:23.243 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.183:0/2025776420", 2024-09-10T19:16:23.243 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 159506 2024-09-10T19:16:23.244 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:23.244 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-10T19:16:23.244 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:23.244 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-10T19:16:23.244 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:06.161820+0000", 2024-09-10T19:16:23.244 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:16:23.244 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:23.244 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:23.244 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-10T19:16:23.244 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:06.161820+0000", 2024-09-10T19:16:23.244 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:16:23.244 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:23.244 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:23.244 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-10T19:16:23.245 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:06.161819+0000", 2024-09-10T19:16:23.245 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999986 2024-09-10T19:16:23.245 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:23.245 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:23.245 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-10T19:16:23.245 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:06.161825+0000", 2024-09-10T19:16:23.245 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.8360000000000003e-06 2024-09-10T19:16:23.245 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:23.245 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:23.245 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-10T19:16:23.245 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:06.161836+0000", 2024-09-10T19:16:23.245 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.0671e-05 2024-09-10T19:16:23.245 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:23.245 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:23.245 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-10T19:16:23.246 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:06.161842+0000", 2024-09-10T19:16:23.246 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.4040000000000003e-06 2024-09-10T19:16:23.246 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:23.246 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:23.246 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:16:23.246 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:06.161984+0000", 2024-09-10T19:16:23.246 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000142244 2024-09-10T19:16:23.246 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:23.246 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:23.246 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-10T19:16:23.246 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:06.162051+0000", 2024-09-10T19:16:23.246 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.7615999999999998e-05 2024-09-10T19:16:23.246 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:23.246 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:23.247 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 3,6", 2024-09-10T19:16:23.247 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:06.162120+0000", 2024-09-10T19:16:23.247 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.9103000000000006e-05 2024-09-10T19:16:23.247 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:23.247 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:23.247 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-09-10T19:16:23.247 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:06.165083+0000", 2024-09-10T19:16:23.247 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.002962868 2024-09-10T19:16:23.247 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:23.247 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:23.247 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:16:23.247 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:06.165196+0000", 2024-09-10T19:16:23.247 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00011301 2024-09-10T19:16:23.247 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:23.247 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:23.248 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:16:23.248 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:06.265255+0000", 2024-09-10T19:16:23.248 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.10005902 2024-09-10T19:16:23.248 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:23.248 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:23.248 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-09-10T19:16:23.248 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:06.265350+0000", 2024-09-10T19:16:23.248 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.4949000000000002e-05 2024-09-10T19:16:23.248 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:23.248 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:23.248 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-10T19:16:23.248 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:06.265394+0000", 2024-09-10T19:16:23.248 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.3180999999999999e-05 2024-09-10T19:16:23.248 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:16:23.248 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-10T19:16:23.249 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:16:23.249 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:23.249 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:23.249 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.14781.0:159514 7.1 7:88e04569:::benchmark_data_smithi183_84160_object19133:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119)", 2024-09-10T19:16:23.249 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-10T19:14:06.166005+0000", 2024-09-10T19:16:23.249 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 136.97371875900001, 2024-09-10T19:16:23.249 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.099527924000000004, 2024-09-10T19:16:23.249 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-10T19:16:23.249 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T19:16:23.249 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-10T19:16:23.249 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.14781", 2024-09-10T19:16:23.249 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.183:0/2025776420", 2024-09-10T19:16:23.249 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 159514 2024-09-10T19:16:23.249 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:23.249 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-10T19:16:23.250 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:23.250 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-10T19:16:23.250 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:06.166005+0000", 2024-09-10T19:16:23.250 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:16:23.250 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:23.250 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:23.250 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-10T19:16:23.250 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:06.166005+0000", 2024-09-10T19:16:23.250 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:16:23.250 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:23.250 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:23.250 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-10T19:16:23.250 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:06.166003+0000", 2024-09-10T19:16:23.250 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999986 2024-09-10T19:16:23.250 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:23.251 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:23.251 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-10T19:16:23.251 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:06.166009+0000", 2024-09-10T19:16:23.251 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.6629999999999998e-06 2024-09-10T19:16:23.251 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:23.251 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:23.251 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-10T19:16:23.251 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:06.166024+0000", 2024-09-10T19:16:23.251 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.4577e-05 2024-09-10T19:16:23.251 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:23.251 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:23.251 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-10T19:16:23.251 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:06.166030+0000", 2024-09-10T19:16:23.251 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.0290000000000002e-06 2024-09-10T19:16:23.251 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:23.252 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:23.252 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:16:23.252 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:06.166084+0000", 2024-09-10T19:16:23.252 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.4799999999999997e-05 2024-09-10T19:16:23.252 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:23.252 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:23.252 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-10T19:16:23.252 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:06.166201+0000", 2024-09-10T19:16:23.252 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00011670100000000001 2024-09-10T19:16:23.252 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:23.252 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:23.252 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 3,6", 2024-09-10T19:16:23.252 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:06.166304+0000", 2024-09-10T19:16:23.252 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000102804 2024-09-10T19:16:23.253 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:23.253 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:23.253 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-09-10T19:16:23.253 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:06.172095+0000", 2024-09-10T19:16:23.253 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0057914250000000002 2024-09-10T19:16:23.253 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:23.253 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:23.253 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:16:23.253 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:06.172540+0000", 2024-09-10T19:16:23.253 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00044485299999999999 2024-09-10T19:16:23.253 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:23.253 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:23.253 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:16:23.253 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:06.265431+0000", 2024-09-10T19:16:23.254 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.092891216999999998 2024-09-10T19:16:23.254 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:23.254 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:23.254 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-09-10T19:16:23.254 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:06.265499+0000", 2024-09-10T19:16:23.254 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.7928000000000004e-05 2024-09-10T19:16:23.254 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:23.254 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:23.254 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-10T19:16:23.254 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:06.265533+0000", 2024-09-10T19:16:23.254 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.3288999999999999e-05 2024-09-10T19:16:23.254 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:16:23.254 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-10T19:16:23.254 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:16:23.254 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:23.255 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:23.255 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.14781.0:159516 7.1 7:87d2a96f:::benchmark_data_smithi183_84160_object19135:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119)", 2024-09-10T19:16:23.255 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-10T19:14:06.169298+0000", 2024-09-10T19:16:23.255 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 136.97042543500001, 2024-09-10T19:16:23.255 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.096447129000000006, 2024-09-10T19:16:23.255 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-10T19:16:23.255 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T19:16:23.255 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-10T19:16:23.255 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.14781", 2024-09-10T19:16:23.255 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.183:0/2025776420", 2024-09-10T19:16:23.255 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 159516 2024-09-10T19:16:23.255 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:23.255 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-10T19:16:23.255 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:23.255 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-10T19:16:23.256 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:06.169298+0000", 2024-09-10T19:16:23.256 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:16:23.256 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:23.256 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:23.256 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-10T19:16:23.256 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:06.169298+0000", 2024-09-10T19:16:23.256 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:16:23.256 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:23.256 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:23.256 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-10T19:16:23.256 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:06.169297+0000", 2024-09-10T19:16:23.256 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999986 2024-09-10T19:16:23.256 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:23.256 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:23.257 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-10T19:16:23.257 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:06.169304+0000", 2024-09-10T19:16:23.257 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.2150000000000004e-06 2024-09-10T19:16:23.257 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:23.257 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:23.257 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-10T19:16:23.257 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:06.169321+0000", 2024-09-10T19:16:23.257 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.7370000000000001e-05 2024-09-10T19:16:23.257 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:23.257 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:23.257 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-10T19:16:23.257 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:06.169327+0000", 2024-09-10T19:16:23.257 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.3160000000000002e-06 2024-09-10T19:16:23.258 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:23.258 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:23.258 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:16:23.258 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:06.169433+0000", 2024-09-10T19:16:23.258 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00010554 2024-09-10T19:16:23.258 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:23.258 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:23.258 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-10T19:16:23.258 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:06.169544+0000", 2024-09-10T19:16:23.258 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000110752 2024-09-10T19:16:23.258 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:23.258 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:23.258 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 3,6", 2024-09-10T19:16:23.258 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:06.169670+0000", 2024-09-10T19:16:23.259 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00012592500000000001 2024-09-10T19:16:23.259 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:23.259 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:23.259 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-09-10T19:16:23.259 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:06.172191+0000", 2024-09-10T19:16:23.259 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.002521527 2024-09-10T19:16:23.259 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:23.259 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:23.259 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:16:23.259 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:06.172662+0000", 2024-09-10T19:16:23.259 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00047115299999999998 2024-09-10T19:16:23.259 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:23.259 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:23.259 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:16:23.259 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:06.265580+0000", 2024-09-10T19:16:23.260 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.092917729000000004 2024-09-10T19:16:23.260 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:23.260 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:23.260 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-09-10T19:16:23.260 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:06.265649+0000", 2024-09-10T19:16:23.260 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.8658000000000006e-05 2024-09-10T19:16:23.260 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:23.260 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:23.260 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-10T19:16:23.260 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:06.265745+0000", 2024-09-10T19:16:23.260 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.6409000000000005e-05 2024-09-10T19:16:23.260 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:16:23.260 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-10T19:16:23.260 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:16:23.260 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:23.261 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:23.261 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.14781.0:159734 7.1 7:80cfc93c:::benchmark_data_smithi183_84160_object19353:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e119)", 2024-09-10T19:16:23.261 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-10T19:14:06.405243+0000", 2024-09-10T19:16:23.261 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 136.73448007600001, 2024-09-10T19:16:23.261 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.095095132999999998, 2024-09-10T19:16:23.261 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-10T19:16:23.261 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T19:16:23.261 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-10T19:16:23.261 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.14781", 2024-09-10T19:16:23.261 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.183:0/2025776420", 2024-09-10T19:16:23.261 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 159734 2024-09-10T19:16:23.261 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:23.261 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-10T19:16:23.261 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:23.261 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-10T19:16:23.262 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:06.405243+0000", 2024-09-10T19:16:23.262 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:16:23.262 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:23.262 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:23.262 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-10T19:16:23.262 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:06.405243+0000", 2024-09-10T19:16:23.262 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:16:23.262 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:23.262 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:23.262 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-10T19:16:23.262 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:06.405241+0000", 2024-09-10T19:16:23.262 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999981 2024-09-10T19:16:23.262 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:23.262 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:23.262 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-10T19:16:23.263 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:06.405249+0000", 2024-09-10T19:16:23.263 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.6350000000000006e-06 2024-09-10T19:16:23.263 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:23.263 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:23.263 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-10T19:16:23.263 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:06.405262+0000", 2024-09-10T19:16:23.263 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.3115000000000001e-05 2024-09-10T19:16:23.263 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:23.263 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:23.263 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-10T19:16:23.263 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:06.405269+0000", 2024-09-10T19:16:23.263 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.6549999999999998e-06 2024-09-10T19:16:23.263 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:23.263 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:23.263 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:16:23.264 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:06.405310+0000", 2024-09-10T19:16:23.264 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.1298999999999999e-05 2024-09-10T19:16:23.264 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:23.264 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:23.264 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-10T19:16:23.264 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:06.405438+0000", 2024-09-10T19:16:23.264 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00012805600000000001 2024-09-10T19:16:23.264 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:23.264 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:23.264 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 3,6", 2024-09-10T19:16:23.264 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:06.405549+0000", 2024-09-10T19:16:23.264 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000111142 2024-09-10T19:16:23.264 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:23.264 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:23.264 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-09-10T19:16:23.265 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:06.408263+0000", 2024-09-10T19:16:23.265 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0027137239999999998 2024-09-10T19:16:23.265 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:23.265 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:23.265 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:16:23.265 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:06.408708+0000", 2024-09-10T19:16:23.265 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00044482099999999998 2024-09-10T19:16:23.265 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:23.265 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:23.265 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:16:23.265 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:06.500210+0000", 2024-09-10T19:16:23.265 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.091502264999999999 2024-09-10T19:16:23.265 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:23.265 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:23.265 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-09-10T19:16:23.266 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:06.500298+0000", 2024-09-10T19:16:23.266 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.7968999999999998e-05 2024-09-10T19:16:23.266 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:23.266 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:23.266 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-10T19:16:23.266 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:06.500339+0000", 2024-09-10T19:16:23.266 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.0398000000000002e-05 2024-09-10T19:16:23.266 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:16:23.266 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-10T19:16:23.266 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:16:23.266 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:23.266 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:23.266 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.14781.0:161360 7.1 7:871a6582:::benchmark_data_smithi183_84160_object20979:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120)", 2024-09-10T19:16:23.266 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-10T19:14:07.740391+0000", 2024-09-10T19:16:23.266 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 135.39933295099999, 2024-09-10T19:16:23.267 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.14222979999999999, 2024-09-10T19:16:23.267 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-10T19:16:23.267 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T19:16:23.267 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-10T19:16:23.267 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.14781", 2024-09-10T19:16:23.267 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.183:0/2025776420", 2024-09-10T19:16:23.267 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 161360 2024-09-10T19:16:23.267 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:23.267 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-10T19:16:23.267 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:23.267 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-10T19:16:23.267 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:07.740391+0000", 2024-09-10T19:16:23.267 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:16:23.267 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:23.267 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:23.268 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-10T19:16:23.268 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:07.740391+0000", 2024-09-10T19:16:23.268 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:16:23.268 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:23.268 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:23.268 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-10T19:16:23.268 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:07.740389+0000", 2024-09-10T19:16:23.268 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.999999 2024-09-10T19:16:23.268 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:23.268 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:23.268 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-10T19:16:23.268 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:07.740395+0000", 2024-09-10T19:16:23.268 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.8660000000000002e-06 2024-09-10T19:16:23.269 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:23.269 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:23.269 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-10T19:16:23.269 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:07.740407+0000", 2024-09-10T19:16:23.269 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.1385e-05 2024-09-10T19:16:23.269 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:23.269 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:23.269 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-10T19:16:23.269 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:07.740411+0000", 2024-09-10T19:16:23.269 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.3100000000000002e-06 2024-09-10T19:16:23.269 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:23.269 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:23.269 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:16:23.269 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:07.740447+0000", 2024-09-10T19:16:23.269 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.5796000000000001e-05 2024-09-10T19:16:23.270 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:23.270 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:23.270 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-10T19:16:23.270 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:07.740524+0000", 2024-09-10T19:16:23.270 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.7362999999999998e-05 2024-09-10T19:16:23.270 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:23.270 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:23.270 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 3,6", 2024-09-10T19:16:23.270 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:07.740592+0000", 2024-09-10T19:16:23.270 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.8367000000000005e-05 2024-09-10T19:16:23.270 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:23.270 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:23.270 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-09-10T19:16:23.270 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:07.743074+0000", 2024-09-10T19:16:23.270 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0024812269999999999 2024-09-10T19:16:23.271 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:23.271 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:23.271 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:16:23.271 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:07.743218+0000", 2024-09-10T19:16:23.271 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000144778 2024-09-10T19:16:23.271 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:23.271 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:23.271 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:16:23.271 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:07.882521+0000", 2024-09-10T19:16:23.271 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.139302329 2024-09-10T19:16:23.271 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:23.271 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:23.271 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-09-10T19:16:23.271 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:07.882569+0000", 2024-09-10T19:16:23.271 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.8505999999999999e-05 2024-09-10T19:16:23.272 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:23.272 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:23.272 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-10T19:16:23.272 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:07.882620+0000", 2024-09-10T19:16:23.272 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.1059e-05 2024-09-10T19:16:23.272 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:16:23.272 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-10T19:16:23.272 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:16:23.272 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:23.272 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:23.272 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.14781.0:168571 7.1 7:8465cf90:::benchmark_data_smithi183_84160_object28190:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121)", 2024-09-10T19:16:23.272 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-10T19:14:13.629412+0000", 2024-09-10T19:16:23.272 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 129.51031156299999, 2024-09-10T19:16:23.272 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.107909929, 2024-09-10T19:16:23.273 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-10T19:16:23.273 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T19:16:23.273 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-10T19:16:23.273 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.14781", 2024-09-10T19:16:23.273 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.183:0/2025776420", 2024-09-10T19:16:23.273 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 168571 2024-09-10T19:16:23.273 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:23.273 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-10T19:16:23.273 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:23.273 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-10T19:16:23.273 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:13.629412+0000", 2024-09-10T19:16:23.273 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:16:23.273 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:23.274 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:23.274 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-10T19:16:23.274 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:13.629412+0000", 2024-09-10T19:16:23.274 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:16:23.274 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:23.274 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:23.274 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-10T19:16:23.274 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:13.629411+0000", 2024-09-10T19:16:23.274 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.999999 2024-09-10T19:16:23.274 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:23.274 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:23.274 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-10T19:16:23.274 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:13.629416+0000", 2024-09-10T19:16:23.274 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.6930000000000004e-06 2024-09-10T19:16:23.275 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:23.275 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:23.275 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-10T19:16:23.275 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:13.629427+0000", 2024-09-10T19:16:23.275 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.0635999999999999e-05 2024-09-10T19:16:23.275 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:23.275 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:23.275 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-10T19:16:23.275 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:13.629431+0000", 2024-09-10T19:16:23.275 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.4070000000000003e-06 2024-09-10T19:16:23.275 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:23.275 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:23.275 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:16:23.275 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:13.629461+0000", 2024-09-10T19:16:23.275 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.9110000000000001e-05 2024-09-10T19:16:23.275 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:23.276 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:23.276 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-10T19:16:23.276 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:13.629531+0000", 2024-09-10T19:16:23.276 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.0209000000000005e-05 2024-09-10T19:16:23.276 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:23.276 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:23.276 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 3,6", 2024-09-10T19:16:23.276 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:13.629618+0000", 2024-09-10T19:16:23.276 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.7678999999999999e-05 2024-09-10T19:16:23.276 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:23.276 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:23.276 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-09-10T19:16:23.276 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:13.635612+0000", 2024-09-10T19:16:23.276 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0059936410000000001 2024-09-10T19:16:23.277 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:23.277 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:23.277 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:16:23.277 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:13.637246+0000", 2024-09-10T19:16:23.277 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.001634021 2024-09-10T19:16:23.277 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:23.277 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:23.277 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:16:23.277 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:13.737242+0000", 2024-09-10T19:16:23.277 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.099995746999999996 2024-09-10T19:16:23.277 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:23.277 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:23.277 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-09-10T19:16:23.277 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:13.737284+0000", 2024-09-10T19:16:23.278 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.214e-05 2024-09-10T19:16:23.278 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:23.278 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:23.278 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-10T19:16:23.278 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:13.737322+0000", 2024-09-10T19:16:23.278 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.7821999999999998e-05 2024-09-10T19:16:23.278 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:16:23.278 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-10T19:16:23.278 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:16:23.278 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:23.278 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:23.278 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.14781.0:168575 7.1 7:8deb5798:::benchmark_data_smithi183_84160_object28194:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121)", 2024-09-10T19:16:23.278 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-10T19:14:13.631950+0000", 2024-09-10T19:16:23.278 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 129.50777326799999, 2024-09-10T19:16:23.278 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.105468507, 2024-09-10T19:16:23.278 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-10T19:16:23.279 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T19:16:23.279 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-10T19:16:23.279 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.14781", 2024-09-10T19:16:23.279 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.183:0/2025776420", 2024-09-10T19:16:23.279 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 168575 2024-09-10T19:16:23.279 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:23.279 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-10T19:16:23.279 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:23.279 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-10T19:16:23.279 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:13.631950+0000", 2024-09-10T19:16:23.279 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:16:23.279 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:23.279 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:23.279 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-10T19:16:23.279 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:13.631950+0000", 2024-09-10T19:16:23.280 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:16:23.280 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:23.280 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:23.280 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-10T19:16:23.280 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:13.631949+0000", 2024-09-10T19:16:23.280 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.999999 2024-09-10T19:16:23.280 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:23.280 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:23.280 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-10T19:16:23.280 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:13.631955+0000", 2024-09-10T19:16:23.280 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.2870000000000004e-06 2024-09-10T19:16:23.280 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:23.280 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:23.280 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-10T19:16:23.281 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:13.631963+0000", 2024-09-10T19:16:23.281 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.5189999999999992e-06 2024-09-10T19:16:23.281 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:23.281 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:23.281 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-10T19:16:23.281 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:13.631967+0000", 2024-09-10T19:16:23.281 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.0940000000000001e-06 2024-09-10T19:16:23.281 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:23.281 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:23.281 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:16:23.281 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:13.631988+0000", 2024-09-10T19:16:23.281 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.1081e-05 2024-09-10T19:16:23.281 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:23.281 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:23.282 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-10T19:16:23.282 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:13.632067+0000", 2024-09-10T19:16:23.282 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.8749000000000001e-05 2024-09-10T19:16:23.282 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:23.282 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:23.282 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 3,6", 2024-09-10T19:16:23.282 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:13.632131+0000", 2024-09-10T19:16:23.282 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.3978000000000003e-05 2024-09-10T19:16:23.282 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:23.282 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:23.282 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-09-10T19:16:23.282 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:13.635646+0000", 2024-09-10T19:16:23.282 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0035145380000000002 2024-09-10T19:16:23.283 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:23.283 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:23.283 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:16:23.283 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:13.639897+0000", 2024-09-10T19:16:23.283 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0042516100000000003 2024-09-10T19:16:23.283 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:23.283 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:23.283 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:16:23.283 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:13.737359+0000", 2024-09-10T19:16:23.284 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.097462282999999997 2024-09-10T19:16:23.284 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:23.284 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:23.284 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-09-10T19:16:23.284 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:13.737395+0000", 2024-09-10T19:16:23.284 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.5855000000000001e-05 2024-09-10T19:16:23.284 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:23.284 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:23.284 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-10T19:16:23.284 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:13.737419+0000", 2024-09-10T19:16:23.284 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.3430000000000001e-05 2024-09-10T19:16:23.285 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:16:23.285 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-10T19:16:23.285 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:16:23.285 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:23.285 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:23.285 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.14781.0:168577 7.1 7:8d2dccd3:::benchmark_data_smithi183_84160_object28196:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e121)", 2024-09-10T19:16:23.285 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-10T19:14:13.633407+0000", 2024-09-10T19:16:23.285 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 129.50631663300001, 2024-09-10T19:16:23.285 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.10409128199999999, 2024-09-10T19:16:23.285 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-10T19:16:23.285 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T19:16:23.285 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-10T19:16:23.285 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.14781", 2024-09-10T19:16:23.285 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.183:0/2025776420", 2024-09-10T19:16:23.286 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 168577 2024-09-10T19:16:23.286 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:23.286 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-10T19:16:23.286 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:23.286 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-10T19:16:23.286 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:13.633407+0000", 2024-09-10T19:16:23.286 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:16:23.286 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:23.286 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:23.286 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-10T19:16:23.286 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:13.633407+0000", 2024-09-10T19:16:23.286 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:16:23.286 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:23.287 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:23.287 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-10T19:16:23.287 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:13.633406+0000", 2024-09-10T19:16:23.287 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.999999 2024-09-10T19:16:23.287 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:23.287 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:23.287 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-10T19:16:23.287 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:13.633411+0000", 2024-09-10T19:16:23.287 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.6269999999999999e-06 2024-09-10T19:16:23.287 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:23.287 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:23.287 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-10T19:16:23.287 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:13.633421+0000", 2024-09-10T19:16:23.287 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.0712e-05 2024-09-10T19:16:23.287 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:23.288 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:23.288 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-10T19:16:23.288 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:13.633426+0000", 2024-09-10T19:16:23.288 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.9699999999999998e-06 2024-09-10T19:16:23.288 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:23.288 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:23.288 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:16:23.288 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:13.633964+0000", 2024-09-10T19:16:23.288 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00053754200000000001 2024-09-10T19:16:23.288 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:23.288 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:23.288 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-10T19:16:23.288 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:13.634028+0000", 2024-09-10T19:16:23.288 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.3949999999999996e-05 2024-09-10T19:16:23.289 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:23.289 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:23.289 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 3,6", 2024-09-10T19:16:23.289 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:13.634089+0000", 2024-09-10T19:16:23.289 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.1056999999999995e-05 2024-09-10T19:16:23.289 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:23.289 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:23.289 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-09-10T19:16:23.289 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:13.635699+0000", 2024-09-10T19:16:23.289 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0016101959999999999 2024-09-10T19:16:23.289 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:23.289 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:23.289 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:16:23.290 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:13.639972+0000", 2024-09-10T19:16:23.290 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0042729689999999997 2024-09-10T19:16:23.290 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:23.290 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:23.290 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:16:23.290 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:13.737448+0000", 2024-09-10T19:16:23.290 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.097476227999999998 2024-09-10T19:16:23.290 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:23.290 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:23.290 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-09-10T19:16:23.290 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:13.737478+0000", 2024-09-10T19:16:23.290 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.0006e-05 2024-09-10T19:16:23.290 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:23.291 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:23.291 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-10T19:16:23.291 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:13.737498+0000", 2024-09-10T19:16:23.291 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.9820999999999999e-05 2024-09-10T19:16:23.291 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:16:23.291 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-10T19:16:23.291 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:16:23.291 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:23.291 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:23.291 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_repop(client.14781.0:168586 7.8 e121/86 7:12b331d0:::benchmark_data_smithi183_84160_object28205:head v 121'10641, mlcod=121'10638)", 2024-09-10T19:16:23.291 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-10T19:14:13.647784+0000", 2024-09-10T19:16:23.291 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 129.491939261, 2024-09-10T19:16:23.291 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.082590319999999995, 2024-09-10T19:16:23.291 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-10T19:16:23.291 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T19:16:23.291 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-10T19:16:23.292 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:23.292 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-10T19:16:23.292 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:13.647784+0000", 2024-09-10T19:16:23.292 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:16:23.292 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:23.292 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:23.292 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-10T19:16:23.292 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:13.647784+0000", 2024-09-10T19:16:23.292 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:16:23.292 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:23.292 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:23.292 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-10T19:16:23.292 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:13.647784+0000", 2024-09-10T19:16:23.292 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999995 2024-09-10T19:16:23.293 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:23.293 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:23.293 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-10T19:16:23.293 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:13.647790+0000", 2024-09-10T19:16:23.293 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.0050000000000001e-06 2024-09-10T19:16:23.293 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:23.293 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:23.293 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-10T19:16:23.293 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:13.647800+0000", 2024-09-10T19:16:23.293 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.0209e-05 2024-09-10T19:16:23.293 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:23.293 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:23.293 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-10T19:16:23.293 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:13.647803+0000", 2024-09-10T19:16:23.294 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.6959999999999998e-06 2024-09-10T19:16:23.294 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:23.294 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:23.294 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:16:23.294 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:13.726046+0000", 2024-09-10T19:16:23.294 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.078242721000000001 2024-09-10T19:16:23.294 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:23.294 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:23.294 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-10T19:16:23.294 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:13.726068+0000", 2024-09-10T19:16:23.294 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.2188000000000002e-05 2024-09-10T19:16:23.294 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:23.294 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:23.294 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-09-10T19:16:23.294 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:13.730299+0000", 2024-09-10T19:16:23.295 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0042301869999999998 2024-09-10T19:16:23.295 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:23.295 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:23.295 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-10T19:16:23.295 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:13.730375+0000", 2024-09-10T19:16:23.295 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.5952999999999996e-05 2024-09-10T19:16:23.295 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:16:23.295 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-10T19:16:23.295 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:16:23.295 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:16:23.295 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-10T19:16:23.295 INFO:teuthology.orchestra.run.smithi191.stdout:} 2024-09-10T19:16:23.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:16:23.296 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:16:23.296 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:16:23.397 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:16:23.418 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:16:23.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:16:23.419 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:16:23.520 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:16:23.541 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:16:23.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:16:23.542 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:16:23.643 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:16:23.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:16:23.665 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:16:23.666 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:16:23.766 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:16:23.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:16:23.788 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:16:23.789 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:16:23.827 INFO:teuthology.orchestra.run.smithi183.stderr:marked out osd.2. 2024-09-10T19:16:23.889 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:16:23.910 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:16:23.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:16:23.911 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:16:23.965 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:16:23 smithi191 ceph-mon[32884]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T19:16:23.966 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:16:23 smithi191 ceph-mon[32884]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_1", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-09-10T19:16:23.966 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:16:23 smithi191 ceph-mon[32884]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-10T19:16:23.966 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:16:23 smithi191 ceph-mon[32884]: osdmap e153: 8 total, 8 up, 7 in 2024-09-10T19:16:23.966 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:16:23 smithi191 ceph-mon[32884]: from='client.? 172.21.15.183:0/841020370' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["2"]}]: dispatch 2024-09-10T19:16:23.966 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:16:23 smithi191 ceph-mon[32884]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["2"]}]: dispatch 2024-09-10T19:16:23.966 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:16:23 smithi191 ceph-mon[32884]: Client client.admin marked osd.2 out, while it was still marked up 2024-09-10T19:16:23.966 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:16:23 smithi191 ceph-mon[32884]: 6.10 scrub starts 2024-09-10T19:16:23.966 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:16:23 smithi191 ceph-mon[32884]: 6.10 scrub ok 2024-09-10T19:16:24.012 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:16:24.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:16:24.035 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:16:24.035 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:16:24.070 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:16:23 smithi183 ceph-mon[29736]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T19:16:24.070 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:16:23 smithi183 ceph-mon[29736]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_1", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-09-10T19:16:24.070 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:16:23 smithi183 ceph-mon[29736]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-10T19:16:24.070 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:16:23 smithi183 ceph-mon[29736]: osdmap e153: 8 total, 8 up, 7 in 2024-09-10T19:16:24.070 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:16:23 smithi183 ceph-mon[29736]: from='client.? 172.21.15.183:0/841020370' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["2"]}]: dispatch 2024-09-10T19:16:24.070 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:16:23 smithi183 ceph-mon[29736]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["2"]}]: dispatch 2024-09-10T19:16:24.070 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:16:23 smithi183 ceph-mon[29736]: Client client.admin marked osd.2 out, while it was still marked up 2024-09-10T19:16:24.070 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:16:23 smithi183 ceph-mon[29736]: 6.10 scrub starts 2024-09-10T19:16:24.070 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:16:23 smithi183 ceph-mon[29736]: 6.10 scrub ok 2024-09-10T19:16:24.070 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:16:23 smithi183 ceph-mon[41562]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T19:16:24.071 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:16:23 smithi183 ceph-mon[41562]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_1", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-09-10T19:16:24.071 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:16:23 smithi183 ceph-mon[41562]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-10T19:16:24.071 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:16:23 smithi183 ceph-mon[41562]: osdmap e153: 8 total, 8 up, 7 in 2024-09-10T19:16:24.071 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:16:23 smithi183 ceph-mon[41562]: from='client.? 172.21.15.183:0/841020370' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["2"]}]: dispatch 2024-09-10T19:16:24.071 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:16:23 smithi183 ceph-mon[41562]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["2"]}]: dispatch 2024-09-10T19:16:24.071 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:16:23 smithi183 ceph-mon[41562]: Client client.admin marked osd.2 out, while it was still marked up 2024-09-10T19:16:24.071 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:16:23 smithi183 ceph-mon[41562]: 6.10 scrub starts 2024-09-10T19:16:24.071 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:16:23 smithi183 ceph-mon[41562]: 6.10 scrub ok 2024-09-10T19:16:24.099 DEBUG:teuthology.orchestra.run.smithi191:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph daemon osd.5 dump_ops_in_flight 2024-09-10T19:16:24.136 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:16:24.157 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:16:24.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:16:24.158 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:16:24.259 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:16:24.280 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:16:24.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:16:24.281 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:16:24.341 INFO:teuthology.orchestra.run.smithi191.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.b/config 2024-09-10T19:16:24.382 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:16:24.403 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:16:24.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:16:24.404 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:16:24.504 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:16:24.526 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:16:24.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:16:24.527 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:16:24.627 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:16:24.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:16:24.649 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:16:24.649 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:16:24.750 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:16:24.766 DEBUG:teuthology.orchestra.run.smithi183:> /bin/sh -c 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --no-log-to-stderr --name client.0 -b 65536 --object-size 65536 -p unique_pool_1 bench 90 write' 2024-09-10T19:16:24.768 INFO:tasks.radosbench:joining radosbench (timing out after 3000s) 2024-09-10T19:16:24.769 INFO:teuthology.orchestra.run:waiting for 3000 2024-09-10T19:16:24.772 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:16:24.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:16:24.772 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:16:24.820 INFO:tasks.radosbench.radosbench.0.smithi183.stdout:hints = 1 2024-09-10T19:16:24.820 INFO:tasks.radosbench.radosbench.0.smithi183.stdout:Maintaining 16 concurrent writes of 65536 bytes to objects of size 65536 for up to 90 seconds or 0 objects 2024-09-10T19:16:24.820 INFO:tasks.radosbench.radosbench.0.smithi183.stdout:Object prefix: benchmark_data_smithi183_114924 2024-09-10T19:16:24.820 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-10T19:16:24.820 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: 0 0 0 0 0 0 - 0 2024-09-10T19:16:24.874 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:16:24.913 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:16:24.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:16:24.914 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:16:25.015 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:16:25.053 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:16:25.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:16:25.054 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:16:25.155 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:16:25.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:16:24 smithi183 ceph-mon[29736]: 2.17 scrub starts 2024-09-10T19:16:25.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:16:24 smithi183 ceph-mon[29736]: 2.17 scrub ok 2024-09-10T19:16:25.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:16:24 smithi183 ceph-mon[29736]: pgmap v253: 148 pgs: 148 active+clean; 582 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 639 B/s rd, 0 op/s 2024-09-10T19:16:25.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:16:24 smithi183 ceph-mon[29736]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-10T19:16:25.182 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:16:24 smithi183 ceph-mon[29736]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["2"]}]': finished 2024-09-10T19:16:25.182 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:16:24 smithi183 ceph-mon[29736]: osdmap e154: 8 total, 8 up, 6 in 2024-09-10T19:16:25.182 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:16:24 smithi183 ceph-mon[29736]: 5.15 scrub starts 2024-09-10T19:16:25.182 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:16:24 smithi183 ceph-mon[29736]: 5.15 scrub ok 2024-09-10T19:16:25.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:16:24 smithi183 ceph-mon[41562]: 2.17 scrub starts 2024-09-10T19:16:25.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:16:24 smithi183 ceph-mon[41562]: 2.17 scrub ok 2024-09-10T19:16:25.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:16:24 smithi183 ceph-mon[41562]: pgmap v253: 148 pgs: 148 active+clean; 582 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 639 B/s rd, 0 op/s 2024-09-10T19:16:25.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:16:24 smithi183 ceph-mon[41562]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-10T19:16:25.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:16:24 smithi183 ceph-mon[41562]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["2"]}]': finished 2024-09-10T19:16:25.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:16:24 smithi183 ceph-mon[41562]: osdmap e154: 8 total, 8 up, 6 in 2024-09-10T19:16:25.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:16:24 smithi183 ceph-mon[41562]: 5.15 scrub starts 2024-09-10T19:16:25.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:16:24 smithi183 ceph-mon[41562]: 5.15 scrub ok 2024-09-10T19:16:25.191 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:16:25.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:16:25.192 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:16:25.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:16:24 smithi191 ceph-mon[32884]: 2.17 scrub starts 2024-09-10T19:16:25.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:16:24 smithi191 ceph-mon[32884]: 2.17 scrub ok 2024-09-10T19:16:25.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:16:24 smithi191 ceph-mon[32884]: pgmap v253: 148 pgs: 148 active+clean; 582 KiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 639 B/s rd, 0 op/s 2024-09-10T19:16:25.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:16:24 smithi191 ceph-mon[32884]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-10T19:16:25.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:16:24 smithi191 ceph-mon[32884]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["2"]}]': finished 2024-09-10T19:16:25.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:16:24 smithi191 ceph-mon[32884]: osdmap e154: 8 total, 8 up, 6 in 2024-09-10T19:16:25.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:16:24 smithi191 ceph-mon[32884]: 5.15 scrub starts 2024-09-10T19:16:25.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:16:24 smithi191 ceph-mon[32884]: 5.15 scrub ok 2024-09-10T19:16:25.293 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:16:25.329 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:16:25.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:16:25.330 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:16:25.431 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:16:25.464 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:16:25.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:16:25.465 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:16:25.565 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:16:25.602 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:16:25.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:16:25.603 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:16:25.704 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:16:25.737 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:16:25.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:16:25.738 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:16:25.820 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: 1 16 3879 3863 241.424 241.438 0.00432492 0.00412343 2024-09-10T19:16:25.839 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:16:25.885 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:16:25.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:16:25.886 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:16:25.986 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:16:26.014 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:16:25 smithi191 ceph-mon[32884]: 5.11 scrub starts 2024-09-10T19:16:26.014 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:16:25 smithi191 ceph-mon[32884]: 5.11 scrub ok 2024-09-10T19:16:26.014 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:16:25 smithi191 ceph-mon[32884]: osdmap e155: 8 total, 8 up, 6 in 2024-09-10T19:16:26.015 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:16:25 smithi191 ceph-mon[32884]: purged_snaps scrub starts 2024-09-10T19:16:26.015 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:16:25 smithi191 ceph-mon[32884]: purged_snaps scrub ok 2024-09-10T19:16:26.020 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:16:26.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:16:26.021 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:16:26.056 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:16:25 smithi183 ceph-mon[29736]: 5.11 scrub starts 2024-09-10T19:16:26.056 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:16:25 smithi183 ceph-mon[29736]: 5.11 scrub ok 2024-09-10T19:16:26.056 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:16:25 smithi183 ceph-mon[29736]: osdmap e155: 8 total, 8 up, 6 in 2024-09-10T19:16:26.056 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:16:25 smithi183 ceph-mon[29736]: purged_snaps scrub starts 2024-09-10T19:16:26.056 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:16:25 smithi183 ceph-mon[29736]: purged_snaps scrub ok 2024-09-10T19:16:26.056 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:16:25 smithi183 ceph-mon[41562]: 5.11 scrub starts 2024-09-10T19:16:26.056 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:16:25 smithi183 ceph-mon[41562]: 5.11 scrub ok 2024-09-10T19:16:26.057 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:16:25 smithi183 ceph-mon[41562]: osdmap e155: 8 total, 8 up, 6 in 2024-09-10T19:16:26.057 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:16:25 smithi183 ceph-mon[41562]: purged_snaps scrub starts 2024-09-10T19:16:26.057 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:16:25 smithi183 ceph-mon[41562]: purged_snaps scrub ok 2024-09-10T19:16:26.121 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:16:26.162 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:16:26.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:16:26.163 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:16:26.263 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:16:26.297 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:16:26.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:16:26.298 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:16:26.399 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:16:26.434 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:16:26.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:16:26.435 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:16:26.504 INFO:teuthology.orchestra.run.smithi191.stdout:{ 2024-09-10T19:16:26.504 INFO:teuthology.orchestra.run.smithi191.stdout: "ops": [ 2024-09-10T19:16:26.504 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:26.504 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_repop(client.15051.0:6291 8.1 e156/154 8:86dc3fff:::benchmark_data_smithi183_114924_object6290:head v 156'401, mlcod=156'399)", 2024-09-10T19:16:26.505 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-10T19:16:26.501435+0000", 2024-09-10T19:16:26.505 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 0.00049797199999999995, 2024-09-10T19:16:26.505 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00052689700000000002, 2024-09-10T19:16:26.505 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-10T19:16:26.505 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "started", 2024-09-10T19:16:26.505 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-10T19:16:26.505 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:26.505 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-10T19:16:26.505 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:16:26.501435+0000", 2024-09-10T19:16:26.505 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:16:26.505 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:26.505 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:26.506 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-10T19:16:26.506 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:16:26.501435+0000", 2024-09-10T19:16:26.506 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:16:26.506 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:26.506 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:26.506 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-10T19:16:26.506 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:16:26.501434+0000", 2024-09-10T19:16:26.506 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.999999 2024-09-10T19:16:26.506 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:26.506 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:26.506 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-10T19:16:26.506 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:16:26.501559+0000", 2024-09-10T19:16:26.506 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000124564 2024-09-10T19:16:26.506 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:26.507 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:26.507 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-10T19:16:26.507 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:16:26.501570+0000", 2024-09-10T19:16:26.507 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.1596999999999999e-05 2024-09-10T19:16:26.507 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:26.507 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:26.507 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-10T19:16:26.507 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:16:26.501576+0000", 2024-09-10T19:16:26.507 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.6949999999999997e-06 2024-09-10T19:16:26.507 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:26.507 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:26.507 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:16:26.507 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:16:26.501631+0000", 2024-09-10T19:16:26.508 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.5331000000000003e-05 2024-09-10T19:16:26.508 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:26.508 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:26.508 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-10T19:16:26.508 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:16:26.501657+0000", 2024-09-10T19:16:26.508 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.5137e-05 2024-09-10T19:16:26.508 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:16:26.508 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-10T19:16:26.508 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:16:26.508 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:26.508 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:26.508 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.15051.0:6292 8.8 8.410edbb8 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e156)", 2024-09-10T19:16:26.509 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-10T19:16:26.501927+0000", 2024-09-10T19:16:26.509 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 6.0179999999999998e-06, 2024-09-10T19:16:26.509 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.8821999999999997e-05, 2024-09-10T19:16:26.509 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-10T19:16:26.509 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "queued for pg", 2024-09-10T19:16:26.509 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-10T19:16:26.509 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.15051", 2024-09-10T19:16:26.509 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.183:0/3632028672", 2024-09-10T19:16:26.509 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 6292 2024-09-10T19:16:26.509 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:26.509 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-10T19:16:26.509 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:26.509 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-10T19:16:26.509 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:16:26.501927+0000", 2024-09-10T19:16:26.510 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:16:26.510 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:26.510 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:26.510 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-10T19:16:26.510 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:16:26.501927+0000", 2024-09-10T19:16:26.510 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:16:26.510 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:26.510 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:26.510 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-10T19:16:26.510 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:16:26.501926+0000", 2024-09-10T19:16:26.510 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.999999 2024-09-10T19:16:26.510 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:26.510 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:26.510 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-10T19:16:26.510 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:16:26.501954+0000", 2024-09-10T19:16:26.511 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.8161999999999999e-05 2024-09-10T19:16:26.511 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:26.511 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:26.511 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-10T19:16:26.511 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:16:26.501965+0000", 2024-09-10T19:16:26.511 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.0308e-05 2024-09-10T19:16:26.511 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:26.511 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:26.511 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-10T19:16:26.511 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:16:26.501971+0000", 2024-09-10T19:16:26.511 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.9329999999999998e-06 2024-09-10T19:16:26.511 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:16:26.511 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-10T19:16:26.511 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:16:26.511 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:26.512 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:26.512 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.15051.0:6278 8.a 8:59dd2c94:::benchmark_data_smithi183_114924_object6277:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e156)", 2024-09-10T19:16:26.512 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-10T19:16:26.497004+0000", 2024-09-10T19:16:26.512 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 0.0049294739999999997, 2024-09-10T19:16:26.512 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0050861680000000003, 2024-09-10T19:16:26.512 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-10T19:16:26.512 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "waiting for sub ops", 2024-09-10T19:16:26.512 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-10T19:16:26.512 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.15051", 2024-09-10T19:16:26.512 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.183:0/3632028672", 2024-09-10T19:16:26.512 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 6278 2024-09-10T19:16:26.512 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:26.512 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-10T19:16:26.512 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:26.513 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-10T19:16:26.513 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:16:26.497004+0000", 2024-09-10T19:16:26.513 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:16:26.513 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:26.513 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:26.513 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-10T19:16:26.513 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:16:26.497004+0000", 2024-09-10T19:16:26.513 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:16:26.513 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:26.513 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:26.513 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-10T19:16:26.513 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:16:26.497003+0000", 2024-09-10T19:16:26.513 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.999999 2024-09-10T19:16:26.514 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:26.514 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:26.514 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-10T19:16:26.514 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:16:26.497044+0000", 2024-09-10T19:16:26.514 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.0871999999999998e-05 2024-09-10T19:16:26.514 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:26.514 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:26.514 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-10T19:16:26.514 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:16:26.497054+0000", 2024-09-10T19:16:26.514 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.0152999999999999e-05 2024-09-10T19:16:26.514 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:26.514 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:26.514 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-10T19:16:26.514 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:16:26.497059+0000", 2024-09-10T19:16:26.515 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.2249999999999999e-06 2024-09-10T19:16:26.515 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:26.515 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:26.515 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:16:26.515 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:16:26.497290+0000", 2024-09-10T19:16:26.515 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00023084700000000001 2024-09-10T19:16:26.515 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:26.515 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:26.515 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-10T19:16:26.515 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:16:26.497412+0000", 2024-09-10T19:16:26.515 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00012267499999999999 2024-09-10T19:16:26.515 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:26.515 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:26.515 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 3,4", 2024-09-10T19:16:26.516 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:16:26.497564+0000", 2024-09-10T19:16:26.516 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00015112900000000001 2024-09-10T19:16:26.516 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:26.516 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:26.516 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-09-10T19:16:26.516 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:16:26.499990+0000", 2024-09-10T19:16:26.516 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0024261280000000001 2024-09-10T19:16:26.516 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:26.516 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:26.516 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:16:26.516 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:16:26.501892+0000", 2024-09-10T19:16:26.516 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0019024490000000001 2024-09-10T19:16:26.516 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:16:26.517 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-10T19:16:26.517 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:16:26.517 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:26.517 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:26.517 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.15051.0:6279 8.d 8:bb881c3e:::benchmark_data_smithi183_114924_object6278:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e156)", 2024-09-10T19:16:26.517 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-10T19:16:26.497069+0000", 2024-09-10T19:16:26.517 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 0.0048638900000000001, 2024-09-10T19:16:26.517 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.005109761, 2024-09-10T19:16:26.517 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-10T19:16:26.517 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "waiting for sub ops", 2024-09-10T19:16:26.517 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-10T19:16:26.517 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.15051", 2024-09-10T19:16:26.517 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.183:0/3632028672", 2024-09-10T19:16:26.518 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 6279 2024-09-10T19:16:26.518 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:26.518 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-10T19:16:26.518 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:26.518 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-10T19:16:26.518 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:16:26.497069+0000", 2024-09-10T19:16:26.518 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:16:26.518 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:26.518 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:26.518 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-10T19:16:26.519 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:16:26.497069+0000", 2024-09-10T19:16:26.519 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:16:26.519 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:26.519 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:26.519 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-10T19:16:26.519 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:16:26.497069+0000", 2024-09-10T19:16:26.519 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999995 2024-09-10T19:16:26.519 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:26.519 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:26.519 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-10T19:16:26.519 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:16:26.497102+0000", 2024-09-10T19:16:26.519 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.3062999999999998e-05 2024-09-10T19:16:26.520 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:26.520 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:26.520 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-10T19:16:26.520 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:16:26.497110+0000", 2024-09-10T19:16:26.520 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.0879999999999995e-06 2024-09-10T19:16:26.520 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:26.520 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:26.520 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-10T19:16:26.520 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:16:26.497115+0000", 2024-09-10T19:16:26.520 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.8550000000000001e-06 2024-09-10T19:16:26.520 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:26.520 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:26.520 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:16:26.521 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:16:26.498729+0000", 2024-09-10T19:16:26.521 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.001613972 2024-09-10T19:16:26.521 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:26.521 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:26.521 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-10T19:16:26.521 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:16:26.498853+0000", 2024-09-10T19:16:26.521 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000124036 2024-09-10T19:16:26.521 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:26.521 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:26.521 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 6,7", 2024-09-10T19:16:26.521 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:16:26.498999+0000", 2024-09-10T19:16:26.521 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00014678900000000001 2024-09-10T19:16:26.521 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:26.522 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:26.522 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-09-10T19:16:26.522 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:16:26.500311+0000", 2024-09-10T19:16:26.522 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0013112009999999999 2024-09-10T19:16:26.522 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:26.522 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:26.522 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:16:26.522 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:16:26.501858+0000", 2024-09-10T19:16:26.522 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0015472940000000001 2024-09-10T19:16:26.522 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:26.522 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:26.522 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:16:26.522 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:16:26.502159+0000", 2024-09-10T19:16:26.522 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000300735 2024-09-10T19:16:26.522 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:16:26.523 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-10T19:16:26.523 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:16:26.523 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:26.523 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:26.523 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_repop(client.15051.0:6285 8.2 e156/148 8:47141e6a:::benchmark_data_smithi183_114924_object6284:head v 156'366, mlcod=156'364)", 2024-09-10T19:16:26.523 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-10T19:16:26.499538+0000", 2024-09-10T19:16:26.523 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 0.002395501, 2024-09-10T19:16:26.523 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.002734163, 2024-09-10T19:16:26.523 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-10T19:16:26.523 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T19:16:26.523 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-10T19:16:26.523 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:26.523 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-10T19:16:26.524 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:16:26.499538+0000", 2024-09-10T19:16:26.524 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:16:26.524 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:26.524 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:26.524 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-10T19:16:26.524 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:16:26.499538+0000", 2024-09-10T19:16:26.524 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:16:26.524 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:26.524 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:26.524 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-10T19:16:26.524 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:16:26.499537+0000", 2024-09-10T19:16:26.524 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999995 2024-09-10T19:16:26.524 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:26.524 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:26.525 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-10T19:16:26.525 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:16:26.499562+0000", 2024-09-10T19:16:26.525 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.5550000000000001e-05 2024-09-10T19:16:26.525 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:26.525 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:26.525 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-10T19:16:26.525 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:16:26.500457+0000", 2024-09-10T19:16:26.525 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00089490900000000005 2024-09-10T19:16:26.525 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:26.525 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:26.525 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-10T19:16:26.525 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:16:26.500466+0000", 2024-09-10T19:16:26.525 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.8319999999999995e-06 2024-09-10T19:16:26.526 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:26.526 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:26.526 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:16:26.526 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:16:26.500660+0000", 2024-09-10T19:16:26.526 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00019388399999999999 2024-09-10T19:16:26.526 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:26.526 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:26.526 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-10T19:16:26.526 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:16:26.500680+0000", 2024-09-10T19:16:26.526 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.9599999999999999e-05 2024-09-10T19:16:26.526 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:26.526 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:26.526 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-09-10T19:16:26.526 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:16:26.502175+0000", 2024-09-10T19:16:26.526 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.001495725 2024-09-10T19:16:26.527 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:16:26.527 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-10T19:16:26.527 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:16:26.527 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:26.527 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:26.527 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.15051.0:6286 8.a 8:597b6472:::benchmark_data_smithi183_114924_object6285:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e156)", 2024-09-10T19:16:26.527 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-10T19:16:26.500615+0000", 2024-09-10T19:16:26.527 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 0.001318133, 2024-09-10T19:16:26.527 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.001724808, 2024-09-10T19:16:26.527 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-10T19:16:26.527 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "waiting for sub ops", 2024-09-10T19:16:26.527 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-10T19:16:26.527 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.15051", 2024-09-10T19:16:26.527 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.183:0/3632028672", 2024-09-10T19:16:26.527 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 6286 2024-09-10T19:16:26.528 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:26.528 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-10T19:16:26.528 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:26.528 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-10T19:16:26.528 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:16:26.500615+0000", 2024-09-10T19:16:26.528 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:16:26.528 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:26.528 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:26.528 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-10T19:16:26.528 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:16:26.500615+0000", 2024-09-10T19:16:26.528 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:16:26.528 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:26.528 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:26.528 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-10T19:16:26.529 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:16:26.500614+0000", 2024-09-10T19:16:26.529 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.999999 2024-09-10T19:16:26.529 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:26.529 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:26.529 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-10T19:16:26.529 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:16:26.500648+0000", 2024-09-10T19:16:26.529 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.4635999999999998e-05 2024-09-10T19:16:26.529 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:26.529 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:26.529 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-10T19:16:26.529 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:16:26.500658+0000", 2024-09-10T19:16:26.529 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.1900000000000001e-06 2024-09-10T19:16:26.529 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:26.529 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:26.529 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-10T19:16:26.530 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:16:26.500663+0000", 2024-09-10T19:16:26.530 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.4260000000000002e-06 2024-09-10T19:16:26.530 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:26.530 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:26.530 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:16:26.530 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:16:26.500760+0000", 2024-09-10T19:16:26.530 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.7114999999999994e-05 2024-09-10T19:16:26.530 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:26.530 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:26.530 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-10T19:16:26.530 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:16:26.500894+0000", 2024-09-10T19:16:26.530 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00013433100000000001 2024-09-10T19:16:26.530 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:26.530 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:26.531 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 3,4", 2024-09-10T19:16:26.531 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:16:26.501021+0000", 2024-09-10T19:16:26.531 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000126888 2024-09-10T19:16:26.531 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:16:26.531 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-10T19:16:26.531 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:16:26.531 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:26.531 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:26.531 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.15051.0:6287 8.4 8:2fbfdabb:::benchmark_data_smithi183_114924_object6286:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e156)", 2024-09-10T19:16:26.531 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-10T19:16:26.500672+0000", 2024-09-10T19:16:26.531 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 0.0012613030000000001, 2024-09-10T19:16:26.531 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0017428210000000001, 2024-09-10T19:16:26.531 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-10T19:16:26.532 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "waiting for sub ops", 2024-09-10T19:16:26.532 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-10T19:16:26.532 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.15051", 2024-09-10T19:16:26.532 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.183:0/3632028672", 2024-09-10T19:16:26.532 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 6287 2024-09-10T19:16:26.532 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:26.532 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-10T19:16:26.532 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:26.532 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-10T19:16:26.532 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:16:26.500672+0000", 2024-09-10T19:16:26.532 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:16:26.532 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:26.532 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:26.533 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-10T19:16:26.533 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:16:26.500672+0000", 2024-09-10T19:16:26.533 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:16:26.533 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:26.533 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:26.533 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-10T19:16:26.533 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:16:26.500671+0000", 2024-09-10T19:16:26.533 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999995 2024-09-10T19:16:26.533 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:26.533 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:26.533 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-10T19:16:26.533 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:16:26.500703+0000", 2024-09-10T19:16:26.533 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.1807000000000002e-05 2024-09-10T19:16:26.533 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:26.534 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:26.534 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-10T19:16:26.534 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:16:26.500711+0000", 2024-09-10T19:16:26.534 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.9189999999999999e-06 2024-09-10T19:16:26.534 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:26.534 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:26.534 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-10T19:16:26.534 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:16:26.500715+0000", 2024-09-10T19:16:26.534 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.3150000000000002e-06 2024-09-10T19:16:26.534 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:26.534 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:26.534 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:16:26.534 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:16:26.501328+0000", 2024-09-10T19:16:26.535 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00061267699999999995 2024-09-10T19:16:26.535 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:26.535 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:26.535 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-10T19:16:26.535 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:16:26.501428+0000", 2024-09-10T19:16:26.535 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.9825000000000001e-05 2024-09-10T19:16:26.535 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:26.535 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:26.535 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 4,6", 2024-09-10T19:16:26.535 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:16:26.501581+0000", 2024-09-10T19:16:26.535 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000153399 2024-09-10T19:16:26.535 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:26.535 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:26.535 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-09-10T19:16:26.535 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:16:26.502227+0000", 2024-09-10T19:16:26.536 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00064596499999999997 2024-09-10T19:16:26.536 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:16:26.536 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-10T19:16:26.536 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:16:26.536 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:16:26.536 INFO:teuthology.orchestra.run.smithi191.stdout: ], 2024-09-10T19:16:26.536 INFO:teuthology.orchestra.run.smithi191.stdout: "num_ops": 7 2024-09-10T19:16:26.536 INFO:teuthology.orchestra.run.smithi191.stdout:} 2024-09-10T19:16:26.536 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:16:26.537 INFO:teuthology.orchestra.run.smithi183.stdout:osd.0: {} 2024-09-10T19:16:26.537 INFO:teuthology.orchestra.run.smithi183.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-10T19:16:26.562 INFO:teuthology.orchestra.run.smithi183.stdout:osd.1: {} 2024-09-10T19:16:26.562 INFO:teuthology.orchestra.run.smithi183.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-10T19:16:26.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:16:26.568 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:16:26.568 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:16:26.570 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-10T19:16:26.591 INFO:teuthology.orchestra.run.smithi183.stdout:osd.2: {} 2024-09-10T19:16:26.591 INFO:teuthology.orchestra.run.smithi183.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-10T19:16:26.621 INFO:teuthology.orchestra.run.smithi183.stdout:osd.3: {} 2024-09-10T19:16:26.622 INFO:teuthology.orchestra.run.smithi183.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-10T19:16:26.654 INFO:teuthology.orchestra.run.smithi183.stdout:osd.4: {} 2024-09-10T19:16:26.655 INFO:teuthology.orchestra.run.smithi183.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-10T19:16:26.669 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:16:26.684 INFO:teuthology.orchestra.run.smithi183.stdout:osd.5: {} 2024-09-10T19:16:26.684 INFO:teuthology.orchestra.run.smithi183.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-10T19:16:26.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:16:26.704 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:16:26.704 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:16:26.718 INFO:teuthology.orchestra.run.smithi183.stdout:osd.6: {} 2024-09-10T19:16:26.718 INFO:teuthology.orchestra.run.smithi183.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-10T19:16:26.749 INFO:teuthology.orchestra.run.smithi183.stdout:osd.7: {} 2024-09-10T19:16:26.750 INFO:teuthology.orchestra.run.smithi183.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-10T19:16:26.805 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:16:26.820 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: 2 16 7355 7339 229.328 217.25 0.00587867 0.00435176 2024-09-10T19:16:26.839 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:16:26.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:16:26.840 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:16:26.931 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:16:26 smithi183 ceph-mon[29736]: 2.7 scrub starts 2024-09-10T19:16:26.931 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:16:26 smithi183 ceph-mon[29736]: 2.7 scrub ok 2024-09-10T19:16:26.931 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:16:26 smithi183 ceph-mon[29736]: 2.1d scrub starts 2024-09-10T19:16:26.932 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:16:26 smithi183 ceph-mon[29736]: 2.1d scrub ok 2024-09-10T19:16:26.932 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:16:26 smithi183 ceph-mon[29736]: pgmap v256: 148 pgs: 37 peering, 111 active+clean; 14 MiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.2 MiB/s wr, 33 op/s; 0 B/s, 2 objects/s recovering 2024-09-10T19:16:26.932 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:16:26 smithi183 ceph-mon[29736]: 4.f scrub starts 2024-09-10T19:16:26.932 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:16:26 smithi183 ceph-mon[29736]: 4.f scrub ok 2024-09-10T19:16:26.932 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:16:26 smithi183 ceph-mon[29736]: Health check failed: Reduced data availability: 3 pgs inactive, 6 pgs peering (PG_AVAILABILITY) 2024-09-10T19:16:26.932 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:16:26 smithi183 ceph-mon[29736]: osdmap e156: 8 total, 8 up, 6 in 2024-09-10T19:16:26.932 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:16:26 smithi183 ceph-mon[29736]: from='client.? 172.21.15.183:0/1527339839' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T19:16:26.932 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:16:26 smithi183 ceph-mon[41562]: 2.7 scrub starts 2024-09-10T19:16:26.932 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:16:26 smithi183 ceph-mon[41562]: 2.7 scrub ok 2024-09-10T19:16:26.932 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:16:26 smithi183 ceph-mon[41562]: 2.1d scrub starts 2024-09-10T19:16:26.932 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:16:26 smithi183 ceph-mon[41562]: 2.1d scrub ok 2024-09-10T19:16:26.933 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:16:26 smithi183 ceph-mon[41562]: pgmap v256: 148 pgs: 37 peering, 111 active+clean; 14 MiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.2 MiB/s wr, 33 op/s; 0 B/s, 2 objects/s recovering 2024-09-10T19:16:26.933 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:16:26 smithi183 ceph-mon[41562]: 4.f scrub starts 2024-09-10T19:16:26.933 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:16:26 smithi183 ceph-mon[41562]: 4.f scrub ok 2024-09-10T19:16:26.933 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:16:26 smithi183 ceph-mon[41562]: Health check failed: Reduced data availability: 3 pgs inactive, 6 pgs peering (PG_AVAILABILITY) 2024-09-10T19:16:26.933 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:16:26 smithi183 ceph-mon[41562]: osdmap e156: 8 total, 8 up, 6 in 2024-09-10T19:16:26.933 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:16:26 smithi183 ceph-mon[41562]: from='client.? 172.21.15.183:0/1527339839' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T19:16:26.940 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:16:26.972 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:16:26.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:16:26.973 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:16:27.074 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:16:27.106 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:16:27.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:16:27.107 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:16:27.208 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:16:27.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:16:26 smithi191 ceph-mon[32884]: 2.7 scrub starts 2024-09-10T19:16:27.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:16:26 smithi191 ceph-mon[32884]: 2.7 scrub ok 2024-09-10T19:16:27.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:16:26 smithi191 ceph-mon[32884]: 2.1d scrub starts 2024-09-10T19:16:27.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:16:26 smithi191 ceph-mon[32884]: 2.1d scrub ok 2024-09-10T19:16:27.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:16:26 smithi191 ceph-mon[32884]: pgmap v256: 148 pgs: 37 peering, 111 active+clean; 14 MiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.2 MiB/s wr, 33 op/s; 0 B/s, 2 objects/s recovering 2024-09-10T19:16:27.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:16:26 smithi191 ceph-mon[32884]: 4.f scrub starts 2024-09-10T19:16:27.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:16:26 smithi191 ceph-mon[32884]: 4.f scrub ok 2024-09-10T19:16:27.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:16:26 smithi191 ceph-mon[32884]: Health check failed: Reduced data availability: 3 pgs inactive, 6 pgs peering (PG_AVAILABILITY) 2024-09-10T19:16:27.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:16:26 smithi191 ceph-mon[32884]: osdmap e156: 8 total, 8 up, 6 in 2024-09-10T19:16:27.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:16:26 smithi191 ceph-mon[32884]: from='client.? 172.21.15.183:0/1527339839' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T19:16:27.239 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:16:27.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:16:27.240 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:16:27.341 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:16:27.379 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:16:27.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:16:27.380 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:16:27.480 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:16:27.518 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:16:27.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:16:27.519 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:16:27.619 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:16:27.633 DEBUG:teuthology.orchestra.run.smithi191:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph daemon osd.5 dump_blocked_ops 2024-09-10T19:16:27.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:16:27.651 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:16:27.651 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:16:27.716 INFO:journalctl@ceph.mgr.x.smithi191.stdout:Sep 10 19:16:27 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-x[34967]: ::ffff:172.21.15.191 - - [10/Sep/2024:19:16:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T19:16:27.738 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:16:27.752 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:16:27.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:16:27.791 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:16:27.791 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:16:27.820 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: 3 16 10733 10717 223.255 211.125 0.00799121 0.00447151 2024-09-10T19:16:27.870 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:16:27 smithi183 ceph-mon[29736]: 5.c scrub starts 2024-09-10T19:16:27.972 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:16:27 smithi183 ceph-mon[29736]: 5.c scrub ok 2024-09-10T19:16:27.972 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:16:27 smithi183 ceph-mon[29736]: 5.b scrub starts 2024-09-10T19:16:27.972 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:16:27 smithi183 ceph-mon[29736]: 5.b scrub ok 2024-09-10T19:16:27.972 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:16:27 smithi183 ceph-mon[29736]: 5.19 scrub starts 2024-09-10T19:16:27.973 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:16:27 smithi183 ceph-mon[29736]: 5.19 scrub ok 2024-09-10T19:16:27.973 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:16:27 smithi183 ceph-mon[41562]: 5.c scrub starts 2024-09-10T19:16:27.973 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:16:27 smithi183 ceph-mon[41562]: 5.c scrub ok 2024-09-10T19:16:27.973 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:16:27 smithi183 ceph-mon[41562]: 5.b scrub starts 2024-09-10T19:16:27.973 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:16:27 smithi183 ceph-mon[41562]: 5.b scrub ok 2024-09-10T19:16:27.973 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:16:27 smithi183 ceph-mon[41562]: 5.19 scrub starts 2024-09-10T19:16:27.973 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:16:27 smithi183 ceph-mon[41562]: 5.19 scrub ok 2024-09-10T19:16:27.974 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:16:28.024 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:16:28.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:16:28.025 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:16:28.035 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:16:27 smithi191 ceph-mon[32884]: 5.c scrub starts 2024-09-10T19:16:28.035 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:16:27 smithi191 ceph-mon[32884]: 5.c scrub ok 2024-09-10T19:16:28.035 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:16:27 smithi191 ceph-mon[32884]: 5.b scrub starts 2024-09-10T19:16:28.036 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:16:27 smithi191 ceph-mon[32884]: 5.b scrub ok 2024-09-10T19:16:28.036 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:16:27 smithi191 ceph-mon[32884]: 5.19 scrub starts 2024-09-10T19:16:28.036 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:16:27 smithi191 ceph-mon[32884]: 5.19 scrub ok 2024-09-10T19:16:28.093 INFO:teuthology.orchestra.run.smithi191.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.b/config 2024-09-10T19:16:28.126 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:16:28.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:16:28.174 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:16:28.174 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:16:28.275 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:16:28.306 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:16:28.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:16:28.307 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:16:28.408 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:16:28.445 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:16:28.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:16:28.446 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:16:28.547 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:16:28.582 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:16:28.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:16:28.583 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:16:28.684 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:16:28.715 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:16:28.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:16:28.716 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:16:28.817 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:16:28.820 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: 4 16 13239 13223 206.594 156.625 0.00481614 0.00483255 2024-09-10T19:16:28.856 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:16:28.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:16:28.858 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:16:28.959 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:16:28.985 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:16:28.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:16:28.986 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:16:29.061 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:16:28 smithi191 ceph-mon[32884]: purged_snaps scrub starts 2024-09-10T19:16:29.061 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:16:28 smithi191 ceph-mon[32884]: purged_snaps scrub ok 2024-09-10T19:16:29.061 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:16:28 smithi191 ceph-mon[32884]: 2.14 scrub starts 2024-09-10T19:16:29.061 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:16:28 smithi191 ceph-mon[32884]: 2.14 scrub ok 2024-09-10T19:16:29.061 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:16:28 smithi191 ceph-mon[32884]: pgmap v258: 148 pgs: 37 peering, 111 active+clean; 14 MiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.7 MiB/s wr, 40 op/s; 0 B/s, 2 objects/s recovering 2024-09-10T19:16:29.062 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:16:28 smithi191 ceph-mon[32884]: osdmap e157: 8 total, 8 up, 6 in 2024-09-10T19:16:29.062 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:16:28 smithi191 ceph-mon[32884]: purged_snaps scrub starts 2024-09-10T19:16:29.062 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:16:28 smithi191 ceph-mon[32884]: purged_snaps scrub ok 2024-09-10T19:16:29.062 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:16:28 smithi191 ceph-mon[32884]: from='mgr.14556 ' entity='mgr.y' 2024-09-10T19:16:29.062 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:16:28 smithi191 ceph-mon[32884]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:16:29.062 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:16:28 smithi191 ceph-mon[32884]: 5.17 scrub starts 2024-09-10T19:16:29.062 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:16:28 smithi191 ceph-mon[32884]: 5.17 scrub ok 2024-09-10T19:16:29.086 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:16:29.117 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:16:29.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:16:29.118 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:16:29.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:16:28 smithi183 ceph-mon[29736]: purged_snaps scrub starts 2024-09-10T19:16:29.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:16:28 smithi183 ceph-mon[29736]: purged_snaps scrub ok 2024-09-10T19:16:29.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:16:28 smithi183 ceph-mon[29736]: 2.14 scrub starts 2024-09-10T19:16:29.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:16:28 smithi183 ceph-mon[29736]: 2.14 scrub ok 2024-09-10T19:16:29.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:16:28 smithi183 ceph-mon[29736]: pgmap v258: 148 pgs: 37 peering, 111 active+clean; 14 MiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.7 MiB/s wr, 40 op/s; 0 B/s, 2 objects/s recovering 2024-09-10T19:16:29.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:16:28 smithi183 ceph-mon[29736]: osdmap e157: 8 total, 8 up, 6 in 2024-09-10T19:16:29.182 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:16:28 smithi183 ceph-mon[29736]: purged_snaps scrub starts 2024-09-10T19:16:29.182 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:16:28 smithi183 ceph-mon[29736]: purged_snaps scrub ok 2024-09-10T19:16:29.182 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:16:28 smithi183 ceph-mon[29736]: from='mgr.14556 ' entity='mgr.y' 2024-09-10T19:16:29.182 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:16:28 smithi183 ceph-mon[29736]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:16:29.182 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:16:28 smithi183 ceph-mon[29736]: 5.17 scrub starts 2024-09-10T19:16:29.182 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:16:28 smithi183 ceph-mon[29736]: 5.17 scrub ok 2024-09-10T19:16:29.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:16:28 smithi183 ceph-mon[41562]: purged_snaps scrub starts 2024-09-10T19:16:29.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:16:28 smithi183 ceph-mon[41562]: purged_snaps scrub ok 2024-09-10T19:16:29.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:16:28 smithi183 ceph-mon[41562]: 2.14 scrub starts 2024-09-10T19:16:29.183 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:16:28 smithi183 ceph-mon[41562]: 2.14 scrub ok 2024-09-10T19:16:29.183 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:16:28 smithi183 ceph-mon[41562]: pgmap v258: 148 pgs: 37 peering, 111 active+clean; 14 MiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.7 MiB/s wr, 40 op/s; 0 B/s, 2 objects/s recovering 2024-09-10T19:16:29.183 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:16:28 smithi183 ceph-mon[41562]: osdmap e157: 8 total, 8 up, 6 in 2024-09-10T19:16:29.183 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:16:28 smithi183 ceph-mon[41562]: purged_snaps scrub starts 2024-09-10T19:16:29.183 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:16:28 smithi183 ceph-mon[41562]: purged_snaps scrub ok 2024-09-10T19:16:29.183 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:16:28 smithi183 ceph-mon[41562]: from='mgr.14556 ' entity='mgr.y' 2024-09-10T19:16:29.183 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:16:28 smithi183 ceph-mon[41562]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:16:29.183 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:16:28 smithi183 ceph-mon[41562]: 5.17 scrub starts 2024-09-10T19:16:29.183 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:16:28 smithi183 ceph-mon[41562]: 5.17 scrub ok 2024-09-10T19:16:29.219 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:16:29.248 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:16:29.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:16:29.249 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:16:29.349 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:16:29.385 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:16:29.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:16:29.386 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:16:29.465 INFO:journalctl@ceph.iscsi.iscsi.a.smithi191.stdout:Sep 10 19:16:29 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-iscsi-iscsi-a[61816]: debug there is no tcmu-runner data available 2024-09-10T19:16:29.487 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:16:29.520 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:16:29.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:16:29.521 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:16:29.622 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:16:29.630 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-10T19:16:29.661 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:16:29.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:16:29.662 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:16:29.762 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:16:29.799 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:16:29.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:16:29.800 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:16:29.821 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: 5 16 15846 15830 197.822 162.938 0.00546482 0.00504568 2024-09-10T19:16:29.901 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:16:29.935 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:16:29.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:16:29.937 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:16:29.965 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:16:29 smithi191 ceph-mon[32884]: 6.4 scrub starts 2024-09-10T19:16:29.965 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:16:29 smithi191 ceph-mon[32884]: 6.4 scrub ok 2024-09-10T19:16:29.966 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:16:29 smithi191 ceph-mon[32884]: osdmap e158: 8 total, 8 up, 6 in 2024-09-10T19:16:29.966 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:16:29 smithi191 ceph-mon[32884]: 2.f scrub starts 2024-09-10T19:16:29.966 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:16:29 smithi191 ceph-mon[32884]: from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:16:30.037 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:16:30.071 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:16:30.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:16:30.072 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:16:30.096 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:16:29 smithi183 ceph-mon[41562]: 6.4 scrub starts 2024-09-10T19:16:30.096 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:16:29 smithi183 ceph-mon[41562]: 6.4 scrub ok 2024-09-10T19:16:30.096 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:16:29 smithi183 ceph-mon[41562]: osdmap e158: 8 total, 8 up, 6 in 2024-09-10T19:16:30.096 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:16:29 smithi183 ceph-mon[41562]: 2.f scrub starts 2024-09-10T19:16:30.096 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:16:29 smithi183 ceph-mon[41562]: from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:16:30.096 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:16:29 smithi183 ceph-mon[29736]: 6.4 scrub starts 2024-09-10T19:16:30.096 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:16:29 smithi183 ceph-mon[29736]: 6.4 scrub ok 2024-09-10T19:16:30.097 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:16:29 smithi183 ceph-mon[29736]: osdmap e158: 8 total, 8 up, 6 in 2024-09-10T19:16:30.097 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:16:29 smithi183 ceph-mon[29736]: 2.f scrub starts 2024-09-10T19:16:30.097 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:16:29 smithi183 ceph-mon[29736]: from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:16:30.168 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:16:30.173 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:16:30.212 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:16:30.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:16:30.214 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:16:30.262 INFO:tasks.thrashosds.thrasher:in_osds: [0, 3, 4, 5, 6, 7] out_osds: [1, 2] dead_osds: [] live_osds: [0, 2, 3, 4, 5, 6, 7, 1] 2024-09-10T19:16:30.262 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-10T19:16:30.262 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap 2024-09-10T19:16:30.262 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph osd dump -f json-pretty 2024-09-10T19:16:30.316 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:16:30.353 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:16:30.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:16:30.354 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:16:30.455 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:16:30.500 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:16:30.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:16:30.501 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:16:30.602 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:16:30.641 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:16:30.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:16:30.642 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:16:30.743 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:16:30.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:16:30.785 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:16:30.785 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:16:30.821 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: 6 16 18480 18464 192.288 164.625 0.00672088 0.00519479 2024-09-10T19:16:30.867 INFO:teuthology.orchestra.run.smithi183.stderr:nodeep-scrub is unset 2024-09-10T19:16:30.886 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:16:30.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:16:30.922 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:16:30.922 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:16:31.023 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:16:31.050 INFO:teuthology.orchestra.run.smithi191.stdout:ERROR: (22) Invalid argument 2024-09-10T19:16:31.050 INFO:teuthology.orchestra.run.smithi191.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-10T19:16:31.054 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:16:31.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:16:31.055 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:16:31.156 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:16:31.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:16:30 smithi183 ceph-mon[29736]: 2.f scrub ok 2024-09-10T19:16:31.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:16:30 smithi183 ceph-mon[29736]: 5.d scrub starts 2024-09-10T19:16:31.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:16:30 smithi183 ceph-mon[29736]: 5.d scrub ok 2024-09-10T19:16:31.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:16:30 smithi183 ceph-mon[29736]: pgmap v261: 148 pgs: 13 peering, 135 active+clean; 375 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail; 2.9 KiB/s rd, 77 MiB/s wr, 1.23k op/s; 92 B/s, 10 objects/s recovering 2024-09-10T19:16:31.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:16:30 smithi183 ceph-mon[29736]: from='client.? 172.21.15.183:0/3305202469' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-10T19:16:31.182 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:16:30 smithi183 ceph-mon[29736]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-10T19:16:31.182 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:16:30 smithi183 ceph-mon[29736]: 2.1e scrub starts 2024-09-10T19:16:31.182 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:16:30 smithi183 ceph-mon[29736]: 2.1e scrub ok 2024-09-10T19:16:31.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:16:30 smithi183 ceph-mon[41562]: 2.f scrub ok 2024-09-10T19:16:31.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:16:30 smithi183 ceph-mon[41562]: 5.d scrub starts 2024-09-10T19:16:31.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:16:30 smithi183 ceph-mon[41562]: 5.d scrub ok 2024-09-10T19:16:31.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:16:30 smithi183 ceph-mon[41562]: pgmap v261: 148 pgs: 13 peering, 135 active+clean; 375 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail; 2.9 KiB/s rd, 77 MiB/s wr, 1.23k op/s; 92 B/s, 10 objects/s recovering 2024-09-10T19:16:31.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:16:30 smithi183 ceph-mon[41562]: from='client.? 172.21.15.183:0/3305202469' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-10T19:16:31.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:16:30 smithi183 ceph-mon[41562]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-10T19:16:31.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:16:30 smithi183 ceph-mon[41562]: 2.1e scrub starts 2024-09-10T19:16:31.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:16:30 smithi183 ceph-mon[41562]: 2.1e scrub ok 2024-09-10T19:16:31.189 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:16:31.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:16:31.191 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:16:31.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:16:30 smithi191 ceph-mon[32884]: 2.f scrub ok 2024-09-10T19:16:31.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:16:30 smithi191 ceph-mon[32884]: 5.d scrub starts 2024-09-10T19:16:31.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:16:30 smithi191 ceph-mon[32884]: 5.d scrub ok 2024-09-10T19:16:31.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:16:30 smithi191 ceph-mon[32884]: pgmap v261: 148 pgs: 13 peering, 135 active+clean; 375 MiB data, 2.7 GiB used, 534 GiB / 536 GiB avail; 2.9 KiB/s rd, 77 MiB/s wr, 1.23k op/s; 92 B/s, 10 objects/s recovering 2024-09-10T19:16:31.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:16:30 smithi191 ceph-mon[32884]: from='client.? 172.21.15.183:0/3305202469' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-10T19:16:31.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:16:30 smithi191 ceph-mon[32884]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-10T19:16:31.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:16:30 smithi191 ceph-mon[32884]: 2.1e scrub starts 2024-09-10T19:16:31.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:16:30 smithi191 ceph-mon[32884]: 2.1e scrub ok 2024-09-10T19:16:31.292 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:16:31.323 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:16:31.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:16:31.324 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:16:31.425 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:16:31.458 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:16:31.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:16:31.459 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:16:31.560 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:16:31.596 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:16:31.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:16:31.597 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:16:31.681 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Sep 10 19:16:31 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-y[29989]: ::ffff:172.21.15.191 - - [10/Sep/2024:19:16:31] "GET /metrics HTTP/1.1" 200 223927 "" "Prometheus/2.43.0" 2024-09-10T19:16:31.698 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:16:31.738 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:16:31.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:16:31.739 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:16:31.764 DEBUG:teuthology.orchestra.run.smithi191:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph daemon osd.5 dump_historic_ops 2024-09-10T19:16:31.821 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: 7 16 21027 21011 187.558 159.188 0.00280293 0.00532531 2024-09-10T19:16:31.840 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:16:31.866 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:16:31.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:16:31.867 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:16:31.968 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:16:32.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:16:32.010 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:16:32.010 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:16:32.111 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:16:32.157 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:16:32.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:16:32.159 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:16:32.181 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:16:31 smithi183 ceph-mon[41562]: 6.2 scrub starts 2024-09-10T19:16:32.181 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:16:31 smithi183 ceph-mon[41562]: 6.2 scrub ok 2024-09-10T19:16:32.181 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:16:31 smithi183 ceph-mon[41562]: 2.1a scrub starts 2024-09-10T19:16:32.181 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:16:31 smithi183 ceph-mon[41562]: 2.1a scrub ok 2024-09-10T19:16:32.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:16:31 smithi183 ceph-mon[41562]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-10T19:16:32.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:16:31 smithi183 ceph-mon[41562]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-10T19:16:32.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:16:31 smithi183 ceph-mon[41562]: osdmap e159: 8 total, 8 up, 6 in 2024-09-10T19:16:32.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:16:31 smithi183 ceph-mon[41562]: 5.1 deep-scrub starts 2024-09-10T19:16:32.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:16:31 smithi183 ceph-mon[41562]: 5.1 deep-scrub ok 2024-09-10T19:16:32.182 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:16:31 smithi183 ceph-mon[29736]: 6.2 scrub starts 2024-09-10T19:16:32.182 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:16:31 smithi183 ceph-mon[29736]: 6.2 scrub ok 2024-09-10T19:16:32.182 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:16:31 smithi183 ceph-mon[29736]: 2.1a scrub starts 2024-09-10T19:16:32.182 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:16:31 smithi183 ceph-mon[29736]: 2.1a scrub ok 2024-09-10T19:16:32.182 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:16:31 smithi183 ceph-mon[29736]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-10T19:16:32.182 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:16:31 smithi183 ceph-mon[29736]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-10T19:16:32.182 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:16:31 smithi183 ceph-mon[29736]: osdmap e159: 8 total, 8 up, 6 in 2024-09-10T19:16:32.183 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:16:31 smithi183 ceph-mon[29736]: 5.1 deep-scrub starts 2024-09-10T19:16:32.183 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:16:31 smithi183 ceph-mon[29736]: 5.1 deep-scrub ok 2024-09-10T19:16:32.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:16:31 smithi191 ceph-mon[32884]: 6.2 scrub starts 2024-09-10T19:16:32.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:16:31 smithi191 ceph-mon[32884]: 6.2 scrub ok 2024-09-10T19:16:32.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:16:31 smithi191 ceph-mon[32884]: 2.1a scrub starts 2024-09-10T19:16:32.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:16:31 smithi191 ceph-mon[32884]: 2.1a scrub ok 2024-09-10T19:16:32.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:16:31 smithi191 ceph-mon[32884]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-10T19:16:32.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:16:31 smithi191 ceph-mon[32884]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-10T19:16:32.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:16:31 smithi191 ceph-mon[32884]: osdmap e159: 8 total, 8 up, 6 in 2024-09-10T19:16:32.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:16:31 smithi191 ceph-mon[32884]: 5.1 deep-scrub starts 2024-09-10T19:16:32.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:16:31 smithi191 ceph-mon[32884]: 5.1 deep-scrub ok 2024-09-10T19:16:32.256 INFO:teuthology.orchestra.run.smithi191.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.b/config 2024-09-10T19:16:32.259 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:16:32.288 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:16:32.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:16:32.289 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:16:32.390 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:16:32.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:16:32.429 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:16:32.429 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:16:32.530 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:16:32.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:16:32.561 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:16:32.561 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:16:32.662 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:16:32.697 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:16:32.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:16:32.698 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:16:32.798 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:16:32.822 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: 8 16 23386 23370 182.532 147.438 0.00921804 0.00547175 2024-09-10T19:16:32.840 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:16:32.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:16:32.841 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:16:32.942 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:16:32.976 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:16:32.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:16:32.978 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:16:33.079 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:16:33.099 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:16:33.103 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:16:32 smithi183 ceph-mon[29736]: pgmap v263: 148 pgs: 148 active+clean; 992 MiB data, 4.4 GiB used, 532 GiB / 536 GiB avail; 3.2 KiB/s rd, 166 MiB/s wr, 2.66k op/s; 205 B/s, 6 objects/s recovering 2024-09-10T19:16:33.103 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:16:32 smithi183 ceph-mon[29736]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 2 pgs peering) 2024-09-10T19:16:33.103 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:16:32 smithi183 ceph-mon[29736]: Cluster is now healthy 2024-09-10T19:16:33.103 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:16:32 smithi183 ceph-mon[29736]: 6.6 scrub starts 2024-09-10T19:16:33.103 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:16:32 smithi183 ceph-mon[29736]: 6.6 scrub ok 2024-09-10T19:16:33.103 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:16:32 smithi183 ceph-mon[41562]: pgmap v263: 148 pgs: 148 active+clean; 992 MiB data, 4.4 GiB used, 532 GiB / 536 GiB avail; 3.2 KiB/s rd, 166 MiB/s wr, 2.66k op/s; 205 B/s, 6 objects/s recovering 2024-09-10T19:16:33.103 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:16:32 smithi183 ceph-mon[41562]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 2 pgs peering) 2024-09-10T19:16:33.103 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:16:32 smithi183 ceph-mon[41562]: Cluster is now healthy 2024-09-10T19:16:33.103 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:16:32 smithi183 ceph-mon[41562]: 6.6 scrub starts 2024-09-10T19:16:33.103 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:16:32 smithi183 ceph-mon[41562]: 6.6 scrub ok 2024-09-10T19:16:33.116 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:16:33.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:16:33.117 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:16:33.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:16:32 smithi191 ceph-mon[32884]: pgmap v263: 148 pgs: 148 active+clean; 992 MiB data, 4.4 GiB used, 532 GiB / 536 GiB avail; 3.2 KiB/s rd, 166 MiB/s wr, 2.66k op/s; 205 B/s, 6 objects/s recovering 2024-09-10T19:16:33.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:16:32 smithi191 ceph-mon[32884]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 2 pgs peering) 2024-09-10T19:16:33.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:16:32 smithi191 ceph-mon[32884]: Cluster is now healthy 2024-09-10T19:16:33.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:16:32 smithi191 ceph-mon[32884]: 6.6 scrub starts 2024-09-10T19:16:33.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:16:32 smithi191 ceph-mon[32884]: 6.6 scrub ok 2024-09-10T19:16:33.218 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:16:33.255 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:16:33.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:16:33.256 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:16:33.356 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:16:33.393 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:16:33.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:16:33.394 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:16:33.431 INFO:teuthology.orchestra.run.smithi183.stdout:osd.0: {} 2024-09-10T19:16:33.431 INFO:teuthology.orchestra.run.smithi183.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-10T19:16:33.457 INFO:teuthology.orchestra.run.smithi183.stdout:osd.1: {} 2024-09-10T19:16:33.458 INFO:teuthology.orchestra.run.smithi183.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-10T19:16:33.481 INFO:teuthology.orchestra.run.smithi183.stdout:osd.2: {} 2024-09-10T19:16:33.481 INFO:teuthology.orchestra.run.smithi183.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-10T19:16:33.495 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:16:33.505 INFO:teuthology.orchestra.run.smithi183.stdout:osd.3: {} 2024-09-10T19:16:33.505 INFO:teuthology.orchestra.run.smithi183.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-10T19:16:33.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:16:33.528 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:16:33.529 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:16:33.531 INFO:teuthology.orchestra.run.smithi183.stdout:osd.4: {} 2024-09-10T19:16:33.531 INFO:teuthology.orchestra.run.smithi183.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-10T19:16:33.561 INFO:teuthology.orchestra.run.smithi183.stdout:osd.5: {} 2024-09-10T19:16:33.561 INFO:teuthology.orchestra.run.smithi183.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-10T19:16:33.586 INFO:teuthology.orchestra.run.smithi183.stdout:osd.6: {} 2024-09-10T19:16:33.587 INFO:teuthology.orchestra.run.smithi183.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-10T19:16:33.611 INFO:teuthology.orchestra.run.smithi183.stdout:osd.7: {} 2024-09-10T19:16:33.611 INFO:teuthology.orchestra.run.smithi183.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-10T19:16:33.630 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:16:33.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:16:33.663 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:16:33.663 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:16:33.764 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:16:33.794 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:16:33.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:16:33.795 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:16:33.822 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: 9 16 25760 25744 178.736 148.375 0.00559811 0.00558914 2024-09-10T19:16:33.895 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:16:33.926 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:16:33.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:16:33.927 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:16:34.028 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:16:34.058 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:16:34.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:16:34.059 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:16:34.159 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:16:34.181 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:16:33 smithi183 ceph-mon[41562]: 3.0 deep-scrub starts 2024-09-10T19:16:34.181 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:16:33 smithi183 ceph-mon[41562]: 6.1c scrub starts 2024-09-10T19:16:34.181 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:16:33 smithi183 ceph-mon[41562]: 6.1c scrub ok 2024-09-10T19:16:34.181 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:16:33 smithi183 ceph-mon[41562]: 3.0 deep-scrub ok 2024-09-10T19:16:34.181 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:16:33 smithi183 ceph-mon[41562]: 4.d scrub starts 2024-09-10T19:16:34.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:16:33 smithi183 ceph-mon[41562]: 4.d scrub ok 2024-09-10T19:16:34.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:16:33 smithi183 ceph-mon[41562]: 4.1e scrub starts 2024-09-10T19:16:34.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:16:33 smithi183 ceph-mon[41562]: 4.1e scrub ok 2024-09-10T19:16:34.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:16:33 smithi183 ceph-mon[41562]: from='client.? 172.21.15.183:0/2591295928' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T19:16:34.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:16:33 smithi183 ceph-mon[41562]: 2.16 scrub starts 2024-09-10T19:16:34.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:16:33 smithi183 ceph-mon[41562]: 2.16 scrub ok 2024-09-10T19:16:34.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:16:33 smithi183 ceph-mon[41562]: 2.1 scrub starts 2024-09-10T19:16:34.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:16:33 smithi183 ceph-mon[41562]: 2.1 scrub ok 2024-09-10T19:16:34.182 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:16:33 smithi183 ceph-mon[29736]: 3.0 deep-scrub starts 2024-09-10T19:16:34.183 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:16:33 smithi183 ceph-mon[29736]: 6.1c scrub starts 2024-09-10T19:16:34.183 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:16:33 smithi183 ceph-mon[29736]: 6.1c scrub ok 2024-09-10T19:16:34.183 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:16:33 smithi183 ceph-mon[29736]: 3.0 deep-scrub ok 2024-09-10T19:16:34.183 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:16:33 smithi183 ceph-mon[29736]: 4.d scrub starts 2024-09-10T19:16:34.183 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:16:33 smithi183 ceph-mon[29736]: 4.d scrub ok 2024-09-10T19:16:34.183 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:16:33 smithi183 ceph-mon[29736]: 4.1e scrub starts 2024-09-10T19:16:34.183 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:16:33 smithi183 ceph-mon[29736]: 4.1e scrub ok 2024-09-10T19:16:34.183 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:16:33 smithi183 ceph-mon[29736]: from='client.? 172.21.15.183:0/2591295928' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T19:16:34.183 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:16:33 smithi183 ceph-mon[29736]: 2.16 scrub starts 2024-09-10T19:16:34.183 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:16:33 smithi183 ceph-mon[29736]: 2.16 scrub ok 2024-09-10T19:16:34.183 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:16:33 smithi183 ceph-mon[29736]: 2.1 scrub starts 2024-09-10T19:16:34.183 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:16:33 smithi183 ceph-mon[29736]: 2.1 scrub ok 2024-09-10T19:16:34.195 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:16:34.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:16:34.196 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:16:34.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:16:33 smithi191 ceph-mon[32884]: 3.0 deep-scrub starts 2024-09-10T19:16:34.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:16:33 smithi191 ceph-mon[32884]: 6.1c scrub starts 2024-09-10T19:16:34.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:16:33 smithi191 ceph-mon[32884]: 6.1c scrub ok 2024-09-10T19:16:34.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:16:33 smithi191 ceph-mon[32884]: 3.0 deep-scrub ok 2024-09-10T19:16:34.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:16:33 smithi191 ceph-mon[32884]: 4.d scrub starts 2024-09-10T19:16:34.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:16:33 smithi191 ceph-mon[32884]: 4.d scrub ok 2024-09-10T19:16:34.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:16:33 smithi191 ceph-mon[32884]: 4.1e scrub starts 2024-09-10T19:16:34.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:16:33 smithi191 ceph-mon[32884]: 4.1e scrub ok 2024-09-10T19:16:34.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:16:33 smithi191 ceph-mon[32884]: from='client.? 172.21.15.183:0/2591295928' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T19:16:34.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:16:33 smithi191 ceph-mon[32884]: 2.16 scrub starts 2024-09-10T19:16:34.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:16:33 smithi191 ceph-mon[32884]: 2.16 scrub ok 2024-09-10T19:16:34.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:16:33 smithi191 ceph-mon[32884]: 2.1 scrub starts 2024-09-10T19:16:34.217 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:16:33 smithi191 ceph-mon[32884]: 2.1 scrub ok 2024-09-10T19:16:34.297 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:16:34.336 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:16:34.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:16:34.337 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:16:34.438 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:16:34.468 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:16:34.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:16:34.469 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:16:34.570 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:16:34.599 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:16:34.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:16:34.600 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:16:34.701 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:16:34.722 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:16:34.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:16:34.723 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:16:34.822 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: 10 16 27643 27627 172.631 117.688 0.00647226 0.00578667 2024-09-10T19:16:34.823 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:16:34.852 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:16:34.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:16:34.853 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:16:34.855 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph osd set noscrub 2024-09-10T19:16:34.954 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:16:34.985 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:16:34.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:16:34.986 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:16:35.087 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:16:35.120 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:16:35.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:16:35.121 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:16:35.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:16:34 smithi183 ceph-mon[29736]: pgmap v264: 148 pgs: 148 active+clean; 992 MiB data, 4.4 GiB used, 532 GiB / 536 GiB avail; 3.2 KiB/s rd, 163 MiB/s wr, 2.61k op/s; 201 B/s, 5 objects/s recovering 2024-09-10T19:16:35.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:16:34 smithi183 ceph-mon[29736]: 5.1e scrub starts 2024-09-10T19:16:35.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:16:34 smithi183 ceph-mon[29736]: 5.1e scrub ok 2024-09-10T19:16:35.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:16:34 smithi183 ceph-mon[29736]: osdmap e160: 8 total, 8 up, 6 in 2024-09-10T19:16:35.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:16:34 smithi183 ceph-mon[41562]: pgmap v264: 148 pgs: 148 active+clean; 992 MiB data, 4.4 GiB used, 532 GiB / 536 GiB avail; 3.2 KiB/s rd, 163 MiB/s wr, 2.61k op/s; 201 B/s, 5 objects/s recovering 2024-09-10T19:16:35.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:16:34 smithi183 ceph-mon[41562]: 5.1e scrub starts 2024-09-10T19:16:35.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:16:34 smithi183 ceph-mon[41562]: 5.1e scrub ok 2024-09-10T19:16:35.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:16:34 smithi183 ceph-mon[41562]: osdmap e160: 8 total, 8 up, 6 in 2024-09-10T19:16:35.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:16:34 smithi191 ceph-mon[32884]: pgmap v264: 148 pgs: 148 active+clean; 992 MiB data, 4.4 GiB used, 532 GiB / 536 GiB avail; 3.2 KiB/s rd, 163 MiB/s wr, 2.61k op/s; 201 B/s, 5 objects/s recovering 2024-09-10T19:16:35.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:16:34 smithi191 ceph-mon[32884]: 5.1e scrub starts 2024-09-10T19:16:35.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:16:34 smithi191 ceph-mon[32884]: 5.1e scrub ok 2024-09-10T19:16:35.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:16:34 smithi191 ceph-mon[32884]: osdmap e160: 8 total, 8 up, 6 in 2024-09-10T19:16:35.222 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:16:35.257 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:16:35.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:16:35.258 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:16:35.358 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:16:35.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:16:35.389 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:16:35.389 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:16:35.490 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:16:35.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:16:35.522 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:16:35.523 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:16:35.624 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:16:35.655 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:16:35.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:16:35.656 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:16:35.756 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:16:35.786 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:16:35.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:16:35.787 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:16:35.822 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: 11 16 29546 29530 167.749 118.938 0.00185301 0.00594179 2024-09-10T19:16:35.888 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:16:35.909 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:16:35.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:16:35.910 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:16:36.011 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:16:36.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:16:36.046 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:16:36.046 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:16:36.147 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:16:36.177 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:16:36.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:16:36.179 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:16:36.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:16:35 smithi183 ceph-mon[29736]: 4.3 scrub starts 2024-09-10T19:16:36.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:16:35 smithi183 ceph-mon[29736]: 4.3 scrub ok 2024-09-10T19:16:36.181 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:16:35 smithi183 ceph-mon[41562]: 4.3 scrub starts 2024-09-10T19:16:36.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:16:35 smithi183 ceph-mon[41562]: 4.3 scrub ok 2024-09-10T19:16:36.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:16:35 smithi191 ceph-mon[32884]: 4.3 scrub starts 2024-09-10T19:16:36.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:16:35 smithi191 ceph-mon[32884]: 4.3 scrub ok 2024-09-10T19:16:36.279 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:16:36.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:16:36.302 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:16:36.302 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:16:36.403 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:16:36.433 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:16:36.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:16:36.434 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:16:36.535 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:16:36.569 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:16:36.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:16:36.571 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:16:36.671 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:16:36.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:16:36.694 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:16:36.695 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:16:36.795 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:16:36.823 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: 12 16 30951 30935 161.079 87.8125 0.006743 0.00620324 2024-09-10T19:16:36.826 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:16:36.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:16:36.827 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:16:36.927 INFO:teuthology.orchestra.run.smithi191.stdout:{ 2024-09-10T19:16:36.927 INFO:teuthology.orchestra.run.smithi191.stdout: "size": 20, 2024-09-10T19:16:36.927 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 600, 2024-09-10T19:16:36.927 INFO:teuthology.orchestra.run.smithi191.stdout: "ops": [ 2024-09-10T19:16:36.927 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:36.927 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.14781.0:205540 7.b 7:d286f7dc:::benchmark_data_smithi183_84160_object65159:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e126)", 2024-09-10T19:16:36.928 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-10T19:14:45.772065+0000", 2024-09-10T19:16:36.928 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 109.777120541, 2024-09-10T19:16:36.928 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.98751803299999996, 2024-09-10T19:16:36.928 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-10T19:16:36.928 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T19:16:36.928 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-10T19:16:36.928 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.14781", 2024-09-10T19:16:36.928 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.183:0/2025776420", 2024-09-10T19:16:36.928 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 205540 2024-09-10T19:16:36.928 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:36.928 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-10T19:16:36.928 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:36.928 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-10T19:16:36.929 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:45.772065+0000", 2024-09-10T19:16:36.929 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:16:36.929 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:36.929 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:36.929 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-10T19:16:36.929 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:45.772065+0000", 2024-09-10T19:16:36.929 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:16:36.929 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:36.929 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:36.929 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-10T19:16:36.929 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:45.772064+0000", 2024-09-10T19:16:36.930 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.999999 2024-09-10T19:16:36.930 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:36.930 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:36.930 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-10T19:16:36.930 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:45.772071+0000", 2024-09-10T19:16:36.930 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.503e-06 2024-09-10T19:16:36.930 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:36.930 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:36.930 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-10T19:16:36.930 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:45.772082+0000", 2024-09-10T19:16:36.930 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.0294e-05 2024-09-10T19:16:36.930 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:36.931 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:36.931 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-10T19:16:36.931 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:45.772087+0000", 2024-09-10T19:16:36.931 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.643e-06 2024-09-10T19:16:36.931 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:36.931 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:36.931 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:16:36.931 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:45.772132+0000", 2024-09-10T19:16:36.931 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.4378000000000001e-05 2024-09-10T19:16:36.931 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:36.931 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:36.931 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for peered", 2024-09-10T19:16:36.932 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:45.772140+0000", 2024-09-10T19:16:36.932 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.7520000000000002e-06 2024-09-10T19:16:36.932 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:36.932 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:36.932 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:16:36.932 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:46.752096+0000", 2024-09-10T19:16:36.932 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.97995554699999998 2024-09-10T19:16:36.932 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:36.932 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:36.932 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-10T19:16:36.932 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:46.752214+0000", 2024-09-10T19:16:36.932 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000118507 2024-09-10T19:16:36.932 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:36.933 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:36.933 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 2,6", 2024-09-10T19:16:36.933 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:46.752288+0000", 2024-09-10T19:16:36.933 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.339e-05 2024-09-10T19:16:36.933 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:36.933 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:36.933 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-09-10T19:16:36.933 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:46.757094+0000", 2024-09-10T19:16:36.933 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0048063569999999998 2024-09-10T19:16:36.933 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:36.933 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:36.933 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:16:36.934 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:46.759427+0000", 2024-09-10T19:16:36.934 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0023326050000000002 2024-09-10T19:16:36.934 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:36.934 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:36.934 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:16:36.934 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:46.759503+0000", 2024-09-10T19:16:36.934 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.5694999999999999e-05 2024-09-10T19:16:36.934 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:36.934 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:36.934 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-09-10T19:16:36.934 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:46.759549+0000", 2024-09-10T19:16:36.934 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.6634000000000001e-05 2024-09-10T19:16:36.934 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:36.935 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:36.935 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-10T19:16:36.935 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:46.759583+0000", 2024-09-10T19:16:36.935 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.3878000000000003e-05 2024-09-10T19:16:36.935 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:16:36.935 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-10T19:16:36.935 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:16:36.935 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:36.935 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:36.935 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.14781.0:205549 7.b 7:d0626604:::benchmark_data_smithi183_84160_object65168:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e126)", 2024-09-10T19:16:36.935 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-10T19:14:45.780874+0000", 2024-09-10T19:16:36.935 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 109.76831122900001, 2024-09-10T19:16:36.936 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.97887205499999996, 2024-09-10T19:16:36.936 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-10T19:16:36.936 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T19:16:36.936 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-10T19:16:36.936 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.14781", 2024-09-10T19:16:36.936 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.183:0/2025776420", 2024-09-10T19:16:36.936 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 205549 2024-09-10T19:16:36.936 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:36.936 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-10T19:16:36.937 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:36.937 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-10T19:16:36.937 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:45.780874+0000", 2024-09-10T19:16:36.937 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:16:36.937 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:36.937 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:36.937 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-10T19:16:36.937 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:45.780874+0000", 2024-09-10T19:16:36.937 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:16:36.937 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:36.937 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:36.937 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-10T19:16:36.938 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:45.780873+0000", 2024-09-10T19:16:36.938 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999986 2024-09-10T19:16:36.938 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:36.938 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:36.938 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-10T19:16:36.938 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:45.780879+0000", 2024-09-10T19:16:36.938 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.083e-06 2024-09-10T19:16:36.938 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:36.938 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:36.938 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-10T19:16:36.938 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:45.780892+0000", 2024-09-10T19:16:36.938 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.2496e-05 2024-09-10T19:16:36.938 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:36.938 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:36.939 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-10T19:16:36.939 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:45.780896+0000", 2024-09-10T19:16:36.939 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.4970000000000003e-06 2024-09-10T19:16:36.939 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:36.939 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:36.939 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:16:36.939 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:45.780945+0000", 2024-09-10T19:16:36.939 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.8618000000000001e-05 2024-09-10T19:16:36.939 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:36.939 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:36.939 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for peered", 2024-09-10T19:16:36.939 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:45.780954+0000", 2024-09-10T19:16:36.939 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.9220000000000003e-06 2024-09-10T19:16:36.939 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:36.940 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:36.940 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:16:36.940 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:46.754818+0000", 2024-09-10T19:16:36.940 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.97386392899999996 2024-09-10T19:16:36.940 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:36.940 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:36.940 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-10T19:16:36.940 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:46.754913+0000", 2024-09-10T19:16:36.940 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.5884000000000006e-05 2024-09-10T19:16:36.940 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:36.940 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:36.940 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 2,6", 2024-09-10T19:16:36.941 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:46.755028+0000", 2024-09-10T19:16:36.941 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00011414 2024-09-10T19:16:36.941 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:36.941 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:36.941 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:16:36.941 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:46.759634+0000", 2024-09-10T19:16:36.941 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0046064060000000004 2024-09-10T19:16:36.941 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:36.941 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:36.941 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-09-10T19:16:36.941 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:46.759643+0000", 2024-09-10T19:16:36.941 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.2660000000000007e-06 2024-09-10T19:16:36.941 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:36.942 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:36.942 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:16:36.942 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:46.759693+0000", 2024-09-10T19:16:36.942 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.9382000000000003e-05 2024-09-10T19:16:36.942 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:36.942 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:36.942 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-09-10T19:16:36.942 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:46.759725+0000", 2024-09-10T19:16:36.942 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.2830999999999998e-05 2024-09-10T19:16:36.942 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:36.942 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:36.942 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-10T19:16:36.942 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:46.759746+0000", 2024-09-10T19:16:36.943 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.0885000000000001e-05 2024-09-10T19:16:36.943 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:16:36.943 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-10T19:16:36.943 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:16:36.943 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:36.943 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:36.943 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.14781.0:205550 7.b 7:d6a1ae8a:::benchmark_data_smithi183_84160_object65169:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e126)", 2024-09-10T19:16:36.943 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-10T19:14:45.780904+0000", 2024-09-10T19:16:36.943 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 109.768281731, 2024-09-10T19:16:36.943 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.98770159000000002, 2024-09-10T19:16:36.943 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-10T19:16:36.943 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T19:16:36.944 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-10T19:16:36.944 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.14781", 2024-09-10T19:16:36.944 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.183:0/2025776420", 2024-09-10T19:16:36.944 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 205550 2024-09-10T19:16:36.944 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:36.944 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-10T19:16:36.944 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:36.944 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-10T19:16:36.944 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:45.780904+0000", 2024-09-10T19:16:36.944 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:16:36.944 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:36.944 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:36.945 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-10T19:16:36.945 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:45.780904+0000", 2024-09-10T19:16:36.945 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:16:36.945 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:36.945 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:36.945 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-10T19:16:36.945 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:45.780903+0000", 2024-09-10T19:16:36.945 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999995 2024-09-10T19:16:36.945 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:36.945 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:36.945 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-10T19:16:36.945 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:45.780906+0000", 2024-09-10T19:16:36.945 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.599e-06 2024-09-10T19:16:36.946 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:36.946 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:36.946 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-10T19:16:36.946 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:45.780911+0000", 2024-09-10T19:16:36.946 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.434e-06 2024-09-10T19:16:36.946 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:36.946 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:36.946 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-10T19:16:36.947 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:45.780915+0000", 2024-09-10T19:16:36.947 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.8070000000000001e-06 2024-09-10T19:16:36.947 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:36.947 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:36.947 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:16:36.947 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:45.780977+0000", 2024-09-10T19:16:36.947 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.2244999999999998e-05 2024-09-10T19:16:36.947 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:36.947 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:36.947 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for peered", 2024-09-10T19:16:36.947 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:45.780982+0000", 2024-09-10T19:16:36.947 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.6609999999999999e-06 2024-09-10T19:16:36.948 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:36.948 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:36.948 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:16:36.948 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:46.757132+0000", 2024-09-10T19:16:36.948 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.97615030000000003 2024-09-10T19:16:36.948 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:36.948 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:36.948 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-10T19:16:36.948 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:46.757251+0000", 2024-09-10T19:16:36.948 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000118434 2024-09-10T19:16:36.948 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:36.948 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:36.948 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 2,6", 2024-09-10T19:16:36.949 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:46.757358+0000", 2024-09-10T19:16:36.949 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000107594 2024-09-10T19:16:36.949 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:36.949 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:36.949 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-09-10T19:16:36.949 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:46.762125+0000", 2024-09-10T19:16:36.949 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0047661559999999997 2024-09-10T19:16:36.949 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:36.949 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:36.949 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:16:36.949 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:46.762162+0000", 2024-09-10T19:16:36.949 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.6940999999999997e-05 2024-09-10T19:16:36.950 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:36.950 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:36.950 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:16:36.950 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:46.768527+0000", 2024-09-10T19:16:36.950 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0063658769999999998 2024-09-10T19:16:36.950 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:36.950 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:36.950 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-09-10T19:16:36.950 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:46.768571+0000", 2024-09-10T19:16:36.950 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.3745000000000002e-05 2024-09-10T19:16:36.950 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:36.950 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:36.950 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-10T19:16:36.951 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:46.768605+0000", 2024-09-10T19:16:36.951 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.4190000000000003e-05 2024-09-10T19:16:36.951 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:16:36.951 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-10T19:16:36.951 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:16:36.951 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:36.951 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:36.951 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.14781.0:205573 7.b 7:dc52318a:::benchmark_data_smithi183_84160_object65192:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e126)", 2024-09-10T19:16:36.951 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-10T19:14:45.813712+0000", 2024-09-10T19:16:36.951 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 109.73547398300001, 2024-09-10T19:16:36.951 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.96058740200000003, 2024-09-10T19:16:36.951 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-10T19:16:36.951 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T19:16:36.952 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-10T19:16:36.952 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.14781", 2024-09-10T19:16:36.952 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.183:0/2025776420", 2024-09-10T19:16:36.952 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 205573 2024-09-10T19:16:36.952 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:36.952 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-10T19:16:36.952 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:36.952 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-10T19:16:36.952 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:45.813712+0000", 2024-09-10T19:16:36.952 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:16:36.952 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:36.952 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:36.952 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-10T19:16:36.953 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:45.813712+0000", 2024-09-10T19:16:36.953 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:16:36.953 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:36.953 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:36.953 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-10T19:16:36.953 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:45.813710+0000", 2024-09-10T19:16:36.953 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999986 2024-09-10T19:16:36.953 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:36.953 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:36.953 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-10T19:16:36.953 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:45.813716+0000", 2024-09-10T19:16:36.953 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.2530000000000001e-06 2024-09-10T19:16:36.953 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:36.954 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:36.954 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-10T19:16:36.954 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:45.813728+0000", 2024-09-10T19:16:36.954 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.1973e-05 2024-09-10T19:16:36.954 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:36.954 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:36.954 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-10T19:16:36.954 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:45.813734+0000", 2024-09-10T19:16:36.954 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.0190000000000003e-06 2024-09-10T19:16:36.954 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:36.954 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:36.954 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:16:36.954 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:45.813784+0000", 2024-09-10T19:16:36.955 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.9521999999999998e-05 2024-09-10T19:16:36.955 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:36.955 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:36.955 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for peered", 2024-09-10T19:16:36.955 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:45.813793+0000", 2024-09-10T19:16:36.955 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.9649999999999997e-06 2024-09-10T19:16:36.955 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:36.955 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:36.955 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:16:36.956 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:46.759812+0000", 2024-09-10T19:16:36.956 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.94601953400000005 2024-09-10T19:16:36.956 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:36.956 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:36.956 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-10T19:16:36.956 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:46.759918+0000", 2024-09-10T19:16:36.956 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000106068 2024-09-10T19:16:36.956 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:36.956 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:36.956 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 2,6", 2024-09-10T19:16:36.956 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:46.760040+0000", 2024-09-10T19:16:36.956 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000121787 2024-09-10T19:16:36.957 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:36.957 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:36.957 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:16:36.957 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:46.768637+0000", 2024-09-10T19:16:36.957 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.008596645 2024-09-10T19:16:36.957 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:36.957 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:36.957 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:16:36.958 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:46.768662+0000", 2024-09-10T19:16:36.958 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.5069e-05 2024-09-10T19:16:36.958 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:36.958 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:36.958 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-09-10T19:16:36.958 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:46.774214+0000", 2024-09-10T19:16:36.958 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0055521820000000001 2024-09-10T19:16:36.958 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:36.958 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:36.958 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-09-10T19:16:36.958 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:46.774261+0000", 2024-09-10T19:16:36.958 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.6937000000000001e-05 2024-09-10T19:16:36.958 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:36.959 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:36.959 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-10T19:16:36.959 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:46.774299+0000", 2024-09-10T19:16:36.959 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.7954000000000003e-05 2024-09-10T19:16:36.959 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:16:36.959 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-10T19:16:36.959 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:16:36.959 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:36.959 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:36.959 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.14781.0:205604 7.b 7:dd1bf265:::benchmark_data_smithi183_84160_object65223:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e126)", 2024-09-10T19:16:36.959 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-10T19:14:45.846095+0000", 2024-09-10T19:16:36.959 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 109.70309093500001, 2024-09-10T19:16:36.960 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.92835953699999996, 2024-09-10T19:16:36.960 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-10T19:16:36.960 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T19:16:36.960 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-10T19:16:36.960 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.14781", 2024-09-10T19:16:36.960 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.183:0/2025776420", 2024-09-10T19:16:36.960 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 205604 2024-09-10T19:16:36.960 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:36.960 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-10T19:16:36.960 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:36.960 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-10T19:16:36.960 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:45.846095+0000", 2024-09-10T19:16:36.960 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:16:36.961 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:36.961 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:36.961 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-10T19:16:36.961 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:45.846095+0000", 2024-09-10T19:16:36.961 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:16:36.961 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:36.961 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:36.961 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-10T19:16:36.961 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:45.846093+0000", 2024-09-10T19:16:36.961 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999986 2024-09-10T19:16:36.961 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:36.961 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:36.962 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-10T19:16:36.962 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:45.846100+0000", 2024-09-10T19:16:36.962 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.4829999999999996e-06 2024-09-10T19:16:36.962 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:36.962 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:36.962 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-10T19:16:36.962 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:45.846114+0000", 2024-09-10T19:16:36.962 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.3798000000000001e-05 2024-09-10T19:16:36.962 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:36.962 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:36.962 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-10T19:16:36.962 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:45.846120+0000", 2024-09-10T19:16:36.962 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.1589999999999998e-06 2024-09-10T19:16:36.963 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:36.963 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:36.963 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:16:36.963 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:45.846173+0000", 2024-09-10T19:16:36.963 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.3520000000000003e-05 2024-09-10T19:16:36.963 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:36.963 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:36.963 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for peered", 2024-09-10T19:16:36.963 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:45.846182+0000", 2024-09-10T19:16:36.963 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.9679999999999995e-06 2024-09-10T19:16:36.963 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:36.963 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:36.963 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:16:36.964 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:46.762183+0000", 2024-09-10T19:16:36.964 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.91600060800000005 2024-09-10T19:16:36.964 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:36.964 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:36.964 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-10T19:16:36.964 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:46.762276+0000", 2024-09-10T19:16:36.964 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.3078000000000006e-05 2024-09-10T19:16:36.964 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:36.964 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:36.964 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 2,6", 2024-09-10T19:16:36.964 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:46.762452+0000", 2024-09-10T19:16:36.964 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00017645200000000001 2024-09-10T19:16:36.965 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:36.965 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:36.965 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:16:36.965 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:46.768688+0000", 2024-09-10T19:16:36.965 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0062354749999999999 2024-09-10T19:16:36.965 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:36.965 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:36.965 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-09-10T19:16:36.965 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:46.774343+0000", 2024-09-10T19:16:36.965 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0056551559999999997 2024-09-10T19:16:36.965 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:36.965 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:36.965 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:16:36.966 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:46.774388+0000", 2024-09-10T19:16:36.966 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.4580999999999999e-05 2024-09-10T19:16:36.966 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:36.966 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:36.966 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-09-10T19:16:36.966 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:46.774422+0000", 2024-09-10T19:16:36.966 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.4808000000000003e-05 2024-09-10T19:16:36.966 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:36.966 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:36.966 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-10T19:16:36.966 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:46.774454+0000", 2024-09-10T19:16:36.966 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.1767999999999998e-05 2024-09-10T19:16:36.966 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:16:36.967 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-10T19:16:36.967 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:16:36.967 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:36.967 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:36.967 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.14781.0:205612 7.b 7:d8a28c2c:::benchmark_data_smithi183_84160_object65231:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e126)", 2024-09-10T19:16:36.967 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-10T19:14:45.852823+0000", 2024-09-10T19:16:36.967 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 109.69636226999999, 2024-09-10T19:16:36.967 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.92173129899999995, 2024-09-10T19:16:36.967 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-10T19:16:36.967 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T19:16:36.968 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-10T19:16:36.968 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.14781", 2024-09-10T19:16:36.968 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.183:0/2025776420", 2024-09-10T19:16:36.968 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 205612 2024-09-10T19:16:36.968 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:36.968 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-10T19:16:36.968 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:36.968 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-10T19:16:36.968 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:45.852823+0000", 2024-09-10T19:16:36.968 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:16:36.968 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:36.969 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:36.969 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-10T19:16:36.969 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:45.852823+0000", 2024-09-10T19:16:36.969 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:16:36.969 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:36.969 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:36.969 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-10T19:16:36.969 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:45.852822+0000", 2024-09-10T19:16:36.969 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999986 2024-09-10T19:16:36.969 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:36.969 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:36.969 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-10T19:16:36.969 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:45.852832+0000", 2024-09-10T19:16:36.970 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.0171e-05 2024-09-10T19:16:36.970 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:36.970 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:36.970 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-10T19:16:36.970 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:45.852848+0000", 2024-09-10T19:16:36.970 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.6138999999999999e-05 2024-09-10T19:16:36.970 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:36.970 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:36.970 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-10T19:16:36.970 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:45.852855+0000", 2024-09-10T19:16:36.970 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.6599999999999998e-06 2024-09-10T19:16:36.970 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:36.970 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:36.970 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:16:36.971 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:45.852914+0000", 2024-09-10T19:16:36.971 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.9175000000000002e-05 2024-09-10T19:16:36.971 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:36.971 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:36.971 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for peered", 2024-09-10T19:16:36.971 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:45.852923+0000", 2024-09-10T19:16:36.971 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.0629999999999995e-06 2024-09-10T19:16:36.971 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:36.971 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:36.971 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:16:36.971 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:46.764866+0000", 2024-09-10T19:16:36.971 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.91194273599999998 2024-09-10T19:16:36.971 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:36.972 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:36.972 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-10T19:16:36.972 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:46.764994+0000", 2024-09-10T19:16:36.972 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00012866700000000001 2024-09-10T19:16:36.972 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:36.972 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:36.972 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 2,6", 2024-09-10T19:16:36.972 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:46.765114+0000", 2024-09-10T19:16:36.972 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000120102 2024-09-10T19:16:36.972 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:36.972 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:36.973 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:16:36.973 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:46.774338+0000", 2024-09-10T19:16:36.973 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0092231819999999999 2024-09-10T19:16:36.973 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:36.973 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:36.973 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-09-10T19:16:36.973 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:46.774347+0000", 2024-09-10T19:16:36.973 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.5219999999999996e-06 2024-09-10T19:16:36.973 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:36.973 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:36.973 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:16:36.973 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:46.774484+0000", 2024-09-10T19:16:36.973 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00013701699999999999 2024-09-10T19:16:36.974 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:36.974 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:36.974 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-09-10T19:16:36.974 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:46.774517+0000", 2024-09-10T19:16:36.974 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.2490999999999998e-05 2024-09-10T19:16:36.974 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:36.974 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:36.974 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-10T19:16:36.974 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:46.774555+0000", 2024-09-10T19:16:36.974 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.7857e-05 2024-09-10T19:16:36.974 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:16:36.974 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-10T19:16:36.974 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:16:36.974 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:36.975 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:36.975 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.14781.0:205654 7.b 7:dcb42d93:::benchmark_data_smithi183_84160_object65273:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e126)", 2024-09-10T19:16:36.975 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-10T19:14:45.896742+0000", 2024-09-10T19:16:36.975 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 109.65244396600001, 2024-09-10T19:16:36.975 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.88562795400000005, 2024-09-10T19:16:36.975 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-10T19:16:36.975 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T19:16:36.975 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-10T19:16:36.975 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.14781", 2024-09-10T19:16:36.975 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.183:0/2025776420", 2024-09-10T19:16:36.975 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 205654 2024-09-10T19:16:36.975 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:36.975 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-10T19:16:36.976 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:36.976 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-10T19:16:36.976 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:45.896742+0000", 2024-09-10T19:16:36.976 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:16:36.976 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:36.976 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:36.976 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-10T19:16:36.976 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:45.896742+0000", 2024-09-10T19:16:36.976 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:16:36.976 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:36.976 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:36.976 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-10T19:16:36.976 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:45.896740+0000", 2024-09-10T19:16:36.977 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.999999 2024-09-10T19:16:36.977 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:36.977 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:36.977 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-10T19:16:36.977 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:45.896747+0000", 2024-09-10T19:16:36.977 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.2890000000000003e-06 2024-09-10T19:16:36.977 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:36.977 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:36.977 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-10T19:16:36.977 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:45.896765+0000", 2024-09-10T19:16:36.977 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.8641999999999999e-05 2024-09-10T19:16:36.977 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:36.977 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:36.978 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-10T19:16:36.978 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:45.896772+0000", 2024-09-10T19:16:36.978 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.1840000000000004e-06 2024-09-10T19:16:36.978 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:36.978 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:36.978 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:16:36.978 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:45.896815+0000", 2024-09-10T19:16:36.978 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.3129e-05 2024-09-10T19:16:36.978 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:36.978 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:36.979 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for peered", 2024-09-10T19:16:36.979 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:45.896822+0000", 2024-09-10T19:16:36.979 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.6190000000000002e-06 2024-09-10T19:16:36.979 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:36.979 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:36.979 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:16:36.979 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:46.768706+0000", 2024-09-10T19:16:36.979 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.87188412299999996 2024-09-10T19:16:36.979 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:36.979 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:36.979 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-10T19:16:36.979 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:46.768799+0000", 2024-09-10T19:16:36.980 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.2752999999999998e-05 2024-09-10T19:16:36.980 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:36.980 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:36.980 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 2,6", 2024-09-10T19:16:36.980 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:46.768896+0000", 2024-09-10T19:16:36.980 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.6756000000000007e-05 2024-09-10T19:16:36.980 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:36.980 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:36.981 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-09-10T19:16:36.981 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:46.774360+0000", 2024-09-10T19:16:36.981 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0054644079999999996 2024-09-10T19:16:36.981 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:36.981 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:36.981 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:16:36.981 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:46.782240+0000", 2024-09-10T19:16:36.981 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0078794420000000004 2024-09-10T19:16:36.981 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:36.981 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:36.982 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:16:36.982 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:46.782282+0000", 2024-09-10T19:16:36.982 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.2252000000000001e-05 2024-09-10T19:16:36.982 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:36.982 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:36.982 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-09-10T19:16:36.982 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:46.782329+0000", 2024-09-10T19:16:36.982 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.6597999999999997e-05 2024-09-10T19:16:36.982 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:36.982 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:36.982 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-10T19:16:36.982 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:46.782370+0000", 2024-09-10T19:16:36.982 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.0883999999999997e-05 2024-09-10T19:16:36.983 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:16:36.983 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-10T19:16:36.983 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:16:36.983 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:36.983 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:36.983 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.14781.0:205665 7.b 7:dcf27368:::benchmark_data_smithi183_84160_object65284:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e126)", 2024-09-10T19:16:36.983 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-10T19:14:45.916525+0000", 2024-09-10T19:16:36.983 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 109.63266017399999, 2024-09-10T19:16:36.983 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.87152486799999995, 2024-09-10T19:16:36.983 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-10T19:16:36.984 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T19:16:36.984 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-10T19:16:36.984 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.14781", 2024-09-10T19:16:36.984 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.183:0/2025776420", 2024-09-10T19:16:36.984 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 205665 2024-09-10T19:16:36.984 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:36.985 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-10T19:16:36.985 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:36.985 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-10T19:16:36.985 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:45.916525+0000", 2024-09-10T19:16:36.985 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:16:36.985 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:36.986 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:36.986 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-10T19:16:36.986 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:45.916525+0000", 2024-09-10T19:16:36.986 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:16:36.986 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:36.986 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:36.987 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-10T19:16:36.987 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:45.916524+0000", 2024-09-10T19:16:36.987 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999986 2024-09-10T19:16:36.987 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:36.987 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:36.987 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-10T19:16:36.987 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:45.916530+0000", 2024-09-10T19:16:36.987 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.1029999999999998e-06 2024-09-10T19:16:36.988 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:36.988 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:36.988 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-10T19:16:36.988 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:45.916541+0000", 2024-09-10T19:16:36.988 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.1276e-05 2024-09-10T19:16:36.988 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:36.988 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:36.989 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-10T19:16:36.989 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:45.916546+0000", 2024-09-10T19:16:36.989 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.6990000000000002e-06 2024-09-10T19:16:36.989 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:36.989 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:36.989 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:16:36.990 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:45.916588+0000", 2024-09-10T19:16:36.990 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.1817999999999998e-05 2024-09-10T19:16:36.990 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:36.990 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:36.990 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for peered", 2024-09-10T19:16:36.990 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:45.916594+0000", 2024-09-10T19:16:36.990 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.728e-06 2024-09-10T19:16:36.991 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:36.991 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:36.991 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:16:36.991 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:46.771613+0000", 2024-09-10T19:16:36.991 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.85501948100000003 2024-09-10T19:16:36.991 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:36.992 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:36.992 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-10T19:16:36.992 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:46.771715+0000", 2024-09-10T19:16:36.992 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000102256 2024-09-10T19:16:36.992 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:36.992 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:36.992 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 2,6", 2024-09-10T19:16:36.992 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:46.771850+0000", 2024-09-10T19:16:36.992 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00013438299999999999 2024-09-10T19:16:36.992 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:36.993 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:36.993 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-09-10T19:16:36.993 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:46.779576+0000", 2024-09-10T19:16:36.993 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0077261949999999999 2024-09-10T19:16:36.993 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:36.993 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:36.993 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:16:36.993 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:46.782430+0000", 2024-09-10T19:16:36.993 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.002853703 2024-09-10T19:16:36.993 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:36.994 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:36.994 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:16:36.994 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:46.787904+0000", 2024-09-10T19:16:36.994 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0054744889999999999 2024-09-10T19:16:36.994 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:36.994 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:36.994 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-09-10T19:16:36.994 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:46.788012+0000", 2024-09-10T19:16:36.994 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000107466 2024-09-10T19:16:36.994 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:36.994 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:36.994 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-10T19:16:36.995 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:46.788050+0000", 2024-09-10T19:16:36.995 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.8546e-05 2024-09-10T19:16:36.995 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:16:36.995 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-10T19:16:36.995 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:16:36.995 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:36.995 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:36.995 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.14781.0:205669 7.b 7:d18f2897:::benchmark_data_smithi183_84160_object65288:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e126)", 2024-09-10T19:16:36.995 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-10T19:14:45.928275+0000", 2024-09-10T19:16:36.995 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 109.620910098, 2024-09-10T19:16:36.995 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.85990154299999999, 2024-09-10T19:16:36.995 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-10T19:16:36.995 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T19:16:36.996 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-10T19:16:36.996 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.14781", 2024-09-10T19:16:36.996 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.183:0/2025776420", 2024-09-10T19:16:36.996 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 205669 2024-09-10T19:16:36.996 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:36.996 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-10T19:16:36.996 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:36.996 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-10T19:16:36.996 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:45.928275+0000", 2024-09-10T19:16:36.996 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:16:36.996 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:36.996 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:36.996 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-10T19:16:36.997 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:45.928275+0000", 2024-09-10T19:16:36.997 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:16:36.997 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:36.997 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:36.997 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-10T19:16:36.997 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:45.928274+0000", 2024-09-10T19:16:36.997 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999986 2024-09-10T19:16:36.997 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:36.997 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:36.997 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-10T19:16:36.997 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:45.928284+0000", 2024-09-10T19:16:36.997 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.8179999999999999e-06 2024-09-10T19:16:36.998 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:36.998 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:36.998 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-10T19:16:36.998 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:45.928299+0000", 2024-09-10T19:16:36.998 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.5158e-05 2024-09-10T19:16:36.998 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:36.998 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:36.998 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-10T19:16:36.998 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:45.928305+0000", 2024-09-10T19:16:36.998 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.528e-06 2024-09-10T19:16:36.998 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:36.998 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:36.998 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:16:36.999 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:45.928358+0000", 2024-09-10T19:16:36.999 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.287e-05 2024-09-10T19:16:36.999 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:36.999 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:36.999 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for peered", 2024-09-10T19:16:36.999 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:45.928368+0000", 2024-09-10T19:16:36.999 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.4210000000000001e-06 2024-09-10T19:16:36.999 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:36.999 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:36.999 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:16:36.999 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:46.774603+0000", 2024-09-10T19:16:36.999 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.84623561000000003 2024-09-10T19:16:36.999 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:36.999 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:37.000 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-10T19:16:37.000 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:46.774704+0000", 2024-09-10T19:16:37.000 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000100661 2024-09-10T19:16:37.000 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:37.000 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:37.000 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 2,6", 2024-09-10T19:16:37.000 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:46.774831+0000", 2024-09-10T19:16:37.000 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00012693399999999999 2024-09-10T19:16:37.000 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:37.000 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:37.000 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-09-10T19:16:37.000 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:46.782384+0000", 2024-09-10T19:16:37.001 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0075529589999999997 2024-09-10T19:16:37.001 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:37.001 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:37.001 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:16:37.001 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:46.782467+0000", 2024-09-10T19:16:37.001 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.2755000000000003e-05 2024-09-10T19:16:37.001 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:37.001 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:37.001 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:16:37.001 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:46.788091+0000", 2024-09-10T19:16:37.001 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.005624558 2024-09-10T19:16:37.001 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:37.002 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:37.002 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-09-10T19:16:37.002 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:46.788141+0000", 2024-09-10T19:16:37.002 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.0040000000000002e-05 2024-09-10T19:16:37.002 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:37.002 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:37.002 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-10T19:16:37.002 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:46.788177+0000", 2024-09-10T19:16:37.002 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.5729000000000003e-05 2024-09-10T19:16:37.002 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:16:37.002 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-10T19:16:37.002 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:16:37.002 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:37.002 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:37.003 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.14781.0:217417 7.0 7:0e3413bc:::benchmark_data_smithi183_84160_object77036:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e131)", 2024-09-10T19:16:37.003 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-10T19:14:56.880641+0000", 2024-09-10T19:16:37.003 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 98.668544769999997, 2024-09-10T19:16:37.003 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.991396897, 2024-09-10T19:16:37.003 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-10T19:16:37.003 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T19:16:37.003 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-10T19:16:37.003 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.14781", 2024-09-10T19:16:37.003 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.183:0/2025776420", 2024-09-10T19:16:37.004 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 217417 2024-09-10T19:16:37.004 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:37.004 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-10T19:16:37.004 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:37.004 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-10T19:16:37.004 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:56.880641+0000", 2024-09-10T19:16:37.004 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:16:37.004 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:37.005 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:37.005 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-10T19:16:37.005 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:56.880641+0000", 2024-09-10T19:16:37.005 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:16:37.005 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:37.005 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:37.005 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-10T19:16:37.005 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:56.880639+0000", 2024-09-10T19:16:37.005 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999981 2024-09-10T19:16:37.005 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:37.005 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:37.006 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-10T19:16:37.006 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:56.880647+0000", 2024-09-10T19:16:37.006 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.9810000000000003e-06 2024-09-10T19:16:37.006 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:37.006 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:37.006 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-10T19:16:37.006 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:56.880664+0000", 2024-09-10T19:16:37.006 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.7589999999999999e-05 2024-09-10T19:16:37.006 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:37.006 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:37.007 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-10T19:16:37.007 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:56.880671+0000", 2024-09-10T19:16:37.007 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.9779999999999999e-06 2024-09-10T19:16:37.007 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:37.007 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:37.007 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:16:37.007 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:56.880736+0000", 2024-09-10T19:16:37.007 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.4858999999999997e-05 2024-09-10T19:16:37.007 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:37.007 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:37.007 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for peered", 2024-09-10T19:16:37.007 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:56.880745+0000", 2024-09-10T19:16:37.007 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.8340000000000005e-06 2024-09-10T19:16:37.008 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:37.008 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:37.008 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:16:37.008 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:57.866422+0000", 2024-09-10T19:16:37.008 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.98567665000000004 2024-09-10T19:16:37.008 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:37.008 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:37.008 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-10T19:16:37.008 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:57.866508+0000", 2024-09-10T19:16:37.008 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.5717000000000006e-05 2024-09-10T19:16:37.008 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:37.008 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:37.009 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 2,3", 2024-09-10T19:16:37.009 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:57.866585+0000", 2024-09-10T19:16:37.009 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.7676999999999995e-05 2024-09-10T19:16:37.009 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:37.009 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:37.009 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-09-10T19:16:37.009 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:57.871908+0000", 2024-09-10T19:16:37.009 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0053224120000000003 2024-09-10T19:16:37.009 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:37.009 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:37.009 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:16:37.009 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:57.871930+0000", 2024-09-10T19:16:37.009 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.2722e-05 2024-09-10T19:16:37.010 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:37.010 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:37.010 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:16:37.010 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:57.871985+0000", 2024-09-10T19:16:37.010 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.4246000000000002e-05 2024-09-10T19:16:37.010 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:37.010 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:37.010 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-09-10T19:16:37.010 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:57.872017+0000", 2024-09-10T19:16:37.010 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.2441000000000003e-05 2024-09-10T19:16:37.010 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:37.010 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:37.011 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-10T19:16:37.011 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:57.872038+0000", 2024-09-10T19:16:37.011 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.0639999999999999e-05 2024-09-10T19:16:37.011 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:16:37.011 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-10T19:16:37.011 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:16:37.011 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:37.011 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:37.011 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.14781.0:217427 7.0 7:0e3d17be:::benchmark_data_smithi183_84160_object77046:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e131)", 2024-09-10T19:16:37.011 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-10T19:14:56.890012+0000", 2024-09-10T19:16:37.011 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 98.659173953999996, 2024-09-10T19:16:37.011 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.98213677700000002, 2024-09-10T19:16:37.011 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-10T19:16:37.011 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T19:16:37.012 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-10T19:16:37.012 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.14781", 2024-09-10T19:16:37.012 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.183:0/2025776420", 2024-09-10T19:16:37.012 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 217427 2024-09-10T19:16:37.012 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:37.012 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-10T19:16:37.012 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:37.012 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-10T19:16:37.012 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:56.890012+0000", 2024-09-10T19:16:37.012 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:16:37.012 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:37.012 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:37.013 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-10T19:16:37.013 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:56.890012+0000", 2024-09-10T19:16:37.013 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:16:37.013 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:37.013 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:37.013 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-10T19:16:37.013 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:56.890010+0000", 2024-09-10T19:16:37.013 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999986 2024-09-10T19:16:37.013 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:37.013 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:37.013 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-10T19:16:37.013 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:56.890017+0000", 2024-09-10T19:16:37.013 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.5259999999999999e-06 2024-09-10T19:16:37.014 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:37.014 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:37.014 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-10T19:16:37.014 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:56.890028+0000", 2024-09-10T19:16:37.014 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.1259e-05 2024-09-10T19:16:37.014 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:37.014 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:37.014 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-10T19:16:37.014 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:56.890032+0000", 2024-09-10T19:16:37.014 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.3769999999999998e-06 2024-09-10T19:16:37.014 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:37.014 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:37.014 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:16:37.015 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:56.890066+0000", 2024-09-10T19:16:37.015 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.3467999999999999e-05 2024-09-10T19:16:37.015 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:37.015 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:37.015 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for peered", 2024-09-10T19:16:37.015 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:56.890072+0000", 2024-09-10T19:16:37.015 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.5400000000000001e-06 2024-09-10T19:16:37.015 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:37.015 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:37.015 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:16:37.015 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:57.868641+0000", 2024-09-10T19:16:37.015 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.97856873 2024-09-10T19:16:37.015 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:37.015 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:37.016 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-10T19:16:37.016 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:57.868700+0000", 2024-09-10T19:16:37.016 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.9086999999999999e-05 2024-09-10T19:16:37.016 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:37.016 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:37.016 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 2,3", 2024-09-10T19:16:37.016 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:57.868767+0000", 2024-09-10T19:16:37.016 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.6742000000000006e-05 2024-09-10T19:16:37.016 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:37.016 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:37.016 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:16:37.016 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:57.872070+0000", 2024-09-10T19:16:37.016 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0033030450000000001 2024-09-10T19:16:37.017 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:37.017 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:37.017 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-09-10T19:16:37.017 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:57.872076+0000", 2024-09-10T19:16:37.017 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.1510000000000001e-06 2024-09-10T19:16:37.017 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:37.017 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:37.017 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:16:37.017 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:57.872111+0000", 2024-09-10T19:16:37.017 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.4857999999999997e-05 2024-09-10T19:16:37.017 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:37.017 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:37.017 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-09-10T19:16:37.018 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:57.872132+0000", 2024-09-10T19:16:37.018 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.0798999999999999e-05 2024-09-10T19:16:37.018 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:37.018 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:37.018 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-10T19:16:37.018 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:57.872148+0000", 2024-09-10T19:16:37.018 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.6449000000000001e-05 2024-09-10T19:16:37.018 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:16:37.018 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-10T19:16:37.018 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:16:37.018 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:37.018 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:37.018 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.14781.0:217459 7.0 7:09cbbd5e:::benchmark_data_smithi183_84160_object77078:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e131)", 2024-09-10T19:16:37.019 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-10T19:14:56.915054+0000", 2024-09-10T19:16:37.019 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 98.634131178000004, 2024-09-10T19:16:37.019 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.96084031700000005, 2024-09-10T19:16:37.019 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-10T19:16:37.019 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T19:16:37.019 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-10T19:16:37.019 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.14781", 2024-09-10T19:16:37.019 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.183:0/2025776420", 2024-09-10T19:16:37.019 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 217459 2024-09-10T19:16:37.019 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:37.019 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-10T19:16:37.019 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:37.019 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-10T19:16:37.019 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:56.915054+0000", 2024-09-10T19:16:37.020 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:16:37.020 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:37.020 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:37.020 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-10T19:16:37.020 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:56.915054+0000", 2024-09-10T19:16:37.020 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:16:37.020 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:37.020 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:37.021 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-10T19:16:37.021 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:56.915053+0000", 2024-09-10T19:16:37.021 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999986 2024-09-10T19:16:37.021 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:37.021 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:37.021 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-10T19:16:37.021 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:56.915060+0000", 2024-09-10T19:16:37.022 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.8109999999999997e-06 2024-09-10T19:16:37.022 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:37.022 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:37.022 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-10T19:16:37.022 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:56.915072+0000", 2024-09-10T19:16:37.022 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.1898e-05 2024-09-10T19:16:37.022 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:37.022 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:37.022 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-10T19:16:37.022 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:56.915077+0000", 2024-09-10T19:16:37.022 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.344e-06 2024-09-10T19:16:37.022 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:37.022 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:37.023 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:16:37.023 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:56.915118+0000", 2024-09-10T19:16:37.023 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.0725000000000001e-05 2024-09-10T19:16:37.023 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:37.023 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:37.023 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for peered", 2024-09-10T19:16:37.023 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:56.915123+0000", 2024-09-10T19:16:37.023 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.6559999999999997e-06 2024-09-10T19:16:37.023 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:37.023 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:37.023 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:16:37.023 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:57.869813+0000", 2024-09-10T19:16:37.024 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.95469002700000005 2024-09-10T19:16:37.024 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:37.024 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:37.024 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-10T19:16:37.024 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:57.869876+0000", 2024-09-10T19:16:37.024 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.2135000000000001e-05 2024-09-10T19:16:37.024 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:37.024 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:37.024 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 2,3", 2024-09-10T19:16:37.024 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:57.869937+0000", 2024-09-10T19:16:37.025 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.1420999999999994e-05 2024-09-10T19:16:37.025 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:37.025 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:37.025 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-09-10T19:16:37.025 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:57.872083+0000", 2024-09-10T19:16:37.025 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.002145545 2024-09-10T19:16:37.025 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:37.025 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:37.025 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:16:37.026 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:57.875737+0000", 2024-09-10T19:16:37.026 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0036545900000000001 2024-09-10T19:16:37.026 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:37.026 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:37.026 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:16:37.026 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:57.875798+0000", 2024-09-10T19:16:37.026 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.0498999999999998e-05 2024-09-10T19:16:37.026 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:37.027 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:37.027 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-09-10T19:16:37.027 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:57.875853+0000", 2024-09-10T19:16:37.027 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.5429000000000001e-05 2024-09-10T19:16:37.027 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:37.027 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:37.027 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-10T19:16:37.027 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:57.875895+0000", 2024-09-10T19:16:37.028 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.1582999999999998e-05 2024-09-10T19:16:37.028 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:16:37.028 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-10T19:16:37.028 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:16:37.028 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:37.028 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:37.028 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.14781.0:217507 7.0 7:01ed5a33:::benchmark_data_smithi183_84160_object77126:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e131)", 2024-09-10T19:16:37.028 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-10T19:14:56.947518+0000", 2024-09-10T19:16:37.029 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 98.601667522, 2024-09-10T19:16:37.029 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.92856619299999998, 2024-09-10T19:16:37.029 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-10T19:16:37.029 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T19:16:37.029 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-10T19:16:37.029 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.14781", 2024-09-10T19:16:37.029 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.183:0/2025776420", 2024-09-10T19:16:37.030 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 217507 2024-09-10T19:16:37.030 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:37.030 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-10T19:16:37.030 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:37.030 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-10T19:16:37.030 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:56.947518+0000", 2024-09-10T19:16:37.030 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:16:37.030 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:37.030 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:37.031 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-10T19:16:37.031 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:56.947518+0000", 2024-09-10T19:16:37.031 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:16:37.031 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:37.031 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:37.031 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-10T19:16:37.031 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:56.947517+0000", 2024-09-10T19:16:37.031 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999986 2024-09-10T19:16:37.032 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:37.032 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:37.032 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-10T19:16:37.032 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:56.947523+0000", 2024-09-10T19:16:37.032 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.3199999999999996e-06 2024-09-10T19:16:37.032 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:37.032 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:37.032 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-10T19:16:37.033 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:56.947534+0000", 2024-09-10T19:16:37.033 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.1139e-05 2024-09-10T19:16:37.033 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:37.033 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:37.033 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-10T19:16:37.033 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:56.947540+0000", 2024-09-10T19:16:37.033 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.2720000000000003e-06 2024-09-10T19:16:37.033 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:37.034 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:37.034 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:16:37.034 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:56.947593+0000", 2024-09-10T19:16:37.034 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.2710000000000002e-05 2024-09-10T19:16:37.034 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:37.034 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:37.034 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for peered", 2024-09-10T19:16:37.034 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:56.947600+0000", 2024-09-10T19:16:37.035 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.0180000000000004e-06 2024-09-10T19:16:37.035 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:37.035 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:37.035 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:16:37.035 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:57.870882+0000", 2024-09-10T19:16:37.035 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.92328237899999999 2024-09-10T19:16:37.035 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:37.035 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:37.035 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-10T19:16:37.036 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:57.870942+0000", 2024-09-10T19:16:37.036 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.9311000000000002e-05 2024-09-10T19:16:37.036 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:37.036 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:37.036 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 2,3", 2024-09-10T19:16:37.036 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:57.871000+0000", 2024-09-10T19:16:37.036 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.7875999999999999e-05 2024-09-10T19:16:37.036 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:37.036 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:37.037 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-09-10T19:16:37.037 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:57.873306+0000", 2024-09-10T19:16:37.037 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0023059539999999998 2024-09-10T19:16:37.037 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:37.037 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:37.037 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:16:37.037 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:57.875946+0000", 2024-09-10T19:16:37.037 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0026407370000000002 2024-09-10T19:16:37.037 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:37.037 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:37.037 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:16:37.037 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:57.875998+0000", 2024-09-10T19:16:37.037 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.1632000000000003e-05 2024-09-10T19:16:37.038 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:37.038 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:37.038 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-09-10T19:16:37.038 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:57.876046+0000", 2024-09-10T19:16:37.038 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.8245999999999999e-05 2024-09-10T19:16:37.038 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:37.038 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:37.038 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-10T19:16:37.038 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:57.876084+0000", 2024-09-10T19:16:37.038 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.8080999999999997e-05 2024-09-10T19:16:37.038 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:16:37.038 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-10T19:16:37.038 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:16:37.038 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:37.039 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:37.039 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.14781.0:217539 7.0 7:0d6da71d:::benchmark_data_smithi183_84160_object77158:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e131)", 2024-09-10T19:16:37.039 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-10T19:14:56.966319+0000", 2024-09-10T19:16:37.039 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 98.582866934999998, 2024-09-10T19:16:37.039 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.91476692999999998, 2024-09-10T19:16:37.039 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-10T19:16:37.039 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T19:16:37.039 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-10T19:16:37.039 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.14781", 2024-09-10T19:16:37.039 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.183:0/2025776420", 2024-09-10T19:16:37.039 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 217539 2024-09-10T19:16:37.039 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:37.039 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-10T19:16:37.039 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:37.040 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-10T19:16:37.040 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:56.966319+0000", 2024-09-10T19:16:37.040 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:16:37.040 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:37.040 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:37.040 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-10T19:16:37.040 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:56.966319+0000", 2024-09-10T19:16:37.040 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:16:37.040 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:37.040 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:37.040 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-10T19:16:37.040 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:56.966317+0000", 2024-09-10T19:16:37.040 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999986 2024-09-10T19:16:37.041 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:37.041 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:37.041 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-10T19:16:37.041 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:56.966323+0000", 2024-09-10T19:16:37.041 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.1099999999999999e-06 2024-09-10T19:16:37.041 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:37.041 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:37.041 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-10T19:16:37.041 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:56.966335+0000", 2024-09-10T19:16:37.041 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.1368e-05 2024-09-10T19:16:37.041 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:37.041 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:37.041 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-10T19:16:37.042 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:56.966339+0000", 2024-09-10T19:16:37.042 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.3880000000000002e-06 2024-09-10T19:16:37.042 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:37.042 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:37.042 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:16:37.042 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:56.966374+0000", 2024-09-10T19:16:37.042 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.5182999999999998e-05 2024-09-10T19:16:37.042 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:37.042 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:37.042 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for peered", 2024-09-10T19:16:37.042 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:56.966381+0000", 2024-09-10T19:16:37.042 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.5760000000000002e-06 2024-09-10T19:16:37.042 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:37.042 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:37.043 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:16:37.043 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:57.872179+0000", 2024-09-10T19:16:37.043 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.90579777100000003 2024-09-10T19:16:37.043 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:37.043 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:37.043 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-10T19:16:37.043 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:57.872241+0000", 2024-09-10T19:16:37.043 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.2650000000000005e-05 2024-09-10T19:16:37.043 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:37.043 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:37.043 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 2,3", 2024-09-10T19:16:37.043 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:57.872302+0000", 2024-09-10T19:16:37.043 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.0810000000000002e-05 2024-09-10T19:16:37.044 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:37.044 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:37.044 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-09-10T19:16:37.044 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:57.875952+0000", 2024-09-10T19:16:37.044 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.003649902 2024-09-10T19:16:37.044 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:37.044 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:37.044 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:16:37.044 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:57.880875+0000", 2024-09-10T19:16:37.044 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0049230159999999997 2024-09-10T19:16:37.044 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:37.045 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:37.045 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:16:37.045 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:57.880985+0000", 2024-09-10T19:16:37.045 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00011011299999999999 2024-09-10T19:16:37.045 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:37.045 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:37.045 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-09-10T19:16:37.045 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:57.881042+0000", 2024-09-10T19:16:37.045 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.7080999999999998e-05 2024-09-10T19:16:37.045 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:37.045 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:37.046 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-10T19:16:37.046 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:57.881086+0000", 2024-09-10T19:16:37.046 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.3420000000000001e-05 2024-09-10T19:16:37.046 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:16:37.046 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-10T19:16:37.046 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:16:37.046 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:37.046 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:37.046 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.14781.0:217543 7.0 7:0ff0ed85:::benchmark_data_smithi183_84160_object77162:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e131)", 2024-09-10T19:16:37.046 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-10T19:14:56.968911+0000", 2024-09-10T19:16:37.046 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 98.580275029999996, 2024-09-10T19:16:37.046 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.91229753700000005, 2024-09-10T19:16:37.046 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-10T19:16:37.046 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T19:16:37.047 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-10T19:16:37.047 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.14781", 2024-09-10T19:16:37.047 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.183:0/2025776420", 2024-09-10T19:16:37.047 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 217543 2024-09-10T19:16:37.047 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:37.047 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-10T19:16:37.047 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:37.047 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-10T19:16:37.047 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:56.968911+0000", 2024-09-10T19:16:37.047 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:16:37.047 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:37.047 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:37.047 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-10T19:16:37.048 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:56.968911+0000", 2024-09-10T19:16:37.048 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:16:37.048 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:37.048 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:37.048 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-10T19:16:37.048 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:56.968909+0000", 2024-09-10T19:16:37.048 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.999999 2024-09-10T19:16:37.048 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:37.048 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:37.048 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-10T19:16:37.048 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:56.968915+0000", 2024-09-10T19:16:37.048 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.8869999999999997e-06 2024-09-10T19:16:37.048 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:37.049 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:37.049 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-10T19:16:37.049 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:56.968929+0000", 2024-09-10T19:16:37.049 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.3413e-05 2024-09-10T19:16:37.049 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:37.049 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:37.049 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-10T19:16:37.049 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:56.968934+0000", 2024-09-10T19:16:37.049 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.8159999999999999e-06 2024-09-10T19:16:37.049 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:37.049 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:37.049 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:16:37.049 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:56.968977+0000", 2024-09-10T19:16:37.050 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.2593999999999999e-05 2024-09-10T19:16:37.050 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:37.050 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:37.050 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for peered", 2024-09-10T19:16:37.050 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:56.968986+0000", 2024-09-10T19:16:37.050 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.1009999999999998e-06 2024-09-10T19:16:37.050 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:37.050 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:37.050 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:16:37.050 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:57.873333+0000", 2024-09-10T19:16:37.050 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.90434694800000004 2024-09-10T19:16:37.050 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:37.050 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:37.050 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-10T19:16:37.051 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:57.873458+0000", 2024-09-10T19:16:37.051 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000125382 2024-09-10T19:16:37.051 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:37.051 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:37.051 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 2,3", 2024-09-10T19:16:37.051 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:57.873595+0000", 2024-09-10T19:16:37.051 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000136217 2024-09-10T19:16:37.051 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:37.051 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:37.051 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-09-10T19:16:37.051 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:57.878366+0000", 2024-09-10T19:16:37.051 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0047711330000000003 2024-09-10T19:16:37.051 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:37.052 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:37.052 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:16:37.052 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:57.880939+0000", 2024-09-10T19:16:37.052 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0025729669999999998 2024-09-10T19:16:37.052 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:37.052 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:37.052 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:16:37.052 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:57.881126+0000", 2024-09-10T19:16:37.052 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00018687099999999999 2024-09-10T19:16:37.052 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:37.052 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:37.052 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-09-10T19:16:37.052 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:57.881161+0000", 2024-09-10T19:16:37.053 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.5775000000000002e-05 2024-09-10T19:16:37.053 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:37.053 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:37.053 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-10T19:16:37.053 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:57.881208+0000", 2024-09-10T19:16:37.053 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.6579000000000002e-05 2024-09-10T19:16:37.053 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:16:37.053 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-10T19:16:37.053 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:16:37.053 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:37.053 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:37.053 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.14781.0:217569 7.0 7:0e5c5ab4:::benchmark_data_smithi183_84160_object77188:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e131)", 2024-09-10T19:16:37.053 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-10T19:14:56.986078+0000", 2024-09-10T19:16:37.053 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 98.563107622000004, 2024-09-10T19:16:37.054 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.90033372899999997, 2024-09-10T19:16:37.054 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-10T19:16:37.054 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T19:16:37.054 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-10T19:16:37.054 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.14781", 2024-09-10T19:16:37.054 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.183:0/2025776420", 2024-09-10T19:16:37.054 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 217569 2024-09-10T19:16:37.054 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:37.054 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-10T19:16:37.054 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:37.054 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-10T19:16:37.054 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:56.986078+0000", 2024-09-10T19:16:37.054 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:16:37.055 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:37.055 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:37.055 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-10T19:16:37.055 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:56.986078+0000", 2024-09-10T19:16:37.055 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:16:37.055 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:37.055 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:37.055 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-10T19:16:37.056 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:56.986077+0000", 2024-09-10T19:16:37.056 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999986 2024-09-10T19:16:37.056 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:37.056 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:37.056 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-10T19:16:37.056 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:56.986083+0000", 2024-09-10T19:16:37.056 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.3409999999999999e-06 2024-09-10T19:16:37.056 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:37.056 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:37.056 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-10T19:16:37.056 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:56.986101+0000", 2024-09-10T19:16:37.057 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.7604999999999998e-05 2024-09-10T19:16:37.057 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:37.057 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:37.057 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-10T19:16:37.057 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:56.986107+0000", 2024-09-10T19:16:37.057 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.6429999999999998e-06 2024-09-10T19:16:37.057 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:37.057 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:37.057 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:16:37.057 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:56.986150+0000", 2024-09-10T19:16:37.057 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.3296e-05 2024-09-10T19:16:37.057 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:37.057 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:37.058 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for peered", 2024-09-10T19:16:37.058 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:56.986157+0000", 2024-09-10T19:16:37.058 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.9450000000000004e-06 2024-09-10T19:16:37.058 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:37.058 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:37.058 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:16:37.058 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:57.876115+0000", 2024-09-10T19:16:37.058 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.88995753399999999 2024-09-10T19:16:37.058 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:37.058 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:37.058 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-10T19:16:37.058 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:57.876180+0000", 2024-09-10T19:16:37.058 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.5535000000000002e-05 2024-09-10T19:16:37.059 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:37.059 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:37.059 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 2,3", 2024-09-10T19:16:37.059 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:57.876255+0000", 2024-09-10T19:16:37.059 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.4130999999999999e-05 2024-09-10T19:16:37.059 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:37.059 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:37.059 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-09-10T19:16:37.059 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:57.880946+0000", 2024-09-10T19:16:37.059 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0046913959999999996 2024-09-10T19:16:37.059 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:37.059 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:37.059 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:16:37.060 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:57.886257+0000", 2024-09-10T19:16:37.060 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0053109059999999998 2024-09-10T19:16:37.060 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:37.060 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:37.060 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:16:37.060 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:57.886325+0000", 2024-09-10T19:16:37.060 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.8233999999999998e-05 2024-09-10T19:16:37.060 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:37.060 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:37.060 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-09-10T19:16:37.060 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:57.886372+0000", 2024-09-10T19:16:37.060 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.6845000000000003e-05 2024-09-10T19:16:37.060 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:37.061 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:37.061 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-10T19:16:37.061 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:57.886412+0000", 2024-09-10T19:16:37.061 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.9671e-05 2024-09-10T19:16:37.061 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:16:37.061 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-10T19:16:37.061 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:16:37.061 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:37.061 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:37.061 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.14781.0:217596 7.0 7:094b0cc1:::benchmark_data_smithi183_84160_object77215:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e131)", 2024-09-10T19:16:37.061 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-10T19:14:57.006462+0000", 2024-09-10T19:16:37.061 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 98.542723679999995, 2024-09-10T19:16:37.061 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.88539582699999997, 2024-09-10T19:16:37.062 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-10T19:16:37.063 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T19:16:37.063 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-10T19:16:37.064 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.14781", 2024-09-10T19:16:37.064 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.183:0/2025776420", 2024-09-10T19:16:37.064 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 217596 2024-09-10T19:16:37.064 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:37.064 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-10T19:16:37.064 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:37.064 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-10T19:16:37.064 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:57.006462+0000", 2024-09-10T19:16:37.064 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:16:37.064 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:37.064 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:37.064 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-10T19:16:37.065 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:57.006462+0000", 2024-09-10T19:16:37.065 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:16:37.065 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:37.065 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:37.065 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-10T19:16:37.065 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:57.006460+0000", 2024-09-10T19:16:37.065 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999986 2024-09-10T19:16:37.065 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:37.065 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:37.065 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-10T19:16:37.065 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:57.006468+0000", 2024-09-10T19:16:37.065 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.1389999999999998e-06 2024-09-10T19:16:37.065 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:37.066 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:37.066 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-10T19:16:37.066 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:57.006483+0000", 2024-09-10T19:16:37.066 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.5625e-05 2024-09-10T19:16:37.066 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:37.066 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:37.066 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-10T19:16:37.066 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:57.006489+0000", 2024-09-10T19:16:37.066 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.246e-06 2024-09-10T19:16:37.066 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:37.066 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:37.066 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:16:37.066 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:57.006537+0000", 2024-09-10T19:16:37.067 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.7423999999999999e-05 2024-09-10T19:16:37.067 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:37.067 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:37.067 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for peered", 2024-09-10T19:16:37.067 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:57.006546+0000", 2024-09-10T19:16:37.067 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.0250000000000008e-06 2024-09-10T19:16:37.067 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:37.067 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:37.067 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:16:37.067 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:57.878397+0000", 2024-09-10T19:16:37.067 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.87185067199999999 2024-09-10T19:16:37.067 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:37.067 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:37.067 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-10T19:16:37.068 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:57.878520+0000", 2024-09-10T19:16:37.068 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000123924 2024-09-10T19:16:37.068 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:37.068 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:37.068 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 2,3", 2024-09-10T19:16:37.068 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:57.878646+0000", 2024-09-10T19:16:37.068 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000125592 2024-09-10T19:16:37.068 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:37.068 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:37.068 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:16:37.068 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:57.886288+0000", 2024-09-10T19:16:37.068 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0076422399999999998 2024-09-10T19:16:37.068 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:37.069 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:37.069 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-09-10T19:16:37.069 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:57.891702+0000", 2024-09-10T19:16:37.069 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0054138629999999997 2024-09-10T19:16:37.069 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:37.069 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:37.069 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:16:37.069 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:57.891738+0000", 2024-09-10T19:16:37.069 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.5969000000000001e-05 2024-09-10T19:16:37.069 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:37.069 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:37.069 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-09-10T19:16:37.069 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:57.891817+0000", 2024-09-10T19:16:37.070 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.8714000000000006e-05 2024-09-10T19:16:37.070 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:37.070 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:37.070 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-10T19:16:37.070 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:57.891858+0000", 2024-09-10T19:16:37.070 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.0802e-05 2024-09-10T19:16:37.070 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:16:37.070 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-10T19:16:37.070 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:16:37.070 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:37.070 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:37.070 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.14781.0:217606 7.0 7:053b9476:::benchmark_data_smithi183_84160_object77225:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e131)", 2024-09-10T19:16:37.070 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-10T19:14:57.012833+0000", 2024-09-10T19:16:37.071 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 98.53635276, 2024-09-10T19:16:37.071 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.87920452199999999, 2024-09-10T19:16:37.071 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-10T19:16:37.071 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T19:16:37.071 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-10T19:16:37.071 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.14781", 2024-09-10T19:16:37.071 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.183:0/2025776420", 2024-09-10T19:16:37.071 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 217606 2024-09-10T19:16:37.071 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:37.071 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-10T19:16:37.071 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:37.071 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-10T19:16:37.071 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:57.012833+0000", 2024-09-10T19:16:37.071 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:16:37.072 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:37.072 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:37.072 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-10T19:16:37.072 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:57.012833+0000", 2024-09-10T19:16:37.072 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:16:37.072 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:37.072 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:37.072 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-10T19:16:37.072 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:57.012831+0000", 2024-09-10T19:16:37.072 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999986 2024-09-10T19:16:37.072 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:37.072 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:37.072 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-10T19:16:37.073 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:57.012839+0000", 2024-09-10T19:16:37.073 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.8290000000000008e-06 2024-09-10T19:16:37.073 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:37.073 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:37.073 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-10T19:16:37.073 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:57.012854+0000", 2024-09-10T19:16:37.073 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.5398000000000001e-05 2024-09-10T19:16:37.073 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:37.073 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:37.073 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-10T19:16:37.073 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:57.012861+0000", 2024-09-10T19:16:37.073 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.6599999999999998e-06 2024-09-10T19:16:37.073 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:37.074 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:37.074 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:16:37.074 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:57.012918+0000", 2024-09-10T19:16:37.074 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.6752999999999997e-05 2024-09-10T19:16:37.074 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:37.074 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:37.074 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for peered", 2024-09-10T19:16:37.074 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:57.012931+0000", 2024-09-10T19:16:37.074 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.3477e-05 2024-09-10T19:16:37.074 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:37.074 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:37.074 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:16:37.074 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:57.881256+0000", 2024-09-10T19:16:37.075 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.86832445000000003 2024-09-10T19:16:37.075 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:37.075 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:37.075 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-10T19:16:37.075 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:57.881353+0000", 2024-09-10T19:16:37.075 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.7676999999999993e-05 2024-09-10T19:16:37.075 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:37.075 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:37.075 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 2,3", 2024-09-10T19:16:37.075 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:57.881456+0000", 2024-09-10T19:16:37.075 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000102324 2024-09-10T19:16:37.075 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:37.075 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:37.076 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-09-10T19:16:37.076 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:57.891867+0000", 2024-09-10T19:16:37.076 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.010411053999999999 2024-09-10T19:16:37.076 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:37.076 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:37.076 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:16:37.076 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:57.891913+0000", 2024-09-10T19:16:37.076 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.5963e-05 2024-09-10T19:16:37.076 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:37.076 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:37.076 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:16:37.076 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:57.891946+0000", 2024-09-10T19:16:37.076 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.3618999999999998e-05 2024-09-10T19:16:37.077 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:37.077 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:37.077 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-09-10T19:16:37.077 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:57.891991+0000", 2024-09-10T19:16:37.077 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.4737000000000002e-05 2024-09-10T19:16:37.077 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:37.077 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:37.077 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-10T19:16:37.077 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:57.892037+0000", 2024-09-10T19:16:37.077 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.6146999999999997e-05 2024-09-10T19:16:37.077 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:16:37.078 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-10T19:16:37.078 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:16:37.078 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:37.078 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:37.078 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.14781.0:217654 7.0 7:0548a005:::benchmark_data_smithi183_84160_object77273:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e131)", 2024-09-10T19:16:37.078 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-10T19:14:57.055737+0000", 2024-09-10T19:16:37.078 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 98.493448732000004, 2024-09-10T19:16:37.078 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.85089322599999995, 2024-09-10T19:16:37.078 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-10T19:16:37.078 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T19:16:37.078 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-10T19:16:37.078 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.14781", 2024-09-10T19:16:37.078 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.183:0/2025776420", 2024-09-10T19:16:37.079 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 217654 2024-09-10T19:16:37.079 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:37.079 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-10T19:16:37.079 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:37.079 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-10T19:16:37.079 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:57.055737+0000", 2024-09-10T19:16:37.079 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:16:37.079 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:37.079 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:37.079 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-10T19:16:37.079 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:57.055737+0000", 2024-09-10T19:16:37.079 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:16:37.079 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:37.080 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:37.080 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-10T19:16:37.080 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:57.055736+0000", 2024-09-10T19:16:37.080 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.999999 2024-09-10T19:16:37.080 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:37.080 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:37.080 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-10T19:16:37.080 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:57.055741+0000", 2024-09-10T19:16:37.080 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.5679999999999999e-06 2024-09-10T19:16:37.080 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:37.080 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:37.080 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-10T19:16:37.080 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:57.055755+0000", 2024-09-10T19:16:37.081 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.3858e-05 2024-09-10T19:16:37.081 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:37.081 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:37.081 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-10T19:16:37.081 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:57.055760+0000", 2024-09-10T19:16:37.081 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.7299999999999996e-06 2024-09-10T19:16:37.081 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:37.081 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:37.081 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:16:37.081 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:57.055791+0000", 2024-09-10T19:16:37.081 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.1198999999999997e-05 2024-09-10T19:16:37.081 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:37.082 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:37.082 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for peered", 2024-09-10T19:16:37.082 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:57.055797+0000", 2024-09-10T19:16:37.082 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.8640000000000001e-06 2024-09-10T19:16:37.082 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:37.082 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:37.082 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:16:37.082 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:57.883829+0000", 2024-09-10T19:16:37.082 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.82803196000000001 2024-09-10T19:16:37.082 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:37.082 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:37.082 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-10T19:16:37.082 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:57.883939+0000", 2024-09-10T19:16:37.083 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000110587 2024-09-10T19:16:37.083 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:37.083 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:37.083 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 2,3", 2024-09-10T19:16:37.083 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:57.884067+0000", 2024-09-10T19:16:37.083 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00012794999999999999 2024-09-10T19:16:37.083 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:37.083 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:37.083 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-09-10T19:16:37.083 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:57.891872+0000", 2024-09-10T19:16:37.083 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0078045559999999998 2024-09-10T19:16:37.083 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:37.083 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:37.083 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:16:37.084 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:57.892078+0000", 2024-09-10T19:16:37.084 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00020619599999999999 2024-09-10T19:16:37.084 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:37.084 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:37.084 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:16:37.084 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:57.906563+0000", 2024-09-10T19:16:37.084 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.014484442 2024-09-10T19:16:37.084 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:37.084 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:37.084 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-09-10T19:16:37.084 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:57.906596+0000", 2024-09-10T19:16:37.085 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.3006999999999998e-05 2024-09-10T19:16:37.085 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:37.085 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:37.085 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-10T19:16:37.085 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:57.906630+0000", 2024-09-10T19:16:37.085 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.4445e-05 2024-09-10T19:16:37.085 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:16:37.085 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-10T19:16:37.085 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:16:37.085 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:37.085 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:37.085 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.14781.0:217660 7.0 7:0d39e53b:::benchmark_data_smithi183_84160_object77279:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e131)", 2024-09-10T19:16:37.086 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-10T19:14:57.060614+0000", 2024-09-10T19:16:37.086 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 98.488571573000002, 2024-09-10T19:16:37.086 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.85285114699999998, 2024-09-10T19:16:37.086 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-10T19:16:37.086 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T19:16:37.086 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-10T19:16:37.086 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.14781", 2024-09-10T19:16:37.086 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.183:0/2025776420", 2024-09-10T19:16:37.086 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 217660 2024-09-10T19:16:37.086 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:37.086 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-10T19:16:37.086 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:37.086 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-10T19:16:37.086 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:57.060614+0000", 2024-09-10T19:16:37.087 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:16:37.087 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:37.087 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:37.087 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-10T19:16:37.087 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:57.060614+0000", 2024-09-10T19:16:37.087 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:16:37.087 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:37.087 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:37.087 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-10T19:16:37.087 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:57.060613+0000", 2024-09-10T19:16:37.087 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.999999 2024-09-10T19:16:37.087 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:37.087 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:37.088 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-10T19:16:37.088 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:57.060622+0000", 2024-09-10T19:16:37.088 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.7579999999999998e-06 2024-09-10T19:16:37.088 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:37.088 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:37.088 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-10T19:16:37.088 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:57.060637+0000", 2024-09-10T19:16:37.088 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.5594999999999999e-05 2024-09-10T19:16:37.088 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:37.088 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:37.088 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-10T19:16:37.088 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:57.060644+0000", 2024-09-10T19:16:37.089 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.2139999999999999e-06 2024-09-10T19:16:37.089 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:37.089 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:37.089 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:16:37.089 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:57.060687+0000", 2024-09-10T19:16:37.089 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.2731000000000001e-05 2024-09-10T19:16:37.089 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:37.089 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:37.089 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for peered", 2024-09-10T19:16:37.089 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:57.060692+0000", 2024-09-10T19:16:37.089 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.1200000000000001e-06 2024-09-10T19:16:37.089 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:37.089 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:37.090 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:16:37.090 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:57.886444+0000", 2024-09-10T19:16:37.090 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.82575209400000005 2024-09-10T19:16:37.090 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:37.090 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:37.090 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-10T19:16:37.090 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:57.886554+0000", 2024-09-10T19:16:37.090 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00010924799999999999 2024-09-10T19:16:37.090 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:37.090 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:37.090 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 2,3", 2024-09-10T19:16:37.090 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:57.886677+0000", 2024-09-10T19:16:37.090 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00012318299999999999 2024-09-10T19:16:37.091 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:37.091 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:37.091 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-09-10T19:16:37.091 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:57.891877+0000", 2024-09-10T19:16:37.091 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0052003750000000001 2024-09-10T19:16:37.091 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:37.091 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:37.091 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:16:37.091 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:57.900165+0000", 2024-09-10T19:16:37.091 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0082874449999999992 2024-09-10T19:16:37.091 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:37.091 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:37.091 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:16:37.092 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:57.913381+0000", 2024-09-10T19:16:37.092 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.013216416 2024-09-10T19:16:37.092 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:37.092 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:37.092 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-09-10T19:16:37.092 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:57.913428+0000", 2024-09-10T19:16:37.092 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.6604999999999998e-05 2024-09-10T19:16:37.092 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:37.092 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:37.092 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-10T19:16:37.092 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:57.913465+0000", 2024-09-10T19:16:37.092 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.7483e-05 2024-09-10T19:16:37.093 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:16:37.093 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-10T19:16:37.093 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:16:37.093 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:16:37.093 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-10T19:16:37.093 INFO:teuthology.orchestra.run.smithi191.stdout:} 2024-09-10T19:16:37.093 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:16:37.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:16:37.115 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:16:37.115 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:16:37.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:16:36 smithi183 ceph-mon[29736]: pgmap v266: 148 pgs: 148 active+clean; 1.7 GiB data, 6.6 GiB used, 530 GiB / 536 GiB avail; 3.5 KiB/s rd, 250 MiB/s wr, 4.01k op/s; 176 B/s, 5 objects/s recovering 2024-09-10T19:16:37.182 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:16:36 smithi183 ceph-mon[29736]: 3.2 scrub starts 2024-09-10T19:16:37.182 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:16:36 smithi183 ceph-mon[29736]: 3.2 scrub ok 2024-09-10T19:16:37.182 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:16:36 smithi183 ceph-mon[29736]: 4.1b scrub starts 2024-09-10T19:16:37.182 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:16:36 smithi183 ceph-mon[29736]: 4.1b scrub ok 2024-09-10T19:16:37.183 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:16:36 smithi183 ceph-mon[41562]: pgmap v266: 148 pgs: 148 active+clean; 1.7 GiB data, 6.6 GiB used, 530 GiB / 536 GiB avail; 3.5 KiB/s rd, 250 MiB/s wr, 4.01k op/s; 176 B/s, 5 objects/s recovering 2024-09-10T19:16:37.183 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:16:36 smithi183 ceph-mon[41562]: 3.2 scrub starts 2024-09-10T19:16:37.183 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:16:36 smithi183 ceph-mon[41562]: 3.2 scrub ok 2024-09-10T19:16:37.183 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:16:36 smithi183 ceph-mon[41562]: 4.1b scrub starts 2024-09-10T19:16:37.183 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:16:36 smithi183 ceph-mon[41562]: 4.1b scrub ok 2024-09-10T19:16:37.215 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:16:37.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:16:36 smithi191 ceph-mon[32884]: pgmap v266: 148 pgs: 148 active+clean; 1.7 GiB data, 6.6 GiB used, 530 GiB / 536 GiB avail; 3.5 KiB/s rd, 250 MiB/s wr, 4.01k op/s; 176 B/s, 5 objects/s recovering 2024-09-10T19:16:37.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:16:36 smithi191 ceph-mon[32884]: 3.2 scrub starts 2024-09-10T19:16:37.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:16:36 smithi191 ceph-mon[32884]: 3.2 scrub ok 2024-09-10T19:16:37.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:16:36 smithi191 ceph-mon[32884]: 4.1b scrub starts 2024-09-10T19:16:37.217 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:16:36 smithi191 ceph-mon[32884]: 4.1b scrub ok 2024-09-10T19:16:37.247 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:16:37.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:16:37.248 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:16:37.348 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:16:37.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:16:37.370 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:16:37.370 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:16:37.471 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:16:37.506 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:16:37.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:16:37.507 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:16:37.608 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:16:37.648 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:16:37.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:16:37.649 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:16:37.715 INFO:journalctl@ceph.mgr.x.smithi191.stdout:Sep 10 19:16:37 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-x[34967]: ::ffff:172.21.15.191 - - [10/Sep/2024:19:16:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T19:16:37.750 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:16:37.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:16:37.772 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:16:37.772 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:16:37.823 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: 13 16 32330 32314 155.319 86.1875 0.0100955 0.00643297 2024-09-10T19:16:37.872 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:16:37.912 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:16:37.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:16:37.914 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:16:38.014 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:16:38.035 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:16:38.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:16:38.036 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:16:38.137 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:16:38.176 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:16:38.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:16:38.177 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:16:38.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:16:37 smithi183 ceph-mon[29736]: 6.1e scrub starts 2024-09-10T19:16:38.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:16:37 smithi183 ceph-mon[29736]: 6.1e scrub ok 2024-09-10T19:16:38.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:16:37 smithi183 ceph-mon[29736]: 5.1b scrub starts 2024-09-10T19:16:38.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:16:37 smithi183 ceph-mon[29736]: 5.1b scrub ok 2024-09-10T19:16:38.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:16:37 smithi183 ceph-mon[29736]: 2.13 scrub starts 2024-09-10T19:16:38.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:16:37 smithi183 ceph-mon[29736]: 2.13 scrub ok 2024-09-10T19:16:38.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:16:37 smithi183 ceph-mon[41562]: 6.1e scrub starts 2024-09-10T19:16:38.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:16:37 smithi183 ceph-mon[41562]: 6.1e scrub ok 2024-09-10T19:16:38.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:16:37 smithi183 ceph-mon[41562]: 5.1b scrub starts 2024-09-10T19:16:38.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:16:37 smithi183 ceph-mon[41562]: 5.1b scrub ok 2024-09-10T19:16:38.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:16:37 smithi183 ceph-mon[41562]: 2.13 scrub starts 2024-09-10T19:16:38.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:16:37 smithi183 ceph-mon[41562]: 2.13 scrub ok 2024-09-10T19:16:38.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:16:37 smithi191 ceph-mon[32884]: 6.1e scrub starts 2024-09-10T19:16:38.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:16:37 smithi191 ceph-mon[32884]: 6.1e scrub ok 2024-09-10T19:16:38.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:16:37 smithi191 ceph-mon[32884]: 5.1b scrub starts 2024-09-10T19:16:38.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:16:37 smithi191 ceph-mon[32884]: 5.1b scrub ok 2024-09-10T19:16:38.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:16:37 smithi191 ceph-mon[32884]: 2.13 scrub starts 2024-09-10T19:16:38.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:16:37 smithi191 ceph-mon[32884]: 2.13 scrub ok 2024-09-10T19:16:38.278 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:16:38.312 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:16:38.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:16:38.313 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:16:38.413 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:16:38.451 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:16:38.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:16:38.452 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:16:38.553 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:16:38.554 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-10T19:16:38.584 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:16:38.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:16:38.585 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:16:38.686 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:16:38.707 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:16:38.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:16:38.708 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:16:38.808 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:16:38.823 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: 14 16 33872 33856 151.109 96.375 0.00921076 0.00661196 2024-09-10T19:16:38.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:16:38.844 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:16:38.844 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:16:38.945 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:16:38.977 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:16:38.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:16:38.978 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:16:38.989 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-09-10T19:16:38.989 INFO:teuthology.orchestra.run.smithi183.stdout:{ 2024-09-10T19:16:38.989 INFO:teuthology.orchestra.run.smithi183.stdout: "epoch": 160, 2024-09-10T19:16:38.989 INFO:teuthology.orchestra.run.smithi183.stdout: "fsid": "320f6df8-6fa7-11ef-bcea-c7b262605968", 2024-09-10T19:16:38.989 INFO:teuthology.orchestra.run.smithi183.stdout: "created": "2024-09-10T19:05:03.688531+0000", 2024-09-10T19:16:38.989 INFO:teuthology.orchestra.run.smithi183.stdout: "modified": "2024-09-10T19:16:34.735829+0000", 2024-09-10T19:16:38.989 INFO:teuthology.orchestra.run.smithi183.stdout: "last_up_change": "2024-09-10T19:15:19.923894+0000", 2024-09-10T19:16:38.989 INFO:teuthology.orchestra.run.smithi183.stdout: "last_in_change": "2024-09-10T19:16:23.815531+0000", 2024-09-10T19:16:38.990 INFO:teuthology.orchestra.run.smithi183.stdout: "flags": "sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-09-10T19:16:38.990 INFO:teuthology.orchestra.run.smithi183.stdout: "flags_num": 5799936, 2024-09-10T19:16:38.990 INFO:teuthology.orchestra.run.smithi183.stdout: "flags_set": [ 2024-09-10T19:16:38.990 INFO:teuthology.orchestra.run.smithi183.stdout: "pglog_hardlimit", 2024-09-10T19:16:38.990 INFO:teuthology.orchestra.run.smithi183.stdout: "purged_snapdirs", 2024-09-10T19:16:38.990 INFO:teuthology.orchestra.run.smithi183.stdout: "recovery_deletes", 2024-09-10T19:16:38.990 INFO:teuthology.orchestra.run.smithi183.stdout: "sortbitwise" 2024-09-10T19:16:38.990 INFO:teuthology.orchestra.run.smithi183.stdout: ], 2024-09-10T19:16:38.990 INFO:teuthology.orchestra.run.smithi183.stdout: "crush_version": 19, 2024-09-10T19:16:38.990 INFO:teuthology.orchestra.run.smithi183.stdout: "full_ratio": 0.94999998807907104, 2024-09-10T19:16:38.990 INFO:teuthology.orchestra.run.smithi183.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-09-10T19:16:38.990 INFO:teuthology.orchestra.run.smithi183.stdout: "nearfull_ratio": 0.85000002384185791, 2024-09-10T19:16:38.990 INFO:teuthology.orchestra.run.smithi183.stdout: "cluster_snapshot": "", 2024-09-10T19:16:38.991 INFO:teuthology.orchestra.run.smithi183.stdout: "pool_max": 8, 2024-09-10T19:16:38.991 INFO:teuthology.orchestra.run.smithi183.stdout: "max_osd": 8, 2024-09-10T19:16:38.991 INFO:teuthology.orchestra.run.smithi183.stdout: "require_min_compat_client": "luminous", 2024-09-10T19:16:38.991 INFO:teuthology.orchestra.run.smithi183.stdout: "min_compat_client": "luminous", 2024-09-10T19:16:38.991 INFO:teuthology.orchestra.run.smithi183.stdout: "require_osd_release": "quincy", 2024-09-10T19:16:38.991 INFO:teuthology.orchestra.run.smithi183.stdout: "pools": [ 2024-09-10T19:16:38.991 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:16:38.991 INFO:teuthology.orchestra.run.smithi183.stdout: "pool": 1, 2024-09-10T19:16:38.991 INFO:teuthology.orchestra.run.smithi183.stdout: "pool_name": ".mgr", 2024-09-10T19:16:38.991 INFO:teuthology.orchestra.run.smithi183.stdout: "create_time": "2024-09-10T19:08:00.456786+0000", 2024-09-10T19:16:38.991 INFO:teuthology.orchestra.run.smithi183.stdout: "flags": 1, 2024-09-10T19:16:38.991 INFO:teuthology.orchestra.run.smithi183.stdout: "flags_names": "hashpspool", 2024-09-10T19:16:38.991 INFO:teuthology.orchestra.run.smithi183.stdout: "type": 1, 2024-09-10T19:16:38.991 INFO:teuthology.orchestra.run.smithi183.stdout: "size": 3, 2024-09-10T19:16:38.992 INFO:teuthology.orchestra.run.smithi183.stdout: "min_size": 2, 2024-09-10T19:16:38.992 INFO:teuthology.orchestra.run.smithi183.stdout: "crush_rule": 0, 2024-09-10T19:16:38.992 INFO:teuthology.orchestra.run.smithi183.stdout: "peering_crush_bucket_count": 0, 2024-09-10T19:16:38.992 INFO:teuthology.orchestra.run.smithi183.stdout: "peering_crush_bucket_target": 0, 2024-09-10T19:16:38.992 INFO:teuthology.orchestra.run.smithi183.stdout: "peering_crush_bucket_barrier": 0, 2024-09-10T19:16:38.992 INFO:teuthology.orchestra.run.smithi183.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-10T19:16:38.992 INFO:teuthology.orchestra.run.smithi183.stdout: "object_hash": 2, 2024-09-10T19:16:38.992 INFO:teuthology.orchestra.run.smithi183.stdout: "pg_autoscale_mode": "off", 2024-09-10T19:16:38.992 INFO:teuthology.orchestra.run.smithi183.stdout: "pg_num": 1, 2024-09-10T19:16:38.992 INFO:teuthology.orchestra.run.smithi183.stdout: "pg_placement_num": 1, 2024-09-10T19:16:38.992 INFO:teuthology.orchestra.run.smithi183.stdout: "pg_placement_num_target": 1, 2024-09-10T19:16:38.992 INFO:teuthology.orchestra.run.smithi183.stdout: "pg_num_target": 1, 2024-09-10T19:16:38.992 INFO:teuthology.orchestra.run.smithi183.stdout: "pg_num_pending": 1, 2024-09-10T19:16:38.992 INFO:teuthology.orchestra.run.smithi183.stdout: "last_pg_merge_meta": { 2024-09-10T19:16:38.993 INFO:teuthology.orchestra.run.smithi183.stdout: "source_pgid": "0.0", 2024-09-10T19:16:38.993 INFO:teuthology.orchestra.run.smithi183.stdout: "ready_epoch": 0, 2024-09-10T19:16:38.993 INFO:teuthology.orchestra.run.smithi183.stdout: "last_epoch_started": 0, 2024-09-10T19:16:38.993 INFO:teuthology.orchestra.run.smithi183.stdout: "last_epoch_clean": 0, 2024-09-10T19:16:38.993 INFO:teuthology.orchestra.run.smithi183.stdout: "source_version": "0'0", 2024-09-10T19:16:38.993 INFO:teuthology.orchestra.run.smithi183.stdout: "target_version": "0'0" 2024-09-10T19:16:38.993 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:16:38.993 INFO:teuthology.orchestra.run.smithi183.stdout: "last_change": "22", 2024-09-10T19:16:38.993 INFO:teuthology.orchestra.run.smithi183.stdout: "last_force_op_resend": "0", 2024-09-10T19:16:38.993 INFO:teuthology.orchestra.run.smithi183.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-10T19:16:38.993 INFO:teuthology.orchestra.run.smithi183.stdout: "last_force_op_resend_preluminous": "0", 2024-09-10T19:16:38.993 INFO:teuthology.orchestra.run.smithi183.stdout: "auid": 0, 2024-09-10T19:16:38.993 INFO:teuthology.orchestra.run.smithi183.stdout: "snap_mode": "selfmanaged", 2024-09-10T19:16:38.994 INFO:teuthology.orchestra.run.smithi183.stdout: "snap_seq": 0, 2024-09-10T19:16:38.994 INFO:teuthology.orchestra.run.smithi183.stdout: "snap_epoch": 0, 2024-09-10T19:16:38.994 INFO:teuthology.orchestra.run.smithi183.stdout: "pool_snaps": [], 2024-09-10T19:16:38.994 INFO:teuthology.orchestra.run.smithi183.stdout: "removed_snaps": "[]", 2024-09-10T19:16:38.994 INFO:teuthology.orchestra.run.smithi183.stdout: "quota_max_bytes": 0, 2024-09-10T19:16:38.994 INFO:teuthology.orchestra.run.smithi183.stdout: "quota_max_objects": 0, 2024-09-10T19:16:38.994 INFO:teuthology.orchestra.run.smithi183.stdout: "tiers": [], 2024-09-10T19:16:38.994 INFO:teuthology.orchestra.run.smithi183.stdout: "tier_of": -1, 2024-09-10T19:16:38.994 INFO:teuthology.orchestra.run.smithi183.stdout: "read_tier": -1, 2024-09-10T19:16:38.994 INFO:teuthology.orchestra.run.smithi183.stdout: "write_tier": -1, 2024-09-10T19:16:38.995 INFO:teuthology.orchestra.run.smithi183.stdout: "cache_mode": "none", 2024-09-10T19:16:38.995 INFO:teuthology.orchestra.run.smithi183.stdout: "target_max_bytes": 0, 2024-09-10T19:16:38.995 INFO:teuthology.orchestra.run.smithi183.stdout: "target_max_objects": 0, 2024-09-10T19:16:38.995 INFO:teuthology.orchestra.run.smithi183.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-10T19:16:38.995 INFO:teuthology.orchestra.run.smithi183.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-10T19:16:38.995 INFO:teuthology.orchestra.run.smithi183.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-10T19:16:38.995 INFO:teuthology.orchestra.run.smithi183.stdout: "cache_min_flush_age": 0, 2024-09-10T19:16:38.995 INFO:teuthology.orchestra.run.smithi183.stdout: "cache_min_evict_age": 0, 2024-09-10T19:16:38.995 INFO:teuthology.orchestra.run.smithi183.stdout: "erasure_code_profile": "", 2024-09-10T19:16:38.995 INFO:teuthology.orchestra.run.smithi183.stdout: "hit_set_params": { 2024-09-10T19:16:38.995 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "none" 2024-09-10T19:16:38.995 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:16:38.995 INFO:teuthology.orchestra.run.smithi183.stdout: "hit_set_period": 0, 2024-09-10T19:16:38.995 INFO:teuthology.orchestra.run.smithi183.stdout: "hit_set_count": 0, 2024-09-10T19:16:38.996 INFO:teuthology.orchestra.run.smithi183.stdout: "use_gmt_hitset": true, 2024-09-10T19:16:38.996 INFO:teuthology.orchestra.run.smithi183.stdout: "min_read_recency_for_promote": 0, 2024-09-10T19:16:38.996 INFO:teuthology.orchestra.run.smithi183.stdout: "min_write_recency_for_promote": 0, 2024-09-10T19:16:38.996 INFO:teuthology.orchestra.run.smithi183.stdout: "hit_set_grade_decay_rate": 0, 2024-09-10T19:16:38.996 INFO:teuthology.orchestra.run.smithi183.stdout: "hit_set_search_last_n": 0, 2024-09-10T19:16:38.996 INFO:teuthology.orchestra.run.smithi183.stdout: "grade_table": [], 2024-09-10T19:16:38.996 INFO:teuthology.orchestra.run.smithi183.stdout: "stripe_width": 0, 2024-09-10T19:16:38.996 INFO:teuthology.orchestra.run.smithi183.stdout: "expected_num_objects": 0, 2024-09-10T19:16:38.996 INFO:teuthology.orchestra.run.smithi183.stdout: "fast_read": false, 2024-09-10T19:16:38.996 INFO:teuthology.orchestra.run.smithi183.stdout: "options": { 2024-09-10T19:16:38.996 INFO:teuthology.orchestra.run.smithi183.stdout: "pg_num_max": 32, 2024-09-10T19:16:38.996 INFO:teuthology.orchestra.run.smithi183.stdout: "pg_num_min": 1 2024-09-10T19:16:38.996 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:16:38.996 INFO:teuthology.orchestra.run.smithi183.stdout: "application_metadata": { 2024-09-10T19:16:38.997 INFO:teuthology.orchestra.run.smithi183.stdout: "mgr": {} 2024-09-10T19:16:38.997 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:16:38.997 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:16:38.997 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:16:38.997 INFO:teuthology.orchestra.run.smithi183.stdout: "pool": 2, 2024-09-10T19:16:38.997 INFO:teuthology.orchestra.run.smithi183.stdout: "pool_name": ".rgw.root", 2024-09-10T19:16:38.997 INFO:teuthology.orchestra.run.smithi183.stdout: "create_time": "2024-09-10T19:10:05.993763+0000", 2024-09-10T19:16:38.997 INFO:teuthology.orchestra.run.smithi183.stdout: "flags": 1, 2024-09-10T19:16:38.997 INFO:teuthology.orchestra.run.smithi183.stdout: "flags_names": "hashpspool", 2024-09-10T19:16:38.997 INFO:teuthology.orchestra.run.smithi183.stdout: "type": 1, 2024-09-10T19:16:38.997 INFO:teuthology.orchestra.run.smithi183.stdout: "size": 3, 2024-09-10T19:16:38.997 INFO:teuthology.orchestra.run.smithi183.stdout: "min_size": 2, 2024-09-10T19:16:38.997 INFO:teuthology.orchestra.run.smithi183.stdout: "crush_rule": 0, 2024-09-10T19:16:38.997 INFO:teuthology.orchestra.run.smithi183.stdout: "peering_crush_bucket_count": 0, 2024-09-10T19:16:38.998 INFO:teuthology.orchestra.run.smithi183.stdout: "peering_crush_bucket_target": 0, 2024-09-10T19:16:38.998 INFO:teuthology.orchestra.run.smithi183.stdout: "peering_crush_bucket_barrier": 0, 2024-09-10T19:16:38.998 INFO:teuthology.orchestra.run.smithi183.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-10T19:16:38.998 INFO:teuthology.orchestra.run.smithi183.stdout: "object_hash": 2, 2024-09-10T19:16:38.998 INFO:teuthology.orchestra.run.smithi183.stdout: "pg_autoscale_mode": "off", 2024-09-10T19:16:38.998 INFO:teuthology.orchestra.run.smithi183.stdout: "pg_num": 32, 2024-09-10T19:16:38.998 INFO:teuthology.orchestra.run.smithi183.stdout: "pg_placement_num": 32, 2024-09-10T19:16:38.998 INFO:teuthology.orchestra.run.smithi183.stdout: "pg_placement_num_target": 32, 2024-09-10T19:16:38.998 INFO:teuthology.orchestra.run.smithi183.stdout: "pg_num_target": 32, 2024-09-10T19:16:38.998 INFO:teuthology.orchestra.run.smithi183.stdout: "pg_num_pending": 32, 2024-09-10T19:16:38.998 INFO:teuthology.orchestra.run.smithi183.stdout: "last_pg_merge_meta": { 2024-09-10T19:16:38.998 INFO:teuthology.orchestra.run.smithi183.stdout: "source_pgid": "0.0", 2024-09-10T19:16:38.998 INFO:teuthology.orchestra.run.smithi183.stdout: "ready_epoch": 0, 2024-09-10T19:16:38.998 INFO:teuthology.orchestra.run.smithi183.stdout: "last_epoch_started": 0, 2024-09-10T19:16:38.999 INFO:teuthology.orchestra.run.smithi183.stdout: "last_epoch_clean": 0, 2024-09-10T19:16:38.999 INFO:teuthology.orchestra.run.smithi183.stdout: "source_version": "0'0", 2024-09-10T19:16:38.999 INFO:teuthology.orchestra.run.smithi183.stdout: "target_version": "0'0" 2024-09-10T19:16:38.999 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:16:38.999 INFO:teuthology.orchestra.run.smithi183.stdout: "last_change": "60", 2024-09-10T19:16:38.999 INFO:teuthology.orchestra.run.smithi183.stdout: "last_force_op_resend": "0", 2024-09-10T19:16:38.999 INFO:teuthology.orchestra.run.smithi183.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-10T19:16:38.999 INFO:teuthology.orchestra.run.smithi183.stdout: "last_force_op_resend_preluminous": "0", 2024-09-10T19:16:38.999 INFO:teuthology.orchestra.run.smithi183.stdout: "auid": 0, 2024-09-10T19:16:38.999 INFO:teuthology.orchestra.run.smithi183.stdout: "snap_mode": "selfmanaged", 2024-09-10T19:16:38.999 INFO:teuthology.orchestra.run.smithi183.stdout: "snap_seq": 0, 2024-09-10T19:16:38.999 INFO:teuthology.orchestra.run.smithi183.stdout: "snap_epoch": 0, 2024-09-10T19:16:38.999 INFO:teuthology.orchestra.run.smithi183.stdout: "pool_snaps": [], 2024-09-10T19:16:38.999 INFO:teuthology.orchestra.run.smithi183.stdout: "removed_snaps": "[]", 2024-09-10T19:16:38.999 INFO:teuthology.orchestra.run.smithi183.stdout: "quota_max_bytes": 0, 2024-09-10T19:16:39.000 INFO:teuthology.orchestra.run.smithi183.stdout: "quota_max_objects": 0, 2024-09-10T19:16:39.000 INFO:teuthology.orchestra.run.smithi183.stdout: "tiers": [], 2024-09-10T19:16:39.000 INFO:teuthology.orchestra.run.smithi183.stdout: "tier_of": -1, 2024-09-10T19:16:39.000 INFO:teuthology.orchestra.run.smithi183.stdout: "read_tier": -1, 2024-09-10T19:16:39.000 INFO:teuthology.orchestra.run.smithi183.stdout: "write_tier": -1, 2024-09-10T19:16:39.000 INFO:teuthology.orchestra.run.smithi183.stdout: "cache_mode": "none", 2024-09-10T19:16:39.000 INFO:teuthology.orchestra.run.smithi183.stdout: "target_max_bytes": 0, 2024-09-10T19:16:39.000 INFO:teuthology.orchestra.run.smithi183.stdout: "target_max_objects": 0, 2024-09-10T19:16:39.000 INFO:teuthology.orchestra.run.smithi183.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-10T19:16:39.000 INFO:teuthology.orchestra.run.smithi183.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-10T19:16:39.000 INFO:teuthology.orchestra.run.smithi183.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-10T19:16:39.000 INFO:teuthology.orchestra.run.smithi183.stdout: "cache_min_flush_age": 0, 2024-09-10T19:16:39.000 INFO:teuthology.orchestra.run.smithi183.stdout: "cache_min_evict_age": 0, 2024-09-10T19:16:39.000 INFO:teuthology.orchestra.run.smithi183.stdout: "erasure_code_profile": "", 2024-09-10T19:16:39.001 INFO:teuthology.orchestra.run.smithi183.stdout: "hit_set_params": { 2024-09-10T19:16:39.001 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "none" 2024-09-10T19:16:39.001 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:16:39.001 INFO:teuthology.orchestra.run.smithi183.stdout: "hit_set_period": 0, 2024-09-10T19:16:39.001 INFO:teuthology.orchestra.run.smithi183.stdout: "hit_set_count": 0, 2024-09-10T19:16:39.001 INFO:teuthology.orchestra.run.smithi183.stdout: "use_gmt_hitset": true, 2024-09-10T19:16:39.001 INFO:teuthology.orchestra.run.smithi183.stdout: "min_read_recency_for_promote": 0, 2024-09-10T19:16:39.001 INFO:teuthology.orchestra.run.smithi183.stdout: "min_write_recency_for_promote": 0, 2024-09-10T19:16:39.002 INFO:teuthology.orchestra.run.smithi183.stdout: "hit_set_grade_decay_rate": 0, 2024-09-10T19:16:39.002 INFO:teuthology.orchestra.run.smithi183.stdout: "hit_set_search_last_n": 0, 2024-09-10T19:16:39.002 INFO:teuthology.orchestra.run.smithi183.stdout: "grade_table": [], 2024-09-10T19:16:39.002 INFO:teuthology.orchestra.run.smithi183.stdout: "stripe_width": 0, 2024-09-10T19:16:39.002 INFO:teuthology.orchestra.run.smithi183.stdout: "expected_num_objects": 0, 2024-09-10T19:16:39.002 INFO:teuthology.orchestra.run.smithi183.stdout: "fast_read": false, 2024-09-10T19:16:39.002 INFO:teuthology.orchestra.run.smithi183.stdout: "options": {}, 2024-09-10T19:16:39.002 INFO:teuthology.orchestra.run.smithi183.stdout: "application_metadata": { 2024-09-10T19:16:39.002 INFO:teuthology.orchestra.run.smithi183.stdout: "rgw": {} 2024-09-10T19:16:39.003 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:16:39.003 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:16:39.003 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:16:39.003 INFO:teuthology.orchestra.run.smithi183.stdout: "pool": 3, 2024-09-10T19:16:39.003 INFO:teuthology.orchestra.run.smithi183.stdout: "pool_name": "datapool", 2024-09-10T19:16:39.003 INFO:teuthology.orchestra.run.smithi183.stdout: "create_time": "2024-09-10T19:10:06.224267+0000", 2024-09-10T19:16:39.003 INFO:teuthology.orchestra.run.smithi183.stdout: "flags": 8193, 2024-09-10T19:16:39.003 INFO:teuthology.orchestra.run.smithi183.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-09-10T19:16:39.003 INFO:teuthology.orchestra.run.smithi183.stdout: "type": 1, 2024-09-10T19:16:39.003 INFO:teuthology.orchestra.run.smithi183.stdout: "size": 3, 2024-09-10T19:16:39.003 INFO:teuthology.orchestra.run.smithi183.stdout: "min_size": 2, 2024-09-10T19:16:39.003 INFO:teuthology.orchestra.run.smithi183.stdout: "crush_rule": 0, 2024-09-10T19:16:39.003 INFO:teuthology.orchestra.run.smithi183.stdout: "peering_crush_bucket_count": 0, 2024-09-10T19:16:39.004 INFO:teuthology.orchestra.run.smithi183.stdout: "peering_crush_bucket_target": 0, 2024-09-10T19:16:39.004 INFO:teuthology.orchestra.run.smithi183.stdout: "peering_crush_bucket_barrier": 0, 2024-09-10T19:16:39.004 INFO:teuthology.orchestra.run.smithi183.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-10T19:16:39.004 INFO:teuthology.orchestra.run.smithi183.stdout: "object_hash": 2, 2024-09-10T19:16:39.004 INFO:teuthology.orchestra.run.smithi183.stdout: "pg_autoscale_mode": "off", 2024-09-10T19:16:39.004 INFO:teuthology.orchestra.run.smithi183.stdout: "pg_num": 3, 2024-09-10T19:16:39.004 INFO:teuthology.orchestra.run.smithi183.stdout: "pg_placement_num": 3, 2024-09-10T19:16:39.004 INFO:teuthology.orchestra.run.smithi183.stdout: "pg_placement_num_target": 3, 2024-09-10T19:16:39.004 INFO:teuthology.orchestra.run.smithi183.stdout: "pg_num_target": 3, 2024-09-10T19:16:39.004 INFO:teuthology.orchestra.run.smithi183.stdout: "pg_num_pending": 3, 2024-09-10T19:16:39.005 INFO:teuthology.orchestra.run.smithi183.stdout: "last_pg_merge_meta": { 2024-09-10T19:16:39.005 INFO:teuthology.orchestra.run.smithi183.stdout: "source_pgid": "0.0", 2024-09-10T19:16:39.005 INFO:teuthology.orchestra.run.smithi183.stdout: "ready_epoch": 0, 2024-09-10T19:16:39.005 INFO:teuthology.orchestra.run.smithi183.stdout: "last_epoch_started": 0, 2024-09-10T19:16:39.005 INFO:teuthology.orchestra.run.smithi183.stdout: "last_epoch_clean": 0, 2024-09-10T19:16:39.005 INFO:teuthology.orchestra.run.smithi183.stdout: "source_version": "0'0", 2024-09-10T19:16:39.005 INFO:teuthology.orchestra.run.smithi183.stdout: "target_version": "0'0" 2024-09-10T19:16:39.005 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:16:39.005 INFO:teuthology.orchestra.run.smithi183.stdout: "last_change": "63", 2024-09-10T19:16:39.005 INFO:teuthology.orchestra.run.smithi183.stdout: "last_force_op_resend": "0", 2024-09-10T19:16:39.005 INFO:teuthology.orchestra.run.smithi183.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-10T19:16:39.005 INFO:teuthology.orchestra.run.smithi183.stdout: "last_force_op_resend_preluminous": "0", 2024-09-10T19:16:39.006 INFO:teuthology.orchestra.run.smithi183.stdout: "auid": 0, 2024-09-10T19:16:39.006 INFO:teuthology.orchestra.run.smithi183.stdout: "snap_mode": "selfmanaged", 2024-09-10T19:16:39.006 INFO:teuthology.orchestra.run.smithi183.stdout: "snap_seq": 3, 2024-09-10T19:16:39.006 INFO:teuthology.orchestra.run.smithi183.stdout: "snap_epoch": 63, 2024-09-10T19:16:39.006 INFO:teuthology.orchestra.run.smithi183.stdout: "pool_snaps": [], 2024-09-10T19:16:39.006 INFO:teuthology.orchestra.run.smithi183.stdout: "removed_snaps": "[]", 2024-09-10T19:16:39.006 INFO:teuthology.orchestra.run.smithi183.stdout: "quota_max_bytes": 0, 2024-09-10T19:16:39.006 INFO:teuthology.orchestra.run.smithi183.stdout: "quota_max_objects": 0, 2024-09-10T19:16:39.006 INFO:teuthology.orchestra.run.smithi183.stdout: "tiers": [], 2024-09-10T19:16:39.006 INFO:teuthology.orchestra.run.smithi183.stdout: "tier_of": -1, 2024-09-10T19:16:39.006 INFO:teuthology.orchestra.run.smithi183.stdout: "read_tier": -1, 2024-09-10T19:16:39.007 INFO:teuthology.orchestra.run.smithi183.stdout: "write_tier": -1, 2024-09-10T19:16:39.007 INFO:teuthology.orchestra.run.smithi183.stdout: "cache_mode": "none", 2024-09-10T19:16:39.007 INFO:teuthology.orchestra.run.smithi183.stdout: "target_max_bytes": 0, 2024-09-10T19:16:39.007 INFO:teuthology.orchestra.run.smithi183.stdout: "target_max_objects": 0, 2024-09-10T19:16:39.007 INFO:teuthology.orchestra.run.smithi183.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-10T19:16:39.007 INFO:teuthology.orchestra.run.smithi183.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-10T19:16:39.007 INFO:teuthology.orchestra.run.smithi183.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-10T19:16:39.007 INFO:teuthology.orchestra.run.smithi183.stdout: "cache_min_flush_age": 0, 2024-09-10T19:16:39.007 INFO:teuthology.orchestra.run.smithi183.stdout: "cache_min_evict_age": 0, 2024-09-10T19:16:39.007 INFO:teuthology.orchestra.run.smithi183.stdout: "erasure_code_profile": "", 2024-09-10T19:16:39.007 INFO:teuthology.orchestra.run.smithi183.stdout: "hit_set_params": { 2024-09-10T19:16:39.007 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "none" 2024-09-10T19:16:39.007 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:16:39.008 INFO:teuthology.orchestra.run.smithi183.stdout: "hit_set_period": 0, 2024-09-10T19:16:39.008 INFO:teuthology.orchestra.run.smithi183.stdout: "hit_set_count": 0, 2024-09-10T19:16:39.008 INFO:teuthology.orchestra.run.smithi183.stdout: "use_gmt_hitset": true, 2024-09-10T19:16:39.008 INFO:teuthology.orchestra.run.smithi183.stdout: "min_read_recency_for_promote": 0, 2024-09-10T19:16:39.008 INFO:teuthology.orchestra.run.smithi183.stdout: "min_write_recency_for_promote": 0, 2024-09-10T19:16:39.008 INFO:teuthology.orchestra.run.smithi183.stdout: "hit_set_grade_decay_rate": 0, 2024-09-10T19:16:39.008 INFO:teuthology.orchestra.run.smithi183.stdout: "hit_set_search_last_n": 0, 2024-09-10T19:16:39.008 INFO:teuthology.orchestra.run.smithi183.stdout: "grade_table": [], 2024-09-10T19:16:39.008 INFO:teuthology.orchestra.run.smithi183.stdout: "stripe_width": 0, 2024-09-10T19:16:39.008 INFO:teuthology.orchestra.run.smithi183.stdout: "expected_num_objects": 0, 2024-09-10T19:16:39.008 INFO:teuthology.orchestra.run.smithi183.stdout: "fast_read": false, 2024-09-10T19:16:39.008 INFO:teuthology.orchestra.run.smithi183.stdout: "options": {}, 2024-09-10T19:16:39.008 INFO:teuthology.orchestra.run.smithi183.stdout: "application_metadata": { 2024-09-10T19:16:39.009 INFO:teuthology.orchestra.run.smithi183.stdout: "rbd": {} 2024-09-10T19:16:39.009 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:16:39.009 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:16:39.009 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:16:39.009 INFO:teuthology.orchestra.run.smithi183.stdout: "pool": 4, 2024-09-10T19:16:39.009 INFO:teuthology.orchestra.run.smithi183.stdout: "pool_name": "default.rgw.log", 2024-09-10T19:16:39.009 INFO:teuthology.orchestra.run.smithi183.stdout: "create_time": "2024-09-10T19:10:07.958284+0000", 2024-09-10T19:16:39.009 INFO:teuthology.orchestra.run.smithi183.stdout: "flags": 1, 2024-09-10T19:16:39.009 INFO:teuthology.orchestra.run.smithi183.stdout: "flags_names": "hashpspool", 2024-09-10T19:16:39.009 INFO:teuthology.orchestra.run.smithi183.stdout: "type": 1, 2024-09-10T19:16:39.009 INFO:teuthology.orchestra.run.smithi183.stdout: "size": 3, 2024-09-10T19:16:39.009 INFO:teuthology.orchestra.run.smithi183.stdout: "min_size": 2, 2024-09-10T19:16:39.009 INFO:teuthology.orchestra.run.smithi183.stdout: "crush_rule": 0, 2024-09-10T19:16:39.009 INFO:teuthology.orchestra.run.smithi183.stdout: "peering_crush_bucket_count": 0, 2024-09-10T19:16:39.010 INFO:teuthology.orchestra.run.smithi183.stdout: "peering_crush_bucket_target": 0, 2024-09-10T19:16:39.010 INFO:teuthology.orchestra.run.smithi183.stdout: "peering_crush_bucket_barrier": 0, 2024-09-10T19:16:39.010 INFO:teuthology.orchestra.run.smithi183.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-10T19:16:39.010 INFO:teuthology.orchestra.run.smithi183.stdout: "object_hash": 2, 2024-09-10T19:16:39.010 INFO:teuthology.orchestra.run.smithi183.stdout: "pg_autoscale_mode": "off", 2024-09-10T19:16:39.010 INFO:teuthology.orchestra.run.smithi183.stdout: "pg_num": 32, 2024-09-10T19:16:39.010 INFO:teuthology.orchestra.run.smithi183.stdout: "pg_placement_num": 32, 2024-09-10T19:16:39.010 INFO:teuthology.orchestra.run.smithi183.stdout: "pg_placement_num_target": 32, 2024-09-10T19:16:39.010 INFO:teuthology.orchestra.run.smithi183.stdout: "pg_num_target": 32, 2024-09-10T19:16:39.010 INFO:teuthology.orchestra.run.smithi183.stdout: "pg_num_pending": 32, 2024-09-10T19:16:39.010 INFO:teuthology.orchestra.run.smithi183.stdout: "last_pg_merge_meta": { 2024-09-10T19:16:39.011 INFO:teuthology.orchestra.run.smithi183.stdout: "source_pgid": "0.0", 2024-09-10T19:16:39.011 INFO:teuthology.orchestra.run.smithi183.stdout: "ready_epoch": 0, 2024-09-10T19:16:39.011 INFO:teuthology.orchestra.run.smithi183.stdout: "last_epoch_started": 0, 2024-09-10T19:16:39.011 INFO:teuthology.orchestra.run.smithi183.stdout: "last_epoch_clean": 0, 2024-09-10T19:16:39.011 INFO:teuthology.orchestra.run.smithi183.stdout: "source_version": "0'0", 2024-09-10T19:16:39.011 INFO:teuthology.orchestra.run.smithi183.stdout: "target_version": "0'0" 2024-09-10T19:16:39.011 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:16:39.011 INFO:teuthology.orchestra.run.smithi183.stdout: "last_change": "62", 2024-09-10T19:16:39.011 INFO:teuthology.orchestra.run.smithi183.stdout: "last_force_op_resend": "0", 2024-09-10T19:16:39.011 INFO:teuthology.orchestra.run.smithi183.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-10T19:16:39.011 INFO:teuthology.orchestra.run.smithi183.stdout: "last_force_op_resend_preluminous": "0", 2024-09-10T19:16:39.011 INFO:teuthology.orchestra.run.smithi183.stdout: "auid": 0, 2024-09-10T19:16:39.011 INFO:teuthology.orchestra.run.smithi183.stdout: "snap_mode": "selfmanaged", 2024-09-10T19:16:39.011 INFO:teuthology.orchestra.run.smithi183.stdout: "snap_seq": 0, 2024-09-10T19:16:39.012 INFO:teuthology.orchestra.run.smithi183.stdout: "snap_epoch": 0, 2024-09-10T19:16:39.012 INFO:teuthology.orchestra.run.smithi183.stdout: "pool_snaps": [], 2024-09-10T19:16:39.012 INFO:teuthology.orchestra.run.smithi183.stdout: "removed_snaps": "[]", 2024-09-10T19:16:39.012 INFO:teuthology.orchestra.run.smithi183.stdout: "quota_max_bytes": 0, 2024-09-10T19:16:39.012 INFO:teuthology.orchestra.run.smithi183.stdout: "quota_max_objects": 0, 2024-09-10T19:16:39.012 INFO:teuthology.orchestra.run.smithi183.stdout: "tiers": [], 2024-09-10T19:16:39.012 INFO:teuthology.orchestra.run.smithi183.stdout: "tier_of": -1, 2024-09-10T19:16:39.012 INFO:teuthology.orchestra.run.smithi183.stdout: "read_tier": -1, 2024-09-10T19:16:39.012 INFO:teuthology.orchestra.run.smithi183.stdout: "write_tier": -1, 2024-09-10T19:16:39.013 INFO:teuthology.orchestra.run.smithi183.stdout: "cache_mode": "none", 2024-09-10T19:16:39.013 INFO:teuthology.orchestra.run.smithi183.stdout: "target_max_bytes": 0, 2024-09-10T19:16:39.013 INFO:teuthology.orchestra.run.smithi183.stdout: "target_max_objects": 0, 2024-09-10T19:16:39.013 INFO:teuthology.orchestra.run.smithi183.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-10T19:16:39.013 INFO:teuthology.orchestra.run.smithi183.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-10T19:16:39.014 INFO:teuthology.orchestra.run.smithi183.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-10T19:16:39.014 INFO:teuthology.orchestra.run.smithi183.stdout: "cache_min_flush_age": 0, 2024-09-10T19:16:39.014 INFO:teuthology.orchestra.run.smithi183.stdout: "cache_min_evict_age": 0, 2024-09-10T19:16:39.014 INFO:teuthology.orchestra.run.smithi183.stdout: "erasure_code_profile": "", 2024-09-10T19:16:39.014 INFO:teuthology.orchestra.run.smithi183.stdout: "hit_set_params": { 2024-09-10T19:16:39.014 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "none" 2024-09-10T19:16:39.015 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:16:39.015 INFO:teuthology.orchestra.run.smithi183.stdout: "hit_set_period": 0, 2024-09-10T19:16:39.015 INFO:teuthology.orchestra.run.smithi183.stdout: "hit_set_count": 0, 2024-09-10T19:16:39.015 INFO:teuthology.orchestra.run.smithi183.stdout: "use_gmt_hitset": true, 2024-09-10T19:16:39.015 INFO:teuthology.orchestra.run.smithi183.stdout: "min_read_recency_for_promote": 0, 2024-09-10T19:16:39.016 INFO:teuthology.orchestra.run.smithi183.stdout: "min_write_recency_for_promote": 0, 2024-09-10T19:16:39.016 INFO:teuthology.orchestra.run.smithi183.stdout: "hit_set_grade_decay_rate": 0, 2024-09-10T19:16:39.016 INFO:teuthology.orchestra.run.smithi183.stdout: "hit_set_search_last_n": 0, 2024-09-10T19:16:39.016 INFO:teuthology.orchestra.run.smithi183.stdout: "grade_table": [], 2024-09-10T19:16:39.016 INFO:teuthology.orchestra.run.smithi183.stdout: "stripe_width": 0, 2024-09-10T19:16:39.016 INFO:teuthology.orchestra.run.smithi183.stdout: "expected_num_objects": 0, 2024-09-10T19:16:39.016 INFO:teuthology.orchestra.run.smithi183.stdout: "fast_read": false, 2024-09-10T19:16:39.016 INFO:teuthology.orchestra.run.smithi183.stdout: "options": {}, 2024-09-10T19:16:39.016 INFO:teuthology.orchestra.run.smithi183.stdout: "application_metadata": { 2024-09-10T19:16:39.016 INFO:teuthology.orchestra.run.smithi183.stdout: "rgw": {} 2024-09-10T19:16:39.016 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:16:39.016 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:16:39.017 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:16:39.017 INFO:teuthology.orchestra.run.smithi183.stdout: "pool": 5, 2024-09-10T19:16:39.017 INFO:teuthology.orchestra.run.smithi183.stdout: "pool_name": "default.rgw.control", 2024-09-10T19:16:39.017 INFO:teuthology.orchestra.run.smithi183.stdout: "create_time": "2024-09-10T19:10:09.933596+0000", 2024-09-10T19:16:39.017 INFO:teuthology.orchestra.run.smithi183.stdout: "flags": 1, 2024-09-10T19:16:39.017 INFO:teuthology.orchestra.run.smithi183.stdout: "flags_names": "hashpspool", 2024-09-10T19:16:39.017 INFO:teuthology.orchestra.run.smithi183.stdout: "type": 1, 2024-09-10T19:16:39.017 INFO:teuthology.orchestra.run.smithi183.stdout: "size": 3, 2024-09-10T19:16:39.017 INFO:teuthology.orchestra.run.smithi183.stdout: "min_size": 2, 2024-09-10T19:16:39.017 INFO:teuthology.orchestra.run.smithi183.stdout: "crush_rule": 0, 2024-09-10T19:16:39.017 INFO:teuthology.orchestra.run.smithi183.stdout: "peering_crush_bucket_count": 0, 2024-09-10T19:16:39.017 INFO:teuthology.orchestra.run.smithi183.stdout: "peering_crush_bucket_target": 0, 2024-09-10T19:16:39.017 INFO:teuthology.orchestra.run.smithi183.stdout: "peering_crush_bucket_barrier": 0, 2024-09-10T19:16:39.017 INFO:teuthology.orchestra.run.smithi183.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-10T19:16:39.018 INFO:teuthology.orchestra.run.smithi183.stdout: "object_hash": 2, 2024-09-10T19:16:39.018 INFO:teuthology.orchestra.run.smithi183.stdout: "pg_autoscale_mode": "off", 2024-09-10T19:16:39.018 INFO:teuthology.orchestra.run.smithi183.stdout: "pg_num": 32, 2024-09-10T19:16:39.018 INFO:teuthology.orchestra.run.smithi183.stdout: "pg_placement_num": 32, 2024-09-10T19:16:39.018 INFO:teuthology.orchestra.run.smithi183.stdout: "pg_placement_num_target": 32, 2024-09-10T19:16:39.018 INFO:teuthology.orchestra.run.smithi183.stdout: "pg_num_target": 32, 2024-09-10T19:16:39.018 INFO:teuthology.orchestra.run.smithi183.stdout: "pg_num_pending": 32, 2024-09-10T19:16:39.018 INFO:teuthology.orchestra.run.smithi183.stdout: "last_pg_merge_meta": { 2024-09-10T19:16:39.018 INFO:teuthology.orchestra.run.smithi183.stdout: "source_pgid": "0.0", 2024-09-10T19:16:39.018 INFO:teuthology.orchestra.run.smithi183.stdout: "ready_epoch": 0, 2024-09-10T19:16:39.018 INFO:teuthology.orchestra.run.smithi183.stdout: "last_epoch_started": 0, 2024-09-10T19:16:39.018 INFO:teuthology.orchestra.run.smithi183.stdout: "last_epoch_clean": 0, 2024-09-10T19:16:39.018 INFO:teuthology.orchestra.run.smithi183.stdout: "source_version": "0'0", 2024-09-10T19:16:39.019 INFO:teuthology.orchestra.run.smithi183.stdout: "target_version": "0'0" 2024-09-10T19:16:39.019 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:16:39.019 INFO:teuthology.orchestra.run.smithi183.stdout: "last_change": "64", 2024-09-10T19:16:39.019 INFO:teuthology.orchestra.run.smithi183.stdout: "last_force_op_resend": "0", 2024-09-10T19:16:39.019 INFO:teuthology.orchestra.run.smithi183.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-10T19:16:39.019 INFO:teuthology.orchestra.run.smithi183.stdout: "last_force_op_resend_preluminous": "0", 2024-09-10T19:16:39.019 INFO:teuthology.orchestra.run.smithi183.stdout: "auid": 0, 2024-09-10T19:16:39.019 INFO:teuthology.orchestra.run.smithi183.stdout: "snap_mode": "selfmanaged", 2024-09-10T19:16:39.019 INFO:teuthology.orchestra.run.smithi183.stdout: "snap_seq": 0, 2024-09-10T19:16:39.019 INFO:teuthology.orchestra.run.smithi183.stdout: "snap_epoch": 0, 2024-09-10T19:16:39.019 INFO:teuthology.orchestra.run.smithi183.stdout: "pool_snaps": [], 2024-09-10T19:16:39.019 INFO:teuthology.orchestra.run.smithi183.stdout: "removed_snaps": "[]", 2024-09-10T19:16:39.020 INFO:teuthology.orchestra.run.smithi183.stdout: "quota_max_bytes": 0, 2024-09-10T19:16:39.020 INFO:teuthology.orchestra.run.smithi183.stdout: "quota_max_objects": 0, 2024-09-10T19:16:39.020 INFO:teuthology.orchestra.run.smithi183.stdout: "tiers": [], 2024-09-10T19:16:39.020 INFO:teuthology.orchestra.run.smithi183.stdout: "tier_of": -1, 2024-09-10T19:16:39.020 INFO:teuthology.orchestra.run.smithi183.stdout: "read_tier": -1, 2024-09-10T19:16:39.020 INFO:teuthology.orchestra.run.smithi183.stdout: "write_tier": -1, 2024-09-10T19:16:39.020 INFO:teuthology.orchestra.run.smithi183.stdout: "cache_mode": "none", 2024-09-10T19:16:39.020 INFO:teuthology.orchestra.run.smithi183.stdout: "target_max_bytes": 0, 2024-09-10T19:16:39.020 INFO:teuthology.orchestra.run.smithi183.stdout: "target_max_objects": 0, 2024-09-10T19:16:39.020 INFO:teuthology.orchestra.run.smithi183.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-10T19:16:39.020 INFO:teuthology.orchestra.run.smithi183.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-10T19:16:39.021 INFO:teuthology.orchestra.run.smithi183.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-10T19:16:39.021 INFO:teuthology.orchestra.run.smithi183.stdout: "cache_min_flush_age": 0, 2024-09-10T19:16:39.021 INFO:teuthology.orchestra.run.smithi183.stdout: "cache_min_evict_age": 0, 2024-09-10T19:16:39.021 INFO:teuthology.orchestra.run.smithi183.stdout: "erasure_code_profile": "", 2024-09-10T19:16:39.021 INFO:teuthology.orchestra.run.smithi183.stdout: "hit_set_params": { 2024-09-10T19:16:39.021 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "none" 2024-09-10T19:16:39.021 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:16:39.021 INFO:teuthology.orchestra.run.smithi183.stdout: "hit_set_period": 0, 2024-09-10T19:16:39.021 INFO:teuthology.orchestra.run.smithi183.stdout: "hit_set_count": 0, 2024-09-10T19:16:39.021 INFO:teuthology.orchestra.run.smithi183.stdout: "use_gmt_hitset": true, 2024-09-10T19:16:39.021 INFO:teuthology.orchestra.run.smithi183.stdout: "min_read_recency_for_promote": 0, 2024-09-10T19:16:39.021 INFO:teuthology.orchestra.run.smithi183.stdout: "min_write_recency_for_promote": 0, 2024-09-10T19:16:39.021 INFO:teuthology.orchestra.run.smithi183.stdout: "hit_set_grade_decay_rate": 0, 2024-09-10T19:16:39.022 INFO:teuthology.orchestra.run.smithi183.stdout: "hit_set_search_last_n": 0, 2024-09-10T19:16:39.022 INFO:teuthology.orchestra.run.smithi183.stdout: "grade_table": [], 2024-09-10T19:16:39.022 INFO:teuthology.orchestra.run.smithi183.stdout: "stripe_width": 0, 2024-09-10T19:16:39.022 INFO:teuthology.orchestra.run.smithi183.stdout: "expected_num_objects": 0, 2024-09-10T19:16:39.022 INFO:teuthology.orchestra.run.smithi183.stdout: "fast_read": false, 2024-09-10T19:16:39.022 INFO:teuthology.orchestra.run.smithi183.stdout: "options": {}, 2024-09-10T19:16:39.022 INFO:teuthology.orchestra.run.smithi183.stdout: "application_metadata": { 2024-09-10T19:16:39.022 INFO:teuthology.orchestra.run.smithi183.stdout: "rgw": {} 2024-09-10T19:16:39.022 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:16:39.022 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:16:39.022 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:16:39.022 INFO:teuthology.orchestra.run.smithi183.stdout: "pool": 6, 2024-09-10T19:16:39.022 INFO:teuthology.orchestra.run.smithi183.stdout: "pool_name": "default.rgw.meta", 2024-09-10T19:16:39.023 INFO:teuthology.orchestra.run.smithi183.stdout: "create_time": "2024-09-10T19:10:11.993712+0000", 2024-09-10T19:16:39.023 INFO:teuthology.orchestra.run.smithi183.stdout: "flags": 1, 2024-09-10T19:16:39.023 INFO:teuthology.orchestra.run.smithi183.stdout: "flags_names": "hashpspool", 2024-09-10T19:16:39.023 INFO:teuthology.orchestra.run.smithi183.stdout: "type": 1, 2024-09-10T19:16:39.023 INFO:teuthology.orchestra.run.smithi183.stdout: "size": 3, 2024-09-10T19:16:39.023 INFO:teuthology.orchestra.run.smithi183.stdout: "min_size": 2, 2024-09-10T19:16:39.023 INFO:teuthology.orchestra.run.smithi183.stdout: "crush_rule": 0, 2024-09-10T19:16:39.023 INFO:teuthology.orchestra.run.smithi183.stdout: "peering_crush_bucket_count": 0, 2024-09-10T19:16:39.023 INFO:teuthology.orchestra.run.smithi183.stdout: "peering_crush_bucket_target": 0, 2024-09-10T19:16:39.023 INFO:teuthology.orchestra.run.smithi183.stdout: "peering_crush_bucket_barrier": 0, 2024-09-10T19:16:39.023 INFO:teuthology.orchestra.run.smithi183.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-10T19:16:39.023 INFO:teuthology.orchestra.run.smithi183.stdout: "object_hash": 2, 2024-09-10T19:16:39.023 INFO:teuthology.orchestra.run.smithi183.stdout: "pg_autoscale_mode": "off", 2024-09-10T19:16:39.023 INFO:teuthology.orchestra.run.smithi183.stdout: "pg_num": 32, 2024-09-10T19:16:39.024 INFO:teuthology.orchestra.run.smithi183.stdout: "pg_placement_num": 32, 2024-09-10T19:16:39.024 INFO:teuthology.orchestra.run.smithi183.stdout: "pg_placement_num_target": 32, 2024-09-10T19:16:39.024 INFO:teuthology.orchestra.run.smithi183.stdout: "pg_num_target": 32, 2024-09-10T19:16:39.024 INFO:teuthology.orchestra.run.smithi183.stdout: "pg_num_pending": 32, 2024-09-10T19:16:39.024 INFO:teuthology.orchestra.run.smithi183.stdout: "last_pg_merge_meta": { 2024-09-10T19:16:39.024 INFO:teuthology.orchestra.run.smithi183.stdout: "source_pgid": "0.0", 2024-09-10T19:16:39.024 INFO:teuthology.orchestra.run.smithi183.stdout: "ready_epoch": 0, 2024-09-10T19:16:39.024 INFO:teuthology.orchestra.run.smithi183.stdout: "last_epoch_started": 0, 2024-09-10T19:16:39.024 INFO:teuthology.orchestra.run.smithi183.stdout: "last_epoch_clean": 0, 2024-09-10T19:16:39.024 INFO:teuthology.orchestra.run.smithi183.stdout: "source_version": "0'0", 2024-09-10T19:16:39.025 INFO:teuthology.orchestra.run.smithi183.stdout: "target_version": "0'0" 2024-09-10T19:16:39.025 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:16:39.025 INFO:teuthology.orchestra.run.smithi183.stdout: "last_change": "66", 2024-09-10T19:16:39.025 INFO:teuthology.orchestra.run.smithi183.stdout: "last_force_op_resend": "0", 2024-09-10T19:16:39.025 INFO:teuthology.orchestra.run.smithi183.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-10T19:16:39.025 INFO:teuthology.orchestra.run.smithi183.stdout: "last_force_op_resend_preluminous": "0", 2024-09-10T19:16:39.025 INFO:teuthology.orchestra.run.smithi183.stdout: "auid": 0, 2024-09-10T19:16:39.025 INFO:teuthology.orchestra.run.smithi183.stdout: "snap_mode": "selfmanaged", 2024-09-10T19:16:39.025 INFO:teuthology.orchestra.run.smithi183.stdout: "snap_seq": 0, 2024-09-10T19:16:39.025 INFO:teuthology.orchestra.run.smithi183.stdout: "snap_epoch": 0, 2024-09-10T19:16:39.025 INFO:teuthology.orchestra.run.smithi183.stdout: "pool_snaps": [], 2024-09-10T19:16:39.026 INFO:teuthology.orchestra.run.smithi183.stdout: "removed_snaps": "[]", 2024-09-10T19:16:39.026 INFO:teuthology.orchestra.run.smithi183.stdout: "quota_max_bytes": 0, 2024-09-10T19:16:39.026 INFO:teuthology.orchestra.run.smithi183.stdout: "quota_max_objects": 0, 2024-09-10T19:16:39.026 INFO:teuthology.orchestra.run.smithi183.stdout: "tiers": [], 2024-09-10T19:16:39.026 INFO:teuthology.orchestra.run.smithi183.stdout: "tier_of": -1, 2024-09-10T19:16:39.026 INFO:teuthology.orchestra.run.smithi183.stdout: "read_tier": -1, 2024-09-10T19:16:39.026 INFO:teuthology.orchestra.run.smithi183.stdout: "write_tier": -1, 2024-09-10T19:16:39.026 INFO:teuthology.orchestra.run.smithi183.stdout: "cache_mode": "none", 2024-09-10T19:16:39.026 INFO:teuthology.orchestra.run.smithi183.stdout: "target_max_bytes": 0, 2024-09-10T19:16:39.026 INFO:teuthology.orchestra.run.smithi183.stdout: "target_max_objects": 0, 2024-09-10T19:16:39.026 INFO:teuthology.orchestra.run.smithi183.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-10T19:16:39.026 INFO:teuthology.orchestra.run.smithi183.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-10T19:16:39.026 INFO:teuthology.orchestra.run.smithi183.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-10T19:16:39.027 INFO:teuthology.orchestra.run.smithi183.stdout: "cache_min_flush_age": 0, 2024-09-10T19:16:39.027 INFO:teuthology.orchestra.run.smithi183.stdout: "cache_min_evict_age": 0, 2024-09-10T19:16:39.027 INFO:teuthology.orchestra.run.smithi183.stdout: "erasure_code_profile": "", 2024-09-10T19:16:39.027 INFO:teuthology.orchestra.run.smithi183.stdout: "hit_set_params": { 2024-09-10T19:16:39.027 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "none" 2024-09-10T19:16:39.027 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:16:39.027 INFO:teuthology.orchestra.run.smithi183.stdout: "hit_set_period": 0, 2024-09-10T19:16:39.027 INFO:teuthology.orchestra.run.smithi183.stdout: "hit_set_count": 0, 2024-09-10T19:16:39.027 INFO:teuthology.orchestra.run.smithi183.stdout: "use_gmt_hitset": true, 2024-09-10T19:16:39.027 INFO:teuthology.orchestra.run.smithi183.stdout: "min_read_recency_for_promote": 0, 2024-09-10T19:16:39.027 INFO:teuthology.orchestra.run.smithi183.stdout: "min_write_recency_for_promote": 0, 2024-09-10T19:16:39.027 INFO:teuthology.orchestra.run.smithi183.stdout: "hit_set_grade_decay_rate": 0, 2024-09-10T19:16:39.028 INFO:teuthology.orchestra.run.smithi183.stdout: "hit_set_search_last_n": 0, 2024-09-10T19:16:39.028 INFO:teuthology.orchestra.run.smithi183.stdout: "grade_table": [], 2024-09-10T19:16:39.028 INFO:teuthology.orchestra.run.smithi183.stdout: "stripe_width": 0, 2024-09-10T19:16:39.028 INFO:teuthology.orchestra.run.smithi183.stdout: "expected_num_objects": 0, 2024-09-10T19:16:39.028 INFO:teuthology.orchestra.run.smithi183.stdout: "fast_read": false, 2024-09-10T19:16:39.028 INFO:teuthology.orchestra.run.smithi183.stdout: "options": { 2024-09-10T19:16:39.028 INFO:teuthology.orchestra.run.smithi183.stdout: "pg_autoscale_bias": 4 2024-09-10T19:16:39.028 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:16:39.028 INFO:teuthology.orchestra.run.smithi183.stdout: "application_metadata": { 2024-09-10T19:16:39.028 INFO:teuthology.orchestra.run.smithi183.stdout: "rgw": {} 2024-09-10T19:16:39.028 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:16:39.028 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:16:39.028 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:16:39.029 INFO:teuthology.orchestra.run.smithi183.stdout: "pool": 8, 2024-09-10T19:16:39.029 INFO:teuthology.orchestra.run.smithi183.stdout: "pool_name": "unique_pool_1", 2024-09-10T19:16:39.029 INFO:teuthology.orchestra.run.smithi183.stdout: "create_time": "2024-09-10T19:16:11.474472+0000", 2024-09-10T19:16:39.029 INFO:teuthology.orchestra.run.smithi183.stdout: "flags": 1, 2024-09-10T19:16:39.029 INFO:teuthology.orchestra.run.smithi183.stdout: "flags_names": "hashpspool", 2024-09-10T19:16:39.029 INFO:teuthology.orchestra.run.smithi183.stdout: "type": 1, 2024-09-10T19:16:39.029 INFO:teuthology.orchestra.run.smithi183.stdout: "size": 3, 2024-09-10T19:16:39.029 INFO:teuthology.orchestra.run.smithi183.stdout: "min_size": 2, 2024-09-10T19:16:39.029 INFO:teuthology.orchestra.run.smithi183.stdout: "crush_rule": 0, 2024-09-10T19:16:39.029 INFO:teuthology.orchestra.run.smithi183.stdout: "peering_crush_bucket_count": 0, 2024-09-10T19:16:39.029 INFO:teuthology.orchestra.run.smithi183.stdout: "peering_crush_bucket_target": 0, 2024-09-10T19:16:39.029 INFO:teuthology.orchestra.run.smithi183.stdout: "peering_crush_bucket_barrier": 0, 2024-09-10T19:16:39.029 INFO:teuthology.orchestra.run.smithi183.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-10T19:16:39.030 INFO:teuthology.orchestra.run.smithi183.stdout: "object_hash": 2, 2024-09-10T19:16:39.030 INFO:teuthology.orchestra.run.smithi183.stdout: "pg_autoscale_mode": "off", 2024-09-10T19:16:39.030 INFO:teuthology.orchestra.run.smithi183.stdout: "pg_num": 16, 2024-09-10T19:16:39.030 INFO:teuthology.orchestra.run.smithi183.stdout: "pg_placement_num": 16, 2024-09-10T19:16:39.030 INFO:teuthology.orchestra.run.smithi183.stdout: "pg_placement_num_target": 16, 2024-09-10T19:16:39.030 INFO:teuthology.orchestra.run.smithi183.stdout: "pg_num_target": 16, 2024-09-10T19:16:39.030 INFO:teuthology.orchestra.run.smithi183.stdout: "pg_num_pending": 16, 2024-09-10T19:16:39.030 INFO:teuthology.orchestra.run.smithi183.stdout: "last_pg_merge_meta": { 2024-09-10T19:16:39.030 INFO:teuthology.orchestra.run.smithi183.stdout: "source_pgid": "0.0", 2024-09-10T19:16:39.030 INFO:teuthology.orchestra.run.smithi183.stdout: "ready_epoch": 0, 2024-09-10T19:16:39.030 INFO:teuthology.orchestra.run.smithi183.stdout: "last_epoch_started": 0, 2024-09-10T19:16:39.030 INFO:teuthology.orchestra.run.smithi183.stdout: "last_epoch_clean": 0, 2024-09-10T19:16:39.030 INFO:teuthology.orchestra.run.smithi183.stdout: "source_version": "0'0", 2024-09-10T19:16:39.030 INFO:teuthology.orchestra.run.smithi183.stdout: "target_version": "0'0" 2024-09-10T19:16:39.031 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:16:39.031 INFO:teuthology.orchestra.run.smithi183.stdout: "last_change": "153", 2024-09-10T19:16:39.031 INFO:teuthology.orchestra.run.smithi183.stdout: "last_force_op_resend": "0", 2024-09-10T19:16:39.031 INFO:teuthology.orchestra.run.smithi183.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-10T19:16:39.031 INFO:teuthology.orchestra.run.smithi183.stdout: "last_force_op_resend_preluminous": "0", 2024-09-10T19:16:39.031 INFO:teuthology.orchestra.run.smithi183.stdout: "auid": 0, 2024-09-10T19:16:39.031 INFO:teuthology.orchestra.run.smithi183.stdout: "snap_mode": "selfmanaged", 2024-09-10T19:16:39.031 INFO:teuthology.orchestra.run.smithi183.stdout: "snap_seq": 0, 2024-09-10T19:16:39.031 INFO:teuthology.orchestra.run.smithi183.stdout: "snap_epoch": 0, 2024-09-10T19:16:39.031 INFO:teuthology.orchestra.run.smithi183.stdout: "pool_snaps": [], 2024-09-10T19:16:39.031 INFO:teuthology.orchestra.run.smithi183.stdout: "removed_snaps": "[]", 2024-09-10T19:16:39.032 INFO:teuthology.orchestra.run.smithi183.stdout: "quota_max_bytes": 0, 2024-09-10T19:16:39.032 INFO:teuthology.orchestra.run.smithi183.stdout: "quota_max_objects": 0, 2024-09-10T19:16:39.032 INFO:teuthology.orchestra.run.smithi183.stdout: "tiers": [], 2024-09-10T19:16:39.032 INFO:teuthology.orchestra.run.smithi183.stdout: "tier_of": -1, 2024-09-10T19:16:39.032 INFO:teuthology.orchestra.run.smithi183.stdout: "read_tier": -1, 2024-09-10T19:16:39.032 INFO:teuthology.orchestra.run.smithi183.stdout: "write_tier": -1, 2024-09-10T19:16:39.032 INFO:teuthology.orchestra.run.smithi183.stdout: "cache_mode": "none", 2024-09-10T19:16:39.032 INFO:teuthology.orchestra.run.smithi183.stdout: "target_max_bytes": 0, 2024-09-10T19:16:39.033 INFO:teuthology.orchestra.run.smithi183.stdout: "target_max_objects": 0, 2024-09-10T19:16:39.033 INFO:teuthology.orchestra.run.smithi183.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-10T19:16:39.033 INFO:teuthology.orchestra.run.smithi183.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-10T19:16:39.033 INFO:teuthology.orchestra.run.smithi183.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-10T19:16:39.033 INFO:teuthology.orchestra.run.smithi183.stdout: "cache_min_flush_age": 0, 2024-09-10T19:16:39.033 INFO:teuthology.orchestra.run.smithi183.stdout: "cache_min_evict_age": 0, 2024-09-10T19:16:39.034 INFO:teuthology.orchestra.run.smithi183.stdout: "erasure_code_profile": "", 2024-09-10T19:16:39.034 INFO:teuthology.orchestra.run.smithi183.stdout: "hit_set_params": { 2024-09-10T19:16:39.034 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "none" 2024-09-10T19:16:39.034 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:16:39.034 INFO:teuthology.orchestra.run.smithi183.stdout: "hit_set_period": 0, 2024-09-10T19:16:39.034 INFO:teuthology.orchestra.run.smithi183.stdout: "hit_set_count": 0, 2024-09-10T19:16:39.035 INFO:teuthology.orchestra.run.smithi183.stdout: "use_gmt_hitset": true, 2024-09-10T19:16:39.035 INFO:teuthology.orchestra.run.smithi183.stdout: "min_read_recency_for_promote": 0, 2024-09-10T19:16:39.035 INFO:teuthology.orchestra.run.smithi183.stdout: "min_write_recency_for_promote": 0, 2024-09-10T19:16:39.035 INFO:teuthology.orchestra.run.smithi183.stdout: "hit_set_grade_decay_rate": 0, 2024-09-10T19:16:39.035 INFO:teuthology.orchestra.run.smithi183.stdout: "hit_set_search_last_n": 0, 2024-09-10T19:16:39.035 INFO:teuthology.orchestra.run.smithi183.stdout: "grade_table": [], 2024-09-10T19:16:39.036 INFO:teuthology.orchestra.run.smithi183.stdout: "stripe_width": 0, 2024-09-10T19:16:39.036 INFO:teuthology.orchestra.run.smithi183.stdout: "expected_num_objects": 0, 2024-09-10T19:16:39.036 INFO:teuthology.orchestra.run.smithi183.stdout: "fast_read": false, 2024-09-10T19:16:39.036 INFO:teuthology.orchestra.run.smithi183.stdout: "options": {}, 2024-09-10T19:16:39.036 INFO:teuthology.orchestra.run.smithi183.stdout: "application_metadata": { 2024-09-10T19:16:39.036 INFO:teuthology.orchestra.run.smithi183.stdout: "rados": {} 2024-09-10T19:16:39.037 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:16:39.037 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:16:39.037 INFO:teuthology.orchestra.run.smithi183.stdout: ], 2024-09-10T19:16:39.037 INFO:teuthology.orchestra.run.smithi183.stdout: "osds": [ 2024-09-10T19:16:39.037 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:16:39.037 INFO:teuthology.orchestra.run.smithi183.stdout: "osd": 0, 2024-09-10T19:16:39.038 INFO:teuthology.orchestra.run.smithi183.stdout: "uuid": "9d323010-394a-49aa-ba9b-244d87faf36e", 2024-09-10T19:16:39.038 INFO:teuthology.orchestra.run.smithi183.stdout: "up": 1, 2024-09-10T19:16:39.038 INFO:teuthology.orchestra.run.smithi183.stdout: "in": 1, 2024-09-10T19:16:39.038 INFO:teuthology.orchestra.run.smithi183.stdout: "weight": 1, 2024-09-10T19:16:39.038 INFO:teuthology.orchestra.run.smithi183.stdout: "primary_affinity": 1, 2024-09-10T19:16:39.039 INFO:teuthology.orchestra.run.smithi183.stdout: "last_clean_begin": 0, 2024-09-10T19:16:39.039 INFO:teuthology.orchestra.run.smithi183.stdout: "last_clean_end": 0, 2024-09-10T19:16:39.039 INFO:teuthology.orchestra.run.smithi183.stdout: "up_from": 8, 2024-09-10T19:16:39.039 INFO:teuthology.orchestra.run.smithi183.stdout: "up_thru": 154, 2024-09-10T19:16:39.039 INFO:teuthology.orchestra.run.smithi183.stdout: "down_at": 0, 2024-09-10T19:16:39.039 INFO:teuthology.orchestra.run.smithi183.stdout: "lost_at": 0, 2024-09-10T19:16:39.040 INFO:teuthology.orchestra.run.smithi183.stdout: "public_addrs": { 2024-09-10T19:16:39.040 INFO:teuthology.orchestra.run.smithi183.stdout: "addrvec": [ 2024-09-10T19:16:39.040 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:16:39.040 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v2", 2024-09-10T19:16:39.040 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.183:6802", 2024-09-10T19:16:39.040 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 2813074952 2024-09-10T19:16:39.041 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:16:39.041 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:16:39.041 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v1", 2024-09-10T19:16:39.041 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.183:6803", 2024-09-10T19:16:39.041 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 2813074952 2024-09-10T19:16:39.042 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:16:39.042 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:16:39.042 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:16:39.042 INFO:teuthology.orchestra.run.smithi183.stdout: "cluster_addrs": { 2024-09-10T19:16:39.042 INFO:teuthology.orchestra.run.smithi183.stdout: "addrvec": [ 2024-09-10T19:16:39.042 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:16:39.043 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v2", 2024-09-10T19:16:39.043 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.183:6804", 2024-09-10T19:16:39.043 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 2813074952 2024-09-10T19:16:39.043 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:16:39.043 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:16:39.043 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v1", 2024-09-10T19:16:39.044 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.183:6805", 2024-09-10T19:16:39.044 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 2813074952 2024-09-10T19:16:39.044 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:16:39.044 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:16:39.044 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:16:39.044 INFO:teuthology.orchestra.run.smithi183.stdout: "heartbeat_back_addrs": { 2024-09-10T19:16:39.045 INFO:teuthology.orchestra.run.smithi183.stdout: "addrvec": [ 2024-09-10T19:16:39.045 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:16:39.045 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v2", 2024-09-10T19:16:39.045 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.183:6808", 2024-09-10T19:16:39.045 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 2813074952 2024-09-10T19:16:39.045 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:16:39.046 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:16:39.046 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v1", 2024-09-10T19:16:39.046 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.183:6809", 2024-09-10T19:16:39.046 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 2813074952 2024-09-10T19:16:39.046 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:16:39.047 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:16:39.047 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:16:39.047 INFO:teuthology.orchestra.run.smithi183.stdout: "heartbeat_front_addrs": { 2024-09-10T19:16:39.047 INFO:teuthology.orchestra.run.smithi183.stdout: "addrvec": [ 2024-09-10T19:16:39.047 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:16:39.047 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v2", 2024-09-10T19:16:39.047 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.183:6806", 2024-09-10T19:16:39.048 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 2813074952 2024-09-10T19:16:39.048 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:16:39.048 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:16:39.048 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v1", 2024-09-10T19:16:39.048 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.183:6807", 2024-09-10T19:16:39.048 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 2813074952 2024-09-10T19:16:39.048 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:16:39.049 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:16:39.049 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:16:39.049 INFO:teuthology.orchestra.run.smithi183.stdout: "public_addr": "172.21.15.183:6803/2813074952", 2024-09-10T19:16:39.049 INFO:teuthology.orchestra.run.smithi183.stdout: "cluster_addr": "172.21.15.183:6805/2813074952", 2024-09-10T19:16:39.049 INFO:teuthology.orchestra.run.smithi183.stdout: "heartbeat_back_addr": "172.21.15.183:6809/2813074952", 2024-09-10T19:16:39.049 INFO:teuthology.orchestra.run.smithi183.stdout: "heartbeat_front_addr": "172.21.15.183:6807/2813074952", 2024-09-10T19:16:39.050 INFO:teuthology.orchestra.run.smithi183.stdout: "state": [ 2024-09-10T19:16:39.050 INFO:teuthology.orchestra.run.smithi183.stdout: "exists", 2024-09-10T19:16:39.050 INFO:teuthology.orchestra.run.smithi183.stdout: "up" 2024-09-10T19:16:39.050 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:16:39.050 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:16:39.050 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:16:39.050 INFO:teuthology.orchestra.run.smithi183.stdout: "osd": 1, 2024-09-10T19:16:39.050 INFO:teuthology.orchestra.run.smithi183.stdout: "uuid": "71a6c5dc-5109-4284-8d20-187d8e7afc9e", 2024-09-10T19:16:39.050 INFO:teuthology.orchestra.run.smithi183.stdout: "up": 1, 2024-09-10T19:16:39.050 INFO:teuthology.orchestra.run.smithi183.stdout: "in": 0, 2024-09-10T19:16:39.050 INFO:teuthology.orchestra.run.smithi183.stdout: "weight": 0, 2024-09-10T19:16:39.051 INFO:teuthology.orchestra.run.smithi183.stdout: "primary_affinity": 1, 2024-09-10T19:16:39.051 INFO:teuthology.orchestra.run.smithi183.stdout: "last_clean_begin": 13, 2024-09-10T19:16:39.051 INFO:teuthology.orchestra.run.smithi183.stdout: "last_clean_end": 98, 2024-09-10T19:16:39.051 INFO:teuthology.orchestra.run.smithi183.stdout: "up_from": 138, 2024-09-10T19:16:39.051 INFO:teuthology.orchestra.run.smithi183.stdout: "up_thru": 86, 2024-09-10T19:16:39.051 INFO:teuthology.orchestra.run.smithi183.stdout: "down_at": 99, 2024-09-10T19:16:39.051 INFO:teuthology.orchestra.run.smithi183.stdout: "lost_at": 0, 2024-09-10T19:16:39.051 INFO:teuthology.orchestra.run.smithi183.stdout: "public_addrs": { 2024-09-10T19:16:39.051 INFO:teuthology.orchestra.run.smithi183.stdout: "addrvec": [ 2024-09-10T19:16:39.051 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:16:39.051 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v2", 2024-09-10T19:16:39.052 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.183:6810", 2024-09-10T19:16:39.052 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 219854798 2024-09-10T19:16:39.052 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:16:39.052 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:16:39.052 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v1", 2024-09-10T19:16:39.052 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.183:6811", 2024-09-10T19:16:39.052 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 219854798 2024-09-10T19:16:39.052 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:16:39.052 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:16:39.052 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:16:39.052 INFO:teuthology.orchestra.run.smithi183.stdout: "cluster_addrs": { 2024-09-10T19:16:39.052 INFO:teuthology.orchestra.run.smithi183.stdout: "addrvec": [ 2024-09-10T19:16:39.053 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:16:39.053 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v2", 2024-09-10T19:16:39.053 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.183:6812", 2024-09-10T19:16:39.053 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 219854798 2024-09-10T19:16:39.053 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:16:39.053 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:16:39.053 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v1", 2024-09-10T19:16:39.053 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.183:6813", 2024-09-10T19:16:39.053 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 219854798 2024-09-10T19:16:39.053 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:16:39.053 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:16:39.053 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:16:39.053 INFO:teuthology.orchestra.run.smithi183.stdout: "heartbeat_back_addrs": { 2024-09-10T19:16:39.054 INFO:teuthology.orchestra.run.smithi183.stdout: "addrvec": [ 2024-09-10T19:16:39.054 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:16:39.054 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v2", 2024-09-10T19:16:39.054 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.183:6816", 2024-09-10T19:16:39.054 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 219854798 2024-09-10T19:16:39.054 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:16:39.054 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:16:39.054 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v1", 2024-09-10T19:16:39.054 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.183:6817", 2024-09-10T19:16:39.054 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 219854798 2024-09-10T19:16:39.054 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:16:39.054 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:16:39.054 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:16:39.054 INFO:teuthology.orchestra.run.smithi183.stdout: "heartbeat_front_addrs": { 2024-09-10T19:16:39.055 INFO:teuthology.orchestra.run.smithi183.stdout: "addrvec": [ 2024-09-10T19:16:39.055 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:16:39.055 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v2", 2024-09-10T19:16:39.055 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.183:6814", 2024-09-10T19:16:39.055 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 219854798 2024-09-10T19:16:39.055 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:16:39.055 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:16:39.055 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v1", 2024-09-10T19:16:39.055 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.183:6815", 2024-09-10T19:16:39.055 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 219854798 2024-09-10T19:16:39.055 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:16:39.055 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:16:39.055 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:16:39.056 INFO:teuthology.orchestra.run.smithi183.stdout: "public_addr": "172.21.15.183:6811/219854798", 2024-09-10T19:16:39.056 INFO:teuthology.orchestra.run.smithi183.stdout: "cluster_addr": "172.21.15.183:6813/219854798", 2024-09-10T19:16:39.056 INFO:teuthology.orchestra.run.smithi183.stdout: "heartbeat_back_addr": "172.21.15.183:6817/219854798", 2024-09-10T19:16:39.056 INFO:teuthology.orchestra.run.smithi183.stdout: "heartbeat_front_addr": "172.21.15.183:6815/219854798", 2024-09-10T19:16:39.056 INFO:teuthology.orchestra.run.smithi183.stdout: "state": [ 2024-09-10T19:16:39.056 INFO:teuthology.orchestra.run.smithi183.stdout: "exists", 2024-09-10T19:16:39.056 INFO:teuthology.orchestra.run.smithi183.stdout: "up" 2024-09-10T19:16:39.057 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:16:39.057 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:16:39.057 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:16:39.057 INFO:teuthology.orchestra.run.smithi183.stdout: "osd": 2, 2024-09-10T19:16:39.057 INFO:teuthology.orchestra.run.smithi183.stdout: "uuid": "fc310d5b-b89b-4655-9989-7e10ba356509", 2024-09-10T19:16:39.057 INFO:teuthology.orchestra.run.smithi183.stdout: "up": 1, 2024-09-10T19:16:39.057 INFO:teuthology.orchestra.run.smithi183.stdout: "in": 0, 2024-09-10T19:16:39.057 INFO:teuthology.orchestra.run.smithi183.stdout: "weight": 0, 2024-09-10T19:16:39.057 INFO:teuthology.orchestra.run.smithi183.stdout: "primary_affinity": 1, 2024-09-10T19:16:39.057 INFO:teuthology.orchestra.run.smithi183.stdout: "last_clean_begin": 0, 2024-09-10T19:16:39.057 INFO:teuthology.orchestra.run.smithi183.stdout: "last_clean_end": 0, 2024-09-10T19:16:39.058 INFO:teuthology.orchestra.run.smithi183.stdout: "up_from": 18, 2024-09-10T19:16:39.058 INFO:teuthology.orchestra.run.smithi183.stdout: "up_thru": 148, 2024-09-10T19:16:39.058 INFO:teuthology.orchestra.run.smithi183.stdout: "down_at": 0, 2024-09-10T19:16:39.058 INFO:teuthology.orchestra.run.smithi183.stdout: "lost_at": 0, 2024-09-10T19:16:39.058 INFO:teuthology.orchestra.run.smithi183.stdout: "public_addrs": { 2024-09-10T19:16:39.058 INFO:teuthology.orchestra.run.smithi183.stdout: "addrvec": [ 2024-09-10T19:16:39.059 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:16:39.059 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v2", 2024-09-10T19:16:39.059 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.183:6818", 2024-09-10T19:16:39.059 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 3730013786 2024-09-10T19:16:39.059 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:16:39.059 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:16:39.059 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v1", 2024-09-10T19:16:39.059 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.183:6819", 2024-09-10T19:16:39.059 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 3730013786 2024-09-10T19:16:39.059 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:16:39.059 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:16:39.059 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:16:39.060 INFO:teuthology.orchestra.run.smithi183.stdout: "cluster_addrs": { 2024-09-10T19:16:39.060 INFO:teuthology.orchestra.run.smithi183.stdout: "addrvec": [ 2024-09-10T19:16:39.060 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:16:39.060 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v2", 2024-09-10T19:16:39.060 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.183:6820", 2024-09-10T19:16:39.060 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 3730013786 2024-09-10T19:16:39.060 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:16:39.060 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:16:39.060 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v1", 2024-09-10T19:16:39.060 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.183:6821", 2024-09-10T19:16:39.060 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 3730013786 2024-09-10T19:16:39.060 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:16:39.060 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:16:39.060 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:16:39.060 INFO:teuthology.orchestra.run.smithi183.stdout: "heartbeat_back_addrs": { 2024-09-10T19:16:39.061 INFO:teuthology.orchestra.run.smithi183.stdout: "addrvec": [ 2024-09-10T19:16:39.061 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:16:39.061 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v2", 2024-09-10T19:16:39.061 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.183:6824", 2024-09-10T19:16:39.061 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 3730013786 2024-09-10T19:16:39.061 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:16:39.061 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:16:39.061 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v1", 2024-09-10T19:16:39.061 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.183:6825", 2024-09-10T19:16:39.061 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 3730013786 2024-09-10T19:16:39.061 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:16:39.061 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:16:39.061 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:16:39.062 INFO:teuthology.orchestra.run.smithi183.stdout: "heartbeat_front_addrs": { 2024-09-10T19:16:39.062 INFO:teuthology.orchestra.run.smithi183.stdout: "addrvec": [ 2024-09-10T19:16:39.062 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:16:39.062 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v2", 2024-09-10T19:16:39.062 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.183:6822", 2024-09-10T19:16:39.062 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 3730013786 2024-09-10T19:16:39.062 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:16:39.062 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:16:39.062 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v1", 2024-09-10T19:16:39.062 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.183:6823", 2024-09-10T19:16:39.062 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 3730013786 2024-09-10T19:16:39.062 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:16:39.062 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:16:39.063 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:16:39.063 INFO:teuthology.orchestra.run.smithi183.stdout: "public_addr": "172.21.15.183:6819/3730013786", 2024-09-10T19:16:39.063 INFO:teuthology.orchestra.run.smithi183.stdout: "cluster_addr": "172.21.15.183:6821/3730013786", 2024-09-10T19:16:39.063 INFO:teuthology.orchestra.run.smithi183.stdout: "heartbeat_back_addr": "172.21.15.183:6825/3730013786", 2024-09-10T19:16:39.063 INFO:teuthology.orchestra.run.smithi183.stdout: "heartbeat_front_addr": "172.21.15.183:6823/3730013786", 2024-09-10T19:16:39.063 INFO:teuthology.orchestra.run.smithi183.stdout: "state": [ 2024-09-10T19:16:39.063 INFO:teuthology.orchestra.run.smithi183.stdout: "exists", 2024-09-10T19:16:39.063 INFO:teuthology.orchestra.run.smithi183.stdout: "up" 2024-09-10T19:16:39.063 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:16:39.063 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:16:39.063 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:16:39.063 INFO:teuthology.orchestra.run.smithi183.stdout: "osd": 3, 2024-09-10T19:16:39.064 INFO:teuthology.orchestra.run.smithi183.stdout: "uuid": "7fd07a95-ad22-43f5-a968-539552283e89", 2024-09-10T19:16:39.064 INFO:teuthology.orchestra.run.smithi183.stdout: "up": 1, 2024-09-10T19:16:39.064 INFO:teuthology.orchestra.run.smithi183.stdout: "in": 1, 2024-09-10T19:16:39.064 INFO:teuthology.orchestra.run.smithi183.stdout: "weight": 1, 2024-09-10T19:16:39.064 INFO:teuthology.orchestra.run.smithi183.stdout: "primary_affinity": 1, 2024-09-10T19:16:39.064 INFO:teuthology.orchestra.run.smithi183.stdout: "last_clean_begin": 0, 2024-09-10T19:16:39.064 INFO:teuthology.orchestra.run.smithi183.stdout: "last_clean_end": 0, 2024-09-10T19:16:39.064 INFO:teuthology.orchestra.run.smithi183.stdout: "up_from": 26, 2024-09-10T19:16:39.064 INFO:teuthology.orchestra.run.smithi183.stdout: "up_thru": 154, 2024-09-10T19:16:39.064 INFO:teuthology.orchestra.run.smithi183.stdout: "down_at": 0, 2024-09-10T19:16:39.064 INFO:teuthology.orchestra.run.smithi183.stdout: "lost_at": 0, 2024-09-10T19:16:39.064 INFO:teuthology.orchestra.run.smithi183.stdout: "public_addrs": { 2024-09-10T19:16:39.065 INFO:teuthology.orchestra.run.smithi183.stdout: "addrvec": [ 2024-09-10T19:16:39.065 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:16:39.065 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v2", 2024-09-10T19:16:39.065 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.183:6826", 2024-09-10T19:16:39.065 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 3522836984 2024-09-10T19:16:39.065 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:16:39.065 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:16:39.065 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v1", 2024-09-10T19:16:39.065 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.183:6827", 2024-09-10T19:16:39.065 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 3522836984 2024-09-10T19:16:39.065 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:16:39.066 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:16:39.066 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:16:39.066 INFO:teuthology.orchestra.run.smithi183.stdout: "cluster_addrs": { 2024-09-10T19:16:39.066 INFO:teuthology.orchestra.run.smithi183.stdout: "addrvec": [ 2024-09-10T19:16:39.066 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:16:39.066 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v2", 2024-09-10T19:16:39.066 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.183:6828", 2024-09-10T19:16:39.066 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 3522836984 2024-09-10T19:16:39.066 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:16:39.066 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:16:39.066 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v1", 2024-09-10T19:16:39.066 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.183:6829", 2024-09-10T19:16:39.066 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 3522836984 2024-09-10T19:16:39.067 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:16:39.067 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:16:39.067 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:16:39.067 INFO:teuthology.orchestra.run.smithi183.stdout: "heartbeat_back_addrs": { 2024-09-10T19:16:39.067 INFO:teuthology.orchestra.run.smithi183.stdout: "addrvec": [ 2024-09-10T19:16:39.067 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:16:39.067 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v2", 2024-09-10T19:16:39.067 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.183:6832", 2024-09-10T19:16:39.067 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 3522836984 2024-09-10T19:16:39.067 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:16:39.067 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:16:39.067 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v1", 2024-09-10T19:16:39.067 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.183:6833", 2024-09-10T19:16:39.068 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 3522836984 2024-09-10T19:16:39.068 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:16:39.068 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:16:39.068 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:16:39.068 INFO:teuthology.orchestra.run.smithi183.stdout: "heartbeat_front_addrs": { 2024-09-10T19:16:39.068 INFO:teuthology.orchestra.run.smithi183.stdout: "addrvec": [ 2024-09-10T19:16:39.068 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:16:39.068 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v2", 2024-09-10T19:16:39.068 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.183:6830", 2024-09-10T19:16:39.068 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 3522836984 2024-09-10T19:16:39.068 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:16:39.068 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:16:39.069 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v1", 2024-09-10T19:16:39.069 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.183:6831", 2024-09-10T19:16:39.069 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 3522836984 2024-09-10T19:16:39.069 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:16:39.069 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:16:39.069 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:16:39.069 INFO:teuthology.orchestra.run.smithi183.stdout: "public_addr": "172.21.15.183:6827/3522836984", 2024-09-10T19:16:39.069 INFO:teuthology.orchestra.run.smithi183.stdout: "cluster_addr": "172.21.15.183:6829/3522836984", 2024-09-10T19:16:39.069 INFO:teuthology.orchestra.run.smithi183.stdout: "heartbeat_back_addr": "172.21.15.183:6833/3522836984", 2024-09-10T19:16:39.069 INFO:teuthology.orchestra.run.smithi183.stdout: "heartbeat_front_addr": "172.21.15.183:6831/3522836984", 2024-09-10T19:16:39.069 INFO:teuthology.orchestra.run.smithi183.stdout: "state": [ 2024-09-10T19:16:39.069 INFO:teuthology.orchestra.run.smithi183.stdout: "exists", 2024-09-10T19:16:39.069 INFO:teuthology.orchestra.run.smithi183.stdout: "up" 2024-09-10T19:16:39.069 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:16:39.070 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:16:39.070 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:16:39.070 INFO:teuthology.orchestra.run.smithi183.stdout: "osd": 4, 2024-09-10T19:16:39.070 INFO:teuthology.orchestra.run.smithi183.stdout: "uuid": "631083fb-e503-44da-9b6b-154e88539ee7", 2024-09-10T19:16:39.070 INFO:teuthology.orchestra.run.smithi183.stdout: "up": 1, 2024-09-10T19:16:39.070 INFO:teuthology.orchestra.run.smithi183.stdout: "in": 1, 2024-09-10T19:16:39.070 INFO:teuthology.orchestra.run.smithi183.stdout: "weight": 1, 2024-09-10T19:16:39.070 INFO:teuthology.orchestra.run.smithi183.stdout: "primary_affinity": 1, 2024-09-10T19:16:39.070 INFO:teuthology.orchestra.run.smithi183.stdout: "last_clean_begin": 0, 2024-09-10T19:16:39.070 INFO:teuthology.orchestra.run.smithi183.stdout: "last_clean_end": 0, 2024-09-10T19:16:39.070 INFO:teuthology.orchestra.run.smithi183.stdout: "up_from": 32, 2024-09-10T19:16:39.070 INFO:teuthology.orchestra.run.smithi183.stdout: "up_thru": 154, 2024-09-10T19:16:39.070 INFO:teuthology.orchestra.run.smithi183.stdout: "down_at": 0, 2024-09-10T19:16:39.071 INFO:teuthology.orchestra.run.smithi183.stdout: "lost_at": 0, 2024-09-10T19:16:39.071 INFO:teuthology.orchestra.run.smithi183.stdout: "public_addrs": { 2024-09-10T19:16:39.071 INFO:teuthology.orchestra.run.smithi183.stdout: "addrvec": [ 2024-09-10T19:16:39.071 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:16:39.071 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v2", 2024-09-10T19:16:39.071 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.191:6800", 2024-09-10T19:16:39.071 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 2674828539 2024-09-10T19:16:39.071 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:16:39.071 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:16:39.071 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v1", 2024-09-10T19:16:39.071 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.191:6801", 2024-09-10T19:16:39.071 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 2674828539 2024-09-10T19:16:39.071 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:16:39.071 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:16:39.072 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:16:39.072 INFO:teuthology.orchestra.run.smithi183.stdout: "cluster_addrs": { 2024-09-10T19:16:39.072 INFO:teuthology.orchestra.run.smithi183.stdout: "addrvec": [ 2024-09-10T19:16:39.072 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:16:39.072 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v2", 2024-09-10T19:16:39.072 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.191:6802", 2024-09-10T19:16:39.072 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 2674828539 2024-09-10T19:16:39.073 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:16:39.073 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:16:39.073 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v1", 2024-09-10T19:16:39.073 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.191:6803", 2024-09-10T19:16:39.073 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 2674828539 2024-09-10T19:16:39.073 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:16:39.073 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:16:39.073 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:16:39.073 INFO:teuthology.orchestra.run.smithi183.stdout: "heartbeat_back_addrs": { 2024-09-10T19:16:39.073 INFO:teuthology.orchestra.run.smithi183.stdout: "addrvec": [ 2024-09-10T19:16:39.073 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:16:39.073 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v2", 2024-09-10T19:16:39.074 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.191:6806", 2024-09-10T19:16:39.074 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 2674828539 2024-09-10T19:16:39.074 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:16:39.074 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:16:39.074 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v1", 2024-09-10T19:16:39.074 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.191:6807", 2024-09-10T19:16:39.074 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 2674828539 2024-09-10T19:16:39.074 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:16:39.074 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:16:39.074 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:16:39.074 INFO:teuthology.orchestra.run.smithi183.stdout: "heartbeat_front_addrs": { 2024-09-10T19:16:39.074 INFO:teuthology.orchestra.run.smithi183.stdout: "addrvec": [ 2024-09-10T19:16:39.074 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:16:39.074 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v2", 2024-09-10T19:16:39.074 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.191:6804", 2024-09-10T19:16:39.075 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 2674828539 2024-09-10T19:16:39.075 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:16:39.075 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:16:39.075 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v1", 2024-09-10T19:16:39.075 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.191:6805", 2024-09-10T19:16:39.075 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 2674828539 2024-09-10T19:16:39.075 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:16:39.075 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:16:39.075 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:16:39.075 INFO:teuthology.orchestra.run.smithi183.stdout: "public_addr": "172.21.15.191:6801/2674828539", 2024-09-10T19:16:39.075 INFO:teuthology.orchestra.run.smithi183.stdout: "cluster_addr": "172.21.15.191:6803/2674828539", 2024-09-10T19:16:39.075 INFO:teuthology.orchestra.run.smithi183.stdout: "heartbeat_back_addr": "172.21.15.191:6807/2674828539", 2024-09-10T19:16:39.075 INFO:teuthology.orchestra.run.smithi183.stdout: "heartbeat_front_addr": "172.21.15.191:6805/2674828539", 2024-09-10T19:16:39.075 INFO:teuthology.orchestra.run.smithi183.stdout: "state": [ 2024-09-10T19:16:39.076 INFO:teuthology.orchestra.run.smithi183.stdout: "exists", 2024-09-10T19:16:39.076 INFO:teuthology.orchestra.run.smithi183.stdout: "up" 2024-09-10T19:16:39.076 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:16:39.076 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:16:39.076 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:16:39.076 INFO:teuthology.orchestra.run.smithi183.stdout: "osd": 5, 2024-09-10T19:16:39.076 INFO:teuthology.orchestra.run.smithi183.stdout: "uuid": "6b963695-3eca-4e6e-9207-05b8010ebff9", 2024-09-10T19:16:39.076 INFO:teuthology.orchestra.run.smithi183.stdout: "up": 1, 2024-09-10T19:16:39.076 INFO:teuthology.orchestra.run.smithi183.stdout: "in": 1, 2024-09-10T19:16:39.076 INFO:teuthology.orchestra.run.smithi183.stdout: "weight": 1, 2024-09-10T19:16:39.076 INFO:teuthology.orchestra.run.smithi183.stdout: "primary_affinity": 1, 2024-09-10T19:16:39.076 INFO:teuthology.orchestra.run.smithi183.stdout: "last_clean_begin": 0, 2024-09-10T19:16:39.077 INFO:teuthology.orchestra.run.smithi183.stdout: "last_clean_end": 0, 2024-09-10T19:16:39.077 INFO:teuthology.orchestra.run.smithi183.stdout: "up_from": 39, 2024-09-10T19:16:39.077 INFO:teuthology.orchestra.run.smithi183.stdout: "up_thru": 154, 2024-09-10T19:16:39.077 INFO:teuthology.orchestra.run.smithi183.stdout: "down_at": 0, 2024-09-10T19:16:39.077 INFO:teuthology.orchestra.run.smithi183.stdout: "lost_at": 0, 2024-09-10T19:16:39.077 INFO:teuthology.orchestra.run.smithi183.stdout: "public_addrs": { 2024-09-10T19:16:39.077 INFO:teuthology.orchestra.run.smithi183.stdout: "addrvec": [ 2024-09-10T19:16:39.077 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:16:39.077 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v2", 2024-09-10T19:16:39.077 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.191:6808", 2024-09-10T19:16:39.077 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 681733378 2024-09-10T19:16:39.077 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:16:39.077 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:16:39.077 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v1", 2024-09-10T19:16:39.078 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.191:6809", 2024-09-10T19:16:39.078 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 681733378 2024-09-10T19:16:39.078 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:16:39.078 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:16:39.078 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:16:39.078 INFO:teuthology.orchestra.run.smithi183.stdout: "cluster_addrs": { 2024-09-10T19:16:39.078 INFO:teuthology.orchestra.run.smithi183.stdout: "addrvec": [ 2024-09-10T19:16:39.078 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:16:39.078 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v2", 2024-09-10T19:16:39.078 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.191:6810", 2024-09-10T19:16:39.078 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 681733378 2024-09-10T19:16:39.078 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:16:39.078 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:16:39.078 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v1", 2024-09-10T19:16:39.079 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.191:6811", 2024-09-10T19:16:39.079 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 681733378 2024-09-10T19:16:39.079 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:16:39.079 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:16:39.079 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:16:39.079 INFO:teuthology.orchestra.run.smithi183.stdout: "heartbeat_back_addrs": { 2024-09-10T19:16:39.079 INFO:teuthology.orchestra.run.smithi183.stdout: "addrvec": [ 2024-09-10T19:16:39.079 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:16:39.079 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v2", 2024-09-10T19:16:39.079 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.191:6814", 2024-09-10T19:16:39.079 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 681733378 2024-09-10T19:16:39.079 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:16:39.079 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:16:39.079 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v1", 2024-09-10T19:16:39.079 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.191:6815", 2024-09-10T19:16:39.080 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 681733378 2024-09-10T19:16:39.080 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:16:39.080 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:16:39.080 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:16:39.080 INFO:teuthology.orchestra.run.smithi183.stdout: "heartbeat_front_addrs": { 2024-09-10T19:16:39.080 INFO:teuthology.orchestra.run.smithi183.stdout: "addrvec": [ 2024-09-10T19:16:39.080 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:16:39.080 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v2", 2024-09-10T19:16:39.080 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.191:6812", 2024-09-10T19:16:39.080 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 681733378 2024-09-10T19:16:39.080 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:16:39.080 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:16:39.081 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v1", 2024-09-10T19:16:39.081 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.191:6813", 2024-09-10T19:16:39.081 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 681733378 2024-09-10T19:16:39.081 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:16:39.081 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:16:39.081 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:16:39.081 INFO:teuthology.orchestra.run.smithi183.stdout: "public_addr": "172.21.15.191:6809/681733378", 2024-09-10T19:16:39.081 INFO:teuthology.orchestra.run.smithi183.stdout: "cluster_addr": "172.21.15.191:6811/681733378", 2024-09-10T19:16:39.081 INFO:teuthology.orchestra.run.smithi183.stdout: "heartbeat_back_addr": "172.21.15.191:6815/681733378", 2024-09-10T19:16:39.081 INFO:teuthology.orchestra.run.smithi183.stdout: "heartbeat_front_addr": "172.21.15.191:6813/681733378", 2024-09-10T19:16:39.081 INFO:teuthology.orchestra.run.smithi183.stdout: "state": [ 2024-09-10T19:16:39.081 INFO:teuthology.orchestra.run.smithi183.stdout: "exists", 2024-09-10T19:16:39.081 INFO:teuthology.orchestra.run.smithi183.stdout: "up" 2024-09-10T19:16:39.082 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:16:39.082 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:16:39.082 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:16:39.082 INFO:teuthology.orchestra.run.smithi183.stdout: "osd": 6, 2024-09-10T19:16:39.082 INFO:teuthology.orchestra.run.smithi183.stdout: "uuid": "2dc9754f-6aef-4e19-8605-2f1591921811", 2024-09-10T19:16:39.082 INFO:teuthology.orchestra.run.smithi183.stdout: "up": 1, 2024-09-10T19:16:39.082 INFO:teuthology.orchestra.run.smithi183.stdout: "in": 1, 2024-09-10T19:16:39.082 INFO:teuthology.orchestra.run.smithi183.stdout: "weight": 1, 2024-09-10T19:16:39.082 INFO:teuthology.orchestra.run.smithi183.stdout: "primary_affinity": 1, 2024-09-10T19:16:39.083 INFO:teuthology.orchestra.run.smithi183.stdout: "last_clean_begin": 0, 2024-09-10T19:16:39.083 INFO:teuthology.orchestra.run.smithi183.stdout: "last_clean_end": 0, 2024-09-10T19:16:39.083 INFO:teuthology.orchestra.run.smithi183.stdout: "up_from": 47, 2024-09-10T19:16:39.083 INFO:teuthology.orchestra.run.smithi183.stdout: "up_thru": 154, 2024-09-10T19:16:39.083 INFO:teuthology.orchestra.run.smithi183.stdout: "down_at": 0, 2024-09-10T19:16:39.083 INFO:teuthology.orchestra.run.smithi183.stdout: "lost_at": 0, 2024-09-10T19:16:39.083 INFO:teuthology.orchestra.run.smithi183.stdout: "public_addrs": { 2024-09-10T19:16:39.084 INFO:teuthology.orchestra.run.smithi183.stdout: "addrvec": [ 2024-09-10T19:16:39.084 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:16:39.084 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v2", 2024-09-10T19:16:39.084 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.191:6816", 2024-09-10T19:16:39.084 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 2825813477 2024-09-10T19:16:39.085 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:16:39.085 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:16:39.085 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v1", 2024-09-10T19:16:39.085 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.191:6817", 2024-09-10T19:16:39.085 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 2825813477 2024-09-10T19:16:39.085 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:16:39.086 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:16:39.086 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:16:39.086 INFO:teuthology.orchestra.run.smithi183.stdout: "cluster_addrs": { 2024-09-10T19:16:39.086 INFO:teuthology.orchestra.run.smithi183.stdout: "addrvec": [ 2024-09-10T19:16:39.086 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:16:39.086 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v2", 2024-09-10T19:16:39.086 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.191:6818", 2024-09-10T19:16:39.086 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 2825813477 2024-09-10T19:16:39.086 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:16:39.086 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:16:39.086 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v1", 2024-09-10T19:16:39.086 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.191:6819", 2024-09-10T19:16:39.086 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 2825813477 2024-09-10T19:16:39.086 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:16:39.086 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:16:39.087 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:16:39.087 INFO:teuthology.orchestra.run.smithi183.stdout: "heartbeat_back_addrs": { 2024-09-10T19:16:39.087 INFO:teuthology.orchestra.run.smithi183.stdout: "addrvec": [ 2024-09-10T19:16:39.087 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:16:39.087 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v2", 2024-09-10T19:16:39.087 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.191:6822", 2024-09-10T19:16:39.087 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 2825813477 2024-09-10T19:16:39.087 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:16:39.087 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:16:39.087 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v1", 2024-09-10T19:16:39.087 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.191:6823", 2024-09-10T19:16:39.087 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 2825813477 2024-09-10T19:16:39.087 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:16:39.087 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:16:39.087 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:16:39.088 INFO:teuthology.orchestra.run.smithi183.stdout: "heartbeat_front_addrs": { 2024-09-10T19:16:39.088 INFO:teuthology.orchestra.run.smithi183.stdout: "addrvec": [ 2024-09-10T19:16:39.088 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:16:39.088 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v2", 2024-09-10T19:16:39.088 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.191:6820", 2024-09-10T19:16:39.088 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 2825813477 2024-09-10T19:16:39.088 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:16:39.088 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:16:39.088 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v1", 2024-09-10T19:16:39.088 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.191:6821", 2024-09-10T19:16:39.088 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 2825813477 2024-09-10T19:16:39.088 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:16:39.089 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:16:39.089 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:16:39.089 INFO:teuthology.orchestra.run.smithi183.stdout: "public_addr": "172.21.15.191:6817/2825813477", 2024-09-10T19:16:39.089 INFO:teuthology.orchestra.run.smithi183.stdout: "cluster_addr": "172.21.15.191:6819/2825813477", 2024-09-10T19:16:39.089 INFO:teuthology.orchestra.run.smithi183.stdout: "heartbeat_back_addr": "172.21.15.191:6823/2825813477", 2024-09-10T19:16:39.089 INFO:teuthology.orchestra.run.smithi183.stdout: "heartbeat_front_addr": "172.21.15.191:6821/2825813477", 2024-09-10T19:16:39.089 INFO:teuthology.orchestra.run.smithi183.stdout: "state": [ 2024-09-10T19:16:39.089 INFO:teuthology.orchestra.run.smithi183.stdout: "exists", 2024-09-10T19:16:39.089 INFO:teuthology.orchestra.run.smithi183.stdout: "up" 2024-09-10T19:16:39.089 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:16:39.089 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:16:39.089 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:16:39.089 INFO:teuthology.orchestra.run.smithi183.stdout: "osd": 7, 2024-09-10T19:16:39.090 INFO:teuthology.orchestra.run.smithi183.stdout: "uuid": "1a9a299e-d5ba-4b9a-85d5-7f6a5bc754e1", 2024-09-10T19:16:39.090 INFO:teuthology.orchestra.run.smithi183.stdout: "up": 1, 2024-09-10T19:16:39.090 INFO:teuthology.orchestra.run.smithi183.stdout: "in": 1, 2024-09-10T19:16:39.090 INFO:teuthology.orchestra.run.smithi183.stdout: "weight": 1, 2024-09-10T19:16:39.090 INFO:teuthology.orchestra.run.smithi183.stdout: "primary_affinity": 1, 2024-09-10T19:16:39.090 INFO:teuthology.orchestra.run.smithi183.stdout: "last_clean_begin": 0, 2024-09-10T19:16:39.090 INFO:teuthology.orchestra.run.smithi183.stdout: "last_clean_end": 0, 2024-09-10T19:16:39.090 INFO:teuthology.orchestra.run.smithi183.stdout: "up_from": 54, 2024-09-10T19:16:39.090 INFO:teuthology.orchestra.run.smithi183.stdout: "up_thru": 154, 2024-09-10T19:16:39.090 INFO:teuthology.orchestra.run.smithi183.stdout: "down_at": 0, 2024-09-10T19:16:39.090 INFO:teuthology.orchestra.run.smithi183.stdout: "lost_at": 0, 2024-09-10T19:16:39.090 INFO:teuthology.orchestra.run.smithi183.stdout: "public_addrs": { 2024-09-10T19:16:39.090 INFO:teuthology.orchestra.run.smithi183.stdout: "addrvec": [ 2024-09-10T19:16:39.091 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:16:39.091 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v2", 2024-09-10T19:16:39.091 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.191:6824", 2024-09-10T19:16:39.091 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 1422610021 2024-09-10T19:16:39.091 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:16:39.091 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:16:39.091 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v1", 2024-09-10T19:16:39.091 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.191:6825", 2024-09-10T19:16:39.091 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 1422610021 2024-09-10T19:16:39.091 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:16:39.091 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:16:39.091 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:16:39.091 INFO:teuthology.orchestra.run.smithi183.stdout: "cluster_addrs": { 2024-09-10T19:16:39.091 INFO:teuthology.orchestra.run.smithi183.stdout: "addrvec": [ 2024-09-10T19:16:39.092 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:16:39.092 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v2", 2024-09-10T19:16:39.092 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.191:6826", 2024-09-10T19:16:39.092 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 1422610021 2024-09-10T19:16:39.092 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:16:39.092 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:16:39.092 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v1", 2024-09-10T19:16:39.092 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.191:6827", 2024-09-10T19:16:39.092 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 1422610021 2024-09-10T19:16:39.092 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:16:39.092 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:16:39.092 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:16:39.092 INFO:teuthology.orchestra.run.smithi183.stdout: "heartbeat_back_addrs": { 2024-09-10T19:16:39.093 INFO:teuthology.orchestra.run.smithi183.stdout: "addrvec": [ 2024-09-10T19:16:39.093 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:16:39.093 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v2", 2024-09-10T19:16:39.093 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.191:6830", 2024-09-10T19:16:39.093 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 1422610021 2024-09-10T19:16:39.093 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:16:39.093 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:16:39.093 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v1", 2024-09-10T19:16:39.093 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.191:6831", 2024-09-10T19:16:39.093 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 1422610021 2024-09-10T19:16:39.093 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:16:39.093 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:16:39.093 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:16:39.093 INFO:teuthology.orchestra.run.smithi183.stdout: "heartbeat_front_addrs": { 2024-09-10T19:16:39.094 INFO:teuthology.orchestra.run.smithi183.stdout: "addrvec": [ 2024-09-10T19:16:39.094 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:16:39.094 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v2", 2024-09-10T19:16:39.094 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.191:6828", 2024-09-10T19:16:39.094 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 1422610021 2024-09-10T19:16:39.094 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:16:39.094 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:16:39.094 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v1", 2024-09-10T19:16:39.094 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.191:6829", 2024-09-10T19:16:39.094 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 1422610021 2024-09-10T19:16:39.094 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:16:39.094 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:16:39.094 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:16:39.094 INFO:teuthology.orchestra.run.smithi183.stdout: "public_addr": "172.21.15.191:6825/1422610021", 2024-09-10T19:16:39.095 INFO:teuthology.orchestra.run.smithi183.stdout: "cluster_addr": "172.21.15.191:6827/1422610021", 2024-09-10T19:16:39.095 INFO:teuthology.orchestra.run.smithi183.stdout: "heartbeat_back_addr": "172.21.15.191:6831/1422610021", 2024-09-10T19:16:39.095 INFO:teuthology.orchestra.run.smithi183.stdout: "heartbeat_front_addr": "172.21.15.191:6829/1422610021", 2024-09-10T19:16:39.095 INFO:teuthology.orchestra.run.smithi183.stdout: "state": [ 2024-09-10T19:16:39.095 INFO:teuthology.orchestra.run.smithi183.stdout: "exists", 2024-09-10T19:16:39.095 INFO:teuthology.orchestra.run.smithi183.stdout: "up" 2024-09-10T19:16:39.095 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:16:39.095 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:16:39.095 INFO:teuthology.orchestra.run.smithi183.stdout: ], 2024-09-10T19:16:39.095 INFO:teuthology.orchestra.run.smithi183.stdout: "osd_xinfo": [ 2024-09-10T19:16:39.095 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:16:39.095 INFO:teuthology.orchestra.run.smithi183.stdout: "osd": 0, 2024-09-10T19:16:39.095 INFO:teuthology.orchestra.run.smithi183.stdout: "down_stamp": "0.000000", 2024-09-10T19:16:39.095 INFO:teuthology.orchestra.run.smithi183.stdout: "laggy_probability": 0, 2024-09-10T19:16:39.095 INFO:teuthology.orchestra.run.smithi183.stdout: "laggy_interval": 0, 2024-09-10T19:16:39.095 INFO:teuthology.orchestra.run.smithi183.stdout: "features": 4540138320759226367, 2024-09-10T19:16:39.096 INFO:teuthology.orchestra.run.smithi183.stdout: "old_weight": 0, 2024-09-10T19:16:39.096 INFO:teuthology.orchestra.run.smithi183.stdout: "last_purged_snaps_scrub": "2024-09-10T19:15:38.282064+0000", 2024-09-10T19:16:39.096 INFO:teuthology.orchestra.run.smithi183.stdout: "dead_epoch": 0 2024-09-10T19:16:39.096 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:16:39.096 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:16:39.096 INFO:teuthology.orchestra.run.smithi183.stdout: "osd": 1, 2024-09-10T19:16:39.096 INFO:teuthology.orchestra.run.smithi183.stdout: "down_stamp": "0.000000", 2024-09-10T19:16:39.096 INFO:teuthology.orchestra.run.smithi183.stdout: "laggy_probability": 0, 2024-09-10T19:16:39.096 INFO:teuthology.orchestra.run.smithi183.stdout: "laggy_interval": 0, 2024-09-10T19:16:39.096 INFO:teuthology.orchestra.run.smithi183.stdout: "features": 4540138320759226367, 2024-09-10T19:16:39.096 INFO:teuthology.orchestra.run.smithi183.stdout: "old_weight": 65536, 2024-09-10T19:16:39.096 INFO:teuthology.orchestra.run.smithi183.stdout: "last_purged_snaps_scrub": "2024-09-10T19:16:25.059015+0000", 2024-09-10T19:16:39.096 INFO:teuthology.orchestra.run.smithi183.stdout: "dead_epoch": 98 2024-09-10T19:16:39.096 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:16:39.097 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:16:39.097 INFO:teuthology.orchestra.run.smithi183.stdout: "osd": 2, 2024-09-10T19:16:39.097 INFO:teuthology.orchestra.run.smithi183.stdout: "down_stamp": "0.000000", 2024-09-10T19:16:39.097 INFO:teuthology.orchestra.run.smithi183.stdout: "laggy_probability": 0, 2024-09-10T19:16:39.097 INFO:teuthology.orchestra.run.smithi183.stdout: "laggy_interval": 0, 2024-09-10T19:16:39.097 INFO:teuthology.orchestra.run.smithi183.stdout: "features": 4540138320759226367, 2024-09-10T19:16:39.097 INFO:teuthology.orchestra.run.smithi183.stdout: "old_weight": 65536, 2024-09-10T19:16:39.097 INFO:teuthology.orchestra.run.smithi183.stdout: "last_purged_snaps_scrub": "2024-09-10T19:15:37.259736+0000", 2024-09-10T19:16:39.097 INFO:teuthology.orchestra.run.smithi183.stdout: "dead_epoch": 0 2024-09-10T19:16:39.097 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:16:39.097 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:16:39.097 INFO:teuthology.orchestra.run.smithi183.stdout: "osd": 3, 2024-09-10T19:16:39.097 INFO:teuthology.orchestra.run.smithi183.stdout: "down_stamp": "0.000000", 2024-09-10T19:16:39.097 INFO:teuthology.orchestra.run.smithi183.stdout: "laggy_probability": 0, 2024-09-10T19:16:39.097 INFO:teuthology.orchestra.run.smithi183.stdout: "laggy_interval": 0, 2024-09-10T19:16:39.098 INFO:teuthology.orchestra.run.smithi183.stdout: "features": 4540138320759226367, 2024-09-10T19:16:39.098 INFO:teuthology.orchestra.run.smithi183.stdout: "old_weight": 0, 2024-09-10T19:16:39.098 INFO:teuthology.orchestra.run.smithi183.stdout: "last_purged_snaps_scrub": "2024-09-10T19:15:57.563651+0000", 2024-09-10T19:16:39.098 INFO:teuthology.orchestra.run.smithi183.stdout: "dead_epoch": 0 2024-09-10T19:16:39.098 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:16:39.098 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:16:39.098 INFO:teuthology.orchestra.run.smithi183.stdout: "osd": 4, 2024-09-10T19:16:39.098 INFO:teuthology.orchestra.run.smithi183.stdout: "down_stamp": "0.000000", 2024-09-10T19:16:39.098 INFO:teuthology.orchestra.run.smithi183.stdout: "laggy_probability": 0, 2024-09-10T19:16:39.098 INFO:teuthology.orchestra.run.smithi183.stdout: "laggy_interval": 0, 2024-09-10T19:16:39.098 INFO:teuthology.orchestra.run.smithi183.stdout: "features": 4540138320759226367, 2024-09-10T19:16:39.098 INFO:teuthology.orchestra.run.smithi183.stdout: "old_weight": 0, 2024-09-10T19:16:39.098 INFO:teuthology.orchestra.run.smithi183.stdout: "last_purged_snaps_scrub": "2024-09-10T19:16:27.222375+0000", 2024-09-10T19:16:39.098 INFO:teuthology.orchestra.run.smithi183.stdout: "dead_epoch": 0 2024-09-10T19:16:39.098 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:16:39.099 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:16:39.099 INFO:teuthology.orchestra.run.smithi183.stdout: "osd": 5, 2024-09-10T19:16:39.099 INFO:teuthology.orchestra.run.smithi183.stdout: "down_stamp": "0.000000", 2024-09-10T19:16:39.099 INFO:teuthology.orchestra.run.smithi183.stdout: "laggy_probability": 0, 2024-09-10T19:16:39.099 INFO:teuthology.orchestra.run.smithi183.stdout: "laggy_interval": 0, 2024-09-10T19:16:39.099 INFO:teuthology.orchestra.run.smithi183.stdout: "features": 4540138320759226367, 2024-09-10T19:16:39.099 INFO:teuthology.orchestra.run.smithi183.stdout: "old_weight": 0, 2024-09-10T19:16:39.099 INFO:teuthology.orchestra.run.smithi183.stdout: "last_purged_snaps_scrub": "2024-09-10T19:16:11.186697+0000", 2024-09-10T19:16:39.099 INFO:teuthology.orchestra.run.smithi183.stdout: "dead_epoch": 0 2024-09-10T19:16:39.099 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:16:39.099 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:16:39.099 INFO:teuthology.orchestra.run.smithi183.stdout: "osd": 6, 2024-09-10T19:16:39.099 INFO:teuthology.orchestra.run.smithi183.stdout: "down_stamp": "0.000000", 2024-09-10T19:16:39.099 INFO:teuthology.orchestra.run.smithi183.stdout: "laggy_probability": 0, 2024-09-10T19:16:39.099 INFO:teuthology.orchestra.run.smithi183.stdout: "laggy_interval": 0, 2024-09-10T19:16:39.100 INFO:teuthology.orchestra.run.smithi183.stdout: "features": 4540138320759226367, 2024-09-10T19:16:39.100 INFO:teuthology.orchestra.run.smithi183.stdout: "old_weight": 0, 2024-09-10T19:16:39.100 INFO:teuthology.orchestra.run.smithi183.stdout: "last_purged_snaps_scrub": "2024-09-10T19:15:59.859226+0000", 2024-09-10T19:16:39.100 INFO:teuthology.orchestra.run.smithi183.stdout: "dead_epoch": 0 2024-09-10T19:16:39.100 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:16:39.100 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:16:39.100 INFO:teuthology.orchestra.run.smithi183.stdout: "osd": 7, 2024-09-10T19:16:39.100 INFO:teuthology.orchestra.run.smithi183.stdout: "down_stamp": "0.000000", 2024-09-10T19:16:39.100 INFO:teuthology.orchestra.run.smithi183.stdout: "laggy_probability": 0, 2024-09-10T19:16:39.100 INFO:teuthology.orchestra.run.smithi183.stdout: "laggy_interval": 0, 2024-09-10T19:16:39.100 INFO:teuthology.orchestra.run.smithi183.stdout: "features": 4540138320759226367, 2024-09-10T19:16:39.100 INFO:teuthology.orchestra.run.smithi183.stdout: "old_weight": 0, 2024-09-10T19:16:39.100 INFO:teuthology.orchestra.run.smithi183.stdout: "last_purged_snaps_scrub": "2024-09-10T19:16:28.114434+0000", 2024-09-10T19:16:39.101 INFO:teuthology.orchestra.run.smithi183.stdout: "dead_epoch": 0 2024-09-10T19:16:39.101 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:16:39.101 INFO:teuthology.orchestra.run.smithi183.stdout: ], 2024-09-10T19:16:39.101 INFO:teuthology.orchestra.run.smithi183.stdout: "pg_upmap": [], 2024-09-10T19:16:39.101 INFO:teuthology.orchestra.run.smithi183.stdout: "pg_upmap_items": [ 2024-09-10T19:16:39.101 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:16:39.101 INFO:teuthology.orchestra.run.smithi183.stdout: "pgid": "4.1", 2024-09-10T19:16:39.101 INFO:teuthology.orchestra.run.smithi183.stdout: "mappings": [ 2024-09-10T19:16:39.101 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:16:39.101 INFO:teuthology.orchestra.run.smithi183.stdout: "from": 6, 2024-09-10T19:16:39.101 INFO:teuthology.orchestra.run.smithi183.stdout: "to": 7 2024-09-10T19:16:39.101 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:16:39.101 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:16:39.102 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:16:39.102 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:16:39.102 INFO:teuthology.orchestra.run.smithi183.stdout: "pgid": "4.11", 2024-09-10T19:16:39.102 INFO:teuthology.orchestra.run.smithi183.stdout: "mappings": [ 2024-09-10T19:16:39.102 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:16:39.102 INFO:teuthology.orchestra.run.smithi183.stdout: "from": 3, 2024-09-10T19:16:39.102 INFO:teuthology.orchestra.run.smithi183.stdout: "to": 5 2024-09-10T19:16:39.102 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:16:39.102 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:16:39.102 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:16:39.102 INFO:teuthology.orchestra.run.smithi183.stdout: ], 2024-09-10T19:16:39.103 INFO:teuthology.orchestra.run.smithi183.stdout: "pg_temp": [], 2024-09-10T19:16:39.103 INFO:teuthology.orchestra.run.smithi183.stdout: "primary_temp": [], 2024-09-10T19:16:39.103 INFO:teuthology.orchestra.run.smithi183.stdout: "blocklist": { 2024-09-10T19:16:39.103 INFO:teuthology.orchestra.run.smithi183.stdout: "172.21.15.183:6801/3311783362": "2024-09-11T19:10:43.576943+0000", 2024-09-10T19:16:39.103 INFO:teuthology.orchestra.run.smithi183.stdout: "172.21.15.183:0/4088553922": "2024-09-11T19:10:43.576943+0000", 2024-09-10T19:16:39.103 INFO:teuthology.orchestra.run.smithi183.stdout: "172.21.15.183:0/3528769414": "2024-09-11T19:05:48.153879+0000", 2024-09-10T19:16:39.103 INFO:teuthology.orchestra.run.smithi183.stdout: "172.21.15.183:0/1766036658": "2024-09-11T19:05:48.153879+0000", 2024-09-10T19:16:39.103 INFO:teuthology.orchestra.run.smithi183.stdout: "172.21.15.183:0/3044170088": "2024-09-11T19:05:25.702354+0000", 2024-09-10T19:16:39.103 INFO:teuthology.orchestra.run.smithi183.stdout: "172.21.15.183:0/3037333778": "2024-09-11T19:05:25.702354+0000", 2024-09-10T19:16:39.103 INFO:teuthology.orchestra.run.smithi183.stdout: "172.21.15.183:6801/3747320570": "2024-09-11T19:05:25.702354+0000", 2024-09-10T19:16:39.103 INFO:teuthology.orchestra.run.smithi183.stdout: "172.21.15.183:0/3982386906": "2024-09-11T19:10:43.576943+0000", 2024-09-10T19:16:39.103 INFO:teuthology.orchestra.run.smithi183.stdout: "172.21.15.183:0/2436014455": "2024-09-11T19:05:25.702354+0000", 2024-09-10T19:16:39.103 INFO:teuthology.orchestra.run.smithi183.stdout: "172.21.15.183:6801/659745765": "2024-09-11T19:05:48.153879+0000", 2024-09-10T19:16:39.103 INFO:teuthology.orchestra.run.smithi183.stdout: "172.21.15.183:6800/3747320570": "2024-09-11T19:05:25.702354+0000", 2024-09-10T19:16:39.103 INFO:teuthology.orchestra.run.smithi183.stdout: "172.21.15.183:0/1652576282": "2024-09-11T19:10:43.576943+0000", 2024-09-10T19:16:39.104 INFO:teuthology.orchestra.run.smithi183.stdout: "172.21.15.183:6800/3311783362": "2024-09-11T19:10:43.576943+0000", 2024-09-10T19:16:39.104 INFO:teuthology.orchestra.run.smithi183.stdout: "172.21.15.183:0/1012899549": "2024-09-11T19:05:48.153879+0000", 2024-09-10T19:16:39.104 INFO:teuthology.orchestra.run.smithi183.stdout: "172.21.15.183:6800/659745765": "2024-09-11T19:05:48.153879+0000", 2024-09-10T19:16:39.104 INFO:teuthology.orchestra.run.smithi183.stdout: "172.21.15.183:0/412044400": "2024-09-11T19:10:43.576943+0000", 2024-09-10T19:16:39.104 INFO:teuthology.orchestra.run.smithi183.stdout: "172.21.15.183:0/3636171401": "2024-09-11T19:10:43.576943+0000" 2024-09-10T19:16:39.104 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:16:39.104 INFO:teuthology.orchestra.run.smithi183.stdout: "range_blocklist": {}, 2024-09-10T19:16:39.104 INFO:teuthology.orchestra.run.smithi183.stdout: "erasure_code_profiles": { 2024-09-10T19:16:39.104 INFO:teuthology.orchestra.run.smithi183.stdout: "default": { 2024-09-10T19:16:39.104 INFO:teuthology.orchestra.run.smithi183.stdout: "crush-failure-domain": "osd", 2024-09-10T19:16:39.104 INFO:teuthology.orchestra.run.smithi183.stdout: "k": "2", 2024-09-10T19:16:39.104 INFO:teuthology.orchestra.run.smithi183.stdout: "m": "1", 2024-09-10T19:16:39.104 INFO:teuthology.orchestra.run.smithi183.stdout: "plugin": "jerasure", 2024-09-10T19:16:39.105 INFO:teuthology.orchestra.run.smithi183.stdout: "technique": "reed_sol_van" 2024-09-10T19:16:39.105 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:16:39.105 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:16:39.105 INFO:teuthology.orchestra.run.smithi183.stdout: "removed_snaps_queue": [], 2024-09-10T19:16:39.105 INFO:teuthology.orchestra.run.smithi183.stdout: "new_removed_snaps": [], 2024-09-10T19:16:39.105 INFO:teuthology.orchestra.run.smithi183.stdout: "new_purged_snaps": [], 2024-09-10T19:16:39.105 INFO:teuthology.orchestra.run.smithi183.stdout: "crush_node_flags": {}, 2024-09-10T19:16:39.105 INFO:teuthology.orchestra.run.smithi183.stdout: "device_class_flags": {}, 2024-09-10T19:16:39.105 INFO:teuthology.orchestra.run.smithi183.stdout: "stretch_mode": { 2024-09-10T19:16:39.105 INFO:teuthology.orchestra.run.smithi183.stdout: "stretch_mode_enabled": false, 2024-09-10T19:16:39.108 INFO:teuthology.orchestra.run.smithi183.stdout: "stretch_bucket_count": 0, 2024-09-10T19:16:39.108 INFO:teuthology.orchestra.run.smithi183.stdout: "degraded_stretch_mode": 0, 2024-09-10T19:16:39.108 INFO:teuthology.orchestra.run.smithi183.stdout: "recovering_stretch_mode": 0, 2024-09-10T19:16:39.108 INFO:teuthology.orchestra.run.smithi183.stdout: "stretch_mode_bucket": 0 2024-09-10T19:16:39.108 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:16:39.108 INFO:teuthology.orchestra.run.smithi183.stdout:} 2024-09-10T19:16:39.108 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:16:39.129 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:16:39.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:16:39.130 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:16:39.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:16:38 smithi183 ceph-mon[29736]: 2.5 deep-scrub starts 2024-09-10T19:16:39.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:16:38 smithi183 ceph-mon[29736]: pgmap v267: 148 pgs: 148 active+clean; 1.7 GiB data, 6.6 GiB used, 530 GiB / 536 GiB avail; 1.2 KiB/s rd, 170 MiB/s wr, 2.72k op/s; 108 B/s, 0 objects/s recovering 2024-09-10T19:16:39.182 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:16:38 smithi183 ceph-mon[29736]: 2.5 deep-scrub ok 2024-09-10T19:16:39.182 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:16:38 smithi183 ceph-mon[29736]: 5.1c scrub starts 2024-09-10T19:16:39.182 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:16:38 smithi183 ceph-mon[29736]: 5.1c scrub ok 2024-09-10T19:16:39.182 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:16:38 smithi183 ceph-mon[29736]: 2.1f deep-scrub starts 2024-09-10T19:16:39.182 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:16:38 smithi183 ceph-mon[29736]: 2.1f deep-scrub ok 2024-09-10T19:16:39.182 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:16:38 smithi183 ceph-mon[29736]: from='client.? 172.21.15.183:0/920550170' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-09-10T19:16:39.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:16:38 smithi183 ceph-mon[41562]: 2.5 deep-scrub starts 2024-09-10T19:16:39.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:16:38 smithi183 ceph-mon[41562]: pgmap v267: 148 pgs: 148 active+clean; 1.7 GiB data, 6.6 GiB used, 530 GiB / 536 GiB avail; 1.2 KiB/s rd, 170 MiB/s wr, 2.72k op/s; 108 B/s, 0 objects/s recovering 2024-09-10T19:16:39.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:16:38 smithi183 ceph-mon[41562]: 2.5 deep-scrub ok 2024-09-10T19:16:39.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:16:38 smithi183 ceph-mon[41562]: 5.1c scrub starts 2024-09-10T19:16:39.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:16:38 smithi183 ceph-mon[41562]: 5.1c scrub ok 2024-09-10T19:16:39.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:16:38 smithi183 ceph-mon[41562]: 2.1f deep-scrub starts 2024-09-10T19:16:39.183 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:16:38 smithi183 ceph-mon[41562]: 2.1f deep-scrub ok 2024-09-10T19:16:39.183 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:16:38 smithi183 ceph-mon[41562]: from='client.? 172.21.15.183:0/920550170' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-09-10T19:16:39.190 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:16:38 smithi191 ceph-mon[32884]: 2.5 deep-scrub starts 2024-09-10T19:16:39.190 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:16:38 smithi191 ceph-mon[32884]: pgmap v267: 148 pgs: 148 active+clean; 1.7 GiB data, 6.6 GiB used, 530 GiB / 536 GiB avail; 1.2 KiB/s rd, 170 MiB/s wr, 2.72k op/s; 108 B/s, 0 objects/s recovering 2024-09-10T19:16:39.190 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:16:38 smithi191 ceph-mon[32884]: 2.5 deep-scrub ok 2024-09-10T19:16:39.190 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:16:38 smithi191 ceph-mon[32884]: 5.1c scrub starts 2024-09-10T19:16:39.190 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:16:38 smithi191 ceph-mon[32884]: 5.1c scrub ok 2024-09-10T19:16:39.190 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:16:38 smithi191 ceph-mon[32884]: 2.1f deep-scrub starts 2024-09-10T19:16:39.190 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:16:38 smithi191 ceph-mon[32884]: 2.1f deep-scrub ok 2024-09-10T19:16:39.190 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:16:38 smithi191 ceph-mon[32884]: from='client.? 172.21.15.183:0/920550170' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-09-10T19:16:39.231 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:16:39.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:16:39.257 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:16:39.258 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:16:39.358 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:16:39.393 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:16:39.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:16:39.395 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:16:39.450 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:16:39.465 INFO:journalctl@ceph.iscsi.iscsi.a.smithi191.stdout:Sep 10 19:16:39 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-iscsi-iscsi-a[61816]: debug there is no tcmu-runner data available 2024-09-10T19:16:39.496 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:16:39.534 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:16:39.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:16:39.536 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:16:39.636 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:16:39.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:16:39.658 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:16:39.659 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:16:39.759 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:16:39.796 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:16:39.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:16:39.797 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:16:39.823 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: 15 16 35231 35215 146.697 84.9375 0.00892958 0.00681198 2024-09-10T19:16:39.897 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:16:39.930 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:16:39.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:16:39.931 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:16:40.032 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:16:40.061 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:16:40.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:16:40.062 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:16:40.162 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:16:40.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:16:39 smithi183 ceph-mon[29736]: 6.9 scrub starts 2024-09-10T19:16:40.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:16:39 smithi183 ceph-mon[29736]: 6.9 scrub ok 2024-09-10T19:16:40.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:16:39 smithi183 ceph-mon[29736]: from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:16:40.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:16:39 smithi183 ceph-mon[41562]: 6.9 scrub starts 2024-09-10T19:16:40.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:16:39 smithi183 ceph-mon[41562]: 6.9 scrub ok 2024-09-10T19:16:40.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:16:39 smithi183 ceph-mon[41562]: from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:16:40.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:16:40.203 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:16:40.203 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:16:40.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:16:39 smithi191 ceph-mon[32884]: 6.9 scrub starts 2024-09-10T19:16:40.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:16:39 smithi191 ceph-mon[32884]: 6.9 scrub ok 2024-09-10T19:16:40.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:16:39 smithi191 ceph-mon[32884]: from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:16:40.304 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:16:40.331 INFO:tasks.thrashosds.thrasher:j is {'epoch': 160, 'fsid': '320f6df8-6fa7-11ef-bcea-c7b262605968', 'created': '2024-09-10T19:05:03.688531+0000', 'modified': '2024-09-10T19:16:34.735829+0000', 'last_up_change': '2024-09-10T19:15:19.923894+0000', 'last_in_change': '2024-09-10T19:16:23.815531+0000', 'flags': 'sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5799936, 'flags_set': ['pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 19, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 8, 'max_osd': 8, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'quincy', 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-10T19:08:00.456786+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}, {'pool': 2, 'pool_name': '.rgw.root', 'create_time': '2024-09-10T19:10:05.993763+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '60', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}}, {'pool': 3, 'pool_name': 'datapool', 'create_time': '2024-09-10T19:10:06.224267+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 3, 'pg_placement_num': 3, 'pg_placement_num_target': 3, 'pg_num_target': 3, 'pg_num_pending': 3, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '63', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 3, 'snap_epoch': 63, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 4, 'pool_name': 'default.rgw.log', 'create_time': '2024-09-10T19:10:07.958284+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '62', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}}, {'pool': 5, 'pool_name': 'default.rgw.control', 'create_time': '2024-09-10T19:10:09.933596+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '64', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}}, {'pool': 6, 'pool_name': 'default.rgw.meta', 'create_time': '2024-09-10T19:10:11.993712+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '66', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4}, 'application_metadata': {'rgw': {}}}, {'pool': 8, 'pool_name': 'unique_pool_1', 'create_time': '2024-09-10T19:16:11.474472+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 16, 'pg_placement_num': 16, 'pg_placement_num_target': 16, 'pg_num_target': 16, 'pg_num_pending': 16, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '153', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}}], 'osds': [{'osd': 0, 'uuid': '9d323010-394a-49aa-ba9b-244d87faf36e', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 8, 'up_thru': 154, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.183:6802', 'nonce': 2813074952}, {'type': 'v1', 'addr': '172.21.15.183:6803', 'nonce': 2813074952}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.183:6804', 'nonce': 2813074952}, {'type': 'v1', 'addr': '172.21.15.183:6805', 'nonce': 2813074952}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.183:6808', 'nonce': 2813074952}, {'type': 'v1', 'addr': '172.21.15.183:6809', 'nonce': 2813074952}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.183:6806', 'nonce': 2813074952}, {'type': 'v1', 'addr': '172.21.15.183:6807', 'nonce': 2813074952}]}, 'public_addr': '172.21.15.183:6803/2813074952', 'cluster_addr': '172.21.15.183:6805/2813074952', 'heartbeat_back_addr': '172.21.15.183:6809/2813074952', 'heartbeat_front_addr': '172.21.15.183:6807/2813074952', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': '71a6c5dc-5109-4284-8d20-187d8e7afc9e', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 1, 'last_clean_begin': 13, 'last_clean_end': 98, 'up_from': 138, 'up_thru': 86, 'down_at': 99, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.183:6810', 'nonce': 219854798}, {'type': 'v1', 'addr': '172.21.15.183:6811', 'nonce': 219854798}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.183:6812', 'nonce': 219854798}, {'type': 'v1', 'addr': '172.21.15.183:6813', 'nonce': 219854798}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.183:6816', 'nonce': 219854798}, {'type': 'v1', 'addr': '172.21.15.183:6817', 'nonce': 219854798}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.183:6814', 'nonce': 219854798}, {'type': 'v1', 'addr': '172.21.15.183:6815', 'nonce': 219854798}]}, 'public_addr': '172.21.15.183:6811/219854798', 'cluster_addr': '172.21.15.183:6813/219854798', 'heartbeat_back_addr': '172.21.15.183:6817/219854798', 'heartbeat_front_addr': '172.21.15.183:6815/219854798', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': 'fc310d5b-b89b-4655-9989-7e10ba356509', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 18, 'up_thru': 148, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.183:6818', 'nonce': 3730013786}, {'type': 'v1', 'addr': '172.21.15.183:6819', 'nonce': 3730013786}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.183:6820', 'nonce': 3730013786}, {'type': 'v1', 'addr': '172.21.15.183:6821', 'nonce': 3730013786}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.183:6824', 'nonce': 3730013786}, {'type': 'v1', 'addr': '172.21.15.183:6825', 'nonce': 3730013786}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.183:6822', 'nonce': 3730013786}, {'type': 'v1', 'addr': '172.21.15.183:6823', 'nonce': 3730013786}]}, 'public_addr': '172.21.15.183:6819/3730013786', 'cluster_addr': '172.21.15.183:6821/3730013786', 'heartbeat_back_addr': '172.21.15.183:6825/3730013786', 'heartbeat_front_addr': '172.21.15.183:6823/3730013786', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': '7fd07a95-ad22-43f5-a968-539552283e89', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 26, 'up_thru': 154, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.183:6826', 'nonce': 3522836984}, {'type': 'v1', 'addr': '172.21.15.183:6827', 'nonce': 3522836984}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.183:6828', 'nonce': 3522836984}, {'type': 'v1', 'addr': '172.21.15.183:6829', 'nonce': 3522836984}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.183:6832', 'nonce': 3522836984}, {'type': 'v1', 'addr': '172.21.15.183:6833', 'nonce': 3522836984}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.183:6830', 'nonce': 3522836984}, {'type': 'v1', 'addr': '172.21.15.183:6831', 'nonce': 3522836984}]}, 'public_addr': '172.21.15.183:6827/3522836984', 'cluster_addr': '172.21.15.183:6829/3522836984', 'heartbeat_back_addr': '172.21.15.183:6833/3522836984', 'heartbeat_front_addr': '172.21.15.183:6831/3522836984', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': '631083fb-e503-44da-9b6b-154e88539ee7', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 32, 'up_thru': 154, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.191:6800', 'nonce': 2674828539}, {'type': 'v1', 'addr': '172.21.15.191:6801', 'nonce': 2674828539}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.191:6802', 'nonce': 2674828539}, {'type': 'v1', 'addr': '172.21.15.191:6803', 'nonce': 2674828539}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.191:6806', 'nonce': 2674828539}, {'type': 'v1', 'addr': '172.21.15.191:6807', 'nonce': 2674828539}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.191:6804', 'nonce': 2674828539}, {'type': 'v1', 'addr': '172.21.15.191:6805', 'nonce': 2674828539}]}, 'public_addr': '172.21.15.191:6801/2674828539', 'cluster_addr': '172.21.15.191:6803/2674828539', 'heartbeat_back_addr': '172.21.15.191:6807/2674828539', 'heartbeat_front_addr': '172.21.15.191:6805/2674828539', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': '6b963695-3eca-4e6e-9207-05b8010ebff9', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 39, 'up_thru': 154, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.191:6808', 'nonce': 681733378}, {'type': 'v1', 'addr': '172.21.15.191:6809', 'nonce': 681733378}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.191:6810', 'nonce': 681733378}, {'type': 'v1', 'addr': '172.21.15.191:6811', 'nonce': 681733378}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.191:6814', 'nonce': 681733378}, {'type': 'v1', 'addr': '172.21.15.191:6815', 'nonce': 681733378}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.191:6812', 'nonce': 681733378}, {'type': 'v1', 'addr': '172.21.15.191:6813', 'nonce': 681733378}]}, 'public_addr': '172.21.15.191:6809/681733378', 'cluster_addr': '172.21.15.191:6811/681733378', 'heartbeat_back_addr': '172.21.15.191:6815/681733378', 'heartbeat_front_addr': '172.21.15.191:6813/681733378', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': '2dc9754f-6aef-4e19-8605-2f1591921811', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 47, 'up_thru': 154, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.191:6816', 'nonce': 2825813477}, {'type': 'v1', 'addr': '172.21.15.191:6817', 'nonce': 2825813477}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.191:6818', 'nonce': 2825813477}, {'type': 'v1', 'addr': '172.21.15.191:6819', 'nonce': 2825813477}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.191:6822', 'nonce': 2825813477}, {'type': 'v1', 'addr': '172.21.15.191:6823', 'nonce': 2825813477}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.191:6820', 'nonce': 2825813477}, {'type': 'v1', 'addr': '172.21.15.191:6821', 'nonce': 2825813477}]}, 'public_addr': '172.21.15.191:6817/2825813477', 'cluster_addr': '172.21.15.191:6819/2825813477', 'heartbeat_back_addr': '172.21.15.191:6823/2825813477', 'heartbeat_front_addr': '172.21.15.191:6821/2825813477', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': '1a9a299e-d5ba-4b9a-85d5-7f6a5bc754e1', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 54, 'up_thru': 154, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.191:6824', 'nonce': 1422610021}, {'type': 'v1', 'addr': '172.21.15.191:6825', 'nonce': 1422610021}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.191:6826', 'nonce': 1422610021}, {'type': 'v1', 'addr': '172.21.15.191:6827', 'nonce': 1422610021}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.191:6830', 'nonce': 1422610021}, {'type': 'v1', 'addr': '172.21.15.191:6831', 'nonce': 1422610021}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.191:6828', 'nonce': 1422610021}, {'type': 'v1', 'addr': '172.21.15.191:6829', 'nonce': 1422610021}]}, 'public_addr': '172.21.15.191:6825/1422610021', 'cluster_addr': '172.21.15.191:6827/1422610021', 'heartbeat_back_addr': '172.21.15.191:6831/1422610021', 'heartbeat_front_addr': '172.21.15.191:6829/1422610021', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138320759226367, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-10T19:15:38.282064+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138320759226367, 'old_weight': 65536, 'last_purged_snaps_scrub': '2024-09-10T19:16:25.059015+0000', 'dead_epoch': 98}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138320759226367, 'old_weight': 65536, 'last_purged_snaps_scrub': '2024-09-10T19:15:37.259736+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138320759226367, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-10T19:15:57.563651+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138320759226367, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-10T19:16:27.222375+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138320759226367, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-10T19:16:11.186697+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138320759226367, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-10T19:15:59.859226+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138320759226367, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-10T19:16:28.114434+0000', 'dead_epoch': 0}], 'pg_upmap': [], 'pg_upmap_items': [{'pgid': '4.1', 'mappings': [{'from': 6, 'to': 7}]}, {'pgid': '4.11', 'mappings': [{'from': 3, 'to': 5}]}], 'pg_temp': [], 'primary_temp': [], 'blocklist': {'172.21.15.183:6801/3311783362': '2024-09-11T19:10:43.576943+0000', '172.21.15.183:0/4088553922': '2024-09-11T19:10:43.576943+0000', '172.21.15.183:0/3528769414': '2024-09-11T19:05:48.153879+0000', '172.21.15.183:0/1766036658': '2024-09-11T19:05:48.153879+0000', '172.21.15.183:0/3044170088': '2024-09-11T19:05:25.702354+0000', '172.21.15.183:0/3037333778': '2024-09-11T19:05:25.702354+0000', '172.21.15.183:6801/3747320570': '2024-09-11T19:05:25.702354+0000', '172.21.15.183:0/3982386906': '2024-09-11T19:10:43.576943+0000', '172.21.15.183:0/2436014455': '2024-09-11T19:05:25.702354+0000', '172.21.15.183:6801/659745765': '2024-09-11T19:05:48.153879+0000', '172.21.15.183:6800/3747320570': '2024-09-11T19:05:25.702354+0000', '172.21.15.183:0/1652576282': '2024-09-11T19:10:43.576943+0000', '172.21.15.183:6800/3311783362': '2024-09-11T19:10:43.576943+0000', '172.21.15.183:0/1012899549': '2024-09-11T19:05:48.153879+0000', '172.21.15.183:6800/659745765': '2024-09-11T19:05:48.153879+0000', '172.21.15.183:0/412044400': '2024-09-11T19:10:43.576943+0000', '172.21.15.183:0/3636171401': '2024-09-11T19:10:43.576943+0000'}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-09-10T19:16:40.332 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph pg dump --format=json 2024-09-10T19:16:40.343 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:16:40.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:16:40.344 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:16:40.445 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:16:40.481 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:16:40.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:16:40.482 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:16:40.583 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:16:40.618 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:16:40.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:16:40.620 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:16:40.720 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:16:40.745 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:16:40.747 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:16:40.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:16:40.748 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:16:40.823 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: 16 15 37016 37001 144.504 111.625 0.0123759 0.00691501 2024-09-10T19:16:40.849 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:16:40.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:16:40.886 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:16:40.887 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:16:40.937 DEBUG:teuthology.orchestra.run.smithi191:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph daemon osd.6 dump_ops_in_flight 2024-09-10T19:16:40.987 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:16:41.015 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:16:41.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:16:41.016 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:16:41.117 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:16:41.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:16:41.157 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:16:41.157 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:16:41.173 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:16:40 smithi183 ceph-mon[29736]: 2.b deep-scrub starts 2024-09-10T19:16:41.173 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:16:40 smithi183 ceph-mon[29736]: 2.b deep-scrub ok 2024-09-10T19:16:41.173 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:16:40 smithi183 ceph-mon[29736]: pgmap v268: 148 pgs: 148 active+clean; 1.7 GiB data, 6.9 GiB used, 529 GiB / 536 GiB avail; 577 B/s rd, 87 MiB/s wr, 1.39k op/s 2024-09-10T19:16:41.173 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:16:40 smithi183 ceph-mon[29736]: 6.16 scrub starts 2024-09-10T19:16:41.174 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:16:40 smithi183 ceph-mon[29736]: 6.16 scrub ok 2024-09-10T19:16:41.174 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:16:40 smithi183 ceph-mon[41562]: 2.b deep-scrub starts 2024-09-10T19:16:41.174 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:16:40 smithi183 ceph-mon[41562]: 2.b deep-scrub ok 2024-09-10T19:16:41.174 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:16:40 smithi183 ceph-mon[41562]: pgmap v268: 148 pgs: 148 active+clean; 1.7 GiB data, 6.9 GiB used, 529 GiB / 536 GiB avail; 577 B/s rd, 87 MiB/s wr, 1.39k op/s 2024-09-10T19:16:41.174 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:16:40 smithi183 ceph-mon[41562]: 6.16 scrub starts 2024-09-10T19:16:41.174 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:16:40 smithi183 ceph-mon[41562]: 6.16 scrub ok 2024-09-10T19:16:41.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:16:40 smithi191 ceph-mon[32884]: 2.b deep-scrub starts 2024-09-10T19:16:41.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:16:40 smithi191 ceph-mon[32884]: 2.b deep-scrub ok 2024-09-10T19:16:41.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:16:40 smithi191 ceph-mon[32884]: pgmap v268: 148 pgs: 148 active+clean; 1.7 GiB data, 6.9 GiB used, 529 GiB / 536 GiB avail; 577 B/s rd, 87 MiB/s wr, 1.39k op/s 2024-09-10T19:16:41.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:16:40 smithi191 ceph-mon[32884]: 6.16 scrub starts 2024-09-10T19:16:41.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:16:40 smithi191 ceph-mon[32884]: 6.16 scrub ok 2024-09-10T19:16:41.258 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:16:41.284 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:16:41.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:16:41.285 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:16:41.373 INFO:teuthology.orchestra.run.smithi191.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.b/config 2024-09-10T19:16:41.386 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:16:41.431 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Sep 10 19:16:41 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-y[29989]: ::ffff:172.21.15.191 - - [10/Sep/2024:19:16:41] "GET /metrics HTTP/1.1" 200 223927 "" "Prometheus/2.43.0" 2024-09-10T19:16:41.446 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:16:41.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:16:41.447 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:16:41.548 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:16:41.587 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:16:41.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:16:41.588 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:16:41.689 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:16:41.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:16:41.728 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:16:41.728 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:16:41.823 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: 17 16 38745 38729 142.357 108 0.00595766 0.00701876 2024-09-10T19:16:41.829 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:16:41.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:16:41.870 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:16:41.870 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:16:41.971 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:16:42.012 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:16:42.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:16:42.013 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:16:42.114 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:16:42.148 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:16:42.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:16:42.149 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:16:42.250 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:16:42.277 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:16:42.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:16:42.277 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:16:42.380 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:16:42.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:16:42.422 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:16:42.422 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:16:42.523 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:16:42.561 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:16:42.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:16:42.562 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:16:42.663 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:16:42.699 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:16:42.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:16:42.700 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:16:42.801 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:16:42.823 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: 18 16 40523 40507 140.621 111.125 0.0158703 0.00710523 2024-09-10T19:16:42.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:16:42.842 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:16:42.843 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:16:42.943 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:16:42.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:16:42.990 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:16:42.990 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:16:43.091 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:16:43.130 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:16:43.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:16:43.131 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:16:43.134 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:16:42 smithi191 ceph-mon[32884]: pgmap v269: 148 pgs: 148 active+clean; 2.2 GiB data, 8.1 GiB used, 528 GiB / 536 GiB avail; 1023 B/s rd, 122 MiB/s wr, 1.95k op/s 2024-09-10T19:16:43.134 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:16:42 smithi191 ceph-mon[32884]: 2.19 scrub starts 2024-09-10T19:16:43.134 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:16:42 smithi191 ceph-mon[32884]: 2.19 scrub ok 2024-09-10T19:16:43.134 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:16:42 smithi191 ceph-mon[32884]: 3.1 scrub starts 2024-09-10T19:16:43.134 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:16:42 smithi191 ceph-mon[32884]: 3.1 scrub ok 2024-09-10T19:16:43.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:16:42 smithi183 ceph-mon[29736]: pgmap v269: 148 pgs: 148 active+clean; 2.2 GiB data, 8.1 GiB used, 528 GiB / 536 GiB avail; 1023 B/s rd, 122 MiB/s wr, 1.95k op/s 2024-09-10T19:16:43.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:16:42 smithi183 ceph-mon[29736]: 2.19 scrub starts 2024-09-10T19:16:43.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:16:42 smithi183 ceph-mon[29736]: 2.19 scrub ok 2024-09-10T19:16:43.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:16:42 smithi183 ceph-mon[29736]: 3.1 scrub starts 2024-09-10T19:16:43.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:16:42 smithi183 ceph-mon[29736]: 3.1 scrub ok 2024-09-10T19:16:43.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:16:42 smithi183 ceph-mon[41562]: pgmap v269: 148 pgs: 148 active+clean; 2.2 GiB data, 8.1 GiB used, 528 GiB / 536 GiB avail; 1023 B/s rd, 122 MiB/s wr, 1.95k op/s 2024-09-10T19:16:43.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:16:42 smithi183 ceph-mon[41562]: 2.19 scrub starts 2024-09-10T19:16:43.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:16:42 smithi183 ceph-mon[41562]: 2.19 scrub ok 2024-09-10T19:16:43.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:16:42 smithi183 ceph-mon[41562]: 3.1 scrub starts 2024-09-10T19:16:43.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:16:42 smithi183 ceph-mon[41562]: 3.1 scrub ok 2024-09-10T19:16:43.232 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:16:43.265 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:16:43.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:16:43.266 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:16:43.367 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:16:43.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:16:43.408 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:16:43.408 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:16:43.447 INFO:teuthology.orchestra.run.smithi191.stdout:{ 2024-09-10T19:16:43.447 INFO:teuthology.orchestra.run.smithi191.stdout: "ops": [ 2024-09-10T19:16:43.447 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:43.447 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.15051.0:41570 8.9 8:9c199939:::benchmark_data_smithi183_114924_object41569:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160)", 2024-09-10T19:16:43.447 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-10T19:16:43.437733+0000", 2024-09-10T19:16:43.447 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 0.0068109590000000001, 2024-09-10T19:16:43.447 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.006845037, 2024-09-10T19:16:43.447 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-10T19:16:43.448 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "waiting for sub ops", 2024-09-10T19:16:43.448 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-10T19:16:43.448 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.15051", 2024-09-10T19:16:43.448 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.183:0/3632028672", 2024-09-10T19:16:43.448 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 41570 2024-09-10T19:16:43.448 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:43.448 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-10T19:16:43.448 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:43.448 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-10T19:16:43.448 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:16:43.437733+0000", 2024-09-10T19:16:43.448 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:16:43.448 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:43.448 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:43.448 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-10T19:16:43.449 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:16:43.437733+0000", 2024-09-10T19:16:43.449 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:16:43.449 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:43.449 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:43.449 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-10T19:16:43.449 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:16:43.437732+0000", 2024-09-10T19:16:43.449 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999986 2024-09-10T19:16:43.449 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:43.449 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:43.449 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-10T19:16:43.449 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:16:43.437782+0000", 2024-09-10T19:16:43.449 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.0701999999999997e-05 2024-09-10T19:16:43.449 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:43.450 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:43.450 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-10T19:16:43.450 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:16:43.437797+0000", 2024-09-10T19:16:43.450 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.4329e-05 2024-09-10T19:16:43.450 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:43.450 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:43.450 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-10T19:16:43.450 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:16:43.437803+0000", 2024-09-10T19:16:43.450 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.6080000000000001e-06 2024-09-10T19:16:43.450 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:43.450 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:43.450 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:16:43.450 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:16:43.438247+0000", 2024-09-10T19:16:43.451 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00044353200000000001 2024-09-10T19:16:43.451 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:43.451 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:43.451 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-10T19:16:43.451 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:16:43.438378+0000", 2024-09-10T19:16:43.451 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000131049 2024-09-10T19:16:43.451 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:43.451 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:43.451 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0,5", 2024-09-10T19:16:43.451 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:16:43.438513+0000", 2024-09-10T19:16:43.451 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00013540199999999999 2024-09-10T19:16:43.451 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:43.452 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:43.452 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-09-10T19:16:43.452 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:16:43.440047+0000", 2024-09-10T19:16:43.452 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0015333339999999999 2024-09-10T19:16:43.452 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:43.452 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:43.452 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:16:43.452 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:16:43.441186+0000", 2024-09-10T19:16:43.452 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00113938 2024-09-10T19:16:43.452 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:16:43.452 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-10T19:16:43.452 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:16:43.452 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:43.453 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:43.453 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.15051.0:41556 8.9 8:9fad780c:::benchmark_data_smithi183_114924_object41555:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160)", 2024-09-10T19:16:43.453 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-10T19:16:43.429046+0000", 2024-09-10T19:16:43.453 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 0.015498352, 2024-09-10T19:16:43.453 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.015625956999999999, 2024-09-10T19:16:43.453 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-10T19:16:43.453 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "waiting for sub ops", 2024-09-10T19:16:43.453 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-10T19:16:43.453 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.15051", 2024-09-10T19:16:43.453 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.183:0/3632028672", 2024-09-10T19:16:43.453 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 41556 2024-09-10T19:16:43.453 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:43.454 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-10T19:16:43.454 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:43.454 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-10T19:16:43.454 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:16:43.429046+0000", 2024-09-10T19:16:43.454 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:16:43.454 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:43.454 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:43.454 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-10T19:16:43.454 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:16:43.429046+0000", 2024-09-10T19:16:43.454 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:16:43.454 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:43.454 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:43.454 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-10T19:16:43.454 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:16:43.429045+0000", 2024-09-10T19:16:43.455 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.999999 2024-09-10T19:16:43.455 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:43.455 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:43.455 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-10T19:16:43.455 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:16:43.429127+0000", 2024-09-10T19:16:43.455 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.2441000000000006e-05 2024-09-10T19:16:43.455 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:43.455 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:43.455 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-10T19:16:43.455 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:16:43.429137+0000", 2024-09-10T19:16:43.455 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.011e-05 2024-09-10T19:16:43.455 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:43.455 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:43.456 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-10T19:16:43.456 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:16:43.429142+0000", 2024-09-10T19:16:43.456 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.0189999999999997e-06 2024-09-10T19:16:43.456 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:43.456 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:43.456 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:16:43.456 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:16:43.433078+0000", 2024-09-10T19:16:43.456 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0039358650000000002 2024-09-10T19:16:43.456 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:43.456 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:43.456 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-10T19:16:43.457 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:16:43.433188+0000", 2024-09-10T19:16:43.457 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000109582 2024-09-10T19:16:43.457 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:43.457 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:43.457 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0,5", 2024-09-10T19:16:43.457 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:16:43.433315+0000", 2024-09-10T19:16:43.457 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00012713899999999999 2024-09-10T19:16:43.457 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:43.457 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:43.457 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-09-10T19:16:43.457 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:16:43.434844+0000", 2024-09-10T19:16:43.457 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0015287930000000001 2024-09-10T19:16:43.457 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:43.458 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:43.458 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:16:43.458 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:16:43.436759+0000", 2024-09-10T19:16:43.458 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0019147089999999999 2024-09-10T19:16:43.458 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:16:43.458 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-10T19:16:43.458 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:16:43.458 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:43.458 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:43.458 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.15051.0:41560 8.9 8:9d7c781a:::benchmark_data_smithi183_114924_object41559:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160)", 2024-09-10T19:16:43.458 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-10T19:16:43.432167+0000", 2024-09-10T19:16:43.458 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 0.012377607000000001, 2024-09-10T19:16:43.458 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.012601533999999999, 2024-09-10T19:16:43.458 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-10T19:16:43.459 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "waiting for sub ops", 2024-09-10T19:16:43.459 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-10T19:16:43.459 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.15051", 2024-09-10T19:16:43.459 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.183:0/3632028672", 2024-09-10T19:16:43.459 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 41560 2024-09-10T19:16:43.459 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:43.459 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-10T19:16:43.459 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:43.459 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-10T19:16:43.459 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:16:43.432167+0000", 2024-09-10T19:16:43.459 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:16:43.459 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:43.459 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:43.459 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-10T19:16:43.460 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:16:43.432167+0000", 2024-09-10T19:16:43.460 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:16:43.460 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:43.460 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:43.460 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-10T19:16:43.460 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:16:43.432165+0000", 2024-09-10T19:16:43.460 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999981 2024-09-10T19:16:43.460 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:43.460 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:43.460 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-10T19:16:43.460 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:16:43.432226+0000", 2024-09-10T19:16:43.460 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.1277000000000003e-05 2024-09-10T19:16:43.460 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:43.460 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:43.461 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-10T19:16:43.461 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:16:43.432241+0000", 2024-09-10T19:16:43.461 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.4701e-05 2024-09-10T19:16:43.461 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:43.461 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:43.461 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-10T19:16:43.461 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:16:43.432247+0000", 2024-09-10T19:16:43.461 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.6229999999999999e-06 2024-09-10T19:16:43.461 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:43.461 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:43.461 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:16:43.461 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:16:43.434863+0000", 2024-09-10T19:16:43.461 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.002615774 2024-09-10T19:16:43.462 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:43.462 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:43.462 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-10T19:16:43.462 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:16:43.434981+0000", 2024-09-10T19:16:43.462 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00011764000000000001 2024-09-10T19:16:43.462 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:43.462 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:43.462 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0,5", 2024-09-10T19:16:43.462 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:16:43.435125+0000", 2024-09-10T19:16:43.462 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00014410900000000001 2024-09-10T19:16:43.462 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:43.462 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:43.462 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-09-10T19:16:43.462 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:16:43.436767+0000", 2024-09-10T19:16:43.462 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.001642499 2024-09-10T19:16:43.463 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:43.463 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:43.463 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:16:43.463 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:16:43.440033+0000", 2024-09-10T19:16:43.463 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0032659289999999999 2024-09-10T19:16:43.463 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:16:43.463 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-10T19:16:43.463 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:16:43.463 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:16:43.463 INFO:teuthology.orchestra.run.smithi191.stdout: ], 2024-09-10T19:16:43.463 INFO:teuthology.orchestra.run.smithi191.stdout: "num_ops": 3 2024-09-10T19:16:43.463 INFO:teuthology.orchestra.run.smithi191.stdout:} 2024-09-10T19:16:43.509 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:16:43.550 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:16:43.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:16:43.551 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:16:43.652 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:16:43.689 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:16:43.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:16:43.690 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:16:43.791 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:16:43.824 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: 19 15 42224 42209 138.818 106.375 0.0134642 0.00719826 2024-09-10T19:16:43.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:16:43.827 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:16:43.827 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:16:43.919 INFO:teuthology.orchestra.run.smithi183.stdout:osd.0: {} 2024-09-10T19:16:43.919 INFO:teuthology.orchestra.run.smithi183.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-10T19:16:43.928 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:16:43.940 INFO:teuthology.orchestra.run.smithi183.stderr:noscrub is set 2024-09-10T19:16:43.945 INFO:teuthology.orchestra.run.smithi183.stdout:osd.1: {} 2024-09-10T19:16:43.945 INFO:teuthology.orchestra.run.smithi183.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-10T19:16:43.964 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:16:43.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:16:43.965 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:16:43.965 INFO:teuthology.orchestra.run.smithi183.stdout:osd.2: {} 2024-09-10T19:16:43.966 INFO:teuthology.orchestra.run.smithi183.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-10T19:16:43.993 INFO:teuthology.orchestra.run.smithi183.stdout:osd.3: {} 2024-09-10T19:16:43.994 INFO:teuthology.orchestra.run.smithi183.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-10T19:16:44.017 INFO:teuthology.orchestra.run.smithi183.stdout:osd.4: {} 2024-09-10T19:16:44.018 INFO:teuthology.orchestra.run.smithi183.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-10T19:16:44.046 INFO:teuthology.orchestra.run.smithi183.stdout:osd.5: {} 2024-09-10T19:16:44.047 INFO:teuthology.orchestra.run.smithi183.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-10T19:16:44.047 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:16:43 smithi183 ceph-mon[41562]: from='client.? 172.21.15.183:0/2027452464' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-10T19:16:44.047 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:16:43 smithi183 ceph-mon[41562]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-10T19:16:44.047 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:16:43 smithi183 ceph-mon[41562]: from='mgr.14556 ' entity='mgr.y' 2024-09-10T19:16:44.047 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:16:43 smithi183 ceph-mon[41562]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:16:44.047 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:16:43 smithi183 ceph-mon[41562]: from='client.? 172.21.15.183:0/2130216445' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T19:16:44.048 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:16:43 smithi183 ceph-mon[29736]: from='client.? 172.21.15.183:0/2027452464' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-10T19:16:44.048 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:16:43 smithi183 ceph-mon[29736]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-10T19:16:44.048 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:16:43 smithi183 ceph-mon[29736]: from='mgr.14556 ' entity='mgr.y' 2024-09-10T19:16:44.048 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:16:43 smithi183 ceph-mon[29736]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:16:44.048 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:16:43 smithi183 ceph-mon[29736]: from='client.? 172.21.15.183:0/2130216445' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T19:16:44.065 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:16:44.072 INFO:teuthology.orchestra.run.smithi183.stdout:osd.6: {} 2024-09-10T19:16:44.072 INFO:teuthology.orchestra.run.smithi183.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-10T19:16:44.099 INFO:teuthology.orchestra.run.smithi183.stdout:osd.7: {} 2024-09-10T19:16:44.099 INFO:teuthology.orchestra.run.smithi183.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-10T19:16:44.105 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:16:44.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:16:44.106 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:16:44.187 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:16:43 smithi191 ceph-mon[32884]: from='client.? 172.21.15.183:0/2027452464' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-10T19:16:44.187 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:16:43 smithi191 ceph-mon[32884]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-10T19:16:44.187 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:16:43 smithi191 ceph-mon[32884]: from='mgr.14556 ' entity='mgr.y' 2024-09-10T19:16:44.187 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:16:43 smithi191 ceph-mon[32884]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:16:44.187 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:16:43 smithi191 ceph-mon[32884]: from='client.? 172.21.15.183:0/2130216445' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T19:16:44.207 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:16:44.232 DEBUG:teuthology.orchestra.run.smithi191:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph daemon osd.6 dump_blocked_ops 2024-09-10T19:16:44.237 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:16:44.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:16:44.238 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:16:44.338 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:16:44.367 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:16:44.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:16:44.368 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:16:44.469 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:16:44.511 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:16:44.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:16:44.512 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:16:44.613 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:16:44.639 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:16:44.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:16:44.640 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:16:44.680 INFO:teuthology.orchestra.run.smithi191.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.b/config 2024-09-10T19:16:44.741 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:16:44.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:16:44.786 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:16:44.787 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:16:44.824 INFO:tasks.radosbench.radosbench.0.smithi183.stdout:2024-09-10T19:16:44.823352+0000 min lat: 0.0011643 max lat: 0.18378 avg lat: 0.00729391 2024-09-10T19:16:44.824 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-10T19:16:44.824 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: 20 15 43862 43847 136.996 102.375 0.007084 0.00729391 2024-09-10T19:16:44.887 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:16:44.932 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:16:44.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:16:44.933 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:16:45.034 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:16:45.062 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:16:45.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:16:45.063 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:16:45.067 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:16:44 smithi191 ceph-mon[32884]: pgmap v270: 148 pgs: 148 active+clean; 2.2 GiB data, 8.1 GiB used, 528 GiB / 536 GiB avail; 1023 B/s rd, 122 MiB/s wr, 1.95k op/s 2024-09-10T19:16:45.068 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:16:44 smithi191 ceph-mon[32884]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T19:16:45.068 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:16:44 smithi191 ceph-mon[32884]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-10T19:16:45.068 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:16:44 smithi191 ceph-mon[32884]: osdmap e161: 8 total, 8 up, 6 in 2024-09-10T19:16:45.068 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:16:44 smithi191 ceph-mon[32884]: 5.6 deep-scrub starts 2024-09-10T19:16:45.068 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:16:44 smithi191 ceph-mon[32884]: 5.6 deep-scrub ok 2024-09-10T19:16:45.068 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:16:44 smithi191 ceph-mon[32884]: osdmap e162: 8 total, 8 up, 6 in 2024-09-10T19:16:45.165 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:16:45.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:16:44 smithi183 ceph-mon[29736]: pgmap v270: 148 pgs: 148 active+clean; 2.2 GiB data, 8.1 GiB used, 528 GiB / 536 GiB avail; 1023 B/s rd, 122 MiB/s wr, 1.95k op/s 2024-09-10T19:16:45.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:16:44 smithi183 ceph-mon[29736]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T19:16:45.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:16:44 smithi183 ceph-mon[29736]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-10T19:16:45.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:16:44 smithi183 ceph-mon[29736]: osdmap e161: 8 total, 8 up, 6 in 2024-09-10T19:16:45.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:16:44 smithi183 ceph-mon[29736]: 5.6 deep-scrub starts 2024-09-10T19:16:45.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:16:44 smithi183 ceph-mon[29736]: 5.6 deep-scrub ok 2024-09-10T19:16:45.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:16:44 smithi183 ceph-mon[29736]: osdmap e162: 8 total, 8 up, 6 in 2024-09-10T19:16:45.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:16:44 smithi183 ceph-mon[41562]: pgmap v270: 148 pgs: 148 active+clean; 2.2 GiB data, 8.1 GiB used, 528 GiB / 536 GiB avail; 1023 B/s rd, 122 MiB/s wr, 1.95k op/s 2024-09-10T19:16:45.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:16:44 smithi183 ceph-mon[41562]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T19:16:45.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:16:44 smithi183 ceph-mon[41562]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-10T19:16:45.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:16:44 smithi183 ceph-mon[41562]: osdmap e161: 8 total, 8 up, 6 in 2024-09-10T19:16:45.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:16:44 smithi183 ceph-mon[41562]: 5.6 deep-scrub starts 2024-09-10T19:16:45.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:16:44 smithi183 ceph-mon[41562]: 5.6 deep-scrub ok 2024-09-10T19:16:45.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:16:44 smithi183 ceph-mon[41562]: osdmap e162: 8 total, 8 up, 6 in 2024-09-10T19:16:45.195 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:16:45.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:16:45.196 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:16:45.297 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:16:45.324 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:16:45.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:16:45.326 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:16:45.426 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:16:45.456 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:16:45.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:16:45.457 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:16:45.487 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:16:45.558 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:16:45.587 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:16:45.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:16:45.588 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:16:45.689 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:16:45.719 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:16:45.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:16:45.720 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:16:45.821 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:16:45.824 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: 21 16 45559 45543 135.52 106 0.00948409 0.00737413 2024-09-10T19:16:45.852 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:16:45.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:16:45.853 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:16:45.953 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:16:45.991 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:16:45.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:16:45.992 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:16:46.093 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:16:46.121 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:16:46.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:16:46.122 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:16:46.223 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:16:46.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:16:46.265 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:16:46.265 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:16:46.366 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:16:46.398 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:16:46.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:16:46.399 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:16:46.500 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:16:46.538 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:16:46.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:16:46.539 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:16:46.640 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:16:46.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:16:46.669 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:16:46.669 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:16:46.770 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:16:46.811 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:16:46.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:16:46.811 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:16:46.824 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: 22 16 47264 47248 134.203 106.562 0.0253936 0.00744646 2024-09-10T19:16:46.912 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:16:46.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:16:46.957 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:16:46.957 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:16:47.058 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:16:47.087 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:16:47.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:16:47.089 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:16:47.128 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:16:46 smithi191 ceph-mon[32884]: pgmap v273: 148 pgs: 148 active+clean; 2.7 GiB data, 9.6 GiB used, 527 GiB / 536 GiB avail; 1.2 KiB/s rd, 126 MiB/s wr, 2.02k op/s 2024-09-10T19:16:47.129 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-10T19:16:47.156 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:16:46 smithi183 ceph-mon[29736]: pgmap v273: 148 pgs: 148 active+clean; 2.7 GiB data, 9.6 GiB used, 527 GiB / 536 GiB avail; 1.2 KiB/s rd, 126 MiB/s wr, 2.02k op/s 2024-09-10T19:16:47.156 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:16:46 smithi183 ceph-mon[41562]: pgmap v273: 148 pgs: 148 active+clean; 2.7 GiB data, 9.6 GiB used, 527 GiB / 536 GiB avail; 1.2 KiB/s rd, 126 MiB/s wr, 2.02k op/s 2024-09-10T19:16:47.189 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:16:47.225 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:16:47.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:16:47.226 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:16:47.287 INFO:teuthology.orchestra.run.smithi191.stdout:ERROR: (22) Invalid argument 2024-09-10T19:16:47.287 INFO:teuthology.orchestra.run.smithi191.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-10T19:16:47.327 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:16:47.365 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:16:47.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:16:47.366 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:16:47.466 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-10T19:16:47.467 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:16:47.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:16:47.502 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:16:47.502 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:16:47.603 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:16:47.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:16:47.640 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:16:47.640 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:16:47.650 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:16:47.715 INFO:journalctl@ceph.mgr.x.smithi191.stdout:Sep 10 19:16:47 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-x[34967]: ::ffff:172.21.15.191 - - [10/Sep/2024:19:16:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T19:16:47.742 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:16:47.778 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:16:47.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:16:47.779 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:16:47.824 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: 23 16 48863 48847 132.713 99.9375 0.0130039 0.00752992 2024-09-10T19:16:47.857 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-09-10T19:16:47.860 INFO:teuthology.orchestra.run.smithi183.stderr:dumped all 2024-09-10T19:16:47.880 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:16:47.914 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:16:47.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:16:47.915 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:16:48.015 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:16:48.037 DEBUG:teuthology.orchestra.run.smithi191:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph daemon osd.6 dump_historic_ops 2024-09-10T19:16:48.042 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:16:48.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:16:48.043 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:16:48.143 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:16:48.169 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:16:48.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:16:48.170 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:16:48.181 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:16:47 smithi183 ceph-mon[41562]: purged_snaps scrub starts 2024-09-10T19:16:48.181 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:16:47 smithi183 ceph-mon[41562]: purged_snaps scrub ok 2024-09-10T19:16:48.181 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:16:47 smithi183 ceph-mon[41562]: osdmap e163: 8 total, 8 up, 6 in 2024-09-10T19:16:48.182 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:16:47 smithi183 ceph-mon[29736]: purged_snaps scrub starts 2024-09-10T19:16:48.182 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:16:47 smithi183 ceph-mon[29736]: purged_snaps scrub ok 2024-09-10T19:16:48.182 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:16:47 smithi183 ceph-mon[29736]: osdmap e163: 8 total, 8 up, 6 in 2024-09-10T19:16:48.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:16:47 smithi191 ceph-mon[32884]: purged_snaps scrub starts 2024-09-10T19:16:48.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:16:47 smithi191 ceph-mon[32884]: purged_snaps scrub ok 2024-09-10T19:16:48.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:16:47 smithi191 ceph-mon[32884]: osdmap e163: 8 total, 8 up, 6 in 2024-09-10T19:16:48.271 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:16:48.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:16:48.314 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:16:48.314 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:16:48.415 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:16:48.442 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:16:48.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:16:48.443 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:16:48.495 INFO:teuthology.orchestra.run.smithi191.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.b/config 2024-09-10T19:16:48.543 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:16:48.571 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:16:48.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:16:48.572 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:16:48.673 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:16:48.715 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:16:48.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:16:48.716 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:16:48.817 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:16:48.824 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: 24 16 50387 50371 131.152 95.25 0.00935871 0.00761973 2024-09-10T19:16:48.860 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:16:48.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:16:48.861 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:16:48.962 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:16:48.990 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:16:48.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:16:48.992 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:16:49.092 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:16:49.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:16:49.127 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:16:49.127 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:16:49.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:16:48 smithi191 ceph-mon[32884]: pgmap v275: 148 pgs: 148 active+clean; 2.7 GiB data, 9.6 GiB used, 527 GiB / 536 GiB avail; 853 B/s rd, 89 MiB/s wr, 1.43k op/s 2024-09-10T19:16:49.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:16:48 smithi191 ceph-mon[32884]: from='client.15090 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-10T19:16:49.216 INFO:journalctl@ceph.iscsi.iscsi.a.smithi191.stdout:Sep 10 19:16:49 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-iscsi-iscsi-a[61816]: debug there is no tcmu-runner data available 2024-09-10T19:16:49.228 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:16:49.263 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:16:48 smithi183 ceph-mon[29736]: pgmap v275: 148 pgs: 148 active+clean; 2.7 GiB data, 9.6 GiB used, 527 GiB / 536 GiB avail; 853 B/s rd, 89 MiB/s wr, 1.43k op/s 2024-09-10T19:16:49.263 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:16:48 smithi183 ceph-mon[29736]: from='client.15090 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-10T19:16:49.264 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:16:48 smithi183 ceph-mon[41562]: pgmap v275: 148 pgs: 148 active+clean; 2.7 GiB data, 9.6 GiB used, 527 GiB / 536 GiB avail; 853 B/s rd, 89 MiB/s wr, 1.43k op/s 2024-09-10T19:16:49.264 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:16:48 smithi183 ceph-mon[41562]: from='client.15090 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-10T19:16:49.265 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:16:49.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:16:49.266 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:16:49.367 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:16:49.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:16:49.404 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:16:49.404 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:16:49.505 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:16:49.546 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:16:49.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:16:49.547 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:16:49.647 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:16:49.676 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:16:49.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:16:49.677 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:16:49.736 INFO:teuthology.orchestra.run.smithi183.stdout:{"pg_ready":true,"pg_map":{"version":275,"stamp":"2024-09-10T19:16:47.726961+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":2878675164,"num_objects":44112,"num_object_clones":0,"num_object_copies":132336,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":44112,"num_whiteouts":0,"num_read":874,"num_read_kb":731,"num_write":44459,"num_write_kb":2812295,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":182,"num_bytes_recovered":594357,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":44529,"ondisk_log_size":44529,"up":444,"acting":444,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":444,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":562470912,"kb_used":10112780,"kb_used_data":8295364,"kb_used_omap":9,"kb_used_meta":1817334,"kb_avail":552358132,"statfs":{"total":575970213888,"available":565614727168,"internally_reserved":0,"allocated":8494452736,"data_stored":8491926286,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":10224,"internal_metadata":1860950032},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,2,7,7,3],"upper_bound":32},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":563019776,"num_objects":8591,"num_object_clones":0,"num_object_copies":25773,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8591,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":8591,"num_write_kb":549824,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001556"},"pg_stats":[{"pgid":"6.1b","version":"0'0","reported_seq":126,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-10T19:16:43.941727+0000","last_change":"2024-09-10T19:13:08.710758+0000","last_active":"2024-09-10T19:16:43.941727+0000","last_peered":"2024-09-10T19:16:43.941727+0000","last_clean":"2024-09-10T19:16:43.941727+0000","last_became_active":"2024-09-10T19:10:14.056187+0000","last_became_peered":"2024-09-10T19:10:14.056187+0000","last_unstale":"2024-09-10T19:16:43.941727+0000","last_undegraded":"2024-09-10T19:16:43.941727+0000","last_fullsized":"2024-09-10T19:16:43.941727+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:13:08.710679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:13:08.710679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0027252499999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":162,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-10T19:16:43.947144+0000","last_change":"2024-09-10T19:16:38.194521+0000","last_active":"2024-09-10T19:16:43.947144+0000","last_peered":"2024-09-10T19:16:43.947144+0000","last_clean":"2024-09-10T19:16:43.947144+0000","last_became_active":"2024-09-10T19:16:24.836961+0000","last_became_peered":"2024-09-10T19:16:24.836961+0000","last_unstale":"2024-09-10T19:16:43.947144+0000","last_undegraded":"2024-09-10T19:16:43.947144+0000","last_fullsized":"2024-09-10T19:16:43.947144+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:16:38.194472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:16:38.194472+0000","last_clean_scrub_stamp":"2024-09-10T19:16:38.194472+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:17:46.775869+0000","scrub_duration":0.0023678900000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.19","version":"66'15","reported_seq":181,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-10T19:16:43.939077+0000","last_change":"2024-09-10T19:16:24.879212+0000","last_active":"2024-09-10T19:16:43.939077+0000","last_peered":"2024-09-10T19:16:43.939077+0000","last_clean":"2024-09-10T19:16:43.939077+0000","last_became_active":"2024-09-10T19:16:24.844951+0000","last_became_peered":"2024-09-10T19:16:24.844951+0000","last_unstale":"2024-09-10T19:16:43.939077+0000","last_undegraded":"2024-09-10T19:16:43.939077+0000","last_fullsized":"2024-09-10T19:16:43.939077+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'15","last_scrub_stamp":"2024-09-10T19:13:47.165848+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:13:47.165848+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.004982841,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":156,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-10T19:16:34.760923+0000","last_change":"2024-09-10T19:16:24.845473+0000","last_active":"2024-09-10T19:16:34.760923+0000","last_peered":"2024-09-10T19:16:34.760923+0000","last_clean":"2024-09-10T19:16:34.760923+0000","last_became_active":"2024-09-10T19:16:24.845111+0000","last_became_peered":"2024-09-10T19:16:24.845111+0000","last_unstale":"2024-09-10T19:16:34.760923+0000","last_undegraded":"2024-09-10T19:16:34.760923+0000","last_fullsized":"2024-09-10T19:16:34.760923+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:15:33.976696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:15:33.976696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0015310580000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,7],"acting":[4,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"59'1","reported_seq":150,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-10T19:16:43.939863+0000","last_change":"2024-09-10T19:16:30.798883+0000","last_active":"2024-09-10T19:16:43.939863+0000","last_peered":"2024-09-10T19:16:43.939863+0000","last_clean":"2024-09-10T19:16:43.939863+0000","last_became_active":"2024-09-10T19:10:07.686143+0000","last_became_peered":"2024-09-10T19:10:07.686143+0000","last_unstale":"2024-09-10T19:16:43.939863+0000","last_undegraded":"2024-09-10T19:16:43.939863+0000","last_fullsized":"2024-09-10T19:16:43.939863+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'1","last_scrub_stamp":"2024-09-10T19:16:30.798795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:16:30.798795+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:17:44.600087+0000","scrub_duration":0.0067002989999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":874,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"66'9","reported_seq":171,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-10T19:16:34.761006+0000","last_change":"2024-09-10T19:14:55.117473+0000","last_active":"2024-09-10T19:16:34.761006+0000","last_peered":"2024-09-10T19:16:34.761006+0000","last_clean":"2024-09-10T19:16:34.761006+0000","last_became_active":"2024-09-10T19:13:54.716117+0000","last_became_peered":"2024-09-10T19:13:54.716117+0000","last_unstale":"2024-09-10T19:16:34.761006+0000","last_undegraded":"2024-09-10T19:16:34.761006+0000","last_fullsized":"2024-09-10T19:16:34.761006+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'9","last_scrub_stamp":"2024-09-10T19:14:55.117387+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:14:55.117387+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0074980940000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,3],"acting":[4,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":102,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-10T19:16:43.948412+0000","last_change":"2024-09-10T19:16:27.753997+0000","last_active":"2024-09-10T19:16:43.948412+0000","last_peered":"2024-09-10T19:16:43.948412+0000","last_clean":"2024-09-10T19:16:43.948412+0000","last_became_active":"2024-09-10T19:16:24.859261+0000","last_became_peered":"2024-09-10T19:16:24.859261+0000","last_unstale":"2024-09-10T19:16:43.948412+0000","last_undegraded":"2024-09-10T19:16:43.948412+0000","last_fullsized":"2024-09-10T19:16:43.948412+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:16:27.753912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:16:27.753912+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:17:49.796558+0000","scrub_duration":0.0029373799999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":144,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-10T19:16:34.761210+0000","last_change":"2024-09-10T19:15:39.910169+0000","last_active":"2024-09-10T19:16:34.761210+0000","last_peered":"2024-09-10T19:16:34.761210+0000","last_clean":"2024-09-10T19:16:34.761210+0000","last_became_active":"2024-09-10T19:13:54.713538+0000","last_became_peered":"2024-09-10T19:13:54.713538+0000","last_unstale":"2024-09-10T19:16:34.761210+0000","last_undegraded":"2024-09-10T19:16:34.761210+0000","last_fullsized":"2024-09-10T19:16:34.761210+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:15:39.910118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:15:39.910118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:16:46.983414+0000","scrub_duration":0.0013551839999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,7],"acting":[4,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":141,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-10T19:16:34.797939+0000","last_change":"2024-09-10T19:16:25.595265+0000","last_active":"2024-09-10T19:16:34.797939+0000","last_peered":"2024-09-10T19:16:34.797939+0000","last_clean":"2024-09-10T19:16:34.797939+0000","last_became_active":"2024-09-10T19:10:07.690728+0000","last_became_peered":"2024-09-10T19:10:07.690728+0000","last_unstale":"2024-09-10T19:16:34.797939+0000","last_undegraded":"2024-09-10T19:16:34.797939+0000","last_fullsized":"2024-09-10T19:16:34.797939+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:16:25.595127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:16:25.595127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:17:51.885390+0000","scrub_duration":0.003922633,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1b","version":"66'5","reported_seq":180,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-10T19:16:35.966509+0000","last_change":"2024-09-10T19:16:35.966457+0000","last_active":"2024-09-10T19:16:35.966509+0000","last_peered":"2024-09-10T19:16:35.966509+0000","last_clean":"2024-09-10T19:16:35.966509+0000","last_became_active":"2024-09-10T19:16:24.843708+0000","last_became_peered":"2024-09-10T19:16:24.843708+0000","last_unstale":"2024-09-10T19:16:35.966509+0000","last_undegraded":"2024-09-10T19:16:35.966509+0000","last_fullsized":"2024-09-10T19:16:35.966509+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'5","last_scrub_stamp":"2024-09-10T19:16:35.966402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:16:35.966402+0000","objects_scrubbed":2,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:17:45.645005+0000","scrub_duration":0.0019325740000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":144,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-10T19:16:34.796601+0000","last_change":"2024-09-10T19:15:33.518718+0000","last_active":"2024-09-10T19:16:34.796601+0000","last_peered":"2024-09-10T19:16:34.796601+0000","last_clean":"2024-09-10T19:16:34.796601+0000","last_became_active":"2024-09-10T19:13:54.719063+0000","last_became_peered":"2024-09-10T19:13:54.719063+0000","last_unstale":"2024-09-10T19:16:34.796601+0000","last_undegraded":"2024-09-10T19:16:34.796601+0000","last_fullsized":"2024-09-10T19:16:34.796601+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:15:33.518601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:15:33.518601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:16:52.587150+0000","scrub_duration":0.0013951650000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":153,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-10T19:16:43.948542+0000","last_change":"2024-09-10T19:16:24.845721+0000","last_active":"2024-09-10T19:16:43.948542+0000","last_peered":"2024-09-10T19:16:43.948542+0000","last_clean":"2024-09-10T19:16:43.948542+0000","last_became_active":"2024-09-10T19:16:24.845130+0000","last_became_peered":"2024-09-10T19:16:24.845130+0000","last_unstale":"2024-09-10T19:16:43.948542+0000","last_undegraded":"2024-09-10T19:16:43.948542+0000","last_fullsized":"2024-09-10T19:16:43.948542+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:15:45.616184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:15:45.616184+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:17:13.888995+0000","scrub_duration":0.0013989510000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,0],"acting":[5,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":141,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-10T19:16:34.797892+0000","last_change":"2024-09-10T19:16:24.859112+0000","last_active":"2024-09-10T19:16:34.797892+0000","last_peered":"2024-09-10T19:16:34.797892+0000","last_clean":"2024-09-10T19:16:34.797892+0000","last_became_active":"2024-09-10T19:16:24.858800+0000","last_became_peered":"2024-09-10T19:16:24.858800+0000","last_unstale":"2024-09-10T19:16:34.797892+0000","last_undegraded":"2024-09-10T19:16:34.797892+0000","last_fullsized":"2024-09-10T19:16:34.797892+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:12:54.477149+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:12:54.477149+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0039768319999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1a","version":"66'9","reported_seq":144,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-10T19:16:34.761635+0000","last_change":"2024-09-10T19:13:02.431528+0000","last_active":"2024-09-10T19:16:34.761635+0000","last_peered":"2024-09-10T19:16:34.761635+0000","last_clean":"2024-09-10T19:16:34.761635+0000","last_became_active":"2024-09-10T19:10:09.688987+0000","last_became_peered":"2024-09-10T19:10:09.688987+0000","last_unstale":"2024-09-10T19:16:34.761635+0000","last_undegraded":"2024-09-10T19:16:34.761635+0000","last_fullsized":"2024-09-10T19:16:34.761635+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'9","last_scrub_stamp":"2024-09-10T19:13:02.431443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:13:02.431443+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0084913290000000006,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":138,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-10T19:16:43.948862+0000","last_change":"2024-09-10T19:16:37.116464+0000","last_active":"2024-09-10T19:16:43.948862+0000","last_peered":"2024-09-10T19:16:43.948862+0000","last_clean":"2024-09-10T19:16:43.948862+0000","last_became_active":"2024-09-10T19:10:11.701799+0000","last_became_peered":"2024-09-10T19:10:11.701799+0000","last_unstale":"2024-09-10T19:16:43.948862+0000","last_undegraded":"2024-09-10T19:16:43.948862+0000","last_fullsized":"2024-09-10T19:16:43.948862+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:16:37.116371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:16:37.116371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:18:03.024881+0000","scrub_duration":0.028100301000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":153,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-10T19:16:43.945302+0000","last_change":"2024-09-10T19:16:24.836814+0000","last_active":"2024-09-10T19:16:43.945302+0000","last_peered":"2024-09-10T19:16:43.945302+0000","last_clean":"2024-09-10T19:16:43.945302+0000","last_became_active":"2024-09-10T19:16:24.836477+0000","last_became_peered":"2024-09-10T19:16:24.836477+0000","last_unstale":"2024-09-10T19:16:43.945302+0000","last_undegraded":"2024-09-10T19:16:43.945302+0000","last_fullsized":"2024-09-10T19:16:43.945302+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:15:33.839248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:15:33.839248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0032019980000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1","version":"162'2732","reported_seq":2764,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-09-10T19:16:44.972762+0000","last_change":"2024-09-10T19:16:24.842553+0000","last_active":"2024-09-10T19:16:44.972762+0000","last_peered":"2024-09-10T19:16:44.972762+0000","last_clean":"2024-09-10T19:16:44.972762+0000","last_became_active":"2024-09-10T19:16:24.842218+0000","last_became_peered":"2024-09-10T19:16:24.842218+0000","last_unstale":"2024-09-10T19:16:44.972762+0000","last_undegraded":"2024-09-10T19:16:44.972762+0000","last_fullsized":"2024-09-10T19:16:44.972762+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:16:11.777014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:16:11.777014+0000","last_clean_scrub_stamp":"2024-09-10T19:16:11.777014+0000","objects_scrubbed":0,"log_size":2732,"log_dups_size":0,"ondisk_log_size":2732,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:17:31.601183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":179044352,"num_objects":2732,"num_object_clones":0,"num_object_copies":8196,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2732,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2732,"num_write_kb":174848,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.d","version":"66'17","reported_seq":229,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-10T19:16:34.791549+0000","last_change":"2024-09-10T19:16:32.999033+0000","last_active":"2024-09-10T19:16:34.791549+0000","last_peered":"2024-09-10T19:16:34.791549+0000","last_clean":"2024-09-10T19:16:34.791549+0000","last_became_active":"2024-09-10T19:16:24.847056+0000","last_became_peered":"2024-09-10T19:16:24.847056+0000","last_unstale":"2024-09-10T19:16:34.791549+0000","last_undegraded":"2024-09-10T19:16:34.791549+0000","last_fullsized":"2024-09-10T19:16:34.791549+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'17","last_scrub_stamp":"2024-09-10T19:16:32.998950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:16:32.998950+0000","objects_scrubbed":8,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:17:37.707354+0000","scrub_duration":0.0095505250000000007,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,0],"acting":[4,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":141,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-10T19:16:39.716938+0000","last_change":"2024-09-10T19:16:39.716870+0000","last_active":"2024-09-10T19:16:39.716938+0000","last_peered":"2024-09-10T19:16:39.716938+0000","last_clean":"2024-09-10T19:16:39.716938+0000","last_became_active":"2024-09-10T19:10:07.691119+0000","last_became_peered":"2024-09-10T19:10:07.691119+0000","last_unstale":"2024-09-10T19:16:39.716938+0000","last_undegraded":"2024-09-10T19:16:39.716938+0000","last_fullsized":"2024-09-10T19:16:39.716938+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:16:39.716674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:16:39.716674+0000","last_clean_scrub_stamp":"2024-09-10T19:16:39.716674+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:17:54.073755+0000","scrub_duration":0.043895318000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":94,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-10T19:16:34.761524+0000","last_change":"2024-09-10T19:16:26.999105+0000","last_active":"2024-09-10T19:16:34.761524+0000","last_peered":"2024-09-10T19:16:34.761524+0000","last_clean":"2024-09-10T19:16:34.761524+0000","last_became_active":"2024-09-10T19:13:54.712883+0000","last_became_peered":"2024-09-10T19:13:54.712883+0000","last_unstale":"2024-09-10T19:16:34.761524+0000","last_undegraded":"2024-09-10T19:16:34.761524+0000","last_fullsized":"2024-09-10T19:16:34.761524+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:16:26.999036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:16:26.999036+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:17:47.793977+0000","scrub_duration":0.0050193119999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":20,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-10T19:16:43.940359+0000","last_change":"2024-09-10T19:16:24.858656+0000","last_active":"2024-09-10T19:16:43.940359+0000","last_peered":"2024-09-10T19:16:43.940359+0000","last_clean":"2024-09-10T19:16:43.940359+0000","last_became_active":"2024-09-10T19:16:24.857843+0000","last_became_peered":"2024-09-10T19:16:24.857843+0000","last_unstale":"2024-09-10T19:16:43.940359+0000","last_undegraded":"2024-09-10T19:16:43.940359+0000","last_fullsized":"2024-09-10T19:16:43.940359+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:15:47.818381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:15:47.818381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:17:06.586846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.0","version":"162'2768","reported_seq":2790,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-09-10T19:16:44.894628+0000","last_change":"2024-09-10T19:16:12.795560+0000","last_active":"2024-09-10T19:16:44.894628+0000","last_peered":"2024-09-10T19:16:44.894628+0000","last_clean":"2024-09-10T19:16:44.894628+0000","last_became_active":"2024-09-10T19:16:12.795331+0000","last_became_peered":"2024-09-10T19:16:12.795331+0000","last_unstale":"2024-09-10T19:16:44.894628+0000","last_undegraded":"2024-09-10T19:16:44.894628+0000","last_fullsized":"2024-09-10T19:16:44.894628+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:16:11.777014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:16:11.777014+0000","last_clean_scrub_stamp":"2024-09-10T19:16:11.777014+0000","objects_scrubbed":0,"log_size":2768,"log_dups_size":0,"ondisk_log_size":2768,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:17:33.649961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":181403648,"num_objects":2768,"num_object_clones":0,"num_object_copies":8304,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2768,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2768,"num_write_kb":177152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,4],"acting":[3,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"66'10","reported_seq":153,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-10T19:16:34.761741+0000","last_change":"2024-09-10T19:13:51.416075+0000","last_active":"2024-09-10T19:16:34.761741+0000","last_peered":"2024-09-10T19:16:34.761741+0000","last_clean":"2024-09-10T19:16:34.761741+0000","last_became_active":"2024-09-10T19:10:09.688321+0000","last_became_peered":"2024-09-10T19:10:09.688321+0000","last_unstale":"2024-09-10T19:16:34.761741+0000","last_undegraded":"2024-09-10T19:16:34.761741+0000","last_fullsized":"2024-09-10T19:16:34.761741+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'10","last_scrub_stamp":"2024-09-10T19:13:51.415998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:13:51.415998+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.005972799,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":92,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-10T19:16:43.939243+0000","last_change":"2024-09-10T19:15:42.119912+0000","last_active":"2024-09-10T19:16:43.939243+0000","last_peered":"2024-09-10T19:16:43.939243+0000","last_clean":"2024-09-10T19:16:43.939243+0000","last_became_active":"2024-09-10T19:13:54.731134+0000","last_became_peered":"2024-09-10T19:13:54.731134+0000","last_unstale":"2024-09-10T19:16:43.939243+0000","last_undegraded":"2024-09-10T19:16:43.939243+0000","last_fullsized":"2024-09-10T19:16:43.939243+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:15:42.119590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:15:42.119590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:16:46.442054+0000","scrub_duration":0.018001347000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,5],"acting":[3,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"155'10","reported_seq":93,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-10T19:16:40.824165+0000","last_change":"2024-09-10T19:16:29.626806+0000","last_active":"2024-09-10T19:16:40.824165+0000","last_peered":"2024-09-10T19:16:40.824165+0000","last_clean":"2024-09-10T19:16:40.824165+0000","last_became_active":"2024-09-10T19:16:24.851236+0000","last_became_peered":"2024-09-10T19:16:24.851236+0000","last_unstale":"2024-09-10T19:16:40.824165+0000","last_undegraded":"2024-09-10T19:16:40.824165+0000","last_fullsized":"2024-09-10T19:16:40.824165+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"155'10","last_scrub_stamp":"2024-09-10T19:16:29.626572+0000","last_deep_scrub":"80'9","last_deep_scrub_stamp":"2024-09-10T19:11:25.156465+0000","last_clean_scrub_stamp":"2024-09-10T19:16:29.626572+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:17:47.526795+0000","scrub_duration":0.0067064810000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,0],"acting":[7,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":157,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-10T19:16:34.791697+0000","last_change":"2024-09-10T19:16:24.852100+0000","last_active":"2024-09-10T19:16:34.791697+0000","last_peered":"2024-09-10T19:16:34.791697+0000","last_clean":"2024-09-10T19:16:34.791697+0000","last_became_active":"2024-09-10T19:16:24.851829+0000","last_became_peered":"2024-09-10T19:16:24.851829+0000","last_unstale":"2024-09-10T19:16:34.791697+0000","last_undegraded":"2024-09-10T19:16:34.791697+0000","last_fullsized":"2024-09-10T19:16:34.791697+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:15:29.988976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:15:29.988976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:16:48.442746+0000","scrub_duration":0.0053787150000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,7],"acting":[4,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.3","version":"162'2882","reported_seq":2914,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-09-10T19:16:44.795668+0000","last_change":"2024-09-10T19:16:24.850732+0000","last_active":"2024-09-10T19:16:44.795668+0000","last_peered":"2024-09-10T19:16:44.795668+0000","last_clean":"2024-09-10T19:16:44.795668+0000","last_became_active":"2024-09-10T19:16:24.850396+0000","last_became_peered":"2024-09-10T19:16:24.850396+0000","last_unstale":"2024-09-10T19:16:44.795668+0000","last_undegraded":"2024-09-10T19:16:44.795668+0000","last_fullsized":"2024-09-10T19:16:44.795668+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:16:11.777014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:16:11.777014+0000","last_clean_scrub_stamp":"2024-09-10T19:16:11.777014+0000","objects_scrubbed":0,"log_size":2882,"log_dups_size":0,"ondisk_log_size":2882,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:17:38.720329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":188874752,"num_objects":2882,"num_object_clones":0,"num_object_copies":8646,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2882,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2882,"num_write_kb":184448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,3],"acting":[5,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.f","version":"66'15","reported_seq":145,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-10T19:16:43.940547+0000","last_change":"2024-09-10T19:16:25.769523+0000","last_active":"2024-09-10T19:16:43.940547+0000","last_peered":"2024-09-10T19:16:43.940547+0000","last_clean":"2024-09-10T19:16:43.940547+0000","last_became_active":"2024-09-10T19:13:54.711634+0000","last_became_peered":"2024-09-10T19:13:54.711634+0000","last_unstale":"2024-09-10T19:16:43.940547+0000","last_undegraded":"2024-09-10T19:16:43.940547+0000","last_fullsized":"2024-09-10T19:16:43.940547+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'15","last_scrub_stamp":"2024-09-10T19:16:25.769436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:16:25.769436+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:17:46.035940+0000","scrub_duration":0.0046254920000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":101,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-10T19:16:34.796263+0000","last_change":"2024-09-10T19:16:24.857467+0000","last_active":"2024-09-10T19:16:34.796263+0000","last_peered":"2024-09-10T19:16:34.796263+0000","last_clean":"2024-09-10T19:16:34.796263+0000","last_became_active":"2024-09-10T19:16:24.857179+0000","last_became_peered":"2024-09-10T19:16:24.857179+0000","last_unstale":"2024-09-10T19:16:34.796263+0000","last_undegraded":"2024-09-10T19:16:34.796263+0000","last_fullsized":"2024-09-10T19:16:34.796263+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:15:29.553448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:15:29.553448+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:16:44.078407+0000","scrub_duration":0.0062080679999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,6],"acting":[7,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.e","version":"66'8","reported_seq":213,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-10T19:16:40.822425+0000","last_change":"2024-09-10T19:13:40.475460+0000","last_active":"2024-09-10T19:16:40.822425+0000","last_peered":"2024-09-10T19:16:40.822425+0000","last_clean":"2024-09-10T19:16:40.822425+0000","last_became_active":"2024-09-10T19:10:11.699232+0000","last_became_peered":"2024-09-10T19:10:11.699232+0000","last_unstale":"2024-09-10T19:16:40.822425+0000","last_undegraded":"2024-09-10T19:16:40.822425+0000","last_fullsized":"2024-09-10T19:16:40.822425+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'8","last_scrub_stamp":"2024-09-10T19:13:40.475375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:13:40.475375+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.035806152000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":153,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-10T19:16:43.950224+0000","last_change":"2024-09-10T19:16:24.855938+0000","last_active":"2024-09-10T19:16:43.950224+0000","last_peered":"2024-09-10T19:16:43.950224+0000","last_clean":"2024-09-10T19:16:43.950224+0000","last_became_active":"2024-09-10T19:16:24.855627+0000","last_became_peered":"2024-09-10T19:16:24.855627+0000","last_unstale":"2024-09-10T19:16:43.950224+0000","last_undegraded":"2024-09-10T19:16:43.950224+0000","last_fullsized":"2024-09-10T19:16:43.950224+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:15:44.633043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:15:44.633043+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:17:08.098975+0000","scrub_duration":0.00368805,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,6],"acting":[5,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.2","version":"161'2731","reported_seq":2752,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-10T19:16:44.703020+0000","last_change":"2024-09-10T19:16:12.794918+0000","last_active":"2024-09-10T19:16:44.703020+0000","last_peered":"2024-09-10T19:16:44.703020+0000","last_clean":"2024-09-10T19:16:44.703020+0000","last_became_active":"2024-09-10T19:16:12.794134+0000","last_became_peered":"2024-09-10T19:16:12.794134+0000","last_unstale":"2024-09-10T19:16:44.703020+0000","last_undegraded":"2024-09-10T19:16:44.703020+0000","last_fullsized":"2024-09-10T19:16:44.703020+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:16:11.777014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:16:11.777014+0000","last_clean_scrub_stamp":"2024-09-10T19:16:11.777014+0000","objects_scrubbed":0,"log_size":2731,"log_dups_size":0,"ondisk_log_size":2731,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:17:40.992082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":178978816,"num_objects":2731,"num_object_clones":0,"num_object_copies":8193,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2731,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2731,"num_write_kb":174784,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,5],"acting":[4,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"84'16","reported_seq":184,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-10T19:16:34.791427+0000","last_change":"2024-09-10T19:15:47.914487+0000","last_active":"2024-09-10T19:16:34.791427+0000","last_peered":"2024-09-10T19:16:34.791427+0000","last_clean":"2024-09-10T19:16:34.791427+0000","last_became_active":"2024-09-10T19:13:54.717022+0000","last_became_peered":"2024-09-10T19:13:54.717022+0000","last_unstale":"2024-09-10T19:16:34.791427+0000","last_undegraded":"2024-09-10T19:16:34.791427+0000","last_fullsized":"2024-09-10T19:16:34.791427+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'16","last_scrub_stamp":"2024-09-10T19:15:47.914420+0000","last_deep_scrub":"84'16","last_deep_scrub_stamp":"2024-09-10T19:11:56.157975+0000","last_clean_scrub_stamp":"2024-09-10T19:15:47.914420+0000","objects_scrubbed":5,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:16:48.311799+0000","scrub_duration":0.0019153390000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,3],"acting":[4,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":148,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-10T19:16:34.796684+0000","last_change":"2024-09-10T19:15:42.450149+0000","last_active":"2024-09-10T19:16:34.796684+0000","last_peered":"2024-09-10T19:16:34.796684+0000","last_clean":"2024-09-10T19:16:34.796684+0000","last_became_active":"2024-09-10T19:13:54.720293+0000","last_became_peered":"2024-09-10T19:13:54.720293+0000","last_unstale":"2024-09-10T19:16:34.796684+0000","last_undegraded":"2024-09-10T19:16:34.796684+0000","last_fullsized":"2024-09-10T19:16:34.796684+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:15:42.450068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:11:53.066849+0000","last_clean_scrub_stamp":"2024-09-10T19:15:42.450068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:16:57.302037+0000","scrub_duration":0.0037708360000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":128,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-10T19:16:43.949377+0000","last_change":"2024-09-10T19:13:06.358188+0000","last_active":"2024-09-10T19:16:43.949377+0000","last_peered":"2024-09-10T19:16:43.949377+0000","last_clean":"2024-09-10T19:16:43.949377+0000","last_became_active":"2024-09-10T19:10:11.692215+0000","last_became_peered":"2024-09-10T19:10:11.692215+0000","last_unstale":"2024-09-10T19:16:43.949377+0000","last_undegraded":"2024-09-10T19:16:43.949377+0000","last_fullsized":"2024-09-10T19:16:43.949377+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:13:06.358109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:13:06.358109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0026724549999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":126,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-10T19:16:43.938815+0000","last_change":"2024-09-10T19:13:54.304493+0000","last_active":"2024-09-10T19:16:43.938815+0000","last_peered":"2024-09-10T19:16:43.938815+0000","last_clean":"2024-09-10T19:16:43.938815+0000","last_became_active":"2024-09-10T19:10:14.053736+0000","last_became_peered":"2024-09-10T19:10:14.053736+0000","last_unstale":"2024-09-10T19:16:43.938815+0000","last_undegraded":"2024-09-10T19:16:43.938815+0000","last_fullsized":"2024-09-10T19:16:43.938815+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:13:54.304435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:13:54.304435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0011043820000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.d","version":"162'2755","reported_seq":2787,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-09-10T19:16:44.771406+0000","last_change":"2024-09-10T19:16:24.858800+0000","last_active":"2024-09-10T19:16:44.771406+0000","last_peered":"2024-09-10T19:16:44.771406+0000","last_clean":"2024-09-10T19:16:44.771406+0000","last_became_active":"2024-09-10T19:16:24.858460+0000","last_became_peered":"2024-09-10T19:16:24.858460+0000","last_unstale":"2024-09-10T19:16:44.771406+0000","last_undegraded":"2024-09-10T19:16:44.771406+0000","last_fullsized":"2024-09-10T19:16:44.771406+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:16:11.777014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:16:11.777014+0000","last_clean_scrub_stamp":"2024-09-10T19:16:11.777014+0000","objects_scrubbed":0,"log_size":2755,"log_dups_size":0,"ondisk_log_size":2755,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:17:15.173931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180551680,"num_objects":2755,"num_object_clones":0,"num_object_copies":8265,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2755,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2755,"num_write_kb":176320,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1","version":"66'14","reported_seq":32,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-10T19:16:34.760541+0000","last_change":"2024-09-10T19:16:24.903236+0000","last_active":"2024-09-10T19:16:34.760541+0000","last_peered":"2024-09-10T19:16:34.760541+0000","last_clean":"2024-09-10T19:16:34.760541+0000","last_became_active":"2024-09-10T19:16:24.852821+0000","last_became_peered":"2024-09-10T19:16:24.852821+0000","last_unstale":"2024-09-10T19:16:34.760541+0000","last_undegraded":"2024-09-10T19:16:34.760541+0000","last_fullsized":"2024-09-10T19:16:34.760541+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'14","last_scrub_stamp":"2024-09-10T19:15:41.748101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:15:41.748101+0000","objects_scrubbed":7,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:16:47.351993+0000","scrub_duration":0.0055403099999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,7],"acting":[4,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":151,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-10T19:16:34.766298+0000","last_change":"2024-09-10T19:16:25.582072+0000","last_active":"2024-09-10T19:16:34.766298+0000","last_peered":"2024-09-10T19:16:34.766298+0000","last_clean":"2024-09-10T19:16:34.766298+0000","last_became_active":"2024-09-10T19:16:24.855861+0000","last_became_peered":"2024-09-10T19:16:24.855861+0000","last_unstale":"2024-09-10T19:16:34.766298+0000","last_undegraded":"2024-09-10T19:16:34.766298+0000","last_fullsized":"2024-09-10T19:16:34.766298+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:16:25.581993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:16:25.581993+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:17:36.903746+0000","scrub_duration":0.0041345990000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.0","version":"117'10","reported_seq":237,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-10T19:16:43.938724+0000","last_change":"2024-09-10T19:15:35.973188+0000","last_active":"2024-09-10T19:16:43.938724+0000","last_peered":"2024-09-10T19:16:43.938724+0000","last_clean":"2024-09-10T19:16:43.938724+0000","last_became_active":"2024-09-10T19:13:54.709846+0000","last_became_peered":"2024-09-10T19:13:54.709846+0000","last_unstale":"2024-09-10T19:16:43.938724+0000","last_undegraded":"2024-09-10T19:16:43.938724+0000","last_fullsized":"2024-09-10T19:16:43.938724+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"117'10","last_scrub_stamp":"2024-09-10T19:15:35.973095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:15:35.973095+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:16:55.086873+0000","scrub_duration":0.0076338860000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,0],"acting":[3,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":135,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-10T19:16:34.797021+0000","last_change":"2024-09-10T19:16:24.858926+0000","last_active":"2024-09-10T19:16:34.797021+0000","last_peered":"2024-09-10T19:16:34.797021+0000","last_clean":"2024-09-10T19:16:34.797021+0000","last_became_active":"2024-09-10T19:16:24.858619+0000","last_became_peered":"2024-09-10T19:16:24.858619+0000","last_unstale":"2024-09-10T19:16:34.797021+0000","last_undegraded":"2024-09-10T19:16:34.797021+0000","last_fullsized":"2024-09-10T19:16:34.797021+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:13:15.350720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:11:41.073833+0000","last_clean_scrub_stamp":"2024-09-10T19:13:15.350720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0014361000000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.c","version":"162'2757","reported_seq":2779,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-09-10T19:16:44.956291+0000","last_change":"2024-09-10T19:16:13.079853+0000","last_active":"2024-09-10T19:16:44.956291+0000","last_peered":"2024-09-10T19:16:44.956291+0000","last_clean":"2024-09-10T19:16:44.956291+0000","last_became_active":"2024-09-10T19:16:13.079527+0000","last_became_peered":"2024-09-10T19:16:13.079527+0000","last_unstale":"2024-09-10T19:16:44.956291+0000","last_undegraded":"2024-09-10T19:16:44.956291+0000","last_fullsized":"2024-09-10T19:16:44.956291+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:16:11.777014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:16:11.777014+0000","last_clean_scrub_stamp":"2024-09-10T19:16:11.777014+0000","objects_scrubbed":0,"log_size":2757,"log_dups_size":0,"ondisk_log_size":2757,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:17:16.133191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180682752,"num_objects":2757,"num_object_clones":0,"num_object_copies":8271,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2757,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2757,"num_write_kb":176448,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.0","version":"66'18","reported_seq":162,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-10T19:16:43.939392+0000","last_change":"2024-09-10T19:12:49.699111+0000","last_active":"2024-09-10T19:16:43.939392+0000","last_peered":"2024-09-10T19:16:43.939392+0000","last_clean":"2024-09-10T19:16:43.939392+0000","last_became_active":"2024-09-10T19:10:09.821836+0000","last_became_peered":"2024-09-10T19:10:09.821836+0000","last_unstale":"2024-09-10T19:16:43.939392+0000","last_undegraded":"2024-09-10T19:16:43.939392+0000","last_fullsized":"2024-09-10T19:16:43.939392+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'18","last_scrub_stamp":"2024-09-10T19:12:49.699054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:12:49.699054+0000","objects_scrubbed":8,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0073070890000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":91,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-10T19:16:34.766078+0000","last_change":"2024-09-10T19:15:42.739879+0000","last_active":"2024-09-10T19:16:34.766078+0000","last_peered":"2024-09-10T19:16:34.766078+0000","last_clean":"2024-09-10T19:16:34.766078+0000","last_became_active":"2024-09-10T19:13:54.710968+0000","last_became_peered":"2024-09-10T19:13:54.710968+0000","last_unstale":"2024-09-10T19:16:34.766078+0000","last_undegraded":"2024-09-10T19:16:34.766078+0000","last_fullsized":"2024-09-10T19:16:34.766078+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:15:42.739801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:15:42.739801+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:17:05.377168+0000","scrub_duration":0.008487784,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,3],"acting":[6,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":136,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-10T19:16:34.760747+0000","last_change":"2024-09-10T19:16:30.991715+0000","last_active":"2024-09-10T19:16:34.760747+0000","last_peered":"2024-09-10T19:16:34.760747+0000","last_clean":"2024-09-10T19:16:34.760747+0000","last_became_active":"2024-09-10T19:10:11.698054+0000","last_became_peered":"2024-09-10T19:10:11.698054+0000","last_unstale":"2024-09-10T19:16:34.760747+0000","last_undegraded":"2024-09-10T19:16:34.760747+0000","last_fullsized":"2024-09-10T19:16:34.760747+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:16:30.991635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:16:30.991635+0000","last_clean_scrub_stamp":"2024-09-10T19:16:30.991635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:17:56.300591+0000","scrub_duration":0.0021810029999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":144,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-10T19:16:34.761558+0000","last_change":"2024-09-10T19:16:30.036748+0000","last_active":"2024-09-10T19:16:34.761558+0000","last_peered":"2024-09-10T19:16:34.761558+0000","last_clean":"2024-09-10T19:16:34.761558+0000","last_became_active":"2024-09-10T19:16:24.848109+0000","last_became_peered":"2024-09-10T19:16:24.848109+0000","last_unstale":"2024-09-10T19:16:34.761558+0000","last_undegraded":"2024-09-10T19:16:34.761558+0000","last_fullsized":"2024-09-10T19:16:34.761558+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:16:30.036662+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:16:30.036662+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:17:40.699772+0000","scrub_duration":0.0044984129999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.f","version":"162'2731","reported_seq":2763,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-09-10T19:16:44.918380+0000","last_change":"2024-09-10T19:16:24.856828+0000","last_active":"2024-09-10T19:16:44.918380+0000","last_peered":"2024-09-10T19:16:44.918380+0000","last_clean":"2024-09-10T19:16:44.918380+0000","last_became_active":"2024-09-10T19:16:24.855393+0000","last_became_peered":"2024-09-10T19:16:24.855393+0000","last_unstale":"2024-09-10T19:16:44.918380+0000","last_undegraded":"2024-09-10T19:16:44.918380+0000","last_fullsized":"2024-09-10T19:16:44.918380+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:16:11.777014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:16:11.777014+0000","last_clean_scrub_stamp":"2024-09-10T19:16:11.777014+0000","objects_scrubbed":0,"log_size":2731,"log_dups_size":0,"ondisk_log_size":2731,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:17:20.171361+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":178978816,"num_objects":2731,"num_object_clones":0,"num_object_copies":8193,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2731,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2731,"num_write_kb":174784,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.3","version":"104'27","reported_seq":184,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-10T19:16:43.945750+0000","last_change":"2024-09-10T19:16:35.246566+0000","last_active":"2024-09-10T19:16:43.945750+0000","last_peered":"2024-09-10T19:16:43.945750+0000","last_clean":"2024-09-10T19:16:43.945750+0000","last_became_active":"2024-09-10T19:10:09.823003+0000","last_became_peered":"2024-09-10T19:10:09.823003+0000","last_unstale":"2024-09-10T19:16:43.945750+0000","last_undegraded":"2024-09-10T19:16:43.945750+0000","last_fullsized":"2024-09-10T19:16:43.945750+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"104'27","last_scrub_stamp":"2024-09-10T19:16:35.246512+0000","last_deep_scrub":"104'27","last_deep_scrub_stamp":"2024-09-10T19:13:02.225944+0000","last_clean_scrub_stamp":"2024-09-10T19:16:35.246512+0000","objects_scrubbed":8,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:17:49.596002+0000","scrub_duration":0.0052296490000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":141,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-10T19:16:37.781488+0000","last_change":"2024-09-10T19:16:37.780086+0000","last_active":"2024-09-10T19:16:37.781488+0000","last_peered":"2024-09-10T19:16:37.781488+0000","last_clean":"2024-09-10T19:16:37.781488+0000","last_became_active":"2024-09-10T19:10:07.690325+0000","last_became_peered":"2024-09-10T19:10:07.690325+0000","last_unstale":"2024-09-10T19:16:37.781488+0000","last_undegraded":"2024-09-10T19:16:37.781488+0000","last_fullsized":"2024-09-10T19:16:37.781488+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:16:37.779967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:16:37.779967+0000","last_clean_scrub_stamp":"2024-09-10T19:16:37.779967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:17:50.595527+0000","scrub_duration":0.067110102000000005,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":137,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-10T19:16:34.765525+0000","last_change":"2024-09-10T19:15:50.669894+0000","last_active":"2024-09-10T19:16:34.765525+0000","last_peered":"2024-09-10T19:16:34.765525+0000","last_clean":"2024-09-10T19:16:34.765525+0000","last_became_active":"2024-09-10T19:10:11.700370+0000","last_became_peered":"2024-09-10T19:10:11.700370+0000","last_unstale":"2024-09-10T19:16:34.765525+0000","last_undegraded":"2024-09-10T19:16:34.765525+0000","last_fullsized":"2024-09-10T19:16:34.765525+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:15:50.669821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:15:50.669821+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:17:00.492456+0000","scrub_duration":0.0015765,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":101,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-10T19:16:34.765405+0000","last_change":"2024-09-10T19:16:24.844993+0000","last_active":"2024-09-10T19:16:34.765405+0000","last_peered":"2024-09-10T19:16:34.765405+0000","last_clean":"2024-09-10T19:16:34.765405+0000","last_became_active":"2024-09-10T19:16:24.844699+0000","last_became_peered":"2024-09-10T19:16:24.844699+0000","last_unstale":"2024-09-10T19:16:34.765405+0000","last_undegraded":"2024-09-10T19:16:34.765405+0000","last_fullsized":"2024-09-10T19:16:34.765405+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:15:41.749338+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:15:41.749338+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:17:01.220554+0000","scrub_duration":0.0018841260000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.e","version":"162'2699","reported_seq":2731,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-09-10T19:16:44.968670+0000","last_change":"2024-09-10T19:16:24.837044+0000","last_active":"2024-09-10T19:16:44.968670+0000","last_peered":"2024-09-10T19:16:44.968670+0000","last_clean":"2024-09-10T19:16:44.968670+0000","last_became_active":"2024-09-10T19:16:24.836458+0000","last_became_peered":"2024-09-10T19:16:24.836458+0000","last_unstale":"2024-09-10T19:16:44.968670+0000","last_undegraded":"2024-09-10T19:16:44.968670+0000","last_fullsized":"2024-09-10T19:16:44.968670+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:16:11.777014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:16:11.777014+0000","last_clean_scrub_stamp":"2024-09-10T19:16:11.777014+0000","objects_scrubbed":0,"log_size":2699,"log_dups_size":0,"ondisk_log_size":2699,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:17:30.166366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":176881664,"num_objects":2699,"num_object_clones":0,"num_object_copies":8097,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2699,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2699,"num_write_kb":172736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"66'10","reported_seq":98,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-10T19:16:43.948517+0000","last_change":"2024-09-10T19:13:54.748881+0000","last_active":"2024-09-10T19:16:43.948517+0000","last_peered":"2024-09-10T19:16:43.948517+0000","last_clean":"2024-09-10T19:16:43.948517+0000","last_became_active":"2024-09-10T19:13:54.714542+0000","last_became_peered":"2024-09-10T19:13:54.714542+0000","last_unstale":"2024-09-10T19:16:43.948517+0000","last_undegraded":"2024-09-10T19:16:43.948517+0000","last_fullsized":"2024-09-10T19:16:43.948517+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'10","last_scrub_stamp":"2024-09-10T19:12:56.360409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:12:56.360409+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":14,"num_read_kb":9,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,0],"acting":[5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":102,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-10T19:16:43.946100+0000","last_change":"2024-09-10T19:16:24.837191+0000","last_active":"2024-09-10T19:16:43.946100+0000","last_peered":"2024-09-10T19:16:43.946100+0000","last_clean":"2024-09-10T19:16:43.946100+0000","last_became_active":"2024-09-10T19:16:24.836898+0000","last_became_peered":"2024-09-10T19:16:24.836898+0000","last_unstale":"2024-09-10T19:16:43.946100+0000","last_undegraded":"2024-09-10T19:16:43.946100+0000","last_fullsized":"2024-09-10T19:16:43.946100+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:15:41.870422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:15:41.870422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:17:10.297611+0000","scrub_duration":0.0061607199999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3","version":"84'9","reported_seq":216,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-10T19:16:43.945577+0000","last_change":"2024-09-10T19:13:45.236207+0000","last_active":"2024-09-10T19:16:43.945577+0000","last_peered":"2024-09-10T19:16:43.945577+0000","last_clean":"2024-09-10T19:16:43.945577+0000","last_became_active":"2024-09-10T19:10:11.699690+0000","last_became_peered":"2024-09-10T19:10:11.699690+0000","last_unstale":"2024-09-10T19:16:43.945577+0000","last_undegraded":"2024-09-10T19:16:43.945577+0000","last_fullsized":"2024-09-10T19:16:43.945577+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'9","last_scrub_stamp":"2024-09-10T19:13:45.236094+0000","last_deep_scrub":"84'9","last_deep_scrub_stamp":"2024-09-10T19:11:53.879272+0000","last_clean_scrub_stamp":"2024-09-10T19:13:45.236094+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.036165602999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":136,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-10T19:16:43.945394+0000","last_change":"2024-09-10T19:16:24.840367+0000","last_active":"2024-09-10T19:16:43.945394+0000","last_peered":"2024-09-10T19:16:43.945394+0000","last_clean":"2024-09-10T19:16:43.945394+0000","last_became_active":"2024-09-10T19:16:24.840096+0000","last_became_peered":"2024-09-10T19:16:24.840096+0000","last_unstale":"2024-09-10T19:16:43.945394+0000","last_undegraded":"2024-09-10T19:16:43.945394+0000","last_fullsized":"2024-09-10T19:16:43.945394+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:13:40.234289+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:13:40.234289+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0086405590000000008,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,5],"acting":[0,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.8","version":"162'2620","reported_seq":2642,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-09-10T19:16:44.789175+0000","last_change":"2024-09-10T19:16:13.081213+0000","last_active":"2024-09-10T19:16:44.789175+0000","last_peered":"2024-09-10T19:16:44.789175+0000","last_clean":"2024-09-10T19:16:44.789175+0000","last_became_active":"2024-09-10T19:16:13.080496+0000","last_became_peered":"2024-09-10T19:16:13.080496+0000","last_unstale":"2024-09-10T19:16:44.789175+0000","last_undegraded":"2024-09-10T19:16:44.789175+0000","last_fullsized":"2024-09-10T19:16:44.789175+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:16:11.777014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:16:11.777014+0000","last_clean_scrub_stamp":"2024-09-10T19:16:11.777014+0000","objects_scrubbed":0,"log_size":2620,"log_dups_size":0,"ondisk_log_size":2620,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:17:12.464052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":171704320,"num_objects":2620,"num_object_clones":0,"num_object_copies":7860,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2620,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2620,"num_write_kb":167680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"155'31","reported_seq":107,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-10T19:16:43.939163+0000","last_change":"2024-09-10T19:16:24.910865+0000","last_active":"2024-09-10T19:16:43.939163+0000","last_peered":"2024-09-10T19:16:43.939163+0000","last_clean":"2024-09-10T19:16:43.939163+0000","last_became_active":"2024-09-10T19:16:24.858154+0000","last_became_peered":"2024-09-10T19:16:24.858154+0000","last_unstale":"2024-09-10T19:16:43.939163+0000","last_undegraded":"2024-09-10T19:16:43.939163+0000","last_fullsized":"2024-09-10T19:16:43.939163+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"117'30","last_scrub_stamp":"2024-09-10T19:15:49.837844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:15:49.837844+0000","objects_scrubbed":0,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:17:09.806255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":47,"num_read_kb":32,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":358,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":149,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-10T19:16:43.948729+0000","last_change":"2024-09-10T19:15:54.746979+0000","last_active":"2024-09-10T19:16:43.948729+0000","last_peered":"2024-09-10T19:16:43.948729+0000","last_clean":"2024-09-10T19:16:43.948729+0000","last_became_active":"2024-09-10T19:13:54.730096+0000","last_became_peered":"2024-09-10T19:13:54.730096+0000","last_unstale":"2024-09-10T19:16:43.948729+0000","last_undegraded":"2024-09-10T19:16:43.948729+0000","last_fullsized":"2024-09-10T19:16:43.948729+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:15:54.746920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:15:54.746920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:17:00.357419+0000","scrub_duration":0.001300037,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":145,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-10T19:16:43.946710+0000","last_change":"2024-09-10T19:15:34.802331+0000","last_active":"2024-09-10T19:16:43.946710+0000","last_peered":"2024-09-10T19:16:43.946710+0000","last_clean":"2024-09-10T19:16:43.946710+0000","last_became_active":"2024-09-10T19:13:54.714017+0000","last_became_peered":"2024-09-10T19:13:54.714017+0000","last_unstale":"2024-09-10T19:16:43.946710+0000","last_undegraded":"2024-09-10T19:16:43.946710+0000","last_fullsized":"2024-09-10T19:16:43.946710+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:15:34.802121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:15:34.802121+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:17:01.582331+0000","scrub_duration":0.0042778080000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,6],"acting":[0,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":126,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-10T19:16:43.939629+0000","last_change":"2024-09-10T19:16:32.818668+0000","last_active":"2024-09-10T19:16:43.939629+0000","last_peered":"2024-09-10T19:16:43.939629+0000","last_clean":"2024-09-10T19:16:43.939629+0000","last_became_active":"2024-09-10T19:10:13.721744+0000","last_became_peered":"2024-09-10T19:10:13.721744+0000","last_unstale":"2024-09-10T19:16:43.939629+0000","last_undegraded":"2024-09-10T19:16:43.939629+0000","last_fullsized":"2024-09-10T19:16:43.939629+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:16:32.818590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:16:32.818590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:17:49.356780+0000","scrub_duration":0.0047149059999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.b","version":"162'2797","reported_seq":2829,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-09-10T19:16:44.897264+0000","last_change":"2024-09-10T19:16:24.852409+0000","last_active":"2024-09-10T19:16:44.897264+0000","last_peered":"2024-09-10T19:16:44.897264+0000","last_clean":"2024-09-10T19:16:44.897264+0000","last_became_active":"2024-09-10T19:16:24.850682+0000","last_became_peered":"2024-09-10T19:16:24.850682+0000","last_unstale":"2024-09-10T19:16:44.897264+0000","last_undegraded":"2024-09-10T19:16:44.897264+0000","last_fullsized":"2024-09-10T19:16:44.897264+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:16:11.777014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:16:11.777014+0000","last_clean_scrub_stamp":"2024-09-10T19:16:11.777014+0000","objects_scrubbed":0,"log_size":2797,"log_dups_size":0,"ondisk_log_size":2797,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:17:26.367282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":183304192,"num_objects":2797,"num_object_clones":0,"num_object_copies":8391,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2797,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2797,"num_write_kb":179008,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.7","version":"82'18","reported_seq":165,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-10T19:16:43.949528+0000","last_change":"2024-09-10T19:15:55.777428+0000","last_active":"2024-09-10T19:16:43.949528+0000","last_peered":"2024-09-10T19:16:43.949528+0000","last_clean":"2024-09-10T19:16:43.949528+0000","last_became_active":"2024-09-10T19:13:54.712065+0000","last_became_peered":"2024-09-10T19:13:54.712065+0000","last_unstale":"2024-09-10T19:16:43.949528+0000","last_undegraded":"2024-09-10T19:16:43.949528+0000","last_fullsized":"2024-09-10T19:16:43.949528+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'18","last_scrub_stamp":"2024-09-10T19:15:55.777361+0000","last_deep_scrub":"82'18","last_deep_scrub_stamp":"2024-09-10T19:15:55.777361+0000","last_clean_scrub_stamp":"2024-09-10T19:15:55.777361+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:17:17.118895+0000","scrub_duration":0.0023560040000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":19,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.0","version":"159'6","reported_seq":82,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-10T19:16:43.901169+0000","last_change":"2024-09-10T19:16:32.650007+0000","last_active":"2024-09-10T19:16:43.901169+0000","last_peered":"2024-09-10T19:16:43.901169+0000","last_clean":"2024-09-10T19:16:43.901169+0000","last_became_active":"2024-09-10T19:16:24.846203+0000","last_became_peered":"2024-09-10T19:16:24.846203+0000","last_unstale":"2024-09-10T19:16:43.901169+0000","last_undegraded":"2024-09-10T19:16:43.901169+0000","last_fullsized":"2024-09-10T19:16:43.901169+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"159'6","last_scrub_stamp":"2024-09-10T19:16:32.649914+0000","last_deep_scrub":"159'6","last_deep_scrub_stamp":"2024-09-10T19:16:32.649914+0000","last_clean_scrub_stamp":"2024-09-10T19:16:32.649914+0000","objects_scrubbed":1,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:17:33.710001+0000","scrub_duration":0.013939817,"objects_trimmed":0,"snaptrim_duration":0.00034454499999999997,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":26,"num_read_kb":21,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":389,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":30,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-10T19:16:43.939150+0000","last_change":"2024-09-10T19:16:33.811962+0000","last_active":"2024-09-10T19:16:43.939150+0000","last_peered":"2024-09-10T19:16:43.939150+0000","last_clean":"2024-09-10T19:16:43.939150+0000","last_became_active":"2024-09-10T19:16:24.857505+0000","last_became_peered":"2024-09-10T19:16:24.857505+0000","last_unstale":"2024-09-10T19:16:43.939150+0000","last_undegraded":"2024-09-10T19:16:43.939150+0000","last_fullsized":"2024-09-10T19:16:43.939150+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:16:33.811869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:16:33.811869+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:17:45.100668+0000","scrub_duration":0.0027920340000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":30,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-10T19:16:43.958372+0000","last_change":"2024-09-10T19:16:43.958311+0000","last_active":"2024-09-10T19:16:43.958372+0000","last_peered":"2024-09-10T19:16:43.958372+0000","last_clean":"2024-09-10T19:16:43.958372+0000","last_became_active":"2024-09-10T19:16:24.861481+0000","last_became_peered":"2024-09-10T19:16:24.861481+0000","last_unstale":"2024-09-10T19:16:43.958372+0000","last_undegraded":"2024-09-10T19:16:43.958372+0000","last_fullsized":"2024-09-10T19:16:43.958372+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:16:43.958224+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:16:43.958224+0000","last_clean_scrub_stamp":"2024-09-10T19:16:43.958224+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:18:09.697349+0000","scrub_duration":0.015970275999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,4],"acting":[5,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":128,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-10T19:16:34.797617+0000","last_change":"2024-09-10T19:15:38.493179+0000","last_active":"2024-09-10T19:16:34.797617+0000","last_peered":"2024-09-10T19:16:34.797617+0000","last_clean":"2024-09-10T19:16:34.797617+0000","last_became_active":"2024-09-10T19:10:14.053697+0000","last_became_peered":"2024-09-10T19:10:14.053697+0000","last_unstale":"2024-09-10T19:16:34.797617+0000","last_undegraded":"2024-09-10T19:16:34.797617+0000","last_fullsized":"2024-09-10T19:16:34.797617+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:15:38.493102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:15:38.493102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:16:48.733504+0000","scrub_duration":0.0015544280000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.a","version":"162'2786","reported_seq":2803,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-09-10T19:16:44.787772+0000","last_change":"2024-09-10T19:16:24.865995+0000","last_active":"2024-09-10T19:16:44.787772+0000","last_peered":"2024-09-10T19:16:44.787772+0000","last_clean":"2024-09-10T19:16:44.787772+0000","last_became_active":"2024-09-10T19:16:24.864959+0000","last_became_peered":"2024-09-10T19:16:24.864959+0000","last_unstale":"2024-09-10T19:16:44.787772+0000","last_undegraded":"2024-09-10T19:16:44.787772+0000","last_fullsized":"2024-09-10T19:16:44.787772+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:16:11.777014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:16:11.777014+0000","last_clean_scrub_stamp":"2024-09-10T19:16:11.777014+0000","objects_scrubbed":0,"log_size":2786,"log_dups_size":0,"ondisk_log_size":2786,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:17:20.676451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":182583296,"num_objects":2786,"num_object_clones":0,"num_object_copies":8358,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2786,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2786,"num_write_kb":178304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.6","version":"66'12","reported_seq":203,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-10T19:16:43.947563+0000","last_change":"2024-09-10T19:16:24.878036+0000","last_active":"2024-09-10T19:16:43.947563+0000","last_peered":"2024-09-10T19:16:43.947563+0000","last_clean":"2024-09-10T19:16:43.947563+0000","last_became_active":"2024-09-10T19:16:24.844323+0000","last_became_peered":"2024-09-10T19:16:24.844323+0000","last_unstale":"2024-09-10T19:16:43.947563+0000","last_undegraded":"2024-09-10T19:16:43.947563+0000","last_fullsized":"2024-09-10T19:16:43.947563+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'12","last_scrub_stamp":"2024-09-10T19:15:40.903279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:15:40.903279+0000","objects_scrubbed":6,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0271238,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,5],"acting":[0,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"61'1","reported_seq":148,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-10T19:16:43.947222+0000","last_change":"2024-09-10T19:16:42.195540+0000","last_active":"2024-09-10T19:16:43.947222+0000","last_peered":"2024-09-10T19:16:43.947222+0000","last_clean":"2024-09-10T19:16:43.947222+0000","last_became_active":"2024-09-10T19:10:07.683259+0000","last_became_peered":"2024-09-10T19:10:07.683259+0000","last_unstale":"2024-09-10T19:16:43.947222+0000","last_undegraded":"2024-09-10T19:16:43.947222+0000","last_fullsized":"2024-09-10T19:16:43.947222+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'1","last_scrub_stamp":"2024-09-10T19:16:42.195463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:16:42.195463+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:17:50.284803+0000","scrub_duration":0.0016622189999999999,"objects_trimmed":0,"snaptrim_duration":0.0014341130000000001,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":148,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-10T19:16:34.796750+0000","last_change":"2024-09-10T19:15:30.542477+0000","last_active":"2024-09-10T19:16:34.796750+0000","last_peered":"2024-09-10T19:16:34.796750+0000","last_clean":"2024-09-10T19:16:34.796750+0000","last_became_active":"2024-09-10T19:13:54.721486+0000","last_became_peered":"2024-09-10T19:13:54.721486+0000","last_unstale":"2024-09-10T19:16:34.796750+0000","last_undegraded":"2024-09-10T19:16:34.796750+0000","last_fullsized":"2024-09-10T19:16:34.796750+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:15:30.542399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:15:30.542399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:17:00.089482+0000","scrub_duration":0.0025142609999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":145,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-10T19:16:43.949452+0000","last_change":"2024-09-10T19:13:54.712428+0000","last_active":"2024-09-10T19:16:43.949452+0000","last_peered":"2024-09-10T19:16:43.949452+0000","last_clean":"2024-09-10T19:16:43.949452+0000","last_became_active":"2024-09-10T19:13:54.711685+0000","last_became_peered":"2024-09-10T19:13:54.711685+0000","last_unstale":"2024-09-10T19:16:43.949452+0000","last_undegraded":"2024-09-10T19:16:43.949452+0000","last_fullsized":"2024-09-10T19:16:43.949452+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:12:44.527561+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:12:44.527561+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.022638367999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,4],"acting":[5,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":92,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-10T19:16:43.950149+0000","last_change":"2024-09-10T19:16:28.786490+0000","last_active":"2024-09-10T19:16:43.950149+0000","last_peered":"2024-09-10T19:16:43.950149+0000","last_clean":"2024-09-10T19:16:43.950149+0000","last_became_active":"2024-09-10T19:13:54.722937+0000","last_became_peered":"2024-09-10T19:13:54.722937+0000","last_unstale":"2024-09-10T19:16:43.950149+0000","last_undegraded":"2024-09-10T19:16:43.950149+0000","last_fullsized":"2024-09-10T19:16:43.950149+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:16:28.786056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:16:28.786056+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:17:51.730597+0000","scrub_duration":0.0082035739999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.9","version":"161'2672","reported_seq":2693,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-10T19:16:44.738264+0000","last_change":"2024-09-10T19:16:12.797816+0000","last_active":"2024-09-10T19:16:44.738264+0000","last_peered":"2024-09-10T19:16:44.738264+0000","last_clean":"2024-09-10T19:16:44.738264+0000","last_became_active":"2024-09-10T19:16:12.797479+0000","last_became_peered":"2024-09-10T19:16:12.797479+0000","last_unstale":"2024-09-10T19:16:44.738264+0000","last_undegraded":"2024-09-10T19:16:44.738264+0000","last_fullsized":"2024-09-10T19:16:44.738264+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:16:11.777014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:16:11.777014+0000","last_clean_scrub_stamp":"2024-09-10T19:16:11.777014+0000","objects_scrubbed":0,"log_size":2672,"log_dups_size":0,"ondisk_log_size":2672,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:17:11.992307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":175112192,"num_objects":2672,"num_object_clones":0,"num_object_copies":8016,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2672,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2672,"num_write_kb":171008,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,0],"acting":[6,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"80'21","reported_seq":168,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-10T19:16:34.766160+0000","last_change":"2024-09-10T19:15:49.682164+0000","last_active":"2024-09-10T19:16:34.766160+0000","last_peered":"2024-09-10T19:16:34.766160+0000","last_clean":"2024-09-10T19:16:34.766160+0000","last_became_active":"2024-09-10T19:10:09.822184+0000","last_became_peered":"2024-09-10T19:10:09.822184+0000","last_unstale":"2024-09-10T19:16:34.766160+0000","last_undegraded":"2024-09-10T19:16:34.766160+0000","last_fullsized":"2024-09-10T19:16:34.766160+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"80'21","last_scrub_stamp":"2024-09-10T19:15:49.682092+0000","last_deep_scrub":"80'21","last_deep_scrub_stamp":"2024-09-10T19:11:39.836144+0000","last_clean_scrub_stamp":"2024-09-10T19:15:49.682092+0000","objects_scrubbed":7,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:17:05.534997+0000","scrub_duration":0.0022142529999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"63'2","reported_seq":149,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-10T19:16:43.939323+0000","last_change":"2024-09-10T19:16:35.962952+0000","last_active":"2024-09-10T19:16:43.939323+0000","last_peered":"2024-09-10T19:16:43.939323+0000","last_clean":"2024-09-10T19:16:43.939323+0000","last_became_active":"2024-09-10T19:10:07.684784+0000","last_became_peered":"2024-09-10T19:10:07.684784+0000","last_unstale":"2024-09-10T19:16:43.939323+0000","last_undegraded":"2024-09-10T19:16:43.939323+0000","last_fullsized":"2024-09-10T19:16:43.939323+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'2","last_scrub_stamp":"2024-09-10T19:16:35.962804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:16:35.962804+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:18:04.954385+0000","scrub_duration":0.105395924,"objects_trimmed":0,"snaptrim_duration":0.00075522999999999996,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":152,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-10T19:16:43.948996+0000","last_change":"2024-09-10T19:16:24.858963+0000","last_active":"2024-09-10T19:16:43.948996+0000","last_peered":"2024-09-10T19:16:43.948996+0000","last_clean":"2024-09-10T19:16:43.948996+0000","last_became_active":"2024-09-10T19:16:24.858662+0000","last_became_peered":"2024-09-10T19:16:24.858662+0000","last_unstale":"2024-09-10T19:16:43.948996+0000","last_undegraded":"2024-09-10T19:16:43.948996+0000","last_fullsized":"2024-09-10T19:16:43.948996+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:13:50.411500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:13:50.411500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0032604869999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"74'79","reported_seq":176,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-10T19:16:34.796908+0000","last_change":"2024-09-10T19:13:46.419002+0000","last_active":"2024-09-10T19:16:34.796908+0000","last_peered":"2024-09-10T19:16:34.796908+0000","last_clean":"2024-09-10T19:16:34.796908+0000","last_became_active":"2024-09-10T19:09:56.634919+0000","last_became_peered":"2024-09-10T19:09:56.634919+0000","last_unstale":"2024-09-10T19:16:34.796908+0000","last_undegraded":"2024-09-10T19:16:34.796908+0000","last_fullsized":"2024-09-10T19:16:34.796908+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'79","last_scrub_stamp":"2024-09-10T19:13:46.418921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:08:00.699315+0000","last_clean_scrub_stamp":"2024-09-10T19:13:46.418921+0000","objects_scrubbed":2,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0029827999999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":137,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-10T19:16:34.797576+0000","last_change":"2024-09-10T19:16:24.852723+0000","last_active":"2024-09-10T19:16:34.797576+0000","last_peered":"2024-09-10T19:16:34.797576+0000","last_clean":"2024-09-10T19:16:34.797576+0000","last_became_active":"2024-09-10T19:16:24.851652+0000","last_became_peered":"2024-09-10T19:16:24.851652+0000","last_unstale":"2024-09-10T19:16:34.797576+0000","last_undegraded":"2024-09-10T19:16:34.797576+0000","last_fullsized":"2024-09-10T19:16:34.797576+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:13:47.453228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:13:47.453228+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.010282463,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,0],"acting":[7,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":126,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-10T19:16:43.949649+0000","last_change":"2024-09-10T19:13:05.390398+0000","last_active":"2024-09-10T19:16:43.949649+0000","last_peered":"2024-09-10T19:16:43.949649+0000","last_clean":"2024-09-10T19:16:43.949649+0000","last_became_active":"2024-09-10T19:10:13.720647+0000","last_became_peered":"2024-09-10T19:10:13.720647+0000","last_unstale":"2024-09-10T19:16:43.949649+0000","last_undegraded":"2024-09-10T19:16:43.949649+0000","last_fullsized":"2024-09-10T19:16:43.949649+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:13:05.390317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:13:05.390317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0058030640000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.6","version":"162'2719","reported_seq":2741,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-09-10T19:16:44.981003+0000","last_change":"2024-09-10T19:16:13.080600+0000","last_active":"2024-09-10T19:16:44.981003+0000","last_peered":"2024-09-10T19:16:44.981003+0000","last_clean":"2024-09-10T19:16:44.981003+0000","last_became_active":"2024-09-10T19:16:13.080292+0000","last_became_peered":"2024-09-10T19:16:13.080292+0000","last_unstale":"2024-09-10T19:16:44.981003+0000","last_undegraded":"2024-09-10T19:16:44.981003+0000","last_fullsized":"2024-09-10T19:16:44.981003+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:16:11.777014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:16:11.777014+0000","last_clean_scrub_stamp":"2024-09-10T19:16:11.777014+0000","objects_scrubbed":0,"log_size":2719,"log_dups_size":0,"ondisk_log_size":2719,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:17:24.544000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":178192384,"num_objects":2719,"num_object_clones":0,"num_object_copies":8157,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2719,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2719,"num_write_kb":174016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,7],"acting":[0,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"66'19","reported_seq":197,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-10T19:16:34.765760+0000","last_change":"2024-09-10T19:15:38.722679+0000","last_active":"2024-09-10T19:16:34.765760+0000","last_peered":"2024-09-10T19:16:34.765760+0000","last_clean":"2024-09-10T19:16:34.765760+0000","last_became_active":"2024-09-10T19:13:54.731274+0000","last_became_peered":"2024-09-10T19:13:54.731274+0000","last_unstale":"2024-09-10T19:16:34.765760+0000","last_undegraded":"2024-09-10T19:16:34.765760+0000","last_fullsized":"2024-09-10T19:16:34.765760+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'19","last_scrub_stamp":"2024-09-10T19:15:38.722581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:15:38.722581+0000","objects_scrubbed":9,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:16:45.004511+0000","scrub_duration":0.0054904940000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,5],"acting":[6,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":20,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-10T19:16:43.950315+0000","last_change":"2024-09-10T19:16:24.847051+0000","last_active":"2024-09-10T19:16:43.950315+0000","last_peered":"2024-09-10T19:16:43.950315+0000","last_clean":"2024-09-10T19:16:43.950315+0000","last_became_active":"2024-09-10T19:16:24.846745+0000","last_became_peered":"2024-09-10T19:16:24.846745+0000","last_unstale":"2024-09-10T19:16:43.950315+0000","last_undegraded":"2024-09-10T19:16:43.950315+0000","last_fullsized":"2024-09-10T19:16:43.950315+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:15:53.803057+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:15:53.803057+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:16:56.459399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3],"acting":[5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":30,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-10T19:16:43.945432+0000","last_change":"2024-09-10T19:16:27.240222+0000","last_active":"2024-09-10T19:16:43.945432+0000","last_peered":"2024-09-10T19:16:43.945432+0000","last_clean":"2024-09-10T19:16:43.945432+0000","last_became_active":"2024-09-10T19:16:24.844295+0000","last_became_peered":"2024-09-10T19:16:24.844295+0000","last_unstale":"2024-09-10T19:16:43.945432+0000","last_undegraded":"2024-09-10T19:16:43.945432+0000","last_fullsized":"2024-09-10T19:16:43.945432+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:16:27.240139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:16:27.240139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:17:44.435081+0000","scrub_duration":0.0020507199999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":135,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-10T19:16:34.796829+0000","last_change":"2024-09-10T19:16:24.858456+0000","last_active":"2024-09-10T19:16:34.796829+0000","last_peered":"2024-09-10T19:16:34.796829+0000","last_clean":"2024-09-10T19:16:34.796829+0000","last_became_active":"2024-09-10T19:16:24.858172+0000","last_became_peered":"2024-09-10T19:16:24.858172+0000","last_unstale":"2024-09-10T19:16:34.796829+0000","last_undegraded":"2024-09-10T19:16:34.796829+0000","last_fullsized":"2024-09-10T19:16:34.796829+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:13:14.311620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:13:14.311620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0054101059999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,6],"acting":[7,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.7","version":"162'2770","reported_seq":2801,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-09-10T19:16:44.926499+0000","last_change":"2024-09-10T19:16:24.858911+0000","last_active":"2024-09-10T19:16:44.926499+0000","last_peered":"2024-09-10T19:16:44.926499+0000","last_clean":"2024-09-10T19:16:44.926499+0000","last_became_active":"2024-09-10T19:16:24.857599+0000","last_became_peered":"2024-09-10T19:16:24.857599+0000","last_unstale":"2024-09-10T19:16:44.926499+0000","last_undegraded":"2024-09-10T19:16:44.926499+0000","last_fullsized":"2024-09-10T19:16:44.926499+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:16:11.777014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:16:11.777014+0000","last_clean_scrub_stamp":"2024-09-10T19:16:11.777014+0000","objects_scrubbed":0,"log_size":2770,"log_dups_size":0,"ondisk_log_size":2770,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:17:40.541111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":181534720,"num_objects":2770,"num_object_clones":0,"num_object_copies":8310,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2770,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2770,"num_write_kb":177280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,4],"acting":[3,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.b","version":"66'9","reported_seq":173,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-10T19:16:43.945668+0000","last_change":"2024-09-10T19:15:37.846895+0000","last_active":"2024-09-10T19:16:43.945668+0000","last_peered":"2024-09-10T19:16:43.945668+0000","last_clean":"2024-09-10T19:16:43.945668+0000","last_became_active":"2024-09-10T19:13:54.710613+0000","last_became_peered":"2024-09-10T19:13:54.710613+0000","last_unstale":"2024-09-10T19:16:43.945668+0000","last_undegraded":"2024-09-10T19:16:43.945668+0000","last_fullsized":"2024-09-10T19:16:43.945668+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'9","last_scrub_stamp":"2024-09-10T19:15:37.846807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:15:37.846807+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:16:45.860729+0000","scrub_duration":0.012189228,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":104,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-10T19:16:34.791677+0000","last_change":"2024-09-10T19:16:24.848979+0000","last_active":"2024-09-10T19:16:34.791677+0000","last_peered":"2024-09-10T19:16:34.791677+0000","last_clean":"2024-09-10T19:16:34.791677+0000","last_became_active":"2024-09-10T19:16:24.848668+0000","last_became_peered":"2024-09-10T19:16:24.848668+0000","last_unstale":"2024-09-10T19:16:34.791677+0000","last_undegraded":"2024-09-10T19:16:34.791677+0000","last_fullsized":"2024-09-10T19:16:34.791677+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:15:50.850576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:11:44.254513+0000","last_clean_scrub_stamp":"2024-09-10T19:15:50.850576+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:16:59.892529+0000","scrub_duration":0.001815419,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":19,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-10T19:16:34.761419+0000","last_change":"2024-09-10T19:16:24.846109+0000","last_active":"2024-09-10T19:16:34.761419+0000","last_peered":"2024-09-10T19:16:34.761419+0000","last_clean":"2024-09-10T19:16:34.761419+0000","last_became_active":"2024-09-10T19:16:24.845709+0000","last_became_peered":"2024-09-10T19:16:24.845709+0000","last_unstale":"2024-09-10T19:16:34.761419+0000","last_undegraded":"2024-09-10T19:16:34.761419+0000","last_fullsized":"2024-09-10T19:16:34.761419+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:13:08.398036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:13:08.398036+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":146,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-10T19:16:43.946555+0000","last_change":"2024-09-10T19:16:39.188737+0000","last_active":"2024-09-10T19:16:43.946555+0000","last_peered":"2024-09-10T19:16:43.946555+0000","last_clean":"2024-09-10T19:16:43.946555+0000","last_became_active":"2024-09-10T19:16:24.837483+0000","last_became_peered":"2024-09-10T19:16:24.837483+0000","last_unstale":"2024-09-10T19:16:43.946555+0000","last_undegraded":"2024-09-10T19:16:43.946555+0000","last_fullsized":"2024-09-10T19:16:43.946555+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:16:39.188682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:16:39.188682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:17:48.097207+0000","scrub_duration":0.0012633589999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.4","version":"162'2764","reported_seq":2796,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-09-10T19:16:44.788432+0000","last_change":"2024-09-10T19:16:24.863099+0000","last_active":"2024-09-10T19:16:44.788432+0000","last_peered":"2024-09-10T19:16:44.788432+0000","last_clean":"2024-09-10T19:16:44.788432+0000","last_became_active":"2024-09-10T19:16:24.862785+0000","last_became_peered":"2024-09-10T19:16:24.862785+0000","last_unstale":"2024-09-10T19:16:44.788432+0000","last_undegraded":"2024-09-10T19:16:44.788432+0000","last_fullsized":"2024-09-10T19:16:44.788432+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:16:11.777014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:16:11.777014+0000","last_clean_scrub_stamp":"2024-09-10T19:16:11.777014+0000","objects_scrubbed":0,"log_size":2764,"log_dups_size":0,"ondisk_log_size":2764,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:17:21.039951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":181141504,"num_objects":2764,"num_object_clones":0,"num_object_copies":8292,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2764,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2764,"num_write_kb":176896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.8","version":"66'15","reported_seq":155,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-10T19:16:43.949090+0000","last_change":"2024-09-10T19:12:50.449684+0000","last_active":"2024-09-10T19:16:43.949090+0000","last_peered":"2024-09-10T19:16:43.949090+0000","last_clean":"2024-09-10T19:16:43.949090+0000","last_became_active":"2024-09-10T19:10:09.820950+0000","last_became_peered":"2024-09-10T19:10:09.820950+0000","last_unstale":"2024-09-10T19:16:43.949090+0000","last_undegraded":"2024-09-10T19:16:43.949090+0000","last_fullsized":"2024-09-10T19:16:43.949090+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'15","last_scrub_stamp":"2024-09-10T19:12:50.449596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:12:50.449596+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0055729239999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":20,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-10T19:16:43.939835+0000","last_change":"2024-09-10T19:16:24.857533+0000","last_active":"2024-09-10T19:16:43.939835+0000","last_peered":"2024-09-10T19:16:43.939835+0000","last_clean":"2024-09-10T19:16:43.939835+0000","last_became_active":"2024-09-10T19:16:24.857253+0000","last_became_peered":"2024-09-10T19:16:24.857253+0000","last_unstale":"2024-09-10T19:16:43.939835+0000","last_undegraded":"2024-09-10T19:16:43.939835+0000","last_fullsized":"2024-09-10T19:16:43.939835+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:15:38.677960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:15:38.677960+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:16:51.127092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.9","version":"80'9","reported_seq":215,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-10T19:16:40.823332+0000","last_change":"2024-09-10T19:13:43.358984+0000","last_active":"2024-09-10T19:16:40.823332+0000","last_peered":"2024-09-10T19:16:40.823332+0000","last_clean":"2024-09-10T19:16:40.823332+0000","last_became_active":"2024-09-10T19:10:11.702878+0000","last_became_peered":"2024-09-10T19:10:11.702878+0000","last_unstale":"2024-09-10T19:16:40.823332+0000","last_undegraded":"2024-09-10T19:16:40.823332+0000","last_fullsized":"2024-09-10T19:16:40.823332+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"80'9","last_scrub_stamp":"2024-09-10T19:13:43.358901+0000","last_deep_scrub":"80'9","last_deep_scrub_stamp":"2024-09-10T19:11:35.174218+0000","last_clean_scrub_stamp":"2024-09-10T19:13:43.358901+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0066143260000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":126,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-10T19:16:43.948216+0000","last_change":"2024-09-10T19:12:57.392898+0000","last_active":"2024-09-10T19:16:43.948216+0000","last_peered":"2024-09-10T19:16:43.948216+0000","last_clean":"2024-09-10T19:16:43.948216+0000","last_became_active":"2024-09-10T19:10:14.053875+0000","last_became_peered":"2024-09-10T19:10:14.053875+0000","last_unstale":"2024-09-10T19:16:43.948216+0000","last_undegraded":"2024-09-10T19:16:43.948216+0000","last_fullsized":"2024-09-10T19:16:43.948216+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:12:57.392844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:12:57.392844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0046180839999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.5","version":"161'2733","reported_seq":2753,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-10T19:16:44.735285+0000","last_change":"2024-09-10T19:16:13.081074+0000","last_active":"2024-09-10T19:16:44.735285+0000","last_peered":"2024-09-10T19:16:44.735285+0000","last_clean":"2024-09-10T19:16:44.735285+0000","last_became_active":"2024-09-10T19:16:13.080762+0000","last_became_peered":"2024-09-10T19:16:13.080762+0000","last_unstale":"2024-09-10T19:16:44.735285+0000","last_undegraded":"2024-09-10T19:16:44.735285+0000","last_fullsized":"2024-09-10T19:16:44.735285+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:16:11.777014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:16:11.777014+0000","last_clean_scrub_stamp":"2024-09-10T19:16:11.777014+0000","objects_scrubbed":0,"log_size":2733,"log_dups_size":0,"ondisk_log_size":2733,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:17:34.160639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":179109888,"num_objects":2733,"num_object_clones":0,"num_object_copies":8199,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2733,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2733,"num_write_kb":174912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.9","version":"66'12","reported_seq":183,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-10T19:16:34.760841+0000","last_change":"2024-09-10T19:15:37.869593+0000","last_active":"2024-09-10T19:16:34.760841+0000","last_peered":"2024-09-10T19:16:34.760841+0000","last_clean":"2024-09-10T19:16:34.760841+0000","last_became_active":"2024-09-10T19:13:54.718152+0000","last_became_peered":"2024-09-10T19:13:54.718152+0000","last_unstale":"2024-09-10T19:16:34.760841+0000","last_undegraded":"2024-09-10T19:16:34.760841+0000","last_fullsized":"2024-09-10T19:16:34.760841+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'12","last_scrub_stamp":"2024-09-10T19:15:37.869451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:15:37.869451+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:16:59.444694+0000","scrub_duration":0.0025441249999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.f","version":"59'2","reported_seq":156,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-10T19:16:34.794114+0000","last_change":"2024-09-10T19:16:29.007627+0000","last_active":"2024-09-10T19:16:34.794114+0000","last_peered":"2024-09-10T19:16:34.794114+0000","last_clean":"2024-09-10T19:16:34.794114+0000","last_became_active":"2024-09-10T19:10:07.681719+0000","last_became_peered":"2024-09-10T19:10:07.681719+0000","last_unstale":"2024-09-10T19:16:34.794114+0000","last_undegraded":"2024-09-10T19:16:34.794114+0000","last_fullsized":"2024-09-10T19:16:34.794114+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'2","last_scrub_stamp":"2024-09-10T19:16:29.007251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:16:29.007251+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:17:56.729256+0000","scrub_duration":0.0079698319999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":14,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":100,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-10T19:16:43.945222+0000","last_change":"2024-09-10T19:16:24.840337+0000","last_active":"2024-09-10T19:16:43.945222+0000","last_peered":"2024-09-10T19:16:43.945222+0000","last_clean":"2024-09-10T19:16:43.945222+0000","last_became_active":"2024-09-10T19:16:24.840091+0000","last_became_peered":"2024-09-10T19:16:24.840091+0000","last_unstale":"2024-09-10T19:16:43.945222+0000","last_undegraded":"2024-09-10T19:16:43.945222+0000","last_fullsized":"2024-09-10T19:16:43.945222+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:15:31.776481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:11:46.587846+0000","last_clean_scrub_stamp":"2024-09-10T19:15:31.776481+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:16:46.797421+0000","scrub_duration":0.001267142,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,4],"acting":[0,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":143,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-10T19:16:43.941807+0000","last_change":"2024-09-10T19:15:41.093923+0000","last_active":"2024-09-10T19:16:43.941807+0000","last_peered":"2024-09-10T19:16:43.941807+0000","last_clean":"2024-09-10T19:16:43.941807+0000","last_became_active":"2024-09-10T19:13:54.711804+0000","last_became_peered":"2024-09-10T19:13:54.711804+0000","last_unstale":"2024-09-10T19:16:43.941807+0000","last_undegraded":"2024-09-10T19:16:43.941807+0000","last_fullsized":"2024-09-10T19:16:43.941807+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:15:41.093783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:15:41.093783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:16:51.502004+0000","scrub_duration":0.0023833050000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.16","version":"66'9","reported_seq":156,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-10T19:16:43.947471+0000","last_change":"2024-09-10T19:13:50.172901+0000","last_active":"2024-09-10T19:16:43.947471+0000","last_peered":"2024-09-10T19:16:43.947471+0000","last_clean":"2024-09-10T19:16:43.947471+0000","last_became_active":"2024-09-10T19:10:09.822994+0000","last_became_peered":"2024-09-10T19:10:09.822994+0000","last_unstale":"2024-09-10T19:16:43.947471+0000","last_undegraded":"2024-09-10T19:16:43.947471+0000","last_fullsized":"2024-09-10T19:16:43.947471+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'9","last_scrub_stamp":"2024-09-10T19:13:50.172825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:13:50.172825+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0094381619999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":104,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-10T19:16:43.945622+0000","last_change":"2024-09-10T19:16:24.844555+0000","last_active":"2024-09-10T19:16:43.945622+0000","last_peered":"2024-09-10T19:16:43.945622+0000","last_clean":"2024-09-10T19:16:43.945622+0000","last_became_active":"2024-09-10T19:16:24.844313+0000","last_became_peered":"2024-09-10T19:16:24.844313+0000","last_unstale":"2024-09-10T19:16:43.945622+0000","last_undegraded":"2024-09-10T19:16:43.945622+0000","last_fullsized":"2024-09-10T19:16:43.945622+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:15:36.662596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:11:41.458523+0000","last_clean_scrub_stamp":"2024-09-10T19:15:36.662596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0012499830000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,6],"acting":[0,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":145,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-10T19:16:43.940559+0000","last_change":"2024-09-10T19:16:28.794727+0000","last_active":"2024-09-10T19:16:43.940559+0000","last_peered":"2024-09-10T19:16:43.940559+0000","last_clean":"2024-09-10T19:16:43.940559+0000","last_became_active":"2024-09-10T19:13:54.712275+0000","last_became_peered":"2024-09-10T19:13:54.712275+0000","last_unstale":"2024-09-10T19:16:43.940559+0000","last_undegraded":"2024-09-10T19:16:43.940559+0000","last_fullsized":"2024-09-10T19:16:43.940559+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:16:28.794649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:16:28.794649+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:17:53.505673+0000","scrub_duration":0.0027500379999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"66'1","reported_seq":32,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-10T19:16:34.761446+0000","last_change":"2024-09-10T19:16:24.862865+0000","last_active":"2024-09-10T19:16:34.761446+0000","last_peered":"2024-09-10T19:16:34.761446+0000","last_clean":"2024-09-10T19:16:34.761446+0000","last_became_active":"2024-09-10T19:16:24.837432+0000","last_became_peered":"2024-09-10T19:16:24.837432+0000","last_unstale":"2024-09-10T19:16:34.761446+0000","last_undegraded":"2024-09-10T19:16:34.761446+0000","last_fullsized":"2024-09-10T19:16:34.761446+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'1","last_scrub_stamp":"2024-09-10T19:15:48.839582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:15:48.839582+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:16:49.914987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":13,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":151,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-10T19:16:34.765562+0000","last_change":"2024-09-10T19:15:36.714392+0000","last_active":"2024-09-10T19:16:34.765562+0000","last_peered":"2024-09-10T19:16:34.765562+0000","last_clean":"2024-09-10T19:16:34.765562+0000","last_became_active":"2024-09-10T19:13:54.724027+0000","last_became_peered":"2024-09-10T19:13:54.724027+0000","last_unstale":"2024-09-10T19:16:34.765562+0000","last_undegraded":"2024-09-10T19:16:34.765562+0000","last_fullsized":"2024-09-10T19:16:34.765562+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:15:36.714327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:15:36.714327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:16:55.977022+0000","scrub_duration":0.001831287,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,5],"acting":[6,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"66'6","reported_seq":172,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-10T19:16:43.940624+0000","last_change":"2024-09-10T19:16:24.891057+0000","last_active":"2024-09-10T19:16:43.940624+0000","last_peered":"2024-09-10T19:16:43.940624+0000","last_clean":"2024-09-10T19:16:43.940624+0000","last_became_active":"2024-09-10T19:16:24.863398+0000","last_became_peered":"2024-09-10T19:16:24.863398+0000","last_unstale":"2024-09-10T19:16:43.940624+0000","last_undegraded":"2024-09-10T19:16:43.940624+0000","last_fullsized":"2024-09-10T19:16:43.940624+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'6","last_scrub_stamp":"2024-09-10T19:13:55.370650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:13:55.370650+0000","objects_scrubbed":3,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.071693797000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":145,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-10T19:16:43.949775+0000","last_change":"2024-09-10T19:15:30.583958+0000","last_active":"2024-09-10T19:16:43.949775+0000","last_peered":"2024-09-10T19:16:43.949775+0000","last_clean":"2024-09-10T19:16:43.949775+0000","last_became_active":"2024-09-10T19:13:54.729917+0000","last_became_peered":"2024-09-10T19:13:54.729917+0000","last_unstale":"2024-09-10T19:16:43.949775+0000","last_undegraded":"2024-09-10T19:16:43.949775+0000","last_fullsized":"2024-09-10T19:16:43.949775+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:15:30.583878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:11:23.264793+0000","last_clean_scrub_stamp":"2024-09-10T19:15:30.583878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:16:56.396822+0000","scrub_duration":0.0015146960000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":125,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-10T19:16:34.797545+0000","last_change":"2024-09-10T19:12:51.508398+0000","last_active":"2024-09-10T19:16:34.797545+0000","last_peered":"2024-09-10T19:16:34.797545+0000","last_clean":"2024-09-10T19:16:34.797545+0000","last_became_active":"2024-09-10T19:10:14.054360+0000","last_became_peered":"2024-09-10T19:10:14.054360+0000","last_unstale":"2024-09-10T19:16:34.797545+0000","last_undegraded":"2024-09-10T19:16:34.797545+0000","last_fullsized":"2024-09-10T19:16:34.797545+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:12:51.508263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:12:51.508263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001748274,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"80'15","reported_seq":201,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-10T19:16:43.939441+0000","last_change":"2024-09-10T19:16:24.881305+0000","last_active":"2024-09-10T19:16:43.939441+0000","last_peered":"2024-09-10T19:16:43.939441+0000","last_clean":"2024-09-10T19:16:43.939441+0000","last_became_active":"2024-09-10T19:16:24.850702+0000","last_became_peered":"2024-09-10T19:16:24.850702+0000","last_unstale":"2024-09-10T19:16:43.939441+0000","last_undegraded":"2024-09-10T19:16:43.939441+0000","last_fullsized":"2024-09-10T19:16:43.939441+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"80'15","last_scrub_stamp":"2024-09-10T19:15:46.019949+0000","last_deep_scrub":"80'15","last_deep_scrub_stamp":"2024-09-10T19:11:41.719310+0000","last_clean_scrub_stamp":"2024-09-10T19:15:46.019949+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:16:48.263693+0000","scrub_duration":0.0021039190000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,5],"acting":[3,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":132,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-10T19:16:43.948748+0000","last_change":"2024-09-10T19:12:46.506129+0000","last_active":"2024-09-10T19:16:43.948748+0000","last_peered":"2024-09-10T19:16:43.948748+0000","last_clean":"2024-09-10T19:16:43.948748+0000","last_became_active":"2024-09-10T19:10:07.685870+0000","last_became_peered":"2024-09-10T19:10:07.685870+0000","last_unstale":"2024-09-10T19:16:43.948748+0000","last_undegraded":"2024-09-10T19:16:43.948748+0000","last_fullsized":"2024-09-10T19:16:43.948748+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:12:46.506071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:12:46.506071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001257935,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"117'10","reported_seq":237,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-10T19:16:43.950352+0000","last_change":"2024-09-10T19:16:23.844790+0000","last_active":"2024-09-10T19:16:43.950352+0000","last_peered":"2024-09-10T19:16:43.950352+0000","last_clean":"2024-09-10T19:16:43.950352+0000","last_became_active":"2024-09-10T19:13:54.724780+0000","last_became_peered":"2024-09-10T19:13:54.724780+0000","last_unstale":"2024-09-10T19:16:43.950352+0000","last_undegraded":"2024-09-10T19:16:43.950352+0000","last_fullsized":"2024-09-10T19:16:43.950352+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"117'10","last_scrub_stamp":"2024-09-10T19:16:23.844724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:16:23.844724+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:17:28.307512+0000","scrub_duration":0.012497955999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3],"acting":[5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":136,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-10T19:16:43.947143+0000","last_change":"2024-09-10T19:16:40.213422+0000","last_active":"2024-09-10T19:16:43.947143+0000","last_peered":"2024-09-10T19:16:43.947143+0000","last_clean":"2024-09-10T19:16:43.947143+0000","last_became_active":"2024-09-10T19:10:13.721035+0000","last_became_peered":"2024-09-10T19:10:13.721035+0000","last_unstale":"2024-09-10T19:16:43.947143+0000","last_undegraded":"2024-09-10T19:16:43.947143+0000","last_fullsized":"2024-09-10T19:16:43.947143+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:16:40.213321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:16:40.213321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:18:03.471480+0000","scrub_duration":0.0051408239999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"66'9","reported_seq":186,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-10T19:16:43.950426+0000","last_change":"2024-09-10T19:16:24.882653+0000","last_active":"2024-09-10T19:16:43.950426+0000","last_peered":"2024-09-10T19:16:43.950426+0000","last_clean":"2024-09-10T19:16:43.950426+0000","last_became_active":"2024-09-10T19:16:24.859740+0000","last_became_peered":"2024-09-10T19:16:24.859740+0000","last_unstale":"2024-09-10T19:16:43.950426+0000","last_undegraded":"2024-09-10T19:16:43.950426+0000","last_fullsized":"2024-09-10T19:16:43.950426+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'9","last_scrub_stamp":"2024-09-10T19:12:52.416748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:12:52.416748+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0047220839999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":142,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-10T19:16:43.945876+0000","last_change":"2024-09-10T19:16:37.243942+0000","last_active":"2024-09-10T19:16:43.945876+0000","last_peered":"2024-09-10T19:16:43.945876+0000","last_clean":"2024-09-10T19:16:43.945876+0000","last_became_active":"2024-09-10T19:10:07.688499+0000","last_became_peered":"2024-09-10T19:10:07.688499+0000","last_unstale":"2024-09-10T19:16:43.945876+0000","last_undegraded":"2024-09-10T19:16:43.945876+0000","last_fullsized":"2024-09-10T19:16:43.945876+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:16:37.243859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:16:37.243859+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:18:05.478908+0000","scrub_duration":0.0052374580000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"155'9","reported_seq":229,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-10T19:16:43.939804+0000","last_change":"2024-09-10T19:16:24.881133+0000","last_active":"2024-09-10T19:16:43.939804+0000","last_peered":"2024-09-10T19:16:43.939804+0000","last_clean":"2024-09-10T19:16:43.939804+0000","last_became_active":"2024-09-10T19:16:24.850850+0000","last_became_peered":"2024-09-10T19:16:24.850850+0000","last_unstale":"2024-09-10T19:16:43.939804+0000","last_undegraded":"2024-09-10T19:16:43.939804+0000","last_fullsized":"2024-09-10T19:16:43.939804+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'8","last_scrub_stamp":"2024-09-10T19:12:59.723712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:12:59.723712+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0083838109999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,4],"acting":[3,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":134,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-10T19:16:34.793019+0000","last_change":"2024-09-10T19:16:24.854571+0000","last_active":"2024-09-10T19:16:34.793019+0000","last_peered":"2024-09-10T19:16:34.793019+0000","last_clean":"2024-09-10T19:16:34.793019+0000","last_became_active":"2024-09-10T19:16:24.854280+0000","last_became_peered":"2024-09-10T19:16:24.854280+0000","last_unstale":"2024-09-10T19:16:34.793019+0000","last_undegraded":"2024-09-10T19:16:34.793019+0000","last_fullsized":"2024-09-10T19:16:34.793019+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:13:09.463001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:13:09.463001+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.012199573,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.12","version":"79'13","reported_seq":168,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-10T19:16:34.765610+0000","last_change":"2024-09-10T19:16:24.896575+0000","last_active":"2024-09-10T19:16:34.765610+0000","last_peered":"2024-09-10T19:16:34.765610+0000","last_clean":"2024-09-10T19:16:34.765610+0000","last_became_active":"2024-09-10T19:16:24.845450+0000","last_became_peered":"2024-09-10T19:16:24.845450+0000","last_unstale":"2024-09-10T19:16:34.765610+0000","last_undegraded":"2024-09-10T19:16:34.765610+0000","last_fullsized":"2024-09-10T19:16:34.765610+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'13","last_scrub_stamp":"2024-09-10T19:15:29.854117+0000","last_deep_scrub":"79'13","last_deep_scrub_stamp":"2024-09-10T19:11:22.072400+0000","last_clean_scrub_stamp":"2024-09-10T19:15:29.854117+0000","objects_scrubbed":4,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:16:53.523439+0000","scrub_duration":0.0066627300000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.14","version":"59'1","reported_seq":148,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-10T19:16:34.766148+0000","last_change":"2024-09-10T19:16:27.594895+0000","last_active":"2024-09-10T19:16:34.766148+0000","last_peered":"2024-09-10T19:16:34.766148+0000","last_clean":"2024-09-10T19:16:34.766148+0000","last_became_active":"2024-09-10T19:10:07.683263+0000","last_became_peered":"2024-09-10T19:10:07.683263+0000","last_unstale":"2024-09-10T19:16:34.766148+0000","last_undegraded":"2024-09-10T19:16:34.766148+0000","last_fullsized":"2024-09-10T19:16:34.766148+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'1","last_scrub_stamp":"2024-09-10T19:16:27.594788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:16:27.594788+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:17:46.479670+0000","scrub_duration":0.0091369840000000008,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":362,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":145,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-10T19:16:43.941847+0000","last_change":"2024-09-10T19:15:55.925313+0000","last_active":"2024-09-10T19:16:43.941847+0000","last_peered":"2024-09-10T19:16:43.941847+0000","last_clean":"2024-09-10T19:16:43.941847+0000","last_became_active":"2024-09-10T19:13:54.730185+0000","last_became_peered":"2024-09-10T19:13:54.730185+0000","last_unstale":"2024-09-10T19:16:43.941847+0000","last_undegraded":"2024-09-10T19:16:43.941847+0000","last_fullsized":"2024-09-10T19:16:43.941847+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:15:55.925228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:15:55.925228+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:17:11.471760+0000","scrub_duration":0.0017202299999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":143,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-10T19:16:43.945141+0000","last_change":"2024-09-10T19:16:23.273671+0000","last_active":"2024-09-10T19:16:43.945141+0000","last_peered":"2024-09-10T19:16:43.945141+0000","last_clean":"2024-09-10T19:16:43.945141+0000","last_became_active":"2024-09-10T19:13:54.714861+0000","last_became_peered":"2024-09-10T19:13:54.714861+0000","last_unstale":"2024-09-10T19:16:43.945141+0000","last_undegraded":"2024-09-10T19:16:43.945141+0000","last_fullsized":"2024-09-10T19:16:43.945141+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:16:23.273615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:16:23.273615+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:17:47.995106+0000","scrub_duration":0.0013972139999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,6],"acting":[0,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"66'11","reported_seq":179,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-10T19:16:34.791321+0000","last_change":"2024-09-10T19:15:53.883613+0000","last_active":"2024-09-10T19:16:34.791321+0000","last_peered":"2024-09-10T19:16:34.791321+0000","last_clean":"2024-09-10T19:16:34.791321+0000","last_became_active":"2024-09-10T19:13:54.720034+0000","last_became_peered":"2024-09-10T19:13:54.720034+0000","last_unstale":"2024-09-10T19:16:34.791321+0000","last_undegraded":"2024-09-10T19:16:34.791321+0000","last_fullsized":"2024-09-10T19:16:34.791321+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'11","last_scrub_stamp":"2024-09-10T19:15:53.883548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:15:53.883548+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:17:04.699778+0000","scrub_duration":0.001858326,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":82,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-10T19:16:43.950502+0000","last_change":"2024-09-10T19:13:54.723475+0000","last_active":"2024-09-10T19:16:43.950502+0000","last_peered":"2024-09-10T19:16:43.950502+0000","last_clean":"2024-09-10T19:16:43.950502+0000","last_became_active":"2024-09-10T19:13:54.722947+0000","last_became_peered":"2024-09-10T19:13:54.722947+0000","last_unstale":"2024-09-10T19:16:43.950502+0000","last_undegraded":"2024-09-10T19:16:43.950502+0000","last_fullsized":"2024-09-10T19:16:43.950502+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:12:44.249765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:11:10.031245+0000","last_clean_scrub_stamp":"2024-09-10T19:12:44.249765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,6],"acting":[5,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":92,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-10T19:16:43.948375+0000","last_change":"2024-09-10T19:15:36.666310+0000","last_active":"2024-09-10T19:16:43.948375+0000","last_peered":"2024-09-10T19:16:43.948375+0000","last_clean":"2024-09-10T19:16:43.948375+0000","last_became_active":"2024-09-10T19:13:54.729703+0000","last_became_peered":"2024-09-10T19:13:54.729703+0000","last_unstale":"2024-09-10T19:16:43.948375+0000","last_undegraded":"2024-09-10T19:16:43.948375+0000","last_fullsized":"2024-09-10T19:16:43.948375+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:15:36.666216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:15:36.666216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:17:03.635276+0000","scrub_duration":0.003489884,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":126,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-10T19:16:43.938611+0000","last_change":"2024-09-10T19:12:54.691886+0000","last_active":"2024-09-10T19:16:43.938611+0000","last_peered":"2024-09-10T19:16:43.938611+0000","last_clean":"2024-09-10T19:16:43.938611+0000","last_became_active":"2024-09-10T19:10:13.724201+0000","last_became_peered":"2024-09-10T19:10:13.724201+0000","last_unstale":"2024-09-10T19:16:43.938611+0000","last_undegraded":"2024-09-10T19:16:43.938611+0000","last_fullsized":"2024-09-10T19:16:43.938611+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:12:54.691810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:12:54.691810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.006702927,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"66'4","reported_seq":159,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-10T19:16:43.938801+0000","last_change":"2024-09-10T19:15:33.995417+0000","last_active":"2024-09-10T19:16:43.938801+0000","last_peered":"2024-09-10T19:16:43.938801+0000","last_clean":"2024-09-10T19:16:43.938801+0000","last_became_active":"2024-09-10T19:13:54.711125+0000","last_became_peered":"2024-09-10T19:13:54.711125+0000","last_unstale":"2024-09-10T19:16:43.938801+0000","last_undegraded":"2024-09-10T19:16:43.938801+0000","last_fullsized":"2024-09-10T19:16:43.938801+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'4","last_scrub_stamp":"2024-09-10T19:15:33.995274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:15:33.995274+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.011108425,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,4],"acting":[3,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":152,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-10T19:16:43.949856+0000","last_change":"2024-09-10T19:16:33.766560+0000","last_active":"2024-09-10T19:16:43.949856+0000","last_peered":"2024-09-10T19:16:43.949856+0000","last_clean":"2024-09-10T19:16:43.949856+0000","last_became_active":"2024-09-10T19:16:24.852460+0000","last_became_peered":"2024-09-10T19:16:24.852460+0000","last_unstale":"2024-09-10T19:16:43.949856+0000","last_undegraded":"2024-09-10T19:16:43.949856+0000","last_fullsized":"2024-09-10T19:16:43.949856+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:16:33.766474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:16:33.766474+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:18:01.711870+0000","scrub_duration":0.0042170250000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,3],"acting":[5,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":137,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-10T19:16:34.765404+0000","last_change":"2024-09-10T19:16:24.545188+0000","last_active":"2024-09-10T19:16:34.765404+0000","last_peered":"2024-09-10T19:16:34.765404+0000","last_clean":"2024-09-10T19:16:34.765404+0000","last_became_active":"2024-09-10T19:10:11.698536+0000","last_became_peered":"2024-09-10T19:10:11.698536+0000","last_unstale":"2024-09-10T19:16:34.765404+0000","last_undegraded":"2024-09-10T19:16:34.765404+0000","last_fullsized":"2024-09-10T19:16:34.765404+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:16:24.545134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:11:31.837944+0000","last_clean_scrub_stamp":"2024-09-10T19:16:24.545134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:17:54.187236+0000","scrub_duration":0.00120104,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":145,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-10T19:16:34.796537+0000","last_change":"2024-09-10T19:16:24.852651+0000","last_active":"2024-09-10T19:16:34.796537+0000","last_peered":"2024-09-10T19:16:34.796537+0000","last_clean":"2024-09-10T19:16:34.796537+0000","last_became_active":"2024-09-10T19:16:24.852356+0000","last_became_peered":"2024-09-10T19:16:24.852356+0000","last_unstale":"2024-09-10T19:16:34.796537+0000","last_undegraded":"2024-09-10T19:16:34.796537+0000","last_fullsized":"2024-09-10T19:16:34.796537+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:15:44.523530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:15:44.523530+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:16:44.534700+0000","scrub_duration":0.027001213999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,3],"acting":[7,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.11","version":"66'11","reported_seq":40,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-10T19:16:43.948226+0000","last_change":"2024-09-10T19:15:45.158623+0000","last_active":"2024-09-10T19:16:43.948226+0000","last_peered":"2024-09-10T19:16:43.948226+0000","last_clean":"2024-09-10T19:16:43.948226+0000","last_became_active":"2024-09-10T19:15:45.149864+0000","last_became_peered":"2024-09-10T19:15:45.149864+0000","last_unstale":"2024-09-10T19:16:43.948226+0000","last_undegraded":"2024-09-10T19:16:43.948226+0000","last_fullsized":"2024-09-10T19:16:43.948226+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'11","last_scrub_stamp":"2024-09-10T19:13:05.729854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:13:05.729854+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.010667395,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":151,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-10T19:16:34.766254+0000","last_change":"2024-09-10T19:16:24.855851+0000","last_active":"2024-09-10T19:16:34.766254+0000","last_peered":"2024-09-10T19:16:34.766254+0000","last_clean":"2024-09-10T19:16:34.766254+0000","last_became_active":"2024-09-10T19:16:24.855541+0000","last_became_peered":"2024-09-10T19:16:24.855541+0000","last_unstale":"2024-09-10T19:16:34.766254+0000","last_undegraded":"2024-09-10T19:16:34.766254+0000","last_fullsized":"2024-09-10T19:16:34.766254+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:16:23.543097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:13:05.876655+0000","last_clean_scrub_stamp":"2024-09-10T19:16:23.543097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:17:31.706207+0000","scrub_duration":0.001181779,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,0],"acting":[6,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":127,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-10T19:16:34.796334+0000","last_change":"2024-09-10T19:13:41.326028+0000","last_active":"2024-09-10T19:16:34.796334+0000","last_peered":"2024-09-10T19:16:34.796334+0000","last_clean":"2024-09-10T19:16:34.796334+0000","last_became_active":"2024-09-10T19:10:11.699013+0000","last_became_peered":"2024-09-10T19:10:11.699013+0000","last_unstale":"2024-09-10T19:16:34.796334+0000","last_undegraded":"2024-09-10T19:16:34.796334+0000","last_fullsized":"2024-09-10T19:16:34.796334+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:13:41.325941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:11:57.112092+0000","last_clean_scrub_stamp":"2024-09-10T19:13:41.325941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001598676,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":126,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-10T19:16:43.941815+0000","last_change":"2024-09-10T19:13:15.812326+0000","last_active":"2024-09-10T19:16:43.941815+0000","last_peered":"2024-09-10T19:16:43.941815+0000","last_clean":"2024-09-10T19:16:43.941815+0000","last_became_active":"2024-09-10T19:10:14.056172+0000","last_became_peered":"2024-09-10T19:10:14.056172+0000","last_unstale":"2024-09-10T19:16:43.941815+0000","last_undegraded":"2024-09-10T19:16:43.941815+0000","last_fullsized":"2024-09-10T19:16:43.941815+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:13:15.812244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:13:15.812244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0048044389999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"80'2","reported_seq":151,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-10T19:16:34.797503+0000","last_change":"2024-09-10T19:16:32.648844+0000","last_active":"2024-09-10T19:16:34.797503+0000","last_peered":"2024-09-10T19:16:34.797503+0000","last_clean":"2024-09-10T19:16:34.797503+0000","last_became_active":"2024-09-10T19:16:24.851220+0000","last_became_peered":"2024-09-10T19:16:24.851220+0000","last_unstale":"2024-09-10T19:16:34.797503+0000","last_undegraded":"2024-09-10T19:16:34.797503+0000","last_fullsized":"2024-09-10T19:16:34.797503+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"80'2","last_scrub_stamp":"2024-09-10T19:16:32.648738+0000","last_deep_scrub":"80'2","last_deep_scrub_stamp":"2024-09-10T19:11:30.272043+0000","last_clean_scrub_stamp":"2024-09-10T19:16:32.648738+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:17:38.161227+0000","scrub_duration":0.0066232920000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":369,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":369,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,3],"acting":[7,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":132,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-10T19:16:43.949352+0000","last_change":"2024-09-10T19:13:14.449643+0000","last_active":"2024-09-10T19:16:43.949352+0000","last_peered":"2024-09-10T19:16:43.949352+0000","last_clean":"2024-09-10T19:16:43.949352+0000","last_became_active":"2024-09-10T19:10:07.689877+0000","last_became_peered":"2024-09-10T19:10:07.689877+0000","last_unstale":"2024-09-10T19:16:43.949352+0000","last_undegraded":"2024-09-10T19:16:43.949352+0000","last_fullsized":"2024-09-10T19:16:43.949352+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:13:14.449593+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:13:14.449593+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001195542,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1e","version":"80'15","reported_seq":160,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-10T19:16:43.947389+0000","last_change":"2024-09-10T19:16:33.258114+0000","last_active":"2024-09-10T19:16:43.947389+0000","last_peered":"2024-09-10T19:16:43.947389+0000","last_clean":"2024-09-10T19:16:43.947389+0000","last_became_active":"2024-09-10T19:10:09.823061+0000","last_became_peered":"2024-09-10T19:10:09.823061+0000","last_unstale":"2024-09-10T19:16:43.947389+0000","last_undegraded":"2024-09-10T19:16:43.947389+0000","last_fullsized":"2024-09-10T19:16:43.947389+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"80'15","last_scrub_stamp":"2024-09-10T19:16:33.258031+0000","last_deep_scrub":"80'15","last_deep_scrub_stamp":"2024-09-10T19:11:25.883919+0000","last_clean_scrub_stamp":"2024-09-10T19:16:33.258031+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:17:57.303783+0000","scrub_duration":0.0040337309999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1f","version":"66'8","reported_seq":227,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-10T19:16:40.823691+0000","last_change":"2024-09-10T19:15:53.664303+0000","last_active":"2024-09-10T19:16:40.823691+0000","last_peered":"2024-09-10T19:16:40.823691+0000","last_clean":"2024-09-10T19:16:40.823691+0000","last_became_active":"2024-09-10T19:10:11.695427+0000","last_became_peered":"2024-09-10T19:10:11.695427+0000","last_unstale":"2024-09-10T19:16:40.823691+0000","last_undegraded":"2024-09-10T19:16:40.823691+0000","last_fullsized":"2024-09-10T19:16:40.823691+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'8","last_scrub_stamp":"2024-09-10T19:15:53.664208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:15:53.664208+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:17:15.336245+0000","scrub_duration":0.001849699,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":102,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-10T19:16:43.950092+0000","last_change":"2024-09-10T19:16:24.863000+0000","last_active":"2024-09-10T19:16:43.950092+0000","last_peered":"2024-09-10T19:16:43.950092+0000","last_clean":"2024-09-10T19:16:43.950092+0000","last_became_active":"2024-09-10T19:16:24.862695+0000","last_became_peered":"2024-09-10T19:16:24.862695+0000","last_unstale":"2024-09-10T19:16:43.950092+0000","last_undegraded":"2024-09-10T19:16:43.950092+0000","last_fullsized":"2024-09-10T19:16:43.950092+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:15:29.580605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:15:29.580605+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:16:49.739914+0000","scrub_duration":0.0047886710000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":142,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-10T19:16:43.939220+0000","last_change":"2024-09-10T19:16:41.881053+0000","last_active":"2024-09-10T19:16:43.939220+0000","last_peered":"2024-09-10T19:16:43.939220+0000","last_clean":"2024-09-10T19:16:43.939220+0000","last_became_active":"2024-09-10T19:10:07.682689+0000","last_became_peered":"2024-09-10T19:10:07.682689+0000","last_unstale":"2024-09-10T19:16:43.939220+0000","last_undegraded":"2024-09-10T19:16:43.939220+0000","last_fullsized":"2024-09-10T19:16:43.939220+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:16:41.880992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:16:41.880992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:17:54.061215+0000","scrub_duration":0.0044984990000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"80'16","reported_seq":207,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-10T19:16:34.766213+0000","last_change":"2024-09-10T19:15:47.700592+0000","last_active":"2024-09-10T19:16:34.766213+0000","last_peered":"2024-09-10T19:16:34.766213+0000","last_clean":"2024-09-10T19:16:34.766213+0000","last_became_active":"2024-09-10T19:13:54.713563+0000","last_became_peered":"2024-09-10T19:13:54.713563+0000","last_unstale":"2024-09-10T19:16:34.766213+0000","last_undegraded":"2024-09-10T19:16:34.766213+0000","last_fullsized":"2024-09-10T19:16:34.766213+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"80'16","last_scrub_stamp":"2024-09-10T19:15:47.700523+0000","last_deep_scrub":"80'16","last_deep_scrub_stamp":"2024-09-10T19:11:40.853611+0000","last_clean_scrub_stamp":"2024-09-10T19:15:47.700523+0000","objects_scrubbed":5,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:16:55.889805+0000","scrub_duration":0.0019149010000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,0],"acting":[6,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":148,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-10T19:16:43.947302+0000","last_change":"2024-09-10T19:16:34.222429+0000","last_active":"2024-09-10T19:16:43.947302+0000","last_peered":"2024-09-10T19:16:43.947302+0000","last_clean":"2024-09-10T19:16:43.947302+0000","last_became_active":"2024-09-10T19:16:24.834713+0000","last_became_peered":"2024-09-10T19:16:24.834713+0000","last_unstale":"2024-09-10T19:16:43.947302+0000","last_undegraded":"2024-09-10T19:16:43.947302+0000","last_fullsized":"2024-09-10T19:16:43.947302+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:16:34.222365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:11:28.822327+0000","last_clean_scrub_stamp":"2024-09-10T19:16:34.222365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:17:37.172660+0000","scrub_duration":0.014893412999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":134,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-10T19:16:36.961019+0000","last_change":"2024-09-10T19:16:36.960796+0000","last_active":"2024-09-10T19:16:36.961019+0000","last_peered":"2024-09-10T19:16:36.961019+0000","last_clean":"2024-09-10T19:16:36.961019+0000","last_became_active":"2024-09-10T19:10:14.053619+0000","last_became_peered":"2024-09-10T19:10:14.053619+0000","last_unstale":"2024-09-10T19:16:36.961019+0000","last_undegraded":"2024-09-10T19:16:36.961019+0000","last_fullsized":"2024-09-10T19:16:36.961019+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:16:36.960695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:16:36.960695+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:17:59.816942+0000","scrub_duration":0.011430734999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":141,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-10T19:16:34.765948+0000","last_change":"2024-09-10T19:16:30.666084+0000","last_active":"2024-09-10T19:16:34.765948+0000","last_peered":"2024-09-10T19:16:34.765948+0000","last_clean":"2024-09-10T19:16:34.765948+0000","last_became_active":"2024-09-10T19:10:07.685997+0000","last_became_peered":"2024-09-10T19:10:07.685997+0000","last_unstale":"2024-09-10T19:16:34.765948+0000","last_undegraded":"2024-09-10T19:16:34.765948+0000","last_fullsized":"2024-09-10T19:16:34.765948+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:16:30.666001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:16:30.666001+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:17:57.903923+0000","scrub_duration":0.0041927830000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1c","version":"78'22","reported_seq":91,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-10T19:16:43.939362+0000","last_change":"2024-09-10T19:16:24.870445+0000","last_active":"2024-09-10T19:16:43.939362+0000","last_peered":"2024-09-10T19:16:43.939362+0000","last_clean":"2024-09-10T19:16:43.939362+0000","last_became_active":"2024-09-10T19:16:24.845764+0000","last_became_peered":"2024-09-10T19:16:24.845764+0000","last_unstale":"2024-09-10T19:16:43.939362+0000","last_undegraded":"2024-09-10T19:16:43.939362+0000","last_fullsized":"2024-09-10T19:16:43.939362+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"78'22","last_scrub_stamp":"2024-09-10T19:15:37.702624+0000","last_deep_scrub":"78'22","last_deep_scrub_stamp":"2024-09-10T19:11:21.118937+0000","last_clean_scrub_stamp":"2024-09-10T19:15:37.702624+0000","objects_scrubbed":7,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:17:03.690099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":94,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-10T19:16:34.791145+0000","last_change":"2024-09-10T19:15:33.008615+0000","last_active":"2024-09-10T19:16:34.791145+0000","last_peered":"2024-09-10T19:16:34.791145+0000","last_clean":"2024-09-10T19:16:34.791145+0000","last_became_active":"2024-09-10T19:13:54.709055+0000","last_became_peered":"2024-09-10T19:13:54.709055+0000","last_unstale":"2024-09-10T19:16:34.791145+0000","last_undegraded":"2024-09-10T19:16:34.791145+0000","last_fullsized":"2024-09-10T19:16:34.791145+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:15:33.008542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:15:33.008542+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:16:51.091294+0000","scrub_duration":0.004557363,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,5],"acting":[4,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":126,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-10T19:16:43.940445+0000","last_change":"2024-09-10T19:12:51.767586+0000","last_active":"2024-09-10T19:16:43.940445+0000","last_peered":"2024-09-10T19:16:43.940445+0000","last_clean":"2024-09-10T19:16:43.940445+0000","last_became_active":"2024-09-10T19:10:14.053698+0000","last_became_peered":"2024-09-10T19:10:14.053698+0000","last_unstale":"2024-09-10T19:16:43.940445+0000","last_undegraded":"2024-09-10T19:16:43.940445+0000","last_fullsized":"2024-09-10T19:16:43.940445+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:12:51.767056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:12:51.767056+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.006796519,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":132,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-10T19:16:43.941931+0000","last_change":"2024-09-10T19:12:50.719167+0000","last_active":"2024-09-10T19:16:43.941931+0000","last_peered":"2024-09-10T19:16:43.941931+0000","last_clean":"2024-09-10T19:16:43.941931+0000","last_became_active":"2024-09-10T19:10:07.682811+0000","last_became_peered":"2024-09-10T19:10:07.682811+0000","last_unstale":"2024-09-10T19:16:43.941931+0000","last_undegraded":"2024-09-10T19:16:43.941931+0000","last_fullsized":"2024-09-10T19:16:43.941931+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:12:50.718663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:12:50.718663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0042138399999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1d","version":"66'12","reported_seq":182,"reported_epoch":161,"state":"active+clean","last_fresh":"2024-09-10T19:16:43.939254+0000","last_change":"2024-09-10T19:15:52.923561+0000","last_active":"2024-09-10T19:16:43.939254+0000","last_peered":"2024-09-10T19:16:43.939254+0000","last_clean":"2024-09-10T19:16:43.939254+0000","last_became_active":"2024-09-10T19:13:54.714629+0000","last_became_peered":"2024-09-10T19:13:54.714629+0000","last_unstale":"2024-09-10T19:16:43.939254+0000","last_undegraded":"2024-09-10T19:16:43.939254+0000","last_fullsized":"2024-09-10T19:16:43.939254+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'12","last_scrub_stamp":"2024-09-10T19:15:52.923498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:15:52.923498+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:17:20.423846+0000","scrub_duration":0.0019450450000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":146,"reported_epoch":160,"state":"active+clean","last_fresh":"2024-09-10T19:16:37.970156+0000","last_change":"2024-09-10T19:16:37.970105+0000","last_active":"2024-09-10T19:16:37.970156+0000","last_peered":"2024-09-10T19:16:37.970156+0000","last_clean":"2024-09-10T19:16:37.970156+0000","last_became_active":"2024-09-10T19:16:24.843013+0000","last_became_peered":"2024-09-10T19:16:24.843013+0000","last_unstale":"2024-09-10T19:16:37.970156+0000","last_undegraded":"2024-09-10T19:16:37.970156+0000","last_fullsized":"2024-09-10T19:16:37.970156+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:16:37.970030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:16:37.970030+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:17:58.177693+0000","scrub_duration":0.003625063,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":8,"num_pg":16,"stat_sum":{"num_bytes":2878078976,"num_objects":43916,"num_object_clones":0,"num_object_copies":131748,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":43916,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":43916,"num_write_kb":2810624,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8620736512,"data_stored":8620736512,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":43916,"ondisk_log_size":43916,"up":48,"acting":48,"num_store_stats":7},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":382,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":1528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":73,"ondisk_log_size":73,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":177,"num_object_clones":0,"num_object_copies":531,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":177,"num_whiteouts":0,"num_read":695,"num_read_kb":452,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":173,"num_bytes_recovered":3218,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":540672,"data_stored":14242,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":445,"ondisk_log_size":445,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":26,"num_read_kb":21,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":389,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":2002,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":9,"ondisk_log_size":9,"up":9,"acting":9,"num_store_stats":7},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1328,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":27,"num_read_kb":27,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":3984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4,"ondisk_log_size":4,"up":96,"acting":96,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79,"ondisk_log_size":79,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":54,"seq":231928234068,"num_pgs":77,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1513456,"kb_used_data":1216100,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":92231696,"statfs":{"total":95995035648,"available":94445256704,"internally_reserved":0,"allocated":1245286400,"data_stored":1244850987,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 19:15:59 2024","interfaces":[{"interface":"back","average":{"1min":0.69899999999999995,"5min":1.7569999999999999,"15min":0.94199999999999995},"min":{"1min":0.28999999999999998,"5min":0.215,"15min":0.215},"max":{"1min":2.7480000000000002,"5min":35.551000000000002,"15min":35.551000000000002},"last":0.89000000000000001},{"interface":"front","average":{"1min":0.69299999999999995,"5min":1.5760000000000001,"15min":0.88400000000000001},"min":{"1min":0.222,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":2.3570000000000002,"5min":35.503999999999998,"15min":35.503999999999998},"last":0.501}]},{"osd":1,"last update":"Tue Sep 10 19:16:21 2024","interfaces":[{"interface":"back","average":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"min":{"1min":0.28100000000000003,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":1.101,"5min":1.101,"15min":1.101},"last":2.7080000000000002},{"interface":"front","average":{"1min":0.70799999999999996,"5min":0.70799999999999996,"15min":0.70799999999999996},"min":{"1min":0.379,"5min":0.379,"15min":0.379},"max":{"1min":3.5819999999999999,"5min":3.5819999999999999,"15min":3.5819999999999999},"last":0.84899999999999998}]},{"osd":2,"last update":"Tue Sep 10 19:15:59 2024","interfaces":[{"interface":"back","average":{"1min":0.56299999999999994,"5min":1.054,"15min":0.70099999999999996},"min":{"1min":0.30199999999999999,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":1.131,"5min":31.940999999999999,"15min":31.940999999999999},"last":0.34699999999999998},{"interface":"front","average":{"1min":1.006,"5min":1.1679999999999999,"15min":0.75600000000000001},"min":{"1min":0.32700000000000001,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":3.613,"5min":25.056000000000001,"15min":25.056000000000001},"last":2.6739999999999999}]},{"osd":3,"last update":"Tue Sep 10 19:15:59 2024","interfaces":[{"interface":"back","average":{"1min":0.86299999999999999,"5min":1.742,"15min":0.93799999999999994},"min":{"1min":0.35699999999999998,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":2.4049999999999998,"5min":35.277999999999999,"15min":35.277999999999999},"last":2.778},{"interface":"front","average":{"1min":0.52200000000000002,"5min":1.657,"15min":0.89900000000000002},"min":{"1min":0.28499999999999998,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":1.151,"5min":36.512999999999998,"15min":36.512999999999998},"last":0.82199999999999995}]},{"osd":4,"last update":"Tue Sep 10 19:15:59 2024","interfaces":[{"interface":"back","average":{"1min":0.61399999999999999,"5min":0.54100000000000004,"15min":0.48299999999999998},"min":{"1min":0.254,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":3.0030000000000001,"5min":3.0030000000000001,"15min":3.0030000000000001},"last":0.44400000000000001},{"interface":"front","average":{"1min":0.437,"5min":0.52500000000000002,"15min":0.48099999999999998},"min":{"1min":0.245,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":0.65800000000000003,"5min":1.5049999999999999,"15min":1.5049999999999999},"last":2.7490000000000001}]},{"osd":5,"last update":"Tue Sep 10 19:15:59 2024","interfaces":[{"interface":"back","average":{"1min":0.55200000000000005,"5min":0.76800000000000002,"15min":0.57599999999999996},"min":{"1min":0.253,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":1.4470000000000001,"5min":11.621,"15min":11.621},"last":0.872},{"interface":"front","average":{"1min":0.67200000000000004,"5min":0.627,"15min":0.52600000000000002},"min":{"1min":0.32500000000000001,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":2.4199999999999999,"5min":2.8980000000000001,"15min":2.8980000000000001},"last":2.613}]},{"osd":6,"last update":"Tue Sep 10 19:15:59 2024","interfaces":[{"interface":"back","average":{"1min":0.73199999999999998,"5min":0.66600000000000004,"15min":0.55500000000000005},"min":{"1min":0.35299999999999998,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":2.3300000000000001,"5min":2.9089999999999998,"15min":2.9089999999999998},"last":0.47599999999999998},{"interface":"front","average":{"1min":0.63900000000000001,"5min":0.59099999999999997,"15min":0.52300000000000002},"min":{"1min":0.35799999999999998,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":2.7490000000000001,"5min":2.7490000000000001,"15min":2.7490000000000001},"last":0.92900000000000005}]}]},{"osd":6,"up_from":47,"seq":201863463001,"num_pgs":75,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1705508,"kb_used_data":1399832,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":92039644,"statfs":{"total":95995035648,"available":94248595456,"internally_reserved":0,"allocated":1433427968,"data_stored":1433004919,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2],"upper_bound":16},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 19:15:45 2024","interfaces":[{"interface":"back","average":{"1min":1.3700000000000001,"5min":2.1629999999999998,"15min":1.069},"min":{"1min":0.20100000000000001,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":12.334,"5min":112.988,"15min":112.988},"last":0.26900000000000002},{"interface":"front","average":{"1min":1.3839999999999999,"5min":2.1469999999999998,"15min":1.0780000000000001},"min":{"1min":0.249,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":12.319000000000001,"5min":113.18300000000001,"15min":113.18300000000001},"last":2.9969999999999999}]},{"osd":1,"last update":"Tue Sep 10 19:16:22 2024","interfaces":[{"interface":"back","average":{"1min":0.51400000000000001,"5min":0.51400000000000001,"15min":0.51400000000000001},"min":{"1min":0.28499999999999998,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":1.4099999999999999,"5min":1.4099999999999999,"15min":1.4099999999999999},"last":0.29899999999999999},{"interface":"front","average":{"1min":0.65400000000000003,"5min":0.65400000000000003,"15min":0.65400000000000003},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":2.996,"5min":2.996,"15min":2.996},"last":0.33300000000000002}]},{"osd":2,"last update":"Tue Sep 10 19:15:45 2024","interfaces":[{"interface":"back","average":{"1min":0.88400000000000001,"5min":1.956,"15min":1.0149999999999999},"min":{"1min":0.25,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":2.7709999999999999,"5min":112.864,"15min":112.864},"last":0.39000000000000001},{"interface":"front","average":{"1min":1.026,"5min":2.069,"15min":1.0289999999999999},"min":{"1min":0.28399999999999997,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":3.456,"5min":112.822,"15min":112.822},"last":3.0369999999999999}]},{"osd":3,"last update":"Tue Sep 10 19:15:45 2024","interfaces":[{"interface":"back","average":{"1min":1.9870000000000001,"5min":2.403,"15min":1.145},"min":{"1min":0.32100000000000001,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":12.539,"5min":125.56399999999999,"15min":125.56399999999999},"last":0.54700000000000004},{"interface":"front","average":{"1min":1.948,"5min":2.4900000000000002,"15min":1.2070000000000001},"min":{"1min":0.438,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":12.513,"5min":125.47,"15min":125.47},"last":0.36199999999999999}]},{"osd":4,"last update":"Tue Sep 10 19:15:45 2024","interfaces":[{"interface":"back","average":{"1min":0.54300000000000004,"5min":0.49099999999999999,"15min":0.39500000000000002},"min":{"1min":0.27600000000000002,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":1.3220000000000001,"5min":3.3410000000000002,"15min":3.3410000000000002},"last":3.0630000000000002},{"interface":"front","average":{"1min":0.59899999999999998,"5min":0.53300000000000003,"15min":0.42699999999999999},"min":{"1min":0.22500000000000001,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":1.831,"5min":2.0710000000000002,"15min":2.0710000000000002},"last":1.276}]},{"osd":5,"last update":"Tue Sep 10 19:15:45 2024","interfaces":[{"interface":"back","average":{"1min":0.60499999999999998,"5min":0.98699999999999999,"15min":0.57899999999999996},"min":{"1min":0.32600000000000001,"5min":0.248,"15min":0.248},"max":{"1min":1.3540000000000001,"5min":44.765000000000001,"15min":44.765000000000001},"last":3.1099999999999999},{"interface":"front","average":{"1min":0.92300000000000004,"5min":1.1160000000000001,"15min":0.621},"min":{"1min":0.28899999999999998,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":4.0430000000000001,"5min":43.036000000000001,"15min":43.036000000000001},"last":2.5489999999999999}]},{"osd":7,"last update":"Tue Sep 10 19:16:05 2024","interfaces":[{"interface":"back","average":{"1min":0.72099999999999997,"5min":0.57699999999999996,"15min":0.48999999999999999},"min":{"1min":0.30599999999999999,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":3.504,"5min":3.504,"15min":3.504},"last":1.4430000000000001},{"interface":"front","average":{"1min":0.64700000000000002,"5min":0.63800000000000001,"15min":0.50900000000000001},"min":{"1min":0.32500000000000001,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":1.218,"5min":2.3260000000000001,"15min":2.3260000000000001},"last":0.57399999999999995}]}]},{"osd":5,"up_from":39,"seq":167503724638,"num_pgs":72,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1836040,"kb_used_data":1530368,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":91909112,"statfs":{"total":95995035648,"available":94114930688,"internally_reserved":0,"allocated":1567096832,"data_stored":1566690624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,3],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 19:16:16 2024","interfaces":[{"interface":"back","average":{"1min":0.94799999999999995,"5min":0.876,"15min":0.61099999999999999},"min":{"1min":0.22500000000000001,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":3.7909999999999999,"5min":5.851,"15min":5.851},"last":1.524},{"interface":"front","average":{"1min":1.008,"5min":0.76800000000000002,"15min":0.57399999999999995},"min":{"1min":0.24099999999999999,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":5.923,"5min":5.923,"15min":5.923},"last":0.41299999999999998}]},{"osd":1,"last update":"Tue Sep 10 19:16:22 2024","interfaces":[{"interface":"back","average":{"1min":0.65200000000000002,"5min":0.65200000000000002,"15min":0.65200000000000002},"min":{"1min":0.27400000000000002,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":2.8370000000000002,"5min":2.8370000000000002,"15min":2.8370000000000002},"last":0.47699999999999998},{"interface":"front","average":{"1min":0.503,"5min":0.503,"15min":0.503},"min":{"1min":0.27100000000000002,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":1.4790000000000001,"5min":1.4790000000000001,"15min":1.4790000000000001},"last":2.5369999999999999}]},{"osd":2,"last update":"Tue Sep 10 19:16:16 2024","interfaces":[{"interface":"back","average":{"1min":0.87,"5min":0.73999999999999999,"15min":0.56599999999999995},"min":{"1min":0.26100000000000001,"5min":0.253,"15min":0.253},"max":{"1min":4.3159999999999998,"5min":4.3159999999999998,"15min":4.3159999999999998},"last":1.556},{"interface":"front","average":{"1min":1.0329999999999999,"5min":0.88800000000000001,"15min":0.625},"min":{"1min":0.29199999999999998,"5min":0.224,"15min":0.224},"max":{"1min":3.1110000000000002,"5min":6.3789999999999996,"15min":6.3789999999999996},"last":0.504}]},{"osd":3,"last update":"Tue Sep 10 19:16:16 2024","interfaces":[{"interface":"back","average":{"1min":0.751,"5min":0.70999999999999996,"15min":0.58099999999999996},"min":{"1min":0.309,"5min":0.224,"15min":0.224},"max":{"1min":2.3460000000000001,"5min":3.351,"15min":3.351},"last":1.577},{"interface":"front","average":{"1min":0.84599999999999997,"5min":0.755,"15min":0.60499999999999998},"min":{"1min":0.33600000000000002,"5min":0.248,"15min":0.248},"max":{"1min":2.8580000000000001,"5min":5.3869999999999996,"15min":5.3869999999999996},"last":0.45200000000000001}]},{"osd":4,"last update":"Tue Sep 10 19:16:16 2024","interfaces":[{"interface":"back","average":{"1min":0.54500000000000004,"5min":0.60199999999999998,"15min":0.441},"min":{"1min":0.25800000000000001,"5min":0.23699999999999999,"15min":0.20699999999999999},"max":{"1min":1.6359999999999999,"5min":4.3280000000000003,"15min":4.3280000000000003},"last":2.5680000000000001},{"interface":"front","average":{"1min":0.58799999999999997,"5min":0.66800000000000004,"15min":0.46999999999999997},"min":{"1min":0.31900000000000001,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":2.02,"5min":5.8899999999999997,"15min":5.8899999999999997},"last":1.7789999999999999}]},{"osd":6,"last update":"Tue Sep 10 19:15:44 2024","interfaces":[{"interface":"back","average":{"1min":0.81599999999999995,"5min":0.77200000000000002,"15min":0.53100000000000003},"min":{"1min":0.29099999999999998,"5min":0.27100000000000002,"15min":0.223},"max":{"1min":2.9399999999999999,"5min":6.617,"15min":6.617},"last":3.6509999999999998},{"interface":"front","average":{"1min":0.80600000000000005,"5min":0.73199999999999998,"15min":0.51400000000000001},"min":{"1min":0.33800000000000002,"5min":0.26800000000000002,"15min":0.25},"max":{"1min":2.794,"5min":5.9349999999999996,"15min":5.9349999999999996},"last":1.194}]},{"osd":7,"last update":"Tue Sep 10 19:16:02 2024","interfaces":[{"interface":"back","average":{"1min":0.72899999999999998,"5min":0.70799999999999996,"15min":0.57599999999999996},"min":{"1min":0.307,"5min":0.307,"15min":0.29099999999999998},"max":{"1min":2.8170000000000002,"5min":5.9660000000000002,"15min":5.9660000000000002},"last":3.097},{"interface":"front","average":{"1min":0.56000000000000005,"5min":0.59899999999999998,"15min":0.51900000000000002},"min":{"1min":0.34899999999999998,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":1.452,"5min":2.415,"15min":2.415},"last":4.9640000000000004}]}]},{"osd":4,"up_from":32,"seq":137438953570,"num_pgs":70,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1838352,"kb_used_data":1536836,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":91906800,"statfs":{"total":95995035648,"available":94112563200,"internally_reserved":0,"allocated":1573720064,"data_stored":1573315928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,2,5,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 19:16:02 2024","interfaces":[{"interface":"back","average":{"1min":3.3929999999999998,"5min":1.1850000000000001,"15min":0.753},"min":{"1min":0.22500000000000001,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":32.533000000000001,"5min":32.533000000000001,"15min":32.533000000000001},"last":0.42999999999999999},{"interface":"front","average":{"1min":3.6709999999999998,"5min":1.2490000000000001,"15min":0.76600000000000001},"min":{"1min":0.27200000000000002,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":32.512999999999998,"5min":32.512999999999998,"15min":32.512999999999998},"last":0.34399999999999997}]},{"osd":1,"last update":"Tue Sep 10 19:16:20 2024","interfaces":[{"interface":"back","average":{"1min":0.628,"5min":0.628,"15min":0.628},"min":{"1min":0.28299999999999997,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":2.5409999999999999,"5min":2.5409999999999999,"15min":2.5409999999999999},"last":0.54300000000000004},{"interface":"front","average":{"1min":0.67600000000000005,"5min":0.67600000000000005,"15min":0.67600000000000005},"min":{"1min":0.32100000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":2.5219999999999998,"5min":2.5219999999999998,"15min":2.5219999999999998},"last":0.497}]},{"osd":2,"last update":"Tue Sep 10 19:16:02 2024","interfaces":[{"interface":"back","average":{"1min":1.3959999999999999,"5min":0.78800000000000003,"15min":0.62},"min":{"1min":0.315,"5min":0.214,"15min":0.214},"max":{"1min":6.8869999999999996,"5min":6.8869999999999996,"15min":6.8869999999999996},"last":0.47499999999999998},{"interface":"front","average":{"1min":1.361,"5min":0.80400000000000005,"15min":0.623},"min":{"1min":0.308,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":7.085,"5min":7.085,"15min":7.085},"last":0.39900000000000002}]},{"osd":3,"last update":"Tue Sep 10 19:16:02 2024","interfaces":[{"interface":"back","average":{"1min":3.1200000000000001,"5min":1.1970000000000001,"15min":0.748},"min":{"1min":0.39500000000000002,"5min":0.252,"15min":0.252},"max":{"1min":35.451999999999998,"5min":35.451999999999998,"15min":35.451999999999998},"last":0.36799999999999999},{"interface":"front","average":{"1min":2.859,"5min":1.232,"15min":0.77600000000000002},"min":{"1min":0.35599999999999998,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":32.880000000000003,"5min":32.880000000000003,"15min":32.880000000000003},"last":0.56200000000000006}]},{"osd":5,"last update":"Tue Sep 10 19:16:22 2024","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":0.51900000000000002,"15min":0.42799999999999999},"min":{"1min":0.255,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":1.2030000000000001,"5min":3.629,"15min":3.629},"last":4.1600000000000001},{"interface":"front","average":{"1min":0.55100000000000005,"5min":0.51100000000000001,"15min":0.438},"min":{"1min":0.27600000000000002,"5min":0.248,"15min":0.223},"max":{"1min":1.8280000000000001,"5min":2.1880000000000002,"15min":2.1880000000000002},"last":3.4159999999999999}]},{"osd":6,"last update":"Tue Sep 10 19:15:48 2024","interfaces":[{"interface":"back","average":{"1min":0.61399999999999999,"5min":0.64000000000000001,"15min":0.48799999999999999},"min":{"1min":0.30599999999999999,"5min":0.253,"15min":0.253},"max":{"1min":1.643,"5min":3.964,"15min":3.964},"last":0.45400000000000001},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.59499999999999997,"15min":0.47399999999999998},"min":{"1min":0.31900000000000001,"5min":0.25,"15min":0.25},"max":{"1min":1.4319999999999999,"5min":3.0129999999999999,"15min":3.0129999999999999},"last":0.52200000000000002}]},{"osd":7,"last update":"Tue Sep 10 19:16:06 2024","interfaces":[{"interface":"back","average":{"1min":0.60299999999999998,"5min":0.92700000000000005,"15min":0.65900000000000003},"min":{"1min":0.308,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":1.2250000000000001,"5min":23.081,"15min":23.081},"last":2.0960000000000001},{"interface":"front","average":{"1min":0.77900000000000003,"5min":0.91200000000000003,"15min":0.63900000000000001},"min":{"1min":0.35699999999999998,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":3.1059999999999999,"5min":22.666,"15min":22.666},"last":1.871}]}]},{"osd":3,"up_from":26,"seq":111669149799,"num_pgs":78,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1892988,"kb_used_data":1591468,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":91852164,"statfs":{"total":95995035648,"available":94056615936,"internally_reserved":0,"allocated":1629663232,"data_stored":1629228221,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 19:16:35 2024","interfaces":[{"interface":"back","average":{"1min":0.57199999999999995,"5min":1.3240000000000001,"15min":0.70599999999999996},"min":{"1min":0.223,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":3.9529999999999998,"5min":17.518000000000001,"15min":17.518000000000001},"last":0.20200000000000001},{"interface":"front","average":{"1min":0.91700000000000004,"5min":1.6060000000000001,"15min":0.80800000000000005},"min":{"1min":0.20599999999999999,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":5.1539999999999999,"5min":17.609000000000002,"15min":17.609000000000002},"last":0.442}]},{"osd":1,"last update":"Tue Sep 10 19:16:20 2024","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"min":{"1min":0.248,"5min":0.248,"15min":0.248},"max":{"1min":2.6059999999999999,"5min":2.6059999999999999,"15min":2.6059999999999999},"last":0.42099999999999999},{"interface":"front","average":{"1min":1.569,"5min":1.569,"15min":1.569},"min":{"1min":0.27700000000000002,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":17.454999999999998,"5min":17.454999999999998,"15min":17.454999999999998},"last":0.63200000000000001}]},{"osd":2,"last update":"Tue Sep 10 19:16:35 2024","interfaces":[{"interface":"back","average":{"1min":0.98199999999999998,"5min":1.04,"15min":0.63500000000000001},"min":{"1min":0.23200000000000001,"5min":0.186,"15min":0.186},"max":{"1min":5.3029999999999999,"5min":5.3029999999999999,"15min":5.3029999999999999},"last":0.45500000000000002},{"interface":"front","average":{"1min":0.52800000000000002,"5min":1.2929999999999999,"15min":0.70599999999999996},"min":{"1min":0.249,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":3.2879999999999998,"5min":17.806000000000001,"15min":17.806000000000001},"last":0.37}]},{"osd":4,"last update":"Tue Sep 10 19:16:01 2024","interfaces":[{"interface":"back","average":{"1min":1.579,"5min":1.278,"15min":0.78900000000000003},"min":{"1min":0.26400000000000001,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":17.834,"5min":17.834,"15min":17.834},"last":0.80900000000000005},{"interface":"front","average":{"1min":1.8799999999999999,"5min":1.6040000000000001,"15min":0.90000000000000002},"min":{"1min":0.24099999999999999,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":17.553000000000001,"5min":17.553000000000001,"15min":17.553000000000001},"last":0.69199999999999995}]},{"osd":5,"last update":"Tue Sep 10 19:16:20 2024","interfaces":[{"interface":"back","average":{"1min":1.3680000000000001,"5min":1.085,"15min":0.73999999999999999},"min":{"1min":0.38200000000000001,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":17.725000000000001,"5min":17.725000000000001,"15min":17.725000000000001},"last":1.637},{"interface":"front","average":{"1min":1.7090000000000001,"5min":1.464,"15min":0.86199999999999999},"min":{"1min":0.435,"5min":0.22,"15min":0.22},"max":{"1min":17.582999999999998,"5min":17.582999999999998,"15min":17.582999999999998},"last":2.7839999999999998}]},{"osd":6,"last update":"Tue Sep 10 19:16:44 2024","interfaces":[{"interface":"back","average":{"1min":0.77700000000000002,"5min":1.373,"15min":0.83999999999999997},"min":{"1min":0.34100000000000003,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":3.2589999999999999,"5min":17.861000000000001,"15min":17.861000000000001},"last":0.51000000000000001},{"interface":"front","average":{"1min":0.89600000000000002,"5min":1.1439999999999999,"15min":0.752},"min":{"1min":0.37,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":4.7569999999999997,"5min":17.687000000000001,"15min":17.687000000000001},"last":0.65800000000000003}]},{"osd":7,"last update":"Tue Sep 10 19:16:10 2024","interfaces":[{"interface":"back","average":{"1min":1.627,"5min":1.204,"15min":0.78800000000000003},"min":{"1min":0.33300000000000002,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":18.093,"5min":18.093,"15min":18.093},"last":1.2410000000000001},{"interface":"front","average":{"1min":1.496,"5min":1.3120000000000001,"15min":0.82599999999999996},"min":{"1min":0.42599999999999999,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":17.780999999999999,"5min":17.780999999999999,"15min":17.780999999999999},"last":1.502}]}]},{"osd":2,"up_from":0,"seq":77309411436,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":0,"seq":592705486866,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":8,"seq":34359738486,"num_pgs":72,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1326436,"kb_used_data":1020760,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":92418716,"statfs":{"total":95995035648,"available":94636765184,"internally_reserved":0,"allocated":1045258240,"data_stored":1044835607,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Sep 10 19:16:20 2024","interfaces":[{"interface":"back","average":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"min":{"1min":0.19,"5min":0.19,"15min":0.19},"max":{"1min":0.68400000000000005,"5min":0.68400000000000005,"15min":0.68400000000000005},"last":0.24199999999999999},{"interface":"front","average":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"min":{"1min":0.26100000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":1.0900000000000001,"5min":1.0900000000000001,"15min":1.0900000000000001},"last":0.25900000000000001}]},{"osd":2,"last update":"Tue Sep 10 19:16:20 2024","interfaces":[{"interface":"back","average":{"1min":0.98499999999999999,"5min":1.01,"15min":0.59299999999999997},"min":{"1min":0.26500000000000001,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":4.2839999999999998,"5min":6.4640000000000004,"15min":6.4640000000000004},"last":0.27400000000000002},{"interface":"front","average":{"1min":0.39800000000000002,"5min":0.98999999999999999,"15min":0.57899999999999996},"min":{"1min":0.25900000000000001,"5min":0.187,"15min":0.187},"max":{"1min":0.70899999999999996,"5min":9.9399999999999995,"15min":9.9399999999999995},"last":0.35099999999999998}]},{"osd":3,"last update":"Tue Sep 10 19:16:43 2024","interfaces":[{"interface":"back","average":{"1min":0.59399999999999997,"5min":1.0740000000000001,"15min":0.63800000000000001},"min":{"1min":0.29199999999999998,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":2.5049999999999999,"5min":6.5590000000000002,"15min":6.5590000000000002},"last":0.38400000000000001},{"interface":"front","average":{"1min":0.34200000000000003,"5min":1.26,"15min":0.68899999999999995},"min":{"1min":0.27200000000000002,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":0.42199999999999999,"5min":7.2140000000000004,"15min":7.2140000000000004},"last":0.40200000000000002}]},{"osd":4,"last update":"Tue Sep 10 19:15:57 2024","interfaces":[{"interface":"back","average":{"1min":0.622,"5min":0.78300000000000003,"15min":0.63700000000000001},"min":{"1min":0.35599999999999998,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":1.72,"5min":5.8499999999999996,"15min":5.8499999999999996},"last":0.85999999999999999},{"interface":"front","average":{"1min":0.83099999999999996,"5min":0.81100000000000005,"15min":0.64500000000000002},"min":{"1min":0.27900000000000003,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":3.8180000000000001,"5min":3.8180000000000001,"15min":3.8180000000000001},"last":0.93300000000000005}]},{"osd":5,"last update":"Tue Sep 10 19:16:20 2024","interfaces":[{"interface":"back","average":{"1min":0.56000000000000005,"5min":0.93799999999999994,"15min":0.70199999999999996},"min":{"1min":0.309,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":1.1870000000000001,"5min":5.3860000000000001,"15min":5.3860000000000001},"last":2.6240000000000001},{"interface":"front","average":{"1min":0.69899999999999995,"5min":0.90900000000000003,"15min":0.68600000000000005},"min":{"1min":0.41199999999999998,"5min":0.253,"15min":0.253},"max":{"1min":1.9610000000000001,"5min":5.9749999999999996,"15min":5.9749999999999996},"last":0.36699999999999999}]},{"osd":6,"last update":"Tue Sep 10 19:16:43 2024","interfaces":[{"interface":"back","average":{"1min":0.75700000000000001,"5min":0.98099999999999998,"15min":0.70199999999999996},"min":{"1min":0.32700000000000001,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":3.0819999999999999,"5min":5.8730000000000002,"15min":5.8730000000000002},"last":3.0819999999999999},{"interface":"front","average":{"1min":0.73899999999999999,"5min":0.98699999999999999,"15min":0.71699999999999997},"min":{"1min":0.38500000000000001,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":2.125,"5min":5.2850000000000001,"15min":5.2850000000000001},"last":1.9730000000000001}]},{"osd":7,"last update":"Tue Sep 10 19:16:08 2024","interfaces":[{"interface":"back","average":{"1min":0.79200000000000004,"5min":0.94999999999999996,"15min":0.70199999999999996},"min":{"1min":0.375,"5min":0.318,"15min":0.318},"max":{"1min":3.8410000000000002,"5min":5.1219999999999999,"15min":5.1219999999999999},"last":0.41499999999999998},{"interface":"front","average":{"1min":0.83999999999999997,"5min":0.91300000000000003,"15min":0.67900000000000005},"min":{"1min":0.40300000000000002,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":1.9870000000000001,"5min":6.0469999999999997,"15min":6.0469999999999997},"last":0.42999999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":966,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1236,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1236,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":362,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":1760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":2118,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1018,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":2118,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1540,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1942,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":2162,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1072758784,"data_stored":1072758784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1631125504,"data_stored":1631125504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1615462400,"data_stored":1615462400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1618739200,"data_stored":1618739200,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1438777344,"data_stored":1438777344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1243873280,"data_stored":1243873280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T19:16:49.743 INFO:tasks.thrashosds.thrasher:Setting 8.3 to [7, 3, 5] 2024-09-10T19:16:49.743 INFO:tasks.thrashosds.thrasher:cmd ['osd', 'pg-upmap', '8.3', '7', '3', '5'] 2024-09-10T19:16:49.743 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph osd pg-upmap 8.3 7 3 5 2024-09-10T19:16:49.778 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:16:49.811 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:16:49.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:16:49.812 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:16:49.824 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: 25 16 51937 51921 129.781 96.875 0.0110082 0.007701 2024-09-10T19:16:49.913 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:16:49.943 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:16:49.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:16:49.944 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:16:50.045 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:16:50.083 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:16:50.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:16:50.085 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:16:50.185 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:16:50.209 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:16:49 smithi183 ceph-mon[41562]: from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:16:50.209 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:16:49 smithi183 ceph-mon[41562]: osdmap e164: 8 total, 8 up, 6 in 2024-09-10T19:16:50.210 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:16:49 smithi183 ceph-mon[29736]: from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:16:50.210 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:16:49 smithi183 ceph-mon[29736]: osdmap e164: 8 total, 8 up, 6 in 2024-09-10T19:16:50.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:16:49 smithi191 ceph-mon[32884]: from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:16:50.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:16:49 smithi191 ceph-mon[32884]: osdmap e164: 8 total, 8 up, 6 in 2024-09-10T19:16:50.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:16:50.222 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:16:50.223 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:16:50.323 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:16:50.365 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:16:50.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:16:50.366 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:16:50.366 INFO:teuthology.orchestra.run.smithi183.stdout:osd.0: {} 2024-09-10T19:16:50.367 INFO:teuthology.orchestra.run.smithi183.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-10T19:16:50.396 INFO:teuthology.orchestra.run.smithi183.stdout:osd.1: {} 2024-09-10T19:16:50.396 INFO:teuthology.orchestra.run.smithi183.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-10T19:16:50.397 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:16:50.426 INFO:teuthology.orchestra.run.smithi183.stdout:osd.2: {} 2024-09-10T19:16:50.426 INFO:teuthology.orchestra.run.smithi183.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-10T19:16:50.442 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:16:50.452 INFO:teuthology.orchestra.run.smithi183.stdout:osd.3: {} 2024-09-10T19:16:50.453 INFO:teuthology.orchestra.run.smithi183.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-10T19:16:50.467 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:16:50.483 INFO:teuthology.orchestra.run.smithi183.stdout:osd.4: {} 2024-09-10T19:16:50.483 INFO:teuthology.orchestra.run.smithi183.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-10T19:16:50.504 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:16:50.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:16:50.505 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:16:50.506 INFO:teuthology.orchestra.run.smithi183.stdout:osd.5: {} 2024-09-10T19:16:50.507 INFO:teuthology.orchestra.run.smithi183.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-10T19:16:50.534 INFO:teuthology.orchestra.run.smithi183.stdout:osd.6: {} 2024-09-10T19:16:50.534 INFO:teuthology.orchestra.run.smithi183.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-10T19:16:50.557 INFO:teuthology.orchestra.run.smithi183.stdout:osd.7: {} 2024-09-10T19:16:50.557 INFO:teuthology.orchestra.run.smithi183.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-10T19:16:50.599 INFO:teuthology.orchestra.run.smithi191.stdout:{ 2024-09-10T19:16:50.599 INFO:teuthology.orchestra.run.smithi191.stdout: "size": 20, 2024-09-10T19:16:50.599 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 600, 2024-09-10T19:16:50.599 INFO:teuthology.orchestra.run.smithi191.stdout: "ops": [ 2024-09-10T19:16:50.599 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:50.599 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.14781.0:217250 7.d 7:bb1ab8d6:::benchmark_data_smithi183_84160_object76869:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e129)", 2024-09-10T19:16:50.600 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-10T19:14:55.829582+0000", 2024-09-10T19:16:50.600 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 114.763027819, 2024-09-10T19:16:50.600 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.035075279, 2024-09-10T19:16:50.600 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-10T19:16:50.600 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "reached pg", 2024-09-10T19:16:50.600 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-10T19:16:50.600 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.14781", 2024-09-10T19:16:50.600 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.183:0/2025776420", 2024-09-10T19:16:50.600 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 217250 2024-09-10T19:16:50.600 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:50.600 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-10T19:16:50.600 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:50.600 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-10T19:16:50.601 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:55.829582+0000", 2024-09-10T19:16:50.601 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:16:50.601 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:50.601 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:50.601 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-10T19:16:50.601 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:55.829582+0000", 2024-09-10T19:16:50.601 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:16:50.601 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:50.601 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:50.601 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-10T19:16:50.601 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:55.829581+0000", 2024-09-10T19:16:50.601 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999986 2024-09-10T19:16:50.601 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:50.602 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:50.602 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-10T19:16:50.602 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:55.829586+0000", 2024-09-10T19:16:50.602 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.5990000000000004e-06 2024-09-10T19:16:50.602 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:50.602 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:50.602 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-10T19:16:50.602 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:55.829597+0000", 2024-09-10T19:16:50.602 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.1605e-05 2024-09-10T19:16:50.602 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:50.602 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:50.602 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-10T19:16:50.602 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:55.829602+0000", 2024-09-10T19:16:50.602 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.617e-06 2024-09-10T19:16:50.602 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:50.603 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:50.603 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:16:50.603 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:55.829644+0000", 2024-09-10T19:16:50.603 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.2304e-05 2024-09-10T19:16:50.603 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:50.603 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:50.603 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-10T19:16:50.603 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:55.829828+0000", 2024-09-10T19:16:50.603 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000184351 2024-09-10T19:16:50.603 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:50.603 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:50.603 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0,3", 2024-09-10T19:16:50.603 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:55.829947+0000", 2024-09-10T19:16:50.603 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000118633 2024-09-10T19:16:50.604 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:50.604 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:50.604 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-09-10T19:16:50.604 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:55.832657+0000", 2024-09-10T19:16:50.604 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0027095650000000002 2024-09-10T19:16:50.604 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:50.604 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:50.604 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:16:50.604 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:55.843209+0000", 2024-09-10T19:16:50.604 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.010552844 2024-09-10T19:16:50.604 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:50.604 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:50.604 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for peered", 2024-09-10T19:16:50.604 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:55.843215+0000", 2024-09-10T19:16:50.605 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.9649999999999996e-06 2024-09-10T19:16:50.605 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:50.605 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:50.605 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:16:50.605 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:56.864581+0000", 2024-09-10T19:16:50.605 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.0213653039999999 2024-09-10T19:16:50.605 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:50.605 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:50.605 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-10T19:16:50.605 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:56.864657+0000", 2024-09-10T19:16:50.605 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.6687999999999995e-05 2024-09-10T19:16:50.605 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:16:50.605 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-10T19:16:50.606 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:16:50.606 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:50.606 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:50.606 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.14781.0:217261 7.d 7:b4bd2e3c:::benchmark_data_smithi183_84160_object76880:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e129)", 2024-09-10T19:16:50.606 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-10T19:14:55.841007+0000", 2024-09-10T19:16:50.606 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 114.751602979, 2024-09-10T19:16:50.606 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.033299427, 2024-09-10T19:16:50.606 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-10T19:16:50.606 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T19:16:50.606 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-10T19:16:50.606 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.14781", 2024-09-10T19:16:50.606 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.183:0/2025776420", 2024-09-10T19:16:50.606 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 217261 2024-09-10T19:16:50.607 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:50.607 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-10T19:16:50.607 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:50.607 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-10T19:16:50.607 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:55.841007+0000", 2024-09-10T19:16:50.607 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:16:50.607 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:50.607 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:50.607 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-10T19:16:50.607 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:55.841007+0000", 2024-09-10T19:16:50.607 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:16:50.607 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:50.607 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:50.607 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-10T19:16:50.607 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:55.841006+0000", 2024-09-10T19:16:50.608 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.999999 2024-09-10T19:16:50.608 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:50.608 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:50.608 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-10T19:16:50.608 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:55.841011+0000", 2024-09-10T19:16:50.608 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.2859999999999999e-06 2024-09-10T19:16:50.608 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:50.608 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:50.608 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-10T19:16:50.608 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:55.841022+0000", 2024-09-10T19:16:50.608 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.0404999999999999e-05 2024-09-10T19:16:50.608 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:50.608 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:50.608 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-10T19:16:50.608 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:55.841027+0000", 2024-09-10T19:16:50.609 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.1869999999999996e-06 2024-09-10T19:16:50.609 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:50.609 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:50.609 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:16:50.609 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:55.843328+0000", 2024-09-10T19:16:50.609 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.002301179 2024-09-10T19:16:50.609 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:50.609 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:50.609 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for peered", 2024-09-10T19:16:50.609 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:55.843332+0000", 2024-09-10T19:16:50.609 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.8550000000000004e-06 2024-09-10T19:16:50.609 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:50.609 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:50.610 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:16:50.610 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:56.864969+0000", 2024-09-10T19:16:50.610 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.021637492 2024-09-10T19:16:50.610 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:50.610 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:50.610 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-10T19:16:50.610 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:56.865098+0000", 2024-09-10T19:16:50.610 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000129052 2024-09-10T19:16:50.610 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:50.610 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:50.610 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0,3", 2024-09-10T19:16:50.610 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:56.865223+0000", 2024-09-10T19:16:50.610 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00012428100000000001 2024-09-10T19:16:50.611 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:50.611 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:50.611 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:16:50.611 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:56.874184+0000", 2024-09-10T19:16:50.611 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0089612579999999997 2024-09-10T19:16:50.611 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:50.611 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:50.611 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-09-10T19:16:50.611 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:56.874192+0000", 2024-09-10T19:16:50.611 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.4290000000000001e-06 2024-09-10T19:16:50.611 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:50.611 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:50.611 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:16:50.611 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:56.874240+0000", 2024-09-10T19:16:50.612 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.7496000000000001e-05 2024-09-10T19:16:50.612 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:50.612 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:50.612 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-09-10T19:16:50.612 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:56.874283+0000", 2024-09-10T19:16:50.612 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.2818999999999998e-05 2024-09-10T19:16:50.612 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:50.612 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:50.612 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-10T19:16:50.612 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:56.874306+0000", 2024-09-10T19:16:50.612 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.3813999999999999e-05 2024-09-10T19:16:50.612 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:16:50.612 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-10T19:16:50.613 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:16:50.613 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:50.613 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:50.613 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.14781.0:217266 7.d 7:b1a2b0a1:::benchmark_data_smithi183_84160_object76885:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e130)", 2024-09-10T19:16:50.613 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-10T19:14:55.846229+0000", 2024-09-10T19:16:50.613 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 114.746381345, 2024-09-10T19:16:50.613 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.0281871330000001, 2024-09-10T19:16:50.613 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-10T19:16:50.613 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T19:16:50.613 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-10T19:16:50.613 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.14781", 2024-09-10T19:16:50.613 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.183:0/2025776420", 2024-09-10T19:16:50.613 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 217266 2024-09-10T19:16:50.614 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:50.614 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-10T19:16:50.614 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:50.614 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-10T19:16:50.614 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:55.846229+0000", 2024-09-10T19:16:50.614 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:16:50.614 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:50.614 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:50.614 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-10T19:16:50.614 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:55.846229+0000", 2024-09-10T19:16:50.614 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:16:50.614 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:50.614 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:50.615 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-10T19:16:50.615 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:55.846227+0000", 2024-09-10T19:16:50.615 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999981 2024-09-10T19:16:50.615 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:50.615 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:50.615 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-10T19:16:50.615 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:55.846233+0000", 2024-09-10T19:16:50.615 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.5719999999999999e-06 2024-09-10T19:16:50.615 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:50.615 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:50.615 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-10T19:16:50.615 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:55.846783+0000", 2024-09-10T19:16:50.615 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00055009500000000001 2024-09-10T19:16:50.615 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:50.615 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:50.616 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-10T19:16:50.616 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:55.846794+0000", 2024-09-10T19:16:50.616 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.0129000000000001e-05 2024-09-10T19:16:50.616 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:50.616 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:50.616 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:16:50.616 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:55.847079+0000", 2024-09-10T19:16:50.616 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00028570199999999998 2024-09-10T19:16:50.616 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:50.616 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:50.616 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for peered", 2024-09-10T19:16:50.616 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:55.847122+0000", 2024-09-10T19:16:50.616 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.2199e-05 2024-09-10T19:16:50.617 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:50.617 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:50.617 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:16:50.617 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:56.867362+0000", 2024-09-10T19:16:50.617 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.020240276 2024-09-10T19:16:50.617 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:50.617 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:50.617 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-10T19:16:50.617 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:56.867446+0000", 2024-09-10T19:16:50.617 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.4536999999999994e-05 2024-09-10T19:16:50.617 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:50.617 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:50.617 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0,3", 2024-09-10T19:16:50.618 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:56.867519+0000", 2024-09-10T19:16:50.618 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.2349999999999997e-05 2024-09-10T19:16:50.618 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:50.618 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:50.618 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-09-10T19:16:50.618 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:56.874198+0000", 2024-09-10T19:16:50.618 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0066797080000000003 2024-09-10T19:16:50.618 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:50.618 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:50.618 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:16:50.618 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:56.874340+0000", 2024-09-10T19:16:50.618 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00014169599999999999 2024-09-10T19:16:50.618 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:50.619 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:50.619 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:16:50.619 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:56.874374+0000", 2024-09-10T19:16:50.619 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.3977999999999999e-05 2024-09-10T19:16:50.619 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:50.619 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:50.619 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-09-10T19:16:50.619 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:56.874397+0000", 2024-09-10T19:16:50.619 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.3280000000000001e-05 2024-09-10T19:16:50.619 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:50.619 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:50.619 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-10T19:16:50.619 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:56.874416+0000", 2024-09-10T19:16:50.619 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.8362000000000001e-05 2024-09-10T19:16:50.620 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:16:50.620 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-10T19:16:50.620 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:16:50.620 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:50.620 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:50.620 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.14781.0:217273 7.d 7:b13a7f3f:::benchmark_data_smithi183_84160_object76892:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e130)", 2024-09-10T19:16:50.620 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-10T19:14:55.855160+0000", 2024-09-10T19:16:50.620 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 114.737449975, 2024-09-10T19:16:50.620 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.0193451490000001, 2024-09-10T19:16:50.620 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-10T19:16:50.620 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T19:16:50.620 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-10T19:16:50.620 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.14781", 2024-09-10T19:16:50.620 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.183:0/2025776420", 2024-09-10T19:16:50.621 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 217273 2024-09-10T19:16:50.621 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:50.621 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-10T19:16:50.621 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:50.621 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-10T19:16:50.621 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:55.855160+0000", 2024-09-10T19:16:50.621 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:16:50.621 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:50.621 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:50.621 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-10T19:16:50.621 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:55.855160+0000", 2024-09-10T19:16:50.621 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:16:50.621 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:50.622 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:50.622 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-10T19:16:50.622 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:55.855158+0000", 2024-09-10T19:16:50.622 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999986 2024-09-10T19:16:50.622 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:50.622 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:50.622 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-10T19:16:50.622 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:55.855179+0000", 2024-09-10T19:16:50.622 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.0724999999999999e-05 2024-09-10T19:16:50.622 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:50.622 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:50.622 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-10T19:16:50.622 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:55.855202+0000", 2024-09-10T19:16:50.622 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.3057999999999998e-05 2024-09-10T19:16:50.622 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:50.623 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:50.623 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-10T19:16:50.623 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:55.855208+0000", 2024-09-10T19:16:50.623 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.0870000000000003e-06 2024-09-10T19:16:50.623 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:50.623 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:50.623 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:16:50.623 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:55.855251+0000", 2024-09-10T19:16:50.623 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.2508e-05 2024-09-10T19:16:50.623 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:50.623 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:50.623 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for peered", 2024-09-10T19:16:50.623 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:55.855258+0000", 2024-09-10T19:16:50.623 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.5599999999999996e-06 2024-09-10T19:16:50.623 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:50.624 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:50.624 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:16:50.624 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:56.869674+0000", 2024-09-10T19:16:50.624 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.014415375 2024-09-10T19:16:50.624 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:50.624 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:50.624 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-10T19:16:50.624 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:56.869782+0000", 2024-09-10T19:16:50.624 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000108618 2024-09-10T19:16:50.624 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:50.624 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:50.624 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0,3", 2024-09-10T19:16:50.624 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:56.869893+0000", 2024-09-10T19:16:50.624 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00011085 2024-09-10T19:16:50.625 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:50.625 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:50.625 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-09-10T19:16:50.625 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:56.874345+0000", 2024-09-10T19:16:50.625 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0044517070000000001 2024-09-10T19:16:50.625 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:50.625 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:50.625 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:16:50.625 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:56.874441+0000", 2024-09-10T19:16:50.625 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.6360999999999994e-05 2024-09-10T19:16:50.625 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:50.625 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:50.625 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:16:50.625 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:56.874468+0000", 2024-09-10T19:16:50.626 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.7307000000000001e-05 2024-09-10T19:16:50.626 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:50.626 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:50.626 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-09-10T19:16:50.626 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:56.874489+0000", 2024-09-10T19:16:50.626 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.0559000000000001e-05 2024-09-10T19:16:50.626 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:50.626 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:50.626 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-10T19:16:50.626 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:56.874505+0000", 2024-09-10T19:16:50.626 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.6082999999999998e-05 2024-09-10T19:16:50.626 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:16:50.626 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-10T19:16:50.626 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:16:50.626 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:50.627 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:50.627 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.14781.0:217302 7.d 7:b1c8efba:::benchmark_data_smithi183_84160_object76921:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e130)", 2024-09-10T19:16:50.627 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:16:50.628 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-10T19:14:55.886976+0000", 2024-09-10T19:16:50.628 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 114.705633569, 2024-09-10T19:16:50.628 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.98768860700000005, 2024-09-10T19:16:50.628 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-10T19:16:50.628 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T19:16:50.628 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-10T19:16:50.629 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.14781", 2024-09-10T19:16:50.629 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.183:0/2025776420", 2024-09-10T19:16:50.629 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 217302 2024-09-10T19:16:50.629 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:50.629 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-10T19:16:50.629 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:50.629 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-10T19:16:50.629 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:55.886976+0000", 2024-09-10T19:16:50.629 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:16:50.629 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:50.629 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:50.629 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-10T19:16:50.629 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:55.886976+0000", 2024-09-10T19:16:50.630 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:16:50.630 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:50.630 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:50.630 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-10T19:16:50.630 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:55.886975+0000", 2024-09-10T19:16:50.630 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999986 2024-09-10T19:16:50.630 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:50.630 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:50.630 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-10T19:16:50.630 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:55.886981+0000", 2024-09-10T19:16:50.630 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.0179999999999998e-06 2024-09-10T19:16:50.630 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:50.630 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:50.630 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-10T19:16:50.631 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:55.886991+0000", 2024-09-10T19:16:50.631 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.0336999999999999e-05 2024-09-10T19:16:50.631 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:50.631 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:50.631 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-10T19:16:50.631 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:55.886996+0000", 2024-09-10T19:16:50.631 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.5800000000000002e-06 2024-09-10T19:16:50.631 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:50.631 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:50.631 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:16:50.631 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:55.887043+0000", 2024-09-10T19:16:50.631 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.6554000000000002e-05 2024-09-10T19:16:50.631 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:50.631 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:50.632 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for peered", 2024-09-10T19:16:50.632 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:55.887050+0000", 2024-09-10T19:16:50.632 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.7419999999999997e-06 2024-09-10T19:16:50.632 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:50.632 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:50.632 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:16:50.632 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:56.871908+0000", 2024-09-10T19:16:50.632 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.98485727300000003 2024-09-10T19:16:50.632 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:50.632 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:50.632 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-10T19:16:50.632 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:56.872011+0000", 2024-09-10T19:16:50.632 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00010307899999999999 2024-09-10T19:16:50.633 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:50.633 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:50.633 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0,3", 2024-09-10T19:16:50.633 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:56.872121+0000", 2024-09-10T19:16:50.633 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000110202 2024-09-10T19:16:50.633 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:50.633 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:50.633 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-09-10T19:16:50.633 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:56.874445+0000", 2024-09-10T19:16:50.633 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0023241830000000001 2024-09-10T19:16:50.633 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:50.633 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:50.633 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:16:50.633 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:56.874552+0000", 2024-09-10T19:16:50.634 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000106612 2024-09-10T19:16:50.634 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:50.634 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:50.634 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:16:50.634 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:56.874598+0000", 2024-09-10T19:16:50.634 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.6468000000000003e-05 2024-09-10T19:16:50.634 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:50.634 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:50.634 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-09-10T19:16:50.634 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:56.874636+0000", 2024-09-10T19:16:50.634 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.7625e-05 2024-09-10T19:16:50.634 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:50.634 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:50.634 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-10T19:16:50.635 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:56.874665+0000", 2024-09-10T19:16:50.635 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.9155000000000001e-05 2024-09-10T19:16:50.635 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:16:50.635 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-10T19:16:50.635 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:16:50.635 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:50.635 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:50.635 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.14781.0:217356 7.d 7:bf6b8576:::benchmark_data_smithi183_84160_object76975:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e130)", 2024-09-10T19:16:50.635 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-10T19:14:55.932959+0000", 2024-09-10T19:16:50.635 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 114.659650727, 2024-09-10T19:16:50.635 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.95670432400000005, 2024-09-10T19:16:50.635 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-10T19:16:50.635 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T19:16:50.635 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-10T19:16:50.635 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.14781", 2024-09-10T19:16:50.636 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.183:0/2025776420", 2024-09-10T19:16:50.636 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 217356 2024-09-10T19:16:50.636 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:50.636 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-10T19:16:50.636 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:50.636 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-10T19:16:50.636 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:55.932959+0000", 2024-09-10T19:16:50.636 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:16:50.636 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:50.636 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:50.636 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-10T19:16:50.636 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:55.932959+0000", 2024-09-10T19:16:50.636 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:16:50.636 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:50.637 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:50.637 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-10T19:16:50.637 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:55.932958+0000", 2024-09-10T19:16:50.637 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.999999 2024-09-10T19:16:50.637 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:50.637 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:50.637 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-10T19:16:50.637 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:55.932963+0000", 2024-09-10T19:16:50.637 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.1039999999999998e-06 2024-09-10T19:16:50.637 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:50.637 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:50.637 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-10T19:16:50.637 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:55.932973+0000", 2024-09-10T19:16:50.637 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.5289999999999997e-06 2024-09-10T19:16:50.638 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:50.638 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:50.638 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-10T19:16:50.638 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:55.932978+0000", 2024-09-10T19:16:50.638 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.7030000000000003e-06 2024-09-10T19:16:50.638 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:50.638 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:50.638 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:16:50.638 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:55.933047+0000", 2024-09-10T19:16:50.638 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.8330000000000005e-05 2024-09-10T19:16:50.638 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:50.638 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:50.638 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for peered", 2024-09-10T19:16:50.639 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:55.933056+0000", 2024-09-10T19:16:50.639 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.3200000000000006e-06 2024-09-10T19:16:50.639 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:50.639 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:50.639 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:16:50.639 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:56.874704+0000", 2024-09-10T19:16:50.639 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.94164784300000004 2024-09-10T19:16:50.639 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:50.639 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:50.639 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-10T19:16:50.639 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:56.874807+0000", 2024-09-10T19:16:50.639 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000102576 2024-09-10T19:16:50.639 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:50.639 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:50.640 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0,3", 2024-09-10T19:16:50.640 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:56.875021+0000", 2024-09-10T19:16:50.640 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00021458999999999999 2024-09-10T19:16:50.640 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:50.640 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:50.640 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:16:50.640 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:56.886743+0000", 2024-09-10T19:16:50.640 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.011721834 2024-09-10T19:16:50.640 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:50.640 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:50.640 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-09-10T19:16:50.640 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:56.886749+0000", 2024-09-10T19:16:50.640 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.8900000000000004e-06 2024-09-10T19:16:50.641 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:50.641 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:50.641 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:16:50.641 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:56.889586+0000", 2024-09-10T19:16:50.641 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0028372340000000001 2024-09-10T19:16:50.641 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:50.641 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:50.641 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-09-10T19:16:50.641 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:56.889622+0000", 2024-09-10T19:16:50.641 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.6399999999999997e-05 2024-09-10T19:16:50.641 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:50.641 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:50.641 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-10T19:16:50.641 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:56.889664+0000", 2024-09-10T19:16:50.642 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.1019999999999997e-05 2024-09-10T19:16:50.642 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:16:50.642 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-10T19:16:50.642 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:16:50.642 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:50.642 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:50.642 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.14781.0:217362 7.d 7:b681ffb2:::benchmark_data_smithi183_84160_object76981:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e130)", 2024-09-10T19:16:50.642 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-10T19:14:55.936260+0000", 2024-09-10T19:16:50.642 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 114.656350151, 2024-09-10T19:16:50.642 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.95911488, 2024-09-10T19:16:50.642 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-10T19:16:50.642 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T19:16:50.642 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-10T19:16:50.642 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.14781", 2024-09-10T19:16:50.643 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.183:0/2025776420", 2024-09-10T19:16:50.643 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 217362 2024-09-10T19:16:50.643 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:50.643 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-10T19:16:50.643 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:50.643 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-10T19:16:50.643 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:55.936260+0000", 2024-09-10T19:16:50.643 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:16:50.643 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:50.644 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:50.644 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-10T19:16:50.644 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:55.936260+0000", 2024-09-10T19:16:50.644 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:16:50.644 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:50.644 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:50.644 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-10T19:16:50.644 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:55.936259+0000", 2024-09-10T19:16:50.644 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.999999 2024-09-10T19:16:50.644 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:50.644 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:50.644 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-10T19:16:50.644 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:55.936264+0000", 2024-09-10T19:16:50.645 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.7239999999999997e-06 2024-09-10T19:16:50.645 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:50.645 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:50.645 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-10T19:16:50.645 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:55.936276+0000", 2024-09-10T19:16:50.645 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.1817000000000001e-05 2024-09-10T19:16:50.645 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:50.645 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:50.645 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-10T19:16:50.645 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:55.936281+0000", 2024-09-10T19:16:50.645 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.5669999999999996e-06 2024-09-10T19:16:50.645 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:50.645 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:50.645 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:16:50.645 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:55.936323+0000", 2024-09-10T19:16:50.646 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.2528999999999999e-05 2024-09-10T19:16:50.646 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:50.646 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:50.646 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for peered", 2024-09-10T19:16:50.646 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:55.936335+0000", 2024-09-10T19:16:50.646 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.1776e-05 2024-09-10T19:16:50.646 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:50.646 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:50.646 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:16:50.646 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:56.877363+0000", 2024-09-10T19:16:50.646 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.94102812700000005 2024-09-10T19:16:50.646 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:50.646 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:50.646 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-10T19:16:50.647 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:56.877465+0000", 2024-09-10T19:16:50.647 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00010194199999999999 2024-09-10T19:16:50.647 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:50.647 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:50.647 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0,3", 2024-09-10T19:16:50.647 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:56.877573+0000", 2024-09-10T19:16:50.647 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000107435 2024-09-10T19:16:50.647 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:50.647 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:50.647 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-09-10T19:16:50.647 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:56.886758+0000", 2024-09-10T19:16:50.647 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0091858319999999997 2024-09-10T19:16:50.647 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:50.648 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:50.648 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:16:50.648 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:56.889720+0000", 2024-09-10T19:16:50.648 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0029615209999999999 2024-09-10T19:16:50.648 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:50.648 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:50.648 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:16:50.648 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:56.895276+0000", 2024-09-10T19:16:50.648 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0055555930000000002 2024-09-10T19:16:50.648 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:50.648 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:50.648 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-09-10T19:16:50.648 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:56.895333+0000", 2024-09-10T19:16:50.648 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.7803000000000002e-05 2024-09-10T19:16:50.649 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:50.649 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:50.649 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-10T19:16:50.649 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:56.895375+0000", 2024-09-10T19:16:50.649 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.1245000000000002e-05 2024-09-10T19:16:50.649 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:16:50.649 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-10T19:16:50.649 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:16:50.649 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:50.649 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:50.649 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.14781.0:217367 7.d 7:bef6592b:::benchmark_data_smithi183_84160_object76986:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e130)", 2024-09-10T19:16:50.649 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-10T19:14:55.940396+0000", 2024-09-10T19:16:50.649 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 114.652213687, 2024-09-10T19:16:50.649 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.95791767100000003, 2024-09-10T19:16:50.649 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-10T19:16:50.650 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T19:16:50.650 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-10T19:16:50.650 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.14781", 2024-09-10T19:16:50.650 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.183:0/2025776420", 2024-09-10T19:16:50.650 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 217367 2024-09-10T19:16:50.650 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:50.650 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-10T19:16:50.650 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:50.650 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-10T19:16:50.650 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:55.940396+0000", 2024-09-10T19:16:50.650 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:16:50.650 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:50.650 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:50.650 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-10T19:16:50.651 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:55.940396+0000", 2024-09-10T19:16:50.651 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:16:50.651 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:50.651 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:50.651 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-10T19:16:50.651 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:55.940395+0000", 2024-09-10T19:16:50.651 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.999999 2024-09-10T19:16:50.651 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:50.651 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:50.651 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-10T19:16:50.651 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:55.940400+0000", 2024-09-10T19:16:50.651 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.4630000000000003e-06 2024-09-10T19:16:50.651 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:50.651 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:50.651 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-10T19:16:50.652 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:55.940409+0000", 2024-09-10T19:16:50.652 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.5440000000000004e-06 2024-09-10T19:16:50.652 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:50.652 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:50.652 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-10T19:16:50.652 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:55.940415+0000", 2024-09-10T19:16:50.652 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.2079999999999999e-06 2024-09-10T19:16:50.652 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:50.652 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:50.652 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:16:50.652 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:55.940451+0000", 2024-09-10T19:16:50.652 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.6643000000000001e-05 2024-09-10T19:16:50.652 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:50.652 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:50.652 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for peered", 2024-09-10T19:16:50.653 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:55.940457+0000", 2024-09-10T19:16:50.653 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.2179999999999998e-06 2024-09-10T19:16:50.653 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:50.653 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:50.653 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:16:50.653 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:56.879384+0000", 2024-09-10T19:16:50.653 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.93892772300000005 2024-09-10T19:16:50.653 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:50.653 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:50.653 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-10T19:16:50.653 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:56.879473+0000", 2024-09-10T19:16:50.653 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.8676000000000003e-05 2024-09-10T19:16:50.653 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:50.654 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:50.654 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0,3", 2024-09-10T19:16:50.654 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:56.879570+0000", 2024-09-10T19:16:50.654 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.7032000000000002e-05 2024-09-10T19:16:50.654 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:50.654 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:50.654 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-09-10T19:16:50.654 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:56.886763+0000", 2024-09-10T19:16:50.654 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0071928160000000003 2024-09-10T19:16:50.654 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:50.654 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:50.654 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:16:50.654 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:56.889787+0000", 2024-09-10T19:16:50.654 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0030244669999999999 2024-09-10T19:16:50.655 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:50.655 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:50.655 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:16:50.655 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:56.898220+0000", 2024-09-10T19:16:50.655 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0084325100000000007 2024-09-10T19:16:50.655 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:50.655 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:50.655 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-09-10T19:16:50.655 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:56.898275+0000", 2024-09-10T19:16:50.655 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.5010999999999999e-05 2024-09-10T19:16:50.655 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:50.655 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:50.655 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-10T19:16:50.656 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:56.898314+0000", 2024-09-10T19:16:50.656 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.9159000000000002e-05 2024-09-10T19:16:50.656 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:16:50.656 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-10T19:16:50.656 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:16:50.656 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:50.656 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:50.656 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.14781.0:217390 7.d 7:bd670288:::benchmark_data_smithi183_84160_object77009:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e130)", 2024-09-10T19:16:50.656 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-10T19:14:55.964568+0000", 2024-09-10T19:16:50.656 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 114.62804212, 2024-09-10T19:16:50.656 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.93384886899999997, 2024-09-10T19:16:50.656 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-10T19:16:50.656 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T19:16:50.656 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-10T19:16:50.657 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.14781", 2024-09-10T19:16:50.657 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.183:0/2025776420", 2024-09-10T19:16:50.657 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 217390 2024-09-10T19:16:50.657 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:50.657 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-10T19:16:50.657 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:50.657 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-10T19:16:50.657 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:55.964568+0000", 2024-09-10T19:16:50.657 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:16:50.657 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:50.657 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:50.657 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-10T19:16:50.657 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:55.964568+0000", 2024-09-10T19:16:50.657 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:16:50.657 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:50.658 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:50.658 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-10T19:16:50.658 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:55.964566+0000", 2024-09-10T19:16:50.658 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999986 2024-09-10T19:16:50.658 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:50.658 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:50.658 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-10T19:16:50.658 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:55.964574+0000", 2024-09-10T19:16:50.658 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.4619999999999999e-06 2024-09-10T19:16:50.658 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:50.658 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:50.658 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-10T19:16:50.658 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:55.964588+0000", 2024-09-10T19:16:50.658 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.4093e-05 2024-09-10T19:16:50.659 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:50.659 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:50.659 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-10T19:16:50.659 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:55.964594+0000", 2024-09-10T19:16:50.659 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.0589999999999999e-06 2024-09-10T19:16:50.659 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:50.659 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:50.659 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:16:50.659 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:55.964636+0000", 2024-09-10T19:16:50.659 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.2154999999999999e-05 2024-09-10T19:16:50.659 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:50.659 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:50.660 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for peered", 2024-09-10T19:16:50.660 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:55.964643+0000", 2024-09-10T19:16:50.660 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.3930000000000002e-06 2024-09-10T19:16:50.660 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:50.660 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:50.660 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:16:50.660 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:56.881418+0000", 2024-09-10T19:16:50.660 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.916775114 2024-09-10T19:16:50.660 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:50.660 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:50.660 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-10T19:16:50.660 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:56.881509+0000", 2024-09-10T19:16:50.660 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.0183e-05 2024-09-10T19:16:50.660 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:50.661 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:50.661 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0,3", 2024-09-10T19:16:50.661 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:56.881615+0000", 2024-09-10T19:16:50.661 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00010611399999999999 2024-09-10T19:16:50.661 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:50.661 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:50.661 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-09-10T19:16:50.661 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:56.886767+0000", 2024-09-10T19:16:50.661 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0051521689999999998 2024-09-10T19:16:50.661 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:50.661 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:50.661 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:16:50.661 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:56.889849+0000", 2024-09-10T19:16:50.662 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0030821429999999999 2024-09-10T19:16:50.662 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:50.662 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:50.662 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:16:50.662 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:56.898350+0000", 2024-09-10T19:16:50.662 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0085007099999999999 2024-09-10T19:16:50.662 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:50.662 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:50.662 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-09-10T19:16:50.662 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:56.898375+0000", 2024-09-10T19:16:50.662 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.5372999999999999e-05 2024-09-10T19:16:50.662 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:50.662 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:50.663 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-10T19:16:50.663 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:56.898417+0000", 2024-09-10T19:16:50.663 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.1516e-05 2024-09-10T19:16:50.663 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:16:50.663 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-10T19:16:50.663 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:16:50.663 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:50.663 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:50.663 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.14781.0:217400 7.d 7:ba739e40:::benchmark_data_smithi183_84160_object77019:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e130)", 2024-09-10T19:16:50.663 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-10T19:14:55.979209+0000", 2024-09-10T19:16:50.663 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 114.613400881, 2024-09-10T19:16:50.663 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.91933015600000001, 2024-09-10T19:16:50.663 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-10T19:16:50.663 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T19:16:50.663 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-10T19:16:50.663 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.14781", 2024-09-10T19:16:50.664 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.183:0/2025776420", 2024-09-10T19:16:50.664 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 217400 2024-09-10T19:16:50.664 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:50.664 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-10T19:16:50.664 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:50.664 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-10T19:16:50.664 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:55.979209+0000", 2024-09-10T19:16:50.664 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:16:50.664 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:50.664 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:50.664 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-10T19:16:50.664 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:55.979209+0000", 2024-09-10T19:16:50.664 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:16:50.664 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:50.664 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:50.665 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-10T19:16:50.665 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:55.979208+0000", 2024-09-10T19:16:50.665 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999986 2024-09-10T19:16:50.665 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:50.665 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:50.665 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-10T19:16:50.665 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:55.979217+0000", 2024-09-10T19:16:50.665 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.0399999999999998e-06 2024-09-10T19:16:50.665 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:50.665 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:50.665 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-10T19:16:50.665 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:55.979229+0000", 2024-09-10T19:16:50.665 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.255e-05 2024-09-10T19:16:50.665 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:50.666 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:50.666 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-10T19:16:50.666 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:55.979234+0000", 2024-09-10T19:16:50.666 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.4440000000000001e-06 2024-09-10T19:16:50.666 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:50.666 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:50.666 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:16:50.666 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:55.979272+0000", 2024-09-10T19:16:50.666 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.8081999999999999e-05 2024-09-10T19:16:50.666 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:50.666 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:50.666 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for peered", 2024-09-10T19:16:50.666 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:55.979278+0000", 2024-09-10T19:16:50.666 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.297e-06 2024-09-10T19:16:50.667 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:50.667 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:50.667 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:16:50.667 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:56.883680+0000", 2024-09-10T19:16:50.667 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.90440144899999997 2024-09-10T19:16:50.667 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:50.667 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:50.667 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-10T19:16:50.667 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:56.883766+0000", 2024-09-10T19:16:50.667 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.6340000000000003e-05 2024-09-10T19:16:50.667 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:50.667 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:50.667 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0,3", 2024-09-10T19:16:50.668 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:56.884028+0000", 2024-09-10T19:16:50.668 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00026236400000000001 2024-09-10T19:16:50.668 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:50.668 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:50.668 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-09-10T19:16:50.668 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:56.889161+0000", 2024-09-10T19:16:50.668 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0051324989999999996 2024-09-10T19:16:50.668 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:50.668 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:50.668 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:16:50.668 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:56.892822+0000", 2024-09-10T19:16:50.668 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0036613779999999999 2024-09-10T19:16:50.668 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:50.668 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:50.668 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:16:50.669 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:56.898455+0000", 2024-09-10T19:16:50.669 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0056326939999999997 2024-09-10T19:16:50.669 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:50.669 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:50.669 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-09-10T19:16:50.669 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:56.898505+0000", 2024-09-10T19:16:50.669 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.9805999999999997e-05 2024-09-10T19:16:50.669 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:50.669 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:50.669 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-10T19:16:50.669 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:56.898539+0000", 2024-09-10T19:16:50.669 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.4570000000000003e-05 2024-09-10T19:16:50.670 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:16:50.670 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-10T19:16:50.671 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:16:50.671 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:50.671 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:50.671 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.14514.0:363 3.0 3:353ec8f9:::gateway.conf:head [getxattr epoch (1) in=5b] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e154)", 2024-09-10T19:16:50.671 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-10T19:16:24.320868+0000", 2024-09-10T19:16:50.671 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 26.271742061000001, 2024-09-10T19:16:50.671 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.52637172899999995, 2024-09-10T19:16:50.671 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-10T19:16:50.671 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "started", 2024-09-10T19:16:50.671 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-10T19:16:50.671 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.14514", 2024-09-10T19:16:50.671 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.191:0/2959168166", 2024-09-10T19:16:50.671 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 363 2024-09-10T19:16:50.671 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:50.672 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-10T19:16:50.672 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:50.672 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-10T19:16:50.672 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:16:24.320868+0000", 2024-09-10T19:16:50.672 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:16:50.672 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:50.672 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:50.672 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-10T19:16:50.672 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:16:24.320868+0000", 2024-09-10T19:16:50.672 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:16:50.672 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:50.672 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:50.672 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-10T19:16:50.672 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:16:24.320866+0000", 2024-09-10T19:16:50.672 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999976 2024-09-10T19:16:50.673 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:50.673 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:50.673 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-10T19:16:50.673 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:16:24.320873+0000", 2024-09-10T19:16:50.673 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.3930000000000002e-06 2024-09-10T19:16:50.673 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:50.673 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:50.673 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-10T19:16:50.673 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:16:24.320884+0000", 2024-09-10T19:16:50.673 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.0749e-05 2024-09-10T19:16:50.673 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:50.673 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:50.673 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-10T19:16:50.673 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:16:24.320892+0000", 2024-09-10T19:16:50.674 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.7409999999999992e-06 2024-09-10T19:16:50.674 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:50.674 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:50.674 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:16:50.674 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:16:24.320955+0000", 2024-09-10T19:16:50.674 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.3847e-05 2024-09-10T19:16:50.674 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:50.674 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:50.674 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for peered", 2024-09-10T19:16:50.674 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:16:24.320966+0000", 2024-09-10T19:16:50.674 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.0801999999999999e-05 2024-09-10T19:16:50.674 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:50.674 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:50.674 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:16:50.674 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:16:24.846991+0000", 2024-09-10T19:16:50.674 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.52602520699999999 2024-09-10T19:16:50.675 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:50.675 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:50.675 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-10T19:16:50.675 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:16:24.847118+0000", 2024-09-10T19:16:50.675 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00012705799999999999 2024-09-10T19:16:50.675 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:50.675 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:50.675 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-10T19:16:50.675 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:16:24.847240+0000", 2024-09-10T19:16:50.675 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000121142 2024-09-10T19:16:50.675 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:16:50.675 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-10T19:16:50.675 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:16:50.675 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:50.675 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:50.676 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.15051.0:29541 8.9 8:968217b7:::benchmark_data_smithi183_114924_object29540:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160)", 2024-09-10T19:16:50.676 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-10T19:16:35.794625+0000", 2024-09-10T19:16:50.676 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 14.797985410999999, 2024-09-10T19:16:50.676 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.16727109700000001, 2024-09-10T19:16:50.676 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-10T19:16:50.676 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T19:16:50.676 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-10T19:16:50.676 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.15051", 2024-09-10T19:16:50.676 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.183:0/3632028672", 2024-09-10T19:16:50.676 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 29541 2024-09-10T19:16:50.676 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:50.676 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-10T19:16:50.676 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:50.676 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-10T19:16:50.677 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:16:35.794625+0000", 2024-09-10T19:16:50.677 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:16:50.677 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:50.677 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:50.677 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-10T19:16:50.677 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:16:35.794625+0000", 2024-09-10T19:16:50.677 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:16:50.677 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:50.677 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:50.677 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-10T19:16:50.677 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:16:35.794623+0000", 2024-09-10T19:16:50.677 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999986 2024-09-10T19:16:50.677 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:50.677 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:50.678 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-10T19:16:50.678 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:16:35.794677+0000", 2024-09-10T19:16:50.678 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.4137e-05 2024-09-10T19:16:50.678 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:50.678 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:50.678 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-10T19:16:50.678 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:16:35.794686+0000", 2024-09-10T19:16:50.678 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.4330000000000004e-06 2024-09-10T19:16:50.678 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:50.678 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:50.678 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-10T19:16:50.678 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:16:35.794690+0000", 2024-09-10T19:16:50.678 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.9389999999999999e-06 2024-09-10T19:16:50.678 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:50.679 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:50.679 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:16:50.679 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:16:35.794733+0000", 2024-09-10T19:16:50.679 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.3662999999999999e-05 2024-09-10T19:16:50.679 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:50.679 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:50.679 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-10T19:16:50.679 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:16:35.794842+0000", 2024-09-10T19:16:50.679 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000108484 2024-09-10T19:16:50.679 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:50.679 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:50.679 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0,5", 2024-09-10T19:16:50.679 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:16:35.794923+0000", 2024-09-10T19:16:50.679 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.1067000000000002e-05 2024-09-10T19:16:50.680 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:50.680 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:50.680 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-09-10T19:16:50.680 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:16:35.796018+0000", 2024-09-10T19:16:50.680 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0010954339999999999 2024-09-10T19:16:50.680 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:50.680 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:50.680 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:16:50.680 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:16:35.797121+0000", 2024-09-10T19:16:50.680 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.001103087 2024-09-10T19:16:50.680 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:50.680 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:50.680 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:16:50.681 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:16:35.961801+0000", 2024-09-10T19:16:50.681 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.164679888 2024-09-10T19:16:50.681 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:50.681 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:50.681 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-09-10T19:16:50.681 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:16:35.961857+0000", 2024-09-10T19:16:50.681 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.6059e-05 2024-09-10T19:16:50.681 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:50.681 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:50.681 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-10T19:16:50.681 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:16:35.961896+0000", 2024-09-10T19:16:50.681 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.8244000000000002e-05 2024-09-10T19:16:50.681 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:16:50.682 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-10T19:16:50.682 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:16:50.682 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:50.682 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:50.682 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.15051.0:31776 8.9 8:963f0e24:::benchmark_data_smithi183_114924_object31775:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160)", 2024-09-10T19:16:50.682 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-10T19:16:37.293644+0000", 2024-09-10T19:16:50.682 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 13.298965483, 2024-09-10T19:16:50.682 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.17371768800000001, 2024-09-10T19:16:50.682 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-10T19:16:50.682 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T19:16:50.682 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-10T19:16:50.682 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.15051", 2024-09-10T19:16:50.682 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.183:0/3632028672", 2024-09-10T19:16:50.682 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 31776 2024-09-10T19:16:50.683 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:50.683 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-10T19:16:50.683 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:50.683 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-10T19:16:50.683 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:16:37.293644+0000", 2024-09-10T19:16:50.683 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:16:50.683 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:50.683 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:50.683 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-10T19:16:50.683 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:16:37.293644+0000", 2024-09-10T19:16:50.683 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:16:50.683 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:50.683 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:50.683 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-10T19:16:50.683 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:16:37.293644+0000", 2024-09-10T19:16:50.684 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.999999 2024-09-10T19:16:50.684 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:50.684 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:50.684 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-10T19:16:50.684 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:16:37.293710+0000", 2024-09-10T19:16:50.684 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.6657999999999998e-05 2024-09-10T19:16:50.684 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:50.684 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:50.684 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-10T19:16:50.684 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:16:37.293720+0000", 2024-09-10T19:16:50.684 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.3870000000000001e-06 2024-09-10T19:16:50.684 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:50.684 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:50.684 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-10T19:16:50.685 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:16:37.293724+0000", 2024-09-10T19:16:50.685 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.6739999999999996e-06 2024-09-10T19:16:50.685 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:50.685 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:50.685 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:16:50.685 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:16:37.294829+0000", 2024-09-10T19:16:50.685 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0011044469999999999 2024-09-10T19:16:50.685 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:50.685 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:50.685 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-10T19:16:50.685 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:16:37.294936+0000", 2024-09-10T19:16:50.685 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000107458 2024-09-10T19:16:50.685 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:50.685 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:50.686 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0,5", 2024-09-10T19:16:50.686 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:16:37.295047+0000", 2024-09-10T19:16:50.686 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000110809 2024-09-10T19:16:50.686 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:50.686 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:50.686 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-09-10T19:16:50.686 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:16:37.299437+0000", 2024-09-10T19:16:50.686 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0043897579999999997 2024-09-10T19:16:50.686 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:50.686 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:50.686 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:16:50.686 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:16:37.299609+0000", 2024-09-10T19:16:50.686 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00017247900000000001 2024-09-10T19:16:50.686 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:50.686 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:50.687 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:16:50.687 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:16:37.467268+0000", 2024-09-10T19:16:50.687 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.16765897599999999 2024-09-10T19:16:50.687 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:50.687 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:50.687 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-09-10T19:16:50.687 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:16:37.467319+0000", 2024-09-10T19:16:50.687 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.0756e-05 2024-09-10T19:16:50.687 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:50.687 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:50.687 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-10T19:16:50.687 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:16:37.467362+0000", 2024-09-10T19:16:50.687 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.3161000000000002e-05 2024-09-10T19:16:50.687 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:16:50.687 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-10T19:16:50.688 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:16:50.688 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:50.688 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:50.688 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.15051.0:31782 8.9 8:9e92f05a:::benchmark_data_smithi183_114924_object31781:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160)", 2024-09-10T19:16:50.688 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-10T19:16:37.294717+0000", 2024-09-10T19:16:50.688 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 13.297893214, 2024-09-10T19:16:50.688 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.172759629, 2024-09-10T19:16:50.688 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-10T19:16:50.688 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T19:16:50.688 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-10T19:16:50.688 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.15051", 2024-09-10T19:16:50.688 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.183:0/3632028672", 2024-09-10T19:16:50.688 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 31782 2024-09-10T19:16:50.688 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:50.689 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-10T19:16:50.689 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:50.689 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-10T19:16:50.689 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:16:37.294717+0000", 2024-09-10T19:16:50.689 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:16:50.689 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:50.689 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:50.689 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-10T19:16:50.689 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:16:37.294717+0000", 2024-09-10T19:16:50.689 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:16:50.689 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:50.689 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:50.689 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-10T19:16:50.689 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:16:37.294716+0000", 2024-09-10T19:16:50.690 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.999999 2024-09-10T19:16:50.690 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:50.690 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:50.690 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-10T19:16:50.690 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:16:37.294747+0000", 2024-09-10T19:16:50.690 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.1563999999999998e-05 2024-09-10T19:16:50.690 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:50.690 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:50.690 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-10T19:16:50.690 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:16:37.294769+0000", 2024-09-10T19:16:50.690 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.2042999999999999e-05 2024-09-10T19:16:50.690 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:50.690 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:50.691 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-10T19:16:50.691 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:16:37.294776+0000", 2024-09-10T19:16:50.691 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.4620000000000001e-06 2024-09-10T19:16:50.691 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:50.691 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:50.691 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:16:50.691 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:16:37.296229+0000", 2024-09-10T19:16:50.691 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.001452854 2024-09-10T19:16:50.691 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:50.691 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:50.691 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-10T19:16:50.691 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:16:37.296336+0000", 2024-09-10T19:16:50.691 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000106933 2024-09-10T19:16:50.692 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:50.692 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:50.692 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0,5", 2024-09-10T19:16:50.692 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:16:37.296454+0000", 2024-09-10T19:16:50.692 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000118099 2024-09-10T19:16:50.692 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:50.692 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:50.692 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-09-10T19:16:50.692 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:16:37.299476+0000", 2024-09-10T19:16:50.692 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00302265 2024-09-10T19:16:50.692 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:50.692 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:50.692 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:16:50.693 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:16:37.299632+0000", 2024-09-10T19:16:50.693 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000155458 2024-09-10T19:16:50.693 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:50.693 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:50.693 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:16:50.693 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:16:37.467404+0000", 2024-09-10T19:16:50.693 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.16777199200000001 2024-09-10T19:16:50.693 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:50.693 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:50.693 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-09-10T19:16:50.693 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:16:37.467439+0000", 2024-09-10T19:16:50.693 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.4807000000000001e-05 2024-09-10T19:16:50.693 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:50.694 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:50.694 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-10T19:16:50.694 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:16:37.467476+0000", 2024-09-10T19:16:50.694 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.7750999999999998e-05 2024-09-10T19:16:50.694 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:16:50.694 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-10T19:16:50.694 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:16:50.694 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:50.694 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:50.694 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.15051.0:31783 8.9 8:9d06511f:::benchmark_data_smithi183_114924_object31782:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160)", 2024-09-10T19:16:50.694 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-10T19:16:37.296061+0000", 2024-09-10T19:16:50.694 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 13.296549275, 2024-09-10T19:16:50.694 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.171561411, 2024-09-10T19:16:50.694 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-10T19:16:50.695 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T19:16:50.695 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-10T19:16:50.695 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.15051", 2024-09-10T19:16:50.695 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.183:0/3632028672", 2024-09-10T19:16:50.695 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 31783 2024-09-10T19:16:50.695 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:50.695 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-10T19:16:50.695 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:50.695 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-10T19:16:50.695 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:16:37.296061+0000", 2024-09-10T19:16:50.695 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:16:50.695 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:50.695 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:50.695 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-10T19:16:50.695 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:16:37.296061+0000", 2024-09-10T19:16:50.695 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:16:50.696 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:50.696 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:50.696 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-10T19:16:50.696 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:16:37.296060+0000", 2024-09-10T19:16:50.696 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.999999 2024-09-10T19:16:50.696 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:50.696 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:50.696 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-10T19:16:50.696 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:16:37.296115+0000", 2024-09-10T19:16:50.696 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.5064e-05 2024-09-10T19:16:50.696 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:50.696 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:50.696 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-10T19:16:50.697 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:16:37.296127+0000", 2024-09-10T19:16:50.697 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.1945e-05 2024-09-10T19:16:50.697 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:50.697 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:50.697 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-10T19:16:50.697 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:16:37.296133+0000", 2024-09-10T19:16:50.697 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.1990000000000003e-06 2024-09-10T19:16:50.697 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:50.697 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:50.697 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:16:50.697 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:16:37.297605+0000", 2024-09-10T19:16:50.697 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0014719360000000001 2024-09-10T19:16:50.697 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:50.697 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:50.698 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-10T19:16:50.698 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:16:37.297711+0000", 2024-09-10T19:16:50.698 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000106174 2024-09-10T19:16:50.698 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:50.698 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:50.698 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0,5", 2024-09-10T19:16:50.698 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:16:37.297833+0000", 2024-09-10T19:16:50.698 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00012257899999999999 2024-09-10T19:16:50.698 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:50.698 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:50.698 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-09-10T19:16:50.698 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:16:37.299481+0000", 2024-09-10T19:16:50.698 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.001647042 2024-09-10T19:16:50.698 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:50.698 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:50.699 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:16:50.699 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:16:37.299659+0000", 2024-09-10T19:16:50.699 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00017869199999999999 2024-09-10T19:16:50.699 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:50.699 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:50.699 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:16:50.699 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:16:37.467552+0000", 2024-09-10T19:16:50.699 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.167893036 2024-09-10T19:16:50.699 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:50.699 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:50.699 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-09-10T19:16:50.699 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:16:37.467595+0000", 2024-09-10T19:16:50.699 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.2265999999999998e-05 2024-09-10T19:16:50.700 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:50.700 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:50.700 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-10T19:16:50.700 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:16:37.467622+0000", 2024-09-10T19:16:50.700 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.7534000000000001e-05 2024-09-10T19:16:50.700 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:16:50.700 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-10T19:16:50.700 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:16:50.700 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:50.700 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:50.700 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.15051.0:31785 8.9 8:9a12b97b:::benchmark_data_smithi183_114924_object31784:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160)", 2024-09-10T19:16:50.700 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-10T19:16:37.296940+0000", 2024-09-10T19:16:50.700 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 13.295670073, 2024-09-10T19:16:50.700 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.170774433, 2024-09-10T19:16:50.700 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-10T19:16:50.701 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T19:16:50.701 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-10T19:16:50.701 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.15051", 2024-09-10T19:16:50.701 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.183:0/3632028672", 2024-09-10T19:16:50.701 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 31785 2024-09-10T19:16:50.701 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:50.701 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-10T19:16:50.701 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:50.701 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-10T19:16:50.701 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:16:37.296940+0000", 2024-09-10T19:16:50.701 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:16:50.701 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:50.701 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:50.701 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-10T19:16:50.702 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:16:37.296940+0000", 2024-09-10T19:16:50.702 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:16:50.702 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:50.702 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:50.702 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-10T19:16:50.702 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:16:37.296939+0000", 2024-09-10T19:16:50.702 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.999999 2024-09-10T19:16:50.702 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:50.702 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:50.702 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-10T19:16:50.702 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:16:37.297001+0000", 2024-09-10T19:16:50.702 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.1847999999999995e-05 2024-09-10T19:16:50.702 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:50.702 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:50.702 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-10T19:16:50.703 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:16:37.297011+0000", 2024-09-10T19:16:50.703 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.0463e-05 2024-09-10T19:16:50.703 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:50.703 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:50.703 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-10T19:16:50.703 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:16:37.297017+0000", 2024-09-10T19:16:50.703 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.429e-06 2024-09-10T19:16:50.703 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:50.703 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:50.703 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:16:50.703 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:16:37.299679+0000", 2024-09-10T19:16:50.703 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0026624579999999999 2024-09-10T19:16:50.703 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:50.703 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:50.704 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-10T19:16:50.704 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:16:37.299792+0000", 2024-09-10T19:16:50.704 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000113157 2024-09-10T19:16:50.704 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:50.704 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:50.704 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0,5", 2024-09-10T19:16:50.704 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:16:37.299912+0000", 2024-09-10T19:16:50.704 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000119497 2024-09-10T19:16:50.704 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:50.704 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:50.704 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-09-10T19:16:50.704 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:16:37.467496+0000", 2024-09-10T19:16:50.704 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.16758398699999999 2024-09-10T19:16:50.704 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:50.705 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:50.705 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:16:50.705 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:16:37.467525+0000", 2024-09-10T19:16:50.705 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.8912e-05 2024-09-10T19:16:50.705 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:50.705 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:50.705 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:16:50.705 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:16:37.467654+0000", 2024-09-10T19:16:50.705 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000129319 2024-09-10T19:16:50.705 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:50.705 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:50.705 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-09-10T19:16:50.705 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:16:37.467688+0000", 2024-09-10T19:16:50.705 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.4159999999999998e-05 2024-09-10T19:16:50.705 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:50.706 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:50.706 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-10T19:16:50.706 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:16:37.467714+0000", 2024-09-10T19:16:50.706 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.6126e-05 2024-09-10T19:16:50.706 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:16:50.706 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-10T19:16:50.706 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:16:50.706 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:50.706 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:50.706 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.15051.0:31786 8.5 8:a7600a19:::benchmark_data_smithi183_114924_object31785:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160)", 2024-09-10T19:16:50.706 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-10T19:16:37.298127+0000", 2024-09-10T19:16:50.706 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 13.294482936, 2024-09-10T19:16:50.706 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.169355383, 2024-09-10T19:16:50.706 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-10T19:16:50.706 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T19:16:50.707 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-10T19:16:50.707 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.15051", 2024-09-10T19:16:50.707 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.183:0/3632028672", 2024-09-10T19:16:50.707 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 31786 2024-09-10T19:16:50.707 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:50.707 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-10T19:16:50.707 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:50.707 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-10T19:16:50.707 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:16:37.298127+0000", 2024-09-10T19:16:50.707 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:16:50.707 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:50.707 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:50.707 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-10T19:16:50.707 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:16:37.298127+0000", 2024-09-10T19:16:50.707 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:16:50.708 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:50.708 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:50.708 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-10T19:16:50.708 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:16:37.298126+0000", 2024-09-10T19:16:50.708 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.999999 2024-09-10T19:16:50.708 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:50.708 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:50.708 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-10T19:16:50.708 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:16:37.298230+0000", 2024-09-10T19:16:50.708 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000103726 2024-09-10T19:16:50.708 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:50.708 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:50.708 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-10T19:16:50.708 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:16:37.298242+0000", 2024-09-10T19:16:50.708 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.171e-05 2024-09-10T19:16:50.709 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:50.709 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:50.709 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-10T19:16:50.709 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:16:37.298248+0000", 2024-09-10T19:16:50.709 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.1419999999999998e-06 2024-09-10T19:16:50.709 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:50.709 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:50.709 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:16:50.709 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:16:37.298844+0000", 2024-09-10T19:16:50.709 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00059611400000000002 2024-09-10T19:16:50.709 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:50.709 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:50.709 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-10T19:16:50.710 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:16:37.298952+0000", 2024-09-10T19:16:50.710 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000108643 2024-09-10T19:16:50.710 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:50.710 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:50.710 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 4,7", 2024-09-10T19:16:50.710 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:16:37.299073+0000", 2024-09-10T19:16:50.710 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00012027399999999999 2024-09-10T19:16:50.710 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:50.710 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:50.710 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-09-10T19:16:50.710 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:16:37.300300+0000", 2024-09-10T19:16:50.710 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.001227536 2024-09-10T19:16:50.710 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:50.710 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:50.711 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:16:50.711 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:16:37.301236+0000", 2024-09-10T19:16:50.711 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00093571799999999999 2024-09-10T19:16:50.711 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:50.712 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:50.712 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:16:50.712 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:16:37.467369+0000", 2024-09-10T19:16:50.712 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.166133005 2024-09-10T19:16:50.712 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:50.712 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:50.712 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-09-10T19:16:50.712 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:16:37.467450+0000", 2024-09-10T19:16:50.712 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.0977999999999996e-05 2024-09-10T19:16:50.712 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:50.712 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:50.712 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-10T19:16:50.713 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:16:37.467482+0000", 2024-09-10T19:16:50.713 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.2419999999999998e-05 2024-09-10T19:16:50.713 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:16:50.713 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-10T19:16:50.713 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:16:50.713 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:50.713 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:50.713 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_repop_reply(client.15051.0:31776 8.9 e160/148 ondisk, result = 0)", 2024-09-10T19:16:50.713 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-10T19:16:37.300420+0000", 2024-09-10T19:16:50.713 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 13.292189706, 2024-09-10T19:16:50.713 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.166951353, 2024-09-10T19:16:50.713 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-10T19:16:50.713 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "started", 2024-09-10T19:16:50.713 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-10T19:16:50.713 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:50.714 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-10T19:16:50.714 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:16:37.300420+0000", 2024-09-10T19:16:50.714 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:16:50.714 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:50.714 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:50.714 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-10T19:16:50.714 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:16:37.300420+0000", 2024-09-10T19:16:50.714 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:16:50.714 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:50.714 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:50.714 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-10T19:16:50.714 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:16:37.300420+0000", 2024-09-10T19:16:50.714 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999995 2024-09-10T19:16:50.714 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:50.715 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:50.715 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-10T19:16:50.715 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:16:37.300425+0000", 2024-09-10T19:16:50.715 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.3900000000000001e-06 2024-09-10T19:16:50.715 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:50.715 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:50.715 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-10T19:16:50.715 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:16:37.300436+0000", 2024-09-10T19:16:50.715 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.1233000000000001e-05 2024-09-10T19:16:50.715 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:50.715 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:50.715 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-10T19:16:50.715 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:16:37.300441+0000", 2024-09-10T19:16:50.715 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.011e-06 2024-09-10T19:16:50.715 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:50.716 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:50.716 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:16:50.716 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:16:37.467252+0000", 2024-09-10T19:16:50.716 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.16681051999999999 2024-09-10T19:16:50.716 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:50.716 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:50.716 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-10T19:16:50.716 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:16:37.467261+0000", 2024-09-10T19:16:50.716 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.7759999999999992e-06 2024-09-10T19:16:50.716 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:50.716 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:50.716 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-10T19:16:50.716 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:16:37.467372+0000", 2024-09-10T19:16:50.716 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000110109 2024-09-10T19:16:50.717 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:16:50.717 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-10T19:16:50.717 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:16:50.717 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:50.717 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:50.717 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_repop_reply(client.15051.0:31782 8.9 e160/148 ondisk, result = 0)", 2024-09-10T19:16:50.717 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-10T19:16:37.300464+0000", 2024-09-10T19:16:50.717 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 13.29214545, 2024-09-10T19:16:50.717 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.16703246399999999, 2024-09-10T19:16:50.717 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-10T19:16:50.717 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "started", 2024-09-10T19:16:50.717 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-10T19:16:50.717 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:50.717 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-10T19:16:50.717 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:16:37.300464+0000", 2024-09-10T19:16:50.717 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:16:50.718 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:50.718 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:50.718 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-10T19:16:50.718 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:16:37.300464+0000", 2024-09-10T19:16:50.718 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:16:50.718 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:50.718 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:50.718 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-10T19:16:50.718 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:16:37.300464+0000", 2024-09-10T19:16:50.718 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999995 2024-09-10T19:16:50.718 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:50.718 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:50.718 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-10T19:16:50.718 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:16:37.300467+0000", 2024-09-10T19:16:50.718 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.757e-06 2024-09-10T19:16:50.719 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:50.719 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:50.719 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-10T19:16:50.719 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:16:37.300473+0000", 2024-09-10T19:16:50.719 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.207e-06 2024-09-10T19:16:50.719 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:50.719 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:50.719 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-10T19:16:50.719 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:16:37.300477+0000", 2024-09-10T19:16:50.719 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.5930000000000002e-06 2024-09-10T19:16:50.719 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:50.719 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:50.719 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:16:50.720 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:16:37.467394+0000", 2024-09-10T19:16:50.720 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.16691710400000001 2024-09-10T19:16:50.720 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:50.720 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:50.720 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-10T19:16:50.720 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:16:37.467399+0000", 2024-09-10T19:16:50.720 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.361e-06 2024-09-10T19:16:50.720 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:50.720 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:50.720 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-10T19:16:50.720 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:16:37.467497+0000", 2024-09-10T19:16:50.720 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.7745999999999996e-05 2024-09-10T19:16:50.720 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:16:50.721 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-10T19:16:50.721 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:16:50.721 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:50.721 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:50.721 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_repop(client.15051.0:31790 8.3 e160/154 8:ccda2a59:::benchmark_data_smithi183_114924_object31789:head v 160'2128, mlcod=160'2126)", 2024-09-10T19:16:50.721 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-10T19:16:37.301214+0000", 2024-09-10T19:16:50.721 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 13.291395483000001, 2024-09-10T19:16:50.721 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.17241159, 2024-09-10T19:16:50.721 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-10T19:16:50.721 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T19:16:50.721 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-10T19:16:50.721 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:50.721 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-10T19:16:50.721 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:16:37.301214+0000", 2024-09-10T19:16:50.721 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:16:50.721 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:50.722 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:50.722 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-10T19:16:50.722 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:16:37.301214+0000", 2024-09-10T19:16:50.722 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:16:50.722 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:50.722 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:50.722 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-10T19:16:50.722 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:16:37.301214+0000", 2024-09-10T19:16:50.722 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.999999 2024-09-10T19:16:50.722 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:50.722 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:50.722 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-10T19:16:50.722 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:16:37.301236+0000", 2024-09-10T19:16:50.722 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.2789000000000002e-05 2024-09-10T19:16:50.722 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:50.723 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:50.723 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-10T19:16:50.723 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:16:37.301247+0000", 2024-09-10T19:16:50.723 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.04e-05 2024-09-10T19:16:50.723 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:50.723 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:50.723 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-10T19:16:50.723 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:16:37.301252+0000", 2024-09-10T19:16:50.723 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.8690000000000003e-06 2024-09-10T19:16:50.723 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:50.723 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:50.723 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:16:50.723 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:16:37.301311+0000", 2024-09-10T19:16:50.723 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.9277999999999997e-05 2024-09-10T19:16:50.723 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:50.723 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:50.724 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-10T19:16:50.724 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:16:37.301329+0000", 2024-09-10T19:16:50.724 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.8142999999999999e-05 2024-09-10T19:16:50.724 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:50.724 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:50.724 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-09-10T19:16:50.724 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:16:37.473571+0000", 2024-09-10T19:16:50.724 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.17224214700000001 2024-09-10T19:16:50.724 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:50.724 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:50.724 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-10T19:16:50.724 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:16:37.473626+0000", 2024-09-10T19:16:50.724 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.4713000000000003e-05 2024-09-10T19:16:50.725 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:16:50.725 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-10T19:16:50.725 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:16:50.725 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:16:50.725 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-10T19:16:50.725 INFO:teuthology.orchestra.run.smithi191.stdout:} 2024-09-10T19:16:50.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:16:50.726 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:16:50.726 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:16:50.824 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: 26 16 53379 53363 128.255 90.125 0.00574696 0.00779188 2024-09-10T19:16:50.827 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:16:50.859 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:16:50.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:16:50.861 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:16:50.961 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:16:51.002 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:16:51.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:16:51.003 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:16:51.104 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:16:51.136 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:16:51.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:16:51.137 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:16:51.158 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:16:50 smithi183 ceph-mon[29736]: pgmap v276: 148 pgs: 148 active+clean; 2.7 GiB data, 9.9 GiB used, 526 GiB / 536 GiB avail; 853 B/s rd, 95 MiB/s wr, 1.51k op/s 2024-09-10T19:16:51.158 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:16:50 smithi183 ceph-mon[29736]: from='client.? 172.21.15.183:0/3206754822' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T19:16:51.159 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:16:50 smithi183 ceph-mon[41562]: pgmap v276: 148 pgs: 148 active+clean; 2.7 GiB data, 9.9 GiB used, 526 GiB / 536 GiB avail; 853 B/s rd, 95 MiB/s wr, 1.51k op/s 2024-09-10T19:16:51.159 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:16:50 smithi183 ceph-mon[41562]: from='client.? 172.21.15.183:0/3206754822' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T19:16:51.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:16:50 smithi191 ceph-mon[32884]: pgmap v276: 148 pgs: 148 active+clean; 2.7 GiB data, 9.9 GiB used, 526 GiB / 536 GiB avail; 853 B/s rd, 95 MiB/s wr, 1.51k op/s 2024-09-10T19:16:51.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:16:50 smithi191 ceph-mon[32884]: from='client.? 172.21.15.183:0/3206754822' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T19:16:51.238 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:16:51.279 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:16:51.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:16:51.280 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:16:51.381 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:16:51.411 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:16:51.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:16:51.412 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:16:51.431 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Sep 10 19:16:51 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-y[29989]: ::ffff:172.21.15.191 - - [10/Sep/2024:19:16:51] "GET /metrics HTTP/1.1" 200 224093 "" "Prometheus/2.43.0" 2024-09-10T19:16:51.513 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:16:51.529 DEBUG:teuthology.orchestra.run.smithi191:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph daemon osd.7 dump_ops_in_flight 2024-09-10T19:16:51.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:16:51.555 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:16:51.555 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:16:51.655 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:16:51.683 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:16:51.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:16:51.684 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:16:51.784 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:16:51.818 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:16:51.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:16:51.819 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:16:51.824 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: 27 16 54793 54777 126.778 88.375 0.00636887 0.00788271 2024-09-10T19:16:51.920 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:16:51.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:16:51.961 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:16:51.962 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:16:52.010 INFO:teuthology.orchestra.run.smithi191.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.b/config 2024-09-10T19:16:52.063 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:16:52.115 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:16:52.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:16:52.115 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:16:52.216 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:16:52.256 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:16:52.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:16:52.257 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:16:52.358 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:16:52.383 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:16:52.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:16:52.384 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:16:52.485 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:16:52.511 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:16:52.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:16:52.512 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:16:52.613 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:16:52.642 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:16:52.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:16:52.643 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:16:52.744 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:16:52.773 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:16:52.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:16:52.774 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:16:52.824 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: 28 16 56241 56225 125.482 90.5 0.00832555 0.00796427 2024-09-10T19:16:52.875 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:16:52.907 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:16:52.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:16:52.908 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:16:53.009 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:16:53.040 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:16:53.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:16:53.041 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:16:53.097 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:16:52 smithi183 ceph-mon[29736]: pgmap v278: 148 pgs: 148 active+clean; 3.2 GiB data, 11 GiB used, 525 GiB / 536 GiB avail; 1.4 KiB/s rd, 144 MiB/s wr, 2.31k op/s 2024-09-10T19:16:53.097 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:16:52 smithi183 ceph-mon[41562]: pgmap v278: 148 pgs: 148 active+clean; 3.2 GiB data, 11 GiB used, 525 GiB / 536 GiB avail; 1.4 KiB/s rd, 144 MiB/s wr, 2.31k op/s 2024-09-10T19:16:53.141 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:16:53.180 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:16:53.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:16:53.180 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:16:53.281 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:16:53.321 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:16:53.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:16:53.322 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:16:53.326 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:16:52 smithi191 ceph-mon[32884]: pgmap v278: 148 pgs: 148 active+clean; 3.2 GiB data, 11 GiB used, 525 GiB / 536 GiB avail; 1.4 KiB/s rd, 144 MiB/s wr, 2.31k op/s 2024-09-10T19:16:53.424 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:16:53.450 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:16:53.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:16:53.451 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:16:53.552 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:16:53.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:16:53.590 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:16:53.590 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:16:53.691 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:16:53.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:16:53.720 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:16:53.720 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:16:53.821 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:16:53.824 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: 29 16 57614 57598 124.114 85.8125 0.00772895 0.00805229 2024-09-10T19:16:53.828 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-10T19:16:53.859 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:16:53.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:16:53.860 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:16:53.961 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:16:54.001 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:16:54.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:16:54.002 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:16:54.103 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:16:54.139 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:16:54.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:16:54.140 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:16:54.241 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:16:54.269 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:16:54.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:16:54.270 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:16:54.370 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:16:54.410 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:16:54.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:16:54.411 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:16:54.512 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:16:54.538 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:16:54.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:16:54.539 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:16:54.640 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:16:54.667 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:16:54.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:16:54.668 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:16:54.769 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:16:54.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:16:54.812 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:16:54.812 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:16:54.824 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: 30 16 58845 58829 122.542 76.9375 0.0094336 0.00815415 2024-09-10T19:16:54.913 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:16:54.936 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:16:54.943 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:16:54.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:16:54.944 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:16:54.959 INFO:teuthology.orchestra.run.smithi191.stdout:{ 2024-09-10T19:16:54.959 INFO:teuthology.orchestra.run.smithi191.stdout: "ops": [ 2024-09-10T19:16:54.960 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:54.960 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_repop(client.15051.0:58998 8.c e164/148 8:37761c29:::benchmark_data_smithi183_114924_object58997:head v 164'3714, mlcod=164'3712)", 2024-09-10T19:16:54.960 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-10T19:16:54.951344+0000", 2024-09-10T19:16:54.960 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 0.006307366, 2024-09-10T19:16:54.960 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0063353660000000003, 2024-09-10T19:16:54.960 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-10T19:16:54.960 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "started", 2024-09-10T19:16:54.960 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-10T19:16:54.960 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:54.960 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-10T19:16:54.960 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:16:54.951344+0000", 2024-09-10T19:16:54.960 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:16:54.960 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:54.961 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:54.961 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-10T19:16:54.961 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:16:54.951344+0000", 2024-09-10T19:16:54.961 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:16:54.961 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:54.961 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:54.961 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-10T19:16:54.961 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:16:54.951344+0000", 2024-09-10T19:16:54.961 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999995 2024-09-10T19:16:54.961 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:54.961 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:54.961 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-10T19:16:54.961 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:16:54.951445+0000", 2024-09-10T19:16:54.961 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000101337 2024-09-10T19:16:54.961 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:54.962 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:54.962 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-10T19:16:54.962 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:16:54.951457+0000", 2024-09-10T19:16:54.962 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.2272e-05 2024-09-10T19:16:54.962 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:54.962 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:54.962 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-10T19:16:54.962 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:16:54.951463+0000", 2024-09-10T19:16:54.962 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.5629999999999999e-06 2024-09-10T19:16:54.962 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:54.962 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:54.962 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:16:54.962 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:16:54.951790+0000", 2024-09-10T19:16:54.962 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00032735400000000002 2024-09-10T19:16:54.963 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:54.963 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:54.963 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-10T19:16:54.963 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:16:54.951843+0000", 2024-09-10T19:16:54.963 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.2629e-05 2024-09-10T19:16:54.963 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:16:54.963 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-10T19:16:54.963 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:16:54.963 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:54.963 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:54.963 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_repop(client.15051.0:59003 8.c e164/148 8:39db1573:::benchmark_data_smithi183_114924_object59002:head v 164'3715, mlcod=164'3712)", 2024-09-10T19:16:54.963 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-10T19:16:54.953844+0000", 2024-09-10T19:16:54.963 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 0.003807567, 2024-09-10T19:16:54.963 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0039043960000000001, 2024-09-10T19:16:54.964 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-10T19:16:54.964 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "started", 2024-09-10T19:16:54.964 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-10T19:16:54.964 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:54.964 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-10T19:16:54.964 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:16:54.953844+0000", 2024-09-10T19:16:54.964 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:16:54.964 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:54.964 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:54.964 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-10T19:16:54.964 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:16:54.953844+0000", 2024-09-10T19:16:54.964 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:16:54.964 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:54.964 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:54.965 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-10T19:16:54.965 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:16:54.953844+0000", 2024-09-10T19:16:54.965 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999995 2024-09-10T19:16:54.965 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:54.965 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:54.965 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-10T19:16:54.965 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:16:54.953872+0000", 2024-09-10T19:16:54.965 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.8034e-05 2024-09-10T19:16:54.965 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:54.965 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:54.965 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-10T19:16:54.965 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:16:54.953879+0000", 2024-09-10T19:16:54.965 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.8789999999999997e-06 2024-09-10T19:16:54.965 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:54.965 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:54.966 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-10T19:16:54.966 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:16:54.953883+0000", 2024-09-10T19:16:54.966 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.0450000000000002e-06 2024-09-10T19:16:54.966 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:54.966 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:54.966 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:16:54.966 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:16:54.955499+0000", 2024-09-10T19:16:54.966 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.001615813 2024-09-10T19:16:54.966 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:54.966 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:54.966 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-10T19:16:54.966 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:16:54.955518+0000", 2024-09-10T19:16:54.966 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.9397000000000001e-05 2024-09-10T19:16:54.966 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:16:54.966 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-10T19:16:54.967 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:16:54.967 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:54.967 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:54.967 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_repop(client.15051.0:59009 8.6 e164/148 8:6764ec17:::benchmark_data_smithi183_114924_object59008:head v 164'3637, mlcod=164'3635)", 2024-09-10T19:16:54.967 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-10T19:16:54.957333+0000", 2024-09-10T19:16:54.967 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 0.00031855300000000001, 2024-09-10T19:16:54.967 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00048257900000000002, 2024-09-10T19:16:54.967 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-10T19:16:54.967 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "started", 2024-09-10T19:16:54.967 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-10T19:16:54.967 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:54.967 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-10T19:16:54.967 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:16:54.957333+0000", 2024-09-10T19:16:54.967 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:16:54.967 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:54.968 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:54.968 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-10T19:16:54.968 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:16:54.957333+0000", 2024-09-10T19:16:54.968 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:16:54.968 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:54.968 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:54.968 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-10T19:16:54.968 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:16:54.957332+0000", 2024-09-10T19:16:54.968 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.999999 2024-09-10T19:16:54.968 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:54.968 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:54.968 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-10T19:16:54.968 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:16:54.957394+0000", 2024-09-10T19:16:54.968 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.1737999999999997e-05 2024-09-10T19:16:54.968 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:54.969 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:54.969 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-10T19:16:54.969 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:16:54.957417+0000", 2024-09-10T19:16:54.969 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.2602000000000001e-05 2024-09-10T19:16:54.969 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:54.969 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:54.969 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-10T19:16:54.969 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:16:54.957424+0000", 2024-09-10T19:16:54.969 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.8669999999999994e-06 2024-09-10T19:16:54.969 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:54.969 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:54.969 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:16:54.969 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:16:54.957619+0000", 2024-09-10T19:16:54.969 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00019414000000000001 2024-09-10T19:16:54.969 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:54.970 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:54.970 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-10T19:16:54.970 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:16:54.957670+0000", 2024-09-10T19:16:54.970 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.1415000000000002e-05 2024-09-10T19:16:54.970 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:16:54.970 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-10T19:16:54.970 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:16:54.970 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:54.970 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:54.970 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_repop(client.15051.0:59007 8.c e164/148)", 2024-09-10T19:16:54.970 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-10T19:16:54.957437+0000", 2024-09-10T19:16:54.970 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 0.00021461000000000001, 2024-09-10T19:16:54.970 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00042540699999999999, 2024-09-10T19:16:54.970 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-10T19:16:54.970 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "queued for pg", 2024-09-10T19:16:54.970 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-10T19:16:54.971 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:54.971 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-10T19:16:54.971 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:16:54.957437+0000", 2024-09-10T19:16:54.971 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:16:54.971 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:54.971 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:54.971 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-10T19:16:54.971 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:16:54.957437+0000", 2024-09-10T19:16:54.971 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:16:54.971 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:54.971 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:54.971 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-10T19:16:54.971 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:16:54.957437+0000", 2024-09-10T19:16:54.971 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999995 2024-09-10T19:16:54.972 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:54.972 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:54.972 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-10T19:16:54.972 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:16:54.957466+0000", 2024-09-10T19:16:54.972 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.9600000000000001e-05 2024-09-10T19:16:54.972 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:54.972 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:54.972 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-10T19:16:54.972 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:16:54.957474+0000", 2024-09-10T19:16:54.972 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.7740000000000004e-06 2024-09-10T19:16:54.972 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:16:54.972 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:16:54.972 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-10T19:16:54.972 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:16:54.957478+0000", 2024-09-10T19:16:54.973 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.9450000000000003e-06 2024-09-10T19:16:54.973 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:16:54.973 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-10T19:16:54.973 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:16:54.973 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:16:54.973 INFO:teuthology.orchestra.run.smithi191.stdout: ], 2024-09-10T19:16:54.973 INFO:teuthology.orchestra.run.smithi191.stdout: "num_ops": 4 2024-09-10T19:16:54.973 INFO:teuthology.orchestra.run.smithi191.stdout:} 2024-09-10T19:16:54.983 INFO:teuthology.orchestra.run.smithi183.stderr:set 8.3 pg_upmap mapping to [7,3,5] 2024-09-10T19:16:55.045 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:16:55.079 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:16:55.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:16:55.080 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:16:55.181 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:16:55.202 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:16:55.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:16:55.203 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:16:55.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:16:54 smithi191 ceph-mon[32884]: pgmap v279: 148 pgs: 148 active+clean; 3.2 GiB data, 11 GiB used, 525 GiB / 536 GiB avail; 639 B/s rd, 63 MiB/s wr, 1.01k op/s 2024-09-10T19:16:55.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:16:54 smithi191 ceph-mon[32884]: from='client.? 172.21.15.183:0/2025064040' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "8.3", "id": [7, 3, 5]}]: dispatch 2024-09-10T19:16:55.304 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:16:55.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:16:55.326 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:16:55.326 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:16:55.427 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:16:55.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:16:54 smithi183 ceph-mon[29736]: pgmap v279: 148 pgs: 148 active+clean; 3.2 GiB data, 11 GiB used, 525 GiB / 536 GiB avail; 639 B/s rd, 63 MiB/s wr, 1.01k op/s 2024-09-10T19:16:55.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:16:54 smithi183 ceph-mon[29736]: from='client.? 172.21.15.183:0/2025064040' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "8.3", "id": [7, 3, 5]}]: dispatch 2024-09-10T19:16:55.431 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:16:54 smithi183 ceph-mon[41562]: pgmap v279: 148 pgs: 148 active+clean; 3.2 GiB data, 11 GiB used, 525 GiB / 536 GiB avail; 639 B/s rd, 63 MiB/s wr, 1.01k op/s 2024-09-10T19:16:55.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:16:54 smithi183 ceph-mon[41562]: from='client.? 172.21.15.183:0/2025064040' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "8.3", "id": [7, 3, 5]}]: dispatch 2024-09-10T19:16:55.448 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:16:55.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:16:55.449 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:16:55.550 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:16:55.570 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:16:55.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:16:55.571 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:16:55.672 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:16:55.697 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:16:55.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:16:55.697 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:16:55.798 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:16:55.819 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:16:55.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:16:55.820 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:16:55.824 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: 31 16 59243 59227 119.391 24.875 0.0040126 0.00817174 2024-09-10T19:16:55.921 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:16:55.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:16:55.943 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:16:55.943 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:16:56.044 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:16:56.065 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:16:56.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:16:56.066 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:16:56.166 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:16:56.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:16:55 smithi183 ceph-mon[29736]: from='client.? 172.21.15.183:0/2025064040' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "8.3", "id": [7, 3, 5]}]': finished 2024-09-10T19:16:56.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:16:55 smithi183 ceph-mon[29736]: osdmap e165: 8 total, 8 up, 6 in 2024-09-10T19:16:56.181 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:16:55 smithi183 ceph-mon[41562]: from='client.? 172.21.15.183:0/2025064040' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "8.3", "id": [7, 3, 5]}]': finished 2024-09-10T19:16:56.181 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:16:55 smithi183 ceph-mon[41562]: osdmap e165: 8 total, 8 up, 6 in 2024-09-10T19:16:56.187 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:16:56.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:16:56.188 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:16:56.289 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:16:56.311 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:16:56.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:16:56.312 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:16:56.412 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:16:56.434 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:16:56.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:16:56.435 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:16:56.465 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:16:55 smithi191 ceph-mon[32884]: from='client.? 172.21.15.183:0/2025064040' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "8.3", "id": [7, 3, 5]}]': finished 2024-09-10T19:16:56.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:16:55 smithi191 ceph-mon[32884]: osdmap e165: 8 total, 8 up, 6 in 2024-09-10T19:16:56.535 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:16:56.560 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:16:56.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:16:56.561 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:16:56.662 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:16:56.684 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:16:56.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:16:56.686 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:16:56.786 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:16:56.811 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:16:56.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:16:56.812 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:16:56.825 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: 32 16 59243 59227 115.66 0 - 0.00817174 2024-09-10T19:16:56.913 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:16:56.935 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:16:56.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:16:56.936 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:16:56.990 INFO:teuthology.orchestra.run.smithi183.stderr:nodeep-scrub is set 2024-09-10T19:16:57.036 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:16:57.060 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:16:57.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:16:57.061 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:16:57.150 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:16:56 smithi183 ceph-mon[29736]: pgmap v281: 148 pgs: 148 active+clean; 3.6 GiB data, 12 GiB used, 524 GiB / 536 GiB avail; 1.2 KiB/s rd, 117 MiB/s wr, 1.87k op/s 2024-09-10T19:16:57.150 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:16:56 smithi183 ceph-mon[29736]: osdmap e166: 8 total, 8 up, 6 in 2024-09-10T19:16:57.150 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:16:56 smithi183 ceph-mon[29736]: from='client.? 172.21.15.183:0/1501162403' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-10T19:16:57.150 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:16:56 smithi183 ceph-mon[29736]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-10T19:16:57.150 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:16:56 smithi183 ceph-mon[41562]: pgmap v281: 148 pgs: 148 active+clean; 3.6 GiB data, 12 GiB used, 524 GiB / 536 GiB avail; 1.2 KiB/s rd, 117 MiB/s wr, 1.87k op/s 2024-09-10T19:16:57.150 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:16:56 smithi183 ceph-mon[41562]: osdmap e166: 8 total, 8 up, 6 in 2024-09-10T19:16:57.150 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:16:56 smithi183 ceph-mon[41562]: from='client.? 172.21.15.183:0/1501162403' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-10T19:16:57.150 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:16:56 smithi183 ceph-mon[41562]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-10T19:16:57.162 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:16:57.183 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:16:57.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:16:57.184 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:16:57.263 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:16:56 smithi191 ceph-mon[32884]: pgmap v281: 148 pgs: 148 active+clean; 3.6 GiB data, 12 GiB used, 524 GiB / 536 GiB avail; 1.2 KiB/s rd, 117 MiB/s wr, 1.87k op/s 2024-09-10T19:16:57.264 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:16:56 smithi191 ceph-mon[32884]: osdmap e166: 8 total, 8 up, 6 in 2024-09-10T19:16:57.264 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:16:56 smithi191 ceph-mon[32884]: from='client.? 172.21.15.183:0/1501162403' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-10T19:16:57.264 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:16:56 smithi191 ceph-mon[32884]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-10T19:16:57.285 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:16:57.293 DEBUG:teuthology.orchestra.run.smithi191:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph daemon osd.7 dump_blocked_ops 2024-09-10T19:16:57.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:16:57.310 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:16:57.311 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:16:57.412 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:16:57.433 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:16:57.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:16:57.434 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:16:57.534 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:16:57.541 INFO:journalctl@ceph.mgr.x.smithi191.stdout:Sep 10 19:16:57 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-x[34967]: ::ffff:172.21.15.191 - - [10/Sep/2024:19:16:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T19:16:57.557 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:16:57.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:16:57.558 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:16:57.567 INFO:teuthology.orchestra.run.smithi191.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.b/config 2024-09-10T19:16:57.659 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:16:57.680 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:16:57.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:16:57.681 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:16:57.781 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:16:57.810 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:16:57.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:16:57.811 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:16:57.824 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: 33 16 59269 59253 112.205 0.8125 0.00386586 0.00863477 2024-09-10T19:16:57.912 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:16:57.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:16:57.955 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:16:57.955 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:16:58.056 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:16:58.093 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:16:58.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:16:58.094 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:16:58.194 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:16:58.201 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:16:58 smithi191 ceph-mon[32884]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T19:16:58.202 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:16:58 smithi191 ceph-mon[32884]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-10T19:16:58.202 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:16:58 smithi191 ceph-mon[32884]: osdmap e167: 8 total, 8 up, 6 in 2024-09-10T19:16:58.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:16:58.248 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:16:58.249 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:16:58.349 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:16:58.389 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:16:58.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:16:58.390 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:16:58.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:16:57 smithi183 ceph-mon[29736]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T19:16:58.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:16:57 smithi183 ceph-mon[29736]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-10T19:16:58.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:16:57 smithi183 ceph-mon[29736]: osdmap e167: 8 total, 8 up, 6 in 2024-09-10T19:16:58.431 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:16:58 smithi183 ceph-mon[41562]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T19:16:58.431 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:16:58 smithi183 ceph-mon[41562]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-10T19:16:58.431 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:16:58 smithi183 ceph-mon[41562]: osdmap e167: 8 total, 8 up, 6 in 2024-09-10T19:16:58.491 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:16:58.519 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:16:58.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:16:58.520 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:16:58.620 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:16:58.660 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:16:58.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:16:58.661 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:16:58.762 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:16:58.805 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:16:58.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:16:58.806 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:16:58.825 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: 34 16 60416 60400 111.013 71.6875 0.00918657 0.00900141 2024-09-10T19:16:58.907 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:16:58.933 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:16:58.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:16:58.934 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:16:59.034 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:16:59.062 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:16:59.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:16:59.063 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:16:59.163 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:16:59.207 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:16:59.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:16:59.209 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:16:59.309 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:16:59.351 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:16:59 smithi191 ceph-mon[32884]: pgmap v284: 148 pgs: 148 active+clean; 3.6 GiB data, 12 GiB used, 524 GiB / 536 GiB avail; 853 B/s rd, 72 MiB/s wr, 1.15k op/s 2024-09-10T19:16:59.351 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:16:59 smithi191 ceph-mon[32884]: from='mgr.14556 ' entity='mgr.y' 2024-09-10T19:16:59.351 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:16:59 smithi191 ceph-mon[32884]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:16:59.352 INFO:journalctl@ceph.iscsi.iscsi.a.smithi191.stdout:Sep 10 19:16:59 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-iscsi-iscsi-a[61816]: debug there is no tcmu-runner data available 2024-09-10T19:16:59.356 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:16:59.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:16:59.357 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:16:59.405 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:16:59 smithi183 ceph-mon[29736]: pgmap v284: 148 pgs: 148 active+clean; 3.6 GiB data, 12 GiB used, 524 GiB / 536 GiB avail; 853 B/s rd, 72 MiB/s wr, 1.15k op/s 2024-09-10T19:16:59.405 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:16:59 smithi183 ceph-mon[29736]: from='mgr.14556 ' entity='mgr.y' 2024-09-10T19:16:59.405 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:16:59 smithi183 ceph-mon[29736]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:16:59.405 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:16:59 smithi183 ceph-mon[41562]: pgmap v284: 148 pgs: 148 active+clean; 3.6 GiB data, 12 GiB used, 524 GiB / 536 GiB avail; 853 B/s rd, 72 MiB/s wr, 1.15k op/s 2024-09-10T19:16:59.405 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:16:59 smithi183 ceph-mon[41562]: from='mgr.14556 ' entity='mgr.y' 2024-09-10T19:16:59.405 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:16:59 smithi183 ceph-mon[41562]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:16:59.458 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:16:59.497 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:16:59.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:16:59.498 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:16:59.599 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:16:59.626 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:16:59.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:16:59.628 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:16:59.729 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:16:59.758 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:16:59.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:16:59.759 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:16:59.825 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: 35 16 61653 61637 110.05 77.3125 0.0192349 0.00908198 2024-09-10T19:16:59.835 INFO:teuthology.orchestra.run.smithi191.stdout:{ 2024-09-10T19:16:59.835 INFO:teuthology.orchestra.run.smithi191.stdout: "ops": [], 2024-09-10T19:16:59.835 INFO:teuthology.orchestra.run.smithi191.stdout: "complaint_time": 30, 2024-09-10T19:16:59.836 INFO:teuthology.orchestra.run.smithi191.stdout: "num_blocked_ops": 0 2024-09-10T19:16:59.836 INFO:teuthology.orchestra.run.smithi191.stdout:} 2024-09-10T19:16:59.860 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:16:59.889 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:16:59.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:16:59.890 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:16:59.968 INFO:teuthology.orchestra.run.smithi183.stdout:osd.0: {} 2024-09-10T19:16:59.968 INFO:teuthology.orchestra.run.smithi183.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-10T19:16:59.982 INFO:teuthology.orchestra.run.smithi183.stdout:osd.1: {} 2024-09-10T19:16:59.983 INFO:teuthology.orchestra.run.smithi183.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-10T19:16:59.991 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:17:00.009 INFO:teuthology.orchestra.run.smithi183.stdout:osd.2: {} 2024-09-10T19:17:00.009 INFO:teuthology.orchestra.run.smithi183.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-10T19:17:00.028 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:17:00.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:17:00.029 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:17:00.036 INFO:teuthology.orchestra.run.smithi183.stdout:osd.3: {} 2024-09-10T19:17:00.036 INFO:teuthology.orchestra.run.smithi183.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-10T19:17:00.060 INFO:teuthology.orchestra.run.smithi183.stdout:osd.4: {} 2024-09-10T19:17:00.060 INFO:teuthology.orchestra.run.smithi183.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-10T19:17:00.090 INFO:teuthology.orchestra.run.smithi183.stdout:osd.5: {} 2024-09-10T19:17:00.090 INFO:teuthology.orchestra.run.smithi183.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-10T19:17:00.119 INFO:teuthology.orchestra.run.smithi183.stdout:osd.6: {} 2024-09-10T19:17:00.119 INFO:teuthology.orchestra.run.smithi183.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-10T19:17:00.129 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:17:00.149 INFO:teuthology.orchestra.run.smithi183.stdout:osd.7: {} 2024-09-10T19:17:00.149 INFO:teuthology.orchestra.run.smithi183.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-10T19:17:00.166 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:17:00.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:17:00.167 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:17:00.267 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:17:00.306 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:17:00.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:17:00.307 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:17:00.407 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:17:00.437 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:17:00.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:17:00.438 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:17:00.539 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:17:00.586 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:17:00.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:17:00.588 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:17:00.602 DEBUG:teuthology.orchestra.run.smithi191:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph daemon osd.7 dump_historic_ops 2024-09-10T19:17:00.688 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:17:00.737 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:17:00.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:17:00.738 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:17:00.825 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: 36 16 62915 62899 109.184 78.875 0.0112217 0.00915392 2024-09-10T19:17:00.839 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:17:00.887 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:17:00.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:17:00.888 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:17:00.923 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:17:00 smithi183 ceph-mon[29736]: from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:17:00.923 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:17:00 smithi183 ceph-mon[29736]: pgmap v285: 148 pgs: 148 active+clean; 3.6 GiB data, 12 GiB used, 524 GiB / 536 GiB avail; 853 B/s rd, 74 MiB/s wr, 1.18k op/s 2024-09-10T19:17:00.923 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:17:00 smithi183 ceph-mon[29736]: osdmap e168: 8 total, 8 up, 6 in 2024-09-10T19:17:00.923 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:17:00 smithi183 ceph-mon[29736]: from='client.? 172.21.15.183:0/2386850975' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T19:17:00.924 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:17:00 smithi183 ceph-mon[41562]: from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:17:00.924 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:17:00 smithi183 ceph-mon[41562]: pgmap v285: 148 pgs: 148 active+clean; 3.6 GiB data, 12 GiB used, 524 GiB / 536 GiB avail; 853 B/s rd, 74 MiB/s wr, 1.18k op/s 2024-09-10T19:17:00.924 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:17:00 smithi183 ceph-mon[41562]: osdmap e168: 8 total, 8 up, 6 in 2024-09-10T19:17:00.924 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:17:00 smithi183 ceph-mon[41562]: from='client.? 172.21.15.183:0/2386850975' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T19:17:00.965 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:17:00 smithi191 ceph-mon[32884]: from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:17:00.965 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:17:00 smithi191 ceph-mon[32884]: pgmap v285: 148 pgs: 148 active+clean; 3.6 GiB data, 12 GiB used, 524 GiB / 536 GiB avail; 853 B/s rd, 74 MiB/s wr, 1.18k op/s 2024-09-10T19:17:00.966 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:17:00 smithi191 ceph-mon[32884]: osdmap e168: 8 total, 8 up, 6 in 2024-09-10T19:17:00.966 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:17:00 smithi191 ceph-mon[32884]: from='client.? 172.21.15.183:0/2386850975' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T19:17:00.989 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:17:01.035 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:17:01.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:17:01.036 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:17:01.105 INFO:teuthology.orchestra.run.smithi191.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.b/config 2024-09-10T19:17:01.137 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:17:01.161 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:17:01.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:17:01.162 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:17:01.263 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:17:01.292 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:17:01.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:17:01.293 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:17:01.394 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:17:01.424 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:17:01.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:17:01.425 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:17:01.441 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph osd unset noscrub 2024-09-10T19:17:01.527 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:17:01.568 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:17:01.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:17:01.569 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:17:01.670 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:17:01.682 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Sep 10 19:17:01 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-y[29989]: ::ffff:172.21.15.191 - - [10/Sep/2024:19:17:01] "GET /metrics HTTP/1.1" 200 224159 "" "Prometheus/2.43.0" 2024-09-10T19:17:01.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:17:01.709 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:17:01.709 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:17:01.771 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:17:01.811 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:17:01.825 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: 37 16 63996 63980 108.059 67.5625 0.00878757 0.00924959 2024-09-10T19:17:01.840 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:17:01.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:17:01.841 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:17:01.942 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:17:01.970 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:17:01.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:17:01.972 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:17:02.072 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:17:02.093 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:17:02.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:17:02.094 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:17:02.195 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:17:02.230 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:17:02.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:17:02.231 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:17:02.332 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:17:02.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:17:02.361 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:17:02.362 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:17:02.463 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:17:02.508 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:17:02.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:17:02.510 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:17:02.610 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:17:02.662 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:17:02.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:17:02.663 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:17:02.763 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:17:02.792 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:17:02.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:17:02.793 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:17:02.825 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: 38 16 65014 64998 106.89 63.625 0.00617064 0.00935018 2024-09-10T19:17:02.894 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:17:02.930 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:17:02.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:17:02.931 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:17:02.960 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-10T19:17:02.965 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:17:02 smithi191 ceph-mon[32884]: pgmap v287: 148 pgs: 1 active+recovering+undersized+remapped, 147 active+clean; 3.8 GiB data, 13 GiB used, 524 GiB / 536 GiB avail; 853 B/s rd, 30 MiB/s wr, 474 op/s; 3811/185787 objects misplaced (2.051%); 736 KiB/s, 11 objects/s recovering 2024-09-10T19:17:02.965 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:17:02 smithi191 ceph-mon[32884]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T19:17:03.032 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:17:03.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:17:03.066 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:17:03.066 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:17:03.070 INFO:tasks.thrashosds.thrasher:in_osds: [0, 3, 4, 5, 6, 7] out_osds: [1, 2] dead_osds: [] live_osds: [0, 2, 3, 4, 5, 6, 7, 1] 2024-09-10T19:17:03.070 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-10T19:17:03.071 INFO:tasks.thrashosds.thrasher:Growing pool unique_pool_1 2024-09-10T19:17:03.071 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph pg dump --format=json 2024-09-10T19:17:03.138 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:17:02 smithi183 ceph-mon[29736]: pgmap v287: 148 pgs: 1 active+recovering+undersized+remapped, 147 active+clean; 3.8 GiB data, 13 GiB used, 524 GiB / 536 GiB avail; 853 B/s rd, 30 MiB/s wr, 474 op/s; 3811/185787 objects misplaced (2.051%); 736 KiB/s, 11 objects/s recovering 2024-09-10T19:17:03.138 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:17:02 smithi183 ceph-mon[29736]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T19:17:03.138 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:17:02 smithi183 ceph-mon[41562]: pgmap v287: 148 pgs: 1 active+recovering+undersized+remapped, 147 active+clean; 3.8 GiB data, 13 GiB used, 524 GiB / 536 GiB avail; 853 B/s rd, 30 MiB/s wr, 474 op/s; 3811/185787 objects misplaced (2.051%); 736 KiB/s, 11 objects/s recovering 2024-09-10T19:17:03.138 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:17:02 smithi183 ceph-mon[41562]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T19:17:03.167 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:17:03.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:17:03.231 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:17:03.231 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:17:03.332 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:17:03.362 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:17:03.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:17:03.363 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:17:03.464 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:17:03.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:17:03.495 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:17:03.495 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:17:03.596 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:17:03.626 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:17:03.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:17:03.627 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:17:03.728 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:17:03.755 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:17:03.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:17:03.756 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:17:03.825 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: 39 16 66164 66148 105.992 71.875 0.00318942 0.00942316 2024-09-10T19:17:03.858 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:17:03.885 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:17:03.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:17:03.886 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:17:03.986 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:17:04.009 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:17:04.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:17:04.010 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:17:04.110 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:17:04.166 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:17:04.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:17:04.167 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:17:04.267 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:17:04.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:17:04.312 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:17:04.312 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:17:04.413 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:17:04.436 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:17:04.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:17:04.437 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:17:04.538 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:17:04.564 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:17:04.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:17:04.565 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:17:04.665 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:17:04.693 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:17:04.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:17:04.694 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:17:04.794 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:17:04.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:17:04.820 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:17:04.820 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:17:04.825 INFO:tasks.radosbench.radosbench.0.smithi183.stdout:2024-09-10T19:17:04.825035+0000 min lat: 0.0011643 max lat: 2.77997 avg lat: 0.00951057 2024-09-10T19:17:04.825 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-10T19:17:04.825 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: 40 16 67265 67249 105.062 68.8125 0.00656514 0.00951057 2024-09-10T19:17:04.921 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:17:04.931 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:17:04 smithi183 ceph-mon[29736]: pgmap v288: 148 pgs: 1 active+recovering+undersized+remapped, 147 active+clean; 3.8 GiB data, 13 GiB used, 524 GiB / 536 GiB avail; 660 B/s rd, 23 MiB/s wr, 367 op/s; 3811/185787 objects misplaced (2.051%); 570 KiB/s, 8 objects/s recovering 2024-09-10T19:17:04.931 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:17:04 smithi183 ceph-mon[29736]: from='mgr.14556 ' entity='mgr.y' 2024-09-10T19:17:04.931 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:17:04 smithi183 ceph-mon[29736]: from='mgr.14556 ' entity='mgr.y' 2024-09-10T19:17:04.931 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:17:04 smithi183 ceph-mon[29736]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T19:17:04.931 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:17:04 smithi183 ceph-mon[29736]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T19:17:04.931 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:17:04 smithi183 ceph-mon[29736]: from='mgr.14556 ' entity='mgr.y' 2024-09-10T19:17:04.932 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:17:04 smithi183 ceph-mon[41562]: pgmap v288: 148 pgs: 1 active+recovering+undersized+remapped, 147 active+clean; 3.8 GiB data, 13 GiB used, 524 GiB / 536 GiB avail; 660 B/s rd, 23 MiB/s wr, 367 op/s; 3811/185787 objects misplaced (2.051%); 570 KiB/s, 8 objects/s recovering 2024-09-10T19:17:04.932 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:17:04 smithi183 ceph-mon[41562]: from='mgr.14556 ' entity='mgr.y' 2024-09-10T19:17:04.932 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:17:04 smithi183 ceph-mon[41562]: from='mgr.14556 ' entity='mgr.y' 2024-09-10T19:17:04.932 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:17:04 smithi183 ceph-mon[41562]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T19:17:04.932 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:17:04 smithi183 ceph-mon[41562]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T19:17:04.932 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:17:04 smithi183 ceph-mon[41562]: from='mgr.14556 ' entity='mgr.y' 2024-09-10T19:17:04.986 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:17:04.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:17:04.987 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:17:05.087 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:17:05.110 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:17:04 smithi191 ceph-mon[32884]: pgmap v288: 148 pgs: 1 active+recovering+undersized+remapped, 147 active+clean; 3.8 GiB data, 13 GiB used, 524 GiB / 536 GiB avail; 660 B/s rd, 23 MiB/s wr, 367 op/s; 3811/185787 objects misplaced (2.051%); 570 KiB/s, 8 objects/s recovering 2024-09-10T19:17:05.110 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:17:04 smithi191 ceph-mon[32884]: from='mgr.14556 ' entity='mgr.y' 2024-09-10T19:17:05.110 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:17:04 smithi191 ceph-mon[32884]: from='mgr.14556 ' entity='mgr.y' 2024-09-10T19:17:05.110 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:17:04 smithi191 ceph-mon[32884]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T19:17:05.110 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:17:04 smithi191 ceph-mon[32884]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T19:17:05.110 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:17:04 smithi191 ceph-mon[32884]: from='mgr.14556 ' entity='mgr.y' 2024-09-10T19:17:05.113 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:17:05.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:17:05.114 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:17:05.214 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:17:05.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:17:05.252 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:17:05.252 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:17:05.353 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:17:05.386 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:17:05.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:17:05.387 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:17:05.488 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:17:05.530 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:17:05.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:17:05.531 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:17:05.570 INFO:teuthology.orchestra.run.smithi191.stdout:ERROR: (22) Invalid argument 2024-09-10T19:17:05.570 INFO:teuthology.orchestra.run.smithi191.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-10T19:17:05.632 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:17:05.665 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:17:05.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:17:05.666 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:17:05.735 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:17:05.766 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:17:05.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:17:05.806 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:17:05.807 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:17:05.825 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: 41 16 68369 68353 104.183 69 0.0131294 0.00959337 2024-09-10T19:17:05.907 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:17:05.943 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:17:05.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:17:05.943 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:17:06.044 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:17:06.080 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:17:06.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:17:06.081 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:17:06.182 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:17:06.219 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:17:06.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:17:06.220 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:17:06.321 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:17:06.346 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:17:06.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:17:06.348 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph daemon osd.1 dump_ops_in_flight 2024-09-10T19:17:06.348 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:17:06.449 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:17:06.481 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:17:06.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:17:06.482 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:17:06.576 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:17:06.582 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:17:06.653 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:17:06.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:17:06.654 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:17:06.755 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:17:06.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:17:06.792 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:17:06.793 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:17:06.825 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: 42 16 69618 69602 103.561 78.0625 0.00884566 0.00965111 2024-09-10T19:17:06.893 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:17:06.922 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:17:06.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:17:06.923 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:17:07.024 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:17:07.032 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:17:06 smithi183 ceph-mon[29736]: pgmap v289: 148 pgs: 1 active+recovering+undersized+remapped, 147 active+clean; 4.1 GiB data, 14 GiB used, 523 GiB / 536 GiB avail; 1.1 KiB/s rd, 60 MiB/s wr, 964 op/s; 3621/202530 objects misplaced (1.788%); 1.9 MiB/s, 29 objects/s recovering 2024-09-10T19:17:07.033 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:17:06 smithi183 ceph-mon[41562]: pgmap v289: 148 pgs: 1 active+recovering+undersized+remapped, 147 active+clean; 4.1 GiB data, 14 GiB used, 523 GiB / 536 GiB avail; 1.1 KiB/s rd, 60 MiB/s wr, 964 op/s; 3621/202530 objects misplaced (1.788%); 1.9 MiB/s, 29 objects/s recovering 2024-09-10T19:17:07.059 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:17:07.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:17:07.060 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:17:07.161 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:17:07.188 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:17:07.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:17:07.189 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:17:07.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:17:06 smithi191 ceph-mon[32884]: pgmap v289: 148 pgs: 1 active+recovering+undersized+remapped, 147 active+clean; 4.1 GiB data, 14 GiB used, 523 GiB / 536 GiB avail; 1.1 KiB/s rd, 60 MiB/s wr, 964 op/s; 3621/202530 objects misplaced (1.788%); 1.9 MiB/s, 29 objects/s recovering 2024-09-10T19:17:07.290 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:17:07.327 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:17:07.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:17:07.328 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:17:07.428 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:17:07.459 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:17:07.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:17:07.460 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:17:07.560 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:17:07.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:17:07.601 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:17:07.601 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:17:07.702 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:17:07.716 INFO:journalctl@ceph.mgr.x.smithi191.stdout:Sep 10 19:17:07 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-x[34967]: ::ffff:172.21.15.191 - - [10/Sep/2024:19:17:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T19:17:07.746 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:17:07.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:17:07.747 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:17:07.825 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: 43 16 70892 70876 103.004 79.625 0.0096178 0.00970157 2024-09-10T19:17:07.847 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:17:07.881 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:17:07.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:17:07.882 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:17:07.983 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:17:08.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:17:08.020 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:17:08.020 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:17:08.120 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:17:08.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:17:08.165 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:17:08.165 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:17:08.266 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:17:08.293 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:17:08.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:17:08.294 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:17:08.395 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:17:08.421 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:17:08.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:17:08.422 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:17:08.523 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:17:08.562 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:17:08.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:17:08.563 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:17:08.664 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:17:08.694 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:17:08.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:17:08.695 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:17:08.796 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:17:08.825 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: 44 16 72172 72156 102.481 80 0.0163484 0.00975343 2024-09-10T19:17:08.833 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:17:08.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:17:08.834 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:17:08.864 INFO:teuthology.orchestra.run.smithi183.stderr:noscrub is unset 2024-09-10T19:17:08.935 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:17:08.966 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:17:08.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:17:08.967 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:17:09.067 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:17:09.110 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:17:09.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:17:09.111 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:17:09.129 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:17:08 smithi191 ceph-mon[32884]: purged_snaps scrub starts 2024-09-10T19:17:09.129 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:17:08 smithi191 ceph-mon[32884]: purged_snaps scrub ok 2024-09-10T19:17:09.129 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:17:08 smithi191 ceph-mon[32884]: pgmap v290: 148 pgs: 1 active+recovering+undersized+remapped, 147 active+clean; 4.1 GiB data, 14 GiB used, 523 GiB / 536 GiB avail; 1023 B/s rd, 53 MiB/s wr, 842 op/s; 3621/202530 objects misplaced (1.788%); 1.6 MiB/s, 25 objects/s recovering 2024-09-10T19:17:09.129 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:17:08 smithi191 ceph-mon[32884]: osdmap e169: 8 total, 8 up, 6 in 2024-09-10T19:17:09.129 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:17:08 smithi191 ceph-mon[32884]: from='client.? 172.21.15.183:0/2584728233' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-10T19:17:09.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:17:08 smithi183 ceph-mon[29736]: purged_snaps scrub starts 2024-09-10T19:17:09.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:17:08 smithi183 ceph-mon[29736]: purged_snaps scrub ok 2024-09-10T19:17:09.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:17:08 smithi183 ceph-mon[29736]: pgmap v290: 148 pgs: 1 active+recovering+undersized+remapped, 147 active+clean; 4.1 GiB data, 14 GiB used, 523 GiB / 536 GiB avail; 1023 B/s rd, 53 MiB/s wr, 842 op/s; 3621/202530 objects misplaced (1.788%); 1.6 MiB/s, 25 objects/s recovering 2024-09-10T19:17:09.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:17:08 smithi183 ceph-mon[29736]: osdmap e169: 8 total, 8 up, 6 in 2024-09-10T19:17:09.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:17:08 smithi183 ceph-mon[29736]: from='client.? 172.21.15.183:0/2584728233' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-10T19:17:09.181 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:17:08 smithi183 ceph-mon[41562]: purged_snaps scrub starts 2024-09-10T19:17:09.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:17:08 smithi183 ceph-mon[41562]: purged_snaps scrub ok 2024-09-10T19:17:09.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:17:08 smithi183 ceph-mon[41562]: pgmap v290: 148 pgs: 1 active+recovering+undersized+remapped, 147 active+clean; 4.1 GiB data, 14 GiB used, 523 GiB / 536 GiB avail; 1023 B/s rd, 53 MiB/s wr, 842 op/s; 3621/202530 objects misplaced (1.788%); 1.6 MiB/s, 25 objects/s recovering 2024-09-10T19:17:09.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:17:08 smithi183 ceph-mon[41562]: osdmap e169: 8 total, 8 up, 6 in 2024-09-10T19:17:09.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:17:08 smithi183 ceph-mon[41562]: from='client.? 172.21.15.183:0/2584728233' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-10T19:17:09.212 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:17:09.249 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:17:09.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:17:09.250 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:17:09.350 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:17:09.378 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:17:09.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:17:09.379 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:17:09.466 INFO:journalctl@ceph.iscsi.iscsi.a.smithi191.stdout:Sep 10 19:17:09 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-iscsi-iscsi-a[61816]: debug there is no tcmu-runner data available 2024-09-10T19:17:09.480 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:17:09.509 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:17:09.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:17:09.510 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:17:09.611 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:17:09.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:17:09.648 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:17:09.648 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:17:09.749 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:17:09.792 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:17:09.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:17:09.793 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:17:09.826 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: 45 16 73424 73408 101.942 78.25 0.0144759 0.00980421 2024-09-10T19:17:09.894 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:17:09.930 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:17:09.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:17:09.930 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:17:10.031 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:17:10.073 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:17:10.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:17:10.074 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:17:10.175 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:17:10.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:17:09 smithi183 ceph-mon[29736]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T19:17:10.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:17:09 smithi183 ceph-mon[29736]: from='client.? 172.21.15.183:0/2584728233' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-10T19:17:10.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:17:09 smithi183 ceph-mon[29736]: osdmap e170: 8 total, 8 up, 6 in 2024-09-10T19:17:10.181 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:17:09 smithi183 ceph-mon[41562]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T19:17:10.181 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:17:09 smithi183 ceph-mon[41562]: from='client.? 172.21.15.183:0/2584728233' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-10T19:17:10.181 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:17:09 smithi183 ceph-mon[41562]: osdmap e170: 8 total, 8 up, 6 in 2024-09-10T19:17:10.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:17:09 smithi191 ceph-mon[32884]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T19:17:10.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:17:09 smithi191 ceph-mon[32884]: from='client.? 172.21.15.183:0/2584728233' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-10T19:17:10.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:17:09 smithi191 ceph-mon[32884]: osdmap e170: 8 total, 8 up, 6 in 2024-09-10T19:17:10.220 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:17:10.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:17:10.221 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:17:10.322 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:17:10.359 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:17:10.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:17:10.360 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:17:10.460 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:17:10.496 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:17:10.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:17:10.497 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:17:10.597 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:17:10.631 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:17:10.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:17:10.632 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:17:10.733 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:17:10.760 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:17:10.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:17:10.761 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:17:10.826 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: 46 16 74689 74673 101.445 79.0625 0.0123848 0.009853 2024-09-10T19:17:10.861 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:17:10.899 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:17:10.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:17:10.900 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:17:11.000 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:17:11.036 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:17:11.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:17:11.037 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:17:11.138 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:17:11.165 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:17:11.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:17:11.166 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:17:11.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:17:10 smithi183 ceph-mon[29736]: purged_snaps scrub starts 2024-09-10T19:17:11.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:17:10 smithi183 ceph-mon[29736]: purged_snaps scrub ok 2024-09-10T19:17:11.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:17:10 smithi183 ceph-mon[29736]: from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:17:11.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:17:10 smithi183 ceph-mon[29736]: pgmap v293: 148 pgs: 1 active+recovering+undersized+remapped, 147 active+clean; 4.1 GiB data, 14 GiB used, 522 GiB / 536 GiB avail; 639 B/s rd, 46 MiB/s wr, 743 op/s; 3621/203622 objects misplaced (1.778%); 1.5 MiB/s, 23 objects/s recovering 2024-09-10T19:17:11.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:17:10 smithi183 ceph-mon[29736]: 5.a scrub starts 2024-09-10T19:17:11.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:17:10 smithi183 ceph-mon[29736]: osdmap e171: 8 total, 8 up, 6 in 2024-09-10T19:17:11.182 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:17:10 smithi183 ceph-mon[29736]: 4.a scrub starts 2024-09-10T19:17:11.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:17:10 smithi183 ceph-mon[41562]: purged_snaps scrub starts 2024-09-10T19:17:11.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:17:10 smithi183 ceph-mon[41562]: purged_snaps scrub ok 2024-09-10T19:17:11.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:17:10 smithi183 ceph-mon[41562]: from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:17:11.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:17:10 smithi183 ceph-mon[41562]: pgmap v293: 148 pgs: 1 active+recovering+undersized+remapped, 147 active+clean; 4.1 GiB data, 14 GiB used, 522 GiB / 536 GiB avail; 639 B/s rd, 46 MiB/s wr, 743 op/s; 3621/203622 objects misplaced (1.778%); 1.5 MiB/s, 23 objects/s recovering 2024-09-10T19:17:11.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:17:10 smithi183 ceph-mon[41562]: 5.a scrub starts 2024-09-10T19:17:11.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:17:10 smithi183 ceph-mon[41562]: osdmap e171: 8 total, 8 up, 6 in 2024-09-10T19:17:11.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:17:10 smithi183 ceph-mon[41562]: 4.a scrub starts 2024-09-10T19:17:11.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:17:10 smithi191 ceph-mon[32884]: purged_snaps scrub starts 2024-09-10T19:17:11.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:17:10 smithi191 ceph-mon[32884]: purged_snaps scrub ok 2024-09-10T19:17:11.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:17:10 smithi191 ceph-mon[32884]: from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:17:11.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:17:10 smithi191 ceph-mon[32884]: pgmap v293: 148 pgs: 1 active+recovering+undersized+remapped, 147 active+clean; 4.1 GiB data, 14 GiB used, 522 GiB / 536 GiB avail; 639 B/s rd, 46 MiB/s wr, 743 op/s; 3621/203622 objects misplaced (1.778%); 1.5 MiB/s, 23 objects/s recovering 2024-09-10T19:17:11.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:17:10 smithi191 ceph-mon[32884]: 5.a scrub starts 2024-09-10T19:17:11.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:17:10 smithi191 ceph-mon[32884]: osdmap e171: 8 total, 8 up, 6 in 2024-09-10T19:17:11.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:17:10 smithi191 ceph-mon[32884]: 4.a scrub starts 2024-09-10T19:17:11.267 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:17:11.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:17:11.296 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:17:11.296 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:17:11.396 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:17:11.432 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:17:11.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:17:11.433 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:17:11.534 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:17:11.561 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:17:11.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:17:11.562 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:17:11.628 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Sep 10 19:17:11 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-y[29989]: ::ffff:172.21.15.191 - - [10/Sep/2024:19:17:11] "GET /metrics HTTP/1.1" 200 224159 "" "Prometheus/2.43.0" 2024-09-10T19:17:11.663 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:17:11.694 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:17:11.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:17:11.695 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:17:11.796 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:17:11.826 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: 47 16 75958 75942 100.974 79.3125 0.00804885 0.00989881 2024-09-10T19:17:11.828 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-10T19:17:11.837 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:17:11.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:17:11.838 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:17:11.931 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:17:11 smithi183 ceph-mon[29736]: 6.17 scrub starts 2024-09-10T19:17:11.931 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:17:11 smithi183 ceph-mon[29736]: 6.0 scrub starts 2024-09-10T19:17:11.931 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:17:11 smithi183 ceph-mon[29736]: 4.12 scrub starts 2024-09-10T19:17:11.931 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:17:11 smithi183 ceph-mon[41562]: 6.17 scrub starts 2024-09-10T19:17:11.931 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:17:11 smithi183 ceph-mon[41562]: 6.0 scrub starts 2024-09-10T19:17:11.932 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:17:11 smithi183 ceph-mon[41562]: 4.12 scrub starts 2024-09-10T19:17:11.938 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:17:11.962 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:17:11.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:17:11.963 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:17:12.064 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:17:12.091 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:17:12.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:17:12.091 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:17:12.192 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:17:12.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:17:11 smithi191 ceph-mon[32884]: 6.17 scrub starts 2024-09-10T19:17:12.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:17:11 smithi191 ceph-mon[32884]: 6.0 scrub starts 2024-09-10T19:17:12.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:17:11 smithi191 ceph-mon[32884]: 4.12 scrub starts 2024-09-10T19:17:12.230 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:17:12.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:17:12.231 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:17:12.332 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:17:12.361 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:17:12.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:17:12.362 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:17:12.463 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:17:12.498 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:17:12.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:17:12.499 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:17:12.600 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:17:12.630 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:17:12.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:17:12.631 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:17:12.732 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:17:12.761 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:17:12.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:17:12.762 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:17:12.827 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: 48 16 77155 77139 100.427 74.8125 0.0107873 0.00995285 2024-09-10T19:17:12.863 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:17:12.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:17:12.892 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:17:12.892 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:17:12.931 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:17:12 smithi183 ceph-mon[29736]: pgmap v295: 148 pgs: 1 active+recovering+undersized+remapped, 147 active+clean; 4.5 GiB data, 15 GiB used, 521 GiB / 536 GiB avail; 853 B/s rd, 65 MiB/s wr, 1.03k op/s; 3425/221124 objects misplaced (1.549%); 2.0 MiB/s, 32 objects/s recovering 2024-09-10T19:17:12.931 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:17:12 smithi183 ceph-mon[29736]: 4.16 scrub starts 2024-09-10T19:17:12.931 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:17:12 smithi183 ceph-mon[41562]: pgmap v295: 148 pgs: 1 active+recovering+undersized+remapped, 147 active+clean; 4.5 GiB data, 15 GiB used, 521 GiB / 536 GiB avail; 853 B/s rd, 65 MiB/s wr, 1.03k op/s; 3425/221124 objects misplaced (1.549%); 2.0 MiB/s, 32 objects/s recovering 2024-09-10T19:17:12.931 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:17:12 smithi183 ceph-mon[41562]: 4.16 scrub starts 2024-09-10T19:17:12.936 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-09-10T19:17:12.938 INFO:teuthology.orchestra.run.smithi183.stderr:dumped all 2024-09-10T19:17:12.993 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:17:13.021 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:17:13.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:17:13.022 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:17:13.123 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:17:13.167 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:17:13.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:17:13.168 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:17:13.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:17:12 smithi191 ceph-mon[32884]: pgmap v295: 148 pgs: 1 active+recovering+undersized+remapped, 147 active+clean; 4.5 GiB data, 15 GiB used, 521 GiB / 536 GiB avail; 853 B/s rd, 65 MiB/s wr, 1.03k op/s; 3425/221124 objects misplaced (1.549%); 2.0 MiB/s, 32 objects/s recovering 2024-09-10T19:17:13.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:17:12 smithi191 ceph-mon[32884]: 4.16 scrub starts 2024-09-10T19:17:13.234 INFO:teuthology.orchestra.run.smithi183.stdout:osd.0: {} 2024-09-10T19:17:13.234 INFO:teuthology.orchestra.run.smithi183.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-10T19:17:13.262 INFO:teuthology.orchestra.run.smithi183.stdout:osd.1: {} 2024-09-10T19:17:13.262 INFO:teuthology.orchestra.run.smithi183.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-10T19:17:13.269 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:17:13.291 INFO:teuthology.orchestra.run.smithi183.stdout:osd.2: {} 2024-09-10T19:17:13.291 INFO:teuthology.orchestra.run.smithi183.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-10T19:17:13.304 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:17:13.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:17:13.305 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:17:13.316 INFO:teuthology.orchestra.run.smithi183.stdout:osd.3: {} 2024-09-10T19:17:13.316 INFO:teuthology.orchestra.run.smithi183.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-10T19:17:13.339 INFO:teuthology.orchestra.run.smithi183.stdout:osd.4: {} 2024-09-10T19:17:13.339 INFO:teuthology.orchestra.run.smithi183.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-10T19:17:13.365 INFO:teuthology.orchestra.run.smithi183.stdout:osd.5: {} 2024-09-10T19:17:13.365 INFO:teuthology.orchestra.run.smithi183.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-10T19:17:13.397 INFO:teuthology.orchestra.run.smithi183.stdout:osd.6: {} 2024-09-10T19:17:13.397 INFO:teuthology.orchestra.run.smithi183.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-10T19:17:13.407 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:17:13.424 INFO:teuthology.orchestra.run.smithi183.stdout:osd.7: {} 2024-09-10T19:17:13.425 INFO:teuthology.orchestra.run.smithi183.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-10T19:17:13.437 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:17:13.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:17:13.438 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:17:13.539 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:17:13.577 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:17:13.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:17:13.578 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:17:13.679 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:17:13.706 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:17:13.714 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:17:13.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:17:13.715 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:17:13.816 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:17:13.827 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: 49 16 78433 78417 100.007 79.875 0.00930984 0.00999332 2024-09-10T19:17:13.859 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:17:13.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:17:13.860 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:17:13.961 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:17:14.000 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:17:14.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:17:14.001 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:17:14.102 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:17:14.135 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:17:14.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:17:14.136 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:17:14.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:17:13 smithi183 ceph-mon[29736]: from='client.15126 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-10T19:17:14.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:17:13 smithi183 ceph-mon[29736]: from='client.? 172.21.15.183:0/4144080302' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T19:17:14.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:17:13 smithi183 ceph-mon[29736]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:17:14.181 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:17:13 smithi183 ceph-mon[41562]: from='client.15126 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-10T19:17:14.181 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:17:13 smithi183 ceph-mon[41562]: from='client.? 172.21.15.183:0/4144080302' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T19:17:14.181 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:17:13 smithi183 ceph-mon[41562]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:17:14.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:17:13 smithi191 ceph-mon[32884]: from='client.15126 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-10T19:17:14.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:17:13 smithi191 ceph-mon[32884]: from='client.? 172.21.15.183:0/4144080302' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T19:17:14.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:17:13 smithi191 ceph-mon[32884]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:17:14.240 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:17:14.270 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:17:14.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:17:14.271 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:17:14.291 INFO:teuthology.orchestra.run.smithi183.stdout:{"pg_ready":true,"pg_map":{"version":295,"stamp":"2024-09-10T19:17:11.732989+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":4818278620,"num_objects":73708,"num_object_clones":0,"num_object_copies":221124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3425,"num_objects_unfound":0,"num_objects_dirty":73708,"num_whiteouts":0,"num_read":899,"num_read_kb":756,"num_write":74055,"num_write_kb":4706439,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":637,"num_bytes_recovered":30413237,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":66525,"ondisk_log_size":66525,"up":444,"acting":443,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":445,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":562470912,"kb_used":15824648,"kb_used_data":13982272,"kb_used_omap":9,"kb_used_meta":1842294,"kb_avail":546646264,"statfs":{"total":575970213888,"available":559765774336,"internally_reserved":0,"allocated":14317846528,"data_stored":14315097574,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":10224,"internal_metadata":1886509072},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":406192128,"num_objects":6198,"num_object_clones":0,"num_object_copies":18594,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-196,"num_objects_unfound":0,"num_objects_dirty":6198,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":6198,"num_write_kb":396672,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":196,"num_bytes_recovered":12845056,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001786"},"pg_stats":[{"pgid":"6.1b","version":"0'0","reported_seq":135,"reported_epoch":170,"state":"active+clean","last_fresh":"2024-09-10T19:17:08.869469+0000","last_change":"2024-09-10T19:13:08.710758+0000","last_active":"2024-09-10T19:17:08.869469+0000","last_peered":"2024-09-10T19:17:08.869469+0000","last_clean":"2024-09-10T19:17:08.869469+0000","last_became_active":"2024-09-10T19:10:14.056187+0000","last_became_peered":"2024-09-10T19:10:14.056187+0000","last_unstale":"2024-09-10T19:17:08.869469+0000","last_undegraded":"2024-09-10T19:17:08.869469+0000","last_fullsized":"2024-09-10T19:17:08.869469+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:13:08.710679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:13:08.710679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0027252499999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":170,"reported_epoch":170,"state":"active+clean","last_fresh":"2024-09-10T19:17:08.864721+0000","last_change":"2024-09-10T19:16:38.194521+0000","last_active":"2024-09-10T19:17:08.864721+0000","last_peered":"2024-09-10T19:17:08.864721+0000","last_clean":"2024-09-10T19:17:08.864721+0000","last_became_active":"2024-09-10T19:16:24.836961+0000","last_became_peered":"2024-09-10T19:16:24.836961+0000","last_unstale":"2024-09-10T19:17:08.864721+0000","last_undegraded":"2024-09-10T19:17:08.864721+0000","last_fullsized":"2024-09-10T19:17:08.864721+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:16:38.194472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:16:38.194472+0000","last_clean_scrub_stamp":"2024-09-10T19:16:38.194472+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:17:46.775869+0000","scrub_duration":0.0023678900000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.19","version":"66'15","reported_seq":190,"reported_epoch":170,"state":"active+clean","last_fresh":"2024-09-10T19:17:08.869205+0000","last_change":"2024-09-10T19:16:24.879212+0000","last_active":"2024-09-10T19:17:08.869205+0000","last_peered":"2024-09-10T19:17:08.869205+0000","last_clean":"2024-09-10T19:17:08.869205+0000","last_became_active":"2024-09-10T19:16:24.844951+0000","last_became_peered":"2024-09-10T19:16:24.844951+0000","last_unstale":"2024-09-10T19:17:08.869205+0000","last_undegraded":"2024-09-10T19:17:08.869205+0000","last_fullsized":"2024-09-10T19:17:08.869205+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'15","last_scrub_stamp":"2024-09-10T19:13:47.165848+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:13:47.165848+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.004982841,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":165,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-09-10T19:17:07.846351+0000","last_change":"2024-09-10T19:16:24.845473+0000","last_active":"2024-09-10T19:17:07.846351+0000","last_peered":"2024-09-10T19:17:07.846351+0000","last_clean":"2024-09-10T19:17:07.846351+0000","last_became_active":"2024-09-10T19:16:24.845111+0000","last_became_peered":"2024-09-10T19:16:24.845111+0000","last_unstale":"2024-09-10T19:17:07.846351+0000","last_undegraded":"2024-09-10T19:17:07.846351+0000","last_fullsized":"2024-09-10T19:17:07.846351+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:15:33.976696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:15:33.976696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0015310580000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,7],"acting":[4,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"59'1","reported_seq":159,"reported_epoch":170,"state":"active+clean","last_fresh":"2024-09-10T19:17:08.869620+0000","last_change":"2024-09-10T19:16:30.798883+0000","last_active":"2024-09-10T19:17:08.869620+0000","last_peered":"2024-09-10T19:17:08.869620+0000","last_clean":"2024-09-10T19:17:08.869620+0000","last_became_active":"2024-09-10T19:10:07.686143+0000","last_became_peered":"2024-09-10T19:10:07.686143+0000","last_unstale":"2024-09-10T19:17:08.869620+0000","last_undegraded":"2024-09-10T19:17:08.869620+0000","last_fullsized":"2024-09-10T19:17:08.869620+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'1","last_scrub_stamp":"2024-09-10T19:16:30.798795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:16:30.798795+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:17:44.600087+0000","scrub_duration":0.0067002989999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":874,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"66'9","reported_seq":180,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-09-10T19:17:07.846439+0000","last_change":"2024-09-10T19:14:55.117473+0000","last_active":"2024-09-10T19:17:07.846439+0000","last_peered":"2024-09-10T19:17:07.846439+0000","last_clean":"2024-09-10T19:17:07.846439+0000","last_became_active":"2024-09-10T19:13:54.716117+0000","last_became_peered":"2024-09-10T19:13:54.716117+0000","last_unstale":"2024-09-10T19:17:07.846439+0000","last_undegraded":"2024-09-10T19:17:07.846439+0000","last_fullsized":"2024-09-10T19:17:07.846439+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'9","last_scrub_stamp":"2024-09-10T19:14:55.117387+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:14:55.117387+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0074980940000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,3],"acting":[4,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":110,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-09-10T19:17:07.848632+0000","last_change":"2024-09-10T19:16:27.753997+0000","last_active":"2024-09-10T19:17:07.848632+0000","last_peered":"2024-09-10T19:17:07.848632+0000","last_clean":"2024-09-10T19:17:07.848632+0000","last_became_active":"2024-09-10T19:16:24.859261+0000","last_became_peered":"2024-09-10T19:16:24.859261+0000","last_unstale":"2024-09-10T19:17:07.848632+0000","last_undegraded":"2024-09-10T19:17:07.848632+0000","last_fullsized":"2024-09-10T19:17:07.848632+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:16:27.753912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:16:27.753912+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:17:49.796558+0000","scrub_duration":0.0029373799999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":153,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-09-10T19:17:07.847815+0000","last_change":"2024-09-10T19:15:39.910169+0000","last_active":"2024-09-10T19:17:07.847815+0000","last_peered":"2024-09-10T19:17:07.847815+0000","last_clean":"2024-09-10T19:17:07.847815+0000","last_became_active":"2024-09-10T19:13:54.713538+0000","last_became_peered":"2024-09-10T19:13:54.713538+0000","last_unstale":"2024-09-10T19:17:07.847815+0000","last_undegraded":"2024-09-10T19:17:07.847815+0000","last_fullsized":"2024-09-10T19:17:07.847815+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:15:39.910118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:15:39.910118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0013551839999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,7],"acting":[4,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":150,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-09-10T19:17:07.855173+0000","last_change":"2024-09-10T19:16:25.595265+0000","last_active":"2024-09-10T19:17:07.855173+0000","last_peered":"2024-09-10T19:17:07.855173+0000","last_clean":"2024-09-10T19:17:07.855173+0000","last_became_active":"2024-09-10T19:10:07.690728+0000","last_became_peered":"2024-09-10T19:10:07.690728+0000","last_unstale":"2024-09-10T19:17:07.855173+0000","last_undegraded":"2024-09-10T19:17:07.855173+0000","last_fullsized":"2024-09-10T19:17:07.855173+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:16:25.595127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:16:25.595127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:17:51.885390+0000","scrub_duration":0.003922633,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1b","version":"66'5","reported_seq":189,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-09-10T19:17:07.850701+0000","last_change":"2024-09-10T19:16:35.966457+0000","last_active":"2024-09-10T19:17:07.850701+0000","last_peered":"2024-09-10T19:17:07.850701+0000","last_clean":"2024-09-10T19:17:07.850701+0000","last_became_active":"2024-09-10T19:16:24.843708+0000","last_became_peered":"2024-09-10T19:16:24.843708+0000","last_unstale":"2024-09-10T19:17:07.850701+0000","last_undegraded":"2024-09-10T19:17:07.850701+0000","last_fullsized":"2024-09-10T19:17:07.850701+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'5","last_scrub_stamp":"2024-09-10T19:16:35.966402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:16:35.966402+0000","objects_scrubbed":2,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:17:45.645005+0000","scrub_duration":0.0019325740000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":153,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-09-10T19:17:07.854741+0000","last_change":"2024-09-10T19:15:33.518718+0000","last_active":"2024-09-10T19:17:07.854741+0000","last_peered":"2024-09-10T19:17:07.854741+0000","last_clean":"2024-09-10T19:17:07.854741+0000","last_became_active":"2024-09-10T19:13:54.719063+0000","last_became_peered":"2024-09-10T19:13:54.719063+0000","last_unstale":"2024-09-10T19:17:07.854741+0000","last_undegraded":"2024-09-10T19:17:07.854741+0000","last_fullsized":"2024-09-10T19:17:07.854741+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:15:33.518601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:15:33.518601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0013951650000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":161,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-09-10T19:17:07.848702+0000","last_change":"2024-09-10T19:16:24.845721+0000","last_active":"2024-09-10T19:17:07.848702+0000","last_peered":"2024-09-10T19:17:07.848702+0000","last_clean":"2024-09-10T19:17:07.848702+0000","last_became_active":"2024-09-10T19:16:24.845130+0000","last_became_peered":"2024-09-10T19:16:24.845130+0000","last_unstale":"2024-09-10T19:17:07.848702+0000","last_undegraded":"2024-09-10T19:17:07.848702+0000","last_fullsized":"2024-09-10T19:17:07.848702+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:15:45.616184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:15:45.616184+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:17:13.888995+0000","scrub_duration":0.0013989510000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,0],"acting":[5,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":150,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-09-10T19:17:07.855638+0000","last_change":"2024-09-10T19:16:24.859112+0000","last_active":"2024-09-10T19:17:07.855638+0000","last_peered":"2024-09-10T19:17:07.855638+0000","last_clean":"2024-09-10T19:17:07.855638+0000","last_became_active":"2024-09-10T19:16:24.858800+0000","last_became_peered":"2024-09-10T19:16:24.858800+0000","last_unstale":"2024-09-10T19:17:07.855638+0000","last_undegraded":"2024-09-10T19:17:07.855638+0000","last_fullsized":"2024-09-10T19:17:07.855638+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:12:54.477149+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:12:54.477149+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0039768319999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1a","version":"66'9","reported_seq":153,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-09-10T19:17:07.848107+0000","last_change":"2024-09-10T19:13:02.431528+0000","last_active":"2024-09-10T19:17:07.848107+0000","last_peered":"2024-09-10T19:17:07.848107+0000","last_clean":"2024-09-10T19:17:07.848107+0000","last_became_active":"2024-09-10T19:10:09.688987+0000","last_became_peered":"2024-09-10T19:10:09.688987+0000","last_unstale":"2024-09-10T19:17:07.848107+0000","last_undegraded":"2024-09-10T19:17:07.848107+0000","last_fullsized":"2024-09-10T19:17:07.848107+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'9","last_scrub_stamp":"2024-09-10T19:13:02.431443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:13:02.431443+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0084913290000000006,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":146,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-09-10T19:17:07.849372+0000","last_change":"2024-09-10T19:16:37.116464+0000","last_active":"2024-09-10T19:17:07.849372+0000","last_peered":"2024-09-10T19:17:07.849372+0000","last_clean":"2024-09-10T19:17:07.849372+0000","last_became_active":"2024-09-10T19:10:11.701799+0000","last_became_peered":"2024-09-10T19:10:11.701799+0000","last_unstale":"2024-09-10T19:17:07.849372+0000","last_undegraded":"2024-09-10T19:17:07.849372+0000","last_fullsized":"2024-09-10T19:17:07.849372+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:16:37.116371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:16:37.116371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:18:03.024881+0000","scrub_duration":0.028100301000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":161,"reported_epoch":170,"state":"active+clean","last_fresh":"2024-09-10T19:17:08.862502+0000","last_change":"2024-09-10T19:16:24.836814+0000","last_active":"2024-09-10T19:17:08.862502+0000","last_peered":"2024-09-10T19:17:08.862502+0000","last_clean":"2024-09-10T19:17:08.862502+0000","last_became_active":"2024-09-10T19:16:24.836477+0000","last_became_peered":"2024-09-10T19:16:24.836477+0000","last_unstale":"2024-09-10T19:17:08.862502+0000","last_undegraded":"2024-09-10T19:17:08.862502+0000","last_fullsized":"2024-09-10T19:17:08.862502+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:15:33.839248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:15:33.839248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0032019980000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1","version":"171'4601","reported_seq":4641,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-09-10T19:17:09.987619+0000","last_change":"2024-09-10T19:16:24.842553+0000","last_active":"2024-09-10T19:17:09.987619+0000","last_peered":"2024-09-10T19:17:09.987619+0000","last_clean":"2024-09-10T19:17:09.987619+0000","last_became_active":"2024-09-10T19:16:24.842218+0000","last_became_peered":"2024-09-10T19:16:24.842218+0000","last_unstale":"2024-09-10T19:17:09.987619+0000","last_undegraded":"2024-09-10T19:17:09.987619+0000","last_fullsized":"2024-09-10T19:17:09.987619+0000","mapping_epoch":154,"log_start":"156'400","ondisk_log_start":"156'400","created":148,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:16:11.777014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:16:11.777014+0000","last_clean_scrub_stamp":"2024-09-10T19:16:11.777014+0000","objects_scrubbed":0,"log_size":4201,"log_dups_size":0,"ondisk_log_size":4201,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:17:31.601183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":301531136,"num_objects":4601,"num_object_clones":0,"num_object_copies":13803,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4601,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4601,"num_write_kb":294464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.d","version":"66'17","reported_seq":238,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-09-10T19:17:07.846635+0000","last_change":"2024-09-10T19:16:32.999033+0000","last_active":"2024-09-10T19:17:07.846635+0000","last_peered":"2024-09-10T19:17:07.846635+0000","last_clean":"2024-09-10T19:17:07.846635+0000","last_became_active":"2024-09-10T19:16:24.847056+0000","last_became_peered":"2024-09-10T19:16:24.847056+0000","last_unstale":"2024-09-10T19:17:07.846635+0000","last_undegraded":"2024-09-10T19:17:07.846635+0000","last_fullsized":"2024-09-10T19:17:07.846635+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'17","last_scrub_stamp":"2024-09-10T19:16:32.998950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:16:32.998950+0000","objects_scrubbed":8,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:17:37.707354+0000","scrub_duration":0.0095505250000000007,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,0],"acting":[4,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":150,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-09-10T19:17:07.857350+0000","last_change":"2024-09-10T19:16:39.716870+0000","last_active":"2024-09-10T19:17:07.857350+0000","last_peered":"2024-09-10T19:17:07.857350+0000","last_clean":"2024-09-10T19:17:07.857350+0000","last_became_active":"2024-09-10T19:10:07.691119+0000","last_became_peered":"2024-09-10T19:10:07.691119+0000","last_unstale":"2024-09-10T19:17:07.857350+0000","last_undegraded":"2024-09-10T19:17:07.857350+0000","last_fullsized":"2024-09-10T19:17:07.857350+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:16:39.716674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:16:39.716674+0000","last_clean_scrub_stamp":"2024-09-10T19:16:39.716674+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:17:54.073755+0000","scrub_duration":0.043895318000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":103,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-09-10T19:17:07.845711+0000","last_change":"2024-09-10T19:16:26.999105+0000","last_active":"2024-09-10T19:17:07.845711+0000","last_peered":"2024-09-10T19:17:07.845711+0000","last_clean":"2024-09-10T19:17:07.845711+0000","last_became_active":"2024-09-10T19:13:54.712883+0000","last_became_peered":"2024-09-10T19:13:54.712883+0000","last_unstale":"2024-09-10T19:17:07.845711+0000","last_undegraded":"2024-09-10T19:17:07.845711+0000","last_fullsized":"2024-09-10T19:17:07.845711+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:16:26.999036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:16:26.999036+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:17:47.793977+0000","scrub_duration":0.0050193119999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":29,"reported_epoch":170,"state":"active+clean","last_fresh":"2024-09-10T19:17:08.869623+0000","last_change":"2024-09-10T19:16:24.858656+0000","last_active":"2024-09-10T19:17:08.869623+0000","last_peered":"2024-09-10T19:17:08.869623+0000","last_clean":"2024-09-10T19:17:08.869623+0000","last_became_active":"2024-09-10T19:16:24.857843+0000","last_became_peered":"2024-09-10T19:16:24.857843+0000","last_unstale":"2024-09-10T19:17:08.869623+0000","last_undegraded":"2024-09-10T19:17:08.869623+0000","last_fullsized":"2024-09-10T19:17:08.869623+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:15:47.818381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:15:47.818381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.0","version":"171'4630","reported_seq":4661,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-09-10T19:17:09.955901+0000","last_change":"2024-09-10T19:16:12.795560+0000","last_active":"2024-09-10T19:17:09.955901+0000","last_peered":"2024-09-10T19:17:09.955901+0000","last_clean":"2024-09-10T19:17:09.955901+0000","last_became_active":"2024-09-10T19:16:12.795331+0000","last_became_peered":"2024-09-10T19:16:12.795331+0000","last_unstale":"2024-09-10T19:17:09.955901+0000","last_undegraded":"2024-09-10T19:17:09.955901+0000","last_fullsized":"2024-09-10T19:17:09.955901+0000","mapping_epoch":148,"log_start":"157'700","ondisk_log_start":"157'700","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:16:11.777014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:16:11.777014+0000","last_clean_scrub_stamp":"2024-09-10T19:16:11.777014+0000","objects_scrubbed":0,"log_size":3930,"log_dups_size":0,"ondisk_log_size":3930,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:17:33.649961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":303431680,"num_objects":4630,"num_object_clones":0,"num_object_copies":13890,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4630,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4630,"num_write_kb":296320,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,4],"acting":[3,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"66'10","reported_seq":162,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-09-10T19:17:07.845894+0000","last_change":"2024-09-10T19:13:51.416075+0000","last_active":"2024-09-10T19:17:07.845894+0000","last_peered":"2024-09-10T19:17:07.845894+0000","last_clean":"2024-09-10T19:17:07.845894+0000","last_became_active":"2024-09-10T19:10:09.688321+0000","last_became_peered":"2024-09-10T19:10:09.688321+0000","last_unstale":"2024-09-10T19:17:07.845894+0000","last_undegraded":"2024-09-10T19:17:07.845894+0000","last_fullsized":"2024-09-10T19:17:07.845894+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'10","last_scrub_stamp":"2024-09-10T19:13:51.415998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:13:51.415998+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.005972799,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":101,"reported_epoch":170,"state":"active+clean","last_fresh":"2024-09-10T19:17:08.868507+0000","last_change":"2024-09-10T19:15:42.119912+0000","last_active":"2024-09-10T19:17:08.868507+0000","last_peered":"2024-09-10T19:17:08.868507+0000","last_clean":"2024-09-10T19:17:08.868507+0000","last_became_active":"2024-09-10T19:13:54.731134+0000","last_became_peered":"2024-09-10T19:13:54.731134+0000","last_unstale":"2024-09-10T19:17:08.868507+0000","last_undegraded":"2024-09-10T19:17:08.868507+0000","last_fullsized":"2024-09-10T19:17:08.868507+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:15:42.119590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:15:42.119590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.018001347000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,5],"acting":[3,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"155'10","reported_seq":107,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-09-10T19:17:07.854971+0000","last_change":"2024-09-10T19:16:29.626806+0000","last_active":"2024-09-10T19:17:07.854971+0000","last_peered":"2024-09-10T19:17:07.854971+0000","last_clean":"2024-09-10T19:17:07.854971+0000","last_became_active":"2024-09-10T19:16:24.851236+0000","last_became_peered":"2024-09-10T19:16:24.851236+0000","last_unstale":"2024-09-10T19:17:07.854971+0000","last_undegraded":"2024-09-10T19:17:07.854971+0000","last_fullsized":"2024-09-10T19:17:07.854971+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"155'10","last_scrub_stamp":"2024-09-10T19:16:29.626572+0000","last_deep_scrub":"80'9","last_deep_scrub_stamp":"2024-09-10T19:11:25.156465+0000","last_clean_scrub_stamp":"2024-09-10T19:16:29.626572+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:17:47.526795+0000","scrub_duration":0.0067064810000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,0],"acting":[7,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":166,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-09-10T19:17:07.846839+0000","last_change":"2024-09-10T19:16:24.852100+0000","last_active":"2024-09-10T19:17:07.846839+0000","last_peered":"2024-09-10T19:17:07.846839+0000","last_clean":"2024-09-10T19:17:07.846839+0000","last_became_active":"2024-09-10T19:16:24.851829+0000","last_became_peered":"2024-09-10T19:16:24.851829+0000","last_unstale":"2024-09-10T19:17:07.846839+0000","last_undegraded":"2024-09-10T19:17:07.846839+0000","last_fullsized":"2024-09-10T19:17:07.846839+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:15:29.988976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:15:29.988976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0053787150000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,7],"acting":[4,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.3","version":"171'4806","reported_seq":5764,"reported_epoch":171,"state":"active+recovering+undersized+remapped","last_fresh":"2024-09-10T19:17:09.955984+0000","last_change":"2024-09-10T19:16:57.881591+0000","last_active":"2024-09-10T19:17:09.955984+0000","last_peered":"2024-09-10T19:17:09.955984+0000","last_clean":"2024-09-10T19:16:54.964816+0000","last_became_active":"2024-09-10T19:16:57.770167+0000","last_became_peered":"2024-09-10T19:16:57.770167+0000","last_unstale":"2024-09-10T19:17:09.955984+0000","last_undegraded":"2024-09-10T19:17:09.955984+0000","last_fullsized":"2024-09-10T19:16:56.987468+0000","mapping_epoch":166,"log_start":"158'900","ondisk_log_start":"158'900","created":148,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:16:11.777014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:16:11.777014+0000","last_clean_scrub_stamp":"2024-09-10T19:16:11.777014+0000","objects_scrubbed":0,"log_size":3906,"log_dups_size":0,"ondisk_log_size":3906,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:17:33.700734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":314966016,"num_objects":4806,"num_object_clones":0,"num_object_copies":14418,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3425,"num_objects_unfound":0,"num_objects_dirty":4806,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4806,"num_write_kb":307584,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":455,"num_bytes_recovered":29818880,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[3,5],"avail_no_missing":["3","5","6"],"object_location_counts":[{"shards":"3,5","objects":1381},{"shards":"3,5,6","objects":3425}],"blocked_by":[],"up_primary":7,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"66'15","reported_seq":154,"reported_epoch":170,"state":"active+clean","last_fresh":"2024-09-10T19:17:08.869831+0000","last_change":"2024-09-10T19:16:25.769523+0000","last_active":"2024-09-10T19:17:08.869831+0000","last_peered":"2024-09-10T19:17:08.869831+0000","last_clean":"2024-09-10T19:17:08.869831+0000","last_became_active":"2024-09-10T19:13:54.711634+0000","last_became_peered":"2024-09-10T19:13:54.711634+0000","last_unstale":"2024-09-10T19:17:08.869831+0000","last_undegraded":"2024-09-10T19:17:08.869831+0000","last_fullsized":"2024-09-10T19:17:08.869831+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'15","last_scrub_stamp":"2024-09-10T19:16:25.769436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:16:25.769436+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:17:46.035940+0000","scrub_duration":0.0046254920000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":110,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-09-10T19:17:07.854095+0000","last_change":"2024-09-10T19:16:24.857467+0000","last_active":"2024-09-10T19:17:07.854095+0000","last_peered":"2024-09-10T19:17:07.854095+0000","last_clean":"2024-09-10T19:17:07.854095+0000","last_became_active":"2024-09-10T19:16:24.857179+0000","last_became_peered":"2024-09-10T19:16:24.857179+0000","last_unstale":"2024-09-10T19:17:07.854095+0000","last_undegraded":"2024-09-10T19:17:07.854095+0000","last_fullsized":"2024-09-10T19:17:07.854095+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:15:29.553448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:15:29.553448+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0062080679999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,6],"acting":[7,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.e","version":"66'8","reported_seq":227,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-09-10T19:17:07.846968+0000","last_change":"2024-09-10T19:13:40.475460+0000","last_active":"2024-09-10T19:17:07.846968+0000","last_peered":"2024-09-10T19:17:07.846968+0000","last_clean":"2024-09-10T19:17:07.846968+0000","last_became_active":"2024-09-10T19:10:11.699232+0000","last_became_peered":"2024-09-10T19:10:11.699232+0000","last_unstale":"2024-09-10T19:17:07.846968+0000","last_undegraded":"2024-09-10T19:17:07.846968+0000","last_fullsized":"2024-09-10T19:17:07.846968+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'8","last_scrub_stamp":"2024-09-10T19:13:40.475375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:13:40.475375+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.035806152000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":161,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-09-10T19:17:07.848682+0000","last_change":"2024-09-10T19:16:24.855938+0000","last_active":"2024-09-10T19:17:07.848682+0000","last_peered":"2024-09-10T19:17:07.848682+0000","last_clean":"2024-09-10T19:17:07.848682+0000","last_became_active":"2024-09-10T19:16:24.855627+0000","last_became_peered":"2024-09-10T19:16:24.855627+0000","last_unstale":"2024-09-10T19:17:07.848682+0000","last_undegraded":"2024-09-10T19:17:07.848682+0000","last_fullsized":"2024-09-10T19:17:07.848682+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:15:44.633043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:15:44.633043+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.00368805,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,6],"acting":[5,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.2","version":"170'4576","reported_seq":4606,"reported_epoch":170,"state":"active+clean","last_fresh":"2024-09-10T19:17:09.697437+0000","last_change":"2024-09-10T19:16:12.794918+0000","last_active":"2024-09-10T19:17:09.697437+0000","last_peered":"2024-09-10T19:17:09.697437+0000","last_clean":"2024-09-10T19:17:09.697437+0000","last_became_active":"2024-09-10T19:16:12.794134+0000","last_became_peered":"2024-09-10T19:16:12.794134+0000","last_unstale":"2024-09-10T19:17:09.697437+0000","last_undegraded":"2024-09-10T19:17:09.697437+0000","last_fullsized":"2024-09-10T19:17:09.697437+0000","mapping_epoch":148,"log_start":"155'200","ondisk_log_start":"155'200","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:16:11.777014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:16:11.777014+0000","last_clean_scrub_stamp":"2024-09-10T19:16:11.777014+0000","objects_scrubbed":0,"log_size":4376,"log_dups_size":0,"ondisk_log_size":4376,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:17:40.992082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":299892736,"num_objects":4576,"num_object_clones":0,"num_object_copies":13728,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4576,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4576,"num_write_kb":292864,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,5],"acting":[4,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"84'16","reported_seq":193,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-09-10T19:17:07.847039+0000","last_change":"2024-09-10T19:15:47.914487+0000","last_active":"2024-09-10T19:17:07.847039+0000","last_peered":"2024-09-10T19:17:07.847039+0000","last_clean":"2024-09-10T19:17:07.847039+0000","last_became_active":"2024-09-10T19:13:54.717022+0000","last_became_peered":"2024-09-10T19:13:54.717022+0000","last_unstale":"2024-09-10T19:17:07.847039+0000","last_undegraded":"2024-09-10T19:17:07.847039+0000","last_fullsized":"2024-09-10T19:17:07.847039+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'16","last_scrub_stamp":"2024-09-10T19:15:47.914420+0000","last_deep_scrub":"84'16","last_deep_scrub_stamp":"2024-09-10T19:11:56.157975+0000","last_clean_scrub_stamp":"2024-09-10T19:15:47.914420+0000","objects_scrubbed":5,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0019153390000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,3],"acting":[4,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":157,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-09-10T19:17:07.853775+0000","last_change":"2024-09-10T19:15:42.450149+0000","last_active":"2024-09-10T19:17:07.853775+0000","last_peered":"2024-09-10T19:17:07.853775+0000","last_clean":"2024-09-10T19:17:07.853775+0000","last_became_active":"2024-09-10T19:13:54.720293+0000","last_became_peered":"2024-09-10T19:13:54.720293+0000","last_unstale":"2024-09-10T19:17:07.853775+0000","last_undegraded":"2024-09-10T19:17:07.853775+0000","last_fullsized":"2024-09-10T19:17:07.853775+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:15:42.450068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:11:53.066849+0000","last_clean_scrub_stamp":"2024-09-10T19:15:42.450068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0037708360000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":136,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-09-10T19:17:07.850747+0000","last_change":"2024-09-10T19:13:06.358188+0000","last_active":"2024-09-10T19:17:07.850747+0000","last_peered":"2024-09-10T19:17:07.850747+0000","last_clean":"2024-09-10T19:17:07.850747+0000","last_became_active":"2024-09-10T19:10:11.692215+0000","last_became_peered":"2024-09-10T19:10:11.692215+0000","last_unstale":"2024-09-10T19:17:07.850747+0000","last_undegraded":"2024-09-10T19:17:07.850747+0000","last_fullsized":"2024-09-10T19:17:07.850747+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:13:06.358109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:13:06.358109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0026724549999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":135,"reported_epoch":170,"state":"active+clean","last_fresh":"2024-09-10T19:17:08.868960+0000","last_change":"2024-09-10T19:13:54.304493+0000","last_active":"2024-09-10T19:17:08.868960+0000","last_peered":"2024-09-10T19:17:08.868960+0000","last_clean":"2024-09-10T19:17:08.868960+0000","last_became_active":"2024-09-10T19:10:14.053736+0000","last_became_peered":"2024-09-10T19:10:14.053736+0000","last_unstale":"2024-09-10T19:17:08.868960+0000","last_undegraded":"2024-09-10T19:17:08.868960+0000","last_fullsized":"2024-09-10T19:17:08.868960+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:13:54.304435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:13:54.304435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0011043820000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.d","version":"170'4616","reported_seq":4656,"reported_epoch":170,"state":"active+clean","last_fresh":"2024-09-10T19:17:09.809556+0000","last_change":"2024-09-10T19:16:24.858800+0000","last_active":"2024-09-10T19:17:09.809556+0000","last_peered":"2024-09-10T19:17:09.809556+0000","last_clean":"2024-09-10T19:17:09.809556+0000","last_became_active":"2024-09-10T19:16:24.858460+0000","last_became_peered":"2024-09-10T19:16:24.858460+0000","last_unstale":"2024-09-10T19:17:09.809556+0000","last_undegraded":"2024-09-10T19:17:09.809556+0000","last_fullsized":"2024-09-10T19:17:09.809556+0000","mapping_epoch":154,"log_start":"156'400","ondisk_log_start":"156'400","created":148,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:16:11.777014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:16:11.777014+0000","last_clean_scrub_stamp":"2024-09-10T19:16:11.777014+0000","objects_scrubbed":0,"log_size":4216,"log_dups_size":0,"ondisk_log_size":4216,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:17:15.173931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":302514176,"num_objects":4616,"num_object_clones":0,"num_object_copies":13848,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4616,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4616,"num_write_kb":295424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1","version":"66'14","reported_seq":41,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-09-10T19:17:07.845477+0000","last_change":"2024-09-10T19:16:24.903236+0000","last_active":"2024-09-10T19:17:07.845477+0000","last_peered":"2024-09-10T19:17:07.845477+0000","last_clean":"2024-09-10T19:17:07.845477+0000","last_became_active":"2024-09-10T19:16:24.852821+0000","last_became_peered":"2024-09-10T19:16:24.852821+0000","last_unstale":"2024-09-10T19:17:07.845477+0000","last_undegraded":"2024-09-10T19:17:07.845477+0000","last_fullsized":"2024-09-10T19:17:07.845477+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'14","last_scrub_stamp":"2024-09-10T19:15:41.748101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:15:41.748101+0000","objects_scrubbed":7,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0055403099999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,7],"acting":[4,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":160,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-09-10T19:17:07.856196+0000","last_change":"2024-09-10T19:16:25.582072+0000","last_active":"2024-09-10T19:17:07.856196+0000","last_peered":"2024-09-10T19:17:07.856196+0000","last_clean":"2024-09-10T19:17:07.856196+0000","last_became_active":"2024-09-10T19:16:24.855861+0000","last_became_peered":"2024-09-10T19:16:24.855861+0000","last_unstale":"2024-09-10T19:17:07.856196+0000","last_undegraded":"2024-09-10T19:17:07.856196+0000","last_fullsized":"2024-09-10T19:17:07.856196+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:16:25.581993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:16:25.581993+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:17:36.903746+0000","scrub_duration":0.0041345990000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.0","version":"117'10","reported_seq":251,"reported_epoch":170,"state":"active+clean","last_fresh":"2024-09-10T19:17:08.868427+0000","last_change":"2024-09-10T19:15:35.973188+0000","last_active":"2024-09-10T19:17:08.868427+0000","last_peered":"2024-09-10T19:17:08.868427+0000","last_clean":"2024-09-10T19:17:08.868427+0000","last_became_active":"2024-09-10T19:13:54.709846+0000","last_became_peered":"2024-09-10T19:13:54.709846+0000","last_unstale":"2024-09-10T19:17:08.868427+0000","last_undegraded":"2024-09-10T19:17:08.868427+0000","last_fullsized":"2024-09-10T19:17:08.868427+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"117'10","last_scrub_stamp":"2024-09-10T19:15:35.973095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:15:35.973095+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0076338860000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,0],"acting":[3,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":144,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-09-10T19:17:07.857387+0000","last_change":"2024-09-10T19:16:24.858926+0000","last_active":"2024-09-10T19:17:07.857387+0000","last_peered":"2024-09-10T19:17:07.857387+0000","last_clean":"2024-09-10T19:17:07.857387+0000","last_became_active":"2024-09-10T19:16:24.858619+0000","last_became_peered":"2024-09-10T19:16:24.858619+0000","last_unstale":"2024-09-10T19:17:07.857387+0000","last_undegraded":"2024-09-10T19:17:07.857387+0000","last_fullsized":"2024-09-10T19:17:07.857387+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:13:15.350720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:11:41.073833+0000","last_clean_scrub_stamp":"2024-09-10T19:13:15.350720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0014361000000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.c","version":"171'4642","reported_seq":4671,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-09-10T19:17:09.998418+0000","last_change":"2024-09-10T19:16:13.079853+0000","last_active":"2024-09-10T19:17:09.998418+0000","last_peered":"2024-09-10T19:17:09.998418+0000","last_clean":"2024-09-10T19:17:09.998418+0000","last_became_active":"2024-09-10T19:16:13.079527+0000","last_became_peered":"2024-09-10T19:16:13.079527+0000","last_unstale":"2024-09-10T19:17:09.998418+0000","last_undegraded":"2024-09-10T19:17:09.998418+0000","last_fullsized":"2024-09-10T19:17:09.998418+0000","mapping_epoch":148,"log_start":"156'400","ondisk_log_start":"156'400","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:16:11.777014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:16:11.777014+0000","last_clean_scrub_stamp":"2024-09-10T19:16:11.777014+0000","objects_scrubbed":0,"log_size":4242,"log_dups_size":0,"ondisk_log_size":4242,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:17:16.133191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":304218112,"num_objects":4642,"num_object_clones":0,"num_object_copies":13926,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4642,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4642,"num_write_kb":297088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.0","version":"66'18","reported_seq":171,"reported_epoch":170,"state":"active+clean","last_fresh":"2024-09-10T19:17:08.868509+0000","last_change":"2024-09-10T19:12:49.699111+0000","last_active":"2024-09-10T19:17:08.868509+0000","last_peered":"2024-09-10T19:17:08.868509+0000","last_clean":"2024-09-10T19:17:08.868509+0000","last_became_active":"2024-09-10T19:10:09.821836+0000","last_became_peered":"2024-09-10T19:10:09.821836+0000","last_unstale":"2024-09-10T19:17:08.868509+0000","last_undegraded":"2024-09-10T19:17:08.868509+0000","last_fullsized":"2024-09-10T19:17:08.868509+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'18","last_scrub_stamp":"2024-09-10T19:12:49.699054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:12:49.699054+0000","objects_scrubbed":8,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0073070890000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":100,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-09-10T19:17:07.855109+0000","last_change":"2024-09-10T19:15:42.739879+0000","last_active":"2024-09-10T19:17:07.855109+0000","last_peered":"2024-09-10T19:17:07.855109+0000","last_clean":"2024-09-10T19:17:07.855109+0000","last_became_active":"2024-09-10T19:13:54.710968+0000","last_became_peered":"2024-09-10T19:13:54.710968+0000","last_unstale":"2024-09-10T19:17:07.855109+0000","last_undegraded":"2024-09-10T19:17:07.855109+0000","last_fullsized":"2024-09-10T19:17:07.855109+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:15:42.739801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:15:42.739801+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.008487784,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,3],"acting":[6,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":145,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-09-10T19:17:07.845660+0000","last_change":"2024-09-10T19:16:30.991715+0000","last_active":"2024-09-10T19:17:07.845660+0000","last_peered":"2024-09-10T19:17:07.845660+0000","last_clean":"2024-09-10T19:17:07.845660+0000","last_became_active":"2024-09-10T19:10:11.698054+0000","last_became_peered":"2024-09-10T19:10:11.698054+0000","last_unstale":"2024-09-10T19:17:07.845660+0000","last_undegraded":"2024-09-10T19:17:07.845660+0000","last_fullsized":"2024-09-10T19:17:07.845660+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:16:30.991635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:16:30.991635+0000","last_clean_scrub_stamp":"2024-09-10T19:16:30.991635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:17:56.300591+0000","scrub_duration":0.0021810029999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":153,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-09-10T19:17:07.848035+0000","last_change":"2024-09-10T19:16:30.036748+0000","last_active":"2024-09-10T19:17:07.848035+0000","last_peered":"2024-09-10T19:17:07.848035+0000","last_clean":"2024-09-10T19:17:07.848035+0000","last_became_active":"2024-09-10T19:16:24.848109+0000","last_became_peered":"2024-09-10T19:16:24.848109+0000","last_unstale":"2024-09-10T19:17:07.848035+0000","last_undegraded":"2024-09-10T19:17:07.848035+0000","last_fullsized":"2024-09-10T19:17:07.848035+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:16:30.036662+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:16:30.036662+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:17:40.699772+0000","scrub_duration":0.0044984129999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.f","version":"171'4526","reported_seq":4567,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-09-10T19:17:09.951540+0000","last_change":"2024-09-10T19:16:24.856828+0000","last_active":"2024-09-10T19:17:09.951540+0000","last_peered":"2024-09-10T19:17:09.951540+0000","last_clean":"2024-09-10T19:17:09.951540+0000","last_became_active":"2024-09-10T19:16:24.855393+0000","last_became_peered":"2024-09-10T19:16:24.855393+0000","last_unstale":"2024-09-10T19:17:09.951540+0000","last_undegraded":"2024-09-10T19:17:09.951540+0000","last_fullsized":"2024-09-10T19:17:09.951540+0000","mapping_epoch":154,"log_start":"156'600","ondisk_log_start":"156'600","created":148,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:16:11.777014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:16:11.777014+0000","last_clean_scrub_stamp":"2024-09-10T19:16:11.777014+0000","objects_scrubbed":0,"log_size":3926,"log_dups_size":0,"ondisk_log_size":3926,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:17:20.171361+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":296615936,"num_objects":4526,"num_object_clones":0,"num_object_copies":13578,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4526,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4526,"num_write_kb":289664,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.3","version":"104'27","reported_seq":192,"reported_epoch":170,"state":"active+clean","last_fresh":"2024-09-10T19:17:08.863502+0000","last_change":"2024-09-10T19:16:35.246566+0000","last_active":"2024-09-10T19:17:08.863502+0000","last_peered":"2024-09-10T19:17:08.863502+0000","last_clean":"2024-09-10T19:17:08.863502+0000","last_became_active":"2024-09-10T19:10:09.823003+0000","last_became_peered":"2024-09-10T19:10:09.823003+0000","last_unstale":"2024-09-10T19:17:08.863502+0000","last_undegraded":"2024-09-10T19:17:08.863502+0000","last_fullsized":"2024-09-10T19:17:08.863502+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"104'27","last_scrub_stamp":"2024-09-10T19:16:35.246512+0000","last_deep_scrub":"104'27","last_deep_scrub_stamp":"2024-09-10T19:13:02.225944+0000","last_clean_scrub_stamp":"2024-09-10T19:16:35.246512+0000","objects_scrubbed":8,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:17:49.596002+0000","scrub_duration":0.0052296490000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":150,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-09-10T19:17:07.855230+0000","last_change":"2024-09-10T19:16:37.780086+0000","last_active":"2024-09-10T19:17:07.855230+0000","last_peered":"2024-09-10T19:17:07.855230+0000","last_clean":"2024-09-10T19:17:07.855230+0000","last_became_active":"2024-09-10T19:10:07.690325+0000","last_became_peered":"2024-09-10T19:10:07.690325+0000","last_unstale":"2024-09-10T19:17:07.855230+0000","last_undegraded":"2024-09-10T19:17:07.855230+0000","last_fullsized":"2024-09-10T19:17:07.855230+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:16:37.779967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:16:37.779967+0000","last_clean_scrub_stamp":"2024-09-10T19:16:37.779967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:17:50.595527+0000","scrub_duration":0.067110102000000005,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":146,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-09-10T19:17:07.854714+0000","last_change":"2024-09-10T19:15:50.669894+0000","last_active":"2024-09-10T19:17:07.854714+0000","last_peered":"2024-09-10T19:17:07.854714+0000","last_clean":"2024-09-10T19:17:07.854714+0000","last_became_active":"2024-09-10T19:10:11.700370+0000","last_became_peered":"2024-09-10T19:10:11.700370+0000","last_unstale":"2024-09-10T19:17:07.854714+0000","last_undegraded":"2024-09-10T19:17:07.854714+0000","last_fullsized":"2024-09-10T19:17:07.854714+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:15:50.669821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:15:50.669821+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0015765,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":110,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-09-10T19:17:07.854565+0000","last_change":"2024-09-10T19:16:24.844993+0000","last_active":"2024-09-10T19:17:07.854565+0000","last_peered":"2024-09-10T19:17:07.854565+0000","last_clean":"2024-09-10T19:17:07.854565+0000","last_became_active":"2024-09-10T19:16:24.844699+0000","last_became_peered":"2024-09-10T19:16:24.844699+0000","last_unstale":"2024-09-10T19:17:07.854565+0000","last_undegraded":"2024-09-10T19:17:07.854565+0000","last_fullsized":"2024-09-10T19:17:07.854565+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:15:41.749338+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:15:41.749338+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0018841260000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.e","version":"171'4550","reported_seq":4590,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-09-10T19:17:09.984676+0000","last_change":"2024-09-10T19:16:24.837044+0000","last_active":"2024-09-10T19:17:09.984676+0000","last_peered":"2024-09-10T19:17:09.984676+0000","last_clean":"2024-09-10T19:17:09.984676+0000","last_became_active":"2024-09-10T19:16:24.836458+0000","last_became_peered":"2024-09-10T19:16:24.836458+0000","last_unstale":"2024-09-10T19:17:09.984676+0000","last_undegraded":"2024-09-10T19:17:09.984676+0000","last_fullsized":"2024-09-10T19:17:09.984676+0000","mapping_epoch":154,"log_start":"156'300","ondisk_log_start":"156'300","created":148,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:16:11.777014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:16:11.777014+0000","last_clean_scrub_stamp":"2024-09-10T19:16:11.777014+0000","objects_scrubbed":0,"log_size":4250,"log_dups_size":0,"ondisk_log_size":4250,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:17:30.166366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":298188800,"num_objects":4550,"num_object_clones":0,"num_object_copies":13650,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4550,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4550,"num_write_kb":291200,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"66'10","reported_seq":106,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-09-10T19:17:07.850724+0000","last_change":"2024-09-10T19:13:54.748881+0000","last_active":"2024-09-10T19:17:07.850724+0000","last_peered":"2024-09-10T19:17:07.850724+0000","last_clean":"2024-09-10T19:17:07.850724+0000","last_became_active":"2024-09-10T19:13:54.714542+0000","last_became_peered":"2024-09-10T19:13:54.714542+0000","last_unstale":"2024-09-10T19:17:07.850724+0000","last_undegraded":"2024-09-10T19:17:07.850724+0000","last_fullsized":"2024-09-10T19:17:07.850724+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'10","last_scrub_stamp":"2024-09-10T19:12:56.360409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:12:56.360409+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":14,"num_read_kb":9,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,0],"acting":[5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":110,"reported_epoch":170,"state":"active+clean","last_fresh":"2024-09-10T19:17:08.864246+0000","last_change":"2024-09-10T19:16:24.837191+0000","last_active":"2024-09-10T19:17:08.864246+0000","last_peered":"2024-09-10T19:17:08.864246+0000","last_clean":"2024-09-10T19:17:08.864246+0000","last_became_active":"2024-09-10T19:16:24.836898+0000","last_became_peered":"2024-09-10T19:16:24.836898+0000","last_unstale":"2024-09-10T19:17:08.864246+0000","last_undegraded":"2024-09-10T19:17:08.864246+0000","last_fullsized":"2024-09-10T19:17:08.864246+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:15:41.870422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:15:41.870422+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:17:10.297611+0000","scrub_duration":0.0061607199999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3","version":"84'9","reported_seq":229,"reported_epoch":170,"state":"active+clean","last_fresh":"2024-09-10T19:17:08.863367+0000","last_change":"2024-09-10T19:13:45.236207+0000","last_active":"2024-09-10T19:17:08.863367+0000","last_peered":"2024-09-10T19:17:08.863367+0000","last_clean":"2024-09-10T19:17:08.863367+0000","last_became_active":"2024-09-10T19:10:11.699690+0000","last_became_peered":"2024-09-10T19:10:11.699690+0000","last_unstale":"2024-09-10T19:17:08.863367+0000","last_undegraded":"2024-09-10T19:17:08.863367+0000","last_fullsized":"2024-09-10T19:17:08.863367+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'9","last_scrub_stamp":"2024-09-10T19:13:45.236094+0000","last_deep_scrub":"84'9","last_deep_scrub_stamp":"2024-09-10T19:11:53.879272+0000","last_clean_scrub_stamp":"2024-09-10T19:13:45.236094+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.036165602999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":144,"reported_epoch":170,"state":"active+clean","last_fresh":"2024-09-10T19:17:08.862361+0000","last_change":"2024-09-10T19:16:24.840367+0000","last_active":"2024-09-10T19:17:08.862361+0000","last_peered":"2024-09-10T19:17:08.862361+0000","last_clean":"2024-09-10T19:17:08.862361+0000","last_became_active":"2024-09-10T19:16:24.840096+0000","last_became_peered":"2024-09-10T19:16:24.840096+0000","last_unstale":"2024-09-10T19:17:08.862361+0000","last_undegraded":"2024-09-10T19:17:08.862361+0000","last_fullsized":"2024-09-10T19:17:08.862361+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:13:40.234289+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:13:40.234289+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0086405590000000008,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,5],"acting":[0,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.8","version":"170'4437","reported_seq":4467,"reported_epoch":170,"state":"active+clean","last_fresh":"2024-09-10T19:17:09.803674+0000","last_change":"2024-09-10T19:16:13.081213+0000","last_active":"2024-09-10T19:17:09.803674+0000","last_peered":"2024-09-10T19:17:09.803674+0000","last_clean":"2024-09-10T19:17:09.803674+0000","last_became_active":"2024-09-10T19:16:13.080496+0000","last_became_peered":"2024-09-10T19:16:13.080496+0000","last_unstale":"2024-09-10T19:17:09.803674+0000","last_undegraded":"2024-09-10T19:17:09.803674+0000","last_fullsized":"2024-09-10T19:17:09.803674+0000","mapping_epoch":148,"log_start":"156'200","ondisk_log_start":"156'200","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:16:11.777014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:16:11.777014+0000","last_clean_scrub_stamp":"2024-09-10T19:16:11.777014+0000","objects_scrubbed":0,"log_size":4237,"log_dups_size":0,"ondisk_log_size":4237,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:17:12.464052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":290783232,"num_objects":4437,"num_object_clones":0,"num_object_copies":13311,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4437,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4437,"num_write_kb":283968,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"155'31","reported_seq":121,"reported_epoch":170,"state":"active+clean","last_fresh":"2024-09-10T19:17:08.869358+0000","last_change":"2024-09-10T19:16:24.910865+0000","last_active":"2024-09-10T19:17:08.869358+0000","last_peered":"2024-09-10T19:17:08.869358+0000","last_clean":"2024-09-10T19:17:08.869358+0000","last_became_active":"2024-09-10T19:16:24.858154+0000","last_became_peered":"2024-09-10T19:16:24.858154+0000","last_unstale":"2024-09-10T19:17:08.869358+0000","last_undegraded":"2024-09-10T19:17:08.869358+0000","last_fullsized":"2024-09-10T19:17:08.869358+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"117'30","last_scrub_stamp":"2024-09-10T19:15:49.837844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:15:49.837844+0000","objects_scrubbed":0,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":47,"num_read_kb":32,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":358,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":157,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-09-10T19:17:07.851045+0000","last_change":"2024-09-10T19:15:54.746979+0000","last_active":"2024-09-10T19:17:07.851045+0000","last_peered":"2024-09-10T19:17:07.851045+0000","last_clean":"2024-09-10T19:17:07.851045+0000","last_became_active":"2024-09-10T19:13:54.730096+0000","last_became_peered":"2024-09-10T19:13:54.730096+0000","last_unstale":"2024-09-10T19:17:07.851045+0000","last_undegraded":"2024-09-10T19:17:07.851045+0000","last_fullsized":"2024-09-10T19:17:07.851045+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:15:54.746920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:15:54.746920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001300037,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":153,"reported_epoch":170,"state":"active+clean","last_fresh":"2024-09-10T19:17:08.863921+0000","last_change":"2024-09-10T19:15:34.802331+0000","last_active":"2024-09-10T19:17:08.863921+0000","last_peered":"2024-09-10T19:17:08.863921+0000","last_clean":"2024-09-10T19:17:08.863921+0000","last_became_active":"2024-09-10T19:13:54.714017+0000","last_became_peered":"2024-09-10T19:13:54.714017+0000","last_unstale":"2024-09-10T19:17:08.863921+0000","last_undegraded":"2024-09-10T19:17:08.863921+0000","last_fullsized":"2024-09-10T19:17:08.863921+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:15:34.802121+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:15:34.802121+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0042778080000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,6],"acting":[0,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":135,"reported_epoch":170,"state":"active+clean","last_fresh":"2024-09-10T19:17:08.869469+0000","last_change":"2024-09-10T19:16:32.818668+0000","last_active":"2024-09-10T19:17:08.869469+0000","last_peered":"2024-09-10T19:17:08.869469+0000","last_clean":"2024-09-10T19:17:08.869469+0000","last_became_active":"2024-09-10T19:10:13.721744+0000","last_became_peered":"2024-09-10T19:10:13.721744+0000","last_unstale":"2024-09-10T19:17:08.869469+0000","last_undegraded":"2024-09-10T19:17:08.869469+0000","last_fullsized":"2024-09-10T19:17:08.869469+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:16:32.818590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:16:32.818590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:17:49.356780+0000","scrub_duration":0.0047149059999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.b","version":"171'4699","reported_seq":4740,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-09-10T19:17:09.943844+0000","last_change":"2024-09-10T19:16:24.852409+0000","last_active":"2024-09-10T19:17:09.943844+0000","last_peered":"2024-09-10T19:17:09.943844+0000","last_clean":"2024-09-10T19:17:09.943844+0000","last_became_active":"2024-09-10T19:16:24.850682+0000","last_became_peered":"2024-09-10T19:16:24.850682+0000","last_unstale":"2024-09-10T19:17:09.943844+0000","last_undegraded":"2024-09-10T19:17:09.943844+0000","last_fullsized":"2024-09-10T19:17:09.943844+0000","mapping_epoch":154,"log_start":"157'800","ondisk_log_start":"157'800","created":148,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:16:11.777014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:16:11.777014+0000","last_clean_scrub_stamp":"2024-09-10T19:16:11.777014+0000","objects_scrubbed":0,"log_size":3899,"log_dups_size":0,"ondisk_log_size":3899,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:17:26.367282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":307953664,"num_objects":4699,"num_object_clones":0,"num_object_copies":14097,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4699,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4699,"num_write_kb":300736,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.7","version":"82'18","reported_seq":173,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-09-10T19:17:07.850886+0000","last_change":"2024-09-10T19:15:55.777428+0000","last_active":"2024-09-10T19:17:07.850886+0000","last_peered":"2024-09-10T19:17:07.850886+0000","last_clean":"2024-09-10T19:17:07.850886+0000","last_became_active":"2024-09-10T19:13:54.712065+0000","last_became_peered":"2024-09-10T19:13:54.712065+0000","last_unstale":"2024-09-10T19:17:07.850886+0000","last_undegraded":"2024-09-10T19:17:07.850886+0000","last_fullsized":"2024-09-10T19:17:07.850886+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'18","last_scrub_stamp":"2024-09-10T19:15:55.777361+0000","last_deep_scrub":"82'18","last_deep_scrub_stamp":"2024-09-10T19:15:55.777361+0000","last_clean_scrub_stamp":"2024-09-10T19:15:55.777361+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:17:17.118895+0000","scrub_duration":0.0023560040000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":19,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.0","version":"159'6","reported_seq":116,"reported_epoch":170,"state":"active+clean","last_fresh":"2024-09-10T19:17:08.869201+0000","last_change":"2024-09-10T19:16:32.650007+0000","last_active":"2024-09-10T19:17:08.869201+0000","last_peered":"2024-09-10T19:17:08.869201+0000","last_clean":"2024-09-10T19:17:08.869201+0000","last_became_active":"2024-09-10T19:16:24.846203+0000","last_became_peered":"2024-09-10T19:16:24.846203+0000","last_unstale":"2024-09-10T19:17:08.869201+0000","last_undegraded":"2024-09-10T19:17:08.869201+0000","last_fullsized":"2024-09-10T19:17:08.869201+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"159'6","last_scrub_stamp":"2024-09-10T19:16:32.649914+0000","last_deep_scrub":"159'6","last_deep_scrub_stamp":"2024-09-10T19:16:32.649914+0000","last_clean_scrub_stamp":"2024-09-10T19:16:32.649914+0000","objects_scrubbed":1,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:17:33.710001+0000","scrub_duration":0.013939817,"objects_trimmed":0,"snaptrim_duration":0.00034454499999999997,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":51,"num_read_kb":46,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":389,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":39,"reported_epoch":170,"state":"active+clean","last_fresh":"2024-09-10T19:17:08.869088+0000","last_change":"2024-09-10T19:16:33.811962+0000","last_active":"2024-09-10T19:17:08.869088+0000","last_peered":"2024-09-10T19:17:08.869088+0000","last_clean":"2024-09-10T19:17:08.869088+0000","last_became_active":"2024-09-10T19:16:24.857505+0000","last_became_peered":"2024-09-10T19:16:24.857505+0000","last_unstale":"2024-09-10T19:17:08.869088+0000","last_undegraded":"2024-09-10T19:17:08.869088+0000","last_fullsized":"2024-09-10T19:17:08.869088+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:16:33.811869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:16:33.811869+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:17:45.100668+0000","scrub_duration":0.0027920340000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":38,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-09-10T19:17:07.848695+0000","last_change":"2024-09-10T19:16:43.958311+0000","last_active":"2024-09-10T19:17:07.848695+0000","last_peered":"2024-09-10T19:17:07.848695+0000","last_clean":"2024-09-10T19:17:07.848695+0000","last_became_active":"2024-09-10T19:16:24.861481+0000","last_became_peered":"2024-09-10T19:16:24.861481+0000","last_unstale":"2024-09-10T19:17:07.848695+0000","last_undegraded":"2024-09-10T19:17:07.848695+0000","last_fullsized":"2024-09-10T19:17:07.848695+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:16:43.958224+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:16:43.958224+0000","last_clean_scrub_stamp":"2024-09-10T19:16:43.958224+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:18:09.697349+0000","scrub_duration":0.015970275999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,4],"acting":[5,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":137,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-09-10T19:17:07.854899+0000","last_change":"2024-09-10T19:15:38.493179+0000","last_active":"2024-09-10T19:17:07.854899+0000","last_peered":"2024-09-10T19:17:07.854899+0000","last_clean":"2024-09-10T19:17:07.854899+0000","last_became_active":"2024-09-10T19:10:14.053697+0000","last_became_peered":"2024-09-10T19:10:14.053697+0000","last_unstale":"2024-09-10T19:17:07.854899+0000","last_undegraded":"2024-09-10T19:17:07.854899+0000","last_fullsized":"2024-09-10T19:17:07.854899+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:15:38.493102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:15:38.493102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0015544280000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.a","version":"170'4648","reported_seq":4676,"reported_epoch":170,"state":"active+clean","last_fresh":"2024-09-10T19:17:09.824321+0000","last_change":"2024-09-10T19:16:24.865995+0000","last_active":"2024-09-10T19:17:09.824321+0000","last_peered":"2024-09-10T19:17:09.824321+0000","last_clean":"2024-09-10T19:17:09.824321+0000","last_became_active":"2024-09-10T19:16:24.864959+0000","last_became_peered":"2024-09-10T19:16:24.864959+0000","last_unstale":"2024-09-10T19:17:09.824321+0000","last_undegraded":"2024-09-10T19:17:09.824321+0000","last_fullsized":"2024-09-10T19:17:09.824321+0000","mapping_epoch":154,"log_start":"156'400","ondisk_log_start":"156'400","created":148,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:16:11.777014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:16:11.777014+0000","last_clean_scrub_stamp":"2024-09-10T19:16:11.777014+0000","objects_scrubbed":0,"log_size":4248,"log_dups_size":0,"ondisk_log_size":4248,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:17:20.676451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":304611328,"num_objects":4648,"num_object_clones":0,"num_object_copies":13944,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4648,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4648,"num_write_kb":297472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.6","version":"66'12","reported_seq":211,"reported_epoch":170,"state":"active+clean","last_fresh":"2024-09-10T19:17:08.864712+0000","last_change":"2024-09-10T19:16:24.878036+0000","last_active":"2024-09-10T19:17:08.864712+0000","last_peered":"2024-09-10T19:17:08.864712+0000","last_clean":"2024-09-10T19:17:08.864712+0000","last_became_active":"2024-09-10T19:16:24.844323+0000","last_became_peered":"2024-09-10T19:16:24.844323+0000","last_unstale":"2024-09-10T19:17:08.864712+0000","last_undegraded":"2024-09-10T19:17:08.864712+0000","last_fullsized":"2024-09-10T19:17:08.864712+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'12","last_scrub_stamp":"2024-09-10T19:15:40.903279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:15:40.903279+0000","objects_scrubbed":6,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0271238,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,5],"acting":[0,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"61'1","reported_seq":156,"reported_epoch":170,"state":"active+clean","last_fresh":"2024-09-10T19:17:08.862877+0000","last_change":"2024-09-10T19:16:42.195540+0000","last_active":"2024-09-10T19:17:08.862877+0000","last_peered":"2024-09-10T19:17:08.862877+0000","last_clean":"2024-09-10T19:17:08.862877+0000","last_became_active":"2024-09-10T19:10:07.683259+0000","last_became_peered":"2024-09-10T19:10:07.683259+0000","last_unstale":"2024-09-10T19:17:08.862877+0000","last_undegraded":"2024-09-10T19:17:08.862877+0000","last_fullsized":"2024-09-10T19:17:08.862877+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'1","last_scrub_stamp":"2024-09-10T19:16:42.195463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:16:42.195463+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:17:50.284803+0000","scrub_duration":0.0016622189999999999,"objects_trimmed":0,"snaptrim_duration":0.0014341130000000001,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":157,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-09-10T19:17:07.853840+0000","last_change":"2024-09-10T19:15:30.542477+0000","last_active":"2024-09-10T19:17:07.853840+0000","last_peered":"2024-09-10T19:17:07.853840+0000","last_clean":"2024-09-10T19:17:07.853840+0000","last_became_active":"2024-09-10T19:13:54.721486+0000","last_became_peered":"2024-09-10T19:13:54.721486+0000","last_unstale":"2024-09-10T19:17:07.853840+0000","last_undegraded":"2024-09-10T19:17:07.853840+0000","last_fullsized":"2024-09-10T19:17:07.853840+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:15:30.542399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:15:30.542399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0025142609999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":153,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-09-10T19:17:07.850817+0000","last_change":"2024-09-10T19:13:54.712428+0000","last_active":"2024-09-10T19:17:07.850817+0000","last_peered":"2024-09-10T19:17:07.850817+0000","last_clean":"2024-09-10T19:17:07.850817+0000","last_became_active":"2024-09-10T19:13:54.711685+0000","last_became_peered":"2024-09-10T19:13:54.711685+0000","last_unstale":"2024-09-10T19:17:07.850817+0000","last_undegraded":"2024-09-10T19:17:07.850817+0000","last_fullsized":"2024-09-10T19:17:07.850817+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:12:44.527561+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:12:44.527561+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.022638367999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,4],"acting":[5,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":100,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-09-10T19:17:07.849063+0000","last_change":"2024-09-10T19:16:28.786490+0000","last_active":"2024-09-10T19:17:07.849063+0000","last_peered":"2024-09-10T19:17:07.849063+0000","last_clean":"2024-09-10T19:17:07.849063+0000","last_became_active":"2024-09-10T19:13:54.722937+0000","last_became_peered":"2024-09-10T19:13:54.722937+0000","last_unstale":"2024-09-10T19:17:07.849063+0000","last_undegraded":"2024-09-10T19:17:07.849063+0000","last_fullsized":"2024-09-10T19:17:07.849063+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:16:28.786056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:16:28.786056+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:17:51.730597+0000","scrub_duration":0.0082035739999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.9","version":"170'4599","reported_seq":4629,"reported_epoch":170,"state":"active+clean","last_fresh":"2024-09-10T19:17:09.729081+0000","last_change":"2024-09-10T19:16:12.797816+0000","last_active":"2024-09-10T19:17:09.729081+0000","last_peered":"2024-09-10T19:17:09.729081+0000","last_clean":"2024-09-10T19:17:09.729081+0000","last_became_active":"2024-09-10T19:16:12.797479+0000","last_became_peered":"2024-09-10T19:16:12.797479+0000","last_unstale":"2024-09-10T19:17:09.729081+0000","last_undegraded":"2024-09-10T19:17:09.729081+0000","last_fullsized":"2024-09-10T19:17:09.729081+0000","mapping_epoch":148,"log_start":"156'500","ondisk_log_start":"156'500","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:16:11.777014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:16:11.777014+0000","last_clean_scrub_stamp":"2024-09-10T19:16:11.777014+0000","objects_scrubbed":0,"log_size":4099,"log_dups_size":0,"ondisk_log_size":4099,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:17:11.992307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":301400064,"num_objects":4599,"num_object_clones":0,"num_object_copies":13797,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4599,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4599,"num_write_kb":294336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,0],"acting":[6,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"80'21","reported_seq":177,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-09-10T19:17:07.855991+0000","last_change":"2024-09-10T19:15:49.682164+0000","last_active":"2024-09-10T19:17:07.855991+0000","last_peered":"2024-09-10T19:17:07.855991+0000","last_clean":"2024-09-10T19:17:07.855991+0000","last_became_active":"2024-09-10T19:10:09.822184+0000","last_became_peered":"2024-09-10T19:10:09.822184+0000","last_unstale":"2024-09-10T19:17:07.855991+0000","last_undegraded":"2024-09-10T19:17:07.855991+0000","last_fullsized":"2024-09-10T19:17:07.855991+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"80'21","last_scrub_stamp":"2024-09-10T19:15:49.682092+0000","last_deep_scrub":"80'21","last_deep_scrub_stamp":"2024-09-10T19:11:39.836144+0000","last_clean_scrub_stamp":"2024-09-10T19:15:49.682092+0000","objects_scrubbed":7,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0022142529999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"63'2","reported_seq":158,"reported_epoch":170,"state":"active+clean","last_fresh":"2024-09-10T19:17:08.868540+0000","last_change":"2024-09-10T19:16:35.962952+0000","last_active":"2024-09-10T19:17:08.868540+0000","last_peered":"2024-09-10T19:17:08.868540+0000","last_clean":"2024-09-10T19:17:08.868540+0000","last_became_active":"2024-09-10T19:10:07.684784+0000","last_became_peered":"2024-09-10T19:10:07.684784+0000","last_unstale":"2024-09-10T19:17:08.868540+0000","last_undegraded":"2024-09-10T19:17:08.868540+0000","last_fullsized":"2024-09-10T19:17:08.868540+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'2","last_scrub_stamp":"2024-09-10T19:16:35.962804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:16:35.962804+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:18:04.954385+0000","scrub_duration":0.105395924,"objects_trimmed":0,"snaptrim_duration":0.00075522999999999996,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":160,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-09-10T19:17:07.849677+0000","last_change":"2024-09-10T19:16:24.858963+0000","last_active":"2024-09-10T19:17:07.849677+0000","last_peered":"2024-09-10T19:17:07.849677+0000","last_clean":"2024-09-10T19:17:07.849677+0000","last_became_active":"2024-09-10T19:16:24.858662+0000","last_became_peered":"2024-09-10T19:16:24.858662+0000","last_unstale":"2024-09-10T19:17:07.849677+0000","last_undegraded":"2024-09-10T19:17:07.849677+0000","last_fullsized":"2024-09-10T19:17:07.849677+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:13:50.411500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:13:50.411500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0032604869999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"74'79","reported_seq":185,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-09-10T19:17:07.853986+0000","last_change":"2024-09-10T19:13:46.419002+0000","last_active":"2024-09-10T19:17:07.853986+0000","last_peered":"2024-09-10T19:17:07.853986+0000","last_clean":"2024-09-10T19:17:07.853986+0000","last_became_active":"2024-09-10T19:09:56.634919+0000","last_became_peered":"2024-09-10T19:09:56.634919+0000","last_unstale":"2024-09-10T19:17:07.853986+0000","last_undegraded":"2024-09-10T19:17:07.853986+0000","last_fullsized":"2024-09-10T19:17:07.853986+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'79","last_scrub_stamp":"2024-09-10T19:13:46.418921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:08:00.699315+0000","last_clean_scrub_stamp":"2024-09-10T19:13:46.418921+0000","objects_scrubbed":2,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0029827999999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":146,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-09-10T19:17:07.855516+0000","last_change":"2024-09-10T19:16:24.852723+0000","last_active":"2024-09-10T19:17:07.855516+0000","last_peered":"2024-09-10T19:17:07.855516+0000","last_clean":"2024-09-10T19:17:07.855516+0000","last_became_active":"2024-09-10T19:16:24.851652+0000","last_became_peered":"2024-09-10T19:16:24.851652+0000","last_unstale":"2024-09-10T19:17:07.855516+0000","last_undegraded":"2024-09-10T19:17:07.855516+0000","last_fullsized":"2024-09-10T19:17:07.855516+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:13:47.453228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:13:47.453228+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.010282463,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,0],"acting":[7,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":134,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-09-10T19:17:07.850949+0000","last_change":"2024-09-10T19:13:05.390398+0000","last_active":"2024-09-10T19:17:07.850949+0000","last_peered":"2024-09-10T19:17:07.850949+0000","last_clean":"2024-09-10T19:17:07.850949+0000","last_became_active":"2024-09-10T19:10:13.720647+0000","last_became_peered":"2024-09-10T19:10:13.720647+0000","last_unstale":"2024-09-10T19:17:07.850949+0000","last_undegraded":"2024-09-10T19:17:07.850949+0000","last_fullsized":"2024-09-10T19:17:07.850949+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:13:05.390317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:13:05.390317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0058030640000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.6","version":"171'4548","reported_seq":4578,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-09-10T19:17:09.998268+0000","last_change":"2024-09-10T19:16:13.080600+0000","last_active":"2024-09-10T19:17:09.998268+0000","last_peered":"2024-09-10T19:17:09.998268+0000","last_clean":"2024-09-10T19:17:09.998268+0000","last_became_active":"2024-09-10T19:16:13.080292+0000","last_became_peered":"2024-09-10T19:16:13.080292+0000","last_unstale":"2024-09-10T19:17:09.998268+0000","last_undegraded":"2024-09-10T19:17:09.998268+0000","last_fullsized":"2024-09-10T19:17:09.998268+0000","mapping_epoch":148,"log_start":"156'300","ondisk_log_start":"156'300","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:16:11.777014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:16:11.777014+0000","last_clean_scrub_stamp":"2024-09-10T19:16:11.777014+0000","objects_scrubbed":0,"log_size":4248,"log_dups_size":0,"ondisk_log_size":4248,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:17:24.544000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":298057728,"num_objects":4548,"num_object_clones":0,"num_object_copies":13644,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4548,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4548,"num_write_kb":291072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,7],"acting":[0,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"66'19","reported_seq":206,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-09-10T19:17:07.855309+0000","last_change":"2024-09-10T19:15:38.722679+0000","last_active":"2024-09-10T19:17:07.855309+0000","last_peered":"2024-09-10T19:17:07.855309+0000","last_clean":"2024-09-10T19:17:07.855309+0000","last_became_active":"2024-09-10T19:13:54.731274+0000","last_became_peered":"2024-09-10T19:13:54.731274+0000","last_unstale":"2024-09-10T19:17:07.855309+0000","last_undegraded":"2024-09-10T19:17:07.855309+0000","last_fullsized":"2024-09-10T19:17:07.855309+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'19","last_scrub_stamp":"2024-09-10T19:15:38.722581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:15:38.722581+0000","objects_scrubbed":9,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0054904940000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,5],"acting":[6,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":28,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-09-10T19:17:07.849226+0000","last_change":"2024-09-10T19:16:24.847051+0000","last_active":"2024-09-10T19:17:07.849226+0000","last_peered":"2024-09-10T19:17:07.849226+0000","last_clean":"2024-09-10T19:17:07.849226+0000","last_became_active":"2024-09-10T19:16:24.846745+0000","last_became_peered":"2024-09-10T19:16:24.846745+0000","last_unstale":"2024-09-10T19:17:07.849226+0000","last_undegraded":"2024-09-10T19:17:07.849226+0000","last_fullsized":"2024-09-10T19:17:07.849226+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:15:53.803057+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:15:53.803057+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3],"acting":[5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":38,"reported_epoch":170,"state":"active+clean","last_fresh":"2024-09-10T19:17:08.863250+0000","last_change":"2024-09-10T19:16:27.240222+0000","last_active":"2024-09-10T19:17:08.863250+0000","last_peered":"2024-09-10T19:17:08.863250+0000","last_clean":"2024-09-10T19:17:08.863250+0000","last_became_active":"2024-09-10T19:16:24.844295+0000","last_became_peered":"2024-09-10T19:16:24.844295+0000","last_unstale":"2024-09-10T19:17:08.863250+0000","last_undegraded":"2024-09-10T19:17:08.863250+0000","last_fullsized":"2024-09-10T19:17:08.863250+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:16:27.240139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:16:27.240139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:17:44.435081+0000","scrub_duration":0.0020507199999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":144,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-09-10T19:17:07.853910+0000","last_change":"2024-09-10T19:16:24.858456+0000","last_active":"2024-09-10T19:17:07.853910+0000","last_peered":"2024-09-10T19:17:07.853910+0000","last_clean":"2024-09-10T19:17:07.853910+0000","last_became_active":"2024-09-10T19:16:24.858172+0000","last_became_peered":"2024-09-10T19:16:24.858172+0000","last_unstale":"2024-09-10T19:17:07.853910+0000","last_undegraded":"2024-09-10T19:17:07.853910+0000","last_fullsized":"2024-09-10T19:17:07.853910+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:13:14.311620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:13:14.311620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0054101059999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,6],"acting":[7,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.7","version":"171'4555","reported_seq":4596,"reported_epoch":171,"state":"active+clean","last_fresh":"2024-09-10T19:17:09.956168+0000","last_change":"2024-09-10T19:16:24.858911+0000","last_active":"2024-09-10T19:17:09.956168+0000","last_peered":"2024-09-10T19:17:09.956168+0000","last_clean":"2024-09-10T19:17:09.956168+0000","last_became_active":"2024-09-10T19:16:24.857599+0000","last_became_peered":"2024-09-10T19:16:24.857599+0000","last_unstale":"2024-09-10T19:17:09.956168+0000","last_undegraded":"2024-09-10T19:17:09.956168+0000","last_fullsized":"2024-09-10T19:17:09.956168+0000","mapping_epoch":154,"log_start":"157'700","ondisk_log_start":"157'700","created":148,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:16:11.777014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:16:11.777014+0000","last_clean_scrub_stamp":"2024-09-10T19:16:11.777014+0000","objects_scrubbed":0,"log_size":3855,"log_dups_size":0,"ondisk_log_size":3855,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:17:40.541111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":298516480,"num_objects":4555,"num_object_clones":0,"num_object_copies":13665,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4555,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4555,"num_write_kb":291520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,4],"acting":[3,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.b","version":"66'9","reported_seq":181,"reported_epoch":170,"state":"active+clean","last_fresh":"2024-09-10T19:17:08.863438+0000","last_change":"2024-09-10T19:15:37.846895+0000","last_active":"2024-09-10T19:17:08.863438+0000","last_peered":"2024-09-10T19:17:08.863438+0000","last_clean":"2024-09-10T19:17:08.863438+0000","last_became_active":"2024-09-10T19:13:54.710613+0000","last_became_peered":"2024-09-10T19:13:54.710613+0000","last_unstale":"2024-09-10T19:17:08.863438+0000","last_undegraded":"2024-09-10T19:17:08.863438+0000","last_fullsized":"2024-09-10T19:17:08.863438+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'9","last_scrub_stamp":"2024-09-10T19:15:37.846807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:15:37.846807+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.012189228,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":113,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-09-10T19:17:07.846683+0000","last_change":"2024-09-10T19:16:24.848979+0000","last_active":"2024-09-10T19:17:07.846683+0000","last_peered":"2024-09-10T19:17:07.846683+0000","last_clean":"2024-09-10T19:17:07.846683+0000","last_became_active":"2024-09-10T19:16:24.848668+0000","last_became_peered":"2024-09-10T19:16:24.848668+0000","last_unstale":"2024-09-10T19:17:07.846683+0000","last_undegraded":"2024-09-10T19:17:07.846683+0000","last_fullsized":"2024-09-10T19:17:07.846683+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:15:50.850576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:11:44.254513+0000","last_clean_scrub_stamp":"2024-09-10T19:15:50.850576+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001815419,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":28,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-09-10T19:17:07.847901+0000","last_change":"2024-09-10T19:16:24.846109+0000","last_active":"2024-09-10T19:17:07.847901+0000","last_peered":"2024-09-10T19:17:07.847901+0000","last_clean":"2024-09-10T19:17:07.847901+0000","last_became_active":"2024-09-10T19:16:24.845709+0000","last_became_peered":"2024-09-10T19:16:24.845709+0000","last_unstale":"2024-09-10T19:17:07.847901+0000","last_undegraded":"2024-09-10T19:17:07.847901+0000","last_fullsized":"2024-09-10T19:17:07.847901+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:13:08.398036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:13:08.398036+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":154,"reported_epoch":170,"state":"active+clean","last_fresh":"2024-09-10T19:17:08.862822+0000","last_change":"2024-09-10T19:16:39.188737+0000","last_active":"2024-09-10T19:17:08.862822+0000","last_peered":"2024-09-10T19:17:08.862822+0000","last_clean":"2024-09-10T19:17:08.862822+0000","last_became_active":"2024-09-10T19:16:24.837483+0000","last_became_peered":"2024-09-10T19:16:24.837483+0000","last_unstale":"2024-09-10T19:17:08.862822+0000","last_undegraded":"2024-09-10T19:17:08.862822+0000","last_fullsized":"2024-09-10T19:17:08.862822+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:16:39.188682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:16:39.188682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:17:48.097207+0000","scrub_duration":0.0012633589999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.4","version":"170'4482","reported_seq":4522,"reported_epoch":170,"state":"active+clean","last_fresh":"2024-09-10T19:17:09.817731+0000","last_change":"2024-09-10T19:16:24.863099+0000","last_active":"2024-09-10T19:17:09.817731+0000","last_peered":"2024-09-10T19:17:09.817731+0000","last_clean":"2024-09-10T19:17:09.817731+0000","last_became_active":"2024-09-10T19:16:24.862785+0000","last_became_peered":"2024-09-10T19:16:24.862785+0000","last_unstale":"2024-09-10T19:17:09.817731+0000","last_undegraded":"2024-09-10T19:17:09.817731+0000","last_fullsized":"2024-09-10T19:17:09.817731+0000","mapping_epoch":154,"log_start":"156'300","ondisk_log_start":"156'300","created":148,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:16:11.777014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:16:11.777014+0000","last_clean_scrub_stamp":"2024-09-10T19:16:11.777014+0000","objects_scrubbed":0,"log_size":4182,"log_dups_size":0,"ondisk_log_size":4182,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:17:21.039951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":293732352,"num_objects":4482,"num_object_clones":0,"num_object_copies":13446,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4482,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4482,"num_write_kb":286848,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.8","version":"66'15","reported_seq":163,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-09-10T19:17:07.850272+0000","last_change":"2024-09-10T19:12:50.449684+0000","last_active":"2024-09-10T19:17:07.850272+0000","last_peered":"2024-09-10T19:17:07.850272+0000","last_clean":"2024-09-10T19:17:07.850272+0000","last_became_active":"2024-09-10T19:10:09.820950+0000","last_became_peered":"2024-09-10T19:10:09.820950+0000","last_unstale":"2024-09-10T19:17:07.850272+0000","last_undegraded":"2024-09-10T19:17:07.850272+0000","last_fullsized":"2024-09-10T19:17:07.850272+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'15","last_scrub_stamp":"2024-09-10T19:12:50.449596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:12:50.449596+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0055729239999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":29,"reported_epoch":170,"state":"active+clean","last_fresh":"2024-09-10T19:17:08.869615+0000","last_change":"2024-09-10T19:16:24.857533+0000","last_active":"2024-09-10T19:17:08.869615+0000","last_peered":"2024-09-10T19:17:08.869615+0000","last_clean":"2024-09-10T19:17:08.869615+0000","last_became_active":"2024-09-10T19:16:24.857253+0000","last_became_peered":"2024-09-10T19:16:24.857253+0000","last_unstale":"2024-09-10T19:17:08.869615+0000","last_undegraded":"2024-09-10T19:17:08.869615+0000","last_fullsized":"2024-09-10T19:17:08.869615+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:15:38.677960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:15:38.677960+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.9","version":"80'9","reported_seq":229,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-09-10T19:17:07.853967+0000","last_change":"2024-09-10T19:13:43.358984+0000","last_active":"2024-09-10T19:17:07.853967+0000","last_peered":"2024-09-10T19:17:07.853967+0000","last_clean":"2024-09-10T19:17:07.853967+0000","last_became_active":"2024-09-10T19:10:11.702878+0000","last_became_peered":"2024-09-10T19:10:11.702878+0000","last_unstale":"2024-09-10T19:17:07.853967+0000","last_undegraded":"2024-09-10T19:17:07.853967+0000","last_fullsized":"2024-09-10T19:17:07.853967+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"80'9","last_scrub_stamp":"2024-09-10T19:13:43.358901+0000","last_deep_scrub":"80'9","last_deep_scrub_stamp":"2024-09-10T19:11:35.174218+0000","last_clean_scrub_stamp":"2024-09-10T19:13:43.358901+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0066143260000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":134,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-09-10T19:17:07.850224+0000","last_change":"2024-09-10T19:12:57.392898+0000","last_active":"2024-09-10T19:17:07.850224+0000","last_peered":"2024-09-10T19:17:07.850224+0000","last_clean":"2024-09-10T19:17:07.850224+0000","last_became_active":"2024-09-10T19:10:14.053875+0000","last_became_peered":"2024-09-10T19:10:14.053875+0000","last_unstale":"2024-09-10T19:17:07.850224+0000","last_undegraded":"2024-09-10T19:17:07.850224+0000","last_fullsized":"2024-09-10T19:17:07.850224+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:12:57.392844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:12:57.392844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0046180839999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.5","version":"170'4597","reported_seq":4627,"reported_epoch":170,"state":"active+clean","last_fresh":"2024-09-10T19:17:09.754077+0000","last_change":"2024-09-10T19:16:13.081074+0000","last_active":"2024-09-10T19:17:09.754077+0000","last_peered":"2024-09-10T19:17:09.754077+0000","last_clean":"2024-09-10T19:17:09.754077+0000","last_became_active":"2024-09-10T19:16:13.080762+0000","last_became_peered":"2024-09-10T19:16:13.080762+0000","last_unstale":"2024-09-10T19:17:09.754077+0000","last_undegraded":"2024-09-10T19:17:09.754077+0000","last_fullsized":"2024-09-10T19:17:09.754077+0000","mapping_epoch":148,"log_start":"156'500","ondisk_log_start":"156'500","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:16:11.777014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:16:11.777014+0000","last_clean_scrub_stamp":"2024-09-10T19:16:11.777014+0000","objects_scrubbed":0,"log_size":4097,"log_dups_size":0,"ondisk_log_size":4097,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:17:34.160639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":301268992,"num_objects":4597,"num_object_clones":0,"num_object_copies":13791,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4597,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4597,"num_write_kb":294208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.9","version":"66'12","reported_seq":192,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-09-10T19:17:07.845740+0000","last_change":"2024-09-10T19:15:37.869593+0000","last_active":"2024-09-10T19:17:07.845740+0000","last_peered":"2024-09-10T19:17:07.845740+0000","last_clean":"2024-09-10T19:17:07.845740+0000","last_became_active":"2024-09-10T19:13:54.718152+0000","last_became_peered":"2024-09-10T19:13:54.718152+0000","last_unstale":"2024-09-10T19:17:07.845740+0000","last_undegraded":"2024-09-10T19:17:07.845740+0000","last_fullsized":"2024-09-10T19:17:07.845740+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'12","last_scrub_stamp":"2024-09-10T19:15:37.869451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:15:37.869451+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0025441249999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.f","version":"59'2","reported_seq":165,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-09-10T19:17:07.847399+0000","last_change":"2024-09-10T19:16:29.007627+0000","last_active":"2024-09-10T19:17:07.847399+0000","last_peered":"2024-09-10T19:17:07.847399+0000","last_clean":"2024-09-10T19:17:07.847399+0000","last_became_active":"2024-09-10T19:10:07.681719+0000","last_became_peered":"2024-09-10T19:10:07.681719+0000","last_unstale":"2024-09-10T19:17:07.847399+0000","last_undegraded":"2024-09-10T19:17:07.847399+0000","last_fullsized":"2024-09-10T19:17:07.847399+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'2","last_scrub_stamp":"2024-09-10T19:16:29.007251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:16:29.007251+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:17:56.729256+0000","scrub_duration":0.0079698319999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":14,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":108,"reported_epoch":170,"state":"active+clean","last_fresh":"2024-09-10T19:17:08.862430+0000","last_change":"2024-09-10T19:16:24.840337+0000","last_active":"2024-09-10T19:17:08.862430+0000","last_peered":"2024-09-10T19:17:08.862430+0000","last_clean":"2024-09-10T19:17:08.862430+0000","last_became_active":"2024-09-10T19:16:24.840091+0000","last_became_peered":"2024-09-10T19:16:24.840091+0000","last_unstale":"2024-09-10T19:17:08.862430+0000","last_undegraded":"2024-09-10T19:17:08.862430+0000","last_fullsized":"2024-09-10T19:17:08.862430+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:15:31.776481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:11:46.587846+0000","last_clean_scrub_stamp":"2024-09-10T19:15:31.776481+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001267142,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,4],"acting":[0,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":152,"reported_epoch":170,"state":"active+clean","last_fresh":"2024-09-10T19:17:08.869707+0000","last_change":"2024-09-10T19:15:41.093923+0000","last_active":"2024-09-10T19:17:08.869707+0000","last_peered":"2024-09-10T19:17:08.869707+0000","last_clean":"2024-09-10T19:17:08.869707+0000","last_became_active":"2024-09-10T19:13:54.711804+0000","last_became_peered":"2024-09-10T19:13:54.711804+0000","last_unstale":"2024-09-10T19:17:08.869707+0000","last_undegraded":"2024-09-10T19:17:08.869707+0000","last_fullsized":"2024-09-10T19:17:08.869707+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:15:41.093783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:15:41.093783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0023833050000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.16","version":"66'9","reported_seq":164,"reported_epoch":170,"state":"active+clean","last_fresh":"2024-09-10T19:17:08.864637+0000","last_change":"2024-09-10T19:13:50.172901+0000","last_active":"2024-09-10T19:17:08.864637+0000","last_peered":"2024-09-10T19:17:08.864637+0000","last_clean":"2024-09-10T19:17:08.864637+0000","last_became_active":"2024-09-10T19:10:09.822994+0000","last_became_peered":"2024-09-10T19:10:09.822994+0000","last_unstale":"2024-09-10T19:17:08.864637+0000","last_undegraded":"2024-09-10T19:17:08.864637+0000","last_fullsized":"2024-09-10T19:17:08.864637+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'9","last_scrub_stamp":"2024-09-10T19:13:50.172825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:13:50.172825+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0094381619999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":112,"reported_epoch":170,"state":"active+clean","last_fresh":"2024-09-10T19:17:08.862785+0000","last_change":"2024-09-10T19:16:24.844555+0000","last_active":"2024-09-10T19:17:08.862785+0000","last_peered":"2024-09-10T19:17:08.862785+0000","last_clean":"2024-09-10T19:17:08.862785+0000","last_became_active":"2024-09-10T19:16:24.844313+0000","last_became_peered":"2024-09-10T19:16:24.844313+0000","last_unstale":"2024-09-10T19:17:08.862785+0000","last_undegraded":"2024-09-10T19:17:08.862785+0000","last_fullsized":"2024-09-10T19:17:08.862785+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:15:36.662596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:11:41.458523+0000","last_clean_scrub_stamp":"2024-09-10T19:15:36.662596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0012499830000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,6],"acting":[0,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":154,"reported_epoch":170,"state":"active+clean","last_fresh":"2024-09-10T19:17:08.869808+0000","last_change":"2024-09-10T19:16:28.794727+0000","last_active":"2024-09-10T19:17:08.869808+0000","last_peered":"2024-09-10T19:17:08.869808+0000","last_clean":"2024-09-10T19:17:08.869808+0000","last_became_active":"2024-09-10T19:13:54.712275+0000","last_became_peered":"2024-09-10T19:13:54.712275+0000","last_unstale":"2024-09-10T19:17:08.869808+0000","last_undegraded":"2024-09-10T19:17:08.869808+0000","last_fullsized":"2024-09-10T19:17:08.869808+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:16:28.794649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:16:28.794649+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:17:53.505673+0000","scrub_duration":0.0027500379999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"66'1","reported_seq":41,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-09-10T19:17:07.845641+0000","last_change":"2024-09-10T19:16:24.862865+0000","last_active":"2024-09-10T19:17:07.845641+0000","last_peered":"2024-09-10T19:17:07.845641+0000","last_clean":"2024-09-10T19:17:07.845641+0000","last_became_active":"2024-09-10T19:16:24.837432+0000","last_became_peered":"2024-09-10T19:16:24.837432+0000","last_unstale":"2024-09-10T19:17:07.845641+0000","last_undegraded":"2024-09-10T19:17:07.845641+0000","last_fullsized":"2024-09-10T19:17:07.845641+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'1","last_scrub_stamp":"2024-09-10T19:15:48.839582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:15:48.839582+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":13,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":160,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-09-10T19:17:07.854896+0000","last_change":"2024-09-10T19:15:36.714392+0000","last_active":"2024-09-10T19:17:07.854896+0000","last_peered":"2024-09-10T19:17:07.854896+0000","last_clean":"2024-09-10T19:17:07.854896+0000","last_became_active":"2024-09-10T19:13:54.724027+0000","last_became_peered":"2024-09-10T19:13:54.724027+0000","last_unstale":"2024-09-10T19:17:07.854896+0000","last_undegraded":"2024-09-10T19:17:07.854896+0000","last_fullsized":"2024-09-10T19:17:07.854896+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:15:36.714327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:15:36.714327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001831287,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,5],"acting":[6,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"66'6","reported_seq":181,"reported_epoch":170,"state":"active+clean","last_fresh":"2024-09-10T19:17:08.869882+0000","last_change":"2024-09-10T19:16:24.891057+0000","last_active":"2024-09-10T19:17:08.869882+0000","last_peered":"2024-09-10T19:17:08.869882+0000","last_clean":"2024-09-10T19:17:08.869882+0000","last_became_active":"2024-09-10T19:16:24.863398+0000","last_became_peered":"2024-09-10T19:16:24.863398+0000","last_unstale":"2024-09-10T19:17:08.869882+0000","last_undegraded":"2024-09-10T19:17:08.869882+0000","last_fullsized":"2024-09-10T19:17:08.869882+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'6","last_scrub_stamp":"2024-09-10T19:13:55.370650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:13:55.370650+0000","objects_scrubbed":3,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.071693797000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":153,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-09-10T19:17:07.848776+0000","last_change":"2024-09-10T19:15:30.583958+0000","last_active":"2024-09-10T19:17:07.848776+0000","last_peered":"2024-09-10T19:17:07.848776+0000","last_clean":"2024-09-10T19:17:07.848776+0000","last_became_active":"2024-09-10T19:13:54.729917+0000","last_became_peered":"2024-09-10T19:13:54.729917+0000","last_unstale":"2024-09-10T19:17:07.848776+0000","last_undegraded":"2024-09-10T19:17:07.848776+0000","last_fullsized":"2024-09-10T19:17:07.848776+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:15:30.583878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:11:23.264793+0000","last_clean_scrub_stamp":"2024-09-10T19:15:30.583878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0015146960000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":134,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-09-10T19:17:07.854831+0000","last_change":"2024-09-10T19:12:51.508398+0000","last_active":"2024-09-10T19:17:07.854831+0000","last_peered":"2024-09-10T19:17:07.854831+0000","last_clean":"2024-09-10T19:17:07.854831+0000","last_became_active":"2024-09-10T19:10:14.054360+0000","last_became_peered":"2024-09-10T19:10:14.054360+0000","last_unstale":"2024-09-10T19:17:07.854831+0000","last_undegraded":"2024-09-10T19:17:07.854831+0000","last_fullsized":"2024-09-10T19:17:07.854831+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:12:51.508263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:12:51.508263+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001748274,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"80'15","reported_seq":210,"reported_epoch":170,"state":"active+clean","last_fresh":"2024-09-10T19:17:08.869461+0000","last_change":"2024-09-10T19:16:24.881305+0000","last_active":"2024-09-10T19:17:08.869461+0000","last_peered":"2024-09-10T19:17:08.869461+0000","last_clean":"2024-09-10T19:17:08.869461+0000","last_became_active":"2024-09-10T19:16:24.850702+0000","last_became_peered":"2024-09-10T19:16:24.850702+0000","last_unstale":"2024-09-10T19:17:08.869461+0000","last_undegraded":"2024-09-10T19:17:08.869461+0000","last_fullsized":"2024-09-10T19:17:08.869461+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"80'15","last_scrub_stamp":"2024-09-10T19:15:46.019949+0000","last_deep_scrub":"80'15","last_deep_scrub_stamp":"2024-09-10T19:11:41.719310+0000","last_clean_scrub_stamp":"2024-09-10T19:15:46.019949+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0021039190000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,5],"acting":[3,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":140,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-09-10T19:17:07.850974+0000","last_change":"2024-09-10T19:12:46.506129+0000","last_active":"2024-09-10T19:17:07.850974+0000","last_peered":"2024-09-10T19:17:07.850974+0000","last_clean":"2024-09-10T19:17:07.850974+0000","last_became_active":"2024-09-10T19:10:07.685870+0000","last_became_peered":"2024-09-10T19:10:07.685870+0000","last_unstale":"2024-09-10T19:17:07.850974+0000","last_undegraded":"2024-09-10T19:17:07.850974+0000","last_fullsized":"2024-09-10T19:17:07.850974+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:12:46.506071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:12:46.506071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001257935,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"117'10","reported_seq":250,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-09-10T19:17:07.849457+0000","last_change":"2024-09-10T19:16:23.844790+0000","last_active":"2024-09-10T19:17:07.849457+0000","last_peered":"2024-09-10T19:17:07.849457+0000","last_clean":"2024-09-10T19:17:07.849457+0000","last_became_active":"2024-09-10T19:13:54.724780+0000","last_became_peered":"2024-09-10T19:13:54.724780+0000","last_unstale":"2024-09-10T19:17:07.849457+0000","last_undegraded":"2024-09-10T19:17:07.849457+0000","last_fullsized":"2024-09-10T19:17:07.849457+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"117'10","last_scrub_stamp":"2024-09-10T19:16:23.844724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:16:23.844724+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:17:28.307512+0000","scrub_duration":0.012497955999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3],"acting":[5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":144,"reported_epoch":170,"state":"active+clean","last_fresh":"2024-09-10T19:17:08.864451+0000","last_change":"2024-09-10T19:16:40.213422+0000","last_active":"2024-09-10T19:17:08.864451+0000","last_peered":"2024-09-10T19:17:08.864451+0000","last_clean":"2024-09-10T19:17:08.864451+0000","last_became_active":"2024-09-10T19:10:13.721035+0000","last_became_peered":"2024-09-10T19:10:13.721035+0000","last_unstale":"2024-09-10T19:17:08.864451+0000","last_undegraded":"2024-09-10T19:17:08.864451+0000","last_fullsized":"2024-09-10T19:17:08.864451+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:16:40.213321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:16:40.213321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:18:03.471480+0000","scrub_duration":0.0051408239999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"66'9","reported_seq":194,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-09-10T19:17:07.849526+0000","last_change":"2024-09-10T19:16:24.882653+0000","last_active":"2024-09-10T19:17:07.849526+0000","last_peered":"2024-09-10T19:17:07.849526+0000","last_clean":"2024-09-10T19:17:07.849526+0000","last_became_active":"2024-09-10T19:16:24.859740+0000","last_became_peered":"2024-09-10T19:16:24.859740+0000","last_unstale":"2024-09-10T19:17:07.849526+0000","last_undegraded":"2024-09-10T19:17:07.849526+0000","last_fullsized":"2024-09-10T19:17:07.849526+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'9","last_scrub_stamp":"2024-09-10T19:12:52.416748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:12:52.416748+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0047220839999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":150,"reported_epoch":170,"state":"active+clean","last_fresh":"2024-09-10T19:17:08.863610+0000","last_change":"2024-09-10T19:16:37.243942+0000","last_active":"2024-09-10T19:17:08.863610+0000","last_peered":"2024-09-10T19:17:08.863610+0000","last_clean":"2024-09-10T19:17:08.863610+0000","last_became_active":"2024-09-10T19:10:07.688499+0000","last_became_peered":"2024-09-10T19:10:07.688499+0000","last_unstale":"2024-09-10T19:17:08.863610+0000","last_undegraded":"2024-09-10T19:17:08.863610+0000","last_fullsized":"2024-09-10T19:17:08.863610+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:16:37.243859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:16:37.243859+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:18:05.478908+0000","scrub_duration":0.0052374580000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"155'9","reported_seq":243,"reported_epoch":170,"state":"active+clean","last_fresh":"2024-09-10T19:17:08.869140+0000","last_change":"2024-09-10T19:16:24.881133+0000","last_active":"2024-09-10T19:17:08.869140+0000","last_peered":"2024-09-10T19:17:08.869140+0000","last_clean":"2024-09-10T19:17:08.869140+0000","last_became_active":"2024-09-10T19:16:24.850850+0000","last_became_peered":"2024-09-10T19:16:24.850850+0000","last_unstale":"2024-09-10T19:17:08.869140+0000","last_undegraded":"2024-09-10T19:17:08.869140+0000","last_fullsized":"2024-09-10T19:17:08.869140+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'8","last_scrub_stamp":"2024-09-10T19:12:59.723712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:12:59.723712+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0083838109999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,4],"acting":[3,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":143,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-09-10T19:17:07.847133+0000","last_change":"2024-09-10T19:16:24.854571+0000","last_active":"2024-09-10T19:17:07.847133+0000","last_peered":"2024-09-10T19:17:07.847133+0000","last_clean":"2024-09-10T19:17:07.847133+0000","last_became_active":"2024-09-10T19:16:24.854280+0000","last_became_peered":"2024-09-10T19:16:24.854280+0000","last_unstale":"2024-09-10T19:17:07.847133+0000","last_undegraded":"2024-09-10T19:17:07.847133+0000","last_fullsized":"2024-09-10T19:17:07.847133+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:13:09.463001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:13:09.463001+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.012199573,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.12","version":"79'13","reported_seq":177,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-09-10T19:17:07.855187+0000","last_change":"2024-09-10T19:16:24.896575+0000","last_active":"2024-09-10T19:17:07.855187+0000","last_peered":"2024-09-10T19:17:07.855187+0000","last_clean":"2024-09-10T19:17:07.855187+0000","last_became_active":"2024-09-10T19:16:24.845450+0000","last_became_peered":"2024-09-10T19:16:24.845450+0000","last_unstale":"2024-09-10T19:17:07.855187+0000","last_undegraded":"2024-09-10T19:17:07.855187+0000","last_fullsized":"2024-09-10T19:17:07.855187+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'13","last_scrub_stamp":"2024-09-10T19:15:29.854117+0000","last_deep_scrub":"79'13","last_deep_scrub_stamp":"2024-09-10T19:11:22.072400+0000","last_clean_scrub_stamp":"2024-09-10T19:15:29.854117+0000","objects_scrubbed":4,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0066627300000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.14","version":"59'1","reported_seq":157,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-09-10T19:17:07.855329+0000","last_change":"2024-09-10T19:16:27.594895+0000","last_active":"2024-09-10T19:17:07.855329+0000","last_peered":"2024-09-10T19:17:07.855329+0000","last_clean":"2024-09-10T19:17:07.855329+0000","last_became_active":"2024-09-10T19:10:07.683263+0000","last_became_peered":"2024-09-10T19:10:07.683263+0000","last_unstale":"2024-09-10T19:17:07.855329+0000","last_undegraded":"2024-09-10T19:17:07.855329+0000","last_fullsized":"2024-09-10T19:17:07.855329+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'1","last_scrub_stamp":"2024-09-10T19:16:27.594788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:16:27.594788+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:17:46.479670+0000","scrub_duration":0.0091369840000000008,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":362,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":154,"reported_epoch":170,"state":"active+clean","last_fresh":"2024-09-10T19:17:08.869773+0000","last_change":"2024-09-10T19:15:55.925313+0000","last_active":"2024-09-10T19:17:08.869773+0000","last_peered":"2024-09-10T19:17:08.869773+0000","last_clean":"2024-09-10T19:17:08.869773+0000","last_became_active":"2024-09-10T19:13:54.730185+0000","last_became_peered":"2024-09-10T19:13:54.730185+0000","last_unstale":"2024-09-10T19:17:08.869773+0000","last_undegraded":"2024-09-10T19:17:08.869773+0000","last_fullsized":"2024-09-10T19:17:08.869773+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:15:55.925228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:15:55.925228+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:17:11.471760+0000","scrub_duration":0.0017202299999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":151,"reported_epoch":170,"state":"active+clean","last_fresh":"2024-09-10T19:17:08.862286+0000","last_change":"2024-09-10T19:16:23.273671+0000","last_active":"2024-09-10T19:17:08.862286+0000","last_peered":"2024-09-10T19:17:08.862286+0000","last_clean":"2024-09-10T19:17:08.862286+0000","last_became_active":"2024-09-10T19:13:54.714861+0000","last_became_peered":"2024-09-10T19:13:54.714861+0000","last_unstale":"2024-09-10T19:17:08.862286+0000","last_undegraded":"2024-09-10T19:17:08.862286+0000","last_fullsized":"2024-09-10T19:17:08.862286+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:16:23.273615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:16:23.273615+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:17:47.995106+0000","scrub_duration":0.0013972139999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,6],"acting":[0,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"66'11","reported_seq":188,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-09-10T19:17:07.850782+0000","last_change":"2024-09-10T19:15:53.883613+0000","last_active":"2024-09-10T19:17:07.850782+0000","last_peered":"2024-09-10T19:17:07.850782+0000","last_clean":"2024-09-10T19:17:07.850782+0000","last_became_active":"2024-09-10T19:13:54.720034+0000","last_became_peered":"2024-09-10T19:13:54.720034+0000","last_unstale":"2024-09-10T19:17:07.850782+0000","last_undegraded":"2024-09-10T19:17:07.850782+0000","last_fullsized":"2024-09-10T19:17:07.850782+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'11","last_scrub_stamp":"2024-09-10T19:15:53.883548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:15:53.883548+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001858326,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":90,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-09-10T19:17:07.849593+0000","last_change":"2024-09-10T19:13:54.723475+0000","last_active":"2024-09-10T19:17:07.849593+0000","last_peered":"2024-09-10T19:17:07.849593+0000","last_clean":"2024-09-10T19:17:07.849593+0000","last_became_active":"2024-09-10T19:13:54.722947+0000","last_became_peered":"2024-09-10T19:13:54.722947+0000","last_unstale":"2024-09-10T19:17:07.849593+0000","last_undegraded":"2024-09-10T19:17:07.849593+0000","last_fullsized":"2024-09-10T19:17:07.849593+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:12:44.249765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:11:10.031245+0000","last_clean_scrub_stamp":"2024-09-10T19:12:44.249765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,6],"acting":[5,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":100,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-09-10T19:17:07.851112+0000","last_change":"2024-09-10T19:15:36.666310+0000","last_active":"2024-09-10T19:17:07.851112+0000","last_peered":"2024-09-10T19:17:07.851112+0000","last_clean":"2024-09-10T19:17:07.851112+0000","last_became_active":"2024-09-10T19:13:54.729703+0000","last_became_peered":"2024-09-10T19:13:54.729703+0000","last_unstale":"2024-09-10T19:17:07.851112+0000","last_undegraded":"2024-09-10T19:17:07.851112+0000","last_fullsized":"2024-09-10T19:17:07.851112+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:15:36.666216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:15:36.666216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.003489884,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":135,"reported_epoch":170,"state":"active+clean","last_fresh":"2024-09-10T19:17:08.868611+0000","last_change":"2024-09-10T19:12:54.691886+0000","last_active":"2024-09-10T19:17:08.868611+0000","last_peered":"2024-09-10T19:17:08.868611+0000","last_clean":"2024-09-10T19:17:08.868611+0000","last_became_active":"2024-09-10T19:10:13.724201+0000","last_became_peered":"2024-09-10T19:10:13.724201+0000","last_unstale":"2024-09-10T19:17:08.868611+0000","last_undegraded":"2024-09-10T19:17:08.868611+0000","last_fullsized":"2024-09-10T19:17:08.868611+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:12:54.691810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:12:54.691810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.006702927,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"66'4","reported_seq":168,"reported_epoch":170,"state":"active+clean","last_fresh":"2024-09-10T19:17:08.868891+0000","last_change":"2024-09-10T19:15:33.995417+0000","last_active":"2024-09-10T19:17:08.868891+0000","last_peered":"2024-09-10T19:17:08.868891+0000","last_clean":"2024-09-10T19:17:08.868891+0000","last_became_active":"2024-09-10T19:13:54.711125+0000","last_became_peered":"2024-09-10T19:13:54.711125+0000","last_unstale":"2024-09-10T19:17:08.868891+0000","last_undegraded":"2024-09-10T19:17:08.868891+0000","last_fullsized":"2024-09-10T19:17:08.868891+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'4","last_scrub_stamp":"2024-09-10T19:15:33.995274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:15:33.995274+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.011108425,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,4],"acting":[3,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":160,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-09-10T19:17:07.849025+0000","last_change":"2024-09-10T19:16:33.766560+0000","last_active":"2024-09-10T19:17:07.849025+0000","last_peered":"2024-09-10T19:17:07.849025+0000","last_clean":"2024-09-10T19:17:07.849025+0000","last_became_active":"2024-09-10T19:16:24.852460+0000","last_became_peered":"2024-09-10T19:16:24.852460+0000","last_unstale":"2024-09-10T19:17:07.849025+0000","last_undegraded":"2024-09-10T19:17:07.849025+0000","last_fullsized":"2024-09-10T19:17:07.849025+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:16:33.766474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:16:33.766474+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:18:01.711870+0000","scrub_duration":0.0042170250000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,3],"acting":[5,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":146,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-09-10T19:17:07.854769+0000","last_change":"2024-09-10T19:16:24.545188+0000","last_active":"2024-09-10T19:17:07.854769+0000","last_peered":"2024-09-10T19:17:07.854769+0000","last_clean":"2024-09-10T19:17:07.854769+0000","last_became_active":"2024-09-10T19:10:11.698536+0000","last_became_peered":"2024-09-10T19:10:11.698536+0000","last_unstale":"2024-09-10T19:17:07.854769+0000","last_undegraded":"2024-09-10T19:17:07.854769+0000","last_fullsized":"2024-09-10T19:17:07.854769+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:16:24.545134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:11:31.837944+0000","last_clean_scrub_stamp":"2024-09-10T19:16:24.545134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:17:54.187236+0000","scrub_duration":0.00120104,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":154,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-09-10T19:17:07.854675+0000","last_change":"2024-09-10T19:16:24.852651+0000","last_active":"2024-09-10T19:17:07.854675+0000","last_peered":"2024-09-10T19:17:07.854675+0000","last_clean":"2024-09-10T19:17:07.854675+0000","last_became_active":"2024-09-10T19:16:24.852356+0000","last_became_peered":"2024-09-10T19:16:24.852356+0000","last_unstale":"2024-09-10T19:17:07.854675+0000","last_undegraded":"2024-09-10T19:17:07.854675+0000","last_fullsized":"2024-09-10T19:17:07.854675+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:15:44.523530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:15:44.523530+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.027001213999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,3],"acting":[7,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.11","version":"66'11","reported_seq":48,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-09-10T19:17:07.848456+0000","last_change":"2024-09-10T19:15:45.158623+0000","last_active":"2024-09-10T19:17:07.848456+0000","last_peered":"2024-09-10T19:17:07.848456+0000","last_clean":"2024-09-10T19:17:07.848456+0000","last_became_active":"2024-09-10T19:15:45.149864+0000","last_became_peered":"2024-09-10T19:15:45.149864+0000","last_unstale":"2024-09-10T19:17:07.848456+0000","last_undegraded":"2024-09-10T19:17:07.848456+0000","last_fullsized":"2024-09-10T19:17:07.848456+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'11","last_scrub_stamp":"2024-09-10T19:13:05.729854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:13:05.729854+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.010667395,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":160,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-09-10T19:17:07.856135+0000","last_change":"2024-09-10T19:16:24.855851+0000","last_active":"2024-09-10T19:17:07.856135+0000","last_peered":"2024-09-10T19:17:07.856135+0000","last_clean":"2024-09-10T19:17:07.856135+0000","last_became_active":"2024-09-10T19:16:24.855541+0000","last_became_peered":"2024-09-10T19:16:24.855541+0000","last_unstale":"2024-09-10T19:17:07.856135+0000","last_undegraded":"2024-09-10T19:17:07.856135+0000","last_fullsized":"2024-09-10T19:17:07.856135+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:16:23.543097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:13:05.876655+0000","last_clean_scrub_stamp":"2024-09-10T19:16:23.543097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:17:31.706207+0000","scrub_duration":0.001181779,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,0],"acting":[6,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":136,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-09-10T19:17:07.853410+0000","last_change":"2024-09-10T19:13:41.326028+0000","last_active":"2024-09-10T19:17:07.853410+0000","last_peered":"2024-09-10T19:17:07.853410+0000","last_clean":"2024-09-10T19:17:07.853410+0000","last_became_active":"2024-09-10T19:10:11.699013+0000","last_became_peered":"2024-09-10T19:10:11.699013+0000","last_unstale":"2024-09-10T19:17:07.853410+0000","last_undegraded":"2024-09-10T19:17:07.853410+0000","last_fullsized":"2024-09-10T19:17:07.853410+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:13:41.325941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:11:57.112092+0000","last_clean_scrub_stamp":"2024-09-10T19:13:41.325941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001598676,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":135,"reported_epoch":170,"state":"active+clean","last_fresh":"2024-09-10T19:17:08.869625+0000","last_change":"2024-09-10T19:13:15.812326+0000","last_active":"2024-09-10T19:17:08.869625+0000","last_peered":"2024-09-10T19:17:08.869625+0000","last_clean":"2024-09-10T19:17:08.869625+0000","last_became_active":"2024-09-10T19:10:14.056172+0000","last_became_peered":"2024-09-10T19:10:14.056172+0000","last_unstale":"2024-09-10T19:17:08.869625+0000","last_undegraded":"2024-09-10T19:17:08.869625+0000","last_fullsized":"2024-09-10T19:17:08.869625+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:13:15.812244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:13:15.812244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0048044389999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"80'2","reported_seq":160,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-09-10T19:17:07.855446+0000","last_change":"2024-09-10T19:16:32.648844+0000","last_active":"2024-09-10T19:17:07.855446+0000","last_peered":"2024-09-10T19:17:07.855446+0000","last_clean":"2024-09-10T19:17:07.855446+0000","last_became_active":"2024-09-10T19:16:24.851220+0000","last_became_peered":"2024-09-10T19:16:24.851220+0000","last_unstale":"2024-09-10T19:17:07.855446+0000","last_undegraded":"2024-09-10T19:17:07.855446+0000","last_fullsized":"2024-09-10T19:17:07.855446+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"80'2","last_scrub_stamp":"2024-09-10T19:16:32.648738+0000","last_deep_scrub":"80'2","last_deep_scrub_stamp":"2024-09-10T19:11:30.272043+0000","last_clean_scrub_stamp":"2024-09-10T19:16:32.648738+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:17:38.161227+0000","scrub_duration":0.0066232920000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":369,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":369,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,3],"acting":[7,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":140,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-09-10T19:17:07.850447+0000","last_change":"2024-09-10T19:13:14.449643+0000","last_active":"2024-09-10T19:17:07.850447+0000","last_peered":"2024-09-10T19:17:07.850447+0000","last_clean":"2024-09-10T19:17:07.850447+0000","last_became_active":"2024-09-10T19:10:07.689877+0000","last_became_peered":"2024-09-10T19:10:07.689877+0000","last_unstale":"2024-09-10T19:17:07.850447+0000","last_undegraded":"2024-09-10T19:17:07.850447+0000","last_fullsized":"2024-09-10T19:17:07.850447+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:13:14.449593+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:13:14.449593+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001195542,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1e","version":"80'15","reported_seq":168,"reported_epoch":170,"state":"active+clean","last_fresh":"2024-09-10T19:17:08.864662+0000","last_change":"2024-09-10T19:16:33.258114+0000","last_active":"2024-09-10T19:17:08.864662+0000","last_peered":"2024-09-10T19:17:08.864662+0000","last_clean":"2024-09-10T19:17:08.864662+0000","last_became_active":"2024-09-10T19:10:09.823061+0000","last_became_peered":"2024-09-10T19:10:09.823061+0000","last_unstale":"2024-09-10T19:17:08.864662+0000","last_undegraded":"2024-09-10T19:17:08.864662+0000","last_fullsized":"2024-09-10T19:17:08.864662+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"80'15","last_scrub_stamp":"2024-09-10T19:16:33.258031+0000","last_deep_scrub":"80'15","last_deep_scrub_stamp":"2024-09-10T19:11:25.883919+0000","last_clean_scrub_stamp":"2024-09-10T19:16:33.258031+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:17:57.303783+0000","scrub_duration":0.0040337309999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1f","version":"66'8","reported_seq":241,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-09-10T19:17:07.856055+0000","last_change":"2024-09-10T19:15:53.664303+0000","last_active":"2024-09-10T19:17:07.856055+0000","last_peered":"2024-09-10T19:17:07.856055+0000","last_clean":"2024-09-10T19:17:07.856055+0000","last_became_active":"2024-09-10T19:10:11.695427+0000","last_became_peered":"2024-09-10T19:10:11.695427+0000","last_unstale":"2024-09-10T19:17:07.856055+0000","last_undegraded":"2024-09-10T19:17:07.856055+0000","last_fullsized":"2024-09-10T19:17:07.856055+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'8","last_scrub_stamp":"2024-09-10T19:15:53.664208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:15:53.664208+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:17:15.336245+0000","scrub_duration":0.001849699,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":110,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-09-10T19:17:07.848543+0000","last_change":"2024-09-10T19:16:24.863000+0000","last_active":"2024-09-10T19:17:07.848543+0000","last_peered":"2024-09-10T19:17:07.848543+0000","last_clean":"2024-09-10T19:17:07.848543+0000","last_became_active":"2024-09-10T19:16:24.862695+0000","last_became_peered":"2024-09-10T19:16:24.862695+0000","last_unstale":"2024-09-10T19:17:07.848543+0000","last_undegraded":"2024-09-10T19:17:07.848543+0000","last_fullsized":"2024-09-10T19:17:07.848543+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:15:29.580605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:15:29.580605+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0047886710000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":151,"reported_epoch":170,"state":"active+clean","last_fresh":"2024-09-10T19:17:08.869155+0000","last_change":"2024-09-10T19:16:41.881053+0000","last_active":"2024-09-10T19:17:08.869155+0000","last_peered":"2024-09-10T19:17:08.869155+0000","last_clean":"2024-09-10T19:17:08.869155+0000","last_became_active":"2024-09-10T19:10:07.682689+0000","last_became_peered":"2024-09-10T19:10:07.682689+0000","last_unstale":"2024-09-10T19:17:08.869155+0000","last_undegraded":"2024-09-10T19:17:08.869155+0000","last_fullsized":"2024-09-10T19:17:08.869155+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:16:41.880992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:16:41.880992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:17:54.061215+0000","scrub_duration":0.0044984990000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"80'16","reported_seq":216,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-09-10T19:17:07.856154+0000","last_change":"2024-09-10T19:15:47.700592+0000","last_active":"2024-09-10T19:17:07.856154+0000","last_peered":"2024-09-10T19:17:07.856154+0000","last_clean":"2024-09-10T19:17:07.856154+0000","last_became_active":"2024-09-10T19:13:54.713563+0000","last_became_peered":"2024-09-10T19:13:54.713563+0000","last_unstale":"2024-09-10T19:17:07.856154+0000","last_undegraded":"2024-09-10T19:17:07.856154+0000","last_fullsized":"2024-09-10T19:17:07.856154+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"80'16","last_scrub_stamp":"2024-09-10T19:15:47.700523+0000","last_deep_scrub":"80'16","last_deep_scrub_stamp":"2024-09-10T19:11:40.853611+0000","last_clean_scrub_stamp":"2024-09-10T19:15:47.700523+0000","objects_scrubbed":5,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0019149010000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,0],"acting":[6,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":156,"reported_epoch":170,"state":"active+clean","last_fresh":"2024-09-10T19:17:08.864563+0000","last_change":"2024-09-10T19:16:34.222429+0000","last_active":"2024-09-10T19:17:08.864563+0000","last_peered":"2024-09-10T19:17:08.864563+0000","last_clean":"2024-09-10T19:17:08.864563+0000","last_became_active":"2024-09-10T19:16:24.834713+0000","last_became_peered":"2024-09-10T19:16:24.834713+0000","last_unstale":"2024-09-10T19:17:08.864563+0000","last_undegraded":"2024-09-10T19:17:08.864563+0000","last_fullsized":"2024-09-10T19:17:08.864563+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:16:34.222365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:11:28.822327+0000","last_clean_scrub_stamp":"2024-09-10T19:16:34.222365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:17:37.172660+0000","scrub_duration":0.014893412999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":143,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-09-10T19:17:07.846901+0000","last_change":"2024-09-10T19:16:36.960796+0000","last_active":"2024-09-10T19:17:07.846901+0000","last_peered":"2024-09-10T19:17:07.846901+0000","last_clean":"2024-09-10T19:17:07.846901+0000","last_became_active":"2024-09-10T19:10:14.053619+0000","last_became_peered":"2024-09-10T19:10:14.053619+0000","last_unstale":"2024-09-10T19:17:07.846901+0000","last_undegraded":"2024-09-10T19:17:07.846901+0000","last_fullsized":"2024-09-10T19:17:07.846901+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:16:36.960695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:16:36.960695+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:17:59.816942+0000","scrub_duration":0.011430734999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":150,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-09-10T19:17:07.855541+0000","last_change":"2024-09-10T19:16:30.666084+0000","last_active":"2024-09-10T19:17:07.855541+0000","last_peered":"2024-09-10T19:17:07.855541+0000","last_clean":"2024-09-10T19:17:07.855541+0000","last_became_active":"2024-09-10T19:10:07.685997+0000","last_became_peered":"2024-09-10T19:10:07.685997+0000","last_unstale":"2024-09-10T19:17:07.855541+0000","last_undegraded":"2024-09-10T19:17:07.855541+0000","last_fullsized":"2024-09-10T19:17:07.855541+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:16:30.666001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:16:30.666001+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:17:57.903923+0000","scrub_duration":0.0041927830000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1c","version":"78'22","reported_seq":100,"reported_epoch":170,"state":"active+clean","last_fresh":"2024-09-10T19:17:08.869363+0000","last_change":"2024-09-10T19:16:24.870445+0000","last_active":"2024-09-10T19:17:08.869363+0000","last_peered":"2024-09-10T19:17:08.869363+0000","last_clean":"2024-09-10T19:17:08.869363+0000","last_became_active":"2024-09-10T19:16:24.845764+0000","last_became_peered":"2024-09-10T19:16:24.845764+0000","last_unstale":"2024-09-10T19:17:08.869363+0000","last_undegraded":"2024-09-10T19:17:08.869363+0000","last_fullsized":"2024-09-10T19:17:08.869363+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"78'22","last_scrub_stamp":"2024-09-10T19:15:37.702624+0000","last_deep_scrub":"78'22","last_deep_scrub_stamp":"2024-09-10T19:11:21.118937+0000","last_clean_scrub_stamp":"2024-09-10T19:15:37.702624+0000","objects_scrubbed":7,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":103,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-09-10T19:17:07.846430+0000","last_change":"2024-09-10T19:15:33.008615+0000","last_active":"2024-09-10T19:17:07.846430+0000","last_peered":"2024-09-10T19:17:07.846430+0000","last_clean":"2024-09-10T19:17:07.846430+0000","last_became_active":"2024-09-10T19:13:54.709055+0000","last_became_peered":"2024-09-10T19:13:54.709055+0000","last_unstale":"2024-09-10T19:17:07.846430+0000","last_undegraded":"2024-09-10T19:17:07.846430+0000","last_fullsized":"2024-09-10T19:17:07.846430+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:15:33.008542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:15:33.008542+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.004557363,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,5],"acting":[4,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":135,"reported_epoch":170,"state":"active+clean","last_fresh":"2024-09-10T19:17:08.869748+0000","last_change":"2024-09-10T19:12:51.767586+0000","last_active":"2024-09-10T19:17:08.869748+0000","last_peered":"2024-09-10T19:17:08.869748+0000","last_clean":"2024-09-10T19:17:08.869748+0000","last_became_active":"2024-09-10T19:10:14.053698+0000","last_became_peered":"2024-09-10T19:10:14.053698+0000","last_unstale":"2024-09-10T19:17:08.869748+0000","last_undegraded":"2024-09-10T19:17:08.869748+0000","last_fullsized":"2024-09-10T19:17:08.869748+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:12:51.767056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:12:51.767056+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.006796519,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":141,"reported_epoch":170,"state":"active+clean","last_fresh":"2024-09-10T19:17:08.869921+0000","last_change":"2024-09-10T19:12:50.719167+0000","last_active":"2024-09-10T19:17:08.869921+0000","last_peered":"2024-09-10T19:17:08.869921+0000","last_clean":"2024-09-10T19:17:08.869921+0000","last_became_active":"2024-09-10T19:10:07.682811+0000","last_became_peered":"2024-09-10T19:10:07.682811+0000","last_unstale":"2024-09-10T19:17:08.869921+0000","last_undegraded":"2024-09-10T19:17:08.869921+0000","last_fullsized":"2024-09-10T19:17:08.869921+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:12:50.718663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:12:50.718663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0042138399999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1d","version":"66'12","reported_seq":191,"reported_epoch":170,"state":"active+clean","last_fresh":"2024-09-10T19:17:08.869154+0000","last_change":"2024-09-10T19:15:52.923561+0000","last_active":"2024-09-10T19:17:08.869154+0000","last_peered":"2024-09-10T19:17:08.869154+0000","last_clean":"2024-09-10T19:17:08.869154+0000","last_became_active":"2024-09-10T19:13:54.714629+0000","last_became_peered":"2024-09-10T19:13:54.714629+0000","last_unstale":"2024-09-10T19:17:08.869154+0000","last_undegraded":"2024-09-10T19:17:08.869154+0000","last_fullsized":"2024-09-10T19:17:08.869154+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'12","last_scrub_stamp":"2024-09-10T19:15:52.923498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:15:52.923498+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:17:20.423846+0000","scrub_duration":0.0019450450000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":155,"reported_epoch":169,"state":"active+clean","last_fresh":"2024-09-10T19:17:07.845830+0000","last_change":"2024-09-10T19:16:37.970105+0000","last_active":"2024-09-10T19:17:07.845830+0000","last_peered":"2024-09-10T19:17:07.845830+0000","last_clean":"2024-09-10T19:17:07.845830+0000","last_became_active":"2024-09-10T19:16:24.843013+0000","last_became_peered":"2024-09-10T19:16:24.843013+0000","last_unstale":"2024-09-10T19:17:07.845830+0000","last_undegraded":"2024-09-10T19:17:07.845830+0000","last_fullsized":"2024-09-10T19:17:07.845830+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:16:37.970030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:16:37.970030+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:17:58.177693+0000","scrub_duration":0.003625063,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":8,"num_pg":16,"stat_sum":{"num_bytes":4817682432,"num_objects":73512,"num_object_clones":0,"num_object_copies":220536,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3425,"num_objects_unfound":0,"num_objects_dirty":73512,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":73512,"num_write_kb":4704768,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":455,"num_bytes_recovered":29818880,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":14467596288,"data_stored":14467596288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":65912,"ondisk_log_size":65912,"up":48,"acting":47,"num_store_stats":7},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":382,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":1528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":73,"ondisk_log_size":73,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":177,"num_object_clones":0,"num_object_copies":531,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":177,"num_whiteouts":0,"num_read":695,"num_read_kb":452,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":173,"num_bytes_recovered":3218,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":540672,"data_stored":14242,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":445,"ondisk_log_size":445,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":51,"num_read_kb":46,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":389,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":2002,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":9,"ondisk_log_size":9,"up":9,"acting":9,"num_store_stats":7},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1328,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":27,"num_read_kb":27,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":3984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4,"ondisk_log_size":4,"up":96,"acting":96,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79,"ondisk_log_size":79,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":54,"seq":231928234073,"num_pgs":78,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2451032,"kb_used_data":2137040,"kb_used_omap":1,"kb_used_meta":313982,"kb_avail":91294120,"statfs":{"total":95995035648,"available":93485178880,"internally_reserved":0,"allocated":2188328960,"data_stored":2187855764,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":321517912},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 19:17:03 2024","interfaces":[{"interface":"back","average":{"1min":1.1379999999999999,"5min":1.8859999999999999,"15min":0.98199999999999998},"min":{"1min":0.246,"5min":0.215,"15min":0.215},"max":{"1min":8.3770000000000007,"5min":35.551000000000002,"15min":35.551000000000002},"last":4.7610000000000001},{"interface":"front","average":{"1min":1.081,"5min":1.6919999999999999,"15min":0.92000000000000004},"min":{"1min":0.19500000000000001,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":8.1440000000000001,"5min":35.503999999999998,"15min":35.503999999999998},"last":0.23699999999999999}]},{"osd":1,"last update":"Tue Sep 10 19:16:21 2024","interfaces":[{"interface":"back","average":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"min":{"1min":0.28100000000000003,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":1.101,"5min":1.101,"15min":1.101},"last":4.702},{"interface":"front","average":{"1min":0.70799999999999996,"5min":0.70799999999999996,"15min":0.70799999999999996},"min":{"1min":0.379,"5min":0.379,"15min":0.379},"max":{"1min":3.5819999999999999,"5min":3.5819999999999999,"15min":3.5819999999999999},"last":4.782}]},{"osd":2,"last update":"Tue Sep 10 19:17:03 2024","interfaces":[{"interface":"back","average":{"1min":0.76300000000000001,"5min":1.109,"15min":0.71699999999999997},"min":{"1min":0.27300000000000002,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":2.0139999999999998,"5min":31.940999999999999,"15min":31.940999999999999},"last":0.65400000000000003},{"interface":"front","average":{"1min":1.274,"5min":1.319,"15min":0.80400000000000005},"min":{"1min":0.35799999999999998,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":8.0839999999999996,"5min":25.056000000000001,"15min":25.056000000000001},"last":4.7329999999999997}]},{"osd":3,"last update":"Tue Sep 10 19:17:03 2024","interfaces":[{"interface":"back","average":{"1min":1.121,"5min":1.8660000000000001,"15min":0.97699999999999998},"min":{"1min":0.217,"5min":0.217,"15min":0.217},"max":{"1min":4.4969999999999999,"5min":35.277999999999999,"15min":35.277999999999999},"last":0.624},{"interface":"front","average":{"1min":0.94299999999999995,"5min":1.738,"15min":0.92700000000000005},"min":{"1min":0.28499999999999998,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":2.4260000000000002,"5min":36.512999999999998,"15min":36.512999999999998},"last":4.8019999999999996}]},{"osd":4,"last update":"Tue Sep 10 19:17:03 2024","interfaces":[{"interface":"back","average":{"1min":1.3380000000000001,"5min":0.71699999999999997,"15min":0.54200000000000004},"min":{"1min":0.23300000000000001,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":8.1859999999999999,"5min":8.1859999999999999,"15min":8.1859999999999999},"last":0.93799999999999994},{"interface":"front","average":{"1min":1.0780000000000001,"5min":0.64600000000000002,"15min":0.52200000000000002},"min":{"1min":0.29399999999999998,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":4.5469999999999997,"5min":4.5469999999999997,"15min":4.5469999999999997},"last":0.67600000000000005}]},{"osd":5,"last update":"Tue Sep 10 19:17:03 2024","interfaces":[{"interface":"back","average":{"1min":1.3700000000000001,"5min":0.93700000000000006,"15min":0.63500000000000001},"min":{"1min":0.32000000000000001,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":8.1660000000000004,"5min":11.621,"15min":11.621},"last":8.1120000000000001},{"interface":"front","average":{"1min":1.859,"5min":0.89500000000000002,"15min":0.61799999999999999},"min":{"1min":0.34699999999999998,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":9.2680000000000007,"5min":9.2680000000000007,"15min":9.2680000000000007},"last":2.4409999999999998}]},{"osd":6,"last update":"Tue Sep 10 19:17:03 2024","interfaces":[{"interface":"back","average":{"1min":1.738,"5min":0.90400000000000003,"15min":0.63800000000000001},"min":{"1min":0.32100000000000001,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":8.2590000000000003,"5min":8.2590000000000003,"15min":8.2590000000000003},"last":1.0840000000000001},{"interface":"front","average":{"1min":1.522,"5min":0.79000000000000004,"15min":0.59099999999999997},"min":{"1min":0.373,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":8.4130000000000003,"5min":8.4130000000000003,"15min":8.4130000000000003},"last":0.70199999999999996}]}]},{"osd":6,"up_from":47,"seq":201863463006,"num_pgs":75,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2582924,"kb_used_data":2281412,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":91162228,"statfs":{"total":95995035648,"available":93350121472,"internally_reserved":0,"allocated":2336165888,"data_stored":2335705056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 19:16:46 2024","interfaces":[{"interface":"back","average":{"1min":0.63800000000000001,"5min":2.194,"15min":1.077},"min":{"1min":0.245,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":2.1509999999999998,"5min":112.988,"15min":112.988},"last":0.20999999999999999},{"interface":"front","average":{"1min":0.82999999999999996,"5min":2.218,"15min":1.097},"min":{"1min":0.34300000000000003,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":2.9969999999999999,"5min":113.18300000000001,"15min":113.18300000000001},"last":0.375}]},{"osd":1,"last update":"Tue Sep 10 19:16:22 2024","interfaces":[{"interface":"back","average":{"1min":0.51400000000000001,"5min":0.51400000000000001,"15min":0.51400000000000001},"min":{"1min":0.28499999999999998,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":1.4099999999999999,"5min":1.4099999999999999,"15min":1.4099999999999999},"last":0.27000000000000002},{"interface":"front","average":{"1min":0.65400000000000003,"5min":0.65400000000000003,"15min":0.65400000000000003},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":2.996,"5min":2.996,"15min":2.996},"last":2.4950000000000001}]},{"osd":2,"last update":"Tue Sep 10 19:16:46 2024","interfaces":[{"interface":"back","average":{"1min":0.78200000000000003,"5min":2.0129999999999999,"15min":1.0309999999999999},"min":{"1min":0.35399999999999998,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":3.7959999999999998,"5min":112.864,"15min":112.864},"last":0.29599999999999999},{"interface":"front","average":{"1min":0.80600000000000005,"5min":2.1299999999999999,"15min":1.0489999999999999},"min":{"1min":0.36199999999999999,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":3.0369999999999999,"5min":112.822,"15min":112.822},"last":0.40400000000000003}]},{"osd":3,"last update":"Tue Sep 10 19:16:46 2024","interfaces":[{"interface":"back","average":{"1min":0.754,"5min":2.4580000000000002,"15min":1.1599999999999999},"min":{"1min":0.34799999999999998,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":2.0209999999999999,"5min":125.56399999999999,"15min":125.56399999999999},"last":2.4430000000000001},{"interface":"front","average":{"1min":0.79300000000000004,"5min":2.5430000000000001,"15min":1.222},"min":{"1min":0.36199999999999999,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":3.8290000000000002,"5min":125.47,"15min":125.47},"last":0.435}]},{"osd":4,"last update":"Tue Sep 10 19:16:46 2024","interfaces":[{"interface":"back","average":{"1min":0.92100000000000004,"5min":0.59399999999999997,"15min":0.434},"min":{"1min":0.316,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":3.0630000000000002,"5min":3.3410000000000002,"15min":3.3410000000000002},"last":0.71699999999999997},{"interface":"front","average":{"1min":0.878,"5min":0.627,"15min":0.46100000000000002},"min":{"1min":0.27400000000000002,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":2.7349999999999999,"5min":2.7349999999999999,"15min":2.7349999999999999},"last":2.5150000000000001}]},{"osd":5,"last update":"Tue Sep 10 19:16:46 2024","interfaces":[{"interface":"back","average":{"1min":1.093,"5min":1.119,"15min":0.626},"min":{"1min":0.30499999999999999,"5min":0.248,"15min":0.248},"max":{"1min":3.1099999999999999,"5min":44.765000000000001,"15min":44.765000000000001},"last":2.4060000000000001},{"interface":"front","average":{"1min":1.3240000000000001,"5min":1.288,"15min":0.68400000000000005},"min":{"1min":0.307,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":3.0590000000000002,"5min":43.036000000000001,"15min":43.036000000000001},"last":2.4740000000000002}]},{"osd":7,"last update":"Tue Sep 10 19:17:07 2024","interfaces":[{"interface":"back","average":{"1min":1.5820000000000001,"5min":0.78900000000000003,"15min":0.56599999999999995},"min":{"1min":0.315,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":8.9390000000000001,"5min":8.9390000000000001,"15min":8.9390000000000001},"last":0.34100000000000003},{"interface":"front","average":{"1min":2.0150000000000001,"5min":0.93899999999999995,"15min":0.61399999999999999},"min":{"1min":0.33300000000000002,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":10.298,"5min":10.298,"15min":10.298},"last":3.4729999999999999}]}]},{"osd":5,"up_from":39,"seq":167503724643,"num_pgs":72,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2922760,"kb_used_data":2612920,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":90822392,"statfs":{"total":95995035648,"available":93002129408,"internally_reserved":0,"allocated":2675630080,"data_stored":2675181833,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 19:16:16 2024","interfaces":[{"interface":"back","average":{"1min":0.94799999999999995,"5min":0.876,"15min":0.61099999999999999},"min":{"1min":0.22500000000000001,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":3.7909999999999999,"5min":5.851,"15min":5.851},"last":0.63},{"interface":"front","average":{"1min":1.008,"5min":0.76800000000000002,"15min":0.57399999999999995},"min":{"1min":0.24099999999999999,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":5.923,"5min":5.923,"15min":5.923},"last":0.48199999999999998}]},{"osd":1,"last update":"Tue Sep 10 19:16:22 2024","interfaces":[{"interface":"back","average":{"1min":0.65200000000000002,"5min":0.65200000000000002,"15min":0.65200000000000002},"min":{"1min":0.27400000000000002,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":2.8370000000000002,"5min":2.8370000000000002,"15min":2.8370000000000002},"last":0.52800000000000002},{"interface":"front","average":{"1min":0.503,"5min":0.503,"15min":0.503},"min":{"1min":0.27100000000000002,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":1.4790000000000001,"5min":1.4790000000000001,"15min":1.4790000000000001},"last":2.48}]},{"osd":2,"last update":"Tue Sep 10 19:16:16 2024","interfaces":[{"interface":"back","average":{"1min":0.87,"5min":0.73999999999999999,"15min":0.56599999999999995},"min":{"1min":0.26100000000000001,"5min":0.253,"15min":0.253},"max":{"1min":4.3159999999999998,"5min":4.3159999999999998,"15min":4.3159999999999998},"last":0.66900000000000004},{"interface":"front","average":{"1min":1.0329999999999999,"5min":0.88800000000000001,"15min":0.625},"min":{"1min":0.29199999999999998,"5min":0.224,"15min":0.224},"max":{"1min":3.1110000000000002,"5min":6.3789999999999996,"15min":6.3789999999999996},"last":0.50900000000000001}]},{"osd":3,"last update":"Tue Sep 10 19:16:16 2024","interfaces":[{"interface":"back","average":{"1min":0.751,"5min":0.70999999999999996,"15min":0.58099999999999996},"min":{"1min":0.309,"5min":0.224,"15min":0.224},"max":{"1min":2.3460000000000001,"5min":3.351,"15min":3.351},"last":0.68999999999999995},{"interface":"front","average":{"1min":0.84599999999999997,"5min":0.755,"15min":0.60499999999999998},"min":{"1min":0.33600000000000002,"5min":0.248,"15min":0.248},"max":{"1min":2.8580000000000001,"5min":5.3869999999999996,"15min":5.3869999999999996},"last":0.71199999999999997}]},{"osd":4,"last update":"Tue Sep 10 19:16:16 2024","interfaces":[{"interface":"back","average":{"1min":0.54500000000000004,"5min":0.60199999999999998,"15min":0.441},"min":{"1min":0.25800000000000001,"5min":0.23699999999999999,"15min":0.20699999999999999},"max":{"1min":1.6359999999999999,"5min":4.3280000000000003,"15min":4.3280000000000003},"last":3.6869999999999998},{"interface":"front","average":{"1min":0.58799999999999997,"5min":0.66800000000000004,"15min":0.46999999999999997},"min":{"1min":0.31900000000000001,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":2.02,"5min":5.8899999999999997,"15min":5.8899999999999997},"last":1.6120000000000001}]},{"osd":6,"last update":"Tue Sep 10 19:16:45 2024","interfaces":[{"interface":"back","average":{"1min":0.89900000000000002,"5min":0.85499999999999998,"15min":0.56299999999999994},"min":{"1min":0.36599999999999999,"5min":0.27100000000000002,"15min":0.223},"max":{"1min":3.6509999999999998,"5min":6.617,"15min":6.617},"last":3.8799999999999999},{"interface":"front","average":{"1min":0.621,"5min":0.76000000000000001,"15min":0.52900000000000003},"min":{"1min":0.34100000000000003,"5min":0.26900000000000002,"15min":0.25},"max":{"1min":1.2250000000000001,"5min":5.9349999999999996,"15min":5.9349999999999996},"last":0.58399999999999996}]},{"osd":7,"last update":"Tue Sep 10 19:17:05 2024","interfaces":[{"interface":"back","average":{"1min":1.3100000000000001,"5min":0.874,"15min":0.629},"min":{"1min":0.35999999999999999,"5min":0.307,"15min":0.29099999999999998},"max":{"1min":5.0640000000000001,"5min":5.9660000000000002,"15min":5.9660000000000002},"last":0.60399999999999998},{"interface":"front","average":{"1min":1.2410000000000001,"5min":0.752,"15min":0.56999999999999995},"min":{"1min":0.28499999999999998,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":5.0149999999999997,"5min":5.0149999999999997,"15min":5.0149999999999997},"last":2.444}]}]},{"osd":4,"up_from":32,"seq":137438953575,"num_pgs":70,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2899376,"kb_used_data":2593700,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":90845776,"statfs":{"total":95995035648,"available":93026074624,"internally_reserved":0,"allocated":2655948800,"data_stored":2655511075,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 19:17:03 2024","interfaces":[{"interface":"back","average":{"1min":0.81599999999999995,"5min":1.2509999999999999,"15min":0.77100000000000002},"min":{"1min":0.187,"5min":0.187,"15min":0.187},"max":{"1min":3.8559999999999999,"5min":32.533000000000001,"15min":32.533000000000001},"last":0.84899999999999998},{"interface":"front","average":{"1min":0.60799999999999998,"5min":1.2729999999999999,"15min":0.77100000000000002},"min":{"1min":0.223,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":2.4540000000000002,"5min":32.512999999999998,"15min":32.512999999999998},"last":1.0109999999999999}]},{"osd":1,"last update":"Tue Sep 10 19:16:20 2024","interfaces":[{"interface":"back","average":{"1min":0.628,"5min":0.628,"15min":0.628},"min":{"1min":0.28299999999999997,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":2.5409999999999999,"5min":2.5409999999999999,"15min":2.5409999999999999},"last":0.86799999999999999},{"interface":"front","average":{"1min":0.67600000000000005,"5min":0.67600000000000005,"15min":0.67600000000000005},"min":{"1min":0.32100000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":2.5219999999999998,"5min":2.5219999999999998,"15min":2.5219999999999998},"last":0.94699999999999995}]},{"osd":2,"last update":"Tue Sep 10 19:17:03 2024","interfaces":[{"interface":"back","average":{"1min":0.98899999999999999,"5min":0.88700000000000001,"15min":0.65000000000000002},"min":{"1min":0.34899999999999998,"5min":0.214,"15min":0.214},"max":{"1min":3.8849999999999998,"5min":6.8869999999999996,"15min":6.8869999999999996},"last":0.92200000000000004},{"interface":"front","average":{"1min":0.63600000000000001,"5min":0.83599999999999997,"15min":0.629},"min":{"1min":0.27800000000000002,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":2.4849999999999999,"5min":7.085,"15min":7.085},"last":0.96099999999999997}]},{"osd":3,"last update":"Tue Sep 10 19:17:03 2024","interfaces":[{"interface":"back","average":{"1min":0.64100000000000001,"5min":1.2290000000000001,"15min":0.75600000000000001},"min":{"1min":0.308,"5min":0.252,"15min":0.252},"max":{"1min":2.5089999999999999,"5min":35.451999999999998,"15min":35.451999999999998},"last":1.032},{"interface":"front","average":{"1min":0.76900000000000002,"5min":1.2849999999999999,"15min":0.79100000000000004},"min":{"1min":0.26400000000000001,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":4.9669999999999996,"5min":32.880000000000003,"15min":32.880000000000003},"last":0.93400000000000005}]},{"osd":5,"last update":"Tue Sep 10 19:16:22 2024","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":0.51900000000000002,"15min":0.42799999999999999},"min":{"1min":0.255,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":1.2030000000000001,"5min":3.629,"15min":3.629},"last":3.6880000000000002},{"interface":"front","average":{"1min":0.55100000000000005,"5min":0.51100000000000001,"15min":0.438},"min":{"1min":0.27600000000000002,"5min":0.248,"15min":0.223},"max":{"1min":1.8280000000000001,"5min":2.1880000000000002,"15min":2.1880000000000002},"last":3.5419999999999998}]},{"osd":6,"last update":"Tue Sep 10 19:16:53 2024","interfaces":[{"interface":"back","average":{"1min":0.79000000000000004,"5min":0.70399999999999996,"15min":0.51300000000000001},"min":{"1min":0.27400000000000002,"5min":0.253,"15min":0.253},"max":{"1min":3.766,"5min":3.964,"15min":3.964},"last":3.5750000000000002},{"interface":"front","average":{"1min":0.68300000000000005,"5min":0.63800000000000001,"15min":0.49199999999999999},"min":{"1min":0.30199999999999999,"5min":0.25,"15min":0.25},"max":{"1min":2.3460000000000001,"5min":3.0129999999999999,"15min":3.0129999999999999},"last":3.722}]},{"osd":7,"last update":"Tue Sep 10 19:17:08 2024","interfaces":[{"interface":"back","average":{"1min":1.113,"5min":1.05,"15min":0.69799999999999995},"min":{"1min":0.28100000000000003,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":4.7889999999999997,"5min":23.081,"15min":23.081},"last":3.6499999999999999},{"interface":"front","average":{"1min":1.3680000000000001,"5min":1.093,"15min":0.69699999999999995},"min":{"1min":0.29099999999999998,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":6.3620000000000001,"5min":22.666,"15min":22.666},"last":3.8380000000000001}]}]},{"osd":3,"up_from":26,"seq":111669149804,"num_pgs":78,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2928604,"kb_used_data":2618764,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":90816548,"statfs":{"total":95995035648,"available":92996145152,"internally_reserved":0,"allocated":2681614336,"data_stored":2681145798,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 19:16:35 2024","interfaces":[{"interface":"back","average":{"1min":0.57199999999999995,"5min":1.3240000000000001,"15min":0.70599999999999996},"min":{"1min":0.223,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":3.9529999999999998,"5min":17.518000000000001,"15min":17.518000000000001},"last":0.20799999999999999},{"interface":"front","average":{"1min":0.91700000000000004,"5min":1.6060000000000001,"15min":0.80800000000000005},"min":{"1min":0.20599999999999999,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":5.1539999999999999,"5min":17.609000000000002,"15min":17.609000000000002},"last":0.224}]},{"osd":1,"last update":"Tue Sep 10 19:16:20 2024","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"min":{"1min":0.248,"5min":0.248,"15min":0.248},"max":{"1min":2.6059999999999999,"5min":2.6059999999999999,"15min":2.6059999999999999},"last":0.307},{"interface":"front","average":{"1min":1.569,"5min":1.569,"15min":1.569},"min":{"1min":0.27700000000000002,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":17.454999999999998,"5min":17.454999999999998,"15min":17.454999999999998},"last":0.255}]},{"osd":2,"last update":"Tue Sep 10 19:16:35 2024","interfaces":[{"interface":"back","average":{"1min":0.98199999999999998,"5min":1.04,"15min":0.63500000000000001},"min":{"1min":0.23200000000000001,"5min":0.186,"15min":0.186},"max":{"1min":5.3029999999999999,"5min":5.3029999999999999,"15min":5.3029999999999999},"last":0.36599999999999999},{"interface":"front","average":{"1min":0.52800000000000002,"5min":1.2929999999999999,"15min":0.70599999999999996},"min":{"1min":0.249,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":3.2879999999999998,"5min":17.806000000000001,"15min":17.806000000000001},"last":0.32500000000000001}]},{"osd":4,"last update":"Tue Sep 10 19:17:04 2024","interfaces":[{"interface":"back","average":{"1min":0.77700000000000002,"5min":1.321,"15min":0.80400000000000005},"min":{"1min":0.32300000000000001,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":3.0249999999999999,"5min":17.834,"15min":17.834},"last":0.63400000000000001},{"interface":"front","average":{"1min":0.872,"5min":1.675,"15min":0.92200000000000004},"min":{"1min":0.35899999999999999,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":3.2879999999999998,"5min":17.553000000000001,"15min":17.553000000000001},"last":0.28100000000000003}]},{"osd":5,"last update":"Tue Sep 10 19:16:20 2024","interfaces":[{"interface":"back","average":{"1min":1.3680000000000001,"5min":1.085,"15min":0.73999999999999999},"min":{"1min":0.38200000000000001,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":17.725000000000001,"5min":17.725000000000001,"15min":17.725000000000001},"last":0.42099999999999999},{"interface":"front","average":{"1min":1.7090000000000001,"5min":1.464,"15min":0.86199999999999999},"min":{"1min":0.435,"5min":0.22,"15min":0.22},"max":{"1min":17.582999999999998,"5min":17.582999999999998,"15min":17.582999999999998},"last":0.85999999999999999}]},{"osd":6,"last update":"Tue Sep 10 19:16:44 2024","interfaces":[{"interface":"back","average":{"1min":0.77700000000000002,"5min":1.373,"15min":0.83999999999999997},"min":{"1min":0.34100000000000003,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":3.2589999999999999,"5min":17.861000000000001,"15min":17.861000000000001},"last":0.82399999999999995},{"interface":"front","average":{"1min":0.89600000000000002,"5min":1.1439999999999999,"15min":0.752},"min":{"1min":0.37,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":4.7569999999999997,"5min":17.687000000000001,"15min":17.687000000000001},"last":0.49199999999999999}]},{"osd":7,"last update":"Tue Sep 10 19:16:10 2024","interfaces":[{"interface":"back","average":{"1min":1.627,"5min":1.204,"15min":0.78800000000000003},"min":{"1min":0.33300000000000002,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":18.093,"5min":18.093,"15min":18.093},"last":0.40699999999999997},{"interface":"front","average":{"1min":1.496,"5min":1.3120000000000001,"15min":0.82599999999999996},"min":{"1min":0.42599999999999999,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":17.780999999999999,"5min":17.780999999999999,"15min":17.780999999999999},"last":0.35599999999999998}]}]},{"osd":2,"up_from":0,"seq":77309411441,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":0,"seq":592705486871,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":8,"seq":34359738491,"num_pgs":72,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2039952,"kb_used_data":1738436,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":91705200,"statfs":{"total":95995035648,"available":93906124800,"internally_reserved":0,"allocated":1780158464,"data_stored":1779698048,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Sep 10 19:16:20 2024","interfaces":[{"interface":"back","average":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"min":{"1min":0.19,"5min":0.19,"15min":0.19},"max":{"1min":0.68400000000000005,"5min":0.68400000000000005,"15min":0.68400000000000005},"last":0.23300000000000001},{"interface":"front","average":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"min":{"1min":0.26100000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":1.0900000000000001,"5min":1.0900000000000001,"15min":1.0900000000000001},"last":0.45500000000000002}]},{"osd":2,"last update":"Tue Sep 10 19:16:20 2024","interfaces":[{"interface":"back","average":{"1min":0.98499999999999999,"5min":1.01,"15min":0.59299999999999997},"min":{"1min":0.26500000000000001,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":4.2839999999999998,"5min":6.4640000000000004,"15min":6.4640000000000004},"last":0.29499999999999998},{"interface":"front","average":{"1min":0.39800000000000002,"5min":0.98999999999999999,"15min":0.57899999999999996},"min":{"1min":0.25900000000000001,"5min":0.187,"15min":0.187},"max":{"1min":0.70899999999999996,"5min":9.9399999999999995,"15min":9.9399999999999995},"last":0.38100000000000001}]},{"osd":3,"last update":"Tue Sep 10 19:16:43 2024","interfaces":[{"interface":"back","average":{"1min":0.59399999999999997,"5min":1.0740000000000001,"15min":0.63800000000000001},"min":{"1min":0.29199999999999998,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":2.5049999999999999,"5min":6.5590000000000002,"15min":6.5590000000000002},"last":0.51600000000000001},{"interface":"front","average":{"1min":0.34200000000000003,"5min":1.26,"15min":0.68899999999999995},"min":{"1min":0.27200000000000002,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":0.42199999999999999,"5min":7.2140000000000004,"15min":7.2140000000000004},"last":0.50700000000000001}]},{"osd":4,"last update":"Tue Sep 10 19:16:59 2024","interfaces":[{"interface":"back","average":{"1min":0.58599999999999997,"5min":0.80100000000000005,"15min":0.63800000000000001},"min":{"1min":0.29099999999999998,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":1.391,"5min":5.8499999999999996,"15min":5.8499999999999996},"last":0.44600000000000001},{"interface":"front","average":{"1min":0.873,"5min":0.88300000000000001,"15min":0.66600000000000004},"min":{"1min":0.29299999999999998,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":5.2640000000000002,"5min":5.2640000000000002,"15min":5.2640000000000002},"last":2.4940000000000002}]},{"osd":5,"last update":"Tue Sep 10 19:16:20 2024","interfaces":[{"interface":"back","average":{"1min":0.56000000000000005,"5min":0.93799999999999994,"15min":0.70199999999999996},"min":{"1min":0.309,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":1.1870000000000001,"5min":5.3860000000000001,"15min":5.3860000000000001},"last":0.32500000000000001},{"interface":"front","average":{"1min":0.69899999999999995,"5min":0.90900000000000003,"15min":0.68600000000000005},"min":{"1min":0.41199999999999998,"5min":0.253,"15min":0.253},"max":{"1min":1.9610000000000001,"5min":5.9749999999999996,"15min":5.9749999999999996},"last":0.53600000000000003}]},{"osd":6,"last update":"Tue Sep 10 19:16:43 2024","interfaces":[{"interface":"back","average":{"1min":0.75700000000000001,"5min":0.98099999999999998,"15min":0.70199999999999996},"min":{"1min":0.32700000000000001,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":3.0819999999999999,"5min":5.8730000000000002,"15min":5.8730000000000002},"last":0.498},{"interface":"front","average":{"1min":0.73899999999999999,"5min":0.98699999999999999,"15min":0.71699999999999997},"min":{"1min":0.38500000000000001,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":2.125,"5min":5.2850000000000001,"15min":5.2850000000000001},"last":0.42799999999999999}]},{"osd":7,"last update":"Tue Sep 10 19:17:09 2024","interfaces":[{"interface":"back","average":{"1min":0.95199999999999996,"5min":1.034,"15min":0.72699999999999998},"min":{"1min":0.40600000000000003,"5min":0.318,"15min":0.318},"max":{"1min":5.1479999999999997,"5min":5.1479999999999997,"15min":5.1479999999999997},"last":0.48399999999999999},{"interface":"front","average":{"1min":1.2230000000000001,"5min":1.056,"15min":0.72299999999999998},"min":{"1min":0.36699999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":4.9900000000000002,"5min":6.0469999999999997,"15min":6.0469999999999997},"last":0.56599999999999995}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":966,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1236,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1236,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":362,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":1760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":2118,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1018,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":2118,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1540,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1942,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":2162,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1806368768,"data_stored":1806368768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":2723872768,"data_stored":2723872768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":2697134080,"data_stored":2697134080,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":2711420928,"data_stored":2711420928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":2342256640,"data_stored":2342256640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":2186543104,"data_stored":2186543104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T19:17:14.298 INFO:tasks.cephadm.ceph_manager.ceph:increase pool size by 10 2024-09-10T19:17:14.298 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph osd pool set unique_pool_1 pg_num 26 2024-09-10T19:17:14.372 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:17:14.407 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:17:14.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:17:14.408 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:17:14.509 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:17:14.542 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:17:14.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:17:14.543 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:17:14.643 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:17:14.679 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:17:14.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:17:14.680 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:17:14.781 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:17:14.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:17:14.818 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:17:14.818 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:17:14.827 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: 50 16 79686 79670 99.5733 78.3125 0.00453382 0.0100379 2024-09-10T19:17:14.918 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:17:14.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:17:14.954 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:17:14.954 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:17:15.022 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:17:15.055 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:17:15.078 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:17:15.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:17:15.079 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:17:15.133 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:17:14 smithi183 ceph-mon[29736]: pgmap v296: 148 pgs: 1 active+recovering+undersized+remapped, 147 active+clean; 4.5 GiB data, 15 GiB used, 521 GiB / 536 GiB avail; 853 B/s rd, 65 MiB/s wr, 1.03k op/s; 3425/221124 objects misplaced (1.549%); 2.0 MiB/s, 32 objects/s recovering 2024-09-10T19:17:15.134 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:17:14 smithi183 ceph-mon[29736]: 5.3 scrub starts 2024-09-10T19:17:15.134 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:17:14 smithi183 ceph-mon[29736]: osdmap e172: 8 total, 8 up, 6 in 2024-09-10T19:17:15.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:17:14 smithi183 ceph-mon[41562]: pgmap v296: 148 pgs: 1 active+recovering+undersized+remapped, 147 active+clean; 4.5 GiB data, 15 GiB used, 521 GiB / 536 GiB avail; 853 B/s rd, 65 MiB/s wr, 1.03k op/s; 3425/221124 objects misplaced (1.549%); 2.0 MiB/s, 32 objects/s recovering 2024-09-10T19:17:15.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:17:14 smithi183 ceph-mon[41562]: 5.3 scrub starts 2024-09-10T19:17:15.134 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:17:14 smithi183 ceph-mon[41562]: osdmap e172: 8 total, 8 up, 6 in 2024-09-10T19:17:15.180 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:17:15.211 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:17:15.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:17:15.212 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:17:15.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:17:14 smithi191 ceph-mon[32884]: pgmap v296: 148 pgs: 1 active+recovering+undersized+remapped, 147 active+clean; 4.5 GiB data, 15 GiB used, 521 GiB / 536 GiB avail; 853 B/s rd, 65 MiB/s wr, 1.03k op/s; 3425/221124 objects misplaced (1.549%); 2.0 MiB/s, 32 objects/s recovering 2024-09-10T19:17:15.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:17:14 smithi191 ceph-mon[32884]: 5.3 scrub starts 2024-09-10T19:17:15.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:17:14 smithi191 ceph-mon[32884]: osdmap e172: 8 total, 8 up, 6 in 2024-09-10T19:17:15.313 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:17:15.350 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:17:15.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:17:15.351 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:17:15.452 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:17:15.479 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:17:15.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:17:15.480 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:17:15.581 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:17:15.631 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:17:15.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:17:15.632 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:17:15.733 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:17:15.761 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:17:15.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:17:15.762 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:17:15.827 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: 51 15 80948 80933 99.1686 78.9375 0.009551 0.0100796 2024-09-10T19:17:15.863 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:17:15.893 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:17:15.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:17:15.894 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:17:15.995 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:17:16.023 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:17:16.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:17:16.024 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:17:16.078 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:17:16.125 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:17:16.159 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:17:16.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:17:16.160 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:17:16.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:17:15 smithi183 ceph-mon[29736]: 4.1a scrub starts 2024-09-10T19:17:16.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:17:15 smithi183 ceph-mon[29736]: 4.1f scrub starts 2024-09-10T19:17:16.181 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:17:15 smithi183 ceph-mon[41562]: 4.1a scrub starts 2024-09-10T19:17:16.181 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:17:15 smithi183 ceph-mon[41562]: 4.1f scrub starts 2024-09-10T19:17:16.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:17:15 smithi191 ceph-mon[32884]: 4.1a scrub starts 2024-09-10T19:17:16.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:17:15 smithi191 ceph-mon[32884]: 4.1f scrub starts 2024-09-10T19:17:16.261 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:17:16.285 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:17:16.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:17:16.286 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:17:16.387 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:17:16.426 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:17:16.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:17:16.427 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:17:16.527 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:17:16.556 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:17:16.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:17:16.558 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:17:16.658 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:17:16.701 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:17:16.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:17:16.702 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:17:16.803 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:17:16.827 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: 52 16 82237 82221 98.8095 80.5 0.00817865 0.0101158 2024-09-10T19:17:16.833 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:17:16.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:17:16.834 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:17:16.935 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:17:16.966 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:17:16.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:17:16.966 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:17:17.067 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:17:17.097 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:17:17.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:17:17.099 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:17:17.099 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:17:16 smithi183 ceph-mon[29736]: purged_snaps scrub starts 2024-09-10T19:17:17.099 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:17:16 smithi183 ceph-mon[29736]: purged_snaps scrub ok 2024-09-10T19:17:17.099 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:17:16 smithi183 ceph-mon[29736]: pgmap v298: 148 pgs: 1 active+recovering+undersized+remapped, 147 active+clean; 4.9 GiB data, 16 GiB used, 520 GiB / 536 GiB avail; 1.5 KiB/s rd, 113 MiB/s wr, 1.81k op/s; 3233/239892 objects misplaced (1.348%); 3.5 MiB/s, 56 objects/s recovering 2024-09-10T19:17:17.099 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:17:16 smithi183 ceph-mon[29736]: osdmap e173: 8 total, 8 up, 6 in 2024-09-10T19:17:17.100 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:17:16 smithi183 ceph-mon[29736]: 2.10 scrub starts 2024-09-10T19:17:17.100 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:17:16 smithi183 ceph-mon[29736]: 2.10 scrub ok 2024-09-10T19:17:17.100 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:17:16 smithi183 ceph-mon[41562]: purged_snaps scrub starts 2024-09-10T19:17:17.100 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:17:16 smithi183 ceph-mon[41562]: purged_snaps scrub ok 2024-09-10T19:17:17.100 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:17:16 smithi183 ceph-mon[41562]: pgmap v298: 148 pgs: 1 active+recovering+undersized+remapped, 147 active+clean; 4.9 GiB data, 16 GiB used, 520 GiB / 536 GiB avail; 1.5 KiB/s rd, 113 MiB/s wr, 1.81k op/s; 3233/239892 objects misplaced (1.348%); 3.5 MiB/s, 56 objects/s recovering 2024-09-10T19:17:17.100 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:17:16 smithi183 ceph-mon[41562]: osdmap e173: 8 total, 8 up, 6 in 2024-09-10T19:17:17.100 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:17:16 smithi183 ceph-mon[41562]: 2.10 scrub starts 2024-09-10T19:17:17.100 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:17:16 smithi183 ceph-mon[41562]: 2.10 scrub ok 2024-09-10T19:17:17.127 INFO:teuthology.orchestra.run.smithi183.stdout:{ 2024-09-10T19:17:17.127 INFO:teuthology.orchestra.run.smithi183.stdout: "ops": [], 2024-09-10T19:17:17.127 INFO:teuthology.orchestra.run.smithi183.stdout: "num_ops": 0 2024-09-10T19:17:17.127 INFO:teuthology.orchestra.run.smithi183.stdout:} 2024-09-10T19:17:17.200 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:17:17.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:17:16 smithi191 ceph-mon[32884]: purged_snaps scrub starts 2024-09-10T19:17:17.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:17:16 smithi191 ceph-mon[32884]: purged_snaps scrub ok 2024-09-10T19:17:17.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:17:16 smithi191 ceph-mon[32884]: pgmap v298: 148 pgs: 1 active+recovering+undersized+remapped, 147 active+clean; 4.9 GiB data, 16 GiB used, 520 GiB / 536 GiB avail; 1.5 KiB/s rd, 113 MiB/s wr, 1.81k op/s; 3233/239892 objects misplaced (1.348%); 3.5 MiB/s, 56 objects/s recovering 2024-09-10T19:17:17.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:17:16 smithi191 ceph-mon[32884]: osdmap e173: 8 total, 8 up, 6 in 2024-09-10T19:17:17.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:17:16 smithi191 ceph-mon[32884]: 2.10 scrub starts 2024-09-10T19:17:17.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:17:16 smithi191 ceph-mon[32884]: 2.10 scrub ok 2024-09-10T19:17:17.233 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:17:17.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:17:17.234 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:17:17.335 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:17:17.365 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:17:17.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:17:17.367 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:17:17.467 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:17:17.490 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:17:17.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:17:17.491 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:17:17.591 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:17:17.613 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-10T19:17:17.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:17:17.627 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:17:17.627 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:17:17.715 INFO:journalctl@ceph.mgr.x.smithi191.stdout:Sep 10 19:17:17 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-x[34967]: ::ffff:172.21.15.191 - - [10/Sep/2024:19:17:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T19:17:17.728 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:17:17.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:17:17.764 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:17:17.764 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:17:17.827 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: 53 16 83448 83432 98.3731 75.6875 0.00788755 0.0101609 2024-09-10T19:17:17.865 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:17:17.905 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:17:17.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:17:17.906 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:17:17.931 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:17:17 smithi183 ceph-mon[29736]: 2.11 scrub starts 2024-09-10T19:17:17.931 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:17:17 smithi183 ceph-mon[41562]: 2.11 scrub starts 2024-09-10T19:17:18.007 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:17:18.051 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:17:18.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:17:18.052 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:17:18.153 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:17:18.181 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:17:18.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:17:18.182 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:17:18.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:17:17 smithi191 ceph-mon[32884]: 2.11 scrub starts 2024-09-10T19:17:18.283 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:17:18.319 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:17:18.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:17:18.320 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:17:18.421 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:17:18.447 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:17:18.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:17:18.448 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:17:18.549 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:17:18.570 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:17:18.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:17:18.571 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:17:18.672 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:17:18.696 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:17:18.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:17:18.697 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:17:18.798 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:17:18.827 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: 54 16 84544 84528 97.8198 68.5 0.0130962 0.0102176 2024-09-10T19:17:18.835 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:17:18.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:17:18.837 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:17:18.937 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:17:18.973 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:17:18.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:17:18.974 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:17:19.075 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:17:19.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:17:19.118 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:17:19.119 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:17:19.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:17:18 smithi183 ceph-mon[29736]: pgmap v300: 148 pgs: 1 active+recovering+undersized+remapped, 147 active+clean; 4.9 GiB data, 16 GiB used, 520 GiB / 536 GiB avail; 650 B/s rd, 78 MiB/s wr, 1.24k op/s; 3233/239892 objects misplaced (1.348%); 3.1 MiB/s, 49 objects/s recovering 2024-09-10T19:17:19.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:17:18 smithi183 ceph-mon[29736]: 6.18 scrub starts 2024-09-10T19:17:19.181 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:17:18 smithi183 ceph-mon[41562]: pgmap v300: 148 pgs: 1 active+recovering+undersized+remapped, 147 active+clean; 4.9 GiB data, 16 GiB used, 520 GiB / 536 GiB avail; 650 B/s rd, 78 MiB/s wr, 1.24k op/s; 3233/239892 objects misplaced (1.348%); 3.1 MiB/s, 49 objects/s recovering 2024-09-10T19:17:19.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:17:18 smithi183 ceph-mon[41562]: 6.18 scrub starts 2024-09-10T19:17:19.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:17:18 smithi191 ceph-mon[32884]: pgmap v300: 148 pgs: 1 active+recovering+undersized+remapped, 147 active+clean; 4.9 GiB data, 16 GiB used, 520 GiB / 536 GiB avail; 650 B/s rd, 78 MiB/s wr, 1.24k op/s; 3233/239892 objects misplaced (1.348%); 3.1 MiB/s, 49 objects/s recovering 2024-09-10T19:17:19.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:17:18 smithi191 ceph-mon[32884]: 6.18 scrub starts 2024-09-10T19:17:19.220 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:17:19.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:17:19.251 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:17:19.251 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:17:19.352 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:17:19.380 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:17:19.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:17:19.381 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:17:19.482 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:17:19.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:17:19.512 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:17:19.512 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:17:19.613 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:17:19.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:17:19.641 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:17:19.641 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:17:19.715 INFO:journalctl@ceph.iscsi.iscsi.a.smithi191.stdout:Sep 10 19:17:19 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-iscsi-iscsi-a[61816]: debug there is no tcmu-runner data available 2024-09-10T19:17:19.743 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:17:19.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:17:19.776 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:17:19.777 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:17:19.827 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: 55 16 85610 85594 97.2526 66.625 0.012208 0.0102779 2024-09-10T19:17:19.878 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:17:19.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:17:19.927 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:17:19.927 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:17:20.028 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:17:20.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:17:20.070 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:17:20.071 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:17:20.171 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:17:20.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:17:19 smithi183 ceph-mon[29736]: 5.e scrub starts 2024-09-10T19:17:20.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:17:19 smithi183 ceph-mon[29736]: 4.6 scrub starts 2024-09-10T19:17:20.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:17:19 smithi183 ceph-mon[29736]: 5.2 scrub starts 2024-09-10T19:17:20.182 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:17:19 smithi183 ceph-mon[29736]: osdmap e174: 8 total, 8 up, 6 in 2024-09-10T19:17:20.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:17:19 smithi183 ceph-mon[41562]: 5.e scrub starts 2024-09-10T19:17:20.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:17:19 smithi183 ceph-mon[41562]: 4.6 scrub starts 2024-09-10T19:17:20.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:17:19 smithi183 ceph-mon[41562]: 5.2 scrub starts 2024-09-10T19:17:20.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:17:19 smithi183 ceph-mon[41562]: osdmap e174: 8 total, 8 up, 6 in 2024-09-10T19:17:20.205 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:17:20.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:17:20.206 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:17:20.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:17:19 smithi191 ceph-mon[32884]: 5.e scrub starts 2024-09-10T19:17:20.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:17:19 smithi191 ceph-mon[32884]: 4.6 scrub starts 2024-09-10T19:17:20.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:17:19 smithi191 ceph-mon[32884]: 5.2 scrub starts 2024-09-10T19:17:20.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:17:19 smithi191 ceph-mon[32884]: osdmap e174: 8 total, 8 up, 6 in 2024-09-10T19:17:20.307 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:17:20.349 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:17:20.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:17:20.350 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:17:20.451 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:17:20.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:17:20.475 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:17:20.475 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:17:20.576 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:17:20.607 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:17:20.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:17:20.609 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:17:20.710 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:17:20.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:17:20.744 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:17:20.744 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:17:20.828 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: 56 16 86760 86744 96.7993 71.875 0.0131747 0.0103263 2024-09-10T19:17:20.845 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:17:20.885 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:17:20.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:17:20.886 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:17:20.987 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:17:21.014 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:17:21.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:17:21.016 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:17:21.107 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:17:20 smithi183 ceph-mon[29736]: from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:17:21.108 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:17:20 smithi183 ceph-mon[29736]: pgmap v301: 148 pgs: 1 active+recovering+undersized+remapped, 147 active+clean; 4.9 GiB data, 16 GiB used, 520 GiB / 536 GiB avail; 639 B/s rd, 52 MiB/s wr, 831 op/s; 3233/241059 objects misplaced (1.341%); 1.5 MiB/s, 23 objects/s recovering 2024-09-10T19:17:21.108 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:17:20 smithi183 ceph-mon[29736]: 4.b scrub starts 2024-09-10T19:17:21.108 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:17:20 smithi183 ceph-mon[41562]: from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:17:21.108 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:17:20 smithi183 ceph-mon[41562]: pgmap v301: 148 pgs: 1 active+recovering+undersized+remapped, 147 active+clean; 4.9 GiB data, 16 GiB used, 520 GiB / 536 GiB avail; 639 B/s rd, 52 MiB/s wr, 831 op/s; 3233/241059 objects misplaced (1.341%); 1.5 MiB/s, 23 objects/s recovering 2024-09-10T19:17:21.108 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:17:20 smithi183 ceph-mon[41562]: 4.b scrub starts 2024-09-10T19:17:21.117 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:17:21.150 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:17:21.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:17:21.151 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:17:21.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:17:20 smithi191 ceph-mon[32884]: from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:17:21.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:17:20 smithi191 ceph-mon[32884]: pgmap v301: 148 pgs: 1 active+recovering+undersized+remapped, 147 active+clean; 4.9 GiB data, 16 GiB used, 520 GiB / 536 GiB avail; 639 B/s rd, 52 MiB/s wr, 831 op/s; 3233/241059 objects misplaced (1.341%); 1.5 MiB/s, 23 objects/s recovering 2024-09-10T19:17:21.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:17:20 smithi191 ceph-mon[32884]: 4.b scrub starts 2024-09-10T19:17:21.252 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:17:21.291 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:17:21.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:17:21.292 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:17:21.393 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:17:21.429 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:17:21.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:17:21.430 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:17:21.431 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Sep 10 19:17:21 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-y[29989]: ::ffff:172.21.15.191 - - [10/Sep/2024:19:17:21] "GET /metrics HTTP/1.1" 200 224196 "" "Prometheus/2.43.0" 2024-09-10T19:17:21.531 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:17:21.564 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:17:21.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:17:21.565 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:17:21.666 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:17:21.704 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:17:21.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:17:21.705 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:17:21.806 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:17:21.828 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: 57 15 88004 87989 96.4661 77.8125 0.0178595 0.0103618 2024-09-10T19:17:21.835 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:17:21.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:17:21.836 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:17:21.937 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:17:21.965 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:17:21.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:17:21.966 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:17:22.067 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:17:22.104 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:17:22.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:17:22.105 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:17:22.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:17:21 smithi183 ceph-mon[29736]: 4.c scrub starts 2024-09-10T19:17:22.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:17:21 smithi183 ceph-mon[41562]: 4.c scrub starts 2024-09-10T19:17:22.205 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:17:22.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:17:21 smithi191 ceph-mon[32884]: 4.c scrub starts 2024-09-10T19:17:22.233 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:17:22.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:17:22.234 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:17:22.335 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:17:22.374 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:17:22.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:17:22.375 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:17:22.475 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:17:22.510 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:17:22.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:17:22.511 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:17:22.612 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:17:22.646 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:17:22.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:17:22.647 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:17:22.748 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:17:22.776 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:17:22.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:17:22.777 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:17:22.828 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: 58 16 89267 89251 96.1627 78.875 0.00642457 0.0103944 2024-09-10T19:17:22.878 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:17:22.917 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:17:22.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:17:22.918 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:17:23.018 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:17:23.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:17:23.054 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:17:23.054 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:17:23.155 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:17:23.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:17:22 smithi183 ceph-mon[29736]: pgmap v303: 148 pgs: 1 active+recovering+undersized+remapped, 147 active+clean; 5.2 GiB data, 17 GiB used, 519 GiB / 536 GiB avail; 1.4 KiB/s rd, 105 MiB/s wr, 1.69k op/s; 3026/257667 objects misplaced (1.174%); 3.6 MiB/s, 57 objects/s recovering 2024-09-10T19:17:23.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:17:22 smithi183 ceph-mon[29736]: 4.18 scrub starts 2024-09-10T19:17:23.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:17:22 smithi183 ceph-mon[29736]: 6.1 scrub starts 2024-09-10T19:17:23.181 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:17:22 smithi183 ceph-mon[41562]: pgmap v303: 148 pgs: 1 active+recovering+undersized+remapped, 147 active+clean; 5.2 GiB data, 17 GiB used, 519 GiB / 536 GiB avail; 1.4 KiB/s rd, 105 MiB/s wr, 1.69k op/s; 3026/257667 objects misplaced (1.174%); 3.6 MiB/s, 57 objects/s recovering 2024-09-10T19:17:23.181 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:17:22 smithi183 ceph-mon[41562]: 4.18 scrub starts 2024-09-10T19:17:23.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:17:22 smithi183 ceph-mon[41562]: 6.1 scrub starts 2024-09-10T19:17:23.183 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:17:23.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:17:23.184 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:17:23.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:17:22 smithi191 ceph-mon[32884]: pgmap v303: 148 pgs: 1 active+recovering+undersized+remapped, 147 active+clean; 5.2 GiB data, 17 GiB used, 519 GiB / 536 GiB avail; 1.4 KiB/s rd, 105 MiB/s wr, 1.69k op/s; 3026/257667 objects misplaced (1.174%); 3.6 MiB/s, 57 objects/s recovering 2024-09-10T19:17:23.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:17:22 smithi191 ceph-mon[32884]: 4.18 scrub starts 2024-09-10T19:17:23.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:17:22 smithi191 ceph-mon[32884]: 6.1 scrub starts 2024-09-10T19:17:23.285 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph daemon osd.1 dump_blocked_ops 2024-09-10T19:17:23.286 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:17:23.310 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:17:23.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:17:23.311 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:17:23.412 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:17:23.452 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:17:23.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:17:23.453 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:17:23.554 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:17:23.587 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:17:23.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:17:23.588 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:17:23.689 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:17:23.726 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:17:23.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:17:23.727 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:17:23.828 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:17:23.829 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: 59 16 90542 90526 95.8834 79.6875 0.00415633 0.0104246 2024-09-10T19:17:23.870 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:17:23.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:17:23.871 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:17:23.926 INFO:teuthology.orchestra.run.smithi183.stderr:nodeep-scrub is unset 2024-09-10T19:17:23.973 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:17:24.006 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:17:24.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:17:24.007 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:17:24.108 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:17:24.145 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:17:24.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:17:24.146 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:17:24.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:17:23 smithi183 ceph-mon[29736]: 5.18 scrub starts 2024-09-10T19:17:24.192 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:17:23 smithi183 ceph-mon[29736]: 5.8 scrub starts 2024-09-10T19:17:24.193 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:17:23 smithi183 ceph-mon[29736]: from='client.? 172.21.15.183:0/3712491340' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-10T19:17:24.193 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:17:23 smithi183 ceph-mon[29736]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-10T19:17:24.193 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:17:23 smithi183 ceph-mon[41562]: 5.18 scrub starts 2024-09-10T19:17:24.193 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:17:23 smithi183 ceph-mon[41562]: 5.8 scrub starts 2024-09-10T19:17:24.193 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:17:23 smithi183 ceph-mon[41562]: from='client.? 172.21.15.183:0/3712491340' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-10T19:17:24.193 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:17:23 smithi183 ceph-mon[41562]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-10T19:17:24.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:17:23 smithi191 ceph-mon[32884]: 5.18 scrub starts 2024-09-10T19:17:24.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:17:23 smithi191 ceph-mon[32884]: 5.8 scrub starts 2024-09-10T19:17:24.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:17:23 smithi191 ceph-mon[32884]: from='client.? 172.21.15.183:0/3712491340' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-10T19:17:24.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:17:23 smithi191 ceph-mon[32884]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-10T19:17:24.247 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:17:24.286 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:17:24.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:17:24.287 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:17:24.353 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:17:24.388 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:17:24.424 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:17:24.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:17:24.425 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:17:24.526 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:17:24.564 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:17:24.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:17:24.565 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:17:24.666 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:17:24.700 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:17:24.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:17:24.701 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:17:24.760 INFO:teuthology.orchestra.run.smithi183.stderr:set pool 8 pg_num to 26 2024-09-10T19:17:24.802 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:17:24.828 INFO:tasks.radosbench.radosbench.0.smithi183.stdout:2024-09-10T19:17:24.827546+0000 min lat: 0.0011643 max lat: 2.77997 avg lat: 0.0104666 2024-09-10T19:17:24.828 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-10T19:17:24.828 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: 60 16 91705 91689 95.4967 72.6875 0.0156583 0.0104666 2024-09-10T19:17:24.841 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:17:24.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:17:24.842 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:17:24.943 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:17:24.980 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:17:24.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:17:24.981 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:17:25.033 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:17:25.082 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:17:25.111 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:17:25.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:17:25.112 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:17:25.114 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:17:24 smithi183 ceph-mon[29736]: pgmap v304: 148 pgs: 1 active+recovering+undersized+remapped, 147 active+clean; 5.2 GiB data, 17 GiB used, 519 GiB / 536 GiB avail; 639 B/s rd, 46 MiB/s wr, 741 op/s; 3026/257667 objects misplaced (1.174%); 1.6 MiB/s, 25 objects/s recovering 2024-09-10T19:17:25.114 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:17:24 smithi183 ceph-mon[29736]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-10T19:17:25.114 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:17:24 smithi183 ceph-mon[29736]: Cluster is now healthy 2024-09-10T19:17:25.114 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:17:24 smithi183 ceph-mon[29736]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-10T19:17:25.115 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:17:24 smithi183 ceph-mon[29736]: osdmap e175: 8 total, 8 up, 6 in 2024-09-10T19:17:25.115 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:17:24 smithi183 ceph-mon[29736]: 6.1a scrub starts 2024-09-10T19:17:25.115 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:17:24 smithi183 ceph-mon[29736]: from='client.? 172.21.15.183:0/703270543' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pg_num", "val": "26"}]: dispatch 2024-09-10T19:17:25.115 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:17:24 smithi183 ceph-mon[29736]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pg_num", "val": "26"}]: dispatch 2024-09-10T19:17:25.115 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:17:24 smithi183 ceph-mon[29736]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pg_num", "val": "26"}]': finished 2024-09-10T19:17:25.115 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:17:24 smithi183 ceph-mon[29736]: osdmap e176: 8 total, 8 up, 6 in 2024-09-10T19:17:25.115 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:17:24 smithi183 ceph-mon[41562]: pgmap v304: 148 pgs: 1 active+recovering+undersized+remapped, 147 active+clean; 5.2 GiB data, 17 GiB used, 519 GiB / 536 GiB avail; 639 B/s rd, 46 MiB/s wr, 741 op/s; 3026/257667 objects misplaced (1.174%); 1.6 MiB/s, 25 objects/s recovering 2024-09-10T19:17:25.115 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:17:24 smithi183 ceph-mon[41562]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-10T19:17:25.115 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:17:24 smithi183 ceph-mon[41562]: Cluster is now healthy 2024-09-10T19:17:25.115 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:17:24 smithi183 ceph-mon[41562]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-10T19:17:25.115 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:17:24 smithi183 ceph-mon[41562]: osdmap e175: 8 total, 8 up, 6 in 2024-09-10T19:17:25.115 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:17:24 smithi183 ceph-mon[41562]: 6.1a scrub starts 2024-09-10T19:17:25.116 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:17:24 smithi183 ceph-mon[41562]: from='client.? 172.21.15.183:0/703270543' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pg_num", "val": "26"}]: dispatch 2024-09-10T19:17:25.116 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:17:24 smithi183 ceph-mon[41562]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pg_num", "val": "26"}]: dispatch 2024-09-10T19:17:25.116 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:17:24 smithi183 ceph-mon[41562]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pg_num", "val": "26"}]': finished 2024-09-10T19:17:25.116 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:17:24 smithi183 ceph-mon[41562]: osdmap e176: 8 total, 8 up, 6 in 2024-09-10T19:17:25.212 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:17:25.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:17:24 smithi191 ceph-mon[32884]: pgmap v304: 148 pgs: 1 active+recovering+undersized+remapped, 147 active+clean; 5.2 GiB data, 17 GiB used, 519 GiB / 536 GiB avail; 639 B/s rd, 46 MiB/s wr, 741 op/s; 3026/257667 objects misplaced (1.174%); 1.6 MiB/s, 25 objects/s recovering 2024-09-10T19:17:25.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:17:24 smithi191 ceph-mon[32884]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-10T19:17:25.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:17:24 smithi191 ceph-mon[32884]: Cluster is now healthy 2024-09-10T19:17:25.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:17:24 smithi191 ceph-mon[32884]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-10T19:17:25.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:17:24 smithi191 ceph-mon[32884]: osdmap e175: 8 total, 8 up, 6 in 2024-09-10T19:17:25.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:17:24 smithi191 ceph-mon[32884]: 6.1a scrub starts 2024-09-10T19:17:25.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:17:24 smithi191 ceph-mon[32884]: from='client.? 172.21.15.183:0/703270543' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pg_num", "val": "26"}]: dispatch 2024-09-10T19:17:25.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:17:24 smithi191 ceph-mon[32884]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pg_num", "val": "26"}]: dispatch 2024-09-10T19:17:25.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:17:24 smithi191 ceph-mon[32884]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pg_num", "val": "26"}]': finished 2024-09-10T19:17:25.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:17:24 smithi191 ceph-mon[32884]: osdmap e176: 8 total, 8 up, 6 in 2024-09-10T19:17:25.256 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:17:25.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:17:25.257 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:17:25.358 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:17:25.399 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:17:25.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:17:25.400 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:17:25.500 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:17:25.532 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:17:25.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:17:25.533 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:17:25.634 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:17:25.662 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:17:25.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:17:25.664 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:17:25.764 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:17:25.801 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:17:25.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:17:25.802 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:17:25.828 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: 61 16 92961 92945 95.218 78.5 0.0104991 0.0104982 2024-09-10T19:17:25.902 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:17:25.936 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:17:25.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:17:25.937 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:17:26.020 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:17:25 smithi183 ceph-mon[29736]: 5.5 scrub starts 2024-09-10T19:17:26.020 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:17:25 smithi183 ceph-mon[29736]: 5.5 scrub ok 2024-09-10T19:17:26.020 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:17:25 smithi183 ceph-mon[29736]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pg_num_actual", "val": "26"}]: dispatch 2024-09-10T19:17:26.020 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:17:25 smithi183 ceph-mon[29736]: from='mgr.14556 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pg_num_actual", "val": "26"}]: dispatch 2024-09-10T19:17:26.020 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:17:25 smithi183 ceph-mon[41562]: 5.5 scrub starts 2024-09-10T19:17:26.021 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:17:25 smithi183 ceph-mon[41562]: 5.5 scrub ok 2024-09-10T19:17:26.021 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:17:25 smithi183 ceph-mon[41562]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pg_num_actual", "val": "26"}]: dispatch 2024-09-10T19:17:26.021 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:17:25 smithi183 ceph-mon[41562]: from='mgr.14556 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pg_num_actual", "val": "26"}]: dispatch 2024-09-10T19:17:26.038 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:17:26.059 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:17:26.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:17:26.060 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:17:26.161 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:17:26.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:17:26.193 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:17:26.194 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:17:26.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:17:25 smithi191 ceph-mon[32884]: 5.5 scrub starts 2024-09-10T19:17:26.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:17:25 smithi191 ceph-mon[32884]: 5.5 scrub ok 2024-09-10T19:17:26.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:17:25 smithi191 ceph-mon[32884]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pg_num_actual", "val": "26"}]: dispatch 2024-09-10T19:17:26.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:17:25 smithi191 ceph-mon[32884]: from='mgr.14556 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pg_num_actual", "val": "26"}]: dispatch 2024-09-10T19:17:26.295 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:17:26.316 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:17:26.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:17:26.317 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:17:26.418 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:17:26.441 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:17:26.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:17:26.442 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:17:26.543 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:17:26.568 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:17:26.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:17:26.569 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:17:26.670 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:17:26.695 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:17:26.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:17:26.696 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:17:26.797 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:17:26.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:17:26.824 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:17:26.824 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:17:26.824 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph osd set noscrub 2024-09-10T19:17:26.828 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: 62 16 93098 93082 93.8204 8.5625 0.0298865 0.0105106 2024-09-10T19:17:26.924 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:17:26.959 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:17:26.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:17:26.961 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:17:27.061 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:17:27.085 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:17:27.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:17:27.086 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:17:27.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:17:26 smithi183 ceph-mon[29736]: pgmap v307: 148 pgs: 1 active+recovering+undersized+remapped, 147 active+clean; 5.6 GiB data, 18 GiB used, 518 GiB / 536 GiB avail; 1.7 KiB/s rd, 121 MiB/s wr, 1.94k op/s; 2827/275955 objects misplaced (1.024%); 4.2 MiB/s, 67 objects/s recovering 2024-09-10T19:17:27.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:17:26 smithi183 ceph-mon[29736]: from='mgr.14556 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pg_num_actual", "val": "26"}]': finished 2024-09-10T19:17:27.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:17:26 smithi183 ceph-mon[29736]: osdmap e177: 8 total, 8 up, 6 in 2024-09-10T19:17:27.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:17:26 smithi183 ceph-mon[29736]: 2.6 deep-scrub starts 2024-09-10T19:17:27.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:17:26 smithi183 ceph-mon[29736]: 2.6 deep-scrub ok 2024-09-10T19:17:27.181 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:17:26 smithi183 ceph-mon[41562]: pgmap v307: 148 pgs: 1 active+recovering+undersized+remapped, 147 active+clean; 5.6 GiB data, 18 GiB used, 518 GiB / 536 GiB avail; 1.7 KiB/s rd, 121 MiB/s wr, 1.94k op/s; 2827/275955 objects misplaced (1.024%); 4.2 MiB/s, 67 objects/s recovering 2024-09-10T19:17:27.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:17:26 smithi183 ceph-mon[41562]: from='mgr.14556 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pg_num_actual", "val": "26"}]': finished 2024-09-10T19:17:27.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:17:26 smithi183 ceph-mon[41562]: osdmap e177: 8 total, 8 up, 6 in 2024-09-10T19:17:27.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:17:26 smithi183 ceph-mon[41562]: 2.6 deep-scrub starts 2024-09-10T19:17:27.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:17:26 smithi183 ceph-mon[41562]: 2.6 deep-scrub ok 2024-09-10T19:17:27.186 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:17:27.209 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:17:27.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:17:27.210 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:17:27.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:17:26 smithi191 ceph-mon[32884]: pgmap v307: 148 pgs: 1 active+recovering+undersized+remapped, 147 active+clean; 5.6 GiB data, 18 GiB used, 518 GiB / 536 GiB avail; 1.7 KiB/s rd, 121 MiB/s wr, 1.94k op/s; 2827/275955 objects misplaced (1.024%); 4.2 MiB/s, 67 objects/s recovering 2024-09-10T19:17:27.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:17:26 smithi191 ceph-mon[32884]: from='mgr.14556 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pg_num_actual", "val": "26"}]': finished 2024-09-10T19:17:27.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:17:26 smithi191 ceph-mon[32884]: osdmap e177: 8 total, 8 up, 6 in 2024-09-10T19:17:27.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:17:26 smithi191 ceph-mon[32884]: 2.6 deep-scrub starts 2024-09-10T19:17:27.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:17:26 smithi191 ceph-mon[32884]: 2.6 deep-scrub ok 2024-09-10T19:17:27.311 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:17:27.332 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:17:27.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:17:27.333 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:17:27.434 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:17:27.455 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:17:27.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:17:27.456 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:17:27.556 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:17:27.579 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:17:27.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:17:27.580 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:17:27.680 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:17:27.702 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:17:27.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:17:27.703 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:17:27.715 INFO:journalctl@ceph.mgr.x.smithi191.stdout:Sep 10 19:17:27 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-x[34967]: ::ffff:172.21.15.191 - - [10/Sep/2024:19:17:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T19:17:27.804 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:17:27.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:17:27.826 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:17:27.826 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:17:27.828 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: 63 16 93350 93334 92.5812 15.75 0.0019447 0.0106469 2024-09-10T19:17:27.926 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:17:27.955 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:17:27.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:17:27.956 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:17:28.057 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:17:28.079 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:17:28.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:17:28.080 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:17:28.181 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:17:28.182 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:17:27 smithi183 ceph-mon[29736]: 6.15 scrub starts 2024-09-10T19:17:28.182 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:17:27 smithi183 ceph-mon[29736]: 6.15 scrub ok 2024-09-10T19:17:28.182 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:17:27 smithi183 ceph-mon[29736]: osdmap e178: 8 total, 8 up, 6 in 2024-09-10T19:17:28.182 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:17:27 smithi183 ceph-mon[29736]: 4.5 scrub starts 2024-09-10T19:17:28.182 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:17:27 smithi183 ceph-mon[29736]: 4.5 scrub ok 2024-09-10T19:17:28.182 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:17:27 smithi183 ceph-mon[29736]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pgp_num_actual", "val": "26"}]: dispatch 2024-09-10T19:17:28.183 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:17:27 smithi183 ceph-mon[29736]: from='mgr.14556 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pgp_num_actual", "val": "26"}]: dispatch 2024-09-10T19:17:28.183 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:17:27 smithi183 ceph-mon[41562]: 6.15 scrub starts 2024-09-10T19:17:28.183 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:17:27 smithi183 ceph-mon[41562]: 6.15 scrub ok 2024-09-10T19:17:28.183 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:17:27 smithi183 ceph-mon[41562]: osdmap e178: 8 total, 8 up, 6 in 2024-09-10T19:17:28.183 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:17:27 smithi183 ceph-mon[41562]: 4.5 scrub starts 2024-09-10T19:17:28.183 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:17:27 smithi183 ceph-mon[41562]: 4.5 scrub ok 2024-09-10T19:17:28.183 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:17:27 smithi183 ceph-mon[41562]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pgp_num_actual", "val": "26"}]: dispatch 2024-09-10T19:17:28.183 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:17:27 smithi183 ceph-mon[41562]: from='mgr.14556 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pgp_num_actual", "val": "26"}]: dispatch 2024-09-10T19:17:28.202 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:17:28.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:17:28.203 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:17:28.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:17:27 smithi191 ceph-mon[32884]: 6.15 scrub starts 2024-09-10T19:17:28.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:17:27 smithi191 ceph-mon[32884]: 6.15 scrub ok 2024-09-10T19:17:28.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:17:27 smithi191 ceph-mon[32884]: osdmap e178: 8 total, 8 up, 6 in 2024-09-10T19:17:28.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:17:27 smithi191 ceph-mon[32884]: 4.5 scrub starts 2024-09-10T19:17:28.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:17:27 smithi191 ceph-mon[32884]: 4.5 scrub ok 2024-09-10T19:17:28.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:17:27 smithi191 ceph-mon[32884]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pgp_num_actual", "val": "26"}]: dispatch 2024-09-10T19:17:28.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:17:27 smithi191 ceph-mon[32884]: from='mgr.14556 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pgp_num_actual", "val": "26"}]: dispatch 2024-09-10T19:17:28.305 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:17:28.326 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:17:28.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:17:28.326 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:17:28.427 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:17:28.448 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:17:28.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:17:28.449 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:17:28.550 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:17:28.571 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:17:28.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:17:28.572 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:17:28.673 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:17:28.694 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:17:28.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:17:28.695 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:17:28.796 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:17:28.817 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:17:28.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:17:28.818 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:17:28.828 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: 64 16 93379 93363 91.1629 1.8125 0.00247645 0.0108088 2024-09-10T19:17:28.919 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:17:28.940 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:17:28.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:17:28.941 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:17:29.042 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:17:29.062 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:17:29.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:17:29.063 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:17:29.164 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:17:29.185 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:17:29.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:17:29.186 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:17:29.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:17:28 smithi191 ceph-mon[32884]: pgmap v310: 158 pgs: 10 unknown, 1 active+recovering+undersized+remapped, 147 active+clean; 5.6 GiB data, 18 GiB used, 518 GiB / 536 GiB avail; 1.2 KiB/s rd, 95 MiB/s wr, 1.52k op/s; 2827/275955 objects misplaced (1.024%); 3.1 MiB/s, 49 objects/s recovering 2024-09-10T19:17:29.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:17:28 smithi191 ceph-mon[32884]: from='mgr.14556 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pgp_num_actual", "val": "26"}]': finished 2024-09-10T19:17:29.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:17:28 smithi191 ceph-mon[32884]: osdmap e179: 8 total, 8 up, 6 in 2024-09-10T19:17:29.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:17:28 smithi191 ceph-mon[32884]: 5.7 scrub starts 2024-09-10T19:17:29.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:17:28 smithi191 ceph-mon[32884]: 5.7 scrub ok 2024-09-10T19:17:29.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:17:28 smithi191 ceph-mon[32884]: purged_snaps scrub starts 2024-09-10T19:17:29.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:17:28 smithi191 ceph-mon[32884]: purged_snaps scrub ok 2024-09-10T19:17:29.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:17:28 smithi191 ceph-mon[32884]: 2.4 scrub starts 2024-09-10T19:17:29.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:17:28 smithi191 ceph-mon[32884]: 2.4 scrub ok 2024-09-10T19:17:29.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:17:28 smithi191 ceph-mon[32884]: from='mgr.14556 ' entity='mgr.y' 2024-09-10T19:17:29.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:17:28 smithi191 ceph-mon[32884]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:17:29.253 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:17:29.263 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:17:28 smithi183 ceph-mon[29736]: pgmap v310: 158 pgs: 10 unknown, 1 active+recovering+undersized+remapped, 147 active+clean; 5.6 GiB data, 18 GiB used, 518 GiB / 536 GiB avail; 1.2 KiB/s rd, 95 MiB/s wr, 1.52k op/s; 2827/275955 objects misplaced (1.024%); 3.1 MiB/s, 49 objects/s recovering 2024-09-10T19:17:29.263 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:17:28 smithi183 ceph-mon[29736]: from='mgr.14556 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pgp_num_actual", "val": "26"}]': finished 2024-09-10T19:17:29.263 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:17:28 smithi183 ceph-mon[29736]: osdmap e179: 8 total, 8 up, 6 in 2024-09-10T19:17:29.263 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:17:28 smithi183 ceph-mon[29736]: 5.7 scrub starts 2024-09-10T19:17:29.263 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:17:28 smithi183 ceph-mon[29736]: 5.7 scrub ok 2024-09-10T19:17:29.263 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:17:28 smithi183 ceph-mon[29736]: purged_snaps scrub starts 2024-09-10T19:17:29.263 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:17:28 smithi183 ceph-mon[29736]: purged_snaps scrub ok 2024-09-10T19:17:29.263 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:17:28 smithi183 ceph-mon[29736]: 2.4 scrub starts 2024-09-10T19:17:29.263 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:17:28 smithi183 ceph-mon[29736]: 2.4 scrub ok 2024-09-10T19:17:29.264 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:17:28 smithi183 ceph-mon[29736]: from='mgr.14556 ' entity='mgr.y' 2024-09-10T19:17:29.264 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:17:28 smithi183 ceph-mon[29736]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:17:29.264 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:17:28 smithi183 ceph-mon[41562]: pgmap v310: 158 pgs: 10 unknown, 1 active+recovering+undersized+remapped, 147 active+clean; 5.6 GiB data, 18 GiB used, 518 GiB / 536 GiB avail; 1.2 KiB/s rd, 95 MiB/s wr, 1.52k op/s; 2827/275955 objects misplaced (1.024%); 3.1 MiB/s, 49 objects/s recovering 2024-09-10T19:17:29.264 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:17:28 smithi183 ceph-mon[41562]: from='mgr.14556 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_1", "var": "pgp_num_actual", "val": "26"}]': finished 2024-09-10T19:17:29.264 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:17:28 smithi183 ceph-mon[41562]: osdmap e179: 8 total, 8 up, 6 in 2024-09-10T19:17:29.264 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:17:28 smithi183 ceph-mon[41562]: 5.7 scrub starts 2024-09-10T19:17:29.264 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:17:28 smithi183 ceph-mon[41562]: 5.7 scrub ok 2024-09-10T19:17:29.264 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:17:28 smithi183 ceph-mon[41562]: purged_snaps scrub starts 2024-09-10T19:17:29.264 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:17:28 smithi183 ceph-mon[41562]: purged_snaps scrub ok 2024-09-10T19:17:29.264 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:17:28 smithi183 ceph-mon[41562]: 2.4 scrub starts 2024-09-10T19:17:29.264 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:17:28 smithi183 ceph-mon[41562]: 2.4 scrub ok 2024-09-10T19:17:29.264 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:17:28 smithi183 ceph-mon[41562]: from='mgr.14556 ' entity='mgr.y' 2024-09-10T19:17:29.265 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:17:28 smithi183 ceph-mon[41562]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:17:29.287 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:17:29.308 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:17:29.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:17:29.309 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:17:29.410 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:17:29.432 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:17:29.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:17:29.433 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:17:29.533 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:17:29.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:17:29.556 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:17:29.556 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:17:29.657 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:17:29.678 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:17:29.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:17:29.679 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:17:29.715 INFO:journalctl@ceph.iscsi.iscsi.a.smithi191.stdout:Sep 10 19:17:29 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-iscsi-iscsi-a[61816]: debug there is no tcmu-runner data available 2024-09-10T19:17:29.780 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:17:29.801 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:17:29.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:17:29.802 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:17:29.828 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: 65 16 93379 93363 89.7605 0 - 0.0108088 2024-09-10T19:17:29.903 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:17:29.925 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:17:29.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:17:29.926 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:17:30.027 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:17:30.031 INFO:teuthology.orchestra.run.smithi183.stdout:osd.0: {} 2024-09-10T19:17:30.032 INFO:teuthology.orchestra.run.smithi183.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-10T19:17:30.032 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:17:29 smithi183 ceph-mon[29736]: 4.1 deep-scrub starts 2024-09-10T19:17:30.032 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:17:29 smithi183 ceph-mon[29736]: osdmap e180: 8 total, 8 up, 6 in 2024-09-10T19:17:30.032 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:17:29 smithi183 ceph-mon[29736]: 8.c deep-scrub starts 2024-09-10T19:17:30.032 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:17:29 smithi183 ceph-mon[41562]: 4.1 deep-scrub starts 2024-09-10T19:17:30.032 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:17:29 smithi183 ceph-mon[41562]: osdmap e180: 8 total, 8 up, 6 in 2024-09-10T19:17:30.033 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:17:29 smithi183 ceph-mon[41562]: 8.c deep-scrub starts 2024-09-10T19:17:30.058 INFO:teuthology.orchestra.run.smithi183.stdout:osd.1: {} 2024-09-10T19:17:30.058 INFO:teuthology.orchestra.run.smithi183.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-10T19:17:30.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:17:30.065 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:17:30.065 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:17:30.079 INFO:teuthology.orchestra.run.smithi183.stdout:osd.2: {} 2024-09-10T19:17:30.080 INFO:teuthology.orchestra.run.smithi183.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-10T19:17:30.101 INFO:teuthology.orchestra.run.smithi183.stdout:osd.3: {} 2024-09-10T19:17:30.101 INFO:teuthology.orchestra.run.smithi183.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-10T19:17:30.131 INFO:teuthology.orchestra.run.smithi183.stdout:osd.4: {} 2024-09-10T19:17:30.131 INFO:teuthology.orchestra.run.smithi183.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-10T19:17:30.160 INFO:teuthology.orchestra.run.smithi183.stdout:osd.5: {} 2024-09-10T19:17:30.161 INFO:teuthology.orchestra.run.smithi183.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-10T19:17:30.165 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:17:30.188 INFO:teuthology.orchestra.run.smithi183.stdout:osd.6: {} 2024-09-10T19:17:30.188 INFO:teuthology.orchestra.run.smithi183.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-10T19:17:30.201 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:17:30.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:17:30.202 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:17:30.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:17:29 smithi191 ceph-mon[32884]: 4.1 deep-scrub starts 2024-09-10T19:17:30.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:17:29 smithi191 ceph-mon[32884]: osdmap e180: 8 total, 8 up, 6 in 2024-09-10T19:17:30.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:17:29 smithi191 ceph-mon[32884]: 8.c deep-scrub starts 2024-09-10T19:17:30.218 INFO:teuthology.orchestra.run.smithi183.stdout:osd.7: {} 2024-09-10T19:17:30.218 INFO:teuthology.orchestra.run.smithi183.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-10T19:17:30.303 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:17:30.337 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:17:30.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:17:30.338 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:17:30.386 INFO:teuthology.orchestra.run.smithi183.stdout:ERROR: (22) Invalid argument 2024-09-10T19:17:30.386 INFO:teuthology.orchestra.run.smithi183.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-10T19:17:30.439 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:17:30.475 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:17:30.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:17:30.476 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:17:30.576 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:17:30.602 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:17:30.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:17:30.603 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:17:30.704 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:17:30.731 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:17:30.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:17:30.732 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:17:30.828 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: 66 16 94298 94282 89.2707 28.7188 0.00529031 0.0111911 2024-09-10T19:17:30.833 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:17:30.867 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:17:30.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:17:30.868 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:17:30.968 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:17:30.996 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:17:30.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:17:30.997 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:17:31.098 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:17:31.124 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:17:31.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:17:31.125 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:17:31.152 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:17:30 smithi183 ceph-mon[29736]: from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:17:31.153 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:17:30 smithi183 ceph-mon[29736]: pgmap v313: 158 pgs: 1 remapped+peering, 9 unknown, 1 active+recovering+undersized+remapped, 147 active+clean; 5.6 GiB data, 19 GiB used, 518 GiB / 536 GiB avail; 840 KiB/s wr, 13 op/s; 2827/276114 objects misplaced (1.024%); 16 KiB/s, 0 objects/s recovering 2024-09-10T19:17:31.153 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:17:30 smithi183 ceph-mon[29736]: 8.14 starting backfill to osd.0 from (0'0,0'0] MAX to 178'5733 2024-09-10T19:17:31.153 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:17:30 smithi183 ceph-mon[29736]: osdmap e181: 8 total, 8 up, 6 in 2024-09-10T19:17:31.153 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:17:30 smithi183 ceph-mon[29736]: 8.10 starting backfill to osd.7 from (0'0,0'0] MAX to 178'5837 2024-09-10T19:17:31.153 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:17:30 smithi183 ceph-mon[29736]: 8.18 starting backfill to osd.4 from (0'0,0'0] MAX to 178'5643 2024-09-10T19:17:31.153 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:17:30 smithi183 ceph-mon[29736]: 8.17 starting backfill to osd.0 from (0'0,0'0] MAX to 178'5804 2024-09-10T19:17:31.153 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:17:30 smithi183 ceph-mon[29736]: 8.13 starting backfill to osd.4 from (0'0,0'0] MAX to 176'6021 2024-09-10T19:17:31.153 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:17:30 smithi183 ceph-mon[29736]: 8.11 starting backfill to osd.6 from (0'0,0'0] MAX to 178'5844 2024-09-10T19:17:31.153 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:17:30 smithi183 ceph-mon[29736]: 8.19 starting backfill to osd.4 from (0'0,0'0] MAX to 178'5795 2024-09-10T19:17:31.153 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:17:30 smithi183 ceph-mon[29736]: 8.12 starting backfill to osd.6 from (0'0,0'0] MAX to 178'5848 2024-09-10T19:17:31.153 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:17:30 smithi183 ceph-mon[29736]: 8.15 starting backfill to osd.0 from (0'0,0'0] MAX to 178'5855 2024-09-10T19:17:31.153 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:17:30 smithi183 ceph-mon[29736]: 8.14 starting backfill to osd.7 from (0'0,0'0] MAX to 178'5733 2024-09-10T19:17:31.154 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:17:30 smithi183 ceph-mon[29736]: 8.16 starting backfill to osd.3 from (0'0,0'0] MAX to 178'5753 2024-09-10T19:17:31.154 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:17:30 smithi183 ceph-mon[29736]: 8.11 starting backfill to osd.7 from (0'0,0'0] MAX to 178'5844 2024-09-10T19:17:31.154 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:17:30 smithi183 ceph-mon[29736]: 8.12 starting backfill to osd.7 from (0'0,0'0] MAX to 178'5848 2024-09-10T19:17:31.154 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:17:30 smithi183 ceph-mon[29736]: 8.16 starting backfill to osd.5 from (0'0,0'0] MAX to 178'5753 2024-09-10T19:17:31.154 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:17:30 smithi183 ceph-mon[29736]: 8.17 starting backfill to osd.7 from (0'0,0'0] MAX to 178'5804 2024-09-10T19:17:31.154 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:17:30 smithi183 ceph-mon[29736]: from='client.? 172.21.15.183:0/1462900328' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T19:17:31.154 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:17:30 smithi183 ceph-mon[41562]: from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:17:31.154 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:17:30 smithi183 ceph-mon[41562]: pgmap v313: 158 pgs: 1 remapped+peering, 9 unknown, 1 active+recovering+undersized+remapped, 147 active+clean; 5.6 GiB data, 19 GiB used, 518 GiB / 536 GiB avail; 840 KiB/s wr, 13 op/s; 2827/276114 objects misplaced (1.024%); 16 KiB/s, 0 objects/s recovering 2024-09-10T19:17:31.154 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:17:30 smithi183 ceph-mon[41562]: 8.14 starting backfill to osd.0 from (0'0,0'0] MAX to 178'5733 2024-09-10T19:17:31.155 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:17:30 smithi183 ceph-mon[41562]: osdmap e181: 8 total, 8 up, 6 in 2024-09-10T19:17:31.155 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:17:30 smithi183 ceph-mon[41562]: 8.10 starting backfill to osd.7 from (0'0,0'0] MAX to 178'5837 2024-09-10T19:17:31.155 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:17:30 smithi183 ceph-mon[41562]: 8.18 starting backfill to osd.4 from (0'0,0'0] MAX to 178'5643 2024-09-10T19:17:31.155 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:17:30 smithi183 ceph-mon[41562]: 8.17 starting backfill to osd.0 from (0'0,0'0] MAX to 178'5804 2024-09-10T19:17:31.155 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:17:30 smithi183 ceph-mon[41562]: 8.13 starting backfill to osd.4 from (0'0,0'0] MAX to 176'6021 2024-09-10T19:17:31.155 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:17:30 smithi183 ceph-mon[41562]: 8.11 starting backfill to osd.6 from (0'0,0'0] MAX to 178'5844 2024-09-10T19:17:31.155 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:17:30 smithi183 ceph-mon[41562]: 8.19 starting backfill to osd.4 from (0'0,0'0] MAX to 178'5795 2024-09-10T19:17:31.155 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:17:30 smithi183 ceph-mon[41562]: 8.12 starting backfill to osd.6 from (0'0,0'0] MAX to 178'5848 2024-09-10T19:17:31.155 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:17:30 smithi183 ceph-mon[41562]: 8.15 starting backfill to osd.0 from (0'0,0'0] MAX to 178'5855 2024-09-10T19:17:31.155 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:17:30 smithi183 ceph-mon[41562]: 8.14 starting backfill to osd.7 from (0'0,0'0] MAX to 178'5733 2024-09-10T19:17:31.155 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:17:30 smithi183 ceph-mon[41562]: 8.16 starting backfill to osd.3 from (0'0,0'0] MAX to 178'5753 2024-09-10T19:17:31.155 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:17:30 smithi183 ceph-mon[41562]: 8.11 starting backfill to osd.7 from (0'0,0'0] MAX to 178'5844 2024-09-10T19:17:31.155 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:17:30 smithi183 ceph-mon[41562]: 8.12 starting backfill to osd.7 from (0'0,0'0] MAX to 178'5848 2024-09-10T19:17:31.155 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:17:30 smithi183 ceph-mon[41562]: 8.16 starting backfill to osd.5 from (0'0,0'0] MAX to 178'5753 2024-09-10T19:17:31.156 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:17:30 smithi183 ceph-mon[41562]: 8.17 starting backfill to osd.7 from (0'0,0'0] MAX to 178'5804 2024-09-10T19:17:31.156 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:17:30 smithi183 ceph-mon[41562]: from='client.? 172.21.15.183:0/1462900328' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T19:17:31.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:17:30 smithi191 ceph-mon[32884]: from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:17:31.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:17:30 smithi191 ceph-mon[32884]: pgmap v313: 158 pgs: 1 remapped+peering, 9 unknown, 1 active+recovering+undersized+remapped, 147 active+clean; 5.6 GiB data, 19 GiB used, 518 GiB / 536 GiB avail; 840 KiB/s wr, 13 op/s; 2827/276114 objects misplaced (1.024%); 16 KiB/s, 0 objects/s recovering 2024-09-10T19:17:31.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:17:30 smithi191 ceph-mon[32884]: 8.14 starting backfill to osd.0 from (0'0,0'0] MAX to 178'5733 2024-09-10T19:17:31.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:17:30 smithi191 ceph-mon[32884]: osdmap e181: 8 total, 8 up, 6 in 2024-09-10T19:17:31.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:17:30 smithi191 ceph-mon[32884]: 8.10 starting backfill to osd.7 from (0'0,0'0] MAX to 178'5837 2024-09-10T19:17:31.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:17:30 smithi191 ceph-mon[32884]: 8.18 starting backfill to osd.4 from (0'0,0'0] MAX to 178'5643 2024-09-10T19:17:31.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:17:30 smithi191 ceph-mon[32884]: 8.17 starting backfill to osd.0 from (0'0,0'0] MAX to 178'5804 2024-09-10T19:17:31.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:17:30 smithi191 ceph-mon[32884]: 8.13 starting backfill to osd.4 from (0'0,0'0] MAX to 176'6021 2024-09-10T19:17:31.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:17:30 smithi191 ceph-mon[32884]: 8.11 starting backfill to osd.6 from (0'0,0'0] MAX to 178'5844 2024-09-10T19:17:31.217 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:17:30 smithi191 ceph-mon[32884]: 8.19 starting backfill to osd.4 from (0'0,0'0] MAX to 178'5795 2024-09-10T19:17:31.217 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:17:30 smithi191 ceph-mon[32884]: 8.12 starting backfill to osd.6 from (0'0,0'0] MAX to 178'5848 2024-09-10T19:17:31.217 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:17:30 smithi191 ceph-mon[32884]: 8.15 starting backfill to osd.0 from (0'0,0'0] MAX to 178'5855 2024-09-10T19:17:31.217 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:17:30 smithi191 ceph-mon[32884]: 8.14 starting backfill to osd.7 from (0'0,0'0] MAX to 178'5733 2024-09-10T19:17:31.217 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:17:30 smithi191 ceph-mon[32884]: 8.16 starting backfill to osd.3 from (0'0,0'0] MAX to 178'5753 2024-09-10T19:17:31.217 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:17:30 smithi191 ceph-mon[32884]: 8.11 starting backfill to osd.7 from (0'0,0'0] MAX to 178'5844 2024-09-10T19:17:31.217 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:17:30 smithi191 ceph-mon[32884]: 8.12 starting backfill to osd.7 from (0'0,0'0] MAX to 178'5848 2024-09-10T19:17:31.217 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:17:30 smithi191 ceph-mon[32884]: 8.16 starting backfill to osd.5 from (0'0,0'0] MAX to 178'5753 2024-09-10T19:17:31.217 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:17:30 smithi191 ceph-mon[32884]: 8.17 starting backfill to osd.7 from (0'0,0'0] MAX to 178'5804 2024-09-10T19:17:31.217 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:17:30 smithi191 ceph-mon[32884]: from='client.? 172.21.15.183:0/1462900328' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T19:17:31.226 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:17:31.253 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:17:31.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:17:31.254 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:17:31.355 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:17:31.381 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:17:31.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:17:31.382 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:17:31.431 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Sep 10 19:17:31 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-y[29989]: ::ffff:172.21.15.191 - - [10/Sep/2024:19:17:31] "GET /metrics HTTP/1.1" 200 224228 "" "Prometheus/2.43.0" 2024-09-10T19:17:31.482 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:17:31.511 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:17:31.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:17:31.512 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:17:31.613 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:17:31.640 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:17:31.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:17:31.641 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:17:31.742 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:17:31.779 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:17:31.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:17:31.780 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:17:31.828 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: 67 16 95341 95325 88.9112 65.1875 0.00509949 0.0112415 2024-09-10T19:17:31.881 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:17:31.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:17:31.918 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:17:31.919 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:17:32.019 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:17:32.061 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:17:32.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:17:32.062 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:17:32.163 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:17:32.188 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:17:32.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:17:32.189 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:17:32.289 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:17:32.331 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:17:32.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:17:32.332 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:17:32.433 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:17:32.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:17:32.456 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:17:32.457 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:17:32.557 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:17:32.587 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:17:32.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:17:32.588 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:17:32.689 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:17:32.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:17:32.716 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:17:32.716 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:17:32.817 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:17:32.828 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: 68 16 96358 96342 88.5384 63.5625 0.149376 0.0112852 2024-09-10T19:17:32.863 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:17:32.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:17:32.864 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:17:32.964 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:17:33.000 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:17:33.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:17:33.001 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:17:33.101 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:17:33.137 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:17:33.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:17:33.138 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:17:33.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:17:32 smithi191 ceph-mon[32884]: pgmap v315: 158 pgs: 6 activating+remapped, 4 remapped+peering, 1 active+recovering, 147 active+clean; 5.7 GiB data, 19 GiB used, 518 GiB / 536 GiB avail; 1.0 KiB/s rd, 20 MiB/s wr, 328 op/s; 1406/280677 objects degraded (0.501%); 26077/280677 objects misplaced (9.291%); 1.8 MiB/s, 28 objects/s recovering 2024-09-10T19:17:33.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:17:32 smithi191 ceph-mon[32884]: 5.1f scrub starts 2024-09-10T19:17:33.239 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:17:33.273 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:17:33.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:17:33.274 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:17:33.307 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:17:32 smithi183 ceph-mon[29736]: pgmap v315: 158 pgs: 6 activating+remapped, 4 remapped+peering, 1 active+recovering, 147 active+clean; 5.7 GiB data, 19 GiB used, 518 GiB / 536 GiB avail; 1.0 KiB/s rd, 20 MiB/s wr, 328 op/s; 1406/280677 objects degraded (0.501%); 26077/280677 objects misplaced (9.291%); 1.8 MiB/s, 28 objects/s recovering 2024-09-10T19:17:33.307 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:17:32 smithi183 ceph-mon[29736]: 5.1f scrub starts 2024-09-10T19:17:33.307 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:17:32 smithi183 ceph-mon[41562]: pgmap v315: 158 pgs: 6 activating+remapped, 4 remapped+peering, 1 active+recovering, 147 active+clean; 5.7 GiB data, 19 GiB used, 518 GiB / 536 GiB avail; 1.0 KiB/s rd, 20 MiB/s wr, 328 op/s; 1406/280677 objects degraded (0.501%); 26077/280677 objects misplaced (9.291%); 1.8 MiB/s, 28 objects/s recovering 2024-09-10T19:17:33.307 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:17:32 smithi183 ceph-mon[41562]: 5.1f scrub starts 2024-09-10T19:17:33.375 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:17:33.415 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:17:33.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:17:33.416 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:17:33.517 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:17:33.547 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:17:33.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:17:33.548 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:17:33.648 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:17:33.679 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:17:33.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:17:33.681 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:17:33.781 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:17:33.815 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:17:33.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:17:33.817 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:17:33.829 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: 69 16 97528 97512 88.3149 73.125 0.00490549 0.0113159 2024-09-10T19:17:33.918 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:17:33.953 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:17:33.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:17:33.954 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:17:33.968 INFO:teuthology.orchestra.run.smithi183.stderr:noscrub is set 2024-09-10T19:17:34.019 INFO:tasks.thrashosds.thrasher:in_osds: [0, 3, 4, 5, 6, 7] out_osds: [1, 2] dead_osds: [] live_osds: [0, 2, 3, 4, 5, 6, 7, 1] 2024-09-10T19:17:34.019 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-10T19:17:34.019 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap 2024-09-10T19:17:34.019 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph osd dump -f json-pretty 2024-09-10T19:17:34.055 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:17:34.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:17:34.095 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:17:34.095 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:17:34.196 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:17:34.214 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:17:33 smithi183 ceph-mon[41562]: from='client.? 172.21.15.183:0/878805087' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-10T19:17:34.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:17:33 smithi191 ceph-mon[32884]: from='client.? 172.21.15.183:0/878805087' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-10T19:17:34.217 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:17:33 smithi183 ceph-mon[29736]: from='client.? 172.21.15.183:0/878805087' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-10T19:17:34.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:17:34.228 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:17:34.228 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:17:34.330 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:17:34.358 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:17:34.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:17:34.359 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:17:34.460 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:17:34.489 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:17:34.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:17:34.490 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:17:34.591 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:17:34.618 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:17:34.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:17:34.620 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:17:34.720 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:17:34.759 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:17:34.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:17:34.760 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:17:34.808 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph daemon osd.1 dump_historic_ops 2024-09-10T19:17:34.829 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: 70 16 98656 98640 88.0603 70.5 0.0123927 0.0113464 2024-09-10T19:17:34.860 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:17:34.897 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:17:34.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:17:34.899 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:17:34.999 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:17:35.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:17:35.037 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:17:35.037 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:17:35.138 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:17:35.174 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:17:35.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:17:35.175 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:17:35.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:17:34 smithi183 ceph-mon[29736]: pgmap v316: 158 pgs: 6 activating+remapped, 4 remapped+peering, 1 active+recovering, 147 active+clean; 5.7 GiB data, 19 GiB used, 518 GiB / 536 GiB avail; 853 B/s rd, 16 MiB/s wr, 263 op/s; 1406/280677 objects degraded (0.501%); 26077/280677 objects misplaced (9.291%); 1.4 MiB/s, 22 objects/s recovering 2024-09-10T19:17:35.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:17:34 smithi183 ceph-mon[29736]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T19:17:35.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:17:34 smithi183 ceph-mon[29736]: from='client.? 172.21.15.183:0/878805087' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-10T19:17:35.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:17:34 smithi183 ceph-mon[29736]: osdmap e182: 8 total, 8 up, 6 in 2024-09-10T19:17:35.181 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:17:34 smithi183 ceph-mon[41562]: pgmap v316: 158 pgs: 6 activating+remapped, 4 remapped+peering, 1 active+recovering, 147 active+clean; 5.7 GiB data, 19 GiB used, 518 GiB / 536 GiB avail; 853 B/s rd, 16 MiB/s wr, 263 op/s; 1406/280677 objects degraded (0.501%); 26077/280677 objects misplaced (9.291%); 1.4 MiB/s, 22 objects/s recovering 2024-09-10T19:17:35.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:17:34 smithi183 ceph-mon[41562]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T19:17:35.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:17:34 smithi183 ceph-mon[41562]: from='client.? 172.21.15.183:0/878805087' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-10T19:17:35.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:17:34 smithi183 ceph-mon[41562]: osdmap e182: 8 total, 8 up, 6 in 2024-09-10T19:17:35.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:17:34 smithi191 ceph-mon[32884]: pgmap v316: 158 pgs: 6 activating+remapped, 4 remapped+peering, 1 active+recovering, 147 active+clean; 5.7 GiB data, 19 GiB used, 518 GiB / 536 GiB avail; 853 B/s rd, 16 MiB/s wr, 263 op/s; 1406/280677 objects degraded (0.501%); 26077/280677 objects misplaced (9.291%); 1.4 MiB/s, 22 objects/s recovering 2024-09-10T19:17:35.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:17:34 smithi191 ceph-mon[32884]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T19:17:35.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:17:34 smithi191 ceph-mon[32884]: from='client.? 172.21.15.183:0/878805087' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-10T19:17:35.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:17:34 smithi191 ceph-mon[32884]: osdmap e182: 8 total, 8 up, 6 in 2024-09-10T19:17:35.277 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:17:35.295 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:17:35.326 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:17:35.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:17:35.328 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:17:35.331 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:17:35.429 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:17:35.467 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:17:35.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:17:35.468 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:17:35.569 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:17:35.606 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:17:35.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:17:35.607 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:17:35.708 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:17:35.747 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:17:35.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:17:35.748 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:17:35.829 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: 71 16 99802 99786 87.8288 71.625 0.0097378 0.0113805 2024-09-10T19:17:35.849 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:17:35.882 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:17:35.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:17:35.883 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:17:35.984 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:17:36.015 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:17:35 smithi183 ceph-mon[29736]: purged_snaps scrub starts 2024-09-10T19:17:36.015 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:17:35 smithi183 ceph-mon[29736]: purged_snaps scrub ok 2024-09-10T19:17:36.016 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:17:35 smithi183 ceph-mon[41562]: purged_snaps scrub starts 2024-09-10T19:17:36.016 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:17:35 smithi183 ceph-mon[41562]: purged_snaps scrub ok 2024-09-10T19:17:36.024 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:17:36.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:17:36.026 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:17:36.126 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:17:36.155 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:17:36.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:17:36.156 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:17:36.257 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:17:36.257 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-10T19:17:36.286 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:17:36.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:17:36.288 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:17:36.389 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:17:36.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:17:36.419 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:17:36.419 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:17:36.465 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:17:35 smithi191 ceph-mon[32884]: purged_snaps scrub starts 2024-09-10T19:17:36.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:17:35 smithi191 ceph-mon[32884]: purged_snaps scrub ok 2024-09-10T19:17:36.520 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:17:36.546 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:17:36.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:17:36.548 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:17:36.648 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:17:36.684 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:17:36.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:17:36.685 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:17:36.786 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:17:36.813 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:17:36.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:17:36.814 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:17:36.829 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: 72 16 100881 100865 87.5455 67.4375 0.0115742 0.0114175 2024-09-10T19:17:36.914 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:17:36.950 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:17:36.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:17:36.953 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:17:37.053 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:17:37.089 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:17:37.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:17:37.090 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:17:37.191 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:17:37.219 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:17:37.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:17:37.220 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:17:37.244 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-10T19:17:37.291 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:17:36 smithi183 ceph-mon[29736]: pgmap v318: 158 pgs: 1 active+recovering+remapped, 4 active+remapped+backfilling, 5 active+remapped+backfill_toofull, 1 active+recovering, 147 active+clean; 6.0 GiB data, 20 GiB used, 516 GiB / 536 GiB avail; 1.5 KiB/s rd, 64 MiB/s wr, 1.02k op/s; 41225/296832 objects misplaced (13.888%); 35 MiB/s, 567 objects/s recovering 2024-09-10T19:17:37.291 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:17:36 smithi183 ceph-mon[29736]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 5 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-09-10T19:17:37.291 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:17:36 smithi183 ceph-mon[29736]: osdmap e183: 8 total, 8 up, 6 in 2024-09-10T19:17:37.291 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:17:36 smithi183 ceph-mon[41562]: pgmap v318: 158 pgs: 1 active+recovering+remapped, 4 active+remapped+backfilling, 5 active+remapped+backfill_toofull, 1 active+recovering, 147 active+clean; 6.0 GiB data, 20 GiB used, 516 GiB / 536 GiB avail; 1.5 KiB/s rd, 64 MiB/s wr, 1.02k op/s; 41225/296832 objects misplaced (13.888%); 35 MiB/s, 567 objects/s recovering 2024-09-10T19:17:37.291 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:17:36 smithi183 ceph-mon[41562]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 5 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-09-10T19:17:37.291 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:17:36 smithi183 ceph-mon[41562]: osdmap e183: 8 total, 8 up, 6 in 2024-09-10T19:17:37.321 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:17:37.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:17:37.361 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:17:37.361 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:17:37.450 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:17:36 smithi191 ceph-mon[32884]: pgmap v318: 158 pgs: 1 active+recovering+remapped, 4 active+remapped+backfilling, 5 active+remapped+backfill_toofull, 1 active+recovering, 147 active+clean; 6.0 GiB data, 20 GiB used, 516 GiB / 536 GiB avail; 1.5 KiB/s rd, 64 MiB/s wr, 1.02k op/s; 41225/296832 objects misplaced (13.888%); 35 MiB/s, 567 objects/s recovering 2024-09-10T19:17:37.450 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:17:36 smithi191 ceph-mon[32884]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 5 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-09-10T19:17:37.451 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:17:36 smithi191 ceph-mon[32884]: osdmap e183: 8 total, 8 up, 6 in 2024-09-10T19:17:37.462 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:17:37.499 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:17:37.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:17:37.500 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:17:37.600 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:17:37.629 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:17:37.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:17:37.630 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:17:37.715 INFO:journalctl@ceph.mgr.x.smithi191.stdout:Sep 10 19:17:37 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-x[34967]: ::ffff:172.21.15.191 - - [10/Sep/2024:19:17:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T19:17:37.731 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:17:37.761 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:17:37.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:17:37.762 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:17:37.829 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: 73 16 101999 101983 87.3034 69.875 0.00715988 0.0114493 2024-09-10T19:17:37.863 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:17:37.905 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:17:37.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:17:37.906 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:17:38.007 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:17:38.035 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:17:38.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:17:38.036 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:17:38.137 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:17:38.164 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:17:38.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:17:38.165 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:17:38.266 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:17:38.294 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:17:38.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:17:38.295 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:17:38.396 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:17:38.432 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:17:38.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:17:38.433 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:17:38.534 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:17:38.570 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:17:38.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:17:38.571 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:17:38.672 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:17:38.701 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:17:38.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:17:38.702 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:17:38.803 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:17:38.829 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: 74 16 103161 103145 87.105 72.625 0.0111834 0.0114753 2024-09-10T19:17:38.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:17:38.846 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:17:38.846 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:17:38.947 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:17:38.977 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:17:38.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:17:38.978 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:17:39.028 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:17:38 smithi183 ceph-mon[29736]: pgmap v320: 158 pgs: 1 active+recovering+remapped, 4 active+remapped+backfilling, 5 active+remapped+backfill_toofull, 1 active+recovering, 147 active+clean; 6.0 GiB data, 20 GiB used, 516 GiB / 536 GiB avail; 657 B/s rd, 96 MiB/s wr, 1.54k op/s; 41225/296832 objects misplaced (13.888%); 34 MiB/s, 547 objects/s recovering 2024-09-10T19:17:39.028 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:17:38 smithi183 ceph-mon[41562]: pgmap v320: 158 pgs: 1 active+recovering+remapped, 4 active+remapped+backfilling, 5 active+remapped+backfill_toofull, 1 active+recovering, 147 active+clean; 6.0 GiB data, 20 GiB used, 516 GiB / 536 GiB avail; 657 B/s rd, 96 MiB/s wr, 1.54k op/s; 41225/296832 objects misplaced (13.888%); 34 MiB/s, 547 objects/s recovering 2024-09-10T19:17:39.079 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:17:39.116 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:17:39.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:17:39.117 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:17:39.217 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:17:39.248 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:17:39.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:17:39.250 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:17:39.256 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:17:38 smithi191 ceph-mon[32884]: pgmap v320: 158 pgs: 1 active+recovering+remapped, 4 active+remapped+backfilling, 5 active+remapped+backfill_toofull, 1 active+recovering, 147 active+clean; 6.0 GiB data, 20 GiB used, 516 GiB / 536 GiB avail; 657 B/s rd, 96 MiB/s wr, 1.54k op/s; 41225/296832 objects misplaced (13.888%); 34 MiB/s, 547 objects/s recovering 2024-09-10T19:17:39.350 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:17:39.376 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:17:39.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:17:39.377 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:17:39.478 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:17:39.513 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:17:39.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:17:39.514 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:17:39.517 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:17:39.540 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-09-10T19:17:39.541 INFO:teuthology.orchestra.run.smithi183.stdout:{ 2024-09-10T19:17:39.541 INFO:teuthology.orchestra.run.smithi183.stdout: "epoch": 183, 2024-09-10T19:17:39.541 INFO:teuthology.orchestra.run.smithi183.stdout: "fsid": "320f6df8-6fa7-11ef-bcea-c7b262605968", 2024-09-10T19:17:39.541 INFO:teuthology.orchestra.run.smithi183.stdout: "created": "2024-09-10T19:05:03.688531+0000", 2024-09-10T19:17:39.541 INFO:teuthology.orchestra.run.smithi183.stdout: "modified": "2024-09-10T19:17:35.958226+0000", 2024-09-10T19:17:39.541 INFO:teuthology.orchestra.run.smithi183.stdout: "last_up_change": "2024-09-10T19:15:19.923894+0000", 2024-09-10T19:17:39.541 INFO:teuthology.orchestra.run.smithi183.stdout: "last_in_change": "2024-09-10T19:16:23.815531+0000", 2024-09-10T19:17:39.541 INFO:teuthology.orchestra.run.smithi183.stdout: "flags": "noscrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-09-10T19:17:39.541 INFO:teuthology.orchestra.run.smithi183.stdout: "flags_num": 5801984, 2024-09-10T19:17:39.541 INFO:teuthology.orchestra.run.smithi183.stdout: "flags_set": [ 2024-09-10T19:17:39.541 INFO:teuthology.orchestra.run.smithi183.stdout: "noscrub", 2024-09-10T19:17:39.542 INFO:teuthology.orchestra.run.smithi183.stdout: "pglog_hardlimit", 2024-09-10T19:17:39.542 INFO:teuthology.orchestra.run.smithi183.stdout: "purged_snapdirs", 2024-09-10T19:17:39.542 INFO:teuthology.orchestra.run.smithi183.stdout: "recovery_deletes", 2024-09-10T19:17:39.542 INFO:teuthology.orchestra.run.smithi183.stdout: "sortbitwise" 2024-09-10T19:17:39.542 INFO:teuthology.orchestra.run.smithi183.stdout: ], 2024-09-10T19:17:39.542 INFO:teuthology.orchestra.run.smithi183.stdout: "crush_version": 19, 2024-09-10T19:17:39.542 INFO:teuthology.orchestra.run.smithi183.stdout: "full_ratio": 0.94999998807907104, 2024-09-10T19:17:39.542 INFO:teuthology.orchestra.run.smithi183.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-09-10T19:17:39.542 INFO:teuthology.orchestra.run.smithi183.stdout: "nearfull_ratio": 0.85000002384185791, 2024-09-10T19:17:39.542 INFO:teuthology.orchestra.run.smithi183.stdout: "cluster_snapshot": "", 2024-09-10T19:17:39.542 INFO:teuthology.orchestra.run.smithi183.stdout: "pool_max": 8, 2024-09-10T19:17:39.542 INFO:teuthology.orchestra.run.smithi183.stdout: "max_osd": 8, 2024-09-10T19:17:39.543 INFO:teuthology.orchestra.run.smithi183.stdout: "require_min_compat_client": "luminous", 2024-09-10T19:17:39.543 INFO:teuthology.orchestra.run.smithi183.stdout: "min_compat_client": "luminous", 2024-09-10T19:17:39.543 INFO:teuthology.orchestra.run.smithi183.stdout: "require_osd_release": "quincy", 2024-09-10T19:17:39.543 INFO:teuthology.orchestra.run.smithi183.stdout: "pools": [ 2024-09-10T19:17:39.543 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:17:39.543 INFO:teuthology.orchestra.run.smithi183.stdout: "pool": 1, 2024-09-10T19:17:39.543 INFO:teuthology.orchestra.run.smithi183.stdout: "pool_name": ".mgr", 2024-09-10T19:17:39.543 INFO:teuthology.orchestra.run.smithi183.stdout: "create_time": "2024-09-10T19:08:00.456786+0000", 2024-09-10T19:17:39.543 INFO:teuthology.orchestra.run.smithi183.stdout: "flags": 1, 2024-09-10T19:17:39.543 INFO:teuthology.orchestra.run.smithi183.stdout: "flags_names": "hashpspool", 2024-09-10T19:17:39.543 INFO:teuthology.orchestra.run.smithi183.stdout: "type": 1, 2024-09-10T19:17:39.543 INFO:teuthology.orchestra.run.smithi183.stdout: "size": 3, 2024-09-10T19:17:39.543 INFO:teuthology.orchestra.run.smithi183.stdout: "min_size": 2, 2024-09-10T19:17:39.544 INFO:teuthology.orchestra.run.smithi183.stdout: "crush_rule": 0, 2024-09-10T19:17:39.544 INFO:teuthology.orchestra.run.smithi183.stdout: "peering_crush_bucket_count": 0, 2024-09-10T19:17:39.544 INFO:teuthology.orchestra.run.smithi183.stdout: "peering_crush_bucket_target": 0, 2024-09-10T19:17:39.544 INFO:teuthology.orchestra.run.smithi183.stdout: "peering_crush_bucket_barrier": 0, 2024-09-10T19:17:39.544 INFO:teuthology.orchestra.run.smithi183.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-10T19:17:39.544 INFO:teuthology.orchestra.run.smithi183.stdout: "object_hash": 2, 2024-09-10T19:17:39.544 INFO:teuthology.orchestra.run.smithi183.stdout: "pg_autoscale_mode": "off", 2024-09-10T19:17:39.544 INFO:teuthology.orchestra.run.smithi183.stdout: "pg_num": 1, 2024-09-10T19:17:39.544 INFO:teuthology.orchestra.run.smithi183.stdout: "pg_placement_num": 1, 2024-09-10T19:17:39.544 INFO:teuthology.orchestra.run.smithi183.stdout: "pg_placement_num_target": 1, 2024-09-10T19:17:39.544 INFO:teuthology.orchestra.run.smithi183.stdout: "pg_num_target": 1, 2024-09-10T19:17:39.544 INFO:teuthology.orchestra.run.smithi183.stdout: "pg_num_pending": 1, 2024-09-10T19:17:39.545 INFO:teuthology.orchestra.run.smithi183.stdout: "last_pg_merge_meta": { 2024-09-10T19:17:39.545 INFO:teuthology.orchestra.run.smithi183.stdout: "source_pgid": "0.0", 2024-09-10T19:17:39.545 INFO:teuthology.orchestra.run.smithi183.stdout: "ready_epoch": 0, 2024-09-10T19:17:39.545 INFO:teuthology.orchestra.run.smithi183.stdout: "last_epoch_started": 0, 2024-09-10T19:17:39.545 INFO:teuthology.orchestra.run.smithi183.stdout: "last_epoch_clean": 0, 2024-09-10T19:17:39.545 INFO:teuthology.orchestra.run.smithi183.stdout: "source_version": "0'0", 2024-09-10T19:17:39.545 INFO:teuthology.orchestra.run.smithi183.stdout: "target_version": "0'0" 2024-09-10T19:17:39.545 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:17:39.545 INFO:teuthology.orchestra.run.smithi183.stdout: "last_change": "22", 2024-09-10T19:17:39.545 INFO:teuthology.orchestra.run.smithi183.stdout: "last_force_op_resend": "0", 2024-09-10T19:17:39.545 INFO:teuthology.orchestra.run.smithi183.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-10T19:17:39.545 INFO:teuthology.orchestra.run.smithi183.stdout: "last_force_op_resend_preluminous": "0", 2024-09-10T19:17:39.545 INFO:teuthology.orchestra.run.smithi183.stdout: "auid": 0, 2024-09-10T19:17:39.545 INFO:teuthology.orchestra.run.smithi183.stdout: "snap_mode": "selfmanaged", 2024-09-10T19:17:39.546 INFO:teuthology.orchestra.run.smithi183.stdout: "snap_seq": 0, 2024-09-10T19:17:39.546 INFO:teuthology.orchestra.run.smithi183.stdout: "snap_epoch": 0, 2024-09-10T19:17:39.546 INFO:teuthology.orchestra.run.smithi183.stdout: "pool_snaps": [], 2024-09-10T19:17:39.546 INFO:teuthology.orchestra.run.smithi183.stdout: "removed_snaps": "[]", 2024-09-10T19:17:39.546 INFO:teuthology.orchestra.run.smithi183.stdout: "quota_max_bytes": 0, 2024-09-10T19:17:39.546 INFO:teuthology.orchestra.run.smithi183.stdout: "quota_max_objects": 0, 2024-09-10T19:17:39.546 INFO:teuthology.orchestra.run.smithi183.stdout: "tiers": [], 2024-09-10T19:17:39.546 INFO:teuthology.orchestra.run.smithi183.stdout: "tier_of": -1, 2024-09-10T19:17:39.546 INFO:teuthology.orchestra.run.smithi183.stdout: "read_tier": -1, 2024-09-10T19:17:39.546 INFO:teuthology.orchestra.run.smithi183.stdout: "write_tier": -1, 2024-09-10T19:17:39.546 INFO:teuthology.orchestra.run.smithi183.stdout: "cache_mode": "none", 2024-09-10T19:17:39.546 INFO:teuthology.orchestra.run.smithi183.stdout: "target_max_bytes": 0, 2024-09-10T19:17:39.547 INFO:teuthology.orchestra.run.smithi183.stdout: "target_max_objects": 0, 2024-09-10T19:17:39.547 INFO:teuthology.orchestra.run.smithi183.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-10T19:17:39.547 INFO:teuthology.orchestra.run.smithi183.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-10T19:17:39.547 INFO:teuthology.orchestra.run.smithi183.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-10T19:17:39.547 INFO:teuthology.orchestra.run.smithi183.stdout: "cache_min_flush_age": 0, 2024-09-10T19:17:39.547 INFO:teuthology.orchestra.run.smithi183.stdout: "cache_min_evict_age": 0, 2024-09-10T19:17:39.547 INFO:teuthology.orchestra.run.smithi183.stdout: "erasure_code_profile": "", 2024-09-10T19:17:39.547 INFO:teuthology.orchestra.run.smithi183.stdout: "hit_set_params": { 2024-09-10T19:17:39.547 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "none" 2024-09-10T19:17:39.547 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:17:39.547 INFO:teuthology.orchestra.run.smithi183.stdout: "hit_set_period": 0, 2024-09-10T19:17:39.547 INFO:teuthology.orchestra.run.smithi183.stdout: "hit_set_count": 0, 2024-09-10T19:17:39.547 INFO:teuthology.orchestra.run.smithi183.stdout: "use_gmt_hitset": true, 2024-09-10T19:17:39.548 INFO:teuthology.orchestra.run.smithi183.stdout: "min_read_recency_for_promote": 0, 2024-09-10T19:17:39.548 INFO:teuthology.orchestra.run.smithi183.stdout: "min_write_recency_for_promote": 0, 2024-09-10T19:17:39.548 INFO:teuthology.orchestra.run.smithi183.stdout: "hit_set_grade_decay_rate": 0, 2024-09-10T19:17:39.548 INFO:teuthology.orchestra.run.smithi183.stdout: "hit_set_search_last_n": 0, 2024-09-10T19:17:39.548 INFO:teuthology.orchestra.run.smithi183.stdout: "grade_table": [], 2024-09-10T19:17:39.548 INFO:teuthology.orchestra.run.smithi183.stdout: "stripe_width": 0, 2024-09-10T19:17:39.548 INFO:teuthology.orchestra.run.smithi183.stdout: "expected_num_objects": 0, 2024-09-10T19:17:39.548 INFO:teuthology.orchestra.run.smithi183.stdout: "fast_read": false, 2024-09-10T19:17:39.548 INFO:teuthology.orchestra.run.smithi183.stdout: "options": { 2024-09-10T19:17:39.548 INFO:teuthology.orchestra.run.smithi183.stdout: "pg_num_max": 32, 2024-09-10T19:17:39.548 INFO:teuthology.orchestra.run.smithi183.stdout: "pg_num_min": 1 2024-09-10T19:17:39.548 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:17:39.548 INFO:teuthology.orchestra.run.smithi183.stdout: "application_metadata": { 2024-09-10T19:17:39.549 INFO:teuthology.orchestra.run.smithi183.stdout: "mgr": {} 2024-09-10T19:17:39.549 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:17:39.549 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:17:39.549 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:17:39.549 INFO:teuthology.orchestra.run.smithi183.stdout: "pool": 2, 2024-09-10T19:17:39.549 INFO:teuthology.orchestra.run.smithi183.stdout: "pool_name": ".rgw.root", 2024-09-10T19:17:39.549 INFO:teuthology.orchestra.run.smithi183.stdout: "create_time": "2024-09-10T19:10:05.993763+0000", 2024-09-10T19:17:39.549 INFO:teuthology.orchestra.run.smithi183.stdout: "flags": 1, 2024-09-10T19:17:39.549 INFO:teuthology.orchestra.run.smithi183.stdout: "flags_names": "hashpspool", 2024-09-10T19:17:39.549 INFO:teuthology.orchestra.run.smithi183.stdout: "type": 1, 2024-09-10T19:17:39.549 INFO:teuthology.orchestra.run.smithi183.stdout: "size": 3, 2024-09-10T19:17:39.549 INFO:teuthology.orchestra.run.smithi183.stdout: "min_size": 2, 2024-09-10T19:17:39.549 INFO:teuthology.orchestra.run.smithi183.stdout: "crush_rule": 0, 2024-09-10T19:17:39.550 INFO:teuthology.orchestra.run.smithi183.stdout: "peering_crush_bucket_count": 0, 2024-09-10T19:17:39.550 INFO:teuthology.orchestra.run.smithi183.stdout: "peering_crush_bucket_target": 0, 2024-09-10T19:17:39.550 INFO:teuthology.orchestra.run.smithi183.stdout: "peering_crush_bucket_barrier": 0, 2024-09-10T19:17:39.550 INFO:teuthology.orchestra.run.smithi183.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-10T19:17:39.550 INFO:teuthology.orchestra.run.smithi183.stdout: "object_hash": 2, 2024-09-10T19:17:39.550 INFO:teuthology.orchestra.run.smithi183.stdout: "pg_autoscale_mode": "off", 2024-09-10T19:17:39.550 INFO:teuthology.orchestra.run.smithi183.stdout: "pg_num": 32, 2024-09-10T19:17:39.550 INFO:teuthology.orchestra.run.smithi183.stdout: "pg_placement_num": 32, 2024-09-10T19:17:39.550 INFO:teuthology.orchestra.run.smithi183.stdout: "pg_placement_num_target": 32, 2024-09-10T19:17:39.550 INFO:teuthology.orchestra.run.smithi183.stdout: "pg_num_target": 32, 2024-09-10T19:17:39.550 INFO:teuthology.orchestra.run.smithi183.stdout: "pg_num_pending": 32, 2024-09-10T19:17:39.550 INFO:teuthology.orchestra.run.smithi183.stdout: "last_pg_merge_meta": { 2024-09-10T19:17:39.550 INFO:teuthology.orchestra.run.smithi183.stdout: "source_pgid": "0.0", 2024-09-10T19:17:39.550 INFO:teuthology.orchestra.run.smithi183.stdout: "ready_epoch": 0, 2024-09-10T19:17:39.551 INFO:teuthology.orchestra.run.smithi183.stdout: "last_epoch_started": 0, 2024-09-10T19:17:39.551 INFO:teuthology.orchestra.run.smithi183.stdout: "last_epoch_clean": 0, 2024-09-10T19:17:39.551 INFO:teuthology.orchestra.run.smithi183.stdout: "source_version": "0'0", 2024-09-10T19:17:39.551 INFO:teuthology.orchestra.run.smithi183.stdout: "target_version": "0'0" 2024-09-10T19:17:39.551 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:17:39.551 INFO:teuthology.orchestra.run.smithi183.stdout: "last_change": "60", 2024-09-10T19:17:39.551 INFO:teuthology.orchestra.run.smithi183.stdout: "last_force_op_resend": "0", 2024-09-10T19:17:39.551 INFO:teuthology.orchestra.run.smithi183.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-10T19:17:39.551 INFO:teuthology.orchestra.run.smithi183.stdout: "last_force_op_resend_preluminous": "0", 2024-09-10T19:17:39.551 INFO:teuthology.orchestra.run.smithi183.stdout: "auid": 0, 2024-09-10T19:17:39.551 INFO:teuthology.orchestra.run.smithi183.stdout: "snap_mode": "selfmanaged", 2024-09-10T19:17:39.551 INFO:teuthology.orchestra.run.smithi183.stdout: "snap_seq": 0, 2024-09-10T19:17:39.551 INFO:teuthology.orchestra.run.smithi183.stdout: "snap_epoch": 0, 2024-09-10T19:17:39.551 INFO:teuthology.orchestra.run.smithi183.stdout: "pool_snaps": [], 2024-09-10T19:17:39.552 INFO:teuthology.orchestra.run.smithi183.stdout: "removed_snaps": "[]", 2024-09-10T19:17:39.552 INFO:teuthology.orchestra.run.smithi183.stdout: "quota_max_bytes": 0, 2024-09-10T19:17:39.552 INFO:teuthology.orchestra.run.smithi183.stdout: "quota_max_objects": 0, 2024-09-10T19:17:39.552 INFO:teuthology.orchestra.run.smithi183.stdout: "tiers": [], 2024-09-10T19:17:39.552 INFO:teuthology.orchestra.run.smithi183.stdout: "tier_of": -1, 2024-09-10T19:17:39.552 INFO:teuthology.orchestra.run.smithi183.stdout: "read_tier": -1, 2024-09-10T19:17:39.552 INFO:teuthology.orchestra.run.smithi183.stdout: "write_tier": -1, 2024-09-10T19:17:39.552 INFO:teuthology.orchestra.run.smithi183.stdout: "cache_mode": "none", 2024-09-10T19:17:39.552 INFO:teuthology.orchestra.run.smithi183.stdout: "target_max_bytes": 0, 2024-09-10T19:17:39.552 INFO:teuthology.orchestra.run.smithi183.stdout: "target_max_objects": 0, 2024-09-10T19:17:39.552 INFO:teuthology.orchestra.run.smithi183.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-10T19:17:39.552 INFO:teuthology.orchestra.run.smithi183.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-10T19:17:39.552 INFO:teuthology.orchestra.run.smithi183.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-10T19:17:39.553 INFO:teuthology.orchestra.run.smithi183.stdout: "cache_min_flush_age": 0, 2024-09-10T19:17:39.553 INFO:teuthology.orchestra.run.smithi183.stdout: "cache_min_evict_age": 0, 2024-09-10T19:17:39.553 INFO:teuthology.orchestra.run.smithi183.stdout: "erasure_code_profile": "", 2024-09-10T19:17:39.553 INFO:teuthology.orchestra.run.smithi183.stdout: "hit_set_params": { 2024-09-10T19:17:39.553 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "none" 2024-09-10T19:17:39.553 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:17:39.553 INFO:teuthology.orchestra.run.smithi183.stdout: "hit_set_period": 0, 2024-09-10T19:17:39.553 INFO:teuthology.orchestra.run.smithi183.stdout: "hit_set_count": 0, 2024-09-10T19:17:39.553 INFO:teuthology.orchestra.run.smithi183.stdout: "use_gmt_hitset": true, 2024-09-10T19:17:39.553 INFO:teuthology.orchestra.run.smithi183.stdout: "min_read_recency_for_promote": 0, 2024-09-10T19:17:39.553 INFO:teuthology.orchestra.run.smithi183.stdout: "min_write_recency_for_promote": 0, 2024-09-10T19:17:39.553 INFO:teuthology.orchestra.run.smithi183.stdout: "hit_set_grade_decay_rate": 0, 2024-09-10T19:17:39.554 INFO:teuthology.orchestra.run.smithi183.stdout: "hit_set_search_last_n": 0, 2024-09-10T19:17:39.554 INFO:teuthology.orchestra.run.smithi183.stdout: "grade_table": [], 2024-09-10T19:17:39.554 INFO:teuthology.orchestra.run.smithi183.stdout: "stripe_width": 0, 2024-09-10T19:17:39.554 INFO:teuthology.orchestra.run.smithi183.stdout: "expected_num_objects": 0, 2024-09-10T19:17:39.554 INFO:teuthology.orchestra.run.smithi183.stdout: "fast_read": false, 2024-09-10T19:17:39.554 INFO:teuthology.orchestra.run.smithi183.stdout: "options": {}, 2024-09-10T19:17:39.554 INFO:teuthology.orchestra.run.smithi183.stdout: "application_metadata": { 2024-09-10T19:17:39.554 INFO:teuthology.orchestra.run.smithi183.stdout: "rgw": {} 2024-09-10T19:17:39.554 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:17:39.554 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:17:39.554 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:17:39.554 INFO:teuthology.orchestra.run.smithi183.stdout: "pool": 3, 2024-09-10T19:17:39.554 INFO:teuthology.orchestra.run.smithi183.stdout: "pool_name": "datapool", 2024-09-10T19:17:39.554 INFO:teuthology.orchestra.run.smithi183.stdout: "create_time": "2024-09-10T19:10:06.224267+0000", 2024-09-10T19:17:39.555 INFO:teuthology.orchestra.run.smithi183.stdout: "flags": 8193, 2024-09-10T19:17:39.555 INFO:teuthology.orchestra.run.smithi183.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-09-10T19:17:39.555 INFO:teuthology.orchestra.run.smithi183.stdout: "type": 1, 2024-09-10T19:17:39.555 INFO:teuthology.orchestra.run.smithi183.stdout: "size": 3, 2024-09-10T19:17:39.555 INFO:teuthology.orchestra.run.smithi183.stdout: "min_size": 2, 2024-09-10T19:17:39.555 INFO:teuthology.orchestra.run.smithi183.stdout: "crush_rule": 0, 2024-09-10T19:17:39.555 INFO:teuthology.orchestra.run.smithi183.stdout: "peering_crush_bucket_count": 0, 2024-09-10T19:17:39.555 INFO:teuthology.orchestra.run.smithi183.stdout: "peering_crush_bucket_target": 0, 2024-09-10T19:17:39.555 INFO:teuthology.orchestra.run.smithi183.stdout: "peering_crush_bucket_barrier": 0, 2024-09-10T19:17:39.555 INFO:teuthology.orchestra.run.smithi183.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-10T19:17:39.555 INFO:teuthology.orchestra.run.smithi183.stdout: "object_hash": 2, 2024-09-10T19:17:39.555 INFO:teuthology.orchestra.run.smithi183.stdout: "pg_autoscale_mode": "off", 2024-09-10T19:17:39.555 INFO:teuthology.orchestra.run.smithi183.stdout: "pg_num": 3, 2024-09-10T19:17:39.555 INFO:teuthology.orchestra.run.smithi183.stdout: "pg_placement_num": 3, 2024-09-10T19:17:39.556 INFO:teuthology.orchestra.run.smithi183.stdout: "pg_placement_num_target": 3, 2024-09-10T19:17:39.556 INFO:teuthology.orchestra.run.smithi183.stdout: "pg_num_target": 3, 2024-09-10T19:17:39.556 INFO:teuthology.orchestra.run.smithi183.stdout: "pg_num_pending": 3, 2024-09-10T19:17:39.556 INFO:teuthology.orchestra.run.smithi183.stdout: "last_pg_merge_meta": { 2024-09-10T19:17:39.556 INFO:teuthology.orchestra.run.smithi183.stdout: "source_pgid": "0.0", 2024-09-10T19:17:39.556 INFO:teuthology.orchestra.run.smithi183.stdout: "ready_epoch": 0, 2024-09-10T19:17:39.556 INFO:teuthology.orchestra.run.smithi183.stdout: "last_epoch_started": 0, 2024-09-10T19:17:39.556 INFO:teuthology.orchestra.run.smithi183.stdout: "last_epoch_clean": 0, 2024-09-10T19:17:39.556 INFO:teuthology.orchestra.run.smithi183.stdout: "source_version": "0'0", 2024-09-10T19:17:39.557 INFO:teuthology.orchestra.run.smithi183.stdout: "target_version": "0'0" 2024-09-10T19:17:39.557 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:17:39.557 INFO:teuthology.orchestra.run.smithi183.stdout: "last_change": "63", 2024-09-10T19:17:39.557 INFO:teuthology.orchestra.run.smithi183.stdout: "last_force_op_resend": "0", 2024-09-10T19:17:39.557 INFO:teuthology.orchestra.run.smithi183.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-10T19:17:39.557 INFO:teuthology.orchestra.run.smithi183.stdout: "last_force_op_resend_preluminous": "0", 2024-09-10T19:17:39.557 INFO:teuthology.orchestra.run.smithi183.stdout: "auid": 0, 2024-09-10T19:17:39.557 INFO:teuthology.orchestra.run.smithi183.stdout: "snap_mode": "selfmanaged", 2024-09-10T19:17:39.557 INFO:teuthology.orchestra.run.smithi183.stdout: "snap_seq": 3, 2024-09-10T19:17:39.557 INFO:teuthology.orchestra.run.smithi183.stdout: "snap_epoch": 63, 2024-09-10T19:17:39.557 INFO:teuthology.orchestra.run.smithi183.stdout: "pool_snaps": [], 2024-09-10T19:17:39.557 INFO:teuthology.orchestra.run.smithi183.stdout: "removed_snaps": "[]", 2024-09-10T19:17:39.557 INFO:teuthology.orchestra.run.smithi183.stdout: "quota_max_bytes": 0, 2024-09-10T19:17:39.558 INFO:teuthology.orchestra.run.smithi183.stdout: "quota_max_objects": 0, 2024-09-10T19:17:39.558 INFO:teuthology.orchestra.run.smithi183.stdout: "tiers": [], 2024-09-10T19:17:39.558 INFO:teuthology.orchestra.run.smithi183.stdout: "tier_of": -1, 2024-09-10T19:17:39.558 INFO:teuthology.orchestra.run.smithi183.stdout: "read_tier": -1, 2024-09-10T19:17:39.558 INFO:teuthology.orchestra.run.smithi183.stdout: "write_tier": -1, 2024-09-10T19:17:39.558 INFO:teuthology.orchestra.run.smithi183.stdout: "cache_mode": "none", 2024-09-10T19:17:39.558 INFO:teuthology.orchestra.run.smithi183.stdout: "target_max_bytes": 0, 2024-09-10T19:17:39.558 INFO:teuthology.orchestra.run.smithi183.stdout: "target_max_objects": 0, 2024-09-10T19:17:39.558 INFO:teuthology.orchestra.run.smithi183.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-10T19:17:39.558 INFO:teuthology.orchestra.run.smithi183.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-10T19:17:39.558 INFO:teuthology.orchestra.run.smithi183.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-10T19:17:39.558 INFO:teuthology.orchestra.run.smithi183.stdout: "cache_min_flush_age": 0, 2024-09-10T19:17:39.558 INFO:teuthology.orchestra.run.smithi183.stdout: "cache_min_evict_age": 0, 2024-09-10T19:17:39.558 INFO:teuthology.orchestra.run.smithi183.stdout: "erasure_code_profile": "", 2024-09-10T19:17:39.559 INFO:teuthology.orchestra.run.smithi183.stdout: "hit_set_params": { 2024-09-10T19:17:39.559 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "none" 2024-09-10T19:17:39.559 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:17:39.559 INFO:teuthology.orchestra.run.smithi183.stdout: "hit_set_period": 0, 2024-09-10T19:17:39.559 INFO:teuthology.orchestra.run.smithi183.stdout: "hit_set_count": 0, 2024-09-10T19:17:39.559 INFO:teuthology.orchestra.run.smithi183.stdout: "use_gmt_hitset": true, 2024-09-10T19:17:39.560 INFO:teuthology.orchestra.run.smithi183.stdout: "min_read_recency_for_promote": 0, 2024-09-10T19:17:39.560 INFO:teuthology.orchestra.run.smithi183.stdout: "min_write_recency_for_promote": 0, 2024-09-10T19:17:39.560 INFO:teuthology.orchestra.run.smithi183.stdout: "hit_set_grade_decay_rate": 0, 2024-09-10T19:17:39.560 INFO:teuthology.orchestra.run.smithi183.stdout: "hit_set_search_last_n": 0, 2024-09-10T19:17:39.560 INFO:teuthology.orchestra.run.smithi183.stdout: "grade_table": [], 2024-09-10T19:17:39.560 INFO:teuthology.orchestra.run.smithi183.stdout: "stripe_width": 0, 2024-09-10T19:17:39.560 INFO:teuthology.orchestra.run.smithi183.stdout: "expected_num_objects": 0, 2024-09-10T19:17:39.560 INFO:teuthology.orchestra.run.smithi183.stdout: "fast_read": false, 2024-09-10T19:17:39.560 INFO:teuthology.orchestra.run.smithi183.stdout: "options": {}, 2024-09-10T19:17:39.560 INFO:teuthology.orchestra.run.smithi183.stdout: "application_metadata": { 2024-09-10T19:17:39.560 INFO:teuthology.orchestra.run.smithi183.stdout: "rbd": {} 2024-09-10T19:17:39.560 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:17:39.560 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:17:39.561 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:17:39.561 INFO:teuthology.orchestra.run.smithi183.stdout: "pool": 4, 2024-09-10T19:17:39.561 INFO:teuthology.orchestra.run.smithi183.stdout: "pool_name": "default.rgw.log", 2024-09-10T19:17:39.561 INFO:teuthology.orchestra.run.smithi183.stdout: "create_time": "2024-09-10T19:10:07.958284+0000", 2024-09-10T19:17:39.561 INFO:teuthology.orchestra.run.smithi183.stdout: "flags": 1, 2024-09-10T19:17:39.561 INFO:teuthology.orchestra.run.smithi183.stdout: "flags_names": "hashpspool", 2024-09-10T19:17:39.561 INFO:teuthology.orchestra.run.smithi183.stdout: "type": 1, 2024-09-10T19:17:39.561 INFO:teuthology.orchestra.run.smithi183.stdout: "size": 3, 2024-09-10T19:17:39.561 INFO:teuthology.orchestra.run.smithi183.stdout: "min_size": 2, 2024-09-10T19:17:39.561 INFO:teuthology.orchestra.run.smithi183.stdout: "crush_rule": 0, 2024-09-10T19:17:39.561 INFO:teuthology.orchestra.run.smithi183.stdout: "peering_crush_bucket_count": 0, 2024-09-10T19:17:39.561 INFO:teuthology.orchestra.run.smithi183.stdout: "peering_crush_bucket_target": 0, 2024-09-10T19:17:39.561 INFO:teuthology.orchestra.run.smithi183.stdout: "peering_crush_bucket_barrier": 0, 2024-09-10T19:17:39.561 INFO:teuthology.orchestra.run.smithi183.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-10T19:17:39.562 INFO:teuthology.orchestra.run.smithi183.stdout: "object_hash": 2, 2024-09-10T19:17:39.562 INFO:teuthology.orchestra.run.smithi183.stdout: "pg_autoscale_mode": "off", 2024-09-10T19:17:39.562 INFO:teuthology.orchestra.run.smithi183.stdout: "pg_num": 32, 2024-09-10T19:17:39.562 INFO:teuthology.orchestra.run.smithi183.stdout: "pg_placement_num": 32, 2024-09-10T19:17:39.562 INFO:teuthology.orchestra.run.smithi183.stdout: "pg_placement_num_target": 32, 2024-09-10T19:17:39.562 INFO:teuthology.orchestra.run.smithi183.stdout: "pg_num_target": 32, 2024-09-10T19:17:39.562 INFO:teuthology.orchestra.run.smithi183.stdout: "pg_num_pending": 32, 2024-09-10T19:17:39.562 INFO:teuthology.orchestra.run.smithi183.stdout: "last_pg_merge_meta": { 2024-09-10T19:17:39.562 INFO:teuthology.orchestra.run.smithi183.stdout: "source_pgid": "0.0", 2024-09-10T19:17:39.562 INFO:teuthology.orchestra.run.smithi183.stdout: "ready_epoch": 0, 2024-09-10T19:17:39.562 INFO:teuthology.orchestra.run.smithi183.stdout: "last_epoch_started": 0, 2024-09-10T19:17:39.562 INFO:teuthology.orchestra.run.smithi183.stdout: "last_epoch_clean": 0, 2024-09-10T19:17:39.563 INFO:teuthology.orchestra.run.smithi183.stdout: "source_version": "0'0", 2024-09-10T19:17:39.563 INFO:teuthology.orchestra.run.smithi183.stdout: "target_version": "0'0" 2024-09-10T19:17:39.563 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:17:39.563 INFO:teuthology.orchestra.run.smithi183.stdout: "last_change": "62", 2024-09-10T19:17:39.563 INFO:teuthology.orchestra.run.smithi183.stdout: "last_force_op_resend": "0", 2024-09-10T19:17:39.563 INFO:teuthology.orchestra.run.smithi183.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-10T19:17:39.563 INFO:teuthology.orchestra.run.smithi183.stdout: "last_force_op_resend_preluminous": "0", 2024-09-10T19:17:39.563 INFO:teuthology.orchestra.run.smithi183.stdout: "auid": 0, 2024-09-10T19:17:39.563 INFO:teuthology.orchestra.run.smithi183.stdout: "snap_mode": "selfmanaged", 2024-09-10T19:17:39.563 INFO:teuthology.orchestra.run.smithi183.stdout: "snap_seq": 0, 2024-09-10T19:17:39.563 INFO:teuthology.orchestra.run.smithi183.stdout: "snap_epoch": 0, 2024-09-10T19:17:39.563 INFO:teuthology.orchestra.run.smithi183.stdout: "pool_snaps": [], 2024-09-10T19:17:39.563 INFO:teuthology.orchestra.run.smithi183.stdout: "removed_snaps": "[]", 2024-09-10T19:17:39.564 INFO:teuthology.orchestra.run.smithi183.stdout: "quota_max_bytes": 0, 2024-09-10T19:17:39.564 INFO:teuthology.orchestra.run.smithi183.stdout: "quota_max_objects": 0, 2024-09-10T19:17:39.564 INFO:teuthology.orchestra.run.smithi183.stdout: "tiers": [], 2024-09-10T19:17:39.564 INFO:teuthology.orchestra.run.smithi183.stdout: "tier_of": -1, 2024-09-10T19:17:39.564 INFO:teuthology.orchestra.run.smithi183.stdout: "read_tier": -1, 2024-09-10T19:17:39.564 INFO:teuthology.orchestra.run.smithi183.stdout: "write_tier": -1, 2024-09-10T19:17:39.564 INFO:teuthology.orchestra.run.smithi183.stdout: "cache_mode": "none", 2024-09-10T19:17:39.564 INFO:teuthology.orchestra.run.smithi183.stdout: "target_max_bytes": 0, 2024-09-10T19:17:39.564 INFO:teuthology.orchestra.run.smithi183.stdout: "target_max_objects": 0, 2024-09-10T19:17:39.564 INFO:teuthology.orchestra.run.smithi183.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-10T19:17:39.564 INFO:teuthology.orchestra.run.smithi183.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-10T19:17:39.564 INFO:teuthology.orchestra.run.smithi183.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-10T19:17:39.565 INFO:teuthology.orchestra.run.smithi183.stdout: "cache_min_flush_age": 0, 2024-09-10T19:17:39.565 INFO:teuthology.orchestra.run.smithi183.stdout: "cache_min_evict_age": 0, 2024-09-10T19:17:39.565 INFO:teuthology.orchestra.run.smithi183.stdout: "erasure_code_profile": "", 2024-09-10T19:17:39.565 INFO:teuthology.orchestra.run.smithi183.stdout: "hit_set_params": { 2024-09-10T19:17:39.565 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "none" 2024-09-10T19:17:39.565 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:17:39.565 INFO:teuthology.orchestra.run.smithi183.stdout: "hit_set_period": 0, 2024-09-10T19:17:39.565 INFO:teuthology.orchestra.run.smithi183.stdout: "hit_set_count": 0, 2024-09-10T19:17:39.565 INFO:teuthology.orchestra.run.smithi183.stdout: "use_gmt_hitset": true, 2024-09-10T19:17:39.565 INFO:teuthology.orchestra.run.smithi183.stdout: "min_read_recency_for_promote": 0, 2024-09-10T19:17:39.565 INFO:teuthology.orchestra.run.smithi183.stdout: "min_write_recency_for_promote": 0, 2024-09-10T19:17:39.565 INFO:teuthology.orchestra.run.smithi183.stdout: "hit_set_grade_decay_rate": 0, 2024-09-10T19:17:39.566 INFO:teuthology.orchestra.run.smithi183.stdout: "hit_set_search_last_n": 0, 2024-09-10T19:17:39.566 INFO:teuthology.orchestra.run.smithi183.stdout: "grade_table": [], 2024-09-10T19:17:39.566 INFO:teuthology.orchestra.run.smithi183.stdout: "stripe_width": 0, 2024-09-10T19:17:39.566 INFO:teuthology.orchestra.run.smithi183.stdout: "expected_num_objects": 0, 2024-09-10T19:17:39.566 INFO:teuthology.orchestra.run.smithi183.stdout: "fast_read": false, 2024-09-10T19:17:39.566 INFO:teuthology.orchestra.run.smithi183.stdout: "options": {}, 2024-09-10T19:17:39.566 INFO:teuthology.orchestra.run.smithi183.stdout: "application_metadata": { 2024-09-10T19:17:39.566 INFO:teuthology.orchestra.run.smithi183.stdout: "rgw": {} 2024-09-10T19:17:39.566 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:17:39.566 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:17:39.566 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:17:39.566 INFO:teuthology.orchestra.run.smithi183.stdout: "pool": 5, 2024-09-10T19:17:39.566 INFO:teuthology.orchestra.run.smithi183.stdout: "pool_name": "default.rgw.control", 2024-09-10T19:17:39.567 INFO:teuthology.orchestra.run.smithi183.stdout: "create_time": "2024-09-10T19:10:09.933596+0000", 2024-09-10T19:17:39.567 INFO:teuthology.orchestra.run.smithi183.stdout: "flags": 1, 2024-09-10T19:17:39.567 INFO:teuthology.orchestra.run.smithi183.stdout: "flags_names": "hashpspool", 2024-09-10T19:17:39.567 INFO:teuthology.orchestra.run.smithi183.stdout: "type": 1, 2024-09-10T19:17:39.567 INFO:teuthology.orchestra.run.smithi183.stdout: "size": 3, 2024-09-10T19:17:39.567 INFO:teuthology.orchestra.run.smithi183.stdout: "min_size": 2, 2024-09-10T19:17:39.567 INFO:teuthology.orchestra.run.smithi183.stdout: "crush_rule": 0, 2024-09-10T19:17:39.567 INFO:teuthology.orchestra.run.smithi183.stdout: "peering_crush_bucket_count": 0, 2024-09-10T19:17:39.567 INFO:teuthology.orchestra.run.smithi183.stdout: "peering_crush_bucket_target": 0, 2024-09-10T19:17:39.567 INFO:teuthology.orchestra.run.smithi183.stdout: "peering_crush_bucket_barrier": 0, 2024-09-10T19:17:39.567 INFO:teuthology.orchestra.run.smithi183.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-10T19:17:39.567 INFO:teuthology.orchestra.run.smithi183.stdout: "object_hash": 2, 2024-09-10T19:17:39.567 INFO:teuthology.orchestra.run.smithi183.stdout: "pg_autoscale_mode": "off", 2024-09-10T19:17:39.568 INFO:teuthology.orchestra.run.smithi183.stdout: "pg_num": 32, 2024-09-10T19:17:39.568 INFO:teuthology.orchestra.run.smithi183.stdout: "pg_placement_num": 32, 2024-09-10T19:17:39.568 INFO:teuthology.orchestra.run.smithi183.stdout: "pg_placement_num_target": 32, 2024-09-10T19:17:39.568 INFO:teuthology.orchestra.run.smithi183.stdout: "pg_num_target": 32, 2024-09-10T19:17:39.568 INFO:teuthology.orchestra.run.smithi183.stdout: "pg_num_pending": 32, 2024-09-10T19:17:39.568 INFO:teuthology.orchestra.run.smithi183.stdout: "last_pg_merge_meta": { 2024-09-10T19:17:39.568 INFO:teuthology.orchestra.run.smithi183.stdout: "source_pgid": "0.0", 2024-09-10T19:17:39.568 INFO:teuthology.orchestra.run.smithi183.stdout: "ready_epoch": 0, 2024-09-10T19:17:39.568 INFO:teuthology.orchestra.run.smithi183.stdout: "last_epoch_started": 0, 2024-09-10T19:17:39.568 INFO:teuthology.orchestra.run.smithi183.stdout: "last_epoch_clean": 0, 2024-09-10T19:17:39.569 INFO:teuthology.orchestra.run.smithi183.stdout: "source_version": "0'0", 2024-09-10T19:17:39.569 INFO:teuthology.orchestra.run.smithi183.stdout: "target_version": "0'0" 2024-09-10T19:17:39.569 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:17:39.569 INFO:teuthology.orchestra.run.smithi183.stdout: "last_change": "64", 2024-09-10T19:17:39.569 INFO:teuthology.orchestra.run.smithi183.stdout: "last_force_op_resend": "0", 2024-09-10T19:17:39.569 INFO:teuthology.orchestra.run.smithi183.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-10T19:17:39.569 INFO:teuthology.orchestra.run.smithi183.stdout: "last_force_op_resend_preluminous": "0", 2024-09-10T19:17:39.569 INFO:teuthology.orchestra.run.smithi183.stdout: "auid": 0, 2024-09-10T19:17:39.569 INFO:teuthology.orchestra.run.smithi183.stdout: "snap_mode": "selfmanaged", 2024-09-10T19:17:39.569 INFO:teuthology.orchestra.run.smithi183.stdout: "snap_seq": 0, 2024-09-10T19:17:39.569 INFO:teuthology.orchestra.run.smithi183.stdout: "snap_epoch": 0, 2024-09-10T19:17:39.569 INFO:teuthology.orchestra.run.smithi183.stdout: "pool_snaps": [], 2024-09-10T19:17:39.569 INFO:teuthology.orchestra.run.smithi183.stdout: "removed_snaps": "[]", 2024-09-10T19:17:39.570 INFO:teuthology.orchestra.run.smithi183.stdout: "quota_max_bytes": 0, 2024-09-10T19:17:39.570 INFO:teuthology.orchestra.run.smithi183.stdout: "quota_max_objects": 0, 2024-09-10T19:17:39.570 INFO:teuthology.orchestra.run.smithi183.stdout: "tiers": [], 2024-09-10T19:17:39.570 INFO:teuthology.orchestra.run.smithi183.stdout: "tier_of": -1, 2024-09-10T19:17:39.570 INFO:teuthology.orchestra.run.smithi183.stdout: "read_tier": -1, 2024-09-10T19:17:39.570 INFO:teuthology.orchestra.run.smithi183.stdout: "write_tier": -1, 2024-09-10T19:17:39.570 INFO:teuthology.orchestra.run.smithi183.stdout: "cache_mode": "none", 2024-09-10T19:17:39.570 INFO:teuthology.orchestra.run.smithi183.stdout: "target_max_bytes": 0, 2024-09-10T19:17:39.570 INFO:teuthology.orchestra.run.smithi183.stdout: "target_max_objects": 0, 2024-09-10T19:17:39.570 INFO:teuthology.orchestra.run.smithi183.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-10T19:17:39.570 INFO:teuthology.orchestra.run.smithi183.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-10T19:17:39.570 INFO:teuthology.orchestra.run.smithi183.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-10T19:17:39.570 INFO:teuthology.orchestra.run.smithi183.stdout: "cache_min_flush_age": 0, 2024-09-10T19:17:39.571 INFO:teuthology.orchestra.run.smithi183.stdout: "cache_min_evict_age": 0, 2024-09-10T19:17:39.571 INFO:teuthology.orchestra.run.smithi183.stdout: "erasure_code_profile": "", 2024-09-10T19:17:39.571 INFO:teuthology.orchestra.run.smithi183.stdout: "hit_set_params": { 2024-09-10T19:17:39.571 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "none" 2024-09-10T19:17:39.571 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:17:39.571 INFO:teuthology.orchestra.run.smithi183.stdout: "hit_set_period": 0, 2024-09-10T19:17:39.571 INFO:teuthology.orchestra.run.smithi183.stdout: "hit_set_count": 0, 2024-09-10T19:17:39.571 INFO:teuthology.orchestra.run.smithi183.stdout: "use_gmt_hitset": true, 2024-09-10T19:17:39.571 INFO:teuthology.orchestra.run.smithi183.stdout: "min_read_recency_for_promote": 0, 2024-09-10T19:17:39.571 INFO:teuthology.orchestra.run.smithi183.stdout: "min_write_recency_for_promote": 0, 2024-09-10T19:17:39.571 INFO:teuthology.orchestra.run.smithi183.stdout: "hit_set_grade_decay_rate": 0, 2024-09-10T19:17:39.571 INFO:teuthology.orchestra.run.smithi183.stdout: "hit_set_search_last_n": 0, 2024-09-10T19:17:39.571 INFO:teuthology.orchestra.run.smithi183.stdout: "grade_table": [], 2024-09-10T19:17:39.572 INFO:teuthology.orchestra.run.smithi183.stdout: "stripe_width": 0, 2024-09-10T19:17:39.572 INFO:teuthology.orchestra.run.smithi183.stdout: "expected_num_objects": 0, 2024-09-10T19:17:39.572 INFO:teuthology.orchestra.run.smithi183.stdout: "fast_read": false, 2024-09-10T19:17:39.572 INFO:teuthology.orchestra.run.smithi183.stdout: "options": {}, 2024-09-10T19:17:39.572 INFO:teuthology.orchestra.run.smithi183.stdout: "application_metadata": { 2024-09-10T19:17:39.572 INFO:teuthology.orchestra.run.smithi183.stdout: "rgw": {} 2024-09-10T19:17:39.572 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:17:39.572 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:17:39.572 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:17:39.572 INFO:teuthology.orchestra.run.smithi183.stdout: "pool": 6, 2024-09-10T19:17:39.572 INFO:teuthology.orchestra.run.smithi183.stdout: "pool_name": "default.rgw.meta", 2024-09-10T19:17:39.572 INFO:teuthology.orchestra.run.smithi183.stdout: "create_time": "2024-09-10T19:10:11.993712+0000", 2024-09-10T19:17:39.572 INFO:teuthology.orchestra.run.smithi183.stdout: "flags": 1, 2024-09-10T19:17:39.573 INFO:teuthology.orchestra.run.smithi183.stdout: "flags_names": "hashpspool", 2024-09-10T19:17:39.573 INFO:teuthology.orchestra.run.smithi183.stdout: "type": 1, 2024-09-10T19:17:39.573 INFO:teuthology.orchestra.run.smithi183.stdout: "size": 3, 2024-09-10T19:17:39.573 INFO:teuthology.orchestra.run.smithi183.stdout: "min_size": 2, 2024-09-10T19:17:39.573 INFO:teuthology.orchestra.run.smithi183.stdout: "crush_rule": 0, 2024-09-10T19:17:39.573 INFO:teuthology.orchestra.run.smithi183.stdout: "peering_crush_bucket_count": 0, 2024-09-10T19:17:39.573 INFO:teuthology.orchestra.run.smithi183.stdout: "peering_crush_bucket_target": 0, 2024-09-10T19:17:39.573 INFO:teuthology.orchestra.run.smithi183.stdout: "peering_crush_bucket_barrier": 0, 2024-09-10T19:17:39.573 INFO:teuthology.orchestra.run.smithi183.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-10T19:17:39.573 INFO:teuthology.orchestra.run.smithi183.stdout: "object_hash": 2, 2024-09-10T19:17:39.573 INFO:teuthology.orchestra.run.smithi183.stdout: "pg_autoscale_mode": "off", 2024-09-10T19:17:39.573 INFO:teuthology.orchestra.run.smithi183.stdout: "pg_num": 32, 2024-09-10T19:17:39.573 INFO:teuthology.orchestra.run.smithi183.stdout: "pg_placement_num": 32, 2024-09-10T19:17:39.574 INFO:teuthology.orchestra.run.smithi183.stdout: "pg_placement_num_target": 32, 2024-09-10T19:17:39.574 INFO:teuthology.orchestra.run.smithi183.stdout: "pg_num_target": 32, 2024-09-10T19:17:39.574 INFO:teuthology.orchestra.run.smithi183.stdout: "pg_num_pending": 32, 2024-09-10T19:17:39.574 INFO:teuthology.orchestra.run.smithi183.stdout: "last_pg_merge_meta": { 2024-09-10T19:17:39.574 INFO:teuthology.orchestra.run.smithi183.stdout: "source_pgid": "0.0", 2024-09-10T19:17:39.574 INFO:teuthology.orchestra.run.smithi183.stdout: "ready_epoch": 0, 2024-09-10T19:17:39.574 INFO:teuthology.orchestra.run.smithi183.stdout: "last_epoch_started": 0, 2024-09-10T19:17:39.574 INFO:teuthology.orchestra.run.smithi183.stdout: "last_epoch_clean": 0, 2024-09-10T19:17:39.574 INFO:teuthology.orchestra.run.smithi183.stdout: "source_version": "0'0", 2024-09-10T19:17:39.574 INFO:teuthology.orchestra.run.smithi183.stdout: "target_version": "0'0" 2024-09-10T19:17:39.574 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:17:39.574 INFO:teuthology.orchestra.run.smithi183.stdout: "last_change": "66", 2024-09-10T19:17:39.574 INFO:teuthology.orchestra.run.smithi183.stdout: "last_force_op_resend": "0", 2024-09-10T19:17:39.574 INFO:teuthology.orchestra.run.smithi183.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-10T19:17:39.575 INFO:teuthology.orchestra.run.smithi183.stdout: "last_force_op_resend_preluminous": "0", 2024-09-10T19:17:39.575 INFO:teuthology.orchestra.run.smithi183.stdout: "auid": 0, 2024-09-10T19:17:39.575 INFO:teuthology.orchestra.run.smithi183.stdout: "snap_mode": "selfmanaged", 2024-09-10T19:17:39.575 INFO:teuthology.orchestra.run.smithi183.stdout: "snap_seq": 0, 2024-09-10T19:17:39.575 INFO:teuthology.orchestra.run.smithi183.stdout: "snap_epoch": 0, 2024-09-10T19:17:39.575 INFO:teuthology.orchestra.run.smithi183.stdout: "pool_snaps": [], 2024-09-10T19:17:39.575 INFO:teuthology.orchestra.run.smithi183.stdout: "removed_snaps": "[]", 2024-09-10T19:17:39.575 INFO:teuthology.orchestra.run.smithi183.stdout: "quota_max_bytes": 0, 2024-09-10T19:17:39.575 INFO:teuthology.orchestra.run.smithi183.stdout: "quota_max_objects": 0, 2024-09-10T19:17:39.575 INFO:teuthology.orchestra.run.smithi183.stdout: "tiers": [], 2024-09-10T19:17:39.575 INFO:teuthology.orchestra.run.smithi183.stdout: "tier_of": -1, 2024-09-10T19:17:39.575 INFO:teuthology.orchestra.run.smithi183.stdout: "read_tier": -1, 2024-09-10T19:17:39.575 INFO:teuthology.orchestra.run.smithi183.stdout: "write_tier": -1, 2024-09-10T19:17:39.575 INFO:teuthology.orchestra.run.smithi183.stdout: "cache_mode": "none", 2024-09-10T19:17:39.576 INFO:teuthology.orchestra.run.smithi183.stdout: "target_max_bytes": 0, 2024-09-10T19:17:39.576 INFO:teuthology.orchestra.run.smithi183.stdout: "target_max_objects": 0, 2024-09-10T19:17:39.576 INFO:teuthology.orchestra.run.smithi183.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-10T19:17:39.576 INFO:teuthology.orchestra.run.smithi183.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-10T19:17:39.576 INFO:teuthology.orchestra.run.smithi183.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-10T19:17:39.576 INFO:teuthology.orchestra.run.smithi183.stdout: "cache_min_flush_age": 0, 2024-09-10T19:17:39.576 INFO:teuthology.orchestra.run.smithi183.stdout: "cache_min_evict_age": 0, 2024-09-10T19:17:39.576 INFO:teuthology.orchestra.run.smithi183.stdout: "erasure_code_profile": "", 2024-09-10T19:17:39.576 INFO:teuthology.orchestra.run.smithi183.stdout: "hit_set_params": { 2024-09-10T19:17:39.576 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "none" 2024-09-10T19:17:39.576 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:17:39.576 INFO:teuthology.orchestra.run.smithi183.stdout: "hit_set_period": 0, 2024-09-10T19:17:39.576 INFO:teuthology.orchestra.run.smithi183.stdout: "hit_set_count": 0, 2024-09-10T19:17:39.577 INFO:teuthology.orchestra.run.smithi183.stdout: "use_gmt_hitset": true, 2024-09-10T19:17:39.577 INFO:teuthology.orchestra.run.smithi183.stdout: "min_read_recency_for_promote": 0, 2024-09-10T19:17:39.577 INFO:teuthology.orchestra.run.smithi183.stdout: "min_write_recency_for_promote": 0, 2024-09-10T19:17:39.577 INFO:teuthology.orchestra.run.smithi183.stdout: "hit_set_grade_decay_rate": 0, 2024-09-10T19:17:39.577 INFO:teuthology.orchestra.run.smithi183.stdout: "hit_set_search_last_n": 0, 2024-09-10T19:17:39.577 INFO:teuthology.orchestra.run.smithi183.stdout: "grade_table": [], 2024-09-10T19:17:39.577 INFO:teuthology.orchestra.run.smithi183.stdout: "stripe_width": 0, 2024-09-10T19:17:39.577 INFO:teuthology.orchestra.run.smithi183.stdout: "expected_num_objects": 0, 2024-09-10T19:17:39.577 INFO:teuthology.orchestra.run.smithi183.stdout: "fast_read": false, 2024-09-10T19:17:39.577 INFO:teuthology.orchestra.run.smithi183.stdout: "options": { 2024-09-10T19:17:39.577 INFO:teuthology.orchestra.run.smithi183.stdout: "pg_autoscale_bias": 4 2024-09-10T19:17:39.577 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:17:39.578 INFO:teuthology.orchestra.run.smithi183.stdout:ERROR: (22) Invalid argument 2024-09-10T19:17:39.578 INFO:teuthology.orchestra.run.smithi183.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-10T19:17:39.578 INFO:teuthology.orchestra.run.smithi183.stdout: "application_metadata": { 2024-09-10T19:17:39.578 INFO:teuthology.orchestra.run.smithi183.stdout: "rgw": {} 2024-09-10T19:17:39.579 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:17:39.579 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:17:39.579 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:17:39.579 INFO:teuthology.orchestra.run.smithi183.stdout: "pool": 8, 2024-09-10T19:17:39.579 INFO:teuthology.orchestra.run.smithi183.stdout: "pool_name": "unique_pool_1", 2024-09-10T19:17:39.579 INFO:teuthology.orchestra.run.smithi183.stdout: "create_time": "2024-09-10T19:16:11.474472+0000", 2024-09-10T19:17:39.579 INFO:teuthology.orchestra.run.smithi183.stdout: "flags": 1, 2024-09-10T19:17:39.579 INFO:teuthology.orchestra.run.smithi183.stdout: "flags_names": "hashpspool", 2024-09-10T19:17:39.579 INFO:teuthology.orchestra.run.smithi183.stdout: "type": 1, 2024-09-10T19:17:39.579 INFO:teuthology.orchestra.run.smithi183.stdout: "size": 3, 2024-09-10T19:17:39.579 INFO:teuthology.orchestra.run.smithi183.stdout: "min_size": 2, 2024-09-10T19:17:39.579 INFO:teuthology.orchestra.run.smithi183.stdout: "crush_rule": 0, 2024-09-10T19:17:39.579 INFO:teuthology.orchestra.run.smithi183.stdout: "peering_crush_bucket_count": 0, 2024-09-10T19:17:39.580 INFO:teuthology.orchestra.run.smithi183.stdout: "peering_crush_bucket_target": 0, 2024-09-10T19:17:39.580 INFO:teuthology.orchestra.run.smithi183.stdout: "peering_crush_bucket_barrier": 0, 2024-09-10T19:17:39.580 INFO:teuthology.orchestra.run.smithi183.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-10T19:17:39.580 INFO:teuthology.orchestra.run.smithi183.stdout: "object_hash": 2, 2024-09-10T19:17:39.580 INFO:teuthology.orchestra.run.smithi183.stdout: "pg_autoscale_mode": "off", 2024-09-10T19:17:39.580 INFO:teuthology.orchestra.run.smithi183.stdout: "pg_num": 26, 2024-09-10T19:17:39.580 INFO:teuthology.orchestra.run.smithi183.stdout: "pg_placement_num": 26, 2024-09-10T19:17:39.580 INFO:teuthology.orchestra.run.smithi183.stdout: "pg_placement_num_target": 26, 2024-09-10T19:17:39.580 INFO:teuthology.orchestra.run.smithi183.stdout: "pg_num_target": 26, 2024-09-10T19:17:39.580 INFO:teuthology.orchestra.run.smithi183.stdout: "pg_num_pending": 26, 2024-09-10T19:17:39.580 INFO:teuthology.orchestra.run.smithi183.stdout: "last_pg_merge_meta": { 2024-09-10T19:17:39.580 INFO:teuthology.orchestra.run.smithi183.stdout: "source_pgid": "0.0", 2024-09-10T19:17:39.581 INFO:teuthology.orchestra.run.smithi183.stdout: "ready_epoch": 0, 2024-09-10T19:17:39.581 INFO:teuthology.orchestra.run.smithi183.stdout: "last_epoch_started": 0, 2024-09-10T19:17:39.581 INFO:teuthology.orchestra.run.smithi183.stdout: "last_epoch_clean": 0, 2024-09-10T19:17:39.581 INFO:teuthology.orchestra.run.smithi183.stdout: "source_version": "0'0", 2024-09-10T19:17:39.581 INFO:teuthology.orchestra.run.smithi183.stdout: "target_version": "0'0" 2024-09-10T19:17:39.581 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:17:39.581 INFO:teuthology.orchestra.run.smithi183.stdout: "last_change": "179", 2024-09-10T19:17:39.581 INFO:teuthology.orchestra.run.smithi183.stdout: "last_force_op_resend": "0", 2024-09-10T19:17:39.581 INFO:teuthology.orchestra.run.smithi183.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-10T19:17:39.581 INFO:teuthology.orchestra.run.smithi183.stdout: "last_force_op_resend_preluminous": "177", 2024-09-10T19:17:39.581 INFO:teuthology.orchestra.run.smithi183.stdout: "auid": 0, 2024-09-10T19:17:39.581 INFO:teuthology.orchestra.run.smithi183.stdout: "snap_mode": "selfmanaged", 2024-09-10T19:17:39.581 INFO:teuthology.orchestra.run.smithi183.stdout: "snap_seq": 0, 2024-09-10T19:17:39.582 INFO:teuthology.orchestra.run.smithi183.stdout: "snap_epoch": 0, 2024-09-10T19:17:39.582 INFO:teuthology.orchestra.run.smithi183.stdout: "pool_snaps": [], 2024-09-10T19:17:39.582 INFO:teuthology.orchestra.run.smithi183.stdout: "removed_snaps": "[]", 2024-09-10T19:17:39.582 INFO:teuthology.orchestra.run.smithi183.stdout: "quota_max_bytes": 0, 2024-09-10T19:17:39.582 INFO:teuthology.orchestra.run.smithi183.stdout: "quota_max_objects": 0, 2024-09-10T19:17:39.582 INFO:teuthology.orchestra.run.smithi183.stdout: "tiers": [], 2024-09-10T19:17:39.582 INFO:teuthology.orchestra.run.smithi183.stdout: "tier_of": -1, 2024-09-10T19:17:39.582 INFO:teuthology.orchestra.run.smithi183.stdout: "read_tier": -1, 2024-09-10T19:17:39.582 INFO:teuthology.orchestra.run.smithi183.stdout: "write_tier": -1, 2024-09-10T19:17:39.582 INFO:teuthology.orchestra.run.smithi183.stdout: "cache_mode": "none", 2024-09-10T19:17:39.582 INFO:teuthology.orchestra.run.smithi183.stdout: "target_max_bytes": 0, 2024-09-10T19:17:39.582 INFO:teuthology.orchestra.run.smithi183.stdout: "target_max_objects": 0, 2024-09-10T19:17:39.582 INFO:teuthology.orchestra.run.smithi183.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-10T19:17:39.582 INFO:teuthology.orchestra.run.smithi183.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-10T19:17:39.583 INFO:teuthology.orchestra.run.smithi183.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-10T19:17:39.583 INFO:teuthology.orchestra.run.smithi183.stdout: "cache_min_flush_age": 0, 2024-09-10T19:17:39.583 INFO:teuthology.orchestra.run.smithi183.stdout: "cache_min_evict_age": 0, 2024-09-10T19:17:39.583 INFO:teuthology.orchestra.run.smithi183.stdout: "erasure_code_profile": "", 2024-09-10T19:17:39.583 INFO:teuthology.orchestra.run.smithi183.stdout: "hit_set_params": { 2024-09-10T19:17:39.583 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "none" 2024-09-10T19:17:39.583 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:17:39.583 INFO:teuthology.orchestra.run.smithi183.stdout: "hit_set_period": 0, 2024-09-10T19:17:39.583 INFO:teuthology.orchestra.run.smithi183.stdout: "hit_set_count": 0, 2024-09-10T19:17:39.583 INFO:teuthology.orchestra.run.smithi183.stdout: "use_gmt_hitset": true, 2024-09-10T19:17:39.583 INFO:teuthology.orchestra.run.smithi183.stdout: "min_read_recency_for_promote": 0, 2024-09-10T19:17:39.583 INFO:teuthology.orchestra.run.smithi183.stdout: "min_write_recency_for_promote": 0, 2024-09-10T19:17:39.583 INFO:teuthology.orchestra.run.smithi183.stdout: "hit_set_grade_decay_rate": 0, 2024-09-10T19:17:39.584 INFO:teuthology.orchestra.run.smithi183.stdout: "hit_set_search_last_n": 0, 2024-09-10T19:17:39.584 INFO:teuthology.orchestra.run.smithi183.stdout: "grade_table": [], 2024-09-10T19:17:39.584 INFO:teuthology.orchestra.run.smithi183.stdout: "stripe_width": 0, 2024-09-10T19:17:39.584 INFO:teuthology.orchestra.run.smithi183.stdout: "expected_num_objects": 0, 2024-09-10T19:17:39.584 INFO:teuthology.orchestra.run.smithi183.stdout: "fast_read": false, 2024-09-10T19:17:39.584 INFO:teuthology.orchestra.run.smithi183.stdout: "options": {}, 2024-09-10T19:17:39.584 INFO:teuthology.orchestra.run.smithi183.stdout: "application_metadata": { 2024-09-10T19:17:39.584 INFO:teuthology.orchestra.run.smithi183.stdout: "rados": {} 2024-09-10T19:17:39.584 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:17:39.584 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:17:39.584 INFO:teuthology.orchestra.run.smithi183.stdout: ], 2024-09-10T19:17:39.584 INFO:teuthology.orchestra.run.smithi183.stdout: "osds": [ 2024-09-10T19:17:39.584 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:17:39.585 INFO:teuthology.orchestra.run.smithi183.stdout: "osd": 0, 2024-09-10T19:17:39.585 INFO:teuthology.orchestra.run.smithi183.stdout: "uuid": "9d323010-394a-49aa-ba9b-244d87faf36e", 2024-09-10T19:17:39.585 INFO:teuthology.orchestra.run.smithi183.stdout: "up": 1, 2024-09-10T19:17:39.585 INFO:teuthology.orchestra.run.smithi183.stdout: "in": 1, 2024-09-10T19:17:39.585 INFO:teuthology.orchestra.run.smithi183.stdout: "weight": 1, 2024-09-10T19:17:39.585 INFO:teuthology.orchestra.run.smithi183.stdout: "primary_affinity": 1, 2024-09-10T19:17:39.585 INFO:teuthology.orchestra.run.smithi183.stdout: "last_clean_begin": 0, 2024-09-10T19:17:39.585 INFO:teuthology.orchestra.run.smithi183.stdout: "last_clean_end": 0, 2024-09-10T19:17:39.585 INFO:teuthology.orchestra.run.smithi183.stdout: "up_from": 8, 2024-09-10T19:17:39.585 INFO:teuthology.orchestra.run.smithi183.stdout: "up_thru": 180, 2024-09-10T19:17:39.585 INFO:teuthology.orchestra.run.smithi183.stdout: "down_at": 0, 2024-09-10T19:17:39.585 INFO:teuthology.orchestra.run.smithi183.stdout: "lost_at": 0, 2024-09-10T19:17:39.585 INFO:teuthology.orchestra.run.smithi183.stdout: "public_addrs": { 2024-09-10T19:17:39.585 INFO:teuthology.orchestra.run.smithi183.stdout: "addrvec": [ 2024-09-10T19:17:39.586 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:17:39.586 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v2", 2024-09-10T19:17:39.586 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.183:6802", 2024-09-10T19:17:39.586 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 2813074952 2024-09-10T19:17:39.586 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:17:39.586 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:17:39.586 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v1", 2024-09-10T19:17:39.586 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.183:6803", 2024-09-10T19:17:39.586 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 2813074952 2024-09-10T19:17:39.586 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:17:39.586 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:17:39.586 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:17:39.586 INFO:teuthology.orchestra.run.smithi183.stdout: "cluster_addrs": { 2024-09-10T19:17:39.587 INFO:teuthology.orchestra.run.smithi183.stdout: "addrvec": [ 2024-09-10T19:17:39.587 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:17:39.587 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v2", 2024-09-10T19:17:39.587 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.183:6804", 2024-09-10T19:17:39.587 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 2813074952 2024-09-10T19:17:39.587 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:17:39.587 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:17:39.587 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v1", 2024-09-10T19:17:39.587 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.183:6805", 2024-09-10T19:17:39.587 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 2813074952 2024-09-10T19:17:39.587 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:17:39.587 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:17:39.587 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:17:39.588 INFO:teuthology.orchestra.run.smithi183.stdout: "heartbeat_back_addrs": { 2024-09-10T19:17:39.588 INFO:teuthology.orchestra.run.smithi183.stdout: "addrvec": [ 2024-09-10T19:17:39.588 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:17:39.588 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v2", 2024-09-10T19:17:39.588 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.183:6808", 2024-09-10T19:17:39.588 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 2813074952 2024-09-10T19:17:39.588 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:17:39.588 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:17:39.588 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v1", 2024-09-10T19:17:39.588 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.183:6809", 2024-09-10T19:17:39.588 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 2813074952 2024-09-10T19:17:39.588 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:17:39.588 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:17:39.588 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:17:39.589 INFO:teuthology.orchestra.run.smithi183.stdout: "heartbeat_front_addrs": { 2024-09-10T19:17:39.589 INFO:teuthology.orchestra.run.smithi183.stdout: "addrvec": [ 2024-09-10T19:17:39.589 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:17:39.589 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v2", 2024-09-10T19:17:39.589 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.183:6806", 2024-09-10T19:17:39.589 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 2813074952 2024-09-10T19:17:39.589 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:17:39.589 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:17:39.589 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v1", 2024-09-10T19:17:39.589 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.183:6807", 2024-09-10T19:17:39.589 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 2813074952 2024-09-10T19:17:39.589 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:17:39.589 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:17:39.590 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:17:39.590 INFO:teuthology.orchestra.run.smithi183.stdout: "public_addr": "172.21.15.183:6803/2813074952", 2024-09-10T19:17:39.590 INFO:teuthology.orchestra.run.smithi183.stdout: "cluster_addr": "172.21.15.183:6805/2813074952", 2024-09-10T19:17:39.590 INFO:teuthology.orchestra.run.smithi183.stdout: "heartbeat_back_addr": "172.21.15.183:6809/2813074952", 2024-09-10T19:17:39.590 INFO:teuthology.orchestra.run.smithi183.stdout: "heartbeat_front_addr": "172.21.15.183:6807/2813074952", 2024-09-10T19:17:39.590 INFO:teuthology.orchestra.run.smithi183.stdout: "state": [ 2024-09-10T19:17:39.590 INFO:teuthology.orchestra.run.smithi183.stdout: "exists", 2024-09-10T19:17:39.590 INFO:teuthology.orchestra.run.smithi183.stdout: "up" 2024-09-10T19:17:39.590 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:17:39.590 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:17:39.590 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:17:39.590 INFO:teuthology.orchestra.run.smithi183.stdout: "osd": 1, 2024-09-10T19:17:39.590 INFO:teuthology.orchestra.run.smithi183.stdout: "uuid": "71a6c5dc-5109-4284-8d20-187d8e7afc9e", 2024-09-10T19:17:39.591 INFO:teuthology.orchestra.run.smithi183.stdout: "up": 1, 2024-09-10T19:17:39.591 INFO:teuthology.orchestra.run.smithi183.stdout: "in": 0, 2024-09-10T19:17:39.591 INFO:teuthology.orchestra.run.smithi183.stdout: "weight": 0, 2024-09-10T19:17:39.591 INFO:teuthology.orchestra.run.smithi183.stdout: "primary_affinity": 1, 2024-09-10T19:17:39.591 INFO:teuthology.orchestra.run.smithi183.stdout: "last_clean_begin": 13, 2024-09-10T19:17:39.591 INFO:teuthology.orchestra.run.smithi183.stdout: "last_clean_end": 98, 2024-09-10T19:17:39.591 INFO:teuthology.orchestra.run.smithi183.stdout: "up_from": 138, 2024-09-10T19:17:39.591 INFO:teuthology.orchestra.run.smithi183.stdout: "up_thru": 86, 2024-09-10T19:17:39.591 INFO:teuthology.orchestra.run.smithi183.stdout: "down_at": 99, 2024-09-10T19:17:39.591 INFO:teuthology.orchestra.run.smithi183.stdout: "lost_at": 0, 2024-09-10T19:17:39.591 INFO:teuthology.orchestra.run.smithi183.stdout: "public_addrs": { 2024-09-10T19:17:39.591 INFO:teuthology.orchestra.run.smithi183.stdout: "addrvec": [ 2024-09-10T19:17:39.591 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:17:39.591 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v2", 2024-09-10T19:17:39.592 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.183:6810", 2024-09-10T19:17:39.592 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 219854798 2024-09-10T19:17:39.592 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:17:39.592 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:17:39.592 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v1", 2024-09-10T19:17:39.592 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.183:6811", 2024-09-10T19:17:39.592 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 219854798 2024-09-10T19:17:39.592 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:17:39.592 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:17:39.592 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:17:39.592 INFO:teuthology.orchestra.run.smithi183.stdout: "cluster_addrs": { 2024-09-10T19:17:39.592 INFO:teuthology.orchestra.run.smithi183.stdout: "addrvec": [ 2024-09-10T19:17:39.592 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:17:39.592 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v2", 2024-09-10T19:17:39.593 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.183:6812", 2024-09-10T19:17:39.593 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 219854798 2024-09-10T19:17:39.593 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:17:39.593 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:17:39.593 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v1", 2024-09-10T19:17:39.593 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.183:6813", 2024-09-10T19:17:39.593 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 219854798 2024-09-10T19:17:39.593 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:17:39.593 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:17:39.593 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:17:39.593 INFO:teuthology.orchestra.run.smithi183.stdout: "heartbeat_back_addrs": { 2024-09-10T19:17:39.593 INFO:teuthology.orchestra.run.smithi183.stdout: "addrvec": [ 2024-09-10T19:17:39.593 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:17:39.593 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v2", 2024-09-10T19:17:39.594 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.183:6816", 2024-09-10T19:17:39.594 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 219854798 2024-09-10T19:17:39.594 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:17:39.594 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:17:39.594 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v1", 2024-09-10T19:17:39.594 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.183:6817", 2024-09-10T19:17:39.594 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 219854798 2024-09-10T19:17:39.594 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:17:39.594 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:17:39.594 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:17:39.594 INFO:teuthology.orchestra.run.smithi183.stdout: "heartbeat_front_addrs": { 2024-09-10T19:17:39.594 INFO:teuthology.orchestra.run.smithi183.stdout: "addrvec": [ 2024-09-10T19:17:39.594 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:17:39.594 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v2", 2024-09-10T19:17:39.595 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.183:6814", 2024-09-10T19:17:39.595 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 219854798 2024-09-10T19:17:39.595 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:17:39.595 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:17:39.595 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v1", 2024-09-10T19:17:39.595 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.183:6815", 2024-09-10T19:17:39.595 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 219854798 2024-09-10T19:17:39.595 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:17:39.595 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:17:39.595 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:17:39.595 INFO:teuthology.orchestra.run.smithi183.stdout: "public_addr": "172.21.15.183:6811/219854798", 2024-09-10T19:17:39.595 INFO:teuthology.orchestra.run.smithi183.stdout: "cluster_addr": "172.21.15.183:6813/219854798", 2024-09-10T19:17:39.595 INFO:teuthology.orchestra.run.smithi183.stdout: "heartbeat_back_addr": "172.21.15.183:6817/219854798", 2024-09-10T19:17:39.595 INFO:teuthology.orchestra.run.smithi183.stdout: "heartbeat_front_addr": "172.21.15.183:6815/219854798", 2024-09-10T19:17:39.596 INFO:teuthology.orchestra.run.smithi183.stdout: "state": [ 2024-09-10T19:17:39.596 INFO:teuthology.orchestra.run.smithi183.stdout: "exists", 2024-09-10T19:17:39.596 INFO:teuthology.orchestra.run.smithi183.stdout: "up" 2024-09-10T19:17:39.596 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:17:39.596 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:17:39.596 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:17:39.596 INFO:teuthology.orchestra.run.smithi183.stdout: "osd": 2, 2024-09-10T19:17:39.596 INFO:teuthology.orchestra.run.smithi183.stdout: "uuid": "fc310d5b-b89b-4655-9989-7e10ba356509", 2024-09-10T19:17:39.596 INFO:teuthology.orchestra.run.smithi183.stdout: "up": 1, 2024-09-10T19:17:39.596 INFO:teuthology.orchestra.run.smithi183.stdout: "in": 0, 2024-09-10T19:17:39.596 INFO:teuthology.orchestra.run.smithi183.stdout: "weight": 0, 2024-09-10T19:17:39.596 INFO:teuthology.orchestra.run.smithi183.stdout: "primary_affinity": 1, 2024-09-10T19:17:39.596 INFO:teuthology.orchestra.run.smithi183.stdout: "last_clean_begin": 0, 2024-09-10T19:17:39.597 INFO:teuthology.orchestra.run.smithi183.stdout: "last_clean_end": 0, 2024-09-10T19:17:39.597 INFO:teuthology.orchestra.run.smithi183.stdout: "up_from": 18, 2024-09-10T19:17:39.597 INFO:teuthology.orchestra.run.smithi183.stdout: "up_thru": 148, 2024-09-10T19:17:39.597 INFO:teuthology.orchestra.run.smithi183.stdout: "down_at": 0, 2024-09-10T19:17:39.597 INFO:teuthology.orchestra.run.smithi183.stdout: "lost_at": 0, 2024-09-10T19:17:39.597 INFO:teuthology.orchestra.run.smithi183.stdout: "public_addrs": { 2024-09-10T19:17:39.597 INFO:teuthology.orchestra.run.smithi183.stdout: "addrvec": [ 2024-09-10T19:17:39.597 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:17:39.597 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v2", 2024-09-10T19:17:39.597 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.183:6818", 2024-09-10T19:17:39.597 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 3730013786 2024-09-10T19:17:39.597 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:17:39.597 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:17:39.597 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v1", 2024-09-10T19:17:39.598 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.183:6819", 2024-09-10T19:17:39.598 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 3730013786 2024-09-10T19:17:39.598 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:17:39.598 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:17:39.598 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:17:39.598 INFO:teuthology.orchestra.run.smithi183.stdout: "cluster_addrs": { 2024-09-10T19:17:39.598 INFO:teuthology.orchestra.run.smithi183.stdout: "addrvec": [ 2024-09-10T19:17:39.598 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:17:39.598 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v2", 2024-09-10T19:17:39.598 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.183:6820", 2024-09-10T19:17:39.598 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 3730013786 2024-09-10T19:17:39.598 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:17:39.598 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:17:39.599 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v1", 2024-09-10T19:17:39.599 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.183:6821", 2024-09-10T19:17:39.599 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 3730013786 2024-09-10T19:17:39.599 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:17:39.599 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:17:39.599 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:17:39.599 INFO:teuthology.orchestra.run.smithi183.stdout: "heartbeat_back_addrs": { 2024-09-10T19:17:39.599 INFO:teuthology.orchestra.run.smithi183.stdout: "addrvec": [ 2024-09-10T19:17:39.599 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:17:39.599 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v2", 2024-09-10T19:17:39.599 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.183:6824", 2024-09-10T19:17:39.599 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 3730013786 2024-09-10T19:17:39.599 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:17:39.599 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:17:39.600 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v1", 2024-09-10T19:17:39.600 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.183:6825", 2024-09-10T19:17:39.600 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 3730013786 2024-09-10T19:17:39.600 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:17:39.600 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:17:39.600 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:17:39.600 INFO:teuthology.orchestra.run.smithi183.stdout: "heartbeat_front_addrs": { 2024-09-10T19:17:39.600 INFO:teuthology.orchestra.run.smithi183.stdout: "addrvec": [ 2024-09-10T19:17:39.600 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:17:39.600 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v2", 2024-09-10T19:17:39.600 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.183:6822", 2024-09-10T19:17:39.600 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 3730013786 2024-09-10T19:17:39.600 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:17:39.601 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:17:39.601 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v1", 2024-09-10T19:17:39.601 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.183:6823", 2024-09-10T19:17:39.601 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 3730013786 2024-09-10T19:17:39.601 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:17:39.601 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:17:39.601 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:17:39.601 INFO:teuthology.orchestra.run.smithi183.stdout: "public_addr": "172.21.15.183:6819/3730013786", 2024-09-10T19:17:39.601 INFO:teuthology.orchestra.run.smithi183.stdout: "cluster_addr": "172.21.15.183:6821/3730013786", 2024-09-10T19:17:39.601 INFO:teuthology.orchestra.run.smithi183.stdout: "heartbeat_back_addr": "172.21.15.183:6825/3730013786", 2024-09-10T19:17:39.603 INFO:teuthology.orchestra.run.smithi183.stdout: "heartbeat_front_addr": "172.21.15.183:6823/3730013786", 2024-09-10T19:17:39.603 INFO:teuthology.orchestra.run.smithi183.stdout: "state": [ 2024-09-10T19:17:39.604 INFO:teuthology.orchestra.run.smithi183.stdout: "exists", 2024-09-10T19:17:39.604 INFO:teuthology.orchestra.run.smithi183.stdout: "up" 2024-09-10T19:17:39.604 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:17:39.604 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:17:39.604 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:17:39.604 INFO:teuthology.orchestra.run.smithi183.stdout: "osd": 3, 2024-09-10T19:17:39.604 INFO:teuthology.orchestra.run.smithi183.stdout: "uuid": "7fd07a95-ad22-43f5-a968-539552283e89", 2024-09-10T19:17:39.604 INFO:teuthology.orchestra.run.smithi183.stdout: "up": 1, 2024-09-10T19:17:39.604 INFO:teuthology.orchestra.run.smithi183.stdout: "in": 1, 2024-09-10T19:17:39.604 INFO:teuthology.orchestra.run.smithi183.stdout: "weight": 1, 2024-09-10T19:17:39.604 INFO:teuthology.orchestra.run.smithi183.stdout: "primary_affinity": 1, 2024-09-10T19:17:39.604 INFO:teuthology.orchestra.run.smithi183.stdout: "last_clean_begin": 0, 2024-09-10T19:17:39.604 INFO:teuthology.orchestra.run.smithi183.stdout: "last_clean_end": 0, 2024-09-10T19:17:39.605 INFO:teuthology.orchestra.run.smithi183.stdout: "up_from": 26, 2024-09-10T19:17:39.605 INFO:teuthology.orchestra.run.smithi183.stdout: "up_thru": 180, 2024-09-10T19:17:39.605 INFO:teuthology.orchestra.run.smithi183.stdout: "down_at": 0, 2024-09-10T19:17:39.605 INFO:teuthology.orchestra.run.smithi183.stdout: "lost_at": 0, 2024-09-10T19:17:39.605 INFO:teuthology.orchestra.run.smithi183.stdout: "public_addrs": { 2024-09-10T19:17:39.605 INFO:teuthology.orchestra.run.smithi183.stdout: "addrvec": [ 2024-09-10T19:17:39.605 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:17:39.605 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v2", 2024-09-10T19:17:39.605 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.183:6826", 2024-09-10T19:17:39.605 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 3522836984 2024-09-10T19:17:39.605 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:17:39.605 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:17:39.605 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v1", 2024-09-10T19:17:39.606 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.183:6827", 2024-09-10T19:17:39.606 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 3522836984 2024-09-10T19:17:39.606 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:17:39.606 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:17:39.606 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:17:39.606 INFO:teuthology.orchestra.run.smithi183.stdout: "cluster_addrs": { 2024-09-10T19:17:39.606 INFO:teuthology.orchestra.run.smithi183.stdout: "addrvec": [ 2024-09-10T19:17:39.606 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:17:39.606 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v2", 2024-09-10T19:17:39.606 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.183:6828", 2024-09-10T19:17:39.606 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 3522836984 2024-09-10T19:17:39.606 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:17:39.606 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:17:39.606 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v1", 2024-09-10T19:17:39.607 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.183:6829", 2024-09-10T19:17:39.607 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 3522836984 2024-09-10T19:17:39.607 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:17:39.607 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:17:39.607 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:17:39.607 INFO:teuthology.orchestra.run.smithi183.stdout: "heartbeat_back_addrs": { 2024-09-10T19:17:39.607 INFO:teuthology.orchestra.run.smithi183.stdout: "addrvec": [ 2024-09-10T19:17:39.607 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:17:39.607 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v2", 2024-09-10T19:17:39.607 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.183:6832", 2024-09-10T19:17:39.607 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 3522836984 2024-09-10T19:17:39.607 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:17:39.607 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:17:39.608 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v1", 2024-09-10T19:17:39.608 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.183:6833", 2024-09-10T19:17:39.608 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 3522836984 2024-09-10T19:17:39.608 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:17:39.608 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:17:39.608 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:17:39.608 INFO:teuthology.orchestra.run.smithi183.stdout: "heartbeat_front_addrs": { 2024-09-10T19:17:39.608 INFO:teuthology.orchestra.run.smithi183.stdout: "addrvec": [ 2024-09-10T19:17:39.608 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:17:39.608 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v2", 2024-09-10T19:17:39.608 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.183:6830", 2024-09-10T19:17:39.608 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 3522836984 2024-09-10T19:17:39.608 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:17:39.609 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:17:39.609 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v1", 2024-09-10T19:17:39.609 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.183:6831", 2024-09-10T19:17:39.609 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 3522836984 2024-09-10T19:17:39.609 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:17:39.609 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:17:39.609 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:17:39.609 INFO:teuthology.orchestra.run.smithi183.stdout: "public_addr": "172.21.15.183:6827/3522836984", 2024-09-10T19:17:39.609 INFO:teuthology.orchestra.run.smithi183.stdout: "cluster_addr": "172.21.15.183:6829/3522836984", 2024-09-10T19:17:39.609 INFO:teuthology.orchestra.run.smithi183.stdout: "heartbeat_back_addr": "172.21.15.183:6833/3522836984", 2024-09-10T19:17:39.609 INFO:teuthology.orchestra.run.smithi183.stdout: "heartbeat_front_addr": "172.21.15.183:6831/3522836984", 2024-09-10T19:17:39.609 INFO:teuthology.orchestra.run.smithi183.stdout: "state": [ 2024-09-10T19:17:39.609 INFO:teuthology.orchestra.run.smithi183.stdout: "exists", 2024-09-10T19:17:39.609 INFO:teuthology.orchestra.run.smithi183.stdout: "up" 2024-09-10T19:17:39.610 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:17:39.610 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:17:39.610 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:17:39.610 INFO:teuthology.orchestra.run.smithi183.stdout: "osd": 4, 2024-09-10T19:17:39.610 INFO:teuthology.orchestra.run.smithi183.stdout: "uuid": "631083fb-e503-44da-9b6b-154e88539ee7", 2024-09-10T19:17:39.610 INFO:teuthology.orchestra.run.smithi183.stdout: "up": 1, 2024-09-10T19:17:39.610 INFO:teuthology.orchestra.run.smithi183.stdout: "in": 1, 2024-09-10T19:17:39.610 INFO:teuthology.orchestra.run.smithi183.stdout: "weight": 1, 2024-09-10T19:17:39.610 INFO:teuthology.orchestra.run.smithi183.stdout: "primary_affinity": 1, 2024-09-10T19:17:39.610 INFO:teuthology.orchestra.run.smithi183.stdout: "last_clean_begin": 0, 2024-09-10T19:17:39.610 INFO:teuthology.orchestra.run.smithi183.stdout: "last_clean_end": 0, 2024-09-10T19:17:39.610 INFO:teuthology.orchestra.run.smithi183.stdout: "up_from": 32, 2024-09-10T19:17:39.610 INFO:teuthology.orchestra.run.smithi183.stdout: "up_thru": 180, 2024-09-10T19:17:39.611 INFO:teuthology.orchestra.run.smithi183.stdout: "down_at": 0, 2024-09-10T19:17:39.611 INFO:teuthology.orchestra.run.smithi183.stdout: "lost_at": 0, 2024-09-10T19:17:39.611 INFO:teuthology.orchestra.run.smithi183.stdout: "public_addrs": { 2024-09-10T19:17:39.611 INFO:teuthology.orchestra.run.smithi183.stdout: "addrvec": [ 2024-09-10T19:17:39.611 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:17:39.611 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v2", 2024-09-10T19:17:39.611 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.191:6800", 2024-09-10T19:17:39.611 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 2674828539 2024-09-10T19:17:39.611 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:17:39.611 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:17:39.611 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v1", 2024-09-10T19:17:39.611 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.191:6801", 2024-09-10T19:17:39.611 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 2674828539 2024-09-10T19:17:39.612 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:17:39.612 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:17:39.612 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:17:39.612 INFO:teuthology.orchestra.run.smithi183.stdout: "cluster_addrs": { 2024-09-10T19:17:39.612 INFO:teuthology.orchestra.run.smithi183.stdout: "addrvec": [ 2024-09-10T19:17:39.612 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:17:39.612 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v2", 2024-09-10T19:17:39.612 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.191:6802", 2024-09-10T19:17:39.612 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 2674828539 2024-09-10T19:17:39.612 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:17:39.612 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:17:39.612 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v1", 2024-09-10T19:17:39.612 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.191:6803", 2024-09-10T19:17:39.612 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 2674828539 2024-09-10T19:17:39.613 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:17:39.613 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:17:39.613 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:17:39.613 INFO:teuthology.orchestra.run.smithi183.stdout: "heartbeat_back_addrs": { 2024-09-10T19:17:39.613 INFO:teuthology.orchestra.run.smithi183.stdout: "addrvec": [ 2024-09-10T19:17:39.613 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:17:39.613 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v2", 2024-09-10T19:17:39.613 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.191:6806", 2024-09-10T19:17:39.613 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 2674828539 2024-09-10T19:17:39.613 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:17:39.613 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:17:39.613 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v1", 2024-09-10T19:17:39.613 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.191:6807", 2024-09-10T19:17:39.614 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 2674828539 2024-09-10T19:17:39.614 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:17:39.614 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:17:39.614 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:17:39.614 INFO:teuthology.orchestra.run.smithi183.stdout: "heartbeat_front_addrs": { 2024-09-10T19:17:39.614 INFO:teuthology.orchestra.run.smithi183.stdout: "addrvec": [ 2024-09-10T19:17:39.614 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:17:39.614 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v2", 2024-09-10T19:17:39.614 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.191:6804", 2024-09-10T19:17:39.614 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 2674828539 2024-09-10T19:17:39.614 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:17:39.614 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:17:39.614 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v1", 2024-09-10T19:17:39.614 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.191:6805", 2024-09-10T19:17:39.615 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 2674828539 2024-09-10T19:17:39.615 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:17:39.615 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:17:39.615 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:17:39.615 INFO:teuthology.orchestra.run.smithi183.stdout: "public_addr": "172.21.15.191:6801/2674828539", 2024-09-10T19:17:39.615 INFO:teuthology.orchestra.run.smithi183.stdout: "cluster_addr": "172.21.15.191:6803/2674828539", 2024-09-10T19:17:39.615 INFO:teuthology.orchestra.run.smithi183.stdout: "heartbeat_back_addr": "172.21.15.191:6807/2674828539", 2024-09-10T19:17:39.615 INFO:teuthology.orchestra.run.smithi183.stdout: "heartbeat_front_addr": "172.21.15.191:6805/2674828539", 2024-09-10T19:17:39.615 INFO:teuthology.orchestra.run.smithi183.stdout: "state": [ 2024-09-10T19:17:39.615 INFO:teuthology.orchestra.run.smithi183.stdout: "exists", 2024-09-10T19:17:39.615 INFO:teuthology.orchestra.run.smithi183.stdout: "up" 2024-09-10T19:17:39.615 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:17:39.615 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:17:39.615 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:17:39.616 INFO:teuthology.orchestra.run.smithi183.stdout: "osd": 5, 2024-09-10T19:17:39.616 INFO:teuthology.orchestra.run.smithi183.stdout: "uuid": "6b963695-3eca-4e6e-9207-05b8010ebff9", 2024-09-10T19:17:39.616 INFO:teuthology.orchestra.run.smithi183.stdout: "up": 1, 2024-09-10T19:17:39.616 INFO:teuthology.orchestra.run.smithi183.stdout: "in": 1, 2024-09-10T19:17:39.616 INFO:teuthology.orchestra.run.smithi183.stdout: "weight": 1, 2024-09-10T19:17:39.616 INFO:teuthology.orchestra.run.smithi183.stdout: "primary_affinity": 1, 2024-09-10T19:17:39.616 INFO:teuthology.orchestra.run.smithi183.stdout: "last_clean_begin": 0, 2024-09-10T19:17:39.616 INFO:teuthology.orchestra.run.smithi183.stdout: "last_clean_end": 0, 2024-09-10T19:17:39.616 INFO:teuthology.orchestra.run.smithi183.stdout: "up_from": 39, 2024-09-10T19:17:39.616 INFO:teuthology.orchestra.run.smithi183.stdout: "up_thru": 180, 2024-09-10T19:17:39.616 INFO:teuthology.orchestra.run.smithi183.stdout: "down_at": 0, 2024-09-10T19:17:39.616 INFO:teuthology.orchestra.run.smithi183.stdout: "lost_at": 0, 2024-09-10T19:17:39.616 INFO:teuthology.orchestra.run.smithi183.stdout: "public_addrs": { 2024-09-10T19:17:39.617 INFO:teuthology.orchestra.run.smithi183.stdout: "addrvec": [ 2024-09-10T19:17:39.617 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:17:39.617 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v2", 2024-09-10T19:17:39.617 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.191:6808", 2024-09-10T19:17:39.617 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 681733378 2024-09-10T19:17:39.617 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:17:39.617 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:17:39.617 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v1", 2024-09-10T19:17:39.617 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.191:6809", 2024-09-10T19:17:39.617 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 681733378 2024-09-10T19:17:39.617 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:17:39.618 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:17:39.618 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:17:39.618 INFO:teuthology.orchestra.run.smithi183.stdout: "cluster_addrs": { 2024-09-10T19:17:39.618 INFO:teuthology.orchestra.run.smithi183.stdout: "addrvec": [ 2024-09-10T19:17:39.618 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:17:39.618 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v2", 2024-09-10T19:17:39.618 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:17:39.619 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.191:6810", 2024-09-10T19:17:39.620 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 681733378 2024-09-10T19:17:39.620 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:17:39.620 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:17:39.620 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v1", 2024-09-10T19:17:39.620 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.191:6811", 2024-09-10T19:17:39.620 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 681733378 2024-09-10T19:17:39.620 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:17:39.620 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:17:39.620 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:17:39.620 INFO:teuthology.orchestra.run.smithi183.stdout: "heartbeat_back_addrs": { 2024-09-10T19:17:39.620 INFO:teuthology.orchestra.run.smithi183.stdout: "addrvec": [ 2024-09-10T19:17:39.621 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:17:39.621 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v2", 2024-09-10T19:17:39.621 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.191:6814", 2024-09-10T19:17:39.621 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 681733378 2024-09-10T19:17:39.621 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:17:39.621 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:17:39.621 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v1", 2024-09-10T19:17:39.621 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.191:6815", 2024-09-10T19:17:39.621 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 681733378 2024-09-10T19:17:39.621 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:17:39.621 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:17:39.621 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:17:39.621 INFO:teuthology.orchestra.run.smithi183.stdout: "heartbeat_front_addrs": { 2024-09-10T19:17:39.622 INFO:teuthology.orchestra.run.smithi183.stdout: "addrvec": [ 2024-09-10T19:17:39.622 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:17:39.622 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v2", 2024-09-10T19:17:39.622 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.191:6812", 2024-09-10T19:17:39.622 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 681733378 2024-09-10T19:17:39.622 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:17:39.622 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:17:39.622 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v1", 2024-09-10T19:17:39.622 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.191:6813", 2024-09-10T19:17:39.622 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 681733378 2024-09-10T19:17:39.622 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:17:39.622 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:17:39.622 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:17:39.622 INFO:teuthology.orchestra.run.smithi183.stdout: "public_addr": "172.21.15.191:6809/681733378", 2024-09-10T19:17:39.623 INFO:teuthology.orchestra.run.smithi183.stdout: "cluster_addr": "172.21.15.191:6811/681733378", 2024-09-10T19:17:39.623 INFO:teuthology.orchestra.run.smithi183.stdout: "heartbeat_back_addr": "172.21.15.191:6815/681733378", 2024-09-10T19:17:39.623 INFO:teuthology.orchestra.run.smithi183.stdout: "heartbeat_front_addr": "172.21.15.191:6813/681733378", 2024-09-10T19:17:39.623 INFO:teuthology.orchestra.run.smithi183.stdout: "state": [ 2024-09-10T19:17:39.623 INFO:teuthology.orchestra.run.smithi183.stdout: "exists", 2024-09-10T19:17:39.623 INFO:teuthology.orchestra.run.smithi183.stdout: "up" 2024-09-10T19:17:39.623 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:17:39.623 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:17:39.623 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:17:39.623 INFO:teuthology.orchestra.run.smithi183.stdout: "osd": 6, 2024-09-10T19:17:39.623 INFO:teuthology.orchestra.run.smithi183.stdout: "uuid": "2dc9754f-6aef-4e19-8605-2f1591921811", 2024-09-10T19:17:39.623 INFO:teuthology.orchestra.run.smithi183.stdout: "up": 1, 2024-09-10T19:17:39.624 INFO:teuthology.orchestra.run.smithi183.stdout: "in": 1, 2024-09-10T19:17:39.624 INFO:teuthology.orchestra.run.smithi183.stdout: "weight": 1, 2024-09-10T19:17:39.624 INFO:teuthology.orchestra.run.smithi183.stdout: "primary_affinity": 1, 2024-09-10T19:17:39.624 INFO:teuthology.orchestra.run.smithi183.stdout: "last_clean_begin": 0, 2024-09-10T19:17:39.624 INFO:teuthology.orchestra.run.smithi183.stdout: "last_clean_end": 0, 2024-09-10T19:17:39.624 INFO:teuthology.orchestra.run.smithi183.stdout: "up_from": 47, 2024-09-10T19:17:39.624 INFO:teuthology.orchestra.run.smithi183.stdout: "up_thru": 179, 2024-09-10T19:17:39.624 INFO:teuthology.orchestra.run.smithi183.stdout: "down_at": 0, 2024-09-10T19:17:39.624 INFO:teuthology.orchestra.run.smithi183.stdout: "lost_at": 0, 2024-09-10T19:17:39.624 INFO:teuthology.orchestra.run.smithi183.stdout: "public_addrs": { 2024-09-10T19:17:39.624 INFO:teuthology.orchestra.run.smithi183.stdout: "addrvec": [ 2024-09-10T19:17:39.624 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:17:39.624 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v2", 2024-09-10T19:17:39.625 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.191:6816", 2024-09-10T19:17:39.625 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 2825813477 2024-09-10T19:17:39.625 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:17:39.625 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:17:39.625 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v1", 2024-09-10T19:17:39.625 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.191:6817", 2024-09-10T19:17:39.625 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 2825813477 2024-09-10T19:17:39.625 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:17:39.625 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:17:39.625 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:17:39.625 INFO:teuthology.orchestra.run.smithi183.stdout: "cluster_addrs": { 2024-09-10T19:17:39.625 INFO:teuthology.orchestra.run.smithi183.stdout: "addrvec": [ 2024-09-10T19:17:39.625 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:17:39.625 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v2", 2024-09-10T19:17:39.626 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.191:6818", 2024-09-10T19:17:39.626 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 2825813477 2024-09-10T19:17:39.626 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:17:39.626 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:17:39.626 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v1", 2024-09-10T19:17:39.626 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.191:6819", 2024-09-10T19:17:39.626 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 2825813477 2024-09-10T19:17:39.626 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:17:39.626 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:17:39.626 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:17:39.626 INFO:teuthology.orchestra.run.smithi183.stdout: "heartbeat_back_addrs": { 2024-09-10T19:17:39.626 INFO:teuthology.orchestra.run.smithi183.stdout: "addrvec": [ 2024-09-10T19:17:39.626 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:17:39.627 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v2", 2024-09-10T19:17:39.627 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.191:6822", 2024-09-10T19:17:39.627 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 2825813477 2024-09-10T19:17:39.627 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:17:39.627 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:17:39.627 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v1", 2024-09-10T19:17:39.627 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.191:6823", 2024-09-10T19:17:39.627 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 2825813477 2024-09-10T19:17:39.627 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:17:39.627 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:17:39.627 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:17:39.627 INFO:teuthology.orchestra.run.smithi183.stdout: "heartbeat_front_addrs": { 2024-09-10T19:17:39.627 INFO:teuthology.orchestra.run.smithi183.stdout: "addrvec": [ 2024-09-10T19:17:39.627 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:17:39.628 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v2", 2024-09-10T19:17:39.628 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.191:6820", 2024-09-10T19:17:39.628 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 2825813477 2024-09-10T19:17:39.628 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:17:39.628 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:17:39.628 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v1", 2024-09-10T19:17:39.628 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.191:6821", 2024-09-10T19:17:39.628 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 2825813477 2024-09-10T19:17:39.628 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:17:39.628 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:17:39.628 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:17:39.628 INFO:teuthology.orchestra.run.smithi183.stdout: "public_addr": "172.21.15.191:6817/2825813477", 2024-09-10T19:17:39.628 INFO:teuthology.orchestra.run.smithi183.stdout: "cluster_addr": "172.21.15.191:6819/2825813477", 2024-09-10T19:17:39.629 INFO:teuthology.orchestra.run.smithi183.stdout: "heartbeat_back_addr": "172.21.15.191:6823/2825813477", 2024-09-10T19:17:39.629 INFO:teuthology.orchestra.run.smithi183.stdout: "heartbeat_front_addr": "172.21.15.191:6821/2825813477", 2024-09-10T19:17:39.629 INFO:teuthology.orchestra.run.smithi183.stdout: "state": [ 2024-09-10T19:17:39.629 INFO:teuthology.orchestra.run.smithi183.stdout: "exists", 2024-09-10T19:17:39.629 INFO:teuthology.orchestra.run.smithi183.stdout: "up" 2024-09-10T19:17:39.629 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:17:39.629 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:17:39.629 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:17:39.629 INFO:teuthology.orchestra.run.smithi183.stdout: "osd": 7, 2024-09-10T19:17:39.629 INFO:teuthology.orchestra.run.smithi183.stdout: "uuid": "1a9a299e-d5ba-4b9a-85d5-7f6a5bc754e1", 2024-09-10T19:17:39.629 INFO:teuthology.orchestra.run.smithi183.stdout: "up": 1, 2024-09-10T19:17:39.629 INFO:teuthology.orchestra.run.smithi183.stdout: "in": 1, 2024-09-10T19:17:39.629 INFO:teuthology.orchestra.run.smithi183.stdout: "weight": 1, 2024-09-10T19:17:39.630 INFO:teuthology.orchestra.run.smithi183.stdout: "primary_affinity": 1, 2024-09-10T19:17:39.630 INFO:teuthology.orchestra.run.smithi183.stdout: "last_clean_begin": 0, 2024-09-10T19:17:39.630 INFO:teuthology.orchestra.run.smithi183.stdout: "last_clean_end": 0, 2024-09-10T19:17:39.630 INFO:teuthology.orchestra.run.smithi183.stdout: "up_from": 54, 2024-09-10T19:17:39.630 INFO:teuthology.orchestra.run.smithi183.stdout: "up_thru": 180, 2024-09-10T19:17:39.630 INFO:teuthology.orchestra.run.smithi183.stdout: "down_at": 0, 2024-09-10T19:17:39.630 INFO:teuthology.orchestra.run.smithi183.stdout: "lost_at": 0, 2024-09-10T19:17:39.630 INFO:teuthology.orchestra.run.smithi183.stdout: "public_addrs": { 2024-09-10T19:17:39.630 INFO:teuthology.orchestra.run.smithi183.stdout: "addrvec": [ 2024-09-10T19:17:39.630 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:17:39.630 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v2", 2024-09-10T19:17:39.630 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.191:6824", 2024-09-10T19:17:39.630 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 1422610021 2024-09-10T19:17:39.631 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:17:39.631 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:17:39.631 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v1", 2024-09-10T19:17:39.631 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.191:6825", 2024-09-10T19:17:39.631 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 1422610021 2024-09-10T19:17:39.631 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:17:39.631 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:17:39.631 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:17:39.631 INFO:teuthology.orchestra.run.smithi183.stdout: "cluster_addrs": { 2024-09-10T19:17:39.631 INFO:teuthology.orchestra.run.smithi183.stdout: "addrvec": [ 2024-09-10T19:17:39.631 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:17:39.631 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v2", 2024-09-10T19:17:39.631 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.191:6826", 2024-09-10T19:17:39.632 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 1422610021 2024-09-10T19:17:39.632 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:17:39.632 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:17:39.632 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v1", 2024-09-10T19:17:39.632 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.191:6827", 2024-09-10T19:17:39.632 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 1422610021 2024-09-10T19:17:39.632 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:17:39.632 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:17:39.632 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:17:39.632 INFO:teuthology.orchestra.run.smithi183.stdout: "heartbeat_back_addrs": { 2024-09-10T19:17:39.632 INFO:teuthology.orchestra.run.smithi183.stdout: "addrvec": [ 2024-09-10T19:17:39.632 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:17:39.633 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v2", 2024-09-10T19:17:39.633 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.191:6830", 2024-09-10T19:17:39.633 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 1422610021 2024-09-10T19:17:39.633 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:17:39.633 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:17:39.633 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v1", 2024-09-10T19:17:39.633 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.191:6831", 2024-09-10T19:17:39.633 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 1422610021 2024-09-10T19:17:39.633 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:17:39.633 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:17:39.633 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:17:39.633 INFO:teuthology.orchestra.run.smithi183.stdout: "heartbeat_front_addrs": { 2024-09-10T19:17:39.633 INFO:teuthology.orchestra.run.smithi183.stdout: "addrvec": [ 2024-09-10T19:17:39.634 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:17:39.634 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v2", 2024-09-10T19:17:39.634 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.191:6828", 2024-09-10T19:17:39.634 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 1422610021 2024-09-10T19:17:39.634 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:17:39.634 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:17:39.634 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v1", 2024-09-10T19:17:39.634 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.191:6829", 2024-09-10T19:17:39.634 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 1422610021 2024-09-10T19:17:39.634 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:17:39.634 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:17:39.634 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:17:39.634 INFO:teuthology.orchestra.run.smithi183.stdout: "public_addr": "172.21.15.191:6825/1422610021", 2024-09-10T19:17:39.634 INFO:teuthology.orchestra.run.smithi183.stdout: "cluster_addr": "172.21.15.191:6827/1422610021", 2024-09-10T19:17:39.634 INFO:teuthology.orchestra.run.smithi183.stdout: "heartbeat_back_addr": "172.21.15.191:6831/1422610021", 2024-09-10T19:17:39.635 INFO:teuthology.orchestra.run.smithi183.stdout: "heartbeat_front_addr": "172.21.15.191:6829/1422610021", 2024-09-10T19:17:39.635 INFO:teuthology.orchestra.run.smithi183.stdout: "state": [ 2024-09-10T19:17:39.635 INFO:teuthology.orchestra.run.smithi183.stdout: "exists", 2024-09-10T19:17:39.635 INFO:teuthology.orchestra.run.smithi183.stdout: "up" 2024-09-10T19:17:39.635 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:17:39.635 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:17:39.635 INFO:teuthology.orchestra.run.smithi183.stdout: ], 2024-09-10T19:17:39.635 INFO:teuthology.orchestra.run.smithi183.stdout: "osd_xinfo": [ 2024-09-10T19:17:39.635 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:17:39.635 INFO:teuthology.orchestra.run.smithi183.stdout: "osd": 0, 2024-09-10T19:17:39.635 INFO:teuthology.orchestra.run.smithi183.stdout: "down_stamp": "0.000000", 2024-09-10T19:17:39.635 INFO:teuthology.orchestra.run.smithi183.stdout: "laggy_probability": 0, 2024-09-10T19:17:39.635 INFO:teuthology.orchestra.run.smithi183.stdout: "laggy_interval": 0, 2024-09-10T19:17:39.635 INFO:teuthology.orchestra.run.smithi183.stdout: "features": 4540138320759226367, 2024-09-10T19:17:39.635 INFO:teuthology.orchestra.run.smithi183.stdout: "old_weight": 0, 2024-09-10T19:17:39.636 INFO:teuthology.orchestra.run.smithi183.stdout: "last_purged_snaps_scrub": "2024-09-10T19:16:46.494740+0000", 2024-09-10T19:17:39.636 INFO:teuthology.orchestra.run.smithi183.stdout: "dead_epoch": 0 2024-09-10T19:17:39.636 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:17:39.636 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:17:39.636 INFO:teuthology.orchestra.run.smithi183.stdout: "osd": 1, 2024-09-10T19:17:39.636 INFO:teuthology.orchestra.run.smithi183.stdout: "down_stamp": "0.000000", 2024-09-10T19:17:39.636 INFO:teuthology.orchestra.run.smithi183.stdout: "laggy_probability": 0, 2024-09-10T19:17:39.636 INFO:teuthology.orchestra.run.smithi183.stdout: "laggy_interval": 0, 2024-09-10T19:17:39.636 INFO:teuthology.orchestra.run.smithi183.stdout: "features": 4540138320759226367, 2024-09-10T19:17:39.636 INFO:teuthology.orchestra.run.smithi183.stdout: "old_weight": 65536, 2024-09-10T19:17:39.636 INFO:teuthology.orchestra.run.smithi183.stdout: "last_purged_snaps_scrub": "2024-09-10T19:16:25.059015+0000", 2024-09-10T19:17:39.636 INFO:teuthology.orchestra.run.smithi183.stdout: "dead_epoch": 98 2024-09-10T19:17:39.636 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:17:39.636 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:17:39.637 INFO:teuthology.orchestra.run.smithi183.stdout: "osd": 2, 2024-09-10T19:17:39.637 INFO:teuthology.orchestra.run.smithi183.stdout: "down_stamp": "0.000000", 2024-09-10T19:17:39.637 INFO:teuthology.orchestra.run.smithi183.stdout: "laggy_probability": 0, 2024-09-10T19:17:39.637 INFO:teuthology.orchestra.run.smithi183.stdout: "laggy_interval": 0, 2024-09-10T19:17:39.637 INFO:teuthology.orchestra.run.smithi183.stdout: "features": 4540138320759226367, 2024-09-10T19:17:39.637 INFO:teuthology.orchestra.run.smithi183.stdout: "old_weight": 65536, 2024-09-10T19:17:39.637 INFO:teuthology.orchestra.run.smithi183.stdout: "last_purged_snaps_scrub": "2024-09-10T19:17:07.299690+0000", 2024-09-10T19:17:39.637 INFO:teuthology.orchestra.run.smithi183.stdout: "dead_epoch": 0 2024-09-10T19:17:39.637 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:17:39.637 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:17:39.637 INFO:teuthology.orchestra.run.smithi183.stdout: "osd": 3, 2024-09-10T19:17:39.637 INFO:teuthology.orchestra.run.smithi183.stdout: "down_stamp": "0.000000", 2024-09-10T19:17:39.637 INFO:teuthology.orchestra.run.smithi183.stdout: "laggy_probability": 0, 2024-09-10T19:17:39.638 INFO:teuthology.orchestra.run.smithi183.stdout: "laggy_interval": 0, 2024-09-10T19:17:39.638 INFO:teuthology.orchestra.run.smithi183.stdout: "features": 4540138320759226367, 2024-09-10T19:17:39.638 INFO:teuthology.orchestra.run.smithi183.stdout: "old_weight": 0, 2024-09-10T19:17:39.638 INFO:teuthology.orchestra.run.smithi183.stdout: "last_purged_snaps_scrub": "2024-09-10T19:17:09.095005+0000", 2024-09-10T19:17:39.638 INFO:teuthology.orchestra.run.smithi183.stdout: "dead_epoch": 0 2024-09-10T19:17:39.638 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:17:39.638 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:17:39.638 INFO:teuthology.orchestra.run.smithi183.stdout: "osd": 4, 2024-09-10T19:17:39.638 INFO:teuthology.orchestra.run.smithi183.stdout: "down_stamp": "0.000000", 2024-09-10T19:17:39.638 INFO:teuthology.orchestra.run.smithi183.stdout: "laggy_probability": 0, 2024-09-10T19:17:39.638 INFO:teuthology.orchestra.run.smithi183.stdout: "laggy_interval": 0, 2024-09-10T19:17:39.638 INFO:teuthology.orchestra.run.smithi183.stdout: "features": 4540138320759226367, 2024-09-10T19:17:39.638 INFO:teuthology.orchestra.run.smithi183.stdout: "old_weight": 0, 2024-09-10T19:17:39.638 INFO:teuthology.orchestra.run.smithi183.stdout: "last_purged_snaps_scrub": "2024-09-10T19:16:27.222375+0000", 2024-09-10T19:17:39.639 INFO:teuthology.orchestra.run.smithi183.stdout: "dead_epoch": 0 2024-09-10T19:17:39.639 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:17:39.639 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:17:39.639 INFO:teuthology.orchestra.run.smithi183.stdout: "osd": 5, 2024-09-10T19:17:39.639 INFO:teuthology.orchestra.run.smithi183.stdout: "down_stamp": "0.000000", 2024-09-10T19:17:39.639 INFO:teuthology.orchestra.run.smithi183.stdout: "laggy_probability": 0, 2024-09-10T19:17:39.639 INFO:teuthology.orchestra.run.smithi183.stdout: "laggy_interval": 0, 2024-09-10T19:17:39.639 INFO:teuthology.orchestra.run.smithi183.stdout: "features": 4540138320759226367, 2024-09-10T19:17:39.639 INFO:teuthology.orchestra.run.smithi183.stdout: "old_weight": 0, 2024-09-10T19:17:39.639 INFO:teuthology.orchestra.run.smithi183.stdout: "last_purged_snaps_scrub": "2024-09-10T19:17:28.197628+0000", 2024-09-10T19:17:39.639 INFO:teuthology.orchestra.run.smithi183.stdout: "dead_epoch": 0 2024-09-10T19:17:39.639 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:17:39.640 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:17:39.640 INFO:teuthology.orchestra.run.smithi183.stdout: "osd": 6, 2024-09-10T19:17:39.640 INFO:teuthology.orchestra.run.smithi183.stdout: "down_stamp": "0.000000", 2024-09-10T19:17:39.640 INFO:teuthology.orchestra.run.smithi183.stdout: "laggy_probability": 0, 2024-09-10T19:17:39.640 INFO:teuthology.orchestra.run.smithi183.stdout: "laggy_interval": 0, 2024-09-10T19:17:39.640 INFO:teuthology.orchestra.run.smithi183.stdout: "features": 4540138320759226367, 2024-09-10T19:17:39.640 INFO:teuthology.orchestra.run.smithi183.stdout: "old_weight": 0, 2024-09-10T19:17:39.640 INFO:teuthology.orchestra.run.smithi183.stdout: "last_purged_snaps_scrub": "2024-09-10T19:17:15.672700+0000", 2024-09-10T19:17:39.640 INFO:teuthology.orchestra.run.smithi183.stdout: "dead_epoch": 0 2024-09-10T19:17:39.640 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:17:39.640 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:17:39.640 INFO:teuthology.orchestra.run.smithi183.stdout: "osd": 7, 2024-09-10T19:17:39.641 INFO:teuthology.orchestra.run.smithi183.stdout: "down_stamp": "0.000000", 2024-09-10T19:17:39.641 INFO:teuthology.orchestra.run.smithi183.stdout: "laggy_probability": 0, 2024-09-10T19:17:39.641 INFO:teuthology.orchestra.run.smithi183.stdout: "laggy_interval": 0, 2024-09-10T19:17:39.641 INFO:teuthology.orchestra.run.smithi183.stdout: "features": 4540138320759226367, 2024-09-10T19:17:39.641 INFO:teuthology.orchestra.run.smithi183.stdout: "old_weight": 0, 2024-09-10T19:17:39.641 INFO:teuthology.orchestra.run.smithi183.stdout: "last_purged_snaps_scrub": "2024-09-10T19:17:35.086065+0000", 2024-09-10T19:17:39.641 INFO:teuthology.orchestra.run.smithi183.stdout: "dead_epoch": 0 2024-09-10T19:17:39.641 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:17:39.641 INFO:teuthology.orchestra.run.smithi183.stdout: ], 2024-09-10T19:17:39.641 INFO:teuthology.orchestra.run.smithi183.stdout: "pg_upmap": [ 2024-09-10T19:17:39.641 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:17:39.641 INFO:teuthology.orchestra.run.smithi183.stdout: "pgid": "8.3", 2024-09-10T19:17:39.641 INFO:teuthology.orchestra.run.smithi183.stdout: "osds": [ 2024-09-10T19:17:39.642 INFO:teuthology.orchestra.run.smithi183.stdout: 7, 2024-09-10T19:17:39.642 INFO:teuthology.orchestra.run.smithi183.stdout: 3, 2024-09-10T19:17:39.642 INFO:teuthology.orchestra.run.smithi183.stdout: 5 2024-09-10T19:17:39.642 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:17:39.642 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:17:39.642 INFO:teuthology.orchestra.run.smithi183.stdout: ], 2024-09-10T19:17:39.642 INFO:teuthology.orchestra.run.smithi183.stdout: "pg_upmap_items": [ 2024-09-10T19:17:39.642 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:17:39.642 INFO:teuthology.orchestra.run.smithi183.stdout: "pgid": "4.1", 2024-09-10T19:17:39.642 INFO:teuthology.orchestra.run.smithi183.stdout: "mappings": [ 2024-09-10T19:17:39.642 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:17:39.642 INFO:teuthology.orchestra.run.smithi183.stdout: "from": 6, 2024-09-10T19:17:39.643 INFO:teuthology.orchestra.run.smithi183.stdout: "to": 7 2024-09-10T19:17:39.643 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:17:39.643 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:17:39.643 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:17:39.643 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:17:39.643 INFO:teuthology.orchestra.run.smithi183.stdout: "pgid": "4.11", 2024-09-10T19:17:39.643 INFO:teuthology.orchestra.run.smithi183.stdout: "mappings": [ 2024-09-10T19:17:39.643 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:17:39.643 INFO:teuthology.orchestra.run.smithi183.stdout: "from": 3, 2024-09-10T19:17:39.643 INFO:teuthology.orchestra.run.smithi183.stdout: "to": 5 2024-09-10T19:17:39.643 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:17:39.643 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:17:39.643 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:17:39.643 INFO:teuthology.orchestra.run.smithi183.stdout: ], 2024-09-10T19:17:39.644 INFO:teuthology.orchestra.run.smithi183.stdout: "pg_temp": [ 2024-09-10T19:17:39.644 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:17:39.644 INFO:teuthology.orchestra.run.smithi183.stdout: "pgid": "8.10", 2024-09-10T19:17:39.644 INFO:teuthology.orchestra.run.smithi183.stdout: "osds": [ 2024-09-10T19:17:39.644 INFO:teuthology.orchestra.run.smithi183.stdout: 3, 2024-09-10T19:17:39.644 INFO:teuthology.orchestra.run.smithi183.stdout: 5, 2024-09-10T19:17:39.644 INFO:teuthology.orchestra.run.smithi183.stdout: 4 2024-09-10T19:17:39.644 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:17:39.644 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:17:39.644 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:17:39.644 INFO:teuthology.orchestra.run.smithi183.stdout: "pgid": "8.11", 2024-09-10T19:17:39.644 INFO:teuthology.orchestra.run.smithi183.stdout: "osds": [ 2024-09-10T19:17:39.644 INFO:teuthology.orchestra.run.smithi183.stdout: 5, 2024-09-10T19:17:39.645 INFO:teuthology.orchestra.run.smithi183.stdout: 0, 2024-09-10T19:17:39.645 INFO:teuthology.orchestra.run.smithi183.stdout: 3 2024-09-10T19:17:39.645 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:17:39.645 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:17:39.645 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:17:39.645 INFO:teuthology.orchestra.run.smithi183.stdout: "pgid": "8.12", 2024-09-10T19:17:39.645 INFO:teuthology.orchestra.run.smithi183.stdout: "osds": [ 2024-09-10T19:17:39.645 INFO:teuthology.orchestra.run.smithi183.stdout: 0, 2024-09-10T19:17:39.645 INFO:teuthology.orchestra.run.smithi183.stdout: 4, 2024-09-10T19:17:39.645 INFO:teuthology.orchestra.run.smithi183.stdout: 5 2024-09-10T19:17:39.645 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:17:39.645 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:17:39.645 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:17:39.645 INFO:teuthology.orchestra.run.smithi183.stdout: "pgid": "8.13", 2024-09-10T19:17:39.645 INFO:teuthology.orchestra.run.smithi183.stdout: "osds": [ 2024-09-10T19:17:39.646 INFO:teuthology.orchestra.run.smithi183.stdout: 3, 2024-09-10T19:17:39.646 INFO:teuthology.orchestra.run.smithi183.stdout: 5, 2024-09-10T19:17:39.646 INFO:teuthology.orchestra.run.smithi183.stdout: 7 2024-09-10T19:17:39.646 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:17:39.646 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:17:39.646 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:17:39.646 INFO:teuthology.orchestra.run.smithi183.stdout: "pgid": "8.14", 2024-09-10T19:17:39.646 INFO:teuthology.orchestra.run.smithi183.stdout: "osds": [ 2024-09-10T19:17:39.646 INFO:teuthology.orchestra.run.smithi183.stdout: 4, 2024-09-10T19:17:39.646 INFO:teuthology.orchestra.run.smithi183.stdout: 5, 2024-09-10T19:17:39.646 INFO:teuthology.orchestra.run.smithi183.stdout: 6 2024-09-10T19:17:39.646 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:17:39.646 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:17:39.646 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:17:39.647 INFO:teuthology.orchestra.run.smithi183.stdout: "pgid": "8.15", 2024-09-10T19:17:39.647 INFO:teuthology.orchestra.run.smithi183.stdout: "osds": [ 2024-09-10T19:17:39.647 INFO:teuthology.orchestra.run.smithi183.stdout: 7, 2024-09-10T19:17:39.647 INFO:teuthology.orchestra.run.smithi183.stdout: 6, 2024-09-10T19:17:39.647 INFO:teuthology.orchestra.run.smithi183.stdout: 4 2024-09-10T19:17:39.647 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:17:39.647 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:17:39.647 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:17:39.647 INFO:teuthology.orchestra.run.smithi183.stdout: "pgid": "8.16", 2024-09-10T19:17:39.647 INFO:teuthology.orchestra.run.smithi183.stdout: "osds": [ 2024-09-10T19:17:39.647 INFO:teuthology.orchestra.run.smithi183.stdout: 0, 2024-09-10T19:17:39.647 INFO:teuthology.orchestra.run.smithi183.stdout: 6, 2024-09-10T19:17:39.647 INFO:teuthology.orchestra.run.smithi183.stdout: 7 2024-09-10T19:17:39.647 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:17:39.647 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:17:39.648 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:17:39.648 INFO:teuthology.orchestra.run.smithi183.stdout: "pgid": "8.17", 2024-09-10T19:17:39.648 INFO:teuthology.orchestra.run.smithi183.stdout: "osds": [ 2024-09-10T19:17:39.648 INFO:teuthology.orchestra.run.smithi183.stdout: 3, 2024-09-10T19:17:39.648 INFO:teuthology.orchestra.run.smithi183.stdout: 6, 2024-09-10T19:17:39.648 INFO:teuthology.orchestra.run.smithi183.stdout: 4 2024-09-10T19:17:39.648 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:17:39.648 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:17:39.648 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:17:39.648 INFO:teuthology.orchestra.run.smithi183.stdout: "pgid": "8.18", 2024-09-10T19:17:39.648 INFO:teuthology.orchestra.run.smithi183.stdout: "osds": [ 2024-09-10T19:17:39.648 INFO:teuthology.orchestra.run.smithi183.stdout: 3, 2024-09-10T19:17:39.649 INFO:teuthology.orchestra.run.smithi183.stdout: 5, 2024-09-10T19:17:39.649 INFO:teuthology.orchestra.run.smithi183.stdout: 7 2024-09-10T19:17:39.649 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:17:39.649 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:17:39.649 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:17:39.649 INFO:teuthology.orchestra.run.smithi183.stdout: "pgid": "8.19", 2024-09-10T19:17:39.649 INFO:teuthology.orchestra.run.smithi183.stdout: "osds": [ 2024-09-10T19:17:39.649 INFO:teuthology.orchestra.run.smithi183.stdout: 5, 2024-09-10T19:17:39.649 INFO:teuthology.orchestra.run.smithi183.stdout: 0, 2024-09-10T19:17:39.649 INFO:teuthology.orchestra.run.smithi183.stdout: 6 2024-09-10T19:17:39.649 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:17:39.649 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:17:39.649 INFO:teuthology.orchestra.run.smithi183.stdout: ], 2024-09-10T19:17:39.649 INFO:teuthology.orchestra.run.smithi183.stdout: "primary_temp": [], 2024-09-10T19:17:39.650 INFO:teuthology.orchestra.run.smithi183.stdout: "blocklist": { 2024-09-10T19:17:39.650 INFO:teuthology.orchestra.run.smithi183.stdout: "172.21.15.183:6801/3311783362": "2024-09-11T19:10:43.576943+0000", 2024-09-10T19:17:39.650 INFO:teuthology.orchestra.run.smithi183.stdout: "172.21.15.183:0/4088553922": "2024-09-11T19:10:43.576943+0000", 2024-09-10T19:17:39.650 INFO:teuthology.orchestra.run.smithi183.stdout: "172.21.15.183:0/3528769414": "2024-09-11T19:05:48.153879+0000", 2024-09-10T19:17:39.650 INFO:teuthology.orchestra.run.smithi183.stdout: "172.21.15.183:0/1766036658": "2024-09-11T19:05:48.153879+0000", 2024-09-10T19:17:39.650 INFO:teuthology.orchestra.run.smithi183.stdout: "172.21.15.183:0/3044170088": "2024-09-11T19:05:25.702354+0000", 2024-09-10T19:17:39.650 INFO:teuthology.orchestra.run.smithi183.stdout: "172.21.15.183:0/3037333778": "2024-09-11T19:05:25.702354+0000", 2024-09-10T19:17:39.650 INFO:teuthology.orchestra.run.smithi183.stdout: "172.21.15.183:6801/3747320570": "2024-09-11T19:05:25.702354+0000", 2024-09-10T19:17:39.650 INFO:teuthology.orchestra.run.smithi183.stdout: "172.21.15.183:0/3982386906": "2024-09-11T19:10:43.576943+0000", 2024-09-10T19:17:39.650 INFO:teuthology.orchestra.run.smithi183.stdout: "172.21.15.183:0/2436014455": "2024-09-11T19:05:25.702354+0000", 2024-09-10T19:17:39.650 INFO:teuthology.orchestra.run.smithi183.stdout: "172.21.15.183:6801/659745765": "2024-09-11T19:05:48.153879+0000", 2024-09-10T19:17:39.650 INFO:teuthology.orchestra.run.smithi183.stdout: "172.21.15.183:6800/3747320570": "2024-09-11T19:05:25.702354+0000", 2024-09-10T19:17:39.650 INFO:teuthology.orchestra.run.smithi183.stdout: "172.21.15.183:0/1652576282": "2024-09-11T19:10:43.576943+0000", 2024-09-10T19:17:39.651 INFO:teuthology.orchestra.run.smithi183.stdout: "172.21.15.183:6800/3311783362": "2024-09-11T19:10:43.576943+0000", 2024-09-10T19:17:39.651 INFO:teuthology.orchestra.run.smithi183.stdout: "172.21.15.183:0/1012899549": "2024-09-11T19:05:48.153879+0000", 2024-09-10T19:17:39.651 INFO:teuthology.orchestra.run.smithi183.stdout: "172.21.15.183:6800/659745765": "2024-09-11T19:05:48.153879+0000", 2024-09-10T19:17:39.651 INFO:teuthology.orchestra.run.smithi183.stdout: "172.21.15.183:0/412044400": "2024-09-11T19:10:43.576943+0000", 2024-09-10T19:17:39.651 INFO:teuthology.orchestra.run.smithi183.stdout: "172.21.15.183:0/3636171401": "2024-09-11T19:10:43.576943+0000" 2024-09-10T19:17:39.651 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:17:39.651 INFO:teuthology.orchestra.run.smithi183.stdout: "range_blocklist": {}, 2024-09-10T19:17:39.651 INFO:teuthology.orchestra.run.smithi183.stdout: "erasure_code_profiles": { 2024-09-10T19:17:39.651 INFO:teuthology.orchestra.run.smithi183.stdout: "default": { 2024-09-10T19:17:39.651 INFO:teuthology.orchestra.run.smithi183.stdout: "crush-failure-domain": "osd", 2024-09-10T19:17:39.651 INFO:teuthology.orchestra.run.smithi183.stdout: "k": "2", 2024-09-10T19:17:39.651 INFO:teuthology.orchestra.run.smithi183.stdout: "m": "1", 2024-09-10T19:17:39.651 INFO:teuthology.orchestra.run.smithi183.stdout: "plugin": "jerasure", 2024-09-10T19:17:39.651 INFO:teuthology.orchestra.run.smithi183.stdout: "technique": "reed_sol_van" 2024-09-10T19:17:39.651 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:17:39.652 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:17:39.652 INFO:teuthology.orchestra.run.smithi183.stdout: "removed_snaps_queue": [], 2024-09-10T19:17:39.652 INFO:teuthology.orchestra.run.smithi183.stdout: "new_removed_snaps": [], 2024-09-10T19:17:39.652 INFO:teuthology.orchestra.run.smithi183.stdout: "new_purged_snaps": [], 2024-09-10T19:17:39.652 INFO:teuthology.orchestra.run.smithi183.stdout: "crush_node_flags": {}, 2024-09-10T19:17:39.652 INFO:teuthology.orchestra.run.smithi183.stdout: "device_class_flags": {}, 2024-09-10T19:17:39.652 INFO:teuthology.orchestra.run.smithi183.stdout: "stretch_mode": { 2024-09-10T19:17:39.652 INFO:teuthology.orchestra.run.smithi183.stdout: "stretch_mode_enabled": false, 2024-09-10T19:17:39.652 INFO:teuthology.orchestra.run.smithi183.stdout: "stretch_bucket_count": 0, 2024-09-10T19:17:39.652 INFO:teuthology.orchestra.run.smithi183.stdout: "degraded_stretch_mode": 0, 2024-09-10T19:17:39.652 INFO:teuthology.orchestra.run.smithi183.stdout: "recovering_stretch_mode": 0, 2024-09-10T19:17:39.652 INFO:teuthology.orchestra.run.smithi183.stdout: "stretch_mode_bucket": 0 2024-09-10T19:17:39.652 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:17:39.652 INFO:teuthology.orchestra.run.smithi183.stdout:} 2024-09-10T19:17:39.681 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:17:39.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:17:39.682 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:17:39.716 INFO:journalctl@ceph.iscsi.iscsi.a.smithi191.stdout:Sep 10 19:17:39 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-iscsi-iscsi-a[61816]: debug there is no tcmu-runner data available 2024-09-10T19:17:39.782 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:17:39.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:17:39.819 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:17:39.820 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:17:39.829 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: 75 16 104141 104125 86.7602 61.25 0.00651132 0.0115111 2024-09-10T19:17:39.920 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:17:39.956 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:17:39.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:17:39.958 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:17:40.059 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:17:40.094 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:17:40.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:17:40.095 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:17:40.196 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:17:40.223 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:17:40.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:17:40.225 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:17:40.260 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:17:39 smithi183 ceph-mon[29736]: from='client.? 172.21.15.183:0/1248905155' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-09-10T19:17:40.260 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:17:39 smithi183 ceph-mon[29736]: osdmap e184: 8 total, 8 up, 6 in 2024-09-10T19:17:40.260 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:17:39 smithi183 ceph-mon[41562]: from='client.? 172.21.15.183:0/1248905155' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-09-10T19:17:40.260 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:17:39 smithi183 ceph-mon[41562]: osdmap e184: 8 total, 8 up, 6 in 2024-09-10T19:17:40.325 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:17:40.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:17:40.362 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:17:40.362 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:17:40.463 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:17:40.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:17:39 smithi191 ceph-mon[32884]: from='client.? 172.21.15.183:0/1248905155' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-09-10T19:17:40.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:17:39 smithi191 ceph-mon[32884]: osdmap e184: 8 total, 8 up, 6 in 2024-09-10T19:17:40.491 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:17:40.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:17:40.492 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:17:40.593 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:17:40.621 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:17:40.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:17:40.622 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:17:40.722 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:17:40.764 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:17:40.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:17:40.765 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:17:40.830 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: 76 16 105172 105156 86.4664 64.4375 0.00336912 0.0115575 2024-09-10T19:17:40.866 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:17:40.892 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:17:40.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:17:40.893 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:17:40.994 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:17:41.028 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:17:41.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:17:41.029 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:17:41.130 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:17:41.166 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:17:41.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:17:41.167 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:17:41.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:17:40 smithi183 ceph-mon[29736]: from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:17:41.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:17:40 smithi183 ceph-mon[29736]: pgmap v321: 158 pgs: 1 active+recovering+remapped, 4 active+remapped+backfilling, 5 active+remapped+backfill_toofull, 1 active+recovering, 147 active+clean; 6.0 GiB data, 20 GiB used, 516 GiB / 536 GiB avail; 639 B/s rd, 42 MiB/s wr, 673 op/s; 41225/296832 objects misplaced (13.888%); 29 MiB/s, 464 objects/s recovering 2024-09-10T19:17:41.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:17:40 smithi183 ceph-mon[29736]: purged_snaps scrub starts 2024-09-10T19:17:41.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:17:40 smithi183 ceph-mon[29736]: purged_snaps scrub ok 2024-09-10T19:17:41.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:17:40 smithi183 ceph-mon[29736]: osdmap e185: 8 total, 8 up, 6 in 2024-09-10T19:17:41.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:17:40 smithi183 ceph-mon[41562]: from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:17:41.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:17:40 smithi183 ceph-mon[41562]: pgmap v321: 158 pgs: 1 active+recovering+remapped, 4 active+remapped+backfilling, 5 active+remapped+backfill_toofull, 1 active+recovering, 147 active+clean; 6.0 GiB data, 20 GiB used, 516 GiB / 536 GiB avail; 639 B/s rd, 42 MiB/s wr, 673 op/s; 41225/296832 objects misplaced (13.888%); 29 MiB/s, 464 objects/s recovering 2024-09-10T19:17:41.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:17:40 smithi183 ceph-mon[41562]: purged_snaps scrub starts 2024-09-10T19:17:41.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:17:40 smithi183 ceph-mon[41562]: purged_snaps scrub ok 2024-09-10T19:17:41.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:17:40 smithi183 ceph-mon[41562]: osdmap e185: 8 total, 8 up, 6 in 2024-09-10T19:17:41.268 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:17:41.297 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:17:41.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:17:41.298 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:17:41.399 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:17:41.428 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:17:41.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:17:41.429 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:17:41.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:17:40 smithi191 ceph-mon[32884]: from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:17:41.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:17:40 smithi191 ceph-mon[32884]: pgmap v321: 158 pgs: 1 active+recovering+remapped, 4 active+remapped+backfilling, 5 active+remapped+backfill_toofull, 1 active+recovering, 147 active+clean; 6.0 GiB data, 20 GiB used, 516 GiB / 536 GiB avail; 639 B/s rd, 42 MiB/s wr, 673 op/s; 41225/296832 objects misplaced (13.888%); 29 MiB/s, 464 objects/s recovering 2024-09-10T19:17:41.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:17:40 smithi191 ceph-mon[32884]: purged_snaps scrub starts 2024-09-10T19:17:41.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:17:40 smithi191 ceph-mon[32884]: purged_snaps scrub ok 2024-09-10T19:17:41.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:17:40 smithi191 ceph-mon[32884]: osdmap e185: 8 total, 8 up, 6 in 2024-09-10T19:17:41.529 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:17:41.543 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:17:41.561 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:17:41.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:17:41.562 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:17:41.648 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Sep 10 19:17:41 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-y[29989]: ::ffff:172.21.15.191 - - [10/Sep/2024:19:17:41] "GET /metrics HTTP/1.1" 200 224228 "" "Prometheus/2.43.0" 2024-09-10T19:17:41.663 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:17:41.706 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:17:41.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:17:41.707 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:17:41.808 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:17:41.829 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: 77 16 106239 106223 86.2095 66.6875 0.0117959 0.0115918 2024-09-10T19:17:41.836 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:17:41.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:17:41.837 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:17:41.938 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:17:41.964 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:17:41.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:17:41.965 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:17:42.034 INFO:tasks.thrashosds.thrasher:j is {'epoch': 183, 'fsid': '320f6df8-6fa7-11ef-bcea-c7b262605968', 'created': '2024-09-10T19:05:03.688531+0000', 'modified': '2024-09-10T19:17:35.958226+0000', 'last_up_change': '2024-09-10T19:15:19.923894+0000', 'last_in_change': '2024-09-10T19:16:23.815531+0000', 'flags': 'noscrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5801984, 'flags_set': ['noscrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 19, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 8, 'max_osd': 8, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'quincy', 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-10T19:08:00.456786+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}, {'pool': 2, 'pool_name': '.rgw.root', 'create_time': '2024-09-10T19:10:05.993763+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '60', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}}, {'pool': 3, 'pool_name': 'datapool', 'create_time': '2024-09-10T19:10:06.224267+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 3, 'pg_placement_num': 3, 'pg_placement_num_target': 3, 'pg_num_target': 3, 'pg_num_pending': 3, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '63', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 3, 'snap_epoch': 63, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 4, 'pool_name': 'default.rgw.log', 'create_time': '2024-09-10T19:10:07.958284+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '62', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}}, {'pool': 5, 'pool_name': 'default.rgw.control', 'create_time': '2024-09-10T19:10:09.933596+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '64', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}}, {'pool': 6, 'pool_name': 'default.rgw.meta', 'create_time': '2024-09-10T19:10:11.993712+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '66', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4}, 'application_metadata': {'rgw': {}}}, {'pool': 8, 'pool_name': 'unique_pool_1', 'create_time': '2024-09-10T19:16:11.474472+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 26, 'pg_placement_num': 26, 'pg_placement_num_target': 26, 'pg_num_target': 26, 'pg_num_pending': 26, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '179', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '177', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}}], 'osds': [{'osd': 0, 'uuid': '9d323010-394a-49aa-ba9b-244d87faf36e', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 8, 'up_thru': 180, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.183:6802', 'nonce': 2813074952}, {'type': 'v1', 'addr': '172.21.15.183:6803', 'nonce': 2813074952}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.183:6804', 'nonce': 2813074952}, {'type': 'v1', 'addr': '172.21.15.183:6805', 'nonce': 2813074952}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.183:6808', 'nonce': 2813074952}, {'type': 'v1', 'addr': '172.21.15.183:6809', 'nonce': 2813074952}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.183:6806', 'nonce': 2813074952}, {'type': 'v1', 'addr': '172.21.15.183:6807', 'nonce': 2813074952}]}, 'public_addr': '172.21.15.183:6803/2813074952', 'cluster_addr': '172.21.15.183:6805/2813074952', 'heartbeat_back_addr': '172.21.15.183:6809/2813074952', 'heartbeat_front_addr': '172.21.15.183:6807/2813074952', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': '71a6c5dc-5109-4284-8d20-187d8e7afc9e', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 1, 'last_clean_begin': 13, 'last_clean_end': 98, 'up_from': 138, 'up_thru': 86, 'down_at': 99, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.183:6810', 'nonce': 219854798}, {'type': 'v1', 'addr': '172.21.15.183:6811', 'nonce': 219854798}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.183:6812', 'nonce': 219854798}, {'type': 'v1', 'addr': '172.21.15.183:6813', 'nonce': 219854798}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.183:6816', 'nonce': 219854798}, {'type': 'v1', 'addr': '172.21.15.183:6817', 'nonce': 219854798}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.183:6814', 'nonce': 219854798}, {'type': 'v1', 'addr': '172.21.15.183:6815', 'nonce': 219854798}]}, 'public_addr': '172.21.15.183:6811/219854798', 'cluster_addr': '172.21.15.183:6813/219854798', 'heartbeat_back_addr': '172.21.15.183:6817/219854798', 'heartbeat_front_addr': '172.21.15.183:6815/219854798', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': 'fc310d5b-b89b-4655-9989-7e10ba356509', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 18, 'up_thru': 148, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.183:6818', 'nonce': 3730013786}, {'type': 'v1', 'addr': '172.21.15.183:6819', 'nonce': 3730013786}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.183:6820', 'nonce': 3730013786}, {'type': 'v1', 'addr': '172.21.15.183:6821', 'nonce': 3730013786}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.183:6824', 'nonce': 3730013786}, {'type': 'v1', 'addr': '172.21.15.183:6825', 'nonce': 3730013786}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.183:6822', 'nonce': 3730013786}, {'type': 'v1', 'addr': '172.21.15.183:6823', 'nonce': 3730013786}]}, 'public_addr': '172.21.15.183:6819/3730013786', 'cluster_addr': '172.21.15.183:6821/3730013786', 'heartbeat_back_addr': '172.21.15.183:6825/3730013786', 'heartbeat_front_addr': '172.21.15.183:6823/3730013786', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': '7fd07a95-ad22-43f5-a968-539552283e89', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 26, 'up_thru': 180, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.183:6826', 'nonce': 3522836984}, {'type': 'v1', 'addr': '172.21.15.183:6827', 'nonce': 3522836984}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.183:6828', 'nonce': 3522836984}, {'type': 'v1', 'addr': '172.21.15.183:6829', 'nonce': 3522836984}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.183:6832', 'nonce': 3522836984}, {'type': 'v1', 'addr': '172.21.15.183:6833', 'nonce': 3522836984}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.183:6830', 'nonce': 3522836984}, {'type': 'v1', 'addr': '172.21.15.183:6831', 'nonce': 3522836984}]}, 'public_addr': '172.21.15.183:6827/3522836984', 'cluster_addr': '172.21.15.183:6829/3522836984', 'heartbeat_back_addr': '172.21.15.183:6833/3522836984', 'heartbeat_front_addr': '172.21.15.183:6831/3522836984', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': '631083fb-e503-44da-9b6b-154e88539ee7', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 32, 'up_thru': 180, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.191:6800', 'nonce': 2674828539}, {'type': 'v1', 'addr': '172.21.15.191:6801', 'nonce': 2674828539}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.191:6802', 'nonce': 2674828539}, {'type': 'v1', 'addr': '172.21.15.191:6803', 'nonce': 2674828539}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.191:6806', 'nonce': 2674828539}, {'type': 'v1', 'addr': '172.21.15.191:6807', 'nonce': 2674828539}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.191:6804', 'nonce': 2674828539}, {'type': 'v1', 'addr': '172.21.15.191:6805', 'nonce': 2674828539}]}, 'public_addr': '172.21.15.191:6801/2674828539', 'cluster_addr': '172.21.15.191:6803/2674828539', 'heartbeat_back_addr': '172.21.15.191:6807/2674828539', 'heartbeat_front_addr': '172.21.15.191:6805/2674828539', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': '6b963695-3eca-4e6e-9207-05b8010ebff9', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 39, 'up_thru': 180, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.191:6808', 'nonce': 681733378}, {'type': 'v1', 'addr': '172.21.15.191:6809', 'nonce': 681733378}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.191:6810', 'nonce': 681733378}, {'type': 'v1', 'addr': '172.21.15.191:6811', 'nonce': 681733378}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.191:6814', 'nonce': 681733378}, {'type': 'v1', 'addr': '172.21.15.191:6815', 'nonce': 681733378}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.191:6812', 'nonce': 681733378}, {'type': 'v1', 'addr': '172.21.15.191:6813', 'nonce': 681733378}]}, 'public_addr': '172.21.15.191:6809/681733378', 'cluster_addr': '172.21.15.191:6811/681733378', 'heartbeat_back_addr': '172.21.15.191:6815/681733378', 'heartbeat_front_addr': '172.21.15.191:6813/681733378', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': '2dc9754f-6aef-4e19-8605-2f1591921811', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 47, 'up_thru': 179, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.191:6816', 'nonce': 2825813477}, {'type': 'v1', 'addr': '172.21.15.191:6817', 'nonce': 2825813477}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.191:6818', 'nonce': 2825813477}, {'type': 'v1', 'addr': '172.21.15.191:6819', 'nonce': 2825813477}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.191:6822', 'nonce': 2825813477}, {'type': 'v1', 'addr': '172.21.15.191:6823', 'nonce': 2825813477}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.191:6820', 'nonce': 2825813477}, {'type': 'v1', 'addr': '172.21.15.191:6821', 'nonce': 2825813477}]}, 'public_addr': '172.21.15.191:6817/2825813477', 'cluster_addr': '172.21.15.191:6819/2825813477', 'heartbeat_back_addr': '172.21.15.191:6823/2825813477', 'heartbeat_front_addr': '172.21.15.191:6821/2825813477', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': '1a9a299e-d5ba-4b9a-85d5-7f6a5bc754e1', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 54, 'up_thru': 180, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.191:6824', 'nonce': 1422610021}, {'type': 'v1', 'addr': '172.21.15.191:6825', 'nonce': 1422610021}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.191:6826', 'nonce': 1422610021}, {'type': 'v1', 'addr': '172.21.15.191:6827', 'nonce': 1422610021}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.191:6830', 'nonce': 1422610021}, {'type': 'v1', 'addr': '172.21.15.191:6831', 'nonce': 1422610021}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.191:6828', 'nonce': 1422610021}, {'type': 'v1', 'addr': '172.21.15.191:6829', 'nonce': 1422610021}]}, 'public_addr': '172.21.15.191:6825/1422610021', 'cluster_addr': '172.21.15.191:6827/1422610021', 'heartbeat_back_addr': '172.21.15.191:6831/1422610021', 'heartbeat_front_addr': '172.21.15.191:6829/1422610021', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138320759226367, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-10T19:16:46.494740+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138320759226367, 'old_weight': 65536, 'last_purged_snaps_scrub': '2024-09-10T19:16:25.059015+0000', 'dead_epoch': 98}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138320759226367, 'old_weight': 65536, 'last_purged_snaps_scrub': '2024-09-10T19:17:07.299690+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138320759226367, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-10T19:17:09.095005+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138320759226367, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-10T19:16:27.222375+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138320759226367, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-10T19:17:28.197628+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138320759226367, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-10T19:17:15.672700+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138320759226367, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-10T19:17:35.086065+0000', 'dead_epoch': 0}], 'pg_upmap': [{'pgid': '8.3', 'osds': [7, 3, 5]}], 'pg_upmap_items': [{'pgid': '4.1', 'mappings': [{'from': 6, 'to': 7}]}, {'pgid': '4.11', 'mappings': [{'from': 3, 'to': 5}]}], 'pg_temp': [{'pgid': '8.10', 'osds': [3, 5, 4]}, {'pgid': '8.11', 'osds': [5, 0, 3]}, {'pgid': '8.12', 'osds': [0, 4, 5]}, {'pgid': '8.13', 'osds': [3, 5, 7]}, {'pgid': '8.14', 'osds': [4, 5, 6]}, {'pgid': '8.15', 'osds': [7, 6, 4]}, {'pgid': '8.16', 'osds': [0, 6, 7]}, {'pgid': '8.17', 'osds': [3, 6, 4]}, {'pgid': '8.18', 'osds': [3, 5, 7]}, {'pgid': '8.19', 'osds': [5, 0, 6]}], 'primary_temp': [], 'blocklist': {'172.21.15.183:6801/3311783362': '2024-09-11T19:10:43.576943+0000', '172.21.15.183:0/4088553922': '2024-09-11T19:10:43.576943+0000', '172.21.15.183:0/3528769414': '2024-09-11T19:05:48.153879+0000', '172.21.15.183:0/1766036658': '2024-09-11T19:05:48.153879+0000', '172.21.15.183:0/3044170088': '2024-09-11T19:05:25.702354+0000', '172.21.15.183:0/3037333778': '2024-09-11T19:05:25.702354+0000', '172.21.15.183:6801/3747320570': '2024-09-11T19:05:25.702354+0000', '172.21.15.183:0/3982386906': '2024-09-11T19:10:43.576943+0000', '172.21.15.183:0/2436014455': '2024-09-11T19:05:25.702354+0000', '172.21.15.183:6801/659745765': '2024-09-11T19:05:48.153879+0000', '172.21.15.183:6800/3747320570': '2024-09-11T19:05:25.702354+0000', '172.21.15.183:0/1652576282': '2024-09-11T19:10:43.576943+0000', '172.21.15.183:6800/3311783362': '2024-09-11T19:10:43.576943+0000', '172.21.15.183:0/1012899549': '2024-09-11T19:05:48.153879+0000', '172.21.15.183:6800/659745765': '2024-09-11T19:05:48.153879+0000', '172.21.15.183:0/412044400': '2024-09-11T19:10:43.576943+0000', '172.21.15.183:0/3636171401': '2024-09-11T19:10:43.576943+0000'}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-09-10T19:17:42.034 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph pg dump --format=json 2024-09-10T19:17:42.066 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:17:42.107 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:17:42.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:17:42.107 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:17:42.208 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:17:42.239 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:17:42.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:17:42.240 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:17:42.341 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:17:42.370 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:17:42.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:17:42.371 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:17:42.472 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:17:42.506 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:17:42.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:17:42.507 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:17:42.607 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:17:42.635 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:17:42.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:17:42.636 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:17:42.733 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph daemon osd.0 dump_ops_in_flight 2024-09-10T19:17:42.737 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:17:42.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:17:42.782 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:17:42.782 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:17:42.829 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: 78 16 107380 107364 86.0184 71.3125 0.00602202 0.0116186 2024-09-10T19:17:42.883 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:17:42.913 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:17:42.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:17:42.914 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:17:43.015 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:17:43.045 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:17:43.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:17:43.046 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:17:43.147 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:17:43.186 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:17:43.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:17:43.187 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:17:43.288 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:17:43.291 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:17:42 smithi183 ceph-mon[29736]: pgmap v324: 158 pgs: 1 active+recovering+remapped, 4 active+remapped+backfilling, 1 active+recovering, 5 active+remapped+backfill_toofull, 147 active+clean; 6.4 GiB data, 21 GiB used, 515 GiB / 536 GiB avail; 853 B/s rd, 57 MiB/s wr, 912 op/s; 38856/313239 objects misplaced (12.405%); 40 MiB/s, 635 objects/s recovering 2024-09-10T19:17:43.292 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:17:42 smithi183 ceph-mon[41562]: pgmap v324: 158 pgs: 1 active+recovering+remapped, 4 active+remapped+backfilling, 1 active+recovering, 5 active+remapped+backfill_toofull, 147 active+clean; 6.4 GiB data, 21 GiB used, 515 GiB / 536 GiB avail; 853 B/s rd, 57 MiB/s wr, 912 op/s; 38856/313239 objects misplaced (12.405%); 40 MiB/s, 635 objects/s recovering 2024-09-10T19:17:43.320 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:17:43.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:17:43.322 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:17:43.422 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:17:43.451 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:17:43.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:17:43.452 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:17:43.465 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:17:42 smithi191 ceph-mon[32884]: pgmap v324: 158 pgs: 1 active+recovering+remapped, 4 active+remapped+backfilling, 1 active+recovering, 5 active+remapped+backfill_toofull, 147 active+clean; 6.4 GiB data, 21 GiB used, 515 GiB / 536 GiB avail; 853 B/s rd, 57 MiB/s wr, 912 op/s; 38856/313239 objects misplaced (12.405%); 40 MiB/s, 635 objects/s recovering 2024-09-10T19:17:43.553 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:17:43.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:17:43.590 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:17:43.590 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:17:43.691 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:17:43.728 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:17:43.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:17:43.729 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:17:43.829 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: 79 16 108384 108368 85.7238 62.75 0.00517579 0.0116551 2024-09-10T19:17:43.830 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:17:43.854 INFO:teuthology.orchestra.run.smithi183.stdout:osd.0: {} 2024-09-10T19:17:43.854 INFO:teuthology.orchestra.run.smithi183.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-10T19:17:43.861 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:17:43.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:17:43.862 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:17:43.868 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:17:43.884 INFO:teuthology.orchestra.run.smithi183.stdout:osd.1: {} 2024-09-10T19:17:43.884 INFO:teuthology.orchestra.run.smithi183.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-10T19:17:43.909 INFO:teuthology.orchestra.run.smithi183.stdout:osd.2: {} 2024-09-10T19:17:43.910 INFO:teuthology.orchestra.run.smithi183.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-10T19:17:43.936 INFO:teuthology.orchestra.run.smithi183.stdout:osd.3: {} 2024-09-10T19:17:43.936 INFO:teuthology.orchestra.run.smithi183.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-10T19:17:43.960 INFO:teuthology.orchestra.run.smithi183.stdout:osd.4: {} 2024-09-10T19:17:43.961 INFO:teuthology.orchestra.run.smithi183.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-10T19:17:43.963 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:17:43.988 INFO:teuthology.orchestra.run.smithi183.stdout:osd.5: {} 2024-09-10T19:17:43.989 INFO:teuthology.orchestra.run.smithi183.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-10T19:17:43.994 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:17:43.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:17:43.995 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:17:44.012 INFO:teuthology.orchestra.run.smithi183.stdout:osd.6: {} 2024-09-10T19:17:44.013 INFO:teuthology.orchestra.run.smithi183.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-10T19:17:44.037 INFO:teuthology.orchestra.run.smithi183.stdout:osd.7: {} 2024-09-10T19:17:44.037 INFO:teuthology.orchestra.run.smithi183.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-10T19:17:44.096 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:17:44.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:17:44.134 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:17:44.134 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:17:44.235 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:17:44.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:17:44.271 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:17:44.271 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:17:44.371 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:17:44.409 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:17:44.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:17:44.410 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:17:44.511 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:17:44.539 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:17:44.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:17:44.540 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:17:44.641 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:17:44.677 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:17:44.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:17:44.678 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:17:44.679 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:17:44 smithi183 ceph-mon[29736]: from='mgr.14556 ' entity='mgr.y' 2024-09-10T19:17:44.679 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:17:44 smithi183 ceph-mon[29736]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:17:44.679 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:17:44 smithi183 ceph-mon[29736]: pgmap v325: 158 pgs: 1 active+recovering+remapped, 4 active+remapped+backfilling, 1 active+recovering, 5 active+remapped+backfill_toofull, 147 active+clean; 6.4 GiB data, 21 GiB used, 515 GiB / 536 GiB avail; 659 B/s rd, 44 MiB/s wr, 704 op/s; 38856/313239 objects misplaced (12.405%); 31 MiB/s, 491 objects/s recovering 2024-09-10T19:17:44.679 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:17:44 smithi183 ceph-mon[29736]: from='client.? 172.21.15.183:0/3581532380' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T19:17:44.679 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:17:44 smithi183 ceph-mon[41562]: from='mgr.14556 ' entity='mgr.y' 2024-09-10T19:17:44.679 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:17:44 smithi183 ceph-mon[41562]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:17:44.679 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:17:44 smithi183 ceph-mon[41562]: pgmap v325: 158 pgs: 1 active+recovering+remapped, 4 active+remapped+backfilling, 1 active+recovering, 5 active+remapped+backfill_toofull, 147 active+clean; 6.4 GiB data, 21 GiB used, 515 GiB / 536 GiB avail; 659 B/s rd, 44 MiB/s wr, 704 op/s; 38856/313239 objects misplaced (12.405%); 31 MiB/s, 491 objects/s recovering 2024-09-10T19:17:44.680 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:17:44 smithi183 ceph-mon[41562]: from='client.? 172.21.15.183:0/3581532380' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T19:17:44.778 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:17:44.805 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:17:44.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:17:44.806 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:17:44.829 INFO:tasks.radosbench.radosbench.0.smithi183.stdout:2024-09-10T19:17:44.829185+0000 min lat: 0.0011643 max lat: 3.03886 avg lat: 0.0116969 2024-09-10T19:17:44.829 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-10T19:17:44.830 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: 80 16 109371 109355 85.4233 61.6875 0.00314397 0.0116969 2024-09-10T19:17:44.906 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:17:44.938 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:17:44.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:17:44.939 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:17:44.965 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:17:44 smithi191 ceph-mon[32884]: from='mgr.14556 ' entity='mgr.y' 2024-09-10T19:17:44.965 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:17:44 smithi191 ceph-mon[32884]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:17:44.965 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:17:44 smithi191 ceph-mon[32884]: pgmap v325: 158 pgs: 1 active+recovering+remapped, 4 active+remapped+backfilling, 1 active+recovering, 5 active+remapped+backfill_toofull, 147 active+clean; 6.4 GiB data, 21 GiB used, 515 GiB / 536 GiB avail; 659 B/s rd, 44 MiB/s wr, 704 op/s; 38856/313239 objects misplaced (12.405%); 31 MiB/s, 491 objects/s recovering 2024-09-10T19:17:44.966 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:17:44 smithi191 ceph-mon[32884]: from='client.? 172.21.15.183:0/3581532380' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T19:17:45.040 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:17:45.065 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:17:45.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:17:45.066 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:17:45.166 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:17:45.191 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:17:45.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:17:45.192 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:17:45.292 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:17:45.324 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:17:45.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:17:45.325 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:17:45.426 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:17:45.454 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:17:45.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:17:45.456 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:17:45.556 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:17:45.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:17:45.582 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:17:45.582 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:17:45.683 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:17:45.718 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:17:45.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:17:45.719 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:17:45.820 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:17:45.829 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: 81 15 109973 109958 84.834 37.6875 0.00590232 0.0117811 2024-09-10T19:17:45.849 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:17:45.855 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:17:45.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:17:45.856 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:17:45.931 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:17:45 smithi183 ceph-mon[29736]: osdmap e186: 8 total, 8 up, 6 in 2024-09-10T19:17:45.931 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:17:45 smithi183 ceph-mon[41562]: osdmap e186: 8 total, 8 up, 6 in 2024-09-10T19:17:45.957 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:17:45.966 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:17:45 smithi191 ceph-mon[32884]: osdmap e186: 8 total, 8 up, 6 in 2024-09-10T19:17:45.987 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:17:45.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:17:45.988 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:17:46.089 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:17:46.115 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:17:46.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:17:46.115 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:17:46.216 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:17:46.242 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:17:46.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:17:46.243 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:17:46.344 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:17:46.368 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:17:46.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:17:46.369 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:17:46.470 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:17:46.495 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:17:46.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:17:46.496 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:17:46.596 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:17:46.618 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:17:46.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:17:46.619 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:17:46.688 INFO:teuthology.orchestra.run.smithi183.stderr:nodeep-scrub is set 2024-09-10T19:17:46.720 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:17:46.749 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:17:46.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:17:46.750 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:17:46.829 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: 82 16 110501 110485 84.2011 32.9375 0.00757215 0.0118705 2024-09-10T19:17:46.851 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:17:46.882 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:17:46.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:17:46.883 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:17:46.931 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:17:46 smithi183 ceph-mon[29736]: osdmap e187: 8 total, 8 up, 6 in 2024-09-10T19:17:46.931 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:17:46 smithi183 ceph-mon[29736]: pgmap v328: 158 pgs: 1 peering, 1 active+recovering+remapped, 3 active+remapped+backfilling, 1 active+recovering, 5 active+remapped+backfill_toofull, 147 active+clean; 6.7 GiB data, 23 GiB used, 514 GiB / 536 GiB avail; 1.7 KiB/s rd, 108 MiB/s wr, 1.73k op/s; 35897/328806 objects misplaced (10.917%); 75 MiB/s, 1.19k objects/s recovering 2024-09-10T19:17:46.931 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:17:46 smithi183 ceph-mon[29736]: from='client.? 172.21.15.183:0/3752793790' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-10T19:17:46.931 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:17:46 smithi183 ceph-mon[29736]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-10T19:17:46.931 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:17:46 smithi183 ceph-mon[41562]: osdmap e187: 8 total, 8 up, 6 in 2024-09-10T19:17:46.931 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:17:46 smithi183 ceph-mon[41562]: pgmap v328: 158 pgs: 1 peering, 1 active+recovering+remapped, 3 active+remapped+backfilling, 1 active+recovering, 5 active+remapped+backfill_toofull, 147 active+clean; 6.7 GiB data, 23 GiB used, 514 GiB / 536 GiB avail; 1.7 KiB/s rd, 108 MiB/s wr, 1.73k op/s; 35897/328806 objects misplaced (10.917%); 75 MiB/s, 1.19k objects/s recovering 2024-09-10T19:17:46.931 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:17:46 smithi183 ceph-mon[41562]: from='client.? 172.21.15.183:0/3752793790' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-10T19:17:46.932 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:17:46 smithi183 ceph-mon[41562]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-10T19:17:46.965 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:17:46 smithi191 ceph-mon[32884]: osdmap e187: 8 total, 8 up, 6 in 2024-09-10T19:17:46.966 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:17:46 smithi191 ceph-mon[32884]: pgmap v328: 158 pgs: 1 peering, 1 active+recovering+remapped, 3 active+remapped+backfilling, 1 active+recovering, 5 active+remapped+backfill_toofull, 147 active+clean; 6.7 GiB data, 23 GiB used, 514 GiB / 536 GiB avail; 1.7 KiB/s rd, 108 MiB/s wr, 1.73k op/s; 35897/328806 objects misplaced (10.917%); 75 MiB/s, 1.19k objects/s recovering 2024-09-10T19:17:46.966 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:17:46 smithi191 ceph-mon[32884]: from='client.? 172.21.15.183:0/3752793790' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-10T19:17:46.966 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:17:46 smithi191 ceph-mon[32884]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-10T19:17:46.984 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:17:47.020 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:17:47.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:17:47.021 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:17:47.122 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:17:47.147 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:17:47.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:17:47.149 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:17:47.250 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:17:47.275 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:17:47.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:17:47.277 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:17:47.378 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:17:47.416 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:17:47.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:17:47.417 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:17:47.518 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:17:47.545 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:17:47.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:17:47.546 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:17:47.647 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:17:47.675 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:17:47.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:17:47.676 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:17:47.715 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:17:47 smithi191 ceph-mon[32884]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T19:17:47.715 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:17:47 smithi191 ceph-mon[32884]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-10T19:17:47.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:17:47 smithi191 ceph-mon[32884]: osdmap e188: 8 total, 8 up, 6 in 2024-09-10T19:17:47.716 INFO:journalctl@ceph.mgr.x.smithi191.stdout:Sep 10 19:17:47 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-x[34967]: ::ffff:172.21.15.191 - - [10/Sep/2024:19:17:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T19:17:47.777 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:17:47.802 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:17:47.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:17:47.804 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:17:47.830 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: 83 16 111724 111708 84.1075 76.4375 0.00345209 0.0118764 2024-09-10T19:17:47.904 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:17:47.931 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:17:47 smithi183 ceph-mon[29736]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T19:17:47.931 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:17:47 smithi183 ceph-mon[29736]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-10T19:17:47.931 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:17:47 smithi183 ceph-mon[29736]: osdmap e188: 8 total, 8 up, 6 in 2024-09-10T19:17:47.931 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:17:47 smithi183 ceph-mon[41562]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T19:17:47.931 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:17:47 smithi183 ceph-mon[41562]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-10T19:17:47.932 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:17:47 smithi183 ceph-mon[41562]: osdmap e188: 8 total, 8 up, 6 in 2024-09-10T19:17:47.932 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:17:47.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:17:47.933 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:17:48.034 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:17:48.058 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:17:48.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:17:48.060 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:17:48.161 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:17:48.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:17:48.189 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:17:48.189 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:17:48.290 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:17:48.320 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:17:48.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:17:48.321 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:17:48.400 INFO:teuthology.orchestra.run.smithi183.stdout:{ 2024-09-10T19:17:48.400 INFO:teuthology.orchestra.run.smithi183.stdout: "ops": [ 2024-09-10T19:17:48.401 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:17:48.401 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.15051.0:112384 8.1 8:82baf408:::benchmark_data_smithi183_114924_object112383:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e188)", 2024-09-10T19:17:48.401 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-10T19:17:48.394797+0000", 2024-09-10T19:17:48.401 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 0.003685261, 2024-09-10T19:17:48.401 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.003707629, 2024-09-10T19:17:48.401 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-10T19:17:48.401 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "waiting for sub ops", 2024-09-10T19:17:48.401 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-10T19:17:48.401 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.15051", 2024-09-10T19:17:48.401 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.183:0/3632028672", 2024-09-10T19:17:48.401 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 112384 2024-09-10T19:17:48.402 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:17:48.402 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-10T19:17:48.402 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:17:48.402 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-10T19:17:48.402 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:17:48.394797+0000", 2024-09-10T19:17:48.402 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:17:48.402 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:17:48.402 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:17:48.402 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-10T19:17:48.402 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:17:48.394797+0000", 2024-09-10T19:17:48.402 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:17:48.402 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:17:48.402 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:17:48.402 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-10T19:17:48.402 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:17:48.394796+0000", 2024-09-10T19:17:48.403 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4294967295.999999 2024-09-10T19:17:48.403 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:17:48.403 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:17:48.403 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-10T19:17:48.403 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:17:48.394814+0000", 2024-09-10T19:17:48.403 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.7708000000000001e-05 2024-09-10T19:17:48.403 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:17:48.403 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:17:48.403 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-10T19:17:48.403 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:17:48.394825+0000", 2024-09-10T19:17:48.403 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.1447000000000001e-05 2024-09-10T19:17:48.403 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:17:48.403 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:17:48.403 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-10T19:17:48.404 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:17:48.394831+0000", 2024-09-10T19:17:48.404 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.9549999999999997e-06 2024-09-10T19:17:48.404 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:17:48.404 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:17:48.404 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-10T19:17:48.404 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:17:48.394878+0000", 2024-09-10T19:17:48.404 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.6761999999999997e-05 2024-09-10T19:17:48.404 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:17:48.404 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:17:48.404 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-10T19:17:48.404 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:17:48.395002+0000", 2024-09-10T19:17:48.404 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000123638 2024-09-10T19:17:48.404 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:17:48.404 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:17:48.405 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 3,5", 2024-09-10T19:17:48.405 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:17:48.395335+0000", 2024-09-10T19:17:48.405 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00033288000000000002 2024-09-10T19:17:48.405 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:17:48.405 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:17:48.405 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-10T19:17:48.405 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:17:48.397125+0000", 2024-09-10T19:17:48.405 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.001790547 2024-09-10T19:17:48.405 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:17:48.405 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:17:48.405 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:17:48.405 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:17:48.397302+0000", 2024-09-10T19:17:48.405 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00017695300000000001 2024-09-10T19:17:48.406 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:17:48.406 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:17:48.406 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:17:48.406 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:17:48.406 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:17:48.406 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_repop(client.15051.0:112385 8.9 e188/177 8:9126060e:::benchmark_data_smithi183_114924_object112384:head v 188'6414, mlcod=188'6411)", 2024-09-10T19:17:48.406 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-10T19:17:48.397774+0000", 2024-09-10T19:17:48.406 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 0.000708222, 2024-09-10T19:17:48.406 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00078968099999999998, 2024-09-10T19:17:48.406 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-10T19:17:48.406 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "started", 2024-09-10T19:17:48.406 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-10T19:17:48.406 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:17:48.406 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-10T19:17:48.407 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:17:48.397774+0000", 2024-09-10T19:17:48.407 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:17:48.407 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:17:48.407 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:17:48.407 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-10T19:17:48.407 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:17:48.397774+0000", 2024-09-10T19:17:48.407 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:17:48.407 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:17:48.407 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:17:48.407 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-10T19:17:48.407 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:17:48.397774+0000", 2024-09-10T19:17:48.407 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4294967295.9999995 2024-09-10T19:17:48.407 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:17:48.407 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:17:48.407 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-10T19:17:48.408 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:17:48.397832+0000", 2024-09-10T19:17:48.408 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.8603000000000001e-05 2024-09-10T19:17:48.408 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:17:48.408 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:17:48.408 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-10T19:17:48.408 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:17:48.397840+0000", 2024-09-10T19:17:48.408 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.4080000000000001e-06 2024-09-10T19:17:48.408 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:17:48.408 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:17:48.408 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-10T19:17:48.408 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:17:48.397843+0000", 2024-09-10T19:17:48.408 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.399e-06 2024-09-10T19:17:48.408 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:17:48.408 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:17:48.409 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-10T19:17:48.409 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:17:48.397878+0000", 2024-09-10T19:17:48.409 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.4511000000000002e-05 2024-09-10T19:17:48.409 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:17:48.409 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:17:48.409 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-10T19:17:48.409 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:17:48.397891+0000", 2024-09-10T19:17:48.409 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.328e-05 2024-09-10T19:17:48.409 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:17:48.409 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:17:48.409 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:17:48.409 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:17:48.409 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:17:48.409 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.15051.0:112387 8.1 8:8710e484:::benchmark_data_smithi183_114924_object112386:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e188)", 2024-09-10T19:17:48.409 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-10T19:17:48.398251+0000", 2024-09-10T19:17:48.410 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 0.00023123999999999999, 2024-09-10T19:17:48.410 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000349043, 2024-09-10T19:17:48.410 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-10T19:17:48.410 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "started", 2024-09-10T19:17:48.410 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-10T19:17:48.410 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.15051", 2024-09-10T19:17:48.410 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.183:0/3632028672", 2024-09-10T19:17:48.410 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 112387 2024-09-10T19:17:48.410 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:17:48.410 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-10T19:17:48.410 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:17:48.410 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-10T19:17:48.410 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:17:48.398251+0000", 2024-09-10T19:17:48.410 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:17:48.411 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:17:48.411 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:17:48.411 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-10T19:17:48.411 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:17:48.398251+0000", 2024-09-10T19:17:48.411 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:17:48.411 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:17:48.411 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:17:48.411 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-10T19:17:48.411 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:17:48.398250+0000", 2024-09-10T19:17:48.411 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4294967295.9999986 2024-09-10T19:17:48.411 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:17:48.411 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:17:48.411 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-10T19:17:48.411 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:17:48.398273+0000", 2024-09-10T19:17:48.411 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.3617000000000001e-05 2024-09-10T19:17:48.412 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:17:48.412 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:17:48.412 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-10T19:17:48.412 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:17:48.398286+0000", 2024-09-10T19:17:48.412 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.2499e-05 2024-09-10T19:17:48.412 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:17:48.412 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:17:48.412 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-10T19:17:48.412 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:17:48.398291+0000", 2024-09-10T19:17:48.412 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.6699999999999999e-06 2024-09-10T19:17:48.412 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:17:48.412 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:17:48.412 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-10T19:17:48.413 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:17:48.398333+0000", 2024-09-10T19:17:48.413 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.1631000000000002e-05 2024-09-10T19:17:48.413 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:17:48.413 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:17:48.413 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-10T19:17:48.413 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:17:48.398448+0000", 2024-09-10T19:17:48.413 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00011446300000000001 2024-09-10T19:17:48.413 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:17:48.413 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:17:48.413 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:17:48.413 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:17:48.413 INFO:teuthology.orchestra.run.smithi183.stdout: ], 2024-09-10T19:17:48.413 INFO:teuthology.orchestra.run.smithi183.stdout: "num_ops": 3 2024-09-10T19:17:48.413 INFO:teuthology.orchestra.run.smithi183.stdout:} 2024-09-10T19:17:48.421 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:17:48.451 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:17:48.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:17:48.452 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:17:48.553 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:17:48.587 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:17:48.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:17:48.588 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:17:48.689 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:17:48.724 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:17:48.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:17:48.725 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:17:48.826 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:17:48.830 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: 84 16 112921 112905 83.9968 74.8125 0.010296 0.0118994 2024-09-10T19:17:48.852 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:17:48.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:17:48.853 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:17:48.898 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-10T19:17:48.954 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:17:48.965 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:17:48 smithi191 ceph-mon[32884]: pgmap v330: 158 pgs: 1 peering, 1 active+recovering+remapped, 3 active+remapped+backfilling, 1 active+recovering, 5 active+remapped+backfill_toofull, 147 active+clean; 6.7 GiB data, 23 GiB used, 514 GiB / 536 GiB avail; 853 B/s rd, 54 MiB/s wr, 865 op/s; 35897/328806 objects misplaced (10.917%); 42 MiB/s, 667 objects/s recovering 2024-09-10T19:17:48.983 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:17:48.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:17:48.984 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:17:49.085 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:17:49.120 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:17:49.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:17:49.120 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:17:49.168 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:17:48 smithi183 ceph-mon[29736]: pgmap v330: 158 pgs: 1 peering, 1 active+recovering+remapped, 3 active+remapped+backfilling, 1 active+recovering, 5 active+remapped+backfill_toofull, 147 active+clean; 6.7 GiB data, 23 GiB used, 514 GiB / 536 GiB avail; 853 B/s rd, 54 MiB/s wr, 865 op/s; 35897/328806 objects misplaced (10.917%); 42 MiB/s, 667 objects/s recovering 2024-09-10T19:17:49.168 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:17:48 smithi183 ceph-mon[41562]: pgmap v330: 158 pgs: 1 peering, 1 active+recovering+remapped, 3 active+remapped+backfilling, 1 active+recovering, 5 active+remapped+backfill_toofull, 147 active+clean; 6.7 GiB data, 23 GiB used, 514 GiB / 536 GiB avail; 853 B/s rd, 54 MiB/s wr, 865 op/s; 35897/328806 objects misplaced (10.917%); 42 MiB/s, 667 objects/s recovering 2024-09-10T19:17:49.221 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:17:49.246 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:17:49.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:17:49.247 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:17:49.348 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:17:49.385 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:17:49.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:17:49.386 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:17:49.486 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:17:49.518 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:17:49.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:17:49.519 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:17:49.620 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:17:49.647 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:17:49.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:17:49.648 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:17:49.715 INFO:journalctl@ceph.iscsi.iscsi.a.smithi191.stdout:Sep 10 19:17:49 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-iscsi-iscsi-a[61816]: debug there is no tcmu-runner data available 2024-09-10T19:17:49.749 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:17:49.775 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:17:49.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:17:49.798 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:17:49.830 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: 85 16 114097 114081 83.8732 73.5 0.00559549 0.0119127 2024-09-10T19:17:49.899 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:17:49.934 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:17:49.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:17:49.935 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:17:49.988 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph daemon osd.0 dump_blocked_ops 2024-09-10T19:17:50.036 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:17:50.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:17:50.072 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:17:50.072 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:17:50.173 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:17:50.207 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:17:50.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:17:50.208 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:17:50.309 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:17:50.338 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:17:50.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:17:50.339 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:17:50.440 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:17:50.469 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:17:50.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:17:50.470 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:17:50.571 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:17:50.607 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:17:50.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:17:50.608 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:17:50.709 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:17:50.739 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:17:50.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:17:50.740 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:17:50.830 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: 86 16 115414 115398 83.855 82.3125 0.00721697 0.0119208 2024-09-10T19:17:50.841 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:17:50.874 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:17:50.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:17:50.875 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:17:50.938 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:17:50 smithi183 ceph-mon[29736]: from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:17:50.938 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:17:50 smithi183 ceph-mon[29736]: osdmap e189: 8 total, 8 up, 6 in 2024-09-10T19:17:50.938 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:17:50 smithi183 ceph-mon[29736]: pgmap v332: 158 pgs: 1 peering, 1 active+recovering+remapped, 3 active+remapped+backfilling, 1 active+recovering, 5 active+remapped+backfill_toofull, 147 active+clean; 6.7 GiB data, 23 GiB used, 514 GiB / 536 GiB avail; 1009 B/s rd, 64 MiB/s wr, 1.02k op/s; 35897/328806 objects misplaced (10.917%); 49 MiB/s, 789 objects/s recovering 2024-09-10T19:17:50.938 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:17:50 smithi183 ceph-mon[29736]: osdmap e190: 8 total, 8 up, 6 in 2024-09-10T19:17:50.939 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:17:50 smithi183 ceph-mon[41562]: from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:17:50.939 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:17:50 smithi183 ceph-mon[41562]: osdmap e189: 8 total, 8 up, 6 in 2024-09-10T19:17:50.939 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:17:50 smithi183 ceph-mon[41562]: pgmap v332: 158 pgs: 1 peering, 1 active+recovering+remapped, 3 active+remapped+backfilling, 1 active+recovering, 5 active+remapped+backfill_toofull, 147 active+clean; 6.7 GiB data, 23 GiB used, 514 GiB / 536 GiB avail; 1009 B/s rd, 64 MiB/s wr, 1.02k op/s; 35897/328806 objects misplaced (10.917%); 49 MiB/s, 789 objects/s recovering 2024-09-10T19:17:50.939 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:17:50 smithi183 ceph-mon[41562]: osdmap e190: 8 total, 8 up, 6 in 2024-09-10T19:17:50.966 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:17:50 smithi191 ceph-mon[32884]: from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:17:50.966 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:17:50 smithi191 ceph-mon[32884]: osdmap e189: 8 total, 8 up, 6 in 2024-09-10T19:17:50.966 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:17:50 smithi191 ceph-mon[32884]: pgmap v332: 158 pgs: 1 peering, 1 active+recovering+remapped, 3 active+remapped+backfilling, 1 active+recovering, 5 active+remapped+backfill_toofull, 147 active+clean; 6.7 GiB data, 23 GiB used, 514 GiB / 536 GiB avail; 1009 B/s rd, 64 MiB/s wr, 1.02k op/s; 35897/328806 objects misplaced (10.917%); 49 MiB/s, 789 objects/s recovering 2024-09-10T19:17:50.966 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:17:50 smithi191 ceph-mon[32884]: osdmap e190: 8 total, 8 up, 6 in 2024-09-10T19:17:50.976 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:17:51.002 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:17:51.017 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:17:51.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:17:51.018 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:17:51.119 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:17:51.151 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:17:51.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:17:51.152 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:17:51.253 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:17:51.285 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:17:51.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:17:51.286 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:17:51.314 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-09-10T19:17:51.318 INFO:teuthology.orchestra.run.smithi183.stderr:dumped all 2024-09-10T19:17:51.387 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:17:51.422 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:17:51.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:17:51.423 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:17:51.524 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:17:51.552 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:17:51.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:17:51.554 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:17:51.654 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:17:51.681 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Sep 10 19:17:51 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-y[29989]: ::ffff:172.21.15.191 - - [10/Sep/2024:19:17:51] "GET /metrics HTTP/1.1" 200 224293 "" "Prometheus/2.43.0" 2024-09-10T19:17:51.687 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:17:51.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:17:51.689 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:17:51.789 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:17:51.820 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:17:51.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:17:51.821 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:17:51.830 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: 87 16 116699 116683 83.8142 80.3125 0.0102009 0.011926 2024-09-10T19:17:51.922 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:17:51.953 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:17:51.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:17:51.954 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:17:52.054 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:17:52.092 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:17:52.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:17:52.093 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:17:52.140 INFO:teuthology.orchestra.run.smithi183.stdout:{"pg_ready":true,"pg_map":{"version":333,"stamp":"2024-09-10T19:17:49.761181+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":7198939356,"num_objects":110034,"num_object_clones":0,"num_object_copies":330102,"num_objects_missing_on_primary":1247,"num_objects_missing":1247,"num_objects_degraded":0,"num_objects_misplaced":33737,"num_objects_unfound":0,"num_objects_dirty":110034,"num_whiteouts":0,"num_read":934,"num_read_kb":791,"num_write":110381,"num_write_kb":7031303,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14034,"num_bytes_recovered":908399029,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":116046,"ondisk_log_size":116046,"up":474,"acting":474,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":491,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":562470912,"kb_used":23969192,"kb_used_data":22114356,"kb_used_omap":9,"kb_used_meta":1854774,"kb_avail":538501720,"statfs":{"total":575970213888,"available":551425761280,"internally_reserved":0,"allocated":22645100544,"data_stored":22641975666,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":10224,"internal_metadata":1899288592},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,2,2,1,2,2,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":28311552,"num_objects":432,"num_object_clones":0,"num_object_copies":1296,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-2160,"num_objects_unfound":0,"num_objects_dirty":432,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":432,"num_write_kb":27648,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1121,"num_bytes_recovered":73465856,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.090607"},"pg_stats":[{"pgid":"8.15","version":"186'6366","reported_seq":6439,"reported_epoch":186,"state":"active+remapped+backfill_toofull","last_fresh":"2024-09-10T19:17:44.678304+0000","last_change":"2024-09-10T19:17:30.008682+0000","last_active":"2024-09-10T19:17:44.678304+0000","last_peered":"2024-09-10T19:17:44.678304+0000","last_clean":"2024-09-10T19:17:27.199333+0000","last_became_active":"2024-09-10T19:17:30.001644+0000","last_became_peered":"2024-09-10T19:17:30.001644+0000","last_unstale":"2024-09-10T19:17:44.678304+0000","last_undegraded":"2024-09-10T19:17:44.678304+0000","last_fullsized":"2024-09-10T19:17:44.678304+0000","mapping_epoch":180,"log_start":"160'1800","ondisk_log_start":"160'1800","created":177,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:16:11.777014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:16:11.777014+0000","last_clean_scrub_stamp":"2024-09-10T19:16:11.777014+0000","objects_scrubbed":0,"log_size":4566,"log_dups_size":0,"ondisk_log_size":4566,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":225574912,"num_objects":3442,"num_object_clones":0,"num_object_copies":10326,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3442,"num_objects_unfound":0,"num_objects_dirty":3442,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3442,"num_write_kb":220288,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,4],"avail_no_missing":["7","4","6"],"object_location_counts":[{"shards":"4,6,7","objects":3442}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":150,"reported_epoch":185,"state":"active+clean","last_fresh":"2024-09-10T19:17:40.767072+0000","last_change":"2024-09-10T19:13:08.710758+0000","last_active":"2024-09-10T19:17:40.767072+0000","last_peered":"2024-09-10T19:17:40.767072+0000","last_clean":"2024-09-10T19:17:40.767072+0000","last_became_active":"2024-09-10T19:10:14.056187+0000","last_became_peered":"2024-09-10T19:10:14.056187+0000","last_unstale":"2024-09-10T19:17:40.767072+0000","last_undegraded":"2024-09-10T19:17:40.767072+0000","last_fullsized":"2024-09-10T19:17:40.767072+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:13:08.710679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:13:08.710679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0027252499999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":185,"reported_epoch":185,"state":"active+clean","last_fresh":"2024-09-10T19:17:40.770958+0000","last_change":"2024-09-10T19:16:38.194521+0000","last_active":"2024-09-10T19:17:40.770958+0000","last_peered":"2024-09-10T19:17:40.770958+0000","last_clean":"2024-09-10T19:17:40.770958+0000","last_became_active":"2024-09-10T19:16:24.836961+0000","last_became_peered":"2024-09-10T19:16:24.836961+0000","last_unstale":"2024-09-10T19:17:40.770958+0000","last_undegraded":"2024-09-10T19:17:40.770958+0000","last_fullsized":"2024-09-10T19:17:40.770958+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:16:38.194472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:16:38.194472+0000","last_clean_scrub_stamp":"2024-09-10T19:16:38.194472+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:17:46.775869+0000","scrub_duration":0.0023678900000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.19","version":"66'15","reported_seq":205,"reported_epoch":185,"state":"active+clean","last_fresh":"2024-09-10T19:17:40.765522+0000","last_change":"2024-09-10T19:16:24.879212+0000","last_active":"2024-09-10T19:17:40.765522+0000","last_peered":"2024-09-10T19:17:40.765522+0000","last_clean":"2024-09-10T19:17:40.765522+0000","last_became_active":"2024-09-10T19:16:24.844951+0000","last_became_peered":"2024-09-10T19:16:24.844951+0000","last_unstale":"2024-09-10T19:17:40.765522+0000","last_undegraded":"2024-09-10T19:17:40.765522+0000","last_fullsized":"2024-09-10T19:17:40.765522+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'15","last_scrub_stamp":"2024-09-10T19:13:47.165848+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:13:47.165848+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.004982841,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":187,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-09-10T19:17:46.691066+0000","last_change":"2024-09-10T19:17:22.938220+0000","last_active":"2024-09-10T19:17:46.691066+0000","last_peered":"2024-09-10T19:17:46.691066+0000","last_clean":"2024-09-10T19:17:46.691066+0000","last_became_active":"2024-09-10T19:16:24.845111+0000","last_became_peered":"2024-09-10T19:16:24.845111+0000","last_unstale":"2024-09-10T19:17:46.691066+0000","last_undegraded":"2024-09-10T19:17:46.691066+0000","last_fullsized":"2024-09-10T19:17:46.691066+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:15:33.976696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:15:33.976696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0015310580000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,7],"acting":[4,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.14","version":"188'6395","reported_seq":16249,"reported_epoch":189,"state":"peering","last_fresh":"2024-09-10T19:17:49.695313+0000","last_change":"2024-09-10T19:17:49.696032+0000","last_active":"2024-09-10T19:17:49.695252+0000","last_peered":"2024-09-10T19:17:49.677249+0000","last_clean":"2024-09-10T19:17:49.677249+0000","last_became_active":"2024-09-10T19:17:30.031917+0000","last_became_peered":"2024-09-10T19:17:30.031917+0000","last_unstale":"2024-09-10T19:17:49.695313+0000","last_undegraded":"2024-09-10T19:17:49.695313+0000","last_fullsized":"2024-09-10T19:17:49.695313+0000","mapping_epoch":189,"log_start":"159'1500","ondisk_log_start":"159'1500","created":177,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:16:11.777014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:16:11.777014+0000","last_clean_scrub_stamp":"2024-09-10T19:16:11.777014+0000","objects_scrubbed":0,"log_size":4895,"log_dups_size":0,"ondisk_log_size":4895,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":231505920,"num_objects":3533,"num_object_clones":0,"num_object_copies":10599,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3533,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3533,"num_write_kb":226080,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3256,"num_bytes_recovered":213385216,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,5,6,7],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"59'1","reported_seq":174,"reported_epoch":185,"state":"active+clean","last_fresh":"2024-09-10T19:17:40.766078+0000","last_change":"2024-09-10T19:16:30.798883+0000","last_active":"2024-09-10T19:17:40.766078+0000","last_peered":"2024-09-10T19:17:40.766078+0000","last_clean":"2024-09-10T19:17:40.766078+0000","last_became_active":"2024-09-10T19:10:07.686143+0000","last_became_peered":"2024-09-10T19:10:07.686143+0000","last_unstale":"2024-09-10T19:17:40.766078+0000","last_undegraded":"2024-09-10T19:17:40.766078+0000","last_fullsized":"2024-09-10T19:17:40.766078+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'1","last_scrub_stamp":"2024-09-10T19:16:30.798795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:16:30.798795+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0067002989999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":874,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"66'9","reported_seq":202,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-09-10T19:17:46.691140+0000","last_change":"2024-09-10T19:17:21.915673+0000","last_active":"2024-09-10T19:17:46.691140+0000","last_peered":"2024-09-10T19:17:46.691140+0000","last_clean":"2024-09-10T19:17:46.691140+0000","last_became_active":"2024-09-10T19:13:54.716117+0000","last_became_peered":"2024-09-10T19:13:54.716117+0000","last_unstale":"2024-09-10T19:17:46.691140+0000","last_undegraded":"2024-09-10T19:17:46.691140+0000","last_fullsized":"2024-09-10T19:17:46.691140+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'9","last_scrub_stamp":"2024-09-10T19:14:55.117387+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:14:55.117387+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0074980940000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,3],"acting":[4,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":126,"reported_epoch":185,"state":"active+clean","last_fresh":"2024-09-10T19:17:40.772613+0000","last_change":"2024-09-10T19:16:27.753997+0000","last_active":"2024-09-10T19:17:40.772613+0000","last_peered":"2024-09-10T19:17:40.772613+0000","last_clean":"2024-09-10T19:17:40.772613+0000","last_became_active":"2024-09-10T19:16:24.859261+0000","last_became_peered":"2024-09-10T19:16:24.859261+0000","last_unstale":"2024-09-10T19:17:40.772613+0000","last_undegraded":"2024-09-10T19:17:40.772613+0000","last_fullsized":"2024-09-10T19:17:40.772613+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:16:27.753912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:16:27.753912+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:17:49.796558+0000","scrub_duration":0.0029373799999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":175,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-09-10T19:17:46.691769+0000","last_change":"2024-09-10T19:17:23.968484+0000","last_active":"2024-09-10T19:17:46.691769+0000","last_peered":"2024-09-10T19:17:46.691769+0000","last_clean":"2024-09-10T19:17:46.691769+0000","last_became_active":"2024-09-10T19:13:54.713538+0000","last_became_peered":"2024-09-10T19:13:54.713538+0000","last_unstale":"2024-09-10T19:17:46.691769+0000","last_undegraded":"2024-09-10T19:17:46.691769+0000","last_fullsized":"2024-09-10T19:17:46.691769+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:15:39.910118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:15:39.910118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0013551839999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,7],"acting":[4,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.17","version":"186'6334","reported_seq":6411,"reported_epoch":186,"state":"active+remapped+backfill_toofull","last_fresh":"2024-09-10T19:17:44.927595+0000","last_change":"2024-09-10T19:17:30.041394+0000","last_active":"2024-09-10T19:17:44.927595+0000","last_peered":"2024-09-10T19:17:44.927595+0000","last_clean":"2024-09-10T19:17:27.195401+0000","last_became_active":"2024-09-10T19:17:30.037948+0000","last_became_peered":"2024-09-10T19:17:30.037948+0000","last_unstale":"2024-09-10T19:17:44.927595+0000","last_undegraded":"2024-09-10T19:17:44.927595+0000","last_fullsized":"2024-09-10T19:17:44.927595+0000","mapping_epoch":180,"log_start":"160'1900","ondisk_log_start":"160'1900","created":177,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:16:11.777014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:16:11.777014+0000","last_clean_scrub_stamp":"2024-09-10T19:16:11.777014+0000","objects_scrubbed":0,"log_size":4434,"log_dups_size":0,"ondisk_log_size":4434,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":225148928,"num_objects":3436,"num_object_clones":0,"num_object_copies":10308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":6872,"num_objects_unfound":0,"num_objects_dirty":3436,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3436,"num_write_kb":219872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[3,6,4],"avail_no_missing":["3","4","6"],"object_location_counts":[{"shards":"3,4,6","objects":3436}],"blocked_by":[],"up_primary":7,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":166,"reported_epoch":185,"state":"active+clean","last_fresh":"2024-09-10T19:17:40.771910+0000","last_change":"2024-09-10T19:16:25.595265+0000","last_active":"2024-09-10T19:17:40.771910+0000","last_peered":"2024-09-10T19:17:40.771910+0000","last_clean":"2024-09-10T19:17:40.771910+0000","last_became_active":"2024-09-10T19:10:07.690728+0000","last_became_peered":"2024-09-10T19:10:07.690728+0000","last_unstale":"2024-09-10T19:17:40.771910+0000","last_undegraded":"2024-09-10T19:17:40.771910+0000","last_fullsized":"2024-09-10T19:17:40.771910+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:16:25.595127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:16:25.595127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:17:51.885390+0000","scrub_duration":0.003922633,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1b","version":"66'5","reported_seq":208,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-09-10T19:17:46.691607+0000","last_change":"2024-09-10T19:16:35.966457+0000","last_active":"2024-09-10T19:17:46.691607+0000","last_peered":"2024-09-10T19:17:46.691607+0000","last_clean":"2024-09-10T19:17:46.691607+0000","last_became_active":"2024-09-10T19:16:24.843708+0000","last_became_peered":"2024-09-10T19:16:24.843708+0000","last_unstale":"2024-09-10T19:17:46.691607+0000","last_undegraded":"2024-09-10T19:17:46.691607+0000","last_fullsized":"2024-09-10T19:17:46.691607+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'5","last_scrub_stamp":"2024-09-10T19:16:35.966402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:16:35.966402+0000","objects_scrubbed":2,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0019325740000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":169,"reported_epoch":185,"state":"active+clean","last_fresh":"2024-09-10T19:17:40.770809+0000","last_change":"2024-09-10T19:15:33.518718+0000","last_active":"2024-09-10T19:17:40.770809+0000","last_peered":"2024-09-10T19:17:40.770809+0000","last_clean":"2024-09-10T19:17:40.770809+0000","last_became_active":"2024-09-10T19:13:54.719063+0000","last_became_peered":"2024-09-10T19:13:54.719063+0000","last_unstale":"2024-09-10T19:17:40.770809+0000","last_undegraded":"2024-09-10T19:17:40.770809+0000","last_fullsized":"2024-09-10T19:17:40.770809+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:15:33.518601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:15:33.518601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0013951650000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":177,"reported_epoch":185,"state":"active+clean","last_fresh":"2024-09-10T19:17:40.772695+0000","last_change":"2024-09-10T19:16:24.845721+0000","last_active":"2024-09-10T19:17:40.772695+0000","last_peered":"2024-09-10T19:17:40.772695+0000","last_clean":"2024-09-10T19:17:40.772695+0000","last_became_active":"2024-09-10T19:16:24.845130+0000","last_became_peered":"2024-09-10T19:16:24.845130+0000","last_unstale":"2024-09-10T19:17:40.772695+0000","last_undegraded":"2024-09-10T19:17:40.772695+0000","last_fullsized":"2024-09-10T19:17:40.772695+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:15:45.616184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:15:45.616184+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0013989510000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,0],"acting":[5,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.16","version":"186'6238","reported_seq":6304,"reported_epoch":186,"state":"active+remapped+backfill_toofull","last_fresh":"2024-09-10T19:17:44.876770+0000","last_change":"2024-09-10T19:17:30.032245+0000","last_active":"2024-09-10T19:17:44.876770+0000","last_peered":"2024-09-10T19:17:44.876770+0000","last_clean":"2024-09-10T19:17:27.143398+0000","last_became_active":"2024-09-10T19:17:30.025679+0000","last_became_peered":"2024-09-10T19:17:30.025679+0000","last_unstale":"2024-09-10T19:17:44.876770+0000","last_undegraded":"2024-09-10T19:17:44.876770+0000","last_fullsized":"2024-09-10T19:17:44.876770+0000","mapping_epoch":180,"log_start":"159'1500","ondisk_log_start":"159'1500","created":177,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:16:11.777014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:16:11.777014+0000","last_clean_scrub_stamp":"2024-09-10T19:16:11.777014+0000","objects_scrubbed":0,"log_size":4738,"log_dups_size":0,"ondisk_log_size":4738,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220561408,"num_objects":3366,"num_object_clones":0,"num_object_copies":10098,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":6732,"num_objects_unfound":0,"num_objects_dirty":3366,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3366,"num_write_kb":215392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,0],"acting":[0,6,7],"avail_no_missing":["0","6","7"],"object_location_counts":[{"shards":"0,6,7","objects":3366}],"blocked_by":[],"up_primary":3,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":166,"reported_epoch":185,"state":"active+clean","last_fresh":"2024-09-10T19:17:40.777606+0000","last_change":"2024-09-10T19:16:24.859112+0000","last_active":"2024-09-10T19:17:40.777606+0000","last_peered":"2024-09-10T19:17:40.777606+0000","last_clean":"2024-09-10T19:17:40.777606+0000","last_became_active":"2024-09-10T19:16:24.858800+0000","last_became_peered":"2024-09-10T19:16:24.858800+0000","last_unstale":"2024-09-10T19:17:40.777606+0000","last_undegraded":"2024-09-10T19:17:40.777606+0000","last_fullsized":"2024-09-10T19:17:40.777606+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:12:54.477149+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:12:54.477149+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0039768319999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1a","version":"66'9","reported_seq":175,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-09-10T19:17:46.691584+0000","last_change":"2024-09-10T19:17:14.981950+0000","last_active":"2024-09-10T19:17:46.691584+0000","last_peered":"2024-09-10T19:17:46.691584+0000","last_clean":"2024-09-10T19:17:46.691584+0000","last_became_active":"2024-09-10T19:10:09.688987+0000","last_became_peered":"2024-09-10T19:10:09.688987+0000","last_unstale":"2024-09-10T19:17:46.691584+0000","last_undegraded":"2024-09-10T19:17:46.691584+0000","last_fullsized":"2024-09-10T19:17:46.691584+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'9","last_scrub_stamp":"2024-09-10T19:13:02.431443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:13:02.431443+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0084913290000000006,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":162,"reported_epoch":185,"state":"active+clean","last_fresh":"2024-09-10T19:17:40.771572+0000","last_change":"2024-09-10T19:16:37.116464+0000","last_active":"2024-09-10T19:17:40.771572+0000","last_peered":"2024-09-10T19:17:40.771572+0000","last_clean":"2024-09-10T19:17:40.771572+0000","last_became_active":"2024-09-10T19:10:11.701799+0000","last_became_peered":"2024-09-10T19:10:11.701799+0000","last_unstale":"2024-09-10T19:17:40.771572+0000","last_undegraded":"2024-09-10T19:17:40.771572+0000","last_fullsized":"2024-09-10T19:17:40.771572+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:16:37.116371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:16:37.116371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:18:03.024881+0000","scrub_duration":0.028100301000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":179,"reported_epoch":185,"state":"active+clean","last_fresh":"2024-09-10T19:17:40.768913+0000","last_change":"2024-09-10T19:17:18.281940+0000","last_active":"2024-09-10T19:17:40.768913+0000","last_peered":"2024-09-10T19:17:40.768913+0000","last_clean":"2024-09-10T19:17:40.768913+0000","last_became_active":"2024-09-10T19:16:24.836477+0000","last_became_peered":"2024-09-10T19:16:24.836477+0000","last_unstale":"2024-09-10T19:17:40.768913+0000","last_undegraded":"2024-09-10T19:17:40.768913+0000","last_fullsized":"2024-09-10T19:17:40.768913+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:15:33.839248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:15:33.839248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0032019980000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1","version":"186'6373","reported_seq":6438,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-09-10T19:17:44.869001+0000","last_change":"2024-09-10T19:17:26.961275+0000","last_active":"2024-09-10T19:17:44.869001+0000","last_peered":"2024-09-10T19:17:44.869001+0000","last_clean":"2024-09-10T19:17:44.869001+0000","last_became_active":"2024-09-10T19:17:26.958866+0000","last_became_peered":"2024-09-10T19:17:26.958866+0000","last_unstale":"2024-09-10T19:17:44.869001+0000","last_undegraded":"2024-09-10T19:17:44.869001+0000","last_fullsized":"2024-09-10T19:17:44.869001+0000","mapping_epoch":154,"log_start":"159'1600","ondisk_log_start":"159'1600","created":148,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:16:11.777014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:16:11.777014+0000","last_clean_scrub_stamp":"2024-09-10T19:16:11.777014+0000","objects_scrubbed":0,"log_size":4773,"log_dups_size":0,"ondisk_log_size":4773,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":226394112,"num_objects":3454,"num_object_clones":0,"num_object_copies":10362,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3454,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3454,"num_write_kb":221088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.d","version":"66'17","reported_seq":257,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-09-10T19:17:46.693367+0000","last_change":"2024-09-10T19:16:32.999033+0000","last_active":"2024-09-10T19:17:46.693367+0000","last_peered":"2024-09-10T19:17:46.693367+0000","last_clean":"2024-09-10T19:17:46.693367+0000","last_became_active":"2024-09-10T19:16:24.847056+0000","last_became_peered":"2024-09-10T19:16:24.847056+0000","last_unstale":"2024-09-10T19:17:46.693367+0000","last_undegraded":"2024-09-10T19:17:46.693367+0000","last_fullsized":"2024-09-10T19:17:46.693367+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'17","last_scrub_stamp":"2024-09-10T19:16:32.998950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:16:32.998950+0000","objects_scrubbed":8,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0095505250000000007,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,0],"acting":[4,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":166,"reported_epoch":185,"state":"active+clean","last_fresh":"2024-09-10T19:17:40.811472+0000","last_change":"2024-09-10T19:16:39.716870+0000","last_active":"2024-09-10T19:17:40.811472+0000","last_peered":"2024-09-10T19:17:40.811472+0000","last_clean":"2024-09-10T19:17:40.811472+0000","last_became_active":"2024-09-10T19:10:07.691119+0000","last_became_peered":"2024-09-10T19:10:07.691119+0000","last_unstale":"2024-09-10T19:17:40.811472+0000","last_undegraded":"2024-09-10T19:17:40.811472+0000","last_fullsized":"2024-09-10T19:17:40.811472+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:16:39.716674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:16:39.716674+0000","last_clean_scrub_stamp":"2024-09-10T19:16:39.716674+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:17:54.073755+0000","scrub_duration":0.043895318000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":122,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-09-10T19:17:46.692120+0000","last_change":"2024-09-10T19:16:26.999105+0000","last_active":"2024-09-10T19:17:46.692120+0000","last_peered":"2024-09-10T19:17:46.692120+0000","last_clean":"2024-09-10T19:17:46.692120+0000","last_became_active":"2024-09-10T19:13:54.712883+0000","last_became_peered":"2024-09-10T19:13:54.712883+0000","last_unstale":"2024-09-10T19:17:46.692120+0000","last_undegraded":"2024-09-10T19:17:46.692120+0000","last_fullsized":"2024-09-10T19:17:46.692120+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:16:26.999036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:16:26.999036+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0050193119999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":44,"reported_epoch":185,"state":"active+clean","last_fresh":"2024-09-10T19:17:40.766300+0000","last_change":"2024-09-10T19:16:24.858656+0000","last_active":"2024-09-10T19:17:40.766300+0000","last_peered":"2024-09-10T19:17:40.766300+0000","last_clean":"2024-09-10T19:17:40.766300+0000","last_became_active":"2024-09-10T19:16:24.857843+0000","last_became_peered":"2024-09-10T19:16:24.857843+0000","last_unstale":"2024-09-10T19:17:40.766300+0000","last_undegraded":"2024-09-10T19:17:40.766300+0000","last_fullsized":"2024-09-10T19:17:40.766300+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:15:47.818381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:15:47.818381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.0","version":"186'6349","reported_seq":6407,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-09-10T19:17:44.944084+0000","last_change":"2024-09-10T19:17:26.971893+0000","last_active":"2024-09-10T19:17:44.944084+0000","last_peered":"2024-09-10T19:17:44.944084+0000","last_clean":"2024-09-10T19:17:44.944084+0000","last_became_active":"2024-09-10T19:17:26.958072+0000","last_became_peered":"2024-09-10T19:17:26.958072+0000","last_unstale":"2024-09-10T19:17:44.944084+0000","last_undegraded":"2024-09-10T19:17:44.944084+0000","last_fullsized":"2024-09-10T19:17:44.944084+0000","mapping_epoch":148,"log_start":"160'1900","ondisk_log_start":"160'1900","created":148,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:16:11.777014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:16:11.777014+0000","last_clean_scrub_stamp":"2024-09-10T19:16:11.777014+0000","objects_scrubbed":0,"log_size":4449,"log_dups_size":0,"ondisk_log_size":4449,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":224985088,"num_objects":3433,"num_object_clones":0,"num_object_copies":10299,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3433,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3433,"num_write_kb":219712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,4],"acting":[3,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"66'10","reported_seq":184,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-09-10T19:17:46.692331+0000","last_change":"2024-09-10T19:17:20.959169+0000","last_active":"2024-09-10T19:17:46.692331+0000","last_peered":"2024-09-10T19:17:46.692331+0000","last_clean":"2024-09-10T19:17:46.692331+0000","last_became_active":"2024-09-10T19:10:09.688321+0000","last_became_peered":"2024-09-10T19:10:09.688321+0000","last_unstale":"2024-09-10T19:17:46.692331+0000","last_undegraded":"2024-09-10T19:17:46.692331+0000","last_fullsized":"2024-09-10T19:17:46.692331+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'10","last_scrub_stamp":"2024-09-10T19:13:51.415998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:13:51.415998+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.005972799,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":116,"reported_epoch":185,"state":"active+clean","last_fresh":"2024-09-10T19:17:40.765436+0000","last_change":"2024-09-10T19:15:42.119912+0000","last_active":"2024-09-10T19:17:40.765436+0000","last_peered":"2024-09-10T19:17:40.765436+0000","last_clean":"2024-09-10T19:17:40.765436+0000","last_became_active":"2024-09-10T19:13:54.731134+0000","last_became_peered":"2024-09-10T19:13:54.731134+0000","last_unstale":"2024-09-10T19:17:40.765436+0000","last_undegraded":"2024-09-10T19:17:40.765436+0000","last_fullsized":"2024-09-10T19:17:40.765436+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:15:42.119590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:15:42.119590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.018001347000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,5],"acting":[3,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"155'10","reported_seq":130,"reported_epoch":185,"state":"active+clean","last_fresh":"2024-09-10T19:17:40.833582+0000","last_change":"2024-09-10T19:16:29.626806+0000","last_active":"2024-09-10T19:17:40.833582+0000","last_peered":"2024-09-10T19:17:40.833582+0000","last_clean":"2024-09-10T19:17:40.833582+0000","last_became_active":"2024-09-10T19:16:24.851236+0000","last_became_peered":"2024-09-10T19:16:24.851236+0000","last_unstale":"2024-09-10T19:17:40.833582+0000","last_undegraded":"2024-09-10T19:17:40.833582+0000","last_fullsized":"2024-09-10T19:17:40.833582+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"155'10","last_scrub_stamp":"2024-09-10T19:16:29.626572+0000","last_deep_scrub":"80'9","last_deep_scrub_stamp":"2024-09-10T19:11:25.156465+0000","last_clean_scrub_stamp":"2024-09-10T19:16:29.626572+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:17:47.526795+0000","scrub_duration":0.0067064810000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,0],"acting":[7,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":185,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-09-10T19:17:46.692228+0000","last_change":"2024-09-10T19:16:24.852100+0000","last_active":"2024-09-10T19:17:46.692228+0000","last_peered":"2024-09-10T19:17:46.692228+0000","last_clean":"2024-09-10T19:17:46.692228+0000","last_became_active":"2024-09-10T19:16:24.851829+0000","last_became_peered":"2024-09-10T19:16:24.851829+0000","last_unstale":"2024-09-10T19:17:46.692228+0000","last_undegraded":"2024-09-10T19:17:46.692228+0000","last_fullsized":"2024-09-10T19:17:46.692228+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:15:29.988976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:15:29.988976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0053787150000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,7],"acting":[4,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.3","version":"186'6548","reported_seq":9070,"reported_epoch":186,"state":"active+recovering","last_fresh":"2024-09-10T19:17:44.783099+0000","last_change":"2024-09-10T19:17:28.441322+0000","last_active":"2024-09-10T19:17:44.783099+0000","last_peered":"2024-09-10T19:17:44.783099+0000","last_clean":"2024-09-10T19:16:54.964816+0000","last_became_active":"2024-09-10T19:17:27.974691+0000","last_became_peered":"2024-09-10T19:17:27.974691+0000","last_unstale":"2024-09-10T19:17:44.783099+0000","last_undegraded":"2024-09-10T19:17:44.783099+0000","last_fullsized":"2024-09-10T19:17:44.783099+0000","mapping_epoch":178,"log_start":"160'2100","ondisk_log_start":"160'2100","created":148,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:16:11.777014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:16:11.777014+0000","last_clean_scrub_stamp":"2024-09-10T19:16:11.777014+0000","objects_scrubbed":0,"log_size":4448,"log_dups_size":0,"ondisk_log_size":4448,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":231833600,"num_objects":3537,"num_object_clones":0,"num_object_copies":10611,"num_objects_missing_on_primary":1247,"num_objects_missing":1247,"num_objects_degraded":0,"num_objects_misplaced":1247,"num_objects_unfound":0,"num_objects_dirty":3537,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3537,"num_write_kb":226400,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":680,"num_bytes_recovered":44597248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":["3","5","6"],"object_location_counts":[{"shards":"3,5,6","objects":1247},{"shards":"3,5,6,7","objects":1},{"shards":"3,5,7","objects":2289}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.f","version":"66'15","reported_seq":169,"reported_epoch":185,"state":"active+clean","last_fresh":"2024-09-10T19:17:40.766803+0000","last_change":"2024-09-10T19:16:25.769523+0000","last_active":"2024-09-10T19:17:40.766803+0000","last_peered":"2024-09-10T19:17:40.766803+0000","last_clean":"2024-09-10T19:17:40.766803+0000","last_became_active":"2024-09-10T19:13:54.711634+0000","last_became_peered":"2024-09-10T19:13:54.711634+0000","last_unstale":"2024-09-10T19:17:40.766803+0000","last_undegraded":"2024-09-10T19:17:40.766803+0000","last_fullsized":"2024-09-10T19:17:40.766803+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'15","last_scrub_stamp":"2024-09-10T19:16:25.769436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:16:25.769436+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:17:46.035940+0000","scrub_duration":0.0046254920000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":126,"reported_epoch":185,"state":"active+clean","last_fresh":"2024-09-10T19:17:40.774766+0000","last_change":"2024-09-10T19:16:24.857467+0000","last_active":"2024-09-10T19:17:40.774766+0000","last_peered":"2024-09-10T19:17:40.774766+0000","last_clean":"2024-09-10T19:17:40.774766+0000","last_became_active":"2024-09-10T19:16:24.857179+0000","last_became_peered":"2024-09-10T19:16:24.857179+0000","last_unstale":"2024-09-10T19:17:40.774766+0000","last_undegraded":"2024-09-10T19:17:40.774766+0000","last_fullsized":"2024-09-10T19:17:40.774766+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:15:29.553448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:15:29.553448+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0062080679999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,6],"acting":[7,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.e","version":"66'8","reported_seq":257,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-09-10T19:17:46.692380+0000","last_change":"2024-09-10T19:17:18.952116+0000","last_active":"2024-09-10T19:17:46.692380+0000","last_peered":"2024-09-10T19:17:46.692380+0000","last_clean":"2024-09-10T19:17:46.692380+0000","last_became_active":"2024-09-10T19:10:11.699232+0000","last_became_peered":"2024-09-10T19:10:11.699232+0000","last_unstale":"2024-09-10T19:17:46.692380+0000","last_undegraded":"2024-09-10T19:17:46.692380+0000","last_fullsized":"2024-09-10T19:17:46.692380+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'8","last_scrub_stamp":"2024-09-10T19:13:40.475375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:13:40.475375+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.035806152000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":177,"reported_epoch":185,"state":"active+clean","last_fresh":"2024-09-10T19:17:40.778973+0000","last_change":"2024-09-10T19:16:24.855938+0000","last_active":"2024-09-10T19:17:40.778973+0000","last_peered":"2024-09-10T19:17:40.778973+0000","last_clean":"2024-09-10T19:17:40.778973+0000","last_became_active":"2024-09-10T19:16:24.855627+0000","last_became_peered":"2024-09-10T19:16:24.855627+0000","last_unstale":"2024-09-10T19:17:40.778973+0000","last_undegraded":"2024-09-10T19:17:40.778973+0000","last_fullsized":"2024-09-10T19:17:40.778973+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:15:44.633043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:15:44.633043+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.00368805,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,6],"acting":[5,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.2","version":"189'6429","reported_seq":6489,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-09-10T19:17:49.727352+0000","last_change":"2024-09-10T19:17:26.956443+0000","last_active":"2024-09-10T19:17:49.727352+0000","last_peered":"2024-09-10T19:17:49.727352+0000","last_clean":"2024-09-10T19:17:49.727352+0000","last_became_active":"2024-09-10T19:17:26.953881+0000","last_became_peered":"2024-09-10T19:17:26.953881+0000","last_unstale":"2024-09-10T19:17:49.727352+0000","last_undegraded":"2024-09-10T19:17:49.727352+0000","last_fullsized":"2024-09-10T19:17:49.727352+0000","mapping_epoch":148,"log_start":"159'1500","ondisk_log_start":"159'1500","created":148,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:16:11.777014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:16:11.777014+0000","last_clean_scrub_stamp":"2024-09-10T19:16:11.777014+0000","objects_scrubbed":0,"log_size":4929,"log_dups_size":0,"ondisk_log_size":4929,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":230162432,"num_objects":3512,"num_object_clones":0,"num_object_copies":10536,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3512,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3512,"num_write_kb":224768,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,5],"acting":[4,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"84'16","reported_seq":212,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-09-10T19:17:46.692460+0000","last_change":"2024-09-10T19:15:47.914487+0000","last_active":"2024-09-10T19:17:46.692460+0000","last_peered":"2024-09-10T19:17:46.692460+0000","last_clean":"2024-09-10T19:17:46.692460+0000","last_became_active":"2024-09-10T19:13:54.717022+0000","last_became_peered":"2024-09-10T19:13:54.717022+0000","last_unstale":"2024-09-10T19:17:46.692460+0000","last_undegraded":"2024-09-10T19:17:46.692460+0000","last_fullsized":"2024-09-10T19:17:46.692460+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'16","last_scrub_stamp":"2024-09-10T19:15:47.914420+0000","last_deep_scrub":"84'16","last_deep_scrub_stamp":"2024-09-10T19:11:56.157975+0000","last_clean_scrub_stamp":"2024-09-10T19:15:47.914420+0000","objects_scrubbed":5,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0019153390000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,3],"acting":[4,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":173,"reported_epoch":185,"state":"active+clean","last_fresh":"2024-09-10T19:17:40.771580+0000","last_change":"2024-09-10T19:15:42.450149+0000","last_active":"2024-09-10T19:17:40.771580+0000","last_peered":"2024-09-10T19:17:40.771580+0000","last_clean":"2024-09-10T19:17:40.771580+0000","last_became_active":"2024-09-10T19:13:54.720293+0000","last_became_peered":"2024-09-10T19:13:54.720293+0000","last_unstale":"2024-09-10T19:17:40.771580+0000","last_undegraded":"2024-09-10T19:17:40.771580+0000","last_fullsized":"2024-09-10T19:17:40.771580+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:15:42.450068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:11:53.066849+0000","last_clean_scrub_stamp":"2024-09-10T19:15:42.450068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0037708360000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":152,"reported_epoch":185,"state":"active+clean","last_fresh":"2024-09-10T19:17:40.778680+0000","last_change":"2024-09-10T19:13:06.358188+0000","last_active":"2024-09-10T19:17:40.778680+0000","last_peered":"2024-09-10T19:17:40.778680+0000","last_clean":"2024-09-10T19:17:40.778680+0000","last_became_active":"2024-09-10T19:10:11.692215+0000","last_became_peered":"2024-09-10T19:10:11.692215+0000","last_unstale":"2024-09-10T19:17:40.778680+0000","last_undegraded":"2024-09-10T19:17:40.778680+0000","last_fullsized":"2024-09-10T19:17:40.778680+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:13:06.358109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:13:06.358109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0026724549999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":150,"reported_epoch":185,"state":"active+clean","last_fresh":"2024-09-10T19:17:40.765957+0000","last_change":"2024-09-10T19:13:54.304493+0000","last_active":"2024-09-10T19:17:40.765957+0000","last_peered":"2024-09-10T19:17:40.765957+0000","last_clean":"2024-09-10T19:17:40.765957+0000","last_became_active":"2024-09-10T19:10:14.053736+0000","last_became_peered":"2024-09-10T19:10:14.053736+0000","last_unstale":"2024-09-10T19:17:40.765957+0000","last_undegraded":"2024-09-10T19:17:40.765957+0000","last_fullsized":"2024-09-10T19:17:40.765957+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:13:54.304435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:13:54.304435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0011043820000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.d","version":"186'6874","reported_seq":6930,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-09-10T19:17:44.815278+0000","last_change":"2024-09-10T19:16:24.858800+0000","last_active":"2024-09-10T19:17:44.815278+0000","last_peered":"2024-09-10T19:17:44.815278+0000","last_clean":"2024-09-10T19:17:44.815278+0000","last_became_active":"2024-09-10T19:16:24.858460+0000","last_became_peered":"2024-09-10T19:16:24.858460+0000","last_unstale":"2024-09-10T19:17:44.815278+0000","last_undegraded":"2024-09-10T19:17:44.815278+0000","last_fullsized":"2024-09-10T19:17:44.815278+0000","mapping_epoch":154,"log_start":"163'3090","ondisk_log_start":"163'3090","created":148,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:16:11.777014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:16:11.777014+0000","last_clean_scrub_stamp":"2024-09-10T19:16:11.777014+0000","objects_scrubbed":0,"log_size":3784,"log_dups_size":0,"ondisk_log_size":3784,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":450494464,"num_objects":6874,"num_object_clones":0,"num_object_copies":20622,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6874,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6874,"num_write_kb":439936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1","version":"66'14","reported_seq":63,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-09-10T19:17:46.690545+0000","last_change":"2024-09-10T19:17:28.925690+0000","last_active":"2024-09-10T19:17:46.690545+0000","last_peered":"2024-09-10T19:17:46.690545+0000","last_clean":"2024-09-10T19:17:46.690545+0000","last_became_active":"2024-09-10T19:16:24.852821+0000","last_became_peered":"2024-09-10T19:16:24.852821+0000","last_unstale":"2024-09-10T19:17:46.690545+0000","last_undegraded":"2024-09-10T19:17:46.690545+0000","last_fullsized":"2024-09-10T19:17:46.690545+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'14","last_scrub_stamp":"2024-09-10T19:15:41.748101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:15:41.748101+0000","objects_scrubbed":7,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0055403099999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,7],"acting":[4,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":176,"reported_epoch":185,"state":"active+clean","last_fresh":"2024-09-10T19:17:40.768291+0000","last_change":"2024-09-10T19:16:25.582072+0000","last_active":"2024-09-10T19:17:40.768291+0000","last_peered":"2024-09-10T19:17:40.768291+0000","last_clean":"2024-09-10T19:17:40.768291+0000","last_became_active":"2024-09-10T19:16:24.855861+0000","last_became_peered":"2024-09-10T19:16:24.855861+0000","last_unstale":"2024-09-10T19:17:40.768291+0000","last_undegraded":"2024-09-10T19:17:40.768291+0000","last_fullsized":"2024-09-10T19:17:40.768291+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:16:25.581993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:16:25.581993+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0041345990000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.0","version":"117'10","reported_seq":273,"reported_epoch":185,"state":"active+clean","last_fresh":"2024-09-10T19:17:40.833355+0000","last_change":"2024-09-10T19:15:35.973188+0000","last_active":"2024-09-10T19:17:40.833355+0000","last_peered":"2024-09-10T19:17:40.833355+0000","last_clean":"2024-09-10T19:17:40.833355+0000","last_became_active":"2024-09-10T19:13:54.709846+0000","last_became_peered":"2024-09-10T19:13:54.709846+0000","last_unstale":"2024-09-10T19:17:40.833355+0000","last_undegraded":"2024-09-10T19:17:40.833355+0000","last_fullsized":"2024-09-10T19:17:40.833355+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"117'10","last_scrub_stamp":"2024-09-10T19:15:35.973095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:15:35.973095+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0076338860000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,0],"acting":[3,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":160,"reported_epoch":185,"state":"active+clean","last_fresh":"2024-09-10T19:17:40.811328+0000","last_change":"2024-09-10T19:16:24.858926+0000","last_active":"2024-09-10T19:17:40.811328+0000","last_peered":"2024-09-10T19:17:40.811328+0000","last_clean":"2024-09-10T19:17:40.811328+0000","last_became_active":"2024-09-10T19:16:24.858619+0000","last_became_peered":"2024-09-10T19:16:24.858619+0000","last_unstale":"2024-09-10T19:17:40.811328+0000","last_undegraded":"2024-09-10T19:17:40.811328+0000","last_fullsized":"2024-09-10T19:17:40.811328+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:13:15.350720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:11:41.073833+0000","last_clean_scrub_stamp":"2024-09-10T19:13:15.350720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0014361000000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.c","version":"186'6887","reported_seq":6935,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-09-10T19:17:44.908245+0000","last_change":"2024-09-10T19:17:29.176247+0000","last_active":"2024-09-10T19:17:44.908245+0000","last_peered":"2024-09-10T19:17:44.908245+0000","last_clean":"2024-09-10T19:17:44.908245+0000","last_became_active":"2024-09-10T19:16:13.079527+0000","last_became_peered":"2024-09-10T19:16:13.079527+0000","last_unstale":"2024-09-10T19:17:44.908245+0000","last_undegraded":"2024-09-10T19:17:44.908245+0000","last_fullsized":"2024-09-10T19:17:44.908245+0000","mapping_epoch":148,"log_start":"163'3073","ondisk_log_start":"163'3073","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:16:11.777014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:16:11.777014+0000","last_clean_scrub_stamp":"2024-09-10T19:16:11.777014+0000","objects_scrubbed":0,"log_size":3814,"log_dups_size":0,"ondisk_log_size":3814,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":451346432,"num_objects":6887,"num_object_clones":0,"num_object_copies":20661,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6887,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6887,"num_write_kb":440768,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.0","version":"66'18","reported_seq":186,"reported_epoch":185,"state":"active+clean","last_fresh":"2024-09-10T19:17:40.766580+0000","last_change":"2024-09-10T19:12:49.699111+0000","last_active":"2024-09-10T19:17:40.766580+0000","last_peered":"2024-09-10T19:17:40.766580+0000","last_clean":"2024-09-10T19:17:40.766580+0000","last_became_active":"2024-09-10T19:10:09.821836+0000","last_became_peered":"2024-09-10T19:10:09.821836+0000","last_unstale":"2024-09-10T19:17:40.766580+0000","last_undegraded":"2024-09-10T19:17:40.766580+0000","last_fullsized":"2024-09-10T19:17:40.766580+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'18","last_scrub_stamp":"2024-09-10T19:12:49.699054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:12:49.699054+0000","objects_scrubbed":8,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0073070890000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":126,"reported_epoch":185,"state":"active+clean","last_fresh":"2024-09-10T19:17:40.769263+0000","last_change":"2024-09-10T19:17:26.374435+0000","last_active":"2024-09-10T19:17:40.769263+0000","last_peered":"2024-09-10T19:17:40.769263+0000","last_clean":"2024-09-10T19:17:40.769263+0000","last_became_active":"2024-09-10T19:13:54.710968+0000","last_became_peered":"2024-09-10T19:13:54.710968+0000","last_unstale":"2024-09-10T19:17:40.769263+0000","last_undegraded":"2024-09-10T19:17:40.769263+0000","last_fullsized":"2024-09-10T19:17:40.769263+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:17:26.374376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:17:26.374376+0000","last_clean_scrub_stamp":"2024-09-10T19:17:26.374376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:18:32.655180+0000","scrub_duration":0.001299815,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,3],"acting":[6,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":164,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-09-10T19:17:46.690867+0000","last_change":"2024-09-10T19:16:30.991715+0000","last_active":"2024-09-10T19:17:46.690867+0000","last_peered":"2024-09-10T19:17:46.690867+0000","last_clean":"2024-09-10T19:17:46.690867+0000","last_became_active":"2024-09-10T19:10:11.698054+0000","last_became_peered":"2024-09-10T19:10:11.698054+0000","last_unstale":"2024-09-10T19:17:46.690867+0000","last_undegraded":"2024-09-10T19:17:46.690867+0000","last_fullsized":"2024-09-10T19:17:46.690867+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:16:30.991635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:16:30.991635+0000","last_clean_scrub_stamp":"2024-09-10T19:16:30.991635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:17:56.300591+0000","scrub_duration":0.0021810029999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":172,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-09-10T19:17:46.691514+0000","last_change":"2024-09-10T19:16:30.036748+0000","last_active":"2024-09-10T19:17:46.691514+0000","last_peered":"2024-09-10T19:17:46.691514+0000","last_clean":"2024-09-10T19:17:46.691514+0000","last_became_active":"2024-09-10T19:16:24.848109+0000","last_became_peered":"2024-09-10T19:16:24.848109+0000","last_unstale":"2024-09-10T19:17:46.691514+0000","last_undegraded":"2024-09-10T19:17:46.691514+0000","last_fullsized":"2024-09-10T19:17:46.691514+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:16:30.036662+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:16:30.036662+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0044984129999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.f","version":"186'6717","reported_seq":6773,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-09-10T19:17:44.975844+0000","last_change":"2024-09-10T19:16:24.856828+0000","last_active":"2024-09-10T19:17:44.975844+0000","last_peered":"2024-09-10T19:17:44.975844+0000","last_clean":"2024-09-10T19:17:44.975844+0000","last_became_active":"2024-09-10T19:16:24.855393+0000","last_became_peered":"2024-09-10T19:16:24.855393+0000","last_unstale":"2024-09-10T19:17:44.975844+0000","last_undegraded":"2024-09-10T19:17:44.975844+0000","last_fullsized":"2024-09-10T19:17:44.975844+0000","mapping_epoch":154,"log_start":"163'3099","ondisk_log_start":"163'3099","created":148,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:16:11.777014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:16:11.777014+0000","last_clean_scrub_stamp":"2024-09-10T19:16:11.777014+0000","objects_scrubbed":0,"log_size":3618,"log_dups_size":0,"ondisk_log_size":3618,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":440205312,"num_objects":6717,"num_object_clones":0,"num_object_copies":20151,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6717,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6717,"num_write_kb":429888,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.3","version":"104'27","reported_seq":207,"reported_epoch":185,"state":"active+clean","last_fresh":"2024-09-10T19:17:40.769752+0000","last_change":"2024-09-10T19:16:35.246566+0000","last_active":"2024-09-10T19:17:40.769752+0000","last_peered":"2024-09-10T19:17:40.769752+0000","last_clean":"2024-09-10T19:17:40.769752+0000","last_became_active":"2024-09-10T19:10:09.823003+0000","last_became_peered":"2024-09-10T19:10:09.823003+0000","last_unstale":"2024-09-10T19:17:40.769752+0000","last_undegraded":"2024-09-10T19:17:40.769752+0000","last_fullsized":"2024-09-10T19:17:40.769752+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"104'27","last_scrub_stamp":"2024-09-10T19:16:35.246512+0000","last_deep_scrub":"104'27","last_deep_scrub_stamp":"2024-09-10T19:13:02.225944+0000","last_clean_scrub_stamp":"2024-09-10T19:16:35.246512+0000","objects_scrubbed":8,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:17:49.596002+0000","scrub_duration":0.0052296490000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":166,"reported_epoch":185,"state":"active+clean","last_fresh":"2024-09-10T19:17:40.771973+0000","last_change":"2024-09-10T19:16:37.780086+0000","last_active":"2024-09-10T19:17:40.771973+0000","last_peered":"2024-09-10T19:17:40.771973+0000","last_clean":"2024-09-10T19:17:40.771973+0000","last_became_active":"2024-09-10T19:10:07.690325+0000","last_became_peered":"2024-09-10T19:10:07.690325+0000","last_unstale":"2024-09-10T19:17:40.771973+0000","last_undegraded":"2024-09-10T19:17:40.771973+0000","last_fullsized":"2024-09-10T19:17:40.771973+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:16:37.779967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:16:37.779967+0000","last_clean_scrub_stamp":"2024-09-10T19:16:37.779967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:17:50.595527+0000","scrub_duration":0.067110102000000005,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":165,"reported_epoch":185,"state":"active+clean","last_fresh":"2024-09-10T19:17:40.766405+0000","last_change":"2024-09-10T19:17:19.463698+0000","last_active":"2024-09-10T19:17:40.766405+0000","last_peered":"2024-09-10T19:17:40.766405+0000","last_clean":"2024-09-10T19:17:40.766405+0000","last_became_active":"2024-09-10T19:10:11.700370+0000","last_became_peered":"2024-09-10T19:10:11.700370+0000","last_unstale":"2024-09-10T19:17:40.766405+0000","last_undegraded":"2024-09-10T19:17:40.766405+0000","last_fullsized":"2024-09-10T19:17:40.766405+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:15:50.669821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:15:50.669821+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0015765,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":129,"reported_epoch":185,"state":"active+clean","last_fresh":"2024-09-10T19:17:40.766765+0000","last_change":"2024-09-10T19:17:22.435156+0000","last_active":"2024-09-10T19:17:40.766765+0000","last_peered":"2024-09-10T19:17:40.766765+0000","last_clean":"2024-09-10T19:17:40.766765+0000","last_became_active":"2024-09-10T19:16:24.844699+0000","last_became_peered":"2024-09-10T19:16:24.844699+0000","last_unstale":"2024-09-10T19:17:40.766765+0000","last_undegraded":"2024-09-10T19:17:40.766765+0000","last_fullsized":"2024-09-10T19:17:40.766765+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:15:41.749338+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:15:41.749338+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0018841260000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.e","version":"186'6761","reported_seq":6816,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-09-10T19:17:45.009669+0000","last_change":"2024-09-10T19:16:24.837044+0000","last_active":"2024-09-10T19:17:45.009669+0000","last_peered":"2024-09-10T19:17:45.009669+0000","last_clean":"2024-09-10T19:17:45.009669+0000","last_became_active":"2024-09-10T19:16:24.836458+0000","last_became_peered":"2024-09-10T19:16:24.836458+0000","last_unstale":"2024-09-10T19:17:45.009669+0000","last_undegraded":"2024-09-10T19:17:45.009669+0000","last_fullsized":"2024-09-10T19:17:45.009669+0000","mapping_epoch":154,"log_start":"162'2884","ondisk_log_start":"162'2884","created":148,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:16:11.777014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:16:11.777014+0000","last_clean_scrub_stamp":"2024-09-10T19:16:11.777014+0000","objects_scrubbed":0,"log_size":3877,"log_dups_size":0,"ondisk_log_size":3877,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":443088896,"num_objects":6761,"num_object_clones":0,"num_object_copies":20283,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6761,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6761,"num_write_kb":432704,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"66'10","reported_seq":122,"reported_epoch":185,"state":"active+clean","last_fresh":"2024-09-10T19:17:40.772449+0000","last_change":"2024-09-10T19:13:54.748881+0000","last_active":"2024-09-10T19:17:40.772449+0000","last_peered":"2024-09-10T19:17:40.772449+0000","last_clean":"2024-09-10T19:17:40.772449+0000","last_became_active":"2024-09-10T19:13:54.714542+0000","last_became_peered":"2024-09-10T19:13:54.714542+0000","last_unstale":"2024-09-10T19:17:40.772449+0000","last_undegraded":"2024-09-10T19:17:40.772449+0000","last_fullsized":"2024-09-10T19:17:40.772449+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'10","last_scrub_stamp":"2024-09-10T19:12:56.360409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:12:56.360409+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":14,"num_read_kb":9,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,0],"acting":[5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":135,"reported_epoch":185,"state":"active+clean","last_fresh":"2024-09-10T19:17:40.770749+0000","last_change":"2024-09-10T19:17:28.203449+0000","last_active":"2024-09-10T19:17:40.770749+0000","last_peered":"2024-09-10T19:17:40.770749+0000","last_clean":"2024-09-10T19:17:40.770749+0000","last_became_active":"2024-09-10T19:16:24.836898+0000","last_became_peered":"2024-09-10T19:16:24.836898+0000","last_unstale":"2024-09-10T19:17:40.770749+0000","last_undegraded":"2024-09-10T19:17:40.770749+0000","last_fullsized":"2024-09-10T19:17:40.770749+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:17:28.203397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:17:28.203397+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:18:28.259497+0000","scrub_duration":0.0011258259999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3","version":"84'9","reported_seq":254,"reported_epoch":185,"state":"active+clean","last_fresh":"2024-09-10T19:17:40.833345+0000","last_change":"2024-09-10T19:17:14.216709+0000","last_active":"2024-09-10T19:17:40.833345+0000","last_peered":"2024-09-10T19:17:40.833345+0000","last_clean":"2024-09-10T19:17:40.833345+0000","last_became_active":"2024-09-10T19:10:11.699690+0000","last_became_peered":"2024-09-10T19:10:11.699690+0000","last_unstale":"2024-09-10T19:17:40.833345+0000","last_undegraded":"2024-09-10T19:17:40.833345+0000","last_fullsized":"2024-09-10T19:17:40.833345+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'9","last_scrub_stamp":"2024-09-10T19:13:45.236094+0000","last_deep_scrub":"84'9","last_deep_scrub_stamp":"2024-09-10T19:11:53.879272+0000","last_clean_scrub_stamp":"2024-09-10T19:13:45.236094+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.036165602999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":162,"reported_epoch":185,"state":"active+clean","last_fresh":"2024-09-10T19:17:40.768836+0000","last_change":"2024-09-10T19:17:11.138938+0000","last_active":"2024-09-10T19:17:40.768836+0000","last_peered":"2024-09-10T19:17:40.768836+0000","last_clean":"2024-09-10T19:17:40.768836+0000","last_became_active":"2024-09-10T19:16:24.840096+0000","last_became_peered":"2024-09-10T19:16:24.840096+0000","last_unstale":"2024-09-10T19:17:40.768836+0000","last_undegraded":"2024-09-10T19:17:40.768836+0000","last_fullsized":"2024-09-10T19:17:40.768836+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:13:40.234289+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:13:40.234289+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0086405590000000008,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,5],"acting":[0,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.8","version":"186'6118","reported_seq":6175,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-09-10T19:17:44.751187+0000","last_change":"2024-09-10T19:17:26.958494+0000","last_active":"2024-09-10T19:17:44.751187+0000","last_peered":"2024-09-10T19:17:44.751187+0000","last_clean":"2024-09-10T19:17:44.751187+0000","last_became_active":"2024-09-10T19:17:26.955617+0000","last_became_peered":"2024-09-10T19:17:26.955617+0000","last_unstale":"2024-09-10T19:17:44.751187+0000","last_undegraded":"2024-09-10T19:17:44.751187+0000","last_fullsized":"2024-09-10T19:17:44.751187+0000","mapping_epoch":148,"log_start":"159'1400","ondisk_log_start":"159'1400","created":148,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:16:11.777014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:16:11.777014+0000","last_clean_scrub_stamp":"2024-09-10T19:16:11.777014+0000","objects_scrubbed":0,"log_size":4718,"log_dups_size":0,"ondisk_log_size":4718,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":216432640,"num_objects":3302,"num_object_clones":0,"num_object_copies":9906,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3302,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3302,"num_write_kb":211360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"155'31","reported_seq":143,"reported_epoch":185,"state":"active+clean","last_fresh":"2024-09-10T19:17:40.833479+0000","last_change":"2024-09-10T19:16:24.910865+0000","last_active":"2024-09-10T19:17:40.833479+0000","last_peered":"2024-09-10T19:17:40.833479+0000","last_clean":"2024-09-10T19:17:40.833479+0000","last_became_active":"2024-09-10T19:16:24.858154+0000","last_became_peered":"2024-09-10T19:16:24.858154+0000","last_unstale":"2024-09-10T19:17:40.833479+0000","last_undegraded":"2024-09-10T19:17:40.833479+0000","last_fullsized":"2024-09-10T19:17:40.833479+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"117'30","last_scrub_stamp":"2024-09-10T19:15:49.837844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:15:49.837844+0000","objects_scrubbed":0,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":47,"num_read_kb":32,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":358,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":173,"reported_epoch":185,"state":"active+clean","last_fresh":"2024-09-10T19:17:40.772165+0000","last_change":"2024-09-10T19:15:54.746979+0000","last_active":"2024-09-10T19:17:40.772165+0000","last_peered":"2024-09-10T19:17:40.772165+0000","last_clean":"2024-09-10T19:17:40.772165+0000","last_became_active":"2024-09-10T19:13:54.730096+0000","last_became_peered":"2024-09-10T19:13:54.730096+0000","last_unstale":"2024-09-10T19:17:40.772165+0000","last_undegraded":"2024-09-10T19:17:40.772165+0000","last_fullsized":"2024-09-10T19:17:40.772165+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:15:54.746920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:15:54.746920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001300037,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":178,"reported_epoch":185,"state":"active+clean","last_fresh":"2024-09-10T19:17:40.769978+0000","last_change":"2024-09-10T19:17:25.147473+0000","last_active":"2024-09-10T19:17:40.769978+0000","last_peered":"2024-09-10T19:17:40.769978+0000","last_clean":"2024-09-10T19:17:40.769978+0000","last_became_active":"2024-09-10T19:13:54.714017+0000","last_became_peered":"2024-09-10T19:13:54.714017+0000","last_unstale":"2024-09-10T19:17:40.769978+0000","last_undegraded":"2024-09-10T19:17:40.769978+0000","last_fullsized":"2024-09-10T19:17:40.769978+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:17:25.147396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:17:25.147396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:18:29.638750+0000","scrub_duration":0.004390054,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,6],"acting":[0,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":150,"reported_epoch":185,"state":"active+clean","last_fresh":"2024-09-10T19:17:40.765936+0000","last_change":"2024-09-10T19:16:32.818668+0000","last_active":"2024-09-10T19:17:40.765936+0000","last_peered":"2024-09-10T19:17:40.765936+0000","last_clean":"2024-09-10T19:17:40.765936+0000","last_became_active":"2024-09-10T19:10:13.721744+0000","last_became_peered":"2024-09-10T19:10:13.721744+0000","last_unstale":"2024-09-10T19:17:40.765936+0000","last_undegraded":"2024-09-10T19:17:40.765936+0000","last_fullsized":"2024-09-10T19:17:40.765936+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:16:32.818590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:16:32.818590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:17:49.356780+0000","scrub_duration":0.0047149059999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.b","version":"186'6873","reported_seq":6929,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-09-10T19:17:44.959167+0000","last_change":"2024-09-10T19:16:24.852409+0000","last_active":"2024-09-10T19:17:44.959167+0000","last_peered":"2024-09-10T19:17:44.959167+0000","last_clean":"2024-09-10T19:17:44.959167+0000","last_became_active":"2024-09-10T19:16:24.850682+0000","last_became_peered":"2024-09-10T19:16:24.850682+0000","last_unstale":"2024-09-10T19:17:44.959167+0000","last_undegraded":"2024-09-10T19:17:44.959167+0000","last_fullsized":"2024-09-10T19:17:44.959167+0000","mapping_epoch":154,"log_start":"163'3288","ondisk_log_start":"163'3288","created":148,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:16:11.777014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:16:11.777014+0000","last_clean_scrub_stamp":"2024-09-10T19:16:11.777014+0000","objects_scrubbed":0,"log_size":3585,"log_dups_size":0,"ondisk_log_size":3585,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":450428928,"num_objects":6873,"num_object_clones":0,"num_object_copies":20619,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6873,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6873,"num_write_kb":439872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.7","version":"82'18","reported_seq":189,"reported_epoch":185,"state":"active+clean","last_fresh":"2024-09-10T19:17:40.778850+0000","last_change":"2024-09-10T19:15:55.777428+0000","last_active":"2024-09-10T19:17:40.778850+0000","last_peered":"2024-09-10T19:17:40.778850+0000","last_clean":"2024-09-10T19:17:40.778850+0000","last_became_active":"2024-09-10T19:13:54.712065+0000","last_became_peered":"2024-09-10T19:13:54.712065+0000","last_unstale":"2024-09-10T19:17:40.778850+0000","last_undegraded":"2024-09-10T19:17:40.778850+0000","last_fullsized":"2024-09-10T19:17:40.778850+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'18","last_scrub_stamp":"2024-09-10T19:15:55.777361+0000","last_deep_scrub":"82'18","last_deep_scrub_stamp":"2024-09-10T19:15:55.777361+0000","last_clean_scrub_stamp":"2024-09-10T19:15:55.777361+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0023560040000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":19,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.0","version":"159'6","reported_seq":167,"reported_epoch":185,"state":"active+clean","last_fresh":"2024-09-10T19:17:44.020378+0000","last_change":"2024-09-10T19:16:32.650007+0000","last_active":"2024-09-10T19:17:44.020378+0000","last_peered":"2024-09-10T19:17:44.020378+0000","last_clean":"2024-09-10T19:17:44.020378+0000","last_became_active":"2024-09-10T19:16:24.846203+0000","last_became_peered":"2024-09-10T19:16:24.846203+0000","last_unstale":"2024-09-10T19:17:44.020378+0000","last_undegraded":"2024-09-10T19:17:44.020378+0000","last_fullsized":"2024-09-10T19:17:44.020378+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"159'6","last_scrub_stamp":"2024-09-10T19:16:32.649914+0000","last_deep_scrub":"159'6","last_deep_scrub_stamp":"2024-09-10T19:16:32.649914+0000","last_clean_scrub_stamp":"2024-09-10T19:16:32.649914+0000","objects_scrubbed":1,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.013939817,"objects_trimmed":0,"snaptrim_duration":0.00034454499999999997,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":86,"num_read_kb":81,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":389,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":54,"reported_epoch":185,"state":"active+clean","last_fresh":"2024-09-10T19:17:40.765642+0000","last_change":"2024-09-10T19:16:33.811962+0000","last_active":"2024-09-10T19:17:40.765642+0000","last_peered":"2024-09-10T19:17:40.765642+0000","last_clean":"2024-09-10T19:17:40.765642+0000","last_became_active":"2024-09-10T19:16:24.857505+0000","last_became_peered":"2024-09-10T19:16:24.857505+0000","last_unstale":"2024-09-10T19:17:40.765642+0000","last_undegraded":"2024-09-10T19:17:40.765642+0000","last_fullsized":"2024-09-10T19:17:40.765642+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:16:33.811869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:16:33.811869+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:17:45.100668+0000","scrub_duration":0.0027920340000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":54,"reported_epoch":185,"state":"active+clean","last_fresh":"2024-09-10T19:17:40.778636+0000","last_change":"2024-09-10T19:16:43.958311+0000","last_active":"2024-09-10T19:17:40.778636+0000","last_peered":"2024-09-10T19:17:40.778636+0000","last_clean":"2024-09-10T19:17:40.778636+0000","last_became_active":"2024-09-10T19:16:24.861481+0000","last_became_peered":"2024-09-10T19:16:24.861481+0000","last_unstale":"2024-09-10T19:17:40.778636+0000","last_undegraded":"2024-09-10T19:17:40.778636+0000","last_fullsized":"2024-09-10T19:17:40.778636+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:16:43.958224+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:16:43.958224+0000","last_clean_scrub_stamp":"2024-09-10T19:16:43.958224+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:18:09.697349+0000","scrub_duration":0.015970275999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,4],"acting":[5,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":153,"reported_epoch":185,"state":"active+clean","last_fresh":"2024-09-10T19:17:40.771592+0000","last_change":"2024-09-10T19:15:38.493179+0000","last_active":"2024-09-10T19:17:40.771592+0000","last_peered":"2024-09-10T19:17:40.771592+0000","last_clean":"2024-09-10T19:17:40.771592+0000","last_became_active":"2024-09-10T19:10:14.053697+0000","last_became_peered":"2024-09-10T19:10:14.053697+0000","last_unstale":"2024-09-10T19:17:40.771592+0000","last_undegraded":"2024-09-10T19:17:40.771592+0000","last_fullsized":"2024-09-10T19:17:40.771592+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:15:38.493102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:15:38.493102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0015544280000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.a","version":"186'7000","reported_seq":7044,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-09-10T19:17:44.832098+0000","last_change":"2024-09-10T19:16:24.865995+0000","last_active":"2024-09-10T19:17:44.832098+0000","last_peered":"2024-09-10T19:17:44.832098+0000","last_clean":"2024-09-10T19:17:44.832098+0000","last_became_active":"2024-09-10T19:16:24.864959+0000","last_became_peered":"2024-09-10T19:16:24.864959+0000","last_unstale":"2024-09-10T19:17:44.832098+0000","last_undegraded":"2024-09-10T19:17:44.832098+0000","last_fullsized":"2024-09-10T19:17:44.832098+0000","mapping_epoch":154,"log_start":"163'3225","ondisk_log_start":"163'3225","created":148,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:16:11.777014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:16:11.777014+0000","last_clean_scrub_stamp":"2024-09-10T19:16:11.777014+0000","objects_scrubbed":0,"log_size":3775,"log_dups_size":0,"ondisk_log_size":3775,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":458752000,"num_objects":7000,"num_object_clones":0,"num_object_copies":21000,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7000,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7000,"num_write_kb":448000,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.6","version":"66'12","reported_seq":229,"reported_epoch":185,"state":"active+clean","last_fresh":"2024-09-10T19:17:40.770751+0000","last_change":"2024-09-10T19:17:19.243613+0000","last_active":"2024-09-10T19:17:40.770751+0000","last_peered":"2024-09-10T19:17:40.770751+0000","last_clean":"2024-09-10T19:17:40.770751+0000","last_became_active":"2024-09-10T19:16:24.844323+0000","last_became_peered":"2024-09-10T19:16:24.844323+0000","last_unstale":"2024-09-10T19:17:40.770751+0000","last_undegraded":"2024-09-10T19:17:40.770751+0000","last_fullsized":"2024-09-10T19:17:40.770751+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'12","last_scrub_stamp":"2024-09-10T19:15:40.903279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:15:40.903279+0000","objects_scrubbed":6,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0271238,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,5],"acting":[0,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"61'1","reported_seq":171,"reported_epoch":185,"state":"active+clean","last_fresh":"2024-09-10T19:17:40.769598+0000","last_change":"2024-09-10T19:16:42.195540+0000","last_active":"2024-09-10T19:17:40.769598+0000","last_peered":"2024-09-10T19:17:40.769598+0000","last_clean":"2024-09-10T19:17:40.769598+0000","last_became_active":"2024-09-10T19:10:07.683259+0000","last_became_peered":"2024-09-10T19:10:07.683259+0000","last_unstale":"2024-09-10T19:17:40.769598+0000","last_undegraded":"2024-09-10T19:17:40.769598+0000","last_fullsized":"2024-09-10T19:17:40.769598+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'1","last_scrub_stamp":"2024-09-10T19:16:42.195463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:16:42.195463+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:17:50.284803+0000","scrub_duration":0.0016622189999999999,"objects_trimmed":0,"snaptrim_duration":0.0014341130000000001,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":173,"reported_epoch":185,"state":"active+clean","last_fresh":"2024-09-10T19:17:40.771653+0000","last_change":"2024-09-10T19:15:30.542477+0000","last_active":"2024-09-10T19:17:40.771653+0000","last_peered":"2024-09-10T19:17:40.771653+0000","last_clean":"2024-09-10T19:17:40.771653+0000","last_became_active":"2024-09-10T19:13:54.721486+0000","last_became_peered":"2024-09-10T19:13:54.721486+0000","last_unstale":"2024-09-10T19:17:40.771653+0000","last_undegraded":"2024-09-10T19:17:40.771653+0000","last_fullsized":"2024-09-10T19:17:40.771653+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:15:30.542399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:15:30.542399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0025142609999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":179,"reported_epoch":185,"state":"active+clean","last_fresh":"2024-09-10T19:17:40.778767+0000","last_change":"2024-09-10T19:17:28.070635+0000","last_active":"2024-09-10T19:17:40.778767+0000","last_peered":"2024-09-10T19:17:40.778767+0000","last_clean":"2024-09-10T19:17:40.778767+0000","last_became_active":"2024-09-10T19:13:54.711685+0000","last_became_peered":"2024-09-10T19:13:54.711685+0000","last_unstale":"2024-09-10T19:17:40.778767+0000","last_undegraded":"2024-09-10T19:17:40.778767+0000","last_fullsized":"2024-09-10T19:17:40.778767+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:17:28.070543+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:17:28.070543+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:18:49.229793+0000","scrub_duration":0.0017508000000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,4],"acting":[5,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":116,"reported_epoch":185,"state":"active+clean","last_fresh":"2024-09-10T19:17:40.778406+0000","last_change":"2024-09-10T19:16:28.786490+0000","last_active":"2024-09-10T19:17:40.778406+0000","last_peered":"2024-09-10T19:17:40.778406+0000","last_clean":"2024-09-10T19:17:40.778406+0000","last_became_active":"2024-09-10T19:13:54.722937+0000","last_became_peered":"2024-09-10T19:13:54.722937+0000","last_unstale":"2024-09-10T19:17:40.778406+0000","last_undegraded":"2024-09-10T19:17:40.778406+0000","last_fullsized":"2024-09-10T19:17:40.778406+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:16:28.786056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:16:28.786056+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:17:51.730597+0000","scrub_duration":0.0082035739999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.9","version":"186'6305","reported_seq":6362,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-09-10T19:17:44.691489+0000","last_change":"2024-09-10T19:17:26.964413+0000","last_active":"2024-09-10T19:17:44.691489+0000","last_peered":"2024-09-10T19:17:44.691489+0000","last_clean":"2024-09-10T19:17:44.691489+0000","last_became_active":"2024-09-10T19:17:26.961919+0000","last_became_peered":"2024-09-10T19:17:26.961919+0000","last_unstale":"2024-09-10T19:17:44.691489+0000","last_undegraded":"2024-09-10T19:17:44.691489+0000","last_fullsized":"2024-09-10T19:17:44.691489+0000","mapping_epoch":148,"log_start":"160'1700","ondisk_log_start":"160'1700","created":148,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:16:11.777014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:16:11.777014+0000","last_clean_scrub_stamp":"2024-09-10T19:16:11.777014+0000","objects_scrubbed":0,"log_size":4605,"log_dups_size":0,"ondisk_log_size":4605,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":223674368,"num_objects":3413,"num_object_clones":0,"num_object_copies":10239,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3413,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3413,"num_write_kb":218432,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,0],"acting":[6,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"80'21","reported_seq":203,"reported_epoch":185,"state":"active+clean","last_fresh":"2024-09-10T19:17:40.768020+0000","last_change":"2024-09-10T19:17:27.420321+0000","last_active":"2024-09-10T19:17:40.768020+0000","last_peered":"2024-09-10T19:17:40.768020+0000","last_clean":"2024-09-10T19:17:40.768020+0000","last_became_active":"2024-09-10T19:10:09.822184+0000","last_became_peered":"2024-09-10T19:10:09.822184+0000","last_unstale":"2024-09-10T19:17:40.768020+0000","last_undegraded":"2024-09-10T19:17:40.768020+0000","last_fullsized":"2024-09-10T19:17:40.768020+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"80'21","last_scrub_stamp":"2024-09-10T19:17:27.420267+0000","last_deep_scrub":"80'21","last_deep_scrub_stamp":"2024-09-10T19:11:39.836144+0000","last_clean_scrub_stamp":"2024-09-10T19:17:27.420267+0000","objects_scrubbed":7,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:18:35.485621+0000","scrub_duration":0.0018427859999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"63'2","reported_seq":173,"reported_epoch":185,"state":"active+clean","last_fresh":"2024-09-10T19:17:40.765458+0000","last_change":"2024-09-10T19:16:35.962952+0000","last_active":"2024-09-10T19:17:40.765458+0000","last_peered":"2024-09-10T19:17:40.765458+0000","last_clean":"2024-09-10T19:17:40.765458+0000","last_became_active":"2024-09-10T19:10:07.684784+0000","last_became_peered":"2024-09-10T19:10:07.684784+0000","last_unstale":"2024-09-10T19:17:40.765458+0000","last_undegraded":"2024-09-10T19:17:40.765458+0000","last_fullsized":"2024-09-10T19:17:40.765458+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'2","last_scrub_stamp":"2024-09-10T19:16:35.962804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:16:35.962804+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:18:04.954385+0000","scrub_duration":0.105395924,"objects_trimmed":0,"snaptrim_duration":0.00075522999999999996,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":176,"reported_epoch":185,"state":"active+clean","last_fresh":"2024-09-10T19:17:40.771935+0000","last_change":"2024-09-10T19:16:24.858963+0000","last_active":"2024-09-10T19:17:40.771935+0000","last_peered":"2024-09-10T19:17:40.771935+0000","last_clean":"2024-09-10T19:17:40.771935+0000","last_became_active":"2024-09-10T19:16:24.858662+0000","last_became_peered":"2024-09-10T19:16:24.858662+0000","last_unstale":"2024-09-10T19:17:40.771935+0000","last_undegraded":"2024-09-10T19:17:40.771935+0000","last_fullsized":"2024-09-10T19:17:40.771935+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:13:50.411500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:13:50.411500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0032604869999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"74'79","reported_seq":201,"reported_epoch":185,"state":"active+clean","last_fresh":"2024-09-10T19:17:40.771818+0000","last_change":"2024-09-10T19:13:46.419002+0000","last_active":"2024-09-10T19:17:40.771818+0000","last_peered":"2024-09-10T19:17:40.771818+0000","last_clean":"2024-09-10T19:17:40.771818+0000","last_became_active":"2024-09-10T19:09:56.634919+0000","last_became_peered":"2024-09-10T19:09:56.634919+0000","last_unstale":"2024-09-10T19:17:40.771818+0000","last_undegraded":"2024-09-10T19:17:40.771818+0000","last_fullsized":"2024-09-10T19:17:40.771818+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'79","last_scrub_stamp":"2024-09-10T19:13:46.418921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:08:00.699315+0000","last_clean_scrub_stamp":"2024-09-10T19:13:46.418921+0000","objects_scrubbed":2,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0029827999999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":162,"reported_epoch":185,"state":"active+clean","last_fresh":"2024-09-10T19:17:40.772266+0000","last_change":"2024-09-10T19:16:24.852723+0000","last_active":"2024-09-10T19:17:40.772266+0000","last_peered":"2024-09-10T19:17:40.772266+0000","last_clean":"2024-09-10T19:17:40.772266+0000","last_became_active":"2024-09-10T19:16:24.851652+0000","last_became_peered":"2024-09-10T19:16:24.851652+0000","last_unstale":"2024-09-10T19:17:40.772266+0000","last_undegraded":"2024-09-10T19:17:40.772266+0000","last_fullsized":"2024-09-10T19:17:40.772266+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:13:47.453228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:13:47.453228+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.010282463,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,0],"acting":[7,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":150,"reported_epoch":185,"state":"active+clean","last_fresh":"2024-09-10T19:17:40.778888+0000","last_change":"2024-09-10T19:13:05.390398+0000","last_active":"2024-09-10T19:17:40.778888+0000","last_peered":"2024-09-10T19:17:40.778888+0000","last_clean":"2024-09-10T19:17:40.778888+0000","last_became_active":"2024-09-10T19:10:13.720647+0000","last_became_peered":"2024-09-10T19:10:13.720647+0000","last_unstale":"2024-09-10T19:17:40.778888+0000","last_undegraded":"2024-09-10T19:17:40.778888+0000","last_fullsized":"2024-09-10T19:17:40.778888+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:13:05.390317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:13:05.390317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0058030640000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.6","version":"186'6262","reported_seq":6317,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-09-10T19:17:44.863033+0000","last_change":"2024-09-10T19:17:26.955780+0000","last_active":"2024-09-10T19:17:44.863033+0000","last_peered":"2024-09-10T19:17:44.863033+0000","last_clean":"2024-09-10T19:17:44.863033+0000","last_became_active":"2024-09-10T19:17:26.953133+0000","last_became_peered":"2024-09-10T19:17:26.953133+0000","last_unstale":"2024-09-10T19:17:44.863033+0000","last_undegraded":"2024-09-10T19:17:44.863033+0000","last_fullsized":"2024-09-10T19:17:44.863033+0000","mapping_epoch":148,"log_start":"159'1500","ondisk_log_start":"159'1500","created":148,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:16:11.777014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:16:11.777014+0000","last_clean_scrub_stamp":"2024-09-10T19:16:11.777014+0000","objects_scrubbed":0,"log_size":4762,"log_dups_size":0,"ondisk_log_size":4762,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":222134272,"num_objects":3389,"num_object_clones":0,"num_object_copies":10167,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3389,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3389,"num_write_kb":216928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,7],"acting":[0,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"66'19","reported_seq":225,"reported_epoch":185,"state":"active+clean","last_fresh":"2024-09-10T19:17:40.766786+0000","last_change":"2024-09-10T19:17:10.484589+0000","last_active":"2024-09-10T19:17:40.766786+0000","last_peered":"2024-09-10T19:17:40.766786+0000","last_clean":"2024-09-10T19:17:40.766786+0000","last_became_active":"2024-09-10T19:13:54.731274+0000","last_became_peered":"2024-09-10T19:13:54.731274+0000","last_unstale":"2024-09-10T19:17:40.766786+0000","last_undegraded":"2024-09-10T19:17:40.766786+0000","last_fullsized":"2024-09-10T19:17:40.766786+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'19","last_scrub_stamp":"2024-09-10T19:15:38.722581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:15:38.722581+0000","objects_scrubbed":9,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0054904940000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,5],"acting":[6,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":44,"reported_epoch":185,"state":"active+clean","last_fresh":"2024-09-10T19:17:40.778511+0000","last_change":"2024-09-10T19:16:24.847051+0000","last_active":"2024-09-10T19:17:40.778511+0000","last_peered":"2024-09-10T19:17:40.778511+0000","last_clean":"2024-09-10T19:17:40.778511+0000","last_became_active":"2024-09-10T19:16:24.846745+0000","last_became_peered":"2024-09-10T19:16:24.846745+0000","last_unstale":"2024-09-10T19:17:40.778511+0000","last_undegraded":"2024-09-10T19:17:40.778511+0000","last_fullsized":"2024-09-10T19:17:40.778511+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:15:53.803057+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:15:53.803057+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3],"acting":[5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":53,"reported_epoch":185,"state":"active+clean","last_fresh":"2024-09-10T19:17:40.769430+0000","last_change":"2024-09-10T19:16:27.240222+0000","last_active":"2024-09-10T19:17:40.769430+0000","last_peered":"2024-09-10T19:17:40.769430+0000","last_clean":"2024-09-10T19:17:40.769430+0000","last_became_active":"2024-09-10T19:16:24.844295+0000","last_became_peered":"2024-09-10T19:16:24.844295+0000","last_unstale":"2024-09-10T19:17:40.769430+0000","last_undegraded":"2024-09-10T19:17:40.769430+0000","last_fullsized":"2024-09-10T19:17:40.769430+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:16:27.240139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:16:27.240139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0020507199999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":160,"reported_epoch":185,"state":"active+clean","last_fresh":"2024-09-10T19:17:40.771732+0000","last_change":"2024-09-10T19:16:24.858456+0000","last_active":"2024-09-10T19:17:40.771732+0000","last_peered":"2024-09-10T19:17:40.771732+0000","last_clean":"2024-09-10T19:17:40.771732+0000","last_became_active":"2024-09-10T19:16:24.858172+0000","last_became_peered":"2024-09-10T19:16:24.858172+0000","last_unstale":"2024-09-10T19:17:40.771732+0000","last_undegraded":"2024-09-10T19:17:40.771732+0000","last_fullsized":"2024-09-10T19:17:40.771732+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:13:14.311620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:13:14.311620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0054101059999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,6],"acting":[7,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.7","version":"186'6269","reported_seq":6335,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-09-10T19:17:44.807862+0000","last_change":"2024-09-10T19:17:26.944772+0000","last_active":"2024-09-10T19:17:44.807862+0000","last_peered":"2024-09-10T19:17:44.807862+0000","last_clean":"2024-09-10T19:17:44.807862+0000","last_became_active":"2024-09-10T19:17:26.942858+0000","last_became_peered":"2024-09-10T19:17:26.942858+0000","last_unstale":"2024-09-10T19:17:44.807862+0000","last_undegraded":"2024-09-10T19:17:44.807862+0000","last_fullsized":"2024-09-10T19:17:44.807862+0000","mapping_epoch":154,"log_start":"160'1900","ondisk_log_start":"160'1900","created":148,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:16:11.777014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:16:11.777014+0000","last_clean_scrub_stamp":"2024-09-10T19:16:11.777014+0000","objects_scrubbed":0,"log_size":4369,"log_dups_size":0,"ondisk_log_size":4369,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220889088,"num_objects":3370,"num_object_clones":0,"num_object_copies":10110,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3370,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3370,"num_write_kb":215712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,4],"acting":[3,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.b","version":"66'9","reported_seq":199,"reported_epoch":185,"state":"active+clean","last_fresh":"2024-09-10T19:17:40.769672+0000","last_change":"2024-09-10T19:17:20.191334+0000","last_active":"2024-09-10T19:17:40.769672+0000","last_peered":"2024-09-10T19:17:40.769672+0000","last_clean":"2024-09-10T19:17:40.769672+0000","last_became_active":"2024-09-10T19:13:54.710613+0000","last_became_peered":"2024-09-10T19:13:54.710613+0000","last_unstale":"2024-09-10T19:17:40.769672+0000","last_undegraded":"2024-09-10T19:17:40.769672+0000","last_fullsized":"2024-09-10T19:17:40.769672+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'9","last_scrub_stamp":"2024-09-10T19:15:37.846807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:15:37.846807+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.012189228,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":132,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-09-10T19:17:46.693440+0000","last_change":"2024-09-10T19:16:24.848979+0000","last_active":"2024-09-10T19:17:46.693440+0000","last_peered":"2024-09-10T19:17:46.693440+0000","last_clean":"2024-09-10T19:17:46.693440+0000","last_became_active":"2024-09-10T19:16:24.848668+0000","last_became_peered":"2024-09-10T19:16:24.848668+0000","last_unstale":"2024-09-10T19:17:46.693440+0000","last_undegraded":"2024-09-10T19:17:46.693440+0000","last_fullsized":"2024-09-10T19:17:46.693440+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:15:50.850576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:11:44.254513+0000","last_clean_scrub_stamp":"2024-09-10T19:15:50.850576+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001815419,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":50,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-09-10T19:17:46.691389+0000","last_change":"2024-09-10T19:17:09.811398+0000","last_active":"2024-09-10T19:17:46.691389+0000","last_peered":"2024-09-10T19:17:46.691389+0000","last_clean":"2024-09-10T19:17:46.691389+0000","last_became_active":"2024-09-10T19:16:24.845709+0000","last_became_peered":"2024-09-10T19:16:24.845709+0000","last_unstale":"2024-09-10T19:17:46.691389+0000","last_undegraded":"2024-09-10T19:17:46.691389+0000","last_fullsized":"2024-09-10T19:17:46.691389+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:13:08.398036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:13:08.398036+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":169,"reported_epoch":185,"state":"active+clean","last_fresh":"2024-09-10T19:17:40.769480+0000","last_change":"2024-09-10T19:16:39.188737+0000","last_active":"2024-09-10T19:17:40.769480+0000","last_peered":"2024-09-10T19:17:40.769480+0000","last_clean":"2024-09-10T19:17:40.769480+0000","last_became_active":"2024-09-10T19:16:24.837483+0000","last_became_peered":"2024-09-10T19:16:24.837483+0000","last_unstale":"2024-09-10T19:17:40.769480+0000","last_undegraded":"2024-09-10T19:17:40.769480+0000","last_fullsized":"2024-09-10T19:17:40.769480+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:16:39.188682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:16:39.188682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:17:48.097207+0000","scrub_duration":0.0012633589999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.4","version":"186'6231","reported_seq":6301,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-09-10T19:17:44.727101+0000","last_change":"2024-09-10T19:17:26.998151+0000","last_active":"2024-09-10T19:17:44.727101+0000","last_peered":"2024-09-10T19:17:44.727101+0000","last_clean":"2024-09-10T19:17:44.727101+0000","last_became_active":"2024-09-10T19:17:26.972029+0000","last_became_peered":"2024-09-10T19:17:26.972029+0000","last_unstale":"2024-09-10T19:17:44.727101+0000","last_undegraded":"2024-09-10T19:17:44.727101+0000","last_fullsized":"2024-09-10T19:17:44.727101+0000","mapping_epoch":154,"log_start":"159'1500","ondisk_log_start":"159'1500","created":148,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:16:11.777014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:16:11.777014+0000","last_clean_scrub_stamp":"2024-09-10T19:16:11.777014+0000","objects_scrubbed":0,"log_size":4731,"log_dups_size":0,"ondisk_log_size":4731,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220758016,"num_objects":3368,"num_object_clones":0,"num_object_copies":10104,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3368,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3368,"num_write_kb":215584,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.8","version":"66'15","reported_seq":179,"reported_epoch":185,"state":"active+clean","last_fresh":"2024-09-10T19:17:40.771289+0000","last_change":"2024-09-10T19:12:50.449684+0000","last_active":"2024-09-10T19:17:40.771289+0000","last_peered":"2024-09-10T19:17:40.771289+0000","last_clean":"2024-09-10T19:17:40.771289+0000","last_became_active":"2024-09-10T19:10:09.820950+0000","last_became_peered":"2024-09-10T19:10:09.820950+0000","last_unstale":"2024-09-10T19:17:40.771289+0000","last_undegraded":"2024-09-10T19:17:40.771289+0000","last_fullsized":"2024-09-10T19:17:40.771289+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'15","last_scrub_stamp":"2024-09-10T19:12:50.449596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:12:50.449596+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0055729239999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":44,"reported_epoch":185,"state":"active+clean","last_fresh":"2024-09-10T19:17:40.766038+0000","last_change":"2024-09-10T19:16:24.857533+0000","last_active":"2024-09-10T19:17:40.766038+0000","last_peered":"2024-09-10T19:17:40.766038+0000","last_clean":"2024-09-10T19:17:40.766038+0000","last_became_active":"2024-09-10T19:16:24.857253+0000","last_became_peered":"2024-09-10T19:16:24.857253+0000","last_unstale":"2024-09-10T19:17:40.766038+0000","last_undegraded":"2024-09-10T19:17:40.766038+0000","last_fullsized":"2024-09-10T19:17:40.766038+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:15:38.677960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:15:38.677960+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.9","version":"80'9","reported_seq":252,"reported_epoch":185,"state":"active+clean","last_fresh":"2024-09-10T19:17:40.833351+0000","last_change":"2024-09-10T19:13:43.358984+0000","last_active":"2024-09-10T19:17:40.833351+0000","last_peered":"2024-09-10T19:17:40.833351+0000","last_clean":"2024-09-10T19:17:40.833351+0000","last_became_active":"2024-09-10T19:10:11.702878+0000","last_became_peered":"2024-09-10T19:10:11.702878+0000","last_unstale":"2024-09-10T19:17:40.833351+0000","last_undegraded":"2024-09-10T19:17:40.833351+0000","last_fullsized":"2024-09-10T19:17:40.833351+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"80'9","last_scrub_stamp":"2024-09-10T19:13:43.358901+0000","last_deep_scrub":"80'9","last_deep_scrub_stamp":"2024-09-10T19:11:35.174218+0000","last_clean_scrub_stamp":"2024-09-10T19:13:43.358901+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0066143260000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":150,"reported_epoch":185,"state":"active+clean","last_fresh":"2024-09-10T19:17:40.772718+0000","last_change":"2024-09-10T19:12:57.392898+0000","last_active":"2024-09-10T19:17:40.772718+0000","last_peered":"2024-09-10T19:17:40.772718+0000","last_clean":"2024-09-10T19:17:40.772718+0000","last_became_active":"2024-09-10T19:10:14.053875+0000","last_became_peered":"2024-09-10T19:10:14.053875+0000","last_unstale":"2024-09-10T19:17:40.772718+0000","last_undegraded":"2024-09-10T19:17:40.772718+0000","last_fullsized":"2024-09-10T19:17:40.772718+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:12:57.392844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:12:57.392844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0046180839999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.5","version":"186'6376","reported_seq":6432,"reported_epoch":186,"state":"active+clean","last_fresh":"2024-09-10T19:17:44.754604+0000","last_change":"2024-09-10T19:17:26.973165+0000","last_active":"2024-09-10T19:17:44.754604+0000","last_peered":"2024-09-10T19:17:44.754604+0000","last_clean":"2024-09-10T19:17:44.754604+0000","last_became_active":"2024-09-10T19:17:26.966286+0000","last_became_peered":"2024-09-10T19:17:26.966286+0000","last_unstale":"2024-09-10T19:17:44.754604+0000","last_undegraded":"2024-09-10T19:17:44.754604+0000","last_fullsized":"2024-09-10T19:17:44.754604+0000","mapping_epoch":148,"log_start":"160'1800","ondisk_log_start":"160'1800","created":148,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:16:11.777014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:16:11.777014+0000","last_clean_scrub_stamp":"2024-09-10T19:16:11.777014+0000","objects_scrubbed":0,"log_size":4576,"log_dups_size":0,"ondisk_log_size":4576,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":226230272,"num_objects":3452,"num_object_clones":0,"num_object_copies":10356,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3452,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3452,"num_write_kb":220928,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.9","version":"66'12","reported_seq":211,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-09-10T19:17:46.690945+0000","last_change":"2024-09-10T19:15:37.869593+0000","last_active":"2024-09-10T19:17:46.690945+0000","last_peered":"2024-09-10T19:17:46.690945+0000","last_clean":"2024-09-10T19:17:46.690945+0000","last_became_active":"2024-09-10T19:13:54.718152+0000","last_became_peered":"2024-09-10T19:13:54.718152+0000","last_unstale":"2024-09-10T19:17:46.690945+0000","last_undegraded":"2024-09-10T19:17:46.690945+0000","last_fullsized":"2024-09-10T19:17:46.690945+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'12","last_scrub_stamp":"2024-09-10T19:15:37.869451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:15:37.869451+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0025441249999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.f","version":"59'2","reported_seq":184,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-09-10T19:17:46.693201+0000","last_change":"2024-09-10T19:16:29.007627+0000","last_active":"2024-09-10T19:17:46.693201+0000","last_peered":"2024-09-10T19:17:46.693201+0000","last_clean":"2024-09-10T19:17:46.693201+0000","last_became_active":"2024-09-10T19:10:07.681719+0000","last_became_peered":"2024-09-10T19:10:07.681719+0000","last_unstale":"2024-09-10T19:17:46.693201+0000","last_undegraded":"2024-09-10T19:17:46.693201+0000","last_fullsized":"2024-09-10T19:17:46.693201+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'2","last_scrub_stamp":"2024-09-10T19:16:29.007251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:16:29.007251+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:17:56.729256+0000","scrub_duration":0.0079698319999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":14,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":126,"reported_epoch":185,"state":"active+clean","last_fresh":"2024-09-10T19:17:40.768872+0000","last_change":"2024-09-10T19:17:23.177960+0000","last_active":"2024-09-10T19:17:40.768872+0000","last_peered":"2024-09-10T19:17:40.768872+0000","last_clean":"2024-09-10T19:17:40.768872+0000","last_became_active":"2024-09-10T19:16:24.840091+0000","last_became_peered":"2024-09-10T19:16:24.840091+0000","last_unstale":"2024-09-10T19:17:40.768872+0000","last_undegraded":"2024-09-10T19:17:40.768872+0000","last_fullsized":"2024-09-10T19:17:40.768872+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:15:31.776481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:11:46.587846+0000","last_clean_scrub_stamp":"2024-09-10T19:15:31.776481+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001267142,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,4],"acting":[0,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":167,"reported_epoch":185,"state":"active+clean","last_fresh":"2024-09-10T19:17:40.767220+0000","last_change":"2024-09-10T19:15:41.093923+0000","last_active":"2024-09-10T19:17:40.767220+0000","last_peered":"2024-09-10T19:17:40.767220+0000","last_clean":"2024-09-10T19:17:40.767220+0000","last_became_active":"2024-09-10T19:13:54.711804+0000","last_became_peered":"2024-09-10T19:13:54.711804+0000","last_unstale":"2024-09-10T19:17:40.767220+0000","last_undegraded":"2024-09-10T19:17:40.767220+0000","last_fullsized":"2024-09-10T19:17:40.767220+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:15:41.093783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:15:41.093783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0023833050000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.16","version":"66'9","reported_seq":182,"reported_epoch":185,"state":"active+clean","last_fresh":"2024-09-10T19:17:40.770672+0000","last_change":"2024-09-10T19:17:12.151591+0000","last_active":"2024-09-10T19:17:40.770672+0000","last_peered":"2024-09-10T19:17:40.770672+0000","last_clean":"2024-09-10T19:17:40.770672+0000","last_became_active":"2024-09-10T19:10:09.822994+0000","last_became_peered":"2024-09-10T19:10:09.822994+0000","last_unstale":"2024-09-10T19:17:40.770672+0000","last_undegraded":"2024-09-10T19:17:40.770672+0000","last_fullsized":"2024-09-10T19:17:40.770672+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'9","last_scrub_stamp":"2024-09-10T19:13:50.172825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:13:50.172825+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0094381619999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":137,"reported_epoch":185,"state":"active+clean","last_fresh":"2024-09-10T19:17:40.769184+0000","last_change":"2024-09-10T19:17:16.305585+0000","last_active":"2024-09-10T19:17:40.769184+0000","last_peered":"2024-09-10T19:17:40.769184+0000","last_clean":"2024-09-10T19:17:40.769184+0000","last_became_active":"2024-09-10T19:16:24.844313+0000","last_became_peered":"2024-09-10T19:16:24.844313+0000","last_unstale":"2024-09-10T19:17:40.769184+0000","last_undegraded":"2024-09-10T19:17:40.769184+0000","last_fullsized":"2024-09-10T19:17:40.769184+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:17:16.305525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:11:41.458523+0000","last_clean_scrub_stamp":"2024-09-10T19:17:16.305525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:18:41.529454+0000","scrub_duration":0.0070771209999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,6],"acting":[0,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":169,"reported_epoch":185,"state":"active+clean","last_fresh":"2024-09-10T19:17:40.766763+0000","last_change":"2024-09-10T19:16:28.794727+0000","last_active":"2024-09-10T19:17:40.766763+0000","last_peered":"2024-09-10T19:17:40.766763+0000","last_clean":"2024-09-10T19:17:40.766763+0000","last_became_active":"2024-09-10T19:13:54.712275+0000","last_became_peered":"2024-09-10T19:13:54.712275+0000","last_unstale":"2024-09-10T19:17:40.766763+0000","last_undegraded":"2024-09-10T19:17:40.766763+0000","last_fullsized":"2024-09-10T19:17:40.766763+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:16:28.794649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:16:28.794649+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:17:53.505673+0000","scrub_duration":0.0027500379999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"66'1","reported_seq":60,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-09-10T19:17:46.692048+0000","last_change":"2024-09-10T19:16:24.862865+0000","last_active":"2024-09-10T19:17:46.692048+0000","last_peered":"2024-09-10T19:17:46.692048+0000","last_clean":"2024-09-10T19:17:46.692048+0000","last_became_active":"2024-09-10T19:16:24.837432+0000","last_became_peered":"2024-09-10T19:16:24.837432+0000","last_unstale":"2024-09-10T19:17:46.692048+0000","last_undegraded":"2024-09-10T19:17:46.692048+0000","last_fullsized":"2024-09-10T19:17:46.692048+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'1","last_scrub_stamp":"2024-09-10T19:15:48.839582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:15:48.839582+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":13,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":179,"reported_epoch":185,"state":"active+clean","last_fresh":"2024-09-10T19:17:40.767095+0000","last_change":"2024-09-10T19:17:17.479805+0000","last_active":"2024-09-10T19:17:40.767095+0000","last_peered":"2024-09-10T19:17:40.767095+0000","last_clean":"2024-09-10T19:17:40.767095+0000","last_became_active":"2024-09-10T19:13:54.724027+0000","last_became_peered":"2024-09-10T19:13:54.724027+0000","last_unstale":"2024-09-10T19:17:40.767095+0000","last_undegraded":"2024-09-10T19:17:40.767095+0000","last_fullsized":"2024-09-10T19:17:40.767095+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:15:36.714327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:15:36.714327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001831287,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,5],"acting":[6,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"66'6","reported_seq":196,"reported_epoch":185,"state":"active+clean","last_fresh":"2024-09-10T19:17:40.766841+0000","last_change":"2024-09-10T19:16:24.891057+0000","last_active":"2024-09-10T19:17:40.766841+0000","last_peered":"2024-09-10T19:17:40.766841+0000","last_clean":"2024-09-10T19:17:40.766841+0000","last_became_active":"2024-09-10T19:16:24.863398+0000","last_became_peered":"2024-09-10T19:16:24.863398+0000","last_unstale":"2024-09-10T19:17:40.766841+0000","last_undegraded":"2024-09-10T19:17:40.766841+0000","last_fullsized":"2024-09-10T19:17:40.766841+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'6","last_scrub_stamp":"2024-09-10T19:13:55.370650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:13:55.370650+0000","objects_scrubbed":3,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.071693797000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":169,"reported_epoch":185,"state":"active+clean","last_fresh":"2024-09-10T19:17:40.778712+0000","last_change":"2024-09-10T19:15:30.583958+0000","last_active":"2024-09-10T19:17:40.778712+0000","last_peered":"2024-09-10T19:17:40.778712+0000","last_clean":"2024-09-10T19:17:40.778712+0000","last_became_active":"2024-09-10T19:13:54.729917+0000","last_became_peered":"2024-09-10T19:13:54.729917+0000","last_unstale":"2024-09-10T19:17:40.778712+0000","last_undegraded":"2024-09-10T19:17:40.778712+0000","last_fullsized":"2024-09-10T19:17:40.778712+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:15:30.583878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:11:23.264793+0000","last_clean_scrub_stamp":"2024-09-10T19:15:30.583878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0015146960000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":160,"reported_epoch":185,"state":"active+clean","last_fresh":"2024-09-10T19:17:40.771335+0000","last_change":"2024-09-10T19:17:26.922378+0000","last_active":"2024-09-10T19:17:40.771335+0000","last_peered":"2024-09-10T19:17:40.771335+0000","last_clean":"2024-09-10T19:17:40.771335+0000","last_became_active":"2024-09-10T19:10:14.054360+0000","last_became_peered":"2024-09-10T19:10:14.054360+0000","last_unstale":"2024-09-10T19:17:40.771335+0000","last_undegraded":"2024-09-10T19:17:40.771335+0000","last_fullsized":"2024-09-10T19:17:40.771335+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:17:26.922305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:17:26.922305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:18:28.472292+0000","scrub_duration":0.0016181940000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.18","version":"189'6277","reported_seq":12499,"reported_epoch":189,"state":"active+clean","last_fresh":"2024-09-10T19:17:49.744638+0000","last_change":"2024-09-10T19:17:45.681866+0000","last_active":"2024-09-10T19:17:49.744638+0000","last_peered":"2024-09-10T19:17:49.744638+0000","last_clean":"2024-09-10T19:17:49.744638+0000","last_became_active":"2024-09-10T19:17:45.677163+0000","last_became_peered":"2024-09-10T19:17:45.677163+0000","last_unstale":"2024-09-10T19:17:49.744638+0000","last_undegraded":"2024-09-10T19:17:49.744638+0000","last_fullsized":"2024-09-10T19:17:49.744638+0000","mapping_epoch":186,"log_start":"159'1400","ondisk_log_start":"159'1400","created":177,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:16:11.777014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:16:11.777014+0000","last_clean_scrub_stamp":"2024-09-10T19:16:11.777014+0000","objects_scrubbed":0,"log_size":4877,"log_dups_size":0,"ondisk_log_size":4877,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":226852864,"num_objects":3462,"num_object_clones":0,"num_object_copies":10386,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3462,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3462,"num_write_kb":221536,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3071,"num_bytes_recovered":201261056,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,5],"acting":[4,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.14","version":"80'15","reported_seq":225,"reported_epoch":185,"state":"active+clean","last_fresh":"2024-09-10T19:17:40.766666+0000","last_change":"2024-09-10T19:16:24.881305+0000","last_active":"2024-09-10T19:17:40.766666+0000","last_peered":"2024-09-10T19:17:40.766666+0000","last_clean":"2024-09-10T19:17:40.766666+0000","last_became_active":"2024-09-10T19:16:24.850702+0000","last_became_peered":"2024-09-10T19:16:24.850702+0000","last_unstale":"2024-09-10T19:17:40.766666+0000","last_undegraded":"2024-09-10T19:17:40.766666+0000","last_fullsized":"2024-09-10T19:17:40.766666+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"80'15","last_scrub_stamp":"2024-09-10T19:15:46.019949+0000","last_deep_scrub":"80'15","last_deep_scrub_stamp":"2024-09-10T19:11:41.719310+0000","last_clean_scrub_stamp":"2024-09-10T19:15:46.019949+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0021039190000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,5],"acting":[3,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":156,"reported_epoch":185,"state":"active+clean","last_fresh":"2024-09-10T19:17:40.772266+0000","last_change":"2024-09-10T19:12:46.506129+0000","last_active":"2024-09-10T19:17:40.772266+0000","last_peered":"2024-09-10T19:17:40.772266+0000","last_clean":"2024-09-10T19:17:40.772266+0000","last_became_active":"2024-09-10T19:10:07.685870+0000","last_became_peered":"2024-09-10T19:10:07.685870+0000","last_unstale":"2024-09-10T19:17:40.772266+0000","last_undegraded":"2024-09-10T19:17:40.772266+0000","last_fullsized":"2024-09-10T19:17:40.772266+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:12:46.506071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:12:46.506071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001257935,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"117'10","reported_seq":273,"reported_epoch":185,"state":"active+clean","last_fresh":"2024-09-10T19:17:40.833461+0000","last_change":"2024-09-10T19:16:23.844790+0000","last_active":"2024-09-10T19:17:40.833461+0000","last_peered":"2024-09-10T19:17:40.833461+0000","last_clean":"2024-09-10T19:17:40.833461+0000","last_became_active":"2024-09-10T19:13:54.724780+0000","last_became_peered":"2024-09-10T19:13:54.724780+0000","last_unstale":"2024-09-10T19:17:40.833461+0000","last_undegraded":"2024-09-10T19:17:40.833461+0000","last_fullsized":"2024-09-10T19:17:40.833461+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"117'10","last_scrub_stamp":"2024-09-10T19:16:23.844724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:16:23.844724+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.012497955999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3],"acting":[5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":159,"reported_epoch":185,"state":"active+clean","last_fresh":"2024-09-10T19:17:40.770312+0000","last_change":"2024-09-10T19:16:40.213422+0000","last_active":"2024-09-10T19:17:40.770312+0000","last_peered":"2024-09-10T19:17:40.770312+0000","last_clean":"2024-09-10T19:17:40.770312+0000","last_became_active":"2024-09-10T19:10:13.721035+0000","last_became_peered":"2024-09-10T19:10:13.721035+0000","last_unstale":"2024-09-10T19:17:40.770312+0000","last_undegraded":"2024-09-10T19:17:40.770312+0000","last_fullsized":"2024-09-10T19:17:40.770312+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:16:40.213321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:16:40.213321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:18:03.471480+0000","scrub_duration":0.0051408239999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.19","version":"186'6290","reported_seq":12250,"reported_epoch":186,"state":"active+remapped+backfilling","last_fresh":"2024-09-10T19:17:44.849336+0000","last_change":"2024-09-10T19:17:30.043970+0000","last_active":"2024-09-10T19:17:44.849336+0000","last_peered":"2024-09-10T19:17:44.849336+0000","last_clean":"2024-09-10T19:17:27.217100+0000","last_became_active":"2024-09-10T19:17:30.026998+0000","last_became_peered":"2024-09-10T19:17:30.026998+0000","last_unstale":"2024-09-10T19:17:44.849336+0000","last_undegraded":"2024-09-10T19:17:44.849336+0000","last_fullsized":"2024-09-10T19:17:44.849336+0000","mapping_epoch":180,"log_start":"160'1700","ondisk_log_start":"160'1700","created":177,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:16:11.777014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:16:11.777014+0000","last_clean_scrub_stamp":"2024-09-10T19:16:11.777014+0000","objects_scrubbed":0,"log_size":4590,"log_dups_size":0,"ondisk_log_size":4590,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":222691328,"num_objects":3398,"num_object_clones":0,"num_object_copies":10194,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":256,"num_objects_unfound":0,"num_objects_dirty":3398,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3398,"num_write_kb":217472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2946,"num_bytes_recovered":193069056,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,0],"acting":[5,0,6],"avail_no_missing":["5","0","6"],"object_location_counts":[{"shards":"0,5,6","objects":3398}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.15","version":"66'9","reported_seq":210,"reported_epoch":185,"state":"active+clean","last_fresh":"2024-09-10T19:17:40.779110+0000","last_change":"2024-09-10T19:16:24.882653+0000","last_active":"2024-09-10T19:17:40.779110+0000","last_peered":"2024-09-10T19:17:40.779110+0000","last_clean":"2024-09-10T19:17:40.779110+0000","last_became_active":"2024-09-10T19:16:24.859740+0000","last_became_peered":"2024-09-10T19:16:24.859740+0000","last_unstale":"2024-09-10T19:17:40.779110+0000","last_undegraded":"2024-09-10T19:17:40.779110+0000","last_fullsized":"2024-09-10T19:17:40.779110+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'9","last_scrub_stamp":"2024-09-10T19:12:52.416748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:12:52.416748+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0047220839999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":165,"reported_epoch":185,"state":"active+clean","last_fresh":"2024-09-10T19:17:40.769878+0000","last_change":"2024-09-10T19:16:37.243942+0000","last_active":"2024-09-10T19:17:40.769878+0000","last_peered":"2024-09-10T19:17:40.769878+0000","last_clean":"2024-09-10T19:17:40.769878+0000","last_became_active":"2024-09-10T19:10:07.688499+0000","last_became_peered":"2024-09-10T19:10:07.688499+0000","last_unstale":"2024-09-10T19:17:40.769878+0000","last_undegraded":"2024-09-10T19:17:40.769878+0000","last_fullsized":"2024-09-10T19:17:40.769878+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:16:37.243859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:16:37.243859+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:18:05.478908+0000","scrub_duration":0.0052374580000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"155'9","reported_seq":265,"reported_epoch":185,"state":"active+clean","last_fresh":"2024-09-10T19:17:40.833406+0000","last_change":"2024-09-10T19:16:24.881133+0000","last_active":"2024-09-10T19:17:40.833406+0000","last_peered":"2024-09-10T19:17:40.833406+0000","last_clean":"2024-09-10T19:17:40.833406+0000","last_became_active":"2024-09-10T19:16:24.850850+0000","last_became_peered":"2024-09-10T19:16:24.850850+0000","last_unstale":"2024-09-10T19:17:40.833406+0000","last_undegraded":"2024-09-10T19:17:40.833406+0000","last_fullsized":"2024-09-10T19:17:40.833406+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'8","last_scrub_stamp":"2024-09-10T19:12:59.723712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:12:59.723712+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0083838109999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,4],"acting":[3,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":165,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-09-10T19:17:46.692924+0000","last_change":"2024-09-10T19:17:10.866121+0000","last_active":"2024-09-10T19:17:46.692924+0000","last_peered":"2024-09-10T19:17:46.692924+0000","last_clean":"2024-09-10T19:17:46.692924+0000","last_became_active":"2024-09-10T19:16:24.854280+0000","last_became_peered":"2024-09-10T19:16:24.854280+0000","last_unstale":"2024-09-10T19:17:46.692924+0000","last_undegraded":"2024-09-10T19:17:46.692924+0000","last_fullsized":"2024-09-10T19:17:46.692924+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:13:09.463001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:13:09.463001+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.012199573,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.12","version":"79'13","reported_seq":196,"reported_epoch":185,"state":"active+clean","last_fresh":"2024-09-10T19:17:40.766529+0000","last_change":"2024-09-10T19:17:11.485802+0000","last_active":"2024-09-10T19:17:40.766529+0000","last_peered":"2024-09-10T19:17:40.766529+0000","last_clean":"2024-09-10T19:17:40.766529+0000","last_became_active":"2024-09-10T19:16:24.845450+0000","last_became_peered":"2024-09-10T19:16:24.845450+0000","last_unstale":"2024-09-10T19:17:40.766529+0000","last_undegraded":"2024-09-10T19:17:40.766529+0000","last_fullsized":"2024-09-10T19:17:40.766529+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'13","last_scrub_stamp":"2024-09-10T19:15:29.854117+0000","last_deep_scrub":"79'13","last_deep_scrub_stamp":"2024-09-10T19:11:22.072400+0000","last_clean_scrub_stamp":"2024-09-10T19:15:29.854117+0000","objects_scrubbed":4,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0066627300000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.14","version":"59'1","reported_seq":173,"reported_epoch":185,"state":"active+clean","last_fresh":"2024-09-10T19:17:40.768332+0000","last_change":"2024-09-10T19:16:27.594895+0000","last_active":"2024-09-10T19:17:40.768332+0000","last_peered":"2024-09-10T19:17:40.768332+0000","last_clean":"2024-09-10T19:17:40.768332+0000","last_became_active":"2024-09-10T19:10:07.683263+0000","last_became_peered":"2024-09-10T19:10:07.683263+0000","last_unstale":"2024-09-10T19:17:40.768332+0000","last_undegraded":"2024-09-10T19:17:40.768332+0000","last_fullsized":"2024-09-10T19:17:40.768332+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'1","last_scrub_stamp":"2024-09-10T19:16:27.594788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:16:27.594788+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:17:46.479670+0000","scrub_duration":0.0091369840000000008,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":362,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":169,"reported_epoch":185,"state":"active+clean","last_fresh":"2024-09-10T19:17:40.767575+0000","last_change":"2024-09-10T19:15:55.925313+0000","last_active":"2024-09-10T19:17:40.767575+0000","last_peered":"2024-09-10T19:17:40.767575+0000","last_clean":"2024-09-10T19:17:40.767575+0000","last_became_active":"2024-09-10T19:13:54.730185+0000","last_became_peered":"2024-09-10T19:13:54.730185+0000","last_unstale":"2024-09-10T19:17:40.767575+0000","last_undegraded":"2024-09-10T19:17:40.767575+0000","last_fullsized":"2024-09-10T19:17:40.767575+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:15:55.925228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:15:55.925228+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0017202299999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":166,"reported_epoch":185,"state":"active+clean","last_fresh":"2024-09-10T19:17:40.768782+0000","last_change":"2024-09-10T19:16:23.273671+0000","last_active":"2024-09-10T19:17:40.768782+0000","last_peered":"2024-09-10T19:17:40.768782+0000","last_clean":"2024-09-10T19:17:40.768782+0000","last_became_active":"2024-09-10T19:13:54.714861+0000","last_became_peered":"2024-09-10T19:13:54.714861+0000","last_unstale":"2024-09-10T19:17:40.768782+0000","last_undegraded":"2024-09-10T19:17:40.768782+0000","last_fullsized":"2024-09-10T19:17:40.768782+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:16:23.273615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:16:23.273615+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:17:47.995106+0000","scrub_duration":0.0013972139999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,6],"acting":[0,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"66'11","reported_seq":207,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-09-10T19:17:46.691691+0000","last_change":"2024-09-10T19:15:53.883613+0000","last_active":"2024-09-10T19:17:46.691691+0000","last_peered":"2024-09-10T19:17:46.691691+0000","last_clean":"2024-09-10T19:17:46.691691+0000","last_became_active":"2024-09-10T19:13:54.720034+0000","last_became_peered":"2024-09-10T19:13:54.720034+0000","last_unstale":"2024-09-10T19:17:46.691691+0000","last_undegraded":"2024-09-10T19:17:46.691691+0000","last_fullsized":"2024-09-10T19:17:46.691691+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'11","last_scrub_stamp":"2024-09-10T19:15:53.883548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:15:53.883548+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001858326,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":106,"reported_epoch":185,"state":"active+clean","last_fresh":"2024-09-10T19:17:40.779156+0000","last_change":"2024-09-10T19:13:54.723475+0000","last_active":"2024-09-10T19:17:40.779156+0000","last_peered":"2024-09-10T19:17:40.779156+0000","last_clean":"2024-09-10T19:17:40.779156+0000","last_became_active":"2024-09-10T19:13:54.722947+0000","last_became_peered":"2024-09-10T19:13:54.722947+0000","last_unstale":"2024-09-10T19:17:40.779156+0000","last_undegraded":"2024-09-10T19:17:40.779156+0000","last_fullsized":"2024-09-10T19:17:40.779156+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:12:44.249765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:11:10.031245+0000","last_clean_scrub_stamp":"2024-09-10T19:12:44.249765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,6],"acting":[5,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":116,"reported_epoch":185,"state":"active+clean","last_fresh":"2024-09-10T19:17:40.772519+0000","last_change":"2024-09-10T19:15:36.666310+0000","last_active":"2024-09-10T19:17:40.772519+0000","last_peered":"2024-09-10T19:17:40.772519+0000","last_clean":"2024-09-10T19:17:40.772519+0000","last_became_active":"2024-09-10T19:13:54.729703+0000","last_became_peered":"2024-09-10T19:13:54.729703+0000","last_unstale":"2024-09-10T19:17:40.772519+0000","last_undegraded":"2024-09-10T19:17:40.772519+0000","last_fullsized":"2024-09-10T19:17:40.772519+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:15:36.666216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:15:36.666216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.003489884,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":150,"reported_epoch":185,"state":"active+clean","last_fresh":"2024-09-10T19:17:40.765100+0000","last_change":"2024-09-10T19:12:54.691886+0000","last_active":"2024-09-10T19:17:40.765100+0000","last_peered":"2024-09-10T19:17:40.765100+0000","last_clean":"2024-09-10T19:17:40.765100+0000","last_became_active":"2024-09-10T19:10:13.724201+0000","last_became_peered":"2024-09-10T19:10:13.724201+0000","last_unstale":"2024-09-10T19:17:40.765100+0000","last_undegraded":"2024-09-10T19:17:40.765100+0000","last_fullsized":"2024-09-10T19:17:40.765100+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:12:54.691810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:12:54.691810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.006702927,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"66'4","reported_seq":183,"reported_epoch":185,"state":"active+clean","last_fresh":"2024-09-10T19:17:40.766526+0000","last_change":"2024-09-10T19:15:33.995417+0000","last_active":"2024-09-10T19:17:40.766526+0000","last_peered":"2024-09-10T19:17:40.766526+0000","last_clean":"2024-09-10T19:17:40.766526+0000","last_became_active":"2024-09-10T19:13:54.711125+0000","last_became_peered":"2024-09-10T19:13:54.711125+0000","last_unstale":"2024-09-10T19:17:40.766526+0000","last_undegraded":"2024-09-10T19:17:40.766526+0000","last_fullsized":"2024-09-10T19:17:40.766526+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'4","last_scrub_stamp":"2024-09-10T19:15:33.995274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:15:33.995274+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.011108425,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,4],"acting":[3,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":176,"reported_epoch":185,"state":"active+clean","last_fresh":"2024-09-10T19:17:40.778908+0000","last_change":"2024-09-10T19:16:33.766560+0000","last_active":"2024-09-10T19:17:40.778908+0000","last_peered":"2024-09-10T19:17:40.778908+0000","last_clean":"2024-09-10T19:17:40.778908+0000","last_became_active":"2024-09-10T19:16:24.852460+0000","last_became_peered":"2024-09-10T19:16:24.852460+0000","last_unstale":"2024-09-10T19:17:40.778908+0000","last_undegraded":"2024-09-10T19:17:40.778908+0000","last_fullsized":"2024-09-10T19:17:40.778908+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:16:33.766474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:16:33.766474+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:18:01.711870+0000","scrub_duration":0.0042170250000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,3],"acting":[5,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":162,"reported_epoch":185,"state":"active+clean","last_fresh":"2024-09-10T19:17:40.766835+0000","last_change":"2024-09-10T19:16:24.545188+0000","last_active":"2024-09-10T19:17:40.766835+0000","last_peered":"2024-09-10T19:17:40.766835+0000","last_clean":"2024-09-10T19:17:40.766835+0000","last_became_active":"2024-09-10T19:10:11.698536+0000","last_became_peered":"2024-09-10T19:10:11.698536+0000","last_unstale":"2024-09-10T19:17:40.766835+0000","last_undegraded":"2024-09-10T19:17:40.766835+0000","last_fullsized":"2024-09-10T19:17:40.766835+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:16:24.545134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:11:31.837944+0000","last_clean_scrub_stamp":"2024-09-10T19:16:24.545134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:17:54.187236+0000","scrub_duration":0.00120104,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":170,"reported_epoch":185,"state":"active+clean","last_fresh":"2024-09-10T19:17:40.770724+0000","last_change":"2024-09-10T19:16:24.852651+0000","last_active":"2024-09-10T19:17:40.770724+0000","last_peered":"2024-09-10T19:17:40.770724+0000","last_clean":"2024-09-10T19:17:40.770724+0000","last_became_active":"2024-09-10T19:16:24.852356+0000","last_became_peered":"2024-09-10T19:16:24.852356+0000","last_unstale":"2024-09-10T19:17:40.770724+0000","last_undegraded":"2024-09-10T19:17:40.770724+0000","last_fullsized":"2024-09-10T19:17:40.770724+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:15:44.523530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:15:44.523530+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.027001213999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,3],"acting":[7,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.11","version":"66'11","reported_seq":64,"reported_epoch":185,"state":"active+clean","last_fresh":"2024-09-10T19:17:40.772474+0000","last_change":"2024-09-10T19:15:45.158623+0000","last_active":"2024-09-10T19:17:40.772474+0000","last_peered":"2024-09-10T19:17:40.772474+0000","last_clean":"2024-09-10T19:17:40.772474+0000","last_became_active":"2024-09-10T19:15:45.149864+0000","last_became_peered":"2024-09-10T19:15:45.149864+0000","last_unstale":"2024-09-10T19:17:40.772474+0000","last_undegraded":"2024-09-10T19:17:40.772474+0000","last_fullsized":"2024-09-10T19:17:40.772474+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'11","last_scrub_stamp":"2024-09-10T19:13:05.729854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:13:05.729854+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.010667395,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":176,"reported_epoch":185,"state":"active+clean","last_fresh":"2024-09-10T19:17:40.768154+0000","last_change":"2024-09-10T19:16:24.855851+0000","last_active":"2024-09-10T19:17:40.768154+0000","last_peered":"2024-09-10T19:17:40.768154+0000","last_clean":"2024-09-10T19:17:40.768154+0000","last_became_active":"2024-09-10T19:16:24.855541+0000","last_became_peered":"2024-09-10T19:16:24.855541+0000","last_unstale":"2024-09-10T19:17:40.768154+0000","last_undegraded":"2024-09-10T19:17:40.768154+0000","last_fullsized":"2024-09-10T19:17:40.768154+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:16:23.543097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:13:05.876655+0000","last_clean_scrub_stamp":"2024-09-10T19:16:23.543097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001181779,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,0],"acting":[6,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":152,"reported_epoch":185,"state":"active+clean","last_fresh":"2024-09-10T19:17:40.771229+0000","last_change":"2024-09-10T19:13:41.326028+0000","last_active":"2024-09-10T19:17:40.771229+0000","last_peered":"2024-09-10T19:17:40.771229+0000","last_clean":"2024-09-10T19:17:40.771229+0000","last_became_active":"2024-09-10T19:10:11.699013+0000","last_became_peered":"2024-09-10T19:10:11.699013+0000","last_unstale":"2024-09-10T19:17:40.771229+0000","last_undegraded":"2024-09-10T19:17:40.771229+0000","last_fullsized":"2024-09-10T19:17:40.771229+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:13:41.325941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:11:57.112092+0000","last_clean_scrub_stamp":"2024-09-10T19:13:41.325941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001598676,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":150,"reported_epoch":185,"state":"active+clean","last_fresh":"2024-09-10T19:17:40.767151+0000","last_change":"2024-09-10T19:13:15.812326+0000","last_active":"2024-09-10T19:17:40.767151+0000","last_peered":"2024-09-10T19:17:40.767151+0000","last_clean":"2024-09-10T19:17:40.767151+0000","last_became_active":"2024-09-10T19:10:14.056172+0000","last_became_peered":"2024-09-10T19:10:14.056172+0000","last_unstale":"2024-09-10T19:17:40.767151+0000","last_undegraded":"2024-09-10T19:17:40.767151+0000","last_fullsized":"2024-09-10T19:17:40.767151+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:13:15.812244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:13:15.812244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0048044389999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.12","version":"186'6379","reported_seq":6446,"reported_epoch":186,"state":"active+remapped+backfill_toofull","last_fresh":"2024-09-10T19:17:44.868300+0000","last_change":"2024-09-10T19:17:30.034851+0000","last_active":"2024-09-10T19:17:44.868300+0000","last_peered":"2024-09-10T19:17:44.868300+0000","last_clean":"2024-09-10T19:17:27.172505+0000","last_became_active":"2024-09-10T19:17:30.027866+0000","last_became_peered":"2024-09-10T19:17:30.027866+0000","last_unstale":"2024-09-10T19:17:44.868300+0000","last_undegraded":"2024-09-10T19:17:44.868300+0000","last_fullsized":"2024-09-10T19:17:44.868300+0000","mapping_epoch":180,"log_start":"159'1500","ondisk_log_start":"159'1500","created":177,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:16:11.777014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:16:11.777014+0000","last_clean_scrub_stamp":"2024-09-10T19:16:11.777014+0000","objects_scrubbed":0,"log_size":4879,"log_dups_size":0,"ondisk_log_size":4879,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":226885632,"num_objects":3462,"num_object_clones":0,"num_object_copies":10386,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":6924,"num_objects_unfound":0,"num_objects_dirty":3462,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3462,"num_write_kb":221568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,4],"acting":[0,4,5],"avail_no_missing":["0","4","5"],"object_location_counts":[{"shards":"0,4,5","objects":3462}],"blocked_by":[],"up_primary":6,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1c","version":"80'2","reported_seq":176,"reported_epoch":185,"state":"active+clean","last_fresh":"2024-09-10T19:17:40.777336+0000","last_change":"2024-09-10T19:16:32.648844+0000","last_active":"2024-09-10T19:17:40.777336+0000","last_peered":"2024-09-10T19:17:40.777336+0000","last_clean":"2024-09-10T19:17:40.777336+0000","last_became_active":"2024-09-10T19:16:24.851220+0000","last_became_peered":"2024-09-10T19:16:24.851220+0000","last_unstale":"2024-09-10T19:17:40.777336+0000","last_undegraded":"2024-09-10T19:17:40.777336+0000","last_fullsized":"2024-09-10T19:17:40.777336+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"80'2","last_scrub_stamp":"2024-09-10T19:16:32.648738+0000","last_deep_scrub":"80'2","last_deep_scrub_stamp":"2024-09-10T19:11:30.272043+0000","last_clean_scrub_stamp":"2024-09-10T19:16:32.648738+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0066232920000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":369,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":369,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,3],"acting":[7,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":156,"reported_epoch":185,"state":"active+clean","last_fresh":"2024-09-10T19:17:40.771590+0000","last_change":"2024-09-10T19:13:14.449643+0000","last_active":"2024-09-10T19:17:40.771590+0000","last_peered":"2024-09-10T19:17:40.771590+0000","last_clean":"2024-09-10T19:17:40.771590+0000","last_became_active":"2024-09-10T19:10:07.689877+0000","last_became_peered":"2024-09-10T19:10:07.689877+0000","last_unstale":"2024-09-10T19:17:40.771590+0000","last_undegraded":"2024-09-10T19:17:40.771590+0000","last_fullsized":"2024-09-10T19:17:40.771590+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:13:14.449593+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:13:14.449593+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001195542,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1e","version":"80'15","reported_seq":183,"reported_epoch":185,"state":"active+clean","last_fresh":"2024-09-10T19:17:40.770592+0000","last_change":"2024-09-10T19:16:33.258114+0000","last_active":"2024-09-10T19:17:40.770592+0000","last_peered":"2024-09-10T19:17:40.770592+0000","last_clean":"2024-09-10T19:17:40.770592+0000","last_became_active":"2024-09-10T19:10:09.823061+0000","last_became_peered":"2024-09-10T19:10:09.823061+0000","last_unstale":"2024-09-10T19:17:40.770592+0000","last_undegraded":"2024-09-10T19:17:40.770592+0000","last_fullsized":"2024-09-10T19:17:40.770592+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"80'15","last_scrub_stamp":"2024-09-10T19:16:33.258031+0000","last_deep_scrub":"80'15","last_deep_scrub_stamp":"2024-09-10T19:11:25.883919+0000","last_clean_scrub_stamp":"2024-09-10T19:16:33.258031+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:17:57.303783+0000","scrub_duration":0.0040337309999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1f","version":"66'8","reported_seq":267,"reported_epoch":185,"state":"active+clean","last_fresh":"2024-09-10T19:17:40.834281+0000","last_change":"2024-09-10T19:17:32.477395+0000","last_active":"2024-09-10T19:17:40.834281+0000","last_peered":"2024-09-10T19:17:40.834281+0000","last_clean":"2024-09-10T19:17:40.834281+0000","last_became_active":"2024-09-10T19:10:11.695427+0000","last_became_peered":"2024-09-10T19:10:11.695427+0000","last_unstale":"2024-09-10T19:17:40.834281+0000","last_undegraded":"2024-09-10T19:17:40.834281+0000","last_fullsized":"2024-09-10T19:17:40.834281+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'8","last_scrub_stamp":"2024-09-10T19:15:53.664208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:15:53.664208+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001849699,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.13","version":"186'6481","reported_seq":8810,"reported_epoch":186,"state":"active+recovering+remapped","last_fresh":"2024-09-10T19:17:45.006825+0000","last_change":"2024-09-10T19:17:30.105506+0000","last_active":"2024-09-10T19:17:45.006825+0000","last_peered":"2024-09-10T19:17:45.006825+0000","last_clean":"2024-09-10T19:16:54.964816+0000","last_became_active":"2024-09-10T19:17:30.042990+0000","last_became_peered":"2024-09-10T19:17:30.042990+0000","last_unstale":"2024-09-10T19:17:45.006825+0000","last_undegraded":"2024-09-10T19:17:45.006825+0000","last_fullsized":"2024-09-10T19:17:45.006825+0000","mapping_epoch":180,"log_start":"160'2100","ondisk_log_start":"160'2100","created":177,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:16:11.777014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:16:11.777014+0000","last_clean_scrub_stamp":"2024-09-10T19:16:11.777014+0000","objects_scrubbed":0,"log_size":4381,"log_dups_size":0,"ondisk_log_size":4381,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":227442688,"num_objects":3471,"num_object_clones":0,"num_object_copies":10413,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1368,"num_objects_unfound":0,"num_objects_dirty":3471,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3471,"num_write_kb":222112,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":584,"num_bytes_recovered":38240256,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,4],"acting":[3,5,7],"avail_no_missing":["3","5","6"],"object_location_counts":[{"shards":"3,5,6","objects":1368},{"shards":"3,5,7","objects":2103}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":126,"reported_epoch":185,"state":"active+clean","last_fresh":"2024-09-10T19:17:40.778696+0000","last_change":"2024-09-10T19:16:24.863000+0000","last_active":"2024-09-10T19:17:40.778696+0000","last_peered":"2024-09-10T19:17:40.778696+0000","last_clean":"2024-09-10T19:17:40.778696+0000","last_became_active":"2024-09-10T19:16:24.862695+0000","last_became_peered":"2024-09-10T19:16:24.862695+0000","last_unstale":"2024-09-10T19:17:40.778696+0000","last_undegraded":"2024-09-10T19:17:40.778696+0000","last_fullsized":"2024-09-10T19:17:40.778696+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:15:29.580605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:15:29.580605+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0047886710000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":166,"reported_epoch":185,"state":"active+clean","last_fresh":"2024-09-10T19:17:40.765706+0000","last_change":"2024-09-10T19:16:41.881053+0000","last_active":"2024-09-10T19:17:40.765706+0000","last_peered":"2024-09-10T19:17:40.765706+0000","last_clean":"2024-09-10T19:17:40.765706+0000","last_became_active":"2024-09-10T19:10:07.682689+0000","last_became_peered":"2024-09-10T19:10:07.682689+0000","last_unstale":"2024-09-10T19:17:40.765706+0000","last_undegraded":"2024-09-10T19:17:40.765706+0000","last_fullsized":"2024-09-10T19:17:40.765706+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:16:41.880992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:16:41.880992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:17:54.061215+0000","scrub_duration":0.0044984990000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"80'16","reported_seq":235,"reported_epoch":185,"state":"active+clean","last_fresh":"2024-09-10T19:17:40.768067+0000","last_change":"2024-09-10T19:17:15.446029+0000","last_active":"2024-09-10T19:17:40.768067+0000","last_peered":"2024-09-10T19:17:40.768067+0000","last_clean":"2024-09-10T19:17:40.768067+0000","last_became_active":"2024-09-10T19:13:54.713563+0000","last_became_peered":"2024-09-10T19:13:54.713563+0000","last_unstale":"2024-09-10T19:17:40.768067+0000","last_undegraded":"2024-09-10T19:17:40.768067+0000","last_fullsized":"2024-09-10T19:17:40.768067+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"80'16","last_scrub_stamp":"2024-09-10T19:15:47.700523+0000","last_deep_scrub":"80'16","last_deep_scrub_stamp":"2024-09-10T19:11:40.853611+0000","last_clean_scrub_stamp":"2024-09-10T19:15:47.700523+0000","objects_scrubbed":5,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0019149010000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,0],"acting":[6,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":171,"reported_epoch":185,"state":"active+clean","last_fresh":"2024-09-10T19:17:40.770512+0000","last_change":"2024-09-10T19:16:34.222429+0000","last_active":"2024-09-10T19:17:40.770512+0000","last_peered":"2024-09-10T19:17:40.770512+0000","last_clean":"2024-09-10T19:17:40.770512+0000","last_became_active":"2024-09-10T19:16:24.834713+0000","last_became_peered":"2024-09-10T19:16:24.834713+0000","last_unstale":"2024-09-10T19:17:40.770512+0000","last_undegraded":"2024-09-10T19:17:40.770512+0000","last_fullsized":"2024-09-10T19:17:40.770512+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:16:34.222365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:11:28.822327+0000","last_clean_scrub_stamp":"2024-09-10T19:16:34.222365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.014893412999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.10","version":"186'6394","reported_seq":13082,"reported_epoch":186,"state":"active+remapped+backfilling","last_fresh":"2024-09-10T19:17:45.015231+0000","last_change":"2024-09-10T19:17:30.042065+0000","last_active":"2024-09-10T19:17:45.015231+0000","last_peered":"2024-09-10T19:17:45.015231+0000","last_clean":"2024-09-10T19:17:27.139201+0000","last_became_active":"2024-09-10T19:17:30.034661+0000","last_became_peered":"2024-09-10T19:17:30.034661+0000","last_unstale":"2024-09-10T19:17:45.015231+0000","last_undegraded":"2024-09-10T19:17:45.015231+0000","last_fullsized":"2024-09-10T19:17:45.015231+0000","mapping_epoch":180,"log_start":"160'1900","ondisk_log_start":"160'1900","created":177,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:16:11.777014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:16:11.777014+0000","last_clean_scrub_stamp":"2024-09-10T19:16:11.777014+0000","objects_scrubbed":0,"log_size":4494,"log_dups_size":0,"ondisk_log_size":4494,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":227934208,"num_objects":3478,"num_object_clones":0,"num_object_copies":10434,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3478,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3478,"num_write_kb":222592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3312,"num_bytes_recovered":217055232,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,4],"acting":[3,5,4],"avail_no_missing":["3","4","5","7"],"object_location_counts":[{"shards":"3,4,5","objects":3478}],"blocked_by":[],"up_primary":7,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":162,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-09-10T19:17:46.692302+0000","last_change":"2024-09-10T19:16:36.960796+0000","last_active":"2024-09-10T19:17:46.692302+0000","last_peered":"2024-09-10T19:17:46.692302+0000","last_clean":"2024-09-10T19:17:46.692302+0000","last_became_active":"2024-09-10T19:10:14.053619+0000","last_became_peered":"2024-09-10T19:10:14.053619+0000","last_unstale":"2024-09-10T19:17:46.692302+0000","last_undegraded":"2024-09-10T19:17:46.692302+0000","last_fullsized":"2024-09-10T19:17:46.692302+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:16:36.960695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:16:36.960695+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:17:59.816942+0000","scrub_duration":0.011430734999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":166,"reported_epoch":185,"state":"active+clean","last_fresh":"2024-09-10T19:17:40.766841+0000","last_change":"2024-09-10T19:16:30.666084+0000","last_active":"2024-09-10T19:17:40.766841+0000","last_peered":"2024-09-10T19:17:40.766841+0000","last_clean":"2024-09-10T19:17:40.766841+0000","last_became_active":"2024-09-10T19:10:07.685997+0000","last_became_peered":"2024-09-10T19:10:07.685997+0000","last_unstale":"2024-09-10T19:17:40.766841+0000","last_undegraded":"2024-09-10T19:17:40.766841+0000","last_fullsized":"2024-09-10T19:17:40.766841+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:16:30.666001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:16:30.666001+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:17:57.903923+0000","scrub_duration":0.0041927830000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1c","version":"78'22","reported_seq":115,"reported_epoch":185,"state":"active+clean","last_fresh":"2024-09-10T19:17:40.766575+0000","last_change":"2024-09-10T19:16:24.870445+0000","last_active":"2024-09-10T19:17:40.766575+0000","last_peered":"2024-09-10T19:17:40.766575+0000","last_clean":"2024-09-10T19:17:40.766575+0000","last_became_active":"2024-09-10T19:16:24.845764+0000","last_became_peered":"2024-09-10T19:16:24.845764+0000","last_unstale":"2024-09-10T19:17:40.766575+0000","last_undegraded":"2024-09-10T19:17:40.766575+0000","last_fullsized":"2024-09-10T19:17:40.766575+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"78'22","last_scrub_stamp":"2024-09-10T19:15:37.702624+0000","last_deep_scrub":"78'22","last_deep_scrub_stamp":"2024-09-10T19:11:21.118937+0000","last_clean_scrub_stamp":"2024-09-10T19:15:37.702624+0000","objects_scrubbed":7,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":122,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-09-10T19:17:46.691611+0000","last_change":"2024-09-10T19:15:33.008615+0000","last_active":"2024-09-10T19:17:46.691611+0000","last_peered":"2024-09-10T19:17:46.691611+0000","last_clean":"2024-09-10T19:17:46.691611+0000","last_became_active":"2024-09-10T19:13:54.709055+0000","last_became_peered":"2024-09-10T19:13:54.709055+0000","last_unstale":"2024-09-10T19:17:46.691611+0000","last_undegraded":"2024-09-10T19:17:46.691611+0000","last_fullsized":"2024-09-10T19:17:46.691611+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:15:33.008542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:15:33.008542+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.004557363,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,5],"acting":[4,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.11","version":"186'6366","reported_seq":6444,"reported_epoch":186,"state":"active+remapped+backfill_toofull","last_fresh":"2024-09-10T19:17:44.796364+0000","last_change":"2024-09-10T19:17:30.055262+0000","last_active":"2024-09-10T19:17:44.796364+0000","last_peered":"2024-09-10T19:17:44.796364+0000","last_clean":"2024-09-10T19:17:27.057735+0000","last_became_active":"2024-09-10T19:17:30.038219+0000","last_became_peered":"2024-09-10T19:17:30.038219+0000","last_unstale":"2024-09-10T19:17:44.796364+0000","last_undegraded":"2024-09-10T19:17:44.796364+0000","last_fullsized":"2024-09-10T19:17:44.796364+0000","mapping_epoch":180,"log_start":"159'1600","ondisk_log_start":"159'1600","created":177,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:16:11.777014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:16:11.777014+0000","last_clean_scrub_stamp":"2024-09-10T19:16:11.777014+0000","objects_scrubbed":0,"log_size":4766,"log_dups_size":0,"ondisk_log_size":4766,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":225935360,"num_objects":3448,"num_object_clones":0,"num_object_copies":10344,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":6896,"num_objects_unfound":0,"num_objects_dirty":3448,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3448,"num_write_kb":220640,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,0,3],"avail_no_missing":["5","0","3"],"object_location_counts":[{"shards":"0,3,5","objects":3448}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":150,"reported_epoch":185,"state":"active+clean","last_fresh":"2024-09-10T19:17:40.766444+0000","last_change":"2024-09-10T19:12:51.767586+0000","last_active":"2024-09-10T19:17:40.766444+0000","last_peered":"2024-09-10T19:17:40.766444+0000","last_clean":"2024-09-10T19:17:40.766444+0000","last_became_active":"2024-09-10T19:10:14.053698+0000","last_became_peered":"2024-09-10T19:10:14.053698+0000","last_unstale":"2024-09-10T19:17:40.766444+0000","last_undegraded":"2024-09-10T19:17:40.766444+0000","last_fullsized":"2024-09-10T19:17:40.766444+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:12:51.767056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:12:51.767056+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.006796519,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":156,"reported_epoch":185,"state":"active+clean","last_fresh":"2024-09-10T19:17:40.767823+0000","last_change":"2024-09-10T19:12:50.719167+0000","last_active":"2024-09-10T19:17:40.767823+0000","last_peered":"2024-09-10T19:17:40.767823+0000","last_clean":"2024-09-10T19:17:40.767823+0000","last_became_active":"2024-09-10T19:10:07.682811+0000","last_became_peered":"2024-09-10T19:10:07.682811+0000","last_unstale":"2024-09-10T19:17:40.767823+0000","last_undegraded":"2024-09-10T19:17:40.767823+0000","last_fullsized":"2024-09-10T19:17:40.767823+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:12:50.718663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:12:50.718663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0042138399999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1d","version":"66'12","reported_seq":206,"reported_epoch":185,"state":"active+clean","last_fresh":"2024-09-10T19:17:40.765732+0000","last_change":"2024-09-10T19:15:52.923561+0000","last_active":"2024-09-10T19:17:40.765732+0000","last_peered":"2024-09-10T19:17:40.765732+0000","last_clean":"2024-09-10T19:17:40.765732+0000","last_became_active":"2024-09-10T19:13:54.714629+0000","last_became_peered":"2024-09-10T19:13:54.714629+0000","last_unstale":"2024-09-10T19:17:40.765732+0000","last_undegraded":"2024-09-10T19:17:40.765732+0000","last_fullsized":"2024-09-10T19:17:40.765732+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'12","last_scrub_stamp":"2024-09-10T19:15:52.923498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:15:52.923498+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0019450450000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":174,"reported_epoch":188,"state":"active+clean","last_fresh":"2024-09-10T19:17:46.692251+0000","last_change":"2024-09-10T19:16:37.970105+0000","last_active":"2024-09-10T19:17:46.692251+0000","last_peered":"2024-09-10T19:17:46.692251+0000","last_clean":"2024-09-10T19:17:46.692251+0000","last_became_active":"2024-09-10T19:16:24.843013+0000","last_became_peered":"2024-09-10T19:16:24.843013+0000","last_unstale":"2024-09-10T19:17:46.692251+0000","last_undegraded":"2024-09-10T19:17:46.692251+0000","last_fullsized":"2024-09-10T19:17:46.692251+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:16:37.970030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:16:37.970030+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:17:58.177693+0000","scrub_duration":0.003625063,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":8,"num_pg":26,"stat_sum":{"num_bytes":7198343168,"num_objects":109838,"num_object_clones":0,"num_object_copies":329514,"num_objects_missing_on_primary":1247,"num_objects_missing":1247,"num_objects_degraded":0,"num_objects_misplaced":33737,"num_objects_unfound":0,"num_objects_dirty":109838,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":109838,"num_write_kb":7029632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13852,"num_bytes_recovered":907804672,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":22757572608,"data_stored":22757572608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":115433,"ondisk_log_size":115433,"up":78,"acting":78,"num_store_stats":7},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":382,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":1528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":73,"ondisk_log_size":73,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":177,"num_object_clones":0,"num_object_copies":531,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":177,"num_whiteouts":0,"num_read":695,"num_read_kb":452,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":173,"num_bytes_recovered":3218,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":540672,"data_stored":14242,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":445,"ondisk_log_size":445,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":86,"num_read_kb":81,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":389,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":2002,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":9,"ondisk_log_size":9,"up":9,"acting":9,"num_store_stats":7},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1328,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":27,"num_read_kb":27,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":3984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4,"ondisk_log_size":4,"up":96,"acting":96,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79,"ondisk_log_size":79,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":54,"seq":231928234080,"num_pgs":87,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":4008796,"kb_used_data":3698960,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":89736356,"statfs":{"total":95995035648,"available":91890028544,"internally_reserved":0,"allocated":3787735040,"data_stored":3787198608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 19:17:03 2024","interfaces":[{"interface":"back","average":{"1min":1.1379999999999999,"5min":1.8859999999999999,"15min":0.98199999999999998},"min":{"1min":0.246,"5min":0.215,"15min":0.215},"max":{"1min":8.3770000000000007,"5min":35.551000000000002,"15min":35.551000000000002},"last":0.34100000000000003},{"interface":"front","average":{"1min":1.081,"5min":1.6919999999999999,"15min":0.92000000000000004},"min":{"1min":0.19500000000000001,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":8.1440000000000001,"5min":35.503999999999998,"15min":35.503999999999998},"last":0.59599999999999997}]},{"osd":1,"last update":"Tue Sep 10 19:17:25 2024","interfaces":[{"interface":"back","average":{"1min":1.9870000000000001,"5min":0.81699999999999995,"15min":0.622},"min":{"1min":0.30099999999999999,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":8.3499999999999996,"5min":8.3499999999999996,"15min":8.3499999999999996},"last":0.38500000000000001},{"interface":"front","average":{"1min":1.482,"5min":0.86199999999999999,"15min":0.75900000000000001},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":5.609,"5min":5.609,"15min":5.609},"last":0.435}]},{"osd":2,"last update":"Tue Sep 10 19:17:03 2024","interfaces":[{"interface":"back","average":{"1min":0.76300000000000001,"5min":1.109,"15min":0.71699999999999997},"min":{"1min":0.27300000000000002,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":2.0139999999999998,"5min":31.940999999999999,"15min":31.940999999999999},"last":0.71899999999999997},{"interface":"front","average":{"1min":1.274,"5min":1.319,"15min":0.80400000000000005},"min":{"1min":0.35799999999999998,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":8.0839999999999996,"5min":25.056000000000001,"15min":25.056000000000001},"last":0.48499999999999999}]},{"osd":3,"last update":"Tue Sep 10 19:17:03 2024","interfaces":[{"interface":"back","average":{"1min":1.121,"5min":1.8660000000000001,"15min":0.97699999999999998},"min":{"1min":0.217,"5min":0.217,"15min":0.217},"max":{"1min":4.4969999999999999,"5min":35.277999999999999,"15min":35.277999999999999},"last":0.69499999999999995},{"interface":"front","average":{"1min":0.94299999999999995,"5min":1.738,"15min":0.92700000000000005},"min":{"1min":0.28499999999999998,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":2.4260000000000002,"5min":36.512999999999998,"15min":36.512999999999998},"last":0.46300000000000002}]},{"osd":4,"last update":"Tue Sep 10 19:17:03 2024","interfaces":[{"interface":"back","average":{"1min":1.3380000000000001,"5min":0.71699999999999997,"15min":0.54200000000000004},"min":{"1min":0.23300000000000001,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":8.1859999999999999,"5min":8.1859999999999999,"15min":8.1859999999999999},"last":1.2569999999999999},{"interface":"front","average":{"1min":1.0780000000000001,"5min":0.64600000000000002,"15min":0.52200000000000002},"min":{"1min":0.29399999999999998,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":4.5469999999999997,"5min":4.5469999999999997,"15min":4.5469999999999997},"last":0.78900000000000003}]},{"osd":5,"last update":"Tue Sep 10 19:17:03 2024","interfaces":[{"interface":"back","average":{"1min":1.3700000000000001,"5min":0.93700000000000006,"15min":0.63500000000000001},"min":{"1min":0.32000000000000001,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":8.1660000000000004,"5min":11.621,"15min":11.621},"last":0.73999999999999999},{"interface":"front","average":{"1min":1.859,"5min":0.89500000000000002,"15min":0.61799999999999999},"min":{"1min":0.34699999999999998,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":9.2680000000000007,"5min":9.2680000000000007,"15min":9.2680000000000007},"last":0.63100000000000001}]},{"osd":6,"last update":"Tue Sep 10 19:17:03 2024","interfaces":[{"interface":"back","average":{"1min":1.738,"5min":0.90400000000000003,"15min":0.63800000000000001},"min":{"1min":0.32100000000000001,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":8.2590000000000003,"5min":8.2590000000000003,"15min":8.2590000000000003},"last":0.65200000000000002},{"interface":"front","average":{"1min":1.522,"5min":0.79000000000000004,"15min":0.59099999999999997},"min":{"1min":0.373,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":8.4130000000000003,"5min":8.4130000000000003,"15min":8.4130000000000003},"last":0.84399999999999997}]}]},{"osd":6,"up_from":47,"seq":201863463013,"num_pgs":83,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3601088,"kb_used_data":3291256,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":90144064,"statfs":{"total":95995035648,"available":92307521536,"internally_reserved":0,"allocated":3370246144,"data_stored":3369725973,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 19:16:46 2024","interfaces":[{"interface":"back","average":{"1min":0.63800000000000001,"5min":2.194,"15min":1.077},"min":{"1min":0.245,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":2.1509999999999998,"5min":112.988,"15min":112.988},"last":0.27400000000000002},{"interface":"front","average":{"1min":0.82999999999999996,"5min":2.218,"15min":1.097},"min":{"1min":0.34300000000000003,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":2.9969999999999999,"5min":113.18300000000001,"15min":113.18300000000001},"last":0.34999999999999998}]},{"osd":1,"last update":"Tue Sep 10 19:17:28 2024","interfaces":[{"interface":"back","average":{"1min":0.878,"5min":0.58599999999999997,"15min":0.53800000000000003},"min":{"1min":0.27000000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":4.2919999999999998,"5min":4.2919999999999998,"15min":4.2919999999999998},"last":0.33000000000000002},{"interface":"front","average":{"1min":1.1379999999999999,"5min":0.75,"15min":0.68600000000000005},"min":{"1min":0.26400000000000001,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":2.5470000000000002,"5min":2.996,"15min":2.996},"last":0.45200000000000001}]},{"osd":2,"last update":"Tue Sep 10 19:16:46 2024","interfaces":[{"interface":"back","average":{"1min":0.78200000000000003,"5min":2.0129999999999999,"15min":1.0309999999999999},"min":{"1min":0.35399999999999998,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":3.7959999999999998,"5min":112.864,"15min":112.864},"last":0.37},{"interface":"front","average":{"1min":0.80600000000000005,"5min":2.1299999999999999,"15min":1.0489999999999999},"min":{"1min":0.36199999999999999,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":3.0369999999999999,"5min":112.822,"15min":112.822},"last":0.41699999999999998}]},{"osd":3,"last update":"Tue Sep 10 19:16:46 2024","interfaces":[{"interface":"back","average":{"1min":0.754,"5min":2.4580000000000002,"15min":1.1599999999999999},"min":{"1min":0.34799999999999998,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":2.0209999999999999,"5min":125.56399999999999,"15min":125.56399999999999},"last":0.67200000000000004},{"interface":"front","average":{"1min":0.79300000000000004,"5min":2.5430000000000001,"15min":1.222},"min":{"1min":0.36199999999999999,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":3.8290000000000002,"5min":125.47,"15min":125.47},"last":0.53500000000000003}]},{"osd":4,"last update":"Tue Sep 10 19:16:46 2024","interfaces":[{"interface":"back","average":{"1min":0.92100000000000004,"5min":0.59399999999999997,"15min":0.434},"min":{"1min":0.316,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":3.0630000000000002,"5min":3.3410000000000002,"15min":3.3410000000000002},"last":0.29999999999999999},{"interface":"front","average":{"1min":0.878,"5min":0.627,"15min":0.46100000000000002},"min":{"1min":0.27400000000000002,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":2.7349999999999999,"5min":2.7349999999999999,"15min":2.7349999999999999},"last":0.71199999999999997}]},{"osd":5,"last update":"Tue Sep 10 19:16:46 2024","interfaces":[{"interface":"back","average":{"1min":1.093,"5min":1.119,"15min":0.626},"min":{"1min":0.30499999999999999,"5min":0.248,"15min":0.248},"max":{"1min":3.1099999999999999,"5min":44.765000000000001,"15min":44.765000000000001},"last":0.65300000000000002},{"interface":"front","average":{"1min":1.3240000000000001,"5min":1.288,"15min":0.68400000000000005},"min":{"1min":0.307,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":3.0590000000000002,"5min":43.036000000000001,"15min":43.036000000000001},"last":0.63400000000000001}]},{"osd":7,"last update":"Tue Sep 10 19:17:07 2024","interfaces":[{"interface":"back","average":{"1min":1.5820000000000001,"5min":0.78900000000000003,"15min":0.56599999999999995},"min":{"1min":0.315,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":8.9390000000000001,"5min":8.9390000000000001,"15min":8.9390000000000001},"last":0.46999999999999997},{"interface":"front","average":{"1min":2.0150000000000001,"5min":0.93899999999999995,"15min":0.61399999999999999},"min":{"1min":0.33300000000000002,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":10.298,"5min":10.298,"15min":10.298},"last":2.125}]}]},{"osd":5,"up_from":39,"seq":167503724650,"num_pgs":80,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":4238388,"kb_used_data":3928556,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":89506764,"statfs":{"total":95995035648,"available":91654926336,"internally_reserved":0,"allocated":4022841344,"data_stored":4022333758,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 19:17:19 2024","interfaces":[{"interface":"back","average":{"1min":1.7869999999999999,"5min":1.1379999999999999,"15min":0.69899999999999995},"min":{"1min":0.35999999999999999,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":5.5519999999999996,"5min":5.851,"15min":5.851},"last":2.6549999999999998},{"interface":"front","average":{"1min":1.1759999999999999,"5min":0.90600000000000003,"15min":0.621},"min":{"1min":0.253,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":3.7130000000000001,"5min":5.923,"15min":5.923},"last":1.1060000000000001}]},{"osd":1,"last update":"Tue Sep 10 19:17:23 2024","interfaces":[{"interface":"back","average":{"1min":1.3140000000000001,"5min":0.78400000000000003,"15min":0.69599999999999995},"min":{"1min":0.28599999999999998,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":3.77,"5min":3.77,"15min":3.77},"last":0.97699999999999998},{"interface":"front","average":{"1min":1.3440000000000001,"5min":0.67100000000000004,"15min":0.55900000000000005},"min":{"1min":0.28399999999999997,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":5.0570000000000004,"5min":5.0570000000000004,"15min":5.0570000000000004},"last":0.42899999999999999}]},{"osd":2,"last update":"Tue Sep 10 19:17:19 2024","interfaces":[{"interface":"back","average":{"1min":2.117,"5min":1.0629999999999999,"15min":0.67600000000000005},"min":{"1min":0.378,"5min":0.253,"15min":0.253},"max":{"1min":9.9399999999999995,"5min":9.9399999999999995,"15min":9.9399999999999995},"last":2.6240000000000001},{"interface":"front","average":{"1min":1.238,"5min":1.0329999999999999,"15min":0.67500000000000004},"min":{"1min":0.33200000000000002,"5min":0.224,"15min":0.224},"max":{"1min":3.7909999999999999,"5min":6.3789999999999996,"15min":6.3789999999999996},"last":1.1359999999999999}]},{"osd":3,"last update":"Tue Sep 10 19:17:19 2024","interfaces":[{"interface":"back","average":{"1min":2.1120000000000001,"5min":1.0309999999999999,"15min":0.68700000000000006},"min":{"1min":0.34100000000000003,"5min":0.224,"15min":0.224},"max":{"1min":9.9019999999999992,"5min":9.9019999999999992,"15min":9.9019999999999992},"last":2.52},{"interface":"front","average":{"1min":1.2869999999999999,"5min":0.90400000000000003,"15min":0.65600000000000003},"min":{"1min":0.29899999999999999,"5min":0.248,"15min":0.248},"max":{"1min":3.7469999999999999,"5min":5.3869999999999996,"15min":5.3869999999999996},"last":1.075}]},{"osd":4,"last update":"Tue Sep 10 19:17:19 2024","interfaces":[{"interface":"back","average":{"1min":2.173,"5min":0.95499999999999996,"15min":0.56299999999999994},"min":{"1min":0.24399999999999999,"5min":0.23699999999999999,"15min":0.20699999999999999},"max":{"1min":10.414,"5min":10.414,"15min":10.414},"last":2.1400000000000001},{"interface":"front","average":{"1min":2.3330000000000002,"5min":1.048,"15min":0.60099999999999998},"min":{"1min":0.313,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":10.518000000000001,"5min":10.518000000000001,"15min":10.518000000000001},"last":4.306}]},{"osd":6,"last update":"Tue Sep 10 19:16:45 2024","interfaces":[{"interface":"back","average":{"1min":0.89900000000000002,"5min":0.85499999999999998,"15min":0.56299999999999994},"min":{"1min":0.36599999999999999,"5min":0.27100000000000002,"15min":0.223},"max":{"1min":3.6509999999999998,"5min":6.617,"15min":6.617},"last":2.6800000000000002},{"interface":"front","average":{"1min":0.621,"5min":0.76000000000000001,"15min":0.52900000000000003},"min":{"1min":0.34100000000000003,"5min":0.26900000000000002,"15min":0.25},"max":{"1min":1.2250000000000001,"5min":5.9349999999999996,"15min":5.9349999999999996},"last":1.0349999999999999}]},{"osd":7,"last update":"Tue Sep 10 19:17:05 2024","interfaces":[{"interface":"back","average":{"1min":1.3100000000000001,"5min":0.874,"15min":0.629},"min":{"1min":0.35999999999999999,"5min":0.307,"15min":0.29099999999999998},"max":{"1min":5.0640000000000001,"5min":5.9660000000000002,"15min":5.9660000000000002},"last":2.6930000000000001},{"interface":"front","average":{"1min":1.2410000000000001,"5min":0.752,"15min":0.56999999999999995},"min":{"1min":0.28499999999999998,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":5.0149999999999997,"5min":5.0149999999999997,"15min":5.0149999999999997},"last":0.39500000000000002}]}]},{"osd":4,"up_from":32,"seq":137438953583,"num_pgs":78,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":4830800,"kb_used_data":4525128,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":88914352,"statfs":{"total":95995035648,"available":91048296448,"internally_reserved":0,"allocated":4633731072,"data_stored":4633218015,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 19:17:03 2024","interfaces":[{"interface":"back","average":{"1min":0.81599999999999995,"5min":1.2509999999999999,"15min":0.77100000000000002},"min":{"1min":0.187,"5min":0.187,"15min":0.187},"max":{"1min":3.8559999999999999,"5min":32.533000000000001,"15min":32.533000000000001},"last":0.77100000000000002},{"interface":"front","average":{"1min":0.60799999999999998,"5min":1.2729999999999999,"15min":0.77100000000000002},"min":{"1min":0.223,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":2.4540000000000002,"5min":32.512999999999998,"15min":32.512999999999998},"last":0.48199999999999998}]},{"osd":1,"last update":"Tue Sep 10 19:17:23 2024","interfaces":[{"interface":"back","average":{"1min":1.296,"5min":0.76100000000000001,"15min":0.67200000000000004},"min":{"1min":0.27800000000000002,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":7.9770000000000003,"5min":7.9770000000000003,"15min":7.9770000000000003},"last":1.206},{"interface":"front","average":{"1min":0.89800000000000002,"5min":0.71999999999999997,"15min":0.68999999999999995},"min":{"1min":0.27700000000000002,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":3.8260000000000001,"5min":3.8260000000000001,"15min":3.8260000000000001},"last":0.80200000000000005}]},{"osd":2,"last update":"Tue Sep 10 19:17:03 2024","interfaces":[{"interface":"back","average":{"1min":0.98899999999999999,"5min":0.88700000000000001,"15min":0.65000000000000002},"min":{"1min":0.34899999999999998,"5min":0.214,"15min":0.214},"max":{"1min":3.8849999999999998,"5min":6.8869999999999996,"15min":6.8869999999999996},"last":0.82699999999999996},{"interface":"front","average":{"1min":0.63600000000000001,"5min":0.83599999999999997,"15min":0.629},"min":{"1min":0.27800000000000002,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":2.4849999999999999,"5min":7.085,"15min":7.085},"last":0.437}]},{"osd":3,"last update":"Tue Sep 10 19:17:03 2024","interfaces":[{"interface":"back","average":{"1min":0.64100000000000001,"5min":1.2290000000000001,"15min":0.75600000000000001},"min":{"1min":0.308,"5min":0.252,"15min":0.252},"max":{"1min":2.5089999999999999,"5min":35.451999999999998,"15min":35.451999999999998},"last":0.40999999999999998},{"interface":"front","average":{"1min":0.76900000000000002,"5min":1.2849999999999999,"15min":0.79100000000000004},"min":{"1min":0.26400000000000001,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":4.9669999999999996,"5min":32.880000000000003,"15min":32.880000000000003},"last":1.2569999999999999}]},{"osd":5,"last update":"Tue Sep 10 19:17:23 2024","interfaces":[{"interface":"back","average":{"1min":5.7119999999999997,"5min":1.577,"15min":0.78400000000000003},"min":{"1min":0.29599999999999999,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":67.379000000000005,"5min":67.379000000000005,"15min":67.379000000000005},"last":1.278},{"interface":"front","average":{"1min":4.9100000000000001,"5min":1.409,"15min":0.73899999999999999},"min":{"1min":0.27500000000000002,"5min":0.248,"15min":0.223},"max":{"1min":67.542000000000002,"5min":67.542000000000002,"15min":67.542000000000002},"last":0.73599999999999999}]},{"osd":6,"last update":"Tue Sep 10 19:16:53 2024","interfaces":[{"interface":"back","average":{"1min":0.79000000000000004,"5min":0.70399999999999996,"15min":0.51300000000000001},"min":{"1min":0.27400000000000002,"5min":0.253,"15min":0.253},"max":{"1min":3.766,"5min":3.964,"15min":3.964},"last":0.84799999999999998},{"interface":"front","average":{"1min":0.68300000000000005,"5min":0.63800000000000001,"15min":0.49199999999999999},"min":{"1min":0.30199999999999999,"5min":0.25,"15min":0.25},"max":{"1min":2.3460000000000001,"5min":3.0129999999999999,"15min":3.0129999999999999},"last":1.236}]},{"osd":7,"last update":"Tue Sep 10 19:17:08 2024","interfaces":[{"interface":"back","average":{"1min":1.113,"5min":1.05,"15min":0.69799999999999995},"min":{"1min":0.28100000000000003,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":4.7889999999999997,"5min":23.081,"15min":23.081},"last":1.3},{"interface":"front","average":{"1min":1.3680000000000001,"5min":1.093,"15min":0.69699999999999995},"min":{"1min":0.29099999999999998,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":6.3620000000000001,"5min":22.666,"15min":22.666},"last":0.86499999999999999}]}]},{"osd":3,"up_from":26,"seq":111669149811,"num_pgs":84,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":4232648,"kb_used_data":3922816,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":89512504,"statfs":{"total":95995035648,"available":91660804096,"internally_reserved":0,"allocated":4016963584,"data_stored":4016435707,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,0,1,2,2,1],"upper_bound":128},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 19:17:37 2024","interfaces":[{"interface":"back","average":{"1min":0.32100000000000001,"5min":1.2929999999999999,"15min":0.69999999999999996},"min":{"1min":0.17499999999999999,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":0.72199999999999998,"5min":17.518000000000001,"15min":17.518000000000001},"last":0.39500000000000002},{"interface":"front","average":{"1min":0.59499999999999997,"5min":1.617,"15min":0.81899999999999995},"min":{"1min":0.224,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":2.3359999999999999,"5min":17.609000000000002,"15min":17.609000000000002},"last":0.59699999999999998}]},{"osd":1,"last update":"Tue Sep 10 19:17:20 2024","interfaces":[{"interface":"back","average":{"1min":0.46800000000000003,"5min":0.52500000000000002,"15min":0.53500000000000003},"min":{"1min":0.22500000000000001,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":1.0509999999999999,"5min":2.6059999999999999,"15min":2.6059999999999999},"last":0.54400000000000004},{"interface":"front","average":{"1min":0.58199999999999996,"5min":1.371,"15min":1.5029999999999999},"min":{"1min":0.255,"5min":0.255,"15min":0.255},"max":{"1min":1.5289999999999999,"5min":17.454999999999998,"15min":17.454999999999998},"last":0.29199999999999998}]},{"osd":2,"last update":"Tue Sep 10 19:17:37 2024","interfaces":[{"interface":"back","average":{"1min":0.66400000000000003,"5min":1.0740000000000001,"15min":0.65000000000000002},"min":{"1min":0.25900000000000001,"5min":0.186,"15min":0.186},"max":{"1min":3.125,"5min":5.3029999999999999,"15min":5.3029999999999999},"last":0.40899999999999997},{"interface":"front","average":{"1min":0.55000000000000004,"5min":1.3080000000000001,"15min":0.71399999999999997},"min":{"1min":0.23599999999999999,"5min":0.23300000000000001,"15min":0.19800000000000001},"max":{"1min":2.266,"5min":17.806000000000001,"15min":17.806000000000001},"last":0.27100000000000002}]},{"osd":4,"last update":"Tue Sep 10 19:17:04 2024","interfaces":[{"interface":"back","average":{"1min":0.77700000000000002,"5min":1.321,"15min":0.80400000000000005},"min":{"1min":0.32300000000000001,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":3.0249999999999999,"5min":17.834,"15min":17.834},"last":1.4610000000000001},{"interface":"front","average":{"1min":0.872,"5min":1.675,"15min":0.92200000000000004},"min":{"1min":0.35899999999999999,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":3.2879999999999998,"5min":17.553000000000001,"15min":17.553000000000001},"last":1.49}]},{"osd":5,"last update":"Tue Sep 10 19:17:20 2024","interfaces":[{"interface":"back","average":{"1min":5.2830000000000004,"5min":2.0369999999999999,"15min":1.0549999999999999},"min":{"1min":0.33300000000000002,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":80.942999999999998,"5min":80.942999999999998,"15min":80.942999999999998},"last":1.6619999999999999},{"interface":"front","average":{"1min":5.6630000000000003,"5min":2.4860000000000002,"15min":1.202},"min":{"1min":0.377,"5min":0.22,"15min":0.22},"max":{"1min":81.489000000000004,"5min":81.489000000000004,"15min":81.489000000000004},"last":2.0899999999999999}]},{"osd":6,"last update":"Tue Sep 10 19:17:44 2024","interfaces":[{"interface":"back","average":{"1min":1.145,"5min":1.464,"15min":0.878},"min":{"1min":0.35599999999999998,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":4.1600000000000001,"5min":17.861000000000001,"15min":17.861000000000001},"last":0.66400000000000003},{"interface":"front","average":{"1min":1.0349999999999999,"5min":1.2090000000000001,"15min":0.78400000000000003},"min":{"1min":0.38400000000000001,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":2.2759999999999998,"5min":17.687000000000001,"15min":17.687000000000001},"last":0.498}]},{"osd":7,"last update":"Tue Sep 10 19:17:12 2024","interfaces":[{"interface":"back","average":{"1min":1.083,"5min":1.3089999999999999,"15min":0.82099999999999995},"min":{"1min":0.40699999999999997,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":4.1059999999999999,"5min":18.093,"15min":18.093},"last":0.442},{"interface":"front","average":{"1min":1.0980000000000001,"5min":1.4139999999999999,"15min":0.86099999999999999},"min":{"1min":0.35599999999999998,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":3.9929999999999999,"5min":17.780999999999999,"15min":17.780999999999999},"last":0.875}]}]},{"osd":2,"up_from":0,"seq":77309411448,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":0,"seq":592705486878,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":8,"seq":34359738498,"num_pgs":79,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3057472,"kb_used_data":2747640,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":90687680,"statfs":{"total":95995035648,"available":92864184320,"internally_reserved":0,"allocated":2813583360,"data_stored":2813063605,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Sep 10 19:17:24 2024","interfaces":[{"interface":"back","average":{"1min":0.38,"5min":0.371,"15min":0.36899999999999999},"min":{"1min":0.217,"5min":0.19,"15min":0.19},"max":{"1min":1.698,"5min":1.698,"15min":1.698},"last":0.253},{"interface":"front","average":{"1min":0.33300000000000002,"5min":0.40600000000000003,"15min":0.41799999999999998},"min":{"1min":0.249,"5min":0.249,"15min":0.249},"max":{"1min":0.72799999999999998,"5min":1.0900000000000001,"15min":1.0900000000000001},"last":0.193}]},{"osd":2,"last update":"Tue Sep 10 19:17:24 2024","interfaces":[{"interface":"back","average":{"1min":0.41199999999999998,"5min":1.028,"15min":0.59499999999999997},"min":{"1min":0.23899999999999999,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":1.339,"5min":6.4640000000000004,"15min":6.4640000000000004},"last":0.22700000000000001},{"interface":"front","average":{"1min":0.42699999999999999,"5min":1.0129999999999999,"15min":0.58199999999999996},"min":{"1min":0.252,"5min":0.193,"15min":0.187},"max":{"1min":0.84499999999999997,"5min":9.9399999999999995,"15min":9.9399999999999995},"last":0.26700000000000002}]},{"osd":3,"last update":"Tue Sep 10 19:17:44 2024","interfaces":[{"interface":"back","average":{"1min":0.48199999999999998,"5min":1.0820000000000001,"15min":0.64200000000000002},"min":{"1min":0.30099999999999999,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.80300000000000005,"5min":6.5590000000000002,"15min":6.5590000000000002},"last":0.39900000000000002},{"interface":"front","average":{"1min":0.41199999999999998,"5min":1.2470000000000001,"15min":0.68899999999999995},"min":{"1min":0.26200000000000001,"5min":0.26200000000000001,"15min":0.23000000000000001},"max":{"1min":0.81599999999999995,"5min":7.2140000000000004,"15min":7.2140000000000004},"last":0.28399999999999997}]},{"osd":4,"last update":"Tue Sep 10 19:16:59 2024","interfaces":[{"interface":"back","average":{"1min":0.58599999999999997,"5min":0.80100000000000005,"15min":0.63800000000000001},"min":{"1min":0.29099999999999998,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":1.391,"5min":5.8499999999999996,"15min":5.8499999999999996},"last":0.30599999999999999},{"interface":"front","average":{"1min":0.873,"5min":0.88300000000000001,"15min":0.66600000000000004},"min":{"1min":0.29299999999999998,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":5.2640000000000002,"5min":5.2640000000000002,"15min":5.2640000000000002},"last":0.86199999999999999}]},{"osd":5,"last update":"Tue Sep 10 19:17:24 2024","interfaces":[{"interface":"back","average":{"1min":1.3580000000000001,"5min":1.1160000000000001,"15min":0.754},"min":{"1min":0.32500000000000001,"5min":0.29499999999999998,"15min":0.28199999999999997},"max":{"1min":2.6240000000000001,"5min":5.3860000000000001,"15min":5.3860000000000001},"last":0.46300000000000002},{"interface":"front","average":{"1min":0.90800000000000003,"5min":0.99199999999999999,"15min":0.70799999999999996},"min":{"1min":0.36699999999999999,"5min":0.253,"15min":0.253},"max":{"1min":4.165,"5min":5.9749999999999996,"15min":5.9749999999999996},"last":0.48799999999999999}]},{"osd":6,"last update":"Tue Sep 10 19:17:44 2024","interfaces":[{"interface":"back","average":{"1min":1.028,"5min":1.0720000000000001,"15min":0.73299999999999998},"min":{"1min":0.35799999999999998,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":2.101,"5min":5.8730000000000002,"15min":5.8730000000000002},"last":0.505},{"interface":"front","average":{"1min":0.81399999999999995,"5min":1.03,"15min":0.73299999999999998},"min":{"1min":0.34399999999999997,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":1.8560000000000001,"5min":5.2850000000000001,"15min":5.2850000000000001},"last":0.65300000000000002}]},{"osd":7,"last update":"Tue Sep 10 19:17:09 2024","interfaces":[{"interface":"back","average":{"1min":0.95199999999999996,"5min":1.034,"15min":0.72699999999999998},"min":{"1min":0.40600000000000003,"5min":0.318,"15min":0.318},"max":{"1min":5.1479999999999997,"5min":5.1479999999999997,"15min":5.1479999999999997},"last":1.0669999999999999},{"interface":"front","average":{"1min":1.2230000000000001,"5min":1.056,"15min":0.72299999999999998},"min":{"1min":0.36699999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":4.9900000000000002,"5min":6.0469999999999997,"15min":6.0469999999999997},"last":0.51600000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":966,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1236,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1236,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":362,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":1760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":2118,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1018,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":2118,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1540,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1942,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":2162,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":2840526848,"data_stored":2840526848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4045996032,"data_stored":4045996032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4664983552,"data_stored":4664983552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4043964416,"data_stored":4043964416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":3372548096,"data_stored":3372548096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":3789553664,"data_stored":3789553664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T19:17:52.210 INFO:tasks.thrashosds.thrasher:Setting 2.13 to [2, 0, 5] 2024-09-10T19:17:52.210 INFO:tasks.thrashosds.thrasher:cmd ['osd', 'pg-upmap', '2.13', '2', '0', '5'] 2024-09-10T19:17:52.210 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph osd pg-upmap 2.13 2 0 5 2024-09-10T19:17:52.211 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:17:52.246 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:17:52.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:17:52.248 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:17:52.349 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:17:52.379 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:17:52.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:17:52.380 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:17:52.481 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:17:52.516 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:17:52.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:17:52.517 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:17:52.564 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph osd unset noscrub 2024-09-10T19:17:52.618 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:17:52.657 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:17:52.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:17:52.659 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:17:52.759 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:17:52.795 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:17:52.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:17:52.796 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:17:52.798 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:17:52.830 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: 88 16 117991 117975 83.7793 80.75 0.00957072 0.0119308 2024-09-10T19:17:52.898 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:17:52.925 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:17:52.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:17:52.926 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:17:53.027 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:17:53.065 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:17:53.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:17:53.066 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:17:53.167 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:17:53.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:17:52 smithi183 ceph-mon[29736]: from='client.15174 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-10T19:17:53.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:17:52 smithi183 ceph-mon[29736]: pgmap v334: 158 pgs: 1 active+recovering+remapped, 1 peering, 1 active+recovering, 5 active+remapped+backfill_toofull, 150 active+clean; 7.0 GiB data, 24 GiB used, 513 GiB / 536 GiB avail; 853 B/s rd, 50 MiB/s wr, 798 op/s; 34646/343161 objects misplaced (10.096%); 16 MiB/s, 263 objects/s recovering 2024-09-10T19:17:53.181 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:17:52 smithi183 ceph-mon[41562]: from='client.15174 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-10T19:17:53.181 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:17:52 smithi183 ceph-mon[41562]: pgmap v334: 158 pgs: 1 active+recovering+remapped, 1 peering, 1 active+recovering, 5 active+remapped+backfill_toofull, 150 active+clean; 7.0 GiB data, 24 GiB used, 513 GiB / 536 GiB avail; 853 B/s rd, 50 MiB/s wr, 798 op/s; 34646/343161 objects misplaced (10.096%); 16 MiB/s, 263 objects/s recovering 2024-09-10T19:17:53.194 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:17:53.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:17:53.195 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:17:53.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:17:52 smithi191 ceph-mon[32884]: from='client.15174 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-10T19:17:53.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:17:52 smithi191 ceph-mon[32884]: pgmap v334: 158 pgs: 1 active+recovering+remapped, 1 peering, 1 active+recovering, 5 active+remapped+backfill_toofull, 150 active+clean; 7.0 GiB data, 24 GiB used, 513 GiB / 536 GiB avail; 853 B/s rd, 50 MiB/s wr, 798 op/s; 34646/343161 objects misplaced (10.096%); 16 MiB/s, 263 objects/s recovering 2024-09-10T19:17:53.296 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:17:53.322 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:17:53.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:17:53.323 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:17:53.424 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:17:53.450 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:17:53.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:17:53.451 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:17:53.552 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:17:53.600 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:17:53.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:17:53.601 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:17:53.702 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:17:53.736 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:17:53.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:17:53.737 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:17:53.830 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: 89 16 119266 119250 83.7333 79.6875 0.0037812 0.0119365 2024-09-10T19:17:53.838 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:17:53.864 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:17:53.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:17:53.865 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:17:53.966 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:17:53.990 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:17:53.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:17:53.991 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:17:54.035 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:17:53 smithi183 ceph-mon[29736]: purged_snaps scrub starts 2024-09-10T19:17:54.035 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:17:53 smithi183 ceph-mon[29736]: purged_snaps scrub ok 2024-09-10T19:17:54.035 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:17:53 smithi183 ceph-mon[41562]: purged_snaps scrub starts 2024-09-10T19:17:54.036 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:17:53 smithi183 ceph-mon[41562]: purged_snaps scrub ok 2024-09-10T19:17:54.092 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:17:54.127 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:17:54.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:17:54.128 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:17:54.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:17:53 smithi191 ceph-mon[32884]: purged_snaps scrub starts 2024-09-10T19:17:54.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:17:53 smithi191 ceph-mon[32884]: purged_snaps scrub ok 2024-09-10T19:17:54.229 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:17:54.258 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:17:54.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:17:54.259 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:17:54.360 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:17:54.385 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:17:54.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:17:54.386 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:17:54.487 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:17:54.512 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:17:54.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:17:54.513 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:17:54.614 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:17:54.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:17:54.644 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:17:54.644 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:17:54.745 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:17:54.767 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:17:54.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:17:54.767 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:17:54.830 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: 90 5 120280 120275 83.5146 64.0625 0.20915 0.0119656 2024-09-10T19:17:54.868 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:17:54.890 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:17:54.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:17:54.891 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:17:54.991 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:17:55.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:17:55.014 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:17:55.014 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:17:55.115 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:17:55.136 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:17:55.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:17:55.137 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:17:55.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:17:54 smithi183 ceph-mon[29736]: pgmap v335: 158 pgs: 1 active+recovering+remapped, 1 peering, 1 active+recovering, 5 active+remapped+backfill_toofull, 150 active+clean; 7.0 GiB data, 24 GiB used, 513 GiB / 536 GiB avail; 725 B/s rd, 42 MiB/s wr, 678 op/s; 34646/343161 objects misplaced (10.096%); 14 MiB/s, 223 objects/s recovering 2024-09-10T19:17:55.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:17:54 smithi183 ceph-mon[29736]: osdmap e191: 8 total, 8 up, 6 in 2024-09-10T19:17:55.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:17:54 smithi183 ceph-mon[29736]: osdmap e192: 8 total, 8 up, 6 in 2024-09-10T19:17:55.181 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:17:54 smithi183 ceph-mon[41562]: pgmap v335: 158 pgs: 1 active+recovering+remapped, 1 peering, 1 active+recovering, 5 active+remapped+backfill_toofull, 150 active+clean; 7.0 GiB data, 24 GiB used, 513 GiB / 536 GiB avail; 725 B/s rd, 42 MiB/s wr, 678 op/s; 34646/343161 objects misplaced (10.096%); 14 MiB/s, 223 objects/s recovering 2024-09-10T19:17:55.181 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:17:54 smithi183 ceph-mon[41562]: osdmap e191: 8 total, 8 up, 6 in 2024-09-10T19:17:55.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:17:54 smithi183 ceph-mon[41562]: osdmap e192: 8 total, 8 up, 6 in 2024-09-10T19:17:55.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:17:54 smithi191 ceph-mon[32884]: pgmap v335: 158 pgs: 1 active+recovering+remapped, 1 peering, 1 active+recovering, 5 active+remapped+backfill_toofull, 150 active+clean; 7.0 GiB data, 24 GiB used, 513 GiB / 536 GiB avail; 725 B/s rd, 42 MiB/s wr, 678 op/s; 34646/343161 objects misplaced (10.096%); 14 MiB/s, 223 objects/s recovering 2024-09-10T19:17:55.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:17:54 smithi191 ceph-mon[32884]: osdmap e191: 8 total, 8 up, 6 in 2024-09-10T19:17:55.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:17:54 smithi191 ceph-mon[32884]: osdmap e192: 8 total, 8 up, 6 in 2024-09-10T19:17:55.238 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:17:55.260 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:17:55.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:17:55.261 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:17:55.362 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:17:55.383 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:17:55.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:17:55.384 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:17:55.485 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:17:55.506 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:17:55.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:17:55.507 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:17:55.608 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:17:55.629 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:17:55.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:17:55.630 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:17:55.730 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:17:55.751 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:17:55.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:17:55.752 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:17:55.831 INFO:tasks.radosbench.radosbench.0.smithi183.stdout:Total time run: 90.0284 2024-09-10T19:17:55.831 INFO:tasks.radosbench.radosbench.0.smithi183.stdout:Total writes made: 120280 2024-09-10T19:17:55.831 INFO:tasks.radosbench.radosbench.0.smithi183.stdout:Write size: 65536 2024-09-10T19:17:55.831 INFO:tasks.radosbench.radosbench.0.smithi183.stdout:Object size: 65536 2024-09-10T19:17:55.831 INFO:tasks.radosbench.radosbench.0.smithi183.stdout:Bandwidth (MB/sec): 83.5014 2024-09-10T19:17:55.831 INFO:tasks.radosbench.radosbench.0.smithi183.stdout:Stddev Bandwidth: 41.6932 2024-09-10T19:17:55.831 INFO:tasks.radosbench.radosbench.0.smithi183.stdout:Max bandwidth (MB/sec): 241.438 2024-09-10T19:17:55.831 INFO:tasks.radosbench.radosbench.0.smithi183.stdout:Min bandwidth (MB/sec): 0 2024-09-10T19:17:55.831 INFO:tasks.radosbench.radosbench.0.smithi183.stdout:Average IOPS: 1336 2024-09-10T19:17:55.832 INFO:tasks.radosbench.radosbench.0.smithi183.stdout:Stddev IOPS: 667.099 2024-09-10T19:17:55.832 INFO:tasks.radosbench.radosbench.0.smithi183.stdout:Max IOPS: 3863 2024-09-10T19:17:55.832 INFO:tasks.radosbench.radosbench.0.smithi183.stdout:Min IOPS: 0 2024-09-10T19:17:55.832 INFO:tasks.radosbench.radosbench.0.smithi183.stdout:Average Latency(s): 0.01197 2024-09-10T19:17:55.832 INFO:tasks.radosbench.radosbench.0.smithi183.stdout:Stddev Latency(s): 0.0482068 2024-09-10T19:17:55.832 INFO:tasks.radosbench.radosbench.0.smithi183.stdout:Max latency(s): 3.03886 2024-09-10T19:17:55.832 INFO:tasks.radosbench.radosbench.0.smithi183.stdout:Min latency(s): 0.0011643 2024-09-10T19:17:55.834 INFO:tasks.radosbench.radosbench.0.smithi183.stdout:Cleaning up (deleting benchmark objects) 2024-09-10T19:17:55.853 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:17:55.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:17:55.875 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:17:55.875 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:17:55.976 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:17:56.022 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:17:56.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:17:56.023 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:17:56.123 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:17:56.157 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:17:56.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:17:56.158 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:17:56.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:17:55 smithi183 ceph-mon[29736]: purged_snaps scrub starts 2024-09-10T19:17:56.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:17:55 smithi183 ceph-mon[29736]: purged_snaps scrub ok 2024-09-10T19:17:56.181 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:17:55 smithi183 ceph-mon[41562]: purged_snaps scrub starts 2024-09-10T19:17:56.181 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:17:55 smithi183 ceph-mon[41562]: purged_snaps scrub ok 2024-09-10T19:17:56.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:17:55 smithi191 ceph-mon[32884]: purged_snaps scrub starts 2024-09-10T19:17:56.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:17:55 smithi191 ceph-mon[32884]: purged_snaps scrub ok 2024-09-10T19:17:56.259 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:17:56.301 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:17:56.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:17:56.302 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:17:56.403 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:17:56.435 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:17:56.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:17:56.435 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:17:56.536 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:17:56.561 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:17:56.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:17:56.562 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:17:56.663 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:17:56.688 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:17:56.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:17:56.690 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:17:56.790 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:17:56.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:17:56.819 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:17:56.819 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:17:56.919 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:17:56.944 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:17:56.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:17:56.945 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:17:57.046 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:17:57.070 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:17:57.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:17:57.071 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:17:57.116 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:17:56 smithi191 ceph-mon[32884]: pgmap v338: 158 pgs: 1 active+recovering+remapped, 1 active+recovering, 5 active+remapped+backfill_toofull, 151 active+clean; 7.3 GiB data, 25 GiB used, 512 GiB / 536 GiB avail; 1.7 KiB/s rd, 113 MiB/s wr, 1.81k op/s; 36203/361422 objects misplaced (10.017%); 17 MiB/s, 275 objects/s recovering 2024-09-10T19:17:57.172 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:17:57.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:17:56 smithi183 ceph-mon[29736]: pgmap v338: 158 pgs: 1 active+recovering+remapped, 1 active+recovering, 5 active+remapped+backfill_toofull, 151 active+clean; 7.3 GiB data, 25 GiB used, 512 GiB / 536 GiB avail; 1.7 KiB/s rd, 113 MiB/s wr, 1.81k op/s; 36203/361422 objects misplaced (10.017%); 17 MiB/s, 275 objects/s recovering 2024-09-10T19:17:57.181 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:17:56 smithi183 ceph-mon[41562]: pgmap v338: 158 pgs: 1 active+recovering+remapped, 1 active+recovering, 5 active+remapped+backfill_toofull, 151 active+clean; 7.3 GiB data, 25 GiB used, 512 GiB / 536 GiB avail; 1.7 KiB/s rd, 113 MiB/s wr, 1.81k op/s; 36203/361422 objects misplaced (10.017%); 17 MiB/s, 275 objects/s recovering 2024-09-10T19:17:57.197 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:17:57.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:17:57.198 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:17:57.298 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:17:57.325 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:17:57.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:17:57.326 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:17:57.427 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:17:57.453 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:17:57.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:17:57.454 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:17:57.555 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:17:57.588 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:17:57.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:17:57.589 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:17:57.690 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:17:57.715 INFO:journalctl@ceph.mgr.x.smithi191.stdout:Sep 10 19:17:57 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-x[34967]: ::ffff:172.21.15.191 - - [10/Sep/2024:19:17:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T19:17:57.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:17:57.727 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:17:57.727 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:17:57.828 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:17:57.853 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:17:57.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:17:57.854 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:17:57.955 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:17:57.995 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:17:57.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:17:57.996 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:17:58.097 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:17:58.131 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:17:58.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:17:58.132 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:17:58.233 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:17:58.265 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:17:58.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:17:58.266 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:17:58.367 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:17:58.389 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:17:58.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:17:58.390 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:17:58.491 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:17:58.517 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:17:58.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:17:58.518 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:17:58.619 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:17:58.640 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:17:58.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:17:58.641 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:17:58.742 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:17:58.782 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:17:58.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:17:58.783 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:17:58.884 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:17:58.906 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:17:58.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:17:58.907 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:17:59.008 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:17:59.043 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:17:59.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:17:59.044 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:17:59.145 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:17:59.173 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:17:59.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:17:59.174 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:17:59.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:17:58 smithi183 ceph-mon[29736]: pgmap v339: 158 pgs: 1 active+recovering+remapped, 1 active+recovering, 5 active+remapped+backfill_toofull, 151 active+clean; 7.3 GiB data, 25 GiB used, 512 GiB / 536 GiB avail; 1.3 KiB/s rd, 82 MiB/s wr, 1.31k op/s; 36203/361422 objects misplaced (10.017%); 4.2 MiB/s, 66 objects/s recovering 2024-09-10T19:17:59.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:17:58 smithi183 ceph-mon[29736]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:17:59.181 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:17:58 smithi183 ceph-mon[41562]: pgmap v339: 158 pgs: 1 active+recovering+remapped, 1 active+recovering, 5 active+remapped+backfill_toofull, 151 active+clean; 7.3 GiB data, 25 GiB used, 512 GiB / 536 GiB avail; 1.3 KiB/s rd, 82 MiB/s wr, 1.31k op/s; 36203/361422 objects misplaced (10.017%); 4.2 MiB/s, 66 objects/s recovering 2024-09-10T19:17:59.181 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:17:58 smithi183 ceph-mon[41562]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:17:59.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:17:58 smithi191 ceph-mon[32884]: pgmap v339: 158 pgs: 1 active+recovering+remapped, 1 active+recovering, 5 active+remapped+backfill_toofull, 151 active+clean; 7.3 GiB data, 25 GiB used, 512 GiB / 536 GiB avail; 1.3 KiB/s rd, 82 MiB/s wr, 1.31k op/s; 36203/361422 objects misplaced (10.017%); 4.2 MiB/s, 66 objects/s recovering 2024-09-10T19:17:59.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:17:58 smithi191 ceph-mon[32884]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:17:59.275 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:17:59.303 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:17:59.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:17:59.304 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:17:59.405 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:17:59.431 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:17:59.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:17:59.432 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:17:59.533 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:17:59.558 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:17:59.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:17:59.559 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:17:59.660 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:17:59.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:17:59.685 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:17:59.685 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:17:59.716 INFO:journalctl@ceph.iscsi.iscsi.a.smithi191.stdout:Sep 10 19:17:59 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-iscsi-iscsi-a[61816]: debug there is no tcmu-runner data available 2024-09-10T19:17:59.786 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:17:59.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:17:59.816 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:17:59.816 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:17:59.917 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:17:59.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:17:59.939 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:17:59.939 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:18:00.040 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:18:00.065 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:18:00.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:18:00.066 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:18:00.166 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:18:00.203 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:18:00.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:18:00.204 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:18:00.305 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:18:00.329 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:18:00.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:18:00.330 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:18:00.431 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:18:00.465 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:18:00.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:18:00.466 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:18:00.567 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:18:00.591 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:18:00.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:18:00.592 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:18:00.693 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:18:00.715 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:18:00.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:18:00.716 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:18:00.817 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:18:00.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:18:00.843 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:18:00.843 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:18:00.944 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:18:00.978 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:18:00.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:18:00.979 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:18:01.080 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:18:01.115 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:18:01.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:18:01.116 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:18:01.144 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:18:00 smithi183 ceph-mon[29736]: from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:18:01.144 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:18:00 smithi183 ceph-mon[29736]: pgmap v340: 158 pgs: 1 active+recovering+remapped, 1 active+recovering, 5 active+remapped+backfill_toofull, 151 active+clean; 7.3 GiB data, 25 GiB used, 512 GiB / 536 GiB avail; 639 B/s rd, 48 MiB/s wr, 761 op/s; 36203/361422 objects misplaced (10.017%); 568 KiB/s, 8 objects/s recovering 2024-09-10T19:18:01.144 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:18:00 smithi183 ceph-mon[41562]: from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:18:01.144 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:18:00 smithi183 ceph-mon[41562]: pgmap v340: 158 pgs: 1 active+recovering+remapped, 1 active+recovering, 5 active+remapped+backfill_toofull, 151 active+clean; 7.3 GiB data, 25 GiB used, 512 GiB / 536 GiB avail; 639 B/s rd, 48 MiB/s wr, 761 op/s; 36203/361422 objects misplaced (10.017%); 568 KiB/s, 8 objects/s recovering 2024-09-10T19:18:01.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:18:00 smithi191 ceph-mon[32884]: from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:18:01.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:18:00 smithi191 ceph-mon[32884]: pgmap v340: 158 pgs: 1 active+recovering+remapped, 1 active+recovering, 5 active+remapped+backfill_toofull, 151 active+clean; 7.3 GiB data, 25 GiB used, 512 GiB / 536 GiB avail; 639 B/s rd, 48 MiB/s wr, 761 op/s; 36203/361422 objects misplaced (10.017%); 568 KiB/s, 8 objects/s recovering 2024-09-10T19:18:01.217 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:18:01.246 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:18:01.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:18:01.247 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:18:01.347 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:18:01.372 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:18:01.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:18:01.373 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:18:01.431 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Sep 10 19:18:01 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-y[29989]: ::ffff:172.21.15.191 - - [10/Sep/2024:19:18:01] "GET /metrics HTTP/1.1" 200 224284 "" "Prometheus/2.43.0" 2024-09-10T19:18:01.473 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:18:01.494 INFO:teuthology.orchestra.run.smithi183.stdout:osd.0: {} 2024-09-10T19:18:01.494 INFO:teuthology.orchestra.run.smithi183.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-10T19:18:01.509 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:18:01.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:18:01.510 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:18:01.517 INFO:teuthology.orchestra.run.smithi183.stdout:osd.1: {} 2024-09-10T19:18:01.517 INFO:teuthology.orchestra.run.smithi183.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-10T19:18:01.537 INFO:teuthology.orchestra.run.smithi183.stdout:osd.2: {} 2024-09-10T19:18:01.538 INFO:teuthology.orchestra.run.smithi183.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-10T19:18:01.546 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:18:01.565 INFO:teuthology.orchestra.run.smithi183.stdout:osd.3: {} 2024-09-10T19:18:01.565 INFO:teuthology.orchestra.run.smithi183.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-10T19:18:01.589 INFO:teuthology.orchestra.run.smithi183.stdout:osd.4: {} 2024-09-10T19:18:01.589 INFO:teuthology.orchestra.run.smithi183.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-10T19:18:01.608 INFO:teuthology.orchestra.run.smithi183.stdout:osd.5: {} 2024-09-10T19:18:01.608 INFO:teuthology.orchestra.run.smithi183.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-10T19:18:01.611 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:18:01.632 INFO:teuthology.orchestra.run.smithi183.stdout:osd.6: {} 2024-09-10T19:18:01.632 INFO:teuthology.orchestra.run.smithi183.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-10T19:18:01.639 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:18:01.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:18:01.640 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:18:01.657 INFO:teuthology.orchestra.run.smithi183.stdout:osd.7: {} 2024-09-10T19:18:01.657 INFO:teuthology.orchestra.run.smithi183.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-10T19:18:01.741 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:18:01.768 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:18:01.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:18:01.769 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:18:01.870 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:18:01.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:18:01.905 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:18:01.906 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:18:02.006 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:18:02.031 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:18:02.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:18:02.032 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:18:02.059 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:18:01 smithi183 ceph-mon[29736]: from='client.? 172.21.15.183:0/2810422841' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T19:18:02.059 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:18:01 smithi183 ceph-mon[41562]: from='client.? 172.21.15.183:0/2810422841' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T19:18:02.133 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:18:02.160 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:18:02.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:18:02.161 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:18:02.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:18:01 smithi191 ceph-mon[32884]: from='client.? 172.21.15.183:0/2810422841' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T19:18:02.262 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:18:02.296 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:18:02.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:18:02.297 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:18:02.310 INFO:teuthology.orchestra.run.smithi183.stdout:ERROR: (22) Invalid argument 2024-09-10T19:18:02.311 INFO:teuthology.orchestra.run.smithi183.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-10T19:18:02.397 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:18:02.424 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:18:02.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:18:02.425 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:18:02.526 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:18:02.564 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:18:02.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:18:02.565 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:18:02.666 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:18:02.701 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:18:02.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:18:02.702 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:18:02.803 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:18:02.838 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:18:02.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:18:02.839 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:18:02.931 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:18:02 smithi183 ceph-mon[29736]: pgmap v341: 158 pgs: 1 active+recovering+remapped, 1 active+recovering, 5 active+remapped+backfill_toofull, 151 active+clean; 7.0 GiB data, 24 GiB used, 512 GiB / 536 GiB avail; 1.4 KiB/s rd, 48 MiB/s wr, 1.36k op/s; 34513/346947 objects misplaced (9.948%); 2.9 MiB/s, 46 objects/s recovering 2024-09-10T19:18:02.931 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:18:02 smithi183 ceph-mon[41562]: pgmap v341: 158 pgs: 1 active+recovering+remapped, 1 active+recovering, 5 active+remapped+backfill_toofull, 151 active+clean; 7.0 GiB data, 24 GiB used, 512 GiB / 536 GiB avail; 1.4 KiB/s rd, 48 MiB/s wr, 1.36k op/s; 34513/346947 objects misplaced (9.948%); 2.9 MiB/s, 46 objects/s recovering 2024-09-10T19:18:02.940 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:18:02.965 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:18:02.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:18:02.966 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:18:03.067 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:18:03.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:18:03.097 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:18:03.097 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:18:03.103 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph daemon osd.0 dump_historic_ops 2024-09-10T19:18:03.198 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:18:03.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:18:02 smithi191 ceph-mon[32884]: pgmap v341: 158 pgs: 1 active+recovering+remapped, 1 active+recovering, 5 active+remapped+backfill_toofull, 151 active+clean; 7.0 GiB data, 24 GiB used, 512 GiB / 536 GiB avail; 1.4 KiB/s rd, 48 MiB/s wr, 1.36k op/s; 34513/346947 objects misplaced (9.948%); 2.9 MiB/s, 46 objects/s recovering 2024-09-10T19:18:03.234 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:18:03.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:18:03.235 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:18:03.336 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:18:03.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:18:03.362 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:18:03.362 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:18:03.462 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:18:03.489 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:18:03.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:18:03.490 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:18:03.591 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:18:03.615 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:18:03.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:18:03.616 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:18:03.702 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:18:03.716 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:18:03.742 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:18:03.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:18:03.743 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:18:03.844 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:18:03.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:18:03.870 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:18:03.870 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:18:03.971 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:18:04.009 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:18:04.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:18:04.010 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:18:04.110 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:18:04.147 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:18:04.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:18:04.149 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:18:04.250 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:18:04.282 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:18:04.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:18:04.283 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:18:04.383 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:18:04.420 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:18:04.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:18:04.421 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:18:04.522 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:18:04.548 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:18:04.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:18:04.549 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:18:04.650 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:18:04.676 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:18:04.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:18:04.677 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:18:04.777 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:18:04.809 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:18:04.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:18:04.810 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:18:04.911 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:18:04.950 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:18:04.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:18:04.951 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:18:05.052 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:18:05.076 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:18:05.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:18:05.077 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:18:05.178 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:18:05.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:18:04 smithi183 ceph-mon[29736]: pgmap v342: 158 pgs: 1 active+recovering+remapped, 1 active+recovering, 5 active+remapped+backfill_toofull, 151 active+clean; 7.0 GiB data, 24 GiB used, 512 GiB / 536 GiB avail; 1.1 KiB/s rd, 38 MiB/s wr, 1.10k op/s; 34513/346947 objects misplaced (9.948%); 2.4 MiB/s, 37 objects/s recovering 2024-09-10T19:18:05.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:18:04 smithi183 ceph-mon[29736]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T19:18:05.181 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:18:04 smithi183 ceph-mon[41562]: pgmap v342: 158 pgs: 1 active+recovering+remapped, 1 active+recovering, 5 active+remapped+backfill_toofull, 151 active+clean; 7.0 GiB data, 24 GiB used, 512 GiB / 536 GiB avail; 1.1 KiB/s rd, 38 MiB/s wr, 1.10k op/s; 34513/346947 objects misplaced (9.948%); 2.4 MiB/s, 37 objects/s recovering 2024-09-10T19:18:05.181 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:18:04 smithi183 ceph-mon[41562]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T19:18:05.200 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:18:05.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:18:05.201 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:18:05.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:18:04 smithi191 ceph-mon[32884]: pgmap v342: 158 pgs: 1 active+recovering+remapped, 1 active+recovering, 5 active+remapped+backfill_toofull, 151 active+clean; 7.0 GiB data, 24 GiB used, 512 GiB / 536 GiB avail; 1.1 KiB/s rd, 38 MiB/s wr, 1.10k op/s; 34513/346947 objects misplaced (9.948%); 2.4 MiB/s, 37 objects/s recovering 2024-09-10T19:18:05.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:18:04 smithi191 ceph-mon[32884]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T19:18:05.302 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:18:05.326 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:18:05.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:18:05.327 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:18:05.428 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:18:05.452 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:18:05.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:18:05.453 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:18:05.554 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:18:05.581 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:18:05.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:18:05.582 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:18:05.683 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:18:05.718 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:18:05.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:18:05.719 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:18:05.820 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:18:05.853 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:18:05.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:18:05.854 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:18:05.955 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:18:05.982 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:18:05.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:18:05.983 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:18:06.076 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:18:05 smithi183 ceph-mon[29736]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 4 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-09-10T19:18:06.076 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:18:05 smithi183 ceph-mon[41562]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 4 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-09-10T19:18:06.084 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:18:06.125 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:18:06.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:18:06.126 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:18:06.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:18:05 smithi191 ceph-mon[32884]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 4 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-09-10T19:18:06.222 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-10T19:18:06.227 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:18:06.293 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:18:06.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:18:06.294 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:18:06.395 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:18:06.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:18:06.439 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:18:06.439 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:18:06.540 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:18:06.576 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:18:06.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:18:06.577 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:18:06.678 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:18:06.719 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:18:06.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:18:06.720 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:18:06.821 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:18:06.847 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:18:06.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:18:06.849 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:18:06.909 INFO:teuthology.orchestra.run.smithi183.stderr:set 2.13 pg_upmap mapping to [2,0,5] 2024-09-10T19:18:06.949 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:18:06.973 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:18:06.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:18:06.974 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:18:07.075 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:18:07.107 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:18:07.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:18:07.109 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:18:07.181 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:18:06 smithi183 ceph-mon[41562]: pgmap v343: 158 pgs: 1 active+recovering+remapped, 1 active+remapped+backfilling, 1 active+recovering, 4 active+remapped+backfill_toofull, 151 active+clean; 6.7 GiB data, 23 GiB used, 513 GiB / 536 GiB avail; 1.5 KiB/s rd, 31 MiB/s wr, 1.48k op/s; 31310/329202 objects misplaced (9.511%); 6.6 MiB/s, 105 objects/s recovering 2024-09-10T19:18:07.181 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:18:06 smithi183 ceph-mon[41562]: from='client.? 172.21.15.183:0/3266797798' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "2.13", "id": [2, 0, 5]}]: dispatch 2024-09-10T19:18:07.181 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:18:06 smithi183 ceph-mon[41562]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "2.13", "id": [2, 0, 5]}]: dispatch 2024-09-10T19:18:07.181 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:18:06 smithi183 ceph-mon[41562]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T19:18:07.181 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:18:06 smithi183 ceph-mon[41562]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T19:18:07.181 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:18:06 smithi183 ceph-mon[41562]: from='mgr.14556 ' entity='mgr.y' 2024-09-10T19:18:07.182 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:18:06 smithi183 ceph-mon[29736]: pgmap v343: 158 pgs: 1 active+recovering+remapped, 1 active+remapped+backfilling, 1 active+recovering, 4 active+remapped+backfill_toofull, 151 active+clean; 6.7 GiB data, 23 GiB used, 513 GiB / 536 GiB avail; 1.5 KiB/s rd, 31 MiB/s wr, 1.48k op/s; 31310/329202 objects misplaced (9.511%); 6.6 MiB/s, 105 objects/s recovering 2024-09-10T19:18:07.182 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:18:06 smithi183 ceph-mon[29736]: from='client.? 172.21.15.183:0/3266797798' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "2.13", "id": [2, 0, 5]}]: dispatch 2024-09-10T19:18:07.182 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:18:06 smithi183 ceph-mon[29736]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "2.13", "id": [2, 0, 5]}]: dispatch 2024-09-10T19:18:07.182 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:18:06 smithi183 ceph-mon[29736]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T19:18:07.182 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:18:06 smithi183 ceph-mon[29736]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T19:18:07.182 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:18:06 smithi183 ceph-mon[29736]: from='mgr.14556 ' entity='mgr.y' 2024-09-10T19:18:07.209 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:18:07.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:18:06 smithi191 ceph-mon[32884]: pgmap v343: 158 pgs: 1 active+recovering+remapped, 1 active+remapped+backfilling, 1 active+recovering, 4 active+remapped+backfill_toofull, 151 active+clean; 6.7 GiB data, 23 GiB used, 513 GiB / 536 GiB avail; 1.5 KiB/s rd, 31 MiB/s wr, 1.48k op/s; 31310/329202 objects misplaced (9.511%); 6.6 MiB/s, 105 objects/s recovering 2024-09-10T19:18:07.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:18:06 smithi191 ceph-mon[32884]: from='client.? 172.21.15.183:0/3266797798' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "2.13", "id": [2, 0, 5]}]: dispatch 2024-09-10T19:18:07.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:18:06 smithi191 ceph-mon[32884]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "2.13", "id": [2, 0, 5]}]: dispatch 2024-09-10T19:18:07.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:18:06 smithi191 ceph-mon[32884]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T19:18:07.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:18:06 smithi191 ceph-mon[32884]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T19:18:07.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:18:06 smithi191 ceph-mon[32884]: from='mgr.14556 ' entity='mgr.y' 2024-09-10T19:18:07.244 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:18:07.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:18:07.245 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:18:07.346 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:18:07.380 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:18:07.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:18:07.381 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:18:07.482 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:18:07.509 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:18:07.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:18:07.510 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:18:07.611 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:18:07.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:18:07.645 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:18:07.645 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:18:07.715 INFO:journalctl@ceph.mgr.x.smithi191.stdout:Sep 10 19:18:07 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-x[34967]: ::ffff:172.21.15.191 - - [10/Sep/2024:19:18:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T19:18:07.746 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:18:07.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:18:07.780 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:18:07.780 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:18:07.867 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:18:07.881 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:18:07.921 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:18:07.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:18:07.922 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:18:08.005 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:18:07 smithi183 ceph-mon[29736]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "2.13", "id": [2, 0, 5]}]': finished 2024-09-10T19:18:08.005 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:18:07 smithi183 ceph-mon[29736]: osdmap e193: 8 total, 8 up, 6 in 2024-09-10T19:18:08.005 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:18:07 smithi183 ceph-mon[41562]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "2.13", "id": [2, 0, 5]}]': finished 2024-09-10T19:18:08.005 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:18:07 smithi183 ceph-mon[41562]: osdmap e193: 8 total, 8 up, 6 in 2024-09-10T19:18:08.022 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:18:08.054 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:18:08.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:18:08.055 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:18:08.156 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:18:08.183 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:18:08.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:18:08.184 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:18:08.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:18:07 smithi191 ceph-mon[32884]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "2.13", "id": [2, 0, 5]}]': finished 2024-09-10T19:18:08.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:18:07 smithi191 ceph-mon[32884]: osdmap e193: 8 total, 8 up, 6 in 2024-09-10T19:18:08.284 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:18:08.320 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:18:08.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:18:08.321 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:18:08.421 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:18:08.448 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:18:08.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:18:08.449 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:18:08.550 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:18:08.551 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:18:08.579 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:18:08.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:18:08.580 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:18:08.681 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:18:08.705 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:18:08.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:18:08.706 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:18:08.807 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:18:08.844 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:18:08.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:18:08.845 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:18:08.935 INFO:teuthology.orchestra.run.smithi183.stderr:noscrub is unset 2024-09-10T19:18:08.945 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:18:08.984 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:18:08.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:18:08.985 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:18:09.055 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:18:08 smithi183 ceph-mon[29736]: pgmap v345: 158 pgs: 1 active+recovering+remapped, 1 active+remapped+backfilling, 1 active+recovering, 4 active+remapped+backfill_toofull, 151 active+clean; 6.7 GiB data, 23 GiB used, 513 GiB / 536 GiB avail; 1.1 KiB/s rd, 13 KiB/s wr, 1.07k op/s; 31310/329202 objects misplaced (9.511%); 6.8 MiB/s, 108 objects/s recovering 2024-09-10T19:18:09.055 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:18:08 smithi183 ceph-mon[29736]: from='client.? 172.21.15.183:0/4117005588' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-10T19:18:09.055 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:18:08 smithi183 ceph-mon[41562]: pgmap v345: 158 pgs: 1 active+recovering+remapped, 1 active+remapped+backfilling, 1 active+recovering, 4 active+remapped+backfill_toofull, 151 active+clean; 6.7 GiB data, 23 GiB used, 513 GiB / 536 GiB avail; 1.1 KiB/s rd, 13 KiB/s wr, 1.07k op/s; 31310/329202 objects misplaced (9.511%); 6.8 MiB/s, 108 objects/s recovering 2024-09-10T19:18:09.055 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:18:08 smithi183 ceph-mon[41562]: from='client.? 172.21.15.183:0/4117005588' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-10T19:18:09.086 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:18:09.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:18:09.115 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:18:09.115 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:18:09.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:18:08 smithi191 ceph-mon[32884]: pgmap v345: 158 pgs: 1 active+recovering+remapped, 1 active+remapped+backfilling, 1 active+recovering, 4 active+remapped+backfill_toofull, 151 active+clean; 6.7 GiB data, 23 GiB used, 513 GiB / 536 GiB avail; 1.1 KiB/s rd, 13 KiB/s wr, 1.07k op/s; 31310/329202 objects misplaced (9.511%); 6.8 MiB/s, 108 objects/s recovering 2024-09-10T19:18:09.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:18:08 smithi191 ceph-mon[32884]: from='client.? 172.21.15.183:0/4117005588' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-10T19:18:09.216 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:18:09.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:18:09.258 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:18:09.259 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:18:09.359 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:18:09.386 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:18:09.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:18:09.389 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:18:09.490 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:18:09.515 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:18:09.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:18:09.516 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:18:09.617 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:18:09.656 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:18:09.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:18:09.657 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:18:09.715 INFO:journalctl@ceph.iscsi.iscsi.a.smithi191.stdout:Sep 10 19:18:09 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-iscsi-iscsi-a[61816]: debug there is no tcmu-runner data available 2024-09-10T19:18:09.758 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:18:09.783 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:18:09.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:18:09.784 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:18:09.885 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:18:09.911 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:18:09.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:18:09.912 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:18:10.013 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:18:10.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:18:10.041 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:18:10.041 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:18:10.084 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:18:09 smithi183 ceph-mon[29736]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T19:18:10.084 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:18:09 smithi183 ceph-mon[29736]: from='client.? 172.21.15.183:0/4117005588' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-10T19:18:10.084 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:18:09 smithi183 ceph-mon[29736]: osdmap e194: 8 total, 8 up, 6 in 2024-09-10T19:18:10.084 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:18:09 smithi183 ceph-mon[29736]: 6.0 scrub starts 2024-09-10T19:18:10.084 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:18:09 smithi183 ceph-mon[41562]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T19:18:10.084 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:18:09 smithi183 ceph-mon[41562]: from='client.? 172.21.15.183:0/4117005588' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-10T19:18:10.084 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:18:09 smithi183 ceph-mon[41562]: osdmap e194: 8 total, 8 up, 6 in 2024-09-10T19:18:10.085 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:18:09 smithi183 ceph-mon[41562]: 6.0 scrub starts 2024-09-10T19:18:10.142 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:18:10.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:18:10.170 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:18:10.170 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:18:10.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:18:09 smithi191 ceph-mon[32884]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T19:18:10.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:18:09 smithi191 ceph-mon[32884]: from='client.? 172.21.15.183:0/4117005588' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-10T19:18:10.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:18:09 smithi191 ceph-mon[32884]: osdmap e194: 8 total, 8 up, 6 in 2024-09-10T19:18:10.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:18:09 smithi191 ceph-mon[32884]: 6.0 scrub starts 2024-09-10T19:18:10.270 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:18:10.312 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:18:10.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:18:10.313 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:18:10.413 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:18:10.446 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:18:10.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:18:10.447 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:18:10.548 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:18:10.590 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:18:10.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:18:10.591 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:18:10.691 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:18:10.724 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:18:10.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:18:10.726 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:18:10.826 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:18:10.872 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:18:10.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:18:10.873 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:18:10.974 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:18:11.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:18:11.016 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:18:11.016 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:18:11.117 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:18:11.142 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:18:11.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:18:11.143 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:18:11.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:18:10 smithi183 ceph-mon[29736]: from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:18:11.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:18:10 smithi183 ceph-mon[29736]: pgmap v347: 158 pgs: 1 active+recovering+remapped, 1 active+remapped+backfilling, 1 active+recovering, 4 active+remapped+backfill_toofull, 151 active+clean; 6.7 GiB data, 23 GiB used, 513 GiB / 536 GiB avail; 639 B/s rd, 0 B/s wr, 739 op/s; 31310/329202 objects misplaced (9.511%); 6.1 MiB/s, 97 objects/s recovering 2024-09-10T19:18:11.181 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:18:10 smithi183 ceph-mon[41562]: from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:18:11.181 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:18:10 smithi183 ceph-mon[41562]: pgmap v347: 158 pgs: 1 active+recovering+remapped, 1 active+remapped+backfilling, 1 active+recovering, 4 active+remapped+backfill_toofull, 151 active+clean; 6.7 GiB data, 23 GiB used, 513 GiB / 536 GiB avail; 639 B/s rd, 0 B/s wr, 739 op/s; 31310/329202 objects misplaced (9.511%); 6.1 MiB/s, 97 objects/s recovering 2024-09-10T19:18:11.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:18:10 smithi191 ceph-mon[32884]: from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:18:11.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:18:10 smithi191 ceph-mon[32884]: pgmap v347: 158 pgs: 1 active+recovering+remapped, 1 active+remapped+backfilling, 1 active+recovering, 4 active+remapped+backfill_toofull, 151 active+clean; 6.7 GiB data, 23 GiB used, 513 GiB / 536 GiB avail; 639 B/s rd, 0 B/s wr, 739 op/s; 31310/329202 objects misplaced (9.511%); 6.1 MiB/s, 97 objects/s recovering 2024-09-10T19:18:11.244 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:18:11.271 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:18:11.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:18:11.272 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:18:11.373 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:18:11.407 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:18:11.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:18:11.408 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:18:11.509 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:18:11.534 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:18:11.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:18:11.536 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:18:11.636 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:18:11.659 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:18:11.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:18:11.660 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:18:11.681 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Sep 10 19:18:11 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-y[29989]: ::ffff:172.21.15.191 - - [10/Sep/2024:19:18:11] "GET /metrics HTTP/1.1" 200 224284 "" "Prometheus/2.43.0" 2024-09-10T19:18:11.761 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:18:11.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:18:11.802 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:18:11.802 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:18:11.846 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-10T19:18:11.903 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:18:11.947 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:18:11.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:18:11.948 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:18:12.049 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:18:12.080 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:18:12.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:18:12.081 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:18:12.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:18:11 smithi183 ceph-mon[29736]: 4.16 scrub starts 2024-09-10T19:18:12.181 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:18:11 smithi183 ceph-mon[41562]: 4.16 scrub starts 2024-09-10T19:18:12.182 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:18:12.206 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:18:12.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:18:12.207 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:18:12.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:18:11 smithi191 ceph-mon[32884]: 4.16 scrub starts 2024-09-10T19:18:12.308 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:18:12.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:18:12.346 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:18:12.346 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:18:12.447 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:18:12.470 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:18:12.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:18:12.471 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:18:12.572 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:18:12.596 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:18:12.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:18:12.597 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:18:12.698 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:18:12.724 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:18:12.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:18:12.725 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:18:12.826 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:18:12.859 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:18:12.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:18:12.860 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:18:12.961 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:18:12.987 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:18:12.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:18:12.988 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:18:13.028 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:18:12 smithi183 ceph-mon[29736]: pgmap v348: 158 pgs: 1 active+recovering+remapped, 1 active+remapped+backfilling, 1 active+recovering, 4 active+remapped+backfill_toofull, 151 active+clean; 6.3 GiB data, 22 GiB used, 514 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.49k op/s; 28341/311145 objects misplaced (9.109%); 12 MiB/s, 190 objects/s recovering 2024-09-10T19:18:13.028 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:18:12 smithi183 ceph-mon[41562]: pgmap v348: 158 pgs: 1 active+recovering+remapped, 1 active+remapped+backfilling, 1 active+recovering, 4 active+remapped+backfill_toofull, 151 active+clean; 6.3 GiB data, 22 GiB used, 514 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.49k op/s; 28341/311145 objects misplaced (9.109%); 12 MiB/s, 190 objects/s recovering 2024-09-10T19:18:13.089 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:18:13.114 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:18:13.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:18:13.115 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:18:13.215 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:18:13.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:18:12 smithi191 ceph-mon[32884]: pgmap v348: 158 pgs: 1 active+recovering+remapped, 1 active+remapped+backfilling, 1 active+recovering, 4 active+remapped+backfill_toofull, 151 active+clean; 6.3 GiB data, 22 GiB used, 514 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.49k op/s; 28341/311145 objects misplaced (9.109%); 12 MiB/s, 190 objects/s recovering 2024-09-10T19:18:13.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:18:13.256 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:18:13.256 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:18:13.357 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:18:13.381 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:18:13.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:18:13.382 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:18:13.480 INFO:tasks.thrashosds.thrasher:in_osds: [0, 3, 4, 5, 6, 7] out_osds: [1, 2] dead_osds: [] live_osds: [0, 2, 3, 4, 5, 6, 7, 1] 2024-09-10T19:18:13.480 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-10T19:18:13.480 INFO:tasks.thrashosds.thrasher:fixing pg num pool .mgr 2024-09-10T19:18:13.481 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph pg dump --format=json 2024-09-10T19:18:13.483 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:18:13.516 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:18:13.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:18:13.517 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:18:13.618 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:18:13.653 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:18:13.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:18:13.654 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:18:13.668 INFO:teuthology.orchestra.run.smithi183.stdout:osd.0: {} 2024-09-10T19:18:13.669 INFO:teuthology.orchestra.run.smithi183.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-10T19:18:13.694 INFO:teuthology.orchestra.run.smithi183.stdout:osd.1: {} 2024-09-10T19:18:13.694 INFO:teuthology.orchestra.run.smithi183.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-10T19:18:13.717 INFO:teuthology.orchestra.run.smithi183.stdout:osd.2: {} 2024-09-10T19:18:13.717 INFO:teuthology.orchestra.run.smithi183.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-10T19:18:13.740 INFO:teuthology.orchestra.run.smithi183.stdout:osd.3: {} 2024-09-10T19:18:13.741 INFO:teuthology.orchestra.run.smithi183.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-10T19:18:13.755 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:18:13.768 INFO:teuthology.orchestra.run.smithi183.stdout:osd.4: {} 2024-09-10T19:18:13.768 INFO:teuthology.orchestra.run.smithi183.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-10T19:18:13.782 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:18:13.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:18:13.783 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:18:13.795 INFO:teuthology.orchestra.run.smithi183.stdout:osd.5: {} 2024-09-10T19:18:13.796 INFO:teuthology.orchestra.run.smithi183.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-10T19:18:13.817 INFO:teuthology.orchestra.run.smithi183.stdout:osd.6: {} 2024-09-10T19:18:13.817 INFO:teuthology.orchestra.run.smithi183.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-10T19:18:13.830 INFO:teuthology.orchestra.run.smithi183.stdout:ERROR: (22) Invalid argument 2024-09-10T19:18:13.830 INFO:teuthology.orchestra.run.smithi183.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-10T19:18:13.844 INFO:teuthology.orchestra.run.smithi183.stdout:osd.7: {} 2024-09-10T19:18:13.845 INFO:teuthology.orchestra.run.smithi183.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-10T19:18:13.884 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:18:13.923 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:18:13.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:18:13.924 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:18:14.025 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:18:14.062 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:18:14.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:18:14.063 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:18:14.164 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:18:14.198 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:18:14.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:18:14.199 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:18:14.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:18:13 smithi191 ceph-mon[32884]: from='client.? 172.21.15.183:0/1321328237' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T19:18:14.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:18:13 smithi191 ceph-mon[32884]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:18:14.300 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:18:14.326 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:18:14.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:18:14.328 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:18:14.403 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:18:13 smithi183 ceph-mon[29736]: from='client.? 172.21.15.183:0/1321328237' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T19:18:14.403 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:18:13 smithi183 ceph-mon[29736]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:18:14.403 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:18:13 smithi183 ceph-mon[41562]: from='client.? 172.21.15.183:0/1321328237' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T19:18:14.403 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:18:13 smithi183 ceph-mon[41562]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:18:14.430 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:18:14.462 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:18:14.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:18:14.463 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:18:14.564 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:18:14.588 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:18:14.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:18:14.589 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:18:14.690 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:18:14.725 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:18:14.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:18:14.726 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:18:14.827 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:18:14.868 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:18:14.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:18:14.869 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:18:14.970 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:18:14.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:18:14.994 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:18:14.995 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:18:15.096 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:18:15.121 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:18:15.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:18:15.122 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:18:15.181 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:18:14 smithi183 ceph-mon[41562]: pgmap v349: 158 pgs: 1 active+recovering+remapped, 1 active+remapped+backfilling, 1 active+recovering, 4 active+remapped+backfill_toofull, 151 active+clean; 6.3 GiB data, 22 GiB used, 514 GiB / 536 GiB avail; 639 B/s rd, 0 B/s wr, 752 op/s; 28341/311145 objects misplaced (9.109%); 5.8 MiB/s, 92 objects/s recovering 2024-09-10T19:18:15.181 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:18:14 smithi183 ceph-mon[41562]: 5.3 scrub starts 2024-09-10T19:18:15.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:18:14 smithi183 ceph-mon[29736]: pgmap v349: 158 pgs: 1 active+recovering+remapped, 1 active+remapped+backfilling, 1 active+recovering, 4 active+remapped+backfill_toofull, 151 active+clean; 6.3 GiB data, 22 GiB used, 514 GiB / 536 GiB avail; 639 B/s rd, 0 B/s wr, 752 op/s; 28341/311145 objects misplaced (9.109%); 5.8 MiB/s, 92 objects/s recovering 2024-09-10T19:18:15.182 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:18:14 smithi183 ceph-mon[29736]: 5.3 scrub starts 2024-09-10T19:18:15.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:18:14 smithi191 ceph-mon[32884]: pgmap v349: 158 pgs: 1 active+recovering+remapped, 1 active+remapped+backfilling, 1 active+recovering, 4 active+remapped+backfill_toofull, 151 active+clean; 6.3 GiB data, 22 GiB used, 514 GiB / 536 GiB avail; 639 B/s rd, 0 B/s wr, 752 op/s; 28341/311145 objects misplaced (9.109%); 5.8 MiB/s, 92 objects/s recovering 2024-09-10T19:18:15.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:18:14 smithi191 ceph-mon[32884]: 5.3 scrub starts 2024-09-10T19:18:15.223 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:18:15.225 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph daemon osd.2 dump_ops_in_flight 2024-09-10T19:18:15.243 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:18:15.260 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:18:15.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:18:15.291 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:18:15.291 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:18:15.392 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:18:15.461 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:18:15.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:18:15.462 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:18:15.563 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:18:15.597 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:18:15.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:18:15.598 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:18:15.699 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:18:15.735 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:18:15.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:18:15.736 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:18:15.837 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:18:15.862 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:18:15.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:18:15.863 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:18:15.963 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:18:15.993 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:18:15.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:18:15.994 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:18:16.094 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:18:16.134 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:18:16.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:18:16.135 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:18:16.235 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:18:16.262 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:18:16.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:18:16.263 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:18:16.363 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:18:16.399 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:18:16.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:18:16.401 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:18:16.501 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:18:16.536 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:18:16.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:18:16.537 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:18:16.638 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:18:16.673 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:18:16.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:18:16.674 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:18:16.758 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-10T19:18:16.775 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:18:16.828 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:18:16.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:18:16.829 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:18:16.930 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:18:16.967 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:18:16.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:18:16.968 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:18:17.069 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:18:17.110 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:18:17.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:18:17.111 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:18:17.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:18:16 smithi183 ceph-mon[29736]: pgmap v350: 158 pgs: 1 active+recovering+remapped, 1 active+remapped+backfilling, 1 active+recovering, 4 active+remapped+backfill_toofull, 151 active+clean; 6.0 GiB data, 21 GiB used, 516 GiB / 536 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.34k op/s; 25555/293661 objects misplaced (8.702%); 10 MiB/s, 163 objects/s recovering 2024-09-10T19:18:17.181 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:18:16 smithi183 ceph-mon[41562]: pgmap v350: 158 pgs: 1 active+recovering+remapped, 1 active+remapped+backfilling, 1 active+recovering, 4 active+remapped+backfill_toofull, 151 active+clean; 6.0 GiB data, 21 GiB used, 516 GiB / 536 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.34k op/s; 25555/293661 objects misplaced (8.702%); 10 MiB/s, 163 objects/s recovering 2024-09-10T19:18:17.212 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:18:17.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:18:16 smithi191 ceph-mon[32884]: pgmap v350: 158 pgs: 1 active+recovering+remapped, 1 active+remapped+backfilling, 1 active+recovering, 4 active+remapped+backfill_toofull, 151 active+clean; 6.0 GiB data, 21 GiB used, 516 GiB / 536 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.34k op/s; 25555/293661 objects misplaced (8.702%); 10 MiB/s, 163 objects/s recovering 2024-09-10T19:18:17.237 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:18:17.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:18:17.238 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:18:17.278 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:18:17.339 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:18:17.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:18:17.381 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:18:17.381 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:18:17.482 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:18:17.506 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:18:17.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:18:17.507 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:18:17.608 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:18:17.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:18:17.634 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:18:17.635 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:18:17.715 INFO:journalctl@ceph.mgr.x.smithi191.stdout:Sep 10 19:18:17 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-x[34967]: ::ffff:172.21.15.191 - - [10/Sep/2024:19:18:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T19:18:17.735 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:18:17.773 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:18:17.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:18:17.774 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:18:17.876 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:18:17.900 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:18:17.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:18:17.901 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:18:18.002 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:18:18.030 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:18:18.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:18:18.031 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:18:18.132 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:18:18.168 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:18:18.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:18:18.169 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:18:18.270 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:18:18.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:18:18.302 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:18:18.303 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:18:18.403 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:18:18.429 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:18:18.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:18:18.430 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:18:18.530 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:18:18.560 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:18:18.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:18:18.561 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:18:18.662 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:18:18.688 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:18:18.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:18:18.689 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:18:18.790 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:18:18.815 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:18:18.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:18:18.816 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:18:18.831 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-09-10T19:18:18.834 INFO:teuthology.orchestra.run.smithi183.stderr:dumped all 2024-09-10T19:18:18.917 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:18:18.963 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:18:18.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:18:18.964 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:18:19.064 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:18:19.089 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:18:19.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:18:19.090 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:18:19.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:18:18 smithi183 ceph-mon[29736]: pgmap v351: 158 pgs: 1 active+recovering+remapped, 1 active+remapped+backfilling, 1 active+recovering, 4 active+remapped+backfill_toofull, 151 active+clean; 6.0 GiB data, 21 GiB used, 516 GiB / 536 GiB avail; 1023 B/s rd, 0 B/s wr, 1.19k op/s; 25555/293661 objects misplaced (8.702%); 9.0 MiB/s, 144 objects/s recovering 2024-09-10T19:18:19.181 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:18:18 smithi183 ceph-mon[41562]: pgmap v351: 158 pgs: 1 active+recovering+remapped, 1 active+remapped+backfilling, 1 active+recovering, 4 active+remapped+backfill_toofull, 151 active+clean; 6.0 GiB data, 21 GiB used, 516 GiB / 536 GiB avail; 1023 B/s rd, 0 B/s wr, 1.19k op/s; 25555/293661 objects misplaced (8.702%); 9.0 MiB/s, 144 objects/s recovering 2024-09-10T19:18:19.191 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:18:19.215 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:18:19.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:18:19.216 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:18:19.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:18:18 smithi191 ceph-mon[32884]: pgmap v351: 158 pgs: 1 active+recovering+remapped, 1 active+remapped+backfilling, 1 active+recovering, 4 active+remapped+backfill_toofull, 151 active+clean; 6.0 GiB data, 21 GiB used, 516 GiB / 536 GiB avail; 1023 B/s rd, 0 B/s wr, 1.19k op/s; 25555/293661 objects misplaced (8.702%); 9.0 MiB/s, 144 objects/s recovering 2024-09-10T19:18:19.316 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:18:19.351 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:18:19.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:18:19.352 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:18:19.453 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:18:19.478 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:18:19.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:18:19.479 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:18:19.580 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:18:19.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:18:19.607 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:18:19.607 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:18:19.708 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:18:19.715 INFO:journalctl@ceph.iscsi.iscsi.a.smithi191.stdout:Sep 10 19:18:19 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-iscsi-iscsi-a[61816]: debug there is no tcmu-runner data available 2024-09-10T19:18:19.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:18:19.748 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:18:19.749 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:18:19.850 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:18:19.877 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:18:19.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:18:19.878 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:18:19.979 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:18:20.012 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:18:20.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:18:20.013 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:18:20.113 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:18:20.155 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:18:20.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:18:20.156 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:18:20.257 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:18:20.298 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:18:20.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:18:20.299 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:18:20.314 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:18:19 smithi183 ceph-mon[29736]: from='client.15204 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-10T19:18:20.314 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:18:19 smithi183 ceph-mon[29736]: purged_snaps scrub starts 2024-09-10T19:18:20.314 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:18:19 smithi183 ceph-mon[29736]: purged_snaps scrub ok 2024-09-10T19:18:20.314 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:18:19 smithi183 ceph-mon[29736]: osdmap e195: 8 total, 8 up, 6 in 2024-09-10T19:18:20.314 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:18:19 smithi183 ceph-mon[29736]: osdmap e196: 8 total, 8 up, 6 in 2024-09-10T19:18:20.314 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:18:19 smithi183 ceph-mon[41562]: from='client.15204 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-10T19:18:20.314 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:18:19 smithi183 ceph-mon[41562]: purged_snaps scrub starts 2024-09-10T19:18:20.314 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:18:19 smithi183 ceph-mon[41562]: purged_snaps scrub ok 2024-09-10T19:18:20.314 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:18:19 smithi183 ceph-mon[41562]: osdmap e195: 8 total, 8 up, 6 in 2024-09-10T19:18:20.315 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:18:19 smithi183 ceph-mon[41562]: osdmap e196: 8 total, 8 up, 6 in 2024-09-10T19:18:20.399 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:18:20.428 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:18:20.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:18:20.429 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:18:20.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:18:19 smithi191 ceph-mon[32884]: from='client.15204 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-10T19:18:20.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:18:19 smithi191 ceph-mon[32884]: purged_snaps scrub starts 2024-09-10T19:18:20.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:18:19 smithi191 ceph-mon[32884]: purged_snaps scrub ok 2024-09-10T19:18:20.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:18:19 smithi191 ceph-mon[32884]: osdmap e195: 8 total, 8 up, 6 in 2024-09-10T19:18:20.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:18:19 smithi191 ceph-mon[32884]: osdmap e196: 8 total, 8 up, 6 in 2024-09-10T19:18:20.530 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:18:20.569 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:18:20.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:18:20.570 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:18:20.671 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:18:20.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:18:20.706 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:18:20.706 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:18:20.807 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:18:20.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:18:20.841 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:18:20.841 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:18:20.942 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:18:20.970 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:18:20.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:18:20.971 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:18:20.989 INFO:teuthology.orchestra.run.smithi183.stderr:nodeep-scrub is unset 2024-09-10T19:18:21.072 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:18:21.098 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:18:21.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:18:21.099 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:18:21.129 INFO:teuthology.orchestra.run.smithi183.stdout:{"pg_ready":true,"pg_map":{"version":351,"stamp":"2024-09-10T19:18:17.751567+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":6402808052,"num_objects":97887,"num_object_clones":0,"num_object_copies":293661,"num_objects_missing_on_primary":993,"num_objects_missing":993,"num_objects_degraded":0,"num_objects_misplaced":25555,"num_objects_unfound":0,"num_objects_dirty":97887,"num_whiteouts":0,"num_read":965,"num_read_kb":822,"num_write":143414,"num_write_kb":7699592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17102,"num_bytes_recovered":1109463477,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":135619,"ondisk_log_size":135619,"up":474,"acting":474,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":491,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":562470912,"kb_used":21922248,"kb_used_data":20092248,"kb_used_omap":9,"kb_used_meta":1829814,"kb_avail":540548664,"statfs":{"total":575970213888,"available":553521831936,"internally_reserved":0,"allocated":20574461952,"data_stored":20571126276,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":10224,"internal_metadata":1873729552},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,3,5,1,4,1,4,2,0,1],"upper_bound":2048},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-776404992,"num_objects":-11847,"num_object_clones":0,"num_object_copies":-35541,"num_objects_missing_on_primary":-48,"num_objects_missing":-48,"num_objects_degraded":0,"num_objects_misplaced":-5755,"num_objects_unfound":0,"num_objects_dirty":-11847,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":11847,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1448,"num_bytes_recovered":94896128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.003405"},"pg_stats":[{"pgid":"8.15","version":"194'7403","reported_seq":7488,"reported_epoch":194,"state":"active+remapped+backfill_toofull","last_fresh":"2024-09-10T19:18:14.823943+0000","last_change":"2024-09-10T19:18:00.012242+0000","last_active":"2024-09-10T19:18:14.823943+0000","last_peered":"2024-09-10T19:18:14.823943+0000","last_clean":"2024-09-10T19:17:27.199333+0000","last_became_active":"2024-09-10T19:17:30.001644+0000","last_became_peered":"2024-09-10T19:17:30.001644+0000","last_unstale":"2024-09-10T19:18:14.823943+0000","last_undegraded":"2024-09-10T19:18:14.823943+0000","last_fullsized":"2024-09-10T19:18:14.823943+0000","mapping_epoch":180,"log_start":"160'1995","ondisk_log_start":"160'1995","created":177,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:16:11.777014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:16:11.777014+0000","last_clean_scrub_stamp":"2024-09-10T19:16:11.777014+0000","objects_scrubbed":0,"log_size":5408,"log_dups_size":0,"ondisk_log_size":5408,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":203620352,"num_objects":3107,"num_object_clones":0,"num_object_copies":9321,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3107,"num_objects_unfound":0,"num_objects_dirty":3107,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4479,"num_write_kb":242752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,4],"avail_no_missing":["7","4","6"],"object_location_counts":[{"shards":"4,6,7","objects":3107}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":158,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-09-10T19:18:06.907451+0000","last_change":"2024-09-10T19:13:08.710758+0000","last_active":"2024-09-10T19:18:06.907451+0000","last_peered":"2024-09-10T19:18:06.907451+0000","last_clean":"2024-09-10T19:18:06.907451+0000","last_became_active":"2024-09-10T19:10:14.056187+0000","last_became_peered":"2024-09-10T19:10:14.056187+0000","last_unstale":"2024-09-10T19:18:06.907451+0000","last_undegraded":"2024-09-10T19:18:06.907451+0000","last_fullsized":"2024-09-10T19:18:06.907451+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:13:08.710679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:13:08.710679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0027252499999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":193,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-09-10T19:18:06.913854+0000","last_change":"2024-09-10T19:16:38.194521+0000","last_active":"2024-09-10T19:18:06.913854+0000","last_peered":"2024-09-10T19:18:06.913854+0000","last_clean":"2024-09-10T19:18:06.913854+0000","last_became_active":"2024-09-10T19:16:24.836961+0000","last_became_peered":"2024-09-10T19:16:24.836961+0000","last_unstale":"2024-09-10T19:18:06.913854+0000","last_undegraded":"2024-09-10T19:18:06.913854+0000","last_fullsized":"2024-09-10T19:18:06.913854+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:16:38.194472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:16:38.194472+0000","last_clean_scrub_stamp":"2024-09-10T19:16:38.194472+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0023678900000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.19","version":"66'15","reported_seq":213,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-09-10T19:18:06.906457+0000","last_change":"2024-09-10T19:16:24.879212+0000","last_active":"2024-09-10T19:18:06.906457+0000","last_peered":"2024-09-10T19:18:06.906457+0000","last_clean":"2024-09-10T19:18:06.906457+0000","last_became_active":"2024-09-10T19:16:24.844951+0000","last_became_peered":"2024-09-10T19:16:24.844951+0000","last_unstale":"2024-09-10T19:18:06.906457+0000","last_undegraded":"2024-09-10T19:18:06.906457+0000","last_fullsized":"2024-09-10T19:18:06.906457+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'15","last_scrub_stamp":"2024-09-10T19:13:47.165848+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:13:47.165848+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.004982841,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":192,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-09-10T19:18:06.912302+0000","last_change":"2024-09-10T19:17:22.938220+0000","last_active":"2024-09-10T19:18:06.912302+0000","last_peered":"2024-09-10T19:18:06.912302+0000","last_clean":"2024-09-10T19:18:06.912302+0000","last_became_active":"2024-09-10T19:16:24.845111+0000","last_became_peered":"2024-09-10T19:16:24.845111+0000","last_unstale":"2024-09-10T19:18:06.912302+0000","last_undegraded":"2024-09-10T19:18:06.912302+0000","last_fullsized":"2024-09-10T19:18:06.912302+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:15:33.976696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:15:33.976696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0015310580000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,7],"acting":[4,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.14","version":"194'7285","reported_seq":17152,"reported_epoch":194,"state":"active+clean","last_fresh":"2024-09-10T19:18:14.716811+0000","last_change":"2024-09-10T19:17:49.772679+0000","last_active":"2024-09-10T19:18:14.716811+0000","last_peered":"2024-09-10T19:18:14.716811+0000","last_clean":"2024-09-10T19:18:14.716811+0000","last_became_active":"2024-09-10T19:17:49.767787+0000","last_became_peered":"2024-09-10T19:17:49.767787+0000","last_unstale":"2024-09-10T19:18:14.716811+0000","last_undegraded":"2024-09-10T19:18:14.716811+0000","last_fullsized":"2024-09-10T19:18:14.716811+0000","mapping_epoch":189,"log_start":"159'1500","ondisk_log_start":"159'1500","created":177,"last_epoch_clean":190,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:16:11.777014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:16:11.777014+0000","last_clean_scrub_stamp":"2024-09-10T19:16:11.777014+0000","objects_scrubbed":0,"log_size":5785,"log_dups_size":0,"ondisk_log_size":5785,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":197689344,"num_objects":3017,"num_object_clones":0,"num_object_copies":9051,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3017,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4423,"num_write_kb":238048,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3256,"num_bytes_recovered":213385216,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"59'1","reported_seq":182,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-09-10T19:18:06.906874+0000","last_change":"2024-09-10T19:16:30.798883+0000","last_active":"2024-09-10T19:18:06.906874+0000","last_peered":"2024-09-10T19:18:06.906874+0000","last_clean":"2024-09-10T19:18:06.906874+0000","last_became_active":"2024-09-10T19:10:07.686143+0000","last_became_peered":"2024-09-10T19:10:07.686143+0000","last_unstale":"2024-09-10T19:18:06.906874+0000","last_undegraded":"2024-09-10T19:18:06.906874+0000","last_fullsized":"2024-09-10T19:18:06.906874+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'1","last_scrub_stamp":"2024-09-10T19:16:30.798795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:16:30.798795+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0067002989999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":874,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"66'9","reported_seq":207,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-09-10T19:18:06.912375+0000","last_change":"2024-09-10T19:17:21.915673+0000","last_active":"2024-09-10T19:18:06.912375+0000","last_peered":"2024-09-10T19:18:06.912375+0000","last_clean":"2024-09-10T19:18:06.912375+0000","last_became_active":"2024-09-10T19:13:54.716117+0000","last_became_peered":"2024-09-10T19:13:54.716117+0000","last_unstale":"2024-09-10T19:18:06.912375+0000","last_undegraded":"2024-09-10T19:18:06.912375+0000","last_fullsized":"2024-09-10T19:18:06.912375+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'9","last_scrub_stamp":"2024-09-10T19:14:55.117387+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:14:55.117387+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0074980940000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,3],"acting":[4,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":134,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-09-10T19:18:06.913726+0000","last_change":"2024-09-10T19:16:27.753997+0000","last_active":"2024-09-10T19:18:06.913726+0000","last_peered":"2024-09-10T19:18:06.913726+0000","last_clean":"2024-09-10T19:18:06.913726+0000","last_became_active":"2024-09-10T19:16:24.859261+0000","last_became_peered":"2024-09-10T19:16:24.859261+0000","last_unstale":"2024-09-10T19:18:06.913726+0000","last_undegraded":"2024-09-10T19:18:06.913726+0000","last_fullsized":"2024-09-10T19:18:06.913726+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:16:27.753912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:16:27.753912+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0029373799999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":180,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-09-10T19:18:06.911214+0000","last_change":"2024-09-10T19:17:23.968484+0000","last_active":"2024-09-10T19:18:06.911214+0000","last_peered":"2024-09-10T19:18:06.911214+0000","last_clean":"2024-09-10T19:18:06.911214+0000","last_became_active":"2024-09-10T19:13:54.713538+0000","last_became_peered":"2024-09-10T19:13:54.713538+0000","last_unstale":"2024-09-10T19:18:06.911214+0000","last_undegraded":"2024-09-10T19:18:06.911214+0000","last_fullsized":"2024-09-10T19:18:06.911214+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:15:39.910118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:15:39.910118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0013551839999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,7],"acting":[4,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.17","version":"194'7378","reported_seq":7467,"reported_epoch":194,"state":"active+remapped+backfill_toofull","last_fresh":"2024-09-10T19:18:14.927519+0000","last_change":"2024-09-10T19:18:00.043042+0000","last_active":"2024-09-10T19:18:14.927519+0000","last_peered":"2024-09-10T19:18:14.927519+0000","last_clean":"2024-09-10T19:17:27.195401+0000","last_became_active":"2024-09-10T19:17:30.037948+0000","last_became_peered":"2024-09-10T19:17:30.037948+0000","last_unstale":"2024-09-10T19:18:14.927519+0000","last_undegraded":"2024-09-10T19:18:14.927519+0000","last_fullsized":"2024-09-10T19:18:14.927519+0000","mapping_epoch":180,"log_start":"160'1900","ondisk_log_start":"160'1900","created":177,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:16:11.777014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:16:11.777014+0000","last_clean_scrub_stamp":"2024-09-10T19:16:11.777014+0000","objects_scrubbed":0,"log_size":5478,"log_dups_size":0,"ondisk_log_size":5478,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":198148096,"num_objects":3024,"num_object_clones":0,"num_object_copies":9072,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":6048,"num_objects_unfound":0,"num_objects_dirty":3024,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4480,"num_write_kb":240096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[3,6,4],"avail_no_missing":["3","4","6"],"object_location_counts":[{"shards":"3,4,6","objects":3024}],"blocked_by":[],"up_primary":7,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":174,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-09-10T19:18:06.914024+0000","last_change":"2024-09-10T19:16:25.595265+0000","last_active":"2024-09-10T19:18:06.914024+0000","last_peered":"2024-09-10T19:18:06.914024+0000","last_clean":"2024-09-10T19:18:06.914024+0000","last_became_active":"2024-09-10T19:10:07.690728+0000","last_became_peered":"2024-09-10T19:10:07.690728+0000","last_unstale":"2024-09-10T19:18:06.914024+0000","last_undegraded":"2024-09-10T19:18:06.914024+0000","last_fullsized":"2024-09-10T19:18:06.914024+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:16:25.595127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:16:25.595127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.003922633,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1b","version":"66'5","reported_seq":213,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-09-10T19:18:06.913144+0000","last_change":"2024-09-10T19:16:35.966457+0000","last_active":"2024-09-10T19:18:06.913144+0000","last_peered":"2024-09-10T19:18:06.913144+0000","last_clean":"2024-09-10T19:18:06.913144+0000","last_became_active":"2024-09-10T19:16:24.843708+0000","last_became_peered":"2024-09-10T19:16:24.843708+0000","last_unstale":"2024-09-10T19:18:06.913144+0000","last_undegraded":"2024-09-10T19:18:06.913144+0000","last_fullsized":"2024-09-10T19:18:06.913144+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'5","last_scrub_stamp":"2024-09-10T19:16:35.966402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:16:35.966402+0000","objects_scrubbed":2,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0019325740000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":177,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-09-10T19:18:06.911741+0000","last_change":"2024-09-10T19:15:33.518718+0000","last_active":"2024-09-10T19:18:06.911741+0000","last_peered":"2024-09-10T19:18:06.911741+0000","last_clean":"2024-09-10T19:18:06.911741+0000","last_became_active":"2024-09-10T19:13:54.719063+0000","last_became_peered":"2024-09-10T19:13:54.719063+0000","last_unstale":"2024-09-10T19:18:06.911741+0000","last_undegraded":"2024-09-10T19:18:06.911741+0000","last_fullsized":"2024-09-10T19:18:06.911741+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:15:33.518601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:15:33.518601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0013951650000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":185,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-09-10T19:18:06.913882+0000","last_change":"2024-09-10T19:16:24.845721+0000","last_active":"2024-09-10T19:18:06.913882+0000","last_peered":"2024-09-10T19:18:06.913882+0000","last_clean":"2024-09-10T19:18:06.913882+0000","last_became_active":"2024-09-10T19:16:24.845130+0000","last_became_peered":"2024-09-10T19:16:24.845130+0000","last_unstale":"2024-09-10T19:18:06.913882+0000","last_undegraded":"2024-09-10T19:18:06.913882+0000","last_fullsized":"2024-09-10T19:18:06.913882+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:15:45.616184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:15:45.616184+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0013989510000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,0],"acting":[5,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.16","version":"194'7293","reported_seq":7371,"reported_epoch":194,"state":"active+remapped+backfill_toofull","last_fresh":"2024-09-10T19:18:14.995984+0000","last_change":"2024-09-10T19:18:00.034968+0000","last_active":"2024-09-10T19:18:14.995984+0000","last_peered":"2024-09-10T19:18:14.995984+0000","last_clean":"2024-09-10T19:17:27.143398+0000","last_became_active":"2024-09-10T19:17:30.025679+0000","last_became_peered":"2024-09-10T19:17:30.025679+0000","last_unstale":"2024-09-10T19:18:14.995984+0000","last_undegraded":"2024-09-10T19:18:14.995984+0000","last_fullsized":"2024-09-10T19:18:14.995984+0000","mapping_epoch":180,"log_start":"159'1500","ondisk_log_start":"159'1500","created":177,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:16:11.777014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:16:11.777014+0000","last_clean_scrub_stamp":"2024-09-10T19:16:11.777014+0000","objects_scrubbed":0,"log_size":5793,"log_dups_size":0,"ondisk_log_size":5793,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":194281472,"num_objects":2965,"num_object_clones":0,"num_object_copies":8895,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":5930,"num_objects_unfound":0,"num_objects_dirty":2965,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4421,"num_write_kb":236320,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,0],"acting":[0,6,7],"avail_no_missing":["0","6","7"],"object_location_counts":[{"shards":"0,6,7","objects":2965}],"blocked_by":[],"up_primary":3,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":174,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-09-10T19:18:06.912649+0000","last_change":"2024-09-10T19:16:24.859112+0000","last_active":"2024-09-10T19:18:06.912649+0000","last_peered":"2024-09-10T19:18:06.912649+0000","last_clean":"2024-09-10T19:18:06.912649+0000","last_became_active":"2024-09-10T19:16:24.858800+0000","last_became_peered":"2024-09-10T19:16:24.858800+0000","last_unstale":"2024-09-10T19:18:06.912649+0000","last_undegraded":"2024-09-10T19:18:06.912649+0000","last_fullsized":"2024-09-10T19:18:06.912649+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:12:54.477149+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:12:54.477149+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0039768319999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1a","version":"66'9","reported_seq":180,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-09-10T19:18:06.911479+0000","last_change":"2024-09-10T19:17:14.981950+0000","last_active":"2024-09-10T19:18:06.911479+0000","last_peered":"2024-09-10T19:18:06.911479+0000","last_clean":"2024-09-10T19:18:06.911479+0000","last_became_active":"2024-09-10T19:10:09.688987+0000","last_became_peered":"2024-09-10T19:10:09.688987+0000","last_unstale":"2024-09-10T19:18:06.911479+0000","last_undegraded":"2024-09-10T19:18:06.911479+0000","last_fullsized":"2024-09-10T19:18:06.911479+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'9","last_scrub_stamp":"2024-09-10T19:13:02.431443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:13:02.431443+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0084913290000000006,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":170,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-09-10T19:18:06.911657+0000","last_change":"2024-09-10T19:16:37.116464+0000","last_active":"2024-09-10T19:18:06.911657+0000","last_peered":"2024-09-10T19:18:06.911657+0000","last_clean":"2024-09-10T19:18:06.911657+0000","last_became_active":"2024-09-10T19:10:11.701799+0000","last_became_peered":"2024-09-10T19:10:11.701799+0000","last_unstale":"2024-09-10T19:18:06.911657+0000","last_undegraded":"2024-09-10T19:18:06.911657+0000","last_fullsized":"2024-09-10T19:18:06.911657+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:16:37.116371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:16:37.116371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.028100301000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":187,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-09-10T19:18:06.910145+0000","last_change":"2024-09-10T19:17:18.281940+0000","last_active":"2024-09-10T19:18:06.910145+0000","last_peered":"2024-09-10T19:18:06.910145+0000","last_clean":"2024-09-10T19:18:06.910145+0000","last_became_active":"2024-09-10T19:16:24.836477+0000","last_became_peered":"2024-09-10T19:16:24.836477+0000","last_unstale":"2024-09-10T19:18:06.910145+0000","last_undegraded":"2024-09-10T19:18:06.910145+0000","last_fullsized":"2024-09-10T19:18:06.910145+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:15:33.839248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:15:33.839248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0032019980000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1","version":"194'7400","reported_seq":7473,"reported_epoch":194,"state":"active+clean","last_fresh":"2024-09-10T19:18:14.913581+0000","last_change":"2024-09-10T19:17:26.961275+0000","last_active":"2024-09-10T19:18:14.913581+0000","last_peered":"2024-09-10T19:18:14.913581+0000","last_clean":"2024-09-10T19:18:14.913581+0000","last_became_active":"2024-09-10T19:17:26.958866+0000","last_became_peered":"2024-09-10T19:17:26.958866+0000","last_unstale":"2024-09-10T19:18:14.913581+0000","last_undegraded":"2024-09-10T19:18:14.913581+0000","last_fullsized":"2024-09-10T19:18:14.913581+0000","mapping_epoch":154,"log_start":"159'1600","ondisk_log_start":"159'1600","created":148,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:16:11.777014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:16:11.777014+0000","last_clean_scrub_stamp":"2024-09-10T19:16:11.777014+0000","objects_scrubbed":0,"log_size":5800,"log_dups_size":0,"ondisk_log_size":5800,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":204308480,"num_objects":3117,"num_object_clones":0,"num_object_copies":9351,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3117,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4481,"num_write_kb":243168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.d","version":"66'17","reported_seq":262,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-09-10T19:18:06.912762+0000","last_change":"2024-09-10T19:16:32.999033+0000","last_active":"2024-09-10T19:18:06.912762+0000","last_peered":"2024-09-10T19:18:06.912762+0000","last_clean":"2024-09-10T19:18:06.912762+0000","last_became_active":"2024-09-10T19:16:24.847056+0000","last_became_peered":"2024-09-10T19:16:24.847056+0000","last_unstale":"2024-09-10T19:18:06.912762+0000","last_undegraded":"2024-09-10T19:18:06.912762+0000","last_fullsized":"2024-09-10T19:18:06.912762+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'17","last_scrub_stamp":"2024-09-10T19:16:32.998950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:16:32.998950+0000","objects_scrubbed":8,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0095505250000000007,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,0],"acting":[4,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":174,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-09-10T19:18:06.931853+0000","last_change":"2024-09-10T19:16:39.716870+0000","last_active":"2024-09-10T19:18:06.931853+0000","last_peered":"2024-09-10T19:18:06.931853+0000","last_clean":"2024-09-10T19:18:06.931853+0000","last_became_active":"2024-09-10T19:10:07.691119+0000","last_became_peered":"2024-09-10T19:10:07.691119+0000","last_unstale":"2024-09-10T19:18:06.931853+0000","last_undegraded":"2024-09-10T19:18:06.931853+0000","last_fullsized":"2024-09-10T19:18:06.931853+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:16:39.716674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:16:39.716674+0000","last_clean_scrub_stamp":"2024-09-10T19:16:39.716674+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.043895318000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":127,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-09-10T19:18:06.910527+0000","last_change":"2024-09-10T19:16:26.999105+0000","last_active":"2024-09-10T19:18:06.910527+0000","last_peered":"2024-09-10T19:18:06.910527+0000","last_clean":"2024-09-10T19:18:06.910527+0000","last_became_active":"2024-09-10T19:13:54.712883+0000","last_became_peered":"2024-09-10T19:13:54.712883+0000","last_unstale":"2024-09-10T19:18:06.910527+0000","last_undegraded":"2024-09-10T19:18:06.910527+0000","last_fullsized":"2024-09-10T19:18:06.910527+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:16:26.999036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:16:26.999036+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0050193119999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":52,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-09-10T19:18:06.907001+0000","last_change":"2024-09-10T19:16:24.858656+0000","last_active":"2024-09-10T19:18:06.907001+0000","last_peered":"2024-09-10T19:18:06.907001+0000","last_clean":"2024-09-10T19:18:06.907001+0000","last_became_active":"2024-09-10T19:16:24.857843+0000","last_became_peered":"2024-09-10T19:16:24.857843+0000","last_unstale":"2024-09-10T19:18:06.907001+0000","last_undegraded":"2024-09-10T19:18:06.907001+0000","last_fullsized":"2024-09-10T19:18:06.907001+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:15:47.818381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:15:47.818381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.0","version":"194'7392","reported_seq":7458,"reported_epoch":194,"state":"active+clean","last_fresh":"2024-09-10T19:18:14.943569+0000","last_change":"2024-09-10T19:17:26.971893+0000","last_active":"2024-09-10T19:18:14.943569+0000","last_peered":"2024-09-10T19:18:14.943569+0000","last_clean":"2024-09-10T19:18:14.943569+0000","last_became_active":"2024-09-10T19:17:26.958072+0000","last_became_peered":"2024-09-10T19:17:26.958072+0000","last_unstale":"2024-09-10T19:18:14.943569+0000","last_undegraded":"2024-09-10T19:18:14.943569+0000","last_fullsized":"2024-09-10T19:18:14.943569+0000","mapping_epoch":148,"log_start":"160'1900","ondisk_log_start":"160'1900","created":148,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:16:11.777014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:16:11.777014+0000","last_clean_scrub_stamp":"2024-09-10T19:16:11.777014+0000","objects_scrubbed":0,"log_size":5492,"log_dups_size":0,"ondisk_log_size":5492,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":202244096,"num_objects":3086,"num_object_clones":0,"num_object_copies":9258,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3086,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4476,"num_write_kb":241984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,4],"acting":[3,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"66'10","reported_seq":189,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-09-10T19:18:06.910722+0000","last_change":"2024-09-10T19:17:20.959169+0000","last_active":"2024-09-10T19:18:06.910722+0000","last_peered":"2024-09-10T19:18:06.910722+0000","last_clean":"2024-09-10T19:18:06.910722+0000","last_became_active":"2024-09-10T19:10:09.688321+0000","last_became_peered":"2024-09-10T19:10:09.688321+0000","last_unstale":"2024-09-10T19:18:06.910722+0000","last_undegraded":"2024-09-10T19:18:06.910722+0000","last_fullsized":"2024-09-10T19:18:06.910722+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'10","last_scrub_stamp":"2024-09-10T19:13:51.415998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:13:51.415998+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.005972799,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":124,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-09-10T19:18:06.906570+0000","last_change":"2024-09-10T19:15:42.119912+0000","last_active":"2024-09-10T19:18:06.906570+0000","last_peered":"2024-09-10T19:18:06.906570+0000","last_clean":"2024-09-10T19:18:06.906570+0000","last_became_active":"2024-09-10T19:13:54.731134+0000","last_became_peered":"2024-09-10T19:13:54.731134+0000","last_unstale":"2024-09-10T19:18:06.906570+0000","last_undegraded":"2024-09-10T19:18:06.906570+0000","last_fullsized":"2024-09-10T19:18:06.906570+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:15:42.119590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:15:42.119590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.018001347000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,5],"acting":[3,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"155'10","reported_seq":144,"reported_epoch":194,"state":"active+clean","last_fresh":"2024-09-10T19:18:08.950482+0000","last_change":"2024-09-10T19:16:29.626806+0000","last_active":"2024-09-10T19:18:08.950482+0000","last_peered":"2024-09-10T19:18:08.950482+0000","last_clean":"2024-09-10T19:18:08.950482+0000","last_became_active":"2024-09-10T19:16:24.851236+0000","last_became_peered":"2024-09-10T19:16:24.851236+0000","last_unstale":"2024-09-10T19:18:08.950482+0000","last_undegraded":"2024-09-10T19:18:08.950482+0000","last_fullsized":"2024-09-10T19:18:08.950482+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"155'10","last_scrub_stamp":"2024-09-10T19:16:29.626572+0000","last_deep_scrub":"80'9","last_deep_scrub_stamp":"2024-09-10T19:11:25.156465+0000","last_clean_scrub_stamp":"2024-09-10T19:16:29.626572+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0067064810000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,0],"acting":[7,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":190,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-09-10T19:18:06.912255+0000","last_change":"2024-09-10T19:16:24.852100+0000","last_active":"2024-09-10T19:18:06.912255+0000","last_peered":"2024-09-10T19:18:06.912255+0000","last_clean":"2024-09-10T19:18:06.912255+0000","last_became_active":"2024-09-10T19:16:24.851829+0000","last_became_peered":"2024-09-10T19:16:24.851829+0000","last_unstale":"2024-09-10T19:18:06.912255+0000","last_undegraded":"2024-09-10T19:18:06.912255+0000","last_fullsized":"2024-09-10T19:18:06.912255+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:15:29.988976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:15:29.988976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0053787150000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,7],"acting":[4,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.3","version":"194'7617","reported_seq":10656,"reported_epoch":194,"state":"active+recovering","last_fresh":"2024-09-10T19:18:14.823296+0000","last_change":"2024-09-10T19:17:28.441322+0000","last_active":"2024-09-10T19:18:14.823296+0000","last_peered":"2024-09-10T19:18:14.823296+0000","last_clean":"2024-09-10T19:16:54.964816+0000","last_became_active":"2024-09-10T19:17:27.974691+0000","last_became_peered":"2024-09-10T19:17:27.974691+0000","last_unstale":"2024-09-10T19:18:14.823296+0000","last_undegraded":"2024-09-10T19:18:14.823296+0000","last_fullsized":"2024-09-10T19:18:14.823296+0000","mapping_epoch":178,"log_start":"160'2257","ondisk_log_start":"160'2257","created":148,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:16:11.777014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:16:11.777014+0000","last_clean_scrub_stamp":"2024-09-10T19:16:11.777014+0000","objects_scrubbed":0,"log_size":5360,"log_dups_size":0,"ondisk_log_size":5360,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":205291520,"num_objects":3132,"num_object_clones":0,"num_object_copies":9396,"num_objects_missing_on_primary":993,"num_objects_missing":993,"num_objects_degraded":0,"num_objects_misplaced":993,"num_objects_unfound":0,"num_objects_dirty":3132,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4606,"num_write_kb":247648,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":934,"num_bytes_recovered":61243392,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":["3","5","6"],"object_location_counts":[{"shards":"3,5,6","objects":993},{"shards":"3,5,6,7","objects":1},{"shards":"3,5,7","objects":2138}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.f","version":"66'15","reported_seq":177,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-09-10T19:18:06.907371+0000","last_change":"2024-09-10T19:16:25.769523+0000","last_active":"2024-09-10T19:18:06.907371+0000","last_peered":"2024-09-10T19:18:06.907371+0000","last_clean":"2024-09-10T19:18:06.907371+0000","last_became_active":"2024-09-10T19:13:54.711634+0000","last_became_peered":"2024-09-10T19:13:54.711634+0000","last_unstale":"2024-09-10T19:18:06.907371+0000","last_undegraded":"2024-09-10T19:18:06.907371+0000","last_fullsized":"2024-09-10T19:18:06.907371+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'15","last_scrub_stamp":"2024-09-10T19:16:25.769436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:16:25.769436+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0046254920000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":134,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-09-10T19:18:06.912111+0000","last_change":"2024-09-10T19:16:24.857467+0000","last_active":"2024-09-10T19:18:06.912111+0000","last_peered":"2024-09-10T19:18:06.912111+0000","last_clean":"2024-09-10T19:18:06.912111+0000","last_became_active":"2024-09-10T19:16:24.857179+0000","last_became_peered":"2024-09-10T19:16:24.857179+0000","last_unstale":"2024-09-10T19:18:06.912111+0000","last_undegraded":"2024-09-10T19:18:06.912111+0000","last_fullsized":"2024-09-10T19:18:06.912111+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:15:29.553448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:15:29.553448+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0062080679999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,6],"acting":[7,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.e","version":"66'8","reported_seq":267,"reported_epoch":194,"state":"active+clean","last_fresh":"2024-09-10T19:18:08.948572+0000","last_change":"2024-09-10T19:17:18.952116+0000","last_active":"2024-09-10T19:18:08.948572+0000","last_peered":"2024-09-10T19:18:08.948572+0000","last_clean":"2024-09-10T19:18:08.948572+0000","last_became_active":"2024-09-10T19:10:11.699232+0000","last_became_peered":"2024-09-10T19:10:11.699232+0000","last_unstale":"2024-09-10T19:18:08.948572+0000","last_undegraded":"2024-09-10T19:18:08.948572+0000","last_fullsized":"2024-09-10T19:18:08.948572+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'8","last_scrub_stamp":"2024-09-10T19:13:40.475375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:13:40.475375+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.035806152000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":185,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-09-10T19:18:06.911193+0000","last_change":"2024-09-10T19:16:24.855938+0000","last_active":"2024-09-10T19:18:06.911193+0000","last_peered":"2024-09-10T19:18:06.911193+0000","last_clean":"2024-09-10T19:18:06.911193+0000","last_became_active":"2024-09-10T19:16:24.855627+0000","last_became_peered":"2024-09-10T19:16:24.855627+0000","last_unstale":"2024-09-10T19:18:06.911193+0000","last_undegraded":"2024-09-10T19:18:06.911193+0000","last_fullsized":"2024-09-10T19:18:06.911193+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:15:44.633043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:15:44.633043+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.00368805,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,6],"acting":[5,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.2","version":"194'7339","reported_seq":7404,"reported_epoch":194,"state":"active+clean","last_fresh":"2024-09-10T19:18:14.712043+0000","last_change":"2024-09-10T19:17:26.956443+0000","last_active":"2024-09-10T19:18:14.712043+0000","last_peered":"2024-09-10T19:18:14.712043+0000","last_clean":"2024-09-10T19:18:14.712043+0000","last_became_active":"2024-09-10T19:17:26.953881+0000","last_became_peered":"2024-09-10T19:17:26.953881+0000","last_unstale":"2024-09-10T19:18:14.712043+0000","last_undegraded":"2024-09-10T19:18:14.712043+0000","last_fullsized":"2024-09-10T19:18:14.712043+0000","mapping_epoch":148,"log_start":"159'1500","ondisk_log_start":"159'1500","created":148,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:16:11.777014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:16:11.777014+0000","last_clean_scrub_stamp":"2024-09-10T19:16:11.777014+0000","objects_scrubbed":0,"log_size":5839,"log_dups_size":0,"ondisk_log_size":5839,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":196214784,"num_objects":2994,"num_object_clones":0,"num_object_copies":8982,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2994,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4422,"num_write_kb":237312,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,5],"acting":[4,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"84'16","reported_seq":217,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-09-10T19:18:06.912612+0000","last_change":"2024-09-10T19:15:47.914487+0000","last_active":"2024-09-10T19:18:06.912612+0000","last_peered":"2024-09-10T19:18:06.912612+0000","last_clean":"2024-09-10T19:18:06.912612+0000","last_became_active":"2024-09-10T19:13:54.717022+0000","last_became_peered":"2024-09-10T19:13:54.717022+0000","last_unstale":"2024-09-10T19:18:06.912612+0000","last_undegraded":"2024-09-10T19:18:06.912612+0000","last_fullsized":"2024-09-10T19:18:06.912612+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'16","last_scrub_stamp":"2024-09-10T19:15:47.914420+0000","last_deep_scrub":"84'16","last_deep_scrub_stamp":"2024-09-10T19:11:56.157975+0000","last_clean_scrub_stamp":"2024-09-10T19:15:47.914420+0000","objects_scrubbed":5,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0019153390000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,3],"acting":[4,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":181,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-09-10T19:18:06.912990+0000","last_change":"2024-09-10T19:15:42.450149+0000","last_active":"2024-09-10T19:18:06.912990+0000","last_peered":"2024-09-10T19:18:06.912990+0000","last_clean":"2024-09-10T19:18:06.912990+0000","last_became_active":"2024-09-10T19:13:54.720293+0000","last_became_peered":"2024-09-10T19:13:54.720293+0000","last_unstale":"2024-09-10T19:18:06.912990+0000","last_undegraded":"2024-09-10T19:18:06.912990+0000","last_fullsized":"2024-09-10T19:18:06.912990+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:15:42.450068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:11:53.066849+0000","last_clean_scrub_stamp":"2024-09-10T19:15:42.450068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0037708360000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":160,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-09-10T19:18:06.910983+0000","last_change":"2024-09-10T19:13:06.358188+0000","last_active":"2024-09-10T19:18:06.910983+0000","last_peered":"2024-09-10T19:18:06.910983+0000","last_clean":"2024-09-10T19:18:06.910983+0000","last_became_active":"2024-09-10T19:10:11.692215+0000","last_became_peered":"2024-09-10T19:10:11.692215+0000","last_unstale":"2024-09-10T19:18:06.910983+0000","last_undegraded":"2024-09-10T19:18:06.910983+0000","last_fullsized":"2024-09-10T19:18:06.910983+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:13:06.358109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:13:06.358109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0026724549999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":158,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-09-10T19:18:06.906178+0000","last_change":"2024-09-10T19:13:54.304493+0000","last_active":"2024-09-10T19:18:06.906178+0000","last_peered":"2024-09-10T19:18:06.906178+0000","last_clean":"2024-09-10T19:18:06.906178+0000","last_became_active":"2024-09-10T19:10:14.053736+0000","last_became_peered":"2024-09-10T19:10:14.053736+0000","last_unstale":"2024-09-10T19:18:06.906178+0000","last_undegraded":"2024-09-10T19:18:06.906178+0000","last_fullsized":"2024-09-10T19:18:06.906178+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:13:54.304435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:13:54.304435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0011043820000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.d","version":"194'8952","reported_seq":9017,"reported_epoch":194,"state":"active+clean","last_fresh":"2024-09-10T19:18:14.851378+0000","last_change":"2024-09-10T19:16:24.858800+0000","last_active":"2024-09-10T19:18:14.851378+0000","last_peered":"2024-09-10T19:18:14.851378+0000","last_clean":"2024-09-10T19:18:14.851378+0000","last_became_active":"2024-09-10T19:16:24.858460+0000","last_became_peered":"2024-09-10T19:16:24.858460+0000","last_unstale":"2024-09-10T19:18:14.851378+0000","last_undegraded":"2024-09-10T19:18:14.851378+0000","last_fullsized":"2024-09-10T19:18:14.851378+0000","mapping_epoch":154,"log_start":"173'5190","ondisk_log_start":"173'5190","created":148,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:16:11.777014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:16:11.777014+0000","last_clean_scrub_stamp":"2024-09-10T19:16:11.777014+0000","objects_scrubbed":0,"log_size":3762,"log_dups_size":0,"ondisk_log_size":3762,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":403505176,"num_objects":6158,"num_object_clones":0,"num_object_copies":18474,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6158,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":8952,"num_write_kb":483457,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1","version":"66'14","reported_seq":68,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-09-10T19:18:06.909525+0000","last_change":"2024-09-10T19:17:28.925690+0000","last_active":"2024-09-10T19:18:06.909525+0000","last_peered":"2024-09-10T19:18:06.909525+0000","last_clean":"2024-09-10T19:18:06.909525+0000","last_became_active":"2024-09-10T19:16:24.852821+0000","last_became_peered":"2024-09-10T19:16:24.852821+0000","last_unstale":"2024-09-10T19:18:06.909525+0000","last_undegraded":"2024-09-10T19:18:06.909525+0000","last_fullsized":"2024-09-10T19:18:06.909525+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'14","last_scrub_stamp":"2024-09-10T19:15:41.748101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:15:41.748101+0000","objects_scrubbed":7,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0055403099999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,7],"acting":[4,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":184,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-09-10T19:18:06.920423+0000","last_change":"2024-09-10T19:16:25.582072+0000","last_active":"2024-09-10T19:18:06.920423+0000","last_peered":"2024-09-10T19:18:06.920423+0000","last_clean":"2024-09-10T19:18:06.920423+0000","last_became_active":"2024-09-10T19:16:24.855861+0000","last_became_peered":"2024-09-10T19:16:24.855861+0000","last_unstale":"2024-09-10T19:18:06.920423+0000","last_undegraded":"2024-09-10T19:18:06.920423+0000","last_fullsized":"2024-09-10T19:18:06.920423+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:16:25.581993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:16:25.581993+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0041345990000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.0","version":"117'10","reported_seq":287,"reported_epoch":194,"state":"active+clean","last_fresh":"2024-09-10T19:18:08.938125+0000","last_change":"2024-09-10T19:15:35.973188+0000","last_active":"2024-09-10T19:18:08.938125+0000","last_peered":"2024-09-10T19:18:08.938125+0000","last_clean":"2024-09-10T19:18:08.938125+0000","last_became_active":"2024-09-10T19:13:54.709846+0000","last_became_peered":"2024-09-10T19:13:54.709846+0000","last_unstale":"2024-09-10T19:18:08.938125+0000","last_undegraded":"2024-09-10T19:18:08.938125+0000","last_fullsized":"2024-09-10T19:18:08.938125+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"117'10","last_scrub_stamp":"2024-09-10T19:15:35.973095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:15:35.973095+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0076338860000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,0],"acting":[3,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":168,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-09-10T19:18:06.931494+0000","last_change":"2024-09-10T19:16:24.858926+0000","last_active":"2024-09-10T19:18:06.931494+0000","last_peered":"2024-09-10T19:18:06.931494+0000","last_clean":"2024-09-10T19:18:06.931494+0000","last_became_active":"2024-09-10T19:16:24.858619+0000","last_became_peered":"2024-09-10T19:16:24.858619+0000","last_unstale":"2024-09-10T19:18:06.931494+0000","last_undegraded":"2024-09-10T19:18:06.931494+0000","last_fullsized":"2024-09-10T19:18:06.931494+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:13:15.350720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:11:41.073833+0000","last_clean_scrub_stamp":"2024-09-10T19:13:15.350720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0014361000000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.c","version":"194'9003","reported_seq":9059,"reported_epoch":194,"state":"active+clean","last_fresh":"2024-09-10T19:18:15.009803+0000","last_change":"2024-09-10T19:17:29.176247+0000","last_active":"2024-09-10T19:18:15.009803+0000","last_peered":"2024-09-10T19:18:15.009803+0000","last_clean":"2024-09-10T19:18:15.009803+0000","last_became_active":"2024-09-10T19:16:13.079527+0000","last_became_peered":"2024-09-10T19:16:13.079527+0000","last_unstale":"2024-09-10T19:18:15.009803+0000","last_undegraded":"2024-09-10T19:18:15.009803+0000","last_fullsized":"2024-09-10T19:18:15.009803+0000","mapping_epoch":148,"log_start":"173'5173","ondisk_log_start":"173'5173","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:16:11.777014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:16:11.777014+0000","last_clean_scrub_stamp":"2024-09-10T19:16:11.777014+0000","objects_scrubbed":0,"log_size":3830,"log_dups_size":0,"ondisk_log_size":3830,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":400752640,"num_objects":6115,"num_object_clones":0,"num_object_copies":18345,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6115,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9003,"num_write_kb":483776,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.0","version":"66'18","reported_seq":194,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-09-10T19:18:06.906866+0000","last_change":"2024-09-10T19:12:49.699111+0000","last_active":"2024-09-10T19:18:06.906866+0000","last_peered":"2024-09-10T19:18:06.906866+0000","last_clean":"2024-09-10T19:18:06.906866+0000","last_became_active":"2024-09-10T19:10:09.821836+0000","last_became_peered":"2024-09-10T19:10:09.821836+0000","last_unstale":"2024-09-10T19:18:06.906866+0000","last_undegraded":"2024-09-10T19:18:06.906866+0000","last_fullsized":"2024-09-10T19:18:06.906866+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'18","last_scrub_stamp":"2024-09-10T19:12:49.699054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:12:49.699054+0000","objects_scrubbed":8,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0073070890000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":134,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-09-10T19:18:06.920523+0000","last_change":"2024-09-10T19:17:26.374435+0000","last_active":"2024-09-10T19:18:06.920523+0000","last_peered":"2024-09-10T19:18:06.920523+0000","last_clean":"2024-09-10T19:18:06.920523+0000","last_became_active":"2024-09-10T19:13:54.710968+0000","last_became_peered":"2024-09-10T19:13:54.710968+0000","last_unstale":"2024-09-10T19:18:06.920523+0000","last_undegraded":"2024-09-10T19:18:06.920523+0000","last_fullsized":"2024-09-10T19:18:06.920523+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:17:26.374376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:17:26.374376+0000","last_clean_scrub_stamp":"2024-09-10T19:17:26.374376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:18:32.655180+0000","scrub_duration":0.001299815,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,3],"acting":[6,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":169,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-09-10T19:18:06.909767+0000","last_change":"2024-09-10T19:16:30.991715+0000","last_active":"2024-09-10T19:18:06.909767+0000","last_peered":"2024-09-10T19:18:06.909767+0000","last_clean":"2024-09-10T19:18:06.909767+0000","last_became_active":"2024-09-10T19:10:11.698054+0000","last_became_peered":"2024-09-10T19:10:11.698054+0000","last_unstale":"2024-09-10T19:18:06.909767+0000","last_undegraded":"2024-09-10T19:18:06.909767+0000","last_fullsized":"2024-09-10T19:18:06.909767+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:16:30.991635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:16:30.991635+0000","last_clean_scrub_stamp":"2024-09-10T19:16:30.991635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0021810029999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":177,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-09-10T19:18:06.911408+0000","last_change":"2024-09-10T19:16:30.036748+0000","last_active":"2024-09-10T19:18:06.911408+0000","last_peered":"2024-09-10T19:18:06.911408+0000","last_clean":"2024-09-10T19:18:06.911408+0000","last_became_active":"2024-09-10T19:16:24.848109+0000","last_became_peered":"2024-09-10T19:16:24.848109+0000","last_unstale":"2024-09-10T19:18:06.911408+0000","last_undegraded":"2024-09-10T19:18:06.911408+0000","last_fullsized":"2024-09-10T19:18:06.911408+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:16:30.036662+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:16:30.036662+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0044984129999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.f","version":"194'8777","reported_seq":8841,"reported_epoch":194,"state":"active+clean","last_fresh":"2024-09-10T19:18:14.976726+0000","last_change":"2024-09-10T19:16:24.856828+0000","last_active":"2024-09-10T19:18:14.976726+0000","last_peered":"2024-09-10T19:18:14.976726+0000","last_clean":"2024-09-10T19:18:14.976726+0000","last_became_active":"2024-09-10T19:16:24.855393+0000","last_became_peered":"2024-09-10T19:16:24.855393+0000","last_unstale":"2024-09-10T19:18:14.976726+0000","last_undegraded":"2024-09-10T19:18:14.976726+0000","last_fullsized":"2024-09-10T19:18:14.976726+0000","mapping_epoch":154,"log_start":"173'5199","ondisk_log_start":"173'5199","created":148,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:16:11.777014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:16:11.777014+0000","last_clean_scrub_stamp":"2024-09-10T19:16:11.777014+0000","objects_scrubbed":0,"log_size":3578,"log_dups_size":0,"ondisk_log_size":3578,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":391577600,"num_objects":5975,"num_object_clones":0,"num_object_copies":17925,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5975,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8777,"num_write_kb":472064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.3","version":"104'27","reported_seq":215,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-09-10T19:18:06.910559+0000","last_change":"2024-09-10T19:16:35.246566+0000","last_active":"2024-09-10T19:18:06.910559+0000","last_peered":"2024-09-10T19:18:06.910559+0000","last_clean":"2024-09-10T19:18:06.910559+0000","last_became_active":"2024-09-10T19:10:09.823003+0000","last_became_peered":"2024-09-10T19:10:09.823003+0000","last_unstale":"2024-09-10T19:18:06.910559+0000","last_undegraded":"2024-09-10T19:18:06.910559+0000","last_fullsized":"2024-09-10T19:18:06.910559+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"104'27","last_scrub_stamp":"2024-09-10T19:16:35.246512+0000","last_deep_scrub":"104'27","last_deep_scrub_stamp":"2024-09-10T19:13:02.225944+0000","last_clean_scrub_stamp":"2024-09-10T19:16:35.246512+0000","objects_scrubbed":8,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0052296490000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":174,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-09-10T19:18:06.914087+0000","last_change":"2024-09-10T19:16:37.780086+0000","last_active":"2024-09-10T19:18:06.914087+0000","last_peered":"2024-09-10T19:18:06.914087+0000","last_clean":"2024-09-10T19:18:06.914087+0000","last_became_active":"2024-09-10T19:10:07.690325+0000","last_became_peered":"2024-09-10T19:10:07.690325+0000","last_unstale":"2024-09-10T19:18:06.914087+0000","last_undegraded":"2024-09-10T19:18:06.914087+0000","last_fullsized":"2024-09-10T19:18:06.914087+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:16:37.779967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:16:37.779967+0000","last_clean_scrub_stamp":"2024-09-10T19:16:37.779967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.067110102000000005,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":173,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-09-10T19:18:06.919149+0000","last_change":"2024-09-10T19:17:19.463698+0000","last_active":"2024-09-10T19:18:06.919149+0000","last_peered":"2024-09-10T19:18:06.919149+0000","last_clean":"2024-09-10T19:18:06.919149+0000","last_became_active":"2024-09-10T19:10:11.700370+0000","last_became_peered":"2024-09-10T19:10:11.700370+0000","last_unstale":"2024-09-10T19:18:06.919149+0000","last_undegraded":"2024-09-10T19:18:06.919149+0000","last_fullsized":"2024-09-10T19:18:06.919149+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:15:50.669821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:15:50.669821+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0015765,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":137,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-09-10T19:18:06.919844+0000","last_change":"2024-09-10T19:17:22.435156+0000","last_active":"2024-09-10T19:18:06.919844+0000","last_peered":"2024-09-10T19:18:06.919844+0000","last_clean":"2024-09-10T19:18:06.919844+0000","last_became_active":"2024-09-10T19:16:24.844699+0000","last_became_peered":"2024-09-10T19:16:24.844699+0000","last_unstale":"2024-09-10T19:18:06.919844+0000","last_undegraded":"2024-09-10T19:18:06.919844+0000","last_fullsized":"2024-09-10T19:18:06.919844+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:15:41.749338+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:15:41.749338+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0018841260000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.e","version":"194'8869","reported_seq":8932,"reported_epoch":194,"state":"active+clean","last_fresh":"2024-09-10T19:18:15.001895+0000","last_change":"2024-09-10T19:16:24.837044+0000","last_active":"2024-09-10T19:18:15.001895+0000","last_peered":"2024-09-10T19:18:15.001895+0000","last_clean":"2024-09-10T19:18:15.001895+0000","last_became_active":"2024-09-10T19:16:24.836458+0000","last_became_peered":"2024-09-10T19:16:24.836458+0000","last_unstale":"2024-09-10T19:18:15.001895+0000","last_undegraded":"2024-09-10T19:18:15.001895+0000","last_fullsized":"2024-09-10T19:18:15.001895+0000","mapping_epoch":154,"log_start":"172'4984","ondisk_log_start":"172'4984","created":148,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:16:11.777014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:16:11.777014+0000","last_clean_scrub_stamp":"2024-09-10T19:16:11.777014+0000","objects_scrubbed":0,"log_size":3885,"log_dups_size":0,"ondisk_log_size":3885,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":392888320,"num_objects":5995,"num_object_clones":0,"num_object_copies":17985,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5995,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8869,"num_write_kb":475648,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"66'10","reported_seq":130,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-09-10T19:18:06.911388+0000","last_change":"2024-09-10T19:13:54.748881+0000","last_active":"2024-09-10T19:18:06.911388+0000","last_peered":"2024-09-10T19:18:06.911388+0000","last_clean":"2024-09-10T19:18:06.911388+0000","last_became_active":"2024-09-10T19:13:54.714542+0000","last_became_peered":"2024-09-10T19:13:54.714542+0000","last_unstale":"2024-09-10T19:18:06.911388+0000","last_undegraded":"2024-09-10T19:18:06.911388+0000","last_fullsized":"2024-09-10T19:18:06.911388+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'10","last_scrub_stamp":"2024-09-10T19:12:56.360409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:12:56.360409+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":14,"num_read_kb":9,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,0],"acting":[5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":143,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-09-10T19:18:06.910910+0000","last_change":"2024-09-10T19:17:28.203449+0000","last_active":"2024-09-10T19:18:06.910910+0000","last_peered":"2024-09-10T19:18:06.910910+0000","last_clean":"2024-09-10T19:18:06.910910+0000","last_became_active":"2024-09-10T19:16:24.836898+0000","last_became_peered":"2024-09-10T19:16:24.836898+0000","last_unstale":"2024-09-10T19:18:06.910910+0000","last_undegraded":"2024-09-10T19:18:06.910910+0000","last_fullsized":"2024-09-10T19:18:06.910910+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:17:28.203397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:17:28.203397+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:18:28.259497+0000","scrub_duration":0.0011258259999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3","version":"84'9","reported_seq":271,"reported_epoch":194,"state":"active+clean","last_fresh":"2024-09-10T19:18:14.048096+0000","last_change":"2024-09-10T19:18:14.048096+0000","last_active":"2024-09-10T19:18:14.048096+0000","last_peered":"2024-09-10T19:18:14.048096+0000","last_clean":"2024-09-10T19:18:14.048096+0000","last_became_active":"2024-09-10T19:10:11.699690+0000","last_became_peered":"2024-09-10T19:10:11.699690+0000","last_unstale":"2024-09-10T19:18:14.048096+0000","last_undegraded":"2024-09-10T19:18:14.048096+0000","last_fullsized":"2024-09-10T19:18:14.048096+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'9","last_scrub_stamp":"2024-09-10T19:13:45.236094+0000","last_deep_scrub":"84'9","last_deep_scrub_stamp":"2024-09-10T19:11:53.879272+0000","last_clean_scrub_stamp":"2024-09-10T19:13:45.236094+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.036165602999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":173,"reported_epoch":194,"state":"active+clean","last_fresh":"2024-09-10T19:18:09.020077+0000","last_change":"2024-09-10T19:18:09.020077+0000","last_active":"2024-09-10T19:18:09.020077+0000","last_peered":"2024-09-10T19:18:09.020077+0000","last_clean":"2024-09-10T19:18:09.020077+0000","last_became_active":"2024-09-10T19:16:24.840096+0000","last_became_peered":"2024-09-10T19:16:24.840096+0000","last_unstale":"2024-09-10T19:18:09.020077+0000","last_undegraded":"2024-09-10T19:18:09.020077+0000","last_fullsized":"2024-09-10T19:18:09.020077+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:13:40.234289+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:13:40.234289+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0086405590000000008,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,5],"acting":[0,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.8","version":"194'7065","reported_seq":7130,"reported_epoch":194,"state":"active+clean","last_fresh":"2024-09-10T19:18:14.829117+0000","last_change":"2024-09-10T19:17:26.958494+0000","last_active":"2024-09-10T19:18:14.829117+0000","last_peered":"2024-09-10T19:18:14.829117+0000","last_clean":"2024-09-10T19:18:14.829117+0000","last_became_active":"2024-09-10T19:17:26.955617+0000","last_became_peered":"2024-09-10T19:17:26.955617+0000","last_unstale":"2024-09-10T19:18:14.829117+0000","last_undegraded":"2024-09-10T19:18:14.829117+0000","last_fullsized":"2024-09-10T19:18:14.829117+0000","mapping_epoch":148,"log_start":"159'1400","ondisk_log_start":"159'1400","created":148,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:16:11.777014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:16:11.777014+0000","last_clean_scrub_stamp":"2024-09-10T19:16:11.777014+0000","objects_scrubbed":0,"log_size":5665,"log_dups_size":0,"ondisk_log_size":5665,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":196575232,"num_objects":2999,"num_object_clones":0,"num_object_copies":8997,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2999,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4249,"num_write_kb":231968,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"155'31","reported_seq":157,"reported_epoch":194,"state":"active+clean","last_fresh":"2024-09-10T19:18:08.938349+0000","last_change":"2024-09-10T19:16:24.910865+0000","last_active":"2024-09-10T19:18:08.938349+0000","last_peered":"2024-09-10T19:18:08.938349+0000","last_clean":"2024-09-10T19:18:08.938349+0000","last_became_active":"2024-09-10T19:16:24.858154+0000","last_became_peered":"2024-09-10T19:16:24.858154+0000","last_unstale":"2024-09-10T19:18:08.938349+0000","last_undegraded":"2024-09-10T19:18:08.938349+0000","last_fullsized":"2024-09-10T19:18:08.938349+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"117'30","last_scrub_stamp":"2024-09-10T19:15:49.837844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:15:49.837844+0000","objects_scrubbed":0,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":47,"num_read_kb":32,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":358,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":181,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-09-10T19:18:06.911138+0000","last_change":"2024-09-10T19:15:54.746979+0000","last_active":"2024-09-10T19:18:06.911138+0000","last_peered":"2024-09-10T19:18:06.911138+0000","last_clean":"2024-09-10T19:18:06.911138+0000","last_became_active":"2024-09-10T19:13:54.730096+0000","last_became_peered":"2024-09-10T19:13:54.730096+0000","last_unstale":"2024-09-10T19:18:06.911138+0000","last_undegraded":"2024-09-10T19:18:06.911138+0000","last_fullsized":"2024-09-10T19:18:06.911138+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:15:54.746920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:15:54.746920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001300037,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":186,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-09-10T19:18:06.912540+0000","last_change":"2024-09-10T19:17:25.147473+0000","last_active":"2024-09-10T19:18:06.912540+0000","last_peered":"2024-09-10T19:18:06.912540+0000","last_clean":"2024-09-10T19:18:06.912540+0000","last_became_active":"2024-09-10T19:13:54.714017+0000","last_became_peered":"2024-09-10T19:13:54.714017+0000","last_unstale":"2024-09-10T19:18:06.912540+0000","last_undegraded":"2024-09-10T19:18:06.912540+0000","last_fullsized":"2024-09-10T19:18:06.912540+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:17:25.147396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:17:25.147396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:18:29.638750+0000","scrub_duration":0.004390054,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,6],"acting":[0,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":158,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-09-10T19:18:06.906688+0000","last_change":"2024-09-10T19:16:32.818668+0000","last_active":"2024-09-10T19:18:06.906688+0000","last_peered":"2024-09-10T19:18:06.906688+0000","last_clean":"2024-09-10T19:18:06.906688+0000","last_became_active":"2024-09-10T19:10:13.721744+0000","last_became_peered":"2024-09-10T19:10:13.721744+0000","last_unstale":"2024-09-10T19:18:06.906688+0000","last_undegraded":"2024-09-10T19:18:06.906688+0000","last_fullsized":"2024-09-10T19:18:06.906688+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:16:32.818590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:16:32.818590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0047149059999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.b","version":"194'9003","reported_seq":9067,"reported_epoch":194,"state":"active+clean","last_fresh":"2024-09-10T19:18:14.922572+0000","last_change":"2024-09-10T19:16:24.852409+0000","last_active":"2024-09-10T19:18:14.922572+0000","last_peered":"2024-09-10T19:18:14.922572+0000","last_clean":"2024-09-10T19:18:14.922572+0000","last_became_active":"2024-09-10T19:16:24.850682+0000","last_became_peered":"2024-09-10T19:16:24.850682+0000","last_unstale":"2024-09-10T19:18:14.922572+0000","last_undegraded":"2024-09-10T19:18:14.922572+0000","last_fullsized":"2024-09-10T19:18:14.922572+0000","mapping_epoch":154,"log_start":"173'5388","ondisk_log_start":"173'5388","created":148,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:16:11.777014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:16:11.777014+0000","last_clean_scrub_stamp":"2024-09-10T19:16:11.777014+0000","objects_scrubbed":0,"log_size":3615,"log_dups_size":0,"ondisk_log_size":3615,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":397213696,"num_objects":6061,"num_object_clones":0,"num_object_copies":18183,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6061,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9003,"num_write_kb":482048,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.7","version":"82'18","reported_seq":197,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-09-10T19:18:06.911140+0000","last_change":"2024-09-10T19:15:55.777428+0000","last_active":"2024-09-10T19:18:06.911140+0000","last_peered":"2024-09-10T19:18:06.911140+0000","last_clean":"2024-09-10T19:18:06.911140+0000","last_became_active":"2024-09-10T19:13:54.712065+0000","last_became_peered":"2024-09-10T19:13:54.712065+0000","last_unstale":"2024-09-10T19:18:06.911140+0000","last_undegraded":"2024-09-10T19:18:06.911140+0000","last_fullsized":"2024-09-10T19:18:06.911140+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'18","last_scrub_stamp":"2024-09-10T19:15:55.777361+0000","last_deep_scrub":"82'18","last_deep_scrub_stamp":"2024-09-10T19:15:55.777361+0000","last_clean_scrub_stamp":"2024-09-10T19:15:55.777361+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0023560040000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":19,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.0","version":"159'6","reported_seq":205,"reported_epoch":194,"state":"active+clean","last_fresh":"2024-09-10T19:18:13.075594+0000","last_change":"2024-09-10T19:16:32.650007+0000","last_active":"2024-09-10T19:18:13.075594+0000","last_peered":"2024-09-10T19:18:13.075594+0000","last_clean":"2024-09-10T19:18:13.075594+0000","last_became_active":"2024-09-10T19:16:24.846203+0000","last_became_peered":"2024-09-10T19:16:24.846203+0000","last_unstale":"2024-09-10T19:18:13.075594+0000","last_undegraded":"2024-09-10T19:18:13.075594+0000","last_fullsized":"2024-09-10T19:18:13.075594+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"159'6","last_scrub_stamp":"2024-09-10T19:16:32.649914+0000","last_deep_scrub":"159'6","last_deep_scrub_stamp":"2024-09-10T19:16:32.649914+0000","last_clean_scrub_stamp":"2024-09-10T19:16:32.649914+0000","objects_scrubbed":1,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.013939817,"objects_trimmed":0,"snaptrim_duration":0.00034454499999999997,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":116,"num_read_kb":111,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":389,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":62,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-09-10T19:18:06.906592+0000","last_change":"2024-09-10T19:16:33.811962+0000","last_active":"2024-09-10T19:18:06.906592+0000","last_peered":"2024-09-10T19:18:06.906592+0000","last_clean":"2024-09-10T19:18:06.906592+0000","last_became_active":"2024-09-10T19:16:24.857505+0000","last_became_peered":"2024-09-10T19:16:24.857505+0000","last_unstale":"2024-09-10T19:18:06.906592+0000","last_undegraded":"2024-09-10T19:18:06.906592+0000","last_fullsized":"2024-09-10T19:18:06.906592+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:16:33.811869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:16:33.811869+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0027920340000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":62,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-09-10T19:18:06.911636+0000","last_change":"2024-09-10T19:16:43.958311+0000","last_active":"2024-09-10T19:18:06.911636+0000","last_peered":"2024-09-10T19:18:06.911636+0000","last_clean":"2024-09-10T19:18:06.911636+0000","last_became_active":"2024-09-10T19:16:24.861481+0000","last_became_peered":"2024-09-10T19:16:24.861481+0000","last_unstale":"2024-09-10T19:18:06.911636+0000","last_undegraded":"2024-09-10T19:18:06.911636+0000","last_fullsized":"2024-09-10T19:18:06.911636+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:16:43.958224+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:16:43.958224+0000","last_clean_scrub_stamp":"2024-09-10T19:16:43.958224+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:18:09.697349+0000","scrub_duration":0.015970275999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,4],"acting":[5,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":161,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-09-10T19:18:06.913622+0000","last_change":"2024-09-10T19:15:38.493179+0000","last_active":"2024-09-10T19:18:06.913622+0000","last_peered":"2024-09-10T19:18:06.913622+0000","last_clean":"2024-09-10T19:18:06.913622+0000","last_became_active":"2024-09-10T19:10:14.053697+0000","last_became_peered":"2024-09-10T19:10:14.053697+0000","last_unstale":"2024-09-10T19:18:06.913622+0000","last_undegraded":"2024-09-10T19:18:06.913622+0000","last_fullsized":"2024-09-10T19:18:06.913622+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:15:38.493102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:15:38.493102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0015544280000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.a","version":"194'9125","reported_seq":9177,"reported_epoch":194,"state":"active+clean","last_fresh":"2024-09-10T19:18:14.846059+0000","last_change":"2024-09-10T19:16:24.865995+0000","last_active":"2024-09-10T19:18:14.846059+0000","last_peered":"2024-09-10T19:18:14.846059+0000","last_clean":"2024-09-10T19:18:14.846059+0000","last_became_active":"2024-09-10T19:16:24.864959+0000","last_became_peered":"2024-09-10T19:16:24.864959+0000","last_unstale":"2024-09-10T19:18:14.846059+0000","last_undegraded":"2024-09-10T19:18:14.846059+0000","last_fullsized":"2024-09-10T19:18:14.846059+0000","mapping_epoch":154,"log_start":"173'5325","ondisk_log_start":"173'5325","created":148,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:16:11.777014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:16:11.777014+0000","last_clean_scrub_stamp":"2024-09-10T19:16:11.777014+0000","objects_scrubbed":0,"log_size":3800,"log_dups_size":0,"ondisk_log_size":3800,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":410583040,"num_objects":6265,"num_object_clones":0,"num_object_copies":18795,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6265,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9125,"num_write_kb":492480,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.6","version":"66'12","reported_seq":237,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-09-10T19:18:06.913346+0000","last_change":"2024-09-10T19:17:19.243613+0000","last_active":"2024-09-10T19:18:06.913346+0000","last_peered":"2024-09-10T19:18:06.913346+0000","last_clean":"2024-09-10T19:18:06.913346+0000","last_became_active":"2024-09-10T19:16:24.844323+0000","last_became_peered":"2024-09-10T19:16:24.844323+0000","last_unstale":"2024-09-10T19:18:06.913346+0000","last_undegraded":"2024-09-10T19:18:06.913346+0000","last_fullsized":"2024-09-10T19:18:06.913346+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'12","last_scrub_stamp":"2024-09-10T19:15:40.903279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:15:40.903279+0000","objects_scrubbed":6,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0271238,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,5],"acting":[0,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"61'1","reported_seq":179,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-09-10T19:18:06.911334+0000","last_change":"2024-09-10T19:16:42.195540+0000","last_active":"2024-09-10T19:18:06.911334+0000","last_peered":"2024-09-10T19:18:06.911334+0000","last_clean":"2024-09-10T19:18:06.911334+0000","last_became_active":"2024-09-10T19:10:07.683259+0000","last_became_peered":"2024-09-10T19:10:07.683259+0000","last_unstale":"2024-09-10T19:18:06.911334+0000","last_undegraded":"2024-09-10T19:18:06.911334+0000","last_fullsized":"2024-09-10T19:18:06.911334+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'1","last_scrub_stamp":"2024-09-10T19:16:42.195463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:16:42.195463+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0016622189999999999,"objects_trimmed":0,"snaptrim_duration":0.0014341130000000001,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":181,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-09-10T19:18:06.913057+0000","last_change":"2024-09-10T19:15:30.542477+0000","last_active":"2024-09-10T19:18:06.913057+0000","last_peered":"2024-09-10T19:18:06.913057+0000","last_clean":"2024-09-10T19:18:06.913057+0000","last_became_active":"2024-09-10T19:13:54.721486+0000","last_became_peered":"2024-09-10T19:13:54.721486+0000","last_unstale":"2024-09-10T19:18:06.913057+0000","last_undegraded":"2024-09-10T19:18:06.913057+0000","last_fullsized":"2024-09-10T19:18:06.913057+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:15:30.542399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:15:30.542399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0025142609999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":187,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-09-10T19:18:06.911060+0000","last_change":"2024-09-10T19:17:28.070635+0000","last_active":"2024-09-10T19:18:06.911060+0000","last_peered":"2024-09-10T19:18:06.911060+0000","last_clean":"2024-09-10T19:18:06.911060+0000","last_became_active":"2024-09-10T19:13:54.711685+0000","last_became_peered":"2024-09-10T19:13:54.711685+0000","last_unstale":"2024-09-10T19:18:06.911060+0000","last_undegraded":"2024-09-10T19:18:06.911060+0000","last_fullsized":"2024-09-10T19:18:06.911060+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:17:28.070543+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:17:28.070543+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:18:49.229793+0000","scrub_duration":0.0017508000000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,4],"acting":[5,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":124,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-09-10T19:18:06.913495+0000","last_change":"2024-09-10T19:16:28.786490+0000","last_active":"2024-09-10T19:18:06.913495+0000","last_peered":"2024-09-10T19:18:06.913495+0000","last_clean":"2024-09-10T19:18:06.913495+0000","last_became_active":"2024-09-10T19:13:54.722937+0000","last_became_peered":"2024-09-10T19:13:54.722937+0000","last_unstale":"2024-09-10T19:18:06.913495+0000","last_undegraded":"2024-09-10T19:18:06.913495+0000","last_fullsized":"2024-09-10T19:18:06.913495+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:16:28.786056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:16:28.786056+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0082035739999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.9","version":"194'7361","reported_seq":7426,"reported_epoch":194,"state":"active+clean","last_fresh":"2024-09-10T19:18:14.725036+0000","last_change":"2024-09-10T19:17:26.964413+0000","last_active":"2024-09-10T19:18:14.725036+0000","last_peered":"2024-09-10T19:18:14.725036+0000","last_clean":"2024-09-10T19:18:14.725036+0000","last_became_active":"2024-09-10T19:17:26.961919+0000","last_became_peered":"2024-09-10T19:17:26.961919+0000","last_unstale":"2024-09-10T19:18:14.725036+0000","last_undegraded":"2024-09-10T19:18:14.725036+0000","last_fullsized":"2024-09-10T19:18:14.725036+0000","mapping_epoch":148,"log_start":"160'1700","ondisk_log_start":"160'1700","created":148,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:16:11.777014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:16:11.777014+0000","last_clean_scrub_stamp":"2024-09-10T19:16:11.777014+0000","objects_scrubbed":0,"log_size":5661,"log_dups_size":0,"ondisk_log_size":5661,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":202833920,"num_objects":3095,"num_object_clones":0,"num_object_copies":9285,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3095,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4469,"num_write_kb":242048,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,0],"acting":[6,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"80'21","reported_seq":211,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-09-10T19:18:06.920084+0000","last_change":"2024-09-10T19:17:27.420321+0000","last_active":"2024-09-10T19:18:06.920084+0000","last_peered":"2024-09-10T19:18:06.920084+0000","last_clean":"2024-09-10T19:18:06.920084+0000","last_became_active":"2024-09-10T19:10:09.822184+0000","last_became_peered":"2024-09-10T19:10:09.822184+0000","last_unstale":"2024-09-10T19:18:06.920084+0000","last_undegraded":"2024-09-10T19:18:06.920084+0000","last_fullsized":"2024-09-10T19:18:06.920084+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"80'21","last_scrub_stamp":"2024-09-10T19:17:27.420267+0000","last_deep_scrub":"80'21","last_deep_scrub_stamp":"2024-09-10T19:11:39.836144+0000","last_clean_scrub_stamp":"2024-09-10T19:17:27.420267+0000","objects_scrubbed":7,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:18:35.485621+0000","scrub_duration":0.0018427859999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"63'2","reported_seq":181,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-09-10T19:18:06.906668+0000","last_change":"2024-09-10T19:16:35.962952+0000","last_active":"2024-09-10T19:18:06.906668+0000","last_peered":"2024-09-10T19:18:06.906668+0000","last_clean":"2024-09-10T19:18:06.906668+0000","last_became_active":"2024-09-10T19:10:07.684784+0000","last_became_peered":"2024-09-10T19:10:07.684784+0000","last_unstale":"2024-09-10T19:18:06.906668+0000","last_undegraded":"2024-09-10T19:18:06.906668+0000","last_fullsized":"2024-09-10T19:18:06.906668+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'2","last_scrub_stamp":"2024-09-10T19:16:35.962804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:16:35.962804+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.105395924,"objects_trimmed":0,"snaptrim_duration":0.00075522999999999996,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":184,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-09-10T19:18:06.914241+0000","last_change":"2024-09-10T19:16:24.858963+0000","last_active":"2024-09-10T19:18:06.914241+0000","last_peered":"2024-09-10T19:18:06.914241+0000","last_clean":"2024-09-10T19:18:06.914241+0000","last_became_active":"2024-09-10T19:16:24.858662+0000","last_became_peered":"2024-09-10T19:16:24.858662+0000","last_unstale":"2024-09-10T19:18:06.914241+0000","last_undegraded":"2024-09-10T19:18:06.914241+0000","last_fullsized":"2024-09-10T19:18:06.914241+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:13:50.411500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:13:50.411500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0032604869999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"74'79","reported_seq":209,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-09-10T19:18:06.913201+0000","last_change":"2024-09-10T19:13:46.419002+0000","last_active":"2024-09-10T19:18:06.913201+0000","last_peered":"2024-09-10T19:18:06.913201+0000","last_clean":"2024-09-10T19:18:06.913201+0000","last_became_active":"2024-09-10T19:09:56.634919+0000","last_became_peered":"2024-09-10T19:09:56.634919+0000","last_unstale":"2024-09-10T19:18:06.913201+0000","last_undegraded":"2024-09-10T19:18:06.913201+0000","last_fullsized":"2024-09-10T19:18:06.913201+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'79","last_scrub_stamp":"2024-09-10T19:13:46.418921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:08:00.699315+0000","last_clean_scrub_stamp":"2024-09-10T19:13:46.418921+0000","objects_scrubbed":2,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0029827999999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":170,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-09-10T19:18:06.912429+0000","last_change":"2024-09-10T19:16:24.852723+0000","last_active":"2024-09-10T19:18:06.912429+0000","last_peered":"2024-09-10T19:18:06.912429+0000","last_clean":"2024-09-10T19:18:06.912429+0000","last_became_active":"2024-09-10T19:16:24.851652+0000","last_became_peered":"2024-09-10T19:16:24.851652+0000","last_unstale":"2024-09-10T19:18:06.912429+0000","last_undegraded":"2024-09-10T19:18:06.912429+0000","last_fullsized":"2024-09-10T19:18:06.912429+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:13:47.453228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:13:47.453228+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.010282463,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,0],"acting":[7,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":158,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-09-10T19:18:06.911225+0000","last_change":"2024-09-10T19:13:05.390398+0000","last_active":"2024-09-10T19:18:06.911225+0000","last_peered":"2024-09-10T19:18:06.911225+0000","last_clean":"2024-09-10T19:18:06.911225+0000","last_became_active":"2024-09-10T19:10:13.720647+0000","last_became_peered":"2024-09-10T19:10:13.720647+0000","last_unstale":"2024-09-10T19:18:06.911225+0000","last_undegraded":"2024-09-10T19:18:06.911225+0000","last_fullsized":"2024-09-10T19:18:06.911225+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:13:05.390317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:13:05.390317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0058030640000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.6","version":"194'7382","reported_seq":7445,"reported_epoch":194,"state":"active+clean","last_fresh":"2024-09-10T19:18:14.979107+0000","last_change":"2024-09-10T19:17:26.955780+0000","last_active":"2024-09-10T19:18:14.979107+0000","last_peered":"2024-09-10T19:18:14.979107+0000","last_clean":"2024-09-10T19:18:14.979107+0000","last_became_active":"2024-09-10T19:17:26.953133+0000","last_became_peered":"2024-09-10T19:17:26.953133+0000","last_unstale":"2024-09-10T19:18:14.979107+0000","last_undegraded":"2024-09-10T19:18:14.979107+0000","last_fullsized":"2024-09-10T19:18:14.979107+0000","mapping_epoch":148,"log_start":"159'1500","ondisk_log_start":"159'1500","created":148,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:16:11.777014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:16:11.777014+0000","last_clean_scrub_stamp":"2024-09-10T19:16:11.777014+0000","objects_scrubbed":0,"log_size":5882,"log_dups_size":0,"ondisk_log_size":5882,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":201031680,"num_objects":3067,"num_object_clones":0,"num_object_copies":9201,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3067,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4509,"num_write_kb":242464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,7],"acting":[0,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"66'19","reported_seq":233,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-09-10T19:18:06.919408+0000","last_change":"2024-09-10T19:17:10.484589+0000","last_active":"2024-09-10T19:18:06.919408+0000","last_peered":"2024-09-10T19:18:06.919408+0000","last_clean":"2024-09-10T19:18:06.919408+0000","last_became_active":"2024-09-10T19:13:54.731274+0000","last_became_peered":"2024-09-10T19:13:54.731274+0000","last_unstale":"2024-09-10T19:18:06.919408+0000","last_undegraded":"2024-09-10T19:18:06.919408+0000","last_fullsized":"2024-09-10T19:18:06.919408+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'19","last_scrub_stamp":"2024-09-10T19:15:38.722581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:15:38.722581+0000","objects_scrubbed":9,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0054904940000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,5],"acting":[6,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":52,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-09-10T19:18:06.913666+0000","last_change":"2024-09-10T19:16:24.847051+0000","last_active":"2024-09-10T19:18:06.913666+0000","last_peered":"2024-09-10T19:18:06.913666+0000","last_clean":"2024-09-10T19:18:06.913666+0000","last_became_active":"2024-09-10T19:16:24.846745+0000","last_became_peered":"2024-09-10T19:16:24.846745+0000","last_unstale":"2024-09-10T19:18:06.913666+0000","last_undegraded":"2024-09-10T19:18:06.913666+0000","last_fullsized":"2024-09-10T19:18:06.913666+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:15:53.803057+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:15:53.803057+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3],"acting":[5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":61,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-09-10T19:18:06.910386+0000","last_change":"2024-09-10T19:16:27.240222+0000","last_active":"2024-09-10T19:18:06.910386+0000","last_peered":"2024-09-10T19:18:06.910386+0000","last_clean":"2024-09-10T19:18:06.910386+0000","last_became_active":"2024-09-10T19:16:24.844295+0000","last_became_peered":"2024-09-10T19:16:24.844295+0000","last_unstale":"2024-09-10T19:18:06.910386+0000","last_undegraded":"2024-09-10T19:18:06.910386+0000","last_fullsized":"2024-09-10T19:18:06.910386+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:16:27.240139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:16:27.240139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0020507199999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":168,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-09-10T19:18:06.913127+0000","last_change":"2024-09-10T19:16:24.858456+0000","last_active":"2024-09-10T19:18:06.913127+0000","last_peered":"2024-09-10T19:18:06.913127+0000","last_clean":"2024-09-10T19:18:06.913127+0000","last_became_active":"2024-09-10T19:16:24.858172+0000","last_became_peered":"2024-09-10T19:16:24.858172+0000","last_unstale":"2024-09-10T19:18:06.913127+0000","last_undegraded":"2024-09-10T19:18:06.913127+0000","last_fullsized":"2024-09-10T19:18:06.913127+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:13:14.311620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:13:14.311620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0054101059999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,6],"acting":[7,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.7","version":"194'7312","reported_seq":7386,"reported_epoch":194,"state":"active+clean","last_fresh":"2024-09-10T19:18:14.908863+0000","last_change":"2024-09-10T19:17:26.944772+0000","last_active":"2024-09-10T19:18:14.908863+0000","last_peered":"2024-09-10T19:18:14.908863+0000","last_clean":"2024-09-10T19:18:14.908863+0000","last_became_active":"2024-09-10T19:17:26.942858+0000","last_became_peered":"2024-09-10T19:17:26.942858+0000","last_unstale":"2024-09-10T19:18:14.908863+0000","last_undegraded":"2024-09-10T19:18:14.908863+0000","last_fullsized":"2024-09-10T19:18:14.908863+0000","mapping_epoch":154,"log_start":"160'1900","ondisk_log_start":"160'1900","created":148,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:16:11.777014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:16:11.777014+0000","last_clean_scrub_stamp":"2024-09-10T19:16:11.777014+0000","objects_scrubbed":0,"log_size":5412,"log_dups_size":0,"ondisk_log_size":5412,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":202080256,"num_objects":3083,"num_object_clones":0,"num_object_copies":9249,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3083,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4413,"num_write_kb":239904,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,4],"acting":[3,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.b","version":"66'9","reported_seq":207,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-09-10T19:18:06.910466+0000","last_change":"2024-09-10T19:17:20.191334+0000","last_active":"2024-09-10T19:18:06.910466+0000","last_peered":"2024-09-10T19:18:06.910466+0000","last_clean":"2024-09-10T19:18:06.910466+0000","last_became_active":"2024-09-10T19:13:54.710613+0000","last_became_peered":"2024-09-10T19:13:54.710613+0000","last_unstale":"2024-09-10T19:18:06.910466+0000","last_undegraded":"2024-09-10T19:18:06.910466+0000","last_fullsized":"2024-09-10T19:18:06.910466+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'9","last_scrub_stamp":"2024-09-10T19:15:37.846807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:15:37.846807+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.012189228,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":137,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-09-10T19:18:06.912879+0000","last_change":"2024-09-10T19:16:24.848979+0000","last_active":"2024-09-10T19:18:06.912879+0000","last_peered":"2024-09-10T19:18:06.912879+0000","last_clean":"2024-09-10T19:18:06.912879+0000","last_became_active":"2024-09-10T19:16:24.848668+0000","last_became_peered":"2024-09-10T19:16:24.848668+0000","last_unstale":"2024-09-10T19:18:06.912879+0000","last_undegraded":"2024-09-10T19:18:06.912879+0000","last_fullsized":"2024-09-10T19:18:06.912879+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:15:50.850576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:11:44.254513+0000","last_clean_scrub_stamp":"2024-09-10T19:15:50.850576+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001815419,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":55,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-09-10T19:18:06.911283+0000","last_change":"2024-09-10T19:17:09.811398+0000","last_active":"2024-09-10T19:18:06.911283+0000","last_peered":"2024-09-10T19:18:06.911283+0000","last_clean":"2024-09-10T19:18:06.911283+0000","last_became_active":"2024-09-10T19:16:24.845709+0000","last_became_peered":"2024-09-10T19:16:24.845709+0000","last_unstale":"2024-09-10T19:18:06.911283+0000","last_undegraded":"2024-09-10T19:18:06.911283+0000","last_fullsized":"2024-09-10T19:18:06.911283+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:13:08.398036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:13:08.398036+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":177,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-09-10T19:18:06.911264+0000","last_change":"2024-09-10T19:16:39.188737+0000","last_active":"2024-09-10T19:18:06.911264+0000","last_peered":"2024-09-10T19:18:06.911264+0000","last_clean":"2024-09-10T19:18:06.911264+0000","last_became_active":"2024-09-10T19:16:24.837483+0000","last_became_peered":"2024-09-10T19:16:24.837483+0000","last_unstale":"2024-09-10T19:18:06.911264+0000","last_undegraded":"2024-09-10T19:18:06.911264+0000","last_fullsized":"2024-09-10T19:18:06.911264+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:16:39.188682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:16:39.188682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0012633589999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.4","version":"194'7244","reported_seq":7322,"reported_epoch":194,"state":"active+clean","last_fresh":"2024-09-10T19:18:14.787955+0000","last_change":"2024-09-10T19:17:26.998151+0000","last_active":"2024-09-10T19:18:14.787955+0000","last_peered":"2024-09-10T19:18:14.787955+0000","last_clean":"2024-09-10T19:18:14.787955+0000","last_became_active":"2024-09-10T19:17:26.972029+0000","last_became_peered":"2024-09-10T19:17:26.972029+0000","last_unstale":"2024-09-10T19:18:14.787955+0000","last_undegraded":"2024-09-10T19:18:14.787955+0000","last_fullsized":"2024-09-10T19:18:14.787955+0000","mapping_epoch":154,"log_start":"159'1500","ondisk_log_start":"159'1500","created":148,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:16:11.777014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:16:11.777014+0000","last_clean_scrub_stamp":"2024-09-10T19:16:11.777014+0000","objects_scrubbed":0,"log_size":5744,"log_dups_size":0,"ondisk_log_size":5744,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":196968448,"num_objects":3005,"num_object_clones":0,"num_object_copies":9015,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3005,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4381,"num_write_kb":236384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.8","version":"66'15","reported_seq":187,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-09-10T19:18:06.914958+0000","last_change":"2024-09-10T19:12:50.449684+0000","last_active":"2024-09-10T19:18:06.914958+0000","last_peered":"2024-09-10T19:18:06.914958+0000","last_clean":"2024-09-10T19:18:06.914958+0000","last_became_active":"2024-09-10T19:10:09.820950+0000","last_became_peered":"2024-09-10T19:10:09.820950+0000","last_unstale":"2024-09-10T19:18:06.914958+0000","last_undegraded":"2024-09-10T19:18:06.914958+0000","last_fullsized":"2024-09-10T19:18:06.914958+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'15","last_scrub_stamp":"2024-09-10T19:12:50.449596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:12:50.449596+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0055729239999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":52,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-09-10T19:18:06.906858+0000","last_change":"2024-09-10T19:16:24.857533+0000","last_active":"2024-09-10T19:18:06.906858+0000","last_peered":"2024-09-10T19:18:06.906858+0000","last_clean":"2024-09-10T19:18:06.906858+0000","last_became_active":"2024-09-10T19:16:24.857253+0000","last_became_peered":"2024-09-10T19:16:24.857253+0000","last_unstale":"2024-09-10T19:18:06.906858+0000","last_undegraded":"2024-09-10T19:18:06.906858+0000","last_fullsized":"2024-09-10T19:18:06.906858+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:15:38.677960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:15:38.677960+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.9","version":"80'9","reported_seq":266,"reported_epoch":194,"state":"active+clean","last_fresh":"2024-09-10T19:18:08.949876+0000","last_change":"2024-09-10T19:13:43.358984+0000","last_active":"2024-09-10T19:18:08.949876+0000","last_peered":"2024-09-10T19:18:08.949876+0000","last_clean":"2024-09-10T19:18:08.949876+0000","last_became_active":"2024-09-10T19:10:11.702878+0000","last_became_peered":"2024-09-10T19:10:11.702878+0000","last_unstale":"2024-09-10T19:18:08.949876+0000","last_undegraded":"2024-09-10T19:18:08.949876+0000","last_fullsized":"2024-09-10T19:18:08.949876+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"80'9","last_scrub_stamp":"2024-09-10T19:13:43.358901+0000","last_deep_scrub":"80'9","last_deep_scrub_stamp":"2024-09-10T19:11:35.174218+0000","last_clean_scrub_stamp":"2024-09-10T19:13:43.358901+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0066143260000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":158,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-09-10T19:18:06.911655+0000","last_change":"2024-09-10T19:12:57.392898+0000","last_active":"2024-09-10T19:18:06.911655+0000","last_peered":"2024-09-10T19:18:06.911655+0000","last_clean":"2024-09-10T19:18:06.911655+0000","last_became_active":"2024-09-10T19:10:14.053875+0000","last_became_peered":"2024-09-10T19:10:14.053875+0000","last_unstale":"2024-09-10T19:18:06.911655+0000","last_undegraded":"2024-09-10T19:18:06.911655+0000","last_fullsized":"2024-09-10T19:18:06.911655+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:12:57.392844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:12:57.392844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0046180839999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.5","version":"194'7383","reported_seq":7447,"reported_epoch":194,"state":"active+clean","last_fresh":"2024-09-10T19:18:14.748181+0000","last_change":"2024-09-10T19:17:26.973165+0000","last_active":"2024-09-10T19:18:14.748181+0000","last_peered":"2024-09-10T19:18:14.748181+0000","last_clean":"2024-09-10T19:18:14.748181+0000","last_became_active":"2024-09-10T19:17:26.966286+0000","last_became_peered":"2024-09-10T19:17:26.966286+0000","last_unstale":"2024-09-10T19:18:14.748181+0000","last_undegraded":"2024-09-10T19:18:14.748181+0000","last_fullsized":"2024-09-10T19:18:14.748181+0000","mapping_epoch":148,"log_start":"160'1800","ondisk_log_start":"160'1800","created":148,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:16:11.777014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:16:11.777014+0000","last_clean_scrub_stamp":"2024-09-10T19:16:11.777014+0000","objects_scrubbed":0,"log_size":5583,"log_dups_size":0,"ondisk_log_size":5583,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":204668928,"num_objects":3123,"num_object_clones":0,"num_object_copies":9369,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3123,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4459,"num_write_kb":242624,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.9","version":"66'12","reported_seq":216,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-09-10T19:18:06.909849+0000","last_change":"2024-09-10T19:15:37.869593+0000","last_active":"2024-09-10T19:18:06.909849+0000","last_peered":"2024-09-10T19:18:06.909849+0000","last_clean":"2024-09-10T19:18:06.909849+0000","last_became_active":"2024-09-10T19:13:54.718152+0000","last_became_peered":"2024-09-10T19:13:54.718152+0000","last_unstale":"2024-09-10T19:18:06.909849+0000","last_undegraded":"2024-09-10T19:18:06.909849+0000","last_fullsized":"2024-09-10T19:18:06.909849+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'12","last_scrub_stamp":"2024-09-10T19:15:37.869451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:15:37.869451+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0025441249999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.f","version":"59'2","reported_seq":189,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-09-10T19:18:06.912203+0000","last_change":"2024-09-10T19:16:29.007627+0000","last_active":"2024-09-10T19:18:06.912203+0000","last_peered":"2024-09-10T19:18:06.912203+0000","last_clean":"2024-09-10T19:18:06.912203+0000","last_became_active":"2024-09-10T19:10:07.681719+0000","last_became_peered":"2024-09-10T19:10:07.681719+0000","last_unstale":"2024-09-10T19:18:06.912203+0000","last_undegraded":"2024-09-10T19:18:06.912203+0000","last_fullsized":"2024-09-10T19:18:06.912203+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'2","last_scrub_stamp":"2024-09-10T19:16:29.007251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:16:29.007251+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0079698319999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":14,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":134,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-09-10T19:18:06.910071+0000","last_change":"2024-09-10T19:17:23.177960+0000","last_active":"2024-09-10T19:18:06.910071+0000","last_peered":"2024-09-10T19:18:06.910071+0000","last_clean":"2024-09-10T19:18:06.910071+0000","last_became_active":"2024-09-10T19:16:24.840091+0000","last_became_peered":"2024-09-10T19:16:24.840091+0000","last_unstale":"2024-09-10T19:18:06.910071+0000","last_undegraded":"2024-09-10T19:18:06.910071+0000","last_fullsized":"2024-09-10T19:18:06.910071+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:15:31.776481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:11:46.587846+0000","last_clean_scrub_stamp":"2024-09-10T19:15:31.776481+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001267142,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,4],"acting":[0,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":175,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-09-10T19:18:06.907659+0000","last_change":"2024-09-10T19:15:41.093923+0000","last_active":"2024-09-10T19:18:06.907659+0000","last_peered":"2024-09-10T19:18:06.907659+0000","last_clean":"2024-09-10T19:18:06.907659+0000","last_became_active":"2024-09-10T19:13:54.711804+0000","last_became_peered":"2024-09-10T19:13:54.711804+0000","last_unstale":"2024-09-10T19:18:06.907659+0000","last_undegraded":"2024-09-10T19:18:06.907659+0000","last_fullsized":"2024-09-10T19:18:06.907659+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:15:41.093783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:15:41.093783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0023833050000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.16","version":"66'9","reported_seq":193,"reported_epoch":194,"state":"active+clean","last_fresh":"2024-09-10T19:18:11.047270+0000","last_change":"2024-09-10T19:18:11.047270+0000","last_active":"2024-09-10T19:18:11.047270+0000","last_peered":"2024-09-10T19:18:11.047270+0000","last_clean":"2024-09-10T19:18:11.047270+0000","last_became_active":"2024-09-10T19:10:09.822994+0000","last_became_peered":"2024-09-10T19:10:09.822994+0000","last_unstale":"2024-09-10T19:18:11.047270+0000","last_undegraded":"2024-09-10T19:18:11.047270+0000","last_fullsized":"2024-09-10T19:18:11.047270+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'9","last_scrub_stamp":"2024-09-10T19:13:50.172825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:13:50.172825+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0094381619999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":145,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-09-10T19:18:06.910462+0000","last_change":"2024-09-10T19:17:16.305585+0000","last_active":"2024-09-10T19:18:06.910462+0000","last_peered":"2024-09-10T19:18:06.910462+0000","last_clean":"2024-09-10T19:18:06.910462+0000","last_became_active":"2024-09-10T19:16:24.844313+0000","last_became_peered":"2024-09-10T19:16:24.844313+0000","last_unstale":"2024-09-10T19:18:06.910462+0000","last_undegraded":"2024-09-10T19:18:06.910462+0000","last_fullsized":"2024-09-10T19:18:06.910462+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:17:16.305525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:11:41.458523+0000","last_clean_scrub_stamp":"2024-09-10T19:17:16.305525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:18:41.529454+0000","scrub_duration":0.0070771209999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,6],"acting":[0,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":177,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-09-10T19:18:06.907223+0000","last_change":"2024-09-10T19:16:28.794727+0000","last_active":"2024-09-10T19:18:06.907223+0000","last_peered":"2024-09-10T19:18:06.907223+0000","last_clean":"2024-09-10T19:18:06.907223+0000","last_became_active":"2024-09-10T19:13:54.712275+0000","last_became_peered":"2024-09-10T19:13:54.712275+0000","last_unstale":"2024-09-10T19:18:06.907223+0000","last_undegraded":"2024-09-10T19:18:06.907223+0000","last_fullsized":"2024-09-10T19:18:06.907223+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:16:28.794649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:16:28.794649+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0027500379999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"66'1","reported_seq":65,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-09-10T19:18:06.910455+0000","last_change":"2024-09-10T19:16:24.862865+0000","last_active":"2024-09-10T19:18:06.910455+0000","last_peered":"2024-09-10T19:18:06.910455+0000","last_clean":"2024-09-10T19:18:06.910455+0000","last_became_active":"2024-09-10T19:16:24.837432+0000","last_became_peered":"2024-09-10T19:16:24.837432+0000","last_unstale":"2024-09-10T19:18:06.910455+0000","last_undegraded":"2024-09-10T19:18:06.910455+0000","last_fullsized":"2024-09-10T19:18:06.910455+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'1","last_scrub_stamp":"2024-09-10T19:15:48.839582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:15:48.839582+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":13,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":187,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-09-10T19:18:06.920142+0000","last_change":"2024-09-10T19:17:17.479805+0000","last_active":"2024-09-10T19:18:06.920142+0000","last_peered":"2024-09-10T19:18:06.920142+0000","last_clean":"2024-09-10T19:18:06.920142+0000","last_became_active":"2024-09-10T19:13:54.724027+0000","last_became_peered":"2024-09-10T19:13:54.724027+0000","last_unstale":"2024-09-10T19:18:06.920142+0000","last_undegraded":"2024-09-10T19:18:06.920142+0000","last_fullsized":"2024-09-10T19:18:06.920142+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:15:36.714327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:15:36.714327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001831287,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,5],"acting":[6,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"66'6","reported_seq":204,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-09-10T19:18:06.907411+0000","last_change":"2024-09-10T19:16:24.891057+0000","last_active":"2024-09-10T19:18:06.907411+0000","last_peered":"2024-09-10T19:18:06.907411+0000","last_clean":"2024-09-10T19:18:06.907411+0000","last_became_active":"2024-09-10T19:16:24.863398+0000","last_became_peered":"2024-09-10T19:16:24.863398+0000","last_unstale":"2024-09-10T19:18:06.907411+0000","last_undegraded":"2024-09-10T19:18:06.907411+0000","last_fullsized":"2024-09-10T19:18:06.907411+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'6","last_scrub_stamp":"2024-09-10T19:13:55.370650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:13:55.370650+0000","objects_scrubbed":3,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.071693797000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":177,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-09-10T19:18:06.911707+0000","last_change":"2024-09-10T19:15:30.583958+0000","last_active":"2024-09-10T19:18:06.911707+0000","last_peered":"2024-09-10T19:18:06.911707+0000","last_clean":"2024-09-10T19:18:06.911707+0000","last_became_active":"2024-09-10T19:13:54.729917+0000","last_became_peered":"2024-09-10T19:13:54.729917+0000","last_unstale":"2024-09-10T19:18:06.911707+0000","last_undegraded":"2024-09-10T19:18:06.911707+0000","last_fullsized":"2024-09-10T19:18:06.911707+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:15:30.583878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:11:23.264793+0000","last_clean_scrub_stamp":"2024-09-10T19:15:30.583878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0015146960000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":168,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-09-10T19:18:06.913515+0000","last_change":"2024-09-10T19:17:26.922378+0000","last_active":"2024-09-10T19:18:06.913515+0000","last_peered":"2024-09-10T19:18:06.913515+0000","last_clean":"2024-09-10T19:18:06.913515+0000","last_became_active":"2024-09-10T19:10:14.054360+0000","last_became_peered":"2024-09-10T19:10:14.054360+0000","last_unstale":"2024-09-10T19:18:06.913515+0000","last_undegraded":"2024-09-10T19:18:06.913515+0000","last_fullsized":"2024-09-10T19:18:06.913515+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:17:26.922305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:17:26.922305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:18:28.472292+0000","scrub_duration":0.0016181940000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.18","version":"194'7181","reported_seq":13408,"reported_epoch":194,"state":"active+clean","last_fresh":"2024-09-10T19:18:14.722742+0000","last_change":"2024-09-10T19:17:45.681866+0000","last_active":"2024-09-10T19:18:14.722742+0000","last_peered":"2024-09-10T19:18:14.722742+0000","last_clean":"2024-09-10T19:18:14.722742+0000","last_became_active":"2024-09-10T19:17:45.677163+0000","last_became_peered":"2024-09-10T19:17:45.677163+0000","last_unstale":"2024-09-10T19:18:14.722742+0000","last_undegraded":"2024-09-10T19:18:14.722742+0000","last_fullsized":"2024-09-10T19:18:14.722742+0000","mapping_epoch":186,"log_start":"159'1400","ondisk_log_start":"159'1400","created":177,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:16:11.777014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:16:11.777014+0000","last_clean_scrub_stamp":"2024-09-10T19:16:11.777014+0000","objects_scrubbed":0,"log_size":5781,"log_dups_size":0,"ondisk_log_size":5781,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":194084864,"num_objects":2962,"num_object_clones":0,"num_object_copies":8886,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2962,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4366,"num_write_kb":234464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3071,"num_bytes_recovered":201261056,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,5],"acting":[4,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.14","version":"80'15","reported_seq":233,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-09-10T19:18:06.906903+0000","last_change":"2024-09-10T19:16:24.881305+0000","last_active":"2024-09-10T19:18:06.906903+0000","last_peered":"2024-09-10T19:18:06.906903+0000","last_clean":"2024-09-10T19:18:06.906903+0000","last_became_active":"2024-09-10T19:16:24.850702+0000","last_became_peered":"2024-09-10T19:16:24.850702+0000","last_unstale":"2024-09-10T19:18:06.906903+0000","last_undegraded":"2024-09-10T19:18:06.906903+0000","last_fullsized":"2024-09-10T19:18:06.906903+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"80'15","last_scrub_stamp":"2024-09-10T19:15:46.019949+0000","last_deep_scrub":"80'15","last_deep_scrub_stamp":"2024-09-10T19:11:41.719310+0000","last_clean_scrub_stamp":"2024-09-10T19:15:46.019949+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0021039190000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,5],"acting":[3,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":164,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-09-10T19:18:06.911206+0000","last_change":"2024-09-10T19:12:46.506129+0000","last_active":"2024-09-10T19:18:06.911206+0000","last_peered":"2024-09-10T19:18:06.911206+0000","last_clean":"2024-09-10T19:18:06.911206+0000","last_became_active":"2024-09-10T19:10:07.685870+0000","last_became_peered":"2024-09-10T19:10:07.685870+0000","last_unstale":"2024-09-10T19:18:06.911206+0000","last_undegraded":"2024-09-10T19:18:06.911206+0000","last_fullsized":"2024-09-10T19:18:06.911206+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:12:46.506071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:12:46.506071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001257935,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"117'10","reported_seq":287,"reported_epoch":194,"state":"active+clean","last_fresh":"2024-09-10T19:18:08.943351+0000","last_change":"2024-09-10T19:16:23.844790+0000","last_active":"2024-09-10T19:18:08.943351+0000","last_peered":"2024-09-10T19:18:08.943351+0000","last_clean":"2024-09-10T19:18:08.943351+0000","last_became_active":"2024-09-10T19:13:54.724780+0000","last_became_peered":"2024-09-10T19:13:54.724780+0000","last_unstale":"2024-09-10T19:18:08.943351+0000","last_undegraded":"2024-09-10T19:18:08.943351+0000","last_fullsized":"2024-09-10T19:18:08.943351+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"117'10","last_scrub_stamp":"2024-09-10T19:16:23.844724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:16:23.844724+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.012497955999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3],"acting":[5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":167,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-09-10T19:18:06.912922+0000","last_change":"2024-09-10T19:16:40.213422+0000","last_active":"2024-09-10T19:18:06.912922+0000","last_peered":"2024-09-10T19:18:06.912922+0000","last_clean":"2024-09-10T19:18:06.912922+0000","last_became_active":"2024-09-10T19:10:13.721035+0000","last_became_peered":"2024-09-10T19:10:13.721035+0000","last_unstale":"2024-09-10T19:18:06.912922+0000","last_undegraded":"2024-09-10T19:18:06.912922+0000","last_fullsized":"2024-09-10T19:18:06.912922+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:16:40.213321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:16:40.213321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0051408239999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.19","version":"194'7310","reported_seq":13819,"reported_epoch":194,"state":"active+clean","last_fresh":"2024-09-10T19:18:14.853881+0000","last_change":"2024-09-10T19:17:46.698410+0000","last_active":"2024-09-10T19:18:14.853881+0000","last_peered":"2024-09-10T19:18:14.853881+0000","last_clean":"2024-09-10T19:18:14.853881+0000","last_became_active":"2024-09-10T19:17:46.694716+0000","last_became_peered":"2024-09-10T19:17:46.694716+0000","last_unstale":"2024-09-10T19:18:14.853881+0000","last_undegraded":"2024-09-10T19:18:14.853881+0000","last_fullsized":"2024-09-10T19:18:14.853881+0000","mapping_epoch":187,"log_start":"160'1700","ondisk_log_start":"160'1700","created":177,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:16:11.777014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:16:11.777014+0000","last_clean_scrub_stamp":"2024-09-10T19:16:11.777014+0000","objects_scrubbed":0,"log_size":5610,"log_dups_size":0,"ondisk_log_size":5610,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":199360512,"num_objects":3042,"num_object_clones":0,"num_object_copies":9126,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3042,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4418,"num_write_kb":238720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3210,"num_bytes_recovered":210370560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,0],"acting":[5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.15","version":"66'9","reported_seq":218,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-09-10T19:18:06.913858+0000","last_change":"2024-09-10T19:16:24.882653+0000","last_active":"2024-09-10T19:18:06.913858+0000","last_peered":"2024-09-10T19:18:06.913858+0000","last_clean":"2024-09-10T19:18:06.913858+0000","last_became_active":"2024-09-10T19:16:24.859740+0000","last_became_peered":"2024-09-10T19:16:24.859740+0000","last_unstale":"2024-09-10T19:18:06.913858+0000","last_undegraded":"2024-09-10T19:18:06.913858+0000","last_fullsized":"2024-09-10T19:18:06.913858+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'9","last_scrub_stamp":"2024-09-10T19:12:52.416748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:12:52.416748+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0047220839999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":173,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-09-10T19:18:06.910680+0000","last_change":"2024-09-10T19:16:37.243942+0000","last_active":"2024-09-10T19:18:06.910680+0000","last_peered":"2024-09-10T19:18:06.910680+0000","last_clean":"2024-09-10T19:18:06.910680+0000","last_became_active":"2024-09-10T19:10:07.688499+0000","last_became_peered":"2024-09-10T19:10:07.688499+0000","last_unstale":"2024-09-10T19:18:06.910680+0000","last_undegraded":"2024-09-10T19:18:06.910680+0000","last_fullsized":"2024-09-10T19:18:06.910680+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:16:37.243859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:16:37.243859+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0052374580000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"155'9","reported_seq":279,"reported_epoch":194,"state":"active+clean","last_fresh":"2024-09-10T19:18:08.937938+0000","last_change":"2024-09-10T19:16:24.881133+0000","last_active":"2024-09-10T19:18:08.937938+0000","last_peered":"2024-09-10T19:18:08.937938+0000","last_clean":"2024-09-10T19:18:08.937938+0000","last_became_active":"2024-09-10T19:16:24.850850+0000","last_became_peered":"2024-09-10T19:16:24.850850+0000","last_unstale":"2024-09-10T19:18:08.937938+0000","last_undegraded":"2024-09-10T19:18:08.937938+0000","last_fullsized":"2024-09-10T19:18:08.937938+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'8","last_scrub_stamp":"2024-09-10T19:12:59.723712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:12:59.723712+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0083838109999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,4],"acting":[3,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":170,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-09-10T19:18:06.911936+0000","last_change":"2024-09-10T19:17:10.866121+0000","last_active":"2024-09-10T19:18:06.911936+0000","last_peered":"2024-09-10T19:18:06.911936+0000","last_clean":"2024-09-10T19:18:06.911936+0000","last_became_active":"2024-09-10T19:16:24.854280+0000","last_became_peered":"2024-09-10T19:16:24.854280+0000","last_unstale":"2024-09-10T19:18:06.911936+0000","last_undegraded":"2024-09-10T19:18:06.911936+0000","last_fullsized":"2024-09-10T19:18:06.911936+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:13:09.463001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:13:09.463001+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.012199573,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.12","version":"79'13","reported_seq":204,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-09-10T19:18:06.919227+0000","last_change":"2024-09-10T19:17:11.485802+0000","last_active":"2024-09-10T19:18:06.919227+0000","last_peered":"2024-09-10T19:18:06.919227+0000","last_clean":"2024-09-10T19:18:06.919227+0000","last_became_active":"2024-09-10T19:16:24.845450+0000","last_became_peered":"2024-09-10T19:16:24.845450+0000","last_unstale":"2024-09-10T19:18:06.919227+0000","last_undegraded":"2024-09-10T19:18:06.919227+0000","last_fullsized":"2024-09-10T19:18:06.919227+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'13","last_scrub_stamp":"2024-09-10T19:15:29.854117+0000","last_deep_scrub":"79'13","last_deep_scrub_stamp":"2024-09-10T19:11:22.072400+0000","last_clean_scrub_stamp":"2024-09-10T19:15:29.854117+0000","objects_scrubbed":4,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0066627300000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.14","version":"59'1","reported_seq":181,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-09-10T19:18:06.919711+0000","last_change":"2024-09-10T19:16:27.594895+0000","last_active":"2024-09-10T19:18:06.919711+0000","last_peered":"2024-09-10T19:18:06.919711+0000","last_clean":"2024-09-10T19:18:06.919711+0000","last_became_active":"2024-09-10T19:10:07.683263+0000","last_became_peered":"2024-09-10T19:10:07.683263+0000","last_unstale":"2024-09-10T19:18:06.919711+0000","last_undegraded":"2024-09-10T19:18:06.919711+0000","last_fullsized":"2024-09-10T19:18:06.919711+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'1","last_scrub_stamp":"2024-09-10T19:16:27.594788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:16:27.594788+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0091369840000000008,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":362,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":177,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-09-10T19:18:06.907727+0000","last_change":"2024-09-10T19:15:55.925313+0000","last_active":"2024-09-10T19:18:06.907727+0000","last_peered":"2024-09-10T19:18:06.907727+0000","last_clean":"2024-09-10T19:18:06.907727+0000","last_became_active":"2024-09-10T19:13:54.730185+0000","last_became_peered":"2024-09-10T19:13:54.730185+0000","last_unstale":"2024-09-10T19:18:06.907727+0000","last_undegraded":"2024-09-10T19:18:06.907727+0000","last_fullsized":"2024-09-10T19:18:06.907727+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:15:55.925228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:15:55.925228+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0017202299999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":174,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-09-10T19:18:06.909917+0000","last_change":"2024-09-10T19:16:23.273671+0000","last_active":"2024-09-10T19:18:06.909917+0000","last_peered":"2024-09-10T19:18:06.909917+0000","last_clean":"2024-09-10T19:18:06.909917+0000","last_became_active":"2024-09-10T19:13:54.714861+0000","last_became_peered":"2024-09-10T19:13:54.714861+0000","last_unstale":"2024-09-10T19:18:06.909917+0000","last_undegraded":"2024-09-10T19:18:06.909917+0000","last_fullsized":"2024-09-10T19:18:06.909917+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:16:23.273615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:16:23.273615+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0013972139999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,6],"acting":[0,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"66'11","reported_seq":212,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-09-10T19:18:06.913219+0000","last_change":"2024-09-10T19:15:53.883613+0000","last_active":"2024-09-10T19:18:06.913219+0000","last_peered":"2024-09-10T19:18:06.913219+0000","last_clean":"2024-09-10T19:18:06.913219+0000","last_became_active":"2024-09-10T19:13:54.720034+0000","last_became_peered":"2024-09-10T19:13:54.720034+0000","last_unstale":"2024-09-10T19:18:06.913219+0000","last_undegraded":"2024-09-10T19:18:06.913219+0000","last_fullsized":"2024-09-10T19:18:06.913219+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'11","last_scrub_stamp":"2024-09-10T19:15:53.883548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:15:53.883548+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001858326,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":114,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-09-10T19:18:06.913826+0000","last_change":"2024-09-10T19:13:54.723475+0000","last_active":"2024-09-10T19:18:06.913826+0000","last_peered":"2024-09-10T19:18:06.913826+0000","last_clean":"2024-09-10T19:18:06.913826+0000","last_became_active":"2024-09-10T19:13:54.722947+0000","last_became_peered":"2024-09-10T19:13:54.722947+0000","last_unstale":"2024-09-10T19:18:06.913826+0000","last_undegraded":"2024-09-10T19:18:06.913826+0000","last_fullsized":"2024-09-10T19:18:06.913826+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:12:44.249765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:11:10.031245+0000","last_clean_scrub_stamp":"2024-09-10T19:12:44.249765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,6],"acting":[5,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":124,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-09-10T19:18:06.911467+0000","last_change":"2024-09-10T19:15:36.666310+0000","last_active":"2024-09-10T19:18:06.911467+0000","last_peered":"2024-09-10T19:18:06.911467+0000","last_clean":"2024-09-10T19:18:06.911467+0000","last_became_active":"2024-09-10T19:13:54.729703+0000","last_became_peered":"2024-09-10T19:13:54.729703+0000","last_unstale":"2024-09-10T19:18:06.911467+0000","last_undegraded":"2024-09-10T19:18:06.911467+0000","last_fullsized":"2024-09-10T19:18:06.911467+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:15:36.666216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:15:36.666216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.003489884,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":158,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-09-10T19:18:06.905973+0000","last_change":"2024-09-10T19:12:54.691886+0000","last_active":"2024-09-10T19:18:06.905973+0000","last_peered":"2024-09-10T19:18:06.905973+0000","last_clean":"2024-09-10T19:18:06.905973+0000","last_became_active":"2024-09-10T19:10:13.724201+0000","last_became_peered":"2024-09-10T19:10:13.724201+0000","last_unstale":"2024-09-10T19:18:06.905973+0000","last_undegraded":"2024-09-10T19:18:06.905973+0000","last_fullsized":"2024-09-10T19:18:06.905973+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:12:54.691810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:12:54.691810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.006702927,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"66'4","reported_seq":191,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-09-10T19:18:06.906695+0000","last_change":"2024-09-10T19:15:33.995417+0000","last_active":"2024-09-10T19:18:06.906695+0000","last_peered":"2024-09-10T19:18:06.906695+0000","last_clean":"2024-09-10T19:18:06.906695+0000","last_became_active":"2024-09-10T19:13:54.711125+0000","last_became_peered":"2024-09-10T19:13:54.711125+0000","last_unstale":"2024-09-10T19:18:06.906695+0000","last_undegraded":"2024-09-10T19:18:06.906695+0000","last_fullsized":"2024-09-10T19:18:06.906695+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'4","last_scrub_stamp":"2024-09-10T19:15:33.995274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:15:33.995274+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.011108425,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,4],"acting":[3,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":184,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-09-10T19:18:06.913935+0000","last_change":"2024-09-10T19:16:33.766560+0000","last_active":"2024-09-10T19:18:06.913935+0000","last_peered":"2024-09-10T19:18:06.913935+0000","last_clean":"2024-09-10T19:18:06.913935+0000","last_became_active":"2024-09-10T19:16:24.852460+0000","last_became_peered":"2024-09-10T19:16:24.852460+0000","last_unstale":"2024-09-10T19:18:06.913935+0000","last_undegraded":"2024-09-10T19:18:06.913935+0000","last_fullsized":"2024-09-10T19:18:06.913935+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:16:33.766474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:16:33.766474+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0042170250000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,3],"acting":[5,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":170,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-09-10T19:18:06.919908+0000","last_change":"2024-09-10T19:16:24.545188+0000","last_active":"2024-09-10T19:18:06.919908+0000","last_peered":"2024-09-10T19:18:06.919908+0000","last_clean":"2024-09-10T19:18:06.919908+0000","last_became_active":"2024-09-10T19:10:11.698536+0000","last_became_peered":"2024-09-10T19:10:11.698536+0000","last_unstale":"2024-09-10T19:18:06.919908+0000","last_undegraded":"2024-09-10T19:18:06.919908+0000","last_fullsized":"2024-09-10T19:18:06.919908+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:16:24.545134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:11:31.837944+0000","last_clean_scrub_stamp":"2024-09-10T19:16:24.545134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.00120104,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":178,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-09-10T19:18:06.911669+0000","last_change":"2024-09-10T19:16:24.852651+0000","last_active":"2024-09-10T19:18:06.911669+0000","last_peered":"2024-09-10T19:18:06.911669+0000","last_clean":"2024-09-10T19:18:06.911669+0000","last_became_active":"2024-09-10T19:16:24.852356+0000","last_became_peered":"2024-09-10T19:16:24.852356+0000","last_unstale":"2024-09-10T19:18:06.911669+0000","last_undegraded":"2024-09-10T19:18:06.911669+0000","last_fullsized":"2024-09-10T19:18:06.911669+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:15:44.523530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:15:44.523530+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.027001213999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,3],"acting":[7,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.11","version":"66'11","reported_seq":72,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-09-10T19:18:06.911714+0000","last_change":"2024-09-10T19:15:45.158623+0000","last_active":"2024-09-10T19:18:06.911714+0000","last_peered":"2024-09-10T19:18:06.911714+0000","last_clean":"2024-09-10T19:18:06.911714+0000","last_became_active":"2024-09-10T19:15:45.149864+0000","last_became_peered":"2024-09-10T19:15:45.149864+0000","last_unstale":"2024-09-10T19:18:06.911714+0000","last_undegraded":"2024-09-10T19:18:06.911714+0000","last_fullsized":"2024-09-10T19:18:06.911714+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'11","last_scrub_stamp":"2024-09-10T19:13:05.729854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:13:05.729854+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.010667395,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":184,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-09-10T19:18:06.920282+0000","last_change":"2024-09-10T19:16:24.855851+0000","last_active":"2024-09-10T19:18:06.920282+0000","last_peered":"2024-09-10T19:18:06.920282+0000","last_clean":"2024-09-10T19:18:06.920282+0000","last_became_active":"2024-09-10T19:16:24.855541+0000","last_became_peered":"2024-09-10T19:16:24.855541+0000","last_unstale":"2024-09-10T19:18:06.920282+0000","last_undegraded":"2024-09-10T19:18:06.920282+0000","last_fullsized":"2024-09-10T19:18:06.920282+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:16:23.543097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:13:05.876655+0000","last_clean_scrub_stamp":"2024-09-10T19:16:23.543097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001181779,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,0],"acting":[6,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":160,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-09-10T19:18:06.912620+0000","last_change":"2024-09-10T19:13:41.326028+0000","last_active":"2024-09-10T19:18:06.912620+0000","last_peered":"2024-09-10T19:18:06.912620+0000","last_clean":"2024-09-10T19:18:06.912620+0000","last_became_active":"2024-09-10T19:10:11.699013+0000","last_became_peered":"2024-09-10T19:10:11.699013+0000","last_unstale":"2024-09-10T19:18:06.912620+0000","last_undegraded":"2024-09-10T19:18:06.912620+0000","last_fullsized":"2024-09-10T19:18:06.912620+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:13:41.325941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:11:57.112092+0000","last_clean_scrub_stamp":"2024-09-10T19:13:41.325941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001598676,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":158,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-09-10T19:18:06.907585+0000","last_change":"2024-09-10T19:13:15.812326+0000","last_active":"2024-09-10T19:18:06.907585+0000","last_peered":"2024-09-10T19:18:06.907585+0000","last_clean":"2024-09-10T19:18:06.907585+0000","last_became_active":"2024-09-10T19:10:14.056172+0000","last_became_peered":"2024-09-10T19:10:14.056172+0000","last_unstale":"2024-09-10T19:18:06.907585+0000","last_undegraded":"2024-09-10T19:18:06.907585+0000","last_fullsized":"2024-09-10T19:18:06.907585+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:13:15.812244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:13:15.812244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0048044389999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.12","version":"194'7418","reported_seq":7497,"reported_epoch":194,"state":"active+remapped+backfill_toofull","last_fresh":"2024-09-10T19:18:14.999408+0000","last_change":"2024-09-10T19:18:00.037116+0000","last_active":"2024-09-10T19:18:14.999408+0000","last_peered":"2024-09-10T19:18:14.999408+0000","last_clean":"2024-09-10T19:17:27.172505+0000","last_became_active":"2024-09-10T19:17:30.027866+0000","last_became_peered":"2024-09-10T19:17:30.027866+0000","last_unstale":"2024-09-10T19:18:14.999408+0000","last_undegraded":"2024-09-10T19:18:14.999408+0000","last_fullsized":"2024-09-10T19:18:14.999408+0000","mapping_epoch":180,"log_start":"159'1500","ondisk_log_start":"159'1500","created":177,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:16:11.777014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:16:11.777014+0000","last_clean_scrub_stamp":"2024-09-10T19:16:11.777014+0000","objects_scrubbed":0,"log_size":5918,"log_dups_size":0,"ondisk_log_size":5918,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":204668928,"num_objects":3123,"num_object_clones":0,"num_object_copies":9369,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":6246,"num_objects_unfound":0,"num_objects_dirty":3123,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4501,"num_write_kb":243968,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,4],"acting":[0,4,5],"avail_no_missing":["0","4","5"],"object_location_counts":[{"shards":"0,4,5","objects":3123}],"blocked_by":[],"up_primary":6,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1c","version":"80'2","reported_seq":184,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-09-10T19:18:06.912355+0000","last_change":"2024-09-10T19:16:32.648844+0000","last_active":"2024-09-10T19:18:06.912355+0000","last_peered":"2024-09-10T19:18:06.912355+0000","last_clean":"2024-09-10T19:18:06.912355+0000","last_became_active":"2024-09-10T19:16:24.851220+0000","last_became_peered":"2024-09-10T19:16:24.851220+0000","last_unstale":"2024-09-10T19:18:06.912355+0000","last_undegraded":"2024-09-10T19:18:06.912355+0000","last_fullsized":"2024-09-10T19:18:06.912355+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"80'2","last_scrub_stamp":"2024-09-10T19:16:32.648738+0000","last_deep_scrub":"80'2","last_deep_scrub_stamp":"2024-09-10T19:11:30.272043+0000","last_clean_scrub_stamp":"2024-09-10T19:16:32.648738+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0066232920000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":369,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":369,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,3],"acting":[7,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":164,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-09-10T19:18:06.915134+0000","last_change":"2024-09-10T19:13:14.449643+0000","last_active":"2024-09-10T19:18:06.915134+0000","last_peered":"2024-09-10T19:18:06.915134+0000","last_clean":"2024-09-10T19:18:06.915134+0000","last_became_active":"2024-09-10T19:10:07.689877+0000","last_became_peered":"2024-09-10T19:10:07.689877+0000","last_unstale":"2024-09-10T19:18:06.915134+0000","last_undegraded":"2024-09-10T19:18:06.915134+0000","last_fullsized":"2024-09-10T19:18:06.915134+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:13:14.449593+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:13:14.449593+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001195542,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1e","version":"80'15","reported_seq":191,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-09-10T19:18:06.913208+0000","last_change":"2024-09-10T19:16:33.258114+0000","last_active":"2024-09-10T19:18:06.913208+0000","last_peered":"2024-09-10T19:18:06.913208+0000","last_clean":"2024-09-10T19:18:06.913208+0000","last_became_active":"2024-09-10T19:10:09.823061+0000","last_became_peered":"2024-09-10T19:10:09.823061+0000","last_unstale":"2024-09-10T19:18:06.913208+0000","last_undegraded":"2024-09-10T19:18:06.913208+0000","last_fullsized":"2024-09-10T19:18:06.913208+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"80'15","last_scrub_stamp":"2024-09-10T19:16:33.258031+0000","last_deep_scrub":"80'15","last_deep_scrub_stamp":"2024-09-10T19:11:25.883919+0000","last_clean_scrub_stamp":"2024-09-10T19:16:33.258031+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0040337309999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1f","version":"66'8","reported_seq":281,"reported_epoch":194,"state":"active+clean","last_fresh":"2024-09-10T19:18:08.937750+0000","last_change":"2024-09-10T19:17:32.477395+0000","last_active":"2024-09-10T19:18:08.937750+0000","last_peered":"2024-09-10T19:18:08.937750+0000","last_clean":"2024-09-10T19:18:08.937750+0000","last_became_active":"2024-09-10T19:10:11.695427+0000","last_became_peered":"2024-09-10T19:10:11.695427+0000","last_unstale":"2024-09-10T19:18:08.937750+0000","last_undegraded":"2024-09-10T19:18:08.937750+0000","last_fullsized":"2024-09-10T19:18:08.937750+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'8","last_scrub_stamp":"2024-09-10T19:15:53.664208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:15:53.664208+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001849699,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.13","version":"194'7578","reported_seq":10853,"reported_epoch":194,"state":"active+recovering+remapped","last_fresh":"2024-09-10T19:18:14.955031+0000","last_change":"2024-09-10T19:17:30.105506+0000","last_active":"2024-09-10T19:18:14.955031+0000","last_peered":"2024-09-10T19:18:14.955031+0000","last_clean":"2024-09-10T19:16:54.964816+0000","last_became_active":"2024-09-10T19:17:30.042990+0000","last_became_peered":"2024-09-10T19:17:30.042990+0000","last_unstale":"2024-09-10T19:18:14.955031+0000","last_undegraded":"2024-09-10T19:18:14.955031+0000","last_fullsized":"2024-09-10T19:18:14.955031+0000","mapping_epoch":180,"log_start":"160'2100","ondisk_log_start":"160'2100","created":177,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:16:11.777014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:16:11.777014+0000","last_clean_scrub_stamp":"2024-09-10T19:16:11.777014+0000","objects_scrubbed":0,"log_size":5478,"log_dups_size":0,"ondisk_log_size":5478,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":199983104,"num_objects":3052,"num_object_clones":0,"num_object_copies":9156,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":899,"num_objects_unfound":0,"num_objects_dirty":3052,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4568,"num_write_kb":243808,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1053,"num_bytes_recovered":68976640,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,4],"acting":[3,5,7],"avail_no_missing":["3","5","6"],"object_location_counts":[{"shards":"3,5,6","objects":899},{"shards":"3,5,7","objects":2153}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":134,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-09-10T19:18:06.911068+0000","last_change":"2024-09-10T19:16:24.863000+0000","last_active":"2024-09-10T19:18:06.911068+0000","last_peered":"2024-09-10T19:18:06.911068+0000","last_clean":"2024-09-10T19:18:06.911068+0000","last_became_active":"2024-09-10T19:16:24.862695+0000","last_became_peered":"2024-09-10T19:16:24.862695+0000","last_unstale":"2024-09-10T19:18:06.911068+0000","last_undegraded":"2024-09-10T19:18:06.911068+0000","last_fullsized":"2024-09-10T19:18:06.911068+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:15:29.580605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:15:29.580605+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0047886710000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":174,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-09-10T19:18:06.906522+0000","last_change":"2024-09-10T19:16:41.881053+0000","last_active":"2024-09-10T19:18:06.906522+0000","last_peered":"2024-09-10T19:18:06.906522+0000","last_clean":"2024-09-10T19:18:06.906522+0000","last_became_active":"2024-09-10T19:10:07.682689+0000","last_became_peered":"2024-09-10T19:10:07.682689+0000","last_unstale":"2024-09-10T19:18:06.906522+0000","last_undegraded":"2024-09-10T19:18:06.906522+0000","last_fullsized":"2024-09-10T19:18:06.906522+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:16:41.880992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:16:41.880992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0044984990000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"80'16","reported_seq":243,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-09-10T19:18:06.920212+0000","last_change":"2024-09-10T19:17:15.446029+0000","last_active":"2024-09-10T19:18:06.920212+0000","last_peered":"2024-09-10T19:18:06.920212+0000","last_clean":"2024-09-10T19:18:06.920212+0000","last_became_active":"2024-09-10T19:13:54.713563+0000","last_became_peered":"2024-09-10T19:13:54.713563+0000","last_unstale":"2024-09-10T19:18:06.920212+0000","last_undegraded":"2024-09-10T19:18:06.920212+0000","last_fullsized":"2024-09-10T19:18:06.920212+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"80'16","last_scrub_stamp":"2024-09-10T19:15:47.700523+0000","last_deep_scrub":"80'16","last_deep_scrub_stamp":"2024-09-10T19:11:40.853611+0000","last_clean_scrub_stamp":"2024-09-10T19:15:47.700523+0000","objects_scrubbed":5,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0019149010000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,0],"acting":[6,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":179,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-09-10T19:18:06.913144+0000","last_change":"2024-09-10T19:16:34.222429+0000","last_active":"2024-09-10T19:18:06.913144+0000","last_peered":"2024-09-10T19:18:06.913144+0000","last_clean":"2024-09-10T19:18:06.913144+0000","last_became_active":"2024-09-10T19:16:24.834713+0000","last_became_peered":"2024-09-10T19:16:24.834713+0000","last_unstale":"2024-09-10T19:18:06.913144+0000","last_undegraded":"2024-09-10T19:18:06.913144+0000","last_fullsized":"2024-09-10T19:18:06.913144+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:16:34.222365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:11:28.822327+0000","last_clean_scrub_stamp":"2024-09-10T19:16:34.222365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.014893412999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.10","version":"194'7468","reported_seq":14239,"reported_epoch":194,"state":"active+clean","last_fresh":"2024-09-10T19:18:14.811570+0000","last_change":"2024-09-10T19:17:46.709272+0000","last_active":"2024-09-10T19:18:14.811570+0000","last_peered":"2024-09-10T19:18:14.811570+0000","last_clean":"2024-09-10T19:18:14.811570+0000","last_became_active":"2024-09-10T19:17:46.694818+0000","last_became_peered":"2024-09-10T19:17:46.694818+0000","last_unstale":"2024-09-10T19:18:14.811570+0000","last_undegraded":"2024-09-10T19:18:14.811570+0000","last_fullsized":"2024-09-10T19:18:14.811570+0000","mapping_epoch":187,"log_start":"160'2408","ondisk_log_start":"160'2408","created":177,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:16:11.777014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:16:11.777014+0000","last_clean_scrub_stamp":"2024-09-10T19:16:11.777014+0000","objects_scrubbed":0,"log_size":5060,"log_dups_size":0,"ondisk_log_size":5060,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":201981952,"num_objects":3082,"num_object_clones":0,"num_object_copies":9246,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3082,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4552,"num_write_kb":244288,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3344,"num_bytes_recovered":219152384,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,4],"acting":[7,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":167,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-09-10T19:18:06.912321+0000","last_change":"2024-09-10T19:16:36.960796+0000","last_active":"2024-09-10T19:18:06.912321+0000","last_peered":"2024-09-10T19:18:06.912321+0000","last_clean":"2024-09-10T19:18:06.912321+0000","last_became_active":"2024-09-10T19:10:14.053619+0000","last_became_peered":"2024-09-10T19:10:14.053619+0000","last_unstale":"2024-09-10T19:18:06.912321+0000","last_undegraded":"2024-09-10T19:18:06.912321+0000","last_fullsized":"2024-09-10T19:18:06.912321+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:16:36.960695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:16:36.960695+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.011430734999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":174,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-09-10T19:18:06.919524+0000","last_change":"2024-09-10T19:16:30.666084+0000","last_active":"2024-09-10T19:18:06.919524+0000","last_peered":"2024-09-10T19:18:06.919524+0000","last_clean":"2024-09-10T19:18:06.919524+0000","last_became_active":"2024-09-10T19:10:07.685997+0000","last_became_peered":"2024-09-10T19:10:07.685997+0000","last_unstale":"2024-09-10T19:18:06.919524+0000","last_undegraded":"2024-09-10T19:18:06.919524+0000","last_fullsized":"2024-09-10T19:18:06.919524+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:16:30.666001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:16:30.666001+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0041927830000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1c","version":"78'22","reported_seq":123,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-09-10T19:18:06.906691+0000","last_change":"2024-09-10T19:16:24.870445+0000","last_active":"2024-09-10T19:18:06.906691+0000","last_peered":"2024-09-10T19:18:06.906691+0000","last_clean":"2024-09-10T19:18:06.906691+0000","last_became_active":"2024-09-10T19:16:24.845764+0000","last_became_peered":"2024-09-10T19:16:24.845764+0000","last_unstale":"2024-09-10T19:18:06.906691+0000","last_undegraded":"2024-09-10T19:18:06.906691+0000","last_fullsized":"2024-09-10T19:18:06.906691+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"78'22","last_scrub_stamp":"2024-09-10T19:15:37.702624+0000","last_deep_scrub":"78'22","last_deep_scrub_stamp":"2024-09-10T19:11:21.118937+0000","last_clean_scrub_stamp":"2024-09-10T19:15:37.702624+0000","objects_scrubbed":7,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":127,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-09-10T19:18:06.909875+0000","last_change":"2024-09-10T19:15:33.008615+0000","last_active":"2024-09-10T19:18:06.909875+0000","last_peered":"2024-09-10T19:18:06.909875+0000","last_clean":"2024-09-10T19:18:06.909875+0000","last_became_active":"2024-09-10T19:13:54.709055+0000","last_became_peered":"2024-09-10T19:13:54.709055+0000","last_unstale":"2024-09-10T19:18:06.909875+0000","last_undegraded":"2024-09-10T19:18:06.909875+0000","last_fullsized":"2024-09-10T19:18:06.909875+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:15:33.008542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:15:33.008542+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.004557363,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,5],"acting":[4,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.11","version":"194'7387","reported_seq":13623,"reported_epoch":194,"state":"active+remapped+backfilling","last_fresh":"2024-09-10T19:18:14.857722+0000","last_change":"2024-09-10T19:18:00.064484+0000","last_active":"2024-09-10T19:18:14.857722+0000","last_peered":"2024-09-10T19:18:14.857722+0000","last_clean":"2024-09-10T19:17:27.057735+0000","last_became_active":"2024-09-10T19:17:30.038219+0000","last_became_peered":"2024-09-10T19:17:30.038219+0000","last_unstale":"2024-09-10T19:18:14.857722+0000","last_undegraded":"2024-09-10T19:18:14.857722+0000","last_fullsized":"2024-09-10T19:18:14.857722+0000","mapping_epoch":180,"log_start":"159'1600","ondisk_log_start":"159'1600","created":177,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:16:11.777014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:16:11.777014+0000","last_clean_scrub_stamp":"2024-09-10T19:16:11.777014+0000","objects_scrubbed":0,"log_size":5787,"log_dups_size":0,"ondisk_log_size":5787,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":199655424,"num_objects":3047,"num_object_clones":0,"num_object_copies":9141,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2332,"num_objects_unfound":0,"num_objects_dirty":3047,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4469,"num_write_kb":240480,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2049,"num_bytes_recovered":134283264,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,0,3],"avail_no_missing":["5","0","3"],"object_location_counts":[{"shards":"0,3,5","objects":3047}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":158,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-09-10T19:18:06.907057+0000","last_change":"2024-09-10T19:12:51.767586+0000","last_active":"2024-09-10T19:18:06.907057+0000","last_peered":"2024-09-10T19:18:06.907057+0000","last_clean":"2024-09-10T19:18:06.907057+0000","last_became_active":"2024-09-10T19:10:14.053698+0000","last_became_peered":"2024-09-10T19:10:14.053698+0000","last_unstale":"2024-09-10T19:18:06.907057+0000","last_undegraded":"2024-09-10T19:18:06.907057+0000","last_fullsized":"2024-09-10T19:18:06.907057+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:12:51.767056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:12:51.767056+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.006796519,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":164,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-09-10T19:18:06.907880+0000","last_change":"2024-09-10T19:12:50.719167+0000","last_active":"2024-09-10T19:18:06.907880+0000","last_peered":"2024-09-10T19:18:06.907880+0000","last_clean":"2024-09-10T19:18:06.907880+0000","last_became_active":"2024-09-10T19:10:07.682811+0000","last_became_peered":"2024-09-10T19:10:07.682811+0000","last_unstale":"2024-09-10T19:18:06.907880+0000","last_undegraded":"2024-09-10T19:18:06.907880+0000","last_fullsized":"2024-09-10T19:18:06.907880+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:12:50.718663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:12:50.718663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0042138399999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1d","version":"66'12","reported_seq":214,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-09-10T19:18:06.906293+0000","last_change":"2024-09-10T19:15:52.923561+0000","last_active":"2024-09-10T19:18:06.906293+0000","last_peered":"2024-09-10T19:18:06.906293+0000","last_clean":"2024-09-10T19:18:06.906293+0000","last_became_active":"2024-09-10T19:13:54.714629+0000","last_became_peered":"2024-09-10T19:13:54.714629+0000","last_unstale":"2024-09-10T19:18:06.906293+0000","last_undegraded":"2024-09-10T19:18:06.906293+0000","last_fullsized":"2024-09-10T19:18:06.906293+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'12","last_scrub_stamp":"2024-09-10T19:15:52.923498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:15:52.923498+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0019450450000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":179,"reported_epoch":193,"state":"active+clean","last_fresh":"2024-09-10T19:18:06.910647+0000","last_change":"2024-09-10T19:16:37.970105+0000","last_active":"2024-09-10T19:18:06.910647+0000","last_peered":"2024-09-10T19:18:06.910647+0000","last_clean":"2024-09-10T19:18:06.910647+0000","last_became_active":"2024-09-10T19:16:24.843013+0000","last_became_peered":"2024-09-10T19:16:24.843013+0000","last_unstale":"2024-09-10T19:18:06.910647+0000","last_undegraded":"2024-09-10T19:18:06.910647+0000","last_fullsized":"2024-09-10T19:18:06.910647+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:16:37.970030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:16:37.970030+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.003625063,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":8,"num_pg":26,"stat_sum":{"num_bytes":6402211864,"num_objects":97691,"num_object_clones":0,"num_object_copies":293073,"num_objects_missing_on_primary":993,"num_objects_missing":993,"num_objects_degraded":0,"num_objects_misplaced":25555,"num_objects_unfound":0,"num_objects_dirty":97691,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":142871,"num_write_kb":7697921,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16920,"num_bytes_recovered":1108869120,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":20465790976,"data_stored":20465778760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":135006,"ondisk_log_size":135006,"up":78,"acting":78,"num_store_stats":7},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":382,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":1528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":73,"ondisk_log_size":73,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":177,"num_object_clones":0,"num_object_copies":531,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":177,"num_whiteouts":0,"num_read":695,"num_read_kb":452,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":173,"num_bytes_recovered":3218,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":540672,"data_stored":14242,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":445,"ondisk_log_size":445,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":116,"num_read_kb":111,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":389,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":2002,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":9,"ondisk_log_size":9,"up":9,"acting":9,"num_store_stats":7},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1328,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":27,"num_read_kb":27,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":3984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4,"ondisk_log_size":4,"up":96,"acting":96,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79,"ondisk_log_size":79,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":54,"seq":231928234086,"num_pgs":87,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3795132,"kb_used_data":3493556,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":89950020,"statfs":{"total":95995035648,"available":92108820480,"internally_reserved":0,"allocated":3577401344,"data_stored":3576829024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,1,2,0,4,1,4,2,0,1],"upper_bound":2048},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 19:18:04 2024","interfaces":[{"interface":"back","average":{"1min":1.629,"5min":1.663,"15min":1.0549999999999999},"min":{"1min":0.20999999999999999,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":5.4470000000000001,"5min":25.640999999999998,"15min":35.551000000000002},"last":1.409},{"interface":"front","average":{"1min":1.246,"5min":1.4530000000000001,"15min":0.96699999999999997},"min":{"1min":0.215,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":5.9119999999999999,"5min":27.870999999999999,"15min":35.503999999999998},"last":0.60299999999999998}]},{"osd":1,"last update":"Tue Sep 10 19:17:25 2024","interfaces":[{"interface":"back","average":{"1min":1.9870000000000001,"5min":0.81699999999999995,"15min":0.622},"min":{"1min":0.30099999999999999,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":8.3499999999999996,"5min":8.3499999999999996,"15min":8.3499999999999996},"last":1.3700000000000001},{"interface":"front","average":{"1min":1.482,"5min":0.86199999999999999,"15min":0.75900000000000001},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":5.609,"5min":5.609,"15min":5.609},"last":1.4550000000000001}]},{"osd":2,"last update":"Tue Sep 10 19:18:04 2024","interfaces":[{"interface":"back","average":{"1min":0.84099999999999997,"5min":1.181,"15min":0.73799999999999999},"min":{"1min":0.27700000000000002,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":2.7519999999999998,"5min":31.940999999999999,"15min":31.940999999999999},"last":1.204},{"interface":"front","average":{"1min":1.6539999999999999,"5min":1.4990000000000001,"15min":0.878},"min":{"1min":0.36299999999999999,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":5.5410000000000004,"5min":25.056000000000001,"15min":25.056000000000001},"last":0.48499999999999999}]},{"osd":3,"last update":"Tue Sep 10 19:18:04 2024","interfaces":[{"interface":"back","average":{"1min":2.3130000000000002,"5min":1.6220000000000001,"15min":1.095},"min":{"1min":0.249,"5min":0.217,"15min":0.217},"max":{"1min":21.074999999999999,"5min":25.443999999999999,"15min":35.277999999999999},"last":0.52300000000000002},{"interface":"front","average":{"1min":2.9660000000000002,"5min":1.6279999999999999,"15min":1.0900000000000001},"min":{"1min":0.29699999999999999,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":18.82,"5min":25.530000000000001,"15min":36.512999999999998},"last":1.4339999999999999}]},{"osd":4,"last update":"Tue Sep 10 19:18:04 2024","interfaces":[{"interface":"back","average":{"1min":2.1040000000000001,"5min":1.0580000000000001,"15min":0.65200000000000002},"min":{"1min":0.34599999999999997,"5min":0.23300000000000001,"15min":0.22800000000000001},"max":{"1min":6.9160000000000004,"5min":8.1859999999999999,"15min":8.1859999999999999},"last":1.823},{"interface":"front","average":{"1min":1.0049999999999999,"5min":0.76400000000000001,"15min":0.55900000000000005},"min":{"1min":0.378,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":3.0339999999999998,"5min":4.5469999999999997,"15min":4.5469999999999997},"last":0.55800000000000005}]},{"osd":5,"last update":"Tue Sep 10 19:18:04 2024","interfaces":[{"interface":"back","average":{"1min":2.1699999999999999,"5min":1.292,"15min":0.748},"min":{"1min":0.32400000000000001,"5min":0.253,"15min":0.23699999999999999},"max":{"1min":8.1120000000000001,"5min":11.621,"15min":11.621},"last":1.474},{"interface":"front","average":{"1min":1.5860000000000001,"5min":1.121,"15min":0.69199999999999995},"min":{"1min":0.372,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":5.9889999999999999,"5min":9.2680000000000007,"15min":9.2680000000000007},"last":0.621}]},{"osd":6,"last update":"Tue Sep 10 19:18:04 2024","interfaces":[{"interface":"back","average":{"1min":1.954,"5min":1.194,"15min":0.73499999999999999},"min":{"1min":0.33900000000000002,"5min":0.29099999999999998,"15min":0.26200000000000001},"max":{"1min":5.9550000000000001,"5min":8.2590000000000003,"15min":8.2590000000000003},"last":0.56999999999999995},{"interface":"front","average":{"1min":1.222,"5min":0.94699999999999995,"15min":0.64000000000000001},"min":{"1min":0.29999999999999999,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":4.2960000000000003,"5min":8.4130000000000003,"15min":8.4130000000000003},"last":0.496}]}]},{"osd":6,"up_from":47,"seq":201863463019,"num_pgs":83,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3373680,"kb_used_data":3063848,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":90371472,"statfs":{"total":95995035648,"available":92540387328,"internally_reserved":0,"allocated":3137380352,"data_stored":3136820396,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 19:17:46 2024","interfaces":[{"interface":"back","average":{"1min":0.504,"5min":2.1970000000000001,"15min":1.075},"min":{"1min":0.20999999999999999,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":2.0169999999999999,"5min":112.988,"15min":112.988},"last":2.1469999999999998},{"interface":"front","average":{"1min":0.90400000000000003,"5min":2.2839999999999998,"15min":1.121},"min":{"1min":0.221,"5min":0.221,"15min":0.221},"max":{"1min":3.742,"5min":113.18300000000001,"15min":113.18300000000001},"last":2.246}]},{"osd":1,"last update":"Tue Sep 10 19:17:28 2024","interfaces":[{"interface":"back","average":{"1min":0.878,"5min":0.58599999999999997,"15min":0.53800000000000003},"min":{"1min":0.27000000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":4.2919999999999998,"5min":4.2919999999999998,"15min":4.2919999999999998},"last":2.4849999999999999},{"interface":"front","average":{"1min":1.1379999999999999,"5min":0.75,"15min":0.68600000000000005},"min":{"1min":0.26400000000000001,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":2.5470000000000002,"5min":2.996,"15min":2.996},"last":0.29199999999999998}]},{"osd":2,"last update":"Tue Sep 10 19:17:46 2024","interfaces":[{"interface":"back","average":{"1min":0.58799999999999997,"5min":2.0209999999999999,"15min":1.034},"min":{"1min":0.29599999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":2.036,"5min":112.864,"15min":112.864},"last":2.4510000000000001},{"interface":"front","average":{"1min":0.93899999999999995,"5min":2.2069999999999999,"15min":1.0780000000000001},"min":{"1min":0.254,"5min":0.254,"15min":0.254},"max":{"1min":3.7789999999999999,"5min":112.822,"15min":112.822},"last":2.2799999999999998}]},{"osd":3,"last update":"Tue Sep 10 19:17:46 2024","interfaces":[{"interface":"back","average":{"1min":2.0350000000000001,"5min":2.7330000000000001,"15min":1.262},"min":{"1min":0.313,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":23.824999999999999,"5min":125.56399999999999,"15min":125.56399999999999},"last":0.33100000000000002},{"interface":"front","average":{"1min":1.9399999999999999,"5min":2.79,"15min":1.3140000000000001},"min":{"1min":0.307,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":24.814,"5min":125.47,"15min":125.47},"last":2.512}]},{"osd":4,"last update":"Tue Sep 10 19:17:46 2024","interfaces":[{"interface":"back","average":{"1min":1.679,"5min":0.84499999999999997,"15min":0.52200000000000002},"min":{"1min":0.29299999999999998,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":4.0330000000000004,"5min":4.0330000000000004,"15min":4.0330000000000004},"last":3.206},{"interface":"front","average":{"1min":1.284,"5min":0.77700000000000002,"15min":0.52200000000000002},"min":{"1min":0.28899999999999998,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":4.0750000000000002,"5min":4.0750000000000002,"15min":4.0750000000000002},"last":0.26100000000000001}]},{"osd":5,"last update":"Tue Sep 10 19:17:46 2024","interfaces":[{"interface":"back","average":{"1min":1.6659999999999999,"5min":1.3600000000000001,"15min":0.71299999999999997},"min":{"1min":0.32100000000000001,"5min":0.248,"15min":0.248},"max":{"1min":3.8029999999999999,"5min":44.765000000000001,"15min":44.765000000000001},"last":3.4140000000000001},{"interface":"front","average":{"1min":1.5780000000000001,"5min":1.4870000000000001,"15min":0.76500000000000001},"min":{"1min":0.34899999999999998,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":6.8230000000000004,"5min":43.036000000000001,"15min":43.036000000000001},"last":0.312}]},{"osd":7,"last update":"Tue Sep 10 19:18:08 2024","interfaces":[{"interface":"back","average":{"1min":1.1359999999999999,"5min":0.92200000000000004,"15min":0.61099999999999999},"min":{"1min":0.35099999999999998,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":3.375,"5min":8.9390000000000001,"15min":8.9390000000000001},"last":2.7629999999999999},{"interface":"front","average":{"1min":1.276,"5min":1.071,"15min":0.66900000000000004},"min":{"1min":0.33800000000000002,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":3.464,"5min":10.298,"15min":10.298},"last":1.6080000000000001}]}]},{"osd":5,"up_from":39,"seq":167503724656,"num_pgs":80,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3849892,"kb_used_data":3540060,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":89895260,"statfs":{"total":95995035648,"available":92052746240,"internally_reserved":0,"allocated":3625021440,"data_stored":3624474069,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,2],"upper_bound":16},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 19:17:19 2024","interfaces":[{"interface":"back","average":{"1min":1.7869999999999999,"5min":1.1379999999999999,"15min":0.69899999999999995},"min":{"1min":0.35999999999999999,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":5.5519999999999996,"5min":5.851,"15min":5.851},"last":0.36199999999999999},{"interface":"front","average":{"1min":1.1759999999999999,"5min":0.90600000000000003,"15min":0.621},"min":{"1min":0.253,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":3.7130000000000001,"5min":5.923,"15min":5.923},"last":1.0469999999999999}]},{"osd":1,"last update":"Tue Sep 10 19:17:23 2024","interfaces":[{"interface":"back","average":{"1min":1.3140000000000001,"5min":0.78400000000000003,"15min":0.69599999999999995},"min":{"1min":0.28599999999999998,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":3.77,"5min":3.77,"15min":3.77},"last":1.121},{"interface":"front","average":{"1min":1.3440000000000001,"5min":0.67100000000000004,"15min":0.55900000000000005},"min":{"1min":0.28399999999999997,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":5.0570000000000004,"5min":5.0570000000000004,"15min":5.0570000000000004},"last":0.41099999999999998}]},{"osd":2,"last update":"Tue Sep 10 19:17:19 2024","interfaces":[{"interface":"back","average":{"1min":2.117,"5min":1.0629999999999999,"15min":0.67600000000000005},"min":{"1min":0.378,"5min":0.253,"15min":0.253},"max":{"1min":9.9399999999999995,"5min":9.9399999999999995,"15min":9.9399999999999995},"last":0.45800000000000002},{"interface":"front","average":{"1min":1.238,"5min":1.0329999999999999,"15min":0.67500000000000004},"min":{"1min":0.33200000000000002,"5min":0.224,"15min":0.224},"max":{"1min":3.7909999999999999,"5min":6.3789999999999996,"15min":6.3789999999999996},"last":1.141}]},{"osd":3,"last update":"Tue Sep 10 19:17:19 2024","interfaces":[{"interface":"back","average":{"1min":2.1120000000000001,"5min":1.0309999999999999,"15min":0.68700000000000006},"min":{"1min":0.34100000000000003,"5min":0.224,"15min":0.224},"max":{"1min":9.9019999999999992,"5min":9.9019999999999992,"15min":9.9019999999999992},"last":0.39000000000000001},{"interface":"front","average":{"1min":1.2869999999999999,"5min":0.90400000000000003,"15min":0.65600000000000003},"min":{"1min":0.29899999999999999,"5min":0.248,"15min":0.248},"max":{"1min":3.7469999999999999,"5min":5.3869999999999996,"15min":5.3869999999999996},"last":1.077}]},{"osd":4,"last update":"Tue Sep 10 19:17:19 2024","interfaces":[{"interface":"back","average":{"1min":2.173,"5min":0.95499999999999996,"15min":0.56299999999999994},"min":{"1min":0.24399999999999999,"5min":0.23699999999999999,"15min":0.20699999999999999},"max":{"1min":10.414,"5min":10.414,"15min":10.414},"last":0.379},{"interface":"front","average":{"1min":2.3330000000000002,"5min":1.048,"15min":0.60099999999999998},"min":{"1min":0.313,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":10.518000000000001,"5min":10.518000000000001,"15min":10.518000000000001},"last":0.42399999999999999}]},{"osd":6,"last update":"Tue Sep 10 19:17:45 2024","interfaces":[{"interface":"back","average":{"1min":2.9289999999999998,"5min":1.331,"15min":0.73099999999999998},"min":{"1min":0.30099999999999999,"5min":0.27100000000000002,"15min":0.223},"max":{"1min":9.9670000000000005,"5min":9.9670000000000005,"15min":9.9670000000000005},"last":0.63700000000000001},{"interface":"front","average":{"1min":2.427,"5min":1.147,"15min":0.66400000000000003},"min":{"1min":0.32100000000000001,"5min":0.26900000000000002,"15min":0.25},"max":{"1min":10.446,"5min":10.446,"15min":10.446},"last":1.2030000000000001}]},{"osd":7,"last update":"Tue Sep 10 19:18:08 2024","interfaces":[{"interface":"back","average":{"1min":1.7430000000000001,"5min":1.125,"15min":0.71099999999999997},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":3.8399999999999999,"5min":5.9660000000000002,"15min":5.9660000000000002},"last":1.1000000000000001},{"interface":"front","average":{"1min":2.0409999999999999,"5min":1.0589999999999999,"15min":0.67400000000000004},"min":{"1min":0.36899999999999999,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":6.8840000000000003,"5min":6.8840000000000003,"15min":6.8840000000000003},"last":0.64500000000000002}]}]},{"osd":4,"up_from":32,"seq":137438953588,"num_pgs":78,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":4240472,"kb_used_data":3934736,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":89504680,"statfs":{"total":95995035648,"available":91652792320,"internally_reserved":0,"allocated":4029169664,"data_stored":4028632693,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 19:18:04 2024","interfaces":[{"interface":"back","average":{"1min":1.7150000000000001,"5min":1.4750000000000001,"15min":0.84899999999999998},"min":{"1min":0.20799999999999999,"5min":0.187,"15min":0.187},"max":{"1min":24.113,"5min":32.533000000000001,"15min":32.533000000000001},"last":0.40600000000000003},{"interface":"front","average":{"1min":1.9670000000000001,"5min":1.54,"15min":0.86699999999999999},"min":{"1min":0.23999999999999999,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":24.132999999999999,"5min":32.512999999999998,"15min":32.512999999999998},"last":0.67200000000000004}]},{"osd":1,"last update":"Tue Sep 10 19:17:23 2024","interfaces":[{"interface":"back","average":{"1min":1.296,"5min":0.76100000000000001,"15min":0.67200000000000004},"min":{"1min":0.27800000000000002,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":7.9770000000000003,"5min":7.9770000000000003,"15min":7.9770000000000003},"last":0.45300000000000001},{"interface":"front","average":{"1min":0.89800000000000002,"5min":0.71999999999999997,"15min":0.68999999999999995},"min":{"1min":0.27700000000000002,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":3.8260000000000001,"5min":3.8260000000000001,"15min":3.8260000000000001},"last":0.52800000000000002}]},{"osd":2,"last update":"Tue Sep 10 19:18:04 2024","interfaces":[{"interface":"back","average":{"1min":0.63700000000000001,"5min":0.91900000000000004,"15min":0.65600000000000003},"min":{"1min":0.32400000000000001,"5min":0.214,"15min":0.214},"max":{"1min":1.6739999999999999,"5min":6.8869999999999996,"15min":6.8869999999999996},"last":0.46500000000000002},{"interface":"front","average":{"1min":0.79600000000000004,"5min":0.88100000000000001,"15min":0.64700000000000002},"min":{"1min":0.28399999999999997,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":2.1960000000000002,"5min":7.085,"15min":7.085},"last":0.70099999999999996}]},{"osd":3,"last update":"Tue Sep 10 19:18:04 2024","interfaces":[{"interface":"back","average":{"1min":1.0880000000000001,"5min":1.304,"15min":0.79300000000000004},"min":{"1min":0.29499999999999998,"5min":0.252,"15min":0.252},"max":{"1min":5.625,"5min":35.451999999999998,"15min":35.451999999999998},"last":0.72199999999999998},{"interface":"front","average":{"1min":1.867,"5min":1.5349999999999999,"15min":0.879},"min":{"1min":0.27600000000000002,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":7.9210000000000003,"5min":32.880000000000003,"15min":32.880000000000003},"last":0.47999999999999998}]},{"osd":5,"last update":"Tue Sep 10 19:17:23 2024","interfaces":[{"interface":"back","average":{"1min":5.7119999999999997,"5min":1.577,"15min":0.78400000000000003},"min":{"1min":0.29599999999999999,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":67.379000000000005,"5min":67.379000000000005,"15min":67.379000000000005},"last":0.41799999999999998},{"interface":"front","average":{"1min":4.9100000000000001,"5min":1.409,"15min":0.73899999999999999},"min":{"1min":0.27500000000000002,"5min":0.248,"15min":0.223},"max":{"1min":67.542000000000002,"5min":67.542000000000002,"15min":67.542000000000002},"last":0.437}]},{"osd":6,"last update":"Tue Sep 10 19:17:55 2024","interfaces":[{"interface":"back","average":{"1min":1.121,"5min":0.82499999999999996,"15min":0.56100000000000005},"min":{"1min":0.315,"5min":0.253,"15min":0.253},"max":{"1min":5.5289999999999999,"5min":5.5289999999999999,"15min":5.5289999999999999},"last":0.59399999999999997},{"interface":"front","average":{"1min":1.6639999999999999,"5min":0.879,"15min":0.57499999999999996},"min":{"1min":0.247,"5min":0.247,"15min":0.247},"max":{"1min":7.9550000000000001,"5min":7.9550000000000001,"15min":7.9550000000000001},"last":0.625}]},{"osd":7,"last update":"Tue Sep 10 19:18:11 2024","interfaces":[{"interface":"back","average":{"1min":2.121,"5min":1.389,"15min":0.80400000000000005},"min":{"1min":0.29899999999999999,"5min":0.28100000000000003,"15min":0.27700000000000002},"max":{"1min":10.33,"5min":23.081,"15min":23.081},"last":0.51300000000000001},{"interface":"front","average":{"1min":1.627,"5min":1.3089999999999999,"15min":0.77100000000000002},"min":{"1min":0.23699999999999999,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":6.0190000000000001,"5min":22.666,"15min":22.666},"last":0.81999999999999995}]}]},{"osd":3,"up_from":26,"seq":111669149817,"num_pgs":84,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3826804,"kb_used_data":3525292,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":89918348,"statfs":{"total":95995035648,"available":92076388352,"internally_reserved":0,"allocated":3609899008,"data_stored":3609335418,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 19:17:37 2024","interfaces":[{"interface":"back","average":{"1min":0.32100000000000001,"5min":1.2929999999999999,"15min":0.69999999999999996},"min":{"1min":0.17499999999999999,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":0.72199999999999998,"5min":17.518000000000001,"15min":17.518000000000001},"last":0.16800000000000001},{"interface":"front","average":{"1min":0.59499999999999997,"5min":1.617,"15min":0.81899999999999995},"min":{"1min":0.224,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":2.3359999999999999,"5min":17.609000000000002,"15min":17.609000000000002},"last":0.40000000000000002}]},{"osd":1,"last update":"Tue Sep 10 19:17:20 2024","interfaces":[{"interface":"back","average":{"1min":0.46800000000000003,"5min":0.52500000000000002,"15min":0.53500000000000003},"min":{"1min":0.22500000000000001,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":1.0509999999999999,"5min":2.6059999999999999,"15min":2.6059999999999999},"last":0.26200000000000001},{"interface":"front","average":{"1min":0.58199999999999996,"5min":1.371,"15min":1.5029999999999999},"min":{"1min":0.255,"5min":0.255,"15min":0.255},"max":{"1min":1.5289999999999999,"5min":17.454999999999998,"15min":17.454999999999998},"last":0.35999999999999999}]},{"osd":2,"last update":"Tue Sep 10 19:17:37 2024","interfaces":[{"interface":"back","average":{"1min":0.66400000000000003,"5min":1.0740000000000001,"15min":0.65000000000000002},"min":{"1min":0.25900000000000001,"5min":0.186,"15min":0.186},"max":{"1min":3.125,"5min":5.3029999999999999,"15min":5.3029999999999999},"last":0.23599999999999999},{"interface":"front","average":{"1min":0.55000000000000004,"5min":1.3080000000000001,"15min":0.71399999999999997},"min":{"1min":0.23599999999999999,"5min":0.23300000000000001,"15min":0.19800000000000001},"max":{"1min":2.266,"5min":17.806000000000001,"15min":17.806000000000001},"last":0.33700000000000002}]},{"osd":4,"last update":"Tue Sep 10 19:18:05 2024","interfaces":[{"interface":"back","average":{"1min":0.91600000000000004,"5min":1.2370000000000001,"15min":0.82899999999999996},"min":{"1min":0.254,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":4.8639999999999999,"5min":17.834,"15min":17.834},"last":0.42799999999999999},{"interface":"front","average":{"1min":0.93899999999999995,"5min":1.496,"15min":0.94799999999999995},"min":{"1min":0.28100000000000003,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":2.3610000000000002,"5min":17.553000000000001,"15min":17.553000000000001},"last":0.52000000000000002}]},{"osd":5,"last update":"Tue Sep 10 19:17:20 2024","interfaces":[{"interface":"back","average":{"1min":5.2830000000000004,"5min":2.0369999999999999,"15min":1.0549999999999999},"min":{"1min":0.33300000000000002,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":80.942999999999998,"5min":80.942999999999998,"15min":80.942999999999998},"last":0.65900000000000003},{"interface":"front","average":{"1min":5.6630000000000003,"5min":2.4860000000000002,"15min":1.202},"min":{"1min":0.377,"5min":0.22,"15min":0.22},"max":{"1min":81.489000000000004,"5min":81.489000000000004,"15min":81.489000000000004},"last":0.623}]},{"osd":6,"last update":"Tue Sep 10 19:17:44 2024","interfaces":[{"interface":"back","average":{"1min":1.145,"5min":1.464,"15min":0.878},"min":{"1min":0.35599999999999998,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":4.1600000000000001,"5min":17.861000000000001,"15min":17.861000000000001},"last":0.38},{"interface":"front","average":{"1min":1.0349999999999999,"5min":1.2090000000000001,"15min":0.78400000000000003},"min":{"1min":0.38400000000000001,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":2.2759999999999998,"5min":17.687000000000001,"15min":17.687000000000001},"last":0.311}]},{"osd":7,"last update":"Tue Sep 10 19:17:12 2024","interfaces":[{"interface":"back","average":{"1min":1.083,"5min":1.3089999999999999,"15min":0.82099999999999995},"min":{"1min":0.40699999999999997,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":4.1059999999999999,"5min":18.093,"15min":18.093},"last":2.0339999999999998},{"interface":"front","average":{"1min":1.0980000000000001,"5min":1.4139999999999999,"15min":0.86099999999999999},"min":{"1min":0.35599999999999998,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":3.9929999999999999,"5min":17.780999999999999,"15min":17.780999999999999},"last":1.145}]}]},{"osd":2,"up_from":0,"seq":77309411454,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":0,"seq":592705486884,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":8,"seq":34359738504,"num_pgs":79,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2836268,"kb_used_data":2534756,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":90908884,"statfs":{"total":95995035648,"available":93090697216,"internally_reserved":0,"allocated":2595590144,"data_stored":2595034676,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Sep 10 19:17:24 2024","interfaces":[{"interface":"back","average":{"1min":0.38,"5min":0.371,"15min":0.36899999999999999},"min":{"1min":0.217,"5min":0.19,"15min":0.19},"max":{"1min":1.698,"5min":1.698,"15min":1.698},"last":0.436},{"interface":"front","average":{"1min":0.33300000000000002,"5min":0.40600000000000003,"15min":0.41799999999999998},"min":{"1min":0.249,"5min":0.249,"15min":0.249},"max":{"1min":0.72799999999999998,"5min":1.0900000000000001,"15min":1.0900000000000001},"last":0.33900000000000002}]},{"osd":2,"last update":"Tue Sep 10 19:17:24 2024","interfaces":[{"interface":"back","average":{"1min":0.41199999999999998,"5min":1.028,"15min":0.59499999999999997},"min":{"1min":0.23899999999999999,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":1.339,"5min":6.4640000000000004,"15min":6.4640000000000004},"last":0.32500000000000001},{"interface":"front","average":{"1min":0.42699999999999999,"5min":1.0129999999999999,"15min":0.58199999999999996},"min":{"1min":0.252,"5min":0.193,"15min":0.187},"max":{"1min":0.84499999999999997,"5min":9.9399999999999995,"15min":9.9399999999999995},"last":0.54400000000000004}]},{"osd":3,"last update":"Tue Sep 10 19:17:44 2024","interfaces":[{"interface":"back","average":{"1min":0.48199999999999998,"5min":1.0820000000000001,"15min":0.64200000000000002},"min":{"1min":0.30099999999999999,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.80300000000000005,"5min":6.5590000000000002,"15min":6.5590000000000002},"last":0.50900000000000001},{"interface":"front","average":{"1min":0.41199999999999998,"5min":1.2470000000000001,"15min":0.68899999999999995},"min":{"1min":0.26200000000000001,"5min":0.26200000000000001,"15min":0.23000000000000001},"max":{"1min":0.81599999999999995,"5min":7.2140000000000004,"15min":7.2140000000000004},"last":0.374}]},{"osd":4,"last update":"Tue Sep 10 19:17:59 2024","interfaces":[{"interface":"back","average":{"1min":1.1419999999999999,"5min":0.90900000000000003,"15min":0.67700000000000005},"min":{"1min":0.29999999999999999,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":5.0789999999999997,"5min":5.8499999999999996,"15min":5.8499999999999996},"last":0.90300000000000002},{"interface":"front","average":{"1min":0.71599999999999997,"5min":0.84499999999999997,"15min":0.67600000000000005},"min":{"1min":0.28899999999999998,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":2.4940000000000002,"5min":5.2640000000000002,"15min":5.2640000000000002},"last":0.60499999999999998}]},{"osd":5,"last update":"Tue Sep 10 19:17:24 2024","interfaces":[{"interface":"back","average":{"1min":1.3580000000000001,"5min":1.1160000000000001,"15min":0.754},"min":{"1min":0.32500000000000001,"5min":0.29499999999999998,"15min":0.28199999999999997},"max":{"1min":2.6240000000000001,"5min":5.3860000000000001,"15min":5.3860000000000001},"last":0.873},{"interface":"front","average":{"1min":0.90800000000000003,"5min":0.99199999999999999,"15min":0.70799999999999996},"min":{"1min":0.36699999999999999,"5min":0.253,"15min":0.253},"max":{"1min":4.165,"5min":5.9749999999999996,"15min":5.9749999999999996},"last":0.42099999999999999}]},{"osd":6,"last update":"Tue Sep 10 19:17:44 2024","interfaces":[{"interface":"back","average":{"1min":1.028,"5min":1.0720000000000001,"15min":0.73299999999999998},"min":{"1min":0.35799999999999998,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":2.101,"5min":5.8730000000000002,"15min":5.8730000000000002},"last":0.49299999999999999},{"interface":"front","average":{"1min":0.81399999999999995,"5min":1.03,"15min":0.73299999999999998},"min":{"1min":0.34399999999999997,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":1.8560000000000001,"5min":5.2850000000000001,"15min":5.2850000000000001},"last":0.45200000000000001}]},{"osd":7,"last update":"Tue Sep 10 19:18:12 2024","interfaces":[{"interface":"back","average":{"1min":0.80600000000000005,"5min":0.98399999999999999,"15min":0.74199999999999999},"min":{"1min":0.35599999999999998,"5min":0.318,"15min":0.318},"max":{"1min":1.837,"5min":5.1479999999999997,"15min":5.1479999999999997},"last":0.57499999999999996},{"interface":"front","average":{"1min":1.228,"5min":1.159,"15min":0.76700000000000002},"min":{"1min":0.36699999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":5.3410000000000002,"5min":6.0469999999999997,"15min":6.0469999999999997},"last":0.56399999999999995}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":966,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1236,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1236,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":362,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":1760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":2118,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1018,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":2118,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1540,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1942,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":2162,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":2593062912,"data_stored":2593062912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":3570728960,"data_stored":3570728960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4001431552,"data_stored":4001431552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":3599437824,"data_stored":3599433752,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":3130855424,"data_stored":3130851352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":3570274304,"data_stored":3570270232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T19:18:21.137 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph osd pool set .mgr pgp_num 1 2024-09-10T19:18:21.200 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:18:21.239 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:18:21.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:18:21.240 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:18:21.315 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:18:20 smithi183 ceph-mon[29736]: from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:18:21.315 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:18:20 smithi183 ceph-mon[29736]: pgmap v353: 158 pgs: 1 active+recovering+remapped, 1 active+remapped+backfilling, 1 active+recovering, 4 active+remapped+backfill_toofull, 151 active+clean; 6.0 GiB data, 21 GiB used, 516 GiB / 536 GiB avail; 1023 B/s rd, 0 B/s wr, 1.19k op/s; 25555/293661 objects misplaced (8.702%); 9.0 MiB/s, 144 objects/s recovering 2024-09-10T19:18:21.315 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:18:20 smithi183 ceph-mon[29736]: 6.18 scrub starts 2024-09-10T19:18:21.315 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:18:20 smithi183 ceph-mon[29736]: from='client.? 172.21.15.183:0/4144838086' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-10T19:18:21.316 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:18:20 smithi183 ceph-mon[41562]: from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:18:21.316 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:18:20 smithi183 ceph-mon[41562]: pgmap v353: 158 pgs: 1 active+recovering+remapped, 1 active+remapped+backfilling, 1 active+recovering, 4 active+remapped+backfill_toofull, 151 active+clean; 6.0 GiB data, 21 GiB used, 516 GiB / 536 GiB avail; 1023 B/s rd, 0 B/s wr, 1.19k op/s; 25555/293661 objects misplaced (8.702%); 9.0 MiB/s, 144 objects/s recovering 2024-09-10T19:18:21.316 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:18:20 smithi183 ceph-mon[41562]: 6.18 scrub starts 2024-09-10T19:18:21.316 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:18:20 smithi183 ceph-mon[41562]: from='client.? 172.21.15.183:0/4144838086' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-10T19:18:21.340 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:18:21.373 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:18:21.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:18:21.374 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:18:21.465 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:18:20 smithi191 ceph-mon[32884]: from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:18:21.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:18:20 smithi191 ceph-mon[32884]: pgmap v353: 158 pgs: 1 active+recovering+remapped, 1 active+remapped+backfilling, 1 active+recovering, 4 active+remapped+backfill_toofull, 151 active+clean; 6.0 GiB data, 21 GiB used, 516 GiB / 536 GiB avail; 1023 B/s rd, 0 B/s wr, 1.19k op/s; 25555/293661 objects misplaced (8.702%); 9.0 MiB/s, 144 objects/s recovering 2024-09-10T19:18:21.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:18:20 smithi191 ceph-mon[32884]: 6.18 scrub starts 2024-09-10T19:18:21.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:18:20 smithi191 ceph-mon[32884]: from='client.? 172.21.15.183:0/4144838086' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-10T19:18:21.475 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:18:21.499 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:18:21.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:18:21.500 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:18:21.600 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:18:21.628 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:18:21.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:18:21.629 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:18:21.681 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Sep 10 19:18:21 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-y[29989]: ::ffff:172.21.15.191 - - [10/Sep/2024:19:18:21] "GET /metrics HTTP/1.1" 200 224301 "" "Prometheus/2.43.0" 2024-09-10T19:18:21.729 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:18:21.754 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:18:21.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:18:21.755 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:18:21.856 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:18:21.888 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:18:21.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:18:21.889 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:18:21.990 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:18:22.022 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:18:22.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:18:22.023 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:18:22.124 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:18:22.151 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:18:22.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:18:22.152 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:18:22.177 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:18:21 smithi183 ceph-mon[29736]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-10T19:18:22.177 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:18:21 smithi183 ceph-mon[29736]: from='client.? 172.21.15.183:0/4144838086' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-10T19:18:22.177 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:18:21 smithi183 ceph-mon[29736]: osdmap e197: 8 total, 8 up, 6 in 2024-09-10T19:18:22.178 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:18:21 smithi183 ceph-mon[41562]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-10T19:18:22.178 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:18:21 smithi183 ceph-mon[41562]: from='client.? 172.21.15.183:0/4144838086' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-10T19:18:22.178 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:18:21 smithi183 ceph-mon[41562]: osdmap e197: 8 total, 8 up, 6 in 2024-09-10T19:18:22.184 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:18:22.197 INFO:teuthology.orchestra.run.smithi183.stdout:{ 2024-09-10T19:18:22.197 INFO:teuthology.orchestra.run.smithi183.stdout: "ops": [], 2024-09-10T19:18:22.197 INFO:teuthology.orchestra.run.smithi183.stdout: "num_ops": 0 2024-09-10T19:18:22.197 INFO:teuthology.orchestra.run.smithi183.stdout:} 2024-09-10T19:18:22.253 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:18:22.277 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:18:22.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:18:22.277 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:18:22.378 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:18:22.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:18:22.408 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:18:22.409 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:18:22.465 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:18:21 smithi191 ceph-mon[32884]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-10T19:18:22.465 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:18:21 smithi191 ceph-mon[32884]: from='client.? 172.21.15.183:0/4144838086' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-10T19:18:22.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:18:21 smithi191 ceph-mon[32884]: osdmap e197: 8 total, 8 up, 6 in 2024-09-10T19:18:22.509 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:18:22.534 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:18:22.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:18:22.535 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:18:22.636 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:18:22.670 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:18:22.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:18:22.671 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:18:22.772 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:18:22.796 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:18:22.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:18:22.797 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:18:22.898 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:18:22.924 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:18:22.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:18:22.925 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:18:22.947 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph daemon osd.2 dump_blocked_ops 2024-09-10T19:18:23.027 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:18:23.056 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:18:23.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:18:23.057 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:18:23.158 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:18:23.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:18:22 smithi183 ceph-mon[29736]: pgmap v356: 158 pgs: 1 active+recovering+remapped, 1 active+remapped+backfilling, 1 active+recovering, 4 active+remapped+backfill_toofull, 151 active+clean; 5.6 GiB data, 20 GiB used, 517 GiB / 536 GiB avail; 852 B/s rd, 0 B/s wr, 967 op/s; 22818/276252 objects misplaced (8.260%); 6.9 MiB/s, 110 objects/s recovering 2024-09-10T19:18:23.181 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:18:22 smithi183 ceph-mon[41562]: pgmap v356: 158 pgs: 1 active+recovering+remapped, 1 active+remapped+backfilling, 1 active+recovering, 4 active+remapped+backfill_toofull, 151 active+clean; 5.6 GiB data, 20 GiB used, 517 GiB / 536 GiB avail; 852 B/s rd, 0 B/s wr, 967 op/s; 22818/276252 objects misplaced (8.260%); 6.9 MiB/s, 110 objects/s recovering 2024-09-10T19:18:23.184 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:18:23.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:18:23.184 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:18:23.285 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:18:23.312 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:18:23.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:18:23.313 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:18:23.413 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:18:23.436 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:18:23.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:18:23.437 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:18:23.465 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:18:22 smithi191 ceph-mon[32884]: pgmap v356: 158 pgs: 1 active+recovering+remapped, 1 active+remapped+backfilling, 1 active+recovering, 4 active+remapped+backfill_toofull, 151 active+clean; 5.6 GiB data, 20 GiB used, 517 GiB / 536 GiB avail; 852 B/s rd, 0 B/s wr, 967 op/s; 22818/276252 objects misplaced (8.260%); 6.9 MiB/s, 110 objects/s recovering 2024-09-10T19:18:23.538 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:18:23.572 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:18:23.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:18:23.572 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:18:23.673 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:18:23.700 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:18:23.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:18:23.701 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:18:23.736 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:18:23.802 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:18:23.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:18:23.823 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:18:23.823 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:18:23.924 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:18:23.947 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:18:23.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:18:23.948 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:18:23.961 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph osd set noscrub 2024-09-10T19:18:24.049 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:18:24.078 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:18:24.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:18:24.079 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:18:24.179 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:18:24.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:18:23 smithi183 ceph-mon[29736]: osdmap e198: 8 total, 8 up, 6 in 2024-09-10T19:18:24.181 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:18:24 smithi183 ceph-mon[41562]: osdmap e198: 8 total, 8 up, 6 in 2024-09-10T19:18:24.208 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:18:24.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:18:24.209 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:18:24.310 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:18:24.339 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:18:24.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:18:24.340 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:18:24.424 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:18:24.441 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:18:24.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:18:24 smithi191 ceph-mon[32884]: osdmap e198: 8 total, 8 up, 6 in 2024-09-10T19:18:24.479 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:18:24.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:18:24.481 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:18:24.581 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:18:24.607 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:18:24.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:18:24.608 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:18:24.709 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:18:24.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:18:24.742 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:18:24.742 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:18:24.843 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:18:24.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:18:24.869 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:18:24.870 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:18:24.970 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:18:24.995 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:18:24.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:18:24.997 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:18:25.097 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:18:25.122 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:18:25.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:18:25.123 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:18:25.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:18:25 smithi183 ceph-mon[29736]: pgmap v358: 158 pgs: 1 active+recovering+remapped, 1 active+remapped+backfilling, 1 active+recovering, 4 active+remapped+backfill_toofull, 151 active+clean; 5.6 GiB data, 20 GiB used, 517 GiB / 536 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1.21k op/s; 22818/276252 objects misplaced (8.260%); 8.7 MiB/s, 139 objects/s recovering 2024-09-10T19:18:25.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:18:25 smithi183 ceph-mon[29736]: osdmap e199: 8 total, 8 up, 6 in 2024-09-10T19:18:25.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:18:25 smithi183 ceph-mon[29736]: osdmap e200: 8 total, 8 up, 6 in 2024-09-10T19:18:25.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:18:25 smithi183 ceph-mon[41562]: pgmap v358: 158 pgs: 1 active+recovering+remapped, 1 active+remapped+backfilling, 1 active+recovering, 4 active+remapped+backfill_toofull, 151 active+clean; 5.6 GiB data, 20 GiB used, 517 GiB / 536 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1.21k op/s; 22818/276252 objects misplaced (8.260%); 8.7 MiB/s, 139 objects/s recovering 2024-09-10T19:18:25.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:18:25 smithi183 ceph-mon[41562]: osdmap e199: 8 total, 8 up, 6 in 2024-09-10T19:18:25.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:18:25 smithi183 ceph-mon[41562]: osdmap e200: 8 total, 8 up, 6 in 2024-09-10T19:18:25.224 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:18:25.246 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:18:25.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:18:25.247 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:18:25.348 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:18:25.381 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:18:25.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:18:25.382 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:18:25.465 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:18:25 smithi191 ceph-mon[32884]: pgmap v358: 158 pgs: 1 active+recovering+remapped, 1 active+remapped+backfilling, 1 active+recovering, 4 active+remapped+backfill_toofull, 151 active+clean; 5.6 GiB data, 20 GiB used, 517 GiB / 536 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1.21k op/s; 22818/276252 objects misplaced (8.260%); 8.7 MiB/s, 139 objects/s recovering 2024-09-10T19:18:25.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:18:25 smithi191 ceph-mon[32884]: osdmap e199: 8 total, 8 up, 6 in 2024-09-10T19:18:25.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:18:25 smithi191 ceph-mon[32884]: osdmap e200: 8 total, 8 up, 6 in 2024-09-10T19:18:25.483 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:18:25.515 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:18:25.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:18:25.516 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:18:25.566 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:18:25.568 INFO:teuthology.orchestra.run.smithi183.stdout:osd.0: {} 2024-09-10T19:18:25.568 INFO:teuthology.orchestra.run.smithi183.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-10T19:18:25.588 INFO:teuthology.orchestra.run.smithi183.stdout:osd.1: {} 2024-09-10T19:18:25.589 INFO:teuthology.orchestra.run.smithi183.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-10T19:18:25.607 INFO:teuthology.orchestra.run.smithi183.stdout:osd.2: {} 2024-09-10T19:18:25.607 INFO:teuthology.orchestra.run.smithi183.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-10T19:18:25.616 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:18:25.624 INFO:teuthology.orchestra.run.smithi183.stdout:osd.3: {} 2024-09-10T19:18:25.625 INFO:teuthology.orchestra.run.smithi183.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-10T19:18:25.646 INFO:teuthology.orchestra.run.smithi183.stdout:osd.4: {} 2024-09-10T19:18:25.646 INFO:teuthology.orchestra.run.smithi183.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-10T19:18:25.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:18:25.647 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:18:25.647 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:18:25.663 INFO:teuthology.orchestra.run.smithi183.stdout:osd.5: {} 2024-09-10T19:18:25.664 INFO:teuthology.orchestra.run.smithi183.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-10T19:18:25.682 INFO:teuthology.orchestra.run.smithi183.stdout:osd.6: {} 2024-09-10T19:18:25.682 INFO:teuthology.orchestra.run.smithi183.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-10T19:18:25.700 INFO:teuthology.orchestra.run.smithi183.stdout:osd.7: {} 2024-09-10T19:18:25.700 INFO:teuthology.orchestra.run.smithi183.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-10T19:18:25.748 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:18:25.771 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:18:25.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:18:25.772 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:18:25.873 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:18:25.897 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:18:25.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:18:25.898 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:18:25.999 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:18:26.026 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:18:26.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:18:26.027 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:18:26.128 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:18:26.151 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:18:26.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:18:26.153 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:18:26.254 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:18:26.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:18:26.281 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:18:26.281 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:18:26.381 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:18:26.411 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:18:26.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:18:26.411 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:18:26.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:18:26 smithi183 ceph-mon[29736]: 4.6 scrub starts 2024-09-10T19:18:26.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:18:26 smithi183 ceph-mon[29736]: from='client.? 172.21.15.183:0/824490169' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T19:18:26.431 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:18:26 smithi183 ceph-mon[41562]: 4.6 scrub starts 2024-09-10T19:18:26.431 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:18:26 smithi183 ceph-mon[41562]: from='client.? 172.21.15.183:0/824490169' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T19:18:26.465 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:18:26 smithi191 ceph-mon[32884]: 4.6 scrub starts 2024-09-10T19:18:26.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:18:26 smithi191 ceph-mon[32884]: from='client.? 172.21.15.183:0/824490169' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T19:18:26.512 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:18:26.546 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:18:26.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:18:26.546 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:18:26.647 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:18:26.683 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:18:26.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:18:26.684 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:18:26.785 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:18:26.812 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:18:26.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:18:26.814 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:18:26.914 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:18:26.948 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:18:26.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:18:26.949 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:18:27.050 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:18:27.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:18:27.079 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:18:27.079 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:18:27.121 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:18:27 smithi183 ceph-mon[29736]: pgmap v361: 158 pgs: 1 active+recovering+remapped, 1 active+recovering, 4 active+remapped+backfill_toofull, 152 active+clean; 5.3 GiB data, 19 GiB used, 517 GiB / 536 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 956 op/s; 20684/262590 objects misplaced (7.877%); 8.2 MiB/s, 130 objects/s recovering 2024-09-10T19:18:27.121 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:18:27 smithi183 ceph-mon[41562]: pgmap v361: 158 pgs: 1 active+recovering+remapped, 1 active+recovering, 4 active+remapped+backfill_toofull, 152 active+clean; 5.3 GiB data, 19 GiB used, 517 GiB / 536 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 956 op/s; 20684/262590 objects misplaced (7.877%); 8.2 MiB/s, 130 objects/s recovering 2024-09-10T19:18:27.180 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:18:27.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:18:27.205 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:18:27.206 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:18:27.306 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:18:27.330 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:18:27.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:18:27.331 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:18:27.366 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:18:27 smithi191 ceph-mon[32884]: pgmap v361: 158 pgs: 1 active+recovering+remapped, 1 active+recovering, 4 active+remapped+backfill_toofull, 152 active+clean; 5.3 GiB data, 19 GiB used, 517 GiB / 536 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 956 op/s; 20684/262590 objects misplaced (7.877%); 8.2 MiB/s, 130 objects/s recovering 2024-09-10T19:18:27.432 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:18:27.463 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:18:27.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:18:27.464 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:18:27.565 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:18:27.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:18:27.597 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:18:27.597 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:18:27.698 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:18:27.715 INFO:journalctl@ceph.mgr.x.smithi191.stdout:Sep 10 19:18:27 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-x[34967]: ::ffff:172.21.15.191 - - [10/Sep/2024:19:18:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T19:18:27.733 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:18:27.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:18:27.734 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:18:27.834 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:18:27.860 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:18:27.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:18:27.861 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:18:27.961 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:18:27.985 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:18:27.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:18:27.986 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:18:28.087 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:18:28.118 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:18:28.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:18:28.119 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:18:28.220 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:18:28.248 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:18:28.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:18:28.249 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:18:28.350 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:18:28.379 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:18:28.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:18:28.380 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:18:28.431 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:18:28 smithi183 ceph-mon[41562]: 4.b scrub starts 2024-09-10T19:18:28.431 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:18:28 smithi183 ceph-mon[41562]: 4.a scrub starts 2024-09-10T19:18:28.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:18:28 smithi183 ceph-mon[29736]: 4.b scrub starts 2024-09-10T19:18:28.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:18:28 smithi183 ceph-mon[29736]: 4.a scrub starts 2024-09-10T19:18:28.465 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:18:28 smithi191 ceph-mon[32884]: 4.b scrub starts 2024-09-10T19:18:28.465 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:18:28 smithi191 ceph-mon[32884]: 4.a scrub starts 2024-09-10T19:18:28.481 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:18:28.505 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:18:28.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:18:28.506 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:18:28.607 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:18:28.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:18:28.632 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:18:28.632 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:18:28.733 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:18:28.759 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:18:28.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:18:28.760 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:18:28.861 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:18:28.886 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:18:28.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:18:28.887 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:18:28.988 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:18:29.022 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:18:29.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:18:29.023 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:18:29.124 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:18:29.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:18:29.152 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:18:29.153 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:18:29.254 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:18:29.277 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:18:29 smithi183 ceph-mon[29736]: pgmap v362: 158 pgs: 1 active+recovering+remapped, 1 active+recovering, 4 active+remapped+backfill_toofull, 152 active+clean; 5.3 GiB data, 19 GiB used, 517 GiB / 536 GiB avail; 853 B/s rd, 0 B/s wr, 759 op/s; 20684/262590 objects misplaced (7.877%); 6.5 MiB/s, 103 objects/s recovering 2024-09-10T19:18:29.277 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:18:29 smithi183 ceph-mon[29736]: 4.12 scrub starts 2024-09-10T19:18:29.277 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:18:29 smithi183 ceph-mon[29736]: from='mgr.14556 ' entity='mgr.y' 2024-09-10T19:18:29.277 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:18:29 smithi183 ceph-mon[29736]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:18:29.278 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:18:29 smithi183 ceph-mon[41562]: pgmap v362: 158 pgs: 1 active+recovering+remapped, 1 active+recovering, 4 active+remapped+backfill_toofull, 152 active+clean; 5.3 GiB data, 19 GiB used, 517 GiB / 536 GiB avail; 853 B/s rd, 0 B/s wr, 759 op/s; 20684/262590 objects misplaced (7.877%); 6.5 MiB/s, 103 objects/s recovering 2024-09-10T19:18:29.278 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:18:29 smithi183 ceph-mon[41562]: 4.12 scrub starts 2024-09-10T19:18:29.278 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:18:29 smithi183 ceph-mon[41562]: from='mgr.14556 ' entity='mgr.y' 2024-09-10T19:18:29.278 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:18:29 smithi183 ceph-mon[41562]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:18:29.284 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:18:29.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:18:29.285 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:18:29.309 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:18:29 smithi191 ceph-mon[32884]: pgmap v362: 158 pgs: 1 active+recovering+remapped, 1 active+recovering, 4 active+remapped+backfill_toofull, 152 active+clean; 5.3 GiB data, 19 GiB used, 517 GiB / 536 GiB avail; 853 B/s rd, 0 B/s wr, 759 op/s; 20684/262590 objects misplaced (7.877%); 6.5 MiB/s, 103 objects/s recovering 2024-09-10T19:18:29.309 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:18:29 smithi191 ceph-mon[32884]: 4.12 scrub starts 2024-09-10T19:18:29.309 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:18:29 smithi191 ceph-mon[32884]: from='mgr.14556 ' entity='mgr.y' 2024-09-10T19:18:29.309 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:18:29 smithi191 ceph-mon[32884]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:18:29.386 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:18:29.413 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:18:29.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:18:29.413 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:18:29.514 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:18:29.539 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:18:29.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:18:29.539 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:18:29.640 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:18:29.667 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:18:29.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:18:29.668 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:18:29.716 INFO:journalctl@ceph.iscsi.iscsi.a.smithi191.stdout:Sep 10 19:18:29 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-iscsi-iscsi-a[61816]: debug there is no tcmu-runner data available 2024-09-10T19:18:29.769 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:18:29.800 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:18:29.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:18:29.801 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:18:29.902 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:18:29.939 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:18:29.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:18:29.941 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:18:30.042 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:18:30.074 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:18:30.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:18:30.075 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:18:30.176 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:18:30.213 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:18:30.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:18:30.214 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:18:30.315 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:18:30.355 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:18:30 smithi183 ceph-mon[29736]: 5.8 scrub starts 2024-09-10T19:18:30.355 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:18:30 smithi183 ceph-mon[29736]: 4.1f scrub starts 2024-09-10T19:18:30.355 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:18:30 smithi183 ceph-mon[41562]: 5.8 scrub starts 2024-09-10T19:18:30.356 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:18:30 smithi183 ceph-mon[41562]: 4.1f scrub starts 2024-09-10T19:18:30.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:18:30.359 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:18:30.359 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:18:30.460 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:18:30.465 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:18:30 smithi191 ceph-mon[32884]: 5.8 scrub starts 2024-09-10T19:18:30.465 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:18:30 smithi191 ceph-mon[32884]: 4.1f scrub starts 2024-09-10T19:18:30.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:18:30.493 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:18:30.493 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:18:30.595 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:18:30.625 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:18:30.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:18:30.626 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:18:30.726 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:18:30.769 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:18:30.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:18:30.770 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:18:30.871 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:18:30.898 INFO:teuthology.orchestra.run.smithi183.stdout:ERROR: (22) Invalid argument 2024-09-10T19:18:30.899 INFO:teuthology.orchestra.run.smithi183.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-10T19:18:30.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:18:30.911 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:18:30.912 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:18:31.013 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:18:31.041 INFO:teuthology.orchestra.run.smithi183.stderr:noscrub is set 2024-09-10T19:18:31.047 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:18:31.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:18:31.048 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:18:31.149 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:18:31.182 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:18:31 smithi183 ceph-mon[29736]: from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:18:31.183 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:18:31 smithi183 ceph-mon[29736]: pgmap v363: 158 pgs: 1 active+recovering+remapped, 1 active+recovering, 4 active+remapped+backfill_toofull, 152 active+clean; 5.3 GiB data, 19 GiB used, 517 GiB / 536 GiB avail; 756 B/s rd, 0 B/s wr, 673 op/s; 20684/262590 objects misplaced (7.877%); 5.7 MiB/s, 91 objects/s recovering 2024-09-10T19:18:31.183 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:18:31 smithi183 ceph-mon[29736]: from='client.? 172.21.15.183:0/751684424' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-10T19:18:31.184 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:18:31 smithi183 ceph-mon[41562]: from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:18:31.184 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:18:31 smithi183 ceph-mon[41562]: pgmap v363: 158 pgs: 1 active+recovering+remapped, 1 active+recovering, 4 active+remapped+backfill_toofull, 152 active+clean; 5.3 GiB data, 19 GiB used, 517 GiB / 536 GiB avail; 756 B/s rd, 0 B/s wr, 673 op/s; 20684/262590 objects misplaced (7.877%); 5.7 MiB/s, 91 objects/s recovering 2024-09-10T19:18:31.184 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:18:31 smithi183 ceph-mon[41562]: from='client.? 172.21.15.183:0/751684424' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-10T19:18:31.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:18:31.186 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:18:31.186 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:18:31.287 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:18:31.321 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:18:31.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:18:31.322 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:18:31.423 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:18:31.457 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:18:31.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:18:31.458 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:18:31.465 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:18:31 smithi191 ceph-mon[32884]: from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:18:31.465 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:18:31 smithi191 ceph-mon[32884]: pgmap v363: 158 pgs: 1 active+recovering+remapped, 1 active+recovering, 4 active+remapped+backfill_toofull, 152 active+clean; 5.3 GiB data, 19 GiB used, 517 GiB / 536 GiB avail; 756 B/s rd, 0 B/s wr, 673 op/s; 20684/262590 objects misplaced (7.877%); 5.7 MiB/s, 91 objects/s recovering 2024-09-10T19:18:31.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:18:31 smithi191 ceph-mon[32884]: from='client.? 172.21.15.183:0/751684424' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-10T19:18:31.559 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:18:31.587 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:18:31.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:18:31.588 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:18:31.681 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Sep 10 19:18:31 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-y[29989]: ::ffff:172.21.15.191 - - [10/Sep/2024:19:18:31] "GET /metrics HTTP/1.1" 200 224286 "" "Prometheus/2.43.0" 2024-09-10T19:18:31.689 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:18:31.720 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:18:31.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:18:31.721 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:18:31.822 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:18:31.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:18:31.846 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:18:31.846 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:18:31.946 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:18:31.984 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:18:31.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:18:31.985 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:18:32.059 INFO:teuthology.orchestra.run.smithi183.stderr:set pool 1 pgp_num to 1 2024-09-10T19:18:32.086 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:18:32.118 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:18:32.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:18:32.119 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:18:32.140 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-10T19:18:32.221 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:18:32.255 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:18:32.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:18:32.257 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:18:32.357 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:18:32.388 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:18:32.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:18:32.389 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:18:32.408 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:18:32 smithi183 ceph-mon[41562]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T19:18:32.408 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:18:32 smithi183 ceph-mon[41562]: from='client.? 172.21.15.183:0/751684424' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-10T19:18:32.409 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:18:32 smithi183 ceph-mon[41562]: osdmap e201: 8 total, 8 up, 6 in 2024-09-10T19:18:32.409 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:18:32 smithi183 ceph-mon[41562]: from='client.? 172.21.15.183:0/2458078523' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "1"}]: dispatch 2024-09-10T19:18:32.409 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:18:32 smithi183 ceph-mon[29736]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T19:18:32.409 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:18:32 smithi183 ceph-mon[29736]: from='client.? 172.21.15.183:0/751684424' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-10T19:18:32.409 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:18:32 smithi183 ceph-mon[29736]: osdmap e201: 8 total, 8 up, 6 in 2024-09-10T19:18:32.409 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:18:32 smithi183 ceph-mon[29736]: from='client.? 172.21.15.183:0/2458078523' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "1"}]: dispatch 2024-09-10T19:18:32.465 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:18:32 smithi191 ceph-mon[32884]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T19:18:32.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:18:32 smithi191 ceph-mon[32884]: from='client.? 172.21.15.183:0/751684424' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-10T19:18:32.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:18:32 smithi191 ceph-mon[32884]: osdmap e201: 8 total, 8 up, 6 in 2024-09-10T19:18:32.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:18:32 smithi191 ceph-mon[32884]: from='client.? 172.21.15.183:0/2458078523' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "1"}]: dispatch 2024-09-10T19:18:32.490 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:18:32.516 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:18:32.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:18:32.517 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:18:32.617 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:18:32.650 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:18:32.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:18:32.651 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:18:32.751 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:18:32.787 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:18:32.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:18:32.789 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:18:32.890 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:18:32.911 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:18:32.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:18:32.912 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:18:33.013 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:18:33.043 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:18:33.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:18:33.044 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:18:33.145 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:18:33.171 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:18:33.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:18:33.172 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:18:33.273 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:18:33.300 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:18:33.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:18:33.301 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:18:33.402 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:18:33.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:18:33 smithi183 ceph-mon[29736]: pgmap v365: 158 pgs: 1 active+recovering+remapped, 1 active+recovering, 4 active+remapped+backfill_toofull, 152 active+clean; 5.0 GiB data, 18 GiB used, 518 GiB / 536 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.25k op/s; 19283/247284 objects misplaced (7.798%); 7.4 MiB/s, 118 objects/s recovering 2024-09-10T19:18:33.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:18:33 smithi183 ceph-mon[29736]: from='client.? 172.21.15.183:0/2458078523' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "1"}]': finished 2024-09-10T19:18:33.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:18:33 smithi183 ceph-mon[29736]: osdmap e202: 8 total, 8 up, 6 in 2024-09-10T19:18:33.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:18:33 smithi183 ceph-mon[29736]: purged_snaps scrub starts 2024-09-10T19:18:33.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:18:33 smithi183 ceph-mon[29736]: purged_snaps scrub ok 2024-09-10T19:18:33.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:18:33 smithi183 ceph-mon[41562]: pgmap v365: 158 pgs: 1 active+recovering+remapped, 1 active+recovering, 4 active+remapped+backfill_toofull, 152 active+clean; 5.0 GiB data, 18 GiB used, 518 GiB / 536 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.25k op/s; 19283/247284 objects misplaced (7.798%); 7.4 MiB/s, 118 objects/s recovering 2024-09-10T19:18:33.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:18:33 smithi183 ceph-mon[41562]: from='client.? 172.21.15.183:0/2458078523' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "1"}]': finished 2024-09-10T19:18:33.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:18:33 smithi183 ceph-mon[41562]: osdmap e202: 8 total, 8 up, 6 in 2024-09-10T19:18:33.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:18:33 smithi183 ceph-mon[41562]: purged_snaps scrub starts 2024-09-10T19:18:33.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:18:33 smithi183 ceph-mon[41562]: purged_snaps scrub ok 2024-09-10T19:18:33.432 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:18:33.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:18:33.433 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:18:33.465 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:18:33 smithi191 ceph-mon[32884]: pgmap v365: 158 pgs: 1 active+recovering+remapped, 1 active+recovering, 4 active+remapped+backfill_toofull, 152 active+clean; 5.0 GiB data, 18 GiB used, 518 GiB / 536 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.25k op/s; 19283/247284 objects misplaced (7.798%); 7.4 MiB/s, 118 objects/s recovering 2024-09-10T19:18:33.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:18:33 smithi191 ceph-mon[32884]: from='client.? 172.21.15.183:0/2458078523' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "1"}]': finished 2024-09-10T19:18:33.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:18:33 smithi191 ceph-mon[32884]: osdmap e202: 8 total, 8 up, 6 in 2024-09-10T19:18:33.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:18:33 smithi191 ceph-mon[32884]: purged_snaps scrub starts 2024-09-10T19:18:33.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:18:33 smithi191 ceph-mon[32884]: purged_snaps scrub ok 2024-09-10T19:18:33.534 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:18:33.561 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:18:33.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:18:33.563 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:18:33.663 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:18:33.693 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:18:33.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:18:33.694 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:18:33.795 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:18:33.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:18:33.828 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:18:33.829 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:18:33.929 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:18:33.957 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:18:33.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:18:33.958 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:18:34.059 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:18:34.089 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:18:34.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:18:34.090 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:18:34.191 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:18:34.221 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:18:34.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:18:34.222 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:18:34.322 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:18:34.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:18:34.349 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:18:34.349 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:18:34.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:18:34 smithi183 ceph-mon[29736]: osdmap e203: 8 total, 8 up, 6 in 2024-09-10T19:18:34.431 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:18:34 smithi183 ceph-mon[41562]: osdmap e203: 8 total, 8 up, 6 in 2024-09-10T19:18:34.450 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:18:34.465 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:18:34 smithi191 ceph-mon[32884]: osdmap e203: 8 total, 8 up, 6 in 2024-09-10T19:18:34.472 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:18:34.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:18:34.473 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:18:34.574 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:18:34.601 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:18:34.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:18:34.603 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:18:34.703 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:18:34.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:18:34.740 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:18:34.740 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:18:34.841 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:18:34.879 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:18:34.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:18:34.880 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:18:34.981 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:18:35.001 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:18:35.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:18:35.002 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:18:35.103 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:18:35.132 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:18:35.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:18:35.133 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:18:35.234 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:18:35.269 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:18:35.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:18:35.269 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:18:35.370 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:18:35.407 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:18:35.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:18:35.408 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:18:35.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:18:35 smithi183 ceph-mon[29736]: pgmap v368: 158 pgs: 1 active+recovering+remapped, 1 active+recovering, 4 active+remapped+backfill_toofull, 152 active+clean; 5.0 GiB data, 18 GiB used, 518 GiB / 536 GiB avail; 853 B/s rd, 0 B/s wr, 850 op/s; 19283/247284 objects misplaced (7.798%); 3.1 MiB/s, 49 objects/s recovering 2024-09-10T19:18:35.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:18:35 smithi183 ceph-mon[29736]: purged_snaps scrub starts 2024-09-10T19:18:35.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:18:35 smithi183 ceph-mon[29736]: purged_snaps scrub ok 2024-09-10T19:18:35.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:18:35 smithi183 ceph-mon[29736]: osdmap e204: 8 total, 8 up, 6 in 2024-09-10T19:18:35.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:18:35 smithi183 ceph-mon[41562]: pgmap v368: 158 pgs: 1 active+recovering+remapped, 1 active+recovering, 4 active+remapped+backfill_toofull, 152 active+clean; 5.0 GiB data, 18 GiB used, 518 GiB / 536 GiB avail; 853 B/s rd, 0 B/s wr, 850 op/s; 19283/247284 objects misplaced (7.798%); 3.1 MiB/s, 49 objects/s recovering 2024-09-10T19:18:35.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:18:35 smithi183 ceph-mon[41562]: purged_snaps scrub starts 2024-09-10T19:18:35.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:18:35 smithi183 ceph-mon[41562]: purged_snaps scrub ok 2024-09-10T19:18:35.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:18:35 smithi183 ceph-mon[41562]: osdmap e204: 8 total, 8 up, 6 in 2024-09-10T19:18:35.465 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:18:35 smithi191 ceph-mon[32884]: pgmap v368: 158 pgs: 1 active+recovering+remapped, 1 active+recovering, 4 active+remapped+backfill_toofull, 152 active+clean; 5.0 GiB data, 18 GiB used, 518 GiB / 536 GiB avail; 853 B/s rd, 0 B/s wr, 850 op/s; 19283/247284 objects misplaced (7.798%); 3.1 MiB/s, 49 objects/s recovering 2024-09-10T19:18:35.465 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:18:35 smithi191 ceph-mon[32884]: purged_snaps scrub starts 2024-09-10T19:18:35.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:18:35 smithi191 ceph-mon[32884]: purged_snaps scrub ok 2024-09-10T19:18:35.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:18:35 smithi191 ceph-mon[32884]: osdmap e204: 8 total, 8 up, 6 in 2024-09-10T19:18:35.509 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:18:35.547 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:18:35.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:18:35.547 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:18:35.648 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:18:35.679 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:18:35.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:18:35.680 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:18:35.781 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:18:35.810 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:18:35.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:18:35.811 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:18:35.912 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:18:35.947 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:18:35.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:18:35.948 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:18:36.048 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:18:36.086 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:18:36.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:18:36.087 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:18:36.188 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:18:36.229 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:18:36.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:18:36.230 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:18:36.330 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:18:36.367 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:18:36.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:18:36.368 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:18:36.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:18:36 smithi183 ceph-mon[29736]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-09-10T19:18:36.431 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:18:36 smithi183 ceph-mon[41562]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-09-10T19:18:36.465 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:18:36 smithi191 ceph-mon[32884]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-09-10T19:18:36.469 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:18:36.497 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:18:36.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:18:36.498 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:18:36.591 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph daemon osd.2 dump_historic_ops 2024-09-10T19:18:36.599 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:18:36.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:18:36.634 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:18:36.634 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:18:36.705 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:18:36.735 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:18:36.771 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:18:36.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:18:36.772 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:18:36.873 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:18:36.906 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:18:36.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:18:36.907 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:18:37.008 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:18:37.038 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:18:37.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:18:37.039 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:18:37.140 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:18:37.169 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:18:37.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:18:37.170 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:18:37.270 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:18:37.299 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:18:37.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:18:37.299 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:18:37.400 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:18:37.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:18:37 smithi183 ceph-mon[29736]: pgmap v370: 158 pgs: 3 active+remapped+backfilling, 1 active+recovering+remapped, 1 active+recovering, 1 active+remapped+backfill_toofull, 152 active+clean; 4.7 GiB data, 17 GiB used, 519 GiB / 536 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.09k op/s; 13235/231834 objects misplaced (5.709%); 42 MiB/s, 664 objects/s recovering 2024-09-10T19:18:37.431 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:18:37 smithi183 ceph-mon[41562]: pgmap v370: 158 pgs: 3 active+remapped+backfilling, 1 active+recovering+remapped, 1 active+recovering, 1 active+remapped+backfill_toofull, 152 active+clean; 4.7 GiB data, 17 GiB used, 519 GiB / 536 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.09k op/s; 13235/231834 objects misplaced (5.709%); 42 MiB/s, 664 objects/s recovering 2024-09-10T19:18:37.433 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:18:37.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:18:37.434 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:18:37.450 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:18:37 smithi191 ceph-mon[32884]: pgmap v370: 158 pgs: 3 active+remapped+backfilling, 1 active+recovering+remapped, 1 active+recovering, 1 active+remapped+backfill_toofull, 152 active+clean; 4.7 GiB data, 17 GiB used, 519 GiB / 536 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.09k op/s; 13235/231834 objects misplaced (5.709%); 42 MiB/s, 664 objects/s recovering 2024-09-10T19:18:37.535 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:18:37.565 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:18:37.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:18:37.566 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:18:37.660 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-10T19:18:37.666 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:18:37.703 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:18:37.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:18:37.704 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:18:37.715 INFO:journalctl@ceph.mgr.x.smithi191.stdout:Sep 10 19:18:37 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-x[34967]: ::ffff:172.21.15.191 - - [10/Sep/2024:19:18:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T19:18:37.805 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:18:37.838 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:18:37.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:18:37.840 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:18:37.940 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:18:37.983 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:18:37.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:18:37.984 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:18:38.085 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:18:38.114 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:18:38.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:18:38.115 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:18:38.198 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:18:38.216 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:18:38.251 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:18:38.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:18:38.252 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:18:38.353 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:18:38.382 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:18:38.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:18:38.383 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:18:38.483 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:18:38.520 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:18:38.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:18:38.521 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:18:38.622 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:18:38.645 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:18:38.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:18:38.646 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:18:38.747 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:18:38.776 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:18:38.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:18:38.777 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:18:38.878 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:18:38.906 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:18:38.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:18:38.907 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:18:39.008 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:18:39.046 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:18:39.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:18:39.046 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:18:39.147 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:18:39.178 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:18:39.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:18:39.179 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:18:39.280 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:18:39.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:18:39.315 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:18:39.315 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:18:39.416 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:18:39.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:18:39 smithi183 ceph-mon[29736]: pgmap v371: 158 pgs: 3 active+remapped+backfilling, 1 active+recovering+remapped, 1 active+recovering, 1 active+remapped+backfill_toofull, 152 active+clean; 4.7 GiB data, 17 GiB used, 519 GiB / 536 GiB avail; 853 B/s rd, 0 B/s wr, 858 op/s; 13235/231834 objects misplaced (5.709%); 33 MiB/s, 522 objects/s recovering 2024-09-10T19:18:39.431 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:18:39 smithi183 ceph-mon[41562]: pgmap v371: 158 pgs: 3 active+remapped+backfilling, 1 active+recovering+remapped, 1 active+recovering, 1 active+remapped+backfill_toofull, 152 active+clean; 4.7 GiB data, 17 GiB used, 519 GiB / 536 GiB avail; 853 B/s rd, 0 B/s wr, 858 op/s; 13235/231834 objects misplaced (5.709%); 33 MiB/s, 522 objects/s recovering 2024-09-10T19:18:39.447 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:18:39.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:18:39.448 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:18:39.465 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:18:39 smithi191 ceph-mon[32884]: pgmap v371: 158 pgs: 3 active+remapped+backfilling, 1 active+recovering+remapped, 1 active+recovering, 1 active+remapped+backfill_toofull, 152 active+clean; 4.7 GiB data, 17 GiB used, 519 GiB / 536 GiB avail; 853 B/s rd, 0 B/s wr, 858 op/s; 13235/231834 objects misplaced (5.709%); 33 MiB/s, 522 objects/s recovering 2024-09-10T19:18:39.466 INFO:journalctl@ceph.iscsi.iscsi.a.smithi191.stdout:Sep 10 19:18:39 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-iscsi-iscsi-a[61816]: debug there is no tcmu-runner data available 2024-09-10T19:18:39.548 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:18:39.579 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:18:39.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:18:39.580 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:18:39.681 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:18:39.711 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:18:39.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:18:39.712 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:18:39.813 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:18:39.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:18:39.853 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:18:39.853 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:18:39.954 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:18:39.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:18:39.979 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:18:39.980 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:18:40.080 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:18:40.113 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:18:40.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:18:40.114 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:18:40.215 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:18:40.243 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:18:40.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:18:40.245 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:18:40.345 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:18:40.378 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:18:40.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:18:40.379 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:18:40.480 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:18:40.523 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:18:40.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:18:40.524 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:18:40.625 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:18:40.657 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:18:40.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:18:40.658 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:18:40.758 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:18:40.786 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:18:40.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:18:40.787 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:18:40.888 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:18:40.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:18:40.917 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:18:40.917 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:18:41.018 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:18:41.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:18:41.049 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:18:41.049 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:18:41.132 INFO:tasks.thrashosds.thrasher:in_osds: [0, 3, 4, 5, 6, 7] out_osds: [1, 2] dead_osds: [] live_osds: [0, 2, 3, 4, 5, 6, 7, 1] 2024-09-10T19:18:41.132 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-10T19:18:41.132 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap_items 2024-09-10T19:18:41.133 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph osd dump -f json-pretty 2024-09-10T19:18:41.151 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:18:41.192 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:18:41.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:18:41.193 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:18:41.294 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:18:41.326 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:18:41.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:18:41.328 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:18:41.428 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:18:41.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:18:41 smithi183 ceph-mon[29736]: from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:18:41.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:18:41 smithi183 ceph-mon[29736]: pgmap v372: 158 pgs: 3 active+remapped+backfilling, 1 active+recovering+remapped, 1 active+recovering, 1 active+remapped+backfill_toofull, 152 active+clean; 4.7 GiB data, 17 GiB used, 519 GiB / 536 GiB avail; 664 B/s rd, 0 B/s wr, 669 op/s; 13235/231834 objects misplaced (5.709%); 25 MiB/s, 407 objects/s recovering 2024-09-10T19:18:41.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:18:41 smithi183 ceph-mon[41562]: from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:18:41.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:18:41 smithi183 ceph-mon[41562]: pgmap v372: 158 pgs: 3 active+remapped+backfilling, 1 active+recovering+remapped, 1 active+recovering, 1 active+remapped+backfill_toofull, 152 active+clean; 4.7 GiB data, 17 GiB used, 519 GiB / 536 GiB avail; 664 B/s rd, 0 B/s wr, 669 op/s; 13235/231834 objects misplaced (5.709%); 25 MiB/s, 407 objects/s recovering 2024-09-10T19:18:41.432 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Sep 10 19:18:41 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-y[29989]: ::ffff:172.21.15.191 - - [10/Sep/2024:19:18:41] "GET /metrics HTTP/1.1" 200 224286 "" "Prometheus/2.43.0" 2024-09-10T19:18:41.460 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:18:41.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:18:41.461 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:18:41.465 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:18:41 smithi191 ceph-mon[32884]: from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:18:41.465 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:18:41 smithi191 ceph-mon[32884]: pgmap v372: 158 pgs: 3 active+remapped+backfilling, 1 active+recovering+remapped, 1 active+recovering, 1 active+remapped+backfill_toofull, 152 active+clean; 4.7 GiB data, 17 GiB used, 519 GiB / 536 GiB avail; 664 B/s rd, 0 B/s wr, 669 op/s; 13235/231834 objects misplaced (5.709%); 25 MiB/s, 407 objects/s recovering 2024-09-10T19:18:41.562 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:18:41.591 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:18:41.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:18:41.592 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:18:41.692 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:18:41.732 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:18:41.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:18:41.733 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:18:41.834 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:18:41.854 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:18:41.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:18:41.855 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:18:41.956 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:18:41.977 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:18:41.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:18:41.978 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:18:42.078 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:18:42.107 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:18:42.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:18:42.108 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:18:42.209 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:18:42.236 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:18:42.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:18:42.238 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:18:42.338 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:18:42.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:18:42.375 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:18:42.375 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:18:42.476 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:18:42.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:18:42.519 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:18:42.519 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:18:42.620 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:18:42.653 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:18:42.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:18:42.655 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:18:42.755 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:18:42.788 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:18:42.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:18:42.789 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:18:42.889 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:18:42.927 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:18:42.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:18:42.928 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:18:43.029 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:18:43.057 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:18:43.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:18:43.058 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:18:43.159 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:18:43.185 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:18:43.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:18:43.186 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:18:43.287 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:18:43.320 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:18:43.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:18:43.321 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:18:43.322 INFO:teuthology.orchestra.run.smithi183.stdout:osd.0: {} 2024-09-10T19:18:43.322 INFO:teuthology.orchestra.run.smithi183.stdout:osd.1: {} 2024-09-10T19:18:43.322 INFO:teuthology.orchestra.run.smithi183.stdout:osd.2: {} 2024-09-10T19:18:43.323 INFO:teuthology.orchestra.run.smithi183.stdout:osd.3: {} 2024-09-10T19:18:43.323 INFO:teuthology.orchestra.run.smithi183.stdout:osd.4: {} 2024-09-10T19:18:43.323 INFO:teuthology.orchestra.run.smithi183.stdout:osd.5: {} 2024-09-10T19:18:43.323 INFO:teuthology.orchestra.run.smithi183.stdout:osd.6: {} 2024-09-10T19:18:43.323 INFO:teuthology.orchestra.run.smithi183.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-10T19:18:43.323 INFO:teuthology.orchestra.run.smithi183.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-10T19:18:43.323 INFO:teuthology.orchestra.run.smithi183.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-10T19:18:43.323 INFO:teuthology.orchestra.run.smithi183.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-10T19:18:43.323 INFO:teuthology.orchestra.run.smithi183.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-10T19:18:43.323 INFO:teuthology.orchestra.run.smithi183.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-10T19:18:43.323 INFO:teuthology.orchestra.run.smithi183.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-10T19:18:43.324 INFO:teuthology.orchestra.run.smithi183.stdout:osd.7: {} 2024-09-10T19:18:43.324 INFO:teuthology.orchestra.run.smithi183.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-10T19:18:43.422 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:18:43.431 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:18:43 smithi183 ceph-mon[41562]: pgmap v373: 158 pgs: 1 active+recovering+remapped, 3 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 1 active+recovering, 152 active+clean; 4.4 GiB data, 17 GiB used, 520 GiB / 536 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.20k op/s; 7141/216018 objects misplaced (3.306%); 47 MiB/s, 747 objects/s recovering 2024-09-10T19:18:43.431 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:18:43 smithi183 ceph-mon[41562]: from='client.? 172.21.15.183:0/3126422244' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T19:18:43.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:18:43 smithi183 ceph-mon[29736]: pgmap v373: 158 pgs: 1 active+recovering+remapped, 3 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 1 active+recovering, 152 active+clean; 4.4 GiB data, 17 GiB used, 520 GiB / 536 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.20k op/s; 7141/216018 objects misplaced (3.306%); 47 MiB/s, 747 objects/s recovering 2024-09-10T19:18:43.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:18:43 smithi183 ceph-mon[29736]: from='client.? 172.21.15.183:0/3126422244' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T19:18:43.446 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:18:43.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:18:43.447 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:18:43.465 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:18:43 smithi191 ceph-mon[32884]: pgmap v373: 158 pgs: 1 active+recovering+remapped, 3 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 1 active+recovering, 152 active+clean; 4.4 GiB data, 17 GiB used, 520 GiB / 536 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.20k op/s; 7141/216018 objects misplaced (3.306%); 47 MiB/s, 747 objects/s recovering 2024-09-10T19:18:43.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:18:43 smithi191 ceph-mon[32884]: from='client.? 172.21.15.183:0/3126422244' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T19:18:43.548 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:18:43.571 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:18:43.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:18:43.572 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:18:43.673 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:18:43.696 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:18:43.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:18:43.697 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:18:43.798 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:18:43.820 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:18:43.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:18:43.821 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:18:43.922 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:18:43.945 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:18:43.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:18:43.946 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:18:44.047 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:18:44.068 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:18:44.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:18:44.069 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:18:44.170 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:18:44.200 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:18:44.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:18:44.201 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:18:44.302 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:18:44.327 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:18:44.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:18:44.328 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:18:44.429 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:18:44.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:18:44 smithi183 ceph-mon[29736]: osdmap e205: 8 total, 8 up, 6 in 2024-09-10T19:18:44.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:18:44 smithi183 ceph-mon[29736]: from='mgr.14556 ' entity='mgr.y' 2024-09-10T19:18:44.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:18:44 smithi183 ceph-mon[29736]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:18:44.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:18:44 smithi183 ceph-mon[41562]: osdmap e205: 8 total, 8 up, 6 in 2024-09-10T19:18:44.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:18:44 smithi183 ceph-mon[41562]: from='mgr.14556 ' entity='mgr.y' 2024-09-10T19:18:44.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:18:44 smithi183 ceph-mon[41562]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:18:44.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:18:44.462 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:18:44.463 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:18:44.465 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:18:44 smithi191 ceph-mon[32884]: osdmap e205: 8 total, 8 up, 6 in 2024-09-10T19:18:44.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:18:44 smithi191 ceph-mon[32884]: from='mgr.14556 ' entity='mgr.y' 2024-09-10T19:18:44.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:18:44 smithi191 ceph-mon[32884]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:18:44.563 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:18:44.592 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:18:44.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:18:44.593 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:18:44.693 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:18:44.720 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:18:44.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:18:44.721 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:18:44.822 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:18:44.846 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:18:44.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:18:44.847 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:18:44.891 INFO:teuthology.orchestra.run.smithi183.stdout:{ 2024-09-10T19:18:44.891 INFO:teuthology.orchestra.run.smithi183.stdout: "size": 20, 2024-09-10T19:18:44.891 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 600, 2024-09-10T19:18:44.891 INFO:teuthology.orchestra.run.smithi183.stdout: "ops": [ 2024-09-10T19:18:44.891 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:18:44.891 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.14781.0:16994 7.e 7:78896f87:::benchmark_data_smithi183_84160_object16993:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e90)", 2024-09-10T19:18:44.892 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-10T19:12:24.659701+0000", 2024-09-10T19:18:44.892 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 380.22681397100001, 2024-09-10T19:18:44.892 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.104560112, 2024-09-10T19:18:44.892 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-10T19:18:44.892 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T19:18:44.892 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-10T19:18:44.892 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.14781", 2024-09-10T19:18:44.892 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.183:0/2025776420", 2024-09-10T19:18:44.892 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 16994 2024-09-10T19:18:44.892 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:18:44.892 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-10T19:18:44.892 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:18:44.892 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-10T19:18:44.893 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:24.659701+0000", 2024-09-10T19:18:44.893 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:18:44.893 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:18:44.893 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:18:44.893 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-10T19:18:44.893 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:24.659701+0000", 2024-09-10T19:18:44.893 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:18:44.893 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:18:44.893 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:18:44.893 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-10T19:18:44.893 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:24.659700+0000", 2024-09-10T19:18:44.893 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4294967295.999999 2024-09-10T19:18:44.893 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:18:44.894 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:18:44.894 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-10T19:18:44.894 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:24.659737+0000", 2024-09-10T19:18:44.894 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.7348999999999998e-05 2024-09-10T19:18:44.894 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:18:44.894 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:18:44.894 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-10T19:18:44.894 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:24.659748+0000", 2024-09-10T19:18:44.894 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.1077000000000001e-05 2024-09-10T19:18:44.894 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:18:44.894 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:18:44.894 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-10T19:18:44.894 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:24.659755+0000", 2024-09-10T19:18:44.894 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.6980000000000001e-06 2024-09-10T19:18:44.895 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:18:44.895 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:18:44.895 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-10T19:18:44.895 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:24.661704+0000", 2024-09-10T19:18:44.895 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0019488050000000001 2024-09-10T19:18:44.895 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:18:44.895 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:18:44.895 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-10T19:18:44.895 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:24.661898+0000", 2024-09-10T19:18:44.895 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00019394799999999999 2024-09-10T19:18:44.895 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:18:44.895 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:18:44.895 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 0,5", 2024-09-10T19:18:44.895 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:24.662695+0000", 2024-09-10T19:18:44.896 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000796789 2024-09-10T19:18:44.896 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:18:44.896 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:18:44.896 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:18:44.896 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:24.663749+0000", 2024-09-10T19:18:44.896 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0010547950000000001 2024-09-10T19:18:44.896 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:18:44.896 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:18:44.896 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-10T19:18:44.896 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:24.663765+0000", 2024-09-10T19:18:44.896 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.5290000000000001e-05 2024-09-10T19:18:44.896 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:18:44.897 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:18:44.897 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:18:44.897 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:24.763895+0000", 2024-09-10T19:18:44.897 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.100130362 2024-09-10T19:18:44.897 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:18:44.897 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:18:44.897 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-10T19:18:44.897 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:24.764223+0000", 2024-09-10T19:18:44.897 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00032781500000000002 2024-09-10T19:18:44.897 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:18:44.897 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:18:44.897 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-10T19:18:44.897 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:24.764261+0000", 2024-09-10T19:18:44.897 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.8194000000000001e-05 2024-09-10T19:18:44.898 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:18:44.898 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:18:44.898 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:18:44.898 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:18:44.898 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:18:44.899 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.14781.0:23975 7.e 7:70e2a0a7:::benchmark_data_smithi183_84160_object23974:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e91)", 2024-09-10T19:18:44.899 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-10T19:12:27.633672+0000", 2024-09-10T19:18:44.899 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 377.252843353, 2024-09-10T19:18:44.899 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.097315150000000003, 2024-09-10T19:18:44.899 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-10T19:18:44.899 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T19:18:44.899 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-10T19:18:44.899 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.14781", 2024-09-10T19:18:44.899 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.183:0/2025776420", 2024-09-10T19:18:44.899 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 23975 2024-09-10T19:18:44.899 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:18:44.899 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-10T19:18:44.899 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:18:44.900 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-10T19:18:44.900 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:27.633672+0000", 2024-09-10T19:18:44.900 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:18:44.900 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:18:44.900 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:18:44.900 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-10T19:18:44.900 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:27.633672+0000", 2024-09-10T19:18:44.900 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:18:44.900 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:18:44.900 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:18:44.900 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-10T19:18:44.900 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:27.633670+0000", 2024-09-10T19:18:44.900 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4294967295.999999 2024-09-10T19:18:44.901 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:18:44.901 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:18:44.901 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-10T19:18:44.901 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:27.633692+0000", 2024-09-10T19:18:44.901 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.2078000000000001e-05 2024-09-10T19:18:44.901 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:18:44.901 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:18:44.901 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-10T19:18:44.901 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:27.633704+0000", 2024-09-10T19:18:44.902 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.1752e-05 2024-09-10T19:18:44.902 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:18:44.902 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:18:44.902 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-10T19:18:44.902 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:27.633710+0000", 2024-09-10T19:18:44.902 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.818e-06 2024-09-10T19:18:44.902 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:18:44.902 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:18:44.902 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-10T19:18:44.902 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:27.633749+0000", 2024-09-10T19:18:44.902 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.9110999999999999e-05 2024-09-10T19:18:44.902 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:18:44.902 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:18:44.903 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-10T19:18:44.903 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:27.633835+0000", 2024-09-10T19:18:44.903 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.6102999999999999e-05 2024-09-10T19:18:44.903 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:18:44.903 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:18:44.903 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 0,5", 2024-09-10T19:18:44.903 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:27.633958+0000", 2024-09-10T19:18:44.903 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00012266499999999999 2024-09-10T19:18:44.903 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:18:44.903 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:18:44.903 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-10T19:18:44.903 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:27.634762+0000", 2024-09-10T19:18:44.903 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00080392400000000002 2024-09-10T19:18:44.904 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:18:44.904 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:18:44.904 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:18:44.904 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:27.635184+0000", 2024-09-10T19:18:44.904 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00042250399999999998 2024-09-10T19:18:44.904 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:18:44.904 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:18:44.904 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:18:44.904 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:27.730851+0000", 2024-09-10T19:18:44.904 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.095666553000000001 2024-09-10T19:18:44.904 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:18:44.904 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:18:44.904 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-10T19:18:44.904 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:27.730948+0000", 2024-09-10T19:18:44.905 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.7418999999999997e-05 2024-09-10T19:18:44.905 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:18:44.905 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:18:44.905 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-10T19:18:44.905 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:12:27.730987+0000", 2024-09-10T19:18:44.905 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.837e-05 2024-09-10T19:18:44.905 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:18:44.905 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:18:44.905 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:18:44.905 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:18:44.905 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:18:44.905 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.14781.0:153221 7.8 7:172abbc3:::benchmark_data_smithi183_84160_object12840:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e118)", 2024-09-10T19:18:44.905 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-10T19:14:00.995085+0000", 2024-09-10T19:18:44.906 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 283.89143006299997, 2024-09-10T19:18:44.906 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.099271308000000003, 2024-09-10T19:18:44.906 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-10T19:18:44.906 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T19:18:44.906 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-10T19:18:44.906 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.14781", 2024-09-10T19:18:44.906 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.183:0/2025776420", 2024-09-10T19:18:44.906 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 153221 2024-09-10T19:18:44.906 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:18:44.906 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-10T19:18:44.906 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:18:44.906 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-10T19:18:44.906 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:00.995085+0000", 2024-09-10T19:18:44.907 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:18:44.907 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:18:44.907 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:18:44.907 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-10T19:18:44.907 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:00.995085+0000", 2024-09-10T19:18:44.907 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:18:44.907 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:18:44.907 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:18:44.907 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-10T19:18:44.907 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:00.995083+0000", 2024-09-10T19:18:44.907 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4294967295.9999986 2024-09-10T19:18:44.907 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:18:44.907 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:18:44.908 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-10T19:18:44.908 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:00.995090+0000", 2024-09-10T19:18:44.908 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.0979999999999996e-06 2024-09-10T19:18:44.908 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:18:44.908 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:18:44.908 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-10T19:18:44.908 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:00.995107+0000", 2024-09-10T19:18:44.908 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.7221999999999999e-05 2024-09-10T19:18:44.908 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:18:44.908 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:18:44.908 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-10T19:18:44.908 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:00.995113+0000", 2024-09-10T19:18:44.908 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.8289999999999996e-06 2024-09-10T19:18:44.909 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:18:44.909 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:18:44.909 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-10T19:18:44.909 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:00.995155+0000", 2024-09-10T19:18:44.909 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.2175000000000002e-05 2024-09-10T19:18:44.909 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:18:44.909 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:18:44.909 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-10T19:18:44.909 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:00.995264+0000", 2024-09-10T19:18:44.909 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000108748 2024-09-10T19:18:44.909 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:18:44.909 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:18:44.909 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 4,7", 2024-09-10T19:18:44.910 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:00.995372+0000", 2024-09-10T19:18:44.910 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000107908 2024-09-10T19:18:44.910 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:18:44.910 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:18:44.910 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-10T19:18:44.910 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:00.999702+0000", 2024-09-10T19:18:44.910 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0043296580000000001 2024-09-10T19:18:44.910 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:18:44.910 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:18:44.910 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:18:44.910 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:01.003291+0000", 2024-09-10T19:18:44.910 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0035892910000000001 2024-09-10T19:18:44.911 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:18:44.911 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:18:44.911 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:18:44.911 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:01.094272+0000", 2024-09-10T19:18:44.911 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.090981399000000004 2024-09-10T19:18:44.911 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:18:44.911 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:18:44.911 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-10T19:18:44.911 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:01.094320+0000", 2024-09-10T19:18:44.911 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.7091e-05 2024-09-10T19:18:44.911 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:18:44.911 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:18:44.911 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-10T19:18:44.912 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:01.094356+0000", 2024-09-10T19:18:44.912 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.6526000000000002e-05 2024-09-10T19:18:44.912 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:18:44.912 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:18:44.912 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:18:44.912 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:18:44.912 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:18:44.912 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.14781.0:161349 7.e 7:7d3e032d:::benchmark_data_smithi183_84160_object20968:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120)", 2024-09-10T19:18:44.912 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-10T19:14:07.731291+0000", 2024-09-10T19:18:44.912 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 277.15522363000002, 2024-09-10T19:18:44.912 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.13911137500000001, 2024-09-10T19:18:44.912 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-10T19:18:44.912 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T19:18:44.913 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-10T19:18:44.913 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.14781", 2024-09-10T19:18:44.913 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.183:0/2025776420", 2024-09-10T19:18:44.913 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 161349 2024-09-10T19:18:44.913 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:18:44.913 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-10T19:18:44.913 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:18:44.913 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-10T19:18:44.913 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:07.731291+0000", 2024-09-10T19:18:44.913 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:18:44.913 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:18:44.913 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:18:44.913 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-10T19:18:44.913 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:07.731291+0000", 2024-09-10T19:18:44.914 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:18:44.914 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:18:44.914 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:18:44.914 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-10T19:18:44.914 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:07.731290+0000", 2024-09-10T19:18:44.914 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4294967295.9999986 2024-09-10T19:18:44.914 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:18:44.914 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:18:44.914 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-10T19:18:44.914 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:07.731303+0000", 2024-09-10T19:18:44.914 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.2734e-05 2024-09-10T19:18:44.914 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:18:44.914 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:18:44.914 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-10T19:18:44.915 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:07.731315+0000", 2024-09-10T19:18:44.915 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.2208999999999999e-05 2024-09-10T19:18:44.915 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:18:44.915 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:18:44.915 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-10T19:18:44.915 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:07.731321+0000", 2024-09-10T19:18:44.915 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.0580000000000003e-06 2024-09-10T19:18:44.915 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:18:44.915 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:18:44.915 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-10T19:18:44.915 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:07.732577+0000", 2024-09-10T19:18:44.915 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0012556690000000001 2024-09-10T19:18:44.915 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:18:44.915 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:18:44.916 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-10T19:18:44.916 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:07.732686+0000", 2024-09-10T19:18:44.916 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00010967300000000001 2024-09-10T19:18:44.916 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:18:44.916 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:18:44.916 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 0,5", 2024-09-10T19:18:44.916 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:07.732792+0000", 2024-09-10T19:18:44.916 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000106204 2024-09-10T19:18:44.916 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:18:44.916 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:18:44.916 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:18:44.916 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:07.739790+0000", 2024-09-10T19:18:44.916 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0069971460000000001 2024-09-10T19:18:44.917 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:18:44.917 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:18:44.917 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-10T19:18:44.917 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:07.739913+0000", 2024-09-10T19:18:44.917 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00012379599999999999 2024-09-10T19:18:44.917 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:18:44.917 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:18:44.917 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:18:44.917 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:07.870322+0000", 2024-09-10T19:18:44.917 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.13040858499999999 2024-09-10T19:18:44.917 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:18:44.917 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:18:44.917 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-10T19:18:44.917 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:07.870367+0000", 2024-09-10T19:18:44.918 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.4666999999999998e-05 2024-09-10T19:18:44.918 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:18:44.918 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:18:44.918 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-10T19:18:44.918 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:07.870403+0000", 2024-09-10T19:18:44.918 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.5917999999999997e-05 2024-09-10T19:18:44.918 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:18:44.918 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:18:44.918 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:18:44.918 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:18:44.918 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:18:44.918 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.14781.0:161353 7.e 7:7f1eb1cd:::benchmark_data_smithi183_84160_object20972:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120)", 2024-09-10T19:18:44.918 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-10T19:14:07.736149+0000", 2024-09-10T19:18:44.918 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 277.15036605900002, 2024-09-10T19:18:44.919 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.134348467, 2024-09-10T19:18:44.919 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-10T19:18:44.919 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T19:18:44.919 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-10T19:18:44.919 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.14781", 2024-09-10T19:18:44.919 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.183:0/2025776420", 2024-09-10T19:18:44.919 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 161353 2024-09-10T19:18:44.919 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:18:44.919 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-10T19:18:44.919 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:18:44.919 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-10T19:18:44.919 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:07.736149+0000", 2024-09-10T19:18:44.919 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:18:44.919 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:18:44.919 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:18:44.920 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-10T19:18:44.920 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:07.736149+0000", 2024-09-10T19:18:44.920 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:18:44.920 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:18:44.920 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:18:44.920 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-10T19:18:44.920 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:07.736147+0000", 2024-09-10T19:18:44.920 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4294967295.9999981 2024-09-10T19:18:44.920 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:18:44.920 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:18:44.920 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-10T19:18:44.920 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:07.736154+0000", 2024-09-10T19:18:44.920 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.9949999999999999e-06 2024-09-10T19:18:44.920 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:18:44.921 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:18:44.921 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-10T19:18:44.921 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:07.736170+0000", 2024-09-10T19:18:44.921 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.6535999999999999e-05 2024-09-10T19:18:44.921 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:18:44.921 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:18:44.921 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-10T19:18:44.921 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:07.736177+0000", 2024-09-10T19:18:44.921 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.3860000000000004e-06 2024-09-10T19:18:44.921 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:18:44.921 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:18:44.921 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-10T19:18:44.921 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:07.736348+0000", 2024-09-10T19:18:44.922 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000171053 2024-09-10T19:18:44.922 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:18:44.922 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:18:44.922 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-10T19:18:44.922 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:07.736451+0000", 2024-09-10T19:18:44.922 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000102731 2024-09-10T19:18:44.922 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:18:44.922 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:18:44.922 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 0,5", 2024-09-10T19:18:44.922 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:07.736555+0000", 2024-09-10T19:18:44.922 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000104147 2024-09-10T19:18:44.922 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:18:44.922 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:18:44.922 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:18:44.922 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:07.739816+0000", 2024-09-10T19:18:44.923 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.003260752 2024-09-10T19:18:44.923 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:18:44.923 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:18:44.923 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-10T19:18:44.923 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:07.740221+0000", 2024-09-10T19:18:44.923 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00040548299999999999 2024-09-10T19:18:44.923 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:18:44.923 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:18:44.923 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:18:44.923 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:07.870433+0000", 2024-09-10T19:18:44.923 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.13021206799999999 2024-09-10T19:18:44.923 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:18:44.923 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:18:44.924 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-10T19:18:44.924 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:07.870467+0000", 2024-09-10T19:18:44.924 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.4207e-05 2024-09-10T19:18:44.924 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:18:44.924 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:18:44.924 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-10T19:18:44.924 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:07.870497+0000", 2024-09-10T19:18:44.924 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.9977000000000001e-05 2024-09-10T19:18:44.924 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:18:44.924 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:18:44.924 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:18:44.924 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:18:44.924 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:18:44.924 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.14781.0:161359 7.e 7:7346129c:::benchmark_data_smithi183_84160_object20978:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120)", 2024-09-10T19:18:44.925 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-10T19:14:07.740514+0000", 2024-09-10T19:18:44.925 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 277.14600052100002, 2024-09-10T19:18:44.925 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.136079479, 2024-09-10T19:18:44.925 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-10T19:18:44.925 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T19:18:44.925 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-10T19:18:44.925 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.14781", 2024-09-10T19:18:44.925 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.183:0/2025776420", 2024-09-10T19:18:44.925 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 161359 2024-09-10T19:18:44.925 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:18:44.925 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-10T19:18:44.925 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:18:44.925 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-10T19:18:44.925 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:07.740514+0000", 2024-09-10T19:18:44.926 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:18:44.926 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:18:44.926 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:18:44.926 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-10T19:18:44.926 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:07.740514+0000", 2024-09-10T19:18:44.926 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:18:44.926 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:18:44.926 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:18:44.926 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-10T19:18:44.926 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:07.740513+0000", 2024-09-10T19:18:44.926 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4294967295.9999986 2024-09-10T19:18:44.926 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:18:44.926 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:18:44.926 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-10T19:18:44.926 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:07.740519+0000", 2024-09-10T19:18:44.927 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.0610000000000001e-06 2024-09-10T19:18:44.927 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:18:44.927 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:18:44.927 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-10T19:18:44.927 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:07.740530+0000", 2024-09-10T19:18:44.927 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.0874e-05 2024-09-10T19:18:44.927 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:18:44.927 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:18:44.927 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-10T19:18:44.927 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:07.740535+0000", 2024-09-10T19:18:44.927 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.4439999999999999e-06 2024-09-10T19:18:44.927 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:18:44.927 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:18:44.927 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-10T19:18:44.928 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:07.740673+0000", 2024-09-10T19:18:44.928 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00013725900000000001 2024-09-10T19:18:44.928 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:18:44.928 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:18:44.928 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-10T19:18:44.928 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:07.740773+0000", 2024-09-10T19:18:44.928 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000100739 2024-09-10T19:18:44.928 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:18:44.928 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:18:44.928 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 0,5", 2024-09-10T19:18:44.928 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:07.740878+0000", 2024-09-10T19:18:44.928 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00010482800000000001 2024-09-10T19:18:44.928 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:18:44.929 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:18:44.929 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:18:44.929 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:07.745774+0000", 2024-09-10T19:18:44.929 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0048961550000000001 2024-09-10T19:18:44.929 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:18:44.929 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:18:44.929 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-10T19:18:44.929 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:07.745798+0000", 2024-09-10T19:18:44.929 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.3125e-05 2024-09-10T19:18:44.929 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:18:44.929 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:18:44.929 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:18:44.929 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:07.876512+0000", 2024-09-10T19:18:44.929 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.130714624 2024-09-10T19:18:44.930 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:18:44.930 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:18:44.930 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-10T19:18:44.930 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:07.876558+0000", 2024-09-10T19:18:44.930 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.6252999999999999e-05 2024-09-10T19:18:44.930 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:18:44.930 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:18:44.930 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-10T19:18:44.930 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:07.876594+0000", 2024-09-10T19:18:44.930 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.5422000000000001e-05 2024-09-10T19:18:44.930 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:18:44.930 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:18:44.930 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:18:44.930 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:18:44.931 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:18:44.931 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.14781.0:161362 7.e 7:736c80f1:::benchmark_data_smithi183_84160_object20981:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120)", 2024-09-10T19:18:44.931 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-10T19:14:07.743069+0000", 2024-09-10T19:18:44.931 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 277.14344559199998, 2024-09-10T19:18:44.931 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.13626217800000001, 2024-09-10T19:18:44.931 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-10T19:18:44.931 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T19:18:44.931 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-10T19:18:44.931 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.14781", 2024-09-10T19:18:44.931 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.183:0/2025776420", 2024-09-10T19:18:44.931 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 161362 2024-09-10T19:18:44.931 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:18:44.931 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-10T19:18:44.931 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:18:44.931 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-10T19:18:44.932 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:07.743069+0000", 2024-09-10T19:18:44.932 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:18:44.932 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:18:44.932 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:18:44.932 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-10T19:18:44.932 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:07.743069+0000", 2024-09-10T19:18:44.932 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:18:44.932 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:18:44.932 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:18:44.932 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-10T19:18:44.932 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:07.743068+0000", 2024-09-10T19:18:44.932 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4294967295.9999986 2024-09-10T19:18:44.932 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:18:44.933 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:18:44.933 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-10T19:18:44.933 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:07.743075+0000", 2024-09-10T19:18:44.933 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.9500000000000004e-06 2024-09-10T19:18:44.933 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:18:44.933 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:18:44.933 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-10T19:18:44.933 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:07.743088+0000", 2024-09-10T19:18:44.933 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.3112000000000001e-05 2024-09-10T19:18:44.933 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:18:44.933 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:18:44.933 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-10T19:18:44.933 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:07.743095+0000", 2024-09-10T19:18:44.933 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.703e-06 2024-09-10T19:18:44.934 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:18:44.934 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:18:44.934 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-10T19:18:44.934 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:07.743684+0000", 2024-09-10T19:18:44.934 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00058946400000000005 2024-09-10T19:18:44.934 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:18:44.934 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:18:44.934 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-10T19:18:44.935 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:18:44.935 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:18:44.936 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:07.743755+0000", 2024-09-10T19:18:44.936 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.1375e-05 2024-09-10T19:18:44.936 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:18:44.936 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:18:44.936 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 0,5", 2024-09-10T19:18:44.936 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:07.743822+0000", 2024-09-10T19:18:44.936 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.6696e-05 2024-09-10T19:18:44.937 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:18:44.937 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:18:44.937 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:18:44.937 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:07.745791+0000", 2024-09-10T19:18:44.937 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0019687049999999998 2024-09-10T19:18:44.937 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:18:44.937 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:18:44.937 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-10T19:18:44.937 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:07.745800+0000", 2024-09-10T19:18:44.937 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.6129999999999993e-06 2024-09-10T19:18:44.937 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:18:44.937 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:18:44.937 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:18:44.938 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:07.879266+0000", 2024-09-10T19:18:44.938 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.133465363 2024-09-10T19:18:44.938 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:18:44.938 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:18:44.938 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-10T19:18:44.938 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:07.879302+0000", 2024-09-10T19:18:44.938 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.5754999999999999e-05 2024-09-10T19:18:44.938 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:18:44.938 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:18:44.938 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-10T19:18:44.938 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:07.879331+0000", 2024-09-10T19:18:44.938 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.9904e-05 2024-09-10T19:18:44.938 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:18:44.938 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:18:44.938 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:18:44.939 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:18:44.939 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:18:44.939 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "MOSDPGPush(7.b 120/116 [PushOp(7:d3313432:::benchmark_data_smithi183_84160_object50232:head, version: 96'3036, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(7:d3313432:::benchmark_data_smithi183_84160_object50232:head@96'3036, size: 65536, copy_subset: [0~65536], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:65536, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-10T19:18:44.939 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-10T19:14:07.749410+0000", 2024-09-10T19:18:44.939 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 277.13710510200002, 2024-09-10T19:18:44.939 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.120505768, 2024-09-10T19:18:44.939 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-10T19:18:44.939 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "started", 2024-09-10T19:18:44.939 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-10T19:18:44.939 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:18:44.939 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-10T19:18:44.939 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:07.749410+0000", 2024-09-10T19:18:44.939 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:18:44.939 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:18:44.940 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:18:44.940 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-10T19:18:44.940 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:07.749410+0000", 2024-09-10T19:18:44.940 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:18:44.940 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:18:44.940 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:18:44.940 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-10T19:18:44.940 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:07.749409+0000", 2024-09-10T19:18:44.940 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4294967295.9999995 2024-09-10T19:18:44.940 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:18:44.940 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:18:44.940 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-10T19:18:44.940 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:07.749556+0000", 2024-09-10T19:18:44.940 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00014702399999999999 2024-09-10T19:18:44.941 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:18:44.941 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:18:44.941 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-10T19:18:44.941 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:07.749567+0000", 2024-09-10T19:18:44.941 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.1066999999999999e-05 2024-09-10T19:18:44.941 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:18:44.941 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:18:44.941 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-10T19:18:44.941 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:07.749573+0000", 2024-09-10T19:18:44.941 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.5849999999999999e-06 2024-09-10T19:18:44.941 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:18:44.941 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:18:44.941 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-10T19:18:44.941 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:07.869617+0000", 2024-09-10T19:18:44.942 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.120043623 2024-09-10T19:18:44.942 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:18:44.942 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:18:44.942 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-10T19:18:44.942 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:07.869626+0000", 2024-09-10T19:18:44.942 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.6889999999999999e-06 2024-09-10T19:18:44.942 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:18:44.942 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:18:44.942 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-10T19:18:44.942 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:07.869916+0000", 2024-09-10T19:18:44.942 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00028926400000000001 2024-09-10T19:18:44.942 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:18:44.942 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:18:44.942 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:18:44.943 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:18:44.943 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:18:44.943 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "pg_backfill(progress 7.b e 120/120 lb 7:d330ac6a:::benchmark_data_smithi183_84160_object135588:head)", 2024-09-10T19:18:44.943 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-10T19:14:07.749584+0000", 2024-09-10T19:18:44.943 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 277.13693060399999, 2024-09-10T19:18:44.943 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.12054490599999999, 2024-09-10T19:18:44.943 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-10T19:18:44.943 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "started", 2024-09-10T19:18:44.943 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-10T19:18:44.943 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:18:44.943 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-10T19:18:44.943 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:07.749584+0000", 2024-09-10T19:18:44.943 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:18:44.943 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:18:44.944 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:18:44.944 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-10T19:18:44.944 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:07.749584+0000", 2024-09-10T19:18:44.944 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:18:44.944 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:18:44.944 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:18:44.944 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-10T19:18:44.944 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:07.749584+0000", 2024-09-10T19:18:44.944 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4294967295.9999995 2024-09-10T19:18:44.944 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:18:44.944 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:18:44.944 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-10T19:18:44.944 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:07.749589+0000", 2024-09-10T19:18:44.944 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.6879999999999998e-06 2024-09-10T19:18:44.944 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:18:44.945 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:18:44.945 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-10T19:18:44.945 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:07.749593+0000", 2024-09-10T19:18:44.945 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.8630000000000001e-06 2024-09-10T19:18:44.945 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:18:44.945 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:18:44.945 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-10T19:18:44.945 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:07.869569+0000", 2024-09-10T19:18:44.945 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.11997672299999999 2024-09-10T19:18:44.945 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:18:44.945 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:18:44.945 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-10T19:18:44.945 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:07.870077+0000", 2024-09-10T19:18:44.945 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00050739600000000002 2024-09-10T19:18:44.945 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:18:44.946 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:18:44.946 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-10T19:18:44.946 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:07.870093+0000", 2024-09-10T19:18:44.946 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.5860000000000001e-05 2024-09-10T19:18:44.946 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:18:44.946 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:18:44.946 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-10T19:18:44.946 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:07.870129+0000", 2024-09-10T19:18:44.946 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.6636e-05 2024-09-10T19:18:44.946 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:18:44.946 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:18:44.946 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:18:44.946 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:18:44.946 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:18:44.947 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.14781.0:161675 7.8 7:1d0bcc03:::benchmark_data_smithi183_84160_object21294:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120)", 2024-09-10T19:18:44.947 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-10T19:14:08.092546+0000", 2024-09-10T19:18:44.947 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 276.79396926499999, 2024-09-10T19:18:44.947 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.143957107, 2024-09-10T19:18:44.947 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-10T19:18:44.947 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T19:18:44.947 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-10T19:18:44.947 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.14781", 2024-09-10T19:18:44.947 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.183:0/2025776420", 2024-09-10T19:18:44.947 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 161675 2024-09-10T19:18:44.947 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:18:44.947 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-10T19:18:44.947 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:18:44.947 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-10T19:18:44.947 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:08.092546+0000", 2024-09-10T19:18:44.947 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:18:44.948 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:18:44.948 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:18:44.948 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-10T19:18:44.948 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:08.092546+0000", 2024-09-10T19:18:44.948 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:18:44.948 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:18:44.948 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:18:44.948 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-10T19:18:44.948 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:08.092544+0000", 2024-09-10T19:18:44.948 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4294967295.9999986 2024-09-10T19:18:44.948 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:18:44.948 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:18:44.948 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-10T19:18:44.948 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:08.092551+0000", 2024-09-10T19:18:44.948 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.4289999999999998e-06 2024-09-10T19:18:44.949 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:18:44.949 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:18:44.949 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-10T19:18:44.949 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:08.092563+0000", 2024-09-10T19:18:44.949 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.22e-05 2024-09-10T19:18:44.949 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:18:44.949 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:18:44.949 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-10T19:18:44.949 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:08.092569+0000", 2024-09-10T19:18:44.949 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.1700000000000002e-06 2024-09-10T19:18:44.949 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:18:44.949 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:18:44.949 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-10T19:18:44.950 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:08.094292+0000", 2024-09-10T19:18:44.950 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.001722433 2024-09-10T19:18:44.950 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:18:44.950 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:18:44.950 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-10T19:18:44.950 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:08.094446+0000", 2024-09-10T19:18:44.950 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00015463900000000001 2024-09-10T19:18:44.950 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:18:44.950 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:18:44.950 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 4,7", 2024-09-10T19:18:44.950 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:08.094552+0000", 2024-09-10T19:18:44.950 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00010600899999999999 2024-09-10T19:18:44.950 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:18:44.951 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:18:44.951 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:18:44.951 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:08.233696+0000", 2024-09-10T19:18:44.951 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.139144136 2024-09-10T19:18:44.951 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:18:44.951 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:18:44.951 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:18:44.951 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:08.233769+0000", 2024-09-10T19:18:44.951 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.2881999999999998e-05 2024-09-10T19:18:44.951 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:18:44.951 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:18:44.951 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-10T19:18:44.951 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:08.236418+0000", 2024-09-10T19:18:44.951 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.002648994 2024-09-10T19:18:44.951 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:18:44.952 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:18:44.952 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-10T19:18:44.952 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:08.236467+0000", 2024-09-10T19:18:44.952 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.8477999999999999e-05 2024-09-10T19:18:44.952 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:18:44.952 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:18:44.952 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-10T19:18:44.952 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:08.236503+0000", 2024-09-10T19:18:44.952 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.6031000000000001e-05 2024-09-10T19:18:44.952 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:18:44.952 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:18:44.952 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:18:44.952 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:18:44.952 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:18:44.953 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.14781.0:161676 7.8 7:1830f4cb:::benchmark_data_smithi183_84160_object21295:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120)", 2024-09-10T19:18:44.953 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-10T19:14:08.092580+0000", 2024-09-10T19:18:44.953 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 276.79393448100001, 2024-09-10T19:18:44.953 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.146075545, 2024-09-10T19:18:44.953 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-10T19:18:44.953 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T19:18:44.953 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-10T19:18:44.953 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.14781", 2024-09-10T19:18:44.953 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.183:0/2025776420", 2024-09-10T19:18:44.953 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 161676 2024-09-10T19:18:44.953 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:18:44.953 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-10T19:18:44.953 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:18:44.954 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-10T19:18:44.954 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:08.092580+0000", 2024-09-10T19:18:44.954 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:18:44.954 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:18:44.954 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:18:44.954 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-10T19:18:44.954 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:08.092580+0000", 2024-09-10T19:18:44.954 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:18:44.954 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:18:44.954 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:18:44.954 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-10T19:18:44.954 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:08.092580+0000", 2024-09-10T19:18:44.954 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4294967295.9999995 2024-09-10T19:18:44.954 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:18:44.955 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:18:44.955 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-10T19:18:44.955 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:08.092583+0000", 2024-09-10T19:18:44.955 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.2669999999999998e-06 2024-09-10T19:18:44.955 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:18:44.955 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:18:44.955 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-10T19:18:44.955 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:08.092590+0000", 2024-09-10T19:18:44.955 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.6630000000000004e-06 2024-09-10T19:18:44.955 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:18:44.955 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:18:44.955 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-10T19:18:44.956 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:08.092594+0000", 2024-09-10T19:18:44.956 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.3279999999999999e-06 2024-09-10T19:18:44.956 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:18:44.956 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:18:44.956 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-10T19:18:44.956 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:08.097367+0000", 2024-09-10T19:18:44.956 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0047731190000000001 2024-09-10T19:18:44.956 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:18:44.956 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:18:44.956 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-10T19:18:44.956 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:08.097460+0000", 2024-09-10T19:18:44.956 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.2694000000000004e-05 2024-09-10T19:18:44.956 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:18:44.956 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:18:44.956 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 4,7", 2024-09-10T19:18:44.957 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:08.097635+0000", 2024-09-10T19:18:44.957 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00017461899999999999 2024-09-10T19:18:44.957 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:18:44.957 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:18:44.957 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:18:44.957 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:08.233732+0000", 2024-09-10T19:18:44.957 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.13609732299999999 2024-09-10T19:18:44.957 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:18:44.957 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:18:44.957 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:18:44.957 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:08.233815+0000", 2024-09-10T19:18:44.957 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.2949999999999997e-05 2024-09-10T19:18:44.957 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:18:44.957 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:18:44.958 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-10T19:18:44.958 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:08.238577+0000", 2024-09-10T19:18:44.958 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0047622319999999999 2024-09-10T19:18:44.958 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:18:44.958 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:18:44.958 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-10T19:18:44.958 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:08.238623+0000", 2024-09-10T19:18:44.958 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.5930999999999998e-05 2024-09-10T19:18:44.958 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:18:44.958 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:18:44.958 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-10T19:18:44.958 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:08.238656+0000", 2024-09-10T19:18:44.958 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.2926999999999998e-05 2024-09-10T19:18:44.958 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:18:44.959 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:18:44.959 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:18:44.959 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:18:44.959 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:18:44.959 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.14781.0:161680 7.8 7:13ecf742:::benchmark_data_smithi183_84160_object21299:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120)", 2024-09-10T19:18:44.959 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-10T19:14:08.092602+0000", 2024-09-10T19:18:44.959 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 276.793912519, 2024-09-10T19:18:44.959 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.16144267000000001, 2024-09-10T19:18:44.959 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-10T19:18:44.959 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T19:18:44.959 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-10T19:18:44.959 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.14781", 2024-09-10T19:18:44.959 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.183:0/2025776420", 2024-09-10T19:18:44.959 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 161680 2024-09-10T19:18:44.959 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:18:44.960 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-10T19:18:44.960 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:18:44.960 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-10T19:18:44.960 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:08.092602+0000", 2024-09-10T19:18:44.960 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:18:44.960 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:18:44.960 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:18:44.960 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-10T19:18:44.960 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:08.092602+0000", 2024-09-10T19:18:44.960 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:18:44.960 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:18:44.960 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:18:44.960 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-10T19:18:44.961 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:08.092602+0000", 2024-09-10T19:18:44.961 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4294967295.9999995 2024-09-10T19:18:44.961 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:18:44.961 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:18:44.961 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-10T19:18:44.961 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:08.092605+0000", 2024-09-10T19:18:44.961 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.7690000000000001e-06 2024-09-10T19:18:44.961 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:18:44.961 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:18:44.961 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-10T19:18:44.961 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:08.092611+0000", 2024-09-10T19:18:44.961 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.1689999999999997e-06 2024-09-10T19:18:44.961 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:18:44.962 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:18:44.962 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-10T19:18:44.962 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:08.092615+0000", 2024-09-10T19:18:44.962 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.6990000000000001e-06 2024-09-10T19:18:44.962 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:18:44.962 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:18:44.962 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-10T19:18:44.962 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:08.244275+0000", 2024-09-10T19:18:44.962 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.15166089299999999 2024-09-10T19:18:44.962 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:18:44.962 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:18:44.962 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-10T19:18:44.962 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:08.244378+0000", 2024-09-10T19:18:44.962 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000102601 2024-09-10T19:18:44.963 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:18:44.963 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:18:44.963 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 4,7", 2024-09-10T19:18:44.963 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:08.244480+0000", 2024-09-10T19:18:44.963 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00010226 2024-09-10T19:18:44.963 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:18:44.963 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:18:44.963 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:18:44.963 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:08.253249+0000", 2024-09-10T19:18:44.963 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0087684270000000005 2024-09-10T19:18:44.963 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:18:44.963 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:18:44.963 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:18:44.964 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:08.253282+0000", 2024-09-10T19:18:44.964 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.2950999999999997e-05 2024-09-10T19:18:44.964 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:18:44.964 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:18:44.964 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-10T19:18:44.964 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:08.253963+0000", 2024-09-10T19:18:44.964 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00068178100000000001 2024-09-10T19:18:44.964 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:18:44.964 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:18:44.964 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-10T19:18:44.964 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:08.254009+0000", 2024-09-10T19:18:44.964 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.5176000000000003e-05 2024-09-10T19:18:44.964 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:18:44.964 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:18:44.965 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-10T19:18:44.965 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:08.254045+0000", 2024-09-10T19:18:44.965 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.6414000000000001e-05 2024-09-10T19:18:44.965 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:18:44.965 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:18:44.965 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:18:44.965 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:18:44.965 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:18:44.965 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_repop(client.14781.0:161681 7.0 e120/116 7:0fcfee5d:::benchmark_data_smithi183_84160_object21300:head v 120'10095, mlcod=120'10093)", 2024-09-10T19:18:44.965 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-10T19:14:08.095281+0000", 2024-09-10T19:18:44.965 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 276.79123380999999, 2024-09-10T19:18:44.965 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.14947492600000001, 2024-09-10T19:18:44.965 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-10T19:18:44.965 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T19:18:44.965 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-10T19:18:44.966 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:18:44.966 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-10T19:18:44.966 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:08.095281+0000", 2024-09-10T19:18:44.966 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:18:44.966 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:18:44.966 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:18:44.966 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-10T19:18:44.966 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:08.095281+0000", 2024-09-10T19:18:44.966 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:18:44.966 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:18:44.966 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:18:44.966 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-10T19:18:44.966 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:08.095280+0000", 2024-09-10T19:18:44.966 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4294967295.9999995 2024-09-10T19:18:44.967 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:18:44.967 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:18:44.967 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-10T19:18:44.967 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:08.095286+0000", 2024-09-10T19:18:44.967 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.1619999999999996e-06 2024-09-10T19:18:44.967 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:18:44.967 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:18:44.967 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-10T19:18:44.967 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:08.095297+0000", 2024-09-10T19:18:44.967 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.0651e-05 2024-09-10T19:18:44.967 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:18:44.967 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:18:44.967 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-10T19:18:44.967 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:08.095302+0000", 2024-09-10T19:18:44.968 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.057e-06 2024-09-10T19:18:44.968 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:18:44.968 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:18:44.968 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-10T19:18:44.968 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:08.097238+0000", 2024-09-10T19:18:44.968 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.001935888 2024-09-10T19:18:44.968 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:18:44.968 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:18:44.968 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-10T19:18:44.968 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:08.097255+0000", 2024-09-10T19:18:44.968 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.7133999999999999e-05 2024-09-10T19:18:44.968 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:18:44.968 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:18:44.969 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-10T19:18:44.969 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:08.244719+0000", 2024-09-10T19:18:44.969 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.14746366 2024-09-10T19:18:44.969 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:18:44.969 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:18:44.969 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-10T19:18:44.969 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:08.244756+0000", 2024-09-10T19:18:44.969 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.7085999999999997e-05 2024-09-10T19:18:44.969 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:18:44.969 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:18:44.969 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:18:44.969 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:18:44.969 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:18:44.969 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.14781.0:161688 7.e 7:76198a01:::benchmark_data_smithi183_84160_object21307:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120)", 2024-09-10T19:18:44.970 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-10T19:14:08.097940+0000", 2024-09-10T19:18:44.970 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 276.78857502800003, 2024-09-10T19:18:44.970 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.144387614, 2024-09-10T19:18:44.970 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-10T19:18:44.970 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T19:18:44.970 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-10T19:18:44.970 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.14781", 2024-09-10T19:18:44.970 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.183:0/2025776420", 2024-09-10T19:18:44.970 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 161688 2024-09-10T19:18:44.970 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:18:44.970 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-10T19:18:44.970 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:18:44.970 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-10T19:18:44.970 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:08.097940+0000", 2024-09-10T19:18:44.971 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:18:44.971 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:18:44.971 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:18:44.971 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-10T19:18:44.971 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:08.097940+0000", 2024-09-10T19:18:44.971 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:18:44.971 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:18:44.971 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:18:44.971 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-10T19:18:44.971 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:08.097939+0000", 2024-09-10T19:18:44.971 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4294967295.9999986 2024-09-10T19:18:44.971 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:18:44.971 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:18:44.972 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-10T19:18:44.972 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:08.097945+0000", 2024-09-10T19:18:44.972 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.6259999999999997e-06 2024-09-10T19:18:44.972 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:18:44.972 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:18:44.972 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-10T19:18:44.972 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:08.097956+0000", 2024-09-10T19:18:44.972 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.0625000000000001e-05 2024-09-10T19:18:44.972 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:18:44.972 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:18:44.972 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-10T19:18:44.972 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:08.097961+0000", 2024-09-10T19:18:44.972 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.5960000000000003e-06 2024-09-10T19:18:44.972 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:18:44.973 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:18:44.973 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-10T19:18:44.973 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:08.098002+0000", 2024-09-10T19:18:44.973 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.0593999999999998e-05 2024-09-10T19:18:44.973 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:18:44.973 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:18:44.973 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-10T19:18:44.973 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:08.098138+0000", 2024-09-10T19:18:44.973 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00013619500000000001 2024-09-10T19:18:44.973 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:18:44.973 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:18:44.973 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 0,5", 2024-09-10T19:18:44.973 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:08.098362+0000", 2024-09-10T19:18:44.974 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00022352299999999999 2024-09-10T19:18:44.974 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:18:44.974 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:18:44.974 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:18:44.974 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:08.233903+0000", 2024-09-10T19:18:44.974 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.13554107600000001 2024-09-10T19:18:44.974 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:18:44.974 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:18:44.974 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-10T19:18:44.974 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:08.233907+0000", 2024-09-10T19:18:44.974 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.2150000000000003e-06 2024-09-10T19:18:44.974 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:18:44.974 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:18:44.974 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:18:44.975 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:08.242091+0000", 2024-09-10T19:18:44.975 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0081839630000000007 2024-09-10T19:18:44.975 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:18:44.975 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:18:44.975 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-10T19:18:44.975 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:08.242286+0000", 2024-09-10T19:18:44.975 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00019465199999999999 2024-09-10T19:18:44.975 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:18:44.975 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:18:44.975 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-10T19:18:44.975 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:08.242327+0000", 2024-09-10T19:18:44.975 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.1820999999999998e-05 2024-09-10T19:18:44.976 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:18:44.976 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:18:44.976 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:18:44.976 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:18:44.976 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:18:44.976 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_repop(client.14781.0:161687 7.5 e120/86 7:a710c747:::benchmark_data_smithi183_84160_object21306:head v 120'10122, mlcod=120'10120)", 2024-09-10T19:18:44.976 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-10T19:14:08.098183+0000", 2024-09-10T19:18:44.976 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 276.78833189599999, 2024-09-10T19:18:44.976 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.13543248199999999, 2024-09-10T19:18:44.976 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-10T19:18:44.976 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T19:18:44.976 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-10T19:18:44.976 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:18:44.977 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-10T19:18:44.977 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:08.098183+0000", 2024-09-10T19:18:44.977 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:18:44.977 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:18:44.977 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:18:44.977 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-10T19:18:44.977 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:08.098183+0000", 2024-09-10T19:18:44.977 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:18:44.977 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:18:44.977 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:18:44.977 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-10T19:18:44.977 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:08.098182+0000", 2024-09-10T19:18:44.977 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4294967295.999999 2024-09-10T19:18:44.977 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:18:44.978 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:18:44.978 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-10T19:18:44.978 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:08.098190+0000", 2024-09-10T19:18:44.978 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.7239999999999992e-06 2024-09-10T19:18:44.978 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:18:44.978 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:18:44.978 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-10T19:18:44.978 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:08.098199+0000", 2024-09-10T19:18:44.978 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.7080000000000003e-06 2024-09-10T19:18:44.978 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:18:44.978 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:18:44.978 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-10T19:18:44.978 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:08.098203+0000", 2024-09-10T19:18:44.979 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.6979999999999997e-06 2024-09-10T19:18:44.979 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:18:44.979 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:18:44.979 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-10T19:18:44.979 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:08.098241+0000", 2024-09-10T19:18:44.979 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.7696e-05 2024-09-10T19:18:44.979 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:18:44.979 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:18:44.979 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-10T19:18:44.979 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:08.098258+0000", 2024-09-10T19:18:44.979 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.7184e-05 2024-09-10T19:18:44.979 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:18:44.979 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:18:44.979 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-10T19:18:44.980 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:08.100539+0000", 2024-09-10T19:18:44.980 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0022809810000000001 2024-09-10T19:18:44.980 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:18:44.980 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:18:44.980 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-10T19:18:44.980 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:08.233615+0000", 2024-09-10T19:18:44.980 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.133076212 2024-09-10T19:18:44.980 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:18:44.980 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:18:44.980 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:18:44.980 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:18:44.980 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:18:44.980 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_repop(client.14781.0:161683 7.0 e120/116 7:085c0f2f:::benchmark_data_smithi183_84160_object21302:head v 120'10096, mlcod=120'10093)", 2024-09-10T19:18:44.981 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-10T19:14:08.098314+0000", 2024-09-10T19:18:44.981 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 276.78820078299998, 2024-09-10T19:18:44.981 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.15554790700000001, 2024-09-10T19:18:44.981 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-10T19:18:44.981 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T19:18:44.981 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-10T19:18:44.981 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:18:44.981 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-10T19:18:44.981 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:08.098314+0000", 2024-09-10T19:18:44.981 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:18:44.981 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:18:44.981 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:18:44.981 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-10T19:18:44.981 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:08.098314+0000", 2024-09-10T19:18:44.982 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:18:44.982 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:18:44.982 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:18:44.982 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-10T19:18:44.982 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:08.098314+0000", 2024-09-10T19:18:44.982 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4294967295.9999995 2024-09-10T19:18:44.982 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:18:44.982 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:18:44.982 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-10T19:18:44.982 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:08.098318+0000", 2024-09-10T19:18:44.982 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.4229999999999998e-06 2024-09-10T19:18:44.982 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:18:44.982 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:18:44.982 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-10T19:18:44.983 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:08.098327+0000", 2024-09-10T19:18:44.983 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.2199999999999998e-06 2024-09-10T19:18:44.983 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:18:44.983 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:18:44.983 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-10T19:18:44.983 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:08.098331+0000", 2024-09-10T19:18:44.983 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.9299999999999996e-06 2024-09-10T19:18:44.983 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:18:44.983 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:18:44.983 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-10T19:18:44.983 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:08.233758+0000", 2024-09-10T19:18:44.983 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.135427031 2024-09-10T19:18:44.983 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:18:44.984 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:18:44.984 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-10T19:18:44.984 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:08.233780+0000", 2024-09-10T19:18:44.984 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.1761999999999999e-05 2024-09-10T19:18:44.984 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:18:44.984 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:18:44.984 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-10T19:18:44.984 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:08.253191+0000", 2024-09-10T19:18:44.984 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.019411069 2024-09-10T19:18:44.984 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:18:44.984 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:18:44.984 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-10T19:18:44.984 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:08.253862+0000", 2024-09-10T19:18:44.984 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00067096099999999997 2024-09-10T19:18:44.985 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:18:44.985 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:18:44.985 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:18:44.985 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:18:44.985 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:18:44.985 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_repop_reply(client.14781.0:161675 7.8 e120/86 ondisk, result = 0)", 2024-09-10T19:18:44.985 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-10T19:14:08.100726+0000", 2024-09-10T19:18:44.985 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 276.78578909100003, 2024-09-10T19:18:44.985 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.13297299300000001, 2024-09-10T19:18:44.985 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-10T19:18:44.985 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "started", 2024-09-10T19:18:44.985 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-10T19:18:44.985 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:18:44.985 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-10T19:18:44.985 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:08.100726+0000", 2024-09-10T19:18:44.986 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:18:44.986 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:18:44.986 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:18:44.986 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-10T19:18:44.986 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:08.100726+0000", 2024-09-10T19:18:44.986 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:18:44.986 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:18:44.986 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:18:44.986 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-10T19:18:44.986 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:08.100725+0000", 2024-09-10T19:18:44.986 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4294967295.999999 2024-09-10T19:18:44.986 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:18:44.986 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:18:44.986 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-10T19:18:44.986 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:08.100733+0000", 2024-09-10T19:18:44.987 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.3739999999999998e-06 2024-09-10T19:18:44.987 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:18:44.987 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:18:44.987 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-10T19:18:44.987 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:08.233540+0000", 2024-09-10T19:18:44.987 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.132806437 2024-09-10T19:18:44.987 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:18:44.987 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:18:44.987 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-10T19:18:44.987 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:08.233556+0000", 2024-09-10T19:18:44.987 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.5962000000000001e-05 2024-09-10T19:18:44.987 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:18:44.988 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:18:44.988 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-10T19:18:44.988 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:08.233678+0000", 2024-09-10T19:18:44.988 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00012219999999999999 2024-09-10T19:18:44.988 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:18:44.988 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:18:44.988 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-10T19:18:44.988 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:08.233688+0000", 2024-09-10T19:18:44.988 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.0138e-05 2024-09-10T19:18:44.988 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:18:44.988 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:18:44.988 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-10T19:18:44.988 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:08.233699+0000", 2024-09-10T19:18:44.989 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.0617e-05 2024-09-10T19:18:44.989 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:18:44.989 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:18:44.989 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:18:44.989 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:18:44.989 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:18:44.989 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.14781.0:184800 7.8 7:1e0ea857:::benchmark_data_smithi183_84160_object44419:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122)", 2024-09-10T19:18:44.989 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-10T19:14:26.531942+0000", 2024-09-10T19:18:44.989 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 258.354572566, 2024-09-10T19:18:44.989 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.099247427999999999, 2024-09-10T19:18:44.989 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-10T19:18:44.989 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T19:18:44.989 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-10T19:18:44.989 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.14781", 2024-09-10T19:18:44.989 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.183:0/2025776420", 2024-09-10T19:18:44.990 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 184800 2024-09-10T19:18:44.990 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:18:44.990 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-10T19:18:44.990 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:18:44.990 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-10T19:18:44.990 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:26.531942+0000", 2024-09-10T19:18:44.990 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:18:44.990 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:18:44.990 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:18:44.990 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-10T19:18:44.990 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:26.531942+0000", 2024-09-10T19:18:44.990 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:18:44.990 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:18:44.991 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:18:44.991 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-10T19:18:44.991 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:26.531942+0000", 2024-09-10T19:18:44.991 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4294967295.9999995 2024-09-10T19:18:44.991 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:18:44.991 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:18:44.991 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-10T19:18:44.991 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:26.531946+0000", 2024-09-10T19:18:44.991 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.743e-06 2024-09-10T19:18:44.991 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:18:44.991 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:18:44.991 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-10T19:18:44.991 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:26.531953+0000", 2024-09-10T19:18:44.991 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.2060000000000001e-06 2024-09-10T19:18:44.991 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:18:44.992 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:18:44.992 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-10T19:18:44.992 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:26.531957+0000", 2024-09-10T19:18:44.992 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.788e-06 2024-09-10T19:18:44.992 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:18:44.992 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:18:44.992 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-10T19:18:44.992 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:26.535574+0000", 2024-09-10T19:18:44.992 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.003617309 2024-09-10T19:18:44.992 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:18:44.992 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:18:44.992 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-10T19:18:44.992 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:26.535674+0000", 2024-09-10T19:18:44.992 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000100146 2024-09-10T19:18:44.992 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:18:44.993 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:18:44.993 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 4,7", 2024-09-10T19:18:44.993 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:26.535795+0000", 2024-09-10T19:18:44.993 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000121418 2024-09-10T19:18:44.993 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:18:44.993 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:18:44.993 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-10T19:18:44.993 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:26.538632+0000", 2024-09-10T19:18:44.993 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0028368500000000001 2024-09-10T19:18:44.993 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:18:44.993 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:18:44.993 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:18:44.993 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:26.549860+0000", 2024-09-10T19:18:44.994 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.01122799 2024-09-10T19:18:44.994 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:18:44.994 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:18:44.994 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:18:44.994 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:26.631100+0000", 2024-09-10T19:18:44.994 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.081239459 2024-09-10T19:18:44.994 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:18:44.994 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:18:44.994 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-10T19:18:44.994 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:26.631153+0000", 2024-09-10T19:18:44.994 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.3025999999999997e-05 2024-09-10T19:18:44.994 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:18:44.994 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:18:44.994 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-10T19:18:44.995 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:26.631190+0000", 2024-09-10T19:18:44.995 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.7020999999999997e-05 2024-09-10T19:18:44.995 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:18:44.995 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:18:44.995 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:18:44.995 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:18:44.995 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:18:44.995 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.14781.0:184815 7.8 7:1248c7ea:::benchmark_data_smithi183_84160_object44434:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e122)", 2024-09-10T19:18:44.995 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-10T19:14:26.540210+0000", 2024-09-10T19:18:44.995 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 258.34630473599998, 2024-09-10T19:18:44.995 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.094977161000000004, 2024-09-10T19:18:44.995 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-10T19:18:44.995 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T19:18:44.995 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-10T19:18:44.995 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.14781", 2024-09-10T19:18:44.995 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.183:0/2025776420", 2024-09-10T19:18:44.996 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 184815 2024-09-10T19:18:44.996 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:18:44.996 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-10T19:18:44.996 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:18:44.996 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-10T19:18:44.996 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:26.540210+0000", 2024-09-10T19:18:44.996 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:18:44.996 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:18:44.996 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:18:44.996 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-10T19:18:44.996 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:26.540210+0000", 2024-09-10T19:18:44.996 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:18:44.996 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:18:44.996 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:18:44.997 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-10T19:18:44.997 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:26.540209+0000", 2024-09-10T19:18:44.997 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4294967295.999999 2024-09-10T19:18:44.997 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:18:44.997 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:18:44.997 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-10T19:18:44.997 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:26.540215+0000", 2024-09-10T19:18:44.997 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.7060000000000001e-06 2024-09-10T19:18:44.997 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:18:44.997 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:18:44.997 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-10T19:18:44.997 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:26.540226+0000", 2024-09-10T19:18:44.997 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.1377e-05 2024-09-10T19:18:44.997 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:18:44.998 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:18:44.998 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-10T19:18:44.998 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:26.540232+0000", 2024-09-10T19:18:44.998 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.913e-06 2024-09-10T19:18:44.998 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:18:44.998 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:18:44.998 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-10T19:18:44.998 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:26.544071+0000", 2024-09-10T19:18:44.998 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.003839103 2024-09-10T19:18:44.998 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:18:44.998 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:18:44.998 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-10T19:18:44.998 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:26.544192+0000", 2024-09-10T19:18:44.998 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000120612 2024-09-10T19:18:44.999 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:18:44.999 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:18:44.999 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 4,7", 2024-09-10T19:18:44.999 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:26.544332+0000", 2024-09-10T19:18:44.999 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000140436 2024-09-10T19:18:44.999 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:18:44.999 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:18:44.999 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-10T19:18:44.999 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:26.547364+0000", 2024-09-10T19:18:44.999 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0030315899999999998 2024-09-10T19:18:44.999 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:18:45.000 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:18:45.000 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:18:45.000 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:26.549943+0000", 2024-09-10T19:18:45.000 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.002579507 2024-09-10T19:18:45.000 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:18:45.000 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:18:45.000 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:18:45.000 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:26.635107+0000", 2024-09-10T19:18:45.000 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.085163191999999999 2024-09-10T19:18:45.000 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:18:45.000 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:18:45.000 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-10T19:18:45.000 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:26.635152+0000", 2024-09-10T19:18:45.000 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.5419e-05 2024-09-10T19:18:45.000 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:18:45.001 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:18:45.001 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-10T19:18:45.001 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:26.635187+0000", 2024-09-10T19:18:45.001 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.5364999999999998e-05 2024-09-10T19:18:45.001 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:18:45.001 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:18:45.001 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:18:45.001 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:18:45.001 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:18:45.001 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.14781.0:218745 7.e 7:76dfdc5d:::benchmark_data_smithi183_84160_object78364:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e132)", 2024-09-10T19:18:45.001 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-10T19:14:58.676347+0000", 2024-09-10T19:18:45.001 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 226.21016834, 2024-09-10T19:18:45.001 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.10844561699999999, 2024-09-10T19:18:45.001 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-10T19:18:45.002 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T19:18:45.002 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-10T19:18:45.002 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.14781", 2024-09-10T19:18:45.002 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.183:0/2025776420", 2024-09-10T19:18:45.002 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 218745 2024-09-10T19:18:45.002 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:18:45.002 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-10T19:18:45.002 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:18:45.002 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-10T19:18:45.002 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:58.676347+0000", 2024-09-10T19:18:45.002 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:18:45.002 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:18:45.002 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:18:45.002 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-10T19:18:45.002 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:58.676347+0000", 2024-09-10T19:18:45.002 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:18:45.003 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:18:45.003 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:18:45.003 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-10T19:18:45.003 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:58.676344+0000", 2024-09-10T19:18:45.003 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4294967295.9999976 2024-09-10T19:18:45.003 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:18:45.003 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:18:45.003 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-10T19:18:45.003 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:58.676352+0000", 2024-09-10T19:18:45.003 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.1280000000000008e-06 2024-09-10T19:18:45.003 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:18:45.003 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:18:45.003 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-10T19:18:45.003 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:58.676366+0000", 2024-09-10T19:18:45.003 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.3864e-05 2024-09-10T19:18:45.004 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:18:45.004 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:18:45.004 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-10T19:18:45.004 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:58.676372+0000", 2024-09-10T19:18:45.004 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.1009999999999997e-06 2024-09-10T19:18:45.004 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:18:45.004 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:18:45.004 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-10T19:18:45.004 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:58.681308+0000", 2024-09-10T19:18:45.004 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0049355149999999997 2024-09-10T19:18:45.004 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:18:45.004 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:18:45.004 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-10T19:18:45.004 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:58.681423+0000", 2024-09-10T19:18:45.004 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00011496 2024-09-10T19:18:45.005 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:18:45.005 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:18:45.005 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 0,5", 2024-09-10T19:18:45.005 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:58.681532+0000", 2024-09-10T19:18:45.005 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00010943899999999999 2024-09-10T19:18:45.005 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:18:45.005 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:18:45.005 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:18:45.005 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:58.686635+0000", 2024-09-10T19:18:45.005 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.005102667 2024-09-10T19:18:45.005 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:18:45.005 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:18:45.005 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-10T19:18:45.005 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:58.688709+0000", 2024-09-10T19:18:45.006 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0020736449999999998 2024-09-10T19:18:45.006 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:18:45.006 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:18:45.006 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:18:45.006 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:58.784711+0000", 2024-09-10T19:18:45.006 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.096002303999999997 2024-09-10T19:18:45.006 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:18:45.006 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:18:45.006 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-10T19:18:45.006 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:58.784753+0000", 2024-09-10T19:18:45.006 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.2373000000000002e-05 2024-09-10T19:18:45.006 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:18:45.006 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:18:45.006 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-10T19:18:45.006 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:58.784792+0000", 2024-09-10T19:18:45.007 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.8924999999999997e-05 2024-09-10T19:18:45.007 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:18:45.007 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:18:45.007 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:18:45.007 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:18:45.007 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:18:45.007 INFO:teuthology.orchestra.run.smithi183.stdout:} 2024-09-10T19:18:45.007 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:18:45.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:18:45.030 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:18:45.030 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:18:45.130 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:18:45.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:18:45.154 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:18:45.155 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:18:45.255 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:18:45.284 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:18:45.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:18:45.285 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:18:45.385 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:18:45.413 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:18:45.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:18:45.414 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:18:45.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:18:45 smithi183 ceph-mon[29736]: pgmap v375: 158 pgs: 1 active+recovering+remapped, 3 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 1 active+recovering, 152 active+clean; 4.4 GiB data, 17 GiB used, 520 GiB / 536 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.16k op/s; 7141/216018 objects misplaced (3.306%); 45 MiB/s, 725 objects/s recovering 2024-09-10T19:18:45.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:18:45 smithi183 ceph-mon[29736]: osdmap e206: 8 total, 8 up, 6 in 2024-09-10T19:18:45.431 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:18:45 smithi183 ceph-mon[41562]: pgmap v375: 158 pgs: 1 active+recovering+remapped, 3 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 1 active+recovering, 152 active+clean; 4.4 GiB data, 17 GiB used, 520 GiB / 536 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.16k op/s; 7141/216018 objects misplaced (3.306%); 45 MiB/s, 725 objects/s recovering 2024-09-10T19:18:45.431 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:18:45 smithi183 ceph-mon[41562]: osdmap e206: 8 total, 8 up, 6 in 2024-09-10T19:18:45.465 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:18:45 smithi191 ceph-mon[32884]: pgmap v375: 158 pgs: 1 active+recovering+remapped, 3 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 1 active+recovering, 152 active+clean; 4.4 GiB data, 17 GiB used, 520 GiB / 536 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.16k op/s; 7141/216018 objects misplaced (3.306%); 45 MiB/s, 725 objects/s recovering 2024-09-10T19:18:45.465 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:18:45 smithi191 ceph-mon[32884]: osdmap e206: 8 total, 8 up, 6 in 2024-09-10T19:18:45.515 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:18:45.541 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:18:45.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:18:45.542 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:18:45.642 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:18:45.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:18:45.677 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:18:45.677 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:18:45.778 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:18:45.814 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:18:45.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:18:45.815 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:18:45.916 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:18:45.955 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:18:45.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:18:45.956 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:18:46.056 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:18:46.083 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:18:46.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:18:46.085 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:18:46.144 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-10T19:18:46.186 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:18:46.214 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:18:46.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:18:46.215 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:18:46.316 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:18:46.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:18:46.356 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:18:46.356 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:18:46.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:18:46 smithi183 ceph-mon[29736]: osdmap e207: 8 total, 8 up, 6 in 2024-09-10T19:18:46.431 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:18:46 smithi183 ceph-mon[41562]: osdmap e207: 8 total, 8 up, 6 in 2024-09-10T19:18:46.457 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:18:46.465 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:18:46 smithi191 ceph-mon[32884]: osdmap e207: 8 total, 8 up, 6 in 2024-09-10T19:18:46.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:18:46.505 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:18:46.505 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:18:46.606 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:18:46.632 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:18:46.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:18:46.634 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:18:46.734 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:18:46.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:18:46.778 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:18:46.778 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:18:46.880 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:18:46.915 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:18:46.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:18:46.916 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:18:47.017 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:18:47.057 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:18:47.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:18:47.058 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:18:47.159 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:18:47.201 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:18:47.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:18:47.202 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:18:47.303 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:18:47.340 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:18:47.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:18:47.341 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:18:47.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:18:47 smithi183 ceph-mon[29736]: pgmap v378: 158 pgs: 1 peering, 1 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 155 active+clean; 4.1 GiB data, 16 GiB used, 521 GiB / 536 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.60k op/s; 5813/202986 objects misplaced (2.864%); 61 MiB/s, 978 objects/s recovering 2024-09-10T19:18:47.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:18:47 smithi183 ceph-mon[29736]: purged_snaps scrub starts 2024-09-10T19:18:47.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:18:47 smithi183 ceph-mon[29736]: purged_snaps scrub ok 2024-09-10T19:18:47.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:18:47 smithi183 ceph-mon[41562]: pgmap v378: 158 pgs: 1 peering, 1 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 155 active+clean; 4.1 GiB data, 16 GiB used, 521 GiB / 536 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.60k op/s; 5813/202986 objects misplaced (2.864%); 61 MiB/s, 978 objects/s recovering 2024-09-10T19:18:47.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:18:47 smithi183 ceph-mon[41562]: purged_snaps scrub starts 2024-09-10T19:18:47.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:18:47 smithi183 ceph-mon[41562]: purged_snaps scrub ok 2024-09-10T19:18:47.441 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:18:47.451 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:18:47 smithi191 ceph-mon[32884]: pgmap v378: 158 pgs: 1 peering, 1 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 155 active+clean; 4.1 GiB data, 16 GiB used, 521 GiB / 536 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.60k op/s; 5813/202986 objects misplaced (2.864%); 61 MiB/s, 978 objects/s recovering 2024-09-10T19:18:47.451 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:18:47 smithi191 ceph-mon[32884]: purged_snaps scrub starts 2024-09-10T19:18:47.451 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:18:47 smithi191 ceph-mon[32884]: purged_snaps scrub ok 2024-09-10T19:18:47.474 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:18:47.475 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph daemon osd.3 dump_ops_in_flight 2024-09-10T19:18:47.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:18:47.476 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:18:47.577 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:18:47.616 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:18:47.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:18:47.617 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:18:47.716 INFO:journalctl@ceph.mgr.x.smithi191.stdout:Sep 10 19:18:47 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-x[34967]: ::ffff:172.21.15.191 - - [10/Sep/2024:19:18:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T19:18:47.718 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:18:47.741 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:18:47.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:18:47.742 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:18:47.843 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:18:47.872 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:18:47.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:18:47.873 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:18:47.974 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:18:48.019 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:18:48.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:18:48.020 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:18:48.121 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:18:48.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:18:48.156 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:18:48.156 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:18:48.257 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:18:48.293 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:18:48.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:18:48.294 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:18:48.395 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:18:48.428 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:18:48.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:18:48.429 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:18:48.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:18:48 smithi183 ceph-mon[29736]: osdmap e208: 8 total, 8 up, 6 in 2024-09-10T19:18:48.431 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:18:48 smithi183 ceph-mon[41562]: osdmap e208: 8 total, 8 up, 6 in 2024-09-10T19:18:48.465 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:18:48 smithi191 ceph-mon[32884]: osdmap e208: 8 total, 8 up, 6 in 2024-09-10T19:18:48.530 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:18:48.556 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:18:48.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:18:48.557 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:18:48.658 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:18:48.695 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:18:48.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:18:48.696 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:18:48.797 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:18:48.824 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:18:48.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:18:48.825 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:18:48.926 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:18:48.951 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:18:48.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:18:48.952 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:18:49.052 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:18:49.093 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:18:49.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:18:49.094 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:18:49.195 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:18:49.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:18:49.225 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:18:49.225 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:18:49.326 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:18:49.326 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:18:49.367 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:18:49.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:18:49.368 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:18:49.372 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:18:49.461 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:18:49 smithi183 ceph-mon[29736]: pgmap v380: 158 pgs: 1 peering, 1 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 155 active+clean; 4.1 GiB data, 16 GiB used, 521 GiB / 536 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 941 op/s; 5813/202986 objects misplaced (2.864%); 34 MiB/s, 540 objects/s recovering 2024-09-10T19:18:49.462 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:18:49 smithi183 ceph-mon[41562]: pgmap v380: 158 pgs: 1 peering, 1 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 155 active+clean; 4.1 GiB data, 16 GiB used, 521 GiB / 536 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 941 op/s; 5813/202986 objects misplaced (2.864%); 34 MiB/s, 540 objects/s recovering 2024-09-10T19:18:49.465 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:18:49 smithi191 ceph-mon[32884]: pgmap v380: 158 pgs: 1 peering, 1 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 155 active+clean; 4.1 GiB data, 16 GiB used, 521 GiB / 536 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 941 op/s; 5813/202986 objects misplaced (2.864%); 34 MiB/s, 540 objects/s recovering 2024-09-10T19:18:49.466 INFO:journalctl@ceph.iscsi.iscsi.a.smithi191.stdout:Sep 10 19:18:49 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-iscsi-iscsi-a[61816]: debug there is no tcmu-runner data available 2024-09-10T19:18:49.469 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:18:49.505 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:18:49.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:18:49.506 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:18:49.607 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:18:49.641 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:18:49.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:18:49.642 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:18:49.742 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:18:49.770 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:18:49.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:18:49.771 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:18:49.872 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:18:49.899 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:18:49.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:18:49.900 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:18:50.001 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:18:50.028 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:18:50.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:18:50.029 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:18:50.130 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:18:50.161 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:18:50.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:18:50.163 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:18:50.198 INFO:teuthology.orchestra.run.smithi183.stderr:nodeep-scrub is set 2024-09-10T19:18:50.264 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:18:50.301 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:18:50.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:18:50.302 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:18:50.376 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:18:50 smithi183 ceph-mon[29736]: from='client.? 172.21.15.183:0/1071007636' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-10T19:18:50.376 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:18:50 smithi183 ceph-mon[41562]: from='client.? 172.21.15.183:0/1071007636' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-10T19:18:50.402 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:18:50.437 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:18:50.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:18:50.438 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:18:50.465 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:18:50 smithi191 ceph-mon[32884]: from='client.? 172.21.15.183:0/1071007636' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-10T19:18:50.539 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:18:50.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:18:50.563 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:18:50.563 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:18:50.664 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:18:50.705 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:18:50.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:18:50.707 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:18:50.807 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:18:50.820 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-09-10T19:18:50.820 INFO:teuthology.orchestra.run.smithi183.stdout:{ 2024-09-10T19:18:50.820 INFO:teuthology.orchestra.run.smithi183.stdout: "epoch": 209, 2024-09-10T19:18:50.820 INFO:teuthology.orchestra.run.smithi183.stdout: "fsid": "320f6df8-6fa7-11ef-bcea-c7b262605968", 2024-09-10T19:18:50.820 INFO:teuthology.orchestra.run.smithi183.stdout: "created": "2024-09-10T19:05:03.688531+0000", 2024-09-10T19:18:50.820 INFO:teuthology.orchestra.run.smithi183.stdout: "modified": "2024-09-10T19:18:50.185351+0000", 2024-09-10T19:18:50.820 INFO:teuthology.orchestra.run.smithi183.stdout: "last_up_change": "2024-09-10T19:15:19.923894+0000", 2024-09-10T19:18:50.820 INFO:teuthology.orchestra.run.smithi183.stdout: "last_in_change": "2024-09-10T19:16:23.815531+0000", 2024-09-10T19:18:50.820 INFO:teuthology.orchestra.run.smithi183.stdout: "flags": "noscrub,nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-09-10T19:18:50.820 INFO:teuthology.orchestra.run.smithi183.stdout: "flags_num": 5806080, 2024-09-10T19:18:50.820 INFO:teuthology.orchestra.run.smithi183.stdout: "flags_set": [ 2024-09-10T19:18:50.821 INFO:teuthology.orchestra.run.smithi183.stdout: "nodeep-scrub", 2024-09-10T19:18:50.821 INFO:teuthology.orchestra.run.smithi183.stdout: "noscrub", 2024-09-10T19:18:50.821 INFO:teuthology.orchestra.run.smithi183.stdout: "pglog_hardlimit", 2024-09-10T19:18:50.821 INFO:teuthology.orchestra.run.smithi183.stdout: "purged_snapdirs", 2024-09-10T19:18:50.821 INFO:teuthology.orchestra.run.smithi183.stdout: "recovery_deletes", 2024-09-10T19:18:50.821 INFO:teuthology.orchestra.run.smithi183.stdout: "sortbitwise" 2024-09-10T19:18:50.821 INFO:teuthology.orchestra.run.smithi183.stdout: ], 2024-09-10T19:18:50.821 INFO:teuthology.orchestra.run.smithi183.stdout: "crush_version": 19, 2024-09-10T19:18:50.821 INFO:teuthology.orchestra.run.smithi183.stdout: "full_ratio": 0.94999998807907104, 2024-09-10T19:18:50.821 INFO:teuthology.orchestra.run.smithi183.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-09-10T19:18:50.821 INFO:teuthology.orchestra.run.smithi183.stdout: "nearfull_ratio": 0.85000002384185791, 2024-09-10T19:18:50.821 INFO:teuthology.orchestra.run.smithi183.stdout: "cluster_snapshot": "", 2024-09-10T19:18:50.821 INFO:teuthology.orchestra.run.smithi183.stdout: "pool_max": 8, 2024-09-10T19:18:50.822 INFO:teuthology.orchestra.run.smithi183.stdout: "max_osd": 8, 2024-09-10T19:18:50.822 INFO:teuthology.orchestra.run.smithi183.stdout: "require_min_compat_client": "luminous", 2024-09-10T19:18:50.822 INFO:teuthology.orchestra.run.smithi183.stdout: "min_compat_client": "luminous", 2024-09-10T19:18:50.822 INFO:teuthology.orchestra.run.smithi183.stdout: "require_osd_release": "quincy", 2024-09-10T19:18:50.822 INFO:teuthology.orchestra.run.smithi183.stdout: "pools": [ 2024-09-10T19:18:50.822 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:18:50.822 INFO:teuthology.orchestra.run.smithi183.stdout: "pool": 1, 2024-09-10T19:18:50.822 INFO:teuthology.orchestra.run.smithi183.stdout: "pool_name": ".mgr", 2024-09-10T19:18:50.822 INFO:teuthology.orchestra.run.smithi183.stdout: "create_time": "2024-09-10T19:08:00.456786+0000", 2024-09-10T19:18:50.822 INFO:teuthology.orchestra.run.smithi183.stdout: "flags": 1, 2024-09-10T19:18:50.822 INFO:teuthology.orchestra.run.smithi183.stdout: "flags_names": "hashpspool", 2024-09-10T19:18:50.822 INFO:teuthology.orchestra.run.smithi183.stdout: "type": 1, 2024-09-10T19:18:50.822 INFO:teuthology.orchestra.run.smithi183.stdout: "size": 3, 2024-09-10T19:18:50.822 INFO:teuthology.orchestra.run.smithi183.stdout: "min_size": 2, 2024-09-10T19:18:50.823 INFO:teuthology.orchestra.run.smithi183.stdout: "crush_rule": 0, 2024-09-10T19:18:50.823 INFO:teuthology.orchestra.run.smithi183.stdout: "peering_crush_bucket_count": 0, 2024-09-10T19:18:50.823 INFO:teuthology.orchestra.run.smithi183.stdout: "peering_crush_bucket_target": 0, 2024-09-10T19:18:50.823 INFO:teuthology.orchestra.run.smithi183.stdout: "peering_crush_bucket_barrier": 0, 2024-09-10T19:18:50.823 INFO:teuthology.orchestra.run.smithi183.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-10T19:18:50.823 INFO:teuthology.orchestra.run.smithi183.stdout: "object_hash": 2, 2024-09-10T19:18:50.823 INFO:teuthology.orchestra.run.smithi183.stdout: "pg_autoscale_mode": "off", 2024-09-10T19:18:50.823 INFO:teuthology.orchestra.run.smithi183.stdout: "pg_num": 1, 2024-09-10T19:18:50.823 INFO:teuthology.orchestra.run.smithi183.stdout: "pg_placement_num": 1, 2024-09-10T19:18:50.823 INFO:teuthology.orchestra.run.smithi183.stdout: "pg_placement_num_target": 1, 2024-09-10T19:18:50.823 INFO:teuthology.orchestra.run.smithi183.stdout: "pg_num_target": 1, 2024-09-10T19:18:50.823 INFO:teuthology.orchestra.run.smithi183.stdout: "pg_num_pending": 1, 2024-09-10T19:18:50.823 INFO:teuthology.orchestra.run.smithi183.stdout: "last_pg_merge_meta": { 2024-09-10T19:18:50.824 INFO:teuthology.orchestra.run.smithi183.stdout: "source_pgid": "0.0", 2024-09-10T19:18:50.824 INFO:teuthology.orchestra.run.smithi183.stdout: "ready_epoch": 0, 2024-09-10T19:18:50.824 INFO:teuthology.orchestra.run.smithi183.stdout: "last_epoch_started": 0, 2024-09-10T19:18:50.824 INFO:teuthology.orchestra.run.smithi183.stdout: "last_epoch_clean": 0, 2024-09-10T19:18:50.824 INFO:teuthology.orchestra.run.smithi183.stdout: "source_version": "0'0", 2024-09-10T19:18:50.824 INFO:teuthology.orchestra.run.smithi183.stdout: "target_version": "0'0" 2024-09-10T19:18:50.824 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:18:50.824 INFO:teuthology.orchestra.run.smithi183.stdout: "last_change": "202", 2024-09-10T19:18:50.824 INFO:teuthology.orchestra.run.smithi183.stdout: "last_force_op_resend": "0", 2024-09-10T19:18:50.824 INFO:teuthology.orchestra.run.smithi183.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-10T19:18:50.824 INFO:teuthology.orchestra.run.smithi183.stdout: "last_force_op_resend_preluminous": "0", 2024-09-10T19:18:50.824 INFO:teuthology.orchestra.run.smithi183.stdout: "auid": 0, 2024-09-10T19:18:50.824 INFO:teuthology.orchestra.run.smithi183.stdout: "snap_mode": "selfmanaged", 2024-09-10T19:18:50.824 INFO:teuthology.orchestra.run.smithi183.stdout: "snap_seq": 0, 2024-09-10T19:18:50.825 INFO:teuthology.orchestra.run.smithi183.stdout: "snap_epoch": 0, 2024-09-10T19:18:50.825 INFO:teuthology.orchestra.run.smithi183.stdout: "pool_snaps": [], 2024-09-10T19:18:50.825 INFO:teuthology.orchestra.run.smithi183.stdout: "removed_snaps": "[]", 2024-09-10T19:18:50.825 INFO:teuthology.orchestra.run.smithi183.stdout: "quota_max_bytes": 0, 2024-09-10T19:18:50.825 INFO:teuthology.orchestra.run.smithi183.stdout: "quota_max_objects": 0, 2024-09-10T19:18:50.825 INFO:teuthology.orchestra.run.smithi183.stdout: "tiers": [], 2024-09-10T19:18:50.825 INFO:teuthology.orchestra.run.smithi183.stdout: "tier_of": -1, 2024-09-10T19:18:50.825 INFO:teuthology.orchestra.run.smithi183.stdout: "read_tier": -1, 2024-09-10T19:18:50.825 INFO:teuthology.orchestra.run.smithi183.stdout: "write_tier": -1, 2024-09-10T19:18:50.825 INFO:teuthology.orchestra.run.smithi183.stdout: "cache_mode": "none", 2024-09-10T19:18:50.825 INFO:teuthology.orchestra.run.smithi183.stdout: "target_max_bytes": 0, 2024-09-10T19:18:50.825 INFO:teuthology.orchestra.run.smithi183.stdout: "target_max_objects": 0, 2024-09-10T19:18:50.825 INFO:teuthology.orchestra.run.smithi183.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-10T19:18:50.826 INFO:teuthology.orchestra.run.smithi183.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-10T19:18:50.826 INFO:teuthology.orchestra.run.smithi183.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-10T19:18:50.826 INFO:teuthology.orchestra.run.smithi183.stdout: "cache_min_flush_age": 0, 2024-09-10T19:18:50.826 INFO:teuthology.orchestra.run.smithi183.stdout: "cache_min_evict_age": 0, 2024-09-10T19:18:50.826 INFO:teuthology.orchestra.run.smithi183.stdout: "erasure_code_profile": "", 2024-09-10T19:18:50.826 INFO:teuthology.orchestra.run.smithi183.stdout: "hit_set_params": { 2024-09-10T19:18:50.826 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "none" 2024-09-10T19:18:50.826 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:18:50.826 INFO:teuthology.orchestra.run.smithi183.stdout: "hit_set_period": 0, 2024-09-10T19:18:50.826 INFO:teuthology.orchestra.run.smithi183.stdout: "hit_set_count": 0, 2024-09-10T19:18:50.826 INFO:teuthology.orchestra.run.smithi183.stdout: "use_gmt_hitset": true, 2024-09-10T19:18:50.826 INFO:teuthology.orchestra.run.smithi183.stdout: "min_read_recency_for_promote": 0, 2024-09-10T19:18:50.826 INFO:teuthology.orchestra.run.smithi183.stdout: "min_write_recency_for_promote": 0, 2024-09-10T19:18:50.826 INFO:teuthology.orchestra.run.smithi183.stdout: "hit_set_grade_decay_rate": 0, 2024-09-10T19:18:50.826 INFO:teuthology.orchestra.run.smithi183.stdout: "hit_set_search_last_n": 0, 2024-09-10T19:18:50.827 INFO:teuthology.orchestra.run.smithi183.stdout: "grade_table": [], 2024-09-10T19:18:50.827 INFO:teuthology.orchestra.run.smithi183.stdout: "stripe_width": 0, 2024-09-10T19:18:50.827 INFO:teuthology.orchestra.run.smithi183.stdout: "expected_num_objects": 0, 2024-09-10T19:18:50.827 INFO:teuthology.orchestra.run.smithi183.stdout: "fast_read": false, 2024-09-10T19:18:50.827 INFO:teuthology.orchestra.run.smithi183.stdout: "options": { 2024-09-10T19:18:50.827 INFO:teuthology.orchestra.run.smithi183.stdout: "pg_num_max": 32, 2024-09-10T19:18:50.827 INFO:teuthology.orchestra.run.smithi183.stdout: "pg_num_min": 1 2024-09-10T19:18:50.827 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:18:50.827 INFO:teuthology.orchestra.run.smithi183.stdout: "application_metadata": { 2024-09-10T19:18:50.827 INFO:teuthology.orchestra.run.smithi183.stdout: "mgr": {} 2024-09-10T19:18:50.827 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:18:50.827 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:18:50.827 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:18:50.828 INFO:teuthology.orchestra.run.smithi183.stdout: "pool": 2, 2024-09-10T19:18:50.828 INFO:teuthology.orchestra.run.smithi183.stdout: "pool_name": ".rgw.root", 2024-09-10T19:18:50.828 INFO:teuthology.orchestra.run.smithi183.stdout: "create_time": "2024-09-10T19:10:05.993763+0000", 2024-09-10T19:18:50.828 INFO:teuthology.orchestra.run.smithi183.stdout: "flags": 1, 2024-09-10T19:18:50.828 INFO:teuthology.orchestra.run.smithi183.stdout: "flags_names": "hashpspool", 2024-09-10T19:18:50.828 INFO:teuthology.orchestra.run.smithi183.stdout: "type": 1, 2024-09-10T19:18:50.828 INFO:teuthology.orchestra.run.smithi183.stdout: "size": 3, 2024-09-10T19:18:50.828 INFO:teuthology.orchestra.run.smithi183.stdout: "min_size": 2, 2024-09-10T19:18:50.828 INFO:teuthology.orchestra.run.smithi183.stdout: "crush_rule": 0, 2024-09-10T19:18:50.828 INFO:teuthology.orchestra.run.smithi183.stdout: "peering_crush_bucket_count": 0, 2024-09-10T19:18:50.828 INFO:teuthology.orchestra.run.smithi183.stdout: "peering_crush_bucket_target": 0, 2024-09-10T19:18:50.828 INFO:teuthology.orchestra.run.smithi183.stdout: "peering_crush_bucket_barrier": 0, 2024-09-10T19:18:50.828 INFO:teuthology.orchestra.run.smithi183.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-10T19:18:50.828 INFO:teuthology.orchestra.run.smithi183.stdout: "object_hash": 2, 2024-09-10T19:18:50.829 INFO:teuthology.orchestra.run.smithi183.stdout: "pg_autoscale_mode": "off", 2024-09-10T19:18:50.829 INFO:teuthology.orchestra.run.smithi183.stdout: "pg_num": 32, 2024-09-10T19:18:50.829 INFO:teuthology.orchestra.run.smithi183.stdout: "pg_placement_num": 32, 2024-09-10T19:18:50.829 INFO:teuthology.orchestra.run.smithi183.stdout: "pg_placement_num_target": 32, 2024-09-10T19:18:50.829 INFO:teuthology.orchestra.run.smithi183.stdout: "pg_num_target": 32, 2024-09-10T19:18:50.829 INFO:teuthology.orchestra.run.smithi183.stdout: "pg_num_pending": 32, 2024-09-10T19:18:50.829 INFO:teuthology.orchestra.run.smithi183.stdout: "last_pg_merge_meta": { 2024-09-10T19:18:50.829 INFO:teuthology.orchestra.run.smithi183.stdout: "source_pgid": "0.0", 2024-09-10T19:18:50.829 INFO:teuthology.orchestra.run.smithi183.stdout: "ready_epoch": 0, 2024-09-10T19:18:50.829 INFO:teuthology.orchestra.run.smithi183.stdout: "last_epoch_started": 0, 2024-09-10T19:18:50.829 INFO:teuthology.orchestra.run.smithi183.stdout: "last_epoch_clean": 0, 2024-09-10T19:18:50.829 INFO:teuthology.orchestra.run.smithi183.stdout: "source_version": "0'0", 2024-09-10T19:18:50.829 INFO:teuthology.orchestra.run.smithi183.stdout: "target_version": "0'0" 2024-09-10T19:18:50.829 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:18:50.830 INFO:teuthology.orchestra.run.smithi183.stdout: "last_change": "60", 2024-09-10T19:18:50.830 INFO:teuthology.orchestra.run.smithi183.stdout: "last_force_op_resend": "0", 2024-09-10T19:18:50.830 INFO:teuthology.orchestra.run.smithi183.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-10T19:18:50.830 INFO:teuthology.orchestra.run.smithi183.stdout: "last_force_op_resend_preluminous": "0", 2024-09-10T19:18:50.830 INFO:teuthology.orchestra.run.smithi183.stdout: "auid": 0, 2024-09-10T19:18:50.830 INFO:teuthology.orchestra.run.smithi183.stdout: "snap_mode": "selfmanaged", 2024-09-10T19:18:50.830 INFO:teuthology.orchestra.run.smithi183.stdout: "snap_seq": 0, 2024-09-10T19:18:50.830 INFO:teuthology.orchestra.run.smithi183.stdout: "snap_epoch": 0, 2024-09-10T19:18:50.830 INFO:teuthology.orchestra.run.smithi183.stdout: "pool_snaps": [], 2024-09-10T19:18:50.830 INFO:teuthology.orchestra.run.smithi183.stdout: "removed_snaps": "[]", 2024-09-10T19:18:50.830 INFO:teuthology.orchestra.run.smithi183.stdout: "quota_max_bytes": 0, 2024-09-10T19:18:50.830 INFO:teuthology.orchestra.run.smithi183.stdout: "quota_max_objects": 0, 2024-09-10T19:18:50.830 INFO:teuthology.orchestra.run.smithi183.stdout: "tiers": [], 2024-09-10T19:18:50.830 INFO:teuthology.orchestra.run.smithi183.stdout: "tier_of": -1, 2024-09-10T19:18:50.831 INFO:teuthology.orchestra.run.smithi183.stdout: "read_tier": -1, 2024-09-10T19:18:50.831 INFO:teuthology.orchestra.run.smithi183.stdout: "write_tier": -1, 2024-09-10T19:18:50.831 INFO:teuthology.orchestra.run.smithi183.stdout: "cache_mode": "none", 2024-09-10T19:18:50.831 INFO:teuthology.orchestra.run.smithi183.stdout: "target_max_bytes": 0, 2024-09-10T19:18:50.831 INFO:teuthology.orchestra.run.smithi183.stdout: "target_max_objects": 0, 2024-09-10T19:18:50.831 INFO:teuthology.orchestra.run.smithi183.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-10T19:18:50.831 INFO:teuthology.orchestra.run.smithi183.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-10T19:18:50.831 INFO:teuthology.orchestra.run.smithi183.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-10T19:18:50.831 INFO:teuthology.orchestra.run.smithi183.stdout: "cache_min_flush_age": 0, 2024-09-10T19:18:50.831 INFO:teuthology.orchestra.run.smithi183.stdout: "cache_min_evict_age": 0, 2024-09-10T19:18:50.831 INFO:teuthology.orchestra.run.smithi183.stdout: "erasure_code_profile": "", 2024-09-10T19:18:50.831 INFO:teuthology.orchestra.run.smithi183.stdout: "hit_set_params": { 2024-09-10T19:18:50.831 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "none" 2024-09-10T19:18:50.832 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:18:50.832 INFO:teuthology.orchestra.run.smithi183.stdout: "hit_set_period": 0, 2024-09-10T19:18:50.832 INFO:teuthology.orchestra.run.smithi183.stdout: "hit_set_count": 0, 2024-09-10T19:18:50.832 INFO:teuthology.orchestra.run.smithi183.stdout: "use_gmt_hitset": true, 2024-09-10T19:18:50.832 INFO:teuthology.orchestra.run.smithi183.stdout: "min_read_recency_for_promote": 0, 2024-09-10T19:18:50.832 INFO:teuthology.orchestra.run.smithi183.stdout: "min_write_recency_for_promote": 0, 2024-09-10T19:18:50.832 INFO:teuthology.orchestra.run.smithi183.stdout: "hit_set_grade_decay_rate": 0, 2024-09-10T19:18:50.832 INFO:teuthology.orchestra.run.smithi183.stdout: "hit_set_search_last_n": 0, 2024-09-10T19:18:50.832 INFO:teuthology.orchestra.run.smithi183.stdout: "grade_table": [], 2024-09-10T19:18:50.832 INFO:teuthology.orchestra.run.smithi183.stdout: "stripe_width": 0, 2024-09-10T19:18:50.832 INFO:teuthology.orchestra.run.smithi183.stdout: "expected_num_objects": 0, 2024-09-10T19:18:50.832 INFO:teuthology.orchestra.run.smithi183.stdout: "fast_read": false, 2024-09-10T19:18:50.832 INFO:teuthology.orchestra.run.smithi183.stdout: "options": {}, 2024-09-10T19:18:50.832 INFO:teuthology.orchestra.run.smithi183.stdout: "application_metadata": { 2024-09-10T19:18:50.833 INFO:teuthology.orchestra.run.smithi183.stdout: "rgw": {} 2024-09-10T19:18:50.833 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:18:50.833 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:18:50.833 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:18:50.833 INFO:teuthology.orchestra.run.smithi183.stdout: "pool": 3, 2024-09-10T19:18:50.833 INFO:teuthology.orchestra.run.smithi183.stdout: "pool_name": "datapool", 2024-09-10T19:18:50.833 INFO:teuthology.orchestra.run.smithi183.stdout: "create_time": "2024-09-10T19:10:06.224267+0000", 2024-09-10T19:18:50.833 INFO:teuthology.orchestra.run.smithi183.stdout: "flags": 8193, 2024-09-10T19:18:50.833 INFO:teuthology.orchestra.run.smithi183.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-09-10T19:18:50.833 INFO:teuthology.orchestra.run.smithi183.stdout: "type": 1, 2024-09-10T19:18:50.833 INFO:teuthology.orchestra.run.smithi183.stdout: "size": 3, 2024-09-10T19:18:50.833 INFO:teuthology.orchestra.run.smithi183.stdout: "min_size": 2, 2024-09-10T19:18:50.833 INFO:teuthology.orchestra.run.smithi183.stdout: "crush_rule": 0, 2024-09-10T19:18:50.833 INFO:teuthology.orchestra.run.smithi183.stdout: "peering_crush_bucket_count": 0, 2024-09-10T19:18:50.834 INFO:teuthology.orchestra.run.smithi183.stdout: "peering_crush_bucket_target": 0, 2024-09-10T19:18:50.834 INFO:teuthology.orchestra.run.smithi183.stdout: "peering_crush_bucket_barrier": 0, 2024-09-10T19:18:50.834 INFO:teuthology.orchestra.run.smithi183.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-10T19:18:50.834 INFO:teuthology.orchestra.run.smithi183.stdout: "object_hash": 2, 2024-09-10T19:18:50.834 INFO:teuthology.orchestra.run.smithi183.stdout: "pg_autoscale_mode": "off", 2024-09-10T19:18:50.834 INFO:teuthology.orchestra.run.smithi183.stdout: "pg_num": 3, 2024-09-10T19:18:50.834 INFO:teuthology.orchestra.run.smithi183.stdout: "pg_placement_num": 3, 2024-09-10T19:18:50.834 INFO:teuthology.orchestra.run.smithi183.stdout: "pg_placement_num_target": 3, 2024-09-10T19:18:50.835 INFO:teuthology.orchestra.run.smithi183.stdout: "pg_num_target": 3, 2024-09-10T19:18:50.836 INFO:teuthology.orchestra.run.smithi183.stdout: "pg_num_pending": 3, 2024-09-10T19:18:50.836 INFO:teuthology.orchestra.run.smithi183.stdout: "last_pg_merge_meta": { 2024-09-10T19:18:50.836 INFO:teuthology.orchestra.run.smithi183.stdout: "source_pgid": "0.0", 2024-09-10T19:18:50.836 INFO:teuthology.orchestra.run.smithi183.stdout: "ready_epoch": 0, 2024-09-10T19:18:50.836 INFO:teuthology.orchestra.run.smithi183.stdout: "last_epoch_started": 0, 2024-09-10T19:18:50.836 INFO:teuthology.orchestra.run.smithi183.stdout: "last_epoch_clean": 0, 2024-09-10T19:18:50.836 INFO:teuthology.orchestra.run.smithi183.stdout: "source_version": "0'0", 2024-09-10T19:18:50.836 INFO:teuthology.orchestra.run.smithi183.stdout: "target_version": "0'0" 2024-09-10T19:18:50.836 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:18:50.836 INFO:teuthology.orchestra.run.smithi183.stdout: "last_change": "63", 2024-09-10T19:18:50.836 INFO:teuthology.orchestra.run.smithi183.stdout: "last_force_op_resend": "0", 2024-09-10T19:18:50.836 INFO:teuthology.orchestra.run.smithi183.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-10T19:18:50.836 INFO:teuthology.orchestra.run.smithi183.stdout: "last_force_op_resend_preluminous": "0", 2024-09-10T19:18:50.837 INFO:teuthology.orchestra.run.smithi183.stdout: "auid": 0, 2024-09-10T19:18:50.837 INFO:teuthology.orchestra.run.smithi183.stdout: "snap_mode": "selfmanaged", 2024-09-10T19:18:50.837 INFO:teuthology.orchestra.run.smithi183.stdout: "snap_seq": 3, 2024-09-10T19:18:50.837 INFO:teuthology.orchestra.run.smithi183.stdout: "snap_epoch": 63, 2024-09-10T19:18:50.837 INFO:teuthology.orchestra.run.smithi183.stdout: "pool_snaps": [], 2024-09-10T19:18:50.837 INFO:teuthology.orchestra.run.smithi183.stdout: "removed_snaps": "[]", 2024-09-10T19:18:50.837 INFO:teuthology.orchestra.run.smithi183.stdout: "quota_max_bytes": 0, 2024-09-10T19:18:50.837 INFO:teuthology.orchestra.run.smithi183.stdout: "quota_max_objects": 0, 2024-09-10T19:18:50.837 INFO:teuthology.orchestra.run.smithi183.stdout: "tiers": [], 2024-09-10T19:18:50.837 INFO:teuthology.orchestra.run.smithi183.stdout: "tier_of": -1, 2024-09-10T19:18:50.837 INFO:teuthology.orchestra.run.smithi183.stdout: "read_tier": -1, 2024-09-10T19:18:50.837 INFO:teuthology.orchestra.run.smithi183.stdout: "write_tier": -1, 2024-09-10T19:18:50.837 INFO:teuthology.orchestra.run.smithi183.stdout: "cache_mode": "none", 2024-09-10T19:18:50.837 INFO:teuthology.orchestra.run.smithi183.stdout: "target_max_bytes": 0, 2024-09-10T19:18:50.838 INFO:teuthology.orchestra.run.smithi183.stdout: "target_max_objects": 0, 2024-09-10T19:18:50.838 INFO:teuthology.orchestra.run.smithi183.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-10T19:18:50.838 INFO:teuthology.orchestra.run.smithi183.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-10T19:18:50.838 INFO:teuthology.orchestra.run.smithi183.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-10T19:18:50.838 INFO:teuthology.orchestra.run.smithi183.stdout: "cache_min_flush_age": 0, 2024-09-10T19:18:50.838 INFO:teuthology.orchestra.run.smithi183.stdout: "cache_min_evict_age": 0, 2024-09-10T19:18:50.838 INFO:teuthology.orchestra.run.smithi183.stdout: "erasure_code_profile": "", 2024-09-10T19:18:50.838 INFO:teuthology.orchestra.run.smithi183.stdout: "hit_set_params": { 2024-09-10T19:18:50.838 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "none" 2024-09-10T19:18:50.838 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:18:50.838 INFO:teuthology.orchestra.run.smithi183.stdout: "hit_set_period": 0, 2024-09-10T19:18:50.838 INFO:teuthology.orchestra.run.smithi183.stdout: "hit_set_count": 0, 2024-09-10T19:18:50.838 INFO:teuthology.orchestra.run.smithi183.stdout: "use_gmt_hitset": true, 2024-09-10T19:18:50.838 INFO:teuthology.orchestra.run.smithi183.stdout: "min_read_recency_for_promote": 0, 2024-09-10T19:18:50.839 INFO:teuthology.orchestra.run.smithi183.stdout: "min_write_recency_for_promote": 0, 2024-09-10T19:18:50.839 INFO:teuthology.orchestra.run.smithi183.stdout: "hit_set_grade_decay_rate": 0, 2024-09-10T19:18:50.839 INFO:teuthology.orchestra.run.smithi183.stdout: "hit_set_search_last_n": 0, 2024-09-10T19:18:50.839 INFO:teuthology.orchestra.run.smithi183.stdout: "grade_table": [], 2024-09-10T19:18:50.839 INFO:teuthology.orchestra.run.smithi183.stdout: "stripe_width": 0, 2024-09-10T19:18:50.839 INFO:teuthology.orchestra.run.smithi183.stdout: "expected_num_objects": 0, 2024-09-10T19:18:50.839 INFO:teuthology.orchestra.run.smithi183.stdout: "fast_read": false, 2024-09-10T19:18:50.839 INFO:teuthology.orchestra.run.smithi183.stdout: "options": {}, 2024-09-10T19:18:50.839 INFO:teuthology.orchestra.run.smithi183.stdout: "application_metadata": { 2024-09-10T19:18:50.839 INFO:teuthology.orchestra.run.smithi183.stdout: "rbd": {} 2024-09-10T19:18:50.839 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:18:50.839 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:18:50.839 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:18:50.839 INFO:teuthology.orchestra.run.smithi183.stdout: "pool": 4, 2024-09-10T19:18:50.840 INFO:teuthology.orchestra.run.smithi183.stdout: "pool_name": "default.rgw.log", 2024-09-10T19:18:50.840 INFO:teuthology.orchestra.run.smithi183.stdout: "create_time": "2024-09-10T19:10:07.958284+0000", 2024-09-10T19:18:50.840 INFO:teuthology.orchestra.run.smithi183.stdout: "flags": 1, 2024-09-10T19:18:50.840 INFO:teuthology.orchestra.run.smithi183.stdout: "flags_names": "hashpspool", 2024-09-10T19:18:50.840 INFO:teuthology.orchestra.run.smithi183.stdout: "type": 1, 2024-09-10T19:18:50.840 INFO:teuthology.orchestra.run.smithi183.stdout: "size": 3, 2024-09-10T19:18:50.840 INFO:teuthology.orchestra.run.smithi183.stdout: "min_size": 2, 2024-09-10T19:18:50.840 INFO:teuthology.orchestra.run.smithi183.stdout: "crush_rule": 0, 2024-09-10T19:18:50.840 INFO:teuthology.orchestra.run.smithi183.stdout: "peering_crush_bucket_count": 0, 2024-09-10T19:18:50.840 INFO:teuthology.orchestra.run.smithi183.stdout: "peering_crush_bucket_target": 0, 2024-09-10T19:18:50.840 INFO:teuthology.orchestra.run.smithi183.stdout: "peering_crush_bucket_barrier": 0, 2024-09-10T19:18:50.840 INFO:teuthology.orchestra.run.smithi183.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-10T19:18:50.840 INFO:teuthology.orchestra.run.smithi183.stdout: "object_hash": 2, 2024-09-10T19:18:50.841 INFO:teuthology.orchestra.run.smithi183.stdout: "pg_autoscale_mode": "off", 2024-09-10T19:18:50.841 INFO:teuthology.orchestra.run.smithi183.stdout: "pg_num": 32, 2024-09-10T19:18:50.841 INFO:teuthology.orchestra.run.smithi183.stdout: "pg_placement_num": 32, 2024-09-10T19:18:50.841 INFO:teuthology.orchestra.run.smithi183.stdout: "pg_placement_num_target": 32, 2024-09-10T19:18:50.841 INFO:teuthology.orchestra.run.smithi183.stdout: "pg_num_target": 32, 2024-09-10T19:18:50.841 INFO:teuthology.orchestra.run.smithi183.stdout: "pg_num_pending": 32, 2024-09-10T19:18:50.841 INFO:teuthology.orchestra.run.smithi183.stdout: "last_pg_merge_meta": { 2024-09-10T19:18:50.841 INFO:teuthology.orchestra.run.smithi183.stdout: "source_pgid": "0.0", 2024-09-10T19:18:50.841 INFO:teuthology.orchestra.run.smithi183.stdout: "ready_epoch": 0, 2024-09-10T19:18:50.841 INFO:teuthology.orchestra.run.smithi183.stdout: "last_epoch_started": 0, 2024-09-10T19:18:50.841 INFO:teuthology.orchestra.run.smithi183.stdout: "last_epoch_clean": 0, 2024-09-10T19:18:50.841 INFO:teuthology.orchestra.run.smithi183.stdout: "source_version": "0'0", 2024-09-10T19:18:50.841 INFO:teuthology.orchestra.run.smithi183.stdout: "target_version": "0'0" 2024-09-10T19:18:50.841 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:18:50.842 INFO:teuthology.orchestra.run.smithi183.stdout: "last_change": "62", 2024-09-10T19:18:50.842 INFO:teuthology.orchestra.run.smithi183.stdout: "last_force_op_resend": "0", 2024-09-10T19:18:50.842 INFO:teuthology.orchestra.run.smithi183.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-10T19:18:50.842 INFO:teuthology.orchestra.run.smithi183.stdout: "last_force_op_resend_preluminous": "0", 2024-09-10T19:18:50.842 INFO:teuthology.orchestra.run.smithi183.stdout: "auid": 0, 2024-09-10T19:18:50.842 INFO:teuthology.orchestra.run.smithi183.stdout: "snap_mode": "selfmanaged", 2024-09-10T19:18:50.842 INFO:teuthology.orchestra.run.smithi183.stdout: "snap_seq": 0, 2024-09-10T19:18:50.842 INFO:teuthology.orchestra.run.smithi183.stdout: "snap_epoch": 0, 2024-09-10T19:18:50.842 INFO:teuthology.orchestra.run.smithi183.stdout: "pool_snaps": [], 2024-09-10T19:18:50.842 INFO:teuthology.orchestra.run.smithi183.stdout: "removed_snaps": "[]", 2024-09-10T19:18:50.842 INFO:teuthology.orchestra.run.smithi183.stdout: "quota_max_bytes": 0, 2024-09-10T19:18:50.842 INFO:teuthology.orchestra.run.smithi183.stdout: "quota_max_objects": 0, 2024-09-10T19:18:50.842 INFO:teuthology.orchestra.run.smithi183.stdout: "tiers": [], 2024-09-10T19:18:50.842 INFO:teuthology.orchestra.run.smithi183.stdout: "tier_of": -1, 2024-09-10T19:18:50.843 INFO:teuthology.orchestra.run.smithi183.stdout: "read_tier": -1, 2024-09-10T19:18:50.843 INFO:teuthology.orchestra.run.smithi183.stdout: "write_tier": -1, 2024-09-10T19:18:50.843 INFO:teuthology.orchestra.run.smithi183.stdout: "cache_mode": "none", 2024-09-10T19:18:50.843 INFO:teuthology.orchestra.run.smithi183.stdout: "target_max_bytes": 0, 2024-09-10T19:18:50.843 INFO:teuthology.orchestra.run.smithi183.stdout: "target_max_objects": 0, 2024-09-10T19:18:50.843 INFO:teuthology.orchestra.run.smithi183.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-10T19:18:50.843 INFO:teuthology.orchestra.run.smithi183.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-10T19:18:50.843 INFO:teuthology.orchestra.run.smithi183.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-10T19:18:50.843 INFO:teuthology.orchestra.run.smithi183.stdout: "cache_min_flush_age": 0, 2024-09-10T19:18:50.843 INFO:teuthology.orchestra.run.smithi183.stdout: "cache_min_evict_age": 0, 2024-09-10T19:18:50.843 INFO:teuthology.orchestra.run.smithi183.stdout: "erasure_code_profile": "", 2024-09-10T19:18:50.843 INFO:teuthology.orchestra.run.smithi183.stdout: "hit_set_params": { 2024-09-10T19:18:50.843 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "none" 2024-09-10T19:18:50.844 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:18:50.844 INFO:teuthology.orchestra.run.smithi183.stdout: "hit_set_period": 0, 2024-09-10T19:18:50.844 INFO:teuthology.orchestra.run.smithi183.stdout: "hit_set_count": 0, 2024-09-10T19:18:50.844 INFO:teuthology.orchestra.run.smithi183.stdout: "use_gmt_hitset": true, 2024-09-10T19:18:50.844 INFO:teuthology.orchestra.run.smithi183.stdout: "min_read_recency_for_promote": 0, 2024-09-10T19:18:50.844 INFO:teuthology.orchestra.run.smithi183.stdout: "min_write_recency_for_promote": 0, 2024-09-10T19:18:50.844 INFO:teuthology.orchestra.run.smithi183.stdout: "hit_set_grade_decay_rate": 0, 2024-09-10T19:18:50.844 INFO:teuthology.orchestra.run.smithi183.stdout: "hit_set_search_last_n": 0, 2024-09-10T19:18:50.844 INFO:teuthology.orchestra.run.smithi183.stdout: "grade_table": [], 2024-09-10T19:18:50.844 INFO:teuthology.orchestra.run.smithi183.stdout: "stripe_width": 0, 2024-09-10T19:18:50.844 INFO:teuthology.orchestra.run.smithi183.stdout: "expected_num_objects": 0, 2024-09-10T19:18:50.844 INFO:teuthology.orchestra.run.smithi183.stdout: "fast_read": false, 2024-09-10T19:18:50.844 INFO:teuthology.orchestra.run.smithi183.stdout: "options": {}, 2024-09-10T19:18:50.845 INFO:teuthology.orchestra.run.smithi183.stdout: "application_metadata": { 2024-09-10T19:18:50.845 INFO:teuthology.orchestra.run.smithi183.stdout: "rgw": {} 2024-09-10T19:18:50.845 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:18:50.845 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:18:50.845 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:18:50.845 INFO:teuthology.orchestra.run.smithi183.stdout: "pool": 5, 2024-09-10T19:18:50.845 INFO:teuthology.orchestra.run.smithi183.stdout: "pool_name": "default.rgw.control", 2024-09-10T19:18:50.845 INFO:teuthology.orchestra.run.smithi183.stdout: "create_time": "2024-09-10T19:10:09.933596+0000", 2024-09-10T19:18:50.845 INFO:teuthology.orchestra.run.smithi183.stdout: "flags": 1, 2024-09-10T19:18:50.845 INFO:teuthology.orchestra.run.smithi183.stdout: "flags_names": "hashpspool", 2024-09-10T19:18:50.845 INFO:teuthology.orchestra.run.smithi183.stdout: "type": 1, 2024-09-10T19:18:50.845 INFO:teuthology.orchestra.run.smithi183.stdout: "size": 3, 2024-09-10T19:18:50.845 INFO:teuthology.orchestra.run.smithi183.stdout: "min_size": 2, 2024-09-10T19:18:50.845 INFO:teuthology.orchestra.run.smithi183.stdout: "crush_rule": 0, 2024-09-10T19:18:50.846 INFO:teuthology.orchestra.run.smithi183.stdout: "peering_crush_bucket_count": 0, 2024-09-10T19:18:50.846 INFO:teuthology.orchestra.run.smithi183.stdout: "peering_crush_bucket_target": 0, 2024-09-10T19:18:50.846 INFO:teuthology.orchestra.run.smithi183.stdout: "peering_crush_bucket_barrier": 0, 2024-09-10T19:18:50.846 INFO:teuthology.orchestra.run.smithi183.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-10T19:18:50.846 INFO:teuthology.orchestra.run.smithi183.stdout: "object_hash": 2, 2024-09-10T19:18:50.846 INFO:teuthology.orchestra.run.smithi183.stdout: "pg_autoscale_mode": "off", 2024-09-10T19:18:50.846 INFO:teuthology.orchestra.run.smithi183.stdout: "pg_num": 32, 2024-09-10T19:18:50.846 INFO:teuthology.orchestra.run.smithi183.stdout: "pg_placement_num": 32, 2024-09-10T19:18:50.846 INFO:teuthology.orchestra.run.smithi183.stdout: "pg_placement_num_target": 32, 2024-09-10T19:18:50.846 INFO:teuthology.orchestra.run.smithi183.stdout: "pg_num_target": 32, 2024-09-10T19:18:50.846 INFO:teuthology.orchestra.run.smithi183.stdout: "pg_num_pending": 32, 2024-09-10T19:18:50.846 INFO:teuthology.orchestra.run.smithi183.stdout: "last_pg_merge_meta": { 2024-09-10T19:18:50.846 INFO:teuthology.orchestra.run.smithi183.stdout: "source_pgid": "0.0", 2024-09-10T19:18:50.846 INFO:teuthology.orchestra.run.smithi183.stdout: "ready_epoch": 0, 2024-09-10T19:18:50.847 INFO:teuthology.orchestra.run.smithi183.stdout: "last_epoch_started": 0, 2024-09-10T19:18:50.847 INFO:teuthology.orchestra.run.smithi183.stdout: "last_epoch_clean": 0, 2024-09-10T19:18:50.847 INFO:teuthology.orchestra.run.smithi183.stdout: "source_version": "0'0", 2024-09-10T19:18:50.847 INFO:teuthology.orchestra.run.smithi183.stdout: "target_version": "0'0" 2024-09-10T19:18:50.847 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:18:50.847 INFO:teuthology.orchestra.run.smithi183.stdout: "last_change": "64", 2024-09-10T19:18:50.847 INFO:teuthology.orchestra.run.smithi183.stdout: "last_force_op_resend": "0", 2024-09-10T19:18:50.847 INFO:teuthology.orchestra.run.smithi183.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-10T19:18:50.847 INFO:teuthology.orchestra.run.smithi183.stdout: "last_force_op_resend_preluminous": "0", 2024-09-10T19:18:50.847 INFO:teuthology.orchestra.run.smithi183.stdout: "auid": 0, 2024-09-10T19:18:50.847 INFO:teuthology.orchestra.run.smithi183.stdout: "snap_mode": "selfmanaged", 2024-09-10T19:18:50.847 INFO:teuthology.orchestra.run.smithi183.stdout: "snap_seq": 0, 2024-09-10T19:18:50.847 INFO:teuthology.orchestra.run.smithi183.stdout: "snap_epoch": 0, 2024-09-10T19:18:50.848 INFO:teuthology.orchestra.run.smithi183.stdout: "pool_snaps": [], 2024-09-10T19:18:50.848 INFO:teuthology.orchestra.run.smithi183.stdout: "removed_snaps": "[]", 2024-09-10T19:18:50.848 INFO:teuthology.orchestra.run.smithi183.stdout: "quota_max_bytes": 0, 2024-09-10T19:18:50.848 INFO:teuthology.orchestra.run.smithi183.stdout: "quota_max_objects": 0, 2024-09-10T19:18:50.848 INFO:teuthology.orchestra.run.smithi183.stdout: "tiers": [], 2024-09-10T19:18:50.848 INFO:teuthology.orchestra.run.smithi183.stdout: "tier_of": -1, 2024-09-10T19:18:50.848 INFO:teuthology.orchestra.run.smithi183.stdout: "read_tier": -1, 2024-09-10T19:18:50.848 INFO:teuthology.orchestra.run.smithi183.stdout: "write_tier": -1, 2024-09-10T19:18:50.848 INFO:teuthology.orchestra.run.smithi183.stdout: "cache_mode": "none", 2024-09-10T19:18:50.848 INFO:teuthology.orchestra.run.smithi183.stdout: "target_max_bytes": 0, 2024-09-10T19:18:50.848 INFO:teuthology.orchestra.run.smithi183.stdout: "target_max_objects": 0, 2024-09-10T19:18:50.848 INFO:teuthology.orchestra.run.smithi183.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-10T19:18:50.848 INFO:teuthology.orchestra.run.smithi183.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-10T19:18:50.849 INFO:teuthology.orchestra.run.smithi183.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-10T19:18:50.849 INFO:teuthology.orchestra.run.smithi183.stdout: "cache_min_flush_age": 0, 2024-09-10T19:18:50.849 INFO:teuthology.orchestra.run.smithi183.stdout: "cache_min_evict_age": 0, 2024-09-10T19:18:50.849 INFO:teuthology.orchestra.run.smithi183.stdout: "erasure_code_profile": "", 2024-09-10T19:18:50.849 INFO:teuthology.orchestra.run.smithi183.stdout: "hit_set_params": { 2024-09-10T19:18:50.849 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "none" 2024-09-10T19:18:50.849 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:18:50.849 INFO:teuthology.orchestra.run.smithi183.stdout: "hit_set_period": 0, 2024-09-10T19:18:50.849 INFO:teuthology.orchestra.run.smithi183.stdout: "hit_set_count": 0, 2024-09-10T19:18:50.849 INFO:teuthology.orchestra.run.smithi183.stdout: "use_gmt_hitset": true, 2024-09-10T19:18:50.849 INFO:teuthology.orchestra.run.smithi183.stdout: "min_read_recency_for_promote": 0, 2024-09-10T19:18:50.849 INFO:teuthology.orchestra.run.smithi183.stdout: "min_write_recency_for_promote": 0, 2024-09-10T19:18:50.849 INFO:teuthology.orchestra.run.smithi183.stdout: "hit_set_grade_decay_rate": 0, 2024-09-10T19:18:50.849 INFO:teuthology.orchestra.run.smithi183.stdout: "hit_set_search_last_n": 0, 2024-09-10T19:18:50.849 INFO:teuthology.orchestra.run.smithi183.stdout: "grade_table": [], 2024-09-10T19:18:50.850 INFO:teuthology.orchestra.run.smithi183.stdout: "stripe_width": 0, 2024-09-10T19:18:50.850 INFO:teuthology.orchestra.run.smithi183.stdout: "expected_num_objects": 0, 2024-09-10T19:18:50.850 INFO:teuthology.orchestra.run.smithi183.stdout: "fast_read": false, 2024-09-10T19:18:50.850 INFO:teuthology.orchestra.run.smithi183.stdout: "options": {}, 2024-09-10T19:18:50.850 INFO:teuthology.orchestra.run.smithi183.stdout: "application_metadata": { 2024-09-10T19:18:50.850 INFO:teuthology.orchestra.run.smithi183.stdout: "rgw": {} 2024-09-10T19:18:50.850 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:18:50.850 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:18:50.850 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:18:50.850 INFO:teuthology.orchestra.run.smithi183.stdout: "pool": 6, 2024-09-10T19:18:50.850 INFO:teuthology.orchestra.run.smithi183.stdout: "pool_name": "default.rgw.meta", 2024-09-10T19:18:50.850 INFO:teuthology.orchestra.run.smithi183.stdout: "create_time": "2024-09-10T19:10:11.993712+0000", 2024-09-10T19:18:50.850 INFO:teuthology.orchestra.run.smithi183.stdout: "flags": 1, 2024-09-10T19:18:50.850 INFO:teuthology.orchestra.run.smithi183.stdout: "flags_names": "hashpspool", 2024-09-10T19:18:50.850 INFO:teuthology.orchestra.run.smithi183.stdout: "type": 1, 2024-09-10T19:18:50.851 INFO:teuthology.orchestra.run.smithi183.stdout: "size": 3, 2024-09-10T19:18:50.851 INFO:teuthology.orchestra.run.smithi183.stdout: "min_size": 2, 2024-09-10T19:18:50.851 INFO:teuthology.orchestra.run.smithi183.stdout: "crush_rule": 0, 2024-09-10T19:18:50.851 INFO:teuthology.orchestra.run.smithi183.stdout: "peering_crush_bucket_count": 0, 2024-09-10T19:18:50.851 INFO:teuthology.orchestra.run.smithi183.stdout: "peering_crush_bucket_target": 0, 2024-09-10T19:18:50.851 INFO:teuthology.orchestra.run.smithi183.stdout: "peering_crush_bucket_barrier": 0, 2024-09-10T19:18:50.851 INFO:teuthology.orchestra.run.smithi183.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-10T19:18:50.851 INFO:teuthology.orchestra.run.smithi183.stdout: "object_hash": 2, 2024-09-10T19:18:50.851 INFO:teuthology.orchestra.run.smithi183.stdout: "pg_autoscale_mode": "off", 2024-09-10T19:18:50.851 INFO:teuthology.orchestra.run.smithi183.stdout: "pg_num": 32, 2024-09-10T19:18:50.851 INFO:teuthology.orchestra.run.smithi183.stdout: "pg_placement_num": 32, 2024-09-10T19:18:50.851 INFO:teuthology.orchestra.run.smithi183.stdout: "pg_placement_num_target": 32, 2024-09-10T19:18:50.851 INFO:teuthology.orchestra.run.smithi183.stdout: "pg_num_target": 32, 2024-09-10T19:18:50.851 INFO:teuthology.orchestra.run.smithi183.stdout: "pg_num_pending": 32, 2024-09-10T19:18:50.852 INFO:teuthology.orchestra.run.smithi183.stdout: "last_pg_merge_meta": { 2024-09-10T19:18:50.852 INFO:teuthology.orchestra.run.smithi183.stdout: "source_pgid": "0.0", 2024-09-10T19:18:50.852 INFO:teuthology.orchestra.run.smithi183.stdout: "ready_epoch": 0, 2024-09-10T19:18:50.852 INFO:teuthology.orchestra.run.smithi183.stdout: "last_epoch_started": 0, 2024-09-10T19:18:50.852 INFO:teuthology.orchestra.run.smithi183.stdout: "last_epoch_clean": 0, 2024-09-10T19:18:50.852 INFO:teuthology.orchestra.run.smithi183.stdout: "source_version": "0'0", 2024-09-10T19:18:50.852 INFO:teuthology.orchestra.run.smithi183.stdout: "target_version": "0'0" 2024-09-10T19:18:50.852 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:18:50.852 INFO:teuthology.orchestra.run.smithi183.stdout: "last_change": "66", 2024-09-10T19:18:50.852 INFO:teuthology.orchestra.run.smithi183.stdout: "last_force_op_resend": "0", 2024-09-10T19:18:50.852 INFO:teuthology.orchestra.run.smithi183.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-10T19:18:50.852 INFO:teuthology.orchestra.run.smithi183.stdout: "last_force_op_resend_preluminous": "0", 2024-09-10T19:18:50.852 INFO:teuthology.orchestra.run.smithi183.stdout: "auid": 0, 2024-09-10T19:18:50.853 INFO:teuthology.orchestra.run.smithi183.stdout: "snap_mode": "selfmanaged", 2024-09-10T19:18:50.853 INFO:teuthology.orchestra.run.smithi183.stdout: "snap_seq": 0, 2024-09-10T19:18:50.853 INFO:teuthology.orchestra.run.smithi183.stdout: "snap_epoch": 0, 2024-09-10T19:18:50.853 INFO:teuthology.orchestra.run.smithi183.stdout: "pool_snaps": [], 2024-09-10T19:18:50.853 INFO:teuthology.orchestra.run.smithi183.stdout: "removed_snaps": "[]", 2024-09-10T19:18:50.853 INFO:teuthology.orchestra.run.smithi183.stdout: "quota_max_bytes": 0, 2024-09-10T19:18:50.853 INFO:teuthology.orchestra.run.smithi183.stdout: "quota_max_objects": 0, 2024-09-10T19:18:50.853 INFO:teuthology.orchestra.run.smithi183.stdout: "tiers": [], 2024-09-10T19:18:50.853 INFO:teuthology.orchestra.run.smithi183.stdout: "tier_of": -1, 2024-09-10T19:18:50.853 INFO:teuthology.orchestra.run.smithi183.stdout: "read_tier": -1, 2024-09-10T19:18:50.853 INFO:teuthology.orchestra.run.smithi183.stdout: "write_tier": -1, 2024-09-10T19:18:50.853 INFO:teuthology.orchestra.run.smithi183.stdout: "cache_mode": "none", 2024-09-10T19:18:50.853 INFO:teuthology.orchestra.run.smithi183.stdout: "target_max_bytes": 0, 2024-09-10T19:18:50.853 INFO:teuthology.orchestra.run.smithi183.stdout: "target_max_objects": 0, 2024-09-10T19:18:50.854 INFO:teuthology.orchestra.run.smithi183.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-10T19:18:50.854 INFO:teuthology.orchestra.run.smithi183.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-10T19:18:50.854 INFO:teuthology.orchestra.run.smithi183.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-10T19:18:50.854 INFO:teuthology.orchestra.run.smithi183.stdout: "cache_min_flush_age": 0, 2024-09-10T19:18:50.854 INFO:teuthology.orchestra.run.smithi183.stdout: "cache_min_evict_age": 0, 2024-09-10T19:18:50.854 INFO:teuthology.orchestra.run.smithi183.stdout: "erasure_code_profile": "", 2024-09-10T19:18:50.854 INFO:teuthology.orchestra.run.smithi183.stdout: "hit_set_params": { 2024-09-10T19:18:50.854 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "none" 2024-09-10T19:18:50.854 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:18:50.854 INFO:teuthology.orchestra.run.smithi183.stdout: "hit_set_period": 0, 2024-09-10T19:18:50.854 INFO:teuthology.orchestra.run.smithi183.stdout: "hit_set_count": 0, 2024-09-10T19:18:50.854 INFO:teuthology.orchestra.run.smithi183.stdout: "use_gmt_hitset": true, 2024-09-10T19:18:50.854 INFO:teuthology.orchestra.run.smithi183.stdout: "min_read_recency_for_promote": 0, 2024-09-10T19:18:50.854 INFO:teuthology.orchestra.run.smithi183.stdout: "min_write_recency_for_promote": 0, 2024-09-10T19:18:50.855 INFO:teuthology.orchestra.run.smithi183.stdout: "hit_set_grade_decay_rate": 0, 2024-09-10T19:18:50.855 INFO:teuthology.orchestra.run.smithi183.stdout: "hit_set_search_last_n": 0, 2024-09-10T19:18:50.855 INFO:teuthology.orchestra.run.smithi183.stdout: "grade_table": [], 2024-09-10T19:18:50.855 INFO:teuthology.orchestra.run.smithi183.stdout: "stripe_width": 0, 2024-09-10T19:18:50.855 INFO:teuthology.orchestra.run.smithi183.stdout: "expected_num_objects": 0, 2024-09-10T19:18:50.855 INFO:teuthology.orchestra.run.smithi183.stdout: "fast_read": false, 2024-09-10T19:18:50.855 INFO:teuthology.orchestra.run.smithi183.stdout: "options": { 2024-09-10T19:18:50.855 INFO:teuthology.orchestra.run.smithi183.stdout: "pg_autoscale_bias": 4 2024-09-10T19:18:50.855 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:18:50.855 INFO:teuthology.orchestra.run.smithi183.stdout: "application_metadata": { 2024-09-10T19:18:50.855 INFO:teuthology.orchestra.run.smithi183.stdout: "rgw": {} 2024-09-10T19:18:50.855 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:18:50.855 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:18:50.856 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:18:50.856 INFO:teuthology.orchestra.run.smithi183.stdout: "pool": 8, 2024-09-10T19:18:50.856 INFO:teuthology.orchestra.run.smithi183.stdout: "pool_name": "unique_pool_1", 2024-09-10T19:18:50.856 INFO:teuthology.orchestra.run.smithi183.stdout: "create_time": "2024-09-10T19:16:11.474472+0000", 2024-09-10T19:18:50.856 INFO:teuthology.orchestra.run.smithi183.stdout: "flags": 1, 2024-09-10T19:18:50.856 INFO:teuthology.orchestra.run.smithi183.stdout: "flags_names": "hashpspool", 2024-09-10T19:18:50.856 INFO:teuthology.orchestra.run.smithi183.stdout: "type": 1, 2024-09-10T19:18:50.856 INFO:teuthology.orchestra.run.smithi183.stdout: "size": 3, 2024-09-10T19:18:50.856 INFO:teuthology.orchestra.run.smithi183.stdout: "min_size": 2, 2024-09-10T19:18:50.856 INFO:teuthology.orchestra.run.smithi183.stdout: "crush_rule": 0, 2024-09-10T19:18:50.856 INFO:teuthology.orchestra.run.smithi183.stdout: "peering_crush_bucket_count": 0, 2024-09-10T19:18:50.856 INFO:teuthology.orchestra.run.smithi183.stdout: "peering_crush_bucket_target": 0, 2024-09-10T19:18:50.856 INFO:teuthology.orchestra.run.smithi183.stdout: "peering_crush_bucket_barrier": 0, 2024-09-10T19:18:50.857 INFO:teuthology.orchestra.run.smithi183.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-10T19:18:50.857 INFO:teuthology.orchestra.run.smithi183.stdout: "object_hash": 2, 2024-09-10T19:18:50.857 INFO:teuthology.orchestra.run.smithi183.stdout: "pg_autoscale_mode": "off", 2024-09-10T19:18:50.857 INFO:teuthology.orchestra.run.smithi183.stdout: "pg_num": 26, 2024-09-10T19:18:50.857 INFO:teuthology.orchestra.run.smithi183.stdout: "pg_placement_num": 26, 2024-09-10T19:18:50.857 INFO:teuthology.orchestra.run.smithi183.stdout: "pg_placement_num_target": 26, 2024-09-10T19:18:50.857 INFO:teuthology.orchestra.run.smithi183.stdout: "pg_num_target": 26, 2024-09-10T19:18:50.857 INFO:teuthology.orchestra.run.smithi183.stdout: "pg_num_pending": 26, 2024-09-10T19:18:50.857 INFO:teuthology.orchestra.run.smithi183.stdout: "last_pg_merge_meta": { 2024-09-10T19:18:50.857 INFO:teuthology.orchestra.run.smithi183.stdout: "source_pgid": "0.0", 2024-09-10T19:18:50.857 INFO:teuthology.orchestra.run.smithi183.stdout: "ready_epoch": 0, 2024-09-10T19:18:50.857 INFO:teuthology.orchestra.run.smithi183.stdout: "last_epoch_started": 0, 2024-09-10T19:18:50.857 INFO:teuthology.orchestra.run.smithi183.stdout: "last_epoch_clean": 0, 2024-09-10T19:18:50.857 INFO:teuthology.orchestra.run.smithi183.stdout: "source_version": "0'0", 2024-09-10T19:18:50.858 INFO:teuthology.orchestra.run.smithi183.stdout: "target_version": "0'0" 2024-09-10T19:18:50.858 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:18:50.858 INFO:teuthology.orchestra.run.smithi183.stdout: "last_change": "179", 2024-09-10T19:18:50.858 INFO:teuthology.orchestra.run.smithi183.stdout: "last_force_op_resend": "0", 2024-09-10T19:18:50.858 INFO:teuthology.orchestra.run.smithi183.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-10T19:18:50.858 INFO:teuthology.orchestra.run.smithi183.stdout: "last_force_op_resend_preluminous": "177", 2024-09-10T19:18:50.858 INFO:teuthology.orchestra.run.smithi183.stdout: "auid": 0, 2024-09-10T19:18:50.858 INFO:teuthology.orchestra.run.smithi183.stdout: "snap_mode": "selfmanaged", 2024-09-10T19:18:50.858 INFO:teuthology.orchestra.run.smithi183.stdout: "snap_seq": 0, 2024-09-10T19:18:50.858 INFO:teuthology.orchestra.run.smithi183.stdout: "snap_epoch": 0, 2024-09-10T19:18:50.858 INFO:teuthology.orchestra.run.smithi183.stdout: "pool_snaps": [], 2024-09-10T19:18:50.858 INFO:teuthology.orchestra.run.smithi183.stdout: "removed_snaps": "[]", 2024-09-10T19:18:50.858 INFO:teuthology.orchestra.run.smithi183.stdout: "quota_max_bytes": 0, 2024-09-10T19:18:50.858 INFO:teuthology.orchestra.run.smithi183.stdout: "quota_max_objects": 0, 2024-09-10T19:18:50.859 INFO:teuthology.orchestra.run.smithi183.stdout: "tiers": [], 2024-09-10T19:18:50.859 INFO:teuthology.orchestra.run.smithi183.stdout: "tier_of": -1, 2024-09-10T19:18:50.859 INFO:teuthology.orchestra.run.smithi183.stdout: "read_tier": -1, 2024-09-10T19:18:50.859 INFO:teuthology.orchestra.run.smithi183.stdout: "write_tier": -1, 2024-09-10T19:18:50.859 INFO:teuthology.orchestra.run.smithi183.stdout: "cache_mode": "none", 2024-09-10T19:18:50.859 INFO:teuthology.orchestra.run.smithi183.stdout: "target_max_bytes": 0, 2024-09-10T19:18:50.859 INFO:teuthology.orchestra.run.smithi183.stdout: "target_max_objects": 0, 2024-09-10T19:18:50.859 INFO:teuthology.orchestra.run.smithi183.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-10T19:18:50.859 INFO:teuthology.orchestra.run.smithi183.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-10T19:18:50.859 INFO:teuthology.orchestra.run.smithi183.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-10T19:18:50.859 INFO:teuthology.orchestra.run.smithi183.stdout: "cache_min_flush_age": 0, 2024-09-10T19:18:50.859 INFO:teuthology.orchestra.run.smithi183.stdout: "cache_min_evict_age": 0, 2024-09-10T19:18:50.859 INFO:teuthology.orchestra.run.smithi183.stdout: "erasure_code_profile": "", 2024-09-10T19:18:50.859 INFO:teuthology.orchestra.run.smithi183.stdout: "hit_set_params": { 2024-09-10T19:18:50.860 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "none" 2024-09-10T19:18:50.860 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:18:50.860 INFO:teuthology.orchestra.run.smithi183.stdout: "hit_set_period": 0, 2024-09-10T19:18:50.860 INFO:teuthology.orchestra.run.smithi183.stdout: "hit_set_count": 0, 2024-09-10T19:18:50.860 INFO:teuthology.orchestra.run.smithi183.stdout: "use_gmt_hitset": true, 2024-09-10T19:18:50.860 INFO:teuthology.orchestra.run.smithi183.stdout: "min_read_recency_for_promote": 0, 2024-09-10T19:18:50.860 INFO:teuthology.orchestra.run.smithi183.stdout: "min_write_recency_for_promote": 0, 2024-09-10T19:18:50.860 INFO:teuthology.orchestra.run.smithi183.stdout: "hit_set_grade_decay_rate": 0, 2024-09-10T19:18:50.860 INFO:teuthology.orchestra.run.smithi183.stdout: "hit_set_search_last_n": 0, 2024-09-10T19:18:50.860 INFO:teuthology.orchestra.run.smithi183.stdout: "grade_table": [], 2024-09-10T19:18:50.860 INFO:teuthology.orchestra.run.smithi183.stdout: "stripe_width": 0, 2024-09-10T19:18:50.860 INFO:teuthology.orchestra.run.smithi183.stdout: "expected_num_objects": 0, 2024-09-10T19:18:50.860 INFO:teuthology.orchestra.run.smithi183.stdout: "fast_read": false, 2024-09-10T19:18:50.861 INFO:teuthology.orchestra.run.smithi183.stdout: "options": {}, 2024-09-10T19:18:50.861 INFO:teuthology.orchestra.run.smithi183.stdout: "application_metadata": { 2024-09-10T19:18:50.861 INFO:teuthology.orchestra.run.smithi183.stdout: "rados": {} 2024-09-10T19:18:50.861 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:18:50.861 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:18:50.861 INFO:teuthology.orchestra.run.smithi183.stdout: ], 2024-09-10T19:18:50.861 INFO:teuthology.orchestra.run.smithi183.stdout: "osds": [ 2024-09-10T19:18:50.861 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:18:50.861 INFO:teuthology.orchestra.run.smithi183.stdout: "osd": 0, 2024-09-10T19:18:50.861 INFO:teuthology.orchestra.run.smithi183.stdout: "uuid": "9d323010-394a-49aa-ba9b-244d87faf36e", 2024-09-10T19:18:50.861 INFO:teuthology.orchestra.run.smithi183.stdout: "up": 1, 2024-09-10T19:18:50.861 INFO:teuthology.orchestra.run.smithi183.stdout: "in": 1, 2024-09-10T19:18:50.861 INFO:teuthology.orchestra.run.smithi183.stdout: "weight": 1, 2024-09-10T19:18:50.862 INFO:teuthology.orchestra.run.smithi183.stdout: "primary_affinity": 1, 2024-09-10T19:18:50.862 INFO:teuthology.orchestra.run.smithi183.stdout: "last_clean_begin": 0, 2024-09-10T19:18:50.862 INFO:teuthology.orchestra.run.smithi183.stdout: "last_clean_end": 0, 2024-09-10T19:18:50.862 INFO:teuthology.orchestra.run.smithi183.stdout: "up_from": 8, 2024-09-10T19:18:50.862 INFO:teuthology.orchestra.run.smithi183.stdout: "up_thru": 204, 2024-09-10T19:18:50.862 INFO:teuthology.orchestra.run.smithi183.stdout: "down_at": 0, 2024-09-10T19:18:50.862 INFO:teuthology.orchestra.run.smithi183.stdout: "lost_at": 0, 2024-09-10T19:18:50.862 INFO:teuthology.orchestra.run.smithi183.stdout: "public_addrs": { 2024-09-10T19:18:50.862 INFO:teuthology.orchestra.run.smithi183.stdout: "addrvec": [ 2024-09-10T19:18:50.862 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:18:50.862 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v2", 2024-09-10T19:18:50.862 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.183:6802", 2024-09-10T19:18:50.862 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 2813074952 2024-09-10T19:18:50.863 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:18:50.863 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:18:50.863 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v1", 2024-09-10T19:18:50.863 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.183:6803", 2024-09-10T19:18:50.863 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 2813074952 2024-09-10T19:18:50.863 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:18:50.863 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:18:50.863 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:18:50.863 INFO:teuthology.orchestra.run.smithi183.stdout: "cluster_addrs": { 2024-09-10T19:18:50.863 INFO:teuthology.orchestra.run.smithi183.stdout: "addrvec": [ 2024-09-10T19:18:50.863 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:18:50.863 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v2", 2024-09-10T19:18:50.863 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.183:6804", 2024-09-10T19:18:50.863 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 2813074952 2024-09-10T19:18:50.864 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:18:50.864 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:18:50.864 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v1", 2024-09-10T19:18:50.864 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.183:6805", 2024-09-10T19:18:50.864 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 2813074952 2024-09-10T19:18:50.864 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:18:50.864 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:18:50.864 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:18:50.864 INFO:teuthology.orchestra.run.smithi183.stdout: "heartbeat_back_addrs": { 2024-09-10T19:18:50.864 INFO:teuthology.orchestra.run.smithi183.stdout: "addrvec": [ 2024-09-10T19:18:50.864 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:18:50.864 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v2", 2024-09-10T19:18:50.864 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.183:6808", 2024-09-10T19:18:50.864 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 2813074952 2024-09-10T19:18:50.865 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:18:50.865 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:18:50.865 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v1", 2024-09-10T19:18:50.865 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.183:6809", 2024-09-10T19:18:50.865 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 2813074952 2024-09-10T19:18:50.865 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:18:50.865 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:18:50.865 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:18:50.865 INFO:teuthology.orchestra.run.smithi183.stdout: "heartbeat_front_addrs": { 2024-09-10T19:18:50.865 INFO:teuthology.orchestra.run.smithi183.stdout: "addrvec": [ 2024-09-10T19:18:50.865 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:18:50.865 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v2", 2024-09-10T19:18:50.865 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.183:6806", 2024-09-10T19:18:50.865 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 2813074952 2024-09-10T19:18:50.866 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:18:50.866 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:18:50.866 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v1", 2024-09-10T19:18:50.866 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.183:6807", 2024-09-10T19:18:50.866 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 2813074952 2024-09-10T19:18:50.866 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:18:50.866 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:18:50.866 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:18:50.866 INFO:teuthology.orchestra.run.smithi183.stdout: "public_addr": "172.21.15.183:6803/2813074952", 2024-09-10T19:18:50.866 INFO:teuthology.orchestra.run.smithi183.stdout: "cluster_addr": "172.21.15.183:6805/2813074952", 2024-09-10T19:18:50.866 INFO:teuthology.orchestra.run.smithi183.stdout: "heartbeat_back_addr": "172.21.15.183:6809/2813074952", 2024-09-10T19:18:50.866 INFO:teuthology.orchestra.run.smithi183.stdout: "heartbeat_front_addr": "172.21.15.183:6807/2813074952", 2024-09-10T19:18:50.866 INFO:teuthology.orchestra.run.smithi183.stdout: "state": [ 2024-09-10T19:18:50.866 INFO:teuthology.orchestra.run.smithi183.stdout: "exists", 2024-09-10T19:18:50.867 INFO:teuthology.orchestra.run.smithi183.stdout: "up" 2024-09-10T19:18:50.867 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:18:50.867 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:18:50.867 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:18:50.867 INFO:teuthology.orchestra.run.smithi183.stdout: "osd": 1, 2024-09-10T19:18:50.867 INFO:teuthology.orchestra.run.smithi183.stdout: "uuid": "71a6c5dc-5109-4284-8d20-187d8e7afc9e", 2024-09-10T19:18:50.867 INFO:teuthology.orchestra.run.smithi183.stdout: "up": 1, 2024-09-10T19:18:50.867 INFO:teuthology.orchestra.run.smithi183.stdout: "in": 0, 2024-09-10T19:18:50.867 INFO:teuthology.orchestra.run.smithi183.stdout: "weight": 0, 2024-09-10T19:18:50.867 INFO:teuthology.orchestra.run.smithi183.stdout: "primary_affinity": 1, 2024-09-10T19:18:50.867 INFO:teuthology.orchestra.run.smithi183.stdout: "last_clean_begin": 13, 2024-09-10T19:18:50.867 INFO:teuthology.orchestra.run.smithi183.stdout: "last_clean_end": 98, 2024-09-10T19:18:50.867 INFO:teuthology.orchestra.run.smithi183.stdout: "up_from": 138, 2024-09-10T19:18:50.867 INFO:teuthology.orchestra.run.smithi183.stdout: "up_thru": 86, 2024-09-10T19:18:50.868 INFO:teuthology.orchestra.run.smithi183.stdout: "down_at": 99, 2024-09-10T19:18:50.868 INFO:teuthology.orchestra.run.smithi183.stdout: "lost_at": 0, 2024-09-10T19:18:50.868 INFO:teuthology.orchestra.run.smithi183.stdout: "public_addrs": { 2024-09-10T19:18:50.868 INFO:teuthology.orchestra.run.smithi183.stdout: "addrvec": [ 2024-09-10T19:18:50.868 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:18:50.868 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v2", 2024-09-10T19:18:50.868 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.183:6810", 2024-09-10T19:18:50.868 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 219854798 2024-09-10T19:18:50.868 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:18:50.868 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:18:50.868 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v1", 2024-09-10T19:18:50.868 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.183:6811", 2024-09-10T19:18:50.868 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 219854798 2024-09-10T19:18:50.869 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:18:50.869 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:18:50.869 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:18:50.869 INFO:teuthology.orchestra.run.smithi183.stdout: "cluster_addrs": { 2024-09-10T19:18:50.869 INFO:teuthology.orchestra.run.smithi183.stdout: "addrvec": [ 2024-09-10T19:18:50.869 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:18:50.869 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v2", 2024-09-10T19:18:50.869 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.183:6812", 2024-09-10T19:18:50.869 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 219854798 2024-09-10T19:18:50.869 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:18:50.869 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:18:50.869 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v1", 2024-09-10T19:18:50.869 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.183:6813", 2024-09-10T19:18:50.869 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 219854798 2024-09-10T19:18:50.870 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:18:50.870 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:18:50.870 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:18:50.870 INFO:teuthology.orchestra.run.smithi183.stdout: "heartbeat_back_addrs": { 2024-09-10T19:18:50.870 INFO:teuthology.orchestra.run.smithi183.stdout: "addrvec": [ 2024-09-10T19:18:50.870 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:18:50.870 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v2", 2024-09-10T19:18:50.870 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.183:6816", 2024-09-10T19:18:50.870 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 219854798 2024-09-10T19:18:50.870 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:18:50.870 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:18:50.870 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v1", 2024-09-10T19:18:50.870 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.183:6817", 2024-09-10T19:18:50.870 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 219854798 2024-09-10T19:18:50.871 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:18:50.871 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:18:50.871 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:18:50.871 INFO:teuthology.orchestra.run.smithi183.stdout: "heartbeat_front_addrs": { 2024-09-10T19:18:50.871 INFO:teuthology.orchestra.run.smithi183.stdout: "addrvec": [ 2024-09-10T19:18:50.871 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:18:50.871 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v2", 2024-09-10T19:18:50.871 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.183:6814", 2024-09-10T19:18:50.871 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 219854798 2024-09-10T19:18:50.871 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:18:50.871 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:18:50.871 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v1", 2024-09-10T19:18:50.871 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.183:6815", 2024-09-10T19:18:50.871 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 219854798 2024-09-10T19:18:50.871 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:18:50.872 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:18:50.872 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:18:50.872 INFO:teuthology.orchestra.run.smithi183.stdout: "public_addr": "172.21.15.183:6811/219854798", 2024-09-10T19:18:50.872 INFO:teuthology.orchestra.run.smithi183.stdout: "cluster_addr": "172.21.15.183:6813/219854798", 2024-09-10T19:18:50.872 INFO:teuthology.orchestra.run.smithi183.stdout: "heartbeat_back_addr": "172.21.15.183:6817/219854798", 2024-09-10T19:18:50.872 INFO:teuthology.orchestra.run.smithi183.stdout: "heartbeat_front_addr": "172.21.15.183:6815/219854798", 2024-09-10T19:18:50.872 INFO:teuthology.orchestra.run.smithi183.stdout: "state": [ 2024-09-10T19:18:50.872 INFO:teuthology.orchestra.run.smithi183.stdout: "exists", 2024-09-10T19:18:50.872 INFO:teuthology.orchestra.run.smithi183.stdout: "up" 2024-09-10T19:18:50.872 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:18:50.872 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:18:50.872 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:18:50.872 INFO:teuthology.orchestra.run.smithi183.stdout: "osd": 2, 2024-09-10T19:18:50.872 INFO:teuthology.orchestra.run.smithi183.stdout: "uuid": "fc310d5b-b89b-4655-9989-7e10ba356509", 2024-09-10T19:18:50.873 INFO:teuthology.orchestra.run.smithi183.stdout: "up": 1, 2024-09-10T19:18:50.873 INFO:teuthology.orchestra.run.smithi183.stdout: "in": 0, 2024-09-10T19:18:50.873 INFO:teuthology.orchestra.run.smithi183.stdout: "weight": 0, 2024-09-10T19:18:50.873 INFO:teuthology.orchestra.run.smithi183.stdout: "primary_affinity": 1, 2024-09-10T19:18:50.873 INFO:teuthology.orchestra.run.smithi183.stdout: "last_clean_begin": 0, 2024-09-10T19:18:50.873 INFO:teuthology.orchestra.run.smithi183.stdout: "last_clean_end": 0, 2024-09-10T19:18:50.873 INFO:teuthology.orchestra.run.smithi183.stdout: "up_from": 18, 2024-09-10T19:18:50.873 INFO:teuthology.orchestra.run.smithi183.stdout: "up_thru": 148, 2024-09-10T19:18:50.873 INFO:teuthology.orchestra.run.smithi183.stdout: "down_at": 0, 2024-09-10T19:18:50.873 INFO:teuthology.orchestra.run.smithi183.stdout: "lost_at": 0, 2024-09-10T19:18:50.873 INFO:teuthology.orchestra.run.smithi183.stdout: "public_addrs": { 2024-09-10T19:18:50.873 INFO:teuthology.orchestra.run.smithi183.stdout: "addrvec": [ 2024-09-10T19:18:50.873 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:18:50.873 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v2", 2024-09-10T19:18:50.874 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.183:6818", 2024-09-10T19:18:50.874 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 3730013786 2024-09-10T19:18:50.874 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:18:50.874 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:18:50.874 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v1", 2024-09-10T19:18:50.874 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.183:6819", 2024-09-10T19:18:50.874 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 3730013786 2024-09-10T19:18:50.874 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:18:50.874 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:18:50.874 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:18:50.874 INFO:teuthology.orchestra.run.smithi183.stdout: "cluster_addrs": { 2024-09-10T19:18:50.874 INFO:teuthology.orchestra.run.smithi183.stdout: "addrvec": [ 2024-09-10T19:18:50.874 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:18:50.874 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v2", 2024-09-10T19:18:50.875 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.183:6820", 2024-09-10T19:18:50.875 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 3730013786 2024-09-10T19:18:50.875 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:18:50.875 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:18:50.875 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v1", 2024-09-10T19:18:50.875 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.183:6821", 2024-09-10T19:18:50.875 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 3730013786 2024-09-10T19:18:50.875 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:18:50.875 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:18:50.875 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:18:50.875 INFO:teuthology.orchestra.run.smithi183.stdout: "heartbeat_back_addrs": { 2024-09-10T19:18:50.875 INFO:teuthology.orchestra.run.smithi183.stdout: "addrvec": [ 2024-09-10T19:18:50.875 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:18:50.875 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v2", 2024-09-10T19:18:50.876 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.183:6824", 2024-09-10T19:18:50.876 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 3730013786 2024-09-10T19:18:50.876 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:18:50.876 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:18:50.876 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v1", 2024-09-10T19:18:50.876 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.183:6825", 2024-09-10T19:18:50.876 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 3730013786 2024-09-10T19:18:50.876 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:18:50.876 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:18:50.876 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:18:50.876 INFO:teuthology.orchestra.run.smithi183.stdout: "heartbeat_front_addrs": { 2024-09-10T19:18:50.876 INFO:teuthology.orchestra.run.smithi183.stdout: "addrvec": [ 2024-09-10T19:18:50.876 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:18:50.877 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v2", 2024-09-10T19:18:50.877 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.183:6822", 2024-09-10T19:18:50.877 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 3730013786 2024-09-10T19:18:50.877 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:18:50.877 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:18:50.877 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v1", 2024-09-10T19:18:50.877 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.183:6823", 2024-09-10T19:18:50.877 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 3730013786 2024-09-10T19:18:50.877 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:18:50.877 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:18:50.877 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:18:50.877 INFO:teuthology.orchestra.run.smithi183.stdout: "public_addr": "172.21.15.183:6819/3730013786", 2024-09-10T19:18:50.877 INFO:teuthology.orchestra.run.smithi183.stdout: "cluster_addr": "172.21.15.183:6821/3730013786", 2024-09-10T19:18:50.877 INFO:teuthology.orchestra.run.smithi183.stdout: "heartbeat_back_addr": "172.21.15.183:6825/3730013786", 2024-09-10T19:18:50.878 INFO:teuthology.orchestra.run.smithi183.stdout: "heartbeat_front_addr": "172.21.15.183:6823/3730013786", 2024-09-10T19:18:50.878 INFO:teuthology.orchestra.run.smithi183.stdout: "state": [ 2024-09-10T19:18:50.878 INFO:teuthology.orchestra.run.smithi183.stdout: "exists", 2024-09-10T19:18:50.878 INFO:teuthology.orchestra.run.smithi183.stdout: "up" 2024-09-10T19:18:50.878 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:18:50.878 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:18:50.878 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:18:50.878 INFO:teuthology.orchestra.run.smithi183.stdout: "osd": 3, 2024-09-10T19:18:50.878 INFO:teuthology.orchestra.run.smithi183.stdout: "uuid": "7fd07a95-ad22-43f5-a968-539552283e89", 2024-09-10T19:18:50.878 INFO:teuthology.orchestra.run.smithi183.stdout: "up": 1, 2024-09-10T19:18:50.878 INFO:teuthology.orchestra.run.smithi183.stdout: "in": 1, 2024-09-10T19:18:50.878 INFO:teuthology.orchestra.run.smithi183.stdout: "weight": 1, 2024-09-10T19:18:50.878 INFO:teuthology.orchestra.run.smithi183.stdout: "primary_affinity": 1, 2024-09-10T19:18:50.878 INFO:teuthology.orchestra.run.smithi183.stdout: "last_clean_begin": 0, 2024-09-10T19:18:50.879 INFO:teuthology.orchestra.run.smithi183.stdout: "last_clean_end": 0, 2024-09-10T19:18:50.879 INFO:teuthology.orchestra.run.smithi183.stdout: "up_from": 26, 2024-09-10T19:18:50.879 INFO:teuthology.orchestra.run.smithi183.stdout: "up_thru": 205, 2024-09-10T19:18:50.879 INFO:teuthology.orchestra.run.smithi183.stdout: "down_at": 0, 2024-09-10T19:18:50.879 INFO:teuthology.orchestra.run.smithi183.stdout: "lost_at": 0, 2024-09-10T19:18:50.879 INFO:teuthology.orchestra.run.smithi183.stdout: "public_addrs": { 2024-09-10T19:18:50.879 INFO:teuthology.orchestra.run.smithi183.stdout: "addrvec": [ 2024-09-10T19:18:50.879 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:18:50.879 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v2", 2024-09-10T19:18:50.879 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.183:6826", 2024-09-10T19:18:50.879 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 3522836984 2024-09-10T19:18:50.879 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:18:50.879 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:18:50.879 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v1", 2024-09-10T19:18:50.880 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.183:6827", 2024-09-10T19:18:50.880 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 3522836984 2024-09-10T19:18:50.880 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:18:50.880 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:18:50.880 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:18:50.880 INFO:teuthology.orchestra.run.smithi183.stdout: "cluster_addrs": { 2024-09-10T19:18:50.880 INFO:teuthology.orchestra.run.smithi183.stdout: "addrvec": [ 2024-09-10T19:18:50.880 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:18:50.880 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v2", 2024-09-10T19:18:50.880 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.183:6828", 2024-09-10T19:18:50.880 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 3522836984 2024-09-10T19:18:50.880 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:18:50.880 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:18:50.880 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v1", 2024-09-10T19:18:50.881 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.183:6829", 2024-09-10T19:18:50.881 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 3522836984 2024-09-10T19:18:50.881 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:18:50.881 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:18:50.881 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:18:50.881 INFO:teuthology.orchestra.run.smithi183.stdout: "heartbeat_back_addrs": { 2024-09-10T19:18:50.881 INFO:teuthology.orchestra.run.smithi183.stdout: "addrvec": [ 2024-09-10T19:18:50.881 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:18:50.881 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v2", 2024-09-10T19:18:50.881 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.183:6832", 2024-09-10T19:18:50.881 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 3522836984 2024-09-10T19:18:50.881 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:18:50.881 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:18:50.881 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v1", 2024-09-10T19:18:50.882 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.183:6833", 2024-09-10T19:18:50.882 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 3522836984 2024-09-10T19:18:50.882 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:18:50.882 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:18:50.882 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:18:50.882 INFO:teuthology.orchestra.run.smithi183.stdout: "heartbeat_front_addrs": { 2024-09-10T19:18:50.882 INFO:teuthology.orchestra.run.smithi183.stdout: "addrvec": [ 2024-09-10T19:18:50.882 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:18:50.882 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v2", 2024-09-10T19:18:50.882 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.183:6830", 2024-09-10T19:18:50.882 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 3522836984 2024-09-10T19:18:50.882 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:18:50.882 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:18:50.882 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v1", 2024-09-10T19:18:50.883 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.183:6831", 2024-09-10T19:18:50.883 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 3522836984 2024-09-10T19:18:50.883 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:18:50.883 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:18:50.883 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:18:50.883 INFO:teuthology.orchestra.run.smithi183.stdout: "public_addr": "172.21.15.183:6827/3522836984", 2024-09-10T19:18:50.883 INFO:teuthology.orchestra.run.smithi183.stdout: "cluster_addr": "172.21.15.183:6829/3522836984", 2024-09-10T19:18:50.883 INFO:teuthology.orchestra.run.smithi183.stdout: "heartbeat_back_addr": "172.21.15.183:6833/3522836984", 2024-09-10T19:18:50.883 INFO:teuthology.orchestra.run.smithi183.stdout: "heartbeat_front_addr": "172.21.15.183:6831/3522836984", 2024-09-10T19:18:50.883 INFO:teuthology.orchestra.run.smithi183.stdout: "state": [ 2024-09-10T19:18:50.883 INFO:teuthology.orchestra.run.smithi183.stdout: "exists", 2024-09-10T19:18:50.883 INFO:teuthology.orchestra.run.smithi183.stdout: "up" 2024-09-10T19:18:50.883 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:18:50.884 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:18:50.884 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:18:50.884 INFO:teuthology.orchestra.run.smithi183.stdout: "osd": 4, 2024-09-10T19:18:50.884 INFO:teuthology.orchestra.run.smithi183.stdout: "uuid": "631083fb-e503-44da-9b6b-154e88539ee7", 2024-09-10T19:18:50.884 INFO:teuthology.orchestra.run.smithi183.stdout: "up": 1, 2024-09-10T19:18:50.884 INFO:teuthology.orchestra.run.smithi183.stdout: "in": 1, 2024-09-10T19:18:50.884 INFO:teuthology.orchestra.run.smithi183.stdout: "weight": 1, 2024-09-10T19:18:50.884 INFO:teuthology.orchestra.run.smithi183.stdout: "primary_affinity": 1, 2024-09-10T19:18:50.884 INFO:teuthology.orchestra.run.smithi183.stdout: "last_clean_begin": 0, 2024-09-10T19:18:50.884 INFO:teuthology.orchestra.run.smithi183.stdout: "last_clean_end": 0, 2024-09-10T19:18:50.884 INFO:teuthology.orchestra.run.smithi183.stdout: "up_from": 32, 2024-09-10T19:18:50.884 INFO:teuthology.orchestra.run.smithi183.stdout: "up_thru": 189, 2024-09-10T19:18:50.884 INFO:teuthology.orchestra.run.smithi183.stdout: "down_at": 0, 2024-09-10T19:18:50.885 INFO:teuthology.orchestra.run.smithi183.stdout: "lost_at": 0, 2024-09-10T19:18:50.885 INFO:teuthology.orchestra.run.smithi183.stdout: "public_addrs": { 2024-09-10T19:18:50.885 INFO:teuthology.orchestra.run.smithi183.stdout: "addrvec": [ 2024-09-10T19:18:50.885 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:18:50.885 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v2", 2024-09-10T19:18:50.885 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.191:6800", 2024-09-10T19:18:50.885 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 2674828539 2024-09-10T19:18:50.885 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:18:50.885 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:18:50.885 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v1", 2024-09-10T19:18:50.885 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.191:6801", 2024-09-10T19:18:50.885 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 2674828539 2024-09-10T19:18:50.885 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:18:50.885 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:18:50.885 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:18:50.886 INFO:teuthology.orchestra.run.smithi183.stdout: "cluster_addrs": { 2024-09-10T19:18:50.886 INFO:teuthology.orchestra.run.smithi183.stdout: "addrvec": [ 2024-09-10T19:18:50.886 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:18:50.886 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v2", 2024-09-10T19:18:50.886 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.191:6802", 2024-09-10T19:18:50.886 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 2674828539 2024-09-10T19:18:50.886 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:18:50.886 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:18:50.886 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v1", 2024-09-10T19:18:50.886 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.191:6803", 2024-09-10T19:18:50.886 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 2674828539 2024-09-10T19:18:50.886 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:18:50.886 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:18:50.887 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:18:50.887 INFO:teuthology.orchestra.run.smithi183.stdout: "heartbeat_back_addrs": { 2024-09-10T19:18:50.887 INFO:teuthology.orchestra.run.smithi183.stdout: "addrvec": [ 2024-09-10T19:18:50.887 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:18:50.887 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v2", 2024-09-10T19:18:50.887 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.191:6806", 2024-09-10T19:18:50.887 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 2674828539 2024-09-10T19:18:50.887 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:18:50.887 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:18:50.887 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v1", 2024-09-10T19:18:50.887 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.191:6807", 2024-09-10T19:18:50.887 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 2674828539 2024-09-10T19:18:50.887 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:18:50.887 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:18:50.888 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:18:50.888 INFO:teuthology.orchestra.run.smithi183.stdout: "heartbeat_front_addrs": { 2024-09-10T19:18:50.888 INFO:teuthology.orchestra.run.smithi183.stdout: "addrvec": [ 2024-09-10T19:18:50.888 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:18:50.888 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v2", 2024-09-10T19:18:50.888 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.191:6804", 2024-09-10T19:18:50.888 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 2674828539 2024-09-10T19:18:50.888 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:18:50.888 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:18:50.888 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v1", 2024-09-10T19:18:50.888 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.191:6805", 2024-09-10T19:18:50.888 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 2674828539 2024-09-10T19:18:50.888 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:18:50.888 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:18:50.889 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:18:50.889 INFO:teuthology.orchestra.run.smithi183.stdout: "public_addr": "172.21.15.191:6801/2674828539", 2024-09-10T19:18:50.889 INFO:teuthology.orchestra.run.smithi183.stdout: "cluster_addr": "172.21.15.191:6803/2674828539", 2024-09-10T19:18:50.889 INFO:teuthology.orchestra.run.smithi183.stdout: "heartbeat_back_addr": "172.21.15.191:6807/2674828539", 2024-09-10T19:18:50.889 INFO:teuthology.orchestra.run.smithi183.stdout: "heartbeat_front_addr": "172.21.15.191:6805/2674828539", 2024-09-10T19:18:50.889 INFO:teuthology.orchestra.run.smithi183.stdout: "state": [ 2024-09-10T19:18:50.889 INFO:teuthology.orchestra.run.smithi183.stdout: "exists", 2024-09-10T19:18:50.889 INFO:teuthology.orchestra.run.smithi183.stdout: "up" 2024-09-10T19:18:50.889 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:18:50.889 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:18:50.889 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:18:50.889 INFO:teuthology.orchestra.run.smithi183.stdout: "osd": 5, 2024-09-10T19:18:50.889 INFO:teuthology.orchestra.run.smithi183.stdout: "uuid": "6b963695-3eca-4e6e-9207-05b8010ebff9", 2024-09-10T19:18:50.889 INFO:teuthology.orchestra.run.smithi183.stdout: "up": 1, 2024-09-10T19:18:50.889 INFO:teuthology.orchestra.run.smithi183.stdout: "in": 1, 2024-09-10T19:18:50.890 INFO:teuthology.orchestra.run.smithi183.stdout: "weight": 1, 2024-09-10T19:18:50.890 INFO:teuthology.orchestra.run.smithi183.stdout: "primary_affinity": 1, 2024-09-10T19:18:50.890 INFO:teuthology.orchestra.run.smithi183.stdout: "last_clean_begin": 0, 2024-09-10T19:18:50.890 INFO:teuthology.orchestra.run.smithi183.stdout: "last_clean_end": 0, 2024-09-10T19:18:50.890 INFO:teuthology.orchestra.run.smithi183.stdout: "up_from": 39, 2024-09-10T19:18:50.890 INFO:teuthology.orchestra.run.smithi183.stdout: "up_thru": 198, 2024-09-10T19:18:50.890 INFO:teuthology.orchestra.run.smithi183.stdout: "down_at": 0, 2024-09-10T19:18:50.890 INFO:teuthology.orchestra.run.smithi183.stdout: "lost_at": 0, 2024-09-10T19:18:50.890 INFO:teuthology.orchestra.run.smithi183.stdout: "public_addrs": { 2024-09-10T19:18:50.890 INFO:teuthology.orchestra.run.smithi183.stdout: "addrvec": [ 2024-09-10T19:18:50.890 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:18:50.890 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v2", 2024-09-10T19:18:50.890 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.191:6808", 2024-09-10T19:18:50.890 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 681733378 2024-09-10T19:18:50.891 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:18:50.891 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:18:50.891 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v1", 2024-09-10T19:18:50.891 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.191:6809", 2024-09-10T19:18:50.891 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 681733378 2024-09-10T19:18:50.891 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:18:50.891 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:18:50.891 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:18:50.891 INFO:teuthology.orchestra.run.smithi183.stdout: "cluster_addrs": { 2024-09-10T19:18:50.891 INFO:teuthology.orchestra.run.smithi183.stdout: "addrvec": [ 2024-09-10T19:18:50.891 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:18:50.891 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v2", 2024-09-10T19:18:50.891 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.191:6810", 2024-09-10T19:18:50.891 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 681733378 2024-09-10T19:18:50.892 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:18:50.892 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:18:50.892 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v1", 2024-09-10T19:18:50.892 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.191:6811", 2024-09-10T19:18:50.892 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 681733378 2024-09-10T19:18:50.892 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:18:50.892 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:18:50.892 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:18:50.892 INFO:teuthology.orchestra.run.smithi183.stdout: "heartbeat_back_addrs": { 2024-09-10T19:18:50.892 INFO:teuthology.orchestra.run.smithi183.stdout: "addrvec": [ 2024-09-10T19:18:50.892 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:18:50.892 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v2", 2024-09-10T19:18:50.892 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.191:6814", 2024-09-10T19:18:50.892 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 681733378 2024-09-10T19:18:50.893 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:18:50.893 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:18:50.893 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v1", 2024-09-10T19:18:50.893 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.191:6815", 2024-09-10T19:18:50.893 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 681733378 2024-09-10T19:18:50.893 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:18:50.893 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:18:50.893 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:18:50.893 INFO:teuthology.orchestra.run.smithi183.stdout: "heartbeat_front_addrs": { 2024-09-10T19:18:50.893 INFO:teuthology.orchestra.run.smithi183.stdout: "addrvec": [ 2024-09-10T19:18:50.893 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:18:50.893 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v2", 2024-09-10T19:18:50.893 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.191:6812", 2024-09-10T19:18:50.894 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 681733378 2024-09-10T19:18:50.894 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:18:50.894 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:18:50.894 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v1", 2024-09-10T19:18:50.894 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.191:6813", 2024-09-10T19:18:50.894 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 681733378 2024-09-10T19:18:50.894 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:18:50.894 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:18:50.894 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:18:50.894 INFO:teuthology.orchestra.run.smithi183.stdout: "public_addr": "172.21.15.191:6809/681733378", 2024-09-10T19:18:50.894 INFO:teuthology.orchestra.run.smithi183.stdout: "cluster_addr": "172.21.15.191:6811/681733378", 2024-09-10T19:18:50.894 INFO:teuthology.orchestra.run.smithi183.stdout: "heartbeat_back_addr": "172.21.15.191:6815/681733378", 2024-09-10T19:18:50.894 INFO:teuthology.orchestra.run.smithi183.stdout: "heartbeat_front_addr": "172.21.15.191:6813/681733378", 2024-09-10T19:18:50.894 INFO:teuthology.orchestra.run.smithi183.stdout: "state": [ 2024-09-10T19:18:50.895 INFO:teuthology.orchestra.run.smithi183.stdout: "exists", 2024-09-10T19:18:50.895 INFO:teuthology.orchestra.run.smithi183.stdout: "up" 2024-09-10T19:18:50.895 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:18:50.895 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:18:50.895 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:18:50.895 INFO:teuthology.orchestra.run.smithi183.stdout: "osd": 6, 2024-09-10T19:18:50.895 INFO:teuthology.orchestra.run.smithi183.stdout: "uuid": "2dc9754f-6aef-4e19-8605-2f1591921811", 2024-09-10T19:18:50.895 INFO:teuthology.orchestra.run.smithi183.stdout: "up": 1, 2024-09-10T19:18:50.895 INFO:teuthology.orchestra.run.smithi183.stdout: "in": 1, 2024-09-10T19:18:50.895 INFO:teuthology.orchestra.run.smithi183.stdout: "weight": 1, 2024-09-10T19:18:50.895 INFO:teuthology.orchestra.run.smithi183.stdout: "primary_affinity": 1, 2024-09-10T19:18:50.895 INFO:teuthology.orchestra.run.smithi183.stdout: "last_clean_begin": 0, 2024-09-10T19:18:50.895 INFO:teuthology.orchestra.run.smithi183.stdout: "last_clean_end": 0, 2024-09-10T19:18:50.895 INFO:teuthology.orchestra.run.smithi183.stdout: "up_from": 47, 2024-09-10T19:18:50.895 INFO:teuthology.orchestra.run.smithi183.stdout: "up_thru": 179, 2024-09-10T19:18:50.896 INFO:teuthology.orchestra.run.smithi183.stdout: "down_at": 0, 2024-09-10T19:18:50.896 INFO:teuthology.orchestra.run.smithi183.stdout: "lost_at": 0, 2024-09-10T19:18:50.896 INFO:teuthology.orchestra.run.smithi183.stdout: "public_addrs": { 2024-09-10T19:18:50.896 INFO:teuthology.orchestra.run.smithi183.stdout: "addrvec": [ 2024-09-10T19:18:50.896 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:18:50.896 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v2", 2024-09-10T19:18:50.896 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.191:6816", 2024-09-10T19:18:50.896 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 2825813477 2024-09-10T19:18:50.896 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:18:50.896 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:18:50.896 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v1", 2024-09-10T19:18:50.896 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.191:6817", 2024-09-10T19:18:50.896 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 2825813477 2024-09-10T19:18:50.897 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:18:50.897 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:18:50.897 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:18:50.897 INFO:teuthology.orchestra.run.smithi183.stdout: "cluster_addrs": { 2024-09-10T19:18:50.897 INFO:teuthology.orchestra.run.smithi183.stdout: "addrvec": [ 2024-09-10T19:18:50.897 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:18:50.897 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v2", 2024-09-10T19:18:50.897 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.191:6818", 2024-09-10T19:18:50.897 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 2825813477 2024-09-10T19:18:50.897 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:18:50.897 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:18:50.897 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v1", 2024-09-10T19:18:50.897 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.191:6819", 2024-09-10T19:18:50.897 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 2825813477 2024-09-10T19:18:50.897 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:18:50.898 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:18:50.898 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:18:50.898 INFO:teuthology.orchestra.run.smithi183.stdout: "heartbeat_back_addrs": { 2024-09-10T19:18:50.898 INFO:teuthology.orchestra.run.smithi183.stdout: "addrvec": [ 2024-09-10T19:18:50.898 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:18:50.898 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v2", 2024-09-10T19:18:50.898 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.191:6822", 2024-09-10T19:18:50.898 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 2825813477 2024-09-10T19:18:50.898 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:18:50.898 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:18:50.898 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v1", 2024-09-10T19:18:50.898 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.191:6823", 2024-09-10T19:18:50.898 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 2825813477 2024-09-10T19:18:50.898 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:18:50.899 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:18:50.899 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:18:50.899 INFO:teuthology.orchestra.run.smithi183.stdout: "heartbeat_front_addrs": { 2024-09-10T19:18:50.899 INFO:teuthology.orchestra.run.smithi183.stdout: "addrvec": [ 2024-09-10T19:18:50.899 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:18:50.899 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v2", 2024-09-10T19:18:50.899 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.191:6820", 2024-09-10T19:18:50.899 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 2825813477 2024-09-10T19:18:50.899 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:18:50.899 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:18:50.899 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v1", 2024-09-10T19:18:50.899 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.191:6821", 2024-09-10T19:18:50.899 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 2825813477 2024-09-10T19:18:50.899 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:18:50.900 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:18:50.900 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:18:50.900 INFO:teuthology.orchestra.run.smithi183.stdout: "public_addr": "172.21.15.191:6817/2825813477", 2024-09-10T19:18:50.900 INFO:teuthology.orchestra.run.smithi183.stdout: "cluster_addr": "172.21.15.191:6819/2825813477", 2024-09-10T19:18:50.900 INFO:teuthology.orchestra.run.smithi183.stdout: "heartbeat_back_addr": "172.21.15.191:6823/2825813477", 2024-09-10T19:18:50.900 INFO:teuthology.orchestra.run.smithi183.stdout: "heartbeat_front_addr": "172.21.15.191:6821/2825813477", 2024-09-10T19:18:50.900 INFO:teuthology.orchestra.run.smithi183.stdout: "state": [ 2024-09-10T19:18:50.900 INFO:teuthology.orchestra.run.smithi183.stdout: "exists", 2024-09-10T19:18:50.900 INFO:teuthology.orchestra.run.smithi183.stdout: "up" 2024-09-10T19:18:50.900 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:18:50.900 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:18:50.900 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:18:50.900 INFO:teuthology.orchestra.run.smithi183.stdout: "osd": 7, 2024-09-10T19:18:50.900 INFO:teuthology.orchestra.run.smithi183.stdout: "uuid": "1a9a299e-d5ba-4b9a-85d5-7f6a5bc754e1", 2024-09-10T19:18:50.901 INFO:teuthology.orchestra.run.smithi183.stdout: "up": 1, 2024-09-10T19:18:50.901 INFO:teuthology.orchestra.run.smithi183.stdout: "in": 1, 2024-09-10T19:18:50.901 INFO:teuthology.orchestra.run.smithi183.stdout: "weight": 1, 2024-09-10T19:18:50.901 INFO:teuthology.orchestra.run.smithi183.stdout: "primary_affinity": 1, 2024-09-10T19:18:50.901 INFO:teuthology.orchestra.run.smithi183.stdout: "last_clean_begin": 0, 2024-09-10T19:18:50.901 INFO:teuthology.orchestra.run.smithi183.stdout: "last_clean_end": 0, 2024-09-10T19:18:50.901 INFO:teuthology.orchestra.run.smithi183.stdout: "up_from": 54, 2024-09-10T19:18:50.901 INFO:teuthology.orchestra.run.smithi183.stdout: "up_thru": 206, 2024-09-10T19:18:50.901 INFO:teuthology.orchestra.run.smithi183.stdout: "down_at": 0, 2024-09-10T19:18:50.901 INFO:teuthology.orchestra.run.smithi183.stdout: "lost_at": 0, 2024-09-10T19:18:50.901 INFO:teuthology.orchestra.run.smithi183.stdout: "public_addrs": { 2024-09-10T19:18:50.901 INFO:teuthology.orchestra.run.smithi183.stdout: "addrvec": [ 2024-09-10T19:18:50.901 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:18:50.901 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v2", 2024-09-10T19:18:50.902 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.191:6824", 2024-09-10T19:18:50.902 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 1422610021 2024-09-10T19:18:50.902 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:18:50.902 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:18:50.902 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v1", 2024-09-10T19:18:50.902 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.191:6825", 2024-09-10T19:18:50.902 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 1422610021 2024-09-10T19:18:50.902 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:18:50.902 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:18:50.902 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:18:50.902 INFO:teuthology.orchestra.run.smithi183.stdout: "cluster_addrs": { 2024-09-10T19:18:50.902 INFO:teuthology.orchestra.run.smithi183.stdout: "addrvec": [ 2024-09-10T19:18:50.902 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:18:50.902 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v2", 2024-09-10T19:18:50.903 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.191:6826", 2024-09-10T19:18:50.903 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 1422610021 2024-09-10T19:18:50.903 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:18:50.903 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:18:50.903 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v1", 2024-09-10T19:18:50.903 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.191:6827", 2024-09-10T19:18:50.903 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 1422610021 2024-09-10T19:18:50.903 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:18:50.903 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:18:50.903 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:18:50.903 INFO:teuthology.orchestra.run.smithi183.stdout: "heartbeat_back_addrs": { 2024-09-10T19:18:50.903 INFO:teuthology.orchestra.run.smithi183.stdout: "addrvec": [ 2024-09-10T19:18:50.903 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:18:50.903 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v2", 2024-09-10T19:18:50.904 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.191:6830", 2024-09-10T19:18:50.904 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 1422610021 2024-09-10T19:18:50.904 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:18:50.904 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:18:50.904 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v1", 2024-09-10T19:18:50.904 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.191:6831", 2024-09-10T19:18:50.904 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 1422610021 2024-09-10T19:18:50.904 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:18:50.904 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:18:50.904 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:18:50.904 INFO:teuthology.orchestra.run.smithi183.stdout: "heartbeat_front_addrs": { 2024-09-10T19:18:50.904 INFO:teuthology.orchestra.run.smithi183.stdout: "addrvec": [ 2024-09-10T19:18:50.904 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:18:50.904 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v2", 2024-09-10T19:18:50.905 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.191:6828", 2024-09-10T19:18:50.905 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 1422610021 2024-09-10T19:18:50.905 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:18:50.905 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:18:50.905 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v1", 2024-09-10T19:18:50.905 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.191:6829", 2024-09-10T19:18:50.905 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 1422610021 2024-09-10T19:18:50.905 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:18:50.905 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:18:50.905 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:18:50.905 INFO:teuthology.orchestra.run.smithi183.stdout: "public_addr": "172.21.15.191:6825/1422610021", 2024-09-10T19:18:50.905 INFO:teuthology.orchestra.run.smithi183.stdout: "cluster_addr": "172.21.15.191:6827/1422610021", 2024-09-10T19:18:50.905 INFO:teuthology.orchestra.run.smithi183.stdout: "heartbeat_back_addr": "172.21.15.191:6831/1422610021", 2024-09-10T19:18:50.905 INFO:teuthology.orchestra.run.smithi183.stdout: "heartbeat_front_addr": "172.21.15.191:6829/1422610021", 2024-09-10T19:18:50.906 INFO:teuthology.orchestra.run.smithi183.stdout: "state": [ 2024-09-10T19:18:50.906 INFO:teuthology.orchestra.run.smithi183.stdout: "exists", 2024-09-10T19:18:50.906 INFO:teuthology.orchestra.run.smithi183.stdout: "up" 2024-09-10T19:18:50.906 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:18:50.906 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:18:50.906 INFO:teuthology.orchestra.run.smithi183.stdout: ], 2024-09-10T19:18:50.906 INFO:teuthology.orchestra.run.smithi183.stdout: "osd_xinfo": [ 2024-09-10T19:18:50.906 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:18:50.906 INFO:teuthology.orchestra.run.smithi183.stdout: "osd": 0, 2024-09-10T19:18:50.906 INFO:teuthology.orchestra.run.smithi183.stdout: "down_stamp": "0.000000", 2024-09-10T19:18:50.906 INFO:teuthology.orchestra.run.smithi183.stdout: "laggy_probability": 0, 2024-09-10T19:18:50.906 INFO:teuthology.orchestra.run.smithi183.stdout: "laggy_interval": 0, 2024-09-10T19:18:50.906 INFO:teuthology.orchestra.run.smithi183.stdout: "features": 4540138320759226367, 2024-09-10T19:18:50.906 INFO:teuthology.orchestra.run.smithi183.stdout: "old_weight": 0, 2024-09-10T19:18:50.907 INFO:teuthology.orchestra.run.smithi183.stdout: "last_purged_snaps_scrub": "2024-09-10T19:17:54.492019+0000", 2024-09-10T19:18:50.907 INFO:teuthology.orchestra.run.smithi183.stdout: "dead_epoch": 0 2024-09-10T19:18:50.907 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:18:50.907 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:18:50.907 INFO:teuthology.orchestra.run.smithi183.stdout: "osd": 1, 2024-09-10T19:18:50.907 INFO:teuthology.orchestra.run.smithi183.stdout: "down_stamp": "0.000000", 2024-09-10T19:18:50.907 INFO:teuthology.orchestra.run.smithi183.stdout: "laggy_probability": 0, 2024-09-10T19:18:50.907 INFO:teuthology.orchestra.run.smithi183.stdout: "laggy_interval": 0, 2024-09-10T19:18:50.907 INFO:teuthology.orchestra.run.smithi183.stdout: "features": 4540138320759226367, 2024-09-10T19:18:50.907 INFO:teuthology.orchestra.run.smithi183.stdout: "old_weight": 65536, 2024-09-10T19:18:50.907 INFO:teuthology.orchestra.run.smithi183.stdout: "last_purged_snaps_scrub": "2024-09-10T19:17:52.905918+0000", 2024-09-10T19:18:50.907 INFO:teuthology.orchestra.run.smithi183.stdout: "dead_epoch": 98 2024-09-10T19:18:50.907 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:18:50.907 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:18:50.908 INFO:teuthology.orchestra.run.smithi183.stdout: "osd": 2, 2024-09-10T19:18:50.908 INFO:teuthology.orchestra.run.smithi183.stdout: "down_stamp": "0.000000", 2024-09-10T19:18:50.908 INFO:teuthology.orchestra.run.smithi183.stdout: "laggy_probability": 0, 2024-09-10T19:18:50.908 INFO:teuthology.orchestra.run.smithi183.stdout: "laggy_interval": 0, 2024-09-10T19:18:50.908 INFO:teuthology.orchestra.run.smithi183.stdout: "features": 4540138320759226367, 2024-09-10T19:18:50.908 INFO:teuthology.orchestra.run.smithi183.stdout: "old_weight": 65536, 2024-09-10T19:18:50.908 INFO:teuthology.orchestra.run.smithi183.stdout: "last_purged_snaps_scrub": "2024-09-10T19:18:18.931307+0000", 2024-09-10T19:18:50.908 INFO:teuthology.orchestra.run.smithi183.stdout: "dead_epoch": 0 2024-09-10T19:18:50.908 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:18:50.908 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:18:50.908 INFO:teuthology.orchestra.run.smithi183.stdout: "osd": 3, 2024-09-10T19:18:50.908 INFO:teuthology.orchestra.run.smithi183.stdout: "down_stamp": "0.000000", 2024-09-10T19:18:50.908 INFO:teuthology.orchestra.run.smithi183.stdout: "laggy_probability": 0, 2024-09-10T19:18:50.908 INFO:teuthology.orchestra.run.smithi183.stdout: "laggy_interval": 0, 2024-09-10T19:18:50.909 INFO:teuthology.orchestra.run.smithi183.stdout: "features": 4540138320759226367, 2024-09-10T19:18:50.909 INFO:teuthology.orchestra.run.smithi183.stdout: "old_weight": 0, 2024-09-10T19:18:50.909 INFO:teuthology.orchestra.run.smithi183.stdout: "last_purged_snaps_scrub": "2024-09-10T19:18:34.335087+0000", 2024-09-10T19:18:50.909 INFO:teuthology.orchestra.run.smithi183.stdout: "dead_epoch": 0 2024-09-10T19:18:50.909 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:18:50.909 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:18:50.909 INFO:teuthology.orchestra.run.smithi183.stdout: "osd": 4, 2024-09-10T19:18:50.909 INFO:teuthology.orchestra.run.smithi183.stdout: "down_stamp": "0.000000", 2024-09-10T19:18:50.909 INFO:teuthology.orchestra.run.smithi183.stdout: "laggy_probability": 0, 2024-09-10T19:18:50.909 INFO:teuthology.orchestra.run.smithi183.stdout: "laggy_interval": 0, 2024-09-10T19:18:50.909 INFO:teuthology.orchestra.run.smithi183.stdout: "features": 4540138320759226367, 2024-09-10T19:18:50.909 INFO:teuthology.orchestra.run.smithi183.stdout: "old_weight": 0, 2024-09-10T19:18:50.909 INFO:teuthology.orchestra.run.smithi183.stdout: "last_purged_snaps_scrub": "2024-09-10T19:18:46.652022+0000", 2024-09-10T19:18:50.909 INFO:teuthology.orchestra.run.smithi183.stdout: "dead_epoch": 0 2024-09-10T19:18:50.910 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:18:50.910 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:18:50.910 INFO:teuthology.orchestra.run.smithi183.stdout: "osd": 5, 2024-09-10T19:18:50.910 INFO:teuthology.orchestra.run.smithi183.stdout: "down_stamp": "0.000000", 2024-09-10T19:18:50.910 INFO:teuthology.orchestra.run.smithi183.stdout: "laggy_probability": 0, 2024-09-10T19:18:50.910 INFO:teuthology.orchestra.run.smithi183.stdout: "laggy_interval": 0, 2024-09-10T19:18:50.910 INFO:teuthology.orchestra.run.smithi183.stdout: "features": 4540138320759226367, 2024-09-10T19:18:50.910 INFO:teuthology.orchestra.run.smithi183.stdout: "old_weight": 0, 2024-09-10T19:18:50.910 INFO:teuthology.orchestra.run.smithi183.stdout: "last_purged_snaps_scrub": "2024-09-10T19:18:50.000086+0000", 2024-09-10T19:18:50.910 INFO:teuthology.orchestra.run.smithi183.stdout: "dead_epoch": 0 2024-09-10T19:18:50.910 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:18:50.910 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:18:50.910 INFO:teuthology.orchestra.run.smithi183.stdout: "osd": 6, 2024-09-10T19:18:50.910 INFO:teuthology.orchestra.run.smithi183.stdout: "down_stamp": "0.000000", 2024-09-10T19:18:50.911 INFO:teuthology.orchestra.run.smithi183.stdout: "laggy_probability": 0, 2024-09-10T19:18:50.911 INFO:teuthology.orchestra.run.smithi183.stdout: "laggy_interval": 0, 2024-09-10T19:18:50.911 INFO:teuthology.orchestra.run.smithi183.stdout: "features": 4540138320759226367, 2024-09-10T19:18:50.911 INFO:teuthology.orchestra.run.smithi183.stdout: "old_weight": 0, 2024-09-10T19:18:50.911 INFO:teuthology.orchestra.run.smithi183.stdout: "last_purged_snaps_scrub": "2024-09-10T19:18:32.489412+0000", 2024-09-10T19:18:50.911 INFO:teuthology.orchestra.run.smithi183.stdout: "dead_epoch": 0 2024-09-10T19:18:50.911 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:18:50.911 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:18:50.911 INFO:teuthology.orchestra.run.smithi183.stdout: "osd": 7, 2024-09-10T19:18:50.911 INFO:teuthology.orchestra.run.smithi183.stdout: "down_stamp": "0.000000", 2024-09-10T19:18:50.911 INFO:teuthology.orchestra.run.smithi183.stdout: "laggy_probability": 0, 2024-09-10T19:18:50.911 INFO:teuthology.orchestra.run.smithi183.stdout: "laggy_interval": 0, 2024-09-10T19:18:50.911 INFO:teuthology.orchestra.run.smithi183.stdout: "features": 4540138320759226367, 2024-09-10T19:18:50.911 INFO:teuthology.orchestra.run.smithi183.stdout: "old_weight": 0, 2024-09-10T19:18:50.912 INFO:teuthology.orchestra.run.smithi183.stdout: "last_purged_snaps_scrub": "2024-09-10T19:17:35.086065+0000", 2024-09-10T19:18:50.912 INFO:teuthology.orchestra.run.smithi183.stdout: "dead_epoch": 0 2024-09-10T19:18:50.912 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:18:50.912 INFO:teuthology.orchestra.run.smithi183.stdout: ], 2024-09-10T19:18:50.912 INFO:teuthology.orchestra.run.smithi183.stdout: "pg_upmap": [ 2024-09-10T19:18:50.912 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:18:50.912 INFO:teuthology.orchestra.run.smithi183.stdout: "pgid": "2.13", 2024-09-10T19:18:50.912 INFO:teuthology.orchestra.run.smithi183.stdout: "osds": [ 2024-09-10T19:18:50.912 INFO:teuthology.orchestra.run.smithi183.stdout: 2, 2024-09-10T19:18:50.912 INFO:teuthology.orchestra.run.smithi183.stdout: 0, 2024-09-10T19:18:50.912 INFO:teuthology.orchestra.run.smithi183.stdout: 5 2024-09-10T19:18:50.912 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:18:50.912 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:18:50.912 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:18:50.913 INFO:teuthology.orchestra.run.smithi183.stdout: "pgid": "8.3", 2024-09-10T19:18:50.913 INFO:teuthology.orchestra.run.smithi183.stdout: "osds": [ 2024-09-10T19:18:50.913 INFO:teuthology.orchestra.run.smithi183.stdout: 7, 2024-09-10T19:18:50.913 INFO:teuthology.orchestra.run.smithi183.stdout: 3, 2024-09-10T19:18:50.913 INFO:teuthology.orchestra.run.smithi183.stdout: 5 2024-09-10T19:18:50.913 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:18:50.913 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:18:50.913 INFO:teuthology.orchestra.run.smithi183.stdout: ], 2024-09-10T19:18:50.913 INFO:teuthology.orchestra.run.smithi183.stdout: "pg_upmap_items": [ 2024-09-10T19:18:50.913 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:18:50.913 INFO:teuthology.orchestra.run.smithi183.stdout: "pgid": "4.1", 2024-09-10T19:18:50.913 INFO:teuthology.orchestra.run.smithi183.stdout: "mappings": [ 2024-09-10T19:18:50.913 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:18:50.914 INFO:teuthology.orchestra.run.smithi183.stdout: "from": 6, 2024-09-10T19:18:50.914 INFO:teuthology.orchestra.run.smithi183.stdout: "to": 7 2024-09-10T19:18:50.914 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:18:50.914 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:18:50.914 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:18:50.914 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:18:50.914 INFO:teuthology.orchestra.run.smithi183.stdout: "pgid": "4.11", 2024-09-10T19:18:50.914 INFO:teuthology.orchestra.run.smithi183.stdout: "mappings": [ 2024-09-10T19:18:50.914 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:18:50.914 INFO:teuthology.orchestra.run.smithi183.stdout: "from": 3, 2024-09-10T19:18:50.914 INFO:teuthology.orchestra.run.smithi183.stdout: "to": 5 2024-09-10T19:18:50.914 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:18:50.914 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:18:50.914 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:18:50.914 INFO:teuthology.orchestra.run.smithi183.stdout: ], 2024-09-10T19:18:50.915 INFO:teuthology.orchestra.run.smithi183.stdout: "pg_temp": [ 2024-09-10T19:18:50.915 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:18:50.915 INFO:teuthology.orchestra.run.smithi183.stdout: "pgid": "8.12", 2024-09-10T19:18:50.915 INFO:teuthology.orchestra.run.smithi183.stdout: "osds": [ 2024-09-10T19:18:50.915 INFO:teuthology.orchestra.run.smithi183.stdout: 0, 2024-09-10T19:18:50.915 INFO:teuthology.orchestra.run.smithi183.stdout: 4, 2024-09-10T19:18:50.915 INFO:teuthology.orchestra.run.smithi183.stdout: 5 2024-09-10T19:18:50.915 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:18:50.915 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:18:50.915 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:18:50.915 INFO:teuthology.orchestra.run.smithi183.stdout: "pgid": "8.13", 2024-09-10T19:18:50.915 INFO:teuthology.orchestra.run.smithi183.stdout: "osds": [ 2024-09-10T19:18:50.915 INFO:teuthology.orchestra.run.smithi183.stdout: 3, 2024-09-10T19:18:50.915 INFO:teuthology.orchestra.run.smithi183.stdout: 5, 2024-09-10T19:18:50.916 INFO:teuthology.orchestra.run.smithi183.stdout: 7 2024-09-10T19:18:50.916 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:18:50.916 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:18:50.916 INFO:teuthology.orchestra.run.smithi183.stdout: ], 2024-09-10T19:18:50.916 INFO:teuthology.orchestra.run.smithi183.stdout: "primary_temp": [], 2024-09-10T19:18:50.916 INFO:teuthology.orchestra.run.smithi183.stdout: "blocklist": { 2024-09-10T19:18:50.916 INFO:teuthology.orchestra.run.smithi183.stdout: "172.21.15.183:6801/3311783362": "2024-09-11T19:10:43.576943+0000", 2024-09-10T19:18:50.916 INFO:teuthology.orchestra.run.smithi183.stdout: "172.21.15.183:0/4088553922": "2024-09-11T19:10:43.576943+0000", 2024-09-10T19:18:50.916 INFO:teuthology.orchestra.run.smithi183.stdout: "172.21.15.183:0/3528769414": "2024-09-11T19:05:48.153879+0000", 2024-09-10T19:18:50.916 INFO:teuthology.orchestra.run.smithi183.stdout: "172.21.15.183:0/1766036658": "2024-09-11T19:05:48.153879+0000", 2024-09-10T19:18:50.916 INFO:teuthology.orchestra.run.smithi183.stdout: "172.21.15.183:0/3044170088": "2024-09-11T19:05:25.702354+0000", 2024-09-10T19:18:50.916 INFO:teuthology.orchestra.run.smithi183.stdout: "172.21.15.183:0/3037333778": "2024-09-11T19:05:25.702354+0000", 2024-09-10T19:18:50.916 INFO:teuthology.orchestra.run.smithi183.stdout: "172.21.15.183:6801/3747320570": "2024-09-11T19:05:25.702354+0000", 2024-09-10T19:18:50.916 INFO:teuthology.orchestra.run.smithi183.stdout: "172.21.15.183:0/3982386906": "2024-09-11T19:10:43.576943+0000", 2024-09-10T19:18:50.917 INFO:teuthology.orchestra.run.smithi183.stdout: "172.21.15.183:0/2436014455": "2024-09-11T19:05:25.702354+0000", 2024-09-10T19:18:50.917 INFO:teuthology.orchestra.run.smithi183.stdout: "172.21.15.183:6801/659745765": "2024-09-11T19:05:48.153879+0000", 2024-09-10T19:18:50.917 INFO:teuthology.orchestra.run.smithi183.stdout: "172.21.15.183:6800/3747320570": "2024-09-11T19:05:25.702354+0000", 2024-09-10T19:18:50.917 INFO:teuthology.orchestra.run.smithi183.stdout: "172.21.15.183:0/1652576282": "2024-09-11T19:10:43.576943+0000", 2024-09-10T19:18:50.917 INFO:teuthology.orchestra.run.smithi183.stdout: "172.21.15.183:6800/3311783362": "2024-09-11T19:10:43.576943+0000", 2024-09-10T19:18:50.917 INFO:teuthology.orchestra.run.smithi183.stdout: "172.21.15.183:0/1012899549": "2024-09-11T19:05:48.153879+0000", 2024-09-10T19:18:50.917 INFO:teuthology.orchestra.run.smithi183.stdout: "172.21.15.183:6800/659745765": "2024-09-11T19:05:48.153879+0000", 2024-09-10T19:18:50.917 INFO:teuthology.orchestra.run.smithi183.stdout: "172.21.15.183:0/412044400": "2024-09-11T19:10:43.576943+0000", 2024-09-10T19:18:50.917 INFO:teuthology.orchestra.run.smithi183.stdout: "172.21.15.183:0/3636171401": "2024-09-11T19:10:43.576943+0000" 2024-09-10T19:18:50.917 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:18:50.917 INFO:teuthology.orchestra.run.smithi183.stdout: "range_blocklist": {}, 2024-09-10T19:18:50.917 INFO:teuthology.orchestra.run.smithi183.stdout: "erasure_code_profiles": { 2024-09-10T19:18:50.917 INFO:teuthology.orchestra.run.smithi183.stdout: "default": { 2024-09-10T19:18:50.917 INFO:teuthology.orchestra.run.smithi183.stdout: "crush-failure-domain": "osd", 2024-09-10T19:18:50.918 INFO:teuthology.orchestra.run.smithi183.stdout: "k": "2", 2024-09-10T19:18:50.918 INFO:teuthology.orchestra.run.smithi183.stdout: "m": "1", 2024-09-10T19:18:50.918 INFO:teuthology.orchestra.run.smithi183.stdout: "plugin": "jerasure", 2024-09-10T19:18:50.918 INFO:teuthology.orchestra.run.smithi183.stdout: "technique": "reed_sol_van" 2024-09-10T19:18:50.918 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:18:50.918 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:18:50.918 INFO:teuthology.orchestra.run.smithi183.stdout: "removed_snaps_queue": [], 2024-09-10T19:18:50.918 INFO:teuthology.orchestra.run.smithi183.stdout: "new_removed_snaps": [], 2024-09-10T19:18:50.918 INFO:teuthology.orchestra.run.smithi183.stdout: "new_purged_snaps": [], 2024-09-10T19:18:50.918 INFO:teuthology.orchestra.run.smithi183.stdout: "crush_node_flags": {}, 2024-09-10T19:18:50.918 INFO:teuthology.orchestra.run.smithi183.stdout: "device_class_flags": {}, 2024-09-10T19:18:50.918 INFO:teuthology.orchestra.run.smithi183.stdout: "stretch_mode": { 2024-09-10T19:18:50.918 INFO:teuthology.orchestra.run.smithi183.stdout: "stretch_mode_enabled": false, 2024-09-10T19:18:50.918 INFO:teuthology.orchestra.run.smithi183.stdout: "stretch_bucket_count": 0, 2024-09-10T19:18:50.919 INFO:teuthology.orchestra.run.smithi183.stdout: "degraded_stretch_mode": 0, 2024-09-10T19:18:50.919 INFO:teuthology.orchestra.run.smithi183.stdout: "recovering_stretch_mode": 0, 2024-09-10T19:18:50.919 INFO:teuthology.orchestra.run.smithi183.stdout: "stretch_mode_bucket": 0 2024-09-10T19:18:50.919 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:18:50.919 INFO:teuthology.orchestra.run.smithi183.stdout:} 2024-09-10T19:18:50.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:18:50.920 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:18:50.920 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:18:51.021 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:18:51.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:18:51.056 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:18:51.056 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:18:51.157 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:18:51.186 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:18:51.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:18:51.187 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:18:51.288 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:18:51.316 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:18:51 smithi183 ceph-mon[29736]: from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:18:51.317 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:18:51 smithi183 ceph-mon[29736]: pgmap v381: 158 pgs: 1 peering, 1 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 155 active+clean; 4.1 GiB data, 16 GiB used, 521 GiB / 536 GiB avail; 853 B/s rd, 0 B/s wr, 724 op/s; 5813/202986 objects misplaced (2.864%); 26 MiB/s, 415 objects/s recovering 2024-09-10T19:18:51.317 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:18:51 smithi183 ceph-mon[29736]: purged_snaps scrub starts 2024-09-10T19:18:51.317 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:18:51 smithi183 ceph-mon[29736]: purged_snaps scrub ok 2024-09-10T19:18:51.317 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:18:51 smithi183 ceph-mon[29736]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T19:18:51.317 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:18:51 smithi183 ceph-mon[29736]: from='client.? 172.21.15.183:0/1071007636' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-10T19:18:51.317 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:18:51 smithi183 ceph-mon[29736]: osdmap e209: 8 total, 8 up, 6 in 2024-09-10T19:18:51.317 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:18:51 smithi183 ceph-mon[29736]: from='client.? 172.21.15.183:0/4121207468' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-09-10T19:18:51.317 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:18:51 smithi183 ceph-mon[41562]: from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:18:51.317 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:18:51 smithi183 ceph-mon[41562]: pgmap v381: 158 pgs: 1 peering, 1 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 155 active+clean; 4.1 GiB data, 16 GiB used, 521 GiB / 536 GiB avail; 853 B/s rd, 0 B/s wr, 724 op/s; 5813/202986 objects misplaced (2.864%); 26 MiB/s, 415 objects/s recovering 2024-09-10T19:18:51.317 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:18:51 smithi183 ceph-mon[41562]: purged_snaps scrub starts 2024-09-10T19:18:51.317 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:18:51 smithi183 ceph-mon[41562]: purged_snaps scrub ok 2024-09-10T19:18:51.318 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:18:51 smithi183 ceph-mon[41562]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T19:18:51.318 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:18:51 smithi183 ceph-mon[41562]: from='client.? 172.21.15.183:0/1071007636' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-10T19:18:51.318 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:18:51 smithi183 ceph-mon[41562]: osdmap e209: 8 total, 8 up, 6 in 2024-09-10T19:18:51.318 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:18:51 smithi183 ceph-mon[41562]: from='client.? 172.21.15.183:0/4121207468' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-09-10T19:18:51.318 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:18:51.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:18:51.319 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:18:51.420 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:18:51.461 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:18:51.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:18:51.462 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:18:51.465 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:18:51 smithi191 ceph-mon[32884]: from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:18:51.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:18:51 smithi191 ceph-mon[32884]: pgmap v381: 158 pgs: 1 peering, 1 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 155 active+clean; 4.1 GiB data, 16 GiB used, 521 GiB / 536 GiB avail; 853 B/s rd, 0 B/s wr, 724 op/s; 5813/202986 objects misplaced (2.864%); 26 MiB/s, 415 objects/s recovering 2024-09-10T19:18:51.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:18:51 smithi191 ceph-mon[32884]: purged_snaps scrub starts 2024-09-10T19:18:51.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:18:51 smithi191 ceph-mon[32884]: purged_snaps scrub ok 2024-09-10T19:18:51.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:18:51 smithi191 ceph-mon[32884]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T19:18:51.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:18:51 smithi191 ceph-mon[32884]: from='client.? 172.21.15.183:0/1071007636' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-10T19:18:51.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:18:51 smithi191 ceph-mon[32884]: osdmap e209: 8 total, 8 up, 6 in 2024-09-10T19:18:51.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:18:51 smithi191 ceph-mon[32884]: from='client.? 172.21.15.183:0/4121207468' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-09-10T19:18:51.563 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:18:51.596 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:18:51.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:18:51.597 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:18:51.677 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Sep 10 19:18:51 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-y[29989]: ::ffff:172.21.15.191 - - [10/Sep/2024:19:18:51] "GET /metrics HTTP/1.1" 200 224345 "" "Prometheus/2.43.0" 2024-09-10T19:18:51.698 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:18:51.730 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:18:51.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:18:51.731 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:18:51.832 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:18:51.864 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:18:51.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:18:51.865 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:18:51.966 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:18:51.992 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:18:51.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:18:51.993 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:18:52.094 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:18:52.132 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:18:52.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:18:52.133 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:18:52.234 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:18:52.272 INFO:tasks.thrashosds.thrasher:j is {'epoch': 209, 'fsid': '320f6df8-6fa7-11ef-bcea-c7b262605968', 'created': '2024-09-10T19:05:03.688531+0000', 'modified': '2024-09-10T19:18:50.185351+0000', 'last_up_change': '2024-09-10T19:15:19.923894+0000', 'last_in_change': '2024-09-10T19:16:23.815531+0000', 'flags': 'noscrub,nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5806080, 'flags_set': ['nodeep-scrub', 'noscrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 19, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 8, 'max_osd': 8, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'quincy', 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-10T19:08:00.456786+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '202', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}, {'pool': 2, 'pool_name': '.rgw.root', 'create_time': '2024-09-10T19:10:05.993763+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '60', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}}, {'pool': 3, 'pool_name': 'datapool', 'create_time': '2024-09-10T19:10:06.224267+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 3, 'pg_placement_num': 3, 'pg_placement_num_target': 3, 'pg_num_target': 3, 'pg_num_pending': 3, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '63', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 3, 'snap_epoch': 63, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 4, 'pool_name': 'default.rgw.log', 'create_time': '2024-09-10T19:10:07.958284+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '62', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}}, {'pool': 5, 'pool_name': 'default.rgw.control', 'create_time': '2024-09-10T19:10:09.933596+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '64', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}}, {'pool': 6, 'pool_name': 'default.rgw.meta', 'create_time': '2024-09-10T19:10:11.993712+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '66', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4}, 'application_metadata': {'rgw': {}}}, {'pool': 8, 'pool_name': 'unique_pool_1', 'create_time': '2024-09-10T19:16:11.474472+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 26, 'pg_placement_num': 26, 'pg_placement_num_target': 26, 'pg_num_target': 26, 'pg_num_pending': 26, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '179', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '177', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}}], 'osds': [{'osd': 0, 'uuid': '9d323010-394a-49aa-ba9b-244d87faf36e', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 8, 'up_thru': 204, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.183:6802', 'nonce': 2813074952}, {'type': 'v1', 'addr': '172.21.15.183:6803', 'nonce': 2813074952}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.183:6804', 'nonce': 2813074952}, {'type': 'v1', 'addr': '172.21.15.183:6805', 'nonce': 2813074952}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.183:6808', 'nonce': 2813074952}, {'type': 'v1', 'addr': '172.21.15.183:6809', 'nonce': 2813074952}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.183:6806', 'nonce': 2813074952}, {'type': 'v1', 'addr': '172.21.15.183:6807', 'nonce': 2813074952}]}, 'public_addr': '172.21.15.183:6803/2813074952', 'cluster_addr': '172.21.15.183:6805/2813074952', 'heartbeat_back_addr': '172.21.15.183:6809/2813074952', 'heartbeat_front_addr': '172.21.15.183:6807/2813074952', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': '71a6c5dc-5109-4284-8d20-187d8e7afc9e', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 1, 'last_clean_begin': 13, 'last_clean_end': 98, 'up_from': 138, 'up_thru': 86, 'down_at': 99, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.183:6810', 'nonce': 219854798}, {'type': 'v1', 'addr': '172.21.15.183:6811', 'nonce': 219854798}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.183:6812', 'nonce': 219854798}, {'type': 'v1', 'addr': '172.21.15.183:6813', 'nonce': 219854798}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.183:6816', 'nonce': 219854798}, {'type': 'v1', 'addr': '172.21.15.183:6817', 'nonce': 219854798}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.183:6814', 'nonce': 219854798}, {'type': 'v1', 'addr': '172.21.15.183:6815', 'nonce': 219854798}]}, 'public_addr': '172.21.15.183:6811/219854798', 'cluster_addr': '172.21.15.183:6813/219854798', 'heartbeat_back_addr': '172.21.15.183:6817/219854798', 'heartbeat_front_addr': '172.21.15.183:6815/219854798', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': 'fc310d5b-b89b-4655-9989-7e10ba356509', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 18, 'up_thru': 148, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.183:6818', 'nonce': 3730013786}, {'type': 'v1', 'addr': '172.21.15.183:6819', 'nonce': 3730013786}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.183:6820', 'nonce': 3730013786}, {'type': 'v1', 'addr': '172.21.15.183:6821', 'nonce': 3730013786}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.183:6824', 'nonce': 3730013786}, {'type': 'v1', 'addr': '172.21.15.183:6825', 'nonce': 3730013786}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.183:6822', 'nonce': 3730013786}, {'type': 'v1', 'addr': '172.21.15.183:6823', 'nonce': 3730013786}]}, 'public_addr': '172.21.15.183:6819/3730013786', 'cluster_addr': '172.21.15.183:6821/3730013786', 'heartbeat_back_addr': '172.21.15.183:6825/3730013786', 'heartbeat_front_addr': '172.21.15.183:6823/3730013786', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': '7fd07a95-ad22-43f5-a968-539552283e89', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 26, 'up_thru': 205, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.183:6826', 'nonce': 3522836984}, {'type': 'v1', 'addr': '172.21.15.183:6827', 'nonce': 3522836984}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.183:6828', 'nonce': 3522836984}, {'type': 'v1', 'addr': '172.21.15.183:6829', 'nonce': 3522836984}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.183:6832', 'nonce': 3522836984}, {'type': 'v1', 'addr': '172.21.15.183:6833', 'nonce': 3522836984}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.183:6830', 'nonce': 3522836984}, {'type': 'v1', 'addr': '172.21.15.183:6831', 'nonce': 3522836984}]}, 'public_addr': '172.21.15.183:6827/3522836984', 'cluster_addr': '172.21.15.183:6829/3522836984', 'heartbeat_back_addr': '172.21.15.183:6833/3522836984', 'heartbeat_front_addr': '172.21.15.183:6831/3522836984', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': '631083fb-e503-44da-9b6b-154e88539ee7', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 32, 'up_thru': 189, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.191:6800', 'nonce': 2674828539}, {'type': 'v1', 'addr': '172.21.15.191:6801', 'nonce': 2674828539}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.191:6802', 'nonce': 2674828539}, {'type': 'v1', 'addr': '172.21.15.191:6803', 'nonce': 2674828539}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.191:6806', 'nonce': 2674828539}, {'type': 'v1', 'addr': '172.21.15.191:6807', 'nonce': 2674828539}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.191:6804', 'nonce': 2674828539}, {'type': 'v1', 'addr': '172.21.15.191:6805', 'nonce': 2674828539}]}, 'public_addr': '172.21.15.191:6801/2674828539', 'cluster_addr': '172.21.15.191:6803/2674828539', 'heartbeat_back_addr': '172.21.15.191:6807/2674828539', 'heartbeat_front_addr': '172.21.15.191:6805/2674828539', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': '6b963695-3eca-4e6e-9207-05b8010ebff9', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 39, 'up_thru': 198, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.191:6808', 'nonce': 681733378}, {'type': 'v1', 'addr': '172.21.15.191:6809', 'nonce': 681733378}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.191:6810', 'nonce': 681733378}, {'type': 'v1', 'addr': '172.21.15.191:6811', 'nonce': 681733378}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.191:6814', 'nonce': 681733378}, {'type': 'v1', 'addr': '172.21.15.191:6815', 'nonce': 681733378}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.191:6812', 'nonce': 681733378}, {'type': 'v1', 'addr': '172.21.15.191:6813', 'nonce': 681733378}]}, 'public_addr': '172.21.15.191:6809/681733378', 'cluster_addr': '172.21.15.191:6811/681733378', 'heartbeat_back_addr': '172.21.15.191:6815/681733378', 'heartbeat_front_addr': '172.21.15.191:6813/681733378', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': '2dc9754f-6aef-4e19-8605-2f1591921811', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 47, 'up_thru': 179, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.191:6816', 'nonce': 2825813477}, {'type': 'v1', 'addr': '172.21.15.191:6817', 'nonce': 2825813477}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.191:6818', 'nonce': 2825813477}, {'type': 'v1', 'addr': '172.21.15.191:6819', 'nonce': 2825813477}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.191:6822', 'nonce': 2825813477}, {'type': 'v1', 'addr': '172.21.15.191:6823', 'nonce': 2825813477}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.191:6820', 'nonce': 2825813477}, {'type': 'v1', 'addr': '172.21.15.191:6821', 'nonce': 2825813477}]}, 'public_addr': '172.21.15.191:6817/2825813477', 'cluster_addr': '172.21.15.191:6819/2825813477', 'heartbeat_back_addr': '172.21.15.191:6823/2825813477', 'heartbeat_front_addr': '172.21.15.191:6821/2825813477', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': '1a9a299e-d5ba-4b9a-85d5-7f6a5bc754e1', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 54, 'up_thru': 206, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.191:6824', 'nonce': 1422610021}, {'type': 'v1', 'addr': '172.21.15.191:6825', 'nonce': 1422610021}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.191:6826', 'nonce': 1422610021}, {'type': 'v1', 'addr': '172.21.15.191:6827', 'nonce': 1422610021}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.191:6830', 'nonce': 1422610021}, {'type': 'v1', 'addr': '172.21.15.191:6831', 'nonce': 1422610021}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.191:6828', 'nonce': 1422610021}, {'type': 'v1', 'addr': '172.21.15.191:6829', 'nonce': 1422610021}]}, 'public_addr': '172.21.15.191:6825/1422610021', 'cluster_addr': '172.21.15.191:6827/1422610021', 'heartbeat_back_addr': '172.21.15.191:6831/1422610021', 'heartbeat_front_addr': '172.21.15.191:6829/1422610021', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138320759226367, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-10T19:17:54.492019+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138320759226367, 'old_weight': 65536, 'last_purged_snaps_scrub': '2024-09-10T19:17:52.905918+0000', 'dead_epoch': 98}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138320759226367, 'old_weight': 65536, 'last_purged_snaps_scrub': '2024-09-10T19:18:18.931307+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138320759226367, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-10T19:18:34.335087+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138320759226367, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-10T19:18:46.652022+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138320759226367, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-10T19:18:50.000086+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138320759226367, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-10T19:18:32.489412+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138320759226367, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-10T19:17:35.086065+0000', 'dead_epoch': 0}], 'pg_upmap': [{'pgid': '2.13', 'osds': [2, 0, 5]}, {'pgid': '8.3', 'osds': [7, 3, 5]}], 'pg_upmap_items': [{'pgid': '4.1', 'mappings': [{'from': 6, 'to': 7}]}, {'pgid': '4.11', 'mappings': [{'from': 3, 'to': 5}]}], 'pg_temp': [{'pgid': '8.12', 'osds': [0, 4, 5]}, {'pgid': '8.13', 'osds': [3, 5, 7]}], 'primary_temp': [], 'blocklist': {'172.21.15.183:6801/3311783362': '2024-09-11T19:10:43.576943+0000', '172.21.15.183:0/4088553922': '2024-09-11T19:10:43.576943+0000', '172.21.15.183:0/3528769414': '2024-09-11T19:05:48.153879+0000', '172.21.15.183:0/1766036658': '2024-09-11T19:05:48.153879+0000', '172.21.15.183:0/3044170088': '2024-09-11T19:05:25.702354+0000', '172.21.15.183:0/3037333778': '2024-09-11T19:05:25.702354+0000', '172.21.15.183:6801/3747320570': '2024-09-11T19:05:25.702354+0000', '172.21.15.183:0/3982386906': '2024-09-11T19:10:43.576943+0000', '172.21.15.183:0/2436014455': '2024-09-11T19:05:25.702354+0000', '172.21.15.183:6801/659745765': '2024-09-11T19:05:48.153879+0000', '172.21.15.183:6800/3747320570': '2024-09-11T19:05:25.702354+0000', '172.21.15.183:0/1652576282': '2024-09-11T19:10:43.576943+0000', '172.21.15.183:6800/3311783362': '2024-09-11T19:10:43.576943+0000', '172.21.15.183:0/1012899549': '2024-09-11T19:05:48.153879+0000', '172.21.15.183:6800/659745765': '2024-09-11T19:05:48.153879+0000', '172.21.15.183:0/412044400': '2024-09-11T19:10:43.576943+0000', '172.21.15.183:0/3636171401': '2024-09-11T19:10:43.576943+0000'}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-09-10T19:18:52.272 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph pg dump --format=json 2024-09-10T19:18:52.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:18:52.273 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:18:52.274 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:18:52.374 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:18:52.408 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:18:52.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:18:52.409 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:18:52.510 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:18:52.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:18:52.550 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:18:52.550 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:18:52.651 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:18:52.679 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:18:52.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:18:52.680 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:18:52.781 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:18:52.825 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:18:52.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:18:52.826 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:18:52.927 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:18:52.963 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:18:52.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:18:52.964 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:18:53.065 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:18:53.101 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:18:53.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:18:53.102 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:18:53.203 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:18:53.244 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:18:53.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:18:53.245 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:18:53.313 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:18:53 smithi183 ceph-mon[29736]: pgmap v383: 158 pgs: 1 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 156 active+clean; 3.7 GiB data, 15 GiB used, 522 GiB / 536 GiB avail; 774 B/s rd, 0 B/s wr, 1.02k op/s; 4526/182838 objects misplaced (2.475%); 7.0 MiB/s, 111 objects/s recovering 2024-09-10T19:18:53.313 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:18:53 smithi183 ceph-mon[41562]: pgmap v383: 158 pgs: 1 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 156 active+clean; 3.7 GiB data, 15 GiB used, 522 GiB / 536 GiB avail; 774 B/s rd, 0 B/s wr, 1.02k op/s; 4526/182838 objects misplaced (2.475%); 7.0 MiB/s, 111 objects/s recovering 2024-09-10T19:18:53.346 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:18:53.374 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:18:53.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:18:53.375 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:18:53.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:18:53 smithi191 ceph-mon[32884]: pgmap v383: 158 pgs: 1 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 156 active+clean; 3.7 GiB data, 15 GiB used, 522 GiB / 536 GiB avail; 774 B/s rd, 0 B/s wr, 1.02k op/s; 4526/182838 objects misplaced (2.475%); 7.0 MiB/s, 111 objects/s recovering 2024-09-10T19:18:53.476 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:18:53.504 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:18:53.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:18:53.505 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:18:53.605 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:18:53.633 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:18:53.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:18:53.634 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:18:53.735 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:18:53.761 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:18:53.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:18:53.762 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:18:53.862 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:18:53.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:18:53.899 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:18:53.899 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:18:54.000 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:18:54.038 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:18:54.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:18:54.040 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:18:54.140 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:18:54.176 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:18:54.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:18:54.177 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:18:54.277 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:18:54.305 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:18:54.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:18:54.308 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:18:54.409 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:18:54.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:18:54.437 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:18:54.437 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:18:54.448 INFO:teuthology.orchestra.run.smithi183.stdout:{ 2024-09-10T19:18:54.448 INFO:teuthology.orchestra.run.smithi183.stdout: "ops": [ 2024-09-10T19:18:54.448 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:18:54.448 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.15051.0:186271 8.f 8:f263db73:::benchmark_data_smithi183_114924_object65988:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e209)", 2024-09-10T19:18:54.448 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-10T19:18:54.429163+0000", 2024-09-10T19:18:54.448 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 0.014778311000000001, 2024-09-10T19:18:54.448 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.014798199, 2024-09-10T19:18:54.448 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-10T19:18:54.448 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "waiting for sub ops", 2024-09-10T19:18:54.448 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-10T19:18:54.449 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.15051", 2024-09-10T19:18:54.449 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.183:0/3632028672", 2024-09-10T19:18:54.449 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 186271 2024-09-10T19:18:54.449 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:18:54.449 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-10T19:18:54.449 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:18:54.449 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-10T19:18:54.449 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:18:54.429163+0000", 2024-09-10T19:18:54.449 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:18:54.449 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:18:54.449 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:18:54.450 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-10T19:18:54.450 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:18:54.429163+0000", 2024-09-10T19:18:54.450 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:18:54.450 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:18:54.450 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:18:54.450 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-10T19:18:54.450 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:18:54.429162+0000", 2024-09-10T19:18:54.450 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4294967295.9999995 2024-09-10T19:18:54.450 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:18:54.450 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:18:54.451 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-10T19:18:54.451 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:18:54.429166+0000", 2024-09-10T19:18:54.451 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.3100000000000002e-06 2024-09-10T19:18:54.451 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:18:54.451 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:18:54.451 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-10T19:18:54.451 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:18:54.429176+0000", 2024-09-10T19:18:54.451 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.3379999999999994e-06 2024-09-10T19:18:54.451 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:18:54.451 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:18:54.451 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-10T19:18:54.451 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:18:54.429180+0000", 2024-09-10T19:18:54.451 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.8210000000000001e-06 2024-09-10T19:18:54.452 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:18:54.452 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:18:54.452 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-10T19:18:54.452 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:18:54.433705+0000", 2024-09-10T19:18:54.452 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0045246949999999996 2024-09-10T19:18:54.452 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:18:54.452 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:18:54.452 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-10T19:18:54.452 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:18:54.433797+0000", 2024-09-10T19:18:54.452 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.1348999999999996e-05 2024-09-10T19:18:54.453 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:18:54.453 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:18:54.453 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 4,6", 2024-09-10T19:18:54.453 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:18:54.433881+0000", 2024-09-10T19:18:54.453 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.4800000000000001e-05 2024-09-10T19:18:54.453 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:18:54.453 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:18:54.453 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-10T19:18:54.453 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:18:54.441354+0000", 2024-09-10T19:18:54.453 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0074731249999999997 2024-09-10T19:18:54.453 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:18:54.453 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:18:54.454 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:18:54.454 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:18:54.443694+0000", 2024-09-10T19:18:54.454 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0023391340000000001 2024-09-10T19:18:54.454 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:18:54.454 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:18:54.454 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:18:54.454 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:18:54.454 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:18:54.454 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.15051.0:186274 8.f 8:fbf1b146:::benchmark_data_smithi183_114924_object65991:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e209)", 2024-09-10T19:18:54.454 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-10T19:18:54.430023+0000", 2024-09-10T19:18:54.454 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 0.013917693, 2024-09-10T19:18:54.454 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.014005013, 2024-09-10T19:18:54.455 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-10T19:18:54.455 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "waiting for sub ops", 2024-09-10T19:18:54.455 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-10T19:18:54.455 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.15051", 2024-09-10T19:18:54.455 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.183:0/3632028672", 2024-09-10T19:18:54.455 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 186274 2024-09-10T19:18:54.455 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:18:54.455 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-10T19:18:54.455 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:18:54.455 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-10T19:18:54.455 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:18:54.430023+0000", 2024-09-10T19:18:54.455 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:18:54.456 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:18:54.456 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:18:54.456 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-10T19:18:54.456 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:18:54.430023+0000", 2024-09-10T19:18:54.456 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:18:54.456 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:18:54.456 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:18:54.456 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-10T19:18:54.456 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:18:54.430022+0000", 2024-09-10T19:18:54.456 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4294967295.999999 2024-09-10T19:18:54.456 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:18:54.457 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:18:54.457 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-10T19:18:54.457 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:18:54.430028+0000", 2024-09-10T19:18:54.457 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.1020000000000002e-06 2024-09-10T19:18:54.457 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:18:54.457 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:18:54.457 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-10T19:18:54.457 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:18:54.430042+0000", 2024-09-10T19:18:54.457 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.3379e-05 2024-09-10T19:18:54.457 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:18:54.457 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:18:54.458 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-10T19:18:54.458 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:18:54.430048+0000", 2024-09-10T19:18:54.458 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.3790000000000003e-06 2024-09-10T19:18:54.458 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:18:54.458 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:18:54.458 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-10T19:18:54.458 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:18:54.436280+0000", 2024-09-10T19:18:54.458 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0062316979999999999 2024-09-10T19:18:54.458 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:18:54.458 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:18:54.458 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-10T19:18:54.459 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:18:54.436402+0000", 2024-09-10T19:18:54.459 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00012240600000000001 2024-09-10T19:18:54.459 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:18:54.459 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:18:54.459 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 4,6", 2024-09-10T19:18:54.459 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:18:54.436516+0000", 2024-09-10T19:18:54.459 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000114282 2024-09-10T19:18:54.459 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:18:54.459 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:18:54.459 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-10T19:18:54.459 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:18:54.441362+0000", 2024-09-10T19:18:54.460 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.004845083 2024-09-10T19:18:54.460 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:18:54.460 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:18:54.460 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:18:54.460 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:18:54.443722+0000", 2024-09-10T19:18:54.460 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0023603000000000001 2024-09-10T19:18:54.460 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:18:54.460 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:18:54.460 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:18:54.461 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:18:54.461 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:18:54.461 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.15051.0:186277 8.f 8:f9c77795:::benchmark_data_smithi183_114924_object65994:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e209)", 2024-09-10T19:18:54.461 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-10T19:18:54.436218+0000", 2024-09-10T19:18:54.461 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 0.0077225870000000004, 2024-09-10T19:18:54.461 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0078669510000000005, 2024-09-10T19:18:54.461 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-10T19:18:54.461 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "waiting for sub ops", 2024-09-10T19:18:54.461 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-10T19:18:54.462 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.15051", 2024-09-10T19:18:54.462 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.183:0/3632028672", 2024-09-10T19:18:54.462 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 186277 2024-09-10T19:18:54.462 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:18:54.462 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-10T19:18:54.462 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:18:54.462 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-10T19:18:54.462 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:18:54.436218+0000", 2024-09-10T19:18:54.462 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:18:54.462 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:18:54.462 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:18:54.462 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-10T19:18:54.462 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:18:54.436218+0000", 2024-09-10T19:18:54.463 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:18:54.463 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:18:54.463 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:18:54.463 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-10T19:18:54.463 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:18:54.436217+0000", 2024-09-10T19:18:54.463 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4294967295.9999986 2024-09-10T19:18:54.463 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:18:54.463 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:18:54.463 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-10T19:18:54.463 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:18:54.436224+0000", 2024-09-10T19:18:54.463 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.1069999999999999e-06 2024-09-10T19:18:54.463 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:18:54.463 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:18:54.463 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-10T19:18:54.464 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:18:54.436234+0000", 2024-09-10T19:18:54.464 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.0203e-05 2024-09-10T19:18:54.464 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:18:54.464 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:18:54.464 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-10T19:18:54.464 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:18:54.436240+0000", 2024-09-10T19:18:54.464 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.5350000000000004e-06 2024-09-10T19:18:54.464 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:18:54.464 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:18:54.464 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-10T19:18:54.464 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:18:54.438506+0000", 2024-09-10T19:18:54.464 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0022657799999999998 2024-09-10T19:18:54.464 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:18:54.465 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:18:54.465 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-10T19:18:54.465 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:18:54.438615+0000", 2024-09-10T19:18:54.465 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000109825 2024-09-10T19:18:54.465 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:18:54.465 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:18:54.465 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 4,6", 2024-09-10T19:18:54.465 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:18:54.438727+0000", 2024-09-10T19:18:54.465 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00011110799999999999 2024-09-10T19:18:54.465 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:18:54.465 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:18:54.465 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-10T19:18:54.466 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:18:54.442256+0000", 2024-09-10T19:18:54.466 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0035299149999999998 2024-09-10T19:18:54.466 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:18:54.466 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:18:54.466 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:18:54.466 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:18:54.466 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:18:54.466 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.15051.0:186283 8.13 8:cf4d7c9c:::benchmark_data_smithi183_114924_object66000:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e209)", 2024-09-10T19:18:54.466 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-10T19:18:54.439370+0000", 2024-09-10T19:18:54.466 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 0.0045714249999999996, 2024-09-10T19:18:54.466 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0047583210000000003, 2024-09-10T19:18:54.466 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-10T19:18:54.466 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "waiting for sub ops", 2024-09-10T19:18:54.467 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-10T19:18:54.467 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.15051", 2024-09-10T19:18:54.467 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.183:0/3632028672", 2024-09-10T19:18:54.467 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 186283 2024-09-10T19:18:54.467 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:18:54.467 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-10T19:18:54.467 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:18:54.467 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-10T19:18:54.467 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:18:54.439370+0000", 2024-09-10T19:18:54.467 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:18:54.467 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:18:54.467 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:18:54.467 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-10T19:18:54.468 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:18:54.439370+0000", 2024-09-10T19:18:54.468 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:18:54.468 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:18:54.468 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:18:54.468 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-10T19:18:54.468 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:18:54.439368+0000", 2024-09-10T19:18:54.468 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4294967295.9999986 2024-09-10T19:18:54.468 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:18:54.468 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:18:54.468 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-10T19:18:54.469 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:18:54.439375+0000", 2024-09-10T19:18:54.469 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.6440000000000003e-06 2024-09-10T19:18:54.469 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:18:54.469 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:18:54.469 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-10T19:18:54.469 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:18:54.439390+0000", 2024-09-10T19:18:54.469 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.5327000000000001e-05 2024-09-10T19:18:54.469 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:18:54.469 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:18:54.469 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-10T19:18:54.469 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:18:54.439396+0000", 2024-09-10T19:18:54.469 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.1800000000000001e-06 2024-09-10T19:18:54.470 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:18:54.470 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:18:54.470 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-10T19:18:54.470 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:18:54.439462+0000", 2024-09-10T19:18:54.470 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.5333999999999995e-05 2024-09-10T19:18:54.470 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:18:54.470 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:18:54.470 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-10T19:18:54.470 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:18:54.439570+0000", 2024-09-10T19:18:54.470 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00010819099999999999 2024-09-10T19:18:54.470 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:18:54.470 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:18:54.470 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 4,5,7", 2024-09-10T19:18:54.470 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:18:54.439936+0000", 2024-09-10T19:18:54.471 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00036591100000000001 2024-09-10T19:18:54.471 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:18:54.471 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:18:54.471 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:18:54.471 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:18:54.471 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:18:54.471 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_repop(client.15051.0:186281 8.c e209/148 8:3ea88c06:::benchmark_data_smithi183_114924_object65998:head v 209'11718, mlcod=209'11716)", 2024-09-10T19:18:54.471 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-10T19:18:54.440813+0000", 2024-09-10T19:18:54.471 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 0.0031275299999999999, 2024-09-10T19:18:54.471 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0033526150000000002, 2024-09-10T19:18:54.471 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-10T19:18:54.471 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "started", 2024-09-10T19:18:54.471 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-10T19:18:54.472 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:18:54.472 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-10T19:18:54.472 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:18:54.440813+0000", 2024-09-10T19:18:54.472 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:18:54.472 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:18:54.472 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:18:54.472 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-10T19:18:54.472 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:18:54.440813+0000", 2024-09-10T19:18:54.472 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:18:54.472 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:18:54.472 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:18:54.472 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-10T19:18:54.472 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:18:54.440813+0000", 2024-09-10T19:18:54.472 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4294967295.9999995 2024-09-10T19:18:54.473 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:18:54.473 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:18:54.473 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-10T19:18:54.473 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:18:54.440817+0000", 2024-09-10T19:18:54.473 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.4149999999999999e-06 2024-09-10T19:18:54.473 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:18:54.473 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:18:54.473 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-10T19:18:54.473 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:18:54.440823+0000", 2024-09-10T19:18:54.473 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.0530000000000003e-06 2024-09-10T19:18:54.473 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:18:54.473 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:18:54.473 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-10T19:18:54.474 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:18:54.440826+0000", 2024-09-10T19:18:54.474 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.2289999999999999e-06 2024-09-10T19:18:54.474 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:18:54.474 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:18:54.474 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-10T19:18:54.474 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:18:54.440862+0000", 2024-09-10T19:18:54.474 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.6261999999999999e-05 2024-09-10T19:18:54.474 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:18:54.474 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:18:54.474 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-10T19:18:54.474 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:18:54.440880+0000", 2024-09-10T19:18:54.474 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.8215000000000001e-05 2024-09-10T19:18:54.475 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:18:54.475 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:18:54.475 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:18:54.475 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:18:54.475 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:18:54.475 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_repop(client.15051.0:186286 8.c e209/148 8:3471f0c5:::benchmark_data_smithi183_114924_object66003:head v 209'11719, mlcod=209'11716)", 2024-09-10T19:18:54.475 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-10T19:18:54.442593+0000", 2024-09-10T19:18:54.475 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 0.0013481249999999999, 2024-09-10T19:18:54.475 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0016030269999999999, 2024-09-10T19:18:54.475 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-10T19:18:54.475 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "started", 2024-09-10T19:18:54.475 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-10T19:18:54.475 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:18:54.476 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-10T19:18:54.476 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:18:54.442593+0000", 2024-09-10T19:18:54.476 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:18:54.476 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:18:54.476 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:18:54.476 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-10T19:18:54.476 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:18:54.442593+0000", 2024-09-10T19:18:54.477 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:18:54.477 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:18:54.477 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:18:54.477 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-10T19:18:54.477 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:18:54.442592+0000", 2024-09-10T19:18:54.477 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4294967295.999999 2024-09-10T19:18:54.478 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:18:54.478 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:18:54.478 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-10T19:18:54.478 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:18:54.442599+0000", 2024-09-10T19:18:54.478 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.0110000000000003e-06 2024-09-10T19:18:54.478 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:18:54.479 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:18:54.479 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-10T19:18:54.479 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:18:54.442610+0000", 2024-09-10T19:18:54.479 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.1358999999999999e-05 2024-09-10T19:18:54.479 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:18:54.479 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:18:54.479 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-10T19:18:54.479 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:18:54.442616+0000", 2024-09-10T19:18:54.480 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.4190000000000001e-06 2024-09-10T19:18:54.480 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:18:54.480 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:18:54.480 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-10T19:18:54.480 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:18:54.442809+0000", 2024-09-10T19:18:54.480 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00019292499999999999 2024-09-10T19:18:54.480 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:18:54.480 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:18:54.481 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-10T19:18:54.481 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:18:54.442825+0000", 2024-09-10T19:18:54.481 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.5795e-05 2024-09-10T19:18:54.481 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:18:54.481 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:18:54.481 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:18:54.481 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:18:54.482 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:18:54.482 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_repop(client.15051.0:186284 8.3 e209/178 8:c7fb0225:::benchmark_data_smithi183_114924_object66001:head v 209'9038, mlcod=209'9036)", 2024-09-10T19:18:54.482 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-10T19:18:54.442801+0000", 2024-09-10T19:18:54.482 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 0.0011397009999999999, 2024-09-10T19:18:54.482 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0014240920000000001, 2024-09-10T19:18:54.482 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-10T19:18:54.483 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "started", 2024-09-10T19:18:54.483 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-10T19:18:54.483 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:18:54.483 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-10T19:18:54.483 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:18:54.442801+0000", 2024-09-10T19:18:54.483 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:18:54.484 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:18:54.484 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:18:54.484 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-10T19:18:54.484 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:18:54.442801+0000", 2024-09-10T19:18:54.484 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:18:54.484 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:18:54.485 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:18:54.485 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-10T19:18:54.485 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:18:54.442801+0000", 2024-09-10T19:18:54.485 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4294967295.9999995 2024-09-10T19:18:54.485 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:18:54.485 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:18:54.485 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-10T19:18:54.485 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:18:54.442807+0000", 2024-09-10T19:18:54.485 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.63e-06 2024-09-10T19:18:54.485 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:18:54.485 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:18:54.486 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-10T19:18:54.486 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:18:54.442818+0000", 2024-09-10T19:18:54.486 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.0358e-05 2024-09-10T19:18:54.486 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:18:54.486 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:18:54.486 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-10T19:18:54.486 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:18:54.442831+0000", 2024-09-10T19:18:54.486 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.3152999999999999e-05 2024-09-10T19:18:54.486 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:18:54.486 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:18:54.486 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-10T19:18:54.486 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:18:54.443104+0000", 2024-09-10T19:18:54.486 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000272947 2024-09-10T19:18:54.487 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:18:54.487 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:18:54.487 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-10T19:18:54.487 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:18:54.443119+0000", 2024-09-10T19:18:54.487 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.5406000000000001e-05 2024-09-10T19:18:54.487 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:18:54.487 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:18:54.487 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:18:54.487 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:18:54.487 INFO:teuthology.orchestra.run.smithi183.stdout: ], 2024-09-10T19:18:54.487 INFO:teuthology.orchestra.run.smithi183.stdout: "num_ops": 7 2024-09-10T19:18:54.487 INFO:teuthology.orchestra.run.smithi183.stdout:} 2024-09-10T19:18:54.538 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:18:54.578 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:18:54.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:18:54.579 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:18:54.680 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:18:54.707 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:18:54.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:18:54.708 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:18:54.809 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:18:54.850 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:18:54.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:18:54.851 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:18:54.951 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:18:54.989 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:18:54.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:18:54.990 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:18:55.090 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:18:55.119 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:18:55.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:18:55.121 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:18:55.225 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:18:55.261 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:18:55.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:18:55.263 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:18:55.363 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:18:55.391 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:18:55.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:18:55.393 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:18:55.465 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:18:55 smithi191 ceph-mon[32884]: pgmap v384: 158 pgs: 1 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 156 active+clean; 3.7 GiB data, 15 GiB used, 522 GiB / 536 GiB avail; 639 B/s rd, 0 B/s wr, 839 op/s; 4526/182838 objects misplaced (2.475%); 5.8 MiB/s, 92 objects/s recovering 2024-09-10T19:18:55.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:18:55 smithi191 ceph-mon[32884]: osdmap e210: 8 total, 8 up, 6 in 2024-09-10T19:18:55.493 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:18:55.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:18:55.532 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:18:55.533 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:18:55.550 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:18:55 smithi183 ceph-mon[29736]: pgmap v384: 158 pgs: 1 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 156 active+clean; 3.7 GiB data, 15 GiB used, 522 GiB / 536 GiB avail; 639 B/s rd, 0 B/s wr, 839 op/s; 4526/182838 objects misplaced (2.475%); 5.8 MiB/s, 92 objects/s recovering 2024-09-10T19:18:55.550 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:18:55 smithi183 ceph-mon[29736]: osdmap e210: 8 total, 8 up, 6 in 2024-09-10T19:18:55.551 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:18:55 smithi183 ceph-mon[41562]: pgmap v384: 158 pgs: 1 active+remapped+backfilling, 1 active+remapped+backfill_toofull, 156 active+clean; 3.7 GiB data, 15 GiB used, 522 GiB / 536 GiB avail; 639 B/s rd, 0 B/s wr, 839 op/s; 4526/182838 objects misplaced (2.475%); 5.8 MiB/s, 92 objects/s recovering 2024-09-10T19:18:55.551 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:18:55 smithi183 ceph-mon[41562]: osdmap e210: 8 total, 8 up, 6 in 2024-09-10T19:18:55.633 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:18:55.668 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:18:55.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:18:55.669 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:18:55.675 INFO:teuthology.orchestra.run.smithi183.stdout:osd.0: {} 2024-09-10T19:18:55.676 INFO:teuthology.orchestra.run.smithi183.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-10T19:18:55.701 INFO:teuthology.orchestra.run.smithi183.stdout:osd.1: {} 2024-09-10T19:18:55.702 INFO:teuthology.orchestra.run.smithi183.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-10T19:18:55.722 INFO:teuthology.orchestra.run.smithi183.stdout:osd.2: {} 2024-09-10T19:18:55.722 INFO:teuthology.orchestra.run.smithi183.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-10T19:18:55.740 INFO:teuthology.orchestra.run.smithi183.stdout:osd.3: {} 2024-09-10T19:18:55.740 INFO:teuthology.orchestra.run.smithi183.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-10T19:18:55.768 INFO:teuthology.orchestra.run.smithi183.stdout:osd.4: {} 2024-09-10T19:18:55.768 INFO:teuthology.orchestra.run.smithi183.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-10T19:18:55.769 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:18:55.802 INFO:teuthology.orchestra.run.smithi183.stdout:osd.5: {} 2024-09-10T19:18:55.802 INFO:teuthology.orchestra.run.smithi183.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-10T19:18:55.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:18:55.805 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:18:55.805 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:18:55.833 INFO:teuthology.orchestra.run.smithi183.stdout:osd.6: {} 2024-09-10T19:18:55.833 INFO:teuthology.orchestra.run.smithi183.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-10T19:18:55.861 INFO:teuthology.orchestra.run.smithi183.stdout:osd.7: {} 2024-09-10T19:18:55.861 INFO:teuthology.orchestra.run.smithi183.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-10T19:18:55.905 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:18:55.908 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:18:55.938 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:18:55.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:18:55.939 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:18:56.040 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:18:56.063 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:18:56.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:18:56.064 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:18:56.165 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:18:56.188 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:18:56.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:18:56.189 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:18:56.290 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:18:56.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:18:56.313 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:18:56.313 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:18:56.414 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:18:56.437 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:18:56.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:18:56.438 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:18:56.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:18:56 smithi191 ceph-mon[32884]: from='client.? 172.21.15.183:0/1106056676' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T19:18:56.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:18:56 smithi191 ceph-mon[32884]: osdmap e211: 8 total, 8 up, 6 in 2024-09-10T19:18:56.540 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:18:56.564 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:18:56.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:18:56.565 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:18:56.666 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:18:56.681 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:18:56 smithi183 ceph-mon[29736]: from='client.? 172.21.15.183:0/1106056676' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T19:18:56.681 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:18:56 smithi183 ceph-mon[29736]: osdmap e211: 8 total, 8 up, 6 in 2024-09-10T19:18:56.681 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:18:56 smithi183 ceph-mon[41562]: from='client.? 172.21.15.183:0/1106056676' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T19:18:56.681 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:18:56 smithi183 ceph-mon[41562]: osdmap e211: 8 total, 8 up, 6 in 2024-09-10T19:18:56.692 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:18:56.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:18:56.693 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:18:56.793 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:18:56.819 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:18:56.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:18:56.820 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:18:56.920 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:18:56.957 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:18:56.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:18:56.958 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:18:57.059 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:18:57.080 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:18:57.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:18:57.081 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:18:57.182 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:18:57.209 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:18:57.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:18:57.210 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:18:57.311 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:18:57.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:18:57.346 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:18:57.346 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:18:57.431 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph osd unset noscrub 2024-09-10T19:18:57.447 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:18:57.460 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:18:57 smithi183 ceph-mon[29736]: pgmap v386: 158 pgs: 1 active+clean+remapped, 1 active+remapped+backfill_toofull, 156 active+clean; 3.3 GiB data, 13 GiB used, 523 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.74k op/s; 3440/161355 objects misplaced (2.132%); 11 MiB/s, 172 objects/s recovering 2024-09-10T19:18:57.461 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:18:57 smithi183 ceph-mon[29736]: osdmap e212: 8 total, 8 up, 6 in 2024-09-10T19:18:57.461 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:18:57 smithi183 ceph-mon[41562]: pgmap v386: 158 pgs: 1 active+clean+remapped, 1 active+remapped+backfill_toofull, 156 active+clean; 3.3 GiB data, 13 GiB used, 523 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.74k op/s; 3440/161355 objects misplaced (2.132%); 11 MiB/s, 172 objects/s recovering 2024-09-10T19:18:57.461 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:18:57 smithi183 ceph-mon[41562]: osdmap e212: 8 total, 8 up, 6 in 2024-09-10T19:18:57.465 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:18:57 smithi191 ceph-mon[32884]: pgmap v386: 158 pgs: 1 active+clean+remapped, 1 active+remapped+backfill_toofull, 156 active+clean; 3.3 GiB data, 13 GiB used, 523 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.74k op/s; 3440/161355 objects misplaced (2.132%); 11 MiB/s, 172 objects/s recovering 2024-09-10T19:18:57.465 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:18:57 smithi191 ceph-mon[32884]: osdmap e212: 8 total, 8 up, 6 in 2024-09-10T19:18:57.466 INFO:journalctl@ceph.mgr.x.smithi191.stdout:Sep 10 19:18:57 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-x[34967]: ::ffff:172.21.15.191 - - [10/Sep/2024:19:18:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T19:18:57.491 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:18:57.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:18:57.492 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:18:57.593 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:18:57.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:18:57.621 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:18:57.621 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:18:57.721 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:18:57.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:18:57.762 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:18:57.762 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:18:57.863 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:18:57.887 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:18:57.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:18:57.888 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:18:57.989 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:18:58.016 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:18:58.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:18:58.017 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:18:58.118 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:18:58.153 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:18:58.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:18:58.154 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:18:58.255 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:18:58.290 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:18:58.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:18:58.291 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:18:58.392 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:18:58.413 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:18:58.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:18:58.414 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:18:58.515 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:18:58.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:18:58.552 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:18:58.552 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:18:58.653 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:18:58.682 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:18:58.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:18:58.683 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:18:58.784 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:18:58.812 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:18:58.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:18:58.813 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:18:58.914 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:18:58.936 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:18:58.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:18:58.937 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:18:59.038 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:18:59.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:18:59.065 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:18:59.065 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:18:59.166 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:18:59.194 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:18:59.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:18:59.195 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:18:59.296 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:18:59.333 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:18:59.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:18:59.334 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:18:59.435 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:18:59.464 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:18:59.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:18:59.465 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:18:59.465 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:18:59 smithi191 ceph-mon[32884]: pgmap v389: 158 pgs: 1 active+clean+remapped, 1 active+remapped+backfill_toofull, 156 active+clean; 3.3 GiB data, 13 GiB used, 523 GiB / 536 GiB avail; 853 B/s rd, 0 B/s wr, 1.19k op/s; 3440/161355 objects misplaced (2.132%); 6.7 MiB/s, 106 objects/s recovering 2024-09-10T19:18:59.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:18:59 smithi191 ceph-mon[32884]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:18:59.466 INFO:journalctl@ceph.iscsi.iscsi.a.smithi191.stdout:Sep 10 19:18:59 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-iscsi-iscsi-a[61816]: debug there is no tcmu-runner data available 2024-09-10T19:18:59.566 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:18:59.592 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:18:59.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:18:59.593 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:18:59.681 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:18:59 smithi183 ceph-mon[29736]: pgmap v389: 158 pgs: 1 active+clean+remapped, 1 active+remapped+backfill_toofull, 156 active+clean; 3.3 GiB data, 13 GiB used, 523 GiB / 536 GiB avail; 853 B/s rd, 0 B/s wr, 1.19k op/s; 3440/161355 objects misplaced (2.132%); 6.7 MiB/s, 106 objects/s recovering 2024-09-10T19:18:59.681 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:18:59 smithi183 ceph-mon[29736]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:18:59.681 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:18:59 smithi183 ceph-mon[41562]: pgmap v389: 158 pgs: 1 active+clean+remapped, 1 active+remapped+backfill_toofull, 156 active+clean; 3.3 GiB data, 13 GiB used, 523 GiB / 536 GiB avail; 853 B/s rd, 0 B/s wr, 1.19k op/s; 3440/161355 objects misplaced (2.132%); 6.7 MiB/s, 106 objects/s recovering 2024-09-10T19:18:59.681 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:18:59 smithi183 ceph-mon[41562]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:18:59.694 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:18:59.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:18:59.731 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:18:59.731 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:18:59.832 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:18:59.874 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:18:59.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:18:59.875 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:18:59.976 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:19:00.007 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph daemon osd.3 dump_blocked_ops 2024-09-10T19:19:00.015 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:19:00.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:00.016 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:19:00.117 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:19:00.147 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:19:00.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:00.148 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:19:00.227 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:19:00 smithi183 ceph-mon[29736]: purged_snaps scrub starts 2024-09-10T19:19:00.227 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:19:00 smithi183 ceph-mon[29736]: purged_snaps scrub ok 2024-09-10T19:19:00.227 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:19:00 smithi183 ceph-mon[29736]: osdmap e213: 8 total, 8 up, 6 in 2024-09-10T19:19:00.227 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:19:00 smithi183 ceph-mon[29736]: osdmap e214: 8 total, 8 up, 6 in 2024-09-10T19:19:00.249 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:19:00.283 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:19:00.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:00.284 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:19:00.385 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:19:00.421 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:19:00.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:00.422 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:19:00.464 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:19:00.495 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:19:00.523 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:19:00.560 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:19:00.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:00.561 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:19:00.581 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:19:00 smithi183 ceph-mon[41562]: purged_snaps scrub starts 2024-09-10T19:19:00.581 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:19:00 smithi183 ceph-mon[41562]: purged_snaps scrub ok 2024-09-10T19:19:00.581 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:19:00 smithi183 ceph-mon[41562]: osdmap e213: 8 total, 8 up, 6 in 2024-09-10T19:19:00.581 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:19:00 smithi183 ceph-mon[41562]: osdmap e214: 8 total, 8 up, 6 in 2024-09-10T19:19:00.662 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:19:00.700 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:19:00.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:00.701 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:19:00.715 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:19:00 smithi191 ceph-mon[32884]: purged_snaps scrub starts 2024-09-10T19:19:00.715 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:19:00 smithi191 ceph-mon[32884]: purged_snaps scrub ok 2024-09-10T19:19:00.715 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:19:00 smithi191 ceph-mon[32884]: osdmap e213: 8 total, 8 up, 6 in 2024-09-10T19:19:00.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:19:00 smithi191 ceph-mon[32884]: osdmap e214: 8 total, 8 up, 6 in 2024-09-10T19:19:00.802 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:19:00.837 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:19:00.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:00.838 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:19:00.939 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:19:00.965 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:19:00.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:00.966 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:19:01.067 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:19:01.095 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:19:01.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:01.096 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:19:01.197 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:19:01.237 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:19:01.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:01.238 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:19:01.316 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:19:01 smithi183 ceph-mon[29736]: from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:19:01.316 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:19:01 smithi183 ceph-mon[29736]: pgmap v391: 158 pgs: 1 active+clean+remapped, 1 active+remapped+backfill_toofull, 156 active+clean; 3.3 GiB data, 13 GiB used, 523 GiB / 536 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1.44k op/s; 3440/161355 objects misplaced (2.132%); 8.0 MiB/s, 128 objects/s recovering 2024-09-10T19:19:01.317 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:19:01 smithi183 ceph-mon[41562]: from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:19:01.317 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:19:01 smithi183 ceph-mon[41562]: pgmap v391: 158 pgs: 1 active+clean+remapped, 1 active+remapped+backfill_toofull, 156 active+clean; 3.3 GiB data, 13 GiB used, 523 GiB / 536 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1.44k op/s; 3440/161355 objects misplaced (2.132%); 8.0 MiB/s, 128 objects/s recovering 2024-09-10T19:19:01.338 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:19:01.369 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:19:01.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:01.370 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:19:01.471 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:19:01.580 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:19:01.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:01.581 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:19:01.681 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Sep 10 19:19:01 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-y[29989]: ::ffff:172.21.15.191 - - [10/Sep/2024:19:19:01] "GET /metrics HTTP/1.1" 200 224337 "" "Prometheus/2.43.0" 2024-09-10T19:19:01.681 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:19:01.715 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:19:01 smithi191 ceph-mon[32884]: from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:19:01.715 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:19:01 smithi191 ceph-mon[32884]: pgmap v391: 158 pgs: 1 active+clean+remapped, 1 active+remapped+backfill_toofull, 156 active+clean; 3.3 GiB data, 13 GiB used, 523 GiB / 536 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1.44k op/s; 3440/161355 objects misplaced (2.132%); 8.0 MiB/s, 128 objects/s recovering 2024-09-10T19:19:01.717 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:19:01.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:01.718 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:19:01.818 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:19:01.844 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:19:01.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:01.845 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:19:01.946 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:19:01.982 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:19:01.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:01.983 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:19:02.084 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:19:02.117 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:19:02.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:02.118 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:19:02.219 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:19:02.247 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:19:02.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:02.248 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:19:02.349 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:19:02.382 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:19:02.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:02.383 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:19:02.426 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:19:02 smithi183 ceph-mon[41562]: pgmap v393: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 2.9 GiB data, 12 GiB used, 524 GiB / 536 GiB avail; 857 B/s rd, 0 B/s wr, 972 op/s; 3052/143940 objects misplaced (2.120%) 2024-09-10T19:19:02.426 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:19:02 smithi183 ceph-mon[29736]: pgmap v393: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 2.9 GiB data, 12 GiB used, 524 GiB / 536 GiB avail; 857 B/s rd, 0 B/s wr, 972 op/s; 3052/143940 objects misplaced (2.120%) 2024-09-10T19:19:02.438 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-10T19:19:02.484 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:19:02.521 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:19:02.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:02.522 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:19:02.622 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:19:02.658 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:19:02.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:02.659 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:19:02.715 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:19:02 smithi191 ceph-mon[32884]: pgmap v393: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 2.9 GiB data, 12 GiB used, 524 GiB / 536 GiB avail; 857 B/s rd, 0 B/s wr, 972 op/s; 3052/143940 objects misplaced (2.120%) 2024-09-10T19:19:02.760 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:19:02.797 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:19:02.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:02.798 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:19:02.899 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:19:02.937 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:19:02.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:02.938 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:19:03.039 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:19:03.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:03.080 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:19:03.080 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:19:03.181 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:19:03.212 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:19:03.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:03.213 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:19:03.244 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-09-10T19:19:03.246 INFO:teuthology.orchestra.run.smithi183.stderr:dumped all 2024-09-10T19:19:03.315 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:19:03.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:03.352 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:19:03.353 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:19:03.453 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:19:03.482 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:19:03.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:03.483 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:19:03.584 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:19:03.619 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:19:03.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:03.620 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:19:03.721 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:19:03.748 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:19:03.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:03.749 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:19:03.849 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:19:03.881 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:19:03.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:03.882 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:19:03.983 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:19:04.022 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:19:04.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:04.023 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:19:04.124 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:19:04.149 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:19:04.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:04.150 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:19:04.251 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:19:04.279 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:19:04.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:04.280 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:19:04.381 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:19:04.417 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:19:04.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:04.418 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:19:04.520 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:19:04.556 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:19:04.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:04.557 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:19:04.658 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:19:04.686 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:19:04.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:04.687 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:19:04.788 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:19:04.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:04.827 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:19:04.827 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:19:04.928 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:19:04.931 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:19:04 smithi183 ceph-mon[29736]: from='client.25148 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-10T19:19:04.931 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:19:04 smithi183 ceph-mon[29736]: pgmap v394: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 2.9 GiB data, 12 GiB used, 524 GiB / 536 GiB avail; 734 B/s rd, 0 B/s wr, 833 op/s; 3052/143940 objects misplaced (2.120%) 2024-09-10T19:19:04.931 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:19:04 smithi183 ceph-mon[41562]: from='client.25148 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-10T19:19:04.932 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:19:04 smithi183 ceph-mon[41562]: pgmap v394: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 2.9 GiB data, 12 GiB used, 524 GiB / 536 GiB avail; 734 B/s rd, 0 B/s wr, 833 op/s; 3052/143940 objects misplaced (2.120%) 2024-09-10T19:19:04.954 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:19:04.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:04.955 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:19:05.056 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:19:05.081 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:19:05.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:05.082 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:19:05.183 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:19:05.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:19:04 smithi191 ceph-mon[32884]: from='client.25148 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-10T19:19:05.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:19:04 smithi191 ceph-mon[32884]: pgmap v394: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 2.9 GiB data, 12 GiB used, 524 GiB / 536 GiB avail; 734 B/s rd, 0 B/s wr, 833 op/s; 3052/143940 objects misplaced (2.120%) 2024-09-10T19:19:05.223 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:19:05.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:05.224 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:19:05.325 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:19:05.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:05.368 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:19:05.368 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:19:05.469 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:19:05.494 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:19:05.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:05.495 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:19:05.596 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:19:05.606 INFO:teuthology.orchestra.run.smithi183.stdout:{"pg_ready":true,"pg_map":{"version":393,"stamp":"2024-09-10T19:19:01.762101+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":3132102900,"num_objects":47980,"num_object_clones":0,"num_object_copies":143940,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3052,"num_objects_unfound":0,"num_objects_dirty":47980,"num_whiteouts":0,"num_read":1010,"num_read_kb":867,"num_write":193321,"num_write_kb":7699592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":29072,"num_bytes_recovered":1893929397,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":107563,"ondisk_log_size":107563,"up":474,"acting":474,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":491,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":562470912,"kb_used":12933944,"kb_used_data":10790256,"kb_used_omap":9,"kb_used_meta":2143158,"kb_avail":549536968,"statfs":{"total":575970213888,"available":562725855232,"internally_reserved":0,"allocated":11049222144,"data_stored":11045427502,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":10224,"internal_metadata":2194593808},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-380436480,"num_objects":-5805,"num_object_clones":0,"num_object_copies":-17415,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-388,"num_objects_unfound":0,"num_objects_dirty":-5805,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":5805,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.971668"},"pg_stats":[{"pgid":"8.15","version":"214'8967","reported_seq":13782,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-09-10T19:18:59.890527+0000","last_change":"2024-09-10T19:18:45.180957+0000","last_active":"2024-09-10T19:18:59.890527+0000","last_peered":"2024-09-10T19:18:59.890527+0000","last_clean":"2024-09-10T19:18:59.890527+0000","last_became_active":"2024-09-10T19:18:45.155095+0000","last_became_peered":"2024-09-10T19:18:45.155095+0000","last_unstale":"2024-09-10T19:18:59.890527+0000","last_undegraded":"2024-09-10T19:18:59.890527+0000","last_fullsized":"2024-09-10T19:18:59.890527+0000","mapping_epoch":206,"log_start":"173'5153","ondisk_log_start":"173'5153","created":177,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:16:11.777014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:16:11.777014+0000","last_clean_scrub_stamp":"2024-09-10T19:16:11.777014+0000","objects_scrubbed":0,"log_size":3814,"log_dups_size":0,"ondisk_log_size":3814,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":101122048,"num_objects":1543,"num_object_clones":0,"num_object_copies":4629,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1543,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6043,"num_write_kb":242752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2348,"num_bytes_recovered":153878528,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":178,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-10T19:18:59.219722+0000","last_change":"2024-09-10T19:13:08.710758+0000","last_active":"2024-09-10T19:18:59.219722+0000","last_peered":"2024-09-10T19:18:59.219722+0000","last_clean":"2024-09-10T19:18:59.219722+0000","last_became_active":"2024-09-10T19:10:14.056187+0000","last_became_peered":"2024-09-10T19:10:14.056187+0000","last_unstale":"2024-09-10T19:18:59.219722+0000","last_undegraded":"2024-09-10T19:18:59.219722+0000","last_fullsized":"2024-09-10T19:18:59.219722+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:13:08.710679+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:13:08.710679+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0027252499999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":213,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-10T19:18:59.224684+0000","last_change":"2024-09-10T19:16:38.194521+0000","last_active":"2024-09-10T19:18:59.224684+0000","last_peered":"2024-09-10T19:18:59.224684+0000","last_clean":"2024-09-10T19:18:59.224684+0000","last_became_active":"2024-09-10T19:16:24.836961+0000","last_became_peered":"2024-09-10T19:16:24.836961+0000","last_unstale":"2024-09-10T19:18:59.224684+0000","last_undegraded":"2024-09-10T19:18:59.224684+0000","last_fullsized":"2024-09-10T19:18:59.224684+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:16:38.194472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:16:38.194472+0000","last_clean_scrub_stamp":"2024-09-10T19:16:38.194472+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0023678900000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.19","version":"66'15","reported_seq":233,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-10T19:18:59.221004+0000","last_change":"2024-09-10T19:16:24.879212+0000","last_active":"2024-09-10T19:18:59.221004+0000","last_peered":"2024-09-10T19:18:59.221004+0000","last_clean":"2024-09-10T19:18:59.221004+0000","last_became_active":"2024-09-10T19:16:24.844951+0000","last_became_peered":"2024-09-10T19:16:24.844951+0000","last_unstale":"2024-09-10T19:18:59.221004+0000","last_undegraded":"2024-09-10T19:18:59.221004+0000","last_fullsized":"2024-09-10T19:18:59.221004+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'15","last_scrub_stamp":"2024-09-10T19:13:47.165848+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:13:47.165848+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.004982841,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":212,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-10T19:18:59.222978+0000","last_change":"2024-09-10T19:17:22.938220+0000","last_active":"2024-09-10T19:18:59.222978+0000","last_peered":"2024-09-10T19:18:59.222978+0000","last_clean":"2024-09-10T19:18:59.222978+0000","last_became_active":"2024-09-10T19:16:24.845111+0000","last_became_peered":"2024-09-10T19:16:24.845111+0000","last_unstale":"2024-09-10T19:18:59.222978+0000","last_undegraded":"2024-09-10T19:18:59.222978+0000","last_fullsized":"2024-09-10T19:18:59.222978+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:15:33.976696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:15:33.976696+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0015310580000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,7],"acting":[4,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.14","version":"213'8800","reported_seq":18687,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-10T19:18:59.729375+0000","last_change":"2024-09-10T19:17:49.772679+0000","last_active":"2024-09-10T19:18:59.729375+0000","last_peered":"2024-09-10T19:18:59.729375+0000","last_clean":"2024-09-10T19:18:59.729375+0000","last_became_active":"2024-09-10T19:17:49.767787+0000","last_became_peered":"2024-09-10T19:17:49.767787+0000","last_unstale":"2024-09-10T19:18:59.729375+0000","last_undegraded":"2024-09-10T19:18:59.729375+0000","last_fullsized":"2024-09-10T19:18:59.729375+0000","mapping_epoch":189,"log_start":"168'4204","ondisk_log_start":"168'4204","created":177,"last_epoch_clean":190,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:16:11.777014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:16:11.777014+0000","last_clean_scrub_stamp":"2024-09-10T19:16:11.777014+0000","objects_scrubbed":0,"log_size":4596,"log_dups_size":0,"ondisk_log_size":4596,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":98402304,"num_objects":1502,"num_object_clones":0,"num_object_copies":4506,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1502,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5938,"num_write_kb":238048,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3256,"num_bytes_recovered":213385216,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"59'1","reported_seq":202,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-10T19:18:59.221341+0000","last_change":"2024-09-10T19:16:30.798883+0000","last_active":"2024-09-10T19:18:59.221341+0000","last_peered":"2024-09-10T19:18:59.221341+0000","last_clean":"2024-09-10T19:18:59.221341+0000","last_became_active":"2024-09-10T19:10:07.686143+0000","last_became_peered":"2024-09-10T19:10:07.686143+0000","last_unstale":"2024-09-10T19:18:59.221341+0000","last_undegraded":"2024-09-10T19:18:59.221341+0000","last_fullsized":"2024-09-10T19:18:59.221341+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'1","last_scrub_stamp":"2024-09-10T19:16:30.798795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:16:30.798795+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0067002989999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":874,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"66'9","reported_seq":227,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-10T19:18:59.223052+0000","last_change":"2024-09-10T19:17:21.915673+0000","last_active":"2024-09-10T19:18:59.223052+0000","last_peered":"2024-09-10T19:18:59.223052+0000","last_clean":"2024-09-10T19:18:59.223052+0000","last_became_active":"2024-09-10T19:13:54.716117+0000","last_became_peered":"2024-09-10T19:13:54.716117+0000","last_unstale":"2024-09-10T19:18:59.223052+0000","last_undegraded":"2024-09-10T19:18:59.223052+0000","last_fullsized":"2024-09-10T19:18:59.223052+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'9","last_scrub_stamp":"2024-09-10T19:14:55.117387+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:14:55.117387+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0074980940000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,3],"acting":[4,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":154,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-10T19:18:59.232460+0000","last_change":"2024-09-10T19:16:27.753997+0000","last_active":"2024-09-10T19:18:59.232460+0000","last_peered":"2024-09-10T19:18:59.232460+0000","last_clean":"2024-09-10T19:18:59.232460+0000","last_became_active":"2024-09-10T19:16:24.859261+0000","last_became_peered":"2024-09-10T19:16:24.859261+0000","last_unstale":"2024-09-10T19:18:59.232460+0000","last_undegraded":"2024-09-10T19:18:59.232460+0000","last_fullsized":"2024-09-10T19:18:59.232460+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:16:27.753912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:16:27.753912+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0029373799999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":200,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-10T19:18:59.223153+0000","last_change":"2024-09-10T19:17:23.968484+0000","last_active":"2024-09-10T19:18:59.223153+0000","last_peered":"2024-09-10T19:18:59.223153+0000","last_clean":"2024-09-10T19:18:59.223153+0000","last_became_active":"2024-09-10T19:13:54.713538+0000","last_became_peered":"2024-09-10T19:13:54.713538+0000","last_unstale":"2024-09-10T19:18:59.223153+0000","last_undegraded":"2024-09-10T19:18:59.223153+0000","last_fullsized":"2024-09-10T19:18:59.223153+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:15:39.910118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:15:39.910118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0013551839999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,7],"acting":[4,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.17","version":"214'8919","reported_seq":16203,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-09-10T19:18:59.859856+0000","last_change":"2024-09-10T19:18:44.173985+0000","last_active":"2024-09-10T19:18:59.859856+0000","last_peered":"2024-09-10T19:18:59.859856+0000","last_clean":"2024-09-10T19:18:59.859856+0000","last_became_active":"2024-09-10T19:18:44.154191+0000","last_became_peered":"2024-09-10T19:18:44.154191+0000","last_unstale":"2024-09-10T19:18:59.859856+0000","last_undegraded":"2024-09-10T19:18:59.859856+0000","last_fullsized":"2024-09-10T19:18:59.859856+0000","mapping_epoch":205,"log_start":"173'5157","ondisk_log_start":"173'5157","created":177,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:16:11.777014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:16:11.777014+0000","last_clean_scrub_stamp":"2024-09-10T19:16:11.777014+0000","objects_scrubbed":0,"log_size":3762,"log_dups_size":0,"ondisk_log_size":3762,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":97157120,"num_objects":1483,"num_object_clones":0,"num_object_copies":4449,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1483,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6021,"num_write_kb":240096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2387,"num_bytes_recovered":156434432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":194,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-10T19:18:59.224962+0000","last_change":"2024-09-10T19:16:25.595265+0000","last_active":"2024-09-10T19:18:59.224962+0000","last_peered":"2024-09-10T19:18:59.224962+0000","last_clean":"2024-09-10T19:18:59.224962+0000","last_became_active":"2024-09-10T19:10:07.690728+0000","last_became_peered":"2024-09-10T19:10:07.690728+0000","last_unstale":"2024-09-10T19:18:59.224962+0000","last_undegraded":"2024-09-10T19:18:59.224962+0000","last_fullsized":"2024-09-10T19:18:59.224962+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:16:25.595127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:16:25.595127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.003922633,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1b","version":"66'5","reported_seq":233,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-10T19:18:59.222883+0000","last_change":"2024-09-10T19:16:35.966457+0000","last_active":"2024-09-10T19:18:59.222883+0000","last_peered":"2024-09-10T19:18:59.222883+0000","last_clean":"2024-09-10T19:18:59.222883+0000","last_became_active":"2024-09-10T19:16:24.843708+0000","last_became_peered":"2024-09-10T19:16:24.843708+0000","last_unstale":"2024-09-10T19:18:59.222883+0000","last_undegraded":"2024-09-10T19:18:59.222883+0000","last_fullsized":"2024-09-10T19:18:59.222883+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'5","last_scrub_stamp":"2024-09-10T19:16:35.966402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:16:35.966402+0000","objects_scrubbed":2,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0019325740000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":197,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-10T19:18:59.225245+0000","last_change":"2024-09-10T19:15:33.518718+0000","last_active":"2024-09-10T19:18:59.225245+0000","last_peered":"2024-09-10T19:18:59.225245+0000","last_clean":"2024-09-10T19:18:59.225245+0000","last_became_active":"2024-09-10T19:13:54.719063+0000","last_became_peered":"2024-09-10T19:13:54.719063+0000","last_unstale":"2024-09-10T19:18:59.225245+0000","last_undegraded":"2024-09-10T19:18:59.225245+0000","last_fullsized":"2024-09-10T19:18:59.225245+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:15:33.518601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:15:33.518601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0013951650000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":205,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-10T19:18:59.232534+0000","last_change":"2024-09-10T19:16:24.845721+0000","last_active":"2024-09-10T19:18:59.232534+0000","last_peered":"2024-09-10T19:18:59.232534+0000","last_clean":"2024-09-10T19:18:59.232534+0000","last_became_active":"2024-09-10T19:16:24.845130+0000","last_became_peered":"2024-09-10T19:16:24.845130+0000","last_unstale":"2024-09-10T19:18:59.232534+0000","last_undegraded":"2024-09-10T19:18:59.232534+0000","last_fullsized":"2024-09-10T19:18:59.232534+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:15:45.616184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:15:45.616184+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0013989510000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,0],"acting":[5,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.16","version":"214'8816","reported_seq":15898,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-09-10T19:18:59.996763+0000","last_change":"2024-09-10T19:18:44.173818+0000","last_active":"2024-09-10T19:18:59.996763+0000","last_peered":"2024-09-10T19:18:59.996763+0000","last_clean":"2024-09-10T19:18:59.996763+0000","last_became_active":"2024-09-10T19:18:44.148801+0000","last_became_peered":"2024-09-10T19:18:44.148801+0000","last_unstale":"2024-09-10T19:18:59.996763+0000","last_undegraded":"2024-09-10T19:18:59.996763+0000","last_fullsized":"2024-09-10T19:18:59.996763+0000","mapping_epoch":205,"log_start":"171'4751","ondisk_log_start":"171'4751","created":177,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:16:11.777014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:16:11.777014+0000","last_clean_scrub_stamp":"2024-09-10T19:16:11.777014+0000","objects_scrubbed":0,"log_size":4065,"log_dups_size":0,"ondisk_log_size":4065,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":94470144,"num_objects":1442,"num_object_clones":0,"num_object_copies":4326,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1442,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5944,"num_write_kb":236320,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2323,"num_bytes_recovered":152240128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,0],"acting":[3,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":194,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-10T19:18:59.230092+0000","last_change":"2024-09-10T19:16:24.859112+0000","last_active":"2024-09-10T19:18:59.230092+0000","last_peered":"2024-09-10T19:18:59.230092+0000","last_clean":"2024-09-10T19:18:59.230092+0000","last_became_active":"2024-09-10T19:16:24.858800+0000","last_became_peered":"2024-09-10T19:16:24.858800+0000","last_unstale":"2024-09-10T19:18:59.230092+0000","last_undegraded":"2024-09-10T19:18:59.230092+0000","last_fullsized":"2024-09-10T19:18:59.230092+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:12:54.477149+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:12:54.477149+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0039768319999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1a","version":"66'9","reported_seq":200,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-10T19:18:59.223410+0000","last_change":"2024-09-10T19:17:14.981950+0000","last_active":"2024-09-10T19:18:59.223410+0000","last_peered":"2024-09-10T19:18:59.223410+0000","last_clean":"2024-09-10T19:18:59.223410+0000","last_became_active":"2024-09-10T19:10:09.688987+0000","last_became_peered":"2024-09-10T19:10:09.688987+0000","last_unstale":"2024-09-10T19:18:59.223410+0000","last_undegraded":"2024-09-10T19:18:59.223410+0000","last_fullsized":"2024-09-10T19:18:59.223410+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'9","last_scrub_stamp":"2024-09-10T19:13:02.431443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:13:02.431443+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0084913290000000006,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":190,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-10T19:18:59.231829+0000","last_change":"2024-09-10T19:16:37.116464+0000","last_active":"2024-09-10T19:18:59.231829+0000","last_peered":"2024-09-10T19:18:59.231829+0000","last_clean":"2024-09-10T19:18:59.231829+0000","last_became_active":"2024-09-10T19:10:11.701799+0000","last_became_peered":"2024-09-10T19:10:11.701799+0000","last_unstale":"2024-09-10T19:18:59.231829+0000","last_undegraded":"2024-09-10T19:18:59.231829+0000","last_fullsized":"2024-09-10T19:18:59.231829+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:16:37.116371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:16:37.116371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.028100301000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":210,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-10T19:18:59.223103+0000","last_change":"2024-09-10T19:18:20.026136+0000","last_active":"2024-09-10T19:18:59.223103+0000","last_peered":"2024-09-10T19:18:59.223103+0000","last_clean":"2024-09-10T19:18:59.223103+0000","last_became_active":"2024-09-10T19:16:24.836477+0000","last_became_peered":"2024-09-10T19:16:24.836477+0000","last_unstale":"2024-09-10T19:18:59.223103+0000","last_undegraded":"2024-09-10T19:18:59.223103+0000","last_fullsized":"2024-09-10T19:18:59.223103+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:15:33.839248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:15:33.839248+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0032019980000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1","version":"214'9002","reported_seq":9095,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-09-10T19:18:59.954421+0000","last_change":"2024-09-10T19:17:26.961275+0000","last_active":"2024-09-10T19:18:59.954421+0000","last_peered":"2024-09-10T19:18:59.954421+0000","last_clean":"2024-09-10T19:18:59.954421+0000","last_became_active":"2024-09-10T19:17:26.958866+0000","last_became_peered":"2024-09-10T19:17:26.958866+0000","last_unstale":"2024-09-10T19:18:59.954421+0000","last_undegraded":"2024-09-10T19:18:59.954421+0000","last_fullsized":"2024-09-10T19:18:59.954421+0000","mapping_epoch":154,"log_start":"171'4597","ondisk_log_start":"171'4597","created":148,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:16:11.777014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:16:11.777014+0000","last_clean_scrub_stamp":"2024-09-10T19:16:11.777014+0000","objects_scrubbed":0,"log_size":4405,"log_dups_size":0,"ondisk_log_size":4405,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":99319808,"num_objects":1515,"num_object_clones":0,"num_object_copies":4545,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1515,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6083,"num_write_kb":243168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.d","version":"66'17","reported_seq":282,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-10T19:18:59.220250+0000","last_change":"2024-09-10T19:16:32.999033+0000","last_active":"2024-09-10T19:18:59.220250+0000","last_peered":"2024-09-10T19:18:59.220250+0000","last_clean":"2024-09-10T19:18:59.220250+0000","last_became_active":"2024-09-10T19:16:24.847056+0000","last_became_peered":"2024-09-10T19:16:24.847056+0000","last_unstale":"2024-09-10T19:18:59.220250+0000","last_undegraded":"2024-09-10T19:18:59.220250+0000","last_fullsized":"2024-09-10T19:18:59.220250+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'17","last_scrub_stamp":"2024-09-10T19:16:32.998950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:16:32.998950+0000","objects_scrubbed":8,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0095505250000000007,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,0],"acting":[4,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":194,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-10T19:18:59.229634+0000","last_change":"2024-09-10T19:16:39.716870+0000","last_active":"2024-09-10T19:18:59.229634+0000","last_peered":"2024-09-10T19:18:59.229634+0000","last_clean":"2024-09-10T19:18:59.229634+0000","last_became_active":"2024-09-10T19:10:07.691119+0000","last_became_peered":"2024-09-10T19:10:07.691119+0000","last_unstale":"2024-09-10T19:18:59.229634+0000","last_undegraded":"2024-09-10T19:18:59.229634+0000","last_fullsized":"2024-09-10T19:18:59.229634+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:16:39.716674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:16:39.716674+0000","last_clean_scrub_stamp":"2024-09-10T19:16:39.716674+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.043895318000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":147,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-10T19:18:59.223590+0000","last_change":"2024-09-10T19:16:26.999105+0000","last_active":"2024-09-10T19:18:59.223590+0000","last_peered":"2024-09-10T19:18:59.223590+0000","last_clean":"2024-09-10T19:18:59.223590+0000","last_became_active":"2024-09-10T19:13:54.712883+0000","last_became_peered":"2024-09-10T19:13:54.712883+0000","last_unstale":"2024-09-10T19:18:59.223590+0000","last_undegraded":"2024-09-10T19:18:59.223590+0000","last_fullsized":"2024-09-10T19:18:59.223590+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:16:26.999036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:16:26.999036+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0050193119999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":72,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-10T19:18:59.221275+0000","last_change":"2024-09-10T19:16:24.858656+0000","last_active":"2024-09-10T19:18:59.221275+0000","last_peered":"2024-09-10T19:18:59.221275+0000","last_clean":"2024-09-10T19:18:59.221275+0000","last_became_active":"2024-09-10T19:16:24.857843+0000","last_became_peered":"2024-09-10T19:16:24.857843+0000","last_unstale":"2024-09-10T19:18:59.221275+0000","last_undegraded":"2024-09-10T19:18:59.221275+0000","last_fullsized":"2024-09-10T19:18:59.221275+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:15:47.818381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:15:47.818381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.0","version":"214'8900","reported_seq":8986,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-09-10T19:18:59.919354+0000","last_change":"2024-09-10T19:17:26.971893+0000","last_active":"2024-09-10T19:18:59.919354+0000","last_peered":"2024-09-10T19:18:59.919354+0000","last_clean":"2024-09-10T19:18:59.919354+0000","last_became_active":"2024-09-10T19:17:26.958072+0000","last_became_peered":"2024-09-10T19:17:26.958072+0000","last_unstale":"2024-09-10T19:18:59.919354+0000","last_undegraded":"2024-09-10T19:18:59.919354+0000","last_fullsized":"2024-09-10T19:18:59.919354+0000","mapping_epoch":148,"log_start":"171'4709","ondisk_log_start":"171'4709","created":148,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:16:11.777014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:16:11.777014+0000","last_clean_scrub_stamp":"2024-09-10T19:16:11.777014+0000","objects_scrubbed":0,"log_size":4191,"log_dups_size":0,"ondisk_log_size":4191,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":103415808,"num_objects":1578,"num_object_clones":0,"num_object_copies":4734,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1578,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5984,"num_write_kb":241984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,4],"acting":[3,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"66'10","reported_seq":209,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-10T19:18:59.226558+0000","last_change":"2024-09-10T19:17:20.959169+0000","last_active":"2024-09-10T19:18:59.226558+0000","last_peered":"2024-09-10T19:18:59.226558+0000","last_clean":"2024-09-10T19:18:59.226558+0000","last_became_active":"2024-09-10T19:10:09.688321+0000","last_became_peered":"2024-09-10T19:10:09.688321+0000","last_unstale":"2024-09-10T19:18:59.226558+0000","last_undegraded":"2024-09-10T19:18:59.226558+0000","last_fullsized":"2024-09-10T19:18:59.226558+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'10","last_scrub_stamp":"2024-09-10T19:13:51.415998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:13:51.415998+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.005972799,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":144,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-10T19:18:59.220489+0000","last_change":"2024-09-10T19:15:42.119912+0000","last_active":"2024-09-10T19:18:59.220489+0000","last_peered":"2024-09-10T19:18:59.220489+0000","last_clean":"2024-09-10T19:18:59.220489+0000","last_became_active":"2024-09-10T19:13:54.731134+0000","last_became_peered":"2024-09-10T19:13:54.731134+0000","last_unstale":"2024-09-10T19:18:59.220489+0000","last_undegraded":"2024-09-10T19:18:59.220489+0000","last_fullsized":"2024-09-10T19:18:59.220489+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:15:42.119590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:15:42.119590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.018001347000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,5],"acting":[3,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"155'10","reported_seq":173,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-10T19:18:59.224717+0000","last_change":"2024-09-10T19:16:29.626806+0000","last_active":"2024-09-10T19:18:59.224717+0000","last_peered":"2024-09-10T19:18:59.224717+0000","last_clean":"2024-09-10T19:18:59.224717+0000","last_became_active":"2024-09-10T19:16:24.851236+0000","last_became_peered":"2024-09-10T19:16:24.851236+0000","last_unstale":"2024-09-10T19:18:59.224717+0000","last_undegraded":"2024-09-10T19:18:59.224717+0000","last_fullsized":"2024-09-10T19:18:59.224717+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"155'10","last_scrub_stamp":"2024-09-10T19:16:29.626572+0000","last_deep_scrub":"80'9","last_deep_scrub_stamp":"2024-09-10T19:11:25.156465+0000","last_clean_scrub_stamp":"2024-09-10T19:16:29.626572+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0067064810000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,0],"acting":[7,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":210,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-10T19:18:59.222386+0000","last_change":"2024-09-10T19:16:24.852100+0000","last_active":"2024-09-10T19:18:59.222386+0000","last_peered":"2024-09-10T19:18:59.222386+0000","last_clean":"2024-09-10T19:18:59.222386+0000","last_became_active":"2024-09-10T19:16:24.851829+0000","last_became_peered":"2024-09-10T19:16:24.851829+0000","last_unstale":"2024-09-10T19:18:59.222386+0000","last_undegraded":"2024-09-10T19:18:59.222386+0000","last_fullsized":"2024-09-10T19:18:59.222386+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:15:29.988976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:15:29.988976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0053787150000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,7],"acting":[4,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.3","version":"214'9239","reported_seq":14288,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-09-10T19:18:59.882590+0000","last_change":"2024-09-10T19:18:43.997699+0000","last_active":"2024-09-10T19:18:59.882590+0000","last_peered":"2024-09-10T19:18:59.882590+0000","last_clean":"2024-09-10T19:18:59.882590+0000","last_became_active":"2024-09-10T19:17:27.974691+0000","last_became_peered":"2024-09-10T19:17:27.974691+0000","last_unstale":"2024-09-10T19:18:59.882590+0000","last_undegraded":"2024-09-10T19:18:59.882590+0000","last_fullsized":"2024-09-10T19:18:59.882590+0000","mapping_epoch":178,"log_start":"173'5538","ondisk_log_start":"173'5538","created":148,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:16:11.777014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:16:11.777014+0000","last_clean_scrub_stamp":"2024-09-10T19:16:11.777014+0000","objects_scrubbed":0,"log_size":3701,"log_dups_size":0,"ondisk_log_size":3701,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":98992128,"num_objects":1510,"num_object_clones":0,"num_object_copies":4530,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1510,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6228,"num_write_kb":247648,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1928,"num_bytes_recovered":126386176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.f","version":"66'15","reported_seq":197,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-10T19:18:59.221492+0000","last_change":"2024-09-10T19:16:25.769523+0000","last_active":"2024-09-10T19:18:59.221492+0000","last_peered":"2024-09-10T19:18:59.221492+0000","last_clean":"2024-09-10T19:18:59.221492+0000","last_became_active":"2024-09-10T19:13:54.711634+0000","last_became_peered":"2024-09-10T19:13:54.711634+0000","last_unstale":"2024-09-10T19:18:59.221492+0000","last_undegraded":"2024-09-10T19:18:59.221492+0000","last_fullsized":"2024-09-10T19:18:59.221492+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'15","last_scrub_stamp":"2024-09-10T19:16:25.769436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:16:25.769436+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0046254920000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":154,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-10T19:18:59.227234+0000","last_change":"2024-09-10T19:16:24.857467+0000","last_active":"2024-09-10T19:18:59.227234+0000","last_peered":"2024-09-10T19:18:59.227234+0000","last_clean":"2024-09-10T19:18:59.227234+0000","last_became_active":"2024-09-10T19:16:24.857179+0000","last_became_peered":"2024-09-10T19:16:24.857179+0000","last_unstale":"2024-09-10T19:18:59.227234+0000","last_undegraded":"2024-09-10T19:18:59.227234+0000","last_fullsized":"2024-09-10T19:18:59.227234+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:15:29.553448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:15:29.553448+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0062080679999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,6],"acting":[7,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.e","version":"66'8","reported_seq":296,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-10T19:18:59.222539+0000","last_change":"2024-09-10T19:17:18.952116+0000","last_active":"2024-09-10T19:18:59.222539+0000","last_peered":"2024-09-10T19:18:59.222539+0000","last_clean":"2024-09-10T19:18:59.222539+0000","last_became_active":"2024-09-10T19:10:11.699232+0000","last_became_peered":"2024-09-10T19:10:11.699232+0000","last_unstale":"2024-09-10T19:18:59.222539+0000","last_undegraded":"2024-09-10T19:18:59.222539+0000","last_fullsized":"2024-09-10T19:18:59.222539+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'8","last_scrub_stamp":"2024-09-10T19:13:40.475375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:13:40.475375+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.035806152000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":205,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-10T19:18:59.236528+0000","last_change":"2024-09-10T19:16:24.855938+0000","last_active":"2024-09-10T19:18:59.236528+0000","last_peered":"2024-09-10T19:18:59.236528+0000","last_clean":"2024-09-10T19:18:59.236528+0000","last_became_active":"2024-09-10T19:16:24.855627+0000","last_became_peered":"2024-09-10T19:16:24.855627+0000","last_unstale":"2024-09-10T19:18:59.236528+0000","last_undegraded":"2024-09-10T19:18:59.236528+0000","last_fullsized":"2024-09-10T19:18:59.236528+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:15:44.633043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:15:44.633043+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.00368805,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,6],"acting":[5,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.2","version":"213'8872","reported_seq":8957,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-10T19:18:59.749165+0000","last_change":"2024-09-10T19:17:26.956443+0000","last_active":"2024-09-10T19:18:59.749165+0000","last_peered":"2024-09-10T19:18:59.749165+0000","last_clean":"2024-09-10T19:18:59.749165+0000","last_became_active":"2024-09-10T19:17:26.953881+0000","last_became_peered":"2024-09-10T19:17:26.953881+0000","last_unstale":"2024-09-10T19:18:59.749165+0000","last_undegraded":"2024-09-10T19:18:59.749165+0000","last_fullsized":"2024-09-10T19:18:59.749165+0000","mapping_epoch":148,"log_start":"168'4232","ondisk_log_start":"168'4232","created":148,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:16:11.777014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:16:11.777014+0000","last_clean_scrub_stamp":"2024-09-10T19:16:11.777014+0000","objects_scrubbed":0,"log_size":4640,"log_dups_size":0,"ondisk_log_size":4640,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":95748096,"num_objects":1461,"num_object_clones":0,"num_object_copies":4383,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1461,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5955,"num_write_kb":237312,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,5],"acting":[4,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"84'16","reported_seq":237,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-10T19:18:59.222620+0000","last_change":"2024-09-10T19:15:47.914487+0000","last_active":"2024-09-10T19:18:59.222620+0000","last_peered":"2024-09-10T19:18:59.222620+0000","last_clean":"2024-09-10T19:18:59.222620+0000","last_became_active":"2024-09-10T19:13:54.717022+0000","last_became_peered":"2024-09-10T19:13:54.717022+0000","last_unstale":"2024-09-10T19:18:59.222620+0000","last_undegraded":"2024-09-10T19:18:59.222620+0000","last_fullsized":"2024-09-10T19:18:59.222620+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'16","last_scrub_stamp":"2024-09-10T19:15:47.914420+0000","last_deep_scrub":"84'16","last_deep_scrub_stamp":"2024-09-10T19:11:56.157975+0000","last_clean_scrub_stamp":"2024-09-10T19:15:47.914420+0000","objects_scrubbed":5,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0019153390000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,3],"acting":[4,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":201,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-10T19:18:59.227784+0000","last_change":"2024-09-10T19:15:42.450149+0000","last_active":"2024-09-10T19:18:59.227784+0000","last_peered":"2024-09-10T19:18:59.227784+0000","last_clean":"2024-09-10T19:18:59.227784+0000","last_became_active":"2024-09-10T19:13:54.720293+0000","last_became_peered":"2024-09-10T19:13:54.720293+0000","last_unstale":"2024-09-10T19:18:59.227784+0000","last_undegraded":"2024-09-10T19:18:59.227784+0000","last_fullsized":"2024-09-10T19:18:59.227784+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:15:42.450068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:11:53.066849+0000","last_clean_scrub_stamp":"2024-09-10T19:15:42.450068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0037708360000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":180,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-10T19:18:59.236153+0000","last_change":"2024-09-10T19:13:06.358188+0000","last_active":"2024-09-10T19:18:59.236153+0000","last_peered":"2024-09-10T19:18:59.236153+0000","last_clean":"2024-09-10T19:18:59.236153+0000","last_became_active":"2024-09-10T19:10:11.692215+0000","last_became_peered":"2024-09-10T19:10:11.692215+0000","last_unstale":"2024-09-10T19:18:59.236153+0000","last_undegraded":"2024-09-10T19:18:59.236153+0000","last_fullsized":"2024-09-10T19:18:59.236153+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:13:06.358109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:13:06.358109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0026724549999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":178,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-10T19:18:59.219520+0000","last_change":"2024-09-10T19:13:54.304493+0000","last_active":"2024-09-10T19:18:59.219520+0000","last_peered":"2024-09-10T19:18:59.219520+0000","last_clean":"2024-09-10T19:18:59.219520+0000","last_became_active":"2024-09-10T19:10:14.053736+0000","last_became_peered":"2024-09-10T19:10:14.053736+0000","last_unstale":"2024-09-10T19:18:59.219520+0000","last_undegraded":"2024-09-10T19:18:59.219520+0000","last_fullsized":"2024-09-10T19:18:59.219520+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:13:54.304435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:13:54.304435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0011043820000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.d","version":"214'12112","reported_seq":12197,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-09-10T19:18:59.852090+0000","last_change":"2024-09-10T19:16:24.858800+0000","last_active":"2024-09-10T19:18:59.852090+0000","last_peered":"2024-09-10T19:18:59.852090+0000","last_clean":"2024-09-10T19:18:59.852090+0000","last_became_active":"2024-09-10T19:16:24.858460+0000","last_became_peered":"2024-09-10T19:16:24.858460+0000","last_unstale":"2024-09-10T19:18:59.852090+0000","last_undegraded":"2024-09-10T19:18:59.852090+0000","last_fullsized":"2024-09-10T19:18:59.852090+0000","mapping_epoch":154,"log_start":"192'8290","ondisk_log_start":"192'8290","created":148,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:16:11.777014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:16:11.777014+0000","last_clean_scrub_stamp":"2024-09-10T19:16:11.777014+0000","objects_scrubbed":0,"log_size":3822,"log_dups_size":0,"ondisk_log_size":3822,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":196411416,"num_objects":2998,"num_object_clones":0,"num_object_copies":8994,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2998,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12112,"num_write_kb":483457,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1","version":"66'14","reported_seq":88,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-10T19:18:59.219339+0000","last_change":"2024-09-10T19:17:28.925690+0000","last_active":"2024-09-10T19:18:59.219339+0000","last_peered":"2024-09-10T19:18:59.219339+0000","last_clean":"2024-09-10T19:18:59.219339+0000","last_became_active":"2024-09-10T19:16:24.852821+0000","last_became_peered":"2024-09-10T19:16:24.852821+0000","last_unstale":"2024-09-10T19:18:59.219339+0000","last_undegraded":"2024-09-10T19:18:59.219339+0000","last_fullsized":"2024-09-10T19:18:59.219339+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'14","last_scrub_stamp":"2024-09-10T19:15:41.748101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:15:41.748101+0000","objects_scrubbed":7,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0055403099999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,7],"acting":[4,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":204,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-10T19:18:59.228758+0000","last_change":"2024-09-10T19:16:25.582072+0000","last_active":"2024-09-10T19:18:59.228758+0000","last_peered":"2024-09-10T19:18:59.228758+0000","last_clean":"2024-09-10T19:18:59.228758+0000","last_became_active":"2024-09-10T19:16:24.855861+0000","last_became_peered":"2024-09-10T19:16:24.855861+0000","last_unstale":"2024-09-10T19:18:59.228758+0000","last_undegraded":"2024-09-10T19:18:59.228758+0000","last_fullsized":"2024-09-10T19:18:59.228758+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:16:25.581993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:16:25.581993+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0041345990000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.0","version":"117'10","reported_seq":316,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-10T19:18:59.219562+0000","last_change":"2024-09-10T19:15:35.973188+0000","last_active":"2024-09-10T19:18:59.219562+0000","last_peered":"2024-09-10T19:18:59.219562+0000","last_clean":"2024-09-10T19:18:59.219562+0000","last_became_active":"2024-09-10T19:13:54.709846+0000","last_became_peered":"2024-09-10T19:13:54.709846+0000","last_unstale":"2024-09-10T19:18:59.219562+0000","last_undegraded":"2024-09-10T19:18:59.219562+0000","last_fullsized":"2024-09-10T19:18:59.219562+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"117'10","last_scrub_stamp":"2024-09-10T19:15:35.973095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:15:35.973095+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0076338860000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,0],"acting":[3,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":188,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-10T19:18:59.229291+0000","last_change":"2024-09-10T19:16:24.858926+0000","last_active":"2024-09-10T19:18:59.229291+0000","last_peered":"2024-09-10T19:18:59.229291+0000","last_clean":"2024-09-10T19:18:59.229291+0000","last_became_active":"2024-09-10T19:16:24.858619+0000","last_became_peered":"2024-09-10T19:16:24.858619+0000","last_unstale":"2024-09-10T19:18:59.229291+0000","last_undegraded":"2024-09-10T19:18:59.229291+0000","last_fullsized":"2024-09-10T19:18:59.229291+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:13:15.350720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:11:41.073833+0000","last_clean_scrub_stamp":"2024-09-10T19:13:15.350720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0014361000000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.c","version":"214'12126","reported_seq":12202,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-09-10T19:19:00.008696+0000","last_change":"2024-09-10T19:17:29.176247+0000","last_active":"2024-09-10T19:19:00.008696+0000","last_peered":"2024-09-10T19:19:00.008696+0000","last_clean":"2024-09-10T19:19:00.008696+0000","last_became_active":"2024-09-10T19:16:13.079527+0000","last_became_peered":"2024-09-10T19:16:13.079527+0000","last_unstale":"2024-09-10T19:19:00.008696+0000","last_undegraded":"2024-09-10T19:19:00.008696+0000","last_fullsized":"2024-09-10T19:19:00.008696+0000","mapping_epoch":148,"log_start":"192'8273","ondisk_log_start":"192'8273","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:16:11.777014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:16:11.777014+0000","last_clean_scrub_stamp":"2024-09-10T19:16:11.777014+0000","objects_scrubbed":0,"log_size":3853,"log_dups_size":0,"ondisk_log_size":3853,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":196083712,"num_objects":2992,"num_object_clones":0,"num_object_copies":8976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2992,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12126,"num_write_kb":483776,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.0","version":"66'18","reported_seq":214,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-10T19:18:59.220098+0000","last_change":"2024-09-10T19:12:49.699111+0000","last_active":"2024-09-10T19:18:59.220098+0000","last_peered":"2024-09-10T19:18:59.220098+0000","last_clean":"2024-09-10T19:18:59.220098+0000","last_became_active":"2024-09-10T19:10:09.821836+0000","last_became_peered":"2024-09-10T19:10:09.821836+0000","last_unstale":"2024-09-10T19:18:59.220098+0000","last_undegraded":"2024-09-10T19:18:59.220098+0000","last_fullsized":"2024-09-10T19:18:59.220098+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'18","last_scrub_stamp":"2024-09-10T19:12:49.699054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:12:49.699054+0000","objects_scrubbed":8,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0073070890000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":154,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-10T19:18:59.227911+0000","last_change":"2024-09-10T19:17:26.374435+0000","last_active":"2024-09-10T19:18:59.227911+0000","last_peered":"2024-09-10T19:18:59.227911+0000","last_clean":"2024-09-10T19:18:59.227911+0000","last_became_active":"2024-09-10T19:13:54.710968+0000","last_became_peered":"2024-09-10T19:13:54.710968+0000","last_unstale":"2024-09-10T19:18:59.227911+0000","last_undegraded":"2024-09-10T19:18:59.227911+0000","last_fullsized":"2024-09-10T19:18:59.227911+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:17:26.374376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:17:26.374376+0000","last_clean_scrub_stamp":"2024-09-10T19:17:26.374376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001299815,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,3],"acting":[6,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":189,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-10T19:18:59.219551+0000","last_change":"2024-09-10T19:16:30.991715+0000","last_active":"2024-09-10T19:18:59.219551+0000","last_peered":"2024-09-10T19:18:59.219551+0000","last_clean":"2024-09-10T19:18:59.219551+0000","last_became_active":"2024-09-10T19:10:11.698054+0000","last_became_peered":"2024-09-10T19:10:11.698054+0000","last_unstale":"2024-09-10T19:18:59.219551+0000","last_undegraded":"2024-09-10T19:18:59.219551+0000","last_fullsized":"2024-09-10T19:18:59.219551+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:16:30.991635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:16:30.991635+0000","last_clean_scrub_stamp":"2024-09-10T19:16:30.991635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0021810029999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":197,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-10T19:18:59.223341+0000","last_change":"2024-09-10T19:16:30.036748+0000","last_active":"2024-09-10T19:18:59.223341+0000","last_peered":"2024-09-10T19:18:59.223341+0000","last_clean":"2024-09-10T19:18:59.223341+0000","last_became_active":"2024-09-10T19:16:24.848109+0000","last_became_peered":"2024-09-10T19:16:24.848109+0000","last_unstale":"2024-09-10T19:18:59.223341+0000","last_undegraded":"2024-09-10T19:18:59.223341+0000","last_fullsized":"2024-09-10T19:18:59.223341+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:16:30.036662+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:16:30.036662+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0044984129999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.f","version":"214'11838","reported_seq":11922,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-09-10T19:19:00.023364+0000","last_change":"2024-09-10T19:16:24.856828+0000","last_active":"2024-09-10T19:19:00.023364+0000","last_peered":"2024-09-10T19:19:00.023364+0000","last_clean":"2024-09-10T19:19:00.023364+0000","last_became_active":"2024-09-10T19:16:24.855393+0000","last_became_peered":"2024-09-10T19:16:24.855393+0000","last_unstale":"2024-09-10T19:19:00.023364+0000","last_undegraded":"2024-09-10T19:19:00.023364+0000","last_fullsized":"2024-09-10T19:19:00.023364+0000","mapping_epoch":154,"log_start":"193'8199","ondisk_log_start":"193'8199","created":148,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:16:11.777014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:16:11.777014+0000","last_clean_scrub_stamp":"2024-09-10T19:16:11.777014+0000","objects_scrubbed":0,"log_size":3639,"log_dups_size":0,"ondisk_log_size":3639,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":190971904,"num_objects":2914,"num_object_clones":0,"num_object_copies":8742,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2914,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11838,"num_write_kb":472064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.3","version":"104'27","reported_seq":235,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-10T19:18:59.224121+0000","last_change":"2024-09-10T19:16:35.246566+0000","last_active":"2024-09-10T19:18:59.224121+0000","last_peered":"2024-09-10T19:18:59.224121+0000","last_clean":"2024-09-10T19:18:59.224121+0000","last_became_active":"2024-09-10T19:10:09.823003+0000","last_became_peered":"2024-09-10T19:10:09.823003+0000","last_unstale":"2024-09-10T19:18:59.224121+0000","last_undegraded":"2024-09-10T19:18:59.224121+0000","last_fullsized":"2024-09-10T19:18:59.224121+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"104'27","last_scrub_stamp":"2024-09-10T19:16:35.246512+0000","last_deep_scrub":"104'27","last_deep_scrub_stamp":"2024-09-10T19:13:02.225944+0000","last_clean_scrub_stamp":"2024-09-10T19:16:35.246512+0000","objects_scrubbed":8,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0052296490000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":194,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-10T19:18:59.225031+0000","last_change":"2024-09-10T19:16:37.780086+0000","last_active":"2024-09-10T19:18:59.225031+0000","last_peered":"2024-09-10T19:18:59.225031+0000","last_clean":"2024-09-10T19:18:59.225031+0000","last_became_active":"2024-09-10T19:10:07.690325+0000","last_became_peered":"2024-09-10T19:10:07.690325+0000","last_unstale":"2024-09-10T19:18:59.225031+0000","last_undegraded":"2024-09-10T19:18:59.225031+0000","last_fullsized":"2024-09-10T19:18:59.225031+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:16:37.779967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:16:37.779967+0000","last_clean_scrub_stamp":"2024-09-10T19:16:37.779967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.067110102000000005,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":193,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-10T19:18:59.226797+0000","last_change":"2024-09-10T19:17:19.463698+0000","last_active":"2024-09-10T19:18:59.226797+0000","last_peered":"2024-09-10T19:18:59.226797+0000","last_clean":"2024-09-10T19:18:59.226797+0000","last_became_active":"2024-09-10T19:10:11.700370+0000","last_became_peered":"2024-09-10T19:10:11.700370+0000","last_unstale":"2024-09-10T19:18:59.226797+0000","last_undegraded":"2024-09-10T19:18:59.226797+0000","last_fullsized":"2024-09-10T19:18:59.226797+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:15:50.669821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:15:50.669821+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0015765,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":157,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-10T19:18:59.228553+0000","last_change":"2024-09-10T19:17:22.435156+0000","last_active":"2024-09-10T19:18:59.228553+0000","last_peered":"2024-09-10T19:18:59.228553+0000","last_clean":"2024-09-10T19:18:59.228553+0000","last_became_active":"2024-09-10T19:16:24.844699+0000","last_became_peered":"2024-09-10T19:16:24.844699+0000","last_unstale":"2024-09-10T19:18:59.228553+0000","last_undegraded":"2024-09-10T19:18:59.228553+0000","last_fullsized":"2024-09-10T19:18:59.228553+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:15:41.749338+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:15:41.749338+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0018841260000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.e","version":"214'11924","reported_seq":12007,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-09-10T19:19:00.020318+0000","last_change":"2024-09-10T19:16:24.837044+0000","last_active":"2024-09-10T19:19:00.020318+0000","last_peered":"2024-09-10T19:19:00.020318+0000","last_clean":"2024-09-10T19:19:00.020318+0000","last_became_active":"2024-09-10T19:16:24.836458+0000","last_became_peered":"2024-09-10T19:16:24.836458+0000","last_unstale":"2024-09-10T19:19:00.020318+0000","last_undegraded":"2024-09-10T19:19:00.020318+0000","last_fullsized":"2024-09-10T19:19:00.020318+0000","mapping_epoch":154,"log_start":"192'8084","ondisk_log_start":"192'8084","created":148,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:16:11.777014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:16:11.777014+0000","last_clean_scrub_stamp":"2024-09-10T19:16:11.777014+0000","objects_scrubbed":0,"log_size":3840,"log_dups_size":0,"ondisk_log_size":3840,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":192675840,"num_objects":2940,"num_object_clones":0,"num_object_copies":8820,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2940,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11924,"num_write_kb":475648,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"66'10","reported_seq":150,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-10T19:18:59.231789+0000","last_change":"2024-09-10T19:13:54.748881+0000","last_active":"2024-09-10T19:18:59.231789+0000","last_peered":"2024-09-10T19:18:59.231789+0000","last_clean":"2024-09-10T19:18:59.231789+0000","last_became_active":"2024-09-10T19:13:54.714542+0000","last_became_peered":"2024-09-10T19:13:54.714542+0000","last_unstale":"2024-09-10T19:18:59.231789+0000","last_undegraded":"2024-09-10T19:18:59.231789+0000","last_fullsized":"2024-09-10T19:18:59.231789+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'10","last_scrub_stamp":"2024-09-10T19:12:56.360409+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:12:56.360409+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":14,"num_read_kb":9,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,0],"acting":[5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":163,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-10T19:18:59.224147+0000","last_change":"2024-09-10T19:17:28.203449+0000","last_active":"2024-09-10T19:18:59.224147+0000","last_peered":"2024-09-10T19:18:59.224147+0000","last_clean":"2024-09-10T19:18:59.224147+0000","last_became_active":"2024-09-10T19:16:24.836898+0000","last_became_peered":"2024-09-10T19:16:24.836898+0000","last_unstale":"2024-09-10T19:18:59.224147+0000","last_undegraded":"2024-09-10T19:18:59.224147+0000","last_fullsized":"2024-09-10T19:18:59.224147+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:17:28.203397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:17:28.203397+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0011258259999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3","version":"84'9","reported_seq":300,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-10T19:18:59.223968+0000","last_change":"2024-09-10T19:18:14.048096+0000","last_active":"2024-09-10T19:18:59.223968+0000","last_peered":"2024-09-10T19:18:59.223968+0000","last_clean":"2024-09-10T19:18:59.223968+0000","last_became_active":"2024-09-10T19:10:11.699690+0000","last_became_peered":"2024-09-10T19:10:11.699690+0000","last_unstale":"2024-09-10T19:18:59.223968+0000","last_undegraded":"2024-09-10T19:18:59.223968+0000","last_fullsized":"2024-09-10T19:18:59.223968+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'9","last_scrub_stamp":"2024-09-10T19:13:45.236094+0000","last_deep_scrub":"84'9","last_deep_scrub_stamp":"2024-09-10T19:11:53.879272+0000","last_clean_scrub_stamp":"2024-09-10T19:13:45.236094+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.036165602999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":193,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-10T19:18:59.222935+0000","last_change":"2024-09-10T19:18:09.020077+0000","last_active":"2024-09-10T19:18:59.222935+0000","last_peered":"2024-09-10T19:18:59.222935+0000","last_clean":"2024-09-10T19:18:59.222935+0000","last_became_active":"2024-09-10T19:16:24.840096+0000","last_became_peered":"2024-09-10T19:16:24.840096+0000","last_unstale":"2024-09-10T19:18:59.222935+0000","last_undegraded":"2024-09-10T19:18:59.222935+0000","last_fullsized":"2024-09-10T19:18:59.222935+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:13:40.234289+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:13:40.234289+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0086405590000000008,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,5],"acting":[0,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.8","version":"213'8633","reported_seq":8718,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-10T19:18:59.775375+0000","last_change":"2024-09-10T19:17:26.958494+0000","last_active":"2024-09-10T19:18:59.775375+0000","last_peered":"2024-09-10T19:18:59.775375+0000","last_clean":"2024-09-10T19:18:59.775375+0000","last_became_active":"2024-09-10T19:17:26.955617+0000","last_became_peered":"2024-09-10T19:17:26.955617+0000","last_unstale":"2024-09-10T19:18:59.775375+0000","last_undegraded":"2024-09-10T19:18:59.775375+0000","last_fullsized":"2024-09-10T19:18:59.775375+0000","mapping_epoch":148,"log_start":"168'4125","ondisk_log_start":"168'4125","created":148,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:16:11.777014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:16:11.777014+0000","last_clean_scrub_stamp":"2024-09-10T19:16:11.777014+0000","objects_scrubbed":0,"log_size":4508,"log_dups_size":0,"ondisk_log_size":4508,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":93814784,"num_objects":1431,"num_object_clones":0,"num_object_copies":4293,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1431,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5817,"num_write_kb":231968,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"155'31","reported_seq":186,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-10T19:18:59.220016+0000","last_change":"2024-09-10T19:16:24.910865+0000","last_active":"2024-09-10T19:18:59.220016+0000","last_peered":"2024-09-10T19:18:59.220016+0000","last_clean":"2024-09-10T19:18:59.220016+0000","last_became_active":"2024-09-10T19:16:24.858154+0000","last_became_peered":"2024-09-10T19:16:24.858154+0000","last_unstale":"2024-09-10T19:18:59.220016+0000","last_undegraded":"2024-09-10T19:18:59.220016+0000","last_fullsized":"2024-09-10T19:18:59.220016+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"117'30","last_scrub_stamp":"2024-09-10T19:15:49.837844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:15:49.837844+0000","objects_scrubbed":0,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":47,"num_read_kb":32,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":358,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":201,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-10T19:18:59.231525+0000","last_change":"2024-09-10T19:15:54.746979+0000","last_active":"2024-09-10T19:18:59.231525+0000","last_peered":"2024-09-10T19:18:59.231525+0000","last_clean":"2024-09-10T19:18:59.231525+0000","last_became_active":"2024-09-10T19:13:54.730096+0000","last_became_peered":"2024-09-10T19:13:54.730096+0000","last_unstale":"2024-09-10T19:18:59.231525+0000","last_undegraded":"2024-09-10T19:18:59.231525+0000","last_fullsized":"2024-09-10T19:18:59.231525+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:15:54.746920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:15:54.746920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001300037,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":206,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-10T19:18:59.224506+0000","last_change":"2024-09-10T19:17:25.147473+0000","last_active":"2024-09-10T19:18:59.224506+0000","last_peered":"2024-09-10T19:18:59.224506+0000","last_clean":"2024-09-10T19:18:59.224506+0000","last_became_active":"2024-09-10T19:13:54.714017+0000","last_became_peered":"2024-09-10T19:13:54.714017+0000","last_unstale":"2024-09-10T19:18:59.224506+0000","last_undegraded":"2024-09-10T19:18:59.224506+0000","last_fullsized":"2024-09-10T19:18:59.224506+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:17:25.147396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:17:25.147396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.004390054,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,6],"acting":[0,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":178,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-10T19:18:59.221205+0000","last_change":"2024-09-10T19:16:32.818668+0000","last_active":"2024-09-10T19:18:59.221205+0000","last_peered":"2024-09-10T19:18:59.221205+0000","last_clean":"2024-09-10T19:18:59.221205+0000","last_became_active":"2024-09-10T19:10:13.721744+0000","last_became_peered":"2024-09-10T19:10:13.721744+0000","last_unstale":"2024-09-10T19:18:59.221205+0000","last_undegraded":"2024-09-10T19:18:59.221205+0000","last_fullsized":"2024-09-10T19:18:59.221205+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:16:32.818590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:16:32.818590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0047149059999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.b","version":"214'12160","reported_seq":12244,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-09-10T19:19:00.006115+0000","last_change":"2024-09-10T19:16:24.852409+0000","last_active":"2024-09-10T19:19:00.006115+0000","last_peered":"2024-09-10T19:19:00.006115+0000","last_clean":"2024-09-10T19:19:00.006115+0000","last_became_active":"2024-09-10T19:16:24.850682+0000","last_became_peered":"2024-09-10T19:16:24.850682+0000","last_unstale":"2024-09-10T19:19:00.006115+0000","last_undegraded":"2024-09-10T19:19:00.006115+0000","last_fullsized":"2024-09-10T19:19:00.006115+0000","mapping_epoch":154,"log_start":"194'8588","ondisk_log_start":"194'8588","created":148,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:16:11.777014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:16:11.777014+0000","last_clean_scrub_stamp":"2024-09-10T19:16:11.777014+0000","objects_scrubbed":0,"log_size":3572,"log_dups_size":0,"ondisk_log_size":3572,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":190316544,"num_objects":2904,"num_object_clones":0,"num_object_copies":8712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2904,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12160,"num_write_kb":482048,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.7","version":"82'18","reported_seq":217,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-10T19:18:59.236307+0000","last_change":"2024-09-10T19:15:55.777428+0000","last_active":"2024-09-10T19:18:59.236307+0000","last_peered":"2024-09-10T19:18:59.236307+0000","last_clean":"2024-09-10T19:18:59.236307+0000","last_became_active":"2024-09-10T19:13:54.712065+0000","last_became_peered":"2024-09-10T19:13:54.712065+0000","last_unstale":"2024-09-10T19:18:59.236307+0000","last_undegraded":"2024-09-10T19:18:59.236307+0000","last_fullsized":"2024-09-10T19:18:59.236307+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'18","last_scrub_stamp":"2024-09-10T19:15:55.777361+0000","last_deep_scrub":"82'18","last_deep_scrub_stamp":"2024-09-10T19:15:55.777361+0000","last_clean_scrub_stamp":"2024-09-10T19:15:55.777361+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0023560040000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":19,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.0","version":"159'6","reported_seq":270,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-10T19:18:59.226534+0000","last_change":"2024-09-10T19:16:32.650007+0000","last_active":"2024-09-10T19:18:59.226534+0000","last_peered":"2024-09-10T19:18:59.226534+0000","last_clean":"2024-09-10T19:18:59.226534+0000","last_became_active":"2024-09-10T19:16:24.846203+0000","last_became_peered":"2024-09-10T19:16:24.846203+0000","last_unstale":"2024-09-10T19:18:59.226534+0000","last_undegraded":"2024-09-10T19:18:59.226534+0000","last_fullsized":"2024-09-10T19:18:59.226534+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"159'6","last_scrub_stamp":"2024-09-10T19:16:32.649914+0000","last_deep_scrub":"159'6","last_deep_scrub_stamp":"2024-09-10T19:16:32.649914+0000","last_clean_scrub_stamp":"2024-09-10T19:16:32.649914+0000","objects_scrubbed":1,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.013939817,"objects_trimmed":0,"snaptrim_duration":0.00034454499999999997,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":161,"num_read_kb":156,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":389,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":82,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-10T19:18:59.221059+0000","last_change":"2024-09-10T19:16:33.811962+0000","last_active":"2024-09-10T19:18:59.221059+0000","last_peered":"2024-09-10T19:18:59.221059+0000","last_clean":"2024-09-10T19:18:59.221059+0000","last_became_active":"2024-09-10T19:16:24.857505+0000","last_became_peered":"2024-09-10T19:16:24.857505+0000","last_unstale":"2024-09-10T19:18:59.221059+0000","last_undegraded":"2024-09-10T19:18:59.221059+0000","last_fullsized":"2024-09-10T19:18:59.221059+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:16:33.811869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:16:33.811869+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0027920340000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":82,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-10T19:18:59.236345+0000","last_change":"2024-09-10T19:16:43.958311+0000","last_active":"2024-09-10T19:18:59.236345+0000","last_peered":"2024-09-10T19:18:59.236345+0000","last_clean":"2024-09-10T19:18:59.236345+0000","last_became_active":"2024-09-10T19:16:24.861481+0000","last_became_peered":"2024-09-10T19:16:24.861481+0000","last_unstale":"2024-09-10T19:18:59.236345+0000","last_undegraded":"2024-09-10T19:18:59.236345+0000","last_fullsized":"2024-09-10T19:18:59.236345+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:16:43.958224+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:16:43.958224+0000","last_clean_scrub_stamp":"2024-09-10T19:16:43.958224+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.015970275999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,4],"acting":[5,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":181,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-10T19:18:59.224645+0000","last_change":"2024-09-10T19:15:38.493179+0000","last_active":"2024-09-10T19:18:59.224645+0000","last_peered":"2024-09-10T19:18:59.224645+0000","last_clean":"2024-09-10T19:18:59.224645+0000","last_became_active":"2024-09-10T19:10:14.053697+0000","last_became_peered":"2024-09-10T19:10:14.053697+0000","last_unstale":"2024-09-10T19:18:59.224645+0000","last_undegraded":"2024-09-10T19:18:59.224645+0000","last_fullsized":"2024-09-10T19:18:59.224645+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:15:38.493102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:15:38.493102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0015544280000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.a","version":"214'12275","reported_seq":12346,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-09-10T19:18:59.869218+0000","last_change":"2024-09-10T19:16:24.865995+0000","last_active":"2024-09-10T19:18:59.869218+0000","last_peered":"2024-09-10T19:18:59.869218+0000","last_clean":"2024-09-10T19:18:59.869218+0000","last_became_active":"2024-09-10T19:16:24.864959+0000","last_became_peered":"2024-09-10T19:16:24.864959+0000","last_unstale":"2024-09-10T19:18:59.869218+0000","last_undegraded":"2024-09-10T19:18:59.869218+0000","last_fullsized":"2024-09-10T19:18:59.869218+0000","mapping_epoch":154,"log_start":"192'8425","ondisk_log_start":"192'8425","created":148,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:16:11.777014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:16:11.777014+0000","last_clean_scrub_stamp":"2024-09-10T19:16:11.777014+0000","objects_scrubbed":0,"log_size":3850,"log_dups_size":0,"ondisk_log_size":3850,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":204144640,"num_objects":3115,"num_object_clones":0,"num_object_copies":9345,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3115,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12275,"num_write_kb":492480,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.6","version":"66'12","reported_seq":260,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-10T19:18:59.225817+0000","last_change":"2024-09-10T19:18:24.968589+0000","last_active":"2024-09-10T19:18:59.225817+0000","last_peered":"2024-09-10T19:18:59.225817+0000","last_clean":"2024-09-10T19:18:59.225817+0000","last_became_active":"2024-09-10T19:16:24.844323+0000","last_became_peered":"2024-09-10T19:16:24.844323+0000","last_unstale":"2024-09-10T19:18:59.225817+0000","last_undegraded":"2024-09-10T19:18:59.225817+0000","last_fullsized":"2024-09-10T19:18:59.225817+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'12","last_scrub_stamp":"2024-09-10T19:15:40.903279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:15:40.903279+0000","objects_scrubbed":6,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0271238,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,5],"acting":[0,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"61'1","reported_seq":199,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-10T19:18:59.223681+0000","last_change":"2024-09-10T19:16:42.195540+0000","last_active":"2024-09-10T19:18:59.223681+0000","last_peered":"2024-09-10T19:18:59.223681+0000","last_clean":"2024-09-10T19:18:59.223681+0000","last_became_active":"2024-09-10T19:10:07.683259+0000","last_became_peered":"2024-09-10T19:10:07.683259+0000","last_unstale":"2024-09-10T19:18:59.223681+0000","last_undegraded":"2024-09-10T19:18:59.223681+0000","last_fullsized":"2024-09-10T19:18:59.223681+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'1","last_scrub_stamp":"2024-09-10T19:16:42.195463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:16:42.195463+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0016622189999999999,"objects_trimmed":0,"snaptrim_duration":0.0014341130000000001,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":201,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-10T19:18:59.227918+0000","last_change":"2024-09-10T19:15:30.542477+0000","last_active":"2024-09-10T19:18:59.227918+0000","last_peered":"2024-09-10T19:18:59.227918+0000","last_clean":"2024-09-10T19:18:59.227918+0000","last_became_active":"2024-09-10T19:13:54.721486+0000","last_became_peered":"2024-09-10T19:13:54.721486+0000","last_unstale":"2024-09-10T19:18:59.227918+0000","last_undegraded":"2024-09-10T19:18:59.227918+0000","last_fullsized":"2024-09-10T19:18:59.227918+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:15:30.542399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:15:30.542399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0025142609999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":207,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-10T19:18:59.236221+0000","last_change":"2024-09-10T19:17:28.070635+0000","last_active":"2024-09-10T19:18:59.236221+0000","last_peered":"2024-09-10T19:18:59.236221+0000","last_clean":"2024-09-10T19:18:59.236221+0000","last_became_active":"2024-09-10T19:13:54.711685+0000","last_became_peered":"2024-09-10T19:13:54.711685+0000","last_unstale":"2024-09-10T19:18:59.236221+0000","last_undegraded":"2024-09-10T19:18:59.236221+0000","last_fullsized":"2024-09-10T19:18:59.236221+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:17:28.070543+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:17:28.070543+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0017508000000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,4],"acting":[5,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":144,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-10T19:18:59.232730+0000","last_change":"2024-09-10T19:16:28.786490+0000","last_active":"2024-09-10T19:18:59.232730+0000","last_peered":"2024-09-10T19:18:59.232730+0000","last_clean":"2024-09-10T19:18:59.232730+0000","last_became_active":"2024-09-10T19:13:54.722937+0000","last_became_peered":"2024-09-10T19:13:54.722937+0000","last_unstale":"2024-09-10T19:18:59.232730+0000","last_undegraded":"2024-09-10T19:18:59.232730+0000","last_fullsized":"2024-09-10T19:18:59.232730+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:16:28.786056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:16:28.786056+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0082035739999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.9","version":"213'8930","reported_seq":9015,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-10T19:18:59.738877+0000","last_change":"2024-09-10T19:17:26.964413+0000","last_active":"2024-09-10T19:18:59.738877+0000","last_peered":"2024-09-10T19:18:59.738877+0000","last_clean":"2024-09-10T19:18:59.738877+0000","last_became_active":"2024-09-10T19:17:26.961919+0000","last_became_peered":"2024-09-10T19:17:26.961919+0000","last_unstale":"2024-09-10T19:18:59.738877+0000","last_undegraded":"2024-09-10T19:18:59.738877+0000","last_fullsized":"2024-09-10T19:18:59.738877+0000","mapping_epoch":148,"log_start":"171'4842","ondisk_log_start":"171'4842","created":148,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:16:11.777014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:16:11.777014+0000","last_clean_scrub_stamp":"2024-09-10T19:16:11.777014+0000","objects_scrubbed":0,"log_size":4088,"log_dups_size":0,"ondisk_log_size":4088,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":100007936,"num_objects":1526,"num_object_clones":0,"num_object_copies":4578,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1526,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6038,"num_write_kb":242048,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,0],"acting":[6,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"80'21","reported_seq":231,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-10T19:18:59.227493+0000","last_change":"2024-09-10T19:17:27.420321+0000","last_active":"2024-09-10T19:18:59.227493+0000","last_peered":"2024-09-10T19:18:59.227493+0000","last_clean":"2024-09-10T19:18:59.227493+0000","last_became_active":"2024-09-10T19:10:09.822184+0000","last_became_peered":"2024-09-10T19:10:09.822184+0000","last_unstale":"2024-09-10T19:18:59.227493+0000","last_undegraded":"2024-09-10T19:18:59.227493+0000","last_fullsized":"2024-09-10T19:18:59.227493+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"80'21","last_scrub_stamp":"2024-09-10T19:17:27.420267+0000","last_deep_scrub":"80'21","last_deep_scrub_stamp":"2024-09-10T19:11:39.836144+0000","last_clean_scrub_stamp":"2024-09-10T19:17:27.420267+0000","objects_scrubbed":7,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0018427859999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"63'2","reported_seq":201,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-10T19:18:59.220528+0000","last_change":"2024-09-10T19:16:35.962952+0000","last_active":"2024-09-10T19:18:59.220528+0000","last_peered":"2024-09-10T19:18:59.220528+0000","last_clean":"2024-09-10T19:18:59.220528+0000","last_became_active":"2024-09-10T19:10:07.684784+0000","last_became_peered":"2024-09-10T19:10:07.684784+0000","last_unstale":"2024-09-10T19:18:59.220528+0000","last_undegraded":"2024-09-10T19:18:59.220528+0000","last_fullsized":"2024-09-10T19:18:59.220528+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'2","last_scrub_stamp":"2024-09-10T19:16:35.962804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:16:35.962804+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.105395924,"objects_trimmed":0,"snaptrim_duration":0.00075522999999999996,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":204,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-10T19:18:59.232141+0000","last_change":"2024-09-10T19:16:24.858963+0000","last_active":"2024-09-10T19:18:59.232141+0000","last_peered":"2024-09-10T19:18:59.232141+0000","last_clean":"2024-09-10T19:18:59.232141+0000","last_became_active":"2024-09-10T19:16:24.858662+0000","last_became_peered":"2024-09-10T19:16:24.858662+0000","last_unstale":"2024-09-10T19:18:59.232141+0000","last_undegraded":"2024-09-10T19:18:59.232141+0000","last_fullsized":"2024-09-10T19:18:59.232141+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:13:50.411500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:13:50.411500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0032604869999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"74'79","reported_seq":229,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-10T19:18:59.228072+0000","last_change":"2024-09-10T19:13:46.419002+0000","last_active":"2024-09-10T19:18:59.228072+0000","last_peered":"2024-09-10T19:18:59.228072+0000","last_clean":"2024-09-10T19:18:59.228072+0000","last_became_active":"2024-09-10T19:09:56.634919+0000","last_became_peered":"2024-09-10T19:09:56.634919+0000","last_unstale":"2024-09-10T19:18:59.228072+0000","last_undegraded":"2024-09-10T19:18:59.228072+0000","last_fullsized":"2024-09-10T19:18:59.228072+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'79","last_scrub_stamp":"2024-09-10T19:13:46.418921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:08:00.699315+0000","last_clean_scrub_stamp":"2024-09-10T19:13:46.418921+0000","objects_scrubbed":2,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0029827999999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":190,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-10T19:18:59.227614+0000","last_change":"2024-09-10T19:16:24.852723+0000","last_active":"2024-09-10T19:18:59.227614+0000","last_peered":"2024-09-10T19:18:59.227614+0000","last_clean":"2024-09-10T19:18:59.227614+0000","last_became_active":"2024-09-10T19:16:24.851652+0000","last_became_peered":"2024-09-10T19:16:24.851652+0000","last_unstale":"2024-09-10T19:18:59.227614+0000","last_undegraded":"2024-09-10T19:18:59.227614+0000","last_fullsized":"2024-09-10T19:18:59.227614+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:13:47.453228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:13:47.453228+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.010282463,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,0],"acting":[7,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":178,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-10T19:18:59.236387+0000","last_change":"2024-09-10T19:13:05.390398+0000","last_active":"2024-09-10T19:18:59.236387+0000","last_peered":"2024-09-10T19:18:59.236387+0000","last_clean":"2024-09-10T19:18:59.236387+0000","last_became_active":"2024-09-10T19:10:13.720647+0000","last_became_peered":"2024-09-10T19:10:13.720647+0000","last_unstale":"2024-09-10T19:18:59.236387+0000","last_undegraded":"2024-09-10T19:18:59.236387+0000","last_fullsized":"2024-09-10T19:18:59.236387+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:13:05.390317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:13:05.390317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0058030640000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.6","version":"214'8892","reported_seq":8974,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-09-10T19:18:59.996043+0000","last_change":"2024-09-10T19:17:26.955780+0000","last_active":"2024-09-10T19:18:59.996043+0000","last_peered":"2024-09-10T19:18:59.996043+0000","last_clean":"2024-09-10T19:18:59.996043+0000","last_became_active":"2024-09-10T19:17:26.953133+0000","last_became_peered":"2024-09-10T19:17:26.953133+0000","last_unstale":"2024-09-10T19:18:59.996043+0000","last_undegraded":"2024-09-10T19:18:59.996043+0000","last_fullsized":"2024-09-10T19:18:59.996043+0000","mapping_epoch":148,"log_start":"169'4426","ondisk_log_start":"169'4426","created":148,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:16:11.777014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:16:11.777014+0000","last_clean_scrub_stamp":"2024-09-10T19:16:11.777014+0000","objects_scrubbed":0,"log_size":4466,"log_dups_size":0,"ondisk_log_size":4466,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":102072320,"num_objects":1557,"num_object_clones":0,"num_object_copies":4671,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1557,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6019,"num_write_kb":242464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,7],"acting":[0,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"66'19","reported_seq":256,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-10T19:18:59.227099+0000","last_change":"2024-09-10T19:18:27.334014+0000","last_active":"2024-09-10T19:18:59.227099+0000","last_peered":"2024-09-10T19:18:59.227099+0000","last_clean":"2024-09-10T19:18:59.227099+0000","last_became_active":"2024-09-10T19:13:54.731274+0000","last_became_peered":"2024-09-10T19:13:54.731274+0000","last_unstale":"2024-09-10T19:18:59.227099+0000","last_undegraded":"2024-09-10T19:18:59.227099+0000","last_fullsized":"2024-09-10T19:18:59.227099+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'19","last_scrub_stamp":"2024-09-10T19:15:38.722581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:15:38.722581+0000","objects_scrubbed":9,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0054904940000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,5],"acting":[6,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":72,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-10T19:18:59.233112+0000","last_change":"2024-09-10T19:16:24.847051+0000","last_active":"2024-09-10T19:18:59.233112+0000","last_peered":"2024-09-10T19:18:59.233112+0000","last_clean":"2024-09-10T19:18:59.233112+0000","last_became_active":"2024-09-10T19:16:24.846745+0000","last_became_peered":"2024-09-10T19:16:24.846745+0000","last_unstale":"2024-09-10T19:18:59.233112+0000","last_undegraded":"2024-09-10T19:18:59.233112+0000","last_fullsized":"2024-09-10T19:18:59.233112+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:15:53.803057+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:15:53.803057+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3],"acting":[5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":81,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-10T19:18:59.223836+0000","last_change":"2024-09-10T19:16:27.240222+0000","last_active":"2024-09-10T19:18:59.223836+0000","last_peered":"2024-09-10T19:18:59.223836+0000","last_clean":"2024-09-10T19:18:59.223836+0000","last_became_active":"2024-09-10T19:16:24.844295+0000","last_became_peered":"2024-09-10T19:16:24.844295+0000","last_unstale":"2024-09-10T19:18:59.223836+0000","last_undegraded":"2024-09-10T19:18:59.223836+0000","last_fullsized":"2024-09-10T19:18:59.223836+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:16:27.240139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:16:27.240139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0020507199999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":188,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-10T19:18:59.227995+0000","last_change":"2024-09-10T19:16:24.858456+0000","last_active":"2024-09-10T19:18:59.227995+0000","last_peered":"2024-09-10T19:18:59.227995+0000","last_clean":"2024-09-10T19:18:59.227995+0000","last_became_active":"2024-09-10T19:16:24.858172+0000","last_became_peered":"2024-09-10T19:16:24.858172+0000","last_unstale":"2024-09-10T19:18:59.227995+0000","last_undegraded":"2024-09-10T19:18:59.227995+0000","last_fullsized":"2024-09-10T19:18:59.227995+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:13:14.311620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:13:14.311620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0054101059999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,6],"acting":[7,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.7","version":"214'8879","reported_seq":8973,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-09-10T19:19:00.017567+0000","last_change":"2024-09-10T19:17:26.944772+0000","last_active":"2024-09-10T19:19:00.017567+0000","last_peered":"2024-09-10T19:19:00.017567+0000","last_clean":"2024-09-10T19:19:00.017567+0000","last_became_active":"2024-09-10T19:17:26.942858+0000","last_became_peered":"2024-09-10T19:17:26.942858+0000","last_unstale":"2024-09-10T19:19:00.017567+0000","last_undegraded":"2024-09-10T19:19:00.017567+0000","last_fullsized":"2024-09-10T19:19:00.017567+0000","mapping_epoch":154,"log_start":"171'4786","ondisk_log_start":"171'4786","created":148,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:16:11.777014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:16:11.777014+0000","last_clean_scrub_stamp":"2024-09-10T19:16:11.777014+0000","objects_scrubbed":0,"log_size":4093,"log_dups_size":0,"ondisk_log_size":4093,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":99385344,"num_objects":1516,"num_object_clones":0,"num_object_copies":4548,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1516,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5980,"num_write_kb":239904,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,4],"acting":[3,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.b","version":"66'9","reported_seq":230,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-10T19:18:59.224049+0000","last_change":"2024-09-10T19:18:26.964644+0000","last_active":"2024-09-10T19:18:59.224049+0000","last_peered":"2024-09-10T19:18:59.224049+0000","last_clean":"2024-09-10T19:18:59.224049+0000","last_became_active":"2024-09-10T19:13:54.710613+0000","last_became_peered":"2024-09-10T19:13:54.710613+0000","last_unstale":"2024-09-10T19:18:59.224049+0000","last_undegraded":"2024-09-10T19:18:59.224049+0000","last_fullsized":"2024-09-10T19:18:59.224049+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'9","last_scrub_stamp":"2024-09-10T19:15:37.846807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:15:37.846807+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.012189228,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":157,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-10T19:18:59.220353+0000","last_change":"2024-09-10T19:16:24.848979+0000","last_active":"2024-09-10T19:18:59.220353+0000","last_peered":"2024-09-10T19:18:59.220353+0000","last_clean":"2024-09-10T19:18:59.220353+0000","last_became_active":"2024-09-10T19:16:24.848668+0000","last_became_peered":"2024-09-10T19:16:24.848668+0000","last_unstale":"2024-09-10T19:18:59.220353+0000","last_undegraded":"2024-09-10T19:18:59.220353+0000","last_fullsized":"2024-09-10T19:18:59.220353+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:15:50.850576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:11:44.254513+0000","last_clean_scrub_stamp":"2024-09-10T19:15:50.850576+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001815419,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":75,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-10T19:18:59.223221+0000","last_change":"2024-09-10T19:17:09.811398+0000","last_active":"2024-09-10T19:18:59.223221+0000","last_peered":"2024-09-10T19:18:59.223221+0000","last_clean":"2024-09-10T19:18:59.223221+0000","last_became_active":"2024-09-10T19:16:24.845709+0000","last_became_peered":"2024-09-10T19:16:24.845709+0000","last_unstale":"2024-09-10T19:18:59.223221+0000","last_undegraded":"2024-09-10T19:18:59.223221+0000","last_fullsized":"2024-09-10T19:18:59.223221+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:13:08.398036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:13:08.398036+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":197,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-10T19:18:59.223661+0000","last_change":"2024-09-10T19:16:39.188737+0000","last_active":"2024-09-10T19:18:59.223661+0000","last_peered":"2024-09-10T19:18:59.223661+0000","last_clean":"2024-09-10T19:18:59.223661+0000","last_became_active":"2024-09-10T19:16:24.837483+0000","last_became_peered":"2024-09-10T19:16:24.837483+0000","last_unstale":"2024-09-10T19:18:59.223661+0000","last_undegraded":"2024-09-10T19:18:59.223661+0000","last_fullsized":"2024-09-10T19:18:59.223661+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:16:39.188682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:16:39.188682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0012633589999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.4","version":"213'8758","reported_seq":8856,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-10T19:18:59.792068+0000","last_change":"2024-09-10T19:17:26.998151+0000","last_active":"2024-09-10T19:18:59.792068+0000","last_peered":"2024-09-10T19:18:59.792068+0000","last_clean":"2024-09-10T19:18:59.792068+0000","last_became_active":"2024-09-10T19:17:26.972029+0000","last_became_peered":"2024-09-10T19:17:26.972029+0000","last_unstale":"2024-09-10T19:18:59.792068+0000","last_undegraded":"2024-09-10T19:18:59.792068+0000","last_fullsized":"2024-09-10T19:18:59.792068+0000","mapping_epoch":154,"log_start":"168'4266","ondisk_log_start":"168'4266","created":148,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:16:11.777014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:16:11.777014+0000","last_clean_scrub_stamp":"2024-09-10T19:16:11.777014+0000","objects_scrubbed":0,"log_size":4492,"log_dups_size":0,"ondisk_log_size":4492,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":97746944,"num_objects":1491,"num_object_clones":0,"num_object_copies":4473,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1491,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5895,"num_write_kb":236384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.8","version":"66'15","reported_seq":207,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-10T19:18:59.231925+0000","last_change":"2024-09-10T19:12:50.449684+0000","last_active":"2024-09-10T19:18:59.231925+0000","last_peered":"2024-09-10T19:18:59.231925+0000","last_clean":"2024-09-10T19:18:59.231925+0000","last_became_active":"2024-09-10T19:10:09.820950+0000","last_became_peered":"2024-09-10T19:10:09.820950+0000","last_unstale":"2024-09-10T19:18:59.231925+0000","last_undegraded":"2024-09-10T19:18:59.231925+0000","last_fullsized":"2024-09-10T19:18:59.231925+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'15","last_scrub_stamp":"2024-09-10T19:12:50.449596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:12:50.449596+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0055729239999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":72,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-10T19:18:59.221288+0000","last_change":"2024-09-10T19:16:24.857533+0000","last_active":"2024-09-10T19:18:59.221288+0000","last_peered":"2024-09-10T19:18:59.221288+0000","last_clean":"2024-09-10T19:18:59.221288+0000","last_became_active":"2024-09-10T19:16:24.857253+0000","last_became_peered":"2024-09-10T19:16:24.857253+0000","last_unstale":"2024-09-10T19:18:59.221288+0000","last_undegraded":"2024-09-10T19:18:59.221288+0000","last_fullsized":"2024-09-10T19:18:59.221288+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:15:38.677960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:15:38.677960+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.9","version":"80'9","reported_seq":295,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-10T19:18:59.227156+0000","last_change":"2024-09-10T19:13:43.358984+0000","last_active":"2024-09-10T19:18:59.227156+0000","last_peered":"2024-09-10T19:18:59.227156+0000","last_clean":"2024-09-10T19:18:59.227156+0000","last_became_active":"2024-09-10T19:10:11.702878+0000","last_became_peered":"2024-09-10T19:10:11.702878+0000","last_unstale":"2024-09-10T19:18:59.227156+0000","last_undegraded":"2024-09-10T19:18:59.227156+0000","last_fullsized":"2024-09-10T19:18:59.227156+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"80'9","last_scrub_stamp":"2024-09-10T19:13:43.358901+0000","last_deep_scrub":"80'9","last_deep_scrub_stamp":"2024-09-10T19:11:35.174218+0000","last_clean_scrub_stamp":"2024-09-10T19:13:43.358901+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0066143260000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":178,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-10T19:18:59.232111+0000","last_change":"2024-09-10T19:12:57.392898+0000","last_active":"2024-09-10T19:18:59.232111+0000","last_peered":"2024-09-10T19:18:59.232111+0000","last_clean":"2024-09-10T19:18:59.232111+0000","last_became_active":"2024-09-10T19:10:14.053875+0000","last_became_peered":"2024-09-10T19:10:14.053875+0000","last_unstale":"2024-09-10T19:18:59.232111+0000","last_undegraded":"2024-09-10T19:18:59.232111+0000","last_fullsized":"2024-09-10T19:18:59.232111+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:12:57.392844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:12:57.392844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0046180839999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.5","version":"213'8998","reported_seq":9082,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-09-10T19:18:59.799328+0000","last_change":"2024-09-10T19:17:26.973165+0000","last_active":"2024-09-10T19:18:59.799328+0000","last_peered":"2024-09-10T19:18:59.799328+0000","last_clean":"2024-09-10T19:18:59.799328+0000","last_became_active":"2024-09-10T19:17:26.966286+0000","last_became_peered":"2024-09-10T19:17:26.966286+0000","last_unstale":"2024-09-10T19:18:59.799328+0000","last_undegraded":"2024-09-10T19:18:59.799328+0000","last_fullsized":"2024-09-10T19:18:59.799328+0000","mapping_epoch":148,"log_start":"171'4958","ondisk_log_start":"171'4958","created":148,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:16:11.777014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:16:11.777014+0000","last_clean_scrub_stamp":"2024-09-10T19:16:11.777014+0000","objects_scrubbed":0,"log_size":4040,"log_dups_size":0,"ondisk_log_size":4040,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":98828288,"num_objects":1508,"num_object_clones":0,"num_object_copies":4524,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1508,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6074,"num_write_kb":242624,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.9","version":"66'12","reported_seq":236,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-10T19:18:59.219617+0000","last_change":"2024-09-10T19:15:37.869593+0000","last_active":"2024-09-10T19:18:59.219617+0000","last_peered":"2024-09-10T19:18:59.219617+0000","last_clean":"2024-09-10T19:18:59.219617+0000","last_became_active":"2024-09-10T19:13:54.718152+0000","last_became_peered":"2024-09-10T19:13:54.718152+0000","last_unstale":"2024-09-10T19:18:59.219617+0000","last_undegraded":"2024-09-10T19:18:59.219617+0000","last_fullsized":"2024-09-10T19:18:59.219617+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'12","last_scrub_stamp":"2024-09-10T19:15:37.869451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:15:37.869451+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0025441249999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.f","version":"59'2","reported_seq":209,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-10T19:18:59.222563+0000","last_change":"2024-09-10T19:16:29.007627+0000","last_active":"2024-09-10T19:18:59.222563+0000","last_peered":"2024-09-10T19:18:59.222563+0000","last_clean":"2024-09-10T19:18:59.222563+0000","last_became_active":"2024-09-10T19:10:07.681719+0000","last_became_peered":"2024-09-10T19:10:07.681719+0000","last_unstale":"2024-09-10T19:18:59.222563+0000","last_undegraded":"2024-09-10T19:18:59.222563+0000","last_fullsized":"2024-09-10T19:18:59.222563+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'2","last_scrub_stamp":"2024-09-10T19:16:29.007251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:16:29.007251+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0079698319999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":14,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":157,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-10T19:18:59.222983+0000","last_change":"2024-09-10T19:18:29.009145+0000","last_active":"2024-09-10T19:18:59.222983+0000","last_peered":"2024-09-10T19:18:59.222983+0000","last_clean":"2024-09-10T19:18:59.222983+0000","last_became_active":"2024-09-10T19:16:24.840091+0000","last_became_peered":"2024-09-10T19:16:24.840091+0000","last_unstale":"2024-09-10T19:18:59.222983+0000","last_undegraded":"2024-09-10T19:18:59.222983+0000","last_fullsized":"2024-09-10T19:18:59.222983+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:15:31.776481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:11:46.587846+0000","last_clean_scrub_stamp":"2024-09-10T19:15:31.776481+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001267142,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,4],"acting":[0,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":195,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-10T19:18:59.219855+0000","last_change":"2024-09-10T19:15:41.093923+0000","last_active":"2024-09-10T19:18:59.219855+0000","last_peered":"2024-09-10T19:18:59.219855+0000","last_clean":"2024-09-10T19:18:59.219855+0000","last_became_active":"2024-09-10T19:13:54.711804+0000","last_became_peered":"2024-09-10T19:13:54.711804+0000","last_unstale":"2024-09-10T19:18:59.219855+0000","last_undegraded":"2024-09-10T19:18:59.219855+0000","last_fullsized":"2024-09-10T19:18:59.219855+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:15:41.093783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:15:41.093783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0023833050000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.16","version":"66'9","reported_seq":213,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-10T19:18:59.225813+0000","last_change":"2024-09-10T19:18:11.047270+0000","last_active":"2024-09-10T19:18:59.225813+0000","last_peered":"2024-09-10T19:18:59.225813+0000","last_clean":"2024-09-10T19:18:59.225813+0000","last_became_active":"2024-09-10T19:10:09.822994+0000","last_became_peered":"2024-09-10T19:10:09.822994+0000","last_unstale":"2024-09-10T19:18:59.225813+0000","last_undegraded":"2024-09-10T19:18:59.225813+0000","last_fullsized":"2024-09-10T19:18:59.225813+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'9","last_scrub_stamp":"2024-09-10T19:13:50.172825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:13:50.172825+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0094381619999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":165,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-10T19:18:59.223277+0000","last_change":"2024-09-10T19:17:16.305585+0000","last_active":"2024-09-10T19:18:59.223277+0000","last_peered":"2024-09-10T19:18:59.223277+0000","last_clean":"2024-09-10T19:18:59.223277+0000","last_became_active":"2024-09-10T19:16:24.844313+0000","last_became_peered":"2024-09-10T19:16:24.844313+0000","last_unstale":"2024-09-10T19:18:59.223277+0000","last_undegraded":"2024-09-10T19:18:59.223277+0000","last_fullsized":"2024-09-10T19:18:59.223277+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:17:16.305525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:11:41.458523+0000","last_clean_scrub_stamp":"2024-09-10T19:17:16.305525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0070771209999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,6],"acting":[0,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":197,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-10T19:18:59.222007+0000","last_change":"2024-09-10T19:16:28.794727+0000","last_active":"2024-09-10T19:18:59.222007+0000","last_peered":"2024-09-10T19:18:59.222007+0000","last_clean":"2024-09-10T19:18:59.222007+0000","last_became_active":"2024-09-10T19:13:54.712275+0000","last_became_peered":"2024-09-10T19:13:54.712275+0000","last_unstale":"2024-09-10T19:18:59.222007+0000","last_undegraded":"2024-09-10T19:18:59.222007+0000","last_fullsized":"2024-09-10T19:18:59.222007+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:16:28.794649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:16:28.794649+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0027500379999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"66'1","reported_seq":85,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-10T19:18:59.223518+0000","last_change":"2024-09-10T19:16:24.862865+0000","last_active":"2024-09-10T19:18:59.223518+0000","last_peered":"2024-09-10T19:18:59.223518+0000","last_clean":"2024-09-10T19:18:59.223518+0000","last_became_active":"2024-09-10T19:16:24.837432+0000","last_became_peered":"2024-09-10T19:16:24.837432+0000","last_unstale":"2024-09-10T19:18:59.223518+0000","last_undegraded":"2024-09-10T19:18:59.223518+0000","last_fullsized":"2024-09-10T19:18:59.223518+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'1","last_scrub_stamp":"2024-09-10T19:15:48.839582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:15:48.839582+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":13,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":207,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-10T19:18:59.228730+0000","last_change":"2024-09-10T19:17:17.479805+0000","last_active":"2024-09-10T19:18:59.228730+0000","last_peered":"2024-09-10T19:18:59.228730+0000","last_clean":"2024-09-10T19:18:59.228730+0000","last_became_active":"2024-09-10T19:13:54.724027+0000","last_became_peered":"2024-09-10T19:13:54.724027+0000","last_unstale":"2024-09-10T19:18:59.228730+0000","last_undegraded":"2024-09-10T19:18:59.228730+0000","last_fullsized":"2024-09-10T19:18:59.228730+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:15:36.714327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:15:36.714327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001831287,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,5],"acting":[6,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"66'6","reported_seq":224,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-10T19:18:59.221675+0000","last_change":"2024-09-10T19:16:24.891057+0000","last_active":"2024-09-10T19:18:59.221675+0000","last_peered":"2024-09-10T19:18:59.221675+0000","last_clean":"2024-09-10T19:18:59.221675+0000","last_became_active":"2024-09-10T19:16:24.863398+0000","last_became_peered":"2024-09-10T19:16:24.863398+0000","last_unstale":"2024-09-10T19:18:59.221675+0000","last_undegraded":"2024-09-10T19:18:59.221675+0000","last_fullsized":"2024-09-10T19:18:59.221675+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'6","last_scrub_stamp":"2024-09-10T19:13:55.370650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:13:55.370650+0000","objects_scrubbed":3,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.071693797000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":197,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-10T19:18:59.236551+0000","last_change":"2024-09-10T19:15:30.583958+0000","last_active":"2024-09-10T19:18:59.236551+0000","last_peered":"2024-09-10T19:18:59.236551+0000","last_clean":"2024-09-10T19:18:59.236551+0000","last_became_active":"2024-09-10T19:13:54.729917+0000","last_became_peered":"2024-09-10T19:13:54.729917+0000","last_unstale":"2024-09-10T19:18:59.236551+0000","last_undegraded":"2024-09-10T19:18:59.236551+0000","last_fullsized":"2024-09-10T19:18:59.236551+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:15:30.583878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:11:23.264793+0000","last_clean_scrub_stamp":"2024-09-10T19:15:30.583878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0015146960000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":188,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-10T19:18:59.224502+0000","last_change":"2024-09-10T19:17:26.922378+0000","last_active":"2024-09-10T19:18:59.224502+0000","last_peered":"2024-09-10T19:18:59.224502+0000","last_clean":"2024-09-10T19:18:59.224502+0000","last_became_active":"2024-09-10T19:10:14.054360+0000","last_became_peered":"2024-09-10T19:10:14.054360+0000","last_unstale":"2024-09-10T19:18:59.224502+0000","last_undegraded":"2024-09-10T19:18:59.224502+0000","last_fullsized":"2024-09-10T19:18:59.224502+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:17:26.922305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:17:26.922305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0016181940000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.18","version":"213'8660","reported_seq":14907,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-10T19:18:59.790707+0000","last_change":"2024-09-10T19:17:45.681866+0000","last_active":"2024-09-10T19:18:59.790707+0000","last_peered":"2024-09-10T19:18:59.790707+0000","last_clean":"2024-09-10T19:18:59.790707+0000","last_became_active":"2024-09-10T19:17:45.677163+0000","last_became_peered":"2024-09-10T19:17:45.677163+0000","last_unstale":"2024-09-10T19:18:59.790707+0000","last_undegraded":"2024-09-10T19:18:59.790707+0000","last_fullsized":"2024-09-10T19:18:59.790707+0000","mapping_epoch":186,"log_start":"168'3995","ondisk_log_start":"168'3995","created":177,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:16:11.777014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:16:11.777014+0000","last_clean_scrub_stamp":"2024-09-10T19:16:11.777014+0000","objects_scrubbed":0,"log_size":4665,"log_dups_size":0,"ondisk_log_size":4665,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":97157120,"num_objects":1483,"num_object_clones":0,"num_object_copies":4449,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1483,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5845,"num_write_kb":234464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3071,"num_bytes_recovered":201261056,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,5],"acting":[4,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.14","version":"80'15","reported_seq":253,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-10T19:18:59.219864+0000","last_change":"2024-09-10T19:16:24.881305+0000","last_active":"2024-09-10T19:18:59.219864+0000","last_peered":"2024-09-10T19:18:59.219864+0000","last_clean":"2024-09-10T19:18:59.219864+0000","last_became_active":"2024-09-10T19:16:24.850702+0000","last_became_peered":"2024-09-10T19:16:24.850702+0000","last_unstale":"2024-09-10T19:18:59.219864+0000","last_undegraded":"2024-09-10T19:18:59.219864+0000","last_fullsized":"2024-09-10T19:18:59.219864+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"80'15","last_scrub_stamp":"2024-09-10T19:15:46.019949+0000","last_deep_scrub":"80'15","last_deep_scrub_stamp":"2024-09-10T19:11:41.719310+0000","last_clean_scrub_stamp":"2024-09-10T19:15:46.019949+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0021039190000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,5],"acting":[3,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":184,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-10T19:18:59.231597+0000","last_change":"2024-09-10T19:12:46.506129+0000","last_active":"2024-09-10T19:18:59.231597+0000","last_peered":"2024-09-10T19:18:59.231597+0000","last_clean":"2024-09-10T19:18:59.231597+0000","last_became_active":"2024-09-10T19:10:07.685870+0000","last_became_peered":"2024-09-10T19:10:07.685870+0000","last_unstale":"2024-09-10T19:18:59.231597+0000","last_undegraded":"2024-09-10T19:18:59.231597+0000","last_fullsized":"2024-09-10T19:18:59.231597+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:12:46.506071+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:12:46.506071+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001257935,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"117'10","reported_seq":316,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-10T19:18:59.236647+0000","last_change":"2024-09-10T19:16:23.844790+0000","last_active":"2024-09-10T19:18:59.236647+0000","last_peered":"2024-09-10T19:18:59.236647+0000","last_clean":"2024-09-10T19:18:59.236647+0000","last_became_active":"2024-09-10T19:13:54.724780+0000","last_became_peered":"2024-09-10T19:13:54.724780+0000","last_unstale":"2024-09-10T19:18:59.236647+0000","last_undegraded":"2024-09-10T19:18:59.236647+0000","last_fullsized":"2024-09-10T19:18:59.236647+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"117'10","last_scrub_stamp":"2024-09-10T19:16:23.844724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:16:23.844724+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.012497955999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3],"acting":[5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":187,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-10T19:18:59.225588+0000","last_change":"2024-09-10T19:16:40.213422+0000","last_active":"2024-09-10T19:18:59.225588+0000","last_peered":"2024-09-10T19:18:59.225588+0000","last_clean":"2024-09-10T19:18:59.225588+0000","last_became_active":"2024-09-10T19:10:13.721035+0000","last_became_peered":"2024-09-10T19:10:13.721035+0000","last_unstale":"2024-09-10T19:18:59.225588+0000","last_undegraded":"2024-09-10T19:18:59.225588+0000","last_fullsized":"2024-09-10T19:18:59.225588+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:16:40.213321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:16:40.213321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0051408239999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.19","version":"214'8906","reported_seq":15435,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-09-10T19:18:59.826302+0000","last_change":"2024-09-10T19:17:46.698410+0000","last_active":"2024-09-10T19:18:59.826302+0000","last_peered":"2024-09-10T19:18:59.826302+0000","last_clean":"2024-09-10T19:18:59.826302+0000","last_became_active":"2024-09-10T19:17:46.694716+0000","last_became_peered":"2024-09-10T19:17:46.694716+0000","last_unstale":"2024-09-10T19:18:59.826302+0000","last_undegraded":"2024-09-10T19:18:59.826302+0000","last_fullsized":"2024-09-10T19:18:59.826302+0000","mapping_epoch":187,"log_start":"169'4522","ondisk_log_start":"169'4522","created":177,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:16:11.777014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:16:11.777014+0000","last_clean_scrub_stamp":"2024-09-10T19:16:11.777014+0000","objects_scrubbed":0,"log_size":4384,"log_dups_size":0,"ondisk_log_size":4384,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":94765056,"num_objects":1446,"num_object_clones":0,"num_object_copies":4338,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1446,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6014,"num_write_kb":238720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3210,"num_bytes_recovered":210370560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,0],"acting":[5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.15","version":"66'9","reported_seq":238,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-10T19:18:59.236716+0000","last_change":"2024-09-10T19:16:24.882653+0000","last_active":"2024-09-10T19:18:59.236716+0000","last_peered":"2024-09-10T19:18:59.236716+0000","last_clean":"2024-09-10T19:18:59.236716+0000","last_became_active":"2024-09-10T19:16:24.859740+0000","last_became_peered":"2024-09-10T19:16:24.859740+0000","last_unstale":"2024-09-10T19:18:59.236716+0000","last_undegraded":"2024-09-10T19:18:59.236716+0000","last_fullsized":"2024-09-10T19:18:59.236716+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'9","last_scrub_stamp":"2024-09-10T19:12:52.416748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:12:52.416748+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0047220839999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":193,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-10T19:18:59.224229+0000","last_change":"2024-09-10T19:16:37.243942+0000","last_active":"2024-09-10T19:18:59.224229+0000","last_peered":"2024-09-10T19:18:59.224229+0000","last_clean":"2024-09-10T19:18:59.224229+0000","last_became_active":"2024-09-10T19:10:07.688499+0000","last_became_peered":"2024-09-10T19:10:07.688499+0000","last_unstale":"2024-09-10T19:18:59.224229+0000","last_undegraded":"2024-09-10T19:18:59.224229+0000","last_fullsized":"2024-09-10T19:18:59.224229+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:16:37.243859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:16:37.243859+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0052374580000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"155'9","reported_seq":308,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-10T19:18:59.219640+0000","last_change":"2024-09-10T19:16:24.881133+0000","last_active":"2024-09-10T19:18:59.219640+0000","last_peered":"2024-09-10T19:18:59.219640+0000","last_clean":"2024-09-10T19:18:59.219640+0000","last_became_active":"2024-09-10T19:16:24.850850+0000","last_became_peered":"2024-09-10T19:16:24.850850+0000","last_unstale":"2024-09-10T19:18:59.219640+0000","last_undegraded":"2024-09-10T19:18:59.219640+0000","last_fullsized":"2024-09-10T19:18:59.219640+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'8","last_scrub_stamp":"2024-09-10T19:12:59.723712+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:12:59.723712+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0083838109999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,4],"acting":[3,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":190,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-10T19:18:59.222312+0000","last_change":"2024-09-10T19:17:10.866121+0000","last_active":"2024-09-10T19:18:59.222312+0000","last_peered":"2024-09-10T19:18:59.222312+0000","last_clean":"2024-09-10T19:18:59.222312+0000","last_became_active":"2024-09-10T19:16:24.854280+0000","last_became_peered":"2024-09-10T19:16:24.854280+0000","last_unstale":"2024-09-10T19:18:59.222312+0000","last_undegraded":"2024-09-10T19:18:59.222312+0000","last_fullsized":"2024-09-10T19:18:59.222312+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:13:09.463001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:13:09.463001+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.012199573,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.12","version":"79'13","reported_seq":227,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-10T19:18:59.226884+0000","last_change":"2024-09-10T19:18:28.288958+0000","last_active":"2024-09-10T19:18:59.226884+0000","last_peered":"2024-09-10T19:18:59.226884+0000","last_clean":"2024-09-10T19:18:59.226884+0000","last_became_active":"2024-09-10T19:16:24.845450+0000","last_became_peered":"2024-09-10T19:16:24.845450+0000","last_unstale":"2024-09-10T19:18:59.226884+0000","last_undegraded":"2024-09-10T19:18:59.226884+0000","last_fullsized":"2024-09-10T19:18:59.226884+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'13","last_scrub_stamp":"2024-09-10T19:15:29.854117+0000","last_deep_scrub":"79'13","last_deep_scrub_stamp":"2024-09-10T19:11:22.072400+0000","last_clean_scrub_stamp":"2024-09-10T19:15:29.854117+0000","objects_scrubbed":4,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0066627300000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.14","version":"59'1","reported_seq":201,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-10T19:18:59.230333+0000","last_change":"2024-09-10T19:16:27.594895+0000","last_active":"2024-09-10T19:18:59.230333+0000","last_peered":"2024-09-10T19:18:59.230333+0000","last_clean":"2024-09-10T19:18:59.230333+0000","last_became_active":"2024-09-10T19:10:07.683263+0000","last_became_peered":"2024-09-10T19:10:07.683263+0000","last_unstale":"2024-09-10T19:18:59.230333+0000","last_undegraded":"2024-09-10T19:18:59.230333+0000","last_fullsized":"2024-09-10T19:18:59.230333+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'1","last_scrub_stamp":"2024-09-10T19:16:27.594788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:16:27.594788+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0091369840000000008,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":362,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":197,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-10T19:18:59.219920+0000","last_change":"2024-09-10T19:15:55.925313+0000","last_active":"2024-09-10T19:18:59.219920+0000","last_peered":"2024-09-10T19:18:59.219920+0000","last_clean":"2024-09-10T19:18:59.219920+0000","last_became_active":"2024-09-10T19:13:54.730185+0000","last_became_peered":"2024-09-10T19:13:54.730185+0000","last_unstale":"2024-09-10T19:18:59.219920+0000","last_undegraded":"2024-09-10T19:18:59.219920+0000","last_fullsized":"2024-09-10T19:18:59.219920+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:15:55.925228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:15:55.925228+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0017202299999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":194,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-10T19:18:59.222893+0000","last_change":"2024-09-10T19:16:23.273671+0000","last_active":"2024-09-10T19:18:59.222893+0000","last_peered":"2024-09-10T19:18:59.222893+0000","last_clean":"2024-09-10T19:18:59.222893+0000","last_became_active":"2024-09-10T19:13:54.714861+0000","last_became_peered":"2024-09-10T19:13:54.714861+0000","last_unstale":"2024-09-10T19:18:59.222893+0000","last_undegraded":"2024-09-10T19:18:59.222893+0000","last_fullsized":"2024-09-10T19:18:59.222893+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:16:23.273615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:16:23.273615+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0013972139999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,6],"acting":[0,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"66'11","reported_seq":232,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-10T19:18:59.222962+0000","last_change":"2024-09-10T19:15:53.883613+0000","last_active":"2024-09-10T19:18:59.222962+0000","last_peered":"2024-09-10T19:18:59.222962+0000","last_clean":"2024-09-10T19:18:59.222962+0000","last_became_active":"2024-09-10T19:13:54.720034+0000","last_became_peered":"2024-09-10T19:13:54.720034+0000","last_unstale":"2024-09-10T19:18:59.222962+0000","last_undegraded":"2024-09-10T19:18:59.222962+0000","last_fullsized":"2024-09-10T19:18:59.222962+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'11","last_scrub_stamp":"2024-09-10T19:15:53.883548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:15:53.883548+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001858326,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":134,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-10T19:18:59.236797+0000","last_change":"2024-09-10T19:13:54.723475+0000","last_active":"2024-09-10T19:18:59.236797+0000","last_peered":"2024-09-10T19:18:59.236797+0000","last_clean":"2024-09-10T19:18:59.236797+0000","last_became_active":"2024-09-10T19:13:54.722947+0000","last_became_peered":"2024-09-10T19:13:54.722947+0000","last_unstale":"2024-09-10T19:18:59.236797+0000","last_undegraded":"2024-09-10T19:18:59.236797+0000","last_fullsized":"2024-09-10T19:18:59.236797+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:12:44.249765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:11:10.031245+0000","last_clean_scrub_stamp":"2024-09-10T19:12:44.249765+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,6],"acting":[5,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":144,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-10T19:18:59.231868+0000","last_change":"2024-09-10T19:15:36.666310+0000","last_active":"2024-09-10T19:18:59.231868+0000","last_peered":"2024-09-10T19:18:59.231868+0000","last_clean":"2024-09-10T19:18:59.231868+0000","last_became_active":"2024-09-10T19:13:54.729703+0000","last_became_peered":"2024-09-10T19:13:54.729703+0000","last_unstale":"2024-09-10T19:18:59.231868+0000","last_undegraded":"2024-09-10T19:18:59.231868+0000","last_fullsized":"2024-09-10T19:18:59.231868+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:15:36.666216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:15:36.666216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.003489884,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":178,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-10T19:18:59.220799+0000","last_change":"2024-09-10T19:12:54.691886+0000","last_active":"2024-09-10T19:18:59.220799+0000","last_peered":"2024-09-10T19:18:59.220799+0000","last_clean":"2024-09-10T19:18:59.220799+0000","last_became_active":"2024-09-10T19:10:13.724201+0000","last_became_peered":"2024-09-10T19:10:13.724201+0000","last_unstale":"2024-09-10T19:18:59.220799+0000","last_undegraded":"2024-09-10T19:18:59.220799+0000","last_fullsized":"2024-09-10T19:18:59.220799+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:12:54.691810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:12:54.691810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.006702927,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"66'4","reported_seq":211,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-10T19:18:59.220011+0000","last_change":"2024-09-10T19:15:33.995417+0000","last_active":"2024-09-10T19:18:59.220011+0000","last_peered":"2024-09-10T19:18:59.220011+0000","last_clean":"2024-09-10T19:18:59.220011+0000","last_became_active":"2024-09-10T19:13:54.711125+0000","last_became_peered":"2024-09-10T19:13:54.711125+0000","last_unstale":"2024-09-10T19:18:59.220011+0000","last_undegraded":"2024-09-10T19:18:59.220011+0000","last_fullsized":"2024-09-10T19:18:59.220011+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'4","last_scrub_stamp":"2024-09-10T19:15:33.995274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:15:33.995274+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.011108425,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,4],"acting":[3,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":204,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-10T19:18:59.236519+0000","last_change":"2024-09-10T19:16:33.766560+0000","last_active":"2024-09-10T19:18:59.236519+0000","last_peered":"2024-09-10T19:18:59.236519+0000","last_clean":"2024-09-10T19:18:59.236519+0000","last_became_active":"2024-09-10T19:16:24.852460+0000","last_became_peered":"2024-09-10T19:16:24.852460+0000","last_unstale":"2024-09-10T19:18:59.236519+0000","last_undegraded":"2024-09-10T19:18:59.236519+0000","last_fullsized":"2024-09-10T19:18:59.236519+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:16:33.766474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:16:33.766474+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0042170250000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,3],"acting":[5,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":190,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-10T19:18:59.228592+0000","last_change":"2024-09-10T19:16:24.545188+0000","last_active":"2024-09-10T19:18:59.228592+0000","last_peered":"2024-09-10T19:18:59.228592+0000","last_clean":"2024-09-10T19:18:59.228592+0000","last_became_active":"2024-09-10T19:10:11.698536+0000","last_became_peered":"2024-09-10T19:10:11.698536+0000","last_unstale":"2024-09-10T19:18:59.228592+0000","last_undegraded":"2024-09-10T19:18:59.228592+0000","last_fullsized":"2024-09-10T19:18:59.228592+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:16:24.545134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:11:31.837944+0000","last_clean_scrub_stamp":"2024-09-10T19:16:24.545134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.00120104,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":198,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-10T19:18:59.225170+0000","last_change":"2024-09-10T19:16:24.852651+0000","last_active":"2024-09-10T19:18:59.225170+0000","last_peered":"2024-09-10T19:18:59.225170+0000","last_clean":"2024-09-10T19:18:59.225170+0000","last_became_active":"2024-09-10T19:16:24.852356+0000","last_became_peered":"2024-09-10T19:16:24.852356+0000","last_unstale":"2024-09-10T19:18:59.225170+0000","last_undegraded":"2024-09-10T19:18:59.225170+0000","last_fullsized":"2024-09-10T19:18:59.225170+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:15:44.523530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:15:44.523530+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.027001213999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,3],"acting":[7,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.11","version":"66'11","reported_seq":92,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-10T19:18:59.232168+0000","last_change":"2024-09-10T19:15:45.158623+0000","last_active":"2024-09-10T19:18:59.232168+0000","last_peered":"2024-09-10T19:18:59.232168+0000","last_clean":"2024-09-10T19:18:59.232168+0000","last_became_active":"2024-09-10T19:15:45.149864+0000","last_became_peered":"2024-09-10T19:15:45.149864+0000","last_unstale":"2024-09-10T19:18:59.232168+0000","last_undegraded":"2024-09-10T19:18:59.232168+0000","last_fullsized":"2024-09-10T19:18:59.232168+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'11","last_scrub_stamp":"2024-09-10T19:13:05.729854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:13:05.729854+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.010667395,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":204,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-10T19:18:59.228623+0000","last_change":"2024-09-10T19:16:24.855851+0000","last_active":"2024-09-10T19:18:59.228623+0000","last_peered":"2024-09-10T19:18:59.228623+0000","last_clean":"2024-09-10T19:18:59.228623+0000","last_became_active":"2024-09-10T19:16:24.855541+0000","last_became_peered":"2024-09-10T19:16:24.855541+0000","last_unstale":"2024-09-10T19:18:59.228623+0000","last_undegraded":"2024-09-10T19:18:59.228623+0000","last_fullsized":"2024-09-10T19:18:59.228623+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:16:23.543097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:13:05.876655+0000","last_clean_scrub_stamp":"2024-09-10T19:16:23.543097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001181779,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,0],"acting":[6,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":180,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-10T19:18:59.225117+0000","last_change":"2024-09-10T19:13:41.326028+0000","last_active":"2024-09-10T19:18:59.225117+0000","last_peered":"2024-09-10T19:18:59.225117+0000","last_clean":"2024-09-10T19:18:59.225117+0000","last_became_active":"2024-09-10T19:10:11.699013+0000","last_became_peered":"2024-09-10T19:10:11.699013+0000","last_unstale":"2024-09-10T19:18:59.225117+0000","last_undegraded":"2024-09-10T19:18:59.225117+0000","last_fullsized":"2024-09-10T19:18:59.225117+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:13:41.325941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:11:57.112092+0000","last_clean_scrub_stamp":"2024-09-10T19:13:41.325941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001598676,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":178,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-10T19:18:59.219787+0000","last_change":"2024-09-10T19:13:15.812326+0000","last_active":"2024-09-10T19:18:59.219787+0000","last_peered":"2024-09-10T19:18:59.219787+0000","last_clean":"2024-09-10T19:18:59.219787+0000","last_became_active":"2024-09-10T19:10:14.056172+0000","last_became_peered":"2024-09-10T19:10:14.056172+0000","last_unstale":"2024-09-10T19:18:59.219787+0000","last_undegraded":"2024-09-10T19:18:59.219787+0000","last_fullsized":"2024-09-10T19:18:59.219787+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:13:15.812244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:13:15.812244+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0048044389999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.12","version":"214'9015","reported_seq":9117,"reported_epoch":214,"state":"active+remapped+backfill_toofull","last_fresh":"2024-09-10T19:18:59.996505+0000","last_change":"2024-09-10T19:18:30.040016+0000","last_active":"2024-09-10T19:18:59.996505+0000","last_peered":"2024-09-10T19:18:59.996505+0000","last_clean":"2024-09-10T19:17:27.172505+0000","last_became_active":"2024-09-10T19:17:30.027866+0000","last_became_peered":"2024-09-10T19:17:30.027866+0000","last_unstale":"2024-09-10T19:18:59.996505+0000","last_undegraded":"2024-09-10T19:18:59.996505+0000","last_fullsized":"2024-09-10T19:18:59.996505+0000","mapping_epoch":180,"log_start":"170'4552","ondisk_log_start":"170'4552","created":177,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:16:11.777014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:16:11.777014+0000","last_clean_scrub_stamp":"2024-09-10T19:16:11.777014+0000","objects_scrubbed":0,"log_size":4463,"log_dups_size":0,"ondisk_log_size":4463,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":100007936,"num_objects":1526,"num_object_clones":0,"num_object_copies":4578,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3052,"num_objects_unfound":0,"num_objects_dirty":1526,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6098,"num_write_kb":243968,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,4],"acting":[0,4,5],"avail_no_missing":["0","4","5"],"object_location_counts":[{"shards":"0,4,5","objects":1526}],"blocked_by":[],"up_primary":6,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1c","version":"80'2","reported_seq":204,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-10T19:18:59.227542+0000","last_change":"2024-09-10T19:16:32.648844+0000","last_active":"2024-09-10T19:18:59.227542+0000","last_peered":"2024-09-10T19:18:59.227542+0000","last_clean":"2024-09-10T19:18:59.227542+0000","last_became_active":"2024-09-10T19:16:24.851220+0000","last_became_peered":"2024-09-10T19:16:24.851220+0000","last_unstale":"2024-09-10T19:18:59.227542+0000","last_undegraded":"2024-09-10T19:18:59.227542+0000","last_fullsized":"2024-09-10T19:18:59.227542+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"80'2","last_scrub_stamp":"2024-09-10T19:16:32.648738+0000","last_deep_scrub":"80'2","last_deep_scrub_stamp":"2024-09-10T19:11:30.272043+0000","last_clean_scrub_stamp":"2024-09-10T19:16:32.648738+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0066232920000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":369,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":369,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,3],"acting":[7,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":184,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-10T19:18:59.233528+0000","last_change":"2024-09-10T19:13:14.449643+0000","last_active":"2024-09-10T19:18:59.233528+0000","last_peered":"2024-09-10T19:18:59.233528+0000","last_clean":"2024-09-10T19:18:59.233528+0000","last_became_active":"2024-09-10T19:10:07.689877+0000","last_became_peered":"2024-09-10T19:10:07.689877+0000","last_unstale":"2024-09-10T19:18:59.233528+0000","last_undegraded":"2024-09-10T19:18:59.233528+0000","last_fullsized":"2024-09-10T19:18:59.233528+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:13:14.449593+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:13:14.449593+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001195542,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1e","version":"80'15","reported_seq":211,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-10T19:18:59.225740+0000","last_change":"2024-09-10T19:16:33.258114+0000","last_active":"2024-09-10T19:18:59.225740+0000","last_peered":"2024-09-10T19:18:59.225740+0000","last_clean":"2024-09-10T19:18:59.225740+0000","last_became_active":"2024-09-10T19:10:09.823061+0000","last_became_peered":"2024-09-10T19:10:09.823061+0000","last_unstale":"2024-09-10T19:18:59.225740+0000","last_undegraded":"2024-09-10T19:18:59.225740+0000","last_fullsized":"2024-09-10T19:18:59.225740+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"80'15","last_scrub_stamp":"2024-09-10T19:16:33.258031+0000","last_deep_scrub":"80'15","last_deep_scrub_stamp":"2024-09-10T19:11:25.883919+0000","last_clean_scrub_stamp":"2024-09-10T19:16:33.258031+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0040337309999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1f","version":"66'8","reported_seq":310,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-10T19:18:59.228467+0000","last_change":"2024-09-10T19:17:32.477395+0000","last_active":"2024-09-10T19:18:59.228467+0000","last_peered":"2024-09-10T19:18:59.228467+0000","last_clean":"2024-09-10T19:18:59.228467+0000","last_became_active":"2024-09-10T19:10:11.695427+0000","last_became_peered":"2024-09-10T19:10:11.695427+0000","last_unstale":"2024-09-10T19:18:59.228467+0000","last_undegraded":"2024-09-10T19:18:59.228467+0000","last_fullsized":"2024-09-10T19:18:59.228467+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'8","last_scrub_stamp":"2024-09-10T19:15:53.664208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:15:53.664208+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001849699,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.13","version":"214'9212","reported_seq":18241,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-09-10T19:19:00.004708+0000","last_change":"2024-09-10T19:18:56.828838+0000","last_active":"2024-09-10T19:19:00.004708+0000","last_peered":"2024-09-10T19:19:00.004708+0000","last_clean":"2024-09-10T19:19:00.004708+0000","last_became_active":"2024-09-10T19:18:56.797038+0000","last_became_peered":"2024-09-10T19:18:56.797038+0000","last_unstale":"2024-09-10T19:19:00.004708+0000","last_undegraded":"2024-09-10T19:19:00.004708+0000","last_fullsized":"2024-09-10T19:19:00.004708+0000","mapping_epoch":211,"log_start":"172'5286","ondisk_log_start":"172'5286","created":177,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:16:11.777014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:16:11.777014+0000","last_clean_scrub_stamp":"2024-09-10T19:16:11.777014+0000","objects_scrubbed":0,"log_size":3926,"log_dups_size":0,"ondisk_log_size":3926,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92897280,"num_objects":1418,"num_object_clones":0,"num_object_copies":4254,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1418,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6202,"num_write_kb":243808,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3911,"num_bytes_recovered":256278528,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,4],"acting":[3,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":154,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-10T19:18:59.236407+0000","last_change":"2024-09-10T19:16:24.863000+0000","last_active":"2024-09-10T19:18:59.236407+0000","last_peered":"2024-09-10T19:18:59.236407+0000","last_clean":"2024-09-10T19:18:59.236407+0000","last_became_active":"2024-09-10T19:16:24.862695+0000","last_became_peered":"2024-09-10T19:16:24.862695+0000","last_unstale":"2024-09-10T19:18:59.236407+0000","last_undegraded":"2024-09-10T19:18:59.236407+0000","last_fullsized":"2024-09-10T19:18:59.236407+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:15:29.580605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:15:29.580605+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0047886710000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":194,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-10T19:18:59.221080+0000","last_change":"2024-09-10T19:16:41.881053+0000","last_active":"2024-09-10T19:18:59.221080+0000","last_peered":"2024-09-10T19:18:59.221080+0000","last_clean":"2024-09-10T19:18:59.221080+0000","last_became_active":"2024-09-10T19:10:07.682689+0000","last_became_peered":"2024-09-10T19:10:07.682689+0000","last_unstale":"2024-09-10T19:18:59.221080+0000","last_undegraded":"2024-09-10T19:18:59.221080+0000","last_fullsized":"2024-09-10T19:18:59.221080+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:16:41.880992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:16:41.880992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0044984990000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"80'16","reported_seq":266,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-10T19:18:59.228549+0000","last_change":"2024-09-10T19:18:29.250077+0000","last_active":"2024-09-10T19:18:59.228549+0000","last_peered":"2024-09-10T19:18:59.228549+0000","last_clean":"2024-09-10T19:18:59.228549+0000","last_became_active":"2024-09-10T19:13:54.713563+0000","last_became_peered":"2024-09-10T19:13:54.713563+0000","last_unstale":"2024-09-10T19:18:59.228549+0000","last_undegraded":"2024-09-10T19:18:59.228549+0000","last_fullsized":"2024-09-10T19:18:59.228549+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"80'16","last_scrub_stamp":"2024-09-10T19:15:47.700523+0000","last_deep_scrub":"80'16","last_deep_scrub_stamp":"2024-09-10T19:11:40.853611+0000","last_clean_scrub_stamp":"2024-09-10T19:15:47.700523+0000","objects_scrubbed":5,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0019149010000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,0],"acting":[6,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":199,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-10T19:18:59.225682+0000","last_change":"2024-09-10T19:16:34.222429+0000","last_active":"2024-09-10T19:18:59.225682+0000","last_peered":"2024-09-10T19:18:59.225682+0000","last_clean":"2024-09-10T19:18:59.225682+0000","last_became_active":"2024-09-10T19:16:24.834713+0000","last_became_peered":"2024-09-10T19:16:24.834713+0000","last_unstale":"2024-09-10T19:18:59.225682+0000","last_undegraded":"2024-09-10T19:18:59.225682+0000","last_fullsized":"2024-09-10T19:18:59.225682+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:16:34.222365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:11:28.822327+0000","last_clean_scrub_stamp":"2024-09-10T19:16:34.222365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.014893412999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.10","version":"214'9093","reported_seq":15884,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-09-10T19:18:59.888161+0000","last_change":"2024-09-10T19:17:46.709272+0000","last_active":"2024-09-10T19:18:59.888161+0000","last_peered":"2024-09-10T19:18:59.888161+0000","last_clean":"2024-09-10T19:18:59.888161+0000","last_became_active":"2024-09-10T19:17:46.694818+0000","last_became_peered":"2024-09-10T19:17:46.694818+0000","last_unstale":"2024-09-10T19:18:59.888161+0000","last_undegraded":"2024-09-10T19:18:59.888161+0000","last_fullsized":"2024-09-10T19:18:59.888161+0000","mapping_epoch":187,"log_start":"174'5492","ondisk_log_start":"174'5492","created":177,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:16:11.777014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:16:11.777014+0000","last_clean_scrub_stamp":"2024-09-10T19:16:11.777014+0000","objects_scrubbed":0,"log_size":3601,"log_dups_size":0,"ondisk_log_size":3601,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":95485952,"num_objects":1457,"num_object_clones":0,"num_object_copies":4371,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1457,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6177,"num_write_kb":244288,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3344,"num_bytes_recovered":219152384,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,4],"acting":[7,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":187,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-10T19:18:59.222464+0000","last_change":"2024-09-10T19:16:36.960796+0000","last_active":"2024-09-10T19:18:59.222464+0000","last_peered":"2024-09-10T19:18:59.222464+0000","last_clean":"2024-09-10T19:18:59.222464+0000","last_became_active":"2024-09-10T19:10:14.053619+0000","last_became_peered":"2024-09-10T19:10:14.053619+0000","last_unstale":"2024-09-10T19:18:59.222464+0000","last_undegraded":"2024-09-10T19:18:59.222464+0000","last_fullsized":"2024-09-10T19:18:59.222464+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:16:36.960695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:16:36.960695+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.011430734999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":194,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-10T19:18:59.227230+0000","last_change":"2024-09-10T19:16:30.666084+0000","last_active":"2024-09-10T19:18:59.227230+0000","last_peered":"2024-09-10T19:18:59.227230+0000","last_clean":"2024-09-10T19:18:59.227230+0000","last_became_active":"2024-09-10T19:10:07.685997+0000","last_became_peered":"2024-09-10T19:10:07.685997+0000","last_unstale":"2024-09-10T19:18:59.227230+0000","last_undegraded":"2024-09-10T19:18:59.227230+0000","last_fullsized":"2024-09-10T19:18:59.227230+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:16:30.666001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:16:30.666001+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0041927830000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1c","version":"78'22","reported_seq":143,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-10T19:18:59.219790+0000","last_change":"2024-09-10T19:16:24.870445+0000","last_active":"2024-09-10T19:18:59.219790+0000","last_peered":"2024-09-10T19:18:59.219790+0000","last_clean":"2024-09-10T19:18:59.219790+0000","last_became_active":"2024-09-10T19:16:24.845764+0000","last_became_peered":"2024-09-10T19:16:24.845764+0000","last_unstale":"2024-09-10T19:18:59.219790+0000","last_undegraded":"2024-09-10T19:18:59.219790+0000","last_fullsized":"2024-09-10T19:18:59.219790+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"78'22","last_scrub_stamp":"2024-09-10T19:15:37.702624+0000","last_deep_scrub":"78'22","last_deep_scrub_stamp":"2024-09-10T19:11:21.118937+0000","last_clean_scrub_stamp":"2024-09-10T19:15:37.702624+0000","objects_scrubbed":7,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":147,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-10T19:18:59.220040+0000","last_change":"2024-09-10T19:15:33.008615+0000","last_active":"2024-09-10T19:18:59.220040+0000","last_peered":"2024-09-10T19:18:59.220040+0000","last_clean":"2024-09-10T19:18:59.220040+0000","last_became_active":"2024-09-10T19:13:54.709055+0000","last_became_peered":"2024-09-10T19:13:54.709055+0000","last_unstale":"2024-09-10T19:18:59.220040+0000","last_undegraded":"2024-09-10T19:18:59.220040+0000","last_fullsized":"2024-09-10T19:18:59.220040+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:15:33.008542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:15:33.008542+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.004557363,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,5],"acting":[4,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.11","version":"214'8906","reported_seq":18358,"reported_epoch":214,"state":"active+clean","last_fresh":"2024-09-10T19:18:59.821466+0000","last_change":"2024-09-10T19:18:24.020721+0000","last_active":"2024-09-10T19:18:59.821466+0000","last_peered":"2024-09-10T19:18:59.821466+0000","last_clean":"2024-09-10T19:18:59.821466+0000","last_became_active":"2024-09-10T19:18:24.009106+0000","last_became_peered":"2024-09-10T19:18:24.009106+0000","last_unstale":"2024-09-10T19:18:59.821466+0000","last_undegraded":"2024-09-10T19:18:59.821466+0000","last_fullsized":"2024-09-10T19:18:59.821466+0000","mapping_epoch":198,"log_start":"169'4432","ondisk_log_start":"169'4432","created":177,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:16:11.777014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:16:11.777014+0000","last_clean_scrub_stamp":"2024-09-10T19:16:11.777014+0000","objects_scrubbed":0,"log_size":4474,"log_dups_size":0,"ondisk_log_size":4474,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":100106240,"num_objects":1528,"num_object_clones":0,"num_object_copies":4584,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1528,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5988,"num_write_kb":240480,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3110,"num_bytes_recovered":203816960,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":178,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-10T19:18:59.221362+0000","last_change":"2024-09-10T19:12:51.767586+0000","last_active":"2024-09-10T19:18:59.221362+0000","last_peered":"2024-09-10T19:18:59.221362+0000","last_clean":"2024-09-10T19:18:59.221362+0000","last_became_active":"2024-09-10T19:10:14.053698+0000","last_became_peered":"2024-09-10T19:10:14.053698+0000","last_unstale":"2024-09-10T19:18:59.221362+0000","last_undegraded":"2024-09-10T19:18:59.221362+0000","last_fullsized":"2024-09-10T19:18:59.221362+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:12:51.767056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:12:51.767056+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.006796519,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":184,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-10T19:18:59.220133+0000","last_change":"2024-09-10T19:12:50.719167+0000","last_active":"2024-09-10T19:18:59.220133+0000","last_peered":"2024-09-10T19:18:59.220133+0000","last_clean":"2024-09-10T19:18:59.220133+0000","last_became_active":"2024-09-10T19:10:07.682811+0000","last_became_peered":"2024-09-10T19:10:07.682811+0000","last_unstale":"2024-09-10T19:18:59.220133+0000","last_undegraded":"2024-09-10T19:18:59.220133+0000","last_fullsized":"2024-09-10T19:18:59.220133+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:12:50.718663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:12:50.718663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0042138399999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1d","version":"66'12","reported_seq":234,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-10T19:18:59.220886+0000","last_change":"2024-09-10T19:15:52.923561+0000","last_active":"2024-09-10T19:18:59.220886+0000","last_peered":"2024-09-10T19:18:59.220886+0000","last_clean":"2024-09-10T19:18:59.220886+0000","last_became_active":"2024-09-10T19:13:54.714629+0000","last_became_peered":"2024-09-10T19:13:54.714629+0000","last_unstale":"2024-09-10T19:18:59.220886+0000","last_undegraded":"2024-09-10T19:18:59.220886+0000","last_fullsized":"2024-09-10T19:18:59.220886+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'12","last_scrub_stamp":"2024-09-10T19:15:52.923498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:15:52.923498+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0019450450000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":199,"reported_epoch":213,"state":"active+clean","last_fresh":"2024-09-10T19:18:59.226476+0000","last_change":"2024-09-10T19:16:37.970105+0000","last_active":"2024-09-10T19:18:59.226476+0000","last_peered":"2024-09-10T19:18:59.226476+0000","last_clean":"2024-09-10T19:18:59.226476+0000","last_became_active":"2024-09-10T19:16:24.843013+0000","last_became_peered":"2024-09-10T19:16:24.843013+0000","last_unstale":"2024-09-10T19:18:59.226476+0000","last_undegraded":"2024-09-10T19:18:59.226476+0000","last_fullsized":"2024-09-10T19:18:59.226476+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:16:37.970030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:16:37.970030+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.003625063,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":8,"num_pg":26,"stat_sum":{"num_bytes":3131506712,"num_objects":47784,"num_object_clones":0,"num_object_copies":143352,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3052,"num_objects_unfound":0,"num_objects_dirty":47784,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":192778,"num_write_kb":7697921,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28890,"num_bytes_recovered":1893335040,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":10912018432,"data_stored":10912006216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":106950,"ondisk_log_size":106950,"up":78,"acting":78,"num_store_stats":7},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":382,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":1528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":73,"ondisk_log_size":73,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":177,"num_object_clones":0,"num_object_copies":531,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":177,"num_whiteouts":0,"num_read":695,"num_read_kb":452,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":173,"num_bytes_recovered":3218,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":540672,"data_stored":14242,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":445,"ondisk_log_size":445,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":161,"num_read_kb":156,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":389,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":2002,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":9,"ondisk_log_size":9,"up":9,"acting":9,"num_store_stats":7},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1328,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":27,"num_read_kb":27,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":3984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4,"ondisk_log_size":4,"up":96,"acting":96,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79,"ondisk_log_size":79,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":54,"seq":231928234095,"num_pgs":87,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2256468,"kb_used_data":1946380,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":91488684,"statfs":{"total":95995035648,"available":93684412416,"internally_reserved":0,"allocated":1993093120,"data_stored":1992448421,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 19:18:04 2024","interfaces":[{"interface":"back","average":{"1min":1.629,"5min":1.663,"15min":1.0549999999999999},"min":{"1min":0.20999999999999999,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":5.4470000000000001,"5min":25.640999999999998,"15min":35.551000000000002},"last":0.21199999999999999},{"interface":"front","average":{"1min":1.246,"5min":1.4530000000000001,"15min":0.96699999999999997},"min":{"1min":0.215,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":5.9119999999999999,"5min":27.870999999999999,"15min":35.503999999999998},"last":1.6279999999999999}]},{"osd":1,"last update":"Tue Sep 10 19:18:29 2024","interfaces":[{"interface":"back","average":{"1min":0.76600000000000001,"5min":0.86499999999999999,"15min":0.63800000000000001},"min":{"1min":0.26000000000000001,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":3.1099999999999999,"5min":8.3499999999999996,"15min":8.3499999999999996},"last":0.28599999999999998},{"interface":"front","average":{"1min":0.78800000000000003,"5min":0.878,"15min":0.76400000000000001},"min":{"1min":0.26000000000000001,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":3.085,"5min":5.609,"15min":5.609},"last":0.35599999999999998}]},{"osd":2,"last update":"Tue Sep 10 19:18:04 2024","interfaces":[{"interface":"back","average":{"1min":0.84099999999999997,"5min":1.181,"15min":0.73799999999999999},"min":{"1min":0.27700000000000002,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":2.7519999999999998,"5min":31.940999999999999,"15min":31.940999999999999},"last":0.629},{"interface":"front","average":{"1min":1.6539999999999999,"5min":1.4990000000000001,"15min":0.878},"min":{"1min":0.36299999999999999,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":5.5410000000000004,"5min":25.056000000000001,"15min":25.056000000000001},"last":0.314}]},{"osd":3,"last update":"Tue Sep 10 19:18:04 2024","interfaces":[{"interface":"back","average":{"1min":2.3130000000000002,"5min":1.6220000000000001,"15min":1.095},"min":{"1min":0.249,"5min":0.217,"15min":0.217},"max":{"1min":21.074999999999999,"5min":25.443999999999999,"15min":35.277999999999999},"last":0.67900000000000005},{"interface":"front","average":{"1min":2.9660000000000002,"5min":1.6279999999999999,"15min":1.0900000000000001},"min":{"1min":0.29699999999999999,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":18.82,"5min":25.530000000000001,"15min":36.512999999999998},"last":0.25700000000000001}]},{"osd":4,"last update":"Tue Sep 10 19:18:04 2024","interfaces":[{"interface":"back","average":{"1min":2.1040000000000001,"5min":1.0580000000000001,"15min":0.65200000000000002},"min":{"1min":0.34599999999999997,"5min":0.23300000000000001,"15min":0.22800000000000001},"max":{"1min":6.9160000000000004,"5min":8.1859999999999999,"15min":8.1859999999999999},"last":3.4780000000000002},{"interface":"front","average":{"1min":1.0049999999999999,"5min":0.76400000000000001,"15min":0.55900000000000005},"min":{"1min":0.378,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":3.0339999999999998,"5min":4.5469999999999997,"15min":4.5469999999999997},"last":2.2440000000000002}]},{"osd":5,"last update":"Tue Sep 10 19:18:04 2024","interfaces":[{"interface":"back","average":{"1min":2.1699999999999999,"5min":1.292,"15min":0.748},"min":{"1min":0.32400000000000001,"5min":0.253,"15min":0.23699999999999999},"max":{"1min":8.1120000000000001,"5min":11.621,"15min":11.621},"last":6.3529999999999998},{"interface":"front","average":{"1min":1.5860000000000001,"5min":1.121,"15min":0.69199999999999995},"min":{"1min":0.372,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":5.9889999999999999,"5min":9.2680000000000007,"15min":9.2680000000000007},"last":1.6579999999999999}]},{"osd":6,"last update":"Tue Sep 10 19:18:04 2024","interfaces":[{"interface":"back","average":{"1min":1.954,"5min":1.194,"15min":0.73499999999999999},"min":{"1min":0.33900000000000002,"5min":0.29099999999999998,"15min":0.26200000000000001},"max":{"1min":5.9550000000000001,"5min":8.2590000000000003,"15min":8.2590000000000003},"last":1.677},{"interface":"front","average":{"1min":1.222,"5min":0.94699999999999995,"15min":0.64000000000000001},"min":{"1min":0.29999999999999999,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":4.2960000000000003,"5min":8.4130000000000003,"15min":8.4130000000000003},"last":0.65900000000000003}]}]},{"osd":6,"up_from":47,"seq":201863463028,"num_pgs":83,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1967768,"kb_used_data":1653772,"kb_used_omap":1,"kb_used_meta":313982,"kb_avail":91777384,"statfs":{"total":95995035648,"available":93980041216,"internally_reserved":0,"allocated":1693462528,"data_stored":1692826086,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":321517912},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 19:18:47 2024","interfaces":[{"interface":"back","average":{"1min":0.996,"5min":2.2149999999999999,"15min":1.107},"min":{"1min":0.19500000000000001,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":5.2350000000000003,"5min":112.988,"15min":112.988},"last":0.35399999999999998},{"interface":"front","average":{"1min":1.139,"5min":2.319,"15min":1.161},"min":{"1min":0.26300000000000001,"5min":0.221,"15min":0.221},"max":{"1min":5.274,"5min":113.18300000000001,"15min":113.18300000000001},"last":0.248}]},{"osd":1,"last update":"Tue Sep 10 19:18:30 2024","interfaces":[{"interface":"back","average":{"1min":0.64600000000000002,"5min":0.61299999999999999,"15min":0.54700000000000004},"min":{"1min":0.23799999999999999,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":2.4849999999999999,"5min":4.2919999999999998,"15min":4.2919999999999998},"last":0.44800000000000001},{"interface":"front","average":{"1min":0.80400000000000005,"5min":0.78000000000000003,"15min":0.69599999999999995},"min":{"1min":0.28100000000000003,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":3.4369999999999998,"5min":3.4369999999999998,"15min":3.4369999999999998},"last":0.50900000000000001}]},{"osd":2,"last update":"Tue Sep 10 19:18:47 2024","interfaces":[{"interface":"back","average":{"1min":0.89500000000000002,"5min":2.0310000000000001,"15min":1.0569999999999999},"min":{"1min":0.27700000000000002,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":2.4510000000000001,"5min":112.864,"15min":112.864},"last":0.40200000000000002},{"interface":"front","average":{"1min":0.93100000000000005,"5min":2.1970000000000001,"15min":1.1060000000000001},"min":{"1min":0.315,"5min":0.254,"15min":0.254},"max":{"1min":2.2799999999999998,"5min":112.822,"15min":112.822},"last":0.55700000000000005}]},{"osd":3,"last update":"Tue Sep 10 19:18:47 2024","interfaces":[{"interface":"back","average":{"1min":1.1839999999999999,"5min":2.7759999999999998,"15min":1.306},"min":{"1min":0.251,"5min":0.251,"15min":0.251},"max":{"1min":5.3419999999999996,"5min":125.56399999999999,"15min":125.56399999999999},"last":0.33800000000000002},{"interface":"front","average":{"1min":1.234,"5min":2.8260000000000001,"15min":1.3580000000000001},"min":{"1min":0.254,"5min":0.254,"15min":0.23200000000000001},"max":{"1min":6.5259999999999998,"5min":125.47,"15min":125.47},"last":0.42599999999999999}]},{"osd":4,"last update":"Tue Sep 10 19:18:47 2024","interfaces":[{"interface":"back","average":{"1min":1.4470000000000001,"5min":1.042,"15min":0.59599999999999997},"min":{"1min":0.27900000000000003,"5min":0.27600000000000002,"15min":0.20699999999999999},"max":{"1min":6.9269999999999996,"5min":6.9269999999999996,"15min":6.9269999999999996},"last":0.52500000000000002},{"interface":"front","average":{"1min":1.1259999999999999,"5min":0.88200000000000001,"15min":0.57199999999999995},"min":{"1min":0.26100000000000001,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":3.492,"5min":4.0750000000000002,"15min":4.0750000000000002},"last":0.48199999999999998}]},{"osd":5,"last update":"Tue Sep 10 19:18:47 2024","interfaces":[{"interface":"back","average":{"1min":1.742,"5min":1.625,"15min":0.80400000000000005},"min":{"1min":0.34799999999999998,"5min":0.30499999999999999,"15min":0.248},"max":{"1min":5.008,"5min":44.765000000000001,"15min":44.765000000000001},"last":0.59099999999999997},{"interface":"front","average":{"1min":1.2969999999999999,"5min":1.6240000000000001,"15min":0.82599999999999996},"min":{"1min":0.29699999999999999,"5min":0.28899999999999998,"15min":0.27500000000000002},"max":{"1min":4.2460000000000004,"5min":43.036000000000001,"15min":43.036000000000001},"last":0.57399999999999995}]},{"osd":7,"last update":"Tue Sep 10 19:18:08 2024","interfaces":[{"interface":"back","average":{"1min":1.1359999999999999,"5min":0.92200000000000004,"15min":0.61099999999999999},"min":{"1min":0.35099999999999998,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":3.375,"5min":8.9390000000000001,"15min":8.9390000000000001},"last":0.69899999999999995},{"interface":"front","average":{"1min":1.276,"5min":1.071,"15min":0.66900000000000004},"min":{"1min":0.33800000000000002,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":3.464,"5min":10.298,"15min":10.298},"last":0.54300000000000004}]}]},{"osd":5,"up_from":39,"seq":167503724665,"num_pgs":80,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2147276,"kb_used_data":1841600,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":91597876,"statfs":{"total":95995035648,"available":93796225024,"internally_reserved":0,"allocated":1885798400,"data_stored":1885174543,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 19:18:20 2024","interfaces":[{"interface":"back","average":{"1min":0.78500000000000003,"5min":1.1639999999999999,"15min":0.71899999999999997},"min":{"1min":0.17999999999999999,"5min":0.17999999999999999,"15min":0.17999999999999999},"max":{"1min":2.6549999999999998,"5min":5.851,"15min":5.851},"last":2.8490000000000002},{"interface":"front","average":{"1min":0.85099999999999998,"5min":0.94799999999999995,"15min":0.64600000000000002},"min":{"1min":0.20899999999999999,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":2.895,"5min":5.923,"15min":5.923},"last":1.1899999999999999}]},{"osd":1,"last update":"Tue Sep 10 19:18:23 2024","interfaces":[{"interface":"back","average":{"1min":0.77400000000000002,"5min":0.80800000000000005,"15min":0.70399999999999996},"min":{"1min":0.249,"5min":0.249,"15min":0.249},"max":{"1min":1.792,"5min":3.77,"15min":3.77},"last":1.1080000000000001},{"interface":"front","average":{"1min":0.85799999999999998,"5min":0.74199999999999999,"15min":0.58199999999999996},"min":{"1min":0.255,"5min":0.255,"15min":0.255},"max":{"1min":3.2130000000000001,"5min":5.0570000000000004,"15min":5.0570000000000004},"last":1.5960000000000001}]},{"osd":2,"last update":"Tue Sep 10 19:18:20 2024","interfaces":[{"interface":"back","average":{"1min":0.80600000000000005,"5min":1.0640000000000001,"15min":0.69799999999999995},"min":{"1min":0.27800000000000002,"5min":0.26100000000000001,"15min":0.253},"max":{"1min":2.6240000000000001,"5min":9.9399999999999995,"15min":9.9399999999999995},"last":2.9529999999999998},{"interface":"front","average":{"1min":1.016,"5min":1.1379999999999999,"15min":0.70999999999999996},"min":{"1min":0.28599999999999998,"5min":0.224,"15min":0.224},"max":{"1min":2.7970000000000002,"5min":6.3789999999999996,"15min":6.3789999999999996},"last":1.1419999999999999}]},{"osd":3,"last update":"Tue Sep 10 19:18:20 2024","interfaces":[{"interface":"back","average":{"1min":0.80900000000000005,"5min":1.0329999999999999,"15min":0.70699999999999996},"min":{"1min":0.39000000000000001,"5min":0.224,"15min":0.224},"max":{"1min":2.52,"5min":9.9019999999999992,"15min":9.9019999999999992},"last":2.9049999999999998},{"interface":"front","average":{"1min":0.97899999999999998,"5min":0.96999999999999997,"15min":0.68600000000000005},"min":{"1min":0.311,"5min":0.251,"15min":0.248},"max":{"1min":2.8260000000000001,"5min":5.3869999999999996,"15min":5.3869999999999996},"last":1.167}]},{"osd":4,"last update":"Tue Sep 10 19:18:20 2024","interfaces":[{"interface":"back","average":{"1min":1.327,"5min":1.139,"15min":0.628},"min":{"1min":0.27100000000000002,"5min":0.24399999999999999,"15min":0.20699999999999999},"max":{"1min":7.4390000000000001,"5min":10.414,"15min":10.414},"last":1.5640000000000001},{"interface":"front","average":{"1min":1.355,"5min":1.1930000000000001,"15min":0.66800000000000004},"min":{"1min":0.26400000000000001,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":4.306,"5min":10.518000000000001,"15min":10.518000000000001},"last":2.9790000000000001}]},{"osd":6,"last update":"Tue Sep 10 19:18:47 2024","interfaces":[{"interface":"back","average":{"1min":0.92400000000000004,"5min":1.393,"15min":0.76600000000000001},"min":{"1min":0.29799999999999999,"5min":0.29099999999999998,"15min":0.223},"max":{"1min":2.0990000000000002,"5min":9.9670000000000005,"15min":9.9670000000000005},"last":2.8820000000000001},{"interface":"front","average":{"1min":1.3440000000000001,"5min":1.296,"15min":0.72599999999999998},"min":{"1min":0.38600000000000001,"5min":0.26900000000000002,"15min":0.25},"max":{"1min":4.7140000000000004,"5min":10.446,"15min":10.446},"last":1.212}]},{"osd":7,"last update":"Tue Sep 10 19:18:08 2024","interfaces":[{"interface":"back","average":{"1min":1.7430000000000001,"5min":1.125,"15min":0.71099999999999997},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":3.8399999999999999,"5min":5.9660000000000002,"15min":5.9660000000000002},"last":3.9510000000000001},{"interface":"front","average":{"1min":2.0409999999999999,"5min":1.0589999999999999,"15min":0.67400000000000004},"min":{"1min":0.36899999999999999,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":6.8840000000000003,"5min":6.8840000000000003,"15min":6.8840000000000003},"last":1.623}]}]},{"osd":4,"up_from":32,"seq":137438953597,"num_pgs":78,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2372032,"kb_used_data":2062004,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":91373120,"statfs":{"total":95995035648,"available":93566074880,"internally_reserved":0,"allocated":2111492096,"data_stored":2110878639,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 19:18:04 2024","interfaces":[{"interface":"back","average":{"1min":1.7150000000000001,"5min":1.4750000000000001,"15min":0.84899999999999998},"min":{"1min":0.20799999999999999,"5min":0.187,"15min":0.187},"max":{"1min":24.113,"5min":32.533000000000001,"15min":32.533000000000001},"last":0.53000000000000003},{"interface":"front","average":{"1min":1.9670000000000001,"5min":1.54,"15min":0.86699999999999999},"min":{"1min":0.23999999999999999,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":24.132999999999999,"5min":32.512999999999998,"15min":32.512999999999998},"last":0.315}]},{"osd":1,"last update":"Tue Sep 10 19:18:25 2024","interfaces":[{"interface":"back","average":{"1min":1.1120000000000001,"5min":0.85799999999999998,"15min":0.70399999999999996},"min":{"1min":0.253,"5min":0.253,"15min":0.253},"max":{"1min":2.9350000000000001,"5min":7.9770000000000003,"15min":7.9770000000000003},"last":0.51900000000000002},{"interface":"front","average":{"1min":0.78600000000000003,"5min":0.74199999999999999,"15min":0.69799999999999995},"min":{"1min":0.30299999999999999,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":2.8540000000000001,"5min":3.8260000000000001,"15min":3.8260000000000001},"last":0.432}]},{"osd":2,"last update":"Tue Sep 10 19:18:04 2024","interfaces":[{"interface":"back","average":{"1min":0.63700000000000001,"5min":0.91900000000000004,"15min":0.65600000000000003},"min":{"1min":0.32400000000000001,"5min":0.214,"15min":0.214},"max":{"1min":1.6739999999999999,"5min":6.8869999999999996,"15min":6.8869999999999996},"last":0.505},{"interface":"front","average":{"1min":0.79600000000000004,"5min":0.88100000000000001,"15min":0.64700000000000002},"min":{"1min":0.28399999999999997,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":2.1960000000000002,"5min":7.085,"15min":7.085},"last":0.33700000000000002}]},{"osd":3,"last update":"Tue Sep 10 19:18:04 2024","interfaces":[{"interface":"back","average":{"1min":1.0880000000000001,"5min":1.304,"15min":0.79300000000000004},"min":{"1min":0.29499999999999998,"5min":0.252,"15min":0.252},"max":{"1min":5.625,"5min":35.451999999999998,"15min":35.451999999999998},"last":0.26900000000000002},{"interface":"front","average":{"1min":1.867,"5min":1.5349999999999999,"15min":0.879},"min":{"1min":0.27600000000000002,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":7.9210000000000003,"5min":32.880000000000003,"15min":32.880000000000003},"last":0.40300000000000002}]},{"osd":5,"last update":"Tue Sep 10 19:18:25 2024","interfaces":[{"interface":"back","average":{"1min":1.3600000000000001,"5min":1.7749999999999999,"15min":0.84999999999999998},"min":{"1min":0.22500000000000001,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":3.1499999999999999,"5min":67.379000000000005,"15min":67.379000000000005},"last":0.372},{"interface":"front","average":{"1min":0.93600000000000005,"5min":1.4970000000000001,"15min":0.77500000000000002},"min":{"1min":0.27800000000000002,"5min":0.248,"15min":0.223},"max":{"1min":2.992,"5min":67.542000000000002,"15min":67.542000000000002},"last":0.55400000000000005}]},{"osd":6,"last update":"Tue Sep 10 19:18:56 2024","interfaces":[{"interface":"back","average":{"1min":1.6399999999999999,"5min":1.044,"15min":0.64200000000000002},"min":{"1min":0.48099999999999998,"5min":0.27400000000000002,"15min":0.253},"max":{"1min":4.7640000000000002,"5min":5.5289999999999999,"15min":5.5289999999999999},"last":0.57799999999999996},{"interface":"front","average":{"1min":1.5840000000000001,"5min":1.099,"15min":0.65300000000000002},"min":{"1min":0.27700000000000002,"5min":0.247,"15min":0.247},"max":{"1min":4.7469999999999999,"5min":7.9550000000000001,"15min":7.9550000000000001},"last":0.49199999999999999}]},{"osd":7,"last update":"Tue Sep 10 19:18:11 2024","interfaces":[{"interface":"back","average":{"1min":2.121,"5min":1.389,"15min":0.80400000000000005},"min":{"1min":0.29899999999999999,"5min":0.28100000000000003,"15min":0.27700000000000002},"max":{"1min":10.33,"5min":23.081,"15min":23.081},"last":0.54300000000000004},{"interface":"front","average":{"1min":1.627,"5min":1.3089999999999999,"15min":0.77100000000000002},"min":{"1min":0.23699999999999999,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":6.0190000000000001,"5min":22.666,"15min":22.666},"last":0.29099999999999998}]}]},{"osd":3,"up_from":26,"seq":111669149826,"num_pgs":84,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2419052,"kb_used_data":1820828,"kb_used_omap":1,"kb_used_meta":598206,"kb_avail":91326100,"statfs":{"total":95995035648,"available":93517926400,"internally_reserved":0,"allocated":1864527872,"data_stored":1863883639,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":612563288},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 19:18:38 2024","interfaces":[{"interface":"back","average":{"1min":0.32100000000000001,"5min":1.0569999999999999,"15min":0.69399999999999995},"min":{"1min":0.157,"5min":0.157,"15min":0.157},"max":{"1min":0.71199999999999997,"5min":17.518000000000001,"15min":17.518000000000001},"last":0.16},{"interface":"front","average":{"1min":0.56599999999999995,"5min":1.3520000000000001,"15min":0.82899999999999996},"min":{"1min":0.17899999999999999,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":2.2850000000000001,"5min":17.609000000000002,"15min":17.609000000000002},"last":0.37}]},{"osd":1,"last update":"Tue Sep 10 19:18:22 2024","interfaces":[{"interface":"back","average":{"1min":0.46400000000000002,"5min":0.51000000000000001,"15min":0.53000000000000003},"min":{"1min":0.219,"5min":0.219,"15min":0.219},"max":{"1min":1.3999999999999999,"5min":2.6059999999999999,"15min":2.6059999999999999},"last":0.20499999999999999},{"interface":"front","average":{"1min":0.56699999999999995,"5min":1.171,"15min":1.4359999999999999},"min":{"1min":0.22500000000000001,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":2.323,"5min":17.454999999999998,"15min":17.454999999999998},"last":0.41299999999999998}]},{"osd":2,"last update":"Tue Sep 10 19:18:38 2024","interfaces":[{"interface":"back","average":{"1min":0.41199999999999998,"5min":0.93799999999999994,"15min":0.64800000000000002},"min":{"1min":0.23599999999999999,"5min":0.20599999999999999,"15min":0.186},"max":{"1min":0.89500000000000002,"5min":5.3029999999999999,"15min":5.3029999999999999},"last":0.34899999999999998},{"interface":"front","average":{"1min":0.39900000000000002,"5min":1.0980000000000001,"15min":0.71299999999999997},"min":{"1min":0.27100000000000002,"5min":0.23300000000000001,"15min":0.19800000000000001},"max":{"1min":0.66900000000000004,"5min":17.806000000000001,"15min":17.806000000000001},"last":0.247}]},{"osd":4,"last update":"Tue Sep 10 19:18:05 2024","interfaces":[{"interface":"back","average":{"1min":0.91600000000000004,"5min":1.2370000000000001,"15min":0.82899999999999996},"min":{"1min":0.254,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":4.8639999999999999,"5min":17.834,"15min":17.834},"last":0.503},{"interface":"front","average":{"1min":0.93899999999999995,"5min":1.496,"15min":0.94799999999999995},"min":{"1min":0.28100000000000003,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":2.3610000000000002,"5min":17.553000000000001,"15min":17.553000000000001},"last":0.442}]},{"osd":5,"last update":"Tue Sep 10 19:18:22 2024","interfaces":[{"interface":"back","average":{"1min":1.0069999999999999,"5min":1.988,"15min":1.0840000000000001},"min":{"1min":0.28299999999999997,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":3.9510000000000001,"5min":80.942999999999998,"15min":80.942999999999998},"last":0.59699999999999998},{"interface":"front","average":{"1min":0.90300000000000002,"5min":2.2970000000000002,"15min":1.2250000000000001},"min":{"1min":0.33600000000000002,"5min":0.22,"15min":0.22},"max":{"1min":2.649,"5min":81.489000000000004,"15min":81.489000000000004},"last":1.7210000000000001}]},{"osd":6,"last update":"Tue Sep 10 19:18:46 2024","interfaces":[{"interface":"back","average":{"1min":0.94799999999999995,"5min":1.3180000000000001,"15min":0.90300000000000002},"min":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.30399999999999999},"max":{"1min":3.7999999999999998,"5min":17.861000000000001,"15min":17.861000000000001},"last":0.432},{"interface":"front","average":{"1min":0.72999999999999998,"5min":1.1759999999999999,"15min":0.79500000000000004},"min":{"1min":0.311,"5min":0.30499999999999999,"15min":0.26800000000000002},"max":{"1min":2.5449999999999999,"5min":17.687000000000001,"15min":17.687000000000001},"last":0.58499999999999996}]},{"osd":7,"last update":"Tue Sep 10 19:18:16 2024","interfaces":[{"interface":"back","average":{"1min":1.2050000000000001,"5min":1.288,"15min":0.86299999999999999},"min":{"1min":0.41399999999999998,"5min":0.31,"15min":0.29399999999999998},"max":{"1min":3.0779999999999998,"5min":18.093,"15min":18.093},"last":0.51900000000000002},{"interface":"front","average":{"1min":1.123,"5min":1.2929999999999999,"15min":0.89700000000000002},"min":{"1min":0.39300000000000002,"5min":0.35499999999999998,"15min":0.34100000000000003},"max":{"1min":3.3069999999999999,"5min":17.780999999999999,"15min":17.780999999999999},"last":0.39100000000000001}]}]},{"osd":2,"up_from":0,"seq":77309411463,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":0,"seq":592705486893,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":8,"seq":34359738513,"num_pgs":79,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1771348,"kb_used_data":1465672,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":91973804,"statfs":{"total":95995035648,"available":94181175296,"internally_reserved":0,"allocated":1500848128,"data_stored":1500216174,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Sep 10 19:18:27 2024","interfaces":[{"interface":"back","average":{"1min":0.32900000000000001,"5min":0.36299999999999999,"15min":0.36699999999999999},"min":{"1min":0.20000000000000001,"5min":0.19,"15min":0.19},"max":{"1min":0.56799999999999995,"5min":1.698,"15min":1.698},"last":0.48599999999999999},{"interface":"front","average":{"1min":0.33500000000000002,"5min":0.38800000000000001,"15min":0.41199999999999998},"min":{"1min":0.193,"5min":0.193,"15min":0.193},"max":{"1min":1.0489999999999999,"5min":1.0900000000000001,"15min":1.0900000000000001},"last":0.38200000000000001}]},{"osd":2,"last update":"Tue Sep 10 19:18:27 2024","interfaces":[{"interface":"back","average":{"1min":0.39300000000000002,"5min":0.90100000000000002,"15min":0.59499999999999997},"min":{"1min":0.22700000000000001,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":0.63200000000000001,"5min":6.4640000000000004,"15min":6.4640000000000004},"last":0.41099999999999998},{"interface":"front","average":{"1min":0.39300000000000002,"5min":0.92800000000000005,"15min":0.58399999999999996},"min":{"1min":0.22900000000000001,"5min":0.216,"15min":0.187},"max":{"1min":0.66100000000000003,"5min":9.9399999999999995,"15min":9.9399999999999995},"last":0.53400000000000003}]},{"osd":3,"last update":"Tue Sep 10 19:18:47 2024","interfaces":[{"interface":"back","average":{"1min":0.46200000000000002,"5min":0.91300000000000003,"15min":0.64400000000000002},"min":{"1min":0.22700000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":0.83099999999999996,"5min":6.5590000000000002,"15min":6.5590000000000002},"last":0.42799999999999999},{"interface":"front","average":{"1min":0.55000000000000004,"5min":1.0800000000000001,"15min":0.69899999999999995},"min":{"1min":0.26400000000000001,"5min":0.26200000000000001,"15min":0.23000000000000001},"max":{"1min":1.2649999999999999,"5min":7.2140000000000004,"15min":7.2140000000000004},"last":0.46999999999999997}]},{"osd":4,"last update":"Tue Sep 10 19:17:59 2024","interfaces":[{"interface":"back","average":{"1min":1.1419999999999999,"5min":0.90900000000000003,"15min":0.67700000000000005},"min":{"1min":0.29999999999999999,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":5.0789999999999997,"5min":5.8499999999999996,"15min":5.8499999999999996},"last":0.56999999999999995},{"interface":"front","average":{"1min":0.71599999999999997,"5min":0.84499999999999997,"15min":0.67600000000000005},"min":{"1min":0.28899999999999998,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":2.4940000000000002,"5min":5.2640000000000002,"15min":5.2640000000000002},"last":0.63700000000000001}]},{"osd":5,"last update":"Tue Sep 10 19:18:27 2024","interfaces":[{"interface":"back","average":{"1min":0.60999999999999999,"5min":1.05,"15min":0.755},"min":{"1min":0.311,"5min":0.29499999999999998,"15min":0.28199999999999997},"max":{"1min":1.3919999999999999,"5min":5.3860000000000001,"15min":5.3860000000000001},"last":0.58299999999999996},{"interface":"front","average":{"1min":0.77700000000000002,"5min":0.93999999999999995,"15min":0.72199999999999998},"min":{"1min":0.30099999999999999,"5min":0.253,"15min":0.253},"max":{"1min":1.6679999999999999,"5min":5.9749999999999996,"15min":5.9749999999999996},"last":0.623}]},{"osd":6,"last update":"Tue Sep 10 19:18:47 2024","interfaces":[{"interface":"back","average":{"1min":0.83599999999999997,"5min":1.008,"15min":0.751},"min":{"1min":0.34799999999999998,"5min":0.312,"15min":0.27200000000000002},"max":{"1min":3.5259999999999998,"5min":5.8730000000000002,"15min":5.8730000000000002},"last":0.66700000000000004},{"interface":"front","average":{"1min":1.036,"5min":0.999,"15min":0.76200000000000001},"min":{"1min":0.378,"5min":0.34399999999999997,"15min":0.28699999999999998},"max":{"1min":4.8300000000000001,"5min":5.2850000000000001,"15min":5.2850000000000001},"last":0.70899999999999996}]},{"osd":7,"last update":"Tue Sep 10 19:18:12 2024","interfaces":[{"interface":"back","average":{"1min":0.80600000000000005,"5min":0.98399999999999999,"15min":0.74199999999999999},"min":{"1min":0.35599999999999998,"5min":0.318,"15min":0.318},"max":{"1min":1.837,"5min":5.1479999999999997,"15min":5.1479999999999997},"last":0.67600000000000005},{"interface":"front","average":{"1min":1.228,"5min":1.159,"15min":0.76700000000000002},"min":{"1min":0.36699999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":5.3410000000000002,"5min":6.0469999999999997,"15min":6.0469999999999997},"last":0.68999999999999995}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":966,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1236,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1236,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":362,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":1760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":2118,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1018,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":2118,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1540,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1942,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":2162,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1480851456,"data_stored":1480851456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1861812224,"data_stored":1861812224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":2083979264,"data_stored":2083979264,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1861423104,"data_stored":1861419032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1679691776,"data_stored":1679687704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1944260608,"data_stored":1944256536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T19:19:05.614 INFO:tasks.thrashosds.thrasher:Setting 2.18 to [6, 3, 7, 0, 5, 4] 2024-09-10T19:19:05.614 INFO:tasks.thrashosds.thrasher:cmd ['osd', 'pg-upmap-items', '2.18', '6', '3', '7', '0', '5', '4'] 2024-09-10T19:19:05.614 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph osd pg-upmap-items 2.18 6 3 7 0 5 4 2024-09-10T19:19:05.631 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:19:05.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:05.632 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:19:05.733 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:19:05.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:05.755 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:19:05.755 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:19:05.856 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:19:05.890 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:19:05.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:05.891 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:19:05.992 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:19:06.020 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:19:06.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:06.021 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:19:06.121 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:19:06.156 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:19:06.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:06.157 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:19:06.257 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:19:06.284 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:19:06.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:06.285 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:19:06.386 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:19:06.413 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:19:06.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:06.414 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:19:06.515 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:19:06.540 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:19:06.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:06.541 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:19:06.642 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:19:06.669 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:19:06.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:06.670 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:19:06.771 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:19:06.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:06.804 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:19:06.804 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:19:06.905 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:19:06.932 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:19:06.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:06.933 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:19:07.027 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:19:06 smithi183 ceph-mon[29736]: pgmap v395: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 2.5 GiB data, 11 GiB used, 526 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.65k op/s; 2632/121818 objects misplaced (2.161%) 2024-09-10T19:19:07.027 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:19:06 smithi183 ceph-mon[29736]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T19:19:07.027 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:19:06 smithi183 ceph-mon[41562]: pgmap v395: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 2.5 GiB data, 11 GiB used, 526 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.65k op/s; 2632/121818 objects misplaced (2.161%) 2024-09-10T19:19:07.028 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:19:06 smithi183 ceph-mon[41562]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T19:19:07.034 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:19:07.066 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:19:07.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:07.067 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:19:07.157 INFO:teuthology.orchestra.run.smithi183.stdout:ERROR: (22) Invalid argument 2024-09-10T19:19:07.157 INFO:teuthology.orchestra.run.smithi183.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-10T19:19:07.168 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:19:07.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:07.211 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:19:07.211 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:19:07.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:19:06 smithi191 ceph-mon[32884]: pgmap v395: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 2.5 GiB data, 11 GiB used, 526 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.65k op/s; 2632/121818 objects misplaced (2.161%) 2024-09-10T19:19:07.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:19:06 smithi191 ceph-mon[32884]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T19:19:07.312 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:19:07.353 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:19:07.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:07.354 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:19:07.449 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:19:07.454 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:19:07.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:07.484 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:19:07.485 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:19:07.585 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:19:07.616 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:19:07.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:07.617 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:19:07.715 INFO:journalctl@ceph.mgr.x.smithi191.stdout:Sep 10 19:19:07 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-x[34967]: ::ffff:172.21.15.191 - - [10/Sep/2024:19:19:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T19:19:07.718 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:19:07.757 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:19:07.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:07.759 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:19:07.837 INFO:teuthology.orchestra.run.smithi183.stderr:noscrub is unset 2024-09-10T19:19:07.859 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:19:07.893 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:19:07.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:07.894 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:19:07.898 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph daemon osd.3 dump_historic_ops 2024-09-10T19:19:07.995 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:19:08.033 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:19:08.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:08.034 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:19:08.135 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:19:08.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:08.172 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:19:08.172 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:19:08.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:19:07 smithi183 ceph-mon[29736]: from='client.? 172.21.15.183:0/3931967263' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-10T19:19:08.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:19:07 smithi183 ceph-mon[29736]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T19:19:08.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:19:07 smithi183 ceph-mon[29736]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T19:19:08.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:19:07 smithi183 ceph-mon[29736]: from='mgr.14556 ' entity='mgr.y' 2024-09-10T19:19:08.181 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:19:07 smithi183 ceph-mon[41562]: from='client.? 172.21.15.183:0/3931967263' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-10T19:19:08.181 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:19:07 smithi183 ceph-mon[41562]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T19:19:08.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:19:07 smithi183 ceph-mon[41562]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T19:19:08.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:19:07 smithi183 ceph-mon[41562]: from='mgr.14556 ' entity='mgr.y' 2024-09-10T19:19:08.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:19:07 smithi191 ceph-mon[32884]: from='client.? 172.21.15.183:0/3931967263' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-10T19:19:08.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:19:07 smithi191 ceph-mon[32884]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T19:19:08.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:19:07 smithi191 ceph-mon[32884]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T19:19:08.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:19:07 smithi191 ceph-mon[32884]: from='mgr.14556 ' entity='mgr.y' 2024-09-10T19:19:08.273 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:19:08.309 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:19:08.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:08.310 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:19:08.411 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:19:08.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:08.445 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:19:08.446 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:19:08.547 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:19:08.572 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:19:08.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:08.573 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:19:08.674 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:19:08.711 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:19:08.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:08.712 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:19:08.812 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:19:08.840 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:19:08.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:08.841 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:19:08.931 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:19:08 smithi183 ceph-mon[29736]: pgmap v396: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 2.5 GiB data, 11 GiB used, 526 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.54k op/s; 2632/121818 objects misplaced (2.161%) 2024-09-10T19:19:08.931 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:19:08 smithi183 ceph-mon[29736]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T19:19:08.931 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:19:08 smithi183 ceph-mon[29736]: from='client.? 172.21.15.183:0/3931967263' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-10T19:19:08.931 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:19:08 smithi183 ceph-mon[29736]: osdmap e215: 8 total, 8 up, 6 in 2024-09-10T19:19:08.932 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:19:08 smithi183 ceph-mon[29736]: 2.1c scrub starts 2024-09-10T19:19:08.932 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:19:08 smithi183 ceph-mon[29736]: 2.1c scrub ok 2024-09-10T19:19:08.932 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:19:08 smithi183 ceph-mon[41562]: pgmap v396: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 2.5 GiB data, 11 GiB used, 526 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.54k op/s; 2632/121818 objects misplaced (2.161%) 2024-09-10T19:19:08.932 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:19:08 smithi183 ceph-mon[41562]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T19:19:08.932 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:19:08 smithi183 ceph-mon[41562]: from='client.? 172.21.15.183:0/3931967263' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-10T19:19:08.932 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:19:08 smithi183 ceph-mon[41562]: osdmap e215: 8 total, 8 up, 6 in 2024-09-10T19:19:08.932 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:19:08 smithi183 ceph-mon[41562]: 2.1c scrub starts 2024-09-10T19:19:08.932 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:19:08 smithi183 ceph-mon[41562]: 2.1c scrub ok 2024-09-10T19:19:08.943 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:19:08.978 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:19:08.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:08.979 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:19:09.080 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:19:09.106 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:19:09.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:09.107 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:19:09.208 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:19:09.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:19:08 smithi191 ceph-mon[32884]: pgmap v396: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 2.5 GiB data, 11 GiB used, 526 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.54k op/s; 2632/121818 objects misplaced (2.161%) 2024-09-10T19:19:09.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:19:08 smithi191 ceph-mon[32884]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T19:19:09.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:19:08 smithi191 ceph-mon[32884]: from='client.? 172.21.15.183:0/3931967263' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-10T19:19:09.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:19:08 smithi191 ceph-mon[32884]: osdmap e215: 8 total, 8 up, 6 in 2024-09-10T19:19:09.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:19:08 smithi191 ceph-mon[32884]: 2.1c scrub starts 2024-09-10T19:19:09.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:19:08 smithi191 ceph-mon[32884]: 2.1c scrub ok 2024-09-10T19:19:09.237 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:19:09.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:09.238 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:19:09.339 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:19:09.378 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:19:09.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:09.379 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:19:09.480 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:19:09.507 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:19:09.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:09.508 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:19:09.609 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:19:09.647 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:19:09.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:09.648 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:19:09.715 INFO:journalctl@ceph.iscsi.iscsi.a.smithi191.stdout:Sep 10 19:19:09 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-iscsi-iscsi-a[61816]: debug there is no tcmu-runner data available 2024-09-10T19:19:09.748 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:19:09.773 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:19:09.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:09.774 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:19:09.874 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:19:09.902 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:19:09.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:09.903 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:19:10.004 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:19:10.032 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:19:10.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:10.033 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:19:10.134 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:19:10.160 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:19:10.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:10.161 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:19:10.261 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:19:10.288 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:19:10.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:10.289 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:19:10.390 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:19:10.418 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:19:10.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:10.419 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:19:10.520 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:19:10.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:10.542 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:19:10.542 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:19:10.643 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:19:10.653 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-10T19:19:10.678 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:19:10.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:10.679 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:19:10.679 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:19:10 smithi183 ceph-mon[29736]: purged_snaps scrub starts 2024-09-10T19:19:10.679 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:19:10 smithi183 ceph-mon[29736]: purged_snaps scrub ok 2024-09-10T19:19:10.679 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:19:10 smithi183 ceph-mon[29736]: 6.8 scrub starts 2024-09-10T19:19:10.680 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:19:10 smithi183 ceph-mon[29736]: 6.8 scrub ok 2024-09-10T19:19:10.680 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:19:10 smithi183 ceph-mon[29736]: 6.1b scrub starts 2024-09-10T19:19:10.680 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:19:10 smithi183 ceph-mon[29736]: 6.1b scrub ok 2024-09-10T19:19:10.680 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:19:10 smithi183 ceph-mon[29736]: osdmap e216: 8 total, 8 up, 6 in 2024-09-10T19:19:10.680 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:19:10 smithi183 ceph-mon[29736]: from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:19:10.680 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:19:10 smithi183 ceph-mon[29736]: 6.0 scrub starts 2024-09-10T19:19:10.680 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:19:10 smithi183 ceph-mon[29736]: 6.0 scrub ok 2024-09-10T19:19:10.680 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:19:10 smithi183 ceph-mon[29736]: pgmap v399: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 2.5 GiB data, 11 GiB used, 526 GiB / 536 GiB avail; 639 B/s rd, 0 B/s wr, 922 op/s; 2632/121818 objects misplaced (2.161%) 2024-09-10T19:19:10.680 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:19:10 smithi183 ceph-mon[41562]: purged_snaps scrub starts 2024-09-10T19:19:10.680 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:19:10 smithi183 ceph-mon[41562]: purged_snaps scrub ok 2024-09-10T19:19:10.680 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:19:10 smithi183 ceph-mon[41562]: 6.8 scrub starts 2024-09-10T19:19:10.680 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:19:10 smithi183 ceph-mon[41562]: 6.8 scrub ok 2024-09-10T19:19:10.681 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:19:10 smithi183 ceph-mon[41562]: 6.1b scrub starts 2024-09-10T19:19:10.681 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:19:10 smithi183 ceph-mon[41562]: 6.1b scrub ok 2024-09-10T19:19:10.681 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:19:10 smithi183 ceph-mon[41562]: osdmap e216: 8 total, 8 up, 6 in 2024-09-10T19:19:10.681 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:19:10 smithi183 ceph-mon[41562]: from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:19:10.681 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:19:10 smithi183 ceph-mon[41562]: 6.0 scrub starts 2024-09-10T19:19:10.681 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:19:10 smithi183 ceph-mon[41562]: 6.0 scrub ok 2024-09-10T19:19:10.681 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:19:10 smithi183 ceph-mon[41562]: pgmap v399: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 2.5 GiB data, 11 GiB used, 526 GiB / 536 GiB avail; 639 B/s rd, 0 B/s wr, 922 op/s; 2632/121818 objects misplaced (2.161%) 2024-09-10T19:19:10.715 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:19:10 smithi191 ceph-mon[32884]: purged_snaps scrub starts 2024-09-10T19:19:10.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:19:10 smithi191 ceph-mon[32884]: purged_snaps scrub ok 2024-09-10T19:19:10.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:19:10 smithi191 ceph-mon[32884]: 6.8 scrub starts 2024-09-10T19:19:10.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:19:10 smithi191 ceph-mon[32884]: 6.8 scrub ok 2024-09-10T19:19:10.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:19:10 smithi191 ceph-mon[32884]: 6.1b scrub starts 2024-09-10T19:19:10.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:19:10 smithi191 ceph-mon[32884]: 6.1b scrub ok 2024-09-10T19:19:10.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:19:10 smithi191 ceph-mon[32884]: osdmap e216: 8 total, 8 up, 6 in 2024-09-10T19:19:10.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:19:10 smithi191 ceph-mon[32884]: from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:19:10.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:19:10 smithi191 ceph-mon[32884]: 6.0 scrub starts 2024-09-10T19:19:10.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:19:10 smithi191 ceph-mon[32884]: 6.0 scrub ok 2024-09-10T19:19:10.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:19:10 smithi191 ceph-mon[32884]: pgmap v399: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 2.5 GiB data, 11 GiB used, 526 GiB / 536 GiB avail; 639 B/s rd, 0 B/s wr, 922 op/s; 2632/121818 objects misplaced (2.161%) 2024-09-10T19:19:10.780 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:19:10.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:10.819 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:19:10.819 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:19:10.920 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:19:10.952 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:19:10.957 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:19:10.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:10.958 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:19:11.059 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:19:11.085 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:19:11.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:11.086 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:19:11.186 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:19:11.224 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:19:11.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:11.225 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:19:11.326 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:19:11.347 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:19:11.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:11.348 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:19:11.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:19:11 smithi183 ceph-mon[29736]: 6.3 scrub starts 2024-09-10T19:19:11.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:19:11 smithi183 ceph-mon[29736]: 6.3 scrub ok 2024-09-10T19:19:11.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:19:11 smithi183 ceph-mon[29736]: 2.15 scrub starts 2024-09-10T19:19:11.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:19:11 smithi183 ceph-mon[29736]: 2.15 scrub ok 2024-09-10T19:19:11.432 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Sep 10 19:19:11 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-y[29989]: ::ffff:172.21.15.191 - - [10/Sep/2024:19:19:11] "GET /metrics HTTP/1.1" 200 224337 "" "Prometheus/2.43.0" 2024-09-10T19:19:11.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:19:11 smithi183 ceph-mon[41562]: 6.3 scrub starts 2024-09-10T19:19:11.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:19:11 smithi183 ceph-mon[41562]: 6.3 scrub ok 2024-09-10T19:19:11.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:19:11 smithi183 ceph-mon[41562]: 2.15 scrub starts 2024-09-10T19:19:11.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:19:11 smithi183 ceph-mon[41562]: 2.15 scrub ok 2024-09-10T19:19:11.450 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:19:11.476 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:19:11.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:11.478 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:19:11.540 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:19:11.579 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:19:11.596 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:19:11.615 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:19:11.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:11.616 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:19:11.715 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:19:11 smithi191 ceph-mon[32884]: 6.3 scrub starts 2024-09-10T19:19:11.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:19:11 smithi191 ceph-mon[32884]: 6.3 scrub ok 2024-09-10T19:19:11.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:19:11 smithi191 ceph-mon[32884]: 2.15 scrub starts 2024-09-10T19:19:11.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:19:11 smithi191 ceph-mon[32884]: 2.15 scrub ok 2024-09-10T19:19:11.717 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:19:11.746 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:19:11.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:11.748 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:19:11.848 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:19:11.883 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:19:11.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:11.884 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:19:11.985 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:19:12.022 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:19:12.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:12.023 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:19:12.124 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:19:12.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:12.162 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:19:12.162 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:19:12.263 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:19:12.287 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:19:12.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:12.289 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:19:12.332 INFO:teuthology.orchestra.run.smithi183.stdout:osd.0: {} 2024-09-10T19:19:12.332 INFO:teuthology.orchestra.run.smithi183.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-10T19:19:12.347 INFO:teuthology.orchestra.run.smithi183.stdout:osd.1: {} 2024-09-10T19:19:12.348 INFO:teuthology.orchestra.run.smithi183.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-10T19:19:12.369 INFO:teuthology.orchestra.run.smithi183.stdout:osd.2: {} 2024-09-10T19:19:12.369 INFO:teuthology.orchestra.run.smithi183.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-10T19:19:12.370 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:19:12 smithi183 ceph-mon[29736]: 1.0 scrub starts 2024-09-10T19:19:12.370 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:19:12 smithi183 ceph-mon[29736]: 1.0 scrub ok 2024-09-10T19:19:12.370 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:19:12 smithi183 ceph-mon[29736]: 6.7 scrub starts 2024-09-10T19:19:12.370 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:19:12 smithi183 ceph-mon[29736]: 6.7 scrub ok 2024-09-10T19:19:12.370 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:19:12 smithi183 ceph-mon[29736]: 5.a scrub starts 2024-09-10T19:19:12.370 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:19:12 smithi183 ceph-mon[29736]: 5.a scrub ok 2024-09-10T19:19:12.370 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:19:12 smithi183 ceph-mon[29736]: pgmap v400: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 2.0 GiB data, 9.7 GiB used, 527 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.80k op/s; 2190/100749 objects misplaced (2.174%) 2024-09-10T19:19:12.370 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:19:12 smithi183 ceph-mon[41562]: 1.0 scrub starts 2024-09-10T19:19:12.370 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:19:12 smithi183 ceph-mon[41562]: 1.0 scrub ok 2024-09-10T19:19:12.370 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:19:12 smithi183 ceph-mon[41562]: 6.7 scrub starts 2024-09-10T19:19:12.370 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:19:12 smithi183 ceph-mon[41562]: 6.7 scrub ok 2024-09-10T19:19:12.371 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:19:12 smithi183 ceph-mon[41562]: 5.a scrub starts 2024-09-10T19:19:12.371 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:19:12 smithi183 ceph-mon[41562]: 5.a scrub ok 2024-09-10T19:19:12.371 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:19:12 smithi183 ceph-mon[41562]: pgmap v400: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 2.0 GiB data, 9.7 GiB used, 527 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.80k op/s; 2190/100749 objects misplaced (2.174%) 2024-09-10T19:19:12.389 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:19:12.392 INFO:teuthology.orchestra.run.smithi183.stdout:osd.3: {} 2024-09-10T19:19:12.392 INFO:teuthology.orchestra.run.smithi183.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-10T19:19:12.418 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:19:12.418 INFO:teuthology.orchestra.run.smithi183.stdout:osd.4: {} 2024-09-10T19:19:12.418 INFO:teuthology.orchestra.run.smithi183.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-10T19:19:12.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:12.419 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:19:12.440 INFO:teuthology.orchestra.run.smithi183.stdout:osd.5: {} 2024-09-10T19:19:12.440 INFO:teuthology.orchestra.run.smithi183.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-10T19:19:12.462 INFO:teuthology.orchestra.run.smithi183.stdout:osd.6: {} 2024-09-10T19:19:12.463 INFO:teuthology.orchestra.run.smithi183.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-10T19:19:12.491 INFO:teuthology.orchestra.run.smithi183.stdout:osd.7: {} 2024-09-10T19:19:12.491 INFO:teuthology.orchestra.run.smithi183.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-10T19:19:12.520 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:19:12.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:12.557 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:19:12.558 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:19:12.658 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:19:12.685 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:19:12.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:12.686 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:19:12.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:19:12 smithi191 ceph-mon[32884]: 1.0 scrub starts 2024-09-10T19:19:12.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:19:12 smithi191 ceph-mon[32884]: 1.0 scrub ok 2024-09-10T19:19:12.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:19:12 smithi191 ceph-mon[32884]: 6.7 scrub starts 2024-09-10T19:19:12.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:19:12 smithi191 ceph-mon[32884]: 6.7 scrub ok 2024-09-10T19:19:12.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:19:12 smithi191 ceph-mon[32884]: 5.a scrub starts 2024-09-10T19:19:12.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:19:12 smithi191 ceph-mon[32884]: 5.a scrub ok 2024-09-10T19:19:12.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:19:12 smithi191 ceph-mon[32884]: pgmap v400: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 2.0 GiB data, 9.7 GiB used, 527 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.80k op/s; 2190/100749 objects misplaced (2.174%) 2024-09-10T19:19:12.787 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:19:12.811 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:19:12.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:12.812 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:19:12.913 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:19:12.940 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:19:12.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:12.941 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:19:13.042 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:19:13.068 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:19:13.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:13.069 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:19:13.170 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:19:13.196 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:19:13.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:13.197 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:19:13.297 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:19:13.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:13.332 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:19:13.332 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:19:13.433 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:19:13.459 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:19:13.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:13.460 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:19:13.560 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:19:13.587 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:19:13.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:13.588 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:19:13.681 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:19:13 smithi183 ceph-mon[29736]: 4.0 scrub starts 2024-09-10T19:19:13.681 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:19:13 smithi183 ceph-mon[29736]: 4.0 scrub ok 2024-09-10T19:19:13.681 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:19:13 smithi183 ceph-mon[29736]: from='client.? 172.21.15.183:0/70078447' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T19:19:13.682 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:19:13 smithi183 ceph-mon[29736]: 6.17 scrub starts 2024-09-10T19:19:13.682 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:19:13 smithi183 ceph-mon[29736]: 6.17 scrub ok 2024-09-10T19:19:13.682 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:19:13 smithi183 ceph-mon[41562]: 4.0 scrub starts 2024-09-10T19:19:13.682 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:19:13 smithi183 ceph-mon[41562]: 4.0 scrub ok 2024-09-10T19:19:13.682 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:19:13 smithi183 ceph-mon[41562]: from='client.? 172.21.15.183:0/70078447' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T19:19:13.682 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:19:13 smithi183 ceph-mon[41562]: 6.17 scrub starts 2024-09-10T19:19:13.682 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:19:13 smithi183 ceph-mon[41562]: 6.17 scrub ok 2024-09-10T19:19:13.688 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:19:13.714 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:19:13.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:13.715 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:19:13.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:19:13 smithi191 ceph-mon[32884]: 4.0 scrub starts 2024-09-10T19:19:13.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:19:13 smithi191 ceph-mon[32884]: 4.0 scrub ok 2024-09-10T19:19:13.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:19:13 smithi191 ceph-mon[32884]: from='client.? 172.21.15.183:0/70078447' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T19:19:13.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:19:13 smithi191 ceph-mon[32884]: 6.17 scrub starts 2024-09-10T19:19:13.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:19:13 smithi191 ceph-mon[32884]: 6.17 scrub ok 2024-09-10T19:19:13.817 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:19:13.842 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:19:13.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:13.843 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:19:13.944 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:19:13.971 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:19:13.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:13.972 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:19:14.073 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:19:14.099 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:19:14.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:14.100 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:19:14.201 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:19:14.236 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:19:14.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:14.237 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:19:14.338 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:19:14.364 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:19:14.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:14.365 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:19:14.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:19:14 smithi183 ceph-mon[29736]: 6.11 scrub starts 2024-09-10T19:19:14.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:19:14 smithi183 ceph-mon[29736]: 6.11 scrub ok 2024-09-10T19:19:14.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:19:14 smithi183 ceph-mon[29736]: 4.1a scrub starts 2024-09-10T19:19:14.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:19:14 smithi183 ceph-mon[29736]: 4.1a scrub ok 2024-09-10T19:19:14.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:19:14 smithi183 ceph-mon[29736]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:19:14.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:19:14 smithi183 ceph-mon[29736]: pgmap v401: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 2.0 GiB data, 9.7 GiB used, 527 GiB / 536 GiB avail; 639 B/s rd, 0 B/s wr, 878 op/s; 2190/100749 objects misplaced (2.174%) 2024-09-10T19:19:14.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:19:14 smithi183 ceph-mon[41562]: 6.11 scrub starts 2024-09-10T19:19:14.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:19:14 smithi183 ceph-mon[41562]: 6.11 scrub ok 2024-09-10T19:19:14.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:19:14 smithi183 ceph-mon[41562]: 4.1a scrub starts 2024-09-10T19:19:14.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:19:14 smithi183 ceph-mon[41562]: 4.1a scrub ok 2024-09-10T19:19:14.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:19:14 smithi183 ceph-mon[41562]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:19:14.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:19:14 smithi183 ceph-mon[41562]: pgmap v401: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 2.0 GiB data, 9.7 GiB used, 527 GiB / 536 GiB avail; 639 B/s rd, 0 B/s wr, 878 op/s; 2190/100749 objects misplaced (2.174%) 2024-09-10T19:19:14.466 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:19:14.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:14.504 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:19:14.504 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:19:14.605 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:19:14.634 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:19:14.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:14.635 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:19:14.715 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:19:14 smithi191 ceph-mon[32884]: 6.11 scrub starts 2024-09-10T19:19:14.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:19:14 smithi191 ceph-mon[32884]: 6.11 scrub ok 2024-09-10T19:19:14.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:19:14 smithi191 ceph-mon[32884]: 4.1a scrub starts 2024-09-10T19:19:14.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:19:14 smithi191 ceph-mon[32884]: 4.1a scrub ok 2024-09-10T19:19:14.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:19:14 smithi191 ceph-mon[32884]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:19:14.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:19:14 smithi191 ceph-mon[32884]: pgmap v401: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 2.0 GiB data, 9.7 GiB used, 527 GiB / 536 GiB avail; 639 B/s rd, 0 B/s wr, 878 op/s; 2190/100749 objects misplaced (2.174%) 2024-09-10T19:19:14.736 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:19:14.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:14.772 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:19:14.773 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:19:14.873 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:19:14.899 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:19:14.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:14.900 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:19:15.000 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:19:15.039 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:19:15.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:15.040 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:19:15.140 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:19:15.173 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:19:15.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:15.174 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:19:15.275 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:19:15.308 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:19:15.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:15.309 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:19:15.410 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:19:15.448 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:19:15.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:15.449 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:19:15.550 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:19:15.575 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:19:15.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:15.576 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:19:15.677 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:19:15.681 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:19:15 smithi183 ceph-mon[41562]: 4.a scrub starts 2024-09-10T19:19:15.681 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:19:15 smithi183 ceph-mon[41562]: 4.a scrub ok 2024-09-10T19:19:15.681 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:19:15 smithi183 ceph-mon[41562]: 5.4 scrub starts 2024-09-10T19:19:15.681 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:19:15 smithi183 ceph-mon[41562]: 5.4 scrub ok 2024-09-10T19:19:15.682 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:19:15 smithi183 ceph-mon[41562]: 6.1f scrub starts 2024-09-10T19:19:15.682 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:19:15 smithi183 ceph-mon[41562]: 6.1f scrub ok 2024-09-10T19:19:15.682 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:19:15 smithi183 ceph-mon[29736]: 4.a scrub starts 2024-09-10T19:19:15.682 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:19:15 smithi183 ceph-mon[29736]: 4.a scrub ok 2024-09-10T19:19:15.682 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:19:15 smithi183 ceph-mon[29736]: 5.4 scrub starts 2024-09-10T19:19:15.682 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:19:15 smithi183 ceph-mon[29736]: 5.4 scrub ok 2024-09-10T19:19:15.682 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:19:15 smithi183 ceph-mon[29736]: 6.1f scrub starts 2024-09-10T19:19:15.682 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:19:15 smithi183 ceph-mon[29736]: 6.1f scrub ok 2024-09-10T19:19:15.706 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:19:15.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:15.707 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:19:15.715 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:19:15 smithi191 ceph-mon[32884]: 4.a scrub starts 2024-09-10T19:19:15.715 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:19:15 smithi191 ceph-mon[32884]: 4.a scrub ok 2024-09-10T19:19:15.715 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:19:15 smithi191 ceph-mon[32884]: 5.4 scrub starts 2024-09-10T19:19:15.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:19:15 smithi191 ceph-mon[32884]: 5.4 scrub ok 2024-09-10T19:19:15.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:19:15 smithi191 ceph-mon[32884]: 6.1f scrub starts 2024-09-10T19:19:15.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:19:15 smithi191 ceph-mon[32884]: 6.1f scrub ok 2024-09-10T19:19:15.808 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:19:15.835 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:19:15.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:15.835 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:19:15.936 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:19:15.965 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:19:15.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:15.966 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:19:16.067 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:19:16.093 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:19:16.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:16.095 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:19:16.195 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:19:16.232 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:19:16.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:16.233 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:19:16.334 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:19:16.363 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:19:16.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:16.364 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:19:16.465 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:19:16.493 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:19:16.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:16.494 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:19:16.595 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:19:16.597 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:19:16 smithi183 ceph-mon[29736]: 2.1b scrub starts 2024-09-10T19:19:16.597 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:19:16 smithi183 ceph-mon[29736]: 2.1b scrub ok 2024-09-10T19:19:16.597 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:19:16 smithi183 ceph-mon[29736]: 6.a scrub starts 2024-09-10T19:19:16.597 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:19:16 smithi183 ceph-mon[29736]: 6.a scrub ok 2024-09-10T19:19:16.597 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:19:16 smithi183 ceph-mon[29736]: pgmap v402: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 1.6 GiB data, 8.3 GiB used, 528 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.76k op/s; 1744/79473 objects misplaced (2.194%) 2024-09-10T19:19:16.597 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:19:16 smithi183 ceph-mon[41562]: 2.1b scrub starts 2024-09-10T19:19:16.598 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:19:16 smithi183 ceph-mon[41562]: 2.1b scrub ok 2024-09-10T19:19:16.598 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:19:16 smithi183 ceph-mon[41562]: 6.a scrub starts 2024-09-10T19:19:16.598 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:19:16 smithi183 ceph-mon[41562]: 6.a scrub ok 2024-09-10T19:19:16.598 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:19:16 smithi183 ceph-mon[41562]: pgmap v402: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 1.6 GiB data, 8.3 GiB used, 528 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.76k op/s; 1744/79473 objects misplaced (2.194%) 2024-09-10T19:19:16.629 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:19:16.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:16.631 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:19:16.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:19:16 smithi191 ceph-mon[32884]: 2.1b scrub starts 2024-09-10T19:19:16.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:19:16 smithi191 ceph-mon[32884]: 2.1b scrub ok 2024-09-10T19:19:16.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:19:16 smithi191 ceph-mon[32884]: 6.a scrub starts 2024-09-10T19:19:16.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:19:16 smithi191 ceph-mon[32884]: 6.a scrub ok 2024-09-10T19:19:16.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:19:16 smithi191 ceph-mon[32884]: pgmap v402: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 1.6 GiB data, 8.3 GiB used, 528 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.76k op/s; 1744/79473 objects misplaced (2.194%) 2024-09-10T19:19:16.731 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:19:16.757 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:19:16.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:16.758 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:19:16.859 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:19:16.886 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:19:16.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:16.887 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:19:16.988 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:19:17.030 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:19:17.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:17.031 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:19:17.132 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:19:17.167 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:19:17.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:17.168 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:19:17.269 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:19:17.301 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:19:17.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:17.302 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:19:17.403 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:19:17.442 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:19:17.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:17.443 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:19:17.521 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:19:17 smithi183 ceph-mon[29736]: 5.14 scrub starts 2024-09-10T19:19:17.521 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:19:17 smithi183 ceph-mon[29736]: 5.14 scrub ok 2024-09-10T19:19:17.521 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:19:17 smithi183 ceph-mon[29736]: 4.16 scrub starts 2024-09-10T19:19:17.521 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:19:17 smithi183 ceph-mon[29736]: 4.16 scrub ok 2024-09-10T19:19:17.521 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:19:17 smithi183 ceph-mon[41562]: 5.14 scrub starts 2024-09-10T19:19:17.521 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:19:17 smithi183 ceph-mon[41562]: 5.14 scrub ok 2024-09-10T19:19:17.521 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:19:17 smithi183 ceph-mon[41562]: 4.16 scrub starts 2024-09-10T19:19:17.521 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:19:17 smithi183 ceph-mon[41562]: 4.16 scrub ok 2024-09-10T19:19:17.544 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:19:17.570 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:19:17.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:17.571 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:19:17.672 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:19:17.675 INFO:teuthology.orchestra.run.smithi183.stdout:{ 2024-09-10T19:19:17.675 INFO:teuthology.orchestra.run.smithi183.stdout: "size": 20, 2024-09-10T19:19:17.675 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 600, 2024-09-10T19:19:17.675 INFO:teuthology.orchestra.run.smithi183.stdout: "ops": [ 2024-09-10T19:19:17.676 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:19:17.676 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.15051.0:59075 8.3 8:cfee2355:::benchmark_data_smithi183_114924_object59074:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e166)", 2024-09-10T19:19:17.676 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-10T19:16:55.988767+0000", 2024-09-10T19:19:17.676 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 141.68023541299999, 2024-09-10T19:19:17.676 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.8076374039999998, 2024-09-10T19:19:17.676 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-10T19:19:17.676 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T19:19:17.676 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-10T19:19:17.676 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.15051", 2024-09-10T19:19:17.676 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.183:0/3632028672", 2024-09-10T19:19:17.676 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 59075 2024-09-10T19:19:17.676 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:19:17.676 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-10T19:19:17.677 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:19:17.677 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-10T19:19:17.677 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:55.988767+0000", 2024-09-10T19:19:17.677 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:19:17.677 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:19:17.677 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:19:17.677 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-10T19:19:17.677 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:55.988767+0000", 2024-09-10T19:19:17.677 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:19:17.677 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:19:17.677 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:19:17.677 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-10T19:19:17.677 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:55.988765+0000", 2024-09-10T19:19:17.678 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4294967295.999999 2024-09-10T19:19:17.678 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:19:17.678 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:19:17.678 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-10T19:19:17.678 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:55.988810+0000", 2024-09-10T19:19:17.678 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.4308999999999999e-05 2024-09-10T19:19:17.678 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:19:17.678 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:19:17.678 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-10T19:19:17.678 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:55.988820+0000", 2024-09-10T19:19:17.678 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.9699999999999994e-06 2024-09-10T19:19:17.678 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:19:17.678 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:19:17.678 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-10T19:19:17.678 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:55.988825+0000", 2024-09-10T19:19:17.678 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.198e-06 2024-09-10T19:19:17.679 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:19:17.679 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:19:17.679 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-10T19:19:17.679 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:55.988871+0000", 2024-09-10T19:19:17.679 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.5540000000000001e-05 2024-09-10T19:19:17.679 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:19:17.679 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:19:17.679 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for peered", 2024-09-10T19:19:17.679 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:55.988881+0000", 2024-09-10T19:19:17.679 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.0849000000000001e-05 2024-09-10T19:19:17.680 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:19:17.680 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:19:17.680 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-10T19:19:17.680 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:57.774576+0000", 2024-09-10T19:19:17.680 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.7856943159999998 2024-09-10T19:19:17.680 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:19:17.680 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:19:17.680 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-10T19:19:17.680 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:57.775578+0000", 2024-09-10T19:19:17.680 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0010023149999999999 2024-09-10T19:19:17.680 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:19:17.680 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:19:17.681 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 5,7", 2024-09-10T19:19:17.681 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:57.775706+0000", 2024-09-10T19:19:17.681 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00012781399999999999 2024-09-10T19:19:17.681 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:19:17.681 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:19:17.681 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-10T19:19:17.681 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:57.781861+0000", 2024-09-10T19:19:17.681 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0061548009999999997 2024-09-10T19:19:17.681 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:19:17.681 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:19:17.681 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:19:17.681 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:57.795208+0000", 2024-09-10T19:19:17.682 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.013347314000000001 2024-09-10T19:19:17.682 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:19:17.682 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:19:17.682 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:19:17.682 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:57.795320+0000", 2024-09-10T19:19:17.682 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000112193 2024-09-10T19:19:17.682 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:19:17.682 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:19:17.682 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-10T19:19:17.682 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:57.795357+0000", 2024-09-10T19:19:17.682 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.6838999999999997e-05 2024-09-10T19:19:17.682 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:19:17.682 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:19:17.682 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-10T19:19:17.682 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:57.796404+0000", 2024-09-10T19:19:17.683 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.001047014 2024-09-10T19:19:17.683 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:19:17.683 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:19:17.683 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:19:17.683 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:19:17.683 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:19:17.683 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.15051.0:59116 8.3 8:ce8335d8:::benchmark_data_smithi183_114924_object59115:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e166)", 2024-09-10T19:19:17.683 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-10T19:16:55.988835+0000", 2024-09-10T19:19:17.683 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 141.68016654199999, 2024-09-10T19:19:17.683 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.808536116, 2024-09-10T19:19:17.683 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-10T19:19:17.683 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T19:19:17.683 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-10T19:19:17.683 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.15051", 2024-09-10T19:19:17.684 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.183:0/3632028672", 2024-09-10T19:19:17.684 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 59116 2024-09-10T19:19:17.684 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:19:17.684 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-10T19:19:17.684 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:19:17.684 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-10T19:19:17.684 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:55.988835+0000", 2024-09-10T19:19:17.684 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:19:17.684 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:19:17.684 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:19:17.684 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-10T19:19:17.684 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:55.988835+0000", 2024-09-10T19:19:17.684 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:19:17.684 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:19:17.684 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:19:17.685 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-10T19:19:17.685 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:55.988835+0000", 2024-09-10T19:19:17.685 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4294967295.9999995 2024-09-10T19:19:17.685 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:19:17.685 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:19:17.685 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-10T19:19:17.685 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:55.988882+0000", 2024-09-10T19:19:17.685 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.6919000000000003e-05 2024-09-10T19:19:17.685 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:19:17.685 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:19:17.685 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-10T19:19:17.686 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:55.988889+0000", 2024-09-10T19:19:17.686 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.7669999999999998e-06 2024-09-10T19:19:17.686 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:19:17.686 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:19:17.686 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-10T19:19:17.686 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:55.988892+0000", 2024-09-10T19:19:17.686 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.6430000000000001e-06 2024-09-10T19:19:17.686 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:19:17.686 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:19:17.686 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-10T19:19:17.686 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:55.988950+0000", 2024-09-10T19:19:17.686 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.8112000000000002e-05 2024-09-10T19:19:17.686 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:19:17.687 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:19:17.687 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for peered", 2024-09-10T19:19:17.687 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:55.988958+0000", 2024-09-10T19:19:17.687 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.7859999999999997e-06 2024-09-10T19:19:17.687 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:19:17.687 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:19:17.687 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-10T19:19:17.687 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:57.777269+0000", 2024-09-10T19:19:17.687 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.788310617 2024-09-10T19:19:17.687 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:19:17.687 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:19:17.687 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-10T19:19:17.687 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:57.777364+0000", 2024-09-10T19:19:17.688 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.5545999999999996e-05 2024-09-10T19:19:17.688 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:19:17.688 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:19:17.688 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 5,7", 2024-09-10T19:19:17.688 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:57.777470+0000", 2024-09-10T19:19:17.688 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000105912 2024-09-10T19:19:17.688 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:19:17.688 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:19:17.688 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-10T19:19:17.688 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:57.783922+0000", 2024-09-10T19:19:17.688 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0064513679999999999 2024-09-10T19:19:17.688 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:19:17.688 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:19:17.689 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:19:17.689 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:57.795246+0000", 2024-09-10T19:19:17.689 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.011324855999999999 2024-09-10T19:19:17.689 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:19:17.689 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:19:17.689 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:19:17.689 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:57.796516+0000", 2024-09-10T19:19:17.689 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0012691950000000001 2024-09-10T19:19:17.689 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:19:17.689 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:19:17.690 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-10T19:19:17.690 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:57.796551+0000", 2024-09-10T19:19:17.690 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.4869000000000001e-05 2024-09-10T19:19:17.690 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:19:17.690 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:19:17.690 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-10T19:19:17.690 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:57.797372+0000", 2024-09-10T19:19:17.690 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00082099899999999999 2024-09-10T19:19:17.690 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:19:17.690 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:19:17.690 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:19:17.690 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:19:17.690 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:19:17.690 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.15051.0:59117 8.3 8:c2a12ec4:::benchmark_data_smithi183_114924_object59116:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e166)", 2024-09-10T19:19:17.690 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-10T19:16:55.988897+0000", 2024-09-10T19:19:17.691 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 141.68010461399999, 2024-09-10T19:19:17.691 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.8096371680000001, 2024-09-10T19:19:17.691 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-10T19:19:17.691 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T19:19:17.691 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-10T19:19:17.691 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.15051", 2024-09-10T19:19:17.691 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.183:0/3632028672", 2024-09-10T19:19:17.691 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 59117 2024-09-10T19:19:17.691 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:19:17.691 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-10T19:19:17.691 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:19:17.692 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-10T19:19:17.692 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:55.988897+0000", 2024-09-10T19:19:17.692 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:19:17.692 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:19:17.692 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:19:17.692 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-10T19:19:17.692 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:55.988897+0000", 2024-09-10T19:19:17.692 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:19:17.692 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:19:17.692 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:19:17.693 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-10T19:19:17.693 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:55.988897+0000", 2024-09-10T19:19:17.693 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4294967295.9999995 2024-09-10T19:19:17.693 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:19:17.693 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:19:17.693 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-10T19:19:17.693 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:55.988947+0000", 2024-09-10T19:19:17.693 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.0318000000000002e-05 2024-09-10T19:19:17.693 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:19:17.693 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:19:17.693 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-10T19:19:17.693 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:55.988952+0000", 2024-09-10T19:19:17.694 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.1440000000000002e-06 2024-09-10T19:19:17.694 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:19:17.694 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:19:17.694 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-10T19:19:17.694 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:55.988955+0000", 2024-09-10T19:19:17.694 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.6850000000000001e-06 2024-09-10T19:19:17.694 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:19:17.694 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:19:17.694 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-10T19:19:17.694 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:55.988986+0000", 2024-09-10T19:19:17.694 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.0629000000000001e-05 2024-09-10T19:19:17.694 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:19:17.694 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:19:17.695 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for peered", 2024-09-10T19:19:17.695 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:55.988990+0000", 2024-09-10T19:19:17.695 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.2729999999999996e-06 2024-09-10T19:19:17.695 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:19:17.695 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:19:17.695 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-10T19:19:17.695 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:57.778622+0000", 2024-09-10T19:19:17.695 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.7896323029999999 2024-09-10T19:19:17.695 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:19:17.695 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:19:17.695 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-10T19:19:17.695 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:57.778713+0000", 2024-09-10T19:19:17.695 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.0404999999999999e-05 2024-09-10T19:19:17.696 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:19:17.696 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:19:17.696 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 5,7", 2024-09-10T19:19:17.696 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:57.778815+0000", 2024-09-10T19:19:17.696 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000102346 2024-09-10T19:19:17.696 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:19:17.696 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:19:17.696 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-10T19:19:17.696 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:57.783939+0000", 2024-09-10T19:19:17.696 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0051238949999999998 2024-09-10T19:19:17.696 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:19:17.696 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:19:17.696 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:19:17.697 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:57.796456+0000", 2024-09-10T19:19:17.697 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.012517240000000001 2024-09-10T19:19:17.697 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:19:17.697 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:19:17.697 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:19:17.697 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:57.797588+0000", 2024-09-10T19:19:17.697 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0011322820000000001 2024-09-10T19:19:17.697 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:19:17.697 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:19:17.697 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-10T19:19:17.697 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:57.797644+0000", 2024-09-10T19:19:17.697 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.6156000000000002e-05 2024-09-10T19:19:17.697 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:19:17.698 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:19:17.698 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-10T19:19:17.698 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:57.798535+0000", 2024-09-10T19:19:17.698 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000890024 2024-09-10T19:19:17.698 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:19:17.698 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:19:17.698 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:19:17.698 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:19:17.698 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:19:17.698 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.15051.0:59124 8.3 8:cab1e759:::benchmark_data_smithi183_114924_object59123:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e166)", 2024-09-10T19:19:17.699 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-10T19:16:55.988960+0000", 2024-09-10T19:19:17.699 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 141.68004227399999, 2024-09-10T19:19:17.699 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.810490784, 2024-09-10T19:19:17.699 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-10T19:19:17.699 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T19:19:17.699 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-10T19:19:17.699 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.15051", 2024-09-10T19:19:17.699 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.183:0/3632028672", 2024-09-10T19:19:17.700 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 59124 2024-09-10T19:19:17.700 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:19:17.700 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-10T19:19:17.700 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:19:17.700 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-10T19:19:17.700 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:55.988960+0000", 2024-09-10T19:19:17.700 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:19:17.700 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:19:17.700 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:19:17.700 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-10T19:19:17.700 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:55.988960+0000", 2024-09-10T19:19:17.700 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:19:17.701 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:19:17.701 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:19:17.701 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-10T19:19:17.701 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:55.988959+0000", 2024-09-10T19:19:17.701 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4294967295.9999995 2024-09-10T19:19:17.701 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:19:17.701 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:19:17.701 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-10T19:19:17.701 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:55.989000+0000", 2024-09-10T19:19:17.701 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.0720000000000003e-05 2024-09-10T19:19:17.701 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:19:17.702 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:19:17.702 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-10T19:19:17.702 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:55.989006+0000", 2024-09-10T19:19:17.702 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.5490000000000003e-06 2024-09-10T19:19:17.702 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:19:17.702 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:19:17.702 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-10T19:19:17.702 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:55.989010+0000", 2024-09-10T19:19:17.702 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.2140000000000001e-06 2024-09-10T19:19:17.703 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:19:17.703 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:19:17.703 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-10T19:19:17.703 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:55.989031+0000", 2024-09-10T19:19:17.703 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.0945999999999999e-05 2024-09-10T19:19:17.703 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:19:17.703 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:19:17.703 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for peered", 2024-09-10T19:19:17.703 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:55.989034+0000", 2024-09-10T19:19:17.703 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.1310000000000002e-06 2024-09-10T19:19:17.703 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:19:17.703 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:19:17.704 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-10T19:19:17.704 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:57.779721+0000", 2024-09-10T19:19:17.704 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.7906875069999999 2024-09-10T19:19:17.704 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:19:17.704 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:19:17.704 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-10T19:19:17.704 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:57.779826+0000", 2024-09-10T19:19:17.704 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000104394 2024-09-10T19:19:17.704 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:19:17.704 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:19:17.704 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 5,7", 2024-09-10T19:19:17.704 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:57.779934+0000", 2024-09-10T19:19:17.704 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000108115 2024-09-10T19:19:17.704 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:19:17.705 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:19:17.705 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-10T19:19:17.705 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:57.783943+0000", 2024-09-10T19:19:17.705 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0040090689999999997 2024-09-10T19:19:17.705 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:19:17.705 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:19:17.705 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:19:17.705 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:57.796489+0000", 2024-09-10T19:19:17.705 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.012546148 2024-09-10T19:19:17.705 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:19:17.705 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:19:17.705 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:19:17.706 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:57.798572+0000", 2024-09-10T19:19:17.706 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0020831249999999999 2024-09-10T19:19:17.706 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:19:17.707 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:19:17.707 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-10T19:19:17.707 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:57.798604+0000", 2024-09-10T19:19:17.707 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.2206000000000003e-05 2024-09-10T19:19:17.707 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:19:17.707 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:19:17.707 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-10T19:19:17.707 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:57.799450+0000", 2024-09-10T19:19:17.707 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00084613600000000005 2024-09-10T19:19:17.707 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:19:17.707 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:19:17.707 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:19:17.707 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:19:17.707 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:19:17.708 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.15051.0:59130 8.3 8:c3849915:::benchmark_data_smithi183_114924_object59129:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e166)", 2024-09-10T19:19:17.708 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-10T19:16:55.989014+0000", 2024-09-10T19:19:17.708 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 141.67998749899999, 2024-09-10T19:19:17.708 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.824075119, 2024-09-10T19:19:17.708 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-10T19:19:17.708 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T19:19:17.708 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-10T19:19:17.708 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.15051", 2024-09-10T19:19:17.708 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.183:0/3632028672", 2024-09-10T19:19:17.708 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 59130 2024-09-10T19:19:17.708 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:19:17.708 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-10T19:19:17.709 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:19:17.709 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-10T19:19:17.709 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:55.989014+0000", 2024-09-10T19:19:17.709 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:19:17.709 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:19:17.709 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:19:17.709 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-10T19:19:17.709 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:55.989014+0000", 2024-09-10T19:19:17.709 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:19:17.709 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:19:17.709 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:19:17.710 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-10T19:19:17.710 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:55.989014+0000", 2024-09-10T19:19:17.710 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4294967295.9999995 2024-09-10T19:19:17.710 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:19:17.710 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:19:17.710 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-10T19:19:17.710 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:55.989054+0000", 2024-09-10T19:19:17.710 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.0370000000000001e-05 2024-09-10T19:19:17.710 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:19:17.710 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:19:17.710 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-10T19:19:17.710 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:55.989061+0000", 2024-09-10T19:19:17.710 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.331e-06 2024-09-10T19:19:17.710 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:19:17.710 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:19:17.711 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-10T19:19:17.711 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:55.989064+0000", 2024-09-10T19:19:17.711 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.0199999999999999e-06 2024-09-10T19:19:17.711 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:19:17.711 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:19:17.711 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-10T19:19:17.711 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:55.989084+0000", 2024-09-10T19:19:17.711 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.0282999999999999e-05 2024-09-10T19:19:17.711 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:19:17.711 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:19:17.711 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for peered", 2024-09-10T19:19:17.711 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:55.989087+0000", 2024-09-10T19:19:17.711 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.7829999999999999e-06 2024-09-10T19:19:17.711 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:19:17.711 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:19:17.711 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-10T19:19:17.712 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:57.780815+0000", 2024-09-10T19:19:17.712 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.791727879 2024-09-10T19:19:17.712 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:19:17.712 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:19:17.712 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-10T19:19:17.712 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:57.780898+0000", 2024-09-10T19:19:17.712 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.3137e-05 2024-09-10T19:19:17.712 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:19:17.712 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:19:17.712 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 5,7", 2024-09-10T19:19:17.712 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:57.781002+0000", 2024-09-10T19:19:17.713 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00010370300000000001 2024-09-10T19:19:17.713 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:19:17.713 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:19:17.713 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-10T19:19:17.713 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:57.786090+0000", 2024-09-10T19:19:17.713 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0050885940000000001 2024-09-10T19:19:17.713 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:19:17.713 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:19:17.714 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:19:17.714 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:57.797412+0000", 2024-09-10T19:19:17.714 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.011321827 2024-09-10T19:19:17.714 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:19:17.714 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:19:17.714 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:19:17.714 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:57.811332+0000", 2024-09-10T19:19:17.714 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.013919617 2024-09-10T19:19:17.714 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:19:17.714 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:19:17.714 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-10T19:19:17.714 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:57.811389+0000", 2024-09-10T19:19:17.715 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.6910999999999998e-05 2024-09-10T19:19:17.715 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:19:17.715 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:19:17.715 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-10T19:19:17.715 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:57.813090+0000", 2024-09-10T19:19:17.715 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.001701073 2024-09-10T19:19:17.715 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:19:17.715 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:19:17.715 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:19:17.715 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:19:17.715 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:19:17.715 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.15051.0:59134 8.3 8:c040b2cd:::benchmark_data_smithi183_114924_object59133:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e166)", 2024-09-10T19:19:17.715 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-10T19:16:55.989069+0000", 2024-09-10T19:19:17.716 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 141.67993337799999, 2024-09-10T19:19:17.716 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.8253441129999999, 2024-09-10T19:19:17.716 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-10T19:19:17.716 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T19:19:17.716 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-10T19:19:17.716 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.15051", 2024-09-10T19:19:17.716 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.183:0/3632028672", 2024-09-10T19:19:17.716 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 59134 2024-09-10T19:19:17.716 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:19:17.716 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-10T19:19:17.716 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:19:17.716 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-10T19:19:17.716 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:55.989069+0000", 2024-09-10T19:19:17.716 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:19:17.716 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:19:17.717 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:19:17.717 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-10T19:19:17.717 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:55.989069+0000", 2024-09-10T19:19:17.717 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:19:17.717 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:19:17.717 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:19:17.717 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-10T19:19:17.717 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:55.989068+0000", 2024-09-10T19:19:17.717 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4294967295.9999995 2024-09-10T19:19:17.717 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:19:17.717 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:19:17.717 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-10T19:19:17.717 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:55.989212+0000", 2024-09-10T19:19:17.718 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000144296 2024-09-10T19:19:17.718 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:19:17.718 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:19:17.718 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-10T19:19:17.718 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:55.989218+0000", 2024-09-10T19:19:17.718 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.6899999999999997e-06 2024-09-10T19:19:17.719 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:19:17.719 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:19:17.719 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-10T19:19:17.719 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:55.989221+0000", 2024-09-10T19:19:17.719 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.6570000000000001e-06 2024-09-10T19:19:17.719 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:19:17.719 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:19:17.719 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-10T19:19:17.719 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:55.989240+0000", 2024-09-10T19:19:17.719 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.906e-05 2024-09-10T19:19:17.719 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:19:17.719 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:19:17.719 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for peered", 2024-09-10T19:19:17.719 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:55.989243+0000", 2024-09-10T19:19:17.719 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.971e-06 2024-09-10T19:19:17.720 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:19:17.720 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:19:17.720 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-10T19:19:17.720 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:57.781893+0000", 2024-09-10T19:19:17.720 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.7926505929999998 2024-09-10T19:19:17.720 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:19:17.720 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:19:17.720 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-10T19:19:17.720 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:57.781985+0000", 2024-09-10T19:19:17.720 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.1576999999999994e-05 2024-09-10T19:19:17.720 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:19:17.720 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:19:17.720 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 5,7", 2024-09-10T19:19:17.720 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:57.782090+0000", 2024-09-10T19:19:17.720 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000104662 2024-09-10T19:19:17.721 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:19:17.721 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:19:17.721 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-10T19:19:17.721 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:57.786095+0000", 2024-09-10T19:19:17.721 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0040054299999999999 2024-09-10T19:19:17.721 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:19:17.721 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:19:17.721 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:19:17.721 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:57.797442+0000", 2024-09-10T19:19:17.721 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.011346827 2024-09-10T19:19:17.721 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:19:17.722 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:19:17.722 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:19:17.722 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:57.813133+0000", 2024-09-10T19:19:17.722 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.015691360000000001 2024-09-10T19:19:17.722 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:19:17.722 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:19:17.722 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-10T19:19:17.722 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:57.813168+0000", 2024-09-10T19:19:17.722 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.4310000000000002e-05 2024-09-10T19:19:17.722 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:19:17.722 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:19:17.722 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-10T19:19:17.722 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:57.814413+0000", 2024-09-10T19:19:17.723 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0012451179999999999 2024-09-10T19:19:17.723 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:19:17.723 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:19:17.723 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:19:17.723 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:19:17.723 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:19:17.723 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.15051.0:59144 8.3 8:c58dd815:::benchmark_data_smithi183_114924_object59143:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e166)", 2024-09-10T19:19:17.723 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-10T19:16:55.989226+0000", 2024-09-10T19:19:17.723 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 141.67977639, 2024-09-10T19:19:17.723 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.8318923599999999, 2024-09-10T19:19:17.723 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-10T19:19:17.724 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T19:19:17.724 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-10T19:19:17.724 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.15051", 2024-09-10T19:19:17.724 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.183:0/3632028672", 2024-09-10T19:19:17.724 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 59144 2024-09-10T19:19:17.724 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:19:17.724 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-10T19:19:17.724 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:19:17.724 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-10T19:19:17.724 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:55.989226+0000", 2024-09-10T19:19:17.724 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:19:17.724 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:19:17.724 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:19:17.724 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-10T19:19:17.725 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:55.989226+0000", 2024-09-10T19:19:17.725 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:19:17.725 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:19:17.725 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:19:17.725 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-10T19:19:17.725 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:55.989225+0000", 2024-09-10T19:19:17.725 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4294967295.9999995 2024-09-10T19:19:17.725 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:19:17.725 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:19:17.725 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-10T19:19:17.725 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:55.989264+0000", 2024-09-10T19:19:17.725 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.8593000000000001e-05 2024-09-10T19:19:17.725 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:19:17.726 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:19:17.726 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-10T19:19:17.726 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:55.989270+0000", 2024-09-10T19:19:17.726 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.1410000000000002e-06 2024-09-10T19:19:17.726 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:19:17.726 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:19:17.726 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-10T19:19:17.726 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:55.989273+0000", 2024-09-10T19:19:17.726 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.0589999999999998e-06 2024-09-10T19:19:17.726 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:19:17.727 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:19:17.727 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-10T19:19:17.727 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:55.989294+0000", 2024-09-10T19:19:17.727 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.1162000000000002e-05 2024-09-10T19:19:17.727 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:19:17.727 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:19:17.727 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for peered", 2024-09-10T19:19:17.727 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:55.989297+0000", 2024-09-10T19:19:17.727 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.773e-06 2024-09-10T19:19:17.727 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:19:17.727 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:19:17.727 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-10T19:19:17.727 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:57.782945+0000", 2024-09-10T19:19:17.728 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.7936485719999999 2024-09-10T19:19:17.728 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:19:17.728 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:19:17.728 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-10T19:19:17.728 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:57.783027+0000", 2024-09-10T19:19:17.728 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.1658999999999999e-05 2024-09-10T19:19:17.728 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:19:17.728 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:19:17.728 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 5,7", 2024-09-10T19:19:17.728 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:57.783126+0000", 2024-09-10T19:19:17.728 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.9234999999999994e-05 2024-09-10T19:19:17.728 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:19:17.729 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:19:17.729 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-10T19:19:17.729 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:57.788338+0000", 2024-09-10T19:19:17.729 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0052119280000000002 2024-09-10T19:19:17.729 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:19:17.729 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:19:17.729 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:19:17.729 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:57.797467+0000", 2024-09-10T19:19:17.729 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0091290439999999994 2024-09-10T19:19:17.729 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:19:17.729 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:19:17.729 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:19:17.729 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:57.819488+0000", 2024-09-10T19:19:17.730 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.022020649 2024-09-10T19:19:17.730 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:19:17.730 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:19:17.730 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-10T19:19:17.730 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:57.819521+0000", 2024-09-10T19:19:17.730 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.3238999999999998e-05 2024-09-10T19:19:17.730 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:19:17.730 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:19:17.730 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-10T19:19:17.730 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:57.821118+0000", 2024-09-10T19:19:17.730 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.001596746 2024-09-10T19:19:17.730 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:19:17.730 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:19:17.730 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:19:17.730 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:19:17.731 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:19:17.731 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.15051.0:59145 8.3 8:c6076f82:::benchmark_data_smithi183_114924_object59144:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e166)", 2024-09-10T19:19:17.731 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-10T19:16:55.989277+0000", 2024-09-10T19:19:17.731 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 141.67972458700001, 2024-09-10T19:19:17.731 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.8331499980000001, 2024-09-10T19:19:17.731 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-10T19:19:17.731 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T19:19:17.731 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-10T19:19:17.731 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.15051", 2024-09-10T19:19:17.731 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.183:0/3632028672", 2024-09-10T19:19:17.732 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 59145 2024-09-10T19:19:17.732 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:19:17.732 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-10T19:19:17.732 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:19:17.732 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-10T19:19:17.732 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:55.989277+0000", 2024-09-10T19:19:17.732 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:19:17.732 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:19:17.732 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:19:17.732 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-10T19:19:17.732 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:55.989277+0000", 2024-09-10T19:19:17.732 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:19:17.732 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:19:17.733 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:19:17.733 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-10T19:19:17.733 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:55.989277+0000", 2024-09-10T19:19:17.733 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4294967295.9999995 2024-09-10T19:19:17.733 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:19:17.733 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:19:17.733 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-10T19:19:17.733 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:55.989314+0000", 2024-09-10T19:19:17.733 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.6698e-05 2024-09-10T19:19:17.733 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:19:17.733 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:19:17.734 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-10T19:19:17.734 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:55.989320+0000", 2024-09-10T19:19:17.734 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.9390000000000002e-06 2024-09-10T19:19:17.734 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:19:17.734 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:19:17.734 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-10T19:19:17.734 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:55.989323+0000", 2024-09-10T19:19:17.734 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.8739999999999999e-06 2024-09-10T19:19:17.734 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:19:17.734 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:19:17.734 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-10T19:19:17.735 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:55.989345+0000", 2024-09-10T19:19:17.735 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.2449000000000001e-05 2024-09-10T19:19:17.735 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:19:17.735 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:19:17.735 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for peered", 2024-09-10T19:19:17.735 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:55.989348+0000", 2024-09-10T19:19:17.735 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.9529999999999999e-06 2024-09-10T19:19:17.735 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:19:17.735 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:19:17.735 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-10T19:19:17.736 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:57.783970+0000", 2024-09-10T19:19:17.736 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.7946222979999999 2024-09-10T19:19:17.736 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:19:17.736 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:19:17.736 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-10T19:19:17.736 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:57.784051+0000", 2024-09-10T19:19:17.736 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.1018000000000003e-05 2024-09-10T19:19:17.736 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:19:17.736 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:19:17.737 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 5,7", 2024-09-10T19:19:17.737 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:57.784150+0000", 2024-09-10T19:19:17.737 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.8586999999999996e-05 2024-09-10T19:19:17.737 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:19:17.737 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:19:17.737 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-10T19:19:17.737 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:57.788350+0000", 2024-09-10T19:19:17.737 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0042002710000000002 2024-09-10T19:19:17.738 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:19:17.738 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:19:17.738 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:19:17.738 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:57.797493+0000", 2024-09-10T19:19:17.738 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0091431740000000004 2024-09-10T19:19:17.738 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:19:17.738 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:19:17.738 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:19:17.738 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:57.821161+0000", 2024-09-10T19:19:17.738 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.023668084999999998 2024-09-10T19:19:17.738 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:19:17.738 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:19:17.738 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-10T19:19:17.738 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:57.821198+0000", 2024-09-10T19:19:17.739 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.6758999999999998e-05 2024-09-10T19:19:17.739 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:19:17.739 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:19:17.739 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-10T19:19:17.739 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:57.822427+0000", 2024-09-10T19:19:17.739 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0012292729999999999 2024-09-10T19:19:17.739 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:19:17.739 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:19:17.740 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:19:17.740 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:19:17.740 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:19:17.740 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.15051.0:59151 8.3 8:cbebd424:::benchmark_data_smithi183_114924_object59150:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e166)", 2024-09-10T19:19:17.740 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-10T19:16:55.989329+0000", 2024-09-10T19:19:17.741 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 141.67967279699999, 2024-09-10T19:19:17.741 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.8345737230000001, 2024-09-10T19:19:17.741 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-10T19:19:17.741 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T19:19:17.741 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-10T19:19:17.741 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.15051", 2024-09-10T19:19:17.742 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.183:0/3632028672", 2024-09-10T19:19:17.742 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 59151 2024-09-10T19:19:17.742 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:19:17.742 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-10T19:19:17.742 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:19:17.742 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-10T19:19:17.742 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:55.989329+0000", 2024-09-10T19:19:17.742 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:19:17.743 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:19:17.743 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:19:17.743 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-10T19:19:17.743 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:55.989329+0000", 2024-09-10T19:19:17.743 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:19:17.743 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:19:17.743 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:19:17.743 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-10T19:19:17.743 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:55.989329+0000", 2024-09-10T19:19:17.743 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4294967295.9999995 2024-09-10T19:19:17.743 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:19:17.743 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:19:17.743 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-10T19:19:17.744 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:55.989364+0000", 2024-09-10T19:19:17.744 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.5280000000000001e-05 2024-09-10T19:19:17.744 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:19:17.744 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:19:17.744 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-10T19:19:17.744 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:55.989370+0000", 2024-09-10T19:19:17.744 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.852e-06 2024-09-10T19:19:17.744 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:19:17.744 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:19:17.744 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-10T19:19:17.744 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:55.989373+0000", 2024-09-10T19:19:17.744 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.9349999999999999e-06 2024-09-10T19:19:17.744 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:19:17.745 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:19:17.745 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-10T19:19:17.745 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:55.989390+0000", 2024-09-10T19:19:17.745 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.7632e-05 2024-09-10T19:19:17.745 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:19:17.745 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:19:17.745 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for peered", 2024-09-10T19:19:17.745 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:55.989393+0000", 2024-09-10T19:19:17.745 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.7719999999999999e-06 2024-09-10T19:19:17.745 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:19:17.746 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:19:17.746 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-10T19:19:17.746 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:57.784999+0000", 2024-09-10T19:19:17.746 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.795605976 2024-09-10T19:19:17.746 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:19:17.746 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:19:17.746 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-10T19:19:17.746 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:57.785091+0000", 2024-09-10T19:19:17.746 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.1812999999999997e-05 2024-09-10T19:19:17.746 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:19:17.747 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:19:17.747 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 5,7", 2024-09-10T19:19:17.747 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:57.785185+0000", 2024-09-10T19:19:17.747 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.4143999999999999e-05 2024-09-10T19:19:17.747 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:19:17.747 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:19:17.747 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-10T19:19:17.747 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:57.792624+0000", 2024-09-10T19:19:17.747 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0074391149999999996 2024-09-10T19:19:17.747 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:19:17.747 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:19:17.747 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:19:17.747 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:57.799494+0000", 2024-09-10T19:19:17.748 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0068695969999999999 2024-09-10T19:19:17.748 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:19:17.748 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:19:17.748 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:19:17.748 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:57.822835+0000", 2024-09-10T19:19:17.748 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.023340831999999999 2024-09-10T19:19:17.748 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:19:17.748 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:19:17.748 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-10T19:19:17.748 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:57.822888+0000", 2024-09-10T19:19:17.749 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.3263999999999997e-05 2024-09-10T19:19:17.749 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:19:17.749 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:19:17.749 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-10T19:19:17.749 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:57.823903+0000", 2024-09-10T19:19:17.749 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0010149309999999999 2024-09-10T19:19:17.749 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:19:17.749 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:19:17.749 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:19:17.749 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:19:17.749 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:19:17.749 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.15051.0:59154 8.3 8:c7c823db:::benchmark_data_smithi183_114924_object59153:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e166)", 2024-09-10T19:19:17.749 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-10T19:16:55.989377+0000", 2024-09-10T19:19:17.750 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 141.679624625, 2024-09-10T19:19:17.750 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.835555351, 2024-09-10T19:19:17.750 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-10T19:19:17.750 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T19:19:17.750 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-10T19:19:17.750 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.15051", 2024-09-10T19:19:17.750 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.183:0/3632028672", 2024-09-10T19:19:17.750 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 59154 2024-09-10T19:19:17.750 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:19:17.750 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-10T19:19:17.750 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:19:17.751 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-10T19:19:17.751 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:55.989377+0000", 2024-09-10T19:19:17.751 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:19:17.751 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:19:17.751 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:19:17.751 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-10T19:19:17.751 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:55.989377+0000", 2024-09-10T19:19:17.751 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:19:17.751 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:19:17.751 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:19:17.751 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-10T19:19:17.751 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:55.989377+0000", 2024-09-10T19:19:17.751 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4294967295.9999995 2024-09-10T19:19:17.752 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:19:17.752 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:19:17.752 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-10T19:19:17.752 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:55.989413+0000", 2024-09-10T19:19:17.752 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.5738000000000002e-05 2024-09-10T19:19:17.752 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:19:17.752 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:19:17.752 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-10T19:19:17.752 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:55.989419+0000", 2024-09-10T19:19:17.752 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.9000000000000003e-06 2024-09-10T19:19:17.752 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:19:17.752 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:19:17.753 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-10T19:19:17.753 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:55.989421+0000", 2024-09-10T19:19:17.753 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.8899999999999999e-06 2024-09-10T19:19:17.753 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:19:17.753 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:19:17.753 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-10T19:19:17.753 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:55.989443+0000", 2024-09-10T19:19:17.753 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.1097000000000001e-05 2024-09-10T19:19:17.753 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:19:17.753 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:19:17.753 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for peered", 2024-09-10T19:19:17.753 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:55.989446+0000", 2024-09-10T19:19:17.754 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.943e-06 2024-09-10T19:19:17.754 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:19:17.754 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:19:17.754 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-10T19:19:17.754 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:57.786124+0000", 2024-09-10T19:19:17.754 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.796678357 2024-09-10T19:19:17.754 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:19:17.754 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:19:17.754 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-10T19:19:17.754 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:57.786202+0000", 2024-09-10T19:19:17.754 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.8150000000000002e-05 2024-09-10T19:19:17.755 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:19:17.755 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:19:17.755 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 5,7", 2024-09-10T19:19:17.755 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:57.786294+0000", 2024-09-10T19:19:17.755 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-10T19:19:17.756 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:19:17 smithi191 ceph-mon[32884]: 5.14 scrub starts 2024-09-10T19:19:17.756 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:19:17 smithi191 ceph-mon[32884]: 5.14 scrub ok 2024-09-10T19:19:17.756 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:19:17 smithi191 ceph-mon[32884]: 4.16 scrub starts 2024-09-10T19:19:17.756 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:19:17 smithi191 ceph-mon[32884]: 4.16 scrub ok 2024-09-10T19:19:17.757 INFO:journalctl@ceph.mgr.x.smithi191.stdout:Sep 10 19:19:17 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-x[34967]: ::ffff:172.21.15.191 - - [10/Sep/2024:19:19:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T19:19:17.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:17.757 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:19:17.758 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:19:17.759 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.1912999999999999e-05 2024-09-10T19:19:17.759 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:19:17.759 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:19:17.759 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-10T19:19:17.759 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:57.792631+0000", 2024-09-10T19:19:17.759 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.006337484 2024-09-10T19:19:17.759 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:19:17.759 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:19:17.759 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:19:17.759 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:57.799517+0000", 2024-09-10T19:19:17.760 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0068852369999999998 2024-09-10T19:19:17.760 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:19:17.760 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:19:17.760 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:19:17.760 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:57.823952+0000", 2024-09-10T19:19:17.760 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.024435651999999999 2024-09-10T19:19:17.760 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:19:17.760 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:19:17.760 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-10T19:19:17.760 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:57.823986+0000", 2024-09-10T19:19:17.760 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.3439000000000003e-05 2024-09-10T19:19:17.760 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:19:17.761 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:19:17.761 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-10T19:19:17.761 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:57.824933+0000", 2024-09-10T19:19:17.761 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00094693100000000003 2024-09-10T19:19:17.761 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:19:17.761 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:19:17.761 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:19:17.761 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:19:17.761 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:19:17.761 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.15051.0:59158 8.3 8:cc8333b1:::benchmark_data_smithi183_114924_object59157:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e166)", 2024-09-10T19:19:17.761 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-10T19:16:55.989426+0000", 2024-09-10T19:19:17.761 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 141.67957595799999, 2024-09-10T19:19:17.761 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.836481646, 2024-09-10T19:19:17.762 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-10T19:19:17.762 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T19:19:17.762 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-10T19:19:17.762 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.15051", 2024-09-10T19:19:17.762 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.183:0/3632028672", 2024-09-10T19:19:17.762 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 59158 2024-09-10T19:19:17.762 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:19:17.762 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-10T19:19:17.762 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:19:17.762 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-10T19:19:17.762 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:55.989426+0000", 2024-09-10T19:19:17.762 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:19:17.762 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:19:17.762 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:19:17.763 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-10T19:19:17.763 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:55.989426+0000", 2024-09-10T19:19:17.763 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:19:17.763 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:19:17.763 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:19:17.763 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-10T19:19:17.763 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:55.989426+0000", 2024-09-10T19:19:17.763 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4294967295.9999995 2024-09-10T19:19:17.763 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:19:17.763 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:19:17.763 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-10T19:19:17.763 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:55.989462+0000", 2024-09-10T19:19:17.763 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.6878999999999997e-05 2024-09-10T19:19:17.764 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:19:17.764 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:19:17.764 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-10T19:19:17.764 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:55.989468+0000", 2024-09-10T19:19:17.764 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.9710000000000001e-06 2024-09-10T19:19:17.764 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:19:17.764 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:19:17.764 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-10T19:19:17.764 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:55.989471+0000", 2024-09-10T19:19:17.764 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.8909999999999999e-06 2024-09-10T19:19:17.764 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:19:17.764 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:19:17.765 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-10T19:19:17.765 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:55.989490+0000", 2024-09-10T19:19:17.765 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.9157e-05 2024-09-10T19:19:17.765 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:19:17.765 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:19:17.765 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for peered", 2024-09-10T19:19:17.765 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:55.989493+0000", 2024-09-10T19:19:17.765 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.6520000000000002e-06 2024-09-10T19:19:17.765 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:19:17.765 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:19:17.765 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-10T19:19:17.765 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:57.787166+0000", 2024-09-10T19:19:17.766 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.7976729169999999 2024-09-10T19:19:17.766 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:19:17.766 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:19:17.766 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-10T19:19:17.766 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:57.787251+0000", 2024-09-10T19:19:17.766 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.4837000000000001e-05 2024-09-10T19:19:17.766 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:19:17.766 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:19:17.766 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 5,7", 2024-09-10T19:19:17.766 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:57.787347+0000", 2024-09-10T19:19:17.766 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.6084999999999999e-05 2024-09-10T19:19:17.766 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:19:17.767 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:19:17.767 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-10T19:19:17.767 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:57.793693+0000", 2024-09-10T19:19:17.767 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0063457749999999997 2024-09-10T19:19:17.767 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:19:17.767 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:19:17.767 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:19:17.767 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:57.805750+0000", 2024-09-10T19:19:17.767 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.012057615000000001 2024-09-10T19:19:17.767 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:19:17.767 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:19:17.767 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:19:17.767 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:57.824981+0000", 2024-09-10T19:19:17.767 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.019230860999999998 2024-09-10T19:19:17.768 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:19:17.768 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:19:17.768 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-10T19:19:17.768 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:57.825014+0000", 2024-09-10T19:19:17.768 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.2435999999999999e-05 2024-09-10T19:19:17.768 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:19:17.768 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:19:17.768 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-10T19:19:17.768 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:57.825908+0000", 2024-09-10T19:19:17.768 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00089398400000000001 2024-09-10T19:19:17.768 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:19:17.768 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:19:17.768 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:19:17.768 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:19:17.769 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:19:17.769 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.15051.0:59203 8.3 8:ccbf45cb:::benchmark_data_smithi183_114924_object59202:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e166)", 2024-09-10T19:19:17.769 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-10T19:16:55.989476+0000", 2024-09-10T19:19:17.769 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 141.679525781, 2024-09-10T19:19:17.769 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.8425899960000001, 2024-09-10T19:19:17.769 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-10T19:19:17.769 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T19:19:17.769 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-10T19:19:17.769 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.15051", 2024-09-10T19:19:17.769 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.183:0/3632028672", 2024-09-10T19:19:17.769 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 59203 2024-09-10T19:19:17.769 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:19:17.769 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-10T19:19:17.770 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:19:17.770 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-10T19:19:17.770 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:55.989476+0000", 2024-09-10T19:19:17.770 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:19:17.770 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:19:17.770 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:19:17.770 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-10T19:19:17.770 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:55.989476+0000", 2024-09-10T19:19:17.770 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:19:17.770 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:19:17.770 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:19:17.770 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-10T19:19:17.770 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:55.989476+0000", 2024-09-10T19:19:17.771 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4294967295.9999995 2024-09-10T19:19:17.771 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:19:17.771 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:19:17.771 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-10T19:19:17.771 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:55.989511+0000", 2024-09-10T19:19:17.771 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.5590000000000003e-05 2024-09-10T19:19:17.771 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:19:17.771 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:19:17.771 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-10T19:19:17.771 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:55.989517+0000", 2024-09-10T19:19:17.771 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.913e-06 2024-09-10T19:19:17.771 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:19:17.771 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:19:17.771 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-10T19:19:17.772 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:55.989520+0000", 2024-09-10T19:19:17.772 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.0419999999999998e-06 2024-09-10T19:19:17.772 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:19:17.772 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:19:17.772 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-10T19:19:17.772 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:55.989539+0000", 2024-09-10T19:19:17.772 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.8624999999999999e-05 2024-09-10T19:19:17.772 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:19:17.772 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:19:17.772 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for peered", 2024-09-10T19:19:17.772 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:55.989542+0000", 2024-09-10T19:19:17.772 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.7250000000000002e-06 2024-09-10T19:19:17.772 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:19:17.773 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:19:17.773 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-10T19:19:17.773 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:57.788382+0000", 2024-09-10T19:19:17.773 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.7988407230000001 2024-09-10T19:19:17.773 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:19:17.773 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:19:17.773 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-10T19:19:17.773 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:57.788461+0000", 2024-09-10T19:19:17.773 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.8881000000000006e-05 2024-09-10T19:19:17.773 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:19:17.773 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:19:17.773 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 5,7", 2024-09-10T19:19:17.774 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:57.788561+0000", 2024-09-10T19:19:17.774 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.9351999999999993e-05 2024-09-10T19:19:17.774 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:19:17.774 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:19:17.774 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-10T19:19:17.774 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:57.793697+0000", 2024-09-10T19:19:17.774 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.005136369 2024-09-10T19:19:17.774 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:19:17.774 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:19:17.774 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:19:17.774 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:57.805782+0000", 2024-09-10T19:19:17.774 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.012085043 2024-09-10T19:19:17.774 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:19:17.774 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:19:17.775 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:19:17.775 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:57.830391+0000", 2024-09-10T19:19:17.775 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.024609273000000001 2024-09-10T19:19:17.775 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:19:17.775 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:19:17.775 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-10T19:19:17.775 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:57.830424+0000", 2024-09-10T19:19:17.775 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.2657999999999998e-05 2024-09-10T19:19:17.775 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:19:17.775 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:19:17.775 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-10T19:19:17.776 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:57.832066+0000", 2024-09-10T19:19:17.776 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.001642225 2024-09-10T19:19:17.776 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:19:17.776 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:19:17.776 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:19:17.776 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:19:17.776 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:19:17.776 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.15051.0:59228 8.3 8:ce38d1dc:::benchmark_data_smithi183_114924_object59227:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e166)", 2024-09-10T19:19:17.776 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-10T19:16:55.989525+0000", 2024-09-10T19:19:17.776 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 141.67947716399999, 2024-09-10T19:19:17.776 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.8572049580000001, 2024-09-10T19:19:17.776 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-10T19:19:17.776 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T19:19:17.777 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-10T19:19:17.777 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.15051", 2024-09-10T19:19:17.777 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.183:0/3632028672", 2024-09-10T19:19:17.777 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 59228 2024-09-10T19:19:17.777 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:19:17.777 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-10T19:19:17.777 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:19:17.777 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-10T19:19:17.777 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:55.989525+0000", 2024-09-10T19:19:17.777 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:19:17.777 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:19:17.778 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:19:17.778 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-10T19:19:17.778 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:55.989525+0000", 2024-09-10T19:19:17.778 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:19:17.778 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:19:17.778 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:19:17.778 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-10T19:19:17.778 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:55.989524+0000", 2024-09-10T19:19:17.778 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4294967295.9999995 2024-09-10T19:19:17.778 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:19:17.778 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:19:17.778 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-10T19:19:17.778 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:55.989559+0000", 2024-09-10T19:19:17.778 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.4261000000000003e-05 2024-09-10T19:19:17.778 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:19:17.779 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:19:17.779 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-10T19:19:17.779 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:55.989565+0000", 2024-09-10T19:19:17.779 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.8470000000000001e-06 2024-09-10T19:19:17.779 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:19:17.779 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:19:17.779 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-10T19:19:17.779 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:55.989567+0000", 2024-09-10T19:19:17.779 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.3630000000000001e-06 2024-09-10T19:19:17.779 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:19:17.779 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:19:17.779 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-10T19:19:17.780 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:55.989586+0000", 2024-09-10T19:19:17.780 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.9576e-05 2024-09-10T19:19:17.780 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:19:17.780 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:19:17.780 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for peered", 2024-09-10T19:19:17.780 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:55.989589+0000", 2024-09-10T19:19:17.780 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.8679999999999999e-06 2024-09-10T19:19:17.780 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:19:17.780 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:19:17.780 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-10T19:19:17.781 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:57.789407+0000", 2024-09-10T19:19:17.781 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.7998178980000001 2024-09-10T19:19:17.781 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:19:17.781 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:19:17.781 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-10T19:19:17.781 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:57.789485+0000", 2024-09-10T19:19:17.781 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.8109999999999996e-05 2024-09-10T19:19:17.781 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:19:17.781 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:19:17.781 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 5,7", 2024-09-10T19:19:17.781 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:57.789580+0000", 2024-09-10T19:19:17.781 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.4176000000000001e-05 2024-09-10T19:19:17.781 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:19:17.781 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:19:17.782 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-10T19:19:17.782 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:57.793702+0000", 2024-09-10T19:19:17.782 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0041221390000000004 2024-09-10T19:19:17.782 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:19:17.782 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:19:17.782 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:19:17.782 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:57.805830+0000", 2024-09-10T19:19:17.782 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.012128144 2024-09-10T19:19:17.782 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:19:17.782 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:19:17.782 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:19:17.782 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:57.845000+0000", 2024-09-10T19:19:17.782 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.039170071000000001 2024-09-10T19:19:17.783 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:19:17.783 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:19:17.783 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-10T19:19:17.783 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:57.845039+0000", 2024-09-10T19:19:17.783 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.8940000000000003e-05 2024-09-10T19:19:17.783 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:19:17.783 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:19:17.783 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-10T19:19:17.783 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:57.846730+0000", 2024-09-10T19:19:17.783 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.001690963 2024-09-10T19:19:17.783 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:19:17.783 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:19:17.783 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:19:17.783 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:19:17.783 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:19:17.784 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.15051.0:59235 8.3 8:c3e38beb:::benchmark_data_smithi183_114924_object59234:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e166)", 2024-09-10T19:19:17.784 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-10T19:16:55.989571+0000", 2024-09-10T19:19:17.784 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 141.679430642, 2024-09-10T19:19:17.784 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.8656851109999999, 2024-09-10T19:19:17.784 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-10T19:19:17.784 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T19:19:17.784 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-10T19:19:17.784 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.15051", 2024-09-10T19:19:17.784 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.183:0/3632028672", 2024-09-10T19:19:17.784 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 59235 2024-09-10T19:19:17.784 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:19:17.784 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-10T19:19:17.785 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:19:17.785 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-10T19:19:17.785 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:55.989571+0000", 2024-09-10T19:19:17.785 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:19:17.785 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:19:17.785 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:19:17.785 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-10T19:19:17.785 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:55.989571+0000", 2024-09-10T19:19:17.785 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:19:17.785 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:19:17.785 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:19:17.785 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-10T19:19:17.785 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:55.989571+0000", 2024-09-10T19:19:17.786 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4294967295.9999995 2024-09-10T19:19:17.786 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:19:17.786 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:19:17.786 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-10T19:19:17.786 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:55.989607+0000", 2024-09-10T19:19:17.786 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.5815000000000002e-05 2024-09-10T19:19:17.786 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:19:17.786 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:19:17.786 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-10T19:19:17.786 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:55.989613+0000", 2024-09-10T19:19:17.786 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.7860000000000002e-06 2024-09-10T19:19:17.786 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:19:17.786 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:19:17.787 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-10T19:19:17.787 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:55.989616+0000", 2024-09-10T19:19:17.787 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.9550000000000001e-06 2024-09-10T19:19:17.787 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:19:17.787 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:19:17.787 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-10T19:19:17.787 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:55.989637+0000", 2024-09-10T19:19:17.787 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.1443000000000001e-05 2024-09-10T19:19:17.787 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:19:17.787 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:19:17.787 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for peered", 2024-09-10T19:19:17.787 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:55.989640+0000", 2024-09-10T19:19:17.787 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.807e-06 2024-09-10T19:19:17.787 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:19:17.787 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:19:17.788 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-10T19:19:17.788 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:57.790482+0000", 2024-09-10T19:19:17.788 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.800842193 2024-09-10T19:19:17.788 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:19:17.788 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:19:17.788 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-10T19:19:17.788 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:57.790563+0000", 2024-09-10T19:19:17.788 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.0902999999999994e-05 2024-09-10T19:19:17.788 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:19:17.788 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:19:17.788 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 5,7", 2024-09-10T19:19:17.788 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:57.790664+0000", 2024-09-10T19:19:17.789 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000101628 2024-09-10T19:19:17.789 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:19:17.789 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:19:17.789 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-10T19:19:17.789 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:57.793706+0000", 2024-09-10T19:19:17.789 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0030410429999999998 2024-09-10T19:19:17.789 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:19:17.789 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:19:17.789 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:19:17.790 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:57.805856+0000", 2024-09-10T19:19:17.790 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.012150213999999999 2024-09-10T19:19:17.790 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:19:17.790 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:19:17.790 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:19:17.790 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:57.853631+0000", 2024-09-10T19:19:17.790 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.047774900000000002 2024-09-10T19:19:17.790 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:19:17.790 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:19:17.790 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-10T19:19:17.790 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:57.853662+0000", 2024-09-10T19:19:17.790 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.1511999999999999e-05 2024-09-10T19:19:17.790 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:19:17.791 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:19:17.791 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-10T19:19:17.791 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:57.855256+0000", 2024-09-10T19:19:17.791 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0015942809999999999 2024-09-10T19:19:17.791 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:19:17.791 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:19:17.791 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:19:17.791 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:19:17.791 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:19:17.791 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.15051.0:59242 8.3 8:c1483ce1:::benchmark_data_smithi183_114924_object59241:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e166)", 2024-09-10T19:19:17.791 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-10T19:16:55.989620+0000", 2024-09-10T19:19:17.791 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 141.679382011, 2024-09-10T19:19:17.791 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.8671676669999999, 2024-09-10T19:19:17.792 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-10T19:19:17.792 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T19:19:17.792 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-10T19:19:17.792 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.15051", 2024-09-10T19:19:17.792 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.183:0/3632028672", 2024-09-10T19:19:17.792 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 59242 2024-09-10T19:19:17.792 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:19:17.792 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-10T19:19:17.792 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:19:17.792 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-10T19:19:17.792 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:55.989620+0000", 2024-09-10T19:19:17.792 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:19:17.792 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:19:17.793 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:19:17.793 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-10T19:19:17.793 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:55.989620+0000", 2024-09-10T19:19:17.793 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:19:17.793 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:19:17.793 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:19:17.793 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-10T19:19:17.793 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:55.989620+0000", 2024-09-10T19:19:17.793 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4294967295.9999995 2024-09-10T19:19:17.794 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:19:17.794 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:19:17.794 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-10T19:19:17.794 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:55.989656+0000", 2024-09-10T19:19:17.794 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.6198999999999997e-05 2024-09-10T19:19:17.794 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:19:17.794 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:19:17.794 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-10T19:19:17.794 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:55.989662+0000", 2024-09-10T19:19:17.794 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.976e-06 2024-09-10T19:19:17.794 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:19:17.794 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:19:17.794 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-10T19:19:17.795 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:55.989665+0000", 2024-09-10T19:19:17.795 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.9129999999999999e-06 2024-09-10T19:19:17.795 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:19:17.795 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:19:17.795 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-10T19:19:17.795 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:55.989684+0000", 2024-09-10T19:19:17.795 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.9235000000000001e-05 2024-09-10T19:19:17.795 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:19:17.795 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:19:17.795 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for peered", 2024-09-10T19:19:17.795 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:55.989687+0000", 2024-09-10T19:19:17.795 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.8109999999999999e-06 2024-09-10T19:19:17.795 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:19:17.795 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:19:17.796 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-10T19:19:17.796 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:57.791617+0000", 2024-09-10T19:19:17.796 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.801930668 2024-09-10T19:19:17.796 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:19:17.796 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:19:17.796 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-10T19:19:17.796 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:57.791701+0000", 2024-09-10T19:19:17.796 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.3627000000000004e-05 2024-09-10T19:19:17.796 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:19:17.796 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:19:17.796 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 5,7", 2024-09-10T19:19:17.796 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:57.791802+0000", 2024-09-10T19:19:17.797 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000100942 2024-09-10T19:19:17.797 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:19:17.797 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:19:17.797 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-10T19:19:17.797 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:57.795215+0000", 2024-09-10T19:19:17.797 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0034128309999999999 2024-09-10T19:19:17.797 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:19:17.797 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:19:17.797 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:19:17.797 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:57.805876+0000", 2024-09-10T19:19:17.797 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.010660849 2024-09-10T19:19:17.798 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:19:17.798 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:19:17.798 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:19:17.798 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:57.855315+0000", 2024-09-10T19:19:17.798 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.049439759 2024-09-10T19:19:17.798 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:19:17.798 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:19:17.798 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-10T19:19:17.798 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:57.855371+0000", 2024-09-10T19:19:17.798 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.5946999999999998e-05 2024-09-10T19:19:17.798 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:19:17.798 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:19:17.798 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-10T19:19:17.798 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:57.856788+0000", 2024-09-10T19:19:17.799 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.001416311 2024-09-10T19:19:17.799 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:19:17.799 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:19:17.799 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:19:17.799 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:19:17.799 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:19:17.799 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.15051.0:59243 8.3 8:c806a870:::benchmark_data_smithi183_114924_object59242:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e166)", 2024-09-10T19:19:17.799 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-10T19:16:55.989669+0000", 2024-09-10T19:19:17.799 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 141.67933287899999, 2024-09-10T19:19:17.800 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.8685301810000001, 2024-09-10T19:19:17.800 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-10T19:19:17.800 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T19:19:17.800 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-10T19:19:17.800 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.15051", 2024-09-10T19:19:17.800 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.183:0/3632028672", 2024-09-10T19:19:17.800 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 59243 2024-09-10T19:19:17.800 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:19:17.800 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-10T19:19:17.800 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:19:17.800 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-10T19:19:17.800 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:55.989669+0000", 2024-09-10T19:19:17.800 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:19:17.801 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:19:17.801 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:19:17.801 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-10T19:19:17.801 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:55.989669+0000", 2024-09-10T19:19:17.801 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:19:17.801 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:19:17.801 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:19:17.801 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-10T19:19:17.801 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:55.989669+0000", 2024-09-10T19:19:17.801 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4294967295.9999995 2024-09-10T19:19:17.801 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:19:17.801 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:19:17.802 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-10T19:19:17.802 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:55.989705+0000", 2024-09-10T19:19:17.802 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.6089e-05 2024-09-10T19:19:17.802 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:19:17.802 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:19:17.802 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-10T19:19:17.802 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:55.989711+0000", 2024-09-10T19:19:17.802 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.1940000000000003e-06 2024-09-10T19:19:17.802 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:19:17.802 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:19:17.802 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-10T19:19:17.802 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:55.989714+0000", 2024-09-10T19:19:17.802 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.7690000000000001e-06 2024-09-10T19:19:17.803 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:19:17.803 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:19:17.803 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-10T19:19:17.803 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:55.989732+0000", 2024-09-10T19:19:17.803 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.8210999999999999e-05 2024-09-10T19:19:17.803 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:19:17.803 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:19:17.803 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for peered", 2024-09-10T19:19:17.803 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:55.989735+0000", 2024-09-10T19:19:17.804 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.8700000000000001e-06 2024-09-10T19:19:17.804 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:19:17.804 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:19:17.804 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-10T19:19:17.804 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:57.792658+0000", 2024-09-10T19:19:17.804 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.8029229409999998 2024-09-10T19:19:17.804 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:19:17.804 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:19:17.804 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-10T19:19:17.804 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:57.792735+0000", 2024-09-10T19:19:17.804 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.6919e-05 2024-09-10T19:19:17.804 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:19:17.804 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:19:17.805 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 5,7", 2024-09-10T19:19:17.805 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:57.792835+0000", 2024-09-10T19:19:17.805 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000100052 2024-09-10T19:19:17.805 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:19:17.805 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:19:17.805 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-10T19:19:17.805 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:57.796414+0000", 2024-09-10T19:19:17.805 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0035788740000000001 2024-09-10T19:19:17.805 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:19:17.805 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:19:17.805 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:19:17.805 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:57.808648+0000", 2024-09-10T19:19:17.805 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.012234134000000001 2024-09-10T19:19:17.805 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:19:17.805 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:19:17.806 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:19:17.806 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:57.856842+0000", 2024-09-10T19:19:17.806 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.048194242999999998 2024-09-10T19:19:17.806 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:19:17.806 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:19:17.806 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-10T19:19:17.806 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:57.856898+0000", 2024-09-10T19:19:17.806 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.6252000000000003e-05 2024-09-10T19:19:17.806 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:19:17.806 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:19:17.806 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-10T19:19:17.806 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:57.858199+0000", 2024-09-10T19:19:17.806 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0013010109999999999 2024-09-10T19:19:17.806 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:19:17.807 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:19:17.807 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:19:17.807 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:19:17.807 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:19:17.807 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.15051.0:93184 8.13 8:cfff3a3d:::benchmark_data_smithi183_114924_object93183:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e180)", 2024-09-10T19:19:17.807 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-10T19:17:28.948455+0000", 2024-09-10T19:19:17.807 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 108.720547377, 2024-09-10T19:19:17.807 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.1285055399999999, 2024-09-10T19:19:17.807 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-10T19:19:17.807 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T19:19:17.807 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-10T19:19:17.807 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.15051", 2024-09-10T19:19:17.807 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.183:0/3632028672", 2024-09-10T19:19:17.807 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 93184 2024-09-10T19:19:17.808 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:19:17.808 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-10T19:19:17.808 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:19:17.808 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-10T19:19:17.808 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:17:28.948455+0000", 2024-09-10T19:19:17.808 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:19:17.808 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:19:17.808 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:19:17.808 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-10T19:19:17.808 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:17:28.948455+0000", 2024-09-10T19:19:17.808 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:19:17.808 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:19:17.808 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:19:17.808 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-10T19:19:17.808 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:17:28.948453+0000", 2024-09-10T19:19:17.809 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4294967295.9999986 2024-09-10T19:19:17.809 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:19:17.809 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:19:17.809 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-10T19:19:17.809 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:17:28.948488+0000", 2024-09-10T19:19:17.809 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.4739000000000001e-05 2024-09-10T19:19:17.809 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:19:17.809 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:19:17.809 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-10T19:19:17.809 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:17:28.948500+0000", 2024-09-10T19:19:17.809 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.1946999999999999e-05 2024-09-10T19:19:17.810 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:19:17.810 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:19:17.810 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-10T19:19:17.810 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:17:28.948506+0000", 2024-09-10T19:19:17.810 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.4400000000000002e-06 2024-09-10T19:19:17.810 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:19:17.810 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:19:17.810 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-10T19:19:17.810 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:17:28.950088+0000", 2024-09-10T19:19:17.810 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.001581671 2024-09-10T19:19:17.810 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:19:17.810 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:19:17.811 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for peered", 2024-09-10T19:19:17.811 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:17:28.950096+0000", 2024-09-10T19:19:17.811 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.8460000000000008e-06 2024-09-10T19:19:17.811 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:19:17.811 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:19:17.811 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-10T19:19:17.811 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:17:30.046045+0000", 2024-09-10T19:19:17.811 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.095949455 2024-09-10T19:19:17.811 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:19:17.811 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:19:17.811 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-10T19:19:17.811 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:17:30.046165+0000", 2024-09-10T19:19:17.811 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00011940800000000001 2024-09-10T19:19:17.811 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:19:17.812 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:19:17.812 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 4,5,7", 2024-09-10T19:19:17.812 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:17:30.046404+0000", 2024-09-10T19:19:17.812 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000239075 2024-09-10T19:19:17.812 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:19:17.812 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:19:17.812 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-10T19:19:17.812 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:17:30.053276+0000", 2024-09-10T19:19:17.812 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0068720839999999997 2024-09-10T19:19:17.812 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:19:17.812 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:19:17.812 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:19:17.812 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:17:30.056804+0000", 2024-09-10T19:19:17.812 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0035281380000000001 2024-09-10T19:19:17.813 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:19:17.813 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:19:17.813 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:19:17.813 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:17:30.057585+0000", 2024-09-10T19:19:17.813 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000781385 2024-09-10T19:19:17.813 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:19:17.813 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:19:17.813 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:19:17.814 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:17:30.075618+0000", 2024-09-10T19:19:17.814 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.018033098000000001 2024-09-10T19:19:17.814 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:19:17.814 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:19:17.814 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-10T19:19:17.814 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:17:30.075655+0000", 2024-09-10T19:19:17.814 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.6934999999999998e-05 2024-09-10T19:19:17.814 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:19:17.814 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:19:17.814 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-10T19:19:17.815 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:17:30.076960+0000", 2024-09-10T19:19:17.815 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.001304838 2024-09-10T19:19:17.815 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:19:17.815 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:19:17.815 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:19:17.815 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:19:17.815 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:19:17.815 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.15051.0:93229 8.13 8:ca7bbcb1:::benchmark_data_smithi183_114924_object93228:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e180)", 2024-09-10T19:19:17.815 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-10T19:17:28.948515+0000", 2024-09-10T19:19:17.815 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 108.72048684000001, 2024-09-10T19:19:17.815 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.1299140519999999, 2024-09-10T19:19:17.815 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-10T19:19:17.815 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T19:19:17.815 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-10T19:19:17.815 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.15051", 2024-09-10T19:19:17.816 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.183:0/3632028672", 2024-09-10T19:19:17.816 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 93229 2024-09-10T19:19:17.816 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:19:17.816 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-10T19:19:17.816 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:19:17.816 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-10T19:19:17.816 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:17:28.948515+0000", 2024-09-10T19:19:17.816 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:19:17.816 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:19:17.816 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:19:17.816 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-10T19:19:17.816 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:17:28.948515+0000", 2024-09-10T19:19:17.817 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:19:17.817 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:19:17.817 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:19:17.817 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-10T19:19:17.817 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:17:28.948515+0000", 2024-09-10T19:19:17.817 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4294967295.9999995 2024-09-10T19:19:17.817 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:19:17.817 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:19:17.817 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-10T19:19:17.817 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:17:28.948535+0000", 2024-09-10T19:19:17.817 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.0781999999999999e-05 2024-09-10T19:19:17.817 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:19:17.817 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:19:17.817 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-10T19:19:17.818 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:17:28.948543+0000", 2024-09-10T19:19:17.818 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.3610000000000003e-06 2024-09-10T19:19:17.818 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:19:17.818 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:19:17.818 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-10T19:19:17.818 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:17:28.948547+0000", 2024-09-10T19:19:17.818 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.2799999999999997e-06 2024-09-10T19:19:17.818 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:19:17.818 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:19:17.818 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-10T19:19:17.819 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:17:28.950137+0000", 2024-09-10T19:19:17.819 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0015896599999999999 2024-09-10T19:19:17.819 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:19:17.819 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:19:17.819 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for peered", 2024-09-10T19:19:17.819 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:17:28.950144+0000", 2024-09-10T19:19:17.819 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.0659999999999998e-06 2024-09-10T19:19:17.819 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:19:17.819 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:19:17.819 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-10T19:19:17.819 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:17:30.047351+0000", 2024-09-10T19:19:17.820 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.0972073360000001 2024-09-10T19:19:17.820 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:19:17.820 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:19:17.820 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-10T19:19:17.820 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:17:30.047471+0000", 2024-09-10T19:19:17.820 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000119522 2024-09-10T19:19:17.820 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:19:17.820 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:19:17.820 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 4,5,7", 2024-09-10T19:19:17.820 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:17:30.047689+0000", 2024-09-10T19:19:17.820 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00021808099999999999 2024-09-10T19:19:17.821 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:19:17.821 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:19:17.821 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-10T19:19:17.821 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:17:30.053285+0000", 2024-09-10T19:19:17.821 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0055959110000000003 2024-09-10T19:19:17.821 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:19:17.821 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:19:17.821 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:19:17.821 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:17:30.056858+0000", 2024-09-10T19:19:17.821 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.003573578 2024-09-10T19:19:17.821 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:19:17.822 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:19:17.822 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:19:17.822 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:17:30.057603+0000", 2024-09-10T19:19:17.822 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00074534299999999998 2024-09-10T19:19:17.822 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:19:17.822 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:19:17.822 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:19:17.822 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:17:30.077300+0000", 2024-09-10T19:19:17.822 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.019696202999999999 2024-09-10T19:19:17.822 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:19:17.822 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:19:17.823 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-10T19:19:17.823 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:17:30.077355+0000", 2024-09-10T19:19:17.823 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.5689999999999997e-05 2024-09-10T19:19:17.823 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:19:17.823 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:19:17.823 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-10T19:19:17.823 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:17:30.078429+0000", 2024-09-10T19:19:17.823 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.001073811 2024-09-10T19:19:17.824 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:19:17.824 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:19:17.824 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:19:17.824 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:19:17.824 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:19:17.824 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.15051.0:93231 8.13 8:ca03a44e:::benchmark_data_smithi183_114924_object93230:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e180)", 2024-09-10T19:19:17.824 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-10T19:17:28.948552+0000", 2024-09-10T19:19:17.824 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 108.720449871, 2024-09-10T19:19:17.824 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.130828164, 2024-09-10T19:19:17.824 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-10T19:19:17.824 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T19:19:17.824 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-10T19:19:17.824 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.15051", 2024-09-10T19:19:17.825 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.183:0/3632028672", 2024-09-10T19:19:17.825 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 93231 2024-09-10T19:19:17.825 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:19:17.825 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-10T19:19:17.825 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:19:17.825 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-10T19:19:17.825 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:17:28.948552+0000", 2024-09-10T19:19:17.825 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:19:17.825 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:19:17.825 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:19:17.825 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-10T19:19:17.825 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:17:28.948552+0000", 2024-09-10T19:19:17.825 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:19:17.826 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:19:17.826 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:19:17.826 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-10T19:19:17.826 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:17:28.948552+0000", 2024-09-10T19:19:17.826 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4294967295.9999995 2024-09-10T19:19:17.826 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:19:17.826 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:19:17.826 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-10T19:19:17.826 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:17:28.948572+0000", 2024-09-10T19:19:17.826 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.0670999999999999e-05 2024-09-10T19:19:17.826 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:19:17.826 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:19:17.826 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-10T19:19:17.827 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:17:28.948579+0000", 2024-09-10T19:19:17.827 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.6579999999999996e-06 2024-09-10T19:19:17.827 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:19:17.827 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:19:17.827 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-10T19:19:17.827 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:17:28.948585+0000", 2024-09-10T19:19:17.827 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.1759999999999998e-06 2024-09-10T19:19:17.827 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:19:17.827 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:19:17.827 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-10T19:19:17.827 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:17:28.950169+0000", 2024-09-10T19:19:17.828 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0015840559999999999 2024-09-10T19:19:17.828 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:19:17.828 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:19:17.828 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for peered", 2024-09-10T19:19:17.828 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:17:28.950175+0000", 2024-09-10T19:19:17.828 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.6180000000000003e-06 2024-09-10T19:19:17.828 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:19:17.828 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:19:17.828 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-10T19:19:17.828 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:17:30.048715+0000", 2024-09-10T19:19:17.828 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.098540498 2024-09-10T19:19:17.828 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:19:17.829 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:19:17.829 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-10T19:19:17.829 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:17:30.048825+0000", 2024-09-10T19:19:17.829 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00010979 2024-09-10T19:19:17.829 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:19:17.829 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:19:17.829 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 4,5,7", 2024-09-10T19:19:17.829 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:17:30.049026+0000", 2024-09-10T19:19:17.829 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00020112 2024-09-10T19:19:17.829 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:19:17.829 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:19:17.829 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-10T19:19:17.830 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:17:30.053293+0000", 2024-09-10T19:19:17.830 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0042666020000000004 2024-09-10T19:19:17.830 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:19:17.830 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:19:17.830 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:19:17.830 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:17:30.057625+0000", 2024-09-10T19:19:17.830 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0043322100000000004 2024-09-10T19:19:17.830 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:19:17.830 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:19:17.830 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:19:17.830 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:17:30.059162+0000", 2024-09-10T19:19:17.830 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.001537184 2024-09-10T19:19:17.830 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:19:17.830 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:19:17.831 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:19:17.831 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:17:30.078486+0000", 2024-09-10T19:19:17.831 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.019324158000000001 2024-09-10T19:19:17.831 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:19:17.831 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:19:17.831 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-10T19:19:17.831 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:17:30.078531+0000", 2024-09-10T19:19:17.831 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.4453999999999998e-05 2024-09-10T19:19:17.831 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:19:17.831 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:19:17.831 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-10T19:19:17.831 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:17:30.079380+0000", 2024-09-10T19:19:17.831 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00084943900000000001 2024-09-10T19:19:17.831 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:19:17.832 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:19:17.832 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:19:17.832 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:19:17.832 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:19:17.832 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.15051.0:93370 8.10 8:0d9c3bdb:::benchmark_data_smithi183_114924_object93369:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e180)", 2024-09-10T19:19:17.832 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-10T19:17:28.948817+0000", 2024-09-10T19:19:17.832 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 108.72018530699999, 2024-09-10T19:19:17.832 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.100337361, 2024-09-10T19:19:17.832 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-10T19:19:17.832 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T19:19:17.832 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-10T19:19:17.832 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.15051", 2024-09-10T19:19:17.833 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.183:0/3632028672", 2024-09-10T19:19:17.833 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 93370 2024-09-10T19:19:17.833 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:19:17.833 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-10T19:19:17.833 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:19:17.833 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-10T19:19:17.833 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:17:28.948817+0000", 2024-09-10T19:19:17.833 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:19:17.833 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:19:17.833 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:19:17.833 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-10T19:19:17.833 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:17:28.948817+0000", 2024-09-10T19:19:17.833 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:19:17.834 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:19:17.834 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:19:17.834 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-10T19:19:17.834 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:17:28.948816+0000", 2024-09-10T19:19:17.834 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4294967295.9999995 2024-09-10T19:19:17.834 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:19:17.834 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:19:17.834 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-10T19:19:17.834 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:17:28.948835+0000", 2024-09-10T19:19:17.834 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.8780999999999999e-05 2024-09-10T19:19:17.834 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:19:17.834 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:19:17.834 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-10T19:19:17.835 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:17:28.948843+0000", 2024-09-10T19:19:17.835 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.6299999999999998e-06 2024-09-10T19:19:17.835 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:19:17.835 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:19:17.835 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-10T19:19:17.835 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:17:28.948847+0000", 2024-09-10T19:19:17.835 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.0069999999999999e-06 2024-09-10T19:19:17.835 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:19:17.835 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:19:17.835 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-10T19:19:17.836 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:17:28.950971+0000", 2024-09-10T19:19:17.836 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0021241039999999999 2024-09-10T19:19:17.836 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:19:17.836 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:19:17.836 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for peered", 2024-09-10T19:19:17.836 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:17:28.950977+0000", 2024-09-10T19:19:17.836 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.2299999999999996e-06 2024-09-10T19:19:17.836 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:19:17.836 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:19:17.836 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-10T19:19:17.836 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:17:30.036595+0000", 2024-09-10T19:19:17.836 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.085617944 2024-09-10T19:19:17.836 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:19:17.837 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:19:17.837 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-10T19:19:17.837 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:17:30.036740+0000", 2024-09-10T19:19:17.837 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00014525000000000001 2024-09-10T19:19:17.837 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:19:17.837 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:19:17.837 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 4,5,7", 2024-09-10T19:19:17.837 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:17:30.036997+0000", 2024-09-10T19:19:17.837 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00025669700000000001 2024-09-10T19:19:17.837 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:19:17.837 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:19:17.837 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-10T19:19:17.837 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:17:30.038616+0000", 2024-09-10T19:19:17.838 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.001619437 2024-09-10T19:19:17.838 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:19:17.838 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:19:17.838 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:19:17.838 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:17:30.042016+0000", 2024-09-10T19:19:17.838 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0033999960000000002 2024-09-10T19:19:17.838 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:19:17.838 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:19:17.838 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:19:17.838 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:17:30.048991+0000", 2024-09-10T19:19:17.838 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0069749520000000004 2024-09-10T19:19:17.838 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:19:17.839 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:19:17.839 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:19:17.839 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:17:30.049015+0000", 2024-09-10T19:19:17.839 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.4096000000000001e-05 2024-09-10T19:19:17.839 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:19:17.839 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:19:17.839 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-10T19:19:17.839 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:17:30.049046+0000", 2024-09-10T19:19:17.839 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.0400999999999999e-05 2024-09-10T19:19:17.839 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:19:17.839 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:19:17.839 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-10T19:19:17.839 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:17:30.049154+0000", 2024-09-10T19:19:17.839 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000108356 2024-09-10T19:19:17.839 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:19:17.840 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:19:17.840 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:19:17.840 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:19:17.840 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:19:17.840 INFO:teuthology.orchestra.run.smithi183.stdout:} 2024-09-10T19:19:17.859 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:19:17.917 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:19:17.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:17.918 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:19:18.019 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:19:18.055 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:19:18.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:18.056 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:19:18.157 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:19:18.184 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:19:18.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:18.185 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:19:18.286 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:19:18.321 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:19:18.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:18.322 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:19:18.355 INFO:teuthology.orchestra.run.smithi183.stderr:set 2.18 pg_upmap_items mapping to [6->3,7->0,5->4] 2024-09-10T19:19:18.423 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:19:18.462 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:19:18.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:18.463 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:19:18.563 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:19:18.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:18.603 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:19:18.603 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:19:18.681 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:19:18 smithi183 ceph-mon[41562]: from='client.? 172.21.15.183:0/3906469181' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "2.18", "id": [6, 3, 7, 0, 5, 4]}]: dispatch 2024-09-10T19:19:18.681 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:19:18 smithi183 ceph-mon[41562]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "2.18", "id": [6, 3, 7, 0, 5, 4]}]: dispatch 2024-09-10T19:19:18.681 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:19:18 smithi183 ceph-mon[41562]: 5.3 scrub starts 2024-09-10T19:19:18.681 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:19:18 smithi183 ceph-mon[41562]: 5.3 scrub ok 2024-09-10T19:19:18.682 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:19:18 smithi183 ceph-mon[41562]: 5.e scrub starts 2024-09-10T19:19:18.682 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:19:18 smithi183 ceph-mon[41562]: 5.e scrub ok 2024-09-10T19:19:18.682 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:19:18 smithi183 ceph-mon[41562]: 2.12 scrub starts 2024-09-10T19:19:18.682 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:19:18 smithi183 ceph-mon[41562]: 2.12 scrub ok 2024-09-10T19:19:18.682 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:19:18 smithi183 ceph-mon[41562]: pgmap v403: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 1.6 GiB data, 8.3 GiB used, 528 GiB / 536 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1.42k op/s; 1744/79473 objects misplaced (2.194%) 2024-09-10T19:19:18.682 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:19:18 smithi183 ceph-mon[29736]: from='client.? 172.21.15.183:0/3906469181' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "2.18", "id": [6, 3, 7, 0, 5, 4]}]: dispatch 2024-09-10T19:19:18.682 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:19:18 smithi183 ceph-mon[29736]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "2.18", "id": [6, 3, 7, 0, 5, 4]}]: dispatch 2024-09-10T19:19:18.682 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:19:18 smithi183 ceph-mon[29736]: 5.3 scrub starts 2024-09-10T19:19:18.682 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:19:18 smithi183 ceph-mon[29736]: 5.3 scrub ok 2024-09-10T19:19:18.682 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:19:18 smithi183 ceph-mon[29736]: 5.e scrub starts 2024-09-10T19:19:18.682 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:19:18 smithi183 ceph-mon[29736]: 5.e scrub ok 2024-09-10T19:19:18.683 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:19:18 smithi183 ceph-mon[29736]: 2.12 scrub starts 2024-09-10T19:19:18.683 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:19:18 smithi183 ceph-mon[29736]: 2.12 scrub ok 2024-09-10T19:19:18.683 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:19:18 smithi183 ceph-mon[29736]: pgmap v403: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 1.6 GiB data, 8.3 GiB used, 528 GiB / 536 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1.42k op/s; 1744/79473 objects misplaced (2.194%) 2024-09-10T19:19:18.704 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:19:18.715 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:19:18 smithi191 ceph-mon[32884]: from='client.? 172.21.15.183:0/3906469181' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "2.18", "id": [6, 3, 7, 0, 5, 4]}]: dispatch 2024-09-10T19:19:18.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:19:18 smithi191 ceph-mon[32884]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "2.18", "id": [6, 3, 7, 0, 5, 4]}]: dispatch 2024-09-10T19:19:18.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:19:18 smithi191 ceph-mon[32884]: 5.3 scrub starts 2024-09-10T19:19:18.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:19:18 smithi191 ceph-mon[32884]: 5.3 scrub ok 2024-09-10T19:19:18.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:19:18 smithi191 ceph-mon[32884]: 5.e scrub starts 2024-09-10T19:19:18.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:19:18 smithi191 ceph-mon[32884]: 5.e scrub ok 2024-09-10T19:19:18.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:19:18 smithi191 ceph-mon[32884]: 2.12 scrub starts 2024-09-10T19:19:18.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:19:18 smithi191 ceph-mon[32884]: 2.12 scrub ok 2024-09-10T19:19:18.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:19:18 smithi191 ceph-mon[32884]: pgmap v403: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 1.6 GiB data, 8.3 GiB used, 528 GiB / 536 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1.42k op/s; 1744/79473 objects misplaced (2.194%) 2024-09-10T19:19:18.741 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:19:18.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:18.742 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:19:18.843 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:19:18.869 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:19:18.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:18.870 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:19:18.971 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:19:19.009 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:19:19.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:19.010 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:19:19.111 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:19:19.150 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:19:19.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:19.151 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:19:19.252 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:19:19.287 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:19:19.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:19.288 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:19:19.389 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:19:19.419 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:19:19.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:19.421 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:19:19.466 DEBUG:teuthology.orchestra.run.smithi191:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph daemon osd.4 dump_ops_in_flight 2024-09-10T19:19:19.521 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:19:19.583 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:19:19.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:19.584 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:19:19.611 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:19:19.681 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:19:19 smithi183 ceph-mon[29736]: 6.13 scrub starts 2024-09-10T19:19:19.681 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:19:19 smithi183 ceph-mon[29736]: 6.13 scrub ok 2024-09-10T19:19:19.681 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:19:19 smithi183 ceph-mon[29736]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "2.18", "id": [6, 3, 7, 0, 5, 4]}]': finished 2024-09-10T19:19:19.682 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:19:19 smithi183 ceph-mon[29736]: osdmap e217: 8 total, 8 up, 6 in 2024-09-10T19:19:19.682 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:19:19 smithi183 ceph-mon[29736]: 4.c scrub starts 2024-09-10T19:19:19.682 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:19:19 smithi183 ceph-mon[29736]: 4.c scrub ok 2024-09-10T19:19:19.682 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:19:19 smithi183 ceph-mon[41562]: 6.13 scrub starts 2024-09-10T19:19:19.682 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:19:19 smithi183 ceph-mon[41562]: 6.13 scrub ok 2024-09-10T19:19:19.682 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:19:19 smithi183 ceph-mon[41562]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "2.18", "id": [6, 3, 7, 0, 5, 4]}]': finished 2024-09-10T19:19:19.682 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:19:19 smithi183 ceph-mon[41562]: osdmap e217: 8 total, 8 up, 6 in 2024-09-10T19:19:19.682 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:19:19 smithi183 ceph-mon[41562]: 4.c scrub starts 2024-09-10T19:19:19.682 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:19:19 smithi183 ceph-mon[41562]: 4.c scrub ok 2024-09-10T19:19:19.685 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:19:19.711 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:19:19.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:19.712 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:19:19.715 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:19:19 smithi191 ceph-mon[32884]: 6.13 scrub starts 2024-09-10T19:19:19.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:19:19 smithi191 ceph-mon[32884]: 6.13 scrub ok 2024-09-10T19:19:19.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:19:19 smithi191 ceph-mon[32884]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "2.18", "id": [6, 3, 7, 0, 5, 4]}]': finished 2024-09-10T19:19:19.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:19:19 smithi191 ceph-mon[32884]: osdmap e217: 8 total, 8 up, 6 in 2024-09-10T19:19:19.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:19:19 smithi191 ceph-mon[32884]: 4.c scrub starts 2024-09-10T19:19:19.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:19:19 smithi191 ceph-mon[32884]: 4.c scrub ok 2024-09-10T19:19:19.716 INFO:journalctl@ceph.iscsi.iscsi.a.smithi191.stdout:Sep 10 19:19:19 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-iscsi-iscsi-a[61816]: debug there is no tcmu-runner data available 2024-09-10T19:19:19.813 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:19:19.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:19.867 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:19:19.867 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:19:19.918 INFO:teuthology.orchestra.run.smithi191.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.b/config 2024-09-10T19:19:19.968 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:19:20.018 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:19:20.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:20.019 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:19:20.120 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:19:20.147 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:19:20.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:20.148 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:19:20.249 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:19:20.286 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:19:20.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:20.287 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:19:20.376 INFO:teuthology.orchestra.run.smithi183.stderr:nodeep-scrub is unset 2024-09-10T19:19:20.387 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:19:20.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:20.420 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:19:20.420 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:19:20.466 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:19:20 smithi183 ceph-mon[29736]: 5.9 scrub starts 2024-09-10T19:19:20.466 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:19:20 smithi183 ceph-mon[29736]: 5.9 scrub ok 2024-09-10T19:19:20.466 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:19:20 smithi183 ceph-mon[29736]: osdmap e218: 8 total, 8 up, 6 in 2024-09-10T19:19:20.466 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:19:20 smithi183 ceph-mon[29736]: from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:19:20.466 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:19:20 smithi183 ceph-mon[29736]: 2.18 scrub starts 2024-09-10T19:19:20.466 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:19:20 smithi183 ceph-mon[29736]: 2.18 scrub ok 2024-09-10T19:19:20.467 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:19:20 smithi183 ceph-mon[29736]: pgmap v406: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 1.6 GiB data, 8.3 GiB used, 528 GiB / 536 GiB avail; 639 B/s rd, 0 B/s wr, 886 op/s; 1744/79473 objects misplaced (2.194%) 2024-09-10T19:19:20.467 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:19:20 smithi183 ceph-mon[29736]: from='client.? 172.21.15.183:0/229735324' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-10T19:19:20.467 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:19:20 smithi183 ceph-mon[41562]: 5.9 scrub starts 2024-09-10T19:19:20.467 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:19:20 smithi183 ceph-mon[41562]: 5.9 scrub ok 2024-09-10T19:19:20.467 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:19:20 smithi183 ceph-mon[41562]: osdmap e218: 8 total, 8 up, 6 in 2024-09-10T19:19:20.467 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:19:20 smithi183 ceph-mon[41562]: from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:19:20.467 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:19:20 smithi183 ceph-mon[41562]: 2.18 scrub starts 2024-09-10T19:19:20.467 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:19:20 smithi183 ceph-mon[41562]: 2.18 scrub ok 2024-09-10T19:19:20.467 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:19:20 smithi183 ceph-mon[41562]: pgmap v406: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 1.6 GiB data, 8.3 GiB used, 528 GiB / 536 GiB avail; 639 B/s rd, 0 B/s wr, 886 op/s; 1744/79473 objects misplaced (2.194%) 2024-09-10T19:19:20.467 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:19:20 smithi183 ceph-mon[41562]: from='client.? 172.21.15.183:0/229735324' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-10T19:19:20.521 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:19:20.548 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:19:20.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:20.549 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:19:20.650 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:19:20.689 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:19:20.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:20.690 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:19:20.715 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:19:20 smithi191 ceph-mon[32884]: 5.9 scrub starts 2024-09-10T19:19:20.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:19:20 smithi191 ceph-mon[32884]: 5.9 scrub ok 2024-09-10T19:19:20.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:19:20 smithi191 ceph-mon[32884]: osdmap e218: 8 total, 8 up, 6 in 2024-09-10T19:19:20.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:19:20 smithi191 ceph-mon[32884]: from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:19:20.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:19:20 smithi191 ceph-mon[32884]: 2.18 scrub starts 2024-09-10T19:19:20.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:19:20 smithi191 ceph-mon[32884]: 2.18 scrub ok 2024-09-10T19:19:20.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:19:20 smithi191 ceph-mon[32884]: pgmap v406: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 1.6 GiB data, 8.3 GiB used, 528 GiB / 536 GiB avail; 639 B/s rd, 0 B/s wr, 886 op/s; 1744/79473 objects misplaced (2.194%) 2024-09-10T19:19:20.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:19:20 smithi191 ceph-mon[32884]: from='client.? 172.21.15.183:0/229735324' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-10T19:19:20.791 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:19:20.817 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:19:20.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:20.819 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:19:20.919 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:19:20.948 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:19:20.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:20.949 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:19:21.050 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:19:21.083 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:19:21.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:21.084 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:19:21.185 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:19:21.215 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:19:21.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:21.216 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:19:21.316 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:19:21.347 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:19:21.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:21.348 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:19:21.449 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:19:21.501 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:19:21.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:21.502 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:19:21.602 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:19:21.649 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:19:21.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:21.651 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:19:21.681 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:19:21 smithi183 ceph-mon[29736]: 5.10 scrub starts 2024-09-10T19:19:21.681 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:19:21 smithi183 ceph-mon[29736]: 5.10 scrub ok 2024-09-10T19:19:21.681 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:19:21 smithi183 ceph-mon[29736]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-10T19:19:21.681 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:19:21 smithi183 ceph-mon[29736]: from='client.? 172.21.15.183:0/229735324' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-10T19:19:21.681 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:19:21 smithi183 ceph-mon[29736]: osdmap e219: 8 total, 8 up, 6 in 2024-09-10T19:19:21.682 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:19:21 smithi183 ceph-mon[41562]: 5.10 scrub starts 2024-09-10T19:19:21.682 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:19:21 smithi183 ceph-mon[41562]: 5.10 scrub ok 2024-09-10T19:19:21.682 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:19:21 smithi183 ceph-mon[41562]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-10T19:19:21.682 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:19:21 smithi183 ceph-mon[41562]: from='client.? 172.21.15.183:0/229735324' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-10T19:19:21.682 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:19:21 smithi183 ceph-mon[41562]: osdmap e219: 8 total, 8 up, 6 in 2024-09-10T19:19:21.682 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Sep 10 19:19:21 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-y[29989]: ::ffff:172.21.15.191 - - [10/Sep/2024:19:19:21] "GET /metrics HTTP/1.1" 200 224289 "" "Prometheus/2.43.0" 2024-09-10T19:19:21.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:19:21 smithi191 ceph-mon[32884]: 5.10 scrub starts 2024-09-10T19:19:21.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:19:21 smithi191 ceph-mon[32884]: 5.10 scrub ok 2024-09-10T19:19:21.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:19:21 smithi191 ceph-mon[32884]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-10T19:19:21.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:19:21 smithi191 ceph-mon[32884]: from='client.? 172.21.15.183:0/229735324' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-10T19:19:21.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:19:21 smithi191 ceph-mon[32884]: osdmap e219: 8 total, 8 up, 6 in 2024-09-10T19:19:21.752 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:19:21.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:21.790 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:19:21.790 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:19:21.891 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:19:21.937 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:19:21.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:21.938 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:19:22.039 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:19:22.040 INFO:teuthology.orchestra.run.smithi191.stdout:{ 2024-09-10T19:19:22.040 INFO:teuthology.orchestra.run.smithi191.stdout: "ops": [ 2024-09-10T19:19:22.040 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:19:22.041 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.15051.0:224194 8.14 8:2f6543a8:::benchmark_data_smithi183_114924_object103911:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e219)", 2024-09-10T19:19:22.041 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-10T19:19:22.024959+0000", 2024-09-10T19:19:22.041 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 0.0092259649999999992, 2024-09-10T19:19:22.041 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0092560890000000003, 2024-09-10T19:19:22.041 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-10T19:19:22.041 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "waiting for sub ops", 2024-09-10T19:19:22.041 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-10T19:19:22.041 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.15051", 2024-09-10T19:19:22.041 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.183:0/3632028672", 2024-09-10T19:19:22.041 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 224194 2024-09-10T19:19:22.041 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:19:22.041 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-10T19:19:22.041 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:19:22.042 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-10T19:19:22.042 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:19:22.024959+0000", 2024-09-10T19:19:22.042 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:19:22.042 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:19:22.042 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:19:22.042 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-10T19:19:22.042 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:19:22.024959+0000", 2024-09-10T19:19:22.042 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:19:22.042 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:19:22.042 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:19:22.042 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-10T19:19:22.042 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:19:22.024957+0000", 2024-09-10T19:19:22.042 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999986 2024-09-10T19:19:22.043 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:19:22.043 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:19:22.043 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-10T19:19:22.043 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:19:22.024962+0000", 2024-09-10T19:19:22.043 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.5929999999999999e-06 2024-09-10T19:19:22.043 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:19:22.043 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:19:22.043 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-10T19:19:22.043 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:19:22.024970+0000", 2024-09-10T19:19:22.043 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.5220000000000007e-06 2024-09-10T19:19:22.043 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:19:22.043 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:19:22.043 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-10T19:19:22.044 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:19:22.024975+0000", 2024-09-10T19:19:22.044 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.2069999999999997e-06 2024-09-10T19:19:22.044 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:19:22.044 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:19:22.044 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:19:22.044 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:19:22.025789+0000", 2024-09-10T19:19:22.044 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00081489799999999999 2024-09-10T19:19:22.044 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:19:22.044 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:19:22.044 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-10T19:19:22.044 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:19:22.025919+0000", 2024-09-10T19:19:22.044 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00013004999999999999 2024-09-10T19:19:22.045 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:19:22.045 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:19:22.045 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0,7", 2024-09-10T19:19:22.045 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:19:22.026030+0000", 2024-09-10T19:19:22.045 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00011099699999999999 2024-09-10T19:19:22.045 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:19:22.045 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:19:22.045 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:19:22.045 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:19:22.030738+0000", 2024-09-10T19:19:22.045 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0047073369999999998 2024-09-10T19:19:22.045 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:19:22.045 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-10T19:19:22.045 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:19:22.045 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:19:22.046 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:19:22.046 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_repop(client.15051.0:224199 8.a e219/154 8:5432d31e:::benchmark_data_smithi183_114924_object103916:head v 219'14327, mlcod=219'14325)", 2024-09-10T19:19:22.046 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-10T19:19:22.028201+0000", 2024-09-10T19:19:22.046 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 0.0059832949999999996, 2024-09-10T19:19:22.046 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0061047219999999999, 2024-09-10T19:19:22.046 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-10T19:19:22.046 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "started", 2024-09-10T19:19:22.046 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-10T19:19:22.046 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:19:22.046 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-10T19:19:22.046 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:19:22.028201+0000", 2024-09-10T19:19:22.046 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:19:22.046 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:19:22.046 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:19:22.046 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-10T19:19:22.047 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:19:22.028201+0000", 2024-09-10T19:19:22.047 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:19:22.047 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:19:22.047 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:19:22.047 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-10T19:19:22.047 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:19:22.028201+0000", 2024-09-10T19:19:22.047 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999995 2024-09-10T19:19:22.047 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:19:22.047 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:19:22.047 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-10T19:19:22.047 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:19:22.028207+0000", 2024-09-10T19:19:22.047 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.8579999999999996e-06 2024-09-10T19:19:22.047 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:19:22.047 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:19:22.048 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-10T19:19:22.048 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:19:22.028216+0000", 2024-09-10T19:19:22.048 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.7890000000000006e-06 2024-09-10T19:19:22.048 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:19:22.048 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:19:22.048 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-10T19:19:22.048 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:19:22.028222+0000", 2024-09-10T19:19:22.048 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.4009999999999996e-06 2024-09-10T19:19:22.048 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:19:22.048 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:19:22.048 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:19:22.048 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:19:22.033064+0000", 2024-09-10T19:19:22.048 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.004842542 2024-09-10T19:19:22.049 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:19:22.049 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:19:22.049 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-10T19:19:22.049 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:19:22.033082+0000", 2024-09-10T19:19:22.049 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.8162e-05 2024-09-10T19:19:22.049 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:19:22.049 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-10T19:19:22.049 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:19:22.049 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:19:22.049 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:19:22.049 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.15051.0:224205 8.14 8:2fc812f9:::benchmark_data_smithi183_114924_object103922:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e219)", 2024-09-10T19:19:22.049 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-10T19:19:22.028371+0000", 2024-09-10T19:19:22.049 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 0.0058139380000000003, 2024-09-10T19:19:22.050 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0059942509999999999, 2024-09-10T19:19:22.050 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-10T19:19:22.050 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "waiting for sub ops", 2024-09-10T19:19:22.050 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-10T19:19:22.050 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.15051", 2024-09-10T19:19:22.050 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.183:0/3632028672", 2024-09-10T19:19:22.050 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 224205 2024-09-10T19:19:22.050 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:19:22.050 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-10T19:19:22.050 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:19:22.050 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-10T19:19:22.050 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:19:22.028371+0000", 2024-09-10T19:19:22.050 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:19:22.050 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:19:22.051 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:19:22.051 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-10T19:19:22.051 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:19:22.028371+0000", 2024-09-10T19:19:22.051 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:19:22.051 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:19:22.051 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:19:22.051 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-10T19:19:22.051 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:19:22.028370+0000", 2024-09-10T19:19:22.051 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.999999 2024-09-10T19:19:22.051 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:19:22.051 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:19:22.051 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-10T19:19:22.051 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:19:22.028374+0000", 2024-09-10T19:19:22.052 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.8049999999999997e-06 2024-09-10T19:19:22.052 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:19:22.052 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:19:22.052 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-10T19:19:22.052 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:19:22.028384+0000", 2024-09-10T19:19:22.052 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.7759999999999992e-06 2024-09-10T19:19:22.052 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:19:22.052 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:19:22.052 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-10T19:19:22.052 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:19:22.028389+0000", 2024-09-10T19:19:22.052 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.2850000000000002e-06 2024-09-10T19:19:22.052 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:19:22.053 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:19:22.053 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:19:22.053 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:19:22.028484+0000", 2024-09-10T19:19:22.053 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.4764000000000003e-05 2024-09-10T19:19:22.053 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:19:22.053 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:19:22.053 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-10T19:19:22.053 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:19:22.028653+0000", 2024-09-10T19:19:22.053 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00016876900000000001 2024-09-10T19:19:22.053 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:19:22.053 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:19:22.053 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0,7", 2024-09-10T19:19:22.054 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:19:22.028769+0000", 2024-09-10T19:19:22.054 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000115918 2024-09-10T19:19:22.054 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:19:22.054 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-10T19:19:22.054 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:19:22.054 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:19:22.054 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:19:22.054 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_repop(client.15051.0:224203 8.4 e219/177 8:26546c58:::benchmark_data_smithi183_114924_object103920:head v 219'9731, mlcod=219'9729)", 2024-09-10T19:19:22.054 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-10T19:19:22.028417+0000", 2024-09-10T19:19:22.054 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 0.0057681130000000001, 2024-09-10T19:19:22.054 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0060281090000000002, 2024-09-10T19:19:22.054 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-10T19:19:22.054 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "started", 2024-09-10T19:19:22.055 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-10T19:19:22.055 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:19:22.055 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-10T19:19:22.055 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:19:22.028417+0000", 2024-09-10T19:19:22.055 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:19:22.055 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:19:22.055 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:19:22.055 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-10T19:19:22.055 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:19:22.028417+0000", 2024-09-10T19:19:22.055 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:19:22.055 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:19:22.055 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:19:22.055 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-10T19:19:22.056 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:19:22.028416+0000", 2024-09-10T19:19:22.056 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999995 2024-09-10T19:19:22.056 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:19:22.056 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:19:22.056 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-10T19:19:22.056 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:19:22.028421+0000", 2024-09-10T19:19:22.056 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.4599999999999996e-06 2024-09-10T19:19:22.056 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:19:22.056 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:19:22.056 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-10T19:19:22.056 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:19:22.028428+0000", 2024-09-10T19:19:22.056 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.8929999999999999e-06 2024-09-10T19:19:22.057 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:19:22.057 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:19:22.057 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-10T19:19:22.057 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:19:22.028432+0000", 2024-09-10T19:19:22.057 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.0799999999999999e-06 2024-09-10T19:19:22.057 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:19:22.057 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:19:22.057 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:19:22.057 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:19:22.030761+0000", 2024-09-10T19:19:22.057 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0023298199999999998 2024-09-10T19:19:22.057 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:19:22.057 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:19:22.057 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-10T19:19:22.058 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:19:22.030777+0000", 2024-09-10T19:19:22.058 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.5231e-05 2024-09-10T19:19:22.058 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:19:22.058 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-10T19:19:22.058 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:19:22.058 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:19:22.058 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:19:22.058 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_repop(client.15051.0:224204 8.7 e219/177 8:e14902d9:::benchmark_data_smithi183_114924_object103921:head v 219'9815, mlcod=219'9813)", 2024-09-10T19:19:22.058 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-10T19:19:22.030774+0000", 2024-09-10T19:19:22.058 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 0.0034102989999999999, 2024-09-10T19:19:22.058 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0037238639999999999, 2024-09-10T19:19:22.058 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-10T19:19:22.059 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "started", 2024-09-10T19:19:22.059 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-10T19:19:22.059 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:19:22.059 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-10T19:19:22.059 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:19:22.030774+0000", 2024-09-10T19:19:22.059 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:19:22.059 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:19:22.059 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:19:22.059 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-10T19:19:22.059 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:19:22.030774+0000", 2024-09-10T19:19:22.059 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:19:22.059 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:19:22.059 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:19:22.060 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-10T19:19:22.060 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:19:22.030773+0000", 2024-09-10T19:19:22.060 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.999999 2024-09-10T19:19:22.060 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:19:22.060 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:19:22.060 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-10T19:19:22.060 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:19:22.030784+0000", 2024-09-10T19:19:22.060 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.0462000000000001e-05 2024-09-10T19:19:22.060 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:19:22.060 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:19:22.060 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-10T19:19:22.060 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:19:22.030796+0000", 2024-09-10T19:19:22.060 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.2266000000000001e-05 2024-09-10T19:19:22.061 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:19:22.061 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:19:22.061 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-10T19:19:22.061 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:19:22.030802+0000", 2024-09-10T19:19:22.061 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.1260000000000003e-06 2024-09-10T19:19:22.061 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:19:22.061 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:19:22.061 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:19:22.061 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:19:22.031080+0000", 2024-09-10T19:19:22.061 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00027765299999999999 2024-09-10T19:19:22.061 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:19:22.061 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:19:22.062 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-10T19:19:22.062 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:19:22.031101+0000", 2024-09-10T19:19:22.062 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.0747e-05 2024-09-10T19:19:22.062 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:19:22.062 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-10T19:19:22.062 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:19:22.062 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:19:22.062 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:19:22.062 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_repop(client.15051.0:224206 8.7 e219/177 8:e007e9d3:::benchmark_data_smithi183_114924_object103923:head v 219'9816, mlcod=219'9813)", 2024-09-10T19:19:22.062 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-10T19:19:22.030816+0000", 2024-09-10T19:19:22.062 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 0.0033691120000000001, 2024-09-10T19:19:22.062 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.003739844, 2024-09-10T19:19:22.062 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-10T19:19:22.062 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "started", 2024-09-10T19:19:22.063 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-10T19:19:22.063 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:19:22.063 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-10T19:19:22.063 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:19:22.030816+0000", 2024-09-10T19:19:22.063 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:19:22.063 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:19:22.063 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:19:22.063 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-10T19:19:22.063 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:19:22.030816+0000", 2024-09-10T19:19:22.063 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:19:22.063 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:19:22.063 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:19:22.063 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-10T19:19:22.063 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:19:22.030815+0000", 2024-09-10T19:19:22.064 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999995 2024-09-10T19:19:22.064 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:19:22.064 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:19:22.064 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-10T19:19:22.064 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:19:22.030819+0000", 2024-09-10T19:19:22.064 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.0600000000000001e-06 2024-09-10T19:19:22.064 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:19:22.064 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:19:22.064 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-10T19:19:22.064 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:19:22.030826+0000", 2024-09-10T19:19:22.064 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.742e-06 2024-09-10T19:19:22.064 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:19:22.064 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:19:22.065 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-10T19:19:22.065 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:19:22.030830+0000", 2024-09-10T19:19:22.065 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.3710000000000002e-06 2024-09-10T19:19:22.065 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:19:22.065 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:19:22.065 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:19:22.065 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:19:22.034225+0000", 2024-09-10T19:19:22.065 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0033942270000000001 2024-09-10T19:19:22.065 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:19:22.065 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:19:22.065 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-10T19:19:22.065 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:19:22.034244+0000", 2024-09-10T19:19:22.065 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.9082000000000001e-05 2024-09-10T19:19:22.066 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:19:22.066 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-10T19:19:22.066 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:19:22.066 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:19:22.066 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:19:22.066 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.15051.0:224207 8.2 8:4458e82c:::benchmark_data_smithi183_114924_object103924:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e219)", 2024-09-10T19:19:22.066 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-10T19:19:22.031268+0000", 2024-09-10T19:19:22.066 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 0.0029171380000000001, 2024-09-10T19:19:22.066 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0033407689999999999, 2024-09-10T19:19:22.066 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-10T19:19:22.066 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "waiting for sub ops", 2024-09-10T19:19:22.066 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-10T19:19:22.066 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.15051", 2024-09-10T19:19:22.066 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.183:0/3632028672", 2024-09-10T19:19:22.067 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 224207 2024-09-10T19:19:22.067 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:19:22.067 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-10T19:19:22.067 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:19:22.067 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-10T19:19:22.067 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:19:22.031268+0000", 2024-09-10T19:19:22.067 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:19:22.067 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:19:22.067 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:19:22.067 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-10T19:19:22.067 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:19:22.031268+0000", 2024-09-10T19:19:22.067 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:19:22.067 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:19:22.067 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:19:22.068 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-10T19:19:22.068 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:19:22.031266+0000", 2024-09-10T19:19:22.068 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.999999 2024-09-10T19:19:22.068 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:19:22.068 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:19:22.068 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-10T19:19:22.068 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:19:22.031272+0000", 2024-09-10T19:19:22.068 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.2619999999999998e-06 2024-09-10T19:19:22.068 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:19:22.068 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:19:22.068 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-10T19:19:22.068 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:19:22.031283+0000", 2024-09-10T19:19:22.068 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.1110999999999999e-05 2024-09-10T19:19:22.068 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:19:22.069 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:19:22.069 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-10T19:19:22.069 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:19:22.031288+0000", 2024-09-10T19:19:22.069 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.9400000000000001e-06 2024-09-10T19:19:22.069 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:19:22.069 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:19:22.069 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:19:22.069 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:19:22.033808+0000", 2024-09-10T19:19:22.069 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0025207419999999999 2024-09-10T19:19:22.069 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:19:22.069 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:19:22.069 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-10T19:19:22.069 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:19:22.033928+0000", 2024-09-10T19:19:22.070 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00011945799999999999 2024-09-10T19:19:22.070 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:19:22.070 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:19:22.070 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0,5", 2024-09-10T19:19:22.070 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:19:22.034040+0000", 2024-09-10T19:19:22.070 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00011171900000000001 2024-09-10T19:19:22.070 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:19:22.071 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-10T19:19:22.071 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:19:22.071 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:19:22.071 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:19:22.071 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_repop(client.15051.0:224197 8.10 e219/187)", 2024-09-10T19:19:22.071 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-10T19:19:22.033783+0000", 2024-09-10T19:19:22.071 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 0.00040144600000000002, 2024-09-10T19:19:22.071 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00088749599999999999, 2024-09-10T19:19:22.071 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-10T19:19:22.071 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "queued for pg", 2024-09-10T19:19:22.071 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-10T19:19:22.071 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:19:22.071 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-10T19:19:22.072 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:19:22.033783+0000", 2024-09-10T19:19:22.072 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:19:22.072 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:19:22.072 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:19:22.072 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-10T19:19:22.072 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:19:22.033783+0000", 2024-09-10T19:19:22.072 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:19:22.072 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:19:22.072 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:19:22.072 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-10T19:19:22.072 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:19:22.033782+0000", 2024-09-10T19:19:22.072 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.999999 2024-09-10T19:19:22.072 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:19:22.073 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:19:22.073 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-10T19:19:22.073 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:19:22.033789+0000", 2024-09-10T19:19:22.073 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.1520000000000003e-06 2024-09-10T19:19:22.073 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:19:22.073 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:19:22.073 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-10T19:19:22.073 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:19:22.033801+0000", 2024-09-10T19:19:22.073 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.1243999999999999e-05 2024-09-10T19:19:22.073 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:19:22.073 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:19:22.073 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-10T19:19:22.073 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:19:22.033807+0000", 2024-09-10T19:19:22.074 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.246e-06 2024-09-10T19:19:22.074 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:19:22.074 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-10T19:19:22.074 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:19:22.074 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:19:22.074 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:19:22.074 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_repop(client.15051.0:224208 8.7 e219/177)", 2024-09-10T19:19:22.074 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-10T19:19:22.033835+0000", 2024-09-10T19:19:22.074 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 0.00034969100000000001, 2024-09-10T19:19:22.074 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00089461100000000002, 2024-09-10T19:19:22.074 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-10T19:19:22.074 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "queued for pg", 2024-09-10T19:19:22.074 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-10T19:19:22.074 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:19:22.075 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-10T19:19:22.075 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:19:22.033835+0000", 2024-09-10T19:19:22.075 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:19:22.075 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:19:22.075 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:19:22.075 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-10T19:19:22.075 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:19:22.033835+0000", 2024-09-10T19:19:22.075 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:19:22.075 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:19:22.075 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:19:22.075 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-10T19:19:22.075 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:19:22.033835+0000", 2024-09-10T19:19:22.075 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999995 2024-09-10T19:19:22.075 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:19:22.075 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:19:22.076 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-10T19:19:22.076 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:19:22.033841+0000", 2024-09-10T19:19:22.076 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.9530000000000002e-06 2024-09-10T19:19:22.076 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:19:22.076 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:19:22.076 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-10T19:19:22.076 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:19:22.033849+0000", 2024-09-10T19:19:22.076 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.4270000000000002e-06 2024-09-10T19:19:22.076 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:19:22.076 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:19:22.076 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-10T19:19:22.076 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:19:22.033853+0000", 2024-09-10T19:19:22.076 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.3340000000000003e-06 2024-09-10T19:19:22.077 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:19:22.077 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-10T19:19:22.077 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:19:22.077 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:19:22.077 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:19:22.077 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_repop_reply(client.15051.0:224205 8.14 e219/189)", 2024-09-10T19:19:22.077 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-10T19:19:22.034330+0000", 2024-09-10T19:19:22.077 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 4294967295.999855, 2024-09-10T19:19:22.077 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00045548000000000002, 2024-09-10T19:19:22.077 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-10T19:19:22.077 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "queued for pg", 2024-09-10T19:19:22.077 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-10T19:19:22.077 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:19:22.078 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-10T19:19:22.078 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:19:22.034330+0000", 2024-09-10T19:19:22.078 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:19:22.078 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:19:22.078 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:19:22.078 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-10T19:19:22.078 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:19:22.034330+0000", 2024-09-10T19:19:22.078 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:19:22.078 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:19:22.078 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:19:22.078 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-10T19:19:22.078 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:19:22.034329+0000", 2024-09-10T19:19:22.078 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.999999 2024-09-10T19:19:22.078 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:19:22.079 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:19:22.079 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-10T19:19:22.079 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:19:22.034334+0000", 2024-09-10T19:19:22.079 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.3199999999999999e-06 2024-09-10T19:19:22.079 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:19:22.079 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:19:22.079 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-10T19:19:22.079 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:19:22.034345+0000", 2024-09-10T19:19:22.079 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.0389e-05 2024-09-10T19:19:22.079 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:19:22.079 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:19:22.079 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-10T19:19:22.079 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:19:22.034350+0000", 2024-09-10T19:19:22.080 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.259e-06 2024-09-10T19:19:22.080 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:19:22.080 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-10T19:19:22.080 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:19:22.080 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:19:22.080 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:19:22.080 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_repop(client.15051.0:224209 8.f e219/154)", 2024-09-10T19:19:22.080 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-10T19:19:22.034715+0000", 2024-09-10T19:19:22.080 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 4294967295.9994698, 2024-09-10T19:19:22.080 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000109623, 2024-09-10T19:19:22.080 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-10T19:19:22.080 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "queued for pg", 2024-09-10T19:19:22.080 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-10T19:19:22.080 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:19:22.081 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-10T19:19:22.081 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:19:22.034715+0000", 2024-09-10T19:19:22.081 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:19:22.081 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:19:22.081 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:19:22.081 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-10T19:19:22.081 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:19:22.034715+0000", 2024-09-10T19:19:22.081 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:19:22.081 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:19:22.081 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:19:22.081 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-10T19:19:22.081 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:19:22.034714+0000", 2024-09-10T19:19:22.081 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999995 2024-09-10T19:19:22.082 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:19:22.082 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:19:22.082 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-10T19:19:22.082 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:19:22.034720+0000", 2024-09-10T19:19:22.082 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.2419999999999997e-06 2024-09-10T19:19:22.082 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:19:22.082 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:19:22.082 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-10T19:19:22.082 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:19:22.034731+0000", 2024-09-10T19:19:22.082 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.0961999999999999e-05 2024-09-10T19:19:22.082 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:19:22.082 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:19:22.082 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-10T19:19:22.083 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:19:22.034736+0000", 2024-09-10T19:19:22.083 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.498e-06 2024-09-10T19:19:22.083 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:19:22.083 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-10T19:19:22.083 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:19:22.083 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:19:22.083 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:19:22.083 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_repop(client.15051.0:224190 8.12 e219/180 8:4f0bc019:::benchmark_data_smithi183_114924_object103907:head v 219'9995, mlcod=219'9992)", 2024-09-10T19:19:22.083 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-10T19:19:22.017934+0000", 2024-09-10T19:19:22.083 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 0.016251170999999998, 2024-09-10T19:19:22.083 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.016935388999999999, 2024-09-10T19:19:22.083 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-10T19:19:22.083 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "started", 2024-09-10T19:19:22.084 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-10T19:19:22.084 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:19:22.084 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-10T19:19:22.084 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:19:22.017934+0000", 2024-09-10T19:19:22.084 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:19:22.084 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:19:22.084 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:19:22.084 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-10T19:19:22.084 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:19:22.017934+0000", 2024-09-10T19:19:22.084 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:19:22.084 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:19:22.084 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:19:22.084 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-10T19:19:22.084 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:19:22.017933+0000", 2024-09-10T19:19:22.085 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967296 2024-09-10T19:19:22.085 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:19:22.085 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:19:22.085 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-10T19:19:22.085 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:19:22.017937+0000", 2024-09-10T19:19:22.085 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.2839999999999998e-06 2024-09-10T19:19:22.085 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:19:22.085 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:19:22.085 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-10T19:19:22.085 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:19:22.017942+0000", 2024-09-10T19:19:22.085 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.0699999999999997e-06 2024-09-10T19:19:22.085 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:19:22.085 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:19:22.086 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-10T19:19:22.086 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:19:22.017945+0000", 2024-09-10T19:19:22.086 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.8499999999999998e-06 2024-09-10T19:19:22.086 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:19:22.086 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:19:22.086 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:19:22.086 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:19:22.026710+0000", 2024-09-10T19:19:22.086 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0087657629999999993 2024-09-10T19:19:22.086 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:19:22.086 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:19:22.086 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-10T19:19:22.086 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:19:22.026730+0000", 2024-09-10T19:19:22.086 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.9262e-05 2024-09-10T19:19:22.086 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:19:22.087 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-10T19:19:22.087 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:19:22.087 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:19:22.087 INFO:teuthology.orchestra.run.smithi191.stdout: ], 2024-09-10T19:19:22.087 INFO:teuthology.orchestra.run.smithi191.stdout: "num_ops": 12 2024-09-10T19:19:22.087 INFO:teuthology.orchestra.run.smithi191.stdout:} 2024-09-10T19:19:22.105 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:19:22.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:22.106 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:19:22.207 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:19:22.232 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:19:22.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:22.233 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:19:22.334 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:19:22.376 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:19:22.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:22.377 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:19:22.478 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:19:22.513 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:19:22.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:22.514 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:19:22.602 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:19:22 smithi183 ceph-mon[29736]: 2.9 scrub starts 2024-09-10T19:19:22.602 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:19:22 smithi183 ceph-mon[29736]: 2.9 scrub ok 2024-09-10T19:19:22.602 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:19:22 smithi183 ceph-mon[29736]: 4.19 scrub starts 2024-09-10T19:19:22.602 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:19:22 smithi183 ceph-mon[29736]: 4.19 scrub ok 2024-09-10T19:19:22.602 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:19:22 smithi183 ceph-mon[29736]: pgmap v408: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 1.2 GiB data, 6.8 GiB used, 530 GiB / 536 GiB avail; 853 B/s rd, 0 B/s wr, 1.19k op/s; 1256/58068 objects misplaced (2.163%) 2024-09-10T19:19:22.602 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:19:22 smithi183 ceph-mon[41562]: 2.9 scrub starts 2024-09-10T19:19:22.602 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:19:22 smithi183 ceph-mon[41562]: 2.9 scrub ok 2024-09-10T19:19:22.603 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:19:22 smithi183 ceph-mon[41562]: 4.19 scrub starts 2024-09-10T19:19:22.603 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:19:22 smithi183 ceph-mon[41562]: 4.19 scrub ok 2024-09-10T19:19:22.603 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:19:22 smithi183 ceph-mon[41562]: pgmap v408: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 1.2 GiB data, 6.8 GiB used, 530 GiB / 536 GiB avail; 853 B/s rd, 0 B/s wr, 1.19k op/s; 1256/58068 objects misplaced (2.163%) 2024-09-10T19:19:22.614 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:19:22.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:22.651 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:19:22.651 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:19:22.715 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:19:22 smithi191 ceph-mon[32884]: 2.9 scrub starts 2024-09-10T19:19:22.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:19:22 smithi191 ceph-mon[32884]: 2.9 scrub ok 2024-09-10T19:19:22.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:19:22 smithi191 ceph-mon[32884]: 4.19 scrub starts 2024-09-10T19:19:22.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:19:22 smithi191 ceph-mon[32884]: 4.19 scrub ok 2024-09-10T19:19:22.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:19:22 smithi191 ceph-mon[32884]: pgmap v408: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 1.2 GiB data, 6.8 GiB used, 530 GiB / 536 GiB avail; 853 B/s rd, 0 B/s wr, 1.19k op/s; 1256/58068 objects misplaced (2.163%) 2024-09-10T19:19:22.752 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:19:22.796 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:19:22.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:22.797 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:19:22.898 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:19:22.932 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:19:22.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:22.933 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:19:23.034 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:19:23.060 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:19:23.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:23.061 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:19:23.162 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:19:23.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:23.190 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:19:23.190 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:19:23.291 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:19:23.331 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:19:23.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:23.332 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:19:23.432 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:19:23.467 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:19:23.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:23.468 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:19:23.569 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:19:23.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:23.616 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:19:23.616 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:19:23.681 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:19:23 smithi183 ceph-mon[29736]: purged_snaps scrub starts 2024-09-10T19:19:23.681 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:19:23 smithi183 ceph-mon[29736]: purged_snaps scrub ok 2024-09-10T19:19:23.681 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:19:23 smithi183 ceph-mon[41562]: purged_snaps scrub starts 2024-09-10T19:19:23.682 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:19:23 smithi183 ceph-mon[41562]: purged_snaps scrub ok 2024-09-10T19:19:23.715 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:19:23 smithi191 ceph-mon[32884]: purged_snaps scrub starts 2024-09-10T19:19:23.715 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:19:23 smithi191 ceph-mon[32884]: purged_snaps scrub ok 2024-09-10T19:19:23.717 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:19:23.755 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:19:23.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:23.756 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:19:23.856 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:19:23.882 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:19:23.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:23.883 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:19:23.983 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:19:24.004 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:19:24.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:24.005 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:19:24.106 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:19:24.133 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:19:24.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:24.134 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:19:24.193 DEBUG:teuthology.orchestra.run.smithi191:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph daemon osd.4 dump_blocked_ops 2024-09-10T19:19:24.235 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:19:24.262 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:19:24.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:24.263 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:19:24.294 INFO:teuthology.orchestra.run.smithi183.stdout:osd.0: {} 2024-09-10T19:19:24.294 INFO:teuthology.orchestra.run.smithi183.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-10T19:19:24.311 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph osd set noscrub 2024-09-10T19:19:24.317 INFO:teuthology.orchestra.run.smithi183.stdout:osd.1: {} 2024-09-10T19:19:24.317 INFO:teuthology.orchestra.run.smithi183.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-10T19:19:24.340 INFO:teuthology.orchestra.run.smithi183.stdout:osd.2: {} 2024-09-10T19:19:24.340 INFO:teuthology.orchestra.run.smithi183.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-10T19:19:24.364 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:19:24.364 INFO:teuthology.orchestra.run.smithi183.stdout:osd.3: {} 2024-09-10T19:19:24.365 INFO:teuthology.orchestra.run.smithi183.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-10T19:19:24.394 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:19:24.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:24.395 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:19:24.399 INFO:teuthology.orchestra.run.smithi183.stdout:osd.4: {} 2024-09-10T19:19:24.399 INFO:teuthology.orchestra.run.smithi183.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-10T19:19:24.437 INFO:teuthology.orchestra.run.smithi183.stdout:osd.5: {} 2024-09-10T19:19:24.437 INFO:teuthology.orchestra.run.smithi183.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-10T19:19:24.463 INFO:teuthology.orchestra.run.smithi183.stdout:osd.6: {} 2024-09-10T19:19:24.463 INFO:teuthology.orchestra.run.smithi183.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-10T19:19:24.465 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:19:24 smithi191 ceph-mon[32884]: 6.12 scrub starts 2024-09-10T19:19:24.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:19:24 smithi191 ceph-mon[32884]: 6.12 scrub ok 2024-09-10T19:19:24.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:19:24 smithi191 ceph-mon[32884]: osdmap e220: 8 total, 8 up, 6 in 2024-09-10T19:19:24.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:19:24 smithi191 ceph-mon[32884]: 4.15 scrub starts 2024-09-10T19:19:24.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:19:24 smithi191 ceph-mon[32884]: 4.15 scrub ok 2024-09-10T19:19:24.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:19:24 smithi191 ceph-mon[32884]: pgmap v410: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 1.2 GiB data, 6.8 GiB used, 530 GiB / 536 GiB avail; 944 B/s rd, 0 B/s wr, 1.32k op/s; 1256/58068 objects misplaced (2.163%) 2024-09-10T19:19:24.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:19:24 smithi191 ceph-mon[32884]: from='client.? 172.21.15.183:0/1726627280' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T19:19:24.486 INFO:teuthology.orchestra.run.smithi183.stdout:osd.7: {} 2024-09-10T19:19:24.486 INFO:teuthology.orchestra.run.smithi183.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-10T19:19:24.496 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:19:24.530 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:19:24.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:24.532 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:19:24.622 INFO:teuthology.orchestra.run.smithi191.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.b/config 2024-09-10T19:19:24.633 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:19:24.659 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:19:24.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:24.660 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:19:24.681 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:19:24 smithi183 ceph-mon[29736]: 6.12 scrub starts 2024-09-10T19:19:24.681 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:19:24 smithi183 ceph-mon[29736]: 6.12 scrub ok 2024-09-10T19:19:24.681 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:19:24 smithi183 ceph-mon[29736]: osdmap e220: 8 total, 8 up, 6 in 2024-09-10T19:19:24.682 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:19:24 smithi183 ceph-mon[29736]: 4.15 scrub starts 2024-09-10T19:19:24.682 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:19:24 smithi183 ceph-mon[29736]: 4.15 scrub ok 2024-09-10T19:19:24.682 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:19:24 smithi183 ceph-mon[29736]: pgmap v410: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 1.2 GiB data, 6.8 GiB used, 530 GiB / 536 GiB avail; 944 B/s rd, 0 B/s wr, 1.32k op/s; 1256/58068 objects misplaced (2.163%) 2024-09-10T19:19:24.682 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:19:24 smithi183 ceph-mon[29736]: from='client.? 172.21.15.183:0/1726627280' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T19:19:24.682 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:19:24 smithi183 ceph-mon[41562]: 6.12 scrub starts 2024-09-10T19:19:24.682 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:19:24 smithi183 ceph-mon[41562]: 6.12 scrub ok 2024-09-10T19:19:24.682 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:19:24 smithi183 ceph-mon[41562]: osdmap e220: 8 total, 8 up, 6 in 2024-09-10T19:19:24.682 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:19:24 smithi183 ceph-mon[41562]: 4.15 scrub starts 2024-09-10T19:19:24.683 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:19:24 smithi183 ceph-mon[41562]: 4.15 scrub ok 2024-09-10T19:19:24.683 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:19:24 smithi183 ceph-mon[41562]: pgmap v410: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 1.2 GiB data, 6.8 GiB used, 530 GiB / 536 GiB avail; 944 B/s rd, 0 B/s wr, 1.32k op/s; 1256/58068 objects misplaced (2.163%) 2024-09-10T19:19:24.683 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:19:24 smithi183 ceph-mon[41562]: from='client.? 172.21.15.183:0/1726627280' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T19:19:24.761 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:19:24.800 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:19:24.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:24.804 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:19:24.905 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:19:24.946 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:19:24.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:24.947 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:19:25.048 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:19:25.076 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:19:25.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:25.077 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:19:25.177 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:19:25.218 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:19:25.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:25.219 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:19:25.279 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:19:25.320 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:19:25.345 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:19:25.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:25.346 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:19:25.447 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:19:25.484 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:19:25.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:25.486 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:19:25.502 INFO:tasks.thrashosds.thrasher:in_osds: [0, 3, 4, 5, 6, 7] out_osds: [1, 2] dead_osds: [] live_osds: [0, 2, 3, 4, 5, 6, 7, 1] 2024-09-10T19:19:25.502 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-10T19:19:25.502 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap_items 2024-09-10T19:19:25.503 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph osd dump -f json-pretty 2024-09-10T19:19:25.587 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:19:25.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:25.619 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:19:25.619 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:19:25.720 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:19:25.758 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:19:25.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:25.759 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:19:25.860 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:19:25.887 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:19:25.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:25.888 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:19:25.966 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:19:25 smithi191 ceph-mon[32884]: 4.18 scrub starts 2024-09-10T19:19:25.966 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:19:25 smithi191 ceph-mon[32884]: 4.18 scrub ok 2024-09-10T19:19:25.966 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:19:25 smithi191 ceph-mon[32884]: 4.8 deep-scrub starts 2024-09-10T19:19:25.966 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:19:25 smithi191 ceph-mon[32884]: 4.8 deep-scrub ok 2024-09-10T19:19:25.966 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:19:25 smithi191 ceph-mon[32884]: osdmap e221: 8 total, 8 up, 6 in 2024-09-10T19:19:25.988 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:19:26.024 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:19:26.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:26.025 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:19:26.125 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:19:26.150 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:19:26.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:26.151 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:19:26.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:19:25 smithi183 ceph-mon[29736]: 4.18 scrub starts 2024-09-10T19:19:26.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:19:25 smithi183 ceph-mon[29736]: 4.18 scrub ok 2024-09-10T19:19:26.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:19:25 smithi183 ceph-mon[29736]: 4.8 deep-scrub starts 2024-09-10T19:19:26.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:19:25 smithi183 ceph-mon[29736]: 4.8 deep-scrub ok 2024-09-10T19:19:26.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:19:25 smithi183 ceph-mon[29736]: osdmap e221: 8 total, 8 up, 6 in 2024-09-10T19:19:26.181 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:19:25 smithi183 ceph-mon[41562]: 4.18 scrub starts 2024-09-10T19:19:26.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:19:25 smithi183 ceph-mon[41562]: 4.18 scrub ok 2024-09-10T19:19:26.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:19:25 smithi183 ceph-mon[41562]: 4.8 deep-scrub starts 2024-09-10T19:19:26.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:19:25 smithi183 ceph-mon[41562]: 4.8 deep-scrub ok 2024-09-10T19:19:26.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:19:25 smithi183 ceph-mon[41562]: osdmap e221: 8 total, 8 up, 6 in 2024-09-10T19:19:26.252 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:19:26.275 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:19:26.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:26.276 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:19:26.377 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:19:26.405 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:19:26.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:26.407 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:19:26.507 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:19:26.533 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:19:26.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:26.534 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:19:26.635 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:19:26.656 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:19:26.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:26.657 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:19:26.710 INFO:teuthology.orchestra.run.smithi191.stdout:ERROR: (22) Invalid argument 2024-09-10T19:19:26.710 INFO:teuthology.orchestra.run.smithi191.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-10T19:19:26.758 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:19:26.794 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:19:26.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:26.795 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:19:26.896 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:19:26.920 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:19:26.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:26.921 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:19:26.931 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:19:26 smithi183 ceph-mon[29736]: 4.17 scrub starts 2024-09-10T19:19:26.931 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:19:26 smithi183 ceph-mon[29736]: 4.17 scrub ok 2024-09-10T19:19:26.931 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:19:26 smithi183 ceph-mon[29736]: 4.11 scrub starts 2024-09-10T19:19:26.931 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:19:26 smithi183 ceph-mon[29736]: 4.11 scrub ok 2024-09-10T19:19:26.931 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:19:26 smithi183 ceph-mon[29736]: pgmap v412: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 785 MiB data, 5.5 GiB used, 531 GiB / 536 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 2.29k op/s; 836/38235 objects misplaced (2.186%) 2024-09-10T19:19:26.931 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:19:26 smithi183 ceph-mon[41562]: 4.17 scrub starts 2024-09-10T19:19:26.932 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:19:26 smithi183 ceph-mon[41562]: 4.17 scrub ok 2024-09-10T19:19:26.932 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:19:26 smithi183 ceph-mon[41562]: 4.11 scrub starts 2024-09-10T19:19:26.932 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:19:26 smithi183 ceph-mon[41562]: 4.11 scrub ok 2024-09-10T19:19:26.932 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:19:26 smithi183 ceph-mon[41562]: pgmap v412: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 785 MiB data, 5.5 GiB used, 531 GiB / 536 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 2.29k op/s; 836/38235 objects misplaced (2.186%) 2024-09-10T19:19:26.966 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:19:26 smithi191 ceph-mon[32884]: 4.17 scrub starts 2024-09-10T19:19:26.966 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:19:26 smithi191 ceph-mon[32884]: 4.17 scrub ok 2024-09-10T19:19:26.966 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:19:26 smithi191 ceph-mon[32884]: 4.11 scrub starts 2024-09-10T19:19:26.967 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:19:26 smithi191 ceph-mon[32884]: 4.11 scrub ok 2024-09-10T19:19:26.967 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:19:26 smithi191 ceph-mon[32884]: pgmap v412: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 785 MiB data, 5.5 GiB used, 531 GiB / 536 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 2.29k op/s; 836/38235 objects misplaced (2.186%) 2024-09-10T19:19:27.022 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:19:27.049 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:19:27.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:27.050 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:19:27.151 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:19:27.177 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:19:27.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:27.178 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:19:27.266 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-10T19:19:27.279 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:19:27.306 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:19:27.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:27.307 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:19:27.407 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:19:27.453 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:19:27.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:27.454 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:19:27.544 DEBUG:teuthology.orchestra.run.smithi191:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph daemon osd.4 dump_historic_ops 2024-09-10T19:19:27.555 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:19:27.632 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:19:27.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:27.634 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:19:27.716 INFO:journalctl@ceph.mgr.x.smithi191.stdout:Sep 10 19:19:27 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-x[34967]: ::ffff:172.21.15.191 - - [10/Sep/2024:19:19:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T19:19:27.735 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:19:27.760 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:19:27.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:27.761 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:19:27.862 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:19:27.887 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:19:27.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:27.888 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:19:27.964 INFO:teuthology.orchestra.run.smithi191.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.b/config 2024-09-10T19:19:27.989 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:19:28.014 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:19:28.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:28.015 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:19:28.110 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:19:27 smithi191 ceph-mon[32884]: 4.2 scrub starts 2024-09-10T19:19:28.110 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:19:27 smithi191 ceph-mon[32884]: 4.2 scrub ok 2024-09-10T19:19:28.110 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:19:27 smithi191 ceph-mon[32884]: 4.12 scrub starts 2024-09-10T19:19:28.110 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:19:27 smithi191 ceph-mon[32884]: 4.12 scrub ok 2024-09-10T19:19:28.116 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:19:28.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:28.158 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:19:28.158 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:19:28.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:19:27 smithi183 ceph-mon[29736]: 4.2 scrub starts 2024-09-10T19:19:28.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:19:27 smithi183 ceph-mon[29736]: 4.2 scrub ok 2024-09-10T19:19:28.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:19:27 smithi183 ceph-mon[29736]: 4.12 scrub starts 2024-09-10T19:19:28.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:19:27 smithi183 ceph-mon[29736]: 4.12 scrub ok 2024-09-10T19:19:28.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:19:27 smithi183 ceph-mon[41562]: 4.2 scrub starts 2024-09-10T19:19:28.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:19:27 smithi183 ceph-mon[41562]: 4.2 scrub ok 2024-09-10T19:19:28.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:19:27 smithi183 ceph-mon[41562]: 4.12 scrub starts 2024-09-10T19:19:28.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:19:27 smithi183 ceph-mon[41562]: 4.12 scrub ok 2024-09-10T19:19:28.258 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:19:28.282 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:19:28.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:28.283 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:19:28.383 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:19:28.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:28.421 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:19:28.421 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:19:28.522 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:19:28.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:28.560 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:19:28.560 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:19:28.661 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:19:28.687 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:19:28.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:28.688 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:19:28.789 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:19:28.822 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:19:28.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:28.823 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:19:28.923 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:19:28.947 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:19:28.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:28.948 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:19:29.049 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:19:29.084 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:19:29.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:29.085 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:19:29.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:19:28 smithi183 ceph-mon[29736]: 6.18 scrub starts 2024-09-10T19:19:29.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:19:28 smithi183 ceph-mon[29736]: 6.18 scrub ok 2024-09-10T19:19:29.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:19:28 smithi183 ceph-mon[29736]: pgmap v413: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 785 MiB data, 5.5 GiB used, 531 GiB / 536 GiB avail; 691 B/s rd, 0 B/s wr, 893 op/s; 836/38235 objects misplaced (2.186%) 2024-09-10T19:19:29.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:19:28 smithi183 ceph-mon[29736]: from='mgr.14556 ' entity='mgr.y' 2024-09-10T19:19:29.182 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:19:28 smithi183 ceph-mon[29736]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:19:29.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:19:28 smithi183 ceph-mon[41562]: 6.18 scrub starts 2024-09-10T19:19:29.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:19:28 smithi183 ceph-mon[41562]: 6.18 scrub ok 2024-09-10T19:19:29.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:19:28 smithi183 ceph-mon[41562]: pgmap v413: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 785 MiB data, 5.5 GiB used, 531 GiB / 536 GiB avail; 691 B/s rd, 0 B/s wr, 893 op/s; 836/38235 objects misplaced (2.186%) 2024-09-10T19:19:29.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:19:28 smithi183 ceph-mon[41562]: from='mgr.14556 ' entity='mgr.y' 2024-09-10T19:19:29.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:19:28 smithi183 ceph-mon[41562]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:19:29.186 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:19:29.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:19:28 smithi191 ceph-mon[32884]: 6.18 scrub starts 2024-09-10T19:19:29.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:19:28 smithi191 ceph-mon[32884]: 6.18 scrub ok 2024-09-10T19:19:29.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:19:28 smithi191 ceph-mon[32884]: pgmap v413: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 785 MiB data, 5.5 GiB used, 531 GiB / 536 GiB avail; 691 B/s rd, 0 B/s wr, 893 op/s; 836/38235 objects misplaced (2.186%) 2024-09-10T19:19:29.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:19:28 smithi191 ceph-mon[32884]: from='mgr.14556 ' entity='mgr.y' 2024-09-10T19:19:29.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:19:28 smithi191 ceph-mon[32884]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:19:29.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:29.230 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:19:29.230 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:19:29.331 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:19:29.355 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:19:29.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:29.356 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:19:29.456 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:19:29.502 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:19:29.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:29.503 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:19:29.603 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:19:29.639 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:19:29.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:29.639 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:19:29.715 INFO:journalctl@ceph.iscsi.iscsi.a.smithi191.stdout:Sep 10 19:19:29 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-iscsi-iscsi-a[61816]: debug there is no tcmu-runner data available 2024-09-10T19:19:29.741 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:19:29.779 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:19:29.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:29.780 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:19:29.881 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:19:29.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:29.903 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:19:29.903 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:19:29.955 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:19:29 smithi183 ceph-mon[29736]: 6.c scrub starts 2024-09-10T19:19:29.955 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:19:29 smithi183 ceph-mon[29736]: 6.c scrub ok 2024-09-10T19:19:29.955 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:19:29 smithi183 ceph-mon[29736]: 5.18 deep-scrub starts 2024-09-10T19:19:29.955 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:19:29 smithi183 ceph-mon[29736]: 5.18 deep-scrub ok 2024-09-10T19:19:29.955 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:19:29 smithi183 ceph-mon[29736]: osdmap e222: 8 total, 8 up, 6 in 2024-09-10T19:19:29.955 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:19:29 smithi183 ceph-mon[41562]: 6.c scrub starts 2024-09-10T19:19:29.956 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:19:29 smithi183 ceph-mon[41562]: 6.c scrub ok 2024-09-10T19:19:29.956 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:19:29 smithi183 ceph-mon[41562]: 5.18 deep-scrub starts 2024-09-10T19:19:29.956 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:19:29 smithi183 ceph-mon[41562]: 5.18 deep-scrub ok 2024-09-10T19:19:29.956 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:19:29 smithi183 ceph-mon[41562]: osdmap e222: 8 total, 8 up, 6 in 2024-09-10T19:19:30.004 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:19:30.026 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:19:30.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:30.027 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:19:30.072 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:19:29 smithi191 ceph-mon[32884]: 6.c scrub starts 2024-09-10T19:19:30.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:19:29 smithi191 ceph-mon[32884]: 6.c scrub ok 2024-09-10T19:19:30.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:19:29 smithi191 ceph-mon[32884]: 5.18 deep-scrub starts 2024-09-10T19:19:30.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:19:29 smithi191 ceph-mon[32884]: 5.18 deep-scrub ok 2024-09-10T19:19:30.073 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:19:29 smithi191 ceph-mon[32884]: osdmap e222: 8 total, 8 up, 6 in 2024-09-10T19:19:30.127 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:19:30.155 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:19:30.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:30.156 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:19:30.257 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:19:30.278 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:19:30.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:30.279 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:19:30.380 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:19:30.403 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:19:30.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:30.404 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:19:30.437 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:19:30.462 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:19:30.504 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:19:30.533 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:19:30.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:30.534 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:19:30.635 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:19:30.668 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:19:30.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:30.669 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:19:30.769 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:19:30.798 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:19:30.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:30.799 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:19:30.823 INFO:teuthology.orchestra.run.smithi183.stderr:noscrub is set 2024-09-10T19:19:30.900 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:19:30.925 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:19:30.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:30.926 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:19:31.027 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:19:31.057 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:19:31.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:31.058 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:19:31.106 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:19:30 smithi183 ceph-mon[41562]: from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:19:31.106 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:19:30 smithi183 ceph-mon[41562]: 5.f scrub starts 2024-09-10T19:19:31.106 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:19:30 smithi183 ceph-mon[41562]: 5.f scrub ok 2024-09-10T19:19:31.106 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:19:30 smithi183 ceph-mon[41562]: 4.6 deep-scrub starts 2024-09-10T19:19:31.106 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:19:30 smithi183 ceph-mon[41562]: 4.6 deep-scrub ok 2024-09-10T19:19:31.107 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:19:30 smithi183 ceph-mon[41562]: pgmap v414: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 785 MiB data, 5.5 GiB used, 531 GiB / 536 GiB avail; 639 B/s rd, 0 B/s wr, 826 op/s; 836/38235 objects misplaced (2.186%) 2024-09-10T19:19:31.107 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:19:30 smithi183 ceph-mon[41562]: from='client.? 172.21.15.183:0/2722725179' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-10T19:19:31.107 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:19:30 smithi183 ceph-mon[29736]: from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:19:31.107 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:19:30 smithi183 ceph-mon[29736]: 5.f scrub starts 2024-09-10T19:19:31.107 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:19:30 smithi183 ceph-mon[29736]: 5.f scrub ok 2024-09-10T19:19:31.107 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:19:30 smithi183 ceph-mon[29736]: 4.6 deep-scrub starts 2024-09-10T19:19:31.107 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:19:30 smithi183 ceph-mon[29736]: 4.6 deep-scrub ok 2024-09-10T19:19:31.107 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:19:30 smithi183 ceph-mon[29736]: pgmap v414: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 785 MiB data, 5.5 GiB used, 531 GiB / 536 GiB avail; 639 B/s rd, 0 B/s wr, 826 op/s; 836/38235 objects misplaced (2.186%) 2024-09-10T19:19:31.107 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:19:30 smithi183 ceph-mon[29736]: from='client.? 172.21.15.183:0/2722725179' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-10T19:19:31.159 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:19:31.184 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:19:31.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:31.185 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:19:31.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:19:30 smithi191 ceph-mon[32884]: from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:19:31.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:19:30 smithi191 ceph-mon[32884]: 5.f scrub starts 2024-09-10T19:19:31.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:19:30 smithi191 ceph-mon[32884]: 5.f scrub ok 2024-09-10T19:19:31.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:19:30 smithi191 ceph-mon[32884]: 4.6 deep-scrub starts 2024-09-10T19:19:31.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:19:30 smithi191 ceph-mon[32884]: 4.6 deep-scrub ok 2024-09-10T19:19:31.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:19:30 smithi191 ceph-mon[32884]: pgmap v414: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 785 MiB data, 5.5 GiB used, 531 GiB / 536 GiB avail; 639 B/s rd, 0 B/s wr, 826 op/s; 836/38235 objects misplaced (2.186%) 2024-09-10T19:19:31.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:19:30 smithi191 ceph-mon[32884]: from='client.? 172.21.15.183:0/2722725179' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-10T19:19:31.285 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:19:31.306 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:19:31.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:31.307 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:19:31.376 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Sep 10 19:19:31 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-y[29989]: ::ffff:172.21.15.191 - - [10/Sep/2024:19:19:31] "GET /metrics HTTP/1.1" 200 224280 "" "Prometheus/2.43.0" 2024-09-10T19:19:31.408 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:19:31.432 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:19:31.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:31.433 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:19:31.534 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:19:31.560 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:19:31.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:31.560 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:19:31.661 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:19:31.688 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:19:31.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:31.689 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:19:31.790 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:19:31.818 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:19:31.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:31.819 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:19:31.920 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:19:31.960 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:19:31.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:31.961 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:19:31.965 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:19:31 smithi191 ceph-mon[32884]: 4.10 deep-scrub starts 2024-09-10T19:19:31.965 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:19:31 smithi191 ceph-mon[32884]: 4.10 deep-scrub ok 2024-09-10T19:19:31.966 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:19:31 smithi191 ceph-mon[32884]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T19:19:31.966 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:19:31 smithi191 ceph-mon[32884]: from='client.? 172.21.15.183:0/2722725179' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-10T19:19:31.966 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:19:31 smithi191 ceph-mon[32884]: osdmap e223: 8 total, 8 up, 6 in 2024-09-10T19:19:32.062 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:19:32.100 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:19:31 smithi183 ceph-mon[29736]: 4.10 deep-scrub starts 2024-09-10T19:19:32.100 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:19:31 smithi183 ceph-mon[29736]: 4.10 deep-scrub ok 2024-09-10T19:19:32.101 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:19:31 smithi183 ceph-mon[29736]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T19:19:32.101 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:19:31 smithi183 ceph-mon[29736]: from='client.? 172.21.15.183:0/2722725179' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-10T19:19:32.101 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:19:31 smithi183 ceph-mon[29736]: osdmap e223: 8 total, 8 up, 6 in 2024-09-10T19:19:32.101 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:19:31 smithi183 ceph-mon[41562]: 4.10 deep-scrub starts 2024-09-10T19:19:32.101 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:19:31 smithi183 ceph-mon[41562]: 4.10 deep-scrub ok 2024-09-10T19:19:32.101 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:19:31 smithi183 ceph-mon[41562]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T19:19:32.101 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:19:31 smithi183 ceph-mon[41562]: from='client.? 172.21.15.183:0/2722725179' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-10T19:19:32.101 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:19:31 smithi183 ceph-mon[41562]: osdmap e223: 8 total, 8 up, 6 in 2024-09-10T19:19:32.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:32.102 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:19:32.102 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:19:32.203 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:19:32.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:32.290 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:19:32.290 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:19:32.362 INFO:teuthology.orchestra.run.smithi191.stdout:ERROR: (22) Invalid argument 2024-09-10T19:19:32.362 INFO:teuthology.orchestra.run.smithi191.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-10T19:19:32.392 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:19:32.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:32.435 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:19:32.435 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:19:32.536 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:19:32.573 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:19:32.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:32.574 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:19:32.675 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:19:32.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:32.718 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:19:32.718 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:19:32.819 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:19:32.858 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:19:32.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:32.859 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:19:32.960 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:19:32.985 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:19:32.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:32.986 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:19:33.087 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:19:33.115 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:19:33.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:33.117 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:19:33.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:19:32 smithi183 ceph-mon[29736]: pgmap v417: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 399 MiB data, 4.2 GiB used, 532 GiB / 536 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 1.83k op/s; 400/19695 objects misplaced (2.031%) 2024-09-10T19:19:33.181 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:19:32 smithi183 ceph-mon[41562]: pgmap v417: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 399 MiB data, 4.2 GiB used, 532 GiB / 536 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 1.83k op/s; 400/19695 objects misplaced (2.031%) 2024-09-10T19:19:33.197 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:19:32 smithi191 ceph-mon[32884]: pgmap v417: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 399 MiB data, 4.2 GiB used, 532 GiB / 536 GiB avail; 1.4 KiB/s rd, 0 B/s wr, 1.83k op/s; 400/19695 objects misplaced (2.031%) 2024-09-10T19:19:33.218 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:19:33.237 DEBUG:teuthology.orchestra.run.smithi191:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph daemon osd.5 dump_ops_in_flight 2024-09-10T19:19:33.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:33.255 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:19:33.255 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:19:33.357 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:19:33.399 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:19:33.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:33.400 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:19:33.501 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:19:33.530 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:19:33.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:33.531 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:19:33.632 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:19:33.690 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:19:33.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:33.691 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:19:33.707 INFO:teuthology.orchestra.run.smithi191.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.b/config 2024-09-10T19:19:33.793 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:19:33.832 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:19:33.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:33.833 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:19:33.934 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:19:33.972 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:19:33.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:33.973 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:19:34.017 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-09-10T19:19:34.017 INFO:teuthology.orchestra.run.smithi183.stdout:{ 2024-09-10T19:19:34.017 INFO:teuthology.orchestra.run.smithi183.stdout: "epoch": 223, 2024-09-10T19:19:34.017 INFO:teuthology.orchestra.run.smithi183.stdout: "fsid": "320f6df8-6fa7-11ef-bcea-c7b262605968", 2024-09-10T19:19:34.017 INFO:teuthology.orchestra.run.smithi183.stdout: "created": "2024-09-10T19:05:03.688531+0000", 2024-09-10T19:19:34.017 INFO:teuthology.orchestra.run.smithi183.stdout: "modified": "2024-09-10T19:19:30.813837+0000", 2024-09-10T19:19:34.017 INFO:teuthology.orchestra.run.smithi183.stdout: "last_up_change": "2024-09-10T19:15:19.923894+0000", 2024-09-10T19:19:34.018 INFO:teuthology.orchestra.run.smithi183.stdout: "last_in_change": "2024-09-10T19:16:23.815531+0000", 2024-09-10T19:19:34.018 INFO:teuthology.orchestra.run.smithi183.stdout: "flags": "noscrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-09-10T19:19:34.018 INFO:teuthology.orchestra.run.smithi183.stdout: "flags_num": 5801984, 2024-09-10T19:19:34.018 INFO:teuthology.orchestra.run.smithi183.stdout: "flags_set": [ 2024-09-10T19:19:34.018 INFO:teuthology.orchestra.run.smithi183.stdout: "noscrub", 2024-09-10T19:19:34.018 INFO:teuthology.orchestra.run.smithi183.stdout: "pglog_hardlimit", 2024-09-10T19:19:34.018 INFO:teuthology.orchestra.run.smithi183.stdout: "purged_snapdirs", 2024-09-10T19:19:34.018 INFO:teuthology.orchestra.run.smithi183.stdout: "recovery_deletes", 2024-09-10T19:19:34.018 INFO:teuthology.orchestra.run.smithi183.stdout: "sortbitwise" 2024-09-10T19:19:34.018 INFO:teuthology.orchestra.run.smithi183.stdout: ], 2024-09-10T19:19:34.018 INFO:teuthology.orchestra.run.smithi183.stdout: "crush_version": 19, 2024-09-10T19:19:34.018 INFO:teuthology.orchestra.run.smithi183.stdout: "full_ratio": 0.94999998807907104, 2024-09-10T19:19:34.018 INFO:teuthology.orchestra.run.smithi183.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-09-10T19:19:34.018 INFO:teuthology.orchestra.run.smithi183.stdout: "nearfull_ratio": 0.85000002384185791, 2024-09-10T19:19:34.019 INFO:teuthology.orchestra.run.smithi183.stdout: "cluster_snapshot": "", 2024-09-10T19:19:34.019 INFO:teuthology.orchestra.run.smithi183.stdout: "pool_max": 8, 2024-09-10T19:19:34.019 INFO:teuthology.orchestra.run.smithi183.stdout: "max_osd": 8, 2024-09-10T19:19:34.019 INFO:teuthology.orchestra.run.smithi183.stdout: "require_min_compat_client": "luminous", 2024-09-10T19:19:34.019 INFO:teuthology.orchestra.run.smithi183.stdout: "min_compat_client": "luminous", 2024-09-10T19:19:34.019 INFO:teuthology.orchestra.run.smithi183.stdout: "require_osd_release": "quincy", 2024-09-10T19:19:34.019 INFO:teuthology.orchestra.run.smithi183.stdout: "pools": [ 2024-09-10T19:19:34.019 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:19:34.019 INFO:teuthology.orchestra.run.smithi183.stdout: "pool": 1, 2024-09-10T19:19:34.019 INFO:teuthology.orchestra.run.smithi183.stdout: "pool_name": ".mgr", 2024-09-10T19:19:34.019 INFO:teuthology.orchestra.run.smithi183.stdout: "create_time": "2024-09-10T19:08:00.456786+0000", 2024-09-10T19:19:34.019 INFO:teuthology.orchestra.run.smithi183.stdout: "flags": 1, 2024-09-10T19:19:34.019 INFO:teuthology.orchestra.run.smithi183.stdout: "flags_names": "hashpspool", 2024-09-10T19:19:34.020 INFO:teuthology.orchestra.run.smithi183.stdout: "type": 1, 2024-09-10T19:19:34.020 INFO:teuthology.orchestra.run.smithi183.stdout: "size": 3, 2024-09-10T19:19:34.020 INFO:teuthology.orchestra.run.smithi183.stdout: "min_size": 2, 2024-09-10T19:19:34.020 INFO:teuthology.orchestra.run.smithi183.stdout: "crush_rule": 0, 2024-09-10T19:19:34.020 INFO:teuthology.orchestra.run.smithi183.stdout: "peering_crush_bucket_count": 0, 2024-09-10T19:19:34.020 INFO:teuthology.orchestra.run.smithi183.stdout: "peering_crush_bucket_target": 0, 2024-09-10T19:19:34.020 INFO:teuthology.orchestra.run.smithi183.stdout: "peering_crush_bucket_barrier": 0, 2024-09-10T19:19:34.020 INFO:teuthology.orchestra.run.smithi183.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-10T19:19:34.020 INFO:teuthology.orchestra.run.smithi183.stdout: "object_hash": 2, 2024-09-10T19:19:34.020 INFO:teuthology.orchestra.run.smithi183.stdout: "pg_autoscale_mode": "off", 2024-09-10T19:19:34.020 INFO:teuthology.orchestra.run.smithi183.stdout: "pg_num": 1, 2024-09-10T19:19:34.020 INFO:teuthology.orchestra.run.smithi183.stdout: "pg_placement_num": 1, 2024-09-10T19:19:34.020 INFO:teuthology.orchestra.run.smithi183.stdout: "pg_placement_num_target": 1, 2024-09-10T19:19:34.020 INFO:teuthology.orchestra.run.smithi183.stdout: "pg_num_target": 1, 2024-09-10T19:19:34.021 INFO:teuthology.orchestra.run.smithi183.stdout: "pg_num_pending": 1, 2024-09-10T19:19:34.021 INFO:teuthology.orchestra.run.smithi183.stdout: "last_pg_merge_meta": { 2024-09-10T19:19:34.021 INFO:teuthology.orchestra.run.smithi183.stdout: "source_pgid": "0.0", 2024-09-10T19:19:34.021 INFO:teuthology.orchestra.run.smithi183.stdout: "ready_epoch": 0, 2024-09-10T19:19:34.021 INFO:teuthology.orchestra.run.smithi183.stdout: "last_epoch_started": 0, 2024-09-10T19:19:34.021 INFO:teuthology.orchestra.run.smithi183.stdout: "last_epoch_clean": 0, 2024-09-10T19:19:34.021 INFO:teuthology.orchestra.run.smithi183.stdout: "source_version": "0'0", 2024-09-10T19:19:34.021 INFO:teuthology.orchestra.run.smithi183.stdout: "target_version": "0'0" 2024-09-10T19:19:34.021 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:19:34.021 INFO:teuthology.orchestra.run.smithi183.stdout: "last_change": "202", 2024-09-10T19:19:34.021 INFO:teuthology.orchestra.run.smithi183.stdout: "last_force_op_resend": "0", 2024-09-10T19:19:34.021 INFO:teuthology.orchestra.run.smithi183.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-10T19:19:34.021 INFO:teuthology.orchestra.run.smithi183.stdout: "last_force_op_resend_preluminous": "0", 2024-09-10T19:19:34.021 INFO:teuthology.orchestra.run.smithi183.stdout: "auid": 0, 2024-09-10T19:19:34.022 INFO:teuthology.orchestra.run.smithi183.stdout: "snap_mode": "selfmanaged", 2024-09-10T19:19:34.024 INFO:teuthology.orchestra.run.smithi183.stdout: "snap_seq": 0, 2024-09-10T19:19:34.024 INFO:teuthology.orchestra.run.smithi183.stdout: "snap_epoch": 0, 2024-09-10T19:19:34.024 INFO:teuthology.orchestra.run.smithi183.stdout: "pool_snaps": [], 2024-09-10T19:19:34.024 INFO:teuthology.orchestra.run.smithi183.stdout: "removed_snaps": "[]", 2024-09-10T19:19:34.024 INFO:teuthology.orchestra.run.smithi183.stdout: "quota_max_bytes": 0, 2024-09-10T19:19:34.024 INFO:teuthology.orchestra.run.smithi183.stdout: "quota_max_objects": 0, 2024-09-10T19:19:34.024 INFO:teuthology.orchestra.run.smithi183.stdout: "tiers": [], 2024-09-10T19:19:34.024 INFO:teuthology.orchestra.run.smithi183.stdout: "tier_of": -1, 2024-09-10T19:19:34.024 INFO:teuthology.orchestra.run.smithi183.stdout: "read_tier": -1, 2024-09-10T19:19:34.024 INFO:teuthology.orchestra.run.smithi183.stdout: "write_tier": -1, 2024-09-10T19:19:34.025 INFO:teuthology.orchestra.run.smithi183.stdout: "cache_mode": "none", 2024-09-10T19:19:34.025 INFO:teuthology.orchestra.run.smithi183.stdout: "target_max_bytes": 0, 2024-09-10T19:19:34.025 INFO:teuthology.orchestra.run.smithi183.stdout: "target_max_objects": 0, 2024-09-10T19:19:34.025 INFO:teuthology.orchestra.run.smithi183.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-10T19:19:34.025 INFO:teuthology.orchestra.run.smithi183.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-10T19:19:34.025 INFO:teuthology.orchestra.run.smithi183.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-10T19:19:34.025 INFO:teuthology.orchestra.run.smithi183.stdout: "cache_min_flush_age": 0, 2024-09-10T19:19:34.025 INFO:teuthology.orchestra.run.smithi183.stdout: "cache_min_evict_age": 0, 2024-09-10T19:19:34.025 INFO:teuthology.orchestra.run.smithi183.stdout: "erasure_code_profile": "", 2024-09-10T19:19:34.025 INFO:teuthology.orchestra.run.smithi183.stdout: "hit_set_params": { 2024-09-10T19:19:34.025 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "none" 2024-09-10T19:19:34.025 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:19:34.025 INFO:teuthology.orchestra.run.smithi183.stdout: "hit_set_period": 0, 2024-09-10T19:19:34.025 INFO:teuthology.orchestra.run.smithi183.stdout: "hit_set_count": 0, 2024-09-10T19:19:34.026 INFO:teuthology.orchestra.run.smithi183.stdout: "use_gmt_hitset": true, 2024-09-10T19:19:34.026 INFO:teuthology.orchestra.run.smithi183.stdout: "min_read_recency_for_promote": 0, 2024-09-10T19:19:34.026 INFO:teuthology.orchestra.run.smithi183.stdout: "min_write_recency_for_promote": 0, 2024-09-10T19:19:34.026 INFO:teuthology.orchestra.run.smithi183.stdout: "hit_set_grade_decay_rate": 0, 2024-09-10T19:19:34.026 INFO:teuthology.orchestra.run.smithi183.stdout: "hit_set_search_last_n": 0, 2024-09-10T19:19:34.026 INFO:teuthology.orchestra.run.smithi183.stdout: "grade_table": [], 2024-09-10T19:19:34.026 INFO:teuthology.orchestra.run.smithi183.stdout: "stripe_width": 0, 2024-09-10T19:19:34.026 INFO:teuthology.orchestra.run.smithi183.stdout: "expected_num_objects": 0, 2024-09-10T19:19:34.026 INFO:teuthology.orchestra.run.smithi183.stdout: "fast_read": false, 2024-09-10T19:19:34.026 INFO:teuthology.orchestra.run.smithi183.stdout: "options": { 2024-09-10T19:19:34.026 INFO:teuthology.orchestra.run.smithi183.stdout: "pg_num_max": 32, 2024-09-10T19:19:34.026 INFO:teuthology.orchestra.run.smithi183.stdout: "pg_num_min": 1 2024-09-10T19:19:34.026 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:19:34.026 INFO:teuthology.orchestra.run.smithi183.stdout: "application_metadata": { 2024-09-10T19:19:34.027 INFO:teuthology.orchestra.run.smithi183.stdout: "mgr": {} 2024-09-10T19:19:34.027 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:19:34.027 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:19:34.027 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:19:34.027 INFO:teuthology.orchestra.run.smithi183.stdout: "pool": 2, 2024-09-10T19:19:34.027 INFO:teuthology.orchestra.run.smithi183.stdout: "pool_name": ".rgw.root", 2024-09-10T19:19:34.027 INFO:teuthology.orchestra.run.smithi183.stdout: "create_time": "2024-09-10T19:10:05.993763+0000", 2024-09-10T19:19:34.027 INFO:teuthology.orchestra.run.smithi183.stdout: "flags": 1, 2024-09-10T19:19:34.027 INFO:teuthology.orchestra.run.smithi183.stdout: "flags_names": "hashpspool", 2024-09-10T19:19:34.027 INFO:teuthology.orchestra.run.smithi183.stdout: "type": 1, 2024-09-10T19:19:34.027 INFO:teuthology.orchestra.run.smithi183.stdout: "size": 3, 2024-09-10T19:19:34.027 INFO:teuthology.orchestra.run.smithi183.stdout: "min_size": 2, 2024-09-10T19:19:34.027 INFO:teuthology.orchestra.run.smithi183.stdout: "crush_rule": 0, 2024-09-10T19:19:34.028 INFO:teuthology.orchestra.run.smithi183.stdout: "peering_crush_bucket_count": 0, 2024-09-10T19:19:34.028 INFO:teuthology.orchestra.run.smithi183.stdout: "peering_crush_bucket_target": 0, 2024-09-10T19:19:34.028 INFO:teuthology.orchestra.run.smithi183.stdout: "peering_crush_bucket_barrier": 0, 2024-09-10T19:19:34.028 INFO:teuthology.orchestra.run.smithi183.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-10T19:19:34.028 INFO:teuthology.orchestra.run.smithi183.stdout: "object_hash": 2, 2024-09-10T19:19:34.028 INFO:teuthology.orchestra.run.smithi183.stdout: "pg_autoscale_mode": "off", 2024-09-10T19:19:34.028 INFO:teuthology.orchestra.run.smithi183.stdout: "pg_num": 32, 2024-09-10T19:19:34.028 INFO:teuthology.orchestra.run.smithi183.stdout: "pg_placement_num": 32, 2024-09-10T19:19:34.028 INFO:teuthology.orchestra.run.smithi183.stdout: "pg_placement_num_target": 32, 2024-09-10T19:19:34.028 INFO:teuthology.orchestra.run.smithi183.stdout: "pg_num_target": 32, 2024-09-10T19:19:34.028 INFO:teuthology.orchestra.run.smithi183.stdout: "pg_num_pending": 32, 2024-09-10T19:19:34.028 INFO:teuthology.orchestra.run.smithi183.stdout: "last_pg_merge_meta": { 2024-09-10T19:19:34.028 INFO:teuthology.orchestra.run.smithi183.stdout: "source_pgid": "0.0", 2024-09-10T19:19:34.029 INFO:teuthology.orchestra.run.smithi183.stdout: "ready_epoch": 0, 2024-09-10T19:19:34.029 INFO:teuthology.orchestra.run.smithi183.stdout: "last_epoch_started": 0, 2024-09-10T19:19:34.029 INFO:teuthology.orchestra.run.smithi183.stdout: "last_epoch_clean": 0, 2024-09-10T19:19:34.029 INFO:teuthology.orchestra.run.smithi183.stdout: "source_version": "0'0", 2024-09-10T19:19:34.029 INFO:teuthology.orchestra.run.smithi183.stdout: "target_version": "0'0" 2024-09-10T19:19:34.029 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:19:34.029 INFO:teuthology.orchestra.run.smithi183.stdout: "last_change": "60", 2024-09-10T19:19:34.029 INFO:teuthology.orchestra.run.smithi183.stdout: "last_force_op_resend": "0", 2024-09-10T19:19:34.029 INFO:teuthology.orchestra.run.smithi183.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-10T19:19:34.029 INFO:teuthology.orchestra.run.smithi183.stdout: "last_force_op_resend_preluminous": "0", 2024-09-10T19:19:34.029 INFO:teuthology.orchestra.run.smithi183.stdout: "auid": 0, 2024-09-10T19:19:34.029 INFO:teuthology.orchestra.run.smithi183.stdout: "snap_mode": "selfmanaged", 2024-09-10T19:19:34.029 INFO:teuthology.orchestra.run.smithi183.stdout: "snap_seq": 0, 2024-09-10T19:19:34.029 INFO:teuthology.orchestra.run.smithi183.stdout: "snap_epoch": 0, 2024-09-10T19:19:34.030 INFO:teuthology.orchestra.run.smithi183.stdout: "pool_snaps": [], 2024-09-10T19:19:34.030 INFO:teuthology.orchestra.run.smithi183.stdout: "removed_snaps": "[]", 2024-09-10T19:19:34.030 INFO:teuthology.orchestra.run.smithi183.stdout: "quota_max_bytes": 0, 2024-09-10T19:19:34.030 INFO:teuthology.orchestra.run.smithi183.stdout: "quota_max_objects": 0, 2024-09-10T19:19:34.030 INFO:teuthology.orchestra.run.smithi183.stdout: "tiers": [], 2024-09-10T19:19:34.030 INFO:teuthology.orchestra.run.smithi183.stdout: "tier_of": -1, 2024-09-10T19:19:34.030 INFO:teuthology.orchestra.run.smithi183.stdout: "read_tier": -1, 2024-09-10T19:19:34.030 INFO:teuthology.orchestra.run.smithi183.stdout: "write_tier": -1, 2024-09-10T19:19:34.030 INFO:teuthology.orchestra.run.smithi183.stdout: "cache_mode": "none", 2024-09-10T19:19:34.030 INFO:teuthology.orchestra.run.smithi183.stdout: "target_max_bytes": 0, 2024-09-10T19:19:34.030 INFO:teuthology.orchestra.run.smithi183.stdout: "target_max_objects": 0, 2024-09-10T19:19:34.030 INFO:teuthology.orchestra.run.smithi183.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-10T19:19:34.030 INFO:teuthology.orchestra.run.smithi183.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-10T19:19:34.030 INFO:teuthology.orchestra.run.smithi183.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-10T19:19:34.030 INFO:teuthology.orchestra.run.smithi183.stdout: "cache_min_flush_age": 0, 2024-09-10T19:19:34.031 INFO:teuthology.orchestra.run.smithi183.stdout: "cache_min_evict_age": 0, 2024-09-10T19:19:34.031 INFO:teuthology.orchestra.run.smithi183.stdout: "erasure_code_profile": "", 2024-09-10T19:19:34.031 INFO:teuthology.orchestra.run.smithi183.stdout: "hit_set_params": { 2024-09-10T19:19:34.031 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "none" 2024-09-10T19:19:34.031 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:19:34.031 INFO:teuthology.orchestra.run.smithi183.stdout: "hit_set_period": 0, 2024-09-10T19:19:34.031 INFO:teuthology.orchestra.run.smithi183.stdout: "hit_set_count": 0, 2024-09-10T19:19:34.031 INFO:teuthology.orchestra.run.smithi183.stdout: "use_gmt_hitset": true, 2024-09-10T19:19:34.031 INFO:teuthology.orchestra.run.smithi183.stdout: "min_read_recency_for_promote": 0, 2024-09-10T19:19:34.031 INFO:teuthology.orchestra.run.smithi183.stdout: "min_write_recency_for_promote": 0, 2024-09-10T19:19:34.031 INFO:teuthology.orchestra.run.smithi183.stdout: "hit_set_grade_decay_rate": 0, 2024-09-10T19:19:34.031 INFO:teuthology.orchestra.run.smithi183.stdout: "hit_set_search_last_n": 0, 2024-09-10T19:19:34.031 INFO:teuthology.orchestra.run.smithi183.stdout: "grade_table": [], 2024-09-10T19:19:34.031 INFO:teuthology.orchestra.run.smithi183.stdout: "stripe_width": 0, 2024-09-10T19:19:34.032 INFO:teuthology.orchestra.run.smithi183.stdout: "expected_num_objects": 0, 2024-09-10T19:19:34.032 INFO:teuthology.orchestra.run.smithi183.stdout: "fast_read": false, 2024-09-10T19:19:34.032 INFO:teuthology.orchestra.run.smithi183.stdout: "options": {}, 2024-09-10T19:19:34.032 INFO:teuthology.orchestra.run.smithi183.stdout: "application_metadata": { 2024-09-10T19:19:34.032 INFO:teuthology.orchestra.run.smithi183.stdout: "rgw": {} 2024-09-10T19:19:34.032 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:19:34.032 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:19:34.032 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:19:34.032 INFO:teuthology.orchestra.run.smithi183.stdout: "pool": 3, 2024-09-10T19:19:34.032 INFO:teuthology.orchestra.run.smithi183.stdout: "pool_name": "datapool", 2024-09-10T19:19:34.032 INFO:teuthology.orchestra.run.smithi183.stdout: "create_time": "2024-09-10T19:10:06.224267+0000", 2024-09-10T19:19:34.032 INFO:teuthology.orchestra.run.smithi183.stdout: "flags": 8193, 2024-09-10T19:19:34.032 INFO:teuthology.orchestra.run.smithi183.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-09-10T19:19:34.032 INFO:teuthology.orchestra.run.smithi183.stdout: "type": 1, 2024-09-10T19:19:34.033 INFO:teuthology.orchestra.run.smithi183.stdout: "size": 3, 2024-09-10T19:19:34.033 INFO:teuthology.orchestra.run.smithi183.stdout: "min_size": 2, 2024-09-10T19:19:34.033 INFO:teuthology.orchestra.run.smithi183.stdout: "crush_rule": 0, 2024-09-10T19:19:34.033 INFO:teuthology.orchestra.run.smithi183.stdout: "peering_crush_bucket_count": 0, 2024-09-10T19:19:34.033 INFO:teuthology.orchestra.run.smithi183.stdout: "peering_crush_bucket_target": 0, 2024-09-10T19:19:34.033 INFO:teuthology.orchestra.run.smithi183.stdout: "peering_crush_bucket_barrier": 0, 2024-09-10T19:19:34.033 INFO:teuthology.orchestra.run.smithi183.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-10T19:19:34.033 INFO:teuthology.orchestra.run.smithi183.stdout: "object_hash": 2, 2024-09-10T19:19:34.033 INFO:teuthology.orchestra.run.smithi183.stdout: "pg_autoscale_mode": "off", 2024-09-10T19:19:34.033 INFO:teuthology.orchestra.run.smithi183.stdout: "pg_num": 3, 2024-09-10T19:19:34.033 INFO:teuthology.orchestra.run.smithi183.stdout: "pg_placement_num": 3, 2024-09-10T19:19:34.033 INFO:teuthology.orchestra.run.smithi183.stdout: "pg_placement_num_target": 3, 2024-09-10T19:19:34.033 INFO:teuthology.orchestra.run.smithi183.stdout: "pg_num_target": 3, 2024-09-10T19:19:34.033 INFO:teuthology.orchestra.run.smithi183.stdout: "pg_num_pending": 3, 2024-09-10T19:19:34.034 INFO:teuthology.orchestra.run.smithi183.stdout: "last_pg_merge_meta": { 2024-09-10T19:19:34.034 INFO:teuthology.orchestra.run.smithi183.stdout: "source_pgid": "0.0", 2024-09-10T19:19:34.034 INFO:teuthology.orchestra.run.smithi183.stdout: "ready_epoch": 0, 2024-09-10T19:19:34.034 INFO:teuthology.orchestra.run.smithi183.stdout: "last_epoch_started": 0, 2024-09-10T19:19:34.034 INFO:teuthology.orchestra.run.smithi183.stdout: "last_epoch_clean": 0, 2024-09-10T19:19:34.034 INFO:teuthology.orchestra.run.smithi183.stdout: "source_version": "0'0", 2024-09-10T19:19:34.034 INFO:teuthology.orchestra.run.smithi183.stdout: "target_version": "0'0" 2024-09-10T19:19:34.034 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:19:34.034 INFO:teuthology.orchestra.run.smithi183.stdout: "last_change": "63", 2024-09-10T19:19:34.034 INFO:teuthology.orchestra.run.smithi183.stdout: "last_force_op_resend": "0", 2024-09-10T19:19:34.034 INFO:teuthology.orchestra.run.smithi183.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-10T19:19:34.034 INFO:teuthology.orchestra.run.smithi183.stdout: "last_force_op_resend_preluminous": "0", 2024-09-10T19:19:34.034 INFO:teuthology.orchestra.run.smithi183.stdout: "auid": 0, 2024-09-10T19:19:34.034 INFO:teuthology.orchestra.run.smithi183.stdout: "snap_mode": "selfmanaged", 2024-09-10T19:19:34.035 INFO:teuthology.orchestra.run.smithi183.stdout: "snap_seq": 3, 2024-09-10T19:19:34.035 INFO:teuthology.orchestra.run.smithi183.stdout: "snap_epoch": 63, 2024-09-10T19:19:34.035 INFO:teuthology.orchestra.run.smithi183.stdout: "pool_snaps": [], 2024-09-10T19:19:34.035 INFO:teuthology.orchestra.run.smithi183.stdout: "removed_snaps": "[]", 2024-09-10T19:19:34.035 INFO:teuthology.orchestra.run.smithi183.stdout: "quota_max_bytes": 0, 2024-09-10T19:19:34.035 INFO:teuthology.orchestra.run.smithi183.stdout: "quota_max_objects": 0, 2024-09-10T19:19:34.035 INFO:teuthology.orchestra.run.smithi183.stdout: "tiers": [], 2024-09-10T19:19:34.035 INFO:teuthology.orchestra.run.smithi183.stdout: "tier_of": -1, 2024-09-10T19:19:34.035 INFO:teuthology.orchestra.run.smithi183.stdout: "read_tier": -1, 2024-09-10T19:19:34.035 INFO:teuthology.orchestra.run.smithi183.stdout: "write_tier": -1, 2024-09-10T19:19:34.035 INFO:teuthology.orchestra.run.smithi183.stdout: "cache_mode": "none", 2024-09-10T19:19:34.035 INFO:teuthology.orchestra.run.smithi183.stdout: "target_max_bytes": 0, 2024-09-10T19:19:34.035 INFO:teuthology.orchestra.run.smithi183.stdout: "target_max_objects": 0, 2024-09-10T19:19:34.035 INFO:teuthology.orchestra.run.smithi183.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-10T19:19:34.036 INFO:teuthology.orchestra.run.smithi183.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-10T19:19:34.036 INFO:teuthology.orchestra.run.smithi183.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-10T19:19:34.036 INFO:teuthology.orchestra.run.smithi183.stdout: "cache_min_flush_age": 0, 2024-09-10T19:19:34.036 INFO:teuthology.orchestra.run.smithi183.stdout: "cache_min_evict_age": 0, 2024-09-10T19:19:34.036 INFO:teuthology.orchestra.run.smithi183.stdout: "erasure_code_profile": "", 2024-09-10T19:19:34.036 INFO:teuthology.orchestra.run.smithi183.stdout: "hit_set_params": { 2024-09-10T19:19:34.036 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "none" 2024-09-10T19:19:34.036 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:19:34.037 INFO:teuthology.orchestra.run.smithi183.stdout: "hit_set_period": 0, 2024-09-10T19:19:34.037 INFO:teuthology.orchestra.run.smithi183.stdout: "hit_set_count": 0, 2024-09-10T19:19:34.037 INFO:teuthology.orchestra.run.smithi183.stdout: "use_gmt_hitset": true, 2024-09-10T19:19:34.037 INFO:teuthology.orchestra.run.smithi183.stdout: "min_read_recency_for_promote": 0, 2024-09-10T19:19:34.037 INFO:teuthology.orchestra.run.smithi183.stdout: "min_write_recency_for_promote": 0, 2024-09-10T19:19:34.037 INFO:teuthology.orchestra.run.smithi183.stdout: "hit_set_grade_decay_rate": 0, 2024-09-10T19:19:34.037 INFO:teuthology.orchestra.run.smithi183.stdout: "hit_set_search_last_n": 0, 2024-09-10T19:19:34.038 INFO:teuthology.orchestra.run.smithi183.stdout: "grade_table": [], 2024-09-10T19:19:34.038 INFO:teuthology.orchestra.run.smithi183.stdout: "stripe_width": 0, 2024-09-10T19:19:34.038 INFO:teuthology.orchestra.run.smithi183.stdout: "expected_num_objects": 0, 2024-09-10T19:19:34.038 INFO:teuthology.orchestra.run.smithi183.stdout: "fast_read": false, 2024-09-10T19:19:34.038 INFO:teuthology.orchestra.run.smithi183.stdout: "options": {}, 2024-09-10T19:19:34.038 INFO:teuthology.orchestra.run.smithi183.stdout: "application_metadata": { 2024-09-10T19:19:34.038 INFO:teuthology.orchestra.run.smithi183.stdout: "rbd": {} 2024-09-10T19:19:34.038 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:19:34.038 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:19:34.038 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:19:34.038 INFO:teuthology.orchestra.run.smithi183.stdout: "pool": 4, 2024-09-10T19:19:34.038 INFO:teuthology.orchestra.run.smithi183.stdout: "pool_name": "default.rgw.log", 2024-09-10T19:19:34.038 INFO:teuthology.orchestra.run.smithi183.stdout: "create_time": "2024-09-10T19:10:07.958284+0000", 2024-09-10T19:19:34.038 INFO:teuthology.orchestra.run.smithi183.stdout: "flags": 1, 2024-09-10T19:19:34.039 INFO:teuthology.orchestra.run.smithi183.stdout: "flags_names": "hashpspool", 2024-09-10T19:19:34.039 INFO:teuthology.orchestra.run.smithi183.stdout: "type": 1, 2024-09-10T19:19:34.039 INFO:teuthology.orchestra.run.smithi183.stdout: "size": 3, 2024-09-10T19:19:34.039 INFO:teuthology.orchestra.run.smithi183.stdout: "min_size": 2, 2024-09-10T19:19:34.039 INFO:teuthology.orchestra.run.smithi183.stdout: "crush_rule": 0, 2024-09-10T19:19:34.039 INFO:teuthology.orchestra.run.smithi183.stdout: "peering_crush_bucket_count": 0, 2024-09-10T19:19:34.039 INFO:teuthology.orchestra.run.smithi183.stdout: "peering_crush_bucket_target": 0, 2024-09-10T19:19:34.039 INFO:teuthology.orchestra.run.smithi183.stdout: "peering_crush_bucket_barrier": 0, 2024-09-10T19:19:34.039 INFO:teuthology.orchestra.run.smithi183.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-10T19:19:34.039 INFO:teuthology.orchestra.run.smithi183.stdout: "object_hash": 2, 2024-09-10T19:19:34.039 INFO:teuthology.orchestra.run.smithi183.stdout: "pg_autoscale_mode": "off", 2024-09-10T19:19:34.039 INFO:teuthology.orchestra.run.smithi183.stdout: "pg_num": 32, 2024-09-10T19:19:34.039 INFO:teuthology.orchestra.run.smithi183.stdout: "pg_placement_num": 32, 2024-09-10T19:19:34.039 INFO:teuthology.orchestra.run.smithi183.stdout: "pg_placement_num_target": 32, 2024-09-10T19:19:34.040 INFO:teuthology.orchestra.run.smithi183.stdout: "pg_num_target": 32, 2024-09-10T19:19:34.040 INFO:teuthology.orchestra.run.smithi183.stdout: "pg_num_pending": 32, 2024-09-10T19:19:34.040 INFO:teuthology.orchestra.run.smithi183.stdout: "last_pg_merge_meta": { 2024-09-10T19:19:34.040 INFO:teuthology.orchestra.run.smithi183.stdout: "source_pgid": "0.0", 2024-09-10T19:19:34.040 INFO:teuthology.orchestra.run.smithi183.stdout: "ready_epoch": 0, 2024-09-10T19:19:34.040 INFO:teuthology.orchestra.run.smithi183.stdout: "last_epoch_started": 0, 2024-09-10T19:19:34.040 INFO:teuthology.orchestra.run.smithi183.stdout: "last_epoch_clean": 0, 2024-09-10T19:19:34.040 INFO:teuthology.orchestra.run.smithi183.stdout: "source_version": "0'0", 2024-09-10T19:19:34.040 INFO:teuthology.orchestra.run.smithi183.stdout: "target_version": "0'0" 2024-09-10T19:19:34.040 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:19:34.040 INFO:teuthology.orchestra.run.smithi183.stdout: "last_change": "62", 2024-09-10T19:19:34.040 INFO:teuthology.orchestra.run.smithi183.stdout: "last_force_op_resend": "0", 2024-09-10T19:19:34.040 INFO:teuthology.orchestra.run.smithi183.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-10T19:19:34.041 INFO:teuthology.orchestra.run.smithi183.stdout: "last_force_op_resend_preluminous": "0", 2024-09-10T19:19:34.041 INFO:teuthology.orchestra.run.smithi183.stdout: "auid": 0, 2024-09-10T19:19:34.041 INFO:teuthology.orchestra.run.smithi183.stdout: "snap_mode": "selfmanaged", 2024-09-10T19:19:34.041 INFO:teuthology.orchestra.run.smithi183.stdout: "snap_seq": 0, 2024-09-10T19:19:34.041 INFO:teuthology.orchestra.run.smithi183.stdout: "snap_epoch": 0, 2024-09-10T19:19:34.041 INFO:teuthology.orchestra.run.smithi183.stdout: "pool_snaps": [], 2024-09-10T19:19:34.041 INFO:teuthology.orchestra.run.smithi183.stdout: "removed_snaps": "[]", 2024-09-10T19:19:34.041 INFO:teuthology.orchestra.run.smithi183.stdout: "quota_max_bytes": 0, 2024-09-10T19:19:34.041 INFO:teuthology.orchestra.run.smithi183.stdout: "quota_max_objects": 0, 2024-09-10T19:19:34.041 INFO:teuthology.orchestra.run.smithi183.stdout: "tiers": [], 2024-09-10T19:19:34.041 INFO:teuthology.orchestra.run.smithi183.stdout: "tier_of": -1, 2024-09-10T19:19:34.041 INFO:teuthology.orchestra.run.smithi183.stdout: "read_tier": -1, 2024-09-10T19:19:34.041 INFO:teuthology.orchestra.run.smithi183.stdout: "write_tier": -1, 2024-09-10T19:19:34.041 INFO:teuthology.orchestra.run.smithi183.stdout: "cache_mode": "none", 2024-09-10T19:19:34.042 INFO:teuthology.orchestra.run.smithi183.stdout: "target_max_bytes": 0, 2024-09-10T19:19:34.042 INFO:teuthology.orchestra.run.smithi183.stdout: "target_max_objects": 0, 2024-09-10T19:19:34.042 INFO:teuthology.orchestra.run.smithi183.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-10T19:19:34.042 INFO:teuthology.orchestra.run.smithi183.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-10T19:19:34.042 INFO:teuthology.orchestra.run.smithi183.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-10T19:19:34.042 INFO:teuthology.orchestra.run.smithi183.stdout: "cache_min_flush_age": 0, 2024-09-10T19:19:34.042 INFO:teuthology.orchestra.run.smithi183.stdout: "cache_min_evict_age": 0, 2024-09-10T19:19:34.042 INFO:teuthology.orchestra.run.smithi183.stdout: "erasure_code_profile": "", 2024-09-10T19:19:34.042 INFO:teuthology.orchestra.run.smithi183.stdout: "hit_set_params": { 2024-09-10T19:19:34.042 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "none" 2024-09-10T19:19:34.042 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:19:34.042 INFO:teuthology.orchestra.run.smithi183.stdout: "hit_set_period": 0, 2024-09-10T19:19:34.042 INFO:teuthology.orchestra.run.smithi183.stdout: "hit_set_count": 0, 2024-09-10T19:19:34.042 INFO:teuthology.orchestra.run.smithi183.stdout: "use_gmt_hitset": true, 2024-09-10T19:19:34.043 INFO:teuthology.orchestra.run.smithi183.stdout: "min_read_recency_for_promote": 0, 2024-09-10T19:19:34.043 INFO:teuthology.orchestra.run.smithi183.stdout: "min_write_recency_for_promote": 0, 2024-09-10T19:19:34.043 INFO:teuthology.orchestra.run.smithi183.stdout: "hit_set_grade_decay_rate": 0, 2024-09-10T19:19:34.043 INFO:teuthology.orchestra.run.smithi183.stdout: "hit_set_search_last_n": 0, 2024-09-10T19:19:34.043 INFO:teuthology.orchestra.run.smithi183.stdout: "grade_table": [], 2024-09-10T19:19:34.043 INFO:teuthology.orchestra.run.smithi183.stdout: "stripe_width": 0, 2024-09-10T19:19:34.043 INFO:teuthology.orchestra.run.smithi183.stdout: "expected_num_objects": 0, 2024-09-10T19:19:34.043 INFO:teuthology.orchestra.run.smithi183.stdout: "fast_read": false, 2024-09-10T19:19:34.043 INFO:teuthology.orchestra.run.smithi183.stdout: "options": {}, 2024-09-10T19:19:34.043 INFO:teuthology.orchestra.run.smithi183.stdout: "application_metadata": { 2024-09-10T19:19:34.043 INFO:teuthology.orchestra.run.smithi183.stdout: "rgw": {} 2024-09-10T19:19:34.043 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:19:34.043 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:19:34.043 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:19:34.044 INFO:teuthology.orchestra.run.smithi183.stdout: "pool": 5, 2024-09-10T19:19:34.044 INFO:teuthology.orchestra.run.smithi183.stdout: "pool_name": "default.rgw.control", 2024-09-10T19:19:34.044 INFO:teuthology.orchestra.run.smithi183.stdout: "create_time": "2024-09-10T19:10:09.933596+0000", 2024-09-10T19:19:34.044 INFO:teuthology.orchestra.run.smithi183.stdout: "flags": 1, 2024-09-10T19:19:34.044 INFO:teuthology.orchestra.run.smithi183.stdout: "flags_names": "hashpspool", 2024-09-10T19:19:34.044 INFO:teuthology.orchestra.run.smithi183.stdout: "type": 1, 2024-09-10T19:19:34.044 INFO:teuthology.orchestra.run.smithi183.stdout: "size": 3, 2024-09-10T19:19:34.044 INFO:teuthology.orchestra.run.smithi183.stdout: "min_size": 2, 2024-09-10T19:19:34.044 INFO:teuthology.orchestra.run.smithi183.stdout: "crush_rule": 0, 2024-09-10T19:19:34.044 INFO:teuthology.orchestra.run.smithi183.stdout: "peering_crush_bucket_count": 0, 2024-09-10T19:19:34.044 INFO:teuthology.orchestra.run.smithi183.stdout: "peering_crush_bucket_target": 0, 2024-09-10T19:19:34.044 INFO:teuthology.orchestra.run.smithi183.stdout: "peering_crush_bucket_barrier": 0, 2024-09-10T19:19:34.044 INFO:teuthology.orchestra.run.smithi183.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-10T19:19:34.045 INFO:teuthology.orchestra.run.smithi183.stdout: "object_hash": 2, 2024-09-10T19:19:34.045 INFO:teuthology.orchestra.run.smithi183.stdout: "pg_autoscale_mode": "off", 2024-09-10T19:19:34.045 INFO:teuthology.orchestra.run.smithi183.stdout: "pg_num": 32, 2024-09-10T19:19:34.045 INFO:teuthology.orchestra.run.smithi183.stdout: "pg_placement_num": 32, 2024-09-10T19:19:34.045 INFO:teuthology.orchestra.run.smithi183.stdout: "pg_placement_num_target": 32, 2024-09-10T19:19:34.045 INFO:teuthology.orchestra.run.smithi183.stdout: "pg_num_target": 32, 2024-09-10T19:19:34.045 INFO:teuthology.orchestra.run.smithi183.stdout: "pg_num_pending": 32, 2024-09-10T19:19:34.045 INFO:teuthology.orchestra.run.smithi183.stdout: "last_pg_merge_meta": { 2024-09-10T19:19:34.045 INFO:teuthology.orchestra.run.smithi183.stdout: "source_pgid": "0.0", 2024-09-10T19:19:34.045 INFO:teuthology.orchestra.run.smithi183.stdout: "ready_epoch": 0, 2024-09-10T19:19:34.045 INFO:teuthology.orchestra.run.smithi183.stdout: "last_epoch_started": 0, 2024-09-10T19:19:34.045 INFO:teuthology.orchestra.run.smithi183.stdout: "last_epoch_clean": 0, 2024-09-10T19:19:34.045 INFO:teuthology.orchestra.run.smithi183.stdout: "source_version": "0'0", 2024-09-10T19:19:34.046 INFO:teuthology.orchestra.run.smithi183.stdout: "target_version": "0'0" 2024-09-10T19:19:34.046 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:19:34.046 INFO:teuthology.orchestra.run.smithi183.stdout: "last_change": "64", 2024-09-10T19:19:34.046 INFO:teuthology.orchestra.run.smithi183.stdout: "last_force_op_resend": "0", 2024-09-10T19:19:34.046 INFO:teuthology.orchestra.run.smithi183.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-10T19:19:34.046 INFO:teuthology.orchestra.run.smithi183.stdout: "last_force_op_resend_preluminous": "0", 2024-09-10T19:19:34.046 INFO:teuthology.orchestra.run.smithi183.stdout: "auid": 0, 2024-09-10T19:19:34.046 INFO:teuthology.orchestra.run.smithi183.stdout: "snap_mode": "selfmanaged", 2024-09-10T19:19:34.046 INFO:teuthology.orchestra.run.smithi183.stdout: "snap_seq": 0, 2024-09-10T19:19:34.046 INFO:teuthology.orchestra.run.smithi183.stdout: "snap_epoch": 0, 2024-09-10T19:19:34.046 INFO:teuthology.orchestra.run.smithi183.stdout: "pool_snaps": [], 2024-09-10T19:19:34.046 INFO:teuthology.orchestra.run.smithi183.stdout: "removed_snaps": "[]", 2024-09-10T19:19:34.046 INFO:teuthology.orchestra.run.smithi183.stdout: "quota_max_bytes": 0, 2024-09-10T19:19:34.046 INFO:teuthology.orchestra.run.smithi183.stdout: "quota_max_objects": 0, 2024-09-10T19:19:34.047 INFO:teuthology.orchestra.run.smithi183.stdout: "tiers": [], 2024-09-10T19:19:34.047 INFO:teuthology.orchestra.run.smithi183.stdout: "tier_of": -1, 2024-09-10T19:19:34.047 INFO:teuthology.orchestra.run.smithi183.stdout: "read_tier": -1, 2024-09-10T19:19:34.047 INFO:teuthology.orchestra.run.smithi183.stdout: "write_tier": -1, 2024-09-10T19:19:34.047 INFO:teuthology.orchestra.run.smithi183.stdout: "cache_mode": "none", 2024-09-10T19:19:34.047 INFO:teuthology.orchestra.run.smithi183.stdout: "target_max_bytes": 0, 2024-09-10T19:19:34.047 INFO:teuthology.orchestra.run.smithi183.stdout: "target_max_objects": 0, 2024-09-10T19:19:34.047 INFO:teuthology.orchestra.run.smithi183.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-10T19:19:34.047 INFO:teuthology.orchestra.run.smithi183.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-10T19:19:34.047 INFO:teuthology.orchestra.run.smithi183.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-10T19:19:34.047 INFO:teuthology.orchestra.run.smithi183.stdout: "cache_min_flush_age": 0, 2024-09-10T19:19:34.047 INFO:teuthology.orchestra.run.smithi183.stdout: "cache_min_evict_age": 0, 2024-09-10T19:19:34.047 INFO:teuthology.orchestra.run.smithi183.stdout: "erasure_code_profile": "", 2024-09-10T19:19:34.048 INFO:teuthology.orchestra.run.smithi183.stdout: "hit_set_params": { 2024-09-10T19:19:34.048 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "none" 2024-09-10T19:19:34.048 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:19:34.048 INFO:teuthology.orchestra.run.smithi183.stdout: "hit_set_period": 0, 2024-09-10T19:19:34.048 INFO:teuthology.orchestra.run.smithi183.stdout: "hit_set_count": 0, 2024-09-10T19:19:34.048 INFO:teuthology.orchestra.run.smithi183.stdout: "use_gmt_hitset": true, 2024-09-10T19:19:34.048 INFO:teuthology.orchestra.run.smithi183.stdout: "min_read_recency_for_promote": 0, 2024-09-10T19:19:34.048 INFO:teuthology.orchestra.run.smithi183.stdout: "min_write_recency_for_promote": 0, 2024-09-10T19:19:34.048 INFO:teuthology.orchestra.run.smithi183.stdout: "hit_set_grade_decay_rate": 0, 2024-09-10T19:19:34.048 INFO:teuthology.orchestra.run.smithi183.stdout: "hit_set_search_last_n": 0, 2024-09-10T19:19:34.048 INFO:teuthology.orchestra.run.smithi183.stdout: "grade_table": [], 2024-09-10T19:19:34.048 INFO:teuthology.orchestra.run.smithi183.stdout: "stripe_width": 0, 2024-09-10T19:19:34.048 INFO:teuthology.orchestra.run.smithi183.stdout: "expected_num_objects": 0, 2024-09-10T19:19:34.048 INFO:teuthology.orchestra.run.smithi183.stdout: "fast_read": false, 2024-09-10T19:19:34.049 INFO:teuthology.orchestra.run.smithi183.stdout: "options": {}, 2024-09-10T19:19:34.049 INFO:teuthology.orchestra.run.smithi183.stdout: "application_metadata": { 2024-09-10T19:19:34.049 INFO:teuthology.orchestra.run.smithi183.stdout: "rgw": {} 2024-09-10T19:19:34.049 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:19:34.049 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:19:34.049 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:19:34.049 INFO:teuthology.orchestra.run.smithi183.stdout: "pool": 6, 2024-09-10T19:19:34.049 INFO:teuthology.orchestra.run.smithi183.stdout: "pool_name": "default.rgw.meta", 2024-09-10T19:19:34.049 INFO:teuthology.orchestra.run.smithi183.stdout: "create_time": "2024-09-10T19:10:11.993712+0000", 2024-09-10T19:19:34.049 INFO:teuthology.orchestra.run.smithi183.stdout: "flags": 1, 2024-09-10T19:19:34.049 INFO:teuthology.orchestra.run.smithi183.stdout: "flags_names": "hashpspool", 2024-09-10T19:19:34.049 INFO:teuthology.orchestra.run.smithi183.stdout: "type": 1, 2024-09-10T19:19:34.049 INFO:teuthology.orchestra.run.smithi183.stdout: "size": 3, 2024-09-10T19:19:34.050 INFO:teuthology.orchestra.run.smithi183.stdout: "min_size": 2, 2024-09-10T19:19:34.050 INFO:teuthology.orchestra.run.smithi183.stdout: "crush_rule": 0, 2024-09-10T19:19:34.050 INFO:teuthology.orchestra.run.smithi183.stdout: "peering_crush_bucket_count": 0, 2024-09-10T19:19:34.050 INFO:teuthology.orchestra.run.smithi183.stdout: "peering_crush_bucket_target": 0, 2024-09-10T19:19:34.050 INFO:teuthology.orchestra.run.smithi183.stdout: "peering_crush_bucket_barrier": 0, 2024-09-10T19:19:34.050 INFO:teuthology.orchestra.run.smithi183.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-10T19:19:34.050 INFO:teuthology.orchestra.run.smithi183.stdout: "object_hash": 2, 2024-09-10T19:19:34.050 INFO:teuthology.orchestra.run.smithi183.stdout: "pg_autoscale_mode": "off", 2024-09-10T19:19:34.050 INFO:teuthology.orchestra.run.smithi183.stdout: "pg_num": 32, 2024-09-10T19:19:34.050 INFO:teuthology.orchestra.run.smithi183.stdout: "pg_placement_num": 32, 2024-09-10T19:19:34.050 INFO:teuthology.orchestra.run.smithi183.stdout: "pg_placement_num_target": 32, 2024-09-10T19:19:34.050 INFO:teuthology.orchestra.run.smithi183.stdout: "pg_num_target": 32, 2024-09-10T19:19:34.050 INFO:teuthology.orchestra.run.smithi183.stdout: "pg_num_pending": 32, 2024-09-10T19:19:34.050 INFO:teuthology.orchestra.run.smithi183.stdout: "last_pg_merge_meta": { 2024-09-10T19:19:34.051 INFO:teuthology.orchestra.run.smithi183.stdout: "source_pgid": "0.0", 2024-09-10T19:19:34.051 INFO:teuthology.orchestra.run.smithi183.stdout: "ready_epoch": 0, 2024-09-10T19:19:34.051 INFO:teuthology.orchestra.run.smithi183.stdout: "last_epoch_started": 0, 2024-09-10T19:19:34.051 INFO:teuthology.orchestra.run.smithi183.stdout: "last_epoch_clean": 0, 2024-09-10T19:19:34.051 INFO:teuthology.orchestra.run.smithi183.stdout: "source_version": "0'0", 2024-09-10T19:19:34.051 INFO:teuthology.orchestra.run.smithi183.stdout: "target_version": "0'0" 2024-09-10T19:19:34.051 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:19:34.051 INFO:teuthology.orchestra.run.smithi183.stdout: "last_change": "66", 2024-09-10T19:19:34.051 INFO:teuthology.orchestra.run.smithi183.stdout: "last_force_op_resend": "0", 2024-09-10T19:19:34.051 INFO:teuthology.orchestra.run.smithi183.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-10T19:19:34.051 INFO:teuthology.orchestra.run.smithi183.stdout: "last_force_op_resend_preluminous": "0", 2024-09-10T19:19:34.051 INFO:teuthology.orchestra.run.smithi183.stdout: "auid": 0, 2024-09-10T19:19:34.051 INFO:teuthology.orchestra.run.smithi183.stdout: "snap_mode": "selfmanaged", 2024-09-10T19:19:34.051 INFO:teuthology.orchestra.run.smithi183.stdout: "snap_seq": 0, 2024-09-10T19:19:34.052 INFO:teuthology.orchestra.run.smithi183.stdout: "snap_epoch": 0, 2024-09-10T19:19:34.052 INFO:teuthology.orchestra.run.smithi183.stdout: "pool_snaps": [], 2024-09-10T19:19:34.052 INFO:teuthology.orchestra.run.smithi183.stdout: "removed_snaps": "[]", 2024-09-10T19:19:34.052 INFO:teuthology.orchestra.run.smithi183.stdout: "quota_max_bytes": 0, 2024-09-10T19:19:34.052 INFO:teuthology.orchestra.run.smithi183.stdout: "quota_max_objects": 0, 2024-09-10T19:19:34.052 INFO:teuthology.orchestra.run.smithi183.stdout: "tiers": [], 2024-09-10T19:19:34.052 INFO:teuthology.orchestra.run.smithi183.stdout: "tier_of": -1, 2024-09-10T19:19:34.052 INFO:teuthology.orchestra.run.smithi183.stdout: "read_tier": -1, 2024-09-10T19:19:34.052 INFO:teuthology.orchestra.run.smithi183.stdout: "write_tier": -1, 2024-09-10T19:19:34.052 INFO:teuthology.orchestra.run.smithi183.stdout: "cache_mode": "none", 2024-09-10T19:19:34.052 INFO:teuthology.orchestra.run.smithi183.stdout: "target_max_bytes": 0, 2024-09-10T19:19:34.052 INFO:teuthology.orchestra.run.smithi183.stdout: "target_max_objects": 0, 2024-09-10T19:19:34.052 INFO:teuthology.orchestra.run.smithi183.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-10T19:19:34.053 INFO:teuthology.orchestra.run.smithi183.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-10T19:19:34.053 INFO:teuthology.orchestra.run.smithi183.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-10T19:19:34.053 INFO:teuthology.orchestra.run.smithi183.stdout: "cache_min_flush_age": 0, 2024-09-10T19:19:34.053 INFO:teuthology.orchestra.run.smithi183.stdout: "cache_min_evict_age": 0, 2024-09-10T19:19:34.053 INFO:teuthology.orchestra.run.smithi183.stdout: "erasure_code_profile": "", 2024-09-10T19:19:34.053 INFO:teuthology.orchestra.run.smithi183.stdout: "hit_set_params": { 2024-09-10T19:19:34.053 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "none" 2024-09-10T19:19:34.053 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:19:34.053 INFO:teuthology.orchestra.run.smithi183.stdout: "hit_set_period": 0, 2024-09-10T19:19:34.053 INFO:teuthology.orchestra.run.smithi183.stdout: "hit_set_count": 0, 2024-09-10T19:19:34.053 INFO:teuthology.orchestra.run.smithi183.stdout: "use_gmt_hitset": true, 2024-09-10T19:19:34.053 INFO:teuthology.orchestra.run.smithi183.stdout: "min_read_recency_for_promote": 0, 2024-09-10T19:19:34.053 INFO:teuthology.orchestra.run.smithi183.stdout: "min_write_recency_for_promote": 0, 2024-09-10T19:19:34.053 INFO:teuthology.orchestra.run.smithi183.stdout: "hit_set_grade_decay_rate": 0, 2024-09-10T19:19:34.054 INFO:teuthology.orchestra.run.smithi183.stdout: "hit_set_search_last_n": 0, 2024-09-10T19:19:34.054 INFO:teuthology.orchestra.run.smithi183.stdout: "grade_table": [], 2024-09-10T19:19:34.054 INFO:teuthology.orchestra.run.smithi183.stdout: "stripe_width": 0, 2024-09-10T19:19:34.054 INFO:teuthology.orchestra.run.smithi183.stdout: "expected_num_objects": 0, 2024-09-10T19:19:34.054 INFO:teuthology.orchestra.run.smithi183.stdout: "fast_read": false, 2024-09-10T19:19:34.054 INFO:teuthology.orchestra.run.smithi183.stdout: "options": { 2024-09-10T19:19:34.054 INFO:teuthology.orchestra.run.smithi183.stdout: "pg_autoscale_bias": 4 2024-09-10T19:19:34.054 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:19:34.054 INFO:teuthology.orchestra.run.smithi183.stdout: "application_metadata": { 2024-09-10T19:19:34.054 INFO:teuthology.orchestra.run.smithi183.stdout: "rgw": {} 2024-09-10T19:19:34.054 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:19:34.054 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:19:34.054 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:19:34.054 INFO:teuthology.orchestra.run.smithi183.stdout: "pool": 8, 2024-09-10T19:19:34.055 INFO:teuthology.orchestra.run.smithi183.stdout: "pool_name": "unique_pool_1", 2024-09-10T19:19:34.055 INFO:teuthology.orchestra.run.smithi183.stdout: "create_time": "2024-09-10T19:16:11.474472+0000", 2024-09-10T19:19:34.055 INFO:teuthology.orchestra.run.smithi183.stdout: "flags": 1, 2024-09-10T19:19:34.055 INFO:teuthology.orchestra.run.smithi183.stdout: "flags_names": "hashpspool", 2024-09-10T19:19:34.055 INFO:teuthology.orchestra.run.smithi183.stdout: "type": 1, 2024-09-10T19:19:34.055 INFO:teuthology.orchestra.run.smithi183.stdout: "size": 3, 2024-09-10T19:19:34.055 INFO:teuthology.orchestra.run.smithi183.stdout: "min_size": 2, 2024-09-10T19:19:34.055 INFO:teuthology.orchestra.run.smithi183.stdout: "crush_rule": 0, 2024-09-10T19:19:34.055 INFO:teuthology.orchestra.run.smithi183.stdout: "peering_crush_bucket_count": 0, 2024-09-10T19:19:34.055 INFO:teuthology.orchestra.run.smithi183.stdout: "peering_crush_bucket_target": 0, 2024-09-10T19:19:34.055 INFO:teuthology.orchestra.run.smithi183.stdout: "peering_crush_bucket_barrier": 0, 2024-09-10T19:19:34.055 INFO:teuthology.orchestra.run.smithi183.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-10T19:19:34.055 INFO:teuthology.orchestra.run.smithi183.stdout: "object_hash": 2, 2024-09-10T19:19:34.055 INFO:teuthology.orchestra.run.smithi183.stdout: "pg_autoscale_mode": "off", 2024-09-10T19:19:34.056 INFO:teuthology.orchestra.run.smithi183.stdout: "pg_num": 26, 2024-09-10T19:19:34.056 INFO:teuthology.orchestra.run.smithi183.stdout: "pg_placement_num": 26, 2024-09-10T19:19:34.056 INFO:teuthology.orchestra.run.smithi183.stdout: "pg_placement_num_target": 26, 2024-09-10T19:19:34.056 INFO:teuthology.orchestra.run.smithi183.stdout: "pg_num_target": 26, 2024-09-10T19:19:34.056 INFO:teuthology.orchestra.run.smithi183.stdout: "pg_num_pending": 26, 2024-09-10T19:19:34.056 INFO:teuthology.orchestra.run.smithi183.stdout: "last_pg_merge_meta": { 2024-09-10T19:19:34.056 INFO:teuthology.orchestra.run.smithi183.stdout: "source_pgid": "0.0", 2024-09-10T19:19:34.056 INFO:teuthology.orchestra.run.smithi183.stdout: "ready_epoch": 0, 2024-09-10T19:19:34.056 INFO:teuthology.orchestra.run.smithi183.stdout: "last_epoch_started": 0, 2024-09-10T19:19:34.056 INFO:teuthology.orchestra.run.smithi183.stdout: "last_epoch_clean": 0, 2024-09-10T19:19:34.056 INFO:teuthology.orchestra.run.smithi183.stdout: "source_version": "0'0", 2024-09-10T19:19:34.056 INFO:teuthology.orchestra.run.smithi183.stdout: "target_version": "0'0" 2024-09-10T19:19:34.056 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:19:34.057 INFO:teuthology.orchestra.run.smithi183.stdout: "last_change": "179", 2024-09-10T19:19:34.057 INFO:teuthology.orchestra.run.smithi183.stdout: "last_force_op_resend": "0", 2024-09-10T19:19:34.057 INFO:teuthology.orchestra.run.smithi183.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-10T19:19:34.057 INFO:teuthology.orchestra.run.smithi183.stdout: "last_force_op_resend_preluminous": "177", 2024-09-10T19:19:34.057 INFO:teuthology.orchestra.run.smithi183.stdout: "auid": 0, 2024-09-10T19:19:34.057 INFO:teuthology.orchestra.run.smithi183.stdout: "snap_mode": "selfmanaged", 2024-09-10T19:19:34.057 INFO:teuthology.orchestra.run.smithi183.stdout: "snap_seq": 0, 2024-09-10T19:19:34.057 INFO:teuthology.orchestra.run.smithi183.stdout: "snap_epoch": 0, 2024-09-10T19:19:34.057 INFO:teuthology.orchestra.run.smithi183.stdout: "pool_snaps": [], 2024-09-10T19:19:34.057 INFO:teuthology.orchestra.run.smithi183.stdout: "removed_snaps": "[]", 2024-09-10T19:19:34.057 INFO:teuthology.orchestra.run.smithi183.stdout: "quota_max_bytes": 0, 2024-09-10T19:19:34.057 INFO:teuthology.orchestra.run.smithi183.stdout: "quota_max_objects": 0, 2024-09-10T19:19:34.057 INFO:teuthology.orchestra.run.smithi183.stdout: "tiers": [], 2024-09-10T19:19:34.057 INFO:teuthology.orchestra.run.smithi183.stdout: "tier_of": -1, 2024-09-10T19:19:34.058 INFO:teuthology.orchestra.run.smithi183.stdout: "read_tier": -1, 2024-09-10T19:19:34.058 INFO:teuthology.orchestra.run.smithi183.stdout: "write_tier": -1, 2024-09-10T19:19:34.058 INFO:teuthology.orchestra.run.smithi183.stdout: "cache_mode": "none", 2024-09-10T19:19:34.058 INFO:teuthology.orchestra.run.smithi183.stdout: "target_max_bytes": 0, 2024-09-10T19:19:34.058 INFO:teuthology.orchestra.run.smithi183.stdout: "target_max_objects": 0, 2024-09-10T19:19:34.058 INFO:teuthology.orchestra.run.smithi183.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-10T19:19:34.058 INFO:teuthology.orchestra.run.smithi183.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-10T19:19:34.058 INFO:teuthology.orchestra.run.smithi183.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-10T19:19:34.058 INFO:teuthology.orchestra.run.smithi183.stdout: "cache_min_flush_age": 0, 2024-09-10T19:19:34.058 INFO:teuthology.orchestra.run.smithi183.stdout: "cache_min_evict_age": 0, 2024-09-10T19:19:34.058 INFO:teuthology.orchestra.run.smithi183.stdout: "erasure_code_profile": "", 2024-09-10T19:19:34.058 INFO:teuthology.orchestra.run.smithi183.stdout: "hit_set_params": { 2024-09-10T19:19:34.058 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "none" 2024-09-10T19:19:34.058 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:19:34.059 INFO:teuthology.orchestra.run.smithi183.stdout: "hit_set_period": 0, 2024-09-10T19:19:34.059 INFO:teuthology.orchestra.run.smithi183.stdout: "hit_set_count": 0, 2024-09-10T19:19:34.059 INFO:teuthology.orchestra.run.smithi183.stdout: "use_gmt_hitset": true, 2024-09-10T19:19:34.059 INFO:teuthology.orchestra.run.smithi183.stdout: "min_read_recency_for_promote": 0, 2024-09-10T19:19:34.059 INFO:teuthology.orchestra.run.smithi183.stdout: "min_write_recency_for_promote": 0, 2024-09-10T19:19:34.059 INFO:teuthology.orchestra.run.smithi183.stdout: "hit_set_grade_decay_rate": 0, 2024-09-10T19:19:34.059 INFO:teuthology.orchestra.run.smithi183.stdout: "hit_set_search_last_n": 0, 2024-09-10T19:19:34.059 INFO:teuthology.orchestra.run.smithi183.stdout: "grade_table": [], 2024-09-10T19:19:34.059 INFO:teuthology.orchestra.run.smithi183.stdout: "stripe_width": 0, 2024-09-10T19:19:34.059 INFO:teuthology.orchestra.run.smithi183.stdout: "expected_num_objects": 0, 2024-09-10T19:19:34.059 INFO:teuthology.orchestra.run.smithi183.stdout: "fast_read": false, 2024-09-10T19:19:34.059 INFO:teuthology.orchestra.run.smithi183.stdout: "options": {}, 2024-09-10T19:19:34.059 INFO:teuthology.orchestra.run.smithi183.stdout: "application_metadata": { 2024-09-10T19:19:34.059 INFO:teuthology.orchestra.run.smithi183.stdout: "rados": {} 2024-09-10T19:19:34.060 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:19:34.060 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:19:34.060 INFO:teuthology.orchestra.run.smithi183.stdout: ], 2024-09-10T19:19:34.060 INFO:teuthology.orchestra.run.smithi183.stdout: "osds": [ 2024-09-10T19:19:34.060 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:19:34.060 INFO:teuthology.orchestra.run.smithi183.stdout: "osd": 0, 2024-09-10T19:19:34.060 INFO:teuthology.orchestra.run.smithi183.stdout: "uuid": "9d323010-394a-49aa-ba9b-244d87faf36e", 2024-09-10T19:19:34.060 INFO:teuthology.orchestra.run.smithi183.stdout: "up": 1, 2024-09-10T19:19:34.060 INFO:teuthology.orchestra.run.smithi183.stdout: "in": 1, 2024-09-10T19:19:34.060 INFO:teuthology.orchestra.run.smithi183.stdout: "weight": 1, 2024-09-10T19:19:34.060 INFO:teuthology.orchestra.run.smithi183.stdout: "primary_affinity": 1, 2024-09-10T19:19:34.060 INFO:teuthology.orchestra.run.smithi183.stdout: "last_clean_begin": 0, 2024-09-10T19:19:34.060 INFO:teuthology.orchestra.run.smithi183.stdout: "last_clean_end": 0, 2024-09-10T19:19:34.061 INFO:teuthology.orchestra.run.smithi183.stdout: "up_from": 8, 2024-09-10T19:19:34.061 INFO:teuthology.orchestra.run.smithi183.stdout: "up_thru": 204, 2024-09-10T19:19:34.061 INFO:teuthology.orchestra.run.smithi183.stdout: "down_at": 0, 2024-09-10T19:19:34.061 INFO:teuthology.orchestra.run.smithi183.stdout: "lost_at": 0, 2024-09-10T19:19:34.061 INFO:teuthology.orchestra.run.smithi183.stdout: "public_addrs": { 2024-09-10T19:19:34.061 INFO:teuthology.orchestra.run.smithi183.stdout: "addrvec": [ 2024-09-10T19:19:34.061 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:19:34.061 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v2", 2024-09-10T19:19:34.061 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.183:6802", 2024-09-10T19:19:34.061 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 2813074952 2024-09-10T19:19:34.061 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:19:34.061 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:19:34.061 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v1", 2024-09-10T19:19:34.061 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.183:6803", 2024-09-10T19:19:34.062 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 2813074952 2024-09-10T19:19:34.062 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:19:34.062 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:19:34.062 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:19:34.062 INFO:teuthology.orchestra.run.smithi183.stdout: "cluster_addrs": { 2024-09-10T19:19:34.062 INFO:teuthology.orchestra.run.smithi183.stdout: "addrvec": [ 2024-09-10T19:19:34.062 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:19:34.062 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v2", 2024-09-10T19:19:34.062 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.183:6804", 2024-09-10T19:19:34.062 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 2813074952 2024-09-10T19:19:34.062 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:19:34.062 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:19:34.062 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v1", 2024-09-10T19:19:34.062 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.183:6805", 2024-09-10T19:19:34.063 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 2813074952 2024-09-10T19:19:34.063 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:19:34.063 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:19:34.063 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:19:34.063 INFO:teuthology.orchestra.run.smithi183.stdout: "heartbeat_back_addrs": { 2024-09-10T19:19:34.063 INFO:teuthology.orchestra.run.smithi183.stdout: "addrvec": [ 2024-09-10T19:19:34.063 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:19:34.063 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v2", 2024-09-10T19:19:34.063 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.183:6808", 2024-09-10T19:19:34.063 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 2813074952 2024-09-10T19:19:34.063 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:19:34.063 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:19:34.063 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v1", 2024-09-10T19:19:34.063 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.183:6809", 2024-09-10T19:19:34.064 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 2813074952 2024-09-10T19:19:34.064 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:19:34.064 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:19:34.064 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:19:34.064 INFO:teuthology.orchestra.run.smithi183.stdout: "heartbeat_front_addrs": { 2024-09-10T19:19:34.064 INFO:teuthology.orchestra.run.smithi183.stdout: "addrvec": [ 2024-09-10T19:19:34.064 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:19:34.064 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v2", 2024-09-10T19:19:34.064 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.183:6806", 2024-09-10T19:19:34.064 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 2813074952 2024-09-10T19:19:34.064 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:19:34.064 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:19:34.064 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v1", 2024-09-10T19:19:34.064 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.183:6807", 2024-09-10T19:19:34.065 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 2813074952 2024-09-10T19:19:34.065 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:19:34.065 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:19:34.065 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:19:34.065 INFO:teuthology.orchestra.run.smithi183.stdout: "public_addr": "172.21.15.183:6803/2813074952", 2024-09-10T19:19:34.065 INFO:teuthology.orchestra.run.smithi183.stdout: "cluster_addr": "172.21.15.183:6805/2813074952", 2024-09-10T19:19:34.065 INFO:teuthology.orchestra.run.smithi183.stdout: "heartbeat_back_addr": "172.21.15.183:6809/2813074952", 2024-09-10T19:19:34.065 INFO:teuthology.orchestra.run.smithi183.stdout: "heartbeat_front_addr": "172.21.15.183:6807/2813074952", 2024-09-10T19:19:34.065 INFO:teuthology.orchestra.run.smithi183.stdout: "state": [ 2024-09-10T19:19:34.065 INFO:teuthology.orchestra.run.smithi183.stdout: "exists", 2024-09-10T19:19:34.065 INFO:teuthology.orchestra.run.smithi183.stdout: "up" 2024-09-10T19:19:34.065 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:19:34.065 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:19:34.065 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:19:34.066 INFO:teuthology.orchestra.run.smithi183.stdout: "osd": 1, 2024-09-10T19:19:34.066 INFO:teuthology.orchestra.run.smithi183.stdout: "uuid": "71a6c5dc-5109-4284-8d20-187d8e7afc9e", 2024-09-10T19:19:34.066 INFO:teuthology.orchestra.run.smithi183.stdout: "up": 1, 2024-09-10T19:19:34.066 INFO:teuthology.orchestra.run.smithi183.stdout: "in": 0, 2024-09-10T19:19:34.066 INFO:teuthology.orchestra.run.smithi183.stdout: "weight": 0, 2024-09-10T19:19:34.066 INFO:teuthology.orchestra.run.smithi183.stdout: "primary_affinity": 1, 2024-09-10T19:19:34.066 INFO:teuthology.orchestra.run.smithi183.stdout: "last_clean_begin": 13, 2024-09-10T19:19:34.066 INFO:teuthology.orchestra.run.smithi183.stdout: "last_clean_end": 98, 2024-09-10T19:19:34.066 INFO:teuthology.orchestra.run.smithi183.stdout: "up_from": 138, 2024-09-10T19:19:34.066 INFO:teuthology.orchestra.run.smithi183.stdout: "up_thru": 86, 2024-09-10T19:19:34.066 INFO:teuthology.orchestra.run.smithi183.stdout: "down_at": 99, 2024-09-10T19:19:34.066 INFO:teuthology.orchestra.run.smithi183.stdout: "lost_at": 0, 2024-09-10T19:19:34.066 INFO:teuthology.orchestra.run.smithi183.stdout: "public_addrs": { 2024-09-10T19:19:34.066 INFO:teuthology.orchestra.run.smithi183.stdout: "addrvec": [ 2024-09-10T19:19:34.067 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:19:34.067 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v2", 2024-09-10T19:19:34.067 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.183:6810", 2024-09-10T19:19:34.067 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 219854798 2024-09-10T19:19:34.067 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:19:34.067 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:19:34.067 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v1", 2024-09-10T19:19:34.067 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.183:6811", 2024-09-10T19:19:34.067 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 219854798 2024-09-10T19:19:34.067 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:19:34.067 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:19:34.067 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:19:34.067 INFO:teuthology.orchestra.run.smithi183.stdout: "cluster_addrs": { 2024-09-10T19:19:34.067 INFO:teuthology.orchestra.run.smithi183.stdout: "addrvec": [ 2024-09-10T19:19:34.068 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:19:34.068 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v2", 2024-09-10T19:19:34.068 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.183:6812", 2024-09-10T19:19:34.068 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 219854798 2024-09-10T19:19:34.068 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:19:34.068 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:19:34.068 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v1", 2024-09-10T19:19:34.068 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.183:6813", 2024-09-10T19:19:34.068 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 219854798 2024-09-10T19:19:34.068 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:19:34.068 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:19:34.068 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:19:34.068 INFO:teuthology.orchestra.run.smithi183.stdout: "heartbeat_back_addrs": { 2024-09-10T19:19:34.068 INFO:teuthology.orchestra.run.smithi183.stdout: "addrvec": [ 2024-09-10T19:19:34.069 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:19:34.069 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v2", 2024-09-10T19:19:34.069 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.183:6816", 2024-09-10T19:19:34.069 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 219854798 2024-09-10T19:19:34.069 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:19:34.069 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:19:34.069 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v1", 2024-09-10T19:19:34.069 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.183:6817", 2024-09-10T19:19:34.069 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 219854798 2024-09-10T19:19:34.069 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:19:34.069 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:19:34.069 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:19:34.069 INFO:teuthology.orchestra.run.smithi183.stdout: "heartbeat_front_addrs": { 2024-09-10T19:19:34.069 INFO:teuthology.orchestra.run.smithi183.stdout: "addrvec": [ 2024-09-10T19:19:34.070 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:19:34.070 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v2", 2024-09-10T19:19:34.070 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.183:6814", 2024-09-10T19:19:34.070 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 219854798 2024-09-10T19:19:34.070 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:19:34.070 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:19:34.070 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v1", 2024-09-10T19:19:34.070 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.183:6815", 2024-09-10T19:19:34.070 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 219854798 2024-09-10T19:19:34.070 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:19:34.070 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:19:34.070 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:19:34.070 INFO:teuthology.orchestra.run.smithi183.stdout: "public_addr": "172.21.15.183:6811/219854798", 2024-09-10T19:19:34.070 INFO:teuthology.orchestra.run.smithi183.stdout: "cluster_addr": "172.21.15.183:6813/219854798", 2024-09-10T19:19:34.071 INFO:teuthology.orchestra.run.smithi183.stdout: "heartbeat_back_addr": "172.21.15.183:6817/219854798", 2024-09-10T19:19:34.071 INFO:teuthology.orchestra.run.smithi183.stdout: "heartbeat_front_addr": "172.21.15.183:6815/219854798", 2024-09-10T19:19:34.071 INFO:teuthology.orchestra.run.smithi183.stdout: "state": [ 2024-09-10T19:19:34.071 INFO:teuthology.orchestra.run.smithi183.stdout: "exists", 2024-09-10T19:19:34.071 INFO:teuthology.orchestra.run.smithi183.stdout: "up" 2024-09-10T19:19:34.071 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:19:34.071 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:19:34.071 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:19:34.071 INFO:teuthology.orchestra.run.smithi183.stdout: "osd": 2, 2024-09-10T19:19:34.071 INFO:teuthology.orchestra.run.smithi183.stdout: "uuid": "fc310d5b-b89b-4655-9989-7e10ba356509", 2024-09-10T19:19:34.071 INFO:teuthology.orchestra.run.smithi183.stdout: "up": 1, 2024-09-10T19:19:34.071 INFO:teuthology.orchestra.run.smithi183.stdout: "in": 0, 2024-09-10T19:19:34.071 INFO:teuthology.orchestra.run.smithi183.stdout: "weight": 0, 2024-09-10T19:19:34.071 INFO:teuthology.orchestra.run.smithi183.stdout: "primary_affinity": 1, 2024-09-10T19:19:34.072 INFO:teuthology.orchestra.run.smithi183.stdout: "last_clean_begin": 0, 2024-09-10T19:19:34.072 INFO:teuthology.orchestra.run.smithi183.stdout: "last_clean_end": 0, 2024-09-10T19:19:34.072 INFO:teuthology.orchestra.run.smithi183.stdout: "up_from": 18, 2024-09-10T19:19:34.072 INFO:teuthology.orchestra.run.smithi183.stdout: "up_thru": 148, 2024-09-10T19:19:34.072 INFO:teuthology.orchestra.run.smithi183.stdout: "down_at": 0, 2024-09-10T19:19:34.072 INFO:teuthology.orchestra.run.smithi183.stdout: "lost_at": 0, 2024-09-10T19:19:34.072 INFO:teuthology.orchestra.run.smithi183.stdout: "public_addrs": { 2024-09-10T19:19:34.072 INFO:teuthology.orchestra.run.smithi183.stdout: "addrvec": [ 2024-09-10T19:19:34.072 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:19:34.072 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v2", 2024-09-10T19:19:34.072 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.183:6818", 2024-09-10T19:19:34.072 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 3730013786 2024-09-10T19:19:34.072 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:19:34.073 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:19:34.073 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v1", 2024-09-10T19:19:34.073 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.183:6819", 2024-09-10T19:19:34.073 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 3730013786 2024-09-10T19:19:34.073 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:19:34.073 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:19:34.073 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:19:34.073 INFO:teuthology.orchestra.run.smithi183.stdout: "cluster_addrs": { 2024-09-10T19:19:34.073 INFO:teuthology.orchestra.run.smithi183.stdout: "addrvec": [ 2024-09-10T19:19:34.073 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:19:34.073 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v2", 2024-09-10T19:19:34.073 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.183:6820", 2024-09-10T19:19:34.073 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 3730013786 2024-09-10T19:19:34.073 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:19:34.074 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:19:34.074 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v1", 2024-09-10T19:19:34.074 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.183:6821", 2024-09-10T19:19:34.074 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 3730013786 2024-09-10T19:19:34.074 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:19:34.074 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:19:34.074 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:19:34.074 INFO:teuthology.orchestra.run.smithi183.stdout: "heartbeat_back_addrs": { 2024-09-10T19:19:34.074 INFO:teuthology.orchestra.run.smithi183.stdout: "addrvec": [ 2024-09-10T19:19:34.074 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:19:34.074 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v2", 2024-09-10T19:19:34.074 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.183:6824", 2024-09-10T19:19:34.074 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 3730013786 2024-09-10T19:19:34.074 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:19:34.075 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:19:34.075 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v1", 2024-09-10T19:19:34.075 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.183:6825", 2024-09-10T19:19:34.075 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 3730013786 2024-09-10T19:19:34.075 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:19:34.075 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:19:34.075 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:19:34.075 INFO:teuthology.orchestra.run.smithi183.stdout: "heartbeat_front_addrs": { 2024-09-10T19:19:34.075 INFO:teuthology.orchestra.run.smithi183.stdout: "addrvec": [ 2024-09-10T19:19:34.075 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:19:34.075 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v2", 2024-09-10T19:19:34.075 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.183:6822", 2024-09-10T19:19:34.075 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 3730013786 2024-09-10T19:19:34.075 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:19:34.076 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:19:34.076 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v1", 2024-09-10T19:19:34.076 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.183:6823", 2024-09-10T19:19:34.076 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 3730013786 2024-09-10T19:19:34.076 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:19:34.076 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:19:34.076 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:19:34.076 INFO:teuthology.orchestra.run.smithi183.stdout: "public_addr": "172.21.15.183:6819/3730013786", 2024-09-10T19:19:34.076 INFO:teuthology.orchestra.run.smithi183.stdout: "cluster_addr": "172.21.15.183:6821/3730013786", 2024-09-10T19:19:34.076 INFO:teuthology.orchestra.run.smithi183.stdout: "heartbeat_back_addr": "172.21.15.183:6825/3730013786", 2024-09-10T19:19:34.076 INFO:teuthology.orchestra.run.smithi183.stdout: "heartbeat_front_addr": "172.21.15.183:6823/3730013786", 2024-09-10T19:19:34.076 INFO:teuthology.orchestra.run.smithi183.stdout: "state": [ 2024-09-10T19:19:34.076 INFO:teuthology.orchestra.run.smithi183.stdout: "exists", 2024-09-10T19:19:34.077 INFO:teuthology.orchestra.run.smithi183.stdout: "up" 2024-09-10T19:19:34.077 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:19:34.077 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:19:34.077 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:19:34.077 INFO:teuthology.orchestra.run.smithi183.stdout: "osd": 3, 2024-09-10T19:19:34.077 INFO:teuthology.orchestra.run.smithi183.stdout: "uuid": "7fd07a95-ad22-43f5-a968-539552283e89", 2024-09-10T19:19:34.077 INFO:teuthology.orchestra.run.smithi183.stdout: "up": 1, 2024-09-10T19:19:34.077 INFO:teuthology.orchestra.run.smithi183.stdout: "in": 1, 2024-09-10T19:19:34.077 INFO:teuthology.orchestra.run.smithi183.stdout: "weight": 1, 2024-09-10T19:19:34.077 INFO:teuthology.orchestra.run.smithi183.stdout: "primary_affinity": 1, 2024-09-10T19:19:34.077 INFO:teuthology.orchestra.run.smithi183.stdout: "last_clean_begin": 0, 2024-09-10T19:19:34.077 INFO:teuthology.orchestra.run.smithi183.stdout: "last_clean_end": 0, 2024-09-10T19:19:34.077 INFO:teuthology.orchestra.run.smithi183.stdout: "up_from": 26, 2024-09-10T19:19:34.077 INFO:teuthology.orchestra.run.smithi183.stdout: "up_thru": 211, 2024-09-10T19:19:34.078 INFO:teuthology.orchestra.run.smithi183.stdout: "down_at": 0, 2024-09-10T19:19:34.078 INFO:teuthology.orchestra.run.smithi183.stdout: "lost_at": 0, 2024-09-10T19:19:34.078 INFO:teuthology.orchestra.run.smithi183.stdout: "public_addrs": { 2024-09-10T19:19:34.078 INFO:teuthology.orchestra.run.smithi183.stdout: "addrvec": [ 2024-09-10T19:19:34.078 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:19:34.078 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v2", 2024-09-10T19:19:34.078 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.183:6826", 2024-09-10T19:19:34.078 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 3522836984 2024-09-10T19:19:34.078 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:19:34.078 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:19:34.078 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v1", 2024-09-10T19:19:34.078 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.183:6827", 2024-09-10T19:19:34.078 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 3522836984 2024-09-10T19:19:34.078 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:19:34.079 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:19:34.079 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:19:34.079 INFO:teuthology.orchestra.run.smithi183.stdout: "cluster_addrs": { 2024-09-10T19:19:34.079 INFO:teuthology.orchestra.run.smithi183.stdout: "addrvec": [ 2024-09-10T19:19:34.079 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:19:34.079 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v2", 2024-09-10T19:19:34.079 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.183:6828", 2024-09-10T19:19:34.079 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 3522836984 2024-09-10T19:19:34.079 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:19:34.079 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:19:34.079 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v1", 2024-09-10T19:19:34.079 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.183:6829", 2024-09-10T19:19:34.079 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 3522836984 2024-09-10T19:19:34.080 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:19:34.080 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:19:34.080 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:19:34.080 INFO:teuthology.orchestra.run.smithi183.stdout: "heartbeat_back_addrs": { 2024-09-10T19:19:34.080 INFO:teuthology.orchestra.run.smithi183.stdout: "addrvec": [ 2024-09-10T19:19:34.080 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:19:34.080 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v2", 2024-09-10T19:19:34.080 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.183:6832", 2024-09-10T19:19:34.080 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 3522836984 2024-09-10T19:19:34.081 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:19:34.081 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:19:34.081 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v1", 2024-09-10T19:19:34.081 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.183:6833", 2024-09-10T19:19:34.081 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 3522836984 2024-09-10T19:19:34.081 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:19:34.081 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:19:34.081 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:19:34.081 INFO:teuthology.orchestra.run.smithi183.stdout: "heartbeat_front_addrs": { 2024-09-10T19:19:34.081 INFO:teuthology.orchestra.run.smithi183.stdout: "addrvec": [ 2024-09-10T19:19:34.081 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:19:34.081 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v2", 2024-09-10T19:19:34.081 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.183:6830", 2024-09-10T19:19:34.081 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 3522836984 2024-09-10T19:19:34.082 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:19:34.082 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:19:34.082 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v1", 2024-09-10T19:19:34.082 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.183:6831", 2024-09-10T19:19:34.082 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 3522836984 2024-09-10T19:19:34.082 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:19:34.082 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:19:34.082 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:19:34.082 INFO:teuthology.orchestra.run.smithi183.stdout: "public_addr": "172.21.15.183:6827/3522836984", 2024-09-10T19:19:34.082 INFO:teuthology.orchestra.run.smithi183.stdout: "cluster_addr": "172.21.15.183:6829/3522836984", 2024-09-10T19:19:34.082 INFO:teuthology.orchestra.run.smithi183.stdout: "heartbeat_back_addr": "172.21.15.183:6833/3522836984", 2024-09-10T19:19:34.082 INFO:teuthology.orchestra.run.smithi183.stdout: "heartbeat_front_addr": "172.21.15.183:6831/3522836984", 2024-09-10T19:19:34.082 INFO:teuthology.orchestra.run.smithi183.stdout: "state": [ 2024-09-10T19:19:34.082 INFO:teuthology.orchestra.run.smithi183.stdout: "exists", 2024-09-10T19:19:34.083 INFO:teuthology.orchestra.run.smithi183.stdout: "up" 2024-09-10T19:19:34.083 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:19:34.083 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:19:34.083 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:19:34.083 INFO:teuthology.orchestra.run.smithi183.stdout: "osd": 4, 2024-09-10T19:19:34.083 INFO:teuthology.orchestra.run.smithi183.stdout: "uuid": "631083fb-e503-44da-9b6b-154e88539ee7", 2024-09-10T19:19:34.083 INFO:teuthology.orchestra.run.smithi183.stdout: "up": 1, 2024-09-10T19:19:34.083 INFO:teuthology.orchestra.run.smithi183.stdout: "in": 1, 2024-09-10T19:19:34.083 INFO:teuthology.orchestra.run.smithi183.stdout: "weight": 1, 2024-09-10T19:19:34.083 INFO:teuthology.orchestra.run.smithi183.stdout: "primary_affinity": 1, 2024-09-10T19:19:34.083 INFO:teuthology.orchestra.run.smithi183.stdout: "last_clean_begin": 0, 2024-09-10T19:19:34.083 INFO:teuthology.orchestra.run.smithi183.stdout: "last_clean_end": 0, 2024-09-10T19:19:34.083 INFO:teuthology.orchestra.run.smithi183.stdout: "up_from": 32, 2024-09-10T19:19:34.083 INFO:teuthology.orchestra.run.smithi183.stdout: "up_thru": 189, 2024-09-10T19:19:34.084 INFO:teuthology.orchestra.run.smithi183.stdout: "down_at": 0, 2024-09-10T19:19:34.084 INFO:teuthology.orchestra.run.smithi183.stdout: "lost_at": 0, 2024-09-10T19:19:34.084 INFO:teuthology.orchestra.run.smithi183.stdout: "public_addrs": { 2024-09-10T19:19:34.084 INFO:teuthology.orchestra.run.smithi183.stdout: "addrvec": [ 2024-09-10T19:19:34.084 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:19:34.084 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v2", 2024-09-10T19:19:34.084 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.191:6800", 2024-09-10T19:19:34.084 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 2674828539 2024-09-10T19:19:34.084 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:19:34.084 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:19:34.084 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v1", 2024-09-10T19:19:34.084 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.191:6801", 2024-09-10T19:19:34.084 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 2674828539 2024-09-10T19:19:34.084 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:19:34.085 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:19:34.085 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:19:34.085 INFO:teuthology.orchestra.run.smithi183.stdout: "cluster_addrs": { 2024-09-10T19:19:34.085 INFO:teuthology.orchestra.run.smithi183.stdout: "addrvec": [ 2024-09-10T19:19:34.085 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:19:34.085 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v2", 2024-09-10T19:19:34.085 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.191:6802", 2024-09-10T19:19:34.085 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 2674828539 2024-09-10T19:19:34.085 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:19:34.085 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:19:34.085 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v1", 2024-09-10T19:19:34.085 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.191:6803", 2024-09-10T19:19:34.085 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 2674828539 2024-09-10T19:19:34.085 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:19:34.086 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:19:34.086 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:19:34.086 INFO:teuthology.orchestra.run.smithi183.stdout: "heartbeat_back_addrs": { 2024-09-10T19:19:34.086 INFO:teuthology.orchestra.run.smithi183.stdout: "addrvec": [ 2024-09-10T19:19:34.086 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:19:34.086 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v2", 2024-09-10T19:19:34.086 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.191:6806", 2024-09-10T19:19:34.086 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 2674828539 2024-09-10T19:19:34.086 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:19:34.086 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:19:34.086 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v1", 2024-09-10T19:19:34.086 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.191:6807", 2024-09-10T19:19:34.086 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 2674828539 2024-09-10T19:19:34.086 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:19:34.087 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:19:34.087 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:19:34.087 INFO:teuthology.orchestra.run.smithi183.stdout: "heartbeat_front_addrs": { 2024-09-10T19:19:34.087 INFO:teuthology.orchestra.run.smithi183.stdout: "addrvec": [ 2024-09-10T19:19:34.087 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:19:34.087 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v2", 2024-09-10T19:19:34.087 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.191:6804", 2024-09-10T19:19:34.087 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 2674828539 2024-09-10T19:19:34.087 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:19:34.087 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:19:34.087 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v1", 2024-09-10T19:19:34.087 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.191:6805", 2024-09-10T19:19:34.087 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 2674828539 2024-09-10T19:19:34.087 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:19:34.087 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:19:34.088 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:19:34.088 INFO:teuthology.orchestra.run.smithi183.stdout: "public_addr": "172.21.15.191:6801/2674828539", 2024-09-10T19:19:34.088 INFO:teuthology.orchestra.run.smithi183.stdout: "cluster_addr": "172.21.15.191:6803/2674828539", 2024-09-10T19:19:34.088 INFO:teuthology.orchestra.run.smithi183.stdout: "heartbeat_back_addr": "172.21.15.191:6807/2674828539", 2024-09-10T19:19:34.088 INFO:teuthology.orchestra.run.smithi183.stdout: "heartbeat_front_addr": "172.21.15.191:6805/2674828539", 2024-09-10T19:19:34.088 INFO:teuthology.orchestra.run.smithi183.stdout: "state": [ 2024-09-10T19:19:34.088 INFO:teuthology.orchestra.run.smithi183.stdout: "exists", 2024-09-10T19:19:34.088 INFO:teuthology.orchestra.run.smithi183.stdout: "up" 2024-09-10T19:19:34.088 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:19:34.088 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:19:34.088 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:19:34.088 INFO:teuthology.orchestra.run.smithi183.stdout: "osd": 5, 2024-09-10T19:19:34.088 INFO:teuthology.orchestra.run.smithi183.stdout: "uuid": "6b963695-3eca-4e6e-9207-05b8010ebff9", 2024-09-10T19:19:34.089 INFO:teuthology.orchestra.run.smithi183.stdout: "up": 1, 2024-09-10T19:19:34.089 INFO:teuthology.orchestra.run.smithi183.stdout: "in": 1, 2024-09-10T19:19:34.089 INFO:teuthology.orchestra.run.smithi183.stdout: "weight": 1, 2024-09-10T19:19:34.089 INFO:teuthology.orchestra.run.smithi183.stdout: "primary_affinity": 1, 2024-09-10T19:19:34.089 INFO:teuthology.orchestra.run.smithi183.stdout: "last_clean_begin": 0, 2024-09-10T19:19:34.089 INFO:teuthology.orchestra.run.smithi183.stdout: "last_clean_end": 0, 2024-09-10T19:19:34.089 INFO:teuthology.orchestra.run.smithi183.stdout: "up_from": 39, 2024-09-10T19:19:34.089 INFO:teuthology.orchestra.run.smithi183.stdout: "up_thru": 217, 2024-09-10T19:19:34.089 INFO:teuthology.orchestra.run.smithi183.stdout: "down_at": 0, 2024-09-10T19:19:34.089 INFO:teuthology.orchestra.run.smithi183.stdout: "lost_at": 0, 2024-09-10T19:19:34.089 INFO:teuthology.orchestra.run.smithi183.stdout: "public_addrs": { 2024-09-10T19:19:34.089 INFO:teuthology.orchestra.run.smithi183.stdout: "addrvec": [ 2024-09-10T19:19:34.089 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:19:34.089 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v2", 2024-09-10T19:19:34.089 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.191:6808", 2024-09-10T19:19:34.090 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 681733378 2024-09-10T19:19:34.090 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:19:34.090 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:19:34.090 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v1", 2024-09-10T19:19:34.090 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.191:6809", 2024-09-10T19:19:34.090 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 681733378 2024-09-10T19:19:34.090 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:19:34.090 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:19:34.090 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:19:34.090 INFO:teuthology.orchestra.run.smithi183.stdout: "cluster_addrs": { 2024-09-10T19:19:34.090 INFO:teuthology.orchestra.run.smithi183.stdout: "addrvec": [ 2024-09-10T19:19:34.090 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:19:34.090 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v2", 2024-09-10T19:19:34.091 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:19:34.092 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.191:6810", 2024-09-10T19:19:34.092 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 681733378 2024-09-10T19:19:34.092 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:19:34.092 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:19:34.092 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v1", 2024-09-10T19:19:34.092 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.191:6811", 2024-09-10T19:19:34.092 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 681733378 2024-09-10T19:19:34.093 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:19:34.093 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:19:34.093 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:19:34.093 INFO:teuthology.orchestra.run.smithi183.stdout: "heartbeat_back_addrs": { 2024-09-10T19:19:34.093 INFO:teuthology.orchestra.run.smithi183.stdout: "addrvec": [ 2024-09-10T19:19:34.093 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:19:34.093 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v2", 2024-09-10T19:19:34.093 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.191:6814", 2024-09-10T19:19:34.093 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 681733378 2024-09-10T19:19:34.093 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:19:34.093 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:19:34.093 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v1", 2024-09-10T19:19:34.093 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.191:6815", 2024-09-10T19:19:34.093 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 681733378 2024-09-10T19:19:34.094 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:19:34.094 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:19:34.094 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:19:34.094 INFO:teuthology.orchestra.run.smithi183.stdout: "heartbeat_front_addrs": { 2024-09-10T19:19:34.094 INFO:teuthology.orchestra.run.smithi183.stdout: "addrvec": [ 2024-09-10T19:19:34.094 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:19:34.094 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v2", 2024-09-10T19:19:34.094 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.191:6812", 2024-09-10T19:19:34.094 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 681733378 2024-09-10T19:19:34.094 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:19:34.094 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:19:34.094 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v1", 2024-09-10T19:19:34.094 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.191:6813", 2024-09-10T19:19:34.094 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 681733378 2024-09-10T19:19:34.095 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:19:34.095 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:19:34.095 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:19:34.095 INFO:teuthology.orchestra.run.smithi183.stdout: "public_addr": "172.21.15.191:6809/681733378", 2024-09-10T19:19:34.095 INFO:teuthology.orchestra.run.smithi183.stdout: "cluster_addr": "172.21.15.191:6811/681733378", 2024-09-10T19:19:34.095 INFO:teuthology.orchestra.run.smithi183.stdout: "heartbeat_back_addr": "172.21.15.191:6815/681733378", 2024-09-10T19:19:34.095 INFO:teuthology.orchestra.run.smithi183.stdout: "heartbeat_front_addr": "172.21.15.191:6813/681733378", 2024-09-10T19:19:34.095 INFO:teuthology.orchestra.run.smithi183.stdout: "state": [ 2024-09-10T19:19:34.095 INFO:teuthology.orchestra.run.smithi183.stdout: "exists", 2024-09-10T19:19:34.095 INFO:teuthology.orchestra.run.smithi183.stdout: "up" 2024-09-10T19:19:34.095 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:19:34.095 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:19:34.095 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:19:34.096 INFO:teuthology.orchestra.run.smithi183.stdout: "osd": 6, 2024-09-10T19:19:34.096 INFO:teuthology.orchestra.run.smithi183.stdout: "uuid": "2dc9754f-6aef-4e19-8605-2f1591921811", 2024-09-10T19:19:34.096 INFO:teuthology.orchestra.run.smithi183.stdout: "up": 1, 2024-09-10T19:19:34.096 INFO:teuthology.orchestra.run.smithi183.stdout: "in": 1, 2024-09-10T19:19:34.096 INFO:teuthology.orchestra.run.smithi183.stdout: "weight": 1, 2024-09-10T19:19:34.096 INFO:teuthology.orchestra.run.smithi183.stdout: "primary_affinity": 1, 2024-09-10T19:19:34.096 INFO:teuthology.orchestra.run.smithi183.stdout: "last_clean_begin": 0, 2024-09-10T19:19:34.096 INFO:teuthology.orchestra.run.smithi183.stdout: "last_clean_end": 0, 2024-09-10T19:19:34.096 INFO:teuthology.orchestra.run.smithi183.stdout: "up_from": 47, 2024-09-10T19:19:34.096 INFO:teuthology.orchestra.run.smithi183.stdout: "up_thru": 179, 2024-09-10T19:19:34.096 INFO:teuthology.orchestra.run.smithi183.stdout: "down_at": 0, 2024-09-10T19:19:34.096 INFO:teuthology.orchestra.run.smithi183.stdout: "lost_at": 0, 2024-09-10T19:19:34.096 INFO:teuthology.orchestra.run.smithi183.stdout: "public_addrs": { 2024-09-10T19:19:34.096 INFO:teuthology.orchestra.run.smithi183.stdout: "addrvec": [ 2024-09-10T19:19:34.097 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:19:34.097 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v2", 2024-09-10T19:19:34.097 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.191:6816", 2024-09-10T19:19:34.097 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 2825813477 2024-09-10T19:19:34.097 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:19:34.097 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:19:34.097 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v1", 2024-09-10T19:19:34.097 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.191:6817", 2024-09-10T19:19:34.097 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 2825813477 2024-09-10T19:19:34.097 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:19:34.097 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:19:34.097 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:19:34.097 INFO:teuthology.orchestra.run.smithi183.stdout: "cluster_addrs": { 2024-09-10T19:19:34.098 INFO:teuthology.orchestra.run.smithi183.stdout: "addrvec": [ 2024-09-10T19:19:34.098 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:19:34.098 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v2", 2024-09-10T19:19:34.098 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.191:6818", 2024-09-10T19:19:34.098 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 2825813477 2024-09-10T19:19:34.098 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:19:34.098 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:19:34.098 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v1", 2024-09-10T19:19:34.098 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.191:6819", 2024-09-10T19:19:34.098 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 2825813477 2024-09-10T19:19:34.098 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:19:34.098 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:19:34.098 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:19:34.098 INFO:teuthology.orchestra.run.smithi183.stdout: "heartbeat_back_addrs": { 2024-09-10T19:19:34.099 INFO:teuthology.orchestra.run.smithi183.stdout: "addrvec": [ 2024-09-10T19:19:34.099 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:19:34.099 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v2", 2024-09-10T19:19:34.099 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.191:6822", 2024-09-10T19:19:34.099 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 2825813477 2024-09-10T19:19:34.099 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:19:34.099 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:19:34.099 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v1", 2024-09-10T19:19:34.099 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.191:6823", 2024-09-10T19:19:34.099 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 2825813477 2024-09-10T19:19:34.099 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:19:34.099 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:19:34.099 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:19:34.099 INFO:teuthology.orchestra.run.smithi183.stdout: "heartbeat_front_addrs": { 2024-09-10T19:19:34.099 INFO:teuthology.orchestra.run.smithi183.stdout: "addrvec": [ 2024-09-10T19:19:34.100 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:19:34.100 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v2", 2024-09-10T19:19:34.100 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.191:6820", 2024-09-10T19:19:34.100 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 2825813477 2024-09-10T19:19:34.100 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:19:34.100 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:19:34.100 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v1", 2024-09-10T19:19:34.100 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.191:6821", 2024-09-10T19:19:34.100 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 2825813477 2024-09-10T19:19:34.100 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:19:34.100 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:19:34.100 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:19:34.100 INFO:teuthology.orchestra.run.smithi183.stdout: "public_addr": "172.21.15.191:6817/2825813477", 2024-09-10T19:19:34.100 INFO:teuthology.orchestra.run.smithi183.stdout: "cluster_addr": "172.21.15.191:6819/2825813477", 2024-09-10T19:19:34.101 INFO:teuthology.orchestra.run.smithi183.stdout: "heartbeat_back_addr": "172.21.15.191:6823/2825813477", 2024-09-10T19:19:34.101 INFO:teuthology.orchestra.run.smithi183.stdout: "heartbeat_front_addr": "172.21.15.191:6821/2825813477", 2024-09-10T19:19:34.101 INFO:teuthology.orchestra.run.smithi183.stdout: "state": [ 2024-09-10T19:19:34.101 INFO:teuthology.orchestra.run.smithi183.stdout: "exists", 2024-09-10T19:19:34.101 INFO:teuthology.orchestra.run.smithi183.stdout: "up" 2024-09-10T19:19:34.101 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:19:34.101 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:19:34.101 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:19:34.101 INFO:teuthology.orchestra.run.smithi183.stdout: "osd": 7, 2024-09-10T19:19:34.101 INFO:teuthology.orchestra.run.smithi183.stdout: "uuid": "1a9a299e-d5ba-4b9a-85d5-7f6a5bc754e1", 2024-09-10T19:19:34.101 INFO:teuthology.orchestra.run.smithi183.stdout: "up": 1, 2024-09-10T19:19:34.101 INFO:teuthology.orchestra.run.smithi183.stdout: "in": 1, 2024-09-10T19:19:34.101 INFO:teuthology.orchestra.run.smithi183.stdout: "weight": 1, 2024-09-10T19:19:34.101 INFO:teuthology.orchestra.run.smithi183.stdout: "primary_affinity": 1, 2024-09-10T19:19:34.102 INFO:teuthology.orchestra.run.smithi183.stdout: "last_clean_begin": 0, 2024-09-10T19:19:34.102 INFO:teuthology.orchestra.run.smithi183.stdout: "last_clean_end": 0, 2024-09-10T19:19:34.102 INFO:teuthology.orchestra.run.smithi183.stdout: "up_from": 54, 2024-09-10T19:19:34.102 INFO:teuthology.orchestra.run.smithi183.stdout: "up_thru": 206, 2024-09-10T19:19:34.102 INFO:teuthology.orchestra.run.smithi183.stdout: "down_at": 0, 2024-09-10T19:19:34.102 INFO:teuthology.orchestra.run.smithi183.stdout: "lost_at": 0, 2024-09-10T19:19:34.102 INFO:teuthology.orchestra.run.smithi183.stdout: "public_addrs": { 2024-09-10T19:19:34.102 INFO:teuthology.orchestra.run.smithi183.stdout: "addrvec": [ 2024-09-10T19:19:34.102 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:19:34.102 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v2", 2024-09-10T19:19:34.102 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.191:6824", 2024-09-10T19:19:34.102 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 1422610021 2024-09-10T19:19:34.102 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:19:34.102 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:19:34.102 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v1", 2024-09-10T19:19:34.103 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.191:6825", 2024-09-10T19:19:34.103 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 1422610021 2024-09-10T19:19:34.103 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:19:34.103 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:19:34.103 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:19:34.103 INFO:teuthology.orchestra.run.smithi183.stdout: "cluster_addrs": { 2024-09-10T19:19:34.103 INFO:teuthology.orchestra.run.smithi183.stdout: "addrvec": [ 2024-09-10T19:19:34.103 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:19:34.103 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v2", 2024-09-10T19:19:34.103 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.191:6826", 2024-09-10T19:19:34.103 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 1422610021 2024-09-10T19:19:34.103 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:19:34.103 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:19:34.103 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v1", 2024-09-10T19:19:34.104 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.191:6827", 2024-09-10T19:19:34.104 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 1422610021 2024-09-10T19:19:34.104 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:19:34.104 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:19:34.104 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:19:34.104 INFO:teuthology.orchestra.run.smithi183.stdout: "heartbeat_back_addrs": { 2024-09-10T19:19:34.104 INFO:teuthology.orchestra.run.smithi183.stdout: "addrvec": [ 2024-09-10T19:19:34.104 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:19:34.104 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v2", 2024-09-10T19:19:34.104 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.191:6830", 2024-09-10T19:19:34.104 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 1422610021 2024-09-10T19:19:34.104 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:19:34.104 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:19:34.105 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v1", 2024-09-10T19:19:34.105 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.191:6831", 2024-09-10T19:19:34.105 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 1422610021 2024-09-10T19:19:34.105 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:19:34.105 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:19:34.105 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:19:34.105 INFO:teuthology.orchestra.run.smithi183.stdout: "heartbeat_front_addrs": { 2024-09-10T19:19:34.105 INFO:teuthology.orchestra.run.smithi183.stdout: "addrvec": [ 2024-09-10T19:19:34.105 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:19:34.105 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v2", 2024-09-10T19:19:34.105 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.191:6828", 2024-09-10T19:19:34.105 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 1422610021 2024-09-10T19:19:34.105 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:19:34.105 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:19:34.106 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v1", 2024-09-10T19:19:34.106 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.191:6829", 2024-09-10T19:19:34.106 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 1422610021 2024-09-10T19:19:34.106 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:19:34.106 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:19:34.106 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:19:34.106 INFO:teuthology.orchestra.run.smithi183.stdout: "public_addr": "172.21.15.191:6825/1422610021", 2024-09-10T19:19:34.106 INFO:teuthology.orchestra.run.smithi183.stdout: "cluster_addr": "172.21.15.191:6827/1422610021", 2024-09-10T19:19:34.106 INFO:teuthology.orchestra.run.smithi183.stdout: "heartbeat_back_addr": "172.21.15.191:6831/1422610021", 2024-09-10T19:19:34.106 INFO:teuthology.orchestra.run.smithi183.stdout: "heartbeat_front_addr": "172.21.15.191:6829/1422610021", 2024-09-10T19:19:34.106 INFO:teuthology.orchestra.run.smithi183.stdout: "state": [ 2024-09-10T19:19:34.106 INFO:teuthology.orchestra.run.smithi183.stdout: "exists", 2024-09-10T19:19:34.106 INFO:teuthology.orchestra.run.smithi183.stdout: "up" 2024-09-10T19:19:34.106 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:19:34.107 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:19:34.107 INFO:teuthology.orchestra.run.smithi183.stdout: ], 2024-09-10T19:19:34.107 INFO:teuthology.orchestra.run.smithi183.stdout: "osd_xinfo": [ 2024-09-10T19:19:34.107 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:19:34.107 INFO:teuthology.orchestra.run.smithi183.stdout: "osd": 0, 2024-09-10T19:19:34.107 INFO:teuthology.orchestra.run.smithi183.stdout: "down_stamp": "0.000000", 2024-09-10T19:19:34.107 INFO:teuthology.orchestra.run.smithi183.stdout: "laggy_probability": 0, 2024-09-10T19:19:34.107 INFO:teuthology.orchestra.run.smithi183.stdout: "laggy_interval": 0, 2024-09-10T19:19:34.107 INFO:teuthology.orchestra.run.smithi183.stdout: "features": 4540138320759226367, 2024-09-10T19:19:34.107 INFO:teuthology.orchestra.run.smithi183.stdout: "old_weight": 0, 2024-09-10T19:19:34.107 INFO:teuthology.orchestra.run.smithi183.stdout: "last_purged_snaps_scrub": "2024-09-10T19:19:08.635604+0000", 2024-09-10T19:19:34.107 INFO:teuthology.orchestra.run.smithi183.stdout: "dead_epoch": 0 2024-09-10T19:19:34.107 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:19:34.107 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:19:34.107 INFO:teuthology.orchestra.run.smithi183.stdout: "osd": 1, 2024-09-10T19:19:34.108 INFO:teuthology.orchestra.run.smithi183.stdout: "down_stamp": "0.000000", 2024-09-10T19:19:34.108 INFO:teuthology.orchestra.run.smithi183.stdout: "laggy_probability": 0, 2024-09-10T19:19:34.108 INFO:teuthology.orchestra.run.smithi183.stdout: "laggy_interval": 0, 2024-09-10T19:19:34.108 INFO:teuthology.orchestra.run.smithi183.stdout: "features": 4540138320759226367, 2024-09-10T19:19:34.108 INFO:teuthology.orchestra.run.smithi183.stdout: "old_weight": 65536, 2024-09-10T19:19:34.108 INFO:teuthology.orchestra.run.smithi183.stdout: "last_purged_snaps_scrub": "2024-09-10T19:19:22.837989+0000", 2024-09-10T19:19:34.108 INFO:teuthology.orchestra.run.smithi183.stdout: "dead_epoch": 98 2024-09-10T19:19:34.108 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:19:34.108 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:19:34.108 INFO:teuthology.orchestra.run.smithi183.stdout: "osd": 2, 2024-09-10T19:19:34.108 INFO:teuthology.orchestra.run.smithi183.stdout: "down_stamp": "0.000000", 2024-09-10T19:19:34.108 INFO:teuthology.orchestra.run.smithi183.stdout: "laggy_probability": 0, 2024-09-10T19:19:34.108 INFO:teuthology.orchestra.run.smithi183.stdout: "laggy_interval": 0, 2024-09-10T19:19:34.108 INFO:teuthology.orchestra.run.smithi183.stdout: "features": 4540138320759226367, 2024-09-10T19:19:34.109 INFO:teuthology.orchestra.run.smithi183.stdout: "old_weight": 65536, 2024-09-10T19:19:34.109 INFO:teuthology.orchestra.run.smithi183.stdout: "last_purged_snaps_scrub": "2024-09-10T19:18:18.931307+0000", 2024-09-10T19:19:34.109 INFO:teuthology.orchestra.run.smithi183.stdout: "dead_epoch": 0 2024-09-10T19:19:34.109 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:19:34.109 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:19:34.109 INFO:teuthology.orchestra.run.smithi183.stdout: "osd": 3, 2024-09-10T19:19:34.109 INFO:teuthology.orchestra.run.smithi183.stdout: "down_stamp": "0.000000", 2024-09-10T19:19:34.109 INFO:teuthology.orchestra.run.smithi183.stdout: "laggy_probability": 0, 2024-09-10T19:19:34.109 INFO:teuthology.orchestra.run.smithi183.stdout: "laggy_interval": 0, 2024-09-10T19:19:34.109 INFO:teuthology.orchestra.run.smithi183.stdout: "features": 4540138320759226367, 2024-09-10T19:19:34.109 INFO:teuthology.orchestra.run.smithi183.stdout: "old_weight": 0, 2024-09-10T19:19:34.109 INFO:teuthology.orchestra.run.smithi183.stdout: "last_purged_snaps_scrub": "2024-09-10T19:18:34.335087+0000", 2024-09-10T19:19:34.109 INFO:teuthology.orchestra.run.smithi183.stdout: "dead_epoch": 0 2024-09-10T19:19:34.109 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:19:34.110 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:19:34.110 INFO:teuthology.orchestra.run.smithi183.stdout: "osd": 4, 2024-09-10T19:19:34.110 INFO:teuthology.orchestra.run.smithi183.stdout: "down_stamp": "0.000000", 2024-09-10T19:19:34.110 INFO:teuthology.orchestra.run.smithi183.stdout: "laggy_probability": 0, 2024-09-10T19:19:34.110 INFO:teuthology.orchestra.run.smithi183.stdout: "laggy_interval": 0, 2024-09-10T19:19:34.110 INFO:teuthology.orchestra.run.smithi183.stdout: "features": 4540138320759226367, 2024-09-10T19:19:34.110 INFO:teuthology.orchestra.run.smithi183.stdout: "old_weight": 0, 2024-09-10T19:19:34.110 INFO:teuthology.orchestra.run.smithi183.stdout: "last_purged_snaps_scrub": "2024-09-10T19:18:46.652022+0000", 2024-09-10T19:19:34.110 INFO:teuthology.orchestra.run.smithi183.stdout: "dead_epoch": 0 2024-09-10T19:19:34.110 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:19:34.110 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:19:34.110 INFO:teuthology.orchestra.run.smithi183.stdout: "osd": 5, 2024-09-10T19:19:34.110 INFO:teuthology.orchestra.run.smithi183.stdout: "down_stamp": "0.000000", 2024-09-10T19:19:34.110 INFO:teuthology.orchestra.run.smithi183.stdout: "laggy_probability": 0, 2024-09-10T19:19:34.111 INFO:teuthology.orchestra.run.smithi183.stdout: "laggy_interval": 0, 2024-09-10T19:19:34.111 INFO:teuthology.orchestra.run.smithi183.stdout: "features": 4540138320759226367, 2024-09-10T19:19:34.111 INFO:teuthology.orchestra.run.smithi183.stdout: "old_weight": 0, 2024-09-10T19:19:34.111 INFO:teuthology.orchestra.run.smithi183.stdout: "last_purged_snaps_scrub": "2024-09-10T19:18:50.000086+0000", 2024-09-10T19:19:34.111 INFO:teuthology.orchestra.run.smithi183.stdout: "dead_epoch": 0 2024-09-10T19:19:34.111 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:19:34.111 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:19:34.111 INFO:teuthology.orchestra.run.smithi183.stdout: "osd": 6, 2024-09-10T19:19:34.111 INFO:teuthology.orchestra.run.smithi183.stdout: "down_stamp": "0.000000", 2024-09-10T19:19:34.111 INFO:teuthology.orchestra.run.smithi183.stdout: "laggy_probability": 0, 2024-09-10T19:19:34.111 INFO:teuthology.orchestra.run.smithi183.stdout: "laggy_interval": 0, 2024-09-10T19:19:34.111 INFO:teuthology.orchestra.run.smithi183.stdout: "features": 4540138320759226367, 2024-09-10T19:19:34.111 INFO:teuthology.orchestra.run.smithi183.stdout: "old_weight": 0, 2024-09-10T19:19:34.111 INFO:teuthology.orchestra.run.smithi183.stdout: "last_purged_snaps_scrub": "2024-09-10T19:18:32.489412+0000", 2024-09-10T19:19:34.112 INFO:teuthology.orchestra.run.smithi183.stdout: "dead_epoch": 0 2024-09-10T19:19:34.112 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:19:34.112 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:19:34.112 INFO:teuthology.orchestra.run.smithi183.stdout: "osd": 7, 2024-09-10T19:19:34.112 INFO:teuthology.orchestra.run.smithi183.stdout: "down_stamp": "0.000000", 2024-09-10T19:19:34.112 INFO:teuthology.orchestra.run.smithi183.stdout: "laggy_probability": 0, 2024-09-10T19:19:34.112 INFO:teuthology.orchestra.run.smithi183.stdout: "laggy_interval": 0, 2024-09-10T19:19:34.112 INFO:teuthology.orchestra.run.smithi183.stdout: "features": 4540138320759226367, 2024-09-10T19:19:34.112 INFO:teuthology.orchestra.run.smithi183.stdout: "old_weight": 0, 2024-09-10T19:19:34.112 INFO:teuthology.orchestra.run.smithi183.stdout: "last_purged_snaps_scrub": "2024-09-10T19:18:58.918466+0000", 2024-09-10T19:19:34.112 INFO:teuthology.orchestra.run.smithi183.stdout: "dead_epoch": 0 2024-09-10T19:19:34.112 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:19:34.112 INFO:teuthology.orchestra.run.smithi183.stdout: ], 2024-09-10T19:19:34.113 INFO:teuthology.orchestra.run.smithi183.stdout: "pg_upmap": [ 2024-09-10T19:19:34.113 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:19:34.113 INFO:teuthology.orchestra.run.smithi183.stdout: "pgid": "2.13", 2024-09-10T19:19:34.113 INFO:teuthology.orchestra.run.smithi183.stdout: "osds": [ 2024-09-10T19:19:34.113 INFO:teuthology.orchestra.run.smithi183.stdout: 2, 2024-09-10T19:19:34.113 INFO:teuthology.orchestra.run.smithi183.stdout: 0, 2024-09-10T19:19:34.113 INFO:teuthology.orchestra.run.smithi183.stdout: 5 2024-09-10T19:19:34.113 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:19:34.113 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:19:34.113 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:19:34.113 INFO:teuthology.orchestra.run.smithi183.stdout: "pgid": "8.3", 2024-09-10T19:19:34.113 INFO:teuthology.orchestra.run.smithi183.stdout: "osds": [ 2024-09-10T19:19:34.113 INFO:teuthology.orchestra.run.smithi183.stdout: 7, 2024-09-10T19:19:34.114 INFO:teuthology.orchestra.run.smithi183.stdout: 3, 2024-09-10T19:19:34.114 INFO:teuthology.orchestra.run.smithi183.stdout: 5 2024-09-10T19:19:34.114 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:19:34.114 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:19:34.114 INFO:teuthology.orchestra.run.smithi183.stdout: ], 2024-09-10T19:19:34.114 INFO:teuthology.orchestra.run.smithi183.stdout: "pg_upmap_items": [ 2024-09-10T19:19:34.114 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:19:34.114 INFO:teuthology.orchestra.run.smithi183.stdout: "pgid": "2.18", 2024-09-10T19:19:34.114 INFO:teuthology.orchestra.run.smithi183.stdout: "mappings": [ 2024-09-10T19:19:34.114 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:19:34.114 INFO:teuthology.orchestra.run.smithi183.stdout: "from": 7, 2024-09-10T19:19:34.114 INFO:teuthology.orchestra.run.smithi183.stdout: "to": 0 2024-09-10T19:19:34.114 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:19:34.114 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:19:34.114 INFO:teuthology.orchestra.run.smithi183.stdout: "from": 5, 2024-09-10T19:19:34.115 INFO:teuthology.orchestra.run.smithi183.stdout: "to": 4 2024-09-10T19:19:34.115 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:19:34.115 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:19:34.115 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:19:34.115 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:19:34.115 INFO:teuthology.orchestra.run.smithi183.stdout: "pgid": "4.1", 2024-09-10T19:19:34.115 INFO:teuthology.orchestra.run.smithi183.stdout: "mappings": [ 2024-09-10T19:19:34.115 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:19:34.115 INFO:teuthology.orchestra.run.smithi183.stdout: "from": 6, 2024-09-10T19:19:34.115 INFO:teuthology.orchestra.run.smithi183.stdout: "to": 7 2024-09-10T19:19:34.115 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:19:34.115 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:19:34.115 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:19:34.115 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:19:34.116 INFO:teuthology.orchestra.run.smithi183.stdout: "pgid": "4.11", 2024-09-10T19:19:34.116 INFO:teuthology.orchestra.run.smithi183.stdout: "mappings": [ 2024-09-10T19:19:34.116 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:19:34.116 INFO:teuthology.orchestra.run.smithi183.stdout: "from": 3, 2024-09-10T19:19:34.116 INFO:teuthology.orchestra.run.smithi183.stdout: "to": 5 2024-09-10T19:19:34.116 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:19:34.116 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:19:34.116 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:19:34.116 INFO:teuthology.orchestra.run.smithi183.stdout: ], 2024-09-10T19:19:34.116 INFO:teuthology.orchestra.run.smithi183.stdout: "pg_temp": [ 2024-09-10T19:19:34.116 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:19:34.116 INFO:teuthology.orchestra.run.smithi183.stdout: "pgid": "8.12", 2024-09-10T19:19:34.116 INFO:teuthology.orchestra.run.smithi183.stdout: "osds": [ 2024-09-10T19:19:34.116 INFO:teuthology.orchestra.run.smithi183.stdout: 0, 2024-09-10T19:19:34.117 INFO:teuthology.orchestra.run.smithi183.stdout: 4, 2024-09-10T19:19:34.117 INFO:teuthology.orchestra.run.smithi183.stdout: 5 2024-09-10T19:19:34.117 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:19:34.117 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:19:34.117 INFO:teuthology.orchestra.run.smithi183.stdout: ], 2024-09-10T19:19:34.117 INFO:teuthology.orchestra.run.smithi183.stdout: "primary_temp": [], 2024-09-10T19:19:34.117 INFO:teuthology.orchestra.run.smithi183.stdout: "blocklist": { 2024-09-10T19:19:34.117 INFO:teuthology.orchestra.run.smithi183.stdout: "172.21.15.183:6801/3311783362": "2024-09-11T19:10:43.576943+0000", 2024-09-10T19:19:34.117 INFO:teuthology.orchestra.run.smithi183.stdout: "172.21.15.183:0/4088553922": "2024-09-11T19:10:43.576943+0000", 2024-09-10T19:19:34.117 INFO:teuthology.orchestra.run.smithi183.stdout: "172.21.15.183:0/3528769414": "2024-09-11T19:05:48.153879+0000", 2024-09-10T19:19:34.117 INFO:teuthology.orchestra.run.smithi183.stdout: "172.21.15.183:0/1766036658": "2024-09-11T19:05:48.153879+0000", 2024-09-10T19:19:34.117 INFO:teuthology.orchestra.run.smithi183.stdout: "172.21.15.183:0/3044170088": "2024-09-11T19:05:25.702354+0000", 2024-09-10T19:19:34.117 INFO:teuthology.orchestra.run.smithi183.stdout: "172.21.15.183:0/3037333778": "2024-09-11T19:05:25.702354+0000", 2024-09-10T19:19:34.117 INFO:teuthology.orchestra.run.smithi183.stdout: "172.21.15.183:6801/3747320570": "2024-09-11T19:05:25.702354+0000", 2024-09-10T19:19:34.118 INFO:teuthology.orchestra.run.smithi183.stdout: "172.21.15.183:0/3982386906": "2024-09-11T19:10:43.576943+0000", 2024-09-10T19:19:34.118 INFO:teuthology.orchestra.run.smithi183.stdout: "172.21.15.183:0/2436014455": "2024-09-11T19:05:25.702354+0000", 2024-09-10T19:19:34.118 INFO:teuthology.orchestra.run.smithi183.stdout: "172.21.15.183:6801/659745765": "2024-09-11T19:05:48.153879+0000", 2024-09-10T19:19:34.118 INFO:teuthology.orchestra.run.smithi183.stdout: "172.21.15.183:6800/3747320570": "2024-09-11T19:05:25.702354+0000", 2024-09-10T19:19:34.118 INFO:teuthology.orchestra.run.smithi183.stdout: "172.21.15.183:0/1652576282": "2024-09-11T19:10:43.576943+0000", 2024-09-10T19:19:34.118 INFO:teuthology.orchestra.run.smithi183.stdout: "172.21.15.183:6800/3311783362": "2024-09-11T19:10:43.576943+0000", 2024-09-10T19:19:34.118 INFO:teuthology.orchestra.run.smithi183.stdout: "172.21.15.183:0/1012899549": "2024-09-11T19:05:48.153879+0000", 2024-09-10T19:19:34.118 INFO:teuthology.orchestra.run.smithi183.stdout: "172.21.15.183:6800/659745765": "2024-09-11T19:05:48.153879+0000", 2024-09-10T19:19:34.118 INFO:teuthology.orchestra.run.smithi183.stdout: "172.21.15.183:0/412044400": "2024-09-11T19:10:43.576943+0000", 2024-09-10T19:19:34.118 INFO:teuthology.orchestra.run.smithi183.stdout: "172.21.15.183:0/3636171401": "2024-09-11T19:10:43.576943+0000" 2024-09-10T19:19:34.118 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:19:34.118 INFO:teuthology.orchestra.run.smithi183.stdout: "range_blocklist": {}, 2024-09-10T19:19:34.118 INFO:teuthology.orchestra.run.smithi183.stdout: "erasure_code_profiles": { 2024-09-10T19:19:34.119 INFO:teuthology.orchestra.run.smithi183.stdout: "default": { 2024-09-10T19:19:34.119 INFO:teuthology.orchestra.run.smithi183.stdout: "crush-failure-domain": "osd", 2024-09-10T19:19:34.119 INFO:teuthology.orchestra.run.smithi183.stdout: "k": "2", 2024-09-10T19:19:34.119 INFO:teuthology.orchestra.run.smithi183.stdout: "m": "1", 2024-09-10T19:19:34.119 INFO:teuthology.orchestra.run.smithi183.stdout: "plugin": "jerasure", 2024-09-10T19:19:34.119 INFO:teuthology.orchestra.run.smithi183.stdout: "technique": "reed_sol_van" 2024-09-10T19:19:34.119 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:19:34.119 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:19:34.119 INFO:teuthology.orchestra.run.smithi183.stdout: "removed_snaps_queue": [], 2024-09-10T19:19:34.119 INFO:teuthology.orchestra.run.smithi183.stdout: "new_removed_snaps": [], 2024-09-10T19:19:34.119 INFO:teuthology.orchestra.run.smithi183.stdout: "new_purged_snaps": [], 2024-09-10T19:19:34.119 INFO:teuthology.orchestra.run.smithi183.stdout: "crush_node_flags": {}, 2024-09-10T19:19:34.119 INFO:teuthology.orchestra.run.smithi183.stdout: "device_class_flags": {}, 2024-09-10T19:19:34.119 INFO:teuthology.orchestra.run.smithi183.stdout: "stretch_mode": { 2024-09-10T19:19:34.120 INFO:teuthology.orchestra.run.smithi183.stdout: "stretch_mode_enabled": false, 2024-09-10T19:19:34.120 INFO:teuthology.orchestra.run.smithi183.stdout: "stretch_bucket_count": 0, 2024-09-10T19:19:34.120 INFO:teuthology.orchestra.run.smithi183.stdout: "degraded_stretch_mode": 0, 2024-09-10T19:19:34.120 INFO:teuthology.orchestra.run.smithi183.stdout: "recovering_stretch_mode": 0, 2024-09-10T19:19:34.120 INFO:teuthology.orchestra.run.smithi183.stdout: "stretch_mode_bucket": 0 2024-09-10T19:19:34.120 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:19:34.120 INFO:teuthology.orchestra.run.smithi183.stdout:} 2024-09-10T19:19:34.125 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:19:34.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:34.126 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:19:34.227 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:19:34.255 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:19:34.255 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-10T19:19:34.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:34.257 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:19:34.357 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:19:34.388 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:19:34.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:34.389 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:19:34.490 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:19:34.511 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:19:34.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:34.512 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:19:34.612 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:19:34.645 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:19:34.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:34.646 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:19:34.747 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:19:34.783 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:19:34.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:34.784 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:19:34.885 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:19:34.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:34.921 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:19:34.921 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:19:34.931 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:19:34 smithi183 ceph-mon[29736]: pgmap v418: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 399 MiB data, 4.2 GiB used, 532 GiB / 536 GiB avail; 639 B/s rd, 0 B/s wr, 772 op/s; 400/19695 objects misplaced (2.031%) 2024-09-10T19:19:34.931 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:19:34 smithi183 ceph-mon[29736]: from='client.? 172.21.15.183:0/414160556' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-09-10T19:19:34.931 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:19:34 smithi183 ceph-mon[29736]: osdmap e224: 8 total, 8 up, 6 in 2024-09-10T19:19:34.931 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:19:34 smithi183 ceph-mon[41562]: pgmap v418: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 399 MiB data, 4.2 GiB used, 532 GiB / 536 GiB avail; 639 B/s rd, 0 B/s wr, 772 op/s; 400/19695 objects misplaced (2.031%) 2024-09-10T19:19:34.931 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:19:34 smithi183 ceph-mon[41562]: from='client.? 172.21.15.183:0/414160556' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-09-10T19:19:34.932 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:19:34 smithi183 ceph-mon[41562]: osdmap e224: 8 total, 8 up, 6 in 2024-09-10T19:19:34.953 INFO:tasks.radosbench.radosbench.0.smithi183.stdout:Removed 120280 objects 2024-09-10T19:19:34.953 INFO:tasks.radosbench.radosbench.0.smithi183.stdout:Clean up completed and total clean up time :99.1194 2024-09-10T19:19:34.965 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:19:34 smithi191 ceph-mon[32884]: pgmap v418: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 399 MiB data, 4.2 GiB used, 532 GiB / 536 GiB avail; 639 B/s rd, 0 B/s wr, 772 op/s; 400/19695 objects misplaced (2.031%) 2024-09-10T19:19:34.965 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:19:34 smithi191 ceph-mon[32884]: from='client.? 172.21.15.183:0/414160556' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-09-10T19:19:34.965 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:19:34 smithi191 ceph-mon[32884]: osdmap e224: 8 total, 8 up, 6 in 2024-09-10T19:19:35.022 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:19:35.044 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:19:35.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:35.048 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:19:35.149 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:19:35.171 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:19:35.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:35.171 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:19:35.272 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:19:35.294 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:19:35.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:35.295 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:19:35.395 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:19:35.418 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:19:35.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:35.419 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:19:35.519 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:19:35.541 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:19:35.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:35.542 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:19:35.643 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:19:35.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:35.664 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:19:35.665 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:19:35.765 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:19:35.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:35.791 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:19:35.791 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:19:35.892 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:19:35.902 INFO:teuthology.orchestra.run.smithi191.stdout:ERROR: (22) Invalid argument 2024-09-10T19:19:35.902 INFO:teuthology.orchestra.run.smithi191.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-10T19:19:35.914 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:19:35.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:35.915 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:19:36.015 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:19:36.037 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:19:36.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:36.038 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:19:36.138 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:19:36.160 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:19:36.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:36.161 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:19:36.262 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:19:36.284 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:19:36.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:36.285 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:19:36.385 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:19:36.402 INFO:tasks.thrashosds.thrasher:j is {'epoch': 223, 'fsid': '320f6df8-6fa7-11ef-bcea-c7b262605968', 'created': '2024-09-10T19:05:03.688531+0000', 'modified': '2024-09-10T19:19:30.813837+0000', 'last_up_change': '2024-09-10T19:15:19.923894+0000', 'last_in_change': '2024-09-10T19:16:23.815531+0000', 'flags': 'noscrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5801984, 'flags_set': ['noscrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 19, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 8, 'max_osd': 8, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'quincy', 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-10T19:08:00.456786+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '202', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}, {'pool': 2, 'pool_name': '.rgw.root', 'create_time': '2024-09-10T19:10:05.993763+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '60', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}}, {'pool': 3, 'pool_name': 'datapool', 'create_time': '2024-09-10T19:10:06.224267+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 3, 'pg_placement_num': 3, 'pg_placement_num_target': 3, 'pg_num_target': 3, 'pg_num_pending': 3, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '63', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 3, 'snap_epoch': 63, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 4, 'pool_name': 'default.rgw.log', 'create_time': '2024-09-10T19:10:07.958284+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '62', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}}, {'pool': 5, 'pool_name': 'default.rgw.control', 'create_time': '2024-09-10T19:10:09.933596+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '64', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}}, {'pool': 6, 'pool_name': 'default.rgw.meta', 'create_time': '2024-09-10T19:10:11.993712+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '66', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4}, 'application_metadata': {'rgw': {}}}, {'pool': 8, 'pool_name': 'unique_pool_1', 'create_time': '2024-09-10T19:16:11.474472+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 26, 'pg_placement_num': 26, 'pg_placement_num_target': 26, 'pg_num_target': 26, 'pg_num_pending': 26, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '179', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '177', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}}], 'osds': [{'osd': 0, 'uuid': '9d323010-394a-49aa-ba9b-244d87faf36e', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 8, 'up_thru': 204, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.183:6802', 'nonce': 2813074952}, {'type': 'v1', 'addr': '172.21.15.183:6803', 'nonce': 2813074952}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.183:6804', 'nonce': 2813074952}, {'type': 'v1', 'addr': '172.21.15.183:6805', 'nonce': 2813074952}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.183:6808', 'nonce': 2813074952}, {'type': 'v1', 'addr': '172.21.15.183:6809', 'nonce': 2813074952}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.183:6806', 'nonce': 2813074952}, {'type': 'v1', 'addr': '172.21.15.183:6807', 'nonce': 2813074952}]}, 'public_addr': '172.21.15.183:6803/2813074952', 'cluster_addr': '172.21.15.183:6805/2813074952', 'heartbeat_back_addr': '172.21.15.183:6809/2813074952', 'heartbeat_front_addr': '172.21.15.183:6807/2813074952', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': '71a6c5dc-5109-4284-8d20-187d8e7afc9e', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 1, 'last_clean_begin': 13, 'last_clean_end': 98, 'up_from': 138, 'up_thru': 86, 'down_at': 99, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.183:6810', 'nonce': 219854798}, {'type': 'v1', 'addr': '172.21.15.183:6811', 'nonce': 219854798}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.183:6812', 'nonce': 219854798}, {'type': 'v1', 'addr': '172.21.15.183:6813', 'nonce': 219854798}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.183:6816', 'nonce': 219854798}, {'type': 'v1', 'addr': '172.21.15.183:6817', 'nonce': 219854798}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.183:6814', 'nonce': 219854798}, {'type': 'v1', 'addr': '172.21.15.183:6815', 'nonce': 219854798}]}, 'public_addr': '172.21.15.183:6811/219854798', 'cluster_addr': '172.21.15.183:6813/219854798', 'heartbeat_back_addr': '172.21.15.183:6817/219854798', 'heartbeat_front_addr': '172.21.15.183:6815/219854798', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': 'fc310d5b-b89b-4655-9989-7e10ba356509', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 18, 'up_thru': 148, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.183:6818', 'nonce': 3730013786}, {'type': 'v1', 'addr': '172.21.15.183:6819', 'nonce': 3730013786}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.183:6820', 'nonce': 3730013786}, {'type': 'v1', 'addr': '172.21.15.183:6821', 'nonce': 3730013786}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.183:6824', 'nonce': 3730013786}, {'type': 'v1', 'addr': '172.21.15.183:6825', 'nonce': 3730013786}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.183:6822', 'nonce': 3730013786}, {'type': 'v1', 'addr': '172.21.15.183:6823', 'nonce': 3730013786}]}, 'public_addr': '172.21.15.183:6819/3730013786', 'cluster_addr': '172.21.15.183:6821/3730013786', 'heartbeat_back_addr': '172.21.15.183:6825/3730013786', 'heartbeat_front_addr': '172.21.15.183:6823/3730013786', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': '7fd07a95-ad22-43f5-a968-539552283e89', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 26, 'up_thru': 211, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.183:6826', 'nonce': 3522836984}, {'type': 'v1', 'addr': '172.21.15.183:6827', 'nonce': 3522836984}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.183:6828', 'nonce': 3522836984}, {'type': 'v1', 'addr': '172.21.15.183:6829', 'nonce': 3522836984}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.183:6832', 'nonce': 3522836984}, {'type': 'v1', 'addr': '172.21.15.183:6833', 'nonce': 3522836984}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.183:6830', 'nonce': 3522836984}, {'type': 'v1', 'addr': '172.21.15.183:6831', 'nonce': 3522836984}]}, 'public_addr': '172.21.15.183:6827/3522836984', 'cluster_addr': '172.21.15.183:6829/3522836984', 'heartbeat_back_addr': '172.21.15.183:6833/3522836984', 'heartbeat_front_addr': '172.21.15.183:6831/3522836984', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': '631083fb-e503-44da-9b6b-154e88539ee7', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 32, 'up_thru': 189, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.191:6800', 'nonce': 2674828539}, {'type': 'v1', 'addr': '172.21.15.191:6801', 'nonce': 2674828539}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.191:6802', 'nonce': 2674828539}, {'type': 'v1', 'addr': '172.21.15.191:6803', 'nonce': 2674828539}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.191:6806', 'nonce': 2674828539}, {'type': 'v1', 'addr': '172.21.15.191:6807', 'nonce': 2674828539}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.191:6804', 'nonce': 2674828539}, {'type': 'v1', 'addr': '172.21.15.191:6805', 'nonce': 2674828539}]}, 'public_addr': '172.21.15.191:6801/2674828539', 'cluster_addr': '172.21.15.191:6803/2674828539', 'heartbeat_back_addr': '172.21.15.191:6807/2674828539', 'heartbeat_front_addr': '172.21.15.191:6805/2674828539', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': '6b963695-3eca-4e6e-9207-05b8010ebff9', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 39, 'up_thru': 217, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.191:6808', 'nonce': 681733378}, {'type': 'v1', 'addr': '172.21.15.191:6809', 'nonce': 681733378}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.191:6810', 'nonce': 681733378}, {'type': 'v1', 'addr': '172.21.15.191:6811', 'nonce': 681733378}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.191:6814', 'nonce': 681733378}, {'type': 'v1', 'addr': '172.21.15.191:6815', 'nonce': 681733378}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.191:6812', 'nonce': 681733378}, {'type': 'v1', 'addr': '172.21.15.191:6813', 'nonce': 681733378}]}, 'public_addr': '172.21.15.191:6809/681733378', 'cluster_addr': '172.21.15.191:6811/681733378', 'heartbeat_back_addr': '172.21.15.191:6815/681733378', 'heartbeat_front_addr': '172.21.15.191:6813/681733378', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': '2dc9754f-6aef-4e19-8605-2f1591921811', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 47, 'up_thru': 179, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.191:6816', 'nonce': 2825813477}, {'type': 'v1', 'addr': '172.21.15.191:6817', 'nonce': 2825813477}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.191:6818', 'nonce': 2825813477}, {'type': 'v1', 'addr': '172.21.15.191:6819', 'nonce': 2825813477}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.191:6822', 'nonce': 2825813477}, {'type': 'v1', 'addr': '172.21.15.191:6823', 'nonce': 2825813477}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.191:6820', 'nonce': 2825813477}, {'type': 'v1', 'addr': '172.21.15.191:6821', 'nonce': 2825813477}]}, 'public_addr': '172.21.15.191:6817/2825813477', 'cluster_addr': '172.21.15.191:6819/2825813477', 'heartbeat_back_addr': '172.21.15.191:6823/2825813477', 'heartbeat_front_addr': '172.21.15.191:6821/2825813477', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': '1a9a299e-d5ba-4b9a-85d5-7f6a5bc754e1', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 54, 'up_thru': 206, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.191:6824', 'nonce': 1422610021}, {'type': 'v1', 'addr': '172.21.15.191:6825', 'nonce': 1422610021}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.191:6826', 'nonce': 1422610021}, {'type': 'v1', 'addr': '172.21.15.191:6827', 'nonce': 1422610021}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.191:6830', 'nonce': 1422610021}, {'type': 'v1', 'addr': '172.21.15.191:6831', 'nonce': 1422610021}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.191:6828', 'nonce': 1422610021}, {'type': 'v1', 'addr': '172.21.15.191:6829', 'nonce': 1422610021}]}, 'public_addr': '172.21.15.191:6825/1422610021', 'cluster_addr': '172.21.15.191:6827/1422610021', 'heartbeat_back_addr': '172.21.15.191:6831/1422610021', 'heartbeat_front_addr': '172.21.15.191:6829/1422610021', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138320759226367, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-10T19:19:08.635604+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138320759226367, 'old_weight': 65536, 'last_purged_snaps_scrub': '2024-09-10T19:19:22.837989+0000', 'dead_epoch': 98}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138320759226367, 'old_weight': 65536, 'last_purged_snaps_scrub': '2024-09-10T19:18:18.931307+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138320759226367, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-10T19:18:34.335087+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138320759226367, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-10T19:18:46.652022+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138320759226367, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-10T19:18:50.000086+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138320759226367, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-10T19:18:32.489412+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138320759226367, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-10T19:18:58.918466+0000', 'dead_epoch': 0}], 'pg_upmap': [{'pgid': '2.13', 'osds': [2, 0, 5]}, {'pgid': '8.3', 'osds': [7, 3, 5]}], 'pg_upmap_items': [{'pgid': '2.18', 'mappings': [{'from': 7, 'to': 0}, {'from': 5, 'to': 4}]}, {'pgid': '4.1', 'mappings': [{'from': 6, 'to': 7}]}, {'pgid': '4.11', 'mappings': [{'from': 3, 'to': 5}]}], 'pg_temp': [{'pgid': '8.12', 'osds': [0, 4, 5]}], 'primary_temp': [], 'blocklist': {'172.21.15.183:6801/3311783362': '2024-09-11T19:10:43.576943+0000', '172.21.15.183:0/4088553922': '2024-09-11T19:10:43.576943+0000', '172.21.15.183:0/3528769414': '2024-09-11T19:05:48.153879+0000', '172.21.15.183:0/1766036658': '2024-09-11T19:05:48.153879+0000', '172.21.15.183:0/3044170088': '2024-09-11T19:05:25.702354+0000', '172.21.15.183:0/3037333778': '2024-09-11T19:05:25.702354+0000', '172.21.15.183:6801/3747320570': '2024-09-11T19:05:25.702354+0000', '172.21.15.183:0/3982386906': '2024-09-11T19:10:43.576943+0000', '172.21.15.183:0/2436014455': '2024-09-11T19:05:25.702354+0000', '172.21.15.183:6801/659745765': '2024-09-11T19:05:48.153879+0000', '172.21.15.183:6800/3747320570': '2024-09-11T19:05:25.702354+0000', '172.21.15.183:0/1652576282': '2024-09-11T19:10:43.576943+0000', '172.21.15.183:6800/3311783362': '2024-09-11T19:10:43.576943+0000', '172.21.15.183:0/1012899549': '2024-09-11T19:05:48.153879+0000', '172.21.15.183:6800/659745765': '2024-09-11T19:05:48.153879+0000', '172.21.15.183:0/412044400': '2024-09-11T19:10:43.576943+0000', '172.21.15.183:0/3636171401': '2024-09-11T19:10:43.576943+0000'}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-09-10T19:19:36.403 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph pg dump --format=json 2024-09-10T19:19:36.407 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:19:36.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:36.408 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:19:36.509 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:19:36.530 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:19:36.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:36.531 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:19:36.580 DEBUG:teuthology.orchestra.run.smithi191:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph daemon osd.5 dump_blocked_ops 2024-09-10T19:19:36.631 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:19:36.653 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:19:36.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:36.653 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:19:36.754 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:19:36.776 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:19:36.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:36.777 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:19:36.821 INFO:teuthology.orchestra.run.smithi191.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.b/config 2024-09-10T19:19:36.878 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:19:36.889 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:19:36.900 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:19:36.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:36.901 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:19:36.919 INFO:teuthology.orchestra.run.smithi183.stdout:osd.0: {} 2024-09-10T19:19:36.919 INFO:teuthology.orchestra.run.smithi183.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-10T19:19:36.922 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:19:36.931 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:19:36 smithi183 ceph-mon[29736]: pgmap v420: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 24 MiB data, 2.9 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 2.09k op/s; 14/1701 objects misplaced (0.823%) 2024-09-10T19:19:36.932 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:19:36 smithi183 ceph-mon[41562]: pgmap v420: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 24 MiB data, 2.9 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 2.09k op/s; 14/1701 objects misplaced (0.823%) 2024-09-10T19:19:36.942 INFO:teuthology.orchestra.run.smithi183.stdout:osd.1: {} 2024-09-10T19:19:36.942 INFO:teuthology.orchestra.run.smithi183.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-10T19:19:36.957 INFO:teuthology.orchestra.run.smithi183.stdout:osd.2: {} 2024-09-10T19:19:36.957 INFO:teuthology.orchestra.run.smithi183.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-10T19:19:36.972 INFO:teuthology.orchestra.run.smithi183.stdout:osd.3: {} 2024-09-10T19:19:36.972 INFO:teuthology.orchestra.run.smithi183.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-10T19:19:36.987 INFO:teuthology.orchestra.run.smithi183.stdout:osd.4: {} 2024-09-10T19:19:36.987 INFO:teuthology.orchestra.run.smithi183.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-10T19:19:37.002 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:19:37.002 INFO:teuthology.orchestra.run.smithi183.stdout:osd.5: {} 2024-09-10T19:19:37.003 INFO:teuthology.orchestra.run.smithi183.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-10T19:19:37.018 INFO:teuthology.orchestra.run.smithi183.stdout:osd.6: {} 2024-09-10T19:19:37.018 INFO:teuthology.orchestra.run.smithi183.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-10T19:19:37.024 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:19:37.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:37.024 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:19:37.033 INFO:teuthology.orchestra.run.smithi183.stdout:osd.7: {} 2024-09-10T19:19:37.033 INFO:teuthology.orchestra.run.smithi183.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-10T19:19:37.110 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:19:36 smithi191 ceph-mon[32884]: pgmap v420: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 24 MiB data, 2.9 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 2.09k op/s; 14/1701 objects misplaced (0.823%) 2024-09-10T19:19:37.125 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:19:37.146 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:19:37.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:37.147 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:19:37.248 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:19:37.269 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:19:37.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:37.270 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:19:37.371 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:19:37.392 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:19:37.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:37.393 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:19:37.494 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:19:37.516 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:19:37.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:37.517 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:19:37.617 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:19:37.639 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:19:37.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:37.640 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:19:37.715 INFO:journalctl@ceph.mgr.x.smithi191.stdout:Sep 10 19:19:37 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-x[34967]: ::ffff:172.21.15.191 - - [10/Sep/2024:19:19:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T19:19:37.741 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:19:37.762 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:19:37.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:37.763 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:19:37.864 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:19:37.886 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:19:37.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:37.887 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:19:37.987 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:19:38.009 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:19:38.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:38.010 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:19:38.085 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:19:37 smithi191 ceph-mon[32884]: from='client.? 172.21.15.183:0/2851456160' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T19:19:38.110 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:19:38.132 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:19:38.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:38.133 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:19:38.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:19:37 smithi183 ceph-mon[29736]: from='client.? 172.21.15.183:0/2851456160' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T19:19:38.181 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:19:37 smithi183 ceph-mon[41562]: from='client.? 172.21.15.183:0/2851456160' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T19:19:38.234 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:19:38.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:38.256 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:19:38.256 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:19:38.357 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:19:38.379 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:19:38.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:38.380 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:19:38.481 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:19:38.503 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:19:38.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:38.504 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:19:38.604 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:19:38.626 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:19:38.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:38.627 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:19:38.640 INFO:teuthology.orchestra.run.smithi191.stdout:{ 2024-09-10T19:19:38.640 INFO:teuthology.orchestra.run.smithi191.stdout: "ops": [], 2024-09-10T19:19:38.640 INFO:teuthology.orchestra.run.smithi191.stdout: "complaint_time": 30, 2024-09-10T19:19:38.640 INFO:teuthology.orchestra.run.smithi191.stdout: "num_blocked_ops": 0 2024-09-10T19:19:38.640 INFO:teuthology.orchestra.run.smithi191.stdout:} 2024-09-10T19:19:38.728 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:19:38.750 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:19:38.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:38.751 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:19:38.851 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:19:38.872 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:19:38.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:38.873 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:19:38.931 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:19:38 smithi183 ceph-mon[29736]: pgmap v421: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 24 MiB data, 2.9 GiB used, 534 GiB / 536 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.57k op/s; 14/1701 objects misplaced (0.823%) 2024-09-10T19:19:38.931 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:19:38 smithi183 ceph-mon[41562]: pgmap v421: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 24 MiB data, 2.9 GiB used, 534 GiB / 536 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.57k op/s; 14/1701 objects misplaced (0.823%) 2024-09-10T19:19:38.965 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:19:38 smithi191 ceph-mon[32884]: pgmap v421: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 24 MiB data, 2.9 GiB used, 534 GiB / 536 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.57k op/s; 14/1701 objects misplaced (0.823%) 2024-09-10T19:19:38.974 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:19:38.995 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:19:38.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:38.996 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:19:39.096 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:19:39.118 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:19:39.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:39.119 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:19:39.219 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:19:39.240 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:19:39.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:39.241 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:19:39.342 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:19:39.359 DEBUG:teuthology.orchestra.run.smithi191:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph daemon osd.5 dump_historic_ops 2024-09-10T19:19:39.363 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:19:39.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:39.364 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:19:39.464 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:19:39.486 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:19:39.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:39.487 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:19:39.587 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:19:39.611 INFO:teuthology.orchestra.run.smithi191.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.b/config 2024-09-10T19:19:39.611 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:19:39.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:39.612 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:19:39.713 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:19:39.716 INFO:journalctl@ceph.iscsi.iscsi.a.smithi191.stdout:Sep 10 19:19:39 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-iscsi-iscsi-a[61816]: debug there is no tcmu-runner data available 2024-09-10T19:19:39.735 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:19:39.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:39.736 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:19:39.837 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:19:39.859 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:19:39.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:39.860 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:19:39.863 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-10T19:19:39.961 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:19:39.982 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:19:39.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:39.983 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:19:40.084 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:19:40.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:40.106 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:19:40.106 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:19:40.138 INFO:tasks.cephadm.ceph_manager.ceph:removing pool_name unique_pool_1 2024-09-10T19:19:40.138 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph osd pool rm unique_pool_1 unique_pool_1 --yes-i-really-really-mean-it 2024-09-10T19:19:40.207 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:19:40.230 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:19:40.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:40.231 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:19:40.331 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:19:40.353 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:19:40.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:40.354 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:19:40.455 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:19:40.478 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:19:40.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:40.479 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:19:40.579 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:19:40.602 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:19:40.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:40.604 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:19:40.704 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:19:40.726 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:19:40.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:40.728 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:19:40.829 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:19:40.850 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:19:40.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:40.851 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:19:40.952 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:19:40.960 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-09-10T19:19:40.963 INFO:teuthology.orchestra.run.smithi183.stderr:dumped all 2024-09-10T19:19:40.975 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:19:40.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:40.975 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:19:41.066 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:19:40 smithi183 ceph-mon[29736]: from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:19:41.067 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:19:40 smithi183 ceph-mon[29736]: pgmap v422: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 24 MiB data, 2.9 GiB used, 534 GiB / 536 GiB avail; 572 B/s rd, 0 B/s wr, 709 op/s; 14/1701 objects misplaced (0.823%) 2024-09-10T19:19:41.067 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:19:40 smithi183 ceph-mon[41562]: from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:19:41.067 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:19:40 smithi183 ceph-mon[41562]: pgmap v422: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 24 MiB data, 2.9 GiB used, 534 GiB / 536 GiB avail; 572 B/s rd, 0 B/s wr, 709 op/s; 14/1701 objects misplaced (0.823%) 2024-09-10T19:19:41.076 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:19:41.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:41.100 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:19:41.100 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:19:41.201 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:19:41.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:19:40 smithi191 ceph-mon[32884]: from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:19:41.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:19:40 smithi191 ceph-mon[32884]: pgmap v422: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 24 MiB data, 2.9 GiB used, 534 GiB / 536 GiB avail; 572 B/s rd, 0 B/s wr, 709 op/s; 14/1701 objects misplaced (0.823%) 2024-09-10T19:19:41.223 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:19:41.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:41.224 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:19:41.325 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:19:41.337 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Sep 10 19:19:41 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-y[29989]: ::ffff:172.21.15.191 - - [10/Sep/2024:19:19:41] "GET /metrics HTTP/1.1" 200 224280 "" "Prometheus/2.43.0" 2024-09-10T19:19:41.346 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:19:41.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:41.347 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:19:41.448 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:19:41.470 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:19:41.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:41.471 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:19:41.572 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:19:41.594 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:19:41.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:41.595 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:19:41.660 INFO:teuthology.orchestra.run.smithi183.stdout:{"pg_ready":true,"pg_map":{"version":422,"stamp":"2024-09-10T19:19:39.771713+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":25041116,"num_objects":567,"num_object_clones":0,"num_object_copies":1701,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":14,"num_objects_unfound":0,"num_objects_dirty":567,"num_whiteouts":0,"num_read":1045,"num_read_kb":902,"num_write":241080,"num_write_kb":7699592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":29072,"num_bytes_recovered":1893929397,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97083,"ondisk_log_size":97083,"up":474,"acting":474,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":491,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":562470912,"kb_used":3001712,"kb_used_data":615616,"kb_used_omap":9,"kb_used_meta":2385782,"kb_avail":559469200,"statfs":{"total":575970213888,"available":572896460800,"internally_reserved":0,"allocated":630390784,"data_stored":628121692,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":10224,"internal_metadata":2443040784},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-392888344,"num_objects":-5998,"num_object_clones":0,"num_object_copies":-17994,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-386,"num_objects_unfound":0,"num_objects_dirty":-5998,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":6344,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.949894"},"pg_stats":[{"pgid":"8.15","version":"224'10468","reported_seq":15293,"reported_epoch":224,"state":"active+clean","last_fresh":"2024-09-10T19:19:34.929417+0000","last_change":"2024-09-10T19:18:45.180957+0000","last_active":"2024-09-10T19:19:34.929417+0000","last_peered":"2024-09-10T19:19:34.929417+0000","last_clean":"2024-09-10T19:19:34.929417+0000","last_became_active":"2024-09-10T19:18:45.155095+0000","last_became_peered":"2024-09-10T19:18:45.155095+0000","last_unstale":"2024-09-10T19:19:34.929417+0000","last_undegraded":"2024-09-10T19:19:34.929417+0000","last_fullsized":"2024-09-10T19:19:34.929417+0000","mapping_epoch":206,"log_start":"192'6942","ondisk_log_start":"192'6942","created":177,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:16:11.777014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:16:11.777014+0000","last_clean_scrub_stamp":"2024-09-10T19:16:11.777014+0000","objects_scrubbed":0,"log_size":3526,"log_dups_size":0,"ondisk_log_size":3526,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2752512,"num_objects":42,"num_object_clones":0,"num_object_copies":126,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":42,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7544,"num_write_kb":242752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2348,"num_bytes_recovered":153878528,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":198,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-09-10T19:19:30.831423+0000","last_change":"2024-09-10T19:19:09.088516+0000","last_active":"2024-09-10T19:19:30.831423+0000","last_peered":"2024-09-10T19:19:30.831423+0000","last_clean":"2024-09-10T19:19:30.831423+0000","last_became_active":"2024-09-10T19:10:14.056187+0000","last_became_peered":"2024-09-10T19:10:14.056187+0000","last_unstale":"2024-09-10T19:19:30.831423+0000","last_undegraded":"2024-09-10T19:19:30.831423+0000","last_fullsized":"2024-09-10T19:19:30.831423+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:19:09.088433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:19:09.088433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:20:17.425778+0000","scrub_duration":0.0029670339999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":223,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-09-10T19:19:30.828267+0000","last_change":"2024-09-10T19:16:38.194521+0000","last_active":"2024-09-10T19:19:30.828267+0000","last_peered":"2024-09-10T19:19:30.828267+0000","last_clean":"2024-09-10T19:19:30.828267+0000","last_became_active":"2024-09-10T19:16:24.836961+0000","last_became_peered":"2024-09-10T19:16:24.836961+0000","last_unstale":"2024-09-10T19:19:30.828267+0000","last_undegraded":"2024-09-10T19:19:30.828267+0000","last_fullsized":"2024-09-10T19:19:30.828267+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:16:38.194472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:16:38.194472+0000","last_clean_scrub_stamp":"2024-09-10T19:16:38.194472+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0023678900000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.19","version":"66'15","reported_seq":253,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-09-10T19:19:30.830357+0000","last_change":"2024-09-10T19:19:21.244588+0000","last_active":"2024-09-10T19:19:30.830357+0000","last_peered":"2024-09-10T19:19:30.830357+0000","last_clean":"2024-09-10T19:19:30.830357+0000","last_became_active":"2024-09-10T19:16:24.844951+0000","last_became_peered":"2024-09-10T19:16:24.844951+0000","last_unstale":"2024-09-10T19:19:30.830357+0000","last_undegraded":"2024-09-10T19:19:30.830357+0000","last_fullsized":"2024-09-10T19:19:30.830357+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'15","last_scrub_stamp":"2024-09-10T19:19:21.244506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:19:21.244506+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:20:40.860636+0000","scrub_duration":0.014405133000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":232,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-09-10T19:19:30.825867+0000","last_change":"2024-09-10T19:19:28.498635+0000","last_active":"2024-09-10T19:19:30.825867+0000","last_peered":"2024-09-10T19:19:30.825867+0000","last_clean":"2024-09-10T19:19:30.825867+0000","last_became_active":"2024-09-10T19:16:24.845111+0000","last_became_peered":"2024-09-10T19:16:24.845111+0000","last_unstale":"2024-09-10T19:19:30.825867+0000","last_undegraded":"2024-09-10T19:19:30.825867+0000","last_fullsized":"2024-09-10T19:19:30.825867+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:19:28.498560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:19:28.498560+0000","last_clean_scrub_stamp":"2024-09-10T19:19:28.498560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:20:53.150820+0000","scrub_duration":0.0071873639999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,7],"acting":[4,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.14","version":"223'10322","reported_seq":20219,"reported_epoch":224,"state":"active+clean","last_fresh":"2024-09-10T19:19:34.804322+0000","last_change":"2024-09-10T19:17:49.772679+0000","last_active":"2024-09-10T19:19:34.804322+0000","last_peered":"2024-09-10T19:19:34.804322+0000","last_clean":"2024-09-10T19:19:34.804322+0000","last_became_active":"2024-09-10T19:17:49.767787+0000","last_became_peered":"2024-09-10T19:17:49.767787+0000","last_unstale":"2024-09-10T19:19:34.804322+0000","last_undegraded":"2024-09-10T19:19:34.804322+0000","last_fullsized":"2024-09-10T19:19:34.804322+0000","mapping_epoch":189,"log_start":"190'6461","ondisk_log_start":"190'6461","created":177,"last_epoch_clean":190,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:16:11.777014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:16:11.777014+0000","last_clean_scrub_stamp":"2024-09-10T19:16:11.777014+0000","objects_scrubbed":0,"log_size":3861,"log_dups_size":0,"ondisk_log_size":3861,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7460,"num_write_kb":238048,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3256,"num_bytes_recovered":213385216,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"59'1","reported_seq":212,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-09-10T19:19:30.834143+0000","last_change":"2024-09-10T19:16:30.798883+0000","last_active":"2024-09-10T19:19:30.834143+0000","last_peered":"2024-09-10T19:19:30.834143+0000","last_clean":"2024-09-10T19:19:30.834143+0000","last_became_active":"2024-09-10T19:10:07.686143+0000","last_became_peered":"2024-09-10T19:10:07.686143+0000","last_unstale":"2024-09-10T19:19:30.834143+0000","last_undegraded":"2024-09-10T19:19:30.834143+0000","last_fullsized":"2024-09-10T19:19:30.834143+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'1","last_scrub_stamp":"2024-09-10T19:16:30.798795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:16:30.798795+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0067002989999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":874,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"66'9","reported_seq":247,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-09-10T19:19:30.826106+0000","last_change":"2024-09-10T19:19:24.417823+0000","last_active":"2024-09-10T19:19:30.826106+0000","last_peered":"2024-09-10T19:19:30.826106+0000","last_clean":"2024-09-10T19:19:30.826106+0000","last_became_active":"2024-09-10T19:13:54.716117+0000","last_became_peered":"2024-09-10T19:13:54.716117+0000","last_unstale":"2024-09-10T19:19:30.826106+0000","last_undegraded":"2024-09-10T19:19:30.826106+0000","last_fullsized":"2024-09-10T19:19:30.826106+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'9","last_scrub_stamp":"2024-09-10T19:19:24.417719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:19:24.417719+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:20:51.013137+0000","scrub_duration":0.023031233000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,3],"acting":[4,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":164,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-09-10T19:19:30.838069+0000","last_change":"2024-09-10T19:16:27.753997+0000","last_active":"2024-09-10T19:19:30.838069+0000","last_peered":"2024-09-10T19:19:30.838069+0000","last_clean":"2024-09-10T19:19:30.838069+0000","last_became_active":"2024-09-10T19:16:24.859261+0000","last_became_peered":"2024-09-10T19:16:24.859261+0000","last_unstale":"2024-09-10T19:19:30.838069+0000","last_undegraded":"2024-09-10T19:19:30.838069+0000","last_fullsized":"2024-09-10T19:19:30.838069+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:16:27.753912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:16:27.753912+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0029373799999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":210,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-09-10T19:19:30.828732+0000","last_change":"2024-09-10T19:17:23.968484+0000","last_active":"2024-09-10T19:19:30.828732+0000","last_peered":"2024-09-10T19:19:30.828732+0000","last_clean":"2024-09-10T19:19:30.828732+0000","last_became_active":"2024-09-10T19:13:54.713538+0000","last_became_peered":"2024-09-10T19:13:54.713538+0000","last_unstale":"2024-09-10T19:19:30.828732+0000","last_undegraded":"2024-09-10T19:19:30.828732+0000","last_fullsized":"2024-09-10T19:19:30.828732+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:15:39.910118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:15:39.910118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0013551839999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,7],"acting":[4,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.17","version":"224'10440","reported_seq":17734,"reported_epoch":224,"state":"active+clean","last_fresh":"2024-09-10T19:19:34.918764+0000","last_change":"2024-09-10T19:18:44.173985+0000","last_active":"2024-09-10T19:19:34.918764+0000","last_peered":"2024-09-10T19:19:34.918764+0000","last_clean":"2024-09-10T19:19:34.918764+0000","last_became_active":"2024-09-10T19:18:44.154191+0000","last_became_peered":"2024-09-10T19:18:44.154191+0000","last_unstale":"2024-09-10T19:19:34.918764+0000","last_undegraded":"2024-09-10T19:19:34.918764+0000","last_fullsized":"2024-09-10T19:19:34.918764+0000","mapping_epoch":205,"log_start":"192'6923","ondisk_log_start":"192'6923","created":177,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:16:11.777014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:16:11.777014+0000","last_clean_scrub_stamp":"2024-09-10T19:16:11.777014+0000","objects_scrubbed":0,"log_size":3517,"log_dups_size":0,"ondisk_log_size":3517,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7542,"num_write_kb":240096,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2387,"num_bytes_recovered":156434432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":204,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-09-10T19:19:30.836879+0000","last_change":"2024-09-10T19:16:25.595265+0000","last_active":"2024-09-10T19:19:30.836879+0000","last_peered":"2024-09-10T19:19:30.836879+0000","last_clean":"2024-09-10T19:19:30.836879+0000","last_became_active":"2024-09-10T19:10:07.690728+0000","last_became_peered":"2024-09-10T19:10:07.690728+0000","last_unstale":"2024-09-10T19:19:30.836879+0000","last_undegraded":"2024-09-10T19:19:30.836879+0000","last_fullsized":"2024-09-10T19:19:30.836879+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:16:25.595127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:16:25.595127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.003922633,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1b","version":"66'5","reported_seq":243,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-09-10T19:19:30.831656+0000","last_change":"2024-09-10T19:16:35.966457+0000","last_active":"2024-09-10T19:19:30.831656+0000","last_peered":"2024-09-10T19:19:30.831656+0000","last_clean":"2024-09-10T19:19:30.831656+0000","last_became_active":"2024-09-10T19:16:24.843708+0000","last_became_peered":"2024-09-10T19:16:24.843708+0000","last_unstale":"2024-09-10T19:19:30.831656+0000","last_undegraded":"2024-09-10T19:19:30.831656+0000","last_fullsized":"2024-09-10T19:19:30.831656+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'5","last_scrub_stamp":"2024-09-10T19:16:35.966402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:16:35.966402+0000","objects_scrubbed":2,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0019325740000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":207,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-09-10T19:19:30.830562+0000","last_change":"2024-09-10T19:15:33.518718+0000","last_active":"2024-09-10T19:19:30.830562+0000","last_peered":"2024-09-10T19:19:30.830562+0000","last_clean":"2024-09-10T19:19:30.830562+0000","last_became_active":"2024-09-10T19:13:54.719063+0000","last_became_peered":"2024-09-10T19:13:54.719063+0000","last_unstale":"2024-09-10T19:19:30.830562+0000","last_undegraded":"2024-09-10T19:19:30.830562+0000","last_fullsized":"2024-09-10T19:19:30.830562+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:15:33.518601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:15:33.518601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0013951650000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":215,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-09-10T19:19:30.838147+0000","last_change":"2024-09-10T19:16:24.845721+0000","last_active":"2024-09-10T19:19:30.838147+0000","last_peered":"2024-09-10T19:19:30.838147+0000","last_clean":"2024-09-10T19:19:30.838147+0000","last_became_active":"2024-09-10T19:16:24.845130+0000","last_became_peered":"2024-09-10T19:16:24.845130+0000","last_unstale":"2024-09-10T19:19:30.838147+0000","last_undegraded":"2024-09-10T19:19:30.838147+0000","last_fullsized":"2024-09-10T19:19:30.838147+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:15:45.616184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:15:45.616184+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0013989510000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,0],"acting":[5,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.16","version":"224'10280","reported_seq":17372,"reported_epoch":224,"state":"active+clean","last_fresh":"2024-09-10T19:19:34.938599+0000","last_change":"2024-09-10T19:18:44.173818+0000","last_active":"2024-09-10T19:19:34.938599+0000","last_peered":"2024-09-10T19:19:34.938599+0000","last_clean":"2024-09-10T19:19:34.938599+0000","last_became_active":"2024-09-10T19:18:44.148801+0000","last_became_peered":"2024-09-10T19:18:44.148801+0000","last_unstale":"2024-09-10T19:19:34.938599+0000","last_undegraded":"2024-09-10T19:19:34.938599+0000","last_fullsized":"2024-09-10T19:19:34.938599+0000","mapping_epoch":205,"log_start":"192'6672","ondisk_log_start":"192'6672","created":177,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:16:11.777014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:16:11.777014+0000","last_clean_scrub_stamp":"2024-09-10T19:16:11.777014+0000","objects_scrubbed":0,"log_size":3608,"log_dups_size":0,"ondisk_log_size":3608,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7408,"num_write_kb":236320,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2323,"num_bytes_recovered":152240128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,0],"acting":[3,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":214,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-09-10T19:19:30.831303+0000","last_change":"2024-09-10T19:19:08.062655+0000","last_active":"2024-09-10T19:19:30.831303+0000","last_peered":"2024-09-10T19:19:30.831303+0000","last_clean":"2024-09-10T19:19:30.831303+0000","last_became_active":"2024-09-10T19:16:24.858800+0000","last_became_peered":"2024-09-10T19:16:24.858800+0000","last_unstale":"2024-09-10T19:19:30.831303+0000","last_undegraded":"2024-09-10T19:19:30.831303+0000","last_fullsized":"2024-09-10T19:19:30.831303+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:19:08.062584+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:19:08.062584+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:20:28.059488+0000","scrub_duration":0.0065614130000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1a","version":"66'9","reported_seq":220,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-09-10T19:19:30.829031+0000","last_change":"2024-09-10T19:19:13.519469+0000","last_active":"2024-09-10T19:19:30.829031+0000","last_peered":"2024-09-10T19:19:30.829031+0000","last_clean":"2024-09-10T19:19:30.829031+0000","last_became_active":"2024-09-10T19:10:09.688987+0000","last_became_peered":"2024-09-10T19:10:09.688987+0000","last_unstale":"2024-09-10T19:19:30.829031+0000","last_undegraded":"2024-09-10T19:19:30.829031+0000","last_fullsized":"2024-09-10T19:19:30.829031+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'9","last_scrub_stamp":"2024-09-10T19:19:13.519390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:19:13.519390+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:20:22.967434+0000","scrub_duration":0.022527794,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":200,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-09-10T19:19:30.835329+0000","last_change":"2024-09-10T19:16:37.116464+0000","last_active":"2024-09-10T19:19:30.835329+0000","last_peered":"2024-09-10T19:19:30.835329+0000","last_clean":"2024-09-10T19:19:30.835329+0000","last_became_active":"2024-09-10T19:10:11.701799+0000","last_became_peered":"2024-09-10T19:10:11.701799+0000","last_unstale":"2024-09-10T19:19:30.835329+0000","last_undegraded":"2024-09-10T19:19:30.835329+0000","last_fullsized":"2024-09-10T19:19:30.835329+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:16:37.116371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:16:37.116371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.028100301000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":230,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-09-10T19:19:30.828461+0000","last_change":"2024-09-10T19:19:27.544760+0000","last_active":"2024-09-10T19:19:30.828461+0000","last_peered":"2024-09-10T19:19:30.828461+0000","last_clean":"2024-09-10T19:19:30.828461+0000","last_became_active":"2024-09-10T19:16:24.836477+0000","last_became_peered":"2024-09-10T19:16:24.836477+0000","last_unstale":"2024-09-10T19:19:30.828461+0000","last_undegraded":"2024-09-10T19:19:30.828461+0000","last_fullsized":"2024-09-10T19:19:30.828461+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:19:27.544704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:19:27.544704+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:20:48.302335+0000","scrub_duration":0.010150773,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1","version":"224'10558","reported_seq":10661,"reported_epoch":224,"state":"active+clean","last_fresh":"2024-09-10T19:19:34.882486+0000","last_change":"2024-09-10T19:17:26.961275+0000","last_active":"2024-09-10T19:19:34.882486+0000","last_peered":"2024-09-10T19:19:34.882486+0000","last_clean":"2024-09-10T19:19:34.882486+0000","last_became_active":"2024-09-10T19:17:26.958866+0000","last_became_peered":"2024-09-10T19:17:26.958866+0000","last_unstale":"2024-09-10T19:19:34.882486+0000","last_undegraded":"2024-09-10T19:19:34.882486+0000","last_fullsized":"2024-09-10T19:19:34.882486+0000","mapping_epoch":154,"log_start":"192'6769","ondisk_log_start":"192'6769","created":148,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:16:11.777014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:16:11.777014+0000","last_clean_scrub_stamp":"2024-09-10T19:16:11.777014+0000","objects_scrubbed":0,"log_size":3789,"log_dups_size":0,"ondisk_log_size":3789,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7639,"num_write_kb":243168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.d","version":"66'17","reported_seq":292,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-09-10T19:19:30.831805+0000","last_change":"2024-09-10T19:16:32.999033+0000","last_active":"2024-09-10T19:19:30.831805+0000","last_peered":"2024-09-10T19:19:30.831805+0000","last_clean":"2024-09-10T19:19:30.831805+0000","last_became_active":"2024-09-10T19:16:24.847056+0000","last_became_peered":"2024-09-10T19:16:24.847056+0000","last_unstale":"2024-09-10T19:19:30.831805+0000","last_undegraded":"2024-09-10T19:19:30.831805+0000","last_fullsized":"2024-09-10T19:19:30.831805+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'17","last_scrub_stamp":"2024-09-10T19:16:32.998950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:16:32.998950+0000","objects_scrubbed":8,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0095505250000000007,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,0],"acting":[4,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":204,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-09-10T19:19:30.832337+0000","last_change":"2024-09-10T19:16:39.716870+0000","last_active":"2024-09-10T19:19:30.832337+0000","last_peered":"2024-09-10T19:19:30.832337+0000","last_clean":"2024-09-10T19:19:30.832337+0000","last_became_active":"2024-09-10T19:10:07.691119+0000","last_became_peered":"2024-09-10T19:10:07.691119+0000","last_unstale":"2024-09-10T19:19:30.832337+0000","last_undegraded":"2024-09-10T19:19:30.832337+0000","last_fullsized":"2024-09-10T19:19:30.832337+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:16:39.716674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:16:39.716674+0000","last_clean_scrub_stamp":"2024-09-10T19:16:39.716674+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.043895318000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":157,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-09-10T19:19:30.830017+0000","last_change":"2024-09-10T19:16:26.999105+0000","last_active":"2024-09-10T19:19:30.830017+0000","last_peered":"2024-09-10T19:19:30.830017+0000","last_clean":"2024-09-10T19:19:30.830017+0000","last_became_active":"2024-09-10T19:13:54.712883+0000","last_became_peered":"2024-09-10T19:13:54.712883+0000","last_unstale":"2024-09-10T19:19:30.830017+0000","last_undegraded":"2024-09-10T19:19:30.830017+0000","last_fullsized":"2024-09-10T19:19:30.830017+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:16:26.999036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:16:26.999036+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0050193119999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":82,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-09-10T19:19:30.833062+0000","last_change":"2024-09-10T19:16:24.858656+0000","last_active":"2024-09-10T19:19:30.833062+0000","last_peered":"2024-09-10T19:19:30.833062+0000","last_clean":"2024-09-10T19:19:30.833062+0000","last_became_active":"2024-09-10T19:16:24.857843+0000","last_became_peered":"2024-09-10T19:16:24.857843+0000","last_unstale":"2024-09-10T19:19:30.833062+0000","last_undegraded":"2024-09-10T19:19:30.833062+0000","last_fullsized":"2024-09-10T19:19:30.833062+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:15:47.818381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:15:47.818381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.0","version":"224'10338","reported_seq":10434,"reported_epoch":224,"state":"active+clean","last_fresh":"2024-09-10T19:19:34.895130+0000","last_change":"2024-09-10T19:17:26.971893+0000","last_active":"2024-09-10T19:19:34.895130+0000","last_peered":"2024-09-10T19:19:34.895130+0000","last_clean":"2024-09-10T19:19:34.895130+0000","last_became_active":"2024-09-10T19:17:26.958072+0000","last_became_peered":"2024-09-10T19:17:26.958072+0000","last_unstale":"2024-09-10T19:19:34.895130+0000","last_undegraded":"2024-09-10T19:19:34.895130+0000","last_fullsized":"2024-09-10T19:19:34.895130+0000","mapping_epoch":148,"log_start":"192'6738","ondisk_log_start":"192'6738","created":148,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:16:11.777014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:16:11.777014+0000","last_clean_scrub_stamp":"2024-09-10T19:16:11.777014+0000","objects_scrubbed":0,"log_size":3600,"log_dups_size":0,"ondisk_log_size":3600,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9175040,"num_objects":140,"num_object_clones":0,"num_object_copies":420,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":140,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7422,"num_write_kb":241984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,4],"acting":[3,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"66'10","reported_seq":229,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-09-10T19:19:30.831655+0000","last_change":"2024-09-10T19:19:18.488264+0000","last_active":"2024-09-10T19:19:30.831655+0000","last_peered":"2024-09-10T19:19:30.831655+0000","last_clean":"2024-09-10T19:19:30.831655+0000","last_became_active":"2024-09-10T19:10:09.688321+0000","last_became_peered":"2024-09-10T19:10:09.688321+0000","last_unstale":"2024-09-10T19:19:30.831655+0000","last_undegraded":"2024-09-10T19:19:30.831655+0000","last_fullsized":"2024-09-10T19:19:30.831655+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'10","last_scrub_stamp":"2024-09-10T19:19:18.488174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:19:18.488174+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:20:28.041787+0000","scrub_duration":0.0042762770000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":154,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-09-10T19:19:30.833127+0000","last_change":"2024-09-10T19:15:42.119912+0000","last_active":"2024-09-10T19:19:30.833127+0000","last_peered":"2024-09-10T19:19:30.833127+0000","last_clean":"2024-09-10T19:19:30.833127+0000","last_became_active":"2024-09-10T19:13:54.731134+0000","last_became_peered":"2024-09-10T19:13:54.731134+0000","last_unstale":"2024-09-10T19:19:30.833127+0000","last_undegraded":"2024-09-10T19:19:30.833127+0000","last_fullsized":"2024-09-10T19:19:30.833127+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:15:42.119590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:15:42.119590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.018001347000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,5],"acting":[3,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"155'10","reported_seq":190,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-09-10T19:19:30.852920+0000","last_change":"2024-09-10T19:16:29.626806+0000","last_active":"2024-09-10T19:19:30.852920+0000","last_peered":"2024-09-10T19:19:30.852920+0000","last_clean":"2024-09-10T19:19:30.852920+0000","last_became_active":"2024-09-10T19:16:24.851236+0000","last_became_peered":"2024-09-10T19:16:24.851236+0000","last_unstale":"2024-09-10T19:19:30.852920+0000","last_undegraded":"2024-09-10T19:19:30.852920+0000","last_fullsized":"2024-09-10T19:19:30.852920+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"155'10","last_scrub_stamp":"2024-09-10T19:16:29.626572+0000","last_deep_scrub":"80'9","last_deep_scrub_stamp":"2024-09-10T19:11:25.156465+0000","last_clean_scrub_stamp":"2024-09-10T19:16:29.626572+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0067064810000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,0],"acting":[7,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":220,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-09-10T19:19:30.828809+0000","last_change":"2024-09-10T19:16:24.852100+0000","last_active":"2024-09-10T19:19:30.828809+0000","last_peered":"2024-09-10T19:19:30.828809+0000","last_clean":"2024-09-10T19:19:30.828809+0000","last_became_active":"2024-09-10T19:16:24.851829+0000","last_became_peered":"2024-09-10T19:16:24.851829+0000","last_unstale":"2024-09-10T19:19:30.828809+0000","last_undegraded":"2024-09-10T19:19:30.828809+0000","last_fullsized":"2024-09-10T19:19:30.828809+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:15:29.988976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:15:29.988976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0053787150000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,7],"acting":[4,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.3","version":"224'10718","reported_seq":15777,"reported_epoch":224,"state":"active+clean","last_fresh":"2024-09-10T19:19:34.901667+0000","last_change":"2024-09-10T19:18:43.997699+0000","last_active":"2024-09-10T19:19:34.901667+0000","last_peered":"2024-09-10T19:19:34.901667+0000","last_clean":"2024-09-10T19:19:34.901667+0000","last_became_active":"2024-09-10T19:17:27.974691+0000","last_became_peered":"2024-09-10T19:17:27.974691+0000","last_unstale":"2024-09-10T19:19:34.901667+0000","last_undegraded":"2024-09-10T19:19:34.901667+0000","last_fullsized":"2024-09-10T19:19:34.901667+0000","mapping_epoch":178,"log_start":"192'7186","ondisk_log_start":"192'7186","created":148,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:16:11.777014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:16:11.777014+0000","last_clean_scrub_stamp":"2024-09-10T19:16:11.777014+0000","objects_scrubbed":0,"log_size":3532,"log_dups_size":0,"ondisk_log_size":3532,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2064384,"num_objects":31,"num_object_clones":0,"num_object_copies":93,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7707,"num_write_kb":247648,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1928,"num_bytes_recovered":126386176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.f","version":"66'15","reported_seq":207,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-09-10T19:19:30.833274+0000","last_change":"2024-09-10T19:16:25.769523+0000","last_active":"2024-09-10T19:19:30.833274+0000","last_peered":"2024-09-10T19:19:30.833274+0000","last_clean":"2024-09-10T19:19:30.833274+0000","last_became_active":"2024-09-10T19:13:54.711634+0000","last_became_peered":"2024-09-10T19:13:54.711634+0000","last_unstale":"2024-09-10T19:19:30.833274+0000","last_undegraded":"2024-09-10T19:19:30.833274+0000","last_fullsized":"2024-09-10T19:19:30.833274+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'15","last_scrub_stamp":"2024-09-10T19:16:25.769436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:16:25.769436+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0046254920000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":174,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-09-10T19:19:30.830732+0000","last_change":"2024-09-10T19:19:21.019996+0000","last_active":"2024-09-10T19:19:30.830732+0000","last_peered":"2024-09-10T19:19:30.830732+0000","last_clean":"2024-09-10T19:19:30.830732+0000","last_became_active":"2024-09-10T19:16:24.857179+0000","last_became_peered":"2024-09-10T19:16:24.857179+0000","last_unstale":"2024-09-10T19:19:30.830732+0000","last_undegraded":"2024-09-10T19:19:30.830732+0000","last_fullsized":"2024-09-10T19:19:30.830732+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:19:21.019912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:19:21.019912+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:20:30.827923+0000","scrub_duration":0.0041693620000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,6],"acting":[7,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.e","version":"66'8","reported_seq":323,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-09-10T19:19:30.855532+0000","last_change":"2024-09-10T19:19:17.519606+0000","last_active":"2024-09-10T19:19:30.855532+0000","last_peered":"2024-09-10T19:19:30.855532+0000","last_clean":"2024-09-10T19:19:30.855532+0000","last_became_active":"2024-09-10T19:10:11.699232+0000","last_became_peered":"2024-09-10T19:10:11.699232+0000","last_unstale":"2024-09-10T19:19:30.855532+0000","last_undegraded":"2024-09-10T19:19:30.855532+0000","last_fullsized":"2024-09-10T19:19:30.855532+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'8","last_scrub_stamp":"2024-09-10T19:19:17.519517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:19:17.519517+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:20:40.560476+0000","scrub_duration":0.0049564470000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":215,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-09-10T19:19:30.835898+0000","last_change":"2024-09-10T19:16:24.855938+0000","last_active":"2024-09-10T19:19:30.835898+0000","last_peered":"2024-09-10T19:19:30.835898+0000","last_clean":"2024-09-10T19:19:30.835898+0000","last_became_active":"2024-09-10T19:16:24.855627+0000","last_became_peered":"2024-09-10T19:16:24.855627+0000","last_unstale":"2024-09-10T19:19:30.835898+0000","last_undegraded":"2024-09-10T19:19:30.835898+0000","last_fullsized":"2024-09-10T19:19:30.835898+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:15:44.633043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:15:44.633043+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.00368805,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,6],"acting":[5,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.2","version":"223'10333","reported_seq":10428,"reported_epoch":224,"state":"active+clean","last_fresh":"2024-09-10T19:19:34.797434+0000","last_change":"2024-09-10T19:17:26.956443+0000","last_active":"2024-09-10T19:19:34.797434+0000","last_peered":"2024-09-10T19:19:34.797434+0000","last_clean":"2024-09-10T19:19:34.797434+0000","last_became_active":"2024-09-10T19:17:26.953881+0000","last_became_peered":"2024-09-10T19:17:26.953881+0000","last_unstale":"2024-09-10T19:19:34.797434+0000","last_undegraded":"2024-09-10T19:19:34.797434+0000","last_fullsized":"2024-09-10T19:19:34.797434+0000","mapping_epoch":148,"log_start":"190'6478","ondisk_log_start":"190'6478","created":148,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:16:11.777014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:16:11.777014+0000","last_clean_scrub_stamp":"2024-09-10T19:16:11.777014+0000","objects_scrubbed":0,"log_size":3855,"log_dups_size":0,"ondisk_log_size":3855,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7416,"num_write_kb":237312,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,5],"acting":[4,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"84'16","reported_seq":247,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-09-10T19:19:30.828977+0000","last_change":"2024-09-10T19:15:47.914487+0000","last_active":"2024-09-10T19:19:30.828977+0000","last_peered":"2024-09-10T19:19:30.828977+0000","last_clean":"2024-09-10T19:19:30.828977+0000","last_became_active":"2024-09-10T19:13:54.717022+0000","last_became_peered":"2024-09-10T19:13:54.717022+0000","last_unstale":"2024-09-10T19:19:30.828977+0000","last_undegraded":"2024-09-10T19:19:30.828977+0000","last_fullsized":"2024-09-10T19:19:30.828977+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'16","last_scrub_stamp":"2024-09-10T19:15:47.914420+0000","last_deep_scrub":"84'16","last_deep_scrub_stamp":"2024-09-10T19:11:56.157975+0000","last_clean_scrub_stamp":"2024-09-10T19:15:47.914420+0000","objects_scrubbed":5,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0019153390000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,3],"acting":[4,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":211,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-09-10T19:19:30.830740+0000","last_change":"2024-09-10T19:15:42.450149+0000","last_active":"2024-09-10T19:19:30.830740+0000","last_peered":"2024-09-10T19:19:30.830740+0000","last_clean":"2024-09-10T19:19:30.830740+0000","last_became_active":"2024-09-10T19:13:54.720293+0000","last_became_peered":"2024-09-10T19:13:54.720293+0000","last_unstale":"2024-09-10T19:19:30.830740+0000","last_undegraded":"2024-09-10T19:19:30.830740+0000","last_fullsized":"2024-09-10T19:19:30.830740+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:15:42.450068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:11:53.066849+0000","last_clean_scrub_stamp":"2024-09-10T19:15:42.450068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0037708360000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":200,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-09-10T19:19:30.835964+0000","last_change":"2024-09-10T19:19:29.566991+0000","last_active":"2024-09-10T19:19:30.835964+0000","last_peered":"2024-09-10T19:19:30.835964+0000","last_clean":"2024-09-10T19:19:30.835964+0000","last_became_active":"2024-09-10T19:10:11.692215+0000","last_became_peered":"2024-09-10T19:10:11.692215+0000","last_unstale":"2024-09-10T19:19:30.835964+0000","last_undegraded":"2024-09-10T19:19:30.835964+0000","last_fullsized":"2024-09-10T19:19:30.835964+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:19:29.566920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:19:29.566920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:20:49.151502+0000","scrub_duration":0.0083507189999999995,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":198,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-09-10T19:19:30.831993+0000","last_change":"2024-09-10T19:19:28.321305+0000","last_active":"2024-09-10T19:19:30.831993+0000","last_peered":"2024-09-10T19:19:30.831993+0000","last_clean":"2024-09-10T19:19:30.831993+0000","last_became_active":"2024-09-10T19:10:14.053736+0000","last_became_peered":"2024-09-10T19:10:14.053736+0000","last_unstale":"2024-09-10T19:19:30.831993+0000","last_undegraded":"2024-09-10T19:19:30.831993+0000","last_fullsized":"2024-09-10T19:19:30.831993+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:19:28.321253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:19:28.321253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:20:53.915436+0000","scrub_duration":0.010088694,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.d","version":"224'15110","reported_seq":15205,"reported_epoch":224,"state":"active+clean","last_fresh":"2024-09-10T19:19:34.951821+0000","last_change":"2024-09-10T19:16:24.858800+0000","last_active":"2024-09-10T19:19:34.951821+0000","last_peered":"2024-09-10T19:19:34.951821+0000","last_clean":"2024-09-10T19:19:34.951821+0000","last_became_active":"2024-09-10T19:16:24.858460+0000","last_became_peered":"2024-09-10T19:16:24.858460+0000","last_unstale":"2024-09-10T19:19:34.951821+0000","last_undegraded":"2024-09-10T19:19:34.951821+0000","last_fullsized":"2024-09-10T19:19:34.951821+0000","mapping_epoch":154,"log_start":"208'11290","ondisk_log_start":"208'11290","created":148,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:16:11.777014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:16:11.777014+0000","last_clean_scrub_stamp":"2024-09-10T19:16:11.777014+0000","objects_scrubbed":0,"log_size":3820,"log_dups_size":0,"ondisk_log_size":3820,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":15110,"num_write_kb":483457,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1","version":"66'14","reported_seq":98,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-09-10T19:19:30.825318+0000","last_change":"2024-09-10T19:17:28.925690+0000","last_active":"2024-09-10T19:19:30.825318+0000","last_peered":"2024-09-10T19:19:30.825318+0000","last_clean":"2024-09-10T19:19:30.825318+0000","last_became_active":"2024-09-10T19:16:24.852821+0000","last_became_peered":"2024-09-10T19:16:24.852821+0000","last_unstale":"2024-09-10T19:19:30.825318+0000","last_undegraded":"2024-09-10T19:19:30.825318+0000","last_fullsized":"2024-09-10T19:19:30.825318+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'14","last_scrub_stamp":"2024-09-10T19:15:41.748101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:15:41.748101+0000","objects_scrubbed":7,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0055403099999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,7],"acting":[4,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":214,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-09-10T19:19:30.828065+0000","last_change":"2024-09-10T19:16:25.582072+0000","last_active":"2024-09-10T19:19:30.828065+0000","last_peered":"2024-09-10T19:19:30.828065+0000","last_clean":"2024-09-10T19:19:30.828065+0000","last_became_active":"2024-09-10T19:16:24.855861+0000","last_became_peered":"2024-09-10T19:16:24.855861+0000","last_unstale":"2024-09-10T19:19:30.828065+0000","last_undegraded":"2024-09-10T19:19:30.828065+0000","last_fullsized":"2024-09-10T19:19:30.828065+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:16:25.581993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:16:25.581993+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0041345990000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.0","version":"117'10","reported_seq":333,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-09-10T19:19:30.852548+0000","last_change":"2024-09-10T19:15:35.973188+0000","last_active":"2024-09-10T19:19:30.852548+0000","last_peered":"2024-09-10T19:19:30.852548+0000","last_clean":"2024-09-10T19:19:30.852548+0000","last_became_active":"2024-09-10T19:13:54.709846+0000","last_became_peered":"2024-09-10T19:13:54.709846+0000","last_unstale":"2024-09-10T19:19:30.852548+0000","last_undegraded":"2024-09-10T19:19:30.852548+0000","last_fullsized":"2024-09-10T19:19:30.852548+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"117'10","last_scrub_stamp":"2024-09-10T19:15:35.973095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:15:35.973095+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0076338860000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,0],"acting":[3,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":208,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-09-10T19:19:30.832000+0000","last_change":"2024-09-10T19:19:10.030136+0000","last_active":"2024-09-10T19:19:30.832000+0000","last_peered":"2024-09-10T19:19:30.832000+0000","last_clean":"2024-09-10T19:19:30.832000+0000","last_became_active":"2024-09-10T19:16:24.858619+0000","last_became_peered":"2024-09-10T19:16:24.858619+0000","last_unstale":"2024-09-10T19:19:30.832000+0000","last_undegraded":"2024-09-10T19:19:30.832000+0000","last_fullsized":"2024-09-10T19:19:30.832000+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:19:10.029951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:11:41.073833+0000","last_clean_scrub_stamp":"2024-09-10T19:19:10.029951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:20:30.253633+0000","scrub_duration":0.01007037,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.c","version":"224'15118","reported_seq":15204,"reported_epoch":224,"state":"active+clean","last_fresh":"2024-09-10T19:19:34.947334+0000","last_change":"2024-09-10T19:17:29.176247+0000","last_active":"2024-09-10T19:19:34.947334+0000","last_peered":"2024-09-10T19:19:34.947334+0000","last_clean":"2024-09-10T19:19:34.947334+0000","last_became_active":"2024-09-10T19:16:13.079527+0000","last_became_peered":"2024-09-10T19:16:13.079527+0000","last_unstale":"2024-09-10T19:19:34.947334+0000","last_undegraded":"2024-09-10T19:19:34.947334+0000","last_fullsized":"2024-09-10T19:19:34.947334+0000","mapping_epoch":148,"log_start":"208'11311","ondisk_log_start":"208'11311","created":148,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:16:11.777014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:16:11.777014+0000","last_clean_scrub_stamp":"2024-09-10T19:16:11.777014+0000","objects_scrubbed":0,"log_size":3807,"log_dups_size":0,"ondisk_log_size":3807,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15118,"num_write_kb":483776,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.0","version":"66'18","reported_seq":234,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-09-10T19:19:30.830088+0000","last_change":"2024-09-10T19:19:12.207158+0000","last_active":"2024-09-10T19:19:30.830088+0000","last_peered":"2024-09-10T19:19:30.830088+0000","last_clean":"2024-09-10T19:19:30.830088+0000","last_became_active":"2024-09-10T19:10:09.821836+0000","last_became_peered":"2024-09-10T19:10:09.821836+0000","last_unstale":"2024-09-10T19:19:30.830088+0000","last_undegraded":"2024-09-10T19:19:30.830088+0000","last_fullsized":"2024-09-10T19:19:30.830088+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'18","last_scrub_stamp":"2024-09-10T19:19:12.207076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:19:12.207076+0000","objects_scrubbed":8,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:20:20.937740+0000","scrub_duration":0.003631156,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":164,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-09-10T19:19:30.827777+0000","last_change":"2024-09-10T19:17:26.374435+0000","last_active":"2024-09-10T19:19:30.827777+0000","last_peered":"2024-09-10T19:19:30.827777+0000","last_clean":"2024-09-10T19:19:30.827777+0000","last_became_active":"2024-09-10T19:13:54.710968+0000","last_became_peered":"2024-09-10T19:13:54.710968+0000","last_unstale":"2024-09-10T19:19:30.827777+0000","last_undegraded":"2024-09-10T19:19:30.827777+0000","last_fullsized":"2024-09-10T19:19:30.827777+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:17:26.374376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:17:26.374376+0000","last_clean_scrub_stamp":"2024-09-10T19:17:26.374376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001299815,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,3],"acting":[6,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":199,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-09-10T19:19:30.825554+0000","last_change":"2024-09-10T19:16:30.991715+0000","last_active":"2024-09-10T19:19:30.825554+0000","last_peered":"2024-09-10T19:19:30.825554+0000","last_clean":"2024-09-10T19:19:30.825554+0000","last_became_active":"2024-09-10T19:10:11.698054+0000","last_became_peered":"2024-09-10T19:10:11.698054+0000","last_unstale":"2024-09-10T19:19:30.825554+0000","last_undegraded":"2024-09-10T19:19:30.825554+0000","last_fullsized":"2024-09-10T19:19:30.825554+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:16:30.991635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:16:30.991635+0000","last_clean_scrub_stamp":"2024-09-10T19:16:30.991635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0021810029999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":207,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-09-10T19:19:30.828953+0000","last_change":"2024-09-10T19:16:30.036748+0000","last_active":"2024-09-10T19:19:30.828953+0000","last_peered":"2024-09-10T19:19:30.828953+0000","last_clean":"2024-09-10T19:19:30.828953+0000","last_became_active":"2024-09-10T19:16:24.848109+0000","last_became_peered":"2024-09-10T19:16:24.848109+0000","last_unstale":"2024-09-10T19:19:30.828953+0000","last_undegraded":"2024-09-10T19:19:30.828953+0000","last_fullsized":"2024-09-10T19:19:30.828953+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:16:30.036662+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:16:30.036662+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0044984129999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.f","version":"224'14752","reported_seq":14846,"reported_epoch":224,"state":"active+clean","last_fresh":"2024-09-10T19:19:34.915317+0000","last_change":"2024-09-10T19:16:24.856828+0000","last_active":"2024-09-10T19:19:34.915317+0000","last_peered":"2024-09-10T19:19:34.915317+0000","last_clean":"2024-09-10T19:19:34.915317+0000","last_became_active":"2024-09-10T19:16:24.855393+0000","last_became_peered":"2024-09-10T19:16:24.855393+0000","last_unstale":"2024-09-10T19:19:34.915317+0000","last_undegraded":"2024-09-10T19:19:34.915317+0000","last_fullsized":"2024-09-10T19:19:34.915317+0000","mapping_epoch":154,"log_start":"209'11099","ondisk_log_start":"209'11099","created":148,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:16:11.777014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:16:11.777014+0000","last_clean_scrub_stamp":"2024-09-10T19:16:11.777014+0000","objects_scrubbed":0,"log_size":3653,"log_dups_size":0,"ondisk_log_size":3653,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14752,"num_write_kb":472064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.3","version":"104'27","reported_seq":245,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-09-10T19:19:30.828142+0000","last_change":"2024-09-10T19:16:35.246566+0000","last_active":"2024-09-10T19:19:30.828142+0000","last_peered":"2024-09-10T19:19:30.828142+0000","last_clean":"2024-09-10T19:19:30.828142+0000","last_became_active":"2024-09-10T19:10:09.823003+0000","last_became_peered":"2024-09-10T19:10:09.823003+0000","last_unstale":"2024-09-10T19:19:30.828142+0000","last_undegraded":"2024-09-10T19:19:30.828142+0000","last_fullsized":"2024-09-10T19:19:30.828142+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"104'27","last_scrub_stamp":"2024-09-10T19:16:35.246512+0000","last_deep_scrub":"104'27","last_deep_scrub_stamp":"2024-09-10T19:13:02.225944+0000","last_clean_scrub_stamp":"2024-09-10T19:16:35.246512+0000","objects_scrubbed":8,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0052296490000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":204,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-09-10T19:19:30.836956+0000","last_change":"2024-09-10T19:16:37.780086+0000","last_active":"2024-09-10T19:19:30.836956+0000","last_peered":"2024-09-10T19:19:30.836956+0000","last_clean":"2024-09-10T19:19:30.836956+0000","last_became_active":"2024-09-10T19:10:07.690325+0000","last_became_peered":"2024-09-10T19:10:07.690325+0000","last_unstale":"2024-09-10T19:19:30.836956+0000","last_undegraded":"2024-09-10T19:19:30.836956+0000","last_fullsized":"2024-09-10T19:19:30.836956+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:16:37.779967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:16:37.779967+0000","last_clean_scrub_stamp":"2024-09-10T19:16:37.779967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.067110102000000005,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":203,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-09-10T19:19:30.826845+0000","last_change":"2024-09-10T19:17:19.463698+0000","last_active":"2024-09-10T19:19:30.826845+0000","last_peered":"2024-09-10T19:19:30.826845+0000","last_clean":"2024-09-10T19:19:30.826845+0000","last_became_active":"2024-09-10T19:10:11.700370+0000","last_became_peered":"2024-09-10T19:10:11.700370+0000","last_unstale":"2024-09-10T19:19:30.826845+0000","last_undegraded":"2024-09-10T19:19:30.826845+0000","last_fullsized":"2024-09-10T19:19:30.826845+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:15:50.669821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:15:50.669821+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0015765,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":167,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-09-10T19:19:30.826586+0000","last_change":"2024-09-10T19:17:22.435156+0000","last_active":"2024-09-10T19:19:30.826586+0000","last_peered":"2024-09-10T19:19:30.826586+0000","last_clean":"2024-09-10T19:19:30.826586+0000","last_became_active":"2024-09-10T19:16:24.844699+0000","last_became_peered":"2024-09-10T19:16:24.844699+0000","last_unstale":"2024-09-10T19:19:30.826586+0000","last_undegraded":"2024-09-10T19:19:30.826586+0000","last_fullsized":"2024-09-10T19:19:30.826586+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:15:41.749338+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:15:41.749338+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0018841260000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.e","version":"224'14864","reported_seq":14957,"reported_epoch":224,"state":"active+clean","last_fresh":"2024-09-10T19:19:34.934473+0000","last_change":"2024-09-10T19:16:24.837044+0000","last_active":"2024-09-10T19:19:34.934473+0000","last_peered":"2024-09-10T19:19:34.934473+0000","last_clean":"2024-09-10T19:19:34.934473+0000","last_became_active":"2024-09-10T19:16:24.836458+0000","last_became_peered":"2024-09-10T19:16:24.836458+0000","last_unstale":"2024-09-10T19:19:34.934473+0000","last_undegraded":"2024-09-10T19:19:34.934473+0000","last_fullsized":"2024-09-10T19:19:34.934473+0000","mapping_epoch":154,"log_start":"208'11084","ondisk_log_start":"208'11084","created":148,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:16:11.777014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:16:11.777014+0000","last_clean_scrub_stamp":"2024-09-10T19:16:11.777014+0000","objects_scrubbed":0,"log_size":3780,"log_dups_size":0,"ondisk_log_size":3780,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14864,"num_write_kb":475648,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"66'10","reported_seq":170,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-09-10T19:19:30.835491+0000","last_change":"2024-09-10T19:19:26.575854+0000","last_active":"2024-09-10T19:19:30.835491+0000","last_peered":"2024-09-10T19:19:30.835491+0000","last_clean":"2024-09-10T19:19:30.835491+0000","last_became_active":"2024-09-10T19:13:54.714542+0000","last_became_peered":"2024-09-10T19:13:54.714542+0000","last_unstale":"2024-09-10T19:19:30.835491+0000","last_undegraded":"2024-09-10T19:19:30.835491+0000","last_fullsized":"2024-09-10T19:19:30.835491+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'10","last_scrub_stamp":"2024-09-10T19:19:26.575769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:19:26.575769+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:20:41.943835+0000","scrub_duration":0.022703509,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":14,"num_read_kb":9,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,0],"acting":[5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":173,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-09-10T19:19:30.828374+0000","last_change":"2024-09-10T19:17:28.203449+0000","last_active":"2024-09-10T19:19:30.828374+0000","last_peered":"2024-09-10T19:19:30.828374+0000","last_clean":"2024-09-10T19:19:30.828374+0000","last_became_active":"2024-09-10T19:16:24.836898+0000","last_became_peered":"2024-09-10T19:16:24.836898+0000","last_unstale":"2024-09-10T19:19:30.828374+0000","last_undegraded":"2024-09-10T19:19:30.828374+0000","last_fullsized":"2024-09-10T19:19:30.828374+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:17:28.203397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:17:28.203397+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0011258259999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3","version":"84'9","reported_seq":327,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-09-10T19:19:30.852432+0000","last_change":"2024-09-10T19:19:17.425038+0000","last_active":"2024-09-10T19:19:30.852432+0000","last_peered":"2024-09-10T19:19:30.852432+0000","last_clean":"2024-09-10T19:19:30.852432+0000","last_became_active":"2024-09-10T19:10:11.699690+0000","last_became_peered":"2024-09-10T19:10:11.699690+0000","last_unstale":"2024-09-10T19:19:30.852432+0000","last_undegraded":"2024-09-10T19:19:30.852432+0000","last_fullsized":"2024-09-10T19:19:30.852432+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'9","last_scrub_stamp":"2024-09-10T19:19:17.424943+0000","last_deep_scrub":"84'9","last_deep_scrub_stamp":"2024-09-10T19:11:53.879272+0000","last_clean_scrub_stamp":"2024-09-10T19:19:17.424943+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:20:32.534406+0000","scrub_duration":0.0083821910000000006,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":213,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-09-10T19:19:30.827392+0000","last_change":"2024-09-10T19:19:09.424372+0000","last_active":"2024-09-10T19:19:30.827392+0000","last_peered":"2024-09-10T19:19:30.827392+0000","last_clean":"2024-09-10T19:19:30.827392+0000","last_became_active":"2024-09-10T19:16:24.840096+0000","last_became_peered":"2024-09-10T19:16:24.840096+0000","last_unstale":"2024-09-10T19:19:30.827392+0000","last_undegraded":"2024-09-10T19:19:30.827392+0000","last_fullsized":"2024-09-10T19:19:30.827392+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:19:09.424289+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:19:09.424289+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:20:28.425816+0000","scrub_duration":0.001622833,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,5],"acting":[0,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.8","version":"224'10054","reported_seq":10149,"reported_epoch":224,"state":"active+clean","last_fresh":"2024-09-10T19:19:34.920569+0000","last_change":"2024-09-10T19:17:26.958494+0000","last_active":"2024-09-10T19:19:34.920569+0000","last_peered":"2024-09-10T19:19:34.920569+0000","last_clean":"2024-09-10T19:19:34.920569+0000","last_became_active":"2024-09-10T19:17:26.955617+0000","last_became_peered":"2024-09-10T19:17:26.955617+0000","last_unstale":"2024-09-10T19:19:34.920569+0000","last_undegraded":"2024-09-10T19:19:34.920569+0000","last_fullsized":"2024-09-10T19:19:34.920569+0000","mapping_epoch":148,"log_start":"190'6259","ondisk_log_start":"190'6259","created":148,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:16:11.777014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:16:11.777014+0000","last_clean_scrub_stamp":"2024-09-10T19:16:11.777014+0000","objects_scrubbed":0,"log_size":3795,"log_dups_size":0,"ondisk_log_size":3795,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":688128,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7238,"num_write_kb":231968,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"155'31","reported_seq":203,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-09-10T19:19:30.852757+0000","last_change":"2024-09-10T19:16:24.910865+0000","last_active":"2024-09-10T19:19:30.852757+0000","last_peered":"2024-09-10T19:19:30.852757+0000","last_clean":"2024-09-10T19:19:30.852757+0000","last_became_active":"2024-09-10T19:16:24.858154+0000","last_became_peered":"2024-09-10T19:16:24.858154+0000","last_unstale":"2024-09-10T19:19:30.852757+0000","last_undegraded":"2024-09-10T19:19:30.852757+0000","last_fullsized":"2024-09-10T19:19:30.852757+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"117'30","last_scrub_stamp":"2024-09-10T19:15:49.837844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:15:49.837844+0000","objects_scrubbed":0,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":47,"num_read_kb":32,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":358,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":211,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-09-10T19:19:30.835217+0000","last_change":"2024-09-10T19:15:54.746979+0000","last_active":"2024-09-10T19:19:30.835217+0000","last_peered":"2024-09-10T19:19:30.835217+0000","last_clean":"2024-09-10T19:19:30.835217+0000","last_became_active":"2024-09-10T19:13:54.730096+0000","last_became_peered":"2024-09-10T19:13:54.730096+0000","last_unstale":"2024-09-10T19:19:30.835217+0000","last_undegraded":"2024-09-10T19:19:30.835217+0000","last_fullsized":"2024-09-10T19:19:30.835217+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:15:54.746920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:15:54.746920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001300037,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":216,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-09-10T19:19:30.828591+0000","last_change":"2024-09-10T19:17:25.147473+0000","last_active":"2024-09-10T19:19:30.828591+0000","last_peered":"2024-09-10T19:19:30.828591+0000","last_clean":"2024-09-10T19:19:30.828591+0000","last_became_active":"2024-09-10T19:13:54.714017+0000","last_became_peered":"2024-09-10T19:13:54.714017+0000","last_unstale":"2024-09-10T19:19:30.828591+0000","last_undegraded":"2024-09-10T19:19:30.828591+0000","last_fullsized":"2024-09-10T19:19:30.828591+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:17:25.147396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:17:25.147396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.004390054,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,6],"acting":[0,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":188,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-09-10T19:19:30.833676+0000","last_change":"2024-09-10T19:16:32.818668+0000","last_active":"2024-09-10T19:19:30.833676+0000","last_peered":"2024-09-10T19:19:30.833676+0000","last_clean":"2024-09-10T19:19:30.833676+0000","last_became_active":"2024-09-10T19:10:13.721744+0000","last_became_peered":"2024-09-10T19:10:13.721744+0000","last_unstale":"2024-09-10T19:19:30.833676+0000","last_undegraded":"2024-09-10T19:19:30.833676+0000","last_fullsized":"2024-09-10T19:19:30.833676+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:16:32.818590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:16:32.818590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0047149059999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.b","version":"224'15064","reported_seq":15158,"reported_epoch":224,"state":"active+clean","last_fresh":"2024-09-10T19:19:34.946703+0000","last_change":"2024-09-10T19:16:24.852409+0000","last_active":"2024-09-10T19:19:34.946703+0000","last_peered":"2024-09-10T19:19:34.946703+0000","last_clean":"2024-09-10T19:19:34.946703+0000","last_became_active":"2024-09-10T19:16:24.850682+0000","last_became_peered":"2024-09-10T19:16:24.850682+0000","last_unstale":"2024-09-10T19:19:34.946703+0000","last_undegraded":"2024-09-10T19:19:34.946703+0000","last_fullsized":"2024-09-10T19:19:34.946703+0000","mapping_epoch":154,"log_start":"209'11488","ondisk_log_start":"209'11488","created":148,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:16:11.777014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:16:11.777014+0000","last_clean_scrub_stamp":"2024-09-10T19:16:11.777014+0000","objects_scrubbed":0,"log_size":3576,"log_dups_size":0,"ondisk_log_size":3576,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15064,"num_write_kb":482048,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.7","version":"82'18","reported_seq":227,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-09-10T19:19:30.836344+0000","last_change":"2024-09-10T19:15:55.777428+0000","last_active":"2024-09-10T19:19:30.836344+0000","last_peered":"2024-09-10T19:19:30.836344+0000","last_clean":"2024-09-10T19:19:30.836344+0000","last_became_active":"2024-09-10T19:13:54.712065+0000","last_became_peered":"2024-09-10T19:13:54.712065+0000","last_unstale":"2024-09-10T19:19:30.836344+0000","last_undegraded":"2024-09-10T19:19:30.836344+0000","last_fullsized":"2024-09-10T19:19:30.836344+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'18","last_scrub_stamp":"2024-09-10T19:15:55.777361+0000","last_deep_scrub":"82'18","last_deep_scrub_stamp":"2024-09-10T19:15:55.777361+0000","last_clean_scrub_stamp":"2024-09-10T19:15:55.777361+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0023560040000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":19,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.0","version":"159'6","reported_seq":315,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-09-10T19:19:34.241758+0000","last_change":"2024-09-10T19:16:32.650007+0000","last_active":"2024-09-10T19:19:34.241758+0000","last_peered":"2024-09-10T19:19:34.241758+0000","last_clean":"2024-09-10T19:19:34.241758+0000","last_became_active":"2024-09-10T19:16:24.846203+0000","last_became_peered":"2024-09-10T19:16:24.846203+0000","last_unstale":"2024-09-10T19:19:34.241758+0000","last_undegraded":"2024-09-10T19:19:34.241758+0000","last_fullsized":"2024-09-10T19:19:34.241758+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"159'6","last_scrub_stamp":"2024-09-10T19:16:32.649914+0000","last_deep_scrub":"159'6","last_deep_scrub_stamp":"2024-09-10T19:16:32.649914+0000","last_clean_scrub_stamp":"2024-09-10T19:16:32.649914+0000","objects_scrubbed":1,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.013939817,"objects_trimmed":0,"snaptrim_duration":0.00034454499999999997,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":196,"num_read_kb":191,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":389,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":92,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-09-10T19:19:30.830395+0000","last_change":"2024-09-10T19:16:33.811962+0000","last_active":"2024-09-10T19:19:30.830395+0000","last_peered":"2024-09-10T19:19:30.830395+0000","last_clean":"2024-09-10T19:19:30.830395+0000","last_became_active":"2024-09-10T19:16:24.857505+0000","last_became_peered":"2024-09-10T19:16:24.857505+0000","last_unstale":"2024-09-10T19:19:30.830395+0000","last_undegraded":"2024-09-10T19:19:30.830395+0000","last_fullsized":"2024-09-10T19:19:30.830395+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:16:33.811869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:16:33.811869+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0027920340000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":92,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-09-10T19:19:30.835819+0000","last_change":"2024-09-10T19:16:43.958311+0000","last_active":"2024-09-10T19:19:30.835819+0000","last_peered":"2024-09-10T19:19:30.835819+0000","last_clean":"2024-09-10T19:19:30.835819+0000","last_became_active":"2024-09-10T19:16:24.861481+0000","last_became_peered":"2024-09-10T19:16:24.861481+0000","last_unstale":"2024-09-10T19:19:30.835819+0000","last_undegraded":"2024-09-10T19:19:30.835819+0000","last_fullsized":"2024-09-10T19:19:30.835819+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:16:43.958224+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:16:43.958224+0000","last_clean_scrub_stamp":"2024-09-10T19:16:43.958224+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.015970275999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,4],"acting":[5,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":191,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-09-10T19:19:30.836500+0000","last_change":"2024-09-10T19:15:38.493179+0000","last_active":"2024-09-10T19:19:30.836500+0000","last_peered":"2024-09-10T19:19:30.836500+0000","last_clean":"2024-09-10T19:19:30.836500+0000","last_became_active":"2024-09-10T19:10:14.053697+0000","last_became_peered":"2024-09-10T19:10:14.053697+0000","last_unstale":"2024-09-10T19:19:30.836500+0000","last_undegraded":"2024-09-10T19:19:30.836500+0000","last_fullsized":"2024-09-10T19:19:30.836500+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:15:38.493102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:15:38.493102+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0015544280000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.a","version":"224'15390","reported_seq":15472,"reported_epoch":224,"state":"active+clean","last_fresh":"2024-09-10T19:19:34.949096+0000","last_change":"2024-09-10T19:16:24.865995+0000","last_active":"2024-09-10T19:19:34.949096+0000","last_peered":"2024-09-10T19:19:34.949096+0000","last_clean":"2024-09-10T19:19:34.949096+0000","last_became_active":"2024-09-10T19:16:24.864959+0000","last_became_peered":"2024-09-10T19:16:24.864959+0000","last_unstale":"2024-09-10T19:19:34.949096+0000","last_undegraded":"2024-09-10T19:19:34.949096+0000","last_fullsized":"2024-09-10T19:19:34.949096+0000","mapping_epoch":154,"log_start":"209'11625","ondisk_log_start":"209'11625","created":148,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:16:11.777014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:16:11.777014+0000","last_clean_scrub_stamp":"2024-09-10T19:16:11.777014+0000","objects_scrubbed":0,"log_size":3765,"log_dups_size":0,"ondisk_log_size":3765,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15390,"num_write_kb":492480,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.6","version":"221'18","reported_seq":286,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-09-10T19:19:30.828900+0000","last_change":"2024-09-10T19:19:29.606828+0000","last_active":"2024-09-10T19:19:30.828900+0000","last_peered":"2024-09-10T19:19:30.828900+0000","last_clean":"2024-09-10T19:19:30.828900+0000","last_became_active":"2024-09-10T19:16:24.844323+0000","last_became_peered":"2024-09-10T19:16:24.844323+0000","last_unstale":"2024-09-10T19:19:30.828900+0000","last_undegraded":"2024-09-10T19:19:30.828900+0000","last_fullsized":"2024-09-10T19:19:30.828900+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"221'18","last_scrub_stamp":"2024-09-10T19:19:29.606748+0000","last_deep_scrub":"221'18","last_deep_scrub_stamp":"2024-09-10T19:19:29.606748+0000","last_clean_scrub_stamp":"2024-09-10T19:19:29.606748+0000","objects_scrubbed":6,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:20:53.113038+0000","scrub_duration":0.020236845,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,5],"acting":[0,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"61'1","reported_seq":209,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-09-10T19:19:30.828760+0000","last_change":"2024-09-10T19:16:42.195540+0000","last_active":"2024-09-10T19:19:30.828760+0000","last_peered":"2024-09-10T19:19:30.828760+0000","last_clean":"2024-09-10T19:19:30.828760+0000","last_became_active":"2024-09-10T19:10:07.683259+0000","last_became_peered":"2024-09-10T19:10:07.683259+0000","last_unstale":"2024-09-10T19:19:30.828760+0000","last_undegraded":"2024-09-10T19:19:30.828760+0000","last_fullsized":"2024-09-10T19:19:30.828760+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'1","last_scrub_stamp":"2024-09-10T19:16:42.195463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:16:42.195463+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0016622189999999999,"objects_trimmed":0,"snaptrim_duration":0.0014341130000000001,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":211,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-09-10T19:19:30.832684+0000","last_change":"2024-09-10T19:15:30.542477+0000","last_active":"2024-09-10T19:19:30.832684+0000","last_peered":"2024-09-10T19:19:30.832684+0000","last_clean":"2024-09-10T19:19:30.832684+0000","last_became_active":"2024-09-10T19:13:54.721486+0000","last_became_peered":"2024-09-10T19:13:54.721486+0000","last_unstale":"2024-09-10T19:19:30.832684+0000","last_undegraded":"2024-09-10T19:19:30.832684+0000","last_fullsized":"2024-09-10T19:19:30.832684+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:15:30.542399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:15:30.542399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0025142609999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":217,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-09-10T19:19:30.835998+0000","last_change":"2024-09-10T19:17:28.070635+0000","last_active":"2024-09-10T19:19:30.835998+0000","last_peered":"2024-09-10T19:19:30.835998+0000","last_clean":"2024-09-10T19:19:30.835998+0000","last_became_active":"2024-09-10T19:13:54.711685+0000","last_became_peered":"2024-09-10T19:13:54.711685+0000","last_unstale":"2024-09-10T19:19:30.835998+0000","last_undegraded":"2024-09-10T19:19:30.835998+0000","last_fullsized":"2024-09-10T19:19:30.835998+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:17:28.070543+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:17:28.070543+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0017508000000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,4],"acting":[5,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":154,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-09-10T19:19:30.836410+0000","last_change":"2024-09-10T19:16:28.786490+0000","last_active":"2024-09-10T19:19:30.836410+0000","last_peered":"2024-09-10T19:19:30.836410+0000","last_clean":"2024-09-10T19:19:30.836410+0000","last_became_active":"2024-09-10T19:13:54.722937+0000","last_became_peered":"2024-09-10T19:13:54.722937+0000","last_unstale":"2024-09-10T19:19:30.836410+0000","last_undegraded":"2024-09-10T19:19:30.836410+0000","last_fullsized":"2024-09-10T19:19:30.836410+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:16:28.786056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:16:28.786056+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0082035739999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.9","version":"224'10440","reported_seq":10535,"reported_epoch":224,"state":"active+clean","last_fresh":"2024-09-10T19:19:34.826657+0000","last_change":"2024-09-10T19:17:26.964413+0000","last_active":"2024-09-10T19:19:34.826657+0000","last_peered":"2024-09-10T19:19:34.826657+0000","last_clean":"2024-09-10T19:19:34.826657+0000","last_became_active":"2024-09-10T19:17:26.961919+0000","last_became_peered":"2024-09-10T19:17:26.961919+0000","last_unstale":"2024-09-10T19:19:34.826657+0000","last_undegraded":"2024-09-10T19:19:34.826657+0000","last_fullsized":"2024-09-10T19:19:34.826657+0000","mapping_epoch":148,"log_start":"192'6730","ondisk_log_start":"192'6730","created":148,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:16:11.777014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:16:11.777014+0000","last_clean_scrub_stamp":"2024-09-10T19:16:11.777014+0000","objects_scrubbed":0,"log_size":3710,"log_dups_size":0,"ondisk_log_size":3710,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1048576,"num_objects":16,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7548,"num_write_kb":242048,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,0],"acting":[6,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"80'21","reported_seq":241,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-09-10T19:19:30.828952+0000","last_change":"2024-09-10T19:17:27.420321+0000","last_active":"2024-09-10T19:19:30.828952+0000","last_peered":"2024-09-10T19:19:30.828952+0000","last_clean":"2024-09-10T19:19:30.828952+0000","last_became_active":"2024-09-10T19:10:09.822184+0000","last_became_peered":"2024-09-10T19:10:09.822184+0000","last_unstale":"2024-09-10T19:19:30.828952+0000","last_undegraded":"2024-09-10T19:19:30.828952+0000","last_fullsized":"2024-09-10T19:19:30.828952+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"80'21","last_scrub_stamp":"2024-09-10T19:17:27.420267+0000","last_deep_scrub":"80'21","last_deep_scrub_stamp":"2024-09-10T19:11:39.836144+0000","last_clean_scrub_stamp":"2024-09-10T19:17:27.420267+0000","objects_scrubbed":7,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0018427859999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"63'2","reported_seq":211,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-09-10T19:19:30.833378+0000","last_change":"2024-09-10T19:16:35.962952+0000","last_active":"2024-09-10T19:19:30.833378+0000","last_peered":"2024-09-10T19:19:30.833378+0000","last_clean":"2024-09-10T19:19:30.833378+0000","last_became_active":"2024-09-10T19:10:07.684784+0000","last_became_peered":"2024-09-10T19:10:07.684784+0000","last_unstale":"2024-09-10T19:19:30.833378+0000","last_undegraded":"2024-09-10T19:19:30.833378+0000","last_fullsized":"2024-09-10T19:19:30.833378+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'2","last_scrub_stamp":"2024-09-10T19:16:35.962804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:16:35.962804+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.105395924,"objects_trimmed":0,"snaptrim_duration":0.00075522999999999996,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":214,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-09-10T19:19:30.835703+0000","last_change":"2024-09-10T19:16:24.858963+0000","last_active":"2024-09-10T19:19:30.835703+0000","last_peered":"2024-09-10T19:19:30.835703+0000","last_clean":"2024-09-10T19:19:30.835703+0000","last_became_active":"2024-09-10T19:16:24.858662+0000","last_became_peered":"2024-09-10T19:16:24.858662+0000","last_unstale":"2024-09-10T19:19:30.835703+0000","last_undegraded":"2024-09-10T19:19:30.835703+0000","last_fullsized":"2024-09-10T19:19:30.835703+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:13:50.411500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:13:50.411500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0032604869999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"74'79","reported_seq":249,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-09-10T19:19:30.830931+0000","last_change":"2024-09-10T19:19:11.070913+0000","last_active":"2024-09-10T19:19:30.830931+0000","last_peered":"2024-09-10T19:19:30.830931+0000","last_clean":"2024-09-10T19:19:30.830931+0000","last_became_active":"2024-09-10T19:09:56.634919+0000","last_became_peered":"2024-09-10T19:09:56.634919+0000","last_unstale":"2024-09-10T19:19:30.830931+0000","last_undegraded":"2024-09-10T19:19:30.830931+0000","last_fullsized":"2024-09-10T19:19:30.830931+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'79","last_scrub_stamp":"2024-09-10T19:19:11.070835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:08:00.699315+0000","last_clean_scrub_stamp":"2024-09-10T19:19:11.070835+0000","objects_scrubbed":2,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:20:23.072445+0000","scrub_duration":0.0076249890000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":210,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-09-10T19:19:30.831075+0000","last_change":"2024-09-10T19:19:14.030325+0000","last_active":"2024-09-10T19:19:30.831075+0000","last_peered":"2024-09-10T19:19:30.831075+0000","last_clean":"2024-09-10T19:19:30.831075+0000","last_became_active":"2024-09-10T19:16:24.851652+0000","last_became_peered":"2024-09-10T19:16:24.851652+0000","last_unstale":"2024-09-10T19:19:30.831075+0000","last_undegraded":"2024-09-10T19:19:30.831075+0000","last_fullsized":"2024-09-10T19:19:30.831075+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:19:14.030181+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:19:14.030181+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:20:22.296791+0000","scrub_duration":0.009598054,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,0],"acting":[7,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":198,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-09-10T19:19:30.836430+0000","last_change":"2024-09-10T19:19:11.381975+0000","last_active":"2024-09-10T19:19:30.836430+0000","last_peered":"2024-09-10T19:19:30.836430+0000","last_clean":"2024-09-10T19:19:30.836430+0000","last_became_active":"2024-09-10T19:10:13.720647+0000","last_became_peered":"2024-09-10T19:10:13.720647+0000","last_unstale":"2024-09-10T19:19:30.836430+0000","last_undegraded":"2024-09-10T19:19:30.836430+0000","last_fullsized":"2024-09-10T19:19:30.836430+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:19:11.381898+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:19:11.381898+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:20:28.772557+0000","scrub_duration":0.0040952130000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.6","version":"224'10427","reported_seq":10520,"reported_epoch":224,"state":"active+clean","last_fresh":"2024-09-10T19:19:34.872012+0000","last_change":"2024-09-10T19:17:26.955780+0000","last_active":"2024-09-10T19:19:34.872012+0000","last_peered":"2024-09-10T19:19:34.872012+0000","last_clean":"2024-09-10T19:19:34.872012+0000","last_became_active":"2024-09-10T19:17:26.953133+0000","last_became_peered":"2024-09-10T19:17:26.953133+0000","last_unstale":"2024-09-10T19:19:34.872012+0000","last_undegraded":"2024-09-10T19:19:34.872012+0000","last_fullsized":"2024-09-10T19:19:34.872012+0000","mapping_epoch":148,"log_start":"190'6600","ondisk_log_start":"190'6600","created":148,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:16:11.777014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:16:11.777014+0000","last_clean_scrub_stamp":"2024-09-10T19:16:11.777014+0000","objects_scrubbed":0,"log_size":3827,"log_dups_size":0,"ondisk_log_size":3827,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1474560,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7554,"num_write_kb":242464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,7],"acting":[0,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"66'19","reported_seq":276,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-09-10T19:19:30.827147+0000","last_change":"2024-09-10T19:19:13.609209+0000","last_active":"2024-09-10T19:19:30.827147+0000","last_peered":"2024-09-10T19:19:30.827147+0000","last_clean":"2024-09-10T19:19:30.827147+0000","last_became_active":"2024-09-10T19:13:54.731274+0000","last_became_peered":"2024-09-10T19:13:54.731274+0000","last_unstale":"2024-09-10T19:19:30.827147+0000","last_undegraded":"2024-09-10T19:19:30.827147+0000","last_fullsized":"2024-09-10T19:19:30.827147+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'19","last_scrub_stamp":"2024-09-10T19:19:13.609135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:19:13.609135+0000","objects_scrubbed":9,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:20:22.695605+0000","scrub_duration":0.010981822,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,5],"acting":[6,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":82,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-09-10T19:19:30.836527+0000","last_change":"2024-09-10T19:16:24.847051+0000","last_active":"2024-09-10T19:19:30.836527+0000","last_peered":"2024-09-10T19:19:30.836527+0000","last_clean":"2024-09-10T19:19:30.836527+0000","last_became_active":"2024-09-10T19:16:24.846745+0000","last_became_peered":"2024-09-10T19:16:24.846745+0000","last_unstale":"2024-09-10T19:19:30.836527+0000","last_undegraded":"2024-09-10T19:19:30.836527+0000","last_fullsized":"2024-09-10T19:19:30.836527+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:15:53.803057+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:15:53.803057+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3],"acting":[5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":91,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-09-10T19:19:30.827884+0000","last_change":"2024-09-10T19:16:27.240222+0000","last_active":"2024-09-10T19:19:30.827884+0000","last_peered":"2024-09-10T19:19:30.827884+0000","last_clean":"2024-09-10T19:19:30.827884+0000","last_became_active":"2024-09-10T19:16:24.844295+0000","last_became_peered":"2024-09-10T19:16:24.844295+0000","last_unstale":"2024-09-10T19:19:30.827884+0000","last_undegraded":"2024-09-10T19:19:30.827884+0000","last_fullsized":"2024-09-10T19:19:30.827884+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:16:27.240139+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:16:27.240139+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0020507199999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":208,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-09-10T19:19:30.830855+0000","last_change":"2024-09-10T19:19:09.055045+0000","last_active":"2024-09-10T19:19:30.830855+0000","last_peered":"2024-09-10T19:19:30.830855+0000","last_clean":"2024-09-10T19:19:30.830855+0000","last_became_active":"2024-09-10T19:16:24.858172+0000","last_became_peered":"2024-09-10T19:16:24.858172+0000","last_unstale":"2024-09-10T19:19:30.830855+0000","last_undegraded":"2024-09-10T19:19:30.830855+0000","last_fullsized":"2024-09-10T19:19:30.830855+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:19:09.054900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:19:09.054900+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:20:28.164015+0000","scrub_duration":0.0066344780000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,6],"acting":[7,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.7","version":"224'10357","reported_seq":10461,"reported_epoch":224,"state":"active+clean","last_fresh":"2024-09-10T19:19:34.949897+0000","last_change":"2024-09-10T19:17:26.944772+0000","last_active":"2024-09-10T19:19:34.949897+0000","last_peered":"2024-09-10T19:19:34.949897+0000","last_clean":"2024-09-10T19:19:34.949897+0000","last_became_active":"2024-09-10T19:17:26.942858+0000","last_became_peered":"2024-09-10T19:17:26.942858+0000","last_unstale":"2024-09-10T19:19:34.949897+0000","last_undegraded":"2024-09-10T19:19:34.949897+0000","last_fullsized":"2024-09-10T19:19:34.949897+0000","mapping_epoch":154,"log_start":"192'6754","ondisk_log_start":"192'6754","created":148,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:16:11.777014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:16:11.777014+0000","last_clean_scrub_stamp":"2024-09-10T19:16:11.777014+0000","objects_scrubbed":0,"log_size":3603,"log_dups_size":0,"ondisk_log_size":3603,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2523136,"num_objects":38,"num_object_clones":0,"num_object_copies":114,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":38,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7458,"num_write_kb":239904,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,4],"acting":[3,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.b","version":"66'9","reported_seq":240,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-09-10T19:19:30.828060+0000","last_change":"2024-09-10T19:18:26.964644+0000","last_active":"2024-09-10T19:19:30.828060+0000","last_peered":"2024-09-10T19:19:30.828060+0000","last_clean":"2024-09-10T19:19:30.828060+0000","last_became_active":"2024-09-10T19:13:54.710613+0000","last_became_peered":"2024-09-10T19:13:54.710613+0000","last_unstale":"2024-09-10T19:19:30.828060+0000","last_undegraded":"2024-09-10T19:19:30.828060+0000","last_fullsized":"2024-09-10T19:19:30.828060+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'9","last_scrub_stamp":"2024-09-10T19:15:37.846807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:15:37.846807+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.012189228,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":167,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-09-10T19:19:30.831913+0000","last_change":"2024-09-10T19:16:24.848979+0000","last_active":"2024-09-10T19:19:30.831913+0000","last_peered":"2024-09-10T19:19:30.831913+0000","last_clean":"2024-09-10T19:19:30.831913+0000","last_became_active":"2024-09-10T19:16:24.848668+0000","last_became_peered":"2024-09-10T19:16:24.848668+0000","last_unstale":"2024-09-10T19:19:30.831913+0000","last_undegraded":"2024-09-10T19:19:30.831913+0000","last_fullsized":"2024-09-10T19:19:30.831913+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:15:50.850576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:11:44.254513+0000","last_clean_scrub_stamp":"2024-09-10T19:15:50.850576+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001815419,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":95,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-09-10T19:19:30.828814+0000","last_change":"2024-09-10T19:19:11.465972+0000","last_active":"2024-09-10T19:19:30.828814+0000","last_peered":"2024-09-10T19:19:30.828814+0000","last_clean":"2024-09-10T19:19:30.828814+0000","last_became_active":"2024-09-10T19:16:24.845709+0000","last_became_peered":"2024-09-10T19:16:24.845709+0000","last_unstale":"2024-09-10T19:19:30.828814+0000","last_undegraded":"2024-09-10T19:19:30.828814+0000","last_fullsized":"2024-09-10T19:19:30.828814+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:19:11.465819+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:19:11.465819+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:20:18.036431+0000","scrub_duration":0.012180921000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":207,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-09-10T19:19:30.828670+0000","last_change":"2024-09-10T19:16:39.188737+0000","last_active":"2024-09-10T19:19:30.828670+0000","last_peered":"2024-09-10T19:19:30.828670+0000","last_clean":"2024-09-10T19:19:30.828670+0000","last_became_active":"2024-09-10T19:16:24.837483+0000","last_became_peered":"2024-09-10T19:16:24.837483+0000","last_unstale":"2024-09-10T19:19:30.828670+0000","last_undegraded":"2024-09-10T19:19:30.828670+0000","last_fullsized":"2024-09-10T19:19:30.828670+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:16:39.188682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:16:39.188682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0012633589999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.4","version":"224'10225","reported_seq":10333,"reported_epoch":224,"state":"active+clean","last_fresh":"2024-09-10T19:19:34.901647+0000","last_change":"2024-09-10T19:17:26.998151+0000","last_active":"2024-09-10T19:19:34.901647+0000","last_peered":"2024-09-10T19:19:34.901647+0000","last_clean":"2024-09-10T19:19:34.901647+0000","last_became_active":"2024-09-10T19:17:26.972029+0000","last_became_peered":"2024-09-10T19:17:26.972029+0000","last_unstale":"2024-09-10T19:19:34.901647+0000","last_undegraded":"2024-09-10T19:19:34.901647+0000","last_fullsized":"2024-09-10T19:19:34.901647+0000","mapping_epoch":154,"log_start":"190'6449","ondisk_log_start":"190'6449","created":148,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:16:11.777014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:16:11.777014+0000","last_clean_scrub_stamp":"2024-09-10T19:16:11.777014+0000","objects_scrubbed":0,"log_size":3776,"log_dups_size":0,"ondisk_log_size":3776,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1605632,"num_objects":24,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7362,"num_write_kb":236384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":65536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.8","version":"220'22","reported_seq":234,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-09-10T19:19:30.835812+0000","last_change":"2024-09-10T19:19:24.597946+0000","last_active":"2024-09-10T19:19:30.835812+0000","last_peered":"2024-09-10T19:19:30.835812+0000","last_clean":"2024-09-10T19:19:30.835812+0000","last_became_active":"2024-09-10T19:10:09.820950+0000","last_became_peered":"2024-09-10T19:10:09.820950+0000","last_unstale":"2024-09-10T19:19:30.835812+0000","last_undegraded":"2024-09-10T19:19:30.835812+0000","last_fullsized":"2024-09-10T19:19:30.835812+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"220'22","last_scrub_stamp":"2024-09-10T19:19:24.597866+0000","last_deep_scrub":"220'22","last_deep_scrub_stamp":"2024-09-10T19:19:24.597866+0000","last_clean_scrub_stamp":"2024-09-10T19:19:24.597866+0000","objects_scrubbed":7,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:20:41.284281+0000","scrub_duration":0.031475547,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":82,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-09-10T19:19:30.834068+0000","last_change":"2024-09-10T19:16:24.857533+0000","last_active":"2024-09-10T19:19:30.834068+0000","last_peered":"2024-09-10T19:19:30.834068+0000","last_clean":"2024-09-10T19:19:30.834068+0000","last_became_active":"2024-09-10T19:16:24.857253+0000","last_became_peered":"2024-09-10T19:16:24.857253+0000","last_unstale":"2024-09-10T19:19:30.834068+0000","last_undegraded":"2024-09-10T19:19:30.834068+0000","last_fullsized":"2024-09-10T19:19:30.834068+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:15:38.677960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:15:38.677960+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.9","version":"80'9","reported_seq":322,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-09-10T19:19:30.852906+0000","last_change":"2024-09-10T19:19:19.008253+0000","last_active":"2024-09-10T19:19:30.852906+0000","last_peered":"2024-09-10T19:19:30.852906+0000","last_clean":"2024-09-10T19:19:30.852906+0000","last_became_active":"2024-09-10T19:10:11.702878+0000","last_became_peered":"2024-09-10T19:10:11.702878+0000","last_unstale":"2024-09-10T19:19:30.852906+0000","last_undegraded":"2024-09-10T19:19:30.852906+0000","last_fullsized":"2024-09-10T19:19:30.852906+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"80'9","last_scrub_stamp":"2024-09-10T19:19:19.008106+0000","last_deep_scrub":"80'9","last_deep_scrub_stamp":"2024-09-10T19:11:35.174218+0000","last_clean_scrub_stamp":"2024-09-10T19:19:19.008106+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:20:25.426864+0000","scrub_duration":0.015626330000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":198,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-09-10T19:19:30.837201+0000","last_change":"2024-09-10T19:19:15.539622+0000","last_active":"2024-09-10T19:19:30.837201+0000","last_peered":"2024-09-10T19:19:30.837201+0000","last_clean":"2024-09-10T19:19:30.837201+0000","last_became_active":"2024-09-10T19:10:14.053875+0000","last_became_peered":"2024-09-10T19:10:14.053875+0000","last_unstale":"2024-09-10T19:19:30.837201+0000","last_undegraded":"2024-09-10T19:19:30.837201+0000","last_fullsized":"2024-09-10T19:19:30.837201+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:19:15.539488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:19:15.539488+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:20:22.964354+0000","scrub_duration":0.033498668000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.5","version":"224'10546","reported_seq":10640,"reported_epoch":224,"state":"active+clean","last_fresh":"2024-09-10T19:19:34.811304+0000","last_change":"2024-09-10T19:17:26.973165+0000","last_active":"2024-09-10T19:19:34.811304+0000","last_peered":"2024-09-10T19:19:34.811304+0000","last_clean":"2024-09-10T19:19:34.811304+0000","last_became_active":"2024-09-10T19:17:26.966286+0000","last_became_peered":"2024-09-10T19:17:26.966286+0000","last_unstale":"2024-09-10T19:19:34.811304+0000","last_undegraded":"2024-09-10T19:19:34.811304+0000","last_fullsized":"2024-09-10T19:19:34.811304+0000","mapping_epoch":148,"log_start":"192'6833","ondisk_log_start":"192'6833","created":148,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:16:11.777014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:16:11.777014+0000","last_clean_scrub_stamp":"2024-09-10T19:16:11.777014+0000","objects_scrubbed":0,"log_size":3713,"log_dups_size":0,"ondisk_log_size":3713,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7622,"num_write_kb":242624,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.9","version":"66'12","reported_seq":246,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-09-10T19:19:30.825636+0000","last_change":"2024-09-10T19:15:37.869593+0000","last_active":"2024-09-10T19:19:30.825636+0000","last_peered":"2024-09-10T19:19:30.825636+0000","last_clean":"2024-09-10T19:19:30.825636+0000","last_became_active":"2024-09-10T19:13:54.718152+0000","last_became_peered":"2024-09-10T19:13:54.718152+0000","last_unstale":"2024-09-10T19:19:30.825636+0000","last_undegraded":"2024-09-10T19:19:30.825636+0000","last_fullsized":"2024-09-10T19:19:30.825636+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'12","last_scrub_stamp":"2024-09-10T19:15:37.869451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:15:37.869451+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0025441249999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.f","version":"59'2","reported_seq":219,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-09-10T19:19:30.829694+0000","last_change":"2024-09-10T19:16:29.007627+0000","last_active":"2024-09-10T19:19:30.829694+0000","last_peered":"2024-09-10T19:19:30.829694+0000","last_clean":"2024-09-10T19:19:30.829694+0000","last_became_active":"2024-09-10T19:10:07.681719+0000","last_became_peered":"2024-09-10T19:10:07.681719+0000","last_unstale":"2024-09-10T19:19:30.829694+0000","last_undegraded":"2024-09-10T19:19:30.829694+0000","last_fullsized":"2024-09-10T19:19:30.829694+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'2","last_scrub_stamp":"2024-09-10T19:16:29.007251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:16:29.007251+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0079698319999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":14,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":167,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-09-10T19:19:30.827813+0000","last_change":"2024-09-10T19:18:29.009145+0000","last_active":"2024-09-10T19:19:30.827813+0000","last_peered":"2024-09-10T19:19:30.827813+0000","last_clean":"2024-09-10T19:19:30.827813+0000","last_became_active":"2024-09-10T19:16:24.840091+0000","last_became_peered":"2024-09-10T19:16:24.840091+0000","last_unstale":"2024-09-10T19:19:30.827813+0000","last_undegraded":"2024-09-10T19:19:30.827813+0000","last_fullsized":"2024-09-10T19:19:30.827813+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:15:31.776481+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:11:46.587846+0000","last_clean_scrub_stamp":"2024-09-10T19:15:31.776481+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001267142,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,4],"acting":[0,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":205,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-09-10T19:19:30.830171+0000","last_change":"2024-09-10T19:15:41.093923+0000","last_active":"2024-09-10T19:19:30.830171+0000","last_peered":"2024-09-10T19:19:30.830171+0000","last_clean":"2024-09-10T19:19:30.830171+0000","last_became_active":"2024-09-10T19:13:54.711804+0000","last_became_peered":"2024-09-10T19:13:54.711804+0000","last_unstale":"2024-09-10T19:19:30.830171+0000","last_undegraded":"2024-09-10T19:19:30.830171+0000","last_fullsized":"2024-09-10T19:19:30.830171+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:15:41.093783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:15:41.093783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0023833050000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.16","version":"66'9","reported_seq":233,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-09-10T19:19:30.828826+0000","last_change":"2024-09-10T19:19:16.444223+0000","last_active":"2024-09-10T19:19:30.828826+0000","last_peered":"2024-09-10T19:19:30.828826+0000","last_clean":"2024-09-10T19:19:30.828826+0000","last_became_active":"2024-09-10T19:10:09.822994+0000","last_became_peered":"2024-09-10T19:10:09.822994+0000","last_unstale":"2024-09-10T19:19:30.828826+0000","last_undegraded":"2024-09-10T19:19:30.828826+0000","last_fullsized":"2024-09-10T19:19:30.828826+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'9","last_scrub_stamp":"2024-09-10T19:19:16.444147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:19:16.444147+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:20:19.419312+0000","scrub_duration":0.0065392059999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":175,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-09-10T19:19:30.828977+0000","last_change":"2024-09-10T19:17:16.305585+0000","last_active":"2024-09-10T19:19:30.828977+0000","last_peered":"2024-09-10T19:19:30.828977+0000","last_clean":"2024-09-10T19:19:30.828977+0000","last_became_active":"2024-09-10T19:16:24.844313+0000","last_became_peered":"2024-09-10T19:16:24.844313+0000","last_unstale":"2024-09-10T19:19:30.828977+0000","last_undegraded":"2024-09-10T19:19:30.828977+0000","last_fullsized":"2024-09-10T19:19:30.828977+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:17:16.305525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:11:41.458523+0000","last_clean_scrub_stamp":"2024-09-10T19:17:16.305525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0070771209999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,6],"acting":[0,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":207,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-09-10T19:19:30.830279+0000","last_change":"2024-09-10T19:16:28.794727+0000","last_active":"2024-09-10T19:19:30.830279+0000","last_peered":"2024-09-10T19:19:30.830279+0000","last_clean":"2024-09-10T19:19:30.830279+0000","last_became_active":"2024-09-10T19:13:54.712275+0000","last_became_peered":"2024-09-10T19:13:54.712275+0000","last_unstale":"2024-09-10T19:19:30.830279+0000","last_undegraded":"2024-09-10T19:19:30.830279+0000","last_fullsized":"2024-09-10T19:19:30.830279+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:16:28.794649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:16:28.794649+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0027500379999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"66'1","reported_seq":95,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-09-10T19:19:30.829953+0000","last_change":"2024-09-10T19:16:24.862865+0000","last_active":"2024-09-10T19:19:30.829953+0000","last_peered":"2024-09-10T19:19:30.829953+0000","last_clean":"2024-09-10T19:19:30.829953+0000","last_became_active":"2024-09-10T19:16:24.837432+0000","last_became_peered":"2024-09-10T19:16:24.837432+0000","last_unstale":"2024-09-10T19:19:30.829953+0000","last_undegraded":"2024-09-10T19:19:30.829953+0000","last_fullsized":"2024-09-10T19:19:30.829953+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'1","last_scrub_stamp":"2024-09-10T19:15:48.839582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:15:48.839582+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":13,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":217,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-09-10T19:19:30.826885+0000","last_change":"2024-09-10T19:17:17.479805+0000","last_active":"2024-09-10T19:19:30.826885+0000","last_peered":"2024-09-10T19:19:30.826885+0000","last_clean":"2024-09-10T19:19:30.826885+0000","last_became_active":"2024-09-10T19:13:54.724027+0000","last_became_peered":"2024-09-10T19:13:54.724027+0000","last_unstale":"2024-09-10T19:19:30.826885+0000","last_undegraded":"2024-09-10T19:19:30.826885+0000","last_fullsized":"2024-09-10T19:19:30.826885+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:15:36.714327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:15:36.714327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001831287,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,5],"acting":[6,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"66'6","reported_seq":244,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-09-10T19:19:30.833294+0000","last_change":"2024-09-10T19:19:25.266265+0000","last_active":"2024-09-10T19:19:30.833294+0000","last_peered":"2024-09-10T19:19:30.833294+0000","last_clean":"2024-09-10T19:19:30.833294+0000","last_became_active":"2024-09-10T19:16:24.863398+0000","last_became_peered":"2024-09-10T19:16:24.863398+0000","last_unstale":"2024-09-10T19:19:30.833294+0000","last_undegraded":"2024-09-10T19:19:30.833294+0000","last_fullsized":"2024-09-10T19:19:30.833294+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'6","last_scrub_stamp":"2024-09-10T19:19:25.266162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:19:25.266162+0000","objects_scrubbed":3,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:20:41.361532+0000","scrub_duration":0.013851005,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":207,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-09-10T19:19:30.835893+0000","last_change":"2024-09-10T19:15:30.583958+0000","last_active":"2024-09-10T19:19:30.835893+0000","last_peered":"2024-09-10T19:19:30.835893+0000","last_clean":"2024-09-10T19:19:30.835893+0000","last_became_active":"2024-09-10T19:13:54.729917+0000","last_became_peered":"2024-09-10T19:13:54.729917+0000","last_unstale":"2024-09-10T19:19:30.835893+0000","last_undegraded":"2024-09-10T19:19:30.835893+0000","last_fullsized":"2024-09-10T19:19:30.835893+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:15:30.583878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:11:23.264793+0000","last_clean_scrub_stamp":"2024-09-10T19:15:30.583878+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0015146960000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":198,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-09-10T19:19:30.836335+0000","last_change":"2024-09-10T19:17:26.922378+0000","last_active":"2024-09-10T19:19:30.836335+0000","last_peered":"2024-09-10T19:19:30.836335+0000","last_clean":"2024-09-10T19:19:30.836335+0000","last_became_active":"2024-09-10T19:10:14.054360+0000","last_became_peered":"2024-09-10T19:10:14.054360+0000","last_unstale":"2024-09-10T19:19:30.836335+0000","last_undegraded":"2024-09-10T19:19:30.836335+0000","last_fullsized":"2024-09-10T19:19:30.836335+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:17:26.922305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:17:26.922305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0016181940000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.18","version":"223'10149","reported_seq":16406,"reported_epoch":224,"state":"active+clean","last_fresh":"2024-09-10T19:19:34.804118+0000","last_change":"2024-09-10T19:17:45.681866+0000","last_active":"2024-09-10T19:19:34.804118+0000","last_peered":"2024-09-10T19:19:34.804118+0000","last_clean":"2024-09-10T19:19:34.804118+0000","last_became_active":"2024-09-10T19:17:45.677163+0000","last_became_peered":"2024-09-10T19:17:45.677163+0000","last_unstale":"2024-09-10T19:19:34.804118+0000","last_undegraded":"2024-09-10T19:19:34.804118+0000","last_fullsized":"2024-09-10T19:19:34.804118+0000","mapping_epoch":186,"log_start":"190'6280","ondisk_log_start":"190'6280","created":177,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:16:11.777014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:16:11.777014+0000","last_clean_scrub_stamp":"2024-09-10T19:16:11.777014+0000","objects_scrubbed":0,"log_size":3869,"log_dups_size":0,"ondisk_log_size":3869,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7334,"num_write_kb":234464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3071,"num_bytes_recovered":201261056,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,5],"acting":[4,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.14","version":"80'15","reported_seq":263,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-09-10T19:19:30.832553+0000","last_change":"2024-09-10T19:16:24.881305+0000","last_active":"2024-09-10T19:19:30.832553+0000","last_peered":"2024-09-10T19:19:30.832553+0000","last_clean":"2024-09-10T19:19:30.832553+0000","last_became_active":"2024-09-10T19:16:24.850702+0000","last_became_peered":"2024-09-10T19:16:24.850702+0000","last_unstale":"2024-09-10T19:19:30.832553+0000","last_undegraded":"2024-09-10T19:19:30.832553+0000","last_fullsized":"2024-09-10T19:19:30.832553+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"80'15","last_scrub_stamp":"2024-09-10T19:15:46.019949+0000","last_deep_scrub":"80'15","last_deep_scrub_stamp":"2024-09-10T19:11:41.719310+0000","last_clean_scrub_stamp":"2024-09-10T19:15:46.019949+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0021039190000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,5],"acting":[3,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":204,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-09-10T19:19:30.835283+0000","last_change":"2024-09-10T19:19:17.588731+0000","last_active":"2024-09-10T19:19:30.835283+0000","last_peered":"2024-09-10T19:19:30.835283+0000","last_clean":"2024-09-10T19:19:30.835283+0000","last_became_active":"2024-09-10T19:10:07.685870+0000","last_became_peered":"2024-09-10T19:10:07.685870+0000","last_unstale":"2024-09-10T19:19:30.835283+0000","last_undegraded":"2024-09-10T19:19:30.835283+0000","last_fullsized":"2024-09-10T19:19:30.835283+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:19:17.588552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:19:17.588552+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:20:29.324857+0000","scrub_duration":0.0065035919999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"117'10","reported_seq":333,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-09-10T19:19:30.855775+0000","last_change":"2024-09-10T19:16:23.844790+0000","last_active":"2024-09-10T19:19:30.855775+0000","last_peered":"2024-09-10T19:19:30.855775+0000","last_clean":"2024-09-10T19:19:30.855775+0000","last_became_active":"2024-09-10T19:13:54.724780+0000","last_became_peered":"2024-09-10T19:13:54.724780+0000","last_unstale":"2024-09-10T19:19:30.855775+0000","last_undegraded":"2024-09-10T19:19:30.855775+0000","last_fullsized":"2024-09-10T19:19:30.855775+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"117'10","last_scrub_stamp":"2024-09-10T19:16:23.844724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:16:23.844724+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.012497955999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3],"acting":[5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":197,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-09-10T19:19:30.828495+0000","last_change":"2024-09-10T19:16:40.213422+0000","last_active":"2024-09-10T19:19:30.828495+0000","last_peered":"2024-09-10T19:19:30.828495+0000","last_clean":"2024-09-10T19:19:30.828495+0000","last_became_active":"2024-09-10T19:10:13.721035+0000","last_became_peered":"2024-09-10T19:10:13.721035+0000","last_unstale":"2024-09-10T19:19:30.828495+0000","last_undegraded":"2024-09-10T19:19:30.828495+0000","last_fullsized":"2024-09-10T19:19:30.828495+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:16:40.213321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:16:40.213321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0051408239999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.19","version":"224'10360","reported_seq":16899,"reported_epoch":224,"state":"active+clean","last_fresh":"2024-09-10T19:19:34.892910+0000","last_change":"2024-09-10T19:17:46.698410+0000","last_active":"2024-09-10T19:19:34.892910+0000","last_peered":"2024-09-10T19:19:34.892910+0000","last_clean":"2024-09-10T19:19:34.892910+0000","last_became_active":"2024-09-10T19:17:46.694716+0000","last_became_peered":"2024-09-10T19:17:46.694716+0000","last_unstale":"2024-09-10T19:19:34.892910+0000","last_undegraded":"2024-09-10T19:19:34.892910+0000","last_fullsized":"2024-09-10T19:19:34.892910+0000","mapping_epoch":187,"log_start":"190'6585","ondisk_log_start":"190'6585","created":177,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:16:11.777014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:16:11.777014+0000","last_clean_scrub_stamp":"2024-09-10T19:16:11.777014+0000","objects_scrubbed":0,"log_size":3775,"log_dups_size":0,"ondisk_log_size":3775,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7468,"num_write_kb":238720,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3210,"num_bytes_recovered":210370560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,0],"acting":[5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.15","version":"66'9","reported_seq":258,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-09-10T19:19:30.836643+0000","last_change":"2024-09-10T19:19:23.577305+0000","last_active":"2024-09-10T19:19:30.836643+0000","last_peered":"2024-09-10T19:19:30.836643+0000","last_clean":"2024-09-10T19:19:30.836643+0000","last_became_active":"2024-09-10T19:16:24.859740+0000","last_became_peered":"2024-09-10T19:16:24.859740+0000","last_unstale":"2024-09-10T19:19:30.836643+0000","last_undegraded":"2024-09-10T19:19:30.836643+0000","last_fullsized":"2024-09-10T19:19:30.836643+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'9","last_scrub_stamp":"2024-09-10T19:19:23.577158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:19:23.577158+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:20:32.990634+0000","scrub_duration":0.011983318,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":203,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-09-10T19:19:30.828274+0000","last_change":"2024-09-10T19:16:37.243942+0000","last_active":"2024-09-10T19:19:30.828274+0000","last_peered":"2024-09-10T19:19:30.828274+0000","last_clean":"2024-09-10T19:19:30.828274+0000","last_became_active":"2024-09-10T19:10:07.688499+0000","last_became_peered":"2024-09-10T19:10:07.688499+0000","last_unstale":"2024-09-10T19:19:30.828274+0000","last_undegraded":"2024-09-10T19:19:30.828274+0000","last_fullsized":"2024-09-10T19:19:30.828274+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:16:37.243859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:16:37.243859+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0052374580000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"155'9","reported_seq":335,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-09-10T19:19:30.852500+0000","last_change":"2024-09-10T19:19:16.257770+0000","last_active":"2024-09-10T19:19:30.852500+0000","last_peered":"2024-09-10T19:19:30.852500+0000","last_clean":"2024-09-10T19:19:30.852500+0000","last_became_active":"2024-09-10T19:16:24.850850+0000","last_became_peered":"2024-09-10T19:16:24.850850+0000","last_unstale":"2024-09-10T19:19:30.852500+0000","last_undegraded":"2024-09-10T19:19:30.852500+0000","last_fullsized":"2024-09-10T19:19:30.852500+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"155'9","last_scrub_stamp":"2024-09-10T19:19:16.257692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:19:16.257692+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:20:20.031544+0000","scrub_duration":0.0076401230000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,4],"acting":[3,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":210,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-09-10T19:19:30.829456+0000","last_change":"2024-09-10T19:19:12.504470+0000","last_active":"2024-09-10T19:19:30.829456+0000","last_peered":"2024-09-10T19:19:30.829456+0000","last_clean":"2024-09-10T19:19:30.829456+0000","last_became_active":"2024-09-10T19:16:24.854280+0000","last_became_peered":"2024-09-10T19:16:24.854280+0000","last_unstale":"2024-09-10T19:19:30.829456+0000","last_undegraded":"2024-09-10T19:19:30.829456+0000","last_fullsized":"2024-09-10T19:19:30.829456+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:19:12.504402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:19:12.504402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:20:41.166660+0000","scrub_duration":0.0034925469999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.12","version":"79'13","reported_seq":247,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-09-10T19:19:30.826971+0000","last_change":"2024-09-10T19:19:27.518153+0000","last_active":"2024-09-10T19:19:30.826971+0000","last_peered":"2024-09-10T19:19:30.826971+0000","last_clean":"2024-09-10T19:19:30.826971+0000","last_became_active":"2024-09-10T19:16:24.845450+0000","last_became_peered":"2024-09-10T19:16:24.845450+0000","last_unstale":"2024-09-10T19:19:30.826971+0000","last_undegraded":"2024-09-10T19:19:30.826971+0000","last_fullsized":"2024-09-10T19:19:30.826971+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'13","last_scrub_stamp":"2024-09-10T19:19:27.518064+0000","last_deep_scrub":"79'13","last_deep_scrub_stamp":"2024-09-10T19:11:22.072400+0000","last_clean_scrub_stamp":"2024-09-10T19:19:27.518064+0000","objects_scrubbed":4,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:20:28.586075+0000","scrub_duration":0.010862685,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.14","version":"59'1","reported_seq":211,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-09-10T19:19:30.827568+0000","last_change":"2024-09-10T19:16:27.594895+0000","last_active":"2024-09-10T19:19:30.827568+0000","last_peered":"2024-09-10T19:19:30.827568+0000","last_clean":"2024-09-10T19:19:30.827568+0000","last_became_active":"2024-09-10T19:10:07.683263+0000","last_became_peered":"2024-09-10T19:10:07.683263+0000","last_unstale":"2024-09-10T19:19:30.827568+0000","last_undegraded":"2024-09-10T19:19:30.827568+0000","last_fullsized":"2024-09-10T19:19:30.827568+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'1","last_scrub_stamp":"2024-09-10T19:16:27.594788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:16:27.594788+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0091369840000000008,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":362,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":207,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-09-10T19:19:30.829861+0000","last_change":"2024-09-10T19:15:55.925313+0000","last_active":"2024-09-10T19:19:30.829861+0000","last_peered":"2024-09-10T19:19:30.829861+0000","last_clean":"2024-09-10T19:19:30.829861+0000","last_became_active":"2024-09-10T19:13:54.730185+0000","last_became_peered":"2024-09-10T19:13:54.730185+0000","last_unstale":"2024-09-10T19:19:30.829861+0000","last_undegraded":"2024-09-10T19:19:30.829861+0000","last_fullsized":"2024-09-10T19:19:30.829861+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:15:55.925228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:15:55.925228+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0017202299999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":204,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-09-10T19:19:30.827462+0000","last_change":"2024-09-10T19:16:23.273671+0000","last_active":"2024-09-10T19:19:30.827462+0000","last_peered":"2024-09-10T19:19:30.827462+0000","last_clean":"2024-09-10T19:19:30.827462+0000","last_became_active":"2024-09-10T19:13:54.714861+0000","last_became_peered":"2024-09-10T19:13:54.714861+0000","last_unstale":"2024-09-10T19:19:30.827462+0000","last_undegraded":"2024-09-10T19:19:30.827462+0000","last_fullsized":"2024-09-10T19:19:30.827462+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:16:23.273615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:16:23.273615+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0013972139999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,6],"acting":[0,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"66'11","reported_seq":242,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-09-10T19:19:30.831737+0000","last_change":"2024-09-10T19:15:53.883613+0000","last_active":"2024-09-10T19:19:30.831737+0000","last_peered":"2024-09-10T19:19:30.831737+0000","last_clean":"2024-09-10T19:19:30.831737+0000","last_became_active":"2024-09-10T19:13:54.720034+0000","last_became_peered":"2024-09-10T19:13:54.720034+0000","last_unstale":"2024-09-10T19:19:30.831737+0000","last_undegraded":"2024-09-10T19:19:30.831737+0000","last_fullsized":"2024-09-10T19:19:30.831737+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'11","last_scrub_stamp":"2024-09-10T19:15:53.883548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:15:53.883548+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001858326,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":154,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-09-10T19:19:30.836715+0000","last_change":"2024-09-10T19:19:10.439796+0000","last_active":"2024-09-10T19:19:30.836715+0000","last_peered":"2024-09-10T19:19:30.836715+0000","last_clean":"2024-09-10T19:19:30.836715+0000","last_became_active":"2024-09-10T19:13:54.722947+0000","last_became_peered":"2024-09-10T19:13:54.722947+0000","last_unstale":"2024-09-10T19:19:30.836715+0000","last_undegraded":"2024-09-10T19:19:30.836715+0000","last_fullsized":"2024-09-10T19:19:30.836715+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:19:10.439659+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:11:10.031245+0000","last_clean_scrub_stamp":"2024-09-10T19:19:10.439659+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:20:17.099888+0000","scrub_duration":0.014001438,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,6],"acting":[5,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":154,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-09-10T19:19:30.835589+0000","last_change":"2024-09-10T19:15:36.666310+0000","last_active":"2024-09-10T19:19:30.835589+0000","last_peered":"2024-09-10T19:19:30.835589+0000","last_clean":"2024-09-10T19:19:30.835589+0000","last_became_active":"2024-09-10T19:13:54.729703+0000","last_became_peered":"2024-09-10T19:13:54.729703+0000","last_unstale":"2024-09-10T19:19:30.835589+0000","last_undegraded":"2024-09-10T19:19:30.835589+0000","last_fullsized":"2024-09-10T19:19:30.835589+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:15:36.666216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:15:36.666216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.003489884,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":198,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-09-10T19:19:30.830028+0000","last_change":"2024-09-10T19:19:13.231447+0000","last_active":"2024-09-10T19:19:30.830028+0000","last_peered":"2024-09-10T19:19:30.830028+0000","last_clean":"2024-09-10T19:19:30.830028+0000","last_became_active":"2024-09-10T19:10:13.724201+0000","last_became_peered":"2024-09-10T19:10:13.724201+0000","last_unstale":"2024-09-10T19:19:30.830028+0000","last_undegraded":"2024-09-10T19:19:30.830028+0000","last_fullsized":"2024-09-10T19:19:30.830028+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:19:13.231390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:19:13.231390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:20:25.623720+0000","scrub_duration":0.0029165770000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"222'6","reported_seq":233,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-09-10T19:19:30.830056+0000","last_change":"2024-09-10T19:19:30.328975+0000","last_active":"2024-09-10T19:19:30.830056+0000","last_peered":"2024-09-10T19:19:30.830056+0000","last_clean":"2024-09-10T19:19:30.830056+0000","last_became_active":"2024-09-10T19:13:54.711125+0000","last_became_peered":"2024-09-10T19:13:54.711125+0000","last_unstale":"2024-09-10T19:19:30.830056+0000","last_undegraded":"2024-09-10T19:19:30.830056+0000","last_fullsized":"2024-09-10T19:19:30.830056+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"222'6","last_scrub_stamp":"2024-09-10T19:19:30.328841+0000","last_deep_scrub":"222'6","last_deep_scrub_stamp":"2024-09-10T19:19:30.328841+0000","last_clean_scrub_stamp":"2024-09-10T19:19:30.328841+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:20:51.291574+0000","scrub_duration":0.011722731,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,4],"acting":[3,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":214,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-09-10T19:19:30.836379+0000","last_change":"2024-09-10T19:16:33.766560+0000","last_active":"2024-09-10T19:19:30.836379+0000","last_peered":"2024-09-10T19:19:30.836379+0000","last_clean":"2024-09-10T19:19:30.836379+0000","last_became_active":"2024-09-10T19:16:24.852460+0000","last_became_peered":"2024-09-10T19:16:24.852460+0000","last_unstale":"2024-09-10T19:19:30.836379+0000","last_undegraded":"2024-09-10T19:19:30.836379+0000","last_fullsized":"2024-09-10T19:19:30.836379+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:16:33.766474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:16:33.766474+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0042170250000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,3],"acting":[5,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":200,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-09-10T19:19:30.826654+0000","last_change":"2024-09-10T19:16:24.545188+0000","last_active":"2024-09-10T19:19:30.826654+0000","last_peered":"2024-09-10T19:19:30.826654+0000","last_clean":"2024-09-10T19:19:30.826654+0000","last_became_active":"2024-09-10T19:10:11.698536+0000","last_became_peered":"2024-09-10T19:10:11.698536+0000","last_unstale":"2024-09-10T19:19:30.826654+0000","last_undegraded":"2024-09-10T19:19:30.826654+0000","last_fullsized":"2024-09-10T19:19:30.826654+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:16:24.545134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:11:31.837944+0000","last_clean_scrub_stamp":"2024-09-10T19:16:24.545134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.00120104,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":218,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-09-10T19:19:30.830343+0000","last_change":"2024-09-10T19:19:23.027867+0000","last_active":"2024-09-10T19:19:30.830343+0000","last_peered":"2024-09-10T19:19:30.830343+0000","last_clean":"2024-09-10T19:19:30.830343+0000","last_became_active":"2024-09-10T19:16:24.852356+0000","last_became_peered":"2024-09-10T19:16:24.852356+0000","last_unstale":"2024-09-10T19:19:30.830343+0000","last_undegraded":"2024-09-10T19:19:30.830343+0000","last_fullsized":"2024-09-10T19:19:30.830343+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:19:23.027800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:19:23.027800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:20:25.338260+0000","scrub_duration":0.014396977,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,3],"acting":[7,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.11","version":"66'11","reported_seq":112,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-09-10T19:19:30.837516+0000","last_change":"2024-09-10T19:19:25.606746+0000","last_active":"2024-09-10T19:19:30.837516+0000","last_peered":"2024-09-10T19:19:30.837516+0000","last_clean":"2024-09-10T19:19:30.837516+0000","last_became_active":"2024-09-10T19:15:45.149864+0000","last_became_peered":"2024-09-10T19:15:45.149864+0000","last_unstale":"2024-09-10T19:19:30.837516+0000","last_undegraded":"2024-09-10T19:19:30.837516+0000","last_fullsized":"2024-09-10T19:19:30.837516+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'11","last_scrub_stamp":"2024-09-10T19:19:25.606598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:19:25.606598+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:20:36.591926+0000","scrub_duration":0.015282657,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":214,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-09-10T19:19:30.827955+0000","last_change":"2024-09-10T19:16:24.855851+0000","last_active":"2024-09-10T19:19:30.827955+0000","last_peered":"2024-09-10T19:19:30.827955+0000","last_clean":"2024-09-10T19:19:30.827955+0000","last_became_active":"2024-09-10T19:16:24.855541+0000","last_became_peered":"2024-09-10T19:16:24.855541+0000","last_unstale":"2024-09-10T19:19:30.827955+0000","last_undegraded":"2024-09-10T19:19:30.827955+0000","last_fullsized":"2024-09-10T19:19:30.827955+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:16:23.543097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:13:05.876655+0000","last_clean_scrub_stamp":"2024-09-10T19:16:23.543097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001181779,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,0],"acting":[6,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":200,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-09-10T19:19:30.830466+0000","last_change":"2024-09-10T19:19:20.045982+0000","last_active":"2024-09-10T19:19:30.830466+0000","last_peered":"2024-09-10T19:19:30.830466+0000","last_clean":"2024-09-10T19:19:30.830466+0000","last_became_active":"2024-09-10T19:10:11.699013+0000","last_became_peered":"2024-09-10T19:10:11.699013+0000","last_unstale":"2024-09-10T19:19:30.830466+0000","last_undegraded":"2024-09-10T19:19:30.830466+0000","last_fullsized":"2024-09-10T19:19:30.830466+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:19:20.045906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:11:57.112092+0000","last_clean_scrub_stamp":"2024-09-10T19:19:20.045906+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:20:42.628925+0000","scrub_duration":0.019773829999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":198,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-09-10T19:19:30.831494+0000","last_change":"2024-09-10T19:19:18.276122+0000","last_active":"2024-09-10T19:19:30.831494+0000","last_peered":"2024-09-10T19:19:30.831494+0000","last_clean":"2024-09-10T19:19:30.831494+0000","last_became_active":"2024-09-10T19:10:14.056172+0000","last_became_peered":"2024-09-10T19:10:14.056172+0000","last_unstale":"2024-09-10T19:19:30.831494+0000","last_undegraded":"2024-09-10T19:19:30.831494+0000","last_fullsized":"2024-09-10T19:19:30.831494+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:19:18.276069+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:19:18.276069+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:20:26.486746+0000","scrub_duration":0.0034003950000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.12","version":"224'10534","reported_seq":10655,"reported_epoch":224,"state":"active+remapped+backfill_toofull","last_fresh":"2024-09-10T19:19:34.866396+0000","last_change":"2024-09-10T19:19:30.042384+0000","last_active":"2024-09-10T19:19:34.866396+0000","last_peered":"2024-09-10T19:19:34.866396+0000","last_clean":"2024-09-10T19:17:27.172505+0000","last_became_active":"2024-09-10T19:17:30.027866+0000","last_became_peered":"2024-09-10T19:17:30.027866+0000","last_unstale":"2024-09-10T19:19:34.866396+0000","last_undegraded":"2024-09-10T19:19:34.866396+0000","last_fullsized":"2024-09-10T19:19:34.866396+0000","mapping_epoch":180,"log_start":"192'6733","ondisk_log_start":"192'6733","created":177,"last_epoch_clean":178,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:16:11.777014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:16:11.777014+0000","last_clean_scrub_stamp":"2024-09-10T19:16:11.777014+0000","objects_scrubbed":0,"log_size":3801,"log_dups_size":0,"ondisk_log_size":3801,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":458752,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":14,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7617,"num_write_kb":243968,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,4],"acting":[0,4,5],"avail_no_missing":["0","4","5"],"object_location_counts":[{"shards":"0,4,5","objects":7}],"blocked_by":[],"up_primary":6,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1c","version":"80'2","reported_seq":214,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-09-10T19:19:30.831055+0000","last_change":"2024-09-10T19:16:32.648844+0000","last_active":"2024-09-10T19:19:30.831055+0000","last_peered":"2024-09-10T19:19:30.831055+0000","last_clean":"2024-09-10T19:19:30.831055+0000","last_became_active":"2024-09-10T19:16:24.851220+0000","last_became_peered":"2024-09-10T19:16:24.851220+0000","last_unstale":"2024-09-10T19:19:30.831055+0000","last_undegraded":"2024-09-10T19:19:30.831055+0000","last_fullsized":"2024-09-10T19:19:30.831055+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"80'2","last_scrub_stamp":"2024-09-10T19:16:32.648738+0000","last_deep_scrub":"80'2","last_deep_scrub_stamp":"2024-09-10T19:11:30.272043+0000","last_clean_scrub_stamp":"2024-09-10T19:16:32.648738+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0066232920000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":369,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":369,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,3],"acting":[7,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":214,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-09-10T19:19:30.836811+0000","last_change":"2024-09-10T19:19:19.581767+0000","last_active":"2024-09-10T19:19:30.836811+0000","last_peered":"2024-09-10T19:19:30.836811+0000","last_clean":"2024-09-10T19:19:30.836811+0000","last_became_active":"2024-09-10T19:19:19.379420+0000","last_became_peered":"2024-09-10T19:19:19.379420+0000","last_unstale":"2024-09-10T19:19:30.836811+0000","last_undegraded":"2024-09-10T19:19:30.836811+0000","last_fullsized":"2024-09-10T19:19:30.836811+0000","mapping_epoch":217,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":218,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:19:19.581564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:19:19.581564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:20:20.362854+0000","scrub_duration":0.01041663,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,0],"acting":[5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1e","version":"80'15","reported_seq":221,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-09-10T19:19:30.828744+0000","last_change":"2024-09-10T19:16:33.258114+0000","last_active":"2024-09-10T19:19:30.828744+0000","last_peered":"2024-09-10T19:19:30.828744+0000","last_clean":"2024-09-10T19:19:30.828744+0000","last_became_active":"2024-09-10T19:10:09.823061+0000","last_became_peered":"2024-09-10T19:10:09.823061+0000","last_unstale":"2024-09-10T19:19:30.828744+0000","last_undegraded":"2024-09-10T19:19:30.828744+0000","last_fullsized":"2024-09-10T19:19:30.828744+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"80'15","last_scrub_stamp":"2024-09-10T19:16:33.258031+0000","last_deep_scrub":"80'15","last_deep_scrub_stamp":"2024-09-10T19:11:25.883919+0000","last_clean_scrub_stamp":"2024-09-10T19:16:33.258031+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0040337309999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1f","version":"66'8","reported_seq":327,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-09-10T19:19:30.852791+0000","last_change":"2024-09-10T19:17:32.477395+0000","last_active":"2024-09-10T19:19:30.852791+0000","last_peered":"2024-09-10T19:19:30.852791+0000","last_clean":"2024-09-10T19:19:30.852791+0000","last_became_active":"2024-09-10T19:10:11.695427+0000","last_became_peered":"2024-09-10T19:10:11.695427+0000","last_unstale":"2024-09-10T19:19:30.852791+0000","last_undegraded":"2024-09-10T19:19:30.852791+0000","last_fullsized":"2024-09-10T19:19:30.852791+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'8","last_scrub_stamp":"2024-09-10T19:15:53.664208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:15:53.664208+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001849699,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.13","version":"224'10661","reported_seq":19700,"reported_epoch":224,"state":"active+clean","last_fresh":"2024-09-10T19:19:34.933404+0000","last_change":"2024-09-10T19:18:56.828838+0000","last_active":"2024-09-10T19:19:34.933404+0000","last_peered":"2024-09-10T19:19:34.933404+0000","last_clean":"2024-09-10T19:19:34.933404+0000","last_became_active":"2024-09-10T19:18:56.797038+0000","last_became_peered":"2024-09-10T19:18:56.797038+0000","last_unstale":"2024-09-10T19:19:34.933404+0000","last_undegraded":"2024-09-10T19:19:34.933404+0000","last_fullsized":"2024-09-10T19:19:34.933404+0000","mapping_epoch":211,"log_start":"192'7054","ondisk_log_start":"192'7054","created":177,"last_epoch_clean":212,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:16:11.777014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:16:11.777014+0000","last_clean_scrub_stamp":"2024-09-10T19:16:11.777014+0000","objects_scrubbed":0,"log_size":3607,"log_dups_size":0,"ondisk_log_size":3607,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7651,"num_write_kb":243808,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3911,"num_bytes_recovered":256278528,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,4],"acting":[3,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":164,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-09-10T19:19:30.835817+0000","last_change":"2024-09-10T19:16:24.863000+0000","last_active":"2024-09-10T19:19:30.835817+0000","last_peered":"2024-09-10T19:19:30.835817+0000","last_clean":"2024-09-10T19:19:30.835817+0000","last_became_active":"2024-09-10T19:16:24.862695+0000","last_became_peered":"2024-09-10T19:16:24.862695+0000","last_unstale":"2024-09-10T19:19:30.835817+0000","last_undegraded":"2024-09-10T19:19:30.835817+0000","last_fullsized":"2024-09-10T19:19:30.835817+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:15:29.580605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:15:29.580605+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0047886710000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":204,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-09-10T19:19:30.830429+0000","last_change":"2024-09-10T19:16:41.881053+0000","last_active":"2024-09-10T19:19:30.830429+0000","last_peered":"2024-09-10T19:19:30.830429+0000","last_clean":"2024-09-10T19:19:30.830429+0000","last_became_active":"2024-09-10T19:10:07.682689+0000","last_became_peered":"2024-09-10T19:10:07.682689+0000","last_unstale":"2024-09-10T19:19:30.830429+0000","last_undegraded":"2024-09-10T19:19:30.830429+0000","last_fullsized":"2024-09-10T19:19:30.830429+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:16:41.880992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:16:41.880992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0044984990000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"80'16","reported_seq":276,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-09-10T19:19:30.827892+0000","last_change":"2024-09-10T19:18:29.250077+0000","last_active":"2024-09-10T19:19:30.827892+0000","last_peered":"2024-09-10T19:19:30.827892+0000","last_clean":"2024-09-10T19:19:30.827892+0000","last_became_active":"2024-09-10T19:13:54.713563+0000","last_became_peered":"2024-09-10T19:13:54.713563+0000","last_unstale":"2024-09-10T19:19:30.827892+0000","last_undegraded":"2024-09-10T19:19:30.827892+0000","last_fullsized":"2024-09-10T19:19:30.827892+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"80'16","last_scrub_stamp":"2024-09-10T19:15:47.700523+0000","last_deep_scrub":"80'16","last_deep_scrub_stamp":"2024-09-10T19:11:40.853611+0000","last_clean_scrub_stamp":"2024-09-10T19:15:47.700523+0000","objects_scrubbed":5,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0019149010000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,0],"acting":[6,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":209,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-09-10T19:19:30.829046+0000","last_change":"2024-09-10T19:16:34.222429+0000","last_active":"2024-09-10T19:19:30.829046+0000","last_peered":"2024-09-10T19:19:30.829046+0000","last_clean":"2024-09-10T19:19:30.829046+0000","last_became_active":"2024-09-10T19:16:24.834713+0000","last_became_peered":"2024-09-10T19:16:24.834713+0000","last_unstale":"2024-09-10T19:19:30.829046+0000","last_undegraded":"2024-09-10T19:19:30.829046+0000","last_fullsized":"2024-09-10T19:19:30.829046+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:16:34.222365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:11:28.822327+0000","last_clean_scrub_stamp":"2024-09-10T19:16:34.222365+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.014893412999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.10","version":"224'10690","reported_seq":17491,"reported_epoch":224,"state":"active+clean","last_fresh":"2024-09-10T19:19:34.941594+0000","last_change":"2024-09-10T19:17:46.709272+0000","last_active":"2024-09-10T19:19:34.941594+0000","last_peered":"2024-09-10T19:19:34.941594+0000","last_clean":"2024-09-10T19:19:34.941594+0000","last_became_active":"2024-09-10T19:17:46.694818+0000","last_became_peered":"2024-09-10T19:17:46.694818+0000","last_unstale":"2024-09-10T19:19:34.941594+0000","last_undegraded":"2024-09-10T19:19:34.941594+0000","last_fullsized":"2024-09-10T19:19:34.941594+0000","mapping_epoch":187,"log_start":"192'7151","ondisk_log_start":"192'7151","created":177,"last_epoch_clean":188,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:16:11.777014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:16:11.777014+0000","last_clean_scrub_stamp":"2024-09-10T19:16:11.777014+0000","objects_scrubbed":0,"log_size":3539,"log_dups_size":0,"ondisk_log_size":3539,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7774,"num_write_kb":244288,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3344,"num_bytes_recovered":219152384,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,4],"acting":[7,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":197,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-09-10T19:19:30.828890+0000","last_change":"2024-09-10T19:16:36.960796+0000","last_active":"2024-09-10T19:19:30.828890+0000","last_peered":"2024-09-10T19:19:30.828890+0000","last_clean":"2024-09-10T19:19:30.828890+0000","last_became_active":"2024-09-10T19:10:14.053619+0000","last_became_peered":"2024-09-10T19:10:14.053619+0000","last_unstale":"2024-09-10T19:19:30.828890+0000","last_undegraded":"2024-09-10T19:19:30.828890+0000","last_fullsized":"2024-09-10T19:19:30.828890+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:16:36.960695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:16:36.960695+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.011430734999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":204,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-09-10T19:19:30.828186+0000","last_change":"2024-09-10T19:16:30.666084+0000","last_active":"2024-09-10T19:19:30.828186+0000","last_peered":"2024-09-10T19:19:30.828186+0000","last_clean":"2024-09-10T19:19:30.828186+0000","last_became_active":"2024-09-10T19:10:07.685997+0000","last_became_peered":"2024-09-10T19:10:07.685997+0000","last_unstale":"2024-09-10T19:19:30.828186+0000","last_undegraded":"2024-09-10T19:19:30.828186+0000","last_fullsized":"2024-09-10T19:19:30.828186+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:16:30.666001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:16:30.666001+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0041927830000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1c","version":"78'22","reported_seq":153,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-09-10T19:19:30.832469+0000","last_change":"2024-09-10T19:16:24.870445+0000","last_active":"2024-09-10T19:19:30.832469+0000","last_peered":"2024-09-10T19:19:30.832469+0000","last_clean":"2024-09-10T19:19:30.832469+0000","last_became_active":"2024-09-10T19:16:24.845764+0000","last_became_peered":"2024-09-10T19:16:24.845764+0000","last_unstale":"2024-09-10T19:19:30.832469+0000","last_undegraded":"2024-09-10T19:19:30.832469+0000","last_fullsized":"2024-09-10T19:19:30.832469+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"78'22","last_scrub_stamp":"2024-09-10T19:15:37.702624+0000","last_deep_scrub":"78'22","last_deep_scrub_stamp":"2024-09-10T19:11:21.118937+0000","last_clean_scrub_stamp":"2024-09-10T19:15:37.702624+0000","objects_scrubbed":7,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":157,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-09-10T19:19:30.831444+0000","last_change":"2024-09-10T19:15:33.008615+0000","last_active":"2024-09-10T19:19:30.831444+0000","last_peered":"2024-09-10T19:19:30.831444+0000","last_clean":"2024-09-10T19:19:30.831444+0000","last_became_active":"2024-09-10T19:13:54.709055+0000","last_became_peered":"2024-09-10T19:13:54.709055+0000","last_unstale":"2024-09-10T19:19:30.831444+0000","last_undegraded":"2024-09-10T19:19:30.831444+0000","last_fullsized":"2024-09-10T19:19:30.831444+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:15:33.008542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:15:33.008542+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.004557363,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,5],"acting":[4,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.11","version":"224'10393","reported_seq":19855,"reported_epoch":224,"state":"active+clean","last_fresh":"2024-09-10T19:19:34.911197+0000","last_change":"2024-09-10T19:18:24.020721+0000","last_active":"2024-09-10T19:19:34.911197+0000","last_peered":"2024-09-10T19:19:34.911197+0000","last_clean":"2024-09-10T19:19:34.911197+0000","last_became_active":"2024-09-10T19:18:24.009106+0000","last_became_peered":"2024-09-10T19:18:24.009106+0000","last_unstale":"2024-09-10T19:19:34.911197+0000","last_undegraded":"2024-09-10T19:19:34.911197+0000","last_fullsized":"2024-09-10T19:19:34.911197+0000","mapping_epoch":198,"log_start":"190'6642","ondisk_log_start":"190'6642","created":177,"last_epoch_clean":199,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:16:11.777014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:16:11.777014+0000","last_clean_scrub_stamp":"2024-09-10T19:16:11.777014+0000","objects_scrubbed":0,"log_size":3751,"log_dups_size":0,"ondisk_log_size":3751,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2654208,"num_objects":41,"num_object_clones":0,"num_object_copies":123,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":41,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7475,"num_write_kb":240480,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3110,"num_bytes_recovered":203816960,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":198,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-09-10T19:19:30.833168+0000","last_change":"2024-09-10T19:19:14.274104+0000","last_active":"2024-09-10T19:19:30.833168+0000","last_peered":"2024-09-10T19:19:30.833168+0000","last_clean":"2024-09-10T19:19:30.833168+0000","last_became_active":"2024-09-10T19:10:14.053698+0000","last_became_peered":"2024-09-10T19:10:14.053698+0000","last_unstale":"2024-09-10T19:19:30.833168+0000","last_undegraded":"2024-09-10T19:19:30.833168+0000","last_fullsized":"2024-09-10T19:19:30.833168+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:19:14.274039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:19:14.274039+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:20:19.701437+0000","scrub_duration":0.0039769159999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":204,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-09-10T19:19:30.831673+0000","last_change":"2024-09-10T19:19:15.244839+0000","last_active":"2024-09-10T19:19:30.831673+0000","last_peered":"2024-09-10T19:19:30.831673+0000","last_clean":"2024-09-10T19:19:30.831673+0000","last_became_active":"2024-09-10T19:10:07.682811+0000","last_became_peered":"2024-09-10T19:10:07.682811+0000","last_unstale":"2024-09-10T19:19:30.831673+0000","last_undegraded":"2024-09-10T19:19:30.831673+0000","last_fullsized":"2024-09-10T19:19:30.831673+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:19:15.244754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:19:15.244754+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:20:24.119370+0000","scrub_duration":0.0077546289999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1d","version":"66'12","reported_seq":244,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-09-10T19:19:30.831547+0000","last_change":"2024-09-10T19:15:52.923561+0000","last_active":"2024-09-10T19:19:30.831547+0000","last_peered":"2024-09-10T19:19:30.831547+0000","last_clean":"2024-09-10T19:19:30.831547+0000","last_became_active":"2024-09-10T19:13:54.714629+0000","last_became_peered":"2024-09-10T19:13:54.714629+0000","last_unstale":"2024-09-10T19:19:30.831547+0000","last_undegraded":"2024-09-10T19:19:30.831547+0000","last_fullsized":"2024-09-10T19:19:30.831547+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'12","last_scrub_stamp":"2024-09-10T19:15:52.923498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:15:52.923498+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0019450450000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":209,"reported_epoch":223,"state":"active+clean","last_fresh":"2024-09-10T19:19:30.831580+0000","last_change":"2024-09-10T19:16:37.970105+0000","last_active":"2024-09-10T19:19:30.831580+0000","last_peered":"2024-09-10T19:19:30.831580+0000","last_clean":"2024-09-10T19:19:30.831580+0000","last_became_active":"2024-09-10T19:16:24.843013+0000","last_became_peered":"2024-09-10T19:16:24.843013+0000","last_unstale":"2024-09-10T19:19:30.831580+0000","last_undegraded":"2024-09-10T19:19:30.831580+0000","last_fullsized":"2024-09-10T19:19:30.831580+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:16:37.970030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:16:37.970030+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.003625063,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":8,"num_pg":26,"stat_sum":{"num_bytes":24444928,"num_objects":371,"num_object_clones":0,"num_object_copies":1113,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":14,"num_objects_unfound":0,"num_objects_dirty":371,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":240537,"num_write_kb":7697921,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28890,"num_bytes_recovered":1893335040,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":487788544,"data_stored":487784472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":96455,"ondisk_log_size":96455,"up":78,"acting":78,"num_store_stats":7},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":382,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":1528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":73,"ondisk_log_size":73,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":177,"num_object_clones":0,"num_object_copies":531,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":177,"num_whiteouts":0,"num_read":695,"num_read_kb":452,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":173,"num_bytes_recovered":3218,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":540672,"data_stored":14242,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":460,"ondisk_log_size":460,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":196,"num_read_kb":191,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":389,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":2002,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":9,"ondisk_log_size":9,"up":9,"acting":9,"num_store_stats":7},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1328,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":27,"num_read_kb":27,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":3984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4,"ondisk_log_size":4,"up":96,"acting":96,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79,"ondisk_log_size":79,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":54,"seq":231928234102,"num_pgs":86,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":700536,"kb_used_data":114800,"kb_used_omap":1,"kb_used_meta":585726,"kb_avail":93044616,"statfs":{"total":95995035648,"available":95277686784,"internally_reserved":0,"allocated":117555200,"data_stored":117160644,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":599783768},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 19:19:05 2024","interfaces":[{"interface":"back","average":{"1min":1.1060000000000001,"5min":1.1259999999999999,"15min":1.093},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":4.5439999999999996,"5min":8.3770000000000007,"15min":35.551000000000002},"last":0.48499999999999999},{"interface":"front","average":{"1min":1.262,"5min":1.099,"15min":1.0149999999999999},"min":{"1min":0.245,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":4.5880000000000001,"5min":8.1440000000000001,"15min":35.503999999999998},"last":0.47499999999999998}]},{"osd":1,"last update":"Tue Sep 10 19:19:29 2024","interfaces":[{"interface":"back","average":{"1min":1.278,"5min":1.016,"15min":0.68799999999999994},"min":{"1min":0.25900000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":4.6589999999999998,"5min":8.3499999999999996,"15min":8.3499999999999996},"last":0.56799999999999995},{"interface":"front","average":{"1min":1.349,"5min":1.0069999999999999,"15min":0.80700000000000005},"min":{"1min":0.308,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":4.5810000000000004,"5min":5.609,"15min":5.609},"last":0.59199999999999997}]},{"osd":2,"last update":"Tue Sep 10 19:19:05 2024","interfaces":[{"interface":"back","average":{"1min":0.83899999999999997,"5min":0.81499999999999995,"15min":0.75900000000000001},"min":{"1min":0.28199999999999997,"5min":0.27300000000000002,"15min":0.25600000000000001},"max":{"1min":2.5569999999999999,"5min":4.21,"15min":31.940999999999999},"last":0.441},{"interface":"front","average":{"1min":1.085,"5min":1.177,"15min":0.91300000000000003},"min":{"1min":0.314,"5min":0.249,"15min":0.24099999999999999},"max":{"1min":4.609,"5min":8.0839999999999996,"15min":25.056000000000001},"last":0.53700000000000003}]},{"osd":3,"last update":"Tue Sep 10 19:19:05 2024","interfaces":[{"interface":"back","average":{"1min":0.90000000000000002,"5min":1.272,"15min":1.119},"min":{"1min":0.33600000000000002,"5min":0.217,"15min":0.217},"max":{"1min":2.343,"5min":21.074999999999999,"15min":35.277999999999999},"last":0.41299999999999998},{"interface":"front","average":{"1min":1.23,"5min":1.2969999999999999,"15min":1.137},"min":{"1min":0.25700000000000001,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":4.6349999999999998,"5min":18.82,"15min":36.512999999999998},"last":0.51700000000000002}]},{"osd":4,"last update":"Tue Sep 10 19:19:05 2024","interfaces":[{"interface":"back","average":{"1min":2.093,"5min":1.357,"15min":0.76100000000000001},"min":{"1min":0.23699999999999999,"5min":0.23300000000000001,"15min":0.22800000000000001},"max":{"1min":5.1150000000000002,"5min":8.1859999999999999,"15min":8.1859999999999999},"last":0.52800000000000002},{"interface":"front","average":{"1min":1.2849999999999999,"5min":0.90500000000000003,"15min":0.61399999999999999},"min":{"1min":0.23100000000000001,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":3.004,"5min":4.5469999999999997,"15min":4.5469999999999997},"last":3.9649999999999999}]},{"osd":5,"last update":"Tue Sep 10 19:19:05 2024","interfaces":[{"interface":"back","average":{"1min":3.4399999999999999,"5min":1.7629999999999999,"15min":0.94499999999999995},"min":{"1min":0.32600000000000001,"5min":0.253,"15min":0.23699999999999999},"max":{"1min":33.521000000000001,"5min":33.521000000000001,"15min":33.521000000000001},"last":1.7370000000000001},{"interface":"front","average":{"1min":3.492,"5min":1.694,"15min":0.89300000000000002},"min":{"1min":0.35199999999999998,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":34.438000000000002,"5min":34.438000000000002,"15min":34.438000000000002},"last":0.55400000000000005}]},{"osd":6,"last update":"Tue Sep 10 19:19:05 2024","interfaces":[{"interface":"back","average":{"1min":1.9370000000000001,"5min":1.472,"15min":0.82999999999999996},"min":{"1min":0.32300000000000001,"5min":0.317,"15min":0.26200000000000001},"max":{"1min":5.1109999999999998,"5min":8.2590000000000003,"15min":8.2590000000000003},"last":0.505},{"interface":"front","average":{"1min":1.3160000000000001,"5min":1.1120000000000001,"15min":0.69499999999999995},"min":{"1min":0.29199999999999998,"5min":0.29199999999999998,"15min":0.28000000000000003},"max":{"1min":3.677,"5min":8.4130000000000003,"15min":8.4130000000000003},"last":4}]}]},{"osd":6,"up_from":47,"seq":201863463035,"num_pgs":83,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":460268,"kb_used_data":162788,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93284884,"statfs":{"total":95995035648,"available":95523721216,"internally_reserved":0,"allocated":166694912,"data_stored":166312720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 19:18:47 2024","interfaces":[{"interface":"back","average":{"1min":0.996,"5min":2.2149999999999999,"15min":1.107},"min":{"1min":0.19500000000000001,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":5.2350000000000003,"5min":112.988,"15min":112.988},"last":0.85599999999999998},{"interface":"front","average":{"1min":1.139,"5min":2.319,"15min":1.161},"min":{"1min":0.26300000000000001,"5min":0.221,"15min":0.221},"max":{"1min":5.274,"5min":113.18300000000001,"15min":113.18300000000001},"last":1.5649999999999999}]},{"osd":1,"last update":"Tue Sep 10 19:19:30 2024","interfaces":[{"interface":"back","average":{"1min":0.997,"5min":0.70899999999999996,"15min":0.57899999999999996},"min":{"1min":0.32900000000000001,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":2.2360000000000002,"5min":4.2919999999999998,"15min":4.2919999999999998},"last":0.88400000000000001},{"interface":"front","average":{"1min":1.3580000000000001,"5min":0.92100000000000004,"15min":0.74299999999999999},"min":{"1min":0.30099999999999999,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":6.5289999999999999,"5min":6.5289999999999999,"15min":6.5289999999999999},"last":0.46000000000000002}]},{"osd":2,"last update":"Tue Sep 10 19:18:47 2024","interfaces":[{"interface":"back","average":{"1min":0.89500000000000002,"5min":2.0310000000000001,"15min":1.0569999999999999},"min":{"1min":0.27700000000000002,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":2.4510000000000001,"5min":112.864,"15min":112.864},"last":1.2050000000000001},{"interface":"front","average":{"1min":0.93100000000000005,"5min":2.1970000000000001,"15min":1.1060000000000001},"min":{"1min":0.315,"5min":0.254,"15min":0.254},"max":{"1min":2.2799999999999998,"5min":112.822,"15min":112.822},"last":1.5980000000000001}]},{"osd":3,"last update":"Tue Sep 10 19:18:47 2024","interfaces":[{"interface":"back","average":{"1min":1.1839999999999999,"5min":2.7759999999999998,"15min":1.306},"min":{"1min":0.251,"5min":0.251,"15min":0.251},"max":{"1min":5.3419999999999996,"5min":125.56399999999999,"15min":125.56399999999999},"last":0.496},{"interface":"front","average":{"1min":1.234,"5min":2.8260000000000001,"15min":1.3580000000000001},"min":{"1min":0.254,"5min":0.254,"15min":0.23200000000000001},"max":{"1min":6.5259999999999998,"5min":125.47,"15min":125.47},"last":1.1499999999999999}]},{"osd":4,"last update":"Tue Sep 10 19:18:47 2024","interfaces":[{"interface":"back","average":{"1min":1.4470000000000001,"5min":1.042,"15min":0.59599999999999997},"min":{"1min":0.27900000000000003,"5min":0.27600000000000002,"15min":0.20699999999999999},"max":{"1min":6.9269999999999996,"5min":6.9269999999999996,"15min":6.9269999999999996},"last":5.6070000000000002},{"interface":"front","average":{"1min":1.1259999999999999,"5min":0.88200000000000001,"15min":0.57199999999999995},"min":{"1min":0.26100000000000001,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":3.492,"5min":4.0750000000000002,"15min":4.0750000000000002},"last":0.58999999999999997}]},{"osd":5,"last update":"Tue Sep 10 19:18:47 2024","interfaces":[{"interface":"back","average":{"1min":1.742,"5min":1.625,"15min":0.80400000000000005},"min":{"1min":0.34799999999999998,"5min":0.30499999999999999,"15min":0.248},"max":{"1min":5.008,"5min":44.765000000000001,"15min":44.765000000000001},"last":2.4510000000000001},{"interface":"front","average":{"1min":1.2969999999999999,"5min":1.6240000000000001,"15min":0.82599999999999996},"min":{"1min":0.29699999999999999,"5min":0.28899999999999998,"15min":0.27500000000000002},"max":{"1min":4.2460000000000004,"5min":43.036000000000001,"15min":43.036000000000001},"last":4.2800000000000002}]},{"osd":7,"last update":"Tue Sep 10 19:19:11 2024","interfaces":[{"interface":"back","average":{"1min":1.452,"5min":1.1180000000000001,"15min":0.67800000000000005},"min":{"1min":0.39000000000000001,"5min":0.30599999999999999,"15min":0.27500000000000002},"max":{"1min":4.1379999999999999,"5min":8.9390000000000001,"15min":8.9390000000000001},"last":1.1819999999999999},{"interface":"front","average":{"1min":1.6870000000000001,"5min":1.28,"15min":0.752},"min":{"1min":0.42099999999999999,"5min":0.31900000000000001,"15min":0.29299999999999998},"max":{"1min":4.9029999999999996,"5min":10.298,"15min":10.298},"last":1.3100000000000001}]}]},{"osd":5,"up_from":39,"seq":167503724672,"num_pgs":80,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":353312,"kb_used_data":47576,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":93391840,"statfs":{"total":95995035648,"available":95633244160,"internally_reserved":0,"allocated":48717824,"data_stored":48348217,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 19:19:20 2024","interfaces":[{"interface":"back","average":{"1min":4.8390000000000004,"5min":1.9299999999999999,"15min":1.01},"min":{"1min":0.19900000000000001,"5min":0.17999999999999999,"15min":0.17999999999999999},"max":{"1min":73.933000000000007,"5min":73.933000000000007,"15min":73.933000000000007},"last":1.294},{"interface":"front","average":{"1min":4.952,"5min":1.742,"15min":0.94499999999999995},"min":{"1min":0.19900000000000001,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":73.769999999999996,"5min":73.769999999999996,"15min":73.769999999999996},"last":0.57099999999999995}]},{"osd":1,"last update":"Tue Sep 10 19:19:24 2024","interfaces":[{"interface":"back","average":{"1min":1.454,"5min":0.96899999999999997,"15min":0.75700000000000001},"min":{"1min":0.26600000000000001,"5min":0.249,"15min":0.249},"max":{"1min":4.3499999999999996,"5min":4.3499999999999996,"15min":4.3499999999999996},"last":0.61699999999999999},{"interface":"front","average":{"1min":1.3109999999999999,"5min":0.90300000000000002,"15min":0.63600000000000001},"min":{"1min":0.26500000000000001,"5min":0.255,"15min":0.255},"max":{"1min":3.8959999999999999,"5min":5.0570000000000004,"15min":5.0570000000000004},"last":0.34100000000000003}]},{"osd":2,"last update":"Tue Sep 10 19:19:20 2024","interfaces":[{"interface":"back","average":{"1min":1.0569999999999999,"5min":1.137,"15min":0.73699999999999999},"min":{"1min":0.27600000000000002,"5min":0.26100000000000001,"15min":0.253},"max":{"1min":3.6819999999999999,"5min":9.9399999999999995,"15min":9.9399999999999995},"last":1.325},{"interface":"front","average":{"1min":1.3700000000000001,"5min":1.083,"15min":0.76900000000000002},"min":{"1min":0.307,"5min":0.24399999999999999,"15min":0.224},"max":{"1min":4.4139999999999997,"5min":4.4139999999999997,"15min":6.3789999999999996},"last":0.67800000000000005}]},{"osd":3,"last update":"Tue Sep 10 19:19:20 2024","interfaces":[{"interface":"back","average":{"1min":1.04,"5min":1.113,"15min":0.74299999999999999},"min":{"1min":0.23300000000000001,"5min":0.224,"15min":0.224},"max":{"1min":3.6619999999999999,"5min":9.9019999999999992,"15min":9.9019999999999992},"last":1.266},{"interface":"front","average":{"1min":1.575,"5min":1.1319999999999999,"15min":0.75600000000000001},"min":{"1min":0.28899999999999998,"5min":0.27800000000000002,"15min":0.248},"max":{"1min":4.4459999999999997,"5min":5.3869999999999996,"15min":5.3869999999999996},"last":0.71699999999999997}]},{"osd":4,"last update":"Tue Sep 10 19:19:20 2024","interfaces":[{"interface":"back","average":{"1min":1.538,"5min":1.2909999999999999,"15min":0.70699999999999996},"min":{"1min":0.27400000000000002,"5min":0.24399999999999999,"15min":0.20699999999999999},"max":{"1min":4.1520000000000001,"5min":10.414,"15min":10.414},"last":5.2480000000000002},{"interface":"front","average":{"1min":1.3640000000000001,"5min":1.2729999999999999,"15min":0.73399999999999999},"min":{"1min":0.23100000000000001,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":4.1870000000000003,"5min":10.518000000000001,"15min":10.518000000000001},"last":1.359}]},{"osd":6,"last update":"Tue Sep 10 19:18:47 2024","interfaces":[{"interface":"back","average":{"1min":0.92400000000000004,"5min":1.393,"15min":0.76600000000000001},"min":{"1min":0.29799999999999999,"5min":0.29099999999999998,"15min":0.223},"max":{"1min":2.0990000000000002,"5min":9.9670000000000005,"15min":9.9670000000000005},"last":1.8009999999999999},{"interface":"front","average":{"1min":1.3440000000000001,"5min":1.296,"15min":0.72599999999999998},"min":{"1min":0.38600000000000001,"5min":0.26900000000000002,"15min":0.25},"max":{"1min":4.7140000000000004,"5min":10.446,"15min":10.446},"last":0.77900000000000003}]},{"osd":7,"last update":"Tue Sep 10 19:19:10 2024","interfaces":[{"interface":"back","average":{"1min":1.798,"5min":1.3400000000000001,"15min":0.79700000000000004},"min":{"1min":0.32400000000000001,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":4.6829999999999998,"5min":5.9660000000000002,"15min":5.9660000000000002},"last":0.59399999999999997},{"interface":"front","average":{"1min":1.552,"5min":1.248,"15min":0.745},"min":{"1min":0.31900000000000001,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":4.9829999999999997,"5min":6.8840000000000003,"15min":6.8840000000000003},"last":0.309}]}]},{"osd":4,"up_from":32,"seq":137438953604,"num_pgs":78,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":412692,"kb_used_data":111180,"kb_used_omap":1,"kb_used_meta":301502,"kb_avail":93332460,"statfs":{"total":95995035648,"available":95572439040,"internally_reserved":0,"allocated":113848320,"data_stored":113489113,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":308738392},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 19:19:06 2024","interfaces":[{"interface":"back","average":{"1min":0.92300000000000004,"5min":1.536,"15min":0.875},"min":{"1min":0.216,"5min":0.187,"15min":0.187},"max":{"1min":3.7330000000000001,"5min":32.533000000000001,"15min":32.533000000000001},"last":2.3780000000000001},{"interface":"front","average":{"1min":0.95699999999999996,"5min":1.6100000000000001,"15min":0.89500000000000002},"min":{"1min":0.219,"5min":0.219,"15min":0.20999999999999999},"max":{"1min":4.6769999999999996,"5min":32.512999999999998,"15min":32.512999999999998},"last":3.258}]},{"osd":1,"last update":"Tue Sep 10 19:19:26 2024","interfaces":[{"interface":"back","average":{"1min":1.0960000000000001,"5min":0.95199999999999996,"15min":0.73599999999999999},"min":{"1min":0.24299999999999999,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":4.5380000000000003,"5min":7.9770000000000003,"15min":7.9770000000000003},"last":0.34100000000000003},{"interface":"front","average":{"1min":1.1339999999999999,"5min":0.83399999999999996,"15min":0.72799999999999998},"min":{"1min":0.27800000000000002,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":3.774,"5min":3.8260000000000001,"15min":3.8260000000000001},"last":2.3399999999999999}]},{"osd":2,"last update":"Tue Sep 10 19:19:06 2024","interfaces":[{"interface":"back","average":{"1min":0.98999999999999999,"5min":0.95099999999999996,"15min":0.68600000000000005},"min":{"1min":0.254,"5min":0.214,"15min":0.214},"max":{"1min":3.798,"5min":6.8869999999999996,"15min":6.8869999999999996},"last":2.4260000000000002},{"interface":"front","average":{"1min":1.0660000000000001,"5min":0.94299999999999995,"15min":0.68200000000000005},"min":{"1min":0.28699999999999998,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":4.7169999999999996,"5min":7.085,"15min":7.085},"last":3.3079999999999998}]},{"osd":3,"last update":"Tue Sep 10 19:19:06 2024","interfaces":[{"interface":"back","average":{"1min":1.1739999999999999,"5min":1.3879999999999999,"15min":0.83599999999999997},"min":{"1min":0.26100000000000001,"5min":0.26100000000000001,"15min":0.252},"max":{"1min":4.8479999999999999,"5min":35.451999999999998,"15min":35.451999999999998},"last":3.2839999999999998},{"interface":"front","average":{"1min":1.3340000000000001,"5min":1.6120000000000001,"15min":0.93100000000000005},"min":{"1min":0.30099999999999999,"5min":0.26300000000000001,"15min":0.26200000000000001},"max":{"1min":4.5060000000000002,"5min":32.880000000000003,"15min":32.880000000000003},"last":0.29099999999999998}]},{"osd":5,"last update":"Tue Sep 10 19:19:26 2024","interfaces":[{"interface":"back","average":{"1min":1.702,"5min":1.9930000000000001,"15min":0.93799999999999994},"min":{"1min":0.224,"5min":0.224,"15min":0.224},"max":{"1min":4.5839999999999996,"5min":67.379000000000005,"15min":67.379000000000005},"last":0.36199999999999999},{"interface":"front","average":{"1min":1.907,"5min":1.7549999999999999,"15min":0.876},"min":{"1min":0.33500000000000002,"5min":0.249,"15min":0.223},"max":{"1min":5.601,"5min":67.542000000000002,"15min":67.542000000000002},"last":2.6789999999999998}]},{"osd":6,"last update":"Tue Sep 10 19:18:56 2024","interfaces":[{"interface":"back","average":{"1min":1.6399999999999999,"5min":1.044,"15min":0.64200000000000002},"min":{"1min":0.48099999999999998,"5min":0.27400000000000002,"15min":0.253},"max":{"1min":4.7640000000000002,"5min":5.5289999999999999,"15min":5.5289999999999999},"last":2.403},{"interface":"front","average":{"1min":1.5840000000000001,"5min":1.099,"15min":0.65300000000000002},"min":{"1min":0.27700000000000002,"5min":0.247,"15min":0.247},"max":{"1min":4.7469999999999999,"5min":7.9550000000000001,"15min":7.9550000000000001},"last":1.427}]},{"osd":7,"last update":"Tue Sep 10 19:19:12 2024","interfaces":[{"interface":"back","average":{"1min":1.2190000000000001,"5min":1.2030000000000001,"15min":0.85099999999999998},"min":{"1min":0.32800000000000001,"5min":0.28100000000000003,"15min":0.27700000000000002},"max":{"1min":4.5629999999999997,"5min":10.33,"15min":23.081},"last":0.318},{"interface":"front","average":{"1min":1.746,"5min":1.232,"15min":0.85399999999999998},"min":{"1min":0.29099999999999998,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":8.8550000000000004,"5min":8.8550000000000004,"15min":22.666},"last":3.2240000000000002}]}]},{"osd":3,"up_from":26,"seq":111669149833,"num_pgs":84,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":717488,"kb_used_data":127528,"kb_used_omap":1,"kb_used_meta":589886,"kb_avail":93027664,"statfs":{"total":95995035648,"available":95260327936,"internally_reserved":0,"allocated":130588672,"data_stored":130202846,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":604043608},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 19:18:38 2024","interfaces":[{"interface":"back","average":{"1min":0.32100000000000001,"5min":1.0569999999999999,"15min":0.69399999999999995},"min":{"1min":0.157,"5min":0.157,"15min":0.157},"max":{"1min":0.71199999999999997,"5min":17.518000000000001,"15min":17.518000000000001},"last":0.754},{"interface":"front","average":{"1min":0.56599999999999995,"5min":1.3520000000000001,"15min":0.82899999999999996},"min":{"1min":0.17899999999999999,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":2.2850000000000001,"5min":17.609000000000002,"15min":17.609000000000002},"last":0.81799999999999995}]},{"osd":1,"last update":"Tue Sep 10 19:19:24 2024","interfaces":[{"interface":"back","average":{"1min":0.377,"5min":0.47699999999999998,"15min":0.51900000000000002},"min":{"1min":0.20499999999999999,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":1.149,"5min":2.6059999999999999,"15min":2.6059999999999999},"last":0.73599999999999999},{"interface":"front","average":{"1min":0.46600000000000003,"5min":0.94999999999999996,"15min":1.3620000000000001},"min":{"1min":0.26800000000000002,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":1.768,"5min":17.454999999999998,"15min":17.454999999999998},"last":0.77000000000000002}]},{"osd":2,"last update":"Tue Sep 10 19:18:38 2024","interfaces":[{"interface":"back","average":{"1min":0.41199999999999998,"5min":0.93799999999999994,"15min":0.64800000000000002},"min":{"1min":0.23599999999999999,"5min":0.20599999999999999,"15min":0.186},"max":{"1min":0.89500000000000002,"5min":5.3029999999999999,"15min":5.3029999999999999},"last":0.80500000000000005},{"interface":"front","average":{"1min":0.39900000000000002,"5min":1.0980000000000001,"15min":0.71299999999999997},"min":{"1min":0.27100000000000002,"5min":0.23300000000000001,"15min":0.19800000000000001},"max":{"1min":0.66900000000000004,"5min":17.806000000000001,"15min":17.806000000000001},"last":0.90600000000000003}]},{"osd":4,"last update":"Tue Sep 10 19:19:09 2024","interfaces":[{"interface":"back","average":{"1min":0.96799999999999997,"5min":1.085,"15min":0.85699999999999998},"min":{"1min":0.28000000000000003,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":2.339,"5min":17.834,"15min":17.834},"last":1.1259999999999999},{"interface":"front","average":{"1min":0.98299999999999998,"5min":1.357,"15min":0.97699999999999998},"min":{"1min":0.28299999999999997,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":3.339,"5min":17.553000000000001,"15min":17.553000000000001},"last":0.99299999999999999}]},{"osd":5,"last update":"Tue Sep 10 19:19:24 2024","interfaces":[{"interface":"back","average":{"1min":0.85899999999999999,"5min":1.8680000000000001,"15min":1.1040000000000001},"min":{"1min":0.27900000000000003,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":3.2709999999999999,"5min":80.942999999999998,"15min":80.942999999999998},"last":5.7210000000000001},{"interface":"front","average":{"1min":1.28,"5min":2.2240000000000002,"15min":1.2729999999999999},"min":{"1min":0.41099999999999998,"5min":0.22,"15min":0.22},"max":{"1min":4.835,"5min":81.489000000000004,"15min":81.489000000000004},"last":1.431}]},{"osd":6,"last update":"Tue Sep 10 19:18:46 2024","interfaces":[{"interface":"back","average":{"1min":0.94799999999999995,"5min":1.3180000000000001,"15min":0.90300000000000002},"min":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.30399999999999999},"max":{"1min":3.7999999999999998,"5min":17.861000000000001,"15min":17.861000000000001},"last":1.004},{"interface":"front","average":{"1min":0.72999999999999998,"5min":1.1759999999999999,"15min":0.79500000000000004},"min":{"1min":0.311,"5min":0.30499999999999999,"15min":0.26800000000000002},"max":{"1min":2.5449999999999999,"5min":17.687000000000001,"15min":17.687000000000001},"last":0.93200000000000005}]},{"osd":7,"last update":"Tue Sep 10 19:19:19 2024","interfaces":[{"interface":"back","average":{"1min":1.1819999999999999,"5min":1.264,"15min":0.90300000000000002},"min":{"1min":0.376,"5min":0.31,"15min":0.29399999999999998},"max":{"1min":3.3969999999999998,"5min":18.093,"15min":18.093},"last":2.3149999999999999},{"interface":"front","average":{"1min":1.835,"5min":1.341,"15min":0.97999999999999998},"min":{"1min":0.39100000000000001,"5min":0.35599999999999998,"15min":0.34100000000000003},"max":{"1min":10.359999999999999,"5min":17.780999999999999,"15min":17.780999999999999},"last":2.0920000000000001}]}]},{"osd":2,"up_from":0,"seq":77309411470,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":0,"seq":592705486900,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":8,"seq":34359738520,"num_pgs":80,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":357416,"kb_used_data":51744,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":93387736,"statfs":{"total":95995035648,"available":95629041664,"internally_reserved":0,"allocated":52985856,"data_stored":52608152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Sep 10 19:19:30 2024","interfaces":[{"interface":"back","average":{"1min":0.60399999999999998,"5min":0.40999999999999998,"15min":0.38200000000000001},"min":{"1min":0.20100000000000001,"5min":0.19,"15min":0.19},"max":{"1min":2.0819999999999999,"5min":2.0819999999999999,"15min":2.0819999999999999},"last":0.69999999999999996},{"interface":"front","average":{"1min":0.56699999999999995,"5min":0.41699999999999998,"15min":0.42199999999999999},"min":{"1min":0.21199999999999999,"5min":0.193,"15min":0.193},"max":{"1min":2.2839999999999998,"5min":2.2839999999999998,"15min":2.2839999999999998},"last":0.503}]},{"osd":2,"last update":"Tue Sep 10 19:19:30 2024","interfaces":[{"interface":"back","average":{"1min":0.68300000000000005,"5min":0.78700000000000003,"15min":0.61599999999999999},"min":{"1min":0.24299999999999999,"5min":0.22700000000000001,"15min":0.17199999999999999},"max":{"1min":2.0510000000000002,"5min":6.4640000000000004,"15min":6.4640000000000004},"last":0.73499999999999999},{"interface":"front","average":{"1min":0.61299999999999999,"5min":0.82699999999999996,"15min":0.59999999999999998},"min":{"1min":0.22600000000000001,"5min":0.22600000000000001,"15min":0.187},"max":{"1min":2.8660000000000001,"5min":9.9399999999999995,"15min":9.9399999999999995},"last":0.312}]},{"osd":3,"last update":"Tue Sep 10 19:18:47 2024","interfaces":[{"interface":"back","average":{"1min":0.46200000000000002,"5min":0.91300000000000003,"15min":0.64400000000000002},"min":{"1min":0.22700000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":0.83099999999999996,"5min":6.5590000000000002,"15min":6.5590000000000002},"last":0.57499999999999996},{"interface":"front","average":{"1min":0.55000000000000004,"5min":1.0800000000000001,"15min":0.69899999999999995},"min":{"1min":0.26400000000000001,"5min":0.26200000000000001,"15min":0.23000000000000001},"max":{"1min":1.2649999999999999,"5min":7.2140000000000004,"15min":7.2140000000000004},"last":0.63100000000000001}]},{"osd":4,"last update":"Tue Sep 10 19:19:02 2024","interfaces":[{"interface":"back","average":{"1min":1.0900000000000001,"5min":0.86199999999999999,"15min":0.71099999999999997},"min":{"1min":0.33100000000000002,"5min":0.28299999999999997,"15min":0.24399999999999999},"max":{"1min":3.6200000000000001,"5min":5.0789999999999997,"15min":5.8499999999999996},"last":1.1220000000000001},{"interface":"front","average":{"1min":0.93999999999999995,"5min":0.83299999999999996,"15min":0.70099999999999996},"min":{"1min":0.29399999999999998,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":3.5569999999999999,"5min":5.2640000000000002,"15min":5.2640000000000002},"last":0.34699999999999998}]},{"osd":5,"last update":"Tue Sep 10 19:19:30 2024","interfaces":[{"interface":"back","average":{"1min":1.5149999999999999,"5min":1.1399999999999999,"15min":0.81699999999999995},"min":{"1min":0.27100000000000002,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":4.327,"5min":5.3860000000000001,"15min":5.3860000000000001},"last":1.1870000000000001},{"interface":"front","average":{"1min":1.2150000000000001,"5min":0.96999999999999997,"15min":0.76400000000000001},"min":{"1min":0.33900000000000002,"5min":0.26000000000000001,"15min":0.253},"max":{"1min":3.5899999999999999,"5min":5.9749999999999996,"15min":5.9749999999999996},"last":0.66200000000000003}]},{"osd":6,"last update":"Tue Sep 10 19:18:47 2024","interfaces":[{"interface":"back","average":{"1min":0.83599999999999997,"5min":1.008,"15min":0.751},"min":{"1min":0.34799999999999998,"5min":0.312,"15min":0.27200000000000002},"max":{"1min":3.5259999999999998,"5min":5.8730000000000002,"15min":5.8730000000000002},"last":0.93600000000000005},{"interface":"front","average":{"1min":1.036,"5min":0.999,"15min":0.76200000000000001},"min":{"1min":0.378,"5min":0.34399999999999997,"15min":0.28699999999999998},"max":{"1min":4.8300000000000001,"5min":5.2850000000000001,"15min":5.2850000000000001},"last":0.60399999999999998}]},{"osd":7,"last update":"Tue Sep 10 19:19:15 2024","interfaces":[{"interface":"back","average":{"1min":1.0529999999999999,"5min":0.98199999999999998,"15min":0.77300000000000002},"min":{"1min":0.36299999999999999,"5min":0.318,"15min":0.318},"max":{"1min":3.7120000000000002,"5min":5.1479999999999997,"15min":5.1479999999999997},"last":0.372},{"interface":"front","average":{"1min":0.71199999999999997,"5min":1.0760000000000001,"15min":0.77700000000000002},"min":{"1min":0.35199999999999998,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":1.3240000000000001,"5min":6.0469999999999997,"15min":6.0469999999999997},"last":0.81399999999999995}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":966,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1236,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1236,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":362,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":1760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":2118,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1018,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":2118,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1540,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1942,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":2162,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":38797312,"data_stored":38797312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":104988672,"data_stored":104988672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":89194496,"data_stored":89194496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":26738688,"data_stored":26738688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":157093888,"data_stored":157089816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":70975488,"data_stored":70975488,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T19:19:41.669 INFO:tasks.thrashosds.thrasher:Setting 8.e to [6, 5, 2, 7, 3, 1] 2024-09-10T19:19:41.670 INFO:tasks.thrashosds.thrasher:cmd ['osd', 'pg-upmap-items', '8.e', '6', '5', '2', '7', '3', '1'] 2024-09-10T19:19:41.670 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph osd pg-upmap-items 8.e 6 5 2 7 3 1 2024-09-10T19:19:41.696 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:19:41.724 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:19:41.727 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:19:41.737 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:19:41.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:41.738 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:19:41.838 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:19:41.862 INFO:teuthology.orchestra.run.smithi183.stderr:nodeep-scrub is set 2024-09-10T19:19:41.862 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:19:41.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:41.863 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:19:41.931 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:19:41 smithi183 ceph-mon[29736]: from='client.15303 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-10T19:19:41.932 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:19:41 smithi183 ceph-mon[29736]: from='client.? 172.21.15.183:0/2774054526' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-10T19:19:41.932 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:19:41 smithi183 ceph-mon[41562]: from='client.15303 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-10T19:19:41.932 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:19:41 smithi183 ceph-mon[41562]: from='client.? 172.21.15.183:0/2774054526' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-10T19:19:41.964 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:19:41.993 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:19:41.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:41.994 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:19:42.095 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:19:42.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:42.118 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:19:42.118 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:19:42.145 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:19:41 smithi191 ceph-mon[32884]: from='client.15303 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-10T19:19:42.145 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:19:41 smithi191 ceph-mon[32884]: from='client.? 172.21.15.183:0/2774054526' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-10T19:19:42.220 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:19:42.244 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:19:42.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:42.245 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:19:42.345 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:19:42.367 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:19:42.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:42.368 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:19:42.386 INFO:teuthology.orchestra.run.smithi191.stdout:{ 2024-09-10T19:19:42.386 INFO:teuthology.orchestra.run.smithi191.stdout: "size": 20, 2024-09-10T19:19:42.386 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 600, 2024-09-10T19:19:42.386 INFO:teuthology.orchestra.run.smithi191.stdout: "ops": [ 2024-09-10T19:19:42.386 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:19:42.386 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.14781.0:205540 7.b 7:d286f7dc:::benchmark_data_smithi183_84160_object65159:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e126)", 2024-09-10T19:19:42.386 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-10T19:14:45.772065+0000", 2024-09-10T19:19:42.386 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 296.61107862199998, 2024-09-10T19:19:42.386 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.98751803299999996, 2024-09-10T19:19:42.387 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-10T19:19:42.387 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T19:19:42.387 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-10T19:19:42.387 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.14781", 2024-09-10T19:19:42.387 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.183:0/2025776420", 2024-09-10T19:19:42.387 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 205540 2024-09-10T19:19:42.387 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:19:42.387 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-10T19:19:42.387 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:19:42.387 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-10T19:19:42.387 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:45.772065+0000", 2024-09-10T19:19:42.387 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:19:42.387 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:19:42.387 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:19:42.388 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-10T19:19:42.388 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:45.772065+0000", 2024-09-10T19:19:42.388 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:19:42.388 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:19:42.388 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:19:42.388 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-10T19:19:42.388 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:45.772064+0000", 2024-09-10T19:19:42.388 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.999999 2024-09-10T19:19:42.388 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:19:42.388 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:19:42.388 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-10T19:19:42.388 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:45.772071+0000", 2024-09-10T19:19:42.388 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.503e-06 2024-09-10T19:19:42.389 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:19:42.389 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:19:42.389 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-10T19:19:42.389 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:45.772082+0000", 2024-09-10T19:19:42.389 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.0294e-05 2024-09-10T19:19:42.389 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:19:42.389 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:19:42.389 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-10T19:19:42.389 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:45.772087+0000", 2024-09-10T19:19:42.389 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.643e-06 2024-09-10T19:19:42.389 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:19:42.389 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:19:42.389 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:19:42.389 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:45.772132+0000", 2024-09-10T19:19:42.390 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.4378000000000001e-05 2024-09-10T19:19:42.390 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:19:42.390 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:19:42.390 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for peered", 2024-09-10T19:19:42.390 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:45.772140+0000", 2024-09-10T19:19:42.390 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.7520000000000002e-06 2024-09-10T19:19:42.390 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:19:42.390 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:19:42.390 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:19:42.390 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:46.752096+0000", 2024-09-10T19:19:42.390 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.97995554699999998 2024-09-10T19:19:42.390 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:19:42.390 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:19:42.390 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-10T19:19:42.391 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:46.752214+0000", 2024-09-10T19:19:42.391 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000118507 2024-09-10T19:19:42.391 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:19:42.391 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:19:42.391 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 2,6", 2024-09-10T19:19:42.391 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:46.752288+0000", 2024-09-10T19:19:42.391 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.339e-05 2024-09-10T19:19:42.391 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:19:42.391 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:19:42.391 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-09-10T19:19:42.391 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:46.757094+0000", 2024-09-10T19:19:42.391 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0048063569999999998 2024-09-10T19:19:42.391 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:19:42.391 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:19:42.392 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:19:42.392 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:46.759427+0000", 2024-09-10T19:19:42.392 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0023326050000000002 2024-09-10T19:19:42.392 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:19:42.392 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:19:42.392 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:19:42.392 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:46.759503+0000", 2024-09-10T19:19:42.392 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.5694999999999999e-05 2024-09-10T19:19:42.392 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:19:42.392 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:19:42.392 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-09-10T19:19:42.392 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:46.759549+0000", 2024-09-10T19:19:42.392 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.6634000000000001e-05 2024-09-10T19:19:42.393 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:19:42.393 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:19:42.393 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-10T19:19:42.393 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:46.759583+0000", 2024-09-10T19:19:42.393 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.3878000000000003e-05 2024-09-10T19:19:42.393 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:19:42.393 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-10T19:19:42.393 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:19:42.393 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:19:42.393 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:19:42.393 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.14781.0:205549 7.b 7:d0626604:::benchmark_data_smithi183_84160_object65168:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e126)", 2024-09-10T19:19:42.393 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-10T19:14:45.780874+0000", 2024-09-10T19:19:42.393 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 296.60226931, 2024-09-10T19:19:42.393 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.97887205499999996, 2024-09-10T19:19:42.394 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-10T19:19:42.394 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T19:19:42.394 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-10T19:19:42.394 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.14781", 2024-09-10T19:19:42.394 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.183:0/2025776420", 2024-09-10T19:19:42.394 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 205549 2024-09-10T19:19:42.394 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:19:42.394 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-10T19:19:42.394 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:19:42.394 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-10T19:19:42.394 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:45.780874+0000", 2024-09-10T19:19:42.394 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:19:42.394 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:19:42.394 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:19:42.394 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-10T19:19:42.395 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:45.780874+0000", 2024-09-10T19:19:42.395 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:19:42.395 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:19:42.395 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:19:42.395 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-10T19:19:42.395 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:45.780873+0000", 2024-09-10T19:19:42.395 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999986 2024-09-10T19:19:42.395 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:19:42.395 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:19:42.395 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-10T19:19:42.395 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:45.780879+0000", 2024-09-10T19:19:42.395 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.083e-06 2024-09-10T19:19:42.395 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:19:42.395 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:19:42.396 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-10T19:19:42.396 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:45.780892+0000", 2024-09-10T19:19:42.396 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.2496e-05 2024-09-10T19:19:42.396 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:19:42.396 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:19:42.396 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-10T19:19:42.396 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:45.780896+0000", 2024-09-10T19:19:42.396 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.4970000000000003e-06 2024-09-10T19:19:42.396 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:19:42.396 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:19:42.396 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:19:42.396 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:45.780945+0000", 2024-09-10T19:19:42.396 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.8618000000000001e-05 2024-09-10T19:19:42.396 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:19:42.397 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:19:42.397 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for peered", 2024-09-10T19:19:42.397 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:45.780954+0000", 2024-09-10T19:19:42.397 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.9220000000000003e-06 2024-09-10T19:19:42.397 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:19:42.397 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:19:42.397 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:19:42.397 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:46.754818+0000", 2024-09-10T19:19:42.397 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.97386392899999996 2024-09-10T19:19:42.397 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:19:42.397 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:19:42.397 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-10T19:19:42.397 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:46.754913+0000", 2024-09-10T19:19:42.398 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.5884000000000006e-05 2024-09-10T19:19:42.398 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:19:42.398 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:19:42.398 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 2,6", 2024-09-10T19:19:42.398 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:46.755028+0000", 2024-09-10T19:19:42.398 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00011414 2024-09-10T19:19:42.398 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:19:42.398 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:19:42.398 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:19:42.398 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:46.759634+0000", 2024-09-10T19:19:42.398 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0046064060000000004 2024-09-10T19:19:42.398 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:19:42.398 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:19:42.398 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-09-10T19:19:42.399 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:46.759643+0000", 2024-09-10T19:19:42.399 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.2660000000000007e-06 2024-09-10T19:19:42.399 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:19:42.399 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:19:42.399 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:19:42.399 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:46.759693+0000", 2024-09-10T19:19:42.399 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.9382000000000003e-05 2024-09-10T19:19:42.399 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:19:42.399 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:19:42.399 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-09-10T19:19:42.399 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:46.759725+0000", 2024-09-10T19:19:42.399 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.2830999999999998e-05 2024-09-10T19:19:42.399 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:19:42.399 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:19:42.399 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-10T19:19:42.400 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:46.759746+0000", 2024-09-10T19:19:42.400 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.0885000000000001e-05 2024-09-10T19:19:42.401 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:19:42.401 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-10T19:19:42.401 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:19:42.401 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:19:42.401 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:19:42.401 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.14781.0:205550 7.b 7:d6a1ae8a:::benchmark_data_smithi183_84160_object65169:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e126)", 2024-09-10T19:19:42.401 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-10T19:14:45.780904+0000", 2024-09-10T19:19:42.401 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 296.60223981199999, 2024-09-10T19:19:42.401 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.98770159000000002, 2024-09-10T19:19:42.401 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-10T19:19:42.401 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T19:19:42.401 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-10T19:19:42.401 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.14781", 2024-09-10T19:19:42.402 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.183:0/2025776420", 2024-09-10T19:19:42.402 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 205550 2024-09-10T19:19:42.402 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:19:42.402 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-10T19:19:42.402 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:19:42.402 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-10T19:19:42.402 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:45.780904+0000", 2024-09-10T19:19:42.402 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:19:42.402 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:19:42.402 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:19:42.402 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-10T19:19:42.402 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:45.780904+0000", 2024-09-10T19:19:42.402 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:19:42.402 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:19:42.402 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:19:42.402 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-10T19:19:42.403 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:45.780903+0000", 2024-09-10T19:19:42.403 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999995 2024-09-10T19:19:42.403 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:19:42.403 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:19:42.403 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-10T19:19:42.403 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:45.780906+0000", 2024-09-10T19:19:42.403 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.599e-06 2024-09-10T19:19:42.403 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:19:42.403 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:19:42.403 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-10T19:19:42.403 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:45.780911+0000", 2024-09-10T19:19:42.403 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.434e-06 2024-09-10T19:19:42.403 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:19:42.403 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:19:42.404 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-10T19:19:42.404 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:45.780915+0000", 2024-09-10T19:19:42.404 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.8070000000000001e-06 2024-09-10T19:19:42.404 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:19:42.404 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:19:42.404 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:19:42.404 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:45.780977+0000", 2024-09-10T19:19:42.404 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.2244999999999998e-05 2024-09-10T19:19:42.404 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:19:42.404 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:19:42.404 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for peered", 2024-09-10T19:19:42.404 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:45.780982+0000", 2024-09-10T19:19:42.404 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.6609999999999999e-06 2024-09-10T19:19:42.405 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:19:42.405 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:19:42.405 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:19:42.405 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:46.757132+0000", 2024-09-10T19:19:42.405 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.97615030000000003 2024-09-10T19:19:42.405 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:19:42.405 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:19:42.405 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-10T19:19:42.405 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:46.757251+0000", 2024-09-10T19:19:42.405 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000118434 2024-09-10T19:19:42.405 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:19:42.405 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:19:42.405 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 2,6", 2024-09-10T19:19:42.405 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:46.757358+0000", 2024-09-10T19:19:42.405 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000107594 2024-09-10T19:19:42.406 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:19:42.406 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:19:42.406 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-09-10T19:19:42.406 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:46.762125+0000", 2024-09-10T19:19:42.406 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0047661559999999997 2024-09-10T19:19:42.406 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:19:42.406 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:19:42.406 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:19:42.406 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:46.762162+0000", 2024-09-10T19:19:42.406 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.6940999999999997e-05 2024-09-10T19:19:42.406 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:19:42.406 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:19:42.406 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:19:42.406 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:46.768527+0000", 2024-09-10T19:19:42.406 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0063658769999999998 2024-09-10T19:19:42.407 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:19:42.407 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:19:42.407 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-09-10T19:19:42.407 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:46.768571+0000", 2024-09-10T19:19:42.407 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.3745000000000002e-05 2024-09-10T19:19:42.407 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:19:42.407 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:19:42.407 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-10T19:19:42.407 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:46.768605+0000", 2024-09-10T19:19:42.407 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.4190000000000003e-05 2024-09-10T19:19:42.407 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:19:42.407 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-10T19:19:42.407 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:19:42.408 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:19:42.408 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:19:42.408 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.14781.0:205573 7.b 7:dc52318a:::benchmark_data_smithi183_84160_object65192:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e126)", 2024-09-10T19:19:42.408 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-10T19:14:45.813712+0000", 2024-09-10T19:19:42.408 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 296.56943206400001, 2024-09-10T19:19:42.408 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.96058740200000003, 2024-09-10T19:19:42.408 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-10T19:19:42.408 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T19:19:42.408 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-10T19:19:42.408 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.14781", 2024-09-10T19:19:42.408 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.183:0/2025776420", 2024-09-10T19:19:42.408 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 205573 2024-09-10T19:19:42.409 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:19:42.409 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-10T19:19:42.409 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:19:42.409 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-10T19:19:42.409 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:45.813712+0000", 2024-09-10T19:19:42.409 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:19:42.409 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:19:42.409 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:19:42.409 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-10T19:19:42.409 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:45.813712+0000", 2024-09-10T19:19:42.409 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:19:42.409 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:19:42.409 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:19:42.409 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-10T19:19:42.410 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:45.813710+0000", 2024-09-10T19:19:42.410 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999986 2024-09-10T19:19:42.410 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:19:42.410 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:19:42.410 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-10T19:19:42.410 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:45.813716+0000", 2024-09-10T19:19:42.410 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.2530000000000001e-06 2024-09-10T19:19:42.410 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:19:42.410 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:19:42.410 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-10T19:19:42.410 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:45.813728+0000", 2024-09-10T19:19:42.410 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.1973e-05 2024-09-10T19:19:42.410 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:19:42.410 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:19:42.411 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-10T19:19:42.411 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:45.813734+0000", 2024-09-10T19:19:42.411 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.0190000000000003e-06 2024-09-10T19:19:42.411 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:19:42.411 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:19:42.411 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:19:42.411 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:45.813784+0000", 2024-09-10T19:19:42.411 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.9521999999999998e-05 2024-09-10T19:19:42.411 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:19:42.411 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:19:42.411 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for peered", 2024-09-10T19:19:42.411 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:45.813793+0000", 2024-09-10T19:19:42.411 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.9649999999999997e-06 2024-09-10T19:19:42.412 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:19:42.412 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:19:42.412 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:19:42.412 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:46.759812+0000", 2024-09-10T19:19:42.412 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.94601953400000005 2024-09-10T19:19:42.412 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:19:42.412 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:19:42.412 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-10T19:19:42.412 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:46.759918+0000", 2024-09-10T19:19:42.412 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000106068 2024-09-10T19:19:42.412 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:19:42.412 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:19:42.412 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 2,6", 2024-09-10T19:19:42.413 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:46.760040+0000", 2024-09-10T19:19:42.413 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000121787 2024-09-10T19:19:42.413 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:19:42.413 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:19:42.413 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:19:42.413 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:46.768637+0000", 2024-09-10T19:19:42.413 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.008596645 2024-09-10T19:19:42.413 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:19:42.413 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:19:42.413 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:19:42.413 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:46.768662+0000", 2024-09-10T19:19:42.413 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.5069e-05 2024-09-10T19:19:42.413 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:19:42.413 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:19:42.414 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-09-10T19:19:42.414 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:46.774214+0000", 2024-09-10T19:19:42.414 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0055521820000000001 2024-09-10T19:19:42.414 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:19:42.414 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:19:42.414 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-09-10T19:19:42.414 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:46.774261+0000", 2024-09-10T19:19:42.414 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.6937000000000001e-05 2024-09-10T19:19:42.414 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:19:42.414 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:19:42.414 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-10T19:19:42.414 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:46.774299+0000", 2024-09-10T19:19:42.414 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.7954000000000003e-05 2024-09-10T19:19:42.414 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:19:42.415 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-10T19:19:42.415 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:19:42.415 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:19:42.415 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:19:42.415 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.14781.0:217417 7.0 7:0e3413bc:::benchmark_data_smithi183_84160_object77036:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e131)", 2024-09-10T19:19:42.415 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-10T19:14:56.880641+0000", 2024-09-10T19:19:42.415 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 285.50250285099997, 2024-09-10T19:19:42.415 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.991396897, 2024-09-10T19:19:42.415 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-10T19:19:42.416 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T19:19:42.416 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-10T19:19:42.416 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.14781", 2024-09-10T19:19:42.416 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.183:0/2025776420", 2024-09-10T19:19:42.416 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 217417 2024-09-10T19:19:42.416 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:19:42.416 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-10T19:19:42.416 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:19:42.416 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-10T19:19:42.417 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:56.880641+0000", 2024-09-10T19:19:42.417 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:19:42.417 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:19:42.417 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:19:42.417 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-10T19:19:42.417 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:56.880641+0000", 2024-09-10T19:19:42.417 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:19:42.417 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:19:42.417 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:19:42.417 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-10T19:19:42.417 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:56.880639+0000", 2024-09-10T19:19:42.417 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999981 2024-09-10T19:19:42.417 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:19:42.418 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:19:42.418 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-10T19:19:42.418 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:56.880647+0000", 2024-09-10T19:19:42.418 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.9810000000000003e-06 2024-09-10T19:19:42.418 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:19:42.418 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:19:42.418 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-10T19:19:42.418 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:56.880664+0000", 2024-09-10T19:19:42.418 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.7589999999999999e-05 2024-09-10T19:19:42.418 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:19:42.418 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:19:42.418 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-10T19:19:42.418 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:56.880671+0000", 2024-09-10T19:19:42.418 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.9779999999999999e-06 2024-09-10T19:19:42.419 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:19:42.419 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:19:42.419 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:19:42.419 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:56.880736+0000", 2024-09-10T19:19:42.419 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.4858999999999997e-05 2024-09-10T19:19:42.419 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:19:42.419 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:19:42.419 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for peered", 2024-09-10T19:19:42.419 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:56.880745+0000", 2024-09-10T19:19:42.419 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.8340000000000005e-06 2024-09-10T19:19:42.419 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:19:42.419 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:19:42.420 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:19:42.420 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:57.866422+0000", 2024-09-10T19:19:42.420 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.98567665000000004 2024-09-10T19:19:42.420 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:19:42.420 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:19:42.420 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-10T19:19:42.420 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:57.866508+0000", 2024-09-10T19:19:42.420 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.5717000000000006e-05 2024-09-10T19:19:42.420 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:19:42.420 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:19:42.420 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 2,3", 2024-09-10T19:19:42.420 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:57.866585+0000", 2024-09-10T19:19:42.420 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.7676999999999995e-05 2024-09-10T19:19:42.421 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:19:42.421 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:19:42.421 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-09-10T19:19:42.421 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:57.871908+0000", 2024-09-10T19:19:42.421 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0053224120000000003 2024-09-10T19:19:42.421 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:19:42.421 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:19:42.421 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:19:42.421 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:57.871930+0000", 2024-09-10T19:19:42.421 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.2722e-05 2024-09-10T19:19:42.421 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:19:42.421 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:19:42.422 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:19:42.422 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:57.871985+0000", 2024-09-10T19:19:42.422 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.4246000000000002e-05 2024-09-10T19:19:42.422 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:19:42.422 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:19:42.422 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-09-10T19:19:42.422 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:57.872017+0000", 2024-09-10T19:19:42.422 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.2441000000000003e-05 2024-09-10T19:19:42.422 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:19:42.422 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:19:42.422 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-10T19:19:42.422 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:57.872038+0000", 2024-09-10T19:19:42.422 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.0639999999999999e-05 2024-09-10T19:19:42.422 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:19:42.423 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-10T19:19:42.423 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:19:42.423 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:19:42.423 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:19:42.423 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.14781.0:217427 7.0 7:0e3d17be:::benchmark_data_smithi183_84160_object77046:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e131)", 2024-09-10T19:19:42.423 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-10T19:14:56.890012+0000", 2024-09-10T19:19:42.423 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 285.49313203499997, 2024-09-10T19:19:42.423 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.98213677700000002, 2024-09-10T19:19:42.423 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-10T19:19:42.423 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T19:19:42.423 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-10T19:19:42.423 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.14781", 2024-09-10T19:19:42.423 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.183:0/2025776420", 2024-09-10T19:19:42.424 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 217427 2024-09-10T19:19:42.424 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:19:42.424 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-10T19:19:42.424 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:19:42.424 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-10T19:19:42.424 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:56.890012+0000", 2024-09-10T19:19:42.424 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:19:42.424 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:19:42.424 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:19:42.424 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-10T19:19:42.424 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:56.890012+0000", 2024-09-10T19:19:42.424 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:19:42.424 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:19:42.425 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:19:42.425 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-10T19:19:42.425 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:56.890010+0000", 2024-09-10T19:19:42.425 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999986 2024-09-10T19:19:42.425 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:19:42.425 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:19:42.425 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-10T19:19:42.425 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:56.890017+0000", 2024-09-10T19:19:42.425 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.5259999999999999e-06 2024-09-10T19:19:42.425 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:19:42.425 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:19:42.425 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-10T19:19:42.425 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:56.890028+0000", 2024-09-10T19:19:42.426 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.1259e-05 2024-09-10T19:19:42.426 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:19:42.426 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:19:42.426 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-10T19:19:42.426 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:56.890032+0000", 2024-09-10T19:19:42.426 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.3769999999999998e-06 2024-09-10T19:19:42.426 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:19:42.426 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:19:42.426 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:19:42.426 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:56.890066+0000", 2024-09-10T19:19:42.426 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.3467999999999999e-05 2024-09-10T19:19:42.426 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:19:42.426 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:19:42.426 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for peered", 2024-09-10T19:19:42.427 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:56.890072+0000", 2024-09-10T19:19:42.427 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.5400000000000001e-06 2024-09-10T19:19:42.427 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:19:42.427 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:19:42.427 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:19:42.427 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:57.868641+0000", 2024-09-10T19:19:42.427 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.97856873 2024-09-10T19:19:42.427 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:19:42.427 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:19:42.427 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-10T19:19:42.427 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:57.868700+0000", 2024-09-10T19:19:42.427 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.9086999999999999e-05 2024-09-10T19:19:42.428 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:19:42.428 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:19:42.428 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 2,3", 2024-09-10T19:19:42.428 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:57.868767+0000", 2024-09-10T19:19:42.428 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.6742000000000006e-05 2024-09-10T19:19:42.428 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:19:42.428 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:19:42.428 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:19:42.428 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:57.872070+0000", 2024-09-10T19:19:42.428 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0033030450000000001 2024-09-10T19:19:42.428 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:19:42.428 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:19:42.428 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-09-10T19:19:42.429 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:57.872076+0000", 2024-09-10T19:19:42.429 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.1510000000000001e-06 2024-09-10T19:19:42.429 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:19:42.429 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:19:42.429 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:19:42.429 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:57.872111+0000", 2024-09-10T19:19:42.429 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.4857999999999997e-05 2024-09-10T19:19:42.429 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:19:42.429 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:19:42.429 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-09-10T19:19:42.429 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:57.872132+0000", 2024-09-10T19:19:42.429 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.0798999999999999e-05 2024-09-10T19:19:42.429 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:19:42.430 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:19:42.430 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-10T19:19:42.430 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:57.872148+0000", 2024-09-10T19:19:42.430 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.6449000000000001e-05 2024-09-10T19:19:42.430 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:19:42.430 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-10T19:19:42.430 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:19:42.430 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:19:42.430 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:19:42.430 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.14781.0:217459 7.0 7:09cbbd5e:::benchmark_data_smithi183_84160_object77078:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e131)", 2024-09-10T19:19:42.430 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-10T19:14:56.915054+0000", 2024-09-10T19:19:42.430 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 285.46808925900001, 2024-09-10T19:19:42.430 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.96084031700000005, 2024-09-10T19:19:42.431 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-10T19:19:42.431 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T19:19:42.431 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-10T19:19:42.431 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.14781", 2024-09-10T19:19:42.431 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.183:0/2025776420", 2024-09-10T19:19:42.431 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 217459 2024-09-10T19:19:42.431 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:19:42.431 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-10T19:19:42.431 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:19:42.431 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-10T19:19:42.431 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:56.915054+0000", 2024-09-10T19:19:42.431 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:19:42.432 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:19:42.432 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:19:42.432 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-10T19:19:42.432 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:56.915054+0000", 2024-09-10T19:19:42.432 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:19:42.432 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:19:42.432 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:19:42.432 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-10T19:19:42.432 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:56.915053+0000", 2024-09-10T19:19:42.432 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999986 2024-09-10T19:19:42.432 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:19:42.432 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:19:42.432 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-10T19:19:42.433 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:56.915060+0000", 2024-09-10T19:19:42.433 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.8109999999999997e-06 2024-09-10T19:19:42.433 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:19:42.433 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:19:42.433 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-10T19:19:42.433 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:56.915072+0000", 2024-09-10T19:19:42.433 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.1898e-05 2024-09-10T19:19:42.433 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:19:42.433 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:19:42.433 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-10T19:19:42.433 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:56.915077+0000", 2024-09-10T19:19:42.433 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.344e-06 2024-09-10T19:19:42.433 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:19:42.433 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:19:42.434 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:19:42.434 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:56.915118+0000", 2024-09-10T19:19:42.434 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.0725000000000001e-05 2024-09-10T19:19:42.434 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:19:42.434 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:19:42.434 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for peered", 2024-09-10T19:19:42.434 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:56.915123+0000", 2024-09-10T19:19:42.434 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.6559999999999997e-06 2024-09-10T19:19:42.434 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:19:42.434 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:19:42.434 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:19:42.434 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:57.869813+0000", 2024-09-10T19:19:42.434 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.95469002700000005 2024-09-10T19:19:42.435 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:19:42.435 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:19:42.435 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-10T19:19:42.435 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:57.869876+0000", 2024-09-10T19:19:42.435 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.2135000000000001e-05 2024-09-10T19:19:42.435 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:19:42.435 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:19:42.435 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 2,3", 2024-09-10T19:19:42.435 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:57.869937+0000", 2024-09-10T19:19:42.435 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.1420999999999994e-05 2024-09-10T19:19:42.435 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:19:42.435 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:19:42.435 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-09-10T19:19:42.435 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:57.872083+0000", 2024-09-10T19:19:42.435 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.002145545 2024-09-10T19:19:42.435 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:19:42.436 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:19:42.436 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:19:42.436 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:57.875737+0000", 2024-09-10T19:19:42.436 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0036545900000000001 2024-09-10T19:19:42.436 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:19:42.436 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:19:42.436 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:19:42.436 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:57.875798+0000", 2024-09-10T19:19:42.436 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.0498999999999998e-05 2024-09-10T19:19:42.436 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:19:42.436 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:19:42.436 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-09-10T19:19:42.436 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:57.875853+0000", 2024-09-10T19:19:42.436 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.5429000000000001e-05 2024-09-10T19:19:42.437 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:19:42.437 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:19:42.437 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-10T19:19:42.437 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:57.875895+0000", 2024-09-10T19:19:42.437 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.1582999999999998e-05 2024-09-10T19:19:42.437 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:19:42.437 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-10T19:19:42.437 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:19:42.437 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:19:42.437 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:19:42.437 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.15051.0:93085 8.14 8:2abf7048:::benchmark_data_smithi183_114924_object93084:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e177)", 2024-09-10T19:19:42.437 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-10T19:17:26.014960+0000", 2024-09-10T19:19:42.437 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 136.368183754, 2024-09-10T19:19:42.438 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.99687631899999996, 2024-09-10T19:19:42.438 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-10T19:19:42.438 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "reached pg", 2024-09-10T19:19:42.438 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-10T19:19:42.438 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.15051", 2024-09-10T19:19:42.438 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.183:0/3632028672", 2024-09-10T19:19:42.438 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 93085 2024-09-10T19:19:42.438 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:19:42.438 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-10T19:19:42.438 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:19:42.438 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-10T19:19:42.438 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:26.014960+0000", 2024-09-10T19:19:42.438 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:19:42.438 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:19:42.438 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:19:42.439 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-10T19:19:42.439 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:26.014960+0000", 2024-09-10T19:19:42.439 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:19:42.439 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:19:42.439 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:19:42.439 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-10T19:19:42.439 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:26.014957+0000", 2024-09-10T19:19:42.439 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999976 2024-09-10T19:19:42.439 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:19:42.439 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:19:42.439 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-10T19:19:42.439 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:26.015041+0000", 2024-09-10T19:19:42.439 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.3499999999999997e-05 2024-09-10T19:19:42.439 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:19:42.439 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:19:42.440 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-10T19:19:42.440 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:26.015055+0000", 2024-09-10T19:19:42.440 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.4613999999999999e-05 2024-09-10T19:19:42.440 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:19:42.440 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:19:42.440 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-10T19:19:42.440 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:26.015064+0000", 2024-09-10T19:19:42.440 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.6070000000000008e-06 2024-09-10T19:19:42.440 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:19:42.440 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:19:42.440 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:19:42.440 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:26.163910+0000", 2024-09-10T19:19:42.440 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.14884554799999999 2024-09-10T19:19:42.440 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:19:42.441 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:19:42.441 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for peered", 2024-09-10T19:19:42.441 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:26.163919+0000", 2024-09-10T19:19:42.441 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.1679999999999993e-06 2024-09-10T19:19:42.441 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:19:42.441 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:19:42.441 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:19:42.441 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:27.006838+0000", 2024-09-10T19:19:42.441 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.84291875999999999 2024-09-10T19:19:42.441 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:19:42.441 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:19:42.441 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for degraded object", 2024-09-10T19:19:42.441 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:27.007794+0000", 2024-09-10T19:19:42.442 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00095682999999999999 2024-09-10T19:19:42.442 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:19:42.442 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:19:42.442 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:19:42.442 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:27.011768+0000", 2024-09-10T19:19:42.442 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0039740390000000004 2024-09-10T19:19:42.442 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:19:42.442 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:19:42.442 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-10T19:19:42.442 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:27.011836+0000", 2024-09-10T19:19:42.442 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.7748000000000003e-05 2024-09-10T19:19:42.442 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:19:42.442 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-10T19:19:42.443 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:19:42.443 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:19:42.443 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:19:42.443 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.15051.0:93088 8.4 8:24544414:::benchmark_data_smithi183_114924_object93087:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e177)", 2024-09-10T19:19:42.443 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-10T19:17:26.015072+0000", 2024-09-10T19:19:42.443 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 136.36807209099999, 2024-09-10T19:19:42.443 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.97135848400000002, 2024-09-10T19:19:42.443 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-10T19:19:42.443 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "reached pg", 2024-09-10T19:19:42.443 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-10T19:19:42.443 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.15051", 2024-09-10T19:19:42.443 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.183:0/3632028672", 2024-09-10T19:19:42.443 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 93088 2024-09-10T19:19:42.443 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:19:42.444 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-10T19:19:42.444 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:19:42.444 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-10T19:19:42.444 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:26.015072+0000", 2024-09-10T19:19:42.444 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:19:42.444 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:19:42.444 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:19:42.444 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-10T19:19:42.444 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:26.015072+0000", 2024-09-10T19:19:42.444 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:19:42.444 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:19:42.444 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:19:42.444 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-10T19:19:42.444 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:26.015071+0000", 2024-09-10T19:19:42.444 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999995 2024-09-10T19:19:42.445 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:19:42.445 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:19:42.445 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-10T19:19:42.445 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:26.015104+0000", 2024-09-10T19:19:42.445 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.3473999999999998e-05 2024-09-10T19:19:42.445 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:19:42.445 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:19:42.445 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-10T19:19:42.445 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:26.015112+0000", 2024-09-10T19:19:42.445 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.8259999999999993e-06 2024-09-10T19:19:42.445 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:19:42.445 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:19:42.445 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-10T19:19:42.445 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:26.015117+0000", 2024-09-10T19:19:42.445 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.4630000000000003e-06 2024-09-10T19:19:42.446 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:19:42.446 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:19:42.446 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:19:42.446 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:26.163978+0000", 2024-09-10T19:19:42.446 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.14886155000000001 2024-09-10T19:19:42.446 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:19:42.446 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:19:42.446 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for peered", 2024-09-10T19:19:42.446 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:26.163985+0000", 2024-09-10T19:19:42.446 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.4559999999999997e-06 2024-09-10T19:19:42.446 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:19:42.446 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:19:42.446 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:19:42.446 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:26.976983+0000", 2024-09-10T19:19:42.447 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.812998357 2024-09-10T19:19:42.447 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:19:42.447 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:19:42.447 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for degraded object", 2024-09-10T19:19:42.447 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:26.978158+0000", 2024-09-10T19:19:42.447 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0011745919999999999 2024-09-10T19:19:42.447 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:19:42.447 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:19:42.447 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:19:42.447 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:26.986359+0000", 2024-09-10T19:19:42.447 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0082010720000000002 2024-09-10T19:19:42.447 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:19:42.447 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:19:42.447 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-10T19:19:42.448 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:26.986430+0000", 2024-09-10T19:19:42.448 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.1358000000000004e-05 2024-09-10T19:19:42.448 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:19:42.448 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-10T19:19:42.448 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:19:42.448 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:19:42.448 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:19:42.448 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.15051.0:93092 8.8 8:1185ec04:::benchmark_data_smithi183_114924_object93091:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e177)", 2024-09-10T19:19:42.448 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-10T19:17:26.031384+0000", 2024-09-10T19:19:42.448 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 136.351759708, 2024-09-10T19:19:42.448 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.95024060600000004, 2024-09-10T19:19:42.448 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-10T19:19:42.448 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T19:19:42.449 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-10T19:19:42.449 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.15051", 2024-09-10T19:19:42.449 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.183:0/3632028672", 2024-09-10T19:19:42.449 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 93092 2024-09-10T19:19:42.449 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:19:42.449 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-10T19:19:42.449 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:19:42.449 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-10T19:19:42.449 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:26.031384+0000", 2024-09-10T19:19:42.449 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:19:42.449 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:19:42.449 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:19:42.449 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-10T19:19:42.449 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:26.031384+0000", 2024-09-10T19:19:42.450 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:19:42.450 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:19:42.450 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:19:42.450 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-10T19:19:42.450 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:26.031381+0000", 2024-09-10T19:19:42.450 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999971 2024-09-10T19:19:42.450 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:19:42.450 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:19:42.450 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-10T19:19:42.450 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:26.031425+0000", 2024-09-10T19:19:42.450 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.4456000000000003e-05 2024-09-10T19:19:42.450 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:19:42.450 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:19:42.451 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-10T19:19:42.451 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:26.031452+0000", 2024-09-10T19:19:42.451 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.6366000000000001e-05 2024-09-10T19:19:42.451 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:19:42.451 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:19:42.451 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-10T19:19:42.451 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:26.031462+0000", 2024-09-10T19:19:42.451 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.0029e-05 2024-09-10T19:19:42.451 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:19:42.451 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:19:42.451 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:19:42.451 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:26.175611+0000", 2024-09-10T19:19:42.451 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.14414951500000001 2024-09-10T19:19:42.452 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:19:42.452 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:19:42.452 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for peered", 2024-09-10T19:19:42.452 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:26.175622+0000", 2024-09-10T19:19:42.452 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.0192e-05 2024-09-10T19:19:42.452 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:19:42.452 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:19:42.452 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:19:42.452 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:26.958588+0000", 2024-09-10T19:19:42.452 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.78296621 2024-09-10T19:19:42.452 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:19:42.452 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:19:42.453 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-10T19:19:42.453 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:26.958763+0000", 2024-09-10T19:19:42.453 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00017493900000000001 2024-09-10T19:19:42.453 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:19:42.453 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:19:42.453 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 3,7", 2024-09-10T19:19:42.453 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:26.959106+0000", 2024-09-10T19:19:42.453 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00034345799999999999 2024-09-10T19:19:42.453 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:19:42.453 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:19:42.453 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-09-10T19:19:42.453 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:26.969476+0000", 2024-09-10T19:19:42.453 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.010370226 2024-09-10T19:19:42.453 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:19:42.454 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:19:42.454 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:19:42.454 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:26.976919+0000", 2024-09-10T19:19:42.454 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0074428430000000002 2024-09-10T19:19:42.454 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:19:42.454 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:19:42.454 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:19:42.454 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:26.981548+0000", 2024-09-10T19:19:42.454 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0046287050000000003 2024-09-10T19:19:42.454 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:19:42.454 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:19:42.454 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-09-10T19:19:42.454 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:26.981590+0000", 2024-09-10T19:19:42.454 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.2546000000000002e-05 2024-09-10T19:19:42.454 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:19:42.455 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:19:42.455 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-10T19:19:42.455 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:26.981625+0000", 2024-09-10T19:19:42.455 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.4071e-05 2024-09-10T19:19:42.455 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:19:42.455 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-10T19:19:42.455 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:19:42.455 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:19:42.455 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:19:42.455 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.15051.0:93184 8.13 8.bc5cfff3 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e178)", 2024-09-10T19:19:42.455 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-10T19:17:27.037577+0000", 2024-09-10T19:19:42.455 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 135.345566937, 2024-09-10T19:19:42.455 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.9100273649999999, 2024-09-10T19:19:42.456 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-10T19:19:42.456 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "reached pg", 2024-09-10T19:19:42.456 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-10T19:19:42.456 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.15051", 2024-09-10T19:19:42.456 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.183:0/3632028672", 2024-09-10T19:19:42.456 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 93184 2024-09-10T19:19:42.456 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:19:42.456 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-10T19:19:42.456 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:19:42.456 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-10T19:19:42.456 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:27.037577+0000", 2024-09-10T19:19:42.456 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:19:42.456 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:19:42.457 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:19:42.457 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-10T19:19:42.457 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:27.037577+0000", 2024-09-10T19:19:42.457 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:19:42.457 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:19:42.457 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:19:42.457 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-10T19:19:42.458 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:27.037576+0000", 2024-09-10T19:19:42.458 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.999999 2024-09-10T19:19:42.458 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:19:42.458 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:19:42.458 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-10T19:19:42.459 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:27.037604+0000", 2024-09-10T19:19:42.459 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.8466000000000002e-05 2024-09-10T19:19:42.459 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:19:42.459 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:19:42.459 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-10T19:19:42.459 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:27.037619+0000", 2024-09-10T19:19:42.459 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.5418e-05 2024-09-10T19:19:42.459 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:19:42.459 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:19:42.459 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-10T19:19:42.459 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:27.037626+0000", 2024-09-10T19:19:42.459 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.426e-06 2024-09-10T19:19:42.460 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:19:42.460 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:19:42.460 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:19:42.460 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:27.037668+0000", 2024-09-10T19:19:42.460 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.1878999999999997e-05 2024-09-10T19:19:42.460 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:19:42.460 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:19:42.460 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for peered", 2024-09-10T19:19:42.460 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:27.037676+0000", 2024-09-10T19:19:42.460 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.8909999999999995e-06 2024-09-10T19:19:42.460 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:19:42.461 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:19:42.461 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:19:42.461 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:27.945157+0000", 2024-09-10T19:19:42.461 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.90748107099999997 2024-09-10T19:19:42.461 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:19:42.461 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:19:42.461 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for peered", 2024-09-10T19:19:42.461 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:27.945165+0000", 2024-09-10T19:19:42.461 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.2579999999999995e-06 2024-09-10T19:19:42.461 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:19:42.461 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:19:42.461 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:19:42.461 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:28.947583+0000", 2024-09-10T19:19:42.462 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.0024182589999999 2024-09-10T19:19:42.462 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:19:42.462 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:19:42.462 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-10T19:19:42.462 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:28.947604+0000", 2024-09-10T19:19:42.462 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.0771000000000002e-05 2024-09-10T19:19:42.462 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:19:42.462 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-10T19:19:42.462 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:19:42.462 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:19:42.462 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:19:42.462 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.15051.0:93229 8.13 8.8d3dde53 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e178)", 2024-09-10T19:19:42.462 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-10T19:17:27.063512+0000", 2024-09-10T19:19:42.462 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 135.31963148400001, 2024-09-10T19:19:42.463 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.8841211410000001, 2024-09-10T19:19:42.463 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-10T19:19:42.463 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "reached pg", 2024-09-10T19:19:42.463 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-10T19:19:42.463 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.15051", 2024-09-10T19:19:42.463 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.183:0/3632028672", 2024-09-10T19:19:42.463 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 93229 2024-09-10T19:19:42.463 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:19:42.463 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-10T19:19:42.463 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:19:42.463 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-10T19:19:42.463 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:27.063512+0000", 2024-09-10T19:19:42.463 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:19:42.463 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:19:42.464 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:19:42.464 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-10T19:19:42.464 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:27.063512+0000", 2024-09-10T19:19:42.464 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:19:42.464 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:19:42.464 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:19:42.464 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-10T19:19:42.464 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:27.063510+0000", 2024-09-10T19:19:42.464 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999981 2024-09-10T19:19:42.464 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:19:42.464 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:19:42.464 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-10T19:19:42.464 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:27.063560+0000", 2024-09-10T19:19:42.464 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.0076e-05 2024-09-10T19:19:42.465 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:19:42.465 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:19:42.465 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-10T19:19:42.465 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:27.063579+0000", 2024-09-10T19:19:42.465 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.8763e-05 2024-09-10T19:19:42.465 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:19:42.465 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:19:42.465 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-10T19:19:42.465 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:27.063587+0000", 2024-09-10T19:19:42.465 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.5020000000000003e-06 2024-09-10T19:19:42.465 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:19:42.465 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:19:42.465 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:19:42.466 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:27.063663+0000", 2024-09-10T19:19:42.466 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.6592000000000001e-05 2024-09-10T19:19:42.466 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:19:42.466 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:19:42.466 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for peered", 2024-09-10T19:19:42.466 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:27.063673+0000", 2024-09-10T19:19:42.466 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.4620000000000002e-06 2024-09-10T19:19:42.466 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:19:42.466 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:19:42.466 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:19:42.466 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:27.945182+0000", 2024-09-10T19:19:42.466 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.88150978000000002 2024-09-10T19:19:42.466 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:19:42.467 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:19:42.467 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for peered", 2024-09-10T19:19:42.467 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:27.945187+0000", 2024-09-10T19:19:42.467 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.7330000000000003e-06 2024-09-10T19:19:42.467 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:19:42.467 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:19:42.467 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:19:42.467 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:28.947626+0000", 2024-09-10T19:19:42.467 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.0024391020000001 2024-09-10T19:19:42.467 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:19:42.467 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:19:42.467 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-10T19:19:42.467 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:28.947633+0000", 2024-09-10T19:19:42.468 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.0890000000000002e-06 2024-09-10T19:19:42.468 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:19:42.468 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-10T19:19:42.468 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:19:42.468 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:19:42.468 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:19:42.468 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.15051.0:93231 8.13 8.7225c053 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e178)", 2024-09-10T19:19:42.468 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-10T19:17:27.065206+0000", 2024-09-10T19:19:42.468 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 135.31793737300001, 2024-09-10T19:19:42.468 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.88245339, 2024-09-10T19:19:42.468 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-10T19:19:42.468 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "reached pg", 2024-09-10T19:19:42.468 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-10T19:19:42.468 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.15051", 2024-09-10T19:19:42.468 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.183:0/3632028672", 2024-09-10T19:19:42.469 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 93231 2024-09-10T19:19:42.469 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:19:42.469 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-10T19:19:42.469 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:19:42.469 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-10T19:19:42.469 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:27.065206+0000", 2024-09-10T19:19:42.469 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:19:42.469 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:19:42.469 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:19:42.469 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-10T19:19:42.469 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:27.065206+0000", 2024-09-10T19:19:42.469 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:19:42.470 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:19:42.470 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:19:42.470 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-10T19:19:42.470 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:27.065205+0000", 2024-09-10T19:19:42.470 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999986 2024-09-10T19:19:42.470 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:19:42.470 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:19:42.470 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-10T19:19:42.470 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:27.065233+0000", 2024-09-10T19:19:42.470 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.8198999999999999e-05 2024-09-10T19:19:42.470 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:19:42.470 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:19:42.470 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-10T19:19:42.471 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:27.065508+0000", 2024-09-10T19:19:42.471 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00027513599999999999 2024-09-10T19:19:42.471 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:19:42.471 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:19:42.471 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-10T19:19:42.471 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:27.065515+0000", 2024-09-10T19:19:42.471 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.5950000000000004e-06 2024-09-10T19:19:42.471 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:19:42.471 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:19:42.471 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:19:42.471 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:27.065569+0000", 2024-09-10T19:19:42.471 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.4144000000000002e-05 2024-09-10T19:19:42.472 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:19:42.472 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:19:42.472 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for peered", 2024-09-10T19:19:42.472 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:27.065576+0000", 2024-09-10T19:19:42.472 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.3050000000000004e-06 2024-09-10T19:19:42.472 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:19:42.472 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:19:42.472 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:19:42.472 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:27.945203+0000", 2024-09-10T19:19:42.472 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.879626876 2024-09-10T19:19:42.472 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:19:42.472 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:19:42.473 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for peered", 2024-09-10T19:19:42.473 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:27.945209+0000", 2024-09-10T19:19:42.473 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.4400000000000002e-06 2024-09-10T19:19:42.473 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:19:42.473 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:19:42.473 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:19:42.473 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:28.947653+0000", 2024-09-10T19:19:42.473 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.0024433909999999 2024-09-10T19:19:42.473 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:19:42.473 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:19:42.473 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-10T19:19:42.473 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:28.947660+0000", 2024-09-10T19:19:42.474 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.7769999999999997e-06 2024-09-10T19:19:42.474 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:19:42.474 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-10T19:19:42.474 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:19:42.474 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:19:42.474 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:19:42.474 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.15051.0:93252 8.13 8.4feb5cf3 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e178)", 2024-09-10T19:19:42.474 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-10T19:17:27.107462+0000", 2024-09-10T19:19:42.474 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 135.275681961, 2024-09-10T19:19:42.474 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.840251692, 2024-09-10T19:19:42.474 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-10T19:19:42.474 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "reached pg", 2024-09-10T19:19:42.474 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-10T19:19:42.474 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.15051", 2024-09-10T19:19:42.475 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.183:0/3632028672", 2024-09-10T19:19:42.475 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 93252 2024-09-10T19:19:42.475 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:19:42.475 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-10T19:19:42.475 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:19:42.475 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-10T19:19:42.475 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:27.107462+0000", 2024-09-10T19:19:42.475 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:19:42.475 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:19:42.475 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:19:42.475 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-10T19:19:42.475 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:27.107462+0000", 2024-09-10T19:19:42.476 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:19:42.476 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:19:42.476 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:19:42.476 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-10T19:19:42.476 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:27.107461+0000", 2024-09-10T19:19:42.476 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999995 2024-09-10T19:19:42.476 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:19:42.476 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:19:42.476 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-10T19:19:42.476 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:27.107578+0000", 2024-09-10T19:19:42.476 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000116752 2024-09-10T19:19:42.476 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:19:42.476 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:19:42.477 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-10T19:19:42.477 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:27.107588+0000", 2024-09-10T19:19:42.477 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.0383e-05 2024-09-10T19:19:42.477 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:19:42.477 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:19:42.477 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-10T19:19:42.477 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:27.107593+0000", 2024-09-10T19:19:42.477 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.3600000000000004e-06 2024-09-10T19:19:42.477 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:19:42.477 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:19:42.477 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:19:42.477 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:27.107667+0000", 2024-09-10T19:19:42.478 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.3631e-05 2024-09-10T19:19:42.478 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:19:42.478 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:19:42.478 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for peered", 2024-09-10T19:19:42.478 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:27.107674+0000", 2024-09-10T19:19:42.478 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.1740000000000003e-06 2024-09-10T19:19:42.478 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:19:42.478 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:19:42.478 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:19:42.478 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:27.945226+0000", 2024-09-10T19:19:42.478 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.83755159300000004 2024-09-10T19:19:42.478 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:19:42.478 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:19:42.478 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for peered", 2024-09-10T19:19:42.479 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:27.945231+0000", 2024-09-10T19:19:42.479 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.5419999999999996e-06 2024-09-10T19:19:42.479 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:19:42.479 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:19:42.479 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:19:42.479 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:28.947706+0000", 2024-09-10T19:19:42.479 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.0024742929999999 2024-09-10T19:19:42.479 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:19:42.479 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:19:42.479 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-10T19:19:42.479 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:28.947713+0000", 2024-09-10T19:19:42.479 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.6410000000000002e-06 2024-09-10T19:19:42.479 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:19:42.479 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-10T19:19:42.480 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:19:42.480 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:19:42.480 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:19:42.480 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.15051.0:93300 8.13 8.7cc11153 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e178)", 2024-09-10T19:19:42.480 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-10T19:17:27.142933+0000", 2024-09-10T19:19:42.480 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 135.240211088, 2024-09-10T19:19:42.480 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.8048062439999999, 2024-09-10T19:19:42.480 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-10T19:19:42.480 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "reached pg", 2024-09-10T19:19:42.480 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-10T19:19:42.480 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.15051", 2024-09-10T19:19:42.480 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.183:0/3632028672", 2024-09-10T19:19:42.480 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 93300 2024-09-10T19:19:42.480 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:19:42.481 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-10T19:19:42.481 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:19:42.481 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-10T19:19:42.481 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:27.142933+0000", 2024-09-10T19:19:42.481 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:19:42.481 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:19:42.481 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:19:42.481 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-10T19:19:42.481 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:27.142933+0000", 2024-09-10T19:19:42.481 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:19:42.481 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:19:42.481 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:19:42.481 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-10T19:19:42.481 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:27.142932+0000", 2024-09-10T19:19:42.482 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999995 2024-09-10T19:19:42.482 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:19:42.482 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:19:42.482 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-10T19:19:42.482 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:27.142964+0000", 2024-09-10T19:19:42.482 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.1646999999999997e-05 2024-09-10T19:19:42.482 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:19:42.482 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:19:42.482 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-10T19:19:42.482 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:27.142970+0000", 2024-09-10T19:19:42.482 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.2090000000000002e-06 2024-09-10T19:19:42.482 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:19:42.482 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:19:42.482 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-10T19:19:42.483 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:27.142973+0000", 2024-09-10T19:19:42.483 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.202e-06 2024-09-10T19:19:42.483 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:19:42.483 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:19:42.483 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:19:42.483 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:27.143014+0000", 2024-09-10T19:19:42.483 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.1029000000000003e-05 2024-09-10T19:19:42.483 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:19:42.483 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:19:42.483 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for peered", 2024-09-10T19:19:42.483 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:27.143020+0000", 2024-09-10T19:19:42.483 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.7969999999999997e-06 2024-09-10T19:19:42.483 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:19:42.483 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:19:42.483 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:19:42.483 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:27.945248+0000", 2024-09-10T19:19:42.484 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.80222796500000004 2024-09-10T19:19:42.484 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:19:42.484 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:19:42.484 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for peered", 2024-09-10T19:19:42.484 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:27.945253+0000", 2024-09-10T19:19:42.484 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.198e-06 2024-09-10T19:19:42.484 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:19:42.484 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:19:42.484 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:19:42.484 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:28.947731+0000", 2024-09-10T19:19:42.484 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.002478379 2024-09-10T19:19:42.484 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:19:42.484 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:19:42.484 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-10T19:19:42.485 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:28.947739+0000", 2024-09-10T19:19:42.485 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.464e-06 2024-09-10T19:19:42.485 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:19:42.485 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-10T19:19:42.485 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:19:42.485 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:19:42.485 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:19:42.485 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.15051.0:93324 8.13 8.bd83f013 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e178)", 2024-09-10T19:19:42.485 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-10T19:17:27.174031+0000", 2024-09-10T19:19:42.485 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 135.209112764, 2024-09-10T19:19:42.485 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.7737433629999999, 2024-09-10T19:19:42.485 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-10T19:19:42.485 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "reached pg", 2024-09-10T19:19:42.485 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-10T19:19:42.486 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.15051", 2024-09-10T19:19:42.486 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.183:0/3632028672", 2024-09-10T19:19:42.486 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 93324 2024-09-10T19:19:42.486 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:19:42.486 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-10T19:19:42.486 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:19:42.486 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-10T19:19:42.486 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:27.174031+0000", 2024-09-10T19:19:42.486 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:19:42.486 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:19:42.486 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:19:42.486 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-10T19:19:42.486 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:27.174031+0000", 2024-09-10T19:19:42.486 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:19:42.486 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:19:42.486 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:19:42.487 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-10T19:19:42.487 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:27.174030+0000", 2024-09-10T19:19:42.487 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.999999 2024-09-10T19:19:42.487 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:19:42.487 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:19:42.487 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-10T19:19:42.487 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:27.174087+0000", 2024-09-10T19:19:42.487 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.6977e-05 2024-09-10T19:19:42.487 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:19:42.487 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:19:42.487 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-10T19:19:42.487 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:27.174098+0000", 2024-09-10T19:19:42.487 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.0872e-05 2024-09-10T19:19:42.487 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:19:42.487 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:19:42.488 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-10T19:19:42.488 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:27.174102+0000", 2024-09-10T19:19:42.488 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.2869999999999998e-06 2024-09-10T19:19:42.488 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:19:42.488 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:19:42.488 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:19:42.488 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:27.174144+0000", 2024-09-10T19:19:42.488 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.1989000000000001e-05 2024-09-10T19:19:42.488 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:19:42.488 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:19:42.488 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for peered", 2024-09-10T19:19:42.488 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:27.174150+0000", 2024-09-10T19:19:42.488 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.8939999999999998e-06 2024-09-10T19:19:42.488 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:19:42.488 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:19:42.489 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:19:42.489 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:27.945270+0000", 2024-09-10T19:19:42.489 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.77111963500000003 2024-09-10T19:19:42.489 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:19:42.489 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:19:42.489 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for peered", 2024-09-10T19:19:42.489 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:27.945275+0000", 2024-09-10T19:19:42.489 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.3229999999999997e-06 2024-09-10T19:19:42.489 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:19:42.489 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:19:42.489 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:19:42.489 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:28.947767+0000", 2024-09-10T19:19:42.489 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.0024922030000001 2024-09-10T19:19:42.489 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:19:42.489 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:19:42.490 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-10T19:19:42.490 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:28.947774+0000", 2024-09-10T19:19:42.490 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.0110000000000003e-06 2024-09-10T19:19:42.490 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:19:42.490 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-10T19:19:42.490 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:19:42.490 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:19:42.490 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:19:42.490 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.15051.0:93356 8.19 8:982b4d7e:::benchmark_data_smithi183_114924_object93355:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e179)", 2024-09-10T19:19:42.490 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-10T19:17:28.369111+0000", 2024-09-10T19:19:42.490 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 134.01403241, 2024-09-10T19:19:42.490 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.6680609639999999, 2024-09-10T19:19:42.490 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-10T19:19:42.491 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T19:19:42.491 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-10T19:19:42.491 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.15051", 2024-09-10T19:19:42.491 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.183:0/3632028672", 2024-09-10T19:19:42.491 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 93356 2024-09-10T19:19:42.491 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:19:42.491 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-10T19:19:42.491 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:19:42.491 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-10T19:19:42.491 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:28.369111+0000", 2024-09-10T19:19:42.491 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:19:42.491 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:19:42.491 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:19:42.492 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-10T19:19:42.492 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:28.369111+0000", 2024-09-10T19:19:42.492 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:19:42.492 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:19:42.492 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:19:42.492 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-10T19:19:42.492 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:28.369109+0000", 2024-09-10T19:19:42.492 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999981 2024-09-10T19:19:42.492 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:19:42.492 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:19:42.492 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-10T19:19:42.492 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:28.369196+0000", 2024-09-10T19:19:42.492 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.7334000000000001e-05 2024-09-10T19:19:42.493 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:19:42.493 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:19:42.493 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-10T19:19:42.493 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:28.369218+0000", 2024-09-10T19:19:42.493 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.1277999999999999e-05 2024-09-10T19:19:42.493 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:19:42.493 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:19:42.493 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-10T19:19:42.493 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:28.369225+0000", 2024-09-10T19:19:42.493 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.2989999999999999e-06 2024-09-10T19:19:42.493 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:19:42.493 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:19:42.493 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:19:42.494 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:28.369283+0000", 2024-09-10T19:19:42.494 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.7565000000000002e-05 2024-09-10T19:19:42.494 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:19:42.494 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:19:42.494 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for peered", 2024-09-10T19:19:42.494 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:28.369290+0000", 2024-09-10T19:19:42.494 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.6240000000000002e-06 2024-09-10T19:19:42.494 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:19:42.494 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:19:42.494 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:19:42.494 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:28.951079+0000", 2024-09-10T19:19:42.494 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.58178845800000001 2024-09-10T19:19:42.494 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:19:42.495 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:19:42.495 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for peered", 2024-09-10T19:19:42.495 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:28.951090+0000", 2024-09-10T19:19:42.495 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.0975000000000001e-05 2024-09-10T19:19:42.495 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:19:42.495 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:19:42.495 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:19:42.495 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:30.030802+0000", 2024-09-10T19:19:42.495 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.0797127639999999 2024-09-10T19:19:42.495 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:19:42.495 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:19:42.495 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-10T19:19:42.495 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:30.030973+0000", 2024-09-10T19:19:42.495 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00017021199999999999 2024-09-10T19:19:42.496 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:19:42.496 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:19:42.496 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0,4,6", 2024-09-10T19:19:42.496 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:30.031240+0000", 2024-09-10T19:19:42.496 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00026700299999999998 2024-09-10T19:19:42.496 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:19:42.496 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:19:42.496 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:19:42.496 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:30.033930+0000", 2024-09-10T19:19:42.496 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0026898680000000002 2024-09-10T19:19:42.496 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:19:42.496 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:19:42.496 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-09-10T19:19:42.496 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:30.033938+0000", 2024-09-10T19:19:42.496 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.5220000000000007e-06 2024-09-10T19:19:42.497 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:19:42.497 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:19:42.497 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:19:42.497 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:30.036650+0000", 2024-09-10T19:19:42.497 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0027118929999999999 2024-09-10T19:19:42.497 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:19:42.497 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:19:42.497 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:19:42.497 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:30.036962+0000", 2024-09-10T19:19:42.497 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00031158899999999997 2024-09-10T19:19:42.497 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:19:42.497 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:19:42.497 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-09-10T19:19:42.498 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:30.037028+0000", 2024-09-10T19:19:42.498 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.6637999999999995e-05 2024-09-10T19:19:42.498 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:19:42.498 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:19:42.498 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-10T19:19:42.498 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:30.037172+0000", 2024-09-10T19:19:42.498 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00014398199999999999 2024-09-10T19:19:42.498 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:19:42.498 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-10T19:19:42.498 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:19:42.498 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:19:42.498 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:19:42.499 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.15051.0:93371 8.19 8:9901a0b3:::benchmark_data_smithi183_114924_object93370:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e179)", 2024-09-10T19:19:42.499 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-10T19:17:28.414412+0000", 2024-09-10T19:19:42.499 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 133.96873179400001, 2024-09-10T19:19:42.499 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.627689602, 2024-09-10T19:19:42.499 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-10T19:19:42.499 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T19:19:42.499 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-10T19:19:42.499 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.15051", 2024-09-10T19:19:42.499 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.183:0/3632028672", 2024-09-10T19:19:42.499 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 93371 2024-09-10T19:19:42.499 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:19:42.499 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-10T19:19:42.499 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:19:42.500 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-10T19:19:42.500 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:28.414412+0000", 2024-09-10T19:19:42.500 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:19:42.500 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:19:42.500 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:19:42.500 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-10T19:19:42.500 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:28.414412+0000", 2024-09-10T19:19:42.500 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:19:42.500 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:19:42.500 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:19:42.500 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-10T19:19:42.500 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:28.414411+0000", 2024-09-10T19:19:42.500 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.999999 2024-09-10T19:19:42.500 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:19:42.501 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:19:42.501 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-10T19:19:42.501 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:28.414465+0000", 2024-09-10T19:19:42.501 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.4323000000000001e-05 2024-09-10T19:19:42.501 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:19:42.501 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:19:42.501 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-10T19:19:42.501 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:28.414477+0000", 2024-09-10T19:19:42.501 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.1323e-05 2024-09-10T19:19:42.501 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:19:42.501 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:19:42.501 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-10T19:19:42.501 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:28.414481+0000", 2024-09-10T19:19:42.501 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.1930000000000003e-06 2024-09-10T19:19:42.502 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:19:42.502 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:19:42.502 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:19:42.502 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:28.414522+0000", 2024-09-10T19:19:42.502 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.1121999999999997e-05 2024-09-10T19:19:42.502 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:19:42.502 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:19:42.502 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for peered", 2024-09-10T19:19:42.502 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:28.414529+0000", 2024-09-10T19:19:42.502 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.7959999999999998e-06 2024-09-10T19:19:42.502 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:19:42.502 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:19:42.502 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:19:42.503 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:28.951123+0000", 2024-09-10T19:19:42.503 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.53659462000000002 2024-09-10T19:19:42.503 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:19:42.503 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:19:42.503 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for peered", 2024-09-10T19:19:42.503 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:28.951131+0000", 2024-09-10T19:19:42.503 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.5070000000000003e-06 2024-09-10T19:19:42.503 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:19:42.503 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:19:42.503 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:19:42.503 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:30.033963+0000", 2024-09-10T19:19:42.503 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.0828317970000001 2024-09-10T19:19:42.503 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:19:42.503 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:19:42.504 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-10T19:19:42.504 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:30.034106+0000", 2024-09-10T19:19:42.504 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000143516 2024-09-10T19:19:42.504 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:19:42.504 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:19:42.504 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0,4,6", 2024-09-10T19:19:42.504 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:30.034378+0000", 2024-09-10T19:19:42.504 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00027172699999999998 2024-09-10T19:19:42.504 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:19:42.504 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:19:42.504 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:19:42.505 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:30.037340+0000", 2024-09-10T19:19:42.505 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0029616719999999998 2024-09-10T19:19:42.505 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:19:42.505 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:19:42.505 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-09-10T19:19:42.505 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:30.040333+0000", 2024-09-10T19:19:42.505 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0029933360000000001 2024-09-10T19:19:42.505 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:19:42.505 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:19:42.505 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:19:42.505 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:30.040365+0000", 2024-09-10T19:19:42.505 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.2110000000000003e-05 2024-09-10T19:19:42.505 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:19:42.506 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:19:42.506 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:19:42.506 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:30.041943+0000", 2024-09-10T19:19:42.506 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0015777359999999999 2024-09-10T19:19:42.506 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:19:42.506 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:19:42.506 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-09-10T19:19:42.506 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:30.041995+0000", 2024-09-10T19:19:42.506 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.1918e-05 2024-09-10T19:19:42.506 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:19:42.506 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:19:42.506 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-10T19:19:42.506 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:30.042101+0000", 2024-09-10T19:19:42.506 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00010680599999999999 2024-09-10T19:19:42.506 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:19:42.507 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-10T19:19:42.507 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:19:42.507 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:19:42.507 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:19:42.507 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.15051.0:151612 8.11 8:89d8a486:::benchmark_data_smithi183_114924_object31329:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e198)", 2024-09-10T19:19:42.507 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-10T19:18:23.050999+0000", 2024-09-10T19:19:42.507 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 79.332144427000003, 2024-09-10T19:19:42.507 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.97268184599999996, 2024-09-10T19:19:42.507 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-10T19:19:42.507 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T19:19:42.507 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-10T19:19:42.507 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.15051", 2024-09-10T19:19:42.507 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.183:0/3632028672", 2024-09-10T19:19:42.507 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 151612 2024-09-10T19:19:42.507 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:19:42.507 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-10T19:19:42.508 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:19:42.508 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-10T19:19:42.508 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:18:23.050999+0000", 2024-09-10T19:19:42.508 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:19:42.508 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:19:42.508 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:19:42.508 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-10T19:19:42.508 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:18:23.050999+0000", 2024-09-10T19:19:42.508 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:19:42.508 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:19:42.508 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:19:42.508 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-10T19:19:42.508 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:18:23.050998+0000", 2024-09-10T19:19:42.509 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.999999 2024-09-10T19:19:42.509 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:19:42.509 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:19:42.509 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-10T19:19:42.509 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:18:23.051003+0000", 2024-09-10T19:19:42.509 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.7589999999999997e-06 2024-09-10T19:19:42.509 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:19:42.509 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:19:42.509 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-10T19:19:42.509 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:18:23.051012+0000", 2024-09-10T19:19:42.509 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.9879999999999993e-06 2024-09-10T19:19:42.509 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:19:42.509 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:19:42.510 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-10T19:19:42.510 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:18:23.051017+0000", 2024-09-10T19:19:42.510 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.7879999999999997e-06 2024-09-10T19:19:42.510 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:19:42.510 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:19:42.510 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:19:42.510 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:18:23.051063+0000", 2024-09-10T19:19:42.510 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.6418999999999997e-05 2024-09-10T19:19:42.510 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:19:42.510 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:19:42.510 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for peered", 2024-09-10T19:19:42.510 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:18:23.051075+0000", 2024-09-10T19:19:42.510 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.2439000000000001e-05 2024-09-10T19:19:42.510 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:19:42.511 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:19:42.511 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:19:42.511 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:18:24.018580+0000", 2024-09-10T19:19:42.511 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.96750444700000005 2024-09-10T19:19:42.511 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:19:42.511 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:19:42.511 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-10T19:19:42.511 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:18:24.018674+0000", 2024-09-10T19:19:42.511 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.4519999999999996e-05 2024-09-10T19:19:42.511 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:19:42.511 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:19:42.511 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 6,7", 2024-09-10T19:19:42.511 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:18:24.018949+0000", 2024-09-10T19:19:42.511 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00027485200000000002 2024-09-10T19:19:42.512 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:19:42.512 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:19:42.512 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:19:42.512 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:18:24.020832+0000", 2024-09-10T19:19:42.512 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0018829529999999999 2024-09-10T19:19:42.512 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:19:42.512 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:19:42.512 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-09-10T19:19:42.512 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:18:24.020841+0000", 2024-09-10T19:19:42.512 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.5199999999999997e-06 2024-09-10T19:19:42.512 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:19:42.512 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:19:42.512 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:19:42.513 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:18:24.023578+0000", 2024-09-10T19:19:42.513 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0027376710000000001 2024-09-10T19:19:42.513 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:19:42.513 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:19:42.513 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-09-10T19:19:42.513 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:18:24.023644+0000", 2024-09-10T19:19:42.513 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.5640000000000002e-05 2024-09-10T19:19:42.513 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:19:42.513 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:19:42.513 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-10T19:19:42.513 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:18:24.023681+0000", 2024-09-10T19:19:42.514 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.6977000000000002e-05 2024-09-10T19:19:42.514 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:19:42.514 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-10T19:19:42.514 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:19:42.514 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:19:42.514 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:19:42.514 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.15051.0:151614 8.11 8:8b9042c4:::benchmark_data_smithi183_114924_object31331:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e198)", 2024-09-10T19:19:42.514 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-10T19:18:23.052048+0000", 2024-09-10T19:19:42.514 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 79.331095387999994, 2024-09-10T19:19:42.514 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.97478914900000002, 2024-09-10T19:19:42.514 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-10T19:19:42.514 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T19:19:42.514 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-10T19:19:42.514 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.15051", 2024-09-10T19:19:42.515 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.183:0/3632028672", 2024-09-10T19:19:42.515 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 151614 2024-09-10T19:19:42.515 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:19:42.515 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-10T19:19:42.515 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:19:42.515 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-10T19:19:42.515 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:18:23.052048+0000", 2024-09-10T19:19:42.515 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:19:42.515 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:19:42.515 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:19:42.516 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-10T19:19:42.516 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:18:23.052048+0000", 2024-09-10T19:19:42.516 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:19:42.516 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:19:42.516 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:19:42.516 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-10T19:19:42.516 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:18:23.052047+0000", 2024-09-10T19:19:42.516 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.999999 2024-09-10T19:19:42.516 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:19:42.516 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:19:42.516 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-10T19:19:42.516 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:18:23.052052+0000", 2024-09-10T19:19:42.516 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.8539999999999996e-06 2024-09-10T19:19:42.517 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:19:42.517 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:19:42.517 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-10T19:19:42.517 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:18:23.052065+0000", 2024-09-10T19:19:42.517 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.2468e-05 2024-09-10T19:19:42.517 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:19:42.517 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:19:42.517 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-10T19:19:42.517 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:18:23.052071+0000", 2024-09-10T19:19:42.517 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.9290000000000003e-06 2024-09-10T19:19:42.517 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:19:42.517 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:19:42.517 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:19:42.518 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:18:23.052109+0000", 2024-09-10T19:19:42.518 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.8807999999999998e-05 2024-09-10T19:19:42.518 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:19:42.518 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:19:42.518 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for peered", 2024-09-10T19:19:42.518 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:18:23.052118+0000", 2024-09-10T19:19:42.518 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.9919999999999996e-06 2024-09-10T19:19:42.518 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:19:42.518 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:19:42.518 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:19:42.518 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:18:24.020885+0000", 2024-09-10T19:19:42.518 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.96876697199999995 2024-09-10T19:19:42.519 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:19:42.519 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:19:42.519 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-10T19:19:42.519 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:18:24.021004+0000", 2024-09-10T19:19:42.519 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000118183 2024-09-10T19:19:42.519 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:19:42.519 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:19:42.519 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 6,7", 2024-09-10T19:19:42.519 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:18:24.021207+0000", 2024-09-10T19:19:42.519 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000203401 2024-09-10T19:19:42.519 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:19:42.519 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:19:42.519 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-09-10T19:19:42.519 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:18:24.023784+0000", 2024-09-10T19:19:42.520 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00257748 2024-09-10T19:19:42.520 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:19:42.520 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:19:42.520 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:19:42.520 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:18:24.023822+0000", 2024-09-10T19:19:42.520 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.7171e-05 2024-09-10T19:19:42.520 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:19:42.520 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:19:42.520 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:19:42.520 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:18:24.026734+0000", 2024-09-10T19:19:42.520 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0029124839999999999 2024-09-10T19:19:42.520 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:19:42.520 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:19:42.520 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-09-10T19:19:42.521 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:18:24.026804+0000", 2024-09-10T19:19:42.521 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.0196000000000004e-05 2024-09-10T19:19:42.521 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:19:42.521 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:19:42.521 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-10T19:19:42.521 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:18:24.026837+0000", 2024-09-10T19:19:42.521 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.3031000000000003e-05 2024-09-10T19:19:42.521 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:19:42.521 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-10T19:19:42.521 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:19:42.521 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:19:42.521 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-10T19:19:42.522 INFO:teuthology.orchestra.run.smithi191.stdout:} 2024-09-10T19:19:42.522 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:19:42.544 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:19:42.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:42.545 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:19:42.645 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:19:42.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:42.668 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:19:42.668 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:19:42.769 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:19:42.790 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:19:42.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:42.791 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:19:42.892 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:19:42.913 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:19:42.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:42.914 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:19:42.931 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:19:42 smithi183 ceph-mon[41562]: pgmap v423: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 23 MiB data, 2.6 GiB used, 534 GiB / 536 GiB avail; 1023 B/s rd, 0 B/s wr, 637 op/s; 14/1677 objects misplaced (0.835%) 2024-09-10T19:19:42.931 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:19:42 smithi183 ceph-mon[41562]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T19:19:42.931 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:19:42 smithi183 ceph-mon[41562]: from='client.? 172.21.15.183:0/2774054526' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-10T19:19:42.931 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:19:42 smithi183 ceph-mon[41562]: osdmap e225: 8 total, 8 up, 6 in 2024-09-10T19:19:42.931 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:19:42 smithi183 ceph-mon[29736]: pgmap v423: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 23 MiB data, 2.6 GiB used, 534 GiB / 536 GiB avail; 1023 B/s rd, 0 B/s wr, 637 op/s; 14/1677 objects misplaced (0.835%) 2024-09-10T19:19:42.932 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:19:42 smithi183 ceph-mon[29736]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T19:19:42.932 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:19:42 smithi183 ceph-mon[29736]: from='client.? 172.21.15.183:0/2774054526' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-10T19:19:42.932 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:19:42 smithi183 ceph-mon[29736]: osdmap e225: 8 total, 8 up, 6 in 2024-09-10T19:19:43.015 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:19:43.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:43.038 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:19:43.038 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:19:43.139 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:19:43.160 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:19:43.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:43.161 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:19:43.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:19:42 smithi191 ceph-mon[32884]: pgmap v423: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 23 MiB data, 2.6 GiB used, 534 GiB / 536 GiB avail; 1023 B/s rd, 0 B/s wr, 637 op/s; 14/1677 objects misplaced (0.835%) 2024-09-10T19:19:43.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:19:42 smithi191 ceph-mon[32884]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T19:19:43.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:19:42 smithi191 ceph-mon[32884]: from='client.? 172.21.15.183:0/2774054526' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-10T19:19:43.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:19:42 smithi191 ceph-mon[32884]: osdmap e225: 8 total, 8 up, 6 in 2024-09-10T19:19:43.262 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:19:43.284 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:19:43.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:43.285 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:19:43.386 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:19:43.407 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:19:43.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:43.408 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:19:43.509 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:19:43.519 DEBUG:teuthology.orchestra.run.smithi191:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph daemon osd.6 dump_ops_in_flight 2024-09-10T19:19:43.531 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:19:43.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:43.532 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:19:43.632 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:19:43.654 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:19:43.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:43.655 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:19:43.756 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:19:43.777 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:19:43.777 INFO:teuthology.orchestra.run.smithi191.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.b/config 2024-09-10T19:19:43.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:43.778 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:19:43.879 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:19:43.900 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:19:43.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:43.901 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:19:44.002 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:19:44.023 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:19:44.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:44.024 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:19:44.125 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:19:44.146 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:19:44.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:44.147 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:19:44.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:19:43 smithi183 ceph-mon[29736]: purged_snaps scrub starts 2024-09-10T19:19:44.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:19:43 smithi183 ceph-mon[29736]: purged_snaps scrub ok 2024-09-10T19:19:44.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:19:43 smithi183 ceph-mon[29736]: osdmap e226: 8 total, 8 up, 6 in 2024-09-10T19:19:44.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:19:43 smithi183 ceph-mon[29736]: from='mgr.14556 ' entity='mgr.y' 2024-09-10T19:19:44.182 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:19:43 smithi183 ceph-mon[29736]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:19:44.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:19:43 smithi183 ceph-mon[41562]: purged_snaps scrub starts 2024-09-10T19:19:44.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:19:43 smithi183 ceph-mon[41562]: purged_snaps scrub ok 2024-09-10T19:19:44.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:19:43 smithi183 ceph-mon[41562]: osdmap e226: 8 total, 8 up, 6 in 2024-09-10T19:19:44.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:19:43 smithi183 ceph-mon[41562]: from='mgr.14556 ' entity='mgr.y' 2024-09-10T19:19:44.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:19:43 smithi183 ceph-mon[41562]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:19:44.207 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:19:43 smithi191 ceph-mon[32884]: purged_snaps scrub starts 2024-09-10T19:19:44.208 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:19:43 smithi191 ceph-mon[32884]: purged_snaps scrub ok 2024-09-10T19:19:44.208 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:19:43 smithi191 ceph-mon[32884]: osdmap e226: 8 total, 8 up, 6 in 2024-09-10T19:19:44.208 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:19:43 smithi191 ceph-mon[32884]: from='mgr.14556 ' entity='mgr.y' 2024-09-10T19:19:44.208 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:19:43 smithi191 ceph-mon[32884]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:19:44.248 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:19:44.270 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:19:44.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:44.271 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:19:44.296 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:19:44.372 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:19:44.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:44.394 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:19:44.395 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:19:44.495 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:19:44.516 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:19:44.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:44.517 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:19:44.618 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:19:44.640 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:19:44.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:44.641 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:19:44.742 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:19:44.764 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:19:44.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:44.765 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:19:44.866 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:19:44.888 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:19:44.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:44.889 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:19:44.990 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:19:45.012 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:19:45.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:45.013 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:19:45.114 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:19:45.121 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph osd unset noscrub 2024-09-10T19:19:45.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:45.135 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:19:45.136 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:19:45.143 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:19:44 smithi183 ceph-mon[29736]: pgmap v426: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 23 MiB data, 2.6 GiB used, 534 GiB / 536 GiB avail; 639 B/s rd, 0 B/s wr, 3 op/s; 14/1677 objects misplaced (0.835%) 2024-09-10T19:19:45.144 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:19:44 smithi183 ceph-mon[41562]: pgmap v426: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 23 MiB data, 2.6 GiB used, 534 GiB / 536 GiB avail; 639 B/s rd, 0 B/s wr, 3 op/s; 14/1677 objects misplaced (0.835%) 2024-09-10T19:19:45.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:19:44 smithi191 ceph-mon[32884]: pgmap v426: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 23 MiB data, 2.6 GiB used, 534 GiB / 536 GiB avail; 639 B/s rd, 0 B/s wr, 3 op/s; 14/1677 objects misplaced (0.835%) 2024-09-10T19:19:45.237 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:19:45.257 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:19:45.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:45.258 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:19:45.359 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:19:45.381 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:19:45.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:45.382 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:19:45.482 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:19:45.504 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:19:45.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:45.505 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:19:45.605 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:19:45.627 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:19:45.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:45.628 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:19:45.729 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:19:45.755 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:19:45.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:45.756 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:19:45.811 INFO:teuthology.orchestra.run.smithi191.stdout:{ 2024-09-10T19:19:45.811 INFO:teuthology.orchestra.run.smithi191.stdout: "ops": [], 2024-09-10T19:19:45.811 INFO:teuthology.orchestra.run.smithi191.stdout: "num_ops": 0 2024-09-10T19:19:45.811 INFO:teuthology.orchestra.run.smithi191.stdout:} 2024-09-10T19:19:45.857 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:19:45.878 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:19:45.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:45.879 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:19:45.980 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:19:46.001 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:19:46.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:46.002 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:19:46.103 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:19:46.125 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:19:46.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:46.126 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:19:46.226 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:19:46.248 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:19:46.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:46.249 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:19:46.349 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:19:46.371 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:19:46.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:46.372 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:19:46.386 INFO:teuthology.orchestra.run.smithi183.stdout:osd.0: {} 2024-09-10T19:19:46.386 INFO:teuthology.orchestra.run.smithi183.stdout:osd.1: {} 2024-09-10T19:19:46.386 INFO:teuthology.orchestra.run.smithi183.stdout:osd.2: {} 2024-09-10T19:19:46.386 INFO:teuthology.orchestra.run.smithi183.stdout:osd.3: {} 2024-09-10T19:19:46.386 INFO:teuthology.orchestra.run.smithi183.stdout:osd.4: {} 2024-09-10T19:19:46.386 INFO:teuthology.orchestra.run.smithi183.stdout:osd.5: {} 2024-09-10T19:19:46.386 INFO:teuthology.orchestra.run.smithi183.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-10T19:19:46.386 INFO:teuthology.orchestra.run.smithi183.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-10T19:19:46.386 INFO:teuthology.orchestra.run.smithi183.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-10T19:19:46.387 INFO:teuthology.orchestra.run.smithi183.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-10T19:19:46.387 INFO:teuthology.orchestra.run.smithi183.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-10T19:19:46.387 INFO:teuthology.orchestra.run.smithi183.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-10T19:19:46.388 INFO:teuthology.orchestra.run.smithi183.stdout:osd.6: {} 2024-09-10T19:19:46.389 INFO:teuthology.orchestra.run.smithi183.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-10T19:19:46.403 INFO:teuthology.orchestra.run.smithi183.stdout:osd.7: {} 2024-09-10T19:19:46.404 INFO:teuthology.orchestra.run.smithi183.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-10T19:19:46.441 DEBUG:teuthology.orchestra.run.smithi191:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph daemon osd.6 dump_blocked_ops 2024-09-10T19:19:46.474 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:19:46.504 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:19:46.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:46.505 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:19:46.606 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:19:46.628 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:19:46.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:46.629 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:19:46.685 INFO:teuthology.orchestra.run.smithi191.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.b/config 2024-09-10T19:19:46.730 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:19:46.751 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:19:46.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:46.752 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:19:46.852 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:19:46.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:46.875 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:19:46.875 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:19:46.976 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:19:46.998 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:19:46.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:46.999 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:19:47.032 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:19:46 smithi183 ceph-mon[29736]: pgmap v427: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 23 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 4 op/s; 14/1677 objects misplaced (0.835%) 2024-09-10T19:19:47.032 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:19:46 smithi183 ceph-mon[29736]: from='client.? 172.21.15.183:0/647307504' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T19:19:47.033 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:19:46 smithi183 ceph-mon[41562]: pgmap v427: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 23 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 4 op/s; 14/1677 objects misplaced (0.835%) 2024-09-10T19:19:47.033 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:19:46 smithi183 ceph-mon[41562]: from='client.? 172.21.15.183:0/647307504' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T19:19:47.100 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:19:47.122 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:19:47.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:47.123 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:19:47.130 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:19:46 smithi191 ceph-mon[32884]: pgmap v427: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 23 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 4 op/s; 14/1677 objects misplaced (0.835%) 2024-09-10T19:19:47.131 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:19:46 smithi191 ceph-mon[32884]: from='client.? 172.21.15.183:0/647307504' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T19:19:47.224 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:19:47.246 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:19:47.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:47.247 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:19:47.348 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:19:47.369 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:19:47.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:47.370 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:19:47.471 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:19:47.493 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:19:47.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:47.494 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:19:47.595 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:19:47.616 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:19:47.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:47.617 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:19:47.716 INFO:journalctl@ceph.mgr.x.smithi191.stdout:Sep 10 19:19:47 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-x[34967]: ::ffff:172.21.15.191 - - [10/Sep/2024:19:19:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T19:19:47.718 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:19:47.739 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:19:47.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:47.741 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:19:47.841 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:19:47.863 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:19:47.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:47.864 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:19:47.869 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:19:47 smithi183 ceph-mon[41562]: from='client.? 172.21.15.183:0/747982072' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_1", "pool2": "unique_pool_1", "yes_i_really_really_mean_it": true}]: dispatch 2024-09-10T19:19:47.869 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:19:47 smithi183 ceph-mon[41562]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_1", "pool2": "unique_pool_1", "yes_i_really_really_mean_it": true}]: dispatch 2024-09-10T19:19:47.874 INFO:teuthology.orchestra.run.smithi183.stderr:pool 'unique_pool_1' removed 2024-09-10T19:19:47.965 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:19:47.988 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:19:47.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:47.989 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:19:48.090 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:19:48.117 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:19:48.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:48.118 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:19:48.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:19:47 smithi183 ceph-mon[29736]: from='client.? 172.21.15.183:0/747982072' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_1", "pool2": "unique_pool_1", "yes_i_really_really_mean_it": true}]: dispatch 2024-09-10T19:19:48.182 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:19:47 smithi183 ceph-mon[29736]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_1", "pool2": "unique_pool_1", "yes_i_really_really_mean_it": true}]: dispatch 2024-09-10T19:19:48.199 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:19:47 smithi191 ceph-mon[32884]: from='client.? 172.21.15.183:0/747982072' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_1", "pool2": "unique_pool_1", "yes_i_really_really_mean_it": true}]: dispatch 2024-09-10T19:19:48.199 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:19:47 smithi191 ceph-mon[32884]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_1", "pool2": "unique_pool_1", "yes_i_really_really_mean_it": true}]: dispatch 2024-09-10T19:19:48.219 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:19:48.240 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:19:48.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:48.241 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:19:48.342 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:19:48.364 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:19:48.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:48.366 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:19:48.458 INFO:teuthology.orchestra.run.smithi191.stdout:ERROR: (22) Invalid argument 2024-09-10T19:19:48.458 INFO:teuthology.orchestra.run.smithi191.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-10T19:19:48.467 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:19:48.489 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:19:48.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:48.489 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:19:48.574 INFO:teuthology.task.full_sequential:In full_sequential, running task radosbench... 2024-09-10T19:19:48.575 INFO:tasks.radosbench:Beginning radosbench... 2024-09-10T19:19:48.575 INFO:tasks.cephadm.ceph_manager.ceph:creating pool_name unique_pool_2 2024-09-10T19:19:48.575 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph osd pool create unique_pool_2 16 2024-09-10T19:19:48.590 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:19:48.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:48.641 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:19:48.642 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:19:48.742 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:19:48.766 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:19:48.767 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:19:48.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:48.768 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:19:48.869 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:19:48.891 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:19:48.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:48.892 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:19:48.947 INFO:teuthology.orchestra.run.smithi183.stderr:Error ENOENT: pg 8.e does not exist 2024-09-10T19:19:48.993 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:19:49.015 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:19:49.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:49.016 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:19:49.098 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:19:48 smithi191 ceph-mon[32884]: pgmap v428: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 23 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 4 op/s; 14/1677 objects misplaced (0.835%) 2024-09-10T19:19:49.099 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:19:48 smithi191 ceph-mon[32884]: purged_snaps scrub starts 2024-09-10T19:19:49.099 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:19:48 smithi191 ceph-mon[32884]: purged_snaps scrub ok 2024-09-10T19:19:49.099 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:19:48 smithi191 ceph-mon[32884]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_1", "pool2": "unique_pool_1", "yes_i_really_really_mean_it": true}]': finished 2024-09-10T19:19:49.099 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:19:48 smithi191 ceph-mon[32884]: osdmap e227: 8 total, 8 up, 6 in 2024-09-10T19:19:49.116 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:19:49.135 DEBUG:teuthology.orchestra.run.smithi191:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph daemon osd.6 dump_historic_ops 2024-09-10T19:19:49.140 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:19:49.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:49.141 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:19:49.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:19:48 smithi183 ceph-mon[29736]: pgmap v428: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 23 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 4 op/s; 14/1677 objects misplaced (0.835%) 2024-09-10T19:19:49.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:19:48 smithi183 ceph-mon[29736]: purged_snaps scrub starts 2024-09-10T19:19:49.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:19:48 smithi183 ceph-mon[29736]: purged_snaps scrub ok 2024-09-10T19:19:49.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:19:48 smithi183 ceph-mon[29736]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_1", "pool2": "unique_pool_1", "yes_i_really_really_mean_it": true}]': finished 2024-09-10T19:19:49.182 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:19:48 smithi183 ceph-mon[29736]: osdmap e227: 8 total, 8 up, 6 in 2024-09-10T19:19:49.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:19:48 smithi183 ceph-mon[41562]: pgmap v428: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 23 MiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 4 op/s; 14/1677 objects misplaced (0.835%) 2024-09-10T19:19:49.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:19:48 smithi183 ceph-mon[41562]: purged_snaps scrub starts 2024-09-10T19:19:49.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:19:48 smithi183 ceph-mon[41562]: purged_snaps scrub ok 2024-09-10T19:19:49.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:19:48 smithi183 ceph-mon[41562]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_1", "pool2": "unique_pool_1", "yes_i_really_really_mean_it": true}]': finished 2024-09-10T19:19:49.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:19:48 smithi183 ceph-mon[41562]: osdmap e227: 8 total, 8 up, 6 in 2024-09-10T19:19:49.243 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:19:49.274 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:19:49.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:49.275 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:19:49.376 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:19:49.399 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:19:49.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:49.400 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:19:49.419 INFO:teuthology.orchestra.run.smithi191.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.b/config 2024-09-10T19:19:49.501 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:19:49.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:49.526 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:19:49.526 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:19:49.627 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:19:49.649 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-10T19:19:49.650 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:19:49.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:49.651 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:19:49.662 DEBUG:teuthology.orchestra.run:got remote process result: 2 2024-09-10T19:19:49.663 INFO:tasks.thrashosds.thrasher:Failed to rm-pg-upmap-items, ignoring 2024-09-10T19:19:49.677 INFO:journalctl@ceph.iscsi.iscsi.a.smithi191.stdout:Sep 10 19:19:49 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-iscsi-iscsi-a[61816]: debug there is no tcmu-runner data available 2024-09-10T19:19:49.752 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:19:49.774 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:19:49.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:49.775 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:19:49.875 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:19:49.897 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:19:49.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:49.898 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:19:49.926 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:19:49 smithi183 ceph-mon[29736]: from='client.? 172.21.15.183:0/316663565' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "8.e", "id": [6, 5, 2, 7, 3, 1]}]: dispatch 2024-09-10T19:19:49.926 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:19:49 smithi183 ceph-mon[29736]: osdmap e228: 8 total, 8 up, 6 in 2024-09-10T19:19:49.926 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:19:49 smithi183 ceph-mon[41562]: from='client.? 172.21.15.183:0/316663565' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "8.e", "id": [6, 5, 2, 7, 3, 1]}]: dispatch 2024-09-10T19:19:49.926 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:19:49 smithi183 ceph-mon[41562]: osdmap e228: 8 total, 8 up, 6 in 2024-09-10T19:19:49.928 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:19:49 smithi191 ceph-mon[32884]: from='client.? 172.21.15.183:0/316663565' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "8.e", "id": [6, 5, 2, 7, 3, 1]}]: dispatch 2024-09-10T19:19:49.929 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:19:49 smithi191 ceph-mon[32884]: osdmap e228: 8 total, 8 up, 6 in 2024-09-10T19:19:49.999 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:19:50.022 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:19:50.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:50.023 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:19:50.124 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:19:50.125 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:19:50.147 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:19:50.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:50.148 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:19:50.249 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:19:50.270 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:19:50.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:50.271 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:19:50.372 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:19:50.395 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:19:50.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:50.396 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:19:50.497 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:19:50.519 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:19:50.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:50.520 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:19:50.621 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:19:50.642 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:19:50.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:50.643 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:19:50.744 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:19:50.766 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:19:50.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:50.767 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:19:50.868 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:19:50.889 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:19:50.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:50.890 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:19:50.991 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:19:51.014 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:19:51.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:51.015 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:19:51.106 INFO:teuthology.orchestra.run.smithi191.stdout:ERROR: (22) Invalid argument 2024-09-10T19:19:51.106 INFO:teuthology.orchestra.run.smithi191.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-10T19:19:51.116 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:19:51.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:51.138 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:19:51.139 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:19:51.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:19:50 smithi183 ceph-mon[29736]: from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:19:51.182 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:19:50 smithi183 ceph-mon[29736]: pgmap v430: 132 pgs: 132 active+clean; 582 KiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 517 B/s rd, 0 op/s 2024-09-10T19:19:51.182 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:19:50 smithi183 ceph-mon[29736]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull) 2024-09-10T19:19:51.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:19:50 smithi183 ceph-mon[41562]: from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:19:51.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:19:50 smithi183 ceph-mon[41562]: pgmap v430: 132 pgs: 132 active+clean; 582 KiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 517 B/s rd, 0 op/s 2024-09-10T19:19:51.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:19:50 smithi183 ceph-mon[41562]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull) 2024-09-10T19:19:51.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:19:50 smithi191 ceph-mon[32884]: from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:19:51.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:19:50 smithi191 ceph-mon[32884]: pgmap v430: 132 pgs: 132 active+clean; 582 KiB data, 2.5 GiB used, 534 GiB / 536 GiB avail; 517 B/s rd, 0 op/s 2024-09-10T19:19:51.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:19:50 smithi191 ceph-mon[32884]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull) 2024-09-10T19:19:51.240 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:19:51.262 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:19:51.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:51.262 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:19:51.363 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:19:51.385 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:19:51.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:51.386 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:19:51.486 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:19:51.505 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Sep 10 19:19:51 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-y[29989]: ::ffff:172.21.15.191 - - [10/Sep/2024:19:19:51] "GET /metrics HTTP/1.1" 200 224218 "" "Prometheus/2.43.0" 2024-09-10T19:19:51.507 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:19:51.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:51.508 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:19:51.608 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:19:51.629 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:19:51.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:51.630 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:19:51.731 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:19:51.753 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:19:51.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:51.753 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:19:51.816 DEBUG:teuthology.orchestra.run.smithi191:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph daemon osd.7 dump_ops_in_flight 2024-09-10T19:19:51.854 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:19:51.877 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:19:51.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:51.878 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:19:51.978 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:19:52.000 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:19:52.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:52.001 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:19:52.060 INFO:teuthology.orchestra.run.smithi191.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.b/config 2024-09-10T19:19:52.102 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:19:52.125 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:19:52.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:52.126 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:19:52.226 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:19:52.248 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:19:52.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:52.249 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:19:52.350 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:19:52.372 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:19:52.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:52.372 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:19:52.473 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:19:52.495 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:19:52.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:52.496 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:19:52.597 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:19:52.620 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:19:52.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:52.621 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:19:52.722 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:19:52.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:52.744 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:19:52.745 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:19:52.846 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:19:52.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:52.870 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:19:52.870 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:19:52.933 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:19:52.971 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:19:52.992 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:19:52.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:52.993 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:19:53.094 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:19:53.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:53.117 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:19:53.117 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:19:53.187 INFO:teuthology.orchestra.run.smithi183.stderr:pool 'unique_pool_2' created 2024-09-10T19:19:53.192 INFO:teuthology.orchestra.run.smithi183.stderr:noscrub is unset 2024-09-10T19:19:53.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:19:53 smithi191 ceph-mon[32884]: pgmap v432: 132 pgs: 132 active+clean; 582 KiB data, 2.4 GiB used, 534 GiB / 536 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-10T19:19:53.218 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:19:53.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:53.241 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:19:53.242 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:19:53.343 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:19:53.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:53.368 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:19:53.368 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:19:53.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:19:53 smithi183 ceph-mon[29736]: pgmap v432: 132 pgs: 132 active+clean; 582 KiB data, 2.4 GiB used, 534 GiB / 536 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-10T19:19:53.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:19:53 smithi183 ceph-mon[41562]: pgmap v432: 132 pgs: 132 active+clean; 582 KiB data, 2.4 GiB used, 534 GiB / 536 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-10T19:19:53.468 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:19:53.490 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:19:53.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:53.491 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:19:53.592 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:19:53.613 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:19:53.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:53.614 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:19:53.715 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:19:53.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:53.737 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:19:53.737 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:19:53.803 INFO:teuthology.orchestra.run.smithi191.stdout:ERROR: (22) Invalid argument 2024-09-10T19:19:53.803 INFO:teuthology.orchestra.run.smithi191.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-10T19:19:53.839 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:19:53.860 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:19:53.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:53.861 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:19:53.962 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:19:53.983 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:19:53.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:53.984 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:19:54.085 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:19:54.106 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:19:54.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:54.108 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:19:54.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:19:54 smithi183 ceph-mon[29736]: from='client.? 172.21.15.183:0/2051641446' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_2", "pg_num": 16}]: dispatch 2024-09-10T19:19:54.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:19:54 smithi183 ceph-mon[29736]: from='client.? 172.21.15.183:0/347453179' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-10T19:19:54.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:19:54 smithi183 ceph-mon[29736]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-10T19:19:54.182 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:19:54 smithi183 ceph-mon[29736]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T19:19:54.182 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:19:54 smithi183 ceph-mon[29736]: from='client.? 172.21.15.183:0/2051641446' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_2", "pg_num": 16}]': finished 2024-09-10T19:19:54.182 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:19:54 smithi183 ceph-mon[29736]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-10T19:19:54.182 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:19:54 smithi183 ceph-mon[29736]: osdmap e229: 8 total, 8 up, 6 in 2024-09-10T19:19:54.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:19:54 smithi183 ceph-mon[41562]: from='client.? 172.21.15.183:0/2051641446' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_2", "pg_num": 16}]: dispatch 2024-09-10T19:19:54.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:19:54 smithi183 ceph-mon[41562]: from='client.? 172.21.15.183:0/347453179' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-10T19:19:54.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:19:54 smithi183 ceph-mon[41562]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-10T19:19:54.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:19:54 smithi183 ceph-mon[41562]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T19:19:54.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:19:54 smithi183 ceph-mon[41562]: from='client.? 172.21.15.183:0/2051641446' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_2", "pg_num": 16}]': finished 2024-09-10T19:19:54.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:19:54 smithi183 ceph-mon[41562]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-10T19:19:54.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:19:54 smithi183 ceph-mon[41562]: osdmap e229: 8 total, 8 up, 6 in 2024-09-10T19:19:54.208 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:19:54.229 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:19:54.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:54.230 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:19:54.331 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:19:54.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:54.354 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:19:54.354 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:19:54.452 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:19:54 smithi191 ceph-mon[32884]: from='client.? 172.21.15.183:0/2051641446' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_2", "pg_num": 16}]: dispatch 2024-09-10T19:19:54.452 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:19:54 smithi191 ceph-mon[32884]: from='client.? 172.21.15.183:0/347453179' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-10T19:19:54.452 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:19:54 smithi191 ceph-mon[32884]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-10T19:19:54.452 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:19:54 smithi191 ceph-mon[32884]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T19:19:54.452 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:19:54 smithi191 ceph-mon[32884]: from='client.? 172.21.15.183:0/2051641446' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_2", "pg_num": 16}]': finished 2024-09-10T19:19:54.452 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:19:54 smithi191 ceph-mon[32884]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-10T19:19:54.452 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:19:54 smithi191 ceph-mon[32884]: osdmap e229: 8 total, 8 up, 6 in 2024-09-10T19:19:54.455 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:19:54.475 DEBUG:teuthology.orchestra.run.smithi191:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph daemon osd.7 dump_blocked_ops 2024-09-10T19:19:54.476 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:19:54.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:54.477 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:19:54.578 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:19:54.600 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:19:54.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:54.601 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:19:54.664 INFO:tasks.thrashosds.thrasher:in_osds: [0, 3, 4, 5, 6, 7] out_osds: [1, 2] dead_osds: [] live_osds: [0, 2, 3, 4, 5, 6, 7, 1] 2024-09-10T19:19:54.665 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-10T19:19:54.701 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:19:54.721 INFO:teuthology.orchestra.run.smithi191.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.b/config 2024-09-10T19:19:54.723 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:19:54.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:54.724 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:19:54.745 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph osd pool application enable unique_pool_2 rados --yes-i-really-mean-it || true 2024-09-10T19:19:54.825 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:19:54.847 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:19:54.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:54.848 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:19:54.948 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:19:54.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:54.970 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:19:54.971 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:19:55.071 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:19:55.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:55.094 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:19:55.094 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:19:55.195 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:19:55.216 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:19:55.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:55.217 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:19:55.318 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:19:55.341 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:19:55.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:55.341 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:19:55.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:19:55 smithi183 ceph-mon[29736]: pgmap v434: 148 pgs: 16 unknown, 132 active+clean; 582 KiB data, 2.4 GiB used, 534 GiB / 536 GiB avail; 682 B/s rd, 0 op/s 2024-09-10T19:19:55.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:19:55 smithi183 ceph-mon[29736]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T19:19:55.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:19:55 smithi183 ceph-mon[29736]: osdmap e230: 8 total, 8 up, 6 in 2024-09-10T19:19:55.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:19:55 smithi183 ceph-mon[29736]: 2.3 scrub starts 2024-09-10T19:19:55.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:19:55 smithi183 ceph-mon[29736]: 2.3 scrub ok 2024-09-10T19:19:55.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:19:55 smithi183 ceph-mon[41562]: pgmap v434: 148 pgs: 16 unknown, 132 active+clean; 582 KiB data, 2.4 GiB used, 534 GiB / 536 GiB avail; 682 B/s rd, 0 op/s 2024-09-10T19:19:55.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:19:55 smithi183 ceph-mon[41562]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T19:19:55.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:19:55 smithi183 ceph-mon[41562]: osdmap e230: 8 total, 8 up, 6 in 2024-09-10T19:19:55.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:19:55 smithi183 ceph-mon[41562]: 2.3 scrub starts 2024-09-10T19:19:55.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:19:55 smithi183 ceph-mon[41562]: 2.3 scrub ok 2024-09-10T19:19:55.443 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:19:55.464 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:19:55.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:55.465 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:19:55.465 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:19:55 smithi191 ceph-mon[32884]: pgmap v434: 148 pgs: 16 unknown, 132 active+clean; 582 KiB data, 2.4 GiB used, 534 GiB / 536 GiB avail; 682 B/s rd, 0 op/s 2024-09-10T19:19:55.465 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:19:55 smithi191 ceph-mon[32884]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T19:19:55.465 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:19:55 smithi191 ceph-mon[32884]: osdmap e230: 8 total, 8 up, 6 in 2024-09-10T19:19:55.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:19:55 smithi191 ceph-mon[32884]: 2.3 scrub starts 2024-09-10T19:19:55.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:19:55 smithi191 ceph-mon[32884]: 2.3 scrub ok 2024-09-10T19:19:55.565 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:19:55.588 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:19:55.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:55.589 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:19:55.690 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:19:55.712 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:19:55.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:55.713 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:19:55.814 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:19:55.835 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:19:55.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:55.836 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:19:55.937 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:19:55.960 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:19:55.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:55.961 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:19:56.062 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:19:56.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:56.084 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:19:56.084 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:19:56.185 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:19:56.206 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:19:56.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:56.207 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:19:56.289 INFO:teuthology.orchestra.run.smithi183.stdout:osd.0: {} 2024-09-10T19:19:56.289 INFO:teuthology.orchestra.run.smithi183.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-10T19:19:56.290 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:19:56 smithi183 ceph-mon[29736]: osdmap e231: 8 total, 8 up, 6 in 2024-09-10T19:19:56.290 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:19:56 smithi183 ceph-mon[41562]: osdmap e231: 8 total, 8 up, 6 in 2024-09-10T19:19:56.295 INFO:teuthology.orchestra.run.smithi183.stdout:osd.1: {} 2024-09-10T19:19:56.296 INFO:teuthology.orchestra.run.smithi183.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-10T19:19:56.296 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:19:56.308 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:19:56.311 INFO:teuthology.orchestra.run.smithi183.stdout:osd.2: {} 2024-09-10T19:19:56.311 INFO:teuthology.orchestra.run.smithi183.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-10T19:19:56.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:56.330 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:19:56.330 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:19:56.332 INFO:teuthology.orchestra.run.smithi183.stdout:osd.3: {} 2024-09-10T19:19:56.333 INFO:teuthology.orchestra.run.smithi183.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-10T19:19:56.348 INFO:teuthology.orchestra.run.smithi183.stdout:osd.4: {} 2024-09-10T19:19:56.348 INFO:teuthology.orchestra.run.smithi183.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-10T19:19:56.363 INFO:teuthology.orchestra.run.smithi183.stdout:osd.5: {} 2024-09-10T19:19:56.363 INFO:teuthology.orchestra.run.smithi183.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-10T19:19:56.378 INFO:teuthology.orchestra.run.smithi183.stdout:osd.6: {} 2024-09-10T19:19:56.378 INFO:teuthology.orchestra.run.smithi183.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-10T19:19:56.393 INFO:teuthology.orchestra.run.smithi183.stdout:osd.7: {} 2024-09-10T19:19:56.394 INFO:teuthology.orchestra.run.smithi183.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-10T19:19:56.431 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:19:56.453 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:19:56.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:56.453 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:19:56.465 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:19:56 smithi191 ceph-mon[32884]: osdmap e231: 8 total, 8 up, 6 in 2024-09-10T19:19:56.555 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:19:56.576 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:19:56.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:56.577 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:19:56.678 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:19:56.699 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:19:56.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:56.700 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:19:56.801 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:19:56.823 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:19:56.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:56.824 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:19:56.924 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:19:56.946 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:19:56.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:56.947 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:19:57.048 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:19:57.069 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:19:57.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:57.070 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:19:57.171 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:19:57.176 INFO:teuthology.orchestra.run.smithi191.stdout:{ 2024-09-10T19:19:57.176 INFO:teuthology.orchestra.run.smithi191.stdout: "ops": [], 2024-09-10T19:19:57.176 INFO:teuthology.orchestra.run.smithi191.stdout: "complaint_time": 30, 2024-09-10T19:19:57.176 INFO:teuthology.orchestra.run.smithi191.stdout: "num_blocked_ops": 0 2024-09-10T19:19:57.176 INFO:teuthology.orchestra.run.smithi191.stdout:} 2024-09-10T19:19:57.194 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:19:57.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:57.194 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:19:57.295 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:19:57.315 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:19:57.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:57.316 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:19:57.417 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:19:57.433 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:19:57 smithi191 ceph-mon[32884]: 6.1a scrub starts 2024-09-10T19:19:57.433 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:19:57 smithi191 ceph-mon[32884]: 6.1a scrub ok 2024-09-10T19:19:57.433 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:19:57 smithi191 ceph-mon[32884]: pgmap v437: 148 pgs: 148 active+clean; 582 KiB data, 2.3 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-09-10T19:19:57.433 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:19:57 smithi191 ceph-mon[32884]: 6.5 scrub starts 2024-09-10T19:19:57.433 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:19:57 smithi191 ceph-mon[32884]: 6.5 scrub ok 2024-09-10T19:19:57.433 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:19:57 smithi191 ceph-mon[32884]: from='client.? 172.21.15.183:0/3151263584' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T19:19:57.438 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:19:57.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:57.439 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:19:57.455 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:19:57 smithi183 ceph-mon[29736]: 6.1a scrub starts 2024-09-10T19:19:57.456 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:19:57 smithi183 ceph-mon[29736]: 6.1a scrub ok 2024-09-10T19:19:57.456 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:19:57 smithi183 ceph-mon[29736]: pgmap v437: 148 pgs: 148 active+clean; 582 KiB data, 2.3 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-09-10T19:19:57.456 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:19:57 smithi183 ceph-mon[29736]: 6.5 scrub starts 2024-09-10T19:19:57.456 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:19:57 smithi183 ceph-mon[29736]: 6.5 scrub ok 2024-09-10T19:19:57.456 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:19:57 smithi183 ceph-mon[29736]: from='client.? 172.21.15.183:0/3151263584' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T19:19:57.456 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:19:57 smithi183 ceph-mon[41562]: 6.1a scrub starts 2024-09-10T19:19:57.456 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:19:57 smithi183 ceph-mon[41562]: 6.1a scrub ok 2024-09-10T19:19:57.456 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:19:57 smithi183 ceph-mon[41562]: pgmap v437: 148 pgs: 148 active+clean; 582 KiB data, 2.3 GiB used, 534 GiB / 536 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-09-10T19:19:57.456 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:19:57 smithi183 ceph-mon[41562]: 6.5 scrub starts 2024-09-10T19:19:57.456 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:19:57 smithi183 ceph-mon[41562]: 6.5 scrub ok 2024-09-10T19:19:57.456 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:19:57 smithi183 ceph-mon[41562]: from='client.? 172.21.15.183:0/3151263584' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T19:19:57.540 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:19:57.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:57.562 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:19:57.562 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:19:57.663 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:19:57.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:57.686 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:19:57.686 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:19:57.715 INFO:journalctl@ceph.mgr.x.smithi191.stdout:Sep 10 19:19:57 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-x[34967]: ::ffff:172.21.15.191 - - [10/Sep/2024:19:19:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T19:19:57.726 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-10T19:19:57.787 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:19:57.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:57.810 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:19:57.811 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:19:57.911 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:19:57.933 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:19:57.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:57.934 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:19:58.035 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:19:58.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:58.057 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:19:58.057 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:19:58.075 DEBUG:teuthology.orchestra.run.smithi191:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph daemon osd.7 dump_historic_ops 2024-09-10T19:19:58.157 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:19:58.179 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:19:58.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:58.180 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:19:58.189 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:19:58.281 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:19:58.304 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:19:58.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:58.306 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:19:58.338 INFO:teuthology.orchestra.run.smithi191.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.b/config 2024-09-10T19:19:58.406 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:19:58.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:58.429 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:19:58.429 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:19:58.431 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:19:58 smithi183 ceph-mon[41562]: 4.1 scrub starts 2024-09-10T19:19:58.431 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:19:58 smithi183 ceph-mon[41562]: 4.1 scrub ok 2024-09-10T19:19:58.431 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:19:58 smithi183 ceph-mon[41562]: 4.b scrub starts 2024-09-10T19:19:58.431 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:19:58 smithi183 ceph-mon[41562]: 4.b scrub ok 2024-09-10T19:19:58.431 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:19:58 smithi183 ceph-mon[41562]: 2.a scrub starts 2024-09-10T19:19:58.431 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:19:58 smithi183 ceph-mon[41562]: 2.a scrub ok 2024-09-10T19:19:58.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:19:58 smithi183 ceph-mon[29736]: 4.1 scrub starts 2024-09-10T19:19:58.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:19:58 smithi183 ceph-mon[29736]: 4.1 scrub ok 2024-09-10T19:19:58.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:19:58 smithi183 ceph-mon[29736]: 4.b scrub starts 2024-09-10T19:19:58.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:19:58 smithi183 ceph-mon[29736]: 4.b scrub ok 2024-09-10T19:19:58.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:19:58 smithi183 ceph-mon[29736]: 2.a scrub starts 2024-09-10T19:19:58.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:19:58 smithi183 ceph-mon[29736]: 2.a scrub ok 2024-09-10T19:19:58.436 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:19:58 smithi191 ceph-mon[32884]: 4.1 scrub starts 2024-09-10T19:19:58.436 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:19:58 smithi191 ceph-mon[32884]: 4.1 scrub ok 2024-09-10T19:19:58.436 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:19:58 smithi191 ceph-mon[32884]: 4.b scrub starts 2024-09-10T19:19:58.436 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:19:58 smithi191 ceph-mon[32884]: 4.b scrub ok 2024-09-10T19:19:58.436 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:19:58 smithi191 ceph-mon[32884]: 2.a scrub starts 2024-09-10T19:19:58.436 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:19:58 smithi191 ceph-mon[32884]: 2.a scrub ok 2024-09-10T19:19:58.529 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:19:58.552 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:19:58.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:58.553 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:19:58.653 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:19:58.676 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:19:58.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:58.677 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:19:58.778 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:19:58.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:58.801 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:19:58.801 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:19:58.902 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:19:58.923 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:19:58.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:58.924 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:19:59.025 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:19:59.048 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:19:59.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:59.049 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:19:59.150 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:19:59.171 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:19:59.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:59.172 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:19:59.214 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-10T19:19:59.273 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:19:59.295 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:19:59.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:59.296 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:19:59.396 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:19:59.417 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:19:59.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:59.418 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:19:59.431 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:19:59 smithi183 ceph-mon[41562]: pgmap v438: 148 pgs: 148 active+clean; 582 KiB data, 2.3 GiB used, 534 GiB / 536 GiB avail; 853 B/s rd, 0 op/s 2024-09-10T19:19:59.431 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:19:59 smithi183 ceph-mon[41562]: 6.1d scrub starts 2024-09-10T19:19:59.431 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:19:59 smithi183 ceph-mon[41562]: 6.1d scrub ok 2024-09-10T19:19:59.431 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:19:59 smithi183 ceph-mon[41562]: from='mgr.14556 ' entity='mgr.y' 2024-09-10T19:19:59.431 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:19:59 smithi183 ceph-mon[41562]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:19:59.431 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:19:59 smithi183 ceph-mon[41562]: from='client.? 172.21.15.183:0/2184205940' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_2", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:19:59.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:19:59 smithi183 ceph-mon[29736]: pgmap v438: 148 pgs: 148 active+clean; 582 KiB data, 2.3 GiB used, 534 GiB / 536 GiB avail; 853 B/s rd, 0 op/s 2024-09-10T19:19:59.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:19:59 smithi183 ceph-mon[29736]: 6.1d scrub starts 2024-09-10T19:19:59.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:19:59 smithi183 ceph-mon[29736]: 6.1d scrub ok 2024-09-10T19:19:59.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:19:59 smithi183 ceph-mon[29736]: from='mgr.14556 ' entity='mgr.y' 2024-09-10T19:19:59.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:19:59 smithi183 ceph-mon[29736]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:19:59.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:19:59 smithi183 ceph-mon[29736]: from='client.? 172.21.15.183:0/2184205940' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_2", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:19:59.465 INFO:journalctl@ceph.iscsi.iscsi.a.smithi191.stdout:Sep 10 19:19:59 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-iscsi-iscsi-a[61816]: debug there is no tcmu-runner data available 2024-09-10T19:19:59.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:19:59 smithi191 ceph-mon[32884]: pgmap v438: 148 pgs: 148 active+clean; 582 KiB data, 2.3 GiB used, 534 GiB / 536 GiB avail; 853 B/s rd, 0 op/s 2024-09-10T19:19:59.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:19:59 smithi191 ceph-mon[32884]: 6.1d scrub starts 2024-09-10T19:19:59.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:19:59 smithi191 ceph-mon[32884]: 6.1d scrub ok 2024-09-10T19:19:59.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:19:59 smithi191 ceph-mon[32884]: from='mgr.14556 ' entity='mgr.y' 2024-09-10T19:19:59.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:19:59 smithi191 ceph-mon[32884]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:19:59.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:19:59 smithi191 ceph-mon[32884]: from='client.? 172.21.15.183:0/2184205940' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_2", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:19:59.519 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:19:59.540 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:19:59.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:59.542 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:19:59.642 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:19:59.665 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:19:59.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:59.666 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:19:59.670 INFO:teuthology.orchestra.run.smithi183.stderr:enabled application 'rados' on pool 'unique_pool_2' 2024-09-10T19:19:59.766 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:19:59.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:59.789 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:19:59.790 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:19:59.890 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:19:59.912 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:19:59.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:19:59.913 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:20:00.014 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:20:00.036 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:20:00.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:20:00.038 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:20:00.109 INFO:teuthology.orchestra.run.smithi191.stdout:{ 2024-09-10T19:20:00.109 INFO:teuthology.orchestra.run.smithi191.stdout: "size": 20, 2024-09-10T19:20:00.109 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 600, 2024-09-10T19:20:00.109 INFO:teuthology.orchestra.run.smithi191.stdout: "ops": [ 2024-09-10T19:20:00.109 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:20:00.109 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.15051.0:59075 8.3 8.aac477f3 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e165)", 2024-09-10T19:20:00.109 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-10T19:16:55.018030+0000", 2024-09-10T19:20:00.109 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 185.06973837699999, 2024-09-10T19:20:00.109 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.97022024399999995, 2024-09-10T19:20:00.109 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-10T19:20:00.109 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "reached pg", 2024-09-10T19:20:00.110 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-10T19:20:00.110 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.15051", 2024-09-10T19:20:00.110 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.183:0/3632028672", 2024-09-10T19:20:00.110 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 59075 2024-09-10T19:20:00.110 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:20:00.110 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-10T19:20:00.110 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:20:00.110 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-10T19:20:00.110 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:16:55.018030+0000", 2024-09-10T19:20:00.110 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:20:00.110 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:20:00.110 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:20:00.111 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-10T19:20:00.111 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:16:55.018030+0000", 2024-09-10T19:20:00.111 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:20:00.111 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:20:00.111 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:20:00.111 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-10T19:20:00.111 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:16:55.018029+0000", 2024-09-10T19:20:00.111 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999981 2024-09-10T19:20:00.111 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:20:00.111 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:20:00.111 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-10T19:20:00.111 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:16:55.018060+0000", 2024-09-10T19:20:00.111 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.1653999999999999e-05 2024-09-10T19:20:00.112 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:20:00.112 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:20:00.112 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-10T19:20:00.112 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:16:55.018072+0000", 2024-09-10T19:20:00.112 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.1960999999999999e-05 2024-09-10T19:20:00.112 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:20:00.112 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:20:00.112 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-10T19:20:00.112 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:16:55.018080+0000", 2024-09-10T19:20:00.112 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.909e-06 2024-09-10T19:20:00.112 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:20:00.112 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:20:00.112 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:20:00.112 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:16:55.018130+0000", 2024-09-10T19:20:00.113 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.0238000000000002e-05 2024-09-10T19:20:00.113 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:20:00.113 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:20:00.113 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for peered", 2024-09-10T19:20:00.113 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:16:55.018136+0000", 2024-09-10T19:20:00.113 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.8039999999999998e-06 2024-09-10T19:20:00.113 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:20:00.113 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:20:00.113 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:20:00.113 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:16:55.988241+0000", 2024-09-10T19:20:00.113 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.97010456899999997 2024-09-10T19:20:00.113 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:20:00.113 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:20:00.114 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-10T19:20:00.114 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:16:55.988251+0000", 2024-09-10T19:20:00.114 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.9059999999999997e-06 2024-09-10T19:20:00.114 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:20:00.114 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-10T19:20:00.114 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:20:00.114 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:20:00.114 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:20:00.114 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.15051.0:59116 8.3 8.1bacc173 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e165)", 2024-09-10T19:20:00.114 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-10T19:16:55.043251+0000", 2024-09-10T19:20:00.114 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 185.04451775000001, 2024-09-10T19:20:00.114 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.945029915, 2024-09-10T19:20:00.114 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-10T19:20:00.114 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "reached pg", 2024-09-10T19:20:00.115 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-10T19:20:00.115 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.15051", 2024-09-10T19:20:00.115 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.183:0/3632028672", 2024-09-10T19:20:00.115 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 59116 2024-09-10T19:20:00.115 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:20:00.115 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-10T19:20:00.115 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:20:00.115 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-10T19:20:00.115 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:16:55.043251+0000", 2024-09-10T19:20:00.115 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:20:00.115 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:20:00.115 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:20:00.115 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-10T19:20:00.116 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:16:55.043251+0000", 2024-09-10T19:20:00.116 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:20:00.116 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:20:00.116 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:20:00.116 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-10T19:20:00.116 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:16:55.043249+0000", 2024-09-10T19:20:00.116 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999981 2024-09-10T19:20:00.116 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:20:00.116 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:20:00.116 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-10T19:20:00.116 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:16:55.043361+0000", 2024-09-10T19:20:00.116 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000112064 2024-09-10T19:20:00.116 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:20:00.116 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:20:00.117 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-10T19:20:00.117 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:16:55.043377+0000", 2024-09-10T19:20:00.117 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.6084000000000001e-05 2024-09-10T19:20:00.117 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:20:00.117 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:20:00.117 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-10T19:20:00.117 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:16:55.043383+0000", 2024-09-10T19:20:00.117 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.066e-06 2024-09-10T19:20:00.117 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:20:00.117 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:20:00.117 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:20:00.117 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:16:55.044608+0000", 2024-09-10T19:20:00.117 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.001224859 2024-09-10T19:20:00.117 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:20:00.118 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:20:00.118 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for peered", 2024-09-10T19:20:00.118 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:16:55.044615+0000", 2024-09-10T19:20:00.118 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.5540000000000003e-06 2024-09-10T19:20:00.118 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:20:00.118 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:20:00.118 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:20:00.118 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:16:55.988274+0000", 2024-09-10T19:20:00.118 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.94365957899999997 2024-09-10T19:20:00.118 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:20:00.118 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:20:00.118 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-10T19:20:00.118 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:16:55.988281+0000", 2024-09-10T19:20:00.118 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.8480000000000003e-06 2024-09-10T19:20:00.119 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:20:00.119 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-10T19:20:00.119 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:20:00.119 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:20:00.119 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:20:00.119 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.15051.0:59117 8.3 8.23748543 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e165)", 2024-09-10T19:20:00.119 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-10T19:16:55.044388+0000", 2024-09-10T19:20:00.119 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 185.043380816, 2024-09-10T19:20:00.119 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.943938842, 2024-09-10T19:20:00.119 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-10T19:20:00.119 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "reached pg", 2024-09-10T19:20:00.119 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-10T19:20:00.119 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.15051", 2024-09-10T19:20:00.119 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.183:0/3632028672", 2024-09-10T19:20:00.120 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 59117 2024-09-10T19:20:00.120 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:20:00.120 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-10T19:20:00.120 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:20:00.120 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-10T19:20:00.120 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:16:55.044388+0000", 2024-09-10T19:20:00.120 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:20:00.120 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:20:00.120 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:20:00.121 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-10T19:20:00.121 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:16:55.044388+0000", 2024-09-10T19:20:00.121 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:20:00.121 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:20:00.121 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:20:00.121 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-10T19:20:00.121 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:16:55.044387+0000", 2024-09-10T19:20:00.121 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999986 2024-09-10T19:20:00.121 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:20:00.121 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:20:00.122 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-10T19:20:00.122 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:16:55.044436+0000", 2024-09-10T19:20:00.122 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.9054000000000002e-05 2024-09-10T19:20:00.122 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:20:00.122 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:20:00.122 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-10T19:20:00.122 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:16:55.044445+0000", 2024-09-10T19:20:00.122 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.5200000000000003e-06 2024-09-10T19:20:00.122 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:20:00.122 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:20:00.122 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-10T19:20:00.122 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:16:55.044451+0000", 2024-09-10T19:20:00.122 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.5709999999999997e-06 2024-09-10T19:20:00.122 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:20:00.123 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:20:00.123 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:20:00.123 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:16:55.044758+0000", 2024-09-10T19:20:00.123 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00030680600000000002 2024-09-10T19:20:00.123 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:20:00.123 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:20:00.123 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for peered", 2024-09-10T19:20:00.123 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:16:55.044763+0000", 2024-09-10T19:20:00.123 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.0980000000000001e-06 2024-09-10T19:20:00.123 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:20:00.123 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:20:00.123 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:20:00.123 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:16:55.988320+0000", 2024-09-10T19:20:00.123 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.94355762899999995 2024-09-10T19:20:00.124 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:20:00.124 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:20:00.124 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-10T19:20:00.124 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:16:55.988327+0000", 2024-09-10T19:20:00.124 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.3799999999999999e-06 2024-09-10T19:20:00.124 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:20:00.124 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-10T19:20:00.124 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:20:00.124 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:20:00.124 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:20:00.124 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.15051.0:93082 8.3 8:c5abc7f7:::benchmark_data_smithi183_114924_object93081:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e178)", 2024-09-10T19:20:00.124 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-10T19:17:26.939355+0000", 2024-09-10T19:20:00.124 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 153.14841358699999, 2024-09-10T19:20:00.125 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.047200065, 2024-09-10T19:20:00.125 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-10T19:20:00.125 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "reached pg", 2024-09-10T19:20:00.125 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-10T19:20:00.125 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.15051", 2024-09-10T19:20:00.125 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.183:0/3632028672", 2024-09-10T19:20:00.125 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 93082 2024-09-10T19:20:00.125 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:20:00.125 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-10T19:20:00.126 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:20:00.126 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-10T19:20:00.126 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:26.939355+0000", 2024-09-10T19:20:00.126 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:20:00.126 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:20:00.126 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:20:00.126 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-10T19:20:00.126 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:26.939355+0000", 2024-09-10T19:20:00.126 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:20:00.126 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:20:00.126 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:20:00.126 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-10T19:20:00.126 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:26.939353+0000", 2024-09-10T19:20:00.127 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999976 2024-09-10T19:20:00.127 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:20:00.127 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:20:00.127 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-10T19:20:00.127 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:26.939539+0000", 2024-09-10T19:20:00.127 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00018558299999999999 2024-09-10T19:20:00.127 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:20:00.127 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:20:00.127 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-10T19:20:00.127 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:26.939567+0000", 2024-09-10T19:20:00.127 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.8345000000000001e-05 2024-09-10T19:20:00.127 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:20:00.127 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:20:00.128 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-10T19:20:00.128 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:26.939576+0000", 2024-09-10T19:20:00.128 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.0389999999999993e-06 2024-09-10T19:20:00.128 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:20:00.128 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:20:00.128 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:20:00.128 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:26.939651+0000", 2024-09-10T19:20:00.128 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.5278999999999995e-05 2024-09-10T19:20:00.128 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:20:00.128 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:20:00.128 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op must wait for map", 2024-09-10T19:20:00.128 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:26.939662+0000", 2024-09-10T19:20:00.128 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.0789e-05 2024-09-10T19:20:00.129 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:20:00.129 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:20:00.129 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:20:00.129 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:26.944034+0000", 2024-09-10T19:20:00.129 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0043717110000000003 2024-09-10T19:20:00.129 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:20:00.129 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:20:00.129 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for peered", 2024-09-10T19:20:00.129 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:26.944047+0000", 2024-09-10T19:20:00.129 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.288e-05 2024-09-10T19:20:00.129 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:20:00.129 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:20:00.129 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:20:00.129 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:27.985144+0000", 2024-09-10T19:20:00.130 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.0410975929999999 2024-09-10T19:20:00.130 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:20:00.130 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:20:00.130 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-10T19:20:00.130 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:27.986555+0000", 2024-09-10T19:20:00.130 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0014110419999999999 2024-09-10T19:20:00.130 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:20:00.130 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-10T19:20:00.130 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:20:00.130 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:20:00.131 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:20:00.131 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.15051.0:93091 8.3 8:c1c9fa55:::benchmark_data_smithi183_114924_object93090:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e178)", 2024-09-10T19:20:00.131 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-10T19:17:26.939596+0000", 2024-09-10T19:20:00.131 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 153.148172848, 2024-09-10T19:20:00.131 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.4256138250000001, 2024-09-10T19:20:00.131 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-10T19:20:00.131 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T19:20:00.131 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-10T19:20:00.131 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.15051", 2024-09-10T19:20:00.131 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.183:0/3632028672", 2024-09-10T19:20:00.131 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 93091 2024-09-10T19:20:00.131 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:20:00.131 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-10T19:20:00.132 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:20:00.132 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-10T19:20:00.132 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:26.939596+0000", 2024-09-10T19:20:00.132 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:20:00.132 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:20:00.132 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:20:00.132 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-10T19:20:00.132 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:26.939596+0000", 2024-09-10T19:20:00.132 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:20:00.132 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:20:00.132 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:20:00.132 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-10T19:20:00.132 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:26.939595+0000", 2024-09-10T19:20:00.132 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.999999 2024-09-10T19:20:00.133 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:20:00.133 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:20:00.133 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-10T19:20:00.133 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:26.939637+0000", 2024-09-10T19:20:00.133 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.2364999999999998e-05 2024-09-10T19:20:00.133 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:20:00.133 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:20:00.133 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-10T19:20:00.133 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:26.939647+0000", 2024-09-10T19:20:00.133 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.8519999999999999e-06 2024-09-10T19:20:00.133 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:20:00.133 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:20:00.134 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-10T19:20:00.134 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:26.939653+0000", 2024-09-10T19:20:00.134 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.8649999999999998e-06 2024-09-10T19:20:00.134 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:20:00.134 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:20:00.134 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:20:00.134 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:26.939821+0000", 2024-09-10T19:20:00.134 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00016759300000000001 2024-09-10T19:20:00.134 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:20:00.134 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:20:00.134 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting_for_map not empty", 2024-09-10T19:20:00.134 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:26.939828+0000", 2024-09-10T19:20:00.134 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.6389999999999992e-06 2024-09-10T19:20:00.134 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:20:00.134 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:20:00.135 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:20:00.135 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:26.944068+0000", 2024-09-10T19:20:00.135 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0042391219999999997 2024-09-10T19:20:00.135 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:20:00.135 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:20:00.135 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for peered", 2024-09-10T19:20:00.135 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:26.944073+0000", 2024-09-10T19:20:00.135 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.0780000000000003e-06 2024-09-10T19:20:00.135 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:20:00.135 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:20:00.135 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:20:00.135 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:27.986811+0000", 2024-09-10T19:20:00.136 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.042738487 2024-09-10T19:20:00.136 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:20:00.136 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:20:00.136 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-10T19:20:00.136 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:27.989675+0000", 2024-09-10T19:20:00.136 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0028639149999999999 2024-09-10T19:20:00.136 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:20:00.136 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:20:00.136 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 3,5", 2024-09-10T19:20:00.136 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:27.994560+0000", 2024-09-10T19:20:00.136 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0048853400000000002 2024-09-10T19:20:00.136 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:20:00.136 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:20:00.136 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-09-10T19:20:00.136 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:28.015341+0000", 2024-09-10T19:20:00.137 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.020780532000000001 2024-09-10T19:20:00.137 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:20:00.137 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:20:00.137 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:20:00.137 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:28.359525+0000", 2024-09-10T19:20:00.137 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.34418381399999998 2024-09-10T19:20:00.137 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:20:00.137 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:20:00.137 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:20:00.137 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:28.360180+0000", 2024-09-10T19:20:00.137 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00065538 2024-09-10T19:20:00.137 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:20:00.137 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:20:00.138 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-09-10T19:20:00.138 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:28.361229+0000", 2024-09-10T19:20:00.138 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.001048998 2024-09-10T19:20:00.138 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:20:00.138 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:20:00.138 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-10T19:20:00.138 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:28.365210+0000", 2024-09-10T19:20:00.138 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0039807380000000002 2024-09-10T19:20:00.138 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:20:00.138 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-10T19:20:00.138 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:20:00.138 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:20:00.138 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:20:00.139 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.15051.0:93120 8.3 8:c0ebb47a:::benchmark_data_smithi183_114924_object93119:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e178)", 2024-09-10T19:20:00.139 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-10T19:17:26.991786+0000", 2024-09-10T19:20:00.139 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 153.09598293799999, 2024-09-10T19:20:00.139 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.3798205450000001, 2024-09-10T19:20:00.139 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-10T19:20:00.139 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T19:20:00.139 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-10T19:20:00.139 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.15051", 2024-09-10T19:20:00.139 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.183:0/3632028672", 2024-09-10T19:20:00.139 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 93120 2024-09-10T19:20:00.139 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:20:00.139 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-10T19:20:00.139 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:20:00.139 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-10T19:20:00.140 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:26.991786+0000", 2024-09-10T19:20:00.140 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:20:00.140 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:20:00.140 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:20:00.140 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-10T19:20:00.140 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:26.991786+0000", 2024-09-10T19:20:00.140 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:20:00.140 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:20:00.140 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:20:00.140 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-10T19:20:00.140 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:26.991784+0000", 2024-09-10T19:20:00.140 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999986 2024-09-10T19:20:00.140 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:20:00.141 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:20:00.141 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-10T19:20:00.141 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:26.991845+0000", 2024-09-10T19:20:00.141 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.0312999999999997e-05 2024-09-10T19:20:00.141 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:20:00.141 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:20:00.141 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-10T19:20:00.141 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:26.991860+0000", 2024-09-10T19:20:00.141 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.5639999999999999e-05 2024-09-10T19:20:00.141 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:20:00.141 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:20:00.141 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-10T19:20:00.141 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:26.991867+0000", 2024-09-10T19:20:00.142 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.5400000000000001e-06 2024-09-10T19:20:00.142 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:20:00.142 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:20:00.142 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:20:00.142 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:26.991919+0000", 2024-09-10T19:20:00.142 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.2629e-05 2024-09-10T19:20:00.142 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:20:00.142 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:20:00.142 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for peered", 2024-09-10T19:20:00.142 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:26.991930+0000", 2024-09-10T19:20:00.142 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.0207000000000001e-05 2024-09-10T19:20:00.142 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:20:00.142 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:20:00.142 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:20:00.143 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:27.997258+0000", 2024-09-10T19:20:00.143 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.0053281890000001 2024-09-10T19:20:00.143 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:20:00.143 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:20:00.143 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-10T19:20:00.143 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:27.999670+0000", 2024-09-10T19:20:00.143 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0024118439999999998 2024-09-10T19:20:00.143 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:20:00.143 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:20:00.143 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 3,5", 2024-09-10T19:20:00.143 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:28.003897+0000", 2024-09-10T19:20:00.143 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0042272050000000004 2024-09-10T19:20:00.143 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:20:00.143 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:20:00.143 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-09-10T19:20:00.144 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:28.033905+0000", 2024-09-10T19:20:00.144 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.030007741000000001 2024-09-10T19:20:00.144 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:20:00.144 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:20:00.144 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:20:00.144 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:28.365817+0000", 2024-09-10T19:20:00.144 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.33191224400000002 2024-09-10T19:20:00.144 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:20:00.144 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:20:00.144 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:20:00.144 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:28.366555+0000", 2024-09-10T19:20:00.144 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00073775299999999995 2024-09-10T19:20:00.144 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:20:00.144 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:20:00.144 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-09-10T19:20:00.145 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:28.367621+0000", 2024-09-10T19:20:00.145 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0010666180000000001 2024-09-10T19:20:00.145 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:20:00.145 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:20:00.145 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-10T19:20:00.145 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:28.371606+0000", 2024-09-10T19:20:00.145 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0039852769999999997 2024-09-10T19:20:00.145 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:20:00.145 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-10T19:20:00.145 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:20:00.145 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:20:00.145 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:20:00.145 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.15051.0:93129 8.3 8:c691b51d:::benchmark_data_smithi183_114924_object93128:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e178)", 2024-09-10T19:20:00.145 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-10T19:17:27.001902+0000", 2024-09-10T19:20:00.146 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 153.08586733199999, 2024-09-10T19:20:00.146 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.3757423389999999, 2024-09-10T19:20:00.146 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-10T19:20:00.146 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T19:20:00.146 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-10T19:20:00.146 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.15051", 2024-09-10T19:20:00.146 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.183:0/3632028672", 2024-09-10T19:20:00.146 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 93129 2024-09-10T19:20:00.146 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:20:00.146 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-10T19:20:00.146 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:20:00.146 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-10T19:20:00.146 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:27.001902+0000", 2024-09-10T19:20:00.146 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:20:00.147 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:20:00.147 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:20:00.147 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-10T19:20:00.147 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:27.001902+0000", 2024-09-10T19:20:00.147 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:20:00.147 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:20:00.147 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:20:00.147 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-10T19:20:00.147 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:27.001900+0000", 2024-09-10T19:20:00.147 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999986 2024-09-10T19:20:00.147 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:20:00.147 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:20:00.147 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-10T19:20:00.148 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:27.001937+0000", 2024-09-10T19:20:00.148 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.6804000000000002e-05 2024-09-10T19:20:00.148 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:20:00.148 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:20:00.148 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-10T19:20:00.148 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:27.001948+0000", 2024-09-10T19:20:00.148 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.1569999999999999e-05 2024-09-10T19:20:00.148 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:20:00.148 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:20:00.148 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-10T19:20:00.148 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:27.001954+0000", 2024-09-10T19:20:00.148 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.5740000000000003e-06 2024-09-10T19:20:00.148 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:20:00.148 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:20:00.149 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:20:00.149 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:27.002077+0000", 2024-09-10T19:20:00.149 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00012273199999999999 2024-09-10T19:20:00.149 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:20:00.149 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:20:00.149 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for peered", 2024-09-10T19:20:00.149 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:27.002086+0000", 2024-09-10T19:20:00.149 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.7499999999999998e-06 2024-09-10T19:20:00.149 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:20:00.149 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:20:00.149 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:20:00.149 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:28.006349+0000", 2024-09-10T19:20:00.149 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.0042625359999999 2024-09-10T19:20:00.150 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:20:00.150 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:20:00.150 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-10T19:20:00.150 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:28.008726+0000", 2024-09-10T19:20:00.150 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0023774859999999998 2024-09-10T19:20:00.150 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:20:00.150 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:20:00.150 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 3,5", 2024-09-10T19:20:00.150 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:28.012885+0000", 2024-09-10T19:20:00.150 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0041590309999999997 2024-09-10T19:20:00.150 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:20:00.150 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:20:00.150 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-09-10T19:20:00.150 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:28.034107+0000", 2024-09-10T19:20:00.151 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.021221998999999998 2024-09-10T19:20:00.151 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:20:00.151 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:20:00.151 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:20:00.151 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:28.372218+0000", 2024-09-10T19:20:00.151 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.338111045 2024-09-10T19:20:00.151 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:20:00.151 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:20:00.151 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:20:00.151 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:28.372817+0000", 2024-09-10T19:20:00.151 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00059862800000000003 2024-09-10T19:20:00.151 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:20:00.151 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:20:00.151 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-09-10T19:20:00.151 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:28.373805+0000", 2024-09-10T19:20:00.152 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00098759499999999997 2024-09-10T19:20:00.152 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:20:00.152 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:20:00.152 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-10T19:20:00.152 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:28.377644+0000", 2024-09-10T19:20:00.152 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0038391969999999999 2024-09-10T19:20:00.152 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:20:00.152 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-10T19:20:00.152 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:20:00.152 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:20:00.152 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:20:00.152 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.15051.0:93169 8.3 8:c5187144:::benchmark_data_smithi183_114924_object93168:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e178)", 2024-09-10T19:20:00.152 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-10T19:17:27.028969+0000", 2024-09-10T19:20:00.153 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 153.05879966800001, 2024-09-10T19:20:00.153 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.354956426, 2024-09-10T19:20:00.153 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-10T19:20:00.153 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T19:20:00.153 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-10T19:20:00.153 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.15051", 2024-09-10T19:20:00.153 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.183:0/3632028672", 2024-09-10T19:20:00.153 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 93169 2024-09-10T19:20:00.153 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:20:00.153 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-10T19:20:00.153 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:20:00.153 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-10T19:20:00.153 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:27.028969+0000", 2024-09-10T19:20:00.154 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:20:00.154 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:20:00.154 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:20:00.154 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-10T19:20:00.154 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:27.028969+0000", 2024-09-10T19:20:00.154 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:20:00.154 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:20:00.154 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:20:00.154 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-10T19:20:00.154 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:27.028967+0000", 2024-09-10T19:20:00.154 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999981 2024-09-10T19:20:00.154 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:20:00.154 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:20:00.155 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-10T19:20:00.155 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:27.029032+0000", 2024-09-10T19:20:00.155 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.4704999999999998e-05 2024-09-10T19:20:00.155 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:20:00.155 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:20:00.155 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-10T19:20:00.155 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:27.029046+0000", 2024-09-10T19:20:00.155 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.3458999999999999e-05 2024-09-10T19:20:00.155 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:20:00.155 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:20:00.155 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-10T19:20:00.155 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:27.029052+0000", 2024-09-10T19:20:00.155 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.8669999999999999e-06 2024-09-10T19:20:00.155 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:20:00.156 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:20:00.156 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:20:00.156 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:27.029118+0000", 2024-09-10T19:20:00.156 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.6771000000000002e-05 2024-09-10T19:20:00.156 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:20:00.156 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:20:00.156 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for peered", 2024-09-10T19:20:00.156 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:27.029136+0000", 2024-09-10T19:20:00.156 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.7257000000000001e-05 2024-09-10T19:20:00.156 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:20:00.156 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:20:00.156 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:20:00.157 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:28.015654+0000", 2024-09-10T19:20:00.157 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.98651808399999996 2024-09-10T19:20:00.157 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:20:00.157 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:20:00.157 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-10T19:20:00.157 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:28.018045+0000", 2024-09-10T19:20:00.157 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0023913710000000002 2024-09-10T19:20:00.157 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:20:00.157 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:20:00.157 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 3,5", 2024-09-10T19:20:00.157 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:28.022199+0000", 2024-09-10T19:20:00.157 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0041539519999999998 2024-09-10T19:20:00.157 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:20:00.158 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:20:00.158 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-09-10T19:20:00.158 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:28.052681+0000", 2024-09-10T19:20:00.158 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.030481590999999999 2024-09-10T19:20:00.158 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:20:00.158 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:20:00.158 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:20:00.158 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:28.378251+0000", 2024-09-10T19:20:00.158 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.32557088899999997 2024-09-10T19:20:00.158 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:20:00.158 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:20:00.158 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:20:00.158 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:28.378848+0000", 2024-09-10T19:20:00.158 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00059637000000000004 2024-09-10T19:20:00.159 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:20:00.159 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:20:00.159 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-09-10T19:20:00.159 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:28.379835+0000", 2024-09-10T19:20:00.159 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00098689999999999997 2024-09-10T19:20:00.159 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:20:00.159 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:20:00.159 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-10T19:20:00.159 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:28.383926+0000", 2024-09-10T19:20:00.159 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0040909129999999998 2024-09-10T19:20:00.159 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:20:00.159 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-10T19:20:00.159 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:20:00.159 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:20:00.159 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:20:00.160 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.15051.0:93200 8.3 8:c1d83c5e:::benchmark_data_smithi183_114924_object93199:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e178)", 2024-09-10T19:20:00.160 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-10T19:17:27.045967+0000", 2024-09-10T19:20:00.160 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 153.04180187599999, 2024-09-10T19:20:00.160 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.3449605410000001, 2024-09-10T19:20:00.160 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-10T19:20:00.160 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T19:20:00.160 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-10T19:20:00.160 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.15051", 2024-09-10T19:20:00.160 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.183:0/3632028672", 2024-09-10T19:20:00.160 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 93200 2024-09-10T19:20:00.160 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:20:00.160 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-10T19:20:00.160 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:20:00.161 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-10T19:20:00.161 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:27.045967+0000", 2024-09-10T19:20:00.161 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:20:00.161 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:20:00.161 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:20:00.161 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-10T19:20:00.161 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:27.045967+0000", 2024-09-10T19:20:00.161 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:20:00.161 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:20:00.161 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:20:00.161 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-10T19:20:00.161 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:27.045965+0000", 2024-09-10T19:20:00.161 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999981 2024-09-10T19:20:00.161 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:20:00.162 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:20:00.162 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-10T19:20:00.162 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:27.046015+0000", 2024-09-10T19:20:00.162 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.0099000000000003e-05 2024-09-10T19:20:00.162 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:20:00.162 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:20:00.162 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-10T19:20:00.162 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:27.046030+0000", 2024-09-10T19:20:00.162 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.4493e-05 2024-09-10T19:20:00.162 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:20:00.162 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:20:00.162 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-10T19:20:00.162 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:27.046035+0000", 2024-09-10T19:20:00.162 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.5999999999999997e-06 2024-09-10T19:20:00.163 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:20:00.163 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:20:00.163 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:20:00.163 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:27.046093+0000", 2024-09-10T19:20:00.163 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.7948999999999997e-05 2024-09-10T19:20:00.163 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:20:00.163 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:20:00.163 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for peered", 2024-09-10T19:20:00.163 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:27.046103+0000", 2024-09-10T19:20:00.163 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.5580000000000006e-06 2024-09-10T19:20:00.163 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:20:00.163 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:20:00.163 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:20:00.163 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:28.024643+0000", 2024-09-10T19:20:00.164 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.97854046100000003 2024-09-10T19:20:00.164 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:20:00.164 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:20:00.164 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-10T19:20:00.164 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:28.027144+0000", 2024-09-10T19:20:00.164 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00250037 2024-09-10T19:20:00.164 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:20:00.164 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:20:00.164 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 3,5", 2024-09-10T19:20:00.164 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:28.031418+0000", 2024-09-10T19:20:00.164 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0042745680000000003 2024-09-10T19:20:00.164 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:20:00.164 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:20:00.165 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-09-10T19:20:00.165 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:28.052882+0000", 2024-09-10T19:20:00.165 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.021464051000000001 2024-09-10T19:20:00.165 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:20:00.165 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:20:00.165 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:20:00.165 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:28.384774+0000", 2024-09-10T19:20:00.165 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.331892038 2024-09-10T19:20:00.165 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:20:00.165 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:20:00.165 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:20:00.165 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:28.385609+0000", 2024-09-10T19:20:00.165 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00083466900000000001 2024-09-10T19:20:00.166 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:20:00.166 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:20:00.166 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-09-10T19:20:00.166 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:28.386728+0000", 2024-09-10T19:20:00.166 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0011185349999999999 2024-09-10T19:20:00.166 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:20:00.166 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:20:00.166 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-10T19:20:00.166 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:28.390928+0000", 2024-09-10T19:20:00.166 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0041999569999999998 2024-09-10T19:20:00.166 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:20:00.166 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-10T19:20:00.166 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:20:00.166 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:20:00.167 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:20:00.167 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.15051.0:93226 8.3 8:c25b1055:::benchmark_data_smithi183_114924_object93225:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e178)", 2024-09-10T19:20:00.167 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-10T19:17:27.062335+0000", 2024-09-10T19:20:00.167 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 153.025433483, 2024-09-10T19:20:00.167 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.335070134, 2024-09-10T19:20:00.167 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-10T19:20:00.167 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T19:20:00.167 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-10T19:20:00.167 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.15051", 2024-09-10T19:20:00.167 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.183:0/3632028672", 2024-09-10T19:20:00.167 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 93226 2024-09-10T19:20:00.167 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:20:00.167 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-10T19:20:00.167 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:20:00.168 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-10T19:20:00.168 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:27.062335+0000", 2024-09-10T19:20:00.168 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:20:00.168 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:20:00.168 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:20:00.168 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-10T19:20:00.168 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:27.062335+0000", 2024-09-10T19:20:00.168 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:20:00.168 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:20:00.168 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:20:00.168 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-10T19:20:00.168 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:27.062334+0000", 2024-09-10T19:20:00.168 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999981 2024-09-10T19:20:00.168 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:20:00.168 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:20:00.169 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-10T19:20:00.169 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:27.062400+0000", 2024-09-10T19:20:00.169 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.5959999999999999e-05 2024-09-10T19:20:00.169 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:20:00.169 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:20:00.169 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-10T19:20:00.169 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:27.062416+0000", 2024-09-10T19:20:00.169 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.6722999999999999e-05 2024-09-10T19:20:00.169 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:20:00.169 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:20:00.169 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-10T19:20:00.169 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:27.062423+0000", 2024-09-10T19:20:00.169 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.8770000000000004e-06 2024-09-10T19:20:00.170 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:20:00.170 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:20:00.170 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:20:00.170 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:27.064828+0000", 2024-09-10T19:20:00.170 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0024045 2024-09-10T19:20:00.170 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:20:00.170 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:20:00.170 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for peered", 2024-09-10T19:20:00.170 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:27.064847+0000", 2024-09-10T19:20:00.170 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.9548999999999999e-05 2024-09-10T19:20:00.170 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:20:00.170 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:20:00.170 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:20:00.170 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:28.034430+0000", 2024-09-10T19:20:00.171 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.96958232899999997 2024-09-10T19:20:00.171 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:20:00.171 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:20:00.171 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-10T19:20:00.171 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:28.036844+0000", 2024-09-10T19:20:00.171 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.002414518 2024-09-10T19:20:00.171 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:20:00.171 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:20:00.171 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 3,5", 2024-09-10T19:20:00.171 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:28.041056+0000", 2024-09-10T19:20:00.171 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0042121169999999996 2024-09-10T19:20:00.171 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:20:00.171 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:20:00.171 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-09-10T19:20:00.171 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:28.073275+0000", 2024-09-10T19:20:00.172 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.032218546000000001 2024-09-10T19:20:00.172 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:20:00.172 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:20:00.172 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:20:00.172 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:28.391594+0000", 2024-09-10T19:20:00.172 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.31831964600000001 2024-09-10T19:20:00.172 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:20:00.172 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:20:00.172 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:20:00.172 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:28.392240+0000", 2024-09-10T19:20:00.172 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00064562500000000004 2024-09-10T19:20:00.172 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:20:00.173 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:20:00.173 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-09-10T19:20:00.173 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:28.393278+0000", 2024-09-10T19:20:00.173 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.001037804 2024-09-10T19:20:00.173 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:20:00.173 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:20:00.173 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-10T19:20:00.173 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:28.397406+0000", 2024-09-10T19:20:00.173 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.004127627 2024-09-10T19:20:00.173 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:20:00.173 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-10T19:20:00.173 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:20:00.173 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:20:00.173 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:20:00.174 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.15051.0:93261 8.3 8:c5a97e06:::benchmark_data_smithi183_114924_object93260:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e178)", 2024-09-10T19:20:00.174 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-10T19:17:27.112265+0000", 2024-09-10T19:20:00.174 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 152.97550344300001, 2024-09-10T19:20:00.174 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.2917055470000001, 2024-09-10T19:20:00.174 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-10T19:20:00.174 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T19:20:00.174 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-10T19:20:00.174 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.15051", 2024-09-10T19:20:00.174 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.183:0/3632028672", 2024-09-10T19:20:00.174 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 93261 2024-09-10T19:20:00.174 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:20:00.174 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-10T19:20:00.174 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:20:00.175 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-10T19:20:00.175 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:27.112265+0000", 2024-09-10T19:20:00.175 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:20:00.175 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:20:00.175 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:20:00.175 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-10T19:20:00.175 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:27.112265+0000", 2024-09-10T19:20:00.175 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:20:00.175 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:20:00.175 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:20:00.175 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-10T19:20:00.175 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:27.112263+0000", 2024-09-10T19:20:00.175 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999981 2024-09-10T19:20:00.175 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:20:00.175 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:20:00.176 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-10T19:20:00.176 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:27.112309+0000", 2024-09-10T19:20:00.176 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.5238999999999998e-05 2024-09-10T19:20:00.176 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:20:00.176 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:20:00.176 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-10T19:20:00.176 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:27.112323+0000", 2024-09-10T19:20:00.176 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.4284000000000001e-05 2024-09-10T19:20:00.176 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:20:00.176 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:20:00.176 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-10T19:20:00.176 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:27.112329+0000", 2024-09-10T19:20:00.176 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.1700000000000002e-06 2024-09-10T19:20:00.176 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:20:00.177 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:20:00.177 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:20:00.177 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:27.112425+0000", 2024-09-10T19:20:00.177 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.5568000000000004e-05 2024-09-10T19:20:00.177 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:20:00.177 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:20:00.177 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for peered", 2024-09-10T19:20:00.177 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:27.112432+0000", 2024-09-10T19:20:00.177 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.4889999999999998e-06 2024-09-10T19:20:00.177 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:20:00.177 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:20:00.177 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:20:00.177 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:28.043578+0000", 2024-09-10T19:20:00.177 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.93114602599999996 2024-09-10T19:20:00.178 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:20:00.178 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:20:00.178 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-10T19:20:00.178 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:28.045997+0000", 2024-09-10T19:20:00.178 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00241902 2024-09-10T19:20:00.178 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:20:00.178 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:20:00.178 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 3,5", 2024-09-10T19:20:00.178 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:28.050259+0000", 2024-09-10T19:20:00.178 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0042620139999999997 2024-09-10T19:20:00.178 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:20:00.178 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:20:00.178 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-09-10T19:20:00.178 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:28.073477+0000", 2024-09-10T19:20:00.179 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.023217686000000001 2024-09-10T19:20:00.179 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:20:00.179 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:20:00.179 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:20:00.179 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:28.398126+0000", 2024-09-10T19:20:00.179 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.32464874399999999 2024-09-10T19:20:00.179 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:20:00.179 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:20:00.179 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:20:00.179 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:28.398842+0000", 2024-09-10T19:20:00.179 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:20:00.181 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000715969 2024-09-10T19:20:00.181 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:20:00.181 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:20:00.181 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-09-10T19:20:00.181 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:28.399834+0000", 2024-09-10T19:20:00.181 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00099297300000000008 2024-09-10T19:20:00.181 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:20:00.181 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:20:00.181 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-10T19:20:00.182 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:28.403971+0000", 2024-09-10T19:20:00.182 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0041364660000000001 2024-09-10T19:20:00.182 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:20:00.182 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-10T19:20:00.182 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:20:00.182 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:20:00.182 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:20:00.182 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.15051.0:93278 8.3 8:c36e9ad5:::benchmark_data_smithi183_114924_object93277:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e178)", 2024-09-10T19:20:00.182 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-10T19:17:27.125300+0000", 2024-09-10T19:20:00.182 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 152.96246872500001, 2024-09-10T19:20:00.182 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.2853549200000001, 2024-09-10T19:20:00.182 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-10T19:20:00.182 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T19:20:00.182 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-10T19:20:00.183 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.15051", 2024-09-10T19:20:00.183 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.183:0/3632028672", 2024-09-10T19:20:00.183 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 93278 2024-09-10T19:20:00.183 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:20:00.183 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-10T19:20:00.183 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:20:00.183 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-10T19:20:00.183 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:27.125300+0000", 2024-09-10T19:20:00.183 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:20:00.183 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:20:00.183 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:20:00.183 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-10T19:20:00.183 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:27.125300+0000", 2024-09-10T19:20:00.183 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:20:00.184 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:20:00.184 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:20:00.184 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-10T19:20:00.184 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:27.125299+0000", 2024-09-10T19:20:00.184 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999986 2024-09-10T19:20:00.184 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:20:00.184 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:20:00.184 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-10T19:20:00.184 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:27.125370+0000", 2024-09-10T19:20:00.184 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.1624000000000004e-05 2024-09-10T19:20:00.184 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:20:00.184 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:20:00.184 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-10T19:20:00.185 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:27.125386+0000", 2024-09-10T19:20:00.185 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.5608e-05 2024-09-10T19:20:00.185 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:20:00.185 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:20:00.185 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-10T19:20:00.185 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:27.125392+0000", 2024-09-10T19:20:00.185 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.4300000000000003e-06 2024-09-10T19:20:00.185 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:20:00.185 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:20:00.185 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:20:00.185 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:27.125449+0000", 2024-09-10T19:20:00.185 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.6604999999999998e-05 2024-09-10T19:20:00.185 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:20:00.185 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:20:00.186 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for peered", 2024-09-10T19:20:00.186 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:27.125459+0000", 2024-09-10T19:20:00.186 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.9699999999999994e-06 2024-09-10T19:20:00.186 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:20:00.186 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:20:00.186 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:20:00.186 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:28.053203+0000", 2024-09-10T19:20:00.186 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.92774412299999998 2024-09-10T19:20:00.186 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:20:00.186 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:20:00.186 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-10T19:20:00.186 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:28.055638+0000", 2024-09-10T19:20:00.186 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0024347959999999999 2024-09-10T19:20:00.187 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:20:00.187 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:20:00.187 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 3,5", 2024-09-10T19:20:00.187 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:28.060056+0000", 2024-09-10T19:20:00.187 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0044181480000000002 2024-09-10T19:20:00.187 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:20:00.187 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:20:00.187 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-09-10T19:20:00.187 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:28.357493+0000", 2024-09-10T19:20:00.187 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.29743763200000001 2024-09-10T19:20:00.187 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:20:00.187 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:20:00.187 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:20:00.187 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:28.404853+0000", 2024-09-10T19:20:00.188 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.047359471 2024-09-10T19:20:00.188 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:20:00.188 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:20:00.188 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:20:00.188 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:28.405508+0000", 2024-09-10T19:20:00.188 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00065505900000000004 2024-09-10T19:20:00.188 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:20:00.188 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:20:00.188 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-09-10T19:20:00.189 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:28.406505+0000", 2024-09-10T19:20:00.189 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00099746100000000005 2024-09-10T19:20:00.189 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:20:00.189 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:20:00.189 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-10T19:20:00.189 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:28.410655+0000", 2024-09-10T19:20:00.189 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0041496019999999996 2024-09-10T19:20:00.189 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:20:00.189 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-10T19:20:00.189 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:20:00.189 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:20:00.189 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:20:00.189 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.15051.0:93350 8.3 8:c18fa4fe:::benchmark_data_smithi183_114924_object93349:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e178)", 2024-09-10T19:20:00.189 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-10T19:17:27.234632+0000", 2024-09-10T19:20:00.190 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 152.85313706400001, 2024-09-10T19:20:00.190 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.1849550170000001, 2024-09-10T19:20:00.190 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-10T19:20:00.190 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T19:20:00.190 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-10T19:20:00.190 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.15051", 2024-09-10T19:20:00.190 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.183:0/3632028672", 2024-09-10T19:20:00.190 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 93350 2024-09-10T19:20:00.190 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:20:00.190 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-10T19:20:00.190 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:20:00.190 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-10T19:20:00.190 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:27.234632+0000", 2024-09-10T19:20:00.190 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:20:00.191 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:20:00.191 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:20:00.191 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-10T19:20:00.191 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:27.234632+0000", 2024-09-10T19:20:00.191 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:20:00.191 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:20:00.191 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:20:00.191 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-10T19:20:00.191 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:27.234630+0000", 2024-09-10T19:20:00.191 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999986 2024-09-10T19:20:00.191 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:20:00.191 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:20:00.191 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-10T19:20:00.191 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:27.234688+0000", 2024-09-10T19:20:00.192 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.7290000000000002e-05 2024-09-10T19:20:00.192 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:20:00.192 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:20:00.192 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-10T19:20:00.192 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:27.234701+0000", 2024-09-10T19:20:00.192 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.2809999999999999e-05 2024-09-10T19:20:00.192 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:20:00.192 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:20:00.192 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-10T19:20:00.192 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:27.234705+0000", 2024-09-10T19:20:00.192 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.5650000000000003e-06 2024-09-10T19:20:00.192 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:20:00.192 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:20:00.193 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:20:00.193 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:27.234750+0000", 2024-09-10T19:20:00.193 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.4950000000000002e-05 2024-09-10T19:20:00.193 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:20:00.193 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:20:00.193 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for peered", 2024-09-10T19:20:00.193 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:27.234768+0000", 2024-09-10T19:20:00.193 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.7472000000000001e-05 2024-09-10T19:20:00.193 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:20:00.193 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:20:00.193 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:20:00.193 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:28.062852+0000", 2024-09-10T19:20:00.193 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.82808465099999995 2024-09-10T19:20:00.193 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:20:00.194 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:20:00.194 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-10T19:20:00.194 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:28.065344+0000", 2024-09-10T19:20:00.194 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0024914149999999999 2024-09-10T19:20:00.194 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:20:00.194 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:20:00.194 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 3,5", 2024-09-10T19:20:00.194 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:28.070338+0000", 2024-09-10T19:20:00.194 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0049941220000000001 2024-09-10T19:20:00.194 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:20:00.194 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:20:00.194 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-09-10T19:20:00.194 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:28.357694+0000", 2024-09-10T19:20:00.194 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.287356587 2024-09-10T19:20:00.195 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:20:00.195 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:20:00.195 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:20:00.195 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:28.411581+0000", 2024-09-10T19:20:00.195 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.053886430999999999 2024-09-10T19:20:00.195 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:20:00.195 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:20:00.195 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:20:00.195 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:28.412526+0000", 2024-09-10T19:20:00.195 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00094505499999999998 2024-09-10T19:20:00.195 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:20:00.195 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:20:00.195 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-09-10T19:20:00.195 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:28.414135+0000", 2024-09-10T19:20:00.195 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0016088039999999999 2024-09-10T19:20:00.196 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:20:00.196 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:20:00.196 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-10T19:20:00.196 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:28.419587+0000", 2024-09-10T19:20:00.196 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0054521830000000002 2024-09-10T19:20:00.196 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:20:00.196 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-10T19:20:00.196 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:20:00.196 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:20:00.196 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:20:00.196 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDPGPush(8.13 188/180 [PushOp(8:c957bfd1:::benchmark_data_smithi183_114924_object20031:head, version: 159'1312, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(8:c957bfd1:::benchmark_data_smithi183_114924_object20031:head@159'1312, size: 65536, copy_subset: [0~65536], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:65536, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-10T19:20:00.196 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-10T19:17:49.291507+0000", 2024-09-10T19:20:00.196 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 130.79626211499999, 2024-09-10T19:20:00.196 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.95131088500000005, 2024-09-10T19:20:00.196 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-10T19:20:00.197 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "started", 2024-09-10T19:20:00.197 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-10T19:20:00.197 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:20:00.197 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-10T19:20:00.197 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:49.291507+0000", 2024-09-10T19:20:00.197 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:20:00.197 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:20:00.197 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:20:00.197 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-10T19:20:00.197 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:49.291507+0000", 2024-09-10T19:20:00.197 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:20:00.197 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:20:00.197 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:20:00.197 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-10T19:20:00.197 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:49.291506+0000", 2024-09-10T19:20:00.198 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999995 2024-09-10T19:20:00.198 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:20:00.198 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:20:00.198 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-10T19:20:00.198 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:49.291542+0000", 2024-09-10T19:20:00.198 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.6282000000000003e-05 2024-09-10T19:20:00.198 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:20:00.198 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:20:00.198 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-10T19:20:00.198 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:49.291567+0000", 2024-09-10T19:20:00.198 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.4939e-05 2024-09-10T19:20:00.198 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:20:00.198 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:20:00.198 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-10T19:20:00.199 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:49.291580+0000", 2024-09-10T19:20:00.199 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.2288e-05 2024-09-10T19:20:00.199 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:20:00.199 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:20:00.199 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:20:00.199 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:50.241711+0000", 2024-09-10T19:20:00.199 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.95013111299999997 2024-09-10T19:20:00.199 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:20:00.199 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:20:00.199 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-10T19:20:00.199 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:50.241718+0000", 2024-09-10T19:20:00.199 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.6399999999999997e-06 2024-09-10T19:20:00.199 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:20:00.200 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:20:00.200 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-10T19:20:00.200 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:50.242818+0000", 2024-09-10T19:20:00.200 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.001099196 2024-09-10T19:20:00.200 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:20:00.200 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-10T19:20:00.200 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:20:00.200 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:20:00.200 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:20:00.200 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDPGPush(8.13 190/180 [PushOp(8:c984e6b4:::benchmark_data_smithi183_114924_object20685:head, version: 159'1352, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(8:c984e6b4:::benchmark_data_smithi183_114924_object20685:head@159'1352, size: 65536, copy_subset: [0~65536], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:65536, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-10T19:20:00.200 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-10T19:17:53.705526+0000", 2024-09-10T19:20:00.200 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 126.382243303, 2024-09-10T19:20:00.200 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.1027652100000001, 2024-09-10T19:20:00.200 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-10T19:20:00.200 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "started", 2024-09-10T19:20:00.201 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-10T19:20:00.201 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:20:00.201 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-10T19:20:00.201 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:53.705526+0000", 2024-09-10T19:20:00.201 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:20:00.201 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:20:00.201 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:20:00.201 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-10T19:20:00.201 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:53.705526+0000", 2024-09-10T19:20:00.201 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:20:00.201 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:20:00.201 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:20:00.201 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-10T19:20:00.201 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:53.705525+0000", 2024-09-10T19:20:00.201 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.999999 2024-09-10T19:20:00.202 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:20:00.202 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:20:00.202 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-10T19:20:00.202 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:53.705571+0000", 2024-09-10T19:20:00.202 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.6814999999999998e-05 2024-09-10T19:20:00.202 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:20:00.202 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:20:00.202 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-10T19:20:00.202 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:53.705590+0000", 2024-09-10T19:20:00.202 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.861e-05 2024-09-10T19:20:00.202 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:20:00.202 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:20:00.202 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-10T19:20:00.202 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:53.705601+0000", 2024-09-10T19:20:00.203 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.0787e-05 2024-09-10T19:20:00.203 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:20:00.203 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:20:00.203 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:20:00.203 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:54.807476+0000", 2024-09-10T19:20:00.203 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.1018752549999999 2024-09-10T19:20:00.203 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:20:00.203 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:20:00.203 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-10T19:20:00.203 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:54.807481+0000", 2024-09-10T19:20:00.203 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.6800000000000001e-06 2024-09-10T19:20:00.203 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:20:00.203 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:20:00.203 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-10T19:20:00.203 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:54.808291+0000", 2024-09-10T19:20:00.204 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00081010900000000005 2024-09-10T19:20:00.204 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:20:00.204 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-10T19:20:00.204 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:20:00.204 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:20:00.204 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:20:00.204 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDPGPush(8.13 191/180 [PushOp(8:ce4937bd:::benchmark_data_smithi183_114924_object20690:head, version: 159'1353, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(8:ce4937bd:::benchmark_data_smithi183_114924_object20690:head@159'1353, size: 65536, copy_subset: [0~65536], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:65536, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-10T19:20:00.204 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-10T19:17:53.824796+0000", 2024-09-10T19:20:00.204 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 126.26297243800001, 2024-09-10T19:20:00.204 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.98403609800000003, 2024-09-10T19:20:00.204 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-10T19:20:00.204 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "started", 2024-09-10T19:20:00.204 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-10T19:20:00.204 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:20:00.205 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-10T19:20:00.205 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:53.824796+0000", 2024-09-10T19:20:00.205 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:20:00.205 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:20:00.205 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:20:00.205 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-10T19:20:00.205 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:53.824796+0000", 2024-09-10T19:20:00.205 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:20:00.205 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:20:00.205 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:20:00.205 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-10T19:20:00.205 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:53.824796+0000", 2024-09-10T19:20:00.205 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.999999 2024-09-10T19:20:00.205 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:20:00.205 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:20:00.206 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-10T19:20:00.206 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:53.824838+0000", 2024-09-10T19:20:00.206 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.1989000000000001e-05 2024-09-10T19:20:00.206 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:20:00.206 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:20:00.206 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-10T19:20:00.206 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:53.824855+0000", 2024-09-10T19:20:00.206 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.6866000000000001e-05 2024-09-10T19:20:00.206 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:20:00.206 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:20:00.206 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-10T19:20:00.206 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:53.824864+0000", 2024-09-10T19:20:00.206 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.6900000000000004e-06 2024-09-10T19:20:00.206 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:20:00.206 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:20:00.207 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:20:00.207 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:54.808328+0000", 2024-09-10T19:20:00.207 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.98346424200000004 2024-09-10T19:20:00.207 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:20:00.207 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:20:00.207 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-10T19:20:00.207 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:54.808333+0000", 2024-09-10T19:20:00.207 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.2150000000000003e-06 2024-09-10T19:20:00.207 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:20:00.207 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:20:00.207 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-10T19:20:00.207 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:54.808833+0000", 2024-09-10T19:20:00.207 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00049981699999999995 2024-09-10T19:20:00.207 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:20:00.207 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-10T19:20:00.208 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:20:00.208 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:20:00.208 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:20:00.208 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDPGPush(8.13 192/180 [PushOp(8:cf919698:::benchmark_data_smithi183_114924_object26566:head, version: 159'1771, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(8:cf919698:::benchmark_data_smithi183_114924_object26566:head@159'1771, size: 65536, copy_subset: [0~65536], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:65536, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-10T19:20:00.208 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-10T19:17:56.866594+0000", 2024-09-10T19:20:00.208 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 123.221175296, 2024-09-10T19:20:00.208 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.49036462, 2024-09-10T19:20:00.208 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-10T19:20:00.208 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "started", 2024-09-10T19:20:00.208 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-10T19:20:00.208 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:20:00.208 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-10T19:20:00.208 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:56.866594+0000", 2024-09-10T19:20:00.208 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:20:00.209 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:20:00.209 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:20:00.209 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-10T19:20:00.209 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:56.866594+0000", 2024-09-10T19:20:00.209 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:20:00.209 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:20:00.209 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:20:00.209 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-10T19:20:00.209 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:56.866593+0000", 2024-09-10T19:20:00.209 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999995 2024-09-10T19:20:00.209 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:20:00.209 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:20:00.209 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-10T19:20:00.210 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:56.866670+0000", 2024-09-10T19:20:00.210 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.6539999999999996e-05 2024-09-10T19:20:00.210 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:20:00.210 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:20:00.210 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-10T19:20:00.210 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:56.866685+0000", 2024-09-10T19:20:00.210 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.5319999999999999e-05 2024-09-10T19:20:00.210 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:20:00.210 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:20:00.210 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-10T19:20:00.210 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:56.866693+0000", 2024-09-10T19:20:00.210 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.4980000000000006e-06 2024-09-10T19:20:00.210 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:20:00.210 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:20:00.210 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:20:00.211 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:58.356122+0000", 2024-09-10T19:20:00.211 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.4894283750000001 2024-09-10T19:20:00.211 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:20:00.211 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:20:00.211 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-10T19:20:00.211 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:58.356126+0000", 2024-09-10T19:20:00.211 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.9260000000000002e-06 2024-09-10T19:20:00.211 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:20:00.211 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:20:00.211 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-10T19:20:00.211 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:58.356958+0000", 2024-09-10T19:20:00.211 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00083252900000000004 2024-09-10T19:20:00.211 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:20:00.211 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-10T19:20:00.212 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:20:00.212 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:20:00.212 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:20:00.212 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDPGPush(8.13 192/180 [PushOp(8:c8afa908:::benchmark_data_smithi183_114924_object26967:head, version: 159'1795, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(8:c8afa908:::benchmark_data_smithi183_114924_object26967:head@159'1795, size: 65536, copy_subset: [0~65536], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:65536, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-10T19:20:00.212 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-10T19:17:59.974190+0000", 2024-09-10T19:20:00.212 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 120.113578907, 2024-09-10T19:20:00.212 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.97933013000000002, 2024-09-10T19:20:00.212 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-10T19:20:00.212 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "started", 2024-09-10T19:20:00.212 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-10T19:20:00.212 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:20:00.212 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-10T19:20:00.212 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:59.974190+0000", 2024-09-10T19:20:00.212 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:20:00.213 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:20:00.213 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:20:00.213 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-10T19:20:00.213 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:59.974190+0000", 2024-09-10T19:20:00.213 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:20:00.213 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:20:00.213 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:20:00.213 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-10T19:20:00.213 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:59.974190+0000", 2024-09-10T19:20:00.213 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999995 2024-09-10T19:20:00.213 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:20:00.213 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:20:00.213 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-10T19:20:00.213 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:59.974226+0000", 2024-09-10T19:20:00.214 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.6442000000000001e-05 2024-09-10T19:20:00.214 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:20:00.214 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:20:00.214 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-10T19:20:00.214 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:59.974243+0000", 2024-09-10T19:20:00.214 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.6586e-05 2024-09-10T19:20:00.214 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:20:00.214 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:20:00.214 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-10T19:20:00.214 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:59.974252+0000", 2024-09-10T19:20:00.214 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.4550000000000001e-06 2024-09-10T19:20:00.214 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:20:00.214 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:20:00.214 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:20:00.215 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:18:00.951928+0000", 2024-09-10T19:20:00.215 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.97767566100000003 2024-09-10T19:20:00.215 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:20:00.215 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:20:00.215 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-10T19:20:00.215 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:18:00.951935+0000", 2024-09-10T19:20:00.215 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.7139999999999996e-06 2024-09-10T19:20:00.215 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:20:00.215 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:20:00.215 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-10T19:20:00.215 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:18:00.953520+0000", 2024-09-10T19:20:00.215 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0015855299999999999 2024-09-10T19:20:00.215 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:20:00.215 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-10T19:20:00.216 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:20:00.216 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:20:00.216 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:20:00.216 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.15051.0:172172 8.17 8:ec5a1e4c:::benchmark_data_smithi183_114924_object51889:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e205)", 2024-09-10T19:20:00.216 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-10T19:18:43.179473+0000", 2024-09-10T19:20:00.216 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 76.908296124000003, 2024-09-10T19:20:00.216 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.99473523600000002, 2024-09-10T19:20:00.216 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-10T19:20:00.216 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T19:20:00.216 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-10T19:20:00.216 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.15051", 2024-09-10T19:20:00.216 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.183:0/3632028672", 2024-09-10T19:20:00.216 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 172172 2024-09-10T19:20:00.217 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:20:00.217 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-10T19:20:00.217 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:20:00.217 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-10T19:20:00.217 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:18:43.179473+0000", 2024-09-10T19:20:00.217 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:20:00.217 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:20:00.217 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:20:00.217 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-10T19:20:00.217 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:18:43.179473+0000", 2024-09-10T19:20:00.217 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:20:00.217 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:20:00.217 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:20:00.217 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-10T19:20:00.218 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:18:43.179471+0000", 2024-09-10T19:20:00.218 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999986 2024-09-10T19:20:00.218 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:20:00.218 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:20:00.218 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-10T19:20:00.218 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:18:43.179478+0000", 2024-09-10T19:20:00.218 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.2910000000000004e-06 2024-09-10T19:20:00.218 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:20:00.218 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:20:00.218 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-10T19:20:00.218 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:18:43.179489+0000", 2024-09-10T19:20:00.218 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.1528e-05 2024-09-10T19:20:00.218 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:20:00.218 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:20:00.219 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-10T19:20:00.219 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:18:43.179495+0000", 2024-09-10T19:20:00.219 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.0480000000000004e-06 2024-09-10T19:20:00.219 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:20:00.219 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:20:00.219 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:20:00.219 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:18:43.179674+0000", 2024-09-10T19:20:00.219 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00017865900000000001 2024-09-10T19:20:00.219 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:20:00.219 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:20:00.219 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for peered", 2024-09-10T19:20:00.219 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:18:43.179687+0000", 2024-09-10T19:20:00.219 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.2853999999999999e-05 2024-09-10T19:20:00.220 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:20:00.220 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:20:00.220 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:20:00.220 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:18:44.165270+0000", 2024-09-10T19:20:00.220 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.98558308299999997 2024-09-10T19:20:00.220 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:20:00.220 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:20:00.220 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-10T19:20:00.220 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:18:44.165366+0000", 2024-09-10T19:20:00.220 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.6162000000000005e-05 2024-09-10T19:20:00.220 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:20:00.220 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:20:00.220 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0,6", 2024-09-10T19:20:00.220 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:18:44.165590+0000", 2024-09-10T19:20:00.221 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00022405299999999999 2024-09-10T19:20:00.221 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:20:00.221 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:20:00.221 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-09-10T19:20:00.221 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:18:44.172416+0000", 2024-09-10T19:20:00.221 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0068258320000000004 2024-09-10T19:20:00.221 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:20:00.221 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:20:00.221 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:20:00.221 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:18:44.174090+0000", 2024-09-10T19:20:00.221 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0016737659999999999 2024-09-10T19:20:00.221 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:20:00.221 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:20:00.221 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:20:00.222 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:18:44.174127+0000", 2024-09-10T19:20:00.222 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.7360999999999997e-05 2024-09-10T19:20:00.222 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:20:00.222 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:20:00.222 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-09-10T19:20:00.222 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:18:44.174171+0000", 2024-09-10T19:20:00.222 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.4270999999999997e-05 2024-09-10T19:20:00.222 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:20:00.222 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:20:00.222 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-10T19:20:00.222 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:18:44.174208+0000", 2024-09-10T19:20:00.222 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.6720000000000001e-05 2024-09-10T19:20:00.222 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:20:00.222 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-10T19:20:00.222 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:20:00.223 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:20:00.223 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:20:00.223 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.15051.0:172461 8.15 8:aff08a5b:::benchmark_data_smithi183_114924_object52178:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e206)", 2024-09-10T19:20:00.223 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-10T19:18:44.222397+0000", 2024-09-10T19:20:00.223 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 75.865371808000006, 2024-09-10T19:20:00.223 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.95883427200000004, 2024-09-10T19:20:00.223 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-10T19:20:00.223 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T19:20:00.223 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-10T19:20:00.223 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.15051", 2024-09-10T19:20:00.223 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.183:0/3632028672", 2024-09-10T19:20:00.223 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 172461 2024-09-10T19:20:00.223 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:20:00.223 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-10T19:20:00.224 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:20:00.224 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-10T19:20:00.224 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:18:44.222397+0000", 2024-09-10T19:20:00.224 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:20:00.224 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:20:00.224 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:20:00.224 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-10T19:20:00.224 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:18:44.222397+0000", 2024-09-10T19:20:00.224 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:20:00.224 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:20:00.224 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:20:00.224 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-10T19:20:00.224 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:18:44.222396+0000", 2024-09-10T19:20:00.224 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999986 2024-09-10T19:20:00.224 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:20:00.225 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:20:00.225 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-10T19:20:00.225 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:18:44.222403+0000", 2024-09-10T19:20:00.225 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.8820000000000003e-06 2024-09-10T19:20:00.225 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:20:00.225 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:20:00.225 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-10T19:20:00.225 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:18:44.222416+0000", 2024-09-10T19:20:00.225 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.3291e-05 2024-09-10T19:20:00.225 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:20:00.225 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:20:00.225 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-10T19:20:00.225 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:18:44.222422+0000", 2024-09-10T19:20:00.225 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.2580000000000001e-06 2024-09-10T19:20:00.226 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:20:00.226 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:20:00.226 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:20:00.226 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:18:44.222507+0000", 2024-09-10T19:20:00.226 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.5118999999999996e-05 2024-09-10T19:20:00.226 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:20:00.226 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:20:00.226 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for peered", 2024-09-10T19:20:00.226 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:18:44.222515+0000", 2024-09-10T19:20:00.226 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.6380000000000004e-06 2024-09-10T19:20:00.226 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:20:00.226 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:20:00.226 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:20:00.226 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:18:45.163741+0000", 2024-09-10T19:20:00.227 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.94122644200000005 2024-09-10T19:20:00.227 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:20:00.227 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:20:00.227 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-10T19:20:00.227 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:18:45.163835+0000", 2024-09-10T19:20:00.227 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.3490000000000001e-05 2024-09-10T19:20:00.227 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:20:00.227 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:20:00.227 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0,6", 2024-09-10T19:20:00.227 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:18:45.164047+0000", 2024-09-10T19:20:00.227 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00021215599999999999 2024-09-10T19:20:00.227 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:20:00.227 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:20:00.227 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-09-10T19:20:00.228 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:18:45.168946+0000", 2024-09-10T19:20:00.228 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0048991410000000001 2024-09-10T19:20:00.228 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:20:00.228 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:20:00.228 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:20:00.228 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:18:45.181058+0000", 2024-09-10T19:20:00.228 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.012112303 2024-09-10T19:20:00.228 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:20:00.228 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:20:00.228 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:20:00.228 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:18:45.181172+0000", 2024-09-10T19:20:00.228 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00011330100000000001 2024-09-10T19:20:00.228 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:20:00.229 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:20:00.229 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-09-10T19:20:00.229 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:18:45.181202+0000", 2024-09-10T19:20:00.229 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.0518999999999997e-05 2024-09-10T19:20:00.229 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:20:00.229 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:20:00.229 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-10T19:20:00.229 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:18:45.181231+0000", 2024-09-10T19:20:00.229 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.9150999999999999e-05 2024-09-10T19:20:00.229 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:20:00.229 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-10T19:20:00.229 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:20:00.229 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:20:00.230 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-10T19:20:00.230 INFO:teuthology.orchestra.run.smithi191.stdout:} 2024-09-10T19:20:00.251 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:20:00.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:20:00.252 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:20:00.352 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:20:00.374 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:20:00.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:20:00.375 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:20:00.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:20:00 smithi191 ceph-mon[32884]: 5.8 scrub starts 2024-09-10T19:20:00.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:20:00 smithi191 ceph-mon[32884]: 5.8 scrub ok 2024-09-10T19:20:00.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:20:00 smithi191 ceph-mon[32884]: purged_snaps scrub starts 2024-09-10T19:20:00.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:20:00 smithi191 ceph-mon[32884]: purged_snaps scrub ok 2024-09-10T19:20:00.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:20:00 smithi191 ceph-mon[32884]: from='client.? 172.21.15.183:0/2184205940' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_2", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-09-10T19:20:00.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:20:00 smithi191 ceph-mon[32884]: osdmap e232: 8 total, 8 up, 6 in 2024-09-10T19:20:00.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:20:00 smithi191 ceph-mon[32884]: Health detail: HEALTH_WARN nodeep-scrub flag(s) set; 1 pool(s) do not have an application enabled 2024-09-10T19:20:00.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:20:00 smithi191 ceph-mon[32884]: [WRN] OSDMAP_FLAGS: nodeep-scrub flag(s) set 2024-09-10T19:20:00.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:20:00 smithi191 ceph-mon[32884]: [WRN] POOL_APP_NOT_ENABLED: 1 pool(s) do not have an application enabled 2024-09-10T19:20:00.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:20:00 smithi191 ceph-mon[32884]: application not enabled on pool 'unique_pool_2' 2024-09-10T19:20:00.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:20:00 smithi191 ceph-mon[32884]: use 'ceph osd pool application enable ', where is 'cephfs', 'rbd', 'rgw', or freeform for custom applications. 2024-09-10T19:20:00.476 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:20:00.497 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:20:00.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:20:00.498 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:20:00.599 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:20:00.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:20:00.621 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:20:00.621 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:20:00.649 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:20:00 smithi183 ceph-mon[29736]: 5.8 scrub starts 2024-09-10T19:20:00.649 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:20:00 smithi183 ceph-mon[29736]: 5.8 scrub ok 2024-09-10T19:20:00.649 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:20:00 smithi183 ceph-mon[29736]: purged_snaps scrub starts 2024-09-10T19:20:00.649 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:20:00 smithi183 ceph-mon[29736]: purged_snaps scrub ok 2024-09-10T19:20:00.649 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:20:00 smithi183 ceph-mon[29736]: from='client.? 172.21.15.183:0/2184205940' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_2", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-09-10T19:20:00.650 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:20:00 smithi183 ceph-mon[29736]: osdmap e232: 8 total, 8 up, 6 in 2024-09-10T19:20:00.650 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:20:00 smithi183 ceph-mon[29736]: Health detail: HEALTH_WARN nodeep-scrub flag(s) set; 1 pool(s) do not have an application enabled 2024-09-10T19:20:00.650 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:20:00 smithi183 ceph-mon[29736]: [WRN] OSDMAP_FLAGS: nodeep-scrub flag(s) set 2024-09-10T19:20:00.650 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:20:00 smithi183 ceph-mon[29736]: [WRN] POOL_APP_NOT_ENABLED: 1 pool(s) do not have an application enabled 2024-09-10T19:20:00.650 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:20:00 smithi183 ceph-mon[29736]: application not enabled on pool 'unique_pool_2' 2024-09-10T19:20:00.650 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:20:00 smithi183 ceph-mon[29736]: use 'ceph osd pool application enable ', where is 'cephfs', 'rbd', 'rgw', or freeform for custom applications. 2024-09-10T19:20:00.650 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:20:00 smithi183 ceph-mon[41562]: 5.8 scrub starts 2024-09-10T19:20:00.650 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:20:00 smithi183 ceph-mon[41562]: 5.8 scrub ok 2024-09-10T19:20:00.650 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:20:00 smithi183 ceph-mon[41562]: purged_snaps scrub starts 2024-09-10T19:20:00.650 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:20:00 smithi183 ceph-mon[41562]: purged_snaps scrub ok 2024-09-10T19:20:00.650 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:20:00 smithi183 ceph-mon[41562]: from='client.? 172.21.15.183:0/2184205940' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_2", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-09-10T19:20:00.651 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:20:00 smithi183 ceph-mon[41562]: osdmap e232: 8 total, 8 up, 6 in 2024-09-10T19:20:00.651 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:20:00 smithi183 ceph-mon[41562]: Health detail: HEALTH_WARN nodeep-scrub flag(s) set; 1 pool(s) do not have an application enabled 2024-09-10T19:20:00.651 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:20:00 smithi183 ceph-mon[41562]: [WRN] OSDMAP_FLAGS: nodeep-scrub flag(s) set 2024-09-10T19:20:00.651 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:20:00 smithi183 ceph-mon[41562]: [WRN] POOL_APP_NOT_ENABLED: 1 pool(s) do not have an application enabled 2024-09-10T19:20:00.651 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:20:00 smithi183 ceph-mon[41562]: application not enabled on pool 'unique_pool_2' 2024-09-10T19:20:00.651 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:20:00 smithi183 ceph-mon[41562]: use 'ceph osd pool application enable ', where is 'cephfs', 'rbd', 'rgw', or freeform for custom applications. 2024-09-10T19:20:00.667 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:20:00.723 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:20:00.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:20:00.745 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:20:00.745 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:20:00.846 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:20:00.868 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:20:00.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:20:00.869 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:20:00.965 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph daemon osd.1 dump_ops_in_flight 2024-09-10T19:20:00.969 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:20:01.029 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:20:01.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:20:01.030 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:20:01.131 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:20:01.152 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:20:01.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:20:01.153 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:20:01.254 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:20:01.275 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:20:01.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:20:01.276 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:20:01.377 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:20:01.398 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:20:01.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:20:01.399 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:20:01.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:20:01 smithi191 ceph-mon[32884]: from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:20:01.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:20:01 smithi191 ceph-mon[32884]: pgmap v440: 148 pgs: 148 active+clean; 582 KiB data, 2.3 GiB used, 534 GiB / 536 GiB avail; 853 B/s rd, 0 op/s 2024-09-10T19:20:01.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:20:01 smithi191 ceph-mon[32884]: 5.16 scrub starts 2024-09-10T19:20:01.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:20:01 smithi191 ceph-mon[32884]: 5.16 scrub ok 2024-09-10T19:20:01.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:20:01 smithi191 ceph-mon[32884]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-10T19:20:01.500 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:20:01.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:20:01.525 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:20:01.525 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:20:01.626 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:20:01.648 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:20:01.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:20:01.649 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:20:01.679 DEBUG:teuthology.orchestra.run.smithi183:> /bin/sh -c 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --no-log-to-stderr --name client.0 -b 65536 --object-size 65536 -p unique_pool_2 bench 90 write' 2024-09-10T19:20:01.680 INFO:tasks.thrashosds.thrasher:Growing pool .mgr 2024-09-10T19:20:01.680 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph pg dump --format=json 2024-09-10T19:20:01.681 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:20:01 smithi183 ceph-mon[29736]: from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:20:01.681 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:20:01 smithi183 ceph-mon[29736]: pgmap v440: 148 pgs: 148 active+clean; 582 KiB data, 2.3 GiB used, 534 GiB / 536 GiB avail; 853 B/s rd, 0 op/s 2024-09-10T19:20:01.681 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:20:01 smithi183 ceph-mon[29736]: 5.16 scrub starts 2024-09-10T19:20:01.681 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:20:01 smithi183 ceph-mon[29736]: 5.16 scrub ok 2024-09-10T19:20:01.682 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:20:01 smithi183 ceph-mon[29736]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-10T19:20:01.684 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Sep 10 19:20:01 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-y[29989]: ::ffff:172.21.15.191 - - [10/Sep/2024:19:20:01] "GET /metrics HTTP/1.1" 200 224108 "" "Prometheus/2.43.0" 2024-09-10T19:20:01.684 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:20:01 smithi183 ceph-mon[41562]: from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:20:01.684 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:20:01 smithi183 ceph-mon[41562]: pgmap v440: 148 pgs: 148 active+clean; 582 KiB data, 2.3 GiB used, 534 GiB / 536 GiB avail; 853 B/s rd, 0 op/s 2024-09-10T19:20:01.684 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:20:01 smithi183 ceph-mon[41562]: 5.16 scrub starts 2024-09-10T19:20:01.685 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:20:01 smithi183 ceph-mon[41562]: 5.16 scrub ok 2024-09-10T19:20:01.685 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:20:01 smithi183 ceph-mon[41562]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-10T19:20:01.685 INFO:tasks.radosbench:joining radosbench (timing out after 3000s) 2024-09-10T19:20:01.685 INFO:teuthology.orchestra.run:waiting for 3000 2024-09-10T19:20:01.729 INFO:tasks.radosbench.radosbench.0.smithi183.stdout:hints = 1 2024-09-10T19:20:01.729 INFO:tasks.radosbench.radosbench.0.smithi183.stdout:Maintaining 16 concurrent writes of 65536 bytes to objects of size 65536 for up to 90 seconds or 0 objects 2024-09-10T19:20:01.729 INFO:tasks.radosbench.radosbench.0.smithi183.stdout:Object prefix: benchmark_data_smithi183_142499 2024-09-10T19:20:01.729 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-10T19:20:01.729 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: 0 0 0 0 0 0 - 0 2024-09-10T19:20:01.750 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:20:01.789 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:20:01.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:20:01.790 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:20:01.891 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:20:01.925 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:20:01.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:20:01.926 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:20:02.027 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:20:02.065 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:20:02.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:20:02.066 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:20:02.167 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:20:02.203 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:20:02.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:20:02.204 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:20:02.305 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:20:02.342 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:20:02.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:20:02.343 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:20:02.431 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:20:02 smithi183 ceph-mon[41562]: 4.1f scrub starts 2024-09-10T19:20:02.431 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:20:02 smithi183 ceph-mon[41562]: 4.1f scrub ok 2024-09-10T19:20:02.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:20:02 smithi183 ceph-mon[29736]: 4.1f scrub starts 2024-09-10T19:20:02.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:20:02 smithi183 ceph-mon[29736]: 4.1f scrub ok 2024-09-10T19:20:02.444 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:20:02.465 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:20:02 smithi191 ceph-mon[32884]: 4.1f scrub starts 2024-09-10T19:20:02.465 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:20:02 smithi191 ceph-mon[32884]: 4.1f scrub ok 2024-09-10T19:20:02.478 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:20:02.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:20:02.479 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:20:02.543 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:20:02.580 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:20:02.614 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:20:02.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:20:02.615 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:20:02.716 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:20:02.729 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: 1 16 3796 3780 236.152 236.25 0.00551438 0.00421701 2024-09-10T19:20:02.753 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:20:02.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:20:02.754 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:20:02.854 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:20:02.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:20:02.972 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:20:02.972 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:20:03.073 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:20:03.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:20:03.112 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:20:03.112 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:20:03.213 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:20:03.222 INFO:teuthology.orchestra.run.smithi183.stderr:nodeep-scrub is unset 2024-09-10T19:20:03.251 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:20:03.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:20:03.252 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:20:03.353 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:20:03.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:20:03.390 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:20:03.390 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:20:03.423 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:20:03 smithi183 ceph-mon[41562]: 5.1e scrub starts 2024-09-10T19:20:03.423 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:20:03 smithi183 ceph-mon[41562]: 5.1e scrub ok 2024-09-10T19:20:03.423 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:20:03 smithi183 ceph-mon[41562]: pgmap v441: 148 pgs: 148 active+clean; 582 KiB data, 2.3 GiB used, 534 GiB / 536 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-09-10T19:20:03.423 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:20:03 smithi183 ceph-mon[41562]: from='client.? 172.21.15.183:0/3535561188' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-10T19:20:03.424 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:20:03 smithi183 ceph-mon[41562]: purged_snaps scrub starts 2024-09-10T19:20:03.424 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:20:03 smithi183 ceph-mon[41562]: purged_snaps scrub ok 2024-09-10T19:20:03.424 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:20:03 smithi183 ceph-mon[41562]: 4.14 scrub starts 2024-09-10T19:20:03.424 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:20:03 smithi183 ceph-mon[41562]: 4.14 scrub ok 2024-09-10T19:20:03.424 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:20:03 smithi183 ceph-mon[29736]: 5.1e scrub starts 2024-09-10T19:20:03.425 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:20:03 smithi183 ceph-mon[29736]: 5.1e scrub ok 2024-09-10T19:20:03.425 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:20:03 smithi183 ceph-mon[29736]: pgmap v441: 148 pgs: 148 active+clean; 582 KiB data, 2.3 GiB used, 534 GiB / 536 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-09-10T19:20:03.425 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:20:03 smithi183 ceph-mon[29736]: from='client.? 172.21.15.183:0/3535561188' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-10T19:20:03.425 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:20:03 smithi183 ceph-mon[29736]: purged_snaps scrub starts 2024-09-10T19:20:03.425 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:20:03 smithi183 ceph-mon[29736]: purged_snaps scrub ok 2024-09-10T19:20:03.426 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:20:03 smithi183 ceph-mon[29736]: 4.14 scrub starts 2024-09-10T19:20:03.426 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:20:03 smithi183 ceph-mon[29736]: 4.14 scrub ok 2024-09-10T19:20:03.491 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:20:03.527 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:20:03.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:20:03.528 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:20:03.629 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:20:03.662 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:20:03.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:20:03.663 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:20:03.715 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:20:03 smithi191 ceph-mon[32884]: 5.1e scrub starts 2024-09-10T19:20:03.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:20:03 smithi191 ceph-mon[32884]: 5.1e scrub ok 2024-09-10T19:20:03.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:20:03 smithi191 ceph-mon[32884]: pgmap v441: 148 pgs: 148 active+clean; 582 KiB data, 2.3 GiB used, 534 GiB / 536 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-09-10T19:20:03.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:20:03 smithi191 ceph-mon[32884]: from='client.? 172.21.15.183:0/3535561188' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-10T19:20:03.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:20:03 smithi191 ceph-mon[32884]: purged_snaps scrub starts 2024-09-10T19:20:03.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:20:03 smithi191 ceph-mon[32884]: purged_snaps scrub ok 2024-09-10T19:20:03.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:20:03 smithi191 ceph-mon[32884]: 4.14 scrub starts 2024-09-10T19:20:03.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:20:03 smithi191 ceph-mon[32884]: 4.14 scrub ok 2024-09-10T19:20:03.732 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: 2 16 7437 7421 231.847 227.562 0.00390941 0.00430235 2024-09-10T19:20:03.764 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:20:03.799 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:20:03.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:20:03.800 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:20:03.883 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:20:03.901 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:20:03.932 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:20:03.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:20:03.933 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:20:04.033 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:20:04.068 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:20:04.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:20:04.069 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:20:04.170 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:20:04.206 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:20:04.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:20:04.208 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:20:04.309 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:20:04.346 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:20:04.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:20:04.347 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:20:04.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:20:04 smithi183 ceph-mon[29736]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-10T19:20:04.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:20:04 smithi183 ceph-mon[29736]: Cluster is now healthy 2024-09-10T19:20:04.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:20:04 smithi183 ceph-mon[29736]: from='client.? 172.21.15.183:0/3535561188' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-10T19:20:04.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:20:04 smithi183 ceph-mon[29736]: osdmap e233: 8 total, 8 up, 6 in 2024-09-10T19:20:04.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:20:04 smithi183 ceph-mon[29736]: 2.11 scrub starts 2024-09-10T19:20:04.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:20:04 smithi183 ceph-mon[29736]: 2.11 scrub ok 2024-09-10T19:20:04.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:20:04 smithi183 ceph-mon[41562]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-10T19:20:04.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:20:04 smithi183 ceph-mon[41562]: Cluster is now healthy 2024-09-10T19:20:04.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:20:04 smithi183 ceph-mon[41562]: from='client.? 172.21.15.183:0/3535561188' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-10T19:20:04.433 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:20:04 smithi183 ceph-mon[41562]: osdmap e233: 8 total, 8 up, 6 in 2024-09-10T19:20:04.433 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:20:04 smithi183 ceph-mon[41562]: 2.11 scrub starts 2024-09-10T19:20:04.433 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:20:04 smithi183 ceph-mon[41562]: 2.11 scrub ok 2024-09-10T19:20:04.447 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:20:04.485 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:20:04.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:20:04.487 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:20:04.588 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:20:04.625 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:20:04.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:20:04.626 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:20:04.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:20:04 smithi191 ceph-mon[32884]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-10T19:20:04.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:20:04 smithi191 ceph-mon[32884]: Cluster is now healthy 2024-09-10T19:20:04.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:20:04 smithi191 ceph-mon[32884]: from='client.? 172.21.15.183:0/3535561188' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-10T19:20:04.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:20:04 smithi191 ceph-mon[32884]: osdmap e233: 8 total, 8 up, 6 in 2024-09-10T19:20:04.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:20:04 smithi191 ceph-mon[32884]: 2.11 scrub starts 2024-09-10T19:20:04.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:20:04 smithi191 ceph-mon[32884]: 2.11 scrub ok 2024-09-10T19:20:04.727 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:20:04.730 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: 3 16 10804 10788 224.706 210.438 0.00432191 0.00444255 2024-09-10T19:20:04.763 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:20:04.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:20:04.764 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:20:04.865 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:20:04.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:20:04.899 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:20:04.900 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:20:05.001 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:20:05.035 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:20:05.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:20:05.036 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:20:05.137 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:20:05.169 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:20:05.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:20:05.170 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:20:05.271 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:20:05.304 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:20:05.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:20:05.305 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:20:05.349 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:20:05 smithi183 ceph-mon[29736]: 5.b scrub starts 2024-09-10T19:20:05.349 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:20:05 smithi183 ceph-mon[29736]: 5.b scrub ok 2024-09-10T19:20:05.349 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:20:05 smithi183 ceph-mon[29736]: pgmap v443: 148 pgs: 148 active+clean; 582 KiB data, 2.3 GiB used, 534 GiB / 536 GiB avail; 639 B/s rd, 0 op/s 2024-09-10T19:20:05.349 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:20:05 smithi183 ceph-mon[29736]: 5.1a scrub starts 2024-09-10T19:20:05.349 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:20:05 smithi183 ceph-mon[29736]: 5.1a scrub ok 2024-09-10T19:20:05.349 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:20:05 smithi183 ceph-mon[29736]: osdmap e234: 8 total, 8 up, 6 in 2024-09-10T19:20:05.350 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:20:05 smithi183 ceph-mon[41562]: 5.b scrub starts 2024-09-10T19:20:05.350 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:20:05 smithi183 ceph-mon[41562]: 5.b scrub ok 2024-09-10T19:20:05.350 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:20:05 smithi183 ceph-mon[41562]: pgmap v443: 148 pgs: 148 active+clean; 582 KiB data, 2.3 GiB used, 534 GiB / 536 GiB avail; 639 B/s rd, 0 op/s 2024-09-10T19:20:05.350 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:20:05 smithi183 ceph-mon[41562]: 5.1a scrub starts 2024-09-10T19:20:05.350 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:20:05 smithi183 ceph-mon[41562]: 5.1a scrub ok 2024-09-10T19:20:05.350 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:20:05 smithi183 ceph-mon[41562]: osdmap e234: 8 total, 8 up, 6 in 2024-09-10T19:20:05.406 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:20:05.440 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:20:05.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:20:05.441 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:20:05.542 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:20:05.579 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:20:05.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:20:05.580 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:20:05.681 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:20:05.707 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:20:05.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:20:05.708 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:20:05.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:20:05 smithi191 ceph-mon[32884]: 5.b scrub starts 2024-09-10T19:20:05.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:20:05 smithi191 ceph-mon[32884]: 5.b scrub ok 2024-09-10T19:20:05.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:20:05 smithi191 ceph-mon[32884]: pgmap v443: 148 pgs: 148 active+clean; 582 KiB data, 2.3 GiB used, 534 GiB / 536 GiB avail; 639 B/s rd, 0 op/s 2024-09-10T19:20:05.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:20:05 smithi191 ceph-mon[32884]: 5.1a scrub starts 2024-09-10T19:20:05.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:20:05 smithi191 ceph-mon[32884]: 5.1a scrub ok 2024-09-10T19:20:05.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:20:05 smithi191 ceph-mon[32884]: osdmap e234: 8 total, 8 up, 6 in 2024-09-10T19:20:05.730 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: 4 16 13861 13845 216.292 191.062 0.00441378 0.00458117 2024-09-10T19:20:05.809 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:20:05.844 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:20:05.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:20:05.845 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:20:05.946 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:20:05.966 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:20:05.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:20:05.967 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:20:06.068 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:20:06.105 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:20:06.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:20:06.106 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:20:06.206 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:20:06.239 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:20:06.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:20:06.240 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:20:06.341 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:20:06.377 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:20:06.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:20:06.378 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:20:06.390 INFO:teuthology.orchestra.run.smithi183.stdout:{ 2024-09-10T19:20:06.390 INFO:teuthology.orchestra.run.smithi183.stdout: "ops": [], 2024-09-10T19:20:06.390 INFO:teuthology.orchestra.run.smithi183.stdout: "num_ops": 0 2024-09-10T19:20:06.390 INFO:teuthology.orchestra.run.smithi183.stdout:} 2024-09-10T19:20:06.479 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:20:06.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:20:06.515 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:20:06.516 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:20:06.574 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:20:06 smithi183 ceph-mon[29736]: 4.e scrub starts 2024-09-10T19:20:06.574 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:20:06 smithi183 ceph-mon[29736]: 4.e scrub ok 2024-09-10T19:20:06.574 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:20:06 smithi183 ceph-mon[29736]: 5.2 scrub starts 2024-09-10T19:20:06.574 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:20:06 smithi183 ceph-mon[29736]: 5.2 scrub ok 2024-09-10T19:20:06.574 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:20:06 smithi183 ceph-mon[29736]: pgmap v445: 148 pgs: 148 active+clean; 729 MiB data, 4.2 GiB used, 532 GiB / 536 GiB avail; 1.6 KiB/s rd, 119 MiB/s wr, 1.91k op/s 2024-09-10T19:20:06.575 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:20:06 smithi183 ceph-mon[41562]: 4.e scrub starts 2024-09-10T19:20:06.575 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:20:06 smithi183 ceph-mon[41562]: 4.e scrub ok 2024-09-10T19:20:06.575 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:20:06 smithi183 ceph-mon[41562]: 5.2 scrub starts 2024-09-10T19:20:06.575 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:20:06 smithi183 ceph-mon[41562]: 5.2 scrub ok 2024-09-10T19:20:06.575 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:20:06 smithi183 ceph-mon[41562]: pgmap v445: 148 pgs: 148 active+clean; 729 MiB data, 4.2 GiB used, 532 GiB / 536 GiB avail; 1.6 KiB/s rd, 119 MiB/s wr, 1.91k op/s 2024-09-10T19:20:06.617 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:20:06.631 INFO:teuthology.orchestra.run.smithi183.stdout:osd.0: {} 2024-09-10T19:20:06.632 INFO:teuthology.orchestra.run.smithi183.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-10T19:20:06.652 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:20:06.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:20:06.653 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:20:06.659 INFO:teuthology.orchestra.run.smithi183.stdout:osd.1: {} 2024-09-10T19:20:06.659 INFO:teuthology.orchestra.run.smithi183.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-10T19:20:06.686 INFO:teuthology.orchestra.run.smithi183.stdout:osd.2: {} 2024-09-10T19:20:06.686 INFO:teuthology.orchestra.run.smithi183.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-10T19:20:06.715 INFO:teuthology.orchestra.run.smithi183.stdout:osd.3: {} 2024-09-10T19:20:06.715 INFO:teuthology.orchestra.run.smithi183.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-10T19:20:06.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:20:06 smithi191 ceph-mon[32884]: 4.e scrub starts 2024-09-10T19:20:06.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:20:06 smithi191 ceph-mon[32884]: 4.e scrub ok 2024-09-10T19:20:06.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:20:06 smithi191 ceph-mon[32884]: 5.2 scrub starts 2024-09-10T19:20:06.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:20:06 smithi191 ceph-mon[32884]: 5.2 scrub ok 2024-09-10T19:20:06.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:20:06 smithi191 ceph-mon[32884]: pgmap v445: 148 pgs: 148 active+clean; 729 MiB data, 4.2 GiB used, 532 GiB / 536 GiB avail; 1.6 KiB/s rd, 119 MiB/s wr, 1.91k op/s 2024-09-10T19:20:06.730 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: 5 16 16649 16633 207.881 174.25 0.00758086 0.00480495 2024-09-10T19:20:06.745 INFO:teuthology.orchestra.run.smithi183.stdout:osd.4: {} 2024-09-10T19:20:06.745 INFO:teuthology.orchestra.run.smithi183.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-10T19:20:06.753 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:20:06.773 INFO:teuthology.orchestra.run.smithi183.stdout:osd.5: {} 2024-09-10T19:20:06.773 INFO:teuthology.orchestra.run.smithi183.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-10T19:20:06.788 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:20:06.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:20:06.789 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:20:06.800 INFO:teuthology.orchestra.run.smithi183.stdout:osd.6: {} 2024-09-10T19:20:06.801 INFO:teuthology.orchestra.run.smithi183.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-10T19:20:06.832 INFO:teuthology.orchestra.run.smithi183.stdout:osd.7: {} 2024-09-10T19:20:06.833 INFO:teuthology.orchestra.run.smithi183.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-10T19:20:06.847 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph osd set noscrub 2024-09-10T19:20:06.890 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:20:06.923 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:20:06.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:20:06.923 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:20:07.024 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:20:07.060 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:20:07.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:20:07.061 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:20:07.162 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:20:07.199 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:20:07.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:20:07.200 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:20:07.300 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:20:07.336 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:20:07.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:20:07.337 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:20:07.438 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:20:07.474 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:20:07.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:20:07.475 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:20:07.576 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:20:07.615 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:20:07.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:20:07.616 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:20:07.681 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:20:07 smithi183 ceph-mon[41562]: 6.e scrub starts 2024-09-10T19:20:07.682 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:20:07 smithi183 ceph-mon[41562]: 6.e scrub ok 2024-09-10T19:20:07.682 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:20:07 smithi183 ceph-mon[41562]: 2.1f scrub starts 2024-09-10T19:20:07.682 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:20:07 smithi183 ceph-mon[41562]: 2.1f scrub ok 2024-09-10T19:20:07.682 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:20:07 smithi183 ceph-mon[41562]: 6.1 scrub starts 2024-09-10T19:20:07.682 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:20:07 smithi183 ceph-mon[41562]: 6.1 scrub ok 2024-09-10T19:20:07.682 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:20:07 smithi183 ceph-mon[41562]: 2.c scrub starts 2024-09-10T19:20:07.682 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:20:07 smithi183 ceph-mon[41562]: 2.c scrub ok 2024-09-10T19:20:07.682 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:20:07 smithi183 ceph-mon[41562]: from='client.? 172.21.15.183:0/3856775519' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T19:20:07.682 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:20:07 smithi183 ceph-mon[29736]: 6.e scrub starts 2024-09-10T19:20:07.682 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:20:07 smithi183 ceph-mon[29736]: 6.e scrub ok 2024-09-10T19:20:07.682 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:20:07 smithi183 ceph-mon[29736]: 2.1f scrub starts 2024-09-10T19:20:07.683 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:20:07 smithi183 ceph-mon[29736]: 2.1f scrub ok 2024-09-10T19:20:07.683 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:20:07 smithi183 ceph-mon[29736]: 6.1 scrub starts 2024-09-10T19:20:07.683 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:20:07 smithi183 ceph-mon[29736]: 6.1 scrub ok 2024-09-10T19:20:07.683 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:20:07 smithi183 ceph-mon[29736]: 2.c scrub starts 2024-09-10T19:20:07.683 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:20:07 smithi183 ceph-mon[29736]: 2.c scrub ok 2024-09-10T19:20:07.683 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:20:07 smithi183 ceph-mon[29736]: from='client.? 172.21.15.183:0/3856775519' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T19:20:07.716 INFO:journalctl@ceph.mgr.x.smithi191.stdout:Sep 10 19:20:07 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-x[34967]: ::ffff:172.21.15.191 - - [10/Sep/2024:19:20:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T19:20:07.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:20:07 smithi191 ceph-mon[32884]: 6.e scrub starts 2024-09-10T19:20:07.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:20:07 smithi191 ceph-mon[32884]: 6.e scrub ok 2024-09-10T19:20:07.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:20:07 smithi191 ceph-mon[32884]: 2.1f scrub starts 2024-09-10T19:20:07.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:20:07 smithi191 ceph-mon[32884]: 2.1f scrub ok 2024-09-10T19:20:07.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:20:07 smithi191 ceph-mon[32884]: 6.1 scrub starts 2024-09-10T19:20:07.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:20:07 smithi191 ceph-mon[32884]: 6.1 scrub ok 2024-09-10T19:20:07.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:20:07 smithi191 ceph-mon[32884]: 2.c scrub starts 2024-09-10T19:20:07.717 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:20:07 smithi191 ceph-mon[32884]: 2.c scrub ok 2024-09-10T19:20:07.717 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:20:07 smithi191 ceph-mon[32884]: from='client.? 172.21.15.183:0/3856775519' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T19:20:07.717 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:20:07.730 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: 6 16 19309 19293 200.94 166.25 0.0157447 0.00496826 2024-09-10T19:20:07.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:20:07.758 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:20:07.758 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:20:07.859 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:20:07.887 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:20:07.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:20:07.888 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:20:07.989 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:20:08.020 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:20:08.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:20:08.021 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:20:08.122 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:20:08.158 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:20:08.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:20:08.159 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:20:08.260 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:20:08.299 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:20:08.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:20:08.300 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:20:08.401 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:20:08.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:20:08 smithi183 ceph-mon[41562]: 2.8 deep-scrub starts 2024-09-10T19:20:08.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:20:08 smithi183 ceph-mon[41562]: 2.8 deep-scrub ok 2024-09-10T19:20:08.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:20:08 smithi183 ceph-mon[41562]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T19:20:08.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:20:08 smithi183 ceph-mon[41562]: 5.1f scrub starts 2024-09-10T19:20:08.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:20:08 smithi183 ceph-mon[41562]: 5.1f scrub ok 2024-09-10T19:20:08.433 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:20:08 smithi183 ceph-mon[41562]: pgmap v446: 148 pgs: 148 active+clean; 729 MiB data, 4.2 GiB used, 532 GiB / 536 GiB avail; 1.2 KiB/s rd, 91 MiB/s wr, 1.46k op/s 2024-09-10T19:20:08.433 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:20:08 smithi183 ceph-mon[29736]: 2.8 deep-scrub starts 2024-09-10T19:20:08.433 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:20:08 smithi183 ceph-mon[29736]: 2.8 deep-scrub ok 2024-09-10T19:20:08.433 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:20:08 smithi183 ceph-mon[29736]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T19:20:08.433 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:20:08 smithi183 ceph-mon[29736]: 5.1f scrub starts 2024-09-10T19:20:08.433 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:20:08 smithi183 ceph-mon[29736]: 5.1f scrub ok 2024-09-10T19:20:08.433 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:20:08 smithi183 ceph-mon[29736]: pgmap v446: 148 pgs: 148 active+clean; 729 MiB data, 4.2 GiB used, 532 GiB / 536 GiB avail; 1.2 KiB/s rd, 91 MiB/s wr, 1.46k op/s 2024-09-10T19:20:08.437 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:20:08.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:20:08.438 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:20:08.538 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:20:08.549 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:20:08 smithi191 ceph-mon[32884]: 2.8 deep-scrub starts 2024-09-10T19:20:08.549 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:20:08 smithi191 ceph-mon[32884]: 2.8 deep-scrub ok 2024-09-10T19:20:08.550 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:20:08 smithi191 ceph-mon[32884]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T19:20:08.550 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:20:08 smithi191 ceph-mon[32884]: 5.1f scrub starts 2024-09-10T19:20:08.550 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:20:08 smithi191 ceph-mon[32884]: 5.1f scrub ok 2024-09-10T19:20:08.550 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:20:08 smithi191 ceph-mon[32884]: pgmap v446: 148 pgs: 148 active+clean; 729 MiB data, 4.2 GiB used, 532 GiB / 536 GiB avail; 1.2 KiB/s rd, 91 MiB/s wr, 1.46k op/s 2024-09-10T19:20:08.570 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:20:08.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:20:08.571 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:20:08.672 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:20:08.706 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:20:08.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:20:08.707 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:20:08.730 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: 7 15 21797 21782 194.456 155.562 0.00560628 0.00513753 2024-09-10T19:20:08.769 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph daemon osd.1 dump_blocked_ops 2024-09-10T19:20:08.808 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:20:08.843 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:20:08.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:20:08.844 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:20:08.945 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:20:08.983 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:20:08.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:20:08.984 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:20:09.039 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:20:09.085 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:20:09.127 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:20:09.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:20:09.128 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:20:09.228 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:20:09.265 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:20:09.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:20:09.266 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:20:09.320 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:20:09 smithi183 ceph-mon[29736]: 6.14 scrub starts 2024-09-10T19:20:09.320 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:20:09 smithi183 ceph-mon[29736]: 6.14 scrub ok 2024-09-10T19:20:09.321 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:20:09 smithi183 ceph-mon[29736]: 6.10 scrub starts 2024-09-10T19:20:09.321 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:20:09 smithi183 ceph-mon[29736]: 6.10 scrub ok 2024-09-10T19:20:09.321 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:20:09 smithi183 ceph-mon[29736]: purged_snaps scrub starts 2024-09-10T19:20:09.321 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:20:09 smithi183 ceph-mon[29736]: purged_snaps scrub ok 2024-09-10T19:20:09.321 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:20:09 smithi183 ceph-mon[29736]: osdmap e235: 8 total, 8 up, 6 in 2024-09-10T19:20:09.321 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:20:09 smithi183 ceph-mon[29736]: 2.17 scrub starts 2024-09-10T19:20:09.321 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:20:09 smithi183 ceph-mon[29736]: 2.17 scrub ok 2024-09-10T19:20:09.321 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:20:09 smithi183 ceph-mon[29736]: 2.e scrub starts 2024-09-10T19:20:09.321 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:20:09 smithi183 ceph-mon[29736]: 2.e scrub ok 2024-09-10T19:20:09.321 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:20:09 smithi183 ceph-mon[41562]: 6.14 scrub starts 2024-09-10T19:20:09.322 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:20:09 smithi183 ceph-mon[41562]: 6.14 scrub ok 2024-09-10T19:20:09.322 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:20:09 smithi183 ceph-mon[41562]: 6.10 scrub starts 2024-09-10T19:20:09.322 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:20:09 smithi183 ceph-mon[41562]: 6.10 scrub ok 2024-09-10T19:20:09.322 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:20:09 smithi183 ceph-mon[41562]: purged_snaps scrub starts 2024-09-10T19:20:09.322 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:20:09 smithi183 ceph-mon[41562]: purged_snaps scrub ok 2024-09-10T19:20:09.322 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:20:09 smithi183 ceph-mon[41562]: osdmap e235: 8 total, 8 up, 6 in 2024-09-10T19:20:09.322 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:20:09 smithi183 ceph-mon[41562]: 2.17 scrub starts 2024-09-10T19:20:09.322 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:20:09 smithi183 ceph-mon[41562]: 2.17 scrub ok 2024-09-10T19:20:09.322 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:20:09 smithi183 ceph-mon[41562]: 2.e scrub starts 2024-09-10T19:20:09.322 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:20:09 smithi183 ceph-mon[41562]: 2.e scrub ok 2024-09-10T19:20:09.367 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:20:09.400 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:20:09.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:20:09.401 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:20:09.502 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:20:09.538 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:20:09.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:20:09.539 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:20:09.640 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:20:09.665 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-09-10T19:20:09.668 INFO:teuthology.orchestra.run.smithi183.stderr:dumped all 2024-09-10T19:20:09.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:20:09.675 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:20:09.675 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:20:09.716 INFO:journalctl@ceph.iscsi.iscsi.a.smithi191.stdout:Sep 10 19:20:09 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-iscsi-iscsi-a[61816]: debug there is no tcmu-runner data available 2024-09-10T19:20:09.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:20:09 smithi191 ceph-mon[32884]: 6.14 scrub starts 2024-09-10T19:20:09.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:20:09 smithi191 ceph-mon[32884]: 6.14 scrub ok 2024-09-10T19:20:09.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:20:09 smithi191 ceph-mon[32884]: 6.10 scrub starts 2024-09-10T19:20:09.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:20:09 smithi191 ceph-mon[32884]: 6.10 scrub ok 2024-09-10T19:20:09.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:20:09 smithi191 ceph-mon[32884]: purged_snaps scrub starts 2024-09-10T19:20:09.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:20:09 smithi191 ceph-mon[32884]: purged_snaps scrub ok 2024-09-10T19:20:09.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:20:09 smithi191 ceph-mon[32884]: osdmap e235: 8 total, 8 up, 6 in 2024-09-10T19:20:09.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:20:09 smithi191 ceph-mon[32884]: 2.17 scrub starts 2024-09-10T19:20:09.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:20:09 smithi191 ceph-mon[32884]: 2.17 scrub ok 2024-09-10T19:20:09.717 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:20:09 smithi191 ceph-mon[32884]: 2.e scrub starts 2024-09-10T19:20:09.717 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:20:09 smithi191 ceph-mon[32884]: 2.e scrub ok 2024-09-10T19:20:09.730 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: 8 16 24284 24268 189.57 155.375 0.00531162 0.00527024 2024-09-10T19:20:09.776 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:20:09.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:20:09.826 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:20:09.827 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:20:09.927 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:20:09.957 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:20:09.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:20:09.958 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:20:10.058 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:20:10.094 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:20:10.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:20:10.095 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:20:10.150 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-10T19:20:10.196 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:20:10.229 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:20:10.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:20:10.231 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:20:10.331 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:20:10.366 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:20:10.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:20:10.367 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:20:10.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:20:10 smithi183 ceph-mon[29736]: 5.1d scrub starts 2024-09-10T19:20:10.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:20:10 smithi183 ceph-mon[29736]: 5.1d scrub ok 2024-09-10T19:20:10.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:20:10 smithi183 ceph-mon[29736]: 6.9 scrub starts 2024-09-10T19:20:10.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:20:10 smithi183 ceph-mon[29736]: 6.9 scrub ok 2024-09-10T19:20:10.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:20:10 smithi183 ceph-mon[29736]: from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:20:10.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:20:10 smithi183 ceph-mon[29736]: 6.b scrub starts 2024-09-10T19:20:10.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:20:10 smithi183 ceph-mon[29736]: 6.b scrub ok 2024-09-10T19:20:10.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:20:10 smithi183 ceph-mon[29736]: from='client.15378 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-10T19:20:10.433 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:20:10 smithi183 ceph-mon[29736]: pgmap v448: 148 pgs: 148 active+clean; 729 MiB data, 4.2 GiB used, 532 GiB / 536 GiB avail; 780 B/s rd, 111 MiB/s wr, 1.77k op/s 2024-09-10T19:20:10.433 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:20:10 smithi183 ceph-mon[29736]: osdmap e236: 8 total, 8 up, 6 in 2024-09-10T19:20:10.433 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:20:10 smithi183 ceph-mon[29736]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T19:20:10.433 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:20:10 smithi183 ceph-mon[29736]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T19:20:10.433 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:20:10 smithi183 ceph-mon[29736]: from='mgr.14556 ' entity='mgr.y' 2024-09-10T19:20:10.433 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:20:10 smithi183 ceph-mon[41562]: 5.1d scrub starts 2024-09-10T19:20:10.433 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:20:10 smithi183 ceph-mon[41562]: 5.1d scrub ok 2024-09-10T19:20:10.433 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:20:10 smithi183 ceph-mon[41562]: 6.9 scrub starts 2024-09-10T19:20:10.433 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:20:10 smithi183 ceph-mon[41562]: 6.9 scrub ok 2024-09-10T19:20:10.433 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:20:10 smithi183 ceph-mon[41562]: from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:20:10.433 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:20:10 smithi183 ceph-mon[41562]: 6.b scrub starts 2024-09-10T19:20:10.434 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:20:10 smithi183 ceph-mon[41562]: 6.b scrub ok 2024-09-10T19:20:10.434 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:20:10 smithi183 ceph-mon[41562]: from='client.15378 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-10T19:20:10.434 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:20:10 smithi183 ceph-mon[41562]: pgmap v448: 148 pgs: 148 active+clean; 729 MiB data, 4.2 GiB used, 532 GiB / 536 GiB avail; 780 B/s rd, 111 MiB/s wr, 1.77k op/s 2024-09-10T19:20:10.434 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:20:10 smithi183 ceph-mon[41562]: osdmap e236: 8 total, 8 up, 6 in 2024-09-10T19:20:10.434 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:20:10 smithi183 ceph-mon[41562]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T19:20:10.434 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:20:10 smithi183 ceph-mon[41562]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T19:20:10.434 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:20:10 smithi183 ceph-mon[41562]: from='mgr.14556 ' entity='mgr.y' 2024-09-10T19:20:10.467 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:20:10.505 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:20:10.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:20:10.506 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:20:10.607 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:20:10.643 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:20:10.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:20:10.644 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:20:10.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:20:10 smithi191 ceph-mon[32884]: 5.1d scrub starts 2024-09-10T19:20:10.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:20:10 smithi191 ceph-mon[32884]: 5.1d scrub ok 2024-09-10T19:20:10.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:20:10 smithi191 ceph-mon[32884]: 6.9 scrub starts 2024-09-10T19:20:10.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:20:10 smithi191 ceph-mon[32884]: 6.9 scrub ok 2024-09-10T19:20:10.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:20:10 smithi191 ceph-mon[32884]: from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:20:10.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:20:10 smithi191 ceph-mon[32884]: 6.b scrub starts 2024-09-10T19:20:10.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:20:10 smithi191 ceph-mon[32884]: 6.b scrub ok 2024-09-10T19:20:10.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:20:10 smithi191 ceph-mon[32884]: from='client.15378 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-10T19:20:10.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:20:10 smithi191 ceph-mon[32884]: pgmap v448: 148 pgs: 148 active+clean; 729 MiB data, 4.2 GiB used, 532 GiB / 536 GiB avail; 780 B/s rd, 111 MiB/s wr, 1.77k op/s 2024-09-10T19:20:10.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:20:10 smithi191 ceph-mon[32884]: osdmap e236: 8 total, 8 up, 6 in 2024-09-10T19:20:10.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:20:10 smithi191 ceph-mon[32884]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T19:20:10.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:20:10 smithi191 ceph-mon[32884]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T19:20:10.717 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:20:10 smithi191 ceph-mon[32884]: from='mgr.14556 ' entity='mgr.y' 2024-09-10T19:20:10.730 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: 9 16 26656 26640 184.978 148.25 0.00540294 0.00539966 2024-09-10T19:20:10.746 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:20:10.781 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:20:10.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:20:10.782 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:20:10.882 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:20:10.913 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:20:10.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:20:10.914 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:20:11.015 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:20:11.050 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:20:11.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:20:11.051 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:20:11.146 INFO:teuthology.orchestra.run.smithi183.stdout:{"pg_ready":true,"pg_map":{"version":447,"stamp":"2024-09-10T19:20:08.261335+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":764221660,"num_objects":11848,"num_object_clones":0,"num_object_copies":35544,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11848,"num_whiteouts":0,"num_read":1074,"num_read_kb":931,"num_write":12195,"num_write_kb":747399,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":182,"num_bytes_recovered":594357,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":12280,"ondisk_log_size":12280,"up":444,"acting":444,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":444,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":562470912,"kb_used":4393960,"kb_used_data":1987304,"kb_used_omap":9,"kb_used_meta":2406582,"kb_avail":558076952,"statfs":{"total":575970213888,"available":571470798848,"internally_reserved":0,"allocated":2034999296,"data_stored":2032469260,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":10224,"internal_metadata":2464339984},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[13,6,3],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":763625472,"num_objects":11652,"num_object_clones":0,"num_object_copies":34956,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11652,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":11652,"num_write_kb":745728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.483192"},"pg_stats":[{"pgid":"6.1b","version":"0'0","reported_seq":208,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-10T19:20:03.220509+0000","last_change":"2024-09-10T19:19:09.088516+0000","last_active":"2024-09-10T19:20:03.220509+0000","last_peered":"2024-09-10T19:20:03.220509+0000","last_clean":"2024-09-10T19:20:03.220509+0000","last_became_active":"2024-09-10T19:10:14.056187+0000","last_became_peered":"2024-09-10T19:10:14.056187+0000","last_unstale":"2024-09-10T19:20:03.220509+0000","last_undegraded":"2024-09-10T19:20:03.220509+0000","last_fullsized":"2024-09-10T19:20:03.220509+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:19:09.088433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:19:09.088433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:20:17.425778+0000","scrub_duration":0.0029670339999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":233,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-10T19:20:03.226472+0000","last_change":"2024-09-10T19:16:38.194521+0000","last_active":"2024-09-10T19:20:03.226472+0000","last_peered":"2024-09-10T19:20:03.226472+0000","last_clean":"2024-09-10T19:20:03.226472+0000","last_became_active":"2024-09-10T19:16:24.836961+0000","last_became_peered":"2024-09-10T19:16:24.836961+0000","last_unstale":"2024-09-10T19:20:03.226472+0000","last_undegraded":"2024-09-10T19:20:03.226472+0000","last_fullsized":"2024-09-10T19:20:03.226472+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:16:38.194472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:16:38.194472+0000","last_clean_scrub_stamp":"2024-09-10T19:16:38.194472+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0023678900000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.19","version":"66'15","reported_seq":263,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-10T19:20:03.221743+0000","last_change":"2024-09-10T19:19:21.244588+0000","last_active":"2024-09-10T19:20:03.221743+0000","last_peered":"2024-09-10T19:20:03.221743+0000","last_clean":"2024-09-10T19:20:03.221743+0000","last_became_active":"2024-09-10T19:16:24.844951+0000","last_became_peered":"2024-09-10T19:16:24.844951+0000","last_unstale":"2024-09-10T19:20:03.221743+0000","last_undegraded":"2024-09-10T19:20:03.221743+0000","last_fullsized":"2024-09-10T19:20:03.221743+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'15","last_scrub_stamp":"2024-09-10T19:19:21.244506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:19:21.244506+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:20:40.860636+0000","scrub_duration":0.014405133000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":242,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-10T19:20:03.225910+0000","last_change":"2024-09-10T19:19:28.498635+0000","last_active":"2024-09-10T19:20:03.225910+0000","last_peered":"2024-09-10T19:20:03.225910+0000","last_clean":"2024-09-10T19:20:03.225910+0000","last_became_active":"2024-09-10T19:16:24.845111+0000","last_became_peered":"2024-09-10T19:16:24.845111+0000","last_unstale":"2024-09-10T19:20:03.225910+0000","last_undegraded":"2024-09-10T19:20:03.225910+0000","last_fullsized":"2024-09-10T19:20:03.225910+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:19:28.498560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:19:28.498560+0000","last_clean_scrub_stamp":"2024-09-10T19:19:28.498560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:20:53.150820+0000","scrub_duration":0.0071873639999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,7],"acting":[4,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"59'1","reported_seq":222,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-10T19:20:03.220775+0000","last_change":"2024-09-10T19:16:30.798883+0000","last_active":"2024-09-10T19:20:03.220775+0000","last_peered":"2024-09-10T19:20:03.220775+0000","last_clean":"2024-09-10T19:20:03.220775+0000","last_became_active":"2024-09-10T19:10:07.686143+0000","last_became_peered":"2024-09-10T19:10:07.686143+0000","last_unstale":"2024-09-10T19:20:03.220775+0000","last_undegraded":"2024-09-10T19:20:03.220775+0000","last_fullsized":"2024-09-10T19:20:03.220775+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'1","last_scrub_stamp":"2024-09-10T19:16:30.798795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:16:30.798795+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0067002989999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":874,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"66'9","reported_seq":257,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-10T19:20:03.225996+0000","last_change":"2024-09-10T19:19:24.417823+0000","last_active":"2024-09-10T19:20:03.225996+0000","last_peered":"2024-09-10T19:20:03.225996+0000","last_clean":"2024-09-10T19:20:03.225996+0000","last_became_active":"2024-09-10T19:13:54.716117+0000","last_became_peered":"2024-09-10T19:13:54.716117+0000","last_unstale":"2024-09-10T19:20:03.225996+0000","last_undegraded":"2024-09-10T19:20:03.225996+0000","last_fullsized":"2024-09-10T19:20:03.225996+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'9","last_scrub_stamp":"2024-09-10T19:19:24.417719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:19:24.417719+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:20:51.013137+0000","scrub_duration":0.023031233000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,3],"acting":[4,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":174,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-10T19:20:03.221588+0000","last_change":"2024-09-10T19:16:27.753997+0000","last_active":"2024-09-10T19:20:03.221588+0000","last_peered":"2024-09-10T19:20:03.221588+0000","last_clean":"2024-09-10T19:20:03.221588+0000","last_became_active":"2024-09-10T19:16:24.859261+0000","last_became_peered":"2024-09-10T19:16:24.859261+0000","last_unstale":"2024-09-10T19:20:03.221588+0000","last_undegraded":"2024-09-10T19:20:03.221588+0000","last_fullsized":"2024-09-10T19:20:03.221588+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:16:27.753912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:16:27.753912+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0029373799999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":230,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-10T19:20:03.226639+0000","last_change":"2024-09-10T19:19:55.380684+0000","last_active":"2024-09-10T19:20:03.226639+0000","last_peered":"2024-09-10T19:20:03.226639+0000","last_clean":"2024-09-10T19:20:03.226639+0000","last_became_active":"2024-09-10T19:13:54.713538+0000","last_became_peered":"2024-09-10T19:13:54.713538+0000","last_unstale":"2024-09-10T19:20:03.226639+0000","last_undegraded":"2024-09-10T19:20:03.226639+0000","last_fullsized":"2024-09-10T19:20:03.226639+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:19:55.380628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:19:55.380628+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:20:55.480928+0000","scrub_duration":0.0040373659999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,7],"acting":[4,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":214,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-10T19:20:03.230257+0000","last_change":"2024-09-10T19:16:25.595265+0000","last_active":"2024-09-10T19:20:03.230257+0000","last_peered":"2024-09-10T19:20:03.230257+0000","last_clean":"2024-09-10T19:20:03.230257+0000","last_became_active":"2024-09-10T19:10:07.690728+0000","last_became_peered":"2024-09-10T19:10:07.690728+0000","last_unstale":"2024-09-10T19:20:03.230257+0000","last_undegraded":"2024-09-10T19:20:03.230257+0000","last_fullsized":"2024-09-10T19:20:03.230257+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:16:25.595127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:16:25.595127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.003922633,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1b","version":"66'5","reported_seq":253,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-10T19:20:03.227191+0000","last_change":"2024-09-10T19:16:35.966457+0000","last_active":"2024-09-10T19:20:03.227191+0000","last_peered":"2024-09-10T19:20:03.227191+0000","last_clean":"2024-09-10T19:20:03.227191+0000","last_became_active":"2024-09-10T19:16:24.843708+0000","last_became_peered":"2024-09-10T19:16:24.843708+0000","last_unstale":"2024-09-10T19:20:03.227191+0000","last_undegraded":"2024-09-10T19:20:03.227191+0000","last_fullsized":"2024-09-10T19:20:03.227191+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'5","last_scrub_stamp":"2024-09-10T19:16:35.966402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:16:35.966402+0000","objects_scrubbed":2,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0019325740000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":227,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-10T19:20:03.903685+0000","last_change":"2024-09-10T19:20:03.903615+0000","last_active":"2024-09-10T19:20:03.903685+0000","last_peered":"2024-09-10T19:20:03.903685+0000","last_clean":"2024-09-10T19:20:03.903685+0000","last_became_active":"2024-09-10T19:13:54.719063+0000","last_became_peered":"2024-09-10T19:13:54.719063+0000","last_unstale":"2024-09-10T19:20:03.903685+0000","last_undegraded":"2024-09-10T19:20:03.903685+0000","last_fullsized":"2024-09-10T19:20:03.903685+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:20:03.903546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:20:03.903546+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:21:23.662091+0000","scrub_duration":0.0066943719999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":225,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-10T19:20:03.221662+0000","last_change":"2024-09-10T19:16:24.845721+0000","last_active":"2024-09-10T19:20:03.221662+0000","last_peered":"2024-09-10T19:20:03.221662+0000","last_clean":"2024-09-10T19:20:03.221662+0000","last_became_active":"2024-09-10T19:16:24.845130+0000","last_became_peered":"2024-09-10T19:16:24.845130+0000","last_unstale":"2024-09-10T19:20:03.221662+0000","last_undegraded":"2024-09-10T19:20:03.221662+0000","last_fullsized":"2024-09-10T19:20:03.221662+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:15:45.616184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:15:45.616184+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0013989510000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,0],"acting":[5,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":224,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-10T19:20:03.224862+0000","last_change":"2024-09-10T19:19:08.062655+0000","last_active":"2024-09-10T19:20:03.224862+0000","last_peered":"2024-09-10T19:20:03.224862+0000","last_clean":"2024-09-10T19:20:03.224862+0000","last_became_active":"2024-09-10T19:16:24.858800+0000","last_became_peered":"2024-09-10T19:16:24.858800+0000","last_unstale":"2024-09-10T19:20:03.224862+0000","last_undegraded":"2024-09-10T19:20:03.224862+0000","last_fullsized":"2024-09-10T19:20:03.224862+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:19:08.062584+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:19:08.062584+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:20:28.059488+0000","scrub_duration":0.0065614130000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1a","version":"66'9","reported_seq":230,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-10T19:20:03.226969+0000","last_change":"2024-09-10T19:19:13.519469+0000","last_active":"2024-09-10T19:20:03.226969+0000","last_peered":"2024-09-10T19:20:03.226969+0000","last_clean":"2024-09-10T19:20:03.226969+0000","last_became_active":"2024-09-10T19:10:09.688987+0000","last_became_peered":"2024-09-10T19:10:09.688987+0000","last_unstale":"2024-09-10T19:20:03.226969+0000","last_undegraded":"2024-09-10T19:20:03.226969+0000","last_fullsized":"2024-09-10T19:20:03.226969+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'9","last_scrub_stamp":"2024-09-10T19:19:13.519390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:19:13.519390+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:20:22.967434+0000","scrub_duration":0.022527794,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":210,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-10T19:20:03.221662+0000","last_change":"2024-09-10T19:16:37.116464+0000","last_active":"2024-09-10T19:20:03.221662+0000","last_peered":"2024-09-10T19:20:03.221662+0000","last_clean":"2024-09-10T19:20:03.221662+0000","last_became_active":"2024-09-10T19:10:11.701799+0000","last_became_peered":"2024-09-10T19:10:11.701799+0000","last_unstale":"2024-09-10T19:20:03.221662+0000","last_undegraded":"2024-09-10T19:20:03.221662+0000","last_fullsized":"2024-09-10T19:20:03.221662+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:16:37.116371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:16:37.116371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.028100301000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":240,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-10T19:20:03.225038+0000","last_change":"2024-09-10T19:19:27.544760+0000","last_active":"2024-09-10T19:20:03.225038+0000","last_peered":"2024-09-10T19:20:03.225038+0000","last_clean":"2024-09-10T19:20:03.225038+0000","last_became_active":"2024-09-10T19:16:24.836477+0000","last_became_peered":"2024-09-10T19:16:24.836477+0000","last_unstale":"2024-09-10T19:20:03.225038+0000","last_undegraded":"2024-09-10T19:20:03.225038+0000","last_fullsized":"2024-09-10T19:20:03.225038+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:19:27.544704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:19:27.544704+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:20:48.302335+0000","scrub_duration":0.010150773,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.0","version":"234'685","reported_seq":698,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-09-10T19:20:04.803256+0000","last_change":"2024-09-10T19:19:54.202041+0000","last_active":"2024-09-10T19:20:04.803256+0000","last_peered":"2024-09-10T19:20:04.803256+0000","last_clean":"2024-09-10T19:20:04.803256+0000","last_became_active":"2024-09-10T19:19:54.200937+0000","last_became_peered":"2024-09-10T19:19:54.200937+0000","last_unstale":"2024-09-10T19:20:04.803256+0000","last_undegraded":"2024-09-10T19:20:04.803256+0000","last_fullsized":"2024-09-10T19:20:04.803256+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:19:53.174811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:19:53.174811+0000","last_clean_scrub_stamp":"2024-09-10T19:19:53.174811+0000","objects_scrubbed":0,"log_size":685,"log_dups_size":0,"ondisk_log_size":685,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:21:17.361255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":44892160,"num_objects":685,"num_object_clones":0,"num_object_copies":2055,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":685,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":685,"num_write_kb":43840,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"66'17","reported_seq":302,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-10T19:20:03.228786+0000","last_change":"2024-09-10T19:16:32.999033+0000","last_active":"2024-09-10T19:20:03.228786+0000","last_peered":"2024-09-10T19:20:03.228786+0000","last_clean":"2024-09-10T19:20:03.228786+0000","last_became_active":"2024-09-10T19:16:24.847056+0000","last_became_peered":"2024-09-10T19:16:24.847056+0000","last_unstale":"2024-09-10T19:20:03.228786+0000","last_undegraded":"2024-09-10T19:20:03.228786+0000","last_fullsized":"2024-09-10T19:20:03.228786+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'17","last_scrub_stamp":"2024-09-10T19:16:32.998950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:16:32.998950+0000","objects_scrubbed":8,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0095505250000000007,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,0],"acting":[4,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":214,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-10T19:20:03.225866+0000","last_change":"2024-09-10T19:16:39.716870+0000","last_active":"2024-09-10T19:20:03.225866+0000","last_peered":"2024-09-10T19:20:03.225866+0000","last_clean":"2024-09-10T19:20:03.225866+0000","last_became_active":"2024-09-10T19:10:07.691119+0000","last_became_peered":"2024-09-10T19:10:07.691119+0000","last_unstale":"2024-09-10T19:20:03.225866+0000","last_undegraded":"2024-09-10T19:20:03.225866+0000","last_fullsized":"2024-09-10T19:20:03.225866+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:16:39.716674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:16:39.716674+0000","last_clean_scrub_stamp":"2024-09-10T19:16:39.716674+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.043895318000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":167,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-10T19:20:03.226332+0000","last_change":"2024-09-10T19:16:26.999105+0000","last_active":"2024-09-10T19:20:03.226332+0000","last_peered":"2024-09-10T19:20:03.226332+0000","last_clean":"2024-09-10T19:20:03.226332+0000","last_became_active":"2024-09-10T19:13:54.712883+0000","last_became_peered":"2024-09-10T19:13:54.712883+0000","last_unstale":"2024-09-10T19:20:03.226332+0000","last_undegraded":"2024-09-10T19:20:03.226332+0000","last_fullsized":"2024-09-10T19:20:03.226332+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:16:26.999036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:16:26.999036+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0050193119999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":92,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-10T19:20:03.222983+0000","last_change":"2024-09-10T19:16:24.858656+0000","last_active":"2024-09-10T19:20:03.222983+0000","last_peered":"2024-09-10T19:20:03.222983+0000","last_clean":"2024-09-10T19:20:03.222983+0000","last_became_active":"2024-09-10T19:16:24.857843+0000","last_became_peered":"2024-09-10T19:16:24.857843+0000","last_unstale":"2024-09-10T19:20:03.222983+0000","last_undegraded":"2024-09-10T19:20:03.222983+0000","last_fullsized":"2024-09-10T19:20:03.222983+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:15:47.818381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:15:47.818381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.1","version":"234'756","reported_seq":769,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-09-10T19:20:05.042084+0000","last_change":"2024-09-10T19:19:54.198962+0000","last_active":"2024-09-10T19:20:05.042084+0000","last_peered":"2024-09-10T19:20:05.042084+0000","last_clean":"2024-09-10T19:20:05.042084+0000","last_became_active":"2024-09-10T19:19:54.198678+0000","last_became_peered":"2024-09-10T19:19:54.198678+0000","last_unstale":"2024-09-10T19:20:05.042084+0000","last_undegraded":"2024-09-10T19:20:05.042084+0000","last_fullsized":"2024-09-10T19:20:05.042084+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:19:53.174811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:19:53.174811+0000","last_clean_scrub_stamp":"2024-09-10T19:19:53.174811+0000","objects_scrubbed":0,"log_size":756,"log_dups_size":0,"ondisk_log_size":756,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:20:53.785271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49545216,"num_objects":756,"num_object_clones":0,"num_object_copies":2268,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":756,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":756,"num_write_kb":48384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"66'10","reported_seq":239,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-10T19:20:03.226488+0000","last_change":"2024-09-10T19:19:18.488264+0000","last_active":"2024-09-10T19:20:03.226488+0000","last_peered":"2024-09-10T19:20:03.226488+0000","last_clean":"2024-09-10T19:20:03.226488+0000","last_became_active":"2024-09-10T19:10:09.688321+0000","last_became_peered":"2024-09-10T19:10:09.688321+0000","last_unstale":"2024-09-10T19:20:03.226488+0000","last_undegraded":"2024-09-10T19:20:03.226488+0000","last_fullsized":"2024-09-10T19:20:03.226488+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'10","last_scrub_stamp":"2024-09-10T19:19:18.488174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:19:18.488174+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:20:28.041787+0000","scrub_duration":0.0042762770000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":174,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-10T19:20:03.220186+0000","last_change":"2024-09-10T19:19:57.419605+0000","last_active":"2024-09-10T19:20:03.220186+0000","last_peered":"2024-09-10T19:20:03.220186+0000","last_clean":"2024-09-10T19:20:03.220186+0000","last_became_active":"2024-09-10T19:13:54.731134+0000","last_became_peered":"2024-09-10T19:13:54.731134+0000","last_unstale":"2024-09-10T19:20:03.220186+0000","last_undegraded":"2024-09-10T19:20:03.220186+0000","last_fullsized":"2024-09-10T19:20:03.220186+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:19:57.419553+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:19:57.419553+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:21:01.300393+0000","scrub_duration":0.001279428,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,5],"acting":[3,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"155'10","reported_seq":206,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-10T19:20:03.224937+0000","last_change":"2024-09-10T19:16:29.626806+0000","last_active":"2024-09-10T19:20:03.224937+0000","last_peered":"2024-09-10T19:20:03.224937+0000","last_clean":"2024-09-10T19:20:03.224937+0000","last_became_active":"2024-09-10T19:16:24.851236+0000","last_became_peered":"2024-09-10T19:16:24.851236+0000","last_unstale":"2024-09-10T19:20:03.224937+0000","last_undegraded":"2024-09-10T19:20:03.224937+0000","last_fullsized":"2024-09-10T19:20:03.224937+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"155'10","last_scrub_stamp":"2024-09-10T19:16:29.626572+0000","last_deep_scrub":"80'9","last_deep_scrub_stamp":"2024-09-10T19:11:25.156465+0000","last_clean_scrub_stamp":"2024-09-10T19:16:29.626572+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0067064810000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,0],"acting":[7,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":230,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-10T19:20:03.228707+0000","last_change":"2024-09-10T19:16:24.852100+0000","last_active":"2024-09-10T19:20:03.228707+0000","last_peered":"2024-09-10T19:20:03.228707+0000","last_clean":"2024-09-10T19:20:03.228707+0000","last_became_active":"2024-09-10T19:16:24.851829+0000","last_became_peered":"2024-09-10T19:16:24.851829+0000","last_unstale":"2024-09-10T19:20:03.228707+0000","last_undegraded":"2024-09-10T19:20:03.228707+0000","last_fullsized":"2024-09-10T19:20:03.228707+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:15:29.988976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:15:29.988976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0053787150000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,7],"acting":[4,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.2","version":"234'743","reported_seq":755,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-09-10T19:20:05.048450+0000","last_change":"2024-09-10T19:19:54.197588+0000","last_active":"2024-09-10T19:20:05.048450+0000","last_peered":"2024-09-10T19:20:05.048450+0000","last_clean":"2024-09-10T19:20:05.048450+0000","last_became_active":"2024-09-10T19:19:54.197290+0000","last_became_peered":"2024-09-10T19:19:54.197290+0000","last_unstale":"2024-09-10T19:20:05.048450+0000","last_undegraded":"2024-09-10T19:20:05.048450+0000","last_fullsized":"2024-09-10T19:20:05.048450+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:19:53.174811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:19:53.174811+0000","last_clean_scrub_stamp":"2024-09-10T19:19:53.174811+0000","objects_scrubbed":0,"log_size":743,"log_dups_size":0,"ondisk_log_size":743,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:21:10.017724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48693248,"num_objects":743,"num_object_clones":0,"num_object_copies":2229,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":743,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":743,"num_write_kb":47552,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"66'15","reported_seq":217,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-10T19:20:03.223580+0000","last_change":"2024-09-10T19:16:25.769523+0000","last_active":"2024-09-10T19:20:03.223580+0000","last_peered":"2024-09-10T19:20:03.223580+0000","last_clean":"2024-09-10T19:20:03.223580+0000","last_became_active":"2024-09-10T19:13:54.711634+0000","last_became_peered":"2024-09-10T19:13:54.711634+0000","last_unstale":"2024-09-10T19:20:03.223580+0000","last_undegraded":"2024-09-10T19:20:03.223580+0000","last_fullsized":"2024-09-10T19:20:03.223580+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'15","last_scrub_stamp":"2024-09-10T19:16:25.769436+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:16:25.769436+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0046254920000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":184,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-10T19:20:03.224664+0000","last_change":"2024-09-10T19:19:21.019996+0000","last_active":"2024-09-10T19:20:03.224664+0000","last_peered":"2024-09-10T19:20:03.224664+0000","last_clean":"2024-09-10T19:20:03.224664+0000","last_became_active":"2024-09-10T19:16:24.857179+0000","last_became_peered":"2024-09-10T19:16:24.857179+0000","last_unstale":"2024-09-10T19:20:03.224664+0000","last_undegraded":"2024-09-10T19:20:03.224664+0000","last_fullsized":"2024-09-10T19:20:03.224664+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:19:21.019912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:19:21.019912+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:20:30.827923+0000","scrub_duration":0.0041693620000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,6],"acting":[7,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.e","version":"66'8","reported_seq":339,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-10T19:20:03.228868+0000","last_change":"2024-09-10T19:19:17.519606+0000","last_active":"2024-09-10T19:20:03.228868+0000","last_peered":"2024-09-10T19:20:03.228868+0000","last_clean":"2024-09-10T19:20:03.228868+0000","last_became_active":"2024-09-10T19:10:11.699232+0000","last_became_peered":"2024-09-10T19:10:11.699232+0000","last_unstale":"2024-09-10T19:20:03.228868+0000","last_undegraded":"2024-09-10T19:20:03.228868+0000","last_fullsized":"2024-09-10T19:20:03.228868+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'8","last_scrub_stamp":"2024-09-10T19:19:17.519517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:19:17.519517+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:20:40.560476+0000","scrub_duration":0.0049564470000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":225,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-10T19:20:03.222149+0000","last_change":"2024-09-10T19:16:24.855938+0000","last_active":"2024-09-10T19:20:03.222149+0000","last_peered":"2024-09-10T19:20:03.222149+0000","last_clean":"2024-09-10T19:20:03.222149+0000","last_became_active":"2024-09-10T19:16:24.855627+0000","last_became_peered":"2024-09-10T19:16:24.855627+0000","last_unstale":"2024-09-10T19:20:03.222149+0000","last_undegraded":"2024-09-10T19:20:03.222149+0000","last_fullsized":"2024-09-10T19:20:03.222149+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:15:44.633043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:15:44.633043+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.00368805,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,6],"acting":[5,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.3","version":"234'654","reported_seq":666,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-09-10T19:20:04.942090+0000","last_change":"2024-09-10T19:19:54.204109+0000","last_active":"2024-09-10T19:20:04.942090+0000","last_peered":"2024-09-10T19:20:04.942090+0000","last_clean":"2024-09-10T19:20:04.942090+0000","last_became_active":"2024-09-10T19:19:54.203780+0000","last_became_peered":"2024-09-10T19:19:54.203780+0000","last_unstale":"2024-09-10T19:20:04.942090+0000","last_undegraded":"2024-09-10T19:20:04.942090+0000","last_fullsized":"2024-09-10T19:20:04.942090+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:19:53.174811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:19:53.174811+0000","last_clean_scrub_stamp":"2024-09-10T19:19:53.174811+0000","objects_scrubbed":0,"log_size":654,"log_dups_size":0,"ondisk_log_size":654,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:21:13.269247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":42860544,"num_objects":654,"num_object_clones":0,"num_object_copies":1962,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":654,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":654,"num_write_kb":41856,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"84'16","reported_seq":267,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-10T19:20:04.434110+0000","last_change":"2024-09-10T19:20:04.433997+0000","last_active":"2024-09-10T19:20:04.434110+0000","last_peered":"2024-09-10T19:20:04.434110+0000","last_clean":"2024-09-10T19:20:04.434110+0000","last_became_active":"2024-09-10T19:13:54.717022+0000","last_became_peered":"2024-09-10T19:13:54.717022+0000","last_unstale":"2024-09-10T19:20:04.434110+0000","last_undegraded":"2024-09-10T19:20:04.434110+0000","last_fullsized":"2024-09-10T19:20:04.434110+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'16","last_scrub_stamp":"2024-09-10T19:20:04.433889+0000","last_deep_scrub":"84'16","last_deep_scrub_stamp":"2024-09-10T19:11:56.157975+0000","last_clean_scrub_stamp":"2024-09-10T19:20:04.433889+0000","objects_scrubbed":5,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:21:20.286902+0000","scrub_duration":0.0029380399999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,3],"acting":[4,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":221,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-10T19:20:03.224220+0000","last_change":"2024-09-10T19:15:42.450149+0000","last_active":"2024-09-10T19:20:03.224220+0000","last_peered":"2024-09-10T19:20:03.224220+0000","last_clean":"2024-09-10T19:20:03.224220+0000","last_became_active":"2024-09-10T19:13:54.720293+0000","last_became_peered":"2024-09-10T19:13:54.720293+0000","last_unstale":"2024-09-10T19:20:03.224220+0000","last_undegraded":"2024-09-10T19:20:03.224220+0000","last_fullsized":"2024-09-10T19:20:03.224220+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:15:42.450068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:11:53.066849+0000","last_clean_scrub_stamp":"2024-09-10T19:15:42.450068+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0037708360000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":210,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-10T19:20:03.222365+0000","last_change":"2024-09-10T19:19:29.566991+0000","last_active":"2024-09-10T19:20:03.222365+0000","last_peered":"2024-09-10T19:20:03.222365+0000","last_clean":"2024-09-10T19:20:03.222365+0000","last_became_active":"2024-09-10T19:10:11.692215+0000","last_became_peered":"2024-09-10T19:10:11.692215+0000","last_unstale":"2024-09-10T19:20:03.222365+0000","last_undegraded":"2024-09-10T19:20:03.222365+0000","last_fullsized":"2024-09-10T19:20:03.222365+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:19:29.566920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:19:29.566920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:20:49.151502+0000","scrub_duration":0.0083507189999999995,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":208,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-10T19:20:03.221002+0000","last_change":"2024-09-10T19:19:28.321305+0000","last_active":"2024-09-10T19:20:03.221002+0000","last_peered":"2024-09-10T19:20:03.221002+0000","last_clean":"2024-09-10T19:20:03.221002+0000","last_became_active":"2024-09-10T19:10:14.053736+0000","last_became_peered":"2024-09-10T19:10:14.053736+0000","last_unstale":"2024-09-10T19:20:03.221002+0000","last_undegraded":"2024-09-10T19:20:03.221002+0000","last_fullsized":"2024-09-10T19:20:03.221002+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:19:28.321253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:19:28.321253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:20:53.915436+0000","scrub_duration":0.010088694,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.c","version":"234'763","reported_seq":776,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-09-10T19:20:05.051316+0000","last_change":"2024-09-10T19:19:54.200758+0000","last_active":"2024-09-10T19:20:05.051316+0000","last_peered":"2024-09-10T19:20:05.051316+0000","last_clean":"2024-09-10T19:20:05.051316+0000","last_became_active":"2024-09-10T19:19:54.200490+0000","last_became_peered":"2024-09-10T19:19:54.200490+0000","last_unstale":"2024-09-10T19:20:05.051316+0000","last_undegraded":"2024-09-10T19:20:05.051316+0000","last_fullsized":"2024-09-10T19:20:05.051316+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:19:53.174811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:19:53.174811+0000","last_clean_scrub_stamp":"2024-09-10T19:19:53.174811+0000","objects_scrubbed":0,"log_size":763,"log_dups_size":0,"ondisk_log_size":763,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:20:59.429111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":50003968,"num_objects":763,"num_object_clones":0,"num_object_copies":2289,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":763,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":763,"num_write_kb":48832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"66'14","reported_seq":118,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-10T19:20:03.226100+0000","last_change":"2024-09-10T19:19:56.359426+0000","last_active":"2024-09-10T19:20:03.226100+0000","last_peered":"2024-09-10T19:20:03.226100+0000","last_clean":"2024-09-10T19:20:03.226100+0000","last_became_active":"2024-09-10T19:16:24.852821+0000","last_became_peered":"2024-09-10T19:16:24.852821+0000","last_unstale":"2024-09-10T19:20:03.226100+0000","last_undegraded":"2024-09-10T19:20:03.226100+0000","last_fullsized":"2024-09-10T19:20:03.226100+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'14","last_scrub_stamp":"2024-09-10T19:19:56.359371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:19:56.359371+0000","objects_scrubbed":7,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:20:57.053542+0000","scrub_duration":0.0016354869999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,7],"acting":[4,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":224,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-10T19:20:03.226010+0000","last_change":"2024-09-10T19:16:25.582072+0000","last_active":"2024-09-10T19:20:03.226010+0000","last_peered":"2024-09-10T19:20:03.226010+0000","last_clean":"2024-09-10T19:20:03.226010+0000","last_became_active":"2024-09-10T19:16:24.855861+0000","last_became_peered":"2024-09-10T19:16:24.855861+0000","last_unstale":"2024-09-10T19:20:03.226010+0000","last_undegraded":"2024-09-10T19:20:03.226010+0000","last_fullsized":"2024-09-10T19:20:03.226010+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:16:25.581993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:16:25.581993+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0041345990000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.0","version":"117'10","reported_seq":349,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-10T19:20:03.219899+0000","last_change":"2024-09-10T19:15:35.973188+0000","last_active":"2024-09-10T19:20:03.219899+0000","last_peered":"2024-09-10T19:20:03.219899+0000","last_clean":"2024-09-10T19:20:03.219899+0000","last_became_active":"2024-09-10T19:13:54.709846+0000","last_became_peered":"2024-09-10T19:13:54.709846+0000","last_unstale":"2024-09-10T19:20:03.219899+0000","last_undegraded":"2024-09-10T19:20:03.219899+0000","last_fullsized":"2024-09-10T19:20:03.219899+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"117'10","last_scrub_stamp":"2024-09-10T19:15:35.973095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:15:35.973095+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0076338860000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,0],"acting":[3,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":218,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-10T19:20:03.225961+0000","last_change":"2024-09-10T19:19:10.030136+0000","last_active":"2024-09-10T19:20:03.225961+0000","last_peered":"2024-09-10T19:20:03.225961+0000","last_clean":"2024-09-10T19:20:03.225961+0000","last_became_active":"2024-09-10T19:16:24.858619+0000","last_became_peered":"2024-09-10T19:16:24.858619+0000","last_unstale":"2024-09-10T19:20:03.225961+0000","last_undegraded":"2024-09-10T19:20:03.225961+0000","last_fullsized":"2024-09-10T19:20:03.225961+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:19:10.029951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:11:41.073833+0000","last_clean_scrub_stamp":"2024-09-10T19:19:10.029951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:20:30.253633+0000","scrub_duration":0.01007037,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.d","version":"234'728","reported_seq":741,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-09-10T19:20:04.959003+0000","last_change":"2024-09-10T19:19:54.202429+0000","last_active":"2024-09-10T19:20:04.959003+0000","last_peered":"2024-09-10T19:20:04.959003+0000","last_clean":"2024-09-10T19:20:04.959003+0000","last_became_active":"2024-09-10T19:19:54.202133+0000","last_became_peered":"2024-09-10T19:19:54.202133+0000","last_unstale":"2024-09-10T19:20:04.959003+0000","last_undegraded":"2024-09-10T19:20:04.959003+0000","last_fullsized":"2024-09-10T19:20:04.959003+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:19:53.174811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:19:53.174811+0000","last_clean_scrub_stamp":"2024-09-10T19:19:53.174811+0000","objects_scrubbed":0,"log_size":728,"log_dups_size":0,"ondisk_log_size":728,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:21:01.900497+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47710208,"num_objects":728,"num_object_clones":0,"num_object_copies":2184,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":728,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":728,"num_write_kb":46592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,4],"acting":[7,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.0","version":"66'18","reported_seq":244,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-10T19:20:03.220069+0000","last_change":"2024-09-10T19:19:12.207158+0000","last_active":"2024-09-10T19:20:03.220069+0000","last_peered":"2024-09-10T19:20:03.220069+0000","last_clean":"2024-09-10T19:20:03.220069+0000","last_became_active":"2024-09-10T19:10:09.821836+0000","last_became_peered":"2024-09-10T19:10:09.821836+0000","last_unstale":"2024-09-10T19:20:03.220069+0000","last_undegraded":"2024-09-10T19:20:03.220069+0000","last_fullsized":"2024-09-10T19:20:03.220069+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'18","last_scrub_stamp":"2024-09-10T19:19:12.207076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:19:12.207076+0000","objects_scrubbed":8,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:20:20.937740+0000","scrub_duration":0.003631156,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":174,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-10T19:20:03.225177+0000","last_change":"2024-09-10T19:17:26.374435+0000","last_active":"2024-09-10T19:20:03.225177+0000","last_peered":"2024-09-10T19:20:03.225177+0000","last_clean":"2024-09-10T19:20:03.225177+0000","last_became_active":"2024-09-10T19:13:54.710968+0000","last_became_peered":"2024-09-10T19:13:54.710968+0000","last_unstale":"2024-09-10T19:20:03.225177+0000","last_undegraded":"2024-09-10T19:20:03.225177+0000","last_fullsized":"2024-09-10T19:20:03.225177+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:17:26.374376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:17:26.374376+0000","last_clean_scrub_stamp":"2024-09-10T19:17:26.374376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001299815,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,3],"acting":[6,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":209,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-10T19:20:03.226267+0000","last_change":"2024-09-10T19:16:30.991715+0000","last_active":"2024-09-10T19:20:03.226267+0000","last_peered":"2024-09-10T19:20:03.226267+0000","last_clean":"2024-09-10T19:20:03.226267+0000","last_became_active":"2024-09-10T19:10:11.698054+0000","last_became_peered":"2024-09-10T19:10:11.698054+0000","last_unstale":"2024-09-10T19:20:03.226267+0000","last_undegraded":"2024-09-10T19:20:03.226267+0000","last_fullsized":"2024-09-10T19:20:03.226267+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:16:30.991635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:16:30.991635+0000","last_clean_scrub_stamp":"2024-09-10T19:16:30.991635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0021810029999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":217,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-10T19:20:03.226894+0000","last_change":"2024-09-10T19:16:30.036748+0000","last_active":"2024-09-10T19:20:03.226894+0000","last_peered":"2024-09-10T19:20:03.226894+0000","last_clean":"2024-09-10T19:20:03.226894+0000","last_became_active":"2024-09-10T19:16:24.848109+0000","last_became_peered":"2024-09-10T19:16:24.848109+0000","last_unstale":"2024-09-10T19:20:03.226894+0000","last_undegraded":"2024-09-10T19:20:03.226894+0000","last_fullsized":"2024-09-10T19:20:03.226894+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:16:30.036662+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:16:30.036662+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0044984129999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.e","version":"234'746","reported_seq":759,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-09-10T19:20:04.961935+0000","last_change":"2024-09-10T19:19:54.201207+0000","last_active":"2024-09-10T19:20:04.961935+0000","last_peered":"2024-09-10T19:20:04.961935+0000","last_clean":"2024-09-10T19:20:04.961935+0000","last_became_active":"2024-09-10T19:19:54.200931+0000","last_became_peered":"2024-09-10T19:19:54.200931+0000","last_unstale":"2024-09-10T19:20:04.961935+0000","last_undegraded":"2024-09-10T19:20:04.961935+0000","last_fullsized":"2024-09-10T19:20:04.961935+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:19:53.174811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:19:53.174811+0000","last_clean_scrub_stamp":"2024-09-10T19:19:53.174811+0000","objects_scrubbed":0,"log_size":746,"log_dups_size":0,"ondisk_log_size":746,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:21:20.412285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48889856,"num_objects":746,"num_object_clones":0,"num_object_copies":2238,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":746,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":746,"num_write_kb":47744,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"104'27","reported_seq":255,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-10T19:20:03.226131+0000","last_change":"2024-09-10T19:16:35.246566+0000","last_active":"2024-09-10T19:20:03.226131+0000","last_peered":"2024-09-10T19:20:03.226131+0000","last_clean":"2024-09-10T19:20:03.226131+0000","last_became_active":"2024-09-10T19:10:09.823003+0000","last_became_peered":"2024-09-10T19:10:09.823003+0000","last_unstale":"2024-09-10T19:20:03.226131+0000","last_undegraded":"2024-09-10T19:20:03.226131+0000","last_fullsized":"2024-09-10T19:20:03.226131+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"104'27","last_scrub_stamp":"2024-09-10T19:16:35.246512+0000","last_deep_scrub":"104'27","last_deep_scrub_stamp":"2024-09-10T19:13:02.225944+0000","last_clean_scrub_stamp":"2024-09-10T19:16:35.246512+0000","objects_scrubbed":8,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0052296490000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":214,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-10T19:20:03.230327+0000","last_change":"2024-09-10T19:16:37.780086+0000","last_active":"2024-09-10T19:20:03.230327+0000","last_peered":"2024-09-10T19:20:03.230327+0000","last_clean":"2024-09-10T19:20:03.230327+0000","last_became_active":"2024-09-10T19:10:07.690325+0000","last_became_peered":"2024-09-10T19:10:07.690325+0000","last_unstale":"2024-09-10T19:20:03.230327+0000","last_undegraded":"2024-09-10T19:20:03.230327+0000","last_fullsized":"2024-09-10T19:20:03.230327+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:16:37.779967+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:16:37.779967+0000","last_clean_scrub_stamp":"2024-09-10T19:16:37.779967+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.067110102000000005,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":213,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-10T19:20:03.223740+0000","last_change":"2024-09-10T19:17:19.463698+0000","last_active":"2024-09-10T19:20:03.223740+0000","last_peered":"2024-09-10T19:20:03.223740+0000","last_clean":"2024-09-10T19:20:03.223740+0000","last_became_active":"2024-09-10T19:10:11.700370+0000","last_became_peered":"2024-09-10T19:10:11.700370+0000","last_unstale":"2024-09-10T19:20:03.223740+0000","last_undegraded":"2024-09-10T19:20:03.223740+0000","last_fullsized":"2024-09-10T19:20:03.223740+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:15:50.669821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:15:50.669821+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0015765,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":177,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-10T19:20:03.223815+0000","last_change":"2024-09-10T19:17:22.435156+0000","last_active":"2024-09-10T19:20:03.223815+0000","last_peered":"2024-09-10T19:20:03.223815+0000","last_clean":"2024-09-10T19:20:03.223815+0000","last_became_active":"2024-09-10T19:16:24.844699+0000","last_became_peered":"2024-09-10T19:16:24.844699+0000","last_unstale":"2024-09-10T19:20:03.223815+0000","last_undegraded":"2024-09-10T19:20:03.223815+0000","last_fullsized":"2024-09-10T19:20:03.223815+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:15:41.749338+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:15:41.749338+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0018841260000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.f","version":"234'721","reported_seq":734,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-09-10T19:20:05.039222+0000","last_change":"2024-09-10T19:19:54.202581+0000","last_active":"2024-09-10T19:20:05.039222+0000","last_peered":"2024-09-10T19:20:05.039222+0000","last_clean":"2024-09-10T19:20:05.039222+0000","last_became_active":"2024-09-10T19:19:54.202323+0000","last_became_peered":"2024-09-10T19:19:54.202323+0000","last_unstale":"2024-09-10T19:20:05.039222+0000","last_undegraded":"2024-09-10T19:20:05.039222+0000","last_fullsized":"2024-09-10T19:20:05.039222+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:19:53.174811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:19:53.174811+0000","last_clean_scrub_stamp":"2024-09-10T19:19:53.174811+0000","objects_scrubbed":0,"log_size":721,"log_dups_size":0,"ondisk_log_size":721,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:21:21.255294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47251456,"num_objects":721,"num_object_clones":0,"num_object_copies":2163,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":721,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":721,"num_write_kb":46144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"66'10","reported_seq":180,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-10T19:20:03.221909+0000","last_change":"2024-09-10T19:19:26.575854+0000","last_active":"2024-09-10T19:20:03.221909+0000","last_peered":"2024-09-10T19:20:03.221909+0000","last_clean":"2024-09-10T19:20:03.221909+0000","last_became_active":"2024-09-10T19:13:54.714542+0000","last_became_peered":"2024-09-10T19:13:54.714542+0000","last_unstale":"2024-09-10T19:20:03.221909+0000","last_undegraded":"2024-09-10T19:20:03.221909+0000","last_fullsized":"2024-09-10T19:20:03.221909+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'10","last_scrub_stamp":"2024-09-10T19:19:26.575769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:19:26.575769+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:20:41.943835+0000","scrub_duration":0.022703509,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":14,"num_read_kb":9,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,0],"acting":[5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":183,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-10T19:20:03.226019+0000","last_change":"2024-09-10T19:17:28.203449+0000","last_active":"2024-09-10T19:20:03.226019+0000","last_peered":"2024-09-10T19:20:03.226019+0000","last_clean":"2024-09-10T19:20:03.226019+0000","last_became_active":"2024-09-10T19:16:24.836898+0000","last_became_peered":"2024-09-10T19:16:24.836898+0000","last_unstale":"2024-09-10T19:20:03.226019+0000","last_undegraded":"2024-09-10T19:20:03.226019+0000","last_fullsized":"2024-09-10T19:20:03.226019+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:17:28.203397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:17:28.203397+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0011258259999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3","version":"84'9","reported_seq":343,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-10T19:20:03.225904+0000","last_change":"2024-09-10T19:19:17.425038+0000","last_active":"2024-09-10T19:20:03.225904+0000","last_peered":"2024-09-10T19:20:03.225904+0000","last_clean":"2024-09-10T19:20:03.225904+0000","last_became_active":"2024-09-10T19:10:11.699690+0000","last_became_peered":"2024-09-10T19:10:11.699690+0000","last_unstale":"2024-09-10T19:20:03.225904+0000","last_undegraded":"2024-09-10T19:20:03.225904+0000","last_fullsized":"2024-09-10T19:20:03.225904+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'9","last_scrub_stamp":"2024-09-10T19:19:17.424943+0000","last_deep_scrub":"84'9","last_deep_scrub_stamp":"2024-09-10T19:11:53.879272+0000","last_clean_scrub_stamp":"2024-09-10T19:19:17.424943+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:20:32.534406+0000","scrub_duration":0.0083821910000000006,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":223,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-10T19:20:03.225189+0000","last_change":"2024-09-10T19:19:09.424372+0000","last_active":"2024-09-10T19:20:03.225189+0000","last_peered":"2024-09-10T19:20:03.225189+0000","last_clean":"2024-09-10T19:20:03.225189+0000","last_became_active":"2024-09-10T19:16:24.840096+0000","last_became_peered":"2024-09-10T19:16:24.840096+0000","last_unstale":"2024-09-10T19:20:03.225189+0000","last_undegraded":"2024-09-10T19:20:03.225189+0000","last_fullsized":"2024-09-10T19:20:03.225189+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:19:09.424289+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:19:09.424289+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:20:28.425816+0000","scrub_duration":0.001622833,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,5],"acting":[0,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.9","version":"234'723","reported_seq":735,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-09-10T19:20:05.032801+0000","last_change":"2024-09-10T19:19:54.200843+0000","last_active":"2024-09-10T19:20:05.032801+0000","last_peered":"2024-09-10T19:20:05.032801+0000","last_clean":"2024-09-10T19:20:05.032801+0000","last_became_active":"2024-09-10T19:19:54.200559+0000","last_became_peered":"2024-09-10T19:19:54.200559+0000","last_unstale":"2024-09-10T19:20:05.032801+0000","last_undegraded":"2024-09-10T19:20:05.032801+0000","last_fullsized":"2024-09-10T19:20:05.032801+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:19:53.174811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:19:53.174811+0000","last_clean_scrub_stamp":"2024-09-10T19:19:53.174811+0000","objects_scrubbed":0,"log_size":723,"log_dups_size":0,"ondisk_log_size":723,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:21:02.539917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47382528,"num_objects":723,"num_object_clones":0,"num_object_copies":2169,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":723,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":723,"num_write_kb":46272,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,5],"acting":[0,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"155'31","reported_seq":219,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-10T19:20:03.220597+0000","last_change":"2024-09-10T19:16:24.910865+0000","last_active":"2024-09-10T19:20:03.220597+0000","last_peered":"2024-09-10T19:20:03.220597+0000","last_clean":"2024-09-10T19:20:03.220597+0000","last_became_active":"2024-09-10T19:16:24.858154+0000","last_became_peered":"2024-09-10T19:16:24.858154+0000","last_unstale":"2024-09-10T19:20:03.220597+0000","last_undegraded":"2024-09-10T19:20:03.220597+0000","last_fullsized":"2024-09-10T19:20:03.220597+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"117'30","last_scrub_stamp":"2024-09-10T19:15:49.837844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:15:49.837844+0000","objects_scrubbed":0,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":47,"num_read_kb":32,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":358,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":221,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-10T19:20:03.221486+0000","last_change":"2024-09-10T19:15:54.746979+0000","last_active":"2024-09-10T19:20:03.221486+0000","last_peered":"2024-09-10T19:20:03.221486+0000","last_clean":"2024-09-10T19:20:03.221486+0000","last_became_active":"2024-09-10T19:13:54.730096+0000","last_became_peered":"2024-09-10T19:13:54.730096+0000","last_unstale":"2024-09-10T19:20:03.221486+0000","last_undegraded":"2024-09-10T19:20:03.221486+0000","last_fullsized":"2024-09-10T19:20:03.221486+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:15:54.746920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:15:54.746920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001300037,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":226,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-10T19:20:03.226191+0000","last_change":"2024-09-10T19:17:25.147473+0000","last_active":"2024-09-10T19:20:03.226191+0000","last_peered":"2024-09-10T19:20:03.226191+0000","last_clean":"2024-09-10T19:20:03.226191+0000","last_became_active":"2024-09-10T19:13:54.714017+0000","last_became_peered":"2024-09-10T19:13:54.714017+0000","last_unstale":"2024-09-10T19:20:03.226191+0000","last_undegraded":"2024-09-10T19:20:03.226191+0000","last_fullsized":"2024-09-10T19:20:03.226191+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:17:25.147396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:17:25.147396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.004390054,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,6],"acting":[0,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":198,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-10T19:20:03.220606+0000","last_change":"2024-09-10T19:16:32.818668+0000","last_active":"2024-09-10T19:20:03.220606+0000","last_peered":"2024-09-10T19:20:03.220606+0000","last_clean":"2024-09-10T19:20:03.220606+0000","last_became_active":"2024-09-10T19:10:13.721744+0000","last_became_peered":"2024-09-10T19:10:13.721744+0000","last_unstale":"2024-09-10T19:20:03.220606+0000","last_undegraded":"2024-09-10T19:20:03.220606+0000","last_fullsized":"2024-09-10T19:20:03.220606+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:16:32.818590+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:16:32.818590+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0047149059999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.a","version":"234'696","reported_seq":707,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-09-10T19:20:05.054994+0000","last_change":"2024-09-10T19:19:54.202692+0000","last_active":"2024-09-10T19:20:05.054994+0000","last_peered":"2024-09-10T19:20:05.054994+0000","last_clean":"2024-09-10T19:20:05.054994+0000","last_became_active":"2024-09-10T19:19:54.202444+0000","last_became_peered":"2024-09-10T19:19:54.202444+0000","last_unstale":"2024-09-10T19:20:05.054994+0000","last_undegraded":"2024-09-10T19:20:05.054994+0000","last_fullsized":"2024-09-10T19:20:05.054994+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:19:53.174811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:19:53.174811+0000","last_clean_scrub_stamp":"2024-09-10T19:19:53.174811+0000","objects_scrubbed":0,"log_size":696,"log_dups_size":0,"ondisk_log_size":696,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:21:02.387145+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":45613056,"num_objects":696,"num_object_clones":0,"num_object_copies":2088,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":696,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":696,"num_write_kb":44544,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.7","version":"82'18","reported_seq":237,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-10T19:20:03.223936+0000","last_change":"2024-09-10T19:15:55.777428+0000","last_active":"2024-09-10T19:20:03.223936+0000","last_peered":"2024-09-10T19:20:03.223936+0000","last_clean":"2024-09-10T19:20:03.223936+0000","last_became_active":"2024-09-10T19:13:54.712065+0000","last_became_peered":"2024-09-10T19:13:54.712065+0000","last_unstale":"2024-09-10T19:20:03.223936+0000","last_undegraded":"2024-09-10T19:20:03.223936+0000","last_fullsized":"2024-09-10T19:20:03.223936+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'18","last_scrub_stamp":"2024-09-10T19:15:55.777361+0000","last_deep_scrub":"82'18","last_deep_scrub_stamp":"2024-09-10T19:15:55.777361+0000","last_clean_scrub_stamp":"2024-09-10T19:15:55.777361+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0023560040000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":19,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.0","version":"159'6","reported_seq":355,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-10T19:20:04.285328+0000","last_change":"2024-09-10T19:16:32.650007+0000","last_active":"2024-09-10T19:20:04.285328+0000","last_peered":"2024-09-10T19:20:04.285328+0000","last_clean":"2024-09-10T19:20:04.285328+0000","last_became_active":"2024-09-10T19:16:24.846203+0000","last_became_peered":"2024-09-10T19:16:24.846203+0000","last_unstale":"2024-09-10T19:20:04.285328+0000","last_undegraded":"2024-09-10T19:20:04.285328+0000","last_fullsized":"2024-09-10T19:20:04.285328+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"159'6","last_scrub_stamp":"2024-09-10T19:16:32.649914+0000","last_deep_scrub":"159'6","last_deep_scrub_stamp":"2024-09-10T19:16:32.649914+0000","last_clean_scrub_stamp":"2024-09-10T19:16:32.649914+0000","objects_scrubbed":1,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.013939817,"objects_trimmed":0,"snaptrim_duration":0.00034454499999999997,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":226,"num_read_kb":221,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":389,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":102,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-10T19:20:03.221827+0000","last_change":"2024-09-10T19:16:33.811962+0000","last_active":"2024-09-10T19:20:03.221827+0000","last_peered":"2024-09-10T19:20:03.221827+0000","last_clean":"2024-09-10T19:20:03.221827+0000","last_became_active":"2024-09-10T19:16:24.857505+0000","last_became_peered":"2024-09-10T19:16:24.857505+0000","last_unstale":"2024-09-10T19:20:03.221827+0000","last_undegraded":"2024-09-10T19:20:03.221827+0000","last_fullsized":"2024-09-10T19:20:03.221827+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:16:33.811869+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:16:33.811869+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0027920340000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":102,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-10T19:20:03.222247+0000","last_change":"2024-09-10T19:16:43.958311+0000","last_active":"2024-09-10T19:20:03.222247+0000","last_peered":"2024-09-10T19:20:03.222247+0000","last_clean":"2024-09-10T19:20:03.222247+0000","last_became_active":"2024-09-10T19:16:24.861481+0000","last_became_peered":"2024-09-10T19:16:24.861481+0000","last_unstale":"2024-09-10T19:20:03.222247+0000","last_undegraded":"2024-09-10T19:20:03.222247+0000","last_fullsized":"2024-09-10T19:20:03.222247+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:16:43.958224+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:16:43.958224+0000","last_clean_scrub_stamp":"2024-09-10T19:16:43.958224+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.015970275999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,4],"acting":[5,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":211,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-10T19:20:03.224808+0000","last_change":"2024-09-10T19:19:56.008860+0000","last_active":"2024-09-10T19:20:03.224808+0000","last_peered":"2024-09-10T19:20:03.224808+0000","last_clean":"2024-09-10T19:20:03.224808+0000","last_became_active":"2024-09-10T19:10:14.053697+0000","last_became_peered":"2024-09-10T19:10:14.053697+0000","last_unstale":"2024-09-10T19:20:03.224808+0000","last_undegraded":"2024-09-10T19:20:03.224808+0000","last_fullsized":"2024-09-10T19:20:03.224808+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:19:56.008805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:19:56.008805+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:21:21.712281+0000","scrub_duration":0.001568308,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.b","version":"234'756","reported_seq":769,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-09-10T19:20:05.028821+0000","last_change":"2024-09-10T19:19:54.199898+0000","last_active":"2024-09-10T19:20:05.028821+0000","last_peered":"2024-09-10T19:20:05.028821+0000","last_clean":"2024-09-10T19:20:05.028821+0000","last_became_active":"2024-09-10T19:19:54.199381+0000","last_became_peered":"2024-09-10T19:19:54.199381+0000","last_unstale":"2024-09-10T19:20:05.028821+0000","last_undegraded":"2024-09-10T19:20:05.028821+0000","last_fullsized":"2024-09-10T19:20:05.028821+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:19:53.174811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:19:53.174811+0000","last_clean_scrub_stamp":"2024-09-10T19:19:53.174811+0000","objects_scrubbed":0,"log_size":756,"log_dups_size":0,"ondisk_log_size":756,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:21:21.459254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49545216,"num_objects":756,"num_object_clones":0,"num_object_copies":2268,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":756,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":756,"num_write_kb":48384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.6","version":"221'18","reported_seq":296,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-10T19:20:03.226918+0000","last_change":"2024-09-10T19:19:29.606828+0000","last_active":"2024-09-10T19:20:03.226918+0000","last_peered":"2024-09-10T19:20:03.226918+0000","last_clean":"2024-09-10T19:20:03.226918+0000","last_became_active":"2024-09-10T19:16:24.844323+0000","last_became_peered":"2024-09-10T19:16:24.844323+0000","last_unstale":"2024-09-10T19:20:03.226918+0000","last_undegraded":"2024-09-10T19:20:03.226918+0000","last_fullsized":"2024-09-10T19:20:03.226918+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"221'18","last_scrub_stamp":"2024-09-10T19:19:29.606748+0000","last_deep_scrub":"221'18","last_deep_scrub_stamp":"2024-09-10T19:19:29.606748+0000","last_clean_scrub_stamp":"2024-09-10T19:19:29.606748+0000","objects_scrubbed":6,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:20:53.113038+0000","scrub_duration":0.020236845,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,5],"acting":[0,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"61'1","reported_seq":219,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-10T19:20:03.225033+0000","last_change":"2024-09-10T19:16:42.195540+0000","last_active":"2024-09-10T19:20:03.225033+0000","last_peered":"2024-09-10T19:20:03.225033+0000","last_clean":"2024-09-10T19:20:03.225033+0000","last_became_active":"2024-09-10T19:10:07.683259+0000","last_became_peered":"2024-09-10T19:10:07.683259+0000","last_unstale":"2024-09-10T19:20:03.225033+0000","last_undegraded":"2024-09-10T19:20:03.225033+0000","last_fullsized":"2024-09-10T19:20:03.225033+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'1","last_scrub_stamp":"2024-09-10T19:16:42.195463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:16:42.195463+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0016622189999999999,"objects_trimmed":0,"snaptrim_duration":0.0014341130000000001,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":221,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-10T19:20:03.224233+0000","last_change":"2024-09-10T19:15:30.542477+0000","last_active":"2024-09-10T19:20:03.224233+0000","last_peered":"2024-09-10T19:20:03.224233+0000","last_clean":"2024-09-10T19:20:03.224233+0000","last_became_active":"2024-09-10T19:13:54.721486+0000","last_became_peered":"2024-09-10T19:13:54.721486+0000","last_unstale":"2024-09-10T19:20:03.224233+0000","last_undegraded":"2024-09-10T19:20:03.224233+0000","last_fullsized":"2024-09-10T19:20:03.224233+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:15:30.542399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:15:30.542399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0025142609999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":227,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-10T19:20:03.222433+0000","last_change":"2024-09-10T19:17:28.070635+0000","last_active":"2024-09-10T19:20:03.222433+0000","last_peered":"2024-09-10T19:20:03.222433+0000","last_clean":"2024-09-10T19:20:03.222433+0000","last_became_active":"2024-09-10T19:13:54.711685+0000","last_became_peered":"2024-09-10T19:13:54.711685+0000","last_unstale":"2024-09-10T19:20:03.222433+0000","last_undegraded":"2024-09-10T19:20:03.222433+0000","last_fullsized":"2024-09-10T19:20:03.222433+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:17:28.070543+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:17:28.070543+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0017508000000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,4],"acting":[5,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":164,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-10T19:20:03.221886+0000","last_change":"2024-09-10T19:16:28.786490+0000","last_active":"2024-09-10T19:20:03.221886+0000","last_peered":"2024-09-10T19:20:03.221886+0000","last_clean":"2024-09-10T19:20:03.221886+0000","last_became_active":"2024-09-10T19:13:54.722937+0000","last_became_peered":"2024-09-10T19:13:54.722937+0000","last_unstale":"2024-09-10T19:20:03.221886+0000","last_undegraded":"2024-09-10T19:20:03.221886+0000","last_fullsized":"2024-09-10T19:20:03.221886+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:16:28.786056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:16:28.786056+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0082035739999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.8","version":"234'745","reported_seq":758,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-09-10T19:20:05.042631+0000","last_change":"2024-09-10T19:19:54.197544+0000","last_active":"2024-09-10T19:20:05.042631+0000","last_peered":"2024-09-10T19:20:05.042631+0000","last_clean":"2024-09-10T19:20:05.042631+0000","last_became_active":"2024-09-10T19:19:54.197258+0000","last_became_peered":"2024-09-10T19:19:54.197258+0000","last_unstale":"2024-09-10T19:20:05.042631+0000","last_undegraded":"2024-09-10T19:20:05.042631+0000","last_fullsized":"2024-09-10T19:20:05.042631+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:19:53.174811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:19:53.174811+0000","last_clean_scrub_stamp":"2024-09-10T19:19:53.174811+0000","objects_scrubbed":0,"log_size":745,"log_dups_size":0,"ondisk_log_size":745,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:21:17.211270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48824320,"num_objects":745,"num_object_clones":0,"num_object_copies":2235,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":745,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":745,"num_write_kb":47680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"80'21","reported_seq":251,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-10T19:20:03.225039+0000","last_change":"2024-09-10T19:17:27.420321+0000","last_active":"2024-09-10T19:20:03.225039+0000","last_peered":"2024-09-10T19:20:03.225039+0000","last_clean":"2024-09-10T19:20:03.225039+0000","last_became_active":"2024-09-10T19:10:09.822184+0000","last_became_peered":"2024-09-10T19:10:09.822184+0000","last_unstale":"2024-09-10T19:20:03.225039+0000","last_undegraded":"2024-09-10T19:20:03.225039+0000","last_fullsized":"2024-09-10T19:20:03.225039+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"80'21","last_scrub_stamp":"2024-09-10T19:17:27.420267+0000","last_deep_scrub":"80'21","last_deep_scrub_stamp":"2024-09-10T19:11:39.836144+0000","last_clean_scrub_stamp":"2024-09-10T19:17:27.420267+0000","objects_scrubbed":7,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0018427859999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"63'2","reported_seq":221,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-10T19:20:03.220261+0000","last_change":"2024-09-10T19:16:35.962952+0000","last_active":"2024-09-10T19:20:03.220261+0000","last_peered":"2024-09-10T19:20:03.220261+0000","last_clean":"2024-09-10T19:20:03.220261+0000","last_became_active":"2024-09-10T19:10:07.684784+0000","last_became_peered":"2024-09-10T19:10:07.684784+0000","last_unstale":"2024-09-10T19:20:03.220261+0000","last_undegraded":"2024-09-10T19:20:03.220261+0000","last_fullsized":"2024-09-10T19:20:03.220261+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'2","last_scrub_stamp":"2024-09-10T19:16:35.962804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:16:35.962804+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.105395924,"objects_trimmed":0,"snaptrim_duration":0.00075522999999999996,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":234,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-10T19:20:03.221858+0000","last_change":"2024-09-10T19:19:54.487118+0000","last_active":"2024-09-10T19:20:03.221858+0000","last_peered":"2024-09-10T19:20:03.221858+0000","last_clean":"2024-09-10T19:20:03.221858+0000","last_became_active":"2024-09-10T19:16:24.858662+0000","last_became_peered":"2024-09-10T19:16:24.858662+0000","last_unstale":"2024-09-10T19:20:03.221858+0000","last_undegraded":"2024-09-10T19:20:03.221858+0000","last_fullsized":"2024-09-10T19:20:03.221858+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:19:54.487065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:19:54.487065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:21:18.777331+0000","scrub_duration":0.0013784279999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"74'79","reported_seq":259,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-10T19:20:03.224309+0000","last_change":"2024-09-10T19:19:11.070913+0000","last_active":"2024-09-10T19:20:03.224309+0000","last_peered":"2024-09-10T19:20:03.224309+0000","last_clean":"2024-09-10T19:20:03.224309+0000","last_became_active":"2024-09-10T19:09:56.634919+0000","last_became_peered":"2024-09-10T19:09:56.634919+0000","last_unstale":"2024-09-10T19:20:03.224309+0000","last_undegraded":"2024-09-10T19:20:03.224309+0000","last_fullsized":"2024-09-10T19:20:03.224309+0000","mapping_epoch":55,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":56,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'79","last_scrub_stamp":"2024-09-10T19:19:11.070835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:08:00.699315+0000","last_clean_scrub_stamp":"2024-09-10T19:19:11.070835+0000","objects_scrubbed":2,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:20:23.072445+0000","scrub_duration":0.0076249890000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":220,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-10T19:20:03.224637+0000","last_change":"2024-09-10T19:19:14.030325+0000","last_active":"2024-09-10T19:20:03.224637+0000","last_peered":"2024-09-10T19:20:03.224637+0000","last_clean":"2024-09-10T19:20:03.224637+0000","last_became_active":"2024-09-10T19:16:24.851652+0000","last_became_peered":"2024-09-10T19:16:24.851652+0000","last_unstale":"2024-09-10T19:20:03.224637+0000","last_undegraded":"2024-09-10T19:20:03.224637+0000","last_fullsized":"2024-09-10T19:20:03.224637+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:19:14.030181+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:19:14.030181+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:20:22.296791+0000","scrub_duration":0.009598054,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,0],"acting":[7,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":208,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-10T19:20:03.223425+0000","last_change":"2024-09-10T19:19:11.381975+0000","last_active":"2024-09-10T19:20:03.223425+0000","last_peered":"2024-09-10T19:20:03.223425+0000","last_clean":"2024-09-10T19:20:03.223425+0000","last_became_active":"2024-09-10T19:10:13.720647+0000","last_became_peered":"2024-09-10T19:10:13.720647+0000","last_unstale":"2024-09-10T19:20:03.223425+0000","last_undegraded":"2024-09-10T19:20:03.223425+0000","last_fullsized":"2024-09-10T19:20:03.223425+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:19:11.381898+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:19:11.381898+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:20:28.772557+0000","scrub_duration":0.0040952130000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.7","version":"234'713","reported_seq":725,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-09-10T19:20:05.054462+0000","last_change":"2024-09-10T19:19:54.203251+0000","last_active":"2024-09-10T19:20:05.054462+0000","last_peered":"2024-09-10T19:20:05.054462+0000","last_clean":"2024-09-10T19:20:05.054462+0000","last_became_active":"2024-09-10T19:19:54.202928+0000","last_became_peered":"2024-09-10T19:19:54.202928+0000","last_unstale":"2024-09-10T19:20:05.054462+0000","last_undegraded":"2024-09-10T19:20:05.054462+0000","last_fullsized":"2024-09-10T19:20:05.054462+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:19:53.174811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:19:53.174811+0000","last_clean_scrub_stamp":"2024-09-10T19:19:53.174811+0000","objects_scrubbed":0,"log_size":713,"log_dups_size":0,"ondisk_log_size":713,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:21:21.507262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46727168,"num_objects":713,"num_object_clones":0,"num_object_copies":2139,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":713,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":713,"num_write_kb":45632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.a","version":"66'19","reported_seq":286,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-10T19:20:03.224013+0000","last_change":"2024-09-10T19:19:13.609209+0000","last_active":"2024-09-10T19:20:03.224013+0000","last_peered":"2024-09-10T19:20:03.224013+0000","last_clean":"2024-09-10T19:20:03.224013+0000","last_became_active":"2024-09-10T19:13:54.731274+0000","last_became_peered":"2024-09-10T19:13:54.731274+0000","last_unstale":"2024-09-10T19:20:03.224013+0000","last_undegraded":"2024-09-10T19:20:03.224013+0000","last_fullsized":"2024-09-10T19:20:03.224013+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'19","last_scrub_stamp":"2024-09-10T19:19:13.609135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:19:13.609135+0000","objects_scrubbed":9,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:20:22.695605+0000","scrub_duration":0.010981822,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,5],"acting":[6,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":92,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-10T19:20:03.221962+0000","last_change":"2024-09-10T19:16:24.847051+0000","last_active":"2024-09-10T19:20:03.221962+0000","last_peered":"2024-09-10T19:20:03.221962+0000","last_clean":"2024-09-10T19:20:03.221962+0000","last_became_active":"2024-09-10T19:16:24.846745+0000","last_became_peered":"2024-09-10T19:16:24.846745+0000","last_unstale":"2024-09-10T19:20:03.221962+0000","last_undegraded":"2024-09-10T19:20:03.221962+0000","last_fullsized":"2024-09-10T19:20:03.221962+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:15:53.803057+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:15:53.803057+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3],"acting":[5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":111,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-10T19:20:03.641532+0000","last_change":"2024-09-10T19:20:03.641455+0000","last_active":"2024-09-10T19:20:03.641532+0000","last_peered":"2024-09-10T19:20:03.641532+0000","last_clean":"2024-09-10T19:20:03.641532+0000","last_became_active":"2024-09-10T19:16:24.844295+0000","last_became_peered":"2024-09-10T19:16:24.844295+0000","last_unstale":"2024-09-10T19:20:03.641532+0000","last_undegraded":"2024-09-10T19:20:03.641532+0000","last_fullsized":"2024-09-10T19:20:03.641532+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:20:03.641376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:20:03.641376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:21:20.714146+0000","scrub_duration":0.0054491740000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":218,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-10T19:20:03.224300+0000","last_change":"2024-09-10T19:19:09.055045+0000","last_active":"2024-09-10T19:20:03.224300+0000","last_peered":"2024-09-10T19:20:03.224300+0000","last_clean":"2024-09-10T19:20:03.224300+0000","last_became_active":"2024-09-10T19:16:24.858172+0000","last_became_peered":"2024-09-10T19:16:24.858172+0000","last_unstale":"2024-09-10T19:20:03.224300+0000","last_undegraded":"2024-09-10T19:20:03.224300+0000","last_fullsized":"2024-09-10T19:20:03.224300+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:19:09.054900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:19:09.054900+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:20:28.164015+0000","scrub_duration":0.0066344780000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,6],"acting":[7,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.6","version":"234'741","reported_seq":754,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-09-10T19:20:05.045034+0000","last_change":"2024-09-10T19:19:54.201451+0000","last_active":"2024-09-10T19:20:05.045034+0000","last_peered":"2024-09-10T19:20:05.045034+0000","last_clean":"2024-09-10T19:20:05.045034+0000","last_became_active":"2024-09-10T19:19:54.201196+0000","last_became_peered":"2024-09-10T19:19:54.201196+0000","last_unstale":"2024-09-10T19:20:05.045034+0000","last_undegraded":"2024-09-10T19:20:05.045034+0000","last_fullsized":"2024-09-10T19:20:05.045034+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:19:53.174811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:19:53.174811+0000","last_clean_scrub_stamp":"2024-09-10T19:19:53.174811+0000","objects_scrubbed":0,"log_size":741,"log_dups_size":0,"ondisk_log_size":741,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:21:08.185799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":48562176,"num_objects":741,"num_object_clones":0,"num_object_copies":2223,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":741,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":741,"num_write_kb":47424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"66'9","reported_seq":260,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-10T19:20:03.226064+0000","last_change":"2024-09-10T19:19:56.711084+0000","last_active":"2024-09-10T19:20:03.226064+0000","last_peered":"2024-09-10T19:20:03.226064+0000","last_clean":"2024-09-10T19:20:03.226064+0000","last_became_active":"2024-09-10T19:13:54.710613+0000","last_became_peered":"2024-09-10T19:13:54.710613+0000","last_unstale":"2024-09-10T19:20:03.226064+0000","last_undegraded":"2024-09-10T19:20:03.226064+0000","last_fullsized":"2024-09-10T19:20:03.226064+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'9","last_scrub_stamp":"2024-09-10T19:19:56.711029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:19:56.711029+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:21:14.743750+0000","scrub_duration":0.001540919,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":177,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-10T19:20:03.228912+0000","last_change":"2024-09-10T19:16:24.848979+0000","last_active":"2024-09-10T19:20:03.228912+0000","last_peered":"2024-09-10T19:20:03.228912+0000","last_clean":"2024-09-10T19:20:03.228912+0000","last_became_active":"2024-09-10T19:16:24.848668+0000","last_became_peered":"2024-09-10T19:16:24.848668+0000","last_unstale":"2024-09-10T19:20:03.228912+0000","last_undegraded":"2024-09-10T19:20:03.228912+0000","last_fullsized":"2024-09-10T19:20:03.228912+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:15:50.850576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:11:44.254513+0000","last_clean_scrub_stamp":"2024-09-10T19:15:50.850576+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001815419,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":105,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-10T19:20:03.226761+0000","last_change":"2024-09-10T19:19:11.465972+0000","last_active":"2024-09-10T19:20:03.226761+0000","last_peered":"2024-09-10T19:20:03.226761+0000","last_clean":"2024-09-10T19:20:03.226761+0000","last_became_active":"2024-09-10T19:16:24.845709+0000","last_became_peered":"2024-09-10T19:16:24.845709+0000","last_unstale":"2024-09-10T19:20:03.226761+0000","last_undegraded":"2024-09-10T19:20:03.226761+0000","last_fullsized":"2024-09-10T19:20:03.226761+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:19:11.465819+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:19:11.465819+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:20:18.036431+0000","scrub_duration":0.012180921000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":217,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-10T19:20:03.224955+0000","last_change":"2024-09-10T19:16:39.188737+0000","last_active":"2024-09-10T19:20:03.224955+0000","last_peered":"2024-09-10T19:20:03.224955+0000","last_clean":"2024-09-10T19:20:03.224955+0000","last_became_active":"2024-09-10T19:16:24.837483+0000","last_became_peered":"2024-09-10T19:16:24.837483+0000","last_unstale":"2024-09-10T19:20:03.224955+0000","last_undegraded":"2024-09-10T19:20:03.224955+0000","last_fullsized":"2024-09-10T19:20:03.224955+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:16:39.188682+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:16:39.188682+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0012633589999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.5","version":"234'758","reported_seq":770,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-09-10T19:20:05.029305+0000","last_change":"2024-09-10T19:19:54.199874+0000","last_active":"2024-09-10T19:20:05.029305+0000","last_peered":"2024-09-10T19:20:05.029305+0000","last_clean":"2024-09-10T19:20:05.029305+0000","last_became_active":"2024-09-10T19:19:54.199543+0000","last_became_peered":"2024-09-10T19:19:54.199543+0000","last_unstale":"2024-09-10T19:20:05.029305+0000","last_undegraded":"2024-09-10T19:20:05.029305+0000","last_fullsized":"2024-09-10T19:20:05.029305+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:19:53.174811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:19:53.174811+0000","last_clean_scrub_stamp":"2024-09-10T19:19:53.174811+0000","objects_scrubbed":0,"log_size":758,"log_dups_size":0,"ondisk_log_size":758,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:21:09.188240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":49676288,"num_objects":758,"num_object_clones":0,"num_object_copies":2274,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":758,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":758,"num_write_kb":48512,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.8","version":"220'22","reported_seq":244,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-10T19:20:03.221436+0000","last_change":"2024-09-10T19:19:24.597946+0000","last_active":"2024-09-10T19:20:03.221436+0000","last_peered":"2024-09-10T19:20:03.221436+0000","last_clean":"2024-09-10T19:20:03.221436+0000","last_became_active":"2024-09-10T19:10:09.820950+0000","last_became_peered":"2024-09-10T19:10:09.820950+0000","last_unstale":"2024-09-10T19:20:03.221436+0000","last_undegraded":"2024-09-10T19:20:03.221436+0000","last_fullsized":"2024-09-10T19:20:03.221436+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"220'22","last_scrub_stamp":"2024-09-10T19:19:24.597866+0000","last_deep_scrub":"220'22","last_deep_scrub_stamp":"2024-09-10T19:19:24.597866+0000","last_clean_scrub_stamp":"2024-09-10T19:19:24.597866+0000","objects_scrubbed":7,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:20:41.284281+0000","scrub_duration":0.031475547,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":92,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-10T19:20:03.220712+0000","last_change":"2024-09-10T19:16:24.857533+0000","last_active":"2024-09-10T19:20:03.220712+0000","last_peered":"2024-09-10T19:20:03.220712+0000","last_clean":"2024-09-10T19:20:03.220712+0000","last_became_active":"2024-09-10T19:16:24.857253+0000","last_became_peered":"2024-09-10T19:16:24.857253+0000","last_unstale":"2024-09-10T19:20:03.220712+0000","last_undegraded":"2024-09-10T19:20:03.220712+0000","last_fullsized":"2024-09-10T19:20:03.220712+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:15:38.677960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:15:38.677960+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.9","version":"80'9","reported_seq":338,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-10T19:20:03.224449+0000","last_change":"2024-09-10T19:19:19.008253+0000","last_active":"2024-09-10T19:20:03.224449+0000","last_peered":"2024-09-10T19:20:03.224449+0000","last_clean":"2024-09-10T19:20:03.224449+0000","last_became_active":"2024-09-10T19:10:11.702878+0000","last_became_peered":"2024-09-10T19:10:11.702878+0000","last_unstale":"2024-09-10T19:20:03.224449+0000","last_undegraded":"2024-09-10T19:20:03.224449+0000","last_fullsized":"2024-09-10T19:20:03.224449+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"80'9","last_scrub_stamp":"2024-09-10T19:19:19.008106+0000","last_deep_scrub":"80'9","last_deep_scrub_stamp":"2024-09-10T19:11:35.174218+0000","last_clean_scrub_stamp":"2024-09-10T19:19:19.008106+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:20:25.426864+0000","scrub_duration":0.015626330000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":208,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-10T19:20:03.222068+0000","last_change":"2024-09-10T19:19:15.539622+0000","last_active":"2024-09-10T19:20:03.222068+0000","last_peered":"2024-09-10T19:20:03.222068+0000","last_clean":"2024-09-10T19:20:03.222068+0000","last_became_active":"2024-09-10T19:10:14.053875+0000","last_became_peered":"2024-09-10T19:10:14.053875+0000","last_unstale":"2024-09-10T19:20:03.222068+0000","last_undegraded":"2024-09-10T19:20:03.222068+0000","last_fullsized":"2024-09-10T19:20:03.222068+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:19:15.539488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:19:15.539488+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:20:22.964354+0000","scrub_duration":0.033498668000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.4","version":"234'724","reported_seq":736,"reported_epoch":234,"state":"active+clean","last_fresh":"2024-09-10T19:20:04.813155+0000","last_change":"2024-09-10T19:19:54.198182+0000","last_active":"2024-09-10T19:20:04.813155+0000","last_peered":"2024-09-10T19:20:04.813155+0000","last_clean":"2024-09-10T19:20:04.813155+0000","last_became_active":"2024-09-10T19:19:54.197469+0000","last_became_peered":"2024-09-10T19:19:54.197469+0000","last_unstale":"2024-09-10T19:20:04.813155+0000","last_undegraded":"2024-09-10T19:20:04.813155+0000","last_fullsized":"2024-09-10T19:20:04.813155+0000","mapping_epoch":229,"log_start":"0'0","ondisk_log_start":"0'0","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:19:53.174811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:19:53.174811+0000","last_clean_scrub_stamp":"2024-09-10T19:19:53.174811+0000","objects_scrubbed":0,"log_size":724,"log_dups_size":0,"ondisk_log_size":724,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:21:09.480428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":47448064,"num_objects":724,"num_object_clones":0,"num_object_copies":2172,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":724,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":724,"num_write_kb":46336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.9","version":"66'12","reported_seq":256,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-10T19:20:03.226341+0000","last_change":"2024-09-10T19:15:37.869593+0000","last_active":"2024-09-10T19:20:03.226341+0000","last_peered":"2024-09-10T19:20:03.226341+0000","last_clean":"2024-09-10T19:20:03.226341+0000","last_became_active":"2024-09-10T19:13:54.718152+0000","last_became_peered":"2024-09-10T19:13:54.718152+0000","last_unstale":"2024-09-10T19:20:03.226341+0000","last_undegraded":"2024-09-10T19:20:03.226341+0000","last_fullsized":"2024-09-10T19:20:03.226341+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'12","last_scrub_stamp":"2024-09-10T19:15:37.869451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:15:37.869451+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0025441249999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.f","version":"59'2","reported_seq":229,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-10T19:20:03.229244+0000","last_change":"2024-09-10T19:16:29.007627+0000","last_active":"2024-09-10T19:20:03.229244+0000","last_peered":"2024-09-10T19:20:03.229244+0000","last_clean":"2024-09-10T19:20:03.229244+0000","last_became_active":"2024-09-10T19:10:07.681719+0000","last_became_peered":"2024-09-10T19:10:07.681719+0000","last_unstale":"2024-09-10T19:20:03.229244+0000","last_undegraded":"2024-09-10T19:20:03.229244+0000","last_fullsized":"2024-09-10T19:20:03.229244+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'2","last_scrub_stamp":"2024-09-10T19:16:29.007251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:16:29.007251+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0079698319999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":14,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":187,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-10T19:20:03.225113+0000","last_change":"2024-09-10T19:19:58.707215+0000","last_active":"2024-09-10T19:20:03.225113+0000","last_peered":"2024-09-10T19:20:03.225113+0000","last_clean":"2024-09-10T19:20:03.225113+0000","last_became_active":"2024-09-10T19:16:24.840091+0000","last_became_peered":"2024-09-10T19:16:24.840091+0000","last_unstale":"2024-09-10T19:20:03.225113+0000","last_undegraded":"2024-09-10T19:20:03.225113+0000","last_fullsized":"2024-09-10T19:20:03.225113+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:19:58.707161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:11:46.587846+0000","last_clean_scrub_stamp":"2024-09-10T19:19:58.707161+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:21:12.543310+0000","scrub_duration":0.0017110669999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,4],"acting":[0,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":215,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-10T19:20:03.220637+0000","last_change":"2024-09-10T19:15:41.093923+0000","last_active":"2024-09-10T19:20:03.220637+0000","last_peered":"2024-09-10T19:20:03.220637+0000","last_clean":"2024-09-10T19:20:03.220637+0000","last_became_active":"2024-09-10T19:13:54.711804+0000","last_became_peered":"2024-09-10T19:13:54.711804+0000","last_unstale":"2024-09-10T19:20:03.220637+0000","last_undegraded":"2024-09-10T19:20:03.220637+0000","last_fullsized":"2024-09-10T19:20:03.220637+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:15:41.093783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:15:41.093783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0023833050000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.16","version":"66'9","reported_seq":243,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-10T19:20:03.226774+0000","last_change":"2024-09-10T19:19:16.444223+0000","last_active":"2024-09-10T19:20:03.226774+0000","last_peered":"2024-09-10T19:20:03.226774+0000","last_clean":"2024-09-10T19:20:03.226774+0000","last_became_active":"2024-09-10T19:10:09.822994+0000","last_became_peered":"2024-09-10T19:10:09.822994+0000","last_unstale":"2024-09-10T19:20:03.226774+0000","last_undegraded":"2024-09-10T19:20:03.226774+0000","last_fullsized":"2024-09-10T19:20:03.226774+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'9","last_scrub_stamp":"2024-09-10T19:19:16.444147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:19:16.444147+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:20:19.419312+0000","scrub_duration":0.0065392059999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":185,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-10T19:20:03.224678+0000","last_change":"2024-09-10T19:17:16.305585+0000","last_active":"2024-09-10T19:20:03.224678+0000","last_peered":"2024-09-10T19:20:03.224678+0000","last_clean":"2024-09-10T19:20:03.224678+0000","last_became_active":"2024-09-10T19:16:24.844313+0000","last_became_peered":"2024-09-10T19:16:24.844313+0000","last_unstale":"2024-09-10T19:20:03.224678+0000","last_undegraded":"2024-09-10T19:20:03.224678+0000","last_fullsized":"2024-09-10T19:20:03.224678+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:17:16.305525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:11:41.458523+0000","last_clean_scrub_stamp":"2024-09-10T19:17:16.305525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0070771209999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,6],"acting":[0,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":217,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-10T19:20:03.223504+0000","last_change":"2024-09-10T19:16:28.794727+0000","last_active":"2024-09-10T19:20:03.223504+0000","last_peered":"2024-09-10T19:20:03.223504+0000","last_clean":"2024-09-10T19:20:03.223504+0000","last_became_active":"2024-09-10T19:13:54.712275+0000","last_became_peered":"2024-09-10T19:13:54.712275+0000","last_unstale":"2024-09-10T19:20:03.223504+0000","last_undegraded":"2024-09-10T19:20:03.223504+0000","last_fullsized":"2024-09-10T19:20:03.223504+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:16:28.794649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:16:28.794649+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0027500379999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"66'1","reported_seq":105,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-10T19:20:03.226262+0000","last_change":"2024-09-10T19:16:24.862865+0000","last_active":"2024-09-10T19:20:03.226262+0000","last_peered":"2024-09-10T19:20:03.226262+0000","last_clean":"2024-09-10T19:20:03.226262+0000","last_became_active":"2024-09-10T19:16:24.837432+0000","last_became_peered":"2024-09-10T19:16:24.837432+0000","last_unstale":"2024-09-10T19:20:03.226262+0000","last_undegraded":"2024-09-10T19:20:03.226262+0000","last_fullsized":"2024-09-10T19:20:03.226262+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'1","last_scrub_stamp":"2024-09-10T19:15:48.839582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:15:48.839582+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":13,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":237,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-10T19:20:03.437998+0000","last_change":"2024-09-10T19:20:03.437843+0000","last_active":"2024-09-10T19:20:03.437998+0000","last_peered":"2024-09-10T19:20:03.437998+0000","last_clean":"2024-09-10T19:20:03.437998+0000","last_became_active":"2024-09-10T19:13:54.724027+0000","last_became_peered":"2024-09-10T19:13:54.724027+0000","last_unstale":"2024-09-10T19:20:03.437998+0000","last_undegraded":"2024-09-10T19:20:03.437998+0000","last_fullsized":"2024-09-10T19:20:03.437998+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:20:03.437749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:20:03.437749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:21:14.797737+0000","scrub_duration":0.0050536629999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,5],"acting":[6,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"66'6","reported_seq":254,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-10T19:20:03.223975+0000","last_change":"2024-09-10T19:19:25.266265+0000","last_active":"2024-09-10T19:20:03.223975+0000","last_peered":"2024-09-10T19:20:03.223975+0000","last_clean":"2024-09-10T19:20:03.223975+0000","last_became_active":"2024-09-10T19:16:24.863398+0000","last_became_peered":"2024-09-10T19:16:24.863398+0000","last_unstale":"2024-09-10T19:20:03.223975+0000","last_undegraded":"2024-09-10T19:20:03.223975+0000","last_fullsized":"2024-09-10T19:20:03.223975+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'6","last_scrub_stamp":"2024-09-10T19:19:25.266162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:19:25.266162+0000","objects_scrubbed":3,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:20:41.361532+0000","scrub_duration":0.013851005,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":227,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-10T19:20:03.222248+0000","last_change":"2024-09-10T19:20:00.494694+0000","last_active":"2024-09-10T19:20:03.222248+0000","last_peered":"2024-09-10T19:20:03.222248+0000","last_clean":"2024-09-10T19:20:03.222248+0000","last_became_active":"2024-09-10T19:13:54.729917+0000","last_became_peered":"2024-09-10T19:13:54.729917+0000","last_unstale":"2024-09-10T19:20:03.222248+0000","last_undegraded":"2024-09-10T19:20:03.222248+0000","last_fullsized":"2024-09-10T19:20:03.222248+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:20:00.494610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:11:23.264793+0000","last_clean_scrub_stamp":"2024-09-10T19:20:00.494610+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:21:12.039036+0000","scrub_duration":0.001639585,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":208,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-10T19:20:03.224730+0000","last_change":"2024-09-10T19:17:26.922378+0000","last_active":"2024-09-10T19:20:03.224730+0000","last_peered":"2024-09-10T19:20:03.224730+0000","last_clean":"2024-09-10T19:20:03.224730+0000","last_became_active":"2024-09-10T19:10:14.054360+0000","last_became_peered":"2024-09-10T19:10:14.054360+0000","last_unstale":"2024-09-10T19:20:03.224730+0000","last_undegraded":"2024-09-10T19:20:03.224730+0000","last_fullsized":"2024-09-10T19:20:03.224730+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:17:26.922305+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:17:26.922305+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0016181940000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"80'15","reported_seq":283,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-10T19:20:03.220398+0000","last_change":"2024-09-10T19:20:02.458352+0000","last_active":"2024-09-10T19:20:03.220398+0000","last_peered":"2024-09-10T19:20:03.220398+0000","last_clean":"2024-09-10T19:20:03.220398+0000","last_became_active":"2024-09-10T19:16:24.850702+0000","last_became_peered":"2024-09-10T19:16:24.850702+0000","last_unstale":"2024-09-10T19:20:03.220398+0000","last_undegraded":"2024-09-10T19:20:03.220398+0000","last_fullsized":"2024-09-10T19:20:03.220398+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"80'15","last_scrub_stamp":"2024-09-10T19:20:02.458275+0000","last_deep_scrub":"80'15","last_deep_scrub_stamp":"2024-09-10T19:11:41.719310+0000","last_clean_scrub_stamp":"2024-09-10T19:20:02.458275+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:21:11.267796+0000","scrub_duration":0.0054307269999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,5],"acting":[3,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":214,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-10T19:20:03.222133+0000","last_change":"2024-09-10T19:19:17.588731+0000","last_active":"2024-09-10T19:20:03.222133+0000","last_peered":"2024-09-10T19:20:03.222133+0000","last_clean":"2024-09-10T19:20:03.222133+0000","last_became_active":"2024-09-10T19:10:07.685870+0000","last_became_peered":"2024-09-10T19:10:07.685870+0000","last_unstale":"2024-09-10T19:20:03.222133+0000","last_undegraded":"2024-09-10T19:20:03.222133+0000","last_fullsized":"2024-09-10T19:20:03.222133+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:19:17.588552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:19:17.588552+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:20:29.324857+0000","scrub_duration":0.0065035919999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"117'10","reported_seq":349,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-10T19:20:03.222499+0000","last_change":"2024-09-10T19:16:23.844790+0000","last_active":"2024-09-10T19:20:03.222499+0000","last_peered":"2024-09-10T19:20:03.222499+0000","last_clean":"2024-09-10T19:20:03.222499+0000","last_became_active":"2024-09-10T19:13:54.724780+0000","last_became_peered":"2024-09-10T19:13:54.724780+0000","last_unstale":"2024-09-10T19:20:03.222499+0000","last_undegraded":"2024-09-10T19:20:03.222499+0000","last_fullsized":"2024-09-10T19:20:03.222499+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"117'10","last_scrub_stamp":"2024-09-10T19:16:23.844724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:16:23.844724+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.012497955999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3],"acting":[5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":207,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-10T19:20:03.226503+0000","last_change":"2024-09-10T19:16:40.213422+0000","last_active":"2024-09-10T19:20:03.226503+0000","last_peered":"2024-09-10T19:20:03.226503+0000","last_clean":"2024-09-10T19:20:03.226503+0000","last_became_active":"2024-09-10T19:10:13.721035+0000","last_became_peered":"2024-09-10T19:10:13.721035+0000","last_unstale":"2024-09-10T19:20:03.226503+0000","last_undegraded":"2024-09-10T19:20:03.226503+0000","last_fullsized":"2024-09-10T19:20:03.226503+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:16:40.213321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:16:40.213321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0051408239999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"66'9","reported_seq":268,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-10T19:20:03.222644+0000","last_change":"2024-09-10T19:19:23.577305+0000","last_active":"2024-09-10T19:20:03.222644+0000","last_peered":"2024-09-10T19:20:03.222644+0000","last_clean":"2024-09-10T19:20:03.222644+0000","last_became_active":"2024-09-10T19:16:24.859740+0000","last_became_peered":"2024-09-10T19:16:24.859740+0000","last_unstale":"2024-09-10T19:20:03.222644+0000","last_undegraded":"2024-09-10T19:20:03.222644+0000","last_fullsized":"2024-09-10T19:20:03.222644+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'9","last_scrub_stamp":"2024-09-10T19:19:23.577158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:19:23.577158+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:20:32.990634+0000","scrub_duration":0.011983318,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":213,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-10T19:20:03.226325+0000","last_change":"2024-09-10T19:16:37.243942+0000","last_active":"2024-09-10T19:20:03.226325+0000","last_peered":"2024-09-10T19:20:03.226325+0000","last_clean":"2024-09-10T19:20:03.226325+0000","last_became_active":"2024-09-10T19:10:07.688499+0000","last_became_peered":"2024-09-10T19:10:07.688499+0000","last_unstale":"2024-09-10T19:20:03.226325+0000","last_undegraded":"2024-09-10T19:20:03.226325+0000","last_fullsized":"2024-09-10T19:20:03.226325+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:16:37.243859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:16:37.243859+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0052374580000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"155'9","reported_seq":351,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-10T19:20:03.220870+0000","last_change":"2024-09-10T19:19:16.257770+0000","last_active":"2024-09-10T19:20:03.220870+0000","last_peered":"2024-09-10T19:20:03.220870+0000","last_clean":"2024-09-10T19:20:03.220870+0000","last_became_active":"2024-09-10T19:16:24.850850+0000","last_became_peered":"2024-09-10T19:16:24.850850+0000","last_unstale":"2024-09-10T19:20:03.220870+0000","last_undegraded":"2024-09-10T19:20:03.220870+0000","last_fullsized":"2024-09-10T19:20:03.220870+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"155'9","last_scrub_stamp":"2024-09-10T19:19:16.257692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:19:16.257692+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:20:20.031544+0000","scrub_duration":0.0076401230000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,4],"acting":[3,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":220,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-10T19:20:03.229073+0000","last_change":"2024-09-10T19:19:12.504470+0000","last_active":"2024-09-10T19:20:03.229073+0000","last_peered":"2024-09-10T19:20:03.229073+0000","last_clean":"2024-09-10T19:20:03.229073+0000","last_became_active":"2024-09-10T19:16:24.854280+0000","last_became_peered":"2024-09-10T19:16:24.854280+0000","last_unstale":"2024-09-10T19:20:03.229073+0000","last_undegraded":"2024-09-10T19:20:03.229073+0000","last_fullsized":"2024-09-10T19:20:03.229073+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:19:12.504402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:19:12.504402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:20:41.166660+0000","scrub_duration":0.0034925469999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.12","version":"79'13","reported_seq":257,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-10T19:20:03.223798+0000","last_change":"2024-09-10T19:19:27.518153+0000","last_active":"2024-09-10T19:20:03.223798+0000","last_peered":"2024-09-10T19:20:03.223798+0000","last_clean":"2024-09-10T19:20:03.223798+0000","last_became_active":"2024-09-10T19:16:24.845450+0000","last_became_peered":"2024-09-10T19:16:24.845450+0000","last_unstale":"2024-09-10T19:20:03.223798+0000","last_undegraded":"2024-09-10T19:20:03.223798+0000","last_fullsized":"2024-09-10T19:20:03.223798+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"79'13","last_scrub_stamp":"2024-09-10T19:19:27.518064+0000","last_deep_scrub":"79'13","last_deep_scrub_stamp":"2024-09-10T19:11:22.072400+0000","last_clean_scrub_stamp":"2024-09-10T19:19:27.518064+0000","objects_scrubbed":4,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:20:28.586075+0000","scrub_duration":0.010862685,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.14","version":"59'1","reported_seq":221,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-10T19:20:03.224900+0000","last_change":"2024-09-10T19:16:27.594895+0000","last_active":"2024-09-10T19:20:03.224900+0000","last_peered":"2024-09-10T19:20:03.224900+0000","last_clean":"2024-09-10T19:20:03.224900+0000","last_became_active":"2024-09-10T19:10:07.683263+0000","last_became_peered":"2024-09-10T19:10:07.683263+0000","last_unstale":"2024-09-10T19:20:03.224900+0000","last_undegraded":"2024-09-10T19:20:03.224900+0000","last_fullsized":"2024-09-10T19:20:03.224900+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'1","last_scrub_stamp":"2024-09-10T19:16:27.594788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:16:27.594788+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0091369840000000008,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":362,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":217,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-10T19:20:03.220701+0000","last_change":"2024-09-10T19:15:55.925313+0000","last_active":"2024-09-10T19:20:03.220701+0000","last_peered":"2024-09-10T19:20:03.220701+0000","last_clean":"2024-09-10T19:20:03.220701+0000","last_became_active":"2024-09-10T19:13:54.730185+0000","last_became_peered":"2024-09-10T19:13:54.730185+0000","last_unstale":"2024-09-10T19:20:03.220701+0000","last_undegraded":"2024-09-10T19:20:03.220701+0000","last_fullsized":"2024-09-10T19:20:03.220701+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:15:55.925228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:15:55.925228+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0017202299999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":214,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-10T19:20:03.225264+0000","last_change":"2024-09-10T19:16:23.273671+0000","last_active":"2024-09-10T19:20:03.225264+0000","last_peered":"2024-09-10T19:20:03.225264+0000","last_clean":"2024-09-10T19:20:03.225264+0000","last_became_active":"2024-09-10T19:13:54.714861+0000","last_became_peered":"2024-09-10T19:13:54.714861+0000","last_unstale":"2024-09-10T19:20:03.225264+0000","last_undegraded":"2024-09-10T19:20:03.225264+0000","last_fullsized":"2024-09-10T19:20:03.225264+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:16:23.273615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:16:23.273615+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0013972139999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,6],"acting":[0,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"66'11","reported_seq":252,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-10T19:20:03.227267+0000","last_change":"2024-09-10T19:15:53.883613+0000","last_active":"2024-09-10T19:20:03.227267+0000","last_peered":"2024-09-10T19:20:03.227267+0000","last_clean":"2024-09-10T19:20:03.227267+0000","last_became_active":"2024-09-10T19:13:54.720034+0000","last_became_peered":"2024-09-10T19:13:54.720034+0000","last_unstale":"2024-09-10T19:20:03.227267+0000","last_undegraded":"2024-09-10T19:20:03.227267+0000","last_fullsized":"2024-09-10T19:20:03.227267+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'11","last_scrub_stamp":"2024-09-10T19:15:53.883548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:15:53.883548+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001858326,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":164,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-10T19:20:03.222583+0000","last_change":"2024-09-10T19:19:10.439796+0000","last_active":"2024-09-10T19:20:03.222583+0000","last_peered":"2024-09-10T19:20:03.222583+0000","last_clean":"2024-09-10T19:20:03.222583+0000","last_became_active":"2024-09-10T19:13:54.722947+0000","last_became_peered":"2024-09-10T19:13:54.722947+0000","last_unstale":"2024-09-10T19:20:03.222583+0000","last_undegraded":"2024-09-10T19:20:03.222583+0000","last_fullsized":"2024-09-10T19:20:03.222583+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:19:10.439659+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:11:10.031245+0000","last_clean_scrub_stamp":"2024-09-10T19:19:10.439659+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:20:17.099888+0000","scrub_duration":0.014001438,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,6],"acting":[5,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":164,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-10T19:20:03.221981+0000","last_change":"2024-09-10T19:15:36.666310+0000","last_active":"2024-09-10T19:20:03.221981+0000","last_peered":"2024-09-10T19:20:03.221981+0000","last_clean":"2024-09-10T19:20:03.221981+0000","last_became_active":"2024-09-10T19:13:54.729703+0000","last_became_peered":"2024-09-10T19:13:54.729703+0000","last_unstale":"2024-09-10T19:20:03.221981+0000","last_undegraded":"2024-09-10T19:20:03.221981+0000","last_fullsized":"2024-09-10T19:20:03.221981+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:15:36.666216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:15:36.666216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.003489884,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":208,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-10T19:20:03.219557+0000","last_change":"2024-09-10T19:19:13.231447+0000","last_active":"2024-09-10T19:20:03.219557+0000","last_peered":"2024-09-10T19:20:03.219557+0000","last_clean":"2024-09-10T19:20:03.219557+0000","last_became_active":"2024-09-10T19:10:13.724201+0000","last_became_peered":"2024-09-10T19:10:13.724201+0000","last_unstale":"2024-09-10T19:20:03.219557+0000","last_undegraded":"2024-09-10T19:20:03.219557+0000","last_fullsized":"2024-09-10T19:20:03.219557+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:19:13.231390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:19:13.231390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:20:25.623720+0000","scrub_duration":0.0029165770000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"222'6","reported_seq":243,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-10T19:20:03.219987+0000","last_change":"2024-09-10T19:19:30.328975+0000","last_active":"2024-09-10T19:20:03.219987+0000","last_peered":"2024-09-10T19:20:03.219987+0000","last_clean":"2024-09-10T19:20:03.219987+0000","last_became_active":"2024-09-10T19:13:54.711125+0000","last_became_peered":"2024-09-10T19:13:54.711125+0000","last_unstale":"2024-09-10T19:20:03.219987+0000","last_undegraded":"2024-09-10T19:20:03.219987+0000","last_fullsized":"2024-09-10T19:20:03.219987+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"222'6","last_scrub_stamp":"2024-09-10T19:19:30.328841+0000","last_deep_scrub":"222'6","last_deep_scrub_stamp":"2024-09-10T19:19:30.328841+0000","last_clean_scrub_stamp":"2024-09-10T19:19:30.328841+0000","objects_scrubbed":2,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:20:51.291574+0000","scrub_duration":0.011722731,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,4],"acting":[3,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":224,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-10T19:20:03.222400+0000","last_change":"2024-09-10T19:16:33.766560+0000","last_active":"2024-09-10T19:20:03.222400+0000","last_peered":"2024-09-10T19:20:03.222400+0000","last_clean":"2024-09-10T19:20:03.222400+0000","last_became_active":"2024-09-10T19:16:24.852460+0000","last_became_peered":"2024-09-10T19:16:24.852460+0000","last_unstale":"2024-09-10T19:20:03.222400+0000","last_undegraded":"2024-09-10T19:20:03.222400+0000","last_fullsized":"2024-09-10T19:20:03.222400+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:16:33.766474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:16:33.766474+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0042170250000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,3],"acting":[5,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":210,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-10T19:20:03.223832+0000","last_change":"2024-09-10T19:16:24.545188+0000","last_active":"2024-09-10T19:20:03.223832+0000","last_peered":"2024-09-10T19:20:03.223832+0000","last_clean":"2024-09-10T19:20:03.223832+0000","last_became_active":"2024-09-10T19:10:11.698536+0000","last_became_peered":"2024-09-10T19:10:11.698536+0000","last_unstale":"2024-09-10T19:20:03.223832+0000","last_undegraded":"2024-09-10T19:20:03.223832+0000","last_fullsized":"2024-09-10T19:20:03.223832+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:16:24.545134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:11:31.837944+0000","last_clean_scrub_stamp":"2024-09-10T19:16:24.545134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.00120104,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":228,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-10T19:20:03.223962+0000","last_change":"2024-09-10T19:19:23.027867+0000","last_active":"2024-09-10T19:20:03.223962+0000","last_peered":"2024-09-10T19:20:03.223962+0000","last_clean":"2024-09-10T19:20:03.223962+0000","last_became_active":"2024-09-10T19:16:24.852356+0000","last_became_peered":"2024-09-10T19:16:24.852356+0000","last_unstale":"2024-09-10T19:20:03.223962+0000","last_undegraded":"2024-09-10T19:20:03.223962+0000","last_fullsized":"2024-09-10T19:20:03.223962+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:19:23.027800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:19:23.027800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:20:25.338260+0000","scrub_duration":0.014396977,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,3],"acting":[7,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.11","version":"66'11","reported_seq":122,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-10T19:20:03.221486+0000","last_change":"2024-09-10T19:19:25.606746+0000","last_active":"2024-09-10T19:20:03.221486+0000","last_peered":"2024-09-10T19:20:03.221486+0000","last_clean":"2024-09-10T19:20:03.221486+0000","last_became_active":"2024-09-10T19:15:45.149864+0000","last_became_peered":"2024-09-10T19:15:45.149864+0000","last_unstale":"2024-09-10T19:20:03.221486+0000","last_undegraded":"2024-09-10T19:20:03.221486+0000","last_fullsized":"2024-09-10T19:20:03.221486+0000","mapping_epoch":143,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":144,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'11","last_scrub_stamp":"2024-09-10T19:19:25.606598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:19:25.606598+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:20:36.591926+0000","scrub_duration":0.015282657,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":224,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-10T19:20:03.225940+0000","last_change":"2024-09-10T19:16:24.855851+0000","last_active":"2024-09-10T19:20:03.225940+0000","last_peered":"2024-09-10T19:20:03.225940+0000","last_clean":"2024-09-10T19:20:03.225940+0000","last_became_active":"2024-09-10T19:16:24.855541+0000","last_became_peered":"2024-09-10T19:16:24.855541+0000","last_unstale":"2024-09-10T19:20:03.225940+0000","last_undegraded":"2024-09-10T19:20:03.225940+0000","last_fullsized":"2024-09-10T19:20:03.225940+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:16:23.543097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:13:05.876655+0000","last_clean_scrub_stamp":"2024-09-10T19:16:23.543097+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001181779,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,0],"acting":[6,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":210,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-10T19:20:03.223960+0000","last_change":"2024-09-10T19:19:20.045982+0000","last_active":"2024-09-10T19:20:03.223960+0000","last_peered":"2024-09-10T19:20:03.223960+0000","last_clean":"2024-09-10T19:20:03.223960+0000","last_became_active":"2024-09-10T19:10:11.699013+0000","last_became_peered":"2024-09-10T19:10:11.699013+0000","last_unstale":"2024-09-10T19:20:03.223960+0000","last_undegraded":"2024-09-10T19:20:03.223960+0000","last_fullsized":"2024-09-10T19:20:03.223960+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:19:20.045906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:11:57.112092+0000","last_clean_scrub_stamp":"2024-09-10T19:19:20.045906+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:20:42.628925+0000","scrub_duration":0.019773829999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":208,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-10T19:20:03.220574+0000","last_change":"2024-09-10T19:19:18.276122+0000","last_active":"2024-09-10T19:20:03.220574+0000","last_peered":"2024-09-10T19:20:03.220574+0000","last_clean":"2024-09-10T19:20:03.220574+0000","last_became_active":"2024-09-10T19:10:14.056172+0000","last_became_peered":"2024-09-10T19:10:14.056172+0000","last_unstale":"2024-09-10T19:20:03.220574+0000","last_undegraded":"2024-09-10T19:20:03.220574+0000","last_fullsized":"2024-09-10T19:20:03.220574+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:19:18.276069+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:19:18.276069+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:20:26.486746+0000","scrub_duration":0.0034003950000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"80'2","reported_seq":224,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-10T19:20:03.224553+0000","last_change":"2024-09-10T19:16:32.648844+0000","last_active":"2024-09-10T19:20:03.224553+0000","last_peered":"2024-09-10T19:20:03.224553+0000","last_clean":"2024-09-10T19:20:03.224553+0000","last_became_active":"2024-09-10T19:16:24.851220+0000","last_became_peered":"2024-09-10T19:16:24.851220+0000","last_unstale":"2024-09-10T19:20:03.224553+0000","last_undegraded":"2024-09-10T19:20:03.224553+0000","last_fullsized":"2024-09-10T19:20:03.224553+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"80'2","last_scrub_stamp":"2024-09-10T19:16:32.648738+0000","last_deep_scrub":"80'2","last_deep_scrub_stamp":"2024-09-10T19:11:30.272043+0000","last_clean_scrub_stamp":"2024-09-10T19:16:32.648738+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0066232920000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":369,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":369,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,3],"acting":[7,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":224,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-10T19:20:03.221493+0000","last_change":"2024-09-10T19:19:19.581767+0000","last_active":"2024-09-10T19:20:03.221493+0000","last_peered":"2024-09-10T19:20:03.221493+0000","last_clean":"2024-09-10T19:20:03.221493+0000","last_became_active":"2024-09-10T19:19:19.379420+0000","last_became_peered":"2024-09-10T19:19:19.379420+0000","last_unstale":"2024-09-10T19:20:03.221493+0000","last_undegraded":"2024-09-10T19:20:03.221493+0000","last_fullsized":"2024-09-10T19:20:03.221493+0000","mapping_epoch":217,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":218,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:19:19.581564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:19:19.581564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:20:20.362854+0000","scrub_duration":0.01041663,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,0],"acting":[5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1e","version":"80'15","reported_seq":231,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-10T19:20:03.226700+0000","last_change":"2024-09-10T19:16:33.258114+0000","last_active":"2024-09-10T19:20:03.226700+0000","last_peered":"2024-09-10T19:20:03.226700+0000","last_clean":"2024-09-10T19:20:03.226700+0000","last_became_active":"2024-09-10T19:10:09.823061+0000","last_became_peered":"2024-09-10T19:10:09.823061+0000","last_unstale":"2024-09-10T19:20:03.226700+0000","last_undegraded":"2024-09-10T19:20:03.226700+0000","last_fullsized":"2024-09-10T19:20:03.226700+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"80'15","last_scrub_stamp":"2024-09-10T19:16:33.258031+0000","last_deep_scrub":"80'15","last_deep_scrub_stamp":"2024-09-10T19:11:25.883919+0000","last_clean_scrub_stamp":"2024-09-10T19:16:33.258031+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0040337309999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1f","version":"66'8","reported_seq":343,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-10T19:20:03.225794+0000","last_change":"2024-09-10T19:17:32.477395+0000","last_active":"2024-09-10T19:20:03.225794+0000","last_peered":"2024-09-10T19:20:03.225794+0000","last_clean":"2024-09-10T19:20:03.225794+0000","last_became_active":"2024-09-10T19:10:11.695427+0000","last_became_peered":"2024-09-10T19:10:11.695427+0000","last_unstale":"2024-09-10T19:20:03.225794+0000","last_undegraded":"2024-09-10T19:20:03.225794+0000","last_fullsized":"2024-09-10T19:20:03.225794+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'8","last_scrub_stamp":"2024-09-10T19:15:53.664208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:15:53.664208+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001849699,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":184,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-10T19:20:03.221871+0000","last_change":"2024-09-10T19:19:58.544196+0000","last_active":"2024-09-10T19:20:03.221871+0000","last_peered":"2024-09-10T19:20:03.221871+0000","last_clean":"2024-09-10T19:20:03.221871+0000","last_became_active":"2024-09-10T19:16:24.862695+0000","last_became_peered":"2024-09-10T19:16:24.862695+0000","last_unstale":"2024-09-10T19:20:03.221871+0000","last_undegraded":"2024-09-10T19:20:03.221871+0000","last_fullsized":"2024-09-10T19:20:03.221871+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:19:58.544137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:19:58.544137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:21:27.203059+0000","scrub_duration":0.0013181899999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":214,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-10T19:20:03.221895+0000","last_change":"2024-09-10T19:16:41.881053+0000","last_active":"2024-09-10T19:20:03.221895+0000","last_peered":"2024-09-10T19:20:03.221895+0000","last_clean":"2024-09-10T19:20:03.221895+0000","last_became_active":"2024-09-10T19:10:07.682689+0000","last_became_peered":"2024-09-10T19:10:07.682689+0000","last_unstale":"2024-09-10T19:20:03.221895+0000","last_undegraded":"2024-09-10T19:20:03.221895+0000","last_fullsized":"2024-09-10T19:20:03.221895+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:16:41.880992+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:16:41.880992+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0044984990000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"80'16","reported_seq":296,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-10T19:20:03.225870+0000","last_change":"2024-09-10T19:20:01.500126+0000","last_active":"2024-09-10T19:20:03.225870+0000","last_peered":"2024-09-10T19:20:03.225870+0000","last_clean":"2024-09-10T19:20:03.225870+0000","last_became_active":"2024-09-10T19:13:54.713563+0000","last_became_peered":"2024-09-10T19:13:54.713563+0000","last_unstale":"2024-09-10T19:20:03.225870+0000","last_undegraded":"2024-09-10T19:20:03.225870+0000","last_fullsized":"2024-09-10T19:20:03.225870+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"80'16","last_scrub_stamp":"2024-09-10T19:20:01.500062+0000","last_deep_scrub":"80'16","last_deep_scrub_stamp":"2024-09-10T19:11:40.853611+0000","last_clean_scrub_stamp":"2024-09-10T19:20:01.500062+0000","objects_scrubbed":5,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:21:12.416074+0000","scrub_duration":0.002579296,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,0],"acting":[6,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":229,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-10T19:20:03.226630+0000","last_change":"2024-09-10T19:20:01.699843+0000","last_active":"2024-09-10T19:20:03.226630+0000","last_peered":"2024-09-10T19:20:03.226630+0000","last_clean":"2024-09-10T19:20:03.226630+0000","last_became_active":"2024-09-10T19:16:24.834713+0000","last_became_peered":"2024-09-10T19:16:24.834713+0000","last_unstale":"2024-09-10T19:20:03.226630+0000","last_undegraded":"2024-09-10T19:20:03.226630+0000","last_fullsized":"2024-09-10T19:20:03.226630+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:20:01.699734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:11:28.822327+0000","last_clean_scrub_stamp":"2024-09-10T19:20:01.699734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:21:27.520631+0000","scrub_duration":0.001702755,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":207,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-10T19:20:03.228787+0000","last_change":"2024-09-10T19:16:36.960796+0000","last_active":"2024-09-10T19:20:03.228787+0000","last_peered":"2024-09-10T19:20:03.228787+0000","last_clean":"2024-09-10T19:20:03.228787+0000","last_became_active":"2024-09-10T19:10:14.053619+0000","last_became_peered":"2024-09-10T19:10:14.053619+0000","last_unstale":"2024-09-10T19:20:03.228787+0000","last_undegraded":"2024-09-10T19:20:03.228787+0000","last_fullsized":"2024-09-10T19:20:03.228787+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:16:36.960695+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:16:36.960695+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.011430734999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":214,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-10T19:20:03.224197+0000","last_change":"2024-09-10T19:16:30.666084+0000","last_active":"2024-09-10T19:20:03.224197+0000","last_peered":"2024-09-10T19:20:03.224197+0000","last_clean":"2024-09-10T19:20:03.224197+0000","last_became_active":"2024-09-10T19:10:07.685997+0000","last_became_peered":"2024-09-10T19:10:07.685997+0000","last_unstale":"2024-09-10T19:20:03.224197+0000","last_undegraded":"2024-09-10T19:20:03.224197+0000","last_fullsized":"2024-09-10T19:20:03.224197+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:16:30.666001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:16:30.666001+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0041927830000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1c","version":"78'22","reported_seq":163,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-10T19:20:03.220492+0000","last_change":"2024-09-10T19:16:24.870445+0000","last_active":"2024-09-10T19:20:03.220492+0000","last_peered":"2024-09-10T19:20:03.220492+0000","last_clean":"2024-09-10T19:20:03.220492+0000","last_became_active":"2024-09-10T19:16:24.845764+0000","last_became_peered":"2024-09-10T19:16:24.845764+0000","last_unstale":"2024-09-10T19:20:03.220492+0000","last_undegraded":"2024-09-10T19:20:03.220492+0000","last_fullsized":"2024-09-10T19:20:03.220492+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"78'22","last_scrub_stamp":"2024-09-10T19:15:37.702624+0000","last_deep_scrub":"78'22","last_deep_scrub_stamp":"2024-09-10T19:11:21.118937+0000","last_clean_scrub_stamp":"2024-09-10T19:15:37.702624+0000","objects_scrubbed":7,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":167,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-10T19:20:03.228525+0000","last_change":"2024-09-10T19:15:33.008615+0000","last_active":"2024-09-10T19:20:03.228525+0000","last_peered":"2024-09-10T19:20:03.228525+0000","last_clean":"2024-09-10T19:20:03.228525+0000","last_became_active":"2024-09-10T19:13:54.709055+0000","last_became_peered":"2024-09-10T19:13:54.709055+0000","last_unstale":"2024-09-10T19:20:03.228525+0000","last_undegraded":"2024-09-10T19:20:03.228525+0000","last_fullsized":"2024-09-10T19:20:03.228525+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:15:33.008542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:15:33.008542+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.004557363,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,5],"acting":[4,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":208,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-10T19:20:03.223487+0000","last_change":"2024-09-10T19:19:14.274104+0000","last_active":"2024-09-10T19:20:03.223487+0000","last_peered":"2024-09-10T19:20:03.223487+0000","last_clean":"2024-09-10T19:20:03.223487+0000","last_became_active":"2024-09-10T19:10:14.053698+0000","last_became_peered":"2024-09-10T19:10:14.053698+0000","last_unstale":"2024-09-10T19:20:03.223487+0000","last_undegraded":"2024-09-10T19:20:03.223487+0000","last_fullsized":"2024-09-10T19:20:03.223487+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:19:14.274039+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:19:14.274039+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:20:19.701437+0000","scrub_duration":0.0039769159999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":214,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-10T19:20:03.220874+0000","last_change":"2024-09-10T19:19:15.244839+0000","last_active":"2024-09-10T19:20:03.220874+0000","last_peered":"2024-09-10T19:20:03.220874+0000","last_clean":"2024-09-10T19:20:03.220874+0000","last_became_active":"2024-09-10T19:10:07.682811+0000","last_became_peered":"2024-09-10T19:10:07.682811+0000","last_unstale":"2024-09-10T19:20:03.220874+0000","last_undegraded":"2024-09-10T19:20:03.220874+0000","last_fullsized":"2024-09-10T19:20:03.220874+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:19:15.244754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:19:15.244754+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:20:24.119370+0000","scrub_duration":0.0077546289999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1d","version":"66'12","reported_seq":254,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-10T19:20:03.219906+0000","last_change":"2024-09-10T19:15:52.923561+0000","last_active":"2024-09-10T19:20:03.219906+0000","last_peered":"2024-09-10T19:20:03.219906+0000","last_clean":"2024-09-10T19:20:03.219906+0000","last_became_active":"2024-09-10T19:13:54.714629+0000","last_became_peered":"2024-09-10T19:13:54.714629+0000","last_unstale":"2024-09-10T19:20:03.219906+0000","last_undegraded":"2024-09-10T19:20:03.219906+0000","last_fullsized":"2024-09-10T19:20:03.219906+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'12","last_scrub_stamp":"2024-09-10T19:15:52.923498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:15:52.923498+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0019450450000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":219,"reported_epoch":233,"state":"active+clean","last_fresh":"2024-09-10T19:20:03.226404+0000","last_change":"2024-09-10T19:16:37.970105+0000","last_active":"2024-09-10T19:20:03.226404+0000","last_peered":"2024-09-10T19:20:03.226404+0000","last_clean":"2024-09-10T19:20:03.226404+0000","last_became_active":"2024-09-10T19:16:24.843013+0000","last_became_peered":"2024-09-10T19:16:24.843013+0000","last_unstale":"2024-09-10T19:20:03.226404+0000","last_undegraded":"2024-09-10T19:20:03.226404+0000","last_fullsized":"2024-09-10T19:20:03.226404+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:16:37.970030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:16:37.970030+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.003625063,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":9,"num_pg":16,"stat_sum":{"num_bytes":763625472,"num_objects":11652,"num_object_clones":0,"num_object_copies":34956,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11652,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11652,"num_write_kb":745728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2277441536,"data_stored":2277441536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":11652,"ondisk_log_size":11652,"up":48,"acting":48,"num_store_stats":6},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":382,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":1528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":73,"ondisk_log_size":73,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":177,"num_object_clones":0,"num_object_copies":531,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":177,"num_whiteouts":0,"num_read":695,"num_read_kb":452,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":173,"num_bytes_recovered":3218,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":540672,"data_stored":14242,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":460,"ondisk_log_size":460,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":226,"num_read_kb":221,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":389,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":2002,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":9,"ondisk_log_size":9,"up":9,"acting":9,"num_store_stats":7},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1328,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":27,"num_read_kb":27,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":3984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4,"ondisk_log_size":4,"up":96,"acting":96,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":231,"num_write":113,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":590368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79,"ondisk_log_size":79,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":54,"seq":231928234108,"num_pgs":79,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1039940,"kb_used_data":450036,"kb_used_omap":1,"kb_used_meta":589886,"kb_avail":92705212,"statfs":{"total":95995035648,"available":94930137088,"internally_reserved":0,"allocated":460836864,"data_stored":460397365,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":604043608},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 19:19:05 2024","interfaces":[{"interface":"back","average":{"1min":1.1060000000000001,"5min":1.1259999999999999,"15min":1.093},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":4.5439999999999996,"5min":8.3770000000000007,"15min":35.551000000000002},"last":0.67800000000000005},{"interface":"front","average":{"1min":1.262,"5min":1.099,"15min":1.0149999999999999},"min":{"1min":0.245,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":4.5880000000000001,"5min":8.1440000000000001,"15min":35.503999999999998},"last":0.57099999999999995}]},{"osd":1,"last update":"Tue Sep 10 19:19:29 2024","interfaces":[{"interface":"back","average":{"1min":1.278,"5min":1.016,"15min":0.68799999999999994},"min":{"1min":0.25900000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":4.6589999999999998,"5min":8.3499999999999996,"15min":8.3499999999999996},"last":0.64500000000000002},{"interface":"front","average":{"1min":1.349,"5min":1.0069999999999999,"15min":0.80700000000000005},"min":{"1min":0.308,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":4.5810000000000004,"5min":5.609,"15min":5.609},"last":0.70599999999999996}]},{"osd":2,"last update":"Tue Sep 10 19:19:05 2024","interfaces":[{"interface":"back","average":{"1min":0.83899999999999997,"5min":0.81499999999999995,"15min":0.75900000000000001},"min":{"1min":0.28199999999999997,"5min":0.27300000000000002,"15min":0.25600000000000001},"max":{"1min":2.5569999999999999,"5min":4.21,"15min":31.940999999999999},"last":1.24},{"interface":"front","average":{"1min":1.085,"5min":1.177,"15min":0.91300000000000003},"min":{"1min":0.314,"5min":0.249,"15min":0.24099999999999999},"max":{"1min":4.609,"5min":8.0839999999999996,"15min":25.056000000000001},"last":1.01}]},{"osd":3,"last update":"Tue Sep 10 19:19:05 2024","interfaces":[{"interface":"back","average":{"1min":0.90000000000000002,"5min":1.272,"15min":1.119},"min":{"1min":0.33600000000000002,"5min":0.217,"15min":0.217},"max":{"1min":2.343,"5min":21.074999999999999,"15min":35.277999999999999},"last":1.617},{"interface":"front","average":{"1min":1.23,"5min":1.2969999999999999,"15min":1.137},"min":{"1min":0.25700000000000001,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":4.6349999999999998,"5min":18.82,"15min":36.512999999999998},"last":0.96499999999999997}]},{"osd":4,"last update":"Tue Sep 10 19:19:05 2024","interfaces":[{"interface":"back","average":{"1min":2.093,"5min":1.357,"15min":0.76100000000000001},"min":{"1min":0.23699999999999999,"5min":0.23300000000000001,"15min":0.22800000000000001},"max":{"1min":5.1150000000000002,"5min":8.1859999999999999,"15min":8.1859999999999999},"last":2.6080000000000001},{"interface":"front","average":{"1min":1.2849999999999999,"5min":0.90500000000000003,"15min":0.61399999999999999},"min":{"1min":0.23100000000000001,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":3.004,"5min":4.5469999999999997,"15min":4.5469999999999997},"last":0.93300000000000005}]},{"osd":5,"last update":"Tue Sep 10 19:19:05 2024","interfaces":[{"interface":"back","average":{"1min":3.4399999999999999,"5min":1.7629999999999999,"15min":0.94499999999999995},"min":{"1min":0.32600000000000001,"5min":0.253,"15min":0.23699999999999999},"max":{"1min":33.521000000000001,"5min":33.521000000000001,"15min":33.521000000000001},"last":0.92000000000000004},{"interface":"front","average":{"1min":3.492,"5min":1.694,"15min":0.89300000000000002},"min":{"1min":0.35199999999999998,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":34.438000000000002,"5min":34.438000000000002,"15min":34.438000000000002},"last":2.5750000000000002}]},{"osd":6,"last update":"Tue Sep 10 19:19:05 2024","interfaces":[{"interface":"back","average":{"1min":1.9370000000000001,"5min":1.472,"15min":0.82999999999999996},"min":{"1min":0.32300000000000001,"5min":0.317,"15min":0.26200000000000001},"max":{"1min":5.1109999999999998,"5min":8.2590000000000003,"15min":8.2590000000000003},"last":2.6320000000000001},{"interface":"front","average":{"1min":1.3160000000000001,"5min":1.1120000000000001,"15min":0.69499999999999995},"min":{"1min":0.29199999999999998,"5min":0.29199999999999998,"15min":0.28000000000000003},"max":{"1min":3.677,"5min":8.4130000000000003,"15min":8.4130000000000003},"last":0.996}]}]},{"osd":6,"up_from":47,"seq":201863463041,"num_pgs":72,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":512808,"kb_used_data":202972,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":93232344,"statfs":{"total":95995035648,"available":95469920256,"internally_reserved":0,"allocated":207843328,"data_stored":207420379,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 19:19:48 2024","interfaces":[{"interface":"back","average":{"1min":0.83399999999999996,"5min":0.86799999999999999,"15min":1.1279999999999999},"min":{"1min":0.23999999999999999,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":2.1619999999999999,"5min":12.334,"15min":112.988},"last":0.376},{"interface":"front","average":{"1min":1.0580000000000001,"5min":1.0629999999999999,"15min":1.1950000000000001},"min":{"1min":0.248,"5min":0.221,"15min":0.221},"max":{"1min":4.9039999999999999,"5min":12.319000000000001,"15min":113.18300000000001},"last":0.65100000000000002}]},{"osd":1,"last update":"Tue Sep 10 19:19:30 2024","interfaces":[{"interface":"back","average":{"1min":0.997,"5min":0.70899999999999996,"15min":0.57899999999999996},"min":{"1min":0.32900000000000001,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":2.2360000000000002,"5min":4.2919999999999998,"15min":4.2919999999999998},"last":0.438},{"interface":"front","average":{"1min":1.3580000000000001,"5min":0.92100000000000004,"15min":0.74299999999999999},"min":{"1min":0.30099999999999999,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":6.5289999999999999,"5min":6.5289999999999999,"15min":6.5289999999999999},"last":0.30399999999999999}]},{"osd":2,"last update":"Tue Sep 10 19:19:48 2024","interfaces":[{"interface":"back","average":{"1min":1.036,"5min":0.83699999999999997,"15min":1.0900000000000001},"min":{"1min":0.40200000000000002,"5min":0.25,"15min":0.20399999999999999},"max":{"1min":2.778,"5min":3.7959999999999998,"15min":112.864},"last":0.47399999999999998},{"interface":"front","average":{"1min":1.3220000000000001,"5min":1.004,"15min":1.1599999999999999},"min":{"1min":0.26500000000000001,"5min":0.254,"15min":0.254},"max":{"1min":5.8300000000000001,"5min":5.8300000000000001,"15min":112.822},"last":0.35199999999999998}]},{"osd":3,"last update":"Tue Sep 10 19:19:48 2024","interfaces":[{"interface":"back","average":{"1min":1.278,"5min":1.4470000000000001,"15min":1.357},"min":{"1min":0.27300000000000002,"5min":0.251,"15min":0.251},"max":{"1min":6.4870000000000001,"5min":23.824999999999999,"15min":125.56399999999999},"last":0.45600000000000002},{"interface":"front","average":{"1min":1.0309999999999999,"5min":1.389,"15min":1.389},"min":{"1min":0.41899999999999998,"5min":0.254,"15min":0.23200000000000001},"max":{"1min":2.7389999999999999,"5min":24.814,"15min":125.47},"last":0.498}]},{"osd":4,"last update":"Tue Sep 10 19:19:48 2024","interfaces":[{"interface":"back","average":{"1min":2.0910000000000002,"5min":1.3360000000000001,"15min":0.71199999999999997},"min":{"1min":0.29999999999999999,"5min":0.27600000000000002,"15min":0.20699999999999999},"max":{"1min":5.9390000000000001,"5min":6.9269999999999996,"15min":6.9269999999999996},"last":0.39200000000000002},{"interface":"front","average":{"1min":1.851,"5min":1.147,"15min":0.67000000000000004},"min":{"1min":0.374,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":6.7279999999999998,"5min":6.7279999999999998,"15min":6.7279999999999998},"last":0.27400000000000002}]},{"osd":5,"last update":"Tue Sep 10 19:19:48 2024","interfaces":[{"interface":"back","average":{"1min":1.8320000000000001,"5min":1.387,"15min":0.90100000000000002},"min":{"1min":0.33200000000000002,"5min":0.30499999999999999,"15min":0.248},"max":{"1min":5.9640000000000004,"5min":5.9640000000000004,"15min":44.765000000000001},"last":0.32200000000000001},{"interface":"front","average":{"1min":2.1890000000000001,"5min":1.462,"15min":0.94699999999999995},"min":{"1min":0.33600000000000002,"5min":0.28899999999999998,"15min":0.27500000000000002},"max":{"1min":7.1369999999999996,"5min":7.1369999999999996,"15min":43.036000000000001},"last":1.8660000000000001}]},{"osd":7,"last update":"Tue Sep 10 19:19:11 2024","interfaces":[{"interface":"back","average":{"1min":1.452,"5min":1.1180000000000001,"15min":0.67800000000000005},"min":{"1min":0.39000000000000001,"5min":0.30599999999999999,"15min":0.27500000000000002},"max":{"1min":4.1379999999999999,"5min":8.9390000000000001,"15min":8.9390000000000001},"last":0.40999999999999998},{"interface":"front","average":{"1min":1.6870000000000001,"5min":1.28,"15min":0.752},"min":{"1min":0.42099999999999999,"5min":0.31900000000000001,"15min":0.29299999999999998},"max":{"1min":4.9029999999999996,"5min":10.298,"15min":10.298},"last":1.0629999999999999}]}]},{"osd":5,"up_from":39,"seq":167503724678,"num_pgs":74,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":735260,"kb_used_data":429584,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":93009892,"statfs":{"total":95995035648,"available":95242129408,"internally_reserved":0,"allocated":439894016,"data_stored":439483652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,3,2],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 19:19:20 2024","interfaces":[{"interface":"back","average":{"1min":4.8390000000000004,"5min":1.9299999999999999,"15min":1.01},"min":{"1min":0.19900000000000001,"5min":0.17999999999999999,"15min":0.17999999999999999},"max":{"1min":73.933000000000007,"5min":73.933000000000007,"15min":73.933000000000007},"last":0.52100000000000002},{"interface":"front","average":{"1min":4.952,"5min":1.742,"15min":0.94499999999999995},"min":{"1min":0.19900000000000001,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":73.769999999999996,"5min":73.769999999999996,"15min":73.769999999999996},"last":2.0630000000000002}]},{"osd":1,"last update":"Tue Sep 10 19:19:24 2024","interfaces":[{"interface":"back","average":{"1min":1.454,"5min":0.96899999999999997,"15min":0.75700000000000001},"min":{"1min":0.26600000000000001,"5min":0.249,"15min":0.249},"max":{"1min":4.3499999999999996,"5min":4.3499999999999996,"15min":4.3499999999999996},"last":2.177},{"interface":"front","average":{"1min":1.3109999999999999,"5min":0.90300000000000002,"15min":0.63600000000000001},"min":{"1min":0.26500000000000001,"5min":0.255,"15min":0.255},"max":{"1min":3.8959999999999999,"5min":5.0570000000000004,"15min":5.0570000000000004},"last":1}]},{"osd":2,"last update":"Tue Sep 10 19:19:20 2024","interfaces":[{"interface":"back","average":{"1min":1.0569999999999999,"5min":1.137,"15min":0.73699999999999999},"min":{"1min":0.27600000000000002,"5min":0.26100000000000001,"15min":0.253},"max":{"1min":3.6819999999999999,"5min":9.9399999999999995,"15min":9.9399999999999995},"last":0.57899999999999996},{"interface":"front","average":{"1min":1.3700000000000001,"5min":1.083,"15min":0.76900000000000002},"min":{"1min":0.307,"5min":0.24399999999999999,"15min":0.224},"max":{"1min":4.4139999999999997,"5min":4.4139999999999997,"15min":6.3789999999999996},"last":2.2050000000000001}]},{"osd":3,"last update":"Tue Sep 10 19:19:20 2024","interfaces":[{"interface":"back","average":{"1min":1.04,"5min":1.113,"15min":0.74299999999999999},"min":{"1min":0.23300000000000001,"5min":0.224,"15min":0.224},"max":{"1min":3.6619999999999999,"5min":9.9019999999999992,"15min":9.9019999999999992},"last":0.60499999999999998},{"interface":"front","average":{"1min":1.575,"5min":1.1319999999999999,"15min":0.75600000000000001},"min":{"1min":0.28899999999999998,"5min":0.27800000000000002,"15min":0.248},"max":{"1min":4.4459999999999997,"5min":5.3869999999999996,"15min":5.3869999999999996},"last":2.1379999999999999}]},{"osd":4,"last update":"Tue Sep 10 19:19:20 2024","interfaces":[{"interface":"back","average":{"1min":1.538,"5min":1.2909999999999999,"15min":0.70699999999999996},"min":{"1min":0.27400000000000002,"5min":0.24399999999999999,"15min":0.20699999999999999},"max":{"1min":4.1520000000000001,"5min":10.414,"15min":10.414},"last":0.46999999999999997},{"interface":"front","average":{"1min":1.3640000000000001,"5min":1.2729999999999999,"15min":0.73399999999999999},"min":{"1min":0.23100000000000001,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":4.1870000000000003,"5min":10.518000000000001,"15min":10.518000000000001},"last":0.85999999999999999}]},{"osd":6,"last update":"Tue Sep 10 19:19:47 2024","interfaces":[{"interface":"back","average":{"1min":1.8220000000000001,"5min":1.478,"15min":0.85999999999999999},"min":{"1min":0.36199999999999999,"5min":0.29099999999999998,"15min":0.223},"max":{"1min":5.5899999999999999,"5min":9.9670000000000005,"15min":9.9670000000000005},"last":0.55300000000000005},{"interface":"front","average":{"1min":1.7350000000000001,"5min":1.3859999999999999,"15min":0.81499999999999995},"min":{"1min":0.32700000000000001,"5min":0.32100000000000001,"15min":0.25},"max":{"1min":4.2750000000000004,"5min":10.446,"15min":10.446},"last":2.1070000000000002}]},{"osd":7,"last update":"Tue Sep 10 19:19:10 2024","interfaces":[{"interface":"back","average":{"1min":1.798,"5min":1.3400000000000001,"15min":0.79700000000000004},"min":{"1min":0.32400000000000001,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":4.6829999999999998,"5min":5.9660000000000002,"15min":5.9660000000000002},"last":2.2280000000000002},{"interface":"front","average":{"1min":1.552,"5min":1.248,"15min":0.745},"min":{"1min":0.31900000000000001,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":4.9829999999999997,"5min":6.8840000000000003,"15min":6.8840000000000003},"last":1.244}]}]},{"osd":4,"up_from":32,"seq":137438953610,"num_pgs":66,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":499472,"kb_used_data":202120,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93245680,"statfs":{"total":95995035648,"available":95483576320,"internally_reserved":0,"allocated":206970880,"data_stored":206566844,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 19:19:06 2024","interfaces":[{"interface":"back","average":{"1min":0.92300000000000004,"5min":1.536,"15min":0.875},"min":{"1min":0.216,"5min":0.187,"15min":0.187},"max":{"1min":3.7330000000000001,"5min":32.533000000000001,"15min":32.533000000000001},"last":0.47599999999999998},{"interface":"front","average":{"1min":0.95699999999999996,"5min":1.6100000000000001,"15min":0.89500000000000002},"min":{"1min":0.219,"5min":0.219,"15min":0.20999999999999999},"max":{"1min":4.6769999999999996,"5min":32.512999999999998,"15min":32.512999999999998},"last":0.55500000000000005}]},{"osd":1,"last update":"Tue Sep 10 19:19:26 2024","interfaces":[{"interface":"back","average":{"1min":1.0960000000000001,"5min":0.95199999999999996,"15min":0.73599999999999999},"min":{"1min":0.24299999999999999,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":4.5380000000000003,"5min":7.9770000000000003,"15min":7.9770000000000003},"last":0.38300000000000001},{"interface":"front","average":{"1min":1.1339999999999999,"5min":0.83399999999999996,"15min":0.72799999999999998},"min":{"1min":0.27800000000000002,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":3.774,"5min":3.8260000000000001,"15min":3.8260000000000001},"last":0.34499999999999997}]},{"osd":2,"last update":"Tue Sep 10 19:19:06 2024","interfaces":[{"interface":"back","average":{"1min":0.98999999999999999,"5min":0.95099999999999996,"15min":0.68600000000000005},"min":{"1min":0.254,"5min":0.214,"15min":0.214},"max":{"1min":3.798,"5min":6.8869999999999996,"15min":6.8869999999999996},"last":0.436},{"interface":"front","average":{"1min":1.0660000000000001,"5min":0.94299999999999995,"15min":0.68200000000000005},"min":{"1min":0.28699999999999998,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":4.7169999999999996,"5min":7.085,"15min":7.085},"last":0.58199999999999996}]},{"osd":3,"last update":"Tue Sep 10 19:19:06 2024","interfaces":[{"interface":"back","average":{"1min":1.1739999999999999,"5min":1.3879999999999999,"15min":0.83599999999999997},"min":{"1min":0.26100000000000001,"5min":0.26100000000000001,"15min":0.252},"max":{"1min":4.8479999999999999,"5min":35.451999999999998,"15min":35.451999999999998},"last":0.89900000000000002},{"interface":"front","average":{"1min":1.3340000000000001,"5min":1.6120000000000001,"15min":0.93100000000000005},"min":{"1min":0.30099999999999999,"5min":0.26300000000000001,"15min":0.26200000000000001},"max":{"1min":4.5060000000000002,"5min":32.880000000000003,"15min":32.880000000000003},"last":0.65700000000000003}]},{"osd":5,"last update":"Tue Sep 10 19:19:26 2024","interfaces":[{"interface":"back","average":{"1min":1.702,"5min":1.9930000000000001,"15min":0.93799999999999994},"min":{"1min":0.224,"5min":0.224,"15min":0.224},"max":{"1min":4.5839999999999996,"5min":67.379000000000005,"15min":67.379000000000005},"last":0.46500000000000002},{"interface":"front","average":{"1min":1.907,"5min":1.7549999999999999,"15min":0.876},"min":{"1min":0.33500000000000002,"5min":0.249,"15min":0.223},"max":{"1min":5.601,"5min":67.542000000000002,"15min":67.542000000000002},"last":2.1749999999999998}]},{"osd":6,"last update":"Tue Sep 10 19:19:59 2024","interfaces":[{"interface":"back","average":{"1min":1.159,"5min":1.0640000000000001,"15min":0.69199999999999995},"min":{"1min":0.312,"5min":0.27400000000000002,"15min":0.253},"max":{"1min":3.1360000000000001,"5min":5.5289999999999999,"15min":5.5289999999999999},"last":0.78900000000000003},{"interface":"front","average":{"1min":1.2090000000000001,"5min":1.1579999999999999,"15min":0.70599999999999996},"min":{"1min":0.25800000000000001,"5min":0.247,"15min":0.247},"max":{"1min":7.3390000000000004,"5min":7.9550000000000001,"15min":7.9550000000000001},"last":0.57199999999999995}]},{"osd":7,"last update":"Tue Sep 10 19:19:12 2024","interfaces":[{"interface":"back","average":{"1min":1.2190000000000001,"5min":1.2030000000000001,"15min":0.85099999999999998},"min":{"1min":0.32800000000000001,"5min":0.28100000000000003,"15min":0.27700000000000002},"max":{"1min":4.5629999999999997,"5min":10.33,"15min":23.081},"last":0.497},{"interface":"front","average":{"1min":1.746,"5min":1.232,"15min":0.85399999999999998},"min":{"1min":0.29099999999999998,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":8.8550000000000004,"5min":8.8550000000000004,"15min":22.666},"last":0.51700000000000002}]}]},{"osd":3,"up_from":26,"seq":111669149839,"num_pgs":78,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":951980,"kb_used_data":357924,"kb_used_omap":1,"kb_used_meta":594046,"kb_avail":92793172,"statfs":{"total":95995035648,"available":95020208128,"internally_reserved":0,"allocated":366514176,"data_stored":366083521,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":608303448},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,1,1],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 19:19:41 2024","interfaces":[{"interface":"back","average":{"1min":0.35299999999999998,"5min":0.76300000000000001,"15min":0.69099999999999995},"min":{"1min":0.16,"5min":0.157,"15min":0.157},"max":{"1min":1.0129999999999999,"5min":17.518000000000001,"15min":17.518000000000001},"last":0.31},{"interface":"front","average":{"1min":0.39400000000000002,"5min":0.95499999999999996,"15min":0.82699999999999996},"min":{"1min":0.20899999999999999,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":1.0149999999999999,"5min":17.609000000000002,"15min":17.609000000000002},"last":0.35499999999999998}]},{"osd":1,"last update":"Tue Sep 10 19:19:24 2024","interfaces":[{"interface":"back","average":{"1min":0.377,"5min":0.47699999999999998,"15min":0.51900000000000002},"min":{"1min":0.20499999999999999,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":1.149,"5min":2.6059999999999999,"15min":2.6059999999999999},"last":0.374},{"interface":"front","average":{"1min":0.46600000000000003,"5min":0.94999999999999996,"15min":1.3620000000000001},"min":{"1min":0.26800000000000002,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":1.768,"5min":17.454999999999998,"15min":17.454999999999998},"last":0.622}]},{"osd":2,"last update":"Tue Sep 10 19:19:41 2024","interfaces":[{"interface":"back","average":{"1min":0.44800000000000001,"5min":0.76200000000000001,"15min":0.64800000000000002},"min":{"1min":0.24099999999999999,"5min":0.218,"15min":0.186},"max":{"1min":1.1819999999999999,"5min":5.3029999999999999,"15min":5.3029999999999999},"last":0.45400000000000001},{"interface":"front","average":{"1min":0.41499999999999998,"5min":0.85399999999999998,"15min":0.71299999999999997},"min":{"1min":0.247,"5min":0.23599999999999999,"15min":0.19800000000000001},"max":{"1min":0.90600000000000003,"5min":17.806000000000001,"15min":17.806000000000001},"last":0.47499999999999998}]},{"osd":4,"last update":"Tue Sep 10 19:19:09 2024","interfaces":[{"interface":"back","average":{"1min":0.96799999999999997,"5min":1.085,"15min":0.85699999999999998},"min":{"1min":0.28000000000000003,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":2.339,"5min":17.834,"15min":17.834},"last":0.54700000000000004},{"interface":"front","average":{"1min":0.98299999999999998,"5min":1.357,"15min":0.97699999999999998},"min":{"1min":0.28299999999999997,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":3.339,"5min":17.553000000000001,"15min":17.553000000000001},"last":0.65000000000000002}]},{"osd":5,"last update":"Tue Sep 10 19:19:24 2024","interfaces":[{"interface":"back","average":{"1min":0.85899999999999999,"5min":1.8680000000000001,"15min":1.1040000000000001},"min":{"1min":0.27900000000000003,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":3.2709999999999999,"5min":80.942999999999998,"15min":80.942999999999998},"last":1.2170000000000001},{"interface":"front","average":{"1min":1.28,"5min":2.2240000000000002,"15min":1.2729999999999999},"min":{"1min":0.41099999999999998,"5min":0.22,"15min":0.22},"max":{"1min":4.835,"5min":81.489000000000004,"15min":81.489000000000004},"last":1.3080000000000001}]},{"osd":6,"last update":"Tue Sep 10 19:19:49 2024","interfaces":[{"interface":"back","average":{"1min":1.137,"5min":1.21,"15min":0.93999999999999995},"min":{"1min":0.30499999999999999,"5min":0.30499999999999999,"15min":0.30399999999999999},"max":{"1min":5.1399999999999997,"5min":17.861000000000001,"15min":17.861000000000001},"last":0.57299999999999995},{"interface":"front","average":{"1min":1.0489999999999999,"5min":1.157,"15min":0.82799999999999996},"min":{"1min":0.33900000000000002,"5min":0.311,"15min":0.26800000000000002},"max":{"1min":2.302,"5min":17.687000000000001,"15min":17.687000000000001},"last":0.63600000000000001}]},{"osd":7,"last update":"Tue Sep 10 19:19:19 2024","interfaces":[{"interface":"back","average":{"1min":1.1819999999999999,"5min":1.264,"15min":0.90300000000000002},"min":{"1min":0.376,"5min":0.31,"15min":0.29399999999999998},"max":{"1min":3.3969999999999998,"5min":18.093,"15min":18.093},"last":0.75900000000000001},{"interface":"front","average":{"1min":1.835,"5min":1.341,"15min":0.97999999999999998},"min":{"1min":0.39100000000000001,"5min":0.35599999999999998,"15min":0.34100000000000003},"max":{"1min":10.359999999999999,"5min":17.780999999999999,"15min":17.780999999999999},"last":0.69499999999999995}]}]},{"osd":2,"up_from":0,"seq":77309411476,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":0,"seq":592705486906,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":8,"seq":34359738526,"num_pgs":75,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":654500,"kb_used_data":344668,"kb_used_omap":1,"kb_used_meta":309822,"kb_avail":93090652,"statfs":{"total":95995035648,"available":95324827648,"internally_reserved":0,"allocated":352940032,"data_stored":352517499,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":317258072},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Sep 10 19:19:30 2024","interfaces":[{"interface":"back","average":{"1min":0.60399999999999998,"5min":0.40999999999999998,"15min":0.38200000000000001},"min":{"1min":0.20100000000000001,"5min":0.19,"15min":0.19},"max":{"1min":2.0819999999999999,"5min":2.0819999999999999,"15min":2.0819999999999999},"last":0.41899999999999998},{"interface":"front","average":{"1min":0.56699999999999995,"5min":0.41699999999999998,"15min":0.42199999999999999},"min":{"1min":0.21199999999999999,"5min":0.193,"15min":0.193},"max":{"1min":2.2839999999999998,"5min":2.2839999999999998,"15min":2.2839999999999998},"last":0.443}]},{"osd":2,"last update":"Tue Sep 10 19:19:30 2024","interfaces":[{"interface":"back","average":{"1min":0.68300000000000005,"5min":0.78700000000000003,"15min":0.61599999999999999},"min":{"1min":0.24299999999999999,"5min":0.22700000000000001,"15min":0.17199999999999999},"max":{"1min":2.0510000000000002,"5min":6.4640000000000004,"15min":6.4640000000000004},"last":0.48299999999999998},{"interface":"front","average":{"1min":0.61299999999999999,"5min":0.82699999999999996,"15min":0.59999999999999998},"min":{"1min":0.22600000000000001,"5min":0.22600000000000001,"15min":0.187},"max":{"1min":2.8660000000000001,"5min":9.9399999999999995,"15min":9.9399999999999995},"last":0.34599999999999997}]},{"osd":3,"last update":"Tue Sep 10 19:19:48 2024","interfaces":[{"interface":"back","average":{"1min":0.87,"5min":0.89700000000000002,"15min":0.67400000000000004},"min":{"1min":0.20300000000000001,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":3.6190000000000002,"5min":6.5590000000000002,"15min":6.5590000000000002},"last":0.39900000000000002},{"interface":"front","average":{"1min":0.505,"5min":0.878,"15min":0.70499999999999996},"min":{"1min":0.29499999999999998,"5min":0.26200000000000001,"15min":0.23000000000000001},"max":{"1min":1.024,"5min":7.2140000000000004,"15min":7.2140000000000004},"last":0.27800000000000002}]},{"osd":4,"last update":"Tue Sep 10 19:20:03 2024","interfaces":[{"interface":"back","average":{"1min":1.278,"5min":0.94299999999999995,"15min":0.75900000000000001},"min":{"1min":0.43099999999999999,"5min":0.29099999999999998,"15min":0.24399999999999999},"max":{"1min":7.4560000000000004,"5min":7.4560000000000004,"15min":7.4560000000000004},"last":0.52500000000000002},{"interface":"front","average":{"1min":1.4019999999999999,"5min":0.95199999999999996,"15min":0.75700000000000001},"min":{"1min":0.28499999999999998,"5min":0.27900000000000003,"15min":0.27100000000000002},"max":{"1min":5.2309999999999999,"5min":5.2640000000000002,"15min":5.2640000000000002},"last":0.46000000000000002}]},{"osd":5,"last update":"Tue Sep 10 19:19:30 2024","interfaces":[{"interface":"back","average":{"1min":1.5149999999999999,"5min":1.1399999999999999,"15min":0.81699999999999995},"min":{"1min":0.27100000000000002,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":4.327,"5min":5.3860000000000001,"15min":5.3860000000000001},"last":0.75600000000000001},{"interface":"front","average":{"1min":1.2150000000000001,"5min":0.96999999999999997,"15min":0.76400000000000001},"min":{"1min":0.33900000000000002,"5min":0.26000000000000001,"15min":0.253},"max":{"1min":3.5899999999999999,"5min":5.9749999999999996,"15min":5.9749999999999996},"last":1.448}]},{"osd":6,"last update":"Tue Sep 10 19:19:48 2024","interfaces":[{"interface":"back","average":{"1min":1.2629999999999999,"5min":0.93999999999999995,"15min":0.79700000000000004},"min":{"1min":0.27200000000000002,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":4.9420000000000002,"5min":4.9420000000000002,"15min":5.8730000000000002},"last":0.55400000000000005},{"interface":"front","average":{"1min":1.1339999999999999,"5min":0.91600000000000004,"15min":0.79900000000000004},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.28699999999999998},"max":{"1min":2.9300000000000002,"5min":4.8300000000000001,"15min":5.2850000000000001},"last":0.434}]},{"osd":7,"last update":"Tue Sep 10 19:19:15 2024","interfaces":[{"interface":"back","average":{"1min":1.0529999999999999,"5min":0.98199999999999998,"15min":0.77300000000000002},"min":{"1min":0.36299999999999999,"5min":0.318,"15min":0.318},"max":{"1min":3.7120000000000002,"5min":5.1479999999999997,"15min":5.1479999999999997},"last":3.0459999999999998},{"interface":"front","average":{"1min":0.71199999999999997,"5min":1.0760000000000001,"15min":0.77700000000000002},"min":{"1min":0.35199999999999998,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":1.3240000000000001,"5min":6.0469999999999997,"15min":6.0469999999999997},"last":0.62}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":966,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1236,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1236,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":362,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":1760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":2118,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1018,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":2118,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1540,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1942,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":2162,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":391118848,"data_stored":391118848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":430899200,"data_stored":430899200,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":232062976,"data_stored":232062976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":525664256,"data_stored":525664256,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":229900288,"data_stored":229900288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":467795968,"data_stored":467795968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T19:20:11.153 INFO:tasks.cephadm.ceph_manager.ceph:increase pool size by 10 2024-09-10T19:20:11.153 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph osd pool set .mgr pg_num 11 2024-09-10T19:20:11.154 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:20:11.186 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:20:11.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:20:11.187 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:20:11.288 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:20:11.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:20:11.322 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:20:11.322 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:20:11.423 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:20:11.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:20:11 smithi183 ceph-mon[29736]: 2.0 scrub starts 2024-09-10T19:20:11.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:20:11 smithi183 ceph-mon[29736]: 2.0 scrub ok 2024-09-10T19:20:11.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:20:11 smithi183 ceph-mon[41562]: 2.0 scrub starts 2024-09-10T19:20:11.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:20:11 smithi183 ceph-mon[41562]: 2.0 scrub ok 2024-09-10T19:20:11.432 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Sep 10 19:20:11 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-y[29989]: ::ffff:172.21.15.191 - - [10/Sep/2024:19:20:11] "GET /metrics HTTP/1.1" 200 224108 "" "Prometheus/2.43.0" 2024-09-10T19:20:11.461 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:20:11.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:20:11.462 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:20:11.563 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:20:11.599 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:20:11.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:20:11.601 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:20:11.701 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:20:11.715 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:20:11 smithi191 ceph-mon[32884]: 2.0 scrub starts 2024-09-10T19:20:11.715 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:20:11 smithi191 ceph-mon[32884]: 2.0 scrub ok 2024-09-10T19:20:11.730 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: 10 16 28975 28959 180.972 144.938 0.00713959 0.00552 2024-09-10T19:20:11.737 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:20:11.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:20:11.738 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:20:11.839 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:20:11.871 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:20:11.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:20:11.872 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:20:11.973 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:20:12.011 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:20:12.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:20:12.012 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:20:12.113 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:20:12.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:20:12.145 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:20:12.145 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:20:12.246 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:20:12.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:20:12.281 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:20:12.281 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:20:12.382 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:20:12.413 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:20:12.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:20:12.414 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:20:12.515 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:20:12.543 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:20:12.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:20:12.544 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:20:12.645 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:20:12.681 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:20:12.681 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:20:12 smithi183 ceph-mon[29736]: pgmap v450: 148 pgs: 148 active+clean; 1.5 GiB data, 6.7 GiB used, 530 GiB / 536 GiB avail; 1.4 KiB/s rd, 223 MiB/s wr, 3.57k op/s 2024-09-10T19:20:12.682 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:20:12 smithi183 ceph-mon[41562]: pgmap v450: 148 pgs: 148 active+clean; 1.5 GiB data, 6.7 GiB used, 530 GiB / 536 GiB avail; 1.4 KiB/s rd, 223 MiB/s wr, 3.57k op/s 2024-09-10T19:20:12.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:20:12.682 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:20:12.715 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:20:12 smithi191 ceph-mon[32884]: pgmap v450: 148 pgs: 148 active+clean; 1.5 GiB data, 6.7 GiB used, 530 GiB / 536 GiB avail; 1.4 KiB/s rd, 223 MiB/s wr, 3.57k op/s 2024-09-10T19:20:12.730 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: 11 16 31016 31000 176.116 127.562 0.00890976 0.00567235 2024-09-10T19:20:12.784 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:20:12.805 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:20:12.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:20:12.806 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:20:12.906 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:20:12.941 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:20:12.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:20:12.942 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:20:13.043 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:20:13.076 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:20:13.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:20:13.077 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:20:13.178 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:20:13.210 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:20:13.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:20:13.211 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:20:13.312 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:20:13.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:20:13.339 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:20:13.339 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:20:13.440 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:20:13.475 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:20:13.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:20:13.476 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:20:13.523 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:20:13.577 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:20:13.606 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:20:13.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:20:13.607 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:20:13.639 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:20:13 smithi183 ceph-mon[29736]: 6.1c scrub starts 2024-09-10T19:20:13.639 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:20:13 smithi183 ceph-mon[29736]: 6.1c scrub ok 2024-09-10T19:20:13.639 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:20:13 smithi183 ceph-mon[41562]: 6.1c scrub starts 2024-09-10T19:20:13.640 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:20:13 smithi183 ceph-mon[41562]: 6.1c scrub ok 2024-09-10T19:20:13.708 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:20:13.715 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:20:13 smithi191 ceph-mon[32884]: 6.1c scrub starts 2024-09-10T19:20:13.715 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:20:13 smithi191 ceph-mon[32884]: 6.1c scrub ok 2024-09-10T19:20:13.730 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: 12 16 32840 32824 170.939 114 0.0107447 0.00584423 2024-09-10T19:20:13.744 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:20:13.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:20:13.745 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:20:13.846 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:20:13.883 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:20:13.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:20:13.884 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:20:13.984 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:20:14.007 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:20:14.028 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:20:14.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:20:14.029 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:20:14.130 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:20:14.167 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:20:14.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:20:14.168 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:20:14.269 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:20:14.308 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:20:14.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:20:14.309 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:20:14.410 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:20:14.445 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:20:14.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:20:14.446 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:20:14.537 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:20:14 smithi183 ceph-mon[29736]: 4.9 scrub starts 2024-09-10T19:20:14.537 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:20:14 smithi183 ceph-mon[29736]: 4.9 scrub ok 2024-09-10T19:20:14.537 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:20:14 smithi183 ceph-mon[29736]: 5.0 scrub starts 2024-09-10T19:20:14.537 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:20:14 smithi183 ceph-mon[29736]: 5.0 scrub ok 2024-09-10T19:20:14.537 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:20:14 smithi183 ceph-mon[29736]: 3.0 scrub starts 2024-09-10T19:20:14.538 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:20:14 smithi183 ceph-mon[29736]: 3.0 scrub ok 2024-09-10T19:20:14.538 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:20:14 smithi183 ceph-mon[29736]: from='mgr.14556 ' entity='mgr.y' 2024-09-10T19:20:14.538 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:20:14 smithi183 ceph-mon[29736]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:20:14.538 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:20:14 smithi183 ceph-mon[29736]: pgmap v451: 148 pgs: 148 active+clean; 1.5 GiB data, 6.7 GiB used, 530 GiB / 536 GiB avail; 639 B/s rd, 104 MiB/s wr, 1.66k op/s 2024-09-10T19:20:14.538 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:20:14 smithi183 ceph-mon[41562]: 4.9 scrub starts 2024-09-10T19:20:14.538 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:20:14 smithi183 ceph-mon[41562]: 4.9 scrub ok 2024-09-10T19:20:14.538 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:20:14 smithi183 ceph-mon[41562]: 5.0 scrub starts 2024-09-10T19:20:14.538 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:20:14 smithi183 ceph-mon[41562]: 5.0 scrub ok 2024-09-10T19:20:14.538 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:20:14 smithi183 ceph-mon[41562]: 3.0 scrub starts 2024-09-10T19:20:14.538 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:20:14 smithi183 ceph-mon[41562]: 3.0 scrub ok 2024-09-10T19:20:14.538 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:20:14 smithi183 ceph-mon[41562]: from='mgr.14556 ' entity='mgr.y' 2024-09-10T19:20:14.539 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:20:14 smithi183 ceph-mon[41562]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:20:14.539 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:20:14 smithi183 ceph-mon[41562]: pgmap v451: 148 pgs: 148 active+clean; 1.5 GiB data, 6.7 GiB used, 530 GiB / 536 GiB avail; 639 B/s rd, 104 MiB/s wr, 1.66k op/s 2024-09-10T19:20:14.547 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:20:14.577 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:20:14.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:20:14.578 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:20:14.679 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:20:14.714 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:20:14.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:20:14.716 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:20:14.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:20:14 smithi191 ceph-mon[32884]: 4.9 scrub starts 2024-09-10T19:20:14.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:20:14 smithi191 ceph-mon[32884]: 4.9 scrub ok 2024-09-10T19:20:14.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:20:14 smithi191 ceph-mon[32884]: 5.0 scrub starts 2024-09-10T19:20:14.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:20:14 smithi191 ceph-mon[32884]: 5.0 scrub ok 2024-09-10T19:20:14.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:20:14 smithi191 ceph-mon[32884]: 3.0 scrub starts 2024-09-10T19:20:14.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:20:14 smithi191 ceph-mon[32884]: 3.0 scrub ok 2024-09-10T19:20:14.717 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:20:14 smithi191 ceph-mon[32884]: from='mgr.14556 ' entity='mgr.y' 2024-09-10T19:20:14.717 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:20:14 smithi191 ceph-mon[32884]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:20:14.717 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:20:14 smithi191 ceph-mon[32884]: pgmap v451: 148 pgs: 148 active+clean; 1.5 GiB data, 6.7 GiB used, 530 GiB / 536 GiB avail; 639 B/s rd, 104 MiB/s wr, 1.66k op/s 2024-09-10T19:20:14.730 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: 13 16 34908 34892 167.732 129.25 0.00674863 0.00595607 2024-09-10T19:20:14.817 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:20:14.850 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:20:14.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:20:14.851 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:20:14.952 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:20:14.989 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:20:14.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:20:14.990 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:20:15.091 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:20:15.125 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:20:15.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:20:15.126 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:20:15.226 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:20:15.261 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:20:15.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:20:15.262 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:20:15.306 INFO:teuthology.orchestra.run.smithi183.stderr:noscrub is set 2024-09-10T19:20:15.362 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:20:15.391 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:20:15.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:20:15.392 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:20:15.493 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:20:15.521 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:20:15.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:20:15.523 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:20:15.623 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:20:15.654 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:20:15 smithi183 ceph-mon[41562]: 2.7 scrub starts 2024-09-10T19:20:15.654 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:20:15 smithi183 ceph-mon[41562]: 2.7 scrub ok 2024-09-10T19:20:15.654 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:20:15 smithi183 ceph-mon[41562]: from='client.? 172.21.15.183:0/3009336162' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-10T19:20:15.654 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:20:15 smithi183 ceph-mon[41562]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-10T19:20:15.654 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:20:15 smithi183 ceph-mon[29736]: 2.7 scrub starts 2024-09-10T19:20:15.654 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:20:15 smithi183 ceph-mon[29736]: 2.7 scrub ok 2024-09-10T19:20:15.654 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:20:15 smithi183 ceph-mon[29736]: from='client.? 172.21.15.183:0/3009336162' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-10T19:20:15.654 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:20:15 smithi183 ceph-mon[29736]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-10T19:20:15.655 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:20:15.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:20:15.656 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:20:15.715 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:20:15 smithi191 ceph-mon[32884]: 2.7 scrub starts 2024-09-10T19:20:15.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:20:15 smithi191 ceph-mon[32884]: 2.7 scrub ok 2024-09-10T19:20:15.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:20:15 smithi191 ceph-mon[32884]: from='client.? 172.21.15.183:0/3009336162' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-10T19:20:15.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:20:15 smithi191 ceph-mon[32884]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-10T19:20:15.730 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: 14 16 36844 36828 164.393 121 0.00468215 0.00607762 2024-09-10T19:20:15.757 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:20:15.792 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:20:15.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:20:15.794 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:20:15.894 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:20:15.931 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:20:15.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:20:15.931 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:20:16.032 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:20:16.053 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:20:16.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:20:16.054 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:20:16.154 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:20:16.192 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:20:16.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:20:16.193 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:20:16.294 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:20:16.322 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:20:16.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:20:16.323 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:20:16.424 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:20:16.460 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:20:16.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:20:16.461 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:20:16.562 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:20:16.590 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:20:16.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:20:16.591 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:20:16.678 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:20:16 smithi183 ceph-mon[29736]: 4.3 scrub starts 2024-09-10T19:20:16.678 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:20:16 smithi183 ceph-mon[29736]: 4.3 scrub ok 2024-09-10T19:20:16.678 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:20:16 smithi183 ceph-mon[29736]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T19:20:16.678 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:20:16 smithi183 ceph-mon[29736]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-10T19:20:16.678 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:20:16 smithi183 ceph-mon[29736]: osdmap e237: 8 total, 8 up, 6 in 2024-09-10T19:20:16.678 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:20:16 smithi183 ceph-mon[29736]: pgmap v453: 148 pgs: 148 active+clean; 2.2 GiB data, 8.9 GiB used, 527 GiB / 536 GiB avail; 5.2 KiB/s rd, 198 MiB/s wr, 3.17k op/s 2024-09-10T19:20:16.679 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:20:16 smithi183 ceph-mon[41562]: 4.3 scrub starts 2024-09-10T19:20:16.679 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:20:16 smithi183 ceph-mon[41562]: 4.3 scrub ok 2024-09-10T19:20:16.679 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:20:16 smithi183 ceph-mon[41562]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T19:20:16.679 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:20:16 smithi183 ceph-mon[41562]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-10T19:20:16.679 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:20:16 smithi183 ceph-mon[41562]: osdmap e237: 8 total, 8 up, 6 in 2024-09-10T19:20:16.679 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:20:16 smithi183 ceph-mon[41562]: pgmap v453: 148 pgs: 148 active+clean; 2.2 GiB data, 8.9 GiB used, 527 GiB / 536 GiB avail; 5.2 KiB/s rd, 198 MiB/s wr, 3.17k op/s 2024-09-10T19:20:16.692 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:20:16.715 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:20:16 smithi191 ceph-mon[32884]: 4.3 scrub starts 2024-09-10T19:20:16.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:20:16 smithi191 ceph-mon[32884]: 4.3 scrub ok 2024-09-10T19:20:16.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:20:16 smithi191 ceph-mon[32884]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T19:20:16.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:20:16 smithi191 ceph-mon[32884]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-10T19:20:16.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:20:16 smithi191 ceph-mon[32884]: osdmap e237: 8 total, 8 up, 6 in 2024-09-10T19:20:16.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:20:16 smithi191 ceph-mon[32884]: pgmap v453: 148 pgs: 148 active+clean; 2.2 GiB data, 8.9 GiB used, 527 GiB / 536 GiB avail; 5.2 KiB/s rd, 198 MiB/s wr, 3.17k op/s 2024-09-10T19:20:16.722 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:20:16.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:20:16.723 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:20:16.731 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: 15 16 38653 38637 160.971 113.062 0.00507495 0.006207 2024-09-10T19:20:16.823 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:20:16.853 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:20:16.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:20:16.854 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:20:16.954 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:20:16.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:20:16.982 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:20:16.983 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:20:17.083 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:20:17.121 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:20:17.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:20:17.122 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:20:17.223 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:20:17.254 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:20:17.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:20:17.257 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:20:17.358 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:20:17.391 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:20:17.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:20:17.392 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:20:17.493 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:20:17.522 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:20:17.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:20:17.523 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:20:17.624 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:20:17.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:20:17.668 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:20:17.669 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:20:17.716 INFO:journalctl@ceph.mgr.x.smithi191.stdout:Sep 10 19:20:17 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-x[34967]: ::ffff:172.21.15.191 - - [10/Sep/2024:19:20:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T19:20:17.731 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: 16 16 40490 40474 158.085 114.812 0.013451 0.00632017 2024-09-10T19:20:17.769 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:20:17.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:20:17.802 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:20:17.802 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:20:17.903 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:20:17.940 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:20:17.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:20:17.941 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:20:18.042 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:20:18.078 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:20:18.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:20:18.079 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:20:18.180 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:20:18.217 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:20:18.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:20:18.218 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:20:18.318 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:20:18.351 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:20:18.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:20:18.352 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:20:18.453 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:20:18.490 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:20:18.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:20:18.491 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:20:18.592 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:20:18.630 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:20:18.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:20:18.631 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:20:18.731 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: 17 16 42312 42296 155.484 113.875 0.0081508 0.00642743 2024-09-10T19:20:18.732 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:20:18.764 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:20:18.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:20:18.765 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:20:18.865 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:20:18.895 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:20:18.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:20:18.896 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:20:18.997 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:20:19.028 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:20:19.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:20:19.029 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:20:19.130 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:20:19.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:20:19.164 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:20:19.165 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:20:19.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:20:18 smithi183 ceph-mon[29736]: pgmap v454: 148 pgs: 148 active+clean; 2.2 GiB data, 8.9 GiB used, 527 GiB / 536 GiB avail; 4.9 KiB/s rd, 186 MiB/s wr, 2.98k op/s 2024-09-10T19:20:19.181 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:20:18 smithi183 ceph-mon[41562]: pgmap v454: 148 pgs: 148 active+clean; 2.2 GiB data, 8.9 GiB used, 527 GiB / 536 GiB avail; 4.9 KiB/s rd, 186 MiB/s wr, 2.98k op/s 2024-09-10T19:20:19.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:20:18 smithi191 ceph-mon[32884]: pgmap v454: 148 pgs: 148 active+clean; 2.2 GiB data, 8.9 GiB used, 527 GiB / 536 GiB avail; 4.9 KiB/s rd, 186 MiB/s wr, 2.98k op/s 2024-09-10T19:20:19.266 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:20:19.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:20:19.314 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:20:19.314 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:20:19.415 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:20:19.444 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:20:19.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:20:19.445 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:20:19.546 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:20:19.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:20:19.585 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:20:19.585 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:20:19.686 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:20:19.716 INFO:journalctl@ceph.iscsi.iscsi.a.smithi191.stdout:Sep 10 19:20:19 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-iscsi-iscsi-a[61816]: debug there is no tcmu-runner data available 2024-09-10T19:20:19.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:20:19.717 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:20:19.717 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:20:19.731 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: 18 15 44144 44129 153.21 114.562 0.00864098 0.00652159 2024-09-10T19:20:19.818 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:20:19.855 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:20:19.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:20:19.856 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:20:19.957 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:20:19.992 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:20:19.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:20:19.993 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:20:20.094 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:20:20.134 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:20:20.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:20:20.136 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:20:20.234 INFO:teuthology.orchestra.run.smithi183.stdout:ERROR: (22) Invalid argument 2024-09-10T19:20:20.234 INFO:teuthology.orchestra.run.smithi183.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-10T19:20:20.236 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:20:20.275 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:20:20.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:20:20.276 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:20:20.377 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:20:20.412 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:20:20.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:20:20.413 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:20:20.485 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:20:20.514 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:20:20.551 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:20:20.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:20:20.552 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:20:20.653 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:20:20.688 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:20:20.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:20:20.689 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:20:20.732 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: 19 16 45886 45870 150.87 108.812 0.0140954 0.00662263 2024-09-10T19:20:20.773 INFO:teuthology.orchestra.run.smithi183.stdout:osd.0: {} 2024-09-10T19:20:20.774 INFO:teuthology.orchestra.run.smithi183.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-10T19:20:20.790 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:20:20.802 INFO:teuthology.orchestra.run.smithi183.stdout:osd.1: {} 2024-09-10T19:20:20.802 INFO:teuthology.orchestra.run.smithi183.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-10T19:20:20.828 INFO:teuthology.orchestra.run.smithi183.stdout:osd.2: {} 2024-09-10T19:20:20.828 INFO:teuthology.orchestra.run.smithi183.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-10T19:20:20.829 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:20:20.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:20:20.830 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:20:20.851 INFO:teuthology.orchestra.run.smithi183.stdout:osd.3: {} 2024-09-10T19:20:20.851 INFO:teuthology.orchestra.run.smithi183.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-10T19:20:20.852 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:20:20 smithi183 ceph-mon[29736]: from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:20:20.852 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:20:20 smithi183 ceph-mon[29736]: pgmap v455: 148 pgs: 148 active+clean; 2.2 GiB data, 8.9 GiB used, 527 GiB / 536 GiB avail; 3.9 KiB/s rd, 149 MiB/s wr, 2.39k op/s 2024-09-10T19:20:20.852 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:20:20 smithi183 ceph-mon[29736]: osdmap e238: 8 total, 8 up, 6 in 2024-09-10T19:20:20.852 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:20:20 smithi183 ceph-mon[29736]: from='client.? 172.21.15.183:0/2784544314' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T19:20:20.852 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:20:20 smithi183 ceph-mon[41562]: from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:20:20.852 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:20:20 smithi183 ceph-mon[41562]: pgmap v455: 148 pgs: 148 active+clean; 2.2 GiB data, 8.9 GiB used, 527 GiB / 536 GiB avail; 3.9 KiB/s rd, 149 MiB/s wr, 2.39k op/s 2024-09-10T19:20:20.852 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:20:20 smithi183 ceph-mon[41562]: osdmap e238: 8 total, 8 up, 6 in 2024-09-10T19:20:20.852 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:20:20 smithi183 ceph-mon[41562]: from='client.? 172.21.15.183:0/2784544314' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T19:20:20.882 INFO:teuthology.orchestra.run.smithi183.stdout:osd.4: {} 2024-09-10T19:20:20.883 INFO:teuthology.orchestra.run.smithi183.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-10T19:20:20.912 INFO:teuthology.orchestra.run.smithi183.stdout:osd.5: {} 2024-09-10T19:20:20.913 INFO:teuthology.orchestra.run.smithi183.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-10T19:20:20.930 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:20:20.941 INFO:teuthology.orchestra.run.smithi183.stdout:osd.6: {} 2024-09-10T19:20:20.942 INFO:teuthology.orchestra.run.smithi183.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-10T19:20:20.969 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:20:20.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:20:20.970 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:20:20.972 INFO:teuthology.orchestra.run.smithi183.stdout:osd.7: {} 2024-09-10T19:20:20.972 INFO:teuthology.orchestra.run.smithi183.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-10T19:20:21.071 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:20:21.105 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:20:21.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:20:21.106 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:20:21.207 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:20:21.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:20:20 smithi191 ceph-mon[32884]: from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:20:21.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:20:20 smithi191 ceph-mon[32884]: pgmap v455: 148 pgs: 148 active+clean; 2.2 GiB data, 8.9 GiB used, 527 GiB / 536 GiB avail; 3.9 KiB/s rd, 149 MiB/s wr, 2.39k op/s 2024-09-10T19:20:21.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:20:20 smithi191 ceph-mon[32884]: osdmap e238: 8 total, 8 up, 6 in 2024-09-10T19:20:21.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:20:20 smithi191 ceph-mon[32884]: from='client.? 172.21.15.183:0/2784544314' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T19:20:21.230 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:20:21.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:20:21.231 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:20:21.316 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-10T19:20:21.332 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:20:21.368 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:20:21.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:20:21.369 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:20:21.470 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:20:21.507 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:20:21.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:20:21.509 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:20:21.537 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph daemon osd.1 dump_historic_ops 2024-09-10T19:20:21.609 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:20:21.618 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Sep 10 19:20:21 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-y[29989]: ::ffff:172.21.15.191 - - [10/Sep/2024:19:20:21] "GET /metrics HTTP/1.1" 200 224291 "" "Prometheus/2.43.0" 2024-09-10T19:20:21.647 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:20:21.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:20:21.648 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:20:21.732 INFO:tasks.radosbench.radosbench.0.smithi183.stdout:2024-09-10T19:20:21.731186+0000 min lat: 0.00139047 max lat: 0.0791893 avg lat: 0.00672894 2024-09-10T19:20:21.732 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-10T19:20:21.732 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: 20 16 47544 47528 148.507 103.625 0.00902557 0.00672894 2024-09-10T19:20:21.748 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:20:21.794 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:20:21.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:20:21.795 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:20:21.896 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:20:21.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:20:21.929 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:20:21.929 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:20:22.030 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:20:22.070 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:20:22.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:20:22.071 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:20:22.172 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:20:22.207 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:20:22.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:20:22.208 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:20:22.309 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:20:22.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:20:22.341 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:20:22.342 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:20:22.443 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:20:22.478 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:20:22.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:20:22.479 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:20:22.579 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:20:22.609 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:20:22.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:20:22.610 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:20:22.710 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:20:22.732 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: 21 15 49227 49212 146.447 105.25 0.00777676 0.00682384 2024-09-10T19:20:22.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:20:22.750 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:20:22.750 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:20:22.851 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:20:22.893 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:20:22.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:20:22.894 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:20:22.995 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:20:23.023 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:20:23.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:20:23.024 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:20:23.125 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:20:23.162 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:20:23.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:20:23.163 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:20:23.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:20:22 smithi183 ceph-mon[29736]: pgmap v457: 148 pgs: 148 active+clean; 2.7 GiB data, 11 GiB used, 526 GiB / 536 GiB avail; 55 KiB/s rd, 154 MiB/s wr, 2.55k op/s 2024-09-10T19:20:23.181 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:20:22 smithi183 ceph-mon[41562]: pgmap v457: 148 pgs: 148 active+clean; 2.7 GiB data, 11 GiB used, 526 GiB / 536 GiB avail; 55 KiB/s rd, 154 MiB/s wr, 2.55k op/s 2024-09-10T19:20:23.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:20:22 smithi191 ceph-mon[32884]: pgmap v457: 148 pgs: 148 active+clean; 2.7 GiB data, 11 GiB used, 526 GiB / 536 GiB avail; 55 KiB/s rd, 154 MiB/s wr, 2.55k op/s 2024-09-10T19:20:23.247 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:20:23.264 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:20:23.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:20:23.304 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:20:23.304 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:20:23.405 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:20:23.435 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:20:23.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:20:23.437 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:20:23.537 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:20:23.578 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:20:23.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:20:23.579 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:20:23.680 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:20:23.717 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:20:23.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:20:23.718 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:20:23.732 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: 22 16 50863 50847 144.435 102.188 0.00770077 0.0069189 2024-09-10T19:20:23.818 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:20:23.848 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:20:23.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:20:23.849 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:20:23.950 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:20:23.992 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:20:23.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:20:23.993 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:20:24.093 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:20:24.099 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:20:24.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:20:24.129 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:20:24.130 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:20:24.230 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:20:24.261 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:20:24.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:20:24.262 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:20:24.363 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:20:24.398 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:20:24.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:20:24.399 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:20:24.420 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-10T19:20:24.499 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:20:24.532 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:20:24.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:20:24.533 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:20:24.634 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:20:24.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:20:24.667 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:20:24.668 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:20:24.732 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: 23 16 52489 52473 142.573 101.625 0.00920813 0.007009 2024-09-10T19:20:24.768 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:20:24.802 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:20:24.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:20:24.803 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:20:24.904 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:20:24.931 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:20:24 smithi183 ceph-mon[29736]: pgmap v458: 148 pgs: 148 active+clean; 2.7 GiB data, 11 GiB used, 526 GiB / 536 GiB avail; 48 KiB/s rd, 68 MiB/s wr, 1.16k op/s 2024-09-10T19:20:24.931 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:20:24 smithi183 ceph-mon[29736]: purged_snaps scrub starts 2024-09-10T19:20:24.931 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:20:24 smithi183 ceph-mon[29736]: purged_snaps scrub ok 2024-09-10T19:20:24.931 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:20:24 smithi183 ceph-mon[29736]: osdmap e239: 8 total, 8 up, 6 in 2024-09-10T19:20:24.931 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:20:24 smithi183 ceph-mon[29736]: osdmap e240: 8 total, 8 up, 6 in 2024-09-10T19:20:24.932 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:20:24 smithi183 ceph-mon[41562]: pgmap v458: 148 pgs: 148 active+clean; 2.7 GiB data, 11 GiB used, 526 GiB / 536 GiB avail; 48 KiB/s rd, 68 MiB/s wr, 1.16k op/s 2024-09-10T19:20:24.932 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:20:24 smithi183 ceph-mon[41562]: purged_snaps scrub starts 2024-09-10T19:20:24.932 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:20:24 smithi183 ceph-mon[41562]: purged_snaps scrub ok 2024-09-10T19:20:24.932 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:20:24 smithi183 ceph-mon[41562]: osdmap e239: 8 total, 8 up, 6 in 2024-09-10T19:20:24.932 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:20:24 smithi183 ceph-mon[41562]: osdmap e240: 8 total, 8 up, 6 in 2024-09-10T19:20:24.946 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:20:24.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:20:24.947 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:20:25.048 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:20:25.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:20:25.089 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:20:25.090 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:20:25.190 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:20:25.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:20:24 smithi191 ceph-mon[32884]: pgmap v458: 148 pgs: 148 active+clean; 2.7 GiB data, 11 GiB used, 526 GiB / 536 GiB avail; 48 KiB/s rd, 68 MiB/s wr, 1.16k op/s 2024-09-10T19:20:25.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:20:24 smithi191 ceph-mon[32884]: purged_snaps scrub starts 2024-09-10T19:20:25.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:20:24 smithi191 ceph-mon[32884]: purged_snaps scrub ok 2024-09-10T19:20:25.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:20:24 smithi191 ceph-mon[32884]: osdmap e239: 8 total, 8 up, 6 in 2024-09-10T19:20:25.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:20:24 smithi191 ceph-mon[32884]: osdmap e240: 8 total, 8 up, 6 in 2024-09-10T19:20:25.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:20:25.229 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:20:25.229 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:20:25.330 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:20:25.359 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:20:25.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:20:25.361 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:20:25.461 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:20:25.503 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:20:25.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:20:25.504 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:20:25.604 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:20:25.642 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:20:25.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:20:25.643 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:20:25.732 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: 24 16 54058 54042 140.718 98.0625 0.0102082 0.00710155 2024-09-10T19:20:25.744 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:20:25.772 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:20:25.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:20:25.773 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:20:25.835 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:20:25 smithi183 ceph-mon[29736]: from='client.? 172.21.15.183:0/4164051854' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"}]: dispatch 2024-09-10T19:20:25.835 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:20:25 smithi183 ceph-mon[29736]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"}]: dispatch 2024-09-10T19:20:25.835 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:20:25 smithi183 ceph-mon[41562]: from='client.? 172.21.15.183:0/4164051854' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"}]: dispatch 2024-09-10T19:20:25.835 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:20:25 smithi183 ceph-mon[41562]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"}]: dispatch 2024-09-10T19:20:25.835 INFO:teuthology.orchestra.run.smithi183.stderr:set pool 1 pg_num to 11 2024-09-10T19:20:25.874 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:20:25.903 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:20:25.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:20:25.904 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:20:26.005 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:20:26.039 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:20:26.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:20:26.040 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:20:26.140 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:20:26.176 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:20:26.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:20:26.177 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:20:26.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:20:25 smithi191 ceph-mon[32884]: from='client.? 172.21.15.183:0/4164051854' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"}]: dispatch 2024-09-10T19:20:26.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:20:25 smithi191 ceph-mon[32884]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"}]: dispatch 2024-09-10T19:20:26.278 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:20:26.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:20:26.309 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:20:26.310 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:20:26.410 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:20:26.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:20:26.449 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:20:26.449 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:20:26.550 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:20:26.583 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:20:26.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:20:26.584 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:20:26.685 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:20:26.719 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:20:26.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:20:26.721 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:20:26.732 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: 25 16 55626 55610 139.009 98 0.00584265 0.00718865 2024-09-10T19:20:26.822 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:20:26.852 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:20:26.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:20:26.853 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:20:26.931 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:20:26 smithi183 ceph-mon[29736]: purged_snaps scrub starts 2024-09-10T19:20:26.931 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:20:26 smithi183 ceph-mon[29736]: purged_snaps scrub ok 2024-09-10T19:20:26.931 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:20:26 smithi183 ceph-mon[29736]: pgmap v461: 148 pgs: 148 active+clean; 3.2 GiB data, 12 GiB used, 524 GiB / 536 GiB avail; 68 KiB/s rd, 182 MiB/s wr, 3.02k op/s 2024-09-10T19:20:26.931 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:20:26 smithi183 ceph-mon[29736]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"}]': finished 2024-09-10T19:20:26.932 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:20:26 smithi183 ceph-mon[29736]: osdmap e241: 8 total, 8 up, 6 in 2024-09-10T19:20:26.932 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:20:26 smithi183 ceph-mon[41562]: purged_snaps scrub starts 2024-09-10T19:20:26.932 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:20:26 smithi183 ceph-mon[41562]: purged_snaps scrub ok 2024-09-10T19:20:26.932 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:20:26 smithi183 ceph-mon[41562]: pgmap v461: 148 pgs: 148 active+clean; 3.2 GiB data, 12 GiB used, 524 GiB / 536 GiB avail; 68 KiB/s rd, 182 MiB/s wr, 3.02k op/s 2024-09-10T19:20:26.932 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:20:26 smithi183 ceph-mon[41562]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"}]': finished 2024-09-10T19:20:26.932 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:20:26 smithi183 ceph-mon[41562]: osdmap e241: 8 total, 8 up, 6 in 2024-09-10T19:20:26.954 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:20:26.994 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:20:26.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:20:26.996 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:20:27.097 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:20:27.117 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:20:26 smithi191 ceph-mon[32884]: purged_snaps scrub starts 2024-09-10T19:20:27.117 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:20:26 smithi191 ceph-mon[32884]: purged_snaps scrub ok 2024-09-10T19:20:27.117 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:20:26 smithi191 ceph-mon[32884]: pgmap v461: 148 pgs: 148 active+clean; 3.2 GiB data, 12 GiB used, 524 GiB / 536 GiB avail; 68 KiB/s rd, 182 MiB/s wr, 3.02k op/s 2024-09-10T19:20:27.117 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:20:26 smithi191 ceph-mon[32884]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"}]': finished 2024-09-10T19:20:27.117 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:20:26 smithi191 ceph-mon[32884]: osdmap e241: 8 total, 8 up, 6 in 2024-09-10T19:20:27.136 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:20:27.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:20:27.137 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:20:27.238 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:20:27.278 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:20:27.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:20:27.279 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:20:27.380 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:20:27.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:20:27.411 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:20:27.412 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:20:27.512 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:20:27.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:20:27.556 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:20:27.556 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:20:27.657 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:20:27.694 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:20:27.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:20:27.695 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:20:27.715 INFO:journalctl@ceph.mgr.x.smithi191.stdout:Sep 10 19:20:27 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-x[34967]: ::ffff:172.21.15.191 - - [10/Sep/2024:19:20:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T19:20:27.732 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: 26 16 57055 57039 137.098 89.3125 0.00971282 0.00728922 2024-09-10T19:20:27.795 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:20:27.839 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:20:27.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:20:27.841 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:20:27.848 INFO:teuthology.orchestra.run.smithi183.stderr:nodeep-scrub is set 2024-09-10T19:20:27.942 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:20:27.976 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:20:27 smithi183 ceph-mon[29736]: from='client.? 172.21.15.183:0/2577460473' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-10T19:20:27.976 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:20:27 smithi183 ceph-mon[29736]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-10T19:20:27.976 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:20:27 smithi183 ceph-mon[29736]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"}]: dispatch 2024-09-10T19:20:27.976 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:20:27 smithi183 ceph-mon[29736]: from='mgr.14556 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"}]: dispatch 2024-09-10T19:20:27.976 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:20:27 smithi183 ceph-mon[41562]: from='client.? 172.21.15.183:0/2577460473' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-10T19:20:27.976 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:20:27 smithi183 ceph-mon[41562]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-10T19:20:27.976 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:20:27 smithi183 ceph-mon[41562]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"}]: dispatch 2024-09-10T19:20:27.977 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:20:27 smithi183 ceph-mon[41562]: from='mgr.14556 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"}]: dispatch 2024-09-10T19:20:27.978 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:20:27.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:20:27.979 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:20:28.080 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:20:28.116 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:20:28.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:20:28.117 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:20:28.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:20:27 smithi191 ceph-mon[32884]: from='client.? 172.21.15.183:0/2577460473' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-10T19:20:28.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:20:27 smithi191 ceph-mon[32884]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-10T19:20:28.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:20:27 smithi191 ceph-mon[32884]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"}]: dispatch 2024-09-10T19:20:28.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:20:27 smithi191 ceph-mon[32884]: from='mgr.14556 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"}]: dispatch 2024-09-10T19:20:28.219 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:20:28.256 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:20:28.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:20:28.257 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:20:28.358 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:20:28.389 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:20:28.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:20:28.390 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:20:28.432 INFO:teuthology.orchestra.run.smithi183.stdout:{ 2024-09-10T19:20:28.432 INFO:teuthology.orchestra.run.smithi183.stdout: "size": 20, 2024-09-10T19:20:28.432 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 600, 2024-09-10T19:20:28.432 INFO:teuthology.orchestra.run.smithi183.stdout: "ops": [] 2024-09-10T19:20:28.433 INFO:teuthology.orchestra.run.smithi183.stdout:} 2024-09-10T19:20:28.490 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:20:28.518 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:20:28.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:20:28.519 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:20:28.619 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:20:28.645 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:20:28.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:20:28.646 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:20:28.732 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: 27 16 58354 58338 135.027 81.1875 0.0185292 0.0074002 2024-09-10T19:20:28.747 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:20:28.774 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:20:28.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:20:28.775 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:20:28.875 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:20:28.901 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:20:28.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:20:28.902 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:20:29.003 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:20:29.039 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:20:29.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:20:29.040 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:20:29.141 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:20:29.167 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:20:29.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:20:29.168 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:20:29.181 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:20:28 smithi183 ceph-mon[41562]: pgmap v463: 148 pgs: 148 active+clean; 3.2 GiB data, 12 GiB used, 524 GiB / 536 GiB avail; 853 B/s rd, 86 MiB/s wr, 1.38k op/s 2024-09-10T19:20:29.181 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:20:28 smithi183 ceph-mon[41562]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T19:20:29.181 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:20:28 smithi183 ceph-mon[41562]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-10T19:20:29.181 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:20:28 smithi183 ceph-mon[41562]: from='mgr.14556 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"}]': finished 2024-09-10T19:20:29.181 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:20:28 smithi183 ceph-mon[41562]: osdmap e242: 8 total, 8 up, 6 in 2024-09-10T19:20:29.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:20:28 smithi183 ceph-mon[41562]: from='mgr.14556 ' entity='mgr.y' 2024-09-10T19:20:29.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:20:28 smithi183 ceph-mon[41562]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:20:29.182 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:20:28 smithi183 ceph-mon[29736]: pgmap v463: 148 pgs: 148 active+clean; 3.2 GiB data, 12 GiB used, 524 GiB / 536 GiB avail; 853 B/s rd, 86 MiB/s wr, 1.38k op/s 2024-09-10T19:20:29.182 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:20:28 smithi183 ceph-mon[29736]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T19:20:29.182 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:20:28 smithi183 ceph-mon[29736]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-10T19:20:29.182 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:20:28 smithi183 ceph-mon[29736]: from='mgr.14556 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"}]': finished 2024-09-10T19:20:29.182 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:20:28 smithi183 ceph-mon[29736]: osdmap e242: 8 total, 8 up, 6 in 2024-09-10T19:20:29.182 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:20:28 smithi183 ceph-mon[29736]: from='mgr.14556 ' entity='mgr.y' 2024-09-10T19:20:29.182 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:20:28 smithi183 ceph-mon[29736]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:20:29.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:20:28 smithi191 ceph-mon[32884]: pgmap v463: 148 pgs: 148 active+clean; 3.2 GiB data, 12 GiB used, 524 GiB / 536 GiB avail; 853 B/s rd, 86 MiB/s wr, 1.38k op/s 2024-09-10T19:20:29.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:20:28 smithi191 ceph-mon[32884]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T19:20:29.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:20:28 smithi191 ceph-mon[32884]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-10T19:20:29.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:20:28 smithi191 ceph-mon[32884]: from='mgr.14556 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"}]': finished 2024-09-10T19:20:29.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:20:28 smithi191 ceph-mon[32884]: osdmap e242: 8 total, 8 up, 6 in 2024-09-10T19:20:29.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:20:28 smithi191 ceph-mon[32884]: from='mgr.14556 ' entity='mgr.y' 2024-09-10T19:20:29.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:20:28 smithi191 ceph-mon[32884]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:20:29.269 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:20:29.298 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:20:29.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:20:29.299 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:20:29.309 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:20:29.401 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:20:29.439 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:20:29.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:20:29.440 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:20:29.541 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:20:29.571 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:20:29.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:20:29.572 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:20:29.673 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:20:29.709 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:20:29.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:20:29.711 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:20:29.715 INFO:journalctl@ceph.iscsi.iscsi.a.smithi191.stdout:Sep 10 19:20:29 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-iscsi-iscsi-a[61816]: debug there is no tcmu-runner data available 2024-09-10T19:20:29.732 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: 28 16 59760 59744 133.343 87.875 0.00538741 0.00749482 2024-09-10T19:20:29.812 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:20:29.846 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:20:29.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:20:29.847 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:20:29.893 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph daemon osd.0 dump_ops_in_flight 2024-09-10T19:20:29.948 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:20:29.989 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:20:29.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:20:29.990 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:20:30.091 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:20:30.128 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:20:30.130 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:20:29 smithi183 ceph-mon[29736]: osdmap e243: 8 total, 8 up, 6 in 2024-09-10T19:20:30.130 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:20:29 smithi183 ceph-mon[29736]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"}]: dispatch 2024-09-10T19:20:30.130 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:20:29 smithi183 ceph-mon[29736]: from='mgr.14556 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"}]: dispatch 2024-09-10T19:20:30.130 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:20:29 smithi183 ceph-mon[29736]: from='mgr.14556 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"}]': finished 2024-09-10T19:20:30.130 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:20:29 smithi183 ceph-mon[29736]: osdmap e244: 8 total, 8 up, 6 in 2024-09-10T19:20:30.131 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:20:29 smithi183 ceph-mon[41562]: osdmap e243: 8 total, 8 up, 6 in 2024-09-10T19:20:30.131 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:20:29 smithi183 ceph-mon[41562]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"}]: dispatch 2024-09-10T19:20:30.131 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:20:29 smithi183 ceph-mon[41562]: from='mgr.14556 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"}]: dispatch 2024-09-10T19:20:30.131 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:20:29 smithi183 ceph-mon[41562]: from='mgr.14556 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"}]': finished 2024-09-10T19:20:30.131 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:20:29 smithi183 ceph-mon[41562]: osdmap e244: 8 total, 8 up, 6 in 2024-09-10T19:20:30.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:20:30.132 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:20:30.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:20:29 smithi191 ceph-mon[32884]: osdmap e243: 8 total, 8 up, 6 in 2024-09-10T19:20:30.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:20:29 smithi191 ceph-mon[32884]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"}]: dispatch 2024-09-10T19:20:30.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:20:29 smithi191 ceph-mon[32884]: from='mgr.14556 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"}]: dispatch 2024-09-10T19:20:30.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:20:29 smithi191 ceph-mon[32884]: from='mgr.14556 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"}]': finished 2024-09-10T19:20:30.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:20:29 smithi191 ceph-mon[32884]: osdmap e244: 8 total, 8 up, 6 in 2024-09-10T19:20:30.232 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:20:30.263 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:20:30.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:20:30.264 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:20:30.365 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:20:30.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:20:30.407 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:20:30.407 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:20:30.508 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:20:30.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:20:30.543 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:20:30.543 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:20:30.644 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:20:30.678 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:20:30.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:20:30.680 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:20:30.733 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: 29 15 61223 61208 131.895 91.5 0.00855984 0.00757683 2024-09-10T19:20:30.780 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:20:30.811 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:20:30.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:20:30.812 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:20:30.913 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:20:30.950 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:20:30.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:20:30.951 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:20:31.021 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph osd unset noscrub 2024-09-10T19:20:31.052 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:20:31.092 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:20:31.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:20:31.093 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:20:31.119 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:20:30 smithi183 ceph-mon[29736]: from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:20:31.119 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:20:30 smithi183 ceph-mon[29736]: pgmap v466: 158 pgs: 10 unknown, 148 active+clean; 3.2 GiB data, 12 GiB used, 524 GiB / 536 GiB avail; 1.0 KiB/s rd, 104 MiB/s wr, 1.67k op/s 2024-09-10T19:20:31.119 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:20:30 smithi183 ceph-mon[29736]: osdmap e245: 8 total, 8 up, 6 in 2024-09-10T19:20:31.119 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:20:30 smithi183 ceph-mon[41562]: from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:20:31.120 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:20:30 smithi183 ceph-mon[41562]: pgmap v466: 158 pgs: 10 unknown, 148 active+clean; 3.2 GiB data, 12 GiB used, 524 GiB / 536 GiB avail; 1.0 KiB/s rd, 104 MiB/s wr, 1.67k op/s 2024-09-10T19:20:31.120 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:20:30 smithi183 ceph-mon[41562]: osdmap e245: 8 total, 8 up, 6 in 2024-09-10T19:20:31.126 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:20:31.194 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:20:31.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:20:30 smithi191 ceph-mon[32884]: from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:20:31.217 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:20:30 smithi191 ceph-mon[32884]: pgmap v466: 158 pgs: 10 unknown, 148 active+clean; 3.2 GiB data, 12 GiB used, 524 GiB / 536 GiB avail; 1.0 KiB/s rd, 104 MiB/s wr, 1.67k op/s 2024-09-10T19:20:31.217 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:20:30 smithi191 ceph-mon[32884]: osdmap e245: 8 total, 8 up, 6 in 2024-09-10T19:20:31.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:20:31.232 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:20:31.233 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:20:31.334 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:20:31.365 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:20:31.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:20:31.366 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:20:31.401 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Sep 10 19:20:31 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-y[29989]: ::ffff:172.21.15.191 - - [10/Sep/2024:19:20:31] "GET /metrics HTTP/1.1" 200 224331 "" "Prometheus/2.43.0" 2024-09-10T19:20:31.467 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:20:31.507 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:20:31.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:20:31.508 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:20:31.609 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:20:31.642 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:20:31.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:20:31.643 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:20:31.734 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: 30 15 62684 62669 130.542 91.3125 0.0110467 0.0076558 2024-09-10T19:20:31.744 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:20:31.777 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:20:31.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:20:31.778 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:20:31.879 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:20:31.907 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:20:31.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:20:31.908 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:20:32.009 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:20:32.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:20:32.040 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:20:32.040 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:20:32.042 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:20:31 smithi183 ceph-mon[29736]: Health check failed: Reduced data availability: 3 pgs peering (PG_AVAILABILITY) 2024-09-10T19:20:32.042 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:20:31 smithi183 ceph-mon[41562]: Health check failed: Reduced data availability: 3 pgs peering (PG_AVAILABILITY) 2024-09-10T19:20:32.141 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:20:32.170 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:20:32.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:20:32.171 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:20:32.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:20:31 smithi191 ceph-mon[32884]: Health check failed: Reduced data availability: 3 pgs peering (PG_AVAILABILITY) 2024-09-10T19:20:32.272 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:20:32.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:20:32.308 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:20:32.308 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:20:32.409 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:20:32.448 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:20:32.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:20:32.449 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:20:32.512 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:20:32.550 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:20:32.560 INFO:teuthology.orchestra.run.smithi183.stdout:osd.0: {} 2024-09-10T19:20:32.560 INFO:teuthology.orchestra.run.smithi183.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-10T19:20:32.585 INFO:teuthology.orchestra.run.smithi183.stdout:osd.1: {} 2024-09-10T19:20:32.585 INFO:teuthology.orchestra.run.smithi183.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-10T19:20:32.593 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:20:32.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:20:32.594 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:20:32.608 INFO:teuthology.orchestra.run.smithi183.stdout:osd.2: {} 2024-09-10T19:20:32.608 INFO:teuthology.orchestra.run.smithi183.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-10T19:20:32.633 INFO:teuthology.orchestra.run.smithi183.stdout:osd.3: {} 2024-09-10T19:20:32.633 INFO:teuthology.orchestra.run.smithi183.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-10T19:20:32.663 INFO:teuthology.orchestra.run.smithi183.stdout:osd.4: {} 2024-09-10T19:20:32.663 INFO:teuthology.orchestra.run.smithi183.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-10T19:20:32.692 INFO:teuthology.orchestra.run.smithi183.stdout:osd.5: {} 2024-09-10T19:20:32.692 INFO:teuthology.orchestra.run.smithi183.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-10T19:20:32.695 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:20:32.716 INFO:teuthology.orchestra.run.smithi183.stdout:osd.6: {} 2024-09-10T19:20:32.716 INFO:teuthology.orchestra.run.smithi183.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-10T19:20:32.728 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:20:32.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:20:32.729 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:20:32.733 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: 31 16 64171 64155 129.326 92.875 0.0176965 0.0077275 2024-09-10T19:20:32.751 INFO:teuthology.orchestra.run.smithi183.stdout:osd.7: {} 2024-09-10T19:20:32.751 INFO:teuthology.orchestra.run.smithi183.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-10T19:20:32.754 INFO:tasks.thrashosds.thrasher:in_osds: [0, 3, 4, 5, 6, 7] out_osds: [1, 2] dead_osds: [] live_osds: [0, 2, 3, 4, 5, 6, 7, 1] 2024-09-10T19:20:32.755 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-10T19:20:32.755 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap_items 2024-09-10T19:20:32.755 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph osd dump -f json-pretty 2024-09-10T19:20:32.830 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:20:32.871 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:20:32.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:20:32.872 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:20:32.931 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:20:32 smithi183 ceph-mon[29736]: pgmap v469: 158 pgs: 10 peering, 148 active+clean; 3.7 GiB data, 14 GiB used, 523 GiB / 536 GiB avail; 113 MiB/s wr, 1.79k op/s 2024-09-10T19:20:32.931 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:20:32 smithi183 ceph-mon[29736]: from='client.? 172.21.15.183:0/2600669984' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T19:20:32.931 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:20:32 smithi183 ceph-mon[41562]: pgmap v469: 158 pgs: 10 peering, 148 active+clean; 3.7 GiB data, 14 GiB used, 523 GiB / 536 GiB avail; 113 MiB/s wr, 1.79k op/s 2024-09-10T19:20:32.932 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:20:32 smithi183 ceph-mon[41562]: from='client.? 172.21.15.183:0/2600669984' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T19:20:32.973 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:20:33.004 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:20:33.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:20:33.005 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:20:33.106 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:20:33.133 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:20:33.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:20:33.134 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:20:33.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:20:32 smithi191 ceph-mon[32884]: pgmap v469: 158 pgs: 10 peering, 148 active+clean; 3.7 GiB data, 14 GiB used, 523 GiB / 536 GiB avail; 113 MiB/s wr, 1.79k op/s 2024-09-10T19:20:33.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:20:32 smithi191 ceph-mon[32884]: from='client.? 172.21.15.183:0/2600669984' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T19:20:33.235 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:20:33.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:20:33.263 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:20:33.263 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:20:33.364 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:20:33.403 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:20:33.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:20:33.404 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:20:33.505 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:20:33.539 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:20:33.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:20:33.540 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:20:33.641 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:20:33.746 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: 32 16 65639 65623 128.152 91.75 0.00771123 0.00779823 2024-09-10T19:20:33.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:20:33.781 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:20:33.781 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:20:33.882 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:20:33.918 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:20:33.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:20:33.919 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:20:34.020 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:20:34.059 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:20:34.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:20:34.060 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:20:34.161 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:20:34.194 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:20:34.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:20:34.195 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:20:34.295 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:20:34.316 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:20:34.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:20:34.317 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:20:34.420 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:20:34.450 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:20:34.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:20:34.451 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:20:34.552 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:20:34.592 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:20:34.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:20:34.593 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:20:34.693 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:20:34.733 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:20:34.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:20:34.734 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:20:34.734 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: 33 16 66988 66972 126.823 84.3125 0.0120885 0.00788025 2024-09-10T19:20:34.835 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:20:34.870 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:20:34.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:20:34.871 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:20:34.972 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:20:35.007 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:20:35.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:20:35.008 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:20:35.109 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:20:35.147 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:20:35.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:20:35.148 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:20:35.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:20:34 smithi183 ceph-mon[29736]: pgmap v470: 158 pgs: 10 peering, 148 active+clean; 3.7 GiB data, 14 GiB used, 523 GiB / 536 GiB avail; 76 MiB/s wr, 1.21k op/s 2024-09-10T19:20:35.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:20:34 smithi183 ceph-mon[29736]: osdmap e246: 8 total, 8 up, 6 in 2024-09-10T19:20:35.181 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:20:34 smithi183 ceph-mon[41562]: pgmap v470: 158 pgs: 10 peering, 148 active+clean; 3.7 GiB data, 14 GiB used, 523 GiB / 536 GiB avail; 76 MiB/s wr, 1.21k op/s 2024-09-10T19:20:35.181 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:20:34 smithi183 ceph-mon[41562]: osdmap e246: 8 total, 8 up, 6 in 2024-09-10T19:20:35.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:20:34 smithi191 ceph-mon[32884]: pgmap v470: 158 pgs: 10 peering, 148 active+clean; 3.7 GiB data, 14 GiB used, 523 GiB / 536 GiB avail; 76 MiB/s wr, 1.21k op/s 2024-09-10T19:20:35.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:20:34 smithi191 ceph-mon[32884]: osdmap e246: 8 total, 8 up, 6 in 2024-09-10T19:20:35.249 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:20:35.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:20:35.287 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:20:35.287 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:20:35.388 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:20:35.415 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:20:35.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:20:35.416 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:20:35.517 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:20:35.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:20:35.556 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:20:35.557 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:20:35.657 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:20:35.686 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:20:35.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:20:35.687 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:20:35.730 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:20:35.734 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: 34 16 68390 68374 125.67 87.625 0.00868087 0.00795267 2024-09-10T19:20:35.789 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:20:35.817 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:20:35.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:20:35.818 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:20:35.919 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:20:35.947 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:20:35.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:20:35.948 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:20:35.964 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-10T19:20:36.049 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:20:36.080 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:20:36.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:20:36.081 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:20:36.182 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:20:36.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:20:36.217 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:20:36.217 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:20:36.230 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:20:35 smithi183 ceph-mon[29736]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 3 pgs peering) 2024-09-10T19:20:36.230 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:20:35 smithi183 ceph-mon[41562]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 3 pgs peering) 2024-09-10T19:20:36.319 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:20:36.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:20:36.354 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:20:36.354 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:20:36.455 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:20:36.468 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:20:35 smithi191 ceph-mon[32884]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 3 pgs peering) 2024-09-10T19:20:36.486 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:20:36.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:20:36.487 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:20:36.588 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:20:36.616 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:20:36.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:20:36.617 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:20:36.718 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:20:36.734 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: 35 16 69841 69825 124.671 90.6875 0.0159499 0.00801613 2024-09-10T19:20:36.752 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:20:36.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:20:36.753 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:20:36.854 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:20:36.882 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:20:36.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:20:36.883 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:20:36.984 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:20:37.015 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:20:37.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:20:37.016 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:20:37.021 INFO:teuthology.orchestra.run.smithi183.stdout:ERROR: (22) Invalid argument 2024-09-10T19:20:37.021 INFO:teuthology.orchestra.run.smithi183.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-10T19:20:37.117 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:20:37.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:20:37.158 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:20:37.158 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:20:37.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:20:36 smithi183 ceph-mon[29736]: pgmap v472: 158 pgs: 158 active+clean; 4.1 GiB data, 15 GiB used, 521 GiB / 536 GiB avail; 151 MiB/s wr, 2.41k op/s; 96 KiB/s, 0 objects/s recovering 2024-09-10T19:20:37.181 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:20:36 smithi183 ceph-mon[41562]: pgmap v472: 158 pgs: 158 active+clean; 4.1 GiB data, 15 GiB used, 521 GiB / 536 GiB avail; 151 MiB/s wr, 2.41k op/s; 96 KiB/s, 0 objects/s recovering 2024-09-10T19:20:37.259 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:20:37.295 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:20:37.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:20:37.296 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:20:37.397 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:20:37.433 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:20:37.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:20:37.434 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:20:37.450 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:20:36 smithi191 ceph-mon[32884]: pgmap v472: 158 pgs: 158 active+clean; 4.1 GiB data, 15 GiB used, 521 GiB / 536 GiB avail; 151 MiB/s wr, 2.41k op/s; 96 KiB/s, 0 objects/s recovering 2024-09-10T19:20:37.535 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:20:37.566 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:20:37.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:20:37.567 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:20:37.668 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:20:37.698 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:20:37.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:20:37.699 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:20:37.715 INFO:journalctl@ceph.mgr.x.smithi191.stdout:Sep 10 19:20:37 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-x[34967]: ::ffff:172.21.15.191 - - [10/Sep/2024:19:20:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T19:20:37.734 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: 36 16 71294 71278 123.73 90.8125 0.00794865 0.00807643 2024-09-10T19:20:37.801 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:20:37.830 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:20:37.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:20:37.831 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:20:37.932 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:20:37.965 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:20:37.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:20:37.966 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:20:38.066 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:20:38.097 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:20:38.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:20:38.098 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:20:38.199 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:20:38.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:20:38.238 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:20:38.238 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:20:38.339 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:20:38.369 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:20:38.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:20:38.370 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:20:38.471 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:20:38.507 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:20:38.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:20:38.508 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:20:38.609 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:20:38.646 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:20:38.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:20:38.647 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:20:38.736 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: 37 16 72738 72722 122.825 90.25 0.0113695 0.00813722 2024-09-10T19:20:38.748 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:20:38.788 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:20:38.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:20:38.789 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:20:38.890 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:20:38.920 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:20:38.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:20:38.922 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:20:39.022 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:20:39.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:20:39.058 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:20:39.058 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:20:39.159 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:20:39.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:20:38 smithi183 ceph-mon[29736]: purged_snaps scrub starts 2024-09-10T19:20:39.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:20:38 smithi183 ceph-mon[29736]: purged_snaps scrub ok 2024-09-10T19:20:39.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:20:38 smithi183 ceph-mon[29736]: pgmap v473: 158 pgs: 158 active+clean; 4.1 GiB data, 15 GiB used, 521 GiB / 536 GiB avail; 113 MiB/s wr, 1.81k op/s; 72 KiB/s, 0 objects/s recovering 2024-09-10T19:20:39.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:20:38 smithi183 ceph-mon[29736]: osdmap e247: 8 total, 8 up, 6 in 2024-09-10T19:20:39.181 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:20:38 smithi183 ceph-mon[41562]: purged_snaps scrub starts 2024-09-10T19:20:39.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:20:38 smithi183 ceph-mon[41562]: purged_snaps scrub ok 2024-09-10T19:20:39.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:20:38 smithi183 ceph-mon[41562]: pgmap v473: 158 pgs: 158 active+clean; 4.1 GiB data, 15 GiB used, 521 GiB / 536 GiB avail; 113 MiB/s wr, 1.81k op/s; 72 KiB/s, 0 objects/s recovering 2024-09-10T19:20:39.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:20:38 smithi183 ceph-mon[41562]: osdmap e247: 8 total, 8 up, 6 in 2024-09-10T19:20:39.188 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:20:39.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:20:39.190 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:20:39.213 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:20:39.230 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph daemon osd.0 dump_blocked_ops 2024-09-10T19:20:39.291 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:20:39.341 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:20:39.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:20:39.343 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:20:39.394 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:20:38 smithi191 ceph-mon[32884]: purged_snaps scrub starts 2024-09-10T19:20:39.394 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:20:38 smithi191 ceph-mon[32884]: purged_snaps scrub ok 2024-09-10T19:20:39.394 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:20:38 smithi191 ceph-mon[32884]: pgmap v473: 158 pgs: 158 active+clean; 4.1 GiB data, 15 GiB used, 521 GiB / 536 GiB avail; 113 MiB/s wr, 1.81k op/s; 72 KiB/s, 0 objects/s recovering 2024-09-10T19:20:39.394 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:20:38 smithi191 ceph-mon[32884]: osdmap e247: 8 total, 8 up, 6 in 2024-09-10T19:20:39.443 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:20:39.480 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:20:39.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:20:39.481 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:20:39.582 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:20:39.609 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:20:39.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:20:39.610 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:20:39.711 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:20:39.715 INFO:journalctl@ceph.iscsi.iscsi.a.smithi191.stdout:Sep 10 19:20:39 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-iscsi-iscsi-a[61816]: debug there is no tcmu-runner data available 2024-09-10T19:20:39.734 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: 38 16 74186 74170 121.974 90.5 0.0230066 0.00819371 2024-09-10T19:20:39.755 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:20:39.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:20:39.756 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:20:39.820 INFO:teuthology.orchestra.run.smithi183.stderr:noscrub is unset 2024-09-10T19:20:39.857 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:20:39.894 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:20:39.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:20:39.895 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:20:39.996 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:20:40.026 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:20:40.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:20:40.027 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:20:40.128 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:20:40.156 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:20:40.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:20:40.160 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:20:40.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:20:39 smithi183 ceph-mon[29736]: from='client.? 172.21.15.183:0/2952834922' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-10T19:20:40.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:20:39 smithi183 ceph-mon[29736]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-10T19:20:40.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:20:39 smithi183 ceph-mon[29736]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T19:20:40.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:20:39 smithi183 ceph-mon[29736]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-10T19:20:40.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:20:39 smithi183 ceph-mon[29736]: osdmap e248: 8 total, 8 up, 6 in 2024-09-10T19:20:40.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:20:39 smithi183 ceph-mon[41562]: from='client.? 172.21.15.183:0/2952834922' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-10T19:20:40.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:20:39 smithi183 ceph-mon[41562]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-10T19:20:40.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:20:39 smithi183 ceph-mon[41562]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T19:20:40.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:20:39 smithi183 ceph-mon[41562]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-10T19:20:40.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:20:39 smithi183 ceph-mon[41562]: osdmap e248: 8 total, 8 up, 6 in 2024-09-10T19:20:40.262 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:20:40.297 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:20:40.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:20:40.298 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:20:40.399 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:20:40.428 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:20:40.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:20:40.429 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:20:40.465 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:20:39 smithi191 ceph-mon[32884]: from='client.? 172.21.15.183:0/2952834922' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-10T19:20:40.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:20:39 smithi191 ceph-mon[32884]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-10T19:20:40.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:20:39 smithi191 ceph-mon[32884]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T19:20:40.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:20:39 smithi191 ceph-mon[32884]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-10T19:20:40.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:20:39 smithi191 ceph-mon[32884]: osdmap e248: 8 total, 8 up, 6 in 2024-09-10T19:20:40.529 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:20:40.565 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:20:40.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:20:40.566 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:20:40.667 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:20:40.703 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:20:40.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:20:40.704 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:20:40.734 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: 39 16 75549 75533 121.031 85.1875 0.00653784 0.0082572 2024-09-10T19:20:40.805 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:20:40.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:20:40.839 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:20:40.839 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:20:40.940 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:20:40.974 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:20:40.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:20:40.975 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:20:41.076 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:20:41.111 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:20:41.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:20:41.112 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:20:41.213 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:20:41.255 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:20:41.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:20:41.256 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:20:41.316 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:20:40 smithi183 ceph-mon[29736]: from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:20:41.316 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:20:40 smithi183 ceph-mon[29736]: pgmap v475: 158 pgs: 158 active+clean; 4.1 GiB data, 15 GiB used, 521 GiB / 536 GiB avail; 13 KiB/s rd, 57 MiB/s wr, 916 op/s; 105 KiB/s, 0 objects/s recovering 2024-09-10T19:20:41.317 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:20:40 smithi183 ceph-mon[29736]: 2.d scrub starts 2024-09-10T19:20:41.317 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:20:40 smithi183 ceph-mon[29736]: 2.d scrub ok 2024-09-10T19:20:41.317 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:20:40 smithi183 ceph-mon[41562]: from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:20:41.317 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:20:40 smithi183 ceph-mon[41562]: pgmap v475: 158 pgs: 158 active+clean; 4.1 GiB data, 15 GiB used, 521 GiB / 536 GiB avail; 13 KiB/s rd, 57 MiB/s wr, 916 op/s; 105 KiB/s, 0 objects/s recovering 2024-09-10T19:20:41.317 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:20:40 smithi183 ceph-mon[41562]: 2.d scrub starts 2024-09-10T19:20:41.317 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:20:40 smithi183 ceph-mon[41562]: 2.d scrub ok 2024-09-10T19:20:41.357 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:20:41.385 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:20:41.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:20:41.386 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:20:41.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:20:40 smithi191 ceph-mon[32884]: from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:20:41.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:20:40 smithi191 ceph-mon[32884]: pgmap v475: 158 pgs: 158 active+clean; 4.1 GiB data, 15 GiB used, 521 GiB / 536 GiB avail; 13 KiB/s rd, 57 MiB/s wr, 916 op/s; 105 KiB/s, 0 objects/s recovering 2024-09-10T19:20:41.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:20:40 smithi191 ceph-mon[32884]: 2.d scrub starts 2024-09-10T19:20:41.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:20:40 smithi191 ceph-mon[32884]: 2.d scrub ok 2024-09-10T19:20:41.486 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:20:41.525 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:20:41.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:20:41.526 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:20:41.627 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:20:41.660 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:20:41.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:20:41.661 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:20:41.681 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Sep 10 19:20:41 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-y[29989]: ::ffff:172.21.15.191 - - [10/Sep/2024:19:20:41] "GET /metrics HTTP/1.1" 200 224331 "" "Prometheus/2.43.0" 2024-09-10T19:20:41.734 INFO:tasks.radosbench.radosbench.0.smithi183.stdout:2024-09-10T19:20:41.734004+0000 min lat: 0.00139047 max lat: 0.0838044 avg lat: 0.0083182 2024-09-10T19:20:41.735 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-10T19:20:41.735 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: 40 16 76926 76910 120.156 86.0625 0.00776931 0.0083182 2024-09-10T19:20:41.762 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:20:41.789 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:20:41.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:20:41.790 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:20:41.891 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:20:41.932 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:20:41.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:20:41.933 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:20:42.034 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:20:42.054 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:20:42.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:20:42.055 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:20:42.156 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:20:42.184 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:20:42.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:20:42.185 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:20:42.286 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:20:42.312 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:20:42.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:20:42.313 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:20:42.414 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:20:42.431 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:20:42 smithi183 ceph-mon[41562]: 4.1c scrub starts 2024-09-10T19:20:42.431 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:20:42 smithi183 ceph-mon[41562]: 4.1c scrub ok 2024-09-10T19:20:42.431 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:20:42 smithi183 ceph-mon[41562]: 3.1 scrub starts 2024-09-10T19:20:42.431 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:20:42 smithi183 ceph-mon[41562]: 3.1 scrub ok 2024-09-10T19:20:42.431 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:20:42 smithi183 ceph-mon[41562]: 4.13 scrub starts 2024-09-10T19:20:42.431 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:20:42 smithi183 ceph-mon[41562]: 4.13 scrub ok 2024-09-10T19:20:42.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:20:42 smithi183 ceph-mon[29736]: 4.1c scrub starts 2024-09-10T19:20:42.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:20:42 smithi183 ceph-mon[29736]: 4.1c scrub ok 2024-09-10T19:20:42.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:20:42 smithi183 ceph-mon[29736]: 3.1 scrub starts 2024-09-10T19:20:42.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:20:42 smithi183 ceph-mon[29736]: 3.1 scrub ok 2024-09-10T19:20:42.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:20:42 smithi183 ceph-mon[29736]: 4.13 scrub starts 2024-09-10T19:20:42.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:20:42 smithi183 ceph-mon[29736]: 4.13 scrub ok 2024-09-10T19:20:42.442 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:20:42.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:20:42.444 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:20:42.465 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:20:42 smithi191 ceph-mon[32884]: 4.1c scrub starts 2024-09-10T19:20:42.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:20:42 smithi191 ceph-mon[32884]: 4.1c scrub ok 2024-09-10T19:20:42.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:20:42 smithi191 ceph-mon[32884]: 3.1 scrub starts 2024-09-10T19:20:42.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:20:42 smithi191 ceph-mon[32884]: 3.1 scrub ok 2024-09-10T19:20:42.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:20:42 smithi191 ceph-mon[32884]: 4.13 scrub starts 2024-09-10T19:20:42.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:20:42 smithi191 ceph-mon[32884]: 4.13 scrub ok 2024-09-10T19:20:42.544 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:20:42.572 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:20:42.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:20:42.573 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:20:42.674 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:20:42.711 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:20:42.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:20:42.712 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:20:42.734 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: 41 16 78208 78192 119.18 80.125 0.00380261 0.0083869 2024-09-10T19:20:42.813 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:20:42.853 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:20:42.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:20:42.854 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:20:42.954 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:20:42.984 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:20:42.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:20:42.985 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:20:43.086 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:20:43.120 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:20:43.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:20:43.121 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:20:43.222 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:20:43.247 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:20:43.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:20:43.248 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:20:43.349 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:20:43.375 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:20:43.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:20:43.376 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:20:43.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:20:43 smithi183 ceph-mon[29736]: pgmap v477: 158 pgs: 158 active+clean; 4.6 GiB data, 17 GiB used, 520 GiB / 536 GiB avail; 16 KiB/s rd, 129 MiB/s wr, 2.09k op/s; 120 KiB/s, 0 objects/s recovering 2024-09-10T19:20:43.431 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:20:43 smithi183 ceph-mon[41562]: pgmap v477: 158 pgs: 158 active+clean; 4.6 GiB data, 17 GiB used, 520 GiB / 536 GiB avail; 16 KiB/s rd, 129 MiB/s wr, 2.09k op/s; 120 KiB/s, 0 objects/s recovering 2024-09-10T19:20:43.465 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:20:43 smithi191 ceph-mon[32884]: pgmap v477: 158 pgs: 158 active+clean; 4.6 GiB data, 17 GiB used, 520 GiB / 536 GiB avail; 16 KiB/s rd, 129 MiB/s wr, 2.09k op/s; 120 KiB/s, 0 objects/s recovering 2024-09-10T19:20:43.477 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:20:43.513 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:20:43.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:20:43.514 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:20:43.615 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:20:43.643 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:20:43.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:20:43.645 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:20:43.735 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: 42 16 79595 79579 118.406 86.6875 0.011943 0.00844131 2024-09-10T19:20:43.746 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:20:43.782 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:20:43.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:20:43.783 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:20:43.884 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:20:43.905 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:20:43.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:20:43.906 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:20:44.007 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:20:44.037 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:20:44.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:20:44.038 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:20:44.139 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:20:44.169 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:20:44.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:20:44.171 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:20:44.272 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:20:44.301 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:20:44.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:20:44.302 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:20:44.375 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:20:44 smithi183 ceph-mon[29736]: 5.d scrub starts 2024-09-10T19:20:44.375 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:20:44 smithi183 ceph-mon[29736]: 5.d scrub ok 2024-09-10T19:20:44.375 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:20:44 smithi183 ceph-mon[29736]: 4.d scrub starts 2024-09-10T19:20:44.375 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:20:44 smithi183 ceph-mon[29736]: 4.d scrub ok 2024-09-10T19:20:44.375 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:20:44 smithi183 ceph-mon[29736]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:20:44.375 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:20:44 smithi183 ceph-mon[41562]: 5.d scrub starts 2024-09-10T19:20:44.376 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:20:44 smithi183 ceph-mon[41562]: 5.d scrub ok 2024-09-10T19:20:44.376 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:20:44 smithi183 ceph-mon[41562]: 4.d scrub starts 2024-09-10T19:20:44.376 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:20:44 smithi183 ceph-mon[41562]: 4.d scrub ok 2024-09-10T19:20:44.376 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:20:44 smithi183 ceph-mon[41562]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:20:44.403 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:20:44.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:20:44.446 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:20:44.446 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:20:44.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:20:44 smithi191 ceph-mon[32884]: 5.d scrub starts 2024-09-10T19:20:44.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:20:44 smithi191 ceph-mon[32884]: 5.d scrub ok 2024-09-10T19:20:44.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:20:44 smithi191 ceph-mon[32884]: 4.d scrub starts 2024-09-10T19:20:44.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:20:44 smithi191 ceph-mon[32884]: 4.d scrub ok 2024-09-10T19:20:44.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:20:44 smithi191 ceph-mon[32884]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:20:44.547 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:20:44.581 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:20:44.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:20:44.582 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:20:44.683 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:20:44.720 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:20:44.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:20:44.721 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:20:44.735 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: 43 16 80979 80963 117.664 86.5 0.00902563 0.00849393 2024-09-10T19:20:44.822 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:20:44.848 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:20:44.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:20:44.849 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:20:44.950 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:20:44.986 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:20:44.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:20:44.987 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:20:45.088 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:20:45.126 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:20:45.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:20:45.127 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:20:45.228 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:20:45.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:20:45.260 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:20:45.260 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:20:45.268 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:20:45 smithi183 ceph-mon[29736]: 2.5 scrub starts 2024-09-10T19:20:45.268 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:20:45 smithi183 ceph-mon[29736]: 2.5 scrub ok 2024-09-10T19:20:45.268 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:20:45 smithi183 ceph-mon[29736]: pgmap v478: 158 pgs: 158 active+clean; 4.6 GiB data, 17 GiB used, 520 GiB / 536 GiB avail; 639 B/s rd, 56 MiB/s wr, 902 op/s 2024-09-10T19:20:45.268 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:20:45 smithi183 ceph-mon[41562]: 2.5 scrub starts 2024-09-10T19:20:45.268 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:20:45 smithi183 ceph-mon[41562]: 2.5 scrub ok 2024-09-10T19:20:45.268 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:20:45 smithi183 ceph-mon[41562]: pgmap v478: 158 pgs: 158 active+clean; 4.6 GiB data, 17 GiB used, 520 GiB / 536 GiB avail; 639 B/s rd, 56 MiB/s wr, 902 op/s 2024-09-10T19:20:45.361 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:20:45.399 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:20:45.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:20:45.400 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:20:45.465 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:20:45 smithi191 ceph-mon[32884]: 2.5 scrub starts 2024-09-10T19:20:45.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:20:45 smithi191 ceph-mon[32884]: 2.5 scrub ok 2024-09-10T19:20:45.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:20:45 smithi191 ceph-mon[32884]: pgmap v478: 158 pgs: 158 active+clean; 4.6 GiB data, 17 GiB used, 520 GiB / 536 GiB avail; 639 B/s rd, 56 MiB/s wr, 902 op/s 2024-09-10T19:20:45.501 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:20:45.528 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:20:45.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:20:45.529 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:20:45.630 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:20:45.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:20:45.660 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:20:45.660 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:20:45.735 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: 44 16 82425 82409 117.044 90.375 0.00975379 0.00853928 2024-09-10T19:20:45.760 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:20:45.790 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:20:45.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:20:45.791 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:20:45.891 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:20:45.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:20:45.932 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:20:45.933 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:20:46.033 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:20:46.047 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:20:46.069 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:20:46.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:20:46.070 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:20:46.171 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:20:46.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:20:46 smithi183 ceph-mon[29736]: 2.2 scrub starts 2024-09-10T19:20:46.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:20:46 smithi183 ceph-mon[29736]: 2.2 scrub ok 2024-09-10T19:20:46.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:20:46 smithi183 ceph-mon[29736]: 4.1e scrub starts 2024-09-10T19:20:46.182 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:20:46 smithi183 ceph-mon[29736]: 4.1e scrub ok 2024-09-10T19:20:46.182 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:20:46 smithi183 ceph-mon[29736]: 6.2 scrub starts 2024-09-10T19:20:46.182 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:20:46 smithi183 ceph-mon[29736]: 6.2 scrub ok 2024-09-10T19:20:46.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:20:46 smithi183 ceph-mon[41562]: 2.2 scrub starts 2024-09-10T19:20:46.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:20:46 smithi183 ceph-mon[41562]: 2.2 scrub ok 2024-09-10T19:20:46.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:20:46 smithi183 ceph-mon[41562]: 4.1e scrub starts 2024-09-10T19:20:46.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:20:46 smithi183 ceph-mon[41562]: 4.1e scrub ok 2024-09-10T19:20:46.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:20:46 smithi183 ceph-mon[41562]: 6.2 scrub starts 2024-09-10T19:20:46.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:20:46 smithi183 ceph-mon[41562]: 6.2 scrub ok 2024-09-10T19:20:46.206 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:20:46.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:20:46.207 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:20:46.211 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-10T19:20:46.307 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:20:46.346 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-09-10T19:20:46.347 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:20:46.347 INFO:teuthology.orchestra.run.smithi183.stdout:{ 2024-09-10T19:20:46.347 INFO:teuthology.orchestra.run.smithi183.stdout: "epoch": 248, 2024-09-10T19:20:46.348 INFO:teuthology.orchestra.run.smithi183.stdout: "fsid": "320f6df8-6fa7-11ef-bcea-c7b262605968", 2024-09-10T19:20:46.348 INFO:teuthology.orchestra.run.smithi183.stdout: "created": "2024-09-10T19:05:03.688531+0000", 2024-09-10T19:20:46.348 INFO:teuthology.orchestra.run.smithi183.stdout: "modified": "2024-09-10T19:20:39.807982+0000", 2024-09-10T19:20:46.348 INFO:teuthology.orchestra.run.smithi183.stdout: "last_up_change": "2024-09-10T19:15:19.923894+0000", 2024-09-10T19:20:46.348 INFO:teuthology.orchestra.run.smithi183.stdout: "last_in_change": "2024-09-10T19:16:23.815531+0000", 2024-09-10T19:20:46.348 INFO:teuthology.orchestra.run.smithi183.stdout: "flags": "nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-09-10T19:20:46.348 INFO:teuthology.orchestra.run.smithi183.stdout: "flags_num": 5804032, 2024-09-10T19:20:46.348 INFO:teuthology.orchestra.run.smithi183.stdout: "flags_set": [ 2024-09-10T19:20:46.348 INFO:teuthology.orchestra.run.smithi183.stdout: "nodeep-scrub", 2024-09-10T19:20:46.348 INFO:teuthology.orchestra.run.smithi183.stdout: "pglog_hardlimit", 2024-09-10T19:20:46.348 INFO:teuthology.orchestra.run.smithi183.stdout: "purged_snapdirs", 2024-09-10T19:20:46.348 INFO:teuthology.orchestra.run.smithi183.stdout: "recovery_deletes", 2024-09-10T19:20:46.349 INFO:teuthology.orchestra.run.smithi183.stdout: "sortbitwise" 2024-09-10T19:20:46.349 INFO:teuthology.orchestra.run.smithi183.stdout: ], 2024-09-10T19:20:46.349 INFO:teuthology.orchestra.run.smithi183.stdout: "crush_version": 19, 2024-09-10T19:20:46.349 INFO:teuthology.orchestra.run.smithi183.stdout: "full_ratio": 0.94999998807907104, 2024-09-10T19:20:46.349 INFO:teuthology.orchestra.run.smithi183.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-09-10T19:20:46.349 INFO:teuthology.orchestra.run.smithi183.stdout: "nearfull_ratio": 0.85000002384185791, 2024-09-10T19:20:46.349 INFO:teuthology.orchestra.run.smithi183.stdout: "cluster_snapshot": "", 2024-09-10T19:20:46.349 INFO:teuthology.orchestra.run.smithi183.stdout: "pool_max": 9, 2024-09-10T19:20:46.349 INFO:teuthology.orchestra.run.smithi183.stdout: "max_osd": 8, 2024-09-10T19:20:46.349 INFO:teuthology.orchestra.run.smithi183.stdout: "require_min_compat_client": "luminous", 2024-09-10T19:20:46.349 INFO:teuthology.orchestra.run.smithi183.stdout: "min_compat_client": "luminous", 2024-09-10T19:20:46.349 INFO:teuthology.orchestra.run.smithi183.stdout: "require_osd_release": "quincy", 2024-09-10T19:20:46.349 INFO:teuthology.orchestra.run.smithi183.stdout: "pools": [ 2024-09-10T19:20:46.349 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:20:46.350 INFO:teuthology.orchestra.run.smithi183.stdout: "pool": 1, 2024-09-10T19:20:46.350 INFO:teuthology.orchestra.run.smithi183.stdout: "pool_name": ".mgr", 2024-09-10T19:20:46.350 INFO:teuthology.orchestra.run.smithi183.stdout: "create_time": "2024-09-10T19:08:00.456786+0000", 2024-09-10T19:20:46.350 INFO:teuthology.orchestra.run.smithi183.stdout: "flags": 1, 2024-09-10T19:20:46.350 INFO:teuthology.orchestra.run.smithi183.stdout: "flags_names": "hashpspool", 2024-09-10T19:20:46.350 INFO:teuthology.orchestra.run.smithi183.stdout: "type": 1, 2024-09-10T19:20:46.350 INFO:teuthology.orchestra.run.smithi183.stdout: "size": 3, 2024-09-10T19:20:46.350 INFO:teuthology.orchestra.run.smithi183.stdout: "min_size": 2, 2024-09-10T19:20:46.350 INFO:teuthology.orchestra.run.smithi183.stdout: "crush_rule": 0, 2024-09-10T19:20:46.350 INFO:teuthology.orchestra.run.smithi183.stdout: "peering_crush_bucket_count": 0, 2024-09-10T19:20:46.350 INFO:teuthology.orchestra.run.smithi183.stdout: "peering_crush_bucket_target": 0, 2024-09-10T19:20:46.350 INFO:teuthology.orchestra.run.smithi183.stdout: "peering_crush_bucket_barrier": 0, 2024-09-10T19:20:46.350 INFO:teuthology.orchestra.run.smithi183.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-10T19:20:46.350 INFO:teuthology.orchestra.run.smithi183.stdout: "object_hash": 2, 2024-09-10T19:20:46.350 INFO:teuthology.orchestra.run.smithi183.stdout: "pg_autoscale_mode": "off", 2024-09-10T19:20:46.351 INFO:teuthology.orchestra.run.smithi183.stdout: "pg_num": 11, 2024-09-10T19:20:46.351 INFO:teuthology.orchestra.run.smithi183.stdout: "pg_placement_num": 11, 2024-09-10T19:20:46.351 INFO:teuthology.orchestra.run.smithi183.stdout: "pg_placement_num_target": 11, 2024-09-10T19:20:46.351 INFO:teuthology.orchestra.run.smithi183.stdout: "pg_num_target": 11, 2024-09-10T19:20:46.351 INFO:teuthology.orchestra.run.smithi183.stdout: "pg_num_pending": 11, 2024-09-10T19:20:46.351 INFO:teuthology.orchestra.run.smithi183.stdout: "last_pg_merge_meta": { 2024-09-10T19:20:46.351 INFO:teuthology.orchestra.run.smithi183.stdout: "source_pgid": "0.0", 2024-09-10T19:20:46.351 INFO:teuthology.orchestra.run.smithi183.stdout: "ready_epoch": 0, 2024-09-10T19:20:46.351 INFO:teuthology.orchestra.run.smithi183.stdout: "last_epoch_started": 0, 2024-09-10T19:20:46.351 INFO:teuthology.orchestra.run.smithi183.stdout: "last_epoch_clean": 0, 2024-09-10T19:20:46.351 INFO:teuthology.orchestra.run.smithi183.stdout: "source_version": "0'0", 2024-09-10T19:20:46.351 INFO:teuthology.orchestra.run.smithi183.stdout: "target_version": "0'0" 2024-09-10T19:20:46.351 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:20:46.351 INFO:teuthology.orchestra.run.smithi183.stdout: "last_change": "244", 2024-09-10T19:20:46.351 INFO:teuthology.orchestra.run.smithi183.stdout: "last_force_op_resend": "0", 2024-09-10T19:20:46.351 INFO:teuthology.orchestra.run.smithi183.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-10T19:20:46.352 INFO:teuthology.orchestra.run.smithi183.stdout: "last_force_op_resend_preluminous": "242", 2024-09-10T19:20:46.352 INFO:teuthology.orchestra.run.smithi183.stdout: "auid": 0, 2024-09-10T19:20:46.352 INFO:teuthology.orchestra.run.smithi183.stdout: "snap_mode": "selfmanaged", 2024-09-10T19:20:46.352 INFO:teuthology.orchestra.run.smithi183.stdout: "snap_seq": 0, 2024-09-10T19:20:46.352 INFO:teuthology.orchestra.run.smithi183.stdout: "snap_epoch": 0, 2024-09-10T19:20:46.352 INFO:teuthology.orchestra.run.smithi183.stdout: "pool_snaps": [], 2024-09-10T19:20:46.352 INFO:teuthology.orchestra.run.smithi183.stdout: "removed_snaps": "[]", 2024-09-10T19:20:46.352 INFO:teuthology.orchestra.run.smithi183.stdout: "quota_max_bytes": 0, 2024-09-10T19:20:46.352 INFO:teuthology.orchestra.run.smithi183.stdout: "quota_max_objects": 0, 2024-09-10T19:20:46.352 INFO:teuthology.orchestra.run.smithi183.stdout: "tiers": [], 2024-09-10T19:20:46.352 INFO:teuthology.orchestra.run.smithi183.stdout: "tier_of": -1, 2024-09-10T19:20:46.352 INFO:teuthology.orchestra.run.smithi183.stdout: "read_tier": -1, 2024-09-10T19:20:46.353 INFO:teuthology.orchestra.run.smithi183.stdout: "write_tier": -1, 2024-09-10T19:20:46.353 INFO:teuthology.orchestra.run.smithi183.stdout: "cache_mode": "none", 2024-09-10T19:20:46.353 INFO:teuthology.orchestra.run.smithi183.stdout: "target_max_bytes": 0, 2024-09-10T19:20:46.353 INFO:teuthology.orchestra.run.smithi183.stdout: "target_max_objects": 0, 2024-09-10T19:20:46.353 INFO:teuthology.orchestra.run.smithi183.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-10T19:20:46.353 INFO:teuthology.orchestra.run.smithi183.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-10T19:20:46.353 INFO:teuthology.orchestra.run.smithi183.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-10T19:20:46.353 INFO:teuthology.orchestra.run.smithi183.stdout: "cache_min_flush_age": 0, 2024-09-10T19:20:46.353 INFO:teuthology.orchestra.run.smithi183.stdout: "cache_min_evict_age": 0, 2024-09-10T19:20:46.353 INFO:teuthology.orchestra.run.smithi183.stdout: "erasure_code_profile": "", 2024-09-10T19:20:46.353 INFO:teuthology.orchestra.run.smithi183.stdout: "hit_set_params": { 2024-09-10T19:20:46.354 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "none" 2024-09-10T19:20:46.354 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:20:46.354 INFO:teuthology.orchestra.run.smithi183.stdout: "hit_set_period": 0, 2024-09-10T19:20:46.354 INFO:teuthology.orchestra.run.smithi183.stdout: "hit_set_count": 0, 2024-09-10T19:20:46.354 INFO:teuthology.orchestra.run.smithi183.stdout: "use_gmt_hitset": true, 2024-09-10T19:20:46.354 INFO:teuthology.orchestra.run.smithi183.stdout: "min_read_recency_for_promote": 0, 2024-09-10T19:20:46.354 INFO:teuthology.orchestra.run.smithi183.stdout: "min_write_recency_for_promote": 0, 2024-09-10T19:20:46.354 INFO:teuthology.orchestra.run.smithi183.stdout: "hit_set_grade_decay_rate": 0, 2024-09-10T19:20:46.354 INFO:teuthology.orchestra.run.smithi183.stdout: "hit_set_search_last_n": 0, 2024-09-10T19:20:46.354 INFO:teuthology.orchestra.run.smithi183.stdout: "grade_table": [], 2024-09-10T19:20:46.354 INFO:teuthology.orchestra.run.smithi183.stdout: "stripe_width": 0, 2024-09-10T19:20:46.354 INFO:teuthology.orchestra.run.smithi183.stdout: "expected_num_objects": 0, 2024-09-10T19:20:46.354 INFO:teuthology.orchestra.run.smithi183.stdout: "fast_read": false, 2024-09-10T19:20:46.354 INFO:teuthology.orchestra.run.smithi183.stdout: "options": { 2024-09-10T19:20:46.355 INFO:teuthology.orchestra.run.smithi183.stdout: "pg_num_max": 32, 2024-09-10T19:20:46.355 INFO:teuthology.orchestra.run.smithi183.stdout: "pg_num_min": 1 2024-09-10T19:20:46.355 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:20:46.355 INFO:teuthology.orchestra.run.smithi183.stdout: "application_metadata": { 2024-09-10T19:20:46.355 INFO:teuthology.orchestra.run.smithi183.stdout: "mgr": {} 2024-09-10T19:20:46.355 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:20:46.355 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:20:46.355 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:20:46.355 INFO:teuthology.orchestra.run.smithi183.stdout: "pool": 2, 2024-09-10T19:20:46.355 INFO:teuthology.orchestra.run.smithi183.stdout: "pool_name": ".rgw.root", 2024-09-10T19:20:46.355 INFO:teuthology.orchestra.run.smithi183.stdout: "create_time": "2024-09-10T19:10:05.993763+0000", 2024-09-10T19:20:46.355 INFO:teuthology.orchestra.run.smithi183.stdout: "flags": 1, 2024-09-10T19:20:46.355 INFO:teuthology.orchestra.run.smithi183.stdout: "flags_names": "hashpspool", 2024-09-10T19:20:46.356 INFO:teuthology.orchestra.run.smithi183.stdout: "type": 1, 2024-09-10T19:20:46.356 INFO:teuthology.orchestra.run.smithi183.stdout: "size": 3, 2024-09-10T19:20:46.356 INFO:teuthology.orchestra.run.smithi183.stdout: "min_size": 2, 2024-09-10T19:20:46.356 INFO:teuthology.orchestra.run.smithi183.stdout: "crush_rule": 0, 2024-09-10T19:20:46.356 INFO:teuthology.orchestra.run.smithi183.stdout: "peering_crush_bucket_count": 0, 2024-09-10T19:20:46.356 INFO:teuthology.orchestra.run.smithi183.stdout: "peering_crush_bucket_target": 0, 2024-09-10T19:20:46.356 INFO:teuthology.orchestra.run.smithi183.stdout: "peering_crush_bucket_barrier": 0, 2024-09-10T19:20:46.356 INFO:teuthology.orchestra.run.smithi183.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-10T19:20:46.356 INFO:teuthology.orchestra.run.smithi183.stdout: "object_hash": 2, 2024-09-10T19:20:46.356 INFO:teuthology.orchestra.run.smithi183.stdout: "pg_autoscale_mode": "off", 2024-09-10T19:20:46.356 INFO:teuthology.orchestra.run.smithi183.stdout: "pg_num": 32, 2024-09-10T19:20:46.356 INFO:teuthology.orchestra.run.smithi183.stdout: "pg_placement_num": 32, 2024-09-10T19:20:46.357 INFO:teuthology.orchestra.run.smithi183.stdout: "pg_placement_num_target": 32, 2024-09-10T19:20:46.357 INFO:teuthology.orchestra.run.smithi183.stdout: "pg_num_target": 32, 2024-09-10T19:20:46.357 INFO:teuthology.orchestra.run.smithi183.stdout: "pg_num_pending": 32, 2024-09-10T19:20:46.357 INFO:teuthology.orchestra.run.smithi183.stdout: "last_pg_merge_meta": { 2024-09-10T19:20:46.357 INFO:teuthology.orchestra.run.smithi183.stdout: "source_pgid": "0.0", 2024-09-10T19:20:46.357 INFO:teuthology.orchestra.run.smithi183.stdout: "ready_epoch": 0, 2024-09-10T19:20:46.357 INFO:teuthology.orchestra.run.smithi183.stdout: "last_epoch_started": 0, 2024-09-10T19:20:46.357 INFO:teuthology.orchestra.run.smithi183.stdout: "last_epoch_clean": 0, 2024-09-10T19:20:46.357 INFO:teuthology.orchestra.run.smithi183.stdout: "source_version": "0'0", 2024-09-10T19:20:46.357 INFO:teuthology.orchestra.run.smithi183.stdout: "target_version": "0'0" 2024-09-10T19:20:46.357 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:20:46.357 INFO:teuthology.orchestra.run.smithi183.stdout: "last_change": "60", 2024-09-10T19:20:46.357 INFO:teuthology.orchestra.run.smithi183.stdout: "last_force_op_resend": "0", 2024-09-10T19:20:46.357 INFO:teuthology.orchestra.run.smithi183.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-10T19:20:46.358 INFO:teuthology.orchestra.run.smithi183.stdout: "last_force_op_resend_preluminous": "0", 2024-09-10T19:20:46.358 INFO:teuthology.orchestra.run.smithi183.stdout: "auid": 0, 2024-09-10T19:20:46.358 INFO:teuthology.orchestra.run.smithi183.stdout: "snap_mode": "selfmanaged", 2024-09-10T19:20:46.358 INFO:teuthology.orchestra.run.smithi183.stdout: "snap_seq": 0, 2024-09-10T19:20:46.358 INFO:teuthology.orchestra.run.smithi183.stdout: "snap_epoch": 0, 2024-09-10T19:20:46.358 INFO:teuthology.orchestra.run.smithi183.stdout: "pool_snaps": [], 2024-09-10T19:20:46.358 INFO:teuthology.orchestra.run.smithi183.stdout: "removed_snaps": "[]", 2024-09-10T19:20:46.358 INFO:teuthology.orchestra.run.smithi183.stdout: "quota_max_bytes": 0, 2024-09-10T19:20:46.358 INFO:teuthology.orchestra.run.smithi183.stdout: "quota_max_objects": 0, 2024-09-10T19:20:46.358 INFO:teuthology.orchestra.run.smithi183.stdout: "tiers": [], 2024-09-10T19:20:46.358 INFO:teuthology.orchestra.run.smithi183.stdout: "tier_of": -1, 2024-09-10T19:20:46.358 INFO:teuthology.orchestra.run.smithi183.stdout: "read_tier": -1, 2024-09-10T19:20:46.358 INFO:teuthology.orchestra.run.smithi183.stdout: "write_tier": -1, 2024-09-10T19:20:46.359 INFO:teuthology.orchestra.run.smithi183.stdout: "cache_mode": "none", 2024-09-10T19:20:46.359 INFO:teuthology.orchestra.run.smithi183.stdout: "target_max_bytes": 0, 2024-09-10T19:20:46.359 INFO:teuthology.orchestra.run.smithi183.stdout: "target_max_objects": 0, 2024-09-10T19:20:46.359 INFO:teuthology.orchestra.run.smithi183.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-10T19:20:46.359 INFO:teuthology.orchestra.run.smithi183.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-10T19:20:46.359 INFO:teuthology.orchestra.run.smithi183.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-10T19:20:46.359 INFO:teuthology.orchestra.run.smithi183.stdout: "cache_min_flush_age": 0, 2024-09-10T19:20:46.359 INFO:teuthology.orchestra.run.smithi183.stdout: "cache_min_evict_age": 0, 2024-09-10T19:20:46.359 INFO:teuthology.orchestra.run.smithi183.stdout: "erasure_code_profile": "", 2024-09-10T19:20:46.359 INFO:teuthology.orchestra.run.smithi183.stdout: "hit_set_params": { 2024-09-10T19:20:46.359 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "none" 2024-09-10T19:20:46.359 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:20:46.359 INFO:teuthology.orchestra.run.smithi183.stdout: "hit_set_period": 0, 2024-09-10T19:20:46.359 INFO:teuthology.orchestra.run.smithi183.stdout: "hit_set_count": 0, 2024-09-10T19:20:46.360 INFO:teuthology.orchestra.run.smithi183.stdout: "use_gmt_hitset": true, 2024-09-10T19:20:46.360 INFO:teuthology.orchestra.run.smithi183.stdout: "min_read_recency_for_promote": 0, 2024-09-10T19:20:46.360 INFO:teuthology.orchestra.run.smithi183.stdout: "min_write_recency_for_promote": 0, 2024-09-10T19:20:46.360 INFO:teuthology.orchestra.run.smithi183.stdout: "hit_set_grade_decay_rate": 0, 2024-09-10T19:20:46.360 INFO:teuthology.orchestra.run.smithi183.stdout: "hit_set_search_last_n": 0, 2024-09-10T19:20:46.360 INFO:teuthology.orchestra.run.smithi183.stdout: "grade_table": [], 2024-09-10T19:20:46.360 INFO:teuthology.orchestra.run.smithi183.stdout: "stripe_width": 0, 2024-09-10T19:20:46.360 INFO:teuthology.orchestra.run.smithi183.stdout: "expected_num_objects": 0, 2024-09-10T19:20:46.360 INFO:teuthology.orchestra.run.smithi183.stdout: "fast_read": false, 2024-09-10T19:20:46.360 INFO:teuthology.orchestra.run.smithi183.stdout: "options": {}, 2024-09-10T19:20:46.361 INFO:teuthology.orchestra.run.smithi183.stdout: "application_metadata": { 2024-09-10T19:20:46.361 INFO:teuthology.orchestra.run.smithi183.stdout: "rgw": {} 2024-09-10T19:20:46.361 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:20:46.361 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:20:46.361 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:20:46.361 INFO:teuthology.orchestra.run.smithi183.stdout: "pool": 3, 2024-09-10T19:20:46.361 INFO:teuthology.orchestra.run.smithi183.stdout: "pool_name": "datapool", 2024-09-10T19:20:46.361 INFO:teuthology.orchestra.run.smithi183.stdout: "create_time": "2024-09-10T19:10:06.224267+0000", 2024-09-10T19:20:46.361 INFO:teuthology.orchestra.run.smithi183.stdout: "flags": 8193, 2024-09-10T19:20:46.361 INFO:teuthology.orchestra.run.smithi183.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-09-10T19:20:46.361 INFO:teuthology.orchestra.run.smithi183.stdout: "type": 1, 2024-09-10T19:20:46.361 INFO:teuthology.orchestra.run.smithi183.stdout: "size": 3, 2024-09-10T19:20:46.362 INFO:teuthology.orchestra.run.smithi183.stdout: "min_size": 2, 2024-09-10T19:20:46.362 INFO:teuthology.orchestra.run.smithi183.stdout: "crush_rule": 0, 2024-09-10T19:20:46.362 INFO:teuthology.orchestra.run.smithi183.stdout: "peering_crush_bucket_count": 0, 2024-09-10T19:20:46.362 INFO:teuthology.orchestra.run.smithi183.stdout: "peering_crush_bucket_target": 0, 2024-09-10T19:20:46.362 INFO:teuthology.orchestra.run.smithi183.stdout: "peering_crush_bucket_barrier": 0, 2024-09-10T19:20:46.362 INFO:teuthology.orchestra.run.smithi183.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-10T19:20:46.362 INFO:teuthology.orchestra.run.smithi183.stdout: "object_hash": 2, 2024-09-10T19:20:46.362 INFO:teuthology.orchestra.run.smithi183.stdout: "pg_autoscale_mode": "off", 2024-09-10T19:20:46.362 INFO:teuthology.orchestra.run.smithi183.stdout: "pg_num": 3, 2024-09-10T19:20:46.362 INFO:teuthology.orchestra.run.smithi183.stdout: "pg_placement_num": 3, 2024-09-10T19:20:46.362 INFO:teuthology.orchestra.run.smithi183.stdout: "pg_placement_num_target": 3, 2024-09-10T19:20:46.362 INFO:teuthology.orchestra.run.smithi183.stdout: "pg_num_target": 3, 2024-09-10T19:20:46.362 INFO:teuthology.orchestra.run.smithi183.stdout: "pg_num_pending": 3, 2024-09-10T19:20:46.363 INFO:teuthology.orchestra.run.smithi183.stdout: "last_pg_merge_meta": { 2024-09-10T19:20:46.363 INFO:teuthology.orchestra.run.smithi183.stdout: "source_pgid": "0.0", 2024-09-10T19:20:46.363 INFO:teuthology.orchestra.run.smithi183.stdout: "ready_epoch": 0, 2024-09-10T19:20:46.363 INFO:teuthology.orchestra.run.smithi183.stdout: "last_epoch_started": 0, 2024-09-10T19:20:46.363 INFO:teuthology.orchestra.run.smithi183.stdout: "last_epoch_clean": 0, 2024-09-10T19:20:46.363 INFO:teuthology.orchestra.run.smithi183.stdout: "source_version": "0'0", 2024-09-10T19:20:46.363 INFO:teuthology.orchestra.run.smithi183.stdout: "target_version": "0'0" 2024-09-10T19:20:46.363 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:20:46.363 INFO:teuthology.orchestra.run.smithi183.stdout: "last_change": "63", 2024-09-10T19:20:46.363 INFO:teuthology.orchestra.run.smithi183.stdout: "last_force_op_resend": "0", 2024-09-10T19:20:46.363 INFO:teuthology.orchestra.run.smithi183.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-10T19:20:46.364 INFO:teuthology.orchestra.run.smithi183.stdout: "last_force_op_resend_preluminous": "0", 2024-09-10T19:20:46.364 INFO:teuthology.orchestra.run.smithi183.stdout: "auid": 0, 2024-09-10T19:20:46.364 INFO:teuthology.orchestra.run.smithi183.stdout: "snap_mode": "selfmanaged", 2024-09-10T19:20:46.364 INFO:teuthology.orchestra.run.smithi183.stdout: "snap_seq": 3, 2024-09-10T19:20:46.364 INFO:teuthology.orchestra.run.smithi183.stdout: "snap_epoch": 63, 2024-09-10T19:20:46.364 INFO:teuthology.orchestra.run.smithi183.stdout: "pool_snaps": [], 2024-09-10T19:20:46.364 INFO:teuthology.orchestra.run.smithi183.stdout: "removed_snaps": "[]", 2024-09-10T19:20:46.364 INFO:teuthology.orchestra.run.smithi183.stdout: "quota_max_bytes": 0, 2024-09-10T19:20:46.364 INFO:teuthology.orchestra.run.smithi183.stdout: "quota_max_objects": 0, 2024-09-10T19:20:46.364 INFO:teuthology.orchestra.run.smithi183.stdout: "tiers": [], 2024-09-10T19:20:46.365 INFO:teuthology.orchestra.run.smithi183.stdout: "tier_of": -1, 2024-09-10T19:20:46.365 INFO:teuthology.orchestra.run.smithi183.stdout: "read_tier": -1, 2024-09-10T19:20:46.365 INFO:teuthology.orchestra.run.smithi183.stdout: "write_tier": -1, 2024-09-10T19:20:46.365 INFO:teuthology.orchestra.run.smithi183.stdout: "cache_mode": "none", 2024-09-10T19:20:46.365 INFO:teuthology.orchestra.run.smithi183.stdout: "target_max_bytes": 0, 2024-09-10T19:20:46.365 INFO:teuthology.orchestra.run.smithi183.stdout: "target_max_objects": 0, 2024-09-10T19:20:46.365 INFO:teuthology.orchestra.run.smithi183.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-10T19:20:46.365 INFO:teuthology.orchestra.run.smithi183.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-10T19:20:46.365 INFO:teuthology.orchestra.run.smithi183.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-10T19:20:46.365 INFO:teuthology.orchestra.run.smithi183.stdout: "cache_min_flush_age": 0, 2024-09-10T19:20:46.365 INFO:teuthology.orchestra.run.smithi183.stdout: "cache_min_evict_age": 0, 2024-09-10T19:20:46.365 INFO:teuthology.orchestra.run.smithi183.stdout: "erasure_code_profile": "", 2024-09-10T19:20:46.365 INFO:teuthology.orchestra.run.smithi183.stdout: "hit_set_params": { 2024-09-10T19:20:46.366 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "none" 2024-09-10T19:20:46.366 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:20:46.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:20:46.367 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:20:46.367 INFO:teuthology.orchestra.run.smithi183.stdout: "hit_set_period": 0, 2024-09-10T19:20:46.367 INFO:teuthology.orchestra.run.smithi183.stdout: "hit_set_count": 0, 2024-09-10T19:20:46.367 INFO:teuthology.orchestra.run.smithi183.stdout: "use_gmt_hitset": true, 2024-09-10T19:20:46.367 INFO:teuthology.orchestra.run.smithi183.stdout: "min_read_recency_for_promote": 0, 2024-09-10T19:20:46.367 INFO:teuthology.orchestra.run.smithi183.stdout: "min_write_recency_for_promote": 0, 2024-09-10T19:20:46.367 INFO:teuthology.orchestra.run.smithi183.stdout: "hit_set_grade_decay_rate": 0, 2024-09-10T19:20:46.368 INFO:teuthology.orchestra.run.smithi183.stdout: "hit_set_search_last_n": 0, 2024-09-10T19:20:46.368 INFO:teuthology.orchestra.run.smithi183.stdout: "grade_table": [], 2024-09-10T19:20:46.368 INFO:teuthology.orchestra.run.smithi183.stdout: "stripe_width": 0, 2024-09-10T19:20:46.368 INFO:teuthology.orchestra.run.smithi183.stdout: "expected_num_objects": 0, 2024-09-10T19:20:46.368 INFO:teuthology.orchestra.run.smithi183.stdout: "fast_read": false, 2024-09-10T19:20:46.368 INFO:teuthology.orchestra.run.smithi183.stdout: "options": {}, 2024-09-10T19:20:46.368 INFO:teuthology.orchestra.run.smithi183.stdout: "application_metadata": { 2024-09-10T19:20:46.368 INFO:teuthology.orchestra.run.smithi183.stdout: "rbd": {} 2024-09-10T19:20:46.368 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:20:46.368 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:20:46.368 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:20:46.368 INFO:teuthology.orchestra.run.smithi183.stdout: "pool": 4, 2024-09-10T19:20:46.368 INFO:teuthology.orchestra.run.smithi183.stdout: "pool_name": "default.rgw.log", 2024-09-10T19:20:46.369 INFO:teuthology.orchestra.run.smithi183.stdout: "create_time": "2024-09-10T19:10:07.958284+0000", 2024-09-10T19:20:46.369 INFO:teuthology.orchestra.run.smithi183.stdout: "flags": 1, 2024-09-10T19:20:46.369 INFO:teuthology.orchestra.run.smithi183.stdout: "flags_names": "hashpspool", 2024-09-10T19:20:46.369 INFO:teuthology.orchestra.run.smithi183.stdout: "type": 1, 2024-09-10T19:20:46.369 INFO:teuthology.orchestra.run.smithi183.stdout: "size": 3, 2024-09-10T19:20:46.369 INFO:teuthology.orchestra.run.smithi183.stdout: "min_size": 2, 2024-09-10T19:20:46.369 INFO:teuthology.orchestra.run.smithi183.stdout: "crush_rule": 0, 2024-09-10T19:20:46.369 INFO:teuthology.orchestra.run.smithi183.stdout: "peering_crush_bucket_count": 0, 2024-09-10T19:20:46.369 INFO:teuthology.orchestra.run.smithi183.stdout: "peering_crush_bucket_target": 0, 2024-09-10T19:20:46.369 INFO:teuthology.orchestra.run.smithi183.stdout: "peering_crush_bucket_barrier": 0, 2024-09-10T19:20:46.369 INFO:teuthology.orchestra.run.smithi183.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-10T19:20:46.370 INFO:teuthology.orchestra.run.smithi183.stdout: "object_hash": 2, 2024-09-10T19:20:46.370 INFO:teuthology.orchestra.run.smithi183.stdout: "pg_autoscale_mode": "off", 2024-09-10T19:20:46.370 INFO:teuthology.orchestra.run.smithi183.stdout: "pg_num": 32, 2024-09-10T19:20:46.370 INFO:teuthology.orchestra.run.smithi183.stdout: "pg_placement_num": 32, 2024-09-10T19:20:46.370 INFO:teuthology.orchestra.run.smithi183.stdout: "pg_placement_num_target": 32, 2024-09-10T19:20:46.370 INFO:teuthology.orchestra.run.smithi183.stdout: "pg_num_target": 32, 2024-09-10T19:20:46.370 INFO:teuthology.orchestra.run.smithi183.stdout: "pg_num_pending": 32, 2024-09-10T19:20:46.370 INFO:teuthology.orchestra.run.smithi183.stdout: "last_pg_merge_meta": { 2024-09-10T19:20:46.370 INFO:teuthology.orchestra.run.smithi183.stdout: "source_pgid": "0.0", 2024-09-10T19:20:46.370 INFO:teuthology.orchestra.run.smithi183.stdout: "ready_epoch": 0, 2024-09-10T19:20:46.370 INFO:teuthology.orchestra.run.smithi183.stdout: "last_epoch_started": 0, 2024-09-10T19:20:46.370 INFO:teuthology.orchestra.run.smithi183.stdout: "last_epoch_clean": 0, 2024-09-10T19:20:46.370 INFO:teuthology.orchestra.run.smithi183.stdout: "source_version": "0'0", 2024-09-10T19:20:46.370 INFO:teuthology.orchestra.run.smithi183.stdout: "target_version": "0'0" 2024-09-10T19:20:46.371 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:20:46.371 INFO:teuthology.orchestra.run.smithi183.stdout: "last_change": "62", 2024-09-10T19:20:46.371 INFO:teuthology.orchestra.run.smithi183.stdout: "last_force_op_resend": "0", 2024-09-10T19:20:46.371 INFO:teuthology.orchestra.run.smithi183.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-10T19:20:46.371 INFO:teuthology.orchestra.run.smithi183.stdout: "last_force_op_resend_preluminous": "0", 2024-09-10T19:20:46.371 INFO:teuthology.orchestra.run.smithi183.stdout: "auid": 0, 2024-09-10T19:20:46.371 INFO:teuthology.orchestra.run.smithi183.stdout: "snap_mode": "selfmanaged", 2024-09-10T19:20:46.371 INFO:teuthology.orchestra.run.smithi183.stdout: "snap_seq": 0, 2024-09-10T19:20:46.371 INFO:teuthology.orchestra.run.smithi183.stdout: "snap_epoch": 0, 2024-09-10T19:20:46.371 INFO:teuthology.orchestra.run.smithi183.stdout: "pool_snaps": [], 2024-09-10T19:20:46.371 INFO:teuthology.orchestra.run.smithi183.stdout: "removed_snaps": "[]", 2024-09-10T19:20:46.371 INFO:teuthology.orchestra.run.smithi183.stdout: "quota_max_bytes": 0, 2024-09-10T19:20:46.371 INFO:teuthology.orchestra.run.smithi183.stdout: "quota_max_objects": 0, 2024-09-10T19:20:46.371 INFO:teuthology.orchestra.run.smithi183.stdout: "tiers": [], 2024-09-10T19:20:46.372 INFO:teuthology.orchestra.run.smithi183.stdout: "tier_of": -1, 2024-09-10T19:20:46.372 INFO:teuthology.orchestra.run.smithi183.stdout: "read_tier": -1, 2024-09-10T19:20:46.372 INFO:teuthology.orchestra.run.smithi183.stdout: "write_tier": -1, 2024-09-10T19:20:46.372 INFO:teuthology.orchestra.run.smithi183.stdout: "cache_mode": "none", 2024-09-10T19:20:46.372 INFO:teuthology.orchestra.run.smithi183.stdout: "target_max_bytes": 0, 2024-09-10T19:20:46.372 INFO:teuthology.orchestra.run.smithi183.stdout: "target_max_objects": 0, 2024-09-10T19:20:46.372 INFO:teuthology.orchestra.run.smithi183.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-10T19:20:46.372 INFO:teuthology.orchestra.run.smithi183.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-10T19:20:46.372 INFO:teuthology.orchestra.run.smithi183.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-10T19:20:46.372 INFO:teuthology.orchestra.run.smithi183.stdout: "cache_min_flush_age": 0, 2024-09-10T19:20:46.372 INFO:teuthology.orchestra.run.smithi183.stdout: "cache_min_evict_age": 0, 2024-09-10T19:20:46.372 INFO:teuthology.orchestra.run.smithi183.stdout: "erasure_code_profile": "", 2024-09-10T19:20:46.372 INFO:teuthology.orchestra.run.smithi183.stdout: "hit_set_params": { 2024-09-10T19:20:46.373 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "none" 2024-09-10T19:20:46.373 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:20:46.373 INFO:teuthology.orchestra.run.smithi183.stdout: "hit_set_period": 0, 2024-09-10T19:20:46.373 INFO:teuthology.orchestra.run.smithi183.stdout: "hit_set_count": 0, 2024-09-10T19:20:46.373 INFO:teuthology.orchestra.run.smithi183.stdout: "use_gmt_hitset": true, 2024-09-10T19:20:46.373 INFO:teuthology.orchestra.run.smithi183.stdout: "min_read_recency_for_promote": 0, 2024-09-10T19:20:46.373 INFO:teuthology.orchestra.run.smithi183.stdout: "min_write_recency_for_promote": 0, 2024-09-10T19:20:46.374 INFO:teuthology.orchestra.run.smithi183.stdout: "hit_set_grade_decay_rate": 0, 2024-09-10T19:20:46.374 INFO:teuthology.orchestra.run.smithi183.stdout: "hit_set_search_last_n": 0, 2024-09-10T19:20:46.374 INFO:teuthology.orchestra.run.smithi183.stdout: "grade_table": [], 2024-09-10T19:20:46.374 INFO:teuthology.orchestra.run.smithi183.stdout: "stripe_width": 0, 2024-09-10T19:20:46.374 INFO:teuthology.orchestra.run.smithi183.stdout: "expected_num_objects": 0, 2024-09-10T19:20:46.374 INFO:teuthology.orchestra.run.smithi183.stdout: "fast_read": false, 2024-09-10T19:20:46.374 INFO:teuthology.orchestra.run.smithi183.stdout: "options": {}, 2024-09-10T19:20:46.374 INFO:teuthology.orchestra.run.smithi183.stdout: "application_metadata": { 2024-09-10T19:20:46.374 INFO:teuthology.orchestra.run.smithi183.stdout: "rgw": {} 2024-09-10T19:20:46.375 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:20:46.375 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:20:46.375 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:20:46.375 INFO:teuthology.orchestra.run.smithi183.stdout: "pool": 5, 2024-09-10T19:20:46.375 INFO:teuthology.orchestra.run.smithi183.stdout: "pool_name": "default.rgw.control", 2024-09-10T19:20:46.375 INFO:teuthology.orchestra.run.smithi183.stdout: "create_time": "2024-09-10T19:10:09.933596+0000", 2024-09-10T19:20:46.375 INFO:teuthology.orchestra.run.smithi183.stdout: "flags": 1, 2024-09-10T19:20:46.375 INFO:teuthology.orchestra.run.smithi183.stdout: "flags_names": "hashpspool", 2024-09-10T19:20:46.375 INFO:teuthology.orchestra.run.smithi183.stdout: "type": 1, 2024-09-10T19:20:46.375 INFO:teuthology.orchestra.run.smithi183.stdout: "size": 3, 2024-09-10T19:20:46.375 INFO:teuthology.orchestra.run.smithi183.stdout: "min_size": 2, 2024-09-10T19:20:46.375 INFO:teuthology.orchestra.run.smithi183.stdout: "crush_rule": 0, 2024-09-10T19:20:46.375 INFO:teuthology.orchestra.run.smithi183.stdout: "peering_crush_bucket_count": 0, 2024-09-10T19:20:46.376 INFO:teuthology.orchestra.run.smithi183.stdout: "peering_crush_bucket_target": 0, 2024-09-10T19:20:46.376 INFO:teuthology.orchestra.run.smithi183.stdout: "peering_crush_bucket_barrier": 0, 2024-09-10T19:20:46.376 INFO:teuthology.orchestra.run.smithi183.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-10T19:20:46.376 INFO:teuthology.orchestra.run.smithi183.stdout: "object_hash": 2, 2024-09-10T19:20:46.376 INFO:teuthology.orchestra.run.smithi183.stdout: "pg_autoscale_mode": "off", 2024-09-10T19:20:46.376 INFO:teuthology.orchestra.run.smithi183.stdout: "pg_num": 32, 2024-09-10T19:20:46.376 INFO:teuthology.orchestra.run.smithi183.stdout: "pg_placement_num": 32, 2024-09-10T19:20:46.376 INFO:teuthology.orchestra.run.smithi183.stdout: "pg_placement_num_target": 32, 2024-09-10T19:20:46.376 INFO:teuthology.orchestra.run.smithi183.stdout: "pg_num_target": 32, 2024-09-10T19:20:46.376 INFO:teuthology.orchestra.run.smithi183.stdout: "pg_num_pending": 32, 2024-09-10T19:20:46.376 INFO:teuthology.orchestra.run.smithi183.stdout: "last_pg_merge_meta": { 2024-09-10T19:20:46.376 INFO:teuthology.orchestra.run.smithi183.stdout: "source_pgid": "0.0", 2024-09-10T19:20:46.376 INFO:teuthology.orchestra.run.smithi183.stdout: "ready_epoch": 0, 2024-09-10T19:20:46.377 INFO:teuthology.orchestra.run.smithi183.stdout: "last_epoch_started": 0, 2024-09-10T19:20:46.377 INFO:teuthology.orchestra.run.smithi183.stdout: "last_epoch_clean": 0, 2024-09-10T19:20:46.377 INFO:teuthology.orchestra.run.smithi183.stdout: "source_version": "0'0", 2024-09-10T19:20:46.377 INFO:teuthology.orchestra.run.smithi183.stdout: "target_version": "0'0" 2024-09-10T19:20:46.377 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:20:46.377 INFO:teuthology.orchestra.run.smithi183.stdout: "last_change": "64", 2024-09-10T19:20:46.377 INFO:teuthology.orchestra.run.smithi183.stdout: "last_force_op_resend": "0", 2024-09-10T19:20:46.377 INFO:teuthology.orchestra.run.smithi183.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-10T19:20:46.377 INFO:teuthology.orchestra.run.smithi183.stdout: "last_force_op_resend_preluminous": "0", 2024-09-10T19:20:46.377 INFO:teuthology.orchestra.run.smithi183.stdout: "auid": 0, 2024-09-10T19:20:46.377 INFO:teuthology.orchestra.run.smithi183.stdout: "snap_mode": "selfmanaged", 2024-09-10T19:20:46.377 INFO:teuthology.orchestra.run.smithi183.stdout: "snap_seq": 0, 2024-09-10T19:20:46.377 INFO:teuthology.orchestra.run.smithi183.stdout: "snap_epoch": 0, 2024-09-10T19:20:46.377 INFO:teuthology.orchestra.run.smithi183.stdout: "pool_snaps": [], 2024-09-10T19:20:46.378 INFO:teuthology.orchestra.run.smithi183.stdout: "removed_snaps": "[]", 2024-09-10T19:20:46.378 INFO:teuthology.orchestra.run.smithi183.stdout: "quota_max_bytes": 0, 2024-09-10T19:20:46.378 INFO:teuthology.orchestra.run.smithi183.stdout: "quota_max_objects": 0, 2024-09-10T19:20:46.378 INFO:teuthology.orchestra.run.smithi183.stdout: "tiers": [], 2024-09-10T19:20:46.378 INFO:teuthology.orchestra.run.smithi183.stdout: "tier_of": -1, 2024-09-10T19:20:46.378 INFO:teuthology.orchestra.run.smithi183.stdout: "read_tier": -1, 2024-09-10T19:20:46.378 INFO:teuthology.orchestra.run.smithi183.stdout: "write_tier": -1, 2024-09-10T19:20:46.378 INFO:teuthology.orchestra.run.smithi183.stdout: "cache_mode": "none", 2024-09-10T19:20:46.378 INFO:teuthology.orchestra.run.smithi183.stdout: "target_max_bytes": 0, 2024-09-10T19:20:46.379 INFO:teuthology.orchestra.run.smithi183.stdout: "target_max_objects": 0, 2024-09-10T19:20:46.379 INFO:teuthology.orchestra.run.smithi183.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-10T19:20:46.379 INFO:teuthology.orchestra.run.smithi183.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-10T19:20:46.379 INFO:teuthology.orchestra.run.smithi183.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-10T19:20:46.379 INFO:teuthology.orchestra.run.smithi183.stdout: "cache_min_flush_age": 0, 2024-09-10T19:20:46.379 INFO:teuthology.orchestra.run.smithi183.stdout: "cache_min_evict_age": 0, 2024-09-10T19:20:46.380 INFO:teuthology.orchestra.run.smithi183.stdout: "erasure_code_profile": "", 2024-09-10T19:20:46.380 INFO:teuthology.orchestra.run.smithi183.stdout: "hit_set_params": { 2024-09-10T19:20:46.380 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "none" 2024-09-10T19:20:46.380 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:20:46.380 INFO:teuthology.orchestra.run.smithi183.stdout: "hit_set_period": 0, 2024-09-10T19:20:46.380 INFO:teuthology.orchestra.run.smithi183.stdout: "hit_set_count": 0, 2024-09-10T19:20:46.380 INFO:teuthology.orchestra.run.smithi183.stdout: "use_gmt_hitset": true, 2024-09-10T19:20:46.381 INFO:teuthology.orchestra.run.smithi183.stdout: "min_read_recency_for_promote": 0, 2024-09-10T19:20:46.381 INFO:teuthology.orchestra.run.smithi183.stdout: "min_write_recency_for_promote": 0, 2024-09-10T19:20:46.381 INFO:teuthology.orchestra.run.smithi183.stdout: "hit_set_grade_decay_rate": 0, 2024-09-10T19:20:46.381 INFO:teuthology.orchestra.run.smithi183.stdout: "hit_set_search_last_n": 0, 2024-09-10T19:20:46.381 INFO:teuthology.orchestra.run.smithi183.stdout: "grade_table": [], 2024-09-10T19:20:46.381 INFO:teuthology.orchestra.run.smithi183.stdout: "stripe_width": 0, 2024-09-10T19:20:46.382 INFO:teuthology.orchestra.run.smithi183.stdout: "expected_num_objects": 0, 2024-09-10T19:20:46.382 INFO:teuthology.orchestra.run.smithi183.stdout: "fast_read": false, 2024-09-10T19:20:46.382 INFO:teuthology.orchestra.run.smithi183.stdout: "options": {}, 2024-09-10T19:20:46.382 INFO:teuthology.orchestra.run.smithi183.stdout: "application_metadata": { 2024-09-10T19:20:46.382 INFO:teuthology.orchestra.run.smithi183.stdout: "rgw": {} 2024-09-10T19:20:46.382 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:20:46.382 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:20:46.383 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:20:46.383 INFO:teuthology.orchestra.run.smithi183.stdout: "pool": 6, 2024-09-10T19:20:46.383 INFO:teuthology.orchestra.run.smithi183.stdout: "pool_name": "default.rgw.meta", 2024-09-10T19:20:46.383 INFO:teuthology.orchestra.run.smithi183.stdout: "create_time": "2024-09-10T19:10:11.993712+0000", 2024-09-10T19:20:46.383 INFO:teuthology.orchestra.run.smithi183.stdout: "flags": 1, 2024-09-10T19:20:46.383 INFO:teuthology.orchestra.run.smithi183.stdout: "flags_names": "hashpspool", 2024-09-10T19:20:46.384 INFO:teuthology.orchestra.run.smithi183.stdout: "type": 1, 2024-09-10T19:20:46.384 INFO:teuthology.orchestra.run.smithi183.stdout: "size": 3, 2024-09-10T19:20:46.384 INFO:teuthology.orchestra.run.smithi183.stdout: "min_size": 2, 2024-09-10T19:20:46.384 INFO:teuthology.orchestra.run.smithi183.stdout: "crush_rule": 0, 2024-09-10T19:20:46.384 INFO:teuthology.orchestra.run.smithi183.stdout: "peering_crush_bucket_count": 0, 2024-09-10T19:20:46.384 INFO:teuthology.orchestra.run.smithi183.stdout: "peering_crush_bucket_target": 0, 2024-09-10T19:20:46.385 INFO:teuthology.orchestra.run.smithi183.stdout: "peering_crush_bucket_barrier": 0, 2024-09-10T19:20:46.385 INFO:teuthology.orchestra.run.smithi183.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-10T19:20:46.385 INFO:teuthology.orchestra.run.smithi183.stdout: "object_hash": 2, 2024-09-10T19:20:46.385 INFO:teuthology.orchestra.run.smithi183.stdout: "pg_autoscale_mode": "off", 2024-09-10T19:20:46.385 INFO:teuthology.orchestra.run.smithi183.stdout: "pg_num": 32, 2024-09-10T19:20:46.385 INFO:teuthology.orchestra.run.smithi183.stdout: "pg_placement_num": 32, 2024-09-10T19:20:46.385 INFO:teuthology.orchestra.run.smithi183.stdout: "pg_placement_num_target": 32, 2024-09-10T19:20:46.385 INFO:teuthology.orchestra.run.smithi183.stdout: "pg_num_target": 32, 2024-09-10T19:20:46.385 INFO:teuthology.orchestra.run.smithi183.stdout: "pg_num_pending": 32, 2024-09-10T19:20:46.385 INFO:teuthology.orchestra.run.smithi183.stdout: "last_pg_merge_meta": { 2024-09-10T19:20:46.385 INFO:teuthology.orchestra.run.smithi183.stdout: "source_pgid": "0.0", 2024-09-10T19:20:46.386 INFO:teuthology.orchestra.run.smithi183.stdout: "ready_epoch": 0, 2024-09-10T19:20:46.386 INFO:teuthology.orchestra.run.smithi183.stdout: "last_epoch_started": 0, 2024-09-10T19:20:46.386 INFO:teuthology.orchestra.run.smithi183.stdout: "last_epoch_clean": 0, 2024-09-10T19:20:46.386 INFO:teuthology.orchestra.run.smithi183.stdout: "source_version": "0'0", 2024-09-10T19:20:46.386 INFO:teuthology.orchestra.run.smithi183.stdout: "target_version": "0'0" 2024-09-10T19:20:46.386 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:20:46.386 INFO:teuthology.orchestra.run.smithi183.stdout: "last_change": "66", 2024-09-10T19:20:46.386 INFO:teuthology.orchestra.run.smithi183.stdout: "last_force_op_resend": "0", 2024-09-10T19:20:46.386 INFO:teuthology.orchestra.run.smithi183.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-10T19:20:46.386 INFO:teuthology.orchestra.run.smithi183.stdout: "last_force_op_resend_preluminous": "0", 2024-09-10T19:20:46.387 INFO:teuthology.orchestra.run.smithi183.stdout: "auid": 0, 2024-09-10T19:20:46.387 INFO:teuthology.orchestra.run.smithi183.stdout: "snap_mode": "selfmanaged", 2024-09-10T19:20:46.387 INFO:teuthology.orchestra.run.smithi183.stdout: "snap_seq": 0, 2024-09-10T19:20:46.387 INFO:teuthology.orchestra.run.smithi183.stdout: "snap_epoch": 0, 2024-09-10T19:20:46.387 INFO:teuthology.orchestra.run.smithi183.stdout: "pool_snaps": [], 2024-09-10T19:20:46.387 INFO:teuthology.orchestra.run.smithi183.stdout: "removed_snaps": "[]", 2024-09-10T19:20:46.387 INFO:teuthology.orchestra.run.smithi183.stdout: "quota_max_bytes": 0, 2024-09-10T19:20:46.387 INFO:teuthology.orchestra.run.smithi183.stdout: "quota_max_objects": 0, 2024-09-10T19:20:46.388 INFO:teuthology.orchestra.run.smithi183.stdout: "tiers": [], 2024-09-10T19:20:46.388 INFO:teuthology.orchestra.run.smithi183.stdout: "tier_of": -1, 2024-09-10T19:20:46.388 INFO:teuthology.orchestra.run.smithi183.stdout: "read_tier": -1, 2024-09-10T19:20:46.388 INFO:teuthology.orchestra.run.smithi183.stdout: "write_tier": -1, 2024-09-10T19:20:46.388 INFO:teuthology.orchestra.run.smithi183.stdout: "cache_mode": "none", 2024-09-10T19:20:46.388 INFO:teuthology.orchestra.run.smithi183.stdout: "target_max_bytes": 0, 2024-09-10T19:20:46.388 INFO:teuthology.orchestra.run.smithi183.stdout: "target_max_objects": 0, 2024-09-10T19:20:46.388 INFO:teuthology.orchestra.run.smithi183.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-10T19:20:46.388 INFO:teuthology.orchestra.run.smithi183.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-10T19:20:46.388 INFO:teuthology.orchestra.run.smithi183.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-10T19:20:46.388 INFO:teuthology.orchestra.run.smithi183.stdout: "cache_min_flush_age": 0, 2024-09-10T19:20:46.388 INFO:teuthology.orchestra.run.smithi183.stdout: "cache_min_evict_age": 0, 2024-09-10T19:20:46.388 INFO:teuthology.orchestra.run.smithi183.stdout: "erasure_code_profile": "", 2024-09-10T19:20:46.389 INFO:teuthology.orchestra.run.smithi183.stdout: "hit_set_params": { 2024-09-10T19:20:46.389 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "none" 2024-09-10T19:20:46.389 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:20:46.389 INFO:teuthology.orchestra.run.smithi183.stdout: "hit_set_period": 0, 2024-09-10T19:20:46.389 INFO:teuthology.orchestra.run.smithi183.stdout: "hit_set_count": 0, 2024-09-10T19:20:46.389 INFO:teuthology.orchestra.run.smithi183.stdout: "use_gmt_hitset": true, 2024-09-10T19:20:46.389 INFO:teuthology.orchestra.run.smithi183.stdout: "min_read_recency_for_promote": 0, 2024-09-10T19:20:46.389 INFO:teuthology.orchestra.run.smithi183.stdout: "min_write_recency_for_promote": 0, 2024-09-10T19:20:46.389 INFO:teuthology.orchestra.run.smithi183.stdout: "hit_set_grade_decay_rate": 0, 2024-09-10T19:20:46.389 INFO:teuthology.orchestra.run.smithi183.stdout: "hit_set_search_last_n": 0, 2024-09-10T19:20:46.389 INFO:teuthology.orchestra.run.smithi183.stdout: "grade_table": [], 2024-09-10T19:20:46.389 INFO:teuthology.orchestra.run.smithi183.stdout: "stripe_width": 0, 2024-09-10T19:20:46.390 INFO:teuthology.orchestra.run.smithi183.stdout: "expected_num_objects": 0, 2024-09-10T19:20:46.390 INFO:teuthology.orchestra.run.smithi183.stdout: "fast_read": false, 2024-09-10T19:20:46.390 INFO:teuthology.orchestra.run.smithi183.stdout: "options": { 2024-09-10T19:20:46.390 INFO:teuthology.orchestra.run.smithi183.stdout: "pg_autoscale_bias": 4 2024-09-10T19:20:46.390 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:20:46.390 INFO:teuthology.orchestra.run.smithi183.stdout: "application_metadata": { 2024-09-10T19:20:46.390 INFO:teuthology.orchestra.run.smithi183.stdout: "rgw": {} 2024-09-10T19:20:46.390 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:20:46.391 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:20:46.391 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:20:46.391 INFO:teuthology.orchestra.run.smithi183.stdout: "pool": 9, 2024-09-10T19:20:46.391 INFO:teuthology.orchestra.run.smithi183.stdout: "pool_name": "unique_pool_2", 2024-09-10T19:20:46.391 INFO:teuthology.orchestra.run.smithi183.stdout: "create_time": "2024-09-10T19:19:53.124664+0000", 2024-09-10T19:20:46.391 INFO:teuthology.orchestra.run.smithi183.stdout: "flags": 1, 2024-09-10T19:20:46.391 INFO:teuthology.orchestra.run.smithi183.stdout: "flags_names": "hashpspool", 2024-09-10T19:20:46.391 INFO:teuthology.orchestra.run.smithi183.stdout: "type": 1, 2024-09-10T19:20:46.391 INFO:teuthology.orchestra.run.smithi183.stdout: "size": 3, 2024-09-10T19:20:46.391 INFO:teuthology.orchestra.run.smithi183.stdout: "min_size": 2, 2024-09-10T19:20:46.391 INFO:teuthology.orchestra.run.smithi183.stdout: "crush_rule": 0, 2024-09-10T19:20:46.391 INFO:teuthology.orchestra.run.smithi183.stdout: "peering_crush_bucket_count": 0, 2024-09-10T19:20:46.391 INFO:teuthology.orchestra.run.smithi183.stdout: "peering_crush_bucket_target": 0, 2024-09-10T19:20:46.391 INFO:teuthology.orchestra.run.smithi183.stdout: "peering_crush_bucket_barrier": 0, 2024-09-10T19:20:46.391 INFO:teuthology.orchestra.run.smithi183.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-10T19:20:46.392 INFO:teuthology.orchestra.run.smithi183.stdout: "object_hash": 2, 2024-09-10T19:20:46.392 INFO:teuthology.orchestra.run.smithi183.stdout: "pg_autoscale_mode": "off", 2024-09-10T19:20:46.392 INFO:teuthology.orchestra.run.smithi183.stdout: "pg_num": 16, 2024-09-10T19:20:46.392 INFO:teuthology.orchestra.run.smithi183.stdout: "pg_placement_num": 16, 2024-09-10T19:20:46.392 INFO:teuthology.orchestra.run.smithi183.stdout: "pg_placement_num_target": 16, 2024-09-10T19:20:46.392 INFO:teuthology.orchestra.run.smithi183.stdout: "pg_num_target": 16, 2024-09-10T19:20:46.392 INFO:teuthology.orchestra.run.smithi183.stdout: "pg_num_pending": 16, 2024-09-10T19:20:46.392 INFO:teuthology.orchestra.run.smithi183.stdout: "last_pg_merge_meta": { 2024-09-10T19:20:46.392 INFO:teuthology.orchestra.run.smithi183.stdout: "source_pgid": "0.0", 2024-09-10T19:20:46.392 INFO:teuthology.orchestra.run.smithi183.stdout: "ready_epoch": 0, 2024-09-10T19:20:46.392 INFO:teuthology.orchestra.run.smithi183.stdout: "last_epoch_started": 0, 2024-09-10T19:20:46.392 INFO:teuthology.orchestra.run.smithi183.stdout: "last_epoch_clean": 0, 2024-09-10T19:20:46.392 INFO:teuthology.orchestra.run.smithi183.stdout: "source_version": "0'0", 2024-09-10T19:20:46.393 INFO:teuthology.orchestra.run.smithi183.stdout: "target_version": "0'0" 2024-09-10T19:20:46.393 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:20:46.393 INFO:teuthology.orchestra.run.smithi183.stdout: "last_change": "232", 2024-09-10T19:20:46.393 INFO:teuthology.orchestra.run.smithi183.stdout: "last_force_op_resend": "0", 2024-09-10T19:20:46.393 INFO:teuthology.orchestra.run.smithi183.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-10T19:20:46.393 INFO:teuthology.orchestra.run.smithi183.stdout: "last_force_op_resend_preluminous": "0", 2024-09-10T19:20:46.393 INFO:teuthology.orchestra.run.smithi183.stdout: "auid": 0, 2024-09-10T19:20:46.393 INFO:teuthology.orchestra.run.smithi183.stdout: "snap_mode": "selfmanaged", 2024-09-10T19:20:46.393 INFO:teuthology.orchestra.run.smithi183.stdout: "snap_seq": 0, 2024-09-10T19:20:46.393 INFO:teuthology.orchestra.run.smithi183.stdout: "snap_epoch": 0, 2024-09-10T19:20:46.393 INFO:teuthology.orchestra.run.smithi183.stdout: "pool_snaps": [], 2024-09-10T19:20:46.394 INFO:teuthology.orchestra.run.smithi183.stdout: "removed_snaps": "[]", 2024-09-10T19:20:46.394 INFO:teuthology.orchestra.run.smithi183.stdout: "quota_max_bytes": 0, 2024-09-10T19:20:46.394 INFO:teuthology.orchestra.run.smithi183.stdout: "quota_max_objects": 0, 2024-09-10T19:20:46.394 INFO:teuthology.orchestra.run.smithi183.stdout: "tiers": [], 2024-09-10T19:20:46.394 INFO:teuthology.orchestra.run.smithi183.stdout: "tier_of": -1, 2024-09-10T19:20:46.394 INFO:teuthology.orchestra.run.smithi183.stdout: "read_tier": -1, 2024-09-10T19:20:46.394 INFO:teuthology.orchestra.run.smithi183.stdout: "write_tier": -1, 2024-09-10T19:20:46.394 INFO:teuthology.orchestra.run.smithi183.stdout: "cache_mode": "none", 2024-09-10T19:20:46.394 INFO:teuthology.orchestra.run.smithi183.stdout: "target_max_bytes": 0, 2024-09-10T19:20:46.394 INFO:teuthology.orchestra.run.smithi183.stdout: "target_max_objects": 0, 2024-09-10T19:20:46.394 INFO:teuthology.orchestra.run.smithi183.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-10T19:20:46.394 INFO:teuthology.orchestra.run.smithi183.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-10T19:20:46.394 INFO:teuthology.orchestra.run.smithi183.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-10T19:20:46.394 INFO:teuthology.orchestra.run.smithi183.stdout: "cache_min_flush_age": 0, 2024-09-10T19:20:46.395 INFO:teuthology.orchestra.run.smithi183.stdout: "cache_min_evict_age": 0, 2024-09-10T19:20:46.395 INFO:teuthology.orchestra.run.smithi183.stdout: "erasure_code_profile": "", 2024-09-10T19:20:46.395 INFO:teuthology.orchestra.run.smithi183.stdout: "hit_set_params": { 2024-09-10T19:20:46.395 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "none" 2024-09-10T19:20:46.395 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:20:46.395 INFO:teuthology.orchestra.run.smithi183.stdout: "hit_set_period": 0, 2024-09-10T19:20:46.395 INFO:teuthology.orchestra.run.smithi183.stdout: "hit_set_count": 0, 2024-09-10T19:20:46.395 INFO:teuthology.orchestra.run.smithi183.stdout: "use_gmt_hitset": true, 2024-09-10T19:20:46.395 INFO:teuthology.orchestra.run.smithi183.stdout: "min_read_recency_for_promote": 0, 2024-09-10T19:20:46.395 INFO:teuthology.orchestra.run.smithi183.stdout: "min_write_recency_for_promote": 0, 2024-09-10T19:20:46.395 INFO:teuthology.orchestra.run.smithi183.stdout: "hit_set_grade_decay_rate": 0, 2024-09-10T19:20:46.395 INFO:teuthology.orchestra.run.smithi183.stdout: "hit_set_search_last_n": 0, 2024-09-10T19:20:46.395 INFO:teuthology.orchestra.run.smithi183.stdout: "grade_table": [], 2024-09-10T19:20:46.396 INFO:teuthology.orchestra.run.smithi183.stdout: "stripe_width": 0, 2024-09-10T19:20:46.396 INFO:teuthology.orchestra.run.smithi183.stdout: "expected_num_objects": 0, 2024-09-10T19:20:46.396 INFO:teuthology.orchestra.run.smithi183.stdout: "fast_read": false, 2024-09-10T19:20:46.396 INFO:teuthology.orchestra.run.smithi183.stdout: "options": {}, 2024-09-10T19:20:46.396 INFO:teuthology.orchestra.run.smithi183.stdout: "application_metadata": { 2024-09-10T19:20:46.396 INFO:teuthology.orchestra.run.smithi183.stdout: "rados": {} 2024-09-10T19:20:46.396 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:20:46.396 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:20:46.396 INFO:teuthology.orchestra.run.smithi183.stdout: ], 2024-09-10T19:20:46.396 INFO:teuthology.orchestra.run.smithi183.stdout: "osds": [ 2024-09-10T19:20:46.396 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:20:46.396 INFO:teuthology.orchestra.run.smithi183.stdout: "osd": 0, 2024-09-10T19:20:46.397 INFO:teuthology.orchestra.run.smithi183.stdout: "uuid": "9d323010-394a-49aa-ba9b-244d87faf36e", 2024-09-10T19:20:46.397 INFO:teuthology.orchestra.run.smithi183.stdout: "up": 1, 2024-09-10T19:20:46.397 INFO:teuthology.orchestra.run.smithi183.stdout: "in": 1, 2024-09-10T19:20:46.397 INFO:teuthology.orchestra.run.smithi183.stdout: "weight": 1, 2024-09-10T19:20:46.397 INFO:teuthology.orchestra.run.smithi183.stdout: "primary_affinity": 1, 2024-09-10T19:20:46.397 INFO:teuthology.orchestra.run.smithi183.stdout: "last_clean_begin": 0, 2024-09-10T19:20:46.397 INFO:teuthology.orchestra.run.smithi183.stdout: "last_clean_end": 0, 2024-09-10T19:20:46.397 INFO:teuthology.orchestra.run.smithi183.stdout: "up_from": 8, 2024-09-10T19:20:46.397 INFO:teuthology.orchestra.run.smithi183.stdout: "up_thru": 229, 2024-09-10T19:20:46.397 INFO:teuthology.orchestra.run.smithi183.stdout: "down_at": 0, 2024-09-10T19:20:46.397 INFO:teuthology.orchestra.run.smithi183.stdout: "lost_at": 0, 2024-09-10T19:20:46.397 INFO:teuthology.orchestra.run.smithi183.stdout: "public_addrs": { 2024-09-10T19:20:46.397 INFO:teuthology.orchestra.run.smithi183.stdout: "addrvec": [ 2024-09-10T19:20:46.398 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:20:46.398 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v2", 2024-09-10T19:20:46.398 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.183:6802", 2024-09-10T19:20:46.398 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 2813074952 2024-09-10T19:20:46.398 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:20:46.398 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:20:46.398 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v1", 2024-09-10T19:20:46.398 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.183:6803", 2024-09-10T19:20:46.398 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 2813074952 2024-09-10T19:20:46.398 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:20:46.398 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:20:46.398 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:20:46.398 INFO:teuthology.orchestra.run.smithi183.stdout: "cluster_addrs": { 2024-09-10T19:20:46.398 INFO:teuthology.orchestra.run.smithi183.stdout: "addrvec": [ 2024-09-10T19:20:46.399 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:20:46.399 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v2", 2024-09-10T19:20:46.399 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.183:6804", 2024-09-10T19:20:46.399 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 2813074952 2024-09-10T19:20:46.399 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:20:46.399 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:20:46.399 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v1", 2024-09-10T19:20:46.399 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.183:6805", 2024-09-10T19:20:46.399 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 2813074952 2024-09-10T19:20:46.399 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:20:46.399 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:20:46.399 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:20:46.399 INFO:teuthology.orchestra.run.smithi183.stdout: "heartbeat_back_addrs": { 2024-09-10T19:20:46.400 INFO:teuthology.orchestra.run.smithi183.stdout: "addrvec": [ 2024-09-10T19:20:46.400 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:20:46.400 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v2", 2024-09-10T19:20:46.400 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.183:6808", 2024-09-10T19:20:46.400 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 2813074952 2024-09-10T19:20:46.400 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:20:46.400 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:20:46.400 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v1", 2024-09-10T19:20:46.400 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.183:6809", 2024-09-10T19:20:46.400 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 2813074952 2024-09-10T19:20:46.400 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:20:46.401 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:20:46.401 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:20:46.401 INFO:teuthology.orchestra.run.smithi183.stdout: "heartbeat_front_addrs": { 2024-09-10T19:20:46.401 INFO:teuthology.orchestra.run.smithi183.stdout: "addrvec": [ 2024-09-10T19:20:46.401 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:20:46.401 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v2", 2024-09-10T19:20:46.401 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.183:6806", 2024-09-10T19:20:46.401 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 2813074952 2024-09-10T19:20:46.401 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:20:46.401 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:20:46.401 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v1", 2024-09-10T19:20:46.401 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.183:6807", 2024-09-10T19:20:46.401 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 2813074952 2024-09-10T19:20:46.402 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:20:46.402 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:20:46.402 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:20:46.402 INFO:teuthology.orchestra.run.smithi183.stdout: "public_addr": "172.21.15.183:6803/2813074952", 2024-09-10T19:20:46.402 INFO:teuthology.orchestra.run.smithi183.stdout: "cluster_addr": "172.21.15.183:6805/2813074952", 2024-09-10T19:20:46.402 INFO:teuthology.orchestra.run.smithi183.stdout: "heartbeat_back_addr": "172.21.15.183:6809/2813074952", 2024-09-10T19:20:46.402 INFO:teuthology.orchestra.run.smithi183.stdout: "heartbeat_front_addr": "172.21.15.183:6807/2813074952", 2024-09-10T19:20:46.402 INFO:teuthology.orchestra.run.smithi183.stdout: "state": [ 2024-09-10T19:20:46.402 INFO:teuthology.orchestra.run.smithi183.stdout: "exists", 2024-09-10T19:20:46.402 INFO:teuthology.orchestra.run.smithi183.stdout: "up" 2024-09-10T19:20:46.402 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:20:46.402 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:20:46.403 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:20:46.403 INFO:teuthology.orchestra.run.smithi183.stdout: "osd": 1, 2024-09-10T19:20:46.403 INFO:teuthology.orchestra.run.smithi183.stdout: "uuid": "71a6c5dc-5109-4284-8d20-187d8e7afc9e", 2024-09-10T19:20:46.403 INFO:teuthology.orchestra.run.smithi183.stdout: "up": 1, 2024-09-10T19:20:46.403 INFO:teuthology.orchestra.run.smithi183.stdout: "in": 0, 2024-09-10T19:20:46.403 INFO:teuthology.orchestra.run.smithi183.stdout: "weight": 0, 2024-09-10T19:20:46.403 INFO:teuthology.orchestra.run.smithi183.stdout: "primary_affinity": 1, 2024-09-10T19:20:46.403 INFO:teuthology.orchestra.run.smithi183.stdout: "last_clean_begin": 13, 2024-09-10T19:20:46.403 INFO:teuthology.orchestra.run.smithi183.stdout: "last_clean_end": 98, 2024-09-10T19:20:46.403 INFO:teuthology.orchestra.run.smithi183.stdout: "up_from": 138, 2024-09-10T19:20:46.403 INFO:teuthology.orchestra.run.smithi183.stdout: "up_thru": 86, 2024-09-10T19:20:46.403 INFO:teuthology.orchestra.run.smithi183.stdout: "down_at": 99, 2024-09-10T19:20:46.403 INFO:teuthology.orchestra.run.smithi183.stdout: "lost_at": 0, 2024-09-10T19:20:46.403 INFO:teuthology.orchestra.run.smithi183.stdout: "public_addrs": { 2024-09-10T19:20:46.404 INFO:teuthology.orchestra.run.smithi183.stdout: "addrvec": [ 2024-09-10T19:20:46.404 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:20:46.404 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v2", 2024-09-10T19:20:46.404 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.183:6810", 2024-09-10T19:20:46.404 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 219854798 2024-09-10T19:20:46.404 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:20:46.404 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:20:46.404 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v1", 2024-09-10T19:20:46.404 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.183:6811", 2024-09-10T19:20:46.404 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 219854798 2024-09-10T19:20:46.404 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:20:46.404 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:20:46.404 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:20:46.405 INFO:teuthology.orchestra.run.smithi183.stdout: "cluster_addrs": { 2024-09-10T19:20:46.405 INFO:teuthology.orchestra.run.smithi183.stdout: "addrvec": [ 2024-09-10T19:20:46.405 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:20:46.405 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v2", 2024-09-10T19:20:46.405 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.183:6812", 2024-09-10T19:20:46.405 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 219854798 2024-09-10T19:20:46.405 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:20:46.405 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:20:46.405 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v1", 2024-09-10T19:20:46.405 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.183:6813", 2024-09-10T19:20:46.405 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 219854798 2024-09-10T19:20:46.405 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:20:46.405 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:20:46.405 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:20:46.406 INFO:teuthology.orchestra.run.smithi183.stdout: "heartbeat_back_addrs": { 2024-09-10T19:20:46.406 INFO:teuthology.orchestra.run.smithi183.stdout: "addrvec": [ 2024-09-10T19:20:46.406 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:20:46.406 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v2", 2024-09-10T19:20:46.406 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.183:6816", 2024-09-10T19:20:46.406 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 219854798 2024-09-10T19:20:46.406 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:20:46.406 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:20:46.406 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v1", 2024-09-10T19:20:46.406 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.183:6817", 2024-09-10T19:20:46.406 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 219854798 2024-09-10T19:20:46.406 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:20:46.406 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:20:46.406 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:20:46.407 INFO:teuthology.orchestra.run.smithi183.stdout: "heartbeat_front_addrs": { 2024-09-10T19:20:46.407 INFO:teuthology.orchestra.run.smithi183.stdout: "addrvec": [ 2024-09-10T19:20:46.407 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:20:46.407 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v2", 2024-09-10T19:20:46.407 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.183:6814", 2024-09-10T19:20:46.407 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 219854798 2024-09-10T19:20:46.407 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:20:46.407 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:20:46.407 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v1", 2024-09-10T19:20:46.407 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.183:6815", 2024-09-10T19:20:46.407 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 219854798 2024-09-10T19:20:46.407 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:20:46.407 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:20:46.407 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:20:46.408 INFO:teuthology.orchestra.run.smithi183.stdout: "public_addr": "172.21.15.183:6811/219854798", 2024-09-10T19:20:46.408 INFO:teuthology.orchestra.run.smithi183.stdout: "cluster_addr": "172.21.15.183:6813/219854798", 2024-09-10T19:20:46.408 INFO:teuthology.orchestra.run.smithi183.stdout: "heartbeat_back_addr": "172.21.15.183:6817/219854798", 2024-09-10T19:20:46.408 INFO:teuthology.orchestra.run.smithi183.stdout: "heartbeat_front_addr": "172.21.15.183:6815/219854798", 2024-09-10T19:20:46.408 INFO:teuthology.orchestra.run.smithi183.stdout: "state": [ 2024-09-10T19:20:46.408 INFO:teuthology.orchestra.run.smithi183.stdout: "exists", 2024-09-10T19:20:46.408 INFO:teuthology.orchestra.run.smithi183.stdout: "up" 2024-09-10T19:20:46.408 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:20:46.408 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:20:46.408 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:20:46.408 INFO:teuthology.orchestra.run.smithi183.stdout: "osd": 2, 2024-09-10T19:20:46.408 INFO:teuthology.orchestra.run.smithi183.stdout: "uuid": "fc310d5b-b89b-4655-9989-7e10ba356509", 2024-09-10T19:20:46.408 INFO:teuthology.orchestra.run.smithi183.stdout: "up": 1, 2024-09-10T19:20:46.409 INFO:teuthology.orchestra.run.smithi183.stdout: "in": 0, 2024-09-10T19:20:46.409 INFO:teuthology.orchestra.run.smithi183.stdout: "weight": 0, 2024-09-10T19:20:46.409 INFO:teuthology.orchestra.run.smithi183.stdout: "primary_affinity": 1, 2024-09-10T19:20:46.409 INFO:teuthology.orchestra.run.smithi183.stdout: "last_clean_begin": 0, 2024-09-10T19:20:46.409 INFO:teuthology.orchestra.run.smithi183.stdout: "last_clean_end": 0, 2024-09-10T19:20:46.409 INFO:teuthology.orchestra.run.smithi183.stdout: "up_from": 18, 2024-09-10T19:20:46.409 INFO:teuthology.orchestra.run.smithi183.stdout: "up_thru": 148, 2024-09-10T19:20:46.409 INFO:teuthology.orchestra.run.smithi183.stdout: "down_at": 0, 2024-09-10T19:20:46.409 INFO:teuthology.orchestra.run.smithi183.stdout: "lost_at": 0, 2024-09-10T19:20:46.409 INFO:teuthology.orchestra.run.smithi183.stdout: "public_addrs": { 2024-09-10T19:20:46.410 INFO:teuthology.orchestra.run.smithi183.stdout: "addrvec": [ 2024-09-10T19:20:46.410 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:20:46.410 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v2", 2024-09-10T19:20:46.410 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.183:6818", 2024-09-10T19:20:46.410 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 3730013786 2024-09-10T19:20:46.410 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:20:46.410 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:20:46.410 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v1", 2024-09-10T19:20:46.411 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.183:6819", 2024-09-10T19:20:46.411 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 3730013786 2024-09-10T19:20:46.411 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:20:46.411 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:20:46.411 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:20:46.411 INFO:teuthology.orchestra.run.smithi183.stdout: "cluster_addrs": { 2024-09-10T19:20:46.411 INFO:teuthology.orchestra.run.smithi183.stdout: "addrvec": [ 2024-09-10T19:20:46.411 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:20:46.411 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v2", 2024-09-10T19:20:46.411 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.183:6820", 2024-09-10T19:20:46.411 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 3730013786 2024-09-10T19:20:46.411 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:20:46.411 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:20:46.411 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v1", 2024-09-10T19:20:46.412 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.183:6821", 2024-09-10T19:20:46.412 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 3730013786 2024-09-10T19:20:46.412 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:20:46.412 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:20:46.412 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:20:46.412 INFO:teuthology.orchestra.run.smithi183.stdout: "heartbeat_back_addrs": { 2024-09-10T19:20:46.412 INFO:teuthology.orchestra.run.smithi183.stdout: "addrvec": [ 2024-09-10T19:20:46.412 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:20:46.412 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v2", 2024-09-10T19:20:46.412 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.183:6824", 2024-09-10T19:20:46.412 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 3730013786 2024-09-10T19:20:46.412 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:20:46.412 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:20:46.412 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v1", 2024-09-10T19:20:46.412 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.183:6825", 2024-09-10T19:20:46.413 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 3730013786 2024-09-10T19:20:46.413 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:20:46.413 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:20:46.413 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:20:46.413 INFO:teuthology.orchestra.run.smithi183.stdout: "heartbeat_front_addrs": { 2024-09-10T19:20:46.413 INFO:teuthology.orchestra.run.smithi183.stdout: "addrvec": [ 2024-09-10T19:20:46.413 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:20:46.413 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v2", 2024-09-10T19:20:46.413 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.183:6822", 2024-09-10T19:20:46.413 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 3730013786 2024-09-10T19:20:46.413 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:20:46.413 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:20:46.414 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v1", 2024-09-10T19:20:46.414 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.183:6823", 2024-09-10T19:20:46.414 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 3730013786 2024-09-10T19:20:46.414 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:20:46.414 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:20:46.414 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:20:46.414 INFO:teuthology.orchestra.run.smithi183.stdout: "public_addr": "172.21.15.183:6819/3730013786", 2024-09-10T19:20:46.414 INFO:teuthology.orchestra.run.smithi183.stdout: "cluster_addr": "172.21.15.183:6821/3730013786", 2024-09-10T19:20:46.414 INFO:teuthology.orchestra.run.smithi183.stdout: "heartbeat_back_addr": "172.21.15.183:6825/3730013786", 2024-09-10T19:20:46.414 INFO:teuthology.orchestra.run.smithi183.stdout: "heartbeat_front_addr": "172.21.15.183:6823/3730013786", 2024-09-10T19:20:46.414 INFO:teuthology.orchestra.run.smithi183.stdout: "state": [ 2024-09-10T19:20:46.414 INFO:teuthology.orchestra.run.smithi183.stdout: "exists", 2024-09-10T19:20:46.415 INFO:teuthology.orchestra.run.smithi183.stdout: "up" 2024-09-10T19:20:46.415 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:20:46.415 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:20:46.415 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:20:46.415 INFO:teuthology.orchestra.run.smithi183.stdout: "osd": 3, 2024-09-10T19:20:46.415 INFO:teuthology.orchestra.run.smithi183.stdout: "uuid": "7fd07a95-ad22-43f5-a968-539552283e89", 2024-09-10T19:20:46.415 INFO:teuthology.orchestra.run.smithi183.stdout: "up": 1, 2024-09-10T19:20:46.415 INFO:teuthology.orchestra.run.smithi183.stdout: "in": 1, 2024-09-10T19:20:46.415 INFO:teuthology.orchestra.run.smithi183.stdout: "weight": 1, 2024-09-10T19:20:46.415 INFO:teuthology.orchestra.run.smithi183.stdout: "primary_affinity": 1, 2024-09-10T19:20:46.415 INFO:teuthology.orchestra.run.smithi183.stdout: "last_clean_begin": 0, 2024-09-10T19:20:46.415 INFO:teuthology.orchestra.run.smithi183.stdout: "last_clean_end": 0, 2024-09-10T19:20:46.416 INFO:teuthology.orchestra.run.smithi183.stdout: "up_from": 26, 2024-09-10T19:20:46.416 INFO:teuthology.orchestra.run.smithi183.stdout: "up_thru": 244, 2024-09-10T19:20:46.416 INFO:teuthology.orchestra.run.smithi183.stdout: "down_at": 0, 2024-09-10T19:20:46.416 INFO:teuthology.orchestra.run.smithi183.stdout: "lost_at": 0, 2024-09-10T19:20:46.416 INFO:teuthology.orchestra.run.smithi183.stdout: "public_addrs": { 2024-09-10T19:20:46.416 INFO:teuthology.orchestra.run.smithi183.stdout: "addrvec": [ 2024-09-10T19:20:46.416 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:20:46.416 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v2", 2024-09-10T19:20:46.416 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.183:6826", 2024-09-10T19:20:46.416 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 3522836984 2024-09-10T19:20:46.416 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:20:46.416 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:20:46.416 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v1", 2024-09-10T19:20:46.417 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.183:6827", 2024-09-10T19:20:46.417 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 3522836984 2024-09-10T19:20:46.417 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:20:46.417 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:20:46.417 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:20:46.417 INFO:teuthology.orchestra.run.smithi183.stdout: "cluster_addrs": { 2024-09-10T19:20:46.417 INFO:teuthology.orchestra.run.smithi183.stdout: "addrvec": [ 2024-09-10T19:20:46.417 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:20:46.417 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v2", 2024-09-10T19:20:46.417 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.183:6828", 2024-09-10T19:20:46.417 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 3522836984 2024-09-10T19:20:46.417 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:20:46.417 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:20:46.417 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v1", 2024-09-10T19:20:46.417 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.183:6829", 2024-09-10T19:20:46.418 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 3522836984 2024-09-10T19:20:46.418 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:20:46.418 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:20:46.418 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:20:46.418 INFO:teuthology.orchestra.run.smithi183.stdout: "heartbeat_back_addrs": { 2024-09-10T19:20:46.418 INFO:teuthology.orchestra.run.smithi183.stdout: "addrvec": [ 2024-09-10T19:20:46.418 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:20:46.418 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v2", 2024-09-10T19:20:46.418 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.183:6832", 2024-09-10T19:20:46.418 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 3522836984 2024-09-10T19:20:46.418 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:20:46.418 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:20:46.418 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v1", 2024-09-10T19:20:46.418 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.183:6833", 2024-09-10T19:20:46.418 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 3522836984 2024-09-10T19:20:46.419 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:20:46.419 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:20:46.419 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:20:46.419 INFO:teuthology.orchestra.run.smithi183.stdout: "heartbeat_front_addrs": { 2024-09-10T19:20:46.419 INFO:teuthology.orchestra.run.smithi183.stdout: "addrvec": [ 2024-09-10T19:20:46.419 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:20:46.419 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v2", 2024-09-10T19:20:46.419 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.183:6830", 2024-09-10T19:20:46.419 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 3522836984 2024-09-10T19:20:46.419 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:20:46.419 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:20:46.419 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v1", 2024-09-10T19:20:46.419 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.183:6831", 2024-09-10T19:20:46.420 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 3522836984 2024-09-10T19:20:46.420 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:20:46.420 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:20:46.420 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:20:46.420 INFO:teuthology.orchestra.run.smithi183.stdout: "public_addr": "172.21.15.183:6827/3522836984", 2024-09-10T19:20:46.420 INFO:teuthology.orchestra.run.smithi183.stdout: "cluster_addr": "172.21.15.183:6829/3522836984", 2024-09-10T19:20:46.420 INFO:teuthology.orchestra.run.smithi183.stdout: "heartbeat_back_addr": "172.21.15.183:6833/3522836984", 2024-09-10T19:20:46.420 INFO:teuthology.orchestra.run.smithi183.stdout: "heartbeat_front_addr": "172.21.15.183:6831/3522836984", 2024-09-10T19:20:46.420 INFO:teuthology.orchestra.run.smithi183.stdout: "state": [ 2024-09-10T19:20:46.420 INFO:teuthology.orchestra.run.smithi183.stdout: "exists", 2024-09-10T19:20:46.421 INFO:teuthology.orchestra.run.smithi183.stdout: "up" 2024-09-10T19:20:46.421 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:20:46.421 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:20:46.421 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:20:46.421 INFO:teuthology.orchestra.run.smithi183.stdout: "osd": 4, 2024-09-10T19:20:46.421 INFO:teuthology.orchestra.run.smithi183.stdout: "uuid": "631083fb-e503-44da-9b6b-154e88539ee7", 2024-09-10T19:20:46.421 INFO:teuthology.orchestra.run.smithi183.stdout: "up": 1, 2024-09-10T19:20:46.421 INFO:teuthology.orchestra.run.smithi183.stdout: "in": 1, 2024-09-10T19:20:46.421 INFO:teuthology.orchestra.run.smithi183.stdout: "weight": 1, 2024-09-10T19:20:46.421 INFO:teuthology.orchestra.run.smithi183.stdout: "primary_affinity": 1, 2024-09-10T19:20:46.421 INFO:teuthology.orchestra.run.smithi183.stdout: "last_clean_begin": 0, 2024-09-10T19:20:46.421 INFO:teuthology.orchestra.run.smithi183.stdout: "last_clean_end": 0, 2024-09-10T19:20:46.422 INFO:teuthology.orchestra.run.smithi183.stdout: "up_from": 32, 2024-09-10T19:20:46.422 INFO:teuthology.orchestra.run.smithi183.stdout: "up_thru": 244, 2024-09-10T19:20:46.422 INFO:teuthology.orchestra.run.smithi183.stdout: "down_at": 0, 2024-09-10T19:20:46.422 INFO:teuthology.orchestra.run.smithi183.stdout: "lost_at": 0, 2024-09-10T19:20:46.422 INFO:teuthology.orchestra.run.smithi183.stdout: "public_addrs": { 2024-09-10T19:20:46.422 INFO:teuthology.orchestra.run.smithi183.stdout: "addrvec": [ 2024-09-10T19:20:46.422 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:20:46.422 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v2", 2024-09-10T19:20:46.422 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.191:6800", 2024-09-10T19:20:46.422 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 2674828539 2024-09-10T19:20:46.422 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:20:46.423 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:20:46.423 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v1", 2024-09-10T19:20:46.423 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.191:6801", 2024-09-10T19:20:46.423 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 2674828539 2024-09-10T19:20:46.423 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:20:46.423 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:20:46.423 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:20:46.423 INFO:teuthology.orchestra.run.smithi183.stdout: "cluster_addrs": { 2024-09-10T19:20:46.423 INFO:teuthology.orchestra.run.smithi183.stdout: "addrvec": [ 2024-09-10T19:20:46.423 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:20:46.423 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v2", 2024-09-10T19:20:46.423 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.191:6802", 2024-09-10T19:20:46.423 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 2674828539 2024-09-10T19:20:46.424 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:20:46.424 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:20:46.424 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v1", 2024-09-10T19:20:46.424 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.191:6803", 2024-09-10T19:20:46.424 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 2674828539 2024-09-10T19:20:46.424 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:20:46.424 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:20:46.424 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:20:46.424 INFO:teuthology.orchestra.run.smithi183.stdout: "heartbeat_back_addrs": { 2024-09-10T19:20:46.424 INFO:teuthology.orchestra.run.smithi183.stdout: "addrvec": [ 2024-09-10T19:20:46.424 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:20:46.424 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v2", 2024-09-10T19:20:46.424 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.191:6806", 2024-09-10T19:20:46.425 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 2674828539 2024-09-10T19:20:46.425 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:20:46.425 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:20:46.425 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v1", 2024-09-10T19:20:46.425 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.191:6807", 2024-09-10T19:20:46.425 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 2674828539 2024-09-10T19:20:46.425 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:20:46.425 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:20:46.425 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:20:46.425 INFO:teuthology.orchestra.run.smithi183.stdout: "heartbeat_front_addrs": { 2024-09-10T19:20:46.425 INFO:teuthology.orchestra.run.smithi183.stdout: "addrvec": [ 2024-09-10T19:20:46.425 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:20:46.425 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v2", 2024-09-10T19:20:46.426 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.191:6804", 2024-09-10T19:20:46.426 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 2674828539 2024-09-10T19:20:46.426 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:20:46.426 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:20:46.426 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v1", 2024-09-10T19:20:46.426 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.191:6805", 2024-09-10T19:20:46.426 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 2674828539 2024-09-10T19:20:46.426 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:20:46.426 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:20:46.426 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:20:46.426 INFO:teuthology.orchestra.run.smithi183.stdout: "public_addr": "172.21.15.191:6801/2674828539", 2024-09-10T19:20:46.427 INFO:teuthology.orchestra.run.smithi183.stdout: "cluster_addr": "172.21.15.191:6803/2674828539", 2024-09-10T19:20:46.427 INFO:teuthology.orchestra.run.smithi183.stdout: "heartbeat_back_addr": "172.21.15.191:6807/2674828539", 2024-09-10T19:20:46.427 INFO:teuthology.orchestra.run.smithi183.stdout: "heartbeat_front_addr": "172.21.15.191:6805/2674828539", 2024-09-10T19:20:46.427 INFO:teuthology.orchestra.run.smithi183.stdout: "state": [ 2024-09-10T19:20:46.427 INFO:teuthology.orchestra.run.smithi183.stdout: "exists", 2024-09-10T19:20:46.427 INFO:teuthology.orchestra.run.smithi183.stdout: "up" 2024-09-10T19:20:46.427 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:20:46.427 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:20:46.427 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:20:46.427 INFO:teuthology.orchestra.run.smithi183.stdout: "osd": 5, 2024-09-10T19:20:46.427 INFO:teuthology.orchestra.run.smithi183.stdout: "uuid": "6b963695-3eca-4e6e-9207-05b8010ebff9", 2024-09-10T19:20:46.427 INFO:teuthology.orchestra.run.smithi183.stdout: "up": 1, 2024-09-10T19:20:46.427 INFO:teuthology.orchestra.run.smithi183.stdout: "in": 1, 2024-09-10T19:20:46.427 INFO:teuthology.orchestra.run.smithi183.stdout: "weight": 1, 2024-09-10T19:20:46.427 INFO:teuthology.orchestra.run.smithi183.stdout: "primary_affinity": 1, 2024-09-10T19:20:46.428 INFO:teuthology.orchestra.run.smithi183.stdout: "last_clean_begin": 0, 2024-09-10T19:20:46.428 INFO:teuthology.orchestra.run.smithi183.stdout: "last_clean_end": 0, 2024-09-10T19:20:46.428 INFO:teuthology.orchestra.run.smithi183.stdout: "up_from": 39, 2024-09-10T19:20:46.428 INFO:teuthology.orchestra.run.smithi183.stdout: "up_thru": 244, 2024-09-10T19:20:46.428 INFO:teuthology.orchestra.run.smithi183.stdout: "down_at": 0, 2024-09-10T19:20:46.428 INFO:teuthology.orchestra.run.smithi183.stdout: "lost_at": 0, 2024-09-10T19:20:46.428 INFO:teuthology.orchestra.run.smithi183.stdout: "public_addrs": { 2024-09-10T19:20:46.428 INFO:teuthology.orchestra.run.smithi183.stdout: "addrvec": [ 2024-09-10T19:20:46.428 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:20:46.428 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v2", 2024-09-10T19:20:46.428 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.191:6808", 2024-09-10T19:20:46.428 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 681733378 2024-09-10T19:20:46.428 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:20:46.428 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:20:46.429 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v1", 2024-09-10T19:20:46.429 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.191:6809", 2024-09-10T19:20:46.429 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 681733378 2024-09-10T19:20:46.429 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:20:46.429 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:20:46.429 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:20:46.429 INFO:teuthology.orchestra.run.smithi183.stdout: "cluster_addrs": { 2024-09-10T19:20:46.429 INFO:teuthology.orchestra.run.smithi183.stdout: "addrvec": [ 2024-09-10T19:20:46.429 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:20:46.429 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v2", 2024-09-10T19:20:46.429 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.191:6810", 2024-09-10T19:20:46.430 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 681733378 2024-09-10T19:20:46.430 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:20:46.430 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:20:46.430 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v1", 2024-09-10T19:20:46.430 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.191:6811", 2024-09-10T19:20:46.430 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 681733378 2024-09-10T19:20:46.430 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:20:46.430 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:20:46.430 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:20:46.430 INFO:teuthology.orchestra.run.smithi183.stdout: "heartbeat_back_addrs": { 2024-09-10T19:20:46.430 INFO:teuthology.orchestra.run.smithi183.stdout: "addrvec": [ 2024-09-10T19:20:46.430 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:20:46.430 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v2", 2024-09-10T19:20:46.430 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.191:6814", 2024-09-10T19:20:46.430 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 681733378 2024-09-10T19:20:46.431 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:20:46.431 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:20:46.431 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v1", 2024-09-10T19:20:46.431 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.191:6815", 2024-09-10T19:20:46.431 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 681733378 2024-09-10T19:20:46.431 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:20:46.431 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:20:46.431 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:20:46.431 INFO:teuthology.orchestra.run.smithi183.stdout: "heartbeat_front_addrs": { 2024-09-10T19:20:46.431 INFO:teuthology.orchestra.run.smithi183.stdout: "addrvec": [ 2024-09-10T19:20:46.431 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:20:46.431 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v2", 2024-09-10T19:20:46.431 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.191:6812", 2024-09-10T19:20:46.432 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 681733378 2024-09-10T19:20:46.432 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:20:46.432 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:20:46.432 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v1", 2024-09-10T19:20:46.432 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.191:6813", 2024-09-10T19:20:46.432 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 681733378 2024-09-10T19:20:46.432 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:20:46.432 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:20:46.432 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:20:46.432 INFO:teuthology.orchestra.run.smithi183.stdout: "public_addr": "172.21.15.191:6809/681733378", 2024-09-10T19:20:46.432 INFO:teuthology.orchestra.run.smithi183.stdout: "cluster_addr": "172.21.15.191:6811/681733378", 2024-09-10T19:20:46.432 INFO:teuthology.orchestra.run.smithi183.stdout: "heartbeat_back_addr": "172.21.15.191:6815/681733378", 2024-09-10T19:20:46.432 INFO:teuthology.orchestra.run.smithi183.stdout: "heartbeat_front_addr": "172.21.15.191:6813/681733378", 2024-09-10T19:20:46.433 INFO:teuthology.orchestra.run.smithi183.stdout: "state": [ 2024-09-10T19:20:46.433 INFO:teuthology.orchestra.run.smithi183.stdout: "exists", 2024-09-10T19:20:46.433 INFO:teuthology.orchestra.run.smithi183.stdout: "up" 2024-09-10T19:20:46.433 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:20:46.433 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:20:46.433 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:20:46.433 INFO:teuthology.orchestra.run.smithi183.stdout: "osd": 6, 2024-09-10T19:20:46.433 INFO:teuthology.orchestra.run.smithi183.stdout: "uuid": "2dc9754f-6aef-4e19-8605-2f1591921811", 2024-09-10T19:20:46.433 INFO:teuthology.orchestra.run.smithi183.stdout: "up": 1, 2024-09-10T19:20:46.433 INFO:teuthology.orchestra.run.smithi183.stdout: "in": 1, 2024-09-10T19:20:46.433 INFO:teuthology.orchestra.run.smithi183.stdout: "weight": 1, 2024-09-10T19:20:46.433 INFO:teuthology.orchestra.run.smithi183.stdout: "primary_affinity": 1, 2024-09-10T19:20:46.433 INFO:teuthology.orchestra.run.smithi183.stdout: "last_clean_begin": 0, 2024-09-10T19:20:46.433 INFO:teuthology.orchestra.run.smithi183.stdout: "last_clean_end": 0, 2024-09-10T19:20:46.434 INFO:teuthology.orchestra.run.smithi183.stdout: "up_from": 47, 2024-09-10T19:20:46.434 INFO:teuthology.orchestra.run.smithi183.stdout: "up_thru": 244, 2024-09-10T19:20:46.434 INFO:teuthology.orchestra.run.smithi183.stdout: "down_at": 0, 2024-09-10T19:20:46.434 INFO:teuthology.orchestra.run.smithi183.stdout: "lost_at": 0, 2024-09-10T19:20:46.434 INFO:teuthology.orchestra.run.smithi183.stdout: "public_addrs": { 2024-09-10T19:20:46.434 INFO:teuthology.orchestra.run.smithi183.stdout: "addrvec": [ 2024-09-10T19:20:46.434 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:20:46.434 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v2", 2024-09-10T19:20:46.434 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.191:6816", 2024-09-10T19:20:46.434 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 2825813477 2024-09-10T19:20:46.434 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:20:46.434 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:20:46.434 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v1", 2024-09-10T19:20:46.434 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.191:6817", 2024-09-10T19:20:46.435 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 2825813477 2024-09-10T19:20:46.435 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:20:46.435 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:20:46.435 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:20:46.435 INFO:teuthology.orchestra.run.smithi183.stdout: "cluster_addrs": { 2024-09-10T19:20:46.435 INFO:teuthology.orchestra.run.smithi183.stdout: "addrvec": [ 2024-09-10T19:20:46.435 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:20:46.435 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v2", 2024-09-10T19:20:46.435 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.191:6818", 2024-09-10T19:20:46.435 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 2825813477 2024-09-10T19:20:46.435 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:20:46.435 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:20:46.435 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v1", 2024-09-10T19:20:46.435 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.191:6819", 2024-09-10T19:20:46.435 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 2825813477 2024-09-10T19:20:46.436 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:20:46.436 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:20:46.436 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:20:46.436 INFO:teuthology.orchestra.run.smithi183.stdout: "heartbeat_back_addrs": { 2024-09-10T19:20:46.436 INFO:teuthology.orchestra.run.smithi183.stdout: "addrvec": [ 2024-09-10T19:20:46.436 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:20:46.436 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v2", 2024-09-10T19:20:46.436 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.191:6822", 2024-09-10T19:20:46.436 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 2825813477 2024-09-10T19:20:46.436 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:20:46.436 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:20:46.436 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v1", 2024-09-10T19:20:46.436 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.191:6823", 2024-09-10T19:20:46.436 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 2825813477 2024-09-10T19:20:46.436 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:20:46.437 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:20:46.437 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:20:46.437 INFO:teuthology.orchestra.run.smithi183.stdout: "heartbeat_front_addrs": { 2024-09-10T19:20:46.437 INFO:teuthology.orchestra.run.smithi183.stdout: "addrvec": [ 2024-09-10T19:20:46.437 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:20:46.437 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v2", 2024-09-10T19:20:46.437 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.191:6820", 2024-09-10T19:20:46.437 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 2825813477 2024-09-10T19:20:46.437 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:20:46.437 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:20:46.437 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v1", 2024-09-10T19:20:46.437 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.191:6821", 2024-09-10T19:20:46.437 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 2825813477 2024-09-10T19:20:46.438 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:20:46.438 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:20:46.438 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:20:46.438 INFO:teuthology.orchestra.run.smithi183.stdout: "public_addr": "172.21.15.191:6817/2825813477", 2024-09-10T19:20:46.438 INFO:teuthology.orchestra.run.smithi183.stdout: "cluster_addr": "172.21.15.191:6819/2825813477", 2024-09-10T19:20:46.438 INFO:teuthology.orchestra.run.smithi183.stdout: "heartbeat_back_addr": "172.21.15.191:6823/2825813477", 2024-09-10T19:20:46.438 INFO:teuthology.orchestra.run.smithi183.stdout: "heartbeat_front_addr": "172.21.15.191:6821/2825813477", 2024-09-10T19:20:46.438 INFO:teuthology.orchestra.run.smithi183.stdout: "state": [ 2024-09-10T19:20:46.438 INFO:teuthology.orchestra.run.smithi183.stdout: "exists", 2024-09-10T19:20:46.438 INFO:teuthology.orchestra.run.smithi183.stdout: "up" 2024-09-10T19:20:46.438 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:20:46.438 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:20:46.439 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:20:46.439 INFO:teuthology.orchestra.run.smithi183.stdout: "osd": 7, 2024-09-10T19:20:46.439 INFO:teuthology.orchestra.run.smithi183.stdout: "uuid": "1a9a299e-d5ba-4b9a-85d5-7f6a5bc754e1", 2024-09-10T19:20:46.439 INFO:teuthology.orchestra.run.smithi183.stdout: "up": 1, 2024-09-10T19:20:46.439 INFO:teuthology.orchestra.run.smithi183.stdout: "in": 1, 2024-09-10T19:20:46.439 INFO:teuthology.orchestra.run.smithi183.stdout: "weight": 1, 2024-09-10T19:20:46.439 INFO:teuthology.orchestra.run.smithi183.stdout: "primary_affinity": 1, 2024-09-10T19:20:46.439 INFO:teuthology.orchestra.run.smithi183.stdout: "last_clean_begin": 0, 2024-09-10T19:20:46.439 INFO:teuthology.orchestra.run.smithi183.stdout: "last_clean_end": 0, 2024-09-10T19:20:46.439 INFO:teuthology.orchestra.run.smithi183.stdout: "up_from": 54, 2024-09-10T19:20:46.439 INFO:teuthology.orchestra.run.smithi183.stdout: "up_thru": 244, 2024-09-10T19:20:46.439 INFO:teuthology.orchestra.run.smithi183.stdout: "down_at": 0, 2024-09-10T19:20:46.439 INFO:teuthology.orchestra.run.smithi183.stdout: "lost_at": 0, 2024-09-10T19:20:46.439 INFO:teuthology.orchestra.run.smithi183.stdout: "public_addrs": { 2024-09-10T19:20:46.440 INFO:teuthology.orchestra.run.smithi183.stdout: "addrvec": [ 2024-09-10T19:20:46.440 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:20:46.440 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v2", 2024-09-10T19:20:46.440 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.191:6824", 2024-09-10T19:20:46.440 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 1422610021 2024-09-10T19:20:46.440 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:20:46.440 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:20:46.440 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v1", 2024-09-10T19:20:46.440 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.191:6825", 2024-09-10T19:20:46.440 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 1422610021 2024-09-10T19:20:46.440 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:20:46.440 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:20:46.440 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:20:46.441 INFO:teuthology.orchestra.run.smithi183.stdout: "cluster_addrs": { 2024-09-10T19:20:46.441 INFO:teuthology.orchestra.run.smithi183.stdout: "addrvec": [ 2024-09-10T19:20:46.441 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:20:46.441 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v2", 2024-09-10T19:20:46.441 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.191:6826", 2024-09-10T19:20:46.441 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 1422610021 2024-09-10T19:20:46.441 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:20:46.441 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:20:46.441 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v1", 2024-09-10T19:20:46.441 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.191:6827", 2024-09-10T19:20:46.441 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 1422610021 2024-09-10T19:20:46.441 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:20:46.441 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:20:46.441 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:20:46.442 INFO:teuthology.orchestra.run.smithi183.stdout: "heartbeat_back_addrs": { 2024-09-10T19:20:46.442 INFO:teuthology.orchestra.run.smithi183.stdout: "addrvec": [ 2024-09-10T19:20:46.442 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:20:46.442 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v2", 2024-09-10T19:20:46.442 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.191:6830", 2024-09-10T19:20:46.442 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 1422610021 2024-09-10T19:20:46.442 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:20:46.442 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:20:46.442 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v1", 2024-09-10T19:20:46.442 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.191:6831", 2024-09-10T19:20:46.442 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 1422610021 2024-09-10T19:20:46.442 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:20:46.443 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:20:46.443 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:20:46.443 INFO:teuthology.orchestra.run.smithi183.stdout: "heartbeat_front_addrs": { 2024-09-10T19:20:46.443 INFO:teuthology.orchestra.run.smithi183.stdout: "addrvec": [ 2024-09-10T19:20:46.443 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:20:46.443 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v2", 2024-09-10T19:20:46.443 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.191:6828", 2024-09-10T19:20:46.443 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 1422610021 2024-09-10T19:20:46.443 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:20:46.443 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:20:46.443 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v1", 2024-09-10T19:20:46.443 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.191:6829", 2024-09-10T19:20:46.443 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 1422610021 2024-09-10T19:20:46.443 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:20:46.443 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:20:46.444 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:20:46.444 INFO:teuthology.orchestra.run.smithi183.stdout: "public_addr": "172.21.15.191:6825/1422610021", 2024-09-10T19:20:46.444 INFO:teuthology.orchestra.run.smithi183.stdout: "cluster_addr": "172.21.15.191:6827/1422610021", 2024-09-10T19:20:46.444 INFO:teuthology.orchestra.run.smithi183.stdout: "heartbeat_back_addr": "172.21.15.191:6831/1422610021", 2024-09-10T19:20:46.444 INFO:teuthology.orchestra.run.smithi183.stdout: "heartbeat_front_addr": "172.21.15.191:6829/1422610021", 2024-09-10T19:20:46.444 INFO:teuthology.orchestra.run.smithi183.stdout: "state": [ 2024-09-10T19:20:46.444 INFO:teuthology.orchestra.run.smithi183.stdout: "exists", 2024-09-10T19:20:46.444 INFO:teuthology.orchestra.run.smithi183.stdout: "up" 2024-09-10T19:20:46.444 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:20:46.444 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:20:46.444 INFO:teuthology.orchestra.run.smithi183.stdout: ], 2024-09-10T19:20:46.444 INFO:teuthology.orchestra.run.smithi183.stdout: "osd_xinfo": [ 2024-09-10T19:20:46.444 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:20:46.444 INFO:teuthology.orchestra.run.smithi183.stdout: "osd": 0, 2024-09-10T19:20:46.444 INFO:teuthology.orchestra.run.smithi183.stdout: "down_stamp": "0.000000", 2024-09-10T19:20:46.445 INFO:teuthology.orchestra.run.smithi183.stdout: "laggy_probability": 0, 2024-09-10T19:20:46.445 INFO:teuthology.orchestra.run.smithi183.stdout: "laggy_interval": 0, 2024-09-10T19:20:46.445 INFO:teuthology.orchestra.run.smithi183.stdout: "features": 4540138320759226367, 2024-09-10T19:20:46.445 INFO:teuthology.orchestra.run.smithi183.stdout: "old_weight": 0, 2024-09-10T19:20:46.445 INFO:teuthology.orchestra.run.smithi183.stdout: "last_purged_snaps_scrub": "2024-09-10T19:20:25.561104+0000", 2024-09-10T19:20:46.445 INFO:teuthology.orchestra.run.smithi183.stdout: "dead_epoch": 0 2024-09-10T19:20:46.445 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:20:46.445 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:20:46.445 INFO:teuthology.orchestra.run.smithi183.stdout: "osd": 1, 2024-09-10T19:20:46.445 INFO:teuthology.orchestra.run.smithi183.stdout: "down_stamp": "0.000000", 2024-09-10T19:20:46.445 INFO:teuthology.orchestra.run.smithi183.stdout: "laggy_probability": 0, 2024-09-10T19:20:46.445 INFO:teuthology.orchestra.run.smithi183.stdout: "laggy_interval": 0, 2024-09-10T19:20:46.445 INFO:teuthology.orchestra.run.smithi183.stdout: "features": 4540138320759226367, 2024-09-10T19:20:46.446 INFO:teuthology.orchestra.run.smithi183.stdout: "old_weight": 65536, 2024-09-10T19:20:46.446 INFO:teuthology.orchestra.run.smithi183.stdout: "last_purged_snaps_scrub": "2024-09-10T19:20:37.497093+0000", 2024-09-10T19:20:46.446 INFO:teuthology.orchestra.run.smithi183.stdout: "dead_epoch": 98 2024-09-10T19:20:46.446 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:20:46.446 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:20:46.446 INFO:teuthology.orchestra.run.smithi183.stdout: "osd": 2, 2024-09-10T19:20:46.446 INFO:teuthology.orchestra.run.smithi183.stdout: "down_stamp": "0.000000", 2024-09-10T19:20:46.446 INFO:teuthology.orchestra.run.smithi183.stdout: "laggy_probability": 0, 2024-09-10T19:20:46.446 INFO:teuthology.orchestra.run.smithi183.stdout: "laggy_interval": 0, 2024-09-10T19:20:46.446 INFO:teuthology.orchestra.run.smithi183.stdout: "features": 4540138320759226367, 2024-09-10T19:20:46.446 INFO:teuthology.orchestra.run.smithi183.stdout: "old_weight": 65536, 2024-09-10T19:20:46.446 INFO:teuthology.orchestra.run.smithi183.stdout: "last_purged_snaps_scrub": "2024-09-10T19:19:42.374103+0000", 2024-09-10T19:20:46.446 INFO:teuthology.orchestra.run.smithi183.stdout: "dead_epoch": 0 2024-09-10T19:20:46.446 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:20:46.447 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:20:46.447 INFO:teuthology.orchestra.run.smithi183.stdout: "osd": 3, 2024-09-10T19:20:46.447 INFO:teuthology.orchestra.run.smithi183.stdout: "down_stamp": "0.000000", 2024-09-10T19:20:46.447 INFO:teuthology.orchestra.run.smithi183.stdout: "laggy_probability": 0, 2024-09-10T19:20:46.447 INFO:teuthology.orchestra.run.smithi183.stdout: "laggy_interval": 0, 2024-09-10T19:20:46.447 INFO:teuthology.orchestra.run.smithi183.stdout: "features": 4540138320759226367, 2024-09-10T19:20:46.447 INFO:teuthology.orchestra.run.smithi183.stdout: "old_weight": 0, 2024-09-10T19:20:46.447 INFO:teuthology.orchestra.run.smithi183.stdout: "last_purged_snaps_scrub": "2024-09-10T19:20:02.376672+0000", 2024-09-10T19:20:46.447 INFO:teuthology.orchestra.run.smithi183.stdout: "dead_epoch": 0 2024-09-10T19:20:46.447 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:20:46.447 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:20:46.447 INFO:teuthology.orchestra.run.smithi183.stdout: "osd": 4, 2024-09-10T19:20:46.447 INFO:teuthology.orchestra.run.smithi183.stdout: "down_stamp": "0.000000", 2024-09-10T19:20:46.447 INFO:teuthology.orchestra.run.smithi183.stdout: "laggy_probability": 0, 2024-09-10T19:20:46.447 INFO:teuthology.orchestra.run.smithi183.stdout: "laggy_interval": 0, 2024-09-10T19:20:46.447 INFO:teuthology.orchestra.run.smithi183.stdout: "features": 4540138320759226367, 2024-09-10T19:20:46.448 INFO:teuthology.orchestra.run.smithi183.stdout: "old_weight": 0, 2024-09-10T19:20:46.448 INFO:teuthology.orchestra.run.smithi183.stdout: "last_purged_snaps_scrub": "2024-09-10T19:20:08.110504+0000", 2024-09-10T19:20:46.448 INFO:teuthology.orchestra.run.smithi183.stdout: "dead_epoch": 0 2024-09-10T19:20:46.448 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:20:46.448 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:20:46.448 INFO:teuthology.orchestra.run.smithi183.stdout: "osd": 5, 2024-09-10T19:20:46.448 INFO:teuthology.orchestra.run.smithi183.stdout: "down_stamp": "0.000000", 2024-09-10T19:20:46.448 INFO:teuthology.orchestra.run.smithi183.stdout: "laggy_probability": 0, 2024-09-10T19:20:46.448 INFO:teuthology.orchestra.run.smithi183.stdout: "laggy_interval": 0, 2024-09-10T19:20:46.448 INFO:teuthology.orchestra.run.smithi183.stdout: "features": 4540138320759226367, 2024-09-10T19:20:46.448 INFO:teuthology.orchestra.run.smithi183.stdout: "old_weight": 0, 2024-09-10T19:20:46.448 INFO:teuthology.orchestra.run.smithi183.stdout: "last_purged_snaps_scrub": "2024-09-10T19:19:59.468537+0000", 2024-09-10T19:20:46.448 INFO:teuthology.orchestra.run.smithi183.stdout: "dead_epoch": 0 2024-09-10T19:20:46.448 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:20:46.449 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:20:46.449 INFO:teuthology.orchestra.run.smithi183.stdout: "osd": 6, 2024-09-10T19:20:46.449 INFO:teuthology.orchestra.run.smithi183.stdout: "down_stamp": "0.000000", 2024-09-10T19:20:46.449 INFO:teuthology.orchestra.run.smithi183.stdout: "laggy_probability": 0, 2024-09-10T19:20:46.449 INFO:teuthology.orchestra.run.smithi183.stdout: "laggy_interval": 0, 2024-09-10T19:20:46.449 INFO:teuthology.orchestra.run.smithi183.stdout: "features": 4540138320759226367, 2024-09-10T19:20:46.449 INFO:teuthology.orchestra.run.smithi183.stdout: "old_weight": 0, 2024-09-10T19:20:46.449 INFO:teuthology.orchestra.run.smithi183.stdout: "last_purged_snaps_scrub": "2024-09-10T19:19:47.856262+0000", 2024-09-10T19:20:46.449 INFO:teuthology.orchestra.run.smithi183.stdout: "dead_epoch": 0 2024-09-10T19:20:46.449 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:20:46.449 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:20:46.449 INFO:teuthology.orchestra.run.smithi183.stdout: "osd": 7, 2024-09-10T19:20:46.449 INFO:teuthology.orchestra.run.smithi183.stdout: "down_stamp": "0.000000", 2024-09-10T19:20:46.449 INFO:teuthology.orchestra.run.smithi183.stdout: "laggy_probability": 0, 2024-09-10T19:20:46.450 INFO:teuthology.orchestra.run.smithi183.stdout: "laggy_interval": 0, 2024-09-10T19:20:46.450 INFO:teuthology.orchestra.run.smithi183.stdout: "features": 4540138320759226367, 2024-09-10T19:20:46.450 INFO:teuthology.orchestra.run.smithi183.stdout: "old_weight": 0, 2024-09-10T19:20:46.450 INFO:teuthology.orchestra.run.smithi183.stdout: "last_purged_snaps_scrub": "2024-09-10T19:20:23.785620+0000", 2024-09-10T19:20:46.450 INFO:teuthology.orchestra.run.smithi183.stdout: "dead_epoch": 0 2024-09-10T19:20:46.450 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:20:46.450 INFO:teuthology.orchestra.run.smithi183.stdout: ], 2024-09-10T19:20:46.450 INFO:teuthology.orchestra.run.smithi183.stdout: "pg_upmap": [ 2024-09-10T19:20:46.450 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:20:46.450 INFO:teuthology.orchestra.run.smithi183.stdout: "pgid": "2.13", 2024-09-10T19:20:46.450 INFO:teuthology.orchestra.run.smithi183.stdout: "osds": [ 2024-09-10T19:20:46.450 INFO:teuthology.orchestra.run.smithi183.stdout: 2, 2024-09-10T19:20:46.450 INFO:teuthology.orchestra.run.smithi183.stdout: 0, 2024-09-10T19:20:46.450 INFO:teuthology.orchestra.run.smithi183.stdout: 5 2024-09-10T19:20:46.450 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:20:46.450 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:20:46.451 INFO:teuthology.orchestra.run.smithi183.stdout: ], 2024-09-10T19:20:46.451 INFO:teuthology.orchestra.run.smithi183.stdout: "pg_upmap_items": [ 2024-09-10T19:20:46.451 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:20:46.451 INFO:teuthology.orchestra.run.smithi183.stdout: "pgid": "2.18", 2024-09-10T19:20:46.451 INFO:teuthology.orchestra.run.smithi183.stdout: "mappings": [ 2024-09-10T19:20:46.451 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:20:46.451 INFO:teuthology.orchestra.run.smithi183.stdout: "from": 7, 2024-09-10T19:20:46.451 INFO:teuthology.orchestra.run.smithi183.stdout: "to": 0 2024-09-10T19:20:46.451 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:20:46.451 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:20:46.451 INFO:teuthology.orchestra.run.smithi183.stdout: "from": 5, 2024-09-10T19:20:46.451 INFO:teuthology.orchestra.run.smithi183.stdout: "to": 4 2024-09-10T19:20:46.451 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:20:46.451 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:20:46.451 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:20:46.452 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:20:46.452 INFO:teuthology.orchestra.run.smithi183.stdout: "pgid": "4.1", 2024-09-10T19:20:46.452 INFO:teuthology.orchestra.run.smithi183.stdout: "mappings": [ 2024-09-10T19:20:46.452 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:20:46.452 INFO:teuthology.orchestra.run.smithi183.stdout: "from": 6, 2024-09-10T19:20:46.452 INFO:teuthology.orchestra.run.smithi183.stdout: "to": 7 2024-09-10T19:20:46.452 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:20:46.452 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:20:46.452 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:20:46.452 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:20:46.452 INFO:teuthology.orchestra.run.smithi183.stdout: "pgid": "4.11", 2024-09-10T19:20:46.452 INFO:teuthology.orchestra.run.smithi183.stdout: "mappings": [ 2024-09-10T19:20:46.452 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:20:46.452 INFO:teuthology.orchestra.run.smithi183.stdout: "from": 3, 2024-09-10T19:20:46.453 INFO:teuthology.orchestra.run.smithi183.stdout: "to": 5 2024-09-10T19:20:46.453 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:20:46.453 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:20:46.453 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:20:46.453 INFO:teuthology.orchestra.run.smithi183.stdout: ], 2024-09-10T19:20:46.453 INFO:teuthology.orchestra.run.smithi183.stdout: "pg_temp": [], 2024-09-10T19:20:46.453 INFO:teuthology.orchestra.run.smithi183.stdout: "primary_temp": [], 2024-09-10T19:20:46.453 INFO:teuthology.orchestra.run.smithi183.stdout: "blocklist": { 2024-09-10T19:20:46.453 INFO:teuthology.orchestra.run.smithi183.stdout: "172.21.15.183:6801/3311783362": "2024-09-11T19:10:43.576943+0000", 2024-09-10T19:20:46.453 INFO:teuthology.orchestra.run.smithi183.stdout: "172.21.15.183:0/4088553922": "2024-09-11T19:10:43.576943+0000", 2024-09-10T19:20:46.453 INFO:teuthology.orchestra.run.smithi183.stdout: "172.21.15.183:0/3528769414": "2024-09-11T19:05:48.153879+0000", 2024-09-10T19:20:46.453 INFO:teuthology.orchestra.run.smithi183.stdout: "172.21.15.183:0/1766036658": "2024-09-11T19:05:48.153879+0000", 2024-09-10T19:20:46.453 INFO:teuthology.orchestra.run.smithi183.stdout: "172.21.15.183:0/3044170088": "2024-09-11T19:05:25.702354+0000", 2024-09-10T19:20:46.453 INFO:teuthology.orchestra.run.smithi183.stdout: "172.21.15.183:0/3037333778": "2024-09-11T19:05:25.702354+0000", 2024-09-10T19:20:46.454 INFO:teuthology.orchestra.run.smithi183.stdout: "172.21.15.183:6801/3747320570": "2024-09-11T19:05:25.702354+0000", 2024-09-10T19:20:46.454 INFO:teuthology.orchestra.run.smithi183.stdout: "172.21.15.183:0/3982386906": "2024-09-11T19:10:43.576943+0000", 2024-09-10T19:20:46.454 INFO:teuthology.orchestra.run.smithi183.stdout: "172.21.15.183:0/2436014455": "2024-09-11T19:05:25.702354+0000", 2024-09-10T19:20:46.454 INFO:teuthology.orchestra.run.smithi183.stdout: "172.21.15.183:6801/659745765": "2024-09-11T19:05:48.153879+0000", 2024-09-10T19:20:46.454 INFO:teuthology.orchestra.run.smithi183.stdout: "172.21.15.183:6800/3747320570": "2024-09-11T19:05:25.702354+0000", 2024-09-10T19:20:46.454 INFO:teuthology.orchestra.run.smithi183.stdout: "172.21.15.183:0/1652576282": "2024-09-11T19:10:43.576943+0000", 2024-09-10T19:20:46.454 INFO:teuthology.orchestra.run.smithi183.stdout: "172.21.15.183:6800/3311783362": "2024-09-11T19:10:43.576943+0000", 2024-09-10T19:20:46.454 INFO:teuthology.orchestra.run.smithi183.stdout: "172.21.15.183:0/1012899549": "2024-09-11T19:05:48.153879+0000", 2024-09-10T19:20:46.454 INFO:teuthology.orchestra.run.smithi183.stdout: "172.21.15.183:6800/659745765": "2024-09-11T19:05:48.153879+0000", 2024-09-10T19:20:46.454 INFO:teuthology.orchestra.run.smithi183.stdout: "172.21.15.183:0/412044400": "2024-09-11T19:10:43.576943+0000", 2024-09-10T19:20:46.454 INFO:teuthology.orchestra.run.smithi183.stdout: "172.21.15.183:0/3636171401": "2024-09-11T19:10:43.576943+0000" 2024-09-10T19:20:46.454 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:20:46.454 INFO:teuthology.orchestra.run.smithi183.stdout: "range_blocklist": {}, 2024-09-10T19:20:46.454 INFO:teuthology.orchestra.run.smithi183.stdout: "erasure_code_profiles": { 2024-09-10T19:20:46.455 INFO:teuthology.orchestra.run.smithi183.stdout: "default": { 2024-09-10T19:20:46.455 INFO:teuthology.orchestra.run.smithi183.stdout: "crush-failure-domain": "osd", 2024-09-10T19:20:46.455 INFO:teuthology.orchestra.run.smithi183.stdout: "k": "2", 2024-09-10T19:20:46.455 INFO:teuthology.orchestra.run.smithi183.stdout: "m": "1", 2024-09-10T19:20:46.455 INFO:teuthology.orchestra.run.smithi183.stdout: "plugin": "jerasure", 2024-09-10T19:20:46.455 INFO:teuthology.orchestra.run.smithi183.stdout: "technique": "reed_sol_van" 2024-09-10T19:20:46.455 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:20:46.455 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:20:46.455 INFO:teuthology.orchestra.run.smithi183.stdout: "removed_snaps_queue": [], 2024-09-10T19:20:46.455 INFO:teuthology.orchestra.run.smithi183.stdout: "new_removed_snaps": [], 2024-09-10T19:20:46.455 INFO:teuthology.orchestra.run.smithi183.stdout: "new_purged_snaps": [], 2024-09-10T19:20:46.455 INFO:teuthology.orchestra.run.smithi183.stdout: "crush_node_flags": {}, 2024-09-10T19:20:46.455 INFO:teuthology.orchestra.run.smithi183.stdout: "device_class_flags": {}, 2024-09-10T19:20:46.455 INFO:teuthology.orchestra.run.smithi183.stdout: "stretch_mode": { 2024-09-10T19:20:46.456 INFO:teuthology.orchestra.run.smithi183.stdout: "stretch_mode_enabled": false, 2024-09-10T19:20:46.456 INFO:teuthology.orchestra.run.smithi183.stdout: "stretch_bucket_count": 0, 2024-09-10T19:20:46.456 INFO:teuthology.orchestra.run.smithi183.stdout: "degraded_stretch_mode": 0, 2024-09-10T19:20:46.456 INFO:teuthology.orchestra.run.smithi183.stdout: "recovering_stretch_mode": 0, 2024-09-10T19:20:46.456 INFO:teuthology.orchestra.run.smithi183.stdout: "stretch_mode_bucket": 0 2024-09-10T19:20:46.456 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:20:46.456 INFO:teuthology.orchestra.run.smithi183.stdout:} 2024-09-10T19:20:46.465 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:20:46 smithi191 ceph-mon[32884]: 2.2 scrub starts 2024-09-10T19:20:46.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:20:46 smithi191 ceph-mon[32884]: 2.2 scrub ok 2024-09-10T19:20:46.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:20:46 smithi191 ceph-mon[32884]: 4.1e scrub starts 2024-09-10T19:20:46.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:20:46 smithi191 ceph-mon[32884]: 4.1e scrub ok 2024-09-10T19:20:46.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:20:46 smithi191 ceph-mon[32884]: 6.2 scrub starts 2024-09-10T19:20:46.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:20:46 smithi191 ceph-mon[32884]: 6.2 scrub ok 2024-09-10T19:20:46.468 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:20:46.507 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:20:46.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:20:46.509 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:20:46.609 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:20:46.637 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:20:46.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:20:46.639 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:20:46.735 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: 45 16 83880 83864 116.463 90.9375 0.0110496 0.00858204 2024-09-10T19:20:46.739 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:20:46.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:20:46.769 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:20:46.770 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:20:46.870 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:20:46.904 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:20:46.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:20:46.905 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:20:47.006 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:20:47.036 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:20:47.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:20:47.037 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:20:47.138 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:20:47.175 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:20:47.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:20:47.177 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:20:47.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:20:47 smithi183 ceph-mon[29736]: 5.12 scrub starts 2024-09-10T19:20:47.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:20:47 smithi183 ceph-mon[29736]: 5.12 scrub ok 2024-09-10T19:20:47.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:20:47 smithi183 ceph-mon[29736]: 6.f scrub starts 2024-09-10T19:20:47.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:20:47 smithi183 ceph-mon[29736]: 6.f scrub ok 2024-09-10T19:20:47.182 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:20:47 smithi183 ceph-mon[29736]: 2.1d scrub starts 2024-09-10T19:20:47.182 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:20:47 smithi183 ceph-mon[29736]: 2.1d scrub ok 2024-09-10T19:20:47.182 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:20:47 smithi183 ceph-mon[29736]: pgmap v479: 158 pgs: 158 active+clean; 5.0 GiB data, 18 GiB used, 518 GiB / 536 GiB avail; 1.2 KiB/s rd, 110 MiB/s wr, 1.75k op/s 2024-09-10T19:20:47.182 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:20:47 smithi183 ceph-mon[29736]: from='client.? 172.21.15.183:0/2958501888' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-09-10T19:20:47.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:20:47 smithi183 ceph-mon[41562]: 5.12 scrub starts 2024-09-10T19:20:47.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:20:47 smithi183 ceph-mon[41562]: 5.12 scrub ok 2024-09-10T19:20:47.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:20:47 smithi183 ceph-mon[41562]: 6.f scrub starts 2024-09-10T19:20:47.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:20:47 smithi183 ceph-mon[41562]: 6.f scrub ok 2024-09-10T19:20:47.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:20:47 smithi183 ceph-mon[41562]: 2.1d scrub starts 2024-09-10T19:20:47.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:20:47 smithi183 ceph-mon[41562]: 2.1d scrub ok 2024-09-10T19:20:47.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:20:47 smithi183 ceph-mon[41562]: pgmap v479: 158 pgs: 158 active+clean; 5.0 GiB data, 18 GiB used, 518 GiB / 536 GiB avail; 1.2 KiB/s rd, 110 MiB/s wr, 1.75k op/s 2024-09-10T19:20:47.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:20:47 smithi183 ceph-mon[41562]: from='client.? 172.21.15.183:0/2958501888' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-09-10T19:20:47.259 INFO:teuthology.orchestra.run.smithi183.stdout:osd.0: {} 2024-09-10T19:20:47.260 INFO:teuthology.orchestra.run.smithi183.stdout:osd.1: {} 2024-09-10T19:20:47.260 INFO:teuthology.orchestra.run.smithi183.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-10T19:20:47.260 INFO:teuthology.orchestra.run.smithi183.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-10T19:20:47.260 INFO:teuthology.orchestra.run.smithi183.stdout:osd.2: {} 2024-09-10T19:20:47.261 INFO:teuthology.orchestra.run.smithi183.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-10T19:20:47.277 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:20:47.288 INFO:teuthology.orchestra.run.smithi183.stdout:osd.3: {} 2024-09-10T19:20:47.288 INFO:teuthology.orchestra.run.smithi183.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-10T19:20:47.312 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:20:47.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:20:47.313 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:20:47.318 INFO:teuthology.orchestra.run.smithi183.stdout:osd.4: {} 2024-09-10T19:20:47.319 INFO:teuthology.orchestra.run.smithi183.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-10T19:20:47.352 INFO:teuthology.orchestra.run.smithi183.stdout:osd.5: {} 2024-09-10T19:20:47.352 INFO:teuthology.orchestra.run.smithi183.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-10T19:20:47.379 INFO:teuthology.orchestra.run.smithi183.stdout:osd.6: {} 2024-09-10T19:20:47.380 INFO:teuthology.orchestra.run.smithi183.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-10T19:20:47.408 INFO:teuthology.orchestra.run.smithi183.stdout:osd.7: {} 2024-09-10T19:20:47.409 INFO:teuthology.orchestra.run.smithi183.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-10T19:20:47.414 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:20:47.450 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:20:47.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:20:47.451 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:20:47.451 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:20:47 smithi191 ceph-mon[32884]: 5.12 scrub starts 2024-09-10T19:20:47.451 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:20:47 smithi191 ceph-mon[32884]: 5.12 scrub ok 2024-09-10T19:20:47.451 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:20:47 smithi191 ceph-mon[32884]: 6.f scrub starts 2024-09-10T19:20:47.451 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:20:47 smithi191 ceph-mon[32884]: 6.f scrub ok 2024-09-10T19:20:47.451 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:20:47 smithi191 ceph-mon[32884]: 2.1d scrub starts 2024-09-10T19:20:47.452 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:20:47 smithi191 ceph-mon[32884]: 2.1d scrub ok 2024-09-10T19:20:47.452 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:20:47 smithi191 ceph-mon[32884]: pgmap v479: 158 pgs: 158 active+clean; 5.0 GiB data, 18 GiB used, 518 GiB / 536 GiB avail; 1.2 KiB/s rd, 110 MiB/s wr, 1.75k op/s 2024-09-10T19:20:47.452 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:20:47 smithi191 ceph-mon[32884]: from='client.? 172.21.15.183:0/2958501888' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-09-10T19:20:47.551 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:20:47.588 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:20:47.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:20:47.589 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:20:47.690 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:20:47.715 INFO:journalctl@ceph.mgr.x.smithi191.stdout:Sep 10 19:20:47 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-x[34967]: ::ffff:172.21.15.191 - - [10/Sep/2024:19:20:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T19:20:47.725 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:20:47.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:20:47.727 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:20:47.735 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: 46 16 85351 85335 115.93 91.9375 0.00684205 0.00862177 2024-09-10T19:20:47.827 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:20:47.868 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:20:47.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:20:47.869 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:20:47.969 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:20:48.008 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:20:48.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:20:48.009 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:20:48.110 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:20:48.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:20:48.153 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:20:48.154 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:20:48.254 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:20:48.287 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:20:48.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:20:48.288 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:20:48.352 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:20:48 smithi183 ceph-mon[29736]: 6.16 scrub starts 2024-09-10T19:20:48.352 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:20:48 smithi183 ceph-mon[29736]: 6.16 scrub ok 2024-09-10T19:20:48.352 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:20:48 smithi183 ceph-mon[29736]: from='client.? 172.21.15.183:0/552839798' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T19:20:48.352 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:20:48 smithi183 ceph-mon[29736]: 2.14 scrub starts 2024-09-10T19:20:48.352 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:20:48 smithi183 ceph-mon[29736]: 2.14 scrub ok 2024-09-10T19:20:48.353 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:20:48 smithi183 ceph-mon[41562]: 6.16 scrub starts 2024-09-10T19:20:48.353 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:20:48 smithi183 ceph-mon[41562]: 6.16 scrub ok 2024-09-10T19:20:48.356 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:20:48 smithi183 ceph-mon[41562]: from='client.? 172.21.15.183:0/552839798' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T19:20:48.356 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:20:48 smithi183 ceph-mon[41562]: 2.14 scrub starts 2024-09-10T19:20:48.356 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:20:48 smithi183 ceph-mon[41562]: 2.14 scrub ok 2024-09-10T19:20:48.389 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:20:48.416 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:20:48.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:20:48.417 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:20:48.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:20:48 smithi191 ceph-mon[32884]: 6.16 scrub starts 2024-09-10T19:20:48.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:20:48 smithi191 ceph-mon[32884]: 6.16 scrub ok 2024-09-10T19:20:48.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:20:48 smithi191 ceph-mon[32884]: from='client.? 172.21.15.183:0/552839798' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T19:20:48.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:20:48 smithi191 ceph-mon[32884]: 2.14 scrub starts 2024-09-10T19:20:48.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:20:48 smithi191 ceph-mon[32884]: 2.14 scrub ok 2024-09-10T19:20:48.518 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:20:48.552 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:20:48.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:20:48.553 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:20:48.654 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:20:48.683 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:20:48.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:20:48.684 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:20:48.735 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: 47 16 86774 86758 115.355 88.9375 0.00766469 0.00866425 2024-09-10T19:20:48.785 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:20:48.819 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:20:48.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:20:48.820 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:20:48.921 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:20:48.963 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:20:48.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:20:48.964 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:20:49.065 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:20:49.100 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:20:49.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:20:49.101 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:20:49.114 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:20:49 smithi183 ceph-mon[29736]: 4.4 scrub starts 2024-09-10T19:20:49.114 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:20:49 smithi183 ceph-mon[29736]: 4.4 scrub ok 2024-09-10T19:20:49.114 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:20:49 smithi183 ceph-mon[29736]: pgmap v480: 158 pgs: 158 active+clean; 5.0 GiB data, 18 GiB used, 518 GiB / 536 GiB avail; 1.0 KiB/s rd, 89 MiB/s wr, 1.43k op/s 2024-09-10T19:20:49.114 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:20:49 smithi183 ceph-mon[29736]: 5.11 scrub starts 2024-09-10T19:20:49.114 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:20:49 smithi183 ceph-mon[29736]: 5.11 scrub ok 2024-09-10T19:20:49.114 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:20:49 smithi183 ceph-mon[41562]: 4.4 scrub starts 2024-09-10T19:20:49.114 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:20:49 smithi183 ceph-mon[41562]: 4.4 scrub ok 2024-09-10T19:20:49.114 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:20:49 smithi183 ceph-mon[41562]: pgmap v480: 158 pgs: 158 active+clean; 5.0 GiB data, 18 GiB used, 518 GiB / 536 GiB avail; 1.0 KiB/s rd, 89 MiB/s wr, 1.43k op/s 2024-09-10T19:20:49.114 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:20:49 smithi183 ceph-mon[41562]: 5.11 scrub starts 2024-09-10T19:20:49.114 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:20:49 smithi183 ceph-mon[41562]: 5.11 scrub ok 2024-09-10T19:20:49.202 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:20:49.239 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:20:49.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:20:49.240 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:20:49.341 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:20:49.372 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:20:49.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:20:49.373 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:20:49.464 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:20:49 smithi191 ceph-mon[32884]: 4.4 scrub starts 2024-09-10T19:20:49.464 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:20:49 smithi191 ceph-mon[32884]: 4.4 scrub ok 2024-09-10T19:20:49.464 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:20:49 smithi191 ceph-mon[32884]: pgmap v480: 158 pgs: 158 active+clean; 5.0 GiB data, 18 GiB used, 518 GiB / 536 GiB avail; 1.0 KiB/s rd, 89 MiB/s wr, 1.43k op/s 2024-09-10T19:20:49.464 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:20:49 smithi191 ceph-mon[32884]: 5.11 scrub starts 2024-09-10T19:20:49.464 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:20:49 smithi191 ceph-mon[32884]: 5.11 scrub ok 2024-09-10T19:20:49.474 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:20:49.512 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:20:49.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:20:49.513 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:20:49.614 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:20:49.643 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:20:49.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:20:49.644 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:20:49.715 INFO:journalctl@ceph.iscsi.iscsi.a.smithi191.stdout:Sep 10 19:20:49 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-iscsi-iscsi-a[61816]: debug there is no tcmu-runner data available 2024-09-10T19:20:49.735 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: 48 16 88131 88115 114.719 84.8125 0.00973081 0.00870731 2024-09-10T19:20:49.745 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:20:49.782 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:20:49.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:20:49.783 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:20:49.884 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:20:49.922 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:20:49.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:20:49.923 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:20:50.024 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:20:50.061 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:20:50.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:20:50.062 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:20:50.145 INFO:tasks.thrashosds.thrasher:j is {'epoch': 248, 'fsid': '320f6df8-6fa7-11ef-bcea-c7b262605968', 'created': '2024-09-10T19:05:03.688531+0000', 'modified': '2024-09-10T19:20:39.807982+0000', 'last_up_change': '2024-09-10T19:15:19.923894+0000', 'last_in_change': '2024-09-10T19:16:23.815531+0000', 'flags': 'nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5804032, 'flags_set': ['nodeep-scrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 19, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 9, 'max_osd': 8, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'quincy', 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-10T19:08:00.456786+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 11, 'pg_placement_num': 11, 'pg_placement_num_target': 11, 'pg_num_target': 11, 'pg_num_pending': 11, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '244', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '242', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}, {'pool': 2, 'pool_name': '.rgw.root', 'create_time': '2024-09-10T19:10:05.993763+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '60', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}}, {'pool': 3, 'pool_name': 'datapool', 'create_time': '2024-09-10T19:10:06.224267+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 3, 'pg_placement_num': 3, 'pg_placement_num_target': 3, 'pg_num_target': 3, 'pg_num_pending': 3, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '63', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 3, 'snap_epoch': 63, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 4, 'pool_name': 'default.rgw.log', 'create_time': '2024-09-10T19:10:07.958284+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '62', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}}, {'pool': 5, 'pool_name': 'default.rgw.control', 'create_time': '2024-09-10T19:10:09.933596+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '64', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}}, {'pool': 6, 'pool_name': 'default.rgw.meta', 'create_time': '2024-09-10T19:10:11.993712+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '66', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4}, 'application_metadata': {'rgw': {}}}, {'pool': 9, 'pool_name': 'unique_pool_2', 'create_time': '2024-09-10T19:19:53.124664+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 16, 'pg_placement_num': 16, 'pg_placement_num_target': 16, 'pg_num_target': 16, 'pg_num_pending': 16, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '232', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}}], 'osds': [{'osd': 0, 'uuid': '9d323010-394a-49aa-ba9b-244d87faf36e', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 8, 'up_thru': 229, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.183:6802', 'nonce': 2813074952}, {'type': 'v1', 'addr': '172.21.15.183:6803', 'nonce': 2813074952}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.183:6804', 'nonce': 2813074952}, {'type': 'v1', 'addr': '172.21.15.183:6805', 'nonce': 2813074952}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.183:6808', 'nonce': 2813074952}, {'type': 'v1', 'addr': '172.21.15.183:6809', 'nonce': 2813074952}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.183:6806', 'nonce': 2813074952}, {'type': 'v1', 'addr': '172.21.15.183:6807', 'nonce': 2813074952}]}, 'public_addr': '172.21.15.183:6803/2813074952', 'cluster_addr': '172.21.15.183:6805/2813074952', 'heartbeat_back_addr': '172.21.15.183:6809/2813074952', 'heartbeat_front_addr': '172.21.15.183:6807/2813074952', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': '71a6c5dc-5109-4284-8d20-187d8e7afc9e', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 1, 'last_clean_begin': 13, 'last_clean_end': 98, 'up_from': 138, 'up_thru': 86, 'down_at': 99, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.183:6810', 'nonce': 219854798}, {'type': 'v1', 'addr': '172.21.15.183:6811', 'nonce': 219854798}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.183:6812', 'nonce': 219854798}, {'type': 'v1', 'addr': '172.21.15.183:6813', 'nonce': 219854798}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.183:6816', 'nonce': 219854798}, {'type': 'v1', 'addr': '172.21.15.183:6817', 'nonce': 219854798}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.183:6814', 'nonce': 219854798}, {'type': 'v1', 'addr': '172.21.15.183:6815', 'nonce': 219854798}]}, 'public_addr': '172.21.15.183:6811/219854798', 'cluster_addr': '172.21.15.183:6813/219854798', 'heartbeat_back_addr': '172.21.15.183:6817/219854798', 'heartbeat_front_addr': '172.21.15.183:6815/219854798', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': 'fc310d5b-b89b-4655-9989-7e10ba356509', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 18, 'up_thru': 148, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.183:6818', 'nonce': 3730013786}, {'type': 'v1', 'addr': '172.21.15.183:6819', 'nonce': 3730013786}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.183:6820', 'nonce': 3730013786}, {'type': 'v1', 'addr': '172.21.15.183:6821', 'nonce': 3730013786}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.183:6824', 'nonce': 3730013786}, {'type': 'v1', 'addr': '172.21.15.183:6825', 'nonce': 3730013786}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.183:6822', 'nonce': 3730013786}, {'type': 'v1', 'addr': '172.21.15.183:6823', 'nonce': 3730013786}]}, 'public_addr': '172.21.15.183:6819/3730013786', 'cluster_addr': '172.21.15.183:6821/3730013786', 'heartbeat_back_addr': '172.21.15.183:6825/3730013786', 'heartbeat_front_addr': '172.21.15.183:6823/3730013786', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': '7fd07a95-ad22-43f5-a968-539552283e89', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 26, 'up_thru': 244, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.183:6826', 'nonce': 3522836984}, {'type': 'v1', 'addr': '172.21.15.183:6827', 'nonce': 3522836984}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.183:6828', 'nonce': 3522836984}, {'type': 'v1', 'addr': '172.21.15.183:6829', 'nonce': 3522836984}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.183:6832', 'nonce': 3522836984}, {'type': 'v1', 'addr': '172.21.15.183:6833', 'nonce': 3522836984}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.183:6830', 'nonce': 3522836984}, {'type': 'v1', 'addr': '172.21.15.183:6831', 'nonce': 3522836984}]}, 'public_addr': '172.21.15.183:6827/3522836984', 'cluster_addr': '172.21.15.183:6829/3522836984', 'heartbeat_back_addr': '172.21.15.183:6833/3522836984', 'heartbeat_front_addr': '172.21.15.183:6831/3522836984', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': '631083fb-e503-44da-9b6b-154e88539ee7', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 32, 'up_thru': 244, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.191:6800', 'nonce': 2674828539}, {'type': 'v1', 'addr': '172.21.15.191:6801', 'nonce': 2674828539}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.191:6802', 'nonce': 2674828539}, {'type': 'v1', 'addr': '172.21.15.191:6803', 'nonce': 2674828539}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.191:6806', 'nonce': 2674828539}, {'type': 'v1', 'addr': '172.21.15.191:6807', 'nonce': 2674828539}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.191:6804', 'nonce': 2674828539}, {'type': 'v1', 'addr': '172.21.15.191:6805', 'nonce': 2674828539}]}, 'public_addr': '172.21.15.191:6801/2674828539', 'cluster_addr': '172.21.15.191:6803/2674828539', 'heartbeat_back_addr': '172.21.15.191:6807/2674828539', 'heartbeat_front_addr': '172.21.15.191:6805/2674828539', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': '6b963695-3eca-4e6e-9207-05b8010ebff9', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 39, 'up_thru': 244, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.191:6808', 'nonce': 681733378}, {'type': 'v1', 'addr': '172.21.15.191:6809', 'nonce': 681733378}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.191:6810', 'nonce': 681733378}, {'type': 'v1', 'addr': '172.21.15.191:6811', 'nonce': 681733378}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.191:6814', 'nonce': 681733378}, {'type': 'v1', 'addr': '172.21.15.191:6815', 'nonce': 681733378}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.191:6812', 'nonce': 681733378}, {'type': 'v1', 'addr': '172.21.15.191:6813', 'nonce': 681733378}]}, 'public_addr': '172.21.15.191:6809/681733378', 'cluster_addr': '172.21.15.191:6811/681733378', 'heartbeat_back_addr': '172.21.15.191:6815/681733378', 'heartbeat_front_addr': '172.21.15.191:6813/681733378', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': '2dc9754f-6aef-4e19-8605-2f1591921811', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 47, 'up_thru': 244, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.191:6816', 'nonce': 2825813477}, {'type': 'v1', 'addr': '172.21.15.191:6817', 'nonce': 2825813477}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.191:6818', 'nonce': 2825813477}, {'type': 'v1', 'addr': '172.21.15.191:6819', 'nonce': 2825813477}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.191:6822', 'nonce': 2825813477}, {'type': 'v1', 'addr': '172.21.15.191:6823', 'nonce': 2825813477}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.191:6820', 'nonce': 2825813477}, {'type': 'v1', 'addr': '172.21.15.191:6821', 'nonce': 2825813477}]}, 'public_addr': '172.21.15.191:6817/2825813477', 'cluster_addr': '172.21.15.191:6819/2825813477', 'heartbeat_back_addr': '172.21.15.191:6823/2825813477', 'heartbeat_front_addr': '172.21.15.191:6821/2825813477', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': '1a9a299e-d5ba-4b9a-85d5-7f6a5bc754e1', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 54, 'up_thru': 244, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.191:6824', 'nonce': 1422610021}, {'type': 'v1', 'addr': '172.21.15.191:6825', 'nonce': 1422610021}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.191:6826', 'nonce': 1422610021}, {'type': 'v1', 'addr': '172.21.15.191:6827', 'nonce': 1422610021}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.191:6830', 'nonce': 1422610021}, {'type': 'v1', 'addr': '172.21.15.191:6831', 'nonce': 1422610021}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.191:6828', 'nonce': 1422610021}, {'type': 'v1', 'addr': '172.21.15.191:6829', 'nonce': 1422610021}]}, 'public_addr': '172.21.15.191:6825/1422610021', 'cluster_addr': '172.21.15.191:6827/1422610021', 'heartbeat_back_addr': '172.21.15.191:6831/1422610021', 'heartbeat_front_addr': '172.21.15.191:6829/1422610021', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138320759226367, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-10T19:20:25.561104+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138320759226367, 'old_weight': 65536, 'last_purged_snaps_scrub': '2024-09-10T19:20:37.497093+0000', 'dead_epoch': 98}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138320759226367, 'old_weight': 65536, 'last_purged_snaps_scrub': '2024-09-10T19:19:42.374103+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138320759226367, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-10T19:20:02.376672+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138320759226367, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-10T19:20:08.110504+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138320759226367, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-10T19:19:59.468537+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138320759226367, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-10T19:19:47.856262+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138320759226367, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-10T19:20:23.785620+0000', 'dead_epoch': 0}], 'pg_upmap': [{'pgid': '2.13', 'osds': [2, 0, 5]}], 'pg_upmap_items': [{'pgid': '2.18', 'mappings': [{'from': 7, 'to': 0}, {'from': 5, 'to': 4}]}, {'pgid': '4.1', 'mappings': [{'from': 6, 'to': 7}]}, {'pgid': '4.11', 'mappings': [{'from': 3, 'to': 5}]}], 'pg_temp': [], 'primary_temp': [], 'blocklist': {'172.21.15.183:6801/3311783362': '2024-09-11T19:10:43.576943+0000', '172.21.15.183:0/4088553922': '2024-09-11T19:10:43.576943+0000', '172.21.15.183:0/3528769414': '2024-09-11T19:05:48.153879+0000', '172.21.15.183:0/1766036658': '2024-09-11T19:05:48.153879+0000', '172.21.15.183:0/3044170088': '2024-09-11T19:05:25.702354+0000', '172.21.15.183:0/3037333778': '2024-09-11T19:05:25.702354+0000', '172.21.15.183:6801/3747320570': '2024-09-11T19:05:25.702354+0000', '172.21.15.183:0/3982386906': '2024-09-11T19:10:43.576943+0000', '172.21.15.183:0/2436014455': '2024-09-11T19:05:25.702354+0000', '172.21.15.183:6801/659745765': '2024-09-11T19:05:48.153879+0000', '172.21.15.183:6800/3747320570': '2024-09-11T19:05:25.702354+0000', '172.21.15.183:0/1652576282': '2024-09-11T19:10:43.576943+0000', '172.21.15.183:6800/3311783362': '2024-09-11T19:10:43.576943+0000', '172.21.15.183:0/1012899549': '2024-09-11T19:05:48.153879+0000', '172.21.15.183:6800/659745765': '2024-09-11T19:05:48.153879+0000', '172.21.15.183:0/412044400': '2024-09-11T19:10:43.576943+0000', '172.21.15.183:0/3636171401': '2024-09-11T19:10:43.576943+0000'}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-09-10T19:20:50.146 INFO:tasks.thrashosds.thrasher:Clearing pg_upmap on 4.11 2024-09-10T19:20:50.146 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph osd rm-pg-upmap-items 4.11 2024-09-10T19:20:50.163 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:20:50.225 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:20:50.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:20:50.226 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:20:50.327 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:20:50.332 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:20:50 smithi183 ceph-mon[41562]: 2.13 scrub starts 2024-09-10T19:20:50.332 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:20:50 smithi183 ceph-mon[41562]: 2.13 scrub ok 2024-09-10T19:20:50.332 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:20:50 smithi183 ceph-mon[29736]: 2.13 scrub starts 2024-09-10T19:20:50.332 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:20:50 smithi183 ceph-mon[29736]: 2.13 scrub ok 2024-09-10T19:20:50.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:20:50.361 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:20:50.362 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:20:50.462 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:20:50.465 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:20:50 smithi191 ceph-mon[32884]: 2.13 scrub starts 2024-09-10T19:20:50.465 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:20:50 smithi191 ceph-mon[32884]: 2.13 scrub ok 2024-09-10T19:20:50.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:20:50.500 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:20:50.500 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:20:50.604 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:20:50.643 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:20:50.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:20:50.644 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:20:50.737 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: 49 16 89573 89557 114.214 90.125 0.00994073 0.00875088 2024-09-10T19:20:50.745 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:20:50.782 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:20:50.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:20:50.784 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:20:50.868 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:20:50.884 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:20:50.885 INFO:teuthology.orchestra.run.smithi183.stdout:{ 2024-09-10T19:20:50.885 INFO:teuthology.orchestra.run.smithi183.stdout: "ops": [], 2024-09-10T19:20:50.885 INFO:teuthology.orchestra.run.smithi183.stdout: "complaint_time": 30, 2024-09-10T19:20:50.885 INFO:teuthology.orchestra.run.smithi183.stdout: "num_blocked_ops": 0 2024-09-10T19:20:50.886 INFO:teuthology.orchestra.run.smithi183.stdout:} 2024-09-10T19:20:50.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:20:50.938 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:20:50.938 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:20:51.039 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:20:51.077 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:20:51.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:20:51.078 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:20:51.179 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:20:51.223 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:20:51.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:20:51.226 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:20:51.316 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:20:51 smithi183 ceph-mon[29736]: 6.d scrub starts 2024-09-10T19:20:51.316 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:20:51 smithi183 ceph-mon[29736]: 6.d scrub ok 2024-09-10T19:20:51.316 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:20:51 smithi183 ceph-mon[29736]: from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:20:51.316 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:20:51 smithi183 ceph-mon[29736]: 2.b scrub starts 2024-09-10T19:20:51.316 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:20:51 smithi183 ceph-mon[29736]: 2.b scrub ok 2024-09-10T19:20:51.316 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:20:51 smithi183 ceph-mon[29736]: pgmap v481: 158 pgs: 158 active+clean; 5.0 GiB data, 18 GiB used, 518 GiB / 536 GiB avail; 1023 B/s rd, 88 MiB/s wr, 1.40k op/s 2024-09-10T19:20:51.316 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:20:51 smithi183 ceph-mon[41562]: 6.d scrub starts 2024-09-10T19:20:51.316 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:20:51 smithi183 ceph-mon[41562]: 6.d scrub ok 2024-09-10T19:20:51.317 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:20:51 smithi183 ceph-mon[41562]: from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:20:51.317 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:20:51 smithi183 ceph-mon[41562]: 2.b scrub starts 2024-09-10T19:20:51.317 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:20:51 smithi183 ceph-mon[41562]: 2.b scrub ok 2024-09-10T19:20:51.317 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:20:51 smithi183 ceph-mon[41562]: pgmap v481: 158 pgs: 158 active+clean; 5.0 GiB data, 18 GiB used, 518 GiB / 536 GiB avail; 1023 B/s rd, 88 MiB/s wr, 1.40k op/s 2024-09-10T19:20:51.327 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:20:51.366 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:20:51.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:20:51.367 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:20:51.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:20:51 smithi191 ceph-mon[32884]: 6.d scrub starts 2024-09-10T19:20:51.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:20:51 smithi191 ceph-mon[32884]: 6.d scrub ok 2024-09-10T19:20:51.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:20:51 smithi191 ceph-mon[32884]: from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:20:51.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:20:51 smithi191 ceph-mon[32884]: 2.b scrub starts 2024-09-10T19:20:51.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:20:51 smithi191 ceph-mon[32884]: 2.b scrub ok 2024-09-10T19:20:51.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:20:51 smithi191 ceph-mon[32884]: pgmap v481: 158 pgs: 158 active+clean; 5.0 GiB data, 18 GiB used, 518 GiB / 536 GiB avail; 1023 B/s rd, 88 MiB/s wr, 1.40k op/s 2024-09-10T19:20:51.468 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:20:51.501 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:20:51.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:20:51.502 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:20:51.570 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Sep 10 19:20:51 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-y[29989]: ::ffff:172.21.15.191 - - [10/Sep/2024:19:20:51] "GET /metrics HTTP/1.1" 200 224393 "" "Prometheus/2.43.0" 2024-09-10T19:20:51.603 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:20:51.643 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:20:51.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:20:51.644 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:20:51.739 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: 50 16 90996 90980 113.702 88.9375 0.0115185 0.00879085 2024-09-10T19:20:51.745 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:20:51.784 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:20:51.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:20:51.785 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:20:51.886 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:20:51.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:20:51.925 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:20:51.925 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:20:52.026 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:20:52.058 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:20:52.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:20:52.059 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:20:52.116 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:20:52 smithi183 ceph-mon[29736]: 6.19 scrub starts 2024-09-10T19:20:52.116 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:20:52 smithi183 ceph-mon[29736]: 6.19 scrub ok 2024-09-10T19:20:52.116 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:20:52 smithi183 ceph-mon[29736]: 5.13 scrub starts 2024-09-10T19:20:52.116 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:20:52 smithi183 ceph-mon[29736]: 5.13 scrub ok 2024-09-10T19:20:52.116 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:20:52 smithi183 ceph-mon[29736]: 6.15 scrub starts 2024-09-10T19:20:52.116 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:20:52 smithi183 ceph-mon[29736]: 6.15 scrub ok 2024-09-10T19:20:52.116 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:20:52 smithi183 ceph-mon[29736]: 2.1a scrub starts 2024-09-10T19:20:52.117 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:20:52 smithi183 ceph-mon[29736]: 2.1a scrub ok 2024-09-10T19:20:52.117 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:20:52 smithi183 ceph-mon[41562]: 6.19 scrub starts 2024-09-10T19:20:52.117 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:20:52 smithi183 ceph-mon[41562]: 6.19 scrub ok 2024-09-10T19:20:52.117 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:20:52 smithi183 ceph-mon[41562]: 5.13 scrub starts 2024-09-10T19:20:52.117 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:20:52 smithi183 ceph-mon[41562]: 5.13 scrub ok 2024-09-10T19:20:52.117 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:20:52 smithi183 ceph-mon[41562]: 6.15 scrub starts 2024-09-10T19:20:52.117 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:20:52 smithi183 ceph-mon[41562]: 6.15 scrub ok 2024-09-10T19:20:52.117 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:20:52 smithi183 ceph-mon[41562]: 2.1a scrub starts 2024-09-10T19:20:52.117 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:20:52 smithi183 ceph-mon[41562]: 2.1a scrub ok 2024-09-10T19:20:52.161 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:20:52.192 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:20:52.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:20:52.193 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:20:52.294 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:20:52.323 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:20:52.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:20:52.324 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:20:52.424 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:20:52.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:20:52 smithi191 ceph-mon[32884]: 6.19 scrub starts 2024-09-10T19:20:52.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:20:52 smithi191 ceph-mon[32884]: 6.19 scrub ok 2024-09-10T19:20:52.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:20:52 smithi191 ceph-mon[32884]: 5.13 scrub starts 2024-09-10T19:20:52.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:20:52 smithi191 ceph-mon[32884]: 5.13 scrub ok 2024-09-10T19:20:52.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:20:52 smithi191 ceph-mon[32884]: 6.15 scrub starts 2024-09-10T19:20:52.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:20:52 smithi191 ceph-mon[32884]: 6.15 scrub ok 2024-09-10T19:20:52.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:20:52 smithi191 ceph-mon[32884]: 2.1a scrub starts 2024-09-10T19:20:52.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:20:52 smithi191 ceph-mon[32884]: 2.1a scrub ok 2024-09-10T19:20:52.467 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:20:52.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:20:52.468 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:20:52.569 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:20:52.578 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph daemon osd.0 dump_historic_ops 2024-09-10T19:20:52.604 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:20:52.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:20:52.605 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:20:52.706 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:20:52.739 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: 51 16 92425 92409 113.224 89.3125 0.0183655 0.00882699 2024-09-10T19:20:52.741 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:20:52.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:20:52.742 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:20:52.843 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:20:52.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:20:52.884 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:20:52.885 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:20:52.985 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:20:53.015 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:20:53.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:20:53.016 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:20:53.117 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:20:53.156 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:20:53.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:20:53.157 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:20:53.258 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:20:53.296 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:20:53.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:20:53.297 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:20:53.307 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:20:53 smithi183 ceph-mon[29736]: 4.7 scrub starts 2024-09-10T19:20:53.307 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:20:53 smithi183 ceph-mon[29736]: 4.7 scrub ok 2024-09-10T19:20:53.307 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:20:53 smithi183 ceph-mon[29736]: pgmap v482: 158 pgs: 158 active+clean; 5.4 GiB data, 20 GiB used, 517 GiB / 536 GiB avail; 1.3 KiB/s rd, 111 MiB/s wr, 1.77k op/s 2024-09-10T19:20:53.307 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:20:53 smithi183 ceph-mon[41562]: 4.7 scrub starts 2024-09-10T19:20:53.307 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:20:53 smithi183 ceph-mon[41562]: 4.7 scrub ok 2024-09-10T19:20:53.307 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:20:53 smithi183 ceph-mon[41562]: pgmap v482: 158 pgs: 158 active+clean; 5.4 GiB data, 20 GiB used, 517 GiB / 536 GiB avail; 1.3 KiB/s rd, 111 MiB/s wr, 1.77k op/s 2024-09-10T19:20:53.366 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:20:53.398 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:20:53.436 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:20:53.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:20:53.437 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:20:53.465 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:20:53 smithi191 ceph-mon[32884]: 4.7 scrub starts 2024-09-10T19:20:53.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:20:53 smithi191 ceph-mon[32884]: 4.7 scrub ok 2024-09-10T19:20:53.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:20:53 smithi191 ceph-mon[32884]: pgmap v482: 158 pgs: 158 active+clean; 5.4 GiB data, 20 GiB used, 517 GiB / 536 GiB avail; 1.3 KiB/s rd, 111 MiB/s wr, 1.77k op/s 2024-09-10T19:20:53.538 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:20:53.579 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:20:53.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:20:53.581 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:20:53.616 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-10T19:20:53.681 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:20:53.719 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:20:53.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:20:53.720 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:20:53.739 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: 52 16 93863 93847 112.774 89.875 0.00614492 0.00886259 2024-09-10T19:20:53.821 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:20:53.857 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:20:53.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:20:53.858 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:20:53.959 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:20:53.988 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:20:53.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:20:53.989 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:20:54.090 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:20:54.127 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:20:54.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:20:54.128 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:20:54.229 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:20:54.264 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:20:54.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:20:54.265 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:20:54.356 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:20:54 smithi183 ceph-mon[29736]: 4.1b scrub starts 2024-09-10T19:20:54.357 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:20:54 smithi183 ceph-mon[29736]: 4.1b scrub ok 2024-09-10T19:20:54.357 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:20:54 smithi183 ceph-mon[29736]: 2.4 scrub starts 2024-09-10T19:20:54.357 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:20:54 smithi183 ceph-mon[29736]: 2.4 scrub ok 2024-09-10T19:20:54.357 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:20:54 smithi183 ceph-mon[29736]: purged_snaps scrub starts 2024-09-10T19:20:54.357 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:20:54 smithi183 ceph-mon[29736]: purged_snaps scrub ok 2024-09-10T19:20:54.357 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:20:54 smithi183 ceph-mon[29736]: osdmap e249: 8 total, 8 up, 6 in 2024-09-10T19:20:54.357 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:20:54 smithi183 ceph-mon[41562]: 4.1b scrub starts 2024-09-10T19:20:54.357 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:20:54 smithi183 ceph-mon[41562]: 4.1b scrub ok 2024-09-10T19:20:54.357 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:20:54 smithi183 ceph-mon[41562]: 2.4 scrub starts 2024-09-10T19:20:54.357 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:20:54 smithi183 ceph-mon[41562]: 2.4 scrub ok 2024-09-10T19:20:54.357 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:20:54 smithi183 ceph-mon[41562]: purged_snaps scrub starts 2024-09-10T19:20:54.358 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:20:54 smithi183 ceph-mon[41562]: purged_snaps scrub ok 2024-09-10T19:20:54.358 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:20:54 smithi183 ceph-mon[41562]: osdmap e249: 8 total, 8 up, 6 in 2024-09-10T19:20:54.366 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:20:54.405 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:20:54.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:20:54.406 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:20:54.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:20:54 smithi191 ceph-mon[32884]: 4.1b scrub starts 2024-09-10T19:20:54.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:20:54 smithi191 ceph-mon[32884]: 4.1b scrub ok 2024-09-10T19:20:54.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:20:54 smithi191 ceph-mon[32884]: 2.4 scrub starts 2024-09-10T19:20:54.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:20:54 smithi191 ceph-mon[32884]: 2.4 scrub ok 2024-09-10T19:20:54.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:20:54 smithi191 ceph-mon[32884]: purged_snaps scrub starts 2024-09-10T19:20:54.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:20:54 smithi191 ceph-mon[32884]: purged_snaps scrub ok 2024-09-10T19:20:54.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:20:54 smithi191 ceph-mon[32884]: osdmap e249: 8 total, 8 up, 6 in 2024-09-10T19:20:54.507 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:20:54.535 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:20:54.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:20:54.536 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:20:54.637 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:20:54.672 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:20:54.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:20:54.673 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:20:54.740 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: 53 16 95303 95287 112.345 90 0.0196601 0.00889701 2024-09-10T19:20:54.774 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:20:54.805 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:20:54.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:20:54.806 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:20:54.908 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:20:54.938 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:20:54.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:20:54.939 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:20:55.040 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:20:55.079 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:20:55.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:20:55.080 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:20:55.181 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:20:55.214 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:20:55.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:20:55.215 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:20:55.316 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:20:55.347 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:20:55.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:20:55.349 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:20:55.352 INFO:teuthology.orchestra.run.smithi183.stderr:nodeep-scrub is unset 2024-09-10T19:20:55.353 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:20:55 smithi183 ceph-mon[29736]: pgmap v484: 158 pgs: 158 active+clean; 5.4 GiB data, 20 GiB used, 517 GiB / 536 GiB avail; 1023 B/s rd, 87 MiB/s wr, 1.40k op/s 2024-09-10T19:20:55.353 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:20:55 smithi183 ceph-mon[29736]: from='client.? 172.21.15.183:0/992047391' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-10T19:20:55.353 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:20:55 smithi183 ceph-mon[29736]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-10T19:20:55.353 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:20:55 smithi183 ceph-mon[29736]: Cluster is now healthy 2024-09-10T19:20:55.353 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:20:55 smithi183 ceph-mon[29736]: from='client.? 172.21.15.183:0/992047391' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-10T19:20:55.353 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:20:55 smithi183 ceph-mon[29736]: osdmap e250: 8 total, 8 up, 6 in 2024-09-10T19:20:55.354 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:20:55 smithi183 ceph-mon[41562]: pgmap v484: 158 pgs: 158 active+clean; 5.4 GiB data, 20 GiB used, 517 GiB / 536 GiB avail; 1023 B/s rd, 87 MiB/s wr, 1.40k op/s 2024-09-10T19:20:55.354 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:20:55 smithi183 ceph-mon[41562]: from='client.? 172.21.15.183:0/992047391' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-10T19:20:55.354 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:20:55 smithi183 ceph-mon[41562]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-10T19:20:55.354 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:20:55 smithi183 ceph-mon[41562]: Cluster is now healthy 2024-09-10T19:20:55.354 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:20:55 smithi183 ceph-mon[41562]: from='client.? 172.21.15.183:0/992047391' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-10T19:20:55.354 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:20:55 smithi183 ceph-mon[41562]: osdmap e250: 8 total, 8 up, 6 in 2024-09-10T19:20:55.450 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:20:55.465 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:20:55 smithi191 ceph-mon[32884]: pgmap v484: 158 pgs: 158 active+clean; 5.4 GiB data, 20 GiB used, 517 GiB / 536 GiB avail; 1023 B/s rd, 87 MiB/s wr, 1.40k op/s 2024-09-10T19:20:55.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:20:55 smithi191 ceph-mon[32884]: from='client.? 172.21.15.183:0/992047391' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-10T19:20:55.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:20:55 smithi191 ceph-mon[32884]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-10T19:20:55.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:20:55 smithi191 ceph-mon[32884]: Cluster is now healthy 2024-09-10T19:20:55.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:20:55 smithi191 ceph-mon[32884]: from='client.? 172.21.15.183:0/992047391' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-10T19:20:55.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:20:55 smithi191 ceph-mon[32884]: osdmap e250: 8 total, 8 up, 6 in 2024-09-10T19:20:55.478 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:20:55.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:20:55.480 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:20:55.570 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:20:55.581 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:20:55.621 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:20:55.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:20:55.622 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:20:55.723 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:20:55.740 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: 54 16 96744 96728 111.932 90.0625 0.00774982 0.00892981 2024-09-10T19:20:55.761 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:20:55.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:20:55.762 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:20:55.863 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:20:55.900 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:20:55.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:20:55.901 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:20:56.001 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:20:56.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:20:56.042 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:20:56.042 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:20:56.143 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:20:56.177 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:20:56.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:20:56.178 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:20:56.279 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:20:56.295 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:20:56 smithi183 ceph-mon[29736]: 5.c scrub starts 2024-09-10T19:20:56.295 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:20:56 smithi183 ceph-mon[29736]: 5.c scrub ok 2024-09-10T19:20:56.295 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:20:56 smithi183 ceph-mon[29736]: 5.4 scrub starts 2024-09-10T19:20:56.295 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:20:56 smithi183 ceph-mon[29736]: 5.4 scrub ok 2024-09-10T19:20:56.295 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:20:56 smithi183 ceph-mon[41562]: 5.c scrub starts 2024-09-10T19:20:56.295 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:20:56 smithi183 ceph-mon[41562]: 5.c scrub ok 2024-09-10T19:20:56.295 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:20:56 smithi183 ceph-mon[41562]: 5.4 scrub starts 2024-09-10T19:20:56.296 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:20:56 smithi183 ceph-mon[41562]: 5.4 scrub ok 2024-09-10T19:20:56.317 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:20:56.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:20:56.318 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:20:56.419 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:20:56.457 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:20:56.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:20:56.459 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:20:56.465 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:20:56 smithi191 ceph-mon[32884]: 5.c scrub starts 2024-09-10T19:20:56.465 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:20:56 smithi191 ceph-mon[32884]: 5.c scrub ok 2024-09-10T19:20:56.465 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:20:56 smithi191 ceph-mon[32884]: 5.4 scrub starts 2024-09-10T19:20:56.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:20:56 smithi191 ceph-mon[32884]: 5.4 scrub ok 2024-09-10T19:20:56.559 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:20:56.594 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:20:56.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:20:56.595 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:20:56.600 INFO:teuthology.orchestra.run.smithi183.stdout:{ 2024-09-10T19:20:56.600 INFO:teuthology.orchestra.run.smithi183.stdout: "size": 20, 2024-09-10T19:20:56.600 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 600, 2024-09-10T19:20:56.600 INFO:teuthology.orchestra.run.smithi183.stdout: "ops": [ 2024-09-10T19:20:56.600 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:20:56.600 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.14781.0:161671 7.6 7:6b18f858:::benchmark_data_smithi183_84160_object21290:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120)", 2024-09-10T19:20:56.600 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-10T19:14:08.088525+0000", 2024-09-10T19:20:56.600 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 408.504134884, 2024-09-10T19:20:56.600 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.15036340400000001, 2024-09-10T19:20:56.601 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-10T19:20:56.601 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T19:20:56.601 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-10T19:20:56.601 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.14781", 2024-09-10T19:20:56.601 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.183:0/2025776420", 2024-09-10T19:20:56.601 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 161671 2024-09-10T19:20:56.601 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:20:56.601 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-10T19:20:56.601 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:20:56.601 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-10T19:20:56.601 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:08.088525+0000", 2024-09-10T19:20:56.601 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:20:56.601 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:20:56.601 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:20:56.602 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-10T19:20:56.602 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:08.088525+0000", 2024-09-10T19:20:56.602 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:20:56.602 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:20:56.602 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:20:56.602 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-10T19:20:56.602 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:08.088524+0000", 2024-09-10T19:20:56.602 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4294967295.999999 2024-09-10T19:20:56.602 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:20:56.602 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:20:56.602 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-10T19:20:56.602 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:08.088529+0000", 2024-09-10T19:20:56.602 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.1340000000000003e-06 2024-09-10T19:20:56.602 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:20:56.603 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:20:56.603 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-10T19:20:56.603 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:08.088539+0000", 2024-09-10T19:20:56.603 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.9760000000000007e-06 2024-09-10T19:20:56.603 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:20:56.603 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:20:56.603 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-10T19:20:56.603 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:08.088544+0000", 2024-09-10T19:20:56.603 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.04e-06 2024-09-10T19:20:56.603 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:20:56.603 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:20:56.603 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-10T19:20:56.603 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:08.090909+0000", 2024-09-10T19:20:56.603 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0023652909999999998 2024-09-10T19:20:56.604 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:20:56.604 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:20:56.604 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-10T19:20:56.604 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:08.091009+0000", 2024-09-10T19:20:56.604 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000100052 2024-09-10T19:20:56.604 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:20:56.604 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:20:56.604 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 3,7", 2024-09-10T19:20:56.604 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:08.091125+0000", 2024-09-10T19:20:56.604 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0001153 2024-09-10T19:20:56.604 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:20:56.604 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:20:56.604 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-10T19:20:56.605 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:08.097804+0000", 2024-09-10T19:20:56.605 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0066796049999999999 2024-09-10T19:20:56.605 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:20:56.605 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:20:56.605 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:20:56.605 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:08.097902+0000", 2024-09-10T19:20:56.605 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.7802999999999999e-05 2024-09-10T19:20:56.605 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:20:56.605 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:20:56.605 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:20:56.605 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:08.238808+0000", 2024-09-10T19:20:56.605 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.140905697 2024-09-10T19:20:56.605 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:20:56.605 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:20:56.606 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-10T19:20:56.606 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:08.238853+0000", 2024-09-10T19:20:56.606 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.5595e-05 2024-09-10T19:20:56.606 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:20:56.606 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:20:56.606 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-10T19:20:56.606 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:08.238888+0000", 2024-09-10T19:20:56.606 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.4829999999999997e-05 2024-09-10T19:20:56.606 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:20:56.606 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:20:56.606 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:20:56.606 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:20:56.606 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:20:56.607 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.14781.0:161673 7.6 7:6af6847e:::benchmark_data_smithi183_84160_object21292:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e120)", 2024-09-10T19:20:56.607 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-10T19:14:08.088863+0000", 2024-09-10T19:20:56.607 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 408.50379686799999, 2024-09-10T19:20:56.607 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.15014227999999999, 2024-09-10T19:20:56.607 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-10T19:20:56.607 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T19:20:56.607 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-10T19:20:56.607 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.14781", 2024-09-10T19:20:56.607 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.183:0/2025776420", 2024-09-10T19:20:56.607 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 161673 2024-09-10T19:20:56.607 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:20:56.607 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-10T19:20:56.607 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:20:56.608 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-10T19:20:56.608 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:08.088863+0000", 2024-09-10T19:20:56.608 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:20:56.608 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:20:56.608 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:20:56.608 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-10T19:20:56.608 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:08.088863+0000", 2024-09-10T19:20:56.608 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:20:56.608 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:20:56.608 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:20:56.608 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-10T19:20:56.608 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:08.088862+0000", 2024-09-10T19:20:56.608 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4294967295.999999 2024-09-10T19:20:56.608 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:20:56.609 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:20:56.609 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-10T19:20:56.609 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:08.088867+0000", 2024-09-10T19:20:56.609 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.5449999999999997e-06 2024-09-10T19:20:56.609 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:20:56.609 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:20:56.609 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-10T19:20:56.609 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:08.088876+0000", 2024-09-10T19:20:56.609 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.9250000000000007e-06 2024-09-10T19:20:56.609 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:20:56.609 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:20:56.609 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-10T19:20:56.609 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:08.088881+0000", 2024-09-10T19:20:56.610 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.8400000000000002e-06 2024-09-10T19:20:56.610 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:20:56.610 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:20:56.610 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-10T19:20:56.610 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:08.094242+0000", 2024-09-10T19:20:56.610 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0053602090000000003 2024-09-10T19:20:56.610 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:20:56.610 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:20:56.610 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-10T19:20:56.610 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:08.094343+0000", 2024-09-10T19:20:56.610 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000101314 2024-09-10T19:20:56.610 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:20:56.610 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:20:56.611 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 3,7", 2024-09-10T19:20:56.611 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:08.094443+0000", 2024-09-10T19:20:56.611 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000100504 2024-09-10T19:20:56.611 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:20:56.611 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:20:56.611 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-10T19:20:56.611 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:08.098325+0000", 2024-09-10T19:20:56.611 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0038816409999999999 2024-09-10T19:20:56.611 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:20:56.611 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:20:56.611 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:20:56.611 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:08.099090+0000", 2024-09-10T19:20:56.611 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00076455999999999996 2024-09-10T19:20:56.612 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:20:56.612 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:20:56.612 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:20:56.612 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:08.238937+0000", 2024-09-10T19:20:56.612 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.13984723299999999 2024-09-10T19:20:56.612 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:20:56.612 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:20:56.612 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-10T19:20:56.612 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:08.238975+0000", 2024-09-10T19:20:56.612 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.7731000000000002e-05 2024-09-10T19:20:56.612 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:20:56.612 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:20:56.612 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-10T19:20:56.612 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:08.239005+0000", 2024-09-10T19:20:56.613 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.0577999999999998e-05 2024-09-10T19:20:56.613 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:20:56.613 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:20:56.613 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:20:56.613 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:20:56.613 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:20:56.613 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.14781.0:217288 7.9 7:9176e26d:::benchmark_data_smithi183_84160_object76907:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e130)", 2024-09-10T19:20:56.613 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-10T19:14:55.869032+0000", 2024-09-10T19:20:56.613 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 360.72362796300001, 2024-09-10T19:20:56.613 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.99896858300000002, 2024-09-10T19:20:56.613 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-10T19:20:56.613 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T19:20:56.613 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-10T19:20:56.614 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.14781", 2024-09-10T19:20:56.614 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.183:0/2025776420", 2024-09-10T19:20:56.614 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 217288 2024-09-10T19:20:56.614 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:20:56.614 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-10T19:20:56.614 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:20:56.614 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-10T19:20:56.614 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:55.869032+0000", 2024-09-10T19:20:56.614 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:20:56.614 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:20:56.614 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:20:56.614 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-10T19:20:56.614 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:55.869032+0000", 2024-09-10T19:20:56.615 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:20:56.615 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:20:56.615 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:20:56.615 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-10T19:20:56.615 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:55.869031+0000", 2024-09-10T19:20:56.615 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4294967295.9999995 2024-09-10T19:20:56.615 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:20:56.615 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:20:56.615 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-10T19:20:56.615 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:55.869035+0000", 2024-09-10T19:20:56.615 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.5839999999999999e-06 2024-09-10T19:20:56.615 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:20:56.615 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:20:56.615 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-10T19:20:56.616 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:55.869042+0000", 2024-09-10T19:20:56.616 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.3679999999999996e-06 2024-09-10T19:20:56.616 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:20:56.616 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:20:56.616 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-10T19:20:56.616 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:55.869046+0000", 2024-09-10T19:20:56.616 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.1790000000000001e-06 2024-09-10T19:20:56.616 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:20:56.616 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:20:56.616 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-10T19:20:56.616 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:55.869093+0000", 2024-09-10T19:20:56.616 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.6292999999999999e-05 2024-09-10T19:20:56.616 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:20:56.616 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:20:56.616 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for peered", 2024-09-10T19:20:56.617 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:55.869102+0000", 2024-09-10T19:20:56.617 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.7229999999999999e-06 2024-09-10T19:20:56.617 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:20:56.617 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:20:56.617 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-10T19:20:56.617 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:56.864757+0000", 2024-09-10T19:20:56.617 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.99565468700000004 2024-09-10T19:20:56.617 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:20:56.617 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:20:56.617 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-10T19:20:56.617 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:56.864889+0000", 2024-09-10T19:20:56.617 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000132223 2024-09-10T19:20:56.617 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:20:56.618 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:20:56.618 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 4,7", 2024-09-10T19:20:56.618 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:56.865028+0000", 2024-09-10T19:20:56.618 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00013886399999999999 2024-09-10T19:20:56.618 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:20:56.618 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:20:56.618 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:20:56.618 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:56.867850+0000", 2024-09-10T19:20:56.618 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0028217210000000001 2024-09-10T19:20:56.618 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:20:56.618 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:20:56.618 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:20:56.618 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:56.867884+0000", 2024-09-10T19:20:56.619 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.4119999999999999e-05 2024-09-10T19:20:56.619 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:20:56.619 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:20:56.619 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-10T19:20:56.619 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:56.867906+0000", 2024-09-10T19:20:56.619 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.2514000000000002e-05 2024-09-10T19:20:56.619 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:20:56.619 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:20:56.619 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-10T19:20:56.619 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:56.867953+0000", 2024-09-10T19:20:56.619 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.6999999999999997e-05 2024-09-10T19:20:56.619 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:20:56.619 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:20:56.620 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-10T19:20:56.620 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:56.868000+0000", 2024-09-10T19:20:56.620 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.6901999999999999e-05 2024-09-10T19:20:56.620 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:20:56.620 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:20:56.620 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:20:56.620 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:20:56.620 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:20:56.620 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.14781.0:217304 7.9 7:9b0d0ed4:::benchmark_data_smithi183_84160_object76923:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e130)", 2024-09-10T19:20:56.620 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-10T19:14:55.887537+0000", 2024-09-10T19:20:56.620 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 360.70512247099998, 2024-09-10T19:20:56.620 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.99279506100000003, 2024-09-10T19:20:56.620 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-10T19:20:56.621 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T19:20:56.621 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-10T19:20:56.621 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.14781", 2024-09-10T19:20:56.621 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.183:0/2025776420", 2024-09-10T19:20:56.621 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 217304 2024-09-10T19:20:56.621 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:20:56.621 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-10T19:20:56.621 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:20:56.621 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-10T19:20:56.621 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:55.887537+0000", 2024-09-10T19:20:56.621 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:20:56.621 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:20:56.621 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:20:56.621 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-10T19:20:56.622 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:55.887537+0000", 2024-09-10T19:20:56.622 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:20:56.622 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:20:56.622 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:20:56.622 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-10T19:20:56.622 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:55.887537+0000", 2024-09-10T19:20:56.622 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4294967295.9999995 2024-09-10T19:20:56.622 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:20:56.622 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:20:56.622 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-10T19:20:56.622 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:55.887540+0000", 2024-09-10T19:20:56.622 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.36e-06 2024-09-10T19:20:56.622 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:20:56.622 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:20:56.622 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-10T19:20:56.623 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:55.887555+0000", 2024-09-10T19:20:56.623 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.4867e-05 2024-09-10T19:20:56.623 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:20:56.623 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:20:56.623 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-10T19:20:56.623 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:55.887559+0000", 2024-09-10T19:20:56.623 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.9600000000000002e-06 2024-09-10T19:20:56.623 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:20:56.623 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:20:56.623 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-10T19:20:56.623 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:55.887605+0000", 2024-09-10T19:20:56.623 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.6431000000000003e-05 2024-09-10T19:20:56.623 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:20:56.624 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:20:56.624 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for peered", 2024-09-10T19:20:56.624 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:55.887614+0000", 2024-09-10T19:20:56.624 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.3059999999999998e-06 2024-09-10T19:20:56.624 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:20:56.624 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:20:56.624 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-10T19:20:56.624 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:56.868048+0000", 2024-09-10T19:20:56.624 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.98043477099999998 2024-09-10T19:20:56.624 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:20:56.624 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:20:56.624 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-10T19:20:56.624 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:56.868142+0000", 2024-09-10T19:20:56.624 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.3259999999999998e-05 2024-09-10T19:20:56.625 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:20:56.625 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:20:56.625 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 4,7", 2024-09-10T19:20:56.625 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:56.868239+0000", 2024-09-10T19:20:56.625 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.7529999999999996e-05 2024-09-10T19:20:56.625 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:20:56.625 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:20:56.625 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:20:56.625 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:56.874910+0000", 2024-09-10T19:20:56.625 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0066706589999999998 2024-09-10T19:20:56.625 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:20:56.625 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:20:56.625 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:20:56.626 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:56.874945+0000", 2024-09-10T19:20:56.626 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.4684999999999997e-05 2024-09-10T19:20:56.626 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:20:56.626 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:20:56.626 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-10T19:20:56.626 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:56.880227+0000", 2024-09-10T19:20:56.626 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0052828099999999998 2024-09-10T19:20:56.626 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:20:56.626 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:20:56.626 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-10T19:20:56.626 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:56.880274+0000", 2024-09-10T19:20:56.626 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.7033000000000002e-05 2024-09-10T19:20:56.627 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:20:56.627 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:20:56.627 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-10T19:20:56.627 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:56.880332+0000", 2024-09-10T19:20:56.627 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.7840000000000002e-05 2024-09-10T19:20:56.627 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:20:56.627 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:20:56.627 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:20:56.627 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:20:56.627 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:20:56.627 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.14781.0:217309 7.9 7:91e6a849:::benchmark_data_smithi183_84160_object76928:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e130)", 2024-09-10T19:20:56.627 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-10T19:14:55.894099+0000", 2024-09-10T19:20:56.628 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 360.69856028800001, 2024-09-10T19:20:56.628 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.98634969800000005, 2024-09-10T19:20:56.628 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-10T19:20:56.628 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T19:20:56.628 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-10T19:20:56.628 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.14781", 2024-09-10T19:20:56.628 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.183:0/2025776420", 2024-09-10T19:20:56.628 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 217309 2024-09-10T19:20:56.628 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:20:56.628 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-10T19:20:56.628 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:20:56.628 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-10T19:20:56.629 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:55.894099+0000", 2024-09-10T19:20:56.629 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:20:56.629 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:20:56.629 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:20:56.629 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-10T19:20:56.629 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:55.894099+0000", 2024-09-10T19:20:56.629 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:20:56.629 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:20:56.629 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:20:56.629 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-10T19:20:56.629 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:55.894098+0000", 2024-09-10T19:20:56.629 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4294967295.999999 2024-09-10T19:20:56.629 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:20:56.630 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:20:56.630 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-10T19:20:56.630 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:55.894104+0000", 2024-09-10T19:20:56.630 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.0730000000000001e-06 2024-09-10T19:20:56.630 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:20:56.630 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:20:56.630 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-10T19:20:56.630 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:55.894120+0000", 2024-09-10T19:20:56.630 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.5389000000000001e-05 2024-09-10T19:20:56.630 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:20:56.630 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:20:56.630 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-10T19:20:56.630 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:55.894127+0000", 2024-09-10T19:20:56.631 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.08e-06 2024-09-10T19:20:56.631 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:20:56.631 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:20:56.631 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-10T19:20:56.631 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:55.894206+0000", 2024-09-10T19:20:56.631 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.8653999999999996e-05 2024-09-10T19:20:56.631 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:20:56.631 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:20:56.631 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for peered", 2024-09-10T19:20:56.631 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:55.894217+0000", 2024-09-10T19:20:56.631 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.1725e-05 2024-09-10T19:20:56.631 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:20:56.631 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:20:56.631 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-10T19:20:56.632 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:56.870602+0000", 2024-09-10T19:20:56.632 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.97638430700000001 2024-09-10T19:20:56.632 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:20:56.632 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:20:56.632 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-10T19:20:56.632 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:56.870692+0000", 2024-09-10T19:20:56.632 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.9975000000000005e-05 2024-09-10T19:20:56.632 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:20:56.632 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:20:56.632 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 4,7", 2024-09-10T19:20:56.632 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:56.870802+0000", 2024-09-10T19:20:56.632 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000110184 2024-09-10T19:20:56.632 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:20:56.633 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:20:56.633 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:20:56.633 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:56.874979+0000", 2024-09-10T19:20:56.633 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0041769219999999996 2024-09-10T19:20:56.633 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:20:56.633 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:20:56.633 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:20:56.633 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:56.880378+0000", 2024-09-10T19:20:56.633 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0053997450000000001 2024-09-10T19:20:56.633 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:20:56.633 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:20:56.633 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-10T19:20:56.633 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:56.880384+0000", 2024-09-10T19:20:56.634 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.4949999999999999e-06 2024-09-10T19:20:56.634 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:20:56.634 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:20:56.634 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-10T19:20:56.634 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:56.880420+0000", 2024-09-10T19:20:56.634 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.6369999999999999e-05 2024-09-10T19:20:56.634 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:20:56.634 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:20:56.634 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-10T19:20:56.634 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:56.880449+0000", 2024-09-10T19:20:56.634 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.8756999999999999e-05 2024-09-10T19:20:56.634 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:20:56.634 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:20:56.634 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:20:56.635 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:20:56.635 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:20:56.635 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.14781.0:217349 7.9 7:919a8f2c:::benchmark_data_smithi183_84160_object76968:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e130)", 2024-09-10T19:20:56.635 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-10T19:14:55.925273+0000", 2024-09-10T19:20:56.635 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 360.66738695800001, 2024-09-10T19:20:56.635 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.95531499399999997, 2024-09-10T19:20:56.635 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-10T19:20:56.635 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T19:20:56.635 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-10T19:20:56.635 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.14781", 2024-09-10T19:20:56.635 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.183:0/2025776420", 2024-09-10T19:20:56.635 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 217349 2024-09-10T19:20:56.635 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:20:56.636 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-10T19:20:56.636 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:20:56.636 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-10T19:20:56.636 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:55.925273+0000", 2024-09-10T19:20:56.636 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:20:56.636 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:20:56.636 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:20:56.636 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-10T19:20:56.636 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:55.925273+0000", 2024-09-10T19:20:56.636 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:20:56.636 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:20:56.636 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:20:56.636 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-10T19:20:56.637 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:55.925271+0000", 2024-09-10T19:20:56.637 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4294967295.9999986 2024-09-10T19:20:56.637 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:20:56.637 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:20:56.637 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-10T19:20:56.637 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:55.925277+0000", 2024-09-10T19:20:56.637 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.7239999999999997e-06 2024-09-10T19:20:56.637 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:20:56.637 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:20:56.637 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-10T19:20:56.637 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:55.925289+0000", 2024-09-10T19:20:56.637 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.1409e-05 2024-09-10T19:20:56.637 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:20:56.638 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:20:56.638 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-10T19:20:56.638 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:55.925293+0000", 2024-09-10T19:20:56.638 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.9650000000000002e-06 2024-09-10T19:20:56.638 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:20:56.638 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:20:56.638 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-10T19:20:56.638 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:55.925329+0000", 2024-09-10T19:20:56.638 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.6730000000000002e-05 2024-09-10T19:20:56.638 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:20:56.638 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:20:56.638 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for peered", 2024-09-10T19:20:56.638 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:55.925335+0000", 2024-09-10T19:20:56.638 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.2270000000000001e-06 2024-09-10T19:20:56.638 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:20:56.639 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:20:56.639 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-10T19:20:56.639 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:56.872890+0000", 2024-09-10T19:20:56.639 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.94755518599999999 2024-09-10T19:20:56.639 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:20:56.639 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:20:56.639 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-10T19:20:56.639 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:56.872961+0000", 2024-09-10T19:20:56.639 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.0977999999999997e-05 2024-09-10T19:20:56.639 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:20:56.639 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:20:56.639 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 4,7", 2024-09-10T19:20:56.639 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:56.873024+0000", 2024-09-10T19:20:56.639 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.3261000000000003e-05 2024-09-10T19:20:56.640 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:20:56.640 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:20:56.640 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-10T19:20:56.640 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:56.880455+0000", 2024-09-10T19:20:56.640 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0074314000000000003 2024-09-10T19:20:56.640 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:20:56.640 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:20:56.640 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:20:56.640 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:56.880494+0000", 2024-09-10T19:20:56.640 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.8763000000000001e-05 2024-09-10T19:20:56.640 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:20:56.640 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:20:56.640 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:20:56.641 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:56.880520+0000", 2024-09-10T19:20:56.641 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.6026000000000001e-05 2024-09-10T19:20:56.641 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:20:56.641 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:20:56.641 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-10T19:20:56.641 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:56.880560+0000", 2024-09-10T19:20:56.641 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.0172000000000001e-05 2024-09-10T19:20:56.641 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:20:56.641 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:20:56.641 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-10T19:20:56.641 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:56.880588+0000", 2024-09-10T19:20:56.641 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.7387999999999999e-05 2024-09-10T19:20:56.641 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:20:56.642 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:20:56.642 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:20:56.642 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:20:56.642 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:20:56.642 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.14781.0:217391 7.9 7:986b6775:::benchmark_data_smithi183_84160_object77010:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e130)", 2024-09-10T19:20:56.642 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-10T19:14:55.967271+0000", 2024-09-10T19:20:56.642 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 360.62538877100002, 2024-09-10T19:20:56.642 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.91717478299999999, 2024-09-10T19:20:56.642 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-10T19:20:56.642 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T19:20:56.642 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-10T19:20:56.643 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.14781", 2024-09-10T19:20:56.643 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.183:0/2025776420", 2024-09-10T19:20:56.644 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 217391 2024-09-10T19:20:56.644 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:20:56.644 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-10T19:20:56.644 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:20:56.644 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-10T19:20:56.644 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:55.967271+0000", 2024-09-10T19:20:56.644 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:20:56.644 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:20:56.644 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:20:56.644 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-10T19:20:56.644 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:55.967271+0000", 2024-09-10T19:20:56.644 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:20:56.644 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:20:56.645 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:20:56.645 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-10T19:20:56.645 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:55.967269+0000", 2024-09-10T19:20:56.645 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4294967295.9999986 2024-09-10T19:20:56.645 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:20:56.645 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:20:56.645 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-10T19:20:56.645 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:55.967277+0000", 2024-09-10T19:20:56.645 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.1010000000000003e-06 2024-09-10T19:20:56.645 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:20:56.645 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:20:56.645 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-10T19:20:56.645 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:55.967292+0000", 2024-09-10T19:20:56.645 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.4996e-05 2024-09-10T19:20:56.646 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:20:56.646 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:20:56.646 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-10T19:20:56.646 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:55.967298+0000", 2024-09-10T19:20:56.646 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.9480000000000002e-06 2024-09-10T19:20:56.646 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:20:56.646 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:20:56.646 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-10T19:20:56.646 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:55.967346+0000", 2024-09-10T19:20:56.646 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.7253000000000003e-05 2024-09-10T19:20:56.646 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:20:56.647 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:20:56.647 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for peered", 2024-09-10T19:20:56.647 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:55.967355+0000", 2024-09-10T19:20:56.647 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.2620000000000004e-06 2024-09-10T19:20:56.647 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:20:56.647 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:20:56.647 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-10T19:20:56.647 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:56.875030+0000", 2024-09-10T19:20:56.647 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.90767522199999995 2024-09-10T19:20:56.647 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:20:56.648 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:20:56.648 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-10T19:20:56.648 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:56.875134+0000", 2024-09-10T19:20:56.648 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000103713 2024-09-10T19:20:56.648 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:20:56.648 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:20:56.648 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 4,7", 2024-09-10T19:20:56.648 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:56.875241+0000", 2024-09-10T19:20:56.648 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00010698 2024-09-10T19:20:56.648 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:20:56.648 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:20:56.648 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-10T19:20:56.648 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:56.880460+0000", 2024-09-10T19:20:56.649 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0052190919999999998 2024-09-10T19:20:56.649 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:20:56.649 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:20:56.649 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:20:56.649 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:56.880960+0000", 2024-09-10T19:20:56.649 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000499848 2024-09-10T19:20:56.649 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:20:56.649 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:20:56.649 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:20:56.649 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:56.884349+0000", 2024-09-10T19:20:56.649 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0033893809999999999 2024-09-10T19:20:56.649 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:20:56.649 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:20:56.650 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-10T19:20:56.650 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:56.884407+0000", 2024-09-10T19:20:56.650 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.7744000000000001e-05 2024-09-10T19:20:56.650 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:20:56.650 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:20:56.650 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-10T19:20:56.650 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:56.884446+0000", 2024-09-10T19:20:56.650 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.8729000000000001e-05 2024-09-10T19:20:56.650 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:20:56.650 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:20:56.650 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:20:56.650 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:20:56.650 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:20:56.650 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.14781.0:217402 7.9 7:954b255b:::benchmark_data_smithi183_84160_object77021:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e130)", 2024-09-10T19:20:56.651 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-10T19:14:55.983665+0000", 2024-09-10T19:20:56.651 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 360.60899438400003, 2024-09-10T19:20:56.651 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.90088200900000004, 2024-09-10T19:20:56.651 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-10T19:20:56.651 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T19:20:56.651 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-10T19:20:56.651 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.14781", 2024-09-10T19:20:56.651 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.183:0/2025776420", 2024-09-10T19:20:56.651 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 217402 2024-09-10T19:20:56.651 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:20:56.651 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-10T19:20:56.651 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:20:56.651 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-10T19:20:56.651 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:55.983665+0000", 2024-09-10T19:20:56.652 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:20:56.652 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:20:56.652 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:20:56.652 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-10T19:20:56.652 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:55.983665+0000", 2024-09-10T19:20:56.652 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:20:56.652 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:20:56.652 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:20:56.652 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-10T19:20:56.652 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:55.983664+0000", 2024-09-10T19:20:56.652 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4294967295.9999986 2024-09-10T19:20:56.652 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:20:56.653 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:20:56.653 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-10T19:20:56.653 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:55.983671+0000", 2024-09-10T19:20:56.653 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.8040000000000004e-06 2024-09-10T19:20:56.653 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:20:56.653 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:20:56.653 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-10T19:20:56.653 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:55.983686+0000", 2024-09-10T19:20:56.653 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.4951e-05 2024-09-10T19:20:56.653 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:20:56.653 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:20:56.653 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-10T19:20:56.653 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:55.983691+0000", 2024-09-10T19:20:56.653 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.383e-06 2024-09-10T19:20:56.654 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:20:56.654 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:20:56.654 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-10T19:20:56.654 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:55.983734+0000", 2024-09-10T19:20:56.654 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.2814e-05 2024-09-10T19:20:56.654 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:20:56.654 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:20:56.654 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for peered", 2024-09-10T19:20:56.654 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:55.983741+0000", 2024-09-10T19:20:56.654 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.1409999999999999e-06 2024-09-10T19:20:56.654 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:20:56.654 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:20:56.654 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-10T19:20:56.655 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:56.877644+0000", 2024-09-10T19:20:56.655 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.89390298899999998 2024-09-10T19:20:56.655 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:20:56.655 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:20:56.655 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-10T19:20:56.655 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:56.877786+0000", 2024-09-10T19:20:56.655 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000141521 2024-09-10T19:20:56.655 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:20:56.655 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:20:56.655 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 4,7", 2024-09-10T19:20:56.655 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:56.877905+0000", 2024-09-10T19:20:56.655 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000119321 2024-09-10T19:20:56.655 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:20:56.656 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:20:56.656 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-10T19:20:56.656 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:56.880464+0000", 2024-09-10T19:20:56.656 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0025592520000000001 2024-09-10T19:20:56.656 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:20:56.656 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:20:56.656 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:20:56.656 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:56.883564+0000", 2024-09-10T19:20:56.656 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0030999959999999998 2024-09-10T19:20:56.656 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:20:56.656 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:20:56.656 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:20:56.656 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:56.884481+0000", 2024-09-10T19:20:56.657 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00091682799999999996 2024-09-10T19:20:56.657 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:20:56.657 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:20:56.657 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-10T19:20:56.657 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:56.884516+0000", 2024-09-10T19:20:56.657 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.4566000000000001e-05 2024-09-10T19:20:56.657 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:20:56.657 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:20:56.657 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-10T19:20:56.657 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:14:56.884547+0000", 2024-09-10T19:20:56.657 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.1665999999999998e-05 2024-09-10T19:20:56.657 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:20:56.657 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:20:56.657 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:20:56.658 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:20:56.658 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:20:56.658 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.15051.0:29527 8.1 8:814c39ba:::benchmark_data_smithi183_114924_object29526:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160)", 2024-09-10T19:20:56.658 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-10T19:16:35.788777+0000", 2024-09-10T19:20:56.658 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 260.80388250099998, 2024-09-10T19:20:56.658 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.16844375, 2024-09-10T19:20:56.658 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-10T19:20:56.658 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T19:20:56.658 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-10T19:20:56.658 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.15051", 2024-09-10T19:20:56.658 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.183:0/3632028672", 2024-09-10T19:20:56.658 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 29527 2024-09-10T19:20:56.658 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:20:56.659 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-10T19:20:56.659 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:20:56.659 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-10T19:20:56.659 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:35.788777+0000", 2024-09-10T19:20:56.659 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:20:56.659 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:20:56.659 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:20:56.659 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-10T19:20:56.659 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:35.788777+0000", 2024-09-10T19:20:56.659 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:20:56.659 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:20:56.659 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:20:56.659 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-10T19:20:56.660 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:35.788776+0000", 2024-09-10T19:20:56.660 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4294967295.9999986 2024-09-10T19:20:56.660 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:20:56.660 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:20:56.660 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-10T19:20:56.660 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:35.788803+0000", 2024-09-10T19:20:56.660 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.6832e-05 2024-09-10T19:20:56.660 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:20:56.660 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:20:56.660 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-10T19:20:56.660 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:35.788814+0000", 2024-09-10T19:20:56.660 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.1636e-05 2024-09-10T19:20:56.660 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:20:56.661 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:20:56.661 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-10T19:20:56.661 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:35.788820+0000", 2024-09-10T19:20:56.661 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.0399999999999998e-06 2024-09-10T19:20:56.661 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:20:56.661 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:20:56.661 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-10T19:20:56.661 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:35.788917+0000", 2024-09-10T19:20:56.661 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.6311000000000007e-05 2024-09-10T19:20:56.661 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:20:56.661 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:20:56.661 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-10T19:20:56.661 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:35.789054+0000", 2024-09-10T19:20:56.662 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000137813 2024-09-10T19:20:56.662 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:20:56.662 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:20:56.662 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 3,5", 2024-09-10T19:20:56.662 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:35.789185+0000", 2024-09-10T19:20:56.662 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000130884 2024-09-10T19:20:56.662 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:20:56.662 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:20:56.662 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-10T19:20:56.662 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:35.791144+0000", 2024-09-10T19:20:56.662 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.001958404 2024-09-10T19:20:56.662 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:20:56.662 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:20:56.662 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:20:56.663 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:35.792387+0000", 2024-09-10T19:20:56.663 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0012435090000000001 2024-09-10T19:20:56.663 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:20:56.663 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:20:56.663 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:20:56.663 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:35.957154+0000", 2024-09-10T19:20:56.663 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.164766564 2024-09-10T19:20:56.663 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:20:56.663 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:20:56.663 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-10T19:20:56.663 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:35.957194+0000", 2024-09-10T19:20:56.663 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.9929999999999999e-05 2024-09-10T19:20:56.663 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:20:56.663 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:20:56.664 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-10T19:20:56.664 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:35.957221+0000", 2024-09-10T19:20:56.664 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.7178e-05 2024-09-10T19:20:56.664 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:20:56.664 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:20:56.664 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:20:56.664 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:20:56.664 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:20:56.664 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.15051.0:29531 8.1 8:8865ba01:::benchmark_data_smithi183_114924_object29530:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160)", 2024-09-10T19:20:56.664 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-10T19:16:35.789769+0000", 2024-09-10T19:20:56.664 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 260.80289065900001, 2024-09-10T19:20:56.665 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.167584333, 2024-09-10T19:20:56.665 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-10T19:20:56.665 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T19:20:56.665 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-10T19:20:56.665 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.15051", 2024-09-10T19:20:56.665 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.183:0/3632028672", 2024-09-10T19:20:56.665 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 29531 2024-09-10T19:20:56.665 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:20:56.665 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-10T19:20:56.665 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:20:56.665 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-10T19:20:56.665 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:35.789769+0000", 2024-09-10T19:20:56.665 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:20:56.666 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:20:56.666 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:20:56.666 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-10T19:20:56.666 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:35.789769+0000", 2024-09-10T19:20:56.666 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:20:56.666 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:20:56.666 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:20:56.666 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-10T19:20:56.666 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:35.789768+0000", 2024-09-10T19:20:56.666 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4294967295.999999 2024-09-10T19:20:56.666 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:20:56.666 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:20:56.666 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-10T19:20:56.667 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:35.789790+0000", 2024-09-10T19:20:56.667 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.2348999999999999e-05 2024-09-10T19:20:56.667 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:20:56.667 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:20:56.667 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-10T19:20:56.667 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:35.789803+0000", 2024-09-10T19:20:56.667 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.2656999999999999e-05 2024-09-10T19:20:56.667 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:20:56.667 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:20:56.667 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-10T19:20:56.667 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:35.789809+0000", 2024-09-10T19:20:56.667 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.6210000000000001e-06 2024-09-10T19:20:56.667 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:20:56.668 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:20:56.668 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-10T19:20:56.668 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:35.790099+0000", 2024-09-10T19:20:56.668 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00029001100000000001 2024-09-10T19:20:56.668 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:20:56.668 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:20:56.668 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-10T19:20:56.668 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:35.790214+0000", 2024-09-10T19:20:56.668 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00011574900000000001 2024-09-10T19:20:56.668 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:20:56.668 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:20:56.668 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 3,5", 2024-09-10T19:20:56.668 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:35.790298+0000", 2024-09-10T19:20:56.668 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.3434000000000001e-05 2024-09-10T19:20:56.669 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:20:56.669 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:20:56.669 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:20:56.669 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:35.792439+0000", 2024-09-10T19:20:56.669 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0021409939999999998 2024-09-10T19:20:56.669 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:20:56.669 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:20:56.669 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-10T19:20:56.669 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:35.792447+0000", 2024-09-10T19:20:56.669 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.0520000000000002e-06 2024-09-10T19:20:56.669 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:20:56.669 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:20:56.669 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:20:56.670 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:35.957265+0000", 2024-09-10T19:20:56.670 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.16481826599999999 2024-09-10T19:20:56.670 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:20:56.670 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:20:56.670 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-10T19:20:56.670 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:35.957315+0000", 2024-09-10T19:20:56.670 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.9421e-05 2024-09-10T19:20:56.670 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:20:56.670 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:20:56.670 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-10T19:20:56.670 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:35.957353+0000", 2024-09-10T19:20:56.670 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.8716e-05 2024-09-10T19:20:56.670 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:20:56.670 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:20:56.671 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:20:56.671 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:20:56.671 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:20:56.671 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.15051.0:29534 8.1 8:8b12dbf9:::benchmark_data_smithi183_114924_object29533:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160)", 2024-09-10T19:20:56.671 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-10T19:16:35.790944+0000", 2024-09-10T19:20:56.671 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 260.80171595899998, 2024-09-10T19:20:56.671 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.17016989699999999, 2024-09-10T19:20:56.671 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-10T19:20:56.671 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T19:20:56.671 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-10T19:20:56.671 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.15051", 2024-09-10T19:20:56.671 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.183:0/3632028672", 2024-09-10T19:20:56.671 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 29534 2024-09-10T19:20:56.671 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:20:56.672 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-10T19:20:56.672 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:20:56.672 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-10T19:20:56.672 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:35.790944+0000", 2024-09-10T19:20:56.672 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:20:56.672 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:20:56.672 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:20:56.672 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-10T19:20:56.672 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:35.790944+0000", 2024-09-10T19:20:56.672 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:20:56.672 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:20:56.672 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:20:56.672 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-10T19:20:56.673 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:35.790943+0000", 2024-09-10T19:20:56.673 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4294967295.999999 2024-09-10T19:20:56.673 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:20:56.673 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:20:56.673 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-10T19:20:56.673 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:35.790961+0000", 2024-09-10T19:20:56.673 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.7615e-05 2024-09-10T19:20:56.673 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:20:56.673 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:20:56.673 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-10T19:20:56.673 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:35.790970+0000", 2024-09-10T19:20:56.673 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.7380000000000006e-06 2024-09-10T19:20:56.673 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:20:56.673 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:20:56.674 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-10T19:20:56.674 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:35.790978+0000", 2024-09-10T19:20:56.674 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.3679999999999996e-06 2024-09-10T19:20:56.674 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:20:56.674 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:20:56.674 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-10T19:20:56.674 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:35.791302+0000", 2024-09-10T19:20:56.674 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00032406600000000002 2024-09-10T19:20:56.674 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:20:56.674 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:20:56.674 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-10T19:20:56.674 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:35.791422+0000", 2024-09-10T19:20:56.674 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00012031099999999999 2024-09-10T19:20:56.674 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:20:56.675 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:20:56.675 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 3,5", 2024-09-10T19:20:56.675 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:35.791554+0000", 2024-09-10T19:20:56.675 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00013224499999999999 2024-09-10T19:20:56.675 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:20:56.675 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:20:56.675 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-10T19:20:56.675 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:35.793309+0000", 2024-09-10T19:20:56.675 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.001754535 2024-09-10T19:20:56.675 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:20:56.675 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:20:56.675 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:20:56.675 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:35.793465+0000", 2024-09-10T19:20:56.675 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000156136 2024-09-10T19:20:56.676 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:20:56.676 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:20:56.676 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:20:56.676 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:35.961049+0000", 2024-09-10T19:20:56.676 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.16758373800000001 2024-09-10T19:20:56.676 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:20:56.676 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:20:56.676 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-10T19:20:56.676 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:35.961083+0000", 2024-09-10T19:20:56.676 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.3881999999999998e-05 2024-09-10T19:20:56.676 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:20:56.676 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:20:56.676 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-10T19:20:56.676 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:35.961114+0000", 2024-09-10T19:20:56.677 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.1025000000000002e-05 2024-09-10T19:20:56.677 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:20:56.677 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:20:56.677 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:20:56.677 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:20:56.677 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:20:56.677 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.15051.0:29535 8.1 8:8d29f2f7:::benchmark_data_smithi183_114924_object29534:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160)", 2024-09-10T19:20:56.677 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-10T19:16:35.791450+0000", 2024-09-10T19:20:56.677 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 260.80120956299999, 2024-09-10T19:20:56.677 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.16979755599999999, 2024-09-10T19:20:56.677 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-10T19:20:56.677 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T19:20:56.677 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-10T19:20:56.677 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.15051", 2024-09-10T19:20:56.678 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.183:0/3632028672", 2024-09-10T19:20:56.678 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 29535 2024-09-10T19:20:56.678 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:20:56.678 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-10T19:20:56.678 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:20:56.678 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-10T19:20:56.678 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:35.791450+0000", 2024-09-10T19:20:56.678 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:20:56.678 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:20:56.678 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:20:56.678 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-10T19:20:56.678 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:35.791450+0000", 2024-09-10T19:20:56.678 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:20:56.678 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:20:56.679 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:20:56.679 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-10T19:20:56.679 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:35.791449+0000", 2024-09-10T19:20:56.679 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4294967295.999999 2024-09-10T19:20:56.679 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:20:56.679 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:20:56.679 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-10T19:20:56.679 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:35.791470+0000", 2024-09-10T19:20:56.679 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.0789e-05 2024-09-10T19:20:56.679 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:20:56.679 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:20:56.679 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-10T19:20:56.679 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:35.791483+0000", 2024-09-10T19:20:56.679 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.2942000000000001e-05 2024-09-10T19:20:56.680 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:20:56.680 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:20:56.680 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-10T19:20:56.680 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:35.791489+0000", 2024-09-10T19:20:56.680 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.6930000000000004e-06 2024-09-10T19:20:56.680 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:20:56.680 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:20:56.680 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-10T19:20:56.680 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:35.792481+0000", 2024-09-10T19:20:56.680 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0009921489999999999 2024-09-10T19:20:56.680 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:20:56.680 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:20:56.680 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-10T19:20:56.680 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:35.792570+0000", 2024-09-10T19:20:56.681 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.9417000000000001e-05 2024-09-10T19:20:56.681 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:20:56.681 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:20:56.681 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 3,5", 2024-09-10T19:20:56.681 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:35.792655+0000", 2024-09-10T19:20:56.681 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.4690000000000004e-05 2024-09-10T19:20:56.681 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:20:56.681 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:20:56.681 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-10T19:20:56.681 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:35.793601+0000", 2024-09-10T19:20:56.681 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00094609200000000005 2024-09-10T19:20:56.681 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:20:56.682 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:20:56.682 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:20:56.682 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:35.794663+0000", 2024-09-10T19:20:56.682 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.001061956 2024-09-10T19:20:56.682 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:20:56.682 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:20:56.682 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:20:56.682 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:35.961162+0000", 2024-09-10T19:20:56.682 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.16649857000000001 2024-09-10T19:20:56.682 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:20:56.682 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:20:56.682 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-10T19:20:56.682 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:35.961211+0000", 2024-09-10T19:20:56.682 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.9756000000000003e-05 2024-09-10T19:20:56.683 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:20:56.683 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:20:56.683 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-10T19:20:56.683 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:35.961248+0000", 2024-09-10T19:20:56.683 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.6319000000000002e-05 2024-09-10T19:20:56.683 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:20:56.683 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:20:56.683 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:20:56.683 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:20:56.683 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:20:56.683 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.15051.0:29540 8.1 8:866c485e:::benchmark_data_smithi183_114924_object29539:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160)", 2024-09-10T19:20:56.683 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-10T19:16:35.794502+0000", 2024-09-10T19:20:56.683 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 260.79815757300003, 2024-09-10T19:20:56.684 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.167192068, 2024-09-10T19:20:56.684 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-10T19:20:56.684 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T19:20:56.684 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-10T19:20:56.684 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.15051", 2024-09-10T19:20:56.684 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.183:0/3632028672", 2024-09-10T19:20:56.684 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 29540 2024-09-10T19:20:56.684 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:20:56.684 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-10T19:20:56.684 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:20:56.684 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-10T19:20:56.684 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:35.794502+0000", 2024-09-10T19:20:56.684 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:20:56.685 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:20:56.685 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:20:56.685 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-10T19:20:56.685 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:35.794502+0000", 2024-09-10T19:20:56.685 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:20:56.685 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:20:56.685 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:20:56.685 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-10T19:20:56.685 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:35.794501+0000", 2024-09-10T19:20:56.685 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4294967295.999999 2024-09-10T19:20:56.685 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:20:56.685 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:20:56.685 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-10T19:20:56.686 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:35.794547+0000", 2024-09-10T19:20:56.686 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.6054999999999999e-05 2024-09-10T19:20:56.686 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:20:56.686 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:20:56.686 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-10T19:20:56.686 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:35.794558+0000", 2024-09-10T19:20:56.686 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.058e-05 2024-09-10T19:20:56.686 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:20:56.686 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:20:56.686 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-10T19:20:56.686 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:35.794563+0000", 2024-09-10T19:20:56.686 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.2800000000000003e-06 2024-09-10T19:20:56.686 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:20:56.686 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:20:56.687 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-10T19:20:56.687 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:35.794774+0000", 2024-09-10T19:20:56.687 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00021090200000000001 2024-09-10T19:20:56.687 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:20:56.687 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:20:56.687 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-10T19:20:56.687 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:35.794948+0000", 2024-09-10T19:20:56.687 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00017378599999999999 2024-09-10T19:20:56.687 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:20:56.687 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:20:56.687 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 3,5", 2024-09-10T19:20:56.687 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:35.795073+0000", 2024-09-10T19:20:56.687 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000125108 2024-09-10T19:20:56.687 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:20:56.688 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:20:56.688 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-10T19:20:56.688 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:35.796373+0000", 2024-09-10T19:20:56.688 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0012999470000000001 2024-09-10T19:20:56.688 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:20:56.688 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:20:56.688 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:20:56.688 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:35.796490+0000", 2024-09-10T19:20:56.688 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000116996 2024-09-10T19:20:56.688 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:20:56.688 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:20:56.688 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:20:56.688 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:35.961615+0000", 2024-09-10T19:20:56.689 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.16512479399999999 2024-09-10T19:20:56.689 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:20:56.689 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:20:56.689 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-10T19:20:56.689 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:35.961660+0000", 2024-09-10T19:20:56.689 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.4968e-05 2024-09-10T19:20:56.689 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:20:56.689 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:20:56.689 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-10T19:20:56.689 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:35.961694+0000", 2024-09-10T19:20:56.689 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.4547e-05 2024-09-10T19:20:56.689 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:20:56.689 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:20:56.690 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:20:56.690 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:20:56.690 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:20:56.690 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.15051.0:30113 8.1 8:8c6dc904:::benchmark_data_smithi183_114924_object30112:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160)", 2024-09-10T19:20:56.690 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-10T19:16:36.215395+0000", 2024-09-10T19:20:56.690 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 260.37726506000001, 2024-09-10T19:20:56.690 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.14840703999999999, 2024-09-10T19:20:56.690 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-10T19:20:56.690 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T19:20:56.690 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-10T19:20:56.690 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.15051", 2024-09-10T19:20:56.690 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.183:0/3632028672", 2024-09-10T19:20:56.690 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 30113 2024-09-10T19:20:56.690 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:20:56.691 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-10T19:20:56.691 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:20:56.691 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-10T19:20:56.691 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:36.215395+0000", 2024-09-10T19:20:56.691 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:20:56.691 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:20:56.691 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:20:56.691 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-10T19:20:56.691 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:36.215395+0000", 2024-09-10T19:20:56.691 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:20:56.691 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:20:56.691 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:20:56.691 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-10T19:20:56.691 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:36.215393+0000", 2024-09-10T19:20:56.692 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4294967295.9999986 2024-09-10T19:20:56.692 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:20:56.692 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:20:56.692 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-10T19:20:56.692 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:36.215420+0000", 2024-09-10T19:20:56.692 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.6577e-05 2024-09-10T19:20:56.692 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:20:56.692 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:20:56.692 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-10T19:20:56.692 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:36.215432+0000", 2024-09-10T19:20:56.692 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.207e-05 2024-09-10T19:20:56.692 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:20:56.692 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:20:56.693 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-10T19:20:56.693 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:36.215436+0000", 2024-09-10T19:20:56.693 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.5700000000000003e-06 2024-09-10T19:20:56.693 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:20:56.693 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:20:56.693 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-10T19:20:56.693 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:36.215465+0000", 2024-09-10T19:20:56.693 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.8479e-05 2024-09-10T19:20:56.693 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:20:56.693 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:20:56.693 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-10T19:20:56.693 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:36.215557+0000", 2024-09-10T19:20:56.693 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.1719999999999996e-05 2024-09-10T19:20:56.694 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:20:56.694 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:20:56.694 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 3,5", 2024-09-10T19:20:56.694 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:36.215637+0000", 2024-09-10T19:20:56.694 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.0350000000000001e-05 2024-09-10T19:20:56.694 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:20:56.694 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:20:56.694 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:20:56.694 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:36.216979+0000", 2024-09-10T19:20:56.694 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00134219 2024-09-10T19:20:56.694 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:20:56.694 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:20:56.694 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-10T19:20:56.694 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:36.217051+0000", 2024-09-10T19:20:56.695 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.1525000000000004e-05 2024-09-10T19:20:56.695 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:20:56.695 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:20:56.695 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:20:56.695 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:36.363718+0000", 2024-09-10T19:20:56.695 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.146667081 2024-09-10T19:20:56.695 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:20:56.695 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:20:56.695 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-10T19:20:56.695 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:36.363765+0000", 2024-09-10T19:20:56.695 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.6860999999999997e-05 2024-09-10T19:20:56.695 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:20:56.695 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:20:56.695 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-10T19:20:56.696 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:36.363802+0000", 2024-09-10T19:20:56.696 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.7051000000000002e-05 2024-09-10T19:20:56.696 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:20:56.696 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:20:56.696 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:20:56.696 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:20:56.696 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:20:56.696 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.15051.0:34258 8.e 8:7d5dcd72:::benchmark_data_smithi183_114924_object34257:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160)", 2024-09-10T19:20:56.696 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-10T19:16:39.104704+0000", 2024-09-10T19:20:56.696 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 257.487955306, 2024-09-10T19:20:56.696 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.147077508, 2024-09-10T19:20:56.696 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-10T19:20:56.696 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T19:20:56.696 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-10T19:20:56.697 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.15051", 2024-09-10T19:20:56.697 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.183:0/3632028672", 2024-09-10T19:20:56.697 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 34258 2024-09-10T19:20:56.697 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:20:56.697 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-10T19:20:56.697 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:20:56.697 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-10T19:20:56.697 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:39.104704+0000", 2024-09-10T19:20:56.697 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:20:56.697 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:20:56.697 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:20:56.697 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-10T19:20:56.697 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:39.104704+0000", 2024-09-10T19:20:56.697 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:20:56.698 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:20:56.698 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:20:56.698 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-10T19:20:56.698 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:39.104704+0000", 2024-09-10T19:20:56.698 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4294967295.999999 2024-09-10T19:20:56.698 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:20:56.698 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:20:56.698 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-10T19:20:56.698 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:39.104721+0000", 2024-09-10T19:20:56.698 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.7745999999999999e-05 2024-09-10T19:20:56.698 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:20:56.698 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:20:56.698 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-10T19:20:56.698 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:39.104730+0000", 2024-09-10T19:20:56.699 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.3990000000000004e-06 2024-09-10T19:20:56.699 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:20:56.699 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:20:56.699 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-10T19:20:56.699 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:39.104735+0000", 2024-09-10T19:20:56.699 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.9930000000000003e-06 2024-09-10T19:20:56.699 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:20:56.699 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:20:56.699 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-10T19:20:56.699 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:39.104771+0000", 2024-09-10T19:20:56.699 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.6269999999999997e-05 2024-09-10T19:20:56.699 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:20:56.699 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:20:56.700 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-10T19:20:56.700 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:39.104895+0000", 2024-09-10T19:20:56.700 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000123564 2024-09-10T19:20:56.700 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:20:56.700 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:20:56.700 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 4,7", 2024-09-10T19:20:56.700 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:20:56.701 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:39.104994+0000", 2024-09-10T19:20:56.702 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.9679000000000006e-05 2024-09-10T19:20:56.702 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:20:56.702 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:20:56.702 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-10T19:20:56.702 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:39.106267+0000", 2024-09-10T19:20:56.702 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0012725939999999999 2024-09-10T19:20:56.702 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:20:56.702 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:20:56.702 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:20:56.702 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:39.106472+0000", 2024-09-10T19:20:56.702 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00020555700000000001 2024-09-10T19:20:56.702 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:20:56.702 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:20:56.702 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:20:56.703 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:39.251697+0000", 2024-09-10T19:20:56.703 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.14522468999999999 2024-09-10T19:20:56.703 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:20:56.703 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:20:56.703 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-10T19:20:56.703 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:39.251746+0000", 2024-09-10T19:20:56.703 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.9193000000000001e-05 2024-09-10T19:20:56.703 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:20:56.703 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:20:56.703 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-10T19:20:56.703 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:39.251782+0000", 2024-09-10T19:20:56.703 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.5578999999999999e-05 2024-09-10T19:20:56.703 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:20:56.703 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:20:56.704 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:20:56.704 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:20:56.704 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:20:56.704 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.15051.0:34261 8.1 8:8c1222a2:::benchmark_data_smithi183_114924_object34260:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e160)", 2024-09-10T19:20:56.704 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-10T19:16:39.105593+0000", 2024-09-10T19:20:56.704 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 257.48706650499997, 2024-09-10T19:20:56.704 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.14712114700000001, 2024-09-10T19:20:56.704 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-10T19:20:56.704 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T19:20:56.704 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-10T19:20:56.704 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.15051", 2024-09-10T19:20:56.704 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.183:0/3632028672", 2024-09-10T19:20:56.704 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 34261 2024-09-10T19:20:56.705 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:20:56.705 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-10T19:20:56.705 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:20:56.705 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-10T19:20:56.705 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:39.105593+0000", 2024-09-10T19:20:56.705 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:20:56.705 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:20:56.705 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:20:56.705 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-10T19:20:56.705 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:39.105593+0000", 2024-09-10T19:20:56.705 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:20:56.705 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:20:56.705 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:20:56.706 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-10T19:20:56.706 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:39.105592+0000", 2024-09-10T19:20:56.706 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4294967295.999999 2024-09-10T19:20:56.706 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:20:56.706 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:20:56.706 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-10T19:20:56.706 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:39.105612+0000", 2024-09-10T19:20:56.706 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.9910000000000001e-05 2024-09-10T19:20:56.706 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:20:56.706 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:20:56.706 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-10T19:20:56.706 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:39.105621+0000", 2024-09-10T19:20:56.706 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.9879999999999993e-06 2024-09-10T19:20:56.706 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:20:56.707 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:20:56.707 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-10T19:20:56.707 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:39.105626+0000", 2024-09-10T19:20:56.707 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.0459999999999996e-06 2024-09-10T19:20:56.707 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:20:56.707 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:20:56.707 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-10T19:20:56.707 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:39.105655+0000", 2024-09-10T19:20:56.707 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.8192999999999999e-05 2024-09-10T19:20:56.707 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:20:56.707 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:20:56.707 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-10T19:20:56.707 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:39.105732+0000", 2024-09-10T19:20:56.708 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.7727999999999998e-05 2024-09-10T19:20:56.708 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:20:56.708 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:20:56.708 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 3,5", 2024-09-10T19:20:56.708 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:39.105829+0000", 2024-09-10T19:20:56.708 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.7070000000000004e-05 2024-09-10T19:20:56.708 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:20:56.708 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:20:56.708 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-10T19:20:56.708 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:39.106970+0000", 2024-09-10T19:20:56.708 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.001140444 2024-09-10T19:20:56.708 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:20:56.708 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:20:56.709 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:20:56.709 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:39.107167+0000", 2024-09-10T19:20:56.709 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00019702899999999999 2024-09-10T19:20:56.709 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:20:56.709 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:20:56.709 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:20:56.709 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:39.252640+0000", 2024-09-10T19:20:56.709 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.145472825 2024-09-10T19:20:56.709 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:20:56.709 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:20:56.709 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-10T19:20:56.709 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:39.252682+0000", 2024-09-10T19:20:56.709 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.2313e-05 2024-09-10T19:20:56.710 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:20:56.710 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:20:56.710 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-10T19:20:56.710 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:39.252714+0000", 2024-09-10T19:20:56.710 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.2379999999999998e-05 2024-09-10T19:20:56.710 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:20:56.710 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:20:56.710 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:20:56.710 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:20:56.710 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:20:56.710 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.15051.0:93071 8.1 8:855a423f:::benchmark_data_smithi183_114924_object93070:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e177)", 2024-09-10T19:20:56.710 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-10T19:17:26.015144+0000", 2024-09-10T19:20:56.710 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 210.57751597699999, 2024-09-10T19:20:56.710 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.94587861100000004, 2024-09-10T19:20:56.710 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-10T19:20:56.711 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "reached pg", 2024-09-10T19:20:56.711 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-10T19:20:56.711 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.15051", 2024-09-10T19:20:56.711 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.183:0/3632028672", 2024-09-10T19:20:56.711 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 93071 2024-09-10T19:20:56.711 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:20:56.711 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-10T19:20:56.711 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:20:56.711 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-10T19:20:56.711 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:17:26.015144+0000", 2024-09-10T19:20:56.711 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:20:56.711 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:20:56.711 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:20:56.711 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-10T19:20:56.712 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:17:26.015144+0000", 2024-09-10T19:20:56.712 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:20:56.712 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:20:56.712 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:20:56.712 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-10T19:20:56.712 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:17:26.015142+0000", 2024-09-10T19:20:56.712 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4294967295.9999986 2024-09-10T19:20:56.712 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:20:56.712 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:20:56.712 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-10T19:20:56.712 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:17:26.015166+0000", 2024-09-10T19:20:56.712 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.4253999999999999e-05 2024-09-10T19:20:56.712 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:20:56.712 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:20:56.713 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-10T19:20:56.713 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:17:26.015185+0000", 2024-09-10T19:20:56.713 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.9139000000000001e-05 2024-09-10T19:20:56.713 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:20:56.713 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:20:56.713 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-10T19:20:56.713 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:17:26.015193+0000", 2024-09-10T19:20:56.713 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.2180000000000002e-06 2024-09-10T19:20:56.713 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:20:56.713 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:20:56.713 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-10T19:20:56.713 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:17:26.018617+0000", 2024-09-10T19:20:56.713 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0034239180000000002 2024-09-10T19:20:56.714 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:20:56.714 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:20:56.714 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for peered", 2024-09-10T19:20:56.714 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:17:26.018625+0000", 2024-09-10T19:20:56.714 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.5809999999999997e-06 2024-09-10T19:20:56.714 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:20:56.714 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:20:56.714 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-10T19:20:56.714 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:17:26.960946+0000", 2024-09-10T19:20:56.714 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.94232036600000002 2024-09-10T19:20:56.714 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:20:56.714 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:20:56.714 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-10T19:20:56.714 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:17:26.961022+0000", 2024-09-10T19:20:56.714 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.6773000000000005e-05 2024-09-10T19:20:56.715 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:20:56.715 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:20:56.715 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:20:56.715 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:20:56.715 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:20:56.715 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.15051.0:93083 8.6 8:66aaa558:::benchmark_data_smithi183_114924_object93082:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e177)", 2024-09-10T19:20:56.715 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-10T19:17:26.015201+0000", 2024-09-10T19:20:56.715 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 210.57745896200001, 2024-09-10T19:20:56.715 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.94040405000000005, 2024-09-10T19:20:56.715 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-10T19:20:56.715 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "reached pg", 2024-09-10T19:20:56.715 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-10T19:20:56.715 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.15051", 2024-09-10T19:20:56.715 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.183:0/3632028672", 2024-09-10T19:20:56.716 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 93083 2024-09-10T19:20:56.716 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:20:56.716 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-10T19:20:56.716 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:20:56.716 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-10T19:20:56.716 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:17:26.015201+0000", 2024-09-10T19:20:56.716 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:20:56.716 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:20:56.716 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:20:56.716 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-10T19:20:56.716 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:17:26.015201+0000", 2024-09-10T19:20:56.716 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:20:56.716 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:20:56.716 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:20:56.717 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-10T19:20:56.717 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:17:26.015200+0000", 2024-09-10T19:20:56.717 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4294967295.9999995 2024-09-10T19:20:56.717 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:20:56.717 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:20:56.717 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-10T19:20:56.717 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:17:26.015220+0000", 2024-09-10T19:20:56.717 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.9629000000000001e-05 2024-09-10T19:20:56.717 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:20:56.717 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:20:56.717 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-10T19:20:56.717 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:17:26.015227+0000", 2024-09-10T19:20:56.717 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.7839999999999998e-06 2024-09-10T19:20:56.718 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:20:56.718 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:20:56.718 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-10T19:20:56.718 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:17:26.015231+0000", 2024-09-10T19:20:56.718 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.6199999999999998e-06 2024-09-10T19:20:56.718 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:20:56.718 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:20:56.718 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-10T19:20:56.718 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:17:26.015389+0000", 2024-09-10T19:20:56.718 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00015741099999999999 2024-09-10T19:20:56.718 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:20:56.718 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:20:56.718 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for peered", 2024-09-10T19:20:56.718 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:17:26.015397+0000", 2024-09-10T19:20:56.718 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.0090000000000007e-06 2024-09-10T19:20:56.719 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:20:56.719 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:20:56.719 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-10T19:20:56.719 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:17:26.955521+0000", 2024-09-10T19:20:56.719 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.94012421800000001 2024-09-10T19:20:56.719 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:20:56.719 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:20:56.719 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-10T19:20:56.719 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:17:26.955605+0000", 2024-09-10T19:20:56.719 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.3910000000000001e-05 2024-09-10T19:20:56.719 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:20:56.719 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:20:56.719 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:20:56.720 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:20:56.720 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:20:56.720 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.15051.0:93089 8.6 8:604b94ad:::benchmark_data_smithi183_114924_object93088:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e177)", 2024-09-10T19:20:56.720 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-10T19:17:26.015240+0000", 2024-09-10T19:20:56.720 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 210.57741963000001, 2024-09-10T19:20:56.720 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.95416877, 2024-09-10T19:20:56.720 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-10T19:20:56.720 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T19:20:56.720 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-10T19:20:56.720 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.15051", 2024-09-10T19:20:56.720 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.183:0/3632028672", 2024-09-10T19:20:56.720 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 93089 2024-09-10T19:20:56.720 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:20:56.720 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-10T19:20:56.721 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:20:56.721 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-10T19:20:56.721 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:17:26.015240+0000", 2024-09-10T19:20:56.721 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:20:56.721 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:20:56.721 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:20:56.721 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-10T19:20:56.721 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:17:26.015240+0000", 2024-09-10T19:20:56.721 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:20:56.721 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:20:56.721 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:20:56.721 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-10T19:20:56.721 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:17:26.015240+0000", 2024-09-10T19:20:56.722 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4294967295.9999995 2024-09-10T19:20:56.722 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:20:56.722 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:20:56.722 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-10T19:20:56.722 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:17:26.015259+0000", 2024-09-10T19:20:56.722 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.9879999999999999e-05 2024-09-10T19:20:56.722 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:20:56.722 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:20:56.722 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-10T19:20:56.722 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:17:26.015266+0000", 2024-09-10T19:20:56.722 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.8109999999999997e-06 2024-09-10T19:20:56.722 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:20:56.722 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:20:56.723 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-10T19:20:56.723 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:17:26.015270+0000", 2024-09-10T19:20:56.723 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.9349999999999996e-06 2024-09-10T19:20:56.723 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:20:56.723 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:20:56.723 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-10T19:20:56.723 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:17:26.018661+0000", 2024-09-10T19:20:56.723 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0033909840000000001 2024-09-10T19:20:56.723 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:20:56.723 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:20:56.723 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for peered", 2024-09-10T19:20:56.723 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:17:26.018669+0000", 2024-09-10T19:20:56.723 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.7000000000000008e-06 2024-09-10T19:20:56.723 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:20:56.723 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:20:56.724 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-10T19:20:56.724 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:17:26.955875+0000", 2024-09-10T19:20:56.724 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.93720624299999999 2024-09-10T19:20:56.724 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:20:56.724 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:20:56.724 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-10T19:20:56.724 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:17:26.956025+0000", 2024-09-10T19:20:56.724 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00014962800000000001 2024-09-10T19:20:56.724 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:20:56.724 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:20:56.724 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 6,7", 2024-09-10T19:20:56.724 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:17:26.956362+0000", 2024-09-10T19:20:56.724 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000337337 2024-09-10T19:20:56.724 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:20:56.725 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:20:56.725 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-10T19:20:56.725 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:17:26.957687+0000", 2024-09-10T19:20:56.725 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0013250619999999999 2024-09-10T19:20:56.725 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:20:56.725 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:20:56.725 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:20:56.725 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:17:26.962591+0000", 2024-09-10T19:20:56.725 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0049041290000000001 2024-09-10T19:20:56.725 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:20:56.725 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:20:56.725 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:20:56.725 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:17:26.969343+0000", 2024-09-10T19:20:56.725 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0067516900000000003 2024-09-10T19:20:56.726 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:20:56.726 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:20:56.726 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-10T19:20:56.726 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:17:26.969386+0000", 2024-09-10T19:20:56.726 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.2775999999999998e-05 2024-09-10T19:20:56.726 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:20:56.726 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:20:56.726 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-10T19:20:56.726 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:17:26.969409+0000", 2024-09-10T19:20:56.726 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.3110000000000001e-05 2024-09-10T19:20:56.726 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:20:56.726 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:20:56.726 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:20:56.726 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:20:56.727 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:20:56.727 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.15051.0:93096 8.16 8:689007e2:::benchmark_data_smithi183_114924_object93095:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e177)", 2024-09-10T19:20:56.727 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-10T19:17:26.181849+0000", 2024-09-10T19:20:56.727 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 210.41081117100001, 2024-09-10T19:20:56.727 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.79545511700000004, 2024-09-10T19:20:56.727 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-10T19:20:56.727 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T19:20:56.727 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-10T19:20:56.727 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.15051", 2024-09-10T19:20:56.727 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.183:0/3632028672", 2024-09-10T19:20:56.727 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 93096 2024-09-10T19:20:56.727 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:20:56.727 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-10T19:20:56.727 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:20:56.728 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-10T19:20:56.728 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:17:26.181849+0000", 2024-09-10T19:20:56.728 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:20:56.728 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:20:56.728 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:20:56.728 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-10T19:20:56.728 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:17:26.181849+0000", 2024-09-10T19:20:56.728 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:20:56.728 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:20:56.728 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:20:56.728 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-10T19:20:56.728 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:17:26.181847+0000", 2024-09-10T19:20:56.728 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4294967295.9999986 2024-09-10T19:20:56.729 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:20:56.729 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:20:56.729 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-10T19:20:56.729 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:17:26.181869+0000", 2024-09-10T19:20:56.729 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.2276000000000002e-05 2024-09-10T19:20:56.729 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:20:56.729 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:20:56.729 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-10T19:20:56.729 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:17:26.181882+0000", 2024-09-10T19:20:56.729 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.238e-05 2024-09-10T19:20:56.729 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:20:56.729 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:20:56.729 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-10T19:20:56.729 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:17:26.181888+0000", 2024-09-10T19:20:56.730 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.9710000000000001e-06 2024-09-10T19:20:56.730 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:20:56.730 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:20:56.730 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-10T19:20:56.730 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:17:26.181930+0000", 2024-09-10T19:20:56.730 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.2710999999999998e-05 2024-09-10T19:20:56.730 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:20:56.730 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:20:56.730 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for peered", 2024-09-10T19:20:56.730 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:17:26.181938+0000", 2024-09-10T19:20:56.730 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.4279999999999999e-06 2024-09-10T19:20:56.730 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:20:56.730 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:20:56.730 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-10T19:20:56.731 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:17:26.971110+0000", 2024-09-10T19:20:56.731 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.789171972 2024-09-10T19:20:56.731 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:20:56.731 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:20:56.731 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-10T19:20:56.731 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:17:26.971272+0000", 2024-09-10T19:20:56.731 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000162411 2024-09-10T19:20:56.731 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:20:56.731 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:20:56.731 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 6,7", 2024-09-10T19:20:56.731 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:17:26.971538+0000", 2024-09-10T19:20:56.731 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00026615200000000002 2024-09-10T19:20:56.731 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:20:56.731 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:20:56.732 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-10T19:20:56.732 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:17:26.973920+0000", 2024-09-10T19:20:56.732 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.002381438 2024-09-10T19:20:56.732 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:20:56.732 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:20:56.732 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:20:56.732 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:17:26.975386+0000", 2024-09-10T19:20:56.732 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0014664420000000001 2024-09-10T19:20:56.732 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:20:56.732 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:20:56.732 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:20:56.732 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:17:26.977234+0000", 2024-09-10T19:20:56.732 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.001848164 2024-09-10T19:20:56.733 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:20:56.733 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:20:56.733 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-10T19:20:56.733 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:17:26.977269+0000", 2024-09-10T19:20:56.733 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.4718999999999997e-05 2024-09-10T19:20:56.733 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:20:56.733 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:20:56.733 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-10T19:20:56.733 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:17:26.977304+0000", 2024-09-10T19:20:56.733 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.4554000000000002e-05 2024-09-10T19:20:56.733 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:20:56.733 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:20:56.733 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:20:56.734 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:20:56.734 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:20:56.734 INFO:teuthology.orchestra.run.smithi183.stdout:} 2024-09-10T19:20:56.740 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: 55 16 98181 98165 111.53 89.8125 0.00865457 0.00896211 2024-09-10T19:20:56.764 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:20:56.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:20:56.765 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:20:56.866 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:20:56.906 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:20:56.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:20:56.907 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:20:57.008 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:20:57.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:20:57.036 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:20:57.036 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:20:57.137 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:20:57.174 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:20:57.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:20:57.175 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:20:57.276 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:20:57.304 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:20:57.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:20:57.306 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:20:57.367 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:20:57 smithi191 ceph-mon[32884]: 5.15 scrub starts 2024-09-10T19:20:57.367 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:20:57 smithi191 ceph-mon[32884]: 5.15 scrub ok 2024-09-10T19:20:57.367 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:20:57 smithi191 ceph-mon[32884]: 5.1 scrub starts 2024-09-10T19:20:57.367 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:20:57 smithi191 ceph-mon[32884]: 5.1 scrub ok 2024-09-10T19:20:57.367 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:20:57 smithi191 ceph-mon[32884]: 4.1d deep-scrub starts 2024-09-10T19:20:57.367 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:20:57 smithi191 ceph-mon[32884]: 4.1d deep-scrub ok 2024-09-10T19:20:57.367 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:20:57 smithi191 ceph-mon[32884]: pgmap v486: 158 pgs: 158 active+clean; 5.8 GiB data, 21 GiB used, 515 GiB / 536 GiB avail; 1.2 KiB/s rd, 112 MiB/s wr, 1.79k op/s 2024-09-10T19:20:57.367 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:20:57 smithi191 ceph-mon[32884]: 6.12 scrub starts 2024-09-10T19:20:57.367 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:20:57 smithi191 ceph-mon[32884]: 6.12 scrub ok 2024-09-10T19:20:57.403 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:20:57 smithi183 ceph-mon[29736]: 5.15 scrub starts 2024-09-10T19:20:57.404 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:20:57 smithi183 ceph-mon[29736]: 5.15 scrub ok 2024-09-10T19:20:57.404 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:20:57 smithi183 ceph-mon[29736]: 5.1 scrub starts 2024-09-10T19:20:57.404 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:20:57 smithi183 ceph-mon[29736]: 5.1 scrub ok 2024-09-10T19:20:57.404 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:20:57 smithi183 ceph-mon[29736]: 4.1d deep-scrub starts 2024-09-10T19:20:57.404 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:20:57 smithi183 ceph-mon[29736]: 4.1d deep-scrub ok 2024-09-10T19:20:57.404 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:20:57 smithi183 ceph-mon[29736]: pgmap v486: 158 pgs: 158 active+clean; 5.8 GiB data, 21 GiB used, 515 GiB / 536 GiB avail; 1.2 KiB/s rd, 112 MiB/s wr, 1.79k op/s 2024-09-10T19:20:57.404 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:20:57 smithi183 ceph-mon[29736]: 6.12 scrub starts 2024-09-10T19:20:57.404 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:20:57 smithi183 ceph-mon[29736]: 6.12 scrub ok 2024-09-10T19:20:57.404 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:20:57 smithi183 ceph-mon[41562]: 5.15 scrub starts 2024-09-10T19:20:57.404 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:20:57 smithi183 ceph-mon[41562]: 5.15 scrub ok 2024-09-10T19:20:57.404 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:20:57 smithi183 ceph-mon[41562]: 5.1 scrub starts 2024-09-10T19:20:57.405 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:20:57 smithi183 ceph-mon[41562]: 5.1 scrub ok 2024-09-10T19:20:57.405 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:20:57 smithi183 ceph-mon[41562]: 4.1d deep-scrub starts 2024-09-10T19:20:57.405 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:20:57 smithi183 ceph-mon[41562]: 4.1d deep-scrub ok 2024-09-10T19:20:57.405 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:20:57 smithi183 ceph-mon[41562]: pgmap v486: 158 pgs: 158 active+clean; 5.8 GiB data, 21 GiB used, 515 GiB / 536 GiB avail; 1.2 KiB/s rd, 112 MiB/s wr, 1.79k op/s 2024-09-10T19:20:57.405 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:20:57 smithi183 ceph-mon[41562]: 6.12 scrub starts 2024-09-10T19:20:57.405 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:20:57 smithi183 ceph-mon[41562]: 6.12 scrub ok 2024-09-10T19:20:57.406 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:20:57.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:20:57.435 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:20:57.435 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:20:57.536 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:20:57.563 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:20:57.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:20:57.564 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:20:57.665 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:20:57.691 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:20:57.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:20:57.692 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:20:57.716 INFO:journalctl@ceph.mgr.x.smithi191.stdout:Sep 10 19:20:57 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-x[34967]: ::ffff:172.21.15.191 - - [10/Sep/2024:19:20:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T19:20:57.740 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: 56 16 99560 99544 111.077 86.1875 0.0152514 0.00899824 2024-09-10T19:20:57.793 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:20:57.832 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:20:57.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:20:57.833 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:20:57.934 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:20:57.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:20:57.977 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:20:57.977 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:20:58.042 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph osd set noscrub 2024-09-10T19:20:58.078 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:20:58.106 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph daemon osd.2 dump_ops_in_flight 2024-09-10T19:20:58.126 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:20:58.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:20:58.127 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:20:58.228 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:20:58.266 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:20:58.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:20:58.267 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:20:58.342 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:20:58 smithi183 ceph-mon[41562]: 2.f scrub starts 2024-09-10T19:20:58.342 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:20:58 smithi183 ceph-mon[41562]: 2.f scrub ok 2024-09-10T19:20:58.342 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:20:58 smithi183 ceph-mon[41562]: 2.1e scrub starts 2024-09-10T19:20:58.342 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:20:58 smithi183 ceph-mon[41562]: 2.1e scrub ok 2024-09-10T19:20:58.342 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:20:58 smithi183 ceph-mon[41562]: 5.5 scrub starts 2024-09-10T19:20:58.342 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:20:58 smithi183 ceph-mon[41562]: 5.5 scrub ok 2024-09-10T19:20:58.343 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:20:58 smithi183 ceph-mon[29736]: 2.f scrub starts 2024-09-10T19:20:58.343 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:20:58 smithi183 ceph-mon[29736]: 2.f scrub ok 2024-09-10T19:20:58.343 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:20:58 smithi183 ceph-mon[29736]: 2.1e scrub starts 2024-09-10T19:20:58.343 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:20:58 smithi183 ceph-mon[29736]: 2.1e scrub ok 2024-09-10T19:20:58.343 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:20:58 smithi183 ceph-mon[29736]: 5.5 scrub starts 2024-09-10T19:20:58.343 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:20:58 smithi183 ceph-mon[29736]: 5.5 scrub ok 2024-09-10T19:20:58.368 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:20:58.406 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:20:58.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:20:58.407 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:20:58.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:20:58 smithi191 ceph-mon[32884]: 2.f scrub starts 2024-09-10T19:20:58.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:20:58 smithi191 ceph-mon[32884]: 2.f scrub ok 2024-09-10T19:20:58.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:20:58 smithi191 ceph-mon[32884]: 2.1e scrub starts 2024-09-10T19:20:58.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:20:58 smithi191 ceph-mon[32884]: 2.1e scrub ok 2024-09-10T19:20:58.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:20:58 smithi191 ceph-mon[32884]: 5.5 scrub starts 2024-09-10T19:20:58.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:20:58 smithi191 ceph-mon[32884]: 5.5 scrub ok 2024-09-10T19:20:58.509 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:20:58.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:20:58.546 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:20:58.546 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:20:58.568 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:20:58.646 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:20:58.682 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:20:58.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:20:58.684 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:20:58.740 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: 57 16 100773 100757 110.458 75.8125 0.00707964 0.00904848 2024-09-10T19:20:58.784 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:20:58.829 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:20:58.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:20:58.830 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:20:58.930 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:20:58.972 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:20:58.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:20:58.973 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:20:59.074 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:20:59.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:20:59.112 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:20:59.112 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:20:59.213 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:20:59.239 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:20:59.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:20:59.241 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:20:59.341 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:20:59.380 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:20:59.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:20:59.381 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:20:59.394 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:20:59 smithi191 ceph-mon[32884]: 5.19 scrub starts 2024-09-10T19:20:59.394 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:20:59 smithi191 ceph-mon[32884]: 5.19 scrub ok 2024-09-10T19:20:59.394 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:20:59 smithi191 ceph-mon[32884]: 2.10 scrub starts 2024-09-10T19:20:59.394 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:20:59 smithi191 ceph-mon[32884]: 2.10 scrub ok 2024-09-10T19:20:59.394 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:20:59 smithi191 ceph-mon[32884]: pgmap v487: 158 pgs: 158 active+clean; 5.8 GiB data, 21 GiB used, 515 GiB / 536 GiB avail; 1.2 KiB/s rd, 112 MiB/s wr, 1.79k op/s 2024-09-10T19:20:59.394 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:20:59 smithi191 ceph-mon[32884]: 5.9 deep-scrub starts 2024-09-10T19:20:59.394 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:20:59 smithi191 ceph-mon[32884]: 5.9 deep-scrub ok 2024-09-10T19:20:59.394 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:20:59 smithi191 ceph-mon[32884]: 2.6 scrub starts 2024-09-10T19:20:59.394 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:20:59 smithi191 ceph-mon[32884]: 2.6 scrub ok 2024-09-10T19:20:59.395 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:20:59 smithi191 ceph-mon[32884]: from='mgr.14556 ' entity='mgr.y' 2024-09-10T19:20:59.395 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:20:59 smithi191 ceph-mon[32884]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:20:59.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:20:59 smithi183 ceph-mon[29736]: 5.19 scrub starts 2024-09-10T19:20:59.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:20:59 smithi183 ceph-mon[29736]: 5.19 scrub ok 2024-09-10T19:20:59.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:20:59 smithi183 ceph-mon[29736]: 2.10 scrub starts 2024-09-10T19:20:59.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:20:59 smithi183 ceph-mon[29736]: 2.10 scrub ok 2024-09-10T19:20:59.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:20:59 smithi183 ceph-mon[29736]: pgmap v487: 158 pgs: 158 active+clean; 5.8 GiB data, 21 GiB used, 515 GiB / 536 GiB avail; 1.2 KiB/s rd, 112 MiB/s wr, 1.79k op/s 2024-09-10T19:20:59.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:20:59 smithi183 ceph-mon[29736]: 5.9 deep-scrub starts 2024-09-10T19:20:59.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:20:59 smithi183 ceph-mon[29736]: 5.9 deep-scrub ok 2024-09-10T19:20:59.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:20:59 smithi183 ceph-mon[29736]: 2.6 scrub starts 2024-09-10T19:20:59.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:20:59 smithi183 ceph-mon[29736]: 2.6 scrub ok 2024-09-10T19:20:59.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:20:59 smithi183 ceph-mon[29736]: from='mgr.14556 ' entity='mgr.y' 2024-09-10T19:20:59.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:20:59 smithi183 ceph-mon[29736]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:20:59.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:20:59 smithi183 ceph-mon[41562]: 5.19 scrub starts 2024-09-10T19:20:59.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:20:59 smithi183 ceph-mon[41562]: 5.19 scrub ok 2024-09-10T19:20:59.433 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:20:59 smithi183 ceph-mon[41562]: 2.10 scrub starts 2024-09-10T19:20:59.433 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:20:59 smithi183 ceph-mon[41562]: 2.10 scrub ok 2024-09-10T19:20:59.433 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:20:59 smithi183 ceph-mon[41562]: pgmap v487: 158 pgs: 158 active+clean; 5.8 GiB data, 21 GiB used, 515 GiB / 536 GiB avail; 1.2 KiB/s rd, 112 MiB/s wr, 1.79k op/s 2024-09-10T19:20:59.433 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:20:59 smithi183 ceph-mon[41562]: 5.9 deep-scrub starts 2024-09-10T19:20:59.433 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:20:59 smithi183 ceph-mon[41562]: 5.9 deep-scrub ok 2024-09-10T19:20:59.433 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:20:59 smithi183 ceph-mon[41562]: 2.6 scrub starts 2024-09-10T19:20:59.433 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:20:59 smithi183 ceph-mon[41562]: 2.6 scrub ok 2024-09-10T19:20:59.433 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:20:59 smithi183 ceph-mon[41562]: from='mgr.14556 ' entity='mgr.y' 2024-09-10T19:20:59.433 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:20:59 smithi183 ceph-mon[41562]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:20:59.482 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:20:59.511 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:20:59.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:20:59.512 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:20:59.613 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:20:59.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:20:59.648 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:20:59.649 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:20:59.716 INFO:journalctl@ceph.iscsi.iscsi.a.smithi191.stdout:Sep 10 19:20:59 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-iscsi-iscsi-a[61816]: debug there is no tcmu-runner data available 2024-09-10T19:20:59.740 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: 58 16 102076 102060 109.958 81.4375 0.00660674 0.00908965 2024-09-10T19:20:59.750 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:20:59.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:20:59.791 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:20:59.791 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:20:59.796 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:20:59.893 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:20:59.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:20:59.934 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:20:59.934 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:21:00.036 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:21:00.066 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:21:00.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:21:00.067 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:21:00.168 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:21:00.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:21:00.197 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:21:00.198 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:21:00.299 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:21:00.335 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:21:00.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:21:00.336 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:21:00.407 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:21:00 smithi183 ceph-mon[29736]: 6.4 scrub starts 2024-09-10T19:21:00.407 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:21:00 smithi183 ceph-mon[29736]: 6.4 scrub ok 2024-09-10T19:21:00.407 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:21:00 smithi183 ceph-mon[41562]: 6.4 scrub starts 2024-09-10T19:21:00.408 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:21:00 smithi183 ceph-mon[41562]: 6.4 scrub ok 2024-09-10T19:21:00.437 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:21:00.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:21:00 smithi191 ceph-mon[32884]: 6.4 scrub starts 2024-09-10T19:21:00.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:21:00 smithi191 ceph-mon[32884]: 6.4 scrub ok 2024-09-10T19:21:00.472 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:21:00.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:21:00.474 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:21:00.575 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:21:00.619 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:21:00.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:21:00.620 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:21:00.721 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:21:00.740 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: 59 16 103508 103492 109.611 89.5 0.00614075 0.00911826 2024-09-10T19:21:00.762 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:21:00.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:21:00.763 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:21:00.864 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:21:00.891 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:21:00.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:21:00.892 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:21:00.993 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:21:01.023 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:21:01.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:21:01.024 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:21:01.124 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:21:01.159 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:21:01.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:21:01.160 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:21:01.244 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:21:01 smithi183 ceph-mon[29736]: 2.16 scrub starts 2024-09-10T19:21:01.244 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:21:01 smithi183 ceph-mon[29736]: 2.16 scrub ok 2024-09-10T19:21:01.244 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:21:01 smithi183 ceph-mon[29736]: from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:21:01.244 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:21:01 smithi183 ceph-mon[29736]: pgmap v488: 158 pgs: 158 active+clean; 5.8 GiB data, 21 GiB used, 515 GiB / 536 GiB avail; 639 B/s rd, 56 MiB/s wr, 897 op/s 2024-09-10T19:21:01.244 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:21:01 smithi183 ceph-mon[29736]: 1.0 scrub starts 2024-09-10T19:21:01.244 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:21:01 smithi183 ceph-mon[29736]: 1.0 scrub ok 2024-09-10T19:21:01.244 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:21:01 smithi183 ceph-mon[29736]: 5.1b scrub starts 2024-09-10T19:21:01.244 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:21:01 smithi183 ceph-mon[29736]: 5.1b scrub ok 2024-09-10T19:21:01.245 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:21:01 smithi183 ceph-mon[41562]: 2.16 scrub starts 2024-09-10T19:21:01.245 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:21:01 smithi183 ceph-mon[41562]: 2.16 scrub ok 2024-09-10T19:21:01.245 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:21:01 smithi183 ceph-mon[41562]: from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:21:01.245 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:21:01 smithi183 ceph-mon[41562]: pgmap v488: 158 pgs: 158 active+clean; 5.8 GiB data, 21 GiB used, 515 GiB / 536 GiB avail; 639 B/s rd, 56 MiB/s wr, 897 op/s 2024-09-10T19:21:01.245 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:21:01 smithi183 ceph-mon[41562]: 1.0 scrub starts 2024-09-10T19:21:01.245 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:21:01 smithi183 ceph-mon[41562]: 1.0 scrub ok 2024-09-10T19:21:01.245 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:21:01 smithi183 ceph-mon[41562]: 5.1b scrub starts 2024-09-10T19:21:01.245 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:21:01 smithi183 ceph-mon[41562]: 5.1b scrub ok 2024-09-10T19:21:01.261 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:21:01.299 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:21:01.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:21:01.300 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:21:01.400 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:21:01.430 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:21:01.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:21:01.431 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:21:01.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:21:01 smithi191 ceph-mon[32884]: 2.16 scrub starts 2024-09-10T19:21:01.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:21:01 smithi191 ceph-mon[32884]: 2.16 scrub ok 2024-09-10T19:21:01.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:21:01 smithi191 ceph-mon[32884]: from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:21:01.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:21:01 smithi191 ceph-mon[32884]: pgmap v488: 158 pgs: 158 active+clean; 5.8 GiB data, 21 GiB used, 515 GiB / 536 GiB avail; 639 B/s rd, 56 MiB/s wr, 897 op/s 2024-09-10T19:21:01.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:21:01 smithi191 ceph-mon[32884]: 1.0 scrub starts 2024-09-10T19:21:01.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:21:01 smithi191 ceph-mon[32884]: 1.0 scrub ok 2024-09-10T19:21:01.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:21:01 smithi191 ceph-mon[32884]: 5.1b scrub starts 2024-09-10T19:21:01.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:21:01 smithi191 ceph-mon[32884]: 5.1b scrub ok 2024-09-10T19:21:01.532 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:21:01.559 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:21:01.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:21:01.560 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:21:01.661 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:21:01.681 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Sep 10 19:21:01 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-y[29989]: ::ffff:172.21.15.191 - - [10/Sep/2024:19:21:01] "GET /metrics HTTP/1.1" 200 224355 "" "Prometheus/2.43.0" 2024-09-10T19:21:01.691 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:21:01.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:21:01.693 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:21:01.740 INFO:tasks.radosbench.radosbench.0.smithi183.stdout:2024-09-10T19:21:01.739805+0000 min lat: 0.00139047 max lat: 0.102872 avg lat: 0.00914315 2024-09-10T19:21:01.741 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-10T19:21:01.741 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: 60 16 104981 104965 109.319 92.0625 0.00765023 0.00914315 2024-09-10T19:21:01.793 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:21:01.826 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:21:01.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:21:01.827 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:21:01.928 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:21:01.966 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:21:01.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:21:01.967 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:21:02.068 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:21:02.098 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:21:02.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:21:02.099 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:21:02.200 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:21:02.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:21:02.236 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:21:02.236 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:21:02.249 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:21:02.337 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:21:02.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:21:02.374 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:21:02.374 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:21:02.377 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:21:02 smithi183 ceph-mon[29736]: 4.5 scrub starts 2024-09-10T19:21:02.377 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:21:02 smithi183 ceph-mon[29736]: 4.5 scrub ok 2024-09-10T19:21:02.377 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:21:02 smithi183 ceph-mon[29736]: 2.1 scrub starts 2024-09-10T19:21:02.377 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:21:02 smithi183 ceph-mon[29736]: 2.1 scrub ok 2024-09-10T19:21:02.377 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:21:02 smithi183 ceph-mon[41562]: 4.5 scrub starts 2024-09-10T19:21:02.377 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:21:02 smithi183 ceph-mon[41562]: 4.5 scrub ok 2024-09-10T19:21:02.377 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:21:02 smithi183 ceph-mon[41562]: 2.1 scrub starts 2024-09-10T19:21:02.377 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:21:02 smithi183 ceph-mon[41562]: 2.1 scrub ok 2024-09-10T19:21:02.465 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:21:02 smithi191 ceph-mon[32884]: 4.5 scrub starts 2024-09-10T19:21:02.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:21:02 smithi191 ceph-mon[32884]: 4.5 scrub ok 2024-09-10T19:21:02.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:21:02 smithi191 ceph-mon[32884]: 2.1 scrub starts 2024-09-10T19:21:02.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:21:02 smithi191 ceph-mon[32884]: 2.1 scrub ok 2024-09-10T19:21:02.475 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:21:02.496 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:21:02.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:21:02.497 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:21:02.598 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:21:02.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:21:02.633 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:21:02.634 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:21:02.735 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:21:02.740 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: 61 16 106289 106273 108.867 81.75 0.0106312 0.00918116 2024-09-10T19:21:02.769 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:21:02.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:21:02.770 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:21:02.871 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:21:02.896 INFO:teuthology.orchestra.run.smithi183.stdout:osd.0: {} 2024-09-10T19:21:02.897 INFO:teuthology.orchestra.run.smithi183.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-10T19:21:02.907 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:21:02.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:21:02.908 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:21:02.918 INFO:teuthology.orchestra.run.smithi183.stdout:osd.1: {} 2024-09-10T19:21:02.918 INFO:teuthology.orchestra.run.smithi183.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-10T19:21:02.940 INFO:teuthology.orchestra.run.smithi183.stdout:osd.2: {} 2024-09-10T19:21:02.940 INFO:teuthology.orchestra.run.smithi183.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-10T19:21:02.963 INFO:teuthology.orchestra.run.smithi183.stdout:osd.3: {} 2024-09-10T19:21:02.964 INFO:teuthology.orchestra.run.smithi183.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-10T19:21:02.994 INFO:teuthology.orchestra.run.smithi183.stdout:osd.4: {} 2024-09-10T19:21:02.995 INFO:teuthology.orchestra.run.smithi183.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-10T19:21:03.008 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:21:03.024 INFO:teuthology.orchestra.run.smithi183.stdout:osd.5: {} 2024-09-10T19:21:03.024 INFO:teuthology.orchestra.run.smithi183.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-10T19:21:03.043 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:21:03.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:21:03.044 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:21:03.047 INFO:teuthology.orchestra.run.smithi183.stdout:osd.6: {} 2024-09-10T19:21:03.048 INFO:teuthology.orchestra.run.smithi183.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-10T19:21:03.075 INFO:teuthology.orchestra.run.smithi183.stdout:osd.7: {} 2024-09-10T19:21:03.076 INFO:teuthology.orchestra.run.smithi183.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-10T19:21:03.131 INFO:teuthology.orchestra.run.smithi183.stderr:clear 4.11 pg_upmap_items mapping 2024-09-10T19:21:03.145 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:21:03.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:21:03.182 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:21:03.183 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:21:03.283 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:21:03.312 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:21:03.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:21:03.313 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:21:03.327 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:21:03 smithi183 ceph-mon[29736]: 4.16 scrub starts 2024-09-10T19:21:03.327 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:21:03 smithi183 ceph-mon[29736]: 4.16 scrub ok 2024-09-10T19:21:03.327 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:21:03 smithi183 ceph-mon[29736]: pgmap v489: 158 pgs: 158 active+clean; 6.3 GiB data, 22 GiB used, 514 GiB / 536 GiB avail; 100 MiB/s wr, 1.60k op/s 2024-09-10T19:21:03.328 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:21:03 smithi183 ceph-mon[29736]: 2.1c scrub starts 2024-09-10T19:21:03.328 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:21:03 smithi183 ceph-mon[29736]: 2.1c scrub ok 2024-09-10T19:21:03.328 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:21:03 smithi183 ceph-mon[29736]: from='client.? 172.21.15.183:0/3813129181' entity='client.admin' cmd=[{"prefix": "osd rm-pg-upmap-items", "pgid": "4.11"}]: dispatch 2024-09-10T19:21:03.328 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:21:03 smithi183 ceph-mon[29736]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd rm-pg-upmap-items", "pgid": "4.11"}]: dispatch 2024-09-10T19:21:03.328 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:21:03 smithi183 ceph-mon[29736]: from='client.? 172.21.15.183:0/413674561' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T19:21:03.328 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:21:03 smithi183 ceph-mon[41562]: 4.16 scrub starts 2024-09-10T19:21:03.328 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:21:03 smithi183 ceph-mon[41562]: 4.16 scrub ok 2024-09-10T19:21:03.328 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:21:03 smithi183 ceph-mon[41562]: pgmap v489: 158 pgs: 158 active+clean; 6.3 GiB data, 22 GiB used, 514 GiB / 536 GiB avail; 100 MiB/s wr, 1.60k op/s 2024-09-10T19:21:03.328 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:21:03 smithi183 ceph-mon[41562]: 2.1c scrub starts 2024-09-10T19:21:03.328 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:21:03 smithi183 ceph-mon[41562]: 2.1c scrub ok 2024-09-10T19:21:03.329 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:21:03 smithi183 ceph-mon[41562]: from='client.? 172.21.15.183:0/3813129181' entity='client.admin' cmd=[{"prefix": "osd rm-pg-upmap-items", "pgid": "4.11"}]: dispatch 2024-09-10T19:21:03.329 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:21:03 smithi183 ceph-mon[41562]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd rm-pg-upmap-items", "pgid": "4.11"}]: dispatch 2024-09-10T19:21:03.329 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:21:03 smithi183 ceph-mon[41562]: from='client.? 172.21.15.183:0/413674561' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T19:21:03.414 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:21:03.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:21:03.452 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:21:03.452 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:21:03.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:21:03 smithi191 ceph-mon[32884]: 4.16 scrub starts 2024-09-10T19:21:03.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:21:03 smithi191 ceph-mon[32884]: 4.16 scrub ok 2024-09-10T19:21:03.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:21:03 smithi191 ceph-mon[32884]: pgmap v489: 158 pgs: 158 active+clean; 6.3 GiB data, 22 GiB used, 514 GiB / 536 GiB avail; 100 MiB/s wr, 1.60k op/s 2024-09-10T19:21:03.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:21:03 smithi191 ceph-mon[32884]: 2.1c scrub starts 2024-09-10T19:21:03.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:21:03 smithi191 ceph-mon[32884]: 2.1c scrub ok 2024-09-10T19:21:03.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:21:03 smithi191 ceph-mon[32884]: from='client.? 172.21.15.183:0/3813129181' entity='client.admin' cmd=[{"prefix": "osd rm-pg-upmap-items", "pgid": "4.11"}]: dispatch 2024-09-10T19:21:03.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:21:03 smithi191 ceph-mon[32884]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd rm-pg-upmap-items", "pgid": "4.11"}]: dispatch 2024-09-10T19:21:03.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:21:03 smithi191 ceph-mon[32884]: from='client.? 172.21.15.183:0/413674561' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T19:21:03.553 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:21:03.593 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:21:03.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:21:03.594 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:21:03.695 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:21:03.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:21:03.725 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:21:03.725 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:21:03.740 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: 62 16 107716 107700 108.549 89.1875 0.0125694 0.00920822 2024-09-10T19:21:03.826 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:21:03.852 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:21:03.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:21:03.854 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:21:03.954 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:21:03.989 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:21:03.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:21:03.990 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:21:04.091 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:21:04.128 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:21:04.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:21:04.129 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:21:04.230 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:21:04.260 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:21:04.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:21:04.261 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:21:04.341 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:21:04 smithi183 ceph-mon[29736]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd rm-pg-upmap-items", "pgid": "4.11"}]': finished 2024-09-10T19:21:04.393 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:21:04 smithi183 ceph-mon[29736]: osdmap e251: 8 total, 8 up, 6 in 2024-09-10T19:21:04.393 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:21:04 smithi183 ceph-mon[41562]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd rm-pg-upmap-items", "pgid": "4.11"}]': finished 2024-09-10T19:21:04.394 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:21:04 smithi183 ceph-mon[41562]: osdmap e251: 8 total, 8 up, 6 in 2024-09-10T19:21:04.394 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:21:04.422 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:21:04.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:21:04.423 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:21:04.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:21:04 smithi191 ceph-mon[32884]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd rm-pg-upmap-items", "pgid": "4.11"}]': finished 2024-09-10T19:21:04.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:21:04 smithi191 ceph-mon[32884]: osdmap e251: 8 total, 8 up, 6 in 2024-09-10T19:21:04.523 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:21:04.557 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:21:04.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:21:04.558 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:21:04.659 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:21:04.686 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:21:04.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:21:04.687 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:21:04.740 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: 63 16 109161 109145 108.26 90.3125 0.00623245 0.00923285 2024-09-10T19:21:04.788 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:21:04.818 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:21:04.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:21:04.819 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:21:04.920 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:21:04.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:21:04.960 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:21:04.960 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:21:05.061 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:21:05.101 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:21:05.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:21:05.102 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:21:05.203 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:21:05.233 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:21:05.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:21:05.234 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:21:05.335 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:21:05.370 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:21:05.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:21:05.371 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:21:05.390 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:21:05 smithi183 ceph-mon[29736]: 6.0 scrub starts 2024-09-10T19:21:05.390 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:21:05 smithi183 ceph-mon[29736]: 6.0 scrub ok 2024-09-10T19:21:05.390 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:21:05 smithi183 ceph-mon[29736]: pgmap v491: 158 pgs: 158 active+clean; 6.3 GiB data, 22 GiB used, 514 GiB / 536 GiB avail; 97 MiB/s wr, 1.55k op/s 2024-09-10T19:21:05.390 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:21:05 smithi183 ceph-mon[29736]: 6.8 deep-scrub starts 2024-09-10T19:21:05.390 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:21:05 smithi183 ceph-mon[29736]: 6.8 deep-scrub ok 2024-09-10T19:21:05.390 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:21:05 smithi183 ceph-mon[29736]: osdmap e252: 8 total, 8 up, 6 in 2024-09-10T19:21:05.390 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:21:05 smithi183 ceph-mon[29736]: 4.a scrub starts 2024-09-10T19:21:05.390 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:21:05 smithi183 ceph-mon[29736]: 4.a scrub ok 2024-09-10T19:21:05.390 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:21:05 smithi183 ceph-mon[29736]: 5.6 scrub starts 2024-09-10T19:21:05.390 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:21:05 smithi183 ceph-mon[29736]: 5.6 scrub ok 2024-09-10T19:21:05.391 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:21:05 smithi183 ceph-mon[41562]: 6.0 scrub starts 2024-09-10T19:21:05.391 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:21:05 smithi183 ceph-mon[41562]: 6.0 scrub ok 2024-09-10T19:21:05.391 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:21:05 smithi183 ceph-mon[41562]: pgmap v491: 158 pgs: 158 active+clean; 6.3 GiB data, 22 GiB used, 514 GiB / 536 GiB avail; 97 MiB/s wr, 1.55k op/s 2024-09-10T19:21:05.391 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:21:05 smithi183 ceph-mon[41562]: 6.8 deep-scrub starts 2024-09-10T19:21:05.391 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:21:05 smithi183 ceph-mon[41562]: 6.8 deep-scrub ok 2024-09-10T19:21:05.391 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:21:05 smithi183 ceph-mon[41562]: osdmap e252: 8 total, 8 up, 6 in 2024-09-10T19:21:05.391 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:21:05 smithi183 ceph-mon[41562]: 4.a scrub starts 2024-09-10T19:21:05.391 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:21:05 smithi183 ceph-mon[41562]: 4.a scrub ok 2024-09-10T19:21:05.391 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:21:05 smithi183 ceph-mon[41562]: 5.6 scrub starts 2024-09-10T19:21:05.391 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:21:05 smithi183 ceph-mon[41562]: 5.6 scrub ok 2024-09-10T19:21:05.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:21:05 smithi191 ceph-mon[32884]: 6.0 scrub starts 2024-09-10T19:21:05.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:21:05 smithi191 ceph-mon[32884]: 6.0 scrub ok 2024-09-10T19:21:05.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:21:05 smithi191 ceph-mon[32884]: pgmap v491: 158 pgs: 158 active+clean; 6.3 GiB data, 22 GiB used, 514 GiB / 536 GiB avail; 97 MiB/s wr, 1.55k op/s 2024-09-10T19:21:05.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:21:05 smithi191 ceph-mon[32884]: 6.8 deep-scrub starts 2024-09-10T19:21:05.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:21:05 smithi191 ceph-mon[32884]: 6.8 deep-scrub ok 2024-09-10T19:21:05.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:21:05 smithi191 ceph-mon[32884]: osdmap e252: 8 total, 8 up, 6 in 2024-09-10T19:21:05.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:21:05 smithi191 ceph-mon[32884]: 4.a scrub starts 2024-09-10T19:21:05.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:21:05 smithi191 ceph-mon[32884]: 4.a scrub ok 2024-09-10T19:21:05.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:21:05 smithi191 ceph-mon[32884]: 5.6 scrub starts 2024-09-10T19:21:05.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:21:05 smithi191 ceph-mon[32884]: 5.6 scrub ok 2024-09-10T19:21:05.472 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:21:05.501 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:21:05.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:21:05.502 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:21:05.603 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:21:05.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:21:05.645 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:21:05.645 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:21:05.741 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: 64 16 110586 110570 107.96 89.0625 0.0145092 0.00925833 2024-09-10T19:21:05.746 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:21:05.774 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:21:05.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:21:05.775 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:21:05.876 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:21:05.907 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:21:05.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:21:05.908 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:21:06.009 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:21:06.042 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:21:06.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:21:06.043 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:21:06.124 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-10T19:21:06.143 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:21:06.160 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:21:06 smithi191 ceph-mon[32884]: 5.1c scrub starts 2024-09-10T19:21:06.160 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:21:06 smithi191 ceph-mon[32884]: 5.1c scrub ok 2024-09-10T19:21:06.160 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:21:06 smithi191 ceph-mon[32884]: 5.3 scrub starts 2024-09-10T19:21:06.160 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:21:06 smithi191 ceph-mon[32884]: 5.3 scrub ok 2024-09-10T19:21:06.160 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:21:06 smithi191 ceph-mon[32884]: 1.2 scrub starts 2024-09-10T19:21:06.160 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:21:06 smithi191 ceph-mon[32884]: 1.2 scrub ok 2024-09-10T19:21:06.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:21:06.186 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:21:06.186 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:21:06.287 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:21:06.314 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:21:06.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:21:06.315 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:21:06.351 INFO:teuthology.orchestra.run.smithi183.stdout:ERROR: (22) Invalid argument 2024-09-10T19:21:06.351 INFO:teuthology.orchestra.run.smithi183.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-10T19:21:06.416 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:21:06.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:21:06 smithi183 ceph-mon[29736]: 5.1c scrub starts 2024-09-10T19:21:06.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:21:06 smithi183 ceph-mon[29736]: 5.1c scrub ok 2024-09-10T19:21:06.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:21:06 smithi183 ceph-mon[29736]: 5.3 scrub starts 2024-09-10T19:21:06.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:21:06 smithi183 ceph-mon[29736]: 5.3 scrub ok 2024-09-10T19:21:06.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:21:06 smithi183 ceph-mon[29736]: 1.2 scrub starts 2024-09-10T19:21:06.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:21:06 smithi183 ceph-mon[29736]: 1.2 scrub ok 2024-09-10T19:21:06.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:21:06 smithi183 ceph-mon[41562]: 5.1c scrub starts 2024-09-10T19:21:06.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:21:06 smithi183 ceph-mon[41562]: 5.1c scrub ok 2024-09-10T19:21:06.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:21:06 smithi183 ceph-mon[41562]: 5.3 scrub starts 2024-09-10T19:21:06.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:21:06 smithi183 ceph-mon[41562]: 5.3 scrub ok 2024-09-10T19:21:06.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:21:06 smithi183 ceph-mon[41562]: 1.2 scrub starts 2024-09-10T19:21:06.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:21:06 smithi183 ceph-mon[41562]: 1.2 scrub ok 2024-09-10T19:21:06.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:21:06.456 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:21:06.456 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:21:06.557 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:21:06.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:21:06.593 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:21:06.593 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:21:06.694 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:21:06.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:21:06.723 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:21:06.723 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:21:06.740 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: 65 16 111912 111896 107.574 82.875 0.0111356 0.00929184 2024-09-10T19:21:06.824 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:21:06.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:21:06.858 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:21:06.859 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:21:06.959 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:21:06.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:21:06.993 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:21:06.994 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:21:07.095 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:21:07.132 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:21:07.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:21:07.133 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:21:07.234 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:21:07.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:21:07.273 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:21:07.273 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:21:07.374 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:21:07.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:21:07.417 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:21:07.417 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:21:07.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:21:07 smithi183 ceph-mon[29736]: 6.18 scrub starts 2024-09-10T19:21:07.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:21:07 smithi183 ceph-mon[29736]: 6.18 scrub ok 2024-09-10T19:21:07.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:21:07 smithi183 ceph-mon[29736]: pgmap v493: 158 pgs: 158 active+clean; 6.7 GiB data, 24 GiB used, 513 GiB / 536 GiB avail; 108 MiB/s wr, 1.73k op/s; 0 B/s, 0 objects/s recovering 2024-09-10T19:21:07.431 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:21:07 smithi183 ceph-mon[41562]: 6.18 scrub starts 2024-09-10T19:21:07.431 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:21:07 smithi183 ceph-mon[41562]: 6.18 scrub ok 2024-09-10T19:21:07.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:21:07 smithi183 ceph-mon[41562]: pgmap v493: 158 pgs: 158 active+clean; 6.7 GiB data, 24 GiB used, 513 GiB / 536 GiB avail; 108 MiB/s wr, 1.73k op/s; 0 B/s, 0 objects/s recovering 2024-09-10T19:21:07.451 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:21:07 smithi191 ceph-mon[32884]: 6.18 scrub starts 2024-09-10T19:21:07.451 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:21:07 smithi191 ceph-mon[32884]: 6.18 scrub ok 2024-09-10T19:21:07.451 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:21:07 smithi191 ceph-mon[32884]: pgmap v493: 158 pgs: 158 active+clean; 6.7 GiB data, 24 GiB used, 513 GiB / 536 GiB avail; 108 MiB/s wr, 1.73k op/s; 0 B/s, 0 objects/s recovering 2024-09-10T19:21:07.517 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:21:07.545 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:21:07.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:21:07.546 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:21:07.647 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:21:07.676 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:21:07.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:21:07.678 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:21:07.716 INFO:journalctl@ceph.mgr.x.smithi191.stdout:Sep 10 19:21:07 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-x[34967]: ::ffff:172.21.15.191 - - [10/Sep/2024:19:21:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T19:21:07.741 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: 66 15 113358 113343 107.314 90.4375 0.0129351 0.00931443 2024-09-10T19:21:07.778 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:21:07.780 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph daemon osd.2 dump_blocked_ops 2024-09-10T19:21:07.821 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:21:07.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:21:07.824 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:21:07.857 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:21:07.925 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:21:07.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:21:07.963 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:21:07.963 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:21:08.064 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:21:08.098 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:21:08.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:21:08.099 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:21:08.200 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:21:08.233 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:21:08.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:21:08.235 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:21:08.336 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:21:08.376 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:21:08.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:21:08.377 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:21:08.380 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:21:08 smithi183 ceph-mon[29736]: 4.12 scrub starts 2024-09-10T19:21:08.380 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:21:08 smithi183 ceph-mon[29736]: 4.12 scrub ok 2024-09-10T19:21:08.380 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:21:08 smithi183 ceph-mon[41562]: 4.12 scrub starts 2024-09-10T19:21:08.380 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:21:08 smithi183 ceph-mon[41562]: 4.12 scrub ok 2024-09-10T19:21:08.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:21:08 smithi191 ceph-mon[32884]: 4.12 scrub starts 2024-09-10T19:21:08.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:21:08 smithi191 ceph-mon[32884]: 4.12 scrub ok 2024-09-10T19:21:08.477 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:21:08.511 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:21:08.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:21:08.512 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:21:08.613 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:21:08.649 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:21:08.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:21:08.650 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:21:08.741 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: 67 16 114770 114754 107.028 88.1875 0.00926416 0.00933897 2024-09-10T19:21:08.753 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:21:08.785 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:21:08.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:21:08.786 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:21:08.887 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:21:08.917 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:21:08.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:21:08.918 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:21:09.018 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:21:09.050 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:21:09.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:21:09.051 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:21:09.152 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:21:09.189 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:21:09.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:21:09.190 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:21:09.291 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:21:09.329 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:21:09.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:21:09.330 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:21:09.362 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:21:09 smithi183 ceph-mon[29736]: 6.1e scrub starts 2024-09-10T19:21:09.362 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:21:09 smithi183 ceph-mon[29736]: 6.1e scrub ok 2024-09-10T19:21:09.362 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:21:09 smithi183 ceph-mon[29736]: 4.6 scrub starts 2024-09-10T19:21:09.362 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:21:09 smithi183 ceph-mon[29736]: 4.6 scrub ok 2024-09-10T19:21:09.362 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:21:09 smithi183 ceph-mon[29736]: pgmap v494: 158 pgs: 158 active+clean; 6.7 GiB data, 24 GiB used, 513 GiB / 536 GiB avail; 108 MiB/s wr, 1.73k op/s; 0 B/s, 0 objects/s recovering 2024-09-10T19:21:09.362 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:21:09 smithi183 ceph-mon[29736]: 1.9 scrub starts 2024-09-10T19:21:09.363 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:21:09 smithi183 ceph-mon[29736]: 1.9 scrub ok 2024-09-10T19:21:09.363 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:21:09 smithi183 ceph-mon[29736]: 5.7 scrub starts 2024-09-10T19:21:09.363 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:21:09 smithi183 ceph-mon[29736]: 5.7 scrub ok 2024-09-10T19:21:09.363 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:21:09 smithi183 ceph-mon[29736]: 1.7 scrub starts 2024-09-10T19:21:09.363 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:21:09 smithi183 ceph-mon[29736]: 1.7 scrub ok 2024-09-10T19:21:09.363 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:21:09 smithi183 ceph-mon[41562]: 6.1e scrub starts 2024-09-10T19:21:09.363 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:21:09 smithi183 ceph-mon[41562]: 6.1e scrub ok 2024-09-10T19:21:09.363 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:21:09 smithi183 ceph-mon[41562]: 4.6 scrub starts 2024-09-10T19:21:09.363 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:21:09 smithi183 ceph-mon[41562]: 4.6 scrub ok 2024-09-10T19:21:09.363 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:21:09 smithi183 ceph-mon[41562]: pgmap v494: 158 pgs: 158 active+clean; 6.7 GiB data, 24 GiB used, 513 GiB / 536 GiB avail; 108 MiB/s wr, 1.73k op/s; 0 B/s, 0 objects/s recovering 2024-09-10T19:21:09.364 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:21:09 smithi183 ceph-mon[41562]: 1.9 scrub starts 2024-09-10T19:21:09.364 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:21:09 smithi183 ceph-mon[41562]: 1.9 scrub ok 2024-09-10T19:21:09.364 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:21:09 smithi183 ceph-mon[41562]: 5.7 scrub starts 2024-09-10T19:21:09.364 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:21:09 smithi183 ceph-mon[41562]: 5.7 scrub ok 2024-09-10T19:21:09.364 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:21:09 smithi183 ceph-mon[41562]: 1.7 scrub starts 2024-09-10T19:21:09.364 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:21:09 smithi183 ceph-mon[41562]: 1.7 scrub ok 2024-09-10T19:21:09.431 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:21:09.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:21:09 smithi191 ceph-mon[32884]: 6.1e scrub starts 2024-09-10T19:21:09.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:21:09 smithi191 ceph-mon[32884]: 6.1e scrub ok 2024-09-10T19:21:09.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:21:09 smithi191 ceph-mon[32884]: 4.6 scrub starts 2024-09-10T19:21:09.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:21:09 smithi191 ceph-mon[32884]: 4.6 scrub ok 2024-09-10T19:21:09.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:21:09 smithi191 ceph-mon[32884]: pgmap v494: 158 pgs: 158 active+clean; 6.7 GiB data, 24 GiB used, 513 GiB / 536 GiB avail; 108 MiB/s wr, 1.73k op/s; 0 B/s, 0 objects/s recovering 2024-09-10T19:21:09.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:21:09 smithi191 ceph-mon[32884]: 1.9 scrub starts 2024-09-10T19:21:09.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:21:09 smithi191 ceph-mon[32884]: 1.9 scrub ok 2024-09-10T19:21:09.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:21:09 smithi191 ceph-mon[32884]: 5.7 scrub starts 2024-09-10T19:21:09.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:21:09 smithi191 ceph-mon[32884]: 5.7 scrub ok 2024-09-10T19:21:09.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:21:09 smithi191 ceph-mon[32884]: 1.7 scrub starts 2024-09-10T19:21:09.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:21:09 smithi191 ceph-mon[32884]: 1.7 scrub ok 2024-09-10T19:21:09.479 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:21:09.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:21:09.480 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:21:09.581 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:21:09.622 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:21:09.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:21:09.623 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:21:09.724 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:21:09.741 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: 68 16 116122 116106 106.697 84.5 0.0170841 0.00936834 2024-09-10T19:21:09.756 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:21:09.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:21:09.757 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:21:09.859 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:21:09.902 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:21:09.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:21:09.903 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:21:09.966 INFO:journalctl@ceph.iscsi.iscsi.a.smithi191.stdout:Sep 10 19:21:09 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-iscsi-iscsi-a[61816]: debug there is no tcmu-runner data available 2024-09-10T19:21:10.004 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:21:10.039 INFO:tasks.thrashosds.thrasher:in_osds: [0, 3, 4, 5, 6, 7] out_osds: [1, 2] dead_osds: [] live_osds: [0, 2, 3, 4, 5, 6, 7, 1] 2024-09-10T19:21:10.039 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-10T19:21:10.039 INFO:tasks.thrashosds.thrasher:Killing osd 1, live_osds are [0, 2, 3, 4, 5, 6, 7, 1] 2024-09-10T19:21:10.040 INFO:tasks.cephadm.osd.1:Stopping osd.1... 2024-09-10T19:21:10.040 DEBUG:teuthology.orchestra.run.smithi183:> sudo systemctl stop ceph-320f6df8-6fa7-11ef-bcea-c7b262605968@osd.1 2024-09-10T19:21:10.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:21:10.047 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:21:10.047 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:21:10.067 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:21:10.148 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:21:10.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:21:10.194 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:21:10.194 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:21:10.195 INFO:teuthology.orchestra.run.smithi183.stderr:noscrub is set 2024-09-10T19:21:10.252 INFO:journalctl@ceph.osd.1.smithi183.stdout:Sep 10 19:21:10 smithi183 systemd[1]: Stopping Ceph osd.1 for 320f6df8-6fa7-11ef-bcea-c7b262605968... 2024-09-10T19:21:10.253 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:21:10 smithi183 ceph-mon[29736]: 9.9 scrub starts 2024-09-10T19:21:10.253 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:21:10 smithi183 ceph-mon[29736]: purged_snaps scrub starts 2024-09-10T19:21:10.253 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:21:10 smithi183 ceph-mon[29736]: purged_snaps scrub ok 2024-09-10T19:21:10.253 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:21:10 smithi183 ceph-mon[29736]: 4.f scrub starts 2024-09-10T19:21:10.253 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:21:10 smithi183 ceph-mon[29736]: 4.f scrub ok 2024-09-10T19:21:10.253 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:21:10 smithi183 ceph-mon[29736]: from='client.? 172.21.15.183:0/3996558970' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-10T19:21:10.253 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:21:10 smithi183 ceph-mon[29736]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-10T19:21:10.253 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:21:10 smithi183 ceph-mon[29736]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T19:21:10.253 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:21:10 smithi183 ceph-mon[41562]: 9.9 scrub starts 2024-09-10T19:21:10.254 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:21:10 smithi183 ceph-mon[41562]: purged_snaps scrub starts 2024-09-10T19:21:10.254 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:21:10 smithi183 ceph-mon[41562]: purged_snaps scrub ok 2024-09-10T19:21:10.254 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:21:10 smithi183 ceph-mon[41562]: 4.f scrub starts 2024-09-10T19:21:10.254 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:21:10 smithi183 ceph-mon[41562]: 4.f scrub ok 2024-09-10T19:21:10.254 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:21:10 smithi183 ceph-mon[41562]: from='client.? 172.21.15.183:0/3996558970' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-10T19:21:10.254 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:21:10 smithi183 ceph-mon[41562]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-10T19:21:10.254 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:21:10 smithi183 ceph-mon[41562]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T19:21:10.295 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:21:10.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:21:10.343 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:21:10.343 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:21:10.444 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:21:10.467 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:21:10 smithi191 ceph-mon[32884]: 9.9 scrub starts 2024-09-10T19:21:10.468 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:21:10 smithi191 ceph-mon[32884]: purged_snaps scrub starts 2024-09-10T19:21:10.468 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:21:10 smithi191 ceph-mon[32884]: purged_snaps scrub ok 2024-09-10T19:21:10.468 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:21:10 smithi191 ceph-mon[32884]: 4.f scrub starts 2024-09-10T19:21:10.468 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:21:10 smithi191 ceph-mon[32884]: 4.f scrub ok 2024-09-10T19:21:10.468 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:21:10 smithi191 ceph-mon[32884]: from='client.? 172.21.15.183:0/3996558970' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-10T19:21:10.468 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:21:10 smithi191 ceph-mon[32884]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-10T19:21:10.468 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:21:10 smithi191 ceph-mon[32884]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T19:21:10.493 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:21:10.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:21:10.494 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:21:10.595 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:21:10.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:21:10.632 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:21:10.633 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:21:10.733 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:21:10.741 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: 69 16 117389 117373 106.298 79.1875 0.00597237 0.009403 2024-09-10T19:21:10.771 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:21:10.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:21:10.772 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:21:10.873 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:21:10.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:21:10.914 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:21:10.914 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:21:11.015 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:21:11.050 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:21:11.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:21:11.051 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:21:11.152 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:21:11.180 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:21:11.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:21:11.181 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:21:11.282 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:21:11.320 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:21:11.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:21:11.321 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:21:11.422 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:21:11.465 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:21:11.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:21:11.466 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:21:11.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:21:11 smithi191 ceph-mon[32884]: 5.a scrub starts 2024-09-10T19:21:11.467 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:21:11 smithi191 ceph-mon[32884]: 5.a scrub ok 2024-09-10T19:21:11.467 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:21:11 smithi191 ceph-mon[32884]: from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:21:11.467 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:21:11 smithi191 ceph-mon[32884]: 6.10 scrub starts 2024-09-10T19:21:11.467 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:21:11 smithi191 ceph-mon[32884]: 6.10 scrub ok 2024-09-10T19:21:11.467 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:21:11 smithi191 ceph-mon[32884]: pgmap v495: 158 pgs: 158 active+clean; 6.7 GiB data, 24 GiB used, 513 GiB / 536 GiB avail; 55 MiB/s wr, 885 op/s; 0 B/s, 0 objects/s recovering 2024-09-10T19:21:11.467 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:21:11 smithi191 ceph-mon[32884]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T19:21:11.467 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:21:11 smithi191 ceph-mon[32884]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-10T19:21:11.467 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:21:11 smithi191 ceph-mon[32884]: osdmap e253: 8 total, 8 up, 6 in 2024-09-10T19:21:11.467 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:21:11 smithi191 ceph-mon[32884]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-10T19:21:11.467 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:21:11 smithi191 ceph-mon[32884]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-10T19:21:11.467 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:21:11 smithi191 ceph-mon[32884]: from='mgr.14556 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-10T19:21:11.467 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:21:11 smithi191 ceph-mon[32884]: from='mgr.14556 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-10T19:21:11.468 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:21:11 smithi191 ceph-mon[32884]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-10T19:21:11.468 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:21:11 smithi191 ceph-mon[32884]: from='mgr.14556 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-10T19:21:11.468 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:21:11 smithi191 ceph-mon[32884]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-10T19:21:11.468 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:21:11 smithi191 ceph-mon[32884]: from='mgr.14556 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-10T19:21:11.468 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:21:11 smithi191 ceph-mon[32884]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-10T19:21:11.468 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:21:11 smithi191 ceph-mon[32884]: from='mgr.14556 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-10T19:21:11.468 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:21:11 smithi191 ceph-mon[32884]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-10T19:21:11.468 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:21:11 smithi191 ceph-mon[32884]: from='mgr.14556 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-10T19:21:11.468 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:21:11 smithi191 ceph-mon[32884]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-10T19:21:11.468 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:21:11 smithi191 ceph-mon[32884]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-10T19:21:11.468 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:21:11 smithi191 ceph-mon[32884]: from='mgr.14556 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-10T19:21:11.468 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:21:11 smithi191 ceph-mon[32884]: from='mgr.14556 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-10T19:21:11.469 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:21:11 smithi191 ceph-mon[32884]: from='mgr.14556 ' entity='mgr.y' 2024-09-10T19:21:11.469 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:21:11 smithi191 ceph-mon[32884]: from='mgr.14556 ' entity='mgr.y' 2024-09-10T19:21:11.469 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:21:11 smithi191 ceph-mon[32884]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T19:21:11.469 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:21:11 smithi191 ceph-mon[32884]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T19:21:11.469 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:21:11 smithi191 ceph-mon[32884]: from='mgr.14556 ' entity='mgr.y' 2024-09-10T19:21:11.469 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:21:11 smithi191 ceph-mon[32884]: purged_snaps scrub starts 2024-09-10T19:21:11.469 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:21:11 smithi191 ceph-mon[32884]: purged_snaps scrub ok 2024-09-10T19:21:11.469 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:21:11 smithi191 ceph-mon[32884]: from='client.? 172.21.15.183:0/254523415' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T19:21:11.567 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:21:11.602 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:21:11.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:21:11.603 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:21:11.681 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Sep 10 19:21:11 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-y[29989]: ::ffff:172.21.15.191 - - [10/Sep/2024:19:21:11] "GET /metrics HTTP/1.1" 200 224355 "" "Prometheus/2.43.0" 2024-09-10T19:21:11.682 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:21:11 smithi183 ceph-mon[41562]: 5.a scrub starts 2024-09-10T19:21:11.682 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:21:11 smithi183 ceph-mon[41562]: 5.a scrub ok 2024-09-10T19:21:11.682 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:21:11 smithi183 ceph-mon[41562]: from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:21:11.682 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:21:11 smithi183 ceph-mon[41562]: 6.10 scrub starts 2024-09-10T19:21:11.682 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:21:11 smithi183 ceph-mon[41562]: 6.10 scrub ok 2024-09-10T19:21:11.682 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:21:11 smithi183 ceph-mon[41562]: pgmap v495: 158 pgs: 158 active+clean; 6.7 GiB data, 24 GiB used, 513 GiB / 536 GiB avail; 55 MiB/s wr, 885 op/s; 0 B/s, 0 objects/s recovering 2024-09-10T19:21:11.682 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:21:11 smithi183 ceph-mon[41562]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T19:21:11.682 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:21:11 smithi183 ceph-mon[41562]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-10T19:21:11.682 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:21:11 smithi183 ceph-mon[41562]: osdmap e253: 8 total, 8 up, 6 in 2024-09-10T19:21:11.682 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:21:11 smithi183 ceph-mon[41562]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-10T19:21:11.682 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:21:11 smithi183 ceph-mon[41562]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-10T19:21:11.682 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:21:11 smithi183 ceph-mon[41562]: from='mgr.14556 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-10T19:21:11.683 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:21:11 smithi183 ceph-mon[41562]: from='mgr.14556 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-10T19:21:11.683 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:21:11 smithi183 ceph-mon[41562]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-10T19:21:11.683 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:21:11 smithi183 ceph-mon[41562]: from='mgr.14556 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-10T19:21:11.683 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:21:11 smithi183 ceph-mon[41562]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-10T19:21:11.683 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:21:11 smithi183 ceph-mon[41562]: from='mgr.14556 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-10T19:21:11.683 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:21:11 smithi183 ceph-mon[41562]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-10T19:21:11.683 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:21:11 smithi183 ceph-mon[41562]: from='mgr.14556 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-10T19:21:11.683 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:21:11 smithi183 ceph-mon[41562]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-10T19:21:11.683 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:21:11 smithi183 ceph-mon[41562]: from='mgr.14556 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-10T19:21:11.683 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:21:11 smithi183 ceph-mon[41562]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-10T19:21:11.683 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:21:11 smithi183 ceph-mon[41562]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-10T19:21:11.683 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:21:11 smithi183 ceph-mon[41562]: from='mgr.14556 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-10T19:21:11.683 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:21:11 smithi183 ceph-mon[41562]: from='mgr.14556 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-10T19:21:11.683 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:21:11 smithi183 ceph-mon[41562]: from='mgr.14556 ' entity='mgr.y' 2024-09-10T19:21:11.684 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:21:11 smithi183 ceph-mon[41562]: from='mgr.14556 ' entity='mgr.y' 2024-09-10T19:21:11.684 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:21:11 smithi183 ceph-mon[41562]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T19:21:11.684 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:21:11 smithi183 ceph-mon[29736]: 5.a scrub starts 2024-09-10T19:21:11.684 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:21:11 smithi183 ceph-mon[29736]: 5.a scrub ok 2024-09-10T19:21:11.684 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:21:11 smithi183 ceph-mon[29736]: from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:21:11.684 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:21:11 smithi183 ceph-mon[29736]: 6.10 scrub starts 2024-09-10T19:21:11.684 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:21:11 smithi183 ceph-mon[29736]: 6.10 scrub ok 2024-09-10T19:21:11.684 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:21:11 smithi183 ceph-mon[29736]: pgmap v495: 158 pgs: 158 active+clean; 6.7 GiB data, 24 GiB used, 513 GiB / 536 GiB avail; 55 MiB/s wr, 885 op/s; 0 B/s, 0 objects/s recovering 2024-09-10T19:21:11.684 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:21:11 smithi183 ceph-mon[29736]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T19:21:11.684 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:21:11 smithi183 ceph-mon[29736]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-10T19:21:11.685 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:21:11 smithi183 ceph-mon[29736]: osdmap e253: 8 total, 8 up, 6 in 2024-09-10T19:21:11.685 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:21:11 smithi183 ceph-mon[29736]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-10T19:21:11.685 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:21:11 smithi183 ceph-mon[29736]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-10T19:21:11.685 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:21:11 smithi183 ceph-mon[29736]: from='mgr.14556 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-10T19:21:11.685 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:21:11 smithi183 ceph-mon[29736]: from='mgr.14556 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-10T19:21:11.685 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:21:11 smithi183 ceph-mon[29736]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-10T19:21:11.685 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:21:11 smithi183 ceph-mon[29736]: from='mgr.14556 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-10T19:21:11.685 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:21:11 smithi183 ceph-mon[29736]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-10T19:21:11.685 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:21:11 smithi183 ceph-mon[29736]: from='mgr.14556 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-10T19:21:11.685 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:21:11 smithi183 ceph-mon[29736]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-10T19:21:11.685 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:21:11 smithi183 ceph-mon[29736]: from='mgr.14556 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-10T19:21:11.685 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:21:11 smithi183 ceph-mon[29736]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-10T19:21:11.685 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:21:11 smithi183 ceph-mon[29736]: from='mgr.14556 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-10T19:21:11.685 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:21:11 smithi183 ceph-mon[29736]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-10T19:21:11.686 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:21:11 smithi183 ceph-mon[29736]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-10T19:21:11.686 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:21:11 smithi183 ceph-mon[29736]: from='mgr.14556 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-10T19:21:11.686 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:21:11 smithi183 ceph-mon[29736]: from='mgr.14556 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-10T19:21:11.686 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:21:11 smithi183 ceph-mon[29736]: from='mgr.14556 ' entity='mgr.y' 2024-09-10T19:21:11.686 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:21:11 smithi183 ceph-mon[29736]: from='mgr.14556 ' entity='mgr.y' 2024-09-10T19:21:11.686 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:21:11 smithi183 ceph-mon[29736]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T19:21:11.686 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:21:11 smithi183 ceph-mon[41562]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T19:21:11.686 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:21:11 smithi183 ceph-mon[41562]: from='mgr.14556 ' entity='mgr.y' 2024-09-10T19:21:11.686 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:21:11 smithi183 ceph-mon[41562]: purged_snaps scrub starts 2024-09-10T19:21:11.686 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:21:11 smithi183 ceph-mon[41562]: purged_snaps scrub ok 2024-09-10T19:21:11.687 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:21:11 smithi183 ceph-mon[41562]: from='client.? 172.21.15.183:0/254523415' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T19:21:11.687 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:21:11 smithi183 ceph-mon[29736]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T19:21:11.687 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:21:11 smithi183 ceph-mon[29736]: from='mgr.14556 ' entity='mgr.y' 2024-09-10T19:21:11.687 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:21:11 smithi183 ceph-mon[29736]: purged_snaps scrub starts 2024-09-10T19:21:11.687 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:21:11 smithi183 ceph-mon[29736]: purged_snaps scrub ok 2024-09-10T19:21:11.687 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:21:11 smithi183 ceph-mon[29736]: from='client.? 172.21.15.183:0/254523415' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T19:21:11.700 INFO:teuthology.orchestra.run.smithi183.stdout:osd.0: {} 2024-09-10T19:21:11.700 INFO:teuthology.orchestra.run.smithi183.stdout:osd.1: {} 2024-09-10T19:21:11.700 INFO:teuthology.orchestra.run.smithi183.stdout:osd.2: {} 2024-09-10T19:21:11.700 INFO:teuthology.orchestra.run.smithi183.stdout:osd.3: {} 2024-09-10T19:21:11.700 INFO:teuthology.orchestra.run.smithi183.stdout:osd.4: {} 2024-09-10T19:21:11.701 INFO:teuthology.orchestra.run.smithi183.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-10T19:21:11.701 INFO:teuthology.orchestra.run.smithi183.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-10T19:21:11.701 INFO:teuthology.orchestra.run.smithi183.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-10T19:21:11.701 INFO:teuthology.orchestra.run.smithi183.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-10T19:21:11.701 INFO:teuthology.orchestra.run.smithi183.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-10T19:21:11.701 INFO:teuthology.orchestra.run.smithi183.stdout:osd.5: {} 2024-09-10T19:21:11.701 INFO:teuthology.orchestra.run.smithi183.stdout:osd.6: {} 2024-09-10T19:21:11.702 INFO:teuthology.orchestra.run.smithi183.stdout:osd.7: {} 2024-09-10T19:21:11.702 INFO:teuthology.orchestra.run.smithi183.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-10T19:21:11.702 INFO:teuthology.orchestra.run.smithi183.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-10T19:21:11.702 INFO:teuthology.orchestra.run.smithi183.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-10T19:21:11.703 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:21:11.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:21:11.733 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:21:11.733 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:21:11.741 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: 70 16 118621 118605 105.879 77 0.00905969 0.00944067 2024-09-10T19:21:11.835 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:21:11.864 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:21:11.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:21:11.865 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:21:11.966 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:21:12.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:21:12.009 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:21:12.009 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:21:12.109 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:21:12.138 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:21:12.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:21:12.139 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:21:12.239 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:21:12.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:21:12.270 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:21:12.270 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:21:12.371 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:21:12.399 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:21:12.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:21:12.400 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:21:12.465 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:21:12 smithi191 ceph-mon[32884]: Adjusting osd_memory_target on smithi191 to 3248M 2024-09-10T19:21:12.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:21:12 smithi191 ceph-mon[32884]: Adjusting osd_memory_target on smithi183 to 3248M 2024-09-10T19:21:12.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:21:12 smithi191 ceph-mon[32884]: osdmap e254: 8 total, 8 up, 6 in 2024-09-10T19:21:12.501 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:21:12.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:21:12.531 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:21:12.532 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:21:12.632 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:21:12.661 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:21:12.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:21:12.662 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:21:12.681 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:21:12 smithi183 ceph-mon[29736]: Adjusting osd_memory_target on smithi191 to 3248M 2024-09-10T19:21:12.681 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:21:12 smithi183 ceph-mon[29736]: Adjusting osd_memory_target on smithi183 to 3248M 2024-09-10T19:21:12.681 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:21:12 smithi183 ceph-mon[29736]: osdmap e254: 8 total, 8 up, 6 in 2024-09-10T19:21:12.682 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:21:12 smithi183 ceph-mon[41562]: Adjusting osd_memory_target on smithi191 to 3248M 2024-09-10T19:21:12.682 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:21:12 smithi183 ceph-mon[41562]: Adjusting osd_memory_target on smithi183 to 3248M 2024-09-10T19:21:12.682 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:21:12 smithi183 ceph-mon[41562]: osdmap e254: 8 total, 8 up, 6 in 2024-09-10T19:21:12.741 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: 71 16 120057 120041 105.652 89.75 0.0119557 0.00946087 2024-09-10T19:21:12.763 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:21:12.802 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:21:12.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:21:12.803 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:21:12.904 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:21:12.941 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:21:12.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:21:12.943 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:21:13.043 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:21:13.073 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:21:13.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:21:13.074 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:21:13.175 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:21:13.213 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:21:13.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:21:13.214 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:21:13.315 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:21:13.344 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:21:13 smithi183 ceph-mon[29736]: pgmap v498: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 7.1 GiB data, 25 GiB used, 512 GiB / 536 GiB avail; 115 MiB/s wr, 1.83k op/s; 0 B/s, 0 objects/s recovering 2024-09-10T19:21:13.345 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:21:13 smithi183 ceph-mon[41562]: pgmap v498: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 7.1 GiB data, 25 GiB used, 512 GiB / 536 GiB avail; 115 MiB/s wr, 1.83k op/s; 0 B/s, 0 objects/s recovering 2024-09-10T19:21:13.351 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:21:13.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:21:13.353 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:21:13.453 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:21:13.465 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:21:13 smithi191 ceph-mon[32884]: pgmap v498: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 7.1 GiB data, 25 GiB used, 512 GiB / 536 GiB avail; 115 MiB/s wr, 1.83k op/s; 0 B/s, 0 objects/s recovering 2024-09-10T19:21:13.481 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:21:13.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:21:13.482 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:21:13.583 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:21:13.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:21:13.623 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:21:13.624 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:21:13.724 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:21:13.741 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: 72 16 121496 121480 105.434 89.9375 0.00700387 0.00948051 2024-09-10T19:21:13.766 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:21:13.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:21:13.767 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:21:13.868 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:21:13.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:21:13.908 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:21:13.908 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:21:14.009 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:21:14.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:21:14.040 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:21:14.040 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:21:14.141 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:21:14.176 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:21:14.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:21:14.180 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:21:14.281 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:21:14.324 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:21:14.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:21:14.325 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:21:14.425 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:21:14.454 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:21:14.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:21:14.455 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:21:14.555 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:21:14.583 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:21:14.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:21:14.584 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:21:14.685 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:21:14.726 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:21:14.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:21:14.727 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:21:14.741 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: 73 16 122803 122787 105.109 81.6875 0.00793351 0.0095096 2024-09-10T19:21:14.828 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:21:14.858 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:21:14.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:21:14.859 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:21:14.931 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:21:14 smithi183 ceph-mon[29736]: from='mgr.14556 ' entity='mgr.y' 2024-09-10T19:21:14.931 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:21:14 smithi183 ceph-mon[29736]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:21:14.931 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:21:14 smithi183 ceph-mon[29736]: pgmap v499: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 7.1 GiB data, 25 GiB used, 512 GiB / 536 GiB avail; 54 MiB/s wr, 865 op/s 2024-09-10T19:21:14.931 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:21:14 smithi183 ceph-mon[41562]: from='mgr.14556 ' entity='mgr.y' 2024-09-10T19:21:14.932 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:21:14 smithi183 ceph-mon[41562]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:21:14.932 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:21:14 smithi183 ceph-mon[41562]: pgmap v499: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 7.1 GiB data, 25 GiB used, 512 GiB / 536 GiB avail; 54 MiB/s wr, 865 op/s 2024-09-10T19:21:14.960 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:21:14.966 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:21:14 smithi191 ceph-mon[32884]: from='mgr.14556 ' entity='mgr.y' 2024-09-10T19:21:14.966 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:21:14 smithi191 ceph-mon[32884]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:21:14.966 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:21:14 smithi191 ceph-mon[32884]: pgmap v499: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 7.1 GiB data, 25 GiB used, 512 GiB / 536 GiB avail; 54 MiB/s wr, 865 op/s 2024-09-10T19:21:14.996 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:21:14.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:21:14.997 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:21:15.098 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:21:15.136 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:21:15.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:21:15.137 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:21:15.238 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:21:15.266 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:21:15.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:21:15.267 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:21:15.368 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:21:15.405 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:21:15.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:21:15.406 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:21:15.506 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:21:15.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:21:15.548 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:21:15.549 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:21:15.649 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:21:15.690 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:21:15.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:21:15.691 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:21:15.741 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: 74 16 124205 124189 104.872 87.625 0.00549153 0.00953106 2024-09-10T19:21:15.792 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:21:15.824 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:21:15.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:21:15.825 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:21:15.926 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:21:15.968 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:21:15.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:21:15.969 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:21:16.070 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:21:16.100 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:21:16.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:21:16.101 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:21:16.149 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-10T19:21:16.202 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:21:16.234 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:21:16.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:21:16.236 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:21:16.336 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:21:16.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:21:16.372 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:21:16.372 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:21:16.473 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:21:16.512 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:21:16.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:21:16.513 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:21:16.614 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:21:16.649 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:21:16.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:21:16.650 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:21:16.741 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: 75 16 125408 125392 104.476 75.1875 0.0163287 0.00956744 2024-09-10T19:21:16.751 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:21:16.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:21:16.790 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:21:16.791 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:21:16.891 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:21:16.923 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:21:16.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:21:16.924 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:21:16.931 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:21:16 smithi183 ceph-mon[29736]: pgmap v500: 158 pgs: 158 active+clean; 7.5 GiB data, 26 GiB used, 510 GiB / 536 GiB avail; 106 MiB/s wr, 1.70k op/s 2024-09-10T19:21:16.931 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:21:16 smithi183 ceph-mon[29736]: purged_snaps scrub starts 2024-09-10T19:21:16.931 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:21:16 smithi183 ceph-mon[29736]: purged_snaps scrub ok 2024-09-10T19:21:16.931 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:21:16 smithi183 ceph-mon[41562]: pgmap v500: 158 pgs: 158 active+clean; 7.5 GiB data, 26 GiB used, 510 GiB / 536 GiB avail; 106 MiB/s wr, 1.70k op/s 2024-09-10T19:21:16.931 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:21:16 smithi183 ceph-mon[41562]: purged_snaps scrub starts 2024-09-10T19:21:16.931 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:21:16 smithi183 ceph-mon[41562]: purged_snaps scrub ok 2024-09-10T19:21:17.025 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:21:17.054 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:21:17.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:21:17.055 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:21:17.156 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:21:17.191 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:21:17.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:21:17.192 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:21:17.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:21:16 smithi191 ceph-mon[32884]: pgmap v500: 158 pgs: 158 active+clean; 7.5 GiB data, 26 GiB used, 510 GiB / 536 GiB avail; 106 MiB/s wr, 1.70k op/s 2024-09-10T19:21:17.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:21:16 smithi191 ceph-mon[32884]: purged_snaps scrub starts 2024-09-10T19:21:17.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:21:16 smithi191 ceph-mon[32884]: purged_snaps scrub ok 2024-09-10T19:21:17.292 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:21:17.332 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:21:17.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:21:17.333 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:21:17.351 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:21:17.434 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:21:17.474 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:21:17.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:21:17.475 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:21:17.576 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:21:17.609 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:21:17.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:21:17.610 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:21:17.711 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:21:17.716 INFO:journalctl@ceph.mgr.x.smithi191.stdout:Sep 10 19:21:17 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-x[34967]: ::ffff:172.21.15.191 - - [10/Sep/2024:19:21:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T19:21:17.741 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: 76 16 126826 126810 104.268 88.625 0.0105972 0.00958639 2024-09-10T19:21:17.750 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:21:17.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:21:17.751 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:21:17.836 INFO:journalctl@ceph.osd.1.smithi183.stdout:Sep 10 19:21:17 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd-1[104866]: 2024-09-10T19:21:17.704+0000 7f3b7c1ea640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-10T19:21:17.837 INFO:journalctl@ceph.osd.1.smithi183.stdout:Sep 10 19:21:17 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd-1[104866]: 2024-09-10T19:21:17.705+0000 7f3b7c1ea640 -1 osd.1 255 *** Got signal Terminated *** 2024-09-10T19:21:17.837 INFO:journalctl@ceph.osd.1.smithi183.stdout:Sep 10 19:21:17 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd-1[104866]: 2024-09-10T19:21:17.705+0000 7f3b7c1ea640 -1 osd.1 255 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-10T19:21:17.853 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:21:17.891 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:21:17.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:21:17.892 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:21:17.985 INFO:teuthology.orchestra.run.smithi183.stdout:{ 2024-09-10T19:21:17.985 INFO:teuthology.orchestra.run.smithi183.stdout: "ops": [], 2024-09-10T19:21:17.985 INFO:teuthology.orchestra.run.smithi183.stdout: "complaint_time": 30, 2024-09-10T19:21:17.985 INFO:teuthology.orchestra.run.smithi183.stdout: "num_blocked_ops": 0 2024-09-10T19:21:17.985 INFO:teuthology.orchestra.run.smithi183.stdout:} 2024-09-10T19:21:17.993 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:21:18.028 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:21:18.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:21:18.029 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:21:18.130 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:21:18.171 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:21:18.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:21:18.172 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:21:18.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:21:17 smithi183 ceph-mon[29736]: osdmap e255: 8 total, 8 up, 6 in 2024-09-10T19:21:18.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:21:17 smithi183 ceph-mon[29736]: osd.1 marked itself down and dead 2024-09-10T19:21:18.181 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:21:17 smithi183 ceph-mon[41562]: osdmap e255: 8 total, 8 up, 6 in 2024-09-10T19:21:18.181 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:21:17 smithi183 ceph-mon[41562]: osd.1 marked itself down and dead 2024-09-10T19:21:18.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:21:17 smithi191 ceph-mon[32884]: osdmap e255: 8 total, 8 up, 6 in 2024-09-10T19:21:18.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:21:17 smithi191 ceph-mon[32884]: osd.1 marked itself down and dead 2024-09-10T19:21:18.273 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:21:18.295 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:21:18.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:21:18.296 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:21:18.397 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:21:18.433 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:21:18.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:21:18.434 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:21:18.534 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:21:18.562 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:21:18.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:21:18.563 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:21:18.664 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:21:18.693 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:21:18.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:21:18.694 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:21:18.711 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-10T19:21:18.741 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: 77 16 128219 128203 104.044 87.0625 0.0181706 0.00960716 2024-09-10T19:21:18.795 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:21:18.830 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:21:18.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:21:18.831 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:21:18.932 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:21:18.965 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:21:18.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:21:18.966 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:21:19.021 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:21:18 smithi183 ceph-mon[29736]: purged_snaps scrub starts 2024-09-10T19:21:19.022 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:21:18 smithi183 ceph-mon[29736]: purged_snaps scrub ok 2024-09-10T19:21:19.022 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:21:18 smithi183 ceph-mon[29736]: pgmap v502: 158 pgs: 158 active+clean; 7.5 GiB data, 26 GiB used, 510 GiB / 536 GiB avail; 672 B/s rd, 55 MiB/s wr, 878 op/s 2024-09-10T19:21:19.022 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:21:18 smithi183 ceph-mon[29736]: osdmap e256: 8 total, 7 up, 6 in 2024-09-10T19:21:19.022 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:21:18 smithi183 ceph-mon[41562]: purged_snaps scrub starts 2024-09-10T19:21:19.022 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:21:18 smithi183 ceph-mon[41562]: purged_snaps scrub ok 2024-09-10T19:21:19.022 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:21:18 smithi183 ceph-mon[41562]: pgmap v502: 158 pgs: 158 active+clean; 7.5 GiB data, 26 GiB used, 510 GiB / 536 GiB avail; 672 B/s rd, 55 MiB/s wr, 878 op/s 2024-09-10T19:21:19.022 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:21:18 smithi183 ceph-mon[41562]: osdmap e256: 8 total, 7 up, 6 in 2024-09-10T19:21:19.054 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph daemon osd.2 dump_historic_ops 2024-09-10T19:21:19.067 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:21:19.124 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:21:19.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:21:19.126 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:21:19.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:21:18 smithi191 ceph-mon[32884]: purged_snaps scrub starts 2024-09-10T19:21:19.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:21:18 smithi191 ceph-mon[32884]: purged_snaps scrub ok 2024-09-10T19:21:19.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:21:18 smithi191 ceph-mon[32884]: pgmap v502: 158 pgs: 158 active+clean; 7.5 GiB data, 26 GiB used, 510 GiB / 536 GiB avail; 672 B/s rd, 55 MiB/s wr, 878 op/s 2024-09-10T19:21:19.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:21:18 smithi191 ceph-mon[32884]: osdmap e256: 8 total, 7 up, 6 in 2024-09-10T19:21:19.229 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:21:19.265 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:21:19.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:21:19.266 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:21:19.346 INFO:journalctl@ceph.osd.1.smithi183.stdout:Sep 10 19:21:19 smithi183 podman[151406]: 2024-09-10 19:21:19.020057869 +0000 UTC m=+1.666487015 container died 01ff17cbf9a0a513ce111980a9e1f53d36e11077aaa39457aaeaeba3cb5b570a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3bfeaf4417585c821cab9ca76e4d95f390b0e769807122a45463640820b5cd16, name=ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd-1, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.vendor=CentOS, RELEASE=wip-guits-quincy-2024-09-09-1810-quincy-5be5739, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-10T19:21:19.368 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:21:19.396 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:21:19.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:21:19.397 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:21:19.497 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:21:19.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:21:19.537 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:21:19.537 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:21:19.638 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:21:19.663 INFO:journalctl@ceph.osd.1.smithi183.stdout:Sep 10 19:21:19 smithi183 podman[151406]: 2024-09-10 19:21:19.435264999 +0000 UTC m=+2.081694146 container cleanup 01ff17cbf9a0a513ce111980a9e1f53d36e11077aaa39457aaeaeba3cb5b570a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3bfeaf4417585c821cab9ca76e4d95f390b0e769807122a45463640820b5cd16, name=ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd-1, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, RELEASE=wip-guits-quincy-2024-09-09-1810-quincy-5be5739, CEPH_POINT_RELEASE=) 2024-09-10T19:21:19.663 INFO:journalctl@ceph.osd.1.smithi183.stdout:Sep 10 19:21:19 smithi183 bash[151406]: ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd-1 2024-09-10T19:21:19.676 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:21:19.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:21:19.677 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:21:19.745 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: 78 16 129653 129637 103.857 89.625 0.0223621 0.00962453 2024-09-10T19:21:19.767 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:21:19.778 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:21:19.826 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:21:19.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:21:19.827 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:21:19.928 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:21:19.932 INFO:journalctl@ceph.osd.1.smithi183.stdout:Sep 10 19:21:19 smithi183 podman[151530]: 2024-09-10 19:21:19.661980923 +0000 UTC m=+1.270195776 container remove 01ff17cbf9a0a513ce111980a9e1f53d36e11077aaa39457aaeaeba3cb5b570a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3bfeaf4417585c821cab9ca76e4d95f390b0e769807122a45463640820b5cd16, name=ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd-1, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, RELEASE=wip-guits-quincy-2024-09-09-1810-quincy-5be5739, GIT_BRANCH=HEAD, ceph=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-10T19:21:19.959 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:21:19.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:21:19.960 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:21:19.966 INFO:journalctl@ceph.iscsi.iscsi.a.smithi191.stdout:Sep 10 19:21:19 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-iscsi-iscsi-a[61816]: debug there is no tcmu-runner data available 2024-09-10T19:21:20.061 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:21:20.095 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:21:20.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:21:20.096 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:21:20.197 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:21:20.234 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:21:20.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:21:20.236 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:21:20.336 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:21:20.365 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:21:20.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:21:20.366 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:21:20.467 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:21:20.503 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:21:20.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:21:20.504 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:21:20.605 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:21:20.640 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:21:20.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:21:20.641 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:21:20.742 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:21:20.743 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: 79 16 131071 131055 103.664 88.625 0.0116153 0.00964265 2024-09-10T19:21:20.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:21:20.778 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:21:20.778 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:21:20.879 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:21:20.908 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:21:20.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:21:20.909 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:21:21.010 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:21:21.040 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:21:21.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:21:21.041 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:21:21.141 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:21:21.174 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:21:21.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:21:21.175 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:21:21.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:21:20 smithi183 ceph-mon[29736]: from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:21:21.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:21:20 smithi183 ceph-mon[29736]: pgmap v504: 158 pgs: 158 active+clean; 7.5 GiB data, 26 GiB used, 510 GiB / 536 GiB avail; 639 B/s rd, 52 MiB/s wr, 836 op/s 2024-09-10T19:21:21.181 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:21:20 smithi183 ceph-mon[41562]: from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:21:21.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:21:20 smithi183 ceph-mon[41562]: pgmap v504: 158 pgs: 158 active+clean; 7.5 GiB data, 26 GiB used, 510 GiB / 536 GiB avail; 639 B/s rd, 52 MiB/s wr, 836 op/s 2024-09-10T19:21:21.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:21:20 smithi191 ceph-mon[32884]: from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:21:21.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:21:20 smithi191 ceph-mon[32884]: pgmap v504: 158 pgs: 158 active+clean; 7.5 GiB data, 26 GiB used, 510 GiB / 536 GiB avail; 639 B/s rd, 52 MiB/s wr, 836 op/s 2024-09-10T19:21:21.276 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:21:21.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:21:21.307 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:21:21.308 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:21:21.408 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:21:21.441 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:21:21.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:21:21.442 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:21:21.542 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:21:21.560 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Sep 10 19:21:21 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-y[29989]: ::ffff:172.21.15.191 - - [10/Sep/2024:19:21:21] "GET /metrics HTTP/1.1" 200 224384 "" "Prometheus/2.43.0" 2024-09-10T19:21:21.571 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:21:21.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:21:21.572 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:21:21.673 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:21:21.714 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:21:21.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:21:21.715 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:21:21.743 INFO:tasks.radosbench.radosbench.0.smithi183.stdout:2024-09-10T19:21:21.743254+0000 min lat: 0.00139047 max lat: 0.109868 avg lat: 0.0096602 2024-09-10T19:21:21.744 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-10T19:21:21.744 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: 80 16 132490 132474 103.477 88.6875 0.0139314 0.0096602 2024-09-10T19:21:21.785 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:21:21.815 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:21:21.848 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:21:21.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:21:21.849 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:21:21.950 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:21:21.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:21:21.987 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:21:21.987 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:21:22.088 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:21:22.121 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:21:22.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:21:22.122 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:21:22.223 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:21:22.256 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:21:22.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:21:22.258 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:21:22.358 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:21:22.388 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:21:22.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:21:22.389 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:21:22.490 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:21:22.526 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:21:22.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:21:22.527 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:21:22.628 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:21:22.664 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:21:22.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:21:22.665 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:21:22.744 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: 81 16 133866 133850 103.261 86 0.00475592 0.00967986 2024-09-10T19:21:22.766 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:21:22.796 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:21:22.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:21:22.797 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:21:22.897 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:21:22.948 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:21:22.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:21:22.949 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:21:23.001 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:21:22 smithi183 ceph-mon[41562]: pgmap v505: 158 pgs: 158 active+clean; 7.9 GiB data, 27 GiB used, 509 GiB / 536 GiB avail; 1.2 KiB/s rd, 106 MiB/s wr, 1.70k op/s 2024-09-10T19:21:23.001 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:21:22 smithi183 ceph-mon[29736]: pgmap v505: 158 pgs: 158 active+clean; 7.9 GiB data, 27 GiB used, 509 GiB / 536 GiB avail; 1.2 KiB/s rd, 106 MiB/s wr, 1.70k op/s 2024-09-10T19:21:23.051 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:21:23.088 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:21:23.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:21:23.089 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:21:23.190 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:21:23.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:21:22 smithi191 ceph-mon[32884]: pgmap v505: 158 pgs: 158 active+clean; 7.9 GiB data, 27 GiB used, 509 GiB / 536 GiB avail; 1.2 KiB/s rd, 106 MiB/s wr, 1.70k op/s 2024-09-10T19:21:23.221 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:21:23.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:21:23.222 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:21:23.322 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:21:23.365 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:21:23.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:21:23.366 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:21:23.467 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:21:23.504 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:21:23.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:21:23.505 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:21:23.605 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:21:23.645 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:21:23.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:21:23.646 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:21:23.744 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: 82 16 135265 135249 103.068 87.4375 0.0175898 0.00969837 2024-09-10T19:21:23.746 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:21:23.787 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:21:23.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:21:23.788 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:21:23.875 INFO:teuthology.orchestra.run.smithi183.stderr:nodeep-scrub is set 2024-09-10T19:21:23.889 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:21:23.926 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:21:23.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:21:23.927 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:21:24.028 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:21:24.064 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:21:24.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:21:24.065 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:21:24.166 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:21:24.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:21:23 smithi183 ceph-mon[29736]: from='client.? 172.21.15.183:0/3827717065' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-10T19:21:24.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:21:23 smithi183 ceph-mon[29736]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-10T19:21:24.181 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:21:23 smithi183 ceph-mon[41562]: from='client.? 172.21.15.183:0/3827717065' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-10T19:21:24.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:21:23 smithi183 ceph-mon[41562]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-10T19:21:24.202 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:21:24.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:21:24.203 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:21:24.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:21:23 smithi191 ceph-mon[32884]: from='client.? 172.21.15.183:0/3827717065' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-10T19:21:24.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:21:23 smithi191 ceph-mon[32884]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-10T19:21:24.304 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:21:24.337 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:21:24.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:21:24.338 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:21:24.439 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:21:24.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:21:24.469 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:21:24.469 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:21:24.571 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:21:24.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:21:24.613 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:21:24.613 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:21:24.672 INFO:journalctl@ceph.osd.1.smithi183.stdout:Sep 10 19:21:24 smithi183 podman[152000]: 2024-09-10 19:21:24.353042029 +0000 UTC m=+2.561223601 container died a2f6029bbe0cfc8b8e68c9b2e23c241dca85fe3b9af8a4800a6ac8fa3851bd9c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588, name=charming_gould, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, org.label-schema.build-date=20240731, GIT_CLEAN=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, RELEASE=wip-guits-quincy-2024-09-09-1810-quincy-5be5739) 2024-09-10T19:21:24.714 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:21:24.744 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: 83 16 136667 136651 102.882 87.625 0.0077673 0.00971596 2024-09-10T19:21:24.748 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:21:24.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:21:24.749 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:21:24.840 INFO:teuthology.orchestra.run.smithi183.stdout:osd.0: {} 2024-09-10T19:21:24.840 INFO:teuthology.orchestra.run.smithi183.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-10T19:21:24.840 INFO:teuthology.orchestra.run.smithi183.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-09-10T19:21:24.850 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:21:24.869 INFO:teuthology.orchestra.run.smithi183.stdout:osd.2: {} 2024-09-10T19:21:24.869 INFO:teuthology.orchestra.run.smithi183.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-10T19:21:24.881 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:21:24.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:21:24.882 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:21:24.894 INFO:teuthology.orchestra.run.smithi183.stdout:osd.3: {} 2024-09-10T19:21:24.894 INFO:teuthology.orchestra.run.smithi183.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-10T19:21:24.924 INFO:teuthology.orchestra.run.smithi183.stdout:osd.4: {} 2024-09-10T19:21:24.925 INFO:teuthology.orchestra.run.smithi183.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-10T19:21:24.925 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:21:24 smithi183 ceph-mon[29736]: pgmap v506: 158 pgs: 158 active+clean; 7.9 GiB data, 27 GiB used, 509 GiB / 536 GiB avail; 639 B/s rd, 54 MiB/s wr, 861 op/s 2024-09-10T19:21:24.925 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:21:24 smithi183 ceph-mon[29736]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T19:21:24.925 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:21:24 smithi183 ceph-mon[29736]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-10T19:21:24.925 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:21:24 smithi183 ceph-mon[29736]: osdmap e257: 8 total, 7 up, 6 in 2024-09-10T19:21:24.925 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:21:24 smithi183 ceph-mon[29736]: from='client.? 172.21.15.183:0/1151538932' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T19:21:24.925 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:21:24 smithi183 ceph-mon[29736]: osdmap e258: 8 total, 7 up, 6 in 2024-09-10T19:21:24.925 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:21:24 smithi183 ceph-mon[41562]: pgmap v506: 158 pgs: 158 active+clean; 7.9 GiB data, 27 GiB used, 509 GiB / 536 GiB avail; 639 B/s rd, 54 MiB/s wr, 861 op/s 2024-09-10T19:21:24.926 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:21:24 smithi183 ceph-mon[41562]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T19:21:24.926 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:21:24 smithi183 ceph-mon[41562]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-10T19:21:24.926 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:21:24 smithi183 ceph-mon[41562]: osdmap e257: 8 total, 7 up, 6 in 2024-09-10T19:21:24.926 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:21:24 smithi183 ceph-mon[41562]: from='client.? 172.21.15.183:0/1151538932' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T19:21:24.926 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:21:24 smithi183 ceph-mon[41562]: osdmap e258: 8 total, 7 up, 6 in 2024-09-10T19:21:24.953 INFO:teuthology.orchestra.run.smithi183.stdout:osd.5: {} 2024-09-10T19:21:24.953 INFO:teuthology.orchestra.run.smithi183.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-10T19:21:24.978 INFO:teuthology.orchestra.run.smithi183.stdout:osd.6: {} 2024-09-10T19:21:24.979 INFO:teuthology.orchestra.run.smithi183.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-10T19:21:24.983 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:21:25.009 INFO:teuthology.orchestra.run.smithi183.stdout:osd.7: {} 2024-09-10T19:21:25.009 INFO:teuthology.orchestra.run.smithi183.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-10T19:21:25.014 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:21:25.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:21:25.015 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:21:25.115 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:21:25.142 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:21:25.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:21:25.143 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:21:25.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:21:24 smithi191 ceph-mon[32884]: pgmap v506: 158 pgs: 158 active+clean; 7.9 GiB data, 27 GiB used, 509 GiB / 536 GiB avail; 639 B/s rd, 54 MiB/s wr, 861 op/s 2024-09-10T19:21:25.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:21:24 smithi191 ceph-mon[32884]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T19:21:25.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:21:24 smithi191 ceph-mon[32884]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-10T19:21:25.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:21:24 smithi191 ceph-mon[32884]: osdmap e257: 8 total, 7 up, 6 in 2024-09-10T19:21:25.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:21:24 smithi191 ceph-mon[32884]: from='client.? 172.21.15.183:0/1151538932' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T19:21:25.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:21:24 smithi191 ceph-mon[32884]: osdmap e258: 8 total, 7 up, 6 in 2024-09-10T19:21:25.243 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:21:25.277 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:21:25.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:21:25.278 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:21:25.378 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:21:25.414 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:21:25.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:21:25.415 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:21:25.516 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:21:25.545 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:21:25.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:21:25.546 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:21:25.599 INFO:journalctl@ceph.osd.1.smithi183.stdout:Sep 10 19:21:25 smithi183 podman[152517]: 2024-09-10 19:21:25.437158238 +0000 UTC m=+0.025863685 image pull 4c55dd4096115e48502f368cff1eab06769f71f030e1692e2611bbc6c6f637e9 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3bfeaf4417585c821cab9ca76e4d95f390b0e769807122a45463640820b5cd16 2024-09-10T19:21:25.647 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:21:25.675 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:21:25.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:21:25.676 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:21:25.744 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: 84 16 138109 138093 102.73 90.125 0.00678049 0.0097303 2024-09-10T19:21:25.777 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:21:25.805 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:21:25.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:21:25.806 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:21:25.907 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:21:25.948 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:21:25.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:21:25.949 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:21:26.050 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:21:26.090 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:21:26.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:21:26.091 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:21:26.191 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:21:26.228 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:21:26.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:21:26.230 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:21:26.330 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:21:26.360 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:21:26.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:21:26.361 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:21:26.462 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:21:26.490 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:21:26.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:21:26.491 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:21:26.591 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:21:26.618 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:21:26.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:21:26.619 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:21:26.717 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-10T19:21:26.718 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-10T19:21:26.720 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:21:26.744 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: 85 16 139565 139549 102.592 91 0.0190923 0.00974349 2024-09-10T19:21:26.757 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:21:26.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:21:26.758 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:21:26.859 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:21:26.904 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:21:26.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:21:26.905 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:21:27.006 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:21:27.018 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:21:26 smithi183 ceph-mon[41562]: pgmap v509: 158 pgs: 158 active+clean; 8.4 GiB data, 29 GiB used, 508 GiB / 536 GiB avail; 1.3 KiB/s rd, 109 MiB/s wr, 1.75k op/s 2024-09-10T19:21:27.019 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:21:26 smithi183 ceph-mon[29736]: pgmap v509: 158 pgs: 158 active+clean; 8.4 GiB data, 29 GiB used, 508 GiB / 536 GiB avail; 1.3 KiB/s rd, 109 MiB/s wr, 1.75k op/s 2024-09-10T19:21:27.050 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:21:27.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:21:27.051 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:21:27.153 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:21:27.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:21:27.187 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:21:27.187 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:21:27.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:21:26 smithi191 ceph-mon[32884]: pgmap v509: 158 pgs: 158 active+clean; 8.4 GiB data, 29 GiB used, 508 GiB / 536 GiB avail; 1.3 KiB/s rd, 109 MiB/s wr, 1.75k op/s 2024-09-10T19:21:27.288 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:21:27.324 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:21:27.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:21:27.325 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:21:27.384 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph osd unset noscrub 2024-09-10T19:21:27.426 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:21:27.462 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:21:27.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:21:27.463 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:21:27.564 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:21:27.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:21:27.596 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:21:27.596 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:21:27.673 INFO:journalctl@ceph.osd.1.smithi183.stdout:Sep 10 19:21:27 smithi183 podman[152517]: 2024-09-10 19:21:27.293845495 +0000 UTC m=+1.882550941 container create f9c0e821b88aeea0d11521203c514dc50c6a0d98bd07192b7f1a57bf71075b3b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3bfeaf4417585c821cab9ca76e4d95f390b0e769807122a45463640820b5cd16, name=ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd-1-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, ceph=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=wip-guits-quincy-2024-09-09-1810-quincy-5be5739, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, org.label-schema.build-date=20240731) 2024-09-10T19:21:27.698 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:21:27.715 INFO:journalctl@ceph.mgr.x.smithi191.stdout:Sep 10 19:21:27 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-x[34967]: ::ffff:172.21.15.191 - - [10/Sep/2024:19:21:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T19:21:27.733 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:21:27.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:21:27.734 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:21:27.745 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: 86 16 140996 140980 102.438 89.4375 0.010506 0.00975799 2024-09-10T19:21:27.835 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:21:27.873 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:21:27.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:21:27.874 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:21:27.931 INFO:journalctl@ceph.osd.1.smithi183.stdout:Sep 10 19:21:27 smithi183 podman[152517]: 2024-09-10 19:21:27.672081414 +0000 UTC m=+2.260786851 container init f9c0e821b88aeea0d11521203c514dc50c6a0d98bd07192b7f1a57bf71075b3b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3bfeaf4417585c821cab9ca76e4d95f390b0e769807122a45463640820b5cd16, name=ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd-1-deactivate, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=wip-guits-quincy-2024-09-09-1810-quincy-5be5739, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.vendor=CentOS) 2024-09-10T19:21:27.932 INFO:journalctl@ceph.osd.1.smithi183.stdout:Sep 10 19:21:27 smithi183 podman[152517]: 2024-09-10 19:21:27.678821892 +0000 UTC m=+2.267527335 container start f9c0e821b88aeea0d11521203c514dc50c6a0d98bd07192b7f1a57bf71075b3b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3bfeaf4417585c821cab9ca76e4d95f390b0e769807122a45463640820b5cd16, name=ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd-1-deactivate, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, RELEASE=wip-guits-quincy-2024-09-09-1810-quincy-5be5739, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True) 2024-09-10T19:21:27.975 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:21:28.010 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:21:28.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:21:28.011 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:21:28.112 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:21:28.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:21:28.158 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:21:28.158 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:21:28.259 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:21:28.296 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:21:28.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:21:28.297 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:21:28.398 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:21:28.431 INFO:journalctl@ceph.osd.1.smithi183.stdout:Sep 10 19:21:28 smithi183 podman[152517]: 2024-09-10 19:21:28.189109166 +0000 UTC m=+2.777814614 container attach f9c0e821b88aeea0d11521203c514dc50c6a0d98bd07192b7f1a57bf71075b3b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3bfeaf4417585c821cab9ca76e4d95f390b0e769807122a45463640820b5cd16, name=ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd-1-deactivate, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, GIT_CLEAN=True, RELEASE=wip-guits-quincy-2024-09-09-1810-quincy-5be5739, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, ceph=True, org.label-schema.build-date=20240731, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7) 2024-09-10T19:21:28.431 INFO:journalctl@ceph.osd.1.smithi183.stdout:Sep 10 19:21:28 smithi183 podman[152517]: 2024-09-10 19:21:28.189474724 +0000 UTC m=+2.778180171 container died f9c0e821b88aeea0d11521203c514dc50c6a0d98bd07192b7f1a57bf71075b3b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3bfeaf4417585c821cab9ca76e4d95f390b0e769807122a45463640820b5cd16, name=ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd-1-deactivate, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=wip-guits-quincy-2024-09-09-1810-quincy-5be5739) 2024-09-10T19:21:28.433 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:21:28.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:21:28.435 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:21:28.535 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:21:28.570 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:21:28.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:21:28.571 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:21:28.672 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:21:28.703 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:21:28.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:21:28.704 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:21:28.719 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-10T19:21:28.745 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: 87 16 142222 142206 102.141 76.625 0.0119247 0.00978654 2024-09-10T19:21:28.805 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:21:28.844 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:21:28.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:21:28.845 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:21:28.931 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:21:28 smithi183 ceph-mon[29736]: pgmap v510: 158 pgs: 158 active+clean; 8.4 GiB data, 29 GiB used, 508 GiB / 536 GiB avail; 1.2 KiB/s rd, 109 MiB/s wr, 1.74k op/s 2024-09-10T19:21:28.931 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:21:28 smithi183 ceph-mon[29736]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:21:28.932 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:21:28 smithi183 ceph-mon[41562]: pgmap v510: 158 pgs: 158 active+clean; 8.4 GiB data, 29 GiB used, 508 GiB / 536 GiB avail; 1.2 KiB/s rd, 109 MiB/s wr, 1.74k op/s 2024-09-10T19:21:28.932 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:21:28 smithi183 ceph-mon[41562]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:21:28.932 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Sep 10 19:21:28 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-y[29989]: 2024-09-10T19:21:28.669+0000 7f9fad6cd640 -1 mgr get_metadata_python Requested missing service osd.1 2024-09-10T19:21:28.945 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:21:28.976 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:21:28.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:21:28.977 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:21:29.078 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:21:29.113 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:21:29.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:21:29.114 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:21:29.215 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:21:29.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:21:28 smithi191 ceph-mon[32884]: pgmap v510: 158 pgs: 158 active+clean; 8.4 GiB data, 29 GiB used, 508 GiB / 536 GiB avail; 1.2 KiB/s rd, 109 MiB/s wr, 1.74k op/s 2024-09-10T19:21:29.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:21:28 smithi191 ceph-mon[32884]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:21:29.242 DEBUG:teuthology.orchestra.run.smithi183:> sudo pkill -f 'journalctl -f -n 0 -u ceph-320f6df8-6fa7-11ef-bcea-c7b262605968@osd.1.service' 2024-09-10T19:21:29.245 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:21:29.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:21:29.246 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:21:29.308 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-10T19:21:29.309 INFO:tasks.cephadm.osd.1:Stopped osd.1 2024-09-10T19:21:29.309 INFO:tasks.thrashosds.thrasher:Testing ceph-objectstore-tool on down osd.1 2024-09-10T19:21:29.309 INFO:tasks.thrashosds.thrasher:remote for osd 1 is ubuntu@smithi183.front.sepia.ceph.com 2024-09-10T19:21:29.309 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell -n osd.1 --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-1 --journal-path /var/lib/ceph/osd/ceph-1/journal --op list-pgs 2024-09-10T19:21:29.347 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:21:29.376 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:21:29.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:21:29.377 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:21:29.441 INFO:teuthology.orchestra.run.smithi183.stdout:ERROR: (22) Invalid argument 2024-09-10T19:21:29.441 INFO:teuthology.orchestra.run.smithi183.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-10T19:21:29.478 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:21:29.517 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:21:29.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:21:29.518 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:21:29.619 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:21:29.644 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/osd.1/config 2024-09-10T19:21:29.657 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:21:29.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:21:29.658 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:21:29.716 INFO:journalctl@ceph.iscsi.iscsi.a.smithi191.stdout:Sep 10 19:21:29 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-iscsi-iscsi-a[61816]: debug there is no tcmu-runner data available 2024-09-10T19:21:29.716 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:21:29 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=sqlstore.transactions t=2024-09-10T19:21:29.421973896Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=0 code="database is locked" 2024-09-10T19:21:29.716 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:21:29 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=cleanup t=2024-09-10T19:21:29.424643777Z level=info msg="Completed cleanup jobs" duration=13.202701ms 2024-09-10T19:21:29.745 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: 88 16 143649 143633 101.994 89.1875 0.00586594 0.00980023 2024-09-10T19:21:29.760 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:21:29.799 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:21:29.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:21:29.800 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:21:29.901 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:21:29.935 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:21:29.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:21:29.937 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:21:30.037 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:21:30.073 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:21:30.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:21:30.074 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:21:30.112 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph daemon osd.3 dump_ops_in_flight 2024-09-10T19:21:30.175 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:21:30.213 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:21:30.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:21:30.214 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:21:30.315 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:21:30.353 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:21:30.356 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:21:30.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:21:30.357 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:21:30.458 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:21:30.499 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:21:30.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:21:30.500 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:21:30.600 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:21:30.640 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:21:30.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:21:30.641 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:21:30.742 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:21:30.745 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: 89 16 145076 145060 101.85 89.1875 0.0124003 0.00981448 2024-09-10T19:21:30.781 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:21:30.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:21:30.782 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:21:30.883 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:21:30.913 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:21:30.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:21:30.914 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:21:30.931 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:21:30 smithi183 ceph-mon[29736]: from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:21:30.931 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:21:30 smithi183 ceph-mon[29736]: pgmap v511: 158 pgs: 158 active+clean; 8.4 GiB data, 29 GiB used, 508 GiB / 536 GiB avail; 639 B/s rd, 55 MiB/s wr, 876 op/s 2024-09-10T19:21:30.931 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:21:30 smithi183 ceph-mon[41562]: from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:21:30.932 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:21:30 smithi183 ceph-mon[41562]: pgmap v511: 158 pgs: 158 active+clean; 8.4 GiB data, 29 GiB used, 508 GiB / 536 GiB avail; 639 B/s rd, 55 MiB/s wr, 876 op/s 2024-09-10T19:21:31.014 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:21:31.043 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:21:31.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:21:31.044 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:21:31.144 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:21:31.174 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:21:31.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:21:31.175 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:21:31.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:21:30 smithi191 ceph-mon[32884]: from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:21:31.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:21:30 smithi191 ceph-mon[32884]: pgmap v511: 158 pgs: 158 active+clean; 8.4 GiB data, 29 GiB used, 508 GiB / 536 GiB avail; 639 B/s rd, 55 MiB/s wr, 876 op/s 2024-09-10T19:21:31.275 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:21:31.312 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:21:31.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:21:31.313 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:21:31.414 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:21:31.457 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:21:31.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:21:31.458 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:21:31.558 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:21:31.595 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:21:31.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:21:31.596 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:21:31.636 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Sep 10 19:21:31 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-y[29989]: ::ffff:172.21.15.191 - - [10/Sep/2024:19:21:31] "GET /metrics HTTP/1.1" 200 209451 "" "Prometheus/2.43.0" 2024-09-10T19:21:31.697 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:21:31.733 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:21:31.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:21:31.734 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:21:31.745 INFO:tasks.radosbench.radosbench.0.smithi183.stdout:Total time run: 90.0076 2024-09-10T19:21:31.745 INFO:tasks.radosbench.radosbench.0.smithi183.stdout:Total writes made: 146486 2024-09-10T19:21:31.745 INFO:tasks.radosbench.radosbench.0.smithi183.stdout:Write size: 65536 2024-09-10T19:21:31.745 INFO:tasks.radosbench.radosbench.0.smithi183.stdout:Object size: 65536 2024-09-10T19:21:31.746 INFO:tasks.radosbench.radosbench.0.smithi183.stdout:Bandwidth (MB/sec): 101.718 2024-09-10T19:21:31.746 INFO:tasks.radosbench.radosbench.0.smithi183.stdout:Stddev Bandwidth: 31.9746 2024-09-10T19:21:31.746 INFO:tasks.radosbench.radosbench.0.smithi183.stdout:Max bandwidth (MB/sec): 236.25 2024-09-10T19:21:31.746 INFO:tasks.radosbench.radosbench.0.smithi183.stdout:Min bandwidth (MB/sec): 75.1875 2024-09-10T19:21:31.746 INFO:tasks.radosbench.radosbench.0.smithi183.stdout:Average IOPS: 1627 2024-09-10T19:21:31.746 INFO:tasks.radosbench.radosbench.0.smithi183.stdout:Stddev IOPS: 511.593 2024-09-10T19:21:31.746 INFO:tasks.radosbench.radosbench.0.smithi183.stdout:Max IOPS: 3780 2024-09-10T19:21:31.746 INFO:tasks.radosbench.radosbench.0.smithi183.stdout:Min IOPS: 1203 2024-09-10T19:21:31.746 INFO:tasks.radosbench.radosbench.0.smithi183.stdout:Average Latency(s): 0.00982751 2024-09-10T19:21:31.746 INFO:tasks.radosbench.radosbench.0.smithi183.stdout:Stddev Latency(s): 0.00567657 2024-09-10T19:21:31.746 INFO:tasks.radosbench.radosbench.0.smithi183.stdout:Max latency(s): 0.109868 2024-09-10T19:21:31.746 INFO:tasks.radosbench.radosbench.0.smithi183.stdout:Min latency(s): 0.00139047 2024-09-10T19:21:31.749 INFO:tasks.radosbench.radosbench.0.smithi183.stdout:Cleaning up (deleting benchmark objects) 2024-09-10T19:21:31.835 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:21:31.866 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:21:31.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:21:31.867 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:21:31.968 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:21:31.997 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:21:31.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:21:31.998 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:21:32.099 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:21:32.106 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:21:32.138 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:21:32.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:21:32.140 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:21:32.240 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:21:32.278 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:21:32.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:21:32.279 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:21:32.380 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:21:32.411 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:21:32.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:21:32.412 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:21:32.513 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:21:32.541 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:21:32.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:21:32.542 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:21:32.643 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:21:32.675 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:21:32.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:21:32.676 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:21:32.777 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:21:32.810 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:21:32.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:21:32.811 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:21:32.912 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:21:32.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:21:32.944 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:21:32.944 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:21:33.045 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:21:33.072 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:21:33.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:21:33.074 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:21:33.160 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:21:32 smithi183 ceph-mon[29736]: pgmap v512: 158 pgs: 158 active+clean; 8.8 GiB data, 30 GiB used, 506 GiB / 536 GiB avail; 1.2 KiB/s rd, 109 MiB/s wr, 1.75k op/s 2024-09-10T19:21:33.160 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:21:32 smithi183 ceph-mon[41562]: pgmap v512: 158 pgs: 158 active+clean; 8.8 GiB data, 30 GiB used, 506 GiB / 536 GiB avail; 1.2 KiB/s rd, 109 MiB/s wr, 1.75k op/s 2024-09-10T19:21:33.174 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:21:33.203 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:21:33.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:21:33.204 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:21:33.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:21:32 smithi191 ceph-mon[32884]: pgmap v512: 158 pgs: 158 active+clean; 8.8 GiB data, 30 GiB used, 506 GiB / 536 GiB avail; 1.2 KiB/s rd, 109 MiB/s wr, 1.75k op/s 2024-09-10T19:21:33.305 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:21:33.337 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:21:33.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:21:33.338 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:21:33.371 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:21:33.439 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:21:33.475 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:21:33.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:21:33.476 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:21:33.577 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:21:33.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:21:33.618 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:21:33.618 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:21:33.719 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:21:33.755 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:21:33.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:21:33.756 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:21:33.856 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:21:33.886 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:21:33.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:21:33.887 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:21:33.987 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:21:34.019 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:21:34.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:21:34.020 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:21:34.121 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:21:34.149 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:21:34.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:21:34.150 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:21:34.251 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:21:34.280 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:21:34.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:21:34.281 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:21:34.382 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:21:34.410 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:21:34.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:21:34.411 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:21:34.512 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:21:34.539 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:21:34.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:21:34.540 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:21:34.641 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:21:34.673 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:21:34.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:21:34.674 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:21:34.774 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:21:34.810 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:21:34.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:21:34.811 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:21:34.912 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:21:34.955 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:21:34.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:21:34.956 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:21:35.057 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:21:35.101 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:21:35.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:21:35.102 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:21:35.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:21:34 smithi183 ceph-mon[29736]: pgmap v513: 158 pgs: 158 active+clean; 8.8 GiB data, 30 GiB used, 506 GiB / 536 GiB avail; 1.0 KiB/s rd, 88 MiB/s wr, 1.41k op/s 2024-09-10T19:21:35.181 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:21:34 smithi183 ceph-mon[41562]: pgmap v513: 158 pgs: 158 active+clean; 8.8 GiB data, 30 GiB used, 506 GiB / 536 GiB avail; 1.0 KiB/s rd, 88 MiB/s wr, 1.41k op/s 2024-09-10T19:21:35.203 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:21:35.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:21:34 smithi191 ceph-mon[32884]: pgmap v513: 158 pgs: 158 active+clean; 8.8 GiB data, 30 GiB used, 506 GiB / 536 GiB avail; 1.0 KiB/s rd, 88 MiB/s wr, 1.41k op/s 2024-09-10T19:21:35.239 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:21:35.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:21:35.240 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:21:35.341 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:21:35.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:21:35.452 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:21:35.452 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:21:35.553 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:21:35.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:21:35.581 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:21:35.582 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:21:35.683 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:21:35.722 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:21:35.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:21:35.723 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:21:35.824 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:21:35.851 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:21:35.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:21:35.853 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:21:35.953 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:21:35.992 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:21:35.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:21:35.993 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:21:36.094 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:21:36.132 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:21:36.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:21:36.133 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:21:36.234 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:21:36.274 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:21:36.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:21:36.275 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:21:36.375 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:21:36.402 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:21:36.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:21:36.403 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:21:36.504 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:21:36.540 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:21:36.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:21:36.541 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:21:36.642 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:21:36.676 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:21:36.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:21:36.677 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:21:36.779 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:21:36.816 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:21:36.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:21:36.817 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:21:36.917 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:21:36.951 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:21:36.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:21:36.952 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:21:37.011 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:21:36 smithi183 ceph-mon[29736]: pgmap v514: 158 pgs: 158 active+clean; 8.6 GiB data, 30 GiB used, 506 GiB / 536 GiB avail; 1.5 KiB/s rd, 93 MiB/s wr, 1.95k op/s 2024-09-10T19:21:37.011 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:21:36 smithi183 ceph-mon[41562]: pgmap v514: 158 pgs: 158 active+clean; 8.6 GiB data, 30 GiB used, 506 GiB / 536 GiB avail; 1.5 KiB/s rd, 93 MiB/s wr, 1.95k op/s 2024-09-10T19:21:37.053 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:21:37.090 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:21:37.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:21:37.091 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:21:37.192 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:21:37.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:21:36 smithi191 ceph-mon[32884]: pgmap v514: 158 pgs: 158 active+clean; 8.6 GiB data, 30 GiB used, 506 GiB / 536 GiB avail; 1.5 KiB/s rd, 93 MiB/s wr, 1.95k op/s 2024-09-10T19:21:37.217 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:21:37.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:21:37.218 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:21:37.319 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:21:37.347 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:21:37.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:21:37.348 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:21:37.449 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:21:37.483 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:21:37.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:21:37.484 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:21:37.584 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:21:37.611 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:21:37.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:21:37.612 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:21:37.686 INFO:tasks.thrashosds.thrasher:No PGs found for osd.1 2024-09-10T19:21:37.713 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:21:37.716 INFO:journalctl@ceph.mgr.x.smithi191.stdout:Sep 10 19:21:37 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-x[34967]: ::ffff:172.21.15.191 - - [10/Sep/2024:19:21:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T19:21:37.744 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:21:37.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:21:37.745 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:21:37.846 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:21:37.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:21:37.883 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:21:37.883 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:21:37.984 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:21:38.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:21:38.024 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:21:38.024 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:21:38.125 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:21:38.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:21:38.153 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:21:38.153 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:21:38.254 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:21:38.286 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:21:38.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:21:38.288 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:21:38.389 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:21:38.416 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:21:38.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:21:38.417 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:21:38.518 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:21:38.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:21:38.557 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:21:38.557 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:21:38.658 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:21:38.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:21:38.687 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:21:38.687 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:21:38.788 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:21:38.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:21:38.827 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:21:38.827 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:21:38.927 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:21:38.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:21:38.950 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:21:38.950 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:21:39.035 INFO:teuthology.orchestra.run.smithi183.stdout:osd.0: {} 2024-09-10T19:21:39.035 INFO:teuthology.orchestra.run.smithi183.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-10T19:21:39.035 INFO:teuthology.orchestra.run.smithi183.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-09-10T19:21:39.051 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:21:39.061 INFO:teuthology.orchestra.run.smithi183.stdout:osd.2: {} 2024-09-10T19:21:39.061 INFO:teuthology.orchestra.run.smithi183.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-10T19:21:39.061 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:21:38 smithi183 ceph-mon[29736]: pgmap v515: 158 pgs: 158 active+clean; 8.6 GiB data, 30 GiB used, 506 GiB / 536 GiB avail; 938 B/s rd, 49 MiB/s wr, 1.20k op/s 2024-09-10T19:21:39.062 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:21:38 smithi183 ceph-mon[29736]: purged_snaps scrub starts 2024-09-10T19:21:39.062 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:21:38 smithi183 ceph-mon[29736]: purged_snaps scrub ok 2024-09-10T19:21:39.062 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:21:38 smithi183 ceph-mon[41562]: pgmap v515: 158 pgs: 158 active+clean; 8.6 GiB data, 30 GiB used, 506 GiB / 536 GiB avail; 938 B/s rd, 49 MiB/s wr, 1.20k op/s 2024-09-10T19:21:39.062 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:21:38 smithi183 ceph-mon[41562]: purged_snaps scrub starts 2024-09-10T19:21:39.062 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:21:38 smithi183 ceph-mon[41562]: purged_snaps scrub ok 2024-09-10T19:21:39.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:21:39.084 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:21:39.085 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:21:39.089 INFO:teuthology.orchestra.run.smithi183.stdout:osd.3: {} 2024-09-10T19:21:39.090 INFO:teuthology.orchestra.run.smithi183.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-10T19:21:39.119 INFO:teuthology.orchestra.run.smithi183.stdout:osd.4: {} 2024-09-10T19:21:39.119 INFO:teuthology.orchestra.run.smithi183.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-10T19:21:39.148 INFO:teuthology.orchestra.run.smithi183.stdout:osd.5: {} 2024-09-10T19:21:39.148 INFO:teuthology.orchestra.run.smithi183.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-10T19:21:39.172 INFO:teuthology.orchestra.run.smithi183.stdout:osd.6: {} 2024-09-10T19:21:39.172 INFO:teuthology.orchestra.run.smithi183.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-10T19:21:39.186 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:21:39.200 INFO:teuthology.orchestra.run.smithi183.stdout:osd.7: {} 2024-09-10T19:21:39.201 INFO:teuthology.orchestra.run.smithi183.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-10T19:21:39.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:21:38 smithi191 ceph-mon[32884]: pgmap v515: 158 pgs: 158 active+clean; 8.6 GiB data, 30 GiB used, 506 GiB / 536 GiB avail; 938 B/s rd, 49 MiB/s wr, 1.20k op/s 2024-09-10T19:21:39.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:21:38 smithi191 ceph-mon[32884]: purged_snaps scrub starts 2024-09-10T19:21:39.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:21:38 smithi191 ceph-mon[32884]: purged_snaps scrub ok 2024-09-10T19:21:39.216 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:21:39.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:21:39.217 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:21:39.318 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:21:39.352 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:21:39.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:21:39.353 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:21:39.454 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:21:39.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:21:39.491 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:21:39.491 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:21:39.592 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:21:39.621 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:21:39.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:21:39.622 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:21:39.722 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:21:39.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:21:39.753 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:21:39.754 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:21:39.854 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:21:39.885 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:21:39.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:21:39.886 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:21:39.919 INFO:journalctl@ceph.iscsi.iscsi.a.smithi191.stdout:Sep 10 19:21:39 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-iscsi-iscsi-a[61816]: debug there is no tcmu-runner data available 2024-09-10T19:21:39.986 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:21:40.025 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:21:40.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:21:40.026 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:21:40.127 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:21:40.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:21:40.161 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:21:40.162 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:21:40.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:21:39 smithi183 ceph-mon[29736]: osdmap e259: 8 total, 7 up, 6 in 2024-09-10T19:21:40.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:21:39 smithi183 ceph-mon[29736]: from='client.? 172.21.15.183:0/2956906271' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T19:21:40.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:21:39 smithi183 ceph-mon[29736]: osdmap e260: 8 total, 7 up, 6 in 2024-09-10T19:21:40.181 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:21:39 smithi183 ceph-mon[41562]: osdmap e259: 8 total, 7 up, 6 in 2024-09-10T19:21:40.181 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:21:39 smithi183 ceph-mon[41562]: from='client.? 172.21.15.183:0/2956906271' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T19:21:40.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:21:39 smithi183 ceph-mon[41562]: osdmap e260: 8 total, 7 up, 6 in 2024-09-10T19:21:40.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:21:39 smithi191 ceph-mon[32884]: osdmap e259: 8 total, 7 up, 6 in 2024-09-10T19:21:40.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:21:39 smithi191 ceph-mon[32884]: from='client.? 172.21.15.183:0/2956906271' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T19:21:40.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:21:39 smithi191 ceph-mon[32884]: osdmap e260: 8 total, 7 up, 6 in 2024-09-10T19:21:40.262 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:21:40.291 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:21:40.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:21:40.292 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:21:40.393 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:21:40.431 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:21:40.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:21:40.434 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:21:40.535 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:21:40.541 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-10T19:21:40.541 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-10T19:21:40.578 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:21:40.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:21:40.579 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:21:40.680 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:21:40.690 INFO:teuthology.orchestra.run.smithi183.stdout:{ 2024-09-10T19:21:40.691 INFO:teuthology.orchestra.run.smithi183.stdout: "ops": [ 2024-09-10T19:21:40.691 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:21:40.692 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.15360.0:158831 9.8 9:1847355c:::benchmark_data_smithi183_142499_object12342:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e259)", 2024-09-10T19:21:40.692 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-10T19:21:39.782827+0000", 2024-09-10T19:21:40.692 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 0.0033601149999999999, 2024-09-10T19:21:40.692 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0033925510000000002, 2024-09-10T19:21:40.692 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-10T19:21:40.692 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "waiting for sub ops", 2024-09-10T19:21:40.692 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-10T19:21:40.692 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.15360", 2024-09-10T19:21:40.692 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.183:0/2364804131", 2024-09-10T19:21:40.692 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 158831 2024-09-10T19:21:40.693 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:21:40.693 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-10T19:21:40.693 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:21:40.693 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-10T19:21:40.693 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:21:39.782827+0000", 2024-09-10T19:21:40.693 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:21:40.693 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:21:40.693 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:21:40.693 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-10T19:21:40.693 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:21:39.782827+0000", 2024-09-10T19:21:40.693 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:21:40.693 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:21:40.693 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:21:40.694 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-10T19:21:40.694 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:21:39.782826+0000", 2024-09-10T19:21:40.694 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4294967295.9999986 2024-09-10T19:21:40.694 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:21:40.694 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:21:40.694 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-10T19:21:40.694 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:21:39.782831+0000", 2024-09-10T19:21:40.694 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.6709999999999998e-06 2024-09-10T19:21:40.694 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:21:40.694 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:21:40.694 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-10T19:21:40.694 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:21:39.782839+0000", 2024-09-10T19:21:40.694 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.7710000000000006e-06 2024-09-10T19:21:40.694 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:21:40.695 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:21:40.695 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-10T19:21:40.695 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:21:39.782843+0000", 2024-09-10T19:21:40.695 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.279e-06 2024-09-10T19:21:40.695 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:21:40.695 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:21:40.695 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-10T19:21:40.695 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:21:39.782876+0000", 2024-09-10T19:21:40.695 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.3485000000000002e-05 2024-09-10T19:21:40.695 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:21:40.695 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:21:40.695 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-10T19:21:40.695 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:21:39.783057+0000", 2024-09-10T19:21:40.695 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000180538 2024-09-10T19:21:40.696 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:21:40.696 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:21:40.696 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 0,5", 2024-09-10T19:21:40.696 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:21:39.783176+0000", 2024-09-10T19:21:40.696 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000119299 2024-09-10T19:21:40.696 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:21:40.696 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:21:40.696 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:21:40.696 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:21:40.696 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:21:40.696 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_repop(client.15360.0:158833 9.7 e259/229 9:e70f2b8d:::benchmark_data_smithi183_142499_object12344:head v 259'9811, mlcod=259'9809)", 2024-09-10T19:21:40.696 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-10T19:21:39.785566+0000", 2024-09-10T19:21:40.696 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 0.00062176500000000003, 2024-09-10T19:21:40.697 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00073994299999999996, 2024-09-10T19:21:40.697 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-10T19:21:40.697 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "started", 2024-09-10T19:21:40.697 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-10T19:21:40.697 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:21:40.697 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-10T19:21:40.697 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:21:39.785566+0000", 2024-09-10T19:21:40.697 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:21:40.697 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:21:40.697 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:21:40.697 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-10T19:21:40.697 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:21:39.785566+0000", 2024-09-10T19:21:40.697 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:21:40.697 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:21:40.698 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:21:40.698 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-10T19:21:40.698 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:21:39.785565+0000", 2024-09-10T19:21:40.698 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4294967295.999999 2024-09-10T19:21:40.698 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:21:40.698 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:21:40.698 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-10T19:21:40.698 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:21:39.785579+0000", 2024-09-10T19:21:40.698 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.4095e-05 2024-09-10T19:21:40.698 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:21:40.698 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:21:40.698 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-10T19:21:40.698 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:21:39.785592+0000", 2024-09-10T19:21:40.698 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.325e-05 2024-09-10T19:21:40.699 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:21:40.699 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:21:40.699 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-10T19:21:40.699 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:21:39.785599+0000", 2024-09-10T19:21:40.699 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.5409999999999997e-06 2024-09-10T19:21:40.699 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:21:40.699 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:21:40.699 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-10T19:21:40.699 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:21:39.785626+0000", 2024-09-10T19:21:40.699 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.7222000000000001e-05 2024-09-10T19:21:40.699 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:21:40.699 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:21:40.699 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-10T19:21:40.699 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:21:39.785644+0000", 2024-09-10T19:21:40.700 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.7978e-05 2024-09-10T19:21:40.700 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:21:40.700 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:21:40.700 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:21:40.700 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:21:40.700 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:21:40.700 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_repop(client.15360.0:158832 9.a e259/229 9:5b86a8ff:::benchmark_data_smithi183_142499_object12343:head v 259'9753, mlcod=259'9750)", 2024-09-10T19:21:40.700 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-10T19:21:39.785611+0000", 2024-09-10T19:21:40.700 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 0.00057681200000000001, 2024-09-10T19:21:40.700 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00075778599999999998, 2024-09-10T19:21:40.700 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-10T19:21:40.700 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "started", 2024-09-10T19:21:40.700 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-10T19:21:40.700 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:21:40.700 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-10T19:21:40.701 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:21:39.785611+0000", 2024-09-10T19:21:40.701 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:21:40.701 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:21:40.701 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:21:40.701 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-10T19:21:40.701 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:21:39.785611+0000", 2024-09-10T19:21:40.701 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:21:40.701 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:21:40.701 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:21:40.701 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-10T19:21:40.701 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:21:39.785610+0000", 2024-09-10T19:21:40.701 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4294967295.9999995 2024-09-10T19:21:40.701 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:21:40.701 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:21:40.702 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-10T19:21:40.702 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:21:39.785614+0000", 2024-09-10T19:21:40.702 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.1200000000000004e-06 2024-09-10T19:21:40.702 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:21:40.702 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:21:40.702 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-10T19:21:40.702 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:21:39.785621+0000", 2024-09-10T19:21:40.702 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.19e-06 2024-09-10T19:21:40.702 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:21:40.702 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:21:40.702 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-10T19:21:40.702 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:21:39.785624+0000", 2024-09-10T19:21:40.702 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.5750000000000001e-06 2024-09-10T19:21:40.702 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:21:40.702 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:21:40.703 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-10T19:21:40.703 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:21:39.785650+0000", 2024-09-10T19:21:40.703 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.5565e-05 2024-09-10T19:21:40.703 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:21:40.703 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:21:40.703 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-10T19:21:40.703 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:21:39.785665+0000", 2024-09-10T19:21:40.703 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.4959e-05 2024-09-10T19:21:40.703 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:21:40.703 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:21:40.703 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:21:40.703 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:21:40.703 INFO:teuthology.orchestra.run.smithi183.stdout: ], 2024-09-10T19:21:40.703 INFO:teuthology.orchestra.run.smithi183.stdout: "num_ops": 3 2024-09-10T19:21:40.704 INFO:teuthology.orchestra.run.smithi183.stdout:} 2024-09-10T19:21:40.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:21:40.709 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:21:40.709 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:21:40.810 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:21:40.850 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:21:40.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:21:40.851 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:21:40.919 INFO:teuthology.orchestra.run.smithi183.stderr:noscrub is unset 2024-09-10T19:21:40.953 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:21:40.987 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:21:40.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:21:40.988 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:21:41.089 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:21:41.118 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:21:41.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:21:41.119 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:21:41.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:21:40 smithi183 ceph-mon[29736]: from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:21:41.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:21:40 smithi183 ceph-mon[29736]: pgmap v517: 158 pgs: 158 active+clean; 8.6 GiB data, 30 GiB used, 506 GiB / 536 GiB avail; 1.1 KiB/s rd, 58 MiB/s wr, 1.44k op/s 2024-09-10T19:21:41.182 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:21:40 smithi183 ceph-mon[29736]: from='client.? 172.21.15.183:0/3437565750' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-10T19:21:41.182 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:21:40 smithi183 ceph-mon[29736]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-10T19:21:41.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:21:40 smithi183 ceph-mon[41562]: from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:21:41.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:21:40 smithi183 ceph-mon[41562]: pgmap v517: 158 pgs: 158 active+clean; 8.6 GiB data, 30 GiB used, 506 GiB / 536 GiB avail; 1.1 KiB/s rd, 58 MiB/s wr, 1.44k op/s 2024-09-10T19:21:41.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:21:40 smithi183 ceph-mon[41562]: from='client.? 172.21.15.183:0/3437565750' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-10T19:21:41.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:21:40 smithi183 ceph-mon[41562]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-10T19:21:41.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:21:40 smithi191 ceph-mon[32884]: from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:21:41.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:21:40 smithi191 ceph-mon[32884]: pgmap v517: 158 pgs: 158 active+clean; 8.6 GiB data, 30 GiB used, 506 GiB / 536 GiB avail; 1.1 KiB/s rd, 58 MiB/s wr, 1.44k op/s 2024-09-10T19:21:41.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:21:40 smithi191 ceph-mon[32884]: from='client.? 172.21.15.183:0/3437565750' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-10T19:21:41.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:21:40 smithi191 ceph-mon[32884]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-10T19:21:41.220 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:21:41.248 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:21:41.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:21:41.249 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:21:41.350 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:21:41.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:21:41.389 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:21:41.389 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:21:41.393 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph daemon osd.3 dump_blocked_ops 2024-09-10T19:21:41.490 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:21:41.522 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:21:41.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:21:41.523 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:21:41.624 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:21:41.662 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:21:41.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:21:41.663 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:21:41.681 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Sep 10 19:21:41 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-y[29989]: ::ffff:172.21.15.191 - - [10/Sep/2024:19:21:41] "GET /metrics HTTP/1.1" 200 209451 "" "Prometheus/2.43.0" 2024-09-10T19:21:41.763 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:21:41.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:21:41.802 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:21:41.802 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:21:41.903 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:21:41.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:21:41.934 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:21:41.934 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:21:42.035 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:21:42.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:21:42.078 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:21:42.078 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:21:42.079 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:21:41 smithi183 ceph-mon[29736]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T19:21:42.079 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:21:41 smithi183 ceph-mon[29736]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-10T19:21:42.079 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:21:41 smithi183 ceph-mon[29736]: osdmap e261: 8 total, 7 up, 6 in 2024-09-10T19:21:42.079 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:21:41 smithi183 ceph-mon[41562]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T19:21:42.079 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:21:41 smithi183 ceph-mon[41562]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-10T19:21:42.079 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:21:41 smithi183 ceph-mon[41562]: osdmap e261: 8 total, 7 up, 6 in 2024-09-10T19:21:42.179 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:21:42.210 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:21:42.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:21:42.211 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:21:42.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:21:41 smithi191 ceph-mon[32884]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T19:21:42.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:21:41 smithi191 ceph-mon[32884]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-10T19:21:42.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:21:41 smithi191 ceph-mon[32884]: osdmap e261: 8 total, 7 up, 6 in 2024-09-10T19:21:42.221 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:21:42.313 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:21:42.354 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:21:42.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:21:42.355 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:21:42.456 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:21:42.482 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:21:42.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:21:42.483 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:21:42.543 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-10T19:21:42.584 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:21:42.621 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:21:42.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:21:42.622 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:21:42.687 INFO:tasks.thrashosds.thrasher:in_osds: [0, 3, 4, 5, 6, 7] out_osds: [1, 2] dead_osds: [1] live_osds: [0, 2, 3, 4, 5, 6, 7] 2024-09-10T19:21:42.687 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-10T19:21:42.688 INFO:tasks.thrashosds.thrasher:Killing osd 0, live_osds are [0, 2, 3, 4, 5, 6, 7] 2024-09-10T19:21:42.688 INFO:tasks.cephadm.osd.0:Stopping osd.0... 2024-09-10T19:21:42.688 DEBUG:teuthology.orchestra.run.smithi183:> sudo systemctl stop ceph-320f6df8-6fa7-11ef-bcea-c7b262605968@osd.0 2024-09-10T19:21:42.722 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:21:42.764 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:21:42.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:21:42.765 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:21:42.866 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:21:42.899 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:21:42.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:21:42.900 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:21:43.001 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:21:43.046 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:21:43.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:21:43.047 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:21:43.148 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:21:43.176 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:21:42 smithi183 ceph-mon[29736]: pgmap v520: 158 pgs: 158 active+clean; 8.2 GiB data, 29 GiB used, 508 GiB / 536 GiB avail; 852 B/s rd, 0 B/s wr, 1.29k op/s 2024-09-10T19:21:43.176 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:21:42 smithi183 ceph-mon[41562]: pgmap v520: 158 pgs: 158 active+clean; 8.2 GiB data, 29 GiB used, 508 GiB / 536 GiB avail; 852 B/s rd, 0 B/s wr, 1.29k op/s 2024-09-10T19:21:43.176 INFO:journalctl@ceph.osd.0.smithi183.stdout:Sep 10 19:21:42 smithi183 systemd[1]: Stopping Ceph osd.0 for 320f6df8-6fa7-11ef-bcea-c7b262605968... 2024-09-10T19:21:43.177 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:21:43.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:21:43.178 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:21:43.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:21:42 smithi191 ceph-mon[32884]: pgmap v520: 158 pgs: 158 active+clean; 8.2 GiB data, 29 GiB used, 508 GiB / 536 GiB avail; 852 B/s rd, 0 B/s wr, 1.29k op/s 2024-09-10T19:21:43.279 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:21:43.306 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:21:43.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:21:43.307 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:21:43.408 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:21:43.437 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:21:43.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:21:43.438 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:21:43.539 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:21:43.575 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:21:43.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:21:43.576 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:21:43.677 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:21:43.714 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:21:43.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:21:43.716 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:21:43.816 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:21:43.848 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:21:43.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:21:43.850 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:21:43.931 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Sep 10 19:21:43 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-y[29989]: 2024-09-10T19:21:43.668+0000 7f9fad6cd640 -1 mgr get_metadata_python Requested missing service osd.1 2024-09-10T19:21:43.951 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:21:43.986 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:21:43.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:21:43.987 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:21:44.050 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:21:44.088 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:21:44.112 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-10T19:21:44.118 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:21:44.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:21:44.119 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:21:44.211 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:21:43 smithi183 ceph-mon[41562]: 4.1a scrub starts 2024-09-10T19:21:44.212 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:21:43 smithi183 ceph-mon[41562]: 4.1a scrub ok 2024-09-10T19:21:44.212 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:21:43 smithi183 ceph-mon[41562]: 6.6 scrub starts 2024-09-10T19:21:44.212 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:21:43 smithi183 ceph-mon[41562]: 6.6 scrub ok 2024-09-10T19:21:44.212 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:21:43 smithi183 ceph-mon[41562]: purged_snaps scrub starts 2024-09-10T19:21:44.212 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:21:43 smithi183 ceph-mon[41562]: purged_snaps scrub ok 2024-09-10T19:21:44.212 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:21:43 smithi183 ceph-mon[41562]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:21:44.212 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:21:43 smithi183 ceph-mon[29736]: 4.1a scrub starts 2024-09-10T19:21:44.212 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:21:43 smithi183 ceph-mon[29736]: 4.1a scrub ok 2024-09-10T19:21:44.212 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:21:43 smithi183 ceph-mon[29736]: 6.6 scrub starts 2024-09-10T19:21:44.213 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:21:43 smithi183 ceph-mon[29736]: 6.6 scrub ok 2024-09-10T19:21:44.213 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:21:43 smithi183 ceph-mon[29736]: purged_snaps scrub starts 2024-09-10T19:21:44.213 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:21:43 smithi183 ceph-mon[29736]: purged_snaps scrub ok 2024-09-10T19:21:44.213 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:21:43 smithi183 ceph-mon[29736]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:21:44.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:21:43 smithi191 ceph-mon[32884]: 4.1a scrub starts 2024-09-10T19:21:44.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:21:43 smithi191 ceph-mon[32884]: 4.1a scrub ok 2024-09-10T19:21:44.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:21:43 smithi191 ceph-mon[32884]: 6.6 scrub starts 2024-09-10T19:21:44.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:21:43 smithi191 ceph-mon[32884]: 6.6 scrub ok 2024-09-10T19:21:44.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:21:43 smithi191 ceph-mon[32884]: purged_snaps scrub starts 2024-09-10T19:21:44.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:21:43 smithi191 ceph-mon[32884]: purged_snaps scrub ok 2024-09-10T19:21:44.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:21:43 smithi191 ceph-mon[32884]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:21:44.220 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:21:44.255 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:21:44.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:21:44.256 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:21:44.357 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:21:44.391 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:21:44.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:21:44.392 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:21:44.493 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:21:44.530 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:21:44.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:21:44.531 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:21:44.632 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:21:44.675 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:21:44.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:21:44.676 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:21:44.776 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:21:44.810 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:21:44.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:21:44.811 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:21:44.912 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:21:44.951 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:21:44.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:21:44.952 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:21:45.053 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:21:45.086 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:21:45.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:21:45.087 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:21:45.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:21:44 smithi183 ceph-mon[29736]: 5.17 scrub starts 2024-09-10T19:21:45.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:21:44 smithi183 ceph-mon[29736]: 5.17 scrub ok 2024-09-10T19:21:45.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:21:44 smithi183 ceph-mon[29736]: pgmap v521: 158 pgs: 158 active+clean; 8.2 GiB data, 29 GiB used, 508 GiB / 536 GiB avail; 852 B/s rd, 0 B/s wr, 1.29k op/s 2024-09-10T19:21:45.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:21:44 smithi183 ceph-mon[29736]: osdmap e262: 8 total, 7 up, 6 in 2024-09-10T19:21:45.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:21:44 smithi183 ceph-mon[41562]: 5.17 scrub starts 2024-09-10T19:21:45.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:21:44 smithi183 ceph-mon[41562]: 5.17 scrub ok 2024-09-10T19:21:45.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:21:44 smithi183 ceph-mon[41562]: pgmap v521: 158 pgs: 158 active+clean; 8.2 GiB data, 29 GiB used, 508 GiB / 536 GiB avail; 852 B/s rd, 0 B/s wr, 1.29k op/s 2024-09-10T19:21:45.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:21:44 smithi183 ceph-mon[41562]: osdmap e262: 8 total, 7 up, 6 in 2024-09-10T19:21:45.187 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:21:45.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:21:44 smithi191 ceph-mon[32884]: 5.17 scrub starts 2024-09-10T19:21:45.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:21:44 smithi191 ceph-mon[32884]: 5.17 scrub ok 2024-09-10T19:21:45.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:21:44 smithi191 ceph-mon[32884]: pgmap v521: 158 pgs: 158 active+clean; 8.2 GiB data, 29 GiB used, 508 GiB / 536 GiB avail; 852 B/s rd, 0 B/s wr, 1.29k op/s 2024-09-10T19:21:45.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:21:44 smithi191 ceph-mon[32884]: osdmap e262: 8 total, 7 up, 6 in 2024-09-10T19:21:45.221 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:21:45.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:21:45.222 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:21:45.322 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:21:45.328 INFO:teuthology.orchestra.run.smithi183.stdout:{ 2024-09-10T19:21:45.329 INFO:teuthology.orchestra.run.smithi183.stdout: "ops": [], 2024-09-10T19:21:45.329 INFO:teuthology.orchestra.run.smithi183.stdout: "complaint_time": 30, 2024-09-10T19:21:45.329 INFO:teuthology.orchestra.run.smithi183.stdout: "num_blocked_ops": 0 2024-09-10T19:21:45.329 INFO:teuthology.orchestra.run.smithi183.stdout:} 2024-09-10T19:21:45.349 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:21:45.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:21:45.350 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:21:45.451 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:21:45.481 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:21:45.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:21:45.482 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:21:45.583 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:21:45.622 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:21:45.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:21:45.623 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:21:45.634 INFO:journalctl@ceph.osd.0.smithi183.stdout:Sep 10 19:21:45 smithi183 podman[154922]: 2024-09-10 19:21:45.328749076 +0000 UTC m=+2.565855919 container died 21ec38ff8daeca88f25c174d1de2dda44ea875a66e9f66e8921fce6c16c3b114 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588, name=gallant_euclid, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , RELEASE=wip-guits-quincy-2024-09-09-1810-quincy-5be5739, org.label-schema.schema-version=1.0, ceph=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS) 2024-09-10T19:21:45.724 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:21:45.751 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:21:45.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:21:45.752 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:21:45.852 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:21:45.895 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:21:45.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:21:45.896 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:21:45.997 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:21:46.033 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:21:46.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:21:46.034 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:21:46.071 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph daemon osd.3 dump_historic_ops 2024-09-10T19:21:46.135 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:21:46.170 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:21:46.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:21:46.171 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:21:46.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:21:45 smithi191 ceph-mon[32884]: 4.c scrub starts 2024-09-10T19:21:46.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:21:45 smithi191 ceph-mon[32884]: 4.c scrub ok 2024-09-10T19:21:46.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:21:45 smithi191 ceph-mon[32884]: 2.19 scrub starts 2024-09-10T19:21:46.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:21:45 smithi191 ceph-mon[32884]: 2.19 scrub ok 2024-09-10T19:21:46.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:21:45 smithi191 ceph-mon[32884]: 9.5 scrub starts 2024-09-10T19:21:46.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:21:45 smithi191 ceph-mon[32884]: 1.6 scrub starts 2024-09-10T19:21:46.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:21:45 smithi191 ceph-mon[32884]: 1.6 scrub ok 2024-09-10T19:21:46.272 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:21:46.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:21:46.312 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:21:46.312 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:21:46.413 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:21:46.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:21:45 smithi183 ceph-mon[29736]: 4.c scrub starts 2024-09-10T19:21:46.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:21:45 smithi183 ceph-mon[29736]: 4.c scrub ok 2024-09-10T19:21:46.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:21:45 smithi183 ceph-mon[29736]: 2.19 scrub starts 2024-09-10T19:21:46.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:21:45 smithi183 ceph-mon[29736]: 2.19 scrub ok 2024-09-10T19:21:46.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:21:45 smithi183 ceph-mon[29736]: 9.5 scrub starts 2024-09-10T19:21:46.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:21:45 smithi183 ceph-mon[29736]: 1.6 scrub starts 2024-09-10T19:21:46.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:21:45 smithi183 ceph-mon[29736]: 1.6 scrub ok 2024-09-10T19:21:46.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:21:45 smithi183 ceph-mon[41562]: 4.c scrub starts 2024-09-10T19:21:46.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:21:45 smithi183 ceph-mon[41562]: 4.c scrub ok 2024-09-10T19:21:46.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:21:45 smithi183 ceph-mon[41562]: 2.19 scrub starts 2024-09-10T19:21:46.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:21:45 smithi183 ceph-mon[41562]: 2.19 scrub ok 2024-09-10T19:21:46.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:21:45 smithi183 ceph-mon[41562]: 9.5 scrub starts 2024-09-10T19:21:46.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:21:45 smithi183 ceph-mon[41562]: 1.6 scrub starts 2024-09-10T19:21:46.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:21:45 smithi183 ceph-mon[41562]: 1.6 scrub ok 2024-09-10T19:21:46.447 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:21:46.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:21:46.448 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:21:46.549 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:21:46.574 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:21:46.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:21:46.575 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:21:46.676 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:21:46.692 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:21:46.708 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:21:46.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:21:46.709 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:21:46.810 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:21:46.843 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:21:46.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:21:46.844 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:21:46.910 INFO:journalctl@ceph.osd.0.smithi183.stdout:Sep 10 19:21:46 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd-0[45861]: 2024-09-10T19:21:46.795+0000 7f96e5c6e640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-10T19:21:46.910 INFO:journalctl@ceph.osd.0.smithi183.stdout:Sep 10 19:21:46 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd-0[45861]: 2024-09-10T19:21:46.795+0000 7f96e5c6e640 -1 osd.0 262 *** Got signal Terminated *** 2024-09-10T19:21:46.910 INFO:journalctl@ceph.osd.0.smithi183.stdout:Sep 10 19:21:46 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd-0[45861]: 2024-09-10T19:21:46.795+0000 7f96e5c6e640 -1 osd.0 262 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-10T19:21:46.945 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:21:46.981 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:21:46.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:21:46.982 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:21:47.083 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:21:47.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:21:47.110 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:21:47.110 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:21:47.182 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:21:46 smithi183 ceph-mon[29736]: 1.5 scrub starts 2024-09-10T19:21:47.182 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:21:46 smithi183 ceph-mon[29736]: 1.5 scrub ok 2024-09-10T19:21:47.182 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:21:46 smithi183 ceph-mon[29736]: pgmap v523: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 7.7 GiB data, 27 GiB used, 509 GiB / 536 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 2.58k op/s 2024-09-10T19:21:47.182 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:21:46 smithi183 ceph-mon[29736]: 5.8 scrub starts 2024-09-10T19:21:47.182 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:21:46 smithi183 ceph-mon[29736]: 5.8 scrub ok 2024-09-10T19:21:47.182 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:21:46 smithi183 ceph-mon[29736]: osd.0 marked itself down and dead 2024-09-10T19:21:47.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:21:46 smithi183 ceph-mon[41562]: 1.5 scrub starts 2024-09-10T19:21:47.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:21:46 smithi183 ceph-mon[41562]: 1.5 scrub ok 2024-09-10T19:21:47.183 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:21:46 smithi183 ceph-mon[41562]: pgmap v523: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 7.7 GiB data, 27 GiB used, 509 GiB / 536 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 2.58k op/s 2024-09-10T19:21:47.183 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:21:46 smithi183 ceph-mon[41562]: 5.8 scrub starts 2024-09-10T19:21:47.183 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:21:46 smithi183 ceph-mon[41562]: 5.8 scrub ok 2024-09-10T19:21:47.183 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:21:46 smithi183 ceph-mon[41562]: osd.0 marked itself down and dead 2024-09-10T19:21:47.211 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:21:47.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:21:46 smithi191 ceph-mon[32884]: 1.5 scrub starts 2024-09-10T19:21:47.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:21:46 smithi191 ceph-mon[32884]: 1.5 scrub ok 2024-09-10T19:21:47.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:21:46 smithi191 ceph-mon[32884]: pgmap v523: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 7.7 GiB data, 27 GiB used, 509 GiB / 536 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 2.58k op/s 2024-09-10T19:21:47.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:21:46 smithi191 ceph-mon[32884]: 5.8 scrub starts 2024-09-10T19:21:47.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:21:46 smithi191 ceph-mon[32884]: 5.8 scrub ok 2024-09-10T19:21:47.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:21:46 smithi191 ceph-mon[32884]: osd.0 marked itself down and dead 2024-09-10T19:21:47.233 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:21:47.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:21:47.234 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:21:47.335 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:21:47.356 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:21:47.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:21:47.357 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:21:47.458 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:21:47.479 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:21:47.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:21:47.480 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:21:47.581 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:21:47.603 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:21:47.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:21:47.604 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:21:47.705 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:21:47.715 INFO:journalctl@ceph.mgr.x.smithi191.stdout:Sep 10 19:21:47 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-x[34967]: ::ffff:172.21.15.191 - - [10/Sep/2024:19:21:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T19:21:47.728 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:21:47.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:21:47.729 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:21:47.830 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:21:47.851 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:21:47.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:21:47.852 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:21:47.953 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:21:47.979 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:21:47.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:21:47.980 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:21:48.080 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:21:48.118 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:21:48.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:21:48.119 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:21:48.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:21:47 smithi183 ceph-mon[29736]: 3.2 scrub starts 2024-09-10T19:21:48.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:21:47 smithi183 ceph-mon[29736]: 3.2 scrub ok 2024-09-10T19:21:48.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:21:47 smithi183 ceph-mon[29736]: Health check failed: 1 osds down (OSD_DOWN) 2024-09-10T19:21:48.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:21:47 smithi183 ceph-mon[29736]: osdmap e263: 8 total, 6 up, 6 in 2024-09-10T19:21:48.182 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:21:47 smithi183 ceph-mon[29736]: 1.3 scrub starts 2024-09-10T19:21:48.182 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:21:47 smithi183 ceph-mon[29736]: 1.3 scrub ok 2024-09-10T19:21:48.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:21:47 smithi183 ceph-mon[41562]: 3.2 scrub starts 2024-09-10T19:21:48.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:21:47 smithi183 ceph-mon[41562]: 3.2 scrub ok 2024-09-10T19:21:48.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:21:47 smithi183 ceph-mon[41562]: Health check failed: 1 osds down (OSD_DOWN) 2024-09-10T19:21:48.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:21:47 smithi183 ceph-mon[41562]: osdmap e263: 8 total, 6 up, 6 in 2024-09-10T19:21:48.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:21:47 smithi183 ceph-mon[41562]: 1.3 scrub starts 2024-09-10T19:21:48.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:21:47 smithi183 ceph-mon[41562]: 1.3 scrub ok 2024-09-10T19:21:48.220 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:21:48.257 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:21:48.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:21:48.258 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:21:48.359 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:21:48.384 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:21:48.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:21:48.385 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:21:48.465 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:21:47 smithi191 ceph-mon[32884]: 3.2 scrub starts 2024-09-10T19:21:48.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:21:47 smithi191 ceph-mon[32884]: 3.2 scrub ok 2024-09-10T19:21:48.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:21:47 smithi191 ceph-mon[32884]: Health check failed: 1 osds down (OSD_DOWN) 2024-09-10T19:21:48.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:21:47 smithi191 ceph-mon[32884]: osdmap e263: 8 total, 6 up, 6 in 2024-09-10T19:21:48.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:21:47 smithi191 ceph-mon[32884]: 1.3 scrub starts 2024-09-10T19:21:48.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:21:47 smithi191 ceph-mon[32884]: 1.3 scrub ok 2024-09-10T19:21:48.486 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:21:48.508 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:21:48.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:21:48.509 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:21:48.610 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:21:48.650 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:21:48.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:21:48.651 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:21:48.753 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:21:48.790 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:21:48.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:21:48.791 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:21:48.892 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:21:48.922 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:21:48.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:21:48.923 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:21:49.024 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:21:49.051 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:21:49.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:21:49.052 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:21:49.112 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:21:48 smithi183 ceph-mon[29736]: 6.7 scrub starts 2024-09-10T19:21:49.112 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:21:48 smithi183 ceph-mon[29736]: 6.7 scrub ok 2024-09-10T19:21:49.112 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:21:48 smithi183 ceph-mon[29736]: pgmap v525: 158 pgs: 1 stale+active+clean+scrubbing, 22 stale+active+clean, 135 active+clean; 7.7 GiB data, 27 GiB used, 509 GiB / 536 GiB avail; 742 B/s rd, 0 B/s wr, 1.12k op/s 2024-09-10T19:21:49.112 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:21:48 smithi183 ceph-mon[29736]: osdmap e264: 8 total, 6 up, 6 in 2024-09-10T19:21:49.112 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:21:48 smithi183 ceph-mon[29736]: 5.1f scrub starts 2024-09-10T19:21:49.112 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:21:48 smithi183 ceph-mon[29736]: 5.1f scrub ok 2024-09-10T19:21:49.112 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:21:48 smithi183 ceph-mon[41562]: 6.7 scrub starts 2024-09-10T19:21:49.112 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:21:48 smithi183 ceph-mon[41562]: 6.7 scrub ok 2024-09-10T19:21:49.112 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:21:48 smithi183 ceph-mon[41562]: pgmap v525: 158 pgs: 1 stale+active+clean+scrubbing, 22 stale+active+clean, 135 active+clean; 7.7 GiB data, 27 GiB used, 509 GiB / 536 GiB avail; 742 B/s rd, 0 B/s wr, 1.12k op/s 2024-09-10T19:21:49.113 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:21:48 smithi183 ceph-mon[41562]: osdmap e264: 8 total, 6 up, 6 in 2024-09-10T19:21:49.113 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:21:48 smithi183 ceph-mon[41562]: 5.1f scrub starts 2024-09-10T19:21:49.113 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:21:48 smithi183 ceph-mon[41562]: 5.1f scrub ok 2024-09-10T19:21:49.153 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:21:49.179 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:21:49.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:21:49.181 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:21:49.281 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:21:49.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:21:49.319 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:21:49.319 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:21:49.420 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:21:49.444 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:21:49.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:21:49.445 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:21:49.465 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:21:48 smithi191 ceph-mon[32884]: 6.7 scrub starts 2024-09-10T19:21:49.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:21:48 smithi191 ceph-mon[32884]: 6.7 scrub ok 2024-09-10T19:21:49.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:21:48 smithi191 ceph-mon[32884]: pgmap v525: 158 pgs: 1 stale+active+clean+scrubbing, 22 stale+active+clean, 135 active+clean; 7.7 GiB data, 27 GiB used, 509 GiB / 536 GiB avail; 742 B/s rd, 0 B/s wr, 1.12k op/s 2024-09-10T19:21:49.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:21:48 smithi191 ceph-mon[32884]: osdmap e264: 8 total, 6 up, 6 in 2024-09-10T19:21:49.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:21:48 smithi191 ceph-mon[32884]: 5.1f scrub starts 2024-09-10T19:21:49.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:21:48 smithi191 ceph-mon[32884]: 5.1f scrub ok 2024-09-10T19:21:49.497 INFO:teuthology.orchestra.run.smithi183.stdout:osd.2: {} 2024-09-10T19:21:49.497 INFO:teuthology.orchestra.run.smithi183.stdout:osd.3: {} 2024-09-10T19:21:49.497 INFO:teuthology.orchestra.run.smithi183.stdout:osd.4: {} 2024-09-10T19:21:49.497 INFO:teuthology.orchestra.run.smithi183.stdout:osd.5: {} 2024-09-10T19:21:49.497 INFO:teuthology.orchestra.run.smithi183.stdout:osd.6: {} 2024-09-10T19:21:49.497 INFO:teuthology.orchestra.run.smithi183.stdout:osd.7: {} 2024-09-10T19:21:49.498 INFO:teuthology.orchestra.run.smithi183.stderr:Error ENXIO: problem getting command descriptions from osd.0 2024-09-10T19:21:49.498 INFO:teuthology.orchestra.run.smithi183.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-09-10T19:21:49.498 INFO:teuthology.orchestra.run.smithi183.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-10T19:21:49.498 INFO:teuthology.orchestra.run.smithi183.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-10T19:21:49.498 INFO:teuthology.orchestra.run.smithi183.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-10T19:21:49.498 INFO:teuthology.orchestra.run.smithi183.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-10T19:21:49.498 INFO:teuthology.orchestra.run.smithi183.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-10T19:21:49.498 INFO:teuthology.orchestra.run.smithi183.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-10T19:21:49.546 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:21:49.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:21:49.579 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:21:49.579 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:21:49.680 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:21:49.714 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:21:49.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:21:49.715 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:21:49.816 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:21:49.859 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:21:49.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:21:49.860 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:21:49.961 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:21:49.965 INFO:journalctl@ceph.iscsi.iscsi.a.smithi191.stdout:Sep 10 19:21:49 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-iscsi-iscsi-a[61816]: debug there is no tcmu-runner data available 2024-09-10T19:21:49.980 INFO:teuthology.orchestra.run.smithi183.stderr:nodeep-scrub is unset 2024-09-10T19:21:49.987 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:21:49.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:21:49.988 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:21:50.089 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:21:50.111 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:21:50.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:21:50.113 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:21:50.213 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:21:50.251 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:21:50.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:21:50.252 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:21:50.353 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:21:50.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:21:50.388 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:21:50.389 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:21:50.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:21:49 smithi183 ceph-mon[29736]: 2.12 scrub starts 2024-09-10T19:21:50.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:21:49 smithi183 ceph-mon[29736]: 2.12 scrub ok 2024-09-10T19:21:50.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:21:49 smithi183 ceph-mon[29736]: 4.18 scrub starts 2024-09-10T19:21:50.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:21:49 smithi183 ceph-mon[29736]: 4.18 scrub ok 2024-09-10T19:21:50.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:21:49 smithi183 ceph-mon[29736]: from='client.? 172.21.15.183:0/2870021038' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-10T19:21:50.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:21:49 smithi183 ceph-mon[29736]: from='client.? 172.21.15.183:0/2811991542' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T19:21:50.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:21:49 smithi183 ceph-mon[29736]: 2.9 scrub starts 2024-09-10T19:21:50.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:21:49 smithi183 ceph-mon[29736]: 2.9 scrub ok 2024-09-10T19:21:50.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:21:49 smithi183 ceph-mon[41562]: 2.12 scrub starts 2024-09-10T19:21:50.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:21:49 smithi183 ceph-mon[41562]: 2.12 scrub ok 2024-09-10T19:21:50.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:21:49 smithi183 ceph-mon[41562]: 4.18 scrub starts 2024-09-10T19:21:50.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:21:49 smithi183 ceph-mon[41562]: 4.18 scrub ok 2024-09-10T19:21:50.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:21:49 smithi183 ceph-mon[41562]: from='client.? 172.21.15.183:0/2870021038' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-10T19:21:50.433 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:21:49 smithi183 ceph-mon[41562]: from='client.? 172.21.15.183:0/2811991542' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T19:21:50.433 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:21:49 smithi183 ceph-mon[41562]: 2.9 scrub starts 2024-09-10T19:21:50.433 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:21:49 smithi183 ceph-mon[41562]: 2.9 scrub ok 2024-09-10T19:21:50.465 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:21:49 smithi191 ceph-mon[32884]: 2.12 scrub starts 2024-09-10T19:21:50.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:21:49 smithi191 ceph-mon[32884]: 2.12 scrub ok 2024-09-10T19:21:50.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:21:49 smithi191 ceph-mon[32884]: 4.18 scrub starts 2024-09-10T19:21:50.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:21:49 smithi191 ceph-mon[32884]: 4.18 scrub ok 2024-09-10T19:21:50.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:21:49 smithi191 ceph-mon[32884]: from='client.? 172.21.15.183:0/2870021038' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-10T19:21:50.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:21:49 smithi191 ceph-mon[32884]: from='client.? 172.21.15.183:0/2811991542' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T19:21:50.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:21:49 smithi191 ceph-mon[32884]: 2.9 scrub starts 2024-09-10T19:21:50.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:21:49 smithi191 ceph-mon[32884]: 2.9 scrub ok 2024-09-10T19:21:50.489 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:21:50.513 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:21:50.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:21:50.514 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:21:50.615 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:21:50.652 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:21:50.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:21:50.653 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:21:50.753 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:21:50.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:21:50.780 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:21:50.780 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:21:50.881 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:21:50.915 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:21:50.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:21:50.916 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:21:51.017 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:21:51.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:21:51.055 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:21:51.056 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:21:51.156 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:21:51.179 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:21:51.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:21:51.180 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:21:51.281 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:21:51.318 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:21:50 smithi183 ceph-mon[29736]: from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:21:51.318 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:21:50 smithi183 ceph-mon[29736]: pgmap v527: 158 pgs: 1 stale+active+clean+scrubbing, 22 stale+active+clean, 135 active+clean; 7.7 GiB data, 27 GiB used, 509 GiB / 536 GiB avail; 853 B/s rd, 0 B/s wr, 1.29k op/s 2024-09-10T19:21:51.318 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:21:50 smithi183 ceph-mon[29736]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-10T19:21:51.318 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:21:50 smithi183 ceph-mon[29736]: from='client.? 172.21.15.183:0/2870021038' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-10T19:21:51.318 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:21:50 smithi183 ceph-mon[29736]: osdmap e265: 8 total, 6 up, 6 in 2024-09-10T19:21:51.318 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:21:50 smithi183 ceph-mon[29736]: 5.10 scrub starts 2024-09-10T19:21:51.318 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:21:50 smithi183 ceph-mon[29736]: 5.10 scrub ok 2024-09-10T19:21:51.318 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:21:50 smithi183 ceph-mon[41562]: from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:21:51.319 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:21:50 smithi183 ceph-mon[41562]: pgmap v527: 158 pgs: 1 stale+active+clean+scrubbing, 22 stale+active+clean, 135 active+clean; 7.7 GiB data, 27 GiB used, 509 GiB / 536 GiB avail; 853 B/s rd, 0 B/s wr, 1.29k op/s 2024-09-10T19:21:51.319 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:21:50 smithi183 ceph-mon[41562]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-10T19:21:51.319 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:21:50 smithi183 ceph-mon[41562]: from='client.? 172.21.15.183:0/2870021038' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-10T19:21:51.319 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:21:50 smithi183 ceph-mon[41562]: osdmap e265: 8 total, 6 up, 6 in 2024-09-10T19:21:51.319 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:21:50 smithi183 ceph-mon[41562]: 5.10 scrub starts 2024-09-10T19:21:51.319 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:21:50 smithi183 ceph-mon[41562]: 5.10 scrub ok 2024-09-10T19:21:51.319 INFO:journalctl@ceph.osd.0.smithi183.stdout:Sep 10 19:21:51 smithi183 podman[155351]: 2024-09-10 19:21:51.208837476 +0000 UTC m=+4.561679756 container died d9768e93e68d6cbaadcb3f963d4de94e38669cd8254ba2e1a218940bc2fb4f42 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3bfeaf4417585c821cab9ca76e4d95f390b0e769807122a45463640820b5cd16, name=ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd-0, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-guits-quincy-2024-09-09-1810-quincy-5be5739, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, ceph=True) 2024-09-10T19:21:51.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:21:51.320 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:21:51.321 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:21:51.421 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:21:51.458 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:21:51.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:21:51.459 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:21:51.465 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:21:50 smithi191 ceph-mon[32884]: from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:21:51.465 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:21:50 smithi191 ceph-mon[32884]: pgmap v527: 158 pgs: 1 stale+active+clean+scrubbing, 22 stale+active+clean, 135 active+clean; 7.7 GiB data, 27 GiB used, 509 GiB / 536 GiB avail; 853 B/s rd, 0 B/s wr, 1.29k op/s 2024-09-10T19:21:51.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:21:50 smithi191 ceph-mon[32884]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-10T19:21:51.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:21:50 smithi191 ceph-mon[32884]: from='client.? 172.21.15.183:0/2870021038' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-10T19:21:51.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:21:50 smithi191 ceph-mon[32884]: osdmap e265: 8 total, 6 up, 6 in 2024-09-10T19:21:51.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:21:50 smithi191 ceph-mon[32884]: 5.10 scrub starts 2024-09-10T19:21:51.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:21:50 smithi191 ceph-mon[32884]: 5.10 scrub ok 2024-09-10T19:21:51.560 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:21:51.591 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:21:51.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:21:51.592 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:21:51.681 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Sep 10 19:21:51 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-y[29989]: ::ffff:172.21.15.191 - - [10/Sep/2024:19:21:51] "GET /metrics HTTP/1.1" 200 209422 "" "Prometheus/2.43.0" 2024-09-10T19:21:51.693 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:21:51.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:21:51.731 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:21:51.731 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:21:51.832 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:21:51.856 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:21:51.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:21:51.857 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:21:51.958 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:21:51.983 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:21:51.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:21:51.984 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:21:52.085 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:21:52.100 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:21:51 smithi183 ceph-mon[29736]: 6.1b scrub starts 2024-09-10T19:21:52.100 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:21:51 smithi183 ceph-mon[29736]: 6.1b scrub ok 2024-09-10T19:21:52.100 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:21:51 smithi183 ceph-mon[41562]: 6.1b scrub starts 2024-09-10T19:21:52.100 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:21:51 smithi183 ceph-mon[41562]: 6.1b scrub ok 2024-09-10T19:21:52.100 INFO:journalctl@ceph.osd.0.smithi183.stdout:Sep 10 19:21:51 smithi183 podman[155351]: 2024-09-10 19:21:51.994404814 +0000 UTC m=+5.347247089 container cleanup d9768e93e68d6cbaadcb3f963d4de94e38669cd8254ba2e1a218940bc2fb4f42 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3bfeaf4417585c821cab9ca76e4d95f390b0e769807122a45463640820b5cd16, name=ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd-0, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , ceph=True, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, RELEASE=wip-guits-quincy-2024-09-09-1810-quincy-5be5739, org.label-schema.vendor=CentOS) 2024-09-10T19:21:52.100 INFO:journalctl@ceph.osd.0.smithi183.stdout:Sep 10 19:21:51 smithi183 bash[155351]: ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd-0 2024-09-10T19:21:52.110 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:21:52.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:21:52.111 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:21:52.212 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:21:52.246 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:21:52.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:21:52.247 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:21:52.348 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:21:52.382 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:21:52.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:21:52.383 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:21:52.392 INFO:journalctl@ceph.osd.0.smithi183.stdout:Sep 10 19:21:52 smithi183 podman[155477]: 2024-09-10 19:21:52.212939384 +0000 UTC m=+4.831345463 container remove d9768e93e68d6cbaadcb3f963d4de94e38669cd8254ba2e1a218940bc2fb4f42 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3bfeaf4417585c821cab9ca76e4d95f390b0e769807122a45463640820b5cd16, name=ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd-0, GIT_CLEAN=True, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-guits-quincy-2024-09-09-1810-quincy-5be5739, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, ceph=True, org.label-schema.license=GPLv2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7) 2024-09-10T19:21:52.465 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:21:51 smithi191 ceph-mon[32884]: 6.1b scrub starts 2024-09-10T19:21:52.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:21:51 smithi191 ceph-mon[32884]: 6.1b scrub ok 2024-09-10T19:21:52.484 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:21:52.516 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:21:52.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:21:52.517 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:21:52.618 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:21:52.655 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:21:52.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:21:52.656 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:21:52.756 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:21:52.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:21:52.792 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:21:52.792 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:21:52.893 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:21:52.920 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:21:52.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:21:52.921 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:21:53.022 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:21:53.024 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:21:53.058 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:21:53.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:21:53.059 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:21:53.159 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:21:53.199 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:21:53.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:21:53.200 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:21:53.301 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:21:53.341 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:21:53.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:21:53.342 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:21:53.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:21:52 smithi183 ceph-mon[29736]: pgmap v529: 158 pgs: 45 active+undersized, 33 active+undersized+degraded, 80 active+clean; 7.3 GiB data, 27 GiB used, 510 GiB / 536 GiB avail; 0 B/s wr, 962 op/s; 60403/361050 objects degraded (16.730%); 0 B/s, 0 objects/s recovering 2024-09-10T19:21:53.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:21:52 smithi183 ceph-mon[29736]: Health check failed: Degraded data redundancy: 60403/361050 objects degraded (16.730%), 33 pgs degraded (PG_DEGRADED) 2024-09-10T19:21:53.431 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:21:52 smithi183 ceph-mon[41562]: pgmap v529: 158 pgs: 45 active+undersized, 33 active+undersized+degraded, 80 active+clean; 7.3 GiB data, 27 GiB used, 510 GiB / 536 GiB avail; 0 B/s wr, 962 op/s; 60403/361050 objects degraded (16.730%); 0 B/s, 0 objects/s recovering 2024-09-10T19:21:53.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:21:52 smithi183 ceph-mon[41562]: Health check failed: Degraded data redundancy: 60403/361050 objects degraded (16.730%), 33 pgs degraded (PG_DEGRADED) 2024-09-10T19:21:53.443 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:21:53.465 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:21:52 smithi191 ceph-mon[32884]: pgmap v529: 158 pgs: 45 active+undersized, 33 active+undersized+degraded, 80 active+clean; 7.3 GiB data, 27 GiB used, 510 GiB / 536 GiB avail; 0 B/s wr, 962 op/s; 60403/361050 objects degraded (16.730%); 0 B/s, 0 objects/s recovering 2024-09-10T19:21:53.465 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:21:52 smithi191 ceph-mon[32884]: Health check failed: Degraded data redundancy: 60403/361050 objects degraded (16.730%), 33 pgs degraded (PG_DEGRADED) 2024-09-10T19:21:53.482 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:21:53.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:21:53.483 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:21:53.575 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-10T19:21:53.575 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-10T19:21:53.584 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:21:53.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:21:53.616 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:21:53.617 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:21:53.717 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:21:53.753 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:21:53.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:21:53.754 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:21:53.855 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:21:53.888 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:21:53.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:21:53.889 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:21:53.990 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:21:54.024 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:21:54.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:21:54.026 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:21:54.101 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:21:53 smithi183 ceph-mon[29736]: 5.18 scrub starts 2024-09-10T19:21:54.101 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:21:53 smithi183 ceph-mon[29736]: 5.18 scrub ok 2024-09-10T19:21:54.101 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:21:53 smithi183 ceph-mon[41562]: 5.18 scrub starts 2024-09-10T19:21:54.101 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:21:53 smithi183 ceph-mon[41562]: 5.18 scrub ok 2024-09-10T19:21:54.126 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:21:54.161 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:21:54.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:21:54.162 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:21:54.262 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:21:54.298 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:21:54.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:21:54.299 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:21:54.400 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:21:54.427 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:21:54.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:21:54.428 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:21:54.431 INFO:journalctl@ceph.osd.0.smithi183.stdout:Sep 10 19:21:54 smithi183 podman[155881]: 2024-09-10 19:21:54.100361532 +0000 UTC m=+2.082064240 container died 63e27cae449c9a770a25118cb4173d25672679bbf7f666db899f46afa651b7da (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588, name=strange_austin, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-guits-quincy-2024-09-09-1810-quincy-5be5739, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, ceph=True, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2) 2024-09-10T19:21:54.465 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:21:53 smithi191 ceph-mon[32884]: 5.18 scrub starts 2024-09-10T19:21:54.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:21:53 smithi191 ceph-mon[32884]: 5.18 scrub ok 2024-09-10T19:21:54.529 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:21:54.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:21:54.568 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:21:54.568 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:21:54.669 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:21:54.706 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:21:54.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:21:54.707 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:21:54.808 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:21:54.850 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:21:54.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:21:54.851 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:21:54.951 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:21:54.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:21:54.990 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:21:54.990 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:21:55.040 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph osd set noscrub 2024-09-10T19:21:55.091 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:21:55.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:21:55.124 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:21:55.124 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:21:55.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:21:54 smithi183 ceph-mon[29736]: 1.a deep-scrub starts 2024-09-10T19:21:55.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:21:54 smithi183 ceph-mon[29736]: 1.a deep-scrub ok 2024-09-10T19:21:55.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:21:54 smithi183 ceph-mon[29736]: 6.1a scrub starts 2024-09-10T19:21:55.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:21:54 smithi183 ceph-mon[29736]: 6.1a scrub ok 2024-09-10T19:21:55.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:21:54 smithi183 ceph-mon[29736]: pgmap v530: 158 pgs: 45 active+undersized, 33 active+undersized+degraded, 80 active+clean; 7.3 GiB data, 27 GiB used, 510 GiB / 536 GiB avail; 0 B/s wr, 842 op/s; 60403/361050 objects degraded (16.730%); 0 B/s, 0 objects/s recovering 2024-09-10T19:21:55.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:21:54 smithi183 ceph-mon[29736]: 2.11 scrub starts 2024-09-10T19:21:55.182 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:21:54 smithi183 ceph-mon[29736]: 2.11 scrub ok 2024-09-10T19:21:55.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:21:54 smithi183 ceph-mon[41562]: 1.a deep-scrub starts 2024-09-10T19:21:55.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:21:54 smithi183 ceph-mon[41562]: 1.a deep-scrub ok 2024-09-10T19:21:55.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:21:54 smithi183 ceph-mon[41562]: 6.1a scrub starts 2024-09-10T19:21:55.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:21:54 smithi183 ceph-mon[41562]: 6.1a scrub ok 2024-09-10T19:21:55.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:21:54 smithi183 ceph-mon[41562]: pgmap v530: 158 pgs: 45 active+undersized, 33 active+undersized+degraded, 80 active+clean; 7.3 GiB data, 27 GiB used, 510 GiB / 536 GiB avail; 0 B/s wr, 842 op/s; 60403/361050 objects degraded (16.730%); 0 B/s, 0 objects/s recovering 2024-09-10T19:21:55.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:21:54 smithi183 ceph-mon[41562]: 2.11 scrub starts 2024-09-10T19:21:55.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:21:54 smithi183 ceph-mon[41562]: 2.11 scrub ok 2024-09-10T19:21:55.225 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:21:55.257 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:21:55.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:21:55.258 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:21:55.359 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:21:55.384 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:21:55.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:21:55.385 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:21:55.465 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:21:54 smithi191 ceph-mon[32884]: 1.a deep-scrub starts 2024-09-10T19:21:55.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:21:54 smithi191 ceph-mon[32884]: 1.a deep-scrub ok 2024-09-10T19:21:55.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:21:54 smithi191 ceph-mon[32884]: 6.1a scrub starts 2024-09-10T19:21:55.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:21:54 smithi191 ceph-mon[32884]: 6.1a scrub ok 2024-09-10T19:21:55.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:21:54 smithi191 ceph-mon[32884]: pgmap v530: 158 pgs: 45 active+undersized, 33 active+undersized+degraded, 80 active+clean; 7.3 GiB data, 27 GiB used, 510 GiB / 536 GiB avail; 0 B/s wr, 842 op/s; 60403/361050 objects degraded (16.730%); 0 B/s, 0 objects/s recovering 2024-09-10T19:21:55.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:21:54 smithi191 ceph-mon[32884]: 2.11 scrub starts 2024-09-10T19:21:55.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:21:54 smithi191 ceph-mon[32884]: 2.11 scrub ok 2024-09-10T19:21:55.485 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:21:55.491 INFO:teuthology.orchestra.run.smithi183.stdout:ERROR: (22) Invalid argument 2024-09-10T19:21:55.491 INFO:teuthology.orchestra.run.smithi183.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-10T19:21:55.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:21:55.526 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:21:55.526 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:21:55.576 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-10T19:21:55.627 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:21:55.666 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:21:55.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:21:55.667 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:21:55.768 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:21:55.800 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:21:55.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:21:55.801 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:21:55.902 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:21:55.941 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:21:55.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:21:55.942 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:21:56.043 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:21:56.066 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:21:56.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:21:56.067 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:21:56.168 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:21:56.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:21:55 smithi183 ceph-mon[29736]: 1.8 scrub starts 2024-09-10T19:21:56.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:21:55 smithi183 ceph-mon[29736]: 1.8 scrub ok 2024-09-10T19:21:56.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:21:55 smithi183 ceph-mon[29736]: 2.7 scrub starts 2024-09-10T19:21:56.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:21:55 smithi183 ceph-mon[29736]: 2.7 scrub ok 2024-09-10T19:21:56.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:21:55 smithi183 ceph-mon[41562]: 1.8 scrub starts 2024-09-10T19:21:56.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:21:55 smithi183 ceph-mon[41562]: 1.8 scrub ok 2024-09-10T19:21:56.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:21:55 smithi183 ceph-mon[41562]: 2.7 scrub starts 2024-09-10T19:21:56.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:21:55 smithi183 ceph-mon[41562]: 2.7 scrub ok 2024-09-10T19:21:56.182 INFO:journalctl@ceph.osd.0.smithi183.stdout:Sep 10 19:21:55 smithi183 podman[156214]: 2024-09-10 19:21:55.764545022 +0000 UTC m=+0.256025640 image pull 4c55dd4096115e48502f368cff1eab06769f71f030e1692e2611bbc6c6f637e9 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3bfeaf4417585c821cab9ca76e4d95f390b0e769807122a45463640820b5cd16 2024-09-10T19:21:56.191 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:21:56.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:21:56.192 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:21:56.293 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:21:56.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:21:56.328 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:21:56.329 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:21:56.354 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:21:56.384 DEBUG:teuthology.orchestra.run.smithi191:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph daemon osd.5 dump_ops_in_flight 2024-09-10T19:21:56.415 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:21:55 smithi191 ceph-mon[32884]: 1.8 scrub starts 2024-09-10T19:21:56.415 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:21:55 smithi191 ceph-mon[32884]: 1.8 scrub ok 2024-09-10T19:21:56.415 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:21:55 smithi191 ceph-mon[32884]: 2.7 scrub starts 2024-09-10T19:21:56.415 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:21:55 smithi191 ceph-mon[32884]: 2.7 scrub ok 2024-09-10T19:21:56.429 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:21:56.452 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:21:56.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:21:56.453 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:21:56.553 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:21:56.598 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:21:56.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:21:56.598 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:21:56.700 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:21:56.724 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:21:56.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:21:56.725 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:21:56.806 INFO:teuthology.orchestra.run.smithi191.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.b/config 2024-09-10T19:21:56.826 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:21:56.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:21:56.863 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:21:56.864 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:21:56.964 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:21:56.998 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:21:56.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:21:57.000 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:21:57.100 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:21:57.136 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:21:57.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:21:57.137 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:21:57.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:21:56 smithi191 ceph-mon[32884]: 4.1 deep-scrub starts 2024-09-10T19:21:57.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:21:56 smithi191 ceph-mon[32884]: 4.1 deep-scrub ok 2024-09-10T19:21:57.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:21:56 smithi191 ceph-mon[32884]: pgmap v531: 158 pgs: 45 active+undersized, 33 active+undersized+degraded, 80 active+clean; 6.9 GiB data, 25 GiB used, 511 GiB / 536 GiB avail; 0 B/s wr, 1.67k op/s; 56578/338205 objects degraded (16.729%); 0 B/s, 0 objects/s recovering 2024-09-10T19:21:57.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:21:56 smithi191 ceph-mon[32884]: 5.11 scrub starts 2024-09-10T19:21:57.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:21:56 smithi191 ceph-mon[32884]: 5.11 scrub ok 2024-09-10T19:21:57.238 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:21:57.259 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:21:57.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:21:57.260 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:21:57.360 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:21:57.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:21:57.382 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:21:57.382 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:21:57.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:21:56 smithi183 ceph-mon[29736]: 4.1 deep-scrub starts 2024-09-10T19:21:57.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:21:56 smithi183 ceph-mon[29736]: 4.1 deep-scrub ok 2024-09-10T19:21:57.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:21:56 smithi183 ceph-mon[29736]: pgmap v531: 158 pgs: 45 active+undersized, 33 active+undersized+degraded, 80 active+clean; 6.9 GiB data, 25 GiB used, 511 GiB / 536 GiB avail; 0 B/s wr, 1.67k op/s; 56578/338205 objects degraded (16.729%); 0 B/s, 0 objects/s recovering 2024-09-10T19:21:57.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:21:56 smithi183 ceph-mon[29736]: 5.11 scrub starts 2024-09-10T19:21:57.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:21:56 smithi183 ceph-mon[29736]: 5.11 scrub ok 2024-09-10T19:21:57.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:21:56 smithi183 ceph-mon[41562]: 4.1 deep-scrub starts 2024-09-10T19:21:57.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:21:56 smithi183 ceph-mon[41562]: 4.1 deep-scrub ok 2024-09-10T19:21:57.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:21:56 smithi183 ceph-mon[41562]: pgmap v531: 158 pgs: 45 active+undersized, 33 active+undersized+degraded, 80 active+clean; 6.9 GiB data, 25 GiB used, 511 GiB / 536 GiB avail; 0 B/s wr, 1.67k op/s; 56578/338205 objects degraded (16.729%); 0 B/s, 0 objects/s recovering 2024-09-10T19:21:57.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:21:56 smithi183 ceph-mon[41562]: 5.11 scrub starts 2024-09-10T19:21:57.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:21:56 smithi183 ceph-mon[41562]: 5.11 scrub ok 2024-09-10T19:21:57.432 INFO:journalctl@ceph.osd.0.smithi183.stdout:Sep 10 19:21:57 smithi183 podman[156214]: 2024-09-10 19:21:57.15103839 +0000 UTC m=+1.642518995 container create 8ac7f983457b51a6f468ea44b6fd13d0131860f36d1238250cf0cf53f54dc7d4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3bfeaf4417585c821cab9ca76e4d95f390b0e769807122a45463640820b5cd16, name=ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd-0-deactivate, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, RELEASE=wip-guits-quincy-2024-09-09-1810-quincy-5be5739, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731) 2024-09-10T19:21:57.483 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:21:57.505 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:21:57.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:21:57.506 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:21:57.607 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:21:57.645 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:21:57.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:21:57.646 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:21:57.694 INFO:journalctl@ceph.mgr.x.smithi191.stdout:Sep 10 19:21:57 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-x[34967]: ::ffff:172.21.15.191 - - [10/Sep/2024:19:21:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T19:21:57.747 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:21:57.780 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:21:57.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:21:57.781 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:21:57.882 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:21:57.907 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:21:57.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:21:57.908 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:21:58.009 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:21:58.052 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:21:58.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:21:58.053 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:21:58.154 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:21:58.175 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:21:58.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:21:58.176 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:21:58.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:21:58 smithi183 ceph-mon[29736]: 9.d deep-scrub starts 2024-09-10T19:21:58.181 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:21:58 smithi183 ceph-mon[41562]: 9.d deep-scrub starts 2024-09-10T19:21:58.277 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:21:58.299 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:21:58.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:21:58.300 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:21:58.400 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:21:58.423 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:21:58.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:21:58.425 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:21:58.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:21:58 smithi191 ceph-mon[32884]: 9.d deep-scrub starts 2024-09-10T19:21:58.525 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:21:58.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:21:58.565 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:21:58.565 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:21:58.666 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:21:58.697 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:21:58.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:21:58.698 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:21:58.799 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:21:58.826 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:21:58.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:21:58.827 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:21:58.927 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:21:58.931 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Sep 10 19:21:58 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-y[29989]: 2024-09-10T19:21:58.673+0000 7f9fad6cd640 -1 mgr get_metadata_python Requested missing service osd.1 2024-09-10T19:21:58.931 INFO:journalctl@ceph.osd.0.smithi183.stdout:Sep 10 19:21:58 smithi183 podman[156214]: 2024-09-10 19:21:58.685542261 +0000 UTC m=+3.177022881 container init 8ac7f983457b51a6f468ea44b6fd13d0131860f36d1238250cf0cf53f54dc7d4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3bfeaf4417585c821cab9ca76e4d95f390b0e769807122a45463640820b5cd16, name=ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd-0-deactivate, io.buildah.version=1.37.2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=wip-guits-quincy-2024-09-09-1810-quincy-5be5739, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, ceph=True, CEPH_POINT_RELEASE=) 2024-09-10T19:21:58.931 INFO:journalctl@ceph.osd.0.smithi183.stdout:Sep 10 19:21:58 smithi183 podman[156214]: 2024-09-10 19:21:58.691532631 +0000 UTC m=+3.183013243 container start 8ac7f983457b51a6f468ea44b6fd13d0131860f36d1238250cf0cf53f54dc7d4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3bfeaf4417585c821cab9ca76e4d95f390b0e769807122a45463640820b5cd16, name=ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd-0-deactivate, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, ceph=True, CEPH_POINT_RELEASE=, RELEASE=wip-guits-quincy-2024-09-09-1810-quincy-5be5739, io.buildah.version=1.37.2, GIT_CLEAN=True) 2024-09-10T19:21:58.951 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:21:58.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:21:58.952 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:21:59.053 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:21:59.077 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:21:59.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:21:59.078 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:21:59.178 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:21:59.201 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:21:59.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:21:59.202 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:21:59.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:21:59 smithi191 ceph-mon[32884]: 4.15 scrub starts 2024-09-10T19:21:59.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:21:59 smithi191 ceph-mon[32884]: 4.15 scrub ok 2024-09-10T19:21:59.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:21:59 smithi191 ceph-mon[32884]: 6.1f scrub starts 2024-09-10T19:21:59.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:21:59 smithi191 ceph-mon[32884]: 6.1f scrub ok 2024-09-10T19:21:59.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:21:59 smithi191 ceph-mon[32884]: pgmap v532: 158 pgs: 45 active+undersized, 33 active+undersized+degraded, 80 active+clean; 6.9 GiB data, 25 GiB used, 511 GiB / 536 GiB avail; 0 B/s wr, 1.36k op/s; 56578/338205 objects degraded (16.729%); 0 B/s, 0 objects/s recovering 2024-09-10T19:21:59.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:21:59 smithi191 ceph-mon[32884]: from='mgr.14556 ' entity='mgr.y' 2024-09-10T19:21:59.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:21:59 smithi191 ceph-mon[32884]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:21:59.281 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:21:59 smithi183 ceph-mon[29736]: 4.15 scrub starts 2024-09-10T19:21:59.281 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:21:59 smithi183 ceph-mon[29736]: 4.15 scrub ok 2024-09-10T19:21:59.281 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:21:59 smithi183 ceph-mon[29736]: 6.1f scrub starts 2024-09-10T19:21:59.281 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:21:59 smithi183 ceph-mon[29736]: 6.1f scrub ok 2024-09-10T19:21:59.281 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:21:59 smithi183 ceph-mon[29736]: pgmap v532: 158 pgs: 45 active+undersized, 33 active+undersized+degraded, 80 active+clean; 6.9 GiB data, 25 GiB used, 511 GiB / 536 GiB avail; 0 B/s wr, 1.36k op/s; 56578/338205 objects degraded (16.729%); 0 B/s, 0 objects/s recovering 2024-09-10T19:21:59.281 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:21:59 smithi183 ceph-mon[29736]: from='mgr.14556 ' entity='mgr.y' 2024-09-10T19:21:59.281 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:21:59 smithi183 ceph-mon[29736]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:21:59.281 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:21:59 smithi183 ceph-mon[41562]: 4.15 scrub starts 2024-09-10T19:21:59.282 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:21:59 smithi183 ceph-mon[41562]: 4.15 scrub ok 2024-09-10T19:21:59.282 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:21:59 smithi183 ceph-mon[41562]: 6.1f scrub starts 2024-09-10T19:21:59.282 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:21:59 smithi183 ceph-mon[41562]: 6.1f scrub ok 2024-09-10T19:21:59.282 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:21:59 smithi183 ceph-mon[41562]: pgmap v532: 158 pgs: 45 active+undersized, 33 active+undersized+degraded, 80 active+clean; 6.9 GiB data, 25 GiB used, 511 GiB / 536 GiB avail; 0 B/s wr, 1.36k op/s; 56578/338205 objects degraded (16.729%); 0 B/s, 0 objects/s recovering 2024-09-10T19:21:59.282 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:21:59 smithi183 ceph-mon[41562]: from='mgr.14556 ' entity='mgr.y' 2024-09-10T19:21:59.282 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:21:59 smithi183 ceph-mon[41562]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:21:59.302 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:21:59.340 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:21:59.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:21:59.341 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:21:59.442 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:21:59.465 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:21:59.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:21:59.466 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:21:59.567 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:21:59.588 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:21:59.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:21:59.590 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:21:59.659 INFO:journalctl@ceph.osd.0.smithi183.stdout:Sep 10 19:21:59 smithi183 podman[156214]: 2024-09-10 19:21:59.279774804 +0000 UTC m=+3.771255415 container attach 8ac7f983457b51a6f468ea44b6fd13d0131860f36d1238250cf0cf53f54dc7d4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3bfeaf4417585c821cab9ca76e4d95f390b0e769807122a45463640820b5cd16, name=ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd-0-deactivate, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, RELEASE=wip-guits-quincy-2024-09-09-1810-quincy-5be5739, org.label-schema.license=GPLv2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-10T19:21:59.659 INFO:journalctl@ceph.osd.0.smithi183.stdout:Sep 10 19:21:59 smithi183 podman[156214]: 2024-09-10 19:21:59.280120709 +0000 UTC m=+3.771601315 container died 8ac7f983457b51a6f468ea44b6fd13d0131860f36d1238250cf0cf53f54dc7d4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3bfeaf4417585c821cab9ca76e4d95f390b0e769807122a45463640820b5cd16, name=ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd-0-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_CLEAN=True, ceph=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, CEPH_POINT_RELEASE=, RELEASE=wip-guits-quincy-2024-09-09-1810-quincy-5be5739, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-10T19:21:59.690 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:21:59.723 INFO:teuthology.orchestra.run.smithi191.stdout:ERROR: (22) Invalid argument 2024-09-10T19:21:59.724 INFO:teuthology.orchestra.run.smithi191.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-10T19:21:59.735 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:21:59.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:21:59.737 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:21:59.838 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:21:59.874 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:21:59.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:21:59.875 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:21:59.965 INFO:journalctl@ceph.iscsi.iscsi.a.smithi191.stdout:Sep 10 19:21:59 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-iscsi-iscsi-a[61816]: debug there is no tcmu-runner data available 2024-09-10T19:21:59.976 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:21:59.997 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:21:59.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:21:59.999 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:22:00.099 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:22:00.131 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:22:00.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:22:00.132 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:22:00.233 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:22:00.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:22:00.268 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:22:00.268 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:22:00.369 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:22:00.393 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:22:00.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:22:00.394 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:22:00.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:22:00 smithi183 ceph-mon[29736]: 4.8 scrub starts 2024-09-10T19:22:00.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:22:00 smithi183 ceph-mon[29736]: 4.8 scrub ok 2024-09-10T19:22:00.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:22:00 smithi183 ceph-mon[29736]: 6.e scrub starts 2024-09-10T19:22:00.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:22:00 smithi183 ceph-mon[29736]: 6.e scrub ok 2024-09-10T19:22:00.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:22:00 smithi183 ceph-mon[29736]: Health check update: Degraded data redundancy: 56578/338205 objects degraded (16.729%), 33 pgs degraded (PG_DEGRADED) 2024-09-10T19:22:00.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:22:00 smithi183 ceph-mon[41562]: 4.8 scrub starts 2024-09-10T19:22:00.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:22:00 smithi183 ceph-mon[41562]: 4.8 scrub ok 2024-09-10T19:22:00.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:22:00 smithi183 ceph-mon[41562]: 6.e scrub starts 2024-09-10T19:22:00.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:22:00 smithi183 ceph-mon[41562]: 6.e scrub ok 2024-09-10T19:22:00.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:22:00 smithi183 ceph-mon[41562]: Health check update: Degraded data redundancy: 56578/338205 objects degraded (16.729%), 33 pgs degraded (PG_DEGRADED) 2024-09-10T19:22:00.465 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:22:00 smithi191 ceph-mon[32884]: 4.8 scrub starts 2024-09-10T19:22:00.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:22:00 smithi191 ceph-mon[32884]: 4.8 scrub ok 2024-09-10T19:22:00.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:22:00 smithi191 ceph-mon[32884]: 6.e scrub starts 2024-09-10T19:22:00.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:22:00 smithi191 ceph-mon[32884]: 6.e scrub ok 2024-09-10T19:22:00.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:22:00 smithi191 ceph-mon[32884]: Health check update: Degraded data redundancy: 56578/338205 objects degraded (16.729%), 33 pgs degraded (PG_DEGRADED) 2024-09-10T19:22:00.495 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:22:00.520 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:22:00.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:22:00.521 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:22:00.621 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:22:00.659 DEBUG:teuthology.orchestra.run.smithi191:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph daemon osd.5 dump_blocked_ops 2024-09-10T19:22:00.678 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:22:00.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:22:00.680 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:22:00.780 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:22:00.819 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:22:00.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:22:00.820 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:22:00.921 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:22:00.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:22:00.962 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:22:00.962 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:22:01.063 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:22:01.087 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:22:01.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:22:01.088 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:22:01.115 INFO:teuthology.orchestra.run.smithi191.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.b/config 2024-09-10T19:22:01.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:22:01 smithi183 ceph-mon[29736]: 4.e scrub starts 2024-09-10T19:22:01.182 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:22:01 smithi183 ceph-mon[29736]: 4.e scrub ok 2024-09-10T19:22:01.182 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:22:01 smithi183 ceph-mon[29736]: from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:22:01.182 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:22:01 smithi183 ceph-mon[29736]: pgmap v533: 158 pgs: 45 active+undersized, 33 active+undersized+degraded, 80 active+clean; 6.9 GiB data, 25 GiB used, 511 GiB / 536 GiB avail; 0 B/s wr, 1.34k op/s; 56578/338205 objects degraded (16.729%); 0 B/s, 0 objects/s recovering 2024-09-10T19:22:01.182 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:22:01 smithi183 ceph-mon[29736]: 9.3 scrub starts 2024-09-10T19:22:01.183 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:22:01 smithi183 ceph-mon[41562]: 4.e scrub starts 2024-09-10T19:22:01.183 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:22:01 smithi183 ceph-mon[41562]: 4.e scrub ok 2024-09-10T19:22:01.183 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:22:01 smithi183 ceph-mon[41562]: from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:22:01.183 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:22:01 smithi183 ceph-mon[41562]: pgmap v533: 158 pgs: 45 active+undersized, 33 active+undersized+degraded, 80 active+clean; 6.9 GiB data, 25 GiB used, 511 GiB / 536 GiB avail; 0 B/s wr, 1.34k op/s; 56578/338205 objects degraded (16.729%); 0 B/s, 0 objects/s recovering 2024-09-10T19:22:01.183 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:22:01 smithi183 ceph-mon[41562]: 9.3 scrub starts 2024-09-10T19:22:01.189 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:22:01.213 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:22:01.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:22:01.214 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:22:01.294 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:22:01 smithi191 ceph-mon[32884]: 4.e scrub starts 2024-09-10T19:22:01.294 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:22:01 smithi191 ceph-mon[32884]: 4.e scrub ok 2024-09-10T19:22:01.294 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:22:01 smithi191 ceph-mon[32884]: from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:22:01.294 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:22:01 smithi191 ceph-mon[32884]: pgmap v533: 158 pgs: 45 active+undersized, 33 active+undersized+degraded, 80 active+clean; 6.9 GiB data, 25 GiB used, 511 GiB / 536 GiB avail; 0 B/s wr, 1.34k op/s; 56578/338205 objects degraded (16.729%); 0 B/s, 0 objects/s recovering 2024-09-10T19:22:01.294 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:22:01 smithi191 ceph-mon[32884]: 9.3 scrub starts 2024-09-10T19:22:01.315 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:22:01.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:22:01.341 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:22:01.341 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:22:01.343 DEBUG:teuthology.orchestra.run.smithi183:> sudo pkill -f 'journalctl -f -n 0 -u ceph-320f6df8-6fa7-11ef-bcea-c7b262605968@osd.0.service' 2024-09-10T19:22:01.427 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-10T19:22:01.427 INFO:tasks.cephadm.osd.0:Stopped osd.0 2024-09-10T19:22:01.427 INFO:tasks.thrashosds.thrasher:Testing ceph-objectstore-tool on down osd.0 2024-09-10T19:22:01.428 INFO:tasks.thrashosds.thrasher:remote for osd 0 is ubuntu@smithi183.front.sepia.ceph.com 2024-09-10T19:22:01.428 INFO:tasks.thrashosds.thrasher:remote for exp osd 1 is ubuntu@smithi183.front.sepia.ceph.com 2024-09-10T19:22:01.428 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell -n osd.1 --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-1 --journal-path /var/lib/ceph/osd/ceph-1/journal --op list-pgs 2024-09-10T19:22:01.442 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:22:01.490 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:22:01.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:22:01.491 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:22:01.580 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Sep 10 19:22:01 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-y[29989]: ::ffff:172.21.15.191 - - [10/Sep/2024:19:22:01] "GET /metrics HTTP/1.1" 200 209492 "" "Prometheus/2.43.0" 2024-09-10T19:22:01.592 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:22:01.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:22:01.619 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:22:01.619 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:22:01.638 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:22:01.720 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:22:01.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:22:01.760 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:22:01.760 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:22:01.861 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:22:01.903 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:22:01.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:22:01.903 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:22:01.928 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/osd.1/config 2024-09-10T19:22:02.004 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:22:02.032 INFO:teuthology.orchestra.run.smithi183.stderr:noscrub is set 2024-09-10T19:22:02.047 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:22:02.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:22:02.048 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:22:02.149 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:22:02.173 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:22:02.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:22:02.174 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:22:02.224 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:22:02 smithi183 ceph-mon[29736]: 4.9 scrub starts 2024-09-10T19:22:02.224 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:22:02 smithi183 ceph-mon[29736]: 4.9 scrub ok 2024-09-10T19:22:02.224 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:22:02 smithi183 ceph-mon[29736]: 5.14 scrub starts 2024-09-10T19:22:02.224 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:22:02 smithi183 ceph-mon[29736]: 5.14 scrub ok 2024-09-10T19:22:02.224 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:22:02 smithi183 ceph-mon[29736]: 6.5 deep-scrub starts 2024-09-10T19:22:02.224 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:22:02 smithi183 ceph-mon[29736]: 6.5 deep-scrub ok 2024-09-10T19:22:02.224 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:22:02 smithi183 ceph-mon[29736]: from='client.? 172.21.15.183:0/1214394894' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-10T19:22:02.224 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:22:02 smithi183 ceph-mon[41562]: 4.9 scrub starts 2024-09-10T19:22:02.224 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:22:02 smithi183 ceph-mon[41562]: 4.9 scrub ok 2024-09-10T19:22:02.225 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:22:02 smithi183 ceph-mon[41562]: 5.14 scrub starts 2024-09-10T19:22:02.225 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:22:02 smithi183 ceph-mon[41562]: 5.14 scrub ok 2024-09-10T19:22:02.225 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:22:02 smithi183 ceph-mon[41562]: 6.5 deep-scrub starts 2024-09-10T19:22:02.225 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:22:02 smithi183 ceph-mon[41562]: 6.5 deep-scrub ok 2024-09-10T19:22:02.225 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:22:02 smithi183 ceph-mon[41562]: from='client.? 172.21.15.183:0/1214394894' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-10T19:22:02.275 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:22:02.306 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:22:02.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:22:02.307 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:22:02.408 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:22:02.443 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:22:02.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:22:02.444 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:22:02.468 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:22:02 smithi191 ceph-mon[32884]: 4.9 scrub starts 2024-09-10T19:22:02.468 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:22:02 smithi191 ceph-mon[32884]: 4.9 scrub ok 2024-09-10T19:22:02.468 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:22:02 smithi191 ceph-mon[32884]: 5.14 scrub starts 2024-09-10T19:22:02.468 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:22:02 smithi191 ceph-mon[32884]: 5.14 scrub ok 2024-09-10T19:22:02.468 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:22:02 smithi191 ceph-mon[32884]: 6.5 deep-scrub starts 2024-09-10T19:22:02.468 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:22:02 smithi191 ceph-mon[32884]: 6.5 deep-scrub ok 2024-09-10T19:22:02.468 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:22:02 smithi191 ceph-mon[32884]: from='client.? 172.21.15.183:0/1214394894' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-10T19:22:02.545 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:22:02.568 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:22:02.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:22:02.569 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:22:02.669 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:22:02.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:22:02.691 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:22:02.691 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:22:02.792 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:22:02.828 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:22:02.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:22:02.829 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:22:02.930 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:22:02.956 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:22:02.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:22:02.957 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:22:03.058 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:22:03.091 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:22:03.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:22:03.093 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:22:03.193 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:22:03.230 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:22:03.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:22:03.231 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:22:03.332 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:22:03.370 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:22:03.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:22:03.371 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:22:03.417 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:22:03 smithi183 ceph-mon[29736]: 2.1b scrub starts 2024-09-10T19:22:03.417 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:22:03 smithi183 ceph-mon[29736]: 2.1b scrub ok 2024-09-10T19:22:03.417 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:22:03 smithi183 ceph-mon[29736]: pgmap v534: 158 pgs: 1 active+clean+scrubbing+deep, 45 active+undersized, 33 active+undersized+degraded, 79 active+clean; 6.5 GiB data, 25 GiB used, 512 GiB / 536 GiB avail; 0 B/s wr, 1.56k op/s; 53455/319857 objects degraded (16.712%) 2024-09-10T19:22:03.417 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:22:03 smithi183 ceph-mon[29736]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T19:22:03.417 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:22:03 smithi183 ceph-mon[29736]: from='client.? 172.21.15.183:0/1214394894' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-10T19:22:03.417 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:22:03 smithi183 ceph-mon[29736]: osdmap e266: 8 total, 6 up, 6 in 2024-09-10T19:22:03.418 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:22:03 smithi183 ceph-mon[41562]: 2.1b scrub starts 2024-09-10T19:22:03.418 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:22:03 smithi183 ceph-mon[41562]: 2.1b scrub ok 2024-09-10T19:22:03.418 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:22:03 smithi183 ceph-mon[41562]: pgmap v534: 158 pgs: 1 active+clean+scrubbing+deep, 45 active+undersized, 33 active+undersized+degraded, 79 active+clean; 6.5 GiB data, 25 GiB used, 512 GiB / 536 GiB avail; 0 B/s wr, 1.56k op/s; 53455/319857 objects degraded (16.712%) 2024-09-10T19:22:03.418 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:22:03 smithi183 ceph-mon[41562]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T19:22:03.418 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:22:03 smithi183 ceph-mon[41562]: from='client.? 172.21.15.183:0/1214394894' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-10T19:22:03.418 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:22:03 smithi183 ceph-mon[41562]: osdmap e266: 8 total, 6 up, 6 in 2024-09-10T19:22:03.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:22:03 smithi191 ceph-mon[32884]: 2.1b scrub starts 2024-09-10T19:22:03.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:22:03 smithi191 ceph-mon[32884]: 2.1b scrub ok 2024-09-10T19:22:03.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:22:03 smithi191 ceph-mon[32884]: pgmap v534: 158 pgs: 1 active+clean+scrubbing+deep, 45 active+undersized, 33 active+undersized+degraded, 79 active+clean; 6.5 GiB data, 25 GiB used, 512 GiB / 536 GiB avail; 0 B/s wr, 1.56k op/s; 53455/319857 objects degraded (16.712%) 2024-09-10T19:22:03.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:22:03 smithi191 ceph-mon[32884]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T19:22:03.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:22:03 smithi191 ceph-mon[32884]: from='client.? 172.21.15.183:0/1214394894' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-10T19:22:03.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:22:03 smithi191 ceph-mon[32884]: osdmap e266: 8 total, 6 up, 6 in 2024-09-10T19:22:03.472 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:22:03.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:22:03.508 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:22:03.508 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:22:03.609 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:22:03.643 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:22:03.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:22:03.644 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:22:03.745 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:22:03.780 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:22:03.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:22:03.781 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:22:03.882 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:22:03.910 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:22:03.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:22:03.911 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:22:04.012 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:22:04.034 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:22:04.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:22:04.035 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:22:04.136 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:22:04.159 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:22:04.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:22:04.160 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:22:04.261 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:22:04.283 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:22:04.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:22:04.284 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:22:04.385 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:22:04.406 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:22:04.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:22:04.407 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:22:04.508 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:22:04.543 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:22:04.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:22:04.544 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:22:04.645 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:22:04.667 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:22:04.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:22:04.669 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:22:04.769 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:22:04.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:22:04.806 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:22:04.806 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:22:04.907 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:22:04.930 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:22:04.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:22:04.931 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:22:05.032 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:22:05.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:22:05.072 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:22:05.072 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:22:05.173 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:22:05.196 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:22:05.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:22:05.197 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:22:05.244 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:22:05 smithi183 ceph-mon[29736]: pgmap v536: 158 pgs: 1 active+clean+scrubbing+deep, 45 active+undersized, 33 active+undersized+degraded, 79 active+clean; 6.5 GiB data, 25 GiB used, 512 GiB / 536 GiB avail; 0 B/s wr, 1.37k op/s; 53455/319857 objects degraded (16.712%) 2024-09-10T19:22:05.244 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:22:05 smithi183 ceph-mon[29736]: Health check update: Degraded data redundancy: 53455/319857 objects degraded (16.712%), 33 pgs degraded (PG_DEGRADED) 2024-09-10T19:22:05.244 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:22:05 smithi183 ceph-mon[41562]: pgmap v536: 158 pgs: 1 active+clean+scrubbing+deep, 45 active+undersized, 33 active+undersized+degraded, 79 active+clean; 6.5 GiB data, 25 GiB used, 512 GiB / 536 GiB avail; 0 B/s wr, 1.37k op/s; 53455/319857 objects degraded (16.712%) 2024-09-10T19:22:05.244 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:22:05 smithi183 ceph-mon[41562]: Health check update: Degraded data redundancy: 53455/319857 objects degraded (16.712%), 33 pgs degraded (PG_DEGRADED) 2024-09-10T19:22:05.298 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:22:05.321 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:22:05.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:22:05.322 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:22:05.364 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:22:05 smithi191 ceph-mon[32884]: pgmap v536: 158 pgs: 1 active+clean+scrubbing+deep, 45 active+undersized, 33 active+undersized+degraded, 79 active+clean; 6.5 GiB data, 25 GiB used, 512 GiB / 536 GiB avail; 0 B/s wr, 1.37k op/s; 53455/319857 objects degraded (16.712%) 2024-09-10T19:22:05.364 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:22:05 smithi191 ceph-mon[32884]: Health check update: Degraded data redundancy: 53455/319857 objects degraded (16.712%), 33 pgs degraded (PG_DEGRADED) 2024-09-10T19:22:05.407 INFO:teuthology.orchestra.run.smithi191.stdout:ERROR: (22) Invalid argument 2024-09-10T19:22:05.407 INFO:teuthology.orchestra.run.smithi191.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-10T19:22:05.423 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:22:05.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:22:05.461 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:22:05.461 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:22:05.562 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:22:05.586 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:22:05.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:22:05.587 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:22:05.687 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:22:05.713 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:22:05.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:22:05.714 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:22:05.757 INFO:teuthology.orchestra.run.smithi183.stdout:osd.2: {} 2024-09-10T19:22:05.758 INFO:teuthology.orchestra.run.smithi183.stderr:Error ENXIO: problem getting command descriptions from osd.0 2024-09-10T19:22:05.758 INFO:teuthology.orchestra.run.smithi183.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-09-10T19:22:05.758 INFO:teuthology.orchestra.run.smithi183.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-10T19:22:05.773 INFO:teuthology.orchestra.run.smithi183.stdout:osd.3: {} 2024-09-10T19:22:05.774 INFO:teuthology.orchestra.run.smithi183.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-10T19:22:05.789 INFO:teuthology.orchestra.run.smithi183.stdout:osd.4: {} 2024-09-10T19:22:05.789 INFO:teuthology.orchestra.run.smithi183.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-10T19:22:05.814 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:22:05.836 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:22:05.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:22:05.837 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:22:05.938 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:22:05.959 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:22:05.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:22:05.960 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:22:06.061 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:22:06.082 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:22:06.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:22:06.083 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:22:06.184 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:22:06.205 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:22:06.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:22:06.206 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:22:06.237 INFO:teuthology.orchestra.run.smithi183.stdout:osd.5: {} 2024-09-10T19:22:06.237 INFO:teuthology.orchestra.run.smithi183.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-10T19:22:06.257 INFO:teuthology.orchestra.run.smithi183.stdout:osd.6: {} 2024-09-10T19:22:06.257 INFO:teuthology.orchestra.run.smithi183.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-10T19:22:06.278 INFO:teuthology.orchestra.run.smithi183.stdout:osd.7: {} 2024-09-10T19:22:06.278 INFO:teuthology.orchestra.run.smithi183.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-10T19:22:06.656 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:22:06.657 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-10T19:22:06.658 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:22:06 smithi191 ceph-mon[32884]: from='client.? 172.21.15.183:0/3533750187' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T19:22:06.658 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:22:06 smithi183 ceph-mon[29736]: from='client.? 172.21.15.183:0/3533750187' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T19:22:06.658 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:22:06 smithi183 ceph-mon[41562]: from='client.? 172.21.15.183:0/3533750187' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T19:22:06.694 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:22:06.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:22:06.695 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:22:06.796 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:22:06.827 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:22:06.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:22:06.828 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:22:06.929 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:22:06.953 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:22:06.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:22:06.954 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:22:06.983 DEBUG:teuthology.orchestra.run.smithi191:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph daemon osd.5 dump_historic_ops 2024-09-10T19:22:07.054 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:22:07.076 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:22:07.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:22:07.077 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:22:07.178 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:22:07.184 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-10T19:22:07.185 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-10T19:22:07.203 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:22:07.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:22:07.204 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:22:07.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:22:07 smithi191 ceph-mon[32884]: pgmap v537: 158 pgs: 1 active+clean+scrubbing+deep, 45 active+undersized, 33 active+undersized+degraded, 79 active+clean; 6.2 GiB data, 24 GiB used, 512 GiB / 536 GiB avail; 1023 B/s rd, 0 B/s wr, 1.12k op/s; 50894/304446 objects degraded (16.717%) 2024-09-10T19:22:07.292 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:22:07 smithi183 ceph-mon[29736]: pgmap v537: 158 pgs: 1 active+clean+scrubbing+deep, 45 active+undersized, 33 active+undersized+degraded, 79 active+clean; 6.2 GiB data, 24 GiB used, 512 GiB / 536 GiB avail; 1023 B/s rd, 0 B/s wr, 1.12k op/s; 50894/304446 objects degraded (16.717%) 2024-09-10T19:22:07.292 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:22:07 smithi183 ceph-mon[41562]: pgmap v537: 158 pgs: 1 active+clean+scrubbing+deep, 45 active+undersized, 33 active+undersized+degraded, 79 active+clean; 6.2 GiB data, 24 GiB used, 512 GiB / 536 GiB avail; 1023 B/s rd, 0 B/s wr, 1.12k op/s; 50894/304446 objects degraded (16.717%) 2024-09-10T19:22:07.304 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:22:07.327 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:22:07.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:22:07.328 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:22:07.428 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:22:07.449 INFO:teuthology.orchestra.run.smithi191.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.b/config 2024-09-10T19:22:07.450 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:22:07.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:22:07.451 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:22:07.552 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:22:07.582 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:22:07.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:22:07.583 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:22:07.684 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:22:07.716 INFO:journalctl@ceph.mgr.x.smithi191.stdout:Sep 10 19:22:07 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-x[34967]: ::ffff:172.21.15.191 - - [10/Sep/2024:19:22:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T19:22:07.717 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:22:07.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:22:07.718 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:22:07.745 INFO:tasks.thrashosds.thrasher:No PGs found for osd.1 2024-09-10T19:22:07.819 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:22:07.855 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:22:07.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:22:07.856 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:22:07.865 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:22:07.957 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:22:08.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:22:08.001 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:22:08.001 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:22:08.102 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:22:08.133 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:22:08.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:22:08.135 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:22:08.236 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:22:08.269 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:22:08.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:22:08.270 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:22:08.371 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:22:08.394 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:22:08.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:22:08.395 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:22:08.497 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:22:08.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:22:08.531 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:22:08.531 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:22:08.631 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:22:08.664 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:22:08.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:22:08.665 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:22:08.766 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:22:08.799 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:22:08.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:22:08.800 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:22:08.901 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:22:08.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:22:08.946 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:22:08.946 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:22:09.047 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:22:09.070 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:22:09.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:22:09.072 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:22:09.072 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:22:09 smithi191 ceph-mon[32884]: pgmap v538: 158 pgs: 1 active+clean+scrubbing+deep, 45 active+undersized, 33 active+undersized+degraded, 79 active+clean; 6.2 GiB data, 24 GiB used, 512 GiB / 536 GiB avail; 1023 B/s rd, 0 B/s wr, 1.12k op/s; 50894/304446 objects degraded (16.717%) 2024-09-10T19:22:09.173 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:22:09.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:22:09 smithi183 ceph-mon[29736]: pgmap v538: 158 pgs: 1 active+clean+scrubbing+deep, 45 active+undersized, 33 active+undersized+degraded, 79 active+clean; 6.2 GiB data, 24 GiB used, 512 GiB / 536 GiB avail; 1023 B/s rd, 0 B/s wr, 1.12k op/s; 50894/304446 objects degraded (16.717%) 2024-09-10T19:22:09.181 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:22:09 smithi183 ceph-mon[41562]: pgmap v538: 158 pgs: 1 active+clean+scrubbing+deep, 45 active+undersized, 33 active+undersized+degraded, 79 active+clean; 6.2 GiB data, 24 GiB used, 512 GiB / 536 GiB avail; 1023 B/s rd, 0 B/s wr, 1.12k op/s; 50894/304446 objects degraded (16.717%) 2024-09-10T19:22:09.186 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-10T19:22:09.194 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:22:09.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:22:09.196 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:22:09.296 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:22:09.320 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:22:09.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:22:09.321 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:22:09.422 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:22:09.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:22:09.446 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:22:09.446 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:22:09.547 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:22:09.583 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:22:09.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:22:09.584 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:22:09.685 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:22:09.715 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:22:09.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:22:09.716 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:22:09.817 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:22:09.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:22:09.844 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:22:09.844 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:22:09.945 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:22:09.966 INFO:journalctl@ceph.iscsi.iscsi.a.smithi191.stdout:Sep 10 19:22:09 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-iscsi-iscsi-a[61816]: debug there is no tcmu-runner data available 2024-09-10T19:22:09.981 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:22:09.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:22:09.982 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:22:10.083 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:22:10.116 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:22:10.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:22:10.117 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:22:10.218 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:22:10.239 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:22:10.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:22:10.240 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:22:10.340 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:22:10.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:22:10.365 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:22:10.365 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:22:10.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:22:10 smithi183 ceph-mon[29736]: Health check update: Degraded data redundancy: 50894/304446 objects degraded (16.717%), 33 pgs degraded (PG_DEGRADED) 2024-09-10T19:22:10.431 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:22:10 smithi183 ceph-mon[41562]: Health check update: Degraded data redundancy: 50894/304446 objects degraded (16.717%), 33 pgs degraded (PG_DEGRADED) 2024-09-10T19:22:10.469 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:22:10 smithi191 ceph-mon[32884]: Health check update: Degraded data redundancy: 50894/304446 objects degraded (16.717%), 33 pgs degraded (PG_DEGRADED) 2024-09-10T19:22:10.469 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:22:10.492 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:22:10.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:22:10.493 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:22:10.594 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:22:10.615 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:22:10.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:22:10.616 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:22:10.717 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:22:10.745 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:22:10.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:22:10.746 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:22:10.834 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:22:10.847 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:22:10.871 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:22:10.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:22:10.872 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:22:10.973 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:22:10.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:22:10.995 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:22:10.995 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:22:11.061 INFO:teuthology.orchestra.run.smithi183.stderr:nodeep-scrub is set 2024-09-10T19:22:11.095 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:22:11.119 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:22:11.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:22:11.120 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:22:11.221 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:22:11.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:22:11.243 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:22:11.243 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:22:11.316 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:22:11 smithi183 ceph-mon[29736]: from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:22:11.317 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:22:11 smithi183 ceph-mon[29736]: pgmap v539: 158 pgs: 1 active+clean+scrubbing+deep, 45 active+undersized, 33 active+undersized+degraded, 79 active+clean; 6.2 GiB data, 24 GiB used, 512 GiB / 536 GiB avail; 1023 B/s rd, 0 B/s wr, 1.12k op/s; 50894/304446 objects degraded (16.717%) 2024-09-10T19:22:11.317 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:22:11 smithi183 ceph-mon[29736]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T19:22:11.317 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:22:11 smithi183 ceph-mon[29736]: from='client.? 172.21.15.183:0/3150067394' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-10T19:22:11.317 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:22:11 smithi183 ceph-mon[29736]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-10T19:22:11.317 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:22:11 smithi183 ceph-mon[41562]: from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:22:11.317 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:22:11 smithi183 ceph-mon[41562]: pgmap v539: 158 pgs: 1 active+clean+scrubbing+deep, 45 active+undersized, 33 active+undersized+degraded, 79 active+clean; 6.2 GiB data, 24 GiB used, 512 GiB / 536 GiB avail; 1023 B/s rd, 0 B/s wr, 1.12k op/s; 50894/304446 objects degraded (16.717%) 2024-09-10T19:22:11.317 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:22:11 smithi183 ceph-mon[41562]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T19:22:11.317 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:22:11 smithi183 ceph-mon[41562]: from='client.? 172.21.15.183:0/3150067394' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-10T19:22:11.317 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:22:11 smithi183 ceph-mon[41562]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-10T19:22:11.318 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Sep 10 19:22:11 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-y[29989]: ::ffff:172.21.15.191 - - [10/Sep/2024:19:22:11] "GET /metrics HTTP/1.1" 200 209492 "" "Prometheus/2.43.0" 2024-09-10T19:22:11.344 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:22:11.385 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:22:11.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:22:11.386 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:22:11.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:22:11 smithi191 ceph-mon[32884]: from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:22:11.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:22:11 smithi191 ceph-mon[32884]: pgmap v539: 158 pgs: 1 active+clean+scrubbing+deep, 45 active+undersized, 33 active+undersized+degraded, 79 active+clean; 6.2 GiB data, 24 GiB used, 512 GiB / 536 GiB avail; 1023 B/s rd, 0 B/s wr, 1.12k op/s; 50894/304446 objects degraded (16.717%) 2024-09-10T19:22:11.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:22:11 smithi191 ceph-mon[32884]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T19:22:11.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:22:11 smithi191 ceph-mon[32884]: from='client.? 172.21.15.183:0/3150067394' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-10T19:22:11.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:22:11 smithi191 ceph-mon[32884]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-10T19:22:11.488 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:22:11.527 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:22:11.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:22:11.528 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:22:11.629 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:22:11.664 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:22:11.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:22:11.665 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:22:11.721 INFO:teuthology.orchestra.run.smithi191.stdout:{ 2024-09-10T19:22:11.721 INFO:teuthology.orchestra.run.smithi191.stdout: "size": 20, 2024-09-10T19:22:11.721 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 600, 2024-09-10T19:22:11.722 INFO:teuthology.orchestra.run.smithi191.stdout: "ops": [ 2024-09-10T19:22:11.722 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:11.722 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.14781.0:205540 7.b 7:d286f7dc:::benchmark_data_smithi183_84160_object65159:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e126)", 2024-09-10T19:22:11.722 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-10T19:14:45.772065+0000", 2024-09-10T19:22:11.722 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 444.46503805499998, 2024-09-10T19:22:11.722 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.98751803299999996, 2024-09-10T19:22:11.723 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-10T19:22:11.723 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T19:22:11.723 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-10T19:22:11.723 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.14781", 2024-09-10T19:22:11.723 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.183:0/2025776420", 2024-09-10T19:22:11.723 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 205540 2024-09-10T19:22:11.724 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:11.724 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-10T19:22:11.724 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:11.724 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-10T19:22:11.724 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:45.772065+0000", 2024-09-10T19:22:11.724 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:22:11.724 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:11.724 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:11.724 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-10T19:22:11.724 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:45.772065+0000", 2024-09-10T19:22:11.724 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:22:11.724 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:11.725 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:11.725 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-10T19:22:11.725 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:45.772064+0000", 2024-09-10T19:22:11.725 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.999999 2024-09-10T19:22:11.725 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:11.725 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:11.725 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-10T19:22:11.725 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:45.772071+0000", 2024-09-10T19:22:11.725 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.503e-06 2024-09-10T19:22:11.725 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:11.725 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:11.725 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-10T19:22:11.725 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:45.772082+0000", 2024-09-10T19:22:11.725 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.0294e-05 2024-09-10T19:22:11.726 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:11.726 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:11.726 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-10T19:22:11.726 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:45.772087+0000", 2024-09-10T19:22:11.726 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.643e-06 2024-09-10T19:22:11.726 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:11.726 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:11.726 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:22:11.726 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:45.772132+0000", 2024-09-10T19:22:11.726 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.4378000000000001e-05 2024-09-10T19:22:11.726 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:11.727 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:11.727 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for peered", 2024-09-10T19:22:11.727 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:45.772140+0000", 2024-09-10T19:22:11.727 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.7520000000000002e-06 2024-09-10T19:22:11.727 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:11.727 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:11.727 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:22:11.727 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:46.752096+0000", 2024-09-10T19:22:11.727 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.97995554699999998 2024-09-10T19:22:11.727 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:11.727 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:11.728 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-10T19:22:11.728 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:46.752214+0000", 2024-09-10T19:22:11.728 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000118507 2024-09-10T19:22:11.728 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:11.728 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:11.728 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 2,6", 2024-09-10T19:22:11.728 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:46.752288+0000", 2024-09-10T19:22:11.728 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.339e-05 2024-09-10T19:22:11.728 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:11.729 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:11.729 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-09-10T19:22:11.729 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:46.757094+0000", 2024-09-10T19:22:11.729 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0048063569999999998 2024-09-10T19:22:11.729 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:11.729 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:11.729 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:22:11.729 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:46.759427+0000", 2024-09-10T19:22:11.729 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0023326050000000002 2024-09-10T19:22:11.730 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:11.730 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:11.730 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:22:11.730 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:46.759503+0000", 2024-09-10T19:22:11.730 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.5694999999999999e-05 2024-09-10T19:22:11.730 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:11.730 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:11.730 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-09-10T19:22:11.730 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:46.759549+0000", 2024-09-10T19:22:11.730 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.6634000000000001e-05 2024-09-10T19:22:11.730 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:11.730 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:11.730 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-10T19:22:11.731 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:46.759583+0000", 2024-09-10T19:22:11.731 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.3878000000000003e-05 2024-09-10T19:22:11.731 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:22:11.731 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-10T19:22:11.731 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:22:11.731 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:11.731 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:11.731 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.14781.0:205549 7.b 7:d0626604:::benchmark_data_smithi183_84160_object65168:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e126)", 2024-09-10T19:22:11.731 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-10T19:14:45.780874+0000", 2024-09-10T19:22:11.731 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 444.456228743, 2024-09-10T19:22:11.731 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.97887205499999996, 2024-09-10T19:22:11.731 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-10T19:22:11.731 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T19:22:11.731 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-10T19:22:11.732 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.14781", 2024-09-10T19:22:11.732 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.183:0/2025776420", 2024-09-10T19:22:11.732 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 205549 2024-09-10T19:22:11.732 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:11.732 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-10T19:22:11.732 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:11.732 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-10T19:22:11.732 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:45.780874+0000", 2024-09-10T19:22:11.732 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:22:11.732 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:11.732 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:11.732 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-10T19:22:11.732 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:45.780874+0000", 2024-09-10T19:22:11.732 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:22:11.733 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:11.733 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:11.733 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-10T19:22:11.733 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:45.780873+0000", 2024-09-10T19:22:11.733 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999986 2024-09-10T19:22:11.733 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:11.733 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:11.733 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-10T19:22:11.733 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:45.780879+0000", 2024-09-10T19:22:11.733 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.083e-06 2024-09-10T19:22:11.733 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:11.733 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:11.733 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-10T19:22:11.734 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:45.780892+0000", 2024-09-10T19:22:11.734 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.2496e-05 2024-09-10T19:22:11.734 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:11.734 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:11.734 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-10T19:22:11.734 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:45.780896+0000", 2024-09-10T19:22:11.734 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.4970000000000003e-06 2024-09-10T19:22:11.734 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:11.734 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:11.734 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:22:11.734 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:45.780945+0000", 2024-09-10T19:22:11.734 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.8618000000000001e-05 2024-09-10T19:22:11.734 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:11.734 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:11.735 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for peered", 2024-09-10T19:22:11.735 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:45.780954+0000", 2024-09-10T19:22:11.735 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.9220000000000003e-06 2024-09-10T19:22:11.735 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:11.735 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:11.735 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:22:11.735 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:46.754818+0000", 2024-09-10T19:22:11.735 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.97386392899999996 2024-09-10T19:22:11.735 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:11.735 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:11.735 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-10T19:22:11.735 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:46.754913+0000", 2024-09-10T19:22:11.735 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.5884000000000006e-05 2024-09-10T19:22:11.735 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:11.736 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:11.736 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 2,6", 2024-09-10T19:22:11.736 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:46.755028+0000", 2024-09-10T19:22:11.736 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00011414 2024-09-10T19:22:11.736 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:11.736 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:11.736 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:22:11.736 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:46.759634+0000", 2024-09-10T19:22:11.736 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0046064060000000004 2024-09-10T19:22:11.736 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:11.736 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:11.736 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-09-10T19:22:11.736 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:46.759643+0000", 2024-09-10T19:22:11.736 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.2660000000000007e-06 2024-09-10T19:22:11.737 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:11.737 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:11.737 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:22:11.737 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:46.759693+0000", 2024-09-10T19:22:11.737 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.9382000000000003e-05 2024-09-10T19:22:11.737 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:11.737 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:11.737 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-09-10T19:22:11.737 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:46.759725+0000", 2024-09-10T19:22:11.737 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.2830999999999998e-05 2024-09-10T19:22:11.738 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:11.738 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:11.738 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-10T19:22:11.738 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:46.759746+0000", 2024-09-10T19:22:11.738 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.0885000000000001e-05 2024-09-10T19:22:11.738 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:22:11.738 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-10T19:22:11.738 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:22:11.738 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:11.738 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:11.738 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.14781.0:205550 7.b 7:d6a1ae8a:::benchmark_data_smithi183_84160_object65169:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e126)", 2024-09-10T19:22:11.738 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-10T19:14:45.780904+0000", 2024-09-10T19:22:11.738 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 444.45619924499999, 2024-09-10T19:22:11.739 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.98770159000000002, 2024-09-10T19:22:11.739 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-10T19:22:11.739 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T19:22:11.739 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-10T19:22:11.739 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.14781", 2024-09-10T19:22:11.739 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.183:0/2025776420", 2024-09-10T19:22:11.739 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 205550 2024-09-10T19:22:11.739 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:11.739 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-10T19:22:11.739 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:11.739 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-10T19:22:11.739 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:45.780904+0000", 2024-09-10T19:22:11.739 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:22:11.740 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:11.740 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:11.740 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-10T19:22:11.740 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:45.780904+0000", 2024-09-10T19:22:11.740 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:22:11.740 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:11.740 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:11.740 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-10T19:22:11.740 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:45.780903+0000", 2024-09-10T19:22:11.740 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999995 2024-09-10T19:22:11.740 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:11.741 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:11.741 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-10T19:22:11.741 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:45.780906+0000", 2024-09-10T19:22:11.741 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.599e-06 2024-09-10T19:22:11.741 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:11.741 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:11.741 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-10T19:22:11.741 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:45.780911+0000", 2024-09-10T19:22:11.741 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.434e-06 2024-09-10T19:22:11.741 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:11.741 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:11.741 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-10T19:22:11.741 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:45.780915+0000", 2024-09-10T19:22:11.742 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.8070000000000001e-06 2024-09-10T19:22:11.742 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:11.742 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:11.742 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:22:11.742 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:45.780977+0000", 2024-09-10T19:22:11.742 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.2244999999999998e-05 2024-09-10T19:22:11.742 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:11.742 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:11.742 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for peered", 2024-09-10T19:22:11.742 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:45.780982+0000", 2024-09-10T19:22:11.742 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.6609999999999999e-06 2024-09-10T19:22:11.742 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:11.742 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:11.743 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:22:11.743 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:46.757132+0000", 2024-09-10T19:22:11.743 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.97615030000000003 2024-09-10T19:22:11.743 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:11.743 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:11.743 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-10T19:22:11.743 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:46.757251+0000", 2024-09-10T19:22:11.744 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000118434 2024-09-10T19:22:11.744 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:11.744 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:11.744 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 2,6", 2024-09-10T19:22:11.744 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:46.757358+0000", 2024-09-10T19:22:11.744 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000107594 2024-09-10T19:22:11.744 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:11.744 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:11.744 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-09-10T19:22:11.744 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:46.762125+0000", 2024-09-10T19:22:11.744 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0047661559999999997 2024-09-10T19:22:11.744 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:11.744 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:11.745 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:22:11.745 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:46.762162+0000", 2024-09-10T19:22:11.745 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.6940999999999997e-05 2024-09-10T19:22:11.745 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:11.745 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:11.745 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:22:11.745 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:46.768527+0000", 2024-09-10T19:22:11.745 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0063658769999999998 2024-09-10T19:22:11.745 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:11.745 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:11.745 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-09-10T19:22:11.745 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:46.768571+0000", 2024-09-10T19:22:11.745 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.3745000000000002e-05 2024-09-10T19:22:11.746 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:11.746 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:11.746 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-10T19:22:11.746 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:46.768605+0000", 2024-09-10T19:22:11.746 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.4190000000000003e-05 2024-09-10T19:22:11.746 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:22:11.746 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-10T19:22:11.746 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:22:11.747 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:11.747 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:11.747 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.14781.0:217417 7.0 7:0e3413bc:::benchmark_data_smithi183_84160_object77036:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e131)", 2024-09-10T19:22:11.747 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-10T19:14:56.880641+0000", 2024-09-10T19:22:11.747 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 433.35646228399997, 2024-09-10T19:22:11.747 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.991396897, 2024-09-10T19:22:11.748 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-10T19:22:11.748 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T19:22:11.748 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-10T19:22:11.748 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.14781", 2024-09-10T19:22:11.748 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.183:0/2025776420", 2024-09-10T19:22:11.748 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 217417 2024-09-10T19:22:11.748 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:11.748 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-10T19:22:11.748 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:11.748 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-10T19:22:11.748 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:56.880641+0000", 2024-09-10T19:22:11.749 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:22:11.749 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:11.749 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:11.749 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-10T19:22:11.749 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:56.880641+0000", 2024-09-10T19:22:11.749 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:22:11.749 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:11.749 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:11.749 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-10T19:22:11.749 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:56.880639+0000", 2024-09-10T19:22:11.749 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999981 2024-09-10T19:22:11.749 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:11.749 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:11.749 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-10T19:22:11.750 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:56.880647+0000", 2024-09-10T19:22:11.750 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.9810000000000003e-06 2024-09-10T19:22:11.750 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:11.750 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:11.750 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-10T19:22:11.750 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:56.880664+0000", 2024-09-10T19:22:11.750 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.7589999999999999e-05 2024-09-10T19:22:11.750 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:11.750 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:11.750 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-10T19:22:11.750 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:56.880671+0000", 2024-09-10T19:22:11.750 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.9779999999999999e-06 2024-09-10T19:22:11.750 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:11.751 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:11.751 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:22:11.751 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:56.880736+0000", 2024-09-10T19:22:11.751 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.4858999999999997e-05 2024-09-10T19:22:11.751 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:11.751 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:11.751 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for peered", 2024-09-10T19:22:11.751 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:56.880745+0000", 2024-09-10T19:22:11.751 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.8340000000000005e-06 2024-09-10T19:22:11.751 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:11.751 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:11.751 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:22:11.751 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:57.866422+0000", 2024-09-10T19:22:11.751 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.98567665000000004 2024-09-10T19:22:11.752 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:11.752 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:11.752 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-10T19:22:11.752 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:57.866508+0000", 2024-09-10T19:22:11.752 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.5717000000000006e-05 2024-09-10T19:22:11.752 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:11.752 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:11.752 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 2,3", 2024-09-10T19:22:11.752 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:57.866585+0000", 2024-09-10T19:22:11.752 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.7676999999999995e-05 2024-09-10T19:22:11.752 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:11.752 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:11.752 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-09-10T19:22:11.753 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:57.871908+0000", 2024-09-10T19:22:11.753 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0053224120000000003 2024-09-10T19:22:11.753 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:11.753 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:11.753 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:22:11.753 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:57.871930+0000", 2024-09-10T19:22:11.753 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.2722e-05 2024-09-10T19:22:11.753 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:11.754 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:11.754 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:22:11.754 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:57.871985+0000", 2024-09-10T19:22:11.754 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.4246000000000002e-05 2024-09-10T19:22:11.754 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:11.754 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:11.754 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-09-10T19:22:11.754 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:57.872017+0000", 2024-09-10T19:22:11.754 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.2441000000000003e-05 2024-09-10T19:22:11.754 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:11.754 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:11.754 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-10T19:22:11.754 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:57.872038+0000", 2024-09-10T19:22:11.755 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.0639999999999999e-05 2024-09-10T19:22:11.755 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:22:11.755 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-10T19:22:11.755 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:22:11.755 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:11.755 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:11.755 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.14781.0:217427 7.0 7:0e3d17be:::benchmark_data_smithi183_84160_object77046:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e131)", 2024-09-10T19:22:11.755 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-10T19:14:56.890012+0000", 2024-09-10T19:22:11.755 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 433.34709146799997, 2024-09-10T19:22:11.755 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.98213677700000002, 2024-09-10T19:22:11.755 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-10T19:22:11.755 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T19:22:11.755 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-10T19:22:11.755 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.14781", 2024-09-10T19:22:11.756 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.183:0/2025776420", 2024-09-10T19:22:11.756 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 217427 2024-09-10T19:22:11.756 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:11.756 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-10T19:22:11.756 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:11.756 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-10T19:22:11.756 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:56.890012+0000", 2024-09-10T19:22:11.756 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:22:11.756 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:11.756 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:11.756 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-10T19:22:11.756 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:56.890012+0000", 2024-09-10T19:22:11.756 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:22:11.757 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:11.757 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:11.757 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-10T19:22:11.757 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:56.890010+0000", 2024-09-10T19:22:11.757 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999986 2024-09-10T19:22:11.757 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:11.757 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:11.757 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-10T19:22:11.757 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:56.890017+0000", 2024-09-10T19:22:11.757 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.5259999999999999e-06 2024-09-10T19:22:11.757 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:11.757 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:11.757 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-10T19:22:11.757 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:56.890028+0000", 2024-09-10T19:22:11.758 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.1259e-05 2024-09-10T19:22:11.758 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:11.758 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:11.758 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-10T19:22:11.758 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:56.890032+0000", 2024-09-10T19:22:11.758 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.3769999999999998e-06 2024-09-10T19:22:11.758 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:11.758 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:11.758 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:22:11.758 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:56.890066+0000", 2024-09-10T19:22:11.758 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.3467999999999999e-05 2024-09-10T19:22:11.758 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:11.758 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:11.759 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for peered", 2024-09-10T19:22:11.759 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:56.890072+0000", 2024-09-10T19:22:11.759 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.5400000000000001e-06 2024-09-10T19:22:11.759 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:11.759 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:11.759 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:22:11.759 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:57.868641+0000", 2024-09-10T19:22:11.759 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.97856873 2024-09-10T19:22:11.759 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:11.759 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:11.759 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-10T19:22:11.759 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:57.868700+0000", 2024-09-10T19:22:11.759 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.9086999999999999e-05 2024-09-10T19:22:11.759 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:11.759 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:11.760 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 2,3", 2024-09-10T19:22:11.760 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:57.868767+0000", 2024-09-10T19:22:11.760 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.6742000000000006e-05 2024-09-10T19:22:11.760 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:11.760 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:11.760 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:22:11.760 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:57.872070+0000", 2024-09-10T19:22:11.760 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0033030450000000001 2024-09-10T19:22:11.760 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:11.760 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:11.760 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-09-10T19:22:11.760 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:57.872076+0000", 2024-09-10T19:22:11.761 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.1510000000000001e-06 2024-09-10T19:22:11.761 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:11.761 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:11.761 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:22:11.761 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:57.872111+0000", 2024-09-10T19:22:11.761 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.4857999999999997e-05 2024-09-10T19:22:11.761 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:11.761 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:11.761 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-09-10T19:22:11.761 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:57.872132+0000", 2024-09-10T19:22:11.761 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.0798999999999999e-05 2024-09-10T19:22:11.761 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:11.761 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:11.761 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-10T19:22:11.762 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:14:57.872148+0000", 2024-09-10T19:22:11.762 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.6449000000000001e-05 2024-09-10T19:22:11.762 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:22:11.762 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-10T19:22:11.762 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:22:11.762 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:11.762 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:11.762 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.15051.0:93085 8.14 8:2abf7048:::benchmark_data_smithi183_114924_object93084:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e177)", 2024-09-10T19:22:11.762 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-10T19:17:26.014960+0000", 2024-09-10T19:22:11.762 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 284.22214318699997, 2024-09-10T19:22:11.762 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.99687631899999996, 2024-09-10T19:22:11.762 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-10T19:22:11.762 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "reached pg", 2024-09-10T19:22:11.762 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-10T19:22:11.763 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.15051", 2024-09-10T19:22:11.763 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.183:0/3632028672", 2024-09-10T19:22:11.763 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 93085 2024-09-10T19:22:11.763 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:11.763 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-10T19:22:11.763 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:11.763 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-10T19:22:11.763 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:26.014960+0000", 2024-09-10T19:22:11.763 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:22:11.763 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:11.763 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:11.763 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-10T19:22:11.763 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:26.014960+0000", 2024-09-10T19:22:11.763 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:22:11.764 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:11.764 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:11.764 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-10T19:22:11.764 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:26.014957+0000", 2024-09-10T19:22:11.764 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999976 2024-09-10T19:22:11.764 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:11.764 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:11.764 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-10T19:22:11.764 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:26.015041+0000", 2024-09-10T19:22:11.764 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.3499999999999997e-05 2024-09-10T19:22:11.764 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:11.764 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:11.765 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-10T19:22:11.765 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:26.015055+0000", 2024-09-10T19:22:11.765 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.4613999999999999e-05 2024-09-10T19:22:11.765 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:11.765 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:11.765 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-10T19:22:11.765 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:26.015064+0000", 2024-09-10T19:22:11.765 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.6070000000000008e-06 2024-09-10T19:22:11.765 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:11.765 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:11.765 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:22:11.765 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:26.163910+0000", 2024-09-10T19:22:11.765 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.14884554799999999 2024-09-10T19:22:11.766 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:11.766 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:11.766 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for peered", 2024-09-10T19:22:11.766 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:26.163919+0000", 2024-09-10T19:22:11.766 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.1679999999999993e-06 2024-09-10T19:22:11.766 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:11.766 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:11.766 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:22:11.766 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:27.006838+0000", 2024-09-10T19:22:11.766 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.84291875999999999 2024-09-10T19:22:11.766 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:11.766 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:11.766 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for degraded object", 2024-09-10T19:22:11.767 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:27.007794+0000", 2024-09-10T19:22:11.767 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00095682999999999999 2024-09-10T19:22:11.767 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:11.767 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:11.767 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:22:11.767 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:27.011768+0000", 2024-09-10T19:22:11.767 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0039740390000000004 2024-09-10T19:22:11.767 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:11.767 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:11.767 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-10T19:22:11.767 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:27.011836+0000", 2024-09-10T19:22:11.767 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.7748000000000003e-05 2024-09-10T19:22:11.767 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:22:11.767 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-10T19:22:11.768 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:22:11.768 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:11.768 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:11.768 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.15051.0:93088 8.4 8:24544414:::benchmark_data_smithi183_114924_object93087:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e177)", 2024-09-10T19:22:11.768 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-10T19:17:26.015072+0000", 2024-09-10T19:22:11.768 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 284.22203152399999, 2024-09-10T19:22:11.768 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.97135848400000002, 2024-09-10T19:22:11.768 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-10T19:22:11.768 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "reached pg", 2024-09-10T19:22:11.768 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-10T19:22:11.768 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.15051", 2024-09-10T19:22:11.768 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.183:0/3632028672", 2024-09-10T19:22:11.768 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 93088 2024-09-10T19:22:11.768 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:11.769 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-10T19:22:11.769 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:11.769 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-10T19:22:11.769 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:26.015072+0000", 2024-09-10T19:22:11.769 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:22:11.769 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:11.769 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:11.769 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-10T19:22:11.769 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:26.015072+0000", 2024-09-10T19:22:11.769 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:22:11.769 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:11.769 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:11.769 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-10T19:22:11.769 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:26.015071+0000", 2024-09-10T19:22:11.770 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999995 2024-09-10T19:22:11.770 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:11.770 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:11.770 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-10T19:22:11.770 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:26.015104+0000", 2024-09-10T19:22:11.770 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.3473999999999998e-05 2024-09-10T19:22:11.770 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:11.770 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:11.770 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-10T19:22:11.770 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:26.015112+0000", 2024-09-10T19:22:11.770 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.8259999999999993e-06 2024-09-10T19:22:11.770 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:11.770 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:11.771 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-10T19:22:11.771 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:26.015117+0000", 2024-09-10T19:22:11.771 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.4630000000000003e-06 2024-09-10T19:22:11.771 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:11.771 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:11.771 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:22:11.771 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:26.163978+0000", 2024-09-10T19:22:11.771 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.14886155000000001 2024-09-10T19:22:11.771 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:11.771 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:11.771 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for peered", 2024-09-10T19:22:11.772 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:26.163985+0000", 2024-09-10T19:22:11.772 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.4559999999999997e-06 2024-09-10T19:22:11.772 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:11.772 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:11.772 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:22:11.772 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:26.976983+0000", 2024-09-10T19:22:11.772 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.812998357 2024-09-10T19:22:11.772 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:11.772 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:11.772 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for degraded object", 2024-09-10T19:22:11.772 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:26.978158+0000", 2024-09-10T19:22:11.772 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0011745919999999999 2024-09-10T19:22:11.773 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:11.773 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:11.773 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:22:11.773 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:26.986359+0000", 2024-09-10T19:22:11.773 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0082010720000000002 2024-09-10T19:22:11.773 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:11.773 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:11.773 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-10T19:22:11.774 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:26.986430+0000", 2024-09-10T19:22:11.774 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.1358000000000004e-05 2024-09-10T19:22:11.774 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:22:11.774 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-10T19:22:11.774 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:22:11.774 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:11.774 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:11.774 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.15051.0:93184 8.13 8.bc5cfff3 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e178)", 2024-09-10T19:22:11.774 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-10T19:17:27.037577+0000", 2024-09-10T19:22:11.774 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 283.19952637, 2024-09-10T19:22:11.774 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.9100273649999999, 2024-09-10T19:22:11.774 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-10T19:22:11.774 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "reached pg", 2024-09-10T19:22:11.775 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-10T19:22:11.775 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.15051", 2024-09-10T19:22:11.775 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.183:0/3632028672", 2024-09-10T19:22:11.775 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 93184 2024-09-10T19:22:11.775 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:11.775 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-10T19:22:11.775 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:11.775 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-10T19:22:11.775 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:27.037577+0000", 2024-09-10T19:22:11.775 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:22:11.775 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:11.775 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:11.775 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-10T19:22:11.776 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:27.037577+0000", 2024-09-10T19:22:11.776 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:22:11.776 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:11.776 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:11.776 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-10T19:22:11.776 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:27.037576+0000", 2024-09-10T19:22:11.776 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.999999 2024-09-10T19:22:11.776 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:11.776 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:11.776 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-10T19:22:11.776 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:27.037604+0000", 2024-09-10T19:22:11.776 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.8466000000000002e-05 2024-09-10T19:22:11.776 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:11.776 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:11.777 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-10T19:22:11.777 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:27.037619+0000", 2024-09-10T19:22:11.777 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.5418e-05 2024-09-10T19:22:11.777 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:11.777 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:11.777 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-10T19:22:11.777 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:27.037626+0000", 2024-09-10T19:22:11.777 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.426e-06 2024-09-10T19:22:11.777 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:11.777 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:11.777 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:22:11.777 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:27.037668+0000", 2024-09-10T19:22:11.777 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.1878999999999997e-05 2024-09-10T19:22:11.778 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:11.778 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:11.778 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for peered", 2024-09-10T19:22:11.778 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:27.037676+0000", 2024-09-10T19:22:11.778 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.8909999999999995e-06 2024-09-10T19:22:11.778 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:11.778 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:11.778 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:22:11.778 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:27.945157+0000", 2024-09-10T19:22:11.778 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.90748107099999997 2024-09-10T19:22:11.778 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:11.778 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:11.778 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for peered", 2024-09-10T19:22:11.779 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:27.945165+0000", 2024-09-10T19:22:11.779 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.2579999999999995e-06 2024-09-10T19:22:11.779 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:11.779 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:11.779 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:22:11.779 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:28.947583+0000", 2024-09-10T19:22:11.779 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.0024182589999999 2024-09-10T19:22:11.779 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:11.779 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:11.779 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-10T19:22:11.779 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:28.947604+0000", 2024-09-10T19:22:11.779 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.0771000000000002e-05 2024-09-10T19:22:11.779 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:22:11.779 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-10T19:22:11.780 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:22:11.780 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:11.780 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:11.780 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.15051.0:93229 8.13 8.8d3dde53 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e178)", 2024-09-10T19:22:11.780 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-10T19:17:27.063512+0000", 2024-09-10T19:22:11.780 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 283.17359091700001, 2024-09-10T19:22:11.780 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.8841211410000001, 2024-09-10T19:22:11.780 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-10T19:22:11.780 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "reached pg", 2024-09-10T19:22:11.780 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-10T19:22:11.780 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.15051", 2024-09-10T19:22:11.780 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.183:0/3632028672", 2024-09-10T19:22:11.780 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 93229 2024-09-10T19:22:11.781 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:11.781 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-10T19:22:11.781 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:11.781 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-10T19:22:11.781 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:27.063512+0000", 2024-09-10T19:22:11.781 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:22:11.781 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:11.781 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:11.781 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-10T19:22:11.781 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:27.063512+0000", 2024-09-10T19:22:11.781 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:22:11.781 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:11.781 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:11.782 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-10T19:22:11.782 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:27.063510+0000", 2024-09-10T19:22:11.782 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999981 2024-09-10T19:22:11.782 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:11.782 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:11.782 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-10T19:22:11.782 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:27.063560+0000", 2024-09-10T19:22:11.782 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.0076e-05 2024-09-10T19:22:11.782 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:11.782 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:11.782 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-10T19:22:11.782 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:27.063579+0000", 2024-09-10T19:22:11.782 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.8763e-05 2024-09-10T19:22:11.782 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:11.783 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:11.783 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-10T19:22:11.783 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:27.063587+0000", 2024-09-10T19:22:11.783 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.5020000000000003e-06 2024-09-10T19:22:11.783 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:11.783 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:11.783 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:22:11.783 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:27.063663+0000", 2024-09-10T19:22:11.783 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.6592000000000001e-05 2024-09-10T19:22:11.783 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:11.783 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:11.783 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for peered", 2024-09-10T19:22:11.783 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:27.063673+0000", 2024-09-10T19:22:11.784 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.4620000000000002e-06 2024-09-10T19:22:11.784 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:11.784 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:11.784 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:22:11.784 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:27.945182+0000", 2024-09-10T19:22:11.784 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.88150978000000002 2024-09-10T19:22:11.784 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:11.784 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:11.784 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for peered", 2024-09-10T19:22:11.784 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:27.945187+0000", 2024-09-10T19:22:11.784 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.7330000000000003e-06 2024-09-10T19:22:11.784 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:11.784 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:11.784 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:22:11.785 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:28.947626+0000", 2024-09-10T19:22:11.785 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.0024391020000001 2024-09-10T19:22:11.785 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:11.785 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:11.785 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-10T19:22:11.785 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:28.947633+0000", 2024-09-10T19:22:11.785 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.0890000000000002e-06 2024-09-10T19:22:11.785 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:22:11.785 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-10T19:22:11.785 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:22:11.785 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:11.785 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:11.785 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.15051.0:93231 8.13 8.7225c053 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e178)", 2024-09-10T19:22:11.786 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-10T19:17:27.065206+0000", 2024-09-10T19:22:11.786 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 283.17189680600001, 2024-09-10T19:22:11.786 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.88245339, 2024-09-10T19:22:11.786 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-10T19:22:11.786 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "reached pg", 2024-09-10T19:22:11.786 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-10T19:22:11.786 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.15051", 2024-09-10T19:22:11.786 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.183:0/3632028672", 2024-09-10T19:22:11.786 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 93231 2024-09-10T19:22:11.786 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:11.786 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-10T19:22:11.787 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:11.787 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-10T19:22:11.787 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:27.065206+0000", 2024-09-10T19:22:11.787 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:22:11.787 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:11.787 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:11.787 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-10T19:22:11.787 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:27.065206+0000", 2024-09-10T19:22:11.787 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:22:11.788 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:11.788 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:11.788 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-10T19:22:11.788 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:27.065205+0000", 2024-09-10T19:22:11.788 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999986 2024-09-10T19:22:11.788 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:11.788 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:11.788 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-10T19:22:11.788 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:27.065233+0000", 2024-09-10T19:22:11.788 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.8198999999999999e-05 2024-09-10T19:22:11.789 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:11.789 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:11.789 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-10T19:22:11.789 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:27.065508+0000", 2024-09-10T19:22:11.789 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00027513599999999999 2024-09-10T19:22:11.789 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:11.789 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:11.789 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-10T19:22:11.789 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:27.065515+0000", 2024-09-10T19:22:11.789 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.5950000000000004e-06 2024-09-10T19:22:11.789 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:11.789 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:11.789 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:22:11.790 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:27.065569+0000", 2024-09-10T19:22:11.790 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.4144000000000002e-05 2024-09-10T19:22:11.790 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:11.790 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:11.790 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for peered", 2024-09-10T19:22:11.790 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:27.065576+0000", 2024-09-10T19:22:11.790 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.3050000000000004e-06 2024-09-10T19:22:11.790 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:11.790 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:11.790 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:22:11.790 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:27.945203+0000", 2024-09-10T19:22:11.790 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.879626876 2024-09-10T19:22:11.790 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:11.791 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:11.791 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for peered", 2024-09-10T19:22:11.791 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:27.945209+0000", 2024-09-10T19:22:11.791 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.4400000000000002e-06 2024-09-10T19:22:11.791 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:11.791 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:11.791 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:22:11.791 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:28.947653+0000", 2024-09-10T19:22:11.791 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.0024433909999999 2024-09-10T19:22:11.791 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:11.791 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:11.791 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-10T19:22:11.791 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:28.947660+0000", 2024-09-10T19:22:11.791 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.7769999999999997e-06 2024-09-10T19:22:11.791 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:22:11.792 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-10T19:22:11.792 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:22:11.792 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:11.792 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:11.792 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.15051.0:93252 8.13 8.4feb5cf3 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e178)", 2024-09-10T19:22:11.792 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-10T19:17:27.107462+0000", 2024-09-10T19:22:11.792 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 283.12964139399998, 2024-09-10T19:22:11.792 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.840251692, 2024-09-10T19:22:11.792 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-10T19:22:11.792 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "reached pg", 2024-09-10T19:22:11.792 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-10T19:22:11.792 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.15051", 2024-09-10T19:22:11.792 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.183:0/3632028672", 2024-09-10T19:22:11.793 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 93252 2024-09-10T19:22:11.793 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:11.793 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-10T19:22:11.793 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:11.793 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-10T19:22:11.793 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:27.107462+0000", 2024-09-10T19:22:11.793 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:22:11.793 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:11.793 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:11.793 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-10T19:22:11.793 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:27.107462+0000", 2024-09-10T19:22:11.793 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:22:11.793 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:11.794 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:11.794 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-10T19:22:11.794 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:27.107461+0000", 2024-09-10T19:22:11.794 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999995 2024-09-10T19:22:11.794 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:11.794 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:11.794 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-10T19:22:11.794 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:27.107578+0000", 2024-09-10T19:22:11.794 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000116752 2024-09-10T19:22:11.794 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:11.794 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:11.794 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-10T19:22:11.794 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:27.107588+0000", 2024-09-10T19:22:11.794 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.0383e-05 2024-09-10T19:22:11.794 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:11.795 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:11.795 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-10T19:22:11.795 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:27.107593+0000", 2024-09-10T19:22:11.795 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.3600000000000004e-06 2024-09-10T19:22:11.795 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:11.795 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:11.795 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:22:11.795 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:27.107667+0000", 2024-09-10T19:22:11.795 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.3631e-05 2024-09-10T19:22:11.795 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:11.795 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:11.795 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for peered", 2024-09-10T19:22:11.795 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:27.107674+0000", 2024-09-10T19:22:11.796 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.1740000000000003e-06 2024-09-10T19:22:11.796 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:11.796 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:11.796 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:22:11.796 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:27.945226+0000", 2024-09-10T19:22:11.796 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.83755159300000004 2024-09-10T19:22:11.796 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:11.796 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:11.796 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for peered", 2024-09-10T19:22:11.796 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:27.945231+0000", 2024-09-10T19:22:11.796 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.5419999999999996e-06 2024-09-10T19:22:11.796 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:11.796 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:11.796 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:22:11.797 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:28.947706+0000", 2024-09-10T19:22:11.797 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.0024742929999999 2024-09-10T19:22:11.797 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:11.797 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:11.797 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-10T19:22:11.797 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:28.947713+0000", 2024-09-10T19:22:11.797 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.6410000000000002e-06 2024-09-10T19:22:11.797 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:22:11.797 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-10T19:22:11.797 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:22:11.797 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:11.797 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:11.797 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.15051.0:93300 8.13 8.7cc11153 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e178)", 2024-09-10T19:22:11.798 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-10T19:17:27.142933+0000", 2024-09-10T19:22:11.798 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 283.09417052100002, 2024-09-10T19:22:11.798 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.8048062439999999, 2024-09-10T19:22:11.798 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-10T19:22:11.798 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "reached pg", 2024-09-10T19:22:11.798 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-10T19:22:11.798 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.15051", 2024-09-10T19:22:11.798 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.183:0/3632028672", 2024-09-10T19:22:11.798 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 93300 2024-09-10T19:22:11.798 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:11.798 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-10T19:22:11.798 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:11.798 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-10T19:22:11.798 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:27.142933+0000", 2024-09-10T19:22:11.799 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:22:11.799 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:11.799 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:11.799 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-10T19:22:11.799 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:27.142933+0000", 2024-09-10T19:22:11.799 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:22:11.799 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:11.799 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:11.799 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-10T19:22:11.799 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:27.142932+0000", 2024-09-10T19:22:11.799 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999995 2024-09-10T19:22:11.799 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:11.799 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:11.800 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-10T19:22:11.800 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:27.142964+0000", 2024-09-10T19:22:11.800 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.1646999999999997e-05 2024-09-10T19:22:11.800 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:11.800 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:11.800 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-10T19:22:11.800 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:27.142970+0000", 2024-09-10T19:22:11.800 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.2090000000000002e-06 2024-09-10T19:22:11.800 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:11.800 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:11.800 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-10T19:22:11.800 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:27.142973+0000", 2024-09-10T19:22:11.800 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.202e-06 2024-09-10T19:22:11.801 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:11.801 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:11.801 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:22:11.801 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:27.143014+0000", 2024-09-10T19:22:11.801 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.1029000000000003e-05 2024-09-10T19:22:11.801 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:11.801 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:11.801 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for peered", 2024-09-10T19:22:11.801 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:27.143020+0000", 2024-09-10T19:22:11.801 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.7969999999999997e-06 2024-09-10T19:22:11.801 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:11.801 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:11.801 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:22:11.802 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:27.945248+0000", 2024-09-10T19:22:11.802 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.80222796500000004 2024-09-10T19:22:11.802 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:11.802 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:11.802 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for peered", 2024-09-10T19:22:11.802 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:27.945253+0000", 2024-09-10T19:22:11.802 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.198e-06 2024-09-10T19:22:11.802 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:11.802 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:11.802 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:22:11.802 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:28.947731+0000", 2024-09-10T19:22:11.802 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.002478379 2024-09-10T19:22:11.802 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:11.802 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:11.803 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-10T19:22:11.803 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:28.947739+0000", 2024-09-10T19:22:11.803 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.464e-06 2024-09-10T19:22:11.803 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:22:11.803 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-10T19:22:11.803 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:22:11.803 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:11.803 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:11.803 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.15051.0:93324 8.13 8.bd83f013 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e178)", 2024-09-10T19:22:11.803 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-10T19:17:27.174031+0000", 2024-09-10T19:22:11.803 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 283.063072197, 2024-09-10T19:22:11.803 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.7737433629999999, 2024-09-10T19:22:11.803 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-10T19:22:11.804 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "reached pg", 2024-09-10T19:22:11.804 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-10T19:22:11.804 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.15051", 2024-09-10T19:22:11.804 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.183:0/3632028672", 2024-09-10T19:22:11.804 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 93324 2024-09-10T19:22:11.804 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:11.804 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-10T19:22:11.804 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:11.804 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-10T19:22:11.804 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:27.174031+0000", 2024-09-10T19:22:11.804 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:22:11.804 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:11.804 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:11.805 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-10T19:22:11.805 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:27.174031+0000", 2024-09-10T19:22:11.805 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:22:11.805 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:11.805 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:11.805 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-10T19:22:11.805 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:27.174030+0000", 2024-09-10T19:22:11.805 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.999999 2024-09-10T19:22:11.805 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:11.805 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:11.805 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-10T19:22:11.805 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:27.174087+0000", 2024-09-10T19:22:11.805 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.6977e-05 2024-09-10T19:22:11.806 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:11.806 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:11.806 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-10T19:22:11.806 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:27.174098+0000", 2024-09-10T19:22:11.806 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.0872e-05 2024-09-10T19:22:11.806 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:11.806 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:11.806 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-10T19:22:11.806 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:27.174102+0000", 2024-09-10T19:22:11.806 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.2869999999999998e-06 2024-09-10T19:22:11.807 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:11.807 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:11.807 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:22:11.807 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:27.174144+0000", 2024-09-10T19:22:11.807 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.1989000000000001e-05 2024-09-10T19:22:11.807 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:11.807 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:11.807 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for peered", 2024-09-10T19:22:11.807 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:27.174150+0000", 2024-09-10T19:22:11.807 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.8939999999999998e-06 2024-09-10T19:22:11.807 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:11.807 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:11.807 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:22:11.807 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:27.945270+0000", 2024-09-10T19:22:11.808 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.77111963500000003 2024-09-10T19:22:11.808 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:11.808 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:11.808 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for peered", 2024-09-10T19:22:11.808 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:27.945275+0000", 2024-09-10T19:22:11.808 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.3229999999999997e-06 2024-09-10T19:22:11.808 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:11.808 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:11.808 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:22:11.808 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:28.947767+0000", 2024-09-10T19:22:11.808 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.0024922030000001 2024-09-10T19:22:11.808 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:11.809 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:11.809 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-10T19:22:11.809 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:28.947774+0000", 2024-09-10T19:22:11.809 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.0110000000000003e-06 2024-09-10T19:22:11.809 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:22:11.809 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-10T19:22:11.810 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:22:11.810 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:11.810 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:11.810 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.15051.0:93356 8.19 8:982b4d7e:::benchmark_data_smithi183_114924_object93355:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e179)", 2024-09-10T19:22:11.810 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-10T19:17:28.369111+0000", 2024-09-10T19:22:11.810 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 281.86799184300003, 2024-09-10T19:22:11.811 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.6680609639999999, 2024-09-10T19:22:11.811 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-10T19:22:11.811 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T19:22:11.811 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-10T19:22:11.811 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.15051", 2024-09-10T19:22:11.811 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.183:0/3632028672", 2024-09-10T19:22:11.811 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 93356 2024-09-10T19:22:11.811 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:11.811 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-10T19:22:11.811 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:11.811 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-10T19:22:11.811 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:28.369111+0000", 2024-09-10T19:22:11.811 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:22:11.811 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:11.812 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:11.812 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-10T19:22:11.812 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:28.369111+0000", 2024-09-10T19:22:11.812 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:22:11.812 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:11.812 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:11.812 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-10T19:22:11.812 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:28.369109+0000", 2024-09-10T19:22:11.812 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999981 2024-09-10T19:22:11.812 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:11.812 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:11.812 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-10T19:22:11.812 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:28.369196+0000", 2024-09-10T19:22:11.813 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.7334000000000001e-05 2024-09-10T19:22:11.813 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:11.813 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:11.813 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-10T19:22:11.813 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:28.369218+0000", 2024-09-10T19:22:11.813 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.1277999999999999e-05 2024-09-10T19:22:11.813 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:11.813 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:11.813 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-10T19:22:11.813 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:28.369225+0000", 2024-09-10T19:22:11.813 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.2989999999999999e-06 2024-09-10T19:22:11.813 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:11.813 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:11.814 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:22:11.814 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:28.369283+0000", 2024-09-10T19:22:11.814 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.7565000000000002e-05 2024-09-10T19:22:11.814 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:11.814 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:11.814 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for peered", 2024-09-10T19:22:11.814 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:28.369290+0000", 2024-09-10T19:22:11.814 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.6240000000000002e-06 2024-09-10T19:22:11.814 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:11.814 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:11.814 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:22:11.815 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:28.951079+0000", 2024-09-10T19:22:11.815 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.58178845800000001 2024-09-10T19:22:11.815 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:11.815 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:11.815 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for peered", 2024-09-10T19:22:11.815 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:28.951090+0000", 2024-09-10T19:22:11.815 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.0975000000000001e-05 2024-09-10T19:22:11.815 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:11.815 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:11.815 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:22:11.815 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:30.030802+0000", 2024-09-10T19:22:11.815 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.0797127639999999 2024-09-10T19:22:11.816 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:11.816 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:11.816 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-10T19:22:11.816 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:30.030973+0000", 2024-09-10T19:22:11.816 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00017021199999999999 2024-09-10T19:22:11.816 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:11.816 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:11.816 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0,4,6", 2024-09-10T19:22:11.816 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:30.031240+0000", 2024-09-10T19:22:11.816 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00026700299999999998 2024-09-10T19:22:11.816 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:11.816 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:11.816 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:22:11.816 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:30.033930+0000", 2024-09-10T19:22:11.817 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0026898680000000002 2024-09-10T19:22:11.817 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:11.817 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:11.817 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-09-10T19:22:11.817 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:30.033938+0000", 2024-09-10T19:22:11.817 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.5220000000000007e-06 2024-09-10T19:22:11.817 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:11.817 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:11.817 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:22:11.817 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:30.036650+0000", 2024-09-10T19:22:11.817 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0027118929999999999 2024-09-10T19:22:11.817 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:11.817 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:11.818 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:22:11.818 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:30.036962+0000", 2024-09-10T19:22:11.818 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00031158899999999997 2024-09-10T19:22:11.818 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:11.818 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:11.818 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-09-10T19:22:11.818 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:30.037028+0000", 2024-09-10T19:22:11.818 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.6637999999999995e-05 2024-09-10T19:22:11.818 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:11.818 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:11.818 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-10T19:22:11.818 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:30.037172+0000", 2024-09-10T19:22:11.818 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00014398199999999999 2024-09-10T19:22:11.818 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:22:11.819 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-10T19:22:11.819 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:22:11.819 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:11.819 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:11.819 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.15051.0:93371 8.19 8:9901a0b3:::benchmark_data_smithi183_114924_object93370:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e179)", 2024-09-10T19:22:11.819 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-10T19:17:28.414412+0000", 2024-09-10T19:22:11.819 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 281.82269122700001, 2024-09-10T19:22:11.819 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.627689602, 2024-09-10T19:22:11.819 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-10T19:22:11.819 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T19:22:11.819 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-10T19:22:11.819 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.15051", 2024-09-10T19:22:11.819 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.183:0/3632028672", 2024-09-10T19:22:11.820 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 93371 2024-09-10T19:22:11.820 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:11.820 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-10T19:22:11.820 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:11.820 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-10T19:22:11.820 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:28.414412+0000", 2024-09-10T19:22:11.820 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:22:11.820 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:11.820 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:11.820 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-10T19:22:11.820 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:28.414412+0000", 2024-09-10T19:22:11.820 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:22:11.820 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:11.821 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:11.821 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-10T19:22:11.821 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:28.414411+0000", 2024-09-10T19:22:11.821 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.999999 2024-09-10T19:22:11.821 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:11.821 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:11.821 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-10T19:22:11.821 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:28.414465+0000", 2024-09-10T19:22:11.821 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.4323000000000001e-05 2024-09-10T19:22:11.821 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:11.821 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:11.821 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-10T19:22:11.821 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:28.414477+0000", 2024-09-10T19:22:11.821 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.1323e-05 2024-09-10T19:22:11.822 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:11.822 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:11.822 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-10T19:22:11.822 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:28.414481+0000", 2024-09-10T19:22:11.822 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.1930000000000003e-06 2024-09-10T19:22:11.822 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:11.822 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:11.822 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:22:11.822 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:28.414522+0000", 2024-09-10T19:22:11.822 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.1121999999999997e-05 2024-09-10T19:22:11.822 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:11.822 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:11.822 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for peered", 2024-09-10T19:22:11.822 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:28.414529+0000", 2024-09-10T19:22:11.823 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.7959999999999998e-06 2024-09-10T19:22:11.823 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:11.823 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:11.823 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:22:11.823 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:28.951123+0000", 2024-09-10T19:22:11.823 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.53659462000000002 2024-09-10T19:22:11.823 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:11.823 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:11.823 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for peered", 2024-09-10T19:22:11.823 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:28.951131+0000", 2024-09-10T19:22:11.823 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.5070000000000003e-06 2024-09-10T19:22:11.823 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:11.824 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:11.824 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:22:11.824 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:30.033963+0000", 2024-09-10T19:22:11.824 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.0828317970000001 2024-09-10T19:22:11.824 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:11.824 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:11.824 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-10T19:22:11.824 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:30.034106+0000", 2024-09-10T19:22:11.824 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000143516 2024-09-10T19:22:11.824 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:11.824 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:11.824 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0,4,6", 2024-09-10T19:22:11.824 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:30.034378+0000", 2024-09-10T19:22:11.825 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00027172699999999998 2024-09-10T19:22:11.825 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:11.825 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:11.825 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:22:11.825 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:30.037340+0000", 2024-09-10T19:22:11.825 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0029616719999999998 2024-09-10T19:22:11.825 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:11.825 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:11.825 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-09-10T19:22:11.825 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:30.040333+0000", 2024-09-10T19:22:11.825 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0029933360000000001 2024-09-10T19:22:11.825 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:11.825 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:11.826 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:22:11.826 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:30.040365+0000", 2024-09-10T19:22:11.826 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.2110000000000003e-05 2024-09-10T19:22:11.826 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:11.826 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:11.826 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:22:11.826 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:30.041943+0000", 2024-09-10T19:22:11.826 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0015777359999999999 2024-09-10T19:22:11.826 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:11.826 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:11.826 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-09-10T19:22:11.826 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:30.041995+0000", 2024-09-10T19:22:11.826 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.1918e-05 2024-09-10T19:22:11.826 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:11.827 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:11.827 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-10T19:22:11.827 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:30.042101+0000", 2024-09-10T19:22:11.827 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00010680599999999999 2024-09-10T19:22:11.827 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:22:11.827 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-10T19:22:11.827 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:22:11.827 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:11.827 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:11.827 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.15051.0:151612 8.11 8:89d8a486:::benchmark_data_smithi183_114924_object31329:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e198)", 2024-09-10T19:22:11.827 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-10T19:18:23.050999+0000", 2024-09-10T19:22:11.827 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 227.18610386, 2024-09-10T19:22:11.827 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.97268184599999996, 2024-09-10T19:22:11.828 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-10T19:22:11.828 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T19:22:11.828 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-10T19:22:11.828 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.15051", 2024-09-10T19:22:11.828 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.183:0/3632028672", 2024-09-10T19:22:11.828 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 151612 2024-09-10T19:22:11.828 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:11.828 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-10T19:22:11.828 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:11.828 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-10T19:22:11.828 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:18:23.050999+0000", 2024-09-10T19:22:11.828 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:22:11.829 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:11.829 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:11.829 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-10T19:22:11.829 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:18:23.050999+0000", 2024-09-10T19:22:11.829 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:22:11.829 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:11.829 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:11.829 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-10T19:22:11.829 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:18:23.050998+0000", 2024-09-10T19:22:11.829 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.999999 2024-09-10T19:22:11.829 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:11.829 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:11.829 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-10T19:22:11.830 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:18:23.051003+0000", 2024-09-10T19:22:11.830 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.7589999999999997e-06 2024-09-10T19:22:11.830 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:11.830 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:11.830 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-10T19:22:11.830 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:18:23.051012+0000", 2024-09-10T19:22:11.830 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.9879999999999993e-06 2024-09-10T19:22:11.830 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:11.830 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:11.830 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-10T19:22:11.830 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:18:23.051017+0000", 2024-09-10T19:22:11.830 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.7879999999999997e-06 2024-09-10T19:22:11.830 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:11.830 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:11.830 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:22:11.831 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:18:23.051063+0000", 2024-09-10T19:22:11.831 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.6418999999999997e-05 2024-09-10T19:22:11.831 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:11.831 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:11.831 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for peered", 2024-09-10T19:22:11.831 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:18:23.051075+0000", 2024-09-10T19:22:11.831 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.2439000000000001e-05 2024-09-10T19:22:11.831 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:11.831 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:11.831 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:22:11.831 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:18:24.018580+0000", 2024-09-10T19:22:11.831 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.96750444700000005 2024-09-10T19:22:11.831 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:11.832 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:11.832 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-10T19:22:11.832 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:18:24.018674+0000", 2024-09-10T19:22:11.832 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.4519999999999996e-05 2024-09-10T19:22:11.832 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:11.832 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:11.832 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 6,7", 2024-09-10T19:22:11.832 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:18:24.018949+0000", 2024-09-10T19:22:11.832 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00027485200000000002 2024-09-10T19:22:11.832 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:11.832 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:11.832 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:22:11.833 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:18:24.020832+0000", 2024-09-10T19:22:11.833 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0018829529999999999 2024-09-10T19:22:11.833 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:11.833 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:11.833 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-09-10T19:22:11.833 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:18:24.020841+0000", 2024-09-10T19:22:11.833 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.5199999999999997e-06 2024-09-10T19:22:11.833 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:11.833 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:11.833 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:22:11.833 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:18:24.023578+0000", 2024-09-10T19:22:11.833 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0027376710000000001 2024-09-10T19:22:11.833 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:11.833 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:11.834 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-09-10T19:22:11.834 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:18:24.023644+0000", 2024-09-10T19:22:11.834 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.5640000000000002e-05 2024-09-10T19:22:11.834 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:11.834 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:11.834 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-10T19:22:11.834 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:18:24.023681+0000", 2024-09-10T19:22:11.834 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.6977000000000002e-05 2024-09-10T19:22:11.834 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:22:11.834 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-10T19:22:11.834 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:22:11.834 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:11.834 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:11.835 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.15051.0:151614 8.11 8:8b9042c4:::benchmark_data_smithi183_114924_object31331:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e198)", 2024-09-10T19:22:11.835 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-10T19:18:23.052048+0000", 2024-09-10T19:22:11.835 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 227.18505482099999, 2024-09-10T19:22:11.835 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.97478914900000002, 2024-09-10T19:22:11.835 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-10T19:22:11.835 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T19:22:11.835 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-10T19:22:11.835 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.15051", 2024-09-10T19:22:11.835 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.183:0/3632028672", 2024-09-10T19:22:11.835 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 151614 2024-09-10T19:22:11.835 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:11.835 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-10T19:22:11.835 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:11.835 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-10T19:22:11.835 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:18:23.052048+0000", 2024-09-10T19:22:11.836 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:22:11.836 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:11.836 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:11.836 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-10T19:22:11.836 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:18:23.052048+0000", 2024-09-10T19:22:11.836 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:22:11.836 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:11.836 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:11.836 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-10T19:22:11.836 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:18:23.052047+0000", 2024-09-10T19:22:11.836 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.999999 2024-09-10T19:22:11.836 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:11.837 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:11.837 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-10T19:22:11.837 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:18:23.052052+0000", 2024-09-10T19:22:11.837 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.8539999999999996e-06 2024-09-10T19:22:11.837 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:11.837 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:11.837 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-10T19:22:11.837 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:18:23.052065+0000", 2024-09-10T19:22:11.837 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.2468e-05 2024-09-10T19:22:11.837 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:11.837 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:11.837 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-10T19:22:11.838 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:18:23.052071+0000", 2024-09-10T19:22:11.838 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.9290000000000003e-06 2024-09-10T19:22:11.838 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:11.838 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:11.838 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:22:11.838 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:18:23.052109+0000", 2024-09-10T19:22:11.838 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.8807999999999998e-05 2024-09-10T19:22:11.838 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:11.838 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:11.838 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for peered", 2024-09-10T19:22:11.838 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:18:23.052118+0000", 2024-09-10T19:22:11.838 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.9919999999999996e-06 2024-09-10T19:22:11.838 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:11.839 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:11.839 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:22:11.839 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:18:24.020885+0000", 2024-09-10T19:22:11.839 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.96876697199999995 2024-09-10T19:22:11.839 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:11.839 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:11.839 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-10T19:22:11.839 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:18:24.021004+0000", 2024-09-10T19:22:11.839 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000118183 2024-09-10T19:22:11.839 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:11.839 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:11.839 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 6,7", 2024-09-10T19:22:11.840 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:18:24.021207+0000", 2024-09-10T19:22:11.840 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000203401 2024-09-10T19:22:11.840 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:11.840 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:11.840 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-09-10T19:22:11.840 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:18:24.023784+0000", 2024-09-10T19:22:11.840 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00257748 2024-09-10T19:22:11.840 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:11.840 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:11.840 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:22:11.840 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:18:24.023822+0000", 2024-09-10T19:22:11.840 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.7171e-05 2024-09-10T19:22:11.840 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:11.841 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:11.841 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:22:11.841 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:18:24.026734+0000", 2024-09-10T19:22:11.841 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0029124839999999999 2024-09-10T19:22:11.841 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:11.841 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:11.841 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-09-10T19:22:11.841 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:18:24.026804+0000", 2024-09-10T19:22:11.841 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.0196000000000004e-05 2024-09-10T19:22:11.841 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:11.841 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:11.841 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-10T19:22:11.841 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:18:24.026837+0000", 2024-09-10T19:22:11.841 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.3031000000000003e-05 2024-09-10T19:22:11.842 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:22:11.842 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-10T19:22:11.842 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:22:11.842 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:11.842 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:11.842 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.15360.0:169771 9.6 9:6a78d762:::benchmark_data_smithi183_142499_object23282:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e262)", 2024-09-10T19:22:11.842 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-10T19:21:46.953119+0000", 2024-09-10T19:22:11.842 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 23.283983672000002, 2024-09-10T19:22:11.842 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.06359206, 2024-09-10T19:22:11.842 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-10T19:22:11.842 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "reached pg", 2024-09-10T19:22:11.842 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-10T19:22:11.843 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.15360", 2024-09-10T19:22:11.843 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.183:0/2364804131", 2024-09-10T19:22:11.843 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 169771 2024-09-10T19:22:11.843 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:11.843 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-10T19:22:11.843 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:11.843 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-10T19:22:11.843 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:21:46.953119+0000", 2024-09-10T19:22:11.843 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:22:11.843 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:11.843 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:11.843 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-10T19:22:11.843 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:21:46.953119+0000", 2024-09-10T19:22:11.844 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:22:11.844 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:11.844 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:11.844 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-10T19:22:11.844 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:21:46.953119+0000", 2024-09-10T19:22:11.844 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999995 2024-09-10T19:22:11.844 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:11.844 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:11.844 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-10T19:22:11.844 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:21:46.953125+0000", 2024-09-10T19:22:11.844 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.9950000000000002e-06 2024-09-10T19:22:11.844 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:11.844 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:11.845 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-10T19:22:11.845 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:21:46.953133+0000", 2024-09-10T19:22:11.845 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.8390000000000007e-06 2024-09-10T19:22:11.845 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:11.845 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:11.845 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-10T19:22:11.845 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:21:46.953137+0000", 2024-09-10T19:22:11.845 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.87e-06 2024-09-10T19:22:11.845 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:11.845 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:11.845 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:22:11.845 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:21:46.954452+0000", 2024-09-10T19:22:11.845 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.001314787 2024-09-10T19:22:11.846 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:11.846 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:11.846 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-10T19:22:11.846 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:21:46.954562+0000", 2024-09-10T19:22:11.846 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00011018000000000001 2024-09-10T19:22:11.846 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:11.846 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:11.846 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0,6", 2024-09-10T19:22:11.846 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:21:46.954675+0000", 2024-09-10T19:22:11.846 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000112895 2024-09-10T19:22:11.846 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:11.846 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:11.846 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-09-10T19:22:11.846 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:21:46.959358+0000", 2024-09-10T19:22:11.847 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0046832080000000003 2024-09-10T19:22:11.847 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:11.847 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:11.847 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:22:11.847 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:21:46.971046+0000", 2024-09-10T19:22:11.847 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.011687362 2024-09-10T19:22:11.847 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:11.847 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:11.847 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:22:11.847 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:21:46.986841+0000", 2024-09-10T19:22:11.847 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.015795317 2024-09-10T19:22:11.847 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:11.847 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:11.848 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for peered", 2024-09-10T19:22:11.848 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:21:46.986849+0000", 2024-09-10T19:22:11.848 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.0059999999999992e-06 2024-09-10T19:22:11.848 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:11.848 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:11.848 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:22:11.848 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:21:48.016648+0000", 2024-09-10T19:22:11.848 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.0297984440000001 2024-09-10T19:22:11.848 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:11.848 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:11.848 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-10T19:22:11.848 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:21:48.016711+0000", 2024-09-10T19:22:11.848 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.3819999999999995e-05 2024-09-10T19:22:11.848 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:22:11.849 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-10T19:22:11.849 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:22:11.849 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:11.849 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:11.849 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.15360.0:169759 9.5 9:a00f2629:::benchmark_data_smithi183_142499_object23270:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e263)", 2024-09-10T19:22:11.849 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-10T19:21:46.960422+0000", 2024-09-10T19:22:11.849 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 23.276681368999999, 2024-09-10T19:22:11.849 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.057974347, 2024-09-10T19:22:11.849 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-10T19:22:11.849 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "reached pg", 2024-09-10T19:22:11.849 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-10T19:22:11.849 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.15360", 2024-09-10T19:22:11.849 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.183:0/2364804131", 2024-09-10T19:22:11.849 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 169759 2024-09-10T19:22:11.850 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:11.850 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-10T19:22:11.850 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:11.850 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-10T19:22:11.850 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:21:46.960422+0000", 2024-09-10T19:22:11.850 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:22:11.850 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:11.850 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:11.850 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-10T19:22:11.850 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:21:46.960422+0000", 2024-09-10T19:22:11.850 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:22:11.850 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:11.850 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:11.850 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-10T19:22:11.851 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:21:46.960420+0000", 2024-09-10T19:22:11.851 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999986 2024-09-10T19:22:11.851 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:11.851 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:11.851 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-10T19:22:11.851 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:21:46.960426+0000", 2024-09-10T19:22:11.851 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.9159999999999998e-06 2024-09-10T19:22:11.851 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:11.851 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:11.851 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-10T19:22:11.851 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:21:46.960436+0000", 2024-09-10T19:22:11.851 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.5380000000000008e-06 2024-09-10T19:22:11.851 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:11.852 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:11.852 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-10T19:22:11.852 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:21:46.960441+0000", 2024-09-10T19:22:11.852 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.271e-06 2024-09-10T19:22:11.852 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:11.852 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:11.852 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:22:11.852 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:21:46.987106+0000", 2024-09-10T19:22:11.852 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.026665287999999999 2024-09-10T19:22:11.852 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:11.852 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:11.852 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for peered", 2024-09-10T19:22:11.852 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:21:46.987113+0000", 2024-09-10T19:22:11.853 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.2129999999999996e-06 2024-09-10T19:22:11.853 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:11.853 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:11.853 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:22:11.853 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:21:48.014573+0000", 2024-09-10T19:22:11.853 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.027460866 2024-09-10T19:22:11.853 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:11.853 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:11.853 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-10T19:22:11.853 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:21:48.018396+0000", 2024-09-10T19:22:11.853 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0038226290000000001 2024-09-10T19:22:11.853 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:22:11.853 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-10T19:22:11.853 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:22:11.854 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:11.854 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:11.854 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.15360.0:169769 9.5 9:aaef8d5a:::benchmark_data_smithi183_142499_object23280:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e263)", 2024-09-10T19:22:11.854 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-10T19:21:46.960447+0000", 2024-09-10T19:22:11.854 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 23.276656122999999, 2024-09-10T19:22:11.854 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.059507046, 2024-09-10T19:22:11.854 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-10T19:22:11.854 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "reached pg", 2024-09-10T19:22:11.854 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-10T19:22:11.854 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.15360", 2024-09-10T19:22:11.854 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.183:0/2364804131", 2024-09-10T19:22:11.854 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 169769 2024-09-10T19:22:11.854 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:11.854 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-10T19:22:11.855 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:11.855 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-10T19:22:11.855 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:21:46.960447+0000", 2024-09-10T19:22:11.855 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:22:11.855 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:11.855 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:11.855 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-10T19:22:11.855 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:21:46.960447+0000", 2024-09-10T19:22:11.855 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:22:11.855 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:11.855 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:11.855 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-10T19:22:11.855 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:21:46.960446+0000", 2024-09-10T19:22:11.856 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999995 2024-09-10T19:22:11.856 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:11.856 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:11.856 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-10T19:22:11.856 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:21:46.960449+0000", 2024-09-10T19:22:11.856 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.6359999999999998e-06 2024-09-10T19:22:11.856 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:11.856 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:11.856 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-10T19:22:11.856 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:21:46.960455+0000", 2024-09-10T19:22:11.856 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.0839999999999997e-06 2024-09-10T19:22:11.856 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:11.856 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:11.856 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-10T19:22:11.857 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:21:46.960459+0000", 2024-09-10T19:22:11.857 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.4419999999999998e-06 2024-09-10T19:22:11.857 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:11.857 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:11.857 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:22:11.857 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:21:46.987136+0000", 2024-09-10T19:22:11.857 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.026677101000000002 2024-09-10T19:22:11.857 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:11.857 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:11.857 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for peered", 2024-09-10T19:22:11.857 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:21:46.987140+0000", 2024-09-10T19:22:11.857 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.3170000000000003e-06 2024-09-10T19:22:11.858 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:11.858 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:11.858 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:22:11.858 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:21:48.019902+0000", 2024-09-10T19:22:11.858 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.03276155 2024-09-10T19:22:11.858 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:11.858 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:11.858 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-10T19:22:11.858 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:21:48.019954+0000", 2024-09-10T19:22:11.858 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.2389000000000003e-05 2024-09-10T19:22:11.858 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:22:11.858 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-10T19:22:11.858 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:22:11.858 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:22:11.859 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-10T19:22:11.859 INFO:teuthology.orchestra.run.smithi191.stdout:} 2024-09-10T19:22:11.859 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:22:11.890 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:22:11.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:22:11.892 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:22:11.992 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:22:12.025 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:22:12.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:22:12.026 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:22:12.127 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:22:12.161 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:22:12.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:22:12.162 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:22:12.263 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:22:12.288 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:22:12.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:22:12.289 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:22:12.376 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:22:12 smithi183 ceph-mon[29736]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T19:22:12.376 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:22:12 smithi183 ceph-mon[29736]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-10T19:22:12.376 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:22:12 smithi183 ceph-mon[29736]: osdmap e267: 8 total, 6 up, 6 in 2024-09-10T19:22:12.376 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:22:12 smithi183 ceph-mon[41562]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T19:22:12.376 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:22:12 smithi183 ceph-mon[41562]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-10T19:22:12.377 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:22:12 smithi183 ceph-mon[41562]: osdmap e267: 8 total, 6 up, 6 in 2024-09-10T19:22:12.390 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:22:12.427 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:22:12.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:22:12.428 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:22:12.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:22:12 smithi191 ceph-mon[32884]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T19:22:12.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:22:12 smithi191 ceph-mon[32884]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-10T19:22:12.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:22:12 smithi191 ceph-mon[32884]: osdmap e267: 8 total, 6 up, 6 in 2024-09-10T19:22:12.529 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:22:12.563 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:22:12.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:22:12.564 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:22:12.664 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:22:12.697 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:22:12.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:22:12.698 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:22:12.746 INFO:tasks.thrashosds.thrasher:in_osds: [0, 3, 4, 5, 6, 7] out_osds: [1, 2] dead_osds: [1, 0] live_osds: [2, 3, 4, 5, 6, 7] 2024-09-10T19:22:12.746 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-10T19:22:12.747 INFO:tasks.thrashosds.thrasher:fixing pg num pool .mgr 2024-09-10T19:22:12.747 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph pg dump --format=json 2024-09-10T19:22:12.798 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:22:12.822 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:22:12.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:22:12.823 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:22:12.924 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:22:12.949 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:22:12.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:22:12.950 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:22:13.051 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:22:13.087 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:22:13.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:22:13.088 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:22:13.189 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:22:13.226 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:22:13.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:22:13.227 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:22:13.296 INFO:teuthology.orchestra.run.smithi183.stderr:Error ENXIO: problem getting command descriptions from osd.0 2024-09-10T19:22:13.297 INFO:teuthology.orchestra.run.smithi183.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-09-10T19:22:13.304 INFO:teuthology.orchestra.run.smithi183.stdout:osd.2: {} 2024-09-10T19:22:13.304 INFO:teuthology.orchestra.run.smithi183.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-10T19:22:13.305 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:22:13 smithi183 ceph-mon[29736]: pgmap v541: 158 pgs: 1 active+clean+scrubbing+deep, 45 active+undersized, 33 active+undersized+degraded, 79 active+clean; 5.9 GiB data, 23 GiB used, 513 GiB / 536 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1.04k op/s; 48346/289263 objects degraded (16.714%) 2024-09-10T19:22:13.305 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:22:13 smithi183 ceph-mon[41562]: pgmap v541: 158 pgs: 1 active+clean+scrubbing+deep, 45 active+undersized, 33 active+undersized+degraded, 79 active+clean; 5.9 GiB data, 23 GiB used, 513 GiB / 536 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1.04k op/s; 48346/289263 objects degraded (16.714%) 2024-09-10T19:22:13.328 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:22:13.334 INFO:teuthology.orchestra.run.smithi183.stdout:osd.3: {} 2024-09-10T19:22:13.335 INFO:teuthology.orchestra.run.smithi183.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-10T19:22:13.349 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:22:13.354 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:22:13.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:22:13.355 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:22:13.361 INFO:teuthology.orchestra.run.smithi183.stdout:osd.4: {} 2024-09-10T19:22:13.361 INFO:teuthology.orchestra.run.smithi183.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-10T19:22:13.388 INFO:teuthology.orchestra.run.smithi183.stdout:osd.5: {} 2024-09-10T19:22:13.388 INFO:teuthology.orchestra.run.smithi183.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-10T19:22:13.408 INFO:teuthology.orchestra.run.smithi183.stdout:osd.6: {} 2024-09-10T19:22:13.408 INFO:teuthology.orchestra.run.smithi183.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-10T19:22:13.435 INFO:teuthology.orchestra.run.smithi183.stdout:osd.7: {} 2024-09-10T19:22:13.435 INFO:teuthology.orchestra.run.smithi183.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-10T19:22:13.456 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:22:13.465 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:22:13 smithi191 ceph-mon[32884]: pgmap v541: 158 pgs: 1 active+clean+scrubbing+deep, 45 active+undersized, 33 active+undersized+degraded, 79 active+clean; 5.9 GiB data, 23 GiB used, 513 GiB / 536 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1.04k op/s; 48346/289263 objects degraded (16.714%) 2024-09-10T19:22:13.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:22:13.491 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:22:13.492 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:22:13.592 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:22:13.616 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:22:13.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:22:13.618 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:22:13.718 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:22:13.760 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:22:13.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:22:13.761 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:22:13.861 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph osd unset noscrub 2024-09-10T19:22:13.862 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:22:13.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:22:13.883 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:22:13.883 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:22:13.931 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Sep 10 19:22:13 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-y[29989]: 2024-09-10T19:22:13.674+0000 7f9fad6cd640 -1 mgr get_metadata_python Requested missing service osd.1 2024-09-10T19:22:13.984 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:22:14.011 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:22:14.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:22:14.013 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:22:14.113 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:22:14.141 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:22:14.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:22:14.142 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:22:14.242 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:22:14.278 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:22:14.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:22:14.279 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:22:14.380 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:22:14.420 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:22:14.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:22:14.421 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:22:14.425 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:22:14 smithi183 ceph-mon[29736]: from='client.? 172.21.15.183:0/1479857911' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T19:22:14.425 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:22:14 smithi183 ceph-mon[29736]: purged_snaps scrub starts 2024-09-10T19:22:14.425 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:22:14 smithi183 ceph-mon[29736]: purged_snaps scrub ok 2024-09-10T19:22:14.425 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:22:14 smithi183 ceph-mon[29736]: from='mgr.14556 ' entity='mgr.y' 2024-09-10T19:22:14.425 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:22:14 smithi183 ceph-mon[29736]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:22:14.425 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:22:14 smithi183 ceph-mon[29736]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T19:22:14.425 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:22:14 smithi183 ceph-mon[29736]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T19:22:14.425 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:22:14 smithi183 ceph-mon[29736]: from='mgr.14556 ' entity='mgr.y' 2024-09-10T19:22:14.425 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:22:14 smithi183 ceph-mon[41562]: from='client.? 172.21.15.183:0/1479857911' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T19:22:14.426 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:22:14 smithi183 ceph-mon[41562]: purged_snaps scrub starts 2024-09-10T19:22:14.426 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:22:14 smithi183 ceph-mon[41562]: purged_snaps scrub ok 2024-09-10T19:22:14.426 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:22:14 smithi183 ceph-mon[41562]: from='mgr.14556 ' entity='mgr.y' 2024-09-10T19:22:14.426 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:22:14 smithi183 ceph-mon[41562]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:22:14.426 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:22:14 smithi183 ceph-mon[41562]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T19:22:14.426 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:22:14 smithi183 ceph-mon[41562]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T19:22:14.426 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:22:14 smithi183 ceph-mon[41562]: from='mgr.14556 ' entity='mgr.y' 2024-09-10T19:22:14.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:22:14 smithi191 ceph-mon[32884]: from='client.? 172.21.15.183:0/1479857911' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T19:22:14.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:22:14 smithi191 ceph-mon[32884]: purged_snaps scrub starts 2024-09-10T19:22:14.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:22:14 smithi191 ceph-mon[32884]: purged_snaps scrub ok 2024-09-10T19:22:14.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:22:14 smithi191 ceph-mon[32884]: from='mgr.14556 ' entity='mgr.y' 2024-09-10T19:22:14.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:22:14 smithi191 ceph-mon[32884]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:22:14.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:22:14 smithi191 ceph-mon[32884]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T19:22:14.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:22:14 smithi191 ceph-mon[32884]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T19:22:14.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:22:14 smithi191 ceph-mon[32884]: from='mgr.14556 ' entity='mgr.y' 2024-09-10T19:22:14.522 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:22:14.546 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:22:14.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:22:14.547 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:22:14.648 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:22:14.674 DEBUG:teuthology.orchestra.run.smithi191:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph daemon osd.6 dump_ops_in_flight 2024-09-10T19:22:14.693 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:22:14.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:22:14.694 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:22:14.794 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:22:14.818 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:22:14.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:22:14.819 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:22:14.882 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-10T19:22:14.882 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-10T19:22:14.920 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:22:14.964 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:22:14.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:22:14.965 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:22:15.065 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:22:15.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:22:15.105 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:22:15.105 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:22:15.153 INFO:teuthology.orchestra.run.smithi191.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.b/config 2024-09-10T19:22:15.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:22:15 smithi183 ceph-mon[29736]: pgmap v542: 158 pgs: 1 active+clean+scrubbing+deep, 45 active+undersized, 33 active+undersized+degraded, 79 active+clean; 5.9 GiB data, 23 GiB used, 513 GiB / 536 GiB avail; 1023 B/s rd, 0 B/s wr, 1.02k op/s; 48346/289263 objects degraded (16.714%) 2024-09-10T19:22:15.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:22:15 smithi183 ceph-mon[29736]: osdmap e268: 8 total, 6 up, 6 in 2024-09-10T19:22:15.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:22:15 smithi183 ceph-mon[29736]: Health check update: Degraded data redundancy: 48346/289263 objects degraded (16.714%), 33 pgs degraded (PG_DEGRADED) 2024-09-10T19:22:15.181 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:22:15 smithi183 ceph-mon[41562]: pgmap v542: 158 pgs: 1 active+clean+scrubbing+deep, 45 active+undersized, 33 active+undersized+degraded, 79 active+clean; 5.9 GiB data, 23 GiB used, 513 GiB / 536 GiB avail; 1023 B/s rd, 0 B/s wr, 1.02k op/s; 48346/289263 objects degraded (16.714%) 2024-09-10T19:22:15.181 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:22:15 smithi183 ceph-mon[41562]: osdmap e268: 8 total, 6 up, 6 in 2024-09-10T19:22:15.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:22:15 smithi183 ceph-mon[41562]: Health check update: Degraded data redundancy: 48346/289263 objects degraded (16.714%), 33 pgs degraded (PG_DEGRADED) 2024-09-10T19:22:15.206 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:22:15.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:22:15.255 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:22:15.255 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:22:15.332 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:22:15 smithi191 ceph-mon[32884]: pgmap v542: 158 pgs: 1 active+clean+scrubbing+deep, 45 active+undersized, 33 active+undersized+degraded, 79 active+clean; 5.9 GiB data, 23 GiB used, 513 GiB / 536 GiB avail; 1023 B/s rd, 0 B/s wr, 1.02k op/s; 48346/289263 objects degraded (16.714%) 2024-09-10T19:22:15.332 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:22:15 smithi191 ceph-mon[32884]: osdmap e268: 8 total, 6 up, 6 in 2024-09-10T19:22:15.332 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:22:15 smithi191 ceph-mon[32884]: Health check update: Degraded data redundancy: 48346/289263 objects degraded (16.714%), 33 pgs degraded (PG_DEGRADED) 2024-09-10T19:22:15.357 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:22:15.377 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:22:15.385 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:22:15.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:22:15.387 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:22:15.488 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:22:15.529 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:22:15.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:22:15.530 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:22:15.630 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:22:15.655 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:22:15.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:22:15.656 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:22:15.756 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:22:15.794 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:22:15.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:22:15.795 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:22:15.895 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:22:15.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:22:15.927 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:22:15.928 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:22:16.028 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:22:16.050 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:22:16.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:22:16.051 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:22:16.151 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:22:16.176 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:22:16.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:22:16.177 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:22:16.277 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:22:16.316 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:22:16.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:22:16.318 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:22:16.418 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:22:16.440 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:22:16.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:22:16.441 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:22:16.542 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:22:16.579 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:22:16.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:22:16.580 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:22:16.680 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:22:16.718 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:22:16.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:22:16.719 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:22:16.822 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:22:16.850 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:22:16.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:22:16.851 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:22:16.883 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-10T19:22:16.952 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:22:16.976 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:22:16.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:22:16.978 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:22:17.079 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:22:17.107 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:22:17.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:22:17.108 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:22:17.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:22:17 smithi183 ceph-mon[29736]: pgmap v544: 158 pgs: 45 active+undersized, 33 active+undersized+degraded, 80 active+clean; 5.5 GiB data, 22 GiB used, 514 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.35k op/s; 45528/271989 objects degraded (16.739%) 2024-09-10T19:22:17.181 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:22:17 smithi183 ceph-mon[41562]: pgmap v544: 158 pgs: 45 active+undersized, 33 active+undersized+degraded, 80 active+clean; 5.5 GiB data, 22 GiB used, 514 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.35k op/s; 45528/271989 objects degraded (16.739%) 2024-09-10T19:22:17.209 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:22:17.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:22:17 smithi191 ceph-mon[32884]: pgmap v544: 158 pgs: 45 active+undersized, 33 active+undersized+degraded, 80 active+clean; 5.5 GiB data, 22 GiB used, 514 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.35k op/s; 45528/271989 objects degraded (16.739%) 2024-09-10T19:22:17.249 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:22:17.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:22:17.250 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:22:17.350 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:22:17.373 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:22:17.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:22:17.374 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:22:17.475 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:22:17.499 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:22:17.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:22:17.500 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:22:17.601 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:22:17.637 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:22:17.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:22:17.641 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:22:17.715 INFO:journalctl@ceph.mgr.x.smithi191.stdout:Sep 10 19:22:17 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-x[34967]: ::ffff:172.21.15.191 - - [10/Sep/2024:19:22:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T19:22:17.742 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:22:17.769 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:22:17.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:22:17.770 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:22:17.871 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:22:17.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:22:17.904 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:22:17.904 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:22:17.978 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-09-10T19:22:17.980 INFO:teuthology.orchestra.run.smithi183.stderr:dumped all 2024-09-10T19:22:18.005 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:22:18.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:22:18.046 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:22:18.047 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:22:18.147 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:22:18.185 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:22:18.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:22:18.186 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:22:18.287 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:22:18.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:22:18.323 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:22:18.323 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:22:18.424 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:22:18.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:22:18.451 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:22:18.452 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:22:18.552 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:22:18.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:22:18.578 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:22:18.578 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:22:18.679 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:22:18.702 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:22:18.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:22:18.703 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:22:18.803 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:22:18.827 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:22:18.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:22:18.828 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:22:18.929 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:22:18.952 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:22:18.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:22:18.953 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:22:19.023 INFO:teuthology.orchestra.run.smithi183.stdout:{"pg_ready":true,"pg_map":{"version":545,"stamp":"2024-09-10T19:22:17.816851+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":5929155648,"num_objects":90663,"num_object_clones":0,"num_object_copies":271989,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":45528,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":90663,"num_whiteouts":0,"num_read":1521,"num_read_kb":1292,"num_write":203244,"num_write_kb":9375444,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":189,"num_bytes_recovered":439119,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":61895,"ondisk_log_size":61895,"up":396,"acting":396,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":474,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":562470912,"kb_used":23417504,"kb_used_data":18655900,"kb_used_omap":9,"kb_used_meta":4761462,"kb_avail":539053408,"statfs":{"total":575970213888,"available":551990689792,"internally_reserved":0,"allocated":19103641600,"data_stored":19101419651,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":10224,"internal_metadata":4875737104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1,0,4,3],"upper_bound":32},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-709033984,"num_objects":-10819,"num_object_clones":0,"num_object_copies":-32457,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-5366,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-10819,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":10819,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002301"},"pg_stats":[{"pgid":"6.1b","version":"0'0","reported_seq":252,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-10T19:22:11.058583+0000","last_change":"2024-09-10T19:21:50.786696+0000","last_active":"2024-09-10T19:22:11.058583+0000","last_peered":"2024-09-10T19:22:11.058583+0000","last_clean":"2024-09-10T19:22:11.058583+0000","last_became_active":"2024-09-10T19:10:14.056187+0000","last_became_peered":"2024-09-10T19:10:14.056187+0000","last_unstale":"2024-09-10T19:22:11.058583+0000","last_undegraded":"2024-09-10T19:22:11.058583+0000","last_fullsized":"2024-09-10T19:22:11.058583+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:21:50.786645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:21:50.786645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:23:12.329028+0000","scrub_duration":0.0043016390000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":150,"reported_epoch":267,"state":"active+undersized","last_fresh":"2024-09-10T19:22:11.063287+0000","last_change":"2024-09-10T19:21:47.983532+0000","last_active":"2024-09-10T19:22:11.063287+0000","last_peered":"2024-09-10T19:22:11.063287+0000","last_clean":"2024-09-10T19:16:23.155764+0000","last_became_active":"2024-09-10T19:21:47.983532+0000","last_became_peered":"2024-09-10T19:21:47.983532+0000","last_unstale":"2024-09-10T19:22:11.063287+0000","last_undegraded":"2024-09-10T19:22:11.063287+0000","last_fullsized":"2024-09-10T19:21:47.980212+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:20:05.748517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:16:38.194472+0000","last_clean_scrub_stamp":"2024-09-10T19:20:05.748517+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.023972094999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":["7","4"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.19","version":"237'27","reported_seq":322,"reported_epoch":267,"state":"active+undersized+degraded","last_fresh":"2024-09-10T19:22:11.058070+0000","last_change":"2024-09-10T19:21:47.993574+0000","last_active":"2024-09-10T19:22:11.058070+0000","last_peered":"2024-09-10T19:22:11.058070+0000","last_clean":"2024-09-10T19:21:43.944958+0000","last_became_active":"2024-09-10T19:21:47.993574+0000","last_became_peered":"2024-09-10T19:21:47.993574+0000","last_unstale":"2024-09-10T19:22:11.058070+0000","last_undegraded":"2024-09-10T19:21:47.976303+0000","last_fullsized":"2024-09-10T19:21:47.975840+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'15","last_scrub_stamp":"2024-09-10T19:19:21.244506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:19:21.244506+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.014405133000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":["3","5"],"object_location_counts":[{"shards":"3,5","objects":7}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":286,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-10T19:22:11.056899+0000","last_change":"2024-09-10T19:21:52.525548+0000","last_active":"2024-09-10T19:22:11.056899+0000","last_peered":"2024-09-10T19:22:11.056899+0000","last_clean":"2024-09-10T19:22:11.056899+0000","last_became_active":"2024-09-10T19:16:24.845111+0000","last_became_peered":"2024-09-10T19:16:24.845111+0000","last_unstale":"2024-09-10T19:22:11.056899+0000","last_undegraded":"2024-09-10T19:22:11.056899+0000","last_fullsized":"2024-09-10T19:22:11.056899+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:21:52.525468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:19:28.498560+0000","last_clean_scrub_stamp":"2024-09-10T19:21:52.525468+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:22:52.914174+0000","scrub_duration":0.0029578730000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,7],"acting":[4,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"59'1","reported_seq":273,"reported_epoch":267,"state":"active+undersized+degraded","last_fresh":"2024-09-10T19:22:11.058558+0000","last_change":"2024-09-10T19:21:47.993529+0000","last_active":"2024-09-10T19:22:11.058558+0000","last_peered":"2024-09-10T19:22:11.058558+0000","last_clean":"2024-09-10T19:21:43.944752+0000","last_became_active":"2024-09-10T19:21:47.993529+0000","last_became_peered":"2024-09-10T19:21:47.993529+0000","last_unstale":"2024-09-10T19:22:11.058558+0000","last_undegraded":"2024-09-10T19:21:47.974676+0000","last_fullsized":"2024-09-10T19:21:47.974527+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'1","last_scrub_stamp":"2024-09-10T19:20:56.540653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:20:56.540653+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:22:20.297346+0000","scrub_duration":0.010998005,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":874,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":["3","5"],"object_location_counts":[{"shards":"3,5","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"237'15","reported_seq":310,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-10T19:22:11.056921+0000","last_change":"2024-09-10T19:21:48.575384+0000","last_active":"2024-09-10T19:22:11.056921+0000","last_peered":"2024-09-10T19:22:11.056921+0000","last_clean":"2024-09-10T19:22:11.056921+0000","last_became_active":"2024-09-10T19:13:54.716117+0000","last_became_peered":"2024-09-10T19:13:54.716117+0000","last_unstale":"2024-09-10T19:22:11.056921+0000","last_undegraded":"2024-09-10T19:22:11.056921+0000","last_fullsized":"2024-09-10T19:22:11.056921+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"237'15","last_scrub_stamp":"2024-09-10T19:21:48.575280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:21:48.575280+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:23:15.773458+0000","scrub_duration":0.0081726820000000006,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,3],"acting":[4,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":218,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-10T19:22:11.062072+0000","last_change":"2024-09-10T19:20:57.313136+0000","last_active":"2024-09-10T19:22:11.062072+0000","last_peered":"2024-09-10T19:22:11.062072+0000","last_clean":"2024-09-10T19:22:11.062072+0000","last_became_active":"2024-09-10T19:16:24.859261+0000","last_became_peered":"2024-09-10T19:16:24.859261+0000","last_unstale":"2024-09-10T19:22:11.062072+0000","last_undegraded":"2024-09-10T19:22:11.062072+0000","last_fullsized":"2024-09-10T19:22:11.062072+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:20:57.313056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:20:57.313056+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:22:15.828453+0000","scrub_duration":0.0079938800000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":274,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-10T19:22:11.057482+0000","last_change":"2024-09-10T19:21:53.485667+0000","last_active":"2024-09-10T19:22:11.057482+0000","last_peered":"2024-09-10T19:22:11.057482+0000","last_clean":"2024-09-10T19:22:11.057482+0000","last_became_active":"2024-09-10T19:13:54.713538+0000","last_became_peered":"2024-09-10T19:13:54.713538+0000","last_unstale":"2024-09-10T19:22:11.057482+0000","last_undegraded":"2024-09-10T19:22:11.057482+0000","last_fullsized":"2024-09-10T19:22:11.057482+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:21:53.485583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:21:53.485583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:23:19.504277+0000","scrub_duration":0.0033055229999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,7],"acting":[4,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":265,"reported_epoch":267,"state":"active+undersized","last_fresh":"2024-09-10T19:22:11.065016+0000","last_change":"2024-09-10T19:21:47.987922+0000","last_active":"2024-09-10T19:22:11.065016+0000","last_peered":"2024-09-10T19:22:11.065016+0000","last_clean":"2024-09-10T19:21:43.955497+0000","last_became_active":"2024-09-10T19:21:47.987922+0000","last_became_peered":"2024-09-10T19:21:47.987922+0000","last_unstale":"2024-09-10T19:22:11.065016+0000","last_undegraded":"2024-09-10T19:22:11.065016+0000","last_fullsized":"2024-09-10T19:21:47.983715+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:20:45.770926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:20:45.770926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:22:14.118127+0000","scrub_duration":0.0040372419999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":["7","6"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1b","version":"237'7","reported_seq":300,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-10T19:22:11.289310+0000","last_change":"2024-09-10T19:20:52.413417+0000","last_active":"2024-09-10T19:22:11.289310+0000","last_peered":"2024-09-10T19:22:11.289310+0000","last_clean":"2024-09-10T19:22:11.289310+0000","last_became_active":"2024-09-10T19:16:24.843708+0000","last_became_peered":"2024-09-10T19:16:24.843708+0000","last_unstale":"2024-09-10T19:22:11.289310+0000","last_undegraded":"2024-09-10T19:22:11.289310+0000","last_fullsized":"2024-09-10T19:22:11.289310+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"237'7","last_scrub_stamp":"2024-09-10T19:20:52.413337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:20:52.413337+0000","objects_scrubbed":2,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:22:17.790560+0000","scrub_duration":0.004717756,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":10,"num_write":8,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":268,"reported_epoch":267,"state":"active+undersized","last_fresh":"2024-09-10T19:22:11.062928+0000","last_change":"2024-09-10T19:21:47.984833+0000","last_active":"2024-09-10T19:22:11.062928+0000","last_peered":"2024-09-10T19:22:11.062928+0000","last_clean":"2024-09-10T19:21:43.956152+0000","last_became_active":"2024-09-10T19:21:47.984833+0000","last_became_peered":"2024-09-10T19:21:47.984833+0000","last_unstale":"2024-09-10T19:22:11.062928+0000","last_undegraded":"2024-09-10T19:22:11.062928+0000","last_fullsized":"2024-09-10T19:21:47.979042+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:20:03.903546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:20:03.903546+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0066943719999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":["7","4"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":276,"reported_epoch":267,"state":"active+undersized","last_fresh":"2024-09-10T19:22:11.062161+0000","last_change":"2024-09-10T19:21:47.983411+0000","last_active":"2024-09-10T19:22:11.062161+0000","last_peered":"2024-09-10T19:22:11.062161+0000","last_clean":"2024-09-10T19:21:43.956084+0000","last_became_active":"2024-09-10T19:21:47.983411+0000","last_became_peered":"2024-09-10T19:21:47.983411+0000","last_unstale":"2024-09-10T19:22:11.062161+0000","last_undegraded":"2024-09-10T19:22:11.062161+0000","last_fullsized":"2024-09-10T19:21:47.973796+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:20:50.336432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:20:50.336432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0077486569999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":["5","3"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":268,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-10T19:22:11.063818+0000","last_change":"2024-09-10T19:21:01.855155+0000","last_active":"2024-09-10T19:22:11.063818+0000","last_peered":"2024-09-10T19:22:11.063818+0000","last_clean":"2024-09-10T19:22:11.063818+0000","last_became_active":"2024-09-10T19:16:24.858800+0000","last_became_peered":"2024-09-10T19:16:24.858800+0000","last_unstale":"2024-09-10T19:22:11.063818+0000","last_undegraded":"2024-09-10T19:22:11.063818+0000","last_fullsized":"2024-09-10T19:22:11.063818+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:21:01.855082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:21:01.855082+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:22:23.949654+0000","scrub_duration":0.008241824,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1a","version":"237'15","reported_seq":290,"reported_epoch":267,"state":"active+undersized+degraded","last_fresh":"2024-09-10T19:22:11.289264+0000","last_change":"2024-09-10T19:21:47.978079+0000","last_active":"2024-09-10T19:22:11.289264+0000","last_peered":"2024-09-10T19:22:11.289264+0000","last_clean":"2024-09-10T19:21:43.946995+0000","last_became_active":"2024-09-10T19:21:47.978079+0000","last_became_peered":"2024-09-10T19:21:47.978079+0000","last_unstale":"2024-09-10T19:22:11.289264+0000","last_undegraded":"2024-09-10T19:21:47.971706+0000","last_fullsized":"2024-09-10T19:21:47.971602+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"237'15","last_scrub_stamp":"2024-09-10T19:21:42.567473+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:21:42.567473+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:22:49.851540+0000","scrub_duration":0.0065782230000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":["4","3"],"object_location_counts":[{"shards":"3,4","objects":4}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":261,"reported_epoch":267,"state":"active+undersized","last_fresh":"2024-09-10T19:22:11.061181+0000","last_change":"2024-09-10T19:21:47.997166+0000","last_active":"2024-09-10T19:22:11.061181+0000","last_peered":"2024-09-10T19:22:11.061181+0000","last_clean":"2024-09-10T19:21:43.951146+0000","last_became_active":"2024-09-10T19:21:47.997166+0000","last_became_peered":"2024-09-10T19:21:47.997166+0000","last_unstale":"2024-09-10T19:22:11.061181+0000","last_undegraded":"2024-09-10T19:22:11.061181+0000","last_fullsized":"2024-09-10T19:21:47.975077+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:21:00.322381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:21:00.322381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.008205258,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":["5","7"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":151,"reported_epoch":267,"state":"active+undersized","last_fresh":"2024-09-10T19:22:11.061847+0000","last_change":"2024-09-10T19:21:47.983632+0000","last_active":"2024-09-10T19:22:11.061847+0000","last_peered":"2024-09-10T19:22:11.061847+0000","last_clean":"2024-09-10T19:16:23.155082+0000","last_became_active":"2024-09-10T19:21:47.983632+0000","last_became_peered":"2024-09-10T19:21:47.983632+0000","last_unstale":"2024-09-10T19:22:11.061847+0000","last_undegraded":"2024-09-10T19:22:11.061847+0000","last_fullsized":"2024-09-10T19:21:47.978395+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:21:05.690833+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:21:05.690833+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:22:20.832256+0000","scrub_duration":0.0032019980000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":["7","4"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.8","version":"74'79","reported_seq":48,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-10T19:22:11.064824+0000","last_change":"2024-09-10T19:21:54.355335+0000","last_active":"2024-09-10T19:22:11.064824+0000","last_peered":"2024-09-10T19:22:11.064824+0000","last_clean":"2024-09-10T19:22:11.064824+0000","last_became_active":"2024-09-10T19:20:30.839050+0000","last_became_peered":"2024-09-10T19:20:30.839050+0000","last_unstale":"2024-09-10T19:22:11.064824+0000","last_undegraded":"2024-09-10T19:22:11.064824+0000","last_fullsized":"2024-09-10T19:22:11.064824+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":4,"last_scrub":"74'79","last_scrub_stamp":"2024-09-10T19:21:54.355151+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:08:00.699315+0000","last_clean_scrub_stamp":"2024-09-10T19:21:54.355151+0000","objects_scrubbed":1,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:23:07.013287+0000","scrub_duration":0.014582322999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.0","version":"268'12569","reported_seq":12628,"reported_epoch":268,"state":"active+undersized+degraded","last_fresh":"2024-09-10T19:22:14.870153+0000","last_change":"2024-09-10T19:21:48.042830+0000","last_active":"2024-09-10T19:22:14.870153+0000","last_peered":"2024-09-10T19:22:14.870153+0000","last_clean":"2024-09-10T19:21:46.973997+0000","last_became_active":"2024-09-10T19:21:48.010936+0000","last_became_peered":"2024-09-10T19:21:48.010936+0000","last_unstale":"2024-09-10T19:22:14.870153+0000","last_undegraded":"2024-09-10T19:21:47.994156+0000","last_fullsized":"2024-09-10T19:21:47.981767+0000","mapping_epoch":263,"log_start":"258'8700","ondisk_log_start":"258'8700","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:19:53.174811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:19:53.174811+0000","last_clean_scrub_stamp":"2024-09-10T19:19:53.174811+0000","objects_scrubbed":0,"log_size":3869,"log_dups_size":0,"ondisk_log_size":3869,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":374669312,"num_objects":5717,"num_object_clones":0,"num_object_copies":17151,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5717,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5717,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12569,"num_write_kb":585152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":["6","5"],"object_location_counts":[{"shards":"5,6","objects":5717}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"237'31","reported_seq":374,"reported_epoch":267,"state":"active+undersized+degraded","last_fresh":"2024-09-10T19:22:11.293618+0000","last_change":"2024-09-10T19:21:47.986499+0000","last_active":"2024-09-10T19:22:11.293618+0000","last_peered":"2024-09-10T19:22:11.293618+0000","last_clean":"2024-09-10T19:21:43.948925+0000","last_became_active":"2024-09-10T19:21:47.986499+0000","last_became_peered":"2024-09-10T19:21:47.986499+0000","last_unstale":"2024-09-10T19:22:11.293618+0000","last_undegraded":"2024-09-10T19:21:47.972336+0000","last_fullsized":"2024-09-10T19:21:47.972236+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"237'31","last_scrub_stamp":"2024-09-10T19:20:43.252273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:20:43.252273+0000","objects_scrubbed":8,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0023272610000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":8,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":50,"num_read_kb":40,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":["4","6"],"object_location_counts":[{"shards":"4,6","objects":8}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":258,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-10T19:22:11.064647+0000","last_change":"2024-09-10T19:20:49.769456+0000","last_active":"2024-09-10T19:22:11.064647+0000","last_peered":"2024-09-10T19:22:11.064647+0000","last_clean":"2024-09-10T19:22:11.064647+0000","last_became_active":"2024-09-10T19:10:07.691119+0000","last_became_peered":"2024-09-10T19:10:07.691119+0000","last_unstale":"2024-09-10T19:22:11.064647+0000","last_undegraded":"2024-09-10T19:22:11.064647+0000","last_fullsized":"2024-09-10T19:22:11.064647+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:20:49.769215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:16:39.716674+0000","last_clean_scrub_stamp":"2024-09-10T19:20:49.769215+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0074469820000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":218,"reported_epoch":267,"state":"active+undersized","last_fresh":"2024-09-10T19:22:11.289791+0000","last_change":"2024-09-10T19:21:48.003034+0000","last_active":"2024-09-10T19:22:11.289791+0000","last_peered":"2024-09-10T19:22:11.289791+0000","last_clean":"2024-09-10T19:21:43.949309+0000","last_became_active":"2024-09-10T19:21:48.003034+0000","last_became_peered":"2024-09-10T19:21:48.003034+0000","last_unstale":"2024-09-10T19:22:11.289791+0000","last_undegraded":"2024-09-10T19:22:11.289791+0000","last_fullsized":"2024-09-10T19:21:47.971412+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:20:54.418649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:20:54.418649+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.006664377,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":["4","6"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":136,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-10T19:22:11.058104+0000","last_change":"2024-09-10T19:20:45.455543+0000","last_active":"2024-09-10T19:22:11.058104+0000","last_peered":"2024-09-10T19:22:11.058104+0000","last_clean":"2024-09-10T19:22:11.058104+0000","last_became_active":"2024-09-10T19:16:24.857843+0000","last_became_peered":"2024-09-10T19:16:24.857843+0000","last_unstale":"2024-09-10T19:22:11.058104+0000","last_undegraded":"2024-09-10T19:22:11.058104+0000","last_fullsized":"2024-09-10T19:22:11.058104+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:20:45.455459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:20:45.455459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0055819499999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.9","version":"74'79","reported_seq":323,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-10T19:22:11.065204+0000","last_change":"2024-09-10T19:21:08.003478+0000","last_active":"2024-09-10T19:22:11.065204+0000","last_peered":"2024-09-10T19:22:11.065204+0000","last_clean":"2024-09-10T19:22:11.065204+0000","last_became_active":"2024-09-10T19:20:30.844108+0000","last_became_peered":"2024-09-10T19:20:30.844108+0000","last_unstale":"2024-09-10T19:22:11.065204+0000","last_undegraded":"2024-09-10T19:22:11.065204+0000","last_fullsized":"2024-09-10T19:22:11.065204+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":4,"last_scrub":"74'79","last_scrub_stamp":"2024-09-10T19:21:08.003401+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:08:00.699315+0000","last_clean_scrub_stamp":"2024-09-10T19:21:08.003401+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:22:17.449863+0000","scrub_duration":0.012490897000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.1","version":"268'12599","reported_seq":12646,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-09-10T19:22:15.098014+0000","last_change":"2024-09-10T19:19:54.198962+0000","last_active":"2024-09-10T19:22:15.098014+0000","last_peered":"2024-09-10T19:22:15.098014+0000","last_clean":"2024-09-10T19:22:15.098014+0000","last_became_active":"2024-09-10T19:19:54.198678+0000","last_became_peered":"2024-09-10T19:19:54.198678+0000","last_unstale":"2024-09-10T19:22:15.098014+0000","last_undegraded":"2024-09-10T19:22:15.098014+0000","last_fullsized":"2024-09-10T19:22:15.098014+0000","mapping_epoch":229,"log_start":"258'8950","ondisk_log_start":"258'8950","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:19:53.174811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:19:53.174811+0000","last_clean_scrub_stamp":"2024-09-10T19:19:53.174811+0000","objects_scrubbed":0,"log_size":3649,"log_dups_size":0,"ondisk_log_size":3649,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":361431040,"num_objects":5515,"num_object_clones":0,"num_object_copies":16545,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5515,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12599,"num_write_kb":579648,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"237'20","reported_seq":298,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-10T19:22:11.289994+0000","last_change":"2024-09-10T19:21:44.614534+0000","last_active":"2024-09-10T19:22:11.289994+0000","last_peered":"2024-09-10T19:22:11.289994+0000","last_clean":"2024-09-10T19:22:11.289994+0000","last_became_active":"2024-09-10T19:10:09.688321+0000","last_became_peered":"2024-09-10T19:10:09.688321+0000","last_unstale":"2024-09-10T19:22:11.289994+0000","last_undegraded":"2024-09-10T19:22:11.289994+0000","last_fullsized":"2024-09-10T19:22:11.289994+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"237'20","last_scrub_stamp":"2024-09-10T19:21:44.614432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:21:44.614432+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:23:02.913848+0000","scrub_duration":0.007970876,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":208,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-10T19:22:11.058962+0000","last_change":"2024-09-10T19:19:57.419605+0000","last_active":"2024-09-10T19:22:11.058962+0000","last_peered":"2024-09-10T19:22:11.058962+0000","last_clean":"2024-09-10T19:22:11.058962+0000","last_became_active":"2024-09-10T19:13:54.731134+0000","last_became_peered":"2024-09-10T19:13:54.731134+0000","last_unstale":"2024-09-10T19:22:11.058962+0000","last_undegraded":"2024-09-10T19:22:11.058962+0000","last_fullsized":"2024-09-10T19:22:11.058962+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:19:57.419553+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:19:57.419553+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001279428,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,5],"acting":[3,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"264'11","reported_seq":284,"reported_epoch":267,"state":"active+undersized+degraded","last_fresh":"2024-09-10T19:22:11.064879+0000","last_change":"2024-09-10T19:21:47.999533+0000","last_active":"2024-09-10T19:22:11.064879+0000","last_peered":"2024-09-10T19:22:11.064879+0000","last_clean":"2024-09-10T19:21:45.875744+0000","last_became_active":"2024-09-10T19:21:47.999533+0000","last_became_peered":"2024-09-10T19:21:47.999533+0000","last_unstale":"2024-09-10T19:22:11.064879+0000","last_undegraded":"2024-09-10T19:21:47.982460+0000","last_fullsized":"2024-09-10T19:21:47.982350+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"155'10","last_scrub_stamp":"2024-09-10T19:20:42.761156+0000","last_deep_scrub":"80'9","last_deep_scrub_stamp":"2024-09-10T19:11:25.156465+0000","last_clean_scrub_stamp":"2024-09-10T19:20:42.761156+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.010904117,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":["7","5"],"object_location_counts":[{"shards":"5,7","objects":1}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":284,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-10T19:22:11.289568+0000","last_change":"2024-09-10T19:21:58.521967+0000","last_active":"2024-09-10T19:22:11.289568+0000","last_peered":"2024-09-10T19:22:11.289568+0000","last_clean":"2024-09-10T19:22:11.289568+0000","last_became_active":"2024-09-10T19:16:24.851829+0000","last_became_peered":"2024-09-10T19:16:24.851829+0000","last_unstale":"2024-09-10T19:22:11.289568+0000","last_undegraded":"2024-09-10T19:22:11.289568+0000","last_fullsized":"2024-09-10T19:22:11.289568+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:21:58.521891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:21:58.521891+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:23:27.912901+0000","scrub_duration":0.0050654960000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,7],"acting":[4,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.a","version":"74'79","reported_seq":43,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-10T19:22:11.060263+0000","last_change":"2024-09-10T19:21:53.326522+0000","last_active":"2024-09-10T19:22:11.060263+0000","last_peered":"2024-09-10T19:22:11.060263+0000","last_clean":"2024-09-10T19:22:11.060263+0000","last_became_active":"2024-09-10T19:20:30.844773+0000","last_became_peered":"2024-09-10T19:20:30.844773+0000","last_unstale":"2024-09-10T19:22:11.060263+0000","last_undegraded":"2024-09-10T19:22:11.060263+0000","last_fullsized":"2024-09-10T19:22:11.060263+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":4,"last_scrub":"74'79","last_scrub_stamp":"2024-09-10T19:21:53.326433+0000","last_deep_scrub":"74'79","last_deep_scrub_stamp":"2024-09-10T19:21:53.326433+0000","last_clean_scrub_stamp":"2024-09-10T19:21:53.326433+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:23:11.639802+0000","scrub_duration":0.0062202960000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,4],"acting":[5,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.2","version":"268'12845","reported_seq":12892,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-09-10T19:22:15.108762+0000","last_change":"2024-09-10T19:19:54.197588+0000","last_active":"2024-09-10T19:22:15.108762+0000","last_peered":"2024-09-10T19:22:15.108762+0000","last_clean":"2024-09-10T19:22:15.108762+0000","last_became_active":"2024-09-10T19:19:54.197290+0000","last_became_peered":"2024-09-10T19:19:54.197290+0000","last_unstale":"2024-09-10T19:22:15.108762+0000","last_undegraded":"2024-09-10T19:22:15.108762+0000","last_fullsized":"2024-09-10T19:22:15.108762+0000","mapping_epoch":229,"log_start":"258'9201","ondisk_log_start":"258'9201","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:19:53.174811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:19:53.174811+0000","last_clean_scrub_stamp":"2024-09-10T19:19:53.174811+0000","objects_scrubbed":0,"log_size":3644,"log_dups_size":0,"ondisk_log_size":3644,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":377946112,"num_objects":5767,"num_object_clones":0,"num_object_copies":17301,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5767,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12845,"num_write_kb":595584,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"237'27","reported_seq":279,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-10T19:22:11.059714+0000","last_change":"2024-09-10T19:21:09.451668+0000","last_active":"2024-09-10T19:22:11.059714+0000","last_peered":"2024-09-10T19:22:11.059714+0000","last_clean":"2024-09-10T19:22:11.059714+0000","last_became_active":"2024-09-10T19:13:54.711634+0000","last_became_peered":"2024-09-10T19:13:54.711634+0000","last_unstale":"2024-09-10T19:22:11.059714+0000","last_undegraded":"2024-09-10T19:22:11.059714+0000","last_fullsized":"2024-09-10T19:22:11.059714+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"237'27","last_scrub_stamp":"2024-09-10T19:21:09.451556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:21:09.451556+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0045196150000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":228,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-10T19:22:11.065394+0000","last_change":"2024-09-10T19:21:49.121081+0000","last_active":"2024-09-10T19:22:11.065394+0000","last_peered":"2024-09-10T19:22:11.065394+0000","last_clean":"2024-09-10T19:22:11.065394+0000","last_became_active":"2024-09-10T19:16:24.857179+0000","last_became_peered":"2024-09-10T19:16:24.857179+0000","last_unstale":"2024-09-10T19:22:11.065394+0000","last_undegraded":"2024-09-10T19:22:11.065394+0000","last_fullsized":"2024-09-10T19:22:11.065394+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:21:49.120987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:21:49.120987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:23:05.286334+0000","scrub_duration":0.0076513420000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,6],"acting":[7,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.e","version":"264'9","reported_seq":407,"reported_epoch":267,"state":"active+undersized+degraded","last_fresh":"2024-09-10T19:22:11.289821+0000","last_change":"2024-09-10T19:21:47.981249+0000","last_active":"2024-09-10T19:22:11.289821+0000","last_peered":"2024-09-10T19:22:11.289821+0000","last_clean":"2024-09-10T19:21:45.875962+0000","last_became_active":"2024-09-10T19:21:47.981249+0000","last_became_peered":"2024-09-10T19:21:47.981249+0000","last_unstale":"2024-09-10T19:22:11.289821+0000","last_undegraded":"2024-09-10T19:21:47.971014+0000","last_fullsized":"2024-09-10T19:21:47.970564+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'8","last_scrub_stamp":"2024-09-10T19:19:17.519517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:19:17.519517+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0049564470000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":["4","5"],"object_location_counts":[{"shards":"4,5","objects":1}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":276,"reported_epoch":267,"state":"active+undersized","last_fresh":"2024-09-10T19:22:11.061810+0000","last_change":"2024-09-10T19:21:47.994832+0000","last_active":"2024-09-10T19:22:11.061810+0000","last_peered":"2024-09-10T19:22:11.061810+0000","last_clean":"2024-09-10T19:21:43.953414+0000","last_became_active":"2024-09-10T19:21:47.994832+0000","last_became_peered":"2024-09-10T19:21:47.994832+0000","last_unstale":"2024-09-10T19:22:11.061810+0000","last_undegraded":"2024-09-10T19:22:11.061810+0000","last_fullsized":"2024-09-10T19:21:47.973238+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:20:49.344566+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:20:49.344566+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0057184669999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":["5","6"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.3","version":"268'12418","reported_seq":12691,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-09-10T19:22:15.139772+0000","last_change":"2024-09-10T19:22:02.040228+0000","last_active":"2024-09-10T19:22:15.139772+0000","last_peered":"2024-09-10T19:22:15.139772+0000","last_clean":"2024-09-10T19:22:15.139772+0000","last_became_active":"2024-09-10T19:19:54.203780+0000","last_became_peered":"2024-09-10T19:19:54.203780+0000","last_unstale":"2024-09-10T19:22:15.139772+0000","last_undegraded":"2024-09-10T19:22:15.139772+0000","last_fullsized":"2024-09-10T19:22:15.139772+0000","mapping_epoch":229,"log_start":"258'8705","ondisk_log_start":"258'8705","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:19:53.174811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:19:53.174811+0000","last_clean_scrub_stamp":"2024-09-10T19:19:53.174811+0000","objects_scrubbed":5486,"log_size":3713,"log_dups_size":0,"ondisk_log_size":3713,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":365428736,"num_objects":5576,"num_object_clones":0,"num_object_copies":16728,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5576,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12418,"num_write_kb":575808,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"237'24","reported_seq":323,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-10T19:22:11.289677+0000","last_change":"2024-09-10T19:21:59.504487+0000","last_active":"2024-09-10T19:22:11.289677+0000","last_peered":"2024-09-10T19:22:11.289677+0000","last_clean":"2024-09-10T19:22:11.289677+0000","last_became_active":"2024-09-10T19:13:54.717022+0000","last_became_peered":"2024-09-10T19:13:54.717022+0000","last_unstale":"2024-09-10T19:22:11.289677+0000","last_undegraded":"2024-09-10T19:22:11.289677+0000","last_fullsized":"2024-09-10T19:22:11.289677+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"237'24","last_scrub_stamp":"2024-09-10T19:21:59.504408+0000","last_deep_scrub":"84'16","last_deep_scrub_stamp":"2024-09-10T19:11:56.157975+0000","last_clean_scrub_stamp":"2024-09-10T19:21:59.504408+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:23:02.276380+0000","scrub_duration":0.0050089319999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,3],"acting":[4,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":272,"reported_epoch":267,"state":"active+undersized","last_fresh":"2024-09-10T19:22:11.062081+0000","last_change":"2024-09-10T19:21:47.983520+0000","last_active":"2024-09-10T19:22:11.062081+0000","last_peered":"2024-09-10T19:22:11.062081+0000","last_clean":"2024-09-10T19:21:43.954695+0000","last_became_active":"2024-09-10T19:21:47.983520+0000","last_became_peered":"2024-09-10T19:21:47.983520+0000","last_unstale":"2024-09-10T19:22:11.062081+0000","last_undegraded":"2024-09-10T19:22:11.062081+0000","last_fullsized":"2024-09-10T19:21:47.978120+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:20:06.874462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:20:06.874462+0000","last_clean_scrub_stamp":"2024-09-10T19:20:06.874462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0052621079999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":["7","4"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":244,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-10T19:22:11.063816+0000","last_change":"2024-09-10T19:19:29.566991+0000","last_active":"2024-09-10T19:22:11.063816+0000","last_peered":"2024-09-10T19:22:11.063816+0000","last_clean":"2024-09-10T19:22:11.063816+0000","last_became_active":"2024-09-10T19:10:11.692215+0000","last_became_peered":"2024-09-10T19:10:11.692215+0000","last_unstale":"2024-09-10T19:22:11.063816+0000","last_undegraded":"2024-09-10T19:22:11.063816+0000","last_fullsized":"2024-09-10T19:22:11.063816+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:19:29.566920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:19:29.566920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0083507189999999995,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":242,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-10T19:22:11.058979+0000","last_change":"2024-09-10T19:19:28.321305+0000","last_active":"2024-09-10T19:22:11.058979+0000","last_peered":"2024-09-10T19:22:11.058979+0000","last_clean":"2024-09-10T19:22:11.058979+0000","last_became_active":"2024-09-10T19:10:14.053736+0000","last_became_peered":"2024-09-10T19:10:14.053736+0000","last_unstale":"2024-09-10T19:22:11.058979+0000","last_undegraded":"2024-09-10T19:22:11.058979+0000","last_fullsized":"2024-09-10T19:22:11.058979+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:19:28.321253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:19:28.321253+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.010088694,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.4","version":"74'79","reported_seq":38,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-10T19:22:11.057631+0000","last_change":"2024-09-10T19:20:30.863699+0000","last_active":"2024-09-10T19:22:11.057631+0000","last_peered":"2024-09-10T19:22:11.057631+0000","last_clean":"2024-09-10T19:22:11.057631+0000","last_became_active":"2024-09-10T19:20:30.838441+0000","last_became_peered":"2024-09-10T19:20:30.838441+0000","last_unstale":"2024-09-10T19:22:11.057631+0000","last_undegraded":"2024-09-10T19:22:11.057631+0000","last_fullsized":"2024-09-10T19:22:11.057631+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":3,"last_scrub":"74'79","last_scrub_stamp":"2024-09-10T19:19:11.070835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:08:00.699315+0000","last_clean_scrub_stamp":"2024-09-10T19:19:11.070835+0000","objects_scrubbed":2,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0030997160000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":20,"num_write":10,"num_write_kb":148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":381350,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.c","version":"268'12754","reported_seq":12801,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-09-10T19:22:15.054154+0000","last_change":"2024-09-10T19:19:54.200758+0000","last_active":"2024-09-10T19:22:15.054154+0000","last_peered":"2024-09-10T19:22:15.054154+0000","last_clean":"2024-09-10T19:22:15.054154+0000","last_became_active":"2024-09-10T19:19:54.200490+0000","last_became_peered":"2024-09-10T19:19:54.200490+0000","last_unstale":"2024-09-10T19:22:15.054154+0000","last_undegraded":"2024-09-10T19:22:15.054154+0000","last_fullsized":"2024-09-10T19:22:15.054154+0000","mapping_epoch":229,"log_start":"258'9155","ondisk_log_start":"258'9155","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:19:53.174811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:19:53.174811+0000","last_clean_scrub_stamp":"2024-09-10T19:19:53.174811+0000","objects_scrubbed":0,"log_size":3599,"log_dups_size":0,"ondisk_log_size":3599,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":372506624,"num_objects":5684,"num_object_clones":0,"num_object_copies":17052,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5684,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12754,"num_write_kb":590016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"265'35","reported_seq":190,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-10T19:22:11.056639+0000","last_change":"2024-09-10T19:21:55.533450+0000","last_active":"2024-09-10T19:22:11.056639+0000","last_peered":"2024-09-10T19:22:11.056639+0000","last_clean":"2024-09-10T19:22:11.056639+0000","last_became_active":"2024-09-10T19:16:24.852821+0000","last_became_peered":"2024-09-10T19:16:24.852821+0000","last_unstale":"2024-09-10T19:22:11.056639+0000","last_undegraded":"2024-09-10T19:22:11.056639+0000","last_fullsized":"2024-09-10T19:22:11.056639+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"265'35","last_scrub_stamp":"2024-09-10T19:21:55.533362+0000","last_deep_scrub":"265'35","last_deep_scrub_stamp":"2024-09-10T19:21:55.533362+0000","last_clean_scrub_stamp":"2024-09-10T19:21:55.533362+0000","objects_scrubbed":7,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:23:17.878002+0000","scrub_duration":0.021732418999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":42,"num_read_kb":35,"num_write":28,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,7],"acting":[4,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":278,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-10T19:22:11.291878+0000","last_change":"2024-09-10T19:21:55.547190+0000","last_active":"2024-09-10T19:22:11.291878+0000","last_peered":"2024-09-10T19:22:11.291878+0000","last_clean":"2024-09-10T19:22:11.291878+0000","last_became_active":"2024-09-10T19:16:24.855861+0000","last_became_peered":"2024-09-10T19:16:24.855861+0000","last_unstale":"2024-09-10T19:22:11.291878+0000","last_undegraded":"2024-09-10T19:22:11.291878+0000","last_fullsized":"2024-09-10T19:22:11.291878+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:21:55.547081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:21:55.547081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:23:22.599506+0000","scrub_duration":0.010304272,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.0","version":"264'11","reported_seq":427,"reported_epoch":267,"state":"active+undersized+degraded","last_fresh":"2024-09-10T19:22:11.057232+0000","last_change":"2024-09-10T19:21:47.975361+0000","last_active":"2024-09-10T19:22:11.057232+0000","last_peered":"2024-09-10T19:22:11.057232+0000","last_clean":"2024-09-10T19:21:45.875415+0000","last_became_active":"2024-09-10T19:21:47.975361+0000","last_became_peered":"2024-09-10T19:21:47.975361+0000","last_unstale":"2024-09-10T19:22:11.057232+0000","last_undegraded":"2024-09-10T19:21:47.972860+0000","last_fullsized":"2024-09-10T19:21:47.972722+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"117'10","last_scrub_stamp":"2024-09-10T19:20:13.462447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:20:13.462447+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0040309090000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":["3","4"],"object_location_counts":[{"shards":"3,4","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":259,"reported_epoch":267,"state":"active+undersized","last_fresh":"2024-09-10T19:22:11.064331+0000","last_change":"2024-09-10T19:21:48.011467+0000","last_active":"2024-09-10T19:22:11.064331+0000","last_peered":"2024-09-10T19:22:11.064331+0000","last_clean":"2024-09-10T19:21:43.955087+0000","last_became_active":"2024-09-10T19:21:48.011467+0000","last_became_peered":"2024-09-10T19:21:48.011467+0000","last_unstale":"2024-09-10T19:22:11.064331+0000","last_undegraded":"2024-09-10T19:22:11.064331+0000","last_fullsized":"2024-09-10T19:21:47.993946+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:19:10.029951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:11:41.073833+0000","last_clean_scrub_stamp":"2024-09-10T19:19:10.029951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.01007037,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":["7","6"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.5","version":"74'79","reported_seq":43,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-10T19:22:11.292967+0000","last_change":"2024-09-10T19:21:45.601945+0000","last_active":"2024-09-10T19:22:11.292967+0000","last_peered":"2024-09-10T19:22:11.292967+0000","last_clean":"2024-09-10T19:22:11.292967+0000","last_became_active":"2024-09-10T19:20:30.838556+0000","last_became_peered":"2024-09-10T19:20:30.838556+0000","last_unstale":"2024-09-10T19:22:11.292967+0000","last_undegraded":"2024-09-10T19:22:11.292967+0000","last_fullsized":"2024-09-10T19:22:11.292967+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":3,"last_scrub":"74'79","last_scrub_stamp":"2024-09-10T19:21:45.601859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:08:00.699315+0000","last_clean_scrub_stamp":"2024-09-10T19:21:45.601859+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:23:03.875761+0000","scrub_duration":0.013353458,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.d","version":"268'15004","reported_seq":15154,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-09-10T19:22:15.148593+0000","last_change":"2024-09-10T19:22:11.294229+0000","last_active":"2024-09-10T19:22:15.148593+0000","last_peered":"2024-09-10T19:22:15.148593+0000","last_clean":"2024-09-10T19:22:15.148593+0000","last_became_active":"2024-09-10T19:19:54.202133+0000","last_became_peered":"2024-09-10T19:19:54.202133+0000","last_unstale":"2024-09-10T19:22:15.148593+0000","last_undegraded":"2024-09-10T19:22:15.148593+0000","last_fullsized":"2024-09-10T19:22:15.148593+0000","mapping_epoch":229,"log_start":"265'11349","ondisk_log_start":"265'11349","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:19:53.174811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:19:53.174811+0000","last_clean_scrub_stamp":"2024-09-10T19:19:53.174811+0000","objects_scrubbed":2352,"log_size":3655,"log_dups_size":0,"ondisk_log_size":3655,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":365166616,"num_objects":5573,"num_object_clones":0,"num_object_copies":16719,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5573,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12677,"num_write_kb":583937,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,4],"acting":[7,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.0","version":"237'30","reported_seq":303,"reported_epoch":267,"state":"active+undersized+degraded","last_fresh":"2024-09-10T19:22:11.057368+0000","last_change":"2024-09-10T19:21:47.983337+0000","last_active":"2024-09-10T19:22:11.057368+0000","last_peered":"2024-09-10T19:22:11.057368+0000","last_clean":"2024-09-10T19:21:43.943548+0000","last_became_active":"2024-09-10T19:21:47.983337+0000","last_became_peered":"2024-09-10T19:21:47.983337+0000","last_unstale":"2024-09-10T19:22:11.057368+0000","last_undegraded":"2024-09-10T19:21:47.975109+0000","last_fullsized":"2024-09-10T19:21:47.974995+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'18","last_scrub_stamp":"2024-09-10T19:19:12.207076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:19:12.207076+0000","objects_scrubbed":8,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.003631156,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":8,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":52,"num_read_kb":40,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":["3","7"],"object_location_counts":[{"shards":"3,7","objects":8}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":218,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-10T19:22:11.296578+0000","last_change":"2024-09-10T19:20:58.271774+0000","last_active":"2024-09-10T19:22:11.296578+0000","last_peered":"2024-09-10T19:22:11.296578+0000","last_clean":"2024-09-10T19:22:11.296578+0000","last_became_active":"2024-09-10T19:13:54.710968+0000","last_became_peered":"2024-09-10T19:13:54.710968+0000","last_unstale":"2024-09-10T19:22:11.296578+0000","last_undegraded":"2024-09-10T19:22:11.296578+0000","last_fullsized":"2024-09-10T19:22:11.296578+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:20:58.271018+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:17:26.374376+0000","last_clean_scrub_stamp":"2024-09-10T19:20:58.271018+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:22:23.077927+0000","scrub_duration":0.014170399,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,3],"acting":[6,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":253,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-10T19:22:11.289618+0000","last_change":"2024-09-10T19:20:55.472162+0000","last_active":"2024-09-10T19:22:11.289618+0000","last_peered":"2024-09-10T19:22:11.289618+0000","last_clean":"2024-09-10T19:22:11.289618+0000","last_became_active":"2024-09-10T19:10:11.698054+0000","last_became_peered":"2024-09-10T19:10:11.698054+0000","last_unstale":"2024-09-10T19:22:11.289618+0000","last_undegraded":"2024-09-10T19:22:11.289618+0000","last_fullsized":"2024-09-10T19:22:11.289618+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:20:55.472063+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:16:30.991635+0000","last_clean_scrub_stamp":"2024-09-10T19:20:55.472063+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:22:16.271595+0000","scrub_duration":0.022379841000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":261,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-10T19:22:11.289169+0000","last_change":"2024-09-10T19:20:45.281458+0000","last_active":"2024-09-10T19:22:11.289169+0000","last_peered":"2024-09-10T19:22:11.289169+0000","last_clean":"2024-09-10T19:22:11.289169+0000","last_became_active":"2024-09-10T19:16:24.848109+0000","last_became_peered":"2024-09-10T19:16:24.848109+0000","last_unstale":"2024-09-10T19:22:11.289169+0000","last_undegraded":"2024-09-10T19:22:11.289169+0000","last_fullsized":"2024-09-10T19:22:11.289169+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:20:45.281383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:20:45.281383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0079035930000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.6","version":"74'79","reported_seq":120,"reported_epoch":267,"state":"active+undersized","last_fresh":"2024-09-10T19:22:11.296402+0000","last_change":"2024-09-10T19:21:47.988079+0000","last_active":"2024-09-10T19:22:11.296402+0000","last_peered":"2024-09-10T19:22:11.296402+0000","last_clean":"2024-09-10T19:21:45.499702+0000","last_became_active":"2024-09-10T19:21:47.988079+0000","last_became_peered":"2024-09-10T19:21:47.988079+0000","last_unstale":"2024-09-10T19:22:11.296402+0000","last_undegraded":"2024-09-10T19:22:11.296402+0000","last_fullsized":"2024-09-10T19:21:47.982186+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":3,"last_scrub":"74'79","last_scrub_stamp":"2024-09-10T19:21:45.499422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:08:00.699315+0000","last_clean_scrub_stamp":"2024-09-10T19:21:45.499422+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:22:57.351711+0000","scrub_duration":0.0024069769999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":["6","3"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.e","version":"268'13013","reported_seq":13067,"reported_epoch":268,"state":"active+undersized+degraded","last_fresh":"2024-09-10T19:22:15.135473+0000","last_change":"2024-09-10T19:21:47.989949+0000","last_active":"2024-09-10T19:22:15.135473+0000","last_peered":"2024-09-10T19:22:15.135473+0000","last_clean":"2024-09-10T19:21:46.952990+0000","last_became_active":"2024-09-10T19:21:47.989949+0000","last_became_peered":"2024-09-10T19:21:47.989949+0000","last_unstale":"2024-09-10T19:22:15.135473+0000","last_undegraded":"2024-09-10T19:21:47.986734+0000","last_fullsized":"2024-09-10T19:21:47.986624+0000","mapping_epoch":263,"log_start":"258'9353","ondisk_log_start":"258'9353","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:19:53.174811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:19:53.174811+0000","last_clean_scrub_stamp":"2024-09-10T19:19:53.174811+0000","objects_scrubbed":0,"log_size":3660,"log_dups_size":0,"ondisk_log_size":3660,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":375062528,"num_objects":5723,"num_object_clones":0,"num_object_copies":17169,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5723,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5723,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13013,"num_write_kb":599552,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":["7","4"],"object_location_counts":[{"shards":"4,7","objects":5723}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"237'37","reported_seq":295,"reported_epoch":267,"state":"active+undersized+degraded","last_fresh":"2024-09-10T19:22:11.061304+0000","last_change":"2024-09-10T19:21:47.998404+0000","last_active":"2024-09-10T19:22:11.061304+0000","last_peered":"2024-09-10T19:22:11.061304+0000","last_clean":"2024-09-10T19:20:16.238914+0000","last_became_active":"2024-09-10T19:21:47.998404+0000","last_became_peered":"2024-09-10T19:21:47.998404+0000","last_unstale":"2024-09-10T19:22:11.061304+0000","last_undegraded":"2024-09-10T19:21:47.977022+0000","last_fullsized":"2024-09-10T19:21:47.976922+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"104'27","last_scrub_stamp":"2024-09-10T19:20:14.754045+0000","last_deep_scrub":"104'27","last_deep_scrub_stamp":"2024-09-10T19:13:02.225944+0000","last_clean_scrub_stamp":"2024-09-10T19:20:14.754045+0000","objects_scrubbed":8,"log_size":37,"log_dups_size":0,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0059406959999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":8,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":53,"num_read_kb":39,"num_write":32,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":["5","7"],"object_location_counts":[{"shards":"5,7","objects":8}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":265,"reported_epoch":267,"state":"active+undersized","last_fresh":"2024-09-10T19:22:11.065023+0000","last_change":"2024-09-10T19:21:47.984335+0000","last_active":"2024-09-10T19:22:11.065023+0000","last_peered":"2024-09-10T19:22:11.065023+0000","last_clean":"2024-09-10T19:21:43.955503+0000","last_became_active":"2024-09-10T19:21:47.984335+0000","last_became_peered":"2024-09-10T19:21:47.984335+0000","last_unstale":"2024-09-10T19:22:11.065023+0000","last_undegraded":"2024-09-10T19:22:11.065023+0000","last_fullsized":"2024-09-10T19:21:47.979308+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:20:43.781741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:16:37.779967+0000","last_clean_scrub_stamp":"2024-09-10T19:20:43.781741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.023161668,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":["7","4"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":264,"reported_epoch":267,"state":"active+undersized","last_fresh":"2024-09-10T19:22:11.294621+0000","last_change":"2024-09-10T19:21:48.002017+0000","last_active":"2024-09-10T19:22:11.294621+0000","last_peered":"2024-09-10T19:22:11.294621+0000","last_clean":"2024-09-10T19:21:43.951362+0000","last_became_active":"2024-09-10T19:21:48.002017+0000","last_became_peered":"2024-09-10T19:21:48.002017+0000","last_unstale":"2024-09-10T19:22:11.294621+0000","last_undegraded":"2024-09-10T19:22:11.294621+0000","last_fullsized":"2024-09-10T19:21:47.979197+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:20:05.474676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:20:05.474676+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0035800150000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":["6","5"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":228,"reported_epoch":267,"state":"active+undersized","last_fresh":"2024-09-10T19:22:11.294758+0000","last_change":"2024-09-10T19:21:47.999030+0000","last_active":"2024-09-10T19:22:11.294758+0000","last_peered":"2024-09-10T19:22:11.294758+0000","last_clean":"2024-09-10T19:21:43.953419+0000","last_became_active":"2024-09-10T19:21:47.999030+0000","last_became_peered":"2024-09-10T19:21:47.999030+0000","last_unstale":"2024-09-10T19:22:11.294758+0000","last_undegraded":"2024-09-10T19:22:11.294758+0000","last_fullsized":"2024-09-10T19:21:47.984292+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:20:06.496950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:20:06.496950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0033688099999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":["6","7"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.7","version":"74'79","reported_seq":113,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-10T19:22:11.291586+0000","last_change":"2024-09-10T19:21:08.340728+0000","last_active":"2024-09-10T19:22:11.291586+0000","last_peered":"2024-09-10T19:22:11.291586+0000","last_clean":"2024-09-10T19:22:11.291586+0000","last_became_active":"2024-09-10T19:20:30.842857+0000","last_became_peered":"2024-09-10T19:20:30.842857+0000","last_unstale":"2024-09-10T19:22:11.291586+0000","last_undegraded":"2024-09-10T19:22:11.291586+0000","last_fullsized":"2024-09-10T19:22:11.291586+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":3,"last_scrub":"74'79","last_scrub_stamp":"2024-09-10T19:21:08.340640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:08:00.699315+0000","last_clean_scrub_stamp":"2024-09-10T19:21:08.340640+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:22:29.386283+0000","scrub_duration":0.012148529,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.f","version":"268'12561","reported_seq":12608,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-09-10T19:22:15.265256+0000","last_change":"2024-09-10T19:19:54.202581+0000","last_active":"2024-09-10T19:22:15.265256+0000","last_peered":"2024-09-10T19:22:15.265256+0000","last_clean":"2024-09-10T19:22:15.265256+0000","last_became_active":"2024-09-10T19:19:54.202323+0000","last_became_peered":"2024-09-10T19:19:54.202323+0000","last_unstale":"2024-09-10T19:22:15.265256+0000","last_undegraded":"2024-09-10T19:22:15.265256+0000","last_fullsized":"2024-09-10T19:22:15.265256+0000","mapping_epoch":229,"log_start":"258'8800","ondisk_log_start":"258'8800","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:19:53.174811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:19:53.174811+0000","last_clean_scrub_stamp":"2024-09-10T19:19:53.174811+0000","objects_scrubbed":0,"log_size":3761,"log_dups_size":0,"ondisk_log_size":3761,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":369950720,"num_objects":5645,"num_object_clones":0,"num_object_copies":16935,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5645,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12561,"num_write_kb":582592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"237'20","reported_seq":236,"reported_epoch":267,"state":"active+undersized+degraded","last_fresh":"2024-09-10T19:22:11.065075+0000","last_change":"2024-09-10T19:21:47.978225+0000","last_active":"2024-09-10T19:22:11.065075+0000","last_peered":"2024-09-10T19:22:11.065075+0000","last_clean":"2024-09-10T19:21:43.955550+0000","last_became_active":"2024-09-10T19:21:47.978225+0000","last_became_peered":"2024-09-10T19:21:47.978225+0000","last_unstale":"2024-09-10T19:22:11.065075+0000","last_undegraded":"2024-09-10T19:21:47.972413+0000","last_fullsized":"2024-09-10T19:21:47.972298+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'10","last_scrub_stamp":"2024-09-10T19:19:26.575769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:19:26.575769+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.022703509,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":24,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":["5","4"],"object_location_counts":[{"shards":"4,5","objects":5}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":100,"reported_epoch":267,"state":"active+undersized","last_fresh":"2024-09-10T19:22:11.063663+0000","last_change":"2024-09-10T19:21:47.983971+0000","last_active":"2024-09-10T19:22:11.063663+0000","last_peered":"2024-09-10T19:22:11.063663+0000","last_clean":"2024-09-10T19:16:23.155488+0000","last_became_active":"2024-09-10T19:21:47.983971+0000","last_became_peered":"2024-09-10T19:21:47.983971+0000","last_unstale":"2024-09-10T19:22:11.063663+0000","last_undegraded":"2024-09-10T19:22:11.063663+0000","last_fullsized":"2024-09-10T19:21:47.979000+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:20:52.748172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:20:52.748172+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0061607199999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":["7","3"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.3","version":"264'10","reported_seq":83,"reported_epoch":267,"state":"active+undersized+degraded","last_fresh":"2024-09-10T19:22:11.295854+0000","last_change":"2024-09-10T19:21:48.008059+0000","last_active":"2024-09-10T19:22:11.295854+0000","last_peered":"2024-09-10T19:22:11.295854+0000","last_clean":"2024-09-10T19:11:53.878374+0000","last_became_active":"2024-09-10T19:21:48.008059+0000","last_became_peered":"2024-09-10T19:21:48.008059+0000","last_unstale":"2024-09-10T19:22:11.295854+0000","last_undegraded":"2024-09-10T19:21:47.990007+0000","last_fullsized":"2024-09-10T19:21:47.989908+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'9","last_scrub_stamp":"2024-09-10T19:21:04.724652+0000","last_deep_scrub":"84'9","last_deep_scrub_stamp":"2024-09-10T19:11:53.879272+0000","last_clean_scrub_stamp":"2024-09-10T19:21:04.724652+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:22:29.505612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":["6","5"],"object_location_counts":[{"shards":"5,6","objects":1}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":134,"reported_epoch":267,"state":"active+undersized","last_fresh":"2024-09-10T19:22:11.056940+0000","last_change":"2024-09-10T19:21:47.995786+0000","last_active":"2024-09-10T19:22:11.056940+0000","last_peered":"2024-09-10T19:22:11.056940+0000","last_clean":"2024-09-10T19:16:23.154999+0000","last_became_active":"2024-09-10T19:21:47.995786+0000","last_became_peered":"2024-09-10T19:21:47.995786+0000","last_unstale":"2024-09-10T19:22:11.056940+0000","last_undegraded":"2024-09-10T19:22:11.056940+0000","last_fullsized":"2024-09-10T19:21:47.973262+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:21:03.766127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:21:03.766127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:22:16.334364+0000","scrub_duration":0.0086405590000000008,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":["3","5"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"74'79","reported_seq":40,"reported_epoch":267,"state":"active+undersized+degraded","last_fresh":"2024-09-10T19:22:11.060283+0000","last_change":"2024-09-10T19:21:47.981812+0000","last_active":"2024-09-10T19:22:11.060283+0000","last_peered":"2024-09-10T19:22:11.060283+0000","last_clean":"2024-09-10T19:21:43.955636+0000","last_became_active":"2024-09-10T19:21:47.981812+0000","last_became_peered":"2024-09-10T19:21:47.981812+0000","last_unstale":"2024-09-10T19:22:11.060283+0000","last_undegraded":"2024-09-10T19:21:47.972771+0000","last_fullsized":"2024-09-10T19:21:47.972653+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":4,"last_scrub":"74'79","last_scrub_stamp":"2024-09-10T19:19:11.070835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:08:00.699315+0000","last_clean_scrub_stamp":"2024-09-10T19:19:11.070835+0000","objects_scrubbed":2,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0030997160000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":11,"num_read_kb":20,"num_write":11,"num_write_kb":148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":53670,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":["5","4"],"object_location_counts":[{"shards":"4,5","objects":1}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.9","version":"268'12582","reported_seq":12816,"reported_epoch":268,"state":"active+undersized+degraded","last_fresh":"2024-09-10T19:22:15.151688+0000","last_change":"2024-09-10T19:21:47.996322+0000","last_active":"2024-09-10T19:22:15.151688+0000","last_peered":"2024-09-10T19:22:15.151688+0000","last_clean":"2024-09-10T19:21:46.936170+0000","last_became_active":"2024-09-10T19:21:47.996322+0000","last_became_peered":"2024-09-10T19:21:47.996322+0000","last_unstale":"2024-09-10T19:22:15.151688+0000","last_undegraded":"2024-09-10T19:21:47.979809+0000","last_fullsized":"2024-09-10T19:21:47.979667+0000","mapping_epoch":263,"log_start":"258'8921","ondisk_log_start":"258'8921","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:19:53.174811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:19:53.174811+0000","last_clean_scrub_stamp":"2024-09-10T19:19:53.174811+0000","objects_scrubbed":4449,"log_size":3661,"log_dups_size":0,"ondisk_log_size":3661,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":363593728,"num_objects":5548,"num_object_clones":0,"num_object_copies":16644,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5548,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5548,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12582,"num_write_kb":580160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":["7","5"],"object_location_counts":[{"shards":"5,7","objects":5548}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.4","version":"237'43","reported_seq":307,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-10T19:22:11.058696+0000","last_change":"2024-09-10T19:20:47.469873+0000","last_active":"2024-09-10T19:22:11.058696+0000","last_peered":"2024-09-10T19:22:11.058696+0000","last_clean":"2024-09-10T19:22:11.058696+0000","last_became_active":"2024-09-10T19:16:24.858154+0000","last_became_peered":"2024-09-10T19:16:24.858154+0000","last_unstale":"2024-09-10T19:22:11.058696+0000","last_undegraded":"2024-09-10T19:22:11.058696+0000","last_fullsized":"2024-09-10T19:22:11.058696+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"237'43","last_scrub_stamp":"2024-09-10T19:20:47.469774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:20:47.469774+0000","objects_scrubbed":10,"log_size":43,"log_dups_size":0,"ondisk_log_size":43,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.002574663,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":65,"num_read_kb":50,"num_write":38,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":358,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":272,"reported_epoch":267,"state":"active+undersized","last_fresh":"2024-09-10T19:22:11.060433+0000","last_change":"2024-09-10T19:21:47.994362+0000","last_active":"2024-09-10T19:22:11.060433+0000","last_peered":"2024-09-10T19:22:11.060433+0000","last_clean":"2024-09-10T19:21:43.955302+0000","last_became_active":"2024-09-10T19:21:47.994362+0000","last_became_peered":"2024-09-10T19:21:47.994362+0000","last_unstale":"2024-09-10T19:22:11.060433+0000","last_undegraded":"2024-09-10T19:22:11.060433+0000","last_fullsized":"2024-09-10T19:21:47.972726+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:20:44.342243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:20:44.342243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0078427040000000007,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":["5","6"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":94,"reported_epoch":267,"state":"active+undersized","last_fresh":"2024-09-10T19:22:11.293153+0000","last_change":"2024-09-10T19:21:47.987054+0000","last_active":"2024-09-10T19:22:11.293153+0000","last_peered":"2024-09-10T19:22:11.293153+0000","last_clean":"2024-09-10T19:12:44.681644+0000","last_became_active":"2024-09-10T19:21:47.987054+0000","last_became_peered":"2024-09-10T19:21:47.987054+0000","last_unstale":"2024-09-10T19:22:11.293153+0000","last_undegraded":"2024-09-10T19:22:11.293153+0000","last_fullsized":"2024-09-10T19:21:47.971744+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:20:56.648680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:20:56.648680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:22:19.017200+0000","scrub_duration":0.001460161,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":["4","6"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":242,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-10T19:22:11.057895+0000","last_change":"2024-09-10T19:21:42.640298+0000","last_active":"2024-09-10T19:22:11.057895+0000","last_peered":"2024-09-10T19:22:11.057895+0000","last_clean":"2024-09-10T19:22:11.057895+0000","last_became_active":"2024-09-10T19:10:13.721744+0000","last_became_peered":"2024-09-10T19:10:13.721744+0000","last_unstale":"2024-09-10T19:22:11.057895+0000","last_undegraded":"2024-09-10T19:22:11.057895+0000","last_fullsized":"2024-09-10T19:22:11.057895+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:21:42.640236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:21:42.640236+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:23:10.323450+0000","scrub_duration":0.0027480289999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"74'79","reported_seq":330,"reported_epoch":267,"state":"active+undersized","last_fresh":"2024-09-10T19:22:11.063446+0000","last_change":"2024-09-10T19:21:47.984511+0000","last_active":"2024-09-10T19:22:11.063446+0000","last_peered":"2024-09-10T19:22:11.063446+0000","last_clean":"2024-09-10T19:21:43.956297+0000","last_became_active":"2024-09-10T19:21:47.984511+0000","last_became_peered":"2024-09-10T19:21:47.984511+0000","last_unstale":"2024-09-10T19:22:11.063446+0000","last_undegraded":"2024-09-10T19:22:11.063446+0000","last_fullsized":"2024-09-10T19:21:47.978708+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":4,"last_scrub":"74'79","last_scrub_stamp":"2024-09-10T19:21:04.941354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:08:00.699315+0000","last_clean_scrub_stamp":"2024-09-10T19:21:04.941354+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0040812920000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":["7","4"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.a","version":"268'12395","reported_seq":12438,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-09-10T19:22:15.243088+0000","last_change":"2024-09-10T19:19:54.202692+0000","last_active":"2024-09-10T19:22:15.243088+0000","last_peered":"2024-09-10T19:22:15.243088+0000","last_clean":"2024-09-10T19:22:15.243088+0000","last_became_active":"2024-09-10T19:19:54.202444+0000","last_became_peered":"2024-09-10T19:19:54.202444+0000","last_unstale":"2024-09-10T19:22:15.243088+0000","last_undegraded":"2024-09-10T19:22:15.243088+0000","last_fullsized":"2024-09-10T19:22:15.243088+0000","mapping_epoch":229,"log_start":"258'8600","ondisk_log_start":"258'8600","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:19:53.174811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:19:53.174811+0000","last_clean_scrub_stamp":"2024-09-10T19:19:53.174811+0000","objects_scrubbed":0,"log_size":3795,"log_dups_size":0,"ondisk_log_size":3795,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":369950720,"num_objects":5645,"num_object_clones":0,"num_object_copies":16935,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5645,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12395,"num_write_kb":577280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.7","version":"237'22","reported_seq":294,"reported_epoch":267,"state":"active+undersized+degraded","last_fresh":"2024-09-10T19:22:11.064003+0000","last_change":"2024-09-10T19:21:47.993567+0000","last_active":"2024-09-10T19:22:11.064003+0000","last_peered":"2024-09-10T19:22:11.064003+0000","last_clean":"2024-09-10T19:21:43.951893+0000","last_became_active":"2024-09-10T19:21:47.993567+0000","last_became_peered":"2024-09-10T19:21:47.993567+0000","last_unstale":"2024-09-10T19:22:11.064003+0000","last_undegraded":"2024-09-10T19:21:47.978296+0000","last_fullsized":"2024-09-10T19:21:47.978196+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"237'22","last_scrub_stamp":"2024-09-10T19:20:51.327668+0000","last_deep_scrub":"82'18","last_deep_scrub_stamp":"2024-09-10T19:15:55.777361+0000","last_clean_scrub_stamp":"2024-09-10T19:20:51.327668+0000","objects_scrubbed":5,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0057025269999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":36,"num_read_kb":25,"num_write":20,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":["5","7"],"object_location_counts":[{"shards":"5,7","objects":5}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.0","version":"159'6","reported_seq":535,"reported_epoch":267,"state":"active+undersized+degraded","last_fresh":"2024-09-10T19:22:14.054118+0000","last_change":"2024-09-10T19:21:48.003078+0000","last_active":"2024-09-10T19:22:14.054118+0000","last_peered":"2024-09-10T19:22:14.054118+0000","last_clean":"2024-09-10T19:21:46.502353+0000","last_became_active":"2024-09-10T19:21:48.003078+0000","last_became_peered":"2024-09-10T19:21:48.003078+0000","last_unstale":"2024-09-10T19:22:14.054118+0000","last_undegraded":"2024-09-10T19:21:47.981464+0000","last_fullsized":"2024-09-10T19:21:47.981365+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"159'6","last_scrub_stamp":"2024-09-10T19:20:13.471084+0000","last_deep_scrub":"159'6","last_deep_scrub_stamp":"2024-09-10T19:16:32.649914+0000","last_clean_scrub_stamp":"2024-09-10T19:20:13.471084+0000","objects_scrubbed":1,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0080331139999999992,"objects_trimmed":0,"snaptrim_duration":0.00034454499999999997,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":353,"num_read_kb":348,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":389,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":["6","5"],"object_location_counts":[{"shards":"5,6","objects":1}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":153,"reported_epoch":267,"state":"active+undersized","last_fresh":"2024-09-10T19:22:11.058156+0000","last_change":"2024-09-10T19:21:47.988021+0000","last_active":"2024-09-10T19:22:11.058156+0000","last_peered":"2024-09-10T19:22:11.058156+0000","last_clean":"2024-09-10T19:21:43.944933+0000","last_became_active":"2024-09-10T19:21:47.988021+0000","last_became_peered":"2024-09-10T19:21:47.988021+0000","last_unstale":"2024-09-10T19:22:11.058156+0000","last_undegraded":"2024-09-10T19:22:11.058156+0000","last_fullsized":"2024-09-10T19:21:47.976328+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:21:01.464138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:21:01.464138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0040256789999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":["3","6"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":146,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-10T19:22:11.062449+0000","last_change":"2024-09-10T19:21:04.357157+0000","last_active":"2024-09-10T19:22:11.062449+0000","last_peered":"2024-09-10T19:22:11.062449+0000","last_clean":"2024-09-10T19:22:11.062449+0000","last_became_active":"2024-09-10T19:16:24.861481+0000","last_became_peered":"2024-09-10T19:16:24.861481+0000","last_unstale":"2024-09-10T19:22:11.062449+0000","last_undegraded":"2024-09-10T19:22:11.062449+0000","last_fullsized":"2024-09-10T19:22:11.062449+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:21:04.357073+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:16:43.958224+0000","last_clean_scrub_stamp":"2024-09-10T19:21:04.357073+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:22:33.580241+0000","scrub_duration":0.0091960070000000008,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,4],"acting":[5,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":255,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-10T19:22:11.063992+0000","last_change":"2024-09-10T19:22:01.208343+0000","last_active":"2024-09-10T19:22:11.063992+0000","last_peered":"2024-09-10T19:22:11.063992+0000","last_clean":"2024-09-10T19:22:11.063992+0000","last_became_active":"2024-09-10T19:10:14.053697+0000","last_became_peered":"2024-09-10T19:10:14.053697+0000","last_unstale":"2024-09-10T19:22:11.063992+0000","last_undegraded":"2024-09-10T19:22:11.063992+0000","last_fullsized":"2024-09-10T19:22:11.063992+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:22:01.208272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:22:01.208272+0000","last_clean_scrub_stamp":"2024-09-10T19:22:01.208272+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:23:06.439276+0000","scrub_duration":0.0042303380000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.3","version":"74'79","reported_seq":113,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-10T19:22:11.289778+0000","last_change":"2024-09-10T19:21:47.511103+0000","last_active":"2024-09-10T19:22:11.289778+0000","last_peered":"2024-09-10T19:22:11.289778+0000","last_clean":"2024-09-10T19:22:11.289778+0000","last_became_active":"2024-09-10T19:20:30.847837+0000","last_became_peered":"2024-09-10T19:20:30.847837+0000","last_unstale":"2024-09-10T19:22:11.289778+0000","last_undegraded":"2024-09-10T19:22:11.289778+0000","last_fullsized":"2024-09-10T19:22:11.289778+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":3,"last_scrub":"74'79","last_scrub_stamp":"2024-09-10T19:21:47.511010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:08:00.699315+0000","last_clean_scrub_stamp":"2024-09-10T19:21:47.511010+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:22:59.355728+0000","scrub_duration":0.0012705069999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,5],"acting":[6,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.b","version":"268'12836","reported_seq":12888,"reported_epoch":268,"state":"active+undersized+degraded","last_fresh":"2024-09-10T19:22:14.859562+0000","last_change":"2024-09-10T19:21:48.003266+0000","last_active":"2024-09-10T19:22:14.859562+0000","last_peered":"2024-09-10T19:22:14.859562+0000","last_clean":"2024-09-10T19:21:46.948308+0000","last_became_active":"2024-09-10T19:21:48.003266+0000","last_became_peered":"2024-09-10T19:21:48.003266+0000","last_unstale":"2024-09-10T19:22:14.859562+0000","last_undegraded":"2024-09-10T19:21:47.987789+0000","last_fullsized":"2024-09-10T19:21:47.985276+0000","mapping_epoch":263,"log_start":"258'8900","ondisk_log_start":"258'8900","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:19:53.174811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:19:53.174811+0000","last_clean_scrub_stamp":"2024-09-10T19:19:53.174811+0000","objects_scrubbed":0,"log_size":3936,"log_dups_size":0,"ondisk_log_size":3936,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":379846656,"num_objects":5796,"num_object_clones":0,"num_object_copies":17388,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5796,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5796,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12836,"num_write_kb":596224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":["6","5"],"object_location_counts":[{"shards":"5,6","objects":5796}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.6","version":"237'30","reported_seq":329,"reported_epoch":267,"state":"active+undersized+degraded","last_fresh":"2024-09-10T19:22:11.063319+0000","last_change":"2024-09-10T19:21:48.008902+0000","last_active":"2024-09-10T19:22:11.063319+0000","last_peered":"2024-09-10T19:22:11.063319+0000","last_clean":"2024-09-10T19:20:16.062994+0000","last_became_active":"2024-09-10T19:21:48.008902+0000","last_became_peered":"2024-09-10T19:21:48.008902+0000","last_unstale":"2024-09-10T19:22:11.063319+0000","last_undegraded":"2024-09-10T19:21:47.986289+0000","last_fullsized":"2024-09-10T19:21:47.986187+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"237'30","last_scrub_stamp":"2024-09-10T19:21:07.733508+0000","last_deep_scrub":"221'18","last_deep_scrub_stamp":"2024-09-10T19:19:29.606748+0000","last_clean_scrub_stamp":"2024-09-10T19:21:07.733508+0000","objects_scrubbed":6,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.020236845,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":["7","5"],"object_location_counts":[{"shards":"5,7","objects":6}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"61'1","reported_seq":24,"reported_epoch":267,"state":"active+undersized+degraded","last_fresh":"2024-09-10T19:22:11.289871+0000","last_change":"2024-09-10T19:21:47.976896+0000","last_active":"2024-09-10T19:22:11.289871+0000","last_peered":"2024-09-10T19:22:11.289871+0000","last_clean":"2024-09-10T19:10:09.679706+0000","last_became_active":"2024-09-10T19:21:47.976896+0000","last_became_peered":"2024-09-10T19:21:47.976896+0000","last_unstale":"2024-09-10T19:22:11.289871+0000","last_undegraded":"2024-09-10T19:21:47.970593+0000","last_fullsized":"2024-09-10T19:21:47.970495+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'1","last_scrub_stamp":"2024-09-10T19:20:40.716730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:20:40.716730+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":["4","3"],"object_location_counts":[{"shards":"3,4","objects":1}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":272,"reported_epoch":267,"state":"active+undersized","last_fresh":"2024-09-10T19:22:11.062200+0000","last_change":"2024-09-10T19:21:47.980682+0000","last_active":"2024-09-10T19:22:11.062200+0000","last_peered":"2024-09-10T19:22:11.062200+0000","last_clean":"2024-09-10T19:21:43.954775+0000","last_became_active":"2024-09-10T19:21:47.980682+0000","last_became_peered":"2024-09-10T19:21:47.980682+0000","last_unstale":"2024-09-10T19:22:11.062200+0000","last_undegraded":"2024-09-10T19:22:11.062200+0000","last_fullsized":"2024-09-10T19:21:47.977818+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:20:09.850369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:20:09.850369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0036440280000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":["7","3"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":278,"reported_epoch":267,"state":"active+undersized","last_fresh":"2024-09-10T19:22:11.063889+0000","last_change":"2024-09-10T19:21:47.980962+0000","last_active":"2024-09-10T19:22:11.063889+0000","last_peered":"2024-09-10T19:22:11.063889+0000","last_clean":"2024-09-10T19:21:43.951818+0000","last_became_active":"2024-09-10T19:21:47.980962+0000","last_became_peered":"2024-09-10T19:21:47.980962+0000","last_unstale":"2024-09-10T19:22:11.063889+0000","last_undegraded":"2024-09-10T19:22:11.063889+0000","last_fullsized":"2024-09-10T19:21:47.972363+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:21:08.210604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:21:08.210604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:22:18.479007+0000","scrub_duration":0.0056788289999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":["5","4"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":208,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-10T19:22:11.061049+0000","last_change":"2024-09-10T19:20:58.275362+0000","last_active":"2024-09-10T19:22:11.061049+0000","last_peered":"2024-09-10T19:22:11.061049+0000","last_clean":"2024-09-10T19:22:11.061049+0000","last_became_active":"2024-09-10T19:13:54.722937+0000","last_became_peered":"2024-09-10T19:13:54.722937+0000","last_unstale":"2024-09-10T19:22:11.061049+0000","last_undegraded":"2024-09-10T19:22:11.061049+0000","last_fullsized":"2024-09-10T19:22:11.061049+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:20:58.275280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:20:58.275280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.005497493,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.8","version":"268'12739","reported_seq":12790,"reported_epoch":268,"state":"active+undersized+degraded","last_fresh":"2024-09-10T19:22:15.114500+0000","last_change":"2024-09-10T19:21:47.996974+0000","last_active":"2024-09-10T19:22:15.114500+0000","last_peered":"2024-09-10T19:22:15.114500+0000","last_clean":"2024-09-10T19:21:46.945944+0000","last_became_active":"2024-09-10T19:21:47.996974+0000","last_became_peered":"2024-09-10T19:21:47.996974+0000","last_unstale":"2024-09-10T19:22:15.114500+0000","last_undegraded":"2024-09-10T19:21:47.973377+0000","last_fullsized":"2024-09-10T19:21:47.973279+0000","mapping_epoch":263,"log_start":"258'9150","ondisk_log_start":"258'9150","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:19:53.174811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:19:53.174811+0000","last_clean_scrub_stamp":"2024-09-10T19:19:53.174811+0000","objects_scrubbed":0,"log_size":3589,"log_dups_size":0,"ondisk_log_size":3589,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":371392512,"num_objects":5667,"num_object_clones":0,"num_object_copies":17001,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5667,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5667,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12739,"num_write_kb":588992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":["3","5"],"object_location_counts":[{"shards":"3,5","objects":5667}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"237'33","reported_seq":313,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-10T19:22:11.291041+0000","last_change":"2024-09-10T19:21:01.348412+0000","last_active":"2024-09-10T19:22:11.291041+0000","last_peered":"2024-09-10T19:22:11.291041+0000","last_clean":"2024-09-10T19:22:11.291041+0000","last_became_active":"2024-09-10T19:10:09.822184+0000","last_became_peered":"2024-09-10T19:10:09.822184+0000","last_unstale":"2024-09-10T19:22:11.291041+0000","last_undegraded":"2024-09-10T19:22:11.291041+0000","last_fullsized":"2024-09-10T19:22:11.291041+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"237'33","last_scrub_stamp":"2024-09-10T19:21:01.348328+0000","last_deep_scrub":"80'21","last_deep_scrub_stamp":"2024-09-10T19:11:39.836144+0000","last_clean_scrub_stamp":"2024-09-10T19:21:01.348328+0000","objects_scrubbed":7,"log_size":33,"log_dups_size":0,"ondisk_log_size":33,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0086338490000000007,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":33,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"63'2","reported_seq":265,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-10T19:22:11.058122+0000","last_change":"2024-09-10T19:21:46.741106+0000","last_active":"2024-09-10T19:22:11.058122+0000","last_peered":"2024-09-10T19:22:11.058122+0000","last_clean":"2024-09-10T19:22:11.058122+0000","last_became_active":"2024-09-10T19:10:07.684784+0000","last_became_peered":"2024-09-10T19:10:07.684784+0000","last_unstale":"2024-09-10T19:22:11.058122+0000","last_undegraded":"2024-09-10T19:22:11.058122+0000","last_fullsized":"2024-09-10T19:22:11.058122+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'2","last_scrub_stamp":"2024-09-10T19:21:46.741014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:21:46.741014+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:23:07.668837+0000","scrub_duration":0.0060747680000000004,"objects_trimmed":0,"snaptrim_duration":0.00075522999999999996,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":268,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-10T19:22:11.061985+0000","last_change":"2024-09-10T19:19:54.487118+0000","last_active":"2024-09-10T19:22:11.061985+0000","last_peered":"2024-09-10T19:22:11.061985+0000","last_clean":"2024-09-10T19:22:11.061985+0000","last_became_active":"2024-09-10T19:16:24.858662+0000","last_became_peered":"2024-09-10T19:16:24.858662+0000","last_unstale":"2024-09-10T19:22:11.061985+0000","last_undegraded":"2024-09-10T19:22:11.061985+0000","last_fullsized":"2024-09-10T19:22:11.061985+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:19:54.487065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:19:54.487065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0013784279999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"74'79","reported_seq":321,"reported_epoch":267,"state":"active+undersized","last_fresh":"2024-09-10T19:22:11.062395+0000","last_change":"2024-09-10T19:21:47.999059+0000","last_active":"2024-09-10T19:22:11.062395+0000","last_peered":"2024-09-10T19:22:11.062395+0000","last_clean":"2024-09-10T19:21:43.954921+0000","last_became_active":"2024-09-10T19:21:47.999059+0000","last_became_peered":"2024-09-10T19:21:47.999059+0000","last_unstale":"2024-09-10T19:22:11.062395+0000","last_undegraded":"2024-09-10T19:22:11.062395+0000","last_fullsized":"2024-09-10T19:21:47.980322+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'79","last_scrub_stamp":"2024-09-10T19:20:59.897158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:08:00.699315+0000","last_clean_scrub_stamp":"2024-09-10T19:20:59.897158+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0081933070000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":["7","6"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":271,"reported_epoch":267,"state":"active+undersized","last_fresh":"2024-09-10T19:22:11.063498+0000","last_change":"2024-09-10T19:21:47.996658+0000","last_active":"2024-09-10T19:22:11.063498+0000","last_peered":"2024-09-10T19:22:11.063498+0000","last_clean":"2024-09-10T19:21:43.955922+0000","last_became_active":"2024-09-10T19:21:47.996658+0000","last_became_peered":"2024-09-10T19:21:47.996658+0000","last_unstale":"2024-09-10T19:22:11.063498+0000","last_undegraded":"2024-09-10T19:22:11.063498+0000","last_fullsized":"2024-09-10T19:21:47.979415+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:20:54.870558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:20:54.870558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0080190729999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":["7","5"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":252,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-10T19:22:11.064215+0000","last_change":"2024-09-10T19:21:47.253156+0000","last_active":"2024-09-10T19:22:11.064215+0000","last_peered":"2024-09-10T19:22:11.064215+0000","last_clean":"2024-09-10T19:22:11.064215+0000","last_became_active":"2024-09-10T19:10:13.720647+0000","last_became_peered":"2024-09-10T19:10:13.720647+0000","last_unstale":"2024-09-10T19:22:11.064215+0000","last_undegraded":"2024-09-10T19:22:11.064215+0000","last_fullsized":"2024-09-10T19:22:11.064215+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:21:47.253100+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:21:47.253100+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:22:49.285115+0000","scrub_duration":0.0012429959999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.7","version":"268'12508","reported_seq":12555,"reported_epoch":268,"state":"active+clean","last_fresh":"2024-09-10T19:22:15.268390+0000","last_change":"2024-09-10T19:19:54.203251+0000","last_active":"2024-09-10T19:22:15.268390+0000","last_peered":"2024-09-10T19:22:15.268390+0000","last_clean":"2024-09-10T19:22:15.268390+0000","last_became_active":"2024-09-10T19:19:54.202928+0000","last_became_peered":"2024-09-10T19:19:54.202928+0000","last_unstale":"2024-09-10T19:22:15.268390+0000","last_undegraded":"2024-09-10T19:22:15.268390+0000","last_fullsized":"2024-09-10T19:22:15.268390+0000","mapping_epoch":229,"log_start":"258'8700","ondisk_log_start":"258'8700","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:19:53.174811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:19:53.174811+0000","last_clean_scrub_stamp":"2024-09-10T19:19:53.174811+0000","objects_scrubbed":0,"log_size":3808,"log_dups_size":0,"ondisk_log_size":3808,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":368705536,"num_objects":5626,"num_object_clones":0,"num_object_copies":16878,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5626,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12508,"num_write_kb":580288,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.a","version":"237'35","reported_seq":354,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-10T19:22:11.294901+0000","last_change":"2024-09-10T19:21:04.316971+0000","last_active":"2024-09-10T19:22:11.294901+0000","last_peered":"2024-09-10T19:22:11.294901+0000","last_clean":"2024-09-10T19:22:11.294901+0000","last_became_active":"2024-09-10T19:13:54.731274+0000","last_became_peered":"2024-09-10T19:13:54.731274+0000","last_unstale":"2024-09-10T19:22:11.294901+0000","last_undegraded":"2024-09-10T19:22:11.294901+0000","last_fullsized":"2024-09-10T19:22:11.294901+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"237'35","last_scrub_stamp":"2024-09-10T19:21:04.316201+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:21:04.316201+0000","objects_scrubbed":9,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:22:15.055269+0000","scrub_duration":0.0082706340000000007,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":56,"num_read_kb":45,"num_write":36,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,5],"acting":[6,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":143,"reported_epoch":267,"state":"active+undersized","last_fresh":"2024-09-10T19:22:11.061763+0000","last_change":"2024-09-10T19:21:47.992941+0000","last_active":"2024-09-10T19:22:11.061763+0000","last_peered":"2024-09-10T19:22:11.061763+0000","last_clean":"2024-09-10T19:21:43.951531+0000","last_became_active":"2024-09-10T19:21:47.992941+0000","last_became_peered":"2024-09-10T19:21:47.992941+0000","last_unstale":"2024-09-10T19:22:11.061763+0000","last_undegraded":"2024-09-10T19:22:11.061763+0000","last_fullsized":"2024-09-10T19:21:47.972931+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:20:06.554210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:20:06.554210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0035560079999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":["5","3"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":18,"reported_epoch":267,"state":"active+undersized","last_fresh":"2024-09-10T19:22:11.061867+0000","last_change":"2024-09-10T19:21:47.998802+0000","last_active":"2024-09-10T19:22:11.061867+0000","last_peered":"2024-09-10T19:22:11.061867+0000","last_clean":"2024-09-10T19:10:10.676876+0000","last_became_active":"2024-09-10T19:21:47.998802+0000","last_became_peered":"2024-09-10T19:21:47.998802+0000","last_unstale":"2024-09-10T19:22:11.061867+0000","last_undegraded":"2024-09-10T19:22:11.061867+0000","last_fullsized":"2024-09-10T19:21:47.977340+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:20:03.641376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:20:03.641376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":["5","7"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":262,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-10T19:22:11.062307+0000","last_change":"2024-09-10T19:21:03.910160+0000","last_active":"2024-09-10T19:22:11.062307+0000","last_peered":"2024-09-10T19:22:11.062307+0000","last_clean":"2024-09-10T19:22:11.062307+0000","last_became_active":"2024-09-10T19:16:24.858172+0000","last_became_peered":"2024-09-10T19:16:24.858172+0000","last_unstale":"2024-09-10T19:22:11.062307+0000","last_undegraded":"2024-09-10T19:22:11.062307+0000","last_fullsized":"2024-09-10T19:22:11.062307+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:21:03.910074+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:21:03.910074+0000","last_clean_scrub_stamp":"2024-09-10T19:21:03.910074+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.017690572000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,6],"acting":[7,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.6","version":"268'12603","reported_seq":12656,"reported_epoch":268,"state":"active+undersized+degraded","last_fresh":"2024-09-10T19:22:15.263226+0000","last_change":"2024-09-10T19:21:48.003207+0000","last_active":"2024-09-10T19:22:15.263226+0000","last_peered":"2024-09-10T19:22:15.263226+0000","last_clean":"2024-09-10T19:21:46.959322+0000","last_became_active":"2024-09-10T19:21:48.003207+0000","last_became_peered":"2024-09-10T19:21:48.003207+0000","last_unstale":"2024-09-10T19:22:15.263226+0000","last_undegraded":"2024-09-10T19:21:47.993011+0000","last_fullsized":"2024-09-10T19:21:47.992906+0000","mapping_epoch":263,"log_start":"258'8800","ondisk_log_start":"258'8800","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:19:53.174811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:19:53.174811+0000","last_clean_scrub_stamp":"2024-09-10T19:19:53.174811+0000","objects_scrubbed":0,"log_size":3803,"log_dups_size":0,"ondisk_log_size":3803,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":369557504,"num_objects":5639,"num_object_clones":0,"num_object_copies":16917,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5639,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5639,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12603,"num_write_kb":583744,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":["5","6"],"object_location_counts":[{"shards":"5,6","objects":5639}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"237'15","reported_seq":284,"reported_epoch":267,"state":"active+undersized+degraded","last_fresh":"2024-09-10T19:22:11.289172+0000","last_change":"2024-09-10T19:21:47.975617+0000","last_active":"2024-09-10T19:22:11.289172+0000","last_peered":"2024-09-10T19:22:11.289172+0000","last_clean":"2024-09-10T19:20:16.225370+0000","last_became_active":"2024-09-10T19:21:47.975617+0000","last_became_peered":"2024-09-10T19:21:47.975617+0000","last_unstale":"2024-09-10T19:22:11.289172+0000","last_undegraded":"2024-09-10T19:21:47.970276+0000","last_fullsized":"2024-09-10T19:21:47.970174+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"237'15","last_scrub_stamp":"2024-09-10T19:21:46.837828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:21:46.837828+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:23:03.505267+0000","scrub_duration":0.001540919,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":["4","3"],"object_location_counts":[{"shards":"3,4","objects":4}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":221,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-10T19:22:11.293681+0000","last_change":"2024-09-10T19:20:40.260748+0000","last_active":"2024-09-10T19:22:11.293681+0000","last_peered":"2024-09-10T19:22:11.293681+0000","last_clean":"2024-09-10T19:22:11.293681+0000","last_became_active":"2024-09-10T19:16:24.848668+0000","last_became_peered":"2024-09-10T19:16:24.848668+0000","last_unstale":"2024-09-10T19:22:11.293681+0000","last_undegraded":"2024-09-10T19:22:11.293681+0000","last_fullsized":"2024-09-10T19:22:11.293681+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:20:40.260536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:11:44.254513+0000","last_clean_scrub_stamp":"2024-09-10T19:20:40.260536+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0068985540000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":156,"reported_epoch":267,"state":"active+undersized","last_fresh":"2024-09-10T19:22:11.057563+0000","last_change":"2024-09-10T19:21:47.977739+0000","last_active":"2024-09-10T19:22:11.057563+0000","last_peered":"2024-09-10T19:22:11.057563+0000","last_clean":"2024-09-10T19:21:43.946899+0000","last_became_active":"2024-09-10T19:21:47.977739+0000","last_became_peered":"2024-09-10T19:21:47.977739+0000","last_unstale":"2024-09-10T19:22:11.057563+0000","last_undegraded":"2024-09-10T19:22:11.057563+0000","last_fullsized":"2024-09-10T19:21:47.969920+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:21:09.367669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:21:09.367669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:22:17.765977+0000","scrub_duration":0.0065456109999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":["4","3"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":134,"reported_epoch":267,"state":"active+undersized","last_fresh":"2024-09-10T19:22:11.064961+0000","last_change":"2024-09-10T19:21:47.990835+0000","last_active":"2024-09-10T19:22:11.064961+0000","last_peered":"2024-09-10T19:22:11.064961+0000","last_clean":"2024-09-10T19:16:23.155267+0000","last_became_active":"2024-09-10T19:21:47.990835+0000","last_became_peered":"2024-09-10T19:21:47.990835+0000","last_unstale":"2024-09-10T19:22:11.064961+0000","last_undegraded":"2024-09-10T19:22:11.064961+0000","last_fullsized":"2024-09-10T19:21:47.981819+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:20:08.746643+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:20:08.746643+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0058765429999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":["7","3"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.5","version":"268'12742","reported_seq":13032,"reported_epoch":268,"state":"active+undersized+degraded","last_fresh":"2024-09-10T19:22:15.268323+0000","last_change":"2024-09-10T19:21:47.994750+0000","last_active":"2024-09-10T19:22:15.268323+0000","last_peered":"2024-09-10T19:22:15.268323+0000","last_clean":"2024-09-10T19:21:46.944630+0000","last_became_active":"2024-09-10T19:21:47.994750+0000","last_became_peered":"2024-09-10T19:21:47.994750+0000","last_unstale":"2024-09-10T19:22:15.268323+0000","last_undegraded":"2024-09-10T19:21:47.973184+0000","last_fullsized":"2024-09-10T19:21:47.972919+0000","mapping_epoch":263,"log_start":"258'8976","ondisk_log_start":"258'8976","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:19:53.174811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:19:53.174811+0000","last_clean_scrub_stamp":"2024-09-10T19:19:53.174811+0000","objects_scrubbed":5823,"log_size":3766,"log_dups_size":0,"ondisk_log_size":3766,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":375128064,"num_objects":5724,"num_object_clones":0,"num_object_copies":17172,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5724,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5724,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12742,"num_write_kb":590912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":["5","7"],"object_location_counts":[{"shards":"5,7","objects":5724}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.8","version":"237'34","reported_seq":306,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-10T19:22:11.065100+0000","last_change":"2024-09-10T19:21:58.440243+0000","last_active":"2024-09-10T19:22:11.065100+0000","last_peered":"2024-09-10T19:22:11.065100+0000","last_clean":"2024-09-10T19:22:11.065100+0000","last_became_active":"2024-09-10T19:10:09.820950+0000","last_became_peered":"2024-09-10T19:10:09.820950+0000","last_unstale":"2024-09-10T19:22:11.065100+0000","last_undegraded":"2024-09-10T19:22:11.065100+0000","last_fullsized":"2024-09-10T19:22:11.065100+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"237'34","last_scrub_stamp":"2024-09-10T19:21:58.440165+0000","last_deep_scrub":"220'22","last_deep_scrub_stamp":"2024-09-10T19:19:24.597866+0000","last_clean_scrub_stamp":"2024-09-10T19:21:58.440165+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:23:13.112083+0000","scrub_duration":0.0050099799999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":136,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-10T19:22:11.058205+0000","last_change":"2024-09-10T19:20:08.533315+0000","last_active":"2024-09-10T19:22:11.058205+0000","last_peered":"2024-09-10T19:22:11.058205+0000","last_clean":"2024-09-10T19:22:11.058205+0000","last_became_active":"2024-09-10T19:16:24.857253+0000","last_became_peered":"2024-09-10T19:16:24.857253+0000","last_unstale":"2024-09-10T19:22:11.058205+0000","last_undegraded":"2024-09-10T19:22:11.058205+0000","last_fullsized":"2024-09-10T19:22:11.058205+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:20:08.533267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:20:08.533267+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0023880849999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.9","version":"80'9","reported_seq":408,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-10T19:22:11.065126+0000","last_change":"2024-09-10T19:20:57.882587+0000","last_active":"2024-09-10T19:22:11.065126+0000","last_peered":"2024-09-10T19:22:11.065126+0000","last_clean":"2024-09-10T19:22:11.065126+0000","last_became_active":"2024-09-10T19:10:11.702878+0000","last_became_peered":"2024-09-10T19:10:11.702878+0000","last_unstale":"2024-09-10T19:22:11.065126+0000","last_undegraded":"2024-09-10T19:22:11.065126+0000","last_fullsized":"2024-09-10T19:22:11.065126+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"80'9","last_scrub_stamp":"2024-09-10T19:20:57.882496+0000","last_deep_scrub":"80'9","last_deep_scrub_stamp":"2024-09-10T19:20:57.882496+0000","last_clean_scrub_stamp":"2024-09-10T19:20:57.882496+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.025045438999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":249,"reported_epoch":267,"state":"active+undersized","last_fresh":"2024-09-10T19:22:11.065132+0000","last_change":"2024-09-10T19:21:47.994405+0000","last_active":"2024-09-10T19:22:11.065132+0000","last_peered":"2024-09-10T19:22:11.065132+0000","last_clean":"2024-09-10T19:21:43.955746+0000","last_became_active":"2024-09-10T19:21:47.994405+0000","last_became_peered":"2024-09-10T19:21:47.994405+0000","last_unstale":"2024-09-10T19:22:11.065132+0000","last_undegraded":"2024-09-10T19:22:11.065132+0000","last_fullsized":"2024-09-10T19:21:47.974191+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:19:15.539488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:19:15.539488+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.033498668000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":["5","6"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.4","version":"268'12666","reported_seq":12720,"reported_epoch":268,"state":"active+undersized+degraded","last_fresh":"2024-09-10T19:22:14.889448+0000","last_change":"2024-09-10T19:21:47.976009+0000","last_active":"2024-09-10T19:22:14.889448+0000","last_peered":"2024-09-10T19:22:14.889448+0000","last_clean":"2024-09-10T19:21:46.956094+0000","last_became_active":"2024-09-10T19:21:47.976009+0000","last_became_peered":"2024-09-10T19:21:47.976009+0000","last_unstale":"2024-09-10T19:22:14.889448+0000","last_undegraded":"2024-09-10T19:21:47.970032+0000","last_fullsized":"2024-09-10T19:21:47.969894+0000","mapping_epoch":263,"log_start":"256'8400","ondisk_log_start":"256'8400","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:19:53.174811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:19:53.174811+0000","last_clean_scrub_stamp":"2024-09-10T19:19:53.174811+0000","objects_scrubbed":0,"log_size":4266,"log_dups_size":0,"ondisk_log_size":4266,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":368443392,"num_objects":5622,"num_object_clones":0,"num_object_copies":16866,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5622,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5622,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12666,"num_write_kb":585216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":["4","3"],"object_location_counts":[{"shards":"3,4","objects":5622}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.9","version":"237'18","reported_seq":319,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-10T19:22:11.289764+0000","last_change":"2024-09-10T19:22:00.500856+0000","last_active":"2024-09-10T19:22:11.289764+0000","last_peered":"2024-09-10T19:22:11.289764+0000","last_clean":"2024-09-10T19:22:11.289764+0000","last_became_active":"2024-09-10T19:13:54.718152+0000","last_became_peered":"2024-09-10T19:13:54.718152+0000","last_unstale":"2024-09-10T19:22:11.289764+0000","last_undegraded":"2024-09-10T19:22:11.289764+0000","last_fullsized":"2024-09-10T19:22:11.289764+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"237'18","last_scrub_stamp":"2024-09-10T19:22:00.500765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:22:00.500765+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:23:22.845668+0000","scrub_duration":0.0093780900000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.f","version":"59'2","reported_seq":280,"reported_epoch":267,"state":"active+undersized+degraded","last_fresh":"2024-09-10T19:22:11.291065+0000","last_change":"2024-09-10T19:21:47.986039+0000","last_active":"2024-09-10T19:22:11.291065+0000","last_peered":"2024-09-10T19:22:11.291065+0000","last_clean":"2024-09-10T19:21:43.952266+0000","last_became_active":"2024-09-10T19:21:47.986039+0000","last_became_peered":"2024-09-10T19:21:47.986039+0000","last_unstale":"2024-09-10T19:22:11.291065+0000","last_undegraded":"2024-09-10T19:21:47.971581+0000","last_fullsized":"2024-09-10T19:21:47.971480+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'2","last_scrub_stamp":"2024-09-10T19:20:56.467569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:20:56.467569+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.013126802,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":14,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":["4","7"],"object_location_counts":[{"shards":"4,7","objects":2}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":98,"reported_epoch":267,"state":"active+undersized","last_fresh":"2024-09-10T19:22:11.065191+0000","last_change":"2024-09-10T19:21:47.983793+0000","last_active":"2024-09-10T19:22:11.065191+0000","last_peered":"2024-09-10T19:22:11.065191+0000","last_clean":"2024-09-10T19:12:44.685930+0000","last_became_active":"2024-09-10T19:21:47.983793+0000","last_became_peered":"2024-09-10T19:21:47.983793+0000","last_unstale":"2024-09-10T19:22:11.065191+0000","last_undegraded":"2024-09-10T19:22:11.065191+0000","last_fullsized":"2024-09-10T19:21:47.972418+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:21:45.880289+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:11:46.587846+0000","last_clean_scrub_stamp":"2024-09-10T19:21:45.880289+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:23:15.161241+0000","scrub_duration":0.001267142,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":["5","4"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":259,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-10T19:22:11.058667+0000","last_change":"2024-09-10T19:20:09.576625+0000","last_active":"2024-09-10T19:22:11.058667+0000","last_peered":"2024-09-10T19:22:11.058667+0000","last_clean":"2024-09-10T19:22:11.058667+0000","last_became_active":"2024-09-10T19:13:54.711804+0000","last_became_peered":"2024-09-10T19:13:54.711804+0000","last_unstale":"2024-09-10T19:22:11.058667+0000","last_undegraded":"2024-09-10T19:22:11.058667+0000","last_fullsized":"2024-09-10T19:22:11.058667+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:20:09.576542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:20:09.576542+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0030134599999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.16","version":"237'15","reported_seq":267,"reported_epoch":267,"state":"active+undersized+degraded","last_fresh":"2024-09-10T19:22:11.058079+0000","last_change":"2024-09-10T19:21:47.988985+0000","last_active":"2024-09-10T19:22:11.058079+0000","last_peered":"2024-09-10T19:22:11.058079+0000","last_clean":"2024-09-10T19:20:15.530776+0000","last_became_active":"2024-09-10T19:21:47.988985+0000","last_became_peered":"2024-09-10T19:21:47.988985+0000","last_unstale":"2024-09-10T19:22:11.058079+0000","last_undegraded":"2024-09-10T19:21:47.974466+0000","last_fullsized":"2024-09-10T19:21:47.974359+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"237'15","last_scrub_stamp":"2024-09-10T19:21:01.691067+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:21:01.691067+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:22:24.835286+0000","scrub_duration":0.0065392059999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":["3","7"],"object_location_counts":[{"shards":"3,7","objects":4}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":102,"reported_epoch":267,"state":"active+undersized","last_fresh":"2024-09-10T19:22:11.056806+0000","last_change":"2024-09-10T19:21:47.995029+0000","last_active":"2024-09-10T19:22:11.056806+0000","last_peered":"2024-09-10T19:22:11.056806+0000","last_clean":"2024-09-10T19:12:44.686063+0000","last_became_active":"2024-09-10T19:21:47.995029+0000","last_became_peered":"2024-09-10T19:21:47.995029+0000","last_unstale":"2024-09-10T19:22:11.056806+0000","last_undegraded":"2024-09-10T19:22:11.056806+0000","last_fullsized":"2024-09-10T19:21:47.969981+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:20:57.698558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:11:41.458523+0000","last_clean_scrub_stamp":"2024-09-10T19:20:57.698558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:22:27.559149+0000","scrub_duration":0.0012499830000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":["4","6"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":268,"reported_epoch":267,"state":"active+undersized","last_fresh":"2024-09-10T19:22:11.059593+0000","last_change":"2024-09-10T19:21:47.984694+0000","last_active":"2024-09-10T19:22:11.059593+0000","last_peered":"2024-09-10T19:22:11.059593+0000","last_clean":"2024-09-10T19:21:43.944868+0000","last_became_active":"2024-09-10T19:21:47.984694+0000","last_became_peered":"2024-09-10T19:21:47.984694+0000","last_unstale":"2024-09-10T19:22:11.059593+0000","last_undegraded":"2024-09-10T19:22:11.059593+0000","last_fullsized":"2024-09-10T19:21:47.974535+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:21:43.633645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:21:43.633645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:22:48.889527+0000","scrub_duration":0.001568039,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":["3","7"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"66'1","reported_seq":156,"reported_epoch":267,"state":"active+undersized+degraded","last_fresh":"2024-09-10T19:22:11.289640+0000","last_change":"2024-09-10T19:21:48.003365+0000","last_active":"2024-09-10T19:22:11.289640+0000","last_peered":"2024-09-10T19:22:11.289640+0000","last_clean":"2024-09-10T19:21:43.949315+0000","last_became_active":"2024-09-10T19:21:48.003365+0000","last_became_peered":"2024-09-10T19:21:48.003365+0000","last_unstale":"2024-09-10T19:22:11.289640+0000","last_undegraded":"2024-09-10T19:21:47.971226+0000","last_fullsized":"2024-09-10T19:21:47.970632+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'1","last_scrub_stamp":"2024-09-10T19:20:07.338085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:20:07.338085+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0056882089999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":13,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":["4","7"],"object_location_counts":[{"shards":"4,7","objects":1}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":281,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-10T19:22:11.295074+0000","last_change":"2024-09-10T19:21:54.567071+0000","last_active":"2024-09-10T19:22:11.295074+0000","last_peered":"2024-09-10T19:22:11.295074+0000","last_clean":"2024-09-10T19:22:11.295074+0000","last_became_active":"2024-09-10T19:13:54.724027+0000","last_became_peered":"2024-09-10T19:13:54.724027+0000","last_unstale":"2024-09-10T19:22:11.295074+0000","last_undegraded":"2024-09-10T19:22:11.295074+0000","last_fullsized":"2024-09-10T19:22:11.295074+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:21:54.566800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:21:54.566800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:23:15.940759+0000","scrub_duration":0.010619585000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,5],"acting":[6,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"237'12","reported_seq":297,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-10T19:22:11.059901+0000","last_change":"2024-09-10T19:19:25.266265+0000","last_active":"2024-09-10T19:22:11.059901+0000","last_peered":"2024-09-10T19:22:11.059901+0000","last_clean":"2024-09-10T19:22:11.059901+0000","last_became_active":"2024-09-10T19:16:24.863398+0000","last_became_peered":"2024-09-10T19:16:24.863398+0000","last_unstale":"2024-09-10T19:22:11.059901+0000","last_undegraded":"2024-09-10T19:22:11.059901+0000","last_fullsized":"2024-09-10T19:22:11.059901+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'6","last_scrub_stamp":"2024-09-10T19:19:25.266162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:19:25.266162+0000","objects_scrubbed":3,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.013851005,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":18,"num_read_kb":15,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":261,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-10T19:22:11.062509+0000","last_change":"2024-09-10T19:20:00.494694+0000","last_active":"2024-09-10T19:22:11.062509+0000","last_peered":"2024-09-10T19:22:11.062509+0000","last_clean":"2024-09-10T19:22:11.062509+0000","last_became_active":"2024-09-10T19:13:54.729917+0000","last_became_peered":"2024-09-10T19:13:54.729917+0000","last_unstale":"2024-09-10T19:22:11.062509+0000","last_undegraded":"2024-09-10T19:22:11.062509+0000","last_fullsized":"2024-09-10T19:22:11.062509+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:20:00.494610+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:11:23.264793+0000","last_clean_scrub_stamp":"2024-09-10T19:20:00.494610+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001639585,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":252,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-10T19:22:11.063920+0000","last_change":"2024-09-10T19:20:50.811637+0000","last_active":"2024-09-10T19:22:11.063920+0000","last_peered":"2024-09-10T19:22:11.063920+0000","last_clean":"2024-09-10T19:22:11.063920+0000","last_became_active":"2024-09-10T19:10:14.054360+0000","last_became_peered":"2024-09-10T19:10:14.054360+0000","last_unstale":"2024-09-10T19:22:11.063920+0000","last_undegraded":"2024-09-10T19:22:11.063920+0000","last_fullsized":"2024-09-10T19:22:11.063920+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:20:50.811559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:20:50.811559+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.014254578,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"237'25","reported_seq":332,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-10T19:22:11.058612+0000","last_change":"2024-09-10T19:20:02.458352+0000","last_active":"2024-09-10T19:22:11.058612+0000","last_peered":"2024-09-10T19:22:11.058612+0000","last_clean":"2024-09-10T19:22:11.058612+0000","last_became_active":"2024-09-10T19:16:24.850702+0000","last_became_peered":"2024-09-10T19:16:24.850702+0000","last_unstale":"2024-09-10T19:22:11.058612+0000","last_undegraded":"2024-09-10T19:22:11.058612+0000","last_fullsized":"2024-09-10T19:22:11.058612+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"80'15","last_scrub_stamp":"2024-09-10T19:20:02.458275+0000","last_deep_scrub":"80'15","last_deep_scrub_stamp":"2024-09-10T19:11:41.719310+0000","last_clean_scrub_stamp":"2024-09-10T19:20:02.458275+0000","objects_scrubbed":5,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0054307269999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,5],"acting":[3,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":258,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-10T19:22:11.060531+0000","last_change":"2024-09-10T19:21:48.298990+0000","last_active":"2024-09-10T19:22:11.060531+0000","last_peered":"2024-09-10T19:22:11.060531+0000","last_clean":"2024-09-10T19:22:11.060531+0000","last_became_active":"2024-09-10T19:10:07.685870+0000","last_became_peered":"2024-09-10T19:10:07.685870+0000","last_unstale":"2024-09-10T19:22:11.060531+0000","last_undegraded":"2024-09-10T19:22:11.060531+0000","last_fullsized":"2024-09-10T19:22:11.060531+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:21:48.298913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:21:48.298913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:22:55.545159+0000","scrub_duration":0.0075260040000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"264'11","reported_seq":427,"reported_epoch":267,"state":"active+undersized+degraded","last_fresh":"2024-09-10T19:22:11.062089+0000","last_change":"2024-09-10T19:21:47.993917+0000","last_active":"2024-09-10T19:22:11.062089+0000","last_peered":"2024-09-10T19:22:11.062089+0000","last_clean":"2024-09-10T19:21:45.876612+0000","last_became_active":"2024-09-10T19:21:47.993917+0000","last_became_peered":"2024-09-10T19:21:47.993917+0000","last_unstale":"2024-09-10T19:22:11.062089+0000","last_undegraded":"2024-09-10T19:21:47.974032+0000","last_fullsized":"2024-09-10T19:21:47.973921+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"117'10","last_scrub_stamp":"2024-09-10T19:20:55.377819+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:20:55.377819+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:22:19.741603+0000","scrub_duration":0.0063951320000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":["5","3"],"object_location_counts":[{"shards":"3,5","objects":1}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":14,"reported_epoch":267,"state":"active+undersized","last_fresh":"2024-09-10T19:22:11.063556+0000","last_change":"2024-09-10T19:21:47.989023+0000","last_active":"2024-09-10T19:22:11.063556+0000","last_peered":"2024-09-10T19:22:11.063556+0000","last_clean":"2024-09-10T19:10:12.693598+0000","last_became_active":"2024-09-10T19:21:47.989023+0000","last_became_peered":"2024-09-10T19:21:47.989023+0000","last_unstale":"2024-09-10T19:22:11.063556+0000","last_undegraded":"2024-09-10T19:22:11.063556+0000","last_fullsized":"2024-09-10T19:21:47.982009+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:20:46.625193+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:20:46.625193+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":["7","3"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.15","version":"237'15","reported_seq":321,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-10T19:22:11.062220+0000","last_change":"2024-09-10T19:21:57.580717+0000","last_active":"2024-09-10T19:22:11.062220+0000","last_peered":"2024-09-10T19:22:11.062220+0000","last_clean":"2024-09-10T19:22:11.062220+0000","last_became_active":"2024-09-10T19:16:24.859740+0000","last_became_peered":"2024-09-10T19:16:24.859740+0000","last_unstale":"2024-09-10T19:22:11.062220+0000","last_undegraded":"2024-09-10T19:22:11.062220+0000","last_fullsized":"2024-09-10T19:22:11.062220+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"237'15","last_scrub_stamp":"2024-09-10T19:21:57.580644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:21:57.580644+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:23:03.202902+0000","scrub_duration":0.11368046,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":14,"reported_epoch":267,"state":"active+undersized","last_fresh":"2024-09-10T19:22:11.289537+0000","last_change":"2024-09-10T19:21:47.975293+0000","last_active":"2024-09-10T19:22:11.289537+0000","last_peered":"2024-09-10T19:22:11.289537+0000","last_clean":"2024-09-10T19:10:06.664709+0000","last_became_active":"2024-09-10T19:21:47.975293+0000","last_became_peered":"2024-09-10T19:21:47.975293+0000","last_unstale":"2024-09-10T19:22:11.289537+0000","last_undegraded":"2024-09-10T19:22:11.289537+0000","last_fullsized":"2024-09-10T19:21:47.970537+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:20:48.709474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:20:48.709474+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":["4","3"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"155'9","reported_seq":421,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-10T19:22:11.058939+0000","last_change":"2024-09-10T19:22:00.716614+0000","last_active":"2024-09-10T19:22:11.058939+0000","last_peered":"2024-09-10T19:22:11.058939+0000","last_clean":"2024-09-10T19:22:11.058939+0000","last_became_active":"2024-09-10T19:16:24.850850+0000","last_became_peered":"2024-09-10T19:16:24.850850+0000","last_unstale":"2024-09-10T19:22:11.058939+0000","last_undegraded":"2024-09-10T19:22:11.058939+0000","last_fullsized":"2024-09-10T19:22:11.058939+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"155'9","last_scrub_stamp":"2024-09-10T19:22:00.716555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:22:00.716555+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:23:29.355333+0000","scrub_duration":0.0020496289999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,4],"acting":[3,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":261,"reported_epoch":267,"state":"active+undersized","last_fresh":"2024-09-10T19:22:11.290290+0000","last_change":"2024-09-10T19:21:47.976028+0000","last_active":"2024-09-10T19:22:11.290290+0000","last_peered":"2024-09-10T19:22:11.290290+0000","last_clean":"2024-09-10T19:21:43.951991+0000","last_became_active":"2024-09-10T19:21:47.976028+0000","last_became_peered":"2024-09-10T19:21:47.976028+0000","last_unstale":"2024-09-10T19:22:11.290290+0000","last_undegraded":"2024-09-10T19:22:11.290290+0000","last_fullsized":"2024-09-10T19:21:47.970502+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:19:12.504402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:19:12.504402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0034925469999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":["4","5"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.12","version":"237'19","reported_seq":317,"reported_epoch":267,"state":"active+undersized+degraded","last_fresh":"2024-09-10T19:22:11.294830+0000","last_change":"2024-09-10T19:21:48.003068+0000","last_active":"2024-09-10T19:22:11.294830+0000","last_peered":"2024-09-10T19:22:11.294830+0000","last_clean":"2024-09-10T19:21:43.951440+0000","last_became_active":"2024-09-10T19:21:48.003068+0000","last_became_peered":"2024-09-10T19:21:48.003068+0000","last_unstale":"2024-09-10T19:22:11.294830+0000","last_undegraded":"2024-09-10T19:21:47.979935+0000","last_fullsized":"2024-09-10T19:21:47.979836+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"237'19","last_scrub_stamp":"2024-09-10T19:21:07.378184+0000","last_deep_scrub":"79'13","last_deep_scrub_stamp":"2024-09-10T19:11:22.072400+0000","last_clean_scrub_stamp":"2024-09-10T19:21:07.378184+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:22:35.491858+0000","scrub_duration":0.004147263,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":["6","5"],"object_location_counts":[{"shards":"5,6","objects":4}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.14","version":"59'1","reported_seq":265,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-10T19:22:11.290855+0000","last_change":"2024-09-10T19:20:47.222380+0000","last_active":"2024-09-10T19:22:11.290855+0000","last_peered":"2024-09-10T19:22:11.290855+0000","last_clean":"2024-09-10T19:22:11.290855+0000","last_became_active":"2024-09-10T19:10:07.683263+0000","last_became_peered":"2024-09-10T19:10:07.683263+0000","last_unstale":"2024-09-10T19:22:11.290855+0000","last_undegraded":"2024-09-10T19:22:11.290855+0000","last_fullsized":"2024-09-10T19:22:11.290855+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'1","last_scrub_stamp":"2024-09-10T19:20:47.222274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:20:47.222274+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0025908440000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":362,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":268,"reported_epoch":267,"state":"active+undersized","last_fresh":"2024-09-10T19:22:11.058751+0000","last_change":"2024-09-10T19:21:47.996741+0000","last_active":"2024-09-10T19:22:11.058751+0000","last_peered":"2024-09-10T19:22:11.058751+0000","last_clean":"2024-09-10T19:21:43.944122+0000","last_became_active":"2024-09-10T19:21:47.996741+0000","last_became_peered":"2024-09-10T19:21:47.996741+0000","last_unstale":"2024-09-10T19:22:11.058751+0000","last_undegraded":"2024-09-10T19:22:11.058751+0000","last_fullsized":"2024-09-10T19:21:47.974561+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:20:50.491473+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:20:50.491473+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:22:16.418021+0000","scrub_duration":0.0066726329999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":["3","5"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":92,"reported_epoch":267,"state":"active+undersized","last_fresh":"2024-09-10T19:22:11.064980+0000","last_change":"2024-09-10T19:21:48.003432+0000","last_active":"2024-09-10T19:22:11.064980+0000","last_peered":"2024-09-10T19:22:11.064980+0000","last_clean":"2024-09-10T19:12:44.680563+0000","last_became_active":"2024-09-10T19:21:48.003432+0000","last_became_peered":"2024-09-10T19:21:48.003432+0000","last_unstale":"2024-09-10T19:22:11.064980+0000","last_undegraded":"2024-09-10T19:22:11.064980+0000","last_fullsized":"2024-09-10T19:21:47.974956+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:21:09.714677+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:21:09.714677+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001865058,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":["5","6"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.13","version":"237'19","reported_seq":308,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-10T19:22:11.289390+0000","last_change":"2024-09-10T19:20:41.226611+0000","last_active":"2024-09-10T19:22:11.289390+0000","last_peered":"2024-09-10T19:22:11.289390+0000","last_clean":"2024-09-10T19:22:11.289390+0000","last_became_active":"2024-09-10T19:13:54.720034+0000","last_became_peered":"2024-09-10T19:13:54.720034+0000","last_unstale":"2024-09-10T19:22:11.289390+0000","last_undegraded":"2024-09-10T19:22:11.289390+0000","last_fullsized":"2024-09-10T19:22:11.289390+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"237'19","last_scrub_stamp":"2024-09-10T19:20:41.226522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:20:41.226522+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.010625466,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":205,"reported_epoch":267,"state":"active+undersized","last_fresh":"2024-09-10T19:22:11.062284+0000","last_change":"2024-09-10T19:21:48.019714+0000","last_active":"2024-09-10T19:22:11.062284+0000","last_peered":"2024-09-10T19:22:11.062284+0000","last_clean":"2024-09-10T19:21:43.954115+0000","last_became_active":"2024-09-10T19:21:48.019714+0000","last_became_peered":"2024-09-10T19:21:48.019714+0000","last_unstale":"2024-09-10T19:22:11.062284+0000","last_undegraded":"2024-09-10T19:22:11.062284+0000","last_fullsized":"2024-09-10T19:21:47.993355+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:19:10.439659+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:11:10.031245+0000","last_clean_scrub_stamp":"2024-09-10T19:19:10.439659+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.014001438,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":["5","6"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":208,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-10T19:22:11.064919+0000","last_change":"2024-09-10T19:20:45.375913+0000","last_active":"2024-09-10T19:22:11.064919+0000","last_peered":"2024-09-10T19:22:11.064919+0000","last_clean":"2024-09-10T19:22:11.064919+0000","last_became_active":"2024-09-10T19:13:54.729703+0000","last_became_peered":"2024-09-10T19:13:54.729703+0000","last_unstale":"2024-09-10T19:22:11.064919+0000","last_undegraded":"2024-09-10T19:22:11.064919+0000","last_fullsized":"2024-09-10T19:22:11.064919+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:20:45.375839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:20:45.375839+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:22:14.395693+0000","scrub_duration":0.0146499,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":249,"reported_epoch":267,"state":"active+undersized","last_fresh":"2024-09-10T19:22:11.057905+0000","last_change":"2024-09-10T19:21:47.993530+0000","last_active":"2024-09-10T19:22:11.057905+0000","last_peered":"2024-09-10T19:22:11.057905+0000","last_clean":"2024-09-10T19:21:43.944067+0000","last_became_active":"2024-09-10T19:21:47.993530+0000","last_became_peered":"2024-09-10T19:21:47.993530+0000","last_unstale":"2024-09-10T19:22:11.057905+0000","last_undegraded":"2024-09-10T19:22:11.057905+0000","last_fullsized":"2024-09-10T19:21:47.975469+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:19:13.231390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:19:13.231390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0029165770000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":["3","5"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"237'10","reported_seq":283,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-10T19:22:11.057281+0000","last_change":"2024-09-10T19:19:30.328975+0000","last_active":"2024-09-10T19:22:11.057281+0000","last_peered":"2024-09-10T19:22:11.057281+0000","last_clean":"2024-09-10T19:22:11.057281+0000","last_became_active":"2024-09-10T19:13:54.711125+0000","last_became_peered":"2024-09-10T19:13:54.711125+0000","last_unstale":"2024-09-10T19:22:11.057281+0000","last_undegraded":"2024-09-10T19:22:11.057281+0000","last_fullsized":"2024-09-10T19:22:11.057281+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"222'6","last_scrub_stamp":"2024-09-10T19:19:30.328841+0000","last_deep_scrub":"222'6","last_deep_scrub_stamp":"2024-09-10T19:19:30.328841+0000","last_clean_scrub_stamp":"2024-09-10T19:19:30.328841+0000","objects_scrubbed":2,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.011722731,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":12,"num_read_kb":10,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,4],"acting":[3,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":268,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-10T19:22:11.062672+0000","last_change":"2024-09-10T19:20:59.297958+0000","last_active":"2024-09-10T19:22:11.062672+0000","last_peered":"2024-09-10T19:22:11.062672+0000","last_clean":"2024-09-10T19:22:11.062672+0000","last_became_active":"2024-09-10T19:16:24.852460+0000","last_became_peered":"2024-09-10T19:16:24.852460+0000","last_unstale":"2024-09-10T19:22:11.062672+0000","last_undegraded":"2024-09-10T19:22:11.062672+0000","last_fullsized":"2024-09-10T19:22:11.062672+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:20:59.297872+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:20:59.297872+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:22:15.377151+0000","scrub_duration":0.0090041549999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,3],"acting":[5,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":264,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-10T19:22:11.294865+0000","last_change":"2024-09-10T19:21:56.530235+0000","last_active":"2024-09-10T19:22:11.294865+0000","last_peered":"2024-09-10T19:22:11.294865+0000","last_clean":"2024-09-10T19:22:11.294865+0000","last_became_active":"2024-09-10T19:10:11.698536+0000","last_became_peered":"2024-09-10T19:10:11.698536+0000","last_unstale":"2024-09-10T19:22:11.294865+0000","last_undegraded":"2024-09-10T19:22:11.294865+0000","last_fullsized":"2024-09-10T19:22:11.294865+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:21:56.529808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:11:31.837944+0000","last_clean_scrub_stamp":"2024-09-10T19:21:56.529808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:23:14.227016+0000","scrub_duration":0.0045868109999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":272,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-10T19:22:11.062850+0000","last_change":"2024-09-10T19:20:55.855671+0000","last_active":"2024-09-10T19:22:11.062850+0000","last_peered":"2024-09-10T19:22:11.062850+0000","last_clean":"2024-09-10T19:22:11.062850+0000","last_became_active":"2024-09-10T19:16:24.852356+0000","last_became_peered":"2024-09-10T19:16:24.852356+0000","last_unstale":"2024-09-10T19:22:11.062850+0000","last_undegraded":"2024-09-10T19:22:11.062850+0000","last_fullsized":"2024-09-10T19:22:11.062850+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:20:55.853297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:20:55.853297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:22:14.421072+0000","scrub_duration":0.017870583999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,3],"acting":[7,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.11","version":"237'19","reported_seq":38,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-10T19:22:11.056863+0000","last_change":"2024-09-10T19:21:04.182417+0000","last_active":"2024-09-10T19:22:11.056863+0000","last_peered":"2024-09-10T19:22:11.056863+0000","last_clean":"2024-09-10T19:22:11.056863+0000","last_became_active":"2024-09-10T19:21:04.169444+0000","last_became_peered":"2024-09-10T19:21:04.169444+0000","last_unstale":"2024-09-10T19:22:11.056863+0000","last_undegraded":"2024-09-10T19:22:11.056863+0000","last_fullsized":"2024-09-10T19:22:11.056863+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'11","last_scrub_stamp":"2024-09-10T19:19:25.606598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:19:25.606598+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.015282657,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":275,"reported_epoch":267,"state":"active+undersized","last_fresh":"2024-09-10T19:22:11.291809+0000","last_change":"2024-09-10T19:21:48.002896+0000","last_active":"2024-09-10T19:22:11.291809+0000","last_peered":"2024-09-10T19:22:11.291809+0000","last_clean":"2024-09-10T19:21:43.952651+0000","last_became_active":"2024-09-10T19:21:48.002896+0000","last_became_peered":"2024-09-10T19:21:48.002896+0000","last_unstale":"2024-09-10T19:22:11.291809+0000","last_undegraded":"2024-09-10T19:22:11.291809+0000","last_fullsized":"2024-09-10T19:21:47.987616+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:20:08.412227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:13:05.876655+0000","last_clean_scrub_stamp":"2024-09-10T19:20:08.412227+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0037109489999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":["6","5"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":254,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-10T19:22:11.061744+0000","last_change":"2024-09-10T19:21:50.103708+0000","last_active":"2024-09-10T19:22:11.061744+0000","last_peered":"2024-09-10T19:22:11.061744+0000","last_clean":"2024-09-10T19:22:11.061744+0000","last_became_active":"2024-09-10T19:10:11.699013+0000","last_became_peered":"2024-09-10T19:10:11.699013+0000","last_unstale":"2024-09-10T19:22:11.061744+0000","last_undegraded":"2024-09-10T19:22:11.061744+0000","last_fullsized":"2024-09-10T19:22:11.061744+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:21:50.103617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:11:57.112092+0000","last_clean_scrub_stamp":"2024-09-10T19:21:50.103617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:23:13.194296+0000","scrub_duration":0.011550181,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":249,"reported_epoch":267,"state":"active+undersized","last_fresh":"2024-09-10T19:22:11.058640+0000","last_change":"2024-09-10T19:21:47.990590+0000","last_active":"2024-09-10T19:22:11.058640+0000","last_peered":"2024-09-10T19:22:11.058640+0000","last_clean":"2024-09-10T19:21:43.944266+0000","last_became_active":"2024-09-10T19:21:47.990590+0000","last_became_peered":"2024-09-10T19:21:47.990590+0000","last_unstale":"2024-09-10T19:22:11.058640+0000","last_undegraded":"2024-09-10T19:22:11.058640+0000","last_fullsized":"2024-09-10T19:21:47.974352+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:19:18.276069+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:19:18.276069+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0034003950000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":["3","6"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"80'2","reported_seq":268,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-10T19:22:11.063382+0000","last_change":"2024-09-10T19:20:11.859906+0000","last_active":"2024-09-10T19:22:11.063382+0000","last_peered":"2024-09-10T19:22:11.063382+0000","last_clean":"2024-09-10T19:22:11.063382+0000","last_became_active":"2024-09-10T19:16:24.851220+0000","last_became_peered":"2024-09-10T19:16:24.851220+0000","last_unstale":"2024-09-10T19:22:11.063382+0000","last_undegraded":"2024-09-10T19:22:11.063382+0000","last_fullsized":"2024-09-10T19:22:11.063382+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"80'2","last_scrub_stamp":"2024-09-10T19:20:11.859816+0000","last_deep_scrub":"80'2","last_deep_scrub_stamp":"2024-09-10T19:11:30.272043+0000","last_clean_scrub_stamp":"2024-09-10T19:20:11.859816+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0068048240000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":369,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":369,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,3],"acting":[7,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":265,"reported_epoch":267,"state":"active+undersized","last_fresh":"2024-09-10T19:22:11.065316+0000","last_change":"2024-09-10T19:21:47.978661+0000","last_active":"2024-09-10T19:22:11.065316+0000","last_peered":"2024-09-10T19:22:11.065316+0000","last_clean":"2024-09-10T19:21:43.952149+0000","last_became_active":"2024-09-10T19:21:47.978661+0000","last_became_peered":"2024-09-10T19:21:47.978661+0000","last_unstale":"2024-09-10T19:22:11.065316+0000","last_undegraded":"2024-09-10T19:22:11.065316+0000","last_fullsized":"2024-09-10T19:21:47.973220+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":218,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:19:19.581564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:19:19.581564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.01041663,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":["5","4"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1e","version":"237'25","reported_seq":261,"reported_epoch":267,"state":"active+undersized+degraded","last_fresh":"2024-09-10T19:22:11.063762+0000","last_change":"2024-09-10T19:21:47.993629+0000","last_active":"2024-09-10T19:22:11.063762+0000","last_peered":"2024-09-10T19:22:11.063762+0000","last_clean":"2024-09-10T19:20:16.344252+0000","last_became_active":"2024-09-10T19:21:47.993629+0000","last_became_peered":"2024-09-10T19:21:47.993629+0000","last_unstale":"2024-09-10T19:22:11.063762+0000","last_undegraded":"2024-09-10T19:21:47.981614+0000","last_fullsized":"2024-09-10T19:21:47.981497+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"237'25","last_scrub_stamp":"2024-09-10T19:20:44.696751+0000","last_deep_scrub":"80'15","last_deep_scrub_stamp":"2024-09-10T19:11:25.883919+0000","last_clean_scrub_stamp":"2024-09-10T19:20:44.696751+0000","objects_scrubbed":5,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0040337309999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":24,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":["7","3"],"object_location_counts":[{"shards":"3,7","objects":5}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"66'8","reported_seq":423,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-10T19:22:11.291713+0000","last_change":"2024-09-10T19:21:48.538842+0000","last_active":"2024-09-10T19:22:11.291713+0000","last_peered":"2024-09-10T19:22:11.291713+0000","last_clean":"2024-09-10T19:22:11.291713+0000","last_became_active":"2024-09-10T19:10:11.695427+0000","last_became_peered":"2024-09-10T19:10:11.695427+0000","last_unstale":"2024-09-10T19:22:11.291713+0000","last_undegraded":"2024-09-10T19:22:11.291713+0000","last_fullsized":"2024-09-10T19:22:11.291713+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'8","last_scrub_stamp":"2024-09-10T19:21:48.538741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:21:48.538741+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:22:56.566083+0000","scrub_duration":0.0059444800000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":218,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-10T19:22:11.061662+0000","last_change":"2024-09-10T19:19:58.544196+0000","last_active":"2024-09-10T19:22:11.061662+0000","last_peered":"2024-09-10T19:22:11.061662+0000","last_clean":"2024-09-10T19:22:11.061662+0000","last_became_active":"2024-09-10T19:16:24.862695+0000","last_became_peered":"2024-09-10T19:16:24.862695+0000","last_unstale":"2024-09-10T19:22:11.061662+0000","last_undegraded":"2024-09-10T19:22:11.061662+0000","last_fullsized":"2024-09-10T19:22:11.061662+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:19:58.544137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:19:58.544137+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0013181899999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":265,"reported_epoch":267,"state":"active+undersized","last_fresh":"2024-09-10T19:22:11.058217+0000","last_change":"2024-09-10T19:21:47.988058+0000","last_active":"2024-09-10T19:22:11.058217+0000","last_peered":"2024-09-10T19:22:11.058217+0000","last_clean":"2024-09-10T19:21:44.676614+0000","last_became_active":"2024-09-10T19:21:47.988058+0000","last_became_peered":"2024-09-10T19:21:47.988058+0000","last_unstale":"2024-09-10T19:22:11.058217+0000","last_undegraded":"2024-09-10T19:22:11.058217+0000","last_fullsized":"2024-09-10T19:21:47.976497+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:21:44.676477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:21:44.676477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:23:05.642583+0000","scrub_duration":0.0017387360000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":["3","6"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"237'24","reported_seq":349,"reported_epoch":267,"state":"active+undersized+degraded","last_fresh":"2024-09-10T19:22:11.291766+0000","last_change":"2024-09-10T19:21:48.002057+0000","last_active":"2024-09-10T19:22:11.291766+0000","last_peered":"2024-09-10T19:22:11.291766+0000","last_clean":"2024-09-10T19:21:43.952583+0000","last_became_active":"2024-09-10T19:21:48.002057+0000","last_became_peered":"2024-09-10T19:21:48.002057+0000","last_unstale":"2024-09-10T19:22:11.291766+0000","last_undegraded":"2024-09-10T19:21:47.987701+0000","last_fullsized":"2024-09-10T19:21:47.985296+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"80'16","last_scrub_stamp":"2024-09-10T19:20:01.500062+0000","last_deep_scrub":"80'16","last_deep_scrub_stamp":"2024-09-10T19:11:40.853611+0000","last_clean_scrub_stamp":"2024-09-10T19:20:01.500062+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.002579296,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":["6","5"],"object_location_counts":[{"shards":"5,6","objects":5}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":136,"reported_epoch":267,"state":"active+undersized","last_fresh":"2024-09-10T19:22:11.063448+0000","last_change":"2024-09-10T19:21:47.989531+0000","last_active":"2024-09-10T19:22:11.063448+0000","last_peered":"2024-09-10T19:22:11.063448+0000","last_clean":"2024-09-10T19:16:23.155700+0000","last_became_active":"2024-09-10T19:21:47.989531+0000","last_became_peered":"2024-09-10T19:21:47.989531+0000","last_unstale":"2024-09-10T19:22:11.063448+0000","last_undegraded":"2024-09-10T19:22:11.063448+0000","last_fullsized":"2024-09-10T19:21:47.982576+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:20:01.699734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:11:28.822327+0000","last_clean_scrub_stamp":"2024-09-10T19:20:01.699734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0022130729999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":["7","3"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":251,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-10T19:22:11.289590+0000","last_change":"2024-09-10T19:21:07.416889+0000","last_active":"2024-09-10T19:22:11.289590+0000","last_peered":"2024-09-10T19:22:11.289590+0000","last_clean":"2024-09-10T19:22:11.289590+0000","last_became_active":"2024-09-10T19:10:14.053619+0000","last_became_peered":"2024-09-10T19:10:14.053619+0000","last_unstale":"2024-09-10T19:22:11.289590+0000","last_undegraded":"2024-09-10T19:22:11.289590+0000","last_fullsized":"2024-09-10T19:22:11.289590+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:21:07.416658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:21:07.416658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:22:36.974440+0000","scrub_duration":0.0096941340000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":258,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-10T19:22:11.295098+0000","last_change":"2024-09-10T19:20:51.313722+0000","last_active":"2024-09-10T19:22:11.295098+0000","last_peered":"2024-09-10T19:22:11.295098+0000","last_clean":"2024-09-10T19:22:11.295098+0000","last_became_active":"2024-09-10T19:10:07.685997+0000","last_became_peered":"2024-09-10T19:10:07.685997+0000","last_unstale":"2024-09-10T19:22:11.295098+0000","last_undegraded":"2024-09-10T19:22:11.295098+0000","last_fullsized":"2024-09-10T19:22:11.295098+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:20:51.313569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:20:51.313569+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:22:20.614023+0000","scrub_duration":0.01586305,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1c","version":"237'34","reported_seq":232,"reported_epoch":267,"state":"active+undersized+degraded","last_fresh":"2024-09-10T19:22:11.058634+0000","last_change":"2024-09-10T19:21:48.003880+0000","last_active":"2024-09-10T19:22:11.058634+0000","last_peered":"2024-09-10T19:22:11.058634+0000","last_clean":"2024-09-10T19:21:43.945320+0000","last_became_active":"2024-09-10T19:21:48.003880+0000","last_became_peered":"2024-09-10T19:21:48.003880+0000","last_unstale":"2024-09-10T19:22:11.058634+0000","last_undegraded":"2024-09-10T19:21:47.973954+0000","last_fullsized":"2024-09-10T19:21:47.973536+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"237'34","last_scrub_stamp":"2024-09-10T19:20:40.452293+0000","last_deep_scrub":"78'22","last_deep_scrub_stamp":"2024-09-10T19:11:21.118937+0000","last_clean_scrub_stamp":"2024-09-10T19:20:40.452293+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0025454940000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":["3","4"],"object_location_counts":[{"shards":"3,4","objects":7}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":218,"reported_epoch":267,"state":"active+undersized","last_fresh":"2024-09-10T19:22:11.293258+0000","last_change":"2024-09-10T19:21:47.981439+0000","last_active":"2024-09-10T19:22:11.293258+0000","last_peered":"2024-09-10T19:22:11.293258+0000","last_clean":"2024-09-10T19:21:43.948694+0000","last_became_active":"2024-09-10T19:21:47.981439+0000","last_became_peered":"2024-09-10T19:21:47.981439+0000","last_unstale":"2024-09-10T19:22:11.293258+0000","last_undegraded":"2024-09-10T19:22:11.293258+0000","last_fullsized":"2024-09-10T19:21:47.972032+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:20:08.342716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:20:08.342716+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.002558444,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":["4","5"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":252,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-10T19:22:11.058520+0000","last_change":"2024-09-10T19:21:57.736056+0000","last_active":"2024-09-10T19:22:11.058520+0000","last_peered":"2024-09-10T19:22:11.058520+0000","last_clean":"2024-09-10T19:22:11.058520+0000","last_became_active":"2024-09-10T19:10:14.053698+0000","last_became_peered":"2024-09-10T19:10:14.053698+0000","last_unstale":"2024-09-10T19:22:11.058520+0000","last_undegraded":"2024-09-10T19:22:11.058520+0000","last_fullsized":"2024-09-10T19:22:11.058520+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:21:57.736006+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:21:57.736006+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:23:27.040783+0000","scrub_duration":0.0017464049999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":258,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-10T19:22:11.058861+0000","last_change":"2024-09-10T19:22:01.763166+0000","last_active":"2024-09-10T19:22:11.058861+0000","last_peered":"2024-09-10T19:22:11.058861+0000","last_clean":"2024-09-10T19:22:11.058861+0000","last_became_active":"2024-09-10T19:10:07.682811+0000","last_became_peered":"2024-09-10T19:10:07.682811+0000","last_unstale":"2024-09-10T19:22:11.058861+0000","last_undegraded":"2024-09-10T19:22:11.058861+0000","last_fullsized":"2024-09-10T19:22:11.058861+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:22:01.763088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:22:01.763088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:23:21.447120+0000","scrub_duration":0.0083058279999999995,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1d","version":"250'23","reported_seq":312,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-10T19:22:11.057778+0000","last_change":"2024-09-10T19:20:55.582263+0000","last_active":"2024-09-10T19:22:11.057778+0000","last_peered":"2024-09-10T19:22:11.057778+0000","last_clean":"2024-09-10T19:22:11.057778+0000","last_became_active":"2024-09-10T19:13:54.714629+0000","last_became_peered":"2024-09-10T19:13:54.714629+0000","last_unstale":"2024-09-10T19:22:11.057778+0000","last_undegraded":"2024-09-10T19:22:11.057778+0000","last_fullsized":"2024-09-10T19:22:11.057778+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"250'23","last_scrub_stamp":"2024-09-10T19:20:55.582173+0000","last_deep_scrub":"250'23","last_deep_scrub_stamp":"2024-09-10T19:20:55.582173+0000","last_clean_scrub_stamp":"2024-09-10T19:20:55.582173+0000","objects_scrubbed":5,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.013381652000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":263,"reported_epoch":267,"state":"active+clean","last_fresh":"2024-09-10T19:22:11.289908+0000","last_change":"2024-09-10T19:21:04.381504+0000","last_active":"2024-09-10T19:22:11.289908+0000","last_peered":"2024-09-10T19:22:11.289908+0000","last_clean":"2024-09-10T19:22:11.289908+0000","last_became_active":"2024-09-10T19:16:24.843013+0000","last_became_peered":"2024-09-10T19:16:24.843013+0000","last_unstale":"2024-09-10T19:22:11.289908+0000","last_undegraded":"2024-09-10T19:22:11.289908+0000","last_fullsized":"2024-09-10T19:22:11.289908+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:21:04.381408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:21:04.381408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:22:15.619575+0000","scrub_duration":0.012414105999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":9,"num_pg":16,"stat_sum":{"num_bytes":5928779800,"num_objects":90467,"num_object_clones":0,"num_object_copies":271401,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":45436,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":90467,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":202507,"num_write_kb":9375105,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":18959773696,"data_stored":18959761480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":60174,"ondisk_log_size":60174,"up":40,"acting":40,"num_store_stats":6},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":382,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":1528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":82,"acting":82,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":78,"ondisk_log_size":78,"up":77,"acting":77,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":177,"num_object_clones":0,"num_object_copies":531,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":80,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":177,"num_whiteouts":0,"num_read":1119,"num_read_kb":876,"num_write":703,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":178,"num_bytes_recovered":3328,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":540672,"data_stored":14242,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":758,"ondisk_log_size":758,"up":82,"acting":82,"num_store_stats":8},{"poolid":3,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":353,"num_read_kb":348,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":389,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":2002,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":9,"ondisk_log_size":9,"up":7,"acting":7,"num_store_stats":7},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1328,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":27,"num_read_kb":27,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":3984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4,"ondisk_log_size":4,"up":79,"acting":79,"num_store_stats":8},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":370028,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":21,"num_read_kb":40,"num_write":21,"num_write_kb":296,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":435020,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":869,"ondisk_log_size":869,"up":29,"acting":29,"num_store_stats":6}],"osd_stats":[{"osd":7,"up_from":54,"seq":231928234134,"num_pgs":83,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":4554668,"kb_used_data":3658208,"kb_used_omap":1,"kb_used_meta":896446,"kb_avail":89190484,"statfs":{"total":95995035648,"available":91331055616,"internally_reserved":0,"allocated":3746004992,"data_stored":3745656371,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":917961048},"hb_peers":[2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,3,3],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 19:21:08 2024","interfaces":[{"interface":"back","average":{"1min":0.61299999999999999,"5min":1.046,"15min":1.113},"min":{"1min":0.215,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":2.3620000000000001,"5min":8.3770000000000007,"15min":35.551000000000002},"last":0.91900000000000004},{"interface":"front","average":{"1min":1.879,"5min":1.2789999999999999,"15min":1.131},"min":{"1min":0.24399999999999999,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":5.0140000000000002,"5min":8.1440000000000001,"15min":35.503999999999998},"last":0.97999999999999998}]},{"osd":1,"last update":"Tue Sep 10 19:20:29 2024","interfaces":[{"interface":"back","average":{"1min":0.751,"5min":1.0609999999999999,"15min":0.70299999999999996},"min":{"1min":0.26000000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":2.5089999999999999,"5min":8.3499999999999996,"15min":8.3499999999999996},"last":2.181},{"interface":"front","average":{"1min":0.78000000000000003,"5min":1.0209999999999999,"15min":0.81200000000000006},"min":{"1min":0.28799999999999998,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":2.5640000000000001,"5min":5.609,"15min":5.609},"last":2.157}]},{"osd":2,"last update":"Tue Sep 10 19:22:09 2024","interfaces":[{"interface":"back","average":{"1min":1.143,"5min":1.0249999999999999,"15min":0.88400000000000001},"min":{"1min":0.27600000000000002,"5min":0.27600000000000002,"15min":0.25600000000000001},"max":{"1min":3.722,"5min":3.77,"15min":31.940999999999999},"last":0.23000000000000001},{"interface":"front","average":{"1min":0.74299999999999999,"5min":1,"15min":0.95399999999999996},"min":{"1min":0.317,"5min":0.28399999999999997,"15min":0.24099999999999999},"max":{"1min":4.4139999999999997,"5min":5.5410000000000004,"15min":25.056000000000001},"last":0.32400000000000001}]},{"osd":3,"last update":"Tue Sep 10 19:22:09 2024","interfaces":[{"interface":"back","average":{"1min":1.375,"5min":1.3580000000000001,"15min":1.2509999999999999},"min":{"1min":0.23899999999999999,"5min":0.23899999999999999,"15min":0.217},"max":{"1min":4.6070000000000002,"5min":21.074999999999999,"15min":35.277999999999999},"last":0.20200000000000001},{"interface":"front","average":{"1min":0.752,"5min":1.343,"15min":1.2010000000000001},"min":{"1min":0.27400000000000002,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":4.4500000000000002,"5min":18.82,"15min":36.512999999999998},"last":0.35399999999999998}]},{"osd":4,"last update":"Tue Sep 10 19:22:09 2024","interfaces":[{"interface":"back","average":{"1min":1.3360000000000001,"5min":1.9670000000000001,"15min":1.046},"min":{"1min":0.222,"5min":0.222,"15min":0.222},"max":{"1min":5.6269999999999998,"5min":8.4399999999999995,"15min":8.4399999999999995},"last":0.877},{"interface":"front","average":{"1min":1.2130000000000001,"5min":1.3680000000000001,"15min":0.82499999999999996},"min":{"1min":0.24399999999999999,"5min":0.23100000000000001,"15min":0.23000000000000001},"max":{"1min":3.8490000000000002,"5min":6.149,"15min":6.149},"last":0.25}]},{"osd":5,"last update":"Tue Sep 10 19:22:09 2024","interfaces":[{"interface":"back","average":{"1min":1.248,"5min":1.845,"15min":1.0900000000000001},"min":{"1min":0.30399999999999999,"5min":0.30399999999999999,"15min":0.23699999999999999},"max":{"1min":4.4749999999999996,"5min":33.521000000000001,"15min":33.521000000000001},"last":0.59999999999999998},{"interface":"front","average":{"1min":1.804,"5min":2.2709999999999999,"15min":1.2170000000000001},"min":{"1min":0.39300000000000002,"5min":0.33700000000000002,"15min":0.26000000000000001},"max":{"1min":5.9029999999999996,"5min":34.438000000000002,"15min":34.438000000000002},"last":0.94199999999999995}]},{"osd":6,"last update":"Tue Sep 10 19:22:09 2024","interfaces":[{"interface":"back","average":{"1min":1.6839999999999999,"5min":1.927,"15min":1.113},"min":{"1min":0.48699999999999999,"5min":0.312,"15min":0.26200000000000001},"max":{"1min":5.7119999999999997,"5min":8.3710000000000004,"15min":8.3710000000000004},"last":0.91100000000000003},{"interface":"front","average":{"1min":1.466,"5min":1.5189999999999999,"15min":0.93500000000000005},"min":{"1min":0.40200000000000002,"5min":0.29199999999999998,"15min":0.28000000000000003},"max":{"1min":3.9119999999999999,"5min":6.8200000000000003,"15min":8.4130000000000003},"last":0.27000000000000002}]}]},{"osd":6,"up_from":47,"seq":201863463067,"num_pgs":78,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2424224,"kb_used_data":1834324,"kb_used_omap":1,"kb_used_meta":589886,"kb_avail":91320928,"statfs":{"total":95995035648,"available":93512630272,"internally_reserved":0,"allocated":1878347776,"data_stored":1878012039,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":604043608},"hb_peers":[2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 19:20:51 2024","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":0.69899999999999995,"15min":1.1279999999999999},"min":{"1min":0.23100000000000001,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":2.2010000000000001,"5min":5.2350000000000003,"15min":112.988},"last":2.3159999999999998},{"interface":"front","average":{"1min":0.59799999999999998,"5min":0.90500000000000003,"15min":1.1990000000000001},"min":{"1min":0.221,"5min":0.221,"15min":0.221},"max":{"1min":2.0009999999999999,"5min":5.274,"15min":113.18300000000001},"last":0.41899999999999998}]},{"osd":1,"last update":"Tue Sep 10 19:20:35 2024","interfaces":[{"interface":"back","average":{"1min":0.51800000000000002,"5min":0.70999999999999996,"15min":0.57899999999999996},"min":{"1min":0.31900000000000001,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":0.995,"5min":4.2919999999999998,"15min":4.2919999999999998},"last":0.33200000000000002},{"interface":"front","average":{"1min":0.59899999999999998,"5min":0.91000000000000003,"15min":0.73899999999999999},"min":{"1min":0.29699999999999999,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":2.0750000000000002,"5min":6.5289999999999999,"15min":6.5289999999999999},"last":0.31900000000000001}]},{"osd":2,"last update":"Tue Sep 10 19:21:52 2024","interfaces":[{"interface":"back","average":{"1min":1.093,"5min":0.83899999999999997,"15min":1.129},"min":{"1min":0.26700000000000002,"5min":0.26700000000000002,"15min":0.20399999999999999},"max":{"1min":3.6099999999999999,"5min":3.6099999999999999,"15min":112.864},"last":0.23799999999999999},{"interface":"front","average":{"1min":1.052,"5min":0.97299999999999998,"15min":1.204},"min":{"1min":0.26800000000000002,"5min":0.254,"15min":0.254},"max":{"1min":3.8660000000000001,"5min":5.8300000000000001,"15min":112.822},"last":0.25600000000000001}]},{"osd":3,"last update":"Tue Sep 10 19:21:52 2024","interfaces":[{"interface":"back","average":{"1min":1.109,"5min":1.26,"15min":1.4079999999999999},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":3.9079999999999999,"5min":23.824999999999999,"15min":125.56399999999999},"last":0.81399999999999995},{"interface":"front","average":{"1min":1.139,"5min":1.1930000000000001,"15min":1.431},"min":{"1min":0.28399999999999997,"5min":0.254,"15min":0.23200000000000001},"max":{"1min":3.794,"5min":24.814,"15min":125.47},"last":0.27600000000000002}]},{"osd":4,"last update":"Tue Sep 10 19:21:52 2024","interfaces":[{"interface":"back","average":{"1min":2.0499999999999998,"5min":1.589,"15min":0.84699999999999998},"min":{"1min":0.29199999999999998,"5min":0.253,"15min":0.20699999999999999},"max":{"1min":6.5330000000000004,"5min":6.9269999999999996,"15min":6.9269999999999996},"last":0.31900000000000001},{"interface":"front","average":{"1min":1.4930000000000001,"5min":1.2969999999999999,"15min":0.76800000000000002},"min":{"1min":0.24099999999999999,"5min":0.24099999999999999,"15min":0.22500000000000001},"max":{"1min":6.907,"5min":6.907,"15min":6.907},"last":2.6869999999999998}]},{"osd":5,"last update":"Tue Sep 10 19:21:52 2024","interfaces":[{"interface":"back","average":{"1min":1.869,"5min":1.6100000000000001,"15min":1.038},"min":{"1min":0.39000000000000001,"5min":0.30299999999999999,"15min":0.248},"max":{"1min":4.4189999999999996,"5min":5.9640000000000004,"15min":44.765000000000001},"last":0.29299999999999998},{"interface":"front","average":{"1min":1.903,"5min":1.603,"15min":1.0940000000000001},"min":{"1min":0.28399999999999997,"5min":0.28399999999999997,"15min":0.27500000000000002},"max":{"1min":6.9459999999999997,"5min":7.1369999999999996,"15min":43.036000000000001},"last":3.2530000000000001}]},{"osd":7,"last update":"Tue Sep 10 19:21:18 2024","interfaces":[{"interface":"back","average":{"1min":1.1599999999999999,"5min":1.24,"15min":0.754},"min":{"1min":0.29899999999999999,"5min":0.29899999999999999,"15min":0.27500000000000002},"max":{"1min":3.831,"5min":8.9390000000000001,"15min":8.9390000000000001},"last":0.443},{"interface":"front","average":{"1min":1.8500000000000001,"5min":1.607,"15min":0.89700000000000002},"min":{"1min":0.48999999999999999,"5min":0.33300000000000002,"15min":0.29299999999999998},"max":{"1min":6.9720000000000004,"5min":10.298,"15min":10.298},"last":0.84899999999999998}]}]},{"osd":5,"up_from":39,"seq":167503724704,"num_pgs":80,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":4911192,"kb_used_data":4009036,"kb_used_omap":1,"kb_used_meta":902078,"kb_avail":88833960,"statfs":{"total":95995035648,"available":90965975040,"internally_reserved":0,"allocated":4105252864,"data_stored":4104926074,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":923728216},"hb_peers":[2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 19:21:23 2024","interfaces":[{"interface":"back","average":{"1min":1.1579999999999999,"5min":1.9570000000000001,"15min":1.1060000000000001},"min":{"1min":0.19600000000000001,"5min":0.17999999999999999,"15min":0.17999999999999999},"max":{"1min":3.9009999999999998,"5min":73.933000000000007,"15min":73.933000000000007},"last":4.4960000000000004},{"interface":"front","average":{"1min":1.286,"5min":1.821,"15min":1.024},"min":{"1min":0.23100000000000001,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":3.9969999999999999,"5min":73.769999999999996,"15min":73.769999999999996},"last":3.2749999999999999}]},{"osd":1,"last update":"Tue Sep 10 19:20:27 2024","interfaces":[{"interface":"back","average":{"1min":0.73299999999999998,"5min":0.98499999999999999,"15min":0.76300000000000001},"min":{"1min":0.23300000000000001,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":2.2050000000000001,"5min":4.3499999999999996,"15min":4.3499999999999996},"last":4.117},{"interface":"front","average":{"1min":0.875,"5min":0.97799999999999998,"15min":0.66100000000000003},"min":{"1min":0.28999999999999998,"5min":0.255,"15min":0.255},"max":{"1min":4.742,"5min":5.0570000000000004,"15min":5.0570000000000004},"last":0.29799999999999999}]},{"osd":2,"last update":"Tue Sep 10 19:21:23 2024","interfaces":[{"interface":"back","average":{"1min":1.3400000000000001,"5min":1.3120000000000001,"15min":0.84599999999999997},"min":{"1min":0.23899999999999999,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":4.3200000000000003,"5min":9.9399999999999995,"15min":9.9399999999999995},"last":1.093},{"interface":"front","average":{"1min":1.347,"5min":1.165,"15min":0.84999999999999998},"min":{"1min":0.30099999999999999,"5min":0.28599999999999998,"15min":0.224},"max":{"1min":4.0910000000000002,"5min":4.4139999999999997,"15min":6.3789999999999996},"last":1.7050000000000001}]},{"osd":3,"last update":"Tue Sep 10 19:21:23 2024","interfaces":[{"interface":"back","average":{"1min":1.204,"5min":1.284,"15min":0.83899999999999997},"min":{"1min":0.26300000000000001,"5min":0.23300000000000001,"15min":0.224},"max":{"1min":3.8580000000000001,"5min":9.9019999999999992,"15min":9.9019999999999992},"last":1.0660000000000001},{"interface":"front","average":{"1min":1.389,"5min":1.2230000000000001,"15min":0.83799999999999997},"min":{"1min":0.36799999999999999,"5min":0.28899999999999998,"15min":0.248},"max":{"1min":4.0609999999999999,"5min":4.4459999999999997,"15min":5.3869999999999996},"last":1.6779999999999999}]},{"osd":4,"last update":"Tue Sep 10 19:21:23 2024","interfaces":[{"interface":"back","average":{"1min":1.329,"5min":1.5129999999999999,"15min":0.82799999999999996},"min":{"1min":0.312,"5min":0.24399999999999999,"15min":0.20699999999999999},"max":{"1min":3.6680000000000001,"5min":10.414,"15min":10.414},"last":1.1930000000000001},{"interface":"front","average":{"1min":1.6739999999999999,"5min":1.623,"15min":0.89100000000000001},"min":{"1min":0.42599999999999999,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":5.1609999999999996,"5min":10.518000000000001,"15min":10.518000000000001},"last":3.419}]},{"osd":6,"last update":"Tue Sep 10 19:21:50 2024","interfaces":[{"interface":"back","average":{"1min":2.0139999999999998,"5min":1.8240000000000001,"15min":1.0349999999999999},"min":{"1min":0.28399999999999997,"5min":0.28399999999999997,"15min":0.223},"max":{"1min":9.6609999999999996,"5min":9.9670000000000005,"15min":9.9670000000000005},"last":3.4510000000000001},{"interface":"front","average":{"1min":1.782,"5min":1.742,"15min":0.97399999999999998},"min":{"1min":0.31,"5min":0.29499999999999998,"15min":0.25},"max":{"1min":5.2679999999999998,"5min":10.446,"15min":10.446},"last":1.635}]},{"osd":7,"last update":"Tue Sep 10 19:21:16 2024","interfaces":[{"interface":"back","average":{"1min":1.5469999999999999,"5min":1.534,"15min":0.91700000000000004},"min":{"1min":0.29999999999999999,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":4.0369999999999999,"5min":5.0640000000000001,"15min":5.9660000000000002},"last":1.75},{"interface":"front","average":{"1min":1.819,"5min":1.6060000000000001,"15min":0.89500000000000002},"min":{"1min":0.27100000000000002,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":9.4909999999999997,"5min":9.4909999999999997,"15min":9.4909999999999997},"last":1.9770000000000001}]}]},{"osd":4,"up_from":32,"seq":137438953636,"num_pgs":71,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2399564,"kb_used_data":1813828,"kb_used_omap":1,"kb_used_meta":585726,"kb_avail":91345588,"statfs":{"total":95995035648,"available":93537882112,"internally_reserved":0,"allocated":1857359872,"data_stored":1857038976,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":599783768},"hb_peers":[2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 19:21:10 2024","interfaces":[{"interface":"back","average":{"1min":1.0209999999999999,"5min":1.224,"15min":0.97999999999999998},"min":{"1min":0.25700000000000001,"5min":0.187,"15min":0.187},"max":{"1min":5.1420000000000003,"5min":24.113,"15min":32.533000000000001},"last":0.53100000000000003},{"interface":"front","average":{"1min":0.82799999999999996,"5min":1.1739999999999999,"15min":0.98099999999999998},"min":{"1min":0.20399999999999999,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":3.3100000000000001,"5min":24.132999999999999,"15min":32.512999999999998},"last":1.6160000000000001}]},{"osd":1,"last update":"Tue Sep 10 19:20:27 2024","interfaces":[{"interface":"back","average":{"1min":0.46700000000000003,"5min":0.91900000000000004,"15min":0.72499999999999998},"min":{"1min":0.23200000000000001,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":1.1779999999999999,"5min":7.9770000000000003,"15min":7.9770000000000003},"last":0.26700000000000002},{"interface":"front","average":{"1min":0.78700000000000003,"5min":0.85599999999999998,"15min":0.73599999999999999},"min":{"1min":0.32300000000000001,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":2.3700000000000001,"5min":3.8260000000000001,"15min":3.8260000000000001},"last":2.0590000000000002}]},{"osd":2,"last update":"Tue Sep 10 19:22:13 2024","interfaces":[{"interface":"back","average":{"1min":0.69399999999999995,"5min":0.876,"15min":0.76200000000000001},"min":{"1min":0.25600000000000001,"5min":0.254,"15min":0.214},"max":{"1min":2.5779999999999998,"5min":5.048,"15min":6.8869999999999996},"last":0.502},{"interface":"front","average":{"1min":1.1699999999999999,"5min":0.98299999999999998,"15min":0.77800000000000002},"min":{"1min":0.26700000000000002,"5min":0.253,"15min":0.23400000000000001},"max":{"1min":3.5030000000000001,"5min":4.7169999999999996,"15min":7.085},"last":0.26700000000000002}]},{"osd":3,"last update":"Tue Sep 10 19:22:13 2024","interfaces":[{"interface":"back","average":{"1min":1.1619999999999999,"5min":1.2370000000000001,"15min":0.99199999999999999},"min":{"1min":0.20999999999999999,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":3.5289999999999999,"5min":15.436,"15min":35.451999999999998},"last":0.20999999999999999},{"interface":"front","average":{"1min":0.88100000000000001,"5min":1.323,"15min":1.048},"min":{"1min":0.247,"5min":0.247,"15min":0.247},"max":{"1min":3.5059999999999998,"5min":14.954000000000001,"15min":32.880000000000003},"last":0.61599999999999999}]},{"osd":5,"last update":"Tue Sep 10 19:21:29 2024","interfaces":[{"interface":"back","average":{"1min":1.871,"5min":2.3879999999999999,"15min":1.099},"min":{"1min":0.35099999999999998,"5min":0.224,"15min":0.224},"max":{"1min":4.0140000000000002,"5min":67.379000000000005,"15min":67.379000000000005},"last":0.67700000000000005},{"interface":"front","average":{"1min":1.9710000000000001,"5min":2.1699999999999999,"15min":1.03},"min":{"1min":0.31900000000000001,"5min":0.27500000000000002,"15min":0.223},"max":{"1min":8.4429999999999996,"5min":67.542000000000002,"15min":67.542000000000002},"last":0.53000000000000003}]},{"osd":6,"last update":"Tue Sep 10 19:22:06 2024","interfaces":[{"interface":"back","average":{"1min":1.2250000000000001,"5min":1.27,"15min":0.79900000000000004},"min":{"1min":0.27800000000000002,"5min":0.27800000000000002,"15min":0.253},"max":{"1min":4.069,"5min":5.5289999999999999,"15min":5.5289999999999999},"last":0.46800000000000003},{"interface":"front","average":{"1min":0.90000000000000002,"5min":1.3240000000000001,"15min":0.79500000000000004},"min":{"1min":0.26800000000000002,"5min":0.247,"15min":0.247},"max":{"1min":3.5499999999999998,"5min":7.9550000000000001,"15min":7.9550000000000001},"last":0.65100000000000002}]},{"osd":7,"last update":"Tue Sep 10 19:21:14 2024","interfaces":[{"interface":"back","average":{"1min":1.764,"5min":1.4650000000000001,"15min":0.97199999999999998},"min":{"1min":0.33000000000000002,"5min":0.28100000000000003,"15min":0.27700000000000002},"max":{"1min":4.1580000000000004,"5min":10.33,"15min":23.081},"last":0.98299999999999998},{"interface":"front","average":{"1min":1.502,"5min":1.419,"15min":0.94399999999999995},"min":{"1min":0.46899999999999997,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":4.1470000000000002,"5min":8.8550000000000004,"15min":22.666},"last":0.34100000000000003}]}]},{"osd":3,"up_from":26,"seq":111669149865,"num_pgs":84,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":4177264,"kb_used_data":3281124,"kb_used_omap":1,"kb_used_meta":896126,"kb_avail":89567888,"statfs":{"total":95995035648,"available":91717517312,"internally_reserved":0,"allocated":3359870976,"data_stored":3359519995,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":917633368},"hb_peers":[2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 19:20:46 2024","interfaces":[{"interface":"back","average":{"1min":0.40799999999999997,"5min":0.39500000000000002,"15min":0.69099999999999995},"min":{"1min":0.16700000000000001,"5min":0.157,"15min":0.157},"max":{"1min":0.77800000000000002,"5min":3.9529999999999998,"15min":17.518000000000001},"last":0.39000000000000001},{"interface":"front","average":{"1min":0.72499999999999998,"5min":0.63900000000000001,"15min":0.84799999999999998},"min":{"1min":0.19400000000000001,"5min":0.17899999999999999,"15min":0.17399999999999999},"max":{"1min":2.3969999999999998,"5min":5.1539999999999999,"15min":17.609000000000002},"last":0.40799999999999997}]},{"osd":1,"last update":"Tue Sep 10 19:20:25 2024","interfaces":[{"interface":"back","average":{"1min":0.43099999999999999,"5min":0.45600000000000002,"15min":0.51200000000000001},"min":{"1min":0.222,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":1.2869999999999999,"5min":2.6059999999999999,"15min":2.6059999999999999},"last":0.27700000000000002},{"interface":"front","average":{"1min":0.438,"5min":0.72399999999999998,"15min":1.2869999999999999},"min":{"1min":0.248,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":1.1919999999999999,"5min":17.454999999999998,"15min":17.454999999999998},"last":0.38500000000000001}]},{"osd":2,"last update":"Tue Sep 10 19:21:50 2024","interfaces":[{"interface":"back","average":{"1min":0.40100000000000002,"5min":0.502,"15min":0.65600000000000003},"min":{"1min":0.215,"5min":0.215,"15min":0.186},"max":{"1min":1.256,"5min":3.125,"15min":5.3029999999999999},"last":0.191},{"interface":"front","average":{"1min":0.46999999999999997,"5min":0.45200000000000001,"15min":0.71699999999999997},"min":{"1min":0.23599999999999999,"5min":0.23599999999999999,"15min":0.19800000000000001},"max":{"1min":2.5649999999999999,"5min":2.5649999999999999,"15min":17.806000000000001},"last":0.23499999999999999}]},{"osd":4,"last update":"Tue Sep 10 19:22:11 2024","interfaces":[{"interface":"back","average":{"1min":0.93000000000000005,"5min":0.89200000000000002,"15min":0.92000000000000004},"min":{"1min":0.26900000000000002,"5min":0.254,"15min":0.23599999999999999},"max":{"1min":2.7229999999999999,"5min":5.2750000000000004,"15min":17.834},"last":2.5779999999999998},{"interface":"front","average":{"1min":1.1899999999999999,"5min":0.97299999999999998,"15min":1.0629999999999999},"min":{"1min":0.22700000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":3.1909999999999998,"5min":3.339,"15min":17.553000000000001},"last":1.3660000000000001}]},{"osd":5,"last update":"Tue Sep 10 19:21:26 2024","interfaces":[{"interface":"back","average":{"1min":1.415,"5min":2.02,"15min":1.2250000000000001},"min":{"1min":0.39700000000000002,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":3.3929999999999998,"5min":80.942999999999998,"15min":80.942999999999998},"last":0.85899999999999999},{"interface":"front","average":{"1min":1.022,"5min":1.978,"15min":1.3340000000000001},"min":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.22},"max":{"1min":2.944,"5min":81.489000000000004,"15min":81.489000000000004},"last":2.2490000000000001}]},{"osd":6,"last update":"Tue Sep 10 19:21:53 2024","interfaces":[{"interface":"back","average":{"1min":1.177,"5min":1.097,"15min":1.0129999999999999},"min":{"1min":0.27900000000000003,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":6.109,"5min":6.109,"15min":17.861000000000001},"last":0.28899999999999998},{"interface":"front","average":{"1min":1.093,"5min":0.95299999999999996,"15min":0.88400000000000001},"min":{"1min":0.28799999999999998,"5min":0.28799999999999998,"15min":0.26800000000000002},"max":{"1min":3.8039999999999998,"5min":3.8039999999999998,"15min":17.687000000000001},"last":0.26800000000000002}]},{"osd":7,"last update":"Tue Sep 10 19:21:22 2024","interfaces":[{"interface":"back","average":{"1min":1.1200000000000001,"5min":1.1160000000000001,"15min":0.96699999999999997},"min":{"1min":0.378,"5min":0.35699999999999998,"15min":0.29399999999999998},"max":{"1min":3.347,"5min":4.1059999999999999,"15min":18.093},"last":1.4510000000000001},{"interface":"front","average":{"1min":1.0169999999999999,"5min":1.181,"15min":1.026},"min":{"1min":0.38700000000000001,"5min":0.35599999999999998,"15min":0.34100000000000003},"max":{"1min":4.1589999999999998,"5min":10.359999999999999,"15min":17.780999999999999},"last":0.26100000000000001}]}]},{"osd":2,"up_from":0,"seq":77309411502,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":0,"seq":592705486920,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":8,"seq":34359738546,"num_pgs":78,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":4950592,"kb_used_data":4059380,"kb_used_omap":1,"kb_used_meta":891198,"kb_avail":88794560,"statfs":{"total":95995035648,"available":90925629440,"internally_reserved":0,"allocated":4156805120,"data_stored":4156266196,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":912587096},"hb_peers":[2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Sep 10 19:20:34 2024","interfaces":[{"interface":"back","average":{"1min":0.50800000000000001,"5min":0.438,"15min":0.39200000000000002},"min":{"1min":0.20999999999999999,"5min":0.19,"15min":0.19},"max":{"1min":1.6799999999999999,"5min":2.0819999999999999,"15min":2.0819999999999999},"last":0.54300000000000004},{"interface":"front","average":{"1min":0.38600000000000001,"5min":0.40899999999999997,"15min":0.41899999999999998},"min":{"1min":0.19900000000000001,"5min":0.193,"15min":0.193},"max":{"1min":0.622,"5min":2.2839999999999998,"15min":2.2839999999999998},"last":0.32900000000000001}]},{"osd":2,"last update":"Tue Sep 10 19:21:39 2024","interfaces":[{"interface":"back","average":{"1min":0.46800000000000003,"5min":0.48999999999999999,"15min":0.629},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.17199999999999999},"max":{"1min":2.0840000000000001,"5min":2.0840000000000001,"15min":6.4640000000000004},"last":0.35199999999999998},{"interface":"front","average":{"1min":0.41399999999999998,"5min":0.46500000000000002,"15min":0.60999999999999999},"min":{"1min":0.22500000000000001,"5min":0.22500000000000001,"15min":0.187},"max":{"1min":1.2090000000000001,"5min":2.8660000000000001,"15min":9.9399999999999995},"last":0.41099999999999998}]},{"osd":3,"last update":"Tue Sep 10 19:20:49 2024","interfaces":[{"interface":"back","average":{"1min":0.51900000000000002,"5min":0.58499999999999996,"15min":0.68000000000000005},"min":{"1min":0.29199999999999998,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":1.0129999999999999,"5min":3.6190000000000002,"15min":6.5590000000000002},"last":0.29699999999999999},{"interface":"front","average":{"1min":0.39200000000000002,"5min":0.44,"15min":0.70399999999999996},"min":{"1min":0.23899999999999999,"5min":0.23899999999999999,"15min":0.23000000000000001},"max":{"1min":0.69799999999999995,"5min":1.2649999999999999,"15min":7.2140000000000004},"last":0.22}]},{"osd":4,"last update":"Tue Sep 10 19:21:07 2024","interfaces":[{"interface":"back","average":{"1min":1.0269999999999999,"5min":1.024,"15min":0.79000000000000004},"min":{"1min":0.29299999999999998,"5min":0.29099999999999998,"15min":0.24399999999999999},"max":{"1min":2.2829999999999999,"5min":7.4560000000000004,"15min":7.4560000000000004},"last":0.438},{"interface":"front","average":{"1min":0.877,"5min":0.96099999999999997,"15min":0.77800000000000002},"min":{"1min":0.31,"5min":0.28499999999999998,"15min":0.27100000000000002},"max":{"1min":2.6080000000000001,"5min":5.2640000000000002,"15min":5.2640000000000002},"last":0.61099999999999999}]},{"osd":5,"last update":"Tue Sep 10 19:21:39 2024","interfaces":[{"interface":"back","average":{"1min":0.82099999999999995,"5min":1.036,"15min":0.85299999999999998},"min":{"1min":0.28199999999999997,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":2.7090000000000001,"5min":4.327,"15min":5.3860000000000001},"last":0.69399999999999995},{"interface":"front","average":{"1min":0.95399999999999996,"5min":0.91700000000000004,"15min":0.80000000000000004},"min":{"1min":0.27700000000000002,"5min":0.27700000000000002,"15min":0.253},"max":{"1min":3.871,"5min":4.165,"15min":5.9749999999999996},"last":0.26700000000000002}]},{"osd":6,"last update":"Tue Sep 10 19:20:49 2024","interfaces":[{"interface":"back","average":{"1min":0.72099999999999997,"5min":0.92100000000000004,"15min":0.80700000000000005},"min":{"1min":0.32000000000000001,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":1.512,"5min":4.9420000000000002,"15min":5.8730000000000002},"last":0.60099999999999998},{"interface":"front","average":{"1min":0.91500000000000004,"5min":0.92700000000000005,"15min":0.82099999999999995},"min":{"1min":0.34300000000000003,"5min":0.32900000000000001,"15min":0.28699999999999998},"max":{"1min":3.6850000000000001,"5min":4.8300000000000001,"15min":5.2850000000000001},"last":0.65600000000000003}]},{"osd":7,"last update":"Tue Sep 10 19:21:19 2024","interfaces":[{"interface":"back","average":{"1min":1.369,"5min":1.0549999999999999,"15min":0.86099999999999999},"min":{"1min":0.317,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":3.1579999999999999,"5min":5.1479999999999997,"15min":5.1479999999999997},"last":0.71999999999999997},{"interface":"front","average":{"1min":1.2470000000000001,"5min":1.0329999999999999,"15min":0.83599999999999997},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.30499999999999999},"max":{"1min":4.5720000000000001,"5min":5.3410000000000002,"15min":6.0469999999999997},"last":1.411}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":966,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1236,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1236,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":362,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":1760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":2118,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1018,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":2228,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1942,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":2162,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4143579136,"data_stored":4143579136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":3328049152,"data_stored":3328049152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1856442368,"data_stored":1856438296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4066840576,"data_stored":4066836504,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1868103680,"data_stored":1868103680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":3696758784,"data_stored":3696754712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T19:22:19.031 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph osd pool set .mgr pgp_num 11 2024-09-10T19:22:19.042 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:22:19.054 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:22:19.085 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:22:19.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:22:19.087 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:22:19.091 INFO:teuthology.orchestra.run.smithi183.stderr:noscrub is unset 2024-09-10T19:22:19.187 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:22:19.239 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:22:19.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:22:19.240 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:22:19.331 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:22:19 smithi183 ceph-mon[29736]: pgmap v545: 158 pgs: 45 active+undersized, 33 active+undersized+degraded, 80 active+clean; 5.5 GiB data, 22 GiB used, 514 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.35k op/s; 45528/271989 objects degraded (16.739%) 2024-09-10T19:22:19.331 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:22:19 smithi183 ceph-mon[29736]: from='client.15516 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-10T19:22:19.331 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:22:19 smithi183 ceph-mon[29736]: from='client.? 172.21.15.183:0/1586589516' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-10T19:22:19.331 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:22:19 smithi183 ceph-mon[41562]: pgmap v545: 158 pgs: 45 active+undersized, 33 active+undersized+degraded, 80 active+clean; 5.5 GiB data, 22 GiB used, 514 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.35k op/s; 45528/271989 objects degraded (16.739%) 2024-09-10T19:22:19.331 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:22:19 smithi183 ceph-mon[41562]: from='client.15516 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-10T19:22:19.331 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:22:19 smithi183 ceph-mon[41562]: from='client.? 172.21.15.183:0/1586589516' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-10T19:22:19.341 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:22:19.369 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:22:19.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:22:19.370 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:22:19.424 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:22:19 smithi191 ceph-mon[32884]: pgmap v545: 158 pgs: 45 active+undersized, 33 active+undersized+degraded, 80 active+clean; 5.5 GiB data, 22 GiB used, 514 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.35k op/s; 45528/271989 objects degraded (16.739%) 2024-09-10T19:22:19.425 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:22:19 smithi191 ceph-mon[32884]: from='client.15516 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-10T19:22:19.425 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:22:19 smithi191 ceph-mon[32884]: from='client.? 172.21.15.183:0/1586589516' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-10T19:22:19.470 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:22:19.513 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:22:19.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:22:19.514 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:22:19.615 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:22:19.638 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:22:19.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:22:19.639 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:22:19.716 INFO:journalctl@ceph.iscsi.iscsi.a.smithi191.stdout:Sep 10 19:22:19 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-iscsi-iscsi-a[61816]: debug there is no tcmu-runner data available 2024-09-10T19:22:19.739 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:22:19.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:22:19.783 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:22:19.784 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:22:19.784 INFO:teuthology.orchestra.run.smithi191.stdout:ERROR: (22) Invalid argument 2024-09-10T19:22:19.784 INFO:teuthology.orchestra.run.smithi191.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-10T19:22:19.885 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:22:19.908 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:22:19.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:22:19.909 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:22:20.010 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:22:20.049 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:22:20.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:22:20.050 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:22:20.151 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:22:20.189 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:22:20.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:22:20.190 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:22:20.291 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:22:20.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:22:20.326 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:22:20.326 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:22:20.373 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:22:20 smithi183 ceph-mon[29736]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T19:22:20.373 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:22:20 smithi183 ceph-mon[29736]: from='client.? 172.21.15.183:0/1586589516' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-10T19:22:20.373 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:22:20 smithi183 ceph-mon[29736]: osdmap e269: 8 total, 6 up, 6 in 2024-09-10T19:22:20.373 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:22:20 smithi183 ceph-mon[29736]: 9.3 scrub starts 2024-09-10T19:22:20.373 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:22:20 smithi183 ceph-mon[29736]: purged_snaps scrub starts 2024-09-10T19:22:20.373 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:22:20 smithi183 ceph-mon[29736]: purged_snaps scrub ok 2024-09-10T19:22:20.373 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:22:20 smithi183 ceph-mon[29736]: Health check update: Degraded data redundancy: 45528/271989 objects degraded (16.739%), 33 pgs degraded (PG_DEGRADED) 2024-09-10T19:22:20.373 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:22:20 smithi183 ceph-mon[41562]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T19:22:20.373 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:22:20 smithi183 ceph-mon[41562]: from='client.? 172.21.15.183:0/1586589516' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-10T19:22:20.373 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:22:20 smithi183 ceph-mon[41562]: osdmap e269: 8 total, 6 up, 6 in 2024-09-10T19:22:20.374 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:22:20 smithi183 ceph-mon[41562]: 9.3 scrub starts 2024-09-10T19:22:20.374 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:22:20 smithi183 ceph-mon[41562]: purged_snaps scrub starts 2024-09-10T19:22:20.374 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:22:20 smithi183 ceph-mon[41562]: purged_snaps scrub ok 2024-09-10T19:22:20.374 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:22:20 smithi183 ceph-mon[41562]: Health check update: Degraded data redundancy: 45528/271989 objects degraded (16.739%), 33 pgs degraded (PG_DEGRADED) 2024-09-10T19:22:20.427 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:22:20.450 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:22:20.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:22:20.451 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:22:20.465 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:22:20 smithi191 ceph-mon[32884]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T19:22:20.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:22:20 smithi191 ceph-mon[32884]: from='client.? 172.21.15.183:0/1586589516' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-10T19:22:20.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:22:20 smithi191 ceph-mon[32884]: osdmap e269: 8 total, 6 up, 6 in 2024-09-10T19:22:20.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:22:20 smithi191 ceph-mon[32884]: 9.3 scrub starts 2024-09-10T19:22:20.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:22:20 smithi191 ceph-mon[32884]: purged_snaps scrub starts 2024-09-10T19:22:20.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:22:20 smithi191 ceph-mon[32884]: purged_snaps scrub ok 2024-09-10T19:22:20.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:22:20 smithi191 ceph-mon[32884]: Health check update: Degraded data redundancy: 45528/271989 objects degraded (16.739%), 33 pgs degraded (PG_DEGRADED) 2024-09-10T19:22:20.546 DEBUG:teuthology.orchestra.run.smithi191:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph daemon osd.6 dump_blocked_ops 2024-09-10T19:22:20.552 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:22:20.575 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:22:20.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:22:20.575 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:22:20.676 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:22:20.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:22:20.723 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:22:20.723 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:22:20.824 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:22:20.876 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:22:20.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:22:20.877 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:22:20.978 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:22:21.000 INFO:teuthology.orchestra.run.smithi191.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.b/config 2024-09-10T19:22:21.000 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:22:21.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:22:21.001 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:22:21.102 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:22:21.126 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:22:21.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:22:21.127 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:22:21.227 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:22:21.267 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:22:21.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:22:21.268 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:22:21.346 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:22:21 smithi183 ceph-mon[29736]: from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:22:21.346 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:22:21 smithi183 ceph-mon[29736]: 5.f scrub starts 2024-09-10T19:22:21.346 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:22:21 smithi183 ceph-mon[29736]: 5.f scrub ok 2024-09-10T19:22:21.346 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:22:21 smithi183 ceph-mon[29736]: pgmap v547: 158 pgs: 45 active+undersized, 33 active+undersized+degraded, 80 active+clean; 5.5 GiB data, 22 GiB used, 514 GiB / 536 GiB avail; 639 B/s rd, 0 B/s wr, 720 op/s; 45528/271989 objects degraded (16.739%) 2024-09-10T19:22:21.346 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:22:21 smithi183 ceph-mon[29736]: osdmap e270: 8 total, 6 up, 6 in 2024-09-10T19:22:21.347 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:22:21 smithi183 ceph-mon[41562]: from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:22:21.347 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:22:21 smithi183 ceph-mon[41562]: 5.f scrub starts 2024-09-10T19:22:21.347 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:22:21 smithi183 ceph-mon[41562]: 5.f scrub ok 2024-09-10T19:22:21.347 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:22:21 smithi183 ceph-mon[41562]: pgmap v547: 158 pgs: 45 active+undersized, 33 active+undersized+degraded, 80 active+clean; 5.5 GiB data, 22 GiB used, 514 GiB / 536 GiB avail; 639 B/s rd, 0 B/s wr, 720 op/s; 45528/271989 objects degraded (16.739%) 2024-09-10T19:22:21.347 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:22:21 smithi183 ceph-mon[41562]: osdmap e270: 8 total, 6 up, 6 in 2024-09-10T19:22:21.347 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Sep 10 19:22:21 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-y[29989]: ::ffff:172.21.15.191 - - [10/Sep/2024:19:22:21] "GET /metrics HTTP/1.1" 200 209508 "" "Prometheus/2.43.0" 2024-09-10T19:22:21.351 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:22:21 smithi191 ceph-mon[32884]: from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:22:21.351 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:22:21 smithi191 ceph-mon[32884]: 5.f scrub starts 2024-09-10T19:22:21.352 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:22:21 smithi191 ceph-mon[32884]: 5.f scrub ok 2024-09-10T19:22:21.352 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:22:21 smithi191 ceph-mon[32884]: pgmap v547: 158 pgs: 45 active+undersized, 33 active+undersized+degraded, 80 active+clean; 5.5 GiB data, 22 GiB used, 514 GiB / 536 GiB avail; 639 B/s rd, 0 B/s wr, 720 op/s; 45528/271989 objects degraded (16.739%) 2024-09-10T19:22:21.352 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:22:21 smithi191 ceph-mon[32884]: osdmap e270: 8 total, 6 up, 6 in 2024-09-10T19:22:21.369 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:22:21.393 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:22:21.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:22:21.394 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:22:21.476 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:22:21.495 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:22:21.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:22:21.529 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:22:21.529 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:22:21.630 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:22:21.651 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:22:21.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:22:21.652 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:22:21.752 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:22:21.774 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:22:21.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:22:21.775 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:22:21.876 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:22:21.899 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:22:21.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:22:21.901 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:22:22.001 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:22:22.045 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:22:22.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:22:22.046 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:22:22.147 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:22:22.181 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:22:22.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:22:22.182 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:22:22.283 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:22:22.306 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:22:22 smithi183 ceph-mon[29736]: 1.4 scrub starts 2024-09-10T19:22:22.307 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:22:22 smithi183 ceph-mon[29736]: 1.4 scrub ok 2024-09-10T19:22:22.307 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:22:22 smithi183 ceph-mon[29736]: 2.14 scrub starts 2024-09-10T19:22:22.307 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:22:22 smithi183 ceph-mon[29736]: 2.14 scrub ok 2024-09-10T19:22:22.307 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:22:22 smithi183 ceph-mon[29736]: 2.d scrub starts 2024-09-10T19:22:22.307 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:22:22 smithi183 ceph-mon[29736]: 2.d scrub ok 2024-09-10T19:22:22.307 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:22:22 smithi183 ceph-mon[41562]: 1.4 scrub starts 2024-09-10T19:22:22.307 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:22:22 smithi183 ceph-mon[41562]: 1.4 scrub ok 2024-09-10T19:22:22.307 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:22:22 smithi183 ceph-mon[41562]: 2.14 scrub starts 2024-09-10T19:22:22.307 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:22:22 smithi183 ceph-mon[41562]: 2.14 scrub ok 2024-09-10T19:22:22.307 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:22:22 smithi183 ceph-mon[41562]: 2.d scrub starts 2024-09-10T19:22:22.308 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:22:22 smithi183 ceph-mon[41562]: 2.d scrub ok 2024-09-10T19:22:22.318 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:22:22.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:22:22.319 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:22:22.414 INFO:teuthology.orchestra.run.smithi183.stderr:Error ENXIO: problem getting command descriptions from osd.0 2024-09-10T19:22:22.414 INFO:teuthology.orchestra.run.smithi183.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-09-10T19:22:22.421 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:22:22.424 INFO:teuthology.orchestra.run.smithi183.stdout:osd.2: {} 2024-09-10T19:22:22.424 INFO:teuthology.orchestra.run.smithi183.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-10T19:22:22.446 INFO:teuthology.orchestra.run.smithi183.stdout:osd.3: {} 2024-09-10T19:22:22.446 INFO:teuthology.orchestra.run.smithi183.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-10T19:22:22.448 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:22:22.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:22:22.449 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:22:22.465 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:22:22 smithi191 ceph-mon[32884]: 1.4 scrub starts 2024-09-10T19:22:22.465 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:22:22 smithi191 ceph-mon[32884]: 1.4 scrub ok 2024-09-10T19:22:22.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:22:22 smithi191 ceph-mon[32884]: 2.14 scrub starts 2024-09-10T19:22:22.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:22:22 smithi191 ceph-mon[32884]: 2.14 scrub ok 2024-09-10T19:22:22.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:22:22 smithi191 ceph-mon[32884]: 2.d scrub starts 2024-09-10T19:22:22.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:22:22 smithi191 ceph-mon[32884]: 2.d scrub ok 2024-09-10T19:22:22.468 INFO:teuthology.orchestra.run.smithi183.stdout:osd.4: {} 2024-09-10T19:22:22.469 INFO:teuthology.orchestra.run.smithi183.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-10T19:22:22.490 INFO:teuthology.orchestra.run.smithi183.stdout:osd.5: {} 2024-09-10T19:22:22.490 INFO:teuthology.orchestra.run.smithi183.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-10T19:22:22.512 INFO:teuthology.orchestra.run.smithi183.stdout:osd.6: {} 2024-09-10T19:22:22.512 INFO:teuthology.orchestra.run.smithi183.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-10T19:22:22.532 INFO:teuthology.orchestra.run.smithi183.stdout:osd.7: {} 2024-09-10T19:22:22.532 INFO:teuthology.orchestra.run.smithi183.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-10T19:22:22.552 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:22:22.576 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:22:22.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:22:22.577 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:22:22.678 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:22:22.704 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:22:22.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:22:22.705 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:22:22.806 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:22:22.829 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:22:22.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:22:22.830 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:22:22.830 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-10T19:22:22.930 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:22:22.955 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:22:22.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:22:22.956 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:22:23.057 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:22:23.093 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:22:23.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:22:23.094 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:22:23.194 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:22:23.229 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:22:23.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:22:23.230 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:22:23.331 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:22:23.367 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:22:23.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:22:23.368 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:22:23.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:22:23 smithi183 ceph-mon[29736]: 9.3 scrub ok 2024-09-10T19:22:23.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:22:23 smithi183 ceph-mon[29736]: pgmap v549: 158 pgs: 1 active+clean+scrubbing, 45 active+undersized, 33 active+undersized+degraded, 79 active+clean; 5.1 GiB data, 21 GiB used, 515 GiB / 536 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.60k op/s; 42168/251982 objects degraded (16.735%) 2024-09-10T19:22:23.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:22:23 smithi183 ceph-mon[29736]: 4.13 scrub starts 2024-09-10T19:22:23.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:22:23 smithi183 ceph-mon[29736]: 4.13 scrub ok 2024-09-10T19:22:23.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:22:23 smithi183 ceph-mon[29736]: 9.d scrub starts 2024-09-10T19:22:23.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:22:23 smithi183 ceph-mon[29736]: from='client.? 172.21.15.183:0/1273608181' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T19:22:23.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:22:23 smithi183 ceph-mon[41562]: 9.3 scrub ok 2024-09-10T19:22:23.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:22:23 smithi183 ceph-mon[41562]: pgmap v549: 158 pgs: 1 active+clean+scrubbing, 45 active+undersized, 33 active+undersized+degraded, 79 active+clean; 5.1 GiB data, 21 GiB used, 515 GiB / 536 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.60k op/s; 42168/251982 objects degraded (16.735%) 2024-09-10T19:22:23.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:22:23 smithi183 ceph-mon[41562]: 4.13 scrub starts 2024-09-10T19:22:23.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:22:23 smithi183 ceph-mon[41562]: 4.13 scrub ok 2024-09-10T19:22:23.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:22:23 smithi183 ceph-mon[41562]: 9.d scrub starts 2024-09-10T19:22:23.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:22:23 smithi183 ceph-mon[41562]: from='client.? 172.21.15.183:0/1273608181' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T19:22:23.465 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:22:23 smithi191 ceph-mon[32884]: 9.3 scrub ok 2024-09-10T19:22:23.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:22:23 smithi191 ceph-mon[32884]: pgmap v549: 158 pgs: 1 active+clean+scrubbing, 45 active+undersized, 33 active+undersized+degraded, 79 active+clean; 5.1 GiB data, 21 GiB used, 515 GiB / 536 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.60k op/s; 42168/251982 objects degraded (16.735%) 2024-09-10T19:22:23.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:22:23 smithi191 ceph-mon[32884]: 4.13 scrub starts 2024-09-10T19:22:23.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:22:23 smithi191 ceph-mon[32884]: 4.13 scrub ok 2024-09-10T19:22:23.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:22:23 smithi191 ceph-mon[32884]: 9.d scrub starts 2024-09-10T19:22:23.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:22:23 smithi191 ceph-mon[32884]: from='client.? 172.21.15.183:0/1273608181' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T19:22:23.469 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:22:23.494 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:22:23.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:22:23.495 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:22:23.509 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-10T19:22:23.510 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-10T19:22:23.596 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:22:23.619 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:22:23.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:22:23.620 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:22:23.720 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:22:23.760 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:22:23.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:22:23.761 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:22:23.862 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:22:23.883 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:22:23.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:22:23.884 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:22:23.985 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:22:24.018 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:22:24.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:22:24.019 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:22:24.120 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:22:24.154 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:22:24.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:22:24.155 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:22:24.256 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:22:24.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:22:24.287 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:22:24.287 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:22:24.368 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:22:24 smithi183 ceph-mon[29736]: 4.11 scrub starts 2024-09-10T19:22:24.368 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:22:24 smithi183 ceph-mon[29736]: 4.11 scrub ok 2024-09-10T19:22:24.368 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:22:24 smithi183 ceph-mon[29736]: 6.2 scrub starts 2024-09-10T19:22:24.368 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:22:24 smithi183 ceph-mon[29736]: 6.2 scrub ok 2024-09-10T19:22:24.369 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:22:24 smithi183 ceph-mon[29736]: 6.1c scrub starts 2024-09-10T19:22:24.369 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:22:24 smithi183 ceph-mon[29736]: 6.1c scrub ok 2024-09-10T19:22:24.369 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:22:24 smithi183 ceph-mon[41562]: 4.11 scrub starts 2024-09-10T19:22:24.369 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:22:24 smithi183 ceph-mon[41562]: 4.11 scrub ok 2024-09-10T19:22:24.369 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:22:24 smithi183 ceph-mon[41562]: 6.2 scrub starts 2024-09-10T19:22:24.369 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:22:24 smithi183 ceph-mon[41562]: 6.2 scrub ok 2024-09-10T19:22:24.369 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:22:24 smithi183 ceph-mon[41562]: 6.1c scrub starts 2024-09-10T19:22:24.369 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:22:24 smithi183 ceph-mon[41562]: 6.1c scrub ok 2024-09-10T19:22:24.388 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:22:24.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:22:24.425 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:22:24.425 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:22:24.440 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:22:24.465 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:22:24 smithi191 ceph-mon[32884]: 4.11 scrub starts 2024-09-10T19:22:24.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:22:24 smithi191 ceph-mon[32884]: 4.11 scrub ok 2024-09-10T19:22:24.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:22:24 smithi191 ceph-mon[32884]: 6.2 scrub starts 2024-09-10T19:22:24.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:22:24 smithi191 ceph-mon[32884]: 6.2 scrub ok 2024-09-10T19:22:24.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:22:24 smithi191 ceph-mon[32884]: 6.1c scrub starts 2024-09-10T19:22:24.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:22:24 smithi191 ceph-mon[32884]: 6.1c scrub ok 2024-09-10T19:22:24.526 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:22:24.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:22:24.551 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:22:24.551 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:22:24.652 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:22:24.683 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:22:24.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:22:24.684 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:22:24.785 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:22:24.809 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:22:24.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:22:24.810 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:22:24.910 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:22:24.937 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:22:24.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:22:24.938 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:22:25.039 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:22:25.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:22:25.084 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:22:25.084 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:22:25.185 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:22:25.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:22:25.231 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:22:25.231 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:22:25.332 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:22:25.333 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:22:25 smithi183 ceph-mon[29736]: pgmap v550: 158 pgs: 1 active+clean+scrubbing, 45 active+undersized, 33 active+undersized+degraded, 79 active+clean; 5.1 GiB data, 21 GiB used, 515 GiB / 536 GiB avail; 639 B/s rd, 0 B/s wr, 833 op/s; 42168/251982 objects degraded (16.735%) 2024-09-10T19:22:25.333 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:22:25 smithi183 ceph-mon[29736]: 2.b scrub starts 2024-09-10T19:22:25.333 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:22:25 smithi183 ceph-mon[29736]: 2.b scrub ok 2024-09-10T19:22:25.334 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:22:25 smithi183 ceph-mon[29736]: 4.5 scrub starts 2024-09-10T19:22:25.334 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:22:25 smithi183 ceph-mon[29736]: 4.5 scrub ok 2024-09-10T19:22:25.334 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:22:25 smithi183 ceph-mon[29736]: Health check update: Degraded data redundancy: 42168/251982 objects degraded (16.735%), 33 pgs degraded (PG_DEGRADED) 2024-09-10T19:22:25.334 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:22:25 smithi183 ceph-mon[41562]: pgmap v550: 158 pgs: 1 active+clean+scrubbing, 45 active+undersized, 33 active+undersized+degraded, 79 active+clean; 5.1 GiB data, 21 GiB used, 515 GiB / 536 GiB avail; 639 B/s rd, 0 B/s wr, 833 op/s; 42168/251982 objects degraded (16.735%) 2024-09-10T19:22:25.334 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:22:25 smithi183 ceph-mon[41562]: 2.b scrub starts 2024-09-10T19:22:25.334 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:22:25 smithi183 ceph-mon[41562]: 2.b scrub ok 2024-09-10T19:22:25.334 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:22:25 smithi183 ceph-mon[41562]: 4.5 scrub starts 2024-09-10T19:22:25.334 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:22:25 smithi183 ceph-mon[41562]: 4.5 scrub ok 2024-09-10T19:22:25.334 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:22:25 smithi183 ceph-mon[41562]: Health check update: Degraded data redundancy: 42168/251982 objects degraded (16.735%), 33 pgs degraded (PG_DEGRADED) 2024-09-10T19:22:25.340 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:22:25 smithi191 ceph-mon[32884]: pgmap v550: 158 pgs: 1 active+clean+scrubbing, 45 active+undersized, 33 active+undersized+degraded, 79 active+clean; 5.1 GiB data, 21 GiB used, 515 GiB / 536 GiB avail; 639 B/s rd, 0 B/s wr, 833 op/s; 42168/251982 objects degraded (16.735%) 2024-09-10T19:22:25.340 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:22:25 smithi191 ceph-mon[32884]: 2.b scrub starts 2024-09-10T19:22:25.340 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:22:25 smithi191 ceph-mon[32884]: 2.b scrub ok 2024-09-10T19:22:25.340 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:22:25 smithi191 ceph-mon[32884]: 4.5 scrub starts 2024-09-10T19:22:25.341 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:22:25 smithi191 ceph-mon[32884]: 4.5 scrub ok 2024-09-10T19:22:25.341 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:22:25 smithi191 ceph-mon[32884]: Health check update: Degraded data redundancy: 42168/251982 objects degraded (16.735%), 33 pgs degraded (PG_DEGRADED) 2024-09-10T19:22:25.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:22:25.374 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:22:25.374 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:22:25.475 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:22:25.491 INFO:teuthology.orchestra.run.smithi191.stdout:{ 2024-09-10T19:22:25.491 INFO:teuthology.orchestra.run.smithi191.stdout: "ops": [], 2024-09-10T19:22:25.491 INFO:teuthology.orchestra.run.smithi191.stdout: "complaint_time": 30, 2024-09-10T19:22:25.491 INFO:teuthology.orchestra.run.smithi191.stdout: "num_blocked_ops": 0 2024-09-10T19:22:25.491 INFO:teuthology.orchestra.run.smithi191.stdout:} 2024-09-10T19:22:25.498 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:22:25.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:22:25.499 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:22:25.510 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-10T19:22:25.600 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:22:25.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:22:25.626 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:22:25.626 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:22:25.726 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:22:25.753 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:22:25.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:22:25.754 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:22:25.855 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:22:25.898 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:22:25.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:22:25.899 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:22:26.000 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:22:26.036 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:22:26.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:22:26.037 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:22:26.122 INFO:teuthology.orchestra.run.smithi183.stderr:set pool 1 pgp_num to 11 2024-09-10T19:22:26.138 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:22:26.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:22:26.166 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:22:26.166 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:22:26.266 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:22:26.304 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:22:26.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:22:26.305 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:22:26.406 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:22:26.430 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:22:26.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:22:26.431 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:22:26.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:22:26 smithi183 ceph-mon[29736]: 9.a scrub starts 2024-09-10T19:22:26.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:22:26 smithi183 ceph-mon[29736]: 9.d scrub ok 2024-09-10T19:22:26.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:22:26 smithi183 ceph-mon[29736]: from='client.? 172.21.15.183:0/3738941961' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "11"}]: dispatch 2024-09-10T19:22:26.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:22:26 smithi183 ceph-mon[41562]: 9.a scrub starts 2024-09-10T19:22:26.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:22:26 smithi183 ceph-mon[41562]: 9.d scrub ok 2024-09-10T19:22:26.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:22:26 smithi183 ceph-mon[41562]: from='client.? 172.21.15.183:0/3738941961' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "11"}]: dispatch 2024-09-10T19:22:26.435 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:22:26 smithi191 ceph-mon[32884]: 9.a scrub starts 2024-09-10T19:22:26.435 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:22:26 smithi191 ceph-mon[32884]: 9.d scrub ok 2024-09-10T19:22:26.435 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:22:26 smithi191 ceph-mon[32884]: from='client.? 172.21.15.183:0/3738941961' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "11"}]: dispatch 2024-09-10T19:22:26.532 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:22:26.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:22:26.558 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:22:26.558 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:22:26.659 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:22:26.698 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:22:26.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:22:26.699 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:22:26.800 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:22:26.830 DEBUG:teuthology.orchestra.run.smithi191:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph daemon osd.6 dump_historic_ops 2024-09-10T19:22:26.848 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:22:26.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:22:26.849 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:22:26.951 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:22:26.989 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:22:26.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:22:26.990 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:22:27.091 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:22:27.115 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:22:27.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:22:27.116 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:22:27.122 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:22:27 smithi191 ceph-mon[32884]: pgmap v551: 158 pgs: 2 active+clean+scrubbing, 45 active+undersized, 33 active+undersized+degraded, 78 active+clean; 4.7 GiB data, 20 GiB used, 516 GiB / 536 GiB avail; 0 B/s wr, 1.59k op/s; 39149/233757 objects degraded (16.748%) 2024-09-10T19:22:27.122 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:22:27 smithi191 ceph-mon[32884]: 5.1c scrub starts 2024-09-10T19:22:27.122 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:22:27 smithi191 ceph-mon[32884]: 5.1c scrub ok 2024-09-10T19:22:27.123 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:22:27 smithi191 ceph-mon[32884]: from='client.? 172.21.15.183:0/3738941961' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "11"}]': finished 2024-09-10T19:22:27.123 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:22:27 smithi191 ceph-mon[32884]: osdmap e271: 8 total, 6 up, 6 in 2024-09-10T19:22:27.123 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:22:27 smithi191 ceph-mon[32884]: 4.17 scrub starts 2024-09-10T19:22:27.123 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:22:27 smithi191 ceph-mon[32884]: 4.17 scrub ok 2024-09-10T19:22:27.216 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:22:27.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:22:27.268 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:22:27.269 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:22:27.285 INFO:teuthology.orchestra.run.smithi191.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.b/config 2024-09-10T19:22:27.370 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:22:27.420 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:22:27.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:22:27.422 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:22:27.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:22:27 smithi183 ceph-mon[29736]: pgmap v551: 158 pgs: 2 active+clean+scrubbing, 45 active+undersized, 33 active+undersized+degraded, 78 active+clean; 4.7 GiB data, 20 GiB used, 516 GiB / 536 GiB avail; 0 B/s wr, 1.59k op/s; 39149/233757 objects degraded (16.748%) 2024-09-10T19:22:27.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:22:27 smithi183 ceph-mon[29736]: 5.1c scrub starts 2024-09-10T19:22:27.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:22:27 smithi183 ceph-mon[29736]: 5.1c scrub ok 2024-09-10T19:22:27.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:22:27 smithi183 ceph-mon[29736]: from='client.? 172.21.15.183:0/3738941961' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "11"}]': finished 2024-09-10T19:22:27.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:22:27 smithi183 ceph-mon[29736]: osdmap e271: 8 total, 6 up, 6 in 2024-09-10T19:22:27.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:22:27 smithi183 ceph-mon[29736]: 4.17 scrub starts 2024-09-10T19:22:27.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:22:27 smithi183 ceph-mon[29736]: 4.17 scrub ok 2024-09-10T19:22:27.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:22:27 smithi183 ceph-mon[41562]: pgmap v551: 158 pgs: 2 active+clean+scrubbing, 45 active+undersized, 33 active+undersized+degraded, 78 active+clean; 4.7 GiB data, 20 GiB used, 516 GiB / 536 GiB avail; 0 B/s wr, 1.59k op/s; 39149/233757 objects degraded (16.748%) 2024-09-10T19:22:27.433 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:22:27 smithi183 ceph-mon[41562]: 5.1c scrub starts 2024-09-10T19:22:27.433 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:22:27 smithi183 ceph-mon[41562]: 5.1c scrub ok 2024-09-10T19:22:27.433 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:22:27 smithi183 ceph-mon[41562]: from='client.? 172.21.15.183:0/3738941961' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "11"}]': finished 2024-09-10T19:22:27.433 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:22:27 smithi183 ceph-mon[41562]: osdmap e271: 8 total, 6 up, 6 in 2024-09-10T19:22:27.433 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:22:27 smithi183 ceph-mon[41562]: 4.17 scrub starts 2024-09-10T19:22:27.433 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:22:27 smithi183 ceph-mon[41562]: 4.17 scrub ok 2024-09-10T19:22:27.523 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:22:27.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:22:27.549 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:22:27.550 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:22:27.651 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:22:27.688 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:22:27.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:22:27.690 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:22:27.715 INFO:journalctl@ceph.mgr.x.smithi191.stdout:Sep 10 19:22:27 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-x[34967]: ::ffff:172.21.15.191 - - [10/Sep/2024:19:22:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T19:22:27.790 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:22:27.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:22:27.824 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:22:27.824 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:22:27.925 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:22:27.964 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:22:27.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:22:27.966 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:22:28.066 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:22:28.088 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:22:28.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:22:28.089 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:22:28.136 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:22:28.191 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:22:28.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:22:28 smithi191 ceph-mon[32884]: 9.a scrub ok 2024-09-10T19:22:28.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:22:28.227 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:22:28.228 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:22:28.328 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:22:28.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:22:28.368 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:22:28.368 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:22:28.401 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:22:28 smithi183 ceph-mon[29736]: 9.a scrub ok 2024-09-10T19:22:28.401 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:22:28 smithi183 ceph-mon[41562]: 9.a scrub ok 2024-09-10T19:22:28.469 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:22:28.509 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:22:28.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:22:28.510 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:22:28.611 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:22:28.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:22:28.646 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:22:28.647 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:22:28.747 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:22:28.781 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:22:28.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:22:28.782 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:22:28.883 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:22:28.926 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:22:28.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:22:28.927 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:22:28.931 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Sep 10 19:22:28 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-y[29989]: 2024-09-10T19:22:28.668+0000 7f9fad6cd640 -1 mgr get_metadata_python Requested missing service osd.1 2024-09-10T19:22:29.028 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:22:29.051 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:22:29.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:22:29.052 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:22:29.139 INFO:teuthology.orchestra.run.smithi183.stderr:nodeep-scrub is unset 2024-09-10T19:22:29.153 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:22:29.190 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:22:29.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:22:29.191 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:22:29.281 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:22:29 smithi191 ceph-mon[32884]: pgmap v553: 158 pgs: 2 active+clean+scrubbing, 45 active+undersized, 33 active+undersized+degraded, 78 active+clean; 4.7 GiB data, 20 GiB used, 516 GiB / 536 GiB avail; 0 B/s wr, 1.59k op/s; 39149/233757 objects degraded (16.748%) 2024-09-10T19:22:29.281 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:22:29 smithi191 ceph-mon[32884]: 5.1 scrub starts 2024-09-10T19:22:29.281 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:22:29 smithi191 ceph-mon[32884]: 5.1 scrub ok 2024-09-10T19:22:29.281 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:22:29 smithi191 ceph-mon[32884]: 6.15 scrub starts 2024-09-10T19:22:29.281 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:22:29 smithi191 ceph-mon[32884]: 6.15 scrub ok 2024-09-10T19:22:29.281 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:22:29 smithi191 ceph-mon[32884]: from='client.? 172.21.15.183:0/1815671106' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-10T19:22:29.281 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:22:29 smithi191 ceph-mon[32884]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-10T19:22:29.281 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:22:29 smithi191 ceph-mon[32884]: 4.10 scrub starts 2024-09-10T19:22:29.282 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:22:29 smithi191 ceph-mon[32884]: 4.10 scrub ok 2024-09-10T19:22:29.282 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:22:29 smithi191 ceph-mon[32884]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:22:29.282 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:22:29 smithi191 ceph-mon[32884]: 4.a scrub starts 2024-09-10T19:22:29.282 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:22:29 smithi191 ceph-mon[32884]: 4.a scrub ok 2024-09-10T19:22:29.291 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:22:29.320 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:22:29.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:22:29.321 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:22:29.422 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:22:29.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:22:29 smithi183 ceph-mon[29736]: pgmap v553: 158 pgs: 2 active+clean+scrubbing, 45 active+undersized, 33 active+undersized+degraded, 78 active+clean; 4.7 GiB data, 20 GiB used, 516 GiB / 536 GiB avail; 0 B/s wr, 1.59k op/s; 39149/233757 objects degraded (16.748%) 2024-09-10T19:22:29.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:22:29 smithi183 ceph-mon[29736]: 5.1 scrub starts 2024-09-10T19:22:29.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:22:29 smithi183 ceph-mon[29736]: 5.1 scrub ok 2024-09-10T19:22:29.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:22:29 smithi183 ceph-mon[29736]: 6.15 scrub starts 2024-09-10T19:22:29.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:22:29 smithi183 ceph-mon[29736]: 6.15 scrub ok 2024-09-10T19:22:29.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:22:29 smithi183 ceph-mon[29736]: from='client.? 172.21.15.183:0/1815671106' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-10T19:22:29.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:22:29 smithi183 ceph-mon[29736]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-10T19:22:29.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:22:29 smithi183 ceph-mon[29736]: 4.10 scrub starts 2024-09-10T19:22:29.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:22:29 smithi183 ceph-mon[29736]: 4.10 scrub ok 2024-09-10T19:22:29.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:22:29 smithi183 ceph-mon[29736]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:22:29.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:22:29 smithi183 ceph-mon[29736]: 4.a scrub starts 2024-09-10T19:22:29.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:22:29 smithi183 ceph-mon[29736]: 4.a scrub ok 2024-09-10T19:22:29.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:22:29 smithi183 ceph-mon[41562]: pgmap v553: 158 pgs: 2 active+clean+scrubbing, 45 active+undersized, 33 active+undersized+degraded, 78 active+clean; 4.7 GiB data, 20 GiB used, 516 GiB / 536 GiB avail; 0 B/s wr, 1.59k op/s; 39149/233757 objects degraded (16.748%) 2024-09-10T19:22:29.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:22:29 smithi183 ceph-mon[41562]: 5.1 scrub starts 2024-09-10T19:22:29.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:22:29 smithi183 ceph-mon[41562]: 5.1 scrub ok 2024-09-10T19:22:29.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:22:29 smithi183 ceph-mon[41562]: 6.15 scrub starts 2024-09-10T19:22:29.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:22:29 smithi183 ceph-mon[41562]: 6.15 scrub ok 2024-09-10T19:22:29.433 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:22:29 smithi183 ceph-mon[41562]: from='client.? 172.21.15.183:0/1815671106' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-10T19:22:29.433 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:22:29 smithi183 ceph-mon[41562]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-10T19:22:29.433 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:22:29 smithi183 ceph-mon[41562]: 4.10 scrub starts 2024-09-10T19:22:29.433 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:22:29 smithi183 ceph-mon[41562]: 4.10 scrub ok 2024-09-10T19:22:29.433 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:22:29 smithi183 ceph-mon[41562]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:22:29.433 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:22:29 smithi183 ceph-mon[41562]: 4.a scrub starts 2024-09-10T19:22:29.433 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:22:29 smithi183 ceph-mon[41562]: 4.a scrub ok 2024-09-10T19:22:29.466 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:22:29.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:22:29.467 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:22:29.568 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:22:29.597 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:22:29.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:22:29.598 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:22:29.699 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:22:29.725 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:22:29.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:22:29.726 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:22:29.827 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:22:29.865 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:22:29.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:22:29.866 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:22:29.965 INFO:journalctl@ceph.iscsi.iscsi.a.smithi191.stdout:Sep 10 19:22:29 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-iscsi-iscsi-a[61816]: debug there is no tcmu-runner data available 2024-09-10T19:22:29.967 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:22:30.006 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:22:30.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:22:30.007 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:22:30.107 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:22:30.135 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:22:30.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:22:30.136 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:22:30.141 INFO:teuthology.orchestra.run.smithi183.stderr:Error ENXIO: problem getting command descriptions from osd.0 2024-09-10T19:22:30.141 INFO:teuthology.orchestra.run.smithi183.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-09-10T19:22:30.141 INFO:teuthology.orchestra.run.smithi183.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-10T19:22:30.142 INFO:teuthology.orchestra.run.smithi183.stdout:osd.2: {} 2024-09-10T19:22:30.152 INFO:teuthology.orchestra.run.smithi183.stdout:osd.3: {} 2024-09-10T19:22:30.153 INFO:teuthology.orchestra.run.smithi183.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-10T19:22:30.171 INFO:teuthology.orchestra.run.smithi183.stdout:osd.4: {} 2024-09-10T19:22:30.171 INFO:teuthology.orchestra.run.smithi183.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-10T19:22:30.194 INFO:teuthology.orchestra.run.smithi183.stdout:osd.5: {} 2024-09-10T19:22:30.194 INFO:teuthology.orchestra.run.smithi183.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-10T19:22:30.218 INFO:teuthology.orchestra.run.smithi183.stdout:osd.6: {} 2024-09-10T19:22:30.218 INFO:teuthology.orchestra.run.smithi183.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-10T19:22:30.236 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:22:30.238 INFO:teuthology.orchestra.run.smithi183.stdout:osd.7: {} 2024-09-10T19:22:30.238 INFO:teuthology.orchestra.run.smithi183.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-10T19:22:30.272 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:22:30.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:22:30.273 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:22:30.374 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:22:30.409 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:22:30.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:22:30.410 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:22:30.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:22:30 smithi183 ceph-mon[29736]: 4.1b scrub starts 2024-09-10T19:22:30.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:22:30 smithi183 ceph-mon[29736]: 4.1b scrub ok 2024-09-10T19:22:30.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:22:30 smithi183 ceph-mon[29736]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-10T19:22:30.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:22:30 smithi183 ceph-mon[29736]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-10T19:22:30.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:22:30 smithi183 ceph-mon[29736]: osdmap e272: 8 total, 6 up, 6 in 2024-09-10T19:22:30.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:22:30 smithi183 ceph-mon[29736]: 5.9 scrub starts 2024-09-10T19:22:30.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:22:30 smithi183 ceph-mon[29736]: 5.9 scrub ok 2024-09-10T19:22:30.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:22:30 smithi183 ceph-mon[29736]: Health check update: Degraded data redundancy: 39149/233757 objects degraded (16.748%), 33 pgs degraded (PG_DEGRADED) 2024-09-10T19:22:30.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:22:30 smithi183 ceph-mon[29736]: from='client.? 172.21.15.183:0/2581623449' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T19:22:30.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:22:30 smithi183 ceph-mon[41562]: 4.1b scrub starts 2024-09-10T19:22:30.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:22:30 smithi183 ceph-mon[41562]: 4.1b scrub ok 2024-09-10T19:22:30.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:22:30 smithi183 ceph-mon[41562]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-10T19:22:30.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:22:30 smithi183 ceph-mon[41562]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-10T19:22:30.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:22:30 smithi183 ceph-mon[41562]: osdmap e272: 8 total, 6 up, 6 in 2024-09-10T19:22:30.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:22:30 smithi183 ceph-mon[41562]: 5.9 scrub starts 2024-09-10T19:22:30.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:22:30 smithi183 ceph-mon[41562]: 5.9 scrub ok 2024-09-10T19:22:30.433 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:22:30 smithi183 ceph-mon[41562]: Health check update: Degraded data redundancy: 39149/233757 objects degraded (16.748%), 33 pgs degraded (PG_DEGRADED) 2024-09-10T19:22:30.433 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:22:30 smithi183 ceph-mon[41562]: from='client.? 172.21.15.183:0/2581623449' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T19:22:30.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:22:30 smithi191 ceph-mon[32884]: 4.1b scrub starts 2024-09-10T19:22:30.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:22:30 smithi191 ceph-mon[32884]: 4.1b scrub ok 2024-09-10T19:22:30.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:22:30 smithi191 ceph-mon[32884]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-10T19:22:30.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:22:30 smithi191 ceph-mon[32884]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-10T19:22:30.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:22:30 smithi191 ceph-mon[32884]: osdmap e272: 8 total, 6 up, 6 in 2024-09-10T19:22:30.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:22:30 smithi191 ceph-mon[32884]: 5.9 scrub starts 2024-09-10T19:22:30.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:22:30 smithi191 ceph-mon[32884]: 5.9 scrub ok 2024-09-10T19:22:30.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:22:30 smithi191 ceph-mon[32884]: Health check update: Degraded data redundancy: 39149/233757 objects degraded (16.748%), 33 pgs degraded (PG_DEGRADED) 2024-09-10T19:22:30.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:22:30 smithi191 ceph-mon[32884]: from='client.? 172.21.15.183:0/2581623449' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T19:22:30.511 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:22:30.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:22:30.549 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:22:30.549 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:22:30.650 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:22:30.673 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:22:30.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:22:30.674 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:22:30.775 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:22:30.798 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:22:30.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:22:30.799 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:22:30.900 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:22:30.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:22:30.924 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:22:30.924 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:22:31.025 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:22:31.066 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:22:31.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:22:31.067 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:22:31.168 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:22:31.207 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:22:31.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:22:31.208 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:22:31.309 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:22:31.316 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:22:31 smithi183 ceph-mon[29736]: from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:22:31.316 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:22:31 smithi183 ceph-mon[29736]: pgmap v555: 158 pgs: 2 active+clean+scrubbing, 45 active+undersized, 33 active+undersized+degraded, 78 active+clean; 4.7 GiB data, 20 GiB used, 516 GiB / 536 GiB avail; 0 B/s wr, 758 op/s; 39149/233757 objects degraded (16.748%) 2024-09-10T19:22:31.317 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:22:31 smithi183 ceph-mon[29736]: 6.8 scrub starts 2024-09-10T19:22:31.317 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:22:31 smithi183 ceph-mon[29736]: 6.8 scrub ok 2024-09-10T19:22:31.317 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:22:31 smithi183 ceph-mon[29736]: 9.1 scrub starts 2024-09-10T19:22:31.317 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:22:31 smithi183 ceph-mon[41562]: from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:22:31.317 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:22:31 smithi183 ceph-mon[41562]: pgmap v555: 158 pgs: 2 active+clean+scrubbing, 45 active+undersized, 33 active+undersized+degraded, 78 active+clean; 4.7 GiB data, 20 GiB used, 516 GiB / 536 GiB avail; 0 B/s wr, 758 op/s; 39149/233757 objects degraded (16.748%) 2024-09-10T19:22:31.317 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:22:31 smithi183 ceph-mon[41562]: 6.8 scrub starts 2024-09-10T19:22:31.317 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:22:31 smithi183 ceph-mon[41562]: 6.8 scrub ok 2024-09-10T19:22:31.317 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:22:31 smithi183 ceph-mon[41562]: 9.1 scrub starts 2024-09-10T19:22:31.333 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:22:31.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:22:31.334 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:22:31.408 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:22:31 smithi191 ceph-mon[32884]: from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:22:31.408 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:22:31 smithi191 ceph-mon[32884]: pgmap v555: 158 pgs: 2 active+clean+scrubbing, 45 active+undersized, 33 active+undersized+degraded, 78 active+clean; 4.7 GiB data, 20 GiB used, 516 GiB / 536 GiB avail; 0 B/s wr, 758 op/s; 39149/233757 objects degraded (16.748%) 2024-09-10T19:22:31.408 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:22:31 smithi191 ceph-mon[32884]: 6.8 scrub starts 2024-09-10T19:22:31.408 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:22:31 smithi191 ceph-mon[32884]: 6.8 scrub ok 2024-09-10T19:22:31.408 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:22:31 smithi191 ceph-mon[32884]: 9.1 scrub starts 2024-09-10T19:22:31.435 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:22:31.472 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:22:31.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:22:31.473 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:22:31.553 INFO:teuthology.orchestra.run.smithi191.stdout:ERROR: (22) Invalid argument 2024-09-10T19:22:31.553 INFO:teuthology.orchestra.run.smithi191.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-10T19:22:31.573 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:22:31.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:22:31.610 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:22:31.610 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:22:31.681 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Sep 10 19:22:31 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-y[29989]: ::ffff:172.21.15.191 - - [10/Sep/2024:19:22:31] "GET /metrics HTTP/1.1" 200 209534 "" "Prometheus/2.43.0" 2024-09-10T19:22:31.710 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:22:31.747 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:22:31.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:22:31.748 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:22:31.848 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:22:31.878 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-10T19:22:31.879 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-10T19:22:31.888 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:22:31.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:22:31.889 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:22:31.990 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:22:32.020 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:22:32.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:22:32.021 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:22:32.122 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:22:32.144 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:22:32.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:22:32.145 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:22:32.246 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:22:32.269 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:22:32.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:22:32.270 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:22:32.287 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:22:32 smithi191 ceph-mon[32884]: 5.16 scrub starts 2024-09-10T19:22:32.287 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:22:32 smithi191 ceph-mon[32884]: 5.16 scrub ok 2024-09-10T19:22:32.287 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:22:32 smithi191 ceph-mon[32884]: 6.12 scrub starts 2024-09-10T19:22:32.287 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:22:32 smithi191 ceph-mon[32884]: 6.12 scrub ok 2024-09-10T19:22:32.287 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:22:32 smithi191 ceph-mon[32884]: 9.1 scrub ok 2024-09-10T19:22:32.327 DEBUG:teuthology.orchestra.run.smithi191:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph daemon osd.7 dump_ops_in_flight 2024-09-10T19:22:32.370 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:22:32.392 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:22:32.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:22:32.393 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:22:32.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:22:32 smithi183 ceph-mon[29736]: 5.16 scrub starts 2024-09-10T19:22:32.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:22:32 smithi183 ceph-mon[29736]: 5.16 scrub ok 2024-09-10T19:22:32.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:22:32 smithi183 ceph-mon[29736]: 6.12 scrub starts 2024-09-10T19:22:32.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:22:32 smithi183 ceph-mon[29736]: 6.12 scrub ok 2024-09-10T19:22:32.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:22:32 smithi183 ceph-mon[29736]: 9.1 scrub ok 2024-09-10T19:22:32.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:22:32 smithi183 ceph-mon[41562]: 5.16 scrub starts 2024-09-10T19:22:32.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:22:32 smithi183 ceph-mon[41562]: 5.16 scrub ok 2024-09-10T19:22:32.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:22:32 smithi183 ceph-mon[41562]: 6.12 scrub starts 2024-09-10T19:22:32.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:22:32 smithi183 ceph-mon[41562]: 6.12 scrub ok 2024-09-10T19:22:32.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:22:32 smithi183 ceph-mon[41562]: 9.1 scrub ok 2024-09-10T19:22:32.495 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:22:32.518 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:22:32.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:22:32.519 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:22:32.620 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:22:32.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:22:32.665 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:22:32.665 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:22:32.761 INFO:teuthology.orchestra.run.smithi191.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.b/config 2024-09-10T19:22:32.765 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:22:32.805 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:22:32.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:22:32.806 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:22:32.863 INFO:tasks.thrashosds.thrasher:in_osds: [0, 3, 4, 5, 6, 7] out_osds: [1, 2] dead_osds: [1, 0] live_osds: [2, 3, 4, 5, 6, 7] 2024-09-10T19:22:32.863 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-10T19:22:32.863 INFO:tasks.thrashosds.thrasher:Reviving osd 1 2024-09-10T19:22:32.863 INFO:tasks.cephadm.osd.1:Restarting osd.1 (starting--it wasn't running)... 2024-09-10T19:22:32.864 DEBUG:teuthology.orchestra.run.smithi183:osd.1> sudo journalctl -f -n 0 -u ceph-320f6df8-6fa7-11ef-bcea-c7b262605968@osd.1.service 2024-09-10T19:22:32.866 DEBUG:teuthology.orchestra.run.smithi183:> sudo systemctl start ceph-320f6df8-6fa7-11ef-bcea-c7b262605968@osd.1 2024-09-10T19:22:32.907 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:22:32.947 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:22:32.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:22:32.948 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:22:32.968 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph osd set noscrub 2024-09-10T19:22:33.048 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:22:33.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:22:33.083 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:22:33.083 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:22:33.152 INFO:journalctl@ceph.osd.1.smithi183.stdout:Sep 10 19:22:32 smithi183 systemd[1]: Starting Ceph osd.1 for 320f6df8-6fa7-11ef-bcea-c7b262605968... 2024-09-10T19:22:33.184 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:22:33.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:22:33 smithi191 ceph-mon[32884]: 2.1a scrub starts 2024-09-10T19:22:33.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:22:33 smithi191 ceph-mon[32884]: 2.1a scrub ok 2024-09-10T19:22:33.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:22:33 smithi191 ceph-mon[32884]: pgmap v556: 158 pgs: 45 active+undersized, 33 active+undersized+degraded, 80 active+clean; 4.4 GiB data, 20 GiB used, 517 GiB / 536 GiB avail; 0 B/s wr, 1.50k op/s; 36170/215910 objects degraded (16.752%) 2024-09-10T19:22:33.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:22:33 smithi191 ceph-mon[32884]: 2.6 scrub starts 2024-09-10T19:22:33.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:22:33.217 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:22:33.217 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:22:33.318 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:22:33.353 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:22:33.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:22:33.355 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:22:33.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:22:33 smithi183 ceph-mon[29736]: 2.1a scrub starts 2024-09-10T19:22:33.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:22:33 smithi183 ceph-mon[29736]: 2.1a scrub ok 2024-09-10T19:22:33.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:22:33 smithi183 ceph-mon[29736]: pgmap v556: 158 pgs: 45 active+undersized, 33 active+undersized+degraded, 80 active+clean; 4.4 GiB data, 20 GiB used, 517 GiB / 536 GiB avail; 0 B/s wr, 1.50k op/s; 36170/215910 objects degraded (16.752%) 2024-09-10T19:22:33.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:22:33 smithi183 ceph-mon[29736]: 2.6 scrub starts 2024-09-10T19:22:33.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:22:33 smithi183 ceph-mon[41562]: 2.1a scrub starts 2024-09-10T19:22:33.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:22:33 smithi183 ceph-mon[41562]: 2.1a scrub ok 2024-09-10T19:22:33.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:22:33 smithi183 ceph-mon[41562]: pgmap v556: 158 pgs: 45 active+undersized, 33 active+undersized+degraded, 80 active+clean; 4.4 GiB data, 20 GiB used, 517 GiB / 536 GiB avail; 0 B/s wr, 1.50k op/s; 36170/215910 objects degraded (16.752%) 2024-09-10T19:22:33.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:22:33 smithi183 ceph-mon[41562]: 2.6 scrub starts 2024-09-10T19:22:33.432 INFO:journalctl@ceph.osd.1.smithi183.stdout:Sep 10 19:22:33 smithi183 podman[160905]: 2024-09-10 19:22:33.088583967 +0000 UTC m=+0.022195250 image pull 4c55dd4096115e48502f368cff1eab06769f71f030e1692e2611bbc6c6f637e9 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3bfeaf4417585c821cab9ca76e4d95f390b0e769807122a45463640820b5cd16 2024-09-10T19:22:33.456 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:22:33.489 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:22:33.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:22:33.490 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:22:33.591 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:22:33.615 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:22:33.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:22:33.616 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:22:33.716 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:22:33.757 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:22:33.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:22:33.758 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:22:33.859 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:22:33.880 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-10T19:22:33.882 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:22:33.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:22:33.883 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:22:33.985 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:22:34.007 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:22:34.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:22:34.008 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:22:34.108 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:22:34.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:22:34.147 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:22:34.147 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:22:34.158 INFO:journalctl@ceph.osd.1.smithi183.stdout:Sep 10 19:22:33 smithi183 podman[160905]: 2024-09-10 19:22:33.785570259 +0000 UTC m=+0.719181518 container create 61fcfaa2a47c8bfd34ffcb50cbc25a434c1b1af910e9b8990ec594936be56399 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3bfeaf4417585c821cab9ca76e4d95f390b0e769807122a45463640820b5cd16, name=ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd-1-activate, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, RELEASE=wip-guits-quincy-2024-09-09-1810-quincy-5be5739, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, ceph=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_CLEAN=True) 2024-09-10T19:22:34.248 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:22:34.270 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:22:34.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:22:34.271 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:22:34.372 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:22:34.386 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:22:34 smithi191 ceph-mon[32884]: 2.6 scrub ok 2024-09-10T19:22:34.410 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:22:34.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:22:34.411 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:22:34.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:22:34 smithi183 ceph-mon[29736]: 2.6 scrub ok 2024-09-10T19:22:34.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:22:34 smithi183 ceph-mon[41562]: 2.6 scrub ok 2024-09-10T19:22:34.432 INFO:journalctl@ceph.osd.1.smithi183.stdout:Sep 10 19:22:34 smithi183 podman[160905]: 2024-09-10 19:22:34.283593174 +0000 UTC m=+1.217204434 container init 61fcfaa2a47c8bfd34ffcb50cbc25a434c1b1af910e9b8990ec594936be56399 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3bfeaf4417585c821cab9ca76e4d95f390b0e769807122a45463640820b5cd16, name=ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd-1-activate, org.label-schema.license=GPLv2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.schema-version=1.0, RELEASE=wip-guits-quincy-2024-09-09-1810-quincy-5be5739, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_CLEAN=True) 2024-09-10T19:22:34.432 INFO:journalctl@ceph.osd.1.smithi183.stdout:Sep 10 19:22:34 smithi183 podman[160905]: 2024-09-10 19:22:34.289783881 +0000 UTC m=+1.223395148 container start 61fcfaa2a47c8bfd34ffcb50cbc25a434c1b1af910e9b8990ec594936be56399 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3bfeaf4417585c821cab9ca76e4d95f390b0e769807122a45463640820b5cd16, name=ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd-1-activate, ceph=True, maintainer=Guillaume Abrioux , RELEASE=wip-guits-quincy-2024-09-09-1810-quincy-5be5739, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0) 2024-09-10T19:22:34.432 INFO:journalctl@ceph.osd.1.smithi183.stdout:Sep 10 19:22:34 smithi183 podman[160905]: 2024-09-10 19:22:34.397165379 +0000 UTC m=+1.330776657 container attach 61fcfaa2a47c8bfd34ffcb50cbc25a434c1b1af910e9b8990ec594936be56399 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3bfeaf4417585c821cab9ca76e4d95f390b0e769807122a45463640820b5cd16, name=ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd-1-activate, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, GIT_CLEAN=True, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, RELEASE=wip-guits-quincy-2024-09-09-1810-quincy-5be5739, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-10T19:22:34.441 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:22:34.512 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:22:34.552 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:22:34.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:22:34.554 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:22:34.656 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:22:34.697 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:22:34.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:22:34.699 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:22:34.799 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:22:34.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:22:34.843 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:22:34.843 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:22:34.853 INFO:teuthology.orchestra.run.smithi191.stdout:ERROR: (22) Invalid argument 2024-09-10T19:22:34.853 INFO:teuthology.orchestra.run.smithi191.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-10T19:22:34.932 INFO:journalctl@ceph.osd.1.smithi183.stdout:Sep 10 19:22:34 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd-1-activate[161015]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-09-10T19:22:34.932 INFO:journalctl@ceph.osd.1.smithi183.stdout:Sep 10 19:22:34 smithi183 bash[160905]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-09-10T19:22:34.932 INFO:journalctl@ceph.osd.1.smithi183.stdout:Sep 10 19:22:34 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd-1-activate[161015]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-1 --no-mon-config --dev /dev/mapper/vg_nvme-lv_3 2024-09-10T19:22:34.932 INFO:journalctl@ceph.osd.1.smithi183.stdout:Sep 10 19:22:34 smithi183 bash[160905]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-1 --no-mon-config --dev /dev/mapper/vg_nvme-lv_3 2024-09-10T19:22:34.932 INFO:journalctl@ceph.osd.1.smithi183.stdout:Sep 10 19:22:34 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd-1-activate[161015]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_3 2024-09-10T19:22:34.932 INFO:journalctl@ceph.osd.1.smithi183.stdout:Sep 10 19:22:34 smithi183 bash[160905]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_3 2024-09-10T19:22:34.932 INFO:journalctl@ceph.osd.1.smithi183.stdout:Sep 10 19:22:34 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd-1-activate[161015]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-2 2024-09-10T19:22:34.932 INFO:journalctl@ceph.osd.1.smithi183.stdout:Sep 10 19:22:34 smithi183 bash[160905]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-2 2024-09-10T19:22:34.932 INFO:journalctl@ceph.osd.1.smithi183.stdout:Sep 10 19:22:34 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd-1-activate[161015]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-1/block 2024-09-10T19:22:34.933 INFO:journalctl@ceph.osd.1.smithi183.stdout:Sep 10 19:22:34 smithi183 bash[160905]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-1/block 2024-09-10T19:22:34.933 INFO:journalctl@ceph.osd.1.smithi183.stdout:Sep 10 19:22:34 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd-1-activate[161015]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-09-10T19:22:34.933 INFO:journalctl@ceph.osd.1.smithi183.stdout:Sep 10 19:22:34 smithi183 bash[160905]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-09-10T19:22:34.933 INFO:journalctl@ceph.osd.1.smithi183.stdout:Sep 10 19:22:34 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd-1-activate[161015]: --> ceph-volume raw activate successful for osd ID: 1 2024-09-10T19:22:34.933 INFO:journalctl@ceph.osd.1.smithi183.stdout:Sep 10 19:22:34 smithi183 bash[160905]: --> ceph-volume raw activate successful for osd ID: 1 2024-09-10T19:22:34.933 INFO:journalctl@ceph.osd.1.smithi183.stdout:Sep 10 19:22:34 smithi183 podman[160905]: 2024-09-10 19:22:34.737161332 +0000 UTC m=+1.670772592 container died 61fcfaa2a47c8bfd34ffcb50cbc25a434c1b1af910e9b8990ec594936be56399 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3bfeaf4417585c821cab9ca76e4d95f390b0e769807122a45463640820b5cd16, name=ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd-1-activate, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , RELEASE=wip-guits-quincy-2024-09-09-1810-quincy-5be5739, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, GIT_CLEAN=True) 2024-09-10T19:22:34.944 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:22:34.979 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:22:34.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:22:34.979 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:22:35.080 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:22:35.120 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:22:35.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:22:35.122 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:22:35.222 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:22:35.262 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:22:35.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:22:35.263 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:22:35.364 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:22:35.403 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:22:35.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:22:35.404 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:22:35.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:22:35 smithi183 ceph-mon[29736]: 2.3 scrub starts 2024-09-10T19:22:35.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:22:35 smithi183 ceph-mon[29736]: 2.3 scrub ok 2024-09-10T19:22:35.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:22:35 smithi183 ceph-mon[29736]: pgmap v557: 158 pgs: 45 active+undersized, 33 active+undersized+degraded, 80 active+clean; 4.4 GiB data, 20 GiB used, 517 GiB / 536 GiB avail; 639 B/s rd, 0 B/s wr, 744 op/s; 36170/215910 objects degraded (16.752%) 2024-09-10T19:22:35.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:22:35 smithi183 ceph-mon[29736]: purged_snaps scrub starts 2024-09-10T19:22:35.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:22:35 smithi183 ceph-mon[29736]: purged_snaps scrub ok 2024-09-10T19:22:35.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:22:35 smithi183 ceph-mon[29736]: Health check update: Degraded data redundancy: 36170/215910 objects degraded (16.752%), 33 pgs degraded (PG_DEGRADED) 2024-09-10T19:22:35.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:22:35 smithi183 ceph-mon[29736]: purged_snaps scrub starts 2024-09-10T19:22:35.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:22:35 smithi183 ceph-mon[29736]: purged_snaps scrub ok 2024-09-10T19:22:35.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:22:35 smithi183 ceph-mon[41562]: 2.3 scrub starts 2024-09-10T19:22:35.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:22:35 smithi183 ceph-mon[41562]: 2.3 scrub ok 2024-09-10T19:22:35.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:22:35 smithi183 ceph-mon[41562]: pgmap v557: 158 pgs: 45 active+undersized, 33 active+undersized+degraded, 80 active+clean; 4.4 GiB data, 20 GiB used, 517 GiB / 536 GiB avail; 639 B/s rd, 0 B/s wr, 744 op/s; 36170/215910 objects degraded (16.752%) 2024-09-10T19:22:35.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:22:35 smithi183 ceph-mon[41562]: purged_snaps scrub starts 2024-09-10T19:22:35.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:22:35 smithi183 ceph-mon[41562]: purged_snaps scrub ok 2024-09-10T19:22:35.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:22:35 smithi183 ceph-mon[41562]: Health check update: Degraded data redundancy: 36170/215910 objects degraded (16.752%), 33 pgs degraded (PG_DEGRADED) 2024-09-10T19:22:35.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:22:35 smithi183 ceph-mon[41562]: purged_snaps scrub starts 2024-09-10T19:22:35.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:22:35 smithi183 ceph-mon[41562]: purged_snaps scrub ok 2024-09-10T19:22:35.465 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:22:35 smithi191 ceph-mon[32884]: 2.3 scrub starts 2024-09-10T19:22:35.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:22:35 smithi191 ceph-mon[32884]: 2.3 scrub ok 2024-09-10T19:22:35.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:22:35 smithi191 ceph-mon[32884]: pgmap v557: 158 pgs: 45 active+undersized, 33 active+undersized+degraded, 80 active+clean; 4.4 GiB data, 20 GiB used, 517 GiB / 536 GiB avail; 639 B/s rd, 0 B/s wr, 744 op/s; 36170/215910 objects degraded (16.752%) 2024-09-10T19:22:35.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:22:35 smithi191 ceph-mon[32884]: purged_snaps scrub starts 2024-09-10T19:22:35.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:22:35 smithi191 ceph-mon[32884]: purged_snaps scrub ok 2024-09-10T19:22:35.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:22:35 smithi191 ceph-mon[32884]: Health check update: Degraded data redundancy: 36170/215910 objects degraded (16.752%), 33 pgs degraded (PG_DEGRADED) 2024-09-10T19:22:35.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:22:35 smithi191 ceph-mon[32884]: purged_snaps scrub starts 2024-09-10T19:22:35.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:22:35 smithi191 ceph-mon[32884]: purged_snaps scrub ok 2024-09-10T19:22:35.505 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:22:35.540 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:22:35.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:22:35.541 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:22:35.641 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:22:35.685 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:22:35.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:22:35.686 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:22:35.787 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:22:35.818 DEBUG:teuthology.orchestra.run.smithi191:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph daemon osd.7 dump_blocked_ops 2024-09-10T19:22:35.834 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:22:35.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:22:35.835 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:22:35.931 INFO:journalctl@ceph.osd.1.smithi183.stdout:Sep 10 19:22:35 smithi183 podman[161217]: 2024-09-10 19:22:35.496217409 +0000 UTC m=+0.830621804 container remove 61fcfaa2a47c8bfd34ffcb50cbc25a434c1b1af910e9b8990ec594936be56399 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3bfeaf4417585c821cab9ca76e4d95f390b0e769807122a45463640820b5cd16, name=ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd-1-activate, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-guits-quincy-2024-09-09-1810-quincy-5be5739, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, ceph=True, io.buildah.version=1.37.2, GIT_CLEAN=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=) 2024-09-10T19:22:35.937 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:22:35.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:22:35.986 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:22:35.986 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:22:36.087 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:22:36.113 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:22:36.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:22:36.114 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:22:36.215 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:22:36.275 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:22:36.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:22:36.276 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:22:36.279 INFO:teuthology.orchestra.run.smithi191.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.b/config 2024-09-10T19:22:36.377 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:22:36.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:22:36.421 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:22:36.422 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:22:36.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:22:36 smithi183 ceph-mon[29736]: 9.f scrub starts 2024-09-10T19:22:36.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:22:36 smithi183 ceph-mon[29736]: 1.7 scrub starts 2024-09-10T19:22:36.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:22:36 smithi183 ceph-mon[29736]: 1.7 scrub ok 2024-09-10T19:22:36.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:22:36 smithi183 ceph-mon[29736]: osdmap e273: 8 total, 6 up, 6 in 2024-09-10T19:22:36.431 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:22:36 smithi183 ceph-mon[41562]: 9.f scrub starts 2024-09-10T19:22:36.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:22:36 smithi183 ceph-mon[41562]: 1.7 scrub starts 2024-09-10T19:22:36.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:22:36 smithi183 ceph-mon[41562]: 1.7 scrub ok 2024-09-10T19:22:36.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:22:36 smithi183 ceph-mon[41562]: osdmap e273: 8 total, 6 up, 6 in 2024-09-10T19:22:36.464 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:22:36 smithi191 ceph-mon[32884]: 9.f scrub starts 2024-09-10T19:22:36.464 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:22:36 smithi191 ceph-mon[32884]: 1.7 scrub starts 2024-09-10T19:22:36.465 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:22:36 smithi191 ceph-mon[32884]: 1.7 scrub ok 2024-09-10T19:22:36.465 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:22:36 smithi191 ceph-mon[32884]: osdmap e273: 8 total, 6 up, 6 in 2024-09-10T19:22:36.522 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:22:36.555 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:22:36.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:22:36.556 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:22:36.611 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:22:36.657 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:22:36.682 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:22:36.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:22:36.683 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:22:36.784 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:22:36.820 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:22:36.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:22:36.821 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:22:36.911 INFO:journalctl@ceph.osd.1.smithi183.stdout:Sep 10 19:22:36 smithi183 podman[161395]: 2024-09-10 19:22:36.667952653 +0000 UTC m=+0.027773893 image pull 4c55dd4096115e48502f368cff1eab06769f71f030e1692e2611bbc6c6f637e9 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3bfeaf4417585c821cab9ca76e4d95f390b0e769807122a45463640820b5cd16 2024-09-10T19:22:36.921 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:22:36.954 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:22:36.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:22:36.955 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:22:37.056 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:22:37.092 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:22:37.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:22:37.093 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:22:37.177 INFO:journalctl@ceph.osd.1.smithi183.stdout:Sep 10 19:22:37 smithi183 podman[161395]: 2024-09-10 19:22:37.139473031 +0000 UTC m=+0.499294257 container create afd815a009427a6cbc46f247f0533f8b8953e92e817987b3d545be5faf222ca6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3bfeaf4417585c821cab9ca76e4d95f390b0e769807122a45463640820b5cd16, name=ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd-1, RELEASE=wip-guits-quincy-2024-09-09-1810-quincy-5be5739, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_CLEAN=True, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0) 2024-09-10T19:22:37.194 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:22:37.219 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:22:37.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:22:37.220 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:22:37.321 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:22:37.370 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:22:37.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:22:37.371 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:22:37.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:22:37 smithi183 ceph-mon[29736]: pgmap v559: 158 pgs: 1 active+clean+scrubbing, 45 active+undersized, 33 active+undersized+degraded, 79 active+clean; 4.0 GiB data, 18 GiB used, 518 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.62k op/s; 32606/194880 objects degraded (16.731%) 2024-09-10T19:22:37.431 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:22:37 smithi183 ceph-mon[41562]: pgmap v559: 158 pgs: 1 active+clean+scrubbing, 45 active+undersized, 33 active+undersized+degraded, 79 active+clean; 4.0 GiB data, 18 GiB used, 518 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.62k op/s; 32606/194880 objects degraded (16.731%) 2024-09-10T19:22:37.443 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:22:37 smithi191 ceph-mon[32884]: pgmap v559: 158 pgs: 1 active+clean+scrubbing, 45 active+undersized, 33 active+undersized+degraded, 79 active+clean; 4.0 GiB data, 18 GiB used, 518 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.62k op/s; 32606/194880 objects degraded (16.731%) 2024-09-10T19:22:37.472 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:22:37.499 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:22:37.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:22:37.500 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:22:37.600 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph daemon osd.1 dump_ops_in_flight 2024-09-10T19:22:37.600 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:22:37.624 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:22:37.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:22:37.626 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:22:37.716 INFO:journalctl@ceph.mgr.x.smithi191.stdout:Sep 10 19:22:37 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-x[34967]: ::ffff:172.21.15.191 - - [10/Sep/2024:19:22:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T19:22:37.726 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:22:37.762 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:22:37.843 INFO:journalctl@ceph.osd.1.smithi183.stdout:Sep 10 19:22:37 smithi183 podman[161395]: 2024-09-10 19:22:37.48541021 +0000 UTC m=+0.845231460 container init afd815a009427a6cbc46f247f0533f8b8953e92e817987b3d545be5faf222ca6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3bfeaf4417585c821cab9ca76e4d95f390b0e769807122a45463640820b5cd16, name=ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd-1, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, RELEASE=wip-guits-quincy-2024-09-09-1810-quincy-5be5739, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, GIT_BRANCH=HEAD, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7) 2024-09-10T19:22:37.843 INFO:journalctl@ceph.osd.1.smithi183.stdout:Sep 10 19:22:37 smithi183 podman[161395]: 2024-09-10 19:22:37.491857177 +0000 UTC m=+0.851678411 container start afd815a009427a6cbc46f247f0533f8b8953e92e817987b3d545be5faf222ca6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3bfeaf4417585c821cab9ca76e4d95f390b0e769807122a45463640820b5cd16, name=ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd-1, ceph=True, org.label-schema.vendor=CentOS, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_CLEAN=True, RELEASE=wip-guits-quincy-2024-09-09-1810-quincy-5be5739, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-10T19:22:37.843 INFO:journalctl@ceph.osd.1.smithi183.stdout:Sep 10 19:22:37 smithi183 ceph-osd[161529]: -- 172.21.15.183:0/3941326085 <== mon.0 v2:172.21.15.183:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55f125714d00 con 0x55f1256e7400 2024-09-10T19:22:37.843 INFO:journalctl@ceph.osd.1.smithi183.stdout:Sep 10 19:22:37 smithi183 bash[161395]: afd815a009427a6cbc46f247f0533f8b8953e92e817987b3d545be5faf222ca6 2024-09-10T19:22:37.843 INFO:journalctl@ceph.osd.1.smithi183.stdout:Sep 10 19:22:37 smithi183 systemd[1]: Started Ceph osd.1 for 320f6df8-6fa7-11ef-bcea-c7b262605968. 2024-09-10T19:22:37.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:22:37.844 INFO:teuthology.orchestra.run.smithi183.stderr:Error ENXIO: problem getting command descriptions from osd.0 2024-09-10T19:22:37.844 INFO:teuthology.orchestra.run.smithi183.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-09-10T19:22:37.845 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:22:37.852 INFO:teuthology.orchestra.run.smithi183.stdout:osd.2: {} 2024-09-10T19:22:37.852 INFO:teuthology.orchestra.run.smithi183.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-10T19:22:37.873 INFO:teuthology.orchestra.run.smithi183.stdout:osd.3: {} 2024-09-10T19:22:37.873 INFO:teuthology.orchestra.run.smithi183.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-10T19:22:37.892 INFO:teuthology.orchestra.run.smithi183.stdout:osd.4: {} 2024-09-10T19:22:37.892 INFO:teuthology.orchestra.run.smithi183.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-10T19:22:37.915 INFO:teuthology.orchestra.run.smithi183.stdout:osd.5: {} 2024-09-10T19:22:37.915 INFO:teuthology.orchestra.run.smithi183.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-10T19:22:37.933 INFO:teuthology.orchestra.run.smithi183.stdout:osd.6: {} 2024-09-10T19:22:37.933 INFO:teuthology.orchestra.run.smithi183.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-10T19:22:37.945 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:22:37.958 INFO:teuthology.orchestra.run.smithi183.stdout:osd.7: {} 2024-09-10T19:22:37.958 INFO:teuthology.orchestra.run.smithi183.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-10T19:22:37.976 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:22:37.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:22:37.977 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:22:38.078 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:22:38.108 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:22:38.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:22:38.109 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:22:38.122 INFO:journalctl@ceph.osd.1.smithi183.stdout:Sep 10 19:22:38 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd-1[161468]: 2024-09-10T19:22:38.036+0000 7f70ce19a440 -1 Falling back to public interface 2024-09-10T19:22:38.210 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:22:38.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:22:38.284 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:22:38.284 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:22:38.385 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:22:38.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:22:38.421 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:22:38.421 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:22:38.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:22:38 smithi183 ceph-mon[29736]: 9.f scrub ok 2024-09-10T19:22:38.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:22:38 smithi183 ceph-mon[29736]: 6.c scrub starts 2024-09-10T19:22:38.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:22:38 smithi183 ceph-mon[29736]: 6.c scrub ok 2024-09-10T19:22:38.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:22:38 smithi183 ceph-mon[29736]: 1.9 scrub starts 2024-09-10T19:22:38.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:22:38 smithi183 ceph-mon[29736]: 1.9 scrub ok 2024-09-10T19:22:38.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:22:38 smithi183 ceph-mon[29736]: from='client.? 172.21.15.183:0/3873136405' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T19:22:38.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:22:38 smithi183 ceph-mon[41562]: 9.f scrub ok 2024-09-10T19:22:38.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:22:38 smithi183 ceph-mon[41562]: 6.c scrub starts 2024-09-10T19:22:38.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:22:38 smithi183 ceph-mon[41562]: 6.c scrub ok 2024-09-10T19:22:38.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:22:38 smithi183 ceph-mon[41562]: 1.9 scrub starts 2024-09-10T19:22:38.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:22:38 smithi183 ceph-mon[41562]: 1.9 scrub ok 2024-09-10T19:22:38.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:22:38 smithi183 ceph-mon[41562]: from='client.? 172.21.15.183:0/3873136405' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T19:22:38.465 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:22:38 smithi191 ceph-mon[32884]: 9.f scrub ok 2024-09-10T19:22:38.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:22:38 smithi191 ceph-mon[32884]: 6.c scrub starts 2024-09-10T19:22:38.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:22:38 smithi191 ceph-mon[32884]: 6.c scrub ok 2024-09-10T19:22:38.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:22:38 smithi191 ceph-mon[32884]: 1.9 scrub starts 2024-09-10T19:22:38.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:22:38 smithi191 ceph-mon[32884]: 1.9 scrub ok 2024-09-10T19:22:38.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:22:38 smithi191 ceph-mon[32884]: from='client.? 172.21.15.183:0/3873136405' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T19:22:38.522 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:22:38.563 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:22:38.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:22:38.564 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:22:38.665 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:22:38.700 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:22:38.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:22:38.701 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:22:38.802 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:22:38.837 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:22:38.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:22:38.838 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:22:38.866 INFO:teuthology.orchestra.run.smithi191.stdout:{ 2024-09-10T19:22:38.867 INFO:teuthology.orchestra.run.smithi191.stdout: "ops": [], 2024-09-10T19:22:38.867 INFO:teuthology.orchestra.run.smithi191.stdout: "complaint_time": 30, 2024-09-10T19:22:38.867 INFO:teuthology.orchestra.run.smithi191.stdout: "num_blocked_ops": 0 2024-09-10T19:22:38.867 INFO:teuthology.orchestra.run.smithi191.stdout:} 2024-09-10T19:22:38.939 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:22:38.963 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:22:38.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:22:38.964 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:22:39.065 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:22:39.110 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:22:39.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:22:39.111 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:22:39.212 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:22:39.251 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:22:39.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:22:39.252 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:22:39.353 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:22:39.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:22:39.389 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:22:39.389 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:22:39.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:22:39 smithi191 ceph-mon[32884]: 9.7 deep-scrub starts 2024-09-10T19:22:39.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:22:39 smithi191 ceph-mon[32884]: pgmap v560: 158 pgs: 1 active+clean+scrubbing, 45 active+undersized, 33 active+undersized+degraded, 79 active+clean; 4.0 GiB data, 18 GiB used, 518 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.49k op/s; 32606/194880 objects degraded (16.731%) 2024-09-10T19:22:39.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:22:39 smithi191 ceph-mon[32884]: 9.c scrub starts 2024-09-10T19:22:39.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:22:39 smithi191 ceph-mon[32884]: 2.1c scrub starts 2024-09-10T19:22:39.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:22:39 smithi191 ceph-mon[32884]: 2.1c scrub ok 2024-09-10T19:22:39.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:22:39 smithi191 ceph-mon[32884]: purged_snaps scrub starts 2024-09-10T19:22:39.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:22:39 smithi191 ceph-mon[32884]: purged_snaps scrub ok 2024-09-10T19:22:39.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:22:39 smithi191 ceph-mon[32884]: 9.c scrub ok 2024-09-10T19:22:39.490 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:22:39.514 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:22:39.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:22:39.515 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:22:39.606 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:22:39 smithi183 ceph-mon[29736]: 9.7 deep-scrub starts 2024-09-10T19:22:39.607 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:22:39 smithi183 ceph-mon[29736]: pgmap v560: 158 pgs: 1 active+clean+scrubbing, 45 active+undersized, 33 active+undersized+degraded, 79 active+clean; 4.0 GiB data, 18 GiB used, 518 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.49k op/s; 32606/194880 objects degraded (16.731%) 2024-09-10T19:22:39.607 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:22:39 smithi183 ceph-mon[29736]: 9.c scrub starts 2024-09-10T19:22:39.607 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:22:39 smithi183 ceph-mon[29736]: 2.1c scrub starts 2024-09-10T19:22:39.607 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:22:39 smithi183 ceph-mon[29736]: 2.1c scrub ok 2024-09-10T19:22:39.607 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:22:39 smithi183 ceph-mon[29736]: purged_snaps scrub starts 2024-09-10T19:22:39.607 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:22:39 smithi183 ceph-mon[29736]: purged_snaps scrub ok 2024-09-10T19:22:39.607 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:22:39 smithi183 ceph-mon[29736]: 9.c scrub ok 2024-09-10T19:22:39.607 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:22:39 smithi183 ceph-mon[41562]: 9.7 deep-scrub starts 2024-09-10T19:22:39.607 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:22:39 smithi183 ceph-mon[41562]: pgmap v560: 158 pgs: 1 active+clean+scrubbing, 45 active+undersized, 33 active+undersized+degraded, 79 active+clean; 4.0 GiB data, 18 GiB used, 518 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.49k op/s; 32606/194880 objects degraded (16.731%) 2024-09-10T19:22:39.607 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:22:39 smithi183 ceph-mon[41562]: 9.c scrub starts 2024-09-10T19:22:39.607 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:22:39 smithi183 ceph-mon[41562]: 2.1c scrub starts 2024-09-10T19:22:39.607 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:22:39 smithi183 ceph-mon[41562]: 2.1c scrub ok 2024-09-10T19:22:39.608 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:22:39 smithi183 ceph-mon[41562]: purged_snaps scrub starts 2024-09-10T19:22:39.608 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:22:39 smithi183 ceph-mon[41562]: purged_snaps scrub ok 2024-09-10T19:22:39.608 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:22:39 smithi183 ceph-mon[41562]: 9.c scrub ok 2024-09-10T19:22:39.610 DEBUG:teuthology.orchestra.run.smithi191:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph daemon osd.7 dump_historic_ops 2024-09-10T19:22:39.616 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:22:39.635 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-10T19:22:39.635 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-10T19:22:39.642 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:22:39.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:22:39.643 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:22:39.744 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:22:39.787 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:22:39.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:22:39.788 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:22:39.889 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:22:39.917 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:22:39.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:22:39.918 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:22:39.966 INFO:journalctl@ceph.iscsi.iscsi.a.smithi191.stdout:Sep 10 19:22:39 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-iscsi-iscsi-a[61816]: debug there is no tcmu-runner data available 2024-09-10T19:22:40.018 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:22:40.040 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:22:40.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:22:40.041 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:22:40.044 INFO:teuthology.orchestra.run.smithi191.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.b/config 2024-09-10T19:22:40.141 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:22:40.168 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:22:40.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:22:40.169 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:22:40.266 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:22:40.270 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:22:40.297 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:22:40.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:22:40.299 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:22:40.399 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:22:40.422 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:22:40.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:22:40.423 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:22:40.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:22:40 smithi183 ceph-mon[29736]: 6.1d scrub starts 2024-09-10T19:22:40.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:22:40 smithi183 ceph-mon[29736]: 6.1d scrub ok 2024-09-10T19:22:40.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:22:40 smithi183 ceph-mon[29736]: 6.1e scrub starts 2024-09-10T19:22:40.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:22:40 smithi183 ceph-mon[29736]: 6.1e scrub ok 2024-09-10T19:22:40.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:22:40 smithi183 ceph-mon[29736]: osdmap e274: 8 total, 6 up, 6 in 2024-09-10T19:22:40.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:22:40 smithi183 ceph-mon[29736]: Health check update: Degraded data redundancy: 32606/194880 objects degraded (16.731%), 33 pgs degraded (PG_DEGRADED) 2024-09-10T19:22:40.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:22:40 smithi183 ceph-mon[41562]: 6.1d scrub starts 2024-09-10T19:22:40.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:22:40 smithi183 ceph-mon[41562]: 6.1d scrub ok 2024-09-10T19:22:40.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:22:40 smithi183 ceph-mon[41562]: 6.1e scrub starts 2024-09-10T19:22:40.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:22:40 smithi183 ceph-mon[41562]: 6.1e scrub ok 2024-09-10T19:22:40.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:22:40 smithi183 ceph-mon[41562]: osdmap e274: 8 total, 6 up, 6 in 2024-09-10T19:22:40.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:22:40 smithi183 ceph-mon[41562]: Health check update: Degraded data redundancy: 32606/194880 objects degraded (16.731%), 33 pgs degraded (PG_DEGRADED) 2024-09-10T19:22:40.465 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:22:40 smithi191 ceph-mon[32884]: 6.1d scrub starts 2024-09-10T19:22:40.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:22:40 smithi191 ceph-mon[32884]: 6.1d scrub ok 2024-09-10T19:22:40.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:22:40 smithi191 ceph-mon[32884]: 6.1e scrub starts 2024-09-10T19:22:40.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:22:40 smithi191 ceph-mon[32884]: 6.1e scrub ok 2024-09-10T19:22:40.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:22:40 smithi191 ceph-mon[32884]: osdmap e274: 8 total, 6 up, 6 in 2024-09-10T19:22:40.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:22:40 smithi191 ceph-mon[32884]: Health check update: Degraded data redundancy: 32606/194880 objects degraded (16.731%), 33 pgs degraded (PG_DEGRADED) 2024-09-10T19:22:40.523 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:22:40.559 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:22:40.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:22:40.560 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:22:40.661 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:22:40.694 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:22:40.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:22:40.695 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:22:40.796 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:22:40.819 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:22:40.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:22:40.820 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:22:40.921 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:22:40.964 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:22:40.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:22:40.965 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:22:41.066 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:22:41.098 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:22:41.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:22:41.099 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:22:41.181 INFO:journalctl@ceph.osd.1.smithi183.stdout:Sep 10 19:22:40 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd-1[161468]: 2024-09-10T19:22:40.714+0000 7f70ce19a440 -1 osd.1 255 log_to_monitors true 2024-09-10T19:22:41.199 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:22:41.211 INFO:teuthology.orchestra.run.smithi183.stderr:noscrub is set 2024-09-10T19:22:41.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:22:41.240 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:22:41.240 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:22:41.340 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:22:41.378 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:22:41.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:22:41.379 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:22:41.379 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:22:41 smithi191 ceph-mon[32884]: from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:22:41.379 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:22:41 smithi191 ceph-mon[32884]: pgmap v562: 158 pgs: 1 active+clean+scrubbing, 45 active+undersized, 33 active+undersized+degraded, 79 active+clean; 4.0 GiB data, 18 GiB used, 518 GiB / 536 GiB avail; 639 B/s rd, 0 B/s wr, 876 op/s; 32606/194880 objects degraded (16.731%) 2024-09-10T19:22:41.380 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:22:41 smithi191 ceph-mon[32884]: from='client.? 172.21.15.183:0/3453357631' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-10T19:22:41.380 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:22:41 smithi191 ceph-mon[32884]: from='osd.1 [v2:172.21.15.183:6802/2764394151,v1:172.21.15.183:6803/2764394151]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-09-10T19:22:41.480 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:22:41.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:22:41.520 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:22:41.520 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:22:41.620 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:22:41.636 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-10T19:22:41.653 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Sep 10 19:22:41 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-y[29989]: ::ffff:172.21.15.191 - - [10/Sep/2024:19:22:41] "GET /metrics HTTP/1.1" 200 209534 "" "Prometheus/2.43.0" 2024-09-10T19:22:41.653 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:22:41 smithi183 ceph-mon[41562]: from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:22:41.653 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:22:41 smithi183 ceph-mon[41562]: pgmap v562: 158 pgs: 1 active+clean+scrubbing, 45 active+undersized, 33 active+undersized+degraded, 79 active+clean; 4.0 GiB data, 18 GiB used, 518 GiB / 536 GiB avail; 639 B/s rd, 0 B/s wr, 876 op/s; 32606/194880 objects degraded (16.731%) 2024-09-10T19:22:41.653 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:22:41 smithi183 ceph-mon[41562]: from='client.? 172.21.15.183:0/3453357631' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-10T19:22:41.653 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:22:41 smithi183 ceph-mon[41562]: from='osd.1 [v2:172.21.15.183:6802/2764394151,v1:172.21.15.183:6803/2764394151]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-09-10T19:22:41.654 INFO:journalctl@ceph.osd.1.smithi183.stdout:Sep 10 19:22:41 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd-1[161468]: 2024-09-10T19:22:41.214+0000 7f70c273f640 -1 osd.1 255 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-10T19:22:41.654 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:22:41 smithi183 ceph-mon[29736]: from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:22:41.654 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:22:41 smithi183 ceph-mon[29736]: pgmap v562: 158 pgs: 1 active+clean+scrubbing, 45 active+undersized, 33 active+undersized+degraded, 79 active+clean; 4.0 GiB data, 18 GiB used, 518 GiB / 536 GiB avail; 639 B/s rd, 0 B/s wr, 876 op/s; 32606/194880 objects degraded (16.731%) 2024-09-10T19:22:41.654 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:22:41 smithi183 ceph-mon[29736]: from='client.? 172.21.15.183:0/3453357631' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-10T19:22:41.654 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:22:41 smithi183 ceph-mon[29736]: from='osd.1 [v2:172.21.15.183:6802/2764394151,v1:172.21.15.183:6803/2764394151]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-09-10T19:22:41.656 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:22:41.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:22:41.657 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:22:41.758 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:22:41.792 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:22:41.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:22:41.793 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:22:41.894 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:22:41.917 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:22:41.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:22:41.918 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:22:42.018 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:22:42.033 INFO:teuthology.orchestra.run.smithi191.stdout:{ 2024-09-10T19:22:42.034 INFO:teuthology.orchestra.run.smithi191.stdout: "size": 20, 2024-09-10T19:22:42.034 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 600, 2024-09-10T19:22:42.034 INFO:teuthology.orchestra.run.smithi191.stdout: "ops": [ 2024-09-10T19:22:42.034 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:42.034 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.15051.0:59075 8.3 8.aac477f3 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e165)", 2024-09-10T19:22:42.034 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-10T19:16:55.018030+0000", 2024-09-10T19:22:42.034 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 347.01152715400002, 2024-09-10T19:22:42.035 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.97022024399999995, 2024-09-10T19:22:42.035 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-10T19:22:42.035 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "reached pg", 2024-09-10T19:22:42.035 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-10T19:22:42.035 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.15051", 2024-09-10T19:22:42.035 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.183:0/3632028672", 2024-09-10T19:22:42.035 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 59075 2024-09-10T19:22:42.035 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:42.035 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-10T19:22:42.035 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:42.035 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-10T19:22:42.035 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:16:55.018030+0000", 2024-09-10T19:22:42.035 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:22:42.035 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:42.036 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:42.036 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-10T19:22:42.036 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:16:55.018030+0000", 2024-09-10T19:22:42.036 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:22:42.036 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:42.036 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:42.036 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-10T19:22:42.036 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:16:55.018029+0000", 2024-09-10T19:22:42.036 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999981 2024-09-10T19:22:42.036 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:42.036 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:42.036 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-10T19:22:42.037 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:16:55.018060+0000", 2024-09-10T19:22:42.037 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.1653999999999999e-05 2024-09-10T19:22:42.037 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:42.037 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:42.037 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-10T19:22:42.037 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:16:55.018072+0000", 2024-09-10T19:22:42.037 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.1960999999999999e-05 2024-09-10T19:22:42.037 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:42.037 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:42.037 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-10T19:22:42.037 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:16:55.018080+0000", 2024-09-10T19:22:42.037 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.909e-06 2024-09-10T19:22:42.037 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:42.038 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:42.038 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:22:42.038 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:16:55.018130+0000", 2024-09-10T19:22:42.038 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.0238000000000002e-05 2024-09-10T19:22:42.038 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:42.038 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:42.038 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for peered", 2024-09-10T19:22:42.038 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:16:55.018136+0000", 2024-09-10T19:22:42.038 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.8039999999999998e-06 2024-09-10T19:22:42.038 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:42.038 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:42.038 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:22:42.038 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:16:55.988241+0000", 2024-09-10T19:22:42.039 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.97010456899999997 2024-09-10T19:22:42.039 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:42.039 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:42.039 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-10T19:22:42.039 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:16:55.988251+0000", 2024-09-10T19:22:42.039 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.9059999999999997e-06 2024-09-10T19:22:42.039 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:22:42.039 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-10T19:22:42.039 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:22:42.039 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:42.039 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:42.039 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.15051.0:93082 8.3 8:c5abc7f7:::benchmark_data_smithi183_114924_object93081:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=2 ondisk+retry+write+known_if_redirected+supports_pool_eio e178)", 2024-09-10T19:22:42.039 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-10T19:17:26.939355+0000", 2024-09-10T19:22:42.039 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 315.09020236399999, 2024-09-10T19:22:42.039 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.047200065, 2024-09-10T19:22:42.040 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-10T19:22:42.040 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "reached pg", 2024-09-10T19:22:42.040 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-10T19:22:42.040 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.15051", 2024-09-10T19:22:42.040 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.183:0/3632028672", 2024-09-10T19:22:42.040 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 93082 2024-09-10T19:22:42.040 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:42.040 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-10T19:22:42.040 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:42.040 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-10T19:22:42.040 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:26.939355+0000", 2024-09-10T19:22:42.040 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:22:42.040 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:42.040 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:42.040 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-10T19:22:42.041 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:26.939355+0000", 2024-09-10T19:22:42.041 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:22:42.041 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:42.041 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:42.041 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-10T19:22:42.041 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:26.939353+0000", 2024-09-10T19:22:42.041 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999976 2024-09-10T19:22:42.041 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:42.041 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:42.041 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-10T19:22:42.041 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:26.939539+0000", 2024-09-10T19:22:42.041 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00018558299999999999 2024-09-10T19:22:42.041 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:42.042 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:42.042 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-10T19:22:42.042 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:26.939567+0000", 2024-09-10T19:22:42.042 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.8345000000000001e-05 2024-09-10T19:22:42.042 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:42.042 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:42.042 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-10T19:22:42.042 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:26.939576+0000", 2024-09-10T19:22:42.042 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.0389999999999993e-06 2024-09-10T19:22:42.042 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:42.042 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:42.042 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:22:42.042 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:26.939651+0000", 2024-09-10T19:22:42.043 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.5278999999999995e-05 2024-09-10T19:22:42.043 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:42.043 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:42.043 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op must wait for map", 2024-09-10T19:22:42.043 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:26.939662+0000", 2024-09-10T19:22:42.043 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.0789e-05 2024-09-10T19:22:42.043 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:42.043 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:42.043 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:22:42.043 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:26.944034+0000", 2024-09-10T19:22:42.043 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0043717110000000003 2024-09-10T19:22:42.043 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:42.043 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:42.043 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for peered", 2024-09-10T19:22:42.043 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:26.944047+0000", 2024-09-10T19:22:42.044 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.288e-05 2024-09-10T19:22:42.044 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:42.044 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:42.044 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:22:42.044 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:27.985144+0000", 2024-09-10T19:22:42.044 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.0410975929999999 2024-09-10T19:22:42.044 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:42.044 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:42.044 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-10T19:22:42.044 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:27.986555+0000", 2024-09-10T19:22:42.044 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0014110419999999999 2024-09-10T19:22:42.044 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:22:42.044 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-10T19:22:42.044 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:22:42.045 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:42.045 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:42.045 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.15051.0:93091 8.3 8:c1c9fa55:::benchmark_data_smithi183_114924_object93090:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e178)", 2024-09-10T19:22:42.045 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-10T19:17:26.939596+0000", 2024-09-10T19:22:42.045 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 315.089961625, 2024-09-10T19:22:42.045 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.4256138250000001, 2024-09-10T19:22:42.045 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-10T19:22:42.045 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T19:22:42.045 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-10T19:22:42.045 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.15051", 2024-09-10T19:22:42.045 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.183:0/3632028672", 2024-09-10T19:22:42.045 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 93091 2024-09-10T19:22:42.045 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:42.045 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-10T19:22:42.046 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:42.046 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-10T19:22:42.046 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:26.939596+0000", 2024-09-10T19:22:42.046 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:22:42.046 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:42.046 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:42.046 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-10T19:22:42.046 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:26.939596+0000", 2024-09-10T19:22:42.046 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:22:42.046 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:42.046 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:42.046 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-10T19:22:42.047 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:26.939595+0000", 2024-09-10T19:22:42.047 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.999999 2024-09-10T19:22:42.047 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:42.047 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:42.047 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-10T19:22:42.047 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:26.939637+0000", 2024-09-10T19:22:42.047 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.2364999999999998e-05 2024-09-10T19:22:42.047 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:42.047 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:42.047 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-10T19:22:42.047 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:26.939647+0000", 2024-09-10T19:22:42.047 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.8519999999999999e-06 2024-09-10T19:22:42.047 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:42.047 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:42.047 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-10T19:22:42.048 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:26.939653+0000", 2024-09-10T19:22:42.048 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.8649999999999998e-06 2024-09-10T19:22:42.048 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:42.048 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:42.048 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:22:42.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:22:42.049 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:26.939821+0000", 2024-09-10T19:22:42.049 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00016759300000000001 2024-09-10T19:22:42.049 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:42.049 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:42.049 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting_for_map not empty", 2024-09-10T19:22:42.049 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:26.939828+0000", 2024-09-10T19:22:42.050 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.6389999999999992e-06 2024-09-10T19:22:42.050 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:42.050 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:42.050 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:22:42.050 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:26.944068+0000", 2024-09-10T19:22:42.050 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0042391219999999997 2024-09-10T19:22:42.050 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:42.050 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:42.050 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for peered", 2024-09-10T19:22:42.050 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:26.944073+0000", 2024-09-10T19:22:42.050 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.0780000000000003e-06 2024-09-10T19:22:42.050 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:42.050 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:42.050 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:22:42.051 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:27.986811+0000", 2024-09-10T19:22:42.051 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.042738487 2024-09-10T19:22:42.051 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:42.051 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:42.051 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-10T19:22:42.051 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:27.989675+0000", 2024-09-10T19:22:42.051 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0028639149999999999 2024-09-10T19:22:42.051 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:42.051 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:42.051 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 3,5", 2024-09-10T19:22:42.051 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:27.994560+0000", 2024-09-10T19:22:42.051 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0048853400000000002 2024-09-10T19:22:42.051 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:42.051 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:42.051 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-09-10T19:22:42.052 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:28.015341+0000", 2024-09-10T19:22:42.052 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.020780532000000001 2024-09-10T19:22:42.052 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:42.052 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:42.052 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:22:42.052 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:28.359525+0000", 2024-09-10T19:22:42.052 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.34418381399999998 2024-09-10T19:22:42.052 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:42.052 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:42.052 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:22:42.052 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:28.360180+0000", 2024-09-10T19:22:42.052 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00065538 2024-09-10T19:22:42.052 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:42.053 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:42.053 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-09-10T19:22:42.053 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:28.361229+0000", 2024-09-10T19:22:42.053 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.001048998 2024-09-10T19:22:42.053 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:42.053 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:42.053 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-10T19:22:42.053 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:28.365210+0000", 2024-09-10T19:22:42.053 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0039807380000000002 2024-09-10T19:22:42.053 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:22:42.053 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-10T19:22:42.053 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:22:42.053 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:42.053 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:42.053 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.15051.0:93120 8.3 8:c0ebb47a:::benchmark_data_smithi183_114924_object93119:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e178)", 2024-09-10T19:22:42.054 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-10T19:17:26.991786+0000", 2024-09-10T19:22:42.054 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 315.03777171500002, 2024-09-10T19:22:42.054 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.3798205450000001, 2024-09-10T19:22:42.054 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-10T19:22:42.054 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T19:22:42.054 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-10T19:22:42.054 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.15051", 2024-09-10T19:22:42.054 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.183:0/3632028672", 2024-09-10T19:22:42.054 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 93120 2024-09-10T19:22:42.054 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:42.054 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-10T19:22:42.054 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:42.054 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-10T19:22:42.055 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:26.991786+0000", 2024-09-10T19:22:42.055 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:22:42.055 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:42.055 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:42.055 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-10T19:22:42.055 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:26.991786+0000", 2024-09-10T19:22:42.055 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:22:42.055 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:42.055 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:42.055 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-10T19:22:42.055 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:26.991784+0000", 2024-09-10T19:22:42.055 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999986 2024-09-10T19:22:42.055 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:42.055 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:42.056 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-10T19:22:42.056 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:26.991845+0000", 2024-09-10T19:22:42.056 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.0312999999999997e-05 2024-09-10T19:22:42.056 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:42.056 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:42.056 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-10T19:22:42.056 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:26.991860+0000", 2024-09-10T19:22:42.056 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.5639999999999999e-05 2024-09-10T19:22:42.056 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:42.056 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:42.056 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-10T19:22:42.056 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:26.991867+0000", 2024-09-10T19:22:42.056 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.5400000000000001e-06 2024-09-10T19:22:42.056 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:42.057 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:42.057 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:22:42.057 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:26.991919+0000", 2024-09-10T19:22:42.057 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.2629e-05 2024-09-10T19:22:42.057 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:42.057 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:42.057 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for peered", 2024-09-10T19:22:42.057 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:26.991930+0000", 2024-09-10T19:22:42.057 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.0207000000000001e-05 2024-09-10T19:22:42.057 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:42.057 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:42.057 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:22:42.057 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:27.997258+0000", 2024-09-10T19:22:42.058 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.0053281890000001 2024-09-10T19:22:42.058 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:42.058 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:42.058 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-10T19:22:42.058 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:27.999670+0000", 2024-09-10T19:22:42.058 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0024118439999999998 2024-09-10T19:22:42.058 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:42.058 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:42.058 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 3,5", 2024-09-10T19:22:42.058 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:28.003897+0000", 2024-09-10T19:22:42.058 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0042272050000000004 2024-09-10T19:22:42.058 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:42.058 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:42.058 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-09-10T19:22:42.058 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:28.033905+0000", 2024-09-10T19:22:42.059 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.030007741000000001 2024-09-10T19:22:42.059 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:42.059 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:42.059 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:22:42.059 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:28.365817+0000", 2024-09-10T19:22:42.059 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.33191224400000002 2024-09-10T19:22:42.059 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:42.059 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:42.059 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:22:42.059 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:28.366555+0000", 2024-09-10T19:22:42.059 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00073775299999999995 2024-09-10T19:22:42.059 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:42.059 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:42.059 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-09-10T19:22:42.059 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:28.367621+0000", 2024-09-10T19:22:42.060 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0010666180000000001 2024-09-10T19:22:42.060 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:42.060 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:42.060 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-10T19:22:42.060 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:28.371606+0000", 2024-09-10T19:22:42.060 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0039852769999999997 2024-09-10T19:22:42.060 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:22:42.060 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-10T19:22:42.060 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:22:42.060 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:42.060 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:42.060 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.15051.0:93129 8.3 8:c691b51d:::benchmark_data_smithi183_114924_object93128:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e178)", 2024-09-10T19:22:42.060 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-10T19:17:27.001902+0000", 2024-09-10T19:22:42.060 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 315.02765610900002, 2024-09-10T19:22:42.060 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.3757423389999999, 2024-09-10T19:22:42.061 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-10T19:22:42.061 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T19:22:42.061 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-10T19:22:42.061 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.15051", 2024-09-10T19:22:42.061 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.183:0/3632028672", 2024-09-10T19:22:42.061 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 93129 2024-09-10T19:22:42.061 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:42.061 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-10T19:22:42.061 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:42.061 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-10T19:22:42.061 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:27.001902+0000", 2024-09-10T19:22:42.061 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:22:42.061 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:42.061 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:42.061 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-10T19:22:42.062 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:27.001902+0000", 2024-09-10T19:22:42.062 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:22:42.062 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:42.062 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:42.062 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-10T19:22:42.062 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:27.001900+0000", 2024-09-10T19:22:42.062 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999986 2024-09-10T19:22:42.062 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:42.062 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:42.062 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-10T19:22:42.062 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:27.001937+0000", 2024-09-10T19:22:42.062 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.6804000000000002e-05 2024-09-10T19:22:42.062 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:42.062 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:42.063 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-10T19:22:42.063 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:27.001948+0000", 2024-09-10T19:22:42.063 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.1569999999999999e-05 2024-09-10T19:22:42.063 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:42.063 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:42.063 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-10T19:22:42.063 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:27.001954+0000", 2024-09-10T19:22:42.063 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.5740000000000003e-06 2024-09-10T19:22:42.063 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:42.063 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:42.063 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:22:42.063 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:27.002077+0000", 2024-09-10T19:22:42.063 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00012273199999999999 2024-09-10T19:22:42.063 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:42.064 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:42.064 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for peered", 2024-09-10T19:22:42.064 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:27.002086+0000", 2024-09-10T19:22:42.064 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.7499999999999998e-06 2024-09-10T19:22:42.064 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:42.064 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:42.064 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:22:42.064 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:28.006349+0000", 2024-09-10T19:22:42.064 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.0042625359999999 2024-09-10T19:22:42.064 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:42.064 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:42.064 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-10T19:22:42.064 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:28.008726+0000", 2024-09-10T19:22:42.065 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0023774859999999998 2024-09-10T19:22:42.065 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:42.065 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:42.065 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 3,5", 2024-09-10T19:22:42.065 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:28.012885+0000", 2024-09-10T19:22:42.065 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0041590309999999997 2024-09-10T19:22:42.065 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:42.065 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:42.065 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-09-10T19:22:42.065 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:28.034107+0000", 2024-09-10T19:22:42.065 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.021221998999999998 2024-09-10T19:22:42.065 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:42.065 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:42.066 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:22:42.066 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:28.372218+0000", 2024-09-10T19:22:42.066 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.338111045 2024-09-10T19:22:42.066 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:42.066 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:42.066 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:22:42.066 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:28.372817+0000", 2024-09-10T19:22:42.066 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00059862800000000003 2024-09-10T19:22:42.066 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:42.066 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:42.066 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-09-10T19:22:42.066 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:28.373805+0000", 2024-09-10T19:22:42.066 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00098759499999999997 2024-09-10T19:22:42.066 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:42.066 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:42.067 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-10T19:22:42.067 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:28.377644+0000", 2024-09-10T19:22:42.067 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0038391969999999999 2024-09-10T19:22:42.067 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:22:42.067 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-10T19:22:42.067 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:22:42.067 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:42.067 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:42.067 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.15051.0:93169 8.3 8:c5187144:::benchmark_data_smithi183_114924_object93168:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e178)", 2024-09-10T19:22:42.067 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-10T19:17:27.028969+0000", 2024-09-10T19:22:42.067 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 315.00058844500001, 2024-09-10T19:22:42.067 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.354956426, 2024-09-10T19:22:42.067 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-10T19:22:42.067 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T19:22:42.068 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-10T19:22:42.068 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.15051", 2024-09-10T19:22:42.068 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.183:0/3632028672", 2024-09-10T19:22:42.068 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 93169 2024-09-10T19:22:42.068 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:42.068 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-10T19:22:42.068 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:42.068 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-10T19:22:42.068 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:27.028969+0000", 2024-09-10T19:22:42.068 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:22:42.068 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:42.068 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:42.068 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-10T19:22:42.068 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:27.028969+0000", 2024-09-10T19:22:42.068 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:22:42.069 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:42.069 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:42.069 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-10T19:22:42.069 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:27.028967+0000", 2024-09-10T19:22:42.069 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999981 2024-09-10T19:22:42.069 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:42.069 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:42.069 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-10T19:22:42.069 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:27.029032+0000", 2024-09-10T19:22:42.069 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.4704999999999998e-05 2024-09-10T19:22:42.069 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:42.069 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:42.069 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-10T19:22:42.069 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:27.029046+0000", 2024-09-10T19:22:42.069 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.3458999999999999e-05 2024-09-10T19:22:42.070 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:42.070 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:42.070 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-10T19:22:42.070 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:27.029052+0000", 2024-09-10T19:22:42.070 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.8669999999999999e-06 2024-09-10T19:22:42.070 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:42.070 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:42.070 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:22:42.070 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:27.029118+0000", 2024-09-10T19:22:42.070 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.6771000000000002e-05 2024-09-10T19:22:42.070 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:42.070 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:42.070 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for peered", 2024-09-10T19:22:42.070 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:27.029136+0000", 2024-09-10T19:22:42.070 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.7257000000000001e-05 2024-09-10T19:22:42.070 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:42.071 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:42.071 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:22:42.071 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:28.015654+0000", 2024-09-10T19:22:42.071 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.98651808399999996 2024-09-10T19:22:42.071 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:42.071 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:42.071 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-10T19:22:42.071 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:28.018045+0000", 2024-09-10T19:22:42.071 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0023913710000000002 2024-09-10T19:22:42.071 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:42.071 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:42.071 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 3,5", 2024-09-10T19:22:42.071 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:28.022199+0000", 2024-09-10T19:22:42.071 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0041539519999999998 2024-09-10T19:22:42.072 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:42.072 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:42.072 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-09-10T19:22:42.072 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:28.052681+0000", 2024-09-10T19:22:42.072 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.030481590999999999 2024-09-10T19:22:42.072 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:42.072 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:42.072 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:22:42.072 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:28.378251+0000", 2024-09-10T19:22:42.072 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.32557088899999997 2024-09-10T19:22:42.072 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:42.072 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:42.072 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:22:42.072 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:28.378848+0000", 2024-09-10T19:22:42.073 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00059637000000000004 2024-09-10T19:22:42.073 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:42.073 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:42.073 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-09-10T19:22:42.073 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:28.379835+0000", 2024-09-10T19:22:42.073 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00098689999999999997 2024-09-10T19:22:42.073 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:42.073 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:42.073 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-10T19:22:42.073 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:28.383926+0000", 2024-09-10T19:22:42.073 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0040909129999999998 2024-09-10T19:22:42.073 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:22:42.073 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-10T19:22:42.073 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:22:42.074 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:42.074 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:42.074 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.15051.0:93200 8.3 8:c1d83c5e:::benchmark_data_smithi183_114924_object93199:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e178)", 2024-09-10T19:22:42.074 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-10T19:17:27.045967+0000", 2024-09-10T19:22:42.074 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 314.98359065300002, 2024-09-10T19:22:42.074 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.3449605410000001, 2024-09-10T19:22:42.074 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-10T19:22:42.074 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T19:22:42.074 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-10T19:22:42.074 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.15051", 2024-09-10T19:22:42.074 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.183:0/3632028672", 2024-09-10T19:22:42.074 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 93200 2024-09-10T19:22:42.074 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:42.074 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-10T19:22:42.074 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:42.075 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-10T19:22:42.075 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:27.045967+0000", 2024-09-10T19:22:42.075 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:22:42.075 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:42.075 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:42.075 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-10T19:22:42.075 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:27.045967+0000", 2024-09-10T19:22:42.075 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:22:42.075 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:42.075 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:42.075 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-10T19:22:42.075 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:27.045965+0000", 2024-09-10T19:22:42.075 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999981 2024-09-10T19:22:42.075 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:42.076 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:42.076 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-10T19:22:42.076 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:27.046015+0000", 2024-09-10T19:22:42.076 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.0099000000000003e-05 2024-09-10T19:22:42.076 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:42.076 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:42.076 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-10T19:22:42.076 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:27.046030+0000", 2024-09-10T19:22:42.076 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.4493e-05 2024-09-10T19:22:42.076 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:42.076 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:42.076 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-10T19:22:42.076 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:27.046035+0000", 2024-09-10T19:22:42.076 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.5999999999999997e-06 2024-09-10T19:22:42.077 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:42.077 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:42.077 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:22:42.077 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:27.046093+0000", 2024-09-10T19:22:42.077 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.7948999999999997e-05 2024-09-10T19:22:42.077 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:42.077 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:42.077 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for peered", 2024-09-10T19:22:42.077 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:27.046103+0000", 2024-09-10T19:22:42.077 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.5580000000000006e-06 2024-09-10T19:22:42.077 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:42.077 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:42.077 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:22:42.077 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:28.024643+0000", 2024-09-10T19:22:42.077 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.97854046100000003 2024-09-10T19:22:42.078 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:42.078 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:42.078 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-10T19:22:42.078 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:28.027144+0000", 2024-09-10T19:22:42.078 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00250037 2024-09-10T19:22:42.078 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:42.078 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:42.078 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 3,5", 2024-09-10T19:22:42.078 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:28.031418+0000", 2024-09-10T19:22:42.078 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0042745680000000003 2024-09-10T19:22:42.078 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:42.078 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:42.078 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-09-10T19:22:42.078 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:28.052882+0000", 2024-09-10T19:22:42.078 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.021464051000000001 2024-09-10T19:22:42.079 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:42.079 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:42.079 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:22:42.079 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:28.384774+0000", 2024-09-10T19:22:42.079 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.331892038 2024-09-10T19:22:42.079 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:42.079 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:42.079 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:22:42.079 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:28.385609+0000", 2024-09-10T19:22:42.079 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00083466900000000001 2024-09-10T19:22:42.079 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:42.079 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:42.079 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-09-10T19:22:42.079 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:28.386728+0000", 2024-09-10T19:22:42.079 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0011185349999999999 2024-09-10T19:22:42.080 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:42.080 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:42.080 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-10T19:22:42.080 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:28.390928+0000", 2024-09-10T19:22:42.080 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0041999569999999998 2024-09-10T19:22:42.080 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:22:42.080 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-10T19:22:42.080 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:22:42.080 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:42.080 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:42.080 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.15051.0:93226 8.3 8:c25b1055:::benchmark_data_smithi183_114924_object93225:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e178)", 2024-09-10T19:22:42.080 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-10T19:17:27.062335+0000", 2024-09-10T19:22:42.080 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 314.96722226000003, 2024-09-10T19:22:42.080 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.335070134, 2024-09-10T19:22:42.081 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-10T19:22:42.081 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T19:22:42.081 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-10T19:22:42.081 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.15051", 2024-09-10T19:22:42.081 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.183:0/3632028672", 2024-09-10T19:22:42.081 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 93226 2024-09-10T19:22:42.081 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:42.081 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-10T19:22:42.081 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:42.081 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-10T19:22:42.081 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:27.062335+0000", 2024-09-10T19:22:42.081 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:22:42.081 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:42.081 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:42.081 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-10T19:22:42.082 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:27.062335+0000", 2024-09-10T19:22:42.082 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:22:42.082 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:42.082 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:42.082 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-10T19:22:42.082 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:27.062334+0000", 2024-09-10T19:22:42.082 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999981 2024-09-10T19:22:42.082 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:42.082 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:42.082 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-10T19:22:42.082 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:27.062400+0000", 2024-09-10T19:22:42.082 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.5959999999999999e-05 2024-09-10T19:22:42.082 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:42.082 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:42.083 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-10T19:22:42.083 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:27.062416+0000", 2024-09-10T19:22:42.083 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.6722999999999999e-05 2024-09-10T19:22:42.083 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:42.083 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:42.083 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-10T19:22:42.083 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:27.062423+0000", 2024-09-10T19:22:42.083 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.8770000000000004e-06 2024-09-10T19:22:42.083 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:42.083 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:42.083 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:22:42.083 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:27.064828+0000", 2024-09-10T19:22:42.083 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0024045 2024-09-10T19:22:42.083 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:42.083 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:42.083 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for peered", 2024-09-10T19:22:42.084 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:27.064847+0000", 2024-09-10T19:22:42.084 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.9548999999999999e-05 2024-09-10T19:22:42.084 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:42.084 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:42.084 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:22:42.084 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:28.034430+0000", 2024-09-10T19:22:42.084 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.96958232899999997 2024-09-10T19:22:42.084 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:42.084 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:42.084 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-10T19:22:42.084 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:28.036844+0000", 2024-09-10T19:22:42.084 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.002414518 2024-09-10T19:22:42.084 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:42.084 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:42.084 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 3,5", 2024-09-10T19:22:42.085 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:28.041056+0000", 2024-09-10T19:22:42.085 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0042121169999999996 2024-09-10T19:22:42.085 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:42.085 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:42.085 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-09-10T19:22:42.085 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:28.073275+0000", 2024-09-10T19:22:42.085 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.032218546000000001 2024-09-10T19:22:42.085 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:42.085 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:42.085 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:22:42.085 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:28.391594+0000", 2024-09-10T19:22:42.085 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.31831964600000001 2024-09-10T19:22:42.085 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:42.085 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:42.086 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:22:42.086 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:28.392240+0000", 2024-09-10T19:22:42.086 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00064562500000000004 2024-09-10T19:22:42.086 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:42.086 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:42.086 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-09-10T19:22:42.086 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:28.393278+0000", 2024-09-10T19:22:42.086 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.001037804 2024-09-10T19:22:42.086 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:42.086 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:42.086 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-10T19:22:42.086 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:28.397406+0000", 2024-09-10T19:22:42.086 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.004127627 2024-09-10T19:22:42.086 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:22:42.087 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-10T19:22:42.087 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:22:42.087 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:42.087 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:42.087 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.15051.0:93261 8.3 8:c5a97e06:::benchmark_data_smithi183_114924_object93260:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e178)", 2024-09-10T19:22:42.087 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-10T19:17:27.112265+0000", 2024-09-10T19:22:42.087 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 314.91729221999998, 2024-09-10T19:22:42.087 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.2917055470000001, 2024-09-10T19:22:42.087 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-10T19:22:42.087 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T19:22:42.087 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-10T19:22:42.087 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.15051", 2024-09-10T19:22:42.087 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.183:0/3632028672", 2024-09-10T19:22:42.087 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 93261 2024-09-10T19:22:42.088 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:42.088 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-10T19:22:42.088 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:42.088 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-10T19:22:42.088 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:27.112265+0000", 2024-09-10T19:22:42.088 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:22:42.088 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:42.088 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:42.088 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:22:42.089 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:22:42.089 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-10T19:22:42.089 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:27.112265+0000", 2024-09-10T19:22:42.089 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:22:42.090 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:42.090 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:42.090 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-10T19:22:42.090 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:27.112263+0000", 2024-09-10T19:22:42.090 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999981 2024-09-10T19:22:42.090 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:42.090 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:42.090 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-10T19:22:42.090 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:27.112309+0000", 2024-09-10T19:22:42.090 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.5238999999999998e-05 2024-09-10T19:22:42.090 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:42.090 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:42.090 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-10T19:22:42.091 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:27.112323+0000", 2024-09-10T19:22:42.091 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.4284000000000001e-05 2024-09-10T19:22:42.091 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:42.091 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:42.091 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-10T19:22:42.091 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:27.112329+0000", 2024-09-10T19:22:42.091 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.1700000000000002e-06 2024-09-10T19:22:42.091 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:42.091 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:42.091 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:22:42.091 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:27.112425+0000", 2024-09-10T19:22:42.091 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.5568000000000004e-05 2024-09-10T19:22:42.091 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:42.091 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:42.091 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for peered", 2024-09-10T19:22:42.092 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:27.112432+0000", 2024-09-10T19:22:42.092 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.4889999999999998e-06 2024-09-10T19:22:42.092 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:42.092 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:42.092 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:22:42.092 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:28.043578+0000", 2024-09-10T19:22:42.092 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.93114602599999996 2024-09-10T19:22:42.092 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:42.092 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:42.092 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-10T19:22:42.092 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:28.045997+0000", 2024-09-10T19:22:42.092 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00241902 2024-09-10T19:22:42.092 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:42.092 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:42.093 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 3,5", 2024-09-10T19:22:42.093 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:28.050259+0000", 2024-09-10T19:22:42.093 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0042620139999999997 2024-09-10T19:22:42.093 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:42.093 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:42.093 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-09-10T19:22:42.093 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:28.073477+0000", 2024-09-10T19:22:42.093 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.023217686000000001 2024-09-10T19:22:42.093 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:42.093 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:42.093 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:22:42.093 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:28.398126+0000", 2024-09-10T19:22:42.093 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.32464874399999999 2024-09-10T19:22:42.094 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:42.094 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:42.094 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:22:42.094 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:28.398842+0000", 2024-09-10T19:22:42.094 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000715969 2024-09-10T19:22:42.094 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:42.094 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:42.094 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-09-10T19:22:42.094 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:28.399834+0000", 2024-09-10T19:22:42.094 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00099297300000000008 2024-09-10T19:22:42.094 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:42.094 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:42.094 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-10T19:22:42.094 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:28.403971+0000", 2024-09-10T19:22:42.095 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0041364660000000001 2024-09-10T19:22:42.095 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:22:42.095 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-10T19:22:42.095 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:22:42.095 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:42.095 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:42.095 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.15051.0:93278 8.3 8:c36e9ad5:::benchmark_data_smithi183_114924_object93277:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e178)", 2024-09-10T19:22:42.095 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-10T19:17:27.125300+0000", 2024-09-10T19:22:42.095 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 314.90425750200001, 2024-09-10T19:22:42.095 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.2853549200000001, 2024-09-10T19:22:42.095 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-10T19:22:42.095 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T19:22:42.095 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-10T19:22:42.095 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.15051", 2024-09-10T19:22:42.096 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.183:0/3632028672", 2024-09-10T19:22:42.096 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 93278 2024-09-10T19:22:42.096 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:42.096 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-10T19:22:42.096 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:42.096 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-10T19:22:42.096 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:27.125300+0000", 2024-09-10T19:22:42.096 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:22:42.096 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:42.096 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:42.096 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-10T19:22:42.096 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:27.125300+0000", 2024-09-10T19:22:42.096 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:22:42.096 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:42.097 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:42.097 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-10T19:22:42.097 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:27.125299+0000", 2024-09-10T19:22:42.097 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999986 2024-09-10T19:22:42.097 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:42.097 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:42.097 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-10T19:22:42.097 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:27.125370+0000", 2024-09-10T19:22:42.097 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.1624000000000004e-05 2024-09-10T19:22:42.097 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:42.097 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:42.097 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-10T19:22:42.097 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:27.125386+0000", 2024-09-10T19:22:42.098 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.5608e-05 2024-09-10T19:22:42.098 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:42.098 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:42.098 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-10T19:22:42.098 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:27.125392+0000", 2024-09-10T19:22:42.098 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.4300000000000003e-06 2024-09-10T19:22:42.098 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:42.098 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:42.098 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:22:42.098 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:27.125449+0000", 2024-09-10T19:22:42.098 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.6604999999999998e-05 2024-09-10T19:22:42.098 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:42.098 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:42.098 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for peered", 2024-09-10T19:22:42.098 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:27.125459+0000", 2024-09-10T19:22:42.099 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.9699999999999994e-06 2024-09-10T19:22:42.099 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:42.099 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:42.099 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:22:42.099 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:28.053203+0000", 2024-09-10T19:22:42.099 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.92774412299999998 2024-09-10T19:22:42.099 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:42.099 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:42.099 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-10T19:22:42.099 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:28.055638+0000", 2024-09-10T19:22:42.099 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0024347959999999999 2024-09-10T19:22:42.099 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:42.099 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:42.099 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 3,5", 2024-09-10T19:22:42.100 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:28.060056+0000", 2024-09-10T19:22:42.100 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0044181480000000002 2024-09-10T19:22:42.100 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:42.100 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:42.100 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-09-10T19:22:42.100 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:28.357493+0000", 2024-09-10T19:22:42.100 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.29743763200000001 2024-09-10T19:22:42.100 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:42.100 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:42.100 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:22:42.100 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:28.404853+0000", 2024-09-10T19:22:42.100 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.047359471 2024-09-10T19:22:42.100 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:42.100 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:42.101 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:22:42.101 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:28.405508+0000", 2024-09-10T19:22:42.101 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00065505900000000004 2024-09-10T19:22:42.101 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:42.101 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:42.101 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-09-10T19:22:42.101 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:28.406505+0000", 2024-09-10T19:22:42.101 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00099746100000000005 2024-09-10T19:22:42.101 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:42.101 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:42.101 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-10T19:22:42.101 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:28.410655+0000", 2024-09-10T19:22:42.101 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0041496019999999996 2024-09-10T19:22:42.101 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:22:42.102 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-10T19:22:42.102 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:22:42.102 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:42.102 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:42.102 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.15051.0:93350 8.3 8:c18fa4fe:::benchmark_data_smithi183_114924_object93349:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e178)", 2024-09-10T19:22:42.102 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-10T19:17:27.234632+0000", 2024-09-10T19:22:42.102 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 314.79492584100001, 2024-09-10T19:22:42.102 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.1849550170000001, 2024-09-10T19:22:42.102 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-10T19:22:42.102 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T19:22:42.102 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-10T19:22:42.102 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.15051", 2024-09-10T19:22:42.102 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.183:0/3632028672", 2024-09-10T19:22:42.103 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 93350 2024-09-10T19:22:42.103 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:42.103 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-10T19:22:42.103 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:42.103 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-10T19:22:42.103 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:27.234632+0000", 2024-09-10T19:22:42.103 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:22:42.103 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:42.103 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:42.103 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-10T19:22:42.103 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:27.234632+0000", 2024-09-10T19:22:42.103 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:22:42.103 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:42.103 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:42.104 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-10T19:22:42.104 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:27.234630+0000", 2024-09-10T19:22:42.104 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999986 2024-09-10T19:22:42.104 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:42.104 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:42.104 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-10T19:22:42.104 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:27.234688+0000", 2024-09-10T19:22:42.104 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.7290000000000002e-05 2024-09-10T19:22:42.104 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:42.104 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:42.104 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-10T19:22:42.104 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:27.234701+0000", 2024-09-10T19:22:42.104 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.2809999999999999e-05 2024-09-10T19:22:42.104 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:42.105 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:42.105 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-10T19:22:42.105 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:27.234705+0000", 2024-09-10T19:22:42.105 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.5650000000000003e-06 2024-09-10T19:22:42.105 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:42.105 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:42.105 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:22:42.105 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:27.234750+0000", 2024-09-10T19:22:42.105 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.4950000000000002e-05 2024-09-10T19:22:42.105 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:42.105 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:42.105 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for peered", 2024-09-10T19:22:42.105 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:27.234768+0000", 2024-09-10T19:22:42.105 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.7472000000000001e-05 2024-09-10T19:22:42.106 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:42.106 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:42.106 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:22:42.106 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:28.062852+0000", 2024-09-10T19:22:42.106 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.82808465099999995 2024-09-10T19:22:42.106 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:42.106 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:42.106 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-10T19:22:42.106 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:28.065344+0000", 2024-09-10T19:22:42.106 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0024914149999999999 2024-09-10T19:22:42.106 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:42.106 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:42.106 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 3,5", 2024-09-10T19:22:42.106 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:28.070338+0000", 2024-09-10T19:22:42.106 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0049941220000000001 2024-09-10T19:22:42.106 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:42.107 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:42.107 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-09-10T19:22:42.107 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:28.357694+0000", 2024-09-10T19:22:42.107 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.287356587 2024-09-10T19:22:42.107 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:42.107 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:42.107 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:22:42.107 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:28.411581+0000", 2024-09-10T19:22:42.107 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.053886430999999999 2024-09-10T19:22:42.107 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:42.107 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:42.107 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:22:42.107 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:28.412526+0000", 2024-09-10T19:22:42.107 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00094505499999999998 2024-09-10T19:22:42.108 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:42.108 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:42.108 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-09-10T19:22:42.108 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:28.414135+0000", 2024-09-10T19:22:42.108 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0016088039999999999 2024-09-10T19:22:42.108 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:42.108 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:42.108 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-10T19:22:42.108 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:28.419587+0000", 2024-09-10T19:22:42.108 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0054521830000000002 2024-09-10T19:22:42.108 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:22:42.108 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-10T19:22:42.108 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:22:42.108 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:42.109 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:42.109 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDPGPush(8.13 190/180 [PushOp(8:c984e6b4:::benchmark_data_smithi183_114924_object20685:head, version: 159'1352, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(8:c984e6b4:::benchmark_data_smithi183_114924_object20685:head@159'1352, size: 65536, copy_subset: [0~65536], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:65536, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-10T19:22:42.109 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-10T19:17:53.705526+0000", 2024-09-10T19:22:42.109 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 288.32403207999999, 2024-09-10T19:22:42.109 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.1027652100000001, 2024-09-10T19:22:42.109 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-10T19:22:42.109 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "started", 2024-09-10T19:22:42.109 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-10T19:22:42.109 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:42.109 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-10T19:22:42.109 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:53.705526+0000", 2024-09-10T19:22:42.109 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:22:42.109 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:42.110 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:42.110 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-10T19:22:42.110 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:53.705526+0000", 2024-09-10T19:22:42.110 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:22:42.110 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:42.110 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:42.110 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-10T19:22:42.110 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:53.705525+0000", 2024-09-10T19:22:42.110 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.999999 2024-09-10T19:22:42.110 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:42.110 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:42.110 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-10T19:22:42.110 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:53.705571+0000", 2024-09-10T19:22:42.110 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.6814999999999998e-05 2024-09-10T19:22:42.111 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:42.111 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:42.111 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-10T19:22:42.111 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:53.705590+0000", 2024-09-10T19:22:42.111 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.861e-05 2024-09-10T19:22:42.111 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:42.111 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:42.111 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-10T19:22:42.111 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:53.705601+0000", 2024-09-10T19:22:42.111 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.0787e-05 2024-09-10T19:22:42.111 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:42.111 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:42.111 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:22:42.111 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:54.807476+0000", 2024-09-10T19:22:42.112 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.1018752549999999 2024-09-10T19:22:42.112 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:42.112 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:42.112 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-10T19:22:42.112 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:54.807481+0000", 2024-09-10T19:22:42.112 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.6800000000000001e-06 2024-09-10T19:22:42.112 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:42.112 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:42.112 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-10T19:22:42.112 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:54.808291+0000", 2024-09-10T19:22:42.112 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00081010900000000005 2024-09-10T19:22:42.112 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:22:42.112 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-10T19:22:42.112 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:22:42.113 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:42.113 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:42.113 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDPGPush(8.13 191/180 [PushOp(8:ce4937bd:::benchmark_data_smithi183_114924_object20690:head, version: 159'1353, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(8:ce4937bd:::benchmark_data_smithi183_114924_object20690:head@159'1353, size: 65536, copy_subset: [0~65536], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:65536, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-10T19:22:42.113 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-10T19:17:53.824796+0000", 2024-09-10T19:22:42.113 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 288.20476121500002, 2024-09-10T19:22:42.113 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.98403609800000003, 2024-09-10T19:22:42.113 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-10T19:22:42.113 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "started", 2024-09-10T19:22:42.113 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-10T19:22:42.113 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:42.113 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-10T19:22:42.113 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:53.824796+0000", 2024-09-10T19:22:42.113 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:22:42.114 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:42.114 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:42.114 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-10T19:22:42.114 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:53.824796+0000", 2024-09-10T19:22:42.114 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:22:42.114 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:42.114 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:42.114 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-10T19:22:42.114 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:53.824796+0000", 2024-09-10T19:22:42.114 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.999999 2024-09-10T19:22:42.114 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:42.114 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:42.114 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-10T19:22:42.114 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:53.824838+0000", 2024-09-10T19:22:42.115 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.1989000000000001e-05 2024-09-10T19:22:42.115 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:42.115 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:42.115 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-10T19:22:42.115 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:53.824855+0000", 2024-09-10T19:22:42.115 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.6866000000000001e-05 2024-09-10T19:22:42.115 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:42.115 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:42.115 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-10T19:22:42.115 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:53.824864+0000", 2024-09-10T19:22:42.115 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.6900000000000004e-06 2024-09-10T19:22:42.115 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:42.115 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:42.115 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:22:42.116 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:54.808328+0000", 2024-09-10T19:22:42.116 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.98346424200000004 2024-09-10T19:22:42.116 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:42.116 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:42.116 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-10T19:22:42.116 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:54.808333+0000", 2024-09-10T19:22:42.116 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.2150000000000003e-06 2024-09-10T19:22:42.116 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:42.116 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:42.116 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-10T19:22:42.116 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:54.808833+0000", 2024-09-10T19:22:42.116 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00049981699999999995 2024-09-10T19:22:42.116 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:22:42.116 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-10T19:22:42.117 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:22:42.117 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:42.117 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:42.117 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDPGPush(8.13 192/180 [PushOp(8:cf919698:::benchmark_data_smithi183_114924_object26566:head, version: 159'1771, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(8:cf919698:::benchmark_data_smithi183_114924_object26566:head@159'1771, size: 65536, copy_subset: [0~65536], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:65536, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-10T19:22:42.117 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-10T19:17:56.866594+0000", 2024-09-10T19:22:42.117 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 285.16296407300001, 2024-09-10T19:22:42.117 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.49036462, 2024-09-10T19:22:42.117 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-10T19:22:42.117 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "started", 2024-09-10T19:22:42.117 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-10T19:22:42.117 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:42.117 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-10T19:22:42.117 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:56.866594+0000", 2024-09-10T19:22:42.118 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:22:42.118 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:42.118 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:42.118 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-10T19:22:42.118 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:56.866594+0000", 2024-09-10T19:22:42.118 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:22:42.118 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:42.118 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:42.118 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-10T19:22:42.118 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:56.866593+0000", 2024-09-10T19:22:42.118 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999995 2024-09-10T19:22:42.118 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:42.118 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:42.118 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-10T19:22:42.118 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:56.866670+0000", 2024-09-10T19:22:42.119 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.6539999999999996e-05 2024-09-10T19:22:42.119 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:42.119 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:42.119 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-10T19:22:42.119 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:56.866685+0000", 2024-09-10T19:22:42.119 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.5319999999999999e-05 2024-09-10T19:22:42.119 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:42.119 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:42.119 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-10T19:22:42.119 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:56.866693+0000", 2024-09-10T19:22:42.119 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.4980000000000006e-06 2024-09-10T19:22:42.119 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:42.119 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:42.119 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:22:42.120 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:58.356122+0000", 2024-09-10T19:22:42.120 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.4894283750000001 2024-09-10T19:22:42.120 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:42.120 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:42.120 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-10T19:22:42.120 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:58.356126+0000", 2024-09-10T19:22:42.120 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.9260000000000002e-06 2024-09-10T19:22:42.120 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:42.120 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:42.120 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-10T19:22:42.120 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:58.356958+0000", 2024-09-10T19:22:42.120 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00083252900000000004 2024-09-10T19:22:42.120 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:22:42.120 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-10T19:22:42.121 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:22:42.121 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:42.121 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:42.121 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDPGPush(8.13 192/180 [PushOp(8:c8afa908:::benchmark_data_smithi183_114924_object26967:head, version: 159'1795, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(8:c8afa908:::benchmark_data_smithi183_114924_object26967:head@159'1795, size: 65536, copy_subset: [0~65536], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:65536, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-10T19:22:42.121 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-10T19:17:59.974190+0000", 2024-09-10T19:22:42.121 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 282.05536768399998, 2024-09-10T19:22:42.121 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.97933013000000002, 2024-09-10T19:22:42.121 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-10T19:22:42.121 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "started", 2024-09-10T19:22:42.121 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-10T19:22:42.121 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:42.121 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-10T19:22:42.121 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:59.974190+0000", 2024-09-10T19:22:42.121 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:22:42.122 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:42.122 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:42.122 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-10T19:22:42.122 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:59.974190+0000", 2024-09-10T19:22:42.122 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:22:42.122 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:42.122 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:42.122 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-10T19:22:42.122 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:59.974190+0000", 2024-09-10T19:22:42.122 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999995 2024-09-10T19:22:42.122 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:42.122 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:42.122 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-10T19:22:42.122 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:59.974226+0000", 2024-09-10T19:22:42.122 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.6442000000000001e-05 2024-09-10T19:22:42.122 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:42.123 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:42.123 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-10T19:22:42.123 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:59.974243+0000", 2024-09-10T19:22:42.123 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.6586e-05 2024-09-10T19:22:42.123 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:42.123 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:42.123 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-10T19:22:42.123 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:59.974252+0000", 2024-09-10T19:22:42.123 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.4550000000000001e-06 2024-09-10T19:22:42.123 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:42.123 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:42.123 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:22:42.123 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:18:00.951928+0000", 2024-09-10T19:22:42.123 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.97767566100000003 2024-09-10T19:22:42.124 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:42.124 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:42.124 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-10T19:22:42.124 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:18:00.951935+0000", 2024-09-10T19:22:42.124 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.7139999999999996e-06 2024-09-10T19:22:42.124 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:42.124 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:42.124 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-10T19:22:42.124 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:18:00.953520+0000", 2024-09-10T19:22:42.124 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0015855299999999999 2024-09-10T19:22:42.124 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:22:42.124 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-10T19:22:42.124 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:22:42.124 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:42.125 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:42.125 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.15051.0:172172 8.17 8:ec5a1e4c:::benchmark_data_smithi183_114924_object51889:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e205)", 2024-09-10T19:22:42.125 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-10T19:18:43.179473+0000", 2024-09-10T19:22:42.125 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 238.850084901, 2024-09-10T19:22:42.125 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.99473523600000002, 2024-09-10T19:22:42.125 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-10T19:22:42.125 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T19:22:42.125 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-10T19:22:42.125 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.15051", 2024-09-10T19:22:42.125 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.183:0/3632028672", 2024-09-10T19:22:42.125 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 172172 2024-09-10T19:22:42.125 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:42.125 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-10T19:22:42.126 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:42.126 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-10T19:22:42.126 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:18:43.179473+0000", 2024-09-10T19:22:42.126 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:22:42.126 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:42.126 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:42.126 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-10T19:22:42.126 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:18:43.179473+0000", 2024-09-10T19:22:42.126 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:22:42.126 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:42.126 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:42.126 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-10T19:22:42.127 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:18:43.179471+0000", 2024-09-10T19:22:42.127 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999986 2024-09-10T19:22:42.127 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:42.127 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:42.127 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-10T19:22:42.127 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:18:43.179478+0000", 2024-09-10T19:22:42.127 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.2910000000000004e-06 2024-09-10T19:22:42.127 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:42.127 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:42.128 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-10T19:22:42.128 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:18:43.179489+0000", 2024-09-10T19:22:42.128 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.1528e-05 2024-09-10T19:22:42.128 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:42.128 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:42.128 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-10T19:22:42.128 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:18:43.179495+0000", 2024-09-10T19:22:42.128 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.0480000000000004e-06 2024-09-10T19:22:42.128 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:42.128 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:42.128 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:22:42.128 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:18:43.179674+0000", 2024-09-10T19:22:42.128 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00017865900000000001 2024-09-10T19:22:42.128 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:42.128 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:42.129 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for peered", 2024-09-10T19:22:42.129 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:18:43.179687+0000", 2024-09-10T19:22:42.129 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.2853999999999999e-05 2024-09-10T19:22:42.129 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:42.129 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:42.129 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:22:42.129 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:18:44.165270+0000", 2024-09-10T19:22:42.129 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.98558308299999997 2024-09-10T19:22:42.129 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:42.129 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:42.129 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-10T19:22:42.129 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:18:44.165366+0000", 2024-09-10T19:22:42.129 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.6162000000000005e-05 2024-09-10T19:22:42.129 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:42.130 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:42.130 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0,6", 2024-09-10T19:22:42.130 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:18:44.165590+0000", 2024-09-10T19:22:42.130 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00022405299999999999 2024-09-10T19:22:42.130 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:42.130 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:42.130 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-09-10T19:22:42.130 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:18:44.172416+0000", 2024-09-10T19:22:42.130 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0068258320000000004 2024-09-10T19:22:42.130 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:42.130 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:42.130 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:22:42.130 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:18:44.174090+0000", 2024-09-10T19:22:42.130 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0016737659999999999 2024-09-10T19:22:42.130 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:42.130 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:42.131 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:22:42.131 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:18:44.174127+0000", 2024-09-10T19:22:42.131 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.7360999999999997e-05 2024-09-10T19:22:42.131 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:42.131 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:42.131 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-09-10T19:22:42.131 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:18:44.174171+0000", 2024-09-10T19:22:42.131 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.4270999999999997e-05 2024-09-10T19:22:42.131 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:42.131 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:42.131 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-10T19:22:42.131 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:18:44.174208+0000", 2024-09-10T19:22:42.131 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.6720000000000001e-05 2024-09-10T19:22:42.131 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:22:42.132 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-10T19:22:42.132 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:22:42.132 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:42.132 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:42.132 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.15051.0:172461 8.15 8:aff08a5b:::benchmark_data_smithi183_114924_object52178:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e206)", 2024-09-10T19:22:42.132 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-10T19:18:44.222397+0000", 2024-09-10T19:22:42.132 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 237.80716058499999, 2024-09-10T19:22:42.132 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.95883427200000004, 2024-09-10T19:22:42.132 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-10T19:22:42.132 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T19:22:42.132 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-10T19:22:42.132 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.15051", 2024-09-10T19:22:42.132 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.183:0/3632028672", 2024-09-10T19:22:42.133 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 172461 2024-09-10T19:22:42.133 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:42.133 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-10T19:22:42.133 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:42.133 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-10T19:22:42.133 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:18:44.222397+0000", 2024-09-10T19:22:42.133 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:22:42.133 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:42.133 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:42.133 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-10T19:22:42.133 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:18:44.222397+0000", 2024-09-10T19:22:42.133 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:22:42.133 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:42.133 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:42.134 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-10T19:22:42.134 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:18:44.222396+0000", 2024-09-10T19:22:42.134 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999986 2024-09-10T19:22:42.134 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:42.134 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:42.134 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-10T19:22:42.134 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:18:44.222403+0000", 2024-09-10T19:22:42.134 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.8820000000000003e-06 2024-09-10T19:22:42.134 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:42.134 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:42.134 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-10T19:22:42.134 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:18:44.222416+0000", 2024-09-10T19:22:42.134 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.3291e-05 2024-09-10T19:22:42.134 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:42.134 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:42.135 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-10T19:22:42.135 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:18:44.222422+0000", 2024-09-10T19:22:42.135 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.2580000000000001e-06 2024-09-10T19:22:42.135 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:42.135 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:42.135 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:22:42.135 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:18:44.222507+0000", 2024-09-10T19:22:42.135 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.5118999999999996e-05 2024-09-10T19:22:42.135 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:42.135 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:42.135 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for peered", 2024-09-10T19:22:42.135 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:18:44.222515+0000", 2024-09-10T19:22:42.135 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.6380000000000004e-06 2024-09-10T19:22:42.135 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:42.135 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:42.136 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:22:42.136 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:18:45.163741+0000", 2024-09-10T19:22:42.136 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.94122644200000005 2024-09-10T19:22:42.136 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:42.136 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:42.136 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-10T19:22:42.136 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:18:45.163835+0000", 2024-09-10T19:22:42.136 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.3490000000000001e-05 2024-09-10T19:22:42.136 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:42.136 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:42.136 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0,6", 2024-09-10T19:22:42.136 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:18:45.164047+0000", 2024-09-10T19:22:42.136 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00021215599999999999 2024-09-10T19:22:42.136 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:42.137 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:42.137 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-09-10T19:22:42.137 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:18:45.168946+0000", 2024-09-10T19:22:42.137 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0048991410000000001 2024-09-10T19:22:42.137 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:42.137 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:42.137 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:22:42.137 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:18:45.181058+0000", 2024-09-10T19:22:42.137 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.012112303 2024-09-10T19:22:42.137 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:42.137 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:42.137 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:22:42.137 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:18:45.181172+0000", 2024-09-10T19:22:42.137 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00011330100000000001 2024-09-10T19:22:42.137 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:42.138 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:42.138 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-09-10T19:22:42.138 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:18:45.181202+0000", 2024-09-10T19:22:42.138 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.0518999999999997e-05 2024-09-10T19:22:42.138 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:42.138 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:42.138 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-10T19:22:42.138 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:18:45.181231+0000", 2024-09-10T19:22:42.138 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.9150999999999999e-05 2024-09-10T19:22:42.138 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:22:42.138 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-10T19:22:42.138 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:22:42.138 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:42.138 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:42.138 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.15360.0:169775 9.9 9:9da34f67:::benchmark_data_smithi183_142499_object23286:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e263)", 2024-09-10T19:22:42.139 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-10T19:21:46.960468+0000", 2024-09-10T19:22:42.139 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 55.069089716000001, 2024-09-10T19:22:42.139 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.0671650109999999, 2024-09-10T19:22:42.139 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-10T19:22:42.139 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T19:22:42.139 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-10T19:22:42.139 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.15360", 2024-09-10T19:22:42.139 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.183:0/2364804131", 2024-09-10T19:22:42.139 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 169775 2024-09-10T19:22:42.139 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:42.139 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-10T19:22:42.139 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:42.139 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-10T19:22:42.139 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:21:46.960468+0000", 2024-09-10T19:22:42.139 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:22:42.139 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:42.139 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:42.140 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-10T19:22:42.140 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:21:46.960468+0000", 2024-09-10T19:22:42.140 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:22:42.140 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:42.140 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:42.140 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-10T19:22:42.140 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:21:46.960466+0000", 2024-09-10T19:22:42.140 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999981 2024-09-10T19:22:42.140 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:42.140 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:42.140 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-10T19:22:42.140 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:21:46.960473+0000", 2024-09-10T19:22:42.140 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.1659999999999997e-06 2024-09-10T19:22:42.140 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:42.141 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:42.141 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-10T19:22:42.141 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:21:46.960486+0000", 2024-09-10T19:22:42.141 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.2972e-05 2024-09-10T19:22:42.141 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:42.141 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:42.141 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-10T19:22:42.141 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:21:46.960493+0000", 2024-09-10T19:22:42.141 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.1910000000000003e-06 2024-09-10T19:22:42.141 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:42.141 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:42.141 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:22:42.141 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:21:46.991408+0000", 2024-09-10T19:22:42.141 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.030915074000000001 2024-09-10T19:22:42.141 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:42.142 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:42.142 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for peered", 2024-09-10T19:22:42.142 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:21:46.991416+0000", 2024-09-10T19:22:42.142 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.5140000000000004e-06 2024-09-10T19:22:42.142 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:42.142 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:42.142 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:22:42.142 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:21:48.013668+0000", 2024-09-10T19:22:42.142 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.022252231 2024-09-10T19:22:42.142 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:42.142 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:42.142 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-10T19:22:42.142 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:21:48.015474+0000", 2024-09-10T19:22:42.143 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0018054760000000001 2024-09-10T19:22:42.143 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:42.143 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:42.143 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 5", 2024-09-10T19:22:42.143 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:21:48.015713+0000", 2024-09-10T19:22:42.143 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00023921599999999999 2024-09-10T19:22:42.143 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:42.143 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:42.143 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-09-10T19:22:42.143 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:21:48.024289+0000", 2024-09-10T19:22:42.143 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0085763880000000008 2024-09-10T19:22:42.143 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:42.143 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:42.143 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:22:42.143 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:21:48.027515+0000", 2024-09-10T19:22:42.144 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0032260079999999998 2024-09-10T19:22:42.144 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:42.144 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:42.144 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-09-10T19:22:42.144 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:21:48.027559+0000", 2024-09-10T19:22:42.144 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.3996999999999999e-05 2024-09-10T19:22:42.144 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:42.144 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:42.144 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-10T19:22:42.144 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:21:48.027633+0000", 2024-09-10T19:22:42.144 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.3619000000000001e-05 2024-09-10T19:22:42.144 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:22:42.144 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-10T19:22:42.144 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:22:42.145 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:42.145 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:42.145 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.15360.0:169783 9.9 9:93db4d72:::benchmark_data_smithi183_142499_object23294:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e263)", 2024-09-10T19:22:42.145 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-10T19:21:46.991598+0000", 2024-09-10T19:22:42.145 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 55.037959862999998, 2024-09-10T19:22:42.145 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.0422353129999999, 2024-09-10T19:22:42.145 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-10T19:22:42.145 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T19:22:42.145 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-10T19:22:42.145 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.15360", 2024-09-10T19:22:42.145 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.183:0/2364804131", 2024-09-10T19:22:42.145 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 169783 2024-09-10T19:22:42.145 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:42.146 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-10T19:22:42.146 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:42.146 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-10T19:22:42.146 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:21:46.991598+0000", 2024-09-10T19:22:42.146 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:22:42.146 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:42.146 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:42.146 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-10T19:22:42.146 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:21:46.991598+0000", 2024-09-10T19:22:42.146 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:22:42.146 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:42.146 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:42.146 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-10T19:22:42.146 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:21:46.991597+0000", 2024-09-10T19:22:42.147 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999986 2024-09-10T19:22:42.147 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:42.147 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:42.147 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-10T19:22:42.147 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:21:46.991602+0000", 2024-09-10T19:22:42.147 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.5380000000000002e-06 2024-09-10T19:22:42.147 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:42.147 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:42.147 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-10T19:22:42.147 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:21:46.991612+0000", 2024-09-10T19:22:42.147 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.5289999999999997e-06 2024-09-10T19:22:42.147 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:42.147 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:42.147 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-10T19:22:42.148 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:21:46.991617+0000", 2024-09-10T19:22:42.148 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.5110000000000003e-06 2024-09-10T19:22:42.148 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:42.148 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:42.148 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:22:42.148 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:21:46.991868+0000", 2024-09-10T19:22:42.148 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000251339 2024-09-10T19:22:42.148 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:42.148 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:42.148 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for peered", 2024-09-10T19:22:42.148 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:21:46.991875+0000", 2024-09-10T19:22:42.148 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.2600000000000002e-06 2024-09-10T19:22:42.148 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:42.148 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:42.148 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:22:42.149 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:21:48.020111+0000", 2024-09-10T19:22:42.149 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.0282362519999999 2024-09-10T19:22:42.149 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:42.149 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:42.149 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-10T19:22:42.149 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:21:48.020206+0000", 2024-09-10T19:22:42.149 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.5179000000000005e-05 2024-09-10T19:22:42.149 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:42.149 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:42.149 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 5", 2024-09-10T19:22:42.149 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:21:48.020309+0000", 2024-09-10T19:22:42.149 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00010310999999999999 2024-09-10T19:22:42.149 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:42.150 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:42.150 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-09-10T19:22:42.150 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:21:48.027641+0000", 2024-09-10T19:22:42.150 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0073316939999999997 2024-09-10T19:22:42.150 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:42.150 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:42.150 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:22:42.150 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:21:48.033698+0000", 2024-09-10T19:22:42.150 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0060571660000000001 2024-09-10T19:22:42.150 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:42.150 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:42.150 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-09-10T19:22:42.150 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:21:48.033745+0000", 2024-09-10T19:22:42.150 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.6461000000000001e-05 2024-09-10T19:22:42.150 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:42.151 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:42.151 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-10T19:22:42.151 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:21:48.033833+0000", 2024-09-10T19:22:42.151 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.8530999999999997e-05 2024-09-10T19:22:42.151 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:22:42.151 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-10T19:22:42.151 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:22:42.151 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:42.151 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:42.151 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.15360.0:169785 9.9 9:9cc1dce6:::benchmark_data_smithi183_142499_object23296:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e263)", 2024-09-10T19:22:42.151 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-10T19:21:46.993227+0000", 2024-09-10T19:22:42.151 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 55.036330956999997, 2024-09-10T19:22:42.151 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.0407537469999999, 2024-09-10T19:22:42.151 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-10T19:22:42.152 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T19:22:42.152 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-10T19:22:42.152 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.15360", 2024-09-10T19:22:42.152 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.183:0/2364804131", 2024-09-10T19:22:42.152 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 169785 2024-09-10T19:22:42.152 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:42.152 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-10T19:22:42.152 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:42.152 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-10T19:22:42.152 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:21:46.993227+0000", 2024-09-10T19:22:42.152 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:22:42.152 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:42.152 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:42.152 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-10T19:22:42.152 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:21:46.993227+0000", 2024-09-10T19:22:42.153 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:22:42.153 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:42.153 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:42.153 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-10T19:22:42.153 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:21:46.993226+0000", 2024-09-10T19:22:42.153 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.999999 2024-09-10T19:22:42.153 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:42.153 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:42.153 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-10T19:22:42.153 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:21:46.993231+0000", 2024-09-10T19:22:42.153 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.5840000000000002e-06 2024-09-10T19:22:42.153 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:42.153 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:42.154 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-10T19:22:42.154 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:21:46.993243+0000", 2024-09-10T19:22:42.154 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.1686e-05 2024-09-10T19:22:42.154 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:42.154 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:42.154 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-10T19:22:42.154 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:21:46.993249+0000", 2024-09-10T19:22:42.154 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.0780000000000001e-06 2024-09-10T19:22:42.154 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:42.154 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:42.154 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:22:42.154 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:21:46.993347+0000", 2024-09-10T19:22:42.154 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.8405000000000004e-05 2024-09-10T19:22:42.154 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:42.154 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:42.155 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for peered", 2024-09-10T19:22:42.155 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:21:46.993354+0000", 2024-09-10T19:22:42.155 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.7390000000000002e-06 2024-09-10T19:22:42.155 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:42.155 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:42.155 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:22:42.155 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:21:48.022236+0000", 2024-09-10T19:22:42.155 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.028881905 2024-09-10T19:22:42.155 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:42.155 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:42.155 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-10T19:22:42.155 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:21:48.022331+0000", 2024-09-10T19:22:42.155 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.4734000000000005e-05 2024-09-10T19:22:42.155 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:42.156 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:42.156 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 5", 2024-09-10T19:22:42.156 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:21:48.022436+0000", 2024-09-10T19:22:42.156 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00010508799999999999 2024-09-10T19:22:42.156 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:42.156 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:42.156 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-09-10T19:22:42.156 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:21:48.027646+0000", 2024-09-10T19:22:42.156 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.005209984 2024-09-10T19:22:42.156 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:42.156 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:42.156 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:22:42.156 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:21:48.033863+0000", 2024-09-10T19:22:42.156 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0062175030000000001 2024-09-10T19:22:42.157 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:42.157 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:42.157 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-09-10T19:22:42.157 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:21:48.033907+0000", 2024-09-10T19:22:42.157 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.3958000000000002e-05 2024-09-10T19:22:42.157 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:22:42.157 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:22:42.157 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-10T19:22:42.157 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:21:48.033980+0000", 2024-09-10T19:22:42.157 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.3159999999999997e-05 2024-09-10T19:22:42.157 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:22:42.157 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-10T19:22:42.157 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:22:42.157 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:22:42.157 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-10T19:22:42.158 INFO:teuthology.orchestra.run.smithi191.stdout:} 2024-09-10T19:22:42.189 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:22:42.209 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:22:42.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:22:42.210 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:22:42.311 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:22:42.345 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:22:42.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:22:42.346 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:22:42.374 INFO:teuthology.orchestra.run.smithi183.stdout:{ 2024-09-10T19:22:42.374 INFO:teuthology.orchestra.run.smithi183.stdout: "ops": [], 2024-09-10T19:22:42.374 INFO:teuthology.orchestra.run.smithi183.stdout: "num_ops": 0 2024-09-10T19:22:42.374 INFO:teuthology.orchestra.run.smithi183.stdout:} 2024-09-10T19:22:42.447 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:22:42.465 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:22:42 smithi191 ceph-mon[32884]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T19:22:42.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:22:42 smithi191 ceph-mon[32884]: from='client.? 172.21.15.183:0/3453357631' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-10T19:22:42.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:22:42 smithi191 ceph-mon[32884]: from='osd.1 [v2:172.21.15.183:6802/2764394151,v1:172.21.15.183:6803/2764394151]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-09-10T19:22:42.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:22:42 smithi191 ceph-mon[32884]: osdmap e275: 8 total, 6 up, 6 in 2024-09-10T19:22:42.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:22:42 smithi191 ceph-mon[32884]: from='osd.1 [v2:172.21.15.183:6802/2764394151,v1:172.21.15.183:6803/2764394151]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi183", "root=default"]}]: dispatch 2024-09-10T19:22:42.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:22:42 smithi191 ceph-mon[32884]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-10T19:22:42.470 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:22:42.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:22:42.471 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:22:42.571 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:22:42.602 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:22:42.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:22:42.603 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:22:42.681 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:22:42 smithi183 ceph-mon[29736]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T19:22:42.681 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:22:42 smithi183 ceph-mon[29736]: from='client.? 172.21.15.183:0/3453357631' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-10T19:22:42.681 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:22:42 smithi183 ceph-mon[29736]: from='osd.1 [v2:172.21.15.183:6802/2764394151,v1:172.21.15.183:6803/2764394151]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-09-10T19:22:42.681 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:22:42 smithi183 ceph-mon[29736]: osdmap e275: 8 total, 6 up, 6 in 2024-09-10T19:22:42.681 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:22:42 smithi183 ceph-mon[29736]: from='osd.1 [v2:172.21.15.183:6802/2764394151,v1:172.21.15.183:6803/2764394151]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi183", "root=default"]}]: dispatch 2024-09-10T19:22:42.681 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:22:42 smithi183 ceph-mon[29736]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-10T19:22:42.682 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:22:42 smithi183 ceph-mon[41562]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T19:22:42.682 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:22:42 smithi183 ceph-mon[41562]: from='client.? 172.21.15.183:0/3453357631' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-10T19:22:42.682 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:22:42 smithi183 ceph-mon[41562]: from='osd.1 [v2:172.21.15.183:6802/2764394151,v1:172.21.15.183:6803/2764394151]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-09-10T19:22:42.682 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:22:42 smithi183 ceph-mon[41562]: osdmap e275: 8 total, 6 up, 6 in 2024-09-10T19:22:42.682 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:22:42 smithi183 ceph-mon[41562]: from='osd.1 [v2:172.21.15.183:6802/2764394151,v1:172.21.15.183:6803/2764394151]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi183", "root=default"]}]: dispatch 2024-09-10T19:22:42.682 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:22:42 smithi183 ceph-mon[41562]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-10T19:22:42.703 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:22:42.727 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:22:42.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:22:42.728 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:22:42.828 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:22:42.850 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:22:42.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:22:42.851 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:22:42.951 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:22:42.974 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:22:42.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:22:42.975 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:22:43.076 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:22:43.091 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:22:43.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:22:43.113 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:22:43.114 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:22:43.214 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:22:43.244 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:22:43.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:22:43.245 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:22:43.346 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:22:43.377 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:22:43.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:22:43.378 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:22:43.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:22:43 smithi183 ceph-mon[29736]: pgmap v564: 158 pgs: 1 active+clean+scrubbing+deep, 45 active+undersized, 33 active+undersized+degraded, 79 active+clean; 3.6 GiB data, 17 GiB used, 519 GiB / 536 GiB avail; 768 B/s rd, 0 B/s wr, 1.54k op/s; 29357/175665 objects degraded (16.712%) 2024-09-10T19:22:43.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:22:43 smithi183 ceph-mon[29736]: osd.1 [v2:172.21.15.183:6802/2764394151,v1:172.21.15.183:6803/2764394151] boot 2024-09-10T19:22:43.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:22:43 smithi183 ceph-mon[29736]: osdmap e276: 8 total, 7 up, 6 in 2024-09-10T19:22:43.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:22:43 smithi183 ceph-mon[29736]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-10T19:22:43.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:22:43 smithi183 ceph-mon[41562]: pgmap v564: 158 pgs: 1 active+clean+scrubbing+deep, 45 active+undersized, 33 active+undersized+degraded, 79 active+clean; 3.6 GiB data, 17 GiB used, 519 GiB / 536 GiB avail; 768 B/s rd, 0 B/s wr, 1.54k op/s; 29357/175665 objects degraded (16.712%) 2024-09-10T19:22:43.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:22:43 smithi183 ceph-mon[41562]: osd.1 [v2:172.21.15.183:6802/2764394151,v1:172.21.15.183:6803/2764394151] boot 2024-09-10T19:22:43.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:22:43 smithi183 ceph-mon[41562]: osdmap e276: 8 total, 7 up, 6 in 2024-09-10T19:22:43.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:22:43 smithi183 ceph-mon[41562]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-10T19:22:43.478 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:22:43.512 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:22:43.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:22:43.513 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:22:43.613 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:22:43.646 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:22:43.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:22:43.647 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:22:43.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:22:43 smithi191 ceph-mon[32884]: pgmap v564: 158 pgs: 1 active+clean+scrubbing+deep, 45 active+undersized, 33 active+undersized+degraded, 79 active+clean; 3.6 GiB data, 17 GiB used, 519 GiB / 536 GiB avail; 768 B/s rd, 0 B/s wr, 1.54k op/s; 29357/175665 objects degraded (16.712%) 2024-09-10T19:22:43.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:22:43 smithi191 ceph-mon[32884]: osd.1 [v2:172.21.15.183:6802/2764394151,v1:172.21.15.183:6803/2764394151] boot 2024-09-10T19:22:43.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:22:43 smithi191 ceph-mon[32884]: osdmap e276: 8 total, 7 up, 6 in 2024-09-10T19:22:43.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:22:43 smithi191 ceph-mon[32884]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-10T19:22:43.747 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:22:43.768 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:22:43.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:22:43.769 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:22:43.870 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:22:43.894 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:22:43.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:22:43.895 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:22:43.995 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:22:44.018 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:22:44.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:22:44.019 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:22:44.120 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:22:44.143 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:22:44.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:22:44.144 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:22:44.244 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:22:44.282 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:22:44.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:22:44.283 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:22:44.384 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:22:44.423 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:22:44.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:22:44.424 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:22:44.524 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:22:44.548 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:22:44.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:22:44.549 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:22:44.590 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:22:44 smithi183 ceph-mon[29736]: purged_snaps scrub starts 2024-09-10T19:22:44.590 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:22:44 smithi183 ceph-mon[29736]: purged_snaps scrub ok 2024-09-10T19:22:44.591 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:22:44 smithi183 ceph-mon[29736]: osdmap e277: 8 total, 7 up, 6 in 2024-09-10T19:22:44.591 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:22:44 smithi183 ceph-mon[29736]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:22:44.591 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:22:44 smithi183 ceph-mon[41562]: purged_snaps scrub starts 2024-09-10T19:22:44.591 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:22:44 smithi183 ceph-mon[41562]: purged_snaps scrub ok 2024-09-10T19:22:44.591 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:22:44 smithi183 ceph-mon[41562]: osdmap e277: 8 total, 7 up, 6 in 2024-09-10T19:22:44.591 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:22:44 smithi183 ceph-mon[41562]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:22:44.650 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:22:44.690 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:22:44.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:22:44.691 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:22:44.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:22:44 smithi191 ceph-mon[32884]: purged_snaps scrub starts 2024-09-10T19:22:44.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:22:44 smithi191 ceph-mon[32884]: purged_snaps scrub ok 2024-09-10T19:22:44.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:22:44 smithi191 ceph-mon[32884]: osdmap e277: 8 total, 7 up, 6 in 2024-09-10T19:22:44.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:22:44 smithi191 ceph-mon[32884]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:22:44.792 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:22:44.815 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:22:44.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:22:44.816 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:22:44.916 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:22:44.942 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:22:44.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:22:44.943 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:22:45.044 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:22:45.071 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:22:45.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:22:45.072 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:22:45.173 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:22:45.204 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:22:45.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:22:45.205 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:22:45.306 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:22:45.342 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:22:45.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:22:45.343 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:22:45.346 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:22:45 smithi183 ceph-mon[29736]: pgmap v567: 158 pgs: 1 active+clean+scrubbing+deep, 45 active+undersized, 33 active+undersized+degraded, 79 active+clean; 3.6 GiB data, 17 GiB used, 519 GiB / 536 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.38k op/s; 29357/175665 objects degraded (16.712%) 2024-09-10T19:22:45.346 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:22:45 smithi183 ceph-mon[29736]: Health check update: Degraded data redundancy: 29357/175665 objects degraded (16.712%), 33 pgs degraded (PG_DEGRADED) 2024-09-10T19:22:45.346 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:22:45 smithi183 ceph-mon[41562]: pgmap v567: 158 pgs: 1 active+clean+scrubbing+deep, 45 active+undersized, 33 active+undersized+degraded, 79 active+clean; 3.6 GiB data, 17 GiB used, 519 GiB / 536 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.38k op/s; 29357/175665 objects degraded (16.712%) 2024-09-10T19:22:45.346 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:22:45 smithi183 ceph-mon[41562]: Health check update: Degraded data redundancy: 29357/175665 objects degraded (16.712%), 33 pgs degraded (PG_DEGRADED) 2024-09-10T19:22:45.444 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:22:45.468 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:22:45.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:22:45.469 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:22:45.573 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:22:45.577 INFO:teuthology.orchestra.run.smithi183.stderr:Error ENXIO: problem getting command descriptions from osd.0 2024-09-10T19:22:45.594 INFO:teuthology.orchestra.run.smithi183.stdout:osd.1: {} 2024-09-10T19:22:45.595 INFO:teuthology.orchestra.run.smithi183.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-10T19:22:45.597 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:22:45.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:22:45.598 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:22:45.616 INFO:teuthology.orchestra.run.smithi183.stdout:osd.2: {} 2024-09-10T19:22:45.617 INFO:teuthology.orchestra.run.smithi183.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-10T19:22:45.629 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-10T19:22:45.635 INFO:teuthology.orchestra.run.smithi183.stdout:osd.3: {} 2024-09-10T19:22:45.635 INFO:teuthology.orchestra.run.smithi183.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-10T19:22:45.700 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:22:45.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:22:45 smithi191 ceph-mon[32884]: pgmap v567: 158 pgs: 1 active+clean+scrubbing+deep, 45 active+undersized, 33 active+undersized+degraded, 79 active+clean; 3.6 GiB data, 17 GiB used, 519 GiB / 536 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.38k op/s; 29357/175665 objects degraded (16.712%) 2024-09-10T19:22:45.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:22:45 smithi191 ceph-mon[32884]: Health check update: Degraded data redundancy: 29357/175665 objects degraded (16.712%), 33 pgs degraded (PG_DEGRADED) 2024-09-10T19:22:45.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:22:45.729 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:22:45.730 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:22:45.732 INFO:teuthology.orchestra.run.smithi183.stdout:osd.4: {} 2024-09-10T19:22:45.733 INFO:teuthology.orchestra.run.smithi183.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-10T19:22:45.754 INFO:teuthology.orchestra.run.smithi183.stdout:osd.5: {} 2024-09-10T19:22:45.755 INFO:teuthology.orchestra.run.smithi183.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-10T19:22:45.770 INFO:teuthology.orchestra.run.smithi183.stdout:osd.6: {} 2024-09-10T19:22:45.771 INFO:teuthology.orchestra.run.smithi183.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-10T19:22:45.791 INFO:teuthology.orchestra.run.smithi183.stdout:osd.7: {} 2024-09-10T19:22:45.791 INFO:teuthology.orchestra.run.smithi183.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-10T19:22:45.830 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:22:45.855 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:22:45.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:22:45.856 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:22:45.957 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:22:45.988 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:22:45.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:22:45.990 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:22:46.091 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:22:46.126 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:22:46.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:22:46.127 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:22:46.133 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:22:46.228 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:22:46.264 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:22:46.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:22:46.266 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:22:46.367 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:22:46.388 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:22:46.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:22:46.389 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:22:46.490 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:22:46.512 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:22:46.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:22:46.513 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:22:46.614 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:22:46.637 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:22:46.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:22:46.638 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:22:46.681 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:22:46 smithi183 ceph-mon[29736]: from='client.? 172.21.15.183:0/3137911730' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T19:22:46.681 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:22:46 smithi183 ceph-mon[41562]: from='client.? 172.21.15.183:0/3137911730' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T19:22:46.715 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:22:46 smithi191 ceph-mon[32884]: from='client.? 172.21.15.183:0/3137911730' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T19:22:46.717 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-10T19:22:46.718 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-10T19:22:46.739 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:22:46.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:22:46.761 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:22:46.761 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:22:46.862 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:22:46.898 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:22:46.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:22:46.899 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:22:47.000 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:22:47.021 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:22:47.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:22:47.022 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:22:47.123 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:22:47.145 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:22:47.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:22:47.146 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:22:47.247 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:22:47.272 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:22:47.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:22:47.273 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:22:47.311 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:22:47 smithi183 ceph-mon[29736]: pgmap v568: 158 pgs: 1 active+clean+scrubbing+deep, 45 active+undersized, 33 active+undersized+degraded, 79 active+clean; 3.2 GiB data, 17 GiB used, 520 GiB / 536 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 2.01k op/s; 26541/158769 objects degraded (16.717%) 2024-09-10T19:22:47.311 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:22:47 smithi183 ceph-mon[41562]: pgmap v568: 158 pgs: 1 active+clean+scrubbing+deep, 45 active+undersized, 33 active+undersized+degraded, 79 active+clean; 3.2 GiB data, 17 GiB used, 520 GiB / 536 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 2.01k op/s; 26541/158769 objects degraded (16.717%) 2024-09-10T19:22:47.373 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:22:47.396 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:22:47.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:22:47.397 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:22:47.497 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:22:47.520 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:22:47.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:22:47.521 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:22:47.621 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:22:47.642 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:22:47.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:22:47.643 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:22:47.715 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:22:47 smithi191 ceph-mon[32884]: pgmap v568: 158 pgs: 1 active+clean+scrubbing+deep, 45 active+undersized, 33 active+undersized+degraded, 79 active+clean; 3.2 GiB data, 17 GiB used, 520 GiB / 536 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 2.01k op/s; 26541/158769 objects degraded (16.717%) 2024-09-10T19:22:47.716 INFO:journalctl@ceph.mgr.x.smithi191.stdout:Sep 10 19:22:47 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-x[34967]: ::ffff:172.21.15.191 - - [10/Sep/2024:19:22:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T19:22:47.744 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:22:47.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:22:47.777 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:22:47.777 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:22:47.878 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:22:47.898 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:22:47.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:22:47.899 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:22:48.000 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:22:48.033 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:22:48.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:22:48.034 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:22:48.114 INFO:tasks.thrashosds.thrasher:in_osds: [0, 3, 4, 5, 6, 7] out_osds: [1, 2] dead_osds: [0] live_osds: [2, 3, 4, 5, 6, 7, 1] 2024-09-10T19:22:48.114 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-10T19:22:48.115 INFO:tasks.thrashosds.thrasher:Shrinking pool unique_pool_2 2024-09-10T19:22:48.115 INFO:tasks.cephadm.ceph_manager.ceph:contract_pool unique_pool_2 by 10 min 8 2024-09-10T19:22:48.115 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph pg dump --format=json 2024-09-10T19:22:48.134 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:22:48.166 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:22:48.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:22:48.167 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:22:48.267 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:22:48.302 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:22:48.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:22:48.303 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:22:48.404 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:22:48.426 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:22:48.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:22:48.427 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:22:48.528 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:22:48.551 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:22:48.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:22:48.552 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:22:48.653 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:22:48.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:22:48.686 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:22:48.686 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:22:48.718 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-10T19:22:48.758 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:22:48.786 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:22:48.810 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:22:48.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:22:48.811 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:22:48.911 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:22:48.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:22:48.950 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:22:48.950 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:22:49.051 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:22:49.077 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:22:49.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:22:49.078 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:22:49.179 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:22:49.201 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:22:49.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:22:49.202 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:22:49.260 INFO:teuthology.orchestra.run.smithi183.stderr:nodeep-scrub is set 2024-09-10T19:22:49.272 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:22:49 smithi183 ceph-mon[29736]: pgmap v569: 158 pgs: 1 active+clean+scrubbing+deep, 45 active+undersized, 33 active+undersized+degraded, 79 active+clean; 3.2 GiB data, 17 GiB used, 520 GiB / 536 GiB avail; 773 B/s rd, 0 B/s wr, 851 op/s; 26541/158769 objects degraded (16.717%) 2024-09-10T19:22:49.272 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:22:49 smithi183 ceph-mon[29736]: from='client.? 172.21.15.183:0/2150214841' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-10T19:22:49.272 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:22:49 smithi183 ceph-mon[29736]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-10T19:22:49.272 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:22:49 smithi183 ceph-mon[41562]: pgmap v569: 158 pgs: 1 active+clean+scrubbing+deep, 45 active+undersized, 33 active+undersized+degraded, 79 active+clean; 3.2 GiB data, 17 GiB used, 520 GiB / 536 GiB avail; 773 B/s rd, 0 B/s wr, 851 op/s; 26541/158769 objects degraded (16.717%) 2024-09-10T19:22:49.272 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:22:49 smithi183 ceph-mon[41562]: from='client.? 172.21.15.183:0/2150214841' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-10T19:22:49.272 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:22:49 smithi183 ceph-mon[41562]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-10T19:22:49.303 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:22:49.332 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:22:49.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:22:49.333 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:22:49.434 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:22:49.455 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:22:49.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:22:49.456 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:22:49.557 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:22:49.579 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:22:49.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:22:49.580 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:22:49.599 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:22:49 smithi191 ceph-mon[32884]: pgmap v569: 158 pgs: 1 active+clean+scrubbing+deep, 45 active+undersized, 33 active+undersized+degraded, 79 active+clean; 3.2 GiB data, 17 GiB used, 520 GiB / 536 GiB avail; 773 B/s rd, 0 B/s wr, 851 op/s; 26541/158769 objects degraded (16.717%) 2024-09-10T19:22:49.599 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:22:49 smithi191 ceph-mon[32884]: from='client.? 172.21.15.183:0/2150214841' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-10T19:22:49.599 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:22:49 smithi191 ceph-mon[32884]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-10T19:22:49.681 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:22:49.706 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:22:49.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:22:49.707 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:22:49.780 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:22:49.809 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:22:49.836 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:22:49.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:22:49.837 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:22:49.937 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:22:49.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:22:49.964 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:22:49.964 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:22:49.965 INFO:journalctl@ceph.iscsi.iscsi.a.smithi191.stdout:Sep 10 19:22:49 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-iscsi-iscsi-a[61816]: debug there is no tcmu-runner data available 2024-09-10T19:22:50.065 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:22:50.086 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:22:50.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:22:50.086 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:22:50.162 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph daemon osd.1 dump_ops_in_flight 2024-09-10T19:22:50.188 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:22:50.229 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:22:50.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:22:50.230 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:22:50.331 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:22:50.369 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:22:50.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:22:50.370 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:22:50.465 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:22:50 smithi191 ceph-mon[32884]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T19:22:50.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:22:50 smithi191 ceph-mon[32884]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-10T19:22:50.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:22:50 smithi191 ceph-mon[32884]: osdmap e278: 8 total, 7 up, 6 in 2024-09-10T19:22:50.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:22:50 smithi191 ceph-mon[32884]: from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:22:50.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:22:50 smithi191 ceph-mon[32884]: pgmap v571: 158 pgs: 1 active+clean+scrubbing+deep, 45 active+undersized, 33 active+undersized+degraded, 79 active+clean; 3.2 GiB data, 17 GiB used, 520 GiB / 536 GiB avail; 673 B/s rd, 0 B/s wr, 741 op/s; 26541/158769 objects degraded (16.717%) 2024-09-10T19:22:50.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:22:50 smithi191 ceph-mon[32884]: Health check update: Degraded data redundancy: 26541/158769 objects degraded (16.717%), 33 pgs degraded (PG_DEGRADED) 2024-09-10T19:22:50.470 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:22:50.495 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:22:50.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:22:50.495 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:22:50.596 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:22:50.633 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:22:50.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:22:50.635 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:22:50.681 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:22:50 smithi183 ceph-mon[29736]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T19:22:50.681 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:22:50 smithi183 ceph-mon[29736]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-10T19:22:50.681 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:22:50 smithi183 ceph-mon[29736]: osdmap e278: 8 total, 7 up, 6 in 2024-09-10T19:22:50.682 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:22:50 smithi183 ceph-mon[29736]: from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:22:50.682 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:22:50 smithi183 ceph-mon[29736]: pgmap v571: 158 pgs: 1 active+clean+scrubbing+deep, 45 active+undersized, 33 active+undersized+degraded, 79 active+clean; 3.2 GiB data, 17 GiB used, 520 GiB / 536 GiB avail; 673 B/s rd, 0 B/s wr, 741 op/s; 26541/158769 objects degraded (16.717%) 2024-09-10T19:22:50.682 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:22:50 smithi183 ceph-mon[29736]: Health check update: Degraded data redundancy: 26541/158769 objects degraded (16.717%), 33 pgs degraded (PG_DEGRADED) 2024-09-10T19:22:50.682 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:22:50 smithi183 ceph-mon[41562]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T19:22:50.682 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:22:50 smithi183 ceph-mon[41562]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-10T19:22:50.682 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:22:50 smithi183 ceph-mon[41562]: osdmap e278: 8 total, 7 up, 6 in 2024-09-10T19:22:50.682 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:22:50 smithi183 ceph-mon[41562]: from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:22:50.682 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:22:50 smithi183 ceph-mon[41562]: pgmap v571: 158 pgs: 1 active+clean+scrubbing+deep, 45 active+undersized, 33 active+undersized+degraded, 79 active+clean; 3.2 GiB data, 17 GiB used, 520 GiB / 536 GiB avail; 673 B/s rd, 0 B/s wr, 741 op/s; 26541/158769 objects degraded (16.717%) 2024-09-10T19:22:50.682 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:22:50 smithi183 ceph-mon[41562]: Health check update: Degraded data redundancy: 26541/158769 objects degraded (16.717%), 33 pgs degraded (PG_DEGRADED) 2024-09-10T19:22:50.736 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:22:50.758 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:22:50.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:22:50.759 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:22:50.860 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:22:50.893 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:22:50.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:22:50.894 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:22:50.995 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:22:51.018 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:22:51.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:22:51.019 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:22:51.120 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:22:51.143 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:22:51.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:22:51.145 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:22:51.245 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:22:51.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:22:51.279 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:22:51.279 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:22:51.380 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:22:51.405 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:22:51.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:22:51.406 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:22:51.507 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:22:51.530 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:22:51.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:22:51.531 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:22:51.571 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:22:51 smithi183 ceph-mon[29736]: purged_snaps scrub starts 2024-09-10T19:22:51.571 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:22:51 smithi183 ceph-mon[29736]: purged_snaps scrub ok 2024-09-10T19:22:51.571 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:22:51 smithi183 ceph-mon[41562]: purged_snaps scrub starts 2024-09-10T19:22:51.571 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:22:51 smithi183 ceph-mon[41562]: purged_snaps scrub ok 2024-09-10T19:22:51.572 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Sep 10 19:22:51 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-y[29989]: ::ffff:172.21.15.191 - - [10/Sep/2024:19:22:51] "GET /metrics HTTP/1.1" 200 224251 "" "Prometheus/2.43.0" 2024-09-10T19:22:51.632 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:22:51.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:22:51.665 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:22:51.666 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:22:51.715 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:22:51 smithi191 ceph-mon[32884]: purged_snaps scrub starts 2024-09-10T19:22:51.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:22:51 smithi191 ceph-mon[32884]: purged_snaps scrub ok 2024-09-10T19:22:51.766 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:22:51.802 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:22:51.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:22:51.803 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:22:51.904 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:22:51.930 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:22:51.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:22:51.931 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:22:51.971 INFO:teuthology.orchestra.run.smithi183.stderr:Error ENXIO: problem getting command descriptions from osd.0 2024-09-10T19:22:51.976 INFO:teuthology.orchestra.run.smithi183.stdout:osd.1: {} 2024-09-10T19:22:51.976 INFO:teuthology.orchestra.run.smithi183.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-10T19:22:51.996 INFO:teuthology.orchestra.run.smithi183.stdout:osd.2: {} 2024-09-10T19:22:51.996 INFO:teuthology.orchestra.run.smithi183.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-10T19:22:52.015 INFO:teuthology.orchestra.run.smithi183.stdout:osd.3: {} 2024-09-10T19:22:52.016 INFO:teuthology.orchestra.run.smithi183.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-10T19:22:52.031 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:22:52.036 INFO:teuthology.orchestra.run.smithi183.stdout:osd.4: {} 2024-09-10T19:22:52.036 INFO:teuthology.orchestra.run.smithi183.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-10T19:22:52.065 INFO:teuthology.orchestra.run.smithi183.stdout:osd.5: {} 2024-09-10T19:22:52.065 INFO:teuthology.orchestra.run.smithi183.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-10T19:22:52.067 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:22:52.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:22:52.068 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:22:52.091 INFO:teuthology.orchestra.run.smithi183.stdout:osd.6: {} 2024-09-10T19:22:52.092 INFO:teuthology.orchestra.run.smithi183.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-10T19:22:52.118 INFO:teuthology.orchestra.run.smithi183.stdout:osd.7: {} 2024-09-10T19:22:52.118 INFO:teuthology.orchestra.run.smithi183.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-10T19:22:52.169 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:22:52.207 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:22:52.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:22:52.208 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:22:52.308 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:22:52.343 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:22:52.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:22:52.344 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:22:52.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:22:52 smithi183 ceph-mon[29736]: osdmap e279: 8 total, 7 up, 6 in 2024-09-10T19:22:52.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:22:52 smithi183 ceph-mon[29736]: pgmap v573: 158 pgs: 45 active+undersized, 33 active+undersized+degraded, 80 active+clean; 2.9 GiB data, 16 GiB used, 521 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.33k op/s; 23995/143679 objects degraded (16.700%) 2024-09-10T19:22:52.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:22:52 smithi183 ceph-mon[29736]: from='client.? 172.21.15.183:0/1820740461' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T19:22:52.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:22:52 smithi183 ceph-mon[41562]: osdmap e279: 8 total, 7 up, 6 in 2024-09-10T19:22:52.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:22:52 smithi183 ceph-mon[41562]: pgmap v573: 158 pgs: 45 active+undersized, 33 active+undersized+degraded, 80 active+clean; 2.9 GiB data, 16 GiB used, 521 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.33k op/s; 23995/143679 objects degraded (16.700%) 2024-09-10T19:22:52.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:22:52 smithi183 ceph-mon[41562]: from='client.? 172.21.15.183:0/1820740461' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T19:22:52.444 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:22:52.482 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:22:52.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:22:52.483 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:22:52.584 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:22:52.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:22:52.621 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:22:52.621 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:22:52.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:22:52 smithi191 ceph-mon[32884]: osdmap e279: 8 total, 7 up, 6 in 2024-09-10T19:22:52.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:22:52 smithi191 ceph-mon[32884]: pgmap v573: 158 pgs: 45 active+undersized, 33 active+undersized+degraded, 80 active+clean; 2.9 GiB data, 16 GiB used, 521 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.33k op/s; 23995/143679 objects degraded (16.700%) 2024-09-10T19:22:52.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:22:52 smithi191 ceph-mon[32884]: from='client.? 172.21.15.183:0/1820740461' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T19:22:52.722 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:22:52.745 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:22:52.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:22:52.746 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:22:52.847 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:22:52.882 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:22:52.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:22:52.883 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:22:52.984 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:22:53.009 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:22:53.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:22:53.011 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:22:53.111 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:22:53.149 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:22:53.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:22:53.150 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:22:53.251 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:22:53.289 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:22:53.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:22:53.290 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:22:53.391 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:22:53.414 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:22:53.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:22:53.415 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:22:53.515 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:22:53.554 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:22:53.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:22:53.555 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:22:53.656 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:22:53.695 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:22:53.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:22:53.696 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:22:53.718 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-10T19:22:53.718 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-10T19:22:53.730 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:22:53.797 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:22:53.835 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:22:53.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:22:53.836 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:22:53.937 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:22:53.969 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:22:53.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:22:53.971 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:22:54.071 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:22:54.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:22:54.107 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:22:54.108 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:22:54.208 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:22:54.244 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:22:54.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:22:54.245 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:22:54.346 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:22:54.371 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:22:54.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:22:54.372 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:22:54.472 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:22:54.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:22:54.497 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:22:54.497 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:22:54.598 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:22:54.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:22:54.633 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:22:54.633 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:22:54.734 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:22:54.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:22:54.771 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:22:54.771 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:22:54.872 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:22:54.908 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:22:54.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:22:54.909 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:22:55.010 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:22:55.027 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph osd unset noscrub 2024-09-10T19:22:55.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:22:55.046 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:22:55.046 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:22:55.147 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:22:55.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:22:54 smithi183 ceph-mon[29736]: pgmap v574: 158 pgs: 45 active+undersized, 33 active+undersized+degraded, 80 active+clean; 2.9 GiB data, 16 GiB used, 521 GiB / 536 GiB avail; 639 B/s rd, 0 B/s wr, 629 op/s; 23995/143679 objects degraded (16.700%) 2024-09-10T19:22:55.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:22:54 smithi183 ceph-mon[29736]: Health check update: Degraded data redundancy: 23995/143679 objects degraded (16.700%), 33 pgs degraded, 78 pgs undersized (PG_DEGRADED) 2024-09-10T19:22:55.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:22:54 smithi183 ceph-mon[41562]: pgmap v574: 158 pgs: 45 active+undersized, 33 active+undersized+degraded, 80 active+clean; 2.9 GiB data, 16 GiB used, 521 GiB / 536 GiB avail; 639 B/s rd, 0 B/s wr, 629 op/s; 23995/143679 objects degraded (16.700%) 2024-09-10T19:22:55.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:22:54 smithi183 ceph-mon[41562]: Health check update: Degraded data redundancy: 23995/143679 objects degraded (16.700%), 33 pgs degraded, 78 pgs undersized (PG_DEGRADED) 2024-09-10T19:22:55.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:22:55.188 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:22:55.189 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:22:55.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:22:54 smithi191 ceph-mon[32884]: pgmap v574: 158 pgs: 45 active+undersized, 33 active+undersized+degraded, 80 active+clean; 2.9 GiB data, 16 GiB used, 521 GiB / 536 GiB avail; 639 B/s rd, 0 B/s wr, 629 op/s; 23995/143679 objects degraded (16.700%) 2024-09-10T19:22:55.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:22:54 smithi191 ceph-mon[32884]: Health check update: Degraded data redundancy: 23995/143679 objects degraded (16.700%), 33 pgs degraded, 78 pgs undersized (PG_DEGRADED) 2024-09-10T19:22:55.289 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:22:55.325 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:22:55.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:22:55.326 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:22:55.427 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:22:55.452 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:22:55.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:22:55.454 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:22:55.554 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:22:55.577 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:22:55.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:22:55.578 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:22:55.679 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:22:55.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:22:55.715 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:22:55.715 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:22:55.719 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-10T19:22:55.816 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:22:55.847 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:22:55.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:22:55.848 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:22:55.949 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:22:55.972 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:22:55.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:22:55.973 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:22:56.074 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:22:56.107 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:22:56.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:22:56.108 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:22:56.209 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:22:56.232 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:22:56.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:22:56.233 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:22:56.334 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:22:56.351 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-09-10T19:22:56.353 INFO:teuthology.orchestra.run.smithi183.stderr:dumped all 2024-09-10T19:22:56.367 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:22:56.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:22:56.369 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:22:56.405 INFO:teuthology.orchestra.run.smithi183.stdout:{ 2024-09-10T19:22:56.405 INFO:teuthology.orchestra.run.smithi183.stdout: "ops": [], 2024-09-10T19:22:56.405 INFO:teuthology.orchestra.run.smithi183.stdout: "num_ops": 0 2024-09-10T19:22:56.406 INFO:teuthology.orchestra.run.smithi183.stdout:} 2024-09-10T19:22:56.469 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:22:56.494 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:22:56.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:22:56.495 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:22:56.596 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:22:56.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:22:56.634 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:22:56.634 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:22:56.736 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:22:56.775 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:22:56.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:22:56.776 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:22:56.877 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:22:56.901 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:22:56.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:22:56.902 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:22:57.003 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:22:57.044 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:22:56 smithi183 ceph-mon[29736]: pgmap v575: 158 pgs: 45 active+undersized, 33 active+undersized+degraded, 80 active+clean; 2.5 GiB data, 15 GiB used, 522 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.56k op/s; 20288/121359 objects degraded (16.717%) 2024-09-10T19:22:57.044 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:22:56 smithi183 ceph-mon[41562]: pgmap v575: 158 pgs: 45 active+undersized, 33 active+undersized+degraded, 80 active+clean; 2.5 GiB data, 15 GiB used, 522 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.56k op/s; 20288/121359 objects degraded (16.717%) 2024-09-10T19:22:57.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:22:57.047 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:22:57.048 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:22:57.070 INFO:teuthology.orchestra.run.smithi183.stdout:{"pg_ready":true,"pg_map":{"version":575,"stamp":"2024-09-10T19:22:55.827491+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":2638801562,"num_objects":40453,"num_object_clones":0,"num_object_copies":121359,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":20288,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40453,"num_whiteouts":0,"num_read":1551,"num_read_kb":1312,"num_write":253445,"num_write_kb":9375296,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":188,"num_bytes_recovered":57769,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":61846,"ondisk_log_size":61846,"up":396,"acting":396,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":474,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":562470912,"kb_used":15633036,"kb_used_data":10581272,"kb_used_omap":9,"kb_used_meta":5051638,"kb_avail":546837876,"statfs":{"total":575970213888,"available":559961985024,"internally_reserved":0,"allocated":10835222528,"data_stored":10832984594,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":10224,"internal_metadata":5172877328},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,3],"upper_bound":16},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-817233920,"num_objects":-12470,"num_object_clones":0,"num_object_copies":-37410,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-6253,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-12470,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":12470,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002323"},"pg_stats":[{"pgid":"6.1b","version":"0'0","reported_seq":263,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-10T19:22:49.257680+0000","last_change":"2024-09-10T19:21:50.786696+0000","last_active":"2024-09-10T19:22:49.257680+0000","last_peered":"2024-09-10T19:22:49.257680+0000","last_clean":"2024-09-10T19:22:49.257680+0000","last_became_active":"2024-09-10T19:10:14.056187+0000","last_became_peered":"2024-09-10T19:10:14.056187+0000","last_unstale":"2024-09-10T19:22:49.257680+0000","last_undegraded":"2024-09-10T19:22:49.257680+0000","last_fullsized":"2024-09-10T19:22:49.257680+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:21:50.786645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:21:50.786645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:23:12.329028+0000","scrub_duration":0.0043016390000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":161,"reported_epoch":278,"state":"active+undersized","last_fresh":"2024-09-10T19:22:49.302016+0000","last_change":"2024-09-10T19:21:47.983532+0000","last_active":"2024-09-10T19:22:49.302016+0000","last_peered":"2024-09-10T19:22:49.302016+0000","last_clean":"2024-09-10T19:16:23.155764+0000","last_became_active":"2024-09-10T19:21:47.983532+0000","last_became_peered":"2024-09-10T19:21:47.983532+0000","last_unstale":"2024-09-10T19:22:49.302016+0000","last_undegraded":"2024-09-10T19:22:49.302016+0000","last_fullsized":"2024-09-10T19:21:47.980212+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:20:05.748517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:16:38.194472+0000","last_clean_scrub_stamp":"2024-09-10T19:20:05.748517+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.023972094999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":["7","4"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.19","version":"237'27","reported_seq":333,"reported_epoch":278,"state":"active+undersized+degraded","last_fresh":"2024-09-10T19:22:49.257967+0000","last_change":"2024-09-10T19:21:47.993574+0000","last_active":"2024-09-10T19:22:49.257967+0000","last_peered":"2024-09-10T19:22:49.257967+0000","last_clean":"2024-09-10T19:21:43.944958+0000","last_became_active":"2024-09-10T19:21:47.993574+0000","last_became_peered":"2024-09-10T19:21:47.993574+0000","last_unstale":"2024-09-10T19:22:49.257967+0000","last_undegraded":"2024-09-10T19:21:47.976303+0000","last_fullsized":"2024-09-10T19:21:47.975840+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'15","last_scrub_stamp":"2024-09-10T19:19:21.244506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:19:21.244506+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.014405133000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":["3","5"],"object_location_counts":[{"shards":"3,5","objects":7}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":297,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-10T19:22:49.280283+0000","last_change":"2024-09-10T19:21:52.525548+0000","last_active":"2024-09-10T19:22:49.280283+0000","last_peered":"2024-09-10T19:22:49.280283+0000","last_clean":"2024-09-10T19:22:49.280283+0000","last_became_active":"2024-09-10T19:16:24.845111+0000","last_became_peered":"2024-09-10T19:16:24.845111+0000","last_unstale":"2024-09-10T19:22:49.280283+0000","last_undegraded":"2024-09-10T19:22:49.280283+0000","last_fullsized":"2024-09-10T19:22:49.280283+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:21:52.525468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:19:28.498560+0000","last_clean_scrub_stamp":"2024-09-10T19:21:52.525468+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:22:52.914174+0000","scrub_duration":0.0029578730000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,7],"acting":[4,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"59'1","reported_seq":284,"reported_epoch":278,"state":"active+undersized+degraded","last_fresh":"2024-09-10T19:22:49.258547+0000","last_change":"2024-09-10T19:21:47.993529+0000","last_active":"2024-09-10T19:22:49.258547+0000","last_peered":"2024-09-10T19:22:49.258547+0000","last_clean":"2024-09-10T19:21:43.944752+0000","last_became_active":"2024-09-10T19:21:47.993529+0000","last_became_peered":"2024-09-10T19:21:47.993529+0000","last_unstale":"2024-09-10T19:22:49.258547+0000","last_undegraded":"2024-09-10T19:21:47.974676+0000","last_fullsized":"2024-09-10T19:21:47.974527+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'1","last_scrub_stamp":"2024-09-10T19:20:56.540653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:20:56.540653+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.010998005,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":874,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":["3","5"],"object_location_counts":[{"shards":"3,5","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"237'15","reported_seq":321,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-10T19:22:49.280033+0000","last_change":"2024-09-10T19:21:48.575384+0000","last_active":"2024-09-10T19:22:49.280033+0000","last_peered":"2024-09-10T19:22:49.280033+0000","last_clean":"2024-09-10T19:22:49.280033+0000","last_became_active":"2024-09-10T19:13:54.716117+0000","last_became_peered":"2024-09-10T19:13:54.716117+0000","last_unstale":"2024-09-10T19:22:49.280033+0000","last_undegraded":"2024-09-10T19:22:49.280033+0000","last_fullsized":"2024-09-10T19:22:49.280033+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"237'15","last_scrub_stamp":"2024-09-10T19:21:48.575280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:21:48.575280+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:23:15.773458+0000","scrub_duration":0.0081726820000000006,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,3],"acting":[4,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":229,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-10T19:22:49.301968+0000","last_change":"2024-09-10T19:20:57.313136+0000","last_active":"2024-09-10T19:22:49.301968+0000","last_peered":"2024-09-10T19:22:49.301968+0000","last_clean":"2024-09-10T19:22:49.301968+0000","last_became_active":"2024-09-10T19:16:24.859261+0000","last_became_peered":"2024-09-10T19:16:24.859261+0000","last_unstale":"2024-09-10T19:22:49.301968+0000","last_undegraded":"2024-09-10T19:22:49.301968+0000","last_fullsized":"2024-09-10T19:22:49.301968+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:20:57.313056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:20:57.313056+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0079938800000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":285,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-10T19:22:49.280307+0000","last_change":"2024-09-10T19:21:53.485667+0000","last_active":"2024-09-10T19:22:49.280307+0000","last_peered":"2024-09-10T19:22:49.280307+0000","last_clean":"2024-09-10T19:22:49.280307+0000","last_became_active":"2024-09-10T19:13:54.713538+0000","last_became_peered":"2024-09-10T19:13:54.713538+0000","last_unstale":"2024-09-10T19:22:49.280307+0000","last_undegraded":"2024-09-10T19:22:49.280307+0000","last_fullsized":"2024-09-10T19:22:49.280307+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:21:53.485583+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:21:53.485583+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:23:19.504277+0000","scrub_duration":0.0033055229999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,7],"acting":[4,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":276,"reported_epoch":278,"state":"active+undersized","last_fresh":"2024-09-10T19:22:49.304093+0000","last_change":"2024-09-10T19:21:47.987922+0000","last_active":"2024-09-10T19:22:49.304093+0000","last_peered":"2024-09-10T19:22:49.304093+0000","last_clean":"2024-09-10T19:21:43.955497+0000","last_became_active":"2024-09-10T19:21:47.987922+0000","last_became_peered":"2024-09-10T19:21:47.987922+0000","last_unstale":"2024-09-10T19:22:49.304093+0000","last_undegraded":"2024-09-10T19:22:49.304093+0000","last_fullsized":"2024-09-10T19:21:47.983715+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:20:45.770926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:20:45.770926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0040372419999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":["7","6"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1b","version":"237'7","reported_seq":321,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-10T19:22:49.280541+0000","last_change":"2024-09-10T19:22:28.987340+0000","last_active":"2024-09-10T19:22:49.280541+0000","last_peered":"2024-09-10T19:22:49.280541+0000","last_clean":"2024-09-10T19:22:49.280541+0000","last_became_active":"2024-09-10T19:16:24.843708+0000","last_became_peered":"2024-09-10T19:16:24.843708+0000","last_unstale":"2024-09-10T19:22:49.280541+0000","last_undegraded":"2024-09-10T19:22:49.280541+0000","last_fullsized":"2024-09-10T19:22:49.280541+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"237'7","last_scrub_stamp":"2024-09-10T19:22:28.987257+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:22:28.987257+0000","objects_scrubbed":2,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:23:36.739968+0000","scrub_duration":0.0078490939999999992,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":10,"num_write":8,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":279,"reported_epoch":278,"state":"active+undersized","last_fresh":"2024-09-10T19:22:49.298448+0000","last_change":"2024-09-10T19:21:47.984833+0000","last_active":"2024-09-10T19:22:49.298448+0000","last_peered":"2024-09-10T19:22:49.298448+0000","last_clean":"2024-09-10T19:21:43.956152+0000","last_became_active":"2024-09-10T19:21:47.984833+0000","last_became_peered":"2024-09-10T19:21:47.984833+0000","last_unstale":"2024-09-10T19:22:49.298448+0000","last_undegraded":"2024-09-10T19:22:49.298448+0000","last_fullsized":"2024-09-10T19:21:47.979042+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:20:03.903546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:20:03.903546+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0066943719999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":["7","4"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":287,"reported_epoch":278,"state":"active+undersized","last_fresh":"2024-09-10T19:22:49.302053+0000","last_change":"2024-09-10T19:21:47.983411+0000","last_active":"2024-09-10T19:22:49.302053+0000","last_peered":"2024-09-10T19:22:49.302053+0000","last_clean":"2024-09-10T19:21:43.956084+0000","last_became_active":"2024-09-10T19:21:47.983411+0000","last_became_peered":"2024-09-10T19:21:47.983411+0000","last_unstale":"2024-09-10T19:22:49.302053+0000","last_undegraded":"2024-09-10T19:22:49.302053+0000","last_fullsized":"2024-09-10T19:21:47.973796+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:20:50.336432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:20:50.336432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0077486569999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":["5","3"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":289,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-10T19:22:49.299941+0000","last_change":"2024-09-10T19:22:38.275940+0000","last_active":"2024-09-10T19:22:49.299941+0000","last_peered":"2024-09-10T19:22:49.299941+0000","last_clean":"2024-09-10T19:22:49.299941+0000","last_became_active":"2024-09-10T19:16:24.858800+0000","last_became_peered":"2024-09-10T19:16:24.858800+0000","last_unstale":"2024-09-10T19:22:49.299941+0000","last_undegraded":"2024-09-10T19:22:49.299941+0000","last_fullsized":"2024-09-10T19:22:49.299941+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:22:38.275863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:22:38.275863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:23:44.801164+0000","scrub_duration":0.010525915,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1a","version":"237'15","reported_seq":301,"reported_epoch":278,"state":"active+undersized+degraded","last_fresh":"2024-09-10T19:22:49.298825+0000","last_change":"2024-09-10T19:21:47.978079+0000","last_active":"2024-09-10T19:22:49.298825+0000","last_peered":"2024-09-10T19:22:49.298825+0000","last_clean":"2024-09-10T19:21:43.946995+0000","last_became_active":"2024-09-10T19:21:47.978079+0000","last_became_peered":"2024-09-10T19:21:47.978079+0000","last_unstale":"2024-09-10T19:22:49.298825+0000","last_undegraded":"2024-09-10T19:21:47.971706+0000","last_fullsized":"2024-09-10T19:21:47.971602+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"237'15","last_scrub_stamp":"2024-09-10T19:21:42.567473+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:21:42.567473+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0065782230000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":["4","3"],"object_location_counts":[{"shards":"3,4","objects":4}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":272,"reported_epoch":278,"state":"active+undersized","last_fresh":"2024-09-10T19:22:49.303106+0000","last_change":"2024-09-10T19:21:47.997166+0000","last_active":"2024-09-10T19:22:49.303106+0000","last_peered":"2024-09-10T19:22:49.303106+0000","last_clean":"2024-09-10T19:21:43.951146+0000","last_became_active":"2024-09-10T19:21:47.997166+0000","last_became_peered":"2024-09-10T19:21:47.997166+0000","last_unstale":"2024-09-10T19:22:49.303106+0000","last_undegraded":"2024-09-10T19:22:49.303106+0000","last_fullsized":"2024-09-10T19:21:47.975077+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:21:00.322381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:21:00.322381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.008205258,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":["5","7"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":162,"reported_epoch":278,"state":"active+undersized","last_fresh":"2024-09-10T19:22:49.297677+0000","last_change":"2024-09-10T19:21:47.983632+0000","last_active":"2024-09-10T19:22:49.297677+0000","last_peered":"2024-09-10T19:22:49.297677+0000","last_clean":"2024-09-10T19:16:23.155082+0000","last_became_active":"2024-09-10T19:21:47.983632+0000","last_became_peered":"2024-09-10T19:21:47.983632+0000","last_unstale":"2024-09-10T19:22:49.297677+0000","last_undegraded":"2024-09-10T19:22:49.297677+0000","last_fullsized":"2024-09-10T19:21:47.978395+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:21:05.690833+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:21:05.690833+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0032019980000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":["7","4"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.8","version":"74'79","reported_seq":59,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-10T19:22:49.302721+0000","last_change":"2024-09-10T19:21:54.355335+0000","last_active":"2024-09-10T19:22:49.302721+0000","last_peered":"2024-09-10T19:22:49.302721+0000","last_clean":"2024-09-10T19:22:49.302721+0000","last_became_active":"2024-09-10T19:20:30.839050+0000","last_became_peered":"2024-09-10T19:20:30.839050+0000","last_unstale":"2024-09-10T19:22:49.302721+0000","last_undegraded":"2024-09-10T19:22:49.302721+0000","last_fullsized":"2024-09-10T19:22:49.302721+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":4,"last_scrub":"74'79","last_scrub_stamp":"2024-09-10T19:21:54.355151+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:08:00.699315+0000","last_clean_scrub_stamp":"2024-09-10T19:21:54.355151+0000","objects_scrubbed":1,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:23:07.013287+0000","scrub_duration":0.014582322999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.0","version":"279'15666","reported_seq":15736,"reported_epoch":279,"state":"active+undersized+degraded","last_fresh":"2024-09-10T19:22:54.898917+0000","last_change":"2024-09-10T19:21:48.042830+0000","last_active":"2024-09-10T19:22:54.898917+0000","last_peered":"2024-09-10T19:22:54.898917+0000","last_clean":"2024-09-10T19:21:46.973997+0000","last_became_active":"2024-09-10T19:21:48.010936+0000","last_became_peered":"2024-09-10T19:21:48.010936+0000","last_unstale":"2024-09-10T19:22:54.898917+0000","last_undegraded":"2024-09-10T19:21:47.994156+0000","last_fullsized":"2024-09-10T19:21:47.981767+0000","mapping_epoch":263,"log_start":"266'11800","ondisk_log_start":"266'11800","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:19:53.174811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:19:53.174811+0000","last_clean_scrub_stamp":"2024-09-10T19:19:53.174811+0000","objects_scrubbed":0,"log_size":3866,"log_dups_size":0,"ondisk_log_size":3866,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":171704320,"num_objects":2620,"num_object_clones":0,"num_object_copies":7860,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2620,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2620,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15666,"num_write_kb":585152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":["6","5"],"object_location_counts":[{"shards":"5,6","objects":2620}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"237'31","reported_seq":385,"reported_epoch":278,"state":"active+undersized+degraded","last_fresh":"2024-09-10T19:22:49.297622+0000","last_change":"2024-09-10T19:21:47.986499+0000","last_active":"2024-09-10T19:22:49.297622+0000","last_peered":"2024-09-10T19:22:49.297622+0000","last_clean":"2024-09-10T19:21:43.948925+0000","last_became_active":"2024-09-10T19:21:47.986499+0000","last_became_peered":"2024-09-10T19:21:47.986499+0000","last_unstale":"2024-09-10T19:22:49.297622+0000","last_undegraded":"2024-09-10T19:21:47.972336+0000","last_fullsized":"2024-09-10T19:21:47.972236+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"237'31","last_scrub_stamp":"2024-09-10T19:20:43.252273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:20:43.252273+0000","objects_scrubbed":8,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0023272610000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":8,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":50,"num_read_kb":40,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":["4","6"],"object_location_counts":[{"shards":"4,6","objects":8}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":279,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-10T19:22:49.297861+0000","last_change":"2024-09-10T19:22:24.285534+0000","last_active":"2024-09-10T19:22:49.297861+0000","last_peered":"2024-09-10T19:22:49.297861+0000","last_clean":"2024-09-10T19:22:49.297861+0000","last_became_active":"2024-09-10T19:10:07.691119+0000","last_became_peered":"2024-09-10T19:10:07.691119+0000","last_unstale":"2024-09-10T19:22:49.297861+0000","last_undegraded":"2024-09-10T19:22:49.297861+0000","last_fullsized":"2024-09-10T19:22:49.297861+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:22:24.285464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:16:39.716674+0000","last_clean_scrub_stamp":"2024-09-10T19:22:24.285464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:23:24.457504+0000","scrub_duration":0.029482613000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":229,"reported_epoch":278,"state":"active+undersized","last_fresh":"2024-09-10T19:22:49.299007+0000","last_change":"2024-09-10T19:21:48.003034+0000","last_active":"2024-09-10T19:22:49.299007+0000","last_peered":"2024-09-10T19:22:49.299007+0000","last_clean":"2024-09-10T19:21:43.949309+0000","last_became_active":"2024-09-10T19:21:48.003034+0000","last_became_peered":"2024-09-10T19:21:48.003034+0000","last_unstale":"2024-09-10T19:22:49.299007+0000","last_undegraded":"2024-09-10T19:22:49.299007+0000","last_fullsized":"2024-09-10T19:21:47.971412+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:20:54.418649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:20:54.418649+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.006664377,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":["4","6"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":147,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-10T19:22:49.257779+0000","last_change":"2024-09-10T19:20:45.455543+0000","last_active":"2024-09-10T19:22:49.257779+0000","last_peered":"2024-09-10T19:22:49.257779+0000","last_clean":"2024-09-10T19:22:49.257779+0000","last_became_active":"2024-09-10T19:16:24.857843+0000","last_became_peered":"2024-09-10T19:16:24.857843+0000","last_unstale":"2024-09-10T19:22:49.257779+0000","last_undegraded":"2024-09-10T19:22:49.257779+0000","last_fullsized":"2024-09-10T19:22:49.257779+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:20:45.455459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:20:45.455459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0055819499999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.9","version":"74'79","reported_seq":344,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-10T19:22:49.303972+0000","last_change":"2024-09-10T19:22:37.284558+0000","last_active":"2024-09-10T19:22:49.303972+0000","last_peered":"2024-09-10T19:22:49.303972+0000","last_clean":"2024-09-10T19:22:49.303972+0000","last_became_active":"2024-09-10T19:20:30.844108+0000","last_became_peered":"2024-09-10T19:20:30.844108+0000","last_unstale":"2024-09-10T19:22:49.303972+0000","last_undegraded":"2024-09-10T19:22:49.303972+0000","last_fullsized":"2024-09-10T19:22:49.303972+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":4,"last_scrub":"74'79","last_scrub_stamp":"2024-09-10T19:22:37.284479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:08:00.699315+0000","last_clean_scrub_stamp":"2024-09-10T19:22:37.284479+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:23:56.506578+0000","scrub_duration":0.014548726999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.1","version":"279'15642","reported_seq":15881,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-09-10T19:22:55.098871+0000","last_change":"2024-09-10T19:22:31.601422+0000","last_active":"2024-09-10T19:22:55.098871+0000","last_peered":"2024-09-10T19:22:55.098871+0000","last_clean":"2024-09-10T19:22:55.098871+0000","last_became_active":"2024-09-10T19:19:54.198678+0000","last_became_peered":"2024-09-10T19:19:54.198678+0000","last_unstale":"2024-09-10T19:22:55.098871+0000","last_undegraded":"2024-09-10T19:22:55.098871+0000","last_fullsized":"2024-09-10T19:22:55.098871+0000","mapping_epoch":229,"log_start":"266'12050","ondisk_log_start":"266'12050","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"272'13865","last_scrub_stamp":"2024-09-10T19:22:31.601360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:19:53.174811+0000","last_clean_scrub_stamp":"2024-09-10T19:22:31.601360+0000","objects_scrubbed":4304,"log_size":3592,"log_dups_size":0,"ondisk_log_size":3592,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:23:51.522073+0000","scrub_duration":1.2956811529999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":162004992,"num_objects":2472,"num_object_clones":0,"num_object_copies":7416,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2472,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15642,"num_write_kb":579648,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"237'20","reported_seq":309,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-10T19:22:49.299137+0000","last_change":"2024-09-10T19:21:44.614534+0000","last_active":"2024-09-10T19:22:49.299137+0000","last_peered":"2024-09-10T19:22:49.299137+0000","last_clean":"2024-09-10T19:22:49.299137+0000","last_became_active":"2024-09-10T19:10:09.688321+0000","last_became_peered":"2024-09-10T19:10:09.688321+0000","last_unstale":"2024-09-10T19:22:49.299137+0000","last_undegraded":"2024-09-10T19:22:49.299137+0000","last_fullsized":"2024-09-10T19:22:49.299137+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"237'20","last_scrub_stamp":"2024-09-10T19:21:44.614432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:21:44.614432+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:23:02.913848+0000","scrub_duration":0.007970876,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":219,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-10T19:22:49.258277+0000","last_change":"2024-09-10T19:19:57.419605+0000","last_active":"2024-09-10T19:22:49.258277+0000","last_peered":"2024-09-10T19:22:49.258277+0000","last_clean":"2024-09-10T19:22:49.258277+0000","last_became_active":"2024-09-10T19:13:54.731134+0000","last_became_peered":"2024-09-10T19:13:54.731134+0000","last_unstale":"2024-09-10T19:22:49.258277+0000","last_undegraded":"2024-09-10T19:22:49.258277+0000","last_fullsized":"2024-09-10T19:22:49.258277+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:19:57.419553+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:19:57.419553+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001279428,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,5],"acting":[3,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"264'11","reported_seq":303,"reported_epoch":278,"state":"active+undersized+degraded","last_fresh":"2024-09-10T19:22:50.897089+0000","last_change":"2024-09-10T19:21:47.999533+0000","last_active":"2024-09-10T19:22:50.897089+0000","last_peered":"2024-09-10T19:22:50.897089+0000","last_clean":"2024-09-10T19:21:45.875744+0000","last_became_active":"2024-09-10T19:21:47.999533+0000","last_became_peered":"2024-09-10T19:21:47.999533+0000","last_unstale":"2024-09-10T19:22:50.897089+0000","last_undegraded":"2024-09-10T19:21:47.982460+0000","last_fullsized":"2024-09-10T19:21:47.982350+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"155'10","last_scrub_stamp":"2024-09-10T19:20:42.761156+0000","last_deep_scrub":"80'9","last_deep_scrub_stamp":"2024-09-10T19:11:25.156465+0000","last_clean_scrub_stamp":"2024-09-10T19:20:42.761156+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.010904117,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":["7","5"],"object_location_counts":[{"shards":"5,7","objects":1}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":295,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-10T19:22:49.280786+0000","last_change":"2024-09-10T19:21:58.521967+0000","last_active":"2024-09-10T19:22:49.280786+0000","last_peered":"2024-09-10T19:22:49.280786+0000","last_clean":"2024-09-10T19:22:49.280786+0000","last_became_active":"2024-09-10T19:16:24.851829+0000","last_became_peered":"2024-09-10T19:16:24.851829+0000","last_unstale":"2024-09-10T19:22:49.280786+0000","last_undegraded":"2024-09-10T19:22:49.280786+0000","last_fullsized":"2024-09-10T19:22:49.280786+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:21:58.521891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:21:58.521891+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:23:27.912901+0000","scrub_duration":0.0050654960000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,7],"acting":[4,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.a","version":"74'79","reported_seq":54,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-10T19:22:49.301495+0000","last_change":"2024-09-10T19:21:53.326522+0000","last_active":"2024-09-10T19:22:49.301495+0000","last_peered":"2024-09-10T19:22:49.301495+0000","last_clean":"2024-09-10T19:22:49.301495+0000","last_became_active":"2024-09-10T19:20:30.844773+0000","last_became_peered":"2024-09-10T19:20:30.844773+0000","last_unstale":"2024-09-10T19:22:49.301495+0000","last_undegraded":"2024-09-10T19:22:49.301495+0000","last_fullsized":"2024-09-10T19:22:49.301495+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":4,"last_scrub":"74'79","last_scrub_stamp":"2024-09-10T19:21:53.326433+0000","last_deep_scrub":"74'79","last_deep_scrub_stamp":"2024-09-10T19:21:53.326433+0000","last_clean_scrub_stamp":"2024-09-10T19:21:53.326433+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:23:11.639802+0000","scrub_duration":0.0062202960000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,4],"acting":[5,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.2","version":"279'16032","reported_seq":16090,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-09-10T19:22:55.118808+0000","last_change":"2024-09-10T19:19:54.197588+0000","last_active":"2024-09-10T19:22:55.118808+0000","last_peered":"2024-09-10T19:22:55.118808+0000","last_clean":"2024-09-10T19:22:55.118808+0000","last_became_active":"2024-09-10T19:19:54.197290+0000","last_became_peered":"2024-09-10T19:19:54.197290+0000","last_unstale":"2024-09-10T19:22:55.118808+0000","last_undegraded":"2024-09-10T19:22:55.118808+0000","last_fullsized":"2024-09-10T19:22:55.118808+0000","mapping_epoch":229,"log_start":"266'12401","ondisk_log_start":"266'12401","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:19:53.174811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:19:53.174811+0000","last_clean_scrub_stamp":"2024-09-10T19:19:53.174811+0000","objects_scrubbed":0,"log_size":3631,"log_dups_size":0,"ondisk_log_size":3631,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":169082880,"num_objects":2580,"num_object_clones":0,"num_object_copies":7740,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2580,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16032,"num_write_kb":595584,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"237'27","reported_seq":290,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-10T19:22:49.258746+0000","last_change":"2024-09-10T19:21:09.451668+0000","last_active":"2024-09-10T19:22:49.258746+0000","last_peered":"2024-09-10T19:22:49.258746+0000","last_clean":"2024-09-10T19:22:49.258746+0000","last_became_active":"2024-09-10T19:13:54.711634+0000","last_became_peered":"2024-09-10T19:13:54.711634+0000","last_unstale":"2024-09-10T19:22:49.258746+0000","last_undegraded":"2024-09-10T19:22:49.258746+0000","last_fullsized":"2024-09-10T19:22:49.258746+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"237'27","last_scrub_stamp":"2024-09-10T19:21:09.451556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:21:09.451556+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0045196150000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":239,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-10T19:22:49.304063+0000","last_change":"2024-09-10T19:21:49.121081+0000","last_active":"2024-09-10T19:22:49.304063+0000","last_peered":"2024-09-10T19:22:49.304063+0000","last_clean":"2024-09-10T19:22:49.304063+0000","last_became_active":"2024-09-10T19:16:24.857179+0000","last_became_peered":"2024-09-10T19:16:24.857179+0000","last_unstale":"2024-09-10T19:22:49.304063+0000","last_undegraded":"2024-09-10T19:22:49.304063+0000","last_fullsized":"2024-09-10T19:22:49.304063+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:21:49.120987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:21:49.120987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:23:05.286334+0000","scrub_duration":0.0076513420000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,6],"acting":[7,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.e","version":"264'9","reported_seq":426,"reported_epoch":278,"state":"active+undersized+degraded","last_fresh":"2024-09-10T19:22:50.890904+0000","last_change":"2024-09-10T19:21:47.981249+0000","last_active":"2024-09-10T19:22:50.890904+0000","last_peered":"2024-09-10T19:22:50.890904+0000","last_clean":"2024-09-10T19:21:45.875962+0000","last_became_active":"2024-09-10T19:21:47.981249+0000","last_became_peered":"2024-09-10T19:21:47.981249+0000","last_unstale":"2024-09-10T19:22:50.890904+0000","last_undegraded":"2024-09-10T19:21:47.971014+0000","last_fullsized":"2024-09-10T19:21:47.970564+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'8","last_scrub_stamp":"2024-09-10T19:19:17.519517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:19:17.519517+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0049564470000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":["4","5"],"object_location_counts":[{"shards":"4,5","objects":1}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":287,"reported_epoch":278,"state":"active+undersized","last_fresh":"2024-09-10T19:22:49.303194+0000","last_change":"2024-09-10T19:21:47.994832+0000","last_active":"2024-09-10T19:22:49.303194+0000","last_peered":"2024-09-10T19:22:49.303194+0000","last_clean":"2024-09-10T19:21:43.953414+0000","last_became_active":"2024-09-10T19:21:47.994832+0000","last_became_peered":"2024-09-10T19:21:47.994832+0000","last_unstale":"2024-09-10T19:22:49.303194+0000","last_undegraded":"2024-09-10T19:22:49.303194+0000","last_fullsized":"2024-09-10T19:21:47.973238+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:20:49.344566+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:20:49.344566+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0057184669999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":["5","6"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.3","version":"279'15549","reported_seq":16048,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-09-10T19:22:55.219585+0000","last_change":"2024-09-10T19:22:21.598654+0000","last_active":"2024-09-10T19:22:55.219585+0000","last_peered":"2024-09-10T19:22:55.219585+0000","last_clean":"2024-09-10T19:22:55.219585+0000","last_became_active":"2024-09-10T19:19:54.203780+0000","last_became_peered":"2024-09-10T19:19:54.203780+0000","last_unstale":"2024-09-10T19:22:55.219585+0000","last_undegraded":"2024-09-10T19:22:55.219585+0000","last_fullsized":"2024-09-10T19:22:55.219585+0000","mapping_epoch":229,"log_start":"266'11905","ondisk_log_start":"266'11905","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"270'12966","last_scrub_stamp":"2024-09-10T19:22:21.598597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:19:53.174811+0000","last_clean_scrub_stamp":"2024-09-10T19:22:21.598597+0000","objects_scrubbed":5109,"log_size":3644,"log_dups_size":0,"ondisk_log_size":3644,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:23:43.809390+0000","scrub_duration":2.297330085,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":160235520,"num_objects":2445,"num_object_clones":0,"num_object_copies":7335,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2445,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15549,"num_write_kb":575808,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"237'24","reported_seq":334,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-10T19:22:49.301045+0000","last_change":"2024-09-10T19:21:59.504487+0000","last_active":"2024-09-10T19:22:49.301045+0000","last_peered":"2024-09-10T19:22:49.301045+0000","last_clean":"2024-09-10T19:22:49.301045+0000","last_became_active":"2024-09-10T19:13:54.717022+0000","last_became_peered":"2024-09-10T19:13:54.717022+0000","last_unstale":"2024-09-10T19:22:49.301045+0000","last_undegraded":"2024-09-10T19:22:49.301045+0000","last_fullsized":"2024-09-10T19:22:49.301045+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"237'24","last_scrub_stamp":"2024-09-10T19:21:59.504408+0000","last_deep_scrub":"84'16","last_deep_scrub_stamp":"2024-09-10T19:11:56.157975+0000","last_clean_scrub_stamp":"2024-09-10T19:21:59.504408+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:23:02.276380+0000","scrub_duration":0.0050089319999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,3],"acting":[4,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":283,"reported_epoch":278,"state":"active+undersized","last_fresh":"2024-09-10T19:22:49.298372+0000","last_change":"2024-09-10T19:21:47.983520+0000","last_active":"2024-09-10T19:22:49.298372+0000","last_peered":"2024-09-10T19:22:49.298372+0000","last_clean":"2024-09-10T19:21:43.954695+0000","last_became_active":"2024-09-10T19:21:47.983520+0000","last_became_peered":"2024-09-10T19:21:47.983520+0000","last_unstale":"2024-09-10T19:22:49.298372+0000","last_undegraded":"2024-09-10T19:22:49.298372+0000","last_fullsized":"2024-09-10T19:21:47.978120+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:20:06.874462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:20:06.874462+0000","last_clean_scrub_stamp":"2024-09-10T19:20:06.874462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0052621079999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":["7","4"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":265,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-10T19:22:49.343870+0000","last_change":"2024-09-10T19:22:19.713111+0000","last_active":"2024-09-10T19:22:49.343870+0000","last_peered":"2024-09-10T19:22:49.343870+0000","last_clean":"2024-09-10T19:22:49.343870+0000","last_became_active":"2024-09-10T19:10:11.692215+0000","last_became_peered":"2024-09-10T19:10:11.692215+0000","last_unstale":"2024-09-10T19:22:49.343870+0000","last_undegraded":"2024-09-10T19:22:49.343870+0000","last_fullsized":"2024-09-10T19:22:49.343870+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:22:19.713013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:22:19.713013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:23:21.216199+0000","scrub_duration":0.029020666000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":263,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-10T19:22:49.258957+0000","last_change":"2024-09-10T19:22:37.144678+0000","last_active":"2024-09-10T19:22:49.258957+0000","last_peered":"2024-09-10T19:22:49.258957+0000","last_clean":"2024-09-10T19:22:49.258957+0000","last_became_active":"2024-09-10T19:10:14.053736+0000","last_became_peered":"2024-09-10T19:10:14.053736+0000","last_unstale":"2024-09-10T19:22:49.258957+0000","last_undegraded":"2024-09-10T19:22:49.258957+0000","last_fullsized":"2024-09-10T19:22:49.258957+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:22:37.144606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:22:37.144606+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:24:02.963330+0000","scrub_duration":0.002180853,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.4","version":"74'79","reported_seq":59,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-10T19:22:49.257568+0000","last_change":"2024-09-10T19:22:20.452660+0000","last_active":"2024-09-10T19:22:49.257568+0000","last_peered":"2024-09-10T19:22:49.257568+0000","last_clean":"2024-09-10T19:22:49.257568+0000","last_became_active":"2024-09-10T19:20:30.838441+0000","last_became_peered":"2024-09-10T19:20:30.838441+0000","last_unstale":"2024-09-10T19:22:49.257568+0000","last_undegraded":"2024-09-10T19:22:49.257568+0000","last_fullsized":"2024-09-10T19:22:49.257568+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":3,"last_scrub":"74'79","last_scrub_stamp":"2024-09-10T19:22:20.452581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:08:00.699315+0000","last_clean_scrub_stamp":"2024-09-10T19:22:20.452581+0000","objects_scrubbed":1,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:23:43.055901+0000","scrub_duration":0.006813782,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.c","version":"279'15831","reported_seq":16052,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-09-10T19:22:55.101466+0000","last_change":"2024-09-10T19:22:38.980045+0000","last_active":"2024-09-10T19:22:55.101466+0000","last_peered":"2024-09-10T19:22:55.101466+0000","last_clean":"2024-09-10T19:22:55.101466+0000","last_became_active":"2024-09-10T19:19:54.200490+0000","last_became_peered":"2024-09-10T19:19:54.200490+0000","last_unstale":"2024-09-10T19:22:55.101466+0000","last_undegraded":"2024-09-10T19:22:55.101466+0000","last_fullsized":"2024-09-10T19:22:55.101466+0000","mapping_epoch":229,"log_start":"266'12255","ondisk_log_start":"266'12255","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"273'14620","last_scrub_stamp":"2024-09-10T19:22:38.979994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:19:53.174811+0000","last_clean_scrub_stamp":"2024-09-10T19:22:38.979994+0000","objects_scrubbed":3836,"log_size":3576,"log_dups_size":0,"ondisk_log_size":3576,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:23:41.164821+0000","scrub_duration":0.79555883599999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":170852352,"num_objects":2607,"num_object_clones":0,"num_object_copies":7821,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2607,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15831,"num_write_kb":590016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"265'35","reported_seq":201,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-10T19:22:49.257246+0000","last_change":"2024-09-10T19:21:55.533450+0000","last_active":"2024-09-10T19:22:49.257246+0000","last_peered":"2024-09-10T19:22:49.257246+0000","last_clean":"2024-09-10T19:22:49.257246+0000","last_became_active":"2024-09-10T19:16:24.852821+0000","last_became_peered":"2024-09-10T19:16:24.852821+0000","last_unstale":"2024-09-10T19:22:49.257246+0000","last_undegraded":"2024-09-10T19:22:49.257246+0000","last_fullsized":"2024-09-10T19:22:49.257246+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"265'35","last_scrub_stamp":"2024-09-10T19:21:55.533362+0000","last_deep_scrub":"265'35","last_deep_scrub_stamp":"2024-09-10T19:21:55.533362+0000","last_clean_scrub_stamp":"2024-09-10T19:21:55.533362+0000","objects_scrubbed":7,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:23:17.878002+0000","scrub_duration":0.021732418999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":42,"num_read_kb":35,"num_write":28,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,7],"acting":[4,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":289,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-10T19:22:49.543736+0000","last_change":"2024-09-10T19:21:55.547190+0000","last_active":"2024-09-10T19:22:49.543736+0000","last_peered":"2024-09-10T19:22:49.543736+0000","last_clean":"2024-09-10T19:22:49.543736+0000","last_became_active":"2024-09-10T19:16:24.855861+0000","last_became_peered":"2024-09-10T19:16:24.855861+0000","last_unstale":"2024-09-10T19:22:49.543736+0000","last_undegraded":"2024-09-10T19:22:49.543736+0000","last_fullsized":"2024-09-10T19:22:49.543736+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:21:55.547081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:21:55.547081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:23:22.599506+0000","scrub_duration":0.010304272,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.0","version":"264'11","reported_seq":446,"reported_epoch":278,"state":"active+undersized+degraded","last_fresh":"2024-09-10T19:22:50.887367+0000","last_change":"2024-09-10T19:21:47.975361+0000","last_active":"2024-09-10T19:22:50.887367+0000","last_peered":"2024-09-10T19:22:50.887367+0000","last_clean":"2024-09-10T19:21:45.875415+0000","last_became_active":"2024-09-10T19:21:47.975361+0000","last_became_peered":"2024-09-10T19:21:47.975361+0000","last_unstale":"2024-09-10T19:22:50.887367+0000","last_undegraded":"2024-09-10T19:21:47.972860+0000","last_fullsized":"2024-09-10T19:21:47.972722+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"117'10","last_scrub_stamp":"2024-09-10T19:20:13.462447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:20:13.462447+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0040309090000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":["3","4"],"object_location_counts":[{"shards":"3,4","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":270,"reported_epoch":278,"state":"active+undersized","last_fresh":"2024-09-10T19:22:49.298354+0000","last_change":"2024-09-10T19:21:48.011467+0000","last_active":"2024-09-10T19:22:49.298354+0000","last_peered":"2024-09-10T19:22:49.298354+0000","last_clean":"2024-09-10T19:21:43.955087+0000","last_became_active":"2024-09-10T19:21:48.011467+0000","last_became_peered":"2024-09-10T19:21:48.011467+0000","last_unstale":"2024-09-10T19:22:49.298354+0000","last_undegraded":"2024-09-10T19:22:49.298354+0000","last_fullsized":"2024-09-10T19:21:47.993946+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:19:10.029951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:11:41.073833+0000","last_clean_scrub_stamp":"2024-09-10T19:19:10.029951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.01007037,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":["7","6"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.5","version":"74'79","reported_seq":54,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-10T19:22:49.280497+0000","last_change":"2024-09-10T19:21:45.601945+0000","last_active":"2024-09-10T19:22:49.280497+0000","last_peered":"2024-09-10T19:22:49.280497+0000","last_clean":"2024-09-10T19:22:49.280497+0000","last_became_active":"2024-09-10T19:20:30.838556+0000","last_became_peered":"2024-09-10T19:20:30.838556+0000","last_unstale":"2024-09-10T19:22:49.280497+0000","last_undegraded":"2024-09-10T19:22:49.280497+0000","last_fullsized":"2024-09-10T19:22:49.280497+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":3,"last_scrub":"74'79","last_scrub_stamp":"2024-09-10T19:21:45.601859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:08:00.699315+0000","last_clean_scrub_stamp":"2024-09-10T19:21:45.601859+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:23:03.875761+0000","scrub_duration":0.013353458,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.d","version":"279'18046","reported_seq":18417,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-09-10T19:22:55.211700+0000","last_change":"2024-09-10T19:22:25.216700+0000","last_active":"2024-09-10T19:22:55.211700+0000","last_peered":"2024-09-10T19:22:55.211700+0000","last_clean":"2024-09-10T19:22:55.211700+0000","last_became_active":"2024-09-10T19:19:54.202133+0000","last_became_peered":"2024-09-10T19:19:54.202133+0000","last_unstale":"2024-09-10T19:22:55.211700+0000","last_undegraded":"2024-09-10T19:22:55.211700+0000","last_fullsized":"2024-09-10T19:22:55.211700+0000","mapping_epoch":229,"log_start":"266'14349","ondisk_log_start":"266'14349","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"270'15773","last_scrub_stamp":"2024-09-10T19:22:25.216491+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:19:53.174811+0000","last_clean_scrub_stamp":"2024-09-10T19:22:25.216491+0000","objects_scrubbed":4931,"log_size":3697,"log_dups_size":0,"ondisk_log_size":3697,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:23:47.031330+0000","scrub_duration":2.9560003789999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":165806104,"num_objects":2531,"num_object_clones":0,"num_object_copies":7593,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2531,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":15719,"num_write_kb":583937,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,4],"acting":[7,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.0","version":"237'30","reported_seq":314,"reported_epoch":278,"state":"active+undersized+degraded","last_fresh":"2024-09-10T19:22:49.258518+0000","last_change":"2024-09-10T19:21:47.983337+0000","last_active":"2024-09-10T19:22:49.258518+0000","last_peered":"2024-09-10T19:22:49.258518+0000","last_clean":"2024-09-10T19:21:43.943548+0000","last_became_active":"2024-09-10T19:21:47.983337+0000","last_became_peered":"2024-09-10T19:21:47.983337+0000","last_unstale":"2024-09-10T19:22:49.258518+0000","last_undegraded":"2024-09-10T19:21:47.975109+0000","last_fullsized":"2024-09-10T19:21:47.974995+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'18","last_scrub_stamp":"2024-09-10T19:19:12.207076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:19:12.207076+0000","objects_scrubbed":8,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.003631156,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":8,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":52,"num_read_kb":40,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":["3","7"],"object_location_counts":[{"shards":"3,7","objects":8}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":239,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-10T19:22:49.543902+0000","last_change":"2024-09-10T19:22:32.696939+0000","last_active":"2024-09-10T19:22:49.543902+0000","last_peered":"2024-09-10T19:22:49.543902+0000","last_clean":"2024-09-10T19:22:49.543902+0000","last_became_active":"2024-09-10T19:13:54.710968+0000","last_became_peered":"2024-09-10T19:13:54.710968+0000","last_unstale":"2024-09-10T19:22:49.543902+0000","last_undegraded":"2024-09-10T19:22:49.543902+0000","last_fullsized":"2024-09-10T19:22:49.543902+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:22:32.696724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:17:26.374376+0000","last_clean_scrub_stamp":"2024-09-10T19:22:32.696724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:23:42.673395+0000","scrub_duration":0.0098896620000000005,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,3],"acting":[6,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":274,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-10T19:22:49.279986+0000","last_change":"2024-09-10T19:22:27.985608+0000","last_active":"2024-09-10T19:22:49.279986+0000","last_peered":"2024-09-10T19:22:49.279986+0000","last_clean":"2024-09-10T19:22:49.279986+0000","last_became_active":"2024-09-10T19:10:11.698054+0000","last_became_peered":"2024-09-10T19:10:11.698054+0000","last_unstale":"2024-09-10T19:22:49.279986+0000","last_undegraded":"2024-09-10T19:22:49.279986+0000","last_fullsized":"2024-09-10T19:22:49.279986+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:22:27.985521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:16:30.991635+0000","last_clean_scrub_stamp":"2024-09-10T19:22:27.985521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:23:45.834053+0000","scrub_duration":0.016026845000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":282,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-10T19:22:49.280576+0000","last_change":"2024-09-10T19:22:23.061813+0000","last_active":"2024-09-10T19:22:49.280576+0000","last_peered":"2024-09-10T19:22:49.280576+0000","last_clean":"2024-09-10T19:22:49.280576+0000","last_became_active":"2024-09-10T19:16:24.848109+0000","last_became_peered":"2024-09-10T19:16:24.848109+0000","last_unstale":"2024-09-10T19:22:49.280576+0000","last_undegraded":"2024-09-10T19:22:49.280576+0000","last_fullsized":"2024-09-10T19:22:49.280576+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:22:23.061715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:22:23.061715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:23:33.978057+0000","scrub_duration":0.0039110009999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.6","version":"74'79","reported_seq":131,"reported_epoch":278,"state":"active+undersized","last_fresh":"2024-09-10T19:22:49.543805+0000","last_change":"2024-09-10T19:21:47.988079+0000","last_active":"2024-09-10T19:22:49.543805+0000","last_peered":"2024-09-10T19:22:49.543805+0000","last_clean":"2024-09-10T19:21:45.499702+0000","last_became_active":"2024-09-10T19:21:47.988079+0000","last_became_peered":"2024-09-10T19:21:47.988079+0000","last_unstale":"2024-09-10T19:22:49.543805+0000","last_undegraded":"2024-09-10T19:22:49.543805+0000","last_fullsized":"2024-09-10T19:21:47.982186+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":3,"last_scrub":"74'79","last_scrub_stamp":"2024-09-10T19:21:45.499422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:08:00.699315+0000","last_clean_scrub_stamp":"2024-09-10T19:21:45.499422+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:22:57.351711+0000","scrub_duration":0.0024069769999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":["6","3"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.e","version":"279'16190","reported_seq":16255,"reported_epoch":279,"state":"active+undersized+degraded","last_fresh":"2024-09-10T19:22:55.190156+0000","last_change":"2024-09-10T19:21:47.989949+0000","last_active":"2024-09-10T19:22:55.190156+0000","last_peered":"2024-09-10T19:22:55.190156+0000","last_clean":"2024-09-10T19:21:46.952990+0000","last_became_active":"2024-09-10T19:21:47.989949+0000","last_became_peered":"2024-09-10T19:21:47.989949+0000","last_unstale":"2024-09-10T19:22:55.190156+0000","last_undegraded":"2024-09-10T19:21:47.986734+0000","last_fullsized":"2024-09-10T19:21:47.986624+0000","mapping_epoch":263,"log_start":"266'12553","ondisk_log_start":"266'12553","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:19:53.174811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:19:53.174811+0000","last_clean_scrub_stamp":"2024-09-10T19:19:53.174811+0000","objects_scrubbed":0,"log_size":3637,"log_dups_size":0,"ondisk_log_size":3637,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":166854656,"num_objects":2546,"num_object_clones":0,"num_object_copies":7638,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2546,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2546,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16190,"num_write_kb":599552,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":["7","4"],"object_location_counts":[{"shards":"4,7","objects":2546}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"237'37","reported_seq":306,"reported_epoch":278,"state":"active+undersized+degraded","last_fresh":"2024-09-10T19:22:49.303225+0000","last_change":"2024-09-10T19:21:47.998404+0000","last_active":"2024-09-10T19:22:49.303225+0000","last_peered":"2024-09-10T19:22:49.303225+0000","last_clean":"2024-09-10T19:20:16.238914+0000","last_became_active":"2024-09-10T19:21:47.998404+0000","last_became_peered":"2024-09-10T19:21:47.998404+0000","last_unstale":"2024-09-10T19:22:49.303225+0000","last_undegraded":"2024-09-10T19:21:47.977022+0000","last_fullsized":"2024-09-10T19:21:47.976922+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"104'27","last_scrub_stamp":"2024-09-10T19:20:14.754045+0000","last_deep_scrub":"104'27","last_deep_scrub_stamp":"2024-09-10T19:13:02.225944+0000","last_clean_scrub_stamp":"2024-09-10T19:20:14.754045+0000","objects_scrubbed":8,"log_size":37,"log_dups_size":0,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0059406959999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":8,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":53,"num_read_kb":39,"num_write":32,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":["5","7"],"object_location_counts":[{"shards":"5,7","objects":8}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":276,"reported_epoch":278,"state":"active+undersized","last_fresh":"2024-09-10T19:22:49.304274+0000","last_change":"2024-09-10T19:21:47.984335+0000","last_active":"2024-09-10T19:22:49.304274+0000","last_peered":"2024-09-10T19:22:49.304274+0000","last_clean":"2024-09-10T19:21:43.955503+0000","last_became_active":"2024-09-10T19:21:47.984335+0000","last_became_peered":"2024-09-10T19:21:47.984335+0000","last_unstale":"2024-09-10T19:22:49.304274+0000","last_undegraded":"2024-09-10T19:22:49.304274+0000","last_fullsized":"2024-09-10T19:21:47.979308+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:20:43.781741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:16:37.779967+0000","last_clean_scrub_stamp":"2024-09-10T19:20:43.781741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.023161668,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":["7","4"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":275,"reported_epoch":278,"state":"active+undersized","last_fresh":"2024-09-10T19:22:49.543550+0000","last_change":"2024-09-10T19:21:48.002017+0000","last_active":"2024-09-10T19:22:49.543550+0000","last_peered":"2024-09-10T19:22:49.543550+0000","last_clean":"2024-09-10T19:21:43.951362+0000","last_became_active":"2024-09-10T19:21:48.002017+0000","last_became_peered":"2024-09-10T19:21:48.002017+0000","last_unstale":"2024-09-10T19:22:49.543550+0000","last_undegraded":"2024-09-10T19:22:49.543550+0000","last_fullsized":"2024-09-10T19:21:47.979197+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:20:05.474676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:20:05.474676+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0035800150000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":["6","5"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":239,"reported_epoch":278,"state":"active+undersized","last_fresh":"2024-09-10T19:22:49.285981+0000","last_change":"2024-09-10T19:21:47.999030+0000","last_active":"2024-09-10T19:22:49.285981+0000","last_peered":"2024-09-10T19:22:49.285981+0000","last_clean":"2024-09-10T19:21:43.953419+0000","last_became_active":"2024-09-10T19:21:47.999030+0000","last_became_peered":"2024-09-10T19:21:47.999030+0000","last_unstale":"2024-09-10T19:22:49.285981+0000","last_undegraded":"2024-09-10T19:22:49.285981+0000","last_fullsized":"2024-09-10T19:21:47.984292+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:20:06.496950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:20:06.496950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0033688099999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":["6","7"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.7","version":"74'79","reported_seq":134,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-10T19:22:49.285886+0000","last_change":"2024-09-10T19:22:34.717383+0000","last_active":"2024-09-10T19:22:49.285886+0000","last_peered":"2024-09-10T19:22:49.285886+0000","last_clean":"2024-09-10T19:22:49.285886+0000","last_became_active":"2024-09-10T19:20:30.842857+0000","last_became_peered":"2024-09-10T19:20:30.842857+0000","last_unstale":"2024-09-10T19:22:49.285886+0000","last_undegraded":"2024-09-10T19:22:49.285886+0000","last_fullsized":"2024-09-10T19:22:49.285886+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":3,"last_scrub":"74'79","last_scrub_stamp":"2024-09-10T19:22:34.717286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:08:00.699315+0000","last_clean_scrub_stamp":"2024-09-10T19:22:34.717286+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:23:46.388918+0000","scrub_duration":0.0060991719999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.f","version":"279'15698","reported_seq":15928,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-09-10T19:22:55.328890+0000","last_change":"2024-09-10T19:22:36.603275+0000","last_active":"2024-09-10T19:22:55.328890+0000","last_peered":"2024-09-10T19:22:55.328890+0000","last_clean":"2024-09-10T19:22:55.328890+0000","last_became_active":"2024-09-10T19:19:54.202323+0000","last_became_peered":"2024-09-10T19:19:54.202323+0000","last_unstale":"2024-09-10T19:22:55.328890+0000","last_undegraded":"2024-09-10T19:22:55.328890+0000","last_fullsized":"2024-09-10T19:22:55.328890+0000","mapping_epoch":229,"log_start":"266'11900","ondisk_log_start":"266'11900","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"273'14243","last_scrub_stamp":"2024-09-10T19:22:36.603192+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:19:53.174811+0000","last_clean_scrub_stamp":"2024-09-10T19:22:36.603192+0000","objects_scrubbed":4047,"log_size":3798,"log_dups_size":0,"ondisk_log_size":3798,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:23:56.410560+0000","scrub_duration":1.99645008,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":164364288,"num_objects":2508,"num_object_clones":0,"num_object_copies":7524,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2508,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15698,"num_write_kb":582592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"237'20","reported_seq":247,"reported_epoch":278,"state":"active+undersized+degraded","last_fresh":"2024-09-10T19:22:49.301946+0000","last_change":"2024-09-10T19:21:47.978225+0000","last_active":"2024-09-10T19:22:49.301946+0000","last_peered":"2024-09-10T19:22:49.301946+0000","last_clean":"2024-09-10T19:21:43.955550+0000","last_became_active":"2024-09-10T19:21:47.978225+0000","last_became_peered":"2024-09-10T19:21:47.978225+0000","last_unstale":"2024-09-10T19:22:49.301946+0000","last_undegraded":"2024-09-10T19:21:47.972413+0000","last_fullsized":"2024-09-10T19:21:47.972298+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'10","last_scrub_stamp":"2024-09-10T19:19:26.575769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:19:26.575769+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.022703509,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":24,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":["5","4"],"object_location_counts":[{"shards":"4,5","objects":5}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":111,"reported_epoch":278,"state":"active+undersized","last_fresh":"2024-09-10T19:22:49.299721+0000","last_change":"2024-09-10T19:21:47.983971+0000","last_active":"2024-09-10T19:22:49.299721+0000","last_peered":"2024-09-10T19:22:49.299721+0000","last_clean":"2024-09-10T19:16:23.155488+0000","last_became_active":"2024-09-10T19:21:47.983971+0000","last_became_peered":"2024-09-10T19:21:47.983971+0000","last_unstale":"2024-09-10T19:22:49.299721+0000","last_undegraded":"2024-09-10T19:22:49.299721+0000","last_fullsized":"2024-09-10T19:21:47.979000+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:20:52.748172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:20:52.748172+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0061607199999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":["7","3"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.3","version":"264'10","reported_seq":102,"reported_epoch":278,"state":"active+undersized+degraded","last_fresh":"2024-09-10T19:22:50.887418+0000","last_change":"2024-09-10T19:21:48.008059+0000","last_active":"2024-09-10T19:22:50.887418+0000","last_peered":"2024-09-10T19:22:50.887418+0000","last_clean":"2024-09-10T19:11:53.878374+0000","last_became_active":"2024-09-10T19:21:48.008059+0000","last_became_peered":"2024-09-10T19:21:48.008059+0000","last_unstale":"2024-09-10T19:22:50.887418+0000","last_undegraded":"2024-09-10T19:21:47.990007+0000","last_fullsized":"2024-09-10T19:21:47.989908+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'9","last_scrub_stamp":"2024-09-10T19:21:04.724652+0000","last_deep_scrub":"84'9","last_deep_scrub_stamp":"2024-09-10T19:11:53.879272+0000","last_clean_scrub_stamp":"2024-09-10T19:21:04.724652+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":["6","5"],"object_location_counts":[{"shards":"5,6","objects":1}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":145,"reported_epoch":278,"state":"active+undersized","last_fresh":"2024-09-10T19:22:49.257324+0000","last_change":"2024-09-10T19:21:47.995786+0000","last_active":"2024-09-10T19:22:49.257324+0000","last_peered":"2024-09-10T19:22:49.257324+0000","last_clean":"2024-09-10T19:16:23.154999+0000","last_became_active":"2024-09-10T19:21:47.995786+0000","last_became_peered":"2024-09-10T19:21:47.995786+0000","last_unstale":"2024-09-10T19:22:49.257324+0000","last_undegraded":"2024-09-10T19:22:49.257324+0000","last_fullsized":"2024-09-10T19:21:47.973262+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:21:03.766127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:21:03.766127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0086405590000000008,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":["3","5"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"74'79","reported_seq":51,"reported_epoch":278,"state":"active+undersized+degraded","last_fresh":"2024-09-10T19:22:49.301490+0000","last_change":"2024-09-10T19:21:47.981812+0000","last_active":"2024-09-10T19:22:49.301490+0000","last_peered":"2024-09-10T19:22:49.301490+0000","last_clean":"2024-09-10T19:21:43.955636+0000","last_became_active":"2024-09-10T19:21:47.981812+0000","last_became_peered":"2024-09-10T19:21:47.981812+0000","last_unstale":"2024-09-10T19:22:49.301490+0000","last_undegraded":"2024-09-10T19:21:47.972771+0000","last_fullsized":"2024-09-10T19:21:47.972653+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":4,"last_scrub":"74'79","last_scrub_stamp":"2024-09-10T19:19:11.070835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:08:00.699315+0000","last_clean_scrub_stamp":"2024-09-10T19:19:11.070835+0000","objects_scrubbed":2,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0030997160000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":11,"num_read_kb":20,"num_write":11,"num_write_kb":148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":53670,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":["5","4"],"object_location_counts":[{"shards":"4,5","objects":1}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.9","version":"279'15677","reported_seq":15924,"reported_epoch":279,"state":"active+undersized+degraded","last_fresh":"2024-09-10T19:22:55.205241+0000","last_change":"2024-09-10T19:21:47.996322+0000","last_active":"2024-09-10T19:22:55.205241+0000","last_peered":"2024-09-10T19:22:55.205241+0000","last_clean":"2024-09-10T19:21:46.936170+0000","last_became_active":"2024-09-10T19:21:47.996322+0000","last_became_peered":"2024-09-10T19:21:47.996322+0000","last_unstale":"2024-09-10T19:22:55.205241+0000","last_undegraded":"2024-09-10T19:21:47.979809+0000","last_fullsized":"2024-09-10T19:21:47.979667+0000","mapping_epoch":263,"log_start":"266'12021","ondisk_log_start":"266'12021","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:19:53.174811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:19:53.174811+0000","last_clean_scrub_stamp":"2024-09-10T19:19:53.174811+0000","objects_scrubbed":4449,"log_size":3656,"log_dups_size":0,"ondisk_log_size":3656,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":160759808,"num_objects":2453,"num_object_clones":0,"num_object_copies":7359,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2453,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2453,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15677,"num_write_kb":580160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":["7","5"],"object_location_counts":[{"shards":"5,7","objects":2453}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.4","version":"237'43","reported_seq":326,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-10T19:22:50.887345+0000","last_change":"2024-09-10T19:20:47.469873+0000","last_active":"2024-09-10T19:22:50.887345+0000","last_peered":"2024-09-10T19:22:50.887345+0000","last_clean":"2024-09-10T19:22:50.887345+0000","last_became_active":"2024-09-10T19:16:24.858154+0000","last_became_peered":"2024-09-10T19:16:24.858154+0000","last_unstale":"2024-09-10T19:22:50.887345+0000","last_undegraded":"2024-09-10T19:22:50.887345+0000","last_fullsized":"2024-09-10T19:22:50.887345+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"237'43","last_scrub_stamp":"2024-09-10T19:20:47.469774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:20:47.469774+0000","objects_scrubbed":10,"log_size":43,"log_dups_size":0,"ondisk_log_size":43,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.002574663,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":65,"num_read_kb":50,"num_write":38,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":358,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":283,"reported_epoch":278,"state":"active+undersized","last_fresh":"2024-09-10T19:22:49.301576+0000","last_change":"2024-09-10T19:21:47.994362+0000","last_active":"2024-09-10T19:22:49.301576+0000","last_peered":"2024-09-10T19:22:49.301576+0000","last_clean":"2024-09-10T19:21:43.955302+0000","last_became_active":"2024-09-10T19:21:47.994362+0000","last_became_peered":"2024-09-10T19:21:47.994362+0000","last_unstale":"2024-09-10T19:22:49.301576+0000","last_undegraded":"2024-09-10T19:22:49.301576+0000","last_fullsized":"2024-09-10T19:21:47.972726+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:20:44.342243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:20:44.342243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0078427040000000007,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":["5","6"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":105,"reported_epoch":278,"state":"active+undersized","last_fresh":"2024-09-10T19:22:49.297327+0000","last_change":"2024-09-10T19:21:47.987054+0000","last_active":"2024-09-10T19:22:49.297327+0000","last_peered":"2024-09-10T19:22:49.297327+0000","last_clean":"2024-09-10T19:12:44.681644+0000","last_became_active":"2024-09-10T19:21:47.987054+0000","last_became_peered":"2024-09-10T19:21:47.987054+0000","last_unstale":"2024-09-10T19:22:49.297327+0000","last_undegraded":"2024-09-10T19:22:49.297327+0000","last_fullsized":"2024-09-10T19:21:47.971744+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:20:56.648680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:20:56.648680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001460161,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":["4","6"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":253,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-10T19:22:49.258231+0000","last_change":"2024-09-10T19:21:42.640298+0000","last_active":"2024-09-10T19:22:49.258231+0000","last_peered":"2024-09-10T19:22:49.258231+0000","last_clean":"2024-09-10T19:22:49.258231+0000","last_became_active":"2024-09-10T19:10:13.721744+0000","last_became_peered":"2024-09-10T19:10:13.721744+0000","last_unstale":"2024-09-10T19:22:49.258231+0000","last_undegraded":"2024-09-10T19:22:49.258231+0000","last_fullsized":"2024-09-10T19:22:49.258231+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:21:42.640236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:21:42.640236+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:23:10.323450+0000","scrub_duration":0.0027480289999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"74'79","reported_seq":341,"reported_epoch":278,"state":"active+undersized","last_fresh":"2024-09-10T19:22:49.298626+0000","last_change":"2024-09-10T19:21:47.984511+0000","last_active":"2024-09-10T19:22:49.298626+0000","last_peered":"2024-09-10T19:22:49.298626+0000","last_clean":"2024-09-10T19:21:43.956297+0000","last_became_active":"2024-09-10T19:21:47.984511+0000","last_became_peered":"2024-09-10T19:21:47.984511+0000","last_unstale":"2024-09-10T19:22:49.298626+0000","last_undegraded":"2024-09-10T19:22:49.298626+0000","last_fullsized":"2024-09-10T19:21:47.978708+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":4,"last_scrub":"74'79","last_scrub_stamp":"2024-09-10T19:21:04.941354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:08:00.699315+0000","last_clean_scrub_stamp":"2024-09-10T19:21:04.941354+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0040812920000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":["7","4"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.a","version":"279'15562","reported_seq":15819,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-09-10T19:22:55.318048+0000","last_change":"2024-09-10T19:22:26.919130+0000","last_active":"2024-09-10T19:22:55.318048+0000","last_peered":"2024-09-10T19:22:55.318048+0000","last_clean":"2024-09-10T19:22:55.318048+0000","last_became_active":"2024-09-10T19:19:54.202444+0000","last_became_peered":"2024-09-10T19:19:54.202444+0000","last_unstale":"2024-09-10T19:22:55.318048+0000","last_undegraded":"2024-09-10T19:22:55.318048+0000","last_fullsized":"2024-09-10T19:22:55.318048+0000","mapping_epoch":229,"log_start":"266'11800","ondisk_log_start":"266'11800","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"271'13343","last_scrub_stamp":"2024-09-10T19:22:26.919040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:19:53.174811+0000","last_clean_scrub_stamp":"2024-09-10T19:22:26.919040+0000","objects_scrubbed":4796,"log_size":3762,"log_dups_size":0,"ondisk_log_size":3762,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:23:37.930761+0000","scrub_duration":2.3267707670000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":162398208,"num_objects":2478,"num_object_clones":0,"num_object_copies":7434,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2478,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15562,"num_write_kb":577280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.7","version":"237'22","reported_seq":305,"reported_epoch":278,"state":"active+undersized+degraded","last_fresh":"2024-09-10T19:22:49.344090+0000","last_change":"2024-09-10T19:21:47.993567+0000","last_active":"2024-09-10T19:22:49.344090+0000","last_peered":"2024-09-10T19:22:49.344090+0000","last_clean":"2024-09-10T19:21:43.951893+0000","last_became_active":"2024-09-10T19:21:47.993567+0000","last_became_peered":"2024-09-10T19:21:47.993567+0000","last_unstale":"2024-09-10T19:22:49.344090+0000","last_undegraded":"2024-09-10T19:21:47.978296+0000","last_fullsized":"2024-09-10T19:21:47.978196+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"237'22","last_scrub_stamp":"2024-09-10T19:20:51.327668+0000","last_deep_scrub":"82'18","last_deep_scrub_stamp":"2024-09-10T19:15:55.777361+0000","last_clean_scrub_stamp":"2024-09-10T19:20:51.327668+0000","objects_scrubbed":5,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0057025269999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":36,"num_read_kb":25,"num_write":20,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":["5","7"],"object_location_counts":[{"shards":"5,7","objects":5}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.0","version":"159'6","reported_seq":586,"reported_epoch":279,"state":"active+undersized+degraded","last_fresh":"2024-09-10T19:22:53.147630+0000","last_change":"2024-09-10T19:21:48.003078+0000","last_active":"2024-09-10T19:22:53.147630+0000","last_peered":"2024-09-10T19:22:53.147630+0000","last_clean":"2024-09-10T19:21:46.502353+0000","last_became_active":"2024-09-10T19:21:48.003078+0000","last_became_peered":"2024-09-10T19:21:48.003078+0000","last_unstale":"2024-09-10T19:22:53.147630+0000","last_undegraded":"2024-09-10T19:21:47.981464+0000","last_fullsized":"2024-09-10T19:21:47.981365+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"159'6","last_scrub_stamp":"2024-09-10T19:20:13.471084+0000","last_deep_scrub":"159'6","last_deep_scrub_stamp":"2024-09-10T19:16:32.649914+0000","last_clean_scrub_stamp":"2024-09-10T19:20:13.471084+0000","objects_scrubbed":1,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0080331139999999992,"objects_trimmed":0,"snaptrim_duration":0.00034454499999999997,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":393,"num_read_kb":388,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":389,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":["6","5"],"object_location_counts":[{"shards":"5,6","objects":1}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":164,"reported_epoch":278,"state":"active+undersized","last_fresh":"2024-09-10T19:22:49.258105+0000","last_change":"2024-09-10T19:21:47.988021+0000","last_active":"2024-09-10T19:22:49.258105+0000","last_peered":"2024-09-10T19:22:49.258105+0000","last_clean":"2024-09-10T19:21:43.944933+0000","last_became_active":"2024-09-10T19:21:47.988021+0000","last_became_peered":"2024-09-10T19:21:47.988021+0000","last_unstale":"2024-09-10T19:22:49.258105+0000","last_undegraded":"2024-09-10T19:22:49.258105+0000","last_fullsized":"2024-09-10T19:21:47.976328+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:21:01.464138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:21:01.464138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0040256789999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":["3","6"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":157,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-10T19:22:49.304640+0000","last_change":"2024-09-10T19:21:04.357157+0000","last_active":"2024-09-10T19:22:49.304640+0000","last_peered":"2024-09-10T19:22:49.304640+0000","last_clean":"2024-09-10T19:22:49.304640+0000","last_became_active":"2024-09-10T19:16:24.861481+0000","last_became_peered":"2024-09-10T19:16:24.861481+0000","last_unstale":"2024-09-10T19:22:49.304640+0000","last_undegraded":"2024-09-10T19:22:49.304640+0000","last_fullsized":"2024-09-10T19:22:49.304640+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:21:04.357073+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:16:43.958224+0000","last_clean_scrub_stamp":"2024-09-10T19:21:04.357073+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0091960070000000008,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,4],"acting":[5,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":266,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-10T19:22:49.298652+0000","last_change":"2024-09-10T19:22:01.208343+0000","last_active":"2024-09-10T19:22:49.298652+0000","last_peered":"2024-09-10T19:22:49.298652+0000","last_clean":"2024-09-10T19:22:49.298652+0000","last_became_active":"2024-09-10T19:10:14.053697+0000","last_became_peered":"2024-09-10T19:10:14.053697+0000","last_unstale":"2024-09-10T19:22:49.298652+0000","last_undegraded":"2024-09-10T19:22:49.298652+0000","last_fullsized":"2024-09-10T19:22:49.298652+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:22:01.208272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:22:01.208272+0000","last_clean_scrub_stamp":"2024-09-10T19:22:01.208272+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:23:06.439276+0000","scrub_duration":0.0042303380000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.3","version":"74'79","reported_seq":124,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-10T19:22:49.543502+0000","last_change":"2024-09-10T19:21:47.511103+0000","last_active":"2024-09-10T19:22:49.543502+0000","last_peered":"2024-09-10T19:22:49.543502+0000","last_clean":"2024-09-10T19:22:49.543502+0000","last_became_active":"2024-09-10T19:20:30.847837+0000","last_became_peered":"2024-09-10T19:20:30.847837+0000","last_unstale":"2024-09-10T19:22:49.543502+0000","last_undegraded":"2024-09-10T19:22:49.543502+0000","last_fullsized":"2024-09-10T19:22:49.543502+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":3,"last_scrub":"74'79","last_scrub_stamp":"2024-09-10T19:21:47.511010+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:08:00.699315+0000","last_clean_scrub_stamp":"2024-09-10T19:21:47.511010+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:22:59.355728+0000","scrub_duration":0.0012705069999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,5],"acting":[6,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.b","version":"279'16081","reported_seq":16143,"reported_epoch":279,"state":"active+undersized+degraded","last_fresh":"2024-09-10T19:22:54.872523+0000","last_change":"2024-09-10T19:21:48.003266+0000","last_active":"2024-09-10T19:22:54.872523+0000","last_peered":"2024-09-10T19:22:54.872523+0000","last_clean":"2024-09-10T19:21:46.948308+0000","last_became_active":"2024-09-10T19:21:48.003266+0000","last_became_peered":"2024-09-10T19:21:48.003266+0000","last_unstale":"2024-09-10T19:22:54.872523+0000","last_undegraded":"2024-09-10T19:21:47.987789+0000","last_fullsized":"2024-09-10T19:21:47.985276+0000","mapping_epoch":263,"log_start":"266'12200","ondisk_log_start":"266'12200","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:19:53.174811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:19:53.174811+0000","last_clean_scrub_stamp":"2024-09-10T19:19:53.174811+0000","objects_scrubbed":0,"log_size":3881,"log_dups_size":0,"ondisk_log_size":3881,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":167182336,"num_objects":2551,"num_object_clones":0,"num_object_copies":7653,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2551,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2551,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16081,"num_write_kb":596224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":["6","5"],"object_location_counts":[{"shards":"5,6","objects":2551}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.6","version":"237'30","reported_seq":340,"reported_epoch":278,"state":"active+undersized+degraded","last_fresh":"2024-09-10T19:22:49.297671+0000","last_change":"2024-09-10T19:21:48.008902+0000","last_active":"2024-09-10T19:22:49.297671+0000","last_peered":"2024-09-10T19:22:49.297671+0000","last_clean":"2024-09-10T19:20:16.062994+0000","last_became_active":"2024-09-10T19:21:48.008902+0000","last_became_peered":"2024-09-10T19:21:48.008902+0000","last_unstale":"2024-09-10T19:22:49.297671+0000","last_undegraded":"2024-09-10T19:21:47.986289+0000","last_fullsized":"2024-09-10T19:21:47.986187+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"237'30","last_scrub_stamp":"2024-09-10T19:21:07.733508+0000","last_deep_scrub":"221'18","last_deep_scrub_stamp":"2024-09-10T19:19:29.606748+0000","last_clean_scrub_stamp":"2024-09-10T19:21:07.733508+0000","objects_scrubbed":6,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.020236845,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":["7","5"],"object_location_counts":[{"shards":"5,7","objects":6}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"61'1","reported_seq":35,"reported_epoch":278,"state":"active+undersized+degraded","last_fresh":"2024-09-10T19:22:49.280161+0000","last_change":"2024-09-10T19:21:47.976896+0000","last_active":"2024-09-10T19:22:49.280161+0000","last_peered":"2024-09-10T19:22:49.280161+0000","last_clean":"2024-09-10T19:10:09.679706+0000","last_became_active":"2024-09-10T19:21:47.976896+0000","last_became_peered":"2024-09-10T19:21:47.976896+0000","last_unstale":"2024-09-10T19:22:49.280161+0000","last_undegraded":"2024-09-10T19:21:47.970593+0000","last_fullsized":"2024-09-10T19:21:47.970495+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'1","last_scrub_stamp":"2024-09-10T19:20:40.716730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:20:40.716730+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":["4","3"],"object_location_counts":[{"shards":"3,4","objects":1}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":283,"reported_epoch":278,"state":"active+undersized","last_fresh":"2024-09-10T19:22:49.297809+0000","last_change":"2024-09-10T19:21:47.980682+0000","last_active":"2024-09-10T19:22:49.297809+0000","last_peered":"2024-09-10T19:22:49.297809+0000","last_clean":"2024-09-10T19:21:43.954775+0000","last_became_active":"2024-09-10T19:21:47.980682+0000","last_became_peered":"2024-09-10T19:21:47.980682+0000","last_unstale":"2024-09-10T19:22:49.297809+0000","last_undegraded":"2024-09-10T19:22:49.297809+0000","last_fullsized":"2024-09-10T19:21:47.977818+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:20:09.850369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:20:09.850369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0036440280000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":["7","3"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":289,"reported_epoch":278,"state":"active+undersized","last_fresh":"2024-09-10T19:22:49.343956+0000","last_change":"2024-09-10T19:21:47.980962+0000","last_active":"2024-09-10T19:22:49.343956+0000","last_peered":"2024-09-10T19:22:49.343956+0000","last_clean":"2024-09-10T19:21:43.951818+0000","last_became_active":"2024-09-10T19:21:47.980962+0000","last_became_peered":"2024-09-10T19:21:47.980962+0000","last_unstale":"2024-09-10T19:22:49.343956+0000","last_undegraded":"2024-09-10T19:22:49.343956+0000","last_fullsized":"2024-09-10T19:21:47.972363+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:21:08.210604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:21:08.210604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0056788289999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":["5","4"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":219,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-10T19:22:49.302235+0000","last_change":"2024-09-10T19:20:58.275362+0000","last_active":"2024-09-10T19:22:49.302235+0000","last_peered":"2024-09-10T19:22:49.302235+0000","last_clean":"2024-09-10T19:22:49.302235+0000","last_became_active":"2024-09-10T19:13:54.722937+0000","last_became_peered":"2024-09-10T19:13:54.722937+0000","last_unstale":"2024-09-10T19:22:49.302235+0000","last_undegraded":"2024-09-10T19:22:49.302235+0000","last_fullsized":"2024-09-10T19:22:49.302235+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:20:58.275280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:20:58.275280+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.005497493,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.8","version":"279'15887","reported_seq":15950,"reported_epoch":279,"state":"active+undersized+degraded","last_fresh":"2024-09-10T19:22:55.097256+0000","last_change":"2024-09-10T19:21:47.996974+0000","last_active":"2024-09-10T19:22:55.097256+0000","last_peered":"2024-09-10T19:22:55.097256+0000","last_clean":"2024-09-10T19:21:46.945944+0000","last_became_active":"2024-09-10T19:21:47.996974+0000","last_became_peered":"2024-09-10T19:21:47.996974+0000","last_unstale":"2024-09-10T19:22:55.097256+0000","last_undegraded":"2024-09-10T19:21:47.973377+0000","last_fullsized":"2024-09-10T19:21:47.973279+0000","mapping_epoch":263,"log_start":"266'12250","ondisk_log_start":"266'12250","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:19:53.174811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:19:53.174811+0000","last_clean_scrub_stamp":"2024-09-10T19:19:53.174811+0000","objects_scrubbed":0,"log_size":3637,"log_dups_size":0,"ondisk_log_size":3637,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":165085184,"num_objects":2519,"num_object_clones":0,"num_object_copies":7557,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2519,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2519,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15887,"num_write_kb":588992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":["3","5"],"object_location_counts":[{"shards":"3,5","objects":2519}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"237'33","reported_seq":334,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-10T19:22:49.285952+0000","last_change":"2024-09-10T19:22:24.646928+0000","last_active":"2024-09-10T19:22:49.285952+0000","last_peered":"2024-09-10T19:22:49.285952+0000","last_clean":"2024-09-10T19:22:49.285952+0000","last_became_active":"2024-09-10T19:10:09.822184+0000","last_became_peered":"2024-09-10T19:10:09.822184+0000","last_unstale":"2024-09-10T19:22:49.285952+0000","last_undegraded":"2024-09-10T19:22:49.285952+0000","last_fullsized":"2024-09-10T19:22:49.285952+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"237'33","last_scrub_stamp":"2024-09-10T19:22:24.646814+0000","last_deep_scrub":"80'21","last_deep_scrub_stamp":"2024-09-10T19:11:39.836144+0000","last_clean_scrub_stamp":"2024-09-10T19:22:24.646814+0000","objects_scrubbed":7,"log_size":33,"log_dups_size":0,"ondisk_log_size":33,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:23:25.255304+0000","scrub_duration":0.0087126990000000008,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":33,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"63'2","reported_seq":276,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-10T19:22:49.258354+0000","last_change":"2024-09-10T19:21:46.741106+0000","last_active":"2024-09-10T19:22:49.258354+0000","last_peered":"2024-09-10T19:22:49.258354+0000","last_clean":"2024-09-10T19:22:49.258354+0000","last_became_active":"2024-09-10T19:10:07.684784+0000","last_became_peered":"2024-09-10T19:10:07.684784+0000","last_unstale":"2024-09-10T19:22:49.258354+0000","last_undegraded":"2024-09-10T19:22:49.258354+0000","last_fullsized":"2024-09-10T19:22:49.258354+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'2","last_scrub_stamp":"2024-09-10T19:21:46.741014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:21:46.741014+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:23:07.668837+0000","scrub_duration":0.0060747680000000004,"objects_trimmed":0,"snaptrim_duration":0.00075522999999999996,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":289,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-10T19:22:49.303539+0000","last_change":"2024-09-10T19:22:33.611730+0000","last_active":"2024-09-10T19:22:49.303539+0000","last_peered":"2024-09-10T19:22:49.303539+0000","last_clean":"2024-09-10T19:22:49.303539+0000","last_became_active":"2024-09-10T19:16:24.858662+0000","last_became_peered":"2024-09-10T19:16:24.858662+0000","last_unstale":"2024-09-10T19:22:49.303539+0000","last_undegraded":"2024-09-10T19:22:49.303539+0000","last_fullsized":"2024-09-10T19:22:49.303539+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:22:33.611658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:22:33.611658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:23:35.533500+0000","scrub_duration":0.012996436,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"74'79","reported_seq":332,"reported_epoch":278,"state":"active+undersized","last_fresh":"2024-09-10T19:22:49.298014+0000","last_change":"2024-09-10T19:21:47.999059+0000","last_active":"2024-09-10T19:22:49.298014+0000","last_peered":"2024-09-10T19:22:49.298014+0000","last_clean":"2024-09-10T19:21:43.954921+0000","last_became_active":"2024-09-10T19:21:47.999059+0000","last_became_peered":"2024-09-10T19:21:47.999059+0000","last_unstale":"2024-09-10T19:22:49.298014+0000","last_undegraded":"2024-09-10T19:22:49.298014+0000","last_fullsized":"2024-09-10T19:21:47.980322+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'79","last_scrub_stamp":"2024-09-10T19:20:59.897158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:08:00.699315+0000","last_clean_scrub_stamp":"2024-09-10T19:20:59.897158+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0081933070000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":["7","6"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":282,"reported_epoch":278,"state":"active+undersized","last_fresh":"2024-09-10T19:22:49.299511+0000","last_change":"2024-09-10T19:21:47.996658+0000","last_active":"2024-09-10T19:22:49.299511+0000","last_peered":"2024-09-10T19:22:49.299511+0000","last_clean":"2024-09-10T19:21:43.955922+0000","last_became_active":"2024-09-10T19:21:47.996658+0000","last_became_peered":"2024-09-10T19:21:47.996658+0000","last_unstale":"2024-09-10T19:22:49.299511+0000","last_undegraded":"2024-09-10T19:22:49.299511+0000","last_fullsized":"2024-09-10T19:21:47.979415+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:20:54.870558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:20:54.870558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0080190729999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":["7","5"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":263,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-10T19:22:49.344258+0000","last_change":"2024-09-10T19:21:47.253156+0000","last_active":"2024-09-10T19:22:49.344258+0000","last_peered":"2024-09-10T19:22:49.344258+0000","last_clean":"2024-09-10T19:22:49.344258+0000","last_became_active":"2024-09-10T19:10:13.720647+0000","last_became_peered":"2024-09-10T19:10:13.720647+0000","last_unstale":"2024-09-10T19:22:49.344258+0000","last_undegraded":"2024-09-10T19:22:49.344258+0000","last_fullsized":"2024-09-10T19:22:49.344258+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:21:47.253100+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:21:47.253100+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0012429959999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.7","version":"279'17536","reported_seq":17676,"reported_epoch":279,"state":"active+clean","last_fresh":"2024-09-10T19:22:55.327369+0000","last_change":"2024-09-10T19:22:49.353337+0000","last_active":"2024-09-10T19:22:55.327369+0000","last_peered":"2024-09-10T19:22:55.327369+0000","last_clean":"2024-09-10T19:22:55.327369+0000","last_became_active":"2024-09-10T19:19:54.202928+0000","last_became_peered":"2024-09-10T19:19:54.202928+0000","last_unstale":"2024-09-10T19:22:55.327369+0000","last_undegraded":"2024-09-10T19:22:55.327369+0000","last_fullsized":"2024-09-10T19:22:55.327369+0000","mapping_epoch":229,"log_start":"272'13701","ondisk_log_start":"272'13701","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:19:53.174811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:19:53.174811+0000","last_clean_scrub_stamp":"2024-09-10T19:19:53.174811+0000","objects_scrubbed":1841,"log_size":3835,"log_dups_size":0,"ondisk_log_size":3835,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":159842304,"num_objects":2439,"num_object_clones":0,"num_object_copies":7317,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2439,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15695,"num_write_kb":580288,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.a","version":"237'35","reported_seq":375,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-10T19:22:49.543913+0000","last_change":"2024-09-10T19:22:28.681277+0000","last_active":"2024-09-10T19:22:49.543913+0000","last_peered":"2024-09-10T19:22:49.543913+0000","last_clean":"2024-09-10T19:22:49.543913+0000","last_became_active":"2024-09-10T19:13:54.731274+0000","last_became_peered":"2024-09-10T19:13:54.731274+0000","last_unstale":"2024-09-10T19:22:49.543913+0000","last_undegraded":"2024-09-10T19:22:49.543913+0000","last_fullsized":"2024-09-10T19:22:49.543913+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"237'35","last_scrub_stamp":"2024-09-10T19:22:28.681169+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:22:28.681169+0000","objects_scrubbed":9,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:23:58.430745+0000","scrub_duration":0.0096326229999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":56,"num_read_kb":45,"num_write":36,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,5],"acting":[6,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":154,"reported_epoch":278,"state":"active+undersized","last_fresh":"2024-09-10T19:22:49.302426+0000","last_change":"2024-09-10T19:21:47.992941+0000","last_active":"2024-09-10T19:22:49.302426+0000","last_peered":"2024-09-10T19:22:49.302426+0000","last_clean":"2024-09-10T19:21:43.951531+0000","last_became_active":"2024-09-10T19:21:47.992941+0000","last_became_peered":"2024-09-10T19:21:47.992941+0000","last_unstale":"2024-09-10T19:22:49.302426+0000","last_undegraded":"2024-09-10T19:22:49.302426+0000","last_fullsized":"2024-09-10T19:21:47.972931+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:20:06.554210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:20:06.554210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0035560079999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":["5","3"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":29,"reported_epoch":278,"state":"active+undersized","last_fresh":"2024-09-10T19:22:49.303441+0000","last_change":"2024-09-10T19:21:47.998802+0000","last_active":"2024-09-10T19:22:49.303441+0000","last_peered":"2024-09-10T19:22:49.303441+0000","last_clean":"2024-09-10T19:10:10.676876+0000","last_became_active":"2024-09-10T19:21:47.998802+0000","last_became_peered":"2024-09-10T19:21:47.998802+0000","last_unstale":"2024-09-10T19:22:49.303441+0000","last_undegraded":"2024-09-10T19:22:49.303441+0000","last_fullsized":"2024-09-10T19:21:47.977340+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:20:03.641376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:20:03.641376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":["5","7"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":283,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-10T19:22:49.297923+0000","last_change":"2024-09-10T19:22:30.285621+0000","last_active":"2024-09-10T19:22:49.297923+0000","last_peered":"2024-09-10T19:22:49.297923+0000","last_clean":"2024-09-10T19:22:49.297923+0000","last_became_active":"2024-09-10T19:16:24.858172+0000","last_became_peered":"2024-09-10T19:16:24.858172+0000","last_unstale":"2024-09-10T19:22:49.297923+0000","last_undegraded":"2024-09-10T19:22:49.297923+0000","last_fullsized":"2024-09-10T19:22:49.297923+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:22:30.285547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:21:03.910074+0000","last_clean_scrub_stamp":"2024-09-10T19:22:30.285547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:23:39.761607+0000","scrub_duration":0.0039370639999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,6],"acting":[7,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.6","version":"279'15704","reported_seq":15768,"reported_epoch":279,"state":"active+undersized+degraded","last_fresh":"2024-09-10T19:22:55.312871+0000","last_change":"2024-09-10T19:21:48.003207+0000","last_active":"2024-09-10T19:22:55.312871+0000","last_peered":"2024-09-10T19:22:55.312871+0000","last_clean":"2024-09-10T19:21:46.959322+0000","last_became_active":"2024-09-10T19:21:48.003207+0000","last_became_peered":"2024-09-10T19:21:48.003207+0000","last_unstale":"2024-09-10T19:22:55.312871+0000","last_undegraded":"2024-09-10T19:21:47.993011+0000","last_fullsized":"2024-09-10T19:21:47.992906+0000","mapping_epoch":263,"log_start":"266'11900","ondisk_log_start":"266'11900","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:19:53.174811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:19:53.174811+0000","last_clean_scrub_stamp":"2024-09-10T19:19:53.174811+0000","objects_scrubbed":0,"log_size":3804,"log_dups_size":0,"ondisk_log_size":3804,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":166330368,"num_objects":2538,"num_object_clones":0,"num_object_copies":7614,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2538,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2538,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15704,"num_write_kb":583744,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":["5","6"],"object_location_counts":[{"shards":"5,6","objects":2538}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"237'15","reported_seq":295,"reported_epoch":278,"state":"active+undersized+degraded","last_fresh":"2024-09-10T19:22:49.280434+0000","last_change":"2024-09-10T19:21:47.975617+0000","last_active":"2024-09-10T19:22:49.280434+0000","last_peered":"2024-09-10T19:22:49.280434+0000","last_clean":"2024-09-10T19:20:16.225370+0000","last_became_active":"2024-09-10T19:21:47.975617+0000","last_became_peered":"2024-09-10T19:21:47.975617+0000","last_unstale":"2024-09-10T19:22:49.280434+0000","last_undegraded":"2024-09-10T19:21:47.970276+0000","last_fullsized":"2024-09-10T19:21:47.970174+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"237'15","last_scrub_stamp":"2024-09-10T19:21:46.837828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:21:46.837828+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:23:03.505267+0000","scrub_duration":0.001540919,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":["4","3"],"object_location_counts":[{"shards":"3,4","objects":4}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":242,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-10T19:22:49.297783+0000","last_change":"2024-09-10T19:22:21.064704+0000","last_active":"2024-09-10T19:22:49.297783+0000","last_peered":"2024-09-10T19:22:49.297783+0000","last_clean":"2024-09-10T19:22:49.297783+0000","last_became_active":"2024-09-10T19:16:24.848668+0000","last_became_peered":"2024-09-10T19:16:24.848668+0000","last_unstale":"2024-09-10T19:22:49.297783+0000","last_undegraded":"2024-09-10T19:22:49.297783+0000","last_fullsized":"2024-09-10T19:22:49.297783+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:22:21.064628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:11:44.254513+0000","last_clean_scrub_stamp":"2024-09-10T19:22:21.064628+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:23:35.513650+0000","scrub_duration":0.0086942800000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":167,"reported_epoch":278,"state":"active+undersized","last_fresh":"2024-09-10T19:22:49.280397+0000","last_change":"2024-09-10T19:21:47.977739+0000","last_active":"2024-09-10T19:22:49.280397+0000","last_peered":"2024-09-10T19:22:49.280397+0000","last_clean":"2024-09-10T19:21:43.946899+0000","last_became_active":"2024-09-10T19:21:47.977739+0000","last_became_peered":"2024-09-10T19:21:47.977739+0000","last_unstale":"2024-09-10T19:22:49.280397+0000","last_undegraded":"2024-09-10T19:22:49.280397+0000","last_fullsized":"2024-09-10T19:21:47.969920+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:21:09.367669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:21:09.367669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0065456109999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":["4","3"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":145,"reported_epoch":278,"state":"active+undersized","last_fresh":"2024-09-10T19:22:49.303849+0000","last_change":"2024-09-10T19:21:47.990835+0000","last_active":"2024-09-10T19:22:49.303849+0000","last_peered":"2024-09-10T19:22:49.303849+0000","last_clean":"2024-09-10T19:16:23.155267+0000","last_became_active":"2024-09-10T19:21:47.990835+0000","last_became_peered":"2024-09-10T19:21:47.990835+0000","last_unstale":"2024-09-10T19:22:49.303849+0000","last_undegraded":"2024-09-10T19:22:49.303849+0000","last_fullsized":"2024-09-10T19:21:47.981819+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:20:08.746643+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:20:08.746643+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0058765429999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":["7","3"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.5","version":"279'15967","reported_seq":16267,"reported_epoch":279,"state":"active+undersized+degraded","last_fresh":"2024-09-10T19:22:55.324027+0000","last_change":"2024-09-10T19:21:47.994750+0000","last_active":"2024-09-10T19:22:55.324027+0000","last_peered":"2024-09-10T19:22:55.324027+0000","last_clean":"2024-09-10T19:21:46.944630+0000","last_became_active":"2024-09-10T19:21:47.994750+0000","last_became_peered":"2024-09-10T19:21:47.994750+0000","last_unstale":"2024-09-10T19:22:55.324027+0000","last_undegraded":"2024-09-10T19:21:47.973184+0000","last_fullsized":"2024-09-10T19:21:47.972919+0000","mapping_epoch":263,"log_start":"266'12176","ondisk_log_start":"266'12176","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:19:53.174811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:19:53.174811+0000","last_clean_scrub_stamp":"2024-09-10T19:19:53.174811+0000","objects_scrubbed":5823,"log_size":3791,"log_dups_size":0,"ondisk_log_size":3791,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":163774464,"num_objects":2499,"num_object_clones":0,"num_object_copies":7497,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2499,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2499,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15967,"num_write_kb":590912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":["5","7"],"object_location_counts":[{"shards":"5,7","objects":2499}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.8","version":"237'34","reported_seq":317,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-10T19:22:49.302366+0000","last_change":"2024-09-10T19:21:58.440243+0000","last_active":"2024-09-10T19:22:49.302366+0000","last_peered":"2024-09-10T19:22:49.302366+0000","last_clean":"2024-09-10T19:22:49.302366+0000","last_became_active":"2024-09-10T19:10:09.820950+0000","last_became_peered":"2024-09-10T19:10:09.820950+0000","last_unstale":"2024-09-10T19:22:49.302366+0000","last_undegraded":"2024-09-10T19:22:49.302366+0000","last_fullsized":"2024-09-10T19:22:49.302366+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"237'34","last_scrub_stamp":"2024-09-10T19:21:58.440165+0000","last_deep_scrub":"220'22","last_deep_scrub_stamp":"2024-09-10T19:19:24.597866+0000","last_clean_scrub_stamp":"2024-09-10T19:21:58.440165+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:23:13.112083+0000","scrub_duration":0.0050099799999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":147,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-10T19:22:49.258382+0000","last_change":"2024-09-10T19:20:08.533315+0000","last_active":"2024-09-10T19:22:49.258382+0000","last_peered":"2024-09-10T19:22:49.258382+0000","last_clean":"2024-09-10T19:22:49.258382+0000","last_became_active":"2024-09-10T19:16:24.857253+0000","last_became_peered":"2024-09-10T19:16:24.857253+0000","last_unstale":"2024-09-10T19:22:49.258382+0000","last_undegraded":"2024-09-10T19:22:49.258382+0000","last_fullsized":"2024-09-10T19:22:49.258382+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:20:08.533267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:20:08.533267+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0023880849999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.9","version":"80'9","reported_seq":437,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-10T19:22:50.896051+0000","last_change":"2024-09-10T19:22:29.301346+0000","last_active":"2024-09-10T19:22:50.896051+0000","last_peered":"2024-09-10T19:22:50.896051+0000","last_clean":"2024-09-10T19:22:50.896051+0000","last_became_active":"2024-09-10T19:10:11.702878+0000","last_became_peered":"2024-09-10T19:10:11.702878+0000","last_unstale":"2024-09-10T19:22:50.896051+0000","last_undegraded":"2024-09-10T19:22:50.896051+0000","last_fullsized":"2024-09-10T19:22:50.896051+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"80'9","last_scrub_stamp":"2024-09-10T19:22:29.301243+0000","last_deep_scrub":"80'9","last_deep_scrub_stamp":"2024-09-10T19:20:57.882496+0000","last_clean_scrub_stamp":"2024-09-10T19:22:29.301243+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:23:41.869017+0000","scrub_duration":0.013296258,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":260,"reported_epoch":278,"state":"active+undersized","last_fresh":"2024-09-10T19:22:49.302419+0000","last_change":"2024-09-10T19:21:47.994405+0000","last_active":"2024-09-10T19:22:49.302419+0000","last_peered":"2024-09-10T19:22:49.302419+0000","last_clean":"2024-09-10T19:21:43.955746+0000","last_became_active":"2024-09-10T19:21:47.994405+0000","last_became_peered":"2024-09-10T19:21:47.994405+0000","last_unstale":"2024-09-10T19:22:49.302419+0000","last_undegraded":"2024-09-10T19:22:49.302419+0000","last_fullsized":"2024-09-10T19:21:47.974191+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:19:15.539488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:19:15.539488+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.033498668000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":["5","6"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.4","version":"279'15818","reported_seq":15881,"reported_epoch":279,"state":"active+undersized+degraded","last_fresh":"2024-09-10T19:22:54.916845+0000","last_change":"2024-09-10T19:21:47.976009+0000","last_active":"2024-09-10T19:22:54.916845+0000","last_peered":"2024-09-10T19:22:54.916845+0000","last_clean":"2024-09-10T19:21:46.956094+0000","last_became_active":"2024-09-10T19:21:47.976009+0000","last_became_peered":"2024-09-10T19:21:47.976009+0000","last_unstale":"2024-09-10T19:22:54.916845+0000","last_undegraded":"2024-09-10T19:21:47.970032+0000","last_fullsized":"2024-09-10T19:21:47.969894+0000","mapping_epoch":263,"log_start":"265'11500","ondisk_log_start":"265'11500","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:19:53.174811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:19:53.174811+0000","last_clean_scrub_stamp":"2024-09-10T19:19:53.174811+0000","objects_scrubbed":0,"log_size":4318,"log_dups_size":0,"ondisk_log_size":4318,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":161873920,"num_objects":2470,"num_object_clones":0,"num_object_copies":7410,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2470,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2470,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15818,"num_write_kb":585216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":["4","3"],"object_location_counts":[{"shards":"3,4","objects":2470}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.9","version":"237'18","reported_seq":330,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-10T19:22:49.280076+0000","last_change":"2024-09-10T19:22:00.500856+0000","last_active":"2024-09-10T19:22:49.280076+0000","last_peered":"2024-09-10T19:22:49.280076+0000","last_clean":"2024-09-10T19:22:49.280076+0000","last_became_active":"2024-09-10T19:13:54.718152+0000","last_became_peered":"2024-09-10T19:13:54.718152+0000","last_unstale":"2024-09-10T19:22:49.280076+0000","last_undegraded":"2024-09-10T19:22:49.280076+0000","last_fullsized":"2024-09-10T19:22:49.280076+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"237'18","last_scrub_stamp":"2024-09-10T19:22:00.500765+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:22:00.500765+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:23:22.845668+0000","scrub_duration":0.0093780900000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.f","version":"59'2","reported_seq":291,"reported_epoch":278,"state":"active+undersized+degraded","last_fresh":"2024-09-10T19:22:49.281110+0000","last_change":"2024-09-10T19:21:47.986039+0000","last_active":"2024-09-10T19:22:49.281110+0000","last_peered":"2024-09-10T19:22:49.281110+0000","last_clean":"2024-09-10T19:21:43.952266+0000","last_became_active":"2024-09-10T19:21:47.986039+0000","last_became_peered":"2024-09-10T19:21:47.986039+0000","last_unstale":"2024-09-10T19:22:49.281110+0000","last_undegraded":"2024-09-10T19:21:47.971581+0000","last_fullsized":"2024-09-10T19:21:47.971480+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'2","last_scrub_stamp":"2024-09-10T19:20:56.467569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:20:56.467569+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.013126802,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":14,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":["4","7"],"object_location_counts":[{"shards":"4,7","objects":2}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":109,"reported_epoch":278,"state":"active+undersized","last_fresh":"2024-09-10T19:22:49.302457+0000","last_change":"2024-09-10T19:21:47.983793+0000","last_active":"2024-09-10T19:22:49.302457+0000","last_peered":"2024-09-10T19:22:49.302457+0000","last_clean":"2024-09-10T19:12:44.685930+0000","last_became_active":"2024-09-10T19:21:47.983793+0000","last_became_peered":"2024-09-10T19:21:47.983793+0000","last_unstale":"2024-09-10T19:22:49.302457+0000","last_undegraded":"2024-09-10T19:22:49.302457+0000","last_fullsized":"2024-09-10T19:21:47.972418+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:21:45.880289+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:11:46.587846+0000","last_clean_scrub_stamp":"2024-09-10T19:21:45.880289+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:23:15.161241+0000","scrub_duration":0.001267142,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":["5","4"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":270,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-10T19:22:49.258479+0000","last_change":"2024-09-10T19:20:09.576625+0000","last_active":"2024-09-10T19:22:49.258479+0000","last_peered":"2024-09-10T19:22:49.258479+0000","last_clean":"2024-09-10T19:22:49.258479+0000","last_became_active":"2024-09-10T19:13:54.711804+0000","last_became_peered":"2024-09-10T19:13:54.711804+0000","last_unstale":"2024-09-10T19:22:49.258479+0000","last_undegraded":"2024-09-10T19:22:49.258479+0000","last_fullsized":"2024-09-10T19:22:49.258479+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:20:09.576542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:20:09.576542+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0030134599999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.16","version":"237'15","reported_seq":278,"reported_epoch":278,"state":"active+undersized+degraded","last_fresh":"2024-09-10T19:22:49.258405+0000","last_change":"2024-09-10T19:21:47.988985+0000","last_active":"2024-09-10T19:22:49.258405+0000","last_peered":"2024-09-10T19:22:49.258405+0000","last_clean":"2024-09-10T19:20:15.530776+0000","last_became_active":"2024-09-10T19:21:47.988985+0000","last_became_peered":"2024-09-10T19:21:47.988985+0000","last_unstale":"2024-09-10T19:22:49.258405+0000","last_undegraded":"2024-09-10T19:21:47.974466+0000","last_fullsized":"2024-09-10T19:21:47.974359+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"237'15","last_scrub_stamp":"2024-09-10T19:21:01.691067+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:21:01.691067+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0065392059999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":["3","7"],"object_location_counts":[{"shards":"3,7","objects":4}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":113,"reported_epoch":278,"state":"active+undersized","last_fresh":"2024-09-10T19:22:49.257261+0000","last_change":"2024-09-10T19:21:47.995029+0000","last_active":"2024-09-10T19:22:49.257261+0000","last_peered":"2024-09-10T19:22:49.257261+0000","last_clean":"2024-09-10T19:12:44.686063+0000","last_became_active":"2024-09-10T19:21:47.995029+0000","last_became_peered":"2024-09-10T19:21:47.995029+0000","last_unstale":"2024-09-10T19:22:49.257261+0000","last_undegraded":"2024-09-10T19:22:49.257261+0000","last_fullsized":"2024-09-10T19:21:47.969981+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:20:57.698558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:11:41.458523+0000","last_clean_scrub_stamp":"2024-09-10T19:20:57.698558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0012499830000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":["4","6"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":279,"reported_epoch":278,"state":"active+undersized","last_fresh":"2024-09-10T19:22:49.258243+0000","last_change":"2024-09-10T19:21:47.984694+0000","last_active":"2024-09-10T19:22:49.258243+0000","last_peered":"2024-09-10T19:22:49.258243+0000","last_clean":"2024-09-10T19:21:43.944868+0000","last_became_active":"2024-09-10T19:21:47.984694+0000","last_became_peered":"2024-09-10T19:21:47.984694+0000","last_unstale":"2024-09-10T19:22:49.258243+0000","last_undegraded":"2024-09-10T19:22:49.258243+0000","last_fullsized":"2024-09-10T19:21:47.974535+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:21:43.633645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:21:43.633645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001568039,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":["3","7"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"66'1","reported_seq":167,"reported_epoch":278,"state":"active+undersized+degraded","last_fresh":"2024-09-10T19:22:49.298956+0000","last_change":"2024-09-10T19:21:48.003365+0000","last_active":"2024-09-10T19:22:49.298956+0000","last_peered":"2024-09-10T19:22:49.298956+0000","last_clean":"2024-09-10T19:21:43.949315+0000","last_became_active":"2024-09-10T19:21:48.003365+0000","last_became_peered":"2024-09-10T19:21:48.003365+0000","last_unstale":"2024-09-10T19:22:49.298956+0000","last_undegraded":"2024-09-10T19:21:47.971226+0000","last_fullsized":"2024-09-10T19:21:47.970632+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'1","last_scrub_stamp":"2024-09-10T19:20:07.338085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:20:07.338085+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0056882089999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":13,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":["4","7"],"object_location_counts":[{"shards":"4,7","objects":1}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":292,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-10T19:22:49.544494+0000","last_change":"2024-09-10T19:21:54.567071+0000","last_active":"2024-09-10T19:22:49.544494+0000","last_peered":"2024-09-10T19:22:49.544494+0000","last_clean":"2024-09-10T19:22:49.544494+0000","last_became_active":"2024-09-10T19:13:54.724027+0000","last_became_peered":"2024-09-10T19:13:54.724027+0000","last_unstale":"2024-09-10T19:22:49.544494+0000","last_undegraded":"2024-09-10T19:22:49.544494+0000","last_fullsized":"2024-09-10T19:22:49.544494+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:21:54.566800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:21:54.566800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:23:15.940759+0000","scrub_duration":0.010619585000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,5],"acting":[6,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"237'12","reported_seq":318,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-10T19:22:49.258505+0000","last_change":"2024-09-10T19:22:26.419563+0000","last_active":"2024-09-10T19:22:49.258505+0000","last_peered":"2024-09-10T19:22:49.258505+0000","last_clean":"2024-09-10T19:22:49.258505+0000","last_became_active":"2024-09-10T19:16:24.863398+0000","last_became_peered":"2024-09-10T19:16:24.863398+0000","last_unstale":"2024-09-10T19:22:49.258505+0000","last_undegraded":"2024-09-10T19:22:49.258505+0000","last_fullsized":"2024-09-10T19:22:49.258505+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"237'12","last_scrub_stamp":"2024-09-10T19:22:26.419497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:22:26.419497+0000","objects_scrubbed":3,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:23:31.536216+0000","scrub_duration":0.0033796080000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":18,"num_read_kb":15,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":282,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-10T19:22:49.304700+0000","last_change":"2024-09-10T19:22:30.626044+0000","last_active":"2024-09-10T19:22:49.304700+0000","last_peered":"2024-09-10T19:22:49.304700+0000","last_clean":"2024-09-10T19:22:49.304700+0000","last_became_active":"2024-09-10T19:13:54.729917+0000","last_became_peered":"2024-09-10T19:13:54.729917+0000","last_unstale":"2024-09-10T19:22:49.304700+0000","last_undegraded":"2024-09-10T19:22:49.304700+0000","last_fullsized":"2024-09-10T19:22:49.304700+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:22:30.625942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:11:23.264793+0000","last_clean_scrub_stamp":"2024-09-10T19:22:30.625942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:23:40.795531+0000","scrub_duration":0.010093893,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":273,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-10T19:22:49.298585+0000","last_change":"2024-09-10T19:22:28.305076+0000","last_active":"2024-09-10T19:22:49.298585+0000","last_peered":"2024-09-10T19:22:49.298585+0000","last_clean":"2024-09-10T19:22:49.298585+0000","last_became_active":"2024-09-10T19:10:14.054360+0000","last_became_peered":"2024-09-10T19:10:14.054360+0000","last_unstale":"2024-09-10T19:22:49.298585+0000","last_undegraded":"2024-09-10T19:22:49.298585+0000","last_fullsized":"2024-09-10T19:22:49.298585+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:22:28.304999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:22:28.304999+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:23:43.703876+0000","scrub_duration":0.0082740739999999993,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"237'25","reported_seq":343,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-10T19:22:49.257793+0000","last_change":"2024-09-10T19:20:02.458352+0000","last_active":"2024-09-10T19:22:49.257793+0000","last_peered":"2024-09-10T19:22:49.257793+0000","last_clean":"2024-09-10T19:22:49.257793+0000","last_became_active":"2024-09-10T19:16:24.850702+0000","last_became_peered":"2024-09-10T19:16:24.850702+0000","last_unstale":"2024-09-10T19:22:49.257793+0000","last_undegraded":"2024-09-10T19:22:49.257793+0000","last_fullsized":"2024-09-10T19:22:49.257793+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"80'15","last_scrub_stamp":"2024-09-10T19:20:02.458275+0000","last_deep_scrub":"80'15","last_deep_scrub_stamp":"2024-09-10T19:11:41.719310+0000","last_clean_scrub_stamp":"2024-09-10T19:20:02.458275+0000","objects_scrubbed":5,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0054307269999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,5],"acting":[3,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":269,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-10T19:22:49.301843+0000","last_change":"2024-09-10T19:21:48.298990+0000","last_active":"2024-09-10T19:22:49.301843+0000","last_peered":"2024-09-10T19:22:49.301843+0000","last_clean":"2024-09-10T19:22:49.301843+0000","last_became_active":"2024-09-10T19:10:07.685870+0000","last_became_peered":"2024-09-10T19:10:07.685870+0000","last_unstale":"2024-09-10T19:22:49.301843+0000","last_undegraded":"2024-09-10T19:22:49.301843+0000","last_fullsized":"2024-09-10T19:22:49.301843+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:21:48.298913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:21:48.298913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:22:55.545159+0000","scrub_duration":0.0075260040000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"264'11","reported_seq":446,"reported_epoch":278,"state":"active+undersized+degraded","last_fresh":"2024-09-10T19:22:50.889591+0000","last_change":"2024-09-10T19:21:47.993917+0000","last_active":"2024-09-10T19:22:50.889591+0000","last_peered":"2024-09-10T19:22:50.889591+0000","last_clean":"2024-09-10T19:21:45.876612+0000","last_became_active":"2024-09-10T19:21:47.993917+0000","last_became_peered":"2024-09-10T19:21:47.993917+0000","last_unstale":"2024-09-10T19:22:50.889591+0000","last_undegraded":"2024-09-10T19:21:47.974032+0000","last_fullsized":"2024-09-10T19:21:47.973921+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"117'10","last_scrub_stamp":"2024-09-10T19:20:55.377819+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:20:55.377819+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0063951320000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":["5","3"],"object_location_counts":[{"shards":"3,5","objects":1}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":25,"reported_epoch":278,"state":"active+undersized","last_fresh":"2024-09-10T19:22:49.297821+0000","last_change":"2024-09-10T19:21:47.989023+0000","last_active":"2024-09-10T19:22:49.297821+0000","last_peered":"2024-09-10T19:22:49.297821+0000","last_clean":"2024-09-10T19:10:12.693598+0000","last_became_active":"2024-09-10T19:21:47.989023+0000","last_became_peered":"2024-09-10T19:21:47.989023+0000","last_unstale":"2024-09-10T19:22:49.297821+0000","last_undegraded":"2024-09-10T19:22:49.297821+0000","last_fullsized":"2024-09-10T19:21:47.982009+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:20:46.625193+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:20:46.625193+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":["7","3"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.15","version":"237'15","reported_seq":332,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-10T19:22:49.303514+0000","last_change":"2024-09-10T19:21:57.580717+0000","last_active":"2024-09-10T19:22:49.303514+0000","last_peered":"2024-09-10T19:22:49.303514+0000","last_clean":"2024-09-10T19:22:49.303514+0000","last_became_active":"2024-09-10T19:16:24.859740+0000","last_became_peered":"2024-09-10T19:16:24.859740+0000","last_unstale":"2024-09-10T19:22:49.303514+0000","last_undegraded":"2024-09-10T19:22:49.303514+0000","last_fullsized":"2024-09-10T19:22:49.303514+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"237'15","last_scrub_stamp":"2024-09-10T19:21:57.580644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:21:57.580644+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:23:03.202902+0000","scrub_duration":0.11368046,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":25,"reported_epoch":278,"state":"active+undersized","last_fresh":"2024-09-10T19:22:49.280884+0000","last_change":"2024-09-10T19:21:47.975293+0000","last_active":"2024-09-10T19:22:49.280884+0000","last_peered":"2024-09-10T19:22:49.280884+0000","last_clean":"2024-09-10T19:10:06.664709+0000","last_became_active":"2024-09-10T19:21:47.975293+0000","last_became_peered":"2024-09-10T19:21:47.975293+0000","last_unstale":"2024-09-10T19:22:49.280884+0000","last_undegraded":"2024-09-10T19:22:49.280884+0000","last_fullsized":"2024-09-10T19:21:47.970537+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:20:48.709474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:20:48.709474+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":["4","3"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"155'9","reported_seq":440,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-10T19:22:50.887338+0000","last_change":"2024-09-10T19:22:00.716614+0000","last_active":"2024-09-10T19:22:50.887338+0000","last_peered":"2024-09-10T19:22:50.887338+0000","last_clean":"2024-09-10T19:22:50.887338+0000","last_became_active":"2024-09-10T19:16:24.850850+0000","last_became_peered":"2024-09-10T19:16:24.850850+0000","last_unstale":"2024-09-10T19:22:50.887338+0000","last_undegraded":"2024-09-10T19:22:50.887338+0000","last_fullsized":"2024-09-10T19:22:50.887338+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"155'9","last_scrub_stamp":"2024-09-10T19:22:00.716555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:22:00.716555+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:23:29.355333+0000","scrub_duration":0.0020496289999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,4],"acting":[3,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":272,"reported_epoch":278,"state":"active+undersized","last_fresh":"2024-09-10T19:22:49.280909+0000","last_change":"2024-09-10T19:21:47.976028+0000","last_active":"2024-09-10T19:22:49.280909+0000","last_peered":"2024-09-10T19:22:49.280909+0000","last_clean":"2024-09-10T19:21:43.951991+0000","last_became_active":"2024-09-10T19:21:47.976028+0000","last_became_peered":"2024-09-10T19:21:47.976028+0000","last_unstale":"2024-09-10T19:22:49.280909+0000","last_undegraded":"2024-09-10T19:22:49.280909+0000","last_fullsized":"2024-09-10T19:21:47.970502+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:19:12.504402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:19:12.504402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0034925469999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":["4","5"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.12","version":"237'19","reported_seq":328,"reported_epoch":278,"state":"active+undersized+degraded","last_fresh":"2024-09-10T19:22:49.543637+0000","last_change":"2024-09-10T19:21:48.003068+0000","last_active":"2024-09-10T19:22:49.543637+0000","last_peered":"2024-09-10T19:22:49.543637+0000","last_clean":"2024-09-10T19:21:43.951440+0000","last_became_active":"2024-09-10T19:21:48.003068+0000","last_became_peered":"2024-09-10T19:21:48.003068+0000","last_unstale":"2024-09-10T19:22:49.543637+0000","last_undegraded":"2024-09-10T19:21:47.979935+0000","last_fullsized":"2024-09-10T19:21:47.979836+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"237'19","last_scrub_stamp":"2024-09-10T19:21:07.378184+0000","last_deep_scrub":"79'13","last_deep_scrub_stamp":"2024-09-10T19:11:22.072400+0000","last_clean_scrub_stamp":"2024-09-10T19:21:07.378184+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.004147263,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":["6","5"],"object_location_counts":[{"shards":"5,6","objects":4}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.14","version":"59'1","reported_seq":286,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-10T19:22:49.543367+0000","last_change":"2024-09-10T19:22:20.715211+0000","last_active":"2024-09-10T19:22:49.543367+0000","last_peered":"2024-09-10T19:22:49.543367+0000","last_clean":"2024-09-10T19:22:49.543367+0000","last_became_active":"2024-09-10T19:10:07.683263+0000","last_became_peered":"2024-09-10T19:10:07.683263+0000","last_unstale":"2024-09-10T19:22:49.543367+0000","last_undegraded":"2024-09-10T19:22:49.543367+0000","last_fullsized":"2024-09-10T19:22:49.543367+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'1","last_scrub_stamp":"2024-09-10T19:22:20.715123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:22:20.715123+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:23:29.620364+0000","scrub_duration":0.0046532379999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":362,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":279,"reported_epoch":278,"state":"active+undersized","last_fresh":"2024-09-10T19:22:49.258576+0000","last_change":"2024-09-10T19:21:47.996741+0000","last_active":"2024-09-10T19:22:49.258576+0000","last_peered":"2024-09-10T19:22:49.258576+0000","last_clean":"2024-09-10T19:21:43.944122+0000","last_became_active":"2024-09-10T19:21:47.996741+0000","last_became_peered":"2024-09-10T19:21:47.996741+0000","last_unstale":"2024-09-10T19:22:49.258576+0000","last_undegraded":"2024-09-10T19:22:49.258576+0000","last_fullsized":"2024-09-10T19:21:47.974561+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:20:50.491473+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:20:50.491473+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0066726329999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":["3","5"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":103,"reported_epoch":278,"state":"active+undersized","last_fresh":"2024-09-10T19:22:49.302246+0000","last_change":"2024-09-10T19:21:48.003432+0000","last_active":"2024-09-10T19:22:49.302246+0000","last_peered":"2024-09-10T19:22:49.302246+0000","last_clean":"2024-09-10T19:12:44.680563+0000","last_became_active":"2024-09-10T19:21:48.003432+0000","last_became_peered":"2024-09-10T19:21:48.003432+0000","last_unstale":"2024-09-10T19:22:49.302246+0000","last_undegraded":"2024-09-10T19:22:49.302246+0000","last_fullsized":"2024-09-10T19:21:47.974956+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:21:09.714677+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:21:09.714677+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001865058,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":["5","6"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.13","version":"237'19","reported_seq":329,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-10T19:22:49.280687+0000","last_change":"2024-09-10T19:22:22.040432+0000","last_active":"2024-09-10T19:22:49.280687+0000","last_peered":"2024-09-10T19:22:49.280687+0000","last_clean":"2024-09-10T19:22:49.280687+0000","last_became_active":"2024-09-10T19:13:54.720034+0000","last_became_peered":"2024-09-10T19:13:54.720034+0000","last_unstale":"2024-09-10T19:22:49.280687+0000","last_undegraded":"2024-09-10T19:22:49.280687+0000","last_fullsized":"2024-09-10T19:22:49.280687+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"237'19","last_scrub_stamp":"2024-09-10T19:22:22.040341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:22:22.040341+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:23:38.655498+0000","scrub_duration":0.013845334000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":216,"reported_epoch":278,"state":"active+undersized","last_fresh":"2024-09-10T19:22:49.303528+0000","last_change":"2024-09-10T19:21:48.019714+0000","last_active":"2024-09-10T19:22:49.303528+0000","last_peered":"2024-09-10T19:22:49.303528+0000","last_clean":"2024-09-10T19:21:43.954115+0000","last_became_active":"2024-09-10T19:21:48.019714+0000","last_became_peered":"2024-09-10T19:21:48.019714+0000","last_unstale":"2024-09-10T19:22:49.303528+0000","last_undegraded":"2024-09-10T19:22:49.303528+0000","last_fullsized":"2024-09-10T19:21:47.993355+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:19:10.439659+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:11:10.031245+0000","last_clean_scrub_stamp":"2024-09-10T19:19:10.439659+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.014001438,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":["5","6"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":219,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-10T19:22:49.302277+0000","last_change":"2024-09-10T19:20:45.375913+0000","last_active":"2024-09-10T19:22:49.302277+0000","last_peered":"2024-09-10T19:22:49.302277+0000","last_clean":"2024-09-10T19:22:49.302277+0000","last_became_active":"2024-09-10T19:13:54.729703+0000","last_became_peered":"2024-09-10T19:13:54.729703+0000","last_unstale":"2024-09-10T19:22:49.302277+0000","last_undegraded":"2024-09-10T19:22:49.302277+0000","last_fullsized":"2024-09-10T19:22:49.302277+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:20:45.375839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:20:45.375839+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0146499,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":260,"reported_epoch":278,"state":"active+undersized","last_fresh":"2024-09-10T19:22:49.257503+0000","last_change":"2024-09-10T19:21:47.993530+0000","last_active":"2024-09-10T19:22:49.257503+0000","last_peered":"2024-09-10T19:22:49.257503+0000","last_clean":"2024-09-10T19:21:43.944067+0000","last_became_active":"2024-09-10T19:21:47.993530+0000","last_became_peered":"2024-09-10T19:21:47.993530+0000","last_unstale":"2024-09-10T19:22:49.257503+0000","last_undegraded":"2024-09-10T19:22:49.257503+0000","last_fullsized":"2024-09-10T19:21:47.975469+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:19:13.231390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:19:13.231390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0029165770000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":["3","5"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"237'10","reported_seq":304,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-10T19:22:49.257566+0000","last_change":"2024-09-10T19:22:28.363094+0000","last_active":"2024-09-10T19:22:49.257566+0000","last_peered":"2024-09-10T19:22:49.257566+0000","last_clean":"2024-09-10T19:22:49.257566+0000","last_became_active":"2024-09-10T19:13:54.711125+0000","last_became_peered":"2024-09-10T19:13:54.711125+0000","last_unstale":"2024-09-10T19:22:49.257566+0000","last_undegraded":"2024-09-10T19:22:49.257566+0000","last_fullsized":"2024-09-10T19:22:49.257566+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"237'10","last_scrub_stamp":"2024-09-10T19:22:28.363041+0000","last_deep_scrub":"222'6","last_deep_scrub_stamp":"2024-09-10T19:19:30.328841+0000","last_clean_scrub_stamp":"2024-09-10T19:22:28.363041+0000","objects_scrubbed":2,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:23:46.463086+0000","scrub_duration":0.0040448380000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":12,"num_read_kb":10,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,4],"acting":[3,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":279,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-10T19:22:49.305015+0000","last_change":"2024-09-10T19:20:59.297958+0000","last_active":"2024-09-10T19:22:49.305015+0000","last_peered":"2024-09-10T19:22:49.305015+0000","last_clean":"2024-09-10T19:22:49.305015+0000","last_became_active":"2024-09-10T19:16:24.852460+0000","last_became_peered":"2024-09-10T19:16:24.852460+0000","last_unstale":"2024-09-10T19:22:49.305015+0000","last_undegraded":"2024-09-10T19:22:49.305015+0000","last_fullsized":"2024-09-10T19:22:49.305015+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:20:59.297872+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:20:59.297872+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0090041549999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,3],"acting":[5,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":275,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-10T19:22:49.285890+0000","last_change":"2024-09-10T19:21:56.530235+0000","last_active":"2024-09-10T19:22:49.285890+0000","last_peered":"2024-09-10T19:22:49.285890+0000","last_clean":"2024-09-10T19:22:49.285890+0000","last_became_active":"2024-09-10T19:10:11.698536+0000","last_became_peered":"2024-09-10T19:10:11.698536+0000","last_unstale":"2024-09-10T19:22:49.285890+0000","last_undegraded":"2024-09-10T19:22:49.285890+0000","last_fullsized":"2024-09-10T19:22:49.285890+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:21:56.529808+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:11:31.837944+0000","last_clean_scrub_stamp":"2024-09-10T19:21:56.529808+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:23:14.227016+0000","scrub_duration":0.0045868109999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":293,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-10T19:22:49.298356+0000","last_change":"2024-09-10T19:22:31.278309+0000","last_active":"2024-09-10T19:22:49.298356+0000","last_peered":"2024-09-10T19:22:49.298356+0000","last_clean":"2024-09-10T19:22:49.298356+0000","last_became_active":"2024-09-10T19:16:24.852356+0000","last_became_peered":"2024-09-10T19:16:24.852356+0000","last_unstale":"2024-09-10T19:22:49.298356+0000","last_undegraded":"2024-09-10T19:22:49.298356+0000","last_fullsized":"2024-09-10T19:22:49.298356+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:22:31.278179+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:22:31.278179+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:23:56.593462+0000","scrub_duration":0.0036472750000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,3],"acting":[7,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.11","version":"237'19","reported_seq":59,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-10T19:22:49.257271+0000","last_change":"2024-09-10T19:22:22.464814+0000","last_active":"2024-09-10T19:22:49.257271+0000","last_peered":"2024-09-10T19:22:49.257271+0000","last_clean":"2024-09-10T19:22:49.257271+0000","last_became_active":"2024-09-10T19:21:04.169444+0000","last_became_peered":"2024-09-10T19:21:04.169444+0000","last_unstale":"2024-09-10T19:22:49.257271+0000","last_undegraded":"2024-09-10T19:22:49.257271+0000","last_fullsized":"2024-09-10T19:22:49.257271+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"237'19","last_scrub_stamp":"2024-09-10T19:22:22.464729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:22:22.464729+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:23:33.416371+0000","scrub_duration":0.0058949659999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":286,"reported_epoch":278,"state":"active+undersized","last_fresh":"2024-09-10T19:22:49.543543+0000","last_change":"2024-09-10T19:21:48.002896+0000","last_active":"2024-09-10T19:22:49.543543+0000","last_peered":"2024-09-10T19:22:49.543543+0000","last_clean":"2024-09-10T19:21:43.952651+0000","last_became_active":"2024-09-10T19:21:48.002896+0000","last_became_peered":"2024-09-10T19:21:48.002896+0000","last_unstale":"2024-09-10T19:22:49.543543+0000","last_undegraded":"2024-09-10T19:22:49.543543+0000","last_fullsized":"2024-09-10T19:21:47.987616+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:20:08.412227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:13:05.876655+0000","last_clean_scrub_stamp":"2024-09-10T19:20:08.412227+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0037109489999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":["6","5"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":265,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-10T19:22:49.297319+0000","last_change":"2024-09-10T19:21:50.103708+0000","last_active":"2024-09-10T19:22:49.297319+0000","last_peered":"2024-09-10T19:22:49.297319+0000","last_clean":"2024-09-10T19:22:49.297319+0000","last_became_active":"2024-09-10T19:10:11.699013+0000","last_became_peered":"2024-09-10T19:10:11.699013+0000","last_unstale":"2024-09-10T19:22:49.297319+0000","last_undegraded":"2024-09-10T19:22:49.297319+0000","last_fullsized":"2024-09-10T19:22:49.297319+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:21:50.103617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:11:57.112092+0000","last_clean_scrub_stamp":"2024-09-10T19:21:50.103617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:23:13.194296+0000","scrub_duration":0.011550181,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":260,"reported_epoch":278,"state":"active+undersized","last_fresh":"2024-09-10T19:22:49.258527+0000","last_change":"2024-09-10T19:21:47.990590+0000","last_active":"2024-09-10T19:22:49.258527+0000","last_peered":"2024-09-10T19:22:49.258527+0000","last_clean":"2024-09-10T19:21:43.944266+0000","last_became_active":"2024-09-10T19:21:47.990590+0000","last_became_peered":"2024-09-10T19:21:47.990590+0000","last_unstale":"2024-09-10T19:22:49.258527+0000","last_undegraded":"2024-09-10T19:22:49.258527+0000","last_fullsized":"2024-09-10T19:21:47.974352+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:19:18.276069+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:19:18.276069+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0034003950000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":["3","6"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"80'2","reported_seq":289,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-10T19:22:49.299416+0000","last_change":"2024-09-10T19:22:23.295914+0000","last_active":"2024-09-10T19:22:49.299416+0000","last_peered":"2024-09-10T19:22:49.299416+0000","last_clean":"2024-09-10T19:22:49.299416+0000","last_became_active":"2024-09-10T19:16:24.851220+0000","last_became_peered":"2024-09-10T19:16:24.851220+0000","last_unstale":"2024-09-10T19:22:49.299416+0000","last_undegraded":"2024-09-10T19:22:49.299416+0000","last_fullsized":"2024-09-10T19:22:49.299416+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"80'2","last_scrub_stamp":"2024-09-10T19:22:23.295840+0000","last_deep_scrub":"80'2","last_deep_scrub_stamp":"2024-09-10T19:11:30.272043+0000","last_clean_scrub_stamp":"2024-09-10T19:22:23.295840+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:23:37.219620+0000","scrub_duration":0.015418665,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":369,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":369,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,3],"acting":[7,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":276,"reported_epoch":278,"state":"active+undersized","last_fresh":"2024-09-10T19:22:49.303037+0000","last_change":"2024-09-10T19:21:47.978661+0000","last_active":"2024-09-10T19:22:49.303037+0000","last_peered":"2024-09-10T19:22:49.303037+0000","last_clean":"2024-09-10T19:21:43.952149+0000","last_became_active":"2024-09-10T19:21:47.978661+0000","last_became_peered":"2024-09-10T19:21:47.978661+0000","last_unstale":"2024-09-10T19:22:49.303037+0000","last_undegraded":"2024-09-10T19:22:49.303037+0000","last_fullsized":"2024-09-10T19:21:47.973220+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":218,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:19:19.581564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:19:19.581564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.01041663,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":["5","4"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1e","version":"237'25","reported_seq":272,"reported_epoch":278,"state":"active+undersized+degraded","last_fresh":"2024-09-10T19:22:49.298043+0000","last_change":"2024-09-10T19:21:47.993629+0000","last_active":"2024-09-10T19:22:49.298043+0000","last_peered":"2024-09-10T19:22:49.298043+0000","last_clean":"2024-09-10T19:20:16.344252+0000","last_became_active":"2024-09-10T19:21:47.993629+0000","last_became_peered":"2024-09-10T19:21:47.993629+0000","last_unstale":"2024-09-10T19:22:49.298043+0000","last_undegraded":"2024-09-10T19:21:47.981614+0000","last_fullsized":"2024-09-10T19:21:47.981497+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"237'25","last_scrub_stamp":"2024-09-10T19:20:44.696751+0000","last_deep_scrub":"80'15","last_deep_scrub_stamp":"2024-09-10T19:11:25.883919+0000","last_clean_scrub_stamp":"2024-09-10T19:20:44.696751+0000","objects_scrubbed":5,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0040337309999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":24,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":["7","3"],"object_location_counts":[{"shards":"3,7","objects":5}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"66'8","reported_seq":442,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-10T19:22:50.887393+0000","last_change":"2024-09-10T19:21:48.538842+0000","last_active":"2024-09-10T19:22:50.887393+0000","last_peered":"2024-09-10T19:22:50.887393+0000","last_clean":"2024-09-10T19:22:50.887393+0000","last_became_active":"2024-09-10T19:10:11.695427+0000","last_became_peered":"2024-09-10T19:10:11.695427+0000","last_unstale":"2024-09-10T19:22:50.887393+0000","last_undegraded":"2024-09-10T19:22:50.887393+0000","last_fullsized":"2024-09-10T19:22:50.887393+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'8","last_scrub_stamp":"2024-09-10T19:21:48.538741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:21:48.538741+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:22:56.566083+0000","scrub_duration":0.0059444800000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":239,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-10T19:22:49.302983+0000","last_change":"2024-09-10T19:22:38.725243+0000","last_active":"2024-09-10T19:22:49.302983+0000","last_peered":"2024-09-10T19:22:49.302983+0000","last_clean":"2024-09-10T19:22:49.302983+0000","last_became_active":"2024-09-10T19:16:24.862695+0000","last_became_peered":"2024-09-10T19:16:24.862695+0000","last_unstale":"2024-09-10T19:22:49.302983+0000","last_undegraded":"2024-09-10T19:22:49.302983+0000","last_fullsized":"2024-09-10T19:22:49.302983+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:22:38.725165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:22:38.725165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:23:44.009816+0000","scrub_duration":0.0099096830000000007,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":276,"reported_epoch":278,"state":"active+undersized","last_fresh":"2024-09-10T19:22:49.259024+0000","last_change":"2024-09-10T19:21:47.988058+0000","last_active":"2024-09-10T19:22:49.259024+0000","last_peered":"2024-09-10T19:22:49.259024+0000","last_clean":"2024-09-10T19:21:44.676614+0000","last_became_active":"2024-09-10T19:21:47.988058+0000","last_became_peered":"2024-09-10T19:21:47.988058+0000","last_unstale":"2024-09-10T19:22:49.259024+0000","last_undegraded":"2024-09-10T19:22:49.259024+0000","last_fullsized":"2024-09-10T19:21:47.976497+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:21:44.676477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:21:44.676477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:23:05.642583+0000","scrub_duration":0.0017387360000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":["3","6"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"237'24","reported_seq":360,"reported_epoch":278,"state":"active+undersized+degraded","last_fresh":"2024-09-10T19:22:49.543401+0000","last_change":"2024-09-10T19:21:48.002057+0000","last_active":"2024-09-10T19:22:49.543401+0000","last_peered":"2024-09-10T19:22:49.543401+0000","last_clean":"2024-09-10T19:21:43.952583+0000","last_became_active":"2024-09-10T19:21:48.002057+0000","last_became_peered":"2024-09-10T19:21:48.002057+0000","last_unstale":"2024-09-10T19:22:49.543401+0000","last_undegraded":"2024-09-10T19:21:47.987701+0000","last_fullsized":"2024-09-10T19:21:47.985296+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"80'16","last_scrub_stamp":"2024-09-10T19:20:01.500062+0000","last_deep_scrub":"80'16","last_deep_scrub_stamp":"2024-09-10T19:11:40.853611+0000","last_clean_scrub_stamp":"2024-09-10T19:20:01.500062+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.002579296,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":["6","5"],"object_location_counts":[{"shards":"5,6","objects":5}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":147,"reported_epoch":278,"state":"active+undersized","last_fresh":"2024-09-10T19:22:49.297686+0000","last_change":"2024-09-10T19:21:47.989531+0000","last_active":"2024-09-10T19:22:49.297686+0000","last_peered":"2024-09-10T19:22:49.297686+0000","last_clean":"2024-09-10T19:16:23.155700+0000","last_became_active":"2024-09-10T19:21:47.989531+0000","last_became_peered":"2024-09-10T19:21:47.989531+0000","last_unstale":"2024-09-10T19:22:49.297686+0000","last_undegraded":"2024-09-10T19:22:49.297686+0000","last_fullsized":"2024-09-10T19:21:47.982576+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:20:01.699734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:11:28.822327+0000","last_clean_scrub_stamp":"2024-09-10T19:20:01.699734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0022130729999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":["7","3"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":272,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-10T19:22:49.280868+0000","last_change":"2024-09-10T19:22:39.093858+0000","last_active":"2024-09-10T19:22:49.280868+0000","last_peered":"2024-09-10T19:22:49.280868+0000","last_clean":"2024-09-10T19:22:49.280868+0000","last_became_active":"2024-09-10T19:10:14.053619+0000","last_became_peered":"2024-09-10T19:10:14.053619+0000","last_unstale":"2024-09-10T19:22:49.280868+0000","last_undegraded":"2024-09-10T19:22:49.280868+0000","last_fullsized":"2024-09-10T19:22:49.280868+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:22:39.093573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:22:39.093573+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:24:02.502576+0000","scrub_duration":0.010663461000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":279,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-10T19:22:49.544797+0000","last_change":"2024-09-10T19:22:31.719863+0000","last_active":"2024-09-10T19:22:49.544797+0000","last_peered":"2024-09-10T19:22:49.544797+0000","last_clean":"2024-09-10T19:22:49.544797+0000","last_became_active":"2024-09-10T19:10:07.685997+0000","last_became_peered":"2024-09-10T19:10:07.685997+0000","last_unstale":"2024-09-10T19:22:49.544797+0000","last_undegraded":"2024-09-10T19:22:49.544797+0000","last_fullsized":"2024-09-10T19:22:49.544797+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:22:31.719788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:22:31.719788+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:23:31.786150+0000","scrub_duration":0.007228801,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1c","version":"237'34","reported_seq":243,"reported_epoch":278,"state":"active+undersized+degraded","last_fresh":"2024-09-10T19:22:49.258298+0000","last_change":"2024-09-10T19:21:48.003880+0000","last_active":"2024-09-10T19:22:49.258298+0000","last_peered":"2024-09-10T19:22:49.258298+0000","last_clean":"2024-09-10T19:21:43.945320+0000","last_became_active":"2024-09-10T19:21:48.003880+0000","last_became_peered":"2024-09-10T19:21:48.003880+0000","last_unstale":"2024-09-10T19:22:49.258298+0000","last_undegraded":"2024-09-10T19:21:47.973954+0000","last_fullsized":"2024-09-10T19:21:47.973536+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"237'34","last_scrub_stamp":"2024-09-10T19:20:40.452293+0000","last_deep_scrub":"78'22","last_deep_scrub_stamp":"2024-09-10T19:11:21.118937+0000","last_clean_scrub_stamp":"2024-09-10T19:20:40.452293+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0025454940000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":["3","4"],"object_location_counts":[{"shards":"3,4","objects":7}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":229,"reported_epoch":278,"state":"active+undersized","last_fresh":"2024-09-10T19:22:49.298804+0000","last_change":"2024-09-10T19:21:47.981439+0000","last_active":"2024-09-10T19:22:49.298804+0000","last_peered":"2024-09-10T19:22:49.298804+0000","last_clean":"2024-09-10T19:21:43.948694+0000","last_became_active":"2024-09-10T19:21:47.981439+0000","last_became_peered":"2024-09-10T19:21:47.981439+0000","last_unstale":"2024-09-10T19:22:49.298804+0000","last_undegraded":"2024-09-10T19:22:49.298804+0000","last_fullsized":"2024-09-10T19:21:47.972032+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:20:08.342716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:20:08.342716+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.002558444,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":["4","5"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":263,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-10T19:22:49.258011+0000","last_change":"2024-09-10T19:21:57.736056+0000","last_active":"2024-09-10T19:22:49.258011+0000","last_peered":"2024-09-10T19:22:49.258011+0000","last_clean":"2024-09-10T19:22:49.258011+0000","last_became_active":"2024-09-10T19:10:14.053698+0000","last_became_peered":"2024-09-10T19:10:14.053698+0000","last_unstale":"2024-09-10T19:22:49.258011+0000","last_undegraded":"2024-09-10T19:22:49.258011+0000","last_fullsized":"2024-09-10T19:22:49.258011+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:21:57.736006+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:21:57.736006+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:23:27.040783+0000","scrub_duration":0.0017464049999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":269,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-10T19:22:49.258717+0000","last_change":"2024-09-10T19:22:01.763166+0000","last_active":"2024-09-10T19:22:49.258717+0000","last_peered":"2024-09-10T19:22:49.258717+0000","last_clean":"2024-09-10T19:22:49.258717+0000","last_became_active":"2024-09-10T19:10:07.682811+0000","last_became_peered":"2024-09-10T19:10:07.682811+0000","last_unstale":"2024-09-10T19:22:49.258717+0000","last_undegraded":"2024-09-10T19:22:49.258717+0000","last_fullsized":"2024-09-10T19:22:49.258717+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:22:01.763088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:22:01.763088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:23:21.447120+0000","scrub_duration":0.0083058279999999995,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1d","version":"250'23","reported_seq":323,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-10T19:22:49.258171+0000","last_change":"2024-09-10T19:20:55.582263+0000","last_active":"2024-09-10T19:22:49.258171+0000","last_peered":"2024-09-10T19:22:49.258171+0000","last_clean":"2024-09-10T19:22:49.258171+0000","last_became_active":"2024-09-10T19:13:54.714629+0000","last_became_peered":"2024-09-10T19:13:54.714629+0000","last_unstale":"2024-09-10T19:22:49.258171+0000","last_undegraded":"2024-09-10T19:22:49.258171+0000","last_fullsized":"2024-09-10T19:22:49.258171+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"250'23","last_scrub_stamp":"2024-09-10T19:20:55.582173+0000","last_deep_scrub":"250'23","last_deep_scrub_stamp":"2024-09-10T19:20:55.582173+0000","last_clean_scrub_stamp":"2024-09-10T19:20:55.582173+0000","objects_scrubbed":5,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.013381652000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":284,"reported_epoch":278,"state":"active+clean","last_fresh":"2024-09-10T19:22:49.299089+0000","last_change":"2024-09-10T19:22:26.010505+0000","last_active":"2024-09-10T19:22:49.299089+0000","last_peered":"2024-09-10T19:22:49.299089+0000","last_clean":"2024-09-10T19:22:49.299089+0000","last_became_active":"2024-09-10T19:16:24.843013+0000","last_became_peered":"2024-09-10T19:16:24.843013+0000","last_unstale":"2024-09-10T19:22:49.299089+0000","last_undegraded":"2024-09-10T19:22:49.299089+0000","last_fullsized":"2024-09-10T19:22:49.299089+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:22:26.010416+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:22:26.010416+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:23:30.838871+0000","scrub_duration":0.012466878000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":9,"num_pg":16,"stat_sum":{"num_bytes":2638151704,"num_objects":40256,"num_object_clones":0,"num_object_copies":120768,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":20196,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40256,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":252718,"num_write_kb":9375105,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":10740183040,"data_stored":10740170824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":60125,"ondisk_log_size":60125,"up":40,"acting":40,"num_store_stats":6},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":382,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":1528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":82,"acting":82,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":78,"ondisk_log_size":78,"up":77,"acting":77,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":177,"num_object_clones":0,"num_object_copies":531,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":80,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":177,"num_whiteouts":0,"num_read":1119,"num_read_kb":876,"num_write":703,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":178,"num_bytes_recovered":3328,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":540672,"data_stored":14242,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":758,"ondisk_log_size":758,"up":82,"acting":82,"num_store_stats":8},{"poolid":3,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":393,"num_read_kb":388,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":389,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":2002,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":9,"ondisk_log_size":9,"up":7,"acting":7,"num_store_stats":7},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1328,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":27,"num_read_kb":27,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":3984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4,"ondisk_log_size":4,"up":79,"acting":79,"num_store_stats":8},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":644038,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":11,"num_read_kb":20,"num_write":11,"num_write_kb":148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":53670,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":869,"ondisk_log_size":869,"up":29,"acting":29,"num_store_stats":6}],"osd_stats":[{"osd":7,"up_from":54,"seq":231928234142,"num_pgs":83,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2499316,"kb_used_data":1607020,"kb_used_omap":1,"kb_used_meta":892286,"kb_avail":91245836,"statfs":{"total":95995035648,"available":93435736064,"internally_reserved":0,"allocated":1645588480,"data_stored":1645236662,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":913701208},"hb_peers":[1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 19:21:08 2024","interfaces":[{"interface":"back","average":{"1min":0.61299999999999999,"5min":1.046,"15min":1.113},"min":{"1min":0.215,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":2.3620000000000001,"5min":8.3770000000000007,"15min":35.551000000000002},"last":0.91900000000000004},{"interface":"front","average":{"1min":1.879,"5min":1.2789999999999999,"15min":1.131},"min":{"1min":0.24399999999999999,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":5.0140000000000002,"5min":8.1440000000000001,"15min":35.503999999999998},"last":0.97999999999999998}]},{"osd":1,"last update":"Tue Sep 10 19:20:29 2024","interfaces":[{"interface":"back","average":{"1min":0.751,"5min":1.0609999999999999,"15min":0.70299999999999996},"min":{"1min":0.26000000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":2.5089999999999999,"5min":8.3499999999999996,"15min":8.3499999999999996},"last":0.28599999999999998},{"interface":"front","average":{"1min":0.78000000000000003,"5min":1.0209999999999999,"15min":0.81200000000000006},"min":{"1min":0.28799999999999998,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":2.5640000000000001,"5min":5.609,"15min":5.609},"last":0.40899999999999997}]},{"osd":2,"last update":"Tue Sep 10 19:22:09 2024","interfaces":[{"interface":"back","average":{"1min":1.143,"5min":1.0249999999999999,"15min":0.88400000000000001},"min":{"1min":0.27600000000000002,"5min":0.27600000000000002,"15min":0.25600000000000001},"max":{"1min":3.722,"5min":3.77,"15min":31.940999999999999},"last":0.375},{"interface":"front","average":{"1min":0.74299999999999999,"5min":1,"15min":0.95399999999999996},"min":{"1min":0.317,"5min":0.28399999999999997,"15min":0.24099999999999999},"max":{"1min":4.4139999999999997,"5min":5.5410000000000004,"15min":25.056000000000001},"last":0.29899999999999999}]},{"osd":3,"last update":"Tue Sep 10 19:22:09 2024","interfaces":[{"interface":"back","average":{"1min":1.375,"5min":1.3580000000000001,"15min":1.2509999999999999},"min":{"1min":0.23899999999999999,"5min":0.23899999999999999,"15min":0.217},"max":{"1min":4.6070000000000002,"5min":21.074999999999999,"15min":35.277999999999999},"last":0.26900000000000002},{"interface":"front","average":{"1min":0.752,"5min":1.343,"15min":1.2010000000000001},"min":{"1min":0.27400000000000002,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":4.4500000000000002,"5min":18.82,"15min":36.512999999999998},"last":0.36099999999999999}]},{"osd":4,"last update":"Tue Sep 10 19:22:09 2024","interfaces":[{"interface":"back","average":{"1min":1.3360000000000001,"5min":1.9670000000000001,"15min":1.046},"min":{"1min":0.222,"5min":0.222,"15min":0.222},"max":{"1min":5.6269999999999998,"5min":8.4399999999999995,"15min":8.4399999999999995},"last":0.23999999999999999},{"interface":"front","average":{"1min":1.2130000000000001,"5min":1.3680000000000001,"15min":0.82499999999999996},"min":{"1min":0.24399999999999999,"5min":0.23100000000000001,"15min":0.23000000000000001},"max":{"1min":3.8490000000000002,"5min":6.149,"15min":6.149},"last":0.34699999999999998}]},{"osd":5,"last update":"Tue Sep 10 19:22:09 2024","interfaces":[{"interface":"back","average":{"1min":1.248,"5min":1.845,"15min":1.0900000000000001},"min":{"1min":0.30399999999999999,"5min":0.30399999999999999,"15min":0.23699999999999999},"max":{"1min":4.4749999999999996,"5min":33.521000000000001,"15min":33.521000000000001},"last":0.38600000000000001},{"interface":"front","average":{"1min":1.804,"5min":2.2709999999999999,"15min":1.2170000000000001},"min":{"1min":0.39300000000000002,"5min":0.33700000000000002,"15min":0.26000000000000001},"max":{"1min":5.9029999999999996,"5min":34.438000000000002,"15min":34.438000000000002},"last":0.33500000000000002}]},{"osd":6,"last update":"Tue Sep 10 19:22:09 2024","interfaces":[{"interface":"back","average":{"1min":1.6839999999999999,"5min":1.927,"15min":1.113},"min":{"1min":0.48699999999999999,"5min":0.312,"15min":0.26200000000000001},"max":{"1min":5.7119999999999997,"5min":8.3710000000000004,"15min":8.3710000000000004},"last":181.53800000000001},{"interface":"front","average":{"1min":1.466,"5min":1.5189999999999999,"15min":0.93500000000000005},"min":{"1min":0.40200000000000002,"5min":0.29199999999999998,"15min":0.28000000000000003},"max":{"1min":3.9119999999999999,"5min":6.8200000000000003,"15min":8.4130000000000003},"last":182.05600000000001}]}]},{"osd":6,"up_from":47,"seq":201863463075,"num_pgs":78,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1418408,"kb_used_data":832672,"kb_used_omap":1,"kb_used_meta":585726,"kb_avail":92326744,"statfs":{"total":95995035648,"available":94542585856,"internally_reserved":0,"allocated":852656128,"data_stored":852317194,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":599783768},"hb_peers":[1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,2],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 19:20:51 2024","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":0.69899999999999995,"15min":1.1279999999999999},"min":{"1min":0.23100000000000001,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":2.2010000000000001,"5min":5.2350000000000003,"15min":112.988},"last":2.3159999999999998},{"interface":"front","average":{"1min":0.59799999999999998,"5min":0.90500000000000003,"15min":1.1990000000000001},"min":{"1min":0.221,"5min":0.221,"15min":0.221},"max":{"1min":2.0009999999999999,"5min":5.274,"15min":113.18300000000001},"last":0.41899999999999998}]},{"osd":1,"last update":"Tue Sep 10 19:20:35 2024","interfaces":[{"interface":"back","average":{"1min":0.51800000000000002,"5min":0.70999999999999996,"15min":0.57899999999999996},"min":{"1min":0.31900000000000001,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":0.995,"5min":4.2919999999999998,"15min":4.2919999999999998},"last":0.45000000000000001},{"interface":"front","average":{"1min":0.59899999999999998,"5min":0.91000000000000003,"15min":0.73899999999999999},"min":{"1min":0.29699999999999999,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":2.0750000000000002,"5min":6.5289999999999999,"15min":6.5289999999999999},"last":0.35999999999999999}]},{"osd":2,"last update":"Tue Sep 10 19:22:53 2024","interfaces":[{"interface":"back","average":{"1min":0.54700000000000004,"5min":0.83099999999999996,"15min":1.129},"min":{"1min":0.23799999999999999,"5min":0.23799999999999999,"15min":0.20399999999999999},"max":{"1min":1.5509999999999999,"5min":3.6099999999999999,"15min":112.864},"last":0.46200000000000002},{"interface":"front","average":{"1min":0.97299999999999998,"5min":0.97999999999999998,"15min":1.2350000000000001},"min":{"1min":0.25600000000000001,"5min":0.25600000000000001,"15min":0.254},"max":{"1min":4.2610000000000001,"5min":5.8300000000000001,"15min":112.822},"last":0.63200000000000001}]},{"osd":3,"last update":"Tue Sep 10 19:22:53 2024","interfaces":[{"interface":"back","average":{"1min":0.55200000000000005,"5min":0.96299999999999997,"15min":1.411},"min":{"1min":0.24099999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":1.6080000000000001,"5min":6.4870000000000001,"15min":125.56399999999999},"last":0.40100000000000002},{"interface":"front","average":{"1min":0.53300000000000003,"5min":0.91100000000000003,"15min":1.429},"min":{"1min":0.20499999999999999,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":1.522,"5min":6.5259999999999998,"15min":125.47},"last":0.32800000000000001}]},{"osd":4,"last update":"Tue Sep 10 19:22:53 2024","interfaces":[{"interface":"back","average":{"1min":1.3029999999999999,"5min":1.5129999999999999,"15min":0.91100000000000003},"min":{"1min":0.29599999999999999,"5min":0.253,"15min":0.20699999999999999},"max":{"1min":4.2240000000000002,"5min":6.9269999999999996,"15min":6.9269999999999996},"last":0.376},{"interface":"front","average":{"1min":1.165,"5min":1.2729999999999999,"15min":0.82099999999999995},"min":{"1min":0.28299999999999997,"5min":0.24099999999999999,"15min":0.22500000000000001},"max":{"1min":3.714,"5min":6.907,"15min":6.907},"last":0.42899999999999999}]},{"osd":5,"last update":"Tue Sep 10 19:22:53 2024","interfaces":[{"interface":"back","average":{"1min":1.278,"5min":1.532,"15min":1.0980000000000001},"min":{"1min":0.29299999999999998,"5min":0.29299999999999998,"15min":0.248},"max":{"1min":4.5519999999999996,"5min":5.9640000000000004,"15min":44.765000000000001},"last":0.85499999999999998},{"interface":"front","average":{"1min":1.3089999999999999,"5min":1.5489999999999999,"15min":1.157},"min":{"1min":0.29599999999999999,"5min":0.28399999999999997,"15min":0.27500000000000002},"max":{"1min":3.7629999999999999,"5min":7.1369999999999996,"15min":43.036000000000001},"last":1.28}]},{"osd":7,"last update":"Tue Sep 10 19:22:18 2024","interfaces":[{"interface":"back","average":{"1min":0.98099999999999998,"5min":1.1200000000000001,"15min":0.79000000000000004},"min":{"1min":0.30099999999999999,"5min":0.29899999999999999,"15min":0.27500000000000002},"max":{"1min":2.6200000000000001,"5min":4.1379999999999999,"15min":8.9390000000000001},"last":1.786},{"interface":"front","average":{"1min":1.6140000000000001,"5min":1.5269999999999999,"15min":0.97399999999999998},"min":{"1min":0.25700000000000001,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":6.016,"5min":6.9720000000000004,"15min":10.298},"last":1.73}]}]},{"osd":5,"up_from":39,"seq":167503724712,"num_pgs":80,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2649248,"kb_used_data":1759576,"kb_used_omap":1,"kb_used_meta":889598,"kb_avail":91095904,"statfs":{"total":95995035648,"available":93282205696,"internally_reserved":0,"allocated":1801805824,"data_stored":1801475837,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":910948696},"hb_peers":[1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 19:21:23 2024","interfaces":[{"interface":"back","average":{"1min":1.1579999999999999,"5min":1.9570000000000001,"15min":1.1060000000000001},"min":{"1min":0.19600000000000001,"5min":0.17999999999999999,"15min":0.17999999999999999},"max":{"1min":3.9009999999999998,"5min":73.933000000000007,"15min":73.933000000000007},"last":4.4960000000000004},{"interface":"front","average":{"1min":1.286,"5min":1.821,"15min":1.024},"min":{"1min":0.23100000000000001,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":3.9969999999999999,"5min":73.769999999999996,"15min":73.769999999999996},"last":3.2749999999999999}]},{"osd":1,"last update":"Tue Sep 10 19:20:27 2024","interfaces":[{"interface":"back","average":{"1min":0.73299999999999998,"5min":0.98499999999999999,"15min":0.76300000000000001},"min":{"1min":0.23300000000000001,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":2.2050000000000001,"5min":4.3499999999999996,"15min":4.3499999999999996},"last":0.51200000000000001},{"interface":"front","average":{"1min":0.875,"5min":0.97799999999999998,"15min":0.66100000000000003},"min":{"1min":0.28999999999999998,"5min":0.255,"15min":0.255},"max":{"1min":4.742,"5min":5.0570000000000004,"15min":5.0570000000000004},"last":0.42499999999999999}]},{"osd":2,"last update":"Tue Sep 10 19:22:23 2024","interfaces":[{"interface":"back","average":{"1min":1.542,"5min":1.1970000000000001,"15min":0.91700000000000004},"min":{"1min":0.22900000000000001,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":8.6920000000000002,"5min":8.6920000000000002,"15min":9.9399999999999995},"last":0.45200000000000001},{"interface":"front","average":{"1min":1.2649999999999999,"5min":1.1699999999999999,"15min":0.90200000000000002},"min":{"1min":0.21199999999999999,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":3.3199999999999998,"5min":4.4139999999999997,"15min":6.3789999999999996},"last":0.56599999999999995}]},{"osd":3,"last update":"Tue Sep 10 19:22:23 2024","interfaces":[{"interface":"back","average":{"1min":1.5409999999999999,"5min":1.1699999999999999,"15min":0.90700000000000003},"min":{"1min":0.19900000000000001,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":8.6660000000000004,"5min":8.6660000000000004,"15min":9.9019999999999992},"last":0.39600000000000002},{"interface":"front","average":{"1min":1.282,"5min":1.222,"15min":0.88900000000000001},"min":{"1min":0.25,"5min":0.25,"15min":0.248},"max":{"1min":3.302,"5min":4.4459999999999997,"15min":5.3869999999999996},"last":0.58999999999999997}]},{"osd":4,"last update":"Tue Sep 10 19:22:23 2024","interfaces":[{"interface":"back","average":{"1min":1.768,"5min":1.4319999999999999,"15min":0.92300000000000004},"min":{"1min":0.30099999999999999,"5min":0.253,"15min":0.20699999999999999},"max":{"1min":6.0970000000000004,"5min":7.4390000000000001,"15min":10.414},"last":2.1890000000000001},{"interface":"front","average":{"1min":1.9410000000000001,"5min":1.5449999999999999,"15min":0.996},"min":{"1min":0.26400000000000001,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":8.6219999999999999,"5min":8.6219999999999999,"15min":10.518000000000001},"last":1.069}]},{"osd":6,"last update":"Tue Sep 10 19:22:51 2024","interfaces":[{"interface":"back","average":{"1min":7.1180000000000003,"5min":2.6619999999999999,"15min":1.482},"min":{"1min":0.317,"5min":0.28399999999999997,"15min":0.223},"max":{"1min":111.328,"5min":111.328,"15min":111.328},"last":1.367},{"interface":"front","average":{"1min":7.0640000000000001,"5min":2.669,"15min":1.4179999999999999},"min":{"1min":0.28799999999999998,"5min":0.28799999999999998,"15min":0.25},"max":{"1min":111.705,"5min":111.705,"15min":111.705},"last":0.60899999999999999}]},{"osd":7,"last update":"Tue Sep 10 19:22:17 2024","interfaces":[{"interface":"back","average":{"1min":1.925,"5min":1.657,"15min":1.012},"min":{"1min":0.38500000000000001,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":6.7869999999999999,"5min":6.7869999999999999,"15min":6.7869999999999999},"last":0.627},{"interface":"front","average":{"1min":2.0870000000000002,"5min":1.7749999999999999,"15min":1.002},"min":{"1min":0.36899999999999999,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":5.4359999999999999,"5min":9.4909999999999997,"15min":9.4909999999999997},"last":0.72599999999999998}]}]},{"osd":4,"up_from":32,"seq":137438953644,"num_pgs":71,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1725464,"kb_used_data":832912,"kb_used_omap":1,"kb_used_meta":892542,"kb_avail":92019688,"statfs":{"total":95995035648,"available":94228160512,"internally_reserved":0,"allocated":852901888,"data_stored":852577795,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":913963352},"hb_peers":[1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 19:21:10 2024","interfaces":[{"interface":"back","average":{"1min":1.0209999999999999,"5min":1.224,"15min":0.97999999999999998},"min":{"1min":0.25700000000000001,"5min":0.187,"15min":0.187},"max":{"1min":5.1420000000000003,"5min":24.113,"15min":32.533000000000001},"last":0.53100000000000003},{"interface":"front","average":{"1min":0.82799999999999996,"5min":1.1739999999999999,"15min":0.98099999999999998},"min":{"1min":0.20399999999999999,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":3.3100000000000001,"5min":24.132999999999999,"15min":32.512999999999998},"last":1.6160000000000001}]},{"osd":1,"last update":"Tue Sep 10 19:20:27 2024","interfaces":[{"interface":"back","average":{"1min":0.46700000000000003,"5min":0.91900000000000004,"15min":0.72499999999999998},"min":{"1min":0.23200000000000001,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":1.1779999999999999,"5min":7.9770000000000003,"15min":7.9770000000000003},"last":1.3759999999999999},{"interface":"front","average":{"1min":0.78700000000000003,"5min":0.85599999999999998,"15min":0.73599999999999999},"min":{"1min":0.32300000000000001,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":2.3700000000000001,"5min":3.8260000000000001,"15min":3.8260000000000001},"last":0.63900000000000001}]},{"osd":2,"last update":"Tue Sep 10 19:22:13 2024","interfaces":[{"interface":"back","average":{"1min":0.69399999999999995,"5min":0.876,"15min":0.76200000000000001},"min":{"1min":0.25600000000000001,"5min":0.254,"15min":0.214},"max":{"1min":2.5779999999999998,"5min":5.048,"15min":6.8869999999999996},"last":1.355},{"interface":"front","average":{"1min":1.1699999999999999,"5min":0.98299999999999998,"15min":0.77800000000000002},"min":{"1min":0.26700000000000002,"5min":0.253,"15min":0.23400000000000001},"max":{"1min":3.5030000000000001,"5min":4.7169999999999996,"15min":7.085},"last":0.68600000000000005}]},{"osd":3,"last update":"Tue Sep 10 19:22:13 2024","interfaces":[{"interface":"back","average":{"1min":1.1619999999999999,"5min":1.2370000000000001,"15min":0.99199999999999999},"min":{"1min":0.20999999999999999,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":3.5289999999999999,"5min":15.436,"15min":35.451999999999998},"last":0.66300000000000003},{"interface":"front","average":{"1min":0.88100000000000001,"5min":1.323,"15min":1.048},"min":{"1min":0.247,"5min":0.247,"15min":0.247},"max":{"1min":3.5059999999999998,"5min":14.954000000000001,"15min":32.880000000000003},"last":0.71099999999999997}]},{"osd":5,"last update":"Tue Sep 10 19:22:31 2024","interfaces":[{"interface":"back","average":{"1min":13.792,"5min":4.0039999999999996,"15min":1.994},"min":{"1min":0.39600000000000002,"5min":0.224,"15min":0.224},"max":{"1min":227.726,"5min":227.726,"15min":227.726},"last":0.90400000000000003},{"interface":"front","average":{"1min":13.819000000000001,"5min":3.952,"15min":1.925},"min":{"1min":0.20699999999999999,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":227.70500000000001,"5min":227.70500000000001,"15min":227.70500000000001},"last":1.323}]},{"osd":6,"last update":"Tue Sep 10 19:22:06 2024","interfaces":[{"interface":"back","average":{"1min":1.2250000000000001,"5min":1.27,"15min":0.79900000000000004},"min":{"1min":0.27800000000000002,"5min":0.27800000000000002,"15min":0.253},"max":{"1min":4.069,"5min":5.5289999999999999,"15min":5.5289999999999999},"last":1.3959999999999999},{"interface":"front","average":{"1min":0.90000000000000002,"5min":1.3240000000000001,"15min":0.79500000000000004},"min":{"1min":0.26800000000000002,"5min":0.247,"15min":0.247},"max":{"1min":3.5499999999999998,"5min":7.9550000000000001,"15min":7.9550000000000001},"last":4.476}]},{"osd":7,"last update":"Tue Sep 10 19:22:18 2024","interfaces":[{"interface":"back","average":{"1min":1.7949999999999999,"5min":1.6020000000000001,"15min":1.0569999999999999},"min":{"1min":0.311,"5min":0.29899999999999999,"15min":0.27700000000000002},"max":{"1min":5.9809999999999999,"5min":10.33,"15min":23.081},"last":4.6630000000000003},{"interface":"front","average":{"1min":2.2050000000000001,"5min":1.5860000000000001,"15min":1.0580000000000001},"min":{"1min":0.32200000000000001,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":6.4729999999999999,"5min":8.8550000000000004,"15min":22.666},"last":0.60799999999999998}]}]},{"osd":3,"up_from":26,"seq":111669149873,"num_pgs":84,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2390008,"kb_used_data":1489712,"kb_used_omap":1,"kb_used_meta":900286,"kb_avail":91355144,"statfs":{"total":95995035648,"available":93547667456,"internally_reserved":0,"allocated":1525465088,"data_stored":1525110910,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":921893208},"hb_peers":[1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 19:20:46 2024","interfaces":[{"interface":"back","average":{"1min":0.40799999999999997,"5min":0.39500000000000002,"15min":0.69099999999999995},"min":{"1min":0.16700000000000001,"5min":0.157,"15min":0.157},"max":{"1min":0.77800000000000002,"5min":3.9529999999999998,"15min":17.518000000000001},"last":0.39000000000000001},{"interface":"front","average":{"1min":0.72499999999999998,"5min":0.63900000000000001,"15min":0.84799999999999998},"min":{"1min":0.19400000000000001,"5min":0.17899999999999999,"15min":0.17399999999999999},"max":{"1min":2.3969999999999998,"5min":5.1539999999999999,"15min":17.609000000000002},"last":0.40799999999999997}]},{"osd":1,"last update":"Tue Sep 10 19:20:25 2024","interfaces":[{"interface":"back","average":{"1min":0.43099999999999999,"5min":0.45600000000000002,"15min":0.51200000000000001},"min":{"1min":0.222,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":1.2869999999999999,"5min":2.6059999999999999,"15min":2.6059999999999999},"last":0.26300000000000001},{"interface":"front","average":{"1min":0.438,"5min":0.72399999999999998,"15min":1.2869999999999999},"min":{"1min":0.248,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":1.1919999999999999,"5min":17.454999999999998,"15min":17.454999999999998},"last":0.35099999999999998}]},{"osd":2,"last update":"Tue Sep 10 19:22:51 2024","interfaces":[{"interface":"back","average":{"1min":0.34300000000000003,"5min":0.438,"15min":0.64900000000000002},"min":{"1min":0.17599999999999999,"5min":0.17599999999999999,"15min":0.17599999999999999},"max":{"1min":2.2530000000000001,"5min":2.2530000000000001,"15min":5.3029999999999999},"last":0.29899999999999999},{"interface":"front","average":{"1min":0.245,"5min":0.39100000000000001,"15min":0.70599999999999996},"min":{"1min":0.16400000000000001,"5min":0.16400000000000001,"15min":0.16400000000000001},"max":{"1min":0.372,"5min":2.5649999999999999,"15min":17.806000000000001},"last":0.372}]},{"osd":4,"last update":"Tue Sep 10 19:22:11 2024","interfaces":[{"interface":"back","average":{"1min":0.93000000000000005,"5min":0.89200000000000002,"15min":0.92000000000000004},"min":{"1min":0.26900000000000002,"5min":0.254,"15min":0.23599999999999999},"max":{"1min":2.7229999999999999,"5min":5.2750000000000004,"15min":17.834},"last":0.33800000000000002},{"interface":"front","average":{"1min":1.1899999999999999,"5min":0.97299999999999998,"15min":1.0629999999999999},"min":{"1min":0.22700000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":3.1909999999999998,"5min":3.339,"15min":17.553000000000001},"last":1.1379999999999999}]},{"osd":5,"last update":"Tue Sep 10 19:22:29 2024","interfaces":[{"interface":"back","average":{"1min":1.542,"5min":1.272,"15min":1.29},"min":{"1min":0.309,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":6.415,"5min":7.2949999999999999,"15min":80.942999999999998},"last":0.28100000000000003},{"interface":"front","average":{"1min":1.3600000000000001,"5min":1.1180000000000001,"15min":1.3879999999999999},"min":{"1min":0.24299999999999999,"5min":0.24299999999999999,"15min":0.22},"max":{"1min":4.4779999999999998,"5min":4.835,"15min":81.489000000000004},"last":0.38600000000000001}]},{"osd":6,"last update":"Tue Sep 10 19:21:53 2024","interfaces":[{"interface":"back","average":{"1min":1.177,"5min":1.097,"15min":1.0129999999999999},"min":{"1min":0.27900000000000003,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":6.109,"5min":6.109,"15min":17.861000000000001},"last":0.40100000000000002},{"interface":"front","average":{"1min":1.093,"5min":0.95299999999999996,"15min":0.88400000000000001},"min":{"1min":0.28799999999999998,"5min":0.28799999999999998,"15min":0.26800000000000002},"max":{"1min":3.8039999999999998,"5min":3.8039999999999998,"15min":17.687000000000001},"last":0.46899999999999997}]},{"osd":7,"last update":"Tue Sep 10 19:22:23 2024","interfaces":[{"interface":"back","average":{"1min":1.379,"5min":1.1759999999999999,"15min":1.02},"min":{"1min":0.26100000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":7.8769999999999998,"5min":7.8769999999999998,"15min":18.093},"last":0.65400000000000003},{"interface":"front","average":{"1min":0.73999999999999999,"5min":1.109,"15min":1.036},"min":{"1min":0.26100000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":1.796,"5min":10.359999999999999,"15min":17.780999999999999},"last":1.2290000000000001}]}]},{"osd":2,"up_from":0,"seq":77309411510,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":0,"seq":1185410973699,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":8,"seq":34359738546,"num_pgs":78,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":4950592,"kb_used_data":4059380,"kb_used_omap":1,"kb_used_meta":891198,"kb_avail":88794560,"statfs":{"total":95995035648,"available":90925629440,"internally_reserved":0,"allocated":4156805120,"data_stored":4156266196,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":912587096},"hb_peers":[2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Sep 10 19:20:34 2024","interfaces":[{"interface":"back","average":{"1min":0.50800000000000001,"5min":0.438,"15min":0.39200000000000002},"min":{"1min":0.20999999999999999,"5min":0.19,"15min":0.19},"max":{"1min":1.6799999999999999,"5min":2.0819999999999999,"15min":2.0819999999999999},"last":0.54300000000000004},{"interface":"front","average":{"1min":0.38600000000000001,"5min":0.40899999999999997,"15min":0.41899999999999998},"min":{"1min":0.19900000000000001,"5min":0.193,"15min":0.193},"max":{"1min":0.622,"5min":2.2839999999999998,"15min":2.2839999999999998},"last":0.32900000000000001}]},{"osd":2,"last update":"Tue Sep 10 19:21:39 2024","interfaces":[{"interface":"back","average":{"1min":0.46800000000000003,"5min":0.48999999999999999,"15min":0.629},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.17199999999999999},"max":{"1min":2.0840000000000001,"5min":2.0840000000000001,"15min":6.4640000000000004},"last":0.35199999999999998},{"interface":"front","average":{"1min":0.41399999999999998,"5min":0.46500000000000002,"15min":0.60999999999999999},"min":{"1min":0.22500000000000001,"5min":0.22500000000000001,"15min":0.187},"max":{"1min":1.2090000000000001,"5min":2.8660000000000001,"15min":9.9399999999999995},"last":0.41099999999999998}]},{"osd":3,"last update":"Tue Sep 10 19:20:49 2024","interfaces":[{"interface":"back","average":{"1min":0.51900000000000002,"5min":0.58499999999999996,"15min":0.68000000000000005},"min":{"1min":0.29199999999999998,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":1.0129999999999999,"5min":3.6190000000000002,"15min":6.5590000000000002},"last":0.29699999999999999},{"interface":"front","average":{"1min":0.39200000000000002,"5min":0.44,"15min":0.70399999999999996},"min":{"1min":0.23899999999999999,"5min":0.23899999999999999,"15min":0.23000000000000001},"max":{"1min":0.69799999999999995,"5min":1.2649999999999999,"15min":7.2140000000000004},"last":0.22}]},{"osd":4,"last update":"Tue Sep 10 19:21:07 2024","interfaces":[{"interface":"back","average":{"1min":1.0269999999999999,"5min":1.024,"15min":0.79000000000000004},"min":{"1min":0.29299999999999998,"5min":0.29099999999999998,"15min":0.24399999999999999},"max":{"1min":2.2829999999999999,"5min":7.4560000000000004,"15min":7.4560000000000004},"last":0.438},{"interface":"front","average":{"1min":0.877,"5min":0.96099999999999997,"15min":0.77800000000000002},"min":{"1min":0.31,"5min":0.28499999999999998,"15min":0.27100000000000002},"max":{"1min":2.6080000000000001,"5min":5.2640000000000002,"15min":5.2640000000000002},"last":0.61099999999999999}]},{"osd":5,"last update":"Tue Sep 10 19:21:39 2024","interfaces":[{"interface":"back","average":{"1min":0.82099999999999995,"5min":1.036,"15min":0.85299999999999998},"min":{"1min":0.28199999999999997,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":2.7090000000000001,"5min":4.327,"15min":5.3860000000000001},"last":0.69399999999999995},{"interface":"front","average":{"1min":0.95399999999999996,"5min":0.91700000000000004,"15min":0.80000000000000004},"min":{"1min":0.27700000000000002,"5min":0.27700000000000002,"15min":0.253},"max":{"1min":3.871,"5min":4.165,"15min":5.9749999999999996},"last":0.26700000000000002}]},{"osd":6,"last update":"Tue Sep 10 19:20:49 2024","interfaces":[{"interface":"back","average":{"1min":0.72099999999999997,"5min":0.92100000000000004,"15min":0.80700000000000005},"min":{"1min":0.32000000000000001,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":1.512,"5min":4.9420000000000002,"15min":5.8730000000000002},"last":0.60099999999999998},{"interface":"front","average":{"1min":0.91500000000000004,"5min":0.92700000000000005,"15min":0.82099999999999995},"min":{"1min":0.34300000000000003,"5min":0.32900000000000001,"15min":0.28699999999999998},"max":{"1min":3.6850000000000001,"5min":4.8300000000000001,"15min":5.2850000000000001},"last":0.65600000000000003}]},{"osd":7,"last update":"Tue Sep 10 19:21:19 2024","interfaces":[{"interface":"back","average":{"1min":1.369,"5min":1.0549999999999999,"15min":0.86099999999999999},"min":{"1min":0.317,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":3.1579999999999999,"5min":5.1479999999999997,"15min":5.1479999999999997},"last":0.71999999999999997},{"interface":"front","average":{"1min":1.2470000000000001,"5min":1.0329999999999999,"15min":0.83599999999999997},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.30499999999999999},"max":{"1min":4.5720000000000001,"5min":5.3410000000000002,"15min":6.0469999999999997},"last":1.411}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":966,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1236,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1236,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":362,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":1760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":2118,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1018,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":2228,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1942,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":2162,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4143579136,"data_stored":4143579136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1479409664,"data_stored":1479409664,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":836177920,"data_stored":836173848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1798377472,"data_stored":1798373400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":841154560,"data_stored":841154560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1641484288,"data_stored":1641480216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T19:22:57.078 INFO:tasks.cephadm.ceph_manager.ceph:would drop below min_pgs, proj 6, currently 16 2024-09-10T19:22:57.149 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:22:57.171 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:22:57.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:22:57.173 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:22:57.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:22:56 smithi191 ceph-mon[32884]: pgmap v575: 158 pgs: 45 active+undersized, 33 active+undersized+degraded, 80 active+clean; 2.5 GiB data, 15 GiB used, 522 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.56k op/s; 20288/121359 objects degraded (16.717%) 2024-09-10T19:22:57.273 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:22:57.317 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:22:57.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:22:57.318 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:22:57.419 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:22:57.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:22:57.461 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:22:57.461 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:22:57.562 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:22:57.586 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:22:57.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:22:57.587 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:22:57.688 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:22:57.716 INFO:journalctl@ceph.mgr.x.smithi191.stdout:Sep 10 19:22:57 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-x[34967]: ::ffff:172.21.15.191 - - [10/Sep/2024:19:22:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T19:22:57.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:22:57.727 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:22:57.727 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:22:57.782 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:22:57.782 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:22:57.815 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph daemon osd.1 dump_blocked_ops 2024-09-10T19:22:57.828 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:22:57.883 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:22:57.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:22:57.884 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:22:57.985 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:22:58.028 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:22:58.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:22:58.029 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:22:58.120 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:22:57 smithi183 ceph-mon[29736]: from='client.15585 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-10T19:22:58.120 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:22:57 smithi183 ceph-mon[41562]: from='client.15585 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-10T19:22:58.130 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:22:58.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:22:58.168 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:22:58.168 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:22:58.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:22:57 smithi191 ceph-mon[32884]: from='client.15585 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-10T19:22:58.269 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:22:58.291 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:22:58.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:22:58.292 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:22:58.392 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:22:58.430 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:22:58.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:22:58.431 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:22:58.532 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:22:58.558 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:22:58.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:22:58.560 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:22:58.661 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:22:58.694 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:22:58.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:22:58.695 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:22:58.796 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:22:58.828 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:22:58.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:22:58.829 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:22:58.930 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:22:58.952 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:22:58.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:22:58.953 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:22:59.054 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:22:59.077 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:22:59.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:22:59.078 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:22:59.179 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:22:59.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:22:58 smithi183 ceph-mon[29736]: pgmap v576: 158 pgs: 45 active+undersized, 33 active+undersized+degraded, 80 active+clean; 2.5 GiB data, 15 GiB used, 522 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.46k op/s; 20288/121359 objects degraded (16.717%) 2024-09-10T19:22:59.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:22:58 smithi183 ceph-mon[29736]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:22:59.181 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:22:58 smithi183 ceph-mon[41562]: pgmap v576: 158 pgs: 45 active+undersized, 33 active+undersized+degraded, 80 active+clean; 2.5 GiB data, 15 GiB used, 522 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.46k op/s; 20288/121359 objects degraded (16.717%) 2024-09-10T19:22:59.181 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:22:58 smithi183 ceph-mon[41562]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:22:59.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:22:58 smithi191 ceph-mon[32884]: pgmap v576: 158 pgs: 45 active+undersized, 33 active+undersized+degraded, 80 active+clean; 2.5 GiB data, 15 GiB used, 522 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.46k op/s; 20288/121359 objects degraded (16.717%) 2024-09-10T19:22:59.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:22:58 smithi191 ceph-mon[32884]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:22:59.218 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:22:59.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:22:59.219 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:22:59.306 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:22:59.319 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:22:59.352 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:22:59.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:22:59.353 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:22:59.454 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:22:59.489 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:22:59.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:22:59.490 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:22:59.591 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:22:59.624 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:22:59.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:22:59.625 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:22:59.716 INFO:journalctl@ceph.iscsi.iscsi.a.smithi191.stdout:Sep 10 19:22:59 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-iscsi-iscsi-a[61816]: debug there is no tcmu-runner data available 2024-09-10T19:22:59.726 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:22:59.751 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:22:59.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:22:59.752 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:22:59.853 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:22:59.886 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:22:59.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:22:59.887 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:22:59.988 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:23:00.000 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:22:59 smithi183 ceph-mon[29736]: Health check update: Degraded data redundancy: 20288/121359 objects degraded (16.717%), 33 pgs degraded, 78 pgs undersized (PG_DEGRADED) 2024-09-10T19:23:00.000 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:22:59 smithi183 ceph-mon[41562]: Health check update: Degraded data redundancy: 20288/121359 objects degraded (16.717%), 33 pgs degraded, 78 pgs undersized (PG_DEGRADED) 2024-09-10T19:23:00.012 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:23:00.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:00.014 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:23:00.114 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:23:00.137 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:23:00.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:00.138 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:23:00.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:22:59 smithi191 ceph-mon[32884]: Health check update: Degraded data redundancy: 20288/121359 objects degraded (16.717%), 33 pgs degraded, 78 pgs undersized (PG_DEGRADED) 2024-09-10T19:23:00.239 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:23:00.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:00.276 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:23:00.277 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:23:00.377 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:23:00.401 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:23:00.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:00.401 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:23:00.502 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:23:00.528 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:23:00.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:00.529 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:23:00.630 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:23:00.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:00.665 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:23:00.665 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:23:00.766 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:23:00.789 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:23:00.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:00.790 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:23:00.891 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:23:00.930 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:23:00.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:00.931 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:23:01.032 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:23:01.063 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:23:01.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:01.064 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:23:01.164 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:23:01.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:23:00 smithi183 ceph-mon[29736]: from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:23:01.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:23:00 smithi183 ceph-mon[29736]: pgmap v577: 158 pgs: 45 active+undersized, 33 active+undersized+degraded, 80 active+clean; 2.5 GiB data, 15 GiB used, 522 GiB / 536 GiB avail; 1023 B/s rd, 0 B/s wr, 1.25k op/s; 20288/121359 objects degraded (16.717%) 2024-09-10T19:23:01.181 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:23:00 smithi183 ceph-mon[41562]: from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:23:01.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:23:00 smithi183 ceph-mon[41562]: pgmap v577: 158 pgs: 45 active+undersized, 33 active+undersized+degraded, 80 active+clean; 2.5 GiB data, 15 GiB used, 522 GiB / 536 GiB avail; 1023 B/s rd, 0 B/s wr, 1.25k op/s; 20288/121359 objects degraded (16.717%) 2024-09-10T19:23:01.189 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:23:01.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:01.190 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:23:01.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:23:00 smithi191 ceph-mon[32884]: from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:23:01.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:23:00 smithi191 ceph-mon[32884]: pgmap v577: 158 pgs: 45 active+undersized, 33 active+undersized+degraded, 80 active+clean; 2.5 GiB data, 15 GiB used, 522 GiB / 536 GiB avail; 1023 B/s rd, 0 B/s wr, 1.25k op/s; 20288/121359 objects degraded (16.717%) 2024-09-10T19:23:01.291 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:23:01.330 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:23:01.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:01.331 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:23:01.432 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:23:01.454 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:23:01.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:01.455 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:23:01.555 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:23:01.589 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:23:01.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:01.590 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:23:01.681 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Sep 10 19:23:01 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-y[29989]: ::ffff:172.21.15.191 - - [10/Sep/2024:19:23:01] "GET /metrics HTTP/1.1" 200 224308 "" "Prometheus/2.43.0" 2024-09-10T19:23:01.690 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:23:01.713 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:23:01.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:01.714 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:23:01.815 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:23:01.838 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:23:01.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:01.840 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:23:01.940 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:23:01.964 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:23:01.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:01.965 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:23:02.065 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:23:02.079 INFO:tasks.thrashosds.thrasher:in_osds: [0, 3, 4, 5, 6, 7] out_osds: [1, 2] dead_osds: [0] live_osds: [2, 3, 4, 5, 6, 7, 1] 2024-09-10T19:23:02.079 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-10T19:23:02.079 INFO:tasks.thrashosds.thrasher:inject_pause on osd.7 2024-09-10T19:23:02.079 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-09-10T19:23:02.079 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-09-10T19:23:02.079 DEBUG:teuthology.orchestra.run.smithi191:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph daemon osd.7 config set filestore_inject_stall 3 2024-09-10T19:23:02.091 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:23:02.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:02.092 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:23:02.193 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:23:02.214 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:23:02.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:02.215 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:23:02.316 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:23:02.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:02.376 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:23:02.376 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:23:02.477 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:23:02.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:02.517 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:23:02.517 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:23:02.524 INFO:teuthology.orchestra.run.smithi191.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.b/config 2024-09-10T19:23:02.617 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:23:02.664 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:23:02.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:02.665 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:23:02.765 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:23:02.790 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:23:02.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:02.791 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:23:02.892 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:23:02.913 INFO:teuthology.orchestra.run.smithi183.stdout:osd.1: {} 2024-09-10T19:23:02.913 INFO:teuthology.orchestra.run.smithi183.stdout:osd.2: {} 2024-09-10T19:23:02.913 INFO:teuthology.orchestra.run.smithi183.stdout:osd.3: {} 2024-09-10T19:23:02.913 INFO:teuthology.orchestra.run.smithi183.stderr:Error ENXIO: problem getting command descriptions from osd.0 2024-09-10T19:23:02.913 INFO:teuthology.orchestra.run.smithi183.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-10T19:23:02.913 INFO:teuthology.orchestra.run.smithi183.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-10T19:23:02.913 INFO:teuthology.orchestra.run.smithi183.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-10T19:23:02.913 INFO:teuthology.orchestra.run.smithi183.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-10T19:23:02.914 INFO:teuthology.orchestra.run.smithi183.stdout:osd.4: {} 2024-09-10T19:23:02.914 INFO:teuthology.orchestra.run.smithi183.stdout:osd.5: {} 2024-09-10T19:23:02.914 INFO:teuthology.orchestra.run.smithi183.stdout:osd.6: {} 2024-09-10T19:23:02.914 INFO:teuthology.orchestra.run.smithi183.stdout:osd.7: {} 2024-09-10T19:23:02.914 INFO:teuthology.orchestra.run.smithi183.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-10T19:23:02.914 INFO:teuthology.orchestra.run.smithi183.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-10T19:23:02.914 INFO:teuthology.orchestra.run.smithi183.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-10T19:23:02.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:02.915 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:23:02.916 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:23:03.016 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:23:03.061 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:23:03.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:03.062 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:23:03.162 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:23:03.181 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:23:02 smithi183 ceph-mon[41562]: pgmap v578: 158 pgs: 45 active+undersized, 33 active+undersized+degraded, 80 active+clean; 2.0 GiB data, 14 GiB used, 523 GiB / 536 GiB avail; 969 B/s rd, 0 B/s wr, 1.36k op/s; 16834/100677 objects degraded (16.721%) 2024-09-10T19:23:03.181 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:23:02 smithi183 ceph-mon[41562]: from='client.? 172.21.15.183:0/4217325711' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T19:23:03.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:23:02 smithi183 ceph-mon[29736]: pgmap v578: 158 pgs: 45 active+undersized, 33 active+undersized+degraded, 80 active+clean; 2.0 GiB data, 14 GiB used, 523 GiB / 536 GiB avail; 969 B/s rd, 0 B/s wr, 1.36k op/s; 16834/100677 objects degraded (16.721%) 2024-09-10T19:23:03.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:23:02 smithi183 ceph-mon[29736]: from='client.? 172.21.15.183:0/4217325711' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T19:23:03.199 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:23:03.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:03.200 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:23:03.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:23:02 smithi191 ceph-mon[32884]: pgmap v578: 158 pgs: 45 active+undersized, 33 active+undersized+degraded, 80 active+clean; 2.0 GiB data, 14 GiB used, 523 GiB / 536 GiB avail; 969 B/s rd, 0 B/s wr, 1.36k op/s; 16834/100677 objects degraded (16.721%) 2024-09-10T19:23:03.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:23:02 smithi191 ceph-mon[32884]: from='client.? 172.21.15.183:0/4217325711' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T19:23:03.301 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:23:03.324 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:23:03.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:03.325 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:23:03.426 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:23:03.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:03.463 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:23:03.463 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:23:03.564 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:23:03.587 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:23:03.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:03.588 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:23:03.689 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:23:03.728 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:23:03.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:03.730 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:23:03.831 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:23:03.868 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:23:03.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:03.872 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:23:03.973 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:23:04.009 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:23:04.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:04.010 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:23:04.110 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:23:04.133 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:23:04.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:04.134 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:23:04.235 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:23:04.238 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-10T19:23:04.239 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-10T19:23:04.259 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:23:04.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:04.260 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:23:04.361 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:23:04.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:04.395 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:23:04.395 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:23:04.496 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:23:04.530 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:23:04.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:04.531 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:23:04.632 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:23:04.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:04.675 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:23:04.675 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:23:04.776 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:23:04.800 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:23:04.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:04.801 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:23:04.902 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:23:04.930 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:23:04 smithi191 ceph-mon[32884]: pgmap v579: 158 pgs: 45 active+undersized, 33 active+undersized+degraded, 80 active+clean; 2.0 GiB data, 14 GiB used, 523 GiB / 536 GiB avail; 853 B/s rd, 0 B/s wr, 1.19k op/s; 16834/100677 objects degraded (16.721%) 2024-09-10T19:23:04.936 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:23:04.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:04.937 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:23:05.038 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:23:05.061 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:23:05.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:05.062 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:23:05.162 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:23:05.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:23:04 smithi183 ceph-mon[29736]: pgmap v579: 158 pgs: 45 active+undersized, 33 active+undersized+degraded, 80 active+clean; 2.0 GiB data, 14 GiB used, 523 GiB / 536 GiB avail; 853 B/s rd, 0 B/s wr, 1.19k op/s; 16834/100677 objects degraded (16.721%) 2024-09-10T19:23:05.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:23:04 smithi183 ceph-mon[29736]: Health check update: Degraded data redundancy: 16834/100677 objects degraded (16.721%), 33 pgs degraded, 78 pgs undersized (PG_DEGRADED) 2024-09-10T19:23:05.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:23:04 smithi183 ceph-mon[41562]: pgmap v579: 158 pgs: 45 active+undersized, 33 active+undersized+degraded, 80 active+clean; 2.0 GiB data, 14 GiB used, 523 GiB / 536 GiB avail; 853 B/s rd, 0 B/s wr, 1.19k op/s; 16834/100677 objects degraded (16.721%) 2024-09-10T19:23:05.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:23:04 smithi183 ceph-mon[41562]: Health check update: Degraded data redundancy: 16834/100677 objects degraded (16.721%), 33 pgs degraded, 78 pgs undersized (PG_DEGRADED) 2024-09-10T19:23:05.186 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:23:05.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:05.187 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:23:05.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:23:04 smithi191 ceph-mon[32884]: Health check update: Degraded data redundancy: 16834/100677 objects degraded (16.721%), 33 pgs degraded, 78 pgs undersized (PG_DEGRADED) 2024-09-10T19:23:05.288 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:23:05.321 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:23:05.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:05.322 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:23:05.423 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:23:05.455 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:23:05.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:05.456 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:23:05.556 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:23:05.594 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:23:05.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:05.595 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:23:05.696 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:23:05.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:05.746 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:23:05.746 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:23:05.847 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:23:05.886 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:23:05.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:05.887 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:23:05.974 INFO:teuthology.orchestra.run.smithi183.stdout:ERROR: (22) Invalid argument 2024-09-10T19:23:05.974 INFO:teuthology.orchestra.run.smithi183.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-10T19:23:05.988 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:23:06.033 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:23:06.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:06.034 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:23:06.135 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:23:06.172 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:23:06.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:06.173 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:23:06.239 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-10T19:23:06.273 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:23:06.320 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:23:06.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:06.321 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:23:06.415 INFO:teuthology.orchestra.run.smithi191.stdout:{ 2024-09-10T19:23:06.415 INFO:teuthology.orchestra.run.smithi191.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-09-10T19:23:06.415 INFO:teuthology.orchestra.run.smithi191.stdout:} 2024-09-10T19:23:06.422 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:23:06.447 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:23:06.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:06.448 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:23:06.549 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:23:06.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:06.583 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:23:06.583 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:23:06.684 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:23:06.706 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:23:06.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:06.707 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:23:06.808 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:23:06.812 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph daemon osd.1 dump_historic_ops 2024-09-10T19:23:06.843 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:23:06.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:06.844 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:23:06.945 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:23:06.979 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:23:06.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:06.980 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:23:07.064 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:23:06 smithi183 ceph-mon[29736]: pgmap v580: 158 pgs: 45 active+undersized, 33 active+undersized+degraded, 80 active+clean; 1.6 GiB data, 13 GiB used, 524 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.77k op/s; 13397/79983 objects degraded (16.750%) 2024-09-10T19:23:07.065 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:23:06 smithi183 ceph-mon[41562]: pgmap v580: 158 pgs: 45 active+undersized, 33 active+undersized+degraded, 80 active+clean; 1.6 GiB data, 13 GiB used, 524 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.77k op/s; 13397/79983 objects degraded (16.750%) 2024-09-10T19:23:07.081 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:23:07.113 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:23:07.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:07.114 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:23:07.117 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:23:07.135 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:23:06 smithi191 ceph-mon[32884]: pgmap v580: 158 pgs: 45 active+undersized, 33 active+undersized+degraded, 80 active+clean; 1.6 GiB data, 13 GiB used, 524 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.77k op/s; 13397/79983 objects degraded (16.750%) 2024-09-10T19:23:07.167 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:23:07.214 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:23:07.249 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:23:07.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:07.250 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:23:07.351 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:23:07.393 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:23:07.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:07.394 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:23:07.495 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:23:07.532 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:23:07.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:07.533 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:23:07.634 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:23:07.669 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:23:07.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:07.670 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:23:07.715 INFO:journalctl@ceph.mgr.x.smithi191.stdout:Sep 10 19:23:07 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-x[34967]: ::ffff:172.21.15.191 - - [10/Sep/2024:19:23:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T19:23:07.771 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:23:07.801 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:23:07.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:07.803 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:23:07.903 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:23:07.939 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:23:07.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:07.940 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:23:07.949 INFO:teuthology.orchestra.run.smithi183.stderr:noscrub is unset 2024-09-10T19:23:08.041 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:23:08.078 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:23:08.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:08.079 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:23:08.180 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:23:08.182 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:23:07 smithi183 ceph-mon[29736]: from='client.? 172.21.15.183:0/2325409962' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-10T19:23:08.182 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:23:07 smithi183 ceph-mon[29736]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-10T19:23:08.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:23:07 smithi183 ceph-mon[41562]: from='client.? 172.21.15.183:0/2325409962' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-10T19:23:08.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:23:07 smithi183 ceph-mon[41562]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-10T19:23:08.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:08.215 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:23:08.215 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:23:08.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:23:07 smithi191 ceph-mon[32884]: from='client.? 172.21.15.183:0/2325409962' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-10T19:23:08.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:23:07 smithi191 ceph-mon[32884]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-10T19:23:08.316 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:23:08.340 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:23:08.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:08.341 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:23:08.441 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:23:08.474 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:23:08.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:08.475 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:23:08.575 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:23:08.618 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:23:08.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:08.619 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:23:08.719 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:23:08.743 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:23:08.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:08.744 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:23:08.845 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:23:08.884 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:23:08.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:08.885 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:23:08.986 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:23:09.018 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:23:09.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:09.019 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:23:09.120 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:23:09.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:09.153 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:23:09.153 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:23:09.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:23:08 smithi191 ceph-mon[32884]: pgmap v581: 158 pgs: 45 active+undersized, 33 active+undersized+degraded, 80 active+clean; 1.6 GiB data, 13 GiB used, 524 GiB / 536 GiB avail; 852 B/s rd, 0 B/s wr, 1.15k op/s; 13397/79983 objects degraded (16.750%) 2024-09-10T19:23:09.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:23:08 smithi191 ceph-mon[32884]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T19:23:09.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:23:08 smithi191 ceph-mon[32884]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-10T19:23:09.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:23:08 smithi191 ceph-mon[32884]: osdmap e280: 8 total, 7 up, 6 in 2024-09-10T19:23:09.254 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:23:09.276 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:23:09.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:09.277 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:23:09.378 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:23:09.399 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:23:09.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:09.400 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:23:09.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:23:08 smithi183 ceph-mon[29736]: pgmap v581: 158 pgs: 45 active+undersized, 33 active+undersized+degraded, 80 active+clean; 1.6 GiB data, 13 GiB used, 524 GiB / 536 GiB avail; 852 B/s rd, 0 B/s wr, 1.15k op/s; 13397/79983 objects degraded (16.750%) 2024-09-10T19:23:09.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:23:08 smithi183 ceph-mon[29736]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T19:23:09.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:23:08 smithi183 ceph-mon[29736]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-10T19:23:09.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:23:08 smithi183 ceph-mon[29736]: osdmap e280: 8 total, 7 up, 6 in 2024-09-10T19:23:09.431 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:23:08 smithi183 ceph-mon[41562]: pgmap v581: 158 pgs: 45 active+undersized, 33 active+undersized+degraded, 80 active+clean; 1.6 GiB data, 13 GiB used, 524 GiB / 536 GiB avail; 852 B/s rd, 0 B/s wr, 1.15k op/s; 13397/79983 objects degraded (16.750%) 2024-09-10T19:23:09.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:23:08 smithi183 ceph-mon[41562]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T19:23:09.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:23:08 smithi183 ceph-mon[41562]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-10T19:23:09.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:23:08 smithi183 ceph-mon[41562]: osdmap e280: 8 total, 7 up, 6 in 2024-09-10T19:23:09.501 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:23:09.529 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:23:09.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:09.530 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:23:09.631 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:23:09.668 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:23:09.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:09.669 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:23:09.770 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:23:09.806 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:23:09.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:09.807 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:23:09.908 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:23:09.945 INFO:journalctl@ceph.iscsi.iscsi.a.smithi191.stdout:Sep 10 19:23:09 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-iscsi-iscsi-a[61816]: debug there is no tcmu-runner data available 2024-09-10T19:23:09.948 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:23:09.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:09.950 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:23:10.051 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:23:10.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:10.087 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:23:10.087 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:23:10.157 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:23:09 smithi183 ceph-mon[29736]: 2.a scrub starts 2024-09-10T19:23:10.157 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:23:09 smithi183 ceph-mon[29736]: 2.a scrub ok 2024-09-10T19:23:10.158 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:23:09 smithi183 ceph-mon[29736]: 2.9 scrub starts 2024-09-10T19:23:10.158 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:23:09 smithi183 ceph-mon[29736]: 2.9 scrub ok 2024-09-10T19:23:10.158 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:23:09 smithi183 ceph-mon[29736]: 5.1f scrub starts 2024-09-10T19:23:10.158 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:23:09 smithi183 ceph-mon[29736]: 5.1f scrub ok 2024-09-10T19:23:10.158 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:23:09 smithi183 ceph-mon[29736]: Health check update: Degraded data redundancy: 13397/79983 objects degraded (16.750%), 33 pgs degraded, 78 pgs undersized (PG_DEGRADED) 2024-09-10T19:23:10.158 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:23:09 smithi183 ceph-mon[41562]: 2.a scrub starts 2024-09-10T19:23:10.158 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:23:09 smithi183 ceph-mon[41562]: 2.a scrub ok 2024-09-10T19:23:10.158 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:23:09 smithi183 ceph-mon[41562]: 2.9 scrub starts 2024-09-10T19:23:10.159 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:23:09 smithi183 ceph-mon[41562]: 2.9 scrub ok 2024-09-10T19:23:10.159 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:23:09 smithi183 ceph-mon[41562]: 5.1f scrub starts 2024-09-10T19:23:10.159 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:23:09 smithi183 ceph-mon[41562]: 5.1f scrub ok 2024-09-10T19:23:10.159 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:23:09 smithi183 ceph-mon[41562]: Health check update: Degraded data redundancy: 13397/79983 objects degraded (16.750%), 33 pgs degraded, 78 pgs undersized (PG_DEGRADED) 2024-09-10T19:23:10.187 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:23:10.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:23:09 smithi191 ceph-mon[32884]: 2.a scrub starts 2024-09-10T19:23:10.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:23:09 smithi191 ceph-mon[32884]: 2.a scrub ok 2024-09-10T19:23:10.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:23:09 smithi191 ceph-mon[32884]: 2.9 scrub starts 2024-09-10T19:23:10.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:23:09 smithi191 ceph-mon[32884]: 2.9 scrub ok 2024-09-10T19:23:10.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:23:09 smithi191 ceph-mon[32884]: 5.1f scrub starts 2024-09-10T19:23:10.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:23:09 smithi191 ceph-mon[32884]: 5.1f scrub ok 2024-09-10T19:23:10.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:23:09 smithi191 ceph-mon[32884]: Health check update: Degraded data redundancy: 13397/79983 objects degraded (16.750%), 33 pgs degraded, 78 pgs undersized (PG_DEGRADED) 2024-09-10T19:23:10.224 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:23:10.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:10.225 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:23:10.326 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:23:10.357 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:23:10.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:10.358 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:23:10.459 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:23:10.483 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:23:10.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:10.484 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:23:10.585 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:23:10.616 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:23:10.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:10.619 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:23:10.719 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:23:10.743 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:23:10.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:10.744 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:23:10.845 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:23:10.868 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:23:10.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:10.869 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:23:10.970 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:23:11.008 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:23:11.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:11.009 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:23:11.110 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:23:11.145 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:23:11.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:11.146 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:23:11.157 INFO:teuthology.orchestra.run.smithi183.stdout:ERROR: (22) Invalid argument 2024-09-10T19:23:11.158 INFO:teuthology.orchestra.run.smithi183.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-10T19:23:11.158 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:23:10 smithi183 ceph-mon[29736]: 5.18 scrub starts 2024-09-10T19:23:11.158 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:23:10 smithi183 ceph-mon[29736]: 5.18 scrub ok 2024-09-10T19:23:11.158 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:23:10 smithi183 ceph-mon[29736]: from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:23:11.158 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:23:10 smithi183 ceph-mon[29736]: pgmap v583: 158 pgs: 45 active+undersized, 33 active+undersized+degraded, 80 active+clean; 1.6 GiB data, 13 GiB used, 524 GiB / 536 GiB avail; 1023 B/s rd, 0 B/s wr, 1.38k op/s; 13397/79983 objects degraded (16.750%) 2024-09-10T19:23:11.158 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:23:10 smithi183 ceph-mon[41562]: 5.18 scrub starts 2024-09-10T19:23:11.158 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:23:10 smithi183 ceph-mon[41562]: 5.18 scrub ok 2024-09-10T19:23:11.159 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:23:10 smithi183 ceph-mon[41562]: from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:23:11.159 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:23:10 smithi183 ceph-mon[41562]: pgmap v583: 158 pgs: 45 active+undersized, 33 active+undersized+degraded, 80 active+clean; 1.6 GiB data, 13 GiB used, 524 GiB / 536 GiB avail; 1023 B/s rd, 0 B/s wr, 1.38k op/s; 13397/79983 objects degraded (16.750%) 2024-09-10T19:23:11.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:23:10 smithi191 ceph-mon[32884]: 5.18 scrub starts 2024-09-10T19:23:11.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:23:10 smithi191 ceph-mon[32884]: 5.18 scrub ok 2024-09-10T19:23:11.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:23:10 smithi191 ceph-mon[32884]: from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:23:11.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:23:10 smithi191 ceph-mon[32884]: pgmap v583: 158 pgs: 45 active+undersized, 33 active+undersized+degraded, 80 active+clean; 1.6 GiB data, 13 GiB used, 524 GiB / 536 GiB avail; 1023 B/s rd, 0 B/s wr, 1.38k op/s; 13397/79983 objects degraded (16.750%) 2024-09-10T19:23:11.248 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:23:11.272 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:23:11.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:11.273 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:23:11.374 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:23:11.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:11.398 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:23:11.398 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:23:11.431 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Sep 10 19:23:11 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-y[29989]: ::ffff:172.21.15.191 - - [10/Sep/2024:19:23:11] "GET /metrics HTTP/1.1" 200 224308 "" "Prometheus/2.43.0" 2024-09-10T19:23:11.500 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:23:11.534 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:23:11.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:11.535 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:23:11.636 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:23:11.676 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:23:11.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:11.677 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:23:11.778 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:23:11.817 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:23:11.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:11.818 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:23:11.886 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph daemon osd.2 dump_ops_in_flight 2024-09-10T19:23:11.919 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:23:11.959 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:23:11.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:11.960 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:23:12.061 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:23:12.097 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:23:12.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:12.098 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:23:12.176 INFO:tasks.thrashosds.thrasher:in_osds: [0, 3, 4, 5, 6, 7] out_osds: [1, 2] dead_osds: [0] live_osds: [2, 3, 4, 5, 6, 7, 1] 2024-09-10T19:23:12.176 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-10T19:23:12.176 INFO:tasks.thrashosds.thrasher:inject_pause on osd.3 2024-09-10T19:23:12.176 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-09-10T19:23:12.177 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-09-10T19:23:12.177 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph daemon osd.3 config set filestore_inject_stall 3 2024-09-10T19:23:12.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:23:11 smithi183 ceph-mon[29736]: 9.2 scrub starts 2024-09-10T19:23:12.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:23:11 smithi183 ceph-mon[29736]: 9.2 scrub ok 2024-09-10T19:23:12.181 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:23:11 smithi183 ceph-mon[41562]: 9.2 scrub starts 2024-09-10T19:23:12.181 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:23:11 smithi183 ceph-mon[41562]: 9.2 scrub ok 2024-09-10T19:23:12.199 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:23:12.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:23:11 smithi191 ceph-mon[32884]: 9.2 scrub starts 2024-09-10T19:23:12.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:23:11 smithi191 ceph-mon[32884]: 9.2 scrub ok 2024-09-10T19:23:12.233 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:23:12.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:12.234 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:23:12.335 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:23:12.367 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:23:12.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:12.368 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:23:12.422 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:23:12.469 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:23:12.493 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:23:12.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:12.494 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:23:12.595 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:23:12.630 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-10T19:23:12.635 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:23:12.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:12.635 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:23:12.736 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:23:12.772 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:23:12.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:12.773 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:23:12.873 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:23:12.902 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:23:12.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:12.903 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:23:12.998 INFO:teuthology.orchestra.run.smithi183.stdout:osd.1: {} 2024-09-10T19:23:12.998 INFO:teuthology.orchestra.run.smithi183.stdout:osd.2: {} 2024-09-10T19:23:12.999 INFO:teuthology.orchestra.run.smithi183.stderr:Error ENXIO: problem getting command descriptions from osd.0 2024-09-10T19:23:12.999 INFO:teuthology.orchestra.run.smithi183.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-10T19:23:12.999 INFO:teuthology.orchestra.run.smithi183.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-10T19:23:13.004 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:23:13.010 INFO:teuthology.orchestra.run.smithi183.stdout:osd.3: {} 2024-09-10T19:23:13.011 INFO:teuthology.orchestra.run.smithi183.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-10T19:23:13.036 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:23:13.036 INFO:teuthology.orchestra.run.smithi183.stdout:osd.4: {} 2024-09-10T19:23:13.037 INFO:teuthology.orchestra.run.smithi183.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-10T19:23:13.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:13.038 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:23:13.058 INFO:teuthology.orchestra.run.smithi183.stdout:osd.5: {} 2024-09-10T19:23:13.058 INFO:teuthology.orchestra.run.smithi183.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-10T19:23:13.069 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:23:13.085 INFO:teuthology.orchestra.run.smithi183.stdout:osd.6: {} 2024-09-10T19:23:13.085 INFO:teuthology.orchestra.run.smithi183.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-10T19:23:13.113 INFO:teuthology.orchestra.run.smithi183.stdout:osd.7: {} 2024-09-10T19:23:13.113 INFO:teuthology.orchestra.run.smithi183.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-10T19:23:13.114 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:23:12 smithi183 ceph-mon[29736]: 9.7 scrub starts 2024-09-10T19:23:13.114 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:23:12 smithi183 ceph-mon[29736]: 4.e scrub starts 2024-09-10T19:23:13.114 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:23:12 smithi183 ceph-mon[29736]: 4.e scrub ok 2024-09-10T19:23:13.114 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:23:12 smithi183 ceph-mon[29736]: pgmap v584: 158 pgs: 45 active+undersized, 33 active+undersized+degraded, 80 active+clean; 1.2 GiB data, 12 GiB used, 525 GiB / 536 GiB avail; 1023 B/s rd, 0 B/s wr, 1.41k op/s; 9798/58248 objects degraded (16.821%) 2024-09-10T19:23:13.114 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:23:12 smithi183 ceph-mon[29736]: 9.7 scrub ok 2024-09-10T19:23:13.114 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:23:12 smithi183 ceph-mon[29736]: from='client.? 172.21.15.183:0/2288115213' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T19:23:13.114 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:23:12 smithi183 ceph-mon[41562]: 9.7 scrub starts 2024-09-10T19:23:13.114 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:23:12 smithi183 ceph-mon[41562]: 4.e scrub starts 2024-09-10T19:23:13.115 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:23:12 smithi183 ceph-mon[41562]: 4.e scrub ok 2024-09-10T19:23:13.115 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:23:12 smithi183 ceph-mon[41562]: pgmap v584: 158 pgs: 45 active+undersized, 33 active+undersized+degraded, 80 active+clean; 1.2 GiB data, 12 GiB used, 525 GiB / 536 GiB avail; 1023 B/s rd, 0 B/s wr, 1.41k op/s; 9798/58248 objects degraded (16.821%) 2024-09-10T19:23:13.115 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:23:12 smithi183 ceph-mon[41562]: 9.7 scrub ok 2024-09-10T19:23:13.115 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:23:12 smithi183 ceph-mon[41562]: from='client.? 172.21.15.183:0/2288115213' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T19:23:13.138 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:23:13.162 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:23:13.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:13.163 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:23:13.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:23:12 smithi191 ceph-mon[32884]: 9.7 scrub starts 2024-09-10T19:23:13.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:23:12 smithi191 ceph-mon[32884]: 4.e scrub starts 2024-09-10T19:23:13.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:23:12 smithi191 ceph-mon[32884]: 4.e scrub ok 2024-09-10T19:23:13.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:23:12 smithi191 ceph-mon[32884]: pgmap v584: 158 pgs: 45 active+undersized, 33 active+undersized+degraded, 80 active+clean; 1.2 GiB data, 12 GiB used, 525 GiB / 536 GiB avail; 1023 B/s rd, 0 B/s wr, 1.41k op/s; 9798/58248 objects degraded (16.821%) 2024-09-10T19:23:13.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:23:12 smithi191 ceph-mon[32884]: 9.7 scrub ok 2024-09-10T19:23:13.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:23:12 smithi191 ceph-mon[32884]: from='client.? 172.21.15.183:0/2288115213' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T19:23:13.264 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:23:13.301 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:23:13.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:13.303 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:23:13.403 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:23:13.439 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:23:13.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:13.441 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:23:13.541 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:23:13.565 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:23:13.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:13.566 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:23:13.667 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:23:13.701 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:23:13.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:13.702 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:23:13.803 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:23:13.827 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:23:13.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:13.828 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:23:13.929 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:23:13.976 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:23:13.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:13.977 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:23:14.078 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:23:14.128 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:23:14.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:14.129 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:23:14.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:23:13 smithi191 ceph-mon[32884]: 4.c scrub starts 2024-09-10T19:23:14.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:23:13 smithi191 ceph-mon[32884]: 4.c scrub ok 2024-09-10T19:23:14.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:23:13 smithi191 ceph-mon[32884]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:23:14.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:23:13 smithi191 ceph-mon[32884]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T19:23:14.230 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:23:14.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:14.276 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:23:14.276 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:23:14.377 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:23:14.391 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:23:13 smithi183 ceph-mon[29736]: 4.c scrub starts 2024-09-10T19:23:14.391 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:23:13 smithi183 ceph-mon[29736]: 4.c scrub ok 2024-09-10T19:23:14.391 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:23:13 smithi183 ceph-mon[29736]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:23:14.391 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:23:13 smithi183 ceph-mon[29736]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T19:23:14.391 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:23:13 smithi183 ceph-mon[41562]: 4.c scrub starts 2024-09-10T19:23:14.391 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:23:13 smithi183 ceph-mon[41562]: 4.c scrub ok 2024-09-10T19:23:14.391 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:23:13 smithi183 ceph-mon[41562]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:23:14.391 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:23:13 smithi183 ceph-mon[41562]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T19:23:14.439 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:23:14.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:14.440 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:23:14.541 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:23:14.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:14.588 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:23:14.588 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:23:14.689 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:23:14.725 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:23:14.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:14.726 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:23:14.827 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:23:14.883 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:23:14.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:14.884 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:23:14.985 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:23:15.035 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:23:15.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:15.037 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:23:15.138 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:23:15.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:15.175 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:23:15.175 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:23:15.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:23:14 smithi191 ceph-mon[32884]: 6.4 scrub starts 2024-09-10T19:23:15.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:23:14 smithi191 ceph-mon[32884]: 6.4 scrub ok 2024-09-10T19:23:15.217 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:23:14 smithi191 ceph-mon[32884]: 1.5 scrub starts 2024-09-10T19:23:15.217 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:23:14 smithi191 ceph-mon[32884]: 1.5 scrub ok 2024-09-10T19:23:15.217 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:23:14 smithi191 ceph-mon[32884]: pgmap v585: 158 pgs: 45 active+undersized, 33 active+undersized+degraded, 80 active+clean; 1.2 GiB data, 12 GiB used, 525 GiB / 536 GiB avail; 1023 B/s rd, 0 B/s wr, 1.41k op/s; 9798/58248 objects degraded (16.821%) 2024-09-10T19:23:15.217 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:23:14 smithi191 ceph-mon[32884]: 1.3 scrub starts 2024-09-10T19:23:15.217 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:23:14 smithi191 ceph-mon[32884]: 1.3 scrub ok 2024-09-10T19:23:15.217 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:23:14 smithi191 ceph-mon[32884]: Health check update: Degraded data redundancy: 9798/58248 objects degraded (16.821%), 33 pgs degraded, 78 pgs undersized (PG_DEGRADED) 2024-09-10T19:23:15.276 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:23:15.302 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:23:15.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:15.304 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:23:15.405 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:23:15.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:23:14 smithi183 ceph-mon[29736]: 6.4 scrub starts 2024-09-10T19:23:15.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:23:14 smithi183 ceph-mon[29736]: 6.4 scrub ok 2024-09-10T19:23:15.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:23:14 smithi183 ceph-mon[29736]: 1.5 scrub starts 2024-09-10T19:23:15.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:23:14 smithi183 ceph-mon[29736]: 1.5 scrub ok 2024-09-10T19:23:15.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:23:14 smithi183 ceph-mon[29736]: pgmap v585: 158 pgs: 45 active+undersized, 33 active+undersized+degraded, 80 active+clean; 1.2 GiB data, 12 GiB used, 525 GiB / 536 GiB avail; 1023 B/s rd, 0 B/s wr, 1.41k op/s; 9798/58248 objects degraded (16.821%) 2024-09-10T19:23:15.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:23:14 smithi183 ceph-mon[29736]: 1.3 scrub starts 2024-09-10T19:23:15.433 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:23:14 smithi183 ceph-mon[29736]: 1.3 scrub ok 2024-09-10T19:23:15.433 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:23:14 smithi183 ceph-mon[29736]: Health check update: Degraded data redundancy: 9798/58248 objects degraded (16.821%), 33 pgs degraded, 78 pgs undersized (PG_DEGRADED) 2024-09-10T19:23:15.433 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:23:14 smithi183 ceph-mon[41562]: 6.4 scrub starts 2024-09-10T19:23:15.433 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:23:14 smithi183 ceph-mon[41562]: 6.4 scrub ok 2024-09-10T19:23:15.434 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:23:14 smithi183 ceph-mon[41562]: 1.5 scrub starts 2024-09-10T19:23:15.434 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:23:14 smithi183 ceph-mon[41562]: 1.5 scrub ok 2024-09-10T19:23:15.434 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:23:14 smithi183 ceph-mon[41562]: pgmap v585: 158 pgs: 45 active+undersized, 33 active+undersized+degraded, 80 active+clean; 1.2 GiB data, 12 GiB used, 525 GiB / 536 GiB avail; 1023 B/s rd, 0 B/s wr, 1.41k op/s; 9798/58248 objects degraded (16.821%) 2024-09-10T19:23:15.434 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:23:14 smithi183 ceph-mon[41562]: 1.3 scrub starts 2024-09-10T19:23:15.434 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:23:14 smithi183 ceph-mon[41562]: 1.3 scrub ok 2024-09-10T19:23:15.434 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:23:14 smithi183 ceph-mon[41562]: Health check update: Degraded data redundancy: 9798/58248 objects degraded (16.821%), 33 pgs degraded, 78 pgs undersized (PG_DEGRADED) 2024-09-10T19:23:15.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:15.454 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:23:15.455 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:23:15.544 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-10T19:23:15.545 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-10T19:23:15.555 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:23:15.595 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:23:15.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:15.596 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:23:15.697 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:23:15.721 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:23:15.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:15.722 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:23:15.823 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:23:15.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:15.847 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:23:15.847 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:23:15.899 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:23:15.949 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:23:15.974 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:23:15.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:15.975 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:23:16.076 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:23:16.125 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:23:16.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:16.126 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:23:16.227 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:23:16.266 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:23:16.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:16.268 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:23:16.368 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:23:16.390 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:23:16.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:16.391 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:23:16.492 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:23:16.524 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:23:16.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:16.525 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:23:16.626 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:23:16.665 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:23:16.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:16.666 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:23:16.767 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:23:16.808 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:23:16.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:16.809 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:23:16.910 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:23:16.949 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:23:16.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:16.950 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:23:17.051 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:23:17.061 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:23:16 smithi183 ceph-mon[29736]: pgmap v586: 158 pgs: 45 active+undersized, 33 active+undersized+degraded, 80 active+clean; 746 MiB data, 11 GiB used, 526 GiB / 536 GiB avail; 1023 B/s rd, 0 B/s wr, 1.45k op/s; 6131/36369 objects degraded (16.858%) 2024-09-10T19:23:17.061 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:23:16 smithi183 ceph-mon[41562]: pgmap v586: 158 pgs: 45 active+undersized, 33 active+undersized+degraded, 80 active+clean; 746 MiB data, 11 GiB used, 526 GiB / 536 GiB avail; 1023 B/s rd, 0 B/s wr, 1.45k op/s; 6131/36369 objects degraded (16.858%) 2024-09-10T19:23:17.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:17.091 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:23:17.091 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:23:17.192 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:23:17.224 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:23:17.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:17.225 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:23:17.326 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:23:17.366 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:23:17.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:17.367 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:23:17.410 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:23:16 smithi191 ceph-mon[32884]: pgmap v586: 158 pgs: 45 active+undersized, 33 active+undersized+degraded, 80 active+clean; 746 MiB data, 11 GiB used, 526 GiB / 536 GiB avail; 1023 B/s rd, 0 B/s wr, 1.45k op/s; 6131/36369 objects degraded (16.858%) 2024-09-10T19:23:17.468 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:23:17.510 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:23:17.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:17.511 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:23:17.545 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-10T19:23:17.611 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:23:17.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:17.647 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:23:17.648 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:23:17.716 INFO:journalctl@ceph.mgr.x.smithi191.stdout:Sep 10 19:23:17 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-x[34967]: ::ffff:172.21.15.191 - - [10/Sep/2024:19:23:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T19:23:17.748 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:23:17.789 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:23:17.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:17.791 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:23:17.892 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:23:17.935 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:23:17.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:17.936 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:23:18.037 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:23:18.058 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:23:18.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:18.059 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:23:18.160 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:23:18.198 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:23:18.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:18.200 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:23:18.301 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:23:18.322 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:23:18.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:18.323 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:23:18.358 INFO:teuthology.orchestra.run.smithi183.stdout:{ 2024-09-10T19:23:18.358 INFO:teuthology.orchestra.run.smithi183.stdout: "ops": [], 2024-09-10T19:23:18.358 INFO:teuthology.orchestra.run.smithi183.stdout: "num_ops": 0 2024-09-10T19:23:18.358 INFO:teuthology.orchestra.run.smithi183.stdout:} 2024-09-10T19:23:18.424 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:23:18.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:18.453 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:23:18.453 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:23:18.553 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:23:18.583 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:23:18.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:18.585 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:23:18.685 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:23:18.714 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:23:18.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:18.715 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:23:18.816 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:23:18.843 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:23:18.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:18.844 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:23:18.944 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:23:18.974 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:23:18.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:18.975 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:23:18.989 INFO:teuthology.orchestra.run.smithi183.stderr:nodeep-scrub is unset 2024-09-10T19:23:19.075 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:23:19.099 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:23:19.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:19.100 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:23:19.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:23:18 smithi183 ceph-mon[29736]: 4.14 scrub starts 2024-09-10T19:23:19.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:23:18 smithi183 ceph-mon[29736]: 4.14 scrub ok 2024-09-10T19:23:19.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:23:18 smithi183 ceph-mon[29736]: 5.12 scrub starts 2024-09-10T19:23:19.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:23:18 smithi183 ceph-mon[29736]: 5.12 scrub ok 2024-09-10T19:23:19.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:23:18 smithi183 ceph-mon[29736]: 4.18 scrub starts 2024-09-10T19:23:19.182 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:23:18 smithi183 ceph-mon[29736]: 4.18 scrub ok 2024-09-10T19:23:19.182 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:23:18 smithi183 ceph-mon[29736]: pgmap v587: 158 pgs: 45 active+undersized, 33 active+undersized+degraded, 80 active+clean; 746 MiB data, 11 GiB used, 526 GiB / 536 GiB avail; 1023 B/s rd, 0 B/s wr, 1.45k op/s; 6131/36369 objects degraded (16.858%) 2024-09-10T19:23:19.182 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:23:18 smithi183 ceph-mon[29736]: from='client.? 172.21.15.183:0/1501091110' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-10T19:23:19.182 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:23:18 smithi183 ceph-mon[29736]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-10T19:23:19.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:23:18 smithi183 ceph-mon[41562]: 4.14 scrub starts 2024-09-10T19:23:19.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:23:18 smithi183 ceph-mon[41562]: 4.14 scrub ok 2024-09-10T19:23:19.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:23:18 smithi183 ceph-mon[41562]: 5.12 scrub starts 2024-09-10T19:23:19.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:23:18 smithi183 ceph-mon[41562]: 5.12 scrub ok 2024-09-10T19:23:19.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:23:18 smithi183 ceph-mon[41562]: 4.18 scrub starts 2024-09-10T19:23:19.183 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:23:18 smithi183 ceph-mon[41562]: 4.18 scrub ok 2024-09-10T19:23:19.183 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:23:18 smithi183 ceph-mon[41562]: pgmap v587: 158 pgs: 45 active+undersized, 33 active+undersized+degraded, 80 active+clean; 746 MiB data, 11 GiB used, 526 GiB / 536 GiB avail; 1023 B/s rd, 0 B/s wr, 1.45k op/s; 6131/36369 objects degraded (16.858%) 2024-09-10T19:23:19.183 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:23:18 smithi183 ceph-mon[41562]: from='client.? 172.21.15.183:0/1501091110' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-10T19:23:19.183 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:23:18 smithi183 ceph-mon[41562]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-10T19:23:19.201 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:23:19.232 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:23:19.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:19.233 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:23:19.334 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:23:19.367 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:23:19.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:19.368 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:23:19.426 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph daemon osd.2 dump_blocked_ops 2024-09-10T19:23:19.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:23:18 smithi191 ceph-mon[32884]: 4.14 scrub starts 2024-09-10T19:23:19.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:23:18 smithi191 ceph-mon[32884]: 4.14 scrub ok 2024-09-10T19:23:19.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:23:18 smithi191 ceph-mon[32884]: 5.12 scrub starts 2024-09-10T19:23:19.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:23:18 smithi191 ceph-mon[32884]: 5.12 scrub ok 2024-09-10T19:23:19.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:23:18 smithi191 ceph-mon[32884]: 4.18 scrub starts 2024-09-10T19:23:19.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:23:18 smithi191 ceph-mon[32884]: 4.18 scrub ok 2024-09-10T19:23:19.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:23:18 smithi191 ceph-mon[32884]: pgmap v587: 158 pgs: 45 active+undersized, 33 active+undersized+degraded, 80 active+clean; 746 MiB data, 11 GiB used, 526 GiB / 536 GiB avail; 1023 B/s rd, 0 B/s wr, 1.45k op/s; 6131/36369 objects degraded (16.858%) 2024-09-10T19:23:19.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:23:18 smithi191 ceph-mon[32884]: from='client.? 172.21.15.183:0/1501091110' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-10T19:23:19.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:23:18 smithi191 ceph-mon[32884]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-10T19:23:19.469 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:23:19.502 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:23:19.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:19.503 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:23:19.604 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:23:19.624 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:23:19.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:19.625 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:23:19.726 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:23:19.764 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:23:19.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:19.765 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:23:19.866 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:23:19.904 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:23:19.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:19.905 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:23:19.965 INFO:journalctl@ceph.iscsi.iscsi.a.smithi191.stdout:Sep 10 19:23:19 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-iscsi-iscsi-a[61816]: debug there is no tcmu-runner data available 2024-09-10T19:23:20.006 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:23:20.047 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:23:20.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:20.048 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:23:20.149 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:23:20.181 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:23:19 smithi183 ceph-mon[41562]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-10T19:23:20.181 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:23:19 smithi183 ceph-mon[41562]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-10T19:23:20.181 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:23:19 smithi183 ceph-mon[41562]: osdmap e281: 8 total, 7 up, 6 in 2024-09-10T19:23:20.181 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:23:19 smithi183 ceph-mon[41562]: Health check update: Degraded data redundancy: 6131/36369 objects degraded (16.858%), 33 pgs degraded, 78 pgs undersized (PG_DEGRADED) 2024-09-10T19:23:20.182 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:23:19 smithi183 ceph-mon[29736]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-10T19:23:20.182 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:23:19 smithi183 ceph-mon[29736]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-10T19:23:20.182 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:23:19 smithi183 ceph-mon[29736]: osdmap e281: 8 total, 7 up, 6 in 2024-09-10T19:23:20.182 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:23:19 smithi183 ceph-mon[29736]: Health check update: Degraded data redundancy: 6131/36369 objects degraded (16.858%), 33 pgs degraded, 78 pgs undersized (PG_DEGRADED) 2024-09-10T19:23:20.183 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:23:20.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:20.184 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:23:20.284 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:23:20.306 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:23:20.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:20.307 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:23:20.408 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:23:20.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:20.445 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:23:20.445 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:23:20.468 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:23:19 smithi191 ceph-mon[32884]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-10T19:23:20.468 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:23:19 smithi191 ceph-mon[32884]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-10T19:23:20.468 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:23:19 smithi191 ceph-mon[32884]: osdmap e281: 8 total, 7 up, 6 in 2024-09-10T19:23:20.468 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:23:19 smithi191 ceph-mon[32884]: Health check update: Degraded data redundancy: 6131/36369 objects degraded (16.858%), 33 pgs degraded, 78 pgs undersized (PG_DEGRADED) 2024-09-10T19:23:20.546 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:23:20.569 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:23:20.570 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:23:20.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:20.571 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:23:20.673 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:23:20.701 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:23:20.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:20.702 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:23:20.802 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:23:20.824 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:23:20.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:20.825 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:23:20.926 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:23:20.946 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:23:20.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:20.947 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:23:21.048 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:23:21.085 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:23:21.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:21.086 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:23:21.187 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:23:21.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:21.229 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:23:21.230 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:23:21.316 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:23:20 smithi183 ceph-mon[29736]: 6.b scrub starts 2024-09-10T19:23:21.316 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:23:20 smithi183 ceph-mon[29736]: 6.b scrub ok 2024-09-10T19:23:21.316 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:23:20 smithi183 ceph-mon[29736]: from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:23:21.316 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:23:20 smithi183 ceph-mon[29736]: 2.16 scrub starts 2024-09-10T19:23:21.316 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:23:20 smithi183 ceph-mon[29736]: 2.16 scrub ok 2024-09-10T19:23:21.316 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:23:20 smithi183 ceph-mon[29736]: pgmap v589: 158 pgs: 45 active+undersized, 33 active+undersized+degraded, 80 active+clean; 746 MiB data, 11 GiB used, 526 GiB / 536 GiB avail; 1023 B/s rd, 0 B/s wr, 1.45k op/s; 6131/36369 objects degraded (16.858%) 2024-09-10T19:23:21.316 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:23:20 smithi183 ceph-mon[41562]: 6.b scrub starts 2024-09-10T19:23:21.317 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:23:20 smithi183 ceph-mon[41562]: 6.b scrub ok 2024-09-10T19:23:21.317 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:23:20 smithi183 ceph-mon[41562]: from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:23:21.317 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:23:20 smithi183 ceph-mon[41562]: 2.16 scrub starts 2024-09-10T19:23:21.317 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:23:20 smithi183 ceph-mon[41562]: 2.16 scrub ok 2024-09-10T19:23:21.317 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:23:20 smithi183 ceph-mon[41562]: pgmap v589: 158 pgs: 45 active+undersized, 33 active+undersized+degraded, 80 active+clean; 746 MiB data, 11 GiB used, 526 GiB / 536 GiB avail; 1023 B/s rd, 0 B/s wr, 1.45k op/s; 6131/36369 objects degraded (16.858%) 2024-09-10T19:23:21.330 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:23:21.353 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:23:21.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:21.354 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:23:21.367 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:23:20 smithi191 ceph-mon[32884]: 6.b scrub starts 2024-09-10T19:23:21.367 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:23:20 smithi191 ceph-mon[32884]: 6.b scrub ok 2024-09-10T19:23:21.368 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:23:20 smithi191 ceph-mon[32884]: from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:23:21.368 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:23:20 smithi191 ceph-mon[32884]: 2.16 scrub starts 2024-09-10T19:23:21.368 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:23:20 smithi191 ceph-mon[32884]: 2.16 scrub ok 2024-09-10T19:23:21.368 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:23:20 smithi191 ceph-mon[32884]: pgmap v589: 158 pgs: 45 active+undersized, 33 active+undersized+degraded, 80 active+clean; 746 MiB data, 11 GiB used, 526 GiB / 536 GiB avail; 1023 B/s rd, 0 B/s wr, 1.45k op/s; 6131/36369 objects degraded (16.858%) 2024-09-10T19:23:21.454 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:23:21.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:21.494 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:23:21.494 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:23:21.568 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Sep 10 19:23:21 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-y[29989]: ::ffff:172.21.15.191 - - [10/Sep/2024:19:23:21] "GET /metrics HTTP/1.1" 200 224305 "" "Prometheus/2.43.0" 2024-09-10T19:23:21.595 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:23:21.628 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:23:21.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:21.629 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:23:21.730 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:23:21.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:21.770 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:23:21.770 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:23:21.871 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:23:21.895 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:23:21.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:21.896 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:23:21.997 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:23:22.023 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:23:22.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:22.024 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:23:22.049 INFO:teuthology.orchestra.run.smithi183.stdout:{ 2024-09-10T19:23:22.049 INFO:teuthology.orchestra.run.smithi183.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-09-10T19:23:22.049 INFO:teuthology.orchestra.run.smithi183.stdout:} 2024-09-10T19:23:22.125 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:23:22.163 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:23:22.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:22.164 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:23:22.265 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:23:22.286 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:23:22.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:22.287 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:23:22.388 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:23:22.411 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:23:22.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:22.412 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:23:22.427 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:23:21 smithi183 ceph-mon[29736]: 2.e scrub starts 2024-09-10T19:23:22.427 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:23:21 smithi183 ceph-mon[29736]: 2.e scrub ok 2024-09-10T19:23:22.427 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:23:21 smithi183 ceph-mon[41562]: 2.e scrub starts 2024-09-10T19:23:22.427 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:23:21 smithi183 ceph-mon[41562]: 2.e scrub ok 2024-09-10T19:23:22.465 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:23:21 smithi191 ceph-mon[32884]: 2.e scrub starts 2024-09-10T19:23:22.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:23:21 smithi191 ceph-mon[32884]: 2.e scrub ok 2024-09-10T19:23:22.516 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:23:22.540 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:23:22.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:22.541 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:23:22.642 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:23:22.666 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:23:22.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:22.667 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:23:22.768 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:23:22.793 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:23:22.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:22.795 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:23:22.895 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:23:22.920 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:23:22.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:22.921 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:23:23.025 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:23:23.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:23.049 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:23:23.049 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:23:23.068 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph osd set noscrub 2024-09-10T19:23:23.151 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:23:23.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:23.172 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:23:23.172 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:23:23.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:23:22 smithi183 ceph-mon[29736]: pgmap v590: 158 pgs: 45 active+undersized, 33 active+undersized+degraded, 80 active+clean; 338 MiB data, 9.6 GiB used, 527 GiB / 536 GiB avail; 1023 B/s rd, 0 B/s wr, 1.38k op/s; 2830/16806 objects degraded (16.839%) 2024-09-10T19:23:23.181 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:23:22 smithi183 ceph-mon[41562]: pgmap v590: 158 pgs: 45 active+undersized, 33 active+undersized+degraded, 80 active+clean; 338 MiB data, 9.6 GiB used, 527 GiB / 536 GiB avail; 1023 B/s rd, 0 B/s wr, 1.38k op/s; 2830/16806 objects degraded (16.839%) 2024-09-10T19:23:23.273 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:23:23.310 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:23:23.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:23.311 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:23:23.412 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:23:23.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:23.433 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:23:23.433 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:23:23.435 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:23:23.465 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:23:22 smithi191 ceph-mon[32884]: pgmap v590: 158 pgs: 45 active+undersized, 33 active+undersized+degraded, 80 active+clean; 338 MiB data, 9.6 GiB used, 527 GiB / 536 GiB avail; 1023 B/s rd, 0 B/s wr, 1.38k op/s; 2830/16806 objects degraded (16.839%) 2024-09-10T19:23:23.536 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:23:23.560 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:23:23.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:23.561 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:23:23.662 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:23:23.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:23.684 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:23:23.684 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:23:23.785 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:23:23.808 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:23:23.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:23.809 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:23:23.910 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:23:23.933 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:23:23.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:23.934 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:23:24.035 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:23:24.075 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:23:24.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:24.076 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:23:24.096 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:23:24.177 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:23:24.200 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:23:24.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:24.201 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:23:24.261 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:23:23 smithi183 ceph-mon[29736]: 4.4 scrub starts 2024-09-10T19:23:24.261 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:23:23 smithi183 ceph-mon[29736]: 4.4 scrub ok 2024-09-10T19:23:24.261 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:23:23 smithi183 ceph-mon[41562]: 4.4 scrub starts 2024-09-10T19:23:24.261 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:23:23 smithi183 ceph-mon[41562]: 4.4 scrub ok 2024-09-10T19:23:24.302 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:23:24.324 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:23:24.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:24.325 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:23:24.426 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:23:24.449 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:23:24.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:24.450 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:23:24.465 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:23:24 smithi191 ceph-mon[32884]: 4.4 scrub starts 2024-09-10T19:23:24.465 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:23:24 smithi191 ceph-mon[32884]: 4.4 scrub ok 2024-09-10T19:23:24.551 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:23:24.587 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:23:24.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:24.588 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:23:24.646 INFO:teuthology.orchestra.run.smithi183.stderr:Error ENXIO: problem getting command descriptions from osd.0 2024-09-10T19:23:24.651 INFO:teuthology.orchestra.run.smithi183.stdout:osd.1: {} 2024-09-10T19:23:24.652 INFO:teuthology.orchestra.run.smithi183.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-10T19:23:24.679 INFO:teuthology.orchestra.run.smithi183.stdout:osd.2: {} 2024-09-10T19:23:24.680 INFO:teuthology.orchestra.run.smithi183.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-10T19:23:24.688 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:23:24.702 INFO:teuthology.orchestra.run.smithi183.stdout:osd.3: {} 2024-09-10T19:23:24.703 INFO:teuthology.orchestra.run.smithi183.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-10T19:23:24.713 INFO:tasks.radosbench.radosbench.0.smithi183.stdout:Removed 146486 objects 2024-09-10T19:23:24.713 INFO:tasks.radosbench.radosbench.0.smithi183.stdout:Clean up completed and total clean up time :112.963 2024-09-10T19:23:24.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:24.714 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:23:24.714 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:23:24.724 INFO:teuthology.orchestra.run.smithi183.stdout:osd.4: {} 2024-09-10T19:23:24.724 INFO:teuthology.orchestra.run.smithi183.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-10T19:23:24.740 INFO:teuthology.orchestra.run.smithi183.stdout:osd.5: {} 2024-09-10T19:23:24.740 INFO:teuthology.orchestra.run.smithi183.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-10T19:23:24.755 INFO:teuthology.orchestra.run.smithi183.stdout:osd.6: {} 2024-09-10T19:23:24.755 INFO:teuthology.orchestra.run.smithi183.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-10T19:23:24.771 INFO:teuthology.orchestra.run.smithi183.stdout:osd.7: {} 2024-09-10T19:23:24.771 INFO:teuthology.orchestra.run.smithi183.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-10T19:23:24.814 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:23:24.836 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:23:24.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:24.837 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:23:24.937 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:23:24.959 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:23:24.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:24.960 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:23:25.061 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:23:25.084 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:23:25.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:25.085 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:23:25.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:23:25 smithi183 ceph-mon[29736]: 6.f scrub starts 2024-09-10T19:23:25.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:23:25 smithi183 ceph-mon[29736]: 6.f scrub ok 2024-09-10T19:23:25.182 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:23:25 smithi183 ceph-mon[29736]: 5.19 scrub starts 2024-09-10T19:23:25.182 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:23:25 smithi183 ceph-mon[29736]: 5.19 scrub ok 2024-09-10T19:23:25.182 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:23:25 smithi183 ceph-mon[29736]: pgmap v591: 158 pgs: 45 active+undersized, 33 active+undersized+degraded, 80 active+clean; 338 MiB data, 9.6 GiB used, 527 GiB / 536 GiB avail; 1023 B/s rd, 0 B/s wr, 1.38k op/s; 2830/16806 objects degraded (16.839%) 2024-09-10T19:23:25.182 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:23:25 smithi183 ceph-mon[29736]: 5.11 scrub starts 2024-09-10T19:23:25.182 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:23:25 smithi183 ceph-mon[29736]: 5.11 scrub ok 2024-09-10T19:23:25.182 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:23:25 smithi183 ceph-mon[29736]: from='client.? 172.21.15.183:0/1609400638' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T19:23:25.182 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:23:25 smithi183 ceph-mon[29736]: Health check update: Degraded data redundancy: 2830/16806 objects degraded (16.839%), 33 pgs degraded, 78 pgs undersized (PG_DEGRADED) 2024-09-10T19:23:25.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:23:25 smithi183 ceph-mon[41562]: 6.f scrub starts 2024-09-10T19:23:25.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:23:25 smithi183 ceph-mon[41562]: 6.f scrub ok 2024-09-10T19:23:25.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:23:25 smithi183 ceph-mon[41562]: 5.19 scrub starts 2024-09-10T19:23:25.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:23:25 smithi183 ceph-mon[41562]: 5.19 scrub ok 2024-09-10T19:23:25.183 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:23:25 smithi183 ceph-mon[41562]: pgmap v591: 158 pgs: 45 active+undersized, 33 active+undersized+degraded, 80 active+clean; 338 MiB data, 9.6 GiB used, 527 GiB / 536 GiB avail; 1023 B/s rd, 0 B/s wr, 1.38k op/s; 2830/16806 objects degraded (16.839%) 2024-09-10T19:23:25.183 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:23:25 smithi183 ceph-mon[41562]: 5.11 scrub starts 2024-09-10T19:23:25.183 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:23:25 smithi183 ceph-mon[41562]: 5.11 scrub ok 2024-09-10T19:23:25.183 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:23:25 smithi183 ceph-mon[41562]: from='client.? 172.21.15.183:0/1609400638' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T19:23:25.183 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:23:25 smithi183 ceph-mon[41562]: Health check update: Degraded data redundancy: 2830/16806 objects degraded (16.839%), 33 pgs degraded, 78 pgs undersized (PG_DEGRADED) 2024-09-10T19:23:25.185 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:23:25.207 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:23:25.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:25.208 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:23:25.309 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:23:25.332 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:23:25.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:25.333 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:23:25.433 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:23:25.457 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:23:25.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:25.458 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:23:25.465 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:23:25 smithi191 ceph-mon[32884]: 6.f scrub starts 2024-09-10T19:23:25.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:23:25 smithi191 ceph-mon[32884]: 6.f scrub ok 2024-09-10T19:23:25.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:23:25 smithi191 ceph-mon[32884]: 5.19 scrub starts 2024-09-10T19:23:25.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:23:25 smithi191 ceph-mon[32884]: 5.19 scrub ok 2024-09-10T19:23:25.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:23:25 smithi191 ceph-mon[32884]: pgmap v591: 158 pgs: 45 active+undersized, 33 active+undersized+degraded, 80 active+clean; 338 MiB data, 9.6 GiB used, 527 GiB / 536 GiB avail; 1023 B/s rd, 0 B/s wr, 1.38k op/s; 2830/16806 objects degraded (16.839%) 2024-09-10T19:23:25.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:23:25 smithi191 ceph-mon[32884]: 5.11 scrub starts 2024-09-10T19:23:25.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:23:25 smithi191 ceph-mon[32884]: 5.11 scrub ok 2024-09-10T19:23:25.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:23:25 smithi191 ceph-mon[32884]: from='client.? 172.21.15.183:0/1609400638' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T19:23:25.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:23:25 smithi191 ceph-mon[32884]: Health check update: Degraded data redundancy: 2830/16806 objects degraded (16.839%), 33 pgs degraded, 78 pgs undersized (PG_DEGRADED) 2024-09-10T19:23:25.559 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:23:25.580 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:23:25.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:25.582 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:23:25.682 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:23:25.704 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:23:25.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:25.705 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:23:25.806 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:23:25.827 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:23:25.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:25.828 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:23:25.929 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:23:25.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:25.953 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:23:25.953 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:23:26.054 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:23:26.075 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:23:26.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:26.076 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:23:26.177 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:23:26.199 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:23:26.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:26.200 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:23:26.301 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:23:26.322 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:23:26.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:26.323 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:23:26.424 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:23:26.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:23:26 smithi183 ceph-mon[29736]: 4.f scrub starts 2024-09-10T19:23:26.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:23:26 smithi183 ceph-mon[29736]: 4.f scrub ok 2024-09-10T19:23:26.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:23:26 smithi183 ceph-mon[29736]: 4.1 scrub starts 2024-09-10T19:23:26.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:23:26 smithi183 ceph-mon[29736]: 4.1 scrub ok 2024-09-10T19:23:26.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:23:26 smithi183 ceph-mon[29736]: 2.11 scrub starts 2024-09-10T19:23:26.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:23:26 smithi183 ceph-mon[29736]: 2.11 scrub ok 2024-09-10T19:23:26.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:23:26 smithi183 ceph-mon[41562]: 4.f scrub starts 2024-09-10T19:23:26.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:23:26 smithi183 ceph-mon[41562]: 4.f scrub ok 2024-09-10T19:23:26.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:23:26 smithi183 ceph-mon[41562]: 4.1 scrub starts 2024-09-10T19:23:26.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:23:26 smithi183 ceph-mon[41562]: 4.1 scrub ok 2024-09-10T19:23:26.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:23:26 smithi183 ceph-mon[41562]: 2.11 scrub starts 2024-09-10T19:23:26.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:23:26 smithi183 ceph-mon[41562]: 2.11 scrub ok 2024-09-10T19:23:26.446 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:23:26.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:26.447 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:23:26.465 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:23:26 smithi191 ceph-mon[32884]: 4.f scrub starts 2024-09-10T19:23:26.465 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:23:26 smithi191 ceph-mon[32884]: 4.f scrub ok 2024-09-10T19:23:26.465 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:23:26 smithi191 ceph-mon[32884]: 4.1 scrub starts 2024-09-10T19:23:26.465 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:23:26 smithi191 ceph-mon[32884]: 4.1 scrub ok 2024-09-10T19:23:26.465 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:23:26 smithi191 ceph-mon[32884]: 2.11 scrub starts 2024-09-10T19:23:26.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:23:26 smithi191 ceph-mon[32884]: 2.11 scrub ok 2024-09-10T19:23:26.499 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-10T19:23:26.500 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-10T19:23:26.548 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:23:26.569 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:23:26.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:26.570 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:23:26.671 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:23:26.692 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:23:26.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:26.693 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:23:26.794 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:23:26.816 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:23:26.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:26.817 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:23:26.918 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:23:26.939 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:23:26.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:26.940 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:23:27.041 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:23:27.062 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:23:27.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:27.063 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:23:27.164 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:23:27.185 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:23:27.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:27.186 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:23:27.287 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:23:27.309 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:23:27.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:27.310 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:23:27.367 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:23:27 smithi191 ceph-mon[32884]: 6.1a scrub starts 2024-09-10T19:23:27.367 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:23:27 smithi191 ceph-mon[32884]: 6.1a scrub ok 2024-09-10T19:23:27.367 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:23:27 smithi191 ceph-mon[32884]: pgmap v592: 158 pgs: 47 active+undersized, 31 active+undersized+degraded, 80 active+clean; 635 KiB data, 8.8 GiB used, 528 GiB / 536 GiB avail; 1023 B/s rd, 0 B/s wr, 1.19k op/s; 92/591 objects degraded (15.567%) 2024-09-10T19:23:27.367 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:23:27 smithi191 ceph-mon[32884]: from='mgr.14556 ' entity='mgr.y' 2024-09-10T19:23:27.367 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:23:27 smithi191 ceph-mon[32884]: 2.7 scrub starts 2024-09-10T19:23:27.367 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:23:27 smithi191 ceph-mon[32884]: 2.7 scrub ok 2024-09-10T19:23:27.367 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:23:27 smithi191 ceph-mon[32884]: from='mgr.14556 ' entity='mgr.y' 2024-09-10T19:23:27.411 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:23:27.422 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:23:27 smithi183 ceph-mon[29736]: 6.1a scrub starts 2024-09-10T19:23:27.422 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:23:27 smithi183 ceph-mon[29736]: 6.1a scrub ok 2024-09-10T19:23:27.422 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:23:27 smithi183 ceph-mon[29736]: pgmap v592: 158 pgs: 47 active+undersized, 31 active+undersized+degraded, 80 active+clean; 635 KiB data, 8.8 GiB used, 528 GiB / 536 GiB avail; 1023 B/s rd, 0 B/s wr, 1.19k op/s; 92/591 objects degraded (15.567%) 2024-09-10T19:23:27.422 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:23:27 smithi183 ceph-mon[29736]: from='mgr.14556 ' entity='mgr.y' 2024-09-10T19:23:27.422 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:23:27 smithi183 ceph-mon[29736]: 2.7 scrub starts 2024-09-10T19:23:27.422 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:23:27 smithi183 ceph-mon[29736]: 2.7 scrub ok 2024-09-10T19:23:27.422 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:23:27 smithi183 ceph-mon[29736]: from='mgr.14556 ' entity='mgr.y' 2024-09-10T19:23:27.422 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:23:27 smithi183 ceph-mon[41562]: 6.1a scrub starts 2024-09-10T19:23:27.422 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:23:27 smithi183 ceph-mon[41562]: 6.1a scrub ok 2024-09-10T19:23:27.423 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:23:27 smithi183 ceph-mon[41562]: pgmap v592: 158 pgs: 47 active+undersized, 31 active+undersized+degraded, 80 active+clean; 635 KiB data, 8.8 GiB used, 528 GiB / 536 GiB avail; 1023 B/s rd, 0 B/s wr, 1.19k op/s; 92/591 objects degraded (15.567%) 2024-09-10T19:23:27.423 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:23:27 smithi183 ceph-mon[41562]: from='mgr.14556 ' entity='mgr.y' 2024-09-10T19:23:27.423 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:23:27 smithi183 ceph-mon[41562]: 2.7 scrub starts 2024-09-10T19:23:27.423 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:23:27 smithi183 ceph-mon[41562]: 2.7 scrub ok 2024-09-10T19:23:27.423 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:23:27 smithi183 ceph-mon[41562]: from='mgr.14556 ' entity='mgr.y' 2024-09-10T19:23:27.433 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:23:27.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:27.434 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:23:27.535 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:23:27.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:27.558 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:23:27.558 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:23:27.659 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:23:27.671 INFO:journalctl@ceph.mgr.x.smithi191.stdout:Sep 10 19:23:27 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-x[34967]: ::ffff:172.21.15.191 - - [10/Sep/2024:19:23:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T19:23:27.681 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:23:27.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:27.682 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:23:27.783 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:23:27.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:27.806 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:23:27.806 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:23:27.907 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:23:27.913 INFO:tasks.thrashosds.thrasher:in_osds: [0, 3, 4, 5, 6, 7] out_osds: [1, 2] dead_osds: [0] live_osds: [2, 3, 4, 5, 6, 7, 1] 2024-09-10T19:23:27.913 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-10T19:23:27.914 INFO:tasks.thrashosds.thrasher:fixing pg num pool unique_pool_2 2024-09-10T19:23:27.914 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph pg dump --format=json 2024-09-10T19:23:27.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:27.931 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:23:27.931 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:23:28.018 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:23:28 smithi183 ceph-mon[29736]: 4.1d scrub starts 2024-09-10T19:23:28.018 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:23:28 smithi183 ceph-mon[29736]: 4.1d scrub ok 2024-09-10T19:23:28.018 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:23:28 smithi183 ceph-mon[29736]: 4.9 scrub starts 2024-09-10T19:23:28.018 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:23:28 smithi183 ceph-mon[29736]: 4.9 scrub ok 2024-09-10T19:23:28.018 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:23:28 smithi183 ceph-mon[29736]: 6.5 scrub starts 2024-09-10T19:23:28.018 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:23:28 smithi183 ceph-mon[29736]: 6.5 scrub ok 2024-09-10T19:23:28.019 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:23:28 smithi183 ceph-mon[41562]: 4.1d scrub starts 2024-09-10T19:23:28.019 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:23:28 smithi183 ceph-mon[41562]: 4.1d scrub ok 2024-09-10T19:23:28.019 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:23:28 smithi183 ceph-mon[41562]: 4.9 scrub starts 2024-09-10T19:23:28.019 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:23:28 smithi183 ceph-mon[41562]: 4.9 scrub ok 2024-09-10T19:23:28.019 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:23:28 smithi183 ceph-mon[41562]: 6.5 scrub starts 2024-09-10T19:23:28.019 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:23:28 smithi183 ceph-mon[41562]: 6.5 scrub ok 2024-09-10T19:23:28.033 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:23:28.055 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:23:28.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:28.056 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:23:28.157 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:23:28.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:28.179 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:23:28.179 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:23:28.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:23:28 smithi191 ceph-mon[32884]: 4.1d scrub starts 2024-09-10T19:23:28.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:23:28 smithi191 ceph-mon[32884]: 4.1d scrub ok 2024-09-10T19:23:28.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:23:28 smithi191 ceph-mon[32884]: 4.9 scrub starts 2024-09-10T19:23:28.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:23:28 smithi191 ceph-mon[32884]: 4.9 scrub ok 2024-09-10T19:23:28.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:23:28 smithi191 ceph-mon[32884]: 6.5 scrub starts 2024-09-10T19:23:28.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:23:28 smithi191 ceph-mon[32884]: 6.5 scrub ok 2024-09-10T19:23:28.281 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:23:28.302 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:23:28.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:28.302 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:23:28.403 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:23:28.425 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:23:28.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:28.426 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:23:28.501 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-10T19:23:28.513 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:23:28.527 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:23:28.548 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:23:28.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:28.549 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:23:28.650 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:23:28.673 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:23:28.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:28.674 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:23:28.775 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:23:28.797 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:23:28.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:28.798 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:23:28.899 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:23:28.920 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:23:28.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:28.921 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:23:29.021 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:23:29.030 INFO:teuthology.orchestra.run.smithi183.stderr:noscrub is set 2024-09-10T19:23:29.043 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:23:29.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:29.044 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:23:29.121 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:23:29 smithi183 ceph-mon[29736]: pgmap v593: 158 pgs: 47 active+undersized, 31 active+undersized+degraded, 80 active+clean; 635 KiB data, 8.8 GiB used, 528 GiB / 536 GiB avail; 1023 B/s rd, 0 B/s wr, 1.19k op/s; 92/591 objects degraded (15.567%) 2024-09-10T19:23:29.121 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:23:29 smithi183 ceph-mon[29736]: from='client.? 172.21.15.183:0/856217242' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-10T19:23:29.121 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:23:29 smithi183 ceph-mon[29736]: 4.5 scrub starts 2024-09-10T19:23:29.121 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:23:29 smithi183 ceph-mon[29736]: 4.5 scrub ok 2024-09-10T19:23:29.122 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:23:29 smithi183 ceph-mon[29736]: from='mgr.14556 ' entity='mgr.y' 2024-09-10T19:23:29.122 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:23:29 smithi183 ceph-mon[29736]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:23:29.122 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:23:29 smithi183 ceph-mon[41562]: pgmap v593: 158 pgs: 47 active+undersized, 31 active+undersized+degraded, 80 active+clean; 635 KiB data, 8.8 GiB used, 528 GiB / 536 GiB avail; 1023 B/s rd, 0 B/s wr, 1.19k op/s; 92/591 objects degraded (15.567%) 2024-09-10T19:23:29.122 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:23:29 smithi183 ceph-mon[41562]: from='client.? 172.21.15.183:0/856217242' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-10T19:23:29.122 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:23:29 smithi183 ceph-mon[41562]: 4.5 scrub starts 2024-09-10T19:23:29.122 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:23:29 smithi183 ceph-mon[41562]: 4.5 scrub ok 2024-09-10T19:23:29.122 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:23:29 smithi183 ceph-mon[41562]: from='mgr.14556 ' entity='mgr.y' 2024-09-10T19:23:29.122 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:23:29 smithi183 ceph-mon[41562]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:23:29.145 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:23:29.166 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:23:29.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:29.167 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:23:29.268 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:23:29.289 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:23:29.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:29.290 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:23:29.391 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:23:29.413 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:23:29.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:29.414 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:23:29.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:23:29 smithi191 ceph-mon[32884]: pgmap v593: 158 pgs: 47 active+undersized, 31 active+undersized+degraded, 80 active+clean; 635 KiB data, 8.8 GiB used, 528 GiB / 536 GiB avail; 1023 B/s rd, 0 B/s wr, 1.19k op/s; 92/591 objects degraded (15.567%) 2024-09-10T19:23:29.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:23:29 smithi191 ceph-mon[32884]: from='client.? 172.21.15.183:0/856217242' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-10T19:23:29.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:23:29 smithi191 ceph-mon[32884]: 4.5 scrub starts 2024-09-10T19:23:29.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:23:29 smithi191 ceph-mon[32884]: 4.5 scrub ok 2024-09-10T19:23:29.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:23:29 smithi191 ceph-mon[32884]: from='mgr.14556 ' entity='mgr.y' 2024-09-10T19:23:29.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:23:29 smithi191 ceph-mon[32884]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:23:29.515 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:23:29.536 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:23:29.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:29.537 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:23:29.638 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:23:29.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:29.661 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:23:29.662 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:23:29.762 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:23:29.784 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:23:29.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:29.785 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:23:29.886 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:23:29.908 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:23:29.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:29.909 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:23:29.965 INFO:journalctl@ceph.iscsi.iscsi.a.smithi191.stdout:Sep 10 19:23:29 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-iscsi-iscsi-a[61816]: debug there is no tcmu-runner data available 2024-09-10T19:23:30.010 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:23:30.032 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:23:30.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:30.033 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:23:30.134 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:23:30.157 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:23:30.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:30.158 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:23:30.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:23:30 smithi183 ceph-mon[29736]: 3.2 scrub starts 2024-09-10T19:23:30.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:23:30 smithi183 ceph-mon[29736]: 3.2 scrub ok 2024-09-10T19:23:30.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:23:30 smithi183 ceph-mon[29736]: 5.6 deep-scrub starts 2024-09-10T19:23:30.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:23:30 smithi183 ceph-mon[29736]: 5.6 deep-scrub ok 2024-09-10T19:23:30.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:23:30 smithi183 ceph-mon[29736]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T19:23:30.182 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:23:30 smithi183 ceph-mon[29736]: from='client.? 172.21.15.183:0/856217242' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-10T19:23:30.182 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:23:30 smithi183 ceph-mon[29736]: osdmap e282: 8 total, 7 up, 6 in 2024-09-10T19:23:30.182 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:23:30 smithi183 ceph-mon[29736]: Health check update: Degraded data redundancy: 92/591 objects degraded (15.567%), 31 pgs degraded, 78 pgs undersized (PG_DEGRADED) 2024-09-10T19:23:30.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:23:30 smithi183 ceph-mon[41562]: 3.2 scrub starts 2024-09-10T19:23:30.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:23:30 smithi183 ceph-mon[41562]: 3.2 scrub ok 2024-09-10T19:23:30.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:23:30 smithi183 ceph-mon[41562]: 5.6 deep-scrub starts 2024-09-10T19:23:30.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:23:30 smithi183 ceph-mon[41562]: 5.6 deep-scrub ok 2024-09-10T19:23:30.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:23:30 smithi183 ceph-mon[41562]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T19:23:30.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:23:30 smithi183 ceph-mon[41562]: from='client.? 172.21.15.183:0/856217242' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-10T19:23:30.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:23:30 smithi183 ceph-mon[41562]: osdmap e282: 8 total, 7 up, 6 in 2024-09-10T19:23:30.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:23:30 smithi183 ceph-mon[41562]: Health check update: Degraded data redundancy: 92/591 objects degraded (15.567%), 31 pgs degraded, 78 pgs undersized (PG_DEGRADED) 2024-09-10T19:23:30.258 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:23:30.282 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:23:30.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:30.283 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:23:30.384 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:23:30.406 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:23:30.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:30.407 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:23:30.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:23:30 smithi191 ceph-mon[32884]: 3.2 scrub starts 2024-09-10T19:23:30.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:23:30 smithi191 ceph-mon[32884]: 3.2 scrub ok 2024-09-10T19:23:30.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:23:30 smithi191 ceph-mon[32884]: 5.6 deep-scrub starts 2024-09-10T19:23:30.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:23:30 smithi191 ceph-mon[32884]: 5.6 deep-scrub ok 2024-09-10T19:23:30.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:23:30 smithi191 ceph-mon[32884]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T19:23:30.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:23:30 smithi191 ceph-mon[32884]: from='client.? 172.21.15.183:0/856217242' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-10T19:23:30.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:23:30 smithi191 ceph-mon[32884]: osdmap e282: 8 total, 7 up, 6 in 2024-09-10T19:23:30.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:23:30 smithi191 ceph-mon[32884]: Health check update: Degraded data redundancy: 92/591 objects degraded (15.567%), 31 pgs degraded, 78 pgs undersized (PG_DEGRADED) 2024-09-10T19:23:30.508 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:23:30.531 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:23:30.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:30.533 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:23:30.634 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:23:30.658 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:23:30.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:30.659 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:23:30.759 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:23:30.782 INFO:teuthology.orchestra.run.smithi183.stdout:ERROR: (22) Invalid argument 2024-09-10T19:23:30.782 INFO:teuthology.orchestra.run.smithi183.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-10T19:23:30.783 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:23:30.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:30.784 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:23:30.885 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:23:30.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:30.910 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:23:30.910 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:23:31.011 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:23:31.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:31.038 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:23:31.038 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:23:31.139 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:23:31.161 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:23:31.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:31.162 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:23:31.262 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:23:31.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:31.286 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:23:31.286 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:23:31.315 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:23:31 smithi183 ceph-mon[29736]: from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:23:31.316 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:23:31 smithi183 ceph-mon[29736]: pgmap v595: 158 pgs: 47 active+undersized, 31 active+undersized+degraded, 80 active+clean; 635 KiB data, 8.8 GiB used, 528 GiB / 536 GiB avail; 1023 B/s rd, 0 B/s wr, 1.19k op/s; 92/591 objects degraded (15.567%) 2024-09-10T19:23:31.316 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:23:31 smithi183 ceph-mon[41562]: from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:23:31.316 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:23:31 smithi183 ceph-mon[41562]: pgmap v595: 158 pgs: 47 active+undersized, 31 active+undersized+degraded, 80 active+clean; 635 KiB data, 8.8 GiB used, 528 GiB / 536 GiB avail; 1023 B/s rd, 0 B/s wr, 1.19k op/s; 92/591 objects degraded (15.567%) 2024-09-10T19:23:31.387 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:23:31.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:31.409 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:23:31.410 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:23:31.465 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:23:31 smithi191 ceph-mon[32884]: from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:23:31.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:23:31 smithi191 ceph-mon[32884]: pgmap v595: 158 pgs: 47 active+undersized, 31 active+undersized+degraded, 80 active+clean; 635 KiB data, 8.8 GiB used, 528 GiB / 536 GiB avail; 1023 B/s rd, 0 B/s wr, 1.19k op/s; 92/591 objects degraded (15.567%) 2024-09-10T19:23:31.510 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:23:31.536 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:23:31.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:31.537 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:23:31.638 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:23:31.663 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:23:31.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:31.664 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:23:31.665 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Sep 10 19:23:31 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-y[29989]: ::ffff:172.21.15.191 - - [10/Sep/2024:19:23:31] "GET /metrics HTTP/1.1" 200 224270 "" "Prometheus/2.43.0" 2024-09-10T19:23:31.765 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:23:31.791 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:23:31.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:31.792 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:23:31.828 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-10T19:23:31.892 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:23:31.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:31.919 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:23:31.920 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:23:32.020 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:23:32.045 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:23:32.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:32.046 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:23:32.062 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph daemon osd.2 dump_historic_ops 2024-09-10T19:23:32.147 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:23:32.173 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:23:32.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:32.174 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:23:32.275 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:23:32.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:32.299 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:23:32.299 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:23:32.326 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:23:32 smithi183 ceph-mon[29736]: purged_snaps scrub starts 2024-09-10T19:23:32.326 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:23:32 smithi183 ceph-mon[29736]: purged_snaps scrub ok 2024-09-10T19:23:32.326 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:23:32 smithi183 ceph-mon[41562]: purged_snaps scrub starts 2024-09-10T19:23:32.326 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:23:32 smithi183 ceph-mon[41562]: purged_snaps scrub ok 2024-09-10T19:23:32.400 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:23:32.425 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:23:32.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:32.426 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:23:32.465 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:23:32 smithi191 ceph-mon[32884]: purged_snaps scrub starts 2024-09-10T19:23:32.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:23:32 smithi191 ceph-mon[32884]: purged_snaps scrub ok 2024-09-10T19:23:32.527 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:23:32.549 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:23:32.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:32.550 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:23:32.650 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:23:32.676 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:23:32.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:32.677 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:23:32.778 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:23:32.802 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:23:32.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:32.803 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:23:32.904 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:23:32.929 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:23:32.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:32.930 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:23:33.031 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:23:33.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:33.053 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:23:33.053 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:23:33.154 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:23:33.179 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:23:33.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:33.180 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:23:33.281 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:23:33.306 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:23:33.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:33.307 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:23:33.408 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:23:33.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:23:33 smithi183 ceph-mon[29736]: pgmap v596: 158 pgs: 53 active+undersized, 25 active+undersized+degraded, 80 active+clean; 635 KiB data, 8.7 GiB used, 528 GiB / 536 GiB avail; 1023 B/s rd, 0 B/s wr, 541 op/s; 92/591 objects degraded (15.567%) 2024-09-10T19:23:33.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:23:33 smithi183 ceph-mon[29736]: purged_snaps scrub starts 2024-09-10T19:23:33.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:23:33 smithi183 ceph-mon[29736]: purged_snaps scrub ok 2024-09-10T19:23:33.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:23:33 smithi183 ceph-mon[29736]: osdmap e283: 8 total, 7 up, 6 in 2024-09-10T19:23:33.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:23:33 smithi183 ceph-mon[29736]: from='mgr.14556 ' entity='mgr.y' 2024-09-10T19:23:33.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:23:33 smithi183 ceph-mon[29736]: from='mgr.14556 ' entity='mgr.y' 2024-09-10T19:23:33.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:23:33 smithi183 ceph-mon[41562]: pgmap v596: 158 pgs: 53 active+undersized, 25 active+undersized+degraded, 80 active+clean; 635 KiB data, 8.7 GiB used, 528 GiB / 536 GiB avail; 1023 B/s rd, 0 B/s wr, 541 op/s; 92/591 objects degraded (15.567%) 2024-09-10T19:23:33.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:23:33 smithi183 ceph-mon[41562]: purged_snaps scrub starts 2024-09-10T19:23:33.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:23:33 smithi183 ceph-mon[41562]: purged_snaps scrub ok 2024-09-10T19:23:33.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:23:33 smithi183 ceph-mon[41562]: osdmap e283: 8 total, 7 up, 6 in 2024-09-10T19:23:33.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:23:33 smithi183 ceph-mon[41562]: from='mgr.14556 ' entity='mgr.y' 2024-09-10T19:23:33.433 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:23:33 smithi183 ceph-mon[41562]: from='mgr.14556 ' entity='mgr.y' 2024-09-10T19:23:33.433 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:23:33.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:33.434 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:23:33.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:23:33 smithi191 ceph-mon[32884]: pgmap v596: 158 pgs: 53 active+undersized, 25 active+undersized+degraded, 80 active+clean; 635 KiB data, 8.7 GiB used, 528 GiB / 536 GiB avail; 1023 B/s rd, 0 B/s wr, 541 op/s; 92/591 objects degraded (15.567%) 2024-09-10T19:23:33.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:23:33 smithi191 ceph-mon[32884]: purged_snaps scrub starts 2024-09-10T19:23:33.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:23:33 smithi191 ceph-mon[32884]: purged_snaps scrub ok 2024-09-10T19:23:33.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:23:33 smithi191 ceph-mon[32884]: osdmap e283: 8 total, 7 up, 6 in 2024-09-10T19:23:33.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:23:33 smithi191 ceph-mon[32884]: from='mgr.14556 ' entity='mgr.y' 2024-09-10T19:23:33.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:23:33 smithi191 ceph-mon[32884]: from='mgr.14556 ' entity='mgr.y' 2024-09-10T19:23:33.496 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:23:33.510 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:23:33.535 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:23:33.557 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:23:33.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:33.558 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:23:33.659 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:23:33.681 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:23:33.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:33.682 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:23:33.783 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:23:33.809 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:23:33.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:33.810 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:23:33.911 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:23:33.933 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:23:33.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:33.934 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:23:34.035 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:23:34.046 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:23:34 smithi183 ceph-mon[29736]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T19:23:34.046 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:23:34 smithi183 ceph-mon[29736]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T19:23:34.046 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:23:34 smithi183 ceph-mon[29736]: from='mgr.14556 ' entity='mgr.y' 2024-09-10T19:23:34.046 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:23:34 smithi183 ceph-mon[41562]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T19:23:34.046 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:23:34 smithi183 ceph-mon[41562]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T19:23:34.046 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:23:34 smithi183 ceph-mon[41562]: from='mgr.14556 ' entity='mgr.y' 2024-09-10T19:23:34.059 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:23:34.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:34.060 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:23:34.161 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:23:34.183 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:23:34.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:34.185 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:23:34.285 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:23:34.307 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:23:34.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:34.308 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:23:34.409 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:23:34.434 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:23:34.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:34.435 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:23:34.465 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:23:34 smithi191 ceph-mon[32884]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T19:23:34.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:23:34 smithi191 ceph-mon[32884]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T19:23:34.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:23:34 smithi191 ceph-mon[32884]: from='mgr.14556 ' entity='mgr.y' 2024-09-10T19:23:34.536 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:23:34.562 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:23:34.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:34.563 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:23:34.664 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:23:34.687 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:23:34.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:34.688 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:23:34.788 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:23:34.810 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:23:34.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:34.811 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:23:34.912 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:23:34.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:34.934 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:23:34.934 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:23:34.998 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-09-10T19:23:35.001 INFO:teuthology.orchestra.run.smithi183.stderr:dumped all 2024-09-10T19:23:35.035 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:23:35.056 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:23:35.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:35.057 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:23:35.158 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:23:35.180 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:23:35.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:35.181 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:23:35.269 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:23:35 smithi183 ceph-mon[29736]: pgmap v598: 158 pgs: 53 active+undersized, 25 active+undersized+degraded, 80 active+clean; 635 KiB data, 8.7 GiB used, 528 GiB / 536 GiB avail; 639 B/s rd, 0 op/s; 92/591 objects degraded (15.567%) 2024-09-10T19:23:35.269 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:23:35 smithi183 ceph-mon[29736]: Health check update: Degraded data redundancy: 92/591 objects degraded (15.567%), 25 pgs degraded, 78 pgs undersized (PG_DEGRADED) 2024-09-10T19:23:35.269 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:23:35 smithi183 ceph-mon[41562]: pgmap v598: 158 pgs: 53 active+undersized, 25 active+undersized+degraded, 80 active+clean; 635 KiB data, 8.7 GiB used, 528 GiB / 536 GiB avail; 639 B/s rd, 0 op/s; 92/591 objects degraded (15.567%) 2024-09-10T19:23:35.269 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:23:35 smithi183 ceph-mon[41562]: Health check update: Degraded data redundancy: 92/591 objects degraded (15.567%), 25 pgs degraded, 78 pgs undersized (PG_DEGRADED) 2024-09-10T19:23:35.282 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:23:35.308 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:23:35.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:35.309 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:23:35.410 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:23:35.432 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:23:35.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:35.433 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:23:35.465 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:23:35 smithi191 ceph-mon[32884]: pgmap v598: 158 pgs: 53 active+undersized, 25 active+undersized+degraded, 80 active+clean; 635 KiB data, 8.7 GiB used, 528 GiB / 536 GiB avail; 639 B/s rd, 0 op/s; 92/591 objects degraded (15.567%) 2024-09-10T19:23:35.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:23:35 smithi191 ceph-mon[32884]: Health check update: Degraded data redundancy: 92/591 objects degraded (15.567%), 25 pgs degraded, 78 pgs undersized (PG_DEGRADED) 2024-09-10T19:23:35.533 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:23:35.559 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:23:35.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:35.560 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:23:35.661 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:23:35.686 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:23:35.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:35.687 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:23:35.787 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:23:35.811 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:23:35.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:35.812 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:23:35.861 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:23:35.913 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:23:35.939 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:23:35.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:35.940 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:23:36.040 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:23:36.064 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:23:36.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:36.065 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:23:36.166 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:23:36.190 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:23:36.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:36.191 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:23:36.291 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:23:36.317 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:23:36.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:36.318 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:23:36.419 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:23:36.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:23:36 smithi183 ceph-mon[29736]: from='client.25532 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-10T19:23:36.431 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:23:36 smithi183 ceph-mon[41562]: from='client.25532 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-10T19:23:36.444 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:23:36.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:36.445 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:23:36.465 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:23:36 smithi191 ceph-mon[32884]: from='client.25532 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-10T19:23:36.546 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:23:36.572 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:23:36.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:36.573 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:23:36.674 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:23:36.700 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:23:36.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:36.701 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:23:36.801 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:23:36.827 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:23:36.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:36.828 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:23:36.929 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:23:36.954 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:23:36.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:36.955 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:23:37.056 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:23:37.082 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:23:37.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:37.083 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:23:37.184 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:23:37.210 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:23:37.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:37.211 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:23:37.312 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:23:37.337 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:23:37.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:37.338 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:23:37.386 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:23:37 smithi183 ceph-mon[29736]: pgmap v599: 158 pgs: 53 active+undersized, 25 active+undersized+degraded, 80 active+clean; 635 KiB data, 8.7 GiB used, 528 GiB / 536 GiB avail; 1.2 KiB/s rd, 1 op/s; 92/591 objects degraded (15.567%) 2024-09-10T19:23:37.386 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:23:37 smithi183 ceph-mon[41562]: pgmap v599: 158 pgs: 53 active+undersized, 25 active+undersized+degraded, 80 active+clean; 635 KiB data, 8.7 GiB used, 528 GiB / 536 GiB avail; 1.2 KiB/s rd, 1 op/s; 92/591 objects degraded (15.567%) 2024-09-10T19:23:37.387 INFO:teuthology.orchestra.run.smithi183.stdout:{"pg_ready":true,"pg_map":{"version":598,"stamp":"2024-09-10T19:23:33.842218+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":649858,"num_objects":197,"num_object_clones":0,"num_object_copies":591,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":92,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":197,"num_whiteouts":0,"num_read":1586,"num_read_kb":1347,"num_write":293701,"num_write_kb":9375296,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":188,"num_bytes_recovered":57769,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":62102,"ondisk_log_size":62102,"up":396,"acting":396,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":474,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":562470912,"kb_used":9133640,"kb_used_data":4065280,"kb_used_omap":9,"kb_used_meta":5068278,"kb_avail":553337272,"statfs":{"total":575970213888,"available":566617366528,"internally_reserved":0,"allocated":4162846720,"data_stored":4160557777,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":10224,"internal_metadata":5189916688},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.003230"},"pg_stats":[{"pgid":"6.1b","version":"0'0","reported_seq":266,"reported_epoch":281,"state":"active+clean","last_fresh":"2024-09-10T19:23:18.985840+0000","last_change":"2024-09-10T19:21:50.786696+0000","last_active":"2024-09-10T19:23:18.985840+0000","last_peered":"2024-09-10T19:23:18.985840+0000","last_clean":"2024-09-10T19:23:18.985840+0000","last_became_active":"2024-09-10T19:10:14.056187+0000","last_became_peered":"2024-09-10T19:10:14.056187+0000","last_unstale":"2024-09-10T19:23:18.985840+0000","last_undegraded":"2024-09-10T19:23:18.985840+0000","last_fullsized":"2024-09-10T19:23:18.985840+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:21:50.786645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:21:50.786645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0043016390000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":164,"reported_epoch":281,"state":"active+undersized","last_fresh":"2024-09-10T19:23:18.993033+0000","last_change":"2024-09-10T19:21:47.983532+0000","last_active":"2024-09-10T19:23:18.993033+0000","last_peered":"2024-09-10T19:23:18.993033+0000","last_clean":"2024-09-10T19:16:23.155764+0000","last_became_active":"2024-09-10T19:21:47.983532+0000","last_became_peered":"2024-09-10T19:21:47.983532+0000","last_unstale":"2024-09-10T19:23:18.993033+0000","last_undegraded":"2024-09-10T19:23:18.993033+0000","last_fullsized":"2024-09-10T19:21:47.980212+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:20:05.748517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:16:38.194472+0000","last_clean_scrub_stamp":"2024-09-10T19:20:05.748517+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.023972094999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":["7","4"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.19","version":"237'27","reported_seq":336,"reported_epoch":281,"state":"active+undersized+degraded","last_fresh":"2024-09-10T19:23:18.986105+0000","last_change":"2024-09-10T19:21:47.993574+0000","last_active":"2024-09-10T19:23:18.986105+0000","last_peered":"2024-09-10T19:23:18.986105+0000","last_clean":"2024-09-10T19:21:43.944958+0000","last_became_active":"2024-09-10T19:21:47.993574+0000","last_became_peered":"2024-09-10T19:21:47.993574+0000","last_unstale":"2024-09-10T19:23:18.986105+0000","last_undegraded":"2024-09-10T19:21:47.976303+0000","last_fullsized":"2024-09-10T19:21:47.975840+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'15","last_scrub_stamp":"2024-09-10T19:19:21.244506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:19:21.244506+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.014405133000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":["3","5"],"object_location_counts":[{"shards":"3,5","objects":7}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":310,"reported_epoch":281,"state":"active+clean","last_fresh":"2024-09-10T19:23:18.992316+0000","last_change":"2024-09-10T19:23:09.564415+0000","last_active":"2024-09-10T19:23:18.992316+0000","last_peered":"2024-09-10T19:23:18.992316+0000","last_clean":"2024-09-10T19:23:18.992316+0000","last_became_active":"2024-09-10T19:16:24.845111+0000","last_became_peered":"2024-09-10T19:16:24.845111+0000","last_unstale":"2024-09-10T19:23:18.992316+0000","last_undegraded":"2024-09-10T19:23:18.992316+0000","last_fullsized":"2024-09-10T19:23:18.992316+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:23:09.564335+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:19:28.498560+0000","last_clean_scrub_stamp":"2024-09-10T19:23:09.564335+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:24:30.271414+0000","scrub_duration":0.0051696729999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,7],"acting":[4,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"59'1","reported_seq":287,"reported_epoch":281,"state":"active+undersized+degraded","last_fresh":"2024-09-10T19:23:18.986505+0000","last_change":"2024-09-10T19:21:47.993529+0000","last_active":"2024-09-10T19:23:18.986505+0000","last_peered":"2024-09-10T19:23:18.986505+0000","last_clean":"2024-09-10T19:21:43.944752+0000","last_became_active":"2024-09-10T19:21:47.993529+0000","last_became_peered":"2024-09-10T19:21:47.993529+0000","last_unstale":"2024-09-10T19:23:18.986505+0000","last_undegraded":"2024-09-10T19:21:47.974676+0000","last_fullsized":"2024-09-10T19:21:47.974527+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'1","last_scrub_stamp":"2024-09-10T19:20:56.540653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:20:56.540653+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.010998005,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":874,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":["3","5"],"object_location_counts":[{"shards":"3,5","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"237'15","reported_seq":334,"reported_epoch":281,"state":"active+clean","last_fresh":"2024-09-10T19:23:18.992387+0000","last_change":"2024-09-10T19:23:17.691402+0000","last_active":"2024-09-10T19:23:18.992387+0000","last_peered":"2024-09-10T19:23:18.992387+0000","last_clean":"2024-09-10T19:23:18.992387+0000","last_became_active":"2024-09-10T19:13:54.716117+0000","last_became_peered":"2024-09-10T19:13:54.716117+0000","last_unstale":"2024-09-10T19:23:18.992387+0000","last_undegraded":"2024-09-10T19:23:18.992387+0000","last_fullsized":"2024-09-10T19:23:18.992387+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"237'15","last_scrub_stamp":"2024-09-10T19:23:17.691323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:23:17.691323+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:24:27.667210+0000","scrub_duration":0.0046562740000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,3],"acting":[4,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":242,"reported_epoch":281,"state":"active+clean","last_fresh":"2024-09-10T19:23:23.819106+0000","last_change":"2024-09-10T19:23:23.819051+0000","last_active":"2024-09-10T19:23:23.819106+0000","last_peered":"2024-09-10T19:23:23.819106+0000","last_clean":"2024-09-10T19:23:23.819106+0000","last_became_active":"2024-09-10T19:16:24.859261+0000","last_became_peered":"2024-09-10T19:16:24.859261+0000","last_unstale":"2024-09-10T19:23:23.819106+0000","last_undegraded":"2024-09-10T19:23:23.819106+0000","last_fullsized":"2024-09-10T19:23:23.819106+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:23:23.816921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:23:23.816921+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:24:26.706556+0000","scrub_duration":0.0075702879999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":298,"reported_epoch":281,"state":"active+clean","last_fresh":"2024-09-10T19:23:25.762899+0000","last_change":"2024-09-10T19:23:25.762839+0000","last_active":"2024-09-10T19:23:25.762899+0000","last_peered":"2024-09-10T19:23:25.762899+0000","last_clean":"2024-09-10T19:23:25.762899+0000","last_became_active":"2024-09-10T19:13:54.713538+0000","last_became_peered":"2024-09-10T19:13:54.713538+0000","last_unstale":"2024-09-10T19:23:25.762899+0000","last_undegraded":"2024-09-10T19:23:25.762899+0000","last_fullsized":"2024-09-10T19:23:25.762899+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:23:25.762778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:23:25.762778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:24:48.915001+0000","scrub_duration":0.001257326,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,7],"acting":[4,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":279,"reported_epoch":281,"state":"active+undersized","last_fresh":"2024-09-10T19:23:18.996063+0000","last_change":"2024-09-10T19:21:47.987922+0000","last_active":"2024-09-10T19:23:18.996063+0000","last_peered":"2024-09-10T19:23:18.996063+0000","last_clean":"2024-09-10T19:21:43.955497+0000","last_became_active":"2024-09-10T19:21:47.987922+0000","last_became_peered":"2024-09-10T19:21:47.987922+0000","last_unstale":"2024-09-10T19:23:18.996063+0000","last_undegraded":"2024-09-10T19:23:18.996063+0000","last_fullsized":"2024-09-10T19:21:47.983715+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:20:45.770926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:20:45.770926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0040372419999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":["7","6"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1b","version":"237'7","reported_seq":324,"reported_epoch":281,"state":"active+clean","last_fresh":"2024-09-10T19:23:18.993770+0000","last_change":"2024-09-10T19:22:28.987340+0000","last_active":"2024-09-10T19:23:18.993770+0000","last_peered":"2024-09-10T19:23:18.993770+0000","last_clean":"2024-09-10T19:23:18.993770+0000","last_became_active":"2024-09-10T19:16:24.843708+0000","last_became_peered":"2024-09-10T19:16:24.843708+0000","last_unstale":"2024-09-10T19:23:18.993770+0000","last_undegraded":"2024-09-10T19:23:18.993770+0000","last_fullsized":"2024-09-10T19:23:18.993770+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"237'7","last_scrub_stamp":"2024-09-10T19:22:28.987257+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:22:28.987257+0000","objects_scrubbed":2,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:23:36.739968+0000","scrub_duration":0.0078490939999999992,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":10,"num_write":8,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":282,"reported_epoch":281,"state":"active+undersized","last_fresh":"2024-09-10T19:23:18.995290+0000","last_change":"2024-09-10T19:21:47.984833+0000","last_active":"2024-09-10T19:23:18.995290+0000","last_peered":"2024-09-10T19:23:18.995290+0000","last_clean":"2024-09-10T19:21:43.956152+0000","last_became_active":"2024-09-10T19:21:47.984833+0000","last_became_peered":"2024-09-10T19:21:47.984833+0000","last_unstale":"2024-09-10T19:23:18.995290+0000","last_undegraded":"2024-09-10T19:23:18.995290+0000","last_fullsized":"2024-09-10T19:21:47.979042+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:20:03.903546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:20:03.903546+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0066943719999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":["7","4"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":290,"reported_epoch":281,"state":"active+undersized","last_fresh":"2024-09-10T19:23:18.988637+0000","last_change":"2024-09-10T19:21:47.983411+0000","last_active":"2024-09-10T19:23:18.988637+0000","last_peered":"2024-09-10T19:23:18.988637+0000","last_clean":"2024-09-10T19:21:43.956084+0000","last_became_active":"2024-09-10T19:21:47.983411+0000","last_became_peered":"2024-09-10T19:21:47.983411+0000","last_unstale":"2024-09-10T19:23:18.988637+0000","last_undegraded":"2024-09-10T19:23:18.988637+0000","last_fullsized":"2024-09-10T19:21:47.973796+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:20:50.336432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:20:50.336432+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0077486569999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":["5","3"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":292,"reported_epoch":281,"state":"active+clean","last_fresh":"2024-09-10T19:23:18.994154+0000","last_change":"2024-09-10T19:22:38.275940+0000","last_active":"2024-09-10T19:23:18.994154+0000","last_peered":"2024-09-10T19:23:18.994154+0000","last_clean":"2024-09-10T19:23:18.994154+0000","last_became_active":"2024-09-10T19:16:24.858800+0000","last_became_peered":"2024-09-10T19:16:24.858800+0000","last_unstale":"2024-09-10T19:23:18.994154+0000","last_undegraded":"2024-09-10T19:23:18.994154+0000","last_fullsized":"2024-09-10T19:23:18.994154+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:22:38.275863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:22:38.275863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:23:44.801164+0000","scrub_duration":0.010525915,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1a","version":"237'15","reported_seq":304,"reported_epoch":281,"state":"active+undersized+degraded","last_fresh":"2024-09-10T19:23:18.993320+0000","last_change":"2024-09-10T19:21:47.978079+0000","last_active":"2024-09-10T19:23:18.993320+0000","last_peered":"2024-09-10T19:23:18.993320+0000","last_clean":"2024-09-10T19:21:43.946995+0000","last_became_active":"2024-09-10T19:21:47.978079+0000","last_became_peered":"2024-09-10T19:21:47.978079+0000","last_unstale":"2024-09-10T19:23:18.993320+0000","last_undegraded":"2024-09-10T19:21:47.971706+0000","last_fullsized":"2024-09-10T19:21:47.971602+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"237'15","last_scrub_stamp":"2024-09-10T19:21:42.567473+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:21:42.567473+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0065782230000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":["4","3"],"object_location_counts":[{"shards":"3,4","objects":4}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":275,"reported_epoch":281,"state":"active+undersized","last_fresh":"2024-09-10T19:23:18.988669+0000","last_change":"2024-09-10T19:21:47.997166+0000","last_active":"2024-09-10T19:23:18.988669+0000","last_peered":"2024-09-10T19:23:18.988669+0000","last_clean":"2024-09-10T19:21:43.951146+0000","last_became_active":"2024-09-10T19:21:47.997166+0000","last_became_peered":"2024-09-10T19:21:47.997166+0000","last_unstale":"2024-09-10T19:23:18.988669+0000","last_undegraded":"2024-09-10T19:23:18.988669+0000","last_fullsized":"2024-09-10T19:21:47.975077+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:21:00.322381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:21:00.322381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.008205258,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":["5","7"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":165,"reported_epoch":281,"state":"active+undersized","last_fresh":"2024-09-10T19:23:18.991851+0000","last_change":"2024-09-10T19:21:47.983632+0000","last_active":"2024-09-10T19:23:18.991851+0000","last_peered":"2024-09-10T19:23:18.991851+0000","last_clean":"2024-09-10T19:16:23.155082+0000","last_became_active":"2024-09-10T19:21:47.983632+0000","last_became_peered":"2024-09-10T19:21:47.983632+0000","last_unstale":"2024-09-10T19:23:18.991851+0000","last_undegraded":"2024-09-10T19:23:18.991851+0000","last_fullsized":"2024-09-10T19:21:47.978395+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:21:05.690833+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:21:05.690833+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0032019980000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":["7","4"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.8","version":"74'79","reported_seq":62,"reported_epoch":281,"state":"active+clean","last_fresh":"2024-09-10T19:23:18.987731+0000","last_change":"2024-09-10T19:21:54.355335+0000","last_active":"2024-09-10T19:23:18.987731+0000","last_peered":"2024-09-10T19:23:18.987731+0000","last_clean":"2024-09-10T19:23:18.987731+0000","last_became_active":"2024-09-10T19:20:30.839050+0000","last_became_peered":"2024-09-10T19:20:30.839050+0000","last_unstale":"2024-09-10T19:23:18.987731+0000","last_undegraded":"2024-09-10T19:23:18.987731+0000","last_fullsized":"2024-09-10T19:23:18.987731+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":4,"last_scrub":"74'79","last_scrub_stamp":"2024-09-10T19:21:54.355151+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:08:00.699315+0000","last_clean_scrub_stamp":"2024-09-10T19:21:54.355151+0000","objects_scrubbed":1,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.014582322999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.0","version":"281'18286","reported_seq":18359,"reported_epoch":281,"state":"active+undersized","last_fresh":"2024-09-10T19:23:24.686395+0000","last_change":"2024-09-10T19:23:29.209801+0000","last_active":"2024-09-10T19:23:24.686395+0000","last_peered":"2024-09-10T19:23:24.686395+0000","last_clean":"2024-09-10T19:21:46.973997+0000","last_became_active":"2024-09-10T19:21:48.010936+0000","last_became_peered":"2024-09-10T19:21:48.010936+0000","last_unstale":"2024-09-10T19:23:24.686395+0000","last_undegraded":"2024-09-10T19:21:47.994156+0000","last_fullsized":"2024-09-10T19:21:47.981767+0000","mapping_epoch":263,"log_start":"273'14400","ondisk_log_start":"273'14400","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:19:53.174811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:19:53.174811+0000","last_clean_scrub_stamp":"2024-09-10T19:19:53.174811+0000","objects_scrubbed":0,"log_size":3886,"log_dups_size":0,"ondisk_log_size":3886,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18286,"num_write_kb":585152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":["6","5"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"237'31","reported_seq":388,"reported_epoch":281,"state":"active+undersized+degraded","last_fresh":"2024-09-10T19:23:18.996762+0000","last_change":"2024-09-10T19:21:47.986499+0000","last_active":"2024-09-10T19:23:18.996762+0000","last_peered":"2024-09-10T19:23:18.996762+0000","last_clean":"2024-09-10T19:21:43.948925+0000","last_became_active":"2024-09-10T19:21:47.986499+0000","last_became_peered":"2024-09-10T19:21:47.986499+0000","last_unstale":"2024-09-10T19:23:18.996762+0000","last_undegraded":"2024-09-10T19:21:47.972336+0000","last_fullsized":"2024-09-10T19:21:47.972236+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"237'31","last_scrub_stamp":"2024-09-10T19:20:43.252273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:20:43.252273+0000","objects_scrubbed":8,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0023272610000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":8,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":50,"num_read_kb":40,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":["4","6"],"object_location_counts":[{"shards":"4,6","objects":8}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":282,"reported_epoch":281,"state":"active+clean","last_fresh":"2024-09-10T19:23:18.994923+0000","last_change":"2024-09-10T19:22:24.285534+0000","last_active":"2024-09-10T19:23:18.994923+0000","last_peered":"2024-09-10T19:23:18.994923+0000","last_clean":"2024-09-10T19:23:18.994923+0000","last_became_active":"2024-09-10T19:10:07.691119+0000","last_became_peered":"2024-09-10T19:10:07.691119+0000","last_unstale":"2024-09-10T19:23:18.994923+0000","last_undegraded":"2024-09-10T19:23:18.994923+0000","last_fullsized":"2024-09-10T19:23:18.994923+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:22:24.285464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:16:39.716674+0000","last_clean_scrub_stamp":"2024-09-10T19:22:24.285464+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.029482613000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":232,"reported_epoch":281,"state":"active+undersized","last_fresh":"2024-09-10T19:23:18.997933+0000","last_change":"2024-09-10T19:21:48.003034+0000","last_active":"2024-09-10T19:23:18.997933+0000","last_peered":"2024-09-10T19:23:18.997933+0000","last_clean":"2024-09-10T19:21:43.949309+0000","last_became_active":"2024-09-10T19:21:48.003034+0000","last_became_peered":"2024-09-10T19:21:48.003034+0000","last_unstale":"2024-09-10T19:23:18.997933+0000","last_undegraded":"2024-09-10T19:23:18.997933+0000","last_fullsized":"2024-09-10T19:21:47.971412+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:20:54.418649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:20:54.418649+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.006664377,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":["4","6"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":160,"reported_epoch":281,"state":"active+clean","last_fresh":"2024-09-10T19:23:23.287262+0000","last_change":"2024-09-10T19:23:23.287193+0000","last_active":"2024-09-10T19:23:23.287262+0000","last_peered":"2024-09-10T19:23:23.287262+0000","last_clean":"2024-09-10T19:23:23.287262+0000","last_became_active":"2024-09-10T19:16:24.857843+0000","last_became_peered":"2024-09-10T19:16:24.857843+0000","last_unstale":"2024-09-10T19:23:23.287262+0000","last_undegraded":"2024-09-10T19:23:23.287262+0000","last_fullsized":"2024-09-10T19:23:23.287262+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:23:23.287110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:23:23.287110+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:24:50.549002+0000","scrub_duration":0.0049096449999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.9","version":"74'79","reported_seq":347,"reported_epoch":281,"state":"active+clean","last_fresh":"2024-09-10T19:23:18.997041+0000","last_change":"2024-09-10T19:22:37.284558+0000","last_active":"2024-09-10T19:23:18.997041+0000","last_peered":"2024-09-10T19:23:18.997041+0000","last_clean":"2024-09-10T19:23:18.997041+0000","last_became_active":"2024-09-10T19:20:30.844108+0000","last_became_peered":"2024-09-10T19:20:30.844108+0000","last_unstale":"2024-09-10T19:23:18.997041+0000","last_undegraded":"2024-09-10T19:23:18.997041+0000","last_fullsized":"2024-09-10T19:23:18.997041+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":4,"last_scrub":"74'79","last_scrub_stamp":"2024-09-10T19:22:37.284479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:08:00.699315+0000","last_clean_scrub_stamp":"2024-09-10T19:22:37.284479+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:23:56.506578+0000","scrub_duration":0.014548726999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.1","version":"281'18114","reported_seq":18357,"reported_epoch":281,"state":"active+clean","last_fresh":"2024-09-10T19:23:24.706894+0000","last_change":"2024-09-10T19:22:31.601422+0000","last_active":"2024-09-10T19:23:24.706894+0000","last_peered":"2024-09-10T19:23:24.706894+0000","last_clean":"2024-09-10T19:23:24.706894+0000","last_became_active":"2024-09-10T19:19:54.198678+0000","last_became_peered":"2024-09-10T19:19:54.198678+0000","last_unstale":"2024-09-10T19:23:24.706894+0000","last_undegraded":"2024-09-10T19:23:24.706894+0000","last_fullsized":"2024-09-10T19:23:24.706894+0000","mapping_epoch":229,"log_start":"273'14450","ondisk_log_start":"273'14450","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"272'13865","last_scrub_stamp":"2024-09-10T19:22:31.601360+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:19:53.174811+0000","last_clean_scrub_stamp":"2024-09-10T19:22:31.601360+0000","objects_scrubbed":4304,"log_size":3664,"log_dups_size":0,"ondisk_log_size":3664,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:23:51.522073+0000","scrub_duration":1.2956811529999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18114,"num_write_kb":579648,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"237'20","reported_seq":322,"reported_epoch":281,"state":"active+clean","last_fresh":"2024-09-10T19:23:18.998127+0000","last_change":"2024-09-10T19:23:12.640206+0000","last_active":"2024-09-10T19:23:18.998127+0000","last_peered":"2024-09-10T19:23:18.998127+0000","last_clean":"2024-09-10T19:23:18.998127+0000","last_became_active":"2024-09-10T19:10:09.688321+0000","last_became_peered":"2024-09-10T19:10:09.688321+0000","last_unstale":"2024-09-10T19:23:18.998127+0000","last_undegraded":"2024-09-10T19:23:18.998127+0000","last_fullsized":"2024-09-10T19:23:18.998127+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"237'20","last_scrub_stamp":"2024-09-10T19:23:12.640129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:23:12.640129+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:24:20.647255+0000","scrub_duration":0.0046113889999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":232,"reported_epoch":281,"state":"active+clean","last_fresh":"2024-09-10T19:23:18.986863+0000","last_change":"2024-09-10T19:23:08.392051+0000","last_active":"2024-09-10T19:23:18.986863+0000","last_peered":"2024-09-10T19:23:18.986863+0000","last_clean":"2024-09-10T19:23:18.986863+0000","last_became_active":"2024-09-10T19:13:54.731134+0000","last_became_peered":"2024-09-10T19:13:54.731134+0000","last_unstale":"2024-09-10T19:23:18.986863+0000","last_undegraded":"2024-09-10T19:23:18.986863+0000","last_fullsized":"2024-09-10T19:23:18.986863+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:23:08.392000+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:23:08.392000+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:24:28.000050+0000","scrub_duration":0.0055299800000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,5],"acting":[3,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"264'11","reported_seq":313,"reported_epoch":281,"state":"active+undersized+degraded","last_fresh":"2024-09-10T19:23:25.893778+0000","last_change":"2024-09-10T19:21:47.999533+0000","last_active":"2024-09-10T19:23:25.893778+0000","last_peered":"2024-09-10T19:23:25.893778+0000","last_clean":"2024-09-10T19:21:45.875744+0000","last_became_active":"2024-09-10T19:21:47.999533+0000","last_became_peered":"2024-09-10T19:21:47.999533+0000","last_unstale":"2024-09-10T19:23:25.893778+0000","last_undegraded":"2024-09-10T19:21:47.982460+0000","last_fullsized":"2024-09-10T19:21:47.982350+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"155'10","last_scrub_stamp":"2024-09-10T19:20:42.761156+0000","last_deep_scrub":"80'9","last_deep_scrub_stamp":"2024-09-10T19:11:25.156465+0000","last_clean_scrub_stamp":"2024-09-10T19:20:42.761156+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.010904117,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":["7","5"],"object_location_counts":[{"shards":"5,7","objects":1}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":298,"reported_epoch":281,"state":"active+clean","last_fresh":"2024-09-10T19:23:18.993467+0000","last_change":"2024-09-10T19:21:58.521967+0000","last_active":"2024-09-10T19:23:18.993467+0000","last_peered":"2024-09-10T19:23:18.993467+0000","last_clean":"2024-09-10T19:23:18.993467+0000","last_became_active":"2024-09-10T19:16:24.851829+0000","last_became_peered":"2024-09-10T19:16:24.851829+0000","last_unstale":"2024-09-10T19:23:18.993467+0000","last_undegraded":"2024-09-10T19:23:18.993467+0000","last_fullsized":"2024-09-10T19:23:18.993467+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:21:58.521891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:21:58.521891+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0050654960000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,7],"acting":[4,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.a","version":"74'79","reported_seq":57,"reported_epoch":281,"state":"active+clean","last_fresh":"2024-09-10T19:23:18.987591+0000","last_change":"2024-09-10T19:21:53.326522+0000","last_active":"2024-09-10T19:23:18.987591+0000","last_peered":"2024-09-10T19:23:18.987591+0000","last_clean":"2024-09-10T19:23:18.987591+0000","last_became_active":"2024-09-10T19:20:30.844773+0000","last_became_peered":"2024-09-10T19:20:30.844773+0000","last_unstale":"2024-09-10T19:23:18.987591+0000","last_undegraded":"2024-09-10T19:23:18.987591+0000","last_fullsized":"2024-09-10T19:23:18.987591+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":4,"last_scrub":"74'79","last_scrub_stamp":"2024-09-10T19:21:53.326433+0000","last_deep_scrub":"74'79","last_deep_scrub_stamp":"2024-09-10T19:21:53.326433+0000","last_clean_scrub_stamp":"2024-09-10T19:21:53.326433+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0062202960000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,4],"acting":[5,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.2","version":"281'18612","reported_seq":18729,"reported_epoch":281,"state":"active+clean","last_fresh":"2024-09-10T19:23:24.706272+0000","last_change":"2024-09-10T19:23:10.803354+0000","last_active":"2024-09-10T19:23:24.706272+0000","last_peered":"2024-09-10T19:23:24.706272+0000","last_clean":"2024-09-10T19:23:24.706272+0000","last_became_active":"2024-09-10T19:19:54.197290+0000","last_became_peered":"2024-09-10T19:19:54.197290+0000","last_unstale":"2024-09-10T19:23:24.706272+0000","last_undegraded":"2024-09-10T19:23:24.706272+0000","last_fullsized":"2024-09-10T19:23:24.706272+0000","mapping_epoch":229,"log_start":"275'15001","ondisk_log_start":"275'15001","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"280'17488","last_scrub_stamp":"2024-09-10T19:23:10.803296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:19:53.174811+0000","last_clean_scrub_stamp":"2024-09-10T19:23:10.803296+0000","objects_scrubbed":1141,"log_size":3611,"log_dups_size":0,"ondisk_log_size":3611,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:24:35.149673+0000","scrub_duration":0.38345378699999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18612,"num_write_kb":595584,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"237'27","reported_seq":303,"reported_epoch":281,"state":"active+clean","last_fresh":"2024-09-10T19:23:24.340124+0000","last_change":"2024-09-10T19:23:24.340051+0000","last_active":"2024-09-10T19:23:24.340124+0000","last_peered":"2024-09-10T19:23:24.340124+0000","last_clean":"2024-09-10T19:23:24.340124+0000","last_became_active":"2024-09-10T19:13:54.711634+0000","last_became_peered":"2024-09-10T19:13:54.711634+0000","last_unstale":"2024-09-10T19:23:24.340124+0000","last_undegraded":"2024-09-10T19:23:24.340124+0000","last_fullsized":"2024-09-10T19:23:24.340124+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"237'27","last_scrub_stamp":"2024-09-10T19:23:24.339960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:23:24.339960+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:24:31.664898+0000","scrub_duration":0.0077172969999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":252,"reported_epoch":281,"state":"active+clean","last_fresh":"2024-09-10T19:23:18.997237+0000","last_change":"2024-09-10T19:23:09.202528+0000","last_active":"2024-09-10T19:23:18.997237+0000","last_peered":"2024-09-10T19:23:18.997237+0000","last_clean":"2024-09-10T19:23:18.997237+0000","last_became_active":"2024-09-10T19:16:24.857179+0000","last_became_peered":"2024-09-10T19:16:24.857179+0000","last_unstale":"2024-09-10T19:23:18.997237+0000","last_undegraded":"2024-09-10T19:23:18.997237+0000","last_fullsized":"2024-09-10T19:23:18.997237+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:23:09.202458+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:23:09.202458+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:24:32.161864+0000","scrub_duration":0.0090519580000000006,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,6],"acting":[7,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.e","version":"264'9","reported_seq":436,"reported_epoch":281,"state":"active+undersized+degraded","last_fresh":"2024-09-10T19:23:25.893789+0000","last_change":"2024-09-10T19:21:47.981249+0000","last_active":"2024-09-10T19:23:25.893789+0000","last_peered":"2024-09-10T19:23:25.893789+0000","last_clean":"2024-09-10T19:21:45.875962+0000","last_became_active":"2024-09-10T19:21:47.981249+0000","last_became_peered":"2024-09-10T19:21:47.981249+0000","last_unstale":"2024-09-10T19:23:25.893789+0000","last_undegraded":"2024-09-10T19:21:47.971014+0000","last_fullsized":"2024-09-10T19:21:47.970564+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'8","last_scrub_stamp":"2024-09-10T19:19:17.519517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:19:17.519517+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0049564470000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":["4","5"],"object_location_counts":[{"shards":"4,5","objects":1}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":290,"reported_epoch":281,"state":"active+undersized","last_fresh":"2024-09-10T19:23:18.989067+0000","last_change":"2024-09-10T19:21:47.994832+0000","last_active":"2024-09-10T19:23:18.989067+0000","last_peered":"2024-09-10T19:23:18.989067+0000","last_clean":"2024-09-10T19:21:43.953414+0000","last_became_active":"2024-09-10T19:21:47.994832+0000","last_became_peered":"2024-09-10T19:21:47.994832+0000","last_unstale":"2024-09-10T19:23:18.989067+0000","last_undegraded":"2024-09-10T19:23:18.989067+0000","last_fullsized":"2024-09-10T19:21:47.973238+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:20:49.344566+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:20:49.344566+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0057184669999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":["5","6"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.3","version":"281'17994","reported_seq":18496,"reported_epoch":281,"state":"active+clean","last_fresh":"2024-09-10T19:23:24.710107+0000","last_change":"2024-09-10T19:22:21.598654+0000","last_active":"2024-09-10T19:23:24.710107+0000","last_peered":"2024-09-10T19:23:24.710107+0000","last_clean":"2024-09-10T19:23:24.710107+0000","last_became_active":"2024-09-10T19:19:54.203780+0000","last_became_peered":"2024-09-10T19:19:54.203780+0000","last_unstale":"2024-09-10T19:23:24.710107+0000","last_undegraded":"2024-09-10T19:23:24.710107+0000","last_fullsized":"2024-09-10T19:23:24.710107+0000","mapping_epoch":229,"log_start":"273'14305","ondisk_log_start":"273'14305","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"270'12966","last_scrub_stamp":"2024-09-10T19:22:21.598597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:19:53.174811+0000","last_clean_scrub_stamp":"2024-09-10T19:22:21.598597+0000","objects_scrubbed":5109,"log_size":3689,"log_dups_size":0,"ondisk_log_size":3689,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:23:43.809390+0000","scrub_duration":2.297330085,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17994,"num_write_kb":575808,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"237'24","reported_seq":347,"reported_epoch":281,"state":"active+clean","last_fresh":"2024-09-10T19:23:18.993757+0000","last_change":"2024-09-10T19:23:11.613306+0000","last_active":"2024-09-10T19:23:18.993757+0000","last_peered":"2024-09-10T19:23:18.993757+0000","last_clean":"2024-09-10T19:23:18.993757+0000","last_became_active":"2024-09-10T19:13:54.717022+0000","last_became_peered":"2024-09-10T19:13:54.717022+0000","last_unstale":"2024-09-10T19:23:18.993757+0000","last_undegraded":"2024-09-10T19:23:18.993757+0000","last_fullsized":"2024-09-10T19:23:18.993757+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"237'24","last_scrub_stamp":"2024-09-10T19:23:11.613027+0000","last_deep_scrub":"84'16","last_deep_scrub_stamp":"2024-09-10T19:11:56.157975+0000","last_clean_scrub_stamp":"2024-09-10T19:23:11.613027+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:24:38.282070+0000","scrub_duration":0.006332631,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,3],"acting":[4,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":286,"reported_epoch":281,"state":"active+undersized","last_fresh":"2024-09-10T19:23:18.992074+0000","last_change":"2024-09-10T19:21:47.983520+0000","last_active":"2024-09-10T19:23:18.992074+0000","last_peered":"2024-09-10T19:23:18.992074+0000","last_clean":"2024-09-10T19:21:43.954695+0000","last_became_active":"2024-09-10T19:21:47.983520+0000","last_became_peered":"2024-09-10T19:21:47.983520+0000","last_unstale":"2024-09-10T19:23:18.992074+0000","last_undegraded":"2024-09-10T19:23:18.992074+0000","last_fullsized":"2024-09-10T19:21:47.978120+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:20:06.874462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:20:06.874462+0000","last_clean_scrub_stamp":"2024-09-10T19:20:06.874462+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0052621079999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":["7","4"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":268,"reported_epoch":281,"state":"active+clean","last_fresh":"2024-09-10T19:23:18.989478+0000","last_change":"2024-09-10T19:22:19.713111+0000","last_active":"2024-09-10T19:23:18.989478+0000","last_peered":"2024-09-10T19:23:18.989478+0000","last_clean":"2024-09-10T19:23:18.989478+0000","last_became_active":"2024-09-10T19:10:11.692215+0000","last_became_peered":"2024-09-10T19:10:11.692215+0000","last_unstale":"2024-09-10T19:23:18.989478+0000","last_undegraded":"2024-09-10T19:23:18.989478+0000","last_fullsized":"2024-09-10T19:23:18.989478+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:22:19.713013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:22:19.713013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.029020666000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":266,"reported_epoch":281,"state":"active+clean","last_fresh":"2024-09-10T19:23:18.987172+0000","last_change":"2024-09-10T19:22:37.144678+0000","last_active":"2024-09-10T19:23:18.987172+0000","last_peered":"2024-09-10T19:23:18.987172+0000","last_clean":"2024-09-10T19:23:18.987172+0000","last_became_active":"2024-09-10T19:10:14.053736+0000","last_became_peered":"2024-09-10T19:10:14.053736+0000","last_unstale":"2024-09-10T19:23:18.987172+0000","last_undegraded":"2024-09-10T19:23:18.987172+0000","last_fullsized":"2024-09-10T19:23:18.987172+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:22:37.144606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:22:37.144606+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:24:02.963330+0000","scrub_duration":0.002180853,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.4","version":"74'79","reported_seq":62,"reported_epoch":281,"state":"active+clean","last_fresh":"2024-09-10T19:23:18.986596+0000","last_change":"2024-09-10T19:22:20.452660+0000","last_active":"2024-09-10T19:23:18.986596+0000","last_peered":"2024-09-10T19:23:18.986596+0000","last_clean":"2024-09-10T19:23:18.986596+0000","last_became_active":"2024-09-10T19:20:30.838441+0000","last_became_peered":"2024-09-10T19:20:30.838441+0000","last_unstale":"2024-09-10T19:23:18.986596+0000","last_undegraded":"2024-09-10T19:23:18.986596+0000","last_fullsized":"2024-09-10T19:23:18.986596+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":3,"last_scrub":"74'79","last_scrub_stamp":"2024-09-10T19:22:20.452581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:08:00.699315+0000","last_clean_scrub_stamp":"2024-09-10T19:22:20.452581+0000","objects_scrubbed":1,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:23:43.055901+0000","scrub_duration":0.006813782,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.c","version":"281'18438","reported_seq":18662,"reported_epoch":281,"state":"active+clean","last_fresh":"2024-09-10T19:23:24.701689+0000","last_change":"2024-09-10T19:22:38.980045+0000","last_active":"2024-09-10T19:23:24.701689+0000","last_peered":"2024-09-10T19:23:24.701689+0000","last_clean":"2024-09-10T19:23:24.701689+0000","last_became_active":"2024-09-10T19:19:54.200490+0000","last_became_peered":"2024-09-10T19:19:54.200490+0000","last_unstale":"2024-09-10T19:23:24.701689+0000","last_undegraded":"2024-09-10T19:23:24.701689+0000","last_fullsized":"2024-09-10T19:23:24.701689+0000","mapping_epoch":229,"log_start":"276'14855","ondisk_log_start":"276'14855","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"273'14620","last_scrub_stamp":"2024-09-10T19:22:38.979994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:19:53.174811+0000","last_clean_scrub_stamp":"2024-09-10T19:22:38.979994+0000","objects_scrubbed":3836,"log_size":3583,"log_dups_size":0,"ondisk_log_size":3583,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:23:41.164821+0000","scrub_duration":0.79555883599999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18438,"num_write_kb":590016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"265'35","reported_seq":214,"reported_epoch":281,"state":"active+clean","last_fresh":"2024-09-10T19:23:24.734335+0000","last_change":"2024-09-10T19:23:24.734284+0000","last_active":"2024-09-10T19:23:24.734335+0000","last_peered":"2024-09-10T19:23:24.734335+0000","last_clean":"2024-09-10T19:23:24.734335+0000","last_became_active":"2024-09-10T19:16:24.852821+0000","last_became_peered":"2024-09-10T19:16:24.852821+0000","last_unstale":"2024-09-10T19:23:24.734335+0000","last_undegraded":"2024-09-10T19:23:24.734335+0000","last_fullsized":"2024-09-10T19:23:24.734335+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"265'35","last_scrub_stamp":"2024-09-10T19:23:24.734231+0000","last_deep_scrub":"265'35","last_deep_scrub_stamp":"2024-09-10T19:21:55.533362+0000","last_clean_scrub_stamp":"2024-09-10T19:23:24.734231+0000","objects_scrubbed":7,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:24:34.519264+0000","scrub_duration":0.0018403409999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":42,"num_read_kb":35,"num_write":28,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,7],"acting":[4,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":302,"reported_epoch":281,"state":"active+clean","last_fresh":"2024-09-10T19:23:26.468298+0000","last_change":"2024-09-10T19:23:26.468261+0000","last_active":"2024-09-10T19:23:26.468298+0000","last_peered":"2024-09-10T19:23:26.468298+0000","last_clean":"2024-09-10T19:23:26.468298+0000","last_became_active":"2024-09-10T19:16:24.855861+0000","last_became_peered":"2024-09-10T19:16:24.855861+0000","last_unstale":"2024-09-10T19:23:26.468298+0000","last_undegraded":"2024-09-10T19:23:26.468298+0000","last_fullsized":"2024-09-10T19:23:26.468298+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:23:26.468206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:23:26.468206+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:24:36.414608+0000","scrub_duration":0.001230769,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.0","version":"264'11","reported_seq":456,"reported_epoch":281,"state":"active+undersized+degraded","last_fresh":"2024-09-10T19:23:25.893704+0000","last_change":"2024-09-10T19:21:47.975361+0000","last_active":"2024-09-10T19:23:25.893704+0000","last_peered":"2024-09-10T19:23:25.893704+0000","last_clean":"2024-09-10T19:21:45.875415+0000","last_became_active":"2024-09-10T19:21:47.975361+0000","last_became_peered":"2024-09-10T19:21:47.975361+0000","last_unstale":"2024-09-10T19:23:25.893704+0000","last_undegraded":"2024-09-10T19:21:47.972860+0000","last_fullsized":"2024-09-10T19:21:47.972722+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"117'10","last_scrub_stamp":"2024-09-10T19:20:13.462447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:20:13.462447+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0040309090000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":["3","4"],"object_location_counts":[{"shards":"3,4","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":273,"reported_epoch":281,"state":"active+undersized","last_fresh":"2024-09-10T19:23:18.994563+0000","last_change":"2024-09-10T19:21:48.011467+0000","last_active":"2024-09-10T19:23:18.994563+0000","last_peered":"2024-09-10T19:23:18.994563+0000","last_clean":"2024-09-10T19:21:43.955087+0000","last_became_active":"2024-09-10T19:21:48.011467+0000","last_became_peered":"2024-09-10T19:21:48.011467+0000","last_unstale":"2024-09-10T19:23:18.994563+0000","last_undegraded":"2024-09-10T19:23:18.994563+0000","last_fullsized":"2024-09-10T19:21:47.993946+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:19:10.029951+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:11:41.073833+0000","last_clean_scrub_stamp":"2024-09-10T19:19:10.029951+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.01007037,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":["7","6"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.5","version":"74'79","reported_seq":67,"reported_epoch":281,"state":"active+clean","last_fresh":"2024-09-10T19:23:18.994380+0000","last_change":"2024-09-10T19:23:13.621573+0000","last_active":"2024-09-10T19:23:18.994380+0000","last_peered":"2024-09-10T19:23:18.994380+0000","last_clean":"2024-09-10T19:23:18.994380+0000","last_became_active":"2024-09-10T19:20:30.838556+0000","last_became_peered":"2024-09-10T19:20:30.838556+0000","last_unstale":"2024-09-10T19:23:18.994380+0000","last_undegraded":"2024-09-10T19:23:18.994380+0000","last_fullsized":"2024-09-10T19:23:18.994380+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":3,"last_scrub":"74'79","last_scrub_stamp":"2024-09-10T19:23:13.621433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:08:00.699315+0000","last_clean_scrub_stamp":"2024-09-10T19:23:13.621433+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:24:20.118465+0000","scrub_duration":0.012192343,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.d","version":"281'20577","reported_seq":20951,"reported_epoch":281,"state":"active+clean","last_fresh":"2024-09-10T19:23:24.714372+0000","last_change":"2024-09-10T19:22:25.216700+0000","last_active":"2024-09-10T19:23:24.714372+0000","last_peered":"2024-09-10T19:23:24.714372+0000","last_clean":"2024-09-10T19:23:24.714372+0000","last_became_active":"2024-09-10T19:19:54.202133+0000","last_became_peered":"2024-09-10T19:19:54.202133+0000","last_unstale":"2024-09-10T19:23:24.714372+0000","last_undegraded":"2024-09-10T19:23:24.714372+0000","last_fullsized":"2024-09-10T19:23:24.714372+0000","mapping_epoch":229,"log_start":"274'16949","ondisk_log_start":"274'16949","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"270'15773","last_scrub_stamp":"2024-09-10T19:22:25.216491+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:19:53.174811+0000","last_clean_scrub_stamp":"2024-09-10T19:22:25.216491+0000","objects_scrubbed":4931,"log_size":3628,"log_dups_size":0,"ondisk_log_size":3628,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:23:47.031330+0000","scrub_duration":2.9560003789999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":18250,"num_write_kb":583937,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,4],"acting":[7,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.0","version":"237'30","reported_seq":317,"reported_epoch":281,"state":"active+undersized+degraded","last_fresh":"2024-09-10T19:23:18.987629+0000","last_change":"2024-09-10T19:21:47.983337+0000","last_active":"2024-09-10T19:23:18.987629+0000","last_peered":"2024-09-10T19:23:18.987629+0000","last_clean":"2024-09-10T19:21:43.943548+0000","last_became_active":"2024-09-10T19:21:47.983337+0000","last_became_peered":"2024-09-10T19:21:47.983337+0000","last_unstale":"2024-09-10T19:23:18.987629+0000","last_undegraded":"2024-09-10T19:21:47.975109+0000","last_fullsized":"2024-09-10T19:21:47.974995+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'18","last_scrub_stamp":"2024-09-10T19:19:12.207076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:19:12.207076+0000","objects_scrubbed":8,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.003631156,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":8,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":52,"num_read_kb":40,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":["3","7"],"object_location_counts":[{"shards":"3,7","objects":8}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":242,"reported_epoch":281,"state":"active+clean","last_fresh":"2024-09-10T19:23:18.996235+0000","last_change":"2024-09-10T19:22:32.696939+0000","last_active":"2024-09-10T19:23:18.996235+0000","last_peered":"2024-09-10T19:23:18.996235+0000","last_clean":"2024-09-10T19:23:18.996235+0000","last_became_active":"2024-09-10T19:13:54.710968+0000","last_became_peered":"2024-09-10T19:13:54.710968+0000","last_unstale":"2024-09-10T19:23:18.996235+0000","last_undegraded":"2024-09-10T19:23:18.996235+0000","last_fullsized":"2024-09-10T19:23:18.996235+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:22:32.696724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:17:26.374376+0000","last_clean_scrub_stamp":"2024-09-10T19:22:32.696724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:23:42.673395+0000","scrub_duration":0.0098896620000000005,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,3],"acting":[6,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":277,"reported_epoch":281,"state":"active+clean","last_fresh":"2024-09-10T19:23:18.992439+0000","last_change":"2024-09-10T19:22:27.985608+0000","last_active":"2024-09-10T19:23:18.992439+0000","last_peered":"2024-09-10T19:23:18.992439+0000","last_clean":"2024-09-10T19:23:18.992439+0000","last_became_active":"2024-09-10T19:10:11.698054+0000","last_became_peered":"2024-09-10T19:10:11.698054+0000","last_unstale":"2024-09-10T19:23:18.992439+0000","last_undegraded":"2024-09-10T19:23:18.992439+0000","last_fullsized":"2024-09-10T19:23:18.992439+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:22:27.985521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:16:30.991635+0000","last_clean_scrub_stamp":"2024-09-10T19:22:27.985521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:23:45.834053+0000","scrub_duration":0.016026845000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":285,"reported_epoch":281,"state":"active+clean","last_fresh":"2024-09-10T19:23:18.993231+0000","last_change":"2024-09-10T19:22:23.061813+0000","last_active":"2024-09-10T19:23:18.993231+0000","last_peered":"2024-09-10T19:23:18.993231+0000","last_clean":"2024-09-10T19:23:18.993231+0000","last_became_active":"2024-09-10T19:16:24.848109+0000","last_became_peered":"2024-09-10T19:16:24.848109+0000","last_unstale":"2024-09-10T19:23:18.993231+0000","last_undegraded":"2024-09-10T19:23:18.993231+0000","last_fullsized":"2024-09-10T19:23:18.993231+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:22:23.061715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:22:23.061715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:23:33.978057+0000","scrub_duration":0.0039110009999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.6","version":"74'79","reported_seq":134,"reported_epoch":281,"state":"active+undersized","last_fresh":"2024-09-10T19:23:18.996063+0000","last_change":"2024-09-10T19:21:47.988079+0000","last_active":"2024-09-10T19:23:18.996063+0000","last_peered":"2024-09-10T19:23:18.996063+0000","last_clean":"2024-09-10T19:21:45.499702+0000","last_became_active":"2024-09-10T19:21:47.988079+0000","last_became_peered":"2024-09-10T19:21:47.988079+0000","last_unstale":"2024-09-10T19:23:18.996063+0000","last_undegraded":"2024-09-10T19:23:18.996063+0000","last_fullsized":"2024-09-10T19:21:47.982186+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":3,"last_scrub":"74'79","last_scrub_stamp":"2024-09-10T19:21:45.499422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:08:00.699315+0000","last_clean_scrub_stamp":"2024-09-10T19:21:45.499422+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0024069769999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":["6","3"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.e","version":"281'18736","reported_seq":18804,"reported_epoch":281,"state":"active+undersized","last_fresh":"2024-09-10T19:23:24.712029+0000","last_change":"2024-09-10T19:23:24.712029+0000","last_active":"2024-09-10T19:23:24.712029+0000","last_peered":"2024-09-10T19:23:24.712029+0000","last_clean":"2024-09-10T19:21:46.952990+0000","last_became_active":"2024-09-10T19:21:47.989949+0000","last_became_peered":"2024-09-10T19:21:47.989949+0000","last_unstale":"2024-09-10T19:23:24.712029+0000","last_undegraded":"2024-09-10T19:23:24.712029+0000","last_fullsized":"2024-09-10T19:21:47.986624+0000","mapping_epoch":263,"log_start":"274'15053","ondisk_log_start":"274'15053","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:19:53.174811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:19:53.174811+0000","last_clean_scrub_stamp":"2024-09-10T19:19:53.174811+0000","objects_scrubbed":0,"log_size":3683,"log_dups_size":0,"ondisk_log_size":3683,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18736,"num_write_kb":599552,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":["7","4"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"237'37","reported_seq":309,"reported_epoch":281,"state":"active+undersized+degraded","last_fresh":"2024-09-10T19:23:18.988791+0000","last_change":"2024-09-10T19:21:47.998404+0000","last_active":"2024-09-10T19:23:18.988791+0000","last_peered":"2024-09-10T19:23:18.988791+0000","last_clean":"2024-09-10T19:20:16.238914+0000","last_became_active":"2024-09-10T19:21:47.998404+0000","last_became_peered":"2024-09-10T19:21:47.998404+0000","last_unstale":"2024-09-10T19:23:18.988791+0000","last_undegraded":"2024-09-10T19:21:47.977022+0000","last_fullsized":"2024-09-10T19:21:47.976922+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"104'27","last_scrub_stamp":"2024-09-10T19:20:14.754045+0000","last_deep_scrub":"104'27","last_deep_scrub_stamp":"2024-09-10T19:13:02.225944+0000","last_clean_scrub_stamp":"2024-09-10T19:20:14.754045+0000","objects_scrubbed":8,"log_size":37,"log_dups_size":0,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0059406959999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":8,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":53,"num_read_kb":39,"num_write":32,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":["5","7"],"object_location_counts":[{"shards":"5,7","objects":8}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":279,"reported_epoch":281,"state":"active+undersized","last_fresh":"2024-09-10T19:23:18.996189+0000","last_change":"2024-09-10T19:21:47.984335+0000","last_active":"2024-09-10T19:23:18.996189+0000","last_peered":"2024-09-10T19:23:18.996189+0000","last_clean":"2024-09-10T19:21:43.955503+0000","last_became_active":"2024-09-10T19:21:47.984335+0000","last_became_peered":"2024-09-10T19:21:47.984335+0000","last_unstale":"2024-09-10T19:23:18.996189+0000","last_undegraded":"2024-09-10T19:23:18.996189+0000","last_fullsized":"2024-09-10T19:21:47.979308+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:20:43.781741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:16:37.779967+0000","last_clean_scrub_stamp":"2024-09-10T19:20:43.781741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.023161668,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":["7","4"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":278,"reported_epoch":281,"state":"active+undersized","last_fresh":"2024-09-10T19:23:18.995333+0000","last_change":"2024-09-10T19:21:48.002017+0000","last_active":"2024-09-10T19:23:18.995333+0000","last_peered":"2024-09-10T19:23:18.995333+0000","last_clean":"2024-09-10T19:21:43.951362+0000","last_became_active":"2024-09-10T19:21:48.002017+0000","last_became_peered":"2024-09-10T19:21:48.002017+0000","last_unstale":"2024-09-10T19:23:18.995333+0000","last_undegraded":"2024-09-10T19:23:18.995333+0000","last_fullsized":"2024-09-10T19:21:47.979197+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:20:05.474676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:20:05.474676+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0035800150000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":["6","5"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":242,"reported_epoch":281,"state":"active+undersized","last_fresh":"2024-09-10T19:23:18.995252+0000","last_change":"2024-09-10T19:21:47.999030+0000","last_active":"2024-09-10T19:23:18.995252+0000","last_peered":"2024-09-10T19:23:18.995252+0000","last_clean":"2024-09-10T19:21:43.953419+0000","last_became_active":"2024-09-10T19:21:47.999030+0000","last_became_peered":"2024-09-10T19:21:47.999030+0000","last_unstale":"2024-09-10T19:23:18.995252+0000","last_undegraded":"2024-09-10T19:23:18.995252+0000","last_fullsized":"2024-09-10T19:21:47.984292+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:20:06.496950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:20:06.496950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0033688099999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":["6","7"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.7","version":"74'79","reported_seq":137,"reported_epoch":281,"state":"active+clean","last_fresh":"2024-09-10T19:23:18.996478+0000","last_change":"2024-09-10T19:22:34.717383+0000","last_active":"2024-09-10T19:23:18.996478+0000","last_peered":"2024-09-10T19:23:18.996478+0000","last_clean":"2024-09-10T19:23:18.996478+0000","last_became_active":"2024-09-10T19:20:30.842857+0000","last_became_peered":"2024-09-10T19:20:30.842857+0000","last_unstale":"2024-09-10T19:23:18.996478+0000","last_undegraded":"2024-09-10T19:23:18.996478+0000","last_fullsized":"2024-09-10T19:23:18.996478+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":3,"last_scrub":"74'79","last_scrub_stamp":"2024-09-10T19:22:34.717286+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:08:00.699315+0000","last_clean_scrub_stamp":"2024-09-10T19:22:34.717286+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:23:46.388918+0000","scrub_duration":0.0060991719999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.f","version":"281'18206","reported_seq":18439,"reported_epoch":281,"state":"active+clean","last_fresh":"2024-09-10T19:23:24.707492+0000","last_change":"2024-09-10T19:22:36.603275+0000","last_active":"2024-09-10T19:23:24.707492+0000","last_peered":"2024-09-10T19:23:24.707492+0000","last_clean":"2024-09-10T19:23:24.707492+0000","last_became_active":"2024-09-10T19:19:54.202323+0000","last_became_peered":"2024-09-10T19:19:54.202323+0000","last_unstale":"2024-09-10T19:23:24.707492+0000","last_undegraded":"2024-09-10T19:23:24.707492+0000","last_fullsized":"2024-09-10T19:23:24.707492+0000","mapping_epoch":229,"log_start":"273'14400","ondisk_log_start":"273'14400","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"273'14243","last_scrub_stamp":"2024-09-10T19:22:36.603192+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:19:53.174811+0000","last_clean_scrub_stamp":"2024-09-10T19:22:36.603192+0000","objects_scrubbed":4047,"log_size":3806,"log_dups_size":0,"ondisk_log_size":3806,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:23:56.410560+0000","scrub_duration":1.99645008,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18206,"num_write_kb":582592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"237'20","reported_seq":250,"reported_epoch":281,"state":"active+undersized+degraded","last_fresh":"2024-09-10T19:23:18.990602+0000","last_change":"2024-09-10T19:21:47.978225+0000","last_active":"2024-09-10T19:23:18.990602+0000","last_peered":"2024-09-10T19:23:18.990602+0000","last_clean":"2024-09-10T19:21:43.955550+0000","last_became_active":"2024-09-10T19:21:47.978225+0000","last_became_peered":"2024-09-10T19:21:47.978225+0000","last_unstale":"2024-09-10T19:23:18.990602+0000","last_undegraded":"2024-09-10T19:21:47.972413+0000","last_fullsized":"2024-09-10T19:21:47.972298+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'10","last_scrub_stamp":"2024-09-10T19:19:26.575769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:19:26.575769+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.022703509,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":24,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":["5","4"],"object_location_counts":[{"shards":"4,5","objects":5}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":114,"reported_epoch":281,"state":"active+undersized","last_fresh":"2024-09-10T19:23:18.993637+0000","last_change":"2024-09-10T19:21:47.983971+0000","last_active":"2024-09-10T19:23:18.993637+0000","last_peered":"2024-09-10T19:23:18.993637+0000","last_clean":"2024-09-10T19:16:23.155488+0000","last_became_active":"2024-09-10T19:21:47.983971+0000","last_became_peered":"2024-09-10T19:21:47.983971+0000","last_unstale":"2024-09-10T19:23:18.993637+0000","last_undegraded":"2024-09-10T19:23:18.993637+0000","last_fullsized":"2024-09-10T19:21:47.979000+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:20:52.748172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:20:52.748172+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0061607199999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":["7","3"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.3","version":"264'10","reported_seq":112,"reported_epoch":281,"state":"active+undersized+degraded","last_fresh":"2024-09-10T19:23:25.893826+0000","last_change":"2024-09-10T19:21:48.008059+0000","last_active":"2024-09-10T19:23:25.893826+0000","last_peered":"2024-09-10T19:23:25.893826+0000","last_clean":"2024-09-10T19:11:53.878374+0000","last_became_active":"2024-09-10T19:21:48.008059+0000","last_became_peered":"2024-09-10T19:21:48.008059+0000","last_unstale":"2024-09-10T19:23:25.893826+0000","last_undegraded":"2024-09-10T19:21:47.990007+0000","last_fullsized":"2024-09-10T19:21:47.989908+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'9","last_scrub_stamp":"2024-09-10T19:21:04.724652+0000","last_deep_scrub":"84'9","last_deep_scrub_stamp":"2024-09-10T19:11:53.879272+0000","last_clean_scrub_stamp":"2024-09-10T19:21:04.724652+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":["6","5"],"object_location_counts":[{"shards":"5,6","objects":1}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":148,"reported_epoch":281,"state":"active+undersized","last_fresh":"2024-09-10T19:23:18.985693+0000","last_change":"2024-09-10T19:21:47.995786+0000","last_active":"2024-09-10T19:23:18.985693+0000","last_peered":"2024-09-10T19:23:18.985693+0000","last_clean":"2024-09-10T19:16:23.154999+0000","last_became_active":"2024-09-10T19:21:47.995786+0000","last_became_peered":"2024-09-10T19:21:47.995786+0000","last_unstale":"2024-09-10T19:23:18.985693+0000","last_undegraded":"2024-09-10T19:23:18.985693+0000","last_fullsized":"2024-09-10T19:21:47.973262+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:21:03.766127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:21:03.766127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0086405590000000008,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":["3","5"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"74'79","reported_seq":54,"reported_epoch":281,"state":"active+undersized+degraded","last_fresh":"2024-09-10T19:23:18.987865+0000","last_change":"2024-09-10T19:21:47.981812+0000","last_active":"2024-09-10T19:23:18.987865+0000","last_peered":"2024-09-10T19:23:18.987865+0000","last_clean":"2024-09-10T19:21:43.955636+0000","last_became_active":"2024-09-10T19:21:47.981812+0000","last_became_peered":"2024-09-10T19:21:47.981812+0000","last_unstale":"2024-09-10T19:23:18.987865+0000","last_undegraded":"2024-09-10T19:21:47.972771+0000","last_fullsized":"2024-09-10T19:21:47.972653+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":4,"last_scrub":"74'79","last_scrub_stamp":"2024-09-10T19:19:11.070835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:08:00.699315+0000","last_clean_scrub_stamp":"2024-09-10T19:19:11.070835+0000","objects_scrubbed":2,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0030997160000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":11,"num_read_kb":20,"num_write":11,"num_write_kb":148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":53670,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":["5","4"],"object_location_counts":[{"shards":"4,5","objects":1}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.9","version":"281'18130","reported_seq":18380,"reported_epoch":281,"state":"active+undersized","last_fresh":"2024-09-10T19:23:24.707438+0000","last_change":"2024-09-10T19:23:29.210347+0000","last_active":"2024-09-10T19:23:24.707438+0000","last_peered":"2024-09-10T19:23:24.707438+0000","last_clean":"2024-09-10T19:21:46.936170+0000","last_became_active":"2024-09-10T19:21:47.996322+0000","last_became_peered":"2024-09-10T19:21:47.996322+0000","last_unstale":"2024-09-10T19:23:24.707438+0000","last_undegraded":"2024-09-10T19:21:47.979809+0000","last_fullsized":"2024-09-10T19:21:47.979667+0000","mapping_epoch":263,"log_start":"273'14421","ondisk_log_start":"273'14421","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:19:53.174811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:19:53.174811+0000","last_clean_scrub_stamp":"2024-09-10T19:19:53.174811+0000","objects_scrubbed":4449,"log_size":3709,"log_dups_size":0,"ondisk_log_size":3709,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18130,"num_write_kb":580160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":["7","5"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.4","version":"237'43","reported_seq":346,"reported_epoch":281,"state":"active+clean","last_fresh":"2024-09-10T19:23:25.893746+0000","last_change":"2024-09-10T19:23:22.309940+0000","last_active":"2024-09-10T19:23:25.893746+0000","last_peered":"2024-09-10T19:23:25.893746+0000","last_clean":"2024-09-10T19:23:25.893746+0000","last_became_active":"2024-09-10T19:16:24.858154+0000","last_became_peered":"2024-09-10T19:16:24.858154+0000","last_unstale":"2024-09-10T19:23:25.893746+0000","last_undegraded":"2024-09-10T19:23:25.893746+0000","last_fullsized":"2024-09-10T19:23:25.893746+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"237'43","last_scrub_stamp":"2024-09-10T19:23:22.309850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:23:22.309850+0000","objects_scrubbed":10,"log_size":43,"log_dups_size":0,"ondisk_log_size":43,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:24:38.590867+0000","scrub_duration":0.047452614999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":65,"num_read_kb":50,"num_write":38,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":358,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":286,"reported_epoch":281,"state":"active+undersized","last_fresh":"2024-09-10T19:23:18.990294+0000","last_change":"2024-09-10T19:21:47.994362+0000","last_active":"2024-09-10T19:23:18.990294+0000","last_peered":"2024-09-10T19:23:18.990294+0000","last_clean":"2024-09-10T19:21:43.955302+0000","last_became_active":"2024-09-10T19:21:47.994362+0000","last_became_peered":"2024-09-10T19:21:47.994362+0000","last_unstale":"2024-09-10T19:23:18.990294+0000","last_undegraded":"2024-09-10T19:23:18.990294+0000","last_fullsized":"2024-09-10T19:21:47.972726+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:20:44.342243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:20:44.342243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0078427040000000007,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":["5","6"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":108,"reported_epoch":281,"state":"active+undersized","last_fresh":"2024-09-10T19:23:18.994555+0000","last_change":"2024-09-10T19:21:47.987054+0000","last_active":"2024-09-10T19:23:18.994555+0000","last_peered":"2024-09-10T19:23:18.994555+0000","last_clean":"2024-09-10T19:12:44.681644+0000","last_became_active":"2024-09-10T19:21:47.987054+0000","last_became_peered":"2024-09-10T19:21:47.987054+0000","last_unstale":"2024-09-10T19:23:18.994555+0000","last_undegraded":"2024-09-10T19:23:18.994555+0000","last_fullsized":"2024-09-10T19:21:47.971744+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:20:56.648680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:20:56.648680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001460161,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":["4","6"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":256,"reported_epoch":281,"state":"active+clean","last_fresh":"2024-09-10T19:23:18.986306+0000","last_change":"2024-09-10T19:21:42.640298+0000","last_active":"2024-09-10T19:23:18.986306+0000","last_peered":"2024-09-10T19:23:18.986306+0000","last_clean":"2024-09-10T19:23:18.986306+0000","last_became_active":"2024-09-10T19:10:13.721744+0000","last_became_peered":"2024-09-10T19:10:13.721744+0000","last_unstale":"2024-09-10T19:23:18.986306+0000","last_undegraded":"2024-09-10T19:23:18.986306+0000","last_fullsized":"2024-09-10T19:23:18.986306+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:21:42.640236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:21:42.640236+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0027480289999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"74'79","reported_seq":344,"reported_epoch":281,"state":"active+undersized","last_fresh":"2024-09-10T19:23:18.995577+0000","last_change":"2024-09-10T19:21:47.984511+0000","last_active":"2024-09-10T19:23:18.995577+0000","last_peered":"2024-09-10T19:23:18.995577+0000","last_clean":"2024-09-10T19:21:43.956297+0000","last_became_active":"2024-09-10T19:21:47.984511+0000","last_became_peered":"2024-09-10T19:21:47.984511+0000","last_unstale":"2024-09-10T19:23:18.995577+0000","last_undegraded":"2024-09-10T19:23:18.995577+0000","last_fullsized":"2024-09-10T19:21:47.978708+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":4,"last_scrub":"74'79","last_scrub_stamp":"2024-09-10T19:21:04.941354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:08:00.699315+0000","last_clean_scrub_stamp":"2024-09-10T19:21:04.941354+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0040812920000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":["7","4"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.a","version":"281'18040","reported_seq":18302,"reported_epoch":281,"state":"active+clean","last_fresh":"2024-09-10T19:23:24.696208+0000","last_change":"2024-09-10T19:22:26.919130+0000","last_active":"2024-09-10T19:23:24.696208+0000","last_peered":"2024-09-10T19:23:24.696208+0000","last_clean":"2024-09-10T19:23:24.696208+0000","last_became_active":"2024-09-10T19:19:54.202444+0000","last_became_peered":"2024-09-10T19:19:54.202444+0000","last_unstale":"2024-09-10T19:23:24.696208+0000","last_undegraded":"2024-09-10T19:23:24.696208+0000","last_fullsized":"2024-09-10T19:23:24.696208+0000","mapping_epoch":229,"log_start":"273'14200","ondisk_log_start":"273'14200","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"271'13343","last_scrub_stamp":"2024-09-10T19:22:26.919040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:19:53.174811+0000","last_clean_scrub_stamp":"2024-09-10T19:22:26.919040+0000","objects_scrubbed":4796,"log_size":3840,"log_dups_size":0,"ondisk_log_size":3840,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:23:37.930761+0000","scrub_duration":2.3267707670000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18040,"num_write_kb":577280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.7","version":"237'22","reported_seq":308,"reported_epoch":281,"state":"active+undersized+degraded","last_fresh":"2024-09-10T19:23:18.989675+0000","last_change":"2024-09-10T19:21:47.993567+0000","last_active":"2024-09-10T19:23:18.989675+0000","last_peered":"2024-09-10T19:23:18.989675+0000","last_clean":"2024-09-10T19:21:43.951893+0000","last_became_active":"2024-09-10T19:21:47.993567+0000","last_became_peered":"2024-09-10T19:21:47.993567+0000","last_unstale":"2024-09-10T19:23:18.989675+0000","last_undegraded":"2024-09-10T19:21:47.978296+0000","last_fullsized":"2024-09-10T19:21:47.978196+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"237'22","last_scrub_stamp":"2024-09-10T19:20:51.327668+0000","last_deep_scrub":"82'18","last_deep_scrub_stamp":"2024-09-10T19:15:55.777361+0000","last_clean_scrub_stamp":"2024-09-10T19:20:51.327668+0000","objects_scrubbed":5,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0057025269999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":36,"num_read_kb":25,"num_write":20,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":["5","7"],"object_location_counts":[{"shards":"5,7","objects":5}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.0","version":"159'6","reported_seq":624,"reported_epoch":282,"state":"active+undersized+degraded","last_fresh":"2024-09-10T19:23:29.213177+0000","last_change":"2024-09-10T19:21:48.003078+0000","last_active":"2024-09-10T19:23:29.213177+0000","last_peered":"2024-09-10T19:23:29.213177+0000","last_clean":"2024-09-10T19:21:46.502353+0000","last_became_active":"2024-09-10T19:21:48.003078+0000","last_became_peered":"2024-09-10T19:21:48.003078+0000","last_unstale":"2024-09-10T19:23:29.213177+0000","last_undegraded":"2024-09-10T19:21:47.981464+0000","last_fullsized":"2024-09-10T19:21:47.981365+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"159'6","last_scrub_stamp":"2024-09-10T19:20:13.471084+0000","last_deep_scrub":"159'6","last_deep_scrub_stamp":"2024-09-10T19:16:32.649914+0000","last_clean_scrub_stamp":"2024-09-10T19:20:13.471084+0000","objects_scrubbed":1,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0080331139999999992,"objects_trimmed":0,"snaptrim_duration":0.00034454499999999997,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":428,"num_read_kb":423,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":389,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":["6","5"],"object_location_counts":[{"shards":"5,6","objects":1}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":167,"reported_epoch":281,"state":"active+undersized","last_fresh":"2024-09-10T19:23:18.986244+0000","last_change":"2024-09-10T19:21:47.988021+0000","last_active":"2024-09-10T19:23:18.986244+0000","last_peered":"2024-09-10T19:23:18.986244+0000","last_clean":"2024-09-10T19:21:43.944933+0000","last_became_active":"2024-09-10T19:21:47.988021+0000","last_became_peered":"2024-09-10T19:21:47.988021+0000","last_unstale":"2024-09-10T19:23:18.986244+0000","last_undegraded":"2024-09-10T19:23:18.986244+0000","last_fullsized":"2024-09-10T19:21:47.976328+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:21:01.464138+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:21:01.464138+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0040256789999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":["3","6"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":170,"reported_epoch":281,"state":"active+clean","last_fresh":"2024-09-10T19:23:28.736589+0000","last_change":"2024-09-10T19:23:28.736556+0000","last_active":"2024-09-10T19:23:28.736589+0000","last_peered":"2024-09-10T19:23:28.736589+0000","last_clean":"2024-09-10T19:23:28.736589+0000","last_became_active":"2024-09-10T19:16:24.861481+0000","last_became_peered":"2024-09-10T19:16:24.861481+0000","last_unstale":"2024-09-10T19:23:28.736589+0000","last_undegraded":"2024-09-10T19:23:28.736589+0000","last_fullsized":"2024-09-10T19:23:28.736589+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:23:28.736502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:23:28.736502+0000","last_clean_scrub_stamp":"2024-09-10T19:23:28.736502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:24:31.837409+0000","scrub_duration":0.0012043900000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,4],"acting":[5,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":279,"reported_epoch":281,"state":"active+clean","last_fresh":"2024-09-10T19:23:27.463853+0000","last_change":"2024-09-10T19:23:27.463818+0000","last_active":"2024-09-10T19:23:27.463853+0000","last_peered":"2024-09-10T19:23:27.463853+0000","last_clean":"2024-09-10T19:23:27.463853+0000","last_became_active":"2024-09-10T19:10:14.053697+0000","last_became_peered":"2024-09-10T19:10:14.053697+0000","last_unstale":"2024-09-10T19:23:27.463853+0000","last_undegraded":"2024-09-10T19:23:27.463853+0000","last_fullsized":"2024-09-10T19:23:27.463853+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:23:27.463763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:22:01.208272+0000","last_clean_scrub_stamp":"2024-09-10T19:23:27.463763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:24:28.432495+0000","scrub_duration":0.0012346900000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.3","version":"74'79","reported_seq":137,"reported_epoch":281,"state":"active+clean","last_fresh":"2024-09-10T19:23:18.995931+0000","last_change":"2024-09-10T19:23:14.563021+0000","last_active":"2024-09-10T19:23:18.995931+0000","last_peered":"2024-09-10T19:23:18.995931+0000","last_clean":"2024-09-10T19:23:18.995931+0000","last_became_active":"2024-09-10T19:20:30.847837+0000","last_became_peered":"2024-09-10T19:20:30.847837+0000","last_unstale":"2024-09-10T19:23:18.995931+0000","last_undegraded":"2024-09-10T19:23:18.995931+0000","last_fullsized":"2024-09-10T19:23:18.995931+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":3,"last_scrub":"74'79","last_scrub_stamp":"2024-09-10T19:23:14.562829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:08:00.699315+0000","last_clean_scrub_stamp":"2024-09-10T19:23:14.562829+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:24:44.111165+0000","scrub_duration":0.008216236,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,5],"acting":[6,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.b","version":"281'18632","reported_seq":18698,"reported_epoch":281,"state":"active+undersized","last_fresh":"2024-09-10T19:23:24.695238+0000","last_change":"2024-09-10T19:23:29.210311+0000","last_active":"2024-09-10T19:23:24.695238+0000","last_peered":"2024-09-10T19:23:24.695238+0000","last_clean":"2024-09-10T19:21:46.948308+0000","last_became_active":"2024-09-10T19:21:48.003266+0000","last_became_peered":"2024-09-10T19:21:48.003266+0000","last_unstale":"2024-09-10T19:23:24.695238+0000","last_undegraded":"2024-09-10T19:21:47.987789+0000","last_fullsized":"2024-09-10T19:21:47.985276+0000","mapping_epoch":263,"log_start":"273'14700","ondisk_log_start":"273'14700","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:19:53.174811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:19:53.174811+0000","last_clean_scrub_stamp":"2024-09-10T19:19:53.174811+0000","objects_scrubbed":0,"log_size":3932,"log_dups_size":0,"ondisk_log_size":3932,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18632,"num_write_kb":596224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":["6","5"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.6","version":"237'30","reported_seq":343,"reported_epoch":281,"state":"active+undersized+degraded","last_fresh":"2024-09-10T19:23:18.993593+0000","last_change":"2024-09-10T19:21:48.008902+0000","last_active":"2024-09-10T19:23:18.993593+0000","last_peered":"2024-09-10T19:23:18.993593+0000","last_clean":"2024-09-10T19:20:16.062994+0000","last_became_active":"2024-09-10T19:21:48.008902+0000","last_became_peered":"2024-09-10T19:21:48.008902+0000","last_unstale":"2024-09-10T19:23:18.993593+0000","last_undegraded":"2024-09-10T19:21:47.986289+0000","last_fullsized":"2024-09-10T19:21:47.986187+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"237'30","last_scrub_stamp":"2024-09-10T19:21:07.733508+0000","last_deep_scrub":"221'18","last_deep_scrub_stamp":"2024-09-10T19:19:29.606748+0000","last_clean_scrub_stamp":"2024-09-10T19:21:07.733508+0000","objects_scrubbed":6,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.020236845,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":["7","5"],"object_location_counts":[{"shards":"5,7","objects":6}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"61'1","reported_seq":38,"reported_epoch":281,"state":"active+undersized+degraded","last_fresh":"2024-09-10T19:23:18.992653+0000","last_change":"2024-09-10T19:21:47.976896+0000","last_active":"2024-09-10T19:23:18.992653+0000","last_peered":"2024-09-10T19:23:18.992653+0000","last_clean":"2024-09-10T19:10:09.679706+0000","last_became_active":"2024-09-10T19:21:47.976896+0000","last_became_peered":"2024-09-10T19:21:47.976896+0000","last_unstale":"2024-09-10T19:23:18.992653+0000","last_undegraded":"2024-09-10T19:21:47.970593+0000","last_fullsized":"2024-09-10T19:21:47.970495+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'1","last_scrub_stamp":"2024-09-10T19:20:40.716730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:20:40.716730+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":["4","3"],"object_location_counts":[{"shards":"3,4","objects":1}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":286,"reported_epoch":281,"state":"active+undersized","last_fresh":"2024-09-10T19:23:18.992187+0000","last_change":"2024-09-10T19:21:47.980682+0000","last_active":"2024-09-10T19:23:18.992187+0000","last_peered":"2024-09-10T19:23:18.992187+0000","last_clean":"2024-09-10T19:21:43.954775+0000","last_became_active":"2024-09-10T19:21:47.980682+0000","last_became_peered":"2024-09-10T19:21:47.980682+0000","last_unstale":"2024-09-10T19:23:18.992187+0000","last_undegraded":"2024-09-10T19:23:18.992187+0000","last_fullsized":"2024-09-10T19:21:47.977818+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:20:09.850369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:20:09.850369+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0036440280000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":["7","3"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":292,"reported_epoch":281,"state":"active+undersized","last_fresh":"2024-09-10T19:23:18.989556+0000","last_change":"2024-09-10T19:21:47.980962+0000","last_active":"2024-09-10T19:23:18.989556+0000","last_peered":"2024-09-10T19:23:18.989556+0000","last_clean":"2024-09-10T19:21:43.951818+0000","last_became_active":"2024-09-10T19:21:47.980962+0000","last_became_peered":"2024-09-10T19:21:47.980962+0000","last_unstale":"2024-09-10T19:23:18.989556+0000","last_undegraded":"2024-09-10T19:23:18.989556+0000","last_fullsized":"2024-09-10T19:21:47.972363+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:21:08.210604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:21:08.210604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0056788289999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":["5","4"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":232,"reported_epoch":281,"state":"active+clean","last_fresh":"2024-09-10T19:23:18.987881+0000","last_change":"2024-09-10T19:23:13.599958+0000","last_active":"2024-09-10T19:23:18.987881+0000","last_peered":"2024-09-10T19:23:18.987881+0000","last_clean":"2024-09-10T19:23:18.987881+0000","last_became_active":"2024-09-10T19:13:54.722937+0000","last_became_peered":"2024-09-10T19:13:54.722937+0000","last_unstale":"2024-09-10T19:23:18.987881+0000","last_undegraded":"2024-09-10T19:23:18.987881+0000","last_fullsized":"2024-09-10T19:23:18.987881+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:23:13.599885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:23:13.599885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:24:17.861774+0000","scrub_duration":0.0045688919999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.8","version":"281'18406","reported_seq":18472,"reported_epoch":281,"state":"active+undersized","last_fresh":"2024-09-10T19:23:24.700509+0000","last_change":"2024-09-10T19:23:29.027851+0000","last_active":"2024-09-10T19:23:24.700509+0000","last_peered":"2024-09-10T19:23:24.700509+0000","last_clean":"2024-09-10T19:21:46.945944+0000","last_became_active":"2024-09-10T19:21:47.996974+0000","last_became_peered":"2024-09-10T19:21:47.996974+0000","last_unstale":"2024-09-10T19:23:24.700509+0000","last_undegraded":"2024-09-10T19:21:47.973377+0000","last_fullsized":"2024-09-10T19:21:47.973279+0000","mapping_epoch":263,"log_start":"274'14750","ondisk_log_start":"274'14750","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:19:53.174811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:19:53.174811+0000","last_clean_scrub_stamp":"2024-09-10T19:19:53.174811+0000","objects_scrubbed":0,"log_size":3656,"log_dups_size":0,"ondisk_log_size":3656,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18406,"num_write_kb":588992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":["3","5"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"237'33","reported_seq":347,"reported_epoch":281,"state":"active+clean","last_fresh":"2024-09-10T19:23:28.405309+0000","last_change":"2024-09-10T19:23:28.405245+0000","last_active":"2024-09-10T19:23:28.405309+0000","last_peered":"2024-09-10T19:23:28.405309+0000","last_clean":"2024-09-10T19:23:28.405309+0000","last_became_active":"2024-09-10T19:10:09.822184+0000","last_became_peered":"2024-09-10T19:10:09.822184+0000","last_unstale":"2024-09-10T19:23:28.405309+0000","last_undegraded":"2024-09-10T19:23:28.405309+0000","last_fullsized":"2024-09-10T19:23:28.405309+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"237'33","last_scrub_stamp":"2024-09-10T19:23:28.405185+0000","last_deep_scrub":"80'21","last_deep_scrub_stamp":"2024-09-10T19:11:39.836144+0000","last_clean_scrub_stamp":"2024-09-10T19:23:28.405185+0000","objects_scrubbed":7,"log_size":33,"log_dups_size":0,"ondisk_log_size":33,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:24:30.180000+0000","scrub_duration":0.0019720100000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":33,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"63'2","reported_seq":289,"reported_epoch":281,"state":"active+clean","last_fresh":"2024-09-10T19:23:28.275673+0000","last_change":"2024-09-10T19:23:28.275635+0000","last_active":"2024-09-10T19:23:28.275673+0000","last_peered":"2024-09-10T19:23:28.275673+0000","last_clean":"2024-09-10T19:23:28.275673+0000","last_became_active":"2024-09-10T19:10:07.684784+0000","last_became_peered":"2024-09-10T19:10:07.684784+0000","last_unstale":"2024-09-10T19:23:28.275673+0000","last_undegraded":"2024-09-10T19:23:28.275673+0000","last_fullsized":"2024-09-10T19:23:28.275673+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'2","last_scrub_stamp":"2024-09-10T19:23:28.275581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:23:28.275581+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:24:38.479093+0000","scrub_duration":0.001514266,"objects_trimmed":0,"snaptrim_duration":0.00075522999999999996,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":292,"reported_epoch":281,"state":"active+clean","last_fresh":"2024-09-10T19:23:18.989144+0000","last_change":"2024-09-10T19:22:33.611730+0000","last_active":"2024-09-10T19:23:18.989144+0000","last_peered":"2024-09-10T19:23:18.989144+0000","last_clean":"2024-09-10T19:23:18.989144+0000","last_became_active":"2024-09-10T19:16:24.858662+0000","last_became_peered":"2024-09-10T19:16:24.858662+0000","last_unstale":"2024-09-10T19:23:18.989144+0000","last_undegraded":"2024-09-10T19:23:18.989144+0000","last_fullsized":"2024-09-10T19:23:18.989144+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:22:33.611658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:22:33.611658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:23:35.533500+0000","scrub_duration":0.012996436,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"74'79","reported_seq":335,"reported_epoch":281,"state":"active+undersized","last_fresh":"2024-09-10T19:23:18.992382+0000","last_change":"2024-09-10T19:21:47.999059+0000","last_active":"2024-09-10T19:23:18.992382+0000","last_peered":"2024-09-10T19:23:18.992382+0000","last_clean":"2024-09-10T19:21:43.954921+0000","last_became_active":"2024-09-10T19:21:47.999059+0000","last_became_peered":"2024-09-10T19:21:47.999059+0000","last_unstale":"2024-09-10T19:23:18.992382+0000","last_undegraded":"2024-09-10T19:23:18.992382+0000","last_fullsized":"2024-09-10T19:21:47.980322+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":243,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'79","last_scrub_stamp":"2024-09-10T19:20:59.897158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:08:00.699315+0000","last_clean_scrub_stamp":"2024-09-10T19:20:59.897158+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0081933070000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":["7","6"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":285,"reported_epoch":281,"state":"active+undersized","last_fresh":"2024-09-10T19:23:18.993453+0000","last_change":"2024-09-10T19:21:47.996658+0000","last_active":"2024-09-10T19:23:18.993453+0000","last_peered":"2024-09-10T19:23:18.993453+0000","last_clean":"2024-09-10T19:21:43.955922+0000","last_became_active":"2024-09-10T19:21:47.996658+0000","last_became_peered":"2024-09-10T19:21:47.996658+0000","last_unstale":"2024-09-10T19:23:18.993453+0000","last_undegraded":"2024-09-10T19:23:18.993453+0000","last_fullsized":"2024-09-10T19:21:47.979415+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:20:54.870558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:20:54.870558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0080190729999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":["7","5"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":266,"reported_epoch":281,"state":"active+clean","last_fresh":"2024-09-10T19:23:18.989881+0000","last_change":"2024-09-10T19:21:47.253156+0000","last_active":"2024-09-10T19:23:18.989881+0000","last_peered":"2024-09-10T19:23:18.989881+0000","last_clean":"2024-09-10T19:23:18.989881+0000","last_became_active":"2024-09-10T19:10:13.720647+0000","last_became_peered":"2024-09-10T19:10:13.720647+0000","last_unstale":"2024-09-10T19:23:18.989881+0000","last_undegraded":"2024-09-10T19:23:18.989881+0000","last_fullsized":"2024-09-10T19:23:18.989881+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:21:47.253100+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:21:47.253100+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0012429959999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.7","version":"281'19975","reported_seq":20169,"reported_epoch":281,"state":"active+clean","last_fresh":"2024-09-10T19:23:24.707048+0000","last_change":"2024-09-10T19:23:12.203648+0000","last_active":"2024-09-10T19:23:24.707048+0000","last_peered":"2024-09-10T19:23:24.707048+0000","last_clean":"2024-09-10T19:23:24.707048+0000","last_became_active":"2024-09-10T19:19:54.202928+0000","last_became_peered":"2024-09-10T19:19:54.202928+0000","last_unstale":"2024-09-10T19:23:24.707048+0000","last_undegraded":"2024-09-10T19:23:24.707048+0000","last_fullsized":"2024-09-10T19:23:24.707048+0000","mapping_epoch":229,"log_start":"277'16201","ondisk_log_start":"277'16201","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"280'18991","last_scrub_stamp":"2024-09-10T19:23:12.203258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:19:53.174811+0000","last_clean_scrub_stamp":"2024-09-10T19:23:12.203258+0000","objects_scrubbed":1008,"log_size":3774,"log_dups_size":0,"ondisk_log_size":3774,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:24:19.252252+0000","scrub_duration":0.62934323800000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18134,"num_write_kb":580288,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.a","version":"237'35","reported_seq":378,"reported_epoch":281,"state":"active+clean","last_fresh":"2024-09-10T19:23:18.995645+0000","last_change":"2024-09-10T19:22:28.681277+0000","last_active":"2024-09-10T19:23:18.995645+0000","last_peered":"2024-09-10T19:23:18.995645+0000","last_clean":"2024-09-10T19:23:18.995645+0000","last_became_active":"2024-09-10T19:13:54.731274+0000","last_became_peered":"2024-09-10T19:13:54.731274+0000","last_unstale":"2024-09-10T19:23:18.995645+0000","last_undegraded":"2024-09-10T19:23:18.995645+0000","last_fullsized":"2024-09-10T19:23:18.995645+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"237'35","last_scrub_stamp":"2024-09-10T19:22:28.681169+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:22:28.681169+0000","objects_scrubbed":9,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:23:58.430745+0000","scrub_duration":0.0096326229999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":56,"num_read_kb":45,"num_write":36,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,5],"acting":[6,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":157,"reported_epoch":281,"state":"active+undersized","last_fresh":"2024-09-10T19:23:18.988048+0000","last_change":"2024-09-10T19:21:47.992941+0000","last_active":"2024-09-10T19:23:18.988048+0000","last_peered":"2024-09-10T19:23:18.988048+0000","last_clean":"2024-09-10T19:21:43.951531+0000","last_became_active":"2024-09-10T19:21:47.992941+0000","last_became_peered":"2024-09-10T19:21:47.992941+0000","last_unstale":"2024-09-10T19:23:18.988048+0000","last_undegraded":"2024-09-10T19:23:18.988048+0000","last_fullsized":"2024-09-10T19:21:47.972931+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:20:06.554210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:20:06.554210+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0035560079999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":["5","3"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":32,"reported_epoch":281,"state":"active+undersized","last_fresh":"2024-09-10T19:23:18.989042+0000","last_change":"2024-09-10T19:21:47.998802+0000","last_active":"2024-09-10T19:23:18.989042+0000","last_peered":"2024-09-10T19:23:18.989042+0000","last_clean":"2024-09-10T19:10:10.676876+0000","last_became_active":"2024-09-10T19:21:47.998802+0000","last_became_peered":"2024-09-10T19:21:47.998802+0000","last_unstale":"2024-09-10T19:23:18.989042+0000","last_undegraded":"2024-09-10T19:23:18.989042+0000","last_fullsized":"2024-09-10T19:21:47.977340+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:20:03.641376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:20:03.641376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":["5","7"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":286,"reported_epoch":281,"state":"active+clean","last_fresh":"2024-09-10T19:23:18.992291+0000","last_change":"2024-09-10T19:22:30.285621+0000","last_active":"2024-09-10T19:23:18.992291+0000","last_peered":"2024-09-10T19:23:18.992291+0000","last_clean":"2024-09-10T19:23:18.992291+0000","last_became_active":"2024-09-10T19:16:24.858172+0000","last_became_peered":"2024-09-10T19:16:24.858172+0000","last_unstale":"2024-09-10T19:23:18.992291+0000","last_undegraded":"2024-09-10T19:23:18.992291+0000","last_fullsized":"2024-09-10T19:23:18.992291+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:22:30.285547+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:21:03.910074+0000","last_clean_scrub_stamp":"2024-09-10T19:22:30.285547+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:23:39.761607+0000","scrub_duration":0.0039370639999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,6],"acting":[7,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.6","version":"281'18242","reported_seq":18309,"reported_epoch":281,"state":"active+undersized","last_fresh":"2024-09-10T19:23:24.706571+0000","last_change":"2024-09-10T19:23:24.706571+0000","last_active":"2024-09-10T19:23:24.706571+0000","last_peered":"2024-09-10T19:23:24.706571+0000","last_clean":"2024-09-10T19:21:46.959322+0000","last_became_active":"2024-09-10T19:21:48.003207+0000","last_became_peered":"2024-09-10T19:21:48.003207+0000","last_unstale":"2024-09-10T19:23:24.706571+0000","last_undegraded":"2024-09-10T19:23:24.706571+0000","last_fullsized":"2024-09-10T19:21:47.992906+0000","mapping_epoch":263,"log_start":"273'14400","ondisk_log_start":"273'14400","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:19:53.174811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:19:53.174811+0000","last_clean_scrub_stamp":"2024-09-10T19:19:53.174811+0000","objects_scrubbed":0,"log_size":3842,"log_dups_size":0,"ondisk_log_size":3842,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18242,"num_write_kb":583744,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":["5","6"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"237'15","reported_seq":298,"reported_epoch":281,"state":"active+undersized+degraded","last_fresh":"2024-09-10T19:23:18.993651+0000","last_change":"2024-09-10T19:21:47.975617+0000","last_active":"2024-09-10T19:23:18.993651+0000","last_peered":"2024-09-10T19:23:18.993651+0000","last_clean":"2024-09-10T19:20:16.225370+0000","last_became_active":"2024-09-10T19:21:47.975617+0000","last_became_peered":"2024-09-10T19:21:47.975617+0000","last_unstale":"2024-09-10T19:23:18.993651+0000","last_undegraded":"2024-09-10T19:21:47.970276+0000","last_fullsized":"2024-09-10T19:21:47.970174+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"237'15","last_scrub_stamp":"2024-09-10T19:21:46.837828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:21:46.837828+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001540919,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":["4","3"],"object_location_counts":[{"shards":"3,4","objects":4}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":245,"reported_epoch":281,"state":"active+clean","last_fresh":"2024-09-10T19:23:18.996930+0000","last_change":"2024-09-10T19:22:21.064704+0000","last_active":"2024-09-10T19:23:18.996930+0000","last_peered":"2024-09-10T19:23:18.996930+0000","last_clean":"2024-09-10T19:23:18.996930+0000","last_became_active":"2024-09-10T19:16:24.848668+0000","last_became_peered":"2024-09-10T19:16:24.848668+0000","last_unstale":"2024-09-10T19:23:18.996930+0000","last_undegraded":"2024-09-10T19:23:18.996930+0000","last_fullsized":"2024-09-10T19:23:18.996930+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:22:21.064628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:11:44.254513+0000","last_clean_scrub_stamp":"2024-09-10T19:22:21.064628+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:23:35.513650+0000","scrub_duration":0.0086942800000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":170,"reported_epoch":281,"state":"active+undersized","last_fresh":"2024-09-10T19:23:18.993076+0000","last_change":"2024-09-10T19:21:47.977739+0000","last_active":"2024-09-10T19:23:18.993076+0000","last_peered":"2024-09-10T19:23:18.993076+0000","last_clean":"2024-09-10T19:21:43.946899+0000","last_became_active":"2024-09-10T19:21:47.977739+0000","last_became_peered":"2024-09-10T19:21:47.977739+0000","last_unstale":"2024-09-10T19:23:18.993076+0000","last_undegraded":"2024-09-10T19:23:18.993076+0000","last_fullsized":"2024-09-10T19:21:47.969920+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:21:09.367669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:21:09.367669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0065456109999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":["4","3"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":148,"reported_epoch":281,"state":"active+undersized","last_fresh":"2024-09-10T19:23:18.996800+0000","last_change":"2024-09-10T19:21:47.990835+0000","last_active":"2024-09-10T19:23:18.996800+0000","last_peered":"2024-09-10T19:23:18.996800+0000","last_clean":"2024-09-10T19:16:23.155267+0000","last_became_active":"2024-09-10T19:21:47.990835+0000","last_became_peered":"2024-09-10T19:21:47.990835+0000","last_unstale":"2024-09-10T19:23:18.996800+0000","last_undegraded":"2024-09-10T19:23:18.996800+0000","last_fullsized":"2024-09-10T19:21:47.981819+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:20:08.746643+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:20:08.746643+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0058765429999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":["7","3"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.5","version":"281'18466","reported_seq":18770,"reported_epoch":281,"state":"active+undersized","last_fresh":"2024-09-10T19:23:24.676553+0000","last_change":"2024-09-10T19:23:29.210080+0000","last_active":"2024-09-10T19:23:24.676553+0000","last_peered":"2024-09-10T19:23:24.676553+0000","last_clean":"2024-09-10T19:21:46.944630+0000","last_became_active":"2024-09-10T19:21:47.994750+0000","last_became_peered":"2024-09-10T19:21:47.994750+0000","last_unstale":"2024-09-10T19:23:24.676553+0000","last_undegraded":"2024-09-10T19:21:47.973184+0000","last_fullsized":"2024-09-10T19:21:47.972919+0000","mapping_epoch":263,"log_start":"274'14676","ondisk_log_start":"274'14676","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:19:53.174811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:19:53.174811+0000","last_clean_scrub_stamp":"2024-09-10T19:19:53.174811+0000","objects_scrubbed":5823,"log_size":3790,"log_dups_size":0,"ondisk_log_size":3790,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18466,"num_write_kb":590912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":["5","7"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.8","version":"237'34","reported_seq":320,"reported_epoch":281,"state":"active+clean","last_fresh":"2024-09-10T19:23:18.987996+0000","last_change":"2024-09-10T19:21:58.440243+0000","last_active":"2024-09-10T19:23:18.987996+0000","last_peered":"2024-09-10T19:23:18.987996+0000","last_clean":"2024-09-10T19:23:18.987996+0000","last_became_active":"2024-09-10T19:10:09.820950+0000","last_became_peered":"2024-09-10T19:10:09.820950+0000","last_unstale":"2024-09-10T19:23:18.987996+0000","last_undegraded":"2024-09-10T19:23:18.987996+0000","last_fullsized":"2024-09-10T19:23:18.987996+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"237'34","last_scrub_stamp":"2024-09-10T19:21:58.440165+0000","last_deep_scrub":"220'22","last_deep_scrub_stamp":"2024-09-10T19:19:24.597866+0000","last_clean_scrub_stamp":"2024-09-10T19:21:58.440165+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0050099799999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":160,"reported_epoch":281,"state":"active+clean","last_fresh":"2024-09-10T19:23:20.339002+0000","last_change":"2024-09-10T19:23:20.338930+0000","last_active":"2024-09-10T19:23:20.339002+0000","last_peered":"2024-09-10T19:23:20.339002+0000","last_clean":"2024-09-10T19:23:20.339002+0000","last_became_active":"2024-09-10T19:16:24.857253+0000","last_became_peered":"2024-09-10T19:16:24.857253+0000","last_unstale":"2024-09-10T19:23:20.339002+0000","last_undegraded":"2024-09-10T19:23:20.339002+0000","last_fullsized":"2024-09-10T19:23:20.339002+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:23:20.338866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:23:20.338866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:24:43.851289+0000","scrub_duration":0.0059949699999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.9","version":"80'9","reported_seq":447,"reported_epoch":281,"state":"active+clean","last_fresh":"2024-09-10T19:23:25.893869+0000","last_change":"2024-09-10T19:22:29.301346+0000","last_active":"2024-09-10T19:23:25.893869+0000","last_peered":"2024-09-10T19:23:25.893869+0000","last_clean":"2024-09-10T19:23:25.893869+0000","last_became_active":"2024-09-10T19:10:11.702878+0000","last_became_peered":"2024-09-10T19:10:11.702878+0000","last_unstale":"2024-09-10T19:23:25.893869+0000","last_undegraded":"2024-09-10T19:23:25.893869+0000","last_fullsized":"2024-09-10T19:23:25.893869+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"80'9","last_scrub_stamp":"2024-09-10T19:22:29.301243+0000","last_deep_scrub":"80'9","last_deep_scrub_stamp":"2024-09-10T19:20:57.882496+0000","last_clean_scrub_stamp":"2024-09-10T19:22:29.301243+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:23:41.869017+0000","scrub_duration":0.013296258,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":263,"reported_epoch":281,"state":"active+undersized","last_fresh":"2024-09-10T19:23:18.990813+0000","last_change":"2024-09-10T19:21:47.994405+0000","last_active":"2024-09-10T19:23:18.990813+0000","last_peered":"2024-09-10T19:23:18.990813+0000","last_clean":"2024-09-10T19:21:43.955746+0000","last_became_active":"2024-09-10T19:21:47.994405+0000","last_became_peered":"2024-09-10T19:21:47.994405+0000","last_unstale":"2024-09-10T19:23:18.990813+0000","last_undegraded":"2024-09-10T19:23:18.990813+0000","last_fullsized":"2024-09-10T19:21:47.974191+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:19:15.539488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:19:15.539488+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.033498668000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":["5","6"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.4","version":"281'18288","reported_seq":18356,"reported_epoch":281,"state":"active+undersized","last_fresh":"2024-09-10T19:23:24.709740+0000","last_change":"2024-09-10T19:23:29.029137+0000","last_active":"2024-09-10T19:23:24.709740+0000","last_peered":"2024-09-10T19:23:24.709740+0000","last_clean":"2024-09-10T19:21:46.956094+0000","last_became_active":"2024-09-10T19:21:47.976009+0000","last_became_peered":"2024-09-10T19:21:47.976009+0000","last_unstale":"2024-09-10T19:23:24.709740+0000","last_undegraded":"2024-09-10T19:21:47.970032+0000","last_fullsized":"2024-09-10T19:21:47.969894+0000","mapping_epoch":263,"log_start":"272'14000","ondisk_log_start":"272'14000","created":229,"last_epoch_clean":230,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:19:53.174811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:19:53.174811+0000","last_clean_scrub_stamp":"2024-09-10T19:19:53.174811+0000","objects_scrubbed":0,"log_size":4288,"log_dups_size":0,"ondisk_log_size":4288,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18288,"num_write_kb":585216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":["4","3"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.9","version":"237'18","reported_seq":343,"reported_epoch":281,"state":"active+clean","last_fresh":"2024-09-10T19:23:26.714655+0000","last_change":"2024-09-10T19:23:26.714612+0000","last_active":"2024-09-10T19:23:26.714655+0000","last_peered":"2024-09-10T19:23:26.714655+0000","last_clean":"2024-09-10T19:23:26.714655+0000","last_became_active":"2024-09-10T19:13:54.718152+0000","last_became_peered":"2024-09-10T19:13:54.718152+0000","last_unstale":"2024-09-10T19:23:26.714655+0000","last_undegraded":"2024-09-10T19:23:26.714655+0000","last_fullsized":"2024-09-10T19:23:26.714655+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"237'18","last_scrub_stamp":"2024-09-10T19:23:26.714557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:23:26.714557+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:24:51.363495+0000","scrub_duration":0.001833859,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.f","version":"59'2","reported_seq":294,"reported_epoch":281,"state":"active+undersized+degraded","last_fresh":"2024-09-10T19:23:18.993118+0000","last_change":"2024-09-10T19:21:47.986039+0000","last_active":"2024-09-10T19:23:18.993118+0000","last_peered":"2024-09-10T19:23:18.993118+0000","last_clean":"2024-09-10T19:21:43.952266+0000","last_became_active":"2024-09-10T19:21:47.986039+0000","last_became_peered":"2024-09-10T19:21:47.986039+0000","last_unstale":"2024-09-10T19:23:18.993118+0000","last_undegraded":"2024-09-10T19:21:47.971581+0000","last_fullsized":"2024-09-10T19:21:47.971480+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'2","last_scrub_stamp":"2024-09-10T19:20:56.467569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:20:56.467569+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.013126802,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":14,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":["4","7"],"object_location_counts":[{"shards":"4,7","objects":2}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":112,"reported_epoch":281,"state":"active+undersized","last_fresh":"2024-09-10T19:23:18.988079+0000","last_change":"2024-09-10T19:21:47.983793+0000","last_active":"2024-09-10T19:23:18.988079+0000","last_peered":"2024-09-10T19:23:18.988079+0000","last_clean":"2024-09-10T19:12:44.685930+0000","last_became_active":"2024-09-10T19:21:47.983793+0000","last_became_peered":"2024-09-10T19:21:47.983793+0000","last_unstale":"2024-09-10T19:23:18.988079+0000","last_undegraded":"2024-09-10T19:23:18.988079+0000","last_fullsized":"2024-09-10T19:21:47.972418+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:21:45.880289+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:11:46.587846+0000","last_clean_scrub_stamp":"2024-09-10T19:21:45.880289+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001267142,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":["5","4"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":283,"reported_epoch":281,"state":"active+clean","last_fresh":"2024-09-10T19:23:19.383560+0000","last_change":"2024-09-10T19:23:19.383517+0000","last_active":"2024-09-10T19:23:19.383560+0000","last_peered":"2024-09-10T19:23:19.383560+0000","last_clean":"2024-09-10T19:23:19.383560+0000","last_became_active":"2024-09-10T19:13:54.711804+0000","last_became_peered":"2024-09-10T19:13:54.711804+0000","last_unstale":"2024-09-10T19:23:19.383560+0000","last_undegraded":"2024-09-10T19:23:19.383560+0000","last_fullsized":"2024-09-10T19:23:19.383560+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:23:19.383453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:23:19.383453+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:24:41.715484+0000","scrub_duration":0.0094206629999999993,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.16","version":"237'15","reported_seq":281,"reported_epoch":281,"state":"active+undersized+degraded","last_fresh":"2024-09-10T19:23:18.986398+0000","last_change":"2024-09-10T19:21:47.988985+0000","last_active":"2024-09-10T19:23:18.986398+0000","last_peered":"2024-09-10T19:23:18.986398+0000","last_clean":"2024-09-10T19:20:15.530776+0000","last_became_active":"2024-09-10T19:21:47.988985+0000","last_became_peered":"2024-09-10T19:21:47.988985+0000","last_unstale":"2024-09-10T19:23:18.986398+0000","last_undegraded":"2024-09-10T19:21:47.974466+0000","last_fullsized":"2024-09-10T19:21:47.974359+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"237'15","last_scrub_stamp":"2024-09-10T19:21:01.691067+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:21:01.691067+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0065392059999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":["3","7"],"object_location_counts":[{"shards":"3,7","objects":4}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":116,"reported_epoch":281,"state":"active+undersized","last_fresh":"2024-09-10T19:23:18.994112+0000","last_change":"2024-09-10T19:21:47.995029+0000","last_active":"2024-09-10T19:23:18.994112+0000","last_peered":"2024-09-10T19:23:18.994112+0000","last_clean":"2024-09-10T19:12:44.686063+0000","last_became_active":"2024-09-10T19:21:47.995029+0000","last_became_peered":"2024-09-10T19:21:47.995029+0000","last_unstale":"2024-09-10T19:23:18.994112+0000","last_undegraded":"2024-09-10T19:23:18.994112+0000","last_fullsized":"2024-09-10T19:21:47.969981+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:20:57.698558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:11:41.458523+0000","last_clean_scrub_stamp":"2024-09-10T19:20:57.698558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0012499830000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":["4","6"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":282,"reported_epoch":281,"state":"active+undersized","last_fresh":"2024-09-10T19:23:18.987523+0000","last_change":"2024-09-10T19:21:47.984694+0000","last_active":"2024-09-10T19:23:18.987523+0000","last_peered":"2024-09-10T19:23:18.987523+0000","last_clean":"2024-09-10T19:21:43.944868+0000","last_became_active":"2024-09-10T19:21:47.984694+0000","last_became_peered":"2024-09-10T19:21:47.984694+0000","last_unstale":"2024-09-10T19:23:18.987523+0000","last_undegraded":"2024-09-10T19:23:18.987523+0000","last_fullsized":"2024-09-10T19:21:47.974535+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:21:43.633645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:21:43.633645+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001568039,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":["3","7"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"66'1","reported_seq":170,"reported_epoch":281,"state":"active+undersized+degraded","last_fresh":"2024-09-10T19:23:18.997772+0000","last_change":"2024-09-10T19:21:48.003365+0000","last_active":"2024-09-10T19:23:18.997772+0000","last_peered":"2024-09-10T19:23:18.997772+0000","last_clean":"2024-09-10T19:21:43.949315+0000","last_became_active":"2024-09-10T19:21:48.003365+0000","last_became_peered":"2024-09-10T19:21:48.003365+0000","last_unstale":"2024-09-10T19:23:18.997772+0000","last_undegraded":"2024-09-10T19:21:47.971226+0000","last_fullsized":"2024-09-10T19:21:47.970632+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'1","last_scrub_stamp":"2024-09-10T19:20:07.338085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:20:07.338085+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0056882089999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":13,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":["4","7"],"object_location_counts":[{"shards":"4,7","objects":1}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":305,"reported_epoch":281,"state":"active+clean","last_fresh":"2024-09-10T19:23:25.466418+0000","last_change":"2024-09-10T19:23:25.466373+0000","last_active":"2024-09-10T19:23:25.466418+0000","last_peered":"2024-09-10T19:23:25.466418+0000","last_clean":"2024-09-10T19:23:25.466418+0000","last_became_active":"2024-09-10T19:13:54.724027+0000","last_became_peered":"2024-09-10T19:13:54.724027+0000","last_unstale":"2024-09-10T19:23:25.466418+0000","last_undegraded":"2024-09-10T19:23:25.466418+0000","last_fullsized":"2024-09-10T19:23:25.466418+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:23:25.466309+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:23:25.466309+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:24:49.822398+0000","scrub_duration":0.001495176,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,5],"acting":[6,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"237'12","reported_seq":321,"reported_epoch":281,"state":"active+clean","last_fresh":"2024-09-10T19:23:18.987714+0000","last_change":"2024-09-10T19:22:26.419563+0000","last_active":"2024-09-10T19:23:18.987714+0000","last_peered":"2024-09-10T19:23:18.987714+0000","last_clean":"2024-09-10T19:23:18.987714+0000","last_became_active":"2024-09-10T19:16:24.863398+0000","last_became_peered":"2024-09-10T19:16:24.863398+0000","last_unstale":"2024-09-10T19:23:18.987714+0000","last_undegraded":"2024-09-10T19:23:18.987714+0000","last_fullsized":"2024-09-10T19:23:18.987714+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"237'12","last_scrub_stamp":"2024-09-10T19:22:26.419497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:22:26.419497+0000","objects_scrubbed":3,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:23:31.536216+0000","scrub_duration":0.0033796080000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":18,"num_read_kb":15,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":285,"reported_epoch":281,"state":"active+clean","last_fresh":"2024-09-10T19:23:18.988369+0000","last_change":"2024-09-10T19:22:30.626044+0000","last_active":"2024-09-10T19:23:18.988369+0000","last_peered":"2024-09-10T19:23:18.988369+0000","last_clean":"2024-09-10T19:23:18.988369+0000","last_became_active":"2024-09-10T19:13:54.729917+0000","last_became_peered":"2024-09-10T19:13:54.729917+0000","last_unstale":"2024-09-10T19:23:18.988369+0000","last_undegraded":"2024-09-10T19:23:18.988369+0000","last_fullsized":"2024-09-10T19:23:18.988369+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:22:30.625942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:11:23.264793+0000","last_clean_scrub_stamp":"2024-09-10T19:22:30.625942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:23:40.795531+0000","scrub_duration":0.010093893,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":276,"reported_epoch":281,"state":"active+clean","last_fresh":"2024-09-10T19:23:18.993206+0000","last_change":"2024-09-10T19:22:28.305076+0000","last_active":"2024-09-10T19:23:18.993206+0000","last_peered":"2024-09-10T19:23:18.993206+0000","last_clean":"2024-09-10T19:23:18.993206+0000","last_became_active":"2024-09-10T19:10:14.054360+0000","last_became_peered":"2024-09-10T19:10:14.054360+0000","last_unstale":"2024-09-10T19:23:18.993206+0000","last_undegraded":"2024-09-10T19:23:18.993206+0000","last_fullsized":"2024-09-10T19:23:18.993206+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:22:28.304999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:22:28.304999+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:23:43.703876+0000","scrub_duration":0.0082740739999999993,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"237'25","reported_seq":356,"reported_epoch":281,"state":"active+clean","last_fresh":"2024-09-10T19:23:18.986756+0000","last_change":"2024-09-10T19:23:17.434150+0000","last_active":"2024-09-10T19:23:18.986756+0000","last_peered":"2024-09-10T19:23:18.986756+0000","last_clean":"2024-09-10T19:23:18.986756+0000","last_became_active":"2024-09-10T19:16:24.850702+0000","last_became_peered":"2024-09-10T19:16:24.850702+0000","last_unstale":"2024-09-10T19:23:18.986756+0000","last_undegraded":"2024-09-10T19:23:18.986756+0000","last_fullsized":"2024-09-10T19:23:18.986756+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"237'25","last_scrub_stamp":"2024-09-10T19:23:17.434065+0000","last_deep_scrub":"80'15","last_deep_scrub_stamp":"2024-09-10T19:11:41.719310+0000","last_clean_scrub_stamp":"2024-09-10T19:23:17.434065+0000","objects_scrubbed":5,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:24:36.811331+0000","scrub_duration":0.0063536310000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,5],"acting":[3,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":272,"reported_epoch":281,"state":"active+clean","last_fresh":"2024-09-10T19:23:18.990408+0000","last_change":"2024-09-10T19:21:48.298990+0000","last_active":"2024-09-10T19:23:18.990408+0000","last_peered":"2024-09-10T19:23:18.990408+0000","last_clean":"2024-09-10T19:23:18.990408+0000","last_became_active":"2024-09-10T19:10:07.685870+0000","last_became_peered":"2024-09-10T19:10:07.685870+0000","last_unstale":"2024-09-10T19:23:18.990408+0000","last_undegraded":"2024-09-10T19:23:18.990408+0000","last_fullsized":"2024-09-10T19:23:18.990408+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:21:48.298913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:21:48.298913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0075260040000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"264'11","reported_seq":456,"reported_epoch":281,"state":"active+undersized+degraded","last_fresh":"2024-09-10T19:23:25.893775+0000","last_change":"2024-09-10T19:21:47.993917+0000","last_active":"2024-09-10T19:23:25.893775+0000","last_peered":"2024-09-10T19:23:25.893775+0000","last_clean":"2024-09-10T19:21:45.876612+0000","last_became_active":"2024-09-10T19:21:47.993917+0000","last_became_peered":"2024-09-10T19:21:47.993917+0000","last_unstale":"2024-09-10T19:23:25.893775+0000","last_undegraded":"2024-09-10T19:21:47.974032+0000","last_fullsized":"2024-09-10T19:21:47.973921+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"117'10","last_scrub_stamp":"2024-09-10T19:20:55.377819+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:20:55.377819+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0063951320000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":["5","3"],"object_location_counts":[{"shards":"3,5","objects":1}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":28,"reported_epoch":281,"state":"active+undersized","last_fresh":"2024-09-10T19:23:18.993852+0000","last_change":"2024-09-10T19:21:47.989023+0000","last_active":"2024-09-10T19:23:18.993852+0000","last_peered":"2024-09-10T19:23:18.993852+0000","last_clean":"2024-09-10T19:10:12.693598+0000","last_became_active":"2024-09-10T19:21:47.989023+0000","last_became_peered":"2024-09-10T19:21:47.989023+0000","last_unstale":"2024-09-10T19:23:18.993852+0000","last_undegraded":"2024-09-10T19:23:18.993852+0000","last_fullsized":"2024-09-10T19:21:47.982009+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:20:46.625193+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:20:46.625193+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":["7","3"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.15","version":"237'15","reported_seq":335,"reported_epoch":281,"state":"active+clean","last_fresh":"2024-09-10T19:23:18.989528+0000","last_change":"2024-09-10T19:21:57.580717+0000","last_active":"2024-09-10T19:23:18.989528+0000","last_peered":"2024-09-10T19:23:18.989528+0000","last_clean":"2024-09-10T19:23:18.989528+0000","last_became_active":"2024-09-10T19:16:24.859740+0000","last_became_peered":"2024-09-10T19:16:24.859740+0000","last_unstale":"2024-09-10T19:23:18.989528+0000","last_undegraded":"2024-09-10T19:23:18.989528+0000","last_fullsized":"2024-09-10T19:23:18.989528+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"237'15","last_scrub_stamp":"2024-09-10T19:21:57.580644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:21:57.580644+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.11368046,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":28,"reported_epoch":281,"state":"active+undersized","last_fresh":"2024-09-10T19:23:18.993961+0000","last_change":"2024-09-10T19:21:47.975293+0000","last_active":"2024-09-10T19:23:18.993961+0000","last_peered":"2024-09-10T19:23:18.993961+0000","last_clean":"2024-09-10T19:10:06.664709+0000","last_became_active":"2024-09-10T19:21:47.975293+0000","last_became_peered":"2024-09-10T19:21:47.975293+0000","last_unstale":"2024-09-10T19:23:18.993961+0000","last_undegraded":"2024-09-10T19:23:18.993961+0000","last_fullsized":"2024-09-10T19:21:47.970537+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:20:48.709474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:20:48.709474+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":["4","3"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"155'9","reported_seq":450,"reported_epoch":281,"state":"active+clean","last_fresh":"2024-09-10T19:23:25.893659+0000","last_change":"2024-09-10T19:22:00.716614+0000","last_active":"2024-09-10T19:23:25.893659+0000","last_peered":"2024-09-10T19:23:25.893659+0000","last_clean":"2024-09-10T19:23:25.893659+0000","last_became_active":"2024-09-10T19:16:24.850850+0000","last_became_peered":"2024-09-10T19:16:24.850850+0000","last_unstale":"2024-09-10T19:23:25.893659+0000","last_undegraded":"2024-09-10T19:23:25.893659+0000","last_fullsized":"2024-09-10T19:23:25.893659+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"155'9","last_scrub_stamp":"2024-09-10T19:22:00.716555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:22:00.716555+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:23:29.355333+0000","scrub_duration":0.0020496289999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,4],"acting":[3,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":275,"reported_epoch":281,"state":"active+undersized","last_fresh":"2024-09-10T19:23:18.992782+0000","last_change":"2024-09-10T19:21:47.976028+0000","last_active":"2024-09-10T19:23:18.992782+0000","last_peered":"2024-09-10T19:23:18.992782+0000","last_clean":"2024-09-10T19:21:43.951991+0000","last_became_active":"2024-09-10T19:21:47.976028+0000","last_became_peered":"2024-09-10T19:21:47.976028+0000","last_unstale":"2024-09-10T19:23:18.992782+0000","last_undegraded":"2024-09-10T19:23:18.992782+0000","last_fullsized":"2024-09-10T19:21:47.970502+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:19:12.504402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:19:12.504402+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0034925469999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":["4","5"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.12","version":"237'19","reported_seq":331,"reported_epoch":281,"state":"active+undersized+degraded","last_fresh":"2024-09-10T19:23:18.995459+0000","last_change":"2024-09-10T19:21:48.003068+0000","last_active":"2024-09-10T19:23:18.995459+0000","last_peered":"2024-09-10T19:23:18.995459+0000","last_clean":"2024-09-10T19:21:43.951440+0000","last_became_active":"2024-09-10T19:21:48.003068+0000","last_became_peered":"2024-09-10T19:21:48.003068+0000","last_unstale":"2024-09-10T19:23:18.995459+0000","last_undegraded":"2024-09-10T19:21:47.979935+0000","last_fullsized":"2024-09-10T19:21:47.979836+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"237'19","last_scrub_stamp":"2024-09-10T19:21:07.378184+0000","last_deep_scrub":"79'13","last_deep_scrub_stamp":"2024-09-10T19:11:22.072400+0000","last_clean_scrub_stamp":"2024-09-10T19:21:07.378184+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.004147263,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":["6","5"],"object_location_counts":[{"shards":"5,6","objects":4}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.14","version":"59'1","reported_seq":289,"reported_epoch":281,"state":"active+clean","last_fresh":"2024-09-10T19:23:18.995740+0000","last_change":"2024-09-10T19:22:20.715211+0000","last_active":"2024-09-10T19:23:18.995740+0000","last_peered":"2024-09-10T19:23:18.995740+0000","last_clean":"2024-09-10T19:23:18.995740+0000","last_became_active":"2024-09-10T19:10:07.683263+0000","last_became_peered":"2024-09-10T19:10:07.683263+0000","last_unstale":"2024-09-10T19:23:18.995740+0000","last_undegraded":"2024-09-10T19:23:18.995740+0000","last_fullsized":"2024-09-10T19:23:18.995740+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'1","last_scrub_stamp":"2024-09-10T19:22:20.715123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:22:20.715123+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:23:29.620364+0000","scrub_duration":0.0046532379999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":362,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":282,"reported_epoch":281,"state":"active+undersized","last_fresh":"2024-09-10T19:23:18.986153+0000","last_change":"2024-09-10T19:21:47.996741+0000","last_active":"2024-09-10T19:23:18.986153+0000","last_peered":"2024-09-10T19:23:18.986153+0000","last_clean":"2024-09-10T19:21:43.944122+0000","last_became_active":"2024-09-10T19:21:47.996741+0000","last_became_peered":"2024-09-10T19:21:47.996741+0000","last_unstale":"2024-09-10T19:23:18.986153+0000","last_undegraded":"2024-09-10T19:23:18.986153+0000","last_fullsized":"2024-09-10T19:21:47.974561+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:20:50.491473+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:20:50.491473+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0066726329999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":["3","5"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":106,"reported_epoch":281,"state":"active+undersized","last_fresh":"2024-09-10T19:23:18.987881+0000","last_change":"2024-09-10T19:21:48.003432+0000","last_active":"2024-09-10T19:23:18.987881+0000","last_peered":"2024-09-10T19:23:18.987881+0000","last_clean":"2024-09-10T19:12:44.680563+0000","last_became_active":"2024-09-10T19:21:48.003432+0000","last_became_peered":"2024-09-10T19:21:48.003432+0000","last_unstale":"2024-09-10T19:23:18.987881+0000","last_undegraded":"2024-09-10T19:23:18.987881+0000","last_fullsized":"2024-09-10T19:21:47.974956+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:21:09.714677+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:21:09.714677+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001865058,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":["5","6"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.13","version":"237'19","reported_seq":332,"reported_epoch":281,"state":"active+clean","last_fresh":"2024-09-10T19:23:18.993846+0000","last_change":"2024-09-10T19:22:22.040432+0000","last_active":"2024-09-10T19:23:18.993846+0000","last_peered":"2024-09-10T19:23:18.993846+0000","last_clean":"2024-09-10T19:23:18.993846+0000","last_became_active":"2024-09-10T19:13:54.720034+0000","last_became_peered":"2024-09-10T19:13:54.720034+0000","last_unstale":"2024-09-10T19:23:18.993846+0000","last_undegraded":"2024-09-10T19:23:18.993846+0000","last_fullsized":"2024-09-10T19:23:18.993846+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"237'19","last_scrub_stamp":"2024-09-10T19:22:22.040341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:22:22.040341+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:23:38.655498+0000","scrub_duration":0.013845334000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":219,"reported_epoch":281,"state":"active+undersized","last_fresh":"2024-09-10T19:23:18.989611+0000","last_change":"2024-09-10T19:21:48.019714+0000","last_active":"2024-09-10T19:23:18.989611+0000","last_peered":"2024-09-10T19:23:18.989611+0000","last_clean":"2024-09-10T19:21:43.954115+0000","last_became_active":"2024-09-10T19:21:48.019714+0000","last_became_peered":"2024-09-10T19:21:48.019714+0000","last_unstale":"2024-09-10T19:23:18.989611+0000","last_undegraded":"2024-09-10T19:23:18.989611+0000","last_fullsized":"2024-09-10T19:21:47.993355+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:19:10.439659+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:11:10.031245+0000","last_clean_scrub_stamp":"2024-09-10T19:19:10.439659+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.014001438,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":["5","6"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":232,"reported_epoch":281,"state":"active+clean","last_fresh":"2024-09-10T19:23:18.990715+0000","last_change":"2024-09-10T19:23:17.622829+0000","last_active":"2024-09-10T19:23:18.990715+0000","last_peered":"2024-09-10T19:23:18.990715+0000","last_clean":"2024-09-10T19:23:18.990715+0000","last_became_active":"2024-09-10T19:13:54.729703+0000","last_became_peered":"2024-09-10T19:13:54.729703+0000","last_unstale":"2024-09-10T19:23:18.990715+0000","last_undegraded":"2024-09-10T19:23:18.990715+0000","last_fullsized":"2024-09-10T19:23:18.990715+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:23:17.622746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:23:17.622746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:24:33.732612+0000","scrub_duration":0.012650075,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":263,"reported_epoch":281,"state":"active+undersized","last_fresh":"2024-09-10T19:23:18.985770+0000","last_change":"2024-09-10T19:21:47.993530+0000","last_active":"2024-09-10T19:23:18.985770+0000","last_peered":"2024-09-10T19:23:18.985770+0000","last_clean":"2024-09-10T19:21:43.944067+0000","last_became_active":"2024-09-10T19:21:47.993530+0000","last_became_peered":"2024-09-10T19:21:47.993530+0000","last_unstale":"2024-09-10T19:23:18.985770+0000","last_undegraded":"2024-09-10T19:23:18.985770+0000","last_fullsized":"2024-09-10T19:21:47.975469+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:19:13.231390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:19:13.231390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0029165770000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":["3","5"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"237'10","reported_seq":307,"reported_epoch":281,"state":"active+clean","last_fresh":"2024-09-10T19:23:18.987409+0000","last_change":"2024-09-10T19:22:28.363094+0000","last_active":"2024-09-10T19:23:18.987409+0000","last_peered":"2024-09-10T19:23:18.987409+0000","last_clean":"2024-09-10T19:23:18.987409+0000","last_became_active":"2024-09-10T19:13:54.711125+0000","last_became_peered":"2024-09-10T19:13:54.711125+0000","last_unstale":"2024-09-10T19:23:18.987409+0000","last_undegraded":"2024-09-10T19:23:18.987409+0000","last_fullsized":"2024-09-10T19:23:18.987409+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"237'10","last_scrub_stamp":"2024-09-10T19:22:28.363041+0000","last_deep_scrub":"222'6","last_deep_scrub_stamp":"2024-09-10T19:19:30.328841+0000","last_clean_scrub_stamp":"2024-09-10T19:22:28.363041+0000","objects_scrubbed":2,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:23:46.463086+0000","scrub_duration":0.0040448380000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":12,"num_read_kb":10,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,4],"acting":[3,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":292,"reported_epoch":281,"state":"active+clean","last_fresh":"2024-09-10T19:23:19.717488+0000","last_change":"2024-09-10T19:23:19.717425+0000","last_active":"2024-09-10T19:23:19.717488+0000","last_peered":"2024-09-10T19:23:19.717488+0000","last_clean":"2024-09-10T19:23:19.717488+0000","last_became_active":"2024-09-10T19:16:24.852460+0000","last_became_peered":"2024-09-10T19:16:24.852460+0000","last_unstale":"2024-09-10T19:23:19.717488+0000","last_undegraded":"2024-09-10T19:23:19.717488+0000","last_fullsized":"2024-09-10T19:23:19.717488+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:23:19.717358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:23:19.717358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:24:38.208507+0000","scrub_duration":0.020480847,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,3],"acting":[5,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":288,"reported_epoch":281,"state":"active+clean","last_fresh":"2024-09-10T19:23:24.462568+0000","last_change":"2024-09-10T19:23:24.462523+0000","last_active":"2024-09-10T19:23:24.462568+0000","last_peered":"2024-09-10T19:23:24.462568+0000","last_clean":"2024-09-10T19:23:24.462568+0000","last_became_active":"2024-09-10T19:10:11.698536+0000","last_became_peered":"2024-09-10T19:10:11.698536+0000","last_unstale":"2024-09-10T19:23:24.462568+0000","last_undegraded":"2024-09-10T19:23:24.462568+0000","last_fullsized":"2024-09-10T19:23:24.462568+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:23:24.462473+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:11:31.837944+0000","last_clean_scrub_stamp":"2024-09-10T19:23:24.462473+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:24:24.902468+0000","scrub_duration":0.010077553,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":296,"reported_epoch":281,"state":"active+clean","last_fresh":"2024-09-10T19:23:18.995212+0000","last_change":"2024-09-10T19:22:31.278309+0000","last_active":"2024-09-10T19:23:18.995212+0000","last_peered":"2024-09-10T19:23:18.995212+0000","last_clean":"2024-09-10T19:23:18.995212+0000","last_became_active":"2024-09-10T19:16:24.852356+0000","last_became_peered":"2024-09-10T19:16:24.852356+0000","last_unstale":"2024-09-10T19:23:18.995212+0000","last_undegraded":"2024-09-10T19:23:18.995212+0000","last_fullsized":"2024-09-10T19:23:18.995212+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:22:31.278179+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:22:31.278179+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:23:56.593462+0000","scrub_duration":0.0036472750000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,3],"acting":[7,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.11","version":"237'19","reported_seq":62,"reported_epoch":281,"state":"active+clean","last_fresh":"2024-09-10T19:23:18.985524+0000","last_change":"2024-09-10T19:22:22.464814+0000","last_active":"2024-09-10T19:23:18.985524+0000","last_peered":"2024-09-10T19:23:18.985524+0000","last_clean":"2024-09-10T19:23:18.985524+0000","last_became_active":"2024-09-10T19:21:04.169444+0000","last_became_peered":"2024-09-10T19:21:04.169444+0000","last_unstale":"2024-09-10T19:23:18.985524+0000","last_undegraded":"2024-09-10T19:23:18.985524+0000","last_fullsized":"2024-09-10T19:23:18.985524+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"237'19","last_scrub_stamp":"2024-09-10T19:22:22.464729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:22:22.464729+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:23:33.416371+0000","scrub_duration":0.0058949659999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":289,"reported_epoch":281,"state":"active+undersized","last_fresh":"2024-09-10T19:23:18.996988+0000","last_change":"2024-09-10T19:21:48.002896+0000","last_active":"2024-09-10T19:23:18.996988+0000","last_peered":"2024-09-10T19:23:18.996988+0000","last_clean":"2024-09-10T19:21:43.952651+0000","last_became_active":"2024-09-10T19:21:48.002896+0000","last_became_peered":"2024-09-10T19:21:48.002896+0000","last_unstale":"2024-09-10T19:23:18.996988+0000","last_undegraded":"2024-09-10T19:23:18.996988+0000","last_fullsized":"2024-09-10T19:21:47.987616+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:20:08.412227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:13:05.876655+0000","last_clean_scrub_stamp":"2024-09-10T19:20:08.412227+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0037109489999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":["6","5"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":268,"reported_epoch":281,"state":"active+clean","last_fresh":"2024-09-10T19:23:18.991758+0000","last_change":"2024-09-10T19:21:50.103708+0000","last_active":"2024-09-10T19:23:18.991758+0000","last_peered":"2024-09-10T19:23:18.991758+0000","last_clean":"2024-09-10T19:23:18.991758+0000","last_became_active":"2024-09-10T19:10:11.699013+0000","last_became_peered":"2024-09-10T19:10:11.699013+0000","last_unstale":"2024-09-10T19:23:18.991758+0000","last_undegraded":"2024-09-10T19:23:18.991758+0000","last_fullsized":"2024-09-10T19:23:18.991758+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:21:50.103617+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:11:57.112092+0000","last_clean_scrub_stamp":"2024-09-10T19:21:50.103617+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.011550181,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":263,"reported_epoch":281,"state":"active+undersized","last_fresh":"2024-09-10T19:23:18.985938+0000","last_change":"2024-09-10T19:21:47.990590+0000","last_active":"2024-09-10T19:23:18.985938+0000","last_peered":"2024-09-10T19:23:18.985938+0000","last_clean":"2024-09-10T19:21:43.944266+0000","last_became_active":"2024-09-10T19:21:47.990590+0000","last_became_peered":"2024-09-10T19:21:47.990590+0000","last_unstale":"2024-09-10T19:23:18.985938+0000","last_undegraded":"2024-09-10T19:23:18.985938+0000","last_fullsized":"2024-09-10T19:21:47.974352+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:19:18.276069+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:19:18.276069+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0034003950000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":["3","6"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"80'2","reported_seq":292,"reported_epoch":281,"state":"active+clean","last_fresh":"2024-09-10T19:23:18.993288+0000","last_change":"2024-09-10T19:22:23.295914+0000","last_active":"2024-09-10T19:23:18.993288+0000","last_peered":"2024-09-10T19:23:18.993288+0000","last_clean":"2024-09-10T19:23:18.993288+0000","last_became_active":"2024-09-10T19:16:24.851220+0000","last_became_peered":"2024-09-10T19:16:24.851220+0000","last_unstale":"2024-09-10T19:23:18.993288+0000","last_undegraded":"2024-09-10T19:23:18.993288+0000","last_fullsized":"2024-09-10T19:23:18.993288+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"80'2","last_scrub_stamp":"2024-09-10T19:22:23.295840+0000","last_deep_scrub":"80'2","last_deep_scrub_stamp":"2024-09-10T19:11:30.272043+0000","last_clean_scrub_stamp":"2024-09-10T19:22:23.295840+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:23:37.219620+0000","scrub_duration":0.015418665,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":369,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":369,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,3],"acting":[7,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":279,"reported_epoch":281,"state":"active+undersized","last_fresh":"2024-09-10T19:23:18.988189+0000","last_change":"2024-09-10T19:21:47.978661+0000","last_active":"2024-09-10T19:23:18.988189+0000","last_peered":"2024-09-10T19:23:18.988189+0000","last_clean":"2024-09-10T19:21:43.952149+0000","last_became_active":"2024-09-10T19:21:47.978661+0000","last_became_peered":"2024-09-10T19:21:47.978661+0000","last_unstale":"2024-09-10T19:23:18.988189+0000","last_undegraded":"2024-09-10T19:23:18.988189+0000","last_fullsized":"2024-09-10T19:21:47.973220+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":218,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:19:19.581564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:19:19.581564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.01041663,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":["5","4"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1e","version":"237'25","reported_seq":275,"reported_epoch":281,"state":"active+undersized+degraded","last_fresh":"2024-09-10T19:23:18.994062+0000","last_change":"2024-09-10T19:21:47.993629+0000","last_active":"2024-09-10T19:23:18.994062+0000","last_peered":"2024-09-10T19:23:18.994062+0000","last_clean":"2024-09-10T19:20:16.344252+0000","last_became_active":"2024-09-10T19:21:47.993629+0000","last_became_peered":"2024-09-10T19:21:47.993629+0000","last_unstale":"2024-09-10T19:23:18.994062+0000","last_undegraded":"2024-09-10T19:21:47.981614+0000","last_fullsized":"2024-09-10T19:21:47.981497+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"237'25","last_scrub_stamp":"2024-09-10T19:20:44.696751+0000","last_deep_scrub":"80'15","last_deep_scrub_stamp":"2024-09-10T19:11:25.883919+0000","last_clean_scrub_stamp":"2024-09-10T19:20:44.696751+0000","objects_scrubbed":5,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0040337309999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":24,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":["7","3"],"object_location_counts":[{"shards":"3,7","objects":5}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"66'8","reported_seq":462,"reported_epoch":281,"state":"active+clean","last_fresh":"2024-09-10T19:23:25.893770+0000","last_change":"2024-09-10T19:23:09.642484+0000","last_active":"2024-09-10T19:23:25.893770+0000","last_peered":"2024-09-10T19:23:25.893770+0000","last_clean":"2024-09-10T19:23:25.893770+0000","last_became_active":"2024-09-10T19:10:11.695427+0000","last_became_peered":"2024-09-10T19:10:11.695427+0000","last_unstale":"2024-09-10T19:23:25.893770+0000","last_undegraded":"2024-09-10T19:23:25.893770+0000","last_fullsized":"2024-09-10T19:23:25.893770+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'8","last_scrub_stamp":"2024-09-10T19:23:09.642403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:23:09.642403+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:24:32.150776+0000","scrub_duration":0.0047206909999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":242,"reported_epoch":281,"state":"active+clean","last_fresh":"2024-09-10T19:23:18.988935+0000","last_change":"2024-09-10T19:22:38.725243+0000","last_active":"2024-09-10T19:23:18.988935+0000","last_peered":"2024-09-10T19:23:18.988935+0000","last_clean":"2024-09-10T19:23:18.988935+0000","last_became_active":"2024-09-10T19:16:24.862695+0000","last_became_peered":"2024-09-10T19:16:24.862695+0000","last_unstale":"2024-09-10T19:23:18.988935+0000","last_undegraded":"2024-09-10T19:23:18.988935+0000","last_fullsized":"2024-09-10T19:23:18.988935+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:22:38.725165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:22:38.725165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:23:44.009816+0000","scrub_duration":0.0099096830000000007,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":279,"reported_epoch":281,"state":"active+undersized","last_fresh":"2024-09-10T19:23:18.986377+0000","last_change":"2024-09-10T19:21:47.988058+0000","last_active":"2024-09-10T19:23:18.986377+0000","last_peered":"2024-09-10T19:23:18.986377+0000","last_clean":"2024-09-10T19:21:44.676614+0000","last_became_active":"2024-09-10T19:21:47.988058+0000","last_became_peered":"2024-09-10T19:21:47.988058+0000","last_unstale":"2024-09-10T19:23:18.986377+0000","last_undegraded":"2024-09-10T19:23:18.986377+0000","last_fullsized":"2024-09-10T19:21:47.976497+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:21:44.676477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:21:44.676477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0017387360000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":["3","6"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"237'24","reported_seq":363,"reported_epoch":281,"state":"active+undersized+degraded","last_fresh":"2024-09-10T19:23:18.996858+0000","last_change":"2024-09-10T19:21:48.002057+0000","last_active":"2024-09-10T19:23:18.996858+0000","last_peered":"2024-09-10T19:23:18.996858+0000","last_clean":"2024-09-10T19:21:43.952583+0000","last_became_active":"2024-09-10T19:21:48.002057+0000","last_became_peered":"2024-09-10T19:21:48.002057+0000","last_unstale":"2024-09-10T19:23:18.996858+0000","last_undegraded":"2024-09-10T19:21:47.987701+0000","last_fullsized":"2024-09-10T19:21:47.985296+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"80'16","last_scrub_stamp":"2024-09-10T19:20:01.500062+0000","last_deep_scrub":"80'16","last_deep_scrub_stamp":"2024-09-10T19:11:40.853611+0000","last_clean_scrub_stamp":"2024-09-10T19:20:01.500062+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.002579296,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":["6","5"],"object_location_counts":[{"shards":"5,6","objects":5}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":150,"reported_epoch":281,"state":"active+undersized","last_fresh":"2024-09-10T19:23:18.993722+0000","last_change":"2024-09-10T19:21:47.989531+0000","last_active":"2024-09-10T19:23:18.993722+0000","last_peered":"2024-09-10T19:23:18.993722+0000","last_clean":"2024-09-10T19:16:23.155700+0000","last_became_active":"2024-09-10T19:21:47.989531+0000","last_became_peered":"2024-09-10T19:21:47.989531+0000","last_unstale":"2024-09-10T19:23:18.993722+0000","last_undegraded":"2024-09-10T19:23:18.993722+0000","last_fullsized":"2024-09-10T19:21:47.982576+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:20:01.699734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:11:28.822327+0000","last_clean_scrub_stamp":"2024-09-10T19:20:01.699734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0022130729999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":["7","3"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":275,"reported_epoch":281,"state":"active+clean","last_fresh":"2024-09-10T19:23:18.993537+0000","last_change":"2024-09-10T19:22:39.093858+0000","last_active":"2024-09-10T19:23:18.993537+0000","last_peered":"2024-09-10T19:23:18.993537+0000","last_clean":"2024-09-10T19:23:18.993537+0000","last_became_active":"2024-09-10T19:10:14.053619+0000","last_became_peered":"2024-09-10T19:10:14.053619+0000","last_unstale":"2024-09-10T19:23:18.993537+0000","last_undegraded":"2024-09-10T19:23:18.993537+0000","last_fullsized":"2024-09-10T19:23:18.993537+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:22:39.093573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:22:39.093573+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:24:02.502576+0000","scrub_duration":0.010663461000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":282,"reported_epoch":281,"state":"active+clean","last_fresh":"2024-09-10T19:23:19.006297+0000","last_change":"2024-09-10T19:22:31.719863+0000","last_active":"2024-09-10T19:23:19.006297+0000","last_peered":"2024-09-10T19:23:19.006297+0000","last_clean":"2024-09-10T19:23:19.006297+0000","last_became_active":"2024-09-10T19:10:07.685997+0000","last_became_peered":"2024-09-10T19:10:07.685997+0000","last_unstale":"2024-09-10T19:23:19.006297+0000","last_undegraded":"2024-09-10T19:23:19.006297+0000","last_fullsized":"2024-09-10T19:23:19.006297+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:22:31.719788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:22:31.719788+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:23:31.786150+0000","scrub_duration":0.007228801,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1c","version":"237'34","reported_seq":246,"reported_epoch":281,"state":"active+undersized+degraded","last_fresh":"2024-09-10T19:23:18.986809+0000","last_change":"2024-09-10T19:21:48.003880+0000","last_active":"2024-09-10T19:23:18.986809+0000","last_peered":"2024-09-10T19:23:18.986809+0000","last_clean":"2024-09-10T19:21:43.945320+0000","last_became_active":"2024-09-10T19:21:48.003880+0000","last_became_peered":"2024-09-10T19:21:48.003880+0000","last_unstale":"2024-09-10T19:23:18.986809+0000","last_undegraded":"2024-09-10T19:21:47.973954+0000","last_fullsized":"2024-09-10T19:21:47.973536+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"237'34","last_scrub_stamp":"2024-09-10T19:20:40.452293+0000","last_deep_scrub":"78'22","last_deep_scrub_stamp":"2024-09-10T19:11:21.118937+0000","last_clean_scrub_stamp":"2024-09-10T19:20:40.452293+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0025454940000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":["3","4"],"object_location_counts":[{"shards":"3,4","objects":7}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":232,"reported_epoch":281,"state":"active+undersized","last_fresh":"2024-09-10T19:23:18.996432+0000","last_change":"2024-09-10T19:21:47.981439+0000","last_active":"2024-09-10T19:23:18.996432+0000","last_peered":"2024-09-10T19:23:18.996432+0000","last_clean":"2024-09-10T19:21:43.948694+0000","last_became_active":"2024-09-10T19:21:47.981439+0000","last_became_peered":"2024-09-10T19:21:47.981439+0000","last_unstale":"2024-09-10T19:23:18.996432+0000","last_undegraded":"2024-09-10T19:23:18.996432+0000","last_fullsized":"2024-09-10T19:21:47.972032+0000","mapping_epoch":263,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:20:08.342716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:20:08.342716+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.002558444,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":["4","5"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":266,"reported_epoch":281,"state":"active+clean","last_fresh":"2024-09-10T19:23:18.987216+0000","last_change":"2024-09-10T19:21:57.736056+0000","last_active":"2024-09-10T19:23:18.987216+0000","last_peered":"2024-09-10T19:23:18.987216+0000","last_clean":"2024-09-10T19:23:18.987216+0000","last_became_active":"2024-09-10T19:10:14.053698+0000","last_became_peered":"2024-09-10T19:10:14.053698+0000","last_unstale":"2024-09-10T19:23:18.987216+0000","last_undegraded":"2024-09-10T19:23:18.987216+0000","last_fullsized":"2024-09-10T19:23:18.987216+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:21:57.736006+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:21:57.736006+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0017464049999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":272,"reported_epoch":281,"state":"active+clean","last_fresh":"2024-09-10T19:23:18.986374+0000","last_change":"2024-09-10T19:22:01.763166+0000","last_active":"2024-09-10T19:23:18.986374+0000","last_peered":"2024-09-10T19:23:18.986374+0000","last_clean":"2024-09-10T19:23:18.986374+0000","last_became_active":"2024-09-10T19:10:07.682811+0000","last_became_peered":"2024-09-10T19:10:07.682811+0000","last_unstale":"2024-09-10T19:23:18.986374+0000","last_undegraded":"2024-09-10T19:23:18.986374+0000","last_fullsized":"2024-09-10T19:23:18.986374+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:22:01.763088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:22:01.763088+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0083058279999999995,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1d","version":"250'23","reported_seq":336,"reported_epoch":281,"state":"active+clean","last_fresh":"2024-09-10T19:23:26.352467+0000","last_change":"2024-09-10T19:23:26.352416+0000","last_active":"2024-09-10T19:23:26.352467+0000","last_peered":"2024-09-10T19:23:26.352467+0000","last_clean":"2024-09-10T19:23:26.352467+0000","last_became_active":"2024-09-10T19:13:54.714629+0000","last_became_peered":"2024-09-10T19:13:54.714629+0000","last_unstale":"2024-09-10T19:23:26.352467+0000","last_undegraded":"2024-09-10T19:23:26.352467+0000","last_fullsized":"2024-09-10T19:23:26.352467+0000","mapping_epoch":115,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":116,"parent":"0.0","parent_split_bits":0,"last_scrub":"250'23","last_scrub_stamp":"2024-09-10T19:23:26.352362+0000","last_deep_scrub":"250'23","last_deep_scrub_stamp":"2024-09-10T19:20:55.582173+0000","last_clean_scrub_stamp":"2024-09-10T19:23:26.352362+0000","objects_scrubbed":5,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:24:38.241915+0000","scrub_duration":0.0017939849999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":287,"reported_epoch":281,"state":"active+clean","last_fresh":"2024-09-10T19:23:18.998053+0000","last_change":"2024-09-10T19:22:26.010505+0000","last_active":"2024-09-10T19:23:18.998053+0000","last_peered":"2024-09-10T19:23:18.998053+0000","last_clean":"2024-09-10T19:23:18.998053+0000","last_became_active":"2024-09-10T19:16:24.843013+0000","last_became_peered":"2024-09-10T19:16:24.843013+0000","last_unstale":"2024-09-10T19:23:18.998053+0000","last_undegraded":"2024-09-10T19:23:18.998053+0000","last_fullsized":"2024-09-10T19:23:18.998053+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:22:26.010416+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:22:26.010416+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:23:30.838871+0000","scrub_duration":0.012466878000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":9,"num_pg":16,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":292974,"num_write_kb":9375105,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4143579136,"data_stored":4143579136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":60381,"ondisk_log_size":60381,"up":40,"acting":40,"num_store_stats":6},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":382,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":1528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":82,"acting":82,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":78,"ondisk_log_size":78,"up":77,"acting":77,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":177,"num_object_clones":0,"num_object_copies":531,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":80,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":177,"num_whiteouts":0,"num_read":1119,"num_read_kb":876,"num_write":703,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":178,"num_bytes_recovered":3328,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":540672,"data_stored":14242,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":758,"ondisk_log_size":758,"up":82,"acting":82,"num_store_stats":8},{"poolid":3,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":428,"num_read_kb":423,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":389,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":2002,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":9,"ondisk_log_size":9,"up":7,"acting":7,"num_store_stats":7},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1328,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":27,"num_read_kb":27,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":3984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4,"ondisk_log_size":4,"up":79,"acting":79,"num_store_stats":8},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":644038,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":11,"num_read_kb":20,"num_write":11,"num_write_kb":148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":53670,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":869,"ondisk_log_size":869,"up":29,"acting":29,"num_store_stats":6}],"osd_stats":[{"osd":7,"up_from":54,"seq":231928234149,"num_pgs":83,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":893464,"kb_used_data":1164,"kb_used_omap":1,"kb_used_meta":892286,"kb_avail":92851688,"statfs":{"total":95995035648,"available":95080128512,"internally_reserved":0,"allocated":1191936,"data_stored":831545,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":913701208},"hb_peers":[1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 19:21:08 2024","interfaces":[{"interface":"back","average":{"1min":0.61299999999999999,"5min":1.046,"15min":1.113},"min":{"1min":0.215,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":2.3620000000000001,"5min":8.3770000000000007,"15min":35.551000000000002},"last":0.91900000000000004},{"interface":"front","average":{"1min":1.879,"5min":1.2789999999999999,"15min":1.131},"min":{"1min":0.24399999999999999,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":5.0140000000000002,"5min":8.1440000000000001,"15min":35.503999999999998},"last":0.97999999999999998}]},{"osd":1,"last update":"Tue Sep 10 19:20:29 2024","interfaces":[{"interface":"back","average":{"1min":0.751,"5min":1.0609999999999999,"15min":0.70299999999999996},"min":{"1min":0.26000000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":2.5089999999999999,"5min":8.3499999999999996,"15min":8.3499999999999996},"last":0.5},{"interface":"front","average":{"1min":0.78000000000000003,"5min":1.0209999999999999,"15min":0.81200000000000006},"min":{"1min":0.28799999999999998,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":2.5640000000000001,"5min":5.609,"15min":5.609},"last":0.59799999999999998}]},{"osd":2,"last update":"Tue Sep 10 19:23:10 2024","interfaces":[{"interface":"back","average":{"1min":1.1859999999999999,"5min":1.0940000000000001,"15min":0.92800000000000005},"min":{"1min":0.23000000000000001,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":4.0190000000000001,"5min":4.0190000000000001,"15min":31.940999999999999},"last":0.51700000000000002},{"interface":"front","average":{"1min":1.0820000000000001,"5min":0.88500000000000001,"15min":0.98899999999999999},"min":{"1min":0.26300000000000001,"5min":0.26300000000000001,"15min":0.24099999999999999},"max":{"1min":3.613,"5min":4.609,"15min":25.056000000000001},"last":0.55300000000000005}]},{"osd":3,"last update":"Tue Sep 10 19:23:10 2024","interfaces":[{"interface":"back","average":{"1min":1.1739999999999999,"5min":1.131,"15min":1.2929999999999999},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":3.984,"5min":4.6070000000000002,"15min":35.277999999999999},"last":0.436},{"interface":"front","average":{"1min":1.0640000000000001,"5min":0.96299999999999997,"15min":1.238},"min":{"1min":0.27800000000000002,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":3.6419999999999999,"5min":5.0119999999999996,"15min":36.512999999999998},"last":0.62}]},{"osd":4,"last update":"Tue Sep 10 19:23:10 2024","interfaces":[{"interface":"back","average":{"1min":2.0990000000000002,"5min":1.966,"15min":1.1559999999999999},"min":{"1min":0.23999999999999999,"5min":0.222,"15min":0.222},"max":{"1min":6.1059999999999999,"5min":8.4399999999999995,"15min":8.4399999999999995},"last":0.53300000000000003},{"interface":"front","average":{"1min":1.843,"5min":1.536,"15min":0.91700000000000004},"min":{"1min":0.25,"5min":0.23100000000000001,"15min":0.23000000000000001},"max":{"1min":5.5389999999999997,"5min":6.149,"15min":6.149},"last":0.47399999999999998}]},{"osd":5,"last update":"Tue Sep 10 19:23:10 2024","interfaces":[{"interface":"back","average":{"1min":5.7569999999999997,"5min":2.5630000000000002,"15min":1.4419999999999999},"min":{"1min":0.34100000000000003,"5min":0.30399999999999999,"15min":0.23699999999999999},"max":{"1min":67.382999999999996,"5min":67.382999999999996,"15min":67.382999999999996},"last":0.63900000000000001},{"interface":"front","average":{"1min":6.4580000000000002,"5min":3.2450000000000001,"15min":1.615},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.26000000000000001},"max":{"1min":67.75,"5min":67.75,"15min":67.75},"last":0.60899999999999999}]},{"osd":6,"last update":"Tue Sep 10 19:23:10 2024","interfaces":[{"interface":"back","average":{"1min":10.581,"5min":3.6520000000000001,"15min":1.7849999999999999},"min":{"1min":0.313,"5min":0.312,"15min":0.26200000000000001},"max":{"1min":181.53800000000001,"5min":181.53800000000001,"15min":181.53800000000001},"last":0.56999999999999995},{"interface":"front","average":{"1min":10.192,"5min":3.3130000000000002,"15min":1.5820000000000001},"min":{"1min":0.27000000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":182.05600000000001,"5min":182.05600000000001,"15min":182.05600000000001},"last":0.58199999999999996}]}]},{"osd":6,"up_from":47,"seq":201863463082,"num_pgs":78,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":603472,"kb_used_data":1092,"kb_used_omap":1,"kb_used_meta":602366,"kb_avail":93141680,"statfs":{"total":95995035648,"available":95377080320,"internally_reserved":0,"allocated":1118208,"data_stored":766629,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":616823128},"hb_peers":[1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 19:20:51 2024","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":0.69899999999999995,"15min":1.1279999999999999},"min":{"1min":0.23100000000000001,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":2.2010000000000001,"5min":5.2350000000000003,"15min":112.988},"last":2.3159999999999998},{"interface":"front","average":{"1min":0.59799999999999998,"5min":0.90500000000000003,"15min":1.1990000000000001},"min":{"1min":0.221,"5min":0.221,"15min":0.221},"max":{"1min":2.0009999999999999,"5min":5.274,"15min":113.18300000000001},"last":0.41899999999999998}]},{"osd":1,"last update":"Tue Sep 10 19:20:35 2024","interfaces":[{"interface":"back","average":{"1min":0.51800000000000002,"5min":0.70999999999999996,"15min":0.57899999999999996},"min":{"1min":0.31900000000000001,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":0.995,"5min":4.2919999999999998,"15min":4.2919999999999998},"last":0.44500000000000001},{"interface":"front","average":{"1min":0.59899999999999998,"5min":0.91000000000000003,"15min":0.73899999999999999},"min":{"1min":0.29699999999999999,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":2.0750000000000002,"5min":6.5289999999999999,"15min":6.5289999999999999},"last":0.51500000000000001}]},{"osd":2,"last update":"Tue Sep 10 19:22:53 2024","interfaces":[{"interface":"back","average":{"1min":0.54700000000000004,"5min":0.83099999999999996,"15min":1.129},"min":{"1min":0.23799999999999999,"5min":0.23799999999999999,"15min":0.20399999999999999},"max":{"1min":1.5509999999999999,"5min":3.6099999999999999,"15min":112.864},"last":0.47699999999999998},{"interface":"front","average":{"1min":0.97299999999999998,"5min":0.97999999999999998,"15min":1.2350000000000001},"min":{"1min":0.25600000000000001,"5min":0.25600000000000001,"15min":0.254},"max":{"1min":4.2610000000000001,"5min":5.8300000000000001,"15min":112.822},"last":0.495}]},{"osd":3,"last update":"Tue Sep 10 19:22:53 2024","interfaces":[{"interface":"back","average":{"1min":0.55200000000000005,"5min":0.96299999999999997,"15min":1.411},"min":{"1min":0.24099999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":1.6080000000000001,"5min":6.4870000000000001,"15min":125.56399999999999},"last":0.37},{"interface":"front","average":{"1min":0.53300000000000003,"5min":0.91100000000000003,"15min":1.429},"min":{"1min":0.20499999999999999,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":1.522,"5min":6.5259999999999998,"15min":125.47},"last":0.55000000000000004}]},{"osd":4,"last update":"Tue Sep 10 19:22:53 2024","interfaces":[{"interface":"back","average":{"1min":1.3029999999999999,"5min":1.5129999999999999,"15min":0.91100000000000003},"min":{"1min":0.29599999999999999,"5min":0.253,"15min":0.20699999999999999},"max":{"1min":4.2240000000000002,"5min":6.9269999999999996,"15min":6.9269999999999996},"last":0.46100000000000002},{"interface":"front","average":{"1min":1.165,"5min":1.2729999999999999,"15min":0.82099999999999995},"min":{"1min":0.28299999999999997,"5min":0.24099999999999999,"15min":0.22500000000000001},"max":{"1min":3.714,"5min":6.907,"15min":6.907},"last":0.41499999999999998}]},{"osd":5,"last update":"Tue Sep 10 19:22:53 2024","interfaces":[{"interface":"back","average":{"1min":1.278,"5min":1.532,"15min":1.0980000000000001},"min":{"1min":0.29299999999999998,"5min":0.29299999999999998,"15min":0.248},"max":{"1min":4.5519999999999996,"5min":5.9640000000000004,"15min":44.765000000000001},"last":0.53800000000000003},{"interface":"front","average":{"1min":1.3089999999999999,"5min":1.5489999999999999,"15min":1.157},"min":{"1min":0.29599999999999999,"5min":0.28399999999999997,"15min":0.27500000000000002},"max":{"1min":3.7629999999999999,"5min":7.1369999999999996,"15min":43.036000000000001},"last":0.48999999999999999}]},{"osd":7,"last update":"Tue Sep 10 19:23:22 2024","interfaces":[{"interface":"back","average":{"1min":1.3080000000000001,"5min":1.1539999999999999,"15min":0.84699999999999998},"min":{"1min":0.29599999999999999,"5min":0.29599999999999999,"15min":0.27500000000000002},"max":{"1min":5.673,"5min":5.673,"15min":8.9390000000000001},"last":0.57199999999999995},{"interface":"front","average":{"1min":1.5660000000000001,"5min":1.585,"15min":1.0489999999999999},"min":{"1min":0.317,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":6.8049999999999997,"5min":6.9720000000000004,"15min":10.298},"last":0.52800000000000002}]}]},{"osd":5,"up_from":39,"seq":167503724719,"num_pgs":80,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":895172,"kb_used_data":1400,"kb_used_omap":1,"kb_used_meta":893758,"kb_avail":92849980,"statfs":{"total":95995035648,"available":95078379520,"internally_reserved":0,"allocated":1433600,"data_stored":1095040,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":915208536},"hb_peers":[1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 19:21:23 2024","interfaces":[{"interface":"back","average":{"1min":1.1579999999999999,"5min":1.9570000000000001,"15min":1.1060000000000001},"min":{"1min":0.19600000000000001,"5min":0.17999999999999999,"15min":0.17999999999999999},"max":{"1min":3.9009999999999998,"5min":73.933000000000007,"15min":73.933000000000007},"last":4.4960000000000004},{"interface":"front","average":{"1min":1.286,"5min":1.821,"15min":1.024},"min":{"1min":0.23100000000000001,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":3.9969999999999999,"5min":73.769999999999996,"15min":73.769999999999996},"last":3.2749999999999999}]},{"osd":1,"last update":"Tue Sep 10 19:20:27 2024","interfaces":[{"interface":"back","average":{"1min":0.73299999999999998,"5min":0.98499999999999999,"15min":0.76300000000000001},"min":{"1min":0.23300000000000001,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":2.2050000000000001,"5min":4.3499999999999996,"15min":4.3499999999999996},"last":0.26300000000000001},{"interface":"front","average":{"1min":0.875,"5min":0.97799999999999998,"15min":0.66100000000000003},"min":{"1min":0.28999999999999998,"5min":0.255,"15min":0.255},"max":{"1min":4.742,"5min":5.0570000000000004,"15min":5.0570000000000004},"last":0.34300000000000003}]},{"osd":2,"last update":"Tue Sep 10 19:23:23 2024","interfaces":[{"interface":"back","average":{"1min":1.103,"5min":1.2569999999999999,"15min":0.95899999999999996},"min":{"1min":0.245,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":2.8980000000000001,"5min":8.6920000000000002,"15min":9.9399999999999995},"last":0.41799999999999998},{"interface":"front","average":{"1min":0.90200000000000002,"5min":1.147,"15min":0.93000000000000005},"min":{"1min":0.22500000000000001,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":3.0880000000000001,"5min":4.4139999999999997,"15min":6.3789999999999996},"last":0.375}]},{"osd":3,"last update":"Tue Sep 10 19:23:23 2024","interfaces":[{"interface":"back","average":{"1min":2.585,"5min":1.5249999999999999,"15min":1.046},"min":{"1min":0.222,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":26.355,"5min":26.355,"15min":26.355},"last":0.38600000000000001},{"interface":"front","average":{"1min":2.4100000000000001,"5min":1.508,"15min":1.0149999999999999},"min":{"1min":0.246,"5min":0.246,"15min":0.246},"max":{"1min":26.327000000000002,"5min":26.327000000000002,"15min":26.327000000000002},"last":0.432}]},{"osd":4,"last update":"Tue Sep 10 19:23:23 2024","interfaces":[{"interface":"back","average":{"1min":8.8089999999999993,"5min":2.9279999999999999,"15min":1.486},"min":{"1min":0.26100000000000001,"5min":0.253,"15min":0.20699999999999999},"max":{"1min":128.08099999999999,"5min":128.08099999999999,"15min":128.08099999999999},"last":0.33000000000000002},{"interface":"front","average":{"1min":8.8759999999999994,"5min":3.0489999999999999,"15min":1.5640000000000001},"min":{"1min":0.30399999999999999,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":126.321,"5min":126.321,"15min":126.321},"last":0.46999999999999997}]},{"osd":6,"last update":"Tue Sep 10 19:22:51 2024","interfaces":[{"interface":"back","average":{"1min":7.1180000000000003,"5min":2.6619999999999999,"15min":1.482},"min":{"1min":0.317,"5min":0.28399999999999997,"15min":0.223},"max":{"1min":111.328,"5min":111.328,"15min":111.328},"last":0.44400000000000001},{"interface":"front","average":{"1min":7.0640000000000001,"5min":2.669,"15min":1.4179999999999999},"min":{"1min":0.28799999999999998,"5min":0.28799999999999998,"15min":0.25},"max":{"1min":111.705,"5min":111.705,"15min":111.705},"last":0.40600000000000003}]},{"osd":7,"last update":"Tue Sep 10 19:23:17 2024","interfaces":[{"interface":"back","average":{"1min":1.911,"5min":1.6899999999999999,"15min":1.105},"min":{"1min":0.313,"5min":0.29999999999999999,"15min":0.28899999999999998},"max":{"1min":6.4790000000000001,"5min":6.7869999999999999,"15min":6.7869999999999999},"last":0.46000000000000002},{"interface":"front","average":{"1min":1.4770000000000001,"5min":1.6619999999999999,"15min":1.0680000000000001},"min":{"1min":0.29299999999999998,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":6.5279999999999996,"5min":9.4909999999999997,"15min":9.4909999999999997},"last":0.39800000000000002}]}]},{"osd":4,"up_from":32,"seq":137438953651,"num_pgs":71,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":893372,"kb_used_data":816,"kb_used_omap":1,"kb_used_meta":892542,"kb_avail":92851780,"statfs":{"total":95995035648,"available":95080222720,"internally_reserved":0,"allocated":835584,"data_stored":502918,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":913963352},"hb_peers":[1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 19:21:10 2024","interfaces":[{"interface":"back","average":{"1min":1.0209999999999999,"5min":1.224,"15min":0.97999999999999998},"min":{"1min":0.25700000000000001,"5min":0.187,"15min":0.187},"max":{"1min":5.1420000000000003,"5min":24.113,"15min":32.533000000000001},"last":0.53100000000000003},{"interface":"front","average":{"1min":0.82799999999999996,"5min":1.1739999999999999,"15min":0.98099999999999998},"min":{"1min":0.20399999999999999,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":3.3100000000000001,"5min":24.132999999999999,"15min":32.512999999999998},"last":1.6160000000000001}]},{"osd":1,"last update":"Tue Sep 10 19:20:27 2024","interfaces":[{"interface":"back","average":{"1min":0.46700000000000003,"5min":0.91900000000000004,"15min":0.72499999999999998},"min":{"1min":0.23200000000000001,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":1.1779999999999999,"5min":7.9770000000000003,"15min":7.9770000000000003},"last":0.61299999999999999},{"interface":"front","average":{"1min":0.78700000000000003,"5min":0.85599999999999998,"15min":0.73599999999999999},"min":{"1min":0.32300000000000001,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":2.3700000000000001,"5min":3.8260000000000001,"15min":3.8260000000000001},"last":0.45200000000000001}]},{"osd":2,"last update":"Tue Sep 10 19:23:17 2024","interfaces":[{"interface":"back","average":{"1min":0.84699999999999998,"5min":0.91800000000000004,"15min":0.78300000000000003},"min":{"1min":0.25900000000000001,"5min":0.254,"15min":0.214},"max":{"1min":2.0750000000000002,"5min":5.048,"15min":6.8869999999999996},"last":0.56999999999999995},{"interface":"front","average":{"1min":0.70999999999999996,"5min":0.96599999999999997,"15min":0.78900000000000003},"min":{"1min":0.25800000000000001,"5min":0.253,"15min":0.23400000000000001},"max":{"1min":3,"5min":4.7169999999999996,"15min":7.085},"last":0.48899999999999999}]},{"osd":3,"last update":"Tue Sep 10 19:23:17 2024","interfaces":[{"interface":"back","average":{"1min":0.69299999999999995,"5min":1.1579999999999999,"15min":1.004},"min":{"1min":0.22900000000000001,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":2.9710000000000001,"5min":15.436,"15min":35.451999999999998},"last":0.54200000000000004},{"interface":"front","average":{"1min":0.82299999999999995,"5min":1.1140000000000001,"15min":1.0660000000000001},"min":{"1min":0.246,"5min":0.246,"15min":0.246},"max":{"1min":2.6400000000000001,"5min":14.954000000000001,"15min":32.880000000000003},"last":0.52500000000000002}]},{"osd":5,"last update":"Tue Sep 10 19:22:31 2024","interfaces":[{"interface":"back","average":{"1min":13.792,"5min":4.0039999999999996,"15min":1.994},"min":{"1min":0.39600000000000002,"5min":0.224,"15min":0.224},"max":{"1min":227.726,"5min":227.726,"15min":227.726},"last":0.40999999999999998},{"interface":"front","average":{"1min":13.819000000000001,"5min":3.952,"15min":1.925},"min":{"1min":0.20699999999999999,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":227.70500000000001,"5min":227.70500000000001,"15min":227.70500000000001},"last":0.30099999999999999}]},{"osd":6,"last update":"Tue Sep 10 19:23:09 2024","interfaces":[{"interface":"back","average":{"1min":1.0649999999999999,"5min":1.2589999999999999,"15min":0.84299999999999997},"min":{"1min":0.38500000000000001,"5min":0.27800000000000002,"15min":0.253},"max":{"1min":2.125,"5min":5.1130000000000004,"15min":5.5289999999999999},"last":0.42599999999999999},{"interface":"front","average":{"1min":1.4830000000000001,"5min":1.288,"15min":0.86699999999999999},"min":{"1min":0.28000000000000003,"5min":0.25800000000000001,"15min":0.247},"max":{"1min":4.476,"5min":7.3390000000000004,"15min":7.9550000000000001},"last":0.35699999999999998}]},{"osd":7,"last update":"Tue Sep 10 19:23:21 2024","interfaces":[{"interface":"back","average":{"1min":1.5700000000000001,"5min":1.492,"15min":1.127},"min":{"1min":0.54400000000000004,"5min":0.308,"15min":0.27700000000000002},"max":{"1min":4.6630000000000003,"5min":5.9809999999999999,"15min":23.081},"last":0.46600000000000003},{"interface":"front","average":{"1min":0.84999999999999998,"5min":1.431,"15min":1.081},"min":{"1min":0.28000000000000003,"5min":0.28000000000000003,"15min":0.23699999999999999},"max":{"1min":2.661,"5min":8.8550000000000004,"15min":22.666},"last":0.38600000000000001}]}]},{"osd":3,"up_from":26,"seq":111669149880,"num_pgs":84,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":897568,"kb_used_data":1428,"kb_used_omap":1,"kb_used_meta":896126,"kb_avail":92847584,"statfs":{"total":95995035648,"available":95075926016,"internally_reserved":0,"allocated":1462272,"data_stored":1095449,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":917633368},"hb_peers":[1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 19:20:46 2024","interfaces":[{"interface":"back","average":{"1min":0.40799999999999997,"5min":0.39500000000000002,"15min":0.69099999999999995},"min":{"1min":0.16700000000000001,"5min":0.157,"15min":0.157},"max":{"1min":0.77800000000000002,"5min":3.9529999999999998,"15min":17.518000000000001},"last":0.39000000000000001},{"interface":"front","average":{"1min":0.72499999999999998,"5min":0.63900000000000001,"15min":0.84799999999999998},"min":{"1min":0.19400000000000001,"5min":0.17899999999999999,"15min":0.17399999999999999},"max":{"1min":2.3969999999999998,"5min":5.1539999999999999,"15min":17.609000000000002},"last":0.40799999999999997}]},{"osd":1,"last update":"Tue Sep 10 19:20:25 2024","interfaces":[{"interface":"back","average":{"1min":0.43099999999999999,"5min":0.45600000000000002,"15min":0.51200000000000001},"min":{"1min":0.222,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":1.2869999999999999,"5min":2.6059999999999999,"15min":2.6059999999999999},"last":0.26900000000000002},{"interface":"front","average":{"1min":0.438,"5min":0.72399999999999998,"15min":1.2869999999999999},"min":{"1min":0.248,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":1.1919999999999999,"5min":17.454999999999998,"15min":17.454999999999998},"last":0.311}]},{"osd":2,"last update":"Tue Sep 10 19:22:51 2024","interfaces":[{"interface":"back","average":{"1min":0.34300000000000003,"5min":0.438,"15min":0.64900000000000002},"min":{"1min":0.17599999999999999,"5min":0.17599999999999999,"15min":0.17599999999999999},"max":{"1min":2.2530000000000001,"5min":2.2530000000000001,"15min":5.3029999999999999},"last":0.28699999999999998},{"interface":"front","average":{"1min":0.245,"5min":0.39100000000000001,"15min":0.70599999999999996},"min":{"1min":0.16400000000000001,"5min":0.16400000000000001,"15min":0.16400000000000001},"max":{"1min":0.372,"5min":2.5649999999999999,"15min":17.806000000000001},"last":0.33300000000000002}]},{"osd":4,"last update":"Tue Sep 10 19:23:14 2024","interfaces":[{"interface":"back","average":{"1min":0.622,"5min":0.83399999999999996,"15min":0.92500000000000004},"min":{"1min":0.28899999999999998,"5min":0.26900000000000002,"15min":0.23599999999999999},"max":{"1min":2.5779999999999998,"5min":5.2750000000000004,"15min":17.834},"last":0.53400000000000003},{"interface":"front","average":{"1min":0.86399999999999999,"5min":0.95799999999999996,"15min":1.0840000000000001},"min":{"1min":0.23899999999999999,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":4.3369999999999997,"5min":4.3369999999999997,"15min":17.553000000000001},"last":0.42899999999999999}]},{"osd":5,"last update":"Tue Sep 10 19:22:29 2024","interfaces":[{"interface":"back","average":{"1min":1.542,"5min":1.272,"15min":1.29},"min":{"1min":0.309,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":6.415,"5min":7.2949999999999999,"15min":80.942999999999998},"last":0.47699999999999998},{"interface":"front","average":{"1min":1.3600000000000001,"5min":1.1180000000000001,"15min":1.3879999999999999},"min":{"1min":0.24299999999999999,"5min":0.24299999999999999,"15min":0.22},"max":{"1min":4.4779999999999998,"5min":4.835,"15min":81.489000000000004},"last":0.39500000000000002}]},{"osd":6,"last update":"Tue Sep 10 19:22:55 2024","interfaces":[{"interface":"back","average":{"1min":8.6829999999999998,"5min":2.605,"15min":1.554},"min":{"1min":0.23699999999999999,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":116.14,"5min":116.14,"15min":116.14},"last":0.441},{"interface":"front","average":{"1min":9.0530000000000008,"5min":2.556,"15min":1.45},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":116.095,"5min":116.095,"15min":116.095},"last":0.38200000000000001}]},{"osd":7,"last update":"Tue Sep 10 19:23:27 2024","interfaces":[{"interface":"back","average":{"1min":5.6470000000000002,"5min":2.0640000000000001,"15min":1.3580000000000001},"min":{"1min":0.314,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":104.749,"5min":104.749,"15min":104.749},"last":0.42099999999999999},{"interface":"front","average":{"1min":5.6749999999999998,"5min":2.02,"15min":1.375},"min":{"1min":0.317,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":104.69499999999999,"5min":104.69499999999999,"15min":104.69499999999999},"last":0.505}]}]},{"osd":2,"up_from":0,"seq":77309411517,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":0,"seq":1185410973707,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":8,"seq":34359738546,"num_pgs":78,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":4950592,"kb_used_data":4059380,"kb_used_omap":1,"kb_used_meta":891198,"kb_avail":88794560,"statfs":{"total":95995035648,"available":90925629440,"internally_reserved":0,"allocated":4156805120,"data_stored":4156266196,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":912587096},"hb_peers":[2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Sep 10 19:20:34 2024","interfaces":[{"interface":"back","average":{"1min":0.50800000000000001,"5min":0.438,"15min":0.39200000000000002},"min":{"1min":0.20999999999999999,"5min":0.19,"15min":0.19},"max":{"1min":1.6799999999999999,"5min":2.0819999999999999,"15min":2.0819999999999999},"last":0.54300000000000004},{"interface":"front","average":{"1min":0.38600000000000001,"5min":0.40899999999999997,"15min":0.41899999999999998},"min":{"1min":0.19900000000000001,"5min":0.193,"15min":0.193},"max":{"1min":0.622,"5min":2.2839999999999998,"15min":2.2839999999999998},"last":0.32900000000000001}]},{"osd":2,"last update":"Tue Sep 10 19:21:39 2024","interfaces":[{"interface":"back","average":{"1min":0.46800000000000003,"5min":0.48999999999999999,"15min":0.629},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.17199999999999999},"max":{"1min":2.0840000000000001,"5min":2.0840000000000001,"15min":6.4640000000000004},"last":0.35199999999999998},{"interface":"front","average":{"1min":0.41399999999999998,"5min":0.46500000000000002,"15min":0.60999999999999999},"min":{"1min":0.22500000000000001,"5min":0.22500000000000001,"15min":0.187},"max":{"1min":1.2090000000000001,"5min":2.8660000000000001,"15min":9.9399999999999995},"last":0.41099999999999998}]},{"osd":3,"last update":"Tue Sep 10 19:20:49 2024","interfaces":[{"interface":"back","average":{"1min":0.51900000000000002,"5min":0.58499999999999996,"15min":0.68000000000000005},"min":{"1min":0.29199999999999998,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":1.0129999999999999,"5min":3.6190000000000002,"15min":6.5590000000000002},"last":0.29699999999999999},{"interface":"front","average":{"1min":0.39200000000000002,"5min":0.44,"15min":0.70399999999999996},"min":{"1min":0.23899999999999999,"5min":0.23899999999999999,"15min":0.23000000000000001},"max":{"1min":0.69799999999999995,"5min":1.2649999999999999,"15min":7.2140000000000004},"last":0.22}]},{"osd":4,"last update":"Tue Sep 10 19:21:07 2024","interfaces":[{"interface":"back","average":{"1min":1.0269999999999999,"5min":1.024,"15min":0.79000000000000004},"min":{"1min":0.29299999999999998,"5min":0.29099999999999998,"15min":0.24399999999999999},"max":{"1min":2.2829999999999999,"5min":7.4560000000000004,"15min":7.4560000000000004},"last":0.438},{"interface":"front","average":{"1min":0.877,"5min":0.96099999999999997,"15min":0.77800000000000002},"min":{"1min":0.31,"5min":0.28499999999999998,"15min":0.27100000000000002},"max":{"1min":2.6080000000000001,"5min":5.2640000000000002,"15min":5.2640000000000002},"last":0.61099999999999999}]},{"osd":5,"last update":"Tue Sep 10 19:21:39 2024","interfaces":[{"interface":"back","average":{"1min":0.82099999999999995,"5min":1.036,"15min":0.85299999999999998},"min":{"1min":0.28199999999999997,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":2.7090000000000001,"5min":4.327,"15min":5.3860000000000001},"last":0.69399999999999995},{"interface":"front","average":{"1min":0.95399999999999996,"5min":0.91700000000000004,"15min":0.80000000000000004},"min":{"1min":0.27700000000000002,"5min":0.27700000000000002,"15min":0.253},"max":{"1min":3.871,"5min":4.165,"15min":5.9749999999999996},"last":0.26700000000000002}]},{"osd":6,"last update":"Tue Sep 10 19:20:49 2024","interfaces":[{"interface":"back","average":{"1min":0.72099999999999997,"5min":0.92100000000000004,"15min":0.80700000000000005},"min":{"1min":0.32000000000000001,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":1.512,"5min":4.9420000000000002,"15min":5.8730000000000002},"last":0.60099999999999998},{"interface":"front","average":{"1min":0.91500000000000004,"5min":0.92700000000000005,"15min":0.82099999999999995},"min":{"1min":0.34300000000000003,"5min":0.32900000000000001,"15min":0.28699999999999998},"max":{"1min":3.6850000000000001,"5min":4.8300000000000001,"15min":5.2850000000000001},"last":0.65600000000000003}]},{"osd":7,"last update":"Tue Sep 10 19:21:19 2024","interfaces":[{"interface":"back","average":{"1min":1.369,"5min":1.0549999999999999,"15min":0.86099999999999999},"min":{"1min":0.317,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":3.1579999999999999,"5min":5.1479999999999997,"15min":5.1479999999999997},"last":0.71999999999999997},{"interface":"front","average":{"1min":1.2470000000000001,"5min":1.0329999999999999,"15min":0.83599999999999997},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.30499999999999999},"max":{"1min":4.5720000000000001,"5min":5.3410000000000002,"15min":6.0469999999999997},"last":1.411}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":966,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1236,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1236,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":362,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":1760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":2118,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1018,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":2228,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1584,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1942,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":2162,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4143579136,"data_stored":4143579136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T19:23:37.395 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph osd pool set unique_pool_2 pgp_num 16 2024-09-10T19:23:37.439 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:23:37.450 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:23:37 smithi191 ceph-mon[32884]: pgmap v599: 158 pgs: 53 active+undersized, 25 active+undersized+degraded, 80 active+clean; 635 KiB data, 8.7 GiB used, 528 GiB / 536 GiB avail; 1.2 KiB/s rd, 1 op/s; 92/591 objects degraded (15.567%) 2024-09-10T19:23:37.462 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:23:37.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:37.463 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:23:37.564 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:23:37.586 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:23:37.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:37.587 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:23:37.688 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:23:37.714 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:23:37.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:37.715 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:23:37.715 INFO:journalctl@ceph.mgr.x.smithi191.stdout:Sep 10 19:23:37 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-x[34967]: ::ffff:172.21.15.191 - - [10/Sep/2024:19:23:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T19:23:37.816 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:23:37.842 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:23:37.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:37.843 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:23:37.944 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:23:37.969 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:23:37.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:37.970 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:23:38.070 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:23:38.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:38.093 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:23:38.093 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:23:38.194 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:23:38.215 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:23:38.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:38.216 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:23:38.317 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:23:38.341 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:23:38.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:38.342 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:23:38.444 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:23:38.465 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:23:38.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:38.467 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:23:38.567 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:23:38.592 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:23:38.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:38.593 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:23:38.694 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:23:38.718 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:23:38.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:38.719 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:23:38.820 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:23:38.841 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:23:38.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:38.842 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:23:38.943 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:23:38.965 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:23:38.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:38.966 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:23:39.066 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:23:39.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:39.090 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:23:39.090 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:23:39.191 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:23:39.212 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:23:39.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:39.311 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:23:39.356 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:23:39 smithi183 ceph-mon[29736]: pgmap v600: 158 pgs: 53 active+undersized, 25 active+undersized+degraded, 80 active+clean; 635 KiB data, 8.7 GiB used, 528 GiB / 536 GiB avail; 1.1 KiB/s rd, 1 op/s; 92/591 objects degraded (15.567%) 2024-09-10T19:23:39.356 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:23:39 smithi183 ceph-mon[41562]: pgmap v600: 158 pgs: 53 active+undersized, 25 active+undersized+degraded, 80 active+clean; 635 KiB data, 8.7 GiB used, 528 GiB / 536 GiB avail; 1.1 KiB/s rd, 1 op/s; 92/591 objects degraded (15.567%) 2024-09-10T19:23:39.412 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:23:39.433 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:23:39.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:39.434 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:23:39.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:23:39 smithi191 ceph-mon[32884]: pgmap v600: 158 pgs: 53 active+undersized, 25 active+undersized+degraded, 80 active+clean; 635 KiB data, 8.7 GiB used, 528 GiB / 536 GiB avail; 1.1 KiB/s rd, 1 op/s; 92/591 objects degraded (15.567%) 2024-09-10T19:23:39.535 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:23:39.558 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:23:39.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:39.559 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:23:39.660 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:23:39.682 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:23:39.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:39.683 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:23:39.784 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:23:39.808 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:23:39.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:39.809 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:23:39.878 INFO:teuthology.orchestra.run.smithi183.stdout:ERROR: (22) Invalid argument 2024-09-10T19:23:39.878 INFO:teuthology.orchestra.run.smithi183.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-10T19:23:39.910 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:23:39.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:39.934 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:23:39.934 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:23:39.965 INFO:journalctl@ceph.iscsi.iscsi.a.smithi191.stdout:Sep 10 19:23:39 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-iscsi-iscsi-a[61816]: debug there is no tcmu-runner data available 2024-09-10T19:23:40.035 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:23:40.057 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:23:40.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:40.058 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:23:40.159 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:23:40.180 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:23:40.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:40.181 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:23:40.282 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:23:40.305 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:23:40.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:40.306 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:23:40.406 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:23:40.429 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:23:40.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:40.430 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:23:40.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:23:40 smithi183 ceph-mon[29736]: purged_snaps scrub starts 2024-09-10T19:23:40.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:23:40 smithi183 ceph-mon[29736]: purged_snaps scrub ok 2024-09-10T19:23:40.431 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:23:40 smithi183 ceph-mon[41562]: purged_snaps scrub starts 2024-09-10T19:23:40.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:23:40 smithi183 ceph-mon[41562]: purged_snaps scrub ok 2024-09-10T19:23:40.465 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:23:40 smithi191 ceph-mon[32884]: purged_snaps scrub starts 2024-09-10T19:23:40.465 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:23:40 smithi191 ceph-mon[32884]: purged_snaps scrub ok 2024-09-10T19:23:40.531 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:23:40.553 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:23:40.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:40.554 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:23:40.655 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:23:40.678 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:23:40.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:40.679 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:23:40.779 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:23:40.800 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:23:40.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:40.801 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:23:40.892 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:23:40.902 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:23:40.926 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:23:40.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:40.927 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:23:41.028 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:23:41.048 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:23:41.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:41.049 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:23:41.143 INFO:teuthology.orchestra.run.smithi183.stderr:nodeep-scrub is set 2024-09-10T19:23:41.150 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:23:41.174 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:23:41.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:41.175 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:23:41.275 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:23:41.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:41.301 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:23:41.301 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:23:41.402 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:23:41.423 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:23:41.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:41.424 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:23:41.431 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Sep 10 19:23:41 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-y[29989]: ::ffff:172.21.15.191 - - [10/Sep/2024:19:23:41] "GET /metrics HTTP/1.1" 200 224270 "" "Prometheus/2.43.0" 2024-09-10T19:23:41.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:23:41 smithi183 ceph-mon[29736]: from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:23:41.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:23:41 smithi183 ceph-mon[29736]: pgmap v601: 158 pgs: 53 active+undersized, 25 active+undersized+degraded, 80 active+clean; 635 KiB data, 8.7 GiB used, 528 GiB / 536 GiB avail; 1023 B/s rd, 0 op/s; 92/591 objects degraded (15.567%) 2024-09-10T19:23:41.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:23:41 smithi183 ceph-mon[29736]: osdmap e284: 8 total, 7 up, 6 in 2024-09-10T19:23:41.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:23:41 smithi183 ceph-mon[29736]: from='client.? 172.21.15.183:0/1141962252' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-10T19:23:41.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:23:41 smithi183 ceph-mon[29736]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-10T19:23:41.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:23:41 smithi183 ceph-mon[41562]: from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:23:41.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:23:41 smithi183 ceph-mon[41562]: pgmap v601: 158 pgs: 53 active+undersized, 25 active+undersized+degraded, 80 active+clean; 635 KiB data, 8.7 GiB used, 528 GiB / 536 GiB avail; 1023 B/s rd, 0 op/s; 92/591 objects degraded (15.567%) 2024-09-10T19:23:41.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:23:41 smithi183 ceph-mon[41562]: osdmap e284: 8 total, 7 up, 6 in 2024-09-10T19:23:41.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:23:41 smithi183 ceph-mon[41562]: from='client.? 172.21.15.183:0/1141962252' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-10T19:23:41.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:23:41 smithi183 ceph-mon[41562]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-10T19:23:41.465 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:23:41 smithi191 ceph-mon[32884]: from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:23:41.465 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:23:41 smithi191 ceph-mon[32884]: pgmap v601: 158 pgs: 53 active+undersized, 25 active+undersized+degraded, 80 active+clean; 635 KiB data, 8.7 GiB used, 528 GiB / 536 GiB avail; 1023 B/s rd, 0 op/s; 92/591 objects degraded (15.567%) 2024-09-10T19:23:41.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:23:41 smithi191 ceph-mon[32884]: osdmap e284: 8 total, 7 up, 6 in 2024-09-10T19:23:41.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:23:41 smithi191 ceph-mon[32884]: from='client.? 172.21.15.183:0/1141962252' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-10T19:23:41.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:23:41 smithi191 ceph-mon[32884]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-10T19:23:41.525 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:23:41.546 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:23:41.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:41.547 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:23:41.648 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:23:41.670 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:23:41.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:41.671 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:23:41.685 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph daemon osd.3 dump_ops_in_flight 2024-09-10T19:23:41.772 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:23:41.798 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:23:41.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:41.799 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:23:41.900 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:23:41.922 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:23:41.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:41.923 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:23:42.024 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:23:42.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:42.046 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:23:42.046 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:23:42.147 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:23:42.170 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:23:42.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:42.171 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:23:42.272 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:23:42.298 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:23:42.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:42.299 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:23:42.344 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:23:42 smithi183 ceph-mon[29736]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T19:23:42.344 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:23:42 smithi183 ceph-mon[29736]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-10T19:23:42.344 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:23:42 smithi183 ceph-mon[29736]: osdmap e285: 8 total, 7 up, 6 in 2024-09-10T19:23:42.345 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:23:42 smithi183 ceph-mon[41562]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T19:23:42.345 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:23:42 smithi183 ceph-mon[41562]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-10T19:23:42.345 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:23:42 smithi183 ceph-mon[41562]: osdmap e285: 8 total, 7 up, 6 in 2024-09-10T19:23:42.399 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:23:42.420 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:23:42.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:42.421 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:23:42.434 INFO:teuthology.orchestra.run.smithi183.stderr:Error ENXIO: problem getting command descriptions from osd.0 2024-09-10T19:23:42.441 INFO:teuthology.orchestra.run.smithi183.stdout:osd.1: {} 2024-09-10T19:23:42.442 INFO:teuthology.orchestra.run.smithi183.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-10T19:23:42.461 INFO:teuthology.orchestra.run.smithi183.stdout:osd.2: {} 2024-09-10T19:23:42.461 INFO:teuthology.orchestra.run.smithi183.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-10T19:23:42.465 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:23:42 smithi191 ceph-mon[32884]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T19:23:42.465 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:23:42 smithi191 ceph-mon[32884]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-10T19:23:42.465 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:23:42 smithi191 ceph-mon[32884]: osdmap e285: 8 total, 7 up, 6 in 2024-09-10T19:23:42.475 INFO:teuthology.orchestra.run.smithi183.stdout:osd.3: {} 2024-09-10T19:23:42.475 INFO:teuthology.orchestra.run.smithi183.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-10T19:23:42.491 INFO:teuthology.orchestra.run.smithi183.stdout:osd.4: {} 2024-09-10T19:23:42.492 INFO:teuthology.orchestra.run.smithi183.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-10T19:23:42.507 INFO:teuthology.orchestra.run.smithi183.stdout:osd.5: {} 2024-09-10T19:23:42.507 INFO:teuthology.orchestra.run.smithi183.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-10T19:23:42.523 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:23:42.523 INFO:teuthology.orchestra.run.smithi183.stdout:osd.6: {} 2024-09-10T19:23:42.524 INFO:teuthology.orchestra.run.smithi183.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-10T19:23:42.539 INFO:teuthology.orchestra.run.smithi183.stdout:osd.7: {} 2024-09-10T19:23:42.539 INFO:teuthology.orchestra.run.smithi183.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-10T19:23:42.544 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:23:42.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:42.545 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:23:42.646 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:23:42.667 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:23:42.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:42.669 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:23:42.769 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:23:42.791 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:23:42.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:42.792 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:23:42.892 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:23:42.913 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:23:42.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:42.914 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:23:43.014 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:23:43.035 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:23:43.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:43.036 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:23:43.137 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:23:43.163 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:23:43.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:43.164 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:23:43.265 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:23:43.290 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:23:43.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:43.291 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:23:43.392 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:23:43.414 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:23:43.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:43.415 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:23:43.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:23:43 smithi183 ceph-mon[29736]: pgmap v604: 158 pgs: 53 active+undersized, 25 active+undersized+degraded, 80 active+clean; 635 KiB data, 8.7 GiB used, 528 GiB / 536 GiB avail; 1.2 KiB/s rd, 1 op/s; 92/591 objects degraded (15.567%) 2024-09-10T19:23:43.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:23:43 smithi183 ceph-mon[29736]: from='client.? 172.21.15.183:0/2322819284' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T19:23:43.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:23:43 smithi183 ceph-mon[41562]: pgmap v604: 158 pgs: 53 active+undersized, 25 active+undersized+degraded, 80 active+clean; 635 KiB data, 8.7 GiB used, 528 GiB / 536 GiB avail; 1.2 KiB/s rd, 1 op/s; 92/591 objects degraded (15.567%) 2024-09-10T19:23:43.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:23:43 smithi183 ceph-mon[41562]: from='client.? 172.21.15.183:0/2322819284' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T19:23:43.465 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:23:43 smithi191 ceph-mon[32884]: pgmap v604: 158 pgs: 53 active+undersized, 25 active+undersized+degraded, 80 active+clean; 635 KiB data, 8.7 GiB used, 528 GiB / 536 GiB avail; 1.2 KiB/s rd, 1 op/s; 92/591 objects degraded (15.567%) 2024-09-10T19:23:43.465 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:23:43 smithi191 ceph-mon[32884]: from='client.? 172.21.15.183:0/2322819284' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T19:23:43.516 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:23:43.538 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:23:43.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:43.539 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:23:43.639 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:23:43.648 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-10T19:23:43.648 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-10T19:23:43.661 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:23:43.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:43.662 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:23:43.763 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:23:43.784 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:23:43.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:43.785 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:23:43.886 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:23:43.912 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:23:43.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:43.913 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:23:44.014 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:23:44.039 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:23:44.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:44.041 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:23:44.070 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:23:44.142 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:23:44.163 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:23:44.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:44.164 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:23:44.265 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:23:44.290 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:23:44.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:44.291 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:23:44.391 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:23:44.417 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:23:44.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:44.418 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:23:44.519 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:23:44.544 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:23:44.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:44.545 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:23:44.646 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:23:44.671 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:23:44.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:44.672 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:23:44.773 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:23:44.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:44.798 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:23:44.799 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:23:44.879 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph osd unset noscrub 2024-09-10T19:23:44.889 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:23:44 smithi183 ceph-mon[29736]: from='mgr.14556 ' entity='mgr.y' 2024-09-10T19:23:44.889 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:23:44 smithi183 ceph-mon[29736]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:23:44.889 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:23:44 smithi183 ceph-mon[29736]: pgmap v605: 158 pgs: 53 active+undersized, 25 active+undersized+degraded, 80 active+clean; 635 KiB data, 8.7 GiB used, 528 GiB / 536 GiB avail; 639 B/s rd, 0 op/s; 92/591 objects degraded (15.567%) 2024-09-10T19:23:44.889 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:23:44 smithi183 ceph-mon[41562]: from='mgr.14556 ' entity='mgr.y' 2024-09-10T19:23:44.889 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:23:44 smithi183 ceph-mon[41562]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:23:44.889 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:23:44 smithi183 ceph-mon[41562]: pgmap v605: 158 pgs: 53 active+undersized, 25 active+undersized+degraded, 80 active+clean; 635 KiB data, 8.7 GiB used, 528 GiB / 536 GiB avail; 639 B/s rd, 0 op/s; 92/591 objects degraded (15.567%) 2024-09-10T19:23:44.900 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:23:44.924 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:23:44.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:44.925 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:23:44.966 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:23:44 smithi191 ceph-mon[32884]: from='mgr.14556 ' entity='mgr.y' 2024-09-10T19:23:44.966 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:23:44 smithi191 ceph-mon[32884]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:23:44.966 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:23:44 smithi191 ceph-mon[32884]: pgmap v605: 158 pgs: 53 active+undersized, 25 active+undersized+degraded, 80 active+clean; 635 KiB data, 8.7 GiB used, 528 GiB / 536 GiB avail; 639 B/s rd, 0 op/s; 92/591 objects degraded (15.567%) 2024-09-10T19:23:45.025 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:23:45.050 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:23:45.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:45.051 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:23:45.151 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:23:45.174 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:23:45.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:45.175 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:23:45.275 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:23:45.297 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:23:45.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:45.298 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:23:45.399 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:23:45.421 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:23:45.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:45.422 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:23:45.523 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:23:45.545 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:23:45.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:45.546 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:23:45.647 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:23:45.649 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-10T19:23:45.669 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:23:45.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:45.670 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:23:45.771 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:23:45.793 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:23:45.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:45.797 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:23:45.897 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:23:45.919 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:23:45.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:45.920 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:23:46.020 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:23:46.042 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:23:46.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:46.043 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:23:46.144 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:23:46.166 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:23:46.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:46.167 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:23:46.268 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:23:46.291 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:23:46.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:46.292 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:23:46.344 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:23:46.393 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:23:46.415 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:23:46.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:46.416 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:23:46.517 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:23:46.539 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:23:46.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:46.540 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:23:46.641 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:23:46.662 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:23:46.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:46.663 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:23:46.764 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:23:46.786 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:23:46.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:46.787 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:23:46.845 INFO:teuthology.orchestra.run.smithi183.stdout:{ 2024-09-10T19:23:46.846 INFO:teuthology.orchestra.run.smithi183.stdout: "ops": [], 2024-09-10T19:23:46.846 INFO:teuthology.orchestra.run.smithi183.stdout: "num_ops": 0 2024-09-10T19:23:46.846 INFO:teuthology.orchestra.run.smithi183.stdout:} 2024-09-10T19:23:46.876 INFO:teuthology.orchestra.run.smithi183.stderr:set pool 9 pgp_num to 16 2024-09-10T19:23:46.888 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:23:46.910 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:23:46.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:46.911 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:23:47.012 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:23:47.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:47.036 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:23:47.037 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:23:47.138 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:23:47.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:47.162 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:23:47.162 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:23:47.181 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:23:46 smithi183 ceph-mon[41562]: purged_snaps scrub starts 2024-09-10T19:23:47.181 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:23:46 smithi183 ceph-mon[41562]: purged_snaps scrub ok 2024-09-10T19:23:47.181 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:23:46 smithi183 ceph-mon[41562]: pgmap v606: 158 pgs: 53 active+undersized, 25 active+undersized+degraded, 80 active+clean; 635 KiB data, 8.7 GiB used, 528 GiB / 536 GiB avail; 1.2 KiB/s rd, 1 op/s; 92/591 objects degraded (15.567%) 2024-09-10T19:23:47.181 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:23:46 smithi183 ceph-mon[41562]: osdmap e286: 8 total, 7 up, 6 in 2024-09-10T19:23:47.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:23:46 smithi183 ceph-mon[41562]: from='client.? 172.21.15.183:0/2056705340' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_2", "var": "pgp_num", "val": "16"}]: dispatch 2024-09-10T19:23:47.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:23:46 smithi183 ceph-mon[41562]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_2", "var": "pgp_num", "val": "16"}]: dispatch 2024-09-10T19:23:47.182 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:23:46 smithi183 ceph-mon[29736]: purged_snaps scrub starts 2024-09-10T19:23:47.182 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:23:46 smithi183 ceph-mon[29736]: purged_snaps scrub ok 2024-09-10T19:23:47.182 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:23:46 smithi183 ceph-mon[29736]: pgmap v606: 158 pgs: 53 active+undersized, 25 active+undersized+degraded, 80 active+clean; 635 KiB data, 8.7 GiB used, 528 GiB / 536 GiB avail; 1.2 KiB/s rd, 1 op/s; 92/591 objects degraded (15.567%) 2024-09-10T19:23:47.182 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:23:46 smithi183 ceph-mon[29736]: osdmap e286: 8 total, 7 up, 6 in 2024-09-10T19:23:47.182 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:23:46 smithi183 ceph-mon[29736]: from='client.? 172.21.15.183:0/2056705340' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_2", "var": "pgp_num", "val": "16"}]: dispatch 2024-09-10T19:23:47.182 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:23:46 smithi183 ceph-mon[29736]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_2", "var": "pgp_num", "val": "16"}]: dispatch 2024-09-10T19:23:47.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:23:46 smithi191 ceph-mon[32884]: purged_snaps scrub starts 2024-09-10T19:23:47.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:23:46 smithi191 ceph-mon[32884]: purged_snaps scrub ok 2024-09-10T19:23:47.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:23:46 smithi191 ceph-mon[32884]: pgmap v606: 158 pgs: 53 active+undersized, 25 active+undersized+degraded, 80 active+clean; 635 KiB data, 8.7 GiB used, 528 GiB / 536 GiB avail; 1.2 KiB/s rd, 1 op/s; 92/591 objects degraded (15.567%) 2024-09-10T19:23:47.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:23:46 smithi191 ceph-mon[32884]: osdmap e286: 8 total, 7 up, 6 in 2024-09-10T19:23:47.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:23:46 smithi191 ceph-mon[32884]: from='client.? 172.21.15.183:0/2056705340' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_2", "var": "pgp_num", "val": "16"}]: dispatch 2024-09-10T19:23:47.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:23:46 smithi191 ceph-mon[32884]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_2", "var": "pgp_num", "val": "16"}]: dispatch 2024-09-10T19:23:47.251 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:23:47.263 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:23:47.286 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:23:47.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:47.287 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:23:47.387 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:23:47.410 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:23:47.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:47.411 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:23:47.511 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:23:47.534 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:23:47.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:47.535 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:23:47.635 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:23:47.659 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:23:47.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:47.660 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:23:47.715 INFO:journalctl@ceph.mgr.x.smithi191.stdout:Sep 10 19:23:47 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-x[34967]: ::ffff:172.21.15.191 - - [10/Sep/2024:19:23:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T19:23:47.761 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:23:47.782 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:23:47.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:47.783 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:23:47.884 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:23:47.907 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:23:47.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:47.908 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:23:48.009 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:23:48.030 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:23:48.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:48.031 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:23:48.132 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:23:48.153 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:23:48.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:48.155 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:23:48.181 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:23:47 smithi183 ceph-mon[41562]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_2", "var": "pgp_num", "val": "16"}]': finished 2024-09-10T19:23:48.181 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:23:47 smithi183 ceph-mon[41562]: osdmap e287: 8 total, 7 up, 6 in 2024-09-10T19:23:48.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:23:47 smithi183 ceph-mon[29736]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_2", "var": "pgp_num", "val": "16"}]': finished 2024-09-10T19:23:48.182 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:23:47 smithi183 ceph-mon[29736]: osdmap e287: 8 total, 7 up, 6 in 2024-09-10T19:23:48.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:23:47 smithi191 ceph-mon[32884]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_2", "var": "pgp_num", "val": "16"}]': finished 2024-09-10T19:23:48.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:23:47 smithi191 ceph-mon[32884]: osdmap e287: 8 total, 7 up, 6 in 2024-09-10T19:23:48.256 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:23:48.278 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:23:48.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:48.279 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:23:48.380 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:23:48.404 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:23:48.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:48.405 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:23:48.506 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:23:48.529 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:23:48.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:48.530 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:23:48.631 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:23:48.653 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:23:48.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:48.654 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:23:48.755 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:23:48.766 INFO:tasks.cephadm.ceph_manager.ceph:removing pool_name unique_pool_2 2024-09-10T19:23:48.766 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph osd pool rm unique_pool_2 unique_pool_2 --yes-i-really-really-mean-it 2024-09-10T19:23:48.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:48.777 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:23:48.778 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:23:48.878 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:23:48.904 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:23:48.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:48.905 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:23:49.006 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:23:49.017 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:23:48 smithi183 ceph-mon[29736]: pgmap v609: 158 pgs: 53 active+undersized, 25 active+undersized+degraded, 80 active+clean; 635 KiB data, 8.7 GiB used, 528 GiB / 536 GiB avail; 763 B/s rd, 0 op/s; 92/591 objects degraded (15.567%) 2024-09-10T19:23:49.017 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:23:48 smithi183 ceph-mon[41562]: pgmap v609: 158 pgs: 53 active+undersized, 25 active+undersized+degraded, 80 active+clean; 635 KiB data, 8.7 GiB used, 528 GiB / 536 GiB avail; 763 B/s rd, 0 op/s; 92/591 objects degraded (15.567%) 2024-09-10T19:23:49.028 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:23:49.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:49.029 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:23:49.130 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:23:49.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:49.157 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:23:49.157 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:23:49.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:23:48 smithi191 ceph-mon[32884]: pgmap v609: 158 pgs: 53 active+undersized, 25 active+undersized+degraded, 80 active+clean; 635 KiB data, 8.7 GiB used, 528 GiB / 536 GiB avail; 763 B/s rd, 0 op/s; 92/591 objects degraded (15.567%) 2024-09-10T19:23:49.258 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:23:49.284 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:23:49.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:49.285 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:23:49.385 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:23:49.410 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:23:49.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:49.411 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:23:49.512 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:23:49.534 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:23:49.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:49.535 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:23:49.636 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:23:49.654 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph daemon osd.3 dump_blocked_ops 2024-09-10T19:23:49.662 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:23:49.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:49.663 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:23:49.763 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:23:49.784 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:23:49.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:49.785 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:23:49.886 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:23:49.911 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:23:49.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:49.912 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:23:49.966 INFO:journalctl@ceph.iscsi.iscsi.a.smithi191.stdout:Sep 10 19:23:49 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-iscsi-iscsi-a[61816]: debug there is no tcmu-runner data available 2024-09-10T19:23:50.013 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:23:50.035 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:23:50.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:50.036 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:23:50.137 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:23:50.159 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:23:50.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:50.160 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:23:50.261 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:23:50.282 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:23:50.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:50.283 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:23:50.384 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:23:50.405 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:23:50.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:50.406 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:23:50.507 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:23:50.528 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:23:50.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:50.529 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:23:50.630 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:23:50.651 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:23:50.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:50.652 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:23:50.753 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:23:50.779 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:23:50.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:50.780 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:23:50.880 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:23:50.908 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:23:50.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:50.909 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:23:51.010 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:23:51.034 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:23:51.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:51.035 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:23:51.070 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:23:50 smithi183 ceph-mon[29736]: purged_snaps scrub starts 2024-09-10T19:23:51.070 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:23:50 smithi183 ceph-mon[29736]: purged_snaps scrub ok 2024-09-10T19:23:51.070 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:23:50 smithi183 ceph-mon[29736]: from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:23:51.070 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:23:50 smithi183 ceph-mon[29736]: pgmap v610: 158 pgs: 53 active+undersized, 25 active+undersized+degraded, 80 active+clean; 635 KiB data, 8.7 GiB used, 528 GiB / 536 GiB avail; 639 B/s rd, 0 op/s; 92/591 objects degraded (15.567%) 2024-09-10T19:23:51.070 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:23:50 smithi183 ceph-mon[29736]: osdmap e288: 8 total, 7 up, 6 in 2024-09-10T19:23:51.070 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:23:50 smithi183 ceph-mon[41562]: purged_snaps scrub starts 2024-09-10T19:23:51.070 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:23:50 smithi183 ceph-mon[41562]: purged_snaps scrub ok 2024-09-10T19:23:51.071 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:23:50 smithi183 ceph-mon[41562]: from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:23:51.071 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:23:50 smithi183 ceph-mon[41562]: pgmap v610: 158 pgs: 53 active+undersized, 25 active+undersized+degraded, 80 active+clean; 635 KiB data, 8.7 GiB used, 528 GiB / 536 GiB avail; 639 B/s rd, 0 op/s; 92/591 objects degraded (15.567%) 2024-09-10T19:23:51.071 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:23:50 smithi183 ceph-mon[41562]: osdmap e288: 8 total, 7 up, 6 in 2024-09-10T19:23:51.135 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:23:51.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:51.159 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:23:51.159 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:23:51.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:23:50 smithi191 ceph-mon[32884]: purged_snaps scrub starts 2024-09-10T19:23:51.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:23:50 smithi191 ceph-mon[32884]: purged_snaps scrub ok 2024-09-10T19:23:51.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:23:50 smithi191 ceph-mon[32884]: from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:23:51.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:23:50 smithi191 ceph-mon[32884]: pgmap v610: 158 pgs: 53 active+undersized, 25 active+undersized+degraded, 80 active+clean; 635 KiB data, 8.7 GiB used, 528 GiB / 536 GiB avail; 639 B/s rd, 0 op/s; 92/591 objects degraded (15.567%) 2024-09-10T19:23:51.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:23:50 smithi191 ceph-mon[32884]: osdmap e288: 8 total, 7 up, 6 in 2024-09-10T19:23:51.260 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:23:51.283 INFO:teuthology.orchestra.run.smithi183.stderr:Error ENXIO: problem getting command descriptions from osd.0 2024-09-10T19:23:51.284 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:23:51.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:51.285 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:23:51.299 INFO:teuthology.orchestra.run.smithi183.stdout:osd.1: {} 2024-09-10T19:23:51.299 INFO:teuthology.orchestra.run.smithi183.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-10T19:23:51.317 INFO:teuthology.orchestra.run.smithi183.stdout:osd.2: {} 2024-09-10T19:23:51.317 INFO:teuthology.orchestra.run.smithi183.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-10T19:23:51.332 INFO:teuthology.orchestra.run.smithi183.stdout:osd.3: {} 2024-09-10T19:23:51.333 INFO:teuthology.orchestra.run.smithi183.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-10T19:23:51.333 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Sep 10 19:23:51 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-y[29989]: ::ffff:172.21.15.191 - - [10/Sep/2024:19:23:51] "GET /metrics HTTP/1.1" 200 224213 "" "Prometheus/2.43.0" 2024-09-10T19:23:51.349 INFO:teuthology.orchestra.run.smithi183.stdout:osd.4: {} 2024-09-10T19:23:51.349 INFO:teuthology.orchestra.run.smithi183.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-10T19:23:51.364 INFO:teuthology.orchestra.run.smithi183.stdout:osd.5: {} 2024-09-10T19:23:51.364 INFO:teuthology.orchestra.run.smithi183.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-10T19:23:51.379 INFO:teuthology.orchestra.run.smithi183.stdout:osd.6: {} 2024-09-10T19:23:51.380 INFO:teuthology.orchestra.run.smithi183.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-10T19:23:51.386 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:23:51.396 INFO:teuthology.orchestra.run.smithi183.stdout:osd.7: {} 2024-09-10T19:23:51.396 INFO:teuthology.orchestra.run.smithi183.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-10T19:23:51.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:51.408 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:23:51.408 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:23:51.509 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:23:51.530 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:23:51.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:51.531 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:23:51.632 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:23:51.654 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:23:51.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:51.655 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:23:51.756 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:23:51.782 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:23:51.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:51.783 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:23:51.884 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:23:51.909 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:23:51.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:51.910 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:23:52.011 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:23:52.032 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:23:52.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:52.033 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:23:52.133 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:23:52.154 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:23:52.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:52.155 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:23:52.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:23:51 smithi183 ceph-mon[29736]: from='client.? 172.21.15.183:0/1133035302' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T19:23:52.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:23:51 smithi183 ceph-mon[41562]: from='client.? 172.21.15.183:0/1133035302' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T19:23:52.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:23:51 smithi191 ceph-mon[32884]: from='client.? 172.21.15.183:0/1133035302' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T19:23:52.256 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:23:52.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:52.283 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:23:52.284 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:23:52.384 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:23:52.407 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:23:52.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:52.408 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:23:52.413 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-10T19:23:52.413 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-10T19:23:52.509 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:23:52.535 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:23:52.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:52.536 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:23:52.636 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:23:52.637 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:23:52.662 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:23:52.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:52.663 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:23:52.669 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:23:52.764 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:23:52.785 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:23:52.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:52.786 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:23:52.887 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:23:52.910 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:23:52.911 INFO:teuthology.orchestra.run.smithi183.stderr:noscrub is unset 2024-09-10T19:23:52.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:52.912 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:23:53.013 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:23:53.019 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:23:52 smithi183 ceph-mon[29736]: pgmap v612: 158 pgs: 53 active+undersized, 25 active+undersized+degraded, 80 active+clean; 635 KiB data, 8.7 GiB used, 528 GiB / 536 GiB avail; 852 B/s rd, 0 op/s; 92/591 objects degraded (15.567%) 2024-09-10T19:23:53.019 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:23:52 smithi183 ceph-mon[29736]: from='client.? 172.21.15.183:0/2347125107' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-10T19:23:53.019 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:23:52 smithi183 ceph-mon[41562]: pgmap v612: 158 pgs: 53 active+undersized, 25 active+undersized+degraded, 80 active+clean; 635 KiB data, 8.7 GiB used, 528 GiB / 536 GiB avail; 852 B/s rd, 0 op/s; 92/591 objects degraded (15.567%) 2024-09-10T19:23:53.019 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:23:52 smithi183 ceph-mon[41562]: from='client.? 172.21.15.183:0/2347125107' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-10T19:23:53.040 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:23:53.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:53.041 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:23:53.141 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:23:53.167 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:23:53.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:53.168 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:23:53.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:23:52 smithi191 ceph-mon[32884]: pgmap v612: 158 pgs: 53 active+undersized, 25 active+undersized+degraded, 80 active+clean; 635 KiB data, 8.7 GiB used, 528 GiB / 536 GiB avail; 852 B/s rd, 0 op/s; 92/591 objects degraded (15.567%) 2024-09-10T19:23:53.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:23:52 smithi191 ceph-mon[32884]: from='client.? 172.21.15.183:0/2347125107' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-10T19:23:53.269 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:23:53.294 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:23:53.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:53.295 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:23:53.396 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:23:53.418 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:23:53.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:53.419 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:23:53.520 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:23:53.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:53.542 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:23:53.542 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:23:53.644 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:23:53.670 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:23:53.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:53.671 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:23:53.766 INFO:tasks.thrashosds.thrasher:in_osds: [0, 3, 4, 5, 6, 7] out_osds: [1, 2] dead_osds: [0] live_osds: [2, 3, 4, 5, 6, 7, 1] 2024-09-10T19:23:53.767 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-10T19:23:53.767 INFO:tasks.thrashosds.thrasher:Adding osd 1 2024-09-10T19:23:53.767 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph osd in 1 2024-09-10T19:23:53.771 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:23:53.797 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:23:53.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:53.798 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:23:53.899 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:23:53.923 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:23:53.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:53.925 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:23:54.025 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:23:54.049 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:23:54.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:54.050 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:23:54.151 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:23:54.174 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:23:54.174 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:23:53 smithi183 ceph-mon[29736]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T19:23:54.174 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:23:53 smithi183 ceph-mon[29736]: from='client.? 172.21.15.183:0/2347125107' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-10T19:23:54.175 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:23:53 smithi183 ceph-mon[29736]: osdmap e289: 8 total, 7 up, 6 in 2024-09-10T19:23:54.175 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:23:53 smithi183 ceph-mon[29736]: purged_snaps scrub starts 2024-09-10T19:23:54.175 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:23:53 smithi183 ceph-mon[29736]: purged_snaps scrub ok 2024-09-10T19:23:54.175 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:23:53 smithi183 ceph-mon[41562]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T19:23:54.176 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:23:53 smithi183 ceph-mon[41562]: from='client.? 172.21.15.183:0/2347125107' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-10T19:23:54.176 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:23:53 smithi183 ceph-mon[41562]: osdmap e289: 8 total, 7 up, 6 in 2024-09-10T19:23:54.176 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:23:53 smithi183 ceph-mon[41562]: purged_snaps scrub starts 2024-09-10T19:23:54.176 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:23:53 smithi183 ceph-mon[41562]: purged_snaps scrub ok 2024-09-10T19:23:54.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:54.178 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:23:54.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:23:53 smithi191 ceph-mon[32884]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T19:23:54.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:23:53 smithi191 ceph-mon[32884]: from='client.? 172.21.15.183:0/2347125107' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-10T19:23:54.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:23:53 smithi191 ceph-mon[32884]: osdmap e289: 8 total, 7 up, 6 in 2024-09-10T19:23:54.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:23:53 smithi191 ceph-mon[32884]: purged_snaps scrub starts 2024-09-10T19:23:54.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:23:53 smithi191 ceph-mon[32884]: purged_snaps scrub ok 2024-09-10T19:23:54.278 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:23:54.301 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:23:54.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:54.302 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:23:54.403 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:23:54.414 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-10T19:23:54.424 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:23:54.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:54.425 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:23:54.527 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:23:54.553 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:23:54.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:54.554 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:23:54.655 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:23:54.681 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:23:54.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:54.682 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:23:54.783 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:23:54.804 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:23:54.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:54.805 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:23:54.906 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:23:54.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:54.928 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:23:54.928 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:23:54.987 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:23:54 smithi183 ceph-mon[29736]: 6.e scrub starts 2024-09-10T19:23:54.988 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:23:54 smithi183 ceph-mon[29736]: 6.e scrub ok 2024-09-10T19:23:54.988 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:23:54 smithi183 ceph-mon[29736]: pgmap v614: 158 pgs: 53 active+undersized, 25 active+undersized+degraded, 80 active+clean; 635 KiB data, 8.7 GiB used, 528 GiB / 536 GiB avail; 733 B/s rd, 0 op/s; 92/591 objects degraded (15.567%) 2024-09-10T19:23:54.988 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:23:54 smithi183 ceph-mon[29736]: osdmap e290: 8 total, 7 up, 6 in 2024-09-10T19:23:54.988 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:23:54 smithi183 ceph-mon[29736]: 5.10 scrub starts 2024-09-10T19:23:54.988 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:23:54 smithi183 ceph-mon[29736]: 5.10 scrub ok 2024-09-10T19:23:54.988 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:23:54 smithi183 ceph-mon[41562]: 6.e scrub starts 2024-09-10T19:23:54.988 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:23:54 smithi183 ceph-mon[41562]: 6.e scrub ok 2024-09-10T19:23:54.988 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:23:54 smithi183 ceph-mon[41562]: pgmap v614: 158 pgs: 53 active+undersized, 25 active+undersized+degraded, 80 active+clean; 635 KiB data, 8.7 GiB used, 528 GiB / 536 GiB avail; 733 B/s rd, 0 op/s; 92/591 objects degraded (15.567%) 2024-09-10T19:23:54.988 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:23:54 smithi183 ceph-mon[41562]: osdmap e290: 8 total, 7 up, 6 in 2024-09-10T19:23:54.988 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:23:54 smithi183 ceph-mon[41562]: 5.10 scrub starts 2024-09-10T19:23:54.989 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:23:54 smithi183 ceph-mon[41562]: 5.10 scrub ok 2024-09-10T19:23:55.029 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:23:55.055 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:23:55.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:55.057 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:23:55.157 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:23:55.183 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:23:55.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:55.184 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:23:55.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:23:54 smithi191 ceph-mon[32884]: 6.e scrub starts 2024-09-10T19:23:55.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:23:54 smithi191 ceph-mon[32884]: 6.e scrub ok 2024-09-10T19:23:55.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:23:54 smithi191 ceph-mon[32884]: pgmap v614: 158 pgs: 53 active+undersized, 25 active+undersized+degraded, 80 active+clean; 635 KiB data, 8.7 GiB used, 528 GiB / 536 GiB avail; 733 B/s rd, 0 op/s; 92/591 objects degraded (15.567%) 2024-09-10T19:23:55.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:23:54 smithi191 ceph-mon[32884]: osdmap e290: 8 total, 7 up, 6 in 2024-09-10T19:23:55.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:23:54 smithi191 ceph-mon[32884]: 5.10 scrub starts 2024-09-10T19:23:55.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:23:54 smithi191 ceph-mon[32884]: 5.10 scrub ok 2024-09-10T19:23:55.244 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:23:55.279 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:23:55.285 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:23:55.306 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:23:55.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:55.308 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:23:55.409 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:23:55.435 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:23:55.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:55.436 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:23:55.537 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:23:55.558 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:23:55.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:55.559 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:23:55.603 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-10T19:23:55.661 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:23:55.687 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:23:55.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:55.688 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:23:55.789 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:23:55.814 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:23:55.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:55.816 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:23:55.916 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:23:55.938 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:23:55.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:55.939 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:23:56.040 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:23:56.065 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:23:56.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:56.066 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:23:56.166 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:23:56.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:23:55 smithi183 ceph-mon[29736]: 5.1c scrub starts 2024-09-10T19:23:56.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:23:55 smithi183 ceph-mon[29736]: 5.1c scrub ok 2024-09-10T19:23:56.181 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:23:55 smithi183 ceph-mon[41562]: 5.1c scrub starts 2024-09-10T19:23:56.181 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:23:55 smithi183 ceph-mon[41562]: 5.1c scrub ok 2024-09-10T19:23:56.188 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:23:56.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:56.188 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:23:56.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:23:55 smithi191 ceph-mon[32884]: 5.1c scrub starts 2024-09-10T19:23:56.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:23:55 smithi191 ceph-mon[32884]: 5.1c scrub ok 2024-09-10T19:23:56.289 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:23:56.313 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:23:56.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:56.314 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:23:56.415 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:23:56.439 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:23:56.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:56.440 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:23:56.522 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:23:56.541 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:23:56.564 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:23:56.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:56.565 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:23:56.666 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:23:56.689 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:23:56.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:56.691 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:23:56.791 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:23:56.815 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:23:56.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:56.816 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:23:56.916 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:23:56.939 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:23:56.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:56.940 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:23:57.041 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:23:57.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:57.067 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:23:57.068 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:23:57.169 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:23:57.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:23:56 smithi183 ceph-mon[29736]: 6.2 scrub starts 2024-09-10T19:23:57.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:23:56 smithi183 ceph-mon[29736]: 6.2 scrub ok 2024-09-10T19:23:57.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:23:56 smithi183 ceph-mon[29736]: 6.6 scrub starts 2024-09-10T19:23:57.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:23:56 smithi183 ceph-mon[29736]: 6.6 scrub ok 2024-09-10T19:23:57.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:23:56 smithi183 ceph-mon[29736]: pgmap v616: 158 pgs: 53 active+undersized, 25 active+undersized+degraded, 80 active+clean; 635 KiB data, 8.7 GiB used, 528 GiB / 536 GiB avail; 1.7 KiB/s rd, 1 op/s; 92/591 objects degraded (15.567%) 2024-09-10T19:23:57.182 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:23:56 smithi183 ceph-mon[29736]: 2.14 scrub starts 2024-09-10T19:23:57.182 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:23:56 smithi183 ceph-mon[29736]: 2.14 scrub ok 2024-09-10T19:23:57.182 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:23:56 smithi183 ceph-mon[29736]: 2.b scrub starts 2024-09-10T19:23:57.182 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:23:56 smithi183 ceph-mon[29736]: 2.b scrub ok 2024-09-10T19:23:57.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:23:56 smithi183 ceph-mon[41562]: 6.2 scrub starts 2024-09-10T19:23:57.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:23:56 smithi183 ceph-mon[41562]: 6.2 scrub ok 2024-09-10T19:23:57.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:23:56 smithi183 ceph-mon[41562]: 6.6 scrub starts 2024-09-10T19:23:57.183 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:23:56 smithi183 ceph-mon[41562]: 6.6 scrub ok 2024-09-10T19:23:57.183 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:23:56 smithi183 ceph-mon[41562]: pgmap v616: 158 pgs: 53 active+undersized, 25 active+undersized+degraded, 80 active+clean; 635 KiB data, 8.7 GiB used, 528 GiB / 536 GiB avail; 1.7 KiB/s rd, 1 op/s; 92/591 objects degraded (15.567%) 2024-09-10T19:23:57.183 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:23:56 smithi183 ceph-mon[41562]: 2.14 scrub starts 2024-09-10T19:23:57.183 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:23:56 smithi183 ceph-mon[41562]: 2.14 scrub ok 2024-09-10T19:23:57.183 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:23:56 smithi183 ceph-mon[41562]: 2.b scrub starts 2024-09-10T19:23:57.183 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:23:56 smithi183 ceph-mon[41562]: 2.b scrub ok 2024-09-10T19:23:57.192 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:23:56 smithi191 ceph-mon[32884]: 6.2 scrub starts 2024-09-10T19:23:57.192 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:23:56 smithi191 ceph-mon[32884]: 6.2 scrub ok 2024-09-10T19:23:57.192 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:23:56 smithi191 ceph-mon[32884]: 6.6 scrub starts 2024-09-10T19:23:57.192 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:23:56 smithi191 ceph-mon[32884]: 6.6 scrub ok 2024-09-10T19:23:57.192 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:23:56 smithi191 ceph-mon[32884]: pgmap v616: 158 pgs: 53 active+undersized, 25 active+undersized+degraded, 80 active+clean; 635 KiB data, 8.7 GiB used, 528 GiB / 536 GiB avail; 1.7 KiB/s rd, 1 op/s; 92/591 objects degraded (15.567%) 2024-09-10T19:23:57.192 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:23:56 smithi191 ceph-mon[32884]: 2.14 scrub starts 2024-09-10T19:23:57.192 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:23:56 smithi191 ceph-mon[32884]: 2.14 scrub ok 2024-09-10T19:23:57.192 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:23:56 smithi191 ceph-mon[32884]: 2.b scrub starts 2024-09-10T19:23:57.192 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:23:56 smithi191 ceph-mon[32884]: 2.b scrub ok 2024-09-10T19:23:57.194 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:23:57.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:57.195 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:23:57.296 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:23:57.318 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:23:57.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:57.319 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:23:57.420 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:23:57.442 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:23:57.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:57.443 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:23:57.544 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:23:57.566 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:23:57.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:57.567 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:23:57.668 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:23:57.693 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:23:57.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:57.694 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:23:57.715 INFO:journalctl@ceph.mgr.x.smithi191.stdout:Sep 10 19:23:57 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-x[34967]: ::ffff:172.21.15.191 - - [10/Sep/2024:19:23:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T19:23:57.795 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:23:57.817 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:23:57.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:57.818 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:23:57.919 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:23:57.942 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:23:57.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:57.943 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:23:58.043 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:23:58.065 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:23:58.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:58.066 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:23:58.166 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:23:58.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:23:57 smithi183 ceph-mon[29736]: 2.d scrub starts 2024-09-10T19:23:58.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:23:57 smithi183 ceph-mon[29736]: 2.d scrub ok 2024-09-10T19:23:58.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:23:57 smithi183 ceph-mon[29736]: 6.1b scrub starts 2024-09-10T19:23:58.182 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:23:57 smithi183 ceph-mon[29736]: 6.1b scrub ok 2024-09-10T19:23:58.182 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:23:57 smithi183 ceph-mon[29736]: 6.1c scrub starts 2024-09-10T19:23:58.182 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:23:57 smithi183 ceph-mon[29736]: 6.1c scrub ok 2024-09-10T19:23:58.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:23:57 smithi183 ceph-mon[41562]: 2.d scrub starts 2024-09-10T19:23:58.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:23:57 smithi183 ceph-mon[41562]: 2.d scrub ok 2024-09-10T19:23:58.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:23:57 smithi183 ceph-mon[41562]: 6.1b scrub starts 2024-09-10T19:23:58.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:23:57 smithi183 ceph-mon[41562]: 6.1b scrub ok 2024-09-10T19:23:58.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:23:57 smithi183 ceph-mon[41562]: 6.1c scrub starts 2024-09-10T19:23:58.183 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:23:57 smithi183 ceph-mon[41562]: 6.1c scrub ok 2024-09-10T19:23:58.187 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:23:58.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:58.188 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:23:58.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:23:57 smithi191 ceph-mon[32884]: 2.d scrub starts 2024-09-10T19:23:58.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:23:57 smithi191 ceph-mon[32884]: 2.d scrub ok 2024-09-10T19:23:58.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:23:57 smithi191 ceph-mon[32884]: 6.1b scrub starts 2024-09-10T19:23:58.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:23:57 smithi191 ceph-mon[32884]: 6.1b scrub ok 2024-09-10T19:23:58.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:23:57 smithi191 ceph-mon[32884]: 6.1c scrub starts 2024-09-10T19:23:58.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:23:57 smithi191 ceph-mon[32884]: 6.1c scrub ok 2024-09-10T19:23:58.289 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:23:58.311 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:23:58.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:58.312 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:23:58.413 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:23:58.434 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:23:58.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:58.435 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:23:58.536 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:23:58.557 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:23:58.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:58.558 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:23:58.659 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:23:58.681 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:23:58.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:58.682 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:23:58.782 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:23:58.805 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:23:58.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:58.806 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:23:58.906 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:23:58.928 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:23:58.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:58.929 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:23:58.945 INFO:teuthology.orchestra.run.smithi183.stderr:pool 'unique_pool_2' removed 2024-09-10T19:23:59.030 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:23:59.051 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:23:59.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:59.052 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:23:59.153 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:23:59.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:59.175 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:23:59.176 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:23:59.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:23:58 smithi183 ceph-mon[29736]: 4.1b scrub starts 2024-09-10T19:23:59.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:23:58 smithi183 ceph-mon[29736]: 4.1b scrub ok 2024-09-10T19:23:59.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:23:58 smithi183 ceph-mon[29736]: pgmap v617: 158 pgs: 53 active+undersized, 25 active+undersized+degraded, 80 active+clean; 635 KiB data, 8.7 GiB used, 528 GiB / 536 GiB avail; 1.3 KiB/s rd, 1 op/s; 92/591 objects degraded (15.567%) 2024-09-10T19:23:59.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:23:58 smithi183 ceph-mon[29736]: from='client.? 172.21.15.183:0/1015951867' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_2", "pool2": "unique_pool_2", "yes_i_really_really_mean_it": true}]: dispatch 2024-09-10T19:23:59.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:23:58 smithi183 ceph-mon[29736]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_2", "pool2": "unique_pool_2", "yes_i_really_really_mean_it": true}]: dispatch 2024-09-10T19:23:59.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:23:58 smithi183 ceph-mon[29736]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:23:59.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:23:58 smithi183 ceph-mon[41562]: 4.1b scrub starts 2024-09-10T19:23:59.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:23:58 smithi183 ceph-mon[41562]: 4.1b scrub ok 2024-09-10T19:23:59.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:23:58 smithi183 ceph-mon[41562]: pgmap v617: 158 pgs: 53 active+undersized, 25 active+undersized+degraded, 80 active+clean; 635 KiB data, 8.7 GiB used, 528 GiB / 536 GiB avail; 1.3 KiB/s rd, 1 op/s; 92/591 objects degraded (15.567%) 2024-09-10T19:23:59.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:23:58 smithi183 ceph-mon[41562]: from='client.? 172.21.15.183:0/1015951867' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_2", "pool2": "unique_pool_2", "yes_i_really_really_mean_it": true}]: dispatch 2024-09-10T19:23:59.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:23:58 smithi183 ceph-mon[41562]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_2", "pool2": "unique_pool_2", "yes_i_really_really_mean_it": true}]: dispatch 2024-09-10T19:23:59.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:23:58 smithi183 ceph-mon[41562]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:23:59.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:23:58 smithi191 ceph-mon[32884]: 4.1b scrub starts 2024-09-10T19:23:59.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:23:58 smithi191 ceph-mon[32884]: 4.1b scrub ok 2024-09-10T19:23:59.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:23:58 smithi191 ceph-mon[32884]: pgmap v617: 158 pgs: 53 active+undersized, 25 active+undersized+degraded, 80 active+clean; 635 KiB data, 8.7 GiB used, 528 GiB / 536 GiB avail; 1.3 KiB/s rd, 1 op/s; 92/591 objects degraded (15.567%) 2024-09-10T19:23:59.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:23:58 smithi191 ceph-mon[32884]: from='client.? 172.21.15.183:0/1015951867' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_2", "pool2": "unique_pool_2", "yes_i_really_really_mean_it": true}]: dispatch 2024-09-10T19:23:59.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:23:58 smithi191 ceph-mon[32884]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_2", "pool2": "unique_pool_2", "yes_i_really_really_mean_it": true}]: dispatch 2024-09-10T19:23:59.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:23:58 smithi191 ceph-mon[32884]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:23:59.276 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:23:59.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:59.298 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:23:59.299 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:23:59.400 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:23:59.421 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:23:59.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:59.422 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:23:59.523 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:23:59.549 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:23:59.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:59.550 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:23:59.651 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:23:59.672 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:23:59.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:59.673 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:23:59.774 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:23:59.796 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:23:59.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:59.797 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:23:59.898 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:23:59.903 INFO:teuthology.task.full_sequential:In full_sequential, running task radosbench... 2024-09-10T19:23:59.904 INFO:tasks.radosbench:Beginning radosbench... 2024-09-10T19:23:59.904 INFO:tasks.cephadm.ceph_manager.ceph:creating pool_name unique_pool_3 2024-09-10T19:23:59.905 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph osd pool create unique_pool_3 16 2024-09-10T19:23:59.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:23:59.920 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:23:59.920 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:23:59.938 INFO:journalctl@ceph.iscsi.iscsi.a.smithi191.stdout:Sep 10 19:23:59 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-iscsi-iscsi-a[61816]: debug there is no tcmu-runner data available 2024-09-10T19:24:00.022 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:24:00.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:24:00.044 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:24:00.044 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:24:00.095 INFO:teuthology.orchestra.run.smithi183.stdout:ERROR: (22) Invalid argument 2024-09-10T19:24:00.095 INFO:teuthology.orchestra.run.smithi183.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-10T19:24:00.145 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:24:00.166 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:24:00.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:24:00.167 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:24:00.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:23:59 smithi191 ceph-mon[32884]: 2.1b scrub starts 2024-09-10T19:24:00.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:23:59 smithi191 ceph-mon[32884]: 2.1b scrub ok 2024-09-10T19:24:00.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:23:59 smithi191 ceph-mon[32884]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_2", "pool2": "unique_pool_2", "yes_i_really_really_mean_it": true}]': finished 2024-09-10T19:24:00.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:23:59 smithi191 ceph-mon[32884]: osdmap e291: 8 total, 7 up, 6 in 2024-09-10T19:24:00.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:23:59 smithi191 ceph-mon[32884]: 6.8 scrub starts 2024-09-10T19:24:00.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:23:59 smithi191 ceph-mon[32884]: 6.8 scrub ok 2024-09-10T19:24:00.267 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:24:00.289 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:24:00.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:24:00.290 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:24:00.391 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:24:00.413 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:24:00.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:24:00.414 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:24:00.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:23:59 smithi183 ceph-mon[29736]: 2.1b scrub starts 2024-09-10T19:24:00.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:23:59 smithi183 ceph-mon[29736]: 2.1b scrub ok 2024-09-10T19:24:00.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:23:59 smithi183 ceph-mon[29736]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_2", "pool2": "unique_pool_2", "yes_i_really_really_mean_it": true}]': finished 2024-09-10T19:24:00.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:23:59 smithi183 ceph-mon[29736]: osdmap e291: 8 total, 7 up, 6 in 2024-09-10T19:24:00.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:23:59 smithi183 ceph-mon[29736]: 6.8 scrub starts 2024-09-10T19:24:00.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:23:59 smithi183 ceph-mon[29736]: 6.8 scrub ok 2024-09-10T19:24:00.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:23:59 smithi183 ceph-mon[41562]: 2.1b scrub starts 2024-09-10T19:24:00.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:23:59 smithi183 ceph-mon[41562]: 2.1b scrub ok 2024-09-10T19:24:00.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:23:59 smithi183 ceph-mon[41562]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_2", "pool2": "unique_pool_2", "yes_i_really_really_mean_it": true}]': finished 2024-09-10T19:24:00.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:23:59 smithi183 ceph-mon[41562]: osdmap e291: 8 total, 7 up, 6 in 2024-09-10T19:24:00.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:23:59 smithi183 ceph-mon[41562]: 6.8 scrub starts 2024-09-10T19:24:00.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:23:59 smithi183 ceph-mon[41562]: 6.8 scrub ok 2024-09-10T19:24:00.515 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:24:00.537 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:24:00.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:24:00.538 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:24:00.638 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:24:00.660 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:24:00.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:24:00.661 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:24:00.761 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:24:00.783 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:24:00.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:24:00.784 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:24:00.885 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:24:00.906 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:24:00.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:24:00.907 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:24:00.952 INFO:teuthology.orchestra.run.smithi183.stderr:marked in osd.1. 2024-09-10T19:24:01.008 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:24:01.029 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:24:01.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:24:01.030 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:24:01.131 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:24:01.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:24:01.153 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:24:01.153 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:24:01.181 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:24:00 smithi183 ceph-mon[41562]: 6.1f scrub starts 2024-09-10T19:24:01.181 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:24:00 smithi183 ceph-mon[41562]: from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:24:01.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:24:00 smithi183 ceph-mon[41562]: pgmap v619: 142 pgs: 45 active+undersized, 25 active+undersized+degraded, 72 active+clean; 635 KiB data, 8.7 GiB used, 528 GiB / 536 GiB avail; 589 B/s rd, 0 op/s; 92/591 objects degraded (15.567%) 2024-09-10T19:24:01.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:24:00 smithi183 ceph-mon[41562]: Health check update: Degraded data redundancy: 92/591 objects degraded (15.567%), 25 pgs degraded, 70 pgs undersized (PG_DEGRADED) 2024-09-10T19:24:01.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:24:00 smithi183 ceph-mon[41562]: from='client.? 172.21.15.183:0/1825035110' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["1"]}]: dispatch 2024-09-10T19:24:01.182 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:24:00 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mon-a[29713]: 2024-09-10T19:24:00.886+0000 7f9f28e29640 -1 mon.a@0(leader).osd e291 definitely_dead 0 2024-09-10T19:24:01.182 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:24:00 smithi183 ceph-mon[29736]: 6.1f scrub starts 2024-09-10T19:24:01.182 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:24:00 smithi183 ceph-mon[29736]: from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:24:01.182 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:24:00 smithi183 ceph-mon[29736]: pgmap v619: 142 pgs: 45 active+undersized, 25 active+undersized+degraded, 72 active+clean; 635 KiB data, 8.7 GiB used, 528 GiB / 536 GiB avail; 589 B/s rd, 0 op/s; 92/591 objects degraded (15.567%) 2024-09-10T19:24:01.182 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:24:00 smithi183 ceph-mon[29736]: Health check update: Degraded data redundancy: 92/591 objects degraded (15.567%), 25 pgs degraded, 70 pgs undersized (PG_DEGRADED) 2024-09-10T19:24:01.182 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:24:00 smithi183 ceph-mon[29736]: from='client.? 172.21.15.183:0/1825035110' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["1"]}]: dispatch 2024-09-10T19:24:01.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:24:00 smithi191 ceph-mon[32884]: 6.1f scrub starts 2024-09-10T19:24:01.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:24:00 smithi191 ceph-mon[32884]: from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:24:01.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:24:00 smithi191 ceph-mon[32884]: pgmap v619: 142 pgs: 45 active+undersized, 25 active+undersized+degraded, 72 active+clean; 635 KiB data, 8.7 GiB used, 528 GiB / 536 GiB avail; 589 B/s rd, 0 op/s; 92/591 objects degraded (15.567%) 2024-09-10T19:24:01.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:24:00 smithi191 ceph-mon[32884]: Health check update: Degraded data redundancy: 92/591 objects degraded (15.567%), 25 pgs degraded, 70 pgs undersized (PG_DEGRADED) 2024-09-10T19:24:01.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:24:00 smithi191 ceph-mon[32884]: from='client.? 172.21.15.183:0/1825035110' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["1"]}]: dispatch 2024-09-10T19:24:01.254 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:24:01.276 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:24:01.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:24:01.277 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:24:01.378 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:24:01.399 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:24:01.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:24:01.400 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:24:01.501 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:24:01.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:24:01.524 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:24:01.524 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:24:01.615 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Sep 10 19:24:01 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-y[29989]: ::ffff:172.21.15.191 - - [10/Sep/2024:19:24:01] "GET /metrics HTTP/1.1" 200 224238 "" "Prometheus/2.43.0" 2024-09-10T19:24:01.626 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:24:01.641 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph daemon osd.3 dump_historic_ops 2024-09-10T19:24:01.647 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:24:01.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:24:01.648 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:24:01.749 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:24:01.770 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:24:01.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:24:01.771 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:24:01.872 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:24:01.894 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:24:01.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:24:01.895 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:24:01.996 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:24:02.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:24:02.019 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:24:02.019 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:24:02.120 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:24:02.141 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:24:02.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:24:02.142 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:24:02.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:24:01 smithi191 ceph-mon[32884]: 6.7 scrub starts 2024-09-10T19:24:02.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:24:01 smithi191 ceph-mon[32884]: 6.7 scrub ok 2024-09-10T19:24:02.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:24:01 smithi191 ceph-mon[32884]: from='client.? 172.21.15.183:0/1825035110' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["1"]}]': finished 2024-09-10T19:24:02.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:24:01 smithi191 ceph-mon[32884]: osdmap e292: 8 total, 7 up, 7 in 2024-09-10T19:24:02.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:24:01 smithi191 ceph-mon[32884]: 2.1a scrub starts 2024-09-10T19:24:02.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:24:01 smithi191 ceph-mon[32884]: 2.1a scrub ok 2024-09-10T19:24:02.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:24:01 smithi191 ceph-mon[32884]: 5.9 scrub starts 2024-09-10T19:24:02.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:24:01 smithi191 ceph-mon[32884]: 5.9 scrub ok 2024-09-10T19:24:02.243 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:24:02.260 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:24:01 smithi183 ceph-mon[29736]: 6.7 scrub starts 2024-09-10T19:24:02.260 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:24:01 smithi183 ceph-mon[29736]: 6.7 scrub ok 2024-09-10T19:24:02.260 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:24:01 smithi183 ceph-mon[29736]: from='client.? 172.21.15.183:0/1825035110' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["1"]}]': finished 2024-09-10T19:24:02.260 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:24:01 smithi183 ceph-mon[29736]: osdmap e292: 8 total, 7 up, 7 in 2024-09-10T19:24:02.260 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:24:01 smithi183 ceph-mon[29736]: 2.1a scrub starts 2024-09-10T19:24:02.260 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:24:01 smithi183 ceph-mon[29736]: 2.1a scrub ok 2024-09-10T19:24:02.260 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:24:01 smithi183 ceph-mon[29736]: 5.9 scrub starts 2024-09-10T19:24:02.261 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:24:01 smithi183 ceph-mon[29736]: 5.9 scrub ok 2024-09-10T19:24:02.261 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:24:01 smithi183 ceph-mon[41562]: 6.7 scrub starts 2024-09-10T19:24:02.261 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:24:01 smithi183 ceph-mon[41562]: 6.7 scrub ok 2024-09-10T19:24:02.261 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:24:01 smithi183 ceph-mon[41562]: from='client.? 172.21.15.183:0/1825035110' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["1"]}]': finished 2024-09-10T19:24:02.261 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:24:01 smithi183 ceph-mon[41562]: osdmap e292: 8 total, 7 up, 7 in 2024-09-10T19:24:02.261 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:24:01 smithi183 ceph-mon[41562]: 2.1a scrub starts 2024-09-10T19:24:02.261 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:24:01 smithi183 ceph-mon[41562]: 2.1a scrub ok 2024-09-10T19:24:02.261 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:24:01 smithi183 ceph-mon[41562]: 5.9 scrub starts 2024-09-10T19:24:02.261 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:24:01 smithi183 ceph-mon[41562]: 5.9 scrub ok 2024-09-10T19:24:02.268 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:24:02.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:24:02.269 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:24:02.370 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:24:02.391 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:24:02.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:24:02.392 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:24:02.493 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:24:02.514 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:24:02.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:24:02.515 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:24:02.616 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:24:02.637 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:24:02.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:24:02.638 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:24:02.669 INFO:tasks.thrashosds.thrasher:Added osd 1 2024-09-10T19:24:02.712 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:24:02.739 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:24:02.761 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:24:02.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:24:02.761 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:24:02.862 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:24:02.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:24:02.885 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:24:02.885 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:24:02.961 INFO:teuthology.orchestra.run.smithi183.stderr:nodeep-scrub is unset 2024-09-10T19:24:02.987 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:24:03.008 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:24:03.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:24:03.009 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:24:03.110 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:24:03.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:24:03.132 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:24:03.132 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:24:03.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:24:02 smithi191 ceph-mon[32884]: pgmap v621: 142 pgs: 45 active+undersized, 25 active+undersized+degraded, 72 active+clean; 635 KiB data, 8.7 GiB used, 528 GiB / 536 GiB avail; 1.1 KiB/s rd, 1 op/s; 92/591 objects degraded (15.567%) 2024-09-10T19:24:03.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:24:02 smithi191 ceph-mon[32884]: osdmap e293: 8 total, 7 up, 7 in 2024-09-10T19:24:03.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:24:02 smithi191 ceph-mon[32884]: 4.2 scrub starts 2024-09-10T19:24:03.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:24:02 smithi191 ceph-mon[32884]: 4.2 scrub ok 2024-09-10T19:24:03.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:24:02 smithi191 ceph-mon[32884]: from='client.? 172.21.15.183:0/1299626122' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-10T19:24:03.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:24:02 smithi191 ceph-mon[32884]: 6.3 scrub starts 2024-09-10T19:24:03.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:24:02 smithi191 ceph-mon[32884]: 6.3 scrub ok 2024-09-10T19:24:03.232 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:24:03.254 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:24:03.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:24:03.255 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:24:03.356 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:24:03.377 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:24:03.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:24:03.378 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:24:03.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:24:02 smithi183 ceph-mon[29736]: pgmap v621: 142 pgs: 45 active+undersized, 25 active+undersized+degraded, 72 active+clean; 635 KiB data, 8.7 GiB used, 528 GiB / 536 GiB avail; 1.1 KiB/s rd, 1 op/s; 92/591 objects degraded (15.567%) 2024-09-10T19:24:03.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:24:02 smithi183 ceph-mon[29736]: osdmap e293: 8 total, 7 up, 7 in 2024-09-10T19:24:03.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:24:02 smithi183 ceph-mon[29736]: 4.2 scrub starts 2024-09-10T19:24:03.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:24:02 smithi183 ceph-mon[29736]: 4.2 scrub ok 2024-09-10T19:24:03.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:24:02 smithi183 ceph-mon[29736]: from='client.? 172.21.15.183:0/1299626122' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-10T19:24:03.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:24:02 smithi183 ceph-mon[29736]: 6.3 scrub starts 2024-09-10T19:24:03.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:24:02 smithi183 ceph-mon[29736]: 6.3 scrub ok 2024-09-10T19:24:03.433 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:24:02 smithi183 ceph-mon[41562]: pgmap v621: 142 pgs: 45 active+undersized, 25 active+undersized+degraded, 72 active+clean; 635 KiB data, 8.7 GiB used, 528 GiB / 536 GiB avail; 1.1 KiB/s rd, 1 op/s; 92/591 objects degraded (15.567%) 2024-09-10T19:24:03.433 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:24:02 smithi183 ceph-mon[41562]: osdmap e293: 8 total, 7 up, 7 in 2024-09-10T19:24:03.433 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:24:02 smithi183 ceph-mon[41562]: 4.2 scrub starts 2024-09-10T19:24:03.433 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:24:02 smithi183 ceph-mon[41562]: 4.2 scrub ok 2024-09-10T19:24:03.433 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:24:02 smithi183 ceph-mon[41562]: from='client.? 172.21.15.183:0/1299626122' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-10T19:24:03.433 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:24:02 smithi183 ceph-mon[41562]: 6.3 scrub starts 2024-09-10T19:24:03.433 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:24:02 smithi183 ceph-mon[41562]: 6.3 scrub ok 2024-09-10T19:24:03.479 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:24:03.501 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:24:03.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:24:03.502 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:24:03.603 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:24:03.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:24:03.626 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:24:03.627 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:24:03.728 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:24:03.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:24:03.751 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:24:03.752 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:24:03.854 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:24:03.876 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:24:03.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:24:03.877 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:24:03.978 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:24:03.999 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:24:04.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:24:04.000 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:24:04.057 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:24:03 smithi183 ceph-mon[29736]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-10T19:24:04.057 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:24:03 smithi183 ceph-mon[29736]: from='client.? 172.21.15.183:0/1299626122' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-10T19:24:04.057 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:24:03 smithi183 ceph-mon[29736]: osdmap e294: 8 total, 7 up, 7 in 2024-09-10T19:24:04.057 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:24:03 smithi183 ceph-mon[29736]: 2.0 scrub starts 2024-09-10T19:24:04.057 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:24:03 smithi183 ceph-mon[29736]: 2.0 scrub ok 2024-09-10T19:24:04.058 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:24:03 smithi183 ceph-mon[41562]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-10T19:24:04.058 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:24:03 smithi183 ceph-mon[41562]: from='client.? 172.21.15.183:0/1299626122' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-10T19:24:04.058 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:24:03 smithi183 ceph-mon[41562]: osdmap e294: 8 total, 7 up, 7 in 2024-09-10T19:24:04.058 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:24:03 smithi183 ceph-mon[41562]: 2.0 scrub starts 2024-09-10T19:24:04.058 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:24:03 smithi183 ceph-mon[41562]: 2.0 scrub ok 2024-09-10T19:24:04.101 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:24:04.123 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:24:04.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:24:04.124 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:24:04.224 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:24:04.246 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:24:04.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:24:04.247 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:24:04.348 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:24:04.369 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:24:04.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:24:04.371 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:24:04.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:24:03 smithi191 ceph-mon[32884]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-10T19:24:04.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:24:03 smithi191 ceph-mon[32884]: from='client.? 172.21.15.183:0/1299626122' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-10T19:24:04.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:24:03 smithi191 ceph-mon[32884]: osdmap e294: 8 total, 7 up, 7 in 2024-09-10T19:24:04.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:24:03 smithi191 ceph-mon[32884]: 2.0 scrub starts 2024-09-10T19:24:04.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:24:03 smithi191 ceph-mon[32884]: 2.0 scrub ok 2024-09-10T19:24:04.471 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:24:04.494 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:24:04.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:24:04.495 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:24:04.596 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:24:04.617 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:24:04.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:24:04.618 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:24:04.719 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:24:04.741 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:24:04.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:24:04.742 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:24:04.760 INFO:teuthology.orchestra.run.smithi183.stderr:Error ENXIO: problem getting command descriptions from osd.0 2024-09-10T19:24:04.762 INFO:teuthology.orchestra.run.smithi183.stdout:osd.1: {} 2024-09-10T19:24:04.763 INFO:teuthology.orchestra.run.smithi183.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-10T19:24:04.778 INFO:teuthology.orchestra.run.smithi183.stdout:osd.2: {} 2024-09-10T19:24:04.778 INFO:teuthology.orchestra.run.smithi183.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-10T19:24:04.793 INFO:teuthology.orchestra.run.smithi183.stdout:osd.3: {} 2024-09-10T19:24:04.793 INFO:teuthology.orchestra.run.smithi183.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-10T19:24:04.809 INFO:teuthology.orchestra.run.smithi183.stdout:osd.4: {} 2024-09-10T19:24:04.809 INFO:teuthology.orchestra.run.smithi183.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-10T19:24:04.824 INFO:teuthology.orchestra.run.smithi183.stdout:osd.5: {} 2024-09-10T19:24:04.824 INFO:teuthology.orchestra.run.smithi183.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-10T19:24:04.839 INFO:teuthology.orchestra.run.smithi183.stdout:osd.6: {} 2024-09-10T19:24:04.839 INFO:teuthology.orchestra.run.smithi183.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-10T19:24:04.843 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:24:04.854 INFO:teuthology.orchestra.run.smithi183.stdout:osd.7: {} 2024-09-10T19:24:04.854 INFO:teuthology.orchestra.run.smithi183.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-10T19:24:04.864 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:24:04.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:24:04.865 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:24:04.965 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:24:04.987 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:24:04.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:24:04.988 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:24:05.089 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:24:05.112 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:24:05.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:24:05.114 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:24:05.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:24:04 smithi183 ceph-mon[29736]: 4.d scrub starts 2024-09-10T19:24:05.182 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:24:04 smithi183 ceph-mon[29736]: 5.0 scrub starts 2024-09-10T19:24:05.182 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:24:04 smithi183 ceph-mon[29736]: 4.d scrub ok 2024-09-10T19:24:05.182 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:24:04 smithi183 ceph-mon[29736]: 5.0 scrub ok 2024-09-10T19:24:05.182 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:24:04 smithi183 ceph-mon[29736]: pgmap v624: 142 pgs: 14 remapped+peering, 36 active+undersized, 18 active+undersized+degraded, 74 active+clean; 686 KiB data, 9.0 GiB used, 617 GiB / 626 GiB avail; 60/510 objects degraded (11.765%); 11 KiB/s, 0 objects/s recovering 2024-09-10T19:24:05.182 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:24:04 smithi183 ceph-mon[29736]: 4.1f scrub starts 2024-09-10T19:24:05.182 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:24:04 smithi183 ceph-mon[29736]: 4.1f scrub ok 2024-09-10T19:24:05.182 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:24:04 smithi183 ceph-mon[29736]: 5.1e scrub starts 2024-09-10T19:24:05.182 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:24:04 smithi183 ceph-mon[29736]: 5.1e scrub ok 2024-09-10T19:24:05.182 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:24:04 smithi183 ceph-mon[29736]: from='client.? 172.21.15.183:0/369454901' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T19:24:05.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:24:04 smithi183 ceph-mon[41562]: 4.d scrub starts 2024-09-10T19:24:05.183 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:24:04 smithi183 ceph-mon[41562]: 5.0 scrub starts 2024-09-10T19:24:05.183 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:24:04 smithi183 ceph-mon[41562]: 4.d scrub ok 2024-09-10T19:24:05.183 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:24:04 smithi183 ceph-mon[41562]: 5.0 scrub ok 2024-09-10T19:24:05.183 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:24:04 smithi183 ceph-mon[41562]: pgmap v624: 142 pgs: 14 remapped+peering, 36 active+undersized, 18 active+undersized+degraded, 74 active+clean; 686 KiB data, 9.0 GiB used, 617 GiB / 626 GiB avail; 60/510 objects degraded (11.765%); 11 KiB/s, 0 objects/s recovering 2024-09-10T19:24:05.183 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:24:04 smithi183 ceph-mon[41562]: 4.1f scrub starts 2024-09-10T19:24:05.183 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:24:04 smithi183 ceph-mon[41562]: 4.1f scrub ok 2024-09-10T19:24:05.183 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:24:04 smithi183 ceph-mon[41562]: 5.1e scrub starts 2024-09-10T19:24:05.183 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:24:04 smithi183 ceph-mon[41562]: 5.1e scrub ok 2024-09-10T19:24:05.183 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:24:04 smithi183 ceph-mon[41562]: from='client.? 172.21.15.183:0/369454901' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T19:24:05.215 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:24:05.239 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:24:05.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:24:05.240 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:24:05.341 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:24:05.363 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:24:05.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:24:05.364 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:24:05.465 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:24:05.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:24:04 smithi191 ceph-mon[32884]: 4.d scrub starts 2024-09-10T19:24:05.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:24:04 smithi191 ceph-mon[32884]: 5.0 scrub starts 2024-09-10T19:24:05.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:24:04 smithi191 ceph-mon[32884]: 4.d scrub ok 2024-09-10T19:24:05.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:24:04 smithi191 ceph-mon[32884]: 5.0 scrub ok 2024-09-10T19:24:05.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:24:04 smithi191 ceph-mon[32884]: pgmap v624: 142 pgs: 14 remapped+peering, 36 active+undersized, 18 active+undersized+degraded, 74 active+clean; 686 KiB data, 9.0 GiB used, 617 GiB / 626 GiB avail; 60/510 objects degraded (11.765%); 11 KiB/s, 0 objects/s recovering 2024-09-10T19:24:05.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:24:04 smithi191 ceph-mon[32884]: 4.1f scrub starts 2024-09-10T19:24:05.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:24:04 smithi191 ceph-mon[32884]: 4.1f scrub ok 2024-09-10T19:24:05.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:24:04 smithi191 ceph-mon[32884]: 5.1e scrub starts 2024-09-10T19:24:05.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:24:04 smithi191 ceph-mon[32884]: 5.1e scrub ok 2024-09-10T19:24:05.467 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:24:04 smithi191 ceph-mon[32884]: from='client.? 172.21.15.183:0/369454901' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T19:24:05.488 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:24:05.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:24:05.489 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:24:05.590 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:24:05.612 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:24:05.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:24:05.613 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:24:05.714 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:24:05.735 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:24:05.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:24:05.736 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:24:05.837 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:24:05.859 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:24:05.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:24:05.860 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:24:05.960 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:24:05.986 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:24:05.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:24:05.987 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:24:06.088 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:24:06.109 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:24:06.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:24:06.110 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:24:06.211 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:24:06.232 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:24:06.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:24:06.233 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:24:06.334 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:24:06.357 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:24:06.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:24:06.359 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:24:06.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:24:06 smithi183 ceph-mon[29736]: Health check update: Degraded data redundancy: 45/510 objects degraded (8.824%), 14 pgs degraded, 35 pgs undersized (PG_DEGRADED) 2024-09-10T19:24:06.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:24:06 smithi183 ceph-mon[41562]: Health check update: Degraded data redundancy: 45/510 objects degraded (8.824%), 14 pgs degraded, 35 pgs undersized (PG_DEGRADED) 2024-09-10T19:24:06.459 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:24:06.465 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:24:06 smithi191 ceph-mon[32884]: Health check update: Degraded data redundancy: 45/510 objects degraded (8.824%), 14 pgs degraded, 35 pgs undersized (PG_DEGRADED) 2024-09-10T19:24:06.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:24:06.482 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:24:06.482 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:24:06.583 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:24:06.604 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:24:06.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:24:06.605 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:24:06.706 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:24:06.732 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:24:06.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:24:06.733 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:24:06.778 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-10T19:24:06.779 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-10T19:24:06.834 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:24:06.856 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:24:06.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:24:06.857 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:24:06.958 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:24:06.984 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:24:06.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:24:06.985 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:24:07.004 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:24:07.085 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:24:07.110 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:24:07.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:24:07.111 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:24:07.211 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:24:07.237 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:24:07.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:24:07.238 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:24:07.285 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:24:07 smithi183 ceph-mon[41562]: 2.1 scrub starts 2024-09-10T19:24:07.285 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:24:07 smithi183 ceph-mon[41562]: 2.1 scrub ok 2024-09-10T19:24:07.285 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:24:07 smithi183 ceph-mon[41562]: 2.c scrub starts 2024-09-10T19:24:07.285 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:24:07 smithi183 ceph-mon[41562]: 2.c scrub ok 2024-09-10T19:24:07.285 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:24:07 smithi183 ceph-mon[41562]: pgmap v625: 142 pgs: 10 active+clean+remapped, 14 remapped+peering, 21 active+undersized, 14 active+undersized+degraded, 83 active+clean; 686 KiB data, 9.3 GiB used, 617 GiB / 626 GiB avail; 45/510 objects degraded (8.824%); 1/510 objects misplaced (0.196%); 8.8 KiB/s, 7 objects/s recovering 2024-09-10T19:24:07.285 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:24:07 smithi183 ceph-mon[29736]: 2.1 scrub starts 2024-09-10T19:24:07.286 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:24:07 smithi183 ceph-mon[29736]: 2.1 scrub ok 2024-09-10T19:24:07.286 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:24:07 smithi183 ceph-mon[29736]: 2.c scrub starts 2024-09-10T19:24:07.286 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:24:07 smithi183 ceph-mon[29736]: 2.c scrub ok 2024-09-10T19:24:07.286 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:24:07 smithi183 ceph-mon[29736]: pgmap v625: 142 pgs: 10 active+clean+remapped, 14 remapped+peering, 21 active+undersized, 14 active+undersized+degraded, 83 active+clean; 686 KiB data, 9.3 GiB used, 617 GiB / 626 GiB avail; 45/510 objects degraded (8.824%); 1/510 objects misplaced (0.196%); 8.8 KiB/s, 7 objects/s recovering 2024-09-10T19:24:07.338 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:24:07.363 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:24:07.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:24:07.364 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:24:07.450 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:24:07 smithi191 ceph-mon[32884]: 2.1 scrub starts 2024-09-10T19:24:07.450 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:24:07 smithi191 ceph-mon[32884]: 2.1 scrub ok 2024-09-10T19:24:07.450 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:24:07 smithi191 ceph-mon[32884]: 2.c scrub starts 2024-09-10T19:24:07.450 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:24:07 smithi191 ceph-mon[32884]: 2.c scrub ok 2024-09-10T19:24:07.450 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:24:07 smithi191 ceph-mon[32884]: pgmap v625: 142 pgs: 10 active+clean+remapped, 14 remapped+peering, 21 active+undersized, 14 active+undersized+degraded, 83 active+clean; 686 KiB data, 9.3 GiB used, 617 GiB / 626 GiB avail; 45/510 objects degraded (8.824%); 1/510 objects misplaced (0.196%); 8.8 KiB/s, 7 objects/s recovering 2024-09-10T19:24:07.466 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:24:07.490 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:24:07.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:24:07.491 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:24:07.592 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:24:07.616 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:24:07.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:24:07.617 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:24:07.670 INFO:tasks.thrashosds.thrasher:in_osds: [0, 3, 4, 5, 6, 7, 1] out_osds: [2] dead_osds: [0] live_osds: [2, 3, 4, 5, 6, 7, 1] 2024-09-10T19:24:07.670 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-10T19:24:07.670 INFO:tasks.thrashosds.thrasher:Reviving osd 0 2024-09-10T19:24:07.670 INFO:tasks.cephadm.osd.0:Restarting osd.0 (starting--it wasn't running)... 2024-09-10T19:24:07.670 DEBUG:teuthology.orchestra.run.smithi183:osd.0> sudo journalctl -f -n 0 -u ceph-320f6df8-6fa7-11ef-bcea-c7b262605968@osd.0.service 2024-09-10T19:24:07.673 DEBUG:teuthology.orchestra.run.smithi183:> sudo systemctl start ceph-320f6df8-6fa7-11ef-bcea-c7b262605968@osd.0 2024-09-10T19:24:07.716 INFO:journalctl@ceph.mgr.x.smithi191.stdout:Sep 10 19:24:07 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-x[34967]: ::ffff:172.21.15.191 - - [10/Sep/2024:19:24:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T19:24:07.718 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:24:07.779 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:24:07.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:24:07.780 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:24:07.881 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:24:07.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:24:07.908 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:24:07.909 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:24:07.971 INFO:journalctl@ceph.osd.0.smithi183.stdout:Sep 10 19:24:07 smithi183 systemd[1]: Starting Ceph osd.0 for 320f6df8-6fa7-11ef-bcea-c7b262605968... 2024-09-10T19:24:08.010 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:24:08.035 INFO:teuthology.orchestra.run.smithi183.stderr:pool 'unique_pool_3' created 2024-09-10T19:24:08.037 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph osd set noscrub 2024-09-10T19:24:08.038 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:24:08.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:24:08.039 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:24:08.140 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:24:08.164 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:24:08.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:24:08.165 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:24:08.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:24:08 smithi191 ceph-mon[32884]: 5.13 scrub starts 2024-09-10T19:24:08.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:24:08 smithi191 ceph-mon[32884]: 5.13 scrub ok 2024-09-10T19:24:08.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:24:08 smithi191 ceph-mon[32884]: from='client.? 172.21.15.183:0/260340201' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_3", "pg_num": 16}]: dispatch 2024-09-10T19:24:08.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:24:08 smithi191 ceph-mon[32884]: 2.15 scrub starts 2024-09-10T19:24:08.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:24:08 smithi191 ceph-mon[32884]: 2.15 scrub ok 2024-09-10T19:24:08.266 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:24:08.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:24:08.291 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:24:08.291 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:24:08.392 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:24:08.414 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:24:08.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:24:08.415 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:24:08.431 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:24:08 smithi183 ceph-mon[41562]: 5.13 scrub starts 2024-09-10T19:24:08.431 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:24:08 smithi183 ceph-mon[41562]: 5.13 scrub ok 2024-09-10T19:24:08.431 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:24:08 smithi183 ceph-mon[41562]: from='client.? 172.21.15.183:0/260340201' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_3", "pg_num": 16}]: dispatch 2024-09-10T19:24:08.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:24:08 smithi183 ceph-mon[41562]: 2.15 scrub starts 2024-09-10T19:24:08.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:24:08 smithi183 ceph-mon[41562]: 2.15 scrub ok 2024-09-10T19:24:08.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:24:08 smithi183 ceph-mon[29736]: 5.13 scrub starts 2024-09-10T19:24:08.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:24:08 smithi183 ceph-mon[29736]: 5.13 scrub ok 2024-09-10T19:24:08.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:24:08 smithi183 ceph-mon[29736]: from='client.? 172.21.15.183:0/260340201' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_3", "pg_num": 16}]: dispatch 2024-09-10T19:24:08.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:24:08 smithi183 ceph-mon[29736]: 2.15 scrub starts 2024-09-10T19:24:08.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:24:08 smithi183 ceph-mon[29736]: 2.15 scrub ok 2024-09-10T19:24:08.432 INFO:journalctl@ceph.osd.0.smithi183.stdout:Sep 10 19:24:07 smithi183 podman[174181]: 2024-09-10 19:24:07.872687374 +0000 UTC m=+0.015446727 image pull 4c55dd4096115e48502f368cff1eab06769f71f030e1692e2611bbc6c6f637e9 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3bfeaf4417585c821cab9ca76e4d95f390b0e769807122a45463640820b5cd16 2024-09-10T19:24:08.433 INFO:journalctl@ceph.osd.0.smithi183.stdout:Sep 10 19:24:07 smithi183 podman[174181]: 2024-09-10 19:24:07.978463564 +0000 UTC m=+0.121222909 container create 422660d45ff1e63318cead9a7377a21ba2c9fc3d6800b10f08c3a417f79f83b5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3bfeaf4417585c821cab9ca76e4d95f390b0e769807122a45463640820b5cd16, name=ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd-0-activate, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-guits-quincy-2024-09-09-1810-quincy-5be5739) 2024-09-10T19:24:08.516 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:24:08.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:24:08.539 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:24:08.539 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:24:08.640 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:24:08.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:24:08.662 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:24:08.662 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:24:08.763 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:24:08.780 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-10T19:24:08.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:24:08.785 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:24:08.785 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:24:08.887 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:24:08.909 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:24:08.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:24:08.910 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:24:08.946 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph osd pool application enable unique_pool_3 rados --yes-i-really-mean-it || true 2024-09-10T19:24:09.010 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:24:09.056 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:24:09.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:24:09.057 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:24:09.059 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:24:09 smithi183 ceph-mon[29736]: 6.19 scrub starts 2024-09-10T19:24:09.060 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:24:09 smithi183 ceph-mon[29736]: 6.19 scrub ok 2024-09-10T19:24:09.060 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:24:09 smithi183 ceph-mon[29736]: pgmap v626: 142 pgs: 24 active+clean+remapped, 21 active+undersized, 14 active+undersized+degraded, 83 active+clean; 687 KiB data, 9.3 GiB used, 617 GiB / 626 GiB avail; 21 KiB/s wr, 1 op/s; 45/591 objects degraded (7.614%); 28/591 objects misplaced (4.738%); 8.0 KiB/s, 15 objects/s recovering 2024-09-10T19:24:09.060 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:24:09 smithi183 ceph-mon[29736]: from='client.? 172.21.15.183:0/260340201' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_3", "pg_num": 16}]': finished 2024-09-10T19:24:09.060 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:24:09 smithi183 ceph-mon[29736]: osdmap e295: 8 total, 7 up, 7 in 2024-09-10T19:24:09.060 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:24:09 smithi183 ceph-mon[29736]: 5.1a scrub starts 2024-09-10T19:24:09.060 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:24:09 smithi183 ceph-mon[29736]: 5.1a scrub ok 2024-09-10T19:24:09.060 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:24:09 smithi183 ceph-mon[29736]: purged_snaps scrub starts 2024-09-10T19:24:09.060 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:24:09 smithi183 ceph-mon[29736]: purged_snaps scrub ok 2024-09-10T19:24:09.061 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:24:09 smithi183 ceph-mon[41562]: 6.19 scrub starts 2024-09-10T19:24:09.061 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:24:09 smithi183 ceph-mon[41562]: 6.19 scrub ok 2024-09-10T19:24:09.061 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:24:09 smithi183 ceph-mon[41562]: pgmap v626: 142 pgs: 24 active+clean+remapped, 21 active+undersized, 14 active+undersized+degraded, 83 active+clean; 687 KiB data, 9.3 GiB used, 617 GiB / 626 GiB avail; 21 KiB/s wr, 1 op/s; 45/591 objects degraded (7.614%); 28/591 objects misplaced (4.738%); 8.0 KiB/s, 15 objects/s recovering 2024-09-10T19:24:09.061 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:24:09 smithi183 ceph-mon[41562]: from='client.? 172.21.15.183:0/260340201' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_3", "pg_num": 16}]': finished 2024-09-10T19:24:09.061 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:24:09 smithi183 ceph-mon[41562]: osdmap e295: 8 total, 7 up, 7 in 2024-09-10T19:24:09.061 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:24:09 smithi183 ceph-mon[41562]: 5.1a scrub starts 2024-09-10T19:24:09.061 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:24:09 smithi183 ceph-mon[41562]: 5.1a scrub ok 2024-09-10T19:24:09.061 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:24:09 smithi183 ceph-mon[41562]: purged_snaps scrub starts 2024-09-10T19:24:09.061 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:24:09 smithi183 ceph-mon[41562]: purged_snaps scrub ok 2024-09-10T19:24:09.061 INFO:journalctl@ceph.osd.0.smithi183.stdout:Sep 10 19:24:09 smithi183 podman[174181]: 2024-09-10 19:24:09.053125052 +0000 UTC m=+1.195884400 container init 422660d45ff1e63318cead9a7377a21ba2c9fc3d6800b10f08c3a417f79f83b5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3bfeaf4417585c821cab9ca76e4d95f390b0e769807122a45463640820b5cd16, name=ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd-0-activate, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_CLEAN=True, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, RELEASE=wip-guits-quincy-2024-09-09-1810-quincy-5be5739, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, org.label-schema.license=GPLv2) 2024-09-10T19:24:09.061 INFO:journalctl@ceph.osd.0.smithi183.stdout:Sep 10 19:24:09 smithi183 podman[174181]: 2024-09-10 19:24:09.058709984 +0000 UTC m=+1.201469333 container start 422660d45ff1e63318cead9a7377a21ba2c9fc3d6800b10f08c3a417f79f83b5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3bfeaf4417585c821cab9ca76e4d95f390b0e769807122a45463640820b5cd16, name=ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd-0-activate, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, ceph=True, RELEASE=wip-guits-quincy-2024-09-09-1810-quincy-5be5739, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731) 2024-09-10T19:24:09.158 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:24:09.181 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:24:09.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:24:09.182 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:24:09.283 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:24:09.309 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:24:09.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:24:09.310 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:24:09.342 INFO:journalctl@ceph.osd.0.smithi183.stdout:Sep 10 19:24:09 smithi183 podman[174181]: 2024-09-10 19:24:09.178243338 +0000 UTC m=+1.321002692 container attach 422660d45ff1e63318cead9a7377a21ba2c9fc3d6800b10f08c3a417f79f83b5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3bfeaf4417585c821cab9ca76e4d95f390b0e769807122a45463640820b5cd16, name=ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd-0-activate, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, ceph=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, RELEASE=wip-guits-quincy-2024-09-09-1810-quincy-5be5739, maintainer=Guillaume Abrioux ) 2024-09-10T19:24:09.411 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:24:09.433 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:24:09.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:24:09.434 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:24:09.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:24:09 smithi191 ceph-mon[32884]: 6.19 scrub starts 2024-09-10T19:24:09.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:24:09 smithi191 ceph-mon[32884]: 6.19 scrub ok 2024-09-10T19:24:09.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:24:09 smithi191 ceph-mon[32884]: pgmap v626: 142 pgs: 24 active+clean+remapped, 21 active+undersized, 14 active+undersized+degraded, 83 active+clean; 687 KiB data, 9.3 GiB used, 617 GiB / 626 GiB avail; 21 KiB/s wr, 1 op/s; 45/591 objects degraded (7.614%); 28/591 objects misplaced (4.738%); 8.0 KiB/s, 15 objects/s recovering 2024-09-10T19:24:09.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:24:09 smithi191 ceph-mon[32884]: from='client.? 172.21.15.183:0/260340201' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_3", "pg_num": 16}]': finished 2024-09-10T19:24:09.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:24:09 smithi191 ceph-mon[32884]: osdmap e295: 8 total, 7 up, 7 in 2024-09-10T19:24:09.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:24:09 smithi191 ceph-mon[32884]: 5.1a scrub starts 2024-09-10T19:24:09.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:24:09 smithi191 ceph-mon[32884]: 5.1a scrub ok 2024-09-10T19:24:09.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:24:09 smithi191 ceph-mon[32884]: purged_snaps scrub starts 2024-09-10T19:24:09.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:24:09 smithi191 ceph-mon[32884]: purged_snaps scrub ok 2024-09-10T19:24:09.534 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:24:09.556 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:24:09.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:24:09.557 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:24:09.657 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:24:09.681 INFO:journalctl@ceph.osd.0.smithi183.stdout:Sep 10 19:24:09 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd-0-activate[174226]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-09-10T19:24:09.682 INFO:journalctl@ceph.osd.0.smithi183.stdout:Sep 10 19:24:09 smithi183 bash[174181]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-09-10T19:24:09.682 INFO:journalctl@ceph.osd.0.smithi183.stdout:Sep 10 19:24:09 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd-0-activate[174226]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-0 --no-mon-config --dev /dev/mapper/vg_nvme-lv_4 2024-09-10T19:24:09.682 INFO:journalctl@ceph.osd.0.smithi183.stdout:Sep 10 19:24:09 smithi183 bash[174181]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-0 --no-mon-config --dev /dev/mapper/vg_nvme-lv_4 2024-09-10T19:24:09.682 INFO:journalctl@ceph.osd.0.smithi183.stdout:Sep 10 19:24:09 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd-0-activate[174226]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_4 2024-09-10T19:24:09.682 INFO:journalctl@ceph.osd.0.smithi183.stdout:Sep 10 19:24:09 smithi183 bash[174181]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_4 2024-09-10T19:24:09.682 INFO:journalctl@ceph.osd.0.smithi183.stdout:Sep 10 19:24:09 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd-0-activate[174226]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-3 2024-09-10T19:24:09.682 INFO:journalctl@ceph.osd.0.smithi183.stdout:Sep 10 19:24:09 smithi183 bash[174181]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-3 2024-09-10T19:24:09.682 INFO:journalctl@ceph.osd.0.smithi183.stdout:Sep 10 19:24:09 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd-0-activate[174226]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-0/block 2024-09-10T19:24:09.682 INFO:journalctl@ceph.osd.0.smithi183.stdout:Sep 10 19:24:09 smithi183 bash[174181]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-0/block 2024-09-10T19:24:09.682 INFO:journalctl@ceph.osd.0.smithi183.stdout:Sep 10 19:24:09 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd-0-activate[174226]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-09-10T19:24:09.682 INFO:journalctl@ceph.osd.0.smithi183.stdout:Sep 10 19:24:09 smithi183 bash[174181]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-09-10T19:24:09.683 INFO:journalctl@ceph.osd.0.smithi183.stdout:Sep 10 19:24:09 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd-0-activate[174226]: --> ceph-volume raw activate successful for osd ID: 0 2024-09-10T19:24:09.683 INFO:journalctl@ceph.osd.0.smithi183.stdout:Sep 10 19:24:09 smithi183 bash[174181]: --> ceph-volume raw activate successful for osd ID: 0 2024-09-10T19:24:09.683 INFO:journalctl@ceph.osd.0.smithi183.stdout:Sep 10 19:24:09 smithi183 podman[174181]: 2024-09-10 19:24:09.38801376 +0000 UTC m=+1.530773110 container died 422660d45ff1e63318cead9a7377a21ba2c9fc3d6800b10f08c3a417f79f83b5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3bfeaf4417585c821cab9ca76e4d95f390b0e769807122a45463640820b5cd16, name=ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd-0-activate, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, RELEASE=wip-guits-quincy-2024-09-09-1810-quincy-5be5739, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-10T19:24:09.683 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:24:09.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:24:09.684 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:24:09.785 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:24:09.807 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:24:09.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:24:09.809 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:24:09.909 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:24:09.935 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:24:09.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:24:09.952 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:24:09.965 INFO:journalctl@ceph.iscsi.iscsi.a.smithi191.stdout:Sep 10 19:24:09 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-iscsi-iscsi-a[61816]: debug there is no tcmu-runner data available 2024-09-10T19:24:10.053 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:24:10.079 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:24:10.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:24:10.080 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:24:10.181 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:24:10.206 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:24:10.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:24:10.207 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:24:10.243 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:24:10 smithi183 ceph-mon[29736]: 6.d scrub starts 2024-09-10T19:24:10.243 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:24:10 smithi183 ceph-mon[29736]: 6.d scrub ok 2024-09-10T19:24:10.243 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:24:10 smithi183 ceph-mon[29736]: osdmap e296: 8 total, 7 up, 7 in 2024-09-10T19:24:10.243 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:24:10 smithi183 ceph-mon[41562]: 6.d scrub starts 2024-09-10T19:24:10.243 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:24:10 smithi183 ceph-mon[41562]: 6.d scrub ok 2024-09-10T19:24:10.244 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:24:10 smithi183 ceph-mon[41562]: osdmap e296: 8 total, 7 up, 7 in 2024-09-10T19:24:10.307 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:24:10.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:24:10.333 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:24:10.333 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:24:10.434 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:24:10.459 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:24:10.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:24:10.460 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:24:10.465 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:24:10 smithi191 ceph-mon[32884]: 6.d scrub starts 2024-09-10T19:24:10.465 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:24:10 smithi191 ceph-mon[32884]: 6.d scrub ok 2024-09-10T19:24:10.465 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:24:10 smithi191 ceph-mon[32884]: osdmap e296: 8 total, 7 up, 7 in 2024-09-10T19:24:10.561 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:24:10.586 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:24:10.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:24:10.587 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:24:10.688 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:24:10.695 INFO:teuthology.orchestra.run.smithi183.stdout:{ 2024-09-10T19:24:10.695 INFO:teuthology.orchestra.run.smithi183.stdout: "size": 20, 2024-09-10T19:24:10.695 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 600, 2024-09-10T19:24:10.695 INFO:teuthology.orchestra.run.smithi183.stdout: "ops": [ 2024-09-10T19:24:10.695 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:10.695 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.15051.0:59075 8.3 8:cfee2355:::benchmark_data_smithi183_114924_object59074:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e166)", 2024-09-10T19:24:10.695 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-10T19:16:55.988767+0000", 2024-09-10T19:24:10.695 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 434.70294562499998, 2024-09-10T19:24:10.695 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.8076374039999998, 2024-09-10T19:24:10.695 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-10T19:24:10.695 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T19:24:10.696 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-10T19:24:10.696 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.15051", 2024-09-10T19:24:10.696 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.183:0/3632028672", 2024-09-10T19:24:10.696 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 59075 2024-09-10T19:24:10.696 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:10.696 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-10T19:24:10.696 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:10.696 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-10T19:24:10.696 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:55.988767+0000", 2024-09-10T19:24:10.696 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:24:10.696 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:10.696 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:10.697 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-10T19:24:10.697 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:55.988767+0000", 2024-09-10T19:24:10.697 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:24:10.697 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:10.697 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:10.697 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-10T19:24:10.697 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:55.988765+0000", 2024-09-10T19:24:10.697 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4294967295.999999 2024-09-10T19:24:10.697 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:10.697 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:10.697 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-10T19:24:10.697 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:55.988810+0000", 2024-09-10T19:24:10.697 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.4308999999999999e-05 2024-09-10T19:24:10.697 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:10.698 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:10.698 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-10T19:24:10.698 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:55.988820+0000", 2024-09-10T19:24:10.698 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.9699999999999994e-06 2024-09-10T19:24:10.698 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:10.698 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:10.698 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-10T19:24:10.698 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:55.988825+0000", 2024-09-10T19:24:10.698 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.198e-06 2024-09-10T19:24:10.698 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:10.698 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:10.698 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-10T19:24:10.698 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:55.988871+0000", 2024-09-10T19:24:10.698 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.5540000000000001e-05 2024-09-10T19:24:10.699 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:10.699 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:10.699 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for peered", 2024-09-10T19:24:10.699 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:55.988881+0000", 2024-09-10T19:24:10.699 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.0849000000000001e-05 2024-09-10T19:24:10.699 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:10.699 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:10.699 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-10T19:24:10.699 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:57.774576+0000", 2024-09-10T19:24:10.699 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.7856943159999998 2024-09-10T19:24:10.699 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:10.699 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:10.699 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-10T19:24:10.700 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:57.775578+0000", 2024-09-10T19:24:10.700 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0010023149999999999 2024-09-10T19:24:10.700 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:10.700 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:10.700 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 5,7", 2024-09-10T19:24:10.700 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:57.775706+0000", 2024-09-10T19:24:10.700 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00012781399999999999 2024-09-10T19:24:10.700 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:10.700 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:10.700 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-10T19:24:10.700 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:57.781861+0000", 2024-09-10T19:24:10.700 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0061548009999999997 2024-09-10T19:24:10.700 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:10.701 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:10.701 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:24:10.701 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:57.795208+0000", 2024-09-10T19:24:10.701 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.013347314000000001 2024-09-10T19:24:10.701 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:10.701 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:10.701 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:24:10.701 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:57.795320+0000", 2024-09-10T19:24:10.701 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000112193 2024-09-10T19:24:10.701 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:10.701 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:10.702 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-10T19:24:10.702 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:57.795357+0000", 2024-09-10T19:24:10.702 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.6838999999999997e-05 2024-09-10T19:24:10.702 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:10.702 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:10.702 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-10T19:24:10.702 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:57.796404+0000", 2024-09-10T19:24:10.702 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.001047014 2024-09-10T19:24:10.703 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:24:10.703 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:24:10.703 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:24:10.703 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:10.703 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:10.703 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.15051.0:59116 8.3 8:ce8335d8:::benchmark_data_smithi183_114924_object59115:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e166)", 2024-09-10T19:24:10.703 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-10T19:16:55.988835+0000", 2024-09-10T19:24:10.703 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 434.70287675399999, 2024-09-10T19:24:10.703 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.808536116, 2024-09-10T19:24:10.703 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-10T19:24:10.703 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T19:24:10.703 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-10T19:24:10.703 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.15051", 2024-09-10T19:24:10.704 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.183:0/3632028672", 2024-09-10T19:24:10.704 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 59116 2024-09-10T19:24:10.704 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:10.704 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-10T19:24:10.704 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:10.704 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-10T19:24:10.704 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:55.988835+0000", 2024-09-10T19:24:10.704 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:24:10.704 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:10.704 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:10.704 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-10T19:24:10.704 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:55.988835+0000", 2024-09-10T19:24:10.704 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:24:10.705 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:10.705 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:10.705 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-10T19:24:10.705 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:55.988835+0000", 2024-09-10T19:24:10.705 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4294967295.9999995 2024-09-10T19:24:10.705 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:10.705 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:10.705 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-10T19:24:10.705 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:55.988882+0000", 2024-09-10T19:24:10.705 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.6919000000000003e-05 2024-09-10T19:24:10.705 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:10.705 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:10.705 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-10T19:24:10.706 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:55.988889+0000", 2024-09-10T19:24:10.706 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.7669999999999998e-06 2024-09-10T19:24:10.706 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:10.706 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:10.706 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-10T19:24:10.706 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:55.988892+0000", 2024-09-10T19:24:10.706 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.6430000000000001e-06 2024-09-10T19:24:10.706 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:10.706 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:10.706 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-10T19:24:10.706 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:55.988950+0000", 2024-09-10T19:24:10.706 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.8112000000000002e-05 2024-09-10T19:24:10.706 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:10.706 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:10.707 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for peered", 2024-09-10T19:24:10.707 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:55.988958+0000", 2024-09-10T19:24:10.707 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.7859999999999997e-06 2024-09-10T19:24:10.707 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:10.707 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:10.707 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-10T19:24:10.707 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:57.777269+0000", 2024-09-10T19:24:10.707 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.788310617 2024-09-10T19:24:10.707 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:10.707 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:10.707 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-10T19:24:10.707 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:57.777364+0000", 2024-09-10T19:24:10.707 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.5545999999999996e-05 2024-09-10T19:24:10.707 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:10.708 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:10.708 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 5,7", 2024-09-10T19:24:10.708 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:57.777470+0000", 2024-09-10T19:24:10.708 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000105912 2024-09-10T19:24:10.708 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:10.708 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:10.708 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-10T19:24:10.708 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:57.783922+0000", 2024-09-10T19:24:10.708 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0064513679999999999 2024-09-10T19:24:10.708 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:10.708 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:10.708 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:24:10.709 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:57.795246+0000", 2024-09-10T19:24:10.709 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.011324855999999999 2024-09-10T19:24:10.709 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:10.709 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:10.709 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:24:10.709 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:57.796516+0000", 2024-09-10T19:24:10.709 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0012691950000000001 2024-09-10T19:24:10.709 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:10.709 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:10.709 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-10T19:24:10.709 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:57.796551+0000", 2024-09-10T19:24:10.710 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.4869000000000001e-05 2024-09-10T19:24:10.710 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:10.710 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:10.711 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-10T19:24:10.711 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:57.797372+0000", 2024-09-10T19:24:10.711 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00082099899999999999 2024-09-10T19:24:10.711 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:24:10.711 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:24:10.711 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:24:10.711 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:10.711 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:10.711 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.15051.0:59117 8.3 8:c2a12ec4:::benchmark_data_smithi183_114924_object59116:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e166)", 2024-09-10T19:24:10.711 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-10T19:16:55.988897+0000", 2024-09-10T19:24:10.711 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 434.70281482600001, 2024-09-10T19:24:10.711 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.8096371680000001, 2024-09-10T19:24:10.711 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-10T19:24:10.712 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T19:24:10.712 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-10T19:24:10.712 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.15051", 2024-09-10T19:24:10.712 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.183:0/3632028672", 2024-09-10T19:24:10.712 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 59117 2024-09-10T19:24:10.712 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:10.712 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-10T19:24:10.712 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:10.712 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-10T19:24:10.712 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:55.988897+0000", 2024-09-10T19:24:10.712 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:24:10.712 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:10.713 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:10.713 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-10T19:24:10.713 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:55.988897+0000", 2024-09-10T19:24:10.713 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:24:10.713 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:10.713 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:10.713 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-10T19:24:10.713 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:55.988897+0000", 2024-09-10T19:24:10.713 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4294967295.9999995 2024-09-10T19:24:10.713 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:10.713 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:10.713 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-10T19:24:10.714 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:55.988947+0000", 2024-09-10T19:24:10.714 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.0318000000000002e-05 2024-09-10T19:24:10.714 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:10.714 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:10.714 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-10T19:24:10.714 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:55.988952+0000", 2024-09-10T19:24:10.714 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.1440000000000002e-06 2024-09-10T19:24:10.714 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:10.714 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:10.714 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-10T19:24:10.714 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:55.988955+0000", 2024-09-10T19:24:10.714 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.6850000000000001e-06 2024-09-10T19:24:10.714 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:10.715 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:10.715 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-10T19:24:10.715 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:55.988986+0000", 2024-09-10T19:24:10.715 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.0629000000000001e-05 2024-09-10T19:24:10.715 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:10.715 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:10.715 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for peered", 2024-09-10T19:24:10.715 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:55.988990+0000", 2024-09-10T19:24:10.715 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.2729999999999996e-06 2024-09-10T19:24:10.715 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:10.715 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:10.715 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-10T19:24:10.716 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:57.778622+0000", 2024-09-10T19:24:10.716 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.7896323029999999 2024-09-10T19:24:10.716 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:10.716 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:10.716 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-10T19:24:10.716 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:57.778713+0000", 2024-09-10T19:24:10.716 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.0404999999999999e-05 2024-09-10T19:24:10.716 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:10.716 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:10.716 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 5,7", 2024-09-10T19:24:10.716 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:57.778815+0000", 2024-09-10T19:24:10.716 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000102346 2024-09-10T19:24:10.716 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:10.716 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:10.717 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-10T19:24:10.717 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:57.783939+0000", 2024-09-10T19:24:10.717 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0051238949999999998 2024-09-10T19:24:10.717 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:10.717 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:10.717 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:24:10.717 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:57.796456+0000", 2024-09-10T19:24:10.717 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.012517240000000001 2024-09-10T19:24:10.717 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:10.717 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:10.717 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:24:10.717 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:57.797588+0000", 2024-09-10T19:24:10.717 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0011322820000000001 2024-09-10T19:24:10.718 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:10.718 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:10.718 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-10T19:24:10.718 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:57.797644+0000", 2024-09-10T19:24:10.718 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.6156000000000002e-05 2024-09-10T19:24:10.718 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:10.718 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:10.718 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-10T19:24:10.718 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:57.798535+0000", 2024-09-10T19:24:10.718 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000890024 2024-09-10T19:24:10.718 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:24:10.718 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:24:10.718 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:24:10.719 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:10.719 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:10.719 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.15051.0:59124 8.3 8:cab1e759:::benchmark_data_smithi183_114924_object59123:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e166)", 2024-09-10T19:24:10.719 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-10T19:16:55.988960+0000", 2024-09-10T19:24:10.719 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 434.70275248600001, 2024-09-10T19:24:10.719 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.810490784, 2024-09-10T19:24:10.719 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-10T19:24:10.719 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T19:24:10.719 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-10T19:24:10.719 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.15051", 2024-09-10T19:24:10.719 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.183:0/3632028672", 2024-09-10T19:24:10.719 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 59124 2024-09-10T19:24:10.719 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:10.720 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-10T19:24:10.720 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:10.720 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-10T19:24:10.720 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:55.988960+0000", 2024-09-10T19:24:10.720 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:24:10.720 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:10.720 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:10.720 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-10T19:24:10.720 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:55.988960+0000", 2024-09-10T19:24:10.720 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:24:10.720 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:10.720 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:10.720 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-10T19:24:10.720 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:55.988959+0000", 2024-09-10T19:24:10.721 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4294967295.9999995 2024-09-10T19:24:10.721 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:10.721 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:10.721 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-10T19:24:10.721 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:55.989000+0000", 2024-09-10T19:24:10.721 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.0720000000000003e-05 2024-09-10T19:24:10.721 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:10.721 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:10.721 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-10T19:24:10.721 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:55.989006+0000", 2024-09-10T19:24:10.721 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.5490000000000003e-06 2024-09-10T19:24:10.721 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:10.721 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:10.722 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-10T19:24:10.722 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:55.989010+0000", 2024-09-10T19:24:10.722 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.2140000000000001e-06 2024-09-10T19:24:10.722 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:10.722 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:10.722 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-10T19:24:10.722 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:55.989031+0000", 2024-09-10T19:24:10.722 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.0945999999999999e-05 2024-09-10T19:24:10.722 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:10.722 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:10.722 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for peered", 2024-09-10T19:24:10.722 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:55.989034+0000", 2024-09-10T19:24:10.722 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.1310000000000002e-06 2024-09-10T19:24:10.722 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:10.723 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:10.723 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-10T19:24:10.723 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:57.779721+0000", 2024-09-10T19:24:10.723 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.7906875069999999 2024-09-10T19:24:10.723 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:10.723 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:10.723 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-10T19:24:10.723 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:57.779826+0000", 2024-09-10T19:24:10.723 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000104394 2024-09-10T19:24:10.723 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:10.723 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:10.723 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 5,7", 2024-09-10T19:24:10.723 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:57.779934+0000", 2024-09-10T19:24:10.723 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000108115 2024-09-10T19:24:10.724 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:10.724 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:10.724 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-10T19:24:10.724 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:57.783943+0000", 2024-09-10T19:24:10.724 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0040090689999999997 2024-09-10T19:24:10.724 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:10.724 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:10.724 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:24:10.724 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:57.796489+0000", 2024-09-10T19:24:10.724 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.012546148 2024-09-10T19:24:10.724 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:10.724 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:10.724 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:24:10.724 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:57.798572+0000", 2024-09-10T19:24:10.725 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0020831249999999999 2024-09-10T19:24:10.725 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:10.725 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:10.725 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-10T19:24:10.725 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:57.798604+0000", 2024-09-10T19:24:10.725 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.2206000000000003e-05 2024-09-10T19:24:10.725 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:10.725 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:10.725 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-10T19:24:10.725 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:57.799450+0000", 2024-09-10T19:24:10.725 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00084613600000000005 2024-09-10T19:24:10.725 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:24:10.725 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:24:10.726 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:24:10.726 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:10.726 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:10.726 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.15051.0:59130 8.3 8:c3849915:::benchmark_data_smithi183_114924_object59129:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e166)", 2024-09-10T19:24:10.726 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-10T19:16:55.989014+0000", 2024-09-10T19:24:10.726 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 434.70269771099998, 2024-09-10T19:24:10.726 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.824075119, 2024-09-10T19:24:10.726 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-10T19:24:10.726 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T19:24:10.726 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-10T19:24:10.726 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.15051", 2024-09-10T19:24:10.726 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.183:0/3632028672", 2024-09-10T19:24:10.726 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 59130 2024-09-10T19:24:10.727 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:10.727 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-10T19:24:10.727 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:10.727 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-10T19:24:10.727 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:55.989014+0000", 2024-09-10T19:24:10.727 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:24:10.727 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:10.727 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:10.727 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-10T19:24:10.727 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:55.989014+0000", 2024-09-10T19:24:10.727 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:24:10.727 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:10.727 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:10.728 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-10T19:24:10.728 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:55.989014+0000", 2024-09-10T19:24:10.728 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4294967295.9999995 2024-09-10T19:24:10.728 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:10.728 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:10.728 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-10T19:24:10.728 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:55.989054+0000", 2024-09-10T19:24:10.728 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.0370000000000001e-05 2024-09-10T19:24:10.728 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:10.728 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:10.728 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-10T19:24:10.728 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:55.989061+0000", 2024-09-10T19:24:10.728 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.331e-06 2024-09-10T19:24:10.728 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:10.729 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:10.729 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-10T19:24:10.729 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:55.989064+0000", 2024-09-10T19:24:10.729 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.0199999999999999e-06 2024-09-10T19:24:10.729 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:10.729 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:10.729 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-10T19:24:10.729 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:55.989084+0000", 2024-09-10T19:24:10.729 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.0282999999999999e-05 2024-09-10T19:24:10.729 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:10.729 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:10.729 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for peered", 2024-09-10T19:24:10.729 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:55.989087+0000", 2024-09-10T19:24:10.730 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.7829999999999999e-06 2024-09-10T19:24:10.730 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:10.730 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:10.730 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-10T19:24:10.730 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:57.780815+0000", 2024-09-10T19:24:10.730 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.791727879 2024-09-10T19:24:10.730 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:10.730 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:10.730 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-10T19:24:10.730 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:57.780898+0000", 2024-09-10T19:24:10.730 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.3137e-05 2024-09-10T19:24:10.730 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:10.730 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:10.731 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 5,7", 2024-09-10T19:24:10.731 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:57.781002+0000", 2024-09-10T19:24:10.731 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00010370300000000001 2024-09-10T19:24:10.731 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:10.731 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:10.731 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-10T19:24:10.731 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:57.786090+0000", 2024-09-10T19:24:10.731 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0050885940000000001 2024-09-10T19:24:10.731 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:10.731 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:10.731 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:24:10.731 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:57.797412+0000", 2024-09-10T19:24:10.731 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.011321827 2024-09-10T19:24:10.732 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:10.732 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:10.732 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:24:10.732 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:57.811332+0000", 2024-09-10T19:24:10.732 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.013919617 2024-09-10T19:24:10.732 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:10.732 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:10.732 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-10T19:24:10.732 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:57.811389+0000", 2024-09-10T19:24:10.732 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.6910999999999998e-05 2024-09-10T19:24:10.732 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:10.732 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:10.732 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-10T19:24:10.733 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:57.813090+0000", 2024-09-10T19:24:10.733 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.001701073 2024-09-10T19:24:10.733 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:24:10.733 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:24:10.733 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:24:10.733 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:10.733 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:10.733 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.15051.0:59134 8.3 8:c040b2cd:::benchmark_data_smithi183_114924_object59133:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e166)", 2024-09-10T19:24:10.733 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-10T19:16:55.989069+0000", 2024-09-10T19:24:10.733 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 434.70264358999998, 2024-09-10T19:24:10.733 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.8253441129999999, 2024-09-10T19:24:10.733 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-10T19:24:10.733 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T19:24:10.734 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-10T19:24:10.734 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.15051", 2024-09-10T19:24:10.734 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.183:0/3632028672", 2024-09-10T19:24:10.734 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 59134 2024-09-10T19:24:10.734 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:10.734 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-10T19:24:10.734 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:10.734 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-10T19:24:10.734 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:55.989069+0000", 2024-09-10T19:24:10.734 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:24:10.734 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:10.734 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:10.734 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-10T19:24:10.735 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:55.989069+0000", 2024-09-10T19:24:10.735 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:24:10.735 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:10.735 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:10.735 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-10T19:24:10.735 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:55.989068+0000", 2024-09-10T19:24:10.735 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4294967295.9999995 2024-09-10T19:24:10.735 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:10.735 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:10.735 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-10T19:24:10.735 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:55.989212+0000", 2024-09-10T19:24:10.735 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000144296 2024-09-10T19:24:10.735 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:10.736 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:10.736 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-10T19:24:10.736 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:55.989218+0000", 2024-09-10T19:24:10.736 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.6899999999999997e-06 2024-09-10T19:24:10.736 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:10.736 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:10.736 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-10T19:24:10.736 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:55.989221+0000", 2024-09-10T19:24:10.736 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.6570000000000001e-06 2024-09-10T19:24:10.736 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:10.736 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:10.736 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-10T19:24:10.736 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:55.989240+0000", 2024-09-10T19:24:10.737 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.906e-05 2024-09-10T19:24:10.737 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:10.737 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:10.737 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for peered", 2024-09-10T19:24:10.737 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:55.989243+0000", 2024-09-10T19:24:10.737 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.971e-06 2024-09-10T19:24:10.737 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:10.737 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:10.737 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-10T19:24:10.737 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:57.781893+0000", 2024-09-10T19:24:10.737 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.7926505929999998 2024-09-10T19:24:10.737 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:10.737 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:10.738 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-10T19:24:10.738 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:57.781985+0000", 2024-09-10T19:24:10.738 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.1576999999999994e-05 2024-09-10T19:24:10.738 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:10.738 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:10.738 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 5,7", 2024-09-10T19:24:10.738 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:57.782090+0000", 2024-09-10T19:24:10.738 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000104662 2024-09-10T19:24:10.738 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:10.738 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:10.738 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-10T19:24:10.738 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:57.786095+0000", 2024-09-10T19:24:10.738 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0040054299999999999 2024-09-10T19:24:10.738 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:10.738 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:10.739 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:24:10.739 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:57.797442+0000", 2024-09-10T19:24:10.739 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.011346827 2024-09-10T19:24:10.739 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:10.739 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:10.739 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:24:10.739 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:57.813133+0000", 2024-09-10T19:24:10.739 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.015691360000000001 2024-09-10T19:24:10.739 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:10.739 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:10.739 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-10T19:24:10.739 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:57.813168+0000", 2024-09-10T19:24:10.739 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.4310000000000002e-05 2024-09-10T19:24:10.740 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:10.740 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:10.740 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-10T19:24:10.740 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:57.814413+0000", 2024-09-10T19:24:10.740 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0012451179999999999 2024-09-10T19:24:10.740 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:24:10.740 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:24:10.740 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:24:10.740 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:10.740 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:10.740 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.15051.0:59144 8.3 8:c58dd815:::benchmark_data_smithi183_114924_object59143:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e166)", 2024-09-10T19:24:10.740 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-10T19:16:55.989226+0000", 2024-09-10T19:24:10.740 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 434.70248660200002, 2024-09-10T19:24:10.740 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.8318923599999999, 2024-09-10T19:24:10.741 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-10T19:24:10.741 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T19:24:10.741 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-10T19:24:10.741 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.15051", 2024-09-10T19:24:10.741 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.183:0/3632028672", 2024-09-10T19:24:10.741 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 59144 2024-09-10T19:24:10.741 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:10.741 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-10T19:24:10.741 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:10.741 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-10T19:24:10.741 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:55.989226+0000", 2024-09-10T19:24:10.741 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:24:10.741 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:10.741 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:10.742 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-10T19:24:10.742 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:55.989226+0000", 2024-09-10T19:24:10.742 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:24:10.742 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:10.742 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:10.742 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-10T19:24:10.742 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:55.989225+0000", 2024-09-10T19:24:10.742 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4294967295.9999995 2024-09-10T19:24:10.742 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:10.742 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:10.742 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-10T19:24:10.742 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:55.989264+0000", 2024-09-10T19:24:10.743 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.8593000000000001e-05 2024-09-10T19:24:10.743 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:10.743 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:10.743 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-10T19:24:10.743 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:55.989270+0000", 2024-09-10T19:24:10.743 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.1410000000000002e-06 2024-09-10T19:24:10.743 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:10.743 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:10.743 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-10T19:24:10.743 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:55.989273+0000", 2024-09-10T19:24:10.743 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.0589999999999998e-06 2024-09-10T19:24:10.743 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:10.743 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:10.743 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-10T19:24:10.744 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:55.989294+0000", 2024-09-10T19:24:10.744 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.1162000000000002e-05 2024-09-10T19:24:10.744 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:10.744 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:10.744 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for peered", 2024-09-10T19:24:10.744 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:55.989297+0000", 2024-09-10T19:24:10.744 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.773e-06 2024-09-10T19:24:10.744 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:10.744 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:10.745 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-10T19:24:10.745 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:57.782945+0000", 2024-09-10T19:24:10.745 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.7936485719999999 2024-09-10T19:24:10.745 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:10.745 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:10.745 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-10T19:24:10.745 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:57.783027+0000", 2024-09-10T19:24:10.745 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.1658999999999999e-05 2024-09-10T19:24:10.745 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:10.745 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:10.745 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 5,7", 2024-09-10T19:24:10.745 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:57.783126+0000", 2024-09-10T19:24:10.745 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.9234999999999994e-05 2024-09-10T19:24:10.745 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:10.746 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:10.746 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-10T19:24:10.746 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:57.788338+0000", 2024-09-10T19:24:10.746 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0052119280000000002 2024-09-10T19:24:10.746 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:10.746 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:10.746 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:24:10.746 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:57.797467+0000", 2024-09-10T19:24:10.746 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0091290439999999994 2024-09-10T19:24:10.746 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:10.746 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:10.746 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:24:10.747 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:57.819488+0000", 2024-09-10T19:24:10.747 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.022020649 2024-09-10T19:24:10.747 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:10.747 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:10.747 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-10T19:24:10.747 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:57.819521+0000", 2024-09-10T19:24:10.747 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.3238999999999998e-05 2024-09-10T19:24:10.747 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:10.747 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:10.747 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-10T19:24:10.747 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:57.821118+0000", 2024-09-10T19:24:10.747 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.001596746 2024-09-10T19:24:10.747 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:24:10.747 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:24:10.748 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:24:10.748 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:10.748 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:10.748 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.15051.0:59145 8.3 8:c6076f82:::benchmark_data_smithi183_114924_object59144:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e166)", 2024-09-10T19:24:10.748 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-10T19:16:55.989277+0000", 2024-09-10T19:24:10.748 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 434.702434799, 2024-09-10T19:24:10.748 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.8331499980000001, 2024-09-10T19:24:10.748 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-10T19:24:10.748 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T19:24:10.748 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-10T19:24:10.748 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.15051", 2024-09-10T19:24:10.748 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.183:0/3632028672", 2024-09-10T19:24:10.748 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 59145 2024-09-10T19:24:10.749 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:10.749 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-10T19:24:10.749 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:10.749 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-10T19:24:10.749 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:55.989277+0000", 2024-09-10T19:24:10.749 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:24:10.749 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:10.749 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:10.749 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-10T19:24:10.749 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:55.989277+0000", 2024-09-10T19:24:10.749 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:24:10.749 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:10.749 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:10.750 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-10T19:24:10.750 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:55.989277+0000", 2024-09-10T19:24:10.750 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4294967295.9999995 2024-09-10T19:24:10.750 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:10.750 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:10.750 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-10T19:24:10.750 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:55.989314+0000", 2024-09-10T19:24:10.750 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.6698e-05 2024-09-10T19:24:10.750 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:10.750 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:10.750 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-10T19:24:10.750 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:55.989320+0000", 2024-09-10T19:24:10.750 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.9390000000000002e-06 2024-09-10T19:24:10.751 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:10.751 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:10.751 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-10T19:24:10.751 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:55.989323+0000", 2024-09-10T19:24:10.751 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.8739999999999999e-06 2024-09-10T19:24:10.751 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:10.751 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:10.751 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-10T19:24:10.751 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:55.989345+0000", 2024-09-10T19:24:10.751 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.2449000000000001e-05 2024-09-10T19:24:10.751 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:10.751 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:10.751 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for peered", 2024-09-10T19:24:10.751 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:55.989348+0000", 2024-09-10T19:24:10.752 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.9529999999999999e-06 2024-09-10T19:24:10.752 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:10.752 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:10.752 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-10T19:24:10.752 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:57.783970+0000", 2024-09-10T19:24:10.752 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.7946222979999999 2024-09-10T19:24:10.752 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:10.752 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:10.752 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-10T19:24:10.752 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:57.784051+0000", 2024-09-10T19:24:10.752 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.1018000000000003e-05 2024-09-10T19:24:10.752 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:10.752 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:10.753 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 5,7", 2024-09-10T19:24:10.753 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:57.784150+0000", 2024-09-10T19:24:10.753 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.8586999999999996e-05 2024-09-10T19:24:10.753 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:10.753 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:10.753 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-10T19:24:10.753 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:57.788350+0000", 2024-09-10T19:24:10.753 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0042002710000000002 2024-09-10T19:24:10.753 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:10.753 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:10.753 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:24:10.753 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:57.797493+0000", 2024-09-10T19:24:10.753 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0091431740000000004 2024-09-10T19:24:10.753 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:10.754 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:10.754 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:24:10.754 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:57.821161+0000", 2024-09-10T19:24:10.754 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.023668084999999998 2024-09-10T19:24:10.754 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:10.754 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:10.754 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-10T19:24:10.754 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:57.821198+0000", 2024-09-10T19:24:10.754 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.6758999999999998e-05 2024-09-10T19:24:10.754 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:10.754 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:10.754 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-10T19:24:10.754 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:57.822427+0000", 2024-09-10T19:24:10.755 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0012292729999999999 2024-09-10T19:24:10.755 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:24:10.755 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:24:10.755 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:24:10.755 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:10.755 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:10.755 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.15051.0:59151 8.3 8:cbebd424:::benchmark_data_smithi183_114924_object59150:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e166)", 2024-09-10T19:24:10.755 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-10T19:16:55.989329+0000", 2024-09-10T19:24:10.755 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 434.70238300900002, 2024-09-10T19:24:10.755 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.8345737230000001, 2024-09-10T19:24:10.755 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-10T19:24:10.755 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T19:24:10.755 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-10T19:24:10.756 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.15051", 2024-09-10T19:24:10.756 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.183:0/3632028672", 2024-09-10T19:24:10.756 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 59151 2024-09-10T19:24:10.756 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:10.756 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-10T19:24:10.756 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:10.756 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-10T19:24:10.756 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:55.989329+0000", 2024-09-10T19:24:10.756 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:24:10.756 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:10.756 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:10.756 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-10T19:24:10.756 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:55.989329+0000", 2024-09-10T19:24:10.757 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:24:10.757 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:10.757 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:10.757 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-10T19:24:10.757 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:55.989329+0000", 2024-09-10T19:24:10.757 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4294967295.9999995 2024-09-10T19:24:10.757 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:10.757 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:10.757 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-10T19:24:10.757 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:55.989364+0000", 2024-09-10T19:24:10.757 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.5280000000000001e-05 2024-09-10T19:24:10.757 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:10.757 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:10.758 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-10T19:24:10.758 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:55.989370+0000", 2024-09-10T19:24:10.758 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.852e-06 2024-09-10T19:24:10.758 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:10.758 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:10.758 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-10T19:24:10.758 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:55.989373+0000", 2024-09-10T19:24:10.758 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.9349999999999999e-06 2024-09-10T19:24:10.758 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:10.758 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:10.758 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-10T19:24:10.758 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:55.989390+0000", 2024-09-10T19:24:10.758 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.7632e-05 2024-09-10T19:24:10.758 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:10.759 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:10.759 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for peered", 2024-09-10T19:24:10.759 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:55.989393+0000", 2024-09-10T19:24:10.759 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.7719999999999999e-06 2024-09-10T19:24:10.759 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:10.759 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:10.759 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-10T19:24:10.759 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:57.784999+0000", 2024-09-10T19:24:10.759 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.795605976 2024-09-10T19:24:10.759 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:10.759 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:10.759 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-10T19:24:10.759 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:57.785091+0000", 2024-09-10T19:24:10.759 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.1812999999999997e-05 2024-09-10T19:24:10.760 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:10.760 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:10.760 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 5,7", 2024-09-10T19:24:10.760 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:57.785185+0000", 2024-09-10T19:24:10.760 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.4143999999999999e-05 2024-09-10T19:24:10.760 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:10.760 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:10.760 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-10T19:24:10.760 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:57.792624+0000", 2024-09-10T19:24:10.760 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0074391149999999996 2024-09-10T19:24:10.760 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:10.760 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:10.760 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:24:10.761 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:57.799494+0000", 2024-09-10T19:24:10.761 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0068695969999999999 2024-09-10T19:24:10.761 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:10.761 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:10.761 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:24:10.761 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:57.822835+0000", 2024-09-10T19:24:10.761 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.023340831999999999 2024-09-10T19:24:10.761 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:10.761 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:10.761 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-10T19:24:10.761 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:57.822888+0000", 2024-09-10T19:24:10.761 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.3263999999999997e-05 2024-09-10T19:24:10.761 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:10.761 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:10.762 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-10T19:24:10.762 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:57.823903+0000", 2024-09-10T19:24:10.762 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0010149309999999999 2024-09-10T19:24:10.762 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:24:10.762 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:24:10.762 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:24:10.762 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:10.762 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:10.762 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.15051.0:59154 8.3 8:c7c823db:::benchmark_data_smithi183_114924_object59153:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e166)", 2024-09-10T19:24:10.762 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-10T19:16:55.989377+0000", 2024-09-10T19:24:10.762 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 434.70233483700002, 2024-09-10T19:24:10.762 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.835555351, 2024-09-10T19:24:10.762 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-10T19:24:10.762 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T19:24:10.763 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-10T19:24:10.763 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.15051", 2024-09-10T19:24:10.763 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.183:0/3632028672", 2024-09-10T19:24:10.763 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 59154 2024-09-10T19:24:10.763 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:10.763 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-10T19:24:10.763 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:10.763 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-10T19:24:10.763 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:55.989377+0000", 2024-09-10T19:24:10.763 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:24:10.763 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:10.763 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:10.763 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-10T19:24:10.764 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:55.989377+0000", 2024-09-10T19:24:10.764 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:24:10.764 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:10.764 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:10.764 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-10T19:24:10.764 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:55.989377+0000", 2024-09-10T19:24:10.764 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4294967295.9999995 2024-09-10T19:24:10.764 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:10.764 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:10.764 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-10T19:24:10.764 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:55.989413+0000", 2024-09-10T19:24:10.764 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.5738000000000002e-05 2024-09-10T19:24:10.764 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:10.764 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:10.765 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-10T19:24:10.765 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:55.989419+0000", 2024-09-10T19:24:10.765 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.9000000000000003e-06 2024-09-10T19:24:10.765 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:10.765 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:10.765 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-10T19:24:10.765 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:55.989421+0000", 2024-09-10T19:24:10.765 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.8899999999999999e-06 2024-09-10T19:24:10.765 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:10.765 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:10.765 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-10T19:24:10.765 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:55.989443+0000", 2024-09-10T19:24:10.765 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.1097000000000001e-05 2024-09-10T19:24:10.765 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:10.766 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:10.766 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for peered", 2024-09-10T19:24:10.766 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:55.989446+0000", 2024-09-10T19:24:10.766 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.943e-06 2024-09-10T19:24:10.766 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:10.766 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:10.766 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-10T19:24:10.766 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:57.786124+0000", 2024-09-10T19:24:10.766 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.796678357 2024-09-10T19:24:10.766 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:10.766 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:10.766 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-10T19:24:10.766 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:57.786202+0000", 2024-09-10T19:24:10.766 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.8150000000000002e-05 2024-09-10T19:24:10.767 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:10.767 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:10.767 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 5,7", 2024-09-10T19:24:10.767 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:57.786294+0000", 2024-09-10T19:24:10.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:24:10.769 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:24:10.769 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.1912999999999999e-05 2024-09-10T19:24:10.769 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:10.769 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:10.769 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-10T19:24:10.769 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:57.792631+0000", 2024-09-10T19:24:10.769 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.006337484 2024-09-10T19:24:10.770 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:10.770 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:10.770 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:24:10.770 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:57.799517+0000", 2024-09-10T19:24:10.770 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0068852369999999998 2024-09-10T19:24:10.770 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:10.770 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:10.770 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:24:10.770 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:57.823952+0000", 2024-09-10T19:24:10.770 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.024435651999999999 2024-09-10T19:24:10.770 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:10.770 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:10.771 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-10T19:24:10.771 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:57.823986+0000", 2024-09-10T19:24:10.771 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.3439000000000003e-05 2024-09-10T19:24:10.771 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:10.771 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:10.771 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-10T19:24:10.771 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:57.824933+0000", 2024-09-10T19:24:10.771 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00094693100000000003 2024-09-10T19:24:10.771 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:24:10.771 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:24:10.771 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:24:10.771 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:10.771 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:10.772 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.15051.0:59158 8.3 8:cc8333b1:::benchmark_data_smithi183_114924_object59157:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e166)", 2024-09-10T19:24:10.772 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-10T19:16:55.989426+0000", 2024-09-10T19:24:10.772 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 434.70228616999998, 2024-09-10T19:24:10.772 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.836481646, 2024-09-10T19:24:10.772 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-10T19:24:10.772 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T19:24:10.772 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-10T19:24:10.772 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.15051", 2024-09-10T19:24:10.772 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.183:0/3632028672", 2024-09-10T19:24:10.772 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 59158 2024-09-10T19:24:10.772 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:10.772 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-10T19:24:10.772 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:10.773 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-10T19:24:10.773 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:55.989426+0000", 2024-09-10T19:24:10.773 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:24:10.773 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:10.773 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:10.773 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-10T19:24:10.773 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:55.989426+0000", 2024-09-10T19:24:10.773 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:24:10.773 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:10.773 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:10.773 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-10T19:24:10.773 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:55.989426+0000", 2024-09-10T19:24:10.773 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4294967295.9999995 2024-09-10T19:24:10.774 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:10.774 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:10.774 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-10T19:24:10.774 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:55.989462+0000", 2024-09-10T19:24:10.774 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.6878999999999997e-05 2024-09-10T19:24:10.774 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:10.774 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:10.774 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-10T19:24:10.774 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:55.989468+0000", 2024-09-10T19:24:10.774 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.9710000000000001e-06 2024-09-10T19:24:10.774 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:10.774 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:10.774 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-10T19:24:10.775 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:55.989471+0000", 2024-09-10T19:24:10.775 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.8909999999999999e-06 2024-09-10T19:24:10.775 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:10.775 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:10.775 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-10T19:24:10.775 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:55.989490+0000", 2024-09-10T19:24:10.775 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.9157e-05 2024-09-10T19:24:10.775 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:10.775 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:10.775 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for peered", 2024-09-10T19:24:10.775 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:55.989493+0000", 2024-09-10T19:24:10.775 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.6520000000000002e-06 2024-09-10T19:24:10.775 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:10.775 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:10.776 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-10T19:24:10.776 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:57.787166+0000", 2024-09-10T19:24:10.776 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.7976729169999999 2024-09-10T19:24:10.776 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:10.776 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:10.776 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-10T19:24:10.776 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:57.787251+0000", 2024-09-10T19:24:10.776 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.4837000000000001e-05 2024-09-10T19:24:10.776 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:10.776 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:10.776 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 5,7", 2024-09-10T19:24:10.776 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:57.787347+0000", 2024-09-10T19:24:10.776 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.6084999999999999e-05 2024-09-10T19:24:10.776 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:10.777 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:10.777 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-10T19:24:10.777 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:57.793693+0000", 2024-09-10T19:24:10.777 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0063457749999999997 2024-09-10T19:24:10.777 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:10.777 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:10.777 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:24:10.777 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:57.805750+0000", 2024-09-10T19:24:10.777 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.012057615000000001 2024-09-10T19:24:10.777 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:10.777 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:10.777 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:24:10.777 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:57.824981+0000", 2024-09-10T19:24:10.778 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.019230860999999998 2024-09-10T19:24:10.778 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:10.778 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:10.778 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-10T19:24:10.778 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:57.825014+0000", 2024-09-10T19:24:10.778 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.2435999999999999e-05 2024-09-10T19:24:10.778 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:10.779 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:10.779 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-10T19:24:10.780 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:57.825908+0000", 2024-09-10T19:24:10.780 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00089398400000000001 2024-09-10T19:24:10.780 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:24:10.780 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:24:10.780 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:24:10.780 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:10.780 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:10.780 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.15051.0:59203 8.3 8:ccbf45cb:::benchmark_data_smithi183_114924_object59202:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e166)", 2024-09-10T19:24:10.780 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-10T19:16:55.989476+0000", 2024-09-10T19:24:10.780 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 434.70223599299999, 2024-09-10T19:24:10.780 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.8425899960000001, 2024-09-10T19:24:10.780 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-10T19:24:10.780 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T19:24:10.781 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-10T19:24:10.781 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.15051", 2024-09-10T19:24:10.781 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.183:0/3632028672", 2024-09-10T19:24:10.781 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 59203 2024-09-10T19:24:10.781 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:10.781 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-10T19:24:10.781 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:10.781 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-10T19:24:10.781 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:55.989476+0000", 2024-09-10T19:24:10.781 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:24:10.781 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:10.781 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:10.781 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-10T19:24:10.781 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:55.989476+0000", 2024-09-10T19:24:10.782 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:24:10.782 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:10.782 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:10.782 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-10T19:24:10.782 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:55.989476+0000", 2024-09-10T19:24:10.782 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4294967295.9999995 2024-09-10T19:24:10.782 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:10.782 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:10.782 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-10T19:24:10.782 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:55.989511+0000", 2024-09-10T19:24:10.782 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.5590000000000003e-05 2024-09-10T19:24:10.782 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:10.782 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:10.783 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-10T19:24:10.783 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:55.989517+0000", 2024-09-10T19:24:10.783 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.913e-06 2024-09-10T19:24:10.783 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:10.783 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:10.783 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-10T19:24:10.783 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:55.989520+0000", 2024-09-10T19:24:10.783 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.0419999999999998e-06 2024-09-10T19:24:10.783 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:10.783 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:10.783 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-10T19:24:10.783 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:55.989539+0000", 2024-09-10T19:24:10.783 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.8624999999999999e-05 2024-09-10T19:24:10.783 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:10.784 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:10.784 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for peered", 2024-09-10T19:24:10.784 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:55.989542+0000", 2024-09-10T19:24:10.784 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.7250000000000002e-06 2024-09-10T19:24:10.784 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:10.784 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:10.784 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-10T19:24:10.784 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:57.788382+0000", 2024-09-10T19:24:10.784 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.7988407230000001 2024-09-10T19:24:10.784 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:10.784 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:10.784 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-10T19:24:10.784 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:57.788461+0000", 2024-09-10T19:24:10.785 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.8881000000000006e-05 2024-09-10T19:24:10.785 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:10.785 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:10.785 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 5,7", 2024-09-10T19:24:10.785 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:57.788561+0000", 2024-09-10T19:24:10.785 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.9351999999999993e-05 2024-09-10T19:24:10.785 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:10.785 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:10.785 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-10T19:24:10.785 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:57.793697+0000", 2024-09-10T19:24:10.785 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.005136369 2024-09-10T19:24:10.785 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:10.785 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:10.786 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:24:10.786 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:57.805782+0000", 2024-09-10T19:24:10.786 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.012085043 2024-09-10T19:24:10.786 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:10.786 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:10.786 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:24:10.786 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:57.830391+0000", 2024-09-10T19:24:10.786 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.024609273000000001 2024-09-10T19:24:10.786 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:10.786 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:10.786 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-10T19:24:10.786 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:57.830424+0000", 2024-09-10T19:24:10.786 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.2657999999999998e-05 2024-09-10T19:24:10.787 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:10.787 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:10.787 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-10T19:24:10.787 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:57.832066+0000", 2024-09-10T19:24:10.787 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.001642225 2024-09-10T19:24:10.787 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:24:10.787 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:24:10.787 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:24:10.787 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:10.787 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:10.787 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.15051.0:59228 8.3 8:ce38d1dc:::benchmark_data_smithi183_114924_object59227:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e166)", 2024-09-10T19:24:10.787 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-10T19:16:55.989525+0000", 2024-09-10T19:24:10.787 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 434.70218737599998, 2024-09-10T19:24:10.788 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.8572049580000001, 2024-09-10T19:24:10.788 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-10T19:24:10.788 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T19:24:10.788 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-10T19:24:10.788 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.15051", 2024-09-10T19:24:10.788 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.183:0/3632028672", 2024-09-10T19:24:10.788 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 59228 2024-09-10T19:24:10.788 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:10.788 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-10T19:24:10.788 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:10.788 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-10T19:24:10.788 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:55.989525+0000", 2024-09-10T19:24:10.788 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:24:10.789 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:10.789 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:10.789 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-10T19:24:10.789 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:55.989525+0000", 2024-09-10T19:24:10.789 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:24:10.789 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:10.789 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:10.789 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-10T19:24:10.789 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:55.989524+0000", 2024-09-10T19:24:10.789 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4294967295.9999995 2024-09-10T19:24:10.789 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:10.789 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:10.789 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-10T19:24:10.789 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:55.989559+0000", 2024-09-10T19:24:10.790 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.4261000000000003e-05 2024-09-10T19:24:10.790 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:10.790 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:10.790 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-10T19:24:10.790 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:55.989565+0000", 2024-09-10T19:24:10.790 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.8470000000000001e-06 2024-09-10T19:24:10.790 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:10.790 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:10.790 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-10T19:24:10.790 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:55.989567+0000", 2024-09-10T19:24:10.790 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.3630000000000001e-06 2024-09-10T19:24:10.790 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:10.790 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:10.791 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-10T19:24:10.791 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:55.989586+0000", 2024-09-10T19:24:10.791 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.9576e-05 2024-09-10T19:24:10.791 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:10.791 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:10.791 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for peered", 2024-09-10T19:24:10.791 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:55.989589+0000", 2024-09-10T19:24:10.791 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.8679999999999999e-06 2024-09-10T19:24:10.791 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:10.791 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:10.791 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-10T19:24:10.791 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:57.789407+0000", 2024-09-10T19:24:10.791 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.7998178980000001 2024-09-10T19:24:10.791 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:10.792 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:10.792 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-10T19:24:10.792 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:57.789485+0000", 2024-09-10T19:24:10.792 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.8109999999999996e-05 2024-09-10T19:24:10.792 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:10.792 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:10.792 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 5,7", 2024-09-10T19:24:10.792 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:57.789580+0000", 2024-09-10T19:24:10.792 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.4176000000000001e-05 2024-09-10T19:24:10.792 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:10.792 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:10.792 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-10T19:24:10.792 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:57.793702+0000", 2024-09-10T19:24:10.793 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0041221390000000004 2024-09-10T19:24:10.793 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:10.793 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:10.793 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:24:10.793 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:57.805830+0000", 2024-09-10T19:24:10.793 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.012128144 2024-09-10T19:24:10.793 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:10.793 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:10.793 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:24:10.793 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:57.845000+0000", 2024-09-10T19:24:10.793 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.039170071000000001 2024-09-10T19:24:10.793 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:10.793 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:10.793 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-10T19:24:10.794 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:57.845039+0000", 2024-09-10T19:24:10.794 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.8940000000000003e-05 2024-09-10T19:24:10.794 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:10.794 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:10.794 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-10T19:24:10.794 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:57.846730+0000", 2024-09-10T19:24:10.794 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.001690963 2024-09-10T19:24:10.794 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:24:10.794 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:24:10.794 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:24:10.794 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:10.794 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:10.794 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.15051.0:59235 8.3 8:c3e38beb:::benchmark_data_smithi183_114924_object59234:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e166)", 2024-09-10T19:24:10.794 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-10T19:16:55.989571+0000", 2024-09-10T19:24:10.795 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 434.70214085399999, 2024-09-10T19:24:10.795 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.8656851109999999, 2024-09-10T19:24:10.795 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-10T19:24:10.795 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T19:24:10.795 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-10T19:24:10.795 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.15051", 2024-09-10T19:24:10.795 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.183:0/3632028672", 2024-09-10T19:24:10.795 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 59235 2024-09-10T19:24:10.795 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:10.795 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-10T19:24:10.795 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:10.795 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-10T19:24:10.795 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:55.989571+0000", 2024-09-10T19:24:10.796 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:24:10.796 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:10.796 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:10.796 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-10T19:24:10.796 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:55.989571+0000", 2024-09-10T19:24:10.796 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:24:10.796 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:10.796 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:10.796 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-10T19:24:10.796 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:55.989571+0000", 2024-09-10T19:24:10.796 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4294967295.9999995 2024-09-10T19:24:10.796 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:10.796 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:10.797 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-10T19:24:10.797 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:55.989607+0000", 2024-09-10T19:24:10.797 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.5815000000000002e-05 2024-09-10T19:24:10.797 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:10.797 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:10.797 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-10T19:24:10.797 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:55.989613+0000", 2024-09-10T19:24:10.797 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.7860000000000002e-06 2024-09-10T19:24:10.797 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:10.797 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:10.797 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-10T19:24:10.797 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:55.989616+0000", 2024-09-10T19:24:10.798 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.9550000000000001e-06 2024-09-10T19:24:10.798 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:10.798 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:10.798 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-10T19:24:10.798 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:55.989637+0000", 2024-09-10T19:24:10.798 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.1443000000000001e-05 2024-09-10T19:24:10.798 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:10.798 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:10.798 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for peered", 2024-09-10T19:24:10.798 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:55.989640+0000", 2024-09-10T19:24:10.798 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.807e-06 2024-09-10T19:24:10.798 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:10.799 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:10.799 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-10T19:24:10.799 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:57.790482+0000", 2024-09-10T19:24:10.799 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.800842193 2024-09-10T19:24:10.799 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:10.799 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:10.799 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-10T19:24:10.799 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:57.790563+0000", 2024-09-10T19:24:10.799 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.0902999999999994e-05 2024-09-10T19:24:10.799 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:10.799 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:10.799 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 5,7", 2024-09-10T19:24:10.799 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:57.790664+0000", 2024-09-10T19:24:10.799 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000101628 2024-09-10T19:24:10.800 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:10.800 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:10.800 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-10T19:24:10.800 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:57.793706+0000", 2024-09-10T19:24:10.800 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0030410429999999998 2024-09-10T19:24:10.800 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:10.800 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:10.800 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:24:10.800 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:57.805856+0000", 2024-09-10T19:24:10.800 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.012150213999999999 2024-09-10T19:24:10.800 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:10.800 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:10.800 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:24:10.800 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:57.853631+0000", 2024-09-10T19:24:10.801 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.047774900000000002 2024-09-10T19:24:10.801 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:10.801 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:10.801 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-10T19:24:10.801 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:57.853662+0000", 2024-09-10T19:24:10.801 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.1511999999999999e-05 2024-09-10T19:24:10.801 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:10.801 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:10.801 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-10T19:24:10.801 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:57.855256+0000", 2024-09-10T19:24:10.801 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0015942809999999999 2024-09-10T19:24:10.801 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:24:10.801 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:24:10.801 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:24:10.802 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:10.802 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:10.802 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.15051.0:59242 8.3 8:c1483ce1:::benchmark_data_smithi183_114924_object59241:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e166)", 2024-09-10T19:24:10.802 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-10T19:16:55.989620+0000", 2024-09-10T19:24:10.802 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 434.70209222300002, 2024-09-10T19:24:10.802 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.8671676669999999, 2024-09-10T19:24:10.802 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-10T19:24:10.802 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T19:24:10.802 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-10T19:24:10.803 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.15051", 2024-09-10T19:24:10.803 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.183:0/3632028672", 2024-09-10T19:24:10.803 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 59242 2024-09-10T19:24:10.803 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:10.803 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-10T19:24:10.803 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:10.803 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-10T19:24:10.803 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:55.989620+0000", 2024-09-10T19:24:10.803 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:24:10.803 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:10.803 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:10.803 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-10T19:24:10.804 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:55.989620+0000", 2024-09-10T19:24:10.804 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:24:10.804 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:10.804 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:10.804 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-10T19:24:10.804 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:55.989620+0000", 2024-09-10T19:24:10.804 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4294967295.9999995 2024-09-10T19:24:10.804 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:10.804 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:10.804 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-10T19:24:10.804 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:55.989656+0000", 2024-09-10T19:24:10.804 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.6198999999999997e-05 2024-09-10T19:24:10.805 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:10.805 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:10.805 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-10T19:24:10.805 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:55.989662+0000", 2024-09-10T19:24:10.805 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.976e-06 2024-09-10T19:24:10.805 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:10.805 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:10.805 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-10T19:24:10.805 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:55.989665+0000", 2024-09-10T19:24:10.805 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.9129999999999999e-06 2024-09-10T19:24:10.805 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:10.805 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:10.805 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-10T19:24:10.806 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:55.989684+0000", 2024-09-10T19:24:10.806 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.9235000000000001e-05 2024-09-10T19:24:10.806 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:10.806 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:10.806 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for peered", 2024-09-10T19:24:10.806 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:55.989687+0000", 2024-09-10T19:24:10.806 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.8109999999999999e-06 2024-09-10T19:24:10.806 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:10.806 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:10.806 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-10T19:24:10.806 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:57.791617+0000", 2024-09-10T19:24:10.806 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.801930668 2024-09-10T19:24:10.806 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:10.807 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:10.807 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-10T19:24:10.807 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:57.791701+0000", 2024-09-10T19:24:10.807 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.3627000000000004e-05 2024-09-10T19:24:10.807 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:10.807 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:10.807 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 5,7", 2024-09-10T19:24:10.807 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:57.791802+0000", 2024-09-10T19:24:10.807 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000100942 2024-09-10T19:24:10.807 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:10.807 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:10.807 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-10T19:24:10.807 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:57.795215+0000", 2024-09-10T19:24:10.807 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0034128309999999999 2024-09-10T19:24:10.808 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:10.808 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:10.808 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:24:10.808 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:57.805876+0000", 2024-09-10T19:24:10.808 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.010660849 2024-09-10T19:24:10.808 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:10.808 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:10.808 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:24:10.808 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:57.855315+0000", 2024-09-10T19:24:10.808 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.049439759 2024-09-10T19:24:10.808 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:10.808 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:10.808 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-10T19:24:10.809 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:57.855371+0000", 2024-09-10T19:24:10.809 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.5946999999999998e-05 2024-09-10T19:24:10.809 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:10.809 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:10.809 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-10T19:24:10.809 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:57.856788+0000", 2024-09-10T19:24:10.809 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.001416311 2024-09-10T19:24:10.809 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:24:10.809 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:24:10.809 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:24:10.809 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:10.809 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:10.810 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.15051.0:59243 8.3 8:c806a870:::benchmark_data_smithi183_114924_object59242:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e166)", 2024-09-10T19:24:10.810 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-10T19:16:55.989669+0000", 2024-09-10T19:24:10.810 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 434.70204309100001, 2024-09-10T19:24:10.810 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.8685301810000001, 2024-09-10T19:24:10.810 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-10T19:24:10.810 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T19:24:10.810 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-10T19:24:10.810 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.15051", 2024-09-10T19:24:10.810 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.183:0/3632028672", 2024-09-10T19:24:10.810 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 59243 2024-09-10T19:24:10.810 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:10.810 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-10T19:24:10.811 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:10.811 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-10T19:24:10.811 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:55.989669+0000", 2024-09-10T19:24:10.811 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:24:10.811 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:10.811 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:10.811 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-10T19:24:10.811 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:55.989669+0000", 2024-09-10T19:24:10.811 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:24:10.811 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:10.811 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:10.811 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-10T19:24:10.812 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:55.989669+0000", 2024-09-10T19:24:10.812 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4294967295.9999995 2024-09-10T19:24:10.812 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:10.812 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:10.812 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-10T19:24:10.812 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:55.989705+0000", 2024-09-10T19:24:10.812 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.6089e-05 2024-09-10T19:24:10.812 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:10.812 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:10.812 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-10T19:24:10.812 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:55.989711+0000", 2024-09-10T19:24:10.813 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.1940000000000003e-06 2024-09-10T19:24:10.813 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:10.813 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:10.813 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-10T19:24:10.813 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:55.989714+0000", 2024-09-10T19:24:10.813 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.7690000000000001e-06 2024-09-10T19:24:10.813 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:10.813 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:10.813 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-10T19:24:10.813 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:55.989732+0000", 2024-09-10T19:24:10.813 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.8210999999999999e-05 2024-09-10T19:24:10.813 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:10.813 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:10.813 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for peered", 2024-09-10T19:24:10.814 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:55.989735+0000", 2024-09-10T19:24:10.814 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.8700000000000001e-06 2024-09-10T19:24:10.814 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:10.814 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:10.814 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-10T19:24:10.814 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:57.792658+0000", 2024-09-10T19:24:10.814 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.8029229409999998 2024-09-10T19:24:10.814 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:10.814 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:10.814 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-10T19:24:10.814 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:57.792735+0000", 2024-09-10T19:24:10.814 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.6919e-05 2024-09-10T19:24:10.814 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:10.815 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:10.815 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 5,7", 2024-09-10T19:24:10.815 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:57.792835+0000", 2024-09-10T19:24:10.815 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000100052 2024-09-10T19:24:10.815 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:10.815 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:10.815 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-10T19:24:10.815 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:57.796414+0000", 2024-09-10T19:24:10.815 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0035788740000000001 2024-09-10T19:24:10.815 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:10.815 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:10.815 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:24:10.815 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:57.808648+0000", 2024-09-10T19:24:10.816 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.012234134000000001 2024-09-10T19:24:10.816 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:10.816 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:10.816 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:24:10.816 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:57.856842+0000", 2024-09-10T19:24:10.816 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.048194242999999998 2024-09-10T19:24:10.816 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:10.816 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:10.816 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-10T19:24:10.816 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:57.856898+0000", 2024-09-10T19:24:10.816 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.6252000000000003e-05 2024-09-10T19:24:10.816 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:10.817 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:10.817 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-10T19:24:10.817 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:16:57.858199+0000", 2024-09-10T19:24:10.817 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0013010109999999999 2024-09-10T19:24:10.817 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:24:10.817 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:24:10.817 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:24:10.817 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:10.817 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:10.817 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.15051.0:93184 8.13 8:cfff3a3d:::benchmark_data_smithi183_114924_object93183:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e180)", 2024-09-10T19:24:10.817 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-10T19:17:28.948455+0000", 2024-09-10T19:24:10.817 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 401.743257589, 2024-09-10T19:24:10.817 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.1285055399999999, 2024-09-10T19:24:10.818 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-10T19:24:10.818 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T19:24:10.818 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-10T19:24:10.818 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.15051", 2024-09-10T19:24:10.818 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.183:0/3632028672", 2024-09-10T19:24:10.818 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 93184 2024-09-10T19:24:10.818 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:10.818 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-10T19:24:10.818 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:10.818 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-10T19:24:10.818 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:17:28.948455+0000", 2024-09-10T19:24:10.818 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:24:10.818 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:10.819 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:10.819 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-10T19:24:10.819 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:17:28.948455+0000", 2024-09-10T19:24:10.819 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:24:10.819 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:10.819 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:10.819 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-10T19:24:10.819 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:17:28.948453+0000", 2024-09-10T19:24:10.819 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4294967295.9999986 2024-09-10T19:24:10.819 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:10.819 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:10.819 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-10T19:24:10.819 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:17:28.948488+0000", 2024-09-10T19:24:10.820 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.4739000000000001e-05 2024-09-10T19:24:10.820 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:10.820 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:10.820 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-10T19:24:10.820 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:17:28.948500+0000", 2024-09-10T19:24:10.820 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.1946999999999999e-05 2024-09-10T19:24:10.820 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:10.820 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:10.820 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-10T19:24:10.820 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:17:28.948506+0000", 2024-09-10T19:24:10.820 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.4400000000000002e-06 2024-09-10T19:24:10.820 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:10.821 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:10.821 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-10T19:24:10.821 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:17:28.950088+0000", 2024-09-10T19:24:10.821 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.001581671 2024-09-10T19:24:10.821 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:10.821 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:10.821 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for peered", 2024-09-10T19:24:10.821 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:17:28.950096+0000", 2024-09-10T19:24:10.821 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.8460000000000008e-06 2024-09-10T19:24:10.821 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:10.821 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:10.821 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-10T19:24:10.821 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:17:30.046045+0000", 2024-09-10T19:24:10.822 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.095949455 2024-09-10T19:24:10.822 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:10.822 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:10.822 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-10T19:24:10.822 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:17:30.046165+0000", 2024-09-10T19:24:10.822 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00011940800000000001 2024-09-10T19:24:10.822 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:10.822 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:10.822 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 4,5,7", 2024-09-10T19:24:10.822 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:17:30.046404+0000", 2024-09-10T19:24:10.822 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000239075 2024-09-10T19:24:10.822 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:10.823 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:10.823 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-10T19:24:10.823 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:17:30.053276+0000", 2024-09-10T19:24:10.823 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0068720839999999997 2024-09-10T19:24:10.823 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:10.823 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:10.823 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:24:10.823 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:17:30.056804+0000", 2024-09-10T19:24:10.823 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0035281380000000001 2024-09-10T19:24:10.823 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:10.823 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:10.823 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:24:10.824 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:17:30.057585+0000", 2024-09-10T19:24:10.824 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000781385 2024-09-10T19:24:10.824 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:10.824 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:10.824 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:24:10.824 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:17:30.075618+0000", 2024-09-10T19:24:10.824 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.018033098000000001 2024-09-10T19:24:10.824 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:10.824 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:10.824 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-10T19:24:10.824 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:17:30.075655+0000", 2024-09-10T19:24:10.824 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.6934999999999998e-05 2024-09-10T19:24:10.824 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:10.825 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:10.825 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-10T19:24:10.825 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:17:30.076960+0000", 2024-09-10T19:24:10.825 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.001304838 2024-09-10T19:24:10.825 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:24:10.825 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:24:10.825 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:24:10.825 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:10.825 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:10.825 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.15051.0:93229 8.13 8:ca7bbcb1:::benchmark_data_smithi183_114924_object93228:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e180)", 2024-09-10T19:24:10.825 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-10T19:17:28.948515+0000", 2024-09-10T19:24:10.825 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 401.74319705200003, 2024-09-10T19:24:10.825 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.1299140519999999, 2024-09-10T19:24:10.825 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-10T19:24:10.826 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T19:24:10.826 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-10T19:24:10.826 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.15051", 2024-09-10T19:24:10.826 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.183:0/3632028672", 2024-09-10T19:24:10.826 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 93229 2024-09-10T19:24:10.826 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:10.826 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-10T19:24:10.826 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:10.826 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-10T19:24:10.826 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:17:28.948515+0000", 2024-09-10T19:24:10.826 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:24:10.826 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:10.826 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:10.827 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-10T19:24:10.827 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:17:28.948515+0000", 2024-09-10T19:24:10.827 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:24:10.827 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:10.827 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:10.827 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-10T19:24:10.827 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:17:28.948515+0000", 2024-09-10T19:24:10.827 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4294967295.9999995 2024-09-10T19:24:10.827 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:10.827 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:10.827 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-10T19:24:10.827 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:17:28.948535+0000", 2024-09-10T19:24:10.827 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.0781999999999999e-05 2024-09-10T19:24:10.828 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:10.828 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:10.828 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-10T19:24:10.828 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:17:28.948543+0000", 2024-09-10T19:24:10.828 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.3610000000000003e-06 2024-09-10T19:24:10.828 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:10.828 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:10.828 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-10T19:24:10.828 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:17:28.948547+0000", 2024-09-10T19:24:10.828 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.2799999999999997e-06 2024-09-10T19:24:10.828 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:10.828 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:10.829 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-10T19:24:10.829 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:17:28.950137+0000", 2024-09-10T19:24:10.829 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0015896599999999999 2024-09-10T19:24:10.829 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:10.829 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:10.829 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for peered", 2024-09-10T19:24:10.829 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:17:28.950144+0000", 2024-09-10T19:24:10.829 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.0659999999999998e-06 2024-09-10T19:24:10.829 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:10.829 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:10.829 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-10T19:24:10.829 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:17:30.047351+0000", 2024-09-10T19:24:10.830 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.0972073360000001 2024-09-10T19:24:10.830 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:10.830 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:10.830 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-10T19:24:10.830 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:17:30.047471+0000", 2024-09-10T19:24:10.830 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000119522 2024-09-10T19:24:10.830 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:10.830 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:10.830 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 4,5,7", 2024-09-10T19:24:10.830 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:17:30.047689+0000", 2024-09-10T19:24:10.830 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00021808099999999999 2024-09-10T19:24:10.830 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:10.830 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:10.831 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-10T19:24:10.831 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:17:30.053285+0000", 2024-09-10T19:24:10.831 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0055959110000000003 2024-09-10T19:24:10.831 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:10.831 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:10.831 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:24:10.831 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:17:30.056858+0000", 2024-09-10T19:24:10.831 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.003573578 2024-09-10T19:24:10.831 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:10.831 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:10.831 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:24:10.831 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:17:30.057603+0000", 2024-09-10T19:24:10.831 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00074534299999999998 2024-09-10T19:24:10.831 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:10.832 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:10.832 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:24:10.832 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:17:30.077300+0000", 2024-09-10T19:24:10.832 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.019696202999999999 2024-09-10T19:24:10.832 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:10.832 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:10.832 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-10T19:24:10.832 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:17:30.077355+0000", 2024-09-10T19:24:10.832 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.5689999999999997e-05 2024-09-10T19:24:10.832 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:10.832 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:10.832 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-10T19:24:10.832 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:17:30.078429+0000", 2024-09-10T19:24:10.832 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.001073811 2024-09-10T19:24:10.833 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:24:10.833 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:24:10.833 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:24:10.833 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:10.833 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:10.833 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.15051.0:93231 8.13 8:ca03a44e:::benchmark_data_smithi183_114924_object93230:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e180)", 2024-09-10T19:24:10.833 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-10T19:17:28.948552+0000", 2024-09-10T19:24:10.833 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 401.74316008300002, 2024-09-10T19:24:10.833 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.130828164, 2024-09-10T19:24:10.833 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-10T19:24:10.833 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T19:24:10.833 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-10T19:24:10.833 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.15051", 2024-09-10T19:24:10.834 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.183:0/3632028672", 2024-09-10T19:24:10.834 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 93231 2024-09-10T19:24:10.834 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:10.834 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-10T19:24:10.834 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:10.834 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-10T19:24:10.834 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:17:28.948552+0000", 2024-09-10T19:24:10.834 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:24:10.834 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:10.834 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:10.834 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-10T19:24:10.834 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:17:28.948552+0000", 2024-09-10T19:24:10.834 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:24:10.834 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:10.835 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:10.835 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-10T19:24:10.835 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:17:28.948552+0000", 2024-09-10T19:24:10.835 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4294967295.9999995 2024-09-10T19:24:10.835 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:10.835 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:10.835 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-10T19:24:10.835 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:17:28.948572+0000", 2024-09-10T19:24:10.835 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.0670999999999999e-05 2024-09-10T19:24:10.835 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:10.835 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:10.835 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-10T19:24:10.835 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:17:28.948579+0000", 2024-09-10T19:24:10.835 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.6579999999999996e-06 2024-09-10T19:24:10.836 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:10.836 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:10.836 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-10T19:24:10.836 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:17:28.948585+0000", 2024-09-10T19:24:10.836 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.1759999999999998e-06 2024-09-10T19:24:10.836 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:10.836 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:10.836 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-10T19:24:10.836 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:17:28.950169+0000", 2024-09-10T19:24:10.836 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0015840559999999999 2024-09-10T19:24:10.836 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:10.836 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:10.836 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for peered", 2024-09-10T19:24:10.837 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:17:28.950175+0000", 2024-09-10T19:24:10.837 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.6180000000000003e-06 2024-09-10T19:24:10.837 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:10.837 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:10.837 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-10T19:24:10.837 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:17:30.048715+0000", 2024-09-10T19:24:10.837 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.098540498 2024-09-10T19:24:10.837 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:10.837 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:10.837 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-10T19:24:10.837 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:17:30.048825+0000", 2024-09-10T19:24:10.837 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00010979 2024-09-10T19:24:10.838 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:10.838 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:10.838 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 4,5,7", 2024-09-10T19:24:10.838 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:17:30.049026+0000", 2024-09-10T19:24:10.838 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00020112 2024-09-10T19:24:10.838 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:10.838 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:10.838 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-10T19:24:10.838 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:17:30.053293+0000", 2024-09-10T19:24:10.838 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0042666020000000004 2024-09-10T19:24:10.838 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:10.838 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:10.838 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:24:10.838 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:17:30.057625+0000", 2024-09-10T19:24:10.839 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0043322100000000004 2024-09-10T19:24:10.839 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:10.839 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:10.839 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:24:10.839 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:17:30.059162+0000", 2024-09-10T19:24:10.839 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.001537184 2024-09-10T19:24:10.839 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:10.839 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:10.839 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:24:10.839 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:17:30.078486+0000", 2024-09-10T19:24:10.839 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.019324158000000001 2024-09-10T19:24:10.839 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:10.839 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:10.839 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-10T19:24:10.840 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:17:30.078531+0000", 2024-09-10T19:24:10.840 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.4453999999999998e-05 2024-09-10T19:24:10.840 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:10.840 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:10.840 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-10T19:24:10.840 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:17:30.079380+0000", 2024-09-10T19:24:10.840 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00084943900000000001 2024-09-10T19:24:10.840 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:24:10.840 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:24:10.840 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:24:10.840 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:10.840 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:10.840 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.15051.0:93370 8.10 8:0d9c3bdb:::benchmark_data_smithi183_114924_object93369:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e180)", 2024-09-10T19:24:10.841 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-10T19:17:28.948817+0000", 2024-09-10T19:24:10.841 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 401.742895519, 2024-09-10T19:24:10.841 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.100337361, 2024-09-10T19:24:10.841 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-10T19:24:10.841 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T19:24:10.841 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-10T19:24:10.841 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.15051", 2024-09-10T19:24:10.841 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.183:0/3632028672", 2024-09-10T19:24:10.841 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 93370 2024-09-10T19:24:10.841 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:10.841 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-10T19:24:10.842 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:10.842 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-10T19:24:10.842 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:17:28.948817+0000", 2024-09-10T19:24:10.842 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:24:10.842 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:10.842 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:10.842 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-10T19:24:10.842 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:17:28.948817+0000", 2024-09-10T19:24:10.842 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:24:10.842 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:10.842 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:10.842 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-10T19:24:10.842 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:17:28.948816+0000", 2024-09-10T19:24:10.843 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4294967295.9999995 2024-09-10T19:24:10.843 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:10.843 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:10.843 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-10T19:24:10.843 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:17:28.948835+0000", 2024-09-10T19:24:10.843 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.8780999999999999e-05 2024-09-10T19:24:10.843 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:10.843 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:10.843 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-10T19:24:10.843 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:17:28.948843+0000", 2024-09-10T19:24:10.843 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.6299999999999998e-06 2024-09-10T19:24:10.843 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:10.843 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:10.843 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-10T19:24:10.844 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:17:28.948847+0000", 2024-09-10T19:24:10.844 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.0069999999999999e-06 2024-09-10T19:24:10.844 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:10.844 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:10.844 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-10T19:24:10.844 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:17:28.950971+0000", 2024-09-10T19:24:10.844 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0021241039999999999 2024-09-10T19:24:10.844 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:10.844 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:10.844 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for peered", 2024-09-10T19:24:10.844 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:17:28.950977+0000", 2024-09-10T19:24:10.844 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.2299999999999996e-06 2024-09-10T19:24:10.844 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:10.845 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:10.845 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-10T19:24:10.845 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:17:30.036595+0000", 2024-09-10T19:24:10.845 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.085617944 2024-09-10T19:24:10.845 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:10.845 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:10.845 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-10T19:24:10.845 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:17:30.036740+0000", 2024-09-10T19:24:10.845 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00014525000000000001 2024-09-10T19:24:10.845 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:10.845 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:10.845 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 4,5,7", 2024-09-10T19:24:10.845 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:17:30.036997+0000", 2024-09-10T19:24:10.846 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00025669700000000001 2024-09-10T19:24:10.846 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:10.846 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:10.846 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-10T19:24:10.846 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:17:30.038616+0000", 2024-09-10T19:24:10.846 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.001619437 2024-09-10T19:24:10.846 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:10.846 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:10.846 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:24:10.846 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:17:30.042016+0000", 2024-09-10T19:24:10.846 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0033999960000000002 2024-09-10T19:24:10.846 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:10.846 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:10.846 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:24:10.847 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:17:30.048991+0000", 2024-09-10T19:24:10.847 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0069749520000000004 2024-09-10T19:24:10.847 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:10.847 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:10.847 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:24:10.847 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:17:30.049015+0000", 2024-09-10T19:24:10.847 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.4096000000000001e-05 2024-09-10T19:24:10.847 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:10.847 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:10.847 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-10T19:24:10.847 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:17:30.049046+0000", 2024-09-10T19:24:10.847 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.0400999999999999e-05 2024-09-10T19:24:10.847 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:10.848 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:10.848 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-10T19:24:10.848 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:17:30.049154+0000", 2024-09-10T19:24:10.848 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000108356 2024-09-10T19:24:10.848 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:24:10.848 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:24:10.848 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:24:10.848 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:24:10.848 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:24:10.848 INFO:teuthology.orchestra.run.smithi183.stdout:} 2024-09-10T19:24:10.848 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:24:10.949 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:24:10.975 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:24:10.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:24:10.976 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:24:11.077 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:24:11.100 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:24:11.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:24:11.101 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:24:11.202 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:24:11.224 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:24:11.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:24:11.225 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:24:11.315 INFO:journalctl@ceph.osd.0.smithi183.stdout:Sep 10 19:24:11 smithi183 podman[174634]: 2024-09-10 19:24:11.232418975 +0000 UTC m=+1.841645217 container remove 422660d45ff1e63318cead9a7377a21ba2c9fc3d6800b10f08c3a417f79f83b5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3bfeaf4417585c821cab9ca76e4d95f390b0e769807122a45463640820b5cd16, name=ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd-0-activate, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, RELEASE=wip-guits-quincy-2024-09-09-1810-quincy-5be5739, ceph=True, maintainer=Guillaume Abrioux ) 2024-09-10T19:24:11.315 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:24:11 smithi183 ceph-mon[41562]: 5.17 deep-scrub starts 2024-09-10T19:24:11.315 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:24:11 smithi183 ceph-mon[41562]: 5.17 deep-scrub ok 2024-09-10T19:24:11.316 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:24:11 smithi183 ceph-mon[41562]: from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:24:11.316 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:24:11 smithi183 ceph-mon[41562]: pgmap v629: 158 pgs: 16 unknown, 24 active+clean+remapped, 21 active+undersized, 14 active+undersized+degraded, 83 active+clean; 687 KiB data, 9.3 GiB used, 617 GiB / 626 GiB avail; 20 KiB/s rd, 1.0 KiB/s wr, 42 op/s; 45/591 objects degraded (7.614%); 28/591 objects misplaced (4.738%); 495 B/s, 18 objects/s recovering 2024-09-10T19:24:11.316 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:24:11 smithi183 ceph-mon[41562]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T19:24:11.316 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:24:11 smithi183 ceph-mon[41562]: osdmap e297: 8 total, 7 up, 7 in 2024-09-10T19:24:11.316 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:24:11 smithi183 ceph-mon[29736]: 5.17 deep-scrub starts 2024-09-10T19:24:11.316 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:24:11 smithi183 ceph-mon[29736]: 5.17 deep-scrub ok 2024-09-10T19:24:11.316 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:24:11 smithi183 ceph-mon[29736]: from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:24:11.316 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:24:11 smithi183 ceph-mon[29736]: pgmap v629: 158 pgs: 16 unknown, 24 active+clean+remapped, 21 active+undersized, 14 active+undersized+degraded, 83 active+clean; 687 KiB data, 9.3 GiB used, 617 GiB / 626 GiB avail; 20 KiB/s rd, 1.0 KiB/s wr, 42 op/s; 45/591 objects degraded (7.614%); 28/591 objects misplaced (4.738%); 495 B/s, 18 objects/s recovering 2024-09-10T19:24:11.316 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:24:11 smithi183 ceph-mon[29736]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T19:24:11.316 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:24:11 smithi183 ceph-mon[29736]: osdmap e297: 8 total, 7 up, 7 in 2024-09-10T19:24:11.327 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:24:11.353 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:24:11.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:24:11.354 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:24:11.455 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:24:11.465 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:24:11 smithi191 ceph-mon[32884]: 5.17 deep-scrub starts 2024-09-10T19:24:11.465 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:24:11 smithi191 ceph-mon[32884]: 5.17 deep-scrub ok 2024-09-10T19:24:11.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:24:11 smithi191 ceph-mon[32884]: from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:24:11.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:24:11 smithi191 ceph-mon[32884]: pgmap v629: 158 pgs: 16 unknown, 24 active+clean+remapped, 21 active+undersized, 14 active+undersized+degraded, 83 active+clean; 687 KiB data, 9.3 GiB used, 617 GiB / 626 GiB avail; 20 KiB/s rd, 1.0 KiB/s wr, 42 op/s; 45/591 objects degraded (7.614%); 28/591 objects misplaced (4.738%); 495 B/s, 18 objects/s recovering 2024-09-10T19:24:11.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:24:11 smithi191 ceph-mon[32884]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T19:24:11.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:24:11 smithi191 ceph-mon[32884]: osdmap e297: 8 total, 7 up, 7 in 2024-09-10T19:24:11.480 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:24:11.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:24:11.481 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:24:11.582 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:24:11.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:24:11.606 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:24:11.607 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:24:11.628 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:24:11.628 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:24:11.628 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:24:11.681 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Sep 10 19:24:11 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-y[29989]: ::ffff:172.21.15.191 - - [10/Sep/2024:19:24:11] "GET /metrics HTTP/1.1" 200 224238 "" "Prometheus/2.43.0" 2024-09-10T19:24:11.708 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:24:11.731 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:24:11.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:24:11.732 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:24:11.833 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:24:11.859 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:24:11.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:24:11.860 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:24:11.961 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:24:11.987 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:24:11.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:24:11.988 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:24:12.089 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:24:12.114 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:24:12.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:24:12.114 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:24:12.215 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:24:12.237 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:24:12.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:24:12.238 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:24:12.339 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:24:12.359 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:24:12 smithi183 ceph-mon[29736]: 4.13 scrub starts 2024-09-10T19:24:12.359 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:24:12 smithi183 ceph-mon[29736]: 4.13 scrub ok 2024-09-10T19:24:12.360 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:24:12 smithi183 ceph-mon[41562]: 4.13 scrub starts 2024-09-10T19:24:12.360 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:24:12 smithi183 ceph-mon[41562]: 4.13 scrub ok 2024-09-10T19:24:12.365 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:24:12.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:24:12.366 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:24:12.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:24:12 smithi191 ceph-mon[32884]: 4.13 scrub starts 2024-09-10T19:24:12.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:24:12 smithi191 ceph-mon[32884]: 4.13 scrub ok 2024-09-10T19:24:12.466 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:24:12.491 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:24:12.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:24:12.492 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:24:12.593 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:24:12.617 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:24:12.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:24:12.620 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:24:12.721 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:24:12.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:24:12.748 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:24:12.749 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:24:12.849 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:24:12.874 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:24:12.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:24:12.875 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:24:12.976 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:24:13.002 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:24:13.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:24:13.003 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:24:13.104 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:24:13.127 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:24:13.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:24:13.129 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:24:13.229 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:24:13.255 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:24:13.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:24:13.256 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:24:13.331 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:24:13 smithi183 ceph-mon[29736]: pgmap v631: 158 pgs: 1 unknown, 24 active+clean+remapped, 30 active+undersized, 14 active+undersized+degraded, 89 active+clean; 687 KiB data, 9.3 GiB used, 617 GiB / 626 GiB avail; 23 KiB/s rd, 1.2 KiB/s wr, 48 op/s; 45/591 objects degraded (7.614%); 28/591 objects misplaced (4.738%); 367 B/s, 10 objects/s recovering 2024-09-10T19:24:13.331 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:24:13 smithi183 ceph-mon[29736]: 2.17 scrub starts 2024-09-10T19:24:13.331 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:24:13 smithi183 ceph-mon[29736]: 2.17 scrub ok 2024-09-10T19:24:13.331 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:24:13 smithi183 ceph-mon[41562]: pgmap v631: 158 pgs: 1 unknown, 24 active+clean+remapped, 30 active+undersized, 14 active+undersized+degraded, 89 active+clean; 687 KiB data, 9.3 GiB used, 617 GiB / 626 GiB avail; 23 KiB/s rd, 1.2 KiB/s wr, 48 op/s; 45/591 objects degraded (7.614%); 28/591 objects misplaced (4.738%); 367 B/s, 10 objects/s recovering 2024-09-10T19:24:13.331 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:24:13 smithi183 ceph-mon[41562]: 2.17 scrub starts 2024-09-10T19:24:13.331 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:24:13 smithi183 ceph-mon[41562]: 2.17 scrub ok 2024-09-10T19:24:13.357 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:24:13.378 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:24:13.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:24:13.379 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:24:13.465 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:24:13 smithi191 ceph-mon[32884]: pgmap v631: 158 pgs: 1 unknown, 24 active+clean+remapped, 30 active+undersized, 14 active+undersized+degraded, 89 active+clean; 687 KiB data, 9.3 GiB used, 617 GiB / 626 GiB avail; 23 KiB/s rd, 1.2 KiB/s wr, 48 op/s; 45/591 objects degraded (7.614%); 28/591 objects misplaced (4.738%); 367 B/s, 10 objects/s recovering 2024-09-10T19:24:13.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:24:13 smithi191 ceph-mon[32884]: 2.17 scrub starts 2024-09-10T19:24:13.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:24:13 smithi191 ceph-mon[32884]: 2.17 scrub ok 2024-09-10T19:24:13.479 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:24:13.501 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:24:13.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:24:13.502 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:24:13.602 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:24:13.624 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:24:13.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:24:13.625 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:24:13.726 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:24:13.747 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:24:13.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:24:13.748 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:24:13.849 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:24:13.870 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:24:13.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:24:13.871 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:24:13.971 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:24:13.997 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:24:13.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:24:13.998 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:24:14.099 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:24:14.121 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:24:14.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:24:14.122 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:24:14.223 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:24:14.249 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:24:14.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:24:14.250 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:24:14.351 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:24:14.376 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:24:14.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:24:14.377 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:24:14.478 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:24:14.504 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:24:14.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:24:14.505 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:24:14.567 DEBUG:teuthology.orchestra.run.smithi191:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph daemon osd.4 dump_ops_in_flight 2024-09-10T19:24:14.606 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:24:14.628 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:24:14.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:24:14.629 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:24:14.730 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:24:14.751 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:24:14.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:24:14.752 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:24:14.799 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:24:14 smithi183 ceph-mon[29736]: 5.14 deep-scrub starts 2024-09-10T19:24:14.799 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:24:14 smithi183 ceph-mon[29736]: 5.14 deep-scrub ok 2024-09-10T19:24:14.799 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:24:14 smithi183 ceph-mon[29736]: 5.1 deep-scrub starts 2024-09-10T19:24:14.800 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:24:14 smithi183 ceph-mon[29736]: 5.1 deep-scrub ok 2024-09-10T19:24:14.800 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:24:14 smithi183 ceph-mon[29736]: 2.8 deep-scrub starts 2024-09-10T19:24:14.800 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:24:14 smithi183 ceph-mon[29736]: 2.8 deep-scrub ok 2024-09-10T19:24:14.800 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:24:14 smithi183 ceph-mon[29736]: from='mgr.14556 ' entity='mgr.y' 2024-09-10T19:24:14.800 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:24:14 smithi183 ceph-mon[29736]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:24:14.800 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:24:14 smithi183 ceph-mon[29736]: pgmap v632: 158 pgs: 24 active+clean+remapped, 31 active+undersized, 14 active+undersized+degraded, 89 active+clean; 687 KiB data, 9.3 GiB used, 617 GiB / 626 GiB avail; 853 B/s rd, 0 op/s; 45/591 objects degraded (7.614%); 28/591 objects misplaced (4.738%) 2024-09-10T19:24:14.800 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:24:14 smithi183 ceph-mon[41562]: 5.14 deep-scrub starts 2024-09-10T19:24:14.800 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:24:14 smithi183 ceph-mon[41562]: 5.14 deep-scrub ok 2024-09-10T19:24:14.800 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:24:14 smithi183 ceph-mon[41562]: 5.1 deep-scrub starts 2024-09-10T19:24:14.800 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:24:14 smithi183 ceph-mon[41562]: 5.1 deep-scrub ok 2024-09-10T19:24:14.800 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:24:14 smithi183 ceph-mon[41562]: 2.8 deep-scrub starts 2024-09-10T19:24:14.801 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:24:14 smithi183 ceph-mon[41562]: 2.8 deep-scrub ok 2024-09-10T19:24:14.801 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:24:14 smithi183 ceph-mon[41562]: from='mgr.14556 ' entity='mgr.y' 2024-09-10T19:24:14.801 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:24:14 smithi183 ceph-mon[41562]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:24:14.801 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:24:14 smithi183 ceph-mon[41562]: pgmap v632: 158 pgs: 24 active+clean+remapped, 31 active+undersized, 14 active+undersized+degraded, 89 active+clean; 687 KiB data, 9.3 GiB used, 617 GiB / 626 GiB avail; 853 B/s rd, 0 op/s; 45/591 objects degraded (7.614%); 28/591 objects misplaced (4.738%) 2024-09-10T19:24:14.825 INFO:teuthology.orchestra.run.smithi191.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.b/config 2024-09-10T19:24:14.853 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:24:14.876 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:24:14.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:24:14.877 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:24:14.947 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:24:14 smithi191 ceph-mon[32884]: 5.14 deep-scrub starts 2024-09-10T19:24:14.947 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:24:14 smithi191 ceph-mon[32884]: 5.14 deep-scrub ok 2024-09-10T19:24:14.947 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:24:14 smithi191 ceph-mon[32884]: 5.1 deep-scrub starts 2024-09-10T19:24:14.947 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:24:14 smithi191 ceph-mon[32884]: 5.1 deep-scrub ok 2024-09-10T19:24:14.948 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:24:14 smithi191 ceph-mon[32884]: 2.8 deep-scrub starts 2024-09-10T19:24:14.948 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:24:14 smithi191 ceph-mon[32884]: 2.8 deep-scrub ok 2024-09-10T19:24:14.948 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:24:14 smithi191 ceph-mon[32884]: from='mgr.14556 ' entity='mgr.y' 2024-09-10T19:24:14.948 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:24:14 smithi191 ceph-mon[32884]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:24:14.948 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:24:14 smithi191 ceph-mon[32884]: pgmap v632: 158 pgs: 24 active+clean+remapped, 31 active+undersized, 14 active+undersized+degraded, 89 active+clean; 687 KiB data, 9.3 GiB used, 617 GiB / 626 GiB avail; 853 B/s rd, 0 op/s; 45/591 objects degraded (7.614%); 28/591 objects misplaced (4.738%) 2024-09-10T19:24:14.978 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:24:15.004 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:24:15.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:24:15.005 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:24:15.106 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:24:15.128 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:24:15.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:24:15.129 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:24:15.230 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:24:15.251 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:24:15.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:24:15.252 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:24:15.353 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:24:15.376 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:24:15.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:24:15.377 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:24:15.478 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:24:15.499 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:24:15.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:24:15.500 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:24:15.601 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:24:15.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:24:15.624 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:24:15.624 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:24:15.681 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:24:15 smithi183 ceph-mon[29736]: Health check update: Degraded data redundancy: 45/591 objects degraded (7.614%), 14 pgs degraded, 35 pgs undersized (PG_DEGRADED) 2024-09-10T19:24:15.682 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:24:15 smithi183 ceph-mon[29736]: from='client.? 172.21.15.183:0/4025920459' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-10T19:24:15.682 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:24:15 smithi183 ceph-mon[41562]: Health check update: Degraded data redundancy: 45/591 objects degraded (7.614%), 14 pgs degraded, 35 pgs undersized (PG_DEGRADED) 2024-09-10T19:24:15.682 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:24:15 smithi183 ceph-mon[41562]: from='client.? 172.21.15.183:0/4025920459' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-10T19:24:15.724 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:24:15.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:24:15.746 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:24:15.746 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:24:15.847 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:24:15.868 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:24:15.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:24:15.869 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:24:15.875 INFO:teuthology.orchestra.run.smithi183.stderr:noscrub is set 2024-09-10T19:24:15.966 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:24:15 smithi191 ceph-mon[32884]: Health check update: Degraded data redundancy: 45/591 objects degraded (7.614%), 14 pgs degraded, 35 pgs undersized (PG_DEGRADED) 2024-09-10T19:24:15.966 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:24:15 smithi191 ceph-mon[32884]: from='client.? 172.21.15.183:0/4025920459' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-10T19:24:15.970 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:24:15.992 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:24:15.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:24:15.993 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:24:16.094 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:24:16.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:24:16.115 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:24:16.116 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:24:16.181 INFO:journalctl@ceph.osd.0.smithi183.stdout:Sep 10 19:24:15 smithi183 podman[175119]: 2024-09-10 19:24:15.643533965 +0000 UTC m=+2.048187076 image pull 4c55dd4096115e48502f368cff1eab06769f71f030e1692e2611bbc6c6f637e9 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3bfeaf4417585c821cab9ca76e4d95f390b0e769807122a45463640820b5cd16 2024-09-10T19:24:16.181 INFO:journalctl@ceph.osd.0.smithi183.stdout:Sep 10 19:24:15 smithi183 podman[175119]: 2024-09-10 19:24:15.779693599 +0000 UTC m=+2.184346698 container create c228d1a8b2b6f4e54923e367c5ec632b01f507707e582a107994c566f8aac7db (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3bfeaf4417585c821cab9ca76e4d95f390b0e769807122a45463640820b5cd16, name=ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd-0, GIT_CLEAN=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, RELEASE=wip-guits-quincy-2024-09-09-1810-quincy-5be5739, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2) 2024-09-10T19:24:16.216 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:24:16.239 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:24:16.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:24:16.240 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:24:16.341 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:24:16.346 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph daemon osd.0 dump_ops_in_flight 2024-09-10T19:24:16.362 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:24:16.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:24:16.363 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:24:16.464 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:24:16.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:24:16.486 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:24:16.486 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:24:16.587 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:24:16.609 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:24:16.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:24:16.610 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:24:16.642 INFO:teuthology.orchestra.run.smithi191.stdout:{ 2024-09-10T19:24:16.642 INFO:teuthology.orchestra.run.smithi191.stdout: "ops": [], 2024-09-10T19:24:16.643 INFO:teuthology.orchestra.run.smithi191.stdout: "num_ops": 0 2024-09-10T19:24:16.643 INFO:teuthology.orchestra.run.smithi191.stdout:} 2024-09-10T19:24:16.680 INFO:journalctl@ceph.osd.0.smithi183.stdout:Sep 10 19:24:16 smithi183 podman[175119]: 2024-09-10 19:24:16.232212472 +0000 UTC m=+2.636865573 container init c228d1a8b2b6f4e54923e367c5ec632b01f507707e582a107994c566f8aac7db (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3bfeaf4417585c821cab9ca76e4d95f390b0e769807122a45463640820b5cd16, name=ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd-0, GIT_BRANCH=HEAD, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, RELEASE=wip-guits-quincy-2024-09-09-1810-quincy-5be5739) 2024-09-10T19:24:16.680 INFO:journalctl@ceph.osd.0.smithi183.stdout:Sep 10 19:24:16 smithi183 podman[175119]: 2024-09-10 19:24:16.238654878 +0000 UTC m=+2.643307977 container start c228d1a8b2b6f4e54923e367c5ec632b01f507707e582a107994c566f8aac7db (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3bfeaf4417585c821cab9ca76e4d95f390b0e769807122a45463640820b5cd16, name=ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd-0, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-guits-quincy-2024-09-09-1810-quincy-5be5739, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, GIT_BRANCH=HEAD, ceph=True, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS) 2024-09-10T19:24:16.680 INFO:journalctl@ceph.osd.0.smithi183.stdout:Sep 10 19:24:16 smithi183 ceph-osd[175434]: -- 172.21.15.183:0/1447243545 <== mon.2 v2:172.21.15.183:3301/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x557e475bad00 con 0x557e4758d400 2024-09-10T19:24:16.680 INFO:journalctl@ceph.osd.0.smithi183.stdout:Sep 10 19:24:16 smithi183 bash[175119]: c228d1a8b2b6f4e54923e367c5ec632b01f507707e582a107994c566f8aac7db 2024-09-10T19:24:16.680 INFO:journalctl@ceph.osd.0.smithi183.stdout:Sep 10 19:24:16 smithi183 systemd[1]: Started Ceph osd.0 for 320f6df8-6fa7-11ef-bcea-c7b262605968. 2024-09-10T19:24:16.711 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:24:16.733 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:24:16.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:24:16.734 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:24:16.835 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:24:16.856 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:24:16.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:24:16.857 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:24:16.932 INFO:journalctl@ceph.osd.0.smithi183.stdout:Sep 10 19:24:16 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd-0[175367]: 2024-09-10T19:24:16.783+0000 7f44f8771440 -1 Falling back to public interface 2024-09-10T19:24:16.932 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:24:16 smithi183 ceph-mon[29736]: 4.a scrub starts 2024-09-10T19:24:16.932 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:24:16 smithi183 ceph-mon[29736]: 4.a scrub ok 2024-09-10T19:24:16.932 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:24:16 smithi183 ceph-mon[29736]: 6.1e scrub starts 2024-09-10T19:24:16.932 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:24:16 smithi183 ceph-mon[29736]: 6.1e scrub ok 2024-09-10T19:24:16.932 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:24:16 smithi183 ceph-mon[29736]: pgmap v633: 158 pgs: 24 active+clean+remapped, 31 active+undersized, 14 active+undersized+degraded, 89 active+clean; 687 KiB data, 9.3 GiB used, 617 GiB / 626 GiB avail; 654 B/s rd, 0 op/s; 45/591 objects degraded (7.614%); 28/591 objects misplaced (4.738%) 2024-09-10T19:24:16.932 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:24:16 smithi183 ceph-mon[29736]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T19:24:16.932 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:24:16 smithi183 ceph-mon[29736]: from='client.? 172.21.15.183:0/4025920459' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-10T19:24:16.933 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:24:16 smithi183 ceph-mon[29736]: osdmap e298: 8 total, 7 up, 7 in 2024-09-10T19:24:16.933 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:24:16 smithi183 ceph-mon[41562]: 4.a scrub starts 2024-09-10T19:24:16.933 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:24:16 smithi183 ceph-mon[41562]: 4.a scrub ok 2024-09-10T19:24:16.933 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:24:16 smithi183 ceph-mon[41562]: 6.1e scrub starts 2024-09-10T19:24:16.933 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:24:16 smithi183 ceph-mon[41562]: 6.1e scrub ok 2024-09-10T19:24:16.933 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:24:16 smithi183 ceph-mon[41562]: pgmap v633: 158 pgs: 24 active+clean+remapped, 31 active+undersized, 14 active+undersized+degraded, 89 active+clean; 687 KiB data, 9.3 GiB used, 617 GiB / 626 GiB avail; 654 B/s rd, 0 op/s; 45/591 objects degraded (7.614%); 28/591 objects misplaced (4.738%) 2024-09-10T19:24:16.933 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:24:16 smithi183 ceph-mon[41562]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T19:24:16.933 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:24:16 smithi183 ceph-mon[41562]: from='client.? 172.21.15.183:0/4025920459' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-10T19:24:16.933 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:24:16 smithi183 ceph-mon[41562]: osdmap e298: 8 total, 7 up, 7 in 2024-09-10T19:24:16.955 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:24:16 smithi191 ceph-mon[32884]: 4.a scrub starts 2024-09-10T19:24:16.955 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:24:16 smithi191 ceph-mon[32884]: 4.a scrub ok 2024-09-10T19:24:16.955 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:24:16 smithi191 ceph-mon[32884]: 6.1e scrub starts 2024-09-10T19:24:16.955 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:24:16 smithi191 ceph-mon[32884]: 6.1e scrub ok 2024-09-10T19:24:16.956 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:24:16 smithi191 ceph-mon[32884]: pgmap v633: 158 pgs: 24 active+clean+remapped, 31 active+undersized, 14 active+undersized+degraded, 89 active+clean; 687 KiB data, 9.3 GiB used, 617 GiB / 626 GiB avail; 654 B/s rd, 0 op/s; 45/591 objects degraded (7.614%); 28/591 objects misplaced (4.738%) 2024-09-10T19:24:16.956 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:24:16 smithi191 ceph-mon[32884]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T19:24:16.956 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:24:16 smithi191 ceph-mon[32884]: from='client.? 172.21.15.183:0/4025920459' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-10T19:24:16.956 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:24:16 smithi191 ceph-mon[32884]: osdmap e298: 8 total, 7 up, 7 in 2024-09-10T19:24:16.958 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:24:16.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:24:16.981 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:24:16.981 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:24:17.082 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:24:17.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:24:17.105 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:24:17.106 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:24:17.206 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:24:17.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:24:17.228 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:24:17.229 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:24:17.330 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:24:17.351 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:24:17.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:24:17.352 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:24:17.393 DEBUG:teuthology.orchestra.run.smithi191:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph daemon osd.4 dump_blocked_ops 2024-09-10T19:24:17.453 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:24:17.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:24:17.477 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:24:17.477 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:24:17.578 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:24:17.599 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:24:17.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:24:17.600 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:24:17.637 INFO:teuthology.orchestra.run.smithi191.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.b/config 2024-09-10T19:24:17.701 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:24:17.715 INFO:journalctl@ceph.mgr.x.smithi191.stdout:Sep 10 19:24:17 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-x[34967]: ::ffff:172.21.15.191 - - [10/Sep/2024:19:24:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T19:24:17.723 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:24:17.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:24:17.724 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:24:17.818 INFO:teuthology.orchestra.run.smithi183.stdout:osd.1: {} 2024-09-10T19:24:17.818 INFO:teuthology.orchestra.run.smithi183.stderr:Error ENXIO: problem getting command descriptions from osd.0 2024-09-10T19:24:17.818 INFO:teuthology.orchestra.run.smithi183.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-10T19:24:17.822 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:24:17.825 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:24:17.833 INFO:teuthology.orchestra.run.smithi183.stdout:osd.2: {} 2024-09-10T19:24:17.833 INFO:teuthology.orchestra.run.smithi183.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-10T19:24:17.847 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:24:17.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:24:17.849 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:24:17.850 INFO:teuthology.orchestra.run.smithi183.stdout:osd.3: {} 2024-09-10T19:24:17.850 INFO:teuthology.orchestra.run.smithi183.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-10T19:24:17.869 INFO:teuthology.orchestra.run.smithi183.stdout:osd.4: {} 2024-09-10T19:24:17.869 INFO:teuthology.orchestra.run.smithi183.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-10T19:24:17.870 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:24:17 smithi183 ceph-mon[29736]: from='client.? 172.21.15.183:0/3103319571' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T19:24:17.870 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:24:17 smithi183 ceph-mon[41562]: from='client.? 172.21.15.183:0/3103319571' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T19:24:17.884 INFO:teuthology.orchestra.run.smithi183.stdout:osd.5: {} 2024-09-10T19:24:17.885 INFO:teuthology.orchestra.run.smithi183.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-10T19:24:17.900 INFO:teuthology.orchestra.run.smithi183.stdout:osd.6: {} 2024-09-10T19:24:17.901 INFO:teuthology.orchestra.run.smithi183.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-10T19:24:17.916 INFO:teuthology.orchestra.run.smithi183.stdout:osd.7: {} 2024-09-10T19:24:17.916 INFO:teuthology.orchestra.run.smithi183.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-10T19:24:17.949 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:24:17.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:24:17.972 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:24:17.972 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:24:18.051 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:24:17 smithi191 ceph-mon[32884]: from='client.? 172.21.15.183:0/3103319571' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T19:24:18.073 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:24:18.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:24:18.096 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:24:18.096 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:24:18.197 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:24:18.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:24:18.221 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:24:18.221 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:24:18.322 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:24:18.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:24:18.345 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:24:18.345 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:24:18.446 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:24:18.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:24:18.469 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:24:18.469 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:24:18.572 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:24:18.594 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:24:18.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:24:18.595 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:24:18.696 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:24:18.718 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:24:18.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:24:18.719 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:24:18.820 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:24:18.841 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:24:18.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:24:18.842 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:24:18.859 INFO:teuthology.orchestra.run.smithi183.stderr:enabled application 'rados' on pool 'unique_pool_3' 2024-09-10T19:24:18.894 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-10T19:24:18.931 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:24:18 smithi191 ceph-mon[32884]: pgmap v635: 158 pgs: 24 active+clean+remapped, 31 active+undersized, 14 active+undersized+degraded, 89 active+clean; 687 KiB data, 9.3 GiB used, 617 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s; 45/591 objects degraded (7.614%); 28/591 objects misplaced (4.738%) 2024-09-10T19:24:18.931 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:24:18 smithi191 ceph-mon[32884]: from='client.? 172.21.15.183:0/1377001418' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:24:18.943 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:24:18.965 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:24:18.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:24:18.966 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:24:19.067 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:24:19.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:24:19.089 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:24:19.090 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:24:19.100 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:24:18 smithi183 ceph-mon[29736]: pgmap v635: 158 pgs: 24 active+clean+remapped, 31 active+undersized, 14 active+undersized+degraded, 89 active+clean; 687 KiB data, 9.3 GiB used, 617 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s; 45/591 objects degraded (7.614%); 28/591 objects misplaced (4.738%) 2024-09-10T19:24:19.100 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:24:18 smithi183 ceph-mon[29736]: from='client.? 172.21.15.183:0/1377001418' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:24:19.100 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:24:18 smithi183 ceph-mon[41562]: pgmap v635: 158 pgs: 24 active+clean+remapped, 31 active+undersized, 14 active+undersized+degraded, 89 active+clean; 687 KiB data, 9.3 GiB used, 617 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s; 45/591 objects degraded (7.614%); 28/591 objects misplaced (4.738%) 2024-09-10T19:24:19.100 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:24:18 smithi183 ceph-mon[41562]: from='client.? 172.21.15.183:0/1377001418' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:24:19.190 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:24:19.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:24:19.213 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:24:19.213 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:24:19.314 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:24:19.336 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:24:19.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:24:19.337 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:24:19.420 INFO:teuthology.orchestra.run.smithi191.stdout:ERROR: (22) Invalid argument 2024-09-10T19:24:19.420 INFO:teuthology.orchestra.run.smithi191.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-10T19:24:19.437 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:24:19.459 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:24:19.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:24:19.460 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:24:19.561 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:24:19.583 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:24:19.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:24:19.584 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:24:19.685 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:24:19.706 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:24:19.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:24:19.707 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:24:19.807 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:24:19.828 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:24:19.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:24:19.829 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:24:19.930 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:24:19.952 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:24:19.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:24:19.953 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:24:19.965 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:24:19 smithi191 ceph-mon[32884]: from='client.? 172.21.15.183:0/1377001418' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-09-10T19:24:19.966 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:24:19 smithi191 ceph-mon[32884]: osdmap e299: 8 total, 7 up, 7 in 2024-09-10T19:24:19.966 INFO:journalctl@ceph.iscsi.iscsi.a.smithi191.stdout:Sep 10 19:24:19 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-iscsi-iscsi-a[61816]: debug there is no tcmu-runner data available 2024-09-10T19:24:20.054 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:24:20.076 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:24:20.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:24:20.077 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:24:20.154 DEBUG:teuthology.orchestra.run.smithi191:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph daemon osd.4 dump_historic_ops 2024-09-10T19:24:20.169 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:24:19 smithi183 ceph-mon[29736]: from='client.? 172.21.15.183:0/1377001418' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-09-10T19:24:20.169 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:24:19 smithi183 ceph-mon[29736]: osdmap e299: 8 total, 7 up, 7 in 2024-09-10T19:24:20.170 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:24:19 smithi183 ceph-mon[41562]: from='client.? 172.21.15.183:0/1377001418' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_3", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-09-10T19:24:20.170 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:24:19 smithi183 ceph-mon[41562]: osdmap e299: 8 total, 7 up, 7 in 2024-09-10T19:24:20.178 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:24:20.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:24:20.216 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:24:20.217 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:24:20.317 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:24:20.339 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:24:20.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:24:20.340 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:24:20.440 INFO:teuthology.orchestra.run.smithi191.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.b/config 2024-09-10T19:24:20.440 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:24:20.462 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:24:20.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:24:20.463 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:24:20.564 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:24:20.579 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-10T19:24:20.580 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-10T19:24:20.586 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:24:20.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:24:20.587 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:24:20.689 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:24:20.710 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:24:20.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:24:20.711 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:24:20.803 DEBUG:teuthology.orchestra.run.smithi183:> /bin/sh -c 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --no-log-to-stderr --name client.0 -b 65536 --object-size 65536 -p unique_pool_3 bench 90 write' 2024-09-10T19:24:20.805 INFO:tasks.radosbench:joining radosbench (timing out after 3000s) 2024-09-10T19:24:20.805 INFO:teuthology.orchestra.run:waiting for 3000 2024-09-10T19:24:20.812 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:24:20.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:24:20.835 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:24:20.835 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:24:20.852 INFO:tasks.radosbench.radosbench.0.smithi183.stdout:hints = 1 2024-09-10T19:24:20.852 INFO:tasks.radosbench.radosbench.0.smithi183.stdout:Maintaining 16 concurrent writes of 65536 bytes to objects of size 65536 for up to 90 seconds or 0 objects 2024-09-10T19:24:20.852 INFO:tasks.radosbench.radosbench.0.smithi183.stdout:Object prefix: benchmark_data_smithi183_176537 2024-09-10T19:24:20.853 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-10T19:24:20.853 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: 0 0 0 0 0 0 - 0 2024-09-10T19:24:20.936 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:24:20.965 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:24:20.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:24:20.966 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:24:21.055 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:24:21.067 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:24:21.104 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:24:21.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:24:21.106 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:24:21.181 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:24:20 smithi183 ceph-mon[41562]: from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:24:21.181 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:24:20 smithi183 ceph-mon[41562]: pgmap v637: 158 pgs: 24 active+clean+remapped, 31 active+undersized, 14 active+undersized+degraded, 89 active+clean; 687 KiB data, 9.3 GiB used, 617 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s; 45/591 objects degraded (7.614%); 28/591 objects misplaced (4.738%) 2024-09-10T19:24:21.182 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:24:20 smithi183 ceph-mon[29736]: from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:24:21.182 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:24:20 smithi183 ceph-mon[29736]: pgmap v637: 158 pgs: 24 active+clean+remapped, 31 active+undersized, 14 active+undersized+degraded, 89 active+clean; 687 KiB data, 9.3 GiB used, 617 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s; 45/591 objects degraded (7.614%); 28/591 objects misplaced (4.738%) 2024-09-10T19:24:21.206 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:24:21.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:24:20 smithi191 ceph-mon[32884]: from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:24:21.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:24:20 smithi191 ceph-mon[32884]: pgmap v637: 158 pgs: 24 active+clean+remapped, 31 active+undersized, 14 active+undersized+degraded, 89 active+clean; 687 KiB data, 9.3 GiB used, 617 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s; 45/591 objects degraded (7.614%); 28/591 objects misplaced (4.738%) 2024-09-10T19:24:21.244 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:24:21.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:24:21.245 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:24:21.345 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:24:21.374 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:24:21.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:24:21.375 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:24:21.476 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:24:21.512 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:24:21.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:24:21.513 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:24:21.560 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Sep 10 19:24:21 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-y[29989]: ::ffff:172.21.15.191 - - [10/Sep/2024:19:24:21] "GET /metrics HTTP/1.1" 200 224473 "" "Prometheus/2.43.0" 2024-09-10T19:24:21.613 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:24:21.649 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:24:21.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:24:21.650 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:24:21.750 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:24:21.782 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:24:21.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:24:21.783 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:24:21.853 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: 1 16 3963 3947 246.675 246.688 0.0072636 0.00403992 2024-09-10T19:24:21.883 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:24:21.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:24:21.928 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:24:21.928 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:24:22.029 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:24:22.059 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:24:21 smithi191 ceph-mon[32884]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-10T19:24:22.060 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:24:21 smithi191 ceph-mon[32884]: from='osd.0 [v2:172.21.15.183:6810/3346539414,v1:172.21.15.183:6811/3346539414]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-09-10T19:24:22.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:24:22.068 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:24:22.068 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:24:22.169 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:24:22.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:24:21 smithi183 ceph-mon[29736]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-10T19:24:22.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:24:21 smithi183 ceph-mon[29736]: from='osd.0 [v2:172.21.15.183:6810/3346539414,v1:172.21.15.183:6811/3346539414]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-09-10T19:24:22.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:24:21 smithi183 ceph-mon[41562]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-10T19:24:22.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:24:21 smithi183 ceph-mon[41562]: from='osd.0 [v2:172.21.15.183:6810/3346539414,v1:172.21.15.183:6811/3346539414]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-09-10T19:24:22.182 INFO:journalctl@ceph.osd.0.smithi183.stdout:Sep 10 19:24:21 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd-0[175367]: 2024-09-10T19:24:21.816+0000 7f44f8771440 -1 osd.0 262 log_to_monitors true 2024-09-10T19:24:22.182 INFO:journalctl@ceph.osd.0.smithi183.stdout:Sep 10 19:24:21 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd-0[175367]: 2024-09-10T19:24:21.899+0000 7f44ef51b640 -1 osd.0 262 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-10T19:24:22.197 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:24:22.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:24:22.198 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:24:22.299 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:24:22.334 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:24:22.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:24:22.335 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:24:22.436 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:24:22.460 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:24:22.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:24:22.461 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:24:22.540 INFO:teuthology.orchestra.run.smithi191.stdout:ERROR: (22) Invalid argument 2024-09-10T19:24:22.540 INFO:teuthology.orchestra.run.smithi191.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-10T19:24:22.562 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:24:22.581 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-10T19:24:22.597 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:24:22.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:24:22.598 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:24:22.699 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:24:22.735 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:24:22.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:24:22.736 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:24:22.837 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:24:22.854 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: 2 16 7295 7279 227.446 208.25 0.00717017 0.00438872 2024-09-10T19:24:22.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:24:22.870 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:24:22.870 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:24:22.971 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:24:22.991 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:24:22.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:24:22.992 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:24:23.093 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:24:23.115 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:24:23.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:24:23.116 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:24:23.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:24:22 smithi183 ceph-mon[29736]: pgmap v638: 158 pgs: 24 active+clean+remapped, 31 active+undersized, 14 active+undersized+degraded, 89 active+clean; 687 KiB data, 9.3 GiB used, 617 GiB / 626 GiB avail; 639 B/s rd, 0 op/s; 45/591 objects degraded (7.614%); 28/591 objects misplaced (4.738%) 2024-09-10T19:24:23.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:24:22 smithi183 ceph-mon[29736]: from='osd.0 [v2:172.21.15.183:6810/3346539414,v1:172.21.15.183:6811/3346539414]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-09-10T19:24:23.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:24:22 smithi183 ceph-mon[29736]: osdmap e300: 8 total, 7 up, 7 in 2024-09-10T19:24:23.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:24:22 smithi183 ceph-mon[29736]: from='osd.0 [v2:172.21.15.183:6810/3346539414,v1:172.21.15.183:6811/3346539414]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi183", "root=default"]}]: dispatch 2024-09-10T19:24:23.181 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:24:22 smithi183 ceph-mon[41562]: pgmap v638: 158 pgs: 24 active+clean+remapped, 31 active+undersized, 14 active+undersized+degraded, 89 active+clean; 687 KiB data, 9.3 GiB used, 617 GiB / 626 GiB avail; 639 B/s rd, 0 op/s; 45/591 objects degraded (7.614%); 28/591 objects misplaced (4.738%) 2024-09-10T19:24:23.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:24:22 smithi183 ceph-mon[41562]: from='osd.0 [v2:172.21.15.183:6810/3346539414,v1:172.21.15.183:6811/3346539414]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-09-10T19:24:23.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:24:22 smithi183 ceph-mon[41562]: osdmap e300: 8 total, 7 up, 7 in 2024-09-10T19:24:23.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:24:22 smithi183 ceph-mon[41562]: from='osd.0 [v2:172.21.15.183:6810/3346539414,v1:172.21.15.183:6811/3346539414]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi183", "root=default"]}]: dispatch 2024-09-10T19:24:23.207 INFO:teuthology.orchestra.run.smithi183.stdout:{ 2024-09-10T19:24:23.207 INFO:teuthology.orchestra.run.smithi183.stdout: "ops": [ 2024-09-10T19:24:23.207 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:23.207 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.25622.0:7383 10.4 10.f7f93b4 (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e301)", 2024-09-10T19:24:23.207 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-10T19:24:22.907252+0000", 2024-09-10T19:24:23.208 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 0.29763478300000001, 2024-09-10T19:24:23.208 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.29766054400000003, 2024-09-10T19:24:23.208 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-10T19:24:23.208 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "waiting for peered", 2024-09-10T19:24:23.208 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-10T19:24:23.208 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.25622", 2024-09-10T19:24:23.208 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.183:0/3685189199", 2024-09-10T19:24:23.208 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 7383 2024-09-10T19:24:23.208 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:23.208 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-10T19:24:23.208 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:23.208 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-10T19:24:23.208 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:24:22.907252+0000", 2024-09-10T19:24:23.208 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:24:23.209 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:23.209 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:23.209 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-10T19:24:23.209 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:24:22.907252+0000", 2024-09-10T19:24:23.209 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:24:23.209 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:23.209 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:23.209 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-10T19:24:23.209 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:24:22.907250+0000", 2024-09-10T19:24:23.209 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4294967295.9999986 2024-09-10T19:24:23.209 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:23.209 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:23.209 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-10T19:24:23.209 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:24:22.907308+0000", 2024-09-10T19:24:23.209 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.7673999999999997e-05 2024-09-10T19:24:23.210 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:23.210 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:23.210 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-10T19:24:23.210 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:24:22.907321+0000", 2024-09-10T19:24:23.210 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.2887e-05 2024-09-10T19:24:23.210 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:23.210 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:23.210 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-10T19:24:23.210 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:24:22.907328+0000", 2024-09-10T19:24:23.210 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.6640000000000001e-06 2024-09-10T19:24:23.210 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:23.210 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:23.210 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-10T19:24:23.210 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:24:22.907367+0000", 2024-09-10T19:24:23.210 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.9017999999999998e-05 2024-09-10T19:24:23.211 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:23.211 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:23.211 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for peered", 2024-09-10T19:24:23.211 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:24:22.907374+0000", 2024-09-10T19:24:23.211 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.0060000000000003e-06 2024-09-10T19:24:23.211 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:24:23.211 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:24:23.211 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:24:23.211 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:23.211 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:23.211 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.25622.0:7387 10.4 10.6dd5c814 (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e301)", 2024-09-10T19:24:23.211 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-10T19:24:22.907340+0000", 2024-09-10T19:24:23.211 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 0.297546792, 2024-09-10T19:24:23.211 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.29764711199999999, 2024-09-10T19:24:23.212 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-10T19:24:23.212 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "waiting for peered", 2024-09-10T19:24:23.212 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-10T19:24:23.212 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.25622", 2024-09-10T19:24:23.212 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.183:0/3685189199", 2024-09-10T19:24:23.212 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 7387 2024-09-10T19:24:23.212 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:23.212 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-10T19:24:23.212 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:23.212 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-10T19:24:23.212 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:24:22.907340+0000", 2024-09-10T19:24:23.212 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:24:23.212 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:23.212 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:23.213 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-10T19:24:23.213 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:24:22.907340+0000", 2024-09-10T19:24:23.213 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:24:23.213 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:23.213 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:23.213 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-10T19:24:23.213 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:24:22.907340+0000", 2024-09-10T19:24:23.213 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4294967295.9999995 2024-09-10T19:24:23.213 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:23.213 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:23.213 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-10T19:24:23.213 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:24:22.907389+0000", 2024-09-10T19:24:23.213 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.9422000000000002e-05 2024-09-10T19:24:23.213 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:23.214 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:23.214 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-10T19:24:23.214 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:24:22.907396+0000", 2024-09-10T19:24:23.214 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.9059999999999996e-06 2024-09-10T19:24:23.214 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:23.214 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:23.214 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-10T19:24:23.214 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:24:22.907401+0000", 2024-09-10T19:24:23.214 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.8940000000000001e-06 2024-09-10T19:24:23.214 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:23.214 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:23.214 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-10T19:24:23.214 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:24:22.907697+0000", 2024-09-10T19:24:23.214 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000296101 2024-09-10T19:24:23.214 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:23.215 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:23.215 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for peered", 2024-09-10T19:24:23.215 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:24:22.907702+0000", 2024-09-10T19:24:23.215 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.5319999999999997e-06 2024-09-10T19:24:23.215 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:24:23.215 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:24:23.215 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:24:23.215 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:23.215 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:23.215 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.25622.0:7397 10.8 10.363724e8 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e301)", 2024-09-10T19:24:23.215 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-10T19:24:22.907971+0000", 2024-09-10T19:24:23.215 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 0.29691574799999998, 2024-09-10T19:24:23.215 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.29705272799999999, 2024-09-10T19:24:23.215 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-10T19:24:23.215 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "waiting for peered", 2024-09-10T19:24:23.216 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-10T19:24:23.216 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.25622", 2024-09-10T19:24:23.216 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.183:0/3685189199", 2024-09-10T19:24:23.216 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 7397 2024-09-10T19:24:23.216 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:23.216 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-10T19:24:23.216 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:23.216 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-10T19:24:23.216 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:24:22.907971+0000", 2024-09-10T19:24:23.216 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:24:23.216 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:23.216 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:23.216 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-10T19:24:23.216 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:24:22.907971+0000", 2024-09-10T19:24:23.216 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:24:23.217 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:23.217 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:23.217 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-10T19:24:23.217 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:24:22.907970+0000", 2024-09-10T19:24:23.217 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4294967295.9999995 2024-09-10T19:24:23.217 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:23.217 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:23.217 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-10T19:24:23.217 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:24:22.908018+0000", 2024-09-10T19:24:23.217 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.7917000000000002e-05 2024-09-10T19:24:23.217 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:23.217 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:23.217 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-10T19:24:23.217 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:24:22.908029+0000", 2024-09-10T19:24:23.218 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.0467e-05 2024-09-10T19:24:23.218 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:23.218 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:23.218 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-10T19:24:23.218 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:24:22.908035+0000", 2024-09-10T19:24:23.218 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.6679999999999998e-06 2024-09-10T19:24:23.218 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:23.218 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:23.218 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-10T19:24:23.218 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:24:22.909515+0000", 2024-09-10T19:24:23.218 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0014800939999999999 2024-09-10T19:24:23.218 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:23.218 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:23.218 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for peered", 2024-09-10T19:24:23.219 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:24:22.909520+0000", 2024-09-10T19:24:23.219 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.6590000000000004e-06 2024-09-10T19:24:23.219 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:24:23.219 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:24:23.219 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:24:23.219 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:23.219 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:23.219 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.25622.0:7401 10.4 10.77620144 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e301)", 2024-09-10T19:24:23.219 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-10T19:24:22.912182+0000", 2024-09-10T19:24:23.219 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 0.29270458700000002, 2024-09-10T19:24:23.219 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.29287607999999998, 2024-09-10T19:24:23.219 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-10T19:24:23.219 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "waiting for peered", 2024-09-10T19:24:23.219 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-10T19:24:23.219 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.25622", 2024-09-10T19:24:23.219 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.183:0/3685189199", 2024-09-10T19:24:23.220 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 7401 2024-09-10T19:24:23.220 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:23.220 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-10T19:24:23.220 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:23.220 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-10T19:24:23.220 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:24:22.912182+0000", 2024-09-10T19:24:23.220 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:24:23.220 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:23.220 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:23.220 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-10T19:24:23.220 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:24:22.912182+0000", 2024-09-10T19:24:23.220 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:24:23.220 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:23.220 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:23.220 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-10T19:24:23.221 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:24:22.912181+0000", 2024-09-10T19:24:23.221 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4294967295.9999986 2024-09-10T19:24:23.221 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:23.221 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:23.221 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-10T19:24:23.221 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:24:22.912235+0000", 2024-09-10T19:24:23.221 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.3838999999999997e-05 2024-09-10T19:24:23.221 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:23.221 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:23.221 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-10T19:24:23.221 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:24:22.912243+0000", 2024-09-10T19:24:23.221 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.5399999999999996e-06 2024-09-10T19:24:23.221 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:23.221 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:23.221 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-10T19:24:23.222 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:24:22.912250+0000", 2024-09-10T19:24:23.222 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.7129999999999999e-06 2024-09-10T19:24:23.222 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:23.222 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:23.222 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-10T19:24:23.222 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:24:22.912408+0000", 2024-09-10T19:24:23.222 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000158412 2024-09-10T19:24:23.222 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:23.222 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:23.222 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for peered", 2024-09-10T19:24:23.222 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:24:22.912416+0000", 2024-09-10T19:24:23.222 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.7160000000000003e-06 2024-09-10T19:24:23.222 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:24:23.222 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:24:23.222 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:24:23.222 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:23.223 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:23.223 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.25622.0:7402 10.5 10.2131c465 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e301)", 2024-09-10T19:24:23.223 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-10T19:24:22.912261+0000", 2024-09-10T19:24:23.223 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 0.29262583199999997, 2024-09-10T19:24:23.223 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.292832439, 2024-09-10T19:24:23.223 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-10T19:24:23.223 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "waiting for peered", 2024-09-10T19:24:23.223 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-10T19:24:23.223 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.25622", 2024-09-10T19:24:23.223 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.183:0/3685189199", 2024-09-10T19:24:23.223 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 7402 2024-09-10T19:24:23.223 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:23.223 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-10T19:24:23.223 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:23.223 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-10T19:24:23.224 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:24:22.912261+0000", 2024-09-10T19:24:23.224 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:24:23.224 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:23.224 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:23.224 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-10T19:24:23.224 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:24:22.912261+0000", 2024-09-10T19:24:23.224 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:24:23.224 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:23.224 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:23.224 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-10T19:24:23.224 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:24:22.912260+0000", 2024-09-10T19:24:23.224 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4294967295.9999995 2024-09-10T19:24:23.224 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:23.224 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:23.224 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-10T19:24:23.225 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:24:22.912502+0000", 2024-09-10T19:24:23.225 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000241458 2024-09-10T19:24:23.225 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:23.225 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:23.225 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-10T19:24:23.225 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:24:22.912511+0000", 2024-09-10T19:24:23.225 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.0510000000000002e-06 2024-09-10T19:24:23.225 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:23.225 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:23.225 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-10T19:24:23.225 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:24:22.912517+0000", 2024-09-10T19:24:23.225 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.015e-06 2024-09-10T19:24:23.225 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:23.225 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:23.225 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-10T19:24:23.226 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:24:22.912546+0000", 2024-09-10T19:24:23.226 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.8824999999999999e-05 2024-09-10T19:24:23.226 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:23.226 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:23.226 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for peered", 2024-09-10T19:24:23.226 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:24:22.912550+0000", 2024-09-10T19:24:23.226 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.5560000000000001e-06 2024-09-10T19:24:23.226 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:24:23.226 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:24:23.226 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:24:23.226 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:23.226 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:23.226 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.25622.0:7403 10.8 10.da3107a8 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e301)", 2024-09-10T19:24:23.226 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-10T19:24:22.913249+0000", 2024-09-10T19:24:23.226 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 0.29163755899999999, 2024-09-10T19:24:23.227 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.29187726800000002, 2024-09-10T19:24:23.227 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-10T19:24:23.227 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "waiting for peered", 2024-09-10T19:24:23.227 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-10T19:24:23.227 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.25622", 2024-09-10T19:24:23.227 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.183:0/3685189199", 2024-09-10T19:24:23.227 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 7403 2024-09-10T19:24:23.227 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:23.227 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-10T19:24:23.227 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:23.227 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-10T19:24:23.227 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:24:22.913249+0000", 2024-09-10T19:24:23.227 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:24:23.227 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:23.228 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:23.228 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-10T19:24:23.228 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:24:22.913249+0000", 2024-09-10T19:24:23.228 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:24:23.228 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:23.228 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:23.228 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-10T19:24:23.228 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:24:22.913248+0000", 2024-09-10T19:24:23.228 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4294967295.999999 2024-09-10T19:24:23.228 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:23.228 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:23.228 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-10T19:24:23.228 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:24:22.913299+0000", 2024-09-10T19:24:23.228 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.0844000000000003e-05 2024-09-10T19:24:23.228 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:23.228 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:23.229 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-10T19:24:23.229 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:24:22.913310+0000", 2024-09-10T19:24:23.229 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.1270000000000001e-05 2024-09-10T19:24:23.229 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:23.229 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:23.229 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-10T19:24:23.229 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:24:22.913316+0000", 2024-09-10T19:24:23.229 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.6890000000000001e-06 2024-09-10T19:24:23.229 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:23.229 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:23.229 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-10T19:24:23.229 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:24:22.913368+0000", 2024-09-10T19:24:23.229 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.2184e-05 2024-09-10T19:24:23.229 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:23.229 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:23.229 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for peered", 2024-09-10T19:24:23.230 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:24:22.913374+0000", 2024-09-10T19:24:23.230 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.7119999999999997e-06 2024-09-10T19:24:23.230 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:24:23.230 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:24:23.230 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:24:23.230 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:24:23.230 INFO:teuthology.orchestra.run.smithi183.stdout: ], 2024-09-10T19:24:23.230 INFO:teuthology.orchestra.run.smithi183.stdout: "num_ops": 6 2024-09-10T19:24:23.230 INFO:teuthology.orchestra.run.smithi183.stdout:} 2024-09-10T19:24:23.230 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:24:23.231 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:24:22 smithi191 ceph-mon[32884]: pgmap v638: 158 pgs: 24 active+clean+remapped, 31 active+undersized, 14 active+undersized+degraded, 89 active+clean; 687 KiB data, 9.3 GiB used, 617 GiB / 626 GiB avail; 639 B/s rd, 0 op/s; 45/591 objects degraded (7.614%); 28/591 objects misplaced (4.738%) 2024-09-10T19:24:23.231 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:24:22 smithi191 ceph-mon[32884]: from='osd.0 [v2:172.21.15.183:6810/3346539414,v1:172.21.15.183:6811/3346539414]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-09-10T19:24:23.231 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:24:22 smithi191 ceph-mon[32884]: osdmap e300: 8 total, 7 up, 7 in 2024-09-10T19:24:23.231 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:24:22 smithi191 ceph-mon[32884]: from='osd.0 [v2:172.21.15.183:6810/3346539414,v1:172.21.15.183:6811/3346539414]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi183", "root=default"]}]: dispatch 2024-09-10T19:24:23.251 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:24:23.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:24:23.252 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:24:23.353 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:24:23.375 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:24:23.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:24:23.376 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:24:23.477 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:24:23.500 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:24:23.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:24:23.501 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:24:23.517 DEBUG:teuthology.orchestra.run.smithi191:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph daemon osd.5 dump_ops_in_flight 2024-09-10T19:24:23.603 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:24:23.624 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:24:23.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:24:23.625 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:24:23.726 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:24:23.748 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:24:23.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:24:23.749 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:24:23.767 INFO:teuthology.orchestra.run.smithi191.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.b/config 2024-09-10T19:24:23.850 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:24:23.853 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: 3 16 7403 7387 153.879 6.75 0.00350807 0.00441759 2024-09-10T19:24:23.871 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:24:23.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:24:23.872 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:24:23.973 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:24:23.995 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:24:23.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:24:23.996 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:24:24.097 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:24:24.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:24:24.118 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:24:24.118 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:24:24.146 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:24:23 smithi191 ceph-mon[32884]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-09-10T19:24:24.147 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:24:23 smithi191 ceph-mon[32884]: osd.0 [v2:172.21.15.183:6810/3346539414,v1:172.21.15.183:6811/3346539414] boot 2024-09-10T19:24:24.147 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:24:23 smithi191 ceph-mon[32884]: osdmap e301: 8 total, 8 up, 7 in 2024-09-10T19:24:24.147 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:24:23 smithi191 ceph-mon[32884]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-10T19:24:24.147 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:24:23 smithi191 ceph-mon[32884]: osdmap e302: 8 total, 8 up, 7 in 2024-09-10T19:24:24.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:24:23 smithi183 ceph-mon[29736]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-09-10T19:24:24.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:24:23 smithi183 ceph-mon[29736]: osd.0 [v2:172.21.15.183:6810/3346539414,v1:172.21.15.183:6811/3346539414] boot 2024-09-10T19:24:24.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:24:23 smithi183 ceph-mon[29736]: osdmap e301: 8 total, 8 up, 7 in 2024-09-10T19:24:24.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:24:23 smithi183 ceph-mon[29736]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-10T19:24:24.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:24:23 smithi183 ceph-mon[29736]: osdmap e302: 8 total, 8 up, 7 in 2024-09-10T19:24:24.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:24:23 smithi183 ceph-mon[41562]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-09-10T19:24:24.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:24:23 smithi183 ceph-mon[41562]: osd.0 [v2:172.21.15.183:6810/3346539414,v1:172.21.15.183:6811/3346539414] boot 2024-09-10T19:24:24.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:24:23 smithi183 ceph-mon[41562]: osdmap e301: 8 total, 8 up, 7 in 2024-09-10T19:24:24.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:24:23 smithi183 ceph-mon[41562]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-10T19:24:24.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:24:23 smithi183 ceph-mon[41562]: osdmap e302: 8 total, 8 up, 7 in 2024-09-10T19:24:24.219 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:24:24.241 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:24:24.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:24:24.242 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:24:24.343 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:24:24.365 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:24:24.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:24:24.366 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:24:24.467 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:24:24.488 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:24:24.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:24:24.489 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:24:24.530 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:24:24.590 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:24:24.613 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:24:24.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:24:24.614 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:24:24.714 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:24:24.736 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:24:24.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:24:24.737 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:24:24.838 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:24:24.853 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: 4 16 7403 7387 115.41 0 - 0.00441759 2024-09-10T19:24:24.860 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:24:24.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:24:24.861 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:24:24.962 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:24:24.994 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:24:24.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:24:24.995 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:24:25.027 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:24:24 smithi183 ceph-mon[29736]: purged_snaps scrub starts 2024-09-10T19:24:25.027 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:24:24 smithi183 ceph-mon[29736]: purged_snaps scrub ok 2024-09-10T19:24:25.027 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:24:24 smithi183 ceph-mon[29736]: pgmap v641: 158 pgs: 24 active+clean+remapped, 30 active+undersized, 15 active+undersized+degraded, 89 active+clean; 25 MiB data, 8.7 GiB used, 617 GiB / 626 GiB avail; 853 B/s rd, 4.0 MiB/s wr, 64 op/s; 428/1740 objects degraded (24.598%); 28/1740 objects misplaced (1.609%) 2024-09-10T19:24:25.027 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:24:24 smithi183 ceph-mon[29736]: Health check update: Degraded data redundancy: 428/1740 objects degraded (24.598%), 15 pgs degraded, 35 pgs undersized (PG_DEGRADED) 2024-09-10T19:24:25.027 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:24:24 smithi183 ceph-mon[29736]: osdmap e303: 8 total, 8 up, 7 in 2024-09-10T19:24:25.027 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:24:24 smithi183 ceph-mon[41562]: purged_snaps scrub starts 2024-09-10T19:24:25.027 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:24:24 smithi183 ceph-mon[41562]: purged_snaps scrub ok 2024-09-10T19:24:25.027 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:24:24 smithi183 ceph-mon[41562]: pgmap v641: 158 pgs: 24 active+clean+remapped, 30 active+undersized, 15 active+undersized+degraded, 89 active+clean; 25 MiB data, 8.7 GiB used, 617 GiB / 626 GiB avail; 853 B/s rd, 4.0 MiB/s wr, 64 op/s; 428/1740 objects degraded (24.598%); 28/1740 objects misplaced (1.609%) 2024-09-10T19:24:25.028 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:24:24 smithi183 ceph-mon[41562]: Health check update: Degraded data redundancy: 428/1740 objects degraded (24.598%), 15 pgs degraded, 35 pgs undersized (PG_DEGRADED) 2024-09-10T19:24:25.028 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:24:24 smithi183 ceph-mon[41562]: osdmap e303: 8 total, 8 up, 7 in 2024-09-10T19:24:25.096 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:24:25.137 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:24:25.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:24:25.138 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:24:25.171 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:24:24 smithi191 ceph-mon[32884]: purged_snaps scrub starts 2024-09-10T19:24:25.171 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:24:24 smithi191 ceph-mon[32884]: purged_snaps scrub ok 2024-09-10T19:24:25.171 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:24:24 smithi191 ceph-mon[32884]: pgmap v641: 158 pgs: 24 active+clean+remapped, 30 active+undersized, 15 active+undersized+degraded, 89 active+clean; 25 MiB data, 8.7 GiB used, 617 GiB / 626 GiB avail; 853 B/s rd, 4.0 MiB/s wr, 64 op/s; 428/1740 objects degraded (24.598%); 28/1740 objects misplaced (1.609%) 2024-09-10T19:24:25.171 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:24:24 smithi191 ceph-mon[32884]: Health check update: Degraded data redundancy: 428/1740 objects degraded (24.598%), 15 pgs degraded, 35 pgs undersized (PG_DEGRADED) 2024-09-10T19:24:25.171 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:24:24 smithi191 ceph-mon[32884]: osdmap e303: 8 total, 8 up, 7 in 2024-09-10T19:24:25.239 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:24:25.278 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:24:25.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:24:25.279 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:24:25.380 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:24:25.422 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:24:25.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:24:25.423 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:24:25.523 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:24:25.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:24:25.568 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:24:25.568 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:24:25.669 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:24:25.679 INFO:teuthology.orchestra.run.smithi191.stdout:ERROR: (22) Invalid argument 2024-09-10T19:24:25.679 INFO:teuthology.orchestra.run.smithi191.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-10T19:24:25.700 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:24:25.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:24:25.701 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:24:25.801 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:24:25.840 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:24:25.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:24:25.841 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:24:25.853 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: 5 16 9699 9683 121.025 71.75 0.00499905 0.00825616 2024-09-10T19:24:25.942 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:24:25.978 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:24:25.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:24:25.979 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:24:26.079 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:24:26.105 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:24:25 smithi183 ceph-mon[29736]: from='client.? 172.21.15.183:0/2696505246' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-10T19:24:26.105 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:24:25 smithi183 ceph-mon[41562]: from='client.? 172.21.15.183:0/2696505246' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-10T19:24:26.113 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:24:26.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:24:26.114 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:24:26.215 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:24:26.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:24:25 smithi191 ceph-mon[32884]: from='client.? 172.21.15.183:0/2696505246' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-10T19:24:26.246 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:24:26.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:24:26.248 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:24:26.348 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:24:26.349 INFO:teuthology.orchestra.run.smithi183.stderr:nodeep-scrub is set 2024-09-10T19:24:26.389 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:24:26.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:24:26.390 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:24:26.490 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:24:26.521 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:24:26.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:24:26.522 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:24:26.623 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:24:26.648 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:24:26.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:24:26.650 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:24:26.730 DEBUG:teuthology.orchestra.run.smithi191:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph daemon osd.5 dump_blocked_ops 2024-09-10T19:24:26.751 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:24:26.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:24:26.814 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:24:26.814 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:24:26.853 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: 6 16 12156 12140 126.446 153.562 0.00554432 0.00789812 2024-09-10T19:24:26.915 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:24:26.946 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:24:26.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:24:26.948 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:24:26.965 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:24:26 smithi191 ceph-mon[32884]: pgmap v644: 158 pgs: 7 active+recovery_wait+undersized+degraded+remapped, 16 active+clean+remapped, 2 active+recovering+undersized+degraded+remapped, 6 active+undersized, 6 active+undersized+degraded, 121 active+clean; 479 MiB data, 9.7 GiB used, 616 GiB / 626 GiB avail; 1.2 KiB/s rd, 120 MiB/s wr, 1.92k op/s; 4475/23571 objects degraded (18.985%); 27/23571 objects misplaced (0.115%); 1.3 MiB/s, 20 objects/s recovering 2024-09-10T19:24:26.966 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:24:26 smithi191 ceph-mon[32884]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T19:24:26.966 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:24:26 smithi191 ceph-mon[32884]: from='client.? 172.21.15.183:0/2696505246' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-10T19:24:26.966 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:24:26 smithi191 ceph-mon[32884]: osdmap e304: 8 total, 8 up, 7 in 2024-09-10T19:24:27.049 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:24:27.075 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:24:27.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:24:27.076 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:24:27.150 INFO:teuthology.orchestra.run.smithi191.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.b/config 2024-09-10T19:24:27.177 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:24:27.181 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:24:26 smithi183 ceph-mon[41562]: pgmap v644: 158 pgs: 7 active+recovery_wait+undersized+degraded+remapped, 16 active+clean+remapped, 2 active+recovering+undersized+degraded+remapped, 6 active+undersized, 6 active+undersized+degraded, 121 active+clean; 479 MiB data, 9.7 GiB used, 616 GiB / 626 GiB avail; 1.2 KiB/s rd, 120 MiB/s wr, 1.92k op/s; 4475/23571 objects degraded (18.985%); 27/23571 objects misplaced (0.115%); 1.3 MiB/s, 20 objects/s recovering 2024-09-10T19:24:27.181 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:24:26 smithi183 ceph-mon[41562]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T19:24:27.181 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:24:26 smithi183 ceph-mon[41562]: from='client.? 172.21.15.183:0/2696505246' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-10T19:24:27.181 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:24:26 smithi183 ceph-mon[41562]: osdmap e304: 8 total, 8 up, 7 in 2024-09-10T19:24:27.182 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:24:26 smithi183 ceph-mon[29736]: pgmap v644: 158 pgs: 7 active+recovery_wait+undersized+degraded+remapped, 16 active+clean+remapped, 2 active+recovering+undersized+degraded+remapped, 6 active+undersized, 6 active+undersized+degraded, 121 active+clean; 479 MiB data, 9.7 GiB used, 616 GiB / 626 GiB avail; 1.2 KiB/s rd, 120 MiB/s wr, 1.92k op/s; 4475/23571 objects degraded (18.985%); 27/23571 objects misplaced (0.115%); 1.3 MiB/s, 20 objects/s recovering 2024-09-10T19:24:27.182 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:24:26 smithi183 ceph-mon[29736]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T19:24:27.182 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:24:26 smithi183 ceph-mon[29736]: from='client.? 172.21.15.183:0/2696505246' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-10T19:24:27.182 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:24:26 smithi183 ceph-mon[29736]: osdmap e304: 8 total, 8 up, 7 in 2024-09-10T19:24:27.204 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:24:27.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:24:27.205 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:24:27.306 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:24:27.328 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:24:27.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:24:27.329 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:24:27.430 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:24:27.453 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:24:27.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:24:27.454 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:24:27.555 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:24:27.579 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:24:27.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:24:27.580 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:24:27.633 INFO:journalctl@ceph.mgr.x.smithi191.stdout:Sep 10 19:24:27 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-x[34967]: ::ffff:172.21.15.191 - - [10/Sep/2024:19:24:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T19:24:27.681 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:24:27.704 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:24:27.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:24:27.705 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:24:27.806 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:24:27.829 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:24:27.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:24:27.831 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:24:27.853 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: 7 16 12580 12564 112.168 26.5 0.00191464 0.00784358 2024-09-10T19:24:27.931 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:24:27.961 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:24:27.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:24:27.962 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:24:28.063 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:24:28.065 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:24:27 smithi183 ceph-mon[29736]: osdmap e305: 8 total, 8 up, 7 in 2024-09-10T19:24:28.065 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:24:27 smithi183 ceph-mon[41562]: osdmap e305: 8 total, 8 up, 7 in 2024-09-10T19:24:28.070 INFO:teuthology.orchestra.run.smithi183.stdout:osd.0: {} 2024-09-10T19:24:28.070 INFO:teuthology.orchestra.run.smithi183.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-10T19:24:28.086 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:24:28.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:24:28.087 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:24:28.090 INFO:teuthology.orchestra.run.smithi183.stdout:osd.1: {} 2024-09-10T19:24:28.090 INFO:teuthology.orchestra.run.smithi183.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-10T19:24:28.109 INFO:teuthology.orchestra.run.smithi183.stdout:osd.2: {} 2024-09-10T19:24:28.109 INFO:teuthology.orchestra.run.smithi183.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-10T19:24:28.128 INFO:teuthology.orchestra.run.smithi183.stdout:osd.3: {} 2024-09-10T19:24:28.129 INFO:teuthology.orchestra.run.smithi183.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-10T19:24:28.147 INFO:teuthology.orchestra.run.smithi183.stdout:osd.4: {} 2024-09-10T19:24:28.147 INFO:teuthology.orchestra.run.smithi183.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-10T19:24:28.165 INFO:teuthology.orchestra.run.smithi183.stdout:osd.5: {} 2024-09-10T19:24:28.165 INFO:teuthology.orchestra.run.smithi183.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-10T19:24:28.182 INFO:teuthology.orchestra.run.smithi183.stdout:osd.6: {} 2024-09-10T19:24:28.183 INFO:teuthology.orchestra.run.smithi183.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-10T19:24:28.187 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:24:28.201 INFO:teuthology.orchestra.run.smithi183.stdout:osd.7: {} 2024-09-10T19:24:28.201 INFO:teuthology.orchestra.run.smithi183.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-10T19:24:28.211 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:24:28.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:24:28.212 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:24:28.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:24:27 smithi191 ceph-mon[32884]: osdmap e305: 8 total, 8 up, 7 in 2024-09-10T19:24:28.314 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:24:28.337 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:24:28.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:24:28.338 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:24:28.439 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:24:28.460 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:24:28.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:24:28.461 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:24:28.562 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:24:28.585 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:24:28.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:24:28.588 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:24:28.689 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:24:28.714 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:24:28.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:24:28.715 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:24:28.816 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:24:28.838 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:24:28.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:24:28.839 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:24:28.853 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: 8 16 12626 12610 98.5065 2.875 0.00178958 0.00900739 2024-09-10T19:24:28.939 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:24:28.971 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:24:28.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:24:28.972 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:24:29.073 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:24:29.094 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:24:29.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:24:29.095 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:24:29.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:24:28 smithi183 ceph-mon[29736]: pgmap v647: 158 pgs: 1 peering, 7 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+degraded+remapped, 148 active+clean; 506 MiB data, 10 GiB used, 616 GiB / 626 GiB avail; 511 B/s rd, 120 MiB/s wr, 1.93k op/s; 4073/24852 objects degraded (16.389%); 9.5 MiB/s, 151 objects/s recovering 2024-09-10T19:24:29.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:24:28 smithi183 ceph-mon[29736]: osdmap e306: 8 total, 8 up, 7 in 2024-09-10T19:24:29.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:24:28 smithi183 ceph-mon[29736]: from='client.? 172.21.15.183:0/641513436' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T19:24:29.182 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:24:28 smithi183 ceph-mon[29736]: from='mgr.14556 ' entity='mgr.y' 2024-09-10T19:24:29.182 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:24:28 smithi183 ceph-mon[29736]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:24:29.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:24:28 smithi183 ceph-mon[41562]: pgmap v647: 158 pgs: 1 peering, 7 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+degraded+remapped, 148 active+clean; 506 MiB data, 10 GiB used, 616 GiB / 626 GiB avail; 511 B/s rd, 120 MiB/s wr, 1.93k op/s; 4073/24852 objects degraded (16.389%); 9.5 MiB/s, 151 objects/s recovering 2024-09-10T19:24:29.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:24:28 smithi183 ceph-mon[41562]: osdmap e306: 8 total, 8 up, 7 in 2024-09-10T19:24:29.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:24:28 smithi183 ceph-mon[41562]: from='client.? 172.21.15.183:0/641513436' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T19:24:29.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:24:28 smithi183 ceph-mon[41562]: from='mgr.14556 ' entity='mgr.y' 2024-09-10T19:24:29.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:24:28 smithi183 ceph-mon[41562]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:24:29.196 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:24:29.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:24:28 smithi191 ceph-mon[32884]: pgmap v647: 158 pgs: 1 peering, 7 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+degraded+remapped, 148 active+clean; 506 MiB data, 10 GiB used, 616 GiB / 626 GiB avail; 511 B/s rd, 120 MiB/s wr, 1.93k op/s; 4073/24852 objects degraded (16.389%); 9.5 MiB/s, 151 objects/s recovering 2024-09-10T19:24:29.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:24:28 smithi191 ceph-mon[32884]: osdmap e306: 8 total, 8 up, 7 in 2024-09-10T19:24:29.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:24:28 smithi191 ceph-mon[32884]: from='client.? 172.21.15.183:0/641513436' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T19:24:29.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:24:28 smithi191 ceph-mon[32884]: from='mgr.14556 ' entity='mgr.y' 2024-09-10T19:24:29.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:24:28 smithi191 ceph-mon[32884]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:24:29.217 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:24:29.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:24:29.218 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:24:29.319 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:24:29.342 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:24:29.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:24:29.343 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:24:29.443 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:24:29.465 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:24:29.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:24:29.466 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:24:29.521 INFO:tasks.thrashosds.thrasher:in_osds: [0, 3, 4, 5, 6, 7, 1] out_osds: [2] dead_osds: [] live_osds: [2, 3, 4, 5, 6, 7, 1, 0] 2024-09-10T19:24:29.521 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-10T19:24:29.521 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap_items 2024-09-10T19:24:29.521 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph osd dump -f json-pretty 2024-09-10T19:24:29.522 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph osd unset noscrub 2024-09-10T19:24:29.568 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:24:29.590 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:24:29.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:24:29.591 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:24:29.692 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:24:29.713 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:24:29.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:24:29.714 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:24:29.779 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:24:29.781 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:24:29.815 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:24:29.836 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:24:29.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:24:29.837 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:24:29.853 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: 9 16 12680 12664 87.9361 3.375 0.00151866 0.0102348 2024-09-10T19:24:29.934 INFO:journalctl@ceph.iscsi.iscsi.a.smithi191.stdout:Sep 10 19:24:29 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-iscsi-iscsi-a[61816]: debug there is no tcmu-runner data available 2024-09-10T19:24:29.938 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:24:29.959 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:24:29.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:24:29.959 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:24:30.060 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:24:30.081 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:24:30.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:24:30.082 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:24:30.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:24:29 smithi183 ceph-mon[29736]: osdmap e307: 8 total, 8 up, 7 in 2024-09-10T19:24:30.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:24:29 smithi183 ceph-mon[29736]: Health check update: Degraded data redundancy: 4073/24852 objects degraded (16.389%), 9 pgs degraded (PG_DEGRADED) 2024-09-10T19:24:30.182 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:24:29 smithi183 ceph-mon[29736]: osdmap e308: 8 total, 8 up, 7 in 2024-09-10T19:24:30.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:24:29 smithi183 ceph-mon[41562]: osdmap e307: 8 total, 8 up, 7 in 2024-09-10T19:24:30.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:24:29 smithi183 ceph-mon[41562]: Health check update: Degraded data redundancy: 4073/24852 objects degraded (16.389%), 9 pgs degraded (PG_DEGRADED) 2024-09-10T19:24:30.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:24:29 smithi183 ceph-mon[41562]: osdmap e308: 8 total, 8 up, 7 in 2024-09-10T19:24:30.183 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:24:30.205 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:24:30.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:24:30.206 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:24:30.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:24:29 smithi191 ceph-mon[32884]: osdmap e307: 8 total, 8 up, 7 in 2024-09-10T19:24:30.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:24:29 smithi191 ceph-mon[32884]: Health check update: Degraded data redundancy: 4073/24852 objects degraded (16.389%), 9 pgs degraded (PG_DEGRADED) 2024-09-10T19:24:30.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:24:29 smithi191 ceph-mon[32884]: osdmap e308: 8 total, 8 up, 7 in 2024-09-10T19:24:30.307 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:24:30.328 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:24:30.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:24:30.329 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:24:30.429 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:24:30.450 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:24:30.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:24:30.451 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:24:30.552 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:24:30.573 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:24:30.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:24:30.574 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:24:30.675 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:24:30.695 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:24:30.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:24:30.696 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:24:30.797 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:24:30.817 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:24:30.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:24:30.818 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:24:30.853 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: 10 16 12828 12812 80.0674 9.25 0.00126486 0.0113332 2024-09-10T19:24:30.897 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-10T19:24:30.918 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:24:30.968 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:24:30.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:24:30.969 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:24:31.011 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:24:30 smithi183 ceph-mon[41562]: from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:24:31.011 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:24:30 smithi183 ceph-mon[41562]: pgmap v650: 158 pgs: 1 peering, 7 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+degraded+remapped, 148 active+clean; 506 MiB data, 10 GiB used, 616 GiB / 626 GiB avail; 511 B/s rd, 6.7 MiB/s wr, 107 op/s; 4073/24852 objects degraded (16.389%); 8.2 MiB/s, 131 objects/s recovering 2024-09-10T19:24:31.011 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:24:30 smithi183 ceph-mon[41562]: osdmap e309: 8 total, 8 up, 7 in 2024-09-10T19:24:31.011 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:24:30 smithi183 ceph-mon[29736]: from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:24:31.011 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:24:30 smithi183 ceph-mon[29736]: pgmap v650: 158 pgs: 1 peering, 7 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+degraded+remapped, 148 active+clean; 506 MiB data, 10 GiB used, 616 GiB / 626 GiB avail; 511 B/s rd, 6.7 MiB/s wr, 107 op/s; 4073/24852 objects degraded (16.389%); 8.2 MiB/s, 131 objects/s recovering 2024-09-10T19:24:31.011 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:24:30 smithi183 ceph-mon[29736]: osdmap e309: 8 total, 8 up, 7 in 2024-09-10T19:24:31.070 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:24:31.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:24:31.109 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:24:31.109 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:24:31.210 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:24:31.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:24:30 smithi191 ceph-mon[32884]: from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:24:31.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:24:30 smithi191 ceph-mon[32884]: pgmap v650: 158 pgs: 1 peering, 7 active+recovery_wait+undersized+degraded+remapped, 2 active+recovering+undersized+degraded+remapped, 148 active+clean; 506 MiB data, 10 GiB used, 616 GiB / 626 GiB avail; 511 B/s rd, 6.7 MiB/s wr, 107 op/s; 4073/24852 objects degraded (16.389%); 8.2 MiB/s, 131 objects/s recovering 2024-09-10T19:24:31.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:24:30 smithi191 ceph-mon[32884]: osdmap e309: 8 total, 8 up, 7 in 2024-09-10T19:24:31.235 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:24:31.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:24:31.236 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:24:31.336 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:24:31.372 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:24:31.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:24:31.373 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:24:31.473 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:24:31.503 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:24:31.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:24:31.504 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:24:31.604 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:24:31.641 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:24:31.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:24:31.642 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:24:31.681 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Sep 10 19:24:31 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-y[29989]: ::ffff:172.21.15.191 - - [10/Sep/2024:19:24:31] "GET /metrics HTTP/1.1" 200 224435 "" "Prometheus/2.43.0" 2024-09-10T19:24:31.743 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:24:31.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:24:31.776 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:24:31.776 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:24:31.853 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: 11 16 15168 15152 86.0829 146.25 0.00589951 0.0116104 2024-09-10T19:24:31.876 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:24:31.909 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:24:31.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:24:31.911 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:24:31.993 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:24:32.012 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:24:32.046 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:24:32.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:24:32.047 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:24:32.148 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:24:32.179 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:24:32.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:24:32.180 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:24:32.281 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:24:32.308 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:24:32.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:24:32.310 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:24:32.410 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:24:32.443 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:24:32.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:24:32.444 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:24:32.544 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:24:32.570 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:24:32.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:24:32.571 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:24:32.672 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:24:32.701 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:24:32.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:24:32.703 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:24:32.803 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:24:32.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:24:32.833 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:24:32.833 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:24:32.853 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: 12 16 17425 17409 90.6634 141.062 0.00305538 0.0110247 2024-09-10T19:24:32.935 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:24:32.974 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:24:32.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:24:32.976 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:24:33.076 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:24:33.113 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:24:33.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:24:33.114 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:24:33.157 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:24:32 smithi183 ceph-mon[29736]: pgmap v653: 158 pgs: 2 peering, 3 active+recovery_wait+undersized+degraded+remapped, 153 active+clean; 741 MiB data, 11 GiB used, 615 GiB / 626 GiB avail; 59 MiB/s wr, 939 op/s; 1338/36126 objects degraded (3.704%); 43 MiB/s, 683 objects/s recovering 2024-09-10T19:24:33.157 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:24:32 smithi183 ceph-mon[29736]: purged_snaps scrub starts 2024-09-10T19:24:33.157 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:24:32 smithi183 ceph-mon[29736]: purged_snaps scrub ok 2024-09-10T19:24:33.157 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:24:32 smithi183 ceph-mon[41562]: pgmap v653: 158 pgs: 2 peering, 3 active+recovery_wait+undersized+degraded+remapped, 153 active+clean; 741 MiB data, 11 GiB used, 615 GiB / 626 GiB avail; 59 MiB/s wr, 939 op/s; 1338/36126 objects degraded (3.704%); 43 MiB/s, 683 objects/s recovering 2024-09-10T19:24:33.157 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:24:32 smithi183 ceph-mon[41562]: purged_snaps scrub starts 2024-09-10T19:24:33.158 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:24:32 smithi183 ceph-mon[41562]: purged_snaps scrub ok 2024-09-10T19:24:33.215 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:24:33.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:24:32 smithi191 ceph-mon[32884]: pgmap v653: 158 pgs: 2 peering, 3 active+recovery_wait+undersized+degraded+remapped, 153 active+clean; 741 MiB data, 11 GiB used, 615 GiB / 626 GiB avail; 59 MiB/s wr, 939 op/s; 1338/36126 objects degraded (3.704%); 43 MiB/s, 683 objects/s recovering 2024-09-10T19:24:33.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:24:32 smithi191 ceph-mon[32884]: purged_snaps scrub starts 2024-09-10T19:24:33.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:24:32 smithi191 ceph-mon[32884]: purged_snaps scrub ok 2024-09-10T19:24:33.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:24:33.253 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:24:33.253 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:24:33.354 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:24:33.375 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:24:33.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:24:33.376 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:24:33.477 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:24:33.497 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:24:33.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:24:33.498 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:24:33.600 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:24:33.636 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:24:33.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:24:33.637 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:24:33.738 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:24:33.768 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:24:33.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:24:33.769 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:24:33.853 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: 13 16 18955 18939 91.0445 95.625 0.00385533 0.0109773 2024-09-10T19:24:33.870 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:24:33.908 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:24:33.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:24:33.909 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:24:34.010 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:24:34.034 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:24:34.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:24:34.035 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:24:34.135 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:24:34.171 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:24:34.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:24:34.173 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:24:34.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:24:33 smithi191 ceph-mon[32884]: osdmap e310: 8 total, 8 up, 7 in 2024-09-10T19:24:34.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:24:33 smithi191 ceph-mon[32884]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T19:24:34.274 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:24:34.306 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:24:34.306 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:24:33 smithi183 ceph-mon[29736]: osdmap e310: 8 total, 8 up, 7 in 2024-09-10T19:24:34.306 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:24:33 smithi183 ceph-mon[29736]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T19:24:34.307 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:24:33 smithi183 ceph-mon[41562]: osdmap e310: 8 total, 8 up, 7 in 2024-09-10T19:24:34.307 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:24:33 smithi183 ceph-mon[41562]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T19:24:34.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:24:34.308 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:24:34.408 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:24:34.440 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:24:34.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:24:34.442 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:24:34.447 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-09-10T19:24:34.447 INFO:teuthology.orchestra.run.smithi183.stdout:{ 2024-09-10T19:24:34.447 INFO:teuthology.orchestra.run.smithi183.stdout: "epoch": 310, 2024-09-10T19:24:34.447 INFO:teuthology.orchestra.run.smithi183.stdout: "fsid": "320f6df8-6fa7-11ef-bcea-c7b262605968", 2024-09-10T19:24:34.448 INFO:teuthology.orchestra.run.smithi183.stdout: "created": "2024-09-10T19:05:03.688531+0000", 2024-09-10T19:24:34.448 INFO:teuthology.orchestra.run.smithi183.stdout: "modified": "2024-09-10T19:24:32.938766+0000", 2024-09-10T19:24:34.448 INFO:teuthology.orchestra.run.smithi183.stdout: "last_up_change": "2024-09-10T19:24:22.870873+0000", 2024-09-10T19:24:34.448 INFO:teuthology.orchestra.run.smithi183.stdout: "last_in_change": "2024-09-10T19:24:00.937328+0000", 2024-09-10T19:24:34.448 INFO:teuthology.orchestra.run.smithi183.stdout: "flags": "noscrub,nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-09-10T19:24:34.448 INFO:teuthology.orchestra.run.smithi183.stdout: "flags_num": 5806080, 2024-09-10T19:24:34.448 INFO:teuthology.orchestra.run.smithi183.stdout: "flags_set": [ 2024-09-10T19:24:34.448 INFO:teuthology.orchestra.run.smithi183.stdout: "nodeep-scrub", 2024-09-10T19:24:34.448 INFO:teuthology.orchestra.run.smithi183.stdout: "noscrub", 2024-09-10T19:24:34.448 INFO:teuthology.orchestra.run.smithi183.stdout: "pglog_hardlimit", 2024-09-10T19:24:34.448 INFO:teuthology.orchestra.run.smithi183.stdout: "purged_snapdirs", 2024-09-10T19:24:34.448 INFO:teuthology.orchestra.run.smithi183.stdout: "recovery_deletes", 2024-09-10T19:24:34.448 INFO:teuthology.orchestra.run.smithi183.stdout: "sortbitwise" 2024-09-10T19:24:34.448 INFO:teuthology.orchestra.run.smithi183.stdout: ], 2024-09-10T19:24:34.449 INFO:teuthology.orchestra.run.smithi183.stdout: "crush_version": 21, 2024-09-10T19:24:34.449 INFO:teuthology.orchestra.run.smithi183.stdout: "full_ratio": 0.94999998807907104, 2024-09-10T19:24:34.449 INFO:teuthology.orchestra.run.smithi183.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-09-10T19:24:34.449 INFO:teuthology.orchestra.run.smithi183.stdout: "nearfull_ratio": 0.85000002384185791, 2024-09-10T19:24:34.449 INFO:teuthology.orchestra.run.smithi183.stdout: "cluster_snapshot": "", 2024-09-10T19:24:34.449 INFO:teuthology.orchestra.run.smithi183.stdout: "pool_max": 10, 2024-09-10T19:24:34.449 INFO:teuthology.orchestra.run.smithi183.stdout: "max_osd": 8, 2024-09-10T19:24:34.449 INFO:teuthology.orchestra.run.smithi183.stdout: "require_min_compat_client": "luminous", 2024-09-10T19:24:34.449 INFO:teuthology.orchestra.run.smithi183.stdout: "min_compat_client": "luminous", 2024-09-10T19:24:34.449 INFO:teuthology.orchestra.run.smithi183.stdout: "require_osd_release": "quincy", 2024-09-10T19:24:34.449 INFO:teuthology.orchestra.run.smithi183.stdout: "pools": [ 2024-09-10T19:24:34.449 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:34.449 INFO:teuthology.orchestra.run.smithi183.stdout: "pool": 1, 2024-09-10T19:24:34.450 INFO:teuthology.orchestra.run.smithi183.stdout: "pool_name": ".mgr", 2024-09-10T19:24:34.450 INFO:teuthology.orchestra.run.smithi183.stdout: "create_time": "2024-09-10T19:08:00.456786+0000", 2024-09-10T19:24:34.450 INFO:teuthology.orchestra.run.smithi183.stdout: "flags": 1, 2024-09-10T19:24:34.450 INFO:teuthology.orchestra.run.smithi183.stdout: "flags_names": "hashpspool", 2024-09-10T19:24:34.450 INFO:teuthology.orchestra.run.smithi183.stdout: "type": 1, 2024-09-10T19:24:34.450 INFO:teuthology.orchestra.run.smithi183.stdout: "size": 3, 2024-09-10T19:24:34.450 INFO:teuthology.orchestra.run.smithi183.stdout: "min_size": 2, 2024-09-10T19:24:34.450 INFO:teuthology.orchestra.run.smithi183.stdout: "crush_rule": 0, 2024-09-10T19:24:34.450 INFO:teuthology.orchestra.run.smithi183.stdout: "peering_crush_bucket_count": 0, 2024-09-10T19:24:34.450 INFO:teuthology.orchestra.run.smithi183.stdout: "peering_crush_bucket_target": 0, 2024-09-10T19:24:34.450 INFO:teuthology.orchestra.run.smithi183.stdout: "peering_crush_bucket_barrier": 0, 2024-09-10T19:24:34.450 INFO:teuthology.orchestra.run.smithi183.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-10T19:24:34.450 INFO:teuthology.orchestra.run.smithi183.stdout: "object_hash": 2, 2024-09-10T19:24:34.450 INFO:teuthology.orchestra.run.smithi183.stdout: "pg_autoscale_mode": "off", 2024-09-10T19:24:34.450 INFO:teuthology.orchestra.run.smithi183.stdout: "pg_num": 11, 2024-09-10T19:24:34.451 INFO:teuthology.orchestra.run.smithi183.stdout: "pg_placement_num": 11, 2024-09-10T19:24:34.451 INFO:teuthology.orchestra.run.smithi183.stdout: "pg_placement_num_target": 11, 2024-09-10T19:24:34.451 INFO:teuthology.orchestra.run.smithi183.stdout: "pg_num_target": 11, 2024-09-10T19:24:34.451 INFO:teuthology.orchestra.run.smithi183.stdout: "pg_num_pending": 11, 2024-09-10T19:24:34.451 INFO:teuthology.orchestra.run.smithi183.stdout: "last_pg_merge_meta": { 2024-09-10T19:24:34.451 INFO:teuthology.orchestra.run.smithi183.stdout: "source_pgid": "0.0", 2024-09-10T19:24:34.451 INFO:teuthology.orchestra.run.smithi183.stdout: "ready_epoch": 0, 2024-09-10T19:24:34.451 INFO:teuthology.orchestra.run.smithi183.stdout: "last_epoch_started": 0, 2024-09-10T19:24:34.451 INFO:teuthology.orchestra.run.smithi183.stdout: "last_epoch_clean": 0, 2024-09-10T19:24:34.451 INFO:teuthology.orchestra.run.smithi183.stdout: "source_version": "0'0", 2024-09-10T19:24:34.451 INFO:teuthology.orchestra.run.smithi183.stdout: "target_version": "0'0" 2024-09-10T19:24:34.452 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:34.452 INFO:teuthology.orchestra.run.smithi183.stdout: "last_change": "271", 2024-09-10T19:24:34.452 INFO:teuthology.orchestra.run.smithi183.stdout: "last_force_op_resend": "0", 2024-09-10T19:24:34.452 INFO:teuthology.orchestra.run.smithi183.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-10T19:24:34.452 INFO:teuthology.orchestra.run.smithi183.stdout: "last_force_op_resend_preluminous": "242", 2024-09-10T19:24:34.452 INFO:teuthology.orchestra.run.smithi183.stdout: "auid": 0, 2024-09-10T19:24:34.452 INFO:teuthology.orchestra.run.smithi183.stdout: "snap_mode": "selfmanaged", 2024-09-10T19:24:34.452 INFO:teuthology.orchestra.run.smithi183.stdout: "snap_seq": 0, 2024-09-10T19:24:34.452 INFO:teuthology.orchestra.run.smithi183.stdout: "snap_epoch": 0, 2024-09-10T19:24:34.452 INFO:teuthology.orchestra.run.smithi183.stdout: "pool_snaps": [], 2024-09-10T19:24:34.452 INFO:teuthology.orchestra.run.smithi183.stdout: "removed_snaps": "[]", 2024-09-10T19:24:34.452 INFO:teuthology.orchestra.run.smithi183.stdout: "quota_max_bytes": 0, 2024-09-10T19:24:34.453 INFO:teuthology.orchestra.run.smithi183.stdout: "quota_max_objects": 0, 2024-09-10T19:24:34.453 INFO:teuthology.orchestra.run.smithi183.stdout: "tiers": [], 2024-09-10T19:24:34.453 INFO:teuthology.orchestra.run.smithi183.stdout: "tier_of": -1, 2024-09-10T19:24:34.453 INFO:teuthology.orchestra.run.smithi183.stdout: "read_tier": -1, 2024-09-10T19:24:34.453 INFO:teuthology.orchestra.run.smithi183.stdout: "write_tier": -1, 2024-09-10T19:24:34.453 INFO:teuthology.orchestra.run.smithi183.stdout: "cache_mode": "none", 2024-09-10T19:24:34.453 INFO:teuthology.orchestra.run.smithi183.stdout: "target_max_bytes": 0, 2024-09-10T19:24:34.453 INFO:teuthology.orchestra.run.smithi183.stdout: "target_max_objects": 0, 2024-09-10T19:24:34.453 INFO:teuthology.orchestra.run.smithi183.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-10T19:24:34.453 INFO:teuthology.orchestra.run.smithi183.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-10T19:24:34.453 INFO:teuthology.orchestra.run.smithi183.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-10T19:24:34.453 INFO:teuthology.orchestra.run.smithi183.stdout: "cache_min_flush_age": 0, 2024-09-10T19:24:34.453 INFO:teuthology.orchestra.run.smithi183.stdout: "cache_min_evict_age": 0, 2024-09-10T19:24:34.454 INFO:teuthology.orchestra.run.smithi183.stdout: "erasure_code_profile": "", 2024-09-10T19:24:34.454 INFO:teuthology.orchestra.run.smithi183.stdout: "hit_set_params": { 2024-09-10T19:24:34.454 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "none" 2024-09-10T19:24:34.454 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:34.454 INFO:teuthology.orchestra.run.smithi183.stdout: "hit_set_period": 0, 2024-09-10T19:24:34.454 INFO:teuthology.orchestra.run.smithi183.stdout: "hit_set_count": 0, 2024-09-10T19:24:34.454 INFO:teuthology.orchestra.run.smithi183.stdout: "use_gmt_hitset": true, 2024-09-10T19:24:34.454 INFO:teuthology.orchestra.run.smithi183.stdout: "min_read_recency_for_promote": 0, 2024-09-10T19:24:34.454 INFO:teuthology.orchestra.run.smithi183.stdout: "min_write_recency_for_promote": 0, 2024-09-10T19:24:34.454 INFO:teuthology.orchestra.run.smithi183.stdout: "hit_set_grade_decay_rate": 0, 2024-09-10T19:24:34.454 INFO:teuthology.orchestra.run.smithi183.stdout: "hit_set_search_last_n": 0, 2024-09-10T19:24:34.454 INFO:teuthology.orchestra.run.smithi183.stdout: "grade_table": [], 2024-09-10T19:24:34.454 INFO:teuthology.orchestra.run.smithi183.stdout: "stripe_width": 0, 2024-09-10T19:24:34.455 INFO:teuthology.orchestra.run.smithi183.stdout: "expected_num_objects": 0, 2024-09-10T19:24:34.455 INFO:teuthology.orchestra.run.smithi183.stdout: "fast_read": false, 2024-09-10T19:24:34.455 INFO:teuthology.orchestra.run.smithi183.stdout: "options": { 2024-09-10T19:24:34.455 INFO:teuthology.orchestra.run.smithi183.stdout: "pg_num_max": 32, 2024-09-10T19:24:34.455 INFO:teuthology.orchestra.run.smithi183.stdout: "pg_num_min": 1 2024-09-10T19:24:34.455 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:34.455 INFO:teuthology.orchestra.run.smithi183.stdout: "application_metadata": { 2024-09-10T19:24:34.455 INFO:teuthology.orchestra.run.smithi183.stdout: "mgr": {} 2024-09-10T19:24:34.455 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:24:34.455 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:34.455 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:34.455 INFO:teuthology.orchestra.run.smithi183.stdout: "pool": 2, 2024-09-10T19:24:34.455 INFO:teuthology.orchestra.run.smithi183.stdout: "pool_name": ".rgw.root", 2024-09-10T19:24:34.456 INFO:teuthology.orchestra.run.smithi183.stdout: "create_time": "2024-09-10T19:10:05.993763+0000", 2024-09-10T19:24:34.456 INFO:teuthology.orchestra.run.smithi183.stdout: "flags": 1, 2024-09-10T19:24:34.456 INFO:teuthology.orchestra.run.smithi183.stdout: "flags_names": "hashpspool", 2024-09-10T19:24:34.456 INFO:teuthology.orchestra.run.smithi183.stdout: "type": 1, 2024-09-10T19:24:34.456 INFO:teuthology.orchestra.run.smithi183.stdout: "size": 3, 2024-09-10T19:24:34.456 INFO:teuthology.orchestra.run.smithi183.stdout: "min_size": 2, 2024-09-10T19:24:34.456 INFO:teuthology.orchestra.run.smithi183.stdout: "crush_rule": 0, 2024-09-10T19:24:34.456 INFO:teuthology.orchestra.run.smithi183.stdout: "peering_crush_bucket_count": 0, 2024-09-10T19:24:34.456 INFO:teuthology.orchestra.run.smithi183.stdout: "peering_crush_bucket_target": 0, 2024-09-10T19:24:34.456 INFO:teuthology.orchestra.run.smithi183.stdout: "peering_crush_bucket_barrier": 0, 2024-09-10T19:24:34.456 INFO:teuthology.orchestra.run.smithi183.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-10T19:24:34.456 INFO:teuthology.orchestra.run.smithi183.stdout: "object_hash": 2, 2024-09-10T19:24:34.456 INFO:teuthology.orchestra.run.smithi183.stdout: "pg_autoscale_mode": "off", 2024-09-10T19:24:34.456 INFO:teuthology.orchestra.run.smithi183.stdout: "pg_num": 32, 2024-09-10T19:24:34.457 INFO:teuthology.orchestra.run.smithi183.stdout: "pg_placement_num": 32, 2024-09-10T19:24:34.457 INFO:teuthology.orchestra.run.smithi183.stdout: "pg_placement_num_target": 32, 2024-09-10T19:24:34.457 INFO:teuthology.orchestra.run.smithi183.stdout: "pg_num_target": 32, 2024-09-10T19:24:34.457 INFO:teuthology.orchestra.run.smithi183.stdout: "pg_num_pending": 32, 2024-09-10T19:24:34.457 INFO:teuthology.orchestra.run.smithi183.stdout: "last_pg_merge_meta": { 2024-09-10T19:24:34.457 INFO:teuthology.orchestra.run.smithi183.stdout: "source_pgid": "0.0", 2024-09-10T19:24:34.457 INFO:teuthology.orchestra.run.smithi183.stdout: "ready_epoch": 0, 2024-09-10T19:24:34.457 INFO:teuthology.orchestra.run.smithi183.stdout: "last_epoch_started": 0, 2024-09-10T19:24:34.457 INFO:teuthology.orchestra.run.smithi183.stdout: "last_epoch_clean": 0, 2024-09-10T19:24:34.457 INFO:teuthology.orchestra.run.smithi183.stdout: "source_version": "0'0", 2024-09-10T19:24:34.457 INFO:teuthology.orchestra.run.smithi183.stdout: "target_version": "0'0" 2024-09-10T19:24:34.457 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:34.457 INFO:teuthology.orchestra.run.smithi183.stdout: "last_change": "60", 2024-09-10T19:24:34.457 INFO:teuthology.orchestra.run.smithi183.stdout: "last_force_op_resend": "0", 2024-09-10T19:24:34.457 INFO:teuthology.orchestra.run.smithi183.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-10T19:24:34.458 INFO:teuthology.orchestra.run.smithi183.stdout: "last_force_op_resend_preluminous": "0", 2024-09-10T19:24:34.458 INFO:teuthology.orchestra.run.smithi183.stdout: "auid": 0, 2024-09-10T19:24:34.458 INFO:teuthology.orchestra.run.smithi183.stdout: "snap_mode": "selfmanaged", 2024-09-10T19:24:34.458 INFO:teuthology.orchestra.run.smithi183.stdout: "snap_seq": 0, 2024-09-10T19:24:34.458 INFO:teuthology.orchestra.run.smithi183.stdout: "snap_epoch": 0, 2024-09-10T19:24:34.458 INFO:teuthology.orchestra.run.smithi183.stdout: "pool_snaps": [], 2024-09-10T19:24:34.458 INFO:teuthology.orchestra.run.smithi183.stdout: "removed_snaps": "[]", 2024-09-10T19:24:34.458 INFO:teuthology.orchestra.run.smithi183.stdout: "quota_max_bytes": 0, 2024-09-10T19:24:34.458 INFO:teuthology.orchestra.run.smithi183.stdout: "quota_max_objects": 0, 2024-09-10T19:24:34.458 INFO:teuthology.orchestra.run.smithi183.stdout: "tiers": [], 2024-09-10T19:24:34.458 INFO:teuthology.orchestra.run.smithi183.stdout: "tier_of": -1, 2024-09-10T19:24:34.458 INFO:teuthology.orchestra.run.smithi183.stdout: "read_tier": -1, 2024-09-10T19:24:34.458 INFO:teuthology.orchestra.run.smithi183.stdout: "write_tier": -1, 2024-09-10T19:24:34.458 INFO:teuthology.orchestra.run.smithi183.stdout: "cache_mode": "none", 2024-09-10T19:24:34.459 INFO:teuthology.orchestra.run.smithi183.stdout: "target_max_bytes": 0, 2024-09-10T19:24:34.459 INFO:teuthology.orchestra.run.smithi183.stdout: "target_max_objects": 0, 2024-09-10T19:24:34.459 INFO:teuthology.orchestra.run.smithi183.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-10T19:24:34.459 INFO:teuthology.orchestra.run.smithi183.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-10T19:24:34.459 INFO:teuthology.orchestra.run.smithi183.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-10T19:24:34.459 INFO:teuthology.orchestra.run.smithi183.stdout: "cache_min_flush_age": 0, 2024-09-10T19:24:34.459 INFO:teuthology.orchestra.run.smithi183.stdout: "cache_min_evict_age": 0, 2024-09-10T19:24:34.459 INFO:teuthology.orchestra.run.smithi183.stdout: "erasure_code_profile": "", 2024-09-10T19:24:34.459 INFO:teuthology.orchestra.run.smithi183.stdout: "hit_set_params": { 2024-09-10T19:24:34.459 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "none" 2024-09-10T19:24:34.459 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:34.459 INFO:teuthology.orchestra.run.smithi183.stdout: "hit_set_period": 0, 2024-09-10T19:24:34.459 INFO:teuthology.orchestra.run.smithi183.stdout: "hit_set_count": 0, 2024-09-10T19:24:34.459 INFO:teuthology.orchestra.run.smithi183.stdout: "use_gmt_hitset": true, 2024-09-10T19:24:34.460 INFO:teuthology.orchestra.run.smithi183.stdout: "min_read_recency_for_promote": 0, 2024-09-10T19:24:34.460 INFO:teuthology.orchestra.run.smithi183.stdout: "min_write_recency_for_promote": 0, 2024-09-10T19:24:34.460 INFO:teuthology.orchestra.run.smithi183.stdout: "hit_set_grade_decay_rate": 0, 2024-09-10T19:24:34.460 INFO:teuthology.orchestra.run.smithi183.stdout: "hit_set_search_last_n": 0, 2024-09-10T19:24:34.460 INFO:teuthology.orchestra.run.smithi183.stdout: "grade_table": [], 2024-09-10T19:24:34.460 INFO:teuthology.orchestra.run.smithi183.stdout: "stripe_width": 0, 2024-09-10T19:24:34.460 INFO:teuthology.orchestra.run.smithi183.stdout: "expected_num_objects": 0, 2024-09-10T19:24:34.460 INFO:teuthology.orchestra.run.smithi183.stdout: "fast_read": false, 2024-09-10T19:24:34.460 INFO:teuthology.orchestra.run.smithi183.stdout: "options": {}, 2024-09-10T19:24:34.460 INFO:teuthology.orchestra.run.smithi183.stdout: "application_metadata": { 2024-09-10T19:24:34.460 INFO:teuthology.orchestra.run.smithi183.stdout: "rgw": {} 2024-09-10T19:24:34.460 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:24:34.461 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:34.461 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:34.461 INFO:teuthology.orchestra.run.smithi183.stdout: "pool": 3, 2024-09-10T19:24:34.461 INFO:teuthology.orchestra.run.smithi183.stdout: "pool_name": "datapool", 2024-09-10T19:24:34.461 INFO:teuthology.orchestra.run.smithi183.stdout: "create_time": "2024-09-10T19:10:06.224267+0000", 2024-09-10T19:24:34.461 INFO:teuthology.orchestra.run.smithi183.stdout: "flags": 8193, 2024-09-10T19:24:34.461 INFO:teuthology.orchestra.run.smithi183.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-09-10T19:24:34.461 INFO:teuthology.orchestra.run.smithi183.stdout: "type": 1, 2024-09-10T19:24:34.461 INFO:teuthology.orchestra.run.smithi183.stdout: "size": 3, 2024-09-10T19:24:34.461 INFO:teuthology.orchestra.run.smithi183.stdout: "min_size": 2, 2024-09-10T19:24:34.461 INFO:teuthology.orchestra.run.smithi183.stdout: "crush_rule": 0, 2024-09-10T19:24:34.461 INFO:teuthology.orchestra.run.smithi183.stdout: "peering_crush_bucket_count": 0, 2024-09-10T19:24:34.461 INFO:teuthology.orchestra.run.smithi183.stdout: "peering_crush_bucket_target": 0, 2024-09-10T19:24:34.462 INFO:teuthology.orchestra.run.smithi183.stdout: "peering_crush_bucket_barrier": 0, 2024-09-10T19:24:34.462 INFO:teuthology.orchestra.run.smithi183.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-10T19:24:34.462 INFO:teuthology.orchestra.run.smithi183.stdout: "object_hash": 2, 2024-09-10T19:24:34.462 INFO:teuthology.orchestra.run.smithi183.stdout: "pg_autoscale_mode": "off", 2024-09-10T19:24:34.462 INFO:teuthology.orchestra.run.smithi183.stdout: "pg_num": 3, 2024-09-10T19:24:34.462 INFO:teuthology.orchestra.run.smithi183.stdout: "pg_placement_num": 3, 2024-09-10T19:24:34.462 INFO:teuthology.orchestra.run.smithi183.stdout: "pg_placement_num_target": 3, 2024-09-10T19:24:34.462 INFO:teuthology.orchestra.run.smithi183.stdout: "pg_num_target": 3, 2024-09-10T19:24:34.462 INFO:teuthology.orchestra.run.smithi183.stdout: "pg_num_pending": 3, 2024-09-10T19:24:34.462 INFO:teuthology.orchestra.run.smithi183.stdout: "last_pg_merge_meta": { 2024-09-10T19:24:34.463 INFO:teuthology.orchestra.run.smithi183.stdout: "source_pgid": "0.0", 2024-09-10T19:24:34.463 INFO:teuthology.orchestra.run.smithi183.stdout: "ready_epoch": 0, 2024-09-10T19:24:34.463 INFO:teuthology.orchestra.run.smithi183.stdout: "last_epoch_started": 0, 2024-09-10T19:24:34.463 INFO:teuthology.orchestra.run.smithi183.stdout: "last_epoch_clean": 0, 2024-09-10T19:24:34.463 INFO:teuthology.orchestra.run.smithi183.stdout: "source_version": "0'0", 2024-09-10T19:24:34.463 INFO:teuthology.orchestra.run.smithi183.stdout: "target_version": "0'0" 2024-09-10T19:24:34.463 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:34.463 INFO:teuthology.orchestra.run.smithi183.stdout: "last_change": "63", 2024-09-10T19:24:34.463 INFO:teuthology.orchestra.run.smithi183.stdout: "last_force_op_resend": "0", 2024-09-10T19:24:34.463 INFO:teuthology.orchestra.run.smithi183.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-10T19:24:34.463 INFO:teuthology.orchestra.run.smithi183.stdout: "last_force_op_resend_preluminous": "0", 2024-09-10T19:24:34.464 INFO:teuthology.orchestra.run.smithi183.stdout: "auid": 0, 2024-09-10T19:24:34.464 INFO:teuthology.orchestra.run.smithi183.stdout: "snap_mode": "selfmanaged", 2024-09-10T19:24:34.464 INFO:teuthology.orchestra.run.smithi183.stdout: "snap_seq": 3, 2024-09-10T19:24:34.464 INFO:teuthology.orchestra.run.smithi183.stdout: "snap_epoch": 63, 2024-09-10T19:24:34.464 INFO:teuthology.orchestra.run.smithi183.stdout: "pool_snaps": [], 2024-09-10T19:24:34.464 INFO:teuthology.orchestra.run.smithi183.stdout: "removed_snaps": "[]", 2024-09-10T19:24:34.464 INFO:teuthology.orchestra.run.smithi183.stdout: "quota_max_bytes": 0, 2024-09-10T19:24:34.464 INFO:teuthology.orchestra.run.smithi183.stdout: "quota_max_objects": 0, 2024-09-10T19:24:34.464 INFO:teuthology.orchestra.run.smithi183.stdout: "tiers": [], 2024-09-10T19:24:34.465 INFO:teuthology.orchestra.run.smithi183.stdout: "tier_of": -1, 2024-09-10T19:24:34.465 INFO:teuthology.orchestra.run.smithi183.stdout: "read_tier": -1, 2024-09-10T19:24:34.465 INFO:teuthology.orchestra.run.smithi183.stdout: "write_tier": -1, 2024-09-10T19:24:34.465 INFO:teuthology.orchestra.run.smithi183.stdout: "cache_mode": "none", 2024-09-10T19:24:34.465 INFO:teuthology.orchestra.run.smithi183.stdout: "target_max_bytes": 0, 2024-09-10T19:24:34.465 INFO:teuthology.orchestra.run.smithi183.stdout: "target_max_objects": 0, 2024-09-10T19:24:34.465 INFO:teuthology.orchestra.run.smithi183.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-10T19:24:34.465 INFO:teuthology.orchestra.run.smithi183.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-10T19:24:34.465 INFO:teuthology.orchestra.run.smithi183.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-10T19:24:34.465 INFO:teuthology.orchestra.run.smithi183.stdout: "cache_min_flush_age": 0, 2024-09-10T19:24:34.465 INFO:teuthology.orchestra.run.smithi183.stdout: "cache_min_evict_age": 0, 2024-09-10T19:24:34.465 INFO:teuthology.orchestra.run.smithi183.stdout: "erasure_code_profile": "", 2024-09-10T19:24:34.466 INFO:teuthology.orchestra.run.smithi183.stdout: "hit_set_params": { 2024-09-10T19:24:34.466 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "none" 2024-09-10T19:24:34.466 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:34.466 INFO:teuthology.orchestra.run.smithi183.stdout: "hit_set_period": 0, 2024-09-10T19:24:34.466 INFO:teuthology.orchestra.run.smithi183.stdout: "hit_set_count": 0, 2024-09-10T19:24:34.466 INFO:teuthology.orchestra.run.smithi183.stdout: "use_gmt_hitset": true, 2024-09-10T19:24:34.466 INFO:teuthology.orchestra.run.smithi183.stdout: "min_read_recency_for_promote": 0, 2024-09-10T19:24:34.466 INFO:teuthology.orchestra.run.smithi183.stdout: "min_write_recency_for_promote": 0, 2024-09-10T19:24:34.466 INFO:teuthology.orchestra.run.smithi183.stdout: "hit_set_grade_decay_rate": 0, 2024-09-10T19:24:34.466 INFO:teuthology.orchestra.run.smithi183.stdout: "hit_set_search_last_n": 0, 2024-09-10T19:24:34.466 INFO:teuthology.orchestra.run.smithi183.stdout: "grade_table": [], 2024-09-10T19:24:34.466 INFO:teuthology.orchestra.run.smithi183.stdout: "stripe_width": 0, 2024-09-10T19:24:34.466 INFO:teuthology.orchestra.run.smithi183.stdout: "expected_num_objects": 0, 2024-09-10T19:24:34.466 INFO:teuthology.orchestra.run.smithi183.stdout: "fast_read": false, 2024-09-10T19:24:34.467 INFO:teuthology.orchestra.run.smithi183.stdout: "options": {}, 2024-09-10T19:24:34.467 INFO:teuthology.orchestra.run.smithi183.stdout: "application_metadata": { 2024-09-10T19:24:34.467 INFO:teuthology.orchestra.run.smithi183.stdout: "rbd": {} 2024-09-10T19:24:34.467 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:24:34.467 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:34.467 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:34.467 INFO:teuthology.orchestra.run.smithi183.stdout: "pool": 4, 2024-09-10T19:24:34.467 INFO:teuthology.orchestra.run.smithi183.stdout: "pool_name": "default.rgw.log", 2024-09-10T19:24:34.467 INFO:teuthology.orchestra.run.smithi183.stdout: "create_time": "2024-09-10T19:10:07.958284+0000", 2024-09-10T19:24:34.467 INFO:teuthology.orchestra.run.smithi183.stdout: "flags": 1, 2024-09-10T19:24:34.467 INFO:teuthology.orchestra.run.smithi183.stdout: "flags_names": "hashpspool", 2024-09-10T19:24:34.467 INFO:teuthology.orchestra.run.smithi183.stdout: "type": 1, 2024-09-10T19:24:34.467 INFO:teuthology.orchestra.run.smithi183.stdout: "size": 3, 2024-09-10T19:24:34.467 INFO:teuthology.orchestra.run.smithi183.stdout: "min_size": 2, 2024-09-10T19:24:34.467 INFO:teuthology.orchestra.run.smithi183.stdout: "crush_rule": 0, 2024-09-10T19:24:34.467 INFO:teuthology.orchestra.run.smithi183.stdout: "peering_crush_bucket_count": 0, 2024-09-10T19:24:34.468 INFO:teuthology.orchestra.run.smithi183.stdout: "peering_crush_bucket_target": 0, 2024-09-10T19:24:34.468 INFO:teuthology.orchestra.run.smithi183.stdout: "peering_crush_bucket_barrier": 0, 2024-09-10T19:24:34.468 INFO:teuthology.orchestra.run.smithi183.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-10T19:24:34.468 INFO:teuthology.orchestra.run.smithi183.stdout: "object_hash": 2, 2024-09-10T19:24:34.468 INFO:teuthology.orchestra.run.smithi183.stdout: "pg_autoscale_mode": "off", 2024-09-10T19:24:34.468 INFO:teuthology.orchestra.run.smithi183.stdout: "pg_num": 32, 2024-09-10T19:24:34.468 INFO:teuthology.orchestra.run.smithi183.stdout: "pg_placement_num": 32, 2024-09-10T19:24:34.468 INFO:teuthology.orchestra.run.smithi183.stdout: "pg_placement_num_target": 32, 2024-09-10T19:24:34.468 INFO:teuthology.orchestra.run.smithi183.stdout: "pg_num_target": 32, 2024-09-10T19:24:34.468 INFO:teuthology.orchestra.run.smithi183.stdout: "pg_num_pending": 32, 2024-09-10T19:24:34.468 INFO:teuthology.orchestra.run.smithi183.stdout: "last_pg_merge_meta": { 2024-09-10T19:24:34.468 INFO:teuthology.orchestra.run.smithi183.stdout: "source_pgid": "0.0", 2024-09-10T19:24:34.468 INFO:teuthology.orchestra.run.smithi183.stdout: "ready_epoch": 0, 2024-09-10T19:24:34.469 INFO:teuthology.orchestra.run.smithi183.stdout: "last_epoch_started": 0, 2024-09-10T19:24:34.469 INFO:teuthology.orchestra.run.smithi183.stdout: "last_epoch_clean": 0, 2024-09-10T19:24:34.469 INFO:teuthology.orchestra.run.smithi183.stdout: "source_version": "0'0", 2024-09-10T19:24:34.469 INFO:teuthology.orchestra.run.smithi183.stdout: "target_version": "0'0" 2024-09-10T19:24:34.469 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:34.469 INFO:teuthology.orchestra.run.smithi183.stdout: "last_change": "62", 2024-09-10T19:24:34.469 INFO:teuthology.orchestra.run.smithi183.stdout: "last_force_op_resend": "0", 2024-09-10T19:24:34.469 INFO:teuthology.orchestra.run.smithi183.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-10T19:24:34.469 INFO:teuthology.orchestra.run.smithi183.stdout: "last_force_op_resend_preluminous": "0", 2024-09-10T19:24:34.469 INFO:teuthology.orchestra.run.smithi183.stdout: "auid": 0, 2024-09-10T19:24:34.469 INFO:teuthology.orchestra.run.smithi183.stdout: "snap_mode": "selfmanaged", 2024-09-10T19:24:34.469 INFO:teuthology.orchestra.run.smithi183.stdout: "snap_seq": 0, 2024-09-10T19:24:34.469 INFO:teuthology.orchestra.run.smithi183.stdout: "snap_epoch": 0, 2024-09-10T19:24:34.469 INFO:teuthology.orchestra.run.smithi183.stdout: "pool_snaps": [], 2024-09-10T19:24:34.470 INFO:teuthology.orchestra.run.smithi183.stdout: "removed_snaps": "[]", 2024-09-10T19:24:34.470 INFO:teuthology.orchestra.run.smithi183.stdout: "quota_max_bytes": 0, 2024-09-10T19:24:34.470 INFO:teuthology.orchestra.run.smithi183.stdout: "quota_max_objects": 0, 2024-09-10T19:24:34.470 INFO:teuthology.orchestra.run.smithi183.stdout: "tiers": [], 2024-09-10T19:24:34.470 INFO:teuthology.orchestra.run.smithi183.stdout: "tier_of": -1, 2024-09-10T19:24:34.470 INFO:teuthology.orchestra.run.smithi183.stdout: "read_tier": -1, 2024-09-10T19:24:34.470 INFO:teuthology.orchestra.run.smithi183.stdout: "write_tier": -1, 2024-09-10T19:24:34.470 INFO:teuthology.orchestra.run.smithi183.stdout: "cache_mode": "none", 2024-09-10T19:24:34.470 INFO:teuthology.orchestra.run.smithi183.stdout: "target_max_bytes": 0, 2024-09-10T19:24:34.470 INFO:teuthology.orchestra.run.smithi183.stdout: "target_max_objects": 0, 2024-09-10T19:24:34.470 INFO:teuthology.orchestra.run.smithi183.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-10T19:24:34.470 INFO:teuthology.orchestra.run.smithi183.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-10T19:24:34.470 INFO:teuthology.orchestra.run.smithi183.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-10T19:24:34.470 INFO:teuthology.orchestra.run.smithi183.stdout: "cache_min_flush_age": 0, 2024-09-10T19:24:34.471 INFO:teuthology.orchestra.run.smithi183.stdout: "cache_min_evict_age": 0, 2024-09-10T19:24:34.471 INFO:teuthology.orchestra.run.smithi183.stdout: "erasure_code_profile": "", 2024-09-10T19:24:34.471 INFO:teuthology.orchestra.run.smithi183.stdout: "hit_set_params": { 2024-09-10T19:24:34.471 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "none" 2024-09-10T19:24:34.471 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:34.471 INFO:teuthology.orchestra.run.smithi183.stdout: "hit_set_period": 0, 2024-09-10T19:24:34.471 INFO:teuthology.orchestra.run.smithi183.stdout: "hit_set_count": 0, 2024-09-10T19:24:34.471 INFO:teuthology.orchestra.run.smithi183.stdout: "use_gmt_hitset": true, 2024-09-10T19:24:34.471 INFO:teuthology.orchestra.run.smithi183.stdout: "min_read_recency_for_promote": 0, 2024-09-10T19:24:34.471 INFO:teuthology.orchestra.run.smithi183.stdout: "min_write_recency_for_promote": 0, 2024-09-10T19:24:34.472 INFO:teuthology.orchestra.run.smithi183.stdout: "hit_set_grade_decay_rate": 0, 2024-09-10T19:24:34.472 INFO:teuthology.orchestra.run.smithi183.stdout: "hit_set_search_last_n": 0, 2024-09-10T19:24:34.472 INFO:teuthology.orchestra.run.smithi183.stdout: "grade_table": [], 2024-09-10T19:24:34.472 INFO:teuthology.orchestra.run.smithi183.stdout: "stripe_width": 0, 2024-09-10T19:24:34.472 INFO:teuthology.orchestra.run.smithi183.stdout: "expected_num_objects": 0, 2024-09-10T19:24:34.472 INFO:teuthology.orchestra.run.smithi183.stdout: "fast_read": false, 2024-09-10T19:24:34.472 INFO:teuthology.orchestra.run.smithi183.stdout: "options": {}, 2024-09-10T19:24:34.472 INFO:teuthology.orchestra.run.smithi183.stdout: "application_metadata": { 2024-09-10T19:24:34.472 INFO:teuthology.orchestra.run.smithi183.stdout: "rgw": {} 2024-09-10T19:24:34.472 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:24:34.472 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:34.472 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:34.472 INFO:teuthology.orchestra.run.smithi183.stdout: "pool": 5, 2024-09-10T19:24:34.473 INFO:teuthology.orchestra.run.smithi183.stdout: "pool_name": "default.rgw.control", 2024-09-10T19:24:34.473 INFO:teuthology.orchestra.run.smithi183.stdout: "create_time": "2024-09-10T19:10:09.933596+0000", 2024-09-10T19:24:34.473 INFO:teuthology.orchestra.run.smithi183.stdout: "flags": 1, 2024-09-10T19:24:34.473 INFO:teuthology.orchestra.run.smithi183.stdout: "flags_names": "hashpspool", 2024-09-10T19:24:34.473 INFO:teuthology.orchestra.run.smithi183.stdout: "type": 1, 2024-09-10T19:24:34.473 INFO:teuthology.orchestra.run.smithi183.stdout: "size": 3, 2024-09-10T19:24:34.473 INFO:teuthology.orchestra.run.smithi183.stdout: "min_size": 2, 2024-09-10T19:24:34.473 INFO:teuthology.orchestra.run.smithi183.stdout: "crush_rule": 0, 2024-09-10T19:24:34.473 INFO:teuthology.orchestra.run.smithi183.stdout: "peering_crush_bucket_count": 0, 2024-09-10T19:24:34.473 INFO:teuthology.orchestra.run.smithi183.stdout: "peering_crush_bucket_target": 0, 2024-09-10T19:24:34.474 INFO:teuthology.orchestra.run.smithi183.stdout: "peering_crush_bucket_barrier": 0, 2024-09-10T19:24:34.474 INFO:teuthology.orchestra.run.smithi183.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-10T19:24:34.474 INFO:teuthology.orchestra.run.smithi183.stdout: "object_hash": 2, 2024-09-10T19:24:34.474 INFO:teuthology.orchestra.run.smithi183.stdout: "pg_autoscale_mode": "off", 2024-09-10T19:24:34.474 INFO:teuthology.orchestra.run.smithi183.stdout: "pg_num": 32, 2024-09-10T19:24:34.474 INFO:teuthology.orchestra.run.smithi183.stdout: "pg_placement_num": 32, 2024-09-10T19:24:34.474 INFO:teuthology.orchestra.run.smithi183.stdout: "pg_placement_num_target": 32, 2024-09-10T19:24:34.474 INFO:teuthology.orchestra.run.smithi183.stdout: "pg_num_target": 32, 2024-09-10T19:24:34.474 INFO:teuthology.orchestra.run.smithi183.stdout: "pg_num_pending": 32, 2024-09-10T19:24:34.474 INFO:teuthology.orchestra.run.smithi183.stdout: "last_pg_merge_meta": { 2024-09-10T19:24:34.474 INFO:teuthology.orchestra.run.smithi183.stdout: "source_pgid": "0.0", 2024-09-10T19:24:34.474 INFO:teuthology.orchestra.run.smithi183.stdout: "ready_epoch": 0, 2024-09-10T19:24:34.474 INFO:teuthology.orchestra.run.smithi183.stdout: "last_epoch_started": 0, 2024-09-10T19:24:34.474 INFO:teuthology.orchestra.run.smithi183.stdout: "last_epoch_clean": 0, 2024-09-10T19:24:34.474 INFO:teuthology.orchestra.run.smithi183.stdout: "source_version": "0'0", 2024-09-10T19:24:34.475 INFO:teuthology.orchestra.run.smithi183.stdout: "target_version": "0'0" 2024-09-10T19:24:34.475 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:34.475 INFO:teuthology.orchestra.run.smithi183.stdout: "last_change": "64", 2024-09-10T19:24:34.475 INFO:teuthology.orchestra.run.smithi183.stdout: "last_force_op_resend": "0", 2024-09-10T19:24:34.475 INFO:teuthology.orchestra.run.smithi183.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-10T19:24:34.475 INFO:teuthology.orchestra.run.smithi183.stdout: "last_force_op_resend_preluminous": "0", 2024-09-10T19:24:34.475 INFO:teuthology.orchestra.run.smithi183.stdout: "auid": 0, 2024-09-10T19:24:34.475 INFO:teuthology.orchestra.run.smithi183.stdout: "snap_mode": "selfmanaged", 2024-09-10T19:24:34.475 INFO:teuthology.orchestra.run.smithi183.stdout: "snap_seq": 0, 2024-09-10T19:24:34.475 INFO:teuthology.orchestra.run.smithi183.stdout: "snap_epoch": 0, 2024-09-10T19:24:34.475 INFO:teuthology.orchestra.run.smithi183.stdout: "pool_snaps": [], 2024-09-10T19:24:34.475 INFO:teuthology.orchestra.run.smithi183.stdout: "removed_snaps": "[]", 2024-09-10T19:24:34.475 INFO:teuthology.orchestra.run.smithi183.stdout: "quota_max_bytes": 0, 2024-09-10T19:24:34.475 INFO:teuthology.orchestra.run.smithi183.stdout: "quota_max_objects": 0, 2024-09-10T19:24:34.476 INFO:teuthology.orchestra.run.smithi183.stdout: "tiers": [], 2024-09-10T19:24:34.476 INFO:teuthology.orchestra.run.smithi183.stdout: "tier_of": -1, 2024-09-10T19:24:34.476 INFO:teuthology.orchestra.run.smithi183.stdout: "read_tier": -1, 2024-09-10T19:24:34.476 INFO:teuthology.orchestra.run.smithi183.stdout: "write_tier": -1, 2024-09-10T19:24:34.476 INFO:teuthology.orchestra.run.smithi183.stdout: "cache_mode": "none", 2024-09-10T19:24:34.476 INFO:teuthology.orchestra.run.smithi183.stdout: "target_max_bytes": 0, 2024-09-10T19:24:34.476 INFO:teuthology.orchestra.run.smithi183.stdout: "target_max_objects": 0, 2024-09-10T19:24:34.476 INFO:teuthology.orchestra.run.smithi183.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-10T19:24:34.476 INFO:teuthology.orchestra.run.smithi183.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-10T19:24:34.476 INFO:teuthology.orchestra.run.smithi183.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-10T19:24:34.476 INFO:teuthology.orchestra.run.smithi183.stdout: "cache_min_flush_age": 0, 2024-09-10T19:24:34.476 INFO:teuthology.orchestra.run.smithi183.stdout: "cache_min_evict_age": 0, 2024-09-10T19:24:34.476 INFO:teuthology.orchestra.run.smithi183.stdout: "erasure_code_profile": "", 2024-09-10T19:24:34.476 INFO:teuthology.orchestra.run.smithi183.stdout: "hit_set_params": { 2024-09-10T19:24:34.476 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "none" 2024-09-10T19:24:34.477 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:34.477 INFO:teuthology.orchestra.run.smithi183.stdout: "hit_set_period": 0, 2024-09-10T19:24:34.477 INFO:teuthology.orchestra.run.smithi183.stdout: "hit_set_count": 0, 2024-09-10T19:24:34.477 INFO:teuthology.orchestra.run.smithi183.stdout: "use_gmt_hitset": true, 2024-09-10T19:24:34.477 INFO:teuthology.orchestra.run.smithi183.stdout: "min_read_recency_for_promote": 0, 2024-09-10T19:24:34.477 INFO:teuthology.orchestra.run.smithi183.stdout: "min_write_recency_for_promote": 0, 2024-09-10T19:24:34.477 INFO:teuthology.orchestra.run.smithi183.stdout: "hit_set_grade_decay_rate": 0, 2024-09-10T19:24:34.477 INFO:teuthology.orchestra.run.smithi183.stdout: "hit_set_search_last_n": 0, 2024-09-10T19:24:34.477 INFO:teuthology.orchestra.run.smithi183.stdout: "grade_table": [], 2024-09-10T19:24:34.477 INFO:teuthology.orchestra.run.smithi183.stdout: "stripe_width": 0, 2024-09-10T19:24:34.477 INFO:teuthology.orchestra.run.smithi183.stdout: "expected_num_objects": 0, 2024-09-10T19:24:34.477 INFO:teuthology.orchestra.run.smithi183.stdout: "fast_read": false, 2024-09-10T19:24:34.477 INFO:teuthology.orchestra.run.smithi183.stdout: "options": {}, 2024-09-10T19:24:34.478 INFO:teuthology.orchestra.run.smithi183.stdout: "application_metadata": { 2024-09-10T19:24:34.478 INFO:teuthology.orchestra.run.smithi183.stdout: "rgw": {} 2024-09-10T19:24:34.478 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:24:34.478 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:34.478 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:34.478 INFO:teuthology.orchestra.run.smithi183.stdout: "pool": 6, 2024-09-10T19:24:34.478 INFO:teuthology.orchestra.run.smithi183.stdout: "pool_name": "default.rgw.meta", 2024-09-10T19:24:34.478 INFO:teuthology.orchestra.run.smithi183.stdout: "create_time": "2024-09-10T19:10:11.993712+0000", 2024-09-10T19:24:34.478 INFO:teuthology.orchestra.run.smithi183.stdout: "flags": 1, 2024-09-10T19:24:34.478 INFO:teuthology.orchestra.run.smithi183.stdout: "flags_names": "hashpspool", 2024-09-10T19:24:34.478 INFO:teuthology.orchestra.run.smithi183.stdout: "type": 1, 2024-09-10T19:24:34.479 INFO:teuthology.orchestra.run.smithi183.stdout: "size": 3, 2024-09-10T19:24:34.479 INFO:teuthology.orchestra.run.smithi183.stdout: "min_size": 2, 2024-09-10T19:24:34.479 INFO:teuthology.orchestra.run.smithi183.stdout: "crush_rule": 0, 2024-09-10T19:24:34.479 INFO:teuthology.orchestra.run.smithi183.stdout: "peering_crush_bucket_count": 0, 2024-09-10T19:24:34.479 INFO:teuthology.orchestra.run.smithi183.stdout: "peering_crush_bucket_target": 0, 2024-09-10T19:24:34.479 INFO:teuthology.orchestra.run.smithi183.stdout: "peering_crush_bucket_barrier": 0, 2024-09-10T19:24:34.479 INFO:teuthology.orchestra.run.smithi183.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-10T19:24:34.479 INFO:teuthology.orchestra.run.smithi183.stdout: "object_hash": 2, 2024-09-10T19:24:34.479 INFO:teuthology.orchestra.run.smithi183.stdout: "pg_autoscale_mode": "off", 2024-09-10T19:24:34.479 INFO:teuthology.orchestra.run.smithi183.stdout: "pg_num": 32, 2024-09-10T19:24:34.479 INFO:teuthology.orchestra.run.smithi183.stdout: "pg_placement_num": 32, 2024-09-10T19:24:34.479 INFO:teuthology.orchestra.run.smithi183.stdout: "pg_placement_num_target": 32, 2024-09-10T19:24:34.479 INFO:teuthology.orchestra.run.smithi183.stdout: "pg_num_target": 32, 2024-09-10T19:24:34.479 INFO:teuthology.orchestra.run.smithi183.stdout: "pg_num_pending": 32, 2024-09-10T19:24:34.479 INFO:teuthology.orchestra.run.smithi183.stdout: "last_pg_merge_meta": { 2024-09-10T19:24:34.480 INFO:teuthology.orchestra.run.smithi183.stdout: "source_pgid": "0.0", 2024-09-10T19:24:34.480 INFO:teuthology.orchestra.run.smithi183.stdout: "ready_epoch": 0, 2024-09-10T19:24:34.480 INFO:teuthology.orchestra.run.smithi183.stdout: "last_epoch_started": 0, 2024-09-10T19:24:34.480 INFO:teuthology.orchestra.run.smithi183.stdout: "last_epoch_clean": 0, 2024-09-10T19:24:34.480 INFO:teuthology.orchestra.run.smithi183.stdout: "source_version": "0'0", 2024-09-10T19:24:34.480 INFO:teuthology.orchestra.run.smithi183.stdout: "target_version": "0'0" 2024-09-10T19:24:34.480 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:34.480 INFO:teuthology.orchestra.run.smithi183.stdout: "last_change": "66", 2024-09-10T19:24:34.480 INFO:teuthology.orchestra.run.smithi183.stdout: "last_force_op_resend": "0", 2024-09-10T19:24:34.480 INFO:teuthology.orchestra.run.smithi183.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-10T19:24:34.480 INFO:teuthology.orchestra.run.smithi183.stdout: "last_force_op_resend_preluminous": "0", 2024-09-10T19:24:34.480 INFO:teuthology.orchestra.run.smithi183.stdout: "auid": 0, 2024-09-10T19:24:34.480 INFO:teuthology.orchestra.run.smithi183.stdout: "snap_mode": "selfmanaged", 2024-09-10T19:24:34.480 INFO:teuthology.orchestra.run.smithi183.stdout: "snap_seq": 0, 2024-09-10T19:24:34.481 INFO:teuthology.orchestra.run.smithi183.stdout: "snap_epoch": 0, 2024-09-10T19:24:34.481 INFO:teuthology.orchestra.run.smithi183.stdout: "pool_snaps": [], 2024-09-10T19:24:34.481 INFO:teuthology.orchestra.run.smithi183.stdout: "removed_snaps": "[]", 2024-09-10T19:24:34.481 INFO:teuthology.orchestra.run.smithi183.stdout: "quota_max_bytes": 0, 2024-09-10T19:24:34.481 INFO:teuthology.orchestra.run.smithi183.stdout: "quota_max_objects": 0, 2024-09-10T19:24:34.481 INFO:teuthology.orchestra.run.smithi183.stdout: "tiers": [], 2024-09-10T19:24:34.481 INFO:teuthology.orchestra.run.smithi183.stdout: "tier_of": -1, 2024-09-10T19:24:34.481 INFO:teuthology.orchestra.run.smithi183.stdout: "read_tier": -1, 2024-09-10T19:24:34.481 INFO:teuthology.orchestra.run.smithi183.stdout: "write_tier": -1, 2024-09-10T19:24:34.481 INFO:teuthology.orchestra.run.smithi183.stdout: "cache_mode": "none", 2024-09-10T19:24:34.481 INFO:teuthology.orchestra.run.smithi183.stdout: "target_max_bytes": 0, 2024-09-10T19:24:34.481 INFO:teuthology.orchestra.run.smithi183.stdout: "target_max_objects": 0, 2024-09-10T19:24:34.481 INFO:teuthology.orchestra.run.smithi183.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-10T19:24:34.481 INFO:teuthology.orchestra.run.smithi183.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-10T19:24:34.481 INFO:teuthology.orchestra.run.smithi183.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-10T19:24:34.481 INFO:teuthology.orchestra.run.smithi183.stdout: "cache_min_flush_age": 0, 2024-09-10T19:24:34.482 INFO:teuthology.orchestra.run.smithi183.stdout: "cache_min_evict_age": 0, 2024-09-10T19:24:34.482 INFO:teuthology.orchestra.run.smithi183.stdout: "erasure_code_profile": "", 2024-09-10T19:24:34.482 INFO:teuthology.orchestra.run.smithi183.stdout: "hit_set_params": { 2024-09-10T19:24:34.482 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "none" 2024-09-10T19:24:34.482 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:34.482 INFO:teuthology.orchestra.run.smithi183.stdout: "hit_set_period": 0, 2024-09-10T19:24:34.482 INFO:teuthology.orchestra.run.smithi183.stdout: "hit_set_count": 0, 2024-09-10T19:24:34.482 INFO:teuthology.orchestra.run.smithi183.stdout: "use_gmt_hitset": true, 2024-09-10T19:24:34.482 INFO:teuthology.orchestra.run.smithi183.stdout: "min_read_recency_for_promote": 0, 2024-09-10T19:24:34.482 INFO:teuthology.orchestra.run.smithi183.stdout: "min_write_recency_for_promote": 0, 2024-09-10T19:24:34.482 INFO:teuthology.orchestra.run.smithi183.stdout: "hit_set_grade_decay_rate": 0, 2024-09-10T19:24:34.482 INFO:teuthology.orchestra.run.smithi183.stdout: "hit_set_search_last_n": 0, 2024-09-10T19:24:34.482 INFO:teuthology.orchestra.run.smithi183.stdout: "grade_table": [], 2024-09-10T19:24:34.482 INFO:teuthology.orchestra.run.smithi183.stdout: "stripe_width": 0, 2024-09-10T19:24:34.482 INFO:teuthology.orchestra.run.smithi183.stdout: "expected_num_objects": 0, 2024-09-10T19:24:34.482 INFO:teuthology.orchestra.run.smithi183.stdout: "fast_read": false, 2024-09-10T19:24:34.483 INFO:teuthology.orchestra.run.smithi183.stdout: "options": { 2024-09-10T19:24:34.483 INFO:teuthology.orchestra.run.smithi183.stdout: "pg_autoscale_bias": 4 2024-09-10T19:24:34.483 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:34.484 INFO:teuthology.orchestra.run.smithi183.stdout: "application_metadata": { 2024-09-10T19:24:34.484 INFO:teuthology.orchestra.run.smithi183.stdout: "rgw": {} 2024-09-10T19:24:34.484 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:24:34.484 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:34.484 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:34.484 INFO:teuthology.orchestra.run.smithi183.stdout: "pool": 10, 2024-09-10T19:24:34.484 INFO:teuthology.orchestra.run.smithi183.stdout: "pool_name": "unique_pool_3", 2024-09-10T19:24:34.484 INFO:teuthology.orchestra.run.smithi183.stdout: "create_time": "2024-09-10T19:24:07.399856+0000", 2024-09-10T19:24:34.484 INFO:teuthology.orchestra.run.smithi183.stdout: "flags": 1, 2024-09-10T19:24:34.484 INFO:teuthology.orchestra.run.smithi183.stdout: "flags_names": "hashpspool", 2024-09-10T19:24:34.484 INFO:teuthology.orchestra.run.smithi183.stdout: "type": 1, 2024-09-10T19:24:34.484 INFO:teuthology.orchestra.run.smithi183.stdout: "size": 3, 2024-09-10T19:24:34.484 INFO:teuthology.orchestra.run.smithi183.stdout: "min_size": 2, 2024-09-10T19:24:34.484 INFO:teuthology.orchestra.run.smithi183.stdout: "crush_rule": 0, 2024-09-10T19:24:34.485 INFO:teuthology.orchestra.run.smithi183.stdout: "peering_crush_bucket_count": 0, 2024-09-10T19:24:34.485 INFO:teuthology.orchestra.run.smithi183.stdout: "peering_crush_bucket_target": 0, 2024-09-10T19:24:34.485 INFO:teuthology.orchestra.run.smithi183.stdout: "peering_crush_bucket_barrier": 0, 2024-09-10T19:24:34.485 INFO:teuthology.orchestra.run.smithi183.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-10T19:24:34.485 INFO:teuthology.orchestra.run.smithi183.stdout: "object_hash": 2, 2024-09-10T19:24:34.485 INFO:teuthology.orchestra.run.smithi183.stdout: "pg_autoscale_mode": "off", 2024-09-10T19:24:34.485 INFO:teuthology.orchestra.run.smithi183.stdout: "pg_num": 16, 2024-09-10T19:24:34.485 INFO:teuthology.orchestra.run.smithi183.stdout: "pg_placement_num": 16, 2024-09-10T19:24:34.485 INFO:teuthology.orchestra.run.smithi183.stdout: "pg_placement_num_target": 16, 2024-09-10T19:24:34.485 INFO:teuthology.orchestra.run.smithi183.stdout: "pg_num_target": 16, 2024-09-10T19:24:34.485 INFO:teuthology.orchestra.run.smithi183.stdout: "pg_num_pending": 16, 2024-09-10T19:24:34.485 INFO:teuthology.orchestra.run.smithi183.stdout: "last_pg_merge_meta": { 2024-09-10T19:24:34.485 INFO:teuthology.orchestra.run.smithi183.stdout: "source_pgid": "0.0", 2024-09-10T19:24:34.485 INFO:teuthology.orchestra.run.smithi183.stdout: "ready_epoch": 0, 2024-09-10T19:24:34.485 INFO:teuthology.orchestra.run.smithi183.stdout: "last_epoch_started": 0, 2024-09-10T19:24:34.486 INFO:teuthology.orchestra.run.smithi183.stdout: "last_epoch_clean": 0, 2024-09-10T19:24:34.486 INFO:teuthology.orchestra.run.smithi183.stdout: "source_version": "0'0", 2024-09-10T19:24:34.486 INFO:teuthology.orchestra.run.smithi183.stdout: "target_version": "0'0" 2024-09-10T19:24:34.486 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:34.486 INFO:teuthology.orchestra.run.smithi183.stdout: "last_change": "299", 2024-09-10T19:24:34.486 INFO:teuthology.orchestra.run.smithi183.stdout: "last_force_op_resend": "0", 2024-09-10T19:24:34.486 INFO:teuthology.orchestra.run.smithi183.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-10T19:24:34.486 INFO:teuthology.orchestra.run.smithi183.stdout: "last_force_op_resend_preluminous": "0", 2024-09-10T19:24:34.486 INFO:teuthology.orchestra.run.smithi183.stdout: "auid": 0, 2024-09-10T19:24:34.486 INFO:teuthology.orchestra.run.smithi183.stdout: "snap_mode": "selfmanaged", 2024-09-10T19:24:34.486 INFO:teuthology.orchestra.run.smithi183.stdout: "snap_seq": 0, 2024-09-10T19:24:34.486 INFO:teuthology.orchestra.run.smithi183.stdout: "snap_epoch": 0, 2024-09-10T19:24:34.486 INFO:teuthology.orchestra.run.smithi183.stdout: "pool_snaps": [], 2024-09-10T19:24:34.486 INFO:teuthology.orchestra.run.smithi183.stdout: "removed_snaps": "[]", 2024-09-10T19:24:34.486 INFO:teuthology.orchestra.run.smithi183.stdout: "quota_max_bytes": 0, 2024-09-10T19:24:34.487 INFO:teuthology.orchestra.run.smithi183.stdout: "quota_max_objects": 0, 2024-09-10T19:24:34.487 INFO:teuthology.orchestra.run.smithi183.stdout: "tiers": [], 2024-09-10T19:24:34.487 INFO:teuthology.orchestra.run.smithi183.stdout: "tier_of": -1, 2024-09-10T19:24:34.487 INFO:teuthology.orchestra.run.smithi183.stdout: "read_tier": -1, 2024-09-10T19:24:34.487 INFO:teuthology.orchestra.run.smithi183.stdout: "write_tier": -1, 2024-09-10T19:24:34.487 INFO:teuthology.orchestra.run.smithi183.stdout: "cache_mode": "none", 2024-09-10T19:24:34.487 INFO:teuthology.orchestra.run.smithi183.stdout: "target_max_bytes": 0, 2024-09-10T19:24:34.487 INFO:teuthology.orchestra.run.smithi183.stdout: "target_max_objects": 0, 2024-09-10T19:24:34.487 INFO:teuthology.orchestra.run.smithi183.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-10T19:24:34.487 INFO:teuthology.orchestra.run.smithi183.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-10T19:24:34.487 INFO:teuthology.orchestra.run.smithi183.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-10T19:24:34.487 INFO:teuthology.orchestra.run.smithi183.stdout: "cache_min_flush_age": 0, 2024-09-10T19:24:34.487 INFO:teuthology.orchestra.run.smithi183.stdout: "cache_min_evict_age": 0, 2024-09-10T19:24:34.487 INFO:teuthology.orchestra.run.smithi183.stdout: "erasure_code_profile": "", 2024-09-10T19:24:34.487 INFO:teuthology.orchestra.run.smithi183.stdout: "hit_set_params": { 2024-09-10T19:24:34.488 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "none" 2024-09-10T19:24:34.488 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:34.488 INFO:teuthology.orchestra.run.smithi183.stdout: "hit_set_period": 0, 2024-09-10T19:24:34.488 INFO:teuthology.orchestra.run.smithi183.stdout: "hit_set_count": 0, 2024-09-10T19:24:34.488 INFO:teuthology.orchestra.run.smithi183.stdout: "use_gmt_hitset": true, 2024-09-10T19:24:34.488 INFO:teuthology.orchestra.run.smithi183.stdout: "min_read_recency_for_promote": 0, 2024-09-10T19:24:34.488 INFO:teuthology.orchestra.run.smithi183.stdout: "min_write_recency_for_promote": 0, 2024-09-10T19:24:34.488 INFO:teuthology.orchestra.run.smithi183.stdout: "hit_set_grade_decay_rate": 0, 2024-09-10T19:24:34.488 INFO:teuthology.orchestra.run.smithi183.stdout: "hit_set_search_last_n": 0, 2024-09-10T19:24:34.488 INFO:teuthology.orchestra.run.smithi183.stdout: "grade_table": [], 2024-09-10T19:24:34.488 INFO:teuthology.orchestra.run.smithi183.stdout: "stripe_width": 0, 2024-09-10T19:24:34.488 INFO:teuthology.orchestra.run.smithi183.stdout: "expected_num_objects": 0, 2024-09-10T19:24:34.488 INFO:teuthology.orchestra.run.smithi183.stdout: "fast_read": false, 2024-09-10T19:24:34.488 INFO:teuthology.orchestra.run.smithi183.stdout: "options": {}, 2024-09-10T19:24:34.488 INFO:teuthology.orchestra.run.smithi183.stdout: "application_metadata": { 2024-09-10T19:24:34.489 INFO:teuthology.orchestra.run.smithi183.stdout: "rados": {} 2024-09-10T19:24:34.489 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:24:34.489 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:24:34.489 INFO:teuthology.orchestra.run.smithi183.stdout: ], 2024-09-10T19:24:34.489 INFO:teuthology.orchestra.run.smithi183.stdout: "osds": [ 2024-09-10T19:24:34.489 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:34.489 INFO:teuthology.orchestra.run.smithi183.stdout: "osd": 0, 2024-09-10T19:24:34.489 INFO:teuthology.orchestra.run.smithi183.stdout: "uuid": "9d323010-394a-49aa-ba9b-244d87faf36e", 2024-09-10T19:24:34.489 INFO:teuthology.orchestra.run.smithi183.stdout: "up": 1, 2024-09-10T19:24:34.489 INFO:teuthology.orchestra.run.smithi183.stdout: "in": 1, 2024-09-10T19:24:34.489 INFO:teuthology.orchestra.run.smithi183.stdout: "weight": 1, 2024-09-10T19:24:34.489 INFO:teuthology.orchestra.run.smithi183.stdout: "primary_affinity": 1, 2024-09-10T19:24:34.489 INFO:teuthology.orchestra.run.smithi183.stdout: "last_clean_begin": 8, 2024-09-10T19:24:34.489 INFO:teuthology.orchestra.run.smithi183.stdout: "last_clean_end": 262, 2024-09-10T19:24:34.490 INFO:teuthology.orchestra.run.smithi183.stdout: "up_from": 301, 2024-09-10T19:24:34.490 INFO:teuthology.orchestra.run.smithi183.stdout: "up_thru": 308, 2024-09-10T19:24:34.490 INFO:teuthology.orchestra.run.smithi183.stdout: "down_at": 263, 2024-09-10T19:24:34.490 INFO:teuthology.orchestra.run.smithi183.stdout: "lost_at": 0, 2024-09-10T19:24:34.490 INFO:teuthology.orchestra.run.smithi183.stdout: "public_addrs": { 2024-09-10T19:24:34.490 INFO:teuthology.orchestra.run.smithi183.stdout: "addrvec": [ 2024-09-10T19:24:34.490 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:34.490 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v2", 2024-09-10T19:24:34.490 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.183:6810", 2024-09-10T19:24:34.490 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 3346539414 2024-09-10T19:24:34.490 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:34.490 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:34.490 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v1", 2024-09-10T19:24:34.490 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.183:6811", 2024-09-10T19:24:34.491 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 3346539414 2024-09-10T19:24:34.491 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:24:34.491 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:24:34.491 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:34.491 INFO:teuthology.orchestra.run.smithi183.stdout: "cluster_addrs": { 2024-09-10T19:24:34.491 INFO:teuthology.orchestra.run.smithi183.stdout: "addrvec": [ 2024-09-10T19:24:34.491 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:34.491 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v2", 2024-09-10T19:24:34.491 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.183:6812", 2024-09-10T19:24:34.491 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 3346539414 2024-09-10T19:24:34.491 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:34.491 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:34.491 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v1", 2024-09-10T19:24:34.491 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.183:6813", 2024-09-10T19:24:34.491 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 3346539414 2024-09-10T19:24:34.492 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:24:34.492 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:24:34.492 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:34.492 INFO:teuthology.orchestra.run.smithi183.stdout: "heartbeat_back_addrs": { 2024-09-10T19:24:34.492 INFO:teuthology.orchestra.run.smithi183.stdout: "addrvec": [ 2024-09-10T19:24:34.492 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:34.492 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v2", 2024-09-10T19:24:34.492 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.183:6816", 2024-09-10T19:24:34.492 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 3346539414 2024-09-10T19:24:34.492 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:34.492 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:34.492 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v1", 2024-09-10T19:24:34.492 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.183:6817", 2024-09-10T19:24:34.492 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 3346539414 2024-09-10T19:24:34.492 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:24:34.493 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:24:34.493 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:34.493 INFO:teuthology.orchestra.run.smithi183.stdout: "heartbeat_front_addrs": { 2024-09-10T19:24:34.493 INFO:teuthology.orchestra.run.smithi183.stdout: "addrvec": [ 2024-09-10T19:24:34.493 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:34.493 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v2", 2024-09-10T19:24:34.493 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.183:6814", 2024-09-10T19:24:34.493 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 3346539414 2024-09-10T19:24:34.493 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:34.493 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:34.493 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v1", 2024-09-10T19:24:34.493 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.183:6815", 2024-09-10T19:24:34.493 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 3346539414 2024-09-10T19:24:34.493 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:24:34.494 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:24:34.494 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:34.494 INFO:teuthology.orchestra.run.smithi183.stdout: "public_addr": "172.21.15.183:6811/3346539414", 2024-09-10T19:24:34.494 INFO:teuthology.orchestra.run.smithi183.stdout: "cluster_addr": "172.21.15.183:6813/3346539414", 2024-09-10T19:24:34.494 INFO:teuthology.orchestra.run.smithi183.stdout: "heartbeat_back_addr": "172.21.15.183:6817/3346539414", 2024-09-10T19:24:34.494 INFO:teuthology.orchestra.run.smithi183.stdout: "heartbeat_front_addr": "172.21.15.183:6815/3346539414", 2024-09-10T19:24:34.494 INFO:teuthology.orchestra.run.smithi183.stdout: "state": [ 2024-09-10T19:24:34.494 INFO:teuthology.orchestra.run.smithi183.stdout: "exists", 2024-09-10T19:24:34.494 INFO:teuthology.orchestra.run.smithi183.stdout: "up" 2024-09-10T19:24:34.494 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:24:34.495 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:34.495 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:34.495 INFO:teuthology.orchestra.run.smithi183.stdout: "osd": 1, 2024-09-10T19:24:34.495 INFO:teuthology.orchestra.run.smithi183.stdout: "uuid": "71a6c5dc-5109-4284-8d20-187d8e7afc9e", 2024-09-10T19:24:34.495 INFO:teuthology.orchestra.run.smithi183.stdout: "up": 1, 2024-09-10T19:24:34.495 INFO:teuthology.orchestra.run.smithi183.stdout: "in": 1, 2024-09-10T19:24:34.495 INFO:teuthology.orchestra.run.smithi183.stdout: "weight": 1, 2024-09-10T19:24:34.495 INFO:teuthology.orchestra.run.smithi183.stdout: "primary_affinity": 1, 2024-09-10T19:24:34.495 INFO:teuthology.orchestra.run.smithi183.stdout: "last_clean_begin": 138, 2024-09-10T19:24:34.495 INFO:teuthology.orchestra.run.smithi183.stdout: "last_clean_end": 255, 2024-09-10T19:24:34.495 INFO:teuthology.orchestra.run.smithi183.stdout: "up_from": 276, 2024-09-10T19:24:34.495 INFO:teuthology.orchestra.run.smithi183.stdout: "up_thru": 305, 2024-09-10T19:24:34.495 INFO:teuthology.orchestra.run.smithi183.stdout: "down_at": 256, 2024-09-10T19:24:34.495 INFO:teuthology.orchestra.run.smithi183.stdout: "lost_at": 0, 2024-09-10T19:24:34.495 INFO:teuthology.orchestra.run.smithi183.stdout: "public_addrs": { 2024-09-10T19:24:34.496 INFO:teuthology.orchestra.run.smithi183.stdout: "addrvec": [ 2024-09-10T19:24:34.496 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:34.496 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v2", 2024-09-10T19:24:34.496 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.183:6802", 2024-09-10T19:24:34.496 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 2764394151 2024-09-10T19:24:34.496 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:34.496 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:34.496 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v1", 2024-09-10T19:24:34.496 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.183:6803", 2024-09-10T19:24:34.496 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 2764394151 2024-09-10T19:24:34.496 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:24:34.496 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:24:34.496 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:34.496 INFO:teuthology.orchestra.run.smithi183.stdout: "cluster_addrs": { 2024-09-10T19:24:34.496 INFO:teuthology.orchestra.run.smithi183.stdout: "addrvec": [ 2024-09-10T19:24:34.497 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:34.497 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v2", 2024-09-10T19:24:34.497 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.183:6804", 2024-09-10T19:24:34.497 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 2764394151 2024-09-10T19:24:34.497 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:34.497 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:34.497 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v1", 2024-09-10T19:24:34.497 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.183:6805", 2024-09-10T19:24:34.497 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 2764394151 2024-09-10T19:24:34.497 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:24:34.497 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:24:34.497 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:34.497 INFO:teuthology.orchestra.run.smithi183.stdout: "heartbeat_back_addrs": { 2024-09-10T19:24:34.498 INFO:teuthology.orchestra.run.smithi183.stdout: "addrvec": [ 2024-09-10T19:24:34.498 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:34.498 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v2", 2024-09-10T19:24:34.498 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.183:6808", 2024-09-10T19:24:34.498 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 2764394151 2024-09-10T19:24:34.498 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:34.498 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:34.498 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v1", 2024-09-10T19:24:34.498 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.183:6809", 2024-09-10T19:24:34.498 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 2764394151 2024-09-10T19:24:34.498 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:24:34.498 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:24:34.498 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:34.499 INFO:teuthology.orchestra.run.smithi183.stdout: "heartbeat_front_addrs": { 2024-09-10T19:24:34.499 INFO:teuthology.orchestra.run.smithi183.stdout: "addrvec": [ 2024-09-10T19:24:34.499 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:34.499 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v2", 2024-09-10T19:24:34.499 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.183:6806", 2024-09-10T19:24:34.499 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 2764394151 2024-09-10T19:24:34.499 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:34.499 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:34.499 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v1", 2024-09-10T19:24:34.499 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.183:6807", 2024-09-10T19:24:34.499 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 2764394151 2024-09-10T19:24:34.499 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:24:34.499 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:24:34.500 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:34.500 INFO:teuthology.orchestra.run.smithi183.stdout: "public_addr": "172.21.15.183:6803/2764394151", 2024-09-10T19:24:34.500 INFO:teuthology.orchestra.run.smithi183.stdout: "cluster_addr": "172.21.15.183:6805/2764394151", 2024-09-10T19:24:34.500 INFO:teuthology.orchestra.run.smithi183.stdout: "heartbeat_back_addr": "172.21.15.183:6809/2764394151", 2024-09-10T19:24:34.500 INFO:teuthology.orchestra.run.smithi183.stdout: "heartbeat_front_addr": "172.21.15.183:6807/2764394151", 2024-09-10T19:24:34.500 INFO:teuthology.orchestra.run.smithi183.stdout: "state": [ 2024-09-10T19:24:34.500 INFO:teuthology.orchestra.run.smithi183.stdout: "exists", 2024-09-10T19:24:34.500 INFO:teuthology.orchestra.run.smithi183.stdout: "up" 2024-09-10T19:24:34.500 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:24:34.500 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:34.500 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:34.500 INFO:teuthology.orchestra.run.smithi183.stdout: "osd": 2, 2024-09-10T19:24:34.500 INFO:teuthology.orchestra.run.smithi183.stdout: "uuid": "fc310d5b-b89b-4655-9989-7e10ba356509", 2024-09-10T19:24:34.501 INFO:teuthology.orchestra.run.smithi183.stdout: "up": 1, 2024-09-10T19:24:34.501 INFO:teuthology.orchestra.run.smithi183.stdout: "in": 0, 2024-09-10T19:24:34.501 INFO:teuthology.orchestra.run.smithi183.stdout: "weight": 0, 2024-09-10T19:24:34.501 INFO:teuthology.orchestra.run.smithi183.stdout: "primary_affinity": 1, 2024-09-10T19:24:34.501 INFO:teuthology.orchestra.run.smithi183.stdout: "last_clean_begin": 0, 2024-09-10T19:24:34.501 INFO:teuthology.orchestra.run.smithi183.stdout: "last_clean_end": 0, 2024-09-10T19:24:34.501 INFO:teuthology.orchestra.run.smithi183.stdout: "up_from": 18, 2024-09-10T19:24:34.501 INFO:teuthology.orchestra.run.smithi183.stdout: "up_thru": 148, 2024-09-10T19:24:34.501 INFO:teuthology.orchestra.run.smithi183.stdout: "down_at": 0, 2024-09-10T19:24:34.501 INFO:teuthology.orchestra.run.smithi183.stdout: "lost_at": 0, 2024-09-10T19:24:34.501 INFO:teuthology.orchestra.run.smithi183.stdout: "public_addrs": { 2024-09-10T19:24:34.501 INFO:teuthology.orchestra.run.smithi183.stdout: "addrvec": [ 2024-09-10T19:24:34.501 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:34.501 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v2", 2024-09-10T19:24:34.501 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.183:6818", 2024-09-10T19:24:34.502 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 3730013786 2024-09-10T19:24:34.502 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:34.502 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:34.502 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v1", 2024-09-10T19:24:34.502 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.183:6819", 2024-09-10T19:24:34.502 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 3730013786 2024-09-10T19:24:34.502 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:24:34.502 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:24:34.502 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:34.502 INFO:teuthology.orchestra.run.smithi183.stdout: "cluster_addrs": { 2024-09-10T19:24:34.502 INFO:teuthology.orchestra.run.smithi183.stdout: "addrvec": [ 2024-09-10T19:24:34.502 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:34.502 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v2", 2024-09-10T19:24:34.503 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.183:6820", 2024-09-10T19:24:34.503 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 3730013786 2024-09-10T19:24:34.503 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:34.503 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:34.503 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v1", 2024-09-10T19:24:34.503 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.183:6821", 2024-09-10T19:24:34.503 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 3730013786 2024-09-10T19:24:34.503 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:24:34.503 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:24:34.503 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:34.503 INFO:teuthology.orchestra.run.smithi183.stdout: "heartbeat_back_addrs": { 2024-09-10T19:24:34.503 INFO:teuthology.orchestra.run.smithi183.stdout: "addrvec": [ 2024-09-10T19:24:34.503 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:34.503 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v2", 2024-09-10T19:24:34.503 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.183:6824", 2024-09-10T19:24:34.503 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 3730013786 2024-09-10T19:24:34.504 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:34.504 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:34.504 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v1", 2024-09-10T19:24:34.504 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.183:6825", 2024-09-10T19:24:34.504 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 3730013786 2024-09-10T19:24:34.504 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:24:34.504 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:24:34.504 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:34.504 INFO:teuthology.orchestra.run.smithi183.stdout: "heartbeat_front_addrs": { 2024-09-10T19:24:34.504 INFO:teuthology.orchestra.run.smithi183.stdout: "addrvec": [ 2024-09-10T19:24:34.505 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:34.505 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v2", 2024-09-10T19:24:34.505 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.183:6822", 2024-09-10T19:24:34.505 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 3730013786 2024-09-10T19:24:34.505 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:34.505 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:34.505 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v1", 2024-09-10T19:24:34.505 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.183:6823", 2024-09-10T19:24:34.506 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 3730013786 2024-09-10T19:24:34.506 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:24:34.506 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:24:34.506 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:34.506 INFO:teuthology.orchestra.run.smithi183.stdout: "public_addr": "172.21.15.183:6819/3730013786", 2024-09-10T19:24:34.506 INFO:teuthology.orchestra.run.smithi183.stdout: "cluster_addr": "172.21.15.183:6821/3730013786", 2024-09-10T19:24:34.506 INFO:teuthology.orchestra.run.smithi183.stdout: "heartbeat_back_addr": "172.21.15.183:6825/3730013786", 2024-09-10T19:24:34.506 INFO:teuthology.orchestra.run.smithi183.stdout: "heartbeat_front_addr": "172.21.15.183:6823/3730013786", 2024-09-10T19:24:34.506 INFO:teuthology.orchestra.run.smithi183.stdout: "state": [ 2024-09-10T19:24:34.506 INFO:teuthology.orchestra.run.smithi183.stdout: "exists", 2024-09-10T19:24:34.506 INFO:teuthology.orchestra.run.smithi183.stdout: "up" 2024-09-10T19:24:34.506 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:24:34.506 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:34.507 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:34.507 INFO:teuthology.orchestra.run.smithi183.stdout: "osd": 3, 2024-09-10T19:24:34.507 INFO:teuthology.orchestra.run.smithi183.stdout: "uuid": "7fd07a95-ad22-43f5-a968-539552283e89", 2024-09-10T19:24:34.507 INFO:teuthology.orchestra.run.smithi183.stdout: "up": 1, 2024-09-10T19:24:34.507 INFO:teuthology.orchestra.run.smithi183.stdout: "in": 1, 2024-09-10T19:24:34.507 INFO:teuthology.orchestra.run.smithi183.stdout: "weight": 1, 2024-09-10T19:24:34.507 INFO:teuthology.orchestra.run.smithi183.stdout: "primary_affinity": 1, 2024-09-10T19:24:34.507 INFO:teuthology.orchestra.run.smithi183.stdout: "last_clean_begin": 0, 2024-09-10T19:24:34.507 INFO:teuthology.orchestra.run.smithi183.stdout: "last_clean_end": 0, 2024-09-10T19:24:34.507 INFO:teuthology.orchestra.run.smithi183.stdout: "up_from": 26, 2024-09-10T19:24:34.507 INFO:teuthology.orchestra.run.smithi183.stdout: "up_thru": 307, 2024-09-10T19:24:34.507 INFO:teuthology.orchestra.run.smithi183.stdout: "down_at": 0, 2024-09-10T19:24:34.507 INFO:teuthology.orchestra.run.smithi183.stdout: "lost_at": 0, 2024-09-10T19:24:34.507 INFO:teuthology.orchestra.run.smithi183.stdout: "public_addrs": { 2024-09-10T19:24:34.507 INFO:teuthology.orchestra.run.smithi183.stdout: "addrvec": [ 2024-09-10T19:24:34.507 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:34.508 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v2", 2024-09-10T19:24:34.508 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.183:6826", 2024-09-10T19:24:34.508 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 3522836984 2024-09-10T19:24:34.508 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:34.508 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:34.508 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v1", 2024-09-10T19:24:34.508 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.183:6827", 2024-09-10T19:24:34.508 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 3522836984 2024-09-10T19:24:34.508 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:24:34.508 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:24:34.508 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:34.508 INFO:teuthology.orchestra.run.smithi183.stdout: "cluster_addrs": { 2024-09-10T19:24:34.508 INFO:teuthology.orchestra.run.smithi183.stdout: "addrvec": [ 2024-09-10T19:24:34.509 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:34.509 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v2", 2024-09-10T19:24:34.509 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.183:6828", 2024-09-10T19:24:34.509 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 3522836984 2024-09-10T19:24:34.509 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:34.509 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:34.509 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v1", 2024-09-10T19:24:34.509 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.183:6829", 2024-09-10T19:24:34.509 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 3522836984 2024-09-10T19:24:34.509 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:24:34.509 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:24:34.509 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:34.509 INFO:teuthology.orchestra.run.smithi183.stdout: "heartbeat_back_addrs": { 2024-09-10T19:24:34.509 INFO:teuthology.orchestra.run.smithi183.stdout: "addrvec": [ 2024-09-10T19:24:34.509 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:34.510 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v2", 2024-09-10T19:24:34.510 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.183:6832", 2024-09-10T19:24:34.510 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 3522836984 2024-09-10T19:24:34.510 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:34.510 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:34.510 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v1", 2024-09-10T19:24:34.510 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.183:6833", 2024-09-10T19:24:34.510 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 3522836984 2024-09-10T19:24:34.510 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:24:34.510 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:24:34.510 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:34.510 INFO:teuthology.orchestra.run.smithi183.stdout: "heartbeat_front_addrs": { 2024-09-10T19:24:34.510 INFO:teuthology.orchestra.run.smithi183.stdout: "addrvec": [ 2024-09-10T19:24:34.510 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:34.511 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v2", 2024-09-10T19:24:34.511 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.183:6830", 2024-09-10T19:24:34.511 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 3522836984 2024-09-10T19:24:34.511 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:34.511 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:34.511 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v1", 2024-09-10T19:24:34.511 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.183:6831", 2024-09-10T19:24:34.511 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 3522836984 2024-09-10T19:24:34.511 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:24:34.511 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:24:34.511 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:34.511 INFO:teuthology.orchestra.run.smithi183.stdout: "public_addr": "172.21.15.183:6827/3522836984", 2024-09-10T19:24:34.511 INFO:teuthology.orchestra.run.smithi183.stdout: "cluster_addr": "172.21.15.183:6829/3522836984", 2024-09-10T19:24:34.511 INFO:teuthology.orchestra.run.smithi183.stdout: "heartbeat_back_addr": "172.21.15.183:6833/3522836984", 2024-09-10T19:24:34.511 INFO:teuthology.orchestra.run.smithi183.stdout: "heartbeat_front_addr": "172.21.15.183:6831/3522836984", 2024-09-10T19:24:34.512 INFO:teuthology.orchestra.run.smithi183.stdout: "state": [ 2024-09-10T19:24:34.512 INFO:teuthology.orchestra.run.smithi183.stdout: "exists", 2024-09-10T19:24:34.512 INFO:teuthology.orchestra.run.smithi183.stdout: "up" 2024-09-10T19:24:34.512 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:24:34.512 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:34.512 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:34.512 INFO:teuthology.orchestra.run.smithi183.stdout: "osd": 4, 2024-09-10T19:24:34.512 INFO:teuthology.orchestra.run.smithi183.stdout: "uuid": "631083fb-e503-44da-9b6b-154e88539ee7", 2024-09-10T19:24:34.512 INFO:teuthology.orchestra.run.smithi183.stdout: "up": 1, 2024-09-10T19:24:34.512 INFO:teuthology.orchestra.run.smithi183.stdout: "in": 1, 2024-09-10T19:24:34.512 INFO:teuthology.orchestra.run.smithi183.stdout: "weight": 1, 2024-09-10T19:24:34.512 INFO:teuthology.orchestra.run.smithi183.stdout: "primary_affinity": 1, 2024-09-10T19:24:34.512 INFO:teuthology.orchestra.run.smithi183.stdout: "last_clean_begin": 0, 2024-09-10T19:24:34.513 INFO:teuthology.orchestra.run.smithi183.stdout: "last_clean_end": 0, 2024-09-10T19:24:34.513 INFO:teuthology.orchestra.run.smithi183.stdout: "up_from": 32, 2024-09-10T19:24:34.513 INFO:teuthology.orchestra.run.smithi183.stdout: "up_thru": 307, 2024-09-10T19:24:34.513 INFO:teuthology.orchestra.run.smithi183.stdout: "down_at": 0, 2024-09-10T19:24:34.513 INFO:teuthology.orchestra.run.smithi183.stdout: "lost_at": 0, 2024-09-10T19:24:34.513 INFO:teuthology.orchestra.run.smithi183.stdout: "public_addrs": { 2024-09-10T19:24:34.513 INFO:teuthology.orchestra.run.smithi183.stdout: "addrvec": [ 2024-09-10T19:24:34.513 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:34.516 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v2", 2024-09-10T19:24:34.516 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.191:6800", 2024-09-10T19:24:34.516 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 2674828539 2024-09-10T19:24:34.516 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:34.516 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:34.516 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v1", 2024-09-10T19:24:34.517 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.191:6801", 2024-09-10T19:24:34.517 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 2674828539 2024-09-10T19:24:34.517 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:24:34.517 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:24:34.517 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:34.517 INFO:teuthology.orchestra.run.smithi183.stdout: "cluster_addrs": { 2024-09-10T19:24:34.517 INFO:teuthology.orchestra.run.smithi183.stdout: "addrvec": [ 2024-09-10T19:24:34.517 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:34.517 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v2", 2024-09-10T19:24:34.517 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.191:6802", 2024-09-10T19:24:34.517 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 2674828539 2024-09-10T19:24:34.517 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:34.518 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:34.518 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v1", 2024-09-10T19:24:34.518 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.191:6803", 2024-09-10T19:24:34.518 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 2674828539 2024-09-10T19:24:34.518 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:24:34.518 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:24:34.518 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:34.518 INFO:teuthology.orchestra.run.smithi183.stdout: "heartbeat_back_addrs": { 2024-09-10T19:24:34.518 INFO:teuthology.orchestra.run.smithi183.stdout: "addrvec": [ 2024-09-10T19:24:34.518 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:34.518 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v2", 2024-09-10T19:24:34.518 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.191:6806", 2024-09-10T19:24:34.518 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 2674828539 2024-09-10T19:24:34.519 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:34.519 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:34.519 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v1", 2024-09-10T19:24:34.519 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.191:6807", 2024-09-10T19:24:34.519 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 2674828539 2024-09-10T19:24:34.519 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:24:34.519 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:24:34.519 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:34.519 INFO:teuthology.orchestra.run.smithi183.stdout: "heartbeat_front_addrs": { 2024-09-10T19:24:34.519 INFO:teuthology.orchestra.run.smithi183.stdout: "addrvec": [ 2024-09-10T19:24:34.519 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:34.519 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v2", 2024-09-10T19:24:34.519 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.191:6804", 2024-09-10T19:24:34.520 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 2674828539 2024-09-10T19:24:34.520 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:34.520 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:34.520 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v1", 2024-09-10T19:24:34.520 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.191:6805", 2024-09-10T19:24:34.520 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 2674828539 2024-09-10T19:24:34.520 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:24:34.520 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:24:34.520 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:34.520 INFO:teuthology.orchestra.run.smithi183.stdout: "public_addr": "172.21.15.191:6801/2674828539", 2024-09-10T19:24:34.521 INFO:teuthology.orchestra.run.smithi183.stdout: "cluster_addr": "172.21.15.191:6803/2674828539", 2024-09-10T19:24:34.521 INFO:teuthology.orchestra.run.smithi183.stdout: "heartbeat_back_addr": "172.21.15.191:6807/2674828539", 2024-09-10T19:24:34.521 INFO:teuthology.orchestra.run.smithi183.stdout: "heartbeat_front_addr": "172.21.15.191:6805/2674828539", 2024-09-10T19:24:34.521 INFO:teuthology.orchestra.run.smithi183.stdout: "state": [ 2024-09-10T19:24:34.521 INFO:teuthology.orchestra.run.smithi183.stdout: "exists", 2024-09-10T19:24:34.521 INFO:teuthology.orchestra.run.smithi183.stdout: "up" 2024-09-10T19:24:34.521 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:24:34.521 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:34.521 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:34.521 INFO:teuthology.orchestra.run.smithi183.stdout: "osd": 5, 2024-09-10T19:24:34.522 INFO:teuthology.orchestra.run.smithi183.stdout: "uuid": "6b963695-3eca-4e6e-9207-05b8010ebff9", 2024-09-10T19:24:34.522 INFO:teuthology.orchestra.run.smithi183.stdout: "up": 1, 2024-09-10T19:24:34.522 INFO:teuthology.orchestra.run.smithi183.stdout: "in": 1, 2024-09-10T19:24:34.522 INFO:teuthology.orchestra.run.smithi183.stdout: "weight": 1, 2024-09-10T19:24:34.522 INFO:teuthology.orchestra.run.smithi183.stdout: "primary_affinity": 1, 2024-09-10T19:24:34.522 INFO:teuthology.orchestra.run.smithi183.stdout: "last_clean_begin": 0, 2024-09-10T19:24:34.522 INFO:teuthology.orchestra.run.smithi183.stdout: "last_clean_end": 0, 2024-09-10T19:24:34.522 INFO:teuthology.orchestra.run.smithi183.stdout: "up_from": 39, 2024-09-10T19:24:34.522 INFO:teuthology.orchestra.run.smithi183.stdout: "up_thru": 308, 2024-09-10T19:24:34.522 INFO:teuthology.orchestra.run.smithi183.stdout: "down_at": 0, 2024-09-10T19:24:34.522 INFO:teuthology.orchestra.run.smithi183.stdout: "lost_at": 0, 2024-09-10T19:24:34.522 INFO:teuthology.orchestra.run.smithi183.stdout: "public_addrs": { 2024-09-10T19:24:34.522 INFO:teuthology.orchestra.run.smithi183.stdout: "addrvec": [ 2024-09-10T19:24:34.522 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:34.523 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v2", 2024-09-10T19:24:34.523 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.191:6808", 2024-09-10T19:24:34.523 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 681733378 2024-09-10T19:24:34.523 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:34.523 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:34.523 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v1", 2024-09-10T19:24:34.523 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.191:6809", 2024-09-10T19:24:34.523 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 681733378 2024-09-10T19:24:34.523 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:24:34.523 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:24:34.523 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:34.523 INFO:teuthology.orchestra.run.smithi183.stdout: "cluster_addrs": { 2024-09-10T19:24:34.523 INFO:teuthology.orchestra.run.smithi183.stdout: "addrvec": [ 2024-09-10T19:24:34.524 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:34.524 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v2", 2024-09-10T19:24:34.524 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.191:6810", 2024-09-10T19:24:34.524 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 681733378 2024-09-10T19:24:34.525 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:34.525 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:34.525 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v1", 2024-09-10T19:24:34.525 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.191:6811", 2024-09-10T19:24:34.525 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 681733378 2024-09-10T19:24:34.525 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:24:34.525 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:24:34.525 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:34.525 INFO:teuthology.orchestra.run.smithi183.stdout: "heartbeat_back_addrs": { 2024-09-10T19:24:34.525 INFO:teuthology.orchestra.run.smithi183.stdout: "addrvec": [ 2024-09-10T19:24:34.525 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:34.525 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v2", 2024-09-10T19:24:34.525 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.191:6814", 2024-09-10T19:24:34.525 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 681733378 2024-09-10T19:24:34.526 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:34.526 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:34.526 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v1", 2024-09-10T19:24:34.526 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.191:6815", 2024-09-10T19:24:34.526 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 681733378 2024-09-10T19:24:34.526 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:24:34.526 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:24:34.526 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:34.526 INFO:teuthology.orchestra.run.smithi183.stdout: "heartbeat_front_addrs": { 2024-09-10T19:24:34.526 INFO:teuthology.orchestra.run.smithi183.stdout: "addrvec": [ 2024-09-10T19:24:34.526 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:34.526 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v2", 2024-09-10T19:24:34.527 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.191:6812", 2024-09-10T19:24:34.527 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 681733378 2024-09-10T19:24:34.527 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:34.527 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:34.527 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v1", 2024-09-10T19:24:34.527 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.191:6813", 2024-09-10T19:24:34.527 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 681733378 2024-09-10T19:24:34.527 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:24:34.527 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:24:34.527 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:34.528 INFO:teuthology.orchestra.run.smithi183.stdout: "public_addr": "172.21.15.191:6809/681733378", 2024-09-10T19:24:34.528 INFO:teuthology.orchestra.run.smithi183.stdout: "cluster_addr": "172.21.15.191:6811/681733378", 2024-09-10T19:24:34.528 INFO:teuthology.orchestra.run.smithi183.stdout: "heartbeat_back_addr": "172.21.15.191:6815/681733378", 2024-09-10T19:24:34.528 INFO:teuthology.orchestra.run.smithi183.stdout: "heartbeat_front_addr": "172.21.15.191:6813/681733378", 2024-09-10T19:24:34.528 INFO:teuthology.orchestra.run.smithi183.stdout: "state": [ 2024-09-10T19:24:34.528 INFO:teuthology.orchestra.run.smithi183.stdout: "exists", 2024-09-10T19:24:34.528 INFO:teuthology.orchestra.run.smithi183.stdout: "up" 2024-09-10T19:24:34.528 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:24:34.528 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:34.528 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:34.528 INFO:teuthology.orchestra.run.smithi183.stdout: "osd": 6, 2024-09-10T19:24:34.528 INFO:teuthology.orchestra.run.smithi183.stdout: "uuid": "2dc9754f-6aef-4e19-8605-2f1591921811", 2024-09-10T19:24:34.528 INFO:teuthology.orchestra.run.smithi183.stdout: "up": 1, 2024-09-10T19:24:34.529 INFO:teuthology.orchestra.run.smithi183.stdout: "in": 1, 2024-09-10T19:24:34.529 INFO:teuthology.orchestra.run.smithi183.stdout: "weight": 1, 2024-09-10T19:24:34.529 INFO:teuthology.orchestra.run.smithi183.stdout: "primary_affinity": 1, 2024-09-10T19:24:34.529 INFO:teuthology.orchestra.run.smithi183.stdout: "last_clean_begin": 0, 2024-09-10T19:24:34.529 INFO:teuthology.orchestra.run.smithi183.stdout: "last_clean_end": 0, 2024-09-10T19:24:34.529 INFO:teuthology.orchestra.run.smithi183.stdout: "up_from": 47, 2024-09-10T19:24:34.529 INFO:teuthology.orchestra.run.smithi183.stdout: "up_thru": 307, 2024-09-10T19:24:34.529 INFO:teuthology.orchestra.run.smithi183.stdout: "down_at": 0, 2024-09-10T19:24:34.529 INFO:teuthology.orchestra.run.smithi183.stdout: "lost_at": 0, 2024-09-10T19:24:34.529 INFO:teuthology.orchestra.run.smithi183.stdout: "public_addrs": { 2024-09-10T19:24:34.529 INFO:teuthology.orchestra.run.smithi183.stdout: "addrvec": [ 2024-09-10T19:24:34.529 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:34.529 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v2", 2024-09-10T19:24:34.530 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.191:6816", 2024-09-10T19:24:34.530 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 2825813477 2024-09-10T19:24:34.530 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:34.530 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:34.530 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v1", 2024-09-10T19:24:34.530 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.191:6817", 2024-09-10T19:24:34.530 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 2825813477 2024-09-10T19:24:34.530 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:24:34.530 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:24:34.530 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:34.530 INFO:teuthology.orchestra.run.smithi183.stdout: "cluster_addrs": { 2024-09-10T19:24:34.530 INFO:teuthology.orchestra.run.smithi183.stdout: "addrvec": [ 2024-09-10T19:24:34.530 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:34.530 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v2", 2024-09-10T19:24:34.531 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.191:6818", 2024-09-10T19:24:34.531 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 2825813477 2024-09-10T19:24:34.531 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:34.531 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:34.531 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v1", 2024-09-10T19:24:34.531 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.191:6819", 2024-09-10T19:24:34.531 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 2825813477 2024-09-10T19:24:34.531 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:24:34.531 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:24:34.531 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:34.531 INFO:teuthology.orchestra.run.smithi183.stdout: "heartbeat_back_addrs": { 2024-09-10T19:24:34.531 INFO:teuthology.orchestra.run.smithi183.stdout: "addrvec": [ 2024-09-10T19:24:34.532 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:34.532 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v2", 2024-09-10T19:24:34.532 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.191:6822", 2024-09-10T19:24:34.532 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 2825813477 2024-09-10T19:24:34.532 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:34.532 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:34.532 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v1", 2024-09-10T19:24:34.532 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.191:6823", 2024-09-10T19:24:34.532 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 2825813477 2024-09-10T19:24:34.532 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:24:34.532 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:24:34.532 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:34.533 INFO:teuthology.orchestra.run.smithi183.stdout: "heartbeat_front_addrs": { 2024-09-10T19:24:34.533 INFO:teuthology.orchestra.run.smithi183.stdout: "addrvec": [ 2024-09-10T19:24:34.533 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:34.533 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v2", 2024-09-10T19:24:34.533 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.191:6820", 2024-09-10T19:24:34.533 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 2825813477 2024-09-10T19:24:34.533 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:34.533 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:34.533 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v1", 2024-09-10T19:24:34.533 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.191:6821", 2024-09-10T19:24:34.533 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 2825813477 2024-09-10T19:24:34.533 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:24:34.533 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:24:34.533 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:34.534 INFO:teuthology.orchestra.run.smithi183.stdout: "public_addr": "172.21.15.191:6817/2825813477", 2024-09-10T19:24:34.534 INFO:teuthology.orchestra.run.smithi183.stdout: "cluster_addr": "172.21.15.191:6819/2825813477", 2024-09-10T19:24:34.534 INFO:teuthology.orchestra.run.smithi183.stdout: "heartbeat_back_addr": "172.21.15.191:6823/2825813477", 2024-09-10T19:24:34.534 INFO:teuthology.orchestra.run.smithi183.stdout: "heartbeat_front_addr": "172.21.15.191:6821/2825813477", 2024-09-10T19:24:34.534 INFO:teuthology.orchestra.run.smithi183.stdout: "state": [ 2024-09-10T19:24:34.534 INFO:teuthology.orchestra.run.smithi183.stdout: "exists", 2024-09-10T19:24:34.534 INFO:teuthology.orchestra.run.smithi183.stdout: "up" 2024-09-10T19:24:34.534 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:24:34.534 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:34.534 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:34.534 INFO:teuthology.orchestra.run.smithi183.stdout: "osd": 7, 2024-09-10T19:24:34.534 INFO:teuthology.orchestra.run.smithi183.stdout: "uuid": "1a9a299e-d5ba-4b9a-85d5-7f6a5bc754e1", 2024-09-10T19:24:34.535 INFO:teuthology.orchestra.run.smithi183.stdout: "up": 1, 2024-09-10T19:24:34.535 INFO:teuthology.orchestra.run.smithi183.stdout: "in": 1, 2024-09-10T19:24:34.535 INFO:teuthology.orchestra.run.smithi183.stdout: "weight": 1, 2024-09-10T19:24:34.535 INFO:teuthology.orchestra.run.smithi183.stdout: "primary_affinity": 1, 2024-09-10T19:24:34.535 INFO:teuthology.orchestra.run.smithi183.stdout: "last_clean_begin": 0, 2024-09-10T19:24:34.535 INFO:teuthology.orchestra.run.smithi183.stdout: "last_clean_end": 0, 2024-09-10T19:24:34.535 INFO:teuthology.orchestra.run.smithi183.stdout: "up_from": 54, 2024-09-10T19:24:34.535 INFO:teuthology.orchestra.run.smithi183.stdout: "up_thru": 306, 2024-09-10T19:24:34.535 INFO:teuthology.orchestra.run.smithi183.stdout: "down_at": 0, 2024-09-10T19:24:34.535 INFO:teuthology.orchestra.run.smithi183.stdout: "lost_at": 0, 2024-09-10T19:24:34.535 INFO:teuthology.orchestra.run.smithi183.stdout: "public_addrs": { 2024-09-10T19:24:34.535 INFO:teuthology.orchestra.run.smithi183.stdout: "addrvec": [ 2024-09-10T19:24:34.535 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:34.535 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v2", 2024-09-10T19:24:34.536 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.191:6824", 2024-09-10T19:24:34.536 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 1422610021 2024-09-10T19:24:34.536 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:34.536 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:34.536 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v1", 2024-09-10T19:24:34.536 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.191:6825", 2024-09-10T19:24:34.536 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 1422610021 2024-09-10T19:24:34.536 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:24:34.536 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:24:34.536 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:34.536 INFO:teuthology.orchestra.run.smithi183.stdout: "cluster_addrs": { 2024-09-10T19:24:34.536 INFO:teuthology.orchestra.run.smithi183.stdout: "addrvec": [ 2024-09-10T19:24:34.536 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:34.536 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v2", 2024-09-10T19:24:34.536 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.191:6826", 2024-09-10T19:24:34.537 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 1422610021 2024-09-10T19:24:34.537 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:34.537 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:34.537 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v1", 2024-09-10T19:24:34.537 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.191:6827", 2024-09-10T19:24:34.537 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 1422610021 2024-09-10T19:24:34.537 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:24:34.537 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:24:34.537 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:34.538 INFO:teuthology.orchestra.run.smithi183.stdout: "heartbeat_back_addrs": { 2024-09-10T19:24:34.538 INFO:teuthology.orchestra.run.smithi183.stdout: "addrvec": [ 2024-09-10T19:24:34.538 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:34.538 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v2", 2024-09-10T19:24:34.538 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.191:6830", 2024-09-10T19:24:34.538 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 1422610021 2024-09-10T19:24:34.538 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:34.538 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:34.538 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v1", 2024-09-10T19:24:34.538 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.191:6831", 2024-09-10T19:24:34.538 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 1422610021 2024-09-10T19:24:34.538 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:24:34.538 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:24:34.539 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:34.539 INFO:teuthology.orchestra.run.smithi183.stdout: "heartbeat_front_addrs": { 2024-09-10T19:24:34.539 INFO:teuthology.orchestra.run.smithi183.stdout: "addrvec": [ 2024-09-10T19:24:34.539 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:34.539 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v2", 2024-09-10T19:24:34.539 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.191:6828", 2024-09-10T19:24:34.539 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 1422610021 2024-09-10T19:24:34.539 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:34.539 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:34.539 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v1", 2024-09-10T19:24:34.539 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.191:6829", 2024-09-10T19:24:34.539 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 1422610021 2024-09-10T19:24:34.539 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:24:34.539 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:24:34.540 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:34.540 INFO:teuthology.orchestra.run.smithi183.stdout: "public_addr": "172.21.15.191:6825/1422610021", 2024-09-10T19:24:34.540 INFO:teuthology.orchestra.run.smithi183.stdout: "cluster_addr": "172.21.15.191:6827/1422610021", 2024-09-10T19:24:34.540 INFO:teuthology.orchestra.run.smithi183.stdout: "heartbeat_back_addr": "172.21.15.191:6831/1422610021", 2024-09-10T19:24:34.540 INFO:teuthology.orchestra.run.smithi183.stdout: "heartbeat_front_addr": "172.21.15.191:6829/1422610021", 2024-09-10T19:24:34.540 INFO:teuthology.orchestra.run.smithi183.stdout: "state": [ 2024-09-10T19:24:34.540 INFO:teuthology.orchestra.run.smithi183.stdout: "exists", 2024-09-10T19:24:34.540 INFO:teuthology.orchestra.run.smithi183.stdout: "up" 2024-09-10T19:24:34.540 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:24:34.540 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:24:34.540 INFO:teuthology.orchestra.run.smithi183.stdout: ], 2024-09-10T19:24:34.540 INFO:teuthology.orchestra.run.smithi183.stdout: "osd_xinfo": [ 2024-09-10T19:24:34.540 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:34.541 INFO:teuthology.orchestra.run.smithi183.stdout: "osd": 0, 2024-09-10T19:24:34.541 INFO:teuthology.orchestra.run.smithi183.stdout: "down_stamp": "0.000000", 2024-09-10T19:24:34.541 INFO:teuthology.orchestra.run.smithi183.stdout: "laggy_probability": 0, 2024-09-10T19:24:34.541 INFO:teuthology.orchestra.run.smithi183.stdout: "laggy_interval": 0, 2024-09-10T19:24:34.541 INFO:teuthology.orchestra.run.smithi183.stdout: "features": 4540138320759226367, 2024-09-10T19:24:34.541 INFO:teuthology.orchestra.run.smithi183.stdout: "old_weight": 0, 2024-09-10T19:24:34.541 INFO:teuthology.orchestra.run.smithi183.stdout: "last_purged_snaps_scrub": "2024-09-10T19:24:22.789037+0000", 2024-09-10T19:24:34.541 INFO:teuthology.orchestra.run.smithi183.stdout: "dead_epoch": 262 2024-09-10T19:24:34.541 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:34.541 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:34.541 INFO:teuthology.orchestra.run.smithi183.stdout: "osd": 1, 2024-09-10T19:24:34.541 INFO:teuthology.orchestra.run.smithi183.stdout: "down_stamp": "0.000000", 2024-09-10T19:24:34.541 INFO:teuthology.orchestra.run.smithi183.stdout: "laggy_probability": 0, 2024-09-10T19:24:34.541 INFO:teuthology.orchestra.run.smithi183.stdout: "laggy_interval": 0, 2024-09-10T19:24:34.542 INFO:teuthology.orchestra.run.smithi183.stdout: "features": 4540138320759226367, 2024-09-10T19:24:34.542 INFO:teuthology.orchestra.run.smithi183.stdout: "old_weight": 0, 2024-09-10T19:24:34.542 INFO:teuthology.orchestra.run.smithi183.stdout: "last_purged_snaps_scrub": "2024-09-10T19:23:53.365179+0000", 2024-09-10T19:24:34.542 INFO:teuthology.orchestra.run.smithi183.stdout: "dead_epoch": 255 2024-09-10T19:24:34.542 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:34.542 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:34.542 INFO:teuthology.orchestra.run.smithi183.stdout: "osd": 2, 2024-09-10T19:24:34.542 INFO:teuthology.orchestra.run.smithi183.stdout: "down_stamp": "0.000000", 2024-09-10T19:24:34.542 INFO:teuthology.orchestra.run.smithi183.stdout: "laggy_probability": 0, 2024-09-10T19:24:34.542 INFO:teuthology.orchestra.run.smithi183.stdout: "laggy_interval": 0, 2024-09-10T19:24:34.542 INFO:teuthology.orchestra.run.smithi183.stdout: "features": 4540138320759226367, 2024-09-10T19:24:34.542 INFO:teuthology.orchestra.run.smithi183.stdout: "old_weight": 65536, 2024-09-10T19:24:34.542 INFO:teuthology.orchestra.run.smithi183.stdout: "last_purged_snaps_scrub": "2024-09-10T19:23:45.722376+0000", 2024-09-10T19:24:34.543 INFO:teuthology.orchestra.run.smithi183.stdout: "dead_epoch": 0 2024-09-10T19:24:34.543 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:34.543 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:34.543 INFO:teuthology.orchestra.run.smithi183.stdout: "osd": 3, 2024-09-10T19:24:34.543 INFO:teuthology.orchestra.run.smithi183.stdout: "down_stamp": "0.000000", 2024-09-10T19:24:34.543 INFO:teuthology.orchestra.run.smithi183.stdout: "laggy_probability": 0, 2024-09-10T19:24:34.543 INFO:teuthology.orchestra.run.smithi183.stdout: "laggy_interval": 0, 2024-09-10T19:24:34.543 INFO:teuthology.orchestra.run.smithi183.stdout: "features": 4540138320759226367, 2024-09-10T19:24:34.543 INFO:teuthology.orchestra.run.smithi183.stdout: "old_weight": 0, 2024-09-10T19:24:34.543 INFO:teuthology.orchestra.run.smithi183.stdout: "last_purged_snaps_scrub": "2024-09-10T19:23:40.018030+0000", 2024-09-10T19:24:34.543 INFO:teuthology.orchestra.run.smithi183.stdout: "dead_epoch": 0 2024-09-10T19:24:34.543 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:34.543 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:34.544 INFO:teuthology.orchestra.run.smithi183.stdout: "osd": 4, 2024-09-10T19:24:34.544 INFO:teuthology.orchestra.run.smithi183.stdout: "down_stamp": "0.000000", 2024-09-10T19:24:34.544 INFO:teuthology.orchestra.run.smithi183.stdout: "laggy_probability": 0, 2024-09-10T19:24:34.544 INFO:teuthology.orchestra.run.smithi183.stdout: "laggy_interval": 0, 2024-09-10T19:24:34.544 INFO:teuthology.orchestra.run.smithi183.stdout: "features": 4540138320759226367, 2024-09-10T19:24:34.544 INFO:teuthology.orchestra.run.smithi183.stdout: "old_weight": 0, 2024-09-10T19:24:34.544 INFO:teuthology.orchestra.run.smithi183.stdout: "last_purged_snaps_scrub": "2024-09-10T19:24:31.957571+0000", 2024-09-10T19:24:34.544 INFO:teuthology.orchestra.run.smithi183.stdout: "dead_epoch": 0 2024-09-10T19:24:34.544 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:34.544 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:34.544 INFO:teuthology.orchestra.run.smithi183.stdout: "osd": 5, 2024-09-10T19:24:34.544 INFO:teuthology.orchestra.run.smithi183.stdout: "down_stamp": "0.000000", 2024-09-10T19:24:34.544 INFO:teuthology.orchestra.run.smithi183.stdout: "laggy_probability": 0, 2024-09-10T19:24:34.544 INFO:teuthology.orchestra.run.smithi183.stdout: "laggy_interval": 0, 2024-09-10T19:24:34.545 INFO:teuthology.orchestra.run.smithi183.stdout: "features": 4540138320759226367, 2024-09-10T19:24:34.545 INFO:teuthology.orchestra.run.smithi183.stdout: "old_weight": 0, 2024-09-10T19:24:34.545 INFO:teuthology.orchestra.run.smithi183.stdout: "last_purged_snaps_scrub": "2024-09-10T19:23:49.488736+0000", 2024-09-10T19:24:34.545 INFO:teuthology.orchestra.run.smithi183.stdout: "dead_epoch": 0 2024-09-10T19:24:34.545 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:34.545 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:34.545 INFO:teuthology.orchestra.run.smithi183.stdout: "osd": 6, 2024-09-10T19:24:34.545 INFO:teuthology.orchestra.run.smithi183.stdout: "down_stamp": "0.000000", 2024-09-10T19:24:34.545 INFO:teuthology.orchestra.run.smithi183.stdout: "laggy_probability": 0, 2024-09-10T19:24:34.545 INFO:teuthology.orchestra.run.smithi183.stdout: "laggy_interval": 0, 2024-09-10T19:24:34.545 INFO:teuthology.orchestra.run.smithi183.stdout: "features": 4540138320759226367, 2024-09-10T19:24:34.545 INFO:teuthology.orchestra.run.smithi183.stdout: "old_weight": 0, 2024-09-10T19:24:34.545 INFO:teuthology.orchestra.run.smithi183.stdout: "last_purged_snaps_scrub": "2024-09-10T19:23:31.994516+0000", 2024-09-10T19:24:34.545 INFO:teuthology.orchestra.run.smithi183.stdout: "dead_epoch": 0 2024-09-10T19:24:34.546 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:34.546 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:34.546 INFO:teuthology.orchestra.run.smithi183.stdout: "osd": 7, 2024-09-10T19:24:34.546 INFO:teuthology.orchestra.run.smithi183.stdout: "down_stamp": "0.000000", 2024-09-10T19:24:34.546 INFO:teuthology.orchestra.run.smithi183.stdout: "laggy_probability": 0, 2024-09-10T19:24:34.546 INFO:teuthology.orchestra.run.smithi183.stdout: "laggy_interval": 0, 2024-09-10T19:24:34.546 INFO:teuthology.orchestra.run.smithi183.stdout: "features": 4540138320759226367, 2024-09-10T19:24:34.546 INFO:teuthology.orchestra.run.smithi183.stdout: "old_weight": 0, 2024-09-10T19:24:34.546 INFO:teuthology.orchestra.run.smithi183.stdout: "last_purged_snaps_scrub": "2024-09-10T19:24:08.748653+0000", 2024-09-10T19:24:34.546 INFO:teuthology.orchestra.run.smithi183.stdout: "dead_epoch": 0 2024-09-10T19:24:34.546 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:24:34.546 INFO:teuthology.orchestra.run.smithi183.stdout: ], 2024-09-10T19:24:34.546 INFO:teuthology.orchestra.run.smithi183.stdout: "pg_upmap": [ 2024-09-10T19:24:34.546 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:34.547 INFO:teuthology.orchestra.run.smithi183.stdout: "pgid": "2.13", 2024-09-10T19:24:34.547 INFO:teuthology.orchestra.run.smithi183.stdout: "osds": [ 2024-09-10T19:24:34.547 INFO:teuthology.orchestra.run.smithi183.stdout: 2, 2024-09-10T19:24:34.547 INFO:teuthology.orchestra.run.smithi183.stdout: 0, 2024-09-10T19:24:34.547 INFO:teuthology.orchestra.run.smithi183.stdout: 5 2024-09-10T19:24:34.547 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:24:34.547 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:24:34.547 INFO:teuthology.orchestra.run.smithi183.stdout: ], 2024-09-10T19:24:34.547 INFO:teuthology.orchestra.run.smithi183.stdout: "pg_upmap_items": [ 2024-09-10T19:24:34.547 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:34.547 INFO:teuthology.orchestra.run.smithi183.stdout: "pgid": "2.18", 2024-09-10T19:24:34.547 INFO:teuthology.orchestra.run.smithi183.stdout: "mappings": [ 2024-09-10T19:24:34.547 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:34.547 INFO:teuthology.orchestra.run.smithi183.stdout: "from": 7, 2024-09-10T19:24:34.548 INFO:teuthology.orchestra.run.smithi183.stdout: "to": 0 2024-09-10T19:24:34.548 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:34.548 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:34.548 INFO:teuthology.orchestra.run.smithi183.stdout: "from": 5, 2024-09-10T19:24:34.548 INFO:teuthology.orchestra.run.smithi183.stdout: "to": 4 2024-09-10T19:24:34.548 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:24:34.548 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:24:34.548 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:34.548 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:34.549 INFO:teuthology.orchestra.run.smithi183.stdout: "pgid": "4.1", 2024-09-10T19:24:34.549 INFO:teuthology.orchestra.run.smithi183.stdout: "mappings": [ 2024-09-10T19:24:34.549 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:24:34.549 INFO:teuthology.orchestra.run.smithi183.stdout: "from": 6, 2024-09-10T19:24:34.549 INFO:teuthology.orchestra.run.smithi183.stdout: "to": 7 2024-09-10T19:24:34.549 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:24:34.549 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:24:34.549 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:24:34.549 INFO:teuthology.orchestra.run.smithi183.stdout: ], 2024-09-10T19:24:34.549 INFO:teuthology.orchestra.run.smithi183.stdout: "pg_temp": [], 2024-09-10T19:24:34.549 INFO:teuthology.orchestra.run.smithi183.stdout: "primary_temp": [], 2024-09-10T19:24:34.549 INFO:teuthology.orchestra.run.smithi183.stdout: "blocklist": { 2024-09-10T19:24:34.549 INFO:teuthology.orchestra.run.smithi183.stdout: "172.21.15.183:6801/3311783362": "2024-09-11T19:10:43.576943+0000", 2024-09-10T19:24:34.550 INFO:teuthology.orchestra.run.smithi183.stdout: "172.21.15.183:0/4088553922": "2024-09-11T19:10:43.576943+0000", 2024-09-10T19:24:34.550 INFO:teuthology.orchestra.run.smithi183.stdout: "172.21.15.183:0/3528769414": "2024-09-11T19:05:48.153879+0000", 2024-09-10T19:24:34.550 INFO:teuthology.orchestra.run.smithi183.stdout: "172.21.15.183:0/1766036658": "2024-09-11T19:05:48.153879+0000", 2024-09-10T19:24:34.550 INFO:teuthology.orchestra.run.smithi183.stdout: "172.21.15.183:0/3044170088": "2024-09-11T19:05:25.702354+0000", 2024-09-10T19:24:34.550 INFO:teuthology.orchestra.run.smithi183.stdout: "172.21.15.183:0/3037333778": "2024-09-11T19:05:25.702354+0000", 2024-09-10T19:24:34.550 INFO:teuthology.orchestra.run.smithi183.stdout: "172.21.15.183:6801/3747320570": "2024-09-11T19:05:25.702354+0000", 2024-09-10T19:24:34.550 INFO:teuthology.orchestra.run.smithi183.stdout: "172.21.15.183:0/3982386906": "2024-09-11T19:10:43.576943+0000", 2024-09-10T19:24:34.550 INFO:teuthology.orchestra.run.smithi183.stdout: "172.21.15.183:0/2436014455": "2024-09-11T19:05:25.702354+0000", 2024-09-10T19:24:34.550 INFO:teuthology.orchestra.run.smithi183.stdout: "172.21.15.183:6801/659745765": "2024-09-11T19:05:48.153879+0000", 2024-09-10T19:24:34.550 INFO:teuthology.orchestra.run.smithi183.stdout: "172.21.15.183:6800/3747320570": "2024-09-11T19:05:25.702354+0000", 2024-09-10T19:24:34.550 INFO:teuthology.orchestra.run.smithi183.stdout: "172.21.15.183:0/1652576282": "2024-09-11T19:10:43.576943+0000", 2024-09-10T19:24:34.550 INFO:teuthology.orchestra.run.smithi183.stdout: "172.21.15.183:6800/3311783362": "2024-09-11T19:10:43.576943+0000", 2024-09-10T19:24:34.550 INFO:teuthology.orchestra.run.smithi183.stdout: "172.21.15.183:0/1012899549": "2024-09-11T19:05:48.153879+0000", 2024-09-10T19:24:34.551 INFO:teuthology.orchestra.run.smithi183.stdout: "172.21.15.183:6800/659745765": "2024-09-11T19:05:48.153879+0000", 2024-09-10T19:24:34.551 INFO:teuthology.orchestra.run.smithi183.stdout: "172.21.15.183:0/412044400": "2024-09-11T19:10:43.576943+0000", 2024-09-10T19:24:34.551 INFO:teuthology.orchestra.run.smithi183.stdout: "172.21.15.183:0/3636171401": "2024-09-11T19:10:43.576943+0000" 2024-09-10T19:24:34.551 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:34.551 INFO:teuthology.orchestra.run.smithi183.stdout: "range_blocklist": {}, 2024-09-10T19:24:34.551 INFO:teuthology.orchestra.run.smithi183.stdout: "erasure_code_profiles": { 2024-09-10T19:24:34.551 INFO:teuthology.orchestra.run.smithi183.stdout: "default": { 2024-09-10T19:24:34.551 INFO:teuthology.orchestra.run.smithi183.stdout: "crush-failure-domain": "osd", 2024-09-10T19:24:34.551 INFO:teuthology.orchestra.run.smithi183.stdout: "k": "2", 2024-09-10T19:24:34.551 INFO:teuthology.orchestra.run.smithi183.stdout: "m": "1", 2024-09-10T19:24:34.551 INFO:teuthology.orchestra.run.smithi183.stdout: "plugin": "jerasure", 2024-09-10T19:24:34.551 INFO:teuthology.orchestra.run.smithi183.stdout: "technique": "reed_sol_van" 2024-09-10T19:24:34.551 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:24:34.551 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:24:34.552 INFO:teuthology.orchestra.run.smithi183.stdout: "removed_snaps_queue": [], 2024-09-10T19:24:34.552 INFO:teuthology.orchestra.run.smithi183.stdout: "new_removed_snaps": [], 2024-09-10T19:24:34.552 INFO:teuthology.orchestra.run.smithi183.stdout: "new_purged_snaps": [], 2024-09-10T19:24:34.552 INFO:teuthology.orchestra.run.smithi183.stdout: "crush_node_flags": {}, 2024-09-10T19:24:34.552 INFO:teuthology.orchestra.run.smithi183.stdout: "device_class_flags": {}, 2024-09-10T19:24:34.552 INFO:teuthology.orchestra.run.smithi183.stdout: "stretch_mode": { 2024-09-10T19:24:34.552 INFO:teuthology.orchestra.run.smithi183.stdout: "stretch_mode_enabled": false, 2024-09-10T19:24:34.552 INFO:teuthology.orchestra.run.smithi183.stdout: "stretch_bucket_count": 0, 2024-09-10T19:24:34.552 INFO:teuthology.orchestra.run.smithi183.stdout: "degraded_stretch_mode": 0, 2024-09-10T19:24:34.552 INFO:teuthology.orchestra.run.smithi183.stdout: "recovering_stretch_mode": 0, 2024-09-10T19:24:34.552 INFO:teuthology.orchestra.run.smithi183.stdout: "stretch_mode_bucket": 0 2024-09-10T19:24:34.552 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:24:34.552 INFO:teuthology.orchestra.run.smithi183.stdout:} 2024-09-10T19:24:34.553 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:24:34.573 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:24:34.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:24:34.574 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:24:34.675 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:24:34.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:24:34.714 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:24:34.714 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:24:34.815 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:24:34.842 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:24:34.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:24:34.843 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:24:34.853 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: 14 16 20797 20781 92.7638 115.125 0.00643064 0.0107735 2024-09-10T19:24:34.876 INFO:teuthology.orchestra.run.smithi183.stderr:noscrub is unset 2024-09-10T19:24:34.944 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:24:34.976 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:24:34.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:24:34.977 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:24:35.078 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:24:35.112 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:24:35.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:24:35.113 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:24:35.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:24:34 smithi183 ceph-mon[29736]: pgmap v655: 158 pgs: 1 peering, 157 active+clean; 844 MiB data, 11 GiB used, 615 GiB / 626 GiB avail; 1.0 KiB/s rd, 69 MiB/s wr, 1.10k op/s; 52 MiB/s, 826 objects/s recovering 2024-09-10T19:24:35.182 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:24:34 smithi183 ceph-mon[29736]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 1338/36126 objects degraded (3.704%), 3 pgs degraded) 2024-09-10T19:24:35.182 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:24:34 smithi183 ceph-mon[29736]: from='client.? 172.21.15.183:0/2180858416' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-09-10T19:24:35.182 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:24:34 smithi183 ceph-mon[29736]: from='client.? 172.21.15.183:0/2755841848' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-10T19:24:35.182 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:24:34 smithi183 ceph-mon[29736]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-10T19:24:35.182 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:24:34 smithi183 ceph-mon[29736]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T19:24:35.182 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:24:34 smithi183 ceph-mon[29736]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-10T19:24:35.182 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:24:34 smithi183 ceph-mon[29736]: osdmap e311: 8 total, 8 up, 7 in 2024-09-10T19:24:35.182 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:24:34 smithi183 ceph-mon[29736]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T19:24:35.182 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:24:34 smithi183 ceph-mon[29736]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T19:24:35.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:24:34 smithi183 ceph-mon[41562]: pgmap v655: 158 pgs: 1 peering, 157 active+clean; 844 MiB data, 11 GiB used, 615 GiB / 626 GiB avail; 1.0 KiB/s rd, 69 MiB/s wr, 1.10k op/s; 52 MiB/s, 826 objects/s recovering 2024-09-10T19:24:35.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:24:34 smithi183 ceph-mon[41562]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 1338/36126 objects degraded (3.704%), 3 pgs degraded) 2024-09-10T19:24:35.183 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:24:34 smithi183 ceph-mon[41562]: from='client.? 172.21.15.183:0/2180858416' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-09-10T19:24:35.183 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:24:34 smithi183 ceph-mon[41562]: from='client.? 172.21.15.183:0/2755841848' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-10T19:24:35.183 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:24:34 smithi183 ceph-mon[41562]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-10T19:24:35.183 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:24:34 smithi183 ceph-mon[41562]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T19:24:35.183 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:24:34 smithi183 ceph-mon[41562]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-10T19:24:35.183 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:24:34 smithi183 ceph-mon[41562]: osdmap e311: 8 total, 8 up, 7 in 2024-09-10T19:24:35.183 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:24:34 smithi183 ceph-mon[41562]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T19:24:35.183 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:24:34 smithi183 ceph-mon[41562]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T19:24:35.213 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:24:35.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:24:34 smithi191 ceph-mon[32884]: pgmap v655: 158 pgs: 1 peering, 157 active+clean; 844 MiB data, 11 GiB used, 615 GiB / 626 GiB avail; 1.0 KiB/s rd, 69 MiB/s wr, 1.10k op/s; 52 MiB/s, 826 objects/s recovering 2024-09-10T19:24:35.217 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:24:34 smithi191 ceph-mon[32884]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 1338/36126 objects degraded (3.704%), 3 pgs degraded) 2024-09-10T19:24:35.217 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:24:34 smithi191 ceph-mon[32884]: from='client.? 172.21.15.183:0/2180858416' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-09-10T19:24:35.217 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:24:34 smithi191 ceph-mon[32884]: from='client.? 172.21.15.183:0/2755841848' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-10T19:24:35.217 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:24:34 smithi191 ceph-mon[32884]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-10T19:24:35.217 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:24:34 smithi191 ceph-mon[32884]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T19:24:35.217 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:24:34 smithi191 ceph-mon[32884]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-10T19:24:35.217 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:24:34 smithi191 ceph-mon[32884]: osdmap e311: 8 total, 8 up, 7 in 2024-09-10T19:24:35.217 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:24:34 smithi191 ceph-mon[32884]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T19:24:35.217 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:24:34 smithi191 ceph-mon[32884]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T19:24:35.249 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:24:35.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:24:35.250 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:24:35.350 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:24:35.388 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:24:35.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:24:35.389 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:24:35.489 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:24:35.520 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:24:35.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:24:35.521 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:24:35.622 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:24:35.660 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:24:35.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:24:35.661 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:24:35.762 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:24:35.782 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:24:35.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:24:35.783 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:24:35.854 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: 15 15 22737 22722 94.6664 121.312 0.0051239 0.0105586 2024-09-10T19:24:35.884 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:24:35.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:24:35.922 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:24:35.922 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:24:36.022 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:24:36.050 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:24:36.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:24:36.051 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:24:36.152 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:24:36.179 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:24:36.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:24:36.180 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:24:36.181 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:24:35 smithi183 ceph-mon[41562]: from='mgr.14556 ' entity='mgr.y' 2024-09-10T19:24:36.181 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:24:35 smithi183 ceph-mon[41562]: 5.1d scrub starts 2024-09-10T19:24:36.181 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:24:35 smithi183 ceph-mon[41562]: 5.1d scrub ok 2024-09-10T19:24:36.181 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:24:35 smithi183 ceph-mon[41562]: 5.2 scrub starts 2024-09-10T19:24:36.181 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:24:35 smithi183 ceph-mon[41562]: 5.2 scrub ok 2024-09-10T19:24:36.182 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:24:35 smithi183 ceph-mon[29736]: from='mgr.14556 ' entity='mgr.y' 2024-09-10T19:24:36.182 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:24:35 smithi183 ceph-mon[29736]: 5.1d scrub starts 2024-09-10T19:24:36.182 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:24:35 smithi183 ceph-mon[29736]: 5.1d scrub ok 2024-09-10T19:24:36.182 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:24:35 smithi183 ceph-mon[29736]: 5.2 scrub starts 2024-09-10T19:24:36.182 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:24:35 smithi183 ceph-mon[29736]: 5.2 scrub ok 2024-09-10T19:24:36.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:24:35 smithi191 ceph-mon[32884]: from='mgr.14556 ' entity='mgr.y' 2024-09-10T19:24:36.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:24:35 smithi191 ceph-mon[32884]: 5.1d scrub starts 2024-09-10T19:24:36.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:24:35 smithi191 ceph-mon[32884]: 5.1d scrub ok 2024-09-10T19:24:36.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:24:35 smithi191 ceph-mon[32884]: 5.2 scrub starts 2024-09-10T19:24:36.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:24:35 smithi191 ceph-mon[32884]: 5.2 scrub ok 2024-09-10T19:24:36.281 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:24:36.315 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:24:36.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:24:36.316 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:24:36.416 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:24:36.455 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:24:36.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:24:36.456 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:24:36.557 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:24:36.588 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:24:36.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:24:36.589 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:24:36.689 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:24:36.719 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:24:36.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:24:36.720 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:24:36.821 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:24:36.849 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:24:36.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:24:36.850 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:24:36.853 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: 16 16 25205 25189 98.3856 154.188 0.00779977 0.0101585 2024-09-10T19:24:36.951 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:24:36.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:24:36.977 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:24:36.977 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:24:37.077 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:24:37.113 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:24:37.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:24:37.115 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:24:37.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:24:36 smithi183 ceph-mon[29736]: 6.13 scrub starts 2024-09-10T19:24:37.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:24:36 smithi183 ceph-mon[29736]: 6.13 scrub ok 2024-09-10T19:24:37.182 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:24:36 smithi183 ceph-mon[29736]: 2.5 scrub starts 2024-09-10T19:24:37.182 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:24:36 smithi183 ceph-mon[29736]: 2.5 scrub ok 2024-09-10T19:24:37.182 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:24:36 smithi183 ceph-mon[29736]: pgmap v657: 158 pgs: 158 active+clean; 1.2 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 854 B/s rd, 123 MiB/s wr, 1.96k op/s; 42 MiB/s, 679 objects/s recovering 2024-09-10T19:24:37.182 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:24:36 smithi183 ceph-mon[29736]: 1.6 scrub starts 2024-09-10T19:24:37.182 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:24:36 smithi183 ceph-mon[29736]: 1.6 scrub ok 2024-09-10T19:24:37.182 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:24:36 smithi183 ceph-mon[29736]: 5.d scrub starts 2024-09-10T19:24:37.182 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:24:36 smithi183 ceph-mon[29736]: 5.d scrub ok 2024-09-10T19:24:37.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:24:36 smithi183 ceph-mon[41562]: 6.13 scrub starts 2024-09-10T19:24:37.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:24:36 smithi183 ceph-mon[41562]: 6.13 scrub ok 2024-09-10T19:24:37.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:24:36 smithi183 ceph-mon[41562]: 2.5 scrub starts 2024-09-10T19:24:37.183 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:24:36 smithi183 ceph-mon[41562]: 2.5 scrub ok 2024-09-10T19:24:37.183 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:24:36 smithi183 ceph-mon[41562]: pgmap v657: 158 pgs: 158 active+clean; 1.2 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 854 B/s rd, 123 MiB/s wr, 1.96k op/s; 42 MiB/s, 679 objects/s recovering 2024-09-10T19:24:37.183 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:24:36 smithi183 ceph-mon[41562]: 1.6 scrub starts 2024-09-10T19:24:37.183 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:24:36 smithi183 ceph-mon[41562]: 1.6 scrub ok 2024-09-10T19:24:37.183 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:24:36 smithi183 ceph-mon[41562]: 5.d scrub starts 2024-09-10T19:24:37.183 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:24:36 smithi183 ceph-mon[41562]: 5.d scrub ok 2024-09-10T19:24:37.215 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:24:37.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:24:36 smithi191 ceph-mon[32884]: 6.13 scrub starts 2024-09-10T19:24:37.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:24:36 smithi191 ceph-mon[32884]: 6.13 scrub ok 2024-09-10T19:24:37.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:24:36 smithi191 ceph-mon[32884]: 2.5 scrub starts 2024-09-10T19:24:37.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:24:36 smithi191 ceph-mon[32884]: 2.5 scrub ok 2024-09-10T19:24:37.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:24:36 smithi191 ceph-mon[32884]: pgmap v657: 158 pgs: 158 active+clean; 1.2 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 854 B/s rd, 123 MiB/s wr, 1.96k op/s; 42 MiB/s, 679 objects/s recovering 2024-09-10T19:24:37.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:24:36 smithi191 ceph-mon[32884]: 1.6 scrub starts 2024-09-10T19:24:37.217 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:24:36 smithi191 ceph-mon[32884]: 1.6 scrub ok 2024-09-10T19:24:37.217 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:24:36 smithi191 ceph-mon[32884]: 5.d scrub starts 2024-09-10T19:24:37.217 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:24:36 smithi191 ceph-mon[32884]: 5.d scrub ok 2024-09-10T19:24:37.246 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:24:37.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:24:37.247 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:24:37.348 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:24:37.375 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:24:37.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:24:37.377 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:24:37.477 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:24:37.509 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:24:37.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:24:37.510 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:24:37.611 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:24:37.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:24:37.643 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:24:37.643 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:24:37.715 INFO:journalctl@ceph.mgr.x.smithi191.stdout:Sep 10 19:24:37 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-x[34967]: ::ffff:172.21.15.191 - - [10/Sep/2024:19:24:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T19:24:37.744 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:24:37.785 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:24:37.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:24:37.786 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:24:37.834 INFO:teuthology.orchestra.run.smithi191.stdout:{ 2024-09-10T19:24:37.835 INFO:teuthology.orchestra.run.smithi191.stdout: "ops": [], 2024-09-10T19:24:37.835 INFO:teuthology.orchestra.run.smithi191.stdout: "complaint_time": 30, 2024-09-10T19:24:37.835 INFO:teuthology.orchestra.run.smithi191.stdout: "num_blocked_ops": 0 2024-09-10T19:24:37.835 INFO:teuthology.orchestra.run.smithi191.stdout:} 2024-09-10T19:24:37.854 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: 17 16 27199 27183 99.9284 124.625 0.00494222 0.0100003 2024-09-10T19:24:37.859 INFO:tasks.thrashosds.thrasher:j is {'epoch': 310, 'fsid': '320f6df8-6fa7-11ef-bcea-c7b262605968', 'created': '2024-09-10T19:05:03.688531+0000', 'modified': '2024-09-10T19:24:32.938766+0000', 'last_up_change': '2024-09-10T19:24:22.870873+0000', 'last_in_change': '2024-09-10T19:24:00.937328+0000', 'flags': 'noscrub,nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5806080, 'flags_set': ['nodeep-scrub', 'noscrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 21, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 10, 'max_osd': 8, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'quincy', 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-10T19:08:00.456786+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 11, 'pg_placement_num': 11, 'pg_placement_num_target': 11, 'pg_num_target': 11, 'pg_num_pending': 11, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '271', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '242', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}, {'pool': 2, 'pool_name': '.rgw.root', 'create_time': '2024-09-10T19:10:05.993763+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '60', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}}, {'pool': 3, 'pool_name': 'datapool', 'create_time': '2024-09-10T19:10:06.224267+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 3, 'pg_placement_num': 3, 'pg_placement_num_target': 3, 'pg_num_target': 3, 'pg_num_pending': 3, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '63', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 3, 'snap_epoch': 63, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 4, 'pool_name': 'default.rgw.log', 'create_time': '2024-09-10T19:10:07.958284+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '62', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}}, {'pool': 5, 'pool_name': 'default.rgw.control', 'create_time': '2024-09-10T19:10:09.933596+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '64', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}}, {'pool': 6, 'pool_name': 'default.rgw.meta', 'create_time': '2024-09-10T19:10:11.993712+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '66', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4}, 'application_metadata': {'rgw': {}}}, {'pool': 10, 'pool_name': 'unique_pool_3', 'create_time': '2024-09-10T19:24:07.399856+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 16, 'pg_placement_num': 16, 'pg_placement_num_target': 16, 'pg_num_target': 16, 'pg_num_pending': 16, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '299', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}}], 'osds': [{'osd': 0, 'uuid': '9d323010-394a-49aa-ba9b-244d87faf36e', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 8, 'last_clean_end': 262, 'up_from': 301, 'up_thru': 308, 'down_at': 263, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.183:6810', 'nonce': 3346539414}, {'type': 'v1', 'addr': '172.21.15.183:6811', 'nonce': 3346539414}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.183:6812', 'nonce': 3346539414}, {'type': 'v1', 'addr': '172.21.15.183:6813', 'nonce': 3346539414}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.183:6816', 'nonce': 3346539414}, {'type': 'v1', 'addr': '172.21.15.183:6817', 'nonce': 3346539414}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.183:6814', 'nonce': 3346539414}, {'type': 'v1', 'addr': '172.21.15.183:6815', 'nonce': 3346539414}]}, 'public_addr': '172.21.15.183:6811/3346539414', 'cluster_addr': '172.21.15.183:6813/3346539414', 'heartbeat_back_addr': '172.21.15.183:6817/3346539414', 'heartbeat_front_addr': '172.21.15.183:6815/3346539414', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': '71a6c5dc-5109-4284-8d20-187d8e7afc9e', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 138, 'last_clean_end': 255, 'up_from': 276, 'up_thru': 305, 'down_at': 256, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.183:6802', 'nonce': 2764394151}, {'type': 'v1', 'addr': '172.21.15.183:6803', 'nonce': 2764394151}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.183:6804', 'nonce': 2764394151}, {'type': 'v1', 'addr': '172.21.15.183:6805', 'nonce': 2764394151}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.183:6808', 'nonce': 2764394151}, {'type': 'v1', 'addr': '172.21.15.183:6809', 'nonce': 2764394151}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.183:6806', 'nonce': 2764394151}, {'type': 'v1', 'addr': '172.21.15.183:6807', 'nonce': 2764394151}]}, 'public_addr': '172.21.15.183:6803/2764394151', 'cluster_addr': '172.21.15.183:6805/2764394151', 'heartbeat_back_addr': '172.21.15.183:6809/2764394151', 'heartbeat_front_addr': '172.21.15.183:6807/2764394151', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': 'fc310d5b-b89b-4655-9989-7e10ba356509', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 18, 'up_thru': 148, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.183:6818', 'nonce': 3730013786}, {'type': 'v1', 'addr': '172.21.15.183:6819', 'nonce': 3730013786}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.183:6820', 'nonce': 3730013786}, {'type': 'v1', 'addr': '172.21.15.183:6821', 'nonce': 3730013786}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.183:6824', 'nonce': 3730013786}, {'type': 'v1', 'addr': '172.21.15.183:6825', 'nonce': 3730013786}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.183:6822', 'nonce': 3730013786}, {'type': 'v1', 'addr': '172.21.15.183:6823', 'nonce': 3730013786}]}, 'public_addr': '172.21.15.183:6819/3730013786', 'cluster_addr': '172.21.15.183:6821/3730013786', 'heartbeat_back_addr': '172.21.15.183:6825/3730013786', 'heartbeat_front_addr': '172.21.15.183:6823/3730013786', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': '7fd07a95-ad22-43f5-a968-539552283e89', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 26, 'up_thru': 307, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.183:6826', 'nonce': 3522836984}, {'type': 'v1', 'addr': '172.21.15.183:6827', 'nonce': 3522836984}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.183:6828', 'nonce': 3522836984}, {'type': 'v1', 'addr': '172.21.15.183:6829', 'nonce': 3522836984}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.183:6832', 'nonce': 3522836984}, {'type': 'v1', 'addr': '172.21.15.183:6833', 'nonce': 3522836984}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.183:6830', 'nonce': 3522836984}, {'type': 'v1', 'addr': '172.21.15.183:6831', 'nonce': 3522836984}]}, 'public_addr': '172.21.15.183:6827/3522836984', 'cluster_addr': '172.21.15.183:6829/3522836984', 'heartbeat_back_addr': '172.21.15.183:6833/3522836984', 'heartbeat_front_addr': '172.21.15.183:6831/3522836984', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': '631083fb-e503-44da-9b6b-154e88539ee7', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 32, 'up_thru': 307, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.191:6800', 'nonce': 2674828539}, {'type': 'v1', 'addr': '172.21.15.191:6801', 'nonce': 2674828539}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.191:6802', 'nonce': 2674828539}, {'type': 'v1', 'addr': '172.21.15.191:6803', 'nonce': 2674828539}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.191:6806', 'nonce': 2674828539}, {'type': 'v1', 'addr': '172.21.15.191:6807', 'nonce': 2674828539}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.191:6804', 'nonce': 2674828539}, {'type': 'v1', 'addr': '172.21.15.191:6805', 'nonce': 2674828539}]}, 'public_addr': '172.21.15.191:6801/2674828539', 'cluster_addr': '172.21.15.191:6803/2674828539', 'heartbeat_back_addr': '172.21.15.191:6807/2674828539', 'heartbeat_front_addr': '172.21.15.191:6805/2674828539', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': '6b963695-3eca-4e6e-9207-05b8010ebff9', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 39, 'up_thru': 308, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.191:6808', 'nonce': 681733378}, {'type': 'v1', 'addr': '172.21.15.191:6809', 'nonce': 681733378}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.191:6810', 'nonce': 681733378}, {'type': 'v1', 'addr': '172.21.15.191:6811', 'nonce': 681733378}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.191:6814', 'nonce': 681733378}, {'type': 'v1', 'addr': '172.21.15.191:6815', 'nonce': 681733378}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.191:6812', 'nonce': 681733378}, {'type': 'v1', 'addr': '172.21.15.191:6813', 'nonce': 681733378}]}, 'public_addr': '172.21.15.191:6809/681733378', 'cluster_addr': '172.21.15.191:6811/681733378', 'heartbeat_back_addr': '172.21.15.191:6815/681733378', 'heartbeat_front_addr': '172.21.15.191:6813/681733378', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': '2dc9754f-6aef-4e19-8605-2f1591921811', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 47, 'up_thru': 307, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.191:6816', 'nonce': 2825813477}, {'type': 'v1', 'addr': '172.21.15.191:6817', 'nonce': 2825813477}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.191:6818', 'nonce': 2825813477}, {'type': 'v1', 'addr': '172.21.15.191:6819', 'nonce': 2825813477}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.191:6822', 'nonce': 2825813477}, {'type': 'v1', 'addr': '172.21.15.191:6823', 'nonce': 2825813477}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.191:6820', 'nonce': 2825813477}, {'type': 'v1', 'addr': '172.21.15.191:6821', 'nonce': 2825813477}]}, 'public_addr': '172.21.15.191:6817/2825813477', 'cluster_addr': '172.21.15.191:6819/2825813477', 'heartbeat_back_addr': '172.21.15.191:6823/2825813477', 'heartbeat_front_addr': '172.21.15.191:6821/2825813477', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': '1a9a299e-d5ba-4b9a-85d5-7f6a5bc754e1', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 54, 'up_thru': 306, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.191:6824', 'nonce': 1422610021}, {'type': 'v1', 'addr': '172.21.15.191:6825', 'nonce': 1422610021}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.191:6826', 'nonce': 1422610021}, {'type': 'v1', 'addr': '172.21.15.191:6827', 'nonce': 1422610021}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.191:6830', 'nonce': 1422610021}, {'type': 'v1', 'addr': '172.21.15.191:6831', 'nonce': 1422610021}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.191:6828', 'nonce': 1422610021}, {'type': 'v1', 'addr': '172.21.15.191:6829', 'nonce': 1422610021}]}, 'public_addr': '172.21.15.191:6825/1422610021', 'cluster_addr': '172.21.15.191:6827/1422610021', 'heartbeat_back_addr': '172.21.15.191:6831/1422610021', 'heartbeat_front_addr': '172.21.15.191:6829/1422610021', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138320759226367, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-10T19:24:22.789037+0000', 'dead_epoch': 262}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138320759226367, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-10T19:23:53.365179+0000', 'dead_epoch': 255}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138320759226367, 'old_weight': 65536, 'last_purged_snaps_scrub': '2024-09-10T19:23:45.722376+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138320759226367, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-10T19:23:40.018030+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138320759226367, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-10T19:24:31.957571+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138320759226367, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-10T19:23:49.488736+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138320759226367, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-10T19:23:31.994516+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138320759226367, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-10T19:24:08.748653+0000', 'dead_epoch': 0}], 'pg_upmap': [{'pgid': '2.13', 'osds': [2, 0, 5]}], 'pg_upmap_items': [{'pgid': '2.18', 'mappings': [{'from': 7, 'to': 0}, {'from': 5, 'to': 4}]}, {'pgid': '4.1', 'mappings': [{'from': 6, 'to': 7}]}], 'pg_temp': [], 'primary_temp': [], 'blocklist': {'172.21.15.183:6801/3311783362': '2024-09-11T19:10:43.576943+0000', '172.21.15.183:0/4088553922': '2024-09-11T19:10:43.576943+0000', '172.21.15.183:0/3528769414': '2024-09-11T19:05:48.153879+0000', '172.21.15.183:0/1766036658': '2024-09-11T19:05:48.153879+0000', '172.21.15.183:0/3044170088': '2024-09-11T19:05:25.702354+0000', '172.21.15.183:0/3037333778': '2024-09-11T19:05:25.702354+0000', '172.21.15.183:6801/3747320570': '2024-09-11T19:05:25.702354+0000', '172.21.15.183:0/3982386906': '2024-09-11T19:10:43.576943+0000', '172.21.15.183:0/2436014455': '2024-09-11T19:05:25.702354+0000', '172.21.15.183:6801/659745765': '2024-09-11T19:05:48.153879+0000', '172.21.15.183:6800/3747320570': '2024-09-11T19:05:25.702354+0000', '172.21.15.183:0/1652576282': '2024-09-11T19:10:43.576943+0000', '172.21.15.183:6800/3311783362': '2024-09-11T19:10:43.576943+0000', '172.21.15.183:0/1012899549': '2024-09-11T19:05:48.153879+0000', '172.21.15.183:6800/659745765': '2024-09-11T19:05:48.153879+0000', '172.21.15.183:0/412044400': '2024-09-11T19:10:43.576943+0000', '172.21.15.183:0/3636171401': '2024-09-11T19:10:43.576943+0000'}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-09-10T19:24:37.860 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph pg dump --format=json 2024-09-10T19:24:37.887 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:24:37.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:24:37.934 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:24:37.935 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:24:38.035 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:24:38.069 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:24:38.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:24:38.070 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:24:38.150 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:24:37 smithi183 ceph-mon[29736]: 6.11 scrub starts 2024-09-10T19:24:38.150 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:24:37 smithi183 ceph-mon[29736]: 6.11 scrub ok 2024-09-10T19:24:38.150 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:24:37 smithi183 ceph-mon[41562]: 6.11 scrub starts 2024-09-10T19:24:38.150 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:24:37 smithi183 ceph-mon[41562]: 6.11 scrub ok 2024-09-10T19:24:38.171 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:24:38.188 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:24:37 smithi191 ceph-mon[32884]: 6.11 scrub starts 2024-09-10T19:24:38.188 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:24:37 smithi191 ceph-mon[32884]: 6.11 scrub ok 2024-09-10T19:24:38.202 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:24:38.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:24:38.204 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:24:38.264 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-10T19:24:38.305 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:24:38.341 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:24:38.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:24:38.342 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:24:38.349 INFO:teuthology.orchestra.run.smithi183.stdout:osd.0: {} 2024-09-10T19:24:38.350 INFO:teuthology.orchestra.run.smithi183.stdout:osd.1: {} 2024-09-10T19:24:38.350 INFO:teuthology.orchestra.run.smithi183.stdout:osd.2: {} 2024-09-10T19:24:38.350 INFO:teuthology.orchestra.run.smithi183.stdout:osd.3: {} 2024-09-10T19:24:38.350 INFO:teuthology.orchestra.run.smithi183.stdout:osd.4: {} 2024-09-10T19:24:38.350 INFO:teuthology.orchestra.run.smithi183.stdout:osd.5: {} 2024-09-10T19:24:38.350 INFO:teuthology.orchestra.run.smithi183.stdout:osd.6: {} 2024-09-10T19:24:38.351 INFO:teuthology.orchestra.run.smithi183.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-10T19:24:38.351 INFO:teuthology.orchestra.run.smithi183.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-10T19:24:38.351 INFO:teuthology.orchestra.run.smithi183.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-10T19:24:38.351 INFO:teuthology.orchestra.run.smithi183.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-10T19:24:38.351 INFO:teuthology.orchestra.run.smithi183.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-10T19:24:38.351 INFO:teuthology.orchestra.run.smithi183.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-10T19:24:38.351 INFO:teuthology.orchestra.run.smithi183.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-10T19:24:38.351 INFO:teuthology.orchestra.run.smithi183.stdout:osd.7: {} 2024-09-10T19:24:38.352 INFO:teuthology.orchestra.run.smithi183.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-10T19:24:38.443 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:24:38.478 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:24:38.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:24:38.480 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:24:38.581 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:24:38.609 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:24:38.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:24:38.610 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:24:38.711 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:24:38.732 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:24:38.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:24:38.733 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:24:38.834 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:24:38.854 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: 18 15 29268 29253 101.564 129.375 0.00940645 0.00984033 2024-09-10T19:24:38.871 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:24:38.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:24:38.872 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:24:38.972 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:24:39.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:24:39.017 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:24:39.017 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:24:39.118 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:24:39.147 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:24:39.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:24:39.148 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:24:39.249 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:24:39.283 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:24:39.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:24:39.284 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:24:39.385 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:24:39.415 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:24:39.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:24:39.416 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:24:39.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:24:38 smithi183 ceph-mon[29736]: 4.0 scrub starts 2024-09-10T19:24:39.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:24:38 smithi183 ceph-mon[29736]: 4.0 scrub ok 2024-09-10T19:24:39.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:24:38 smithi183 ceph-mon[29736]: 6.17 scrub starts 2024-09-10T19:24:39.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:24:38 smithi183 ceph-mon[29736]: 6.17 scrub ok 2024-09-10T19:24:39.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:24:38 smithi183 ceph-mon[29736]: pgmap v658: 158 pgs: 158 active+clean; 1.4 GiB data, 12 GiB used, 613 GiB / 626 GiB avail; 1.4 KiB/s rd, 94 MiB/s wr, 1.50k op/s; 12 MiB/s, 191 objects/s recovering 2024-09-10T19:24:39.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:24:38 smithi183 ceph-mon[29736]: from='client.? 172.21.15.183:0/2030584730' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T19:24:39.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:24:38 smithi183 ceph-mon[41562]: 4.0 scrub starts 2024-09-10T19:24:39.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:24:38 smithi183 ceph-mon[41562]: 4.0 scrub ok 2024-09-10T19:24:39.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:24:38 smithi183 ceph-mon[41562]: 6.17 scrub starts 2024-09-10T19:24:39.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:24:38 smithi183 ceph-mon[41562]: 6.17 scrub ok 2024-09-10T19:24:39.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:24:38 smithi183 ceph-mon[41562]: pgmap v658: 158 pgs: 158 active+clean; 1.4 GiB data, 12 GiB used, 613 GiB / 626 GiB avail; 1.4 KiB/s rd, 94 MiB/s wr, 1.50k op/s; 12 MiB/s, 191 objects/s recovering 2024-09-10T19:24:39.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:24:38 smithi183 ceph-mon[41562]: from='client.? 172.21.15.183:0/2030584730' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T19:24:39.465 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:24:38 smithi191 ceph-mon[32884]: 4.0 scrub starts 2024-09-10T19:24:39.465 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:24:38 smithi191 ceph-mon[32884]: 4.0 scrub ok 2024-09-10T19:24:39.465 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:24:38 smithi191 ceph-mon[32884]: 6.17 scrub starts 2024-09-10T19:24:39.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:24:38 smithi191 ceph-mon[32884]: 6.17 scrub ok 2024-09-10T19:24:39.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:24:38 smithi191 ceph-mon[32884]: pgmap v658: 158 pgs: 158 active+clean; 1.4 GiB data, 12 GiB used, 613 GiB / 626 GiB avail; 1.4 KiB/s rd, 94 MiB/s wr, 1.50k op/s; 12 MiB/s, 191 objects/s recovering 2024-09-10T19:24:39.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:24:38 smithi191 ceph-mon[32884]: from='client.? 172.21.15.183:0/2030584730' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T19:24:39.516 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:24:39.551 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:24:39.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:24:39.552 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:24:39.612 DEBUG:teuthology.orchestra.run.smithi191:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph daemon osd.5 dump_historic_ops 2024-09-10T19:24:39.653 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:24:39.683 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:24:39.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:24:39.684 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:24:39.784 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:24:39.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:24:39.827 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:24:39.827 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:24:39.854 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: 19 16 31265 31249 102.783 124.75 0.00817013 0.00972369 2024-09-10T19:24:39.928 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:24:39.955 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:24:39.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:24:39.956 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:24:39.965 INFO:journalctl@ceph.iscsi.iscsi.a.smithi191.stdout:Sep 10 19:24:39 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-iscsi-iscsi-a[61816]: debug there is no tcmu-runner data available 2024-09-10T19:24:40.057 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:24:40.058 INFO:teuthology.orchestra.run.smithi191.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.b/config 2024-09-10T19:24:40.096 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:24:40.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:24:40.097 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:24:40.180 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:24:40.198 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:24:40.239 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:24:40.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:24:40.240 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:24:40.266 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:24:39 smithi191 ceph-mon[32884]: 6.a scrub starts 2024-09-10T19:24:40.266 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:24:39 smithi191 ceph-mon[32884]: 6.a scrub ok 2024-09-10T19:24:40.267 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:24:39 smithi191 ceph-mon[32884]: 1.7 scrub starts 2024-09-10T19:24:40.267 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:24:39 smithi191 ceph-mon[32884]: 1.7 scrub ok 2024-09-10T19:24:40.267 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:24:39 smithi191 ceph-mon[32884]: osdmap e312: 8 total, 8 up, 7 in 2024-09-10T19:24:40.312 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:24:39 smithi183 ceph-mon[29736]: 6.a scrub starts 2024-09-10T19:24:40.312 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:24:39 smithi183 ceph-mon[29736]: 6.a scrub ok 2024-09-10T19:24:40.312 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:24:39 smithi183 ceph-mon[29736]: 1.7 scrub starts 2024-09-10T19:24:40.313 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:24:39 smithi183 ceph-mon[29736]: 1.7 scrub ok 2024-09-10T19:24:40.313 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:24:39 smithi183 ceph-mon[29736]: osdmap e312: 8 total, 8 up, 7 in 2024-09-10T19:24:40.313 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:24:39 smithi183 ceph-mon[41562]: 6.a scrub starts 2024-09-10T19:24:40.313 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:24:39 smithi183 ceph-mon[41562]: 6.a scrub ok 2024-09-10T19:24:40.313 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:24:39 smithi183 ceph-mon[41562]: 1.7 scrub starts 2024-09-10T19:24:40.313 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:24:39 smithi183 ceph-mon[41562]: 1.7 scrub ok 2024-09-10T19:24:40.313 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:24:39 smithi183 ceph-mon[41562]: osdmap e312: 8 total, 8 up, 7 in 2024-09-10T19:24:40.342 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:24:40.378 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:24:40.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:24:40.380 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:24:40.480 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:24:40.501 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:24:40.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:24:40.502 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:24:40.603 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:24:40.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:24:40.641 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:24:40.641 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:24:40.742 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:24:41.123 INFO:tasks.radosbench.radosbench.0.smithi183.stdout:2024-09-10T19:24:40.853626+0000 min lat: 0.00110744 max lat: 2.06031 avg lat: 0.00968065 2024-09-10T19:24:41.123 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-10T19:24:41.123 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: 20 15 33057 33042 103.247 112.062 0.00449588 0.00968065 2024-09-10T19:24:41.131 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:24:40 smithi191 ceph-mon[32884]: from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:24:41.131 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:24:40 smithi191 ceph-mon[32884]: pgmap v659: 158 pgs: 158 active+clean; 1.4 GiB data, 12 GiB used, 613 GiB / 626 GiB avail; 1.2 KiB/s rd, 82 MiB/s wr, 1.31k op/s; 10 MiB/s, 167 objects/s recovering 2024-09-10T19:24:41.131 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:24:40 smithi191 ceph-mon[32884]: 1.0 scrub starts 2024-09-10T19:24:41.131 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:24:40 smithi191 ceph-mon[32884]: 1.0 scrub ok 2024-09-10T19:24:41.143 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:24:40 smithi183 ceph-mon[29736]: from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:24:41.144 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:24:40 smithi183 ceph-mon[29736]: pgmap v659: 158 pgs: 158 active+clean; 1.4 GiB data, 12 GiB used, 613 GiB / 626 GiB avail; 1.2 KiB/s rd, 82 MiB/s wr, 1.31k op/s; 10 MiB/s, 167 objects/s recovering 2024-09-10T19:24:41.144 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:24:40 smithi183 ceph-mon[29736]: 1.0 scrub starts 2024-09-10T19:24:41.144 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:24:40 smithi183 ceph-mon[29736]: 1.0 scrub ok 2024-09-10T19:24:41.144 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:24:40 smithi183 ceph-mon[41562]: from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:24:41.144 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:24:40 smithi183 ceph-mon[41562]: pgmap v659: 158 pgs: 158 active+clean; 1.4 GiB data, 12 GiB used, 613 GiB / 626 GiB avail; 1.2 KiB/s rd, 82 MiB/s wr, 1.31k op/s; 10 MiB/s, 167 objects/s recovering 2024-09-10T19:24:41.144 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:24:40 smithi183 ceph-mon[41562]: 1.0 scrub starts 2024-09-10T19:24:41.144 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:24:40 smithi183 ceph-mon[41562]: 1.0 scrub ok 2024-09-10T19:24:41.149 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:24:41.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:24:41.151 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:24:41.251 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:24:41.287 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:24:41.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:24:41.288 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:24:41.389 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:24:41.417 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:24:41.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:24:41.418 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:24:41.431 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Sep 10 19:24:41 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-y[29989]: ::ffff:172.21.15.191 - - [10/Sep/2024:19:24:41] "GET /metrics HTTP/1.1" 200 224435 "" "Prometheus/2.43.0" 2024-09-10T19:24:41.460 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:24:41.519 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:24:41.547 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:24:41.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:24:41.548 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:24:41.649 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:24:41.685 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:24:41.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:24:41.686 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:24:41.787 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:24:41.825 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:24:41.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:24:41.826 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:24:41.854 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: 21 16 34965 34949 104.005 119.188 0.00896768 0.00960992 2024-09-10T19:24:41.926 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:24:41.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:24:41.965 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:24:41.965 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:24:42.066 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:24:42.102 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:24:42.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:24:42.103 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:24:42.122 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-10T19:24:42.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:24:41 smithi183 ceph-mon[29736]: 4.19 scrub starts 2024-09-10T19:24:42.182 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:24:41 smithi183 ceph-mon[29736]: 4.19 scrub ok 2024-09-10T19:24:42.182 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:24:41 smithi183 ceph-mon[29736]: purged_snaps scrub starts 2024-09-10T19:24:42.182 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:24:41 smithi183 ceph-mon[29736]: purged_snaps scrub ok 2024-09-10T19:24:42.182 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:24:41 smithi183 ceph-mon[29736]: 2.18 scrub starts 2024-09-10T19:24:42.182 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:24:41 smithi183 ceph-mon[29736]: 2.18 scrub ok 2024-09-10T19:24:42.182 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:24:41 smithi183 ceph-mon[29736]: 5.4 scrub starts 2024-09-10T19:24:42.182 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:24:41 smithi183 ceph-mon[29736]: 5.4 scrub ok 2024-09-10T19:24:42.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:24:41 smithi183 ceph-mon[41562]: 4.19 scrub starts 2024-09-10T19:24:42.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:24:41 smithi183 ceph-mon[41562]: 4.19 scrub ok 2024-09-10T19:24:42.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:24:41 smithi183 ceph-mon[41562]: purged_snaps scrub starts 2024-09-10T19:24:42.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:24:41 smithi183 ceph-mon[41562]: purged_snaps scrub ok 2024-09-10T19:24:42.183 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:24:41 smithi183 ceph-mon[41562]: 2.18 scrub starts 2024-09-10T19:24:42.183 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:24:41 smithi183 ceph-mon[41562]: 2.18 scrub ok 2024-09-10T19:24:42.183 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:24:41 smithi183 ceph-mon[41562]: 5.4 scrub starts 2024-09-10T19:24:42.183 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:24:41 smithi183 ceph-mon[41562]: 5.4 scrub ok 2024-09-10T19:24:42.204 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:24:42.237 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:24:42.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:24:42.240 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:24:42.341 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:24:42.377 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:24:42.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:24:42.378 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:24:42.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:24:41 smithi191 ceph-mon[32884]: 4.19 scrub starts 2024-09-10T19:24:42.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:24:41 smithi191 ceph-mon[32884]: 4.19 scrub ok 2024-09-10T19:24:42.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:24:41 smithi191 ceph-mon[32884]: purged_snaps scrub starts 2024-09-10T19:24:42.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:24:41 smithi191 ceph-mon[32884]: purged_snaps scrub ok 2024-09-10T19:24:42.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:24:41 smithi191 ceph-mon[32884]: 2.18 scrub starts 2024-09-10T19:24:42.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:24:41 smithi191 ceph-mon[32884]: 2.18 scrub ok 2024-09-10T19:24:42.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:24:41 smithi191 ceph-mon[32884]: 5.4 scrub starts 2024-09-10T19:24:42.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:24:41 smithi191 ceph-mon[32884]: 5.4 scrub ok 2024-09-10T19:24:42.479 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:24:42.514 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:24:42.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:24:42.515 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:24:42.616 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:24:42.645 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:24:42.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:24:42.646 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:24:42.747 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:24:42.789 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:24:42.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:24:42.790 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:24:42.854 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: 22 16 36863 36847 104.67 118.625 0.00818922 0.00954934 2024-09-10T19:24:42.891 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:24:42.918 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:24:42.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:24:42.919 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:24:43.020 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:24:43.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:24:43.068 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:24:43.068 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:24:43.159 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:24:42 smithi191 ceph-mon[32884]: 5.e scrub starts 2024-09-10T19:24:43.159 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:24:42 smithi191 ceph-mon[32884]: 5.e scrub ok 2024-09-10T19:24:43.159 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:24:42 smithi191 ceph-mon[32884]: pgmap v661: 158 pgs: 158 active+clean; 1.8 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 639 B/s rd, 131 MiB/s wr, 2.09k op/s 2024-09-10T19:24:43.159 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:24:42 smithi191 ceph-mon[32884]: osdmap e313: 8 total, 8 up, 7 in 2024-09-10T19:24:43.159 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:24:42 smithi191 ceph-mon[32884]: 6.1 scrub starts 2024-09-10T19:24:43.159 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:24:42 smithi191 ceph-mon[32884]: 6.1 scrub ok 2024-09-10T19:24:43.159 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:24:42 smithi191 ceph-mon[32884]: 2.2 scrub starts 2024-09-10T19:24:43.159 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:24:42 smithi191 ceph-mon[32884]: 2.2 scrub ok 2024-09-10T19:24:43.160 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:24:42 smithi191 ceph-mon[32884]: 2.1f scrub starts 2024-09-10T19:24:43.160 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:24:42 smithi191 ceph-mon[32884]: 2.1f scrub ok 2024-09-10T19:24:43.169 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:24:43.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:24:42 smithi183 ceph-mon[29736]: 5.e scrub starts 2024-09-10T19:24:43.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:24:42 smithi183 ceph-mon[29736]: 5.e scrub ok 2024-09-10T19:24:43.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:24:42 smithi183 ceph-mon[29736]: pgmap v661: 158 pgs: 158 active+clean; 1.8 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 639 B/s rd, 131 MiB/s wr, 2.09k op/s 2024-09-10T19:24:43.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:24:42 smithi183 ceph-mon[29736]: osdmap e313: 8 total, 8 up, 7 in 2024-09-10T19:24:43.182 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:24:42 smithi183 ceph-mon[29736]: 6.1 scrub starts 2024-09-10T19:24:43.182 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:24:42 smithi183 ceph-mon[29736]: 6.1 scrub ok 2024-09-10T19:24:43.182 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:24:42 smithi183 ceph-mon[29736]: 2.2 scrub starts 2024-09-10T19:24:43.182 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:24:42 smithi183 ceph-mon[29736]: 2.2 scrub ok 2024-09-10T19:24:43.182 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:24:42 smithi183 ceph-mon[29736]: 2.1f scrub starts 2024-09-10T19:24:43.182 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:24:42 smithi183 ceph-mon[29736]: 2.1f scrub ok 2024-09-10T19:24:43.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:24:42 smithi183 ceph-mon[41562]: 5.e scrub starts 2024-09-10T19:24:43.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:24:42 smithi183 ceph-mon[41562]: 5.e scrub ok 2024-09-10T19:24:43.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:24:42 smithi183 ceph-mon[41562]: pgmap v661: 158 pgs: 158 active+clean; 1.8 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 639 B/s rd, 131 MiB/s wr, 2.09k op/s 2024-09-10T19:24:43.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:24:42 smithi183 ceph-mon[41562]: osdmap e313: 8 total, 8 up, 7 in 2024-09-10T19:24:43.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:24:42 smithi183 ceph-mon[41562]: 6.1 scrub starts 2024-09-10T19:24:43.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:24:42 smithi183 ceph-mon[41562]: 6.1 scrub ok 2024-09-10T19:24:43.183 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:24:42 smithi183 ceph-mon[41562]: 2.2 scrub starts 2024-09-10T19:24:43.183 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:24:42 smithi183 ceph-mon[41562]: 2.2 scrub ok 2024-09-10T19:24:43.183 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:24:42 smithi183 ceph-mon[41562]: 2.1f scrub starts 2024-09-10T19:24:43.183 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:24:42 smithi183 ceph-mon[41562]: 2.1f scrub ok 2024-09-10T19:24:43.197 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:24:43.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:24:43.198 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:24:43.199 INFO:teuthology.orchestra.run.smithi191.stdout:ERROR: (22) Invalid argument 2024-09-10T19:24:43.199 INFO:teuthology.orchestra.run.smithi191.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-10T19:24:43.299 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:24:43.326 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:24:43.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:24:43.327 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:24:43.428 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:24:43.458 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:24:43.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:24:43.459 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:24:43.560 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:24:43.588 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:24:43.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:24:43.589 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:24:43.690 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:24:43.731 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:24:43.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:24:43.732 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:24:43.833 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:24:43.855 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: 23 16 38746 38730 105.233 117.688 0.0080571 0.00949791 2024-09-10T19:24:43.867 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:24:43.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:24:43.868 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:24:43.969 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:24:43.972 DEBUG:teuthology.orchestra.run.smithi191:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph daemon osd.6 dump_ops_in_flight 2024-09-10T19:24:44.003 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:24:44.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:24:44.004 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:24:44.105 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:24:44.147 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:24:44.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:24:44.148 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:24:44.249 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:24:44.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:24:44.286 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:24:44.286 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:24:44.387 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:24:44.416 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:24:44.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:24:44.417 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:24:44.454 INFO:teuthology.orchestra.run.smithi191.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.b/config 2024-09-10T19:24:44.518 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:24:44.565 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:24:44.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:24:44.566 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:24:44.651 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:24:44.667 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:24:44.696 INFO:teuthology.orchestra.run.smithi183.stderr:nodeep-scrub is unset 2024-09-10T19:24:44.711 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:24:44.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:24:44.712 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:24:44.813 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:24:44.837 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-09-10T19:24:44.840 INFO:teuthology.orchestra.run.smithi183.stderr:dumped all 2024-09-10T19:24:44.845 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:24:44.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:24:44.846 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:24:44.855 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: 24 16 40444 40428 105.27 106.125 0.0141005 0.00949479 2024-09-10T19:24:44.948 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:24:44.966 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:24:44 smithi191 ceph-mon[32884]: 5.11 scrub starts 2024-09-10T19:24:44.966 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:24:44 smithi191 ceph-mon[32884]: 5.11 scrub ok 2024-09-10T19:24:44.966 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:24:44 smithi191 ceph-mon[32884]: 3.0 scrub starts 2024-09-10T19:24:44.966 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:24:44 smithi191 ceph-mon[32884]: 3.0 scrub ok 2024-09-10T19:24:44.966 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:24:44 smithi191 ceph-mon[32884]: 2.1e scrub starts 2024-09-10T19:24:44.967 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:24:44 smithi191 ceph-mon[32884]: 2.1e scrub ok 2024-09-10T19:24:44.967 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:24:44 smithi191 ceph-mon[32884]: 6.14 scrub starts 2024-09-10T19:24:44.967 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:24:44 smithi191 ceph-mon[32884]: 6.14 scrub ok 2024-09-10T19:24:44.967 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:24:44 smithi191 ceph-mon[32884]: from='mgr.14556 ' entity='mgr.y' 2024-09-10T19:24:44.967 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:24:44 smithi191 ceph-mon[32884]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:24:44.967 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:24:44 smithi191 ceph-mon[32884]: 2.1d scrub starts 2024-09-10T19:24:44.967 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:24:44 smithi191 ceph-mon[32884]: 2.1d scrub ok 2024-09-10T19:24:44.967 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:24:44 smithi191 ceph-mon[32884]: pgmap v663: 158 pgs: 158 active+clean; 2.0 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 1.2 KiB/s rd, 100 MiB/s wr, 1.60k op/s 2024-09-10T19:24:44.967 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:24:44 smithi191 ceph-mon[32884]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.b", "id": [1, 5]}]: dispatch 2024-09-10T19:24:44.967 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:24:44 smithi191 ceph-mon[32884]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.e", "id": [1, 5]}]: dispatch 2024-09-10T19:24:44.967 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:24:44 smithi191 ceph-mon[32884]: from='mgr.14556 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.b", "id": [1, 5]}]: dispatch 2024-09-10T19:24:44.967 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:24:44 smithi191 ceph-mon[32884]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.19", "id": [1, 6]}]: dispatch 2024-09-10T19:24:44.967 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:24:44 smithi191 ceph-mon[32884]: from='mgr.14556 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.e", "id": [1, 5]}]: dispatch 2024-09-10T19:24:44.967 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:24:44 smithi191 ceph-mon[32884]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.1b", "id": [3, 0]}]: dispatch 2024-09-10T19:24:44.968 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:24:44 smithi191 ceph-mon[32884]: from='mgr.14556 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.19", "id": [1, 6]}]: dispatch 2024-09-10T19:24:44.968 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:24:44 smithi191 ceph-mon[32884]: from='mgr.14556 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.1b", "id": [3, 0]}]: dispatch 2024-09-10T19:24:44.968 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:24:44 smithi191 ceph-mon[32884]: from='client.? 172.21.15.183:0/688005994' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-10T19:24:44.968 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:24:44 smithi191 ceph-mon[32884]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-10T19:24:44.969 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:24:44.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:24:44.970 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:24:45.034 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:24:44 smithi183 ceph-mon[29736]: 5.11 scrub starts 2024-09-10T19:24:45.034 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:24:44 smithi183 ceph-mon[29736]: 5.11 scrub ok 2024-09-10T19:24:45.034 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:24:44 smithi183 ceph-mon[29736]: 3.0 scrub starts 2024-09-10T19:24:45.034 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:24:44 smithi183 ceph-mon[29736]: 3.0 scrub ok 2024-09-10T19:24:45.034 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:24:44 smithi183 ceph-mon[29736]: 2.1e scrub starts 2024-09-10T19:24:45.034 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:24:44 smithi183 ceph-mon[29736]: 2.1e scrub ok 2024-09-10T19:24:45.034 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:24:44 smithi183 ceph-mon[29736]: 6.14 scrub starts 2024-09-10T19:24:45.034 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:24:44 smithi183 ceph-mon[29736]: 6.14 scrub ok 2024-09-10T19:24:45.034 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:24:44 smithi183 ceph-mon[29736]: from='mgr.14556 ' entity='mgr.y' 2024-09-10T19:24:45.034 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:24:44 smithi183 ceph-mon[29736]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:24:45.034 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:24:44 smithi183 ceph-mon[29736]: 2.1d scrub starts 2024-09-10T19:24:45.035 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:24:44 smithi183 ceph-mon[29736]: 2.1d scrub ok 2024-09-10T19:24:45.035 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:24:44 smithi183 ceph-mon[29736]: pgmap v663: 158 pgs: 158 active+clean; 2.0 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 1.2 KiB/s rd, 100 MiB/s wr, 1.60k op/s 2024-09-10T19:24:45.035 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:24:44 smithi183 ceph-mon[29736]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.b", "id": [1, 5]}]: dispatch 2024-09-10T19:24:45.035 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:24:44 smithi183 ceph-mon[29736]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.e", "id": [1, 5]}]: dispatch 2024-09-10T19:24:45.035 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:24:44 smithi183 ceph-mon[29736]: from='mgr.14556 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.b", "id": [1, 5]}]: dispatch 2024-09-10T19:24:45.035 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:24:44 smithi183 ceph-mon[29736]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.19", "id": [1, 6]}]: dispatch 2024-09-10T19:24:45.035 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:24:44 smithi183 ceph-mon[29736]: from='mgr.14556 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.e", "id": [1, 5]}]: dispatch 2024-09-10T19:24:45.035 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:24:44 smithi183 ceph-mon[29736]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.1b", "id": [3, 0]}]: dispatch 2024-09-10T19:24:45.035 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:24:44 smithi183 ceph-mon[29736]: from='mgr.14556 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.19", "id": [1, 6]}]: dispatch 2024-09-10T19:24:45.035 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:24:44 smithi183 ceph-mon[29736]: from='mgr.14556 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.1b", "id": [3, 0]}]: dispatch 2024-09-10T19:24:45.035 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:24:44 smithi183 ceph-mon[29736]: from='client.? 172.21.15.183:0/688005994' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-10T19:24:45.035 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:24:44 smithi183 ceph-mon[29736]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-10T19:24:45.036 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:24:44 smithi183 ceph-mon[41562]: 5.11 scrub starts 2024-09-10T19:24:45.036 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:24:44 smithi183 ceph-mon[41562]: 5.11 scrub ok 2024-09-10T19:24:45.036 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:24:44 smithi183 ceph-mon[41562]: 3.0 scrub starts 2024-09-10T19:24:45.036 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:24:44 smithi183 ceph-mon[41562]: 3.0 scrub ok 2024-09-10T19:24:45.036 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:24:44 smithi183 ceph-mon[41562]: 2.1e scrub starts 2024-09-10T19:24:45.036 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:24:44 smithi183 ceph-mon[41562]: 2.1e scrub ok 2024-09-10T19:24:45.036 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:24:44 smithi183 ceph-mon[41562]: 6.14 scrub starts 2024-09-10T19:24:45.036 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:24:44 smithi183 ceph-mon[41562]: 6.14 scrub ok 2024-09-10T19:24:45.036 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:24:44 smithi183 ceph-mon[41562]: from='mgr.14556 ' entity='mgr.y' 2024-09-10T19:24:45.036 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:24:44 smithi183 ceph-mon[41562]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:24:45.036 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:24:44 smithi183 ceph-mon[41562]: 2.1d scrub starts 2024-09-10T19:24:45.036 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:24:44 smithi183 ceph-mon[41562]: 2.1d scrub ok 2024-09-10T19:24:45.037 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:24:44 smithi183 ceph-mon[41562]: pgmap v663: 158 pgs: 158 active+clean; 2.0 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 1.2 KiB/s rd, 100 MiB/s wr, 1.60k op/s 2024-09-10T19:24:45.037 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:24:44 smithi183 ceph-mon[41562]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.b", "id": [1, 5]}]: dispatch 2024-09-10T19:24:45.037 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:24:44 smithi183 ceph-mon[41562]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.e", "id": [1, 5]}]: dispatch 2024-09-10T19:24:45.037 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:24:44 smithi183 ceph-mon[41562]: from='mgr.14556 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.b", "id": [1, 5]}]: dispatch 2024-09-10T19:24:45.037 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:24:44 smithi183 ceph-mon[41562]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.19", "id": [1, 6]}]: dispatch 2024-09-10T19:24:45.037 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:24:44 smithi183 ceph-mon[41562]: from='mgr.14556 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.e", "id": [1, 5]}]: dispatch 2024-09-10T19:24:45.037 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:24:44 smithi183 ceph-mon[41562]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.1b", "id": [3, 0]}]: dispatch 2024-09-10T19:24:45.037 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:24:44 smithi183 ceph-mon[41562]: from='mgr.14556 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.19", "id": [1, 6]}]: dispatch 2024-09-10T19:24:45.037 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:24:44 smithi183 ceph-mon[41562]: from='mgr.14556 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.1b", "id": [3, 0]}]: dispatch 2024-09-10T19:24:45.037 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:24:44 smithi183 ceph-mon[41562]: from='client.? 172.21.15.183:0/688005994' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-10T19:24:45.037 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:24:44 smithi183 ceph-mon[41562]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-10T19:24:45.071 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:24:45.108 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:24:45.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:24:45.109 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:24:45.210 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:24:45.242 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:24:45.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:24:45.243 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:24:45.344 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:24:45.372 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:24:45.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:24:45.373 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:24:45.474 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:24:45.499 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:24:45.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:24:45.500 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:24:45.601 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:24:45.633 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:24:45.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:24:45.634 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:24:45.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:24:45 smithi191 ceph-mon[32884]: 4.1c scrub starts 2024-09-10T19:24:45.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:24:45 smithi191 ceph-mon[32884]: 4.1c scrub ok 2024-09-10T19:24:45.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:24:45 smithi191 ceph-mon[32884]: 2.19 scrub starts 2024-09-10T19:24:45.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:24:45 smithi191 ceph-mon[32884]: 2.19 scrub ok 2024-09-10T19:24:45.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:24:45 smithi191 ceph-mon[32884]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-10T19:24:45.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:24:45 smithi191 ceph-mon[32884]: Cluster is now healthy 2024-09-10T19:24:45.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:24:45 smithi191 ceph-mon[32884]: from='mgr.14556 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.b", "id": [1, 5]}]': finished 2024-09-10T19:24:45.717 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:24:45 smithi191 ceph-mon[32884]: from='mgr.14556 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.e", "id": [1, 5]}]': finished 2024-09-10T19:24:45.717 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:24:45 smithi191 ceph-mon[32884]: from='mgr.14556 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.19", "id": [1, 6]}]': finished 2024-09-10T19:24:45.717 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:24:45 smithi191 ceph-mon[32884]: from='mgr.14556 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.1b", "id": [3, 0]}]': finished 2024-09-10T19:24:45.717 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:24:45 smithi191 ceph-mon[32884]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-10T19:24:45.717 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:24:45 smithi191 ceph-mon[32884]: osdmap e314: 8 total, 8 up, 7 in 2024-09-10T19:24:45.717 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:24:45 smithi191 ceph-mon[32884]: from='client.15762 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-10T19:24:45.717 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:24:45 smithi191 ceph-mon[32884]: osdmap e315: 8 total, 8 up, 7 in 2024-09-10T19:24:45.735 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:24:45.772 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:24:45.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:24:45.773 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:24:45.855 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: 25 16 41934 41918 104.784 93.125 0.0459252 0.00952756 2024-09-10T19:24:45.875 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:24:45.885 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:24:45 smithi183 ceph-mon[29736]: 4.1c scrub starts 2024-09-10T19:24:45.885 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:24:45 smithi183 ceph-mon[29736]: 4.1c scrub ok 2024-09-10T19:24:45.886 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:24:45 smithi183 ceph-mon[29736]: 2.19 scrub starts 2024-09-10T19:24:45.886 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:24:45 smithi183 ceph-mon[29736]: 2.19 scrub ok 2024-09-10T19:24:45.886 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:24:45 smithi183 ceph-mon[29736]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-10T19:24:45.886 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:24:45 smithi183 ceph-mon[29736]: Cluster is now healthy 2024-09-10T19:24:45.886 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:24:45 smithi183 ceph-mon[29736]: from='mgr.14556 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.b", "id": [1, 5]}]': finished 2024-09-10T19:24:45.886 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:24:45 smithi183 ceph-mon[29736]: from='mgr.14556 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.e", "id": [1, 5]}]': finished 2024-09-10T19:24:45.886 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:24:45 smithi183 ceph-mon[29736]: from='mgr.14556 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.19", "id": [1, 6]}]': finished 2024-09-10T19:24:45.886 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:24:45 smithi183 ceph-mon[29736]: from='mgr.14556 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.1b", "id": [3, 0]}]': finished 2024-09-10T19:24:45.886 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:24:45 smithi183 ceph-mon[29736]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-10T19:24:45.886 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:24:45 smithi183 ceph-mon[29736]: osdmap e314: 8 total, 8 up, 7 in 2024-09-10T19:24:45.886 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:24:45 smithi183 ceph-mon[29736]: from='client.15762 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-10T19:24:45.886 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:24:45 smithi183 ceph-mon[29736]: osdmap e315: 8 total, 8 up, 7 in 2024-09-10T19:24:45.887 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:24:45 smithi183 ceph-mon[41562]: 4.1c scrub starts 2024-09-10T19:24:45.887 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:24:45 smithi183 ceph-mon[41562]: 4.1c scrub ok 2024-09-10T19:24:45.887 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:24:45 smithi183 ceph-mon[41562]: 2.19 scrub starts 2024-09-10T19:24:45.887 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:24:45 smithi183 ceph-mon[41562]: 2.19 scrub ok 2024-09-10T19:24:45.887 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:24:45 smithi183 ceph-mon[41562]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-10T19:24:45.887 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:24:45 smithi183 ceph-mon[41562]: Cluster is now healthy 2024-09-10T19:24:45.887 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:24:45 smithi183 ceph-mon[41562]: from='mgr.14556 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.b", "id": [1, 5]}]': finished 2024-09-10T19:24:45.887 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:24:45 smithi183 ceph-mon[41562]: from='mgr.14556 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.e", "id": [1, 5]}]': finished 2024-09-10T19:24:45.887 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:24:45 smithi183 ceph-mon[41562]: from='mgr.14556 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.19", "id": [1, 6]}]': finished 2024-09-10T19:24:45.887 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:24:45 smithi183 ceph-mon[41562]: from='mgr.14556 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.1b", "id": [3, 0]}]': finished 2024-09-10T19:24:45.887 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:24:45 smithi183 ceph-mon[41562]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-10T19:24:45.887 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:24:45 smithi183 ceph-mon[41562]: osdmap e314: 8 total, 8 up, 7 in 2024-09-10T19:24:45.887 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:24:45 smithi183 ceph-mon[41562]: from='client.15762 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-10T19:24:45.888 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:24:45 smithi183 ceph-mon[41562]: osdmap e315: 8 total, 8 up, 7 in 2024-09-10T19:24:45.909 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:24:45.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:24:45.910 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:24:46.011 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:24:46.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:24:46.042 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:24:46.042 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:24:46.086 INFO:teuthology.orchestra.run.smithi183.stdout:{"pg_ready":true,"pg_map":{"version":664,"stamp":"2024-09-10T19:24:44.683740+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":2140126248,"num_objects":32842,"num_object_clones":0,"num_object_copies":98526,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32842,"num_whiteouts":0,"num_read":1214,"num_read_kb":985,"num_write":33383,"num_write_kb":2089619,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4976,"num_bytes_recovered":306822646,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":34377,"ondisk_log_size":34377,"up":474,"acting":474,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":482,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":656216064,"kb_used":14976944,"kb_used_data":9793268,"kb_used_omap":19,"kb_used_meta":5183596,"kb_avail":641239120,"statfs":{"total":671965249536,"available":656628858880,"internally_reserved":0,"allocated":10028306432,"data_stored":10025530300,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19457,"internal_metadata":5308003327},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[3,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":675217408,"num_objects":10303,"num_object_clones":0,"num_object_copies":30909,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10303,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":10303,"num_write_kb":659392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.820006"},"pg_stats":[{"pgid":"6.1b","version":"0'0","reported_seq":306,"reported_epoch":311,"state":"active+clean","last_fresh":"2024-09-10T19:24:34.877475+0000","last_change":"2024-09-10T19:23:56.530595+0000","last_active":"2024-09-10T19:24:34.877475+0000","last_peered":"2024-09-10T19:24:34.877475+0000","last_clean":"2024-09-10T19:24:34.877475+0000","last_became_active":"2024-09-10T19:10:14.056187+0000","last_became_peered":"2024-09-10T19:10:14.056187+0000","last_unstale":"2024-09-10T19:24:34.877475+0000","last_undegraded":"2024-09-10T19:24:34.877475+0000","last_fullsized":"2024-09-10T19:24:34.877475+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:23:56.530535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:23:56.530535+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:25:18.116907+0000","scrub_duration":0.0013991299999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":265,"reported_epoch":311,"state":"active+clean","last_fresh":"2024-09-10T19:24:34.880504+0000","last_change":"2024-09-10T19:24:23.893696+0000","last_active":"2024-09-10T19:24:34.880504+0000","last_peered":"2024-09-10T19:24:34.880504+0000","last_clean":"2024-09-10T19:24:34.880504+0000","last_became_active":"2024-09-10T19:24:23.892621+0000","last_became_peered":"2024-09-10T19:24:23.892621+0000","last_unstale":"2024-09-10T19:24:34.880504+0000","last_undegraded":"2024-09-10T19:24:34.880504+0000","last_fullsized":"2024-09-10T19:24:34.880504+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:20:05.748517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:16:38.194472+0000","last_clean_scrub_stamp":"2024-09-10T19:20:05.748517+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.023972094999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.19","version":"237'27","reported_seq":376,"reported_epoch":311,"state":"active+clean","last_fresh":"2024-09-10T19:24:34.877790+0000","last_change":"2024-09-10T19:24:23.907209+0000","last_active":"2024-09-10T19:24:34.877790+0000","last_peered":"2024-09-10T19:24:34.877790+0000","last_clean":"2024-09-10T19:24:34.877790+0000","last_became_active":"2024-09-10T19:24:23.906592+0000","last_became_peered":"2024-09-10T19:24:23.906592+0000","last_unstale":"2024-09-10T19:24:34.877790+0000","last_undegraded":"2024-09-10T19:24:34.877790+0000","last_fullsized":"2024-09-10T19:24:34.877790+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'15","last_scrub_stamp":"2024-09-10T19:19:21.244506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:19:21.244506+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.014405133000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":350,"reported_epoch":311,"state":"active+clean","last_fresh":"2024-09-10T19:24:34.879568+0000","last_change":"2024-09-10T19:24:01.977174+0000","last_active":"2024-09-10T19:24:34.879568+0000","last_peered":"2024-09-10T19:24:34.879568+0000","last_clean":"2024-09-10T19:24:34.879568+0000","last_became_active":"2024-09-10T19:24:01.976841+0000","last_became_peered":"2024-09-10T19:24:01.976841+0000","last_unstale":"2024-09-10T19:24:34.879568+0000","last_undegraded":"2024-09-10T19:24:34.879568+0000","last_fullsized":"2024-09-10T19:24:34.879568+0000","mapping_epoch":292,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":293,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:23:09.564335+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:19:28.498560+0000","last_clean_scrub_stamp":"2024-09-10T19:23:09.564335+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0051696729999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"59'1","reported_seq":327,"reported_epoch":311,"state":"active+clean","last_fresh":"2024-09-10T19:24:34.878224+0000","last_change":"2024-09-10T19:24:23.909007+0000","last_active":"2024-09-10T19:24:34.878224+0000","last_peered":"2024-09-10T19:24:34.878224+0000","last_clean":"2024-09-10T19:24:34.878224+0000","last_became_active":"2024-09-10T19:24:23.908614+0000","last_became_peered":"2024-09-10T19:24:23.908614+0000","last_unstale":"2024-09-10T19:24:34.878224+0000","last_undegraded":"2024-09-10T19:24:34.878224+0000","last_fullsized":"2024-09-10T19:24:34.878224+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'1","last_scrub_stamp":"2024-09-10T19:20:56.540653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:20:56.540653+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.010998005,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":874,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"237'15","reported_seq":384,"reported_epoch":311,"state":"active+clean","last_fresh":"2024-09-10T19:24:34.879641+0000","last_change":"2024-09-10T19:24:02.005253+0000","last_active":"2024-09-10T19:24:34.879641+0000","last_peered":"2024-09-10T19:24:34.879641+0000","last_clean":"2024-09-10T19:24:34.879641+0000","last_became_active":"2024-09-10T19:24:01.976339+0000","last_became_peered":"2024-09-10T19:24:01.976339+0000","last_unstale":"2024-09-10T19:24:34.879641+0000","last_undegraded":"2024-09-10T19:24:34.879641+0000","last_fullsized":"2024-09-10T19:24:34.879641+0000","mapping_epoch":292,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":293,"parent":"0.0","parent_split_bits":0,"last_scrub":"237'15","last_scrub_stamp":"2024-09-10T19:23:17.691323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:23:17.691323+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0046562740000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":30,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-09-10T19:24:39.878686+0000","last_change":"2024-09-10T19:24:01.974827+0000","last_active":"2024-09-10T19:24:39.878686+0000","last_peered":"2024-09-10T19:24:39.878686+0000","last_clean":"2024-09-10T19:24:39.878686+0000","last_became_active":"2024-09-10T19:24:01.973868+0000","last_became_peered":"2024-09-10T19:24:01.973868+0000","last_unstale":"2024-09-10T19:24:39.878686+0000","last_undegraded":"2024-09-10T19:24:39.878686+0000","last_fullsized":"2024-09-10T19:24:39.878686+0000","mapping_epoch":292,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":293,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:23:23.816921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:23:23.816921+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:24:49.725812+0000","scrub_duration":0.0075702879999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":338,"reported_epoch":311,"state":"active+clean","last_fresh":"2024-09-10T19:24:34.879258+0000","last_change":"2024-09-10T19:24:01.972221+0000","last_active":"2024-09-10T19:24:34.879258+0000","last_peered":"2024-09-10T19:24:34.879258+0000","last_clean":"2024-09-10T19:24:34.879258+0000","last_became_active":"2024-09-10T19:24:01.971921+0000","last_became_peered":"2024-09-10T19:24:01.971921+0000","last_unstale":"2024-09-10T19:24:34.879258+0000","last_undegraded":"2024-09-10T19:24:34.879258+0000","last_fullsized":"2024-09-10T19:24:34.879258+0000","mapping_epoch":292,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":293,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:23:25.762778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:23:25.762778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001257326,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":319,"reported_epoch":311,"state":"active+clean","last_fresh":"2024-09-10T19:24:34.881118+0000","last_change":"2024-09-10T19:24:23.902544+0000","last_active":"2024-09-10T19:24:34.881118+0000","last_peered":"2024-09-10T19:24:34.881118+0000","last_clean":"2024-09-10T19:24:34.881118+0000","last_became_active":"2024-09-10T19:24:23.902270+0000","last_became_peered":"2024-09-10T19:24:23.902270+0000","last_unstale":"2024-09-10T19:24:34.881118+0000","last_undegraded":"2024-09-10T19:24:34.881118+0000","last_fullsized":"2024-09-10T19:24:34.881118+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:20:45.770926+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:20:45.770926+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0040372419999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1b","version":"237'7","reported_seq":380,"reported_epoch":311,"state":"active+clean","last_fresh":"2024-09-10T19:24:34.878775+0000","last_change":"2024-09-10T19:24:01.991414+0000","last_active":"2024-09-10T19:24:34.878775+0000","last_peered":"2024-09-10T19:24:34.878775+0000","last_clean":"2024-09-10T19:24:34.878775+0000","last_became_active":"2024-09-10T19:24:01.974813+0000","last_became_peered":"2024-09-10T19:24:01.974813+0000","last_unstale":"2024-09-10T19:24:34.878775+0000","last_undegraded":"2024-09-10T19:24:34.878775+0000","last_fullsized":"2024-09-10T19:24:34.878775+0000","mapping_epoch":292,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":293,"parent":"0.0","parent_split_bits":0,"last_scrub":"237'7","last_scrub_stamp":"2024-09-10T19:23:57.471417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:23:57.471417+0000","objects_scrubbed":2,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:25:14.500050+0000","scrub_duration":0.001783548,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":10,"num_write":8,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":332,"reported_epoch":311,"state":"active+clean","last_fresh":"2024-09-10T19:24:34.879133+0000","last_change":"2024-09-10T19:24:08.570505+0000","last_active":"2024-09-10T19:24:34.879133+0000","last_peered":"2024-09-10T19:24:34.879133+0000","last_clean":"2024-09-10T19:24:34.879133+0000","last_became_active":"2024-09-10T19:24:01.976587+0000","last_became_peered":"2024-09-10T19:24:01.976587+0000","last_unstale":"2024-09-10T19:24:34.879133+0000","last_undegraded":"2024-09-10T19:24:34.879133+0000","last_fullsized":"2024-09-10T19:24:34.879133+0000","mapping_epoch":292,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":293,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:24:08.570444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:24:08.570444+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:25:21.550103+0000","scrub_duration":0.001488262,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":340,"reported_epoch":311,"state":"active+clean","last_fresh":"2024-09-10T19:24:34.877617+0000","last_change":"2024-09-10T19:24:07.659325+0000","last_active":"2024-09-10T19:24:34.877617+0000","last_peered":"2024-09-10T19:24:34.877617+0000","last_clean":"2024-09-10T19:24:34.877617+0000","last_became_active":"2024-09-10T19:24:01.970118+0000","last_became_peered":"2024-09-10T19:24:01.970118+0000","last_unstale":"2024-09-10T19:24:34.877617+0000","last_undegraded":"2024-09-10T19:24:34.877617+0000","last_fullsized":"2024-09-10T19:24:34.877617+0000","mapping_epoch":292,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":293,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:24:07.659224+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:24:07.659224+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:25:31.910243+0000","scrub_duration":0.001912307,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":322,"reported_epoch":311,"state":"active+clean","last_fresh":"2024-09-10T19:24:34.883000+0000","last_change":"2024-09-10T19:22:38.275940+0000","last_active":"2024-09-10T19:24:34.883000+0000","last_peered":"2024-09-10T19:24:34.883000+0000","last_clean":"2024-09-10T19:24:34.883000+0000","last_became_active":"2024-09-10T19:16:24.858800+0000","last_became_peered":"2024-09-10T19:16:24.858800+0000","last_unstale":"2024-09-10T19:24:34.883000+0000","last_undegraded":"2024-09-10T19:24:34.883000+0000","last_fullsized":"2024-09-10T19:24:34.883000+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:22:38.275863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:22:38.275863+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.010525915,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1a","version":"237'15","reported_seq":344,"reported_epoch":311,"state":"active+clean","last_fresh":"2024-09-10T19:24:34.879519+0000","last_change":"2024-09-10T19:24:23.891561+0000","last_active":"2024-09-10T19:24:34.879519+0000","last_peered":"2024-09-10T19:24:34.879519+0000","last_clean":"2024-09-10T19:24:34.879519+0000","last_became_active":"2024-09-10T19:24:23.891213+0000","last_became_peered":"2024-09-10T19:24:23.891213+0000","last_unstale":"2024-09-10T19:24:34.879519+0000","last_undegraded":"2024-09-10T19:24:34.879519+0000","last_fullsized":"2024-09-10T19:24:34.879519+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"237'15","last_scrub_stamp":"2024-09-10T19:21:42.567473+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:21:42.567473+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0065782230000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":315,"reported_epoch":311,"state":"active+clean","last_fresh":"2024-09-10T19:24:34.876202+0000","last_change":"2024-09-10T19:24:23.908699+0000","last_active":"2024-09-10T19:24:34.876202+0000","last_peered":"2024-09-10T19:24:34.876202+0000","last_clean":"2024-09-10T19:24:34.876202+0000","last_became_active":"2024-09-10T19:24:23.907653+0000","last_became_peered":"2024-09-10T19:24:23.907653+0000","last_unstale":"2024-09-10T19:24:34.876202+0000","last_undegraded":"2024-09-10T19:24:34.876202+0000","last_fullsized":"2024-09-10T19:24:34.876202+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:21:00.322381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:21:00.322381+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.008205258,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":288,"reported_epoch":311,"state":"active+clean","last_fresh":"2024-09-10T19:24:34.879963+0000","last_change":"2024-09-10T19:24:24.908498+0000","last_active":"2024-09-10T19:24:34.879963+0000","last_peered":"2024-09-10T19:24:34.879963+0000","last_clean":"2024-09-10T19:24:34.879963+0000","last_became_active":"2024-09-10T19:24:24.907431+0000","last_became_peered":"2024-09-10T19:24:24.907431+0000","last_unstale":"2024-09-10T19:24:34.879963+0000","last_undegraded":"2024-09-10T19:24:34.879963+0000","last_fullsized":"2024-09-10T19:24:34.879963+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:21:05.690833+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:21:05.690833+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0032019980000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.3","version":"312'2002","reported_seq":2027,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-09-10T19:24:40.343664+0000","last_change":"2024-09-10T19:24:09.045117+0000","last_active":"2024-09-10T19:24:40.343664+0000","last_peered":"2024-09-10T19:24:40.343664+0000","last_clean":"2024-09-10T19:24:40.343664+0000","last_became_active":"2024-09-10T19:24:09.043872+0000","last_became_peered":"2024-09-10T19:24:09.043872+0000","last_unstale":"2024-09-10T19:24:40.343664+0000","last_undegraded":"2024-09-10T19:24:40.343664+0000","last_fullsized":"2024-09-10T19:24:40.343664+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":295,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:24:08.023080+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:24:08.023080+0000","last_clean_scrub_stamp":"2024-09-10T19:24:08.023080+0000","objects_scrubbed":0,"log_size":2002,"log_dups_size":0,"ondisk_log_size":2002,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:25:28.521483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":131203072,"num_objects":2002,"num_object_clones":0,"num_object_copies":6006,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2002,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2002,"num_write_kb":128128,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.8","version":"74'79","reported_seq":92,"reported_epoch":311,"state":"active+clean","last_fresh":"2024-09-10T19:24:34.877110+0000","last_change":"2024-09-10T19:21:54.355335+0000","last_active":"2024-09-10T19:24:34.877110+0000","last_peered":"2024-09-10T19:24:34.877110+0000","last_clean":"2024-09-10T19:24:34.877110+0000","last_became_active":"2024-09-10T19:20:30.839050+0000","last_became_peered":"2024-09-10T19:20:30.839050+0000","last_unstale":"2024-09-10T19:24:34.877110+0000","last_undegraded":"2024-09-10T19:24:34.877110+0000","last_fullsized":"2024-09-10T19:24:34.877110+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":4,"last_scrub":"74'79","last_scrub_stamp":"2024-09-10T19:21:54.355151+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:08:00.699315+0000","last_clean_scrub_stamp":"2024-09-10T19:21:54.355151+0000","objects_scrubbed":1,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.014582322999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.d","version":"237'31","reported_seq":456,"reported_epoch":311,"state":"active+clean","last_fresh":"2024-09-10T19:24:34.879181+0000","last_change":"2024-09-10T19:24:03.465731+0000","last_active":"2024-09-10T19:24:34.879181+0000","last_peered":"2024-09-10T19:24:34.879181+0000","last_clean":"2024-09-10T19:24:34.879181+0000","last_became_active":"2024-09-10T19:24:01.975691+0000","last_became_peered":"2024-09-10T19:24:01.975691+0000","last_unstale":"2024-09-10T19:24:34.879181+0000","last_undegraded":"2024-09-10T19:24:34.879181+0000","last_fullsized":"2024-09-10T19:24:34.879181+0000","mapping_epoch":292,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":293,"parent":"0.0","parent_split_bits":0,"last_scrub":"237'31","last_scrub_stamp":"2024-09-10T19:24:03.465667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:24:03.465667+0000","objects_scrubbed":8,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:25:19.543452+0000","scrub_duration":0.0019330739999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":50,"num_read_kb":40,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,6],"acting":[4,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":322,"reported_epoch":311,"state":"active+clean","last_fresh":"2024-09-10T19:24:34.879849+0000","last_change":"2024-09-10T19:23:56.564075+0000","last_active":"2024-09-10T19:24:34.879849+0000","last_peered":"2024-09-10T19:24:34.879849+0000","last_clean":"2024-09-10T19:24:34.879849+0000","last_became_active":"2024-09-10T19:10:07.691119+0000","last_became_peered":"2024-09-10T19:10:07.691119+0000","last_unstale":"2024-09-10T19:24:34.879849+0000","last_undegraded":"2024-09-10T19:24:34.879849+0000","last_fullsized":"2024-09-10T19:24:34.879849+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:23:56.564014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:16:39.716674+0000","last_clean_scrub_stamp":"2024-09-10T19:23:56.564014+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:25:12.005393+0000","scrub_duration":0.001453333,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":44,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-09-10T19:24:39.878750+0000","last_change":"2024-09-10T19:24:24.886811+0000","last_active":"2024-09-10T19:24:39.878750+0000","last_peered":"2024-09-10T19:24:39.878750+0000","last_clean":"2024-09-10T19:24:39.878750+0000","last_became_active":"2024-09-10T19:24:24.886479+0000","last_became_peered":"2024-09-10T19:24:24.886479+0000","last_unstale":"2024-09-10T19:24:39.878750+0000","last_undegraded":"2024-09-10T19:24:39.878750+0000","last_fullsized":"2024-09-10T19:24:39.878750+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:20:54.418649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:20:54.418649+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.006664377,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":190,"reported_epoch":311,"state":"active+clean","last_fresh":"2024-09-10T19:24:34.878101+0000","last_change":"2024-09-10T19:23:23.287193+0000","last_active":"2024-09-10T19:24:34.878101+0000","last_peered":"2024-09-10T19:24:34.878101+0000","last_clean":"2024-09-10T19:24:34.878101+0000","last_became_active":"2024-09-10T19:16:24.857843+0000","last_became_peered":"2024-09-10T19:16:24.857843+0000","last_unstale":"2024-09-10T19:24:34.878101+0000","last_undegraded":"2024-09-10T19:24:34.878101+0000","last_fullsized":"2024-09-10T19:24:34.878101+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:23:23.287110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:23:23.287110+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:24:50.549002+0000","scrub_duration":0.0049096449999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.2","version":"313'2278","reported_seq":3376,"reported_epoch":313,"state":"active+clean","last_fresh":"2024-09-10T19:24:42.194617+0000","last_change":"2024-09-10T19:24:27.944339+0000","last_active":"2024-09-10T19:24:42.194617+0000","last_peered":"2024-09-10T19:24:42.194617+0000","last_clean":"2024-09-10T19:24:42.194617+0000","last_became_active":"2024-09-10T19:24:27.932777+0000","last_became_peered":"2024-09-10T19:24:27.932777+0000","last_unstale":"2024-09-10T19:24:42.194617+0000","last_undegraded":"2024-09-10T19:24:42.194617+0000","last_fullsized":"2024-09-10T19:24:42.194617+0000","mapping_epoch":305,"log_start":"0'0","ondisk_log_start":"0'0","created":295,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:24:08.023080+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:24:08.023080+0000","last_clean_scrub_stamp":"2024-09-10T19:24:08.023080+0000","objects_scrubbed":0,"log_size":2278,"log_dups_size":0,"ondisk_log_size":2278,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:25:27.464893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":149291008,"num_objects":2278,"num_object_clones":0,"num_object_copies":6834,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2278,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2278,"num_write_kb":145792,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":526,"num_bytes_recovered":34471936,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"74'79","reported_seq":377,"reported_epoch":311,"state":"active+clean","last_fresh":"2024-09-10T19:24:34.880356+0000","last_change":"2024-09-10T19:22:37.284558+0000","last_active":"2024-09-10T19:24:34.880356+0000","last_peered":"2024-09-10T19:24:34.880356+0000","last_clean":"2024-09-10T19:24:34.880356+0000","last_became_active":"2024-09-10T19:20:30.844108+0000","last_became_peered":"2024-09-10T19:20:30.844108+0000","last_unstale":"2024-09-10T19:24:34.880356+0000","last_undegraded":"2024-09-10T19:24:34.880356+0000","last_fullsized":"2024-09-10T19:24:34.880356+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":4,"last_scrub":"74'79","last_scrub_stamp":"2024-09-10T19:22:37.284479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:08:00.699315+0000","last_clean_scrub_stamp":"2024-09-10T19:22:37.284479+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.014548726999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.c","version":"237'20","reported_seq":352,"reported_epoch":311,"state":"active+clean","last_fresh":"2024-09-10T19:24:34.881039+0000","last_change":"2024-09-10T19:23:12.640206+0000","last_active":"2024-09-10T19:24:34.881039+0000","last_peered":"2024-09-10T19:24:34.881039+0000","last_clean":"2024-09-10T19:24:34.881039+0000","last_became_active":"2024-09-10T19:10:09.688321+0000","last_became_peered":"2024-09-10T19:10:09.688321+0000","last_unstale":"2024-09-10T19:24:34.881039+0000","last_undegraded":"2024-09-10T19:24:34.881039+0000","last_fullsized":"2024-09-10T19:24:34.881039+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"237'20","last_scrub_stamp":"2024-09-10T19:23:12.640129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:23:12.640129+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0046113889999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":30,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-09-10T19:24:39.878863+0000","last_change":"2024-09-10T19:24:01.974837+0000","last_active":"2024-09-10T19:24:39.878863+0000","last_peered":"2024-09-10T19:24:39.878863+0000","last_clean":"2024-09-10T19:24:39.878863+0000","last_became_active":"2024-09-10T19:24:01.973848+0000","last_became_peered":"2024-09-10T19:24:01.973848+0000","last_unstale":"2024-09-10T19:24:39.878863+0000","last_undegraded":"2024-09-10T19:24:39.878863+0000","last_fullsized":"2024-09-10T19:24:39.878863+0000","mapping_epoch":292,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":293,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:23:08.392000+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:23:08.392000+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0055299800000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.d","version":"303'12","reported_seq":378,"reported_epoch":311,"state":"active+clean","last_fresh":"2024-09-10T19:24:36.790833+0000","last_change":"2024-09-10T19:24:36.790767+0000","last_active":"2024-09-10T19:24:36.790833+0000","last_peered":"2024-09-10T19:24:36.790833+0000","last_clean":"2024-09-10T19:24:36.790833+0000","last_became_active":"2024-09-10T19:24:23.906819+0000","last_became_peered":"2024-09-10T19:24:23.906819+0000","last_unstale":"2024-09-10T19:24:36.790833+0000","last_undegraded":"2024-09-10T19:24:36.790833+0000","last_fullsized":"2024-09-10T19:24:36.790833+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"303'12","last_scrub_stamp":"2024-09-10T19:24:36.790665+0000","last_deep_scrub":"80'9","last_deep_scrub_stamp":"2024-09-10T19:11:25.156465+0000","last_clean_scrub_stamp":"2024-09-10T19:24:36.790665+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:26:06.003849+0000","scrub_duration":0.0028602430000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,0],"acting":[7,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":348,"reported_epoch":311,"state":"active+clean","last_fresh":"2024-09-10T19:24:34.880603+0000","last_change":"2024-09-10T19:24:01.972939+0000","last_active":"2024-09-10T19:24:34.880603+0000","last_peered":"2024-09-10T19:24:34.880603+0000","last_clean":"2024-09-10T19:24:34.880603+0000","last_became_active":"2024-09-10T19:24:01.972593+0000","last_became_peered":"2024-09-10T19:24:01.972593+0000","last_unstale":"2024-09-10T19:24:34.880603+0000","last_undegraded":"2024-09-10T19:24:34.880603+0000","last_fullsized":"2024-09-10T19:24:34.880603+0000","mapping_epoch":292,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":293,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:23:53.486675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:23:53.486675+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:25:01.795593+0000","scrub_duration":0.001538228,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,5],"acting":[4,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.1","version":"312'2026","reported_seq":2961,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-09-10T19:24:39.974797+0000","last_change":"2024-09-10T19:24:29.878172+0000","last_active":"2024-09-10T19:24:39.974797+0000","last_peered":"2024-09-10T19:24:39.974797+0000","last_clean":"2024-09-10T19:24:39.974797+0000","last_became_active":"2024-09-10T19:24:29.875287+0000","last_became_peered":"2024-09-10T19:24:29.875287+0000","last_unstale":"2024-09-10T19:24:39.974797+0000","last_undegraded":"2024-09-10T19:24:39.974797+0000","last_fullsized":"2024-09-10T19:24:39.974797+0000","mapping_epoch":307,"log_start":"0'0","ondisk_log_start":"0'0","created":295,"last_epoch_clean":308,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:24:08.023080+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:24:08.023080+0000","last_clean_scrub_stamp":"2024-09-10T19:24:08.023080+0000","objects_scrubbed":0,"log_size":2026,"log_dups_size":0,"ondisk_log_size":2026,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:25:31.460917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":132775936,"num_objects":2026,"num_object_clones":0,"num_object_copies":6078,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2026,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2026,"num_write_kb":129664,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":446,"num_bytes_recovered":29229056,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.a","version":"74'79","reported_seq":87,"reported_epoch":311,"state":"active+clean","last_fresh":"2024-09-10T19:24:34.874358+0000","last_change":"2024-09-10T19:21:53.326522+0000","last_active":"2024-09-10T19:24:34.874358+0000","last_peered":"2024-09-10T19:24:34.874358+0000","last_clean":"2024-09-10T19:24:34.874358+0000","last_became_active":"2024-09-10T19:20:30.844773+0000","last_became_peered":"2024-09-10T19:20:30.844773+0000","last_unstale":"2024-09-10T19:24:34.874358+0000","last_undegraded":"2024-09-10T19:24:34.874358+0000","last_fullsized":"2024-09-10T19:24:34.874358+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":4,"last_scrub":"74'79","last_scrub_stamp":"2024-09-10T19:21:53.326433+0000","last_deep_scrub":"74'79","last_deep_scrub_stamp":"2024-09-10T19:21:53.326433+0000","last_clean_scrub_stamp":"2024-09-10T19:21:53.326433+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0062202960000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,4],"acting":[5,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.f","version":"237'27","reported_seq":46,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-09-10T19:24:39.880169+0000","last_change":"2024-09-10T19:24:01.990847+0000","last_active":"2024-09-10T19:24:39.880169+0000","last_peered":"2024-09-10T19:24:39.880169+0000","last_clean":"2024-09-10T19:24:39.880169+0000","last_became_active":"2024-09-10T19:24:01.979187+0000","last_became_peered":"2024-09-10T19:24:01.979187+0000","last_unstale":"2024-09-10T19:24:39.880169+0000","last_undegraded":"2024-09-10T19:24:39.880169+0000","last_fullsized":"2024-09-10T19:24:39.880169+0000","mapping_epoch":292,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":293,"parent":"0.0","parent_split_bits":0,"last_scrub":"237'27","last_scrub_stamp":"2024-09-10T19:23:24.339960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:23:24.339960+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:24:46.515227+0000","scrub_duration":0.0077172969999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":30,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-09-10T19:24:39.878865+0000","last_change":"2024-09-10T19:24:01.975072+0000","last_active":"2024-09-10T19:24:39.878865+0000","last_peered":"2024-09-10T19:24:39.878865+0000","last_clean":"2024-09-10T19:24:39.878865+0000","last_became_active":"2024-09-10T19:24:01.974773+0000","last_became_peered":"2024-09-10T19:24:01.974773+0000","last_unstale":"2024-09-10T19:24:39.878865+0000","last_undegraded":"2024-09-10T19:24:39.878865+0000","last_fullsized":"2024-09-10T19:24:39.878865+0000","mapping_epoch":292,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":293,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:23:09.202458+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:23:09.202458+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0062080679999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.e","version":"303'10","reported_seq":491,"reported_epoch":311,"state":"active+clean","last_fresh":"2024-09-10T19:24:35.906896+0000","last_change":"2024-09-10T19:24:23.909143+0000","last_active":"2024-09-10T19:24:35.906896+0000","last_peered":"2024-09-10T19:24:35.906896+0000","last_clean":"2024-09-10T19:24:35.906896+0000","last_became_active":"2024-09-10T19:24:23.908760+0000","last_became_peered":"2024-09-10T19:24:23.908760+0000","last_unstale":"2024-09-10T19:24:35.906896+0000","last_undegraded":"2024-09-10T19:24:35.906896+0000","last_fullsized":"2024-09-10T19:24:35.906896+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'8","last_scrub_stamp":"2024-09-10T19:19:17.519517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:19:17.519517+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0049564470000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":354,"reported_epoch":311,"state":"active+clean","last_fresh":"2024-09-10T19:24:34.875363+0000","last_change":"2024-09-10T19:24:24.917431+0000","last_active":"2024-09-10T19:24:34.875363+0000","last_peered":"2024-09-10T19:24:34.875363+0000","last_clean":"2024-09-10T19:24:34.875363+0000","last_became_active":"2024-09-10T19:24:24.917109+0000","last_became_peered":"2024-09-10T19:24:24.917109+0000","last_unstale":"2024-09-10T19:24:34.875363+0000","last_undegraded":"2024-09-10T19:24:34.875363+0000","last_fullsized":"2024-09-10T19:24:34.875363+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:24:08.626072+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:24:08.626072+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:25:23.263723+0000","scrub_duration":0.001608906,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.0","version":"312'1941","reported_seq":1965,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-09-10T19:24:40.021859+0000","last_change":"2024-09-10T19:24:09.047016+0000","last_active":"2024-09-10T19:24:40.021859+0000","last_peered":"2024-09-10T19:24:40.021859+0000","last_clean":"2024-09-10T19:24:40.021859+0000","last_became_active":"2024-09-10T19:24:09.046656+0000","last_became_peered":"2024-09-10T19:24:09.046656+0000","last_unstale":"2024-09-10T19:24:40.021859+0000","last_undegraded":"2024-09-10T19:24:40.021859+0000","last_fullsized":"2024-09-10T19:24:40.021859+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":295,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:24:08.023080+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:24:08.023080+0000","last_clean_scrub_stamp":"2024-09-10T19:24:08.023080+0000","objects_scrubbed":0,"log_size":1941,"log_dups_size":0,"ondisk_log_size":1941,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:25:14.897639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":127205376,"num_objects":1941,"num_object_clones":0,"num_object_copies":5823,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1941,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1941,"num_write_kb":124224,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.e","version":"237'24","reported_seq":399,"reported_epoch":311,"state":"active+clean","last_fresh":"2024-09-10T19:24:34.880830+0000","last_change":"2024-09-10T19:24:02.003262+0000","last_active":"2024-09-10T19:24:34.880830+0000","last_peered":"2024-09-10T19:24:34.880830+0000","last_clean":"2024-09-10T19:24:34.880830+0000","last_became_active":"2024-09-10T19:24:01.976149+0000","last_became_peered":"2024-09-10T19:24:01.976149+0000","last_unstale":"2024-09-10T19:24:34.880830+0000","last_undegraded":"2024-09-10T19:24:34.880830+0000","last_fullsized":"2024-09-10T19:24:34.880830+0000","mapping_epoch":292,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":293,"parent":"0.0","parent_split_bits":0,"last_scrub":"237'24","last_scrub_stamp":"2024-09-10T19:23:11.613027+0000","last_deep_scrub":"84'16","last_deep_scrub_stamp":"2024-09-10T19:11:56.157975+0000","last_clean_scrub_stamp":"2024-09-10T19:23:11.613027+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.006332631,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":350,"reported_epoch":311,"state":"active+clean","last_fresh":"2024-09-10T19:24:34.877359+0000","last_change":"2024-09-10T19:24:24.887918+0000","last_active":"2024-09-10T19:24:34.877359+0000","last_peered":"2024-09-10T19:24:34.877359+0000","last_clean":"2024-09-10T19:24:34.877359+0000","last_became_active":"2024-09-10T19:24:24.887602+0000","last_became_peered":"2024-09-10T19:24:24.887602+0000","last_unstale":"2024-09-10T19:24:34.877359+0000","last_undegraded":"2024-09-10T19:24:34.877359+0000","last_fullsized":"2024-09-10T19:24:34.877359+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:24:13.636249+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:24:13.636249+0000","last_clean_scrub_stamp":"2024-09-10T19:24:13.636249+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:25:25.549624+0000","scrub_duration":0.001732363,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":298,"reported_epoch":311,"state":"active+clean","last_fresh":"2024-09-10T19:24:34.876287+0000","last_change":"2024-09-10T19:22:19.713111+0000","last_active":"2024-09-10T19:24:34.876287+0000","last_peered":"2024-09-10T19:24:34.876287+0000","last_clean":"2024-09-10T19:24:34.876287+0000","last_became_active":"2024-09-10T19:10:11.692215+0000","last_became_peered":"2024-09-10T19:10:11.692215+0000","last_unstale":"2024-09-10T19:24:34.876287+0000","last_undegraded":"2024-09-10T19:24:34.876287+0000","last_fullsized":"2024-09-10T19:24:34.876287+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:22:19.713013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:22:19.713013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.029020666000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":296,"reported_epoch":311,"state":"active+clean","last_fresh":"2024-09-10T19:24:34.877803+0000","last_change":"2024-09-10T19:22:37.144678+0000","last_active":"2024-09-10T19:24:34.877803+0000","last_peered":"2024-09-10T19:24:34.877803+0000","last_clean":"2024-09-10T19:24:34.877803+0000","last_became_active":"2024-09-10T19:10:14.053736+0000","last_became_peered":"2024-09-10T19:10:14.053736+0000","last_unstale":"2024-09-10T19:24:34.877803+0000","last_undegraded":"2024-09-10T19:24:34.877803+0000","last_fullsized":"2024-09-10T19:24:34.877803+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:22:37.144606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:22:37.144606+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.002180853,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.f","version":"312'1967","reported_seq":1992,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-09-10T19:24:40.018459+0000","last_change":"2024-09-10T19:24:09.047508+0000","last_active":"2024-09-10T19:24:40.018459+0000","last_peered":"2024-09-10T19:24:40.018459+0000","last_clean":"2024-09-10T19:24:40.018459+0000","last_became_active":"2024-09-10T19:24:09.047213+0000","last_became_peered":"2024-09-10T19:24:09.047213+0000","last_unstale":"2024-09-10T19:24:40.018459+0000","last_undegraded":"2024-09-10T19:24:40.018459+0000","last_fullsized":"2024-09-10T19:24:40.018459+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":295,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:24:08.023080+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:24:08.023080+0000","last_clean_scrub_stamp":"2024-09-10T19:24:08.023080+0000","objects_scrubbed":0,"log_size":1967,"log_dups_size":0,"ondisk_log_size":1967,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:25:10.895945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":128909312,"num_objects":1967,"num_object_clones":0,"num_object_copies":5901,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1967,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1967,"num_write_kb":125888,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,4],"acting":[6,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.4","version":"74'79","reported_seq":92,"reported_epoch":311,"state":"active+clean","last_fresh":"2024-09-10T19:24:34.877168+0000","last_change":"2024-09-10T19:22:20.452660+0000","last_active":"2024-09-10T19:24:34.877168+0000","last_peered":"2024-09-10T19:24:34.877168+0000","last_clean":"2024-09-10T19:24:34.877168+0000","last_became_active":"2024-09-10T19:20:30.838441+0000","last_became_peered":"2024-09-10T19:20:30.838441+0000","last_unstale":"2024-09-10T19:24:34.877168+0000","last_undegraded":"2024-09-10T19:24:34.877168+0000","last_fullsized":"2024-09-10T19:24:34.877168+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":3,"last_scrub":"74'79","last_scrub_stamp":"2024-09-10T19:22:20.452581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:08:00.699315+0000","last_clean_scrub_stamp":"2024-09-10T19:22:20.452581+0000","objects_scrubbed":1,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.006813782,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"265'35","reported_seq":244,"reported_epoch":311,"state":"active+clean","last_fresh":"2024-09-10T19:24:34.880593+0000","last_change":"2024-09-10T19:23:24.734284+0000","last_active":"2024-09-10T19:24:34.880593+0000","last_peered":"2024-09-10T19:24:34.880593+0000","last_clean":"2024-09-10T19:24:34.880593+0000","last_became_active":"2024-09-10T19:16:24.852821+0000","last_became_peered":"2024-09-10T19:16:24.852821+0000","last_unstale":"2024-09-10T19:24:34.880593+0000","last_undegraded":"2024-09-10T19:24:34.880593+0000","last_fullsized":"2024-09-10T19:24:34.880593+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"265'35","last_scrub_stamp":"2024-09-10T19:23:24.734231+0000","last_deep_scrub":"265'35","last_deep_scrub_stamp":"2024-09-10T19:21:55.533362+0000","last_clean_scrub_stamp":"2024-09-10T19:23:24.734231+0000","objects_scrubbed":7,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0018403409999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":42,"num_read_kb":35,"num_write":28,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,7],"acting":[4,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":332,"reported_epoch":311,"state":"active+clean","last_fresh":"2024-09-10T19:24:34.878678+0000","last_change":"2024-09-10T19:23:26.468261+0000","last_active":"2024-09-10T19:24:34.878678+0000","last_peered":"2024-09-10T19:24:34.878678+0000","last_clean":"2024-09-10T19:24:34.878678+0000","last_became_active":"2024-09-10T19:16:24.855861+0000","last_became_peered":"2024-09-10T19:16:24.855861+0000","last_unstale":"2024-09-10T19:24:34.878678+0000","last_undegraded":"2024-09-10T19:24:34.878678+0000","last_fullsized":"2024-09-10T19:24:34.878678+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:23:26.468206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:23:26.468206+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001230769,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.0","version":"294'12","reported_seq":524,"reported_epoch":311,"state":"active+clean","last_fresh":"2024-09-10T19:24:35.905600+0000","last_change":"2024-09-10T19:24:03.465787+0000","last_active":"2024-09-10T19:24:35.905600+0000","last_peered":"2024-09-10T19:24:35.905600+0000","last_clean":"2024-09-10T19:24:35.905600+0000","last_became_active":"2024-09-10T19:24:01.972645+0000","last_became_peered":"2024-09-10T19:24:01.972645+0000","last_unstale":"2024-09-10T19:24:35.905600+0000","last_undegraded":"2024-09-10T19:24:35.905600+0000","last_fullsized":"2024-09-10T19:24:35.905600+0000","mapping_epoch":292,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":293,"parent":"0.0","parent_split_bits":0,"last_scrub":"264'11","last_scrub_stamp":"2024-09-10T19:24:03.465696+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:24:03.465696+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:25:29.907384+0000","scrub_duration":0.001484055,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":323,"reported_epoch":311,"state":"active+clean","last_fresh":"2024-09-10T19:24:34.879577+0000","last_change":"2024-09-10T19:24:02.643075+0000","last_active":"2024-09-10T19:24:34.879577+0000","last_peered":"2024-09-10T19:24:34.879577+0000","last_clean":"2024-09-10T19:24:34.879577+0000","last_became_active":"2024-09-10T19:24:01.974621+0000","last_became_peered":"2024-09-10T19:24:01.974621+0000","last_unstale":"2024-09-10T19:24:34.879577+0000","last_undegraded":"2024-09-10T19:24:34.879577+0000","last_fullsized":"2024-09-10T19:24:34.879577+0000","mapping_epoch":292,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":293,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:24:02.642987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:11:41.073833+0000","last_clean_scrub_stamp":"2024-09-10T19:24:02.642987+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:25:22.266695+0000","scrub_duration":0.0014517099999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.e","version":"312'1990","reported_seq":2015,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-09-10T19:24:39.968273+0000","last_change":"2024-09-10T19:24:09.046030+0000","last_active":"2024-09-10T19:24:39.968273+0000","last_peered":"2024-09-10T19:24:39.968273+0000","last_clean":"2024-09-10T19:24:39.968273+0000","last_became_active":"2024-09-10T19:24:09.045762+0000","last_became_peered":"2024-09-10T19:24:09.045762+0000","last_unstale":"2024-09-10T19:24:39.968273+0000","last_undegraded":"2024-09-10T19:24:39.968273+0000","last_fullsized":"2024-09-10T19:24:39.968273+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":295,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:24:08.023080+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:24:08.023080+0000","last_clean_scrub_stamp":"2024-09-10T19:24:08.023080+0000","objects_scrubbed":0,"log_size":1990,"log_dups_size":0,"ondisk_log_size":1990,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:25:25.918737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":130416640,"num_objects":1990,"num_object_clones":0,"num_object_copies":5970,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1990,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1990,"num_write_kb":127360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.5","version":"74'79","reported_seq":97,"reported_epoch":311,"state":"active+clean","last_fresh":"2024-09-10T19:24:34.878790+0000","last_change":"2024-09-10T19:23:13.621573+0000","last_active":"2024-09-10T19:24:34.878790+0000","last_peered":"2024-09-10T19:24:34.878790+0000","last_clean":"2024-09-10T19:24:34.878790+0000","last_became_active":"2024-09-10T19:20:30.838556+0000","last_became_peered":"2024-09-10T19:20:30.838556+0000","last_unstale":"2024-09-10T19:24:34.878790+0000","last_undegraded":"2024-09-10T19:24:34.878790+0000","last_fullsized":"2024-09-10T19:24:34.878790+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":3,"last_scrub":"74'79","last_scrub_stamp":"2024-09-10T19:23:13.621433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:08:00.699315+0000","last_clean_scrub_stamp":"2024-09-10T19:23:13.621433+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.012192343,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"237'30","reported_seq":367,"reported_epoch":311,"state":"active+clean","last_fresh":"2024-09-10T19:24:37.476320+0000","last_change":"2024-09-10T19:24:37.476248+0000","last_active":"2024-09-10T19:24:37.476320+0000","last_peered":"2024-09-10T19:24:37.476320+0000","last_clean":"2024-09-10T19:24:37.476320+0000","last_became_active":"2024-09-10T19:24:23.903784+0000","last_became_peered":"2024-09-10T19:24:23.903784+0000","last_unstale":"2024-09-10T19:24:37.476320+0000","last_undegraded":"2024-09-10T19:24:37.476320+0000","last_fullsized":"2024-09-10T19:24:37.476320+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"237'30","last_scrub_stamp":"2024-09-10T19:24:37.476174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:24:37.476174+0000","objects_scrubbed":8,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:26:03.990729+0000","scrub_duration":0.0032561880000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":52,"num_read_kb":40,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":30,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-09-10T19:24:39.879898+0000","last_change":"2024-09-10T19:24:01.979270+0000","last_active":"2024-09-10T19:24:39.879898+0000","last_peered":"2024-09-10T19:24:39.879898+0000","last_clean":"2024-09-10T19:24:39.879898+0000","last_became_active":"2024-09-10T19:24:01.978995+0000","last_became_peered":"2024-09-10T19:24:01.978995+0000","last_unstale":"2024-09-10T19:24:39.879898+0000","last_undegraded":"2024-09-10T19:24:39.879898+0000","last_fullsized":"2024-09-10T19:24:39.879898+0000","mapping_epoch":292,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":293,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:22:32.696724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:17:26.374376+0000","last_clean_scrub_stamp":"2024-09-10T19:22:32.696724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":317,"reported_epoch":311,"state":"active+clean","last_fresh":"2024-09-10T19:24:34.880795+0000","last_change":"2024-09-10T19:24:13.564497+0000","last_active":"2024-09-10T19:24:34.880795+0000","last_peered":"2024-09-10T19:24:34.880795+0000","last_clean":"2024-09-10T19:24:34.880795+0000","last_became_active":"2024-09-10T19:10:11.698054+0000","last_became_peered":"2024-09-10T19:10:11.698054+0000","last_unstale":"2024-09-10T19:24:34.880795+0000","last_undegraded":"2024-09-10T19:24:34.880795+0000","last_fullsized":"2024-09-10T19:24:34.880795+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:24:13.564429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:24:13.564429+0000","last_clean_scrub_stamp":"2024-09-10T19:24:13.564429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:25:18.364651+0000","scrub_duration":0.0014190920000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":325,"reported_epoch":311,"state":"active+clean","last_fresh":"2024-09-10T19:24:34.879446+0000","last_change":"2024-09-10T19:23:55.452716+0000","last_active":"2024-09-10T19:24:34.879446+0000","last_peered":"2024-09-10T19:24:34.879446+0000","last_clean":"2024-09-10T19:24:34.879446+0000","last_became_active":"2024-09-10T19:16:24.848109+0000","last_became_peered":"2024-09-10T19:16:24.848109+0000","last_unstale":"2024-09-10T19:24:34.879446+0000","last_undegraded":"2024-09-10T19:24:34.879446+0000","last_fullsized":"2024-09-10T19:24:34.879446+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:23:55.452654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:23:55.452654+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:25:25.076106+0000","scrub_duration":0.0018215709999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.d","version":"312'1878","reported_seq":2835,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-09-10T19:24:39.974264+0000","last_change":"2024-09-10T19:24:28.943745+0000","last_active":"2024-09-10T19:24:39.974264+0000","last_peered":"2024-09-10T19:24:39.974264+0000","last_clean":"2024-09-10T19:24:39.974264+0000","last_became_active":"2024-09-10T19:24:28.941711+0000","last_became_peered":"2024-09-10T19:24:28.941711+0000","last_unstale":"2024-09-10T19:24:39.974264+0000","last_undegraded":"2024-09-10T19:24:39.974264+0000","last_fullsized":"2024-09-10T19:24:39.974264+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":295,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:24:08.023080+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:24:08.023080+0000","last_clean_scrub_stamp":"2024-09-10T19:24:08.023080+0000","objects_scrubbed":0,"log_size":1878,"log_dups_size":0,"ondisk_log_size":1878,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:25:24.513557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":123076608,"num_objects":1878,"num_object_clones":0,"num_object_copies":5634,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1878,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1878,"num_write_kb":120192,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":456,"num_bytes_recovered":29884416,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.6","version":"74'79","reported_seq":184,"reported_epoch":311,"state":"active+clean","last_fresh":"2024-09-10T19:24:36.645208+0000","last_change":"2024-09-10T19:24:36.645141+0000","last_active":"2024-09-10T19:24:36.645208+0000","last_peered":"2024-09-10T19:24:36.645208+0000","last_clean":"2024-09-10T19:24:36.645208+0000","last_became_active":"2024-09-10T19:24:23.902290+0000","last_became_peered":"2024-09-10T19:24:23.902290+0000","last_unstale":"2024-09-10T19:24:36.645208+0000","last_undegraded":"2024-09-10T19:24:36.645208+0000","last_fullsized":"2024-09-10T19:24:36.645208+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":3,"last_scrub":"74'79","last_scrub_stamp":"2024-09-10T19:24:36.645049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:08:00.699315+0000","last_clean_scrub_stamp":"2024-09-10T19:24:36.645049+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:25:40.707623+0000","scrub_duration":0.0029320190000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"237'37","reported_seq":307,"reported_epoch":311,"state":"active+clean","last_fresh":"2024-09-10T19:24:34.880185+0000","last_change":"2024-09-10T19:24:23.902349+0000","last_active":"2024-09-10T19:24:34.880185+0000","last_peered":"2024-09-10T19:24:34.880185+0000","last_clean":"2024-09-10T19:24:34.880185+0000","last_became_active":"2024-09-10T19:24:23.901836+0000","last_became_peered":"2024-09-10T19:24:23.901836+0000","last_unstale":"2024-09-10T19:24:34.880185+0000","last_undegraded":"2024-09-10T19:24:34.880185+0000","last_fullsized":"2024-09-10T19:24:34.880185+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"104'27","last_scrub_stamp":"2024-09-10T19:20:14.754045+0000","last_deep_scrub":"104'27","last_deep_scrub_stamp":"2024-09-10T19:13:02.225944+0000","last_clean_scrub_stamp":"2024-09-10T19:20:14.754045+0000","objects_scrubbed":8,"log_size":37,"log_dups_size":0,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0059406959999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":53,"num_read_kb":39,"num_write":32,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":329,"reported_epoch":311,"state":"active+clean","last_fresh":"2024-09-10T19:24:35.803717+0000","last_change":"2024-09-10T19:24:35.803375+0000","last_active":"2024-09-10T19:24:35.803717+0000","last_peered":"2024-09-10T19:24:35.803717+0000","last_clean":"2024-09-10T19:24:35.803717+0000","last_became_active":"2024-09-10T19:24:23.899556+0000","last_became_peered":"2024-09-10T19:24:23.899556+0000","last_unstale":"2024-09-10T19:24:35.803717+0000","last_undegraded":"2024-09-10T19:24:35.803717+0000","last_fullsized":"2024-09-10T19:24:35.803717+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:24:35.803239+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:16:37.779967+0000","last_clean_scrub_stamp":"2024-09-10T19:24:35.803239+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:25:41.618836+0000","scrub_duration":0.011234631,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":328,"reported_epoch":311,"state":"active+clean","last_fresh":"2024-09-10T19:24:35.649342+0000","last_change":"2024-09-10T19:24:35.649237+0000","last_active":"2024-09-10T19:24:35.649342+0000","last_peered":"2024-09-10T19:24:35.649342+0000","last_clean":"2024-09-10T19:24:35.649342+0000","last_became_active":"2024-09-10T19:24:23.901937+0000","last_became_peered":"2024-09-10T19:24:23.901937+0000","last_unstale":"2024-09-10T19:24:35.649342+0000","last_undegraded":"2024-09-10T19:24:35.649342+0000","last_fullsized":"2024-09-10T19:24:35.649342+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:24:35.649162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:24:35.649162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:25:37.623155+0000","scrub_duration":0.005766831,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":30,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-09-10T19:24:39.878760+0000","last_change":"2024-09-10T19:24:01.975007+0000","last_active":"2024-09-10T19:24:39.878760+0000","last_peered":"2024-09-10T19:24:39.878760+0000","last_clean":"2024-09-10T19:24:39.878760+0000","last_became_active":"2024-09-10T19:24:01.974076+0000","last_became_peered":"2024-09-10T19:24:01.974076+0000","last_unstale":"2024-09-10T19:24:39.878760+0000","last_undegraded":"2024-09-10T19:24:39.878760+0000","last_fullsized":"2024-09-10T19:24:39.878760+0000","mapping_epoch":292,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":293,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:20:06.496950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:20:06.496950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0033688099999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,7],"acting":[1,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.c","version":"312'1971","reported_seq":3017,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-09-10T19:24:40.306886+0000","last_change":"2024-09-10T19:24:28.947516+0000","last_active":"2024-09-10T19:24:40.306886+0000","last_peered":"2024-09-10T19:24:40.306886+0000","last_clean":"2024-09-10T19:24:40.306886+0000","last_became_active":"2024-09-10T19:24:28.941128+0000","last_became_peered":"2024-09-10T19:24:28.941128+0000","last_unstale":"2024-09-10T19:24:40.306886+0000","last_undegraded":"2024-09-10T19:24:40.306886+0000","last_fullsized":"2024-09-10T19:24:40.306886+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":295,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:24:08.023080+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:24:08.023080+0000","last_clean_scrub_stamp":"2024-09-10T19:24:08.023080+0000","objects_scrubbed":0,"log_size":1971,"log_dups_size":0,"ondisk_log_size":1971,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:25:19.144151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":129171456,"num_objects":1971,"num_object_clones":0,"num_object_copies":5913,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1971,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1971,"num_write_kb":126144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":501,"num_bytes_recovered":32833536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.7","version":"74'79","reported_seq":177,"reported_epoch":311,"state":"active+clean","last_fresh":"2024-09-10T19:24:39.714613+0000","last_change":"2024-09-10T19:24:39.714196+0000","last_active":"2024-09-10T19:24:39.714613+0000","last_peered":"2024-09-10T19:24:39.714613+0000","last_clean":"2024-09-10T19:24:39.714613+0000","last_became_active":"2024-09-10T19:20:30.842857+0000","last_became_peered":"2024-09-10T19:20:30.842857+0000","last_unstale":"2024-09-10T19:24:39.714613+0000","last_undegraded":"2024-09-10T19:24:39.714613+0000","last_fullsized":"2024-09-10T19:24:39.714613+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":3,"last_scrub":"74'79","last_scrub_stamp":"2024-09-10T19:24:39.714098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:08:00.699315+0000","last_clean_scrub_stamp":"2024-09-10T19:24:39.714098+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:25:45.032074+0000","scrub_duration":0.0081359229999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.2","version":"237'20","reported_seq":52,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-09-10T19:24:39.879020+0000","last_change":"2024-09-10T19:24:02.309283+0000","last_active":"2024-09-10T19:24:39.879020+0000","last_peered":"2024-09-10T19:24:39.879020+0000","last_clean":"2024-09-10T19:24:39.879020+0000","last_became_active":"2024-09-10T19:24:01.976841+0000","last_became_peered":"2024-09-10T19:24:01.976841+0000","last_unstale":"2024-09-10T19:24:39.879020+0000","last_undegraded":"2024-09-10T19:24:39.879020+0000","last_fullsized":"2024-09-10T19:24:39.879020+0000","mapping_epoch":292,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":293,"parent":"0.0","parent_split_bits":0,"last_scrub":"237'20","last_scrub_stamp":"2024-09-10T19:24:02.309226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:24:02.309226+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:25:29.710630+0000","scrub_duration":0.0022310799999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":28,"num_read_kb":23,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":44,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-09-10T19:24:39.878837+0000","last_change":"2024-09-10T19:24:24.910911+0000","last_active":"2024-09-10T19:24:39.878837+0000","last_peered":"2024-09-10T19:24:39.878837+0000","last_clean":"2024-09-10T19:24:39.878837+0000","last_became_active":"2024-09-10T19:24:24.910173+0000","last_became_peered":"2024-09-10T19:24:24.910173+0000","last_unstale":"2024-09-10T19:24:39.878837+0000","last_undegraded":"2024-09-10T19:24:39.878837+0000","last_fullsized":"2024-09-10T19:24:39.878837+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:20:52.748172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:20:52.748172+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0061607199999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"303'11","reported_seq":409,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-09-10T19:24:39.879029+0000","last_change":"2024-09-10T19:24:23.905909+0000","last_active":"2024-09-10T19:24:39.879029+0000","last_peered":"2024-09-10T19:24:39.879029+0000","last_clean":"2024-09-10T19:24:39.879029+0000","last_became_active":"2024-09-10T19:24:23.905107+0000","last_became_peered":"2024-09-10T19:24:23.905107+0000","last_unstale":"2024-09-10T19:24:39.879029+0000","last_undegraded":"2024-09-10T19:24:39.879029+0000","last_fullsized":"2024-09-10T19:24:39.879029+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'9","last_scrub_stamp":"2024-09-10T19:21:04.724652+0000","last_deep_scrub":"84'9","last_deep_scrub_stamp":"2024-09-10T19:11:53.879272+0000","last_clean_scrub_stamp":"2024-09-10T19:21:04.724652+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":272,"reported_epoch":311,"state":"active+clean","last_fresh":"2024-09-10T19:24:34.882580+0000","last_change":"2024-09-10T19:24:23.906739+0000","last_active":"2024-09-10T19:24:34.882580+0000","last_peered":"2024-09-10T19:24:34.882580+0000","last_clean":"2024-09-10T19:24:34.882580+0000","last_became_active":"2024-09-10T19:24:23.905977+0000","last_became_peered":"2024-09-10T19:24:23.905977+0000","last_unstale":"2024-09-10T19:24:34.882580+0000","last_undegraded":"2024-09-10T19:24:34.882580+0000","last_fullsized":"2024-09-10T19:24:34.882580+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:21:03.766127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:21:03.766127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0086405590000000008,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,5],"acting":[0,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.a","version":"312'2016","reported_seq":2945,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-09-10T19:24:40.026935+0000","last_change":"2024-09-10T19:24:29.883357+0000","last_active":"2024-09-10T19:24:40.026935+0000","last_peered":"2024-09-10T19:24:40.026935+0000","last_clean":"2024-09-10T19:24:40.026935+0000","last_became_active":"2024-09-10T19:24:29.876670+0000","last_became_peered":"2024-09-10T19:24:29.876670+0000","last_unstale":"2024-09-10T19:24:40.026935+0000","last_undegraded":"2024-09-10T19:24:40.026935+0000","last_fullsized":"2024-09-10T19:24:40.026935+0000","mapping_epoch":307,"log_start":"0'0","ondisk_log_start":"0'0","created":295,"last_epoch_clean":308,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:24:08.023080+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:24:08.023080+0000","last_clean_scrub_stamp":"2024-09-10T19:24:08.023080+0000","objects_scrubbed":0,"log_size":2016,"log_dups_size":0,"ondisk_log_size":2016,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:25:11.881070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":132120576,"num_objects":2016,"num_object_clones":0,"num_object_copies":6048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2016,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2016,"num_write_kb":129024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":443,"num_bytes_recovered":29032448,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.1","version":"74'79","reported_seq":94,"reported_epoch":311,"state":"active+clean","last_fresh":"2024-09-10T19:24:34.877138+0000","last_change":"2024-09-10T19:24:23.908124+0000","last_active":"2024-09-10T19:24:34.877138+0000","last_peered":"2024-09-10T19:24:34.877138+0000","last_clean":"2024-09-10T19:24:34.877138+0000","last_became_active":"2024-09-10T19:24:23.907264+0000","last_became_peered":"2024-09-10T19:24:23.907264+0000","last_unstale":"2024-09-10T19:24:34.877138+0000","last_undegraded":"2024-09-10T19:24:34.877138+0000","last_fullsized":"2024-09-10T19:24:34.877138+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":4,"last_scrub":"74'79","last_scrub_stamp":"2024-09-10T19:19:11.070835+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:08:00.699315+0000","last_clean_scrub_stamp":"2024-09-10T19:19:11.070835+0000","objects_scrubbed":2,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0030997160000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":11,"num_read_kb":20,"num_write":11,"num_write_kb":148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":53670,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,4],"acting":[5,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"294'44","reported_seq":60,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-09-10T19:24:40.906440+0000","last_change":"2024-09-10T19:24:01.994430+0000","last_active":"2024-09-10T19:24:40.906440+0000","last_peered":"2024-09-10T19:24:40.906440+0000","last_clean":"2024-09-10T19:24:40.906440+0000","last_became_active":"2024-09-10T19:24:01.972831+0000","last_became_peered":"2024-09-10T19:24:01.972831+0000","last_unstale":"2024-09-10T19:24:40.906440+0000","last_undegraded":"2024-09-10T19:24:40.906440+0000","last_fullsized":"2024-09-10T19:24:40.906440+0000","mapping_epoch":292,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":293,"parent":"0.0","parent_split_bits":0,"last_scrub":"237'43","last_scrub_stamp":"2024-09-10T19:23:22.309850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:23:22.309850+0000","objects_scrubbed":10,"log_size":44,"log_dups_size":0,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.047452614999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":65,"num_read_kb":50,"num_write":38,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":716,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":326,"reported_epoch":311,"state":"active+clean","last_fresh":"2024-09-10T19:24:34.874528+0000","last_change":"2024-09-10T19:24:01.978606+0000","last_active":"2024-09-10T19:24:34.874528+0000","last_peered":"2024-09-10T19:24:34.874528+0000","last_clean":"2024-09-10T19:24:34.874528+0000","last_became_active":"2024-09-10T19:24:01.978187+0000","last_became_peered":"2024-09-10T19:24:01.978187+0000","last_unstale":"2024-09-10T19:24:34.874528+0000","last_undegraded":"2024-09-10T19:24:34.874528+0000","last_fullsized":"2024-09-10T19:24:34.874528+0000","mapping_epoch":292,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":293,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:20:44.342243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:20:44.342243+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0078427040000000007,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":272,"reported_epoch":311,"state":"active+clean","last_fresh":"2024-09-10T19:24:34.880703+0000","last_change":"2024-09-10T19:24:24.893028+0000","last_active":"2024-09-10T19:24:34.880703+0000","last_peered":"2024-09-10T19:24:34.880703+0000","last_clean":"2024-09-10T19:24:34.880703+0000","last_became_active":"2024-09-10T19:24:24.892692+0000","last_became_peered":"2024-09-10T19:24:24.892692+0000","last_unstale":"2024-09-10T19:24:34.880703+0000","last_undegraded":"2024-09-10T19:24:34.880703+0000","last_fullsized":"2024-09-10T19:24:34.880703+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:20:56.648680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:20:56.648680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001460161,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":296,"reported_epoch":311,"state":"active+clean","last_fresh":"2024-09-10T19:24:34.877951+0000","last_change":"2024-09-10T19:23:55.501216+0000","last_active":"2024-09-10T19:24:34.877951+0000","last_peered":"2024-09-10T19:24:34.877951+0000","last_clean":"2024-09-10T19:24:34.877951+0000","last_became_active":"2024-09-10T19:10:13.721744+0000","last_became_peered":"2024-09-10T19:10:13.721744+0000","last_unstale":"2024-09-10T19:24:34.877951+0000","last_undegraded":"2024-09-10T19:24:34.877951+0000","last_fullsized":"2024-09-10T19:24:34.877951+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:23:55.501166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:23:55.501166+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:24:57.562916+0000","scrub_duration":0.0012138850000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.9","version":"312'2023","reported_seq":2047,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-09-10T19:24:40.369688+0000","last_change":"2024-09-10T19:24:09.043291+0000","last_active":"2024-09-10T19:24:40.369688+0000","last_peered":"2024-09-10T19:24:40.369688+0000","last_clean":"2024-09-10T19:24:40.369688+0000","last_became_active":"2024-09-10T19:24:09.042606+0000","last_became_peered":"2024-09-10T19:24:09.042606+0000","last_unstale":"2024-09-10T19:24:40.369688+0000","last_undegraded":"2024-09-10T19:24:40.369688+0000","last_fullsized":"2024-09-10T19:24:40.369688+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":295,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:24:08.023080+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:24:08.023080+0000","last_clean_scrub_stamp":"2024-09-10T19:24:08.023080+0000","objects_scrubbed":0,"log_size":2023,"log_dups_size":0,"ondisk_log_size":2023,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:25:23.620732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":132579328,"num_objects":2023,"num_object_clones":0,"num_object_copies":6069,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2023,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2023,"num_write_kb":129472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,4],"acting":[5,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.2","version":"74'79","reported_seq":384,"reported_epoch":311,"state":"active+clean","last_fresh":"2024-09-10T19:24:34.879261+0000","last_change":"2024-09-10T19:24:23.895649+0000","last_active":"2024-09-10T19:24:34.879261+0000","last_peered":"2024-09-10T19:24:34.879261+0000","last_clean":"2024-09-10T19:24:34.879261+0000","last_became_active":"2024-09-10T19:24:23.895247+0000","last_became_peered":"2024-09-10T19:24:23.895247+0000","last_unstale":"2024-09-10T19:24:34.879261+0000","last_undegraded":"2024-09-10T19:24:34.879261+0000","last_fullsized":"2024-09-10T19:24:34.879261+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":4,"last_scrub":"74'79","last_scrub_stamp":"2024-09-10T19:21:04.941354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:08:00.699315+0000","last_clean_scrub_stamp":"2024-09-10T19:21:04.941354+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0040812920000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.7","version":"237'22","reported_seq":56,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-09-10T19:24:39.880245+0000","last_change":"2024-09-10T19:24:24.894634+0000","last_active":"2024-09-10T19:24:39.880245+0000","last_peered":"2024-09-10T19:24:39.880245+0000","last_clean":"2024-09-10T19:24:39.880245+0000","last_became_active":"2024-09-10T19:24:24.894293+0000","last_became_peered":"2024-09-10T19:24:24.894293+0000","last_unstale":"2024-09-10T19:24:39.880245+0000","last_undegraded":"2024-09-10T19:24:39.880245+0000","last_fullsized":"2024-09-10T19:24:39.880245+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"237'22","last_scrub_stamp":"2024-09-10T19:20:51.327668+0000","last_deep_scrub":"82'18","last_deep_scrub_stamp":"2024-09-10T19:15:55.777361+0000","last_clean_scrub_stamp":"2024-09-10T19:20:51.327668+0000","objects_scrubbed":5,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0057025269999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":36,"num_read_kb":25,"num_write":20,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"159'6","reported_seq":86,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-09-10T19:24:41.916189+0000","last_change":"2024-09-10T19:24:24.895131+0000","last_active":"2024-09-10T19:24:41.916189+0000","last_peered":"2024-09-10T19:24:41.916189+0000","last_clean":"2024-09-10T19:24:41.916189+0000","last_became_active":"2024-09-10T19:24:24.887336+0000","last_became_peered":"2024-09-10T19:24:24.887336+0000","last_unstale":"2024-09-10T19:24:41.916189+0000","last_undegraded":"2024-09-10T19:24:41.916189+0000","last_fullsized":"2024-09-10T19:24:41.916189+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"159'6","last_scrub_stamp":"2024-09-10T19:20:13.471084+0000","last_deep_scrub":"159'6","last_deep_scrub_stamp":"2024-09-10T19:16:32.649914+0000","last_clean_scrub_stamp":"2024-09-10T19:20:13.471084+0000","objects_scrubbed":1,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00034454499999999997,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":49,"num_read_kb":44,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":778,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":231,"reported_epoch":311,"state":"active+clean","last_fresh":"2024-09-10T19:24:34.877694+0000","last_change":"2024-09-10T19:24:24.925153+0000","last_active":"2024-09-10T19:24:34.877694+0000","last_peered":"2024-09-10T19:24:34.877694+0000","last_clean":"2024-09-10T19:24:34.877694+0000","last_became_active":"2024-09-10T19:24:24.924859+0000","last_became_peered":"2024-09-10T19:24:24.924859+0000","last_unstale":"2024-09-10T19:24:34.877694+0000","last_undegraded":"2024-09-10T19:24:34.877694+0000","last_fullsized":"2024-09-10T19:24:34.877694+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:24:05.437725+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:24:05.437725+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:25:06.046071+0000","scrub_duration":0.001691627,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":200,"reported_epoch":311,"state":"active+clean","last_fresh":"2024-09-10T19:24:34.875546+0000","last_change":"2024-09-10T19:23:28.736556+0000","last_active":"2024-09-10T19:24:34.875546+0000","last_peered":"2024-09-10T19:24:34.875546+0000","last_clean":"2024-09-10T19:24:34.875546+0000","last_became_active":"2024-09-10T19:16:24.861481+0000","last_became_peered":"2024-09-10T19:16:24.861481+0000","last_unstale":"2024-09-10T19:24:34.875546+0000","last_undegraded":"2024-09-10T19:24:34.875546+0000","last_fullsized":"2024-09-10T19:24:34.875546+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:23:28.736502+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:23:28.736502+0000","last_clean_scrub_stamp":"2024-09-10T19:23:28.736502+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0012043900000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,4],"acting":[5,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":309,"reported_epoch":311,"state":"active+clean","last_fresh":"2024-09-10T19:24:34.880546+0000","last_change":"2024-09-10T19:23:27.463818+0000","last_active":"2024-09-10T19:24:34.880546+0000","last_peered":"2024-09-10T19:24:34.880546+0000","last_clean":"2024-09-10T19:24:34.880546+0000","last_became_active":"2024-09-10T19:10:14.053697+0000","last_became_peered":"2024-09-10T19:10:14.053697+0000","last_unstale":"2024-09-10T19:24:34.880546+0000","last_undegraded":"2024-09-10T19:24:34.880546+0000","last_fullsized":"2024-09-10T19:24:34.880546+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:23:27.463763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:22:01.208272+0000","last_clean_scrub_stamp":"2024-09-10T19:23:27.463763+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0012346900000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.8","version":"312'2180","reported_seq":2219,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-09-10T19:24:41.866404+0000","last_change":"2024-09-10T19:24:29.882455+0000","last_active":"2024-09-10T19:24:41.866404+0000","last_peered":"2024-09-10T19:24:41.866404+0000","last_clean":"2024-09-10T19:24:41.866404+0000","last_became_active":"2024-09-10T19:24:29.877601+0000","last_became_peered":"2024-09-10T19:24:29.877601+0000","last_unstale":"2024-09-10T19:24:41.866404+0000","last_undegraded":"2024-09-10T19:24:41.866404+0000","last_fullsized":"2024-09-10T19:24:41.866404+0000","mapping_epoch":307,"log_start":"0'0","ondisk_log_start":"0'0","created":295,"last_epoch_clean":308,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:24:08.023080+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:24:08.023080+0000","last_clean_scrub_stamp":"2024-09-10T19:24:08.023080+0000","objects_scrubbed":0,"log_size":2180,"log_dups_size":0,"ondisk_log_size":2180,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:25:29.489867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":142868480,"num_objects":2180,"num_object_clones":0,"num_object_copies":6540,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2180,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2180,"num_write_kb":139520,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":455,"num_bytes_recovered":29818880,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,5],"acting":[0,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"74'79","reported_seq":30,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-09-10T19:24:39.879352+0000","last_change":"2024-09-10T19:24:01.977897+0000","last_active":"2024-09-10T19:24:39.879352+0000","last_peered":"2024-09-10T19:24:39.879352+0000","last_clean":"2024-09-10T19:24:39.879352+0000","last_became_active":"2024-09-10T19:24:01.977477+0000","last_became_peered":"2024-09-10T19:24:01.977477+0000","last_unstale":"2024-09-10T19:24:39.879352+0000","last_undegraded":"2024-09-10T19:24:39.879352+0000","last_fullsized":"2024-09-10T19:24:39.879352+0000","mapping_epoch":292,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":293,"parent":"0.0","parent_split_bits":3,"last_scrub":"74'79","last_scrub_stamp":"2024-09-10T19:23:14.562829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:08:00.699315+0000","last_clean_scrub_stamp":"2024-09-10T19:23:14.562829+0000","objects_scrubbed":2,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0030997160000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":20,"num_write":11,"num_write_kb":148,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":53670,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"237'30","reported_seq":362,"reported_epoch":311,"state":"active+clean","last_fresh":"2024-09-10T19:24:34.881835+0000","last_change":"2024-09-10T19:24:24.907065+0000","last_active":"2024-09-10T19:24:34.881835+0000","last_peered":"2024-09-10T19:24:34.881835+0000","last_clean":"2024-09-10T19:24:34.881835+0000","last_became_active":"2024-09-10T19:24:24.899711+0000","last_became_peered":"2024-09-10T19:24:24.899711+0000","last_unstale":"2024-09-10T19:24:34.881835+0000","last_undegraded":"2024-09-10T19:24:34.881835+0000","last_fullsized":"2024-09-10T19:24:34.881835+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"237'30","last_scrub_stamp":"2024-09-10T19:21:07.733508+0000","last_deep_scrub":"221'18","last_deep_scrub_stamp":"2024-09-10T19:19:29.606748+0000","last_clean_scrub_stamp":"2024-09-10T19:21:07.733508+0000","objects_scrubbed":6,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.020236845,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"61'1","reported_seq":265,"reported_epoch":311,"state":"active+clean","last_fresh":"2024-09-10T19:24:34.880218+0000","last_change":"2024-09-10T19:24:23.899583+0000","last_active":"2024-09-10T19:24:34.880218+0000","last_peered":"2024-09-10T19:24:34.880218+0000","last_clean":"2024-09-10T19:24:34.880218+0000","last_became_active":"2024-09-10T19:24:23.897412+0000","last_became_peered":"2024-09-10T19:24:23.897412+0000","last_unstale":"2024-09-10T19:24:34.880218+0000","last_undegraded":"2024-09-10T19:24:34.880218+0000","last_fullsized":"2024-09-10T19:24:34.880218+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'1","last_scrub_stamp":"2024-09-10T19:20:40.716730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:20:40.716730+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":350,"reported_epoch":311,"state":"active+clean","last_fresh":"2024-09-10T19:24:34.878148+0000","last_change":"2024-09-10T19:24:24.896465+0000","last_active":"2024-09-10T19:24:34.878148+0000","last_peered":"2024-09-10T19:24:34.878148+0000","last_clean":"2024-09-10T19:24:34.878148+0000","last_became_active":"2024-09-10T19:24:24.896201+0000","last_became_peered":"2024-09-10T19:24:24.896201+0000","last_unstale":"2024-09-10T19:24:34.878148+0000","last_undegraded":"2024-09-10T19:24:34.878148+0000","last_fullsized":"2024-09-10T19:24:34.878148+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:24:03.653347+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:24:03.653347+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:25:07.643899+0000","scrub_duration":0.001322178,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":346,"reported_epoch":311,"state":"active+clean","last_fresh":"2024-09-10T19:24:34.876351+0000","last_change":"2024-09-10T19:24:24.914893+0000","last_active":"2024-09-10T19:24:34.876351+0000","last_peered":"2024-09-10T19:24:34.876351+0000","last_clean":"2024-09-10T19:24:34.876351+0000","last_became_active":"2024-09-10T19:24:24.914680+0000","last_became_peered":"2024-09-10T19:24:24.914680+0000","last_unstale":"2024-09-10T19:24:34.876351+0000","last_undegraded":"2024-09-10T19:24:34.876351+0000","last_fullsized":"2024-09-10T19:24:34.876351+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:21:08.210604+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:21:08.210604+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0056788289999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":30,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-09-10T19:24:39.879947+0000","last_change":"2024-09-10T19:24:01.974860+0000","last_active":"2024-09-10T19:24:39.879947+0000","last_peered":"2024-09-10T19:24:39.879947+0000","last_clean":"2024-09-10T19:24:39.879947+0000","last_became_active":"2024-09-10T19:24:01.974039+0000","last_became_peered":"2024-09-10T19:24:01.974039+0000","last_unstale":"2024-09-10T19:24:39.879947+0000","last_undegraded":"2024-09-10T19:24:39.879947+0000","last_fullsized":"2024-09-10T19:24:39.879947+0000","mapping_epoch":292,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":293,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:23:13.599885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:23:13.599885+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.b","version":"312'2056","reported_seq":3056,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-09-10T19:24:40.003282+0000","last_change":"2024-09-10T19:24:28.945977+0000","last_active":"2024-09-10T19:24:40.003282+0000","last_peered":"2024-09-10T19:24:40.003282+0000","last_clean":"2024-09-10T19:24:40.003282+0000","last_became_active":"2024-09-10T19:24:28.940329+0000","last_became_peered":"2024-09-10T19:24:28.940329+0000","last_unstale":"2024-09-10T19:24:40.003282+0000","last_undegraded":"2024-09-10T19:24:40.003282+0000","last_fullsized":"2024-09-10T19:24:40.003282+0000","mapping_epoch":306,"log_start":"0'0","ondisk_log_start":"0'0","created":295,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:24:08.023080+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:24:08.023080+0000","last_clean_scrub_stamp":"2024-09-10T19:24:08.023080+0000","objects_scrubbed":0,"log_size":2056,"log_dups_size":0,"ondisk_log_size":2056,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:25:28.671247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":134742016,"num_objects":2056,"num_object_clones":0,"num_object_copies":6168,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2056,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2056,"num_write_kb":131584,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":478,"num_bytes_recovered":31326208,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"237'33","reported_seq":377,"reported_epoch":311,"state":"active+clean","last_fresh":"2024-09-10T19:24:34.877773+0000","last_change":"2024-09-10T19:23:28.405245+0000","last_active":"2024-09-10T19:24:34.877773+0000","last_peered":"2024-09-10T19:24:34.877773+0000","last_clean":"2024-09-10T19:24:34.877773+0000","last_became_active":"2024-09-10T19:10:09.822184+0000","last_became_peered":"2024-09-10T19:10:09.822184+0000","last_unstale":"2024-09-10T19:24:34.877773+0000","last_undegraded":"2024-09-10T19:24:34.877773+0000","last_fullsized":"2024-09-10T19:24:34.877773+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"237'33","last_scrub_stamp":"2024-09-10T19:23:28.405185+0000","last_deep_scrub":"80'21","last_deep_scrub_stamp":"2024-09-10T19:11:39.836144+0000","last_clean_scrub_stamp":"2024-09-10T19:23:28.405185+0000","objects_scrubbed":7,"log_size":33,"log_dups_size":0,"ondisk_log_size":33,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0019720100000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":33,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"63'2","reported_seq":319,"reported_epoch":311,"state":"active+clean","last_fresh":"2024-09-10T19:24:34.877616+0000","last_change":"2024-09-10T19:23:28.275635+0000","last_active":"2024-09-10T19:24:34.877616+0000","last_peered":"2024-09-10T19:24:34.877616+0000","last_clean":"2024-09-10T19:24:34.877616+0000","last_became_active":"2024-09-10T19:10:07.684784+0000","last_became_peered":"2024-09-10T19:10:07.684784+0000","last_unstale":"2024-09-10T19:24:34.877616+0000","last_undegraded":"2024-09-10T19:24:34.877616+0000","last_fullsized":"2024-09-10T19:24:34.877616+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'2","last_scrub_stamp":"2024-09-10T19:23:28.275581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:23:28.275581+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001514266,"objects_trimmed":0,"snaptrim_duration":0.00075522999999999996,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":322,"reported_epoch":311,"state":"active+clean","last_fresh":"2024-09-10T19:24:34.876507+0000","last_change":"2024-09-10T19:22:33.611730+0000","last_active":"2024-09-10T19:24:34.876507+0000","last_peered":"2024-09-10T19:24:34.876507+0000","last_clean":"2024-09-10T19:24:34.876507+0000","last_became_active":"2024-09-10T19:16:24.858662+0000","last_became_peered":"2024-09-10T19:16:24.858662+0000","last_unstale":"2024-09-10T19:24:34.876507+0000","last_undegraded":"2024-09-10T19:24:34.876507+0000","last_fullsized":"2024-09-10T19:24:34.876507+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:22:33.611658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:22:33.611658+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.012996436,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"74'79","reported_seq":375,"reported_epoch":311,"state":"active+clean","last_fresh":"2024-09-10T19:24:34.878307+0000","last_change":"2024-09-10T19:24:23.904928+0000","last_active":"2024-09-10T19:24:34.878307+0000","last_peered":"2024-09-10T19:24:34.878307+0000","last_clean":"2024-09-10T19:24:34.878307+0000","last_became_active":"2024-09-10T19:24:23.904267+0000","last_became_peered":"2024-09-10T19:24:23.904267+0000","last_unstale":"2024-09-10T19:24:34.878307+0000","last_undegraded":"2024-09-10T19:24:34.878307+0000","last_fullsized":"2024-09-10T19:24:34.878307+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'79","last_scrub_stamp":"2024-09-10T19:20:59.897158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:08:00.699315+0000","last_clean_scrub_stamp":"2024-09-10T19:20:59.897158+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0081933070000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":325,"reported_epoch":311,"state":"active+clean","last_fresh":"2024-09-10T19:24:34.882686+0000","last_change":"2024-09-10T19:24:23.904160+0000","last_active":"2024-09-10T19:24:34.882686+0000","last_peered":"2024-09-10T19:24:34.882686+0000","last_clean":"2024-09-10T19:24:34.882686+0000","last_became_active":"2024-09-10T19:24:23.903868+0000","last_became_peered":"2024-09-10T19:24:23.903868+0000","last_unstale":"2024-09-10T19:24:34.882686+0000","last_undegraded":"2024-09-10T19:24:34.882686+0000","last_fullsized":"2024-09-10T19:24:34.882686+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:20:54.870558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:20:54.870558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0080190729999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,0],"acting":[7,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":306,"reported_epoch":311,"state":"active+clean","last_fresh":"2024-09-10T19:24:34.876479+0000","last_change":"2024-09-10T19:24:00.725537+0000","last_active":"2024-09-10T19:24:34.876479+0000","last_peered":"2024-09-10T19:24:34.876479+0000","last_clean":"2024-09-10T19:24:34.876479+0000","last_became_active":"2024-09-10T19:10:13.720647+0000","last_became_peered":"2024-09-10T19:10:13.720647+0000","last_unstale":"2024-09-10T19:24:34.876479+0000","last_undegraded":"2024-09-10T19:24:34.876479+0000","last_fullsized":"2024-09-10T19:24:34.876479+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:24:00.725477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:24:00.725477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:25:10.261661+0000","scrub_duration":0.0012023559999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.4","version":"312'2179","reported_seq":2501,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-09-10T19:24:41.876526+0000","last_change":"2024-09-10T19:24:30.886617+0000","last_active":"2024-09-10T19:24:41.876526+0000","last_peered":"2024-09-10T19:24:41.876526+0000","last_clean":"2024-09-10T19:24:41.876526+0000","last_became_active":"2024-09-10T19:24:30.884803+0000","last_became_peered":"2024-09-10T19:24:30.884803+0000","last_unstale":"2024-09-10T19:24:41.876526+0000","last_undegraded":"2024-09-10T19:24:41.876526+0000","last_fullsized":"2024-09-10T19:24:41.876526+0000","mapping_epoch":308,"log_start":"0'0","ondisk_log_start":"0'0","created":295,"last_epoch_clean":309,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:24:08.023080+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:24:08.023080+0000","last_clean_scrub_stamp":"2024-09-10T19:24:08.023080+0000","objects_scrubbed":0,"log_size":2179,"log_dups_size":0,"ondisk_log_size":2179,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:25:36.711965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":142802944,"num_objects":2179,"num_object_clones":0,"num_object_copies":6537,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2179,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2179,"num_write_kb":139456,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":441,"num_bytes_recovered":28901376,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"237'35","reported_seq":448,"reported_epoch":311,"state":"active+clean","last_fresh":"2024-09-10T19:24:34.879361+0000","last_change":"2024-09-10T19:24:15.543273+0000","last_active":"2024-09-10T19:24:34.879361+0000","last_peered":"2024-09-10T19:24:34.879361+0000","last_clean":"2024-09-10T19:24:34.879361+0000","last_became_active":"2024-09-10T19:24:01.973201+0000","last_became_peered":"2024-09-10T19:24:01.973201+0000","last_unstale":"2024-09-10T19:24:34.879361+0000","last_undegraded":"2024-09-10T19:24:34.879361+0000","last_fullsized":"2024-09-10T19:24:34.879361+0000","mapping_epoch":292,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":293,"parent":"0.0","parent_split_bits":0,"last_scrub":"237'35","last_scrub_stamp":"2024-09-10T19:24:15.543217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:24:15.543217+0000","objects_scrubbed":9,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:25:45.036371+0000","scrub_duration":0.001850832,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":56,"num_read_kb":45,"num_write":36,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":221,"reported_epoch":311,"state":"active+clean","last_fresh":"2024-09-10T19:24:34.875142+0000","last_change":"2024-09-10T19:24:24.913980+0000","last_active":"2024-09-10T19:24:34.875142+0000","last_peered":"2024-09-10T19:24:34.875142+0000","last_clean":"2024-09-10T19:24:34.875142+0000","last_became_active":"2024-09-10T19:24:24.913709+0000","last_became_peered":"2024-09-10T19:24:24.913709+0000","last_unstale":"2024-09-10T19:24:34.875142+0000","last_undegraded":"2024-09-10T19:24:34.875142+0000","last_fullsized":"2024-09-10T19:24:34.875142+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:24:05.723155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:24:05.723155+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:25:24.092483+0000","scrub_duration":0.0015060060000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,1],"acting":[5,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":132,"reported_epoch":311,"state":"active+clean","last_fresh":"2024-09-10T19:24:34.880027+0000","last_change":"2024-09-10T19:24:23.902155+0000","last_active":"2024-09-10T19:24:34.880027+0000","last_peered":"2024-09-10T19:24:34.880027+0000","last_clean":"2024-09-10T19:24:34.880027+0000","last_became_active":"2024-09-10T19:24:23.901415+0000","last_became_peered":"2024-09-10T19:24:23.901415+0000","last_unstale":"2024-09-10T19:24:34.880027+0000","last_undegraded":"2024-09-10T19:24:34.880027+0000","last_fullsized":"2024-09-10T19:24:34.880027+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:20:03.641376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:20:03.641376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":326,"reported_epoch":311,"state":"active+clean","last_fresh":"2024-09-10T19:24:34.878230+0000","last_change":"2024-09-10T19:23:59.577506+0000","last_active":"2024-09-10T19:24:34.878230+0000","last_peered":"2024-09-10T19:24:34.878230+0000","last_clean":"2024-09-10T19:24:34.878230+0000","last_became_active":"2024-09-10T19:16:24.858172+0000","last_became_peered":"2024-09-10T19:16:24.858172+0000","last_unstale":"2024-09-10T19:24:34.878230+0000","last_undegraded":"2024-09-10T19:24:34.878230+0000","last_fullsized":"2024-09-10T19:24:34.878230+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:23:59.577451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:21:03.910074+0000","last_clean_scrub_stamp":"2024-09-10T19:23:59.577451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:25:11.613302+0000","scrub_duration":0.001240019,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,6],"acting":[7,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.5","version":"312'2222","reported_seq":2353,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-09-10T19:24:41.880625+0000","last_change":"2024-09-10T19:24:29.879080+0000","last_active":"2024-09-10T19:24:41.880625+0000","last_peered":"2024-09-10T19:24:41.880625+0000","last_clean":"2024-09-10T19:24:41.880625+0000","last_became_active":"2024-09-10T19:24:29.877183+0000","last_became_peered":"2024-09-10T19:24:29.877183+0000","last_unstale":"2024-09-10T19:24:41.880625+0000","last_undegraded":"2024-09-10T19:24:41.880625+0000","last_fullsized":"2024-09-10T19:24:41.880625+0000","mapping_epoch":307,"log_start":"0'0","ondisk_log_start":"0'0","created":295,"last_epoch_clean":308,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:24:08.023080+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:24:08.023080+0000","last_clean_scrub_stamp":"2024-09-10T19:24:08.023080+0000","objects_scrubbed":0,"log_size":2222,"log_dups_size":0,"ondisk_log_size":2222,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:25:15.626460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":145620992,"num_objects":2222,"num_object_clones":0,"num_object_copies":6666,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2222,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2222,"num_write_kb":142208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":446,"num_bytes_recovered":29229056,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.b","version":"237'15","reported_seq":326,"reported_epoch":311,"state":"active+clean","last_fresh":"2024-09-10T19:24:34.880494+0000","last_change":"2024-09-10T19:24:24.892009+0000","last_active":"2024-09-10T19:24:34.880494+0000","last_peered":"2024-09-10T19:24:34.880494+0000","last_clean":"2024-09-10T19:24:34.880494+0000","last_became_active":"2024-09-10T19:24:24.891636+0000","last_became_peered":"2024-09-10T19:24:24.891636+0000","last_unstale":"2024-09-10T19:24:34.880494+0000","last_undegraded":"2024-09-10T19:24:34.880494+0000","last_fullsized":"2024-09-10T19:24:34.880494+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"237'15","last_scrub_stamp":"2024-09-10T19:21:46.837828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:21:46.837828+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001540919,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":30,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-09-10T19:24:39.879587+0000","last_change":"2024-09-10T19:24:01.980716+0000","last_active":"2024-09-10T19:24:39.879587+0000","last_peered":"2024-09-10T19:24:39.879587+0000","last_clean":"2024-09-10T19:24:39.879587+0000","last_became_active":"2024-09-10T19:24:01.980348+0000","last_became_peered":"2024-09-10T19:24:01.980348+0000","last_unstale":"2024-09-10T19:24:39.879587+0000","last_undegraded":"2024-09-10T19:24:39.879587+0000","last_fullsized":"2024-09-10T19:24:39.879587+0000","mapping_epoch":292,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":293,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:23:56.445943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:11:44.254513+0000","last_clean_scrub_stamp":"2024-09-10T19:23:56.445943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:25:20.978913+0000","scrub_duration":0.001815419,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":210,"reported_epoch":311,"state":"active+clean","last_fresh":"2024-09-10T19:24:34.879323+0000","last_change":"2024-09-10T19:24:23.891447+0000","last_active":"2024-09-10T19:24:34.879323+0000","last_peered":"2024-09-10T19:24:34.879323+0000","last_clean":"2024-09-10T19:24:34.879323+0000","last_became_active":"2024-09-10T19:24:23.891161+0000","last_became_peered":"2024-09-10T19:24:23.891161+0000","last_unstale":"2024-09-10T19:24:34.879323+0000","last_undegraded":"2024-09-10T19:24:34.879323+0000","last_fullsized":"2024-09-10T19:24:34.879323+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:21:09.367669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:21:09.367669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0065456109999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":250,"reported_epoch":311,"state":"active+clean","last_fresh":"2024-09-10T19:24:34.880299+0000","last_change":"2024-09-10T19:24:23.905598+0000","last_active":"2024-09-10T19:24:34.880299+0000","last_peered":"2024-09-10T19:24:34.880299+0000","last_clean":"2024-09-10T19:24:34.880299+0000","last_became_active":"2024-09-10T19:24:23.905112+0000","last_became_peered":"2024-09-10T19:24:23.905112+0000","last_unstale":"2024-09-10T19:24:34.880299+0000","last_undegraded":"2024-09-10T19:24:34.880299+0000","last_fullsized":"2024-09-10T19:24:34.880299+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:20:08.746643+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:20:08.746643+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0058765429999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.6","version":"312'2018","reported_seq":3038,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-09-10T19:24:40.369450+0000","last_change":"2024-09-10T19:24:30.889054+0000","last_active":"2024-09-10T19:24:40.369450+0000","last_peered":"2024-09-10T19:24:40.369450+0000","last_clean":"2024-09-10T19:24:40.369450+0000","last_became_active":"2024-09-10T19:24:30.883926+0000","last_became_peered":"2024-09-10T19:24:30.883926+0000","last_unstale":"2024-09-10T19:24:40.369450+0000","last_undegraded":"2024-09-10T19:24:40.369450+0000","last_fullsized":"2024-09-10T19:24:40.369450+0000","mapping_epoch":308,"log_start":"0'0","ondisk_log_start":"0'0","created":295,"last_epoch_clean":309,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:24:08.023080+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:24:08.023080+0000","last_clean_scrub_stamp":"2024-09-10T19:24:08.023080+0000","objects_scrubbed":0,"log_size":2018,"log_dups_size":0,"ondisk_log_size":2018,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:25:32.830579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":132251648,"num_objects":2018,"num_object_clones":0,"num_object_copies":6054,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2018,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2018,"num_write_kb":129152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":488,"num_bytes_recovered":31981568,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,4],"acting":[5,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.8","version":"237'34","reported_seq":350,"reported_epoch":311,"state":"active+clean","last_fresh":"2024-09-10T19:24:34.880216+0000","last_change":"2024-09-10T19:21:58.440243+0000","last_active":"2024-09-10T19:24:34.880216+0000","last_peered":"2024-09-10T19:24:34.880216+0000","last_clean":"2024-09-10T19:24:34.880216+0000","last_became_active":"2024-09-10T19:10:09.820950+0000","last_became_peered":"2024-09-10T19:10:09.820950+0000","last_unstale":"2024-09-10T19:24:34.880216+0000","last_undegraded":"2024-09-10T19:24:34.880216+0000","last_fullsized":"2024-09-10T19:24:34.880216+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"237'34","last_scrub_stamp":"2024-09-10T19:21:58.440165+0000","last_deep_scrub":"220'22","last_deep_scrub_stamp":"2024-09-10T19:19:24.597866+0000","last_clean_scrub_stamp":"2024-09-10T19:21:58.440165+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0050099799999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":190,"reported_epoch":311,"state":"active+clean","last_fresh":"2024-09-10T19:24:34.878148+0000","last_change":"2024-09-10T19:23:20.338930+0000","last_active":"2024-09-10T19:24:34.878148+0000","last_peered":"2024-09-10T19:24:34.878148+0000","last_clean":"2024-09-10T19:24:34.878148+0000","last_became_active":"2024-09-10T19:16:24.857253+0000","last_became_peered":"2024-09-10T19:16:24.857253+0000","last_unstale":"2024-09-10T19:24:34.878148+0000","last_undegraded":"2024-09-10T19:24:34.878148+0000","last_fullsized":"2024-09-10T19:24:34.878148+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:23:20.338866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:23:20.338866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:24:43.851289+0000","scrub_duration":0.0059949699999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.9","version":"80'9","reported_seq":501,"reported_epoch":311,"state":"active+clean","last_fresh":"2024-09-10T19:24:35.905662+0000","last_change":"2024-09-10T19:24:01.602598+0000","last_active":"2024-09-10T19:24:35.905662+0000","last_peered":"2024-09-10T19:24:35.905662+0000","last_clean":"2024-09-10T19:24:35.905662+0000","last_became_active":"2024-09-10T19:10:11.702878+0000","last_became_peered":"2024-09-10T19:10:11.702878+0000","last_unstale":"2024-09-10T19:24:35.905662+0000","last_undegraded":"2024-09-10T19:24:35.905662+0000","last_fullsized":"2024-09-10T19:24:35.905662+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"80'9","last_scrub_stamp":"2024-09-10T19:24:01.602542+0000","last_deep_scrub":"80'9","last_deep_scrub_stamp":"2024-09-10T19:20:57.882496+0000","last_clean_scrub_stamp":"2024-09-10T19:24:01.602542+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:25:08.483478+0000","scrub_duration":0.001381452,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":313,"reported_epoch":311,"state":"active+clean","last_fresh":"2024-09-10T19:24:38.483915+0000","last_change":"2024-09-10T19:24:38.483717+0000","last_active":"2024-09-10T19:24:38.483915+0000","last_peered":"2024-09-10T19:24:38.483915+0000","last_clean":"2024-09-10T19:24:38.483915+0000","last_became_active":"2024-09-10T19:24:23.905606+0000","last_became_peered":"2024-09-10T19:24:23.905606+0000","last_unstale":"2024-09-10T19:24:38.483915+0000","last_undegraded":"2024-09-10T19:24:38.483915+0000","last_fullsized":"2024-09-10T19:24:38.483915+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:24:38.483596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:24:38.483596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:25:48.069076+0000","scrub_duration":0.0049697939999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.7","version":"312'1898","reported_seq":1923,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-09-10T19:24:40.017869+0000","last_change":"2024-09-10T19:24:09.045892+0000","last_active":"2024-09-10T19:24:40.017869+0000","last_peered":"2024-09-10T19:24:40.017869+0000","last_clean":"2024-09-10T19:24:40.017869+0000","last_became_active":"2024-09-10T19:24:09.045557+0000","last_became_peered":"2024-09-10T19:24:09.045557+0000","last_unstale":"2024-09-10T19:24:40.017869+0000","last_undegraded":"2024-09-10T19:24:40.017869+0000","last_fullsized":"2024-09-10T19:24:40.017869+0000","mapping_epoch":295,"log_start":"0'0","ondisk_log_start":"0'0","created":295,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:24:08.023080+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:24:08.023080+0000","last_clean_scrub_stamp":"2024-09-10T19:24:08.023080+0000","objects_scrubbed":0,"log_size":1898,"log_dups_size":0,"ondisk_log_size":1898,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:25:14.046129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":124387328,"num_objects":1898,"num_object_clones":0,"num_object_copies":5694,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1898,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1898,"num_write_kb":121472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.9","version":"237'18","reported_seq":395,"reported_epoch":311,"state":"active+clean","last_fresh":"2024-09-10T19:24:34.880923+0000","last_change":"2024-09-10T19:24:01.997134+0000","last_active":"2024-09-10T19:24:34.880923+0000","last_peered":"2024-09-10T19:24:34.880923+0000","last_clean":"2024-09-10T19:24:34.880923+0000","last_became_active":"2024-09-10T19:24:01.974878+0000","last_became_peered":"2024-09-10T19:24:01.974878+0000","last_unstale":"2024-09-10T19:24:34.880923+0000","last_undegraded":"2024-09-10T19:24:34.880923+0000","last_fullsized":"2024-09-10T19:24:34.880923+0000","mapping_epoch":292,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":293,"parent":"0.0","parent_split_bits":0,"last_scrub":"237'18","last_scrub_stamp":"2024-09-10T19:23:26.714557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:23:26.714557+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:24:43.891802+0000","scrub_duration":0.001833859,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.f","version":"59'2","reported_seq":334,"reported_epoch":311,"state":"active+clean","last_fresh":"2024-09-10T19:24:34.879400+0000","last_change":"2024-09-10T19:24:23.894221+0000","last_active":"2024-09-10T19:24:34.879400+0000","last_peered":"2024-09-10T19:24:34.879400+0000","last_clean":"2024-09-10T19:24:34.879400+0000","last_became_active":"2024-09-10T19:24:23.893796+0000","last_became_peered":"2024-09-10T19:24:23.893796+0000","last_unstale":"2024-09-10T19:24:34.879400+0000","last_undegraded":"2024-09-10T19:24:34.879400+0000","last_fullsized":"2024-09-10T19:24:34.879400+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'2","last_scrub_stamp":"2024-09-10T19:20:56.467569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:20:56.467569+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.013126802,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":14,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":244,"reported_epoch":311,"state":"active+clean","last_fresh":"2024-09-10T19:24:34.882308+0000","last_change":"2024-09-10T19:24:24.917262+0000","last_active":"2024-09-10T19:24:34.882308+0000","last_peered":"2024-09-10T19:24:34.882308+0000","last_clean":"2024-09-10T19:24:34.882308+0000","last_became_active":"2024-09-10T19:24:24.914252+0000","last_became_peered":"2024-09-10T19:24:24.914252+0000","last_unstale":"2024-09-10T19:24:34.882308+0000","last_undegraded":"2024-09-10T19:24:34.882308+0000","last_fullsized":"2024-09-10T19:24:34.882308+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:21:45.880289+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:11:46.587846+0000","last_clean_scrub_stamp":"2024-09-10T19:21:45.880289+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001267142,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,5],"acting":[0,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":323,"reported_epoch":311,"state":"active+clean","last_fresh":"2024-09-10T19:24:34.877636+0000","last_change":"2024-09-10T19:24:01.971525+0000","last_active":"2024-09-10T19:24:34.877636+0000","last_peered":"2024-09-10T19:24:34.877636+0000","last_clean":"2024-09-10T19:24:34.877636+0000","last_became_active":"2024-09-10T19:24:01.971131+0000","last_became_peered":"2024-09-10T19:24:01.971131+0000","last_unstale":"2024-09-10T19:24:34.877636+0000","last_undegraded":"2024-09-10T19:24:34.877636+0000","last_fullsized":"2024-09-10T19:24:34.877636+0000","mapping_epoch":292,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":293,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:23:19.383453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:23:19.383453+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0094206629999999993,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.16","version":"237'15","reported_seq":300,"reported_epoch":311,"state":"active+clean","last_fresh":"2024-09-10T19:24:34.881069+0000","last_change":"2024-09-10T19:24:23.901956+0000","last_active":"2024-09-10T19:24:34.881069+0000","last_peered":"2024-09-10T19:24:34.881069+0000","last_clean":"2024-09-10T19:24:34.881069+0000","last_became_active":"2024-09-10T19:24:23.901610+0000","last_became_peered":"2024-09-10T19:24:23.901610+0000","last_unstale":"2024-09-10T19:24:34.881069+0000","last_undegraded":"2024-09-10T19:24:34.881069+0000","last_fullsized":"2024-09-10T19:24:34.881069+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"237'15","last_scrub_stamp":"2024-09-10T19:21:01.691067+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:21:01.691067+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0065392059999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":44,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-09-10T19:24:39.879295+0000","last_change":"2024-09-10T19:24:24.895309+0000","last_active":"2024-09-10T19:24:39.879295+0000","last_peered":"2024-09-10T19:24:39.879295+0000","last_clean":"2024-09-10T19:24:39.879295+0000","last_became_active":"2024-09-10T19:24:24.894815+0000","last_became_peered":"2024-09-10T19:24:24.894815+0000","last_unstale":"2024-09-10T19:24:39.879295+0000","last_undegraded":"2024-09-10T19:24:39.879295+0000","last_fullsized":"2024-09-10T19:24:39.879295+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:20:57.698558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:11:41.458523+0000","last_clean_scrub_stamp":"2024-09-10T19:20:57.698558+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0012499830000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,4],"acting":[1,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":332,"reported_epoch":311,"state":"active+clean","last_fresh":"2024-09-10T19:24:34.878377+0000","last_change":"2024-09-10T19:24:09.367710+0000","last_active":"2024-09-10T19:24:34.878377+0000","last_peered":"2024-09-10T19:24:34.878377+0000","last_clean":"2024-09-10T19:24:34.878377+0000","last_became_active":"2024-09-10T19:24:01.970325+0000","last_became_peered":"2024-09-10T19:24:01.970325+0000","last_unstale":"2024-09-10T19:24:34.878377+0000","last_undegraded":"2024-09-10T19:24:34.878377+0000","last_fullsized":"2024-09-10T19:24:34.878377+0000","mapping_epoch":292,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":293,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:24:09.367654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:24:09.367654+0000","last_clean_scrub_stamp":"2024-09-10T19:24:09.367654+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:25:35.362846+0000","scrub_duration":0.0016005710000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"66'1","reported_seq":210,"reported_epoch":311,"state":"active+clean","last_fresh":"2024-09-10T19:24:34.879620+0000","last_change":"2024-09-10T19:24:23.898172+0000","last_active":"2024-09-10T19:24:34.879620+0000","last_peered":"2024-09-10T19:24:34.879620+0000","last_clean":"2024-09-10T19:24:34.879620+0000","last_became_active":"2024-09-10T19:24:23.897858+0000","last_became_peered":"2024-09-10T19:24:23.897858+0000","last_unstale":"2024-09-10T19:24:34.879620+0000","last_undegraded":"2024-09-10T19:24:34.879620+0000","last_fullsized":"2024-09-10T19:24:34.879620+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'1","last_scrub_stamp":"2024-09-10T19:20:07.338085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:20:07.338085+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0056882089999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":13,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":345,"reported_epoch":311,"state":"active+clean","last_fresh":"2024-09-10T19:24:34.877849+0000","last_change":"2024-09-10T19:24:01.976040+0000","last_active":"2024-09-10T19:24:34.877849+0000","last_peered":"2024-09-10T19:24:34.877849+0000","last_clean":"2024-09-10T19:24:34.877849+0000","last_became_active":"2024-09-10T19:24:01.975706+0000","last_became_peered":"2024-09-10T19:24:01.975706+0000","last_unstale":"2024-09-10T19:24:34.877849+0000","last_undegraded":"2024-09-10T19:24:34.877849+0000","last_fullsized":"2024-09-10T19:24:34.877849+0000","mapping_epoch":292,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":293,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:23:25.466309+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:23:25.466309+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:24:48.383321+0000","scrub_duration":0.001495176,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"237'12","reported_seq":369,"reported_epoch":311,"state":"active+clean","last_fresh":"2024-09-10T19:24:34.878543+0000","last_change":"2024-09-10T19:24:01.991983+0000","last_active":"2024-09-10T19:24:34.878543+0000","last_peered":"2024-09-10T19:24:34.878543+0000","last_clean":"2024-09-10T19:24:34.878543+0000","last_became_active":"2024-09-10T19:24:01.971492+0000","last_became_peered":"2024-09-10T19:24:01.971492+0000","last_unstale":"2024-09-10T19:24:34.878543+0000","last_undegraded":"2024-09-10T19:24:34.878543+0000","last_fullsized":"2024-09-10T19:24:34.878543+0000","mapping_epoch":292,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":293,"parent":"0.0","parent_split_bits":0,"last_scrub":"237'12","last_scrub_stamp":"2024-09-10T19:22:26.419497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:22:26.419497+0000","objects_scrubbed":3,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0033796080000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":18,"num_read_kb":15,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":325,"reported_epoch":311,"state":"active+clean","last_fresh":"2024-09-10T19:24:34.875611+0000","last_change":"2024-09-10T19:24:01.978524+0000","last_active":"2024-09-10T19:24:34.875611+0000","last_peered":"2024-09-10T19:24:34.875611+0000","last_clean":"2024-09-10T19:24:34.875611+0000","last_became_active":"2024-09-10T19:24:01.978215+0000","last_became_peered":"2024-09-10T19:24:01.978215+0000","last_unstale":"2024-09-10T19:24:34.875611+0000","last_undegraded":"2024-09-10T19:24:34.875611+0000","last_fullsized":"2024-09-10T19:24:34.875611+0000","mapping_epoch":292,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":293,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:22:30.625942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:11:23.264793+0000","last_clean_scrub_stamp":"2024-09-10T19:22:30.625942+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.010093893,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":306,"reported_epoch":311,"state":"active+clean","last_fresh":"2024-09-10T19:24:34.880159+0000","last_change":"2024-09-10T19:22:28.305076+0000","last_active":"2024-09-10T19:24:34.880159+0000","last_peered":"2024-09-10T19:24:34.880159+0000","last_clean":"2024-09-10T19:24:34.880159+0000","last_became_active":"2024-09-10T19:10:14.054360+0000","last_became_peered":"2024-09-10T19:10:14.054360+0000","last_unstale":"2024-09-10T19:24:34.880159+0000","last_undegraded":"2024-09-10T19:24:34.880159+0000","last_fullsized":"2024-09-10T19:24:34.880159+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:22:28.304999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:22:28.304999+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0082740739999999993,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"237'25","reported_seq":408,"reported_epoch":311,"state":"active+clean","last_fresh":"2024-09-10T19:24:34.877369+0000","last_change":"2024-09-10T19:24:01.988172+0000","last_active":"2024-09-10T19:24:34.877369+0000","last_peered":"2024-09-10T19:24:34.877369+0000","last_clean":"2024-09-10T19:24:34.877369+0000","last_became_active":"2024-09-10T19:24:01.971173+0000","last_became_peered":"2024-09-10T19:24:01.971173+0000","last_unstale":"2024-09-10T19:24:34.877369+0000","last_undegraded":"2024-09-10T19:24:34.877369+0000","last_fullsized":"2024-09-10T19:24:34.877369+0000","mapping_epoch":292,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":293,"parent":"0.0","parent_split_bits":0,"last_scrub":"237'25","last_scrub_stamp":"2024-09-10T19:23:17.434065+0000","last_deep_scrub":"80'15","last_deep_scrub_stamp":"2024-09-10T19:11:41.719310+0000","last_clean_scrub_stamp":"2024-09-10T19:23:17.434065+0000","objects_scrubbed":5,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0063536310000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":302,"reported_epoch":311,"state":"active+clean","last_fresh":"2024-09-10T19:24:34.876703+0000","last_change":"2024-09-10T19:21:48.298990+0000","last_active":"2024-09-10T19:24:34.876703+0000","last_peered":"2024-09-10T19:24:34.876703+0000","last_clean":"2024-09-10T19:24:34.876703+0000","last_became_active":"2024-09-10T19:10:07.685870+0000","last_became_peered":"2024-09-10T19:10:07.685870+0000","last_unstale":"2024-09-10T19:24:34.876703+0000","last_undegraded":"2024-09-10T19:24:34.876703+0000","last_fullsized":"2024-09-10T19:24:34.876703+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:21:48.298913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:21:48.298913+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0075260040000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"303'15","reported_seq":531,"reported_epoch":311,"state":"active+clean","last_fresh":"2024-09-10T19:24:35.905545+0000","last_change":"2024-09-10T19:24:24.917423+0000","last_active":"2024-09-10T19:24:35.905545+0000","last_peered":"2024-09-10T19:24:35.905545+0000","last_clean":"2024-09-10T19:24:35.905545+0000","last_became_active":"2024-09-10T19:24:24.917099+0000","last_became_peered":"2024-09-10T19:24:24.917099+0000","last_unstale":"2024-09-10T19:24:35.905545+0000","last_undegraded":"2024-09-10T19:24:35.905545+0000","last_fullsized":"2024-09-10T19:24:35.905545+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"117'10","last_scrub_stamp":"2024-09-10T19:20:55.377819+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:20:55.377819+0000","objects_scrubbed":1,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0063951320000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":253,"reported_epoch":311,"state":"active+clean","last_fresh":"2024-09-10T19:24:34.881703+0000","last_change":"2024-09-10T19:24:23.901175+0000","last_active":"2024-09-10T19:24:34.881703+0000","last_peered":"2024-09-10T19:24:34.881703+0000","last_clean":"2024-09-10T19:24:34.881703+0000","last_became_active":"2024-09-10T19:24:23.900513+0000","last_became_peered":"2024-09-10T19:24:23.900513+0000","last_unstale":"2024-09-10T19:24:34.881703+0000","last_undegraded":"2024-09-10T19:24:34.881703+0000","last_fullsized":"2024-09-10T19:24:34.881703+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:20:46.625193+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:20:46.625193+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"237'15","reported_seq":365,"reported_epoch":311,"state":"active+clean","last_fresh":"2024-09-10T19:24:34.875614+0000","last_change":"2024-09-10T19:21:57.580717+0000","last_active":"2024-09-10T19:24:34.875614+0000","last_peered":"2024-09-10T19:24:34.875614+0000","last_clean":"2024-09-10T19:24:34.875614+0000","last_became_active":"2024-09-10T19:16:24.859740+0000","last_became_peered":"2024-09-10T19:16:24.859740+0000","last_unstale":"2024-09-10T19:24:34.875614+0000","last_undegraded":"2024-09-10T19:24:34.875614+0000","last_fullsized":"2024-09-10T19:24:34.875614+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"237'15","last_scrub_stamp":"2024-09-10T19:21:57.580644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:21:57.580644+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.11368046,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":259,"reported_epoch":311,"state":"active+clean","last_fresh":"2024-09-10T19:24:34.880102+0000","last_change":"2024-09-10T19:24:23.895505+0000","last_active":"2024-09-10T19:24:34.880102+0000","last_peered":"2024-09-10T19:24:34.880102+0000","last_clean":"2024-09-10T19:24:34.880102+0000","last_became_active":"2024-09-10T19:24:23.895163+0000","last_became_peered":"2024-09-10T19:24:23.895163+0000","last_unstale":"2024-09-10T19:24:34.880102+0000","last_undegraded":"2024-09-10T19:24:34.880102+0000","last_fullsized":"2024-09-10T19:24:34.880102+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:20:48.709474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:20:48.709474+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"297'11","reported_seq":519,"reported_epoch":311,"state":"active+clean","last_fresh":"2024-09-10T19:24:35.905785+0000","last_change":"2024-09-10T19:24:13.318181+0000","last_active":"2024-09-10T19:24:35.905785+0000","last_peered":"2024-09-10T19:24:35.905785+0000","last_clean":"2024-09-10T19:24:35.905785+0000","last_became_active":"2024-09-10T19:24:01.974816+0000","last_became_peered":"2024-09-10T19:24:01.974816+0000","last_unstale":"2024-09-10T19:24:35.905785+0000","last_undegraded":"2024-09-10T19:24:35.905785+0000","last_fullsized":"2024-09-10T19:24:35.905785+0000","mapping_epoch":292,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":293,"parent":"0.0","parent_split_bits":0,"last_scrub":"297'11","last_scrub_stamp":"2024-09-10T19:24:13.318130+0000","last_deep_scrub":"297'11","last_deep_scrub_stamp":"2024-09-10T19:24:13.318130+0000","last_clean_scrub_stamp":"2024-09-10T19:24:13.318130+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:25:25.133218+0000","scrub_duration":0.0029092940000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":325,"reported_epoch":311,"state":"active+clean","last_fresh":"2024-09-10T19:24:37.671977+0000","last_change":"2024-09-10T19:24:37.671881+0000","last_active":"2024-09-10T19:24:37.671977+0000","last_peered":"2024-09-10T19:24:37.671977+0000","last_clean":"2024-09-10T19:24:37.671977+0000","last_became_active":"2024-09-10T19:24:23.903779+0000","last_became_peered":"2024-09-10T19:24:23.903779+0000","last_unstale":"2024-09-10T19:24:37.671977+0000","last_undegraded":"2024-09-10T19:24:37.671977+0000","last_fullsized":"2024-09-10T19:24:37.671977+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:24:37.671797+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:24:37.671797+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:26:07.384408+0000","scrub_duration":0.002698401,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.12","version":"237'19","reported_seq":54,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-09-10T19:24:39.878704+0000","last_change":"2024-09-10T19:24:24.913544+0000","last_active":"2024-09-10T19:24:39.878704+0000","last_peered":"2024-09-10T19:24:39.878704+0000","last_clean":"2024-09-10T19:24:39.878704+0000","last_became_active":"2024-09-10T19:24:24.913130+0000","last_became_peered":"2024-09-10T19:24:24.913130+0000","last_unstale":"2024-09-10T19:24:39.878704+0000","last_undegraded":"2024-09-10T19:24:39.878704+0000","last_fullsized":"2024-09-10T19:24:39.878704+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"237'19","last_scrub_stamp":"2024-09-10T19:21:07.378184+0000","last_deep_scrub":"79'13","last_deep_scrub_stamp":"2024-09-10T19:11:22.072400+0000","last_clean_scrub_stamp":"2024-09-10T19:21:07.378184+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.010862685,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"59'1","reported_seq":329,"reported_epoch":311,"state":"active+clean","last_fresh":"2024-09-10T19:24:34.877626+0000","last_change":"2024-09-10T19:23:56.511161+0000","last_active":"2024-09-10T19:24:34.877626+0000","last_peered":"2024-09-10T19:24:34.877626+0000","last_clean":"2024-09-10T19:24:34.877626+0000","last_became_active":"2024-09-10T19:10:07.683263+0000","last_became_peered":"2024-09-10T19:10:07.683263+0000","last_unstale":"2024-09-10T19:24:34.877626+0000","last_undegraded":"2024-09-10T19:24:34.877626+0000","last_fullsized":"2024-09-10T19:24:34.877626+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'1","last_scrub_stamp":"2024-09-10T19:23:56.511101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:23:56.511101+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:25:10.784807+0000","scrub_duration":0.0015306409999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":362,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":346,"reported_epoch":311,"state":"active+clean","last_fresh":"2024-09-10T19:24:34.877767+0000","last_change":"2024-09-10T19:24:24.924193+0000","last_active":"2024-09-10T19:24:34.877767+0000","last_peered":"2024-09-10T19:24:34.877767+0000","last_clean":"2024-09-10T19:24:34.877767+0000","last_became_active":"2024-09-10T19:24:24.923804+0000","last_became_peered":"2024-09-10T19:24:24.923804+0000","last_unstale":"2024-09-10T19:24:34.877767+0000","last_undegraded":"2024-09-10T19:24:34.877767+0000","last_fullsized":"2024-09-10T19:24:34.877767+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:24:06.468264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:24:06.468264+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:25:31.137084+0000","scrub_duration":0.001379861,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":272,"reported_epoch":311,"state":"active+clean","last_fresh":"2024-09-10T19:24:34.879770+0000","last_change":"2024-09-10T19:24:24.917522+0000","last_active":"2024-09-10T19:24:34.879770+0000","last_peered":"2024-09-10T19:24:34.879770+0000","last_clean":"2024-09-10T19:24:34.879770+0000","last_became_active":"2024-09-10T19:24:24.916105+0000","last_became_peered":"2024-09-10T19:24:24.916105+0000","last_unstale":"2024-09-10T19:24:34.879770+0000","last_undegraded":"2024-09-10T19:24:34.879770+0000","last_fullsized":"2024-09-10T19:24:34.879770+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:21:09.714677+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:21:09.714677+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001865058,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"237'19","reported_seq":394,"reported_epoch":311,"state":"active+clean","last_fresh":"2024-09-10T19:24:34.878858+0000","last_change":"2024-09-10T19:24:10.592443+0000","last_active":"2024-09-10T19:24:34.878858+0000","last_peered":"2024-09-10T19:24:34.878858+0000","last_clean":"2024-09-10T19:24:34.878858+0000","last_became_active":"2024-09-10T19:24:01.975520+0000","last_became_peered":"2024-09-10T19:24:01.975520+0000","last_unstale":"2024-09-10T19:24:34.878858+0000","last_undegraded":"2024-09-10T19:24:34.878858+0000","last_fullsized":"2024-09-10T19:24:34.878858+0000","mapping_epoch":292,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":293,"parent":"0.0","parent_split_bits":0,"last_scrub":"237'19","last_scrub_stamp":"2024-09-10T19:24:10.592332+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:24:10.592332+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:25:23.686728+0000","scrub_duration":0.0027032150000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":54,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-09-10T19:24:39.879520+0000","last_change":"2024-09-10T19:24:24.909841+0000","last_active":"2024-09-10T19:24:39.879520+0000","last_peered":"2024-09-10T19:24:39.879520+0000","last_clean":"2024-09-10T19:24:39.879520+0000","last_became_active":"2024-09-10T19:24:24.909540+0000","last_became_peered":"2024-09-10T19:24:24.909540+0000","last_unstale":"2024-09-10T19:24:39.879520+0000","last_undegraded":"2024-09-10T19:24:39.879520+0000","last_fullsized":"2024-09-10T19:24:39.879520+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:24:07.365451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:11:10.031245+0000","last_clean_scrub_stamp":"2024-09-10T19:24:07.365451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:25:24.636709+0000","scrub_duration":0.001455275,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":30,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-09-10T19:24:39.878941+0000","last_change":"2024-09-10T19:24:01.975862+0000","last_active":"2024-09-10T19:24:39.878941+0000","last_peered":"2024-09-10T19:24:39.878941+0000","last_clean":"2024-09-10T19:24:39.878941+0000","last_became_active":"2024-09-10T19:24:01.975520+0000","last_became_peered":"2024-09-10T19:24:01.975520+0000","last_unstale":"2024-09-10T19:24:39.878941+0000","last_undegraded":"2024-09-10T19:24:39.878941+0000","last_fullsized":"2024-09-10T19:24:39.878941+0000","mapping_epoch":292,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":293,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:23:17.622746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:23:17.622746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:24:45.911205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":313,"reported_epoch":311,"state":"active+clean","last_fresh":"2024-09-10T19:24:36.519454+0000","last_change":"2024-09-10T19:24:36.519411+0000","last_active":"2024-09-10T19:24:36.519454+0000","last_peered":"2024-09-10T19:24:36.519454+0000","last_clean":"2024-09-10T19:24:36.519454+0000","last_became_active":"2024-09-10T19:24:23.907136+0000","last_became_peered":"2024-09-10T19:24:23.907136+0000","last_unstale":"2024-09-10T19:24:36.519454+0000","last_undegraded":"2024-09-10T19:24:36.519454+0000","last_fullsized":"2024-09-10T19:24:36.519454+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:24:36.519345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:24:36.519345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:25:57.167296+0000","scrub_duration":0.0057329820000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"237'10","reported_seq":353,"reported_epoch":311,"state":"active+clean","last_fresh":"2024-09-10T19:24:34.876642+0000","last_change":"2024-09-10T19:24:01.982269+0000","last_active":"2024-09-10T19:24:34.876642+0000","last_peered":"2024-09-10T19:24:34.876642+0000","last_clean":"2024-09-10T19:24:34.876642+0000","last_became_active":"2024-09-10T19:24:01.970758+0000","last_became_peered":"2024-09-10T19:24:01.970758+0000","last_unstale":"2024-09-10T19:24:34.876642+0000","last_undegraded":"2024-09-10T19:24:34.876642+0000","last_fullsized":"2024-09-10T19:24:34.876642+0000","mapping_epoch":292,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":293,"parent":"0.0","parent_split_bits":0,"last_scrub":"237'10","last_scrub_stamp":"2024-09-10T19:22:28.363041+0000","last_deep_scrub":"222'6","last_deep_scrub_stamp":"2024-09-10T19:19:30.328841+0000","last_clean_scrub_stamp":"2024-09-10T19:22:28.363041+0000","objects_scrubbed":2,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0040448380000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":12,"num_read_kb":10,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":332,"reported_epoch":311,"state":"active+clean","last_fresh":"2024-09-10T19:24:34.877228+0000","last_change":"2024-09-10T19:24:01.978852+0000","last_active":"2024-09-10T19:24:34.877228+0000","last_peered":"2024-09-10T19:24:34.877228+0000","last_clean":"2024-09-10T19:24:34.877228+0000","last_became_active":"2024-09-10T19:24:01.978534+0000","last_became_peered":"2024-09-10T19:24:01.978534+0000","last_unstale":"2024-09-10T19:24:34.877228+0000","last_undegraded":"2024-09-10T19:24:34.877228+0000","last_fullsized":"2024-09-10T19:24:34.877228+0000","mapping_epoch":292,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":293,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:23:19.717358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:23:19.717358+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:24:41.865008+0000","scrub_duration":0.020480847,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,1],"acting":[5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":318,"reported_epoch":311,"state":"active+clean","last_fresh":"2024-09-10T19:24:34.877608+0000","last_change":"2024-09-10T19:23:24.462523+0000","last_active":"2024-09-10T19:24:34.877608+0000","last_peered":"2024-09-10T19:24:34.877608+0000","last_clean":"2024-09-10T19:24:34.877608+0000","last_became_active":"2024-09-10T19:10:11.698536+0000","last_became_peered":"2024-09-10T19:10:11.698536+0000","last_unstale":"2024-09-10T19:24:34.877608+0000","last_undegraded":"2024-09-10T19:24:34.877608+0000","last_fullsized":"2024-09-10T19:24:34.877608+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:23:24.462473+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:11:31.837944+0000","last_clean_scrub_stamp":"2024-09-10T19:23:24.462473+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.010077553,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":326,"reported_epoch":311,"state":"active+clean","last_fresh":"2024-09-10T19:24:34.879063+0000","last_change":"2024-09-10T19:22:31.278309+0000","last_active":"2024-09-10T19:24:34.879063+0000","last_peered":"2024-09-10T19:24:34.879063+0000","last_clean":"2024-09-10T19:24:34.879063+0000","last_became_active":"2024-09-10T19:16:24.852356+0000","last_became_peered":"2024-09-10T19:16:24.852356+0000","last_unstale":"2024-09-10T19:24:34.879063+0000","last_undegraded":"2024-09-10T19:24:34.879063+0000","last_fullsized":"2024-09-10T19:24:34.879063+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:22:31.278179+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:22:31.278179+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0036472750000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,3],"acting":[7,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.11","version":"237'19","reported_seq":92,"reported_epoch":311,"state":"active+clean","last_fresh":"2024-09-10T19:24:34.876566+0000","last_change":"2024-09-10T19:22:22.464814+0000","last_active":"2024-09-10T19:24:34.876566+0000","last_peered":"2024-09-10T19:24:34.876566+0000","last_clean":"2024-09-10T19:24:34.876566+0000","last_became_active":"2024-09-10T19:21:04.169444+0000","last_became_peered":"2024-09-10T19:21:04.169444+0000","last_unstale":"2024-09-10T19:24:34.876566+0000","last_undegraded":"2024-09-10T19:24:34.876566+0000","last_fullsized":"2024-09-10T19:24:34.876566+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"237'19","last_scrub_stamp":"2024-09-10T19:22:22.464729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:22:22.464729+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0058949659999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":339,"reported_epoch":311,"state":"active+clean","last_fresh":"2024-09-10T19:24:34.878533+0000","last_change":"2024-09-10T19:24:12.505521+0000","last_active":"2024-09-10T19:24:34.878533+0000","last_peered":"2024-09-10T19:24:34.878533+0000","last_clean":"2024-09-10T19:24:34.878533+0000","last_became_active":"2024-09-10T19:24:01.974565+0000","last_became_peered":"2024-09-10T19:24:01.974565+0000","last_unstale":"2024-09-10T19:24:34.878533+0000","last_undegraded":"2024-09-10T19:24:34.878533+0000","last_fullsized":"2024-09-10T19:24:34.878533+0000","mapping_epoch":292,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":293,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:24:12.505459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:13:05.876655+0000","last_clean_scrub_stamp":"2024-09-10T19:24:12.505459+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:25:16.858325+0000","scrub_duration":0.0014517989999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":308,"reported_epoch":311,"state":"active+clean","last_fresh":"2024-09-10T19:24:34.877119+0000","last_change":"2024-09-10T19:23:54.472121+0000","last_active":"2024-09-10T19:24:34.877119+0000","last_peered":"2024-09-10T19:24:34.877119+0000","last_clean":"2024-09-10T19:24:34.877119+0000","last_became_active":"2024-09-10T19:10:11.699013+0000","last_became_peered":"2024-09-10T19:10:11.699013+0000","last_unstale":"2024-09-10T19:24:34.877119+0000","last_undegraded":"2024-09-10T19:24:34.877119+0000","last_fullsized":"2024-09-10T19:24:34.877119+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:23:54.472020+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:11:57.112092+0000","last_clean_scrub_stamp":"2024-09-10T19:23:54.472020+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:25:00.811901+0000","scrub_duration":0.002132419,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":313,"reported_epoch":311,"state":"active+clean","last_fresh":"2024-09-10T19:24:35.541740+0000","last_change":"2024-09-10T19:24:35.541687+0000","last_active":"2024-09-10T19:24:35.541740+0000","last_peered":"2024-09-10T19:24:35.541740+0000","last_clean":"2024-09-10T19:24:35.541740+0000","last_became_active":"2024-09-10T19:24:23.896454+0000","last_became_peered":"2024-09-10T19:24:23.896454+0000","last_unstale":"2024-09-10T19:24:35.541740+0000","last_undegraded":"2024-09-10T19:24:35.541740+0000","last_fullsized":"2024-09-10T19:24:35.541740+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:24:35.541618+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:24:35.541618+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:25:52.657891+0000","scrub_duration":0.0020099889999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"80'2","reported_seq":332,"reported_epoch":311,"state":"active+clean","last_fresh":"2024-09-10T19:24:34.882612+0000","last_change":"2024-09-10T19:23:57.565620+0000","last_active":"2024-09-10T19:24:34.882612+0000","last_peered":"2024-09-10T19:24:34.882612+0000","last_clean":"2024-09-10T19:24:34.882612+0000","last_became_active":"2024-09-10T19:16:24.851220+0000","last_became_peered":"2024-09-10T19:16:24.851220+0000","last_unstale":"2024-09-10T19:24:34.882612+0000","last_undegraded":"2024-09-10T19:24:34.882612+0000","last_fullsized":"2024-09-10T19:24:34.882612+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"80'2","last_scrub_stamp":"2024-09-10T19:23:57.565527+0000","last_deep_scrub":"80'2","last_deep_scrub_stamp":"2024-09-10T19:11:30.272043+0000","last_clean_scrub_stamp":"2024-09-10T19:23:57.565527+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:25:22.117867+0000","scrub_duration":0.0015365120000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":369,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":369,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,3],"acting":[7,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":319,"reported_epoch":311,"state":"active+clean","last_fresh":"2024-09-10T19:24:34.880325+0000","last_change":"2024-09-10T19:24:23.907057+0000","last_active":"2024-09-10T19:24:34.880325+0000","last_peered":"2024-09-10T19:24:34.880325+0000","last_clean":"2024-09-10T19:24:34.880325+0000","last_became_active":"2024-09-10T19:24:23.906812+0000","last_became_peered":"2024-09-10T19:24:23.906812+0000","last_unstale":"2024-09-10T19:24:34.880325+0000","last_undegraded":"2024-09-10T19:24:34.880325+0000","last_fullsized":"2024-09-10T19:24:34.880325+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:19:19.581564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:19:19.581564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.01041663,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,0],"acting":[5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1e","version":"237'25","reported_seq":292,"reported_epoch":311,"state":"active+clean","last_fresh":"2024-09-10T19:24:34.880990+0000","last_change":"2024-09-10T19:24:23.900468+0000","last_active":"2024-09-10T19:24:34.880990+0000","last_peered":"2024-09-10T19:24:34.880990+0000","last_clean":"2024-09-10T19:24:34.880990+0000","last_became_active":"2024-09-10T19:24:23.899507+0000","last_became_peered":"2024-09-10T19:24:23.899507+0000","last_unstale":"2024-09-10T19:24:34.880990+0000","last_undegraded":"2024-09-10T19:24:34.880990+0000","last_fullsized":"2024-09-10T19:24:34.880990+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"237'25","last_scrub_stamp":"2024-09-10T19:20:44.696751+0000","last_deep_scrub":"80'15","last_deep_scrub_stamp":"2024-09-10T19:11:25.883919+0000","last_clean_scrub_stamp":"2024-09-10T19:20:44.696751+0000","objects_scrubbed":5,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0040337309999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":24,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1f","version":"66'8","reported_seq":506,"reported_epoch":311,"state":"active+clean","last_fresh":"2024-09-10T19:24:35.906650+0000","last_change":"2024-09-10T19:23:09.642484+0000","last_active":"2024-09-10T19:24:35.906650+0000","last_peered":"2024-09-10T19:24:35.906650+0000","last_clean":"2024-09-10T19:24:35.906650+0000","last_became_active":"2024-09-10T19:10:11.695427+0000","last_became_peered":"2024-09-10T19:10:11.695427+0000","last_unstale":"2024-09-10T19:24:35.906650+0000","last_undegraded":"2024-09-10T19:24:35.906650+0000","last_fullsized":"2024-09-10T19:24:35.906650+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'8","last_scrub_stamp":"2024-09-10T19:23:09.642403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:23:09.642403+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0047206909999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":30,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-09-10T19:24:39.879658+0000","last_change":"2024-09-10T19:24:01.980713+0000","last_active":"2024-09-10T19:24:39.879658+0000","last_peered":"2024-09-10T19:24:39.879658+0000","last_clean":"2024-09-10T19:24:39.879658+0000","last_became_active":"2024-09-10T19:24:01.980391+0000","last_became_peered":"2024-09-10T19:24:01.980391+0000","last_unstale":"2024-09-10T19:24:39.879658+0000","last_undegraded":"2024-09-10T19:24:39.879658+0000","last_fullsized":"2024-09-10T19:24:39.879658+0000","mapping_epoch":292,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":293,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:22:38.725165+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:22:38.725165+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0047886710000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":319,"reported_epoch":311,"state":"active+clean","last_fresh":"2024-09-10T19:24:34.877756+0000","last_change":"2024-09-10T19:24:23.906138+0000","last_active":"2024-09-10T19:24:34.877756+0000","last_peered":"2024-09-10T19:24:34.877756+0000","last_clean":"2024-09-10T19:24:34.877756+0000","last_became_active":"2024-09-10T19:24:23.905869+0000","last_became_peered":"2024-09-10T19:24:23.905869+0000","last_unstale":"2024-09-10T19:24:34.877756+0000","last_undegraded":"2024-09-10T19:24:34.877756+0000","last_fullsized":"2024-09-10T19:24:34.877756+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:21:44.676477+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:21:44.676477+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0017387360000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"237'24","reported_seq":425,"reported_epoch":311,"state":"active+clean","last_fresh":"2024-09-10T19:24:34.878461+0000","last_change":"2024-09-10T19:24:04.569983+0000","last_active":"2024-09-10T19:24:34.878461+0000","last_peered":"2024-09-10T19:24:34.878461+0000","last_clean":"2024-09-10T19:24:34.878461+0000","last_became_active":"2024-09-10T19:24:01.974988+0000","last_became_peered":"2024-09-10T19:24:01.974988+0000","last_unstale":"2024-09-10T19:24:34.878461+0000","last_undegraded":"2024-09-10T19:24:34.878461+0000","last_fullsized":"2024-09-10T19:24:34.878461+0000","mapping_epoch":292,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":293,"parent":"0.0","parent_split_bits":0,"last_scrub":"237'24","last_scrub_stamp":"2024-09-10T19:24:04.569927+0000","last_deep_scrub":"80'16","last_deep_scrub_stamp":"2024-09-10T19:11:40.853611+0000","last_clean_scrub_stamp":"2024-09-10T19:24:04.569927+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:25:15.306779+0000","scrub_duration":0.0019487459999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":247,"reported_epoch":311,"state":"active+clean","last_fresh":"2024-09-10T19:24:34.880905+0000","last_change":"2024-09-10T19:24:24.901089+0000","last_active":"2024-09-10T19:24:34.880905+0000","last_peered":"2024-09-10T19:24:34.880905+0000","last_clean":"2024-09-10T19:24:34.880905+0000","last_became_active":"2024-09-10T19:24:24.899153+0000","last_became_peered":"2024-09-10T19:24:24.899153+0000","last_unstale":"2024-09-10T19:24:34.880905+0000","last_undegraded":"2024-09-10T19:24:34.880905+0000","last_fullsized":"2024-09-10T19:24:34.880905+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:24:04.653486+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:11:28.822327+0000","last_clean_scrub_stamp":"2024-09-10T19:24:04.653486+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:25:16.272696+0000","scrub_duration":0.0022130729999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":315,"reported_epoch":311,"state":"active+clean","last_fresh":"2024-09-10T19:24:34.880673+0000","last_change":"2024-09-10T19:24:15.621642+0000","last_active":"2024-09-10T19:24:34.880673+0000","last_peered":"2024-09-10T19:24:34.880673+0000","last_clean":"2024-09-10T19:24:34.880673+0000","last_became_active":"2024-09-10T19:10:14.053619+0000","last_became_peered":"2024-09-10T19:10:14.053619+0000","last_unstale":"2024-09-10T19:24:34.880673+0000","last_undegraded":"2024-09-10T19:24:34.880673+0000","last_fullsized":"2024-09-10T19:24:34.880673+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:24:15.621548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:24:15.621548+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:25:23.591706+0000","scrub_duration":0.0015495050000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":322,"reported_epoch":311,"state":"active+clean","last_fresh":"2024-09-10T19:24:34.879540+0000","last_change":"2024-09-10T19:24:01.436167+0000","last_active":"2024-09-10T19:24:34.879540+0000","last_peered":"2024-09-10T19:24:34.879540+0000","last_clean":"2024-09-10T19:24:34.879540+0000","last_became_active":"2024-09-10T19:10:07.685997+0000","last_became_peered":"2024-09-10T19:10:07.685997+0000","last_unstale":"2024-09-10T19:24:34.879540+0000","last_undegraded":"2024-09-10T19:24:34.879540+0000","last_fullsized":"2024-09-10T19:24:34.879540+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:24:01.436116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:24:01.436116+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:25:14.136209+0000","scrub_duration":0.0012242259999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1c","version":"237'34","reported_seq":60,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-09-10T19:24:39.879024+0000","last_change":"2024-09-10T19:24:24.911052+0000","last_active":"2024-09-10T19:24:39.879024+0000","last_peered":"2024-09-10T19:24:39.879024+0000","last_clean":"2024-09-10T19:24:39.879024+0000","last_became_active":"2024-09-10T19:24:24.910577+0000","last_became_peered":"2024-09-10T19:24:24.910577+0000","last_unstale":"2024-09-10T19:24:39.879024+0000","last_undegraded":"2024-09-10T19:24:39.879024+0000","last_fullsized":"2024-09-10T19:24:39.879024+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"237'34","last_scrub_stamp":"2024-09-10T19:20:40.452293+0000","last_deep_scrub":"78'22","last_deep_scrub_stamp":"2024-09-10T19:11:21.118937+0000","last_clean_scrub_stamp":"2024-09-10T19:20:40.452293+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0025454940000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":54,"reported_epoch":312,"state":"active+clean","last_fresh":"2024-09-10T19:24:39.880378+0000","last_change":"2024-09-10T19:24:35.438665+0000","last_active":"2024-09-10T19:24:39.880378+0000","last_peered":"2024-09-10T19:24:39.880378+0000","last_clean":"2024-09-10T19:24:39.880378+0000","last_became_active":"2024-09-10T19:24:24.887205+0000","last_became_peered":"2024-09-10T19:24:24.887205+0000","last_unstale":"2024-09-10T19:24:39.880378+0000","last_undegraded":"2024-09-10T19:24:39.880378+0000","last_fullsized":"2024-09-10T19:24:39.880378+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:24:35.438586+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:24:35.438586+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:25:42.686433+0000","scrub_duration":0.0036432579999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":299,"reported_epoch":311,"state":"active+clean","last_fresh":"2024-09-10T19:24:34.878217+0000","last_change":"2024-09-10T19:23:59.456853+0000","last_active":"2024-09-10T19:24:34.878217+0000","last_peered":"2024-09-10T19:24:34.878217+0000","last_clean":"2024-09-10T19:24:34.878217+0000","last_became_active":"2024-09-10T19:10:14.053698+0000","last_became_peered":"2024-09-10T19:10:14.053698+0000","last_unstale":"2024-09-10T19:24:34.878217+0000","last_undegraded":"2024-09-10T19:24:34.878217+0000","last_fullsized":"2024-09-10T19:24:34.878217+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:21:57.736006+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:21:57.736006+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0017464049999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":312,"reported_epoch":311,"state":"active+clean","last_fresh":"2024-09-10T19:24:34.877955+0000","last_change":"2024-09-10T19:23:58.467298+0000","last_active":"2024-09-10T19:24:34.877955+0000","last_peered":"2024-09-10T19:24:34.877955+0000","last_clean":"2024-09-10T19:24:34.877955+0000","last_became_active":"2024-09-10T19:10:07.682811+0000","last_became_peered":"2024-09-10T19:10:07.682811+0000","last_unstale":"2024-09-10T19:24:34.877955+0000","last_undegraded":"2024-09-10T19:24:34.877955+0000","last_fullsized":"2024-09-10T19:24:34.877955+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:23:58.467240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:23:58.467240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:25:17.579970+0000","scrub_duration":0.001352088,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1d","version":"250'23","reported_seq":388,"reported_epoch":311,"state":"active+clean","last_fresh":"2024-09-10T19:24:34.877264+0000","last_change":"2024-09-10T19:24:02.001489+0000","last_active":"2024-09-10T19:24:34.877264+0000","last_peered":"2024-09-10T19:24:34.877264+0000","last_clean":"2024-09-10T19:24:34.877264+0000","last_became_active":"2024-09-10T19:24:01.976664+0000","last_became_peered":"2024-09-10T19:24:01.976664+0000","last_unstale":"2024-09-10T19:24:34.877264+0000","last_undegraded":"2024-09-10T19:24:34.877264+0000","last_fullsized":"2024-09-10T19:24:34.877264+0000","mapping_epoch":292,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":293,"parent":"0.0","parent_split_bits":0,"last_scrub":"250'23","last_scrub_stamp":"2024-09-10T19:23:26.352362+0000","last_deep_scrub":"250'23","last_deep_scrub_stamp":"2024-09-10T19:20:55.582173+0000","last_clean_scrub_stamp":"2024-09-10T19:23:26.352362+0000","objects_scrubbed":5,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:24:45.931194+0000","scrub_duration":0.0017939849999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":327,"reported_epoch":311,"state":"active+clean","last_fresh":"2024-09-10T19:24:34.881505+0000","last_change":"2024-09-10T19:23:54.495283+0000","last_active":"2024-09-10T19:24:34.881505+0000","last_peered":"2024-09-10T19:24:34.881505+0000","last_clean":"2024-09-10T19:24:34.881505+0000","last_became_active":"2024-09-10T19:16:24.843013+0000","last_became_peered":"2024-09-10T19:16:24.843013+0000","last_unstale":"2024-09-10T19:24:34.881505+0000","last_undegraded":"2024-09-10T19:24:34.881505+0000","last_fullsized":"2024-09-10T19:24:34.881505+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:23:54.495223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:23:54.495223+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:25:23.347070+0000","scrub_duration":0.0013286420000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":10,"num_pg":16,"stat_sum":{"num_bytes":2139422720,"num_objects":32645,"num_object_clones":0,"num_object_copies":97935,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32645,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":32645,"num_write_kb":2089280,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4680,"num_bytes_recovered":306708480,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":6436945920,"data_stored":6436945920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":32645,"ondisk_log_size":32645,"up":48,"acting":48,"num_store_stats":7},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":382,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":1528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":88,"ondisk_log_size":88,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":177,"num_object_clones":0,"num_object_copies":531,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":177,"num_whiteouts":0,"num_read":1117,"num_read_kb":874,"num_write":703,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":284,"num_bytes_recovered":5666,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":458752,"data_stored":12124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":759,"ondisk_log_size":759,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":49,"num_read_kb":44,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":778,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1613,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":9,"ondisk_log_size":9,"up":9,"acting":9,"num_store_stats":7},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1328,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":27,"num_read_kb":27,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":3984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4,"ondisk_log_size":4,"up":96,"acting":96,"num_store_stats":8},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":697708,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":21,"num_read_kb":40,"num_write":22,"num_write_kb":296,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":107340,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":869,"ondisk_log_size":869,"up":33,"acting":33,"num_store_stats":7}],"osd_stats":[{"osd":7,"up_from":54,"seq":231928234163,"num_pgs":70,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1519724,"kb_used_data":623268,"kb_used_omap":1,"kb_used_meta":896446,"kb_avail":92225428,"statfs":{"total":95995035648,"available":94438838272,"internally_reserved":0,"allocated":638226432,"data_stored":637844090,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":917961048},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 19:21:08 2024","interfaces":[{"interface":"back","average":{"1min":0.61299999999999999,"5min":1.046,"15min":1.113},"min":{"1min":0.215,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":2.3620000000000001,"5min":8.3770000000000007,"15min":35.551000000000002},"last":0.48199999999999998},{"interface":"front","average":{"1min":1.879,"5min":1.2789999999999999,"15min":1.131},"min":{"1min":0.24399999999999999,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":5.0140000000000002,"5min":8.1440000000000001,"15min":35.503999999999998},"last":0.378}]},{"osd":1,"last update":"Tue Sep 10 19:23:46 2024","interfaces":[{"interface":"back","average":{"1min":1.036,"5min":1.036,"15min":1.036},"min":{"1min":0.23899999999999999,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":4.2240000000000002,"5min":4.2240000000000002,"15min":4.2240000000000002},"last":0.50800000000000001},{"interface":"front","average":{"1min":0.83699999999999997,"5min":0.83699999999999997,"15min":0.83699999999999997},"min":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":3.5459999999999998,"5min":3.5459999999999998,"15min":3.5459999999999998},"last":0.71399999999999997}]},{"osd":2,"last update":"Tue Sep 10 19:24:10 2024","interfaces":[{"interface":"back","average":{"1min":0.51900000000000002,"5min":1.03,"15min":0.92700000000000005},"min":{"1min":0.36399999999999999,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":1.2450000000000001,"5min":4.0190000000000001,"15min":31.940999999999999},"last":0.45200000000000001},{"interface":"front","average":{"1min":0.47399999999999998,"5min":0.76300000000000001,"15min":0.98399999999999999},"min":{"1min":0.312,"5min":0.26300000000000001,"15min":0.24099999999999999},"max":{"1min":0.64600000000000002,"5min":4.4139999999999997,"15min":25.056000000000001},"last":0.77600000000000002}]},{"osd":3,"last update":"Tue Sep 10 19:24:10 2024","interfaces":[{"interface":"back","average":{"1min":0.49199999999999999,"5min":1.0489999999999999,"15min":1.29},"min":{"1min":0.33300000000000002,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":1.2729999999999999,"5min":4.6070000000000002,"15min":35.277999999999999},"last":0.42499999999999999},{"interface":"front","average":{"1min":0.48499999999999999,"5min":0.81399999999999995,"15min":1.2350000000000001},"min":{"1min":0.34200000000000003,"5min":0.27400000000000002,"15min":0.25700000000000001},"max":{"1min":0.629,"5min":5.0119999999999996,"15min":36.512999999999998},"last":0.75}]},{"osd":4,"last update":"Tue Sep 10 19:24:10 2024","interfaces":[{"interface":"back","average":{"1min":13.311,"5min":4.21,"15min":2.0129999999999999},"min":{"1min":0.249,"5min":0.222,"15min":0.222},"max":{"1min":231.923,"5min":231.923,"15min":231.923},"last":1.1799999999999999},{"interface":"front","average":{"1min":13.355,"5min":3.9500000000000002,"15min":1.7769999999999999},"min":{"1min":0.30099999999999999,"5min":0.24399999999999999,"15min":0.23000000000000001},"max":{"1min":231.322,"5min":231.322,"15min":231.322},"last":0.66800000000000004}]},{"osd":5,"last update":"Tue Sep 10 19:24:10 2024","interfaces":[{"interface":"back","average":{"1min":0.46300000000000002,"5min":1.9670000000000001,"15min":1.4410000000000001},"min":{"1min":0.24099999999999999,"5min":0.24099999999999999,"15min":0.23699999999999999},"max":{"1min":0.89200000000000002,"5min":67.382999999999996,"15min":67.382999999999996},"last":1.218},{"interface":"front","average":{"1min":0.48399999999999999,"5min":2.6429999999999998,"15min":1.6160000000000001},"min":{"1min":0.253,"5min":0.253,"15min":0.253},"max":{"1min":1.05,"5min":67.75,"15min":67.75},"last":1.7350000000000001}]},{"osd":6,"last update":"Tue Sep 10 19:24:10 2024","interfaces":[{"interface":"back","average":{"1min":0.60499999999999998,"5min":3.3860000000000001,"15min":1.792},"min":{"1min":0.28000000000000003,"5min":0.28000000000000003,"15min":0.26200000000000001},"max":{"1min":2.609,"5min":181.53800000000001,"15min":181.53800000000001},"last":4.5940000000000003},{"interface":"front","average":{"1min":0.51600000000000001,"5min":3.153,"15min":1.5840000000000001},"min":{"1min":0.30299999999999999,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":1.427,"5min":182.05600000000001,"15min":182.05600000000001},"last":1.639}]}]},{"osd":6,"up_from":47,"seq":201863463096,"num_pgs":63,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1759480,"kb_used_data":864688,"kb_used_omap":1,"kb_used_meta":894782,"kb_avail":91985672,"statfs":{"total":95995035648,"available":94193328128,"internally_reserved":0,"allocated":885440512,"data_stored":885046970,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":916257112},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 19:20:51 2024","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":0.69899999999999995,"15min":1.1279999999999999},"min":{"1min":0.23100000000000001,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":2.2010000000000001,"5min":5.2350000000000003,"15min":112.988},"last":0.27200000000000002},{"interface":"front","average":{"1min":0.59799999999999998,"5min":0.90500000000000003,"15min":1.1990000000000001},"min":{"1min":0.221,"5min":0.221,"15min":0.221},"max":{"1min":2.0009999999999999,"5min":5.274,"15min":113.18300000000001},"last":0.85399999999999998}]},{"osd":1,"last update":"Tue Sep 10 19:23:47 2024","interfaces":[{"interface":"back","average":{"1min":0.72299999999999998,"5min":0.72299999999999998,"15min":0.72299999999999998},"min":{"1min":0.23799999999999999,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":4.9359999999999999,"5min":4.9359999999999999,"15min":4.9359999999999999},"last":0.30399999999999999},{"interface":"front","average":{"1min":0.753,"5min":0.753,"15min":0.753},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":3.1040000000000001,"5min":3.1040000000000001,"15min":3.1040000000000001},"last":0.90400000000000003}]},{"osd":2,"last update":"Tue Sep 10 19:23:56 2024","interfaces":[{"interface":"back","average":{"1min":0.72199999999999998,"5min":0.79600000000000004,"15min":1.141},"min":{"1min":0.25700000000000001,"5min":0.23799999999999999,"15min":0.20399999999999999},"max":{"1min":3.1680000000000001,"5min":3.6099999999999999,"15min":112.864},"last":0.998},{"interface":"front","average":{"1min":0.52600000000000002,"5min":0.89900000000000002,"15min":1.236},"min":{"1min":0.28399999999999997,"5min":0.25600000000000001,"15min":0.254},"max":{"1min":1.476,"5min":5.8300000000000001,"15min":112.822},"last":0.69199999999999995}]},{"osd":3,"last update":"Tue Sep 10 19:23:56 2024","interfaces":[{"interface":"back","average":{"1min":0.71999999999999997,"5min":0.87,"15min":1.4239999999999999},"min":{"1min":0.34699999999999998,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":4.9029999999999996,"5min":6.4870000000000001,"15min":125.56399999999999},"last":0.33200000000000002},{"interface":"front","average":{"1min":0.90300000000000002,"5min":0.84499999999999997,"15min":1.4510000000000001},"min":{"1min":0.247,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":3.141,"5min":3.794,"15min":125.47},"last":0.95899999999999996}]},{"osd":4,"last update":"Tue Sep 10 19:23:56 2024","interfaces":[{"interface":"back","average":{"1min":0.65600000000000003,"5min":1.355,"15min":0.93100000000000005},"min":{"1min":0.313,"5min":0.253,"15min":0.20699999999999999},"max":{"1min":2.5249999999999999,"5min":6.5330000000000004,"15min":6.9269999999999996},"last":0.73199999999999998},{"interface":"front","average":{"1min":1.2310000000000001,"5min":1.294,"15min":0.878},"min":{"1min":0.33200000000000002,"5min":0.24099999999999999,"15min":0.22500000000000001},"max":{"1min":8.6419999999999995,"5min":8.6419999999999995,"15min":8.6419999999999995},"last":0.54400000000000004}]},{"osd":5,"last update":"Tue Sep 10 19:23:56 2024","interfaces":[{"interface":"back","average":{"1min":1.1020000000000001,"5min":1.4039999999999999,"15min":1.147},"min":{"1min":0.28100000000000003,"5min":0.28100000000000003,"15min":0.248},"max":{"1min":6.3730000000000002,"5min":6.3730000000000002,"15min":44.765000000000001},"last":0.76400000000000001},{"interface":"front","average":{"1min":1.1279999999999999,"5min":1.516,"15min":1.2070000000000001},"min":{"1min":0.35599999999999998,"5min":0.28399999999999997,"15min":0.27500000000000002},"max":{"1min":6.3710000000000004,"5min":7.1369999999999996,"15min":43.036000000000001},"last":1.2050000000000001}]},{"osd":7,"last update":"Tue Sep 10 19:24:24 2024","interfaces":[{"interface":"back","average":{"1min":0.75700000000000001,"5min":1.0149999999999999,"15min":0.86799999999999999},"min":{"1min":0.28100000000000003,"5min":0.28100000000000003,"15min":0.27500000000000002},"max":{"1min":6.0099999999999998,"5min":6.0099999999999998,"15min":8.9390000000000001},"last":0.93300000000000005},{"interface":"front","average":{"1min":0.59899999999999998,"5min":1.367,"15min":1.0589999999999999},"min":{"1min":0.317,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":2.7730000000000001,"5min":6.9720000000000004,"15min":10.298},"last":0.48799999999999999}]}]},{"osd":5,"up_from":39,"seq":167503724733,"num_pgs":65,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1761452,"kb_used_data":855204,"kb_used_omap":1,"kb_used_meta":906238,"kb_avail":91983700,"statfs":{"total":95995035648,"available":94191308800,"internally_reserved":0,"allocated":875728896,"data_stored":875348094,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":927988056},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 19:21:23 2024","interfaces":[{"interface":"back","average":{"1min":1.1579999999999999,"5min":1.9570000000000001,"15min":1.1060000000000001},"min":{"1min":0.19600000000000001,"5min":0.17999999999999999,"15min":0.17999999999999999},"max":{"1min":3.9009999999999998,"5min":73.933000000000007,"15min":73.933000000000007},"last":0.504},{"interface":"front","average":{"1min":1.286,"5min":1.821,"15min":1.024},"min":{"1min":0.23100000000000001,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":3.9969999999999999,"5min":73.769999999999996,"15min":73.769999999999996},"last":3.8580000000000001}]},{"osd":1,"last update":"Tue Sep 10 19:23:42 2024","interfaces":[{"interface":"back","average":{"1min":0.99099999999999999,"5min":0.99099999999999999,"15min":0.99099999999999999},"min":{"1min":0.26300000000000001,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":3.4420000000000002,"5min":3.4420000000000002,"15min":3.4420000000000002},"last":0.33500000000000002},{"interface":"front","average":{"1min":1.0640000000000001,"5min":1.0640000000000001,"15min":1.0640000000000001},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":2.8460000000000001,"5min":2.8460000000000001,"15min":2.8460000000000001},"last":0.45400000000000001}]},{"osd":2,"last update":"Tue Sep 10 19:24:24 2024","interfaces":[{"interface":"back","average":{"1min":0.50600000000000001,"5min":1.1459999999999999,"15min":0.96099999999999997},"min":{"1min":0.35699999999999998,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":0.71499999999999997,"5min":8.6920000000000002,"15min":9.9399999999999995},"last":0.48099999999999998},{"interface":"front","average":{"1min":0.46300000000000002,"5min":0.96599999999999997,"15min":0.92800000000000005},"min":{"1min":0.32000000000000001,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":0.59299999999999997,"5min":4.0910000000000002,"15min":6.3789999999999996},"last":3.8319999999999999}]},{"osd":3,"last update":"Tue Sep 10 19:24:24 2024","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":1.4330000000000001,"15min":1.05},"min":{"1min":0.377,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":1.738,"5min":26.355,"15min":26.355},"last":0.42299999999999999},{"interface":"front","average":{"1min":0.504,"5min":1.294,"15min":1.014},"min":{"1min":0.28999999999999998,"5min":0.246,"15min":0.246},"max":{"1min":0.69499999999999995,"5min":26.327000000000002,"15min":26.327000000000002},"last":3.7530000000000001}]},{"osd":4,"last update":"Tue Sep 10 19:24:24 2024","interfaces":[{"interface":"back","average":{"1min":0.38100000000000001,"5min":2.6970000000000001,"15min":1.488},"min":{"1min":0.28199999999999997,"5min":0.253,"15min":0.20699999999999999},"max":{"1min":0.57599999999999996,"5min":128.08099999999999,"15min":128.08099999999999},"last":3.6909999999999998},{"interface":"front","average":{"1min":0.48599999999999999,"5min":2.8730000000000002,"15min":1.5720000000000001},"min":{"1min":0.217,"5min":0.217,"15min":0.217},"max":{"1min":1.704,"5min":126.321,"15min":126.321},"last":1.2709999999999999}]},{"osd":6,"last update":"Tue Sep 10 19:23:55 2024","interfaces":[{"interface":"back","average":{"1min":1.123,"5min":2.702,"15min":1.5289999999999999},"min":{"1min":0.36599999999999999,"5min":0.28399999999999997,"15min":0.223},"max":{"1min":2.9239999999999999,"5min":111.328,"15min":111.328},"last":1.0880000000000001},{"interface":"front","average":{"1min":1.151,"5min":2.6299999999999999,"15min":1.468},"min":{"1min":0.27500000000000002,"5min":0.27500000000000002,"15min":0.25},"max":{"1min":4.6989999999999998,"5min":111.705,"15min":111.705},"last":3.7850000000000001}]},{"osd":7,"last update":"Tue Sep 10 19:24:20 2024","interfaces":[{"interface":"back","average":{"1min":0.55000000000000004,"5min":1.4410000000000001,"15min":1.1080000000000001},"min":{"1min":0.28599999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":2.335,"5min":6.7869999999999999,"15min":6.7869999999999999},"last":3.8090000000000002},{"interface":"front","average":{"1min":0.52200000000000002,"5min":1.456,"15min":1.071},"min":{"1min":0.32700000000000001,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":2.0219999999999998,"5min":9.4909999999999997,"15min":9.4909999999999997},"last":3.6539999999999999}]}]},{"osd":4,"up_from":32,"seq":137438953665,"num_pgs":70,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2255216,"kb_used_data":1358504,"kb_used_omap":1,"kb_used_meta":896702,"kb_avail":91489936,"statfs":{"total":95995035648,"available":93685694464,"internally_reserved":0,"allocated":1391108096,"data_stored":1390721509,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":918223192},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 19:21:10 2024","interfaces":[{"interface":"back","average":{"1min":1.0209999999999999,"5min":1.224,"15min":0.97999999999999998},"min":{"1min":0.25700000000000001,"5min":0.187,"15min":0.187},"max":{"1min":5.1420000000000003,"5min":24.113,"15min":32.533000000000001},"last":1.669},{"interface":"front","average":{"1min":0.82799999999999996,"5min":1.1739999999999999,"15min":0.98099999999999998},"min":{"1min":0.20399999999999999,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":3.3100000000000001,"5min":24.132999999999999,"15min":32.512999999999998},"last":1.599}]},{"osd":1,"last update":"Tue Sep 10 19:23:44 2024","interfaces":[{"interface":"back","average":{"1min":0.70499999999999996,"5min":0.70499999999999996,"15min":0.70499999999999996},"min":{"1min":0.29499999999999998,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":1.516,"5min":1.516,"15min":1.516},"last":1.619},{"interface":"front","average":{"1min":0.60699999999999998,"5min":0.60699999999999998,"15min":0.60699999999999998},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":1.6779999999999999,"5min":1.6779999999999999,"15min":1.6779999999999999},"last":0.90200000000000002}]},{"osd":2,"last update":"Tue Sep 10 19:24:20 2024","interfaces":[{"interface":"back","average":{"1min":0.495,"5min":0.81899999999999995,"15min":0.78000000000000003},"min":{"1min":0.34200000000000003,"5min":0.25600000000000001,"15min":0.214},"max":{"1min":0.628,"5min":5.048,"15min":6.8869999999999996},"last":1.6399999999999999},{"interface":"front","average":{"1min":0.504,"5min":0.85399999999999998,"15min":0.78700000000000003},"min":{"1min":0.315,"5min":0.253,"15min":0.23400000000000001},"max":{"1min":0.91300000000000003,"5min":3.5030000000000001,"15min":7.085},"last":0.92300000000000004}]},{"osd":3,"last update":"Tue Sep 10 19:24:20 2024","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":1.0289999999999999,"15min":1.0029999999999999},"min":{"1min":0.32700000000000001,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":0.88900000000000001,"5min":15.436,"15min":35.451999999999998},"last":0.876},{"interface":"front","average":{"1min":0.47799999999999998,"5min":0.94299999999999995,"15min":1.0609999999999999},"min":{"1min":0.29999999999999999,"5min":0.246,"15min":0.246},"max":{"1min":0.60499999999999998,"5min":14.954000000000001,"15min":32.880000000000003},"last":0.52200000000000002}]},{"osd":5,"last update":"Tue Sep 10 19:24:36 2024","interfaces":[{"interface":"back","average":{"1min":0.69399999999999995,"5min":3.7250000000000001,"15min":2.0550000000000002},"min":{"1min":0.32300000000000001,"5min":0.29999999999999999,"15min":0.224},"max":{"1min":2.319,"5min":227.726,"15min":227.726},"last":1.7},{"interface":"front","average":{"1min":0.59499999999999997,"5min":3.7149999999999999,"15min":1.9830000000000001},"min":{"1min":0.29299999999999998,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":1.958,"5min":227.70500000000001,"15min":227.70500000000001},"last":1.571}]},{"osd":6,"last update":"Tue Sep 10 19:24:11 2024","interfaces":[{"interface":"back","average":{"1min":0.54800000000000004,"5min":1.0409999999999999,"15min":0.85199999999999998},"min":{"1min":0.35599999999999998,"5min":0.27800000000000002,"15min":0.253},"max":{"1min":1.2569999999999999,"5min":5.1130000000000004,"15min":5.5289999999999999},"last":1.7450000000000001},{"interface":"front","average":{"1min":0.56299999999999994,"5min":1.083,"15min":0.877},"min":{"1min":0.309,"5min":0.25800000000000001,"15min":0.247},"max":{"1min":1.3799999999999999,"5min":7.3390000000000004,"15min":7.9550000000000001},"last":0.78900000000000003}]},{"osd":7,"last update":"Tue Sep 10 19:24:25 2024","interfaces":[{"interface":"back","average":{"1min":0.47799999999999998,"5min":1.343,"15min":1.1240000000000001},"min":{"1min":0.35199999999999998,"5min":0.308,"15min":0.27700000000000002},"max":{"1min":0.65600000000000003,"5min":5.9809999999999999,"15min":23.081},"last":0.49399999999999999},{"interface":"front","average":{"1min":0.52100000000000002,"5min":1.1859999999999999,"15min":1.0820000000000001},"min":{"1min":0.26100000000000001,"5min":0.26100000000000001,"15min":0.23699999999999999},"max":{"1min":1.3080000000000001,"5min":6.4729999999999999,"15min":22.666},"last":0.83799999999999997}]}]},{"osd":3,"up_from":26,"seq":111669149894,"num_pgs":70,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1508684,"kb_used_data":608388,"kb_used_omap":1,"kb_used_meta":900286,"kb_avail":92236468,"statfs":{"total":95995035648,"available":94450143232,"internally_reserved":0,"allocated":622989312,"data_stored":622576540,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":921893208},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 19:20:46 2024","interfaces":[{"interface":"back","average":{"1min":0.40799999999999997,"5min":0.39500000000000002,"15min":0.69099999999999995},"min":{"1min":0.16700000000000001,"5min":0.157,"15min":0.157},"max":{"1min":0.77800000000000002,"5min":3.9529999999999998,"15min":17.518000000000001},"last":0.35799999999999998},{"interface":"front","average":{"1min":0.72499999999999998,"5min":0.63900000000000001,"15min":0.84799999999999998},"min":{"1min":0.19400000000000001,"5min":0.17899999999999999,"15min":0.17399999999999999},"max":{"1min":2.3969999999999998,"5min":5.1539999999999999,"15min":17.609000000000002},"last":0.55100000000000005}]},{"osd":1,"last update":"Tue Sep 10 19:23:43 2024","interfaces":[{"interface":"back","average":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"min":{"1min":0.20300000000000001,"5min":0.20300000000000001,"15min":0.20300000000000001},"max":{"1min":0.443,"5min":0.443,"15min":0.443},"last":0.20399999999999999},{"interface":"front","average":{"1min":0.314,"5min":0.314,"15min":0.314},"min":{"1min":0.21299999999999999,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":0.745,"5min":0.745,"15min":0.745},"last":0.46700000000000003}]},{"osd":2,"last update":"Tue Sep 10 19:23:55 2024","interfaces":[{"interface":"back","average":{"1min":0.33400000000000002,"5min":0.42199999999999999,"15min":0.64200000000000002},"min":{"1min":0.23000000000000001,"5min":0.17599999999999999,"15min":0.17599999999999999},"max":{"1min":0.77900000000000003,"5min":2.2530000000000001,"15min":5.3029999999999999},"last":0.41799999999999998},{"interface":"front","average":{"1min":0.30199999999999999,"5min":0.372,"15min":0.69799999999999995},"min":{"1min":0.20100000000000001,"5min":0.16400000000000001,"15min":0.16400000000000001},"max":{"1min":0.44500000000000001,"5min":2.5649999999999999,"15min":17.806000000000001},"last":0.34300000000000003}]},{"osd":4,"last update":"Tue Sep 10 19:24:15 2024","interfaces":[{"interface":"back","average":{"1min":0.55000000000000004,"5min":0.75,"15min":0.92600000000000005},"min":{"1min":0.32300000000000001,"5min":0.26900000000000002,"15min":0.23599999999999999},"max":{"1min":0.95299999999999996,"5min":5.2750000000000004,"15min":17.834},"last":0.95799999999999996},{"interface":"front","average":{"1min":0.55900000000000005,"5min":0.873,"15min":1.085},"min":{"1min":0.28999999999999998,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":1.9490000000000001,"5min":4.3369999999999997,"15min":17.553000000000001},"last":0.876}]},{"osd":5,"last update":"Tue Sep 10 19:24:33 2024","interfaces":[{"interface":"back","average":{"1min":0.504,"5min":2.1030000000000002,"15min":1.6160000000000001},"min":{"1min":0.38900000000000001,"5min":0.28100000000000003,"15min":0.27900000000000003},"max":{"1min":0.64400000000000002,"5min":104.672,"15min":104.672},"last":0.36699999999999999},{"interface":"front","average":{"1min":0.48099999999999998,"5min":1.887,"15min":1.7150000000000001},"min":{"1min":0.28899999999999998,"5min":0.24299999999999999,"15min":0.22},"max":{"1min":0.63900000000000001,"5min":105.971,"15min":105.971},"last":0.501}]},{"osd":6,"last update":"Tue Sep 10 19:23:56 2024","interfaces":[{"interface":"back","average":{"1min":0.69099999999999995,"5min":2.5529999999999999,"15min":1.5609999999999999},"min":{"1min":0.28000000000000003,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":1.9990000000000001,"5min":116.14,"15min":116.14},"last":0.85899999999999999},{"interface":"front","average":{"1min":0.57899999999999996,"5min":2.5259999999999998,"15min":1.452},"min":{"1min":0.29299999999999998,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":2.1059999999999999,"5min":116.095,"15min":116.095},"last":0.83699999999999997}]},{"osd":7,"last update":"Tue Sep 10 19:24:30 2024","interfaces":[{"interface":"back","average":{"1min":0.502,"5min":1.9279999999999999,"15min":1.3520000000000001},"min":{"1min":0.32800000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.68899999999999995,"5min":104.749,"15min":104.749},"last":0.52500000000000002},{"interface":"front","average":{"1min":0.53100000000000003,"5min":1.7589999999999999,"15min":1.3720000000000001},"min":{"1min":0.42299999999999999,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.65700000000000003,"5min":104.69499999999999,"15min":104.69499999999999},"last":0.59999999999999998}]}]},{"osd":2,"up_from":0,"seq":77309411531,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":276,"seq":1185410973721,"num_pgs":67,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":734680,"kb_used_data":416336,"kb_used_omap":6,"kb_used_meta":318329,"kb_avail":93010472,"statfs":{"total":95995035648,"available":95242723328,"internally_reserved":0,"allocated":426328064,"data_stored":425909738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6335,"internal_metadata":325969729},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34599999999999997}]},{"osd":2,"last update":"Tue Sep 10 19:23:43 2024","interfaces":[{"interface":"back","average":{"1min":0.28799999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"min":{"1min":0.19,"5min":0.19,"15min":0.19},"max":{"1min":0.50900000000000001,"5min":0.50900000000000001,"15min":0.50900000000000001},"last":0.41899999999999998},{"interface":"front","average":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"min":{"1min":0.222,"5min":0.222,"15min":0.222},"max":{"1min":0.55500000000000005,"5min":0.55500000000000005,"15min":0.55500000000000005},"last":0.35999999999999999}]},{"osd":3,"last update":"Tue Sep 10 19:23:43 2024","interfaces":[{"interface":"back","average":{"1min":0.30399999999999999,"5min":0.30399999999999999,"15min":0.30399999999999999},"min":{"1min":0.221,"5min":0.221,"15min":0.221},"max":{"1min":0.56100000000000005,"5min":0.56100000000000005,"15min":0.56100000000000005},"last":0.39000000000000001},{"interface":"front","average":{"1min":0.30099999999999999,"5min":0.30099999999999999,"15min":0.30099999999999999},"min":{"1min":0.17599999999999999,"5min":0.17599999999999999,"15min":0.17599999999999999},"max":{"1min":0.48399999999999999,"5min":0.48399999999999999,"15min":0.48399999999999999},"last":0.33500000000000002}]},{"osd":4,"last update":"Tue Sep 10 19:23:43 2024","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.53200000000000003,"15min":0.53200000000000003},"min":{"1min":0.28000000000000003,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":1.9199999999999999,"5min":1.9199999999999999,"15min":1.9199999999999999},"last":0.28100000000000003},{"interface":"front","average":{"1min":0.61599999999999999,"5min":0.61599999999999999,"15min":0.61599999999999999},"min":{"1min":0.27600000000000002,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":2.0110000000000001,"5min":2.0110000000000001,"15min":2.0110000000000001},"last":0.46999999999999997}]},{"osd":5,"last update":"Tue Sep 10 19:23:43 2024","interfaces":[{"interface":"back","average":{"1min":0.99399999999999999,"5min":0.99399999999999999,"15min":0.99399999999999999},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":4.7519999999999998,"5min":4.7519999999999998,"15min":4.7519999999999998},"last":0.48399999999999999},{"interface":"front","average":{"1min":1.4330000000000001,"5min":1.4330000000000001,"15min":1.4330000000000001},"min":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":5.617,"5min":5.617,"15min":5.617},"last":0.44700000000000001}]},{"osd":6,"last update":"Tue Sep 10 19:23:43 2024","interfaces":[{"interface":"back","average":{"1min":2.3439999999999999,"5min":2.3439999999999999,"15min":2.3439999999999999},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":27.379999999999999,"5min":27.379999999999999,"15min":27.379999999999999},"last":0.55900000000000005},{"interface":"front","average":{"1min":2.3410000000000002,"5min":2.3410000000000002,"15min":2.3410000000000002},"min":{"1min":0.311,"5min":0.311,"15min":0.311},"max":{"1min":26.812999999999999,"5min":26.812999999999999,"15min":26.812999999999999},"last":3.2370000000000001}]},{"osd":7,"last update":"Tue Sep 10 19:23:43 2024","interfaces":[{"interface":"back","average":{"1min":1.0349999999999999,"5min":1.0349999999999999,"15min":1.0349999999999999},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":5.6260000000000003,"5min":5.6260000000000003,"15min":5.6260000000000003},"last":0.58799999999999997},{"interface":"front","average":{"1min":1.046,"5min":1.046,"15min":1.046},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":5.6050000000000004,"5min":5.6050000000000004,"15min":5.6050000000000004},"last":0.54500000000000004}]}]},{"osd":0,"up_from":301,"seq":1292785156101,"num_pgs":77,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":5437708,"kb_used_data":5066880,"kb_used_omap":4,"kb_used_meta":370811,"kb_avail":88307444,"statfs":{"total":95995035648,"available":90426822656,"internally_reserved":0,"allocated":5188485120,"data_stored":5188083359,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":379710982},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.29299999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.33900000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.313}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.97399999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.81999999999999995}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.6480000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72699999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":966,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1236,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1236,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":362,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1540,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":2338,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1018,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1502,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1445986304,"data_stored":1445986304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":443416576,"data_stored":443416576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":646971392,"data_stored":646971392,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1414725632,"data_stored":1414725632,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":922615808,"data_stored":922615808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":901316608,"data_stored":901316608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":661913600,"data_stored":661913600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T19:24:46.094 INFO:tasks.thrashosds.thrasher:Setting 2.8 to [0, 5, 1, 3, 7, 4] 2024-09-10T19:24:46.094 INFO:tasks.thrashosds.thrasher:cmd ['osd', 'pg-upmap-items', '2.8', '0', '5', '1', '3', '7', '4'] 2024-09-10T19:24:46.095 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph osd pg-upmap-items 2.8 0 5 1 3 7 4 2024-09-10T19:24:46.143 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:24:46.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:24:46.165 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:24:46.165 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:24:46.266 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:24:46.297 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:24:46.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:24:46.299 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:24:46.399 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:24:46.426 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:24:46.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:24:46.427 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:24:46.528 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:24:46.549 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:24:46.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:24:46.550 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:24:46.650 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:24:46.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:24:46.692 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:24:46.692 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:24:46.793 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:24:46.831 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:24:46.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:24:46.832 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:24:46.855 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: 26 15 43227 43212 103.864 80.875 0.00407507 0.00962266 2024-09-10T19:24:46.931 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:24:46 smithi183 ceph-mon[29736]: 4.5 scrub starts 2024-09-10T19:24:46.931 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:24:46 smithi183 ceph-mon[29736]: 4.5 scrub ok 2024-09-10T19:24:46.931 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:24:46 smithi183 ceph-mon[29736]: 2.4 scrub starts 2024-09-10T19:24:46.932 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:24:46 smithi183 ceph-mon[29736]: 2.4 scrub ok 2024-09-10T19:24:46.932 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:24:46 smithi183 ceph-mon[29736]: 1.2 scrub starts 2024-09-10T19:24:46.932 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:24:46 smithi183 ceph-mon[29736]: 1.2 scrub ok 2024-09-10T19:24:46.932 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:24:46 smithi183 ceph-mon[29736]: pgmap v666: 158 pgs: 158 active+clean; 2.4 GiB data, 15 GiB used, 610 GiB / 626 GiB avail; 854 B/s rd, 177 MiB/s wr, 2.84k op/s; 18 B/s, 0 objects/s recovering 2024-09-10T19:24:46.932 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:24:46 smithi183 ceph-mon[41562]: 4.5 scrub starts 2024-09-10T19:24:46.932 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:24:46 smithi183 ceph-mon[41562]: 4.5 scrub ok 2024-09-10T19:24:46.932 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:24:46 smithi183 ceph-mon[41562]: 2.4 scrub starts 2024-09-10T19:24:46.932 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:24:46 smithi183 ceph-mon[41562]: 2.4 scrub ok 2024-09-10T19:24:46.932 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:24:46 smithi183 ceph-mon[41562]: 1.2 scrub starts 2024-09-10T19:24:46.932 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:24:46 smithi183 ceph-mon[41562]: 1.2 scrub ok 2024-09-10T19:24:46.932 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:24:46 smithi183 ceph-mon[41562]: pgmap v666: 158 pgs: 158 active+clean; 2.4 GiB data, 15 GiB used, 610 GiB / 626 GiB avail; 854 B/s rd, 177 MiB/s wr, 2.84k op/s; 18 B/s, 0 objects/s recovering 2024-09-10T19:24:46.933 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:24:46.965 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:24:46 smithi191 ceph-mon[32884]: 4.5 scrub starts 2024-09-10T19:24:46.966 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:24:46 smithi191 ceph-mon[32884]: 4.5 scrub ok 2024-09-10T19:24:46.966 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:24:46 smithi191 ceph-mon[32884]: 2.4 scrub starts 2024-09-10T19:24:46.966 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:24:46 smithi191 ceph-mon[32884]: 2.4 scrub ok 2024-09-10T19:24:46.966 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:24:46 smithi191 ceph-mon[32884]: 1.2 scrub starts 2024-09-10T19:24:46.966 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:24:46 smithi191 ceph-mon[32884]: 1.2 scrub ok 2024-09-10T19:24:46.966 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:24:46 smithi191 ceph-mon[32884]: pgmap v666: 158 pgs: 158 active+clean; 2.4 GiB data, 15 GiB used, 610 GiB / 626 GiB avail; 854 B/s rd, 177 MiB/s wr, 2.84k op/s; 18 B/s, 0 objects/s recovering 2024-09-10T19:24:46.967 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:24:46.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:24:46.968 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:24:47.069 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:24:47.099 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:24:47.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:24:47.100 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:24:47.201 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:24:47.222 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:24:47.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:24:47.223 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:24:47.323 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:24:47.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:24:47.363 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:24:47.363 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:24:47.464 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:24:47.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:24:47.502 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:24:47.503 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:24:47.603 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:24:47.628 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:24:47.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:24:47.629 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:24:47.630 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:24:47.704 INFO:journalctl@ceph.mgr.x.smithi191.stdout:Sep 10 19:24:47 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-x[34967]: ::ffff:172.21.15.191 - - [10/Sep/2024:19:24:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T19:24:47.731 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:24:47.767 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:24:47.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:24:47.768 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:24:47.855 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: 27 16 44725 44709 103.482 93.5625 0.00403136 0.00965877 2024-09-10T19:24:47.870 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:24:47.875 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:24:47 smithi183 ceph-mon[29736]: 5.c scrub starts 2024-09-10T19:24:47.876 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:24:47 smithi183 ceph-mon[29736]: 5.c scrub ok 2024-09-10T19:24:47.876 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:24:47 smithi183 ceph-mon[29736]: 6.1f deep-scrub starts 2024-09-10T19:24:47.876 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:24:47 smithi183 ceph-mon[29736]: 6.1f deep-scrub ok 2024-09-10T19:24:47.876 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:24:47 smithi183 ceph-mon[29736]: 5.15 scrub starts 2024-09-10T19:24:47.876 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:24:47 smithi183 ceph-mon[29736]: 5.15 scrub ok 2024-09-10T19:24:47.876 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:24:47 smithi183 ceph-mon[41562]: 5.c scrub starts 2024-09-10T19:24:47.876 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:24:47 smithi183 ceph-mon[41562]: 5.c scrub ok 2024-09-10T19:24:47.876 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:24:47 smithi183 ceph-mon[41562]: 6.1f deep-scrub starts 2024-09-10T19:24:47.876 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:24:47 smithi183 ceph-mon[41562]: 6.1f deep-scrub ok 2024-09-10T19:24:47.876 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:24:47 smithi183 ceph-mon[41562]: 5.15 scrub starts 2024-09-10T19:24:47.876 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:24:47 smithi183 ceph-mon[41562]: 5.15 scrub ok 2024-09-10T19:24:47.908 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:24:47.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:24:47.909 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:24:47.966 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:24:47 smithi191 ceph-mon[32884]: 5.c scrub starts 2024-09-10T19:24:47.966 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:24:47 smithi191 ceph-mon[32884]: 5.c scrub ok 2024-09-10T19:24:47.966 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:24:47 smithi191 ceph-mon[32884]: 6.1f deep-scrub starts 2024-09-10T19:24:47.966 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:24:47 smithi191 ceph-mon[32884]: 6.1f deep-scrub ok 2024-09-10T19:24:47.966 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:24:47 smithi191 ceph-mon[32884]: 5.15 scrub starts 2024-09-10T19:24:47.966 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:24:47 smithi191 ceph-mon[32884]: 5.15 scrub ok 2024-09-10T19:24:48.010 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:24:48.045 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:24:48.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:24:48.046 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:24:48.147 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:24:48.180 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:24:48.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:24:48.181 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:24:48.282 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:24:48.310 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:24:48.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:24:48.312 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:24:48.412 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:24:48.450 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:24:48.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:24:48.451 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:24:48.552 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:24:48.581 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:24:48.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:24:48.583 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:24:48.683 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:24:48.714 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:24:48.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:24:48.715 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:24:48.781 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:24:48 smithi183 ceph-mon[29736]: 5.1f scrub starts 2024-09-10T19:24:48.781 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:24:48 smithi183 ceph-mon[29736]: 5.1f scrub ok 2024-09-10T19:24:48.781 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:24:48 smithi183 ceph-mon[29736]: purged_snaps scrub starts 2024-09-10T19:24:48.781 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:24:48 smithi183 ceph-mon[29736]: purged_snaps scrub ok 2024-09-10T19:24:48.781 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:24:48 smithi183 ceph-mon[29736]: 4.12 scrub starts 2024-09-10T19:24:48.782 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:24:48 smithi183 ceph-mon[29736]: 4.12 scrub ok 2024-09-10T19:24:48.782 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:24:48 smithi183 ceph-mon[29736]: osdmap e316: 8 total, 8 up, 7 in 2024-09-10T19:24:48.782 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:24:48 smithi183 ceph-mon[29736]: 5.b scrub starts 2024-09-10T19:24:48.782 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:24:48 smithi183 ceph-mon[29736]: 5.b scrub ok 2024-09-10T19:24:48.782 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:24:48 smithi183 ceph-mon[29736]: pgmap v668: 158 pgs: 158 active+clean; 2.5 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 1.7 KiB/s rd, 100 MiB/s wr, 1.60k op/s; 37 B/s, 1 objects/s recovering 2024-09-10T19:24:48.782 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:24:48 smithi183 ceph-mon[41562]: 5.1f scrub starts 2024-09-10T19:24:48.782 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:24:48 smithi183 ceph-mon[41562]: 5.1f scrub ok 2024-09-10T19:24:48.782 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:24:48 smithi183 ceph-mon[41562]: purged_snaps scrub starts 2024-09-10T19:24:48.782 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:24:48 smithi183 ceph-mon[41562]: purged_snaps scrub ok 2024-09-10T19:24:48.782 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:24:48 smithi183 ceph-mon[41562]: 4.12 scrub starts 2024-09-10T19:24:48.782 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:24:48 smithi183 ceph-mon[41562]: 4.12 scrub ok 2024-09-10T19:24:48.783 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:24:48 smithi183 ceph-mon[41562]: osdmap e316: 8 total, 8 up, 7 in 2024-09-10T19:24:48.783 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:24:48 smithi183 ceph-mon[41562]: 5.b scrub starts 2024-09-10T19:24:48.783 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:24:48 smithi183 ceph-mon[41562]: 5.b scrub ok 2024-09-10T19:24:48.783 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:24:48 smithi183 ceph-mon[41562]: pgmap v668: 158 pgs: 158 active+clean; 2.5 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 1.7 KiB/s rd, 100 MiB/s wr, 1.60k op/s; 37 B/s, 1 objects/s recovering 2024-09-10T19:24:48.816 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:24:48.855 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: 28 16 46404 46388 103.534 104.938 0.0051421 0.00965298 2024-09-10T19:24:48.857 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:24:48.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:24:48.858 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:24:48.959 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:24:48.965 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:24:48 smithi191 ceph-mon[32884]: 5.1f scrub starts 2024-09-10T19:24:48.966 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:24:48 smithi191 ceph-mon[32884]: 5.1f scrub ok 2024-09-10T19:24:48.966 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:24:48 smithi191 ceph-mon[32884]: purged_snaps scrub starts 2024-09-10T19:24:48.966 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:24:48 smithi191 ceph-mon[32884]: purged_snaps scrub ok 2024-09-10T19:24:48.966 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:24:48 smithi191 ceph-mon[32884]: 4.12 scrub starts 2024-09-10T19:24:48.966 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:24:48 smithi191 ceph-mon[32884]: 4.12 scrub ok 2024-09-10T19:24:48.966 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:24:48 smithi191 ceph-mon[32884]: osdmap e316: 8 total, 8 up, 7 in 2024-09-10T19:24:48.966 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:24:48 smithi191 ceph-mon[32884]: 5.b scrub starts 2024-09-10T19:24:48.966 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:24:48 smithi191 ceph-mon[32884]: 5.b scrub ok 2024-09-10T19:24:48.966 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:24:48 smithi191 ceph-mon[32884]: pgmap v668: 158 pgs: 158 active+clean; 2.5 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 1.7 KiB/s rd, 100 MiB/s wr, 1.60k op/s; 37 B/s, 1 objects/s recovering 2024-09-10T19:24:48.992 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:24:48.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:24:48.993 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:24:49.062 INFO:teuthology.orchestra.run.smithi183.stdout:osd.0: {} 2024-09-10T19:24:49.062 INFO:teuthology.orchestra.run.smithi183.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-10T19:24:49.064 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph osd set noscrub 2024-09-10T19:24:49.088 INFO:teuthology.orchestra.run.smithi183.stdout:osd.1: {} 2024-09-10T19:24:49.088 INFO:teuthology.orchestra.run.smithi183.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-10T19:24:49.094 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:24:49.113 INFO:teuthology.orchestra.run.smithi183.stdout:osd.2: {} 2024-09-10T19:24:49.113 INFO:teuthology.orchestra.run.smithi183.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-10T19:24:49.130 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:24:49.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:24:49.131 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:24:49.144 INFO:teuthology.orchestra.run.smithi183.stdout:osd.3: {} 2024-09-10T19:24:49.144 INFO:teuthology.orchestra.run.smithi183.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-10T19:24:49.175 INFO:teuthology.orchestra.run.smithi183.stdout:osd.4: {} 2024-09-10T19:24:49.175 INFO:teuthology.orchestra.run.smithi183.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-10T19:24:49.209 INFO:teuthology.orchestra.run.smithi183.stdout:osd.5: {} 2024-09-10T19:24:49.209 INFO:teuthology.orchestra.run.smithi183.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-10T19:24:49.231 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:24:49.240 INFO:teuthology.orchestra.run.smithi183.stdout:osd.6: {} 2024-09-10T19:24:49.241 INFO:teuthology.orchestra.run.smithi183.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-10T19:24:49.273 INFO:teuthology.orchestra.run.smithi183.stdout:osd.7: {} 2024-09-10T19:24:49.274 INFO:teuthology.orchestra.run.smithi183.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-10T19:24:49.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:24:49.281 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:24:49.281 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:24:49.382 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:24:49.426 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:24:49.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:24:49.427 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:24:49.528 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:24:49.571 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:24:49.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:24:49.572 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:24:49.673 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:24:49.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:24:49.703 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:24:49.703 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:24:49.805 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:24:49.848 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:24:49.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:24:49.849 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:24:49.855 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: 29 16 48026 48010 103.459 101.375 0.0092732 0.00965897 2024-09-10T19:24:49.873 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:24:49 smithi183 ceph-mon[29736]: from='client.? 172.21.15.183:0/1672482754' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T19:24:49.873 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:24:49 smithi183 ceph-mon[29736]: 5.7 scrub starts 2024-09-10T19:24:49.874 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:24:49 smithi183 ceph-mon[29736]: 5.7 scrub ok 2024-09-10T19:24:49.874 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:24:49 smithi191 ceph-mon[32884]: from='client.? 172.21.15.183:0/1672482754' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T19:24:49.874 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:24:49 smithi191 ceph-mon[32884]: 5.7 scrub starts 2024-09-10T19:24:49.874 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:24:49 smithi191 ceph-mon[32884]: 5.7 scrub ok 2024-09-10T19:24:49.874 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:24:49 smithi183 ceph-mon[41562]: from='client.? 172.21.15.183:0/1672482754' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T19:24:49.875 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:24:49 smithi183 ceph-mon[41562]: 5.7 scrub starts 2024-09-10T19:24:49.875 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:24:49 smithi183 ceph-mon[41562]: 5.7 scrub ok 2024-09-10T19:24:49.875 INFO:journalctl@ceph.iscsi.iscsi.a.smithi191.stdout:Sep 10 19:24:49 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-iscsi-iscsi-a[61816]: debug there is no tcmu-runner data available 2024-09-10T19:24:49.930 INFO:teuthology.orchestra.run.smithi191.stdout:{ 2024-09-10T19:24:49.930 INFO:teuthology.orchestra.run.smithi191.stdout: "ops": [ 2024-09-10T19:24:49.930 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:24:49.930 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.25622.0:48125 10.b 10:d753f774:::benchmark_data_smithi183_176537_object48124:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e317)", 2024-09-10T19:24:49.930 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-10T19:24:49.920216+0000", 2024-09-10T19:24:49.930 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 0.0064554219999999997, 2024-09-10T19:24:49.930 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0064786369999999998, 2024-09-10T19:24:49.930 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-10T19:24:49.930 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "waiting for sub ops", 2024-09-10T19:24:49.930 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-10T19:24:49.931 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.25622", 2024-09-10T19:24:49.931 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.183:0/3685189199", 2024-09-10T19:24:49.931 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 48125 2024-09-10T19:24:49.931 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:24:49.931 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-10T19:24:49.931 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:24:49.931 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-10T19:24:49.931 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:24:49.920216+0000", 2024-09-10T19:24:49.931 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:24:49.931 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:24:49.931 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:24:49.931 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-10T19:24:49.931 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:24:49.920216+0000", 2024-09-10T19:24:49.931 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:24:49.931 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:24:49.932 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:24:49.932 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-10T19:24:49.932 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:24:49.920215+0000", 2024-09-10T19:24:49.932 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.999999 2024-09-10T19:24:49.932 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:24:49.932 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:24:49.932 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-10T19:24:49.932 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:24:49.920276+0000", 2024-09-10T19:24:49.932 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.1524000000000003e-05 2024-09-10T19:24:49.932 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:24:49.932 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:24:49.932 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-10T19:24:49.932 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:24:49.920287+0000", 2024-09-10T19:24:49.932 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.0567999999999999e-05 2024-09-10T19:24:49.932 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:24:49.933 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:24:49.933 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-10T19:24:49.933 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:24:49.920292+0000", 2024-09-10T19:24:49.933 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.6409999999999999e-06 2024-09-10T19:24:49.933 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:24:49.933 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:24:49.933 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:24:49.933 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:24:49.920433+0000", 2024-09-10T19:24:49.933 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000140581 2024-09-10T19:24:49.933 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:24:49.933 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:24:49.933 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-10T19:24:49.933 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:24:49.920577+0000", 2024-09-10T19:24:49.934 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000143731 2024-09-10T19:24:49.934 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:24:49.934 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:24:49.934 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0,1", 2024-09-10T19:24:49.934 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:24:49.920688+0000", 2024-09-10T19:24:49.934 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00011073699999999999 2024-09-10T19:24:49.934 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:24:49.934 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-10T19:24:49.934 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:24:49.934 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:24:49.934 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:24:49.934 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.25622.0:48126 10.b 10:d894a342:::benchmark_data_smithi183_176537_object48125:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e317)", 2024-09-10T19:24:49.935 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-10T19:24:49.920306+0000", 2024-09-10T19:24:49.935 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 0.006365084, 2024-09-10T19:24:49.935 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00643555, 2024-09-10T19:24:49.935 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-10T19:24:49.935 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "waiting for sub ops", 2024-09-10T19:24:49.935 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-10T19:24:49.935 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.25622", 2024-09-10T19:24:49.935 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.183:0/3685189199", 2024-09-10T19:24:49.935 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 48126 2024-09-10T19:24:49.935 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:24:49.935 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-10T19:24:49.935 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:24:49.935 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-10T19:24:49.935 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:24:49.920306+0000", 2024-09-10T19:24:49.936 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:24:49.936 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:24:49.936 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:24:49.936 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-10T19:24:49.936 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:24:49.920306+0000", 2024-09-10T19:24:49.936 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:24:49.936 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:24:49.936 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:24:49.936 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-10T19:24:49.936 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:24:49.920306+0000", 2024-09-10T19:24:49.936 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999995 2024-09-10T19:24:49.936 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:24:49.936 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:24:49.937 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-10T19:24:49.937 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:24:49.920350+0000", 2024-09-10T19:24:49.937 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.4660999999999998e-05 2024-09-10T19:24:49.937 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:24:49.937 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:24:49.937 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-10T19:24:49.937 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:24:49.920357+0000", 2024-09-10T19:24:49.937 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.7880000000000001e-06 2024-09-10T19:24:49.937 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:24:49.937 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:24:49.937 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-10T19:24:49.937 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:24:49.920361+0000", 2024-09-10T19:24:49.937 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.9820000000000002e-06 2024-09-10T19:24:49.937 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:24:49.938 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:24:49.938 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:24:49.938 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:24:49.922356+0000", 2024-09-10T19:24:49.938 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0019945480000000001 2024-09-10T19:24:49.938 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:24:49.938 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:24:49.938 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-10T19:24:49.938 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:24:49.922467+0000", 2024-09-10T19:24:49.938 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00011171399999999999 2024-09-10T19:24:49.938 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:24:49.938 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:24:49.938 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0,1", 2024-09-10T19:24:49.938 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:24:49.922600+0000", 2024-09-10T19:24:49.938 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00013274000000000001 2024-09-10T19:24:49.939 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:24:49.939 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-10T19:24:49.939 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:24:49.939 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:24:49.939 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:24:49.939 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_repop_reply(client.25622.0:48125 10.b e317/306)", 2024-09-10T19:24:49.939 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-10T19:24:49.922905+0000", 2024-09-10T19:24:49.939 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 0.0037659339999999999, 2024-09-10T19:24:49.939 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0038874320000000001, 2024-09-10T19:24:49.939 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-10T19:24:49.939 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "queued for pg", 2024-09-10T19:24:49.939 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-10T19:24:49.939 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:24:49.939 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-10T19:24:49.939 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:24:49.922905+0000", 2024-09-10T19:24:49.940 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:24:49.940 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:24:49.940 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:24:49.940 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-10T19:24:49.940 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:24:49.922905+0000", 2024-09-10T19:24:49.940 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:24:49.940 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:24:49.940 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:24:49.940 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-10T19:24:49.940 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:24:49.922905+0000", 2024-09-10T19:24:49.940 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999995 2024-09-10T19:24:49.940 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:24:49.940 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:24:49.940 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-10T19:24:49.941 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:24:49.922908+0000", 2024-09-10T19:24:49.941 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.7700000000000002e-06 2024-09-10T19:24:49.941 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:24:49.941 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:24:49.941 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-10T19:24:49.941 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:24:49.922916+0000", 2024-09-10T19:24:49.941 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.7069999999999998e-06 2024-09-10T19:24:49.941 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:24:49.941 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:24:49.941 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-10T19:24:49.941 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:24:49.922920+0000", 2024-09-10T19:24:49.941 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.1069999999999998e-06 2024-09-10T19:24:49.941 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:24:49.942 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-10T19:24:49.942 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:24:49.942 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:24:49.942 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:24:49.942 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.25622.0:48106 10.d 10:bb57b68d:::benchmark_data_smithi183_176537_object48105:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e317)", 2024-09-10T19:24:49.942 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-10T19:24:49.905827+0000", 2024-09-10T19:24:49.942 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 0.020844623, 2024-09-10T19:24:49.942 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.020994247000000001, 2024-09-10T19:24:49.942 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-10T19:24:49.942 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "waiting for sub ops", 2024-09-10T19:24:49.942 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-10T19:24:49.942 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.25622", 2024-09-10T19:24:49.942 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.183:0/3685189199", 2024-09-10T19:24:49.942 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 48106 2024-09-10T19:24:49.943 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:24:49.943 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-10T19:24:49.943 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:24:49.943 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-10T19:24:49.943 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:24:49.905827+0000", 2024-09-10T19:24:49.943 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:24:49.943 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:24:49.943 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:24:49.943 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-10T19:24:49.943 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:24:49.905827+0000", 2024-09-10T19:24:49.943 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:24:49.943 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:24:49.943 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:24:49.944 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-10T19:24:49.944 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:24:49.905826+0000", 2024-09-10T19:24:49.944 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999995 2024-09-10T19:24:49.944 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:24:49.944 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:24:49.944 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-10T19:24:49.945 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:24:49.905946+0000", 2024-09-10T19:24:49.945 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000120049 2024-09-10T19:24:49.945 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:24:49.945 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:24:49.945 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-10T19:24:49.945 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:24:49.905954+0000", 2024-09-10T19:24:49.945 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.0800000000000006e-06 2024-09-10T19:24:49.945 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:24:49.945 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:24:49.945 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-10T19:24:49.945 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:24:49.905959+0000", 2024-09-10T19:24:49.945 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.4900000000000002e-06 2024-09-10T19:24:49.945 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:24:49.945 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:24:49.946 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:24:49.946 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:24:49.915472+0000", 2024-09-10T19:24:49.946 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0095139660000000004 2024-09-10T19:24:49.946 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:24:49.946 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:24:49.946 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-10T19:24:49.946 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:24:49.915596+0000", 2024-09-10T19:24:49.946 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00012364900000000001 2024-09-10T19:24:49.946 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:24:49.946 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:24:49.946 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0,4", 2024-09-10T19:24:49.946 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:24:49.915727+0000", 2024-09-10T19:24:49.946 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00013089300000000001 2024-09-10T19:24:49.946 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:24:49.947 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:24:49.947 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:24:49.947 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:24:49.918175+0000", 2024-09-10T19:24:49.947 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0024475349999999998 2024-09-10T19:24:49.947 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:24:49.947 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:24:49.947 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-09-10T19:24:49.947 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:24:49.925509+0000", 2024-09-10T19:24:49.947 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.007334747 2024-09-10T19:24:49.947 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:24:49.947 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-10T19:24:49.947 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:24:49.947 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:24:49.947 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:24:49.948 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_repop_reply(client.25622.0:48126 10.b e317/306)", 2024-09-10T19:24:49.948 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-10T19:24:49.924053+0000", 2024-09-10T19:24:49.948 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 0.0026182110000000001, 2024-09-10T19:24:49.948 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0028134229999999998, 2024-09-10T19:24:49.948 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-10T19:24:49.948 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "queued for pg", 2024-09-10T19:24:49.948 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-10T19:24:49.948 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:24:49.948 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-10T19:24:49.948 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:24:49.924053+0000", 2024-09-10T19:24:49.948 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:24:49.948 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:24:49.948 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:24:49.948 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-10T19:24:49.949 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:24:49.924053+0000", 2024-09-10T19:24:49.949 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:24:49.949 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:24:49.949 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:24:49.949 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-10T19:24:49.949 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:24:49.924052+0000", 2024-09-10T19:24:49.949 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.999999 2024-09-10T19:24:49.949 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:24:49.949 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:24:49.949 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-10T19:24:49.949 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:24:49.924057+0000", 2024-09-10T19:24:49.949 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.3139999999999997e-06 2024-09-10T19:24:49.949 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:24:49.949 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:24:49.950 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-10T19:24:49.950 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:24:49.924070+0000", 2024-09-10T19:24:49.950 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.3399e-05 2024-09-10T19:24:49.950 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:24:49.950 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:24:49.950 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-10T19:24:49.950 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:24:49.924075+0000", 2024-09-10T19:24:49.950 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.0819999999999998e-06 2024-09-10T19:24:49.950 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:24:49.950 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-10T19:24:49.950 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:24:49.950 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:24:49.950 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:24:49.950 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.25622.0:48107 10.d 10:b310cf1e:::benchmark_data_smithi183_176537_object48106:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e317)", 2024-09-10T19:24:49.950 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-10T19:24:49.905964+0000", 2024-09-10T19:24:49.951 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 0.020707625, 2024-09-10T19:24:49.951 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.020930029999999999, 2024-09-10T19:24:49.951 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-10T19:24:49.951 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "waiting for sub ops", 2024-09-10T19:24:49.951 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-10T19:24:49.951 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.25622", 2024-09-10T19:24:49.951 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.183:0/3685189199", 2024-09-10T19:24:49.951 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 48107 2024-09-10T19:24:49.951 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:24:49.951 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-10T19:24:49.951 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:24:49.951 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-10T19:24:49.951 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:24:49.905964+0000", 2024-09-10T19:24:49.951 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:24:49.952 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:24:49.952 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:24:49.952 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-10T19:24:49.952 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:24:49.905964+0000", 2024-09-10T19:24:49.952 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:24:49.952 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:24:49.952 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:24:49.952 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-10T19:24:49.952 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:24:49.905963+0000", 2024-09-10T19:24:49.952 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999995 2024-09-10T19:24:49.952 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:24:49.952 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:24:49.952 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-10T19:24:49.953 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:24:49.906014+0000", 2024-09-10T19:24:49.953 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.1493000000000003e-05 2024-09-10T19:24:49.953 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:24:49.953 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:24:49.953 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-10T19:24:49.953 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:24:49.906024+0000", 2024-09-10T19:24:49.953 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.0990000000000005e-06 2024-09-10T19:24:49.953 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:24:49.953 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:24:49.953 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-10T19:24:49.953 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:24:49.906028+0000", 2024-09-10T19:24:49.953 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.9169999999999997e-06 2024-09-10T19:24:49.953 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:24:49.953 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:24:49.954 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:24:49.954 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:24:49.918196+0000", 2024-09-10T19:24:49.954 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.012167446 2024-09-10T19:24:49.954 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:24:49.954 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:24:49.954 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-10T19:24:49.954 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:24:49.918312+0000", 2024-09-10T19:24:49.954 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000116538 2024-09-10T19:24:49.954 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:24:49.954 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:24:49.954 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0,4", 2024-09-10T19:24:49.954 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:24:49.918446+0000", 2024-09-10T19:24:49.954 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00013335699999999999 2024-09-10T19:24:49.954 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:24:49.955 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:24:49.955 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-09-10T19:24:49.955 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:24:49.925514+0000", 2024-09-10T19:24:49.955 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0070679499999999999 2024-09-10T19:24:49.955 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:24:49.955 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:24:49.955 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:24:49.955 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:24:49.925573+0000", 2024-09-10T19:24:49.955 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.9499000000000001e-05 2024-09-10T19:24:49.955 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:24:49.955 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-10T19:24:49.955 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:24:49.955 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:24:49.955 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:24:49.956 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_repop_reply(client.25622.0:48125 10.b e317/306)", 2024-09-10T19:24:49.956 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-10T19:24:49.924665+0000", 2024-09-10T19:24:49.956 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 0.0020061189999999998, 2024-09-10T19:24:49.956 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0022690119999999999, 2024-09-10T19:24:49.956 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-10T19:24:49.956 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "queued for pg", 2024-09-10T19:24:49.956 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-10T19:24:49.956 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:24:49.956 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-10T19:24:49.956 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:24:49.924665+0000", 2024-09-10T19:24:49.956 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:24:49.956 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:24:49.956 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:24:49.956 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-10T19:24:49.957 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:24:49.924665+0000", 2024-09-10T19:24:49.957 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:24:49.957 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:24:49.957 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:24:49.957 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-10T19:24:49.957 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:24:49.924665+0000", 2024-09-10T19:24:49.957 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999995 2024-09-10T19:24:49.957 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:24:49.957 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:24:49.957 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-10T19:24:49.957 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:24:49.924667+0000", 2024-09-10T19:24:49.957 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.6010000000000002e-06 2024-09-10T19:24:49.957 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:24:49.957 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:24:49.958 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-10T19:24:49.958 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:24:49.924711+0000", 2024-09-10T19:24:49.958 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.4110999999999998e-05 2024-09-10T19:24:49.958 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:24:49.958 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:24:49.958 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-10T19:24:49.958 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:24:49.924717+0000", 2024-09-10T19:24:49.958 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.6699999999999999e-06 2024-09-10T19:24:49.958 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:24:49.958 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-10T19:24:49.958 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:24:49.958 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:24:49.958 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:24:49.958 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_repop_reply(client.25622.0:48126 10.b e317/306)", 2024-09-10T19:24:49.959 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-10T19:24:49.926781+0000", 2024-09-10T19:24:49.959 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 4294967295.9998899, 2024-09-10T19:24:49.959 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00017806699999999999, 2024-09-10T19:24:49.959 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-10T19:24:49.959 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "queued for pg", 2024-09-10T19:24:49.959 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-10T19:24:49.959 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:24:49.959 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-10T19:24:49.959 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:24:49.926781+0000", 2024-09-10T19:24:49.959 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:24:49.959 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:24:49.959 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:24:49.959 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-10T19:24:49.959 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:24:49.926781+0000", 2024-09-10T19:24:49.960 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:24:49.960 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:24:49.960 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:24:49.960 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-10T19:24:49.960 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:24:49.926780+0000", 2024-09-10T19:24:49.960 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.999999 2024-09-10T19:24:49.960 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:24:49.960 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:24:49.960 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-10T19:24:49.960 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:24:49.926786+0000", 2024-09-10T19:24:49.960 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.5269999999999998e-06 2024-09-10T19:24:49.960 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:24:49.960 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:24:49.960 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-10T19:24:49.960 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:24:49.926796+0000", 2024-09-10T19:24:49.961 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.0516999999999999e-05 2024-09-10T19:24:49.961 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:24:49.961 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:24:49.961 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-10T19:24:49.961 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:24:49.926802+0000", 2024-09-10T19:24:49.961 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.4049999999999999e-06 2024-09-10T19:24:49.961 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:24:49.961 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-10T19:24:49.961 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:24:49.961 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:24:49.961 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:24:49.961 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_repop_reply(client.25622.0:48106 10.d e317/306)", 2024-09-10T19:24:49.961 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-10T19:24:49.926844+0000", 2024-09-10T19:24:49.961 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 4294967295.9998269, 2024-09-10T19:24:49.962 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00013786199999999999, 2024-09-10T19:24:49.962 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-10T19:24:49.962 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "reached pg", 2024-09-10T19:24:49.962 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-10T19:24:49.962 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:24:49.962 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-10T19:24:49.962 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:24:49.926844+0000", 2024-09-10T19:24:49.962 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:24:49.962 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:24:49.962 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:24:49.962 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-10T19:24:49.962 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:24:49.926844+0000", 2024-09-10T19:24:49.962 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:24:49.962 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:24:49.963 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:24:49.963 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-10T19:24:49.963 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:24:49.926844+0000", 2024-09-10T19:24:49.963 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999995 2024-09-10T19:24:49.963 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:24:49.963 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:24:49.963 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-10T19:24:49.963 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:24:49.926846+0000", 2024-09-10T19:24:49.963 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.4669999999999998e-06 2024-09-10T19:24:49.963 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:24:49.963 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:24:49.963 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-10T19:24:49.963 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:24:49.926854+0000", 2024-09-10T19:24:49.963 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.3930000000000002e-06 2024-09-10T19:24:49.964 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:24:49.964 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:24:49.964 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-10T19:24:49.964 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:24:49.926857+0000", 2024-09-10T19:24:49.964 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.4769999999999999e-06 2024-09-10T19:24:49.964 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:24:49.964 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:24:49.964 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:24:49.964 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:24:49.926981+0000", 2024-09-10T19:24:49.964 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00012356300000000001 2024-09-10T19:24:49.964 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:24:49.964 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-10T19:24:49.964 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:24:49.964 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:24:49.965 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:24:49.965 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_repop_reply(client.25622.0:48107 10.d e317/306)", 2024-09-10T19:24:49.965 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-10T19:24:49.926867+0000", 2024-09-10T19:24:49.965 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 4294967295.999804, 2024-09-10T19:24:49.965 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00014382899999999999, 2024-09-10T19:24:49.965 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-10T19:24:49.965 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "queued for pg", 2024-09-10T19:24:49.965 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-10T19:24:49.965 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:24:49.965 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-10T19:24:49.965 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:24:49.926867+0000", 2024-09-10T19:24:49.965 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:24:49.965 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:24:49.965 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:24:49.966 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-10T19:24:49.966 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:24:49.926867+0000", 2024-09-10T19:24:49.966 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:24:49.966 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:24:49.966 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:24:49.966 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-10T19:24:49.966 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:24:49.926867+0000", 2024-09-10T19:24:49.966 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967296 2024-09-10T19:24:49.966 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:24:49.966 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:24:49.966 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-10T19:24:49.966 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:24:49.926869+0000", 2024-09-10T19:24:49.966 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.73e-06 2024-09-10T19:24:49.966 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:24:49.967 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:24:49.967 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-10T19:24:49.967 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:24:49.926874+0000", 2024-09-10T19:24:49.967 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.6800000000000001e-06 2024-09-10T19:24:49.967 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:24:49.967 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:24:49.967 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-10T19:24:49.967 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:24:49.926876+0000", 2024-09-10T19:24:49.967 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.8760000000000001e-06 2024-09-10T19:24:49.967 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:24:49.967 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-10T19:24:49.967 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:24:49.967 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:24:49.967 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:24:49.967 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_repop_reply(client.25622.0:48123 10.f e317/295)", 2024-09-10T19:24:49.968 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-10T19:24:49.926879+0000", 2024-09-10T19:24:49.968 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 4294967295.9997916, 2024-09-10T19:24:49.968 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000154387, 2024-09-10T19:24:49.968 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-10T19:24:49.968 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "queued for pg", 2024-09-10T19:24:49.968 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-10T19:24:49.968 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:24:49.968 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-10T19:24:49.968 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:24:49.926879+0000", 2024-09-10T19:24:49.968 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:24:49.968 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:24:49.968 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:24:49.968 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-10T19:24:49.969 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:24:49.926879+0000", 2024-09-10T19:24:49.969 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:24:49.969 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:24:49.969 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:24:49.969 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-10T19:24:49.969 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:24:49.926879+0000", 2024-09-10T19:24:49.969 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967296 2024-09-10T19:24:49.969 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:24:49.969 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:24:49.969 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-10T19:24:49.969 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:24:49.926881+0000", 2024-09-10T19:24:49.969 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.3289999999999999e-06 2024-09-10T19:24:49.969 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:24:49.969 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:24:49.969 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-10T19:24:49.970 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:24:49.926885+0000", 2024-09-10T19:24:49.970 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.2549999999999999e-06 2024-09-10T19:24:49.970 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:24:49.970 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:24:49.970 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-10T19:24:49.970 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:24:49.926888+0000", 2024-09-10T19:24:49.970 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.6840000000000001e-06 2024-09-10T19:24:49.970 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:24:49.970 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-10T19:24:49.970 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:24:49.970 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:24:49.970 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:24:49.970 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_repop_reply(client.25622.0:48119 10.0 e317/295 ondisk, result = 0)", 2024-09-10T19:24:49.970 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-10T19:24:49.926895+0000", 2024-09-10T19:24:49.971 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 4294967295.9997759, 2024-09-10T19:24:49.971 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000161605, 2024-09-10T19:24:49.971 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-10T19:24:49.971 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "started", 2024-09-10T19:24:49.971 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-10T19:24:49.971 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:24:49.971 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-10T19:24:49.971 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:24:49.926895+0000", 2024-09-10T19:24:49.971 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:24:49.971 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:24:49.971 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:24:49.971 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-10T19:24:49.971 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:24:49.926895+0000", 2024-09-10T19:24:49.971 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:24:49.972 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:24:49.972 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:24:49.972 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-10T19:24:49.972 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:24:49.926895+0000", 2024-09-10T19:24:49.972 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967296 2024-09-10T19:24:49.972 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:24:49.972 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:24:49.972 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-10T19:24:49.972 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:24:49.926897+0000", 2024-09-10T19:24:49.972 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.5910000000000001e-06 2024-09-10T19:24:49.972 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:24:49.972 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:24:49.973 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-10T19:24:49.973 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:24:49.974 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:24:49.926901+0000", 2024-09-10T19:24:49.974 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.3959999999999999e-06 2024-09-10T19:24:49.974 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:24:49.974 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:24:49.974 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-10T19:24:49.974 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:24:49.926905+0000", 2024-09-10T19:24:49.974 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.692e-06 2024-09-10T19:24:49.974 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:24:49.974 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:24:49.974 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:24:49.974 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:24:49.927006+0000", 2024-09-10T19:24:49.975 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00010052699999999999 2024-09-10T19:24:49.975 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:24:49.975 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:24:49.975 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-10T19:24:49.975 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:24:49.927010+0000", 2024-09-10T19:24:49.975 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.7310000000000001e-06 2024-09-10T19:24:49.975 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:24:49.975 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-10T19:24:49.975 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:24:49.975 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:24:49.975 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:24:49.975 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.25622.0:48119 10.0 10:023f5456:::benchmark_data_smithi183_176537_object48118:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e317)", 2024-09-10T19:24:49.975 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-10T19:24:49.919597+0000", 2024-09-10T19:24:49.975 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 0.0070742069999999999, 2024-09-10T19:24:49.976 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0075001, 2024-09-10T19:24:49.976 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-10T19:24:49.976 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T19:24:49.976 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-10T19:24:49.976 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.25622", 2024-09-10T19:24:49.976 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.183:0/3685189199", 2024-09-10T19:24:49.976 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 48119 2024-09-10T19:24:49.976 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:24:49.976 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-10T19:24:49.976 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:24:49.976 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-10T19:24:49.976 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:24:49.919597+0000", 2024-09-10T19:24:49.976 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:24:49.977 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:24:49.977 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:24:49.977 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-10T19:24:49.977 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:24:49.919597+0000", 2024-09-10T19:24:49.977 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:24:49.977 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:24:49.977 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:24:49.977 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-10T19:24:49.977 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:24:49.919595+0000", 2024-09-10T19:24:49.977 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999986 2024-09-10T19:24:49.977 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:24:49.977 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:24:49.977 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-10T19:24:49.977 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:24:49.919636+0000", 2024-09-10T19:24:49.978 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.0420000000000003e-05 2024-09-10T19:24:49.978 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:24:49.978 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:24:49.978 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-10T19:24:49.978 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:24:49.919651+0000", 2024-09-10T19:24:49.978 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.4871999999999999e-05 2024-09-10T19:24:49.978 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:24:49.978 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:24:49.978 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-10T19:24:49.978 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:24:49.919657+0000", 2024-09-10T19:24:49.978 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.0950000000000001e-06 2024-09-10T19:24:49.978 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:24:49.978 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:24:49.978 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:24:49.979 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:24:49.920797+0000", 2024-09-10T19:24:49.979 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0011399349999999999 2024-09-10T19:24:49.979 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:24:49.979 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:24:49.979 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-10T19:24:49.979 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:24:49.920915+0000", 2024-09-10T19:24:49.979 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000118491 2024-09-10T19:24:49.979 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:24:49.979 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:24:49.979 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 1,4", 2024-09-10T19:24:49.979 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:24:49.921053+0000", 2024-09-10T19:24:49.979 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000137625 2024-09-10T19:24:49.979 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:24:49.979 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:24:49.980 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:24:49.980 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:24:49.924321+0000", 2024-09-10T19:24:49.980 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0032679530000000001 2024-09-10T19:24:49.980 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:24:49.980 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:24:49.980 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-09-10T19:24:49.980 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:24:49.925528+0000", 2024-09-10T19:24:49.980 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0012075289999999999 2024-09-10T19:24:49.980 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:24:49.980 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:24:49.980 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:24:49.980 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:24:49.927015+0000", 2024-09-10T19:24:49.980 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0014871280000000001 2024-09-10T19:24:49.980 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:24:49.981 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:24:49.981 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-09-10T19:24:49.981 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:24:49.927054+0000", 2024-09-10T19:24:49.981 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.8148000000000001e-05 2024-09-10T19:24:49.981 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:24:49.981 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-10T19:24:49.981 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:24:49.981 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:24:49.981 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:24:49.981 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.25622.0:48123 10.f 10:f6aa9e83:::benchmark_data_smithi183_176537_object48122:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536 in=65536b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e317)", 2024-09-10T19:24:49.981 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-10T19:24:49.919732+0000", 2024-09-10T19:24:49.981 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 0.0069389639999999997, 2024-09-10T19:24:49.981 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0074164360000000002, 2024-09-10T19:24:49.982 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-10T19:24:49.982 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "waiting for sub ops", 2024-09-10T19:24:49.982 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-10T19:24:49.982 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.25622", 2024-09-10T19:24:49.982 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.183:0/3685189199", 2024-09-10T19:24:49.982 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 48123 2024-09-10T19:24:49.982 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:24:49.982 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-10T19:24:49.982 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:24:49.982 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-10T19:24:49.982 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:24:49.919732+0000", 2024-09-10T19:24:49.982 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:24:49.982 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:24:49.982 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:24:49.983 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-10T19:24:49.983 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:24:49.919732+0000", 2024-09-10T19:24:49.983 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:24:49.983 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:24:49.983 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:24:49.983 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-10T19:24:49.983 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:24:49.919732+0000", 2024-09-10T19:24:49.983 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999995 2024-09-10T19:24:49.983 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:24:49.983 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:24:49.983 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-10T19:24:49.983 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:24:49.919801+0000", 2024-09-10T19:24:49.983 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.9072999999999994e-05 2024-09-10T19:24:49.984 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:24:49.984 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:24:49.984 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-10T19:24:49.984 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:24:49.919808+0000", 2024-09-10T19:24:49.984 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.6550000000000004e-06 2024-09-10T19:24:49.984 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:24:49.984 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:24:49.984 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-10T19:24:49.984 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:24:49.919813+0000", 2024-09-10T19:24:49.984 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.7809999999999996e-06 2024-09-10T19:24:49.984 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:24:49.984 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:24:49.984 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:24:49.984 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:24:49.919879+0000", 2024-09-10T19:24:49.985 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.5703000000000004e-05 2024-09-10T19:24:49.985 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:24:49.985 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:24:49.985 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-10T19:24:49.985 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:24:49.920009+0000", 2024-09-10T19:24:49.985 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00013028900000000001 2024-09-10T19:24:49.985 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:24:49.985 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:24:49.985 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 3,4", 2024-09-10T19:24:49.985 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:24:49.920230+0000", 2024-09-10T19:24:49.985 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00022096099999999999 2024-09-10T19:24:49.985 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:24:49.985 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:24:49.986 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:24:49.986 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:24:49.925898+0000", 2024-09-10T19:24:49.986 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.005668101 2024-09-10T19:24:49.986 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:24:49.986 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:24:49.986 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-09-10T19:24:49.986 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:24:49.926136+0000", 2024-09-10T19:24:49.986 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00023798800000000001 2024-09-10T19:24:49.986 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:24:49.986 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-10T19:24:49.986 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:24:49.986 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:24:49.986 INFO:teuthology.orchestra.run.smithi191.stdout: ], 2024-09-10T19:24:49.986 INFO:teuthology.orchestra.run.smithi191.stdout: "num_ops": 14 2024-09-10T19:24:49.987 INFO:teuthology.orchestra.run.smithi191.stdout:} 2024-09-10T19:24:50.020 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:24:50.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:24:50.021 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:24:50.122 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:24:50.159 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:24:50.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:24:50.161 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:24:50.262 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:24:50.289 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:24:50.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:24:50.290 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:24:50.391 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:24:50.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:24:50.426 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:24:50.426 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:24:50.527 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:24:50.566 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:24:50.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:24:50.567 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:24:50.667 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:24:50.704 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:24:50.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:24:50.705 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:24:50.806 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:24:50.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:24:50.835 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:24:50.835 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:24:50.855 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: 30 16 49597 49581 103.283 98.1875 0.01002 0.00967675 2024-09-10T19:24:50.894 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:24:50 smithi183 ceph-mon[29736]: 2.10 scrub starts 2024-09-10T19:24:50.894 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:24:50 smithi183 ceph-mon[29736]: 2.10 scrub ok 2024-09-10T19:24:50.894 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:24:50 smithi183 ceph-mon[29736]: from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:24:50.894 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:24:50 smithi183 ceph-mon[29736]: 6.15 scrub starts 2024-09-10T19:24:50.894 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:24:50 smithi183 ceph-mon[29736]: 6.9 scrub starts 2024-09-10T19:24:50.894 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:24:50 smithi183 ceph-mon[29736]: 6.9 scrub ok 2024-09-10T19:24:50.895 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:24:50 smithi183 ceph-mon[29736]: pgmap v669: 158 pgs: 158 active+clean; 2.5 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 853 B/s rd, 91 MiB/s wr, 1.46k op/s; 36 B/s, 1 objects/s recovering 2024-09-10T19:24:50.895 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:24:50 smithi183 ceph-mon[29736]: osdmap e317: 8 total, 8 up, 7 in 2024-09-10T19:24:50.895 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:24:50 smithi183 ceph-mon[41562]: 2.10 scrub starts 2024-09-10T19:24:50.895 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:24:50 smithi183 ceph-mon[41562]: 2.10 scrub ok 2024-09-10T19:24:50.895 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:24:50 smithi183 ceph-mon[41562]: from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:24:50.895 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:24:50 smithi183 ceph-mon[41562]: 6.15 scrub starts 2024-09-10T19:24:50.895 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:24:50 smithi183 ceph-mon[41562]: 6.9 scrub starts 2024-09-10T19:24:50.895 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:24:50 smithi183 ceph-mon[41562]: 6.9 scrub ok 2024-09-10T19:24:50.895 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:24:50 smithi183 ceph-mon[41562]: pgmap v669: 158 pgs: 158 active+clean; 2.5 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 853 B/s rd, 91 MiB/s wr, 1.46k op/s; 36 B/s, 1 objects/s recovering 2024-09-10T19:24:50.895 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:24:50 smithi183 ceph-mon[41562]: osdmap e317: 8 total, 8 up, 7 in 2024-09-10T19:24:50.936 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:24:50.956 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:24:50.974 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:24:50.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:24:50.975 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:24:51.075 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:24:51.108 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:24:51.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:24:51.112 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:24:51.214 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:24:51.217 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:24:50 smithi191 ceph-mon[32884]: 2.10 scrub starts 2024-09-10T19:24:51.217 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:24:50 smithi191 ceph-mon[32884]: 2.10 scrub ok 2024-09-10T19:24:51.217 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:24:50 smithi191 ceph-mon[32884]: from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:24:51.217 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:24:50 smithi191 ceph-mon[32884]: 6.15 scrub starts 2024-09-10T19:24:51.217 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:24:50 smithi191 ceph-mon[32884]: 6.9 scrub starts 2024-09-10T19:24:51.218 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:24:50 smithi191 ceph-mon[32884]: 6.9 scrub ok 2024-09-10T19:24:51.218 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:24:50 smithi191 ceph-mon[32884]: pgmap v669: 158 pgs: 158 active+clean; 2.5 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 853 B/s rd, 91 MiB/s wr, 1.46k op/s; 36 B/s, 1 objects/s recovering 2024-09-10T19:24:51.218 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:24:50 smithi191 ceph-mon[32884]: osdmap e317: 8 total, 8 up, 7 in 2024-09-10T19:24:51.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:24:51.251 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:24:51.251 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:24:51.352 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:24:51.384 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:24:51.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:24:51.385 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:24:51.486 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:24:51.498 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Sep 10 19:24:51 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-y[29989]: ::ffff:172.21.15.191 - - [10/Sep/2024:19:24:51] "GET /metrics HTTP/1.1" 200 224660 "" "Prometheus/2.43.0" 2024-09-10T19:24:51.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:24:51.522 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:24:51.522 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:24:51.623 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:24:51.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:24:51.662 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:24:51.662 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:24:51.763 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:24:51.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:24:51.799 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:24:51.800 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:24:51.829 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:24:51 smithi183 ceph-mon[29736]: 6.15 scrub ok 2024-09-10T19:24:51.829 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:24:51 smithi183 ceph-mon[29736]: 2.7 scrub starts 2024-09-10T19:24:51.829 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:24:51 smithi183 ceph-mon[29736]: 2.7 scrub ok 2024-09-10T19:24:51.829 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:24:51 smithi183 ceph-mon[29736]: 4.11 scrub starts 2024-09-10T19:24:51.829 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:24:51 smithi183 ceph-mon[29736]: 4.11 scrub ok 2024-09-10T19:24:51.830 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:24:51 smithi183 ceph-mon[41562]: 6.15 scrub ok 2024-09-10T19:24:51.830 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:24:51 smithi183 ceph-mon[41562]: 2.7 scrub starts 2024-09-10T19:24:51.830 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:24:51 smithi183 ceph-mon[41562]: 2.7 scrub ok 2024-09-10T19:24:51.830 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:24:51 smithi183 ceph-mon[41562]: 4.11 scrub starts 2024-09-10T19:24:51.830 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:24:51 smithi183 ceph-mon[41562]: 4.11 scrub ok 2024-09-10T19:24:51.856 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: 31 16 51077 51061 102.934 92.5 0.00698391 0.0097098 2024-09-10T19:24:51.900 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:24:51.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:24:51.939 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:24:51.940 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:24:52.040 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:24:52.068 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:24:52.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:24:52.071 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:24:52.083 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:24:51 smithi191 ceph-mon[32884]: 6.15 scrub ok 2024-09-10T19:24:52.083 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:24:51 smithi191 ceph-mon[32884]: 2.7 scrub starts 2024-09-10T19:24:52.083 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:24:51 smithi191 ceph-mon[32884]: 2.7 scrub ok 2024-09-10T19:24:52.083 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:24:51 smithi191 ceph-mon[32884]: 4.11 scrub starts 2024-09-10T19:24:52.083 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:24:51 smithi191 ceph-mon[32884]: 4.11 scrub ok 2024-09-10T19:24:52.124 DEBUG:teuthology.orchestra.run.smithi191:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph daemon osd.6 dump_blocked_ops 2024-09-10T19:24:52.172 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:24:52.197 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:24:52.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:24:52.199 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:24:52.299 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:24:52.325 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:24:52.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:24:52.326 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:24:52.329 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-10T19:24:52.427 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:24:52.456 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:24:52.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:24:52.457 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:24:52.558 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:24:52.573 INFO:teuthology.orchestra.run.smithi191.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.b/config 2024-09-10T19:24:52.614 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:24:52.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:24:52.615 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:24:52.716 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:24:52.739 INFO:teuthology.orchestra.run.smithi183.stderr:set 2.8 pg_upmap_items mapping to [0->5,1->3,7->4] 2024-09-10T19:24:52.758 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:24:52.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:24:52.759 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:24:52.856 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: 32 16 52541 52525 102.577 91.5 0.00571552 0.00974349 2024-09-10T19:24:52.860 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:24:52.889 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:24:52.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:24:52.890 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:24:52.966 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:24:52 smithi191 ceph-mon[32884]: 1.4 deep-scrub starts 2024-09-10T19:24:52.966 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:24:52 smithi191 ceph-mon[32884]: 1.4 deep-scrub ok 2024-09-10T19:24:52.966 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:24:52 smithi191 ceph-mon[32884]: pgmap v671: 158 pgs: 158 active+clean; 2.9 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 732 B/s rd, 130 MiB/s wr, 2.08k op/s; 31 B/s, 1 objects/s recovering 2024-09-10T19:24:52.966 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:24:52 smithi191 ceph-mon[32884]: from='client.? 172.21.15.183:0/186271119' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "2.8", "id": [0, 5, 1, 3, 7, 4]}]: dispatch 2024-09-10T19:24:52.966 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:24:52 smithi191 ceph-mon[32884]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "2.8", "id": [0, 5, 1, 3, 7, 4]}]: dispatch 2024-09-10T19:24:52.991 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:24:53.028 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:24:53.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:24:53.029 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:24:53.130 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:24:53.137 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:24:52 smithi183 ceph-mon[41562]: 1.4 deep-scrub starts 2024-09-10T19:24:53.137 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:24:52 smithi183 ceph-mon[41562]: 1.4 deep-scrub ok 2024-09-10T19:24:53.137 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:24:52 smithi183 ceph-mon[41562]: pgmap v671: 158 pgs: 158 active+clean; 2.9 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 732 B/s rd, 130 MiB/s wr, 2.08k op/s; 31 B/s, 1 objects/s recovering 2024-09-10T19:24:53.137 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:24:52 smithi183 ceph-mon[41562]: from='client.? 172.21.15.183:0/186271119' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "2.8", "id": [0, 5, 1, 3, 7, 4]}]: dispatch 2024-09-10T19:24:53.138 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:24:52 smithi183 ceph-mon[41562]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "2.8", "id": [0, 5, 1, 3, 7, 4]}]: dispatch 2024-09-10T19:24:53.138 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:24:52 smithi183 ceph-mon[29736]: 1.4 deep-scrub starts 2024-09-10T19:24:53.138 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:24:52 smithi183 ceph-mon[29736]: 1.4 deep-scrub ok 2024-09-10T19:24:53.138 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:24:52 smithi183 ceph-mon[29736]: pgmap v671: 158 pgs: 158 active+clean; 2.9 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 732 B/s rd, 130 MiB/s wr, 2.08k op/s; 31 B/s, 1 objects/s recovering 2024-09-10T19:24:53.138 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:24:52 smithi183 ceph-mon[29736]: from='client.? 172.21.15.183:0/186271119' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "2.8", "id": [0, 5, 1, 3, 7, 4]}]: dispatch 2024-09-10T19:24:53.138 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:24:52 smithi183 ceph-mon[29736]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "2.8", "id": [0, 5, 1, 3, 7, 4]}]: dispatch 2024-09-10T19:24:53.171 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:24:53.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:24:53.172 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:24:53.273 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:24:53.311 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:24:53.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:24:53.312 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:24:53.413 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:24:53.441 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:24:53.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:24:53.442 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:24:53.543 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:24:53.571 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:24:53.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:24:53.572 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:24:53.673 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:24:53.703 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:24:53.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:24:53.704 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:24:53.804 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:24:53.831 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:24:53.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:24:53.833 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:24:53.856 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: 33 16 54076 54060 102.375 95.9375 0.00577361 0.00976172 2024-09-10T19:24:53.861 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:24:53.875 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:24:53 smithi183 ceph-mon[41562]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "2.8", "id": [0, 5, 1, 3, 7, 4]}]': finished 2024-09-10T19:24:53.876 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:24:53 smithi183 ceph-mon[41562]: osdmap e318: 8 total, 8 up, 7 in 2024-09-10T19:24:53.876 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:24:53 smithi183 ceph-mon[41562]: 2.f scrub starts 2024-09-10T19:24:53.876 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:24:53 smithi183 ceph-mon[41562]: 2.f scrub ok 2024-09-10T19:24:53.876 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:24:53 smithi183 ceph-mon[29736]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "2.8", "id": [0, 5, 1, 3, 7, 4]}]': finished 2024-09-10T19:24:53.876 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:24:53 smithi183 ceph-mon[29736]: osdmap e318: 8 total, 8 up, 7 in 2024-09-10T19:24:53.876 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:24:53 smithi183 ceph-mon[29736]: 2.f scrub starts 2024-09-10T19:24:53.876 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:24:53 smithi183 ceph-mon[29736]: 2.f scrub ok 2024-09-10T19:24:53.934 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:24:53.964 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:24:53.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:24:53.965 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:24:53.966 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:24:53 smithi191 ceph-mon[32884]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "2.8", "id": [0, 5, 1, 3, 7, 4]}]': finished 2024-09-10T19:24:53.966 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:24:53 smithi191 ceph-mon[32884]: osdmap e318: 8 total, 8 up, 7 in 2024-09-10T19:24:53.966 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:24:53 smithi191 ceph-mon[32884]: 2.f scrub starts 2024-09-10T19:24:53.966 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:24:53 smithi191 ceph-mon[32884]: 2.f scrub ok 2024-09-10T19:24:54.067 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:24:54.094 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:24:54.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:24:54.095 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:24:54.196 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:24:54.226 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:24:54.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:24:54.227 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:24:54.327 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:24:54.358 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:24:54.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:24:54.359 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:24:54.460 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:24:54.506 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:24:54.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:24:54.507 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:24:54.608 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:24:54.644 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:24:54.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:24:54.645 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:24:54.746 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:24:54.756 INFO:teuthology.orchestra.run.smithi183.stderr:noscrub is set 2024-09-10T19:24:54.757 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:24:54 smithi183 ceph-mon[29736]: 4.10 scrub starts 2024-09-10T19:24:54.757 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:24:54 smithi183 ceph-mon[29736]: 4.10 scrub ok 2024-09-10T19:24:54.757 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:24:54 smithi183 ceph-mon[29736]: osdmap e319: 8 total, 8 up, 7 in 2024-09-10T19:24:54.757 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:24:54 smithi183 ceph-mon[29736]: pgmap v674: 158 pgs: 158 active+clean; 3.0 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 853 B/s rd, 81 MiB/s wr, 1.29k op/s 2024-09-10T19:24:54.757 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:24:54 smithi183 ceph-mon[29736]: 5.a scrub starts 2024-09-10T19:24:54.757 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:24:54 smithi183 ceph-mon[29736]: 5.a scrub ok 2024-09-10T19:24:54.757 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:24:54 smithi183 ceph-mon[29736]: from='client.? 172.21.15.183:0/1358867505' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-10T19:24:54.758 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:24:54 smithi183 ceph-mon[41562]: 4.10 scrub starts 2024-09-10T19:24:54.758 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:24:54 smithi183 ceph-mon[41562]: 4.10 scrub ok 2024-09-10T19:24:54.758 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:24:54 smithi183 ceph-mon[41562]: osdmap e319: 8 total, 8 up, 7 in 2024-09-10T19:24:54.758 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:24:54 smithi183 ceph-mon[41562]: pgmap v674: 158 pgs: 158 active+clean; 3.0 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 853 B/s rd, 81 MiB/s wr, 1.29k op/s 2024-09-10T19:24:54.758 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:24:54 smithi183 ceph-mon[41562]: 5.a scrub starts 2024-09-10T19:24:54.758 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:24:54 smithi183 ceph-mon[41562]: 5.a scrub ok 2024-09-10T19:24:54.758 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:24:54 smithi183 ceph-mon[41562]: from='client.? 172.21.15.183:0/1358867505' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-10T19:24:54.779 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:24:54.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:24:54.780 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:24:54.809 INFO:teuthology.orchestra.run.smithi191.stdout:{ 2024-09-10T19:24:54.809 INFO:teuthology.orchestra.run.smithi191.stdout: "ops": [], 2024-09-10T19:24:54.809 INFO:teuthology.orchestra.run.smithi191.stdout: "complaint_time": 30, 2024-09-10T19:24:54.809 INFO:teuthology.orchestra.run.smithi191.stdout: "num_blocked_ops": 0 2024-09-10T19:24:54.809 INFO:teuthology.orchestra.run.smithi191.stdout:} 2024-09-10T19:24:54.856 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: 34 16 55541 55525 102.057 91.5625 0.00412439 0.0097941 2024-09-10T19:24:54.880 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:24:54.911 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:24:54.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:24:54.912 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:24:54.966 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:24:54 smithi191 ceph-mon[32884]: 4.10 scrub starts 2024-09-10T19:24:54.966 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:24:54 smithi191 ceph-mon[32884]: 4.10 scrub ok 2024-09-10T19:24:54.966 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:24:54 smithi191 ceph-mon[32884]: osdmap e319: 8 total, 8 up, 7 in 2024-09-10T19:24:54.966 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:24:54 smithi191 ceph-mon[32884]: pgmap v674: 158 pgs: 158 active+clean; 3.0 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 853 B/s rd, 81 MiB/s wr, 1.29k op/s 2024-09-10T19:24:54.966 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:24:54 smithi191 ceph-mon[32884]: 5.a scrub starts 2024-09-10T19:24:54.966 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:24:54 smithi191 ceph-mon[32884]: 5.a scrub ok 2024-09-10T19:24:54.966 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:24:54 smithi191 ceph-mon[32884]: from='client.? 172.21.15.183:0/1358867505' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-10T19:24:55.012 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:24:55.050 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:24:55.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:24:55.051 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:24:55.152 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:24:55.191 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:24:55.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:24:55.192 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:24:55.292 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:24:55.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:24:55.322 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:24:55.322 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:24:55.423 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:24:55.459 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:24:55.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:24:55.460 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:24:55.561 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:24:55.588 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:24:55.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:24:55.589 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:24:55.633 DEBUG:teuthology.orchestra.run.smithi191:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph daemon osd.6 dump_historic_ops 2024-09-10T19:24:55.690 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:24:55.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:24:55.741 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:24:55.742 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:24:55.842 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:24:55.856 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: 35 15 56997 56982 101.743 91.0625 0.011706 0.00982226 2024-09-10T19:24:55.868 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:24:55.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:24:55.869 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:24:55.931 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:24:55 smithi183 ceph-mon[29736]: 2.1c scrub starts 2024-09-10T19:24:55.931 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:24:55 smithi183 ceph-mon[29736]: 2.1c scrub ok 2024-09-10T19:24:55.932 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:24:55 smithi183 ceph-mon[29736]: 5.19 scrub starts 2024-09-10T19:24:55.932 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:24:55 smithi183 ceph-mon[29736]: 5.19 scrub ok 2024-09-10T19:24:55.932 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:24:55 smithi183 ceph-mon[29736]: 4.17 scrub starts 2024-09-10T19:24:55.932 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:24:55 smithi183 ceph-mon[29736]: 4.17 scrub ok 2024-09-10T19:24:55.932 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:24:55 smithi183 ceph-mon[29736]: purged_snaps scrub starts 2024-09-10T19:24:55.932 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:24:55 smithi183 ceph-mon[29736]: purged_snaps scrub ok 2024-09-10T19:24:55.932 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:24:55 smithi183 ceph-mon[29736]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T19:24:55.932 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:24:55 smithi183 ceph-mon[29736]: from='client.? 172.21.15.183:0/1358867505' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-10T19:24:55.932 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:24:55 smithi183 ceph-mon[29736]: osdmap e320: 8 total, 8 up, 7 in 2024-09-10T19:24:55.932 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:24:55 smithi183 ceph-mon[29736]: osdmap e321: 8 total, 8 up, 7 in 2024-09-10T19:24:55.933 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:24:55 smithi183 ceph-mon[41562]: 2.1c scrub starts 2024-09-10T19:24:55.933 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:24:55 smithi183 ceph-mon[41562]: 2.1c scrub ok 2024-09-10T19:24:55.933 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:24:55 smithi183 ceph-mon[41562]: 5.19 scrub starts 2024-09-10T19:24:55.933 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:24:55 smithi183 ceph-mon[41562]: 5.19 scrub ok 2024-09-10T19:24:55.933 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:24:55 smithi183 ceph-mon[41562]: 4.17 scrub starts 2024-09-10T19:24:55.933 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:24:55 smithi183 ceph-mon[41562]: 4.17 scrub ok 2024-09-10T19:24:55.933 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:24:55 smithi183 ceph-mon[41562]: purged_snaps scrub starts 2024-09-10T19:24:55.933 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:24:55 smithi183 ceph-mon[41562]: purged_snaps scrub ok 2024-09-10T19:24:55.933 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:24:55 smithi183 ceph-mon[41562]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T19:24:55.933 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:24:55 smithi183 ceph-mon[41562]: from='client.? 172.21.15.183:0/1358867505' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-10T19:24:55.933 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:24:55 smithi183 ceph-mon[41562]: osdmap e320: 8 total, 8 up, 7 in 2024-09-10T19:24:55.934 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:24:55 smithi183 ceph-mon[41562]: osdmap e321: 8 total, 8 up, 7 in 2024-09-10T19:24:55.966 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:24:55 smithi191 ceph-mon[32884]: 2.1c scrub starts 2024-09-10T19:24:55.966 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:24:55 smithi191 ceph-mon[32884]: 2.1c scrub ok 2024-09-10T19:24:55.966 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:24:55 smithi191 ceph-mon[32884]: 5.19 scrub starts 2024-09-10T19:24:55.966 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:24:55 smithi191 ceph-mon[32884]: 5.19 scrub ok 2024-09-10T19:24:55.966 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:24:55 smithi191 ceph-mon[32884]: 4.17 scrub starts 2024-09-10T19:24:55.966 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:24:55 smithi191 ceph-mon[32884]: 4.17 scrub ok 2024-09-10T19:24:55.966 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:24:55 smithi191 ceph-mon[32884]: purged_snaps scrub starts 2024-09-10T19:24:55.966 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:24:55 smithi191 ceph-mon[32884]: purged_snaps scrub ok 2024-09-10T19:24:55.966 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:24:55 smithi191 ceph-mon[32884]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T19:24:55.966 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:24:55 smithi191 ceph-mon[32884]: from='client.? 172.21.15.183:0/1358867505' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-10T19:24:55.967 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:24:55 smithi191 ceph-mon[32884]: osdmap e320: 8 total, 8 up, 7 in 2024-09-10T19:24:55.967 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:24:55 smithi191 ceph-mon[32884]: osdmap e321: 8 total, 8 up, 7 in 2024-09-10T19:24:55.969 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:24:56.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:24:56.017 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:24:56.017 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:24:56.093 INFO:teuthology.orchestra.run.smithi191.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.b/config 2024-09-10T19:24:56.118 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:24:56.146 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:24:56.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:24:56.147 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:24:56.248 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:24:56.276 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:24:56.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:24:56.277 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:24:56.378 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:24:56.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:24:56.421 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:24:56.421 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:24:56.522 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:24:56.576 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:24:56.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:24:56.577 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:24:56.678 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:24:56.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:24:56.706 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:24:56.706 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:24:56.807 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:24:56.835 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:24:56 smithi191 ceph-mon[32884]: pgmap v677: 158 pgs: 158 active+clean; 3.3 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 1.2 KiB/s rd, 120 MiB/s wr, 1.91k op/s 2024-09-10T19:24:56.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:24:56.841 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:24:56.841 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:24:56.856 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: 36 16 58429 58413 101.401 89.4375 0.010426 0.00985751 2024-09-10T19:24:56.943 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:24:56.978 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:24:56.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:24:56.979 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:24:57.012 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:24:56 smithi183 ceph-mon[29736]: pgmap v677: 158 pgs: 158 active+clean; 3.3 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 1.2 KiB/s rd, 120 MiB/s wr, 1.91k op/s 2024-09-10T19:24:57.013 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:24:56 smithi183 ceph-mon[41562]: pgmap v677: 158 pgs: 158 active+clean; 3.3 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 1.2 KiB/s rd, 120 MiB/s wr, 1.91k op/s 2024-09-10T19:24:57.047 INFO:teuthology.orchestra.run.smithi183.stdout:osd.0: {} 2024-09-10T19:24:57.047 INFO:teuthology.orchestra.run.smithi183.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-10T19:24:57.072 INFO:teuthology.orchestra.run.smithi183.stdout:osd.1: {} 2024-09-10T19:24:57.072 INFO:teuthology.orchestra.run.smithi183.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-10T19:24:57.080 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:24:57.093 INFO:teuthology.orchestra.run.smithi183.stdout:osd.2: {} 2024-09-10T19:24:57.093 INFO:teuthology.orchestra.run.smithi183.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-10T19:24:57.115 INFO:teuthology.orchestra.run.smithi183.stdout:osd.3: {} 2024-09-10T19:24:57.115 INFO:teuthology.orchestra.run.smithi183.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-10T19:24:57.119 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:24:57.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:24:57.121 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:24:57.138 INFO:teuthology.orchestra.run.smithi183.stdout:osd.4: {} 2024-09-10T19:24:57.138 INFO:teuthology.orchestra.run.smithi183.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-10T19:24:57.162 INFO:teuthology.orchestra.run.smithi183.stdout:osd.5: {} 2024-09-10T19:24:57.162 INFO:teuthology.orchestra.run.smithi183.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-10T19:24:57.197 INFO:teuthology.orchestra.run.smithi183.stdout:osd.6: {} 2024-09-10T19:24:57.197 INFO:teuthology.orchestra.run.smithi183.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-10T19:24:57.221 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:24:57.222 INFO:teuthology.orchestra.run.smithi183.stdout:osd.7: {} 2024-09-10T19:24:57.222 INFO:teuthology.orchestra.run.smithi183.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-10T19:24:57.259 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:24:57.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:24:57.261 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:24:57.362 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:24:57.388 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:24:57.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:24:57.390 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:24:57.490 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:24:57.522 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:24:57.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:24:57.523 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:24:57.624 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:24:57.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:24:57.662 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:24:57.662 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:24:57.701 INFO:journalctl@ceph.mgr.x.smithi191.stdout:Sep 10 19:24:57 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-x[34967]: ::ffff:172.21.15.191 - - [10/Sep/2024:19:24:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T19:24:57.764 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:24:57.790 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:24:57.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:24:57.793 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:24:57.856 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: 37 16 59874 59858 101.101 90.3125 0.0140076 0.00988603 2024-09-10T19:24:57.895 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:24:57.922 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:24:57.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:24:57.923 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:24:57.931 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:24:57 smithi183 ceph-mon[29736]: from='client.? 172.21.15.183:0/3587066125' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T19:24:57.931 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:24:57 smithi183 ceph-mon[41562]: from='client.? 172.21.15.183:0/3587066125' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T19:24:57.965 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:24:57 smithi191 ceph-mon[32884]: from='client.? 172.21.15.183:0/3587066125' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T19:24:58.024 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:24:58.056 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:24:58.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:24:58.057 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:24:58.158 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:24:58.198 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:24:58.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:24:58.199 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:24:58.300 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:24:58.324 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:24:58.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:24:58.325 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:24:58.426 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:24:58.450 INFO:teuthology.orchestra.run.smithi191.stdout:ERROR: (22) Invalid argument 2024-09-10T19:24:58.450 INFO:teuthology.orchestra.run.smithi191.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-10T19:24:58.451 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:24:58.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:24:58.452 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:24:58.510 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-10T19:24:58.552 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:24:58.582 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:24:58.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:24:58.583 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:24:58.612 INFO:tasks.thrashosds.thrasher:in_osds: [0, 3, 4, 5, 6, 7, 1] out_osds: [2] dead_osds: [] live_osds: [2, 3, 4, 5, 6, 7, 1, 0] 2024-09-10T19:24:58.612 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-10T19:24:58.612 INFO:tasks.thrashosds.thrasher:primary_affinity 2024-09-10T19:24:58.612 INFO:tasks.thrashosds.thrasher:Setting osd 3 primary_affinity to 1.000000 2024-09-10T19:24:58.612 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph osd primary-affinity 3 1 2024-09-10T19:24:58.684 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:24:58.723 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:24:58.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:24:58.724 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:24:58.825 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:24:58.856 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: 38 16 61123 61107 100.494 78.0625 0.0193352 0.00994506 2024-09-10T19:24:58.857 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:24:58.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:24:58.858 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:24:58.859 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:24:58.931 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:24:58 smithi183 ceph-mon[29736]: purged_snaps scrub starts 2024-09-10T19:24:58.931 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:24:58 smithi183 ceph-mon[29736]: purged_snaps scrub ok 2024-09-10T19:24:58.932 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:24:58 smithi183 ceph-mon[29736]: osdmap e322: 8 total, 8 up, 7 in 2024-09-10T19:24:58.932 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:24:58 smithi183 ceph-mon[29736]: pgmap v679: 158 pgs: 158 active+clean; 3.5 GiB data, 18 GiB used, 607 GiB / 626 GiB avail; 1.2 KiB/s rd, 114 MiB/s wr, 1.82k op/s 2024-09-10T19:24:58.932 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:24:58 smithi183 ceph-mon[29736]: purged_snaps scrub starts 2024-09-10T19:24:58.932 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:24:58 smithi183 ceph-mon[29736]: purged_snaps scrub ok 2024-09-10T19:24:58.932 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:24:58 smithi183 ceph-mon[29736]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:24:58.932 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:24:58 smithi183 ceph-mon[41562]: purged_snaps scrub starts 2024-09-10T19:24:58.932 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:24:58 smithi183 ceph-mon[41562]: purged_snaps scrub ok 2024-09-10T19:24:58.932 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:24:58 smithi183 ceph-mon[41562]: osdmap e322: 8 total, 8 up, 7 in 2024-09-10T19:24:58.932 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:24:58 smithi183 ceph-mon[41562]: pgmap v679: 158 pgs: 158 active+clean; 3.5 GiB data, 18 GiB used, 607 GiB / 626 GiB avail; 1.2 KiB/s rd, 114 MiB/s wr, 1.82k op/s 2024-09-10T19:24:58.932 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:24:58 smithi183 ceph-mon[41562]: purged_snaps scrub starts 2024-09-10T19:24:58.933 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:24:58 smithi183 ceph-mon[41562]: purged_snaps scrub ok 2024-09-10T19:24:58.933 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:24:58 smithi183 ceph-mon[41562]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:24:58.958 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:24:58.988 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:24:58.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:24:58.990 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:24:59.090 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:24:59.115 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:24:59.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:24:59.116 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:24:59.124 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:24:58 smithi191 ceph-mon[32884]: purged_snaps scrub starts 2024-09-10T19:24:59.124 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:24:58 smithi191 ceph-mon[32884]: purged_snaps scrub ok 2024-09-10T19:24:59.124 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:24:58 smithi191 ceph-mon[32884]: osdmap e322: 8 total, 8 up, 7 in 2024-09-10T19:24:59.124 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:24:58 smithi191 ceph-mon[32884]: pgmap v679: 158 pgs: 158 active+clean; 3.5 GiB data, 18 GiB used, 607 GiB / 626 GiB avail; 1.2 KiB/s rd, 114 MiB/s wr, 1.82k op/s 2024-09-10T19:24:59.124 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:24:58 smithi191 ceph-mon[32884]: purged_snaps scrub starts 2024-09-10T19:24:59.124 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:24:58 smithi191 ceph-mon[32884]: purged_snaps scrub ok 2024-09-10T19:24:59.124 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:24:58 smithi191 ceph-mon[32884]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:24:59.217 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:24:59.245 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:24:59.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:24:59.246 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:24:59.347 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:24:59.361 DEBUG:teuthology.orchestra.run.smithi191:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph daemon osd.7 dump_ops_in_flight 2024-09-10T19:24:59.374 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:24:59.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:24:59.375 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:24:59.476 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:24:59.520 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:24:59.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:24:59.521 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:24:59.622 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:24:59.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:24:59.661 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:24:59.661 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:24:59.716 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:24:59.762 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:24:59.789 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:24:59.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:24:59.790 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:24:59.810 INFO:teuthology.orchestra.run.smithi191.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.b/config 2024-09-10T19:24:59.856 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: 39 16 62325 62309 99.8437 75.125 0.00467961 0.0100101 2024-09-10T19:24:59.891 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:24:59.920 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:24:59.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:24:59.921 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:25:00.022 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:25:00.061 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:25:00.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:25:00.062 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:25:00.076 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-10T19:25:00.105 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:24:59 smithi183 ceph-mon[29736]: osdmap e323: 8 total, 8 up, 7 in 2024-09-10T19:25:00.105 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:24:59 smithi183 ceph-mon[41562]: osdmap e323: 8 total, 8 up, 7 in 2024-09-10T19:25:00.163 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:25:00.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:25:00.198 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:25:00.198 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:25:00.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:24:59 smithi191 ceph-mon[32884]: osdmap e323: 8 total, 8 up, 7 in 2024-09-10T19:25:00.216 INFO:journalctl@ceph.iscsi.iscsi.a.smithi191.stdout:Sep 10 19:24:59 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-iscsi-iscsi-a[61816]: debug there is no tcmu-runner data available 2024-09-10T19:25:00.299 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:25:00.331 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:25:00.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:25:00.332 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:25:00.433 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:25:00.458 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:25:00.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:25:00.459 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:25:00.560 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:25:00.600 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:25:00.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:25:00.601 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:25:00.702 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:25:00.728 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:25:00.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:25:00.729 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:25:00.830 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:25:00.850 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:25:00.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:25:00.851 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:25:00.856 INFO:tasks.radosbench.radosbench.0.smithi183.stdout:2024-09-10T19:25:00.855937+0000 min lat: 0.00110744 max lat: 2.06031 avg lat: 0.0100507 2024-09-10T19:25:00.856 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-10T19:25:00.856 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: 40 16 63629 63613 99.3851 81.5 0.0572002 0.0100507 2024-09-10T19:25:00.952 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:25:00.966 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:25:00 smithi191 ceph-mon[32884]: from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:25:00.966 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:25:00 smithi191 ceph-mon[32884]: pgmap v681: 158 pgs: 158 active+clean; 3.5 GiB data, 18 GiB used, 607 GiB / 626 GiB avail; 999 B/s rd, 92 MiB/s wr, 1.47k op/s 2024-09-10T19:25:00.988 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:25:00.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:25:00.989 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:25:01.089 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:25:01.123 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:25:01.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:25:01.124 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:25:01.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:25:00 smithi183 ceph-mon[29736]: from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:25:01.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:25:00 smithi183 ceph-mon[29736]: pgmap v681: 158 pgs: 158 active+clean; 3.5 GiB data, 18 GiB used, 607 GiB / 626 GiB avail; 999 B/s rd, 92 MiB/s wr, 1.47k op/s 2024-09-10T19:25:01.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:25:00 smithi183 ceph-mon[41562]: from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:25:01.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:25:00 smithi183 ceph-mon[41562]: pgmap v681: 158 pgs: 158 active+clean; 3.5 GiB data, 18 GiB used, 607 GiB / 626 GiB avail; 999 B/s rd, 92 MiB/s wr, 1.47k op/s 2024-09-10T19:25:01.225 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:25:01.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:25:01.261 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:25:01.261 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:25:01.362 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:25:01.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:25:01.395 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:25:01.395 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:25:01.496 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:25:01.523 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:25:01.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:25:01.524 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:25:01.623 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:25:01.625 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:25:01.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:25:01.668 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:25:01.669 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:25:01.681 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Sep 10 19:25:01 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-y[29989]: ::ffff:172.21.15.191 - - [10/Sep/2024:19:25:01] "GET /metrics HTTP/1.1" 200 224729 "" "Prometheus/2.43.0" 2024-09-10T19:25:01.769 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:25:01.797 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:25:01.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:25:01.798 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:25:01.857 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: 41 16 64833 64817 98.7949 75.25 0.00481572 0.0101158 2024-09-10T19:25:01.899 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:25:01.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:25:01.946 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:25:01.946 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:25:02.047 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:25:02.073 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:25:02.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:25:02.074 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:25:02.175 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:25:02.197 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:25:02.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:25:02.198 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:25:02.299 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:25:02.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:25:02.338 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:25:02.339 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:25:02.364 INFO:teuthology.orchestra.run.smithi191.stdout:ERROR: (22) Invalid argument 2024-09-10T19:25:02.364 INFO:teuthology.orchestra.run.smithi191.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-10T19:25:02.440 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:25:02.479 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:25:02.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:25:02.480 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:25:02.580 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:25:02.620 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:25:02.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:25:02.621 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:25:02.722 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:25:02.744 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:25:02.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:25:02.746 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:25:02.846 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:25:02.857 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: 42 16 66015 65999 98.2014 73.875 0.00988779 0.0101791 2024-09-10T19:25:02.873 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:25:02.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:25:02.874 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:25:02.975 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:25:03.012 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:25:03.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:25:03.013 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:25:03.060 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:25:02 smithi191 ceph-mon[32884]: pgmap v682: 158 pgs: 158 active+clean; 3.8 GiB data, 19 GiB used, 606 GiB / 626 GiB avail; 732 B/s rd, 112 MiB/s wr, 1.79k op/s 2024-09-10T19:25:03.101 DEBUG:teuthology.orchestra.run.smithi191:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph daemon osd.7 dump_blocked_ops 2024-09-10T19:25:03.114 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:25:03.136 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:25:02 smithi183 ceph-mon[29736]: pgmap v682: 158 pgs: 158 active+clean; 3.8 GiB data, 19 GiB used, 606 GiB / 626 GiB avail; 732 B/s rd, 112 MiB/s wr, 1.79k op/s 2024-09-10T19:25:03.137 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:25:02 smithi183 ceph-mon[41562]: pgmap v682: 158 pgs: 158 active+clean; 3.8 GiB data, 19 GiB used, 606 GiB / 626 GiB avail; 732 B/s rd, 112 MiB/s wr, 1.79k op/s 2024-09-10T19:25:03.182 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:25:03.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:25:03.185 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:25:03.286 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:25:03.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:25:03.337 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:25:03.338 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:25:03.438 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:25:03.467 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:25:03.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:25:03.468 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:25:03.543 INFO:teuthology.orchestra.run.smithi191.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.b/config 2024-09-10T19:25:03.569 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:25:03.621 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:25:03.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:25:03.622 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:25:03.723 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:25:03.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:25:03.766 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:25:03.766 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:25:03.857 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: 43 16 67277 67261 97.7518 78.875 0.0205062 0.0102256 2024-09-10T19:25:03.867 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:25:03.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:25:03.909 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:25:03.909 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:25:03.946 INFO:teuthology.orchestra.run.smithi183.stderr:nodeep-scrub is set 2024-09-10T19:25:03.965 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:25:03 smithi191 ceph-mon[32884]: from='client.? 172.21.15.183:0/344443512' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-10T19:25:03.965 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:25:03 smithi191 ceph-mon[32884]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-10T19:25:04.010 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:25:04.044 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:25:04.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:25:04.045 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:25:04.146 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:25:04.161 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:25:03 smithi183 ceph-mon[29736]: from='client.? 172.21.15.183:0/344443512' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-10T19:25:04.161 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:25:03 smithi183 ceph-mon[29736]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-10T19:25:04.161 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:25:03 smithi183 ceph-mon[41562]: from='client.? 172.21.15.183:0/344443512' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-10T19:25:04.162 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:25:03 smithi183 ceph-mon[41562]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-10T19:25:04.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:25:04.184 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:25:04.185 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:25:04.285 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:25:04.312 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:25:04.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:25:04.313 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:25:04.414 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:25:04.442 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:25:04.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:25:04.444 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:25:04.544 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:25:04.584 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:25:04.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:25:04.585 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:25:04.686 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:25:04.720 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:25:04.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:25:04.721 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:25:04.822 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:25:04.848 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:25:04.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:25:04.849 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:25:04.857 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: 44 16 68602 68586 97.4121 82.8125 0.00507019 0.0102609 2024-09-10T19:25:04.950 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:25:04.978 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:25:04.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:25:04.979 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:25:05.080 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:25:05.105 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:25:05.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:25:05.106 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:25:05.180 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:25:04 smithi183 ceph-mon[29736]: pgmap v683: 158 pgs: 158 active+clean; 3.9 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 1.2 KiB/s rd, 66 MiB/s wr, 1.06k op/s 2024-09-10T19:25:05.180 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:25:04 smithi183 ceph-mon[29736]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T19:25:05.180 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:25:04 smithi183 ceph-mon[29736]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-10T19:25:05.180 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:25:04 smithi183 ceph-mon[29736]: osdmap e324: 8 total, 8 up, 7 in 2024-09-10T19:25:05.180 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:25:04 smithi183 ceph-mon[29736]: osdmap e325: 8 total, 8 up, 7 in 2024-09-10T19:25:05.181 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:25:04 smithi183 ceph-mon[41562]: pgmap v683: 158 pgs: 158 active+clean; 3.9 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 1.2 KiB/s rd, 66 MiB/s wr, 1.06k op/s 2024-09-10T19:25:05.181 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:25:04 smithi183 ceph-mon[41562]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T19:25:05.181 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:25:04 smithi183 ceph-mon[41562]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-10T19:25:05.181 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:25:04 smithi183 ceph-mon[41562]: osdmap e324: 8 total, 8 up, 7 in 2024-09-10T19:25:05.181 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:25:04 smithi183 ceph-mon[41562]: osdmap e325: 8 total, 8 up, 7 in 2024-09-10T19:25:05.207 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:25:05.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:25:04 smithi191 ceph-mon[32884]: pgmap v683: 158 pgs: 158 active+clean; 3.9 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 1.2 KiB/s rd, 66 MiB/s wr, 1.06k op/s 2024-09-10T19:25:05.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:25:04 smithi191 ceph-mon[32884]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T19:25:05.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:25:04 smithi191 ceph-mon[32884]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-10T19:25:05.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:25:04 smithi191 ceph-mon[32884]: osdmap e324: 8 total, 8 up, 7 in 2024-09-10T19:25:05.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:25:04 smithi191 ceph-mon[32884]: osdmap e325: 8 total, 8 up, 7 in 2024-09-10T19:25:05.238 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:25:05.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:25:05.239 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:25:05.340 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:25:05.361 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:25:05.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:25:05.362 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:25:05.463 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:25:05.491 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:25:05.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:25:05.492 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:25:05.592 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:25:05.622 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:25:05.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:25:05.623 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:25:05.724 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:25:05.753 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:25:05.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:25:05.754 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:25:05.855 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:25:05.858 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: 45 16 69873 69857 97.0126 79.4375 0.00486643 0.0103022 2024-09-10T19:25:05.882 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:25:05.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:25:05.884 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:25:05.984 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:25:06.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:25:06.006 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:25:06.006 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:25:06.107 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:25:06.135 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:25:06.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:25:06.137 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:25:06.238 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:25:06.266 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:25:06.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:25:06.267 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:25:06.368 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:25:06.403 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:25:06.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:25:06.404 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:25:06.505 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:25:06.541 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:25:06.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:25:06.542 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:25:06.643 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:25:06.664 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:25:06.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:25:06.665 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:25:06.766 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:25:06.797 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:25:06.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:25:06.798 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:25:06.857 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: 46 15 71028 71013 96.4741 72.25 0.008115 0.0103611 2024-09-10T19:25:06.899 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:25:06.937 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:25:06.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:25:06.938 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:25:06.941 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:25:06 smithi183 ceph-mon[41562]: pgmap v686: 158 pgs: 158 active+clean; 4.1 GiB data, 20 GiB used, 605 GiB / 626 GiB avail; 721 B/s rd, 100 MiB/s wr, 1.59k op/s 2024-09-10T19:25:06.941 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:25:06 smithi183 ceph-mon[29736]: pgmap v686: 158 pgs: 158 active+clean; 4.1 GiB data, 20 GiB used, 605 GiB / 626 GiB avail; 721 B/s rd, 100 MiB/s wr, 1.59k op/s 2024-09-10T19:25:07.040 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:25:07.061 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:25:07.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:25:07.062 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:25:07.163 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:25:07.188 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:25:07.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:25:07.189 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:25:07.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:25:06 smithi191 ceph-mon[32884]: pgmap v686: 158 pgs: 158 active+clean; 4.1 GiB data, 20 GiB used, 605 GiB / 626 GiB avail; 721 B/s rd, 100 MiB/s wr, 1.59k op/s 2024-09-10T19:25:07.290 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:25:07.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:25:07.332 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:25:07.332 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:25:07.433 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:25:07.462 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:25:07.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:25:07.463 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:25:07.564 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:25:07.585 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:25:07.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:25:07.586 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:25:07.649 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph osd unset noscrub 2024-09-10T19:25:07.687 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:25:07.715 INFO:journalctl@ceph.mgr.x.smithi191.stdout:Sep 10 19:25:07 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-x[34967]: ::ffff:172.21.15.191 - - [10/Sep/2024:19:25:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T19:25:07.723 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:25:07.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:25:07.724 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:25:07.825 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:25:07.857 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:25:07.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:25:07.858 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:25:07.858 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: 47 16 72014 71998 95.7313 61.5625 0.0108343 0.0104411 2024-09-10T19:25:07.959 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:25:08.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:25:08.001 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:25:08.001 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:25:08.102 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:25:08.130 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:25:08.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:25:08.131 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:25:08.232 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:25:08.269 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:25:08.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:25:08.270 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:25:08.354 INFO:teuthology.orchestra.run.smithi183.stdout:osd.0: {} 2024-09-10T19:25:08.355 INFO:teuthology.orchestra.run.smithi183.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-10T19:25:08.371 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:25:08.371 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:25:08.382 INFO:teuthology.orchestra.run.smithi183.stdout:osd.1: {} 2024-09-10T19:25:08.382 INFO:teuthology.orchestra.run.smithi183.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-10T19:25:08.405 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:25:08.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:25:08.406 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:25:08.411 INFO:teuthology.orchestra.run.smithi183.stdout:osd.2: {} 2024-09-10T19:25:08.411 INFO:teuthology.orchestra.run.smithi183.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-10T19:25:08.442 INFO:teuthology.orchestra.run.smithi183.stdout:osd.3: {} 2024-09-10T19:25:08.442 INFO:teuthology.orchestra.run.smithi183.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-10T19:25:08.465 INFO:teuthology.orchestra.run.smithi183.stdout:osd.4: {} 2024-09-10T19:25:08.465 INFO:teuthology.orchestra.run.smithi183.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-10T19:25:08.506 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:25:08.528 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:25:08.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:25:08.529 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:25:08.540 INFO:teuthology.orchestra.run.smithi183.stdout:osd.5: {} 2024-09-10T19:25:08.541 INFO:teuthology.orchestra.run.smithi183.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-10T19:25:08.573 INFO:teuthology.orchestra.run.smithi183.stdout:osd.6: {} 2024-09-10T19:25:08.573 INFO:teuthology.orchestra.run.smithi183.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-10T19:25:08.596 INFO:teuthology.orchestra.run.smithi183.stdout:osd.7: {} 2024-09-10T19:25:08.597 INFO:teuthology.orchestra.run.smithi183.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-10T19:25:08.630 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:25:08.662 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:25:08.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:25:08.663 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:25:08.764 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:25:08.797 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:25:08.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:25:08.798 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:25:08.858 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: 48 16 73244 73228 95.3383 76.875 0.0135358 0.0104835 2024-09-10T19:25:08.898 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:25:08.924 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:25:08.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:25:08.925 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:25:08.943 INFO:teuthology.orchestra.run.smithi183.stderr:set osd.3 primary-affinity to 1 (8655362) 2024-09-10T19:25:09.027 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:25:09.054 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:25:09.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:25:09.055 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:25:09.156 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:25:09.178 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:25:09.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:25:09.179 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:25:09.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:25:08 smithi191 ceph-mon[32884]: pgmap v687: 158 pgs: 158 active+clean; 4.2 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 1.2 KiB/s rd, 100 MiB/s wr, 1.60k op/s 2024-09-10T19:25:09.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:25:08 smithi191 ceph-mon[32884]: from='client.? 172.21.15.183:0/232208774' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T19:25:09.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:25:08 smithi191 ceph-mon[32884]: from='client.? 172.21.15.183:0/2132334543' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]: dispatch 2024-09-10T19:25:09.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:25:08 smithi191 ceph-mon[32884]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]: dispatch 2024-09-10T19:25:09.280 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:25:09.308 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:25:09.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:25:09.309 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:25:09.410 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:25:09.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:25:08 smithi183 ceph-mon[29736]: pgmap v687: 158 pgs: 158 active+clean; 4.2 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 1.2 KiB/s rd, 100 MiB/s wr, 1.60k op/s 2024-09-10T19:25:09.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:25:08 smithi183 ceph-mon[29736]: from='client.? 172.21.15.183:0/232208774' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T19:25:09.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:25:08 smithi183 ceph-mon[29736]: from='client.? 172.21.15.183:0/2132334543' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]: dispatch 2024-09-10T19:25:09.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:25:08 smithi183 ceph-mon[29736]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]: dispatch 2024-09-10T19:25:09.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:25:08 smithi183 ceph-mon[41562]: pgmap v687: 158 pgs: 158 active+clean; 4.2 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 1.2 KiB/s rd, 100 MiB/s wr, 1.60k op/s 2024-09-10T19:25:09.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:25:08 smithi183 ceph-mon[41562]: from='client.? 172.21.15.183:0/232208774' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T19:25:09.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:25:08 smithi183 ceph-mon[41562]: from='client.? 172.21.15.183:0/2132334543' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]: dispatch 2024-09-10T19:25:09.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:25:08 smithi183 ceph-mon[41562]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]: dispatch 2024-09-10T19:25:09.432 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:25:09.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:25:09.434 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:25:09.534 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:25:09.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:25:09.572 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:25:09.572 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:25:09.673 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:25:09.712 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:25:09.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:25:09.713 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:25:09.814 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:25:09.835 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:25:09.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:25:09.836 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:25:09.857 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: 49 16 74122 74106 94.5125 54.875 0.077528 0.0105651 2024-09-10T19:25:09.937 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:25:09.945 INFO:journalctl@ceph.iscsi.iscsi.a.smithi191.stdout:Sep 10 19:25:09 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-iscsi-iscsi-a[61816]: debug there is no tcmu-runner data available 2024-09-10T19:25:09.959 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:25:09.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:25:09.960 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:25:10.061 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:25:10.083 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:25:10.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:25:10.084 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:25:10.185 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:25:10.202 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:25:09 smithi183 ceph-mon[29736]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]': finished 2024-09-10T19:25:10.202 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:25:09 smithi183 ceph-mon[29736]: osdmap e326: 8 total, 8 up, 7 in 2024-09-10T19:25:10.202 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:25:09 smithi183 ceph-mon[29736]: osdmap e327: 8 total, 8 up, 7 in 2024-09-10T19:25:10.202 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:25:09 smithi183 ceph-mon[41562]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]': finished 2024-09-10T19:25:10.202 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:25:09 smithi183 ceph-mon[41562]: osdmap e326: 8 total, 8 up, 7 in 2024-09-10T19:25:10.202 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:25:09 smithi183 ceph-mon[41562]: osdmap e327: 8 total, 8 up, 7 in 2024-09-10T19:25:10.212 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:25:10.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:25:10.213 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:25:10.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:25:09 smithi191 ceph-mon[32884]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]': finished 2024-09-10T19:25:10.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:25:09 smithi191 ceph-mon[32884]: osdmap e326: 8 total, 8 up, 7 in 2024-09-10T19:25:10.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:25:09 smithi191 ceph-mon[32884]: osdmap e327: 8 total, 8 up, 7 in 2024-09-10T19:25:10.314 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:25:10.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:25:10.346 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:25:10.346 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:25:10.447 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:25:10.476 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:25:10.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:25:10.477 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:25:10.578 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:25:10.604 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:25:10.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:25:10.605 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:25:10.706 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:25:10.731 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:25:10.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:25:10.732 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:25:10.833 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:25:10.854 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:25:10.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:25:10.855 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:25:10.858 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: 50 16 74936 74920 93.6397 50.875 0.00241444 0.0106687 2024-09-10T19:25:10.955 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:25:10.982 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:25:10.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:25:10.984 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:25:11.084 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:25:11.107 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:25:11.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:25:11.108 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:25:11.209 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:25:11.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:25:10 smithi191 ceph-mon[32884]: from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:25:11.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:25:10 smithi191 ceph-mon[32884]: pgmap v689: 158 pgs: 158 active+clean; 4.2 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 853 B/s rd, 65 MiB/s wr, 1.03k op/s 2024-09-10T19:25:11.217 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:25:10 smithi183 ceph-mon[29736]: from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:25:11.217 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:25:10 smithi183 ceph-mon[29736]: pgmap v689: 158 pgs: 158 active+clean; 4.2 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 853 B/s rd, 65 MiB/s wr, 1.03k op/s 2024-09-10T19:25:11.218 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:25:10 smithi183 ceph-mon[41562]: from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:25:11.218 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:25:10 smithi183 ceph-mon[41562]: pgmap v689: 158 pgs: 158 active+clean; 4.2 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 853 B/s rd, 65 MiB/s wr, 1.03k op/s 2024-09-10T19:25:11.238 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:25:11.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:25:11.239 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:25:11.340 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:25:11.376 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:25:11.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:25:11.434 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:25:11.479 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Sep 10 19:25:11 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-y[29989]: ::ffff:172.21.15.191 - - [10/Sep/2024:19:25:11] "GET /metrics HTTP/1.1" 200 224729 "" "Prometheus/2.43.0" 2024-09-10T19:25:11.535 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:25:11.556 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:25:11.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:25:11.557 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:25:11.658 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:25:11.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:25:11.685 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:25:11.685 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:25:11.786 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:25:11.837 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:25:11.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:25:11.839 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:25:11.858 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: 51 16 75840 75824 92.9114 56.5 0.00679687 0.0107581 2024-09-10T19:25:11.867 INFO:teuthology.orchestra.run.smithi191.stdout:{ 2024-09-10T19:25:11.867 INFO:teuthology.orchestra.run.smithi191.stdout: "ops": [], 2024-09-10T19:25:11.867 INFO:teuthology.orchestra.run.smithi191.stdout: "complaint_time": 30, 2024-09-10T19:25:11.867 INFO:teuthology.orchestra.run.smithi191.stdout: "num_blocked_ops": 0 2024-09-10T19:25:11.867 INFO:teuthology.orchestra.run.smithi191.stdout:} 2024-09-10T19:25:11.939 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:25:11.962 INFO:teuthology.orchestra.run.smithi183.stderr:noscrub is unset 2024-09-10T19:25:11.963 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:25:11.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:25:11.964 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:25:11.990 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-10T19:25:12.065 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:25:12.096 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:25:12.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:25:12.097 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:25:12.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:25:11 smithi183 ceph-mon[29736]: from='client.? 172.21.15.183:0/766674056' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-10T19:25:12.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:25:11 smithi183 ceph-mon[41562]: from='client.? 172.21.15.183:0/766674056' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-10T19:25:12.198 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:25:12.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:25:11 smithi191 ceph-mon[32884]: from='client.? 172.21.15.183:0/766674056' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-10T19:25:12.219 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:25:12.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:25:12.220 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:25:12.320 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:25:12.348 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:25:12.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:25:12.349 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:25:12.450 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:25:12.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:25:12.472 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:25:12.472 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:25:12.573 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:25:12.601 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:25:12.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:25:12.602 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:25:12.703 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:25:12.738 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:25:12.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:25:12.739 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:25:12.840 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:25:12.858 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: 52 16 76911 76895 92.4118 66.9375 0.0129326 0.0108166 2024-09-10T19:25:12.879 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:25:12.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:25:12.880 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:25:12.981 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:25:13.004 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:25:13.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:25:13.008 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:25:13.109 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:25:13.145 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:25:13.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:25:13.147 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:25:13.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:25:12 smithi191 ceph-mon[32884]: pgmap v691: 158 pgs: 158 active+clean; 4.5 GiB data, 21 GiB used, 604 GiB / 626 GiB avail; 732 B/s rd, 91 MiB/s wr, 1.46k op/s 2024-09-10T19:25:13.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:25:12 smithi191 ceph-mon[32884]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T19:25:13.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:25:12 smithi191 ceph-mon[32884]: from='client.? 172.21.15.183:0/766674056' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-10T19:25:13.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:25:12 smithi191 ceph-mon[32884]: osdmap e328: 8 total, 8 up, 7 in 2024-09-10T19:25:13.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:25:12 smithi191 ceph-mon[32884]: 4.3 scrub starts 2024-09-10T19:25:13.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:25:12 smithi191 ceph-mon[32884]: 4.3 scrub ok 2024-09-10T19:25:13.247 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:25:13.270 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:25:13.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:25:13.271 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:25:13.372 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:25:13.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:25:13.417 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:25:13.417 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:25:13.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:25:12 smithi183 ceph-mon[29736]: pgmap v691: 158 pgs: 158 active+clean; 4.5 GiB data, 21 GiB used, 604 GiB / 626 GiB avail; 732 B/s rd, 91 MiB/s wr, 1.46k op/s 2024-09-10T19:25:13.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:25:12 smithi183 ceph-mon[29736]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T19:25:13.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:25:12 smithi183 ceph-mon[29736]: from='client.? 172.21.15.183:0/766674056' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-10T19:25:13.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:25:12 smithi183 ceph-mon[29736]: osdmap e328: 8 total, 8 up, 7 in 2024-09-10T19:25:13.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:25:12 smithi183 ceph-mon[29736]: 4.3 scrub starts 2024-09-10T19:25:13.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:25:12 smithi183 ceph-mon[29736]: 4.3 scrub ok 2024-09-10T19:25:13.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:25:12 smithi183 ceph-mon[41562]: pgmap v691: 158 pgs: 158 active+clean; 4.5 GiB data, 21 GiB used, 604 GiB / 626 GiB avail; 732 B/s rd, 91 MiB/s wr, 1.46k op/s 2024-09-10T19:25:13.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:25:12 smithi183 ceph-mon[41562]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T19:25:13.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:25:12 smithi183 ceph-mon[41562]: from='client.? 172.21.15.183:0/766674056' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-10T19:25:13.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:25:12 smithi183 ceph-mon[41562]: osdmap e328: 8 total, 8 up, 7 in 2024-09-10T19:25:13.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:25:12 smithi183 ceph-mon[41562]: 4.3 scrub starts 2024-09-10T19:25:13.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:25:12 smithi183 ceph-mon[41562]: 4.3 scrub ok 2024-09-10T19:25:13.496 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:25:13.518 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:25:13.544 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:25:13.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:25:13.545 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:25:13.646 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:25:13.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:25:13.672 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:25:13.672 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:25:13.773 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:25:13.807 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:25:13.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:25:13.808 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:25:13.858 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: 53 16 78089 78073 92.0572 73.625 0.00383623 0.0108575 2024-09-10T19:25:13.909 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:25:13.936 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:25:13.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:25:13.937 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:25:14.038 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:25:14.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:25:14.067 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:25:14.067 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:25:14.168 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:25:14.206 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:25:14.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:25:14.208 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:25:14.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:25:13 smithi191 ceph-mon[32884]: 1.9 scrub starts 2024-09-10T19:25:14.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:25:13 smithi191 ceph-mon[32884]: 1.9 scrub ok 2024-09-10T19:25:14.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:25:13 smithi191 ceph-mon[32884]: 3.1 scrub starts 2024-09-10T19:25:14.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:25:13 smithi191 ceph-mon[32884]: 3.1 scrub ok 2024-09-10T19:25:14.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:25:13 smithi191 ceph-mon[32884]: 4.7 scrub starts 2024-09-10T19:25:14.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:25:13 smithi191 ceph-mon[32884]: 4.7 scrub ok 2024-09-10T19:25:14.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:25:13 smithi191 ceph-mon[32884]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:25:14.236 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:25:13 smithi183 ceph-mon[29736]: 1.9 scrub starts 2024-09-10T19:25:14.236 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:25:13 smithi183 ceph-mon[29736]: 1.9 scrub ok 2024-09-10T19:25:14.236 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:25:13 smithi183 ceph-mon[29736]: 3.1 scrub starts 2024-09-10T19:25:14.236 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:25:13 smithi183 ceph-mon[29736]: 3.1 scrub ok 2024-09-10T19:25:14.236 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:25:13 smithi183 ceph-mon[29736]: 4.7 scrub starts 2024-09-10T19:25:14.236 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:25:13 smithi183 ceph-mon[29736]: 4.7 scrub ok 2024-09-10T19:25:14.237 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:25:13 smithi183 ceph-mon[29736]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:25:14.237 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:25:13 smithi183 ceph-mon[41562]: 1.9 scrub starts 2024-09-10T19:25:14.237 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:25:13 smithi183 ceph-mon[41562]: 1.9 scrub ok 2024-09-10T19:25:14.237 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:25:13 smithi183 ceph-mon[41562]: 3.1 scrub starts 2024-09-10T19:25:14.237 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:25:13 smithi183 ceph-mon[41562]: 3.1 scrub ok 2024-09-10T19:25:14.237 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:25:13 smithi183 ceph-mon[41562]: 4.7 scrub starts 2024-09-10T19:25:14.237 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:25:13 smithi183 ceph-mon[41562]: 4.7 scrub ok 2024-09-10T19:25:14.237 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:25:13 smithi183 ceph-mon[41562]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:25:14.309 DEBUG:teuthology.orchestra.run.smithi191:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph daemon osd.7 dump_historic_ops 2024-09-10T19:25:14.309 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:25:14.340 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:25:14.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:25:14.341 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:25:14.442 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:25:14.470 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:25:14.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:25:14.471 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:25:14.572 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:25:14.596 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:25:14.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:25:14.597 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:25:14.698 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:25:14.726 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:25:14.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:25:14.728 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:25:14.811 INFO:teuthology.orchestra.run.smithi191.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.b/config 2024-09-10T19:25:14.828 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:25:14.858 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: 54 16 79267 79251 91.7158 73.625 0.0205688 0.010898 2024-09-10T19:25:14.878 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:25:14.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:25:14.879 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:25:14.979 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:25:15.007 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:25:15.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:25:15.008 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:25:15.108 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:25:15.135 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:25:15.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:25:15.136 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:25:15.162 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:25:14 smithi183 ceph-mon[29736]: 6.c scrub starts 2024-09-10T19:25:15.162 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:25:14 smithi183 ceph-mon[29736]: 6.c scrub ok 2024-09-10T19:25:15.162 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:25:14 smithi183 ceph-mon[29736]: 5.1b scrub starts 2024-09-10T19:25:15.162 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:25:14 smithi183 ceph-mon[29736]: 5.1b scrub ok 2024-09-10T19:25:15.162 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:25:14 smithi183 ceph-mon[29736]: pgmap v693: 158 pgs: 158 active+clean; 4.6 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 853 B/s rd, 54 MiB/s wr, 870 op/s 2024-09-10T19:25:15.162 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:25:14 smithi183 ceph-mon[29736]: 2.11 scrub starts 2024-09-10T19:25:15.162 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:25:14 smithi183 ceph-mon[29736]: 2.11 scrub ok 2024-09-10T19:25:15.162 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:25:14 smithi183 ceph-mon[29736]: 2.6 scrub starts 2024-09-10T19:25:15.162 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:25:14 smithi183 ceph-mon[29736]: 2.6 scrub ok 2024-09-10T19:25:15.162 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:25:14 smithi183 ceph-mon[29736]: osdmap e329: 8 total, 8 up, 7 in 2024-09-10T19:25:15.163 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:25:14 smithi183 ceph-mon[41562]: 6.c scrub starts 2024-09-10T19:25:15.163 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:25:14 smithi183 ceph-mon[41562]: 6.c scrub ok 2024-09-10T19:25:15.163 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:25:14 smithi183 ceph-mon[41562]: 5.1b scrub starts 2024-09-10T19:25:15.163 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:25:14 smithi183 ceph-mon[41562]: 5.1b scrub ok 2024-09-10T19:25:15.163 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:25:14 smithi183 ceph-mon[41562]: pgmap v693: 158 pgs: 158 active+clean; 4.6 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 853 B/s rd, 54 MiB/s wr, 870 op/s 2024-09-10T19:25:15.163 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:25:14 smithi183 ceph-mon[41562]: 2.11 scrub starts 2024-09-10T19:25:15.163 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:25:14 smithi183 ceph-mon[41562]: 2.11 scrub ok 2024-09-10T19:25:15.163 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:25:14 smithi183 ceph-mon[41562]: 2.6 scrub starts 2024-09-10T19:25:15.163 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:25:14 smithi183 ceph-mon[41562]: 2.6 scrub ok 2024-09-10T19:25:15.163 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:25:14 smithi183 ceph-mon[41562]: osdmap e329: 8 total, 8 up, 7 in 2024-09-10T19:25:15.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:25:14 smithi191 ceph-mon[32884]: 6.c scrub starts 2024-09-10T19:25:15.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:25:14 smithi191 ceph-mon[32884]: 6.c scrub ok 2024-09-10T19:25:15.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:25:14 smithi191 ceph-mon[32884]: 5.1b scrub starts 2024-09-10T19:25:15.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:25:14 smithi191 ceph-mon[32884]: 5.1b scrub ok 2024-09-10T19:25:15.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:25:14 smithi191 ceph-mon[32884]: pgmap v693: 158 pgs: 158 active+clean; 4.6 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 853 B/s rd, 54 MiB/s wr, 870 op/s 2024-09-10T19:25:15.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:25:14 smithi191 ceph-mon[32884]: 2.11 scrub starts 2024-09-10T19:25:15.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:25:14 smithi191 ceph-mon[32884]: 2.11 scrub ok 2024-09-10T19:25:15.217 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:25:14 smithi191 ceph-mon[32884]: 2.6 scrub starts 2024-09-10T19:25:15.217 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:25:14 smithi191 ceph-mon[32884]: 2.6 scrub ok 2024-09-10T19:25:15.217 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:25:14 smithi191 ceph-mon[32884]: osdmap e329: 8 total, 8 up, 7 in 2024-09-10T19:25:15.237 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:25:15.271 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:25:15.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:25:15.273 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:25:15.373 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:25:15.407 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:25:15.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:25:15.408 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:25:15.497 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-10T19:25:15.509 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:25:15.543 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:25:15.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:25:15.544 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:25:15.645 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:25:15.680 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:25:15.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:25:15.681 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:25:15.782 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:25:15.789 INFO:teuthology.orchestra.run.smithi183.stdout:osd.0: {} 2024-09-10T19:25:15.789 INFO:teuthology.orchestra.run.smithi183.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-10T19:25:15.820 INFO:teuthology.orchestra.run.smithi183.stdout:osd.1: {} 2024-09-10T19:25:15.820 INFO:teuthology.orchestra.run.smithi183.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-10T19:25:15.823 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:25:15.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:25:15.824 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:25:15.850 INFO:teuthology.orchestra.run.smithi183.stdout:osd.2: {} 2024-09-10T19:25:15.850 INFO:teuthology.orchestra.run.smithi183.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-10T19:25:15.858 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: 55 16 80525 80509 91.4773 78.625 0.00705008 0.0109273 2024-09-10T19:25:15.878 INFO:teuthology.orchestra.run.smithi183.stdout:osd.3: {} 2024-09-10T19:25:15.878 INFO:teuthology.orchestra.run.smithi183.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-10T19:25:15.879 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:25:15.914 INFO:teuthology.orchestra.run.smithi183.stdout:osd.4: {} 2024-09-10T19:25:15.914 INFO:teuthology.orchestra.run.smithi183.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-10T19:25:15.925 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:25:15.937 INFO:teuthology.orchestra.run.smithi183.stdout:osd.5: {} 2024-09-10T19:25:15.937 INFO:teuthology.orchestra.run.smithi183.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-10T19:25:15.967 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:25:15.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:25:15.968 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:25:15.970 INFO:teuthology.orchestra.run.smithi183.stdout:osd.6: {} 2024-09-10T19:25:15.970 INFO:teuthology.orchestra.run.smithi183.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-10T19:25:16.001 INFO:teuthology.orchestra.run.smithi183.stdout:osd.7: {} 2024-09-10T19:25:16.001 INFO:teuthology.orchestra.run.smithi183.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-10T19:25:16.002 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:25:15 smithi183 ceph-mon[29736]: 2.12 scrub starts 2024-09-10T19:25:16.002 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:25:15 smithi183 ceph-mon[29736]: 2.12 scrub ok 2024-09-10T19:25:16.002 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:25:15 smithi183 ceph-mon[29736]: 4.1a scrub starts 2024-09-10T19:25:16.002 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:25:15 smithi183 ceph-mon[29736]: 4.1a scrub ok 2024-09-10T19:25:16.002 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:25:15 smithi183 ceph-mon[29736]: 4.1e scrub starts 2024-09-10T19:25:16.002 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:25:15 smithi183 ceph-mon[29736]: 4.1e scrub ok 2024-09-10T19:25:16.002 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:25:15 smithi183 ceph-mon[29736]: from='client.? 172.21.15.183:0/1701906782' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T19:25:16.002 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:25:15 smithi183 ceph-mon[41562]: 2.12 scrub starts 2024-09-10T19:25:16.002 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:25:15 smithi183 ceph-mon[41562]: 2.12 scrub ok 2024-09-10T19:25:16.003 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:25:15 smithi183 ceph-mon[41562]: 4.1a scrub starts 2024-09-10T19:25:16.003 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:25:15 smithi183 ceph-mon[41562]: 4.1a scrub ok 2024-09-10T19:25:16.003 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:25:15 smithi183 ceph-mon[41562]: 4.1e scrub starts 2024-09-10T19:25:16.003 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:25:15 smithi183 ceph-mon[41562]: 4.1e scrub ok 2024-09-10T19:25:16.003 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:25:15 smithi183 ceph-mon[41562]: from='client.? 172.21.15.183:0/1701906782' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T19:25:16.069 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:25:16.107 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:25:16.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:25:16.108 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:25:16.209 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:25:16.210 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:25:15 smithi191 ceph-mon[32884]: 2.12 scrub starts 2024-09-10T19:25:16.210 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:25:15 smithi191 ceph-mon[32884]: 2.12 scrub ok 2024-09-10T19:25:16.210 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:25:15 smithi191 ceph-mon[32884]: 4.1a scrub starts 2024-09-10T19:25:16.210 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:25:15 smithi191 ceph-mon[32884]: 4.1a scrub ok 2024-09-10T19:25:16.211 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:25:15 smithi191 ceph-mon[32884]: 4.1e scrub starts 2024-09-10T19:25:16.211 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:25:15 smithi191 ceph-mon[32884]: 4.1e scrub ok 2024-09-10T19:25:16.211 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:25:15 smithi191 ceph-mon[32884]: from='client.? 172.21.15.183:0/1701906782' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T19:25:16.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:25:16.252 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:25:16.253 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:25:16.353 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:25:16.383 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:25:16.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:25:16.384 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:25:16.485 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:25:16.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:25:16.526 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:25:16.526 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:25:16.627 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:25:16.651 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:25:16.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:25:16.652 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:25:16.753 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:25:16.780 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:25:16.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:25:16.781 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:25:16.858 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: 56 16 81771 81755 91.2343 77.875 0.0075711 0.0109561 2024-09-10T19:25:16.882 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:25:16.925 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:25:16.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:25:16.926 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:25:17.027 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:25:17.030 INFO:teuthology.orchestra.run.smithi191.stdout:ERROR: (22) Invalid argument 2024-09-10T19:25:17.031 INFO:teuthology.orchestra.run.smithi191.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-10T19:25:17.054 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:25:17.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:25:17.055 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:25:17.156 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:25:17.192 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:25:17.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:25:17.193 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:25:17.200 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:25:16 smithi183 ceph-mon[29736]: 1.5 scrub starts 2024-09-10T19:25:17.200 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:25:16 smithi183 ceph-mon[29736]: 1.5 scrub ok 2024-09-10T19:25:17.200 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:25:16 smithi183 ceph-mon[29736]: pgmap v695: 158 pgs: 158 active+clean; 4.8 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 853 B/s rd, 96 MiB/s wr, 1.53k op/s 2024-09-10T19:25:17.200 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:25:16 smithi183 ceph-mon[29736]: 5.5 scrub starts 2024-09-10T19:25:17.200 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:25:16 smithi183 ceph-mon[29736]: 5.5 scrub ok 2024-09-10T19:25:17.200 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:25:16 smithi183 ceph-mon[29736]: 6.12 scrub starts 2024-09-10T19:25:17.200 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:25:16 smithi183 ceph-mon[29736]: 6.12 scrub ok 2024-09-10T19:25:17.200 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:25:16 smithi183 ceph-mon[41562]: 1.5 scrub starts 2024-09-10T19:25:17.200 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:25:16 smithi183 ceph-mon[41562]: 1.5 scrub ok 2024-09-10T19:25:17.200 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:25:16 smithi183 ceph-mon[41562]: pgmap v695: 158 pgs: 158 active+clean; 4.8 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 853 B/s rd, 96 MiB/s wr, 1.53k op/s 2024-09-10T19:25:17.201 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:25:16 smithi183 ceph-mon[41562]: 5.5 scrub starts 2024-09-10T19:25:17.201 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:25:16 smithi183 ceph-mon[41562]: 5.5 scrub ok 2024-09-10T19:25:17.201 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:25:16 smithi183 ceph-mon[41562]: 6.12 scrub starts 2024-09-10T19:25:17.201 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:25:16 smithi183 ceph-mon[41562]: 6.12 scrub ok 2024-09-10T19:25:17.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:25:16 smithi191 ceph-mon[32884]: 1.5 scrub starts 2024-09-10T19:25:17.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:25:16 smithi191 ceph-mon[32884]: 1.5 scrub ok 2024-09-10T19:25:17.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:25:16 smithi191 ceph-mon[32884]: pgmap v695: 158 pgs: 158 active+clean; 4.8 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 853 B/s rd, 96 MiB/s wr, 1.53k op/s 2024-09-10T19:25:17.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:25:16 smithi191 ceph-mon[32884]: 5.5 scrub starts 2024-09-10T19:25:17.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:25:16 smithi191 ceph-mon[32884]: 5.5 scrub ok 2024-09-10T19:25:17.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:25:16 smithi191 ceph-mon[32884]: 6.12 scrub starts 2024-09-10T19:25:17.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:25:16 smithi191 ceph-mon[32884]: 6.12 scrub ok 2024-09-10T19:25:17.233 INFO:tasks.thrashosds.thrasher:in_osds: [0, 3, 4, 5, 6, 7, 1] out_osds: [2] dead_osds: [] live_osds: [2, 3, 4, 5, 6, 7, 1, 0] 2024-09-10T19:25:17.233 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph osd reweight 0 1 2024-09-10T19:25:17.294 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:25:17.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:25:17.331 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:25:17.331 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:25:17.432 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:25:17.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:25:17.461 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:25:17.461 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:25:17.561 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:25:17.600 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:25:17.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:25:17.601 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:25:17.701 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:25:17.715 INFO:journalctl@ceph.mgr.x.smithi191.stdout:Sep 10 19:25:17 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-x[34967]: ::ffff:172.21.15.191 - - [10/Sep/2024:19:25:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T19:25:17.735 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:25:17.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:25:17.736 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:25:17.837 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:25:17.847 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph daemon osd.1 dump_ops_in_flight 2024-09-10T19:25:17.858 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: 57 15 83003 82988 90.9856 77.0625 0.0138666 0.0109859 2024-09-10T19:25:17.863 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:25:17.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:25:17.865 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:25:17.965 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:25:18.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:25:18.010 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:25:18.010 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:25:18.111 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:25:18.143 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:25:18.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:25:18.144 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:25:18.173 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:25:17 smithi183 ceph-mon[41562]: 4.15 scrub starts 2024-09-10T19:25:18.173 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:25:17 smithi183 ceph-mon[41562]: 4.15 scrub ok 2024-09-10T19:25:18.173 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:25:17 smithi183 ceph-mon[41562]: 4.14 scrub starts 2024-09-10T19:25:18.173 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:25:17 smithi183 ceph-mon[41562]: 4.14 scrub ok 2024-09-10T19:25:18.173 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:25:17 smithi183 ceph-mon[41562]: 4.c scrub starts 2024-09-10T19:25:18.173 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:25:17 smithi183 ceph-mon[41562]: 4.c scrub ok 2024-09-10T19:25:18.173 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:25:17 smithi183 ceph-mon[41562]: 2.14 scrub starts 2024-09-10T19:25:18.174 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:25:17 smithi183 ceph-mon[41562]: 2.14 scrub ok 2024-09-10T19:25:18.174 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:25:17 smithi183 ceph-mon[41562]: 1.3 scrub starts 2024-09-10T19:25:18.174 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:25:17 smithi183 ceph-mon[29736]: 4.15 scrub starts 2024-09-10T19:25:18.174 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:25:17 smithi183 ceph-mon[29736]: 4.15 scrub ok 2024-09-10T19:25:18.174 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:25:17 smithi183 ceph-mon[29736]: 4.14 scrub starts 2024-09-10T19:25:18.174 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:25:17 smithi183 ceph-mon[29736]: 4.14 scrub ok 2024-09-10T19:25:18.174 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:25:17 smithi183 ceph-mon[29736]: 4.c scrub starts 2024-09-10T19:25:18.174 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:25:17 smithi183 ceph-mon[29736]: 4.c scrub ok 2024-09-10T19:25:18.174 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:25:17 smithi183 ceph-mon[29736]: 2.14 scrub starts 2024-09-10T19:25:18.175 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:25:17 smithi183 ceph-mon[29736]: 2.14 scrub ok 2024-09-10T19:25:18.175 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:25:17 smithi183 ceph-mon[29736]: 1.3 scrub starts 2024-09-10T19:25:18.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:25:17 smithi191 ceph-mon[32884]: 4.15 scrub starts 2024-09-10T19:25:18.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:25:17 smithi191 ceph-mon[32884]: 4.15 scrub ok 2024-09-10T19:25:18.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:25:17 smithi191 ceph-mon[32884]: 4.14 scrub starts 2024-09-10T19:25:18.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:25:17 smithi191 ceph-mon[32884]: 4.14 scrub ok 2024-09-10T19:25:18.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:25:17 smithi191 ceph-mon[32884]: 4.c scrub starts 2024-09-10T19:25:18.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:25:17 smithi191 ceph-mon[32884]: 4.c scrub ok 2024-09-10T19:25:18.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:25:17 smithi191 ceph-mon[32884]: 2.14 scrub starts 2024-09-10T19:25:18.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:25:17 smithi191 ceph-mon[32884]: 2.14 scrub ok 2024-09-10T19:25:18.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:25:17 smithi191 ceph-mon[32884]: 1.3 scrub starts 2024-09-10T19:25:18.246 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:25:18.276 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:25:18.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:25:18.277 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:25:18.378 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:25:18.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:25:18.409 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:25:18.409 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:25:18.509 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:25:18.545 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:25:18.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:25:18.546 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:25:18.591 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:25:18.648 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:25:18.649 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:25:18.687 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:25:18.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:25:18.688 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:25:18.788 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:25:18.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:25:18.821 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:25:18.821 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:25:18.859 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: 58 16 84267 84251 90.7778 78.9375 0.014504 0.0110114 2024-09-10T19:25:18.922 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:25:18.953 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:25:18.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:25:18.954 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:25:18.975 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-10T19:25:18.989 INFO:teuthology.orchestra.run.smithi183.stderr:nodeep-scrub is unset 2024-09-10T19:25:19.054 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:25:19.093 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:25:19.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:25:19.094 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:25:19.195 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:25:19.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:25:19.230 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:25:19.230 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:25:19.301 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:25:18 smithi183 ceph-mon[29736]: 1.3 scrub ok 2024-09-10T19:25:19.302 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:25:18 smithi183 ceph-mon[29736]: 1.8 scrub starts 2024-09-10T19:25:19.302 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:25:18 smithi183 ceph-mon[29736]: 1.8 scrub ok 2024-09-10T19:25:19.302 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:25:18 smithi183 ceph-mon[29736]: pgmap v696: 158 pgs: 158 active+clean; 4.9 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 1.3 KiB/s rd, 84 MiB/s wr, 1.34k op/s 2024-09-10T19:25:19.302 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:25:18 smithi183 ceph-mon[29736]: 10.f scrub starts 2024-09-10T19:25:19.302 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:25:18 smithi183 ceph-mon[29736]: from='client.? 172.21.15.183:0/4140410319' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-10T19:25:19.302 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:25:18 smithi183 ceph-mon[41562]: 1.3 scrub ok 2024-09-10T19:25:19.302 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:25:18 smithi183 ceph-mon[41562]: 1.8 scrub starts 2024-09-10T19:25:19.302 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:25:18 smithi183 ceph-mon[41562]: 1.8 scrub ok 2024-09-10T19:25:19.302 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:25:18 smithi183 ceph-mon[41562]: pgmap v696: 158 pgs: 158 active+clean; 4.9 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 1.3 KiB/s rd, 84 MiB/s wr, 1.34k op/s 2024-09-10T19:25:19.302 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:25:18 smithi183 ceph-mon[41562]: 10.f scrub starts 2024-09-10T19:25:19.303 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:25:18 smithi183 ceph-mon[41562]: from='client.? 172.21.15.183:0/4140410319' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-10T19:25:19.331 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:25:19.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:25:19.363 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:25:19.363 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:25:19.464 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:25:19.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:25:18 smithi191 ceph-mon[32884]: 1.3 scrub ok 2024-09-10T19:25:19.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:25:18 smithi191 ceph-mon[32884]: 1.8 scrub starts 2024-09-10T19:25:19.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:25:18 smithi191 ceph-mon[32884]: 1.8 scrub ok 2024-09-10T19:25:19.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:25:18 smithi191 ceph-mon[32884]: pgmap v696: 158 pgs: 158 active+clean; 4.9 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 1.3 KiB/s rd, 84 MiB/s wr, 1.34k op/s 2024-09-10T19:25:19.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:25:18 smithi191 ceph-mon[32884]: 10.f scrub starts 2024-09-10T19:25:19.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:25:18 smithi191 ceph-mon[32884]: from='client.? 172.21.15.183:0/4140410319' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-10T19:25:19.492 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:25:19.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:25:19.493 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:25:19.594 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:25:19.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:25:19.634 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:25:19.635 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:25:19.736 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:25:19.762 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:25:19.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:25:19.763 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:25:19.859 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: 59 16 85449 85433 90.4912 73.875 0.0211954 0.0110466 2024-09-10T19:25:19.863 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:25:19.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:25:19.897 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:25:19.897 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:25:19.998 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:25:19.999 INFO:journalctl@ceph.iscsi.iscsi.a.smithi191.stdout:Sep 10 19:25:19 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-iscsi-iscsi-a[61816]: debug there is no tcmu-runner data available 2024-09-10T19:25:20.027 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:25:20.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:25:20.029 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:25:20.130 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:25:20.176 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:25:20.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:25:20.177 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:25:20.278 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:25:20.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:25:20.320 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:25:20.320 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:25:20.393 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:25:19 smithi183 ceph-mon[29736]: 6.1d scrub starts 2024-09-10T19:25:20.393 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:25:19 smithi183 ceph-mon[29736]: 6.1d scrub ok 2024-09-10T19:25:20.393 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:25:19 smithi183 ceph-mon[29736]: 1.a scrub starts 2024-09-10T19:25:20.393 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:25:19 smithi183 ceph-mon[29736]: 1.a scrub ok 2024-09-10T19:25:20.394 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:25:19 smithi183 ceph-mon[29736]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-10T19:25:20.394 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:25:19 smithi183 ceph-mon[29736]: Cluster is now healthy 2024-09-10T19:25:20.394 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:25:19 smithi183 ceph-mon[29736]: from='client.? 172.21.15.183:0/4140410319' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-10T19:25:20.394 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:25:19 smithi183 ceph-mon[29736]: osdmap e330: 8 total, 8 up, 7 in 2024-09-10T19:25:20.394 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:25:19 smithi183 ceph-mon[29736]: 10.a scrub starts 2024-09-10T19:25:20.394 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:25:19 smithi183 ceph-mon[29736]: 6.16 deep-scrub starts 2024-09-10T19:25:20.394 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:25:19 smithi183 ceph-mon[29736]: 6.16 deep-scrub ok 2024-09-10T19:25:20.394 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:25:19 smithi183 ceph-mon[29736]: osdmap e331: 8 total, 8 up, 7 in 2024-09-10T19:25:20.394 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:25:19 smithi183 ceph-mon[41562]: 6.1d scrub starts 2024-09-10T19:25:20.395 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:25:19 smithi183 ceph-mon[41562]: 6.1d scrub ok 2024-09-10T19:25:20.395 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:25:19 smithi183 ceph-mon[41562]: 1.a scrub starts 2024-09-10T19:25:20.395 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:25:19 smithi183 ceph-mon[41562]: 1.a scrub ok 2024-09-10T19:25:20.395 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:25:19 smithi183 ceph-mon[41562]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-10T19:25:20.395 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:25:19 smithi183 ceph-mon[41562]: Cluster is now healthy 2024-09-10T19:25:20.395 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:25:19 smithi183 ceph-mon[41562]: from='client.? 172.21.15.183:0/4140410319' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-10T19:25:20.395 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:25:19 smithi183 ceph-mon[41562]: osdmap e330: 8 total, 8 up, 7 in 2024-09-10T19:25:20.395 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:25:19 smithi183 ceph-mon[41562]: 10.a scrub starts 2024-09-10T19:25:20.395 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:25:19 smithi183 ceph-mon[41562]: 6.16 deep-scrub starts 2024-09-10T19:25:20.395 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:25:19 smithi183 ceph-mon[41562]: 6.16 deep-scrub ok 2024-09-10T19:25:20.395 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:25:19 smithi183 ceph-mon[41562]: osdmap e331: 8 total, 8 up, 7 in 2024-09-10T19:25:20.421 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:25:20.458 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:25:20.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:25:20.459 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:25:20.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:25:19 smithi191 ceph-mon[32884]: 6.1d scrub starts 2024-09-10T19:25:20.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:25:19 smithi191 ceph-mon[32884]: 6.1d scrub ok 2024-09-10T19:25:20.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:25:19 smithi191 ceph-mon[32884]: 1.a scrub starts 2024-09-10T19:25:20.467 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:25:19 smithi191 ceph-mon[32884]: 1.a scrub ok 2024-09-10T19:25:20.467 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:25:19 smithi191 ceph-mon[32884]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-10T19:25:20.467 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:25:19 smithi191 ceph-mon[32884]: Cluster is now healthy 2024-09-10T19:25:20.467 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:25:19 smithi191 ceph-mon[32884]: from='client.? 172.21.15.183:0/4140410319' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-10T19:25:20.467 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:25:19 smithi191 ceph-mon[32884]: osdmap e330: 8 total, 8 up, 7 in 2024-09-10T19:25:20.467 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:25:19 smithi191 ceph-mon[32884]: 10.a scrub starts 2024-09-10T19:25:20.467 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:25:19 smithi191 ceph-mon[32884]: 6.16 deep-scrub starts 2024-09-10T19:25:20.467 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:25:19 smithi191 ceph-mon[32884]: 6.16 deep-scrub ok 2024-09-10T19:25:20.467 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:25:19 smithi191 ceph-mon[32884]: osdmap e331: 8 total, 8 up, 7 in 2024-09-10T19:25:20.560 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:25:20.588 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:25:20.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:25:20.589 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:25:20.690 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:25:20.717 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:25:20.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:25:20.718 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:25:20.819 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:25:20.856 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:25:20.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:25:20.857 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:25:20.859 INFO:tasks.radosbench.radosbench.0.smithi183.stdout:2024-09-10T19:25:20.858424+0000 min lat: 0.00110744 max lat: 2.06031 avg lat: 0.0110865 2024-09-10T19:25:20.859 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-10T19:25:20.859 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: 60 16 86581 86565 90.162 70.75 0.0166697 0.0110865 2024-09-10T19:25:20.959 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:25:20.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:25:21.000 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:25:21.000 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:25:21.101 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:25:21.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:25:21.139 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:25:21.140 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:25:21.241 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:25:21.269 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:25:21.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:25:21.270 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:25:21.316 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:25:20 smithi183 ceph-mon[29736]: from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:25:21.316 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:25:20 smithi183 ceph-mon[29736]: pgmap v698: 158 pgs: 158 active+clean; 4.9 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 1.3 KiB/s rd, 46 MiB/s wr, 729 op/s 2024-09-10T19:25:21.316 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:25:20 smithi183 ceph-mon[29736]: 2.13 scrub starts 2024-09-10T19:25:21.316 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:25:20 smithi183 ceph-mon[29736]: 2.13 scrub ok 2024-09-10T19:25:21.316 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:25:21 smithi183 ceph-mon[41562]: from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:25:21.316 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:25:21 smithi183 ceph-mon[41562]: pgmap v698: 158 pgs: 158 active+clean; 4.9 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 1.3 KiB/s rd, 46 MiB/s wr, 729 op/s 2024-09-10T19:25:21.316 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:25:21 smithi183 ceph-mon[41562]: 2.13 scrub starts 2024-09-10T19:25:21.317 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:25:21 smithi183 ceph-mon[41562]: 2.13 scrub ok 2024-09-10T19:25:21.371 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:25:21.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:25:21.403 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:25:21.403 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:25:21.465 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:25:21 smithi191 ceph-mon[32884]: from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:25:21.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:25:21 smithi191 ceph-mon[32884]: pgmap v698: 158 pgs: 158 active+clean; 4.9 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 1.3 KiB/s rd, 46 MiB/s wr, 729 op/s 2024-09-10T19:25:21.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:25:21 smithi191 ceph-mon[32884]: 2.13 scrub starts 2024-09-10T19:25:21.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:25:21 smithi191 ceph-mon[32884]: 2.13 scrub ok 2024-09-10T19:25:21.504 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:25:21.550 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:25:21.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:25:21.551 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:25:21.577 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Sep 10 19:25:21 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-y[29989]: ::ffff:172.21.15.191 - - [10/Sep/2024:19:25:21] "GET /metrics HTTP/1.1" 200 224745 "" "Prometheus/2.43.0" 2024-09-10T19:25:21.652 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:25:21.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:25:21.686 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:25:21.686 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:25:21.787 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:25:21.814 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:25:21.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:25:21.815 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:25:21.859 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: 61 16 87831 87815 89.9645 78.125 0.00659526 0.0111107 2024-09-10T19:25:21.916 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:25:21.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:25:21.957 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:25:21.958 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:25:22.058 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:25:22.089 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:25:22.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:25:22.090 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:25:22.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:25:22 smithi183 ceph-mon[29736]: 4.18 scrub starts 2024-09-10T19:25:22.182 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:25:22 smithi183 ceph-mon[29736]: 4.18 scrub ok 2024-09-10T19:25:22.182 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:25:22 smithi183 ceph-mon[29736]: 10.f scrub ok 2024-09-10T19:25:22.182 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:25:22 smithi183 ceph-mon[29736]: 10.a scrub ok 2024-09-10T19:25:22.182 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:25:22 smithi183 ceph-mon[29736]: 6.0 scrub starts 2024-09-10T19:25:22.182 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:25:22 smithi183 ceph-mon[29736]: 6.0 scrub ok 2024-09-10T19:25:22.182 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:25:22 smithi183 ceph-mon[29736]: purged_snaps scrub starts 2024-09-10T19:25:22.182 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:25:22 smithi183 ceph-mon[29736]: purged_snaps scrub ok 2024-09-10T19:25:22.182 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:25:22 smithi183 ceph-mon[29736]: 6.5 scrub starts 2024-09-10T19:25:22.182 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:25:22 smithi183 ceph-mon[29736]: 6.5 scrub ok 2024-09-10T19:25:22.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:25:22 smithi183 ceph-mon[41562]: 4.18 scrub starts 2024-09-10T19:25:22.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:25:22 smithi183 ceph-mon[41562]: 4.18 scrub ok 2024-09-10T19:25:22.183 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:25:22 smithi183 ceph-mon[41562]: 10.f scrub ok 2024-09-10T19:25:22.183 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:25:22 smithi183 ceph-mon[41562]: 10.a scrub ok 2024-09-10T19:25:22.183 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:25:22 smithi183 ceph-mon[41562]: 6.0 scrub starts 2024-09-10T19:25:22.183 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:25:22 smithi183 ceph-mon[41562]: 6.0 scrub ok 2024-09-10T19:25:22.183 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:25:22 smithi183 ceph-mon[41562]: purged_snaps scrub starts 2024-09-10T19:25:22.183 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:25:22 smithi183 ceph-mon[41562]: purged_snaps scrub ok 2024-09-10T19:25:22.183 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:25:22 smithi183 ceph-mon[41562]: 6.5 scrub starts 2024-09-10T19:25:22.183 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:25:22 smithi183 ceph-mon[41562]: 6.5 scrub ok 2024-09-10T19:25:22.190 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:25:22.219 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:25:22.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:25:22.220 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:25:22.321 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:25:22.348 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:25:22.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:25:22.349 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:25:22.450 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:25:22.465 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:25:22 smithi191 ceph-mon[32884]: 4.18 scrub starts 2024-09-10T19:25:22.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:25:22 smithi191 ceph-mon[32884]: 4.18 scrub ok 2024-09-10T19:25:22.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:25:22 smithi191 ceph-mon[32884]: 10.f scrub ok 2024-09-10T19:25:22.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:25:22 smithi191 ceph-mon[32884]: 10.a scrub ok 2024-09-10T19:25:22.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:25:22 smithi191 ceph-mon[32884]: 6.0 scrub starts 2024-09-10T19:25:22.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:25:22 smithi191 ceph-mon[32884]: 6.0 scrub ok 2024-09-10T19:25:22.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:25:22 smithi191 ceph-mon[32884]: purged_snaps scrub starts 2024-09-10T19:25:22.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:25:22 smithi191 ceph-mon[32884]: purged_snaps scrub ok 2024-09-10T19:25:22.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:25:22 smithi191 ceph-mon[32884]: 6.5 scrub starts 2024-09-10T19:25:22.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:25:22 smithi191 ceph-mon[32884]: 6.5 scrub ok 2024-09-10T19:25:22.492 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:25:22.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:25:22.493 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:25:22.508 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:25:22.594 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:25:22.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:25:22.625 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:25:22.625 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:25:22.726 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:25:22.752 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:25:22.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:25:22.753 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:25:22.854 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:25:22.859 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: 62 16 89116 89100 89.8087 80.3125 0.00557758 0.0111304 2024-09-10T19:25:22.890 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:25:22.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:25:22.891 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:25:22.992 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:25:23.018 INFO:teuthology.orchestra.run.smithi183.stderr:reweighted osd.0 to 1 (10000) 2024-09-10T19:25:23.024 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:25:23.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:25:23.025 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:25:23.091 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:25:23 smithi183 ceph-mon[29736]: 6.b scrub starts 2024-09-10T19:25:23.092 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:25:23 smithi183 ceph-mon[29736]: 6.b scrub ok 2024-09-10T19:25:23.092 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:25:23 smithi183 ceph-mon[29736]: pgmap v700: 158 pgs: 2 active+clean+scrubbing, 156 active+clean; 5.2 GiB data, 23 GiB used, 602 GiB / 626 GiB avail; 732 B/s rd, 90 MiB/s wr, 1.45k op/s 2024-09-10T19:25:23.092 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:25:23 smithi183 ceph-mon[29736]: osdmap e332: 8 total, 8 up, 7 in 2024-09-10T19:25:23.092 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:25:23 smithi183 ceph-mon[29736]: from='client.? 172.21.15.183:0/1605759303' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-09-10T19:25:23.093 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:25:23 smithi183 ceph-mon[41562]: 6.b scrub starts 2024-09-10T19:25:23.093 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:25:23 smithi183 ceph-mon[41562]: 6.b scrub ok 2024-09-10T19:25:23.093 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:25:23 smithi183 ceph-mon[41562]: pgmap v700: 158 pgs: 2 active+clean+scrubbing, 156 active+clean; 5.2 GiB data, 23 GiB used, 602 GiB / 626 GiB avail; 732 B/s rd, 90 MiB/s wr, 1.45k op/s 2024-09-10T19:25:23.093 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:25:23 smithi183 ceph-mon[41562]: osdmap e332: 8 total, 8 up, 7 in 2024-09-10T19:25:23.094 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:25:23 smithi183 ceph-mon[41562]: from='client.? 172.21.15.183:0/1605759303' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-09-10T19:25:23.126 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:25:23.158 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:25:23.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:25:23.159 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:25:23.260 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:25:23.289 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:25:23.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:25:23.290 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:25:23.391 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:25:23.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:25:23.428 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:25:23.429 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:25:23.440 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph osd set noscrub 2024-09-10T19:25:23.465 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:25:23 smithi191 ceph-mon[32884]: 6.b scrub starts 2024-09-10T19:25:23.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:25:23 smithi191 ceph-mon[32884]: 6.b scrub ok 2024-09-10T19:25:23.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:25:23 smithi191 ceph-mon[32884]: pgmap v700: 158 pgs: 2 active+clean+scrubbing, 156 active+clean; 5.2 GiB data, 23 GiB used, 602 GiB / 626 GiB avail; 732 B/s rd, 90 MiB/s wr, 1.45k op/s 2024-09-10T19:25:23.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:25:23 smithi191 ceph-mon[32884]: osdmap e332: 8 total, 8 up, 7 in 2024-09-10T19:25:23.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:25:23 smithi191 ceph-mon[32884]: from='client.? 172.21.15.183:0/1605759303' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-09-10T19:25:23.529 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:25:23.560 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:25:23.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:25:23.561 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:25:23.662 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:25:23.697 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:25:23.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:25:23.699 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:25:23.800 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:25:23.843 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:25:23.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:25:23.844 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:25:23.859 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: 63 16 90412 90396 89.6688 81 0.0248719 0.0111479 2024-09-10T19:25:23.945 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:25:23.986 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:25:23.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:25:23.987 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:25:24.003 INFO:teuthology.orchestra.run.smithi183.stdout:ERROR: (22) Invalid argument 2024-09-10T19:25:24.003 INFO:teuthology.orchestra.run.smithi183.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-10T19:25:24.089 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:25:24.125 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:25:24.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:25:24.126 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:25:24.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:25:24 smithi183 ceph-mon[29736]: 6.4 scrub starts 2024-09-10T19:25:24.182 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:25:24 smithi183 ceph-mon[29736]: 6.4 scrub ok 2024-09-10T19:25:24.182 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:25:24 smithi183 ceph-mon[29736]: 4.e scrub starts 2024-09-10T19:25:24.182 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:25:24 smithi183 ceph-mon[29736]: 4.e scrub ok 2024-09-10T19:25:24.182 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:25:24 smithi183 ceph-mon[29736]: from='client.? 172.21.15.183:0/1605759303' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-09-10T19:25:24.182 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:25:24 smithi183 ceph-mon[29736]: osdmap e333: 8 total, 8 up, 7 in 2024-09-10T19:25:24.182 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:25:24 smithi183 ceph-mon[29736]: 5.10 scrub starts 2024-09-10T19:25:24.182 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:25:24 smithi183 ceph-mon[29736]: 5.10 scrub ok 2024-09-10T19:25:24.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:25:24 smithi183 ceph-mon[41562]: 6.4 scrub starts 2024-09-10T19:25:24.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:25:24 smithi183 ceph-mon[41562]: 6.4 scrub ok 2024-09-10T19:25:24.183 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:25:24 smithi183 ceph-mon[41562]: 4.e scrub starts 2024-09-10T19:25:24.183 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:25:24 smithi183 ceph-mon[41562]: 4.e scrub ok 2024-09-10T19:25:24.183 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:25:24 smithi183 ceph-mon[41562]: from='client.? 172.21.15.183:0/1605759303' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-09-10T19:25:24.183 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:25:24 smithi183 ceph-mon[41562]: osdmap e333: 8 total, 8 up, 7 in 2024-09-10T19:25:24.183 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:25:24 smithi183 ceph-mon[41562]: 5.10 scrub starts 2024-09-10T19:25:24.183 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:25:24 smithi183 ceph-mon[41562]: 5.10 scrub ok 2024-09-10T19:25:24.227 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:25:24.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:25:24.267 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:25:24.267 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:25:24.368 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:25:24.398 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:25:24.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:25:24.399 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:25:24.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:25:24 smithi191 ceph-mon[32884]: 6.4 scrub starts 2024-09-10T19:25:24.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:25:24 smithi191 ceph-mon[32884]: 6.4 scrub ok 2024-09-10T19:25:24.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:25:24 smithi191 ceph-mon[32884]: 4.e scrub starts 2024-09-10T19:25:24.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:25:24 smithi191 ceph-mon[32884]: 4.e scrub ok 2024-09-10T19:25:24.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:25:24 smithi191 ceph-mon[32884]: from='client.? 172.21.15.183:0/1605759303' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-09-10T19:25:24.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:25:24 smithi191 ceph-mon[32884]: osdmap e333: 8 total, 8 up, 7 in 2024-09-10T19:25:24.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:25:24 smithi191 ceph-mon[32884]: 5.10 scrub starts 2024-09-10T19:25:24.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:25:24 smithi191 ceph-mon[32884]: 5.10 scrub ok 2024-09-10T19:25:24.500 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:25:24.526 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:25:24.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:25:24.527 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:25:24.628 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:25:24.659 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:25:24.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:25:24.660 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:25:24.761 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:25:24.797 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:25:24.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:25:24.798 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:25:24.859 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: 64 16 91690 91674 89.5157 79.875 0.0107772 0.0111662 2024-09-10T19:25:24.899 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:25:24.934 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:25:24.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:25:24.935 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:25:25.036 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:25:25.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:25:25.073 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:25:25.073 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:25:25.174 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:25:25.214 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:25:25.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:25:25.215 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:25:25.315 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:25:25.351 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:25:25.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:25:25.352 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:25:25.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:25:25 smithi183 ceph-mon[29736]: 3.2 scrub starts 2024-09-10T19:25:25.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:25:25 smithi183 ceph-mon[29736]: 3.2 scrub ok 2024-09-10T19:25:25.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:25:25 smithi183 ceph-mon[29736]: pgmap v703: 158 pgs: 2 active+clean+scrubbing, 156 active+clean; 5.3 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 78 MiB/s wr, 1.25k op/s 2024-09-10T19:25:25.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:25:25 smithi183 ceph-mon[41562]: 3.2 scrub starts 2024-09-10T19:25:25.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:25:25 smithi183 ceph-mon[41562]: 3.2 scrub ok 2024-09-10T19:25:25.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:25:25 smithi183 ceph-mon[41562]: pgmap v703: 158 pgs: 2 active+clean+scrubbing, 156 active+clean; 5.3 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 78 MiB/s wr, 1.25k op/s 2024-09-10T19:25:25.452 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:25:25.465 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:25:25 smithi191 ceph-mon[32884]: 3.2 scrub starts 2024-09-10T19:25:25.465 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:25:25 smithi191 ceph-mon[32884]: 3.2 scrub ok 2024-09-10T19:25:25.465 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:25:25 smithi191 ceph-mon[32884]: pgmap v703: 158 pgs: 2 active+clean+scrubbing, 156 active+clean; 5.3 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 78 MiB/s wr, 1.25k op/s 2024-09-10T19:25:25.472 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:25:25.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:25:25.473 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:25:25.574 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:25:25.595 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:25:25.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:25:25.596 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:25:25.643 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph daemon osd.1 dump_blocked_ops 2024-09-10T19:25:25.697 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:25:25.729 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:25:25.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:25:25.730 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:25:25.831 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:25:25.860 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: 65 16 92722 92706 89.1307 64.5 0.0115676 0.011215 2024-09-10T19:25:25.866 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:25:25.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:25:25.867 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:25:25.967 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:25:26.013 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:25:26.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:25:26.016 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:25:26.086 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph osd reweight 3 1 2024-09-10T19:25:26.117 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:25:26.160 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:25:26.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:25:26.161 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:25:26.262 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:25:26.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:25:26.298 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:25:26.298 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:25:26.321 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:25:26 smithi183 ceph-mon[29736]: 2.e scrub starts 2024-09-10T19:25:26.321 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:25:26 smithi183 ceph-mon[29736]: 2.e scrub ok 2024-09-10T19:25:26.321 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:25:26 smithi183 ceph-mon[29736]: 10.7 scrub starts 2024-09-10T19:25:26.322 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:25:26 smithi183 ceph-mon[41562]: 2.e scrub starts 2024-09-10T19:25:26.322 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:25:26 smithi183 ceph-mon[41562]: 2.e scrub ok 2024-09-10T19:25:26.322 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:25:26 smithi183 ceph-mon[41562]: 10.7 scrub starts 2024-09-10T19:25:26.398 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:25:26.430 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:25:26.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:25:26.431 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:25:26.465 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:25:26 smithi191 ceph-mon[32884]: 2.e scrub starts 2024-09-10T19:25:26.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:25:26 smithi191 ceph-mon[32884]: 2.e scrub ok 2024-09-10T19:25:26.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:25:26 smithi191 ceph-mon[32884]: 10.7 scrub starts 2024-09-10T19:25:26.531 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:25:26.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:25:26.562 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:25:26.562 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:25:26.591 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:25:26.617 INFO:teuthology.orchestra.run.smithi183.stdout:osd.0: {} 2024-09-10T19:25:26.618 INFO:teuthology.orchestra.run.smithi183.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-10T19:25:26.656 INFO:teuthology.orchestra.run.smithi183.stdout:osd.1: {} 2024-09-10T19:25:26.656 INFO:teuthology.orchestra.run.smithi183.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-10T19:25:26.663 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:25:26.684 INFO:teuthology.orchestra.run.smithi183.stdout:osd.2: {} 2024-09-10T19:25:26.685 INFO:teuthology.orchestra.run.smithi183.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-10T19:25:26.699 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:25:26.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:25:26.700 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:25:26.711 INFO:teuthology.orchestra.run.smithi183.stdout:osd.3: {} 2024-09-10T19:25:26.711 INFO:teuthology.orchestra.run.smithi183.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-10T19:25:26.743 INFO:teuthology.orchestra.run.smithi183.stdout:osd.4: {} 2024-09-10T19:25:26.743 INFO:teuthology.orchestra.run.smithi183.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-10T19:25:26.774 INFO:teuthology.orchestra.run.smithi183.stdout:osd.5: {} 2024-09-10T19:25:26.774 INFO:teuthology.orchestra.run.smithi183.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-10T19:25:26.797 INFO:teuthology.orchestra.run.smithi183.stdout:osd.6: {} 2024-09-10T19:25:26.797 INFO:teuthology.orchestra.run.smithi183.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-10T19:25:26.800 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:25:26.820 INFO:teuthology.orchestra.run.smithi183.stdout:osd.7: {} 2024-09-10T19:25:26.821 INFO:teuthology.orchestra.run.smithi183.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-10T19:25:26.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:25:26.842 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:25:26.842 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:25:26.859 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: 66 16 93846 93830 88.8446 70.25 0.0239859 0.0112513 2024-09-10T19:25:26.943 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:25:26.986 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:25:26.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:25:26.987 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:25:27.088 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:25:27.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:25:27.115 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:25:27.115 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:25:27.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:25:27 smithi183 ceph-mon[29736]: pgmap v704: 158 pgs: 158 active+clean; 5.6 GiB data, 24 GiB used, 601 GiB / 626 GiB avail; 113 MiB/s wr, 1.81k op/s 2024-09-10T19:25:27.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:25:27 smithi183 ceph-mon[29736]: 2.1a scrub starts 2024-09-10T19:25:27.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:25:27 smithi183 ceph-mon[29736]: 2.1a scrub ok 2024-09-10T19:25:27.182 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:25:27 smithi183 ceph-mon[29736]: from='client.? 172.21.15.183:0/102164168' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T19:25:27.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:25:27 smithi183 ceph-mon[41562]: pgmap v704: 158 pgs: 158 active+clean; 5.6 GiB data, 24 GiB used, 601 GiB / 626 GiB avail; 113 MiB/s wr, 1.81k op/s 2024-09-10T19:25:27.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:25:27 smithi183 ceph-mon[41562]: 2.1a scrub starts 2024-09-10T19:25:27.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:25:27 smithi183 ceph-mon[41562]: 2.1a scrub ok 2024-09-10T19:25:27.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:25:27 smithi183 ceph-mon[41562]: from='client.? 172.21.15.183:0/102164168' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T19:25:27.216 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:25:27.253 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:25:27.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:25:27.254 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:25:27.355 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:25:27.367 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:25:27 smithi191 ceph-mon[32884]: pgmap v704: 158 pgs: 158 active+clean; 5.6 GiB data, 24 GiB used, 601 GiB / 626 GiB avail; 113 MiB/s wr, 1.81k op/s 2024-09-10T19:25:27.367 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:25:27 smithi191 ceph-mon[32884]: 2.1a scrub starts 2024-09-10T19:25:27.367 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:25:27 smithi191 ceph-mon[32884]: 2.1a scrub ok 2024-09-10T19:25:27.367 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:25:27 smithi191 ceph-mon[32884]: from='client.? 172.21.15.183:0/102164168' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T19:25:27.383 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:25:27.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:25:27.384 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:25:27.485 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:25:27.518 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:25:27.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:25:27.520 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:25:27.620 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:25:27.649 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:25:27.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:25:27.650 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:25:27.716 INFO:journalctl@ceph.mgr.x.smithi191.stdout:Sep 10 19:25:27 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-x[34967]: ::ffff:172.21.15.191 - - [10/Sep/2024:19:25:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T19:25:27.751 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:25:27.785 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:25:27.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:25:27.786 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:25:27.859 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: 67 16 94883 94867 88.4858 64.8125 0.0286809 0.0112966 2024-09-10T19:25:27.887 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:25:27.916 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:25:27.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:25:27.917 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:25:28.018 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:25:28.049 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:25:28.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:25:28.050 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:25:28.119 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:25:28.119 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:25:28.151 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:25:28.174 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:25:28.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:25:28.175 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:25:28.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:25:28 smithi183 ceph-mon[29736]: 6.10 deep-scrub starts 2024-09-10T19:25:28.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:25:28 smithi183 ceph-mon[29736]: 6.10 deep-scrub ok 2024-09-10T19:25:28.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:25:28 smithi183 ceph-mon[29736]: 10.7 scrub ok 2024-09-10T19:25:28.181 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:25:28 smithi183 ceph-mon[41562]: 6.10 deep-scrub starts 2024-09-10T19:25:28.181 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:25:28 smithi183 ceph-mon[41562]: 6.10 deep-scrub ok 2024-09-10T19:25:28.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:25:28 smithi183 ceph-mon[41562]: 10.7 scrub ok 2024-09-10T19:25:28.276 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:25:28.305 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:25:28.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:25:28.306 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:25:28.407 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:25:28.443 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:25:28.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:25:28.444 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:25:28.465 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:25:28 smithi191 ceph-mon[32884]: 6.10 deep-scrub starts 2024-09-10T19:25:28.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:25:28 smithi191 ceph-mon[32884]: 6.10 deep-scrub ok 2024-09-10T19:25:28.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:25:28 smithi191 ceph-mon[32884]: 10.7 scrub ok 2024-09-10T19:25:28.545 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:25:28.581 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:25:28.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:25:28.583 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:25:28.683 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:25:28.722 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:25:28.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:25:28.723 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:25:28.824 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:25:28.860 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: 68 16 96002 95986 88.213 69.9375 0.0236553 0.0113315 2024-09-10T19:25:28.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:25:28.865 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:25:28.866 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:25:28.967 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:25:28.993 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:25:28.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:25:28.994 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:25:29.095 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:25:29.120 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:25:29.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:25:29.121 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:25:29.221 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:25:29.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:25:29.265 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:25:29.265 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:25:29.366 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:25:29.403 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:25:29.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:25:29.404 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:25:29.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:25:29 smithi183 ceph-mon[29736]: 4.8 scrub starts 2024-09-10T19:25:29.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:25:29 smithi183 ceph-mon[29736]: 4.8 scrub ok 2024-09-10T19:25:29.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:25:29 smithi183 ceph-mon[29736]: pgmap v705: 158 pgs: 158 active+clean; 5.7 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 97 MiB/s wr, 1.55k op/s 2024-09-10T19:25:29.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:25:29 smithi183 ceph-mon[29736]: 10.0 scrub starts 2024-09-10T19:25:29.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:25:29 smithi183 ceph-mon[29736]: 4.16 scrub starts 2024-09-10T19:25:29.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:25:29 smithi183 ceph-mon[29736]: 4.16 scrub ok 2024-09-10T19:25:29.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:25:29 smithi183 ceph-mon[29736]: from='mgr.14556 ' entity='mgr.y' 2024-09-10T19:25:29.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:25:29 smithi183 ceph-mon[29736]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:25:29.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:25:29 smithi183 ceph-mon[29736]: 2.0 deep-scrub starts 2024-09-10T19:25:29.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:25:29 smithi183 ceph-mon[29736]: 2.0 deep-scrub ok 2024-09-10T19:25:29.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:25:29 smithi183 ceph-mon[41562]: 4.8 scrub starts 2024-09-10T19:25:29.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:25:29 smithi183 ceph-mon[41562]: 4.8 scrub ok 2024-09-10T19:25:29.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:25:29 smithi183 ceph-mon[41562]: pgmap v705: 158 pgs: 158 active+clean; 5.7 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 97 MiB/s wr, 1.55k op/s 2024-09-10T19:25:29.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:25:29 smithi183 ceph-mon[41562]: 10.0 scrub starts 2024-09-10T19:25:29.433 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:25:29 smithi183 ceph-mon[41562]: 4.16 scrub starts 2024-09-10T19:25:29.433 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:25:29 smithi183 ceph-mon[41562]: 4.16 scrub ok 2024-09-10T19:25:29.433 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:25:29 smithi183 ceph-mon[41562]: from='mgr.14556 ' entity='mgr.y' 2024-09-10T19:25:29.433 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:25:29 smithi183 ceph-mon[41562]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:25:29.433 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:25:29 smithi183 ceph-mon[41562]: 2.0 deep-scrub starts 2024-09-10T19:25:29.433 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:25:29 smithi183 ceph-mon[41562]: 2.0 deep-scrub ok 2024-09-10T19:25:29.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:25:29 smithi191 ceph-mon[32884]: 4.8 scrub starts 2024-09-10T19:25:29.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:25:29 smithi191 ceph-mon[32884]: 4.8 scrub ok 2024-09-10T19:25:29.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:25:29 smithi191 ceph-mon[32884]: pgmap v705: 158 pgs: 158 active+clean; 5.7 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 97 MiB/s wr, 1.55k op/s 2024-09-10T19:25:29.467 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:25:29 smithi191 ceph-mon[32884]: 10.0 scrub starts 2024-09-10T19:25:29.467 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:25:29 smithi191 ceph-mon[32884]: 4.16 scrub starts 2024-09-10T19:25:29.467 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:25:29 smithi191 ceph-mon[32884]: 4.16 scrub ok 2024-09-10T19:25:29.467 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:25:29 smithi191 ceph-mon[32884]: from='mgr.14556 ' entity='mgr.y' 2024-09-10T19:25:29.467 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:25:29 smithi191 ceph-mon[32884]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:25:29.467 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:25:29 smithi191 ceph-mon[32884]: 2.0 deep-scrub starts 2024-09-10T19:25:29.468 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:25:29 smithi191 ceph-mon[32884]: 2.0 deep-scrub ok 2024-09-10T19:25:29.505 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:25:29.534 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:25:29.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:25:29.535 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:25:29.630 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-10T19:25:29.636 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:25:29.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:25:29.710 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:25:29.710 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:25:29.811 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:25:29.841 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:25:29.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:25:29.842 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:25:29.860 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: 69 15 97169 97154 87.9922 73 0.00735009 0.0113599 2024-09-10T19:25:29.942 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:25:29.966 INFO:journalctl@ceph.iscsi.iscsi.a.smithi191.stdout:Sep 10 19:25:29 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-iscsi-iscsi-a[61816]: debug there is no tcmu-runner data available 2024-09-10T19:25:29.988 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:25:29.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:25:29.989 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:25:30.090 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:25:30.133 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:25:30.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:25:30.134 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:25:30.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:25:30 smithi183 ceph-mon[29736]: 5.f scrub starts 2024-09-10T19:25:30.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:25:30 smithi183 ceph-mon[29736]: 5.9 deep-scrub starts 2024-09-10T19:25:30.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:25:30 smithi183 ceph-mon[41562]: 5.f scrub starts 2024-09-10T19:25:30.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:25:30 smithi183 ceph-mon[41562]: 5.9 deep-scrub starts 2024-09-10T19:25:30.235 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:25:30.261 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:25:30.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:25:30.262 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:25:30.363 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:25:30.388 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:25:30.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:25:30.390 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:25:30.465 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:25:30 smithi191 ceph-mon[32884]: 5.f scrub starts 2024-09-10T19:25:30.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:25:30 smithi191 ceph-mon[32884]: 5.9 deep-scrub starts 2024-09-10T19:25:30.490 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:25:30.532 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:25:30.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:25:30.533 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:25:30.634 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:25:30.675 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:25:30.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:25:30.676 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:25:30.777 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:25:30.809 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:25:30.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:25:30.810 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:25:30.860 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: 70 16 98223 98207 87.6752 65.8125 0.011317 0.0114007 2024-09-10T19:25:30.912 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:25:30.941 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:25:30.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:25:30.942 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:25:31.043 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:25:31.086 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:25:31.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:25:31.087 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:25:31.188 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:25:31.217 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:25:31.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:25:31.219 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:25:31.320 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:25:31.345 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Sep 10 19:25:31 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-y[29989]: ::ffff:172.21.15.191 - - [10/Sep/2024:19:25:31] "GET /metrics HTTP/1.1" 200 224771 "" "Prometheus/2.43.0" 2024-09-10T19:25:31.345 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:25:31 smithi183 ceph-mon[29736]: 5.f scrub ok 2024-09-10T19:25:31.345 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:25:31 smithi183 ceph-mon[29736]: from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:25:31.346 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:25:31 smithi183 ceph-mon[29736]: 2.a scrub starts 2024-09-10T19:25:31.346 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:25:31 smithi183 ceph-mon[29736]: 2.a scrub ok 2024-09-10T19:25:31.346 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:25:31 smithi183 ceph-mon[29736]: 5.9 deep-scrub ok 2024-09-10T19:25:31.346 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:25:31 smithi183 ceph-mon[29736]: 4.1 deep-scrub starts 2024-09-10T19:25:31.346 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:25:31 smithi183 ceph-mon[29736]: 4.1 deep-scrub ok 2024-09-10T19:25:31.346 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:25:31 smithi183 ceph-mon[29736]: pgmap v706: 158 pgs: 158 active+clean; 5.7 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 61 MiB/s wr, 966 op/s 2024-09-10T19:25:31.346 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:25:31 smithi183 ceph-mon[41562]: 5.f scrub ok 2024-09-10T19:25:31.346 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:25:31 smithi183 ceph-mon[41562]: from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:25:31.346 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:25:31 smithi183 ceph-mon[41562]: 2.a scrub starts 2024-09-10T19:25:31.346 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:25:31 smithi183 ceph-mon[41562]: 2.a scrub ok 2024-09-10T19:25:31.346 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:25:31 smithi183 ceph-mon[41562]: 5.9 deep-scrub ok 2024-09-10T19:25:31.346 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:25:31 smithi183 ceph-mon[41562]: 4.1 deep-scrub starts 2024-09-10T19:25:31.347 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:25:31 smithi183 ceph-mon[41562]: 4.1 deep-scrub ok 2024-09-10T19:25:31.347 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:25:31 smithi183 ceph-mon[41562]: pgmap v706: 158 pgs: 158 active+clean; 5.7 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 61 MiB/s wr, 966 op/s 2024-09-10T19:25:31.352 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:25:31.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:25:31.353 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:25:31.454 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:25:31.465 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:25:31 smithi191 ceph-mon[32884]: 5.f scrub ok 2024-09-10T19:25:31.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:25:31 smithi191 ceph-mon[32884]: from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:25:31.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:25:31 smithi191 ceph-mon[32884]: 2.a scrub starts 2024-09-10T19:25:31.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:25:31 smithi191 ceph-mon[32884]: 2.a scrub ok 2024-09-10T19:25:31.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:25:31 smithi191 ceph-mon[32884]: 5.9 deep-scrub ok 2024-09-10T19:25:31.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:25:31 smithi191 ceph-mon[32884]: 4.1 deep-scrub starts 2024-09-10T19:25:31.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:25:31 smithi191 ceph-mon[32884]: 4.1 deep-scrub ok 2024-09-10T19:25:31.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:25:31 smithi191 ceph-mon[32884]: pgmap v706: 158 pgs: 158 active+clean; 5.7 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 61 MiB/s wr, 966 op/s 2024-09-10T19:25:31.487 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:25:31.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:25:31.488 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:25:31.589 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:25:31.617 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:25:31.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:25:31.618 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:25:31.719 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:25:31.747 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:25:31.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:25:31.749 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:25:31.852 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:25:31.860 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: 71 16 99467 99451 87.5349 77.75 0.0115729 0.0114196 2024-09-10T19:25:31.890 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:25:31.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:25:31.891 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:25:31.992 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:25:32.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:25:32.018 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:25:32.019 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:25:32.090 INFO:teuthology.orchestra.run.smithi183.stderr:noscrub is set 2024-09-10T19:25:32.120 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:25:32.158 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:25:32.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:25:32.159 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:25:32.260 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:25:32.296 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:25:32.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:25:32.298 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:25:32.399 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:25:32.423 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:25:32 smithi183 ceph-mon[29736]: 6.8 scrub starts 2024-09-10T19:25:32.423 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:25:32 smithi183 ceph-mon[29736]: 6.8 scrub ok 2024-09-10T19:25:32.424 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:25:32 smithi183 ceph-mon[29736]: from='client.? 172.21.15.183:0/125121056' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-10T19:25:32.424 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:25:32 smithi183 ceph-mon[29736]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-10T19:25:32.424 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:25:32 smithi183 ceph-mon[29736]: 10.0 scrub ok 2024-09-10T19:25:32.424 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:25:32 smithi183 ceph-mon[29736]: 4.1f scrub starts 2024-09-10T19:25:32.424 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:25:32 smithi183 ceph-mon[29736]: 4.1f scrub ok 2024-09-10T19:25:32.424 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:25:32 smithi183 ceph-mon[41562]: 6.8 scrub starts 2024-09-10T19:25:32.424 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:25:32 smithi183 ceph-mon[41562]: 6.8 scrub ok 2024-09-10T19:25:32.424 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:25:32 smithi183 ceph-mon[41562]: from='client.? 172.21.15.183:0/125121056' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-10T19:25:32.424 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:25:32 smithi183 ceph-mon[41562]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-10T19:25:32.424 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:25:32 smithi183 ceph-mon[41562]: 10.0 scrub ok 2024-09-10T19:25:32.424 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:25:32 smithi183 ceph-mon[41562]: 4.1f scrub starts 2024-09-10T19:25:32.424 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:25:32 smithi183 ceph-mon[41562]: 4.1f scrub ok 2024-09-10T19:25:32.436 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:25:32.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:25:32.437 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:25:32.465 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:25:32 smithi191 ceph-mon[32884]: 6.8 scrub starts 2024-09-10T19:25:32.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:25:32 smithi191 ceph-mon[32884]: 6.8 scrub ok 2024-09-10T19:25:32.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:25:32 smithi191 ceph-mon[32884]: from='client.? 172.21.15.183:0/125121056' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-10T19:25:32.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:25:32 smithi191 ceph-mon[32884]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-10T19:25:32.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:25:32 smithi191 ceph-mon[32884]: 10.0 scrub ok 2024-09-10T19:25:32.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:25:32 smithi191 ceph-mon[32884]: 4.1f scrub starts 2024-09-10T19:25:32.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:25:32 smithi191 ceph-mon[32884]: 4.1f scrub ok 2024-09-10T19:25:32.539 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:25:32.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:25:32.568 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:25:32.568 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:25:32.669 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:25:32.709 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:25:32.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:25:32.710 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:25:32.811 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:25:32.845 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:25:32.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:25:32.846 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:25:32.860 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: 72 16 100747 100731 87.4301 80 0.0140142 0.0114333 2024-09-10T19:25:32.947 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:25:32.982 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:25:32.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:25:32.983 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:25:33.084 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:25:33.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:25:33.125 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:25:33.126 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:25:33.226 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:25:33.257 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:25:33.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:25:33.258 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:25:33.354 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:25:33 smithi183 ceph-mon[29736]: 4.1d scrub starts 2024-09-10T19:25:33.354 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:25:33 smithi183 ceph-mon[29736]: 4.1d scrub ok 2024-09-10T19:25:33.354 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:25:33 smithi183 ceph-mon[29736]: pgmap v707: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 5.9 GiB data, 25 GiB used, 600 GiB / 626 GiB avail; 67 MiB/s wr, 1.08k op/s 2024-09-10T19:25:33.355 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:25:33 smithi183 ceph-mon[29736]: 6.1a scrub starts 2024-09-10T19:25:33.355 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:25:33 smithi183 ceph-mon[29736]: 6.1a scrub ok 2024-09-10T19:25:33.355 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:25:33 smithi183 ceph-mon[29736]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T19:25:33.355 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:25:33 smithi183 ceph-mon[29736]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-10T19:25:33.355 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:25:33 smithi183 ceph-mon[29736]: osdmap e334: 8 total, 8 up, 7 in 2024-09-10T19:25:33.355 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:25:33 smithi183 ceph-mon[41562]: 4.1d scrub starts 2024-09-10T19:25:33.355 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:25:33 smithi183 ceph-mon[41562]: 4.1d scrub ok 2024-09-10T19:25:33.356 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:25:33 smithi183 ceph-mon[41562]: pgmap v707: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 5.9 GiB data, 25 GiB used, 600 GiB / 626 GiB avail; 67 MiB/s wr, 1.08k op/s 2024-09-10T19:25:33.356 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:25:33 smithi183 ceph-mon[41562]: 6.1a scrub starts 2024-09-10T19:25:33.356 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:25:33 smithi183 ceph-mon[41562]: 6.1a scrub ok 2024-09-10T19:25:33.356 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:25:33 smithi183 ceph-mon[41562]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T19:25:33.356 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:25:33 smithi183 ceph-mon[41562]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-10T19:25:33.356 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:25:33 smithi183 ceph-mon[41562]: osdmap e334: 8 total, 8 up, 7 in 2024-09-10T19:25:33.359 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:25:33.395 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:25:33.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:25:33.396 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:25:33.465 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:25:33 smithi191 ceph-mon[32884]: 4.1d scrub starts 2024-09-10T19:25:33.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:25:33 smithi191 ceph-mon[32884]: 4.1d scrub ok 2024-09-10T19:25:33.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:25:33 smithi191 ceph-mon[32884]: pgmap v707: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 5.9 GiB data, 25 GiB used, 600 GiB / 626 GiB avail; 67 MiB/s wr, 1.08k op/s 2024-09-10T19:25:33.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:25:33 smithi191 ceph-mon[32884]: 6.1a scrub starts 2024-09-10T19:25:33.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:25:33 smithi191 ceph-mon[32884]: 6.1a scrub ok 2024-09-10T19:25:33.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:25:33 smithi191 ceph-mon[32884]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T19:25:33.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:25:33 smithi191 ceph-mon[32884]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-10T19:25:33.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:25:33 smithi191 ceph-mon[32884]: osdmap e334: 8 total, 8 up, 7 in 2024-09-10T19:25:33.497 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:25:33.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:25:33.531 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:25:33.531 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:25:33.632 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:25:33.657 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:25:33.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:25:33.658 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:25:33.759 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:25:33.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:25:33.798 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:25:33.798 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:25:33.861 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: 73 16 102015 101999 87.318 79.25 0.00577921 0.011448 2024-09-10T19:25:33.900 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:25:33.933 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:25:33.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:25:33.934 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:25:34.035 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:25:34.072 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:25:34.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:25:34.074 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:25:34.110 INFO:teuthology.orchestra.run.smithi183.stdout:{ 2024-09-10T19:25:34.111 INFO:teuthology.orchestra.run.smithi183.stdout: "ops": [], 2024-09-10T19:25:34.111 INFO:teuthology.orchestra.run.smithi183.stdout: "complaint_time": 30, 2024-09-10T19:25:34.111 INFO:teuthology.orchestra.run.smithi183.stdout: "num_blocked_ops": 0 2024-09-10T19:25:34.111 INFO:teuthology.orchestra.run.smithi183.stdout:} 2024-09-10T19:25:34.175 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:25:34.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:25:34.213 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:25:34.214 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:25:34.314 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:25:34.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:25:34.343 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:25:34.343 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:25:34.444 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:25:34.475 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:25:34.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:25:34.477 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:25:34.578 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:25:34.611 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:25:34.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:25:34.612 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:25:34.713 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:25:34.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:25:34.741 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:25:34.741 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:25:34.843 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:25:34.860 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: 74 16 103203 103187 87.1413 74.25 0.00774218 0.0114713 2024-09-10T19:25:34.877 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:25:34.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:25:34.878 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:25:34.979 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:25:35.004 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:25:35.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:25:35.005 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:25:35.106 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:25:35.134 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:25:35.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:25:35.136 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:25:35.236 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:25:35.258 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:25:35.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:25:35.259 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:25:35.359 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:25:35.416 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:25:35.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:25:35.418 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:25:35.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:25:35 smithi183 ceph-mon[29736]: pgmap v709: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 6.0 GiB data, 26 GiB used, 600 GiB / 626 GiB avail; 1023 B/s rd, 73 MiB/s wr, 1.17k op/s 2024-09-10T19:25:35.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:25:35 smithi183 ceph-mon[29736]: osdmap e335: 8 total, 8 up, 7 in 2024-09-10T19:25:35.431 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:25:35 smithi183 ceph-mon[41562]: pgmap v709: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 6.0 GiB data, 26 GiB used, 600 GiB / 626 GiB avail; 1023 B/s rd, 73 MiB/s wr, 1.17k op/s 2024-09-10T19:25:35.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:25:35 smithi183 ceph-mon[41562]: osdmap e335: 8 total, 8 up, 7 in 2024-09-10T19:25:35.465 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:25:35 smithi191 ceph-mon[32884]: pgmap v709: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 6.0 GiB data, 26 GiB used, 600 GiB / 626 GiB avail; 1023 B/s rd, 73 MiB/s wr, 1.17k op/s 2024-09-10T19:25:35.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:25:35 smithi191 ceph-mon[32884]: osdmap e335: 8 total, 8 up, 7 in 2024-09-10T19:25:35.519 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:25:35.548 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:25:35.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:25:35.549 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:25:35.650 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:25:35.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:25:35.679 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:25:35.679 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:25:35.780 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:25:35.821 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-10T19:25:35.829 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:25:35.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:25:35.830 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:25:35.861 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: 75 16 104240 104224 86.8435 64.8125 0.0106784 0.0115105 2024-09-10T19:25:35.930 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:25:35.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:25:35.970 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:25:35.971 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:25:36.071 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:25:36.099 INFO:teuthology.orchestra.run.smithi183.stderr:reweighted osd.3 to 1 (10000) 2024-09-10T19:25:36.100 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:25:36 smithi183 ceph-mon[29736]: from='client.? 172.21.15.183:0/2487865266' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-09-10T19:25:36.100 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:25:36 smithi183 ceph-mon[29736]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T19:25:36.100 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:25:36 smithi183 ceph-mon[41562]: from='client.? 172.21.15.183:0/2487865266' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-09-10T19:25:36.100 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:25:36 smithi183 ceph-mon[41562]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T19:25:36.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:25:36.102 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:25:36.102 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:25:36.202 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:25:36.239 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:25:36.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:25:36.240 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:25:36.341 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:25:36.382 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:25:36.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:25:36.383 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:25:36.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:25:36 smithi191 ceph-mon[32884]: from='client.? 172.21.15.183:0/2487865266' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-09-10T19:25:36.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:25:36 smithi191 ceph-mon[32884]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T19:25:36.484 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:25:36.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:25:36.520 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:25:36.521 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:25:36.621 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:25:36.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:25:36.660 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:25:36.660 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:25:36.761 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:25:36.798 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:25:36.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:25:36.799 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:25:36.861 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: 76 16 105426 105410 86.6761 74.125 0.0103441 0.0115332 2024-09-10T19:25:36.900 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:25:36.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:25:36.935 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:25:36.935 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:25:37.036 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:25:37.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:25:37.075 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:25:37.075 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:25:37.176 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:25:37.211 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:25:37.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:25:37.212 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:25:37.312 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:25:37.338 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:25:37.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:25:37.339 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:25:37.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:25:37 smithi183 ceph-mon[29736]: pgmap v711: 158 pgs: 158 active+clean; 6.3 GiB data, 26 GiB used, 599 GiB / 626 GiB avail; 639 B/s rd, 78 MiB/s wr, 1.25k op/s 2024-09-10T19:25:37.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:25:37 smithi183 ceph-mon[29736]: from='client.? 172.21.15.183:0/2487865266' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-09-10T19:25:37.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:25:37 smithi183 ceph-mon[29736]: osdmap e336: 8 total, 8 up, 7 in 2024-09-10T19:25:37.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:25:37 smithi183 ceph-mon[41562]: pgmap v711: 158 pgs: 158 active+clean; 6.3 GiB data, 26 GiB used, 599 GiB / 626 GiB avail; 639 B/s rd, 78 MiB/s wr, 1.25k op/s 2024-09-10T19:25:37.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:25:37 smithi183 ceph-mon[41562]: from='client.? 172.21.15.183:0/2487865266' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-09-10T19:25:37.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:25:37 smithi183 ceph-mon[41562]: osdmap e336: 8 total, 8 up, 7 in 2024-09-10T19:25:37.440 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:25:37.450 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:25:37 smithi191 ceph-mon[32884]: pgmap v711: 158 pgs: 158 active+clean; 6.3 GiB data, 26 GiB used, 599 GiB / 626 GiB avail; 639 B/s rd, 78 MiB/s wr, 1.25k op/s 2024-09-10T19:25:37.450 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:25:37 smithi191 ceph-mon[32884]: from='client.? 172.21.15.183:0/2487865266' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-09-10T19:25:37.450 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:25:37 smithi191 ceph-mon[32884]: osdmap e336: 8 total, 8 up, 7 in 2024-09-10T19:25:37.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:25:37.462 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:25:37.462 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:25:37.563 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:25:37.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:25:37.604 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:25:37.604 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:25:37.705 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:25:37.715 INFO:journalctl@ceph.mgr.x.smithi191.stdout:Sep 10 19:25:37 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-x[34967]: ::ffff:172.21.15.191 - - [10/Sep/2024:19:25:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T19:25:37.747 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:25:37.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:25:37.748 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:25:37.849 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:25:37.861 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: 77 16 106613 106597 86.5139 74.1875 0.0163159 0.0115547 2024-09-10T19:25:37.884 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:25:37.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:25:37.885 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:25:37.919 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph daemon osd.1 dump_historic_ops 2024-09-10T19:25:37.986 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:25:38.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:25:38.021 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:25:38.021 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:25:38.122 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:25:38.159 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:25:38.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:25:38.160 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:25:38.213 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:25:38.261 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:25:38.296 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:25:38.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:25:38.297 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:25:38.397 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:25:38.435 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:25:38.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:25:38.436 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:25:38.537 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:25:38.570 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:25:38.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:25:38.571 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:25:38.672 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:25:38.701 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:25:38.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:25:38.702 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:25:38.803 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:25:38.831 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:25:38.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:25:38.832 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:25:38.861 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: 78 16 107887 107871 86.4255 79.625 0.00740198 0.0115666 2024-09-10T19:25:38.933 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:25:38.961 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:25:38.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:25:38.962 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:25:39.062 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:25:39.088 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph osd reweight 4 1 2024-09-10T19:25:39.110 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:25:39.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:25:39.111 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:25:39.212 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:25:39.244 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:25:39.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:25:39.245 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:25:39.307 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:25:39 smithi183 ceph-mon[29736]: pgmap v713: 158 pgs: 158 active+clean; 6.3 GiB data, 27 GiB used, 599 GiB / 626 GiB avail; 1.7 KiB/s rd, 77 MiB/s wr, 1.24k op/s 2024-09-10T19:25:39.307 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:25:39 smithi183 ceph-mon[29736]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T19:25:39.307 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:25:39 smithi183 ceph-mon[29736]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T19:25:39.307 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:25:39 smithi183 ceph-mon[29736]: from='mgr.14556 ' entity='mgr.y' 2024-09-10T19:25:39.307 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:25:39 smithi183 ceph-mon[41562]: pgmap v713: 158 pgs: 158 active+clean; 6.3 GiB data, 27 GiB used, 599 GiB / 626 GiB avail; 1.7 KiB/s rd, 77 MiB/s wr, 1.24k op/s 2024-09-10T19:25:39.307 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:25:39 smithi183 ceph-mon[41562]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T19:25:39.307 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:25:39 smithi183 ceph-mon[41562]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T19:25:39.307 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:25:39 smithi183 ceph-mon[41562]: from='mgr.14556 ' entity='mgr.y' 2024-09-10T19:25:39.346 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:25:39.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:25:39.383 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:25:39.384 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:25:39.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:25:39 smithi191 ceph-mon[32884]: pgmap v713: 158 pgs: 158 active+clean; 6.3 GiB data, 27 GiB used, 599 GiB / 626 GiB avail; 1.7 KiB/s rd, 77 MiB/s wr, 1.24k op/s 2024-09-10T19:25:39.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:25:39 smithi191 ceph-mon[32884]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T19:25:39.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:25:39 smithi191 ceph-mon[32884]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T19:25:39.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:25:39 smithi191 ceph-mon[32884]: from='mgr.14556 ' entity='mgr.y' 2024-09-10T19:25:39.484 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:25:39.497 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:25:39.521 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:25:39.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:25:39.522 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:25:39.543 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:25:39.623 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:25:39.653 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:25:39.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:25:39.654 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:25:39.755 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:25:39.783 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:25:39.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:25:39.784 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:25:39.861 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: 79 16 109165 109149 86.3425 79.875 0.0187259 0.0115778 2024-09-10T19:25:39.885 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:25:39.915 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:25:39.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:25:39.916 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:25:40.017 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:25:40.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:25:40.047 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:25:40.047 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:25:40.105 INFO:journalctl@ceph.iscsi.iscsi.a.smithi191.stdout:Sep 10 19:25:39 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-iscsi-iscsi-a[61816]: debug there is no tcmu-runner data available 2024-09-10T19:25:40.148 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:25:40.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:25:40 smithi183 ceph-mon[29736]: purged_snaps scrub starts 2024-09-10T19:25:40.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:25:40 smithi183 ceph-mon[29736]: purged_snaps scrub ok 2024-09-10T19:25:40.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:25:40 smithi183 ceph-mon[29736]: osdmap e337: 8 total, 8 up, 7 in 2024-09-10T19:25:40.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:25:40 smithi183 ceph-mon[41562]: purged_snaps scrub starts 2024-09-10T19:25:40.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:25:40 smithi183 ceph-mon[41562]: purged_snaps scrub ok 2024-09-10T19:25:40.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:25:40 smithi183 ceph-mon[41562]: osdmap e337: 8 total, 8 up, 7 in 2024-09-10T19:25:40.188 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:25:40.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:25:40.189 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:25:40.290 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:25:40.329 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:25:40.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:25:40.330 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:25:40.431 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:25:40.465 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:25:40 smithi191 ceph-mon[32884]: purged_snaps scrub starts 2024-09-10T19:25:40.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:25:40 smithi191 ceph-mon[32884]: purged_snaps scrub ok 2024-09-10T19:25:40.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:25:40 smithi191 ceph-mon[32884]: osdmap e337: 8 total, 8 up, 7 in 2024-09-10T19:25:40.469 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:25:40.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:25:40.470 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:25:40.571 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:25:40.601 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:25:40.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:25:40.602 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:25:40.703 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:25:40.740 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:25:40.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:25:40.741 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:25:40.842 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:25:40.861 INFO:tasks.radosbench.radosbench.0.smithi183.stdout:2024-09-10T19:25:40.860693+0000 min lat: 0.00110744 max lat: 2.06031 avg lat: 0.0115906 2024-09-10T19:25:40.861 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-10T19:25:40.861 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: 80 16 110423 110407 86.2459 78.625 0.0081144 0.0115906 2024-09-10T19:25:40.873 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:25:40.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:25:40.874 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:25:40.975 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:25:41.001 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:25:41.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:25:41.002 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:25:41.103 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:25:41.138 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:25:41.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:25:41.139 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:25:41.239 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:25:41.270 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:25:41.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:25:41.271 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:25:41.313 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:25:41 smithi183 ceph-mon[29736]: from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:25:41.314 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:25:41 smithi183 ceph-mon[29736]: pgmap v714: 158 pgs: 158 active+clean; 6.3 GiB data, 27 GiB used, 599 GiB / 626 GiB avail; 1.3 KiB/s rd, 51 MiB/s wr, 818 op/s 2024-09-10T19:25:41.314 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:25:41 smithi183 ceph-mon[41562]: from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:25:41.314 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:25:41 smithi183 ceph-mon[41562]: pgmap v714: 158 pgs: 158 active+clean; 6.3 GiB data, 27 GiB used, 599 GiB / 626 GiB avail; 1.3 KiB/s rd, 51 MiB/s wr, 818 op/s 2024-09-10T19:25:41.371 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:25:41.404 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:25:41.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:25:41.405 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:25:41.465 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:25:41 smithi191 ceph-mon[32884]: from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:25:41.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:25:41 smithi191 ceph-mon[32884]: pgmap v714: 158 pgs: 158 active+clean; 6.3 GiB data, 27 GiB used, 599 GiB / 626 GiB avail; 1.3 KiB/s rd, 51 MiB/s wr, 818 op/s 2024-09-10T19:25:41.506 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:25:41.546 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:25:41.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:25:41.547 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:25:41.618 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Sep 10 19:25:41 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-y[29989]: ::ffff:172.21.15.191 - - [10/Sep/2024:19:25:41] "GET /metrics HTTP/1.1" 200 224771 "" "Prometheus/2.43.0" 2024-09-10T19:25:41.647 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:25:41.684 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:25:41.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:25:41.686 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:25:41.786 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:25:41.816 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:25:41.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:25:41.817 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:25:41.861 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: 81 16 111711 111695 86.1749 80.5 0.00770665 0.0116003 2024-09-10T19:25:41.918 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:25:41.948 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:25:41.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:25:41.949 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:25:42.050 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:25:42.097 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:25:42.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:25:42.098 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:25:42.199 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:25:42.234 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:25:42.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:25:42.236 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:25:42.332 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:25:42.337 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:25:42.373 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:25:42.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:25:42.374 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:25:42.475 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:25:42.514 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:25:42.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:25:42.515 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:25:42.616 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:25:42.642 INFO:teuthology.orchestra.run.smithi183.stdout:{ 2024-09-10T19:25:42.642 INFO:teuthology.orchestra.run.smithi183.stdout: "size": 20, 2024-09-10T19:25:42.642 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 600, 2024-09-10T19:25:42.642 INFO:teuthology.orchestra.run.smithi183.stdout: "ops": [ 2024-09-10T19:25:42.643 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:25:42.643 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.25622.0:16143 10.2 10:47fd5a5f:::benchmark_data_smithi183_176537_object16142:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e309)", 2024-09-10T19:25:42.643 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-10T19:24:32.193051+0000", 2024-09-10T19:25:42.643 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 70.445463371000002, 2024-09-10T19:25:42.643 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.043770879999999998, 2024-09-10T19:25:42.643 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-10T19:25:42.643 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T19:25:42.643 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-10T19:25:42.643 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.25622", 2024-09-10T19:25:42.643 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.183:0/3685189199", 2024-09-10T19:25:42.643 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 16143 2024-09-10T19:25:42.643 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:25:42.643 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-10T19:25:42.643 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:25:42.644 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-10T19:25:42.644 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:24:32.193051+0000", 2024-09-10T19:25:42.644 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:25:42.644 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:25:42.644 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:25:42.644 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-10T19:25:42.644 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:24:32.193051+0000", 2024-09-10T19:25:42.644 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:25:42.644 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:25:42.644 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:25:42.644 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-10T19:25:42.644 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:24:32.193050+0000", 2024-09-10T19:25:42.644 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4294967295.999999 2024-09-10T19:25:42.645 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:25:42.645 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:25:42.645 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-10T19:25:42.645 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:24:32.193073+0000", 2024-09-10T19:25:42.645 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.2412000000000002e-05 2024-09-10T19:25:42.645 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:25:42.645 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:25:42.645 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-10T19:25:42.645 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:24:32.193084+0000", 2024-09-10T19:25:42.645 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.1911e-05 2024-09-10T19:25:42.645 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:25:42.645 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:25:42.646 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-10T19:25:42.646 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:24:32.193091+0000", 2024-09-10T19:25:42.646 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.224e-06 2024-09-10T19:25:42.646 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:25:42.646 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:25:42.646 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-10T19:25:42.646 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:24:32.193784+0000", 2024-09-10T19:25:42.646 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000693693 2024-09-10T19:25:42.646 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:25:42.646 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:25:42.646 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-10T19:25:42.647 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:24:32.193905+0000", 2024-09-10T19:25:42.647 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00012050299999999999 2024-09-10T19:25:42.647 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:25:42.647 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:25:42.647 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 0,4", 2024-09-10T19:25:42.647 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:24:32.194028+0000", 2024-09-10T19:25:42.647 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00012284000000000001 2024-09-10T19:25:42.647 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:25:42.647 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:25:42.647 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-10T19:25:42.647 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:24:32.194807+0000", 2024-09-10T19:25:42.647 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000779395 2024-09-10T19:25:42.647 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:25:42.648 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:25:42.648 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:25:42.648 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:24:32.195275+0000", 2024-09-10T19:25:42.648 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00046822600000000003 2024-09-10T19:25:42.648 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:25:42.648 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:25:42.648 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:25:42.648 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:24:32.236768+0000", 2024-09-10T19:25:42.648 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.041492656000000003 2024-09-10T19:25:42.648 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:25:42.648 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:25:42.648 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-10T19:25:42.648 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:24:32.236802+0000", 2024-09-10T19:25:42.649 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.3732000000000002e-05 2024-09-10T19:25:42.649 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:25:42.649 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:25:42.649 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-10T19:25:42.649 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:24:32.236822+0000", 2024-09-10T19:25:42.649 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.0429999999999999e-05 2024-09-10T19:25:42.649 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:25:42.649 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:25:42.649 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:25:42.649 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:25:42.649 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:25:42.650 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.25622.0:18276 10.2 10:4431d057:::benchmark_data_smithi183_176537_object18275:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e310)", 2024-09-10T19:25:42.650 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-10T19:24:33.290677+0000", 2024-09-10T19:25:42.650 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 69.347837736000002, 2024-09-10T19:25:42.650 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.31871982599999998, 2024-09-10T19:25:42.650 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-10T19:25:42.650 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T19:25:42.650 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-10T19:25:42.650 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.25622", 2024-09-10T19:25:42.650 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.183:0/3685189199", 2024-09-10T19:25:42.650 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 18276 2024-09-10T19:25:42.650 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:25:42.650 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-10T19:25:42.650 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:25:42.650 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-10T19:25:42.651 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:24:33.290677+0000", 2024-09-10T19:25:42.651 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:25:42.651 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:25:42.651 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:25:42.651 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-10T19:25:42.651 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:24:33.290677+0000", 2024-09-10T19:25:42.651 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:25:42.651 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:25:42.651 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:25:42.651 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-10T19:25:42.651 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:24:33.290675+0000", 2024-09-10T19:25:42.651 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4294967295.9999986 2024-09-10T19:25:42.651 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:25:42.651 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:25:42.651 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-10T19:25:42.652 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:24:33.290707+0000", 2024-09-10T19:25:42.652 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.1223000000000002e-05 2024-09-10T19:25:42.652 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:25:42.652 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:25:42.652 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-10T19:25:42.652 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:24:33.290723+0000", 2024-09-10T19:25:42.652 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.6351999999999999e-05 2024-09-10T19:25:42.652 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:25:42.652 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:25:42.652 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-10T19:25:42.652 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:24:33.290729+0000", 2024-09-10T19:25:42.652 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.7950000000000004e-06 2024-09-10T19:25:42.652 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:25:42.652 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:25:42.652 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-10T19:25:42.653 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:24:33.290782+0000", 2024-09-10T19:25:42.653 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.3128999999999999e-05 2024-09-10T19:25:42.653 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:25:42.653 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:25:42.653 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-10T19:25:42.653 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:24:33.290910+0000", 2024-09-10T19:25:42.653 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00012813899999999999 2024-09-10T19:25:42.653 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:25:42.653 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:25:42.653 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 0,4", 2024-09-10T19:25:42.653 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:24:33.291013+0000", 2024-09-10T19:25:42.653 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000103427 2024-09-10T19:25:42.654 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:25:42.654 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:25:42.654 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-10T19:25:42.654 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:24:33.292511+0000", 2024-09-10T19:25:42.654 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0014975349999999999 2024-09-10T19:25:42.654 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:25:42.654 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:25:42.654 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:25:42.654 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:24:33.292542+0000", 2024-09-10T19:25:42.654 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.1058e-05 2024-09-10T19:25:42.654 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:25:42.654 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:25:42.654 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:25:42.654 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:24:33.609342+0000", 2024-09-10T19:25:42.655 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.316800366 2024-09-10T19:25:42.655 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:25:42.655 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:25:42.655 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-10T19:25:42.655 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:24:33.609375+0000", 2024-09-10T19:25:42.655 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.2215000000000002e-05 2024-09-10T19:25:42.655 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:25:42.655 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:25:42.655 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-10T19:25:42.655 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:24:33.609397+0000", 2024-09-10T19:25:42.655 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.2082999999999998e-05 2024-09-10T19:25:42.655 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:25:42.655 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:25:42.656 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:25:42.656 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:25:42.656 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:25:42.656 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.25622.0:18277 10.2 10:431b5bb5:::benchmark_data_smithi183_176537_object18276:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e310)", 2024-09-10T19:25:42.656 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-10T19:24:33.290739+0000", 2024-09-10T19:25:42.656 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 69.347775221999996, 2024-09-10T19:25:42.656 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.31878863099999999, 2024-09-10T19:25:42.656 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-10T19:25:42.656 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T19:25:42.656 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-10T19:25:42.656 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.25622", 2024-09-10T19:25:42.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:25:42.658 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.183:0/3685189199", 2024-09-10T19:25:42.658 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 18277 2024-09-10T19:25:42.659 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:25:42.659 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-10T19:25:42.659 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:25:42.659 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-10T19:25:42.659 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:24:33.290739+0000", 2024-09-10T19:25:42.659 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:25:42.659 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:25:42.659 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:25:42.659 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-10T19:25:42.659 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:24:33.290739+0000", 2024-09-10T19:25:42.660 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:25:42.660 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:25:42.660 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:25:42.660 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-10T19:25:42.660 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:24:33.290739+0000", 2024-09-10T19:25:42.660 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4294967295.9999995 2024-09-10T19:25:42.660 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:25:42.660 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:25:42.660 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-10T19:25:42.660 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:24:33.290756+0000", 2024-09-10T19:25:42.660 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.6871000000000002e-05 2024-09-10T19:25:42.660 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:25:42.660 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:25:42.661 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-10T19:25:42.661 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:24:33.290760+0000", 2024-09-10T19:25:42.661 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.4889999999999997e-06 2024-09-10T19:25:42.661 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:25:42.661 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:25:42.661 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-10T19:25:42.661 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:24:33.290763+0000", 2024-09-10T19:25:42.661 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.8590000000000001e-06 2024-09-10T19:25:42.661 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:25:42.661 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:25:42.661 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-10T19:25:42.661 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:24:33.291659+0000", 2024-09-10T19:25:42.661 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00089572800000000004 2024-09-10T19:25:42.662 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:25:42.662 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:25:42.662 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-10T19:25:42.662 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:24:33.291779+0000", 2024-09-10T19:25:42.662 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000120222 2024-09-10T19:25:42.662 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:25:42.662 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:25:42.662 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 0,4", 2024-09-10T19:25:42.662 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:24:33.291881+0000", 2024-09-10T19:25:42.662 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000101563 2024-09-10T19:25:42.662 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:25:42.663 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:25:42.663 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-10T19:25:42.663 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:24:33.292666+0000", 2024-09-10T19:25:42.663 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00078504499999999995 2024-09-10T19:25:42.663 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:25:42.663 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:25:42.663 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:25:42.663 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:24:33.292866+0000", 2024-09-10T19:25:42.663 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00020038400000000001 2024-09-10T19:25:42.663 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:25:42.663 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:25:42.663 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:25:42.663 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:24:33.609447+0000", 2024-09-10T19:25:42.664 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.31658076499999999 2024-09-10T19:25:42.664 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:25:42.664 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:25:42.664 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-10T19:25:42.664 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:24:33.609496+0000", 2024-09-10T19:25:42.664 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.9487999999999998e-05 2024-09-10T19:25:42.664 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:25:42.664 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:25:42.664 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-10T19:25:42.664 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:24:33.609528+0000", 2024-09-10T19:25:42.664 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.1609000000000002e-05 2024-09-10T19:25:42.664 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:25:42.664 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:25:42.665 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:25:42.665 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:25:42.665 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:25:42.665 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.25622.0:22562 10.2 10:49b34d59:::benchmark_data_smithi183_176537_object22561:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e311)", 2024-09-10T19:25:42.665 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-10T19:24:35.687974+0000", 2024-09-10T19:25:42.665 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 66.950540666999999, 2024-09-10T19:25:42.665 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.106139646, 2024-09-10T19:25:42.665 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-10T19:25:42.665 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T19:25:42.665 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-10T19:25:42.665 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.25622", 2024-09-10T19:25:42.665 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.183:0/3685189199", 2024-09-10T19:25:42.665 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 22562 2024-09-10T19:25:42.666 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:25:42.666 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-10T19:25:42.666 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:25:42.666 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-10T19:25:42.666 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:24:35.687974+0000", 2024-09-10T19:25:42.666 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:25:42.666 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:25:42.666 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:25:42.666 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-10T19:25:42.666 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:24:35.687974+0000", 2024-09-10T19:25:42.666 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:25:42.666 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:25:42.666 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:25:42.667 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-10T19:25:42.667 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:24:35.687973+0000", 2024-09-10T19:25:42.667 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4294967295.9999995 2024-09-10T19:25:42.667 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:25:42.667 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:25:42.667 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-10T19:25:42.667 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:24:35.687993+0000", 2024-09-10T19:25:42.667 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.9619e-05 2024-09-10T19:25:42.667 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:25:42.667 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:25:42.667 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-10T19:25:42.667 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:24:35.688000+0000", 2024-09-10T19:25:42.667 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.1280000000000002e-06 2024-09-10T19:25:42.667 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:25:42.667 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:25:42.668 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-10T19:25:42.668 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:24:35.688004+0000", 2024-09-10T19:25:42.668 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.9949999999999999e-06 2024-09-10T19:25:42.668 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:25:42.668 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:25:42.668 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-10T19:25:42.668 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:24:35.689233+0000", 2024-09-10T19:25:42.668 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.001228411 2024-09-10T19:25:42.668 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:25:42.668 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:25:42.668 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-10T19:25:42.669 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:24:35.689353+0000", 2024-09-10T19:25:42.669 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00012029800000000001 2024-09-10T19:25:42.669 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:25:42.669 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:25:42.669 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 0,4", 2024-09-10T19:25:42.669 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:24:35.689485+0000", 2024-09-10T19:25:42.669 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000131719 2024-09-10T19:25:42.669 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:25:42.669 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:25:42.669 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-10T19:25:42.670 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:24:35.690585+0000", 2024-09-10T19:25:42.670 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0011003199999999999 2024-09-10T19:25:42.670 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:25:42.670 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:25:42.670 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:25:42.670 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:24:35.692719+0000", 2024-09-10T19:25:42.670 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0021343790000000001 2024-09-10T19:25:42.670 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:25:42.670 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:25:42.670 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:25:42.670 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:24:35.794052+0000", 2024-09-10T19:25:42.670 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.101333065 2024-09-10T19:25:42.670 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:25:42.670 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:25:42.671 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-10T19:25:42.671 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:24:35.794088+0000", 2024-09-10T19:25:42.671 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.6106999999999998e-05 2024-09-10T19:25:42.671 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:25:42.671 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:25:42.671 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-10T19:25:42.671 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:24:35.794114+0000", 2024-09-10T19:25:42.671 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.5191999999999999e-05 2024-09-10T19:25:42.671 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:25:42.671 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:25:42.671 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:25:42.672 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:25:42.672 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:25:42.672 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.25622.0:50252 10.2 10:4b58b089:::benchmark_data_smithi183_176537_object50251:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e317)", 2024-09-10T19:25:42.672 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-10T19:24:51.279940+0000", 2024-09-10T19:25:42.672 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 51.358574396999998, 2024-09-10T19:25:42.672 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.054643285999999999, 2024-09-10T19:25:42.672 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-10T19:25:42.672 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T19:25:42.672 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-10T19:25:42.672 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.25622", 2024-09-10T19:25:42.672 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.183:0/3685189199", 2024-09-10T19:25:42.672 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 50252 2024-09-10T19:25:42.672 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:25:42.673 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-10T19:25:42.673 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:25:42.673 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-10T19:25:42.673 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:24:51.279940+0000", 2024-09-10T19:25:42.673 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:25:42.673 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:25:42.673 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:25:42.673 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-10T19:25:42.673 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:24:51.279940+0000", 2024-09-10T19:25:42.673 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:25:42.673 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:25:42.673 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:25:42.673 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-10T19:25:42.674 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:24:51.279938+0000", 2024-09-10T19:25:42.674 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4294967295.9999981 2024-09-10T19:25:42.674 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:25:42.674 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:25:42.674 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-10T19:25:42.674 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:24:51.279964+0000", 2024-09-10T19:25:42.674 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.5899000000000001e-05 2024-09-10T19:25:42.674 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:25:42.674 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:25:42.674 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-10T19:25:42.674 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:24:51.279981+0000", 2024-09-10T19:25:42.674 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.6892999999999999e-05 2024-09-10T19:25:42.674 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:25:42.674 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:25:42.675 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-10T19:25:42.675 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:24:51.279987+0000", 2024-09-10T19:25:42.675 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.4110000000000001e-06 2024-09-10T19:25:42.675 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:25:42.675 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:25:42.675 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-10T19:25:42.675 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:24:51.280040+0000", 2024-09-10T19:25:42.675 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.2938e-05 2024-09-10T19:25:42.675 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:25:42.675 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:25:42.675 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-10T19:25:42.675 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:24:51.280143+0000", 2024-09-10T19:25:42.675 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000102496 2024-09-10T19:25:42.675 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:25:42.676 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:25:42.676 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 0,4", 2024-09-10T19:25:42.676 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:24:51.280232+0000", 2024-09-10T19:25:42.676 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.9295999999999993e-05 2024-09-10T19:25:42.676 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:25:42.676 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:25:42.676 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-10T19:25:42.676 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:24:51.281790+0000", 2024-09-10T19:25:42.676 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.001558074 2024-09-10T19:25:42.676 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:25:42.676 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:25:42.676 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:25:42.677 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:24:51.281955+0000", 2024-09-10T19:25:42.677 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000164711 2024-09-10T19:25:42.677 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:25:42.677 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:25:42.677 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:25:42.677 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:24:51.334489+0000", 2024-09-10T19:25:42.677 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.052534096000000002 2024-09-10T19:25:42.677 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:25:42.677 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:25:42.677 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-10T19:25:42.677 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:24:51.334542+0000", 2024-09-10T19:25:42.677 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.2584999999999999e-05 2024-09-10T19:25:42.677 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:25:42.678 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:25:42.678 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-10T19:25:42.678 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:24:51.334583+0000", 2024-09-10T19:25:42.678 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.1899e-05 2024-09-10T19:25:42.678 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:25:42.678 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:25:42.678 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:25:42.678 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:25:42.678 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:25:42.678 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.25622.0:72808 10.2 10:4d66b71e:::benchmark_data_smithi183_176537_object72807:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e325)", 2024-09-10T19:25:42.678 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-10T19:25:08.485958+0000", 2024-09-10T19:25:42.678 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 34.152556658000002, 2024-09-10T19:25:42.678 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.064086874000000002, 2024-09-10T19:25:42.678 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-10T19:25:42.679 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T19:25:42.679 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-10T19:25:42.679 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.25622", 2024-09-10T19:25:42.679 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.183:0/3685189199", 2024-09-10T19:25:42.679 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 72808 2024-09-10T19:25:42.679 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:25:42.679 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-10T19:25:42.679 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:25:42.679 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-10T19:25:42.679 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:25:08.485958+0000", 2024-09-10T19:25:42.679 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:25:42.679 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:25:42.679 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:25:42.680 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-10T19:25:42.680 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:25:08.485958+0000", 2024-09-10T19:25:42.680 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:25:42.680 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:25:42.680 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:25:42.680 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-10T19:25:42.680 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:25:08.485956+0000", 2024-09-10T19:25:42.680 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4294967295.9999976 2024-09-10T19:25:42.680 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:25:42.680 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:25:42.680 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-10T19:25:42.680 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:25:08.485980+0000", 2024-09-10T19:25:42.680 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.4556999999999999e-05 2024-09-10T19:25:42.681 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:25:42.681 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:25:42.681 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-10T19:25:42.681 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:25:08.485998+0000", 2024-09-10T19:25:42.681 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.7385999999999999e-05 2024-09-10T19:25:42.681 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:25:42.681 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:25:42.681 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-10T19:25:42.681 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:25:08.486004+0000", 2024-09-10T19:25:42.681 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.0249999999999999e-06 2024-09-10T19:25:42.681 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:25:42.682 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:25:42.682 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-10T19:25:42.682 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:25:08.486050+0000", 2024-09-10T19:25:42.682 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.6650000000000002e-05 2024-09-10T19:25:42.682 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:25:42.682 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:25:42.682 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-10T19:25:42.682 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:25:08.486185+0000", 2024-09-10T19:25:42.682 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000135084 2024-09-10T19:25:42.682 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:25:42.682 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:25:42.682 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 0,4", 2024-09-10T19:25:42.682 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:25:08.486319+0000", 2024-09-10T19:25:42.683 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00013374599999999999 2024-09-10T19:25:42.683 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:25:42.683 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:25:42.683 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-10T19:25:42.683 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:25:08.488368+0000", 2024-09-10T19:25:42.683 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.002049192 2024-09-10T19:25:42.683 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:25:42.683 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:25:42.683 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:25:42.683 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:25:08.489322+0000", 2024-09-10T19:25:42.683 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00095376200000000003 2024-09-10T19:25:42.683 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:25:42.683 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:25:42.683 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:25:42.684 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:25:08.549952+0000", 2024-09-10T19:25:42.684 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.060629741000000001 2024-09-10T19:25:42.684 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:25:42.684 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:25:42.684 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-10T19:25:42.684 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:25:08.550003+0000", 2024-09-10T19:25:42.684 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.1048999999999999e-05 2024-09-10T19:25:42.684 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:25:42.684 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:25:42.684 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-10T19:25:42.684 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:25:08.550045+0000", 2024-09-10T19:25:42.684 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.1835999999999997e-05 2024-09-10T19:25:42.684 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:25:42.684 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:25:42.685 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:25:42.685 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:25:42.685 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:25:42.685 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.25622.0:73535 10.2 10:42044870:::benchmark_data_smithi183_176537_object73534:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e326)", 2024-09-10T19:25:42.685 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-10T19:25:09.158006+0000", 2024-09-10T19:25:42.685 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 33.480508139000001, 2024-09-10T19:25:42.685 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.059738140000000002, 2024-09-10T19:25:42.685 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-10T19:25:42.685 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T19:25:42.685 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-10T19:25:42.685 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.25622", 2024-09-10T19:25:42.685 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.183:0/3685189199", 2024-09-10T19:25:42.686 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 73535 2024-09-10T19:25:42.686 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:25:42.686 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-10T19:25:42.686 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:25:42.686 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-10T19:25:42.686 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:25:09.158006+0000", 2024-09-10T19:25:42.686 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:25:42.686 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:25:42.686 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:25:42.686 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-10T19:25:42.686 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:25:09.158006+0000", 2024-09-10T19:25:42.686 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:25:42.687 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:25:42.687 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:25:42.687 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-10T19:25:42.687 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:25:09.158005+0000", 2024-09-10T19:25:42.687 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4294967295.9999981 2024-09-10T19:25:42.687 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:25:42.687 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:25:42.687 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-10T19:25:42.687 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:25:09.158027+0000", 2024-09-10T19:25:42.687 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.2405e-05 2024-09-10T19:25:42.687 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:25:42.687 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:25:42.687 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-10T19:25:42.687 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:25:09.158040+0000", 2024-09-10T19:25:42.688 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.2639999999999999e-05 2024-09-10T19:25:42.688 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:25:42.688 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:25:42.688 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-10T19:25:42.688 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:25:09.158046+0000", 2024-09-10T19:25:42.688 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.0519999999999998e-06 2024-09-10T19:25:42.688 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:25:42.688 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:25:42.688 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-10T19:25:42.688 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:25:09.158088+0000", 2024-09-10T19:25:42.688 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.2348000000000002e-05 2024-09-10T19:25:42.688 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:25:42.688 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:25:42.689 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-10T19:25:42.689 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:25:09.158203+0000", 2024-09-10T19:25:42.689 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00011537800000000001 2024-09-10T19:25:42.689 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:25:42.689 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:25:42.689 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 0,4", 2024-09-10T19:25:42.689 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:25:09.158335+0000", 2024-09-10T19:25:42.689 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00013165400000000001 2024-09-10T19:25:42.689 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:25:42.689 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:25:42.689 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-10T19:25:42.689 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:25:09.160614+0000", 2024-09-10T19:25:42.689 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0022788940000000001 2024-09-10T19:25:42.689 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:25:42.690 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:25:42.690 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:25:42.690 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:25:09.164422+0000", 2024-09-10T19:25:42.690 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0038079450000000001 2024-09-10T19:25:42.690 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:25:42.690 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:25:42.690 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:25:42.690 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:25:09.217666+0000", 2024-09-10T19:25:42.690 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.053244167000000002 2024-09-10T19:25:42.690 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:25:42.690 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:25:42.690 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-10T19:25:42.690 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:25:09.217718+0000", 2024-09-10T19:25:42.690 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.215e-05 2024-09-10T19:25:42.691 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:25:42.691 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:25:42.691 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-10T19:25:42.691 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:25:09.217745+0000", 2024-09-10T19:25:42.691 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.6349000000000001e-05 2024-09-10T19:25:42.691 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:25:42.691 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:25:42.691 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:25:42.691 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:25:42.691 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:25:42.691 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.25622.0:74118 10.2 10:46a3abd0:::benchmark_data_smithi183_176537_object74117:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e326)", 2024-09-10T19:25:42.691 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-10T19:25:09.827597+0000", 2024-09-10T19:25:42.691 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 32.810917815000003, 2024-09-10T19:25:42.691 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.043150797999999997, 2024-09-10T19:25:42.692 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-10T19:25:42.692 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T19:25:42.692 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-10T19:25:42.692 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.25622", 2024-09-10T19:25:42.692 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.183:0/3685189199", 2024-09-10T19:25:42.692 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 74118 2024-09-10T19:25:42.692 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:25:42.692 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-10T19:25:42.692 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:25:42.692 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-10T19:25:42.692 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:25:09.827597+0000", 2024-09-10T19:25:42.692 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:25:42.692 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:25:42.692 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:25:42.693 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-10T19:25:42.693 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:25:09.827597+0000", 2024-09-10T19:25:42.693 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:25:42.693 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:25:42.693 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:25:42.693 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-10T19:25:42.693 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:25:09.827595+0000", 2024-09-10T19:25:42.693 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4294967295.9999986 2024-09-10T19:25:42.693 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:25:42.693 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:25:42.693 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-10T19:25:42.693 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:25:09.827626+0000", 2024-09-10T19:25:42.693 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.0443e-05 2024-09-10T19:25:42.694 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:25:42.694 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:25:42.694 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-10T19:25:42.694 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:25:09.827642+0000", 2024-09-10T19:25:42.694 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.6702e-05 2024-09-10T19:25:42.694 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:25:42.694 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:25:42.694 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-10T19:25:42.694 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:25:09.827649+0000", 2024-09-10T19:25:42.694 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.421e-06 2024-09-10T19:25:42.694 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:25:42.694 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:25:42.695 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-10T19:25:42.695 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:25:09.827703+0000", 2024-09-10T19:25:42.695 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.4317999999999997e-05 2024-09-10T19:25:42.695 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:25:42.695 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:25:42.695 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-10T19:25:42.695 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:25:09.827815+0000", 2024-09-10T19:25:42.695 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000111526 2024-09-10T19:25:42.695 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:25:42.695 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:25:42.695 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 0,4", 2024-09-10T19:25:42.695 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:25:09.827912+0000", 2024-09-10T19:25:42.695 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.7765999999999999e-05 2024-09-10T19:25:42.696 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:25:42.696 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:25:42.696 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-10T19:25:42.696 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:25:09.829889+0000", 2024-09-10T19:25:42.696 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0019769900000000001 2024-09-10T19:25:42.696 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:25:42.696 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:25:42.696 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:25:42.696 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:25:09.830078+0000", 2024-09-10T19:25:42.696 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00018855 2024-09-10T19:25:42.697 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:25:42.697 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:25:42.697 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:25:42.697 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:25:09.870578+0000", 2024-09-10T19:25:42.697 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.040499968999999997 2024-09-10T19:25:42.697 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:25:42.697 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:25:42.697 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-10T19:25:42.697 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:25:09.870629+0000", 2024-09-10T19:25:42.697 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.0788999999999998e-05 2024-09-10T19:25:42.698 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:25:42.698 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:25:42.698 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-10T19:25:42.698 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:25:09.870748+0000", 2024-09-10T19:25:42.698 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00011880299999999999 2024-09-10T19:25:42.698 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:25:42.698 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:25:42.698 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:25:42.698 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:25:42.699 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:25:42.699 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.25622.0:77065 10.2 10:463f93c5:::benchmark_data_smithi183_176537_object77064:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e328)", 2024-09-10T19:25:42.699 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-10T19:25:12.979061+0000", 2024-09-10T19:25:42.699 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 29.659453153000001, 2024-09-10T19:25:42.699 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.058905055999999997, 2024-09-10T19:25:42.699 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-10T19:25:42.699 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T19:25:42.699 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-10T19:25:42.699 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.25622", 2024-09-10T19:25:42.700 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.183:0/3685189199", 2024-09-10T19:25:42.700 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 77065 2024-09-10T19:25:42.700 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:25:42.700 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-10T19:25:42.700 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:25:42.700 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-10T19:25:42.700 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:25:12.979061+0000", 2024-09-10T19:25:42.700 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:25:42.701 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:25:42.701 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:25:42.701 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-10T19:25:42.701 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:25:12.979061+0000", 2024-09-10T19:25:42.701 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:25:42.701 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:25:42.701 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:25:42.701 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-10T19:25:42.702 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:25:12.979060+0000", 2024-09-10T19:25:42.702 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4294967295.9999981 2024-09-10T19:25:42.702 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:25:42.702 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:25:42.702 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-10T19:25:42.702 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:25:12.979084+0000", 2024-09-10T19:25:42.702 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.4666000000000001e-05 2024-09-10T19:25:42.702 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:25:42.702 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:25:42.703 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-10T19:25:42.703 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:25:12.979099+0000", 2024-09-10T19:25:42.703 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.4722999999999999e-05 2024-09-10T19:25:42.703 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:25:42.703 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:25:42.703 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-10T19:25:42.703 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:25:12.979105+0000", 2024-09-10T19:25:42.703 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.3439999999999997e-06 2024-09-10T19:25:42.703 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:25:42.704 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:25:42.704 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-10T19:25:42.704 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:25:12.979152+0000", 2024-09-10T19:25:42.704 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.6879000000000003e-05 2024-09-10T19:25:42.704 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:25:42.704 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:25:42.704 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-10T19:25:42.705 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:25:12.979253+0000", 2024-09-10T19:25:42.705 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00010098300000000001 2024-09-10T19:25:42.705 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:25:42.705 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:25:42.705 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 0,4", 2024-09-10T19:25:42.705 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:25:12.979351+0000", 2024-09-10T19:25:42.705 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.8023999999999995e-05 2024-09-10T19:25:42.706 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:25:42.706 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:25:42.708 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:25:42.708 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:25:42.708 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-10T19:25:42.708 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:25:12.981938+0000", 2024-09-10T19:25:42.708 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.002586442 2024-09-10T19:25:42.709 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:25:42.709 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:25:42.709 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:25:42.709 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:25:12.982075+0000", 2024-09-10T19:25:42.709 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00013692399999999999 2024-09-10T19:25:42.709 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:25:42.709 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:25:42.709 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:25:42.710 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:25:13.037865+0000", 2024-09-10T19:25:42.710 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.055790520000000003 2024-09-10T19:25:42.710 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:25:42.710 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:25:42.710 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-10T19:25:42.710 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:25:13.037935+0000", 2024-09-10T19:25:42.710 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.9968999999999994e-05 2024-09-10T19:25:42.710 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:25:42.710 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:25:42.711 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-10T19:25:42.711 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:25:13.037967+0000", 2024-09-10T19:25:42.711 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.1313999999999999e-05 2024-09-10T19:25:42.711 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:25:42.711 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:25:42.711 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:25:42.711 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:25:42.711 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:25:42.711 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.25622.0:78077 10.2 10:43d24a26:::benchmark_data_smithi183_176537_object78076:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e328)", 2024-09-10T19:25:42.711 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-10T19:25:13.846534+0000", 2024-09-10T19:25:42.711 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 28.791980305999999, 2024-09-10T19:25:42.712 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.039644276999999999, 2024-09-10T19:25:42.712 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-10T19:25:42.712 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T19:25:42.712 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-10T19:25:42.712 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.25622", 2024-09-10T19:25:42.712 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.183:0/3685189199", 2024-09-10T19:25:42.712 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 78077 2024-09-10T19:25:42.712 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:25:42.712 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-10T19:25:42.712 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:25:42.712 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-10T19:25:42.712 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:25:13.846534+0000", 2024-09-10T19:25:42.712 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:25:42.713 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:25:42.713 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:25:42.713 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-10T19:25:42.713 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:25:13.846534+0000", 2024-09-10T19:25:42.713 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:25:42.713 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:25:42.713 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:25:42.713 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-10T19:25:42.713 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:25:13.846531+0000", 2024-09-10T19:25:42.713 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4294967295.9999971 2024-09-10T19:25:42.713 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:25:42.713 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:25:42.713 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-10T19:25:42.713 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:25:13.846561+0000", 2024-09-10T19:25:42.714 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.9798000000000002e-05 2024-09-10T19:25:42.714 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:25:42.714 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:25:42.714 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-10T19:25:42.714 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:25:13.846572+0000", 2024-09-10T19:25:42.714 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.0550999999999999e-05 2024-09-10T19:25:42.714 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:25:42.714 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:25:42.714 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-10T19:25:42.714 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:25:13.846578+0000", 2024-09-10T19:25:42.714 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.8880000000000002e-06 2024-09-10T19:25:42.714 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:25:42.714 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:25:42.715 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-10T19:25:42.715 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:25:13.846610+0000", 2024-09-10T19:25:42.715 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.2804e-05 2024-09-10T19:25:42.715 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:25:42.715 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:25:42.715 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-10T19:25:42.715 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:25:13.846696+0000", 2024-09-10T19:25:42.715 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.5903999999999996e-05 2024-09-10T19:25:42.715 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:25:42.715 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:25:42.715 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 0,4", 2024-09-10T19:25:42.715 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:25:13.846778+0000", 2024-09-10T19:25:42.715 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.1607999999999995e-05 2024-09-10T19:25:42.716 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:25:42.716 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:25:42.716 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-10T19:25:42.716 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:25:13.848729+0000", 2024-09-10T19:25:42.716 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.001951113 2024-09-10T19:25:42.716 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:25:42.716 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:25:42.716 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:25:42.716 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:25:13.849983+0000", 2024-09-10T19:25:42.716 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0012541760000000001 2024-09-10T19:25:42.716 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:25:42.717 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:25:42.717 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:25:42.717 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:25:13.885937+0000", 2024-09-10T19:25:42.717 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.035954180000000002 2024-09-10T19:25:42.717 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:25:42.717 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:25:42.717 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-10T19:25:42.717 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:25:13.886140+0000", 2024-09-10T19:25:42.717 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000203062 2024-09-10T19:25:42.717 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:25:42.717 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:25:42.717 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-10T19:25:42.717 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:25:13.886179+0000", 2024-09-10T19:25:42.718 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.8080000000000001e-05 2024-09-10T19:25:42.718 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:25:42.718 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:25:42.718 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:25:42.718 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:25:42.718 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:25:42.718 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.25622.0:93856 10.2 10:47f7b6f9:::benchmark_data_smithi183_176537_object93855:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e333)", 2024-09-10T19:25:42.718 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-10T19:25:26.867119+0000", 2024-09-10T19:25:42.718 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 15.771395601, 2024-09-10T19:25:42.718 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.049869756000000001, 2024-09-10T19:25:42.718 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-10T19:25:42.719 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T19:25:42.719 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-10T19:25:42.719 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.25622", 2024-09-10T19:25:42.719 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.183:0/3685189199", 2024-09-10T19:25:42.719 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 93856 2024-09-10T19:25:42.719 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:25:42.719 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-10T19:25:42.719 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:25:42.719 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-10T19:25:42.719 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:25:26.867119+0000", 2024-09-10T19:25:42.719 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:25:42.719 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:25:42.719 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:25:42.719 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-10T19:25:42.720 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:25:26.867119+0000", 2024-09-10T19:25:42.720 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:25:42.720 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:25:42.720 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:25:42.720 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-10T19:25:42.720 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:25:26.867117+0000", 2024-09-10T19:25:42.720 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4294967295.9999981 2024-09-10T19:25:42.720 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:25:42.720 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:25:42.720 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-10T19:25:42.720 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:25:26.867146+0000", 2024-09-10T19:25:42.721 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.8473e-05 2024-09-10T19:25:42.721 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:25:42.721 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:25:42.721 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-10T19:25:42.721 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:25:26.867164+0000", 2024-09-10T19:25:42.721 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.8275999999999999e-05 2024-09-10T19:25:42.721 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:25:42.721 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:25:42.721 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-10T19:25:42.721 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:25:26.867171+0000", 2024-09-10T19:25:42.721 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.7469999999999999e-06 2024-09-10T19:25:42.721 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:25:42.722 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:25:42.722 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-10T19:25:42.722 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:25:26.867225+0000", 2024-09-10T19:25:42.722 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.4243000000000002e-05 2024-09-10T19:25:42.722 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:25:42.722 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:25:42.722 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-10T19:25:42.722 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:25:26.867364+0000", 2024-09-10T19:25:42.722 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00013904400000000001 2024-09-10T19:25:42.722 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:25:42.723 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:25:42.723 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 0,4", 2024-09-10T19:25:42.723 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:25:26.867524+0000", 2024-09-10T19:25:42.723 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00015987400000000001 2024-09-10T19:25:42.723 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:25:42.723 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:25:42.723 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-10T19:25:42.723 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:25:26.870009+0000", 2024-09-10T19:25:42.723 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0024847710000000002 2024-09-10T19:25:42.723 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:25:42.723 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:25:42.723 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:25:42.724 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:25:26.872707+0000", 2024-09-10T19:25:42.724 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0026987650000000001 2024-09-10T19:25:42.724 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:25:42.724 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:25:42.724 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:25:42.724 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:25:26.916651+0000", 2024-09-10T19:25:42.724 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.043943307000000001 2024-09-10T19:25:42.724 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:25:42.724 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:25:42.724 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-10T19:25:42.724 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:25:26.916944+0000", 2024-09-10T19:25:42.724 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00029348200000000002 2024-09-10T19:25:42.724 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:25:42.725 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:25:42.725 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-10T19:25:42.725 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:25:26.916989+0000", 2024-09-10T19:25:42.725 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.4604999999999997e-05 2024-09-10T19:25:42.725 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:25:42.725 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:25:42.725 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:25:42.725 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:25:42.725 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:25:42.725 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.25622.0:102416 10.2 10:41e82bf9:::benchmark_data_smithi183_176537_object102415:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e334)", 2024-09-10T19:25:42.725 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-10T19:25:34.203544+0000", 2024-09-10T19:25:42.725 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 8.434970216, 2024-09-10T19:25:42.726 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.053507496000000002, 2024-09-10T19:25:42.726 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-10T19:25:42.726 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T19:25:42.726 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-10T19:25:42.726 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.25622", 2024-09-10T19:25:42.726 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.183:0/3685189199", 2024-09-10T19:25:42.726 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 102416 2024-09-10T19:25:42.726 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:25:42.726 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-10T19:25:42.726 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:25:42.726 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-10T19:25:42.726 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:25:34.203544+0000", 2024-09-10T19:25:42.727 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:25:42.727 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:25:42.727 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:25:42.727 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-10T19:25:42.727 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:25:34.203544+0000", 2024-09-10T19:25:42.727 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:25:42.727 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:25:42.727 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:25:42.727 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-10T19:25:42.727 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:25:34.203543+0000", 2024-09-10T19:25:42.727 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4294967295.9999986 2024-09-10T19:25:42.727 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:25:42.727 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:25:42.728 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-10T19:25:42.728 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:25:34.203567+0000", 2024-09-10T19:25:42.728 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.4430000000000002e-05 2024-09-10T19:25:42.728 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:25:42.728 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:25:42.728 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-10T19:25:42.728 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:25:34.203584+0000", 2024-09-10T19:25:42.728 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.6038000000000001e-05 2024-09-10T19:25:42.728 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:25:42.728 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:25:42.728 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-10T19:25:42.728 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:25:34.203590+0000", 2024-09-10T19:25:42.729 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.776e-06 2024-09-10T19:25:42.729 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:25:42.729 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:25:42.729 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-10T19:25:42.729 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:25:34.203634+0000", 2024-09-10T19:25:42.729 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.4168000000000001e-05 2024-09-10T19:25:42.729 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:25:42.729 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:25:42.729 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-10T19:25:42.729 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:25:34.203761+0000", 2024-09-10T19:25:42.729 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00012672000000000001 2024-09-10T19:25:42.729 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:25:42.730 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:25:42.730 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 0,4", 2024-09-10T19:25:42.730 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:25:34.203899+0000", 2024-09-10T19:25:42.730 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00013750899999999999 2024-09-10T19:25:42.730 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:25:42.730 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:25:42.730 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-10T19:25:42.730 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:25:34.206502+0000", 2024-09-10T19:25:42.730 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0026029149999999999 2024-09-10T19:25:42.730 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:25:42.730 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:25:42.730 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:25:42.730 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:25:34.206897+0000", 2024-09-10T19:25:42.730 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00039512500000000002 2024-09-10T19:25:42.730 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:25:42.731 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:25:42.731 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:25:42.731 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:25:34.256955+0000", 2024-09-10T19:25:42.731 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.050057959999999999 2024-09-10T19:25:42.731 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:25:42.731 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:25:42.731 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-10T19:25:42.731 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:25:34.257008+0000", 2024-09-10T19:25:42.731 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.3671999999999997e-05 2024-09-10T19:25:42.731 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:25:42.731 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:25:42.731 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-10T19:25:42.731 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:25:34.257052+0000", 2024-09-10T19:25:42.732 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.3513999999999997e-05 2024-09-10T19:25:42.732 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:25:42.732 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:25:42.732 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:25:42.732 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:25:42.732 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:25:42.732 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.25622.0:102425 10.2 10:48fb2aa4:::benchmark_data_smithi183_176537_object102424:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e334)", 2024-09-10T19:25:42.732 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-10T19:25:34.214106+0000", 2024-09-10T19:25:42.732 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 8.4244081919999996, 2024-09-10T19:25:42.732 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.051308227999999997, 2024-09-10T19:25:42.732 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-10T19:25:42.732 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T19:25:42.733 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-10T19:25:42.733 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.25622", 2024-09-10T19:25:42.733 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.183:0/3685189199", 2024-09-10T19:25:42.733 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 102425 2024-09-10T19:25:42.733 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:25:42.733 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-10T19:25:42.733 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:25:42.733 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-10T19:25:42.733 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:25:34.214106+0000", 2024-09-10T19:25:42.733 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:25:42.733 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:25:42.733 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:25:42.733 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-10T19:25:42.734 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:25:34.214106+0000", 2024-09-10T19:25:42.734 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:25:42.734 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:25:42.734 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:25:42.734 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-10T19:25:42.734 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:25:34.214105+0000", 2024-09-10T19:25:42.734 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4294967295.9999986 2024-09-10T19:25:42.734 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:25:42.734 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:25:42.734 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-10T19:25:42.734 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:25:34.214127+0000", 2024-09-10T19:25:42.734 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.1552999999999998e-05 2024-09-10T19:25:42.734 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:25:42.734 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:25:42.735 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-10T19:25:42.735 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:25:34.214139+0000", 2024-09-10T19:25:42.735 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.2064e-05 2024-09-10T19:25:42.735 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:25:42.735 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:25:42.735 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-10T19:25:42.735 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:25:34.214144+0000", 2024-09-10T19:25:42.735 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.8189999999999999e-06 2024-09-10T19:25:42.735 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:25:42.735 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:25:42.735 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-10T19:25:42.735 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:25:34.214180+0000", 2024-09-10T19:25:42.735 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.5979999999999998e-05 2024-09-10T19:25:42.736 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:25:42.736 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:25:42.736 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-10T19:25:42.736 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:25:34.214286+0000", 2024-09-10T19:25:42.736 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00010632999999999999 2024-09-10T19:25:42.736 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:25:42.736 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:25:42.736 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 0,4", 2024-09-10T19:25:42.736 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:25:34.214387+0000", 2024-09-10T19:25:42.736 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000101517 2024-09-10T19:25:42.736 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:25:42.736 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:25:42.737 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-10T19:25:42.737 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:25:34.216254+0000", 2024-09-10T19:25:42.737 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0018666290000000001 2024-09-10T19:25:42.737 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:25:42.737 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:25:42.737 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:25:42.737 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:25:34.216500+0000", 2024-09-10T19:25:42.737 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00024594999999999999 2024-09-10T19:25:42.737 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:25:42.737 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:25:42.737 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:25:42.737 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:25:34.265320+0000", 2024-09-10T19:25:42.737 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.048819512000000002 2024-09-10T19:25:42.737 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:25:42.738 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:25:42.738 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-10T19:25:42.738 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:25:34.265371+0000", 2024-09-10T19:25:42.738 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.1900000000000001e-05 2024-09-10T19:25:42.738 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:25:42.738 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:25:42.738 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-10T19:25:42.738 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:25:34.265415+0000", 2024-09-10T19:25:42.738 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.3235999999999997e-05 2024-09-10T19:25:42.738 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:25:42.738 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:25:42.738 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:25:42.739 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:25:42.739 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:25:42.739 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.25622.0:103323 10.2 10:4d3dfff8:::benchmark_data_smithi183_176537_object103322:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e335)", 2024-09-10T19:25:42.739 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-10T19:25:34.962493+0000", 2024-09-10T19:25:42.739 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 7.6760216840000002, 2024-09-10T19:25:42.739 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.040309338, 2024-09-10T19:25:42.739 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-10T19:25:42.739 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T19:25:42.739 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-10T19:25:42.739 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.25622", 2024-09-10T19:25:42.739 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.183:0/3685189199", 2024-09-10T19:25:42.740 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 103323 2024-09-10T19:25:42.740 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:25:42.740 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-10T19:25:42.740 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:25:42.740 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-10T19:25:42.740 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:25:34.962493+0000", 2024-09-10T19:25:42.740 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:25:42.740 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:25:42.740 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:25:42.740 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-10T19:25:42.740 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:25:34.962493+0000", 2024-09-10T19:25:42.740 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:25:42.740 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:25:42.740 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:25:42.741 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-10T19:25:42.741 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:25:34.962491+0000", 2024-09-10T19:25:42.741 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4294967295.9999986 2024-09-10T19:25:42.741 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:25:42.741 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:25:42.741 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-10T19:25:42.741 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:25:34.962513+0000", 2024-09-10T19:25:42.741 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.1793e-05 2024-09-10T19:25:42.741 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:25:42.741 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:25:42.741 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-10T19:25:42.741 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:25:34.962524+0000", 2024-09-10T19:25:42.741 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.1102e-05 2024-09-10T19:25:42.742 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:25:42.742 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:25:42.742 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-10T19:25:42.742 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:25:34.962529+0000", 2024-09-10T19:25:42.742 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.3680000000000004e-06 2024-09-10T19:25:42.742 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:25:42.742 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:25:42.742 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-10T19:25:42.742 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:25:34.962570+0000", 2024-09-10T19:25:42.742 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.1072999999999998e-05 2024-09-10T19:25:42.742 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:25:42.742 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:25:42.742 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-10T19:25:42.742 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:25:34.962706+0000", 2024-09-10T19:25:42.743 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00013618199999999999 2024-09-10T19:25:42.743 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:25:42.743 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:25:42.743 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 0,4", 2024-09-10T19:25:42.743 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:25:34.962880+0000", 2024-09-10T19:25:42.743 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000173733 2024-09-10T19:25:42.743 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:25:42.743 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:25:42.743 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-10T19:25:42.743 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:25:34.965977+0000", 2024-09-10T19:25:42.743 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0030972420000000001 2024-09-10T19:25:42.743 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:25:42.743 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:25:42.743 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:25:42.744 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:25:34.966498+0000", 2024-09-10T19:25:42.744 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00052136299999999995 2024-09-10T19:25:42.744 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:25:42.744 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:25:42.744 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:25:42.744 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:25:35.002696+0000", 2024-09-10T19:25:42.744 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.036197499000000001 2024-09-10T19:25:42.744 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:25:42.744 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:25:42.744 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-10T19:25:42.744 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:25:35.002757+0000", 2024-09-10T19:25:42.744 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.1371000000000006e-05 2024-09-10T19:25:42.744 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:25:42.745 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:25:42.745 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-10T19:25:42.745 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:25:35.002802+0000", 2024-09-10T19:25:42.745 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.5102000000000003e-05 2024-09-10T19:25:42.745 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:25:42.745 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:25:42.745 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:25:42.745 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:25:42.745 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:25:42.745 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.25622.0:103581 10.2 10:4ad26711:::benchmark_data_smithi183_176537_object103580:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e335)", 2024-09-10T19:25:42.745 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-10T19:25:35.186694+0000", 2024-09-10T19:25:42.745 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 7.4518205420000001, 2024-09-10T19:25:42.745 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.084840092000000006, 2024-09-10T19:25:42.746 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-10T19:25:42.746 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T19:25:42.746 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-10T19:25:42.746 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.25622", 2024-09-10T19:25:42.746 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.183:0/3685189199", 2024-09-10T19:25:42.746 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 103581 2024-09-10T19:25:42.746 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:25:42.746 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-10T19:25:42.746 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:25:42.746 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-10T19:25:42.746 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:25:35.186694+0000", 2024-09-10T19:25:42.746 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:25:42.746 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:25:42.747 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:25:42.747 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-10T19:25:42.747 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:25:35.186694+0000", 2024-09-10T19:25:42.747 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:25:42.747 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:25:42.747 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:25:42.747 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-10T19:25:42.747 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:25:35.186692+0000", 2024-09-10T19:25:42.747 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4294967295.9999981 2024-09-10T19:25:42.747 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:25:42.747 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:25:42.747 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-10T19:25:42.747 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:25:35.186716+0000", 2024-09-10T19:25:42.748 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.3974000000000001e-05 2024-09-10T19:25:42.748 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:25:42.748 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:25:42.748 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-10T19:25:42.748 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:25:35.186729+0000", 2024-09-10T19:25:42.748 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.2404e-05 2024-09-10T19:25:42.748 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:25:42.748 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:25:42.748 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-10T19:25:42.748 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:25:35.186734+0000", 2024-09-10T19:25:42.748 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.7119999999999997e-06 2024-09-10T19:25:42.748 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:25:42.748 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:25:42.749 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-10T19:25:42.749 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:25:35.186781+0000", 2024-09-10T19:25:42.749 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.6962000000000002e-05 2024-09-10T19:25:42.749 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:25:42.749 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:25:42.749 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-10T19:25:42.749 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:25:35.186878+0000", 2024-09-10T19:25:42.749 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.6929999999999995e-05 2024-09-10T19:25:42.749 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:25:42.749 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:25:42.749 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 0,4", 2024-09-10T19:25:42.749 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:25:35.186978+0000", 2024-09-10T19:25:42.749 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.9217999999999998e-05 2024-09-10T19:25:42.749 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:25:42.750 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:25:42.750 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-10T19:25:42.750 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:25:35.189990+0000", 2024-09-10T19:25:42.750 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0030127349999999999 2024-09-10T19:25:42.750 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:25:42.750 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:25:42.750 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:25:42.750 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:25:35.194124+0000", 2024-09-10T19:25:42.750 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0041334939999999997 2024-09-10T19:25:42.750 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:25:42.750 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:25:42.750 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:25:42.750 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:25:35.271448+0000", 2024-09-10T19:25:42.750 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.077324534 2024-09-10T19:25:42.750 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:25:42.751 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:25:42.751 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-10T19:25:42.751 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:25:35.271500+0000", 2024-09-10T19:25:42.751 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.1638999999999998e-05 2024-09-10T19:25:42.751 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:25:42.751 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:25:42.751 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-10T19:25:42.751 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:25:35.271534+0000", 2024-09-10T19:25:42.751 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.4159999999999998e-05 2024-09-10T19:25:42.751 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:25:42.751 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:25:42.751 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:25:42.751 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:25:42.752 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:25:42.752 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.25622.0:103582 10.2 10:40ee5ea4:::benchmark_data_smithi183_176537_object103581:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e335)", 2024-09-10T19:25:42.752 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-10T19:25:35.187396+0000", 2024-09-10T19:25:42.752 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 7.4511186250000003, 2024-09-10T19:25:42.752 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.084313805000000006, 2024-09-10T19:25:42.752 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-10T19:25:42.752 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T19:25:42.752 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-10T19:25:42.752 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.25622", 2024-09-10T19:25:42.752 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.183:0/3685189199", 2024-09-10T19:25:42.752 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 103582 2024-09-10T19:25:42.753 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:25:42.753 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-10T19:25:42.753 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:25:42.753 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-10T19:25:42.753 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:25:35.187396+0000", 2024-09-10T19:25:42.753 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:25:42.753 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:25:42.753 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:25:42.753 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-10T19:25:42.753 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:25:35.187396+0000", 2024-09-10T19:25:42.753 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:25:42.753 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:25:42.753 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:25:42.754 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-10T19:25:42.754 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:25:35.187395+0000", 2024-09-10T19:25:42.754 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4294967295.9999995 2024-09-10T19:25:42.754 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:25:42.754 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:25:42.754 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-10T19:25:42.754 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:25:35.187417+0000", 2024-09-10T19:25:42.754 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.128e-05 2024-09-10T19:25:42.754 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:25:42.754 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:25:42.754 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-10T19:25:42.754 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:25:35.187426+0000", 2024-09-10T19:25:42.754 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.329e-06 2024-09-10T19:25:42.755 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:25:42.755 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:25:42.755 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-10T19:25:42.755 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:25:35.187431+0000", 2024-09-10T19:25:42.755 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.1189999999999996e-06 2024-09-10T19:25:42.755 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:25:42.755 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:25:42.755 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-10T19:25:42.755 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:25:35.188665+0000", 2024-09-10T19:25:42.755 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0012342799999999999 2024-09-10T19:25:42.755 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:25:42.755 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:25:42.755 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-10T19:25:42.756 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:25:35.188739+0000", 2024-09-10T19:25:42.756 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.3805000000000002e-05 2024-09-10T19:25:42.756 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:25:42.756 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:25:42.756 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 0,4", 2024-09-10T19:25:42.756 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:25:35.188825+0000", 2024-09-10T19:25:42.756 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.6178000000000001e-05 2024-09-10T19:25:42.756 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:25:42.756 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:25:42.756 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-10T19:25:42.756 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:25:35.190138+0000", 2024-09-10T19:25:42.756 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0013131309999999999 2024-09-10T19:25:42.756 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:25:42.757 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:25:42.757 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:25:42.757 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:25:35.194154+0000", 2024-09-10T19:25:42.757 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0040156499999999999 2024-09-10T19:25:42.757 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:25:42.757 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:25:42.757 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:25:42.757 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:25:35.271572+0000", 2024-09-10T19:25:42.757 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.077417764 2024-09-10T19:25:42.757 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:25:42.757 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:25:42.757 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-10T19:25:42.757 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:25:35.271676+0000", 2024-09-10T19:25:42.757 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000103996 2024-09-10T19:25:42.758 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:25:42.758 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:25:42.758 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-10T19:25:42.758 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:25:35.271710+0000", 2024-09-10T19:25:42.758 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.3949999999999999e-05 2024-09-10T19:25:42.758 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:25:42.758 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:25:42.758 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:25:42.758 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:25:42.758 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:25:42.758 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.25622.0:103587 10.2 10:494dc3e8:::benchmark_data_smithi183_176537_object103586:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e335)", 2024-09-10T19:25:42.758 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-10T19:25:35.194361+0000", 2024-09-10T19:25:42.758 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 7.4441537259999997, 2024-09-10T19:25:42.759 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.077448487999999996, 2024-09-10T19:25:42.759 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-10T19:25:42.759 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T19:25:42.759 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-10T19:25:42.759 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.25622", 2024-09-10T19:25:42.759 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.183:0/3685189199", 2024-09-10T19:25:42.759 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 103587 2024-09-10T19:25:42.759 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:25:42.759 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-10T19:25:42.759 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:25:42.759 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-10T19:25:42.759 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:25:35.194361+0000", 2024-09-10T19:25:42.759 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:25:42.759 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:25:42.760 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:25:42.760 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-10T19:25:42.760 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:25:35.194361+0000", 2024-09-10T19:25:42.760 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:25:42.760 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:25:42.760 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:25:42.760 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-10T19:25:42.760 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:25:35.194360+0000", 2024-09-10T19:25:42.760 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4294967295.9999986 2024-09-10T19:25:42.760 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:25:42.760 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:25:42.760 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-10T19:25:42.760 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:25:35.194382+0000", 2024-09-10T19:25:42.761 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.2130999999999999e-05 2024-09-10T19:25:42.761 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:25:42.761 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:25:42.761 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-10T19:25:42.761 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:25:35.194391+0000", 2024-09-10T19:25:42.761 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.8009999999999998e-06 2024-09-10T19:25:42.761 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:25:42.761 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:25:42.761 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-10T19:25:42.761 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:25:35.194396+0000", 2024-09-10T19:25:42.761 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.7639999999999996e-06 2024-09-10T19:25:42.761 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:25:42.761 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:25:42.762 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-10T19:25:42.762 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:25:35.194422+0000", 2024-09-10T19:25:42.762 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.6026000000000001e-05 2024-09-10T19:25:42.762 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:25:42.762 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:25:42.762 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-10T19:25:42.762 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:25:35.194509+0000", 2024-09-10T19:25:42.762 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.6537000000000002e-05 2024-09-10T19:25:42.762 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:25:42.762 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:25:42.762 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 0,4", 2024-09-10T19:25:42.762 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:25:35.194622+0000", 2024-09-10T19:25:42.762 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000113324 2024-09-10T19:25:42.762 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:25:42.763 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:25:42.763 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-10T19:25:42.763 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:25:35.196193+0000", 2024-09-10T19:25:42.763 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0015705820000000001 2024-09-10T19:25:42.763 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:25:42.763 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:25:42.763 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:25:42.763 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:25:35.197960+0000", 2024-09-10T19:25:42.763 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0017671519999999999 2024-09-10T19:25:42.763 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:25:42.763 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:25:42.763 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:25:42.763 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:25:35.271746+0000", 2024-09-10T19:25:42.764 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.073786518999999995 2024-09-10T19:25:42.764 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:25:42.764 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:25:42.764 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-10T19:25:42.764 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:25:35.271782+0000", 2024-09-10T19:25:42.764 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.5454999999999999e-05 2024-09-10T19:25:42.764 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:25:42.764 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:25:42.764 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-10T19:25:42.764 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:25:35.271809+0000", 2024-09-10T19:25:42.764 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.7538e-05 2024-09-10T19:25:42.764 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:25:42.764 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:25:42.764 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:25:42.765 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:25:42.765 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:25:42.765 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.25622.0:103600 10.2 10:47587c5a:::benchmark_data_smithi183_176537_object103599:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e335)", 2024-09-10T19:25:42.765 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-10T19:25:35.208553+0000", 2024-09-10T19:25:42.765 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 7.4299620060000002, 2024-09-10T19:25:42.765 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.063349866000000005, 2024-09-10T19:25:42.765 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-10T19:25:42.765 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T19:25:42.765 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-10T19:25:42.765 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.25622", 2024-09-10T19:25:42.765 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.183:0/3685189199", 2024-09-10T19:25:42.765 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 103600 2024-09-10T19:25:42.765 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:25:42.766 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-10T19:25:42.766 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:25:42.766 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-10T19:25:42.766 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:25:35.208553+0000", 2024-09-10T19:25:42.766 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:25:42.766 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:25:42.766 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:25:42.766 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-10T19:25:42.766 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:25:35.208553+0000", 2024-09-10T19:25:42.766 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:25:42.766 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:25:42.766 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:25:42.766 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-10T19:25:42.766 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:25:35.208552+0000", 2024-09-10T19:25:42.767 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4294967295.999999 2024-09-10T19:25:42.767 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:25:42.767 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:25:42.767 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-10T19:25:42.767 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:25:35.208570+0000", 2024-09-10T19:25:42.767 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.8783e-05 2024-09-10T19:25:42.767 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:25:42.767 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:25:42.767 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-10T19:25:42.767 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:25:35.208579+0000", 2024-09-10T19:25:42.767 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.6230000000000003e-06 2024-09-10T19:25:42.767 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:25:42.767 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:25:42.767 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-10T19:25:42.768 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:25:35.208583+0000", 2024-09-10T19:25:42.768 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.6370000000000001e-06 2024-09-10T19:25:42.768 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:25:42.768 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:25:42.768 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-10T19:25:42.768 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:25:35.208631+0000", 2024-09-10T19:25:42.768 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.8384999999999998e-05 2024-09-10T19:25:42.768 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:25:42.768 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:25:42.768 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-10T19:25:42.768 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:25:35.208726+0000", 2024-09-10T19:25:42.768 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.4500000000000007e-05 2024-09-10T19:25:42.768 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:25:42.768 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:25:42.769 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 0,4", 2024-09-10T19:25:42.769 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:25:35.208816+0000", 2024-09-10T19:25:42.769 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.0326000000000002e-05 2024-09-10T19:25:42.769 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:25:42.769 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:25:42.769 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-10T19:25:42.769 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:25:35.210557+0000", 2024-09-10T19:25:42.769 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.001740983 2024-09-10T19:25:42.769 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:25:42.769 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:25:42.769 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:25:42.769 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:25:35.211771+0000", 2024-09-10T19:25:42.769 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0012137199999999999 2024-09-10T19:25:42.769 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:25:42.770 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:25:42.770 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:25:42.770 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:25:35.271842+0000", 2024-09-10T19:25:42.770 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.060071167000000002 2024-09-10T19:25:42.770 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:25:42.770 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:25:42.770 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-10T19:25:42.770 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:25:35.271876+0000", 2024-09-10T19:25:42.770 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.3977999999999999e-05 2024-09-10T19:25:42.770 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:25:42.770 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:25:42.770 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-10T19:25:42.770 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:25:35.271902+0000", 2024-09-10T19:25:42.771 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.6791000000000001e-05 2024-09-10T19:25:42.771 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:25:42.771 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:25:42.771 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:25:42.771 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:25:42.771 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:25:42.771 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.25622.0:105052 10.2 10:40a42238:::benchmark_data_smithi183_176537_object105051:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e336)", 2024-09-10T19:25:42.771 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-10T19:25:36.527006+0000", 2024-09-10T19:25:42.771 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 6.1115083549999998, 2024-09-10T19:25:42.771 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.058512954999999998, 2024-09-10T19:25:42.771 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-10T19:25:42.771 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T19:25:42.771 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-10T19:25:42.772 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.25622", 2024-09-10T19:25:42.772 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.183:0/3685189199", 2024-09-10T19:25:42.772 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 105052 2024-09-10T19:25:42.772 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:25:42.772 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-10T19:25:42.772 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:25:42.772 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-10T19:25:42.772 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:25:36.527006+0000", 2024-09-10T19:25:42.772 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:25:42.772 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:25:42.772 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:25:42.772 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-10T19:25:42.772 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:25:36.527006+0000", 2024-09-10T19:25:42.772 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:25:42.773 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:25:42.773 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:25:42.773 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-10T19:25:42.773 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:25:36.527005+0000", 2024-09-10T19:25:42.773 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4294967295.9999986 2024-09-10T19:25:42.773 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:25:42.773 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:25:42.773 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-10T19:25:42.773 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:25:36.527025+0000", 2024-09-10T19:25:42.773 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.9775999999999998e-05 2024-09-10T19:25:42.773 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:25:42.773 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:25:42.773 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-10T19:25:42.774 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:25:36.527034+0000", 2024-09-10T19:25:42.774 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.5049999999999996e-06 2024-09-10T19:25:42.774 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:25:42.774 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:25:42.774 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-10T19:25:42.774 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:25:36.527038+0000", 2024-09-10T19:25:42.774 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.07e-06 2024-09-10T19:25:42.774 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:25:42.774 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:25:42.774 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-10T19:25:42.774 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:25:36.527074+0000", 2024-09-10T19:25:42.774 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.5546999999999997e-05 2024-09-10T19:25:42.774 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:25:42.775 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:25:42.775 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-10T19:25:42.775 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:25:36.527166+0000", 2024-09-10T19:25:42.775 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.2058000000000005e-05 2024-09-10T19:25:42.775 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:25:42.775 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:25:42.775 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 0,4", 2024-09-10T19:25:42.775 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:25:36.527248+0000", 2024-09-10T19:25:42.775 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.1798000000000005e-05 2024-09-10T19:25:42.775 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:25:42.775 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:25:42.775 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-10T19:25:42.776 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:25:36.529241+0000", 2024-09-10T19:25:42.776 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.001993369 2024-09-10T19:25:42.776 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:25:42.776 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:25:42.776 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:25:42.776 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:25:36.531126+0000", 2024-09-10T19:25:42.776 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.001884482 2024-09-10T19:25:42.776 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:25:42.776 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:25:42.776 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:25:42.776 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:25:36.585279+0000", 2024-09-10T19:25:42.776 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.054153933000000001 2024-09-10T19:25:42.776 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:25:42.777 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:25:42.777 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-10T19:25:42.777 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:25:36.585474+0000", 2024-09-10T19:25:42.777 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00019414399999999999 2024-09-10T19:25:42.777 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:25:42.777 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:25:42.777 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-10T19:25:42.777 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:25:36.585519+0000", 2024-09-10T19:25:42.777 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.5589e-05 2024-09-10T19:25:42.777 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:25:42.777 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:25:42.777 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:25:42.777 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:25:42.777 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:25:42.778 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.25622.0:106063 10.2 10:4f92400a:::benchmark_data_smithi183_176537_object106062:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e336)", 2024-09-10T19:25:42.778 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-10T19:25:37.350305+0000", 2024-09-10T19:25:42.778 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 5.2882100789999997, 2024-09-10T19:25:42.778 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.050092276999999998, 2024-09-10T19:25:42.778 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-10T19:25:42.778 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T19:25:42.778 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-10T19:25:42.778 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.25622", 2024-09-10T19:25:42.778 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.183:0/3685189199", 2024-09-10T19:25:42.778 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 106063 2024-09-10T19:25:42.778 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:25:42.778 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-10T19:25:42.778 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:25:42.778 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-10T19:25:42.779 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:25:37.350305+0000", 2024-09-10T19:25:42.779 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:25:42.779 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:25:42.779 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:25:42.779 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-10T19:25:42.779 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:25:37.350305+0000", 2024-09-10T19:25:42.779 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:25:42.779 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:25:42.779 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:25:42.779 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-10T19:25:42.779 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:25:37.350303+0000", 2024-09-10T19:25:42.779 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4294967295.999999 2024-09-10T19:25:42.779 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:25:42.779 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:25:42.780 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-10T19:25:42.780 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:25:37.350327+0000", 2024-09-10T19:25:42.780 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.3283000000000001e-05 2024-09-10T19:25:42.780 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:25:42.780 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:25:42.780 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-10T19:25:42.780 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:25:37.350336+0000", 2024-09-10T19:25:42.780 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.0280000000000006e-06 2024-09-10T19:25:42.780 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:25:42.780 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:25:42.780 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-10T19:25:42.780 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:25:37.350340+0000", 2024-09-10T19:25:42.781 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.2640000000000002e-06 2024-09-10T19:25:42.781 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:25:42.781 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:25:42.781 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-10T19:25:42.781 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:25:37.350381+0000", 2024-09-10T19:25:42.781 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.1124000000000001e-05 2024-09-10T19:25:42.781 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:25:42.781 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:25:42.781 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-10T19:25:42.781 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:25:37.350495+0000", 2024-09-10T19:25:42.781 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000114121 2024-09-10T19:25:42.782 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:25:42.782 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:25:42.782 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 0,4", 2024-09-10T19:25:42.782 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:25:37.350581+0000", 2024-09-10T19:25:42.782 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.5836999999999998e-05 2024-09-10T19:25:42.782 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:25:42.782 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:25:42.782 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-10T19:25:42.782 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:25:37.352568+0000", 2024-09-10T19:25:42.782 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0019865059999999999 2024-09-10T19:25:42.782 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:25:42.782 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:25:42.783 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:25:42.783 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:25:37.354802+0000", 2024-09-10T19:25:42.783 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0022341869999999999 2024-09-10T19:25:42.783 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:25:42.783 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:25:42.783 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:25:42.783 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:25:37.400310+0000", 2024-09-10T19:25:42.783 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.045508316 2024-09-10T19:25:42.783 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:25:42.783 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:25:42.783 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-10T19:25:42.783 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:25:37.400358+0000", 2024-09-10T19:25:42.783 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.8053000000000002e-05 2024-09-10T19:25:42.783 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:25:42.784 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:25:42.784 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-10T19:25:42.784 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:25:37.400397+0000", 2024-09-10T19:25:42.784 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.8708000000000003e-05 2024-09-10T19:25:42.784 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:25:42.784 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:25:42.784 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:25:42.784 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:25:42.784 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:25:42.784 INFO:teuthology.orchestra.run.smithi183.stdout:} 2024-09-10T19:25:42.807 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:25:42.843 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:25:42.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:25:42.844 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:25:42.861 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: 82 16 113012 112996 86.1155 81.3125 0.00765649 0.0116079 2024-09-10T19:25:42.945 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:25:42.989 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:25:42.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:25:42.990 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:25:43.091 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:25:43.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:25:43.134 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:25:43.134 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:25:43.235 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:25:43.262 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:25:43.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:25:43.263 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:25:43.364 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:25:43.400 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:25:43.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:25:43.401 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:25:43.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:25:43 smithi183 ceph-mon[29736]: pgmap v716: 158 pgs: 158 active+clean; 6.6 GiB data, 27 GiB used, 598 GiB / 626 GiB avail; 732 B/s rd, 95 MiB/s wr, 1.51k op/s 2024-09-10T19:25:43.431 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:25:43 smithi183 ceph-mon[41562]: pgmap v716: 158 pgs: 158 active+clean; 6.6 GiB data, 27 GiB used, 598 GiB / 626 GiB avail; 732 B/s rd, 95 MiB/s wr, 1.51k op/s 2024-09-10T19:25:43.465 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:25:43 smithi191 ceph-mon[32884]: pgmap v716: 158 pgs: 158 active+clean; 6.6 GiB data, 27 GiB used, 598 GiB / 626 GiB avail; 732 B/s rd, 95 MiB/s wr, 1.51k op/s 2024-09-10T19:25:43.502 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:25:43.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:25:43.543 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:25:43.543 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:25:43.645 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:25:43.679 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:25:43.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:25:43.680 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:25:43.781 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:25:43.808 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:25:43.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:25:43.809 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:25:43.861 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: 83 16 114315 114299 86.0591 81.4375 0.0133614 0.0116156 2024-09-10T19:25:43.911 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:25:43.947 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:25:43.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:25:43.948 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:25:44.049 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:25:44.079 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph daemon osd.0 dump_ops_in_flight 2024-09-10T19:25:44.087 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:25:44.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:25:44.088 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:25:44.189 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:25:44.222 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:25:44.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:25:44.223 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:25:44.323 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:25:44.354 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:25:44.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:25:44.355 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:25:44.455 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:25:44.482 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:25:44.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:25:44.483 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:25:44.583 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:25:44.618 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:25:44.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:25:44.619 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:25:44.720 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:25:44.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:25:44.759 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:25:44.759 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:25:44.860 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:25:44.861 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: 84 16 115597 115581 85.9884 80.125 0.0180486 0.0116255 2024-09-10T19:25:44.892 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:25:44.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:25:44.893 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:25:44.931 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:25:44 smithi183 ceph-mon[29736]: from='mgr.14556 ' entity='mgr.y' 2024-09-10T19:25:44.932 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:25:44 smithi183 ceph-mon[29736]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:25:44.932 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:25:44 smithi183 ceph-mon[29736]: pgmap v717: 158 pgs: 158 active+clean; 6.7 GiB data, 28 GiB used, 598 GiB / 626 GiB avail; 1.2 KiB/s rd, 59 MiB/s wr, 943 op/s 2024-09-10T19:25:44.932 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:25:44 smithi183 ceph-mon[41562]: from='mgr.14556 ' entity='mgr.y' 2024-09-10T19:25:44.932 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:25:44 smithi183 ceph-mon[41562]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:25:44.933 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:25:44 smithi183 ceph-mon[41562]: pgmap v717: 158 pgs: 158 active+clean; 6.7 GiB data, 28 GiB used, 598 GiB / 626 GiB avail; 1.2 KiB/s rd, 59 MiB/s wr, 943 op/s 2024-09-10T19:25:44.965 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:25:44 smithi191 ceph-mon[32884]: from='mgr.14556 ' entity='mgr.y' 2024-09-10T19:25:44.966 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:25:44 smithi191 ceph-mon[32884]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:25:44.966 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:25:44 smithi191 ceph-mon[32884]: pgmap v717: 158 pgs: 158 active+clean; 6.7 GiB data, 28 GiB used, 598 GiB / 626 GiB avail; 1.2 KiB/s rd, 59 MiB/s wr, 943 op/s 2024-09-10T19:25:44.994 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:25:45.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:25:45.032 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:25:45.032 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:25:45.133 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:25:45.177 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:25:45.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:25:45.179 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:25:45.279 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:25:45.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:25:45.315 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:25:45.316 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:25:45.416 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:25:45.449 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:25:45.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:25:45.450 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:25:45.551 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:25:45.592 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:25:45.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:25:45.593 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:25:45.694 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:25:45.732 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:25:45.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:25:45.733 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:25:45.834 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:25:45.861 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: 85 16 116809 116793 85.8678 75.75 0.0103012 0.0116418 2024-09-10T19:25:45.866 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:25:45.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:25:45.867 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:25:45.967 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:25:45.996 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:25:45.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:25:45.997 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:25:46.098 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:25:46.140 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:25:46.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:25:46.141 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:25:46.235 INFO:teuthology.orchestra.run.smithi183.stdout:osd.0: {} 2024-09-10T19:25:46.236 INFO:teuthology.orchestra.run.smithi183.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-10T19:25:46.241 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:25:46.260 INFO:teuthology.orchestra.run.smithi183.stdout:osd.1: {} 2024-09-10T19:25:46.260 INFO:teuthology.orchestra.run.smithi183.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-10T19:25:46.277 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:25:46.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:25:46.278 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:25:46.281 INFO:teuthology.orchestra.run.smithi183.stdout:osd.2: {} 2024-09-10T19:25:46.281 INFO:teuthology.orchestra.run.smithi183.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-10T19:25:46.302 INFO:teuthology.orchestra.run.smithi183.stdout:osd.3: {} 2024-09-10T19:25:46.302 INFO:teuthology.orchestra.run.smithi183.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-10T19:25:46.329 INFO:teuthology.orchestra.run.smithi183.stdout:osd.4: {} 2024-09-10T19:25:46.329 INFO:teuthology.orchestra.run.smithi183.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-10T19:25:46.359 INFO:teuthology.orchestra.run.smithi183.stdout:osd.5: {} 2024-09-10T19:25:46.359 INFO:teuthology.orchestra.run.smithi183.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-10T19:25:46.379 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:25:46.383 INFO:teuthology.orchestra.run.smithi183.stdout:osd.6: {} 2024-09-10T19:25:46.383 INFO:teuthology.orchestra.run.smithi183.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-10T19:25:46.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:25:46.412 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:25:46.413 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:25:46.415 INFO:teuthology.orchestra.run.smithi183.stdout:osd.7: {} 2024-09-10T19:25:46.415 INFO:teuthology.orchestra.run.smithi183.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-10T19:25:46.514 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:25:46.539 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:25:46.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:25:46.540 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:25:46.640 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:25:46.674 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:25:46.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:25:46.675 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:25:46.775 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:25:46.803 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:25:46.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:25:46.804 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:25:46.862 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: 86 16 118104 118088 85.8104 80.9375 0.00670113 0.0116494 2024-09-10T19:25:46.905 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:25:46.945 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:25:46.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:25:46.946 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:25:47.047 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:25:47.077 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:25:47.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:25:47.078 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:25:47.178 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:25:47.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:25:46 smithi183 ceph-mon[29736]: pgmap v718: 158 pgs: 158 active+clean; 7.0 GiB data, 28 GiB used, 597 GiB / 626 GiB avail; 1.0 KiB/s rd, 77 MiB/s wr, 1.24k op/s 2024-09-10T19:25:47.182 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:25:46 smithi183 ceph-mon[29736]: from='client.? 172.21.15.183:0/1192476927' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T19:25:47.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:25:46 smithi183 ceph-mon[41562]: pgmap v718: 158 pgs: 158 active+clean; 7.0 GiB data, 28 GiB used, 597 GiB / 626 GiB avail; 1.0 KiB/s rd, 77 MiB/s wr, 1.24k op/s 2024-09-10T19:25:47.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:25:46 smithi183 ceph-mon[41562]: from='client.? 172.21.15.183:0/1192476927' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T19:25:47.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:25:47.213 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:25:47.213 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:25:47.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:25:46 smithi191 ceph-mon[32884]: pgmap v718: 158 pgs: 158 active+clean; 7.0 GiB data, 28 GiB used, 597 GiB / 626 GiB avail; 1.0 KiB/s rd, 77 MiB/s wr, 1.24k op/s 2024-09-10T19:25:47.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:25:46 smithi191 ceph-mon[32884]: from='client.? 172.21.15.183:0/1192476927' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T19:25:47.314 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:25:47.341 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:25:47.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:25:47.342 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:25:47.443 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:25:47.468 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:25:47.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:25:47.469 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:25:47.570 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:25:47.600 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:25:47.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:25:47.601 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:25:47.701 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:25:47.715 INFO:journalctl@ceph.mgr.x.smithi191.stdout:Sep 10 19:25:47 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-x[34967]: ::ffff:172.21.15.191 - - [10/Sep/2024:19:25:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T19:25:47.735 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:25:47.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:25:47.736 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:25:47.837 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:25:47.861 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: 87 16 119394 119378 85.7508 80.625 0.0186833 0.0116575 2024-09-10T19:25:47.869 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:25:47.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:25:47.871 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:25:47.971 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:25:48.012 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:25:48.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:25:48.013 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:25:48.017 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:25:48.114 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:25:48.143 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:25:48.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:25:48.144 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:25:48.245 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:25:48.281 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:25:48.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:25:48.282 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:25:48.383 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:25:48.421 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:25:48.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:25:48.422 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:25:48.523 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:25:48.551 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:25:48.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:25:48.553 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:25:48.654 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:25:48.683 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:25:48.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:25:48.684 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:25:48.785 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:25:48.813 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:25:48.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:25:48.814 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:25:48.862 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: 88 16 120695 120679 85.7002 81.3125 0.00397741 0.0116645 2024-09-10T19:25:48.914 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:25:48.935 INFO:teuthology.orchestra.run.smithi183.stderr:nodeep-scrub is set 2024-09-10T19:25:48.947 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:25:48.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:25:48.948 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:25:49.049 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:25:49.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:25:49.087 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:25:49.087 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:25:49.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:25:48 smithi183 ceph-mon[29736]: pgmap v719: 158 pgs: 158 active+clean; 7.1 GiB data, 29 GiB used, 597 GiB / 626 GiB avail; 1023 B/s rd, 77 MiB/s wr, 1.23k op/s 2024-09-10T19:25:49.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:25:48 smithi183 ceph-mon[29736]: from='client.? 172.21.15.183:0/3773833454' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-10T19:25:49.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:25:48 smithi183 ceph-mon[29736]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-10T19:25:49.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:25:48 smithi183 ceph-mon[41562]: pgmap v719: 158 pgs: 158 active+clean; 7.1 GiB data, 29 GiB used, 597 GiB / 626 GiB avail; 1023 B/s rd, 77 MiB/s wr, 1.23k op/s 2024-09-10T19:25:49.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:25:48 smithi183 ceph-mon[41562]: from='client.? 172.21.15.183:0/3773833454' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-10T19:25:49.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:25:48 smithi183 ceph-mon[41562]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-10T19:25:49.188 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:25:49.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:25:48 smithi191 ceph-mon[32884]: pgmap v719: 158 pgs: 158 active+clean; 7.1 GiB data, 29 GiB used, 597 GiB / 626 GiB avail; 1023 B/s rd, 77 MiB/s wr, 1.23k op/s 2024-09-10T19:25:49.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:25:48 smithi191 ceph-mon[32884]: from='client.? 172.21.15.183:0/3773833454' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-10T19:25:49.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:25:48 smithi191 ceph-mon[32884]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-10T19:25:49.228 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:25:49.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:25:49.230 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:25:49.330 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:25:49.369 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:25:49.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:25:49.370 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:25:49.471 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:25:49.506 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:25:49.506 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-10T19:25:49.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:25:49.509 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:25:49.610 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:25:49.643 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:25:49.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:25:49.644 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:25:49.745 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:25:49.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:25:49.777 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:25:49.778 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:25:49.862 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: 89 16 121934 121918 85.6072 77.4375 0.0101036 0.0116772 2024-09-10T19:25:49.879 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:25:49.912 INFO:teuthology.orchestra.run.smithi183.stderr:reweighted osd.4 to 1 (10000) 2024-09-10T19:25:49.918 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:25:49.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:25:49.919 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:25:50.020 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:25:50.058 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:25:50.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:25:50.059 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:25:50.160 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:25:50.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:25:49 smithi183 ceph-mon[29736]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T19:25:50.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:25:49 smithi183 ceph-mon[29736]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-10T19:25:50.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:25:49 smithi183 ceph-mon[29736]: osdmap e338: 8 total, 8 up, 7 in 2024-09-10T19:25:50.182 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:25:49 smithi183 ceph-mon[29736]: purged_snaps scrub starts 2024-09-10T19:25:50.182 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:25:49 smithi183 ceph-mon[29736]: purged_snaps scrub ok 2024-09-10T19:25:50.182 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:25:49 smithi183 ceph-mon[29736]: from='client.? 172.21.15.183:0/2967078660' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-09-10T19:25:50.182 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:25:49 smithi183 ceph-mon[29736]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-09-10T19:25:50.182 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:25:49 smithi183 ceph-mon[29736]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-09-10T19:25:50.182 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:25:49 smithi183 ceph-mon[29736]: osdmap e339: 8 total, 8 up, 7 in 2024-09-10T19:25:50.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:25:49 smithi183 ceph-mon[41562]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T19:25:50.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:25:49 smithi183 ceph-mon[41562]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-10T19:25:50.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:25:49 smithi183 ceph-mon[41562]: osdmap e338: 8 total, 8 up, 7 in 2024-09-10T19:25:50.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:25:49 smithi183 ceph-mon[41562]: purged_snaps scrub starts 2024-09-10T19:25:50.183 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:25:49 smithi183 ceph-mon[41562]: purged_snaps scrub ok 2024-09-10T19:25:50.183 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:25:49 smithi183 ceph-mon[41562]: from='client.? 172.21.15.183:0/2967078660' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-09-10T19:25:50.183 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:25:49 smithi183 ceph-mon[41562]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-09-10T19:25:50.183 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:25:49 smithi183 ceph-mon[41562]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-09-10T19:25:50.183 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:25:49 smithi183 ceph-mon[41562]: osdmap e339: 8 total, 8 up, 7 in 2024-09-10T19:25:50.198 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:25:50.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:25:50.199 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:25:50.216 INFO:journalctl@ceph.iscsi.iscsi.a.smithi191.stdout:Sep 10 19:25:49 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-iscsi-iscsi-a[61816]: debug there is no tcmu-runner data available 2024-09-10T19:25:50.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:25:49 smithi191 ceph-mon[32884]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T19:25:50.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:25:49 smithi191 ceph-mon[32884]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-10T19:25:50.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:25:49 smithi191 ceph-mon[32884]: osdmap e338: 8 total, 8 up, 7 in 2024-09-10T19:25:50.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:25:49 smithi191 ceph-mon[32884]: purged_snaps scrub starts 2024-09-10T19:25:50.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:25:49 smithi191 ceph-mon[32884]: purged_snaps scrub ok 2024-09-10T19:25:50.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:25:49 smithi191 ceph-mon[32884]: from='client.? 172.21.15.183:0/2967078660' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-09-10T19:25:50.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:25:49 smithi191 ceph-mon[32884]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-09-10T19:25:50.217 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:25:49 smithi191 ceph-mon[32884]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-09-10T19:25:50.217 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:25:49 smithi191 ceph-mon[32884]: osdmap e339: 8 total, 8 up, 7 in 2024-09-10T19:25:50.300 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:25:50.342 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:25:50.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:25:50.343 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:25:50.443 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:25:50.472 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:25:50.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:25:50.473 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:25:50.574 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:25:50.599 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:25:50.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:25:50.600 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:25:50.700 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:25:50.737 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:25:50.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:25:50.738 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:25:50.839 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:25:50.862 INFO:tasks.radosbench.radosbench.0.smithi183.stdout:Total time run: 90.0083 2024-09-10T19:25:50.862 INFO:tasks.radosbench.radosbench.0.smithi183.stdout:Total writes made: 123205 2024-09-10T19:25:50.862 INFO:tasks.radosbench.radosbench.0.smithi183.stdout:Write size: 65536 2024-09-10T19:25:50.863 INFO:tasks.radosbench.radosbench.0.smithi183.stdout:Object size: 65536 2024-09-10T19:25:50.863 INFO:tasks.radosbench.radosbench.0.smithi183.stdout:Bandwidth (MB/sec): 85.5511 2024-09-10T19:25:50.863 INFO:tasks.radosbench.radosbench.0.smithi183.stdout:Stddev Bandwidth: 36.2266 2024-09-10T19:25:50.863 INFO:tasks.radosbench.radosbench.0.smithi183.stdout:Max bandwidth (MB/sec): 246.688 2024-09-10T19:25:50.863 INFO:tasks.radosbench.radosbench.0.smithi183.stdout:Min bandwidth (MB/sec): 0 2024-09-10T19:25:50.863 INFO:tasks.radosbench.radosbench.0.smithi183.stdout:Average IOPS: 1368 2024-09-10T19:25:50.863 INFO:tasks.radosbench.radosbench.0.smithi183.stdout:Stddev IOPS: 579.626 2024-09-10T19:25:50.863 INFO:tasks.radosbench.radosbench.0.smithi183.stdout:Max IOPS: 3947 2024-09-10T19:25:50.863 INFO:tasks.radosbench.radosbench.0.smithi183.stdout:Min IOPS: 0 2024-09-10T19:25:50.863 INFO:tasks.radosbench.radosbench.0.smithi183.stdout:Average Latency(s): 0.011685 2024-09-10T19:25:50.863 INFO:tasks.radosbench.radosbench.0.smithi183.stdout:Stddev Latency(s): 0.033164 2024-09-10T19:25:50.863 INFO:tasks.radosbench.radosbench.0.smithi183.stdout:Max latency(s): 2.06031 2024-09-10T19:25:50.864 INFO:tasks.radosbench.radosbench.0.smithi183.stdout:Min latency(s): 0.00110744 2024-09-10T19:25:50.866 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:25:50.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:25:50.868 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:25:50.869 INFO:tasks.radosbench.radosbench.0.smithi183.stdout:Cleaning up (deleting benchmark objects) 2024-09-10T19:25:50.968 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:25:51.005 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:25:51.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:25:51.007 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:25:51.108 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:25:51.140 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:25:51.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:25:51.141 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:25:51.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:25:50 smithi183 ceph-mon[29736]: from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:25:51.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:25:50 smithi183 ceph-mon[29736]: pgmap v721: 158 pgs: 158 active+clean; 7.2 GiB data, 29 GiB used, 597 GiB / 626 GiB avail; 1.0 KiB/s rd, 92 MiB/s wr, 1.47k op/s 2024-09-10T19:25:51.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:25:50 smithi183 ceph-mon[41562]: from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:25:51.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:25:50 smithi183 ceph-mon[41562]: pgmap v721: 158 pgs: 158 active+clean; 7.2 GiB data, 29 GiB used, 597 GiB / 626 GiB avail; 1.0 KiB/s rd, 92 MiB/s wr, 1.47k op/s 2024-09-10T19:25:51.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:25:50 smithi191 ceph-mon[32884]: from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:25:51.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:25:50 smithi191 ceph-mon[32884]: pgmap v721: 158 pgs: 158 active+clean; 7.2 GiB data, 29 GiB used, 597 GiB / 626 GiB avail; 1.0 KiB/s rd, 92 MiB/s wr, 1.47k op/s 2024-09-10T19:25:51.242 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:25:51.277 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:25:51.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:25:51.278 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:25:51.379 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:25:51.416 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:25:51.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:25:51.418 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:25:51.519 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:25:51.557 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:25:51.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:25:51.558 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:25:51.659 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:25:51.681 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Sep 10 19:25:51 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-y[29989]: ::ffff:172.21.15.191 - - [10/Sep/2024:19:25:51] "GET /metrics HTTP/1.1" 200 224790 "" "Prometheus/2.43.0" 2024-09-10T19:25:51.695 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:25:51.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:25:51.696 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:25:51.797 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:25:51.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:25:51.823 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:25:51.823 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:25:51.924 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:25:51.952 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:25:51.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:25:51.953 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:25:52.054 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:25:52.056 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph osd reweight 5 1 2024-09-10T19:25:52.094 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:25:52.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:25:52.096 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:25:52.196 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:25:52.226 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:25:52.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:25:52.227 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:25:52.328 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:25:52.357 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:25:52.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:25:52.358 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:25:52.370 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:25:52.458 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:25:52.486 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:25:52.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:25:52.487 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:25:52.588 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:25:52.627 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:25:52.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:25:52.628 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:25:52.729 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:25:52.761 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:25:52.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:25:52.762 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:25:52.849 INFO:teuthology.orchestra.run.smithi183.stdout:ERROR: (22) Invalid argument 2024-09-10T19:25:52.850 INFO:teuthology.orchestra.run.smithi183.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-10T19:25:52.863 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:25:52.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:25:52.900 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:25:52.900 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:25:52.962 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph osd unset noscrub 2024-09-10T19:25:52.966 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:25:52 smithi183 ceph-mon[29736]: pgmap v723: 158 pgs: 158 active+clean; 7.4 GiB data, 30 GiB used, 596 GiB / 626 GiB avail; 639 B/s rd, 86 MiB/s wr, 1.37k op/s 2024-09-10T19:25:52.966 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:25:52 smithi183 ceph-mon[29736]: purged_snaps scrub starts 2024-09-10T19:25:52.966 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:25:52 smithi183 ceph-mon[29736]: purged_snaps scrub ok 2024-09-10T19:25:52.966 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:25:52 smithi183 ceph-mon[29736]: osdmap e340: 8 total, 8 up, 7 in 2024-09-10T19:25:52.966 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:25:52 smithi183 ceph-mon[41562]: pgmap v723: 158 pgs: 158 active+clean; 7.4 GiB data, 30 GiB used, 596 GiB / 626 GiB avail; 639 B/s rd, 86 MiB/s wr, 1.37k op/s 2024-09-10T19:25:52.967 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:25:52 smithi183 ceph-mon[41562]: purged_snaps scrub starts 2024-09-10T19:25:52.967 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:25:52 smithi183 ceph-mon[41562]: purged_snaps scrub ok 2024-09-10T19:25:52.967 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:25:52 smithi183 ceph-mon[41562]: osdmap e340: 8 total, 8 up, 7 in 2024-09-10T19:25:53.000 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:25:53.038 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:25:53.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:25:53.039 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:25:53.140 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:25:53.174 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:25:53.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:25:53.175 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:25:53.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:25:52 smithi191 ceph-mon[32884]: pgmap v723: 158 pgs: 158 active+clean; 7.4 GiB data, 30 GiB used, 596 GiB / 626 GiB avail; 639 B/s rd, 86 MiB/s wr, 1.37k op/s 2024-09-10T19:25:53.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:25:52 smithi191 ceph-mon[32884]: purged_snaps scrub starts 2024-09-10T19:25:53.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:25:52 smithi191 ceph-mon[32884]: purged_snaps scrub ok 2024-09-10T19:25:53.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:25:52 smithi191 ceph-mon[32884]: osdmap e340: 8 total, 8 up, 7 in 2024-09-10T19:25:53.275 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:25:53.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:25:53.313 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:25:53.314 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:25:53.414 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:25:53.453 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:25:53.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:25:53.454 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:25:53.555 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:25:53.590 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:25:53.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:25:53.591 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:25:53.692 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:25:53.714 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph daemon osd.0 dump_blocked_ops 2024-09-10T19:25:53.724 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:25:53.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:25:53.725 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:25:53.826 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:25:53.859 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:25:53.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:25:53.861 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:25:53.962 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:25:54.005 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:25:54.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:25:54.006 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:25:54.107 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:25:54.133 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:25:54.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:25:54.134 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:25:54.235 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:25:54.263 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:25:54.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:25:54.265 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:25:54.366 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:25:54.405 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:25:54.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:25:54.406 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:25:54.507 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:25:54.546 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:25:54.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:25:54.548 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:25:54.649 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:25:54.679 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:25:54.679 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:25:54.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:25:54.681 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:25:54.782 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:25:54.805 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:25:54.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:25:54.807 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:25:54.907 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:25:54.951 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:25:54.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:25:54.952 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:25:55.053 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:25:55.079 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:25:55.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:25:55.080 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:25:55.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:25:54 smithi183 ceph-mon[29736]: pgmap v725: 158 pgs: 158 active+clean; 7.4 GiB data, 30 GiB used, 596 GiB / 626 GiB avail; 853 B/s rd, 62 MiB/s wr, 1.07k op/s 2024-09-10T19:25:55.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:25:54 smithi183 ceph-mon[29736]: osdmap e341: 8 total, 8 up, 7 in 2024-09-10T19:25:55.181 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:25:54 smithi183 ceph-mon[41562]: pgmap v725: 158 pgs: 158 active+clean; 7.4 GiB data, 30 GiB used, 596 GiB / 626 GiB avail; 853 B/s rd, 62 MiB/s wr, 1.07k op/s 2024-09-10T19:25:55.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:25:54 smithi183 ceph-mon[41562]: osdmap e341: 8 total, 8 up, 7 in 2024-09-10T19:25:55.182 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:25:55.209 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:25:55.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:25:55.210 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:25:55.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:25:54 smithi191 ceph-mon[32884]: pgmap v725: 158 pgs: 158 active+clean; 7.4 GiB data, 30 GiB used, 596 GiB / 626 GiB avail; 853 B/s rd, 62 MiB/s wr, 1.07k op/s 2024-09-10T19:25:55.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:25:54 smithi191 ceph-mon[32884]: osdmap e341: 8 total, 8 up, 7 in 2024-09-10T19:25:55.311 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:25:55.342 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:25:55.352 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:25:55.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:25:55.354 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:25:55.454 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:25:55.481 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:25:55.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:25:55.482 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:25:55.583 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:25:55.621 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:25:55.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:25:55.622 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:25:55.723 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:25:55.748 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:25:55.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:25:55.749 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:25:55.850 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:25:55.877 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:25:55.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:25:55.878 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:25:55.979 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:25:56.005 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:25:56.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:25:56.006 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:25:56.106 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:25:56.136 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:25:56.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:25:56.137 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:25:56.238 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:25:56.271 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:25:56.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:25:56.272 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:25:56.373 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:25:56.410 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:25:56.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:25:56.412 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:25:56.512 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:25:56.551 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:25:56.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:25:56.552 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:25:56.653 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:25:56.693 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:25:56.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:25:56.694 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:25:56.795 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:25:56.828 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:25:56.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:25:56.829 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:25:56.930 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:25:56.967 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:25:56.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:25:56.968 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:25:57.069 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:25:57.097 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:25:57.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:25:57.098 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:25:57.199 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:25:57.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:25:56 smithi191 ceph-mon[32884]: pgmap v727: 158 pgs: 158 active+clean; 7.2 GiB data, 29 GiB used, 597 GiB / 626 GiB avail; 1023 B/s rd, 46 MiB/s wr, 1.52k op/s 2024-09-10T19:25:57.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:25:56 smithi191 ceph-mon[32884]: from='client.? 172.21.15.183:0/1280933021' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T19:25:57.222 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:25:56 smithi183 ceph-mon[29736]: pgmap v727: 158 pgs: 158 active+clean; 7.2 GiB data, 29 GiB used, 597 GiB / 626 GiB avail; 1023 B/s rd, 46 MiB/s wr, 1.52k op/s 2024-09-10T19:25:57.222 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:25:56 smithi183 ceph-mon[29736]: from='client.? 172.21.15.183:0/1280933021' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T19:25:57.222 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:25:56 smithi183 ceph-mon[41562]: pgmap v727: 158 pgs: 158 active+clean; 7.2 GiB data, 29 GiB used, 597 GiB / 626 GiB avail; 1023 B/s rd, 46 MiB/s wr, 1.52k op/s 2024-09-10T19:25:57.223 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:25:56 smithi183 ceph-mon[41562]: from='client.? 172.21.15.183:0/1280933021' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T19:25:57.223 INFO:teuthology.orchestra.run.smithi183.stdout:osd.0: {} 2024-09-10T19:25:57.223 INFO:teuthology.orchestra.run.smithi183.stdout:osd.1: {} 2024-09-10T19:25:57.223 INFO:teuthology.orchestra.run.smithi183.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-10T19:25:57.223 INFO:teuthology.orchestra.run.smithi183.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-10T19:25:57.224 INFO:teuthology.orchestra.run.smithi183.stdout:osd.2: {} 2024-09-10T19:25:57.224 INFO:teuthology.orchestra.run.smithi183.stdout:osd.3: {} 2024-09-10T19:25:57.224 INFO:teuthology.orchestra.run.smithi183.stdout:osd.4: {} 2024-09-10T19:25:57.224 INFO:teuthology.orchestra.run.smithi183.stdout:osd.5: {} 2024-09-10T19:25:57.224 INFO:teuthology.orchestra.run.smithi183.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-10T19:25:57.224 INFO:teuthology.orchestra.run.smithi183.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-10T19:25:57.224 INFO:teuthology.orchestra.run.smithi183.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-10T19:25:57.225 INFO:teuthology.orchestra.run.smithi183.stdout:osd.6: {} 2024-09-10T19:25:57.225 INFO:teuthology.orchestra.run.smithi183.stdout:osd.7: {} 2024-09-10T19:25:57.225 INFO:teuthology.orchestra.run.smithi183.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-10T19:25:57.225 INFO:teuthology.orchestra.run.smithi183.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-10T19:25:57.225 INFO:teuthology.orchestra.run.smithi183.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-10T19:25:57.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:25:57.237 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:25:57.237 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:25:57.338 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:25:57.363 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:25:57.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:25:57.364 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:25:57.465 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:25:57.488 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:25:57.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:25:57.489 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:25:57.590 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:25:57.615 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:25:57.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:25:57.617 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:25:57.716 INFO:journalctl@ceph.mgr.x.smithi191.stdout:Sep 10 19:25:57 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-x[34967]: ::ffff:172.21.15.191 - - [10/Sep/2024:19:25:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T19:25:57.717 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:25:57.743 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:25:57.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:25:57.744 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:25:57.845 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:25:57.872 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:25:57.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:25:57.873 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:25:57.973 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:25:58.005 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:25:58.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:25:58.006 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:25:58.106 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:25:58.147 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:25:58.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:25:58.148 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:25:58.249 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:25:58.290 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:25:58.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:25:58.291 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:25:58.392 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:25:58.428 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:25:58.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:25:58.429 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:25:58.530 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:25:58.561 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:25:58.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:25:58.562 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:25:58.663 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:25:58.689 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:25:58.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:25:58.690 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:25:58.791 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:25:58.835 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:25:58.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:25:58.836 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:25:58.936 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:25:58.963 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:25:58.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:25:58.965 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:25:59.065 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:25:59.091 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:25:59.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:25:59.093 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:25:59.193 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:25:59.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:25:58 smithi191 ceph-mon[32884]: pgmap v728: 158 pgs: 158 active+clean; 7.1 GiB data, 29 GiB used, 597 GiB / 626 GiB avail; 1.4 KiB/s rd, 35 MiB/s wr, 1.35k op/s 2024-09-10T19:25:59.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:25:58 smithi191 ceph-mon[32884]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:25:59.222 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:25:59.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:25:59.223 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:25:59.324 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:25:59.360 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:25:59.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:25:59.361 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:25:59.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:25:58 smithi183 ceph-mon[29736]: pgmap v728: 158 pgs: 158 active+clean; 7.1 GiB data, 29 GiB used, 597 GiB / 626 GiB avail; 1.4 KiB/s rd, 35 MiB/s wr, 1.35k op/s 2024-09-10T19:25:59.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:25:58 smithi183 ceph-mon[29736]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:25:59.431 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:25:58 smithi183 ceph-mon[41562]: pgmap v728: 158 pgs: 158 active+clean; 7.1 GiB data, 29 GiB used, 597 GiB / 626 GiB avail; 1.4 KiB/s rd, 35 MiB/s wr, 1.35k op/s 2024-09-10T19:25:59.431 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:25:58 smithi183 ceph-mon[41562]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:25:59.462 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:25:59.489 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:25:59.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:25:59.490 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:25:59.591 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:25:59.625 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:25:59.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:25:59.626 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:25:59.727 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:25:59.759 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:25:59.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:25:59.760 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:25:59.861 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:25:59.886 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:25:59.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:25:59.887 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:25:59.988 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:26:00.012 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:26:00.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:26:00.013 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:26:00.114 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:26:00.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:26:00.153 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:26:00.153 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:26:00.215 INFO:journalctl@ceph.iscsi.iscsi.a.smithi191.stdout:Sep 10 19:25:59 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-iscsi-iscsi-a[61816]: debug there is no tcmu-runner data available 2024-09-10T19:26:00.254 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:26:00.282 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:26:00.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:26:00.283 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:26:00.383 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:26:00.423 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:26:00.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:26:00.424 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:26:00.510 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:26:00.524 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:26:00.557 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:26:00.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:26:00.558 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:26:00.659 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:26:00.691 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:26:00.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:26:00.692 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:26:00.793 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:26:00.826 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:26:00.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:26:00.827 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:26:00.928 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:26:00.969 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:26:00.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:26:00.970 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:26:01.018 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:26:00 smithi183 ceph-mon[29736]: from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:26:01.018 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:26:00 smithi183 ceph-mon[29736]: pgmap v729: 158 pgs: 158 active+clean; 7.0 GiB data, 29 GiB used, 597 GiB / 626 GiB avail; 1.4 KiB/s rd, 16 MiB/s wr, 1.36k op/s 2024-09-10T19:26:01.018 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:26:00 smithi183 ceph-mon[41562]: from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:26:01.018 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:26:00 smithi183 ceph-mon[41562]: pgmap v729: 158 pgs: 158 active+clean; 7.0 GiB data, 29 GiB used, 597 GiB / 626 GiB avail; 1.4 KiB/s rd, 16 MiB/s wr, 1.36k op/s 2024-09-10T19:26:01.070 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:26:01.110 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:26:01.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:26:01.111 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:26:01.212 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:26:01.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:26:00 smithi191 ceph-mon[32884]: from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:26:01.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:26:00 smithi191 ceph-mon[32884]: pgmap v729: 158 pgs: 158 active+clean; 7.0 GiB data, 29 GiB used, 597 GiB / 626 GiB avail; 1.4 KiB/s rd, 16 MiB/s wr, 1.36k op/s 2024-09-10T19:26:01.244 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:26:01.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:26:01.246 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:26:01.251 INFO:teuthology.orchestra.run.smithi183.stdout:{ 2024-09-10T19:26:01.251 INFO:teuthology.orchestra.run.smithi183.stdout: "ops": [], 2024-09-10T19:26:01.251 INFO:teuthology.orchestra.run.smithi183.stdout: "complaint_time": 30, 2024-09-10T19:26:01.251 INFO:teuthology.orchestra.run.smithi183.stdout: "num_blocked_ops": 0 2024-09-10T19:26:01.251 INFO:teuthology.orchestra.run.smithi183.stdout:} 2024-09-10T19:26:01.347 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:26:01.373 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:26:01.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:26:01.374 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:26:01.475 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:26:01.513 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:26:01.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:26:01.514 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:26:01.615 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:26:01.645 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:26:01.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:26:01.646 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:26:01.681 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Sep 10 19:26:01 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-y[29989]: ::ffff:172.21.15.191 - - [10/Sep/2024:19:26:01] "GET /metrics HTTP/1.1" 200 224797 "" "Prometheus/2.43.0" 2024-09-10T19:26:01.747 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:26:01.774 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:26:01.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:26:01.775 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:26:01.876 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:26:01.902 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:26:01.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:26:01.903 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:26:02.004 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:26:02.036 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:26:02.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:26:02.037 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:26:02.139 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:26:02.175 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:26:02.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:26:02.176 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:26:02.277 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:26:02.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:26:02.305 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:26:02.305 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:26:02.406 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:26:02.431 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:26:02.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:26:02.432 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:26:02.485 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph daemon osd.0 dump_historic_ops 2024-09-10T19:26:02.486 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-10T19:26:02.533 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:26:02.573 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:26:02.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:26:02.622 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:26:02.722 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:26:02.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:26:02.786 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:26:02.786 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:26:02.887 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:26:02.922 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:26:02.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:26:02.923 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:26:03.024 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:26:03.051 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:26:03.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:26:03.052 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:26:03.153 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:26:03.181 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:26:03.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:26:02 smithi183 ceph-mon[29736]: pgmap v730: 158 pgs: 158 active+clean; 6.8 GiB data, 28 GiB used, 598 GiB / 626 GiB avail; 1.1 KiB/s rd, 7.4 MiB/s wr, 1.22k op/s 2024-09-10T19:26:03.181 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:26:02 smithi183 ceph-mon[41562]: pgmap v730: 158 pgs: 158 active+clean; 6.8 GiB data, 28 GiB used, 598 GiB / 626 GiB avail; 1.1 KiB/s rd, 7.4 MiB/s wr, 1.22k op/s 2024-09-10T19:26:03.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:26:03.182 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:26:03.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:26:02 smithi191 ceph-mon[32884]: pgmap v730: 158 pgs: 158 active+clean; 6.8 GiB data, 28 GiB used, 598 GiB / 626 GiB avail; 1.1 KiB/s rd, 7.4 MiB/s wr, 1.22k op/s 2024-09-10T19:26:03.283 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:26:03.314 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:26:03.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:26:03.316 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:26:03.325 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:26:03.328 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:26:03.416 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:26:03.457 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:26:03.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:26:03.458 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:26:03.559 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:26:03.599 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:26:03.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:26:03.600 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:26:03.701 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:26:03.728 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:26:03.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:26:03.729 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:26:03.829 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:26:03.862 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:26:03.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:26:03.863 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:26:03.963 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:26:03.983 INFO:teuthology.orchestra.run.smithi183.stderr:reweighted osd.5 to 1 (10000) 2024-09-10T19:26:04.000 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:26:04.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:26:04.001 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:26:04.102 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:26:04.136 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:26:04.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:26:04.137 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:26:04.163 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:26:03 smithi183 ceph-mon[29736]: from='client.? 172.21.15.183:0/3780099086' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-09-10T19:26:04.163 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:26:03 smithi183 ceph-mon[29736]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-09-10T19:26:04.164 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:26:03 smithi183 ceph-mon[29736]: purged_snaps scrub starts 2024-09-10T19:26:04.164 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:26:03 smithi183 ceph-mon[29736]: purged_snaps scrub ok 2024-09-10T19:26:04.164 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:26:03 smithi183 ceph-mon[41562]: from='client.? 172.21.15.183:0/3780099086' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-09-10T19:26:04.164 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:26:03 smithi183 ceph-mon[41562]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-09-10T19:26:04.164 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:26:03 smithi183 ceph-mon[41562]: purged_snaps scrub starts 2024-09-10T19:26:04.164 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:26:03 smithi183 ceph-mon[41562]: purged_snaps scrub ok 2024-09-10T19:26:04.237 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:26:04.268 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:26:04.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:26:04.269 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:26:04.370 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:26:04.398 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:26:04.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:26:04.399 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:26:04.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:26:03 smithi191 ceph-mon[32884]: from='client.? 172.21.15.183:0/3780099086' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-09-10T19:26:04.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:26:03 smithi191 ceph-mon[32884]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-09-10T19:26:04.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:26:03 smithi191 ceph-mon[32884]: purged_snaps scrub starts 2024-09-10T19:26:04.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:26:03 smithi191 ceph-mon[32884]: purged_snaps scrub ok 2024-09-10T19:26:04.500 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:26:04.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:26:04.541 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:26:04.542 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:26:04.642 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:26:04.668 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:26:04.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:26:04.669 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:26:04.770 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:26:04.797 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:26:04.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:26:04.798 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:26:04.854 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph osd reweight 6 1 2024-09-10T19:26:04.899 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:26:04.945 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:26:04.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:26:04.946 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:26:05.047 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:26:05.079 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:26:05.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:26:05.080 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:26:05.181 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:26:05.182 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:26:04 smithi183 ceph-mon[29736]: pgmap v731: 158 pgs: 158 active+clean; 6.7 GiB data, 27 GiB used, 598 GiB / 626 GiB avail; 1.1 KiB/s rd, 5.5 MiB/s wr, 1.33k op/s 2024-09-10T19:26:05.182 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:26:04 smithi183 ceph-mon[29736]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-09-10T19:26:05.182 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:26:04 smithi183 ceph-mon[29736]: osdmap e342: 8 total, 8 up, 7 in 2024-09-10T19:26:05.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:26:04 smithi183 ceph-mon[41562]: pgmap v731: 158 pgs: 158 active+clean; 6.7 GiB data, 27 GiB used, 598 GiB / 626 GiB avail; 1.1 KiB/s rd, 5.5 MiB/s wr, 1.33k op/s 2024-09-10T19:26:05.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:26:04 smithi183 ceph-mon[41562]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-09-10T19:26:05.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:26:04 smithi183 ceph-mon[41562]: osdmap e342: 8 total, 8 up, 7 in 2024-09-10T19:26:05.211 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:26:05.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:26:05.212 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:26:05.313 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:26:05.348 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:26:05.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:26:05.350 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:26:05.450 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:26:05.465 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:26:04 smithi191 ceph-mon[32884]: pgmap v731: 158 pgs: 158 active+clean; 6.7 GiB data, 27 GiB used, 598 GiB / 626 GiB avail; 1.1 KiB/s rd, 5.5 MiB/s wr, 1.33k op/s 2024-09-10T19:26:05.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:26:04 smithi191 ceph-mon[32884]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-09-10T19:26:05.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:26:04 smithi191 ceph-mon[32884]: osdmap e342: 8 total, 8 up, 7 in 2024-09-10T19:26:05.489 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:26:05.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:26:05.490 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:26:05.591 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:26:05.629 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:26:05.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:26:05.630 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:26:05.731 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:26:05.768 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:26:05.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:26:05.769 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:26:05.870 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:26:05.902 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:26:05.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:26:05.903 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:26:06.004 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:26:06.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:26:06.035 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:26:06.035 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:26:06.136 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:26:06.165 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:26:06.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:26:06.166 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:26:06.267 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:26:06.305 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:26:06.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:26:06.306 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:26:06.407 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:26:06.437 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:26:06.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:26:06.438 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:26:06.539 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:26:06.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:26:06.578 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:26:06.578 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:26:06.679 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:26:06.710 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:26:06.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:26:06.711 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:26:06.811 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:26:06.850 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:26:06.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:26:06.851 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:26:06.951 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:26:06.991 INFO:teuthology.orchestra.run.smithi183.stderr:noscrub is unset 2024-09-10T19:26:06.992 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:26:06.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:26:06.994 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:26:07.094 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:26:07.134 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:26:07.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:26:07.135 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:26:07.182 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:26:06 smithi183 ceph-mon[29736]: pgmap v733: 158 pgs: 158 active+clean; 6.4 GiB data, 27 GiB used, 599 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.34k op/s 2024-09-10T19:26:07.182 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:26:06 smithi183 ceph-mon[29736]: from='client.? 172.21.15.183:0/593882806' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-10T19:26:07.182 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:26:06 smithi183 ceph-mon[29736]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-10T19:26:07.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:26:06 smithi183 ceph-mon[41562]: pgmap v733: 158 pgs: 158 active+clean; 6.4 GiB data, 27 GiB used, 599 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.34k op/s 2024-09-10T19:26:07.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:26:06 smithi183 ceph-mon[41562]: from='client.? 172.21.15.183:0/593882806' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-10T19:26:07.183 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:26:06 smithi183 ceph-mon[41562]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-10T19:26:07.236 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:26:07.274 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:26:07.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:26:07.275 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:26:07.376 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:26:07.405 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:26:07.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:26:07.406 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:26:07.450 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:26:06 smithi191 ceph-mon[32884]: pgmap v733: 158 pgs: 158 active+clean; 6.4 GiB data, 27 GiB used, 599 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.34k op/s 2024-09-10T19:26:07.451 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:26:06 smithi191 ceph-mon[32884]: from='client.? 172.21.15.183:0/593882806' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-10T19:26:07.451 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:26:06 smithi191 ceph-mon[32884]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-10T19:26:07.506 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:26:07.541 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:26:07.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:26:07.542 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:26:07.643 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:26:07.672 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:26:07.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:26:07.673 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:26:07.716 INFO:journalctl@ceph.mgr.x.smithi191.stdout:Sep 10 19:26:07 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-x[34967]: ::ffff:172.21.15.191 - - [10/Sep/2024:19:26:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T19:26:07.750 INFO:teuthology.orchestra.run.smithi183.stdout:{ 2024-09-10T19:26:07.750 INFO:teuthology.orchestra.run.smithi183.stdout: "size": 20, 2024-09-10T19:26:07.751 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 600, 2024-09-10T19:26:07.751 INFO:teuthology.orchestra.run.smithi183.stdout: "ops": [ 2024-09-10T19:26:07.751 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:26:07.751 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.25622.0:7383 10.4 10.f7f93b4 (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e301)", 2024-09-10T19:26:07.751 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-10T19:24:22.907252+0000", 2024-09-10T19:26:07.751 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 104.838844833, 2024-09-10T19:26:07.751 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.986354693, 2024-09-10T19:26:07.751 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-10T19:26:07.751 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "reached pg", 2024-09-10T19:26:07.751 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-10T19:26:07.751 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.25622", 2024-09-10T19:26:07.751 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.183:0/3685189199", 2024-09-10T19:26:07.752 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 7383 2024-09-10T19:26:07.752 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:26:07.752 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-10T19:26:07.752 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:26:07.752 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-10T19:26:07.752 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:24:22.907252+0000", 2024-09-10T19:26:07.752 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:26:07.752 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:26:07.752 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:26:07.752 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-10T19:26:07.752 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:24:22.907252+0000", 2024-09-10T19:26:07.752 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:26:07.753 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:26:07.753 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:26:07.753 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-10T19:26:07.753 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:24:22.907250+0000", 2024-09-10T19:26:07.753 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4294967295.9999986 2024-09-10T19:26:07.753 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:26:07.753 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:26:07.753 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-10T19:26:07.753 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:24:22.907308+0000", 2024-09-10T19:26:07.753 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.7673999999999997e-05 2024-09-10T19:26:07.753 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:26:07.753 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:26:07.753 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-10T19:26:07.753 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:24:22.907321+0000", 2024-09-10T19:26:07.754 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.2887e-05 2024-09-10T19:26:07.754 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:26:07.754 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:26:07.754 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-10T19:26:07.754 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:24:22.907328+0000", 2024-09-10T19:26:07.754 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.6640000000000001e-06 2024-09-10T19:26:07.754 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:26:07.754 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:26:07.754 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-10T19:26:07.754 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:24:22.907367+0000", 2024-09-10T19:26:07.754 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.9017999999999998e-05 2024-09-10T19:26:07.754 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:26:07.754 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:26:07.754 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for peered", 2024-09-10T19:26:07.755 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:24:22.907374+0000", 2024-09-10T19:26:07.755 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.0060000000000003e-06 2024-09-10T19:26:07.755 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:26:07.755 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:26:07.755 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-10T19:26:07.755 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:24:23.893599+0000", 2024-09-10T19:26:07.755 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.98622488699999999 2024-09-10T19:26:07.755 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:26:07.755 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:26:07.755 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-10T19:26:07.755 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:24:23.893607+0000", 2024-09-10T19:26:07.755 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.1939999999999998e-06 2024-09-10T19:26:07.755 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:26:07.756 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:26:07.756 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:26:07.756 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:26:07.756 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:26:07.756 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.25622.0:7387 10.4 10.6dd5c814 (undecoded) ondisk+retry+write+known_if_redirected+supports_pool_eio e301)", 2024-09-10T19:26:07.756 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-10T19:24:22.907340+0000", 2024-09-10T19:26:07.756 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 104.838756842, 2024-09-10T19:26:07.756 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.98629274300000003, 2024-09-10T19:26:07.756 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-10T19:26:07.756 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "reached pg", 2024-09-10T19:26:07.756 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-10T19:26:07.756 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.25622", 2024-09-10T19:26:07.757 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.183:0/3685189199", 2024-09-10T19:26:07.757 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 7387 2024-09-10T19:26:07.757 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:26:07.757 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-10T19:26:07.757 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:26:07.757 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-10T19:26:07.757 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:24:22.907340+0000", 2024-09-10T19:26:07.757 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:26:07.757 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:26:07.757 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:26:07.757 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-10T19:26:07.757 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:24:22.907340+0000", 2024-09-10T19:26:07.758 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:26:07.758 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:26:07.758 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:26:07.758 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-10T19:26:07.758 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:24:22.907340+0000", 2024-09-10T19:26:07.758 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4294967295.9999995 2024-09-10T19:26:07.758 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:26:07.758 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:26:07.758 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-10T19:26:07.758 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:24:22.907389+0000", 2024-09-10T19:26:07.758 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.9422000000000002e-05 2024-09-10T19:26:07.758 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:26:07.758 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:26:07.759 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-10T19:26:07.759 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:24:22.907396+0000", 2024-09-10T19:26:07.759 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.9059999999999996e-06 2024-09-10T19:26:07.759 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:26:07.759 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:26:07.759 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-10T19:26:07.759 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:24:22.907401+0000", 2024-09-10T19:26:07.759 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.8940000000000001e-06 2024-09-10T19:26:07.759 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:26:07.759 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:26:07.759 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-10T19:26:07.759 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:24:22.907697+0000", 2024-09-10T19:26:07.759 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000296101 2024-09-10T19:26:07.760 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:26:07.760 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:26:07.760 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for peered", 2024-09-10T19:26:07.760 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:24:22.907702+0000", 2024-09-10T19:26:07.760 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.5319999999999997e-06 2024-09-10T19:26:07.760 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:26:07.760 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:26:07.760 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-10T19:26:07.760 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:24:23.893627+0000", 2024-09-10T19:26:07.760 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.98592454699999998 2024-09-10T19:26:07.760 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:26:07.760 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:26:07.761 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-10T19:26:07.761 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:24:23.893633+0000", 2024-09-10T19:26:07.761 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.8289999999999996e-06 2024-09-10T19:26:07.761 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:26:07.761 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:26:07.761 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:26:07.761 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:26:07.761 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:26:07.761 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.25622.0:7397 10.8 10.363724e8 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e301)", 2024-09-10T19:26:07.761 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-10T19:24:22.907971+0000", 2024-09-10T19:26:07.761 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 104.83812579799999, 2024-09-10T19:26:07.761 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.977870405, 2024-09-10T19:26:07.762 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-10T19:26:07.762 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "reached pg", 2024-09-10T19:26:07.762 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-10T19:26:07.762 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.25622", 2024-09-10T19:26:07.762 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.183:0/3685189199", 2024-09-10T19:26:07.762 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 7397 2024-09-10T19:26:07.762 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:26:07.762 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-10T19:26:07.762 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:26:07.762 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-10T19:26:07.762 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:24:22.907971+0000", 2024-09-10T19:26:07.762 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:26:07.763 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:26:07.763 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:26:07.763 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-10T19:26:07.763 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:24:22.907971+0000", 2024-09-10T19:26:07.763 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:26:07.763 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:26:07.763 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:26:07.763 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-10T19:26:07.763 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:24:22.907970+0000", 2024-09-10T19:26:07.763 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4294967295.9999995 2024-09-10T19:26:07.763 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:26:07.763 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:26:07.763 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-10T19:26:07.764 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:24:22.908018+0000", 2024-09-10T19:26:07.764 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.7917000000000002e-05 2024-09-10T19:26:07.764 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:26:07.764 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:26:07.764 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-10T19:26:07.764 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:24:22.908029+0000", 2024-09-10T19:26:07.764 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.0467e-05 2024-09-10T19:26:07.764 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:26:07.764 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:26:07.764 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-10T19:26:07.764 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:24:22.908035+0000", 2024-09-10T19:26:07.764 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.6679999999999998e-06 2024-09-10T19:26:07.765 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:26:07.765 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:26:07.765 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-10T19:26:07.765 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:24:22.909515+0000", 2024-09-10T19:26:07.765 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0014800939999999999 2024-09-10T19:26:07.765 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:26:07.765 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:26:07.765 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for peered", 2024-09-10T19:26:07.765 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:24:22.909520+0000", 2024-09-10T19:26:07.765 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.6590000000000004e-06 2024-09-10T19:26:07.765 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:26:07.765 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:26:07.765 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-10T19:26:07.766 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:24:23.885833+0000", 2024-09-10T19:26:07.766 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.976313074 2024-09-10T19:26:07.766 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:26:07.766 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:26:07.766 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-10T19:26:07.766 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:24:23.885842+0000", 2024-09-10T19:26:07.766 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.1999999999999994e-06 2024-09-10T19:26:07.766 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:26:07.766 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:26:07.766 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:26:07.766 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:26:07.766 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:26:07.766 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.25622.0:7401 10.4 10.77620144 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e301)", 2024-09-10T19:26:07.766 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-10T19:24:22.912182+0000", 2024-09-10T19:26:07.767 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 104.83391463700001, 2024-09-10T19:26:07.767 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.98149294499999995, 2024-09-10T19:26:07.767 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-10T19:26:07.767 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "reached pg", 2024-09-10T19:26:07.767 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-10T19:26:07.767 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.25622", 2024-09-10T19:26:07.767 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.183:0/3685189199", 2024-09-10T19:26:07.767 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 7401 2024-09-10T19:26:07.767 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:26:07.767 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-10T19:26:07.767 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:26:07.767 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-10T19:26:07.767 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:24:22.912182+0000", 2024-09-10T19:26:07.767 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:26:07.767 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:26:07.768 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:26:07.768 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-10T19:26:07.768 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:24:22.912182+0000", 2024-09-10T19:26:07.768 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:26:07.768 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:26:07.768 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:26:07.768 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-10T19:26:07.768 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:24:22.912181+0000", 2024-09-10T19:26:07.768 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4294967295.9999986 2024-09-10T19:26:07.768 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:26:07.768 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:26:07.768 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-10T19:26:07.768 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:24:22.912235+0000", 2024-09-10T19:26:07.768 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.3838999999999997e-05 2024-09-10T19:26:07.769 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:26:07.769 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:26:07.769 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-10T19:26:07.769 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:24:22.912243+0000", 2024-09-10T19:26:07.769 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.5399999999999996e-06 2024-09-10T19:26:07.769 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:26:07.769 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:26:07.769 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-10T19:26:07.769 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:24:22.912250+0000", 2024-09-10T19:26:07.769 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.7129999999999999e-06 2024-09-10T19:26:07.769 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:26:07.770 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:26:07.770 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-10T19:26:07.770 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:24:22.912408+0000", 2024-09-10T19:26:07.770 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000158412 2024-09-10T19:26:07.770 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:26:07.770 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:26:07.770 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for peered", 2024-09-10T19:26:07.770 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:24:22.912416+0000", 2024-09-10T19:26:07.770 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.7160000000000003e-06 2024-09-10T19:26:07.770 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:26:07.770 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:26:07.770 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-10T19:26:07.771 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:24:23.893669+0000", 2024-09-10T19:26:07.771 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.98125279799999998 2024-09-10T19:26:07.771 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:26:07.771 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:26:07.771 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-10T19:26:07.771 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:24:23.893675+0000", 2024-09-10T19:26:07.771 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.4579999999999998e-06 2024-09-10T19:26:07.771 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:26:07.771 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:26:07.771 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:26:07.771 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:26:07.771 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:26:07.771 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.25622.0:7402 10.5 10.2131c465 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e301)", 2024-09-10T19:26:07.771 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-10T19:24:22.912261+0000", 2024-09-10T19:26:07.772 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 104.833835882, 2024-09-10T19:26:07.772 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.98468117899999996, 2024-09-10T19:26:07.772 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-10T19:26:07.772 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "reached pg", 2024-09-10T19:26:07.772 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-10T19:26:07.772 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.25622", 2024-09-10T19:26:07.772 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.183:0/3685189199", 2024-09-10T19:26:07.772 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 7402 2024-09-10T19:26:07.772 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:26:07.772 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-10T19:26:07.772 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:26:07.772 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-10T19:26:07.772 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:24:22.912261+0000", 2024-09-10T19:26:07.773 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:26:07.773 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:26:07.773 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:26:07.773 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-10T19:26:07.773 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:24:22.912261+0000", 2024-09-10T19:26:07.773 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:26:07.773 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:26:07.773 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:26:07.774 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-10T19:26:07.774 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:24:22.912260+0000", 2024-09-10T19:26:07.774 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4294967295.9999995 2024-09-10T19:26:07.774 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:26:07.774 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:26:07.774 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-10T19:26:07.774 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:24:22.912502+0000", 2024-09-10T19:26:07.774 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000241458 2024-09-10T19:26:07.774 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:26:07.774 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:26:07.774 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-10T19:26:07.774 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:24:22.912511+0000", 2024-09-10T19:26:07.774 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.0510000000000002e-06 2024-09-10T19:26:07.775 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:26:07.775 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:26:07.775 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-10T19:26:07.775 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:24:22.912517+0000", 2024-09-10T19:26:07.775 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.015e-06 2024-09-10T19:26:07.775 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:26:07.775 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:26:07.775 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-10T19:26:07.775 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:24:22.912546+0000", 2024-09-10T19:26:07.775 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.8824999999999999e-05 2024-09-10T19:26:07.775 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:26:07.775 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:26:07.775 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for peered", 2024-09-10T19:26:07.776 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:24:22.912550+0000", 2024-09-10T19:26:07.776 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.5560000000000001e-06 2024-09-10T19:26:07.776 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:26:07.776 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:26:07.776 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-10T19:26:07.776 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:24:23.896923+0000", 2024-09-10T19:26:07.776 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.984372739 2024-09-10T19:26:07.776 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:26:07.776 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:26:07.776 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-10T19:26:07.776 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:24:23.896942+0000", 2024-09-10T19:26:07.776 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.9097999999999999e-05 2024-09-10T19:26:07.777 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:26:07.777 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:26:07.777 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:26:07.777 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:26:07.777 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:26:07.777 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.25622.0:7403 10.8 10.da3107a8 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e301)", 2024-09-10T19:26:07.777 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-10T19:24:22.913249+0000", 2024-09-10T19:26:07.777 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 104.832847609, 2024-09-10T19:26:07.777 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.97262312500000003, 2024-09-10T19:26:07.777 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-10T19:26:07.777 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "reached pg", 2024-09-10T19:26:07.777 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-10T19:26:07.778 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.25622", 2024-09-10T19:26:07.778 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.183:0/3685189199", 2024-09-10T19:26:07.778 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 7403 2024-09-10T19:26:07.778 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:26:07.778 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-10T19:26:07.778 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:26:07.778 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-10T19:26:07.778 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:24:22.913249+0000", 2024-09-10T19:26:07.778 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:26:07.778 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:26:07.778 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:26:07.778 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-10T19:26:07.778 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:24:22.913249+0000", 2024-09-10T19:26:07.778 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:26:07.779 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:26:07.779 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:26:07.779 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-10T19:26:07.779 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:24:22.913248+0000", 2024-09-10T19:26:07.779 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4294967295.999999 2024-09-10T19:26:07.779 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:26:07.779 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:26:07.779 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-10T19:26:07.779 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:24:22.913299+0000", 2024-09-10T19:26:07.779 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.0844000000000003e-05 2024-09-10T19:26:07.779 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:26:07.779 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:26:07.780 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-10T19:26:07.780 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:24:22.913310+0000", 2024-09-10T19:26:07.780 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.1270000000000001e-05 2024-09-10T19:26:07.780 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:26:07.780 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:26:07.780 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-10T19:26:07.780 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:24:22.913316+0000", 2024-09-10T19:26:07.780 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.6890000000000001e-06 2024-09-10T19:26:07.780 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:26:07.780 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:26:07.780 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-10T19:26:07.780 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:24:22.913368+0000", 2024-09-10T19:26:07.780 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.2184e-05 2024-09-10T19:26:07.781 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:26:07.781 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:26:07.781 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for peered", 2024-09-10T19:26:07.781 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:24:22.913374+0000", 2024-09-10T19:26:07.781 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.7119999999999997e-06 2024-09-10T19:26:07.781 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:26:07.781 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:26:07.781 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-10T19:26:07.781 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:24:23.885866+0000", 2024-09-10T19:26:07.781 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.97249230600000003 2024-09-10T19:26:07.781 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:26:07.781 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:26:07.782 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-10T19:26:07.782 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:26:07.783 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:24:23.885872+0000", 2024-09-10T19:26:07.783 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.2709999999999998e-06 2024-09-10T19:26:07.783 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:26:07.783 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:26:07.783 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:26:07.783 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:26:07.783 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:26:07.783 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.25622.0:12634 10.5 10:aa6e2e99:::benchmark_data_smithi183_176537_object12633:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e307)", 2024-09-10T19:26:07.784 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-10T19:24:28.948588+0000", 2024-09-10T19:26:07.784 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 98.797508828000005, 2024-09-10T19:26:07.784 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.93226618900000002, 2024-09-10T19:26:07.784 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-10T19:26:07.784 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T19:26:07.784 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-10T19:26:07.784 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.25622", 2024-09-10T19:26:07.784 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.183:0/3685189199", 2024-09-10T19:26:07.784 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 12634 2024-09-10T19:26:07.784 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:26:07.784 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-10T19:26:07.784 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:26:07.784 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-10T19:26:07.785 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:24:28.948588+0000", 2024-09-10T19:26:07.785 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:26:07.785 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:26:07.785 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:26:07.785 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-10T19:26:07.785 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:24:28.948588+0000", 2024-09-10T19:26:07.785 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:26:07.785 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:26:07.785 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:26:07.785 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-10T19:26:07.785 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:24:28.948587+0000", 2024-09-10T19:26:07.785 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4294967295.9999986 2024-09-10T19:26:07.785 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:26:07.786 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:26:07.786 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-10T19:26:07.786 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:24:28.948606+0000", 2024-09-10T19:26:07.786 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.9497e-05 2024-09-10T19:26:07.786 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:26:07.786 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:26:07.786 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-10T19:26:07.786 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:24:28.948619+0000", 2024-09-10T19:26:07.786 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.2549e-05 2024-09-10T19:26:07.786 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:26:07.786 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:26:07.786 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-10T19:26:07.786 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:24:28.948625+0000", 2024-09-10T19:26:07.786 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.6930000000000001e-06 2024-09-10T19:26:07.786 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:26:07.787 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:26:07.787 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-10T19:26:07.787 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:24:28.948672+0000", 2024-09-10T19:26:07.787 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.7009999999999999e-05 2024-09-10T19:26:07.787 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:26:07.787 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:26:07.787 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for peered", 2024-09-10T19:26:07.787 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:24:28.948682+0000", 2024-09-10T19:26:07.787 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.7480000000000005e-06 2024-09-10T19:26:07.787 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:26:07.787 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:26:07.787 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-10T19:26:07.787 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:24:29.878002+0000", 2024-09-10T19:26:07.788 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.929319644 2024-09-10T19:26:07.788 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:26:07.788 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:26:07.788 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-10T19:26:07.788 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:24:29.878131+0000", 2024-09-10T19:26:07.788 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000129593 2024-09-10T19:26:07.788 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:26:07.788 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:26:07.788 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 3,7", 2024-09-10T19:26:07.788 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:24:29.878274+0000", 2024-09-10T19:26:07.788 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00014287499999999999 2024-09-10T19:26:07.788 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:26:07.788 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:26:07.788 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:26:07.788 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:24:29.880706+0000", 2024-09-10T19:26:07.789 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0024313709999999999 2024-09-10T19:26:07.789 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:26:07.789 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:26:07.789 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-10T19:26:07.789 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:24:29.880716+0000", 2024-09-10T19:26:07.789 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.0791999999999999e-05 2024-09-10T19:26:07.789 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:26:07.789 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:26:07.789 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:26:07.789 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:24:29.880775+0000", 2024-09-10T19:26:07.789 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.8226000000000001e-05 2024-09-10T19:26:07.789 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:26:07.789 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:26:07.789 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-10T19:26:07.790 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:24:29.880820+0000", 2024-09-10T19:26:07.790 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.5374999999999998e-05 2024-09-10T19:26:07.790 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:26:07.790 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:26:07.790 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-10T19:26:07.790 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:24:29.880854+0000", 2024-09-10T19:26:07.790 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.4292999999999998e-05 2024-09-10T19:26:07.790 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:26:07.790 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:26:07.790 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:26:07.790 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:26:07.790 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:26:07.790 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.25622.0:12647 10.8 10:1c39f374:::benchmark_data_smithi183_176537_object12646:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e307)", 2024-09-10T19:26:07.791 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-10T19:24:28.953435+0000", 2024-09-10T19:26:07.791 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 98.792662331000002, 2024-09-10T19:26:07.791 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.92933013600000003, 2024-09-10T19:26:07.791 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-10T19:26:07.791 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T19:26:07.791 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-10T19:26:07.791 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.25622", 2024-09-10T19:26:07.791 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.183:0/3685189199", 2024-09-10T19:26:07.791 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 12647 2024-09-10T19:26:07.791 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:26:07.791 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-10T19:26:07.791 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:26:07.791 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-10T19:26:07.791 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:24:28.953435+0000", 2024-09-10T19:26:07.792 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:26:07.792 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:26:07.792 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:26:07.792 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-10T19:26:07.792 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:24:28.953435+0000", 2024-09-10T19:26:07.792 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:26:07.792 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:26:07.792 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:26:07.792 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-10T19:26:07.792 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:24:28.953433+0000", 2024-09-10T19:26:07.792 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4294967295.9999981 2024-09-10T19:26:07.792 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:26:07.792 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:26:07.793 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-10T19:26:07.793 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:24:28.953455+0000", 2024-09-10T19:26:07.793 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.1696e-05 2024-09-10T19:26:07.793 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:26:07.793 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:26:07.793 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-10T19:26:07.793 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:24:28.953467+0000", 2024-09-10T19:26:07.793 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.2489e-05 2024-09-10T19:26:07.793 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:26:07.793 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:26:07.793 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-10T19:26:07.793 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:24:28.953471+0000", 2024-09-10T19:26:07.794 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.296e-06 2024-09-10T19:26:07.794 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:26:07.794 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:26:07.794 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-10T19:26:07.794 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:24:28.953526+0000", 2024-09-10T19:26:07.794 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.4459000000000002e-05 2024-09-10T19:26:07.794 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:26:07.794 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:26:07.794 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for peered", 2024-09-10T19:26:07.794 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:24:28.953535+0000", 2024-09-10T19:26:07.794 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.8759999999999994e-06 2024-09-10T19:26:07.794 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:26:07.794 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:26:07.794 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-10T19:26:07.795 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:24:29.878325+0000", 2024-09-10T19:26:07.795 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.92479075899999996 2024-09-10T19:26:07.795 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:26:07.795 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:26:07.795 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-10T19:26:07.795 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:24:29.878421+0000", 2024-09-10T19:26:07.795 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.5447999999999998e-05 2024-09-10T19:26:07.795 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:26:07.795 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:26:07.795 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 5,7", 2024-09-10T19:26:07.795 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:24:29.878539+0000", 2024-09-10T19:26:07.795 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000118536 2024-09-10T19:26:07.795 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:26:07.795 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:26:07.796 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-10T19:26:07.796 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:24:29.881168+0000", 2024-09-10T19:26:07.796 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0026286500000000002 2024-09-10T19:26:07.796 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:26:07.796 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:26:07.796 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:26:07.796 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:24:29.882634+0000", 2024-09-10T19:26:07.796 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0014661780000000001 2024-09-10T19:26:07.796 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:26:07.796 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:26:07.796 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:26:07.796 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:24:29.882685+0000", 2024-09-10T19:26:07.797 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.0655999999999997e-05 2024-09-10T19:26:07.797 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:26:07.797 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:26:07.797 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-10T19:26:07.797 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:24:29.882731+0000", 2024-09-10T19:26:07.797 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.5936000000000002e-05 2024-09-10T19:26:07.797 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:26:07.797 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:26:07.797 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-10T19:26:07.797 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:24:29.882765+0000", 2024-09-10T19:26:07.797 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.3853999999999998e-05 2024-09-10T19:26:07.797 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:26:07.798 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:26:07.798 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:26:07.798 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:26:07.798 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:26:07.798 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.25622.0:12650 10.8 10:16608410:::benchmark_data_smithi183_176537_object12649:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e307)", 2024-09-10T19:26:07.798 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-10T19:24:28.953623+0000", 2024-09-10T19:26:07.798 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 98.792473725999997, 2024-09-10T19:26:07.798 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.93028344799999996, 2024-09-10T19:26:07.798 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-10T19:26:07.798 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T19:26:07.798 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-10T19:26:07.798 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.25622", 2024-09-10T19:26:07.798 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.183:0/3685189199", 2024-09-10T19:26:07.799 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 12650 2024-09-10T19:26:07.799 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:26:07.799 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-10T19:26:07.799 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:26:07.799 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-10T19:26:07.799 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:24:28.953623+0000", 2024-09-10T19:26:07.799 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:26:07.799 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:26:07.799 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:26:07.799 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-10T19:26:07.799 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:24:28.953623+0000", 2024-09-10T19:26:07.799 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:26:07.799 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:26:07.800 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:26:07.800 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-10T19:26:07.800 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:24:28.953623+0000", 2024-09-10T19:26:07.800 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4294967295.9999995 2024-09-10T19:26:07.800 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:26:07.800 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:26:07.800 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-10T19:26:07.800 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:24:28.953639+0000", 2024-09-10T19:26:07.800 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.6889e-05 2024-09-10T19:26:07.800 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:26:07.800 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:26:07.800 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-10T19:26:07.800 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:24:28.953650+0000", 2024-09-10T19:26:07.801 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.0896999999999999e-05 2024-09-10T19:26:07.801 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:26:07.801 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:26:07.801 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-10T19:26:07.801 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:24:28.953655+0000", 2024-09-10T19:26:07.801 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.0300000000000001e-06 2024-09-10T19:26:07.801 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:26:07.801 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:26:07.801 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-10T19:26:07.801 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:24:28.953702+0000", 2024-09-10T19:26:07.801 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.6635000000000003e-05 2024-09-10T19:26:07.801 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:26:07.802 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:26:07.802 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for peered", 2024-09-10T19:26:07.802 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:24:28.953710+0000", 2024-09-10T19:26:07.802 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.6899999999999992e-06 2024-09-10T19:26:07.802 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:26:07.802 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:26:07.802 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-10T19:26:07.802 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:24:29.879124+0000", 2024-09-10T19:26:07.802 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.92541420600000002 2024-09-10T19:26:07.802 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:26:07.802 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:26:07.802 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-10T19:26:07.802 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:24:29.879219+0000", 2024-09-10T19:26:07.803 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.4981000000000004e-05 2024-09-10T19:26:07.803 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:26:07.803 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:26:07.803 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 5,7", 2024-09-10T19:26:07.803 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:24:29.879335+0000", 2024-09-10T19:26:07.803 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000115962 2024-09-10T19:26:07.803 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:26:07.803 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:26:07.803 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-10T19:26:07.803 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:24:29.881175+0000", 2024-09-10T19:26:07.803 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0018398360000000001 2024-09-10T19:26:07.803 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:26:07.803 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:26:07.804 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:26:07.804 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:24:29.883805+0000", 2024-09-10T19:26:07.804 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0026300490000000002 2024-09-10T19:26:07.804 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:26:07.804 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:26:07.804 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:26:07.804 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:24:29.883830+0000", 2024-09-10T19:26:07.804 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.5473000000000002e-05 2024-09-10T19:26:07.804 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:26:07.804 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:26:07.804 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-10T19:26:07.804 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:24:29.883875+0000", 2024-09-10T19:26:07.804 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.4607999999999997e-05 2024-09-10T19:26:07.805 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:26:07.805 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:26:07.805 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-10T19:26:07.805 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:24:29.883907+0000", 2024-09-10T19:26:07.805 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.1826000000000003e-05 2024-09-10T19:26:07.805 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:26:07.805 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:26:07.805 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:26:07.805 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:26:07.805 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:26:07.805 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.25622.0:12655 10.8 10:1c684d15:::benchmark_data_smithi183_176537_object12654:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e307)", 2024-09-10T19:26:07.805 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-10T19:24:28.954556+0000", 2024-09-10T19:26:07.805 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 98.791541421000005, 2024-09-10T19:26:07.806 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.93041465700000003, 2024-09-10T19:26:07.806 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-10T19:26:07.806 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T19:26:07.806 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-10T19:26:07.806 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.25622", 2024-09-10T19:26:07.806 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.183:0/3685189199", 2024-09-10T19:26:07.806 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 12655 2024-09-10T19:26:07.806 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:26:07.806 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-10T19:26:07.806 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:26:07.806 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-10T19:26:07.806 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:24:28.954556+0000", 2024-09-10T19:26:07.806 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:26:07.807 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:26:07.807 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:26:07.807 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-10T19:26:07.807 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:24:28.954556+0000", 2024-09-10T19:26:07.807 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:26:07.807 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:26:07.807 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:26:07.807 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-10T19:26:07.807 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:24:28.954554+0000", 2024-09-10T19:26:07.807 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4294967295.9999986 2024-09-10T19:26:07.807 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:26:07.807 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:26:07.808 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-10T19:26:07.808 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:24:28.954578+0000", 2024-09-10T19:26:07.808 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.3394e-05 2024-09-10T19:26:07.808 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:26:07.808 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:26:07.808 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-10T19:26:07.808 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:24:28.954598+0000", 2024-09-10T19:26:07.808 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.014e-05 2024-09-10T19:26:07.808 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:26:07.808 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:26:07.808 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-10T19:26:07.808 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:24:28.954603+0000", 2024-09-10T19:26:07.808 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.5820000000000001e-06 2024-09-10T19:26:07.809 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:26:07.809 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:26:07.809 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-10T19:26:07.809 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:24:28.954658+0000", 2024-09-10T19:26:07.809 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.5167000000000002e-05 2024-09-10T19:26:07.809 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:26:07.809 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:26:07.809 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for peered", 2024-09-10T19:26:07.809 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:24:28.954666+0000", 2024-09-10T19:26:07.809 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.8539999999999997e-06 2024-09-10T19:26:07.809 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:26:07.809 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:26:07.810 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-10T19:26:07.810 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:24:29.879901+0000", 2024-09-10T19:26:07.810 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.92523418800000001 2024-09-10T19:26:07.810 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:26:07.810 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:26:07.810 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-10T19:26:07.810 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:24:29.880014+0000", 2024-09-10T19:26:07.810 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000113089 2024-09-10T19:26:07.810 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:26:07.810 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:26:07.810 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 5,7", 2024-09-10T19:26:07.810 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:24:29.880143+0000", 2024-09-10T19:26:07.810 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000129822 2024-09-10T19:26:07.811 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:26:07.811 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:26:07.811 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-10T19:26:07.811 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:24:29.882578+0000", 2024-09-10T19:26:07.811 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0024347940000000001 2024-09-10T19:26:07.811 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:26:07.811 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:26:07.811 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:26:07.811 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:24:29.883971+0000", 2024-09-10T19:26:07.811 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0013926539999999999 2024-09-10T19:26:07.811 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:26:07.811 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:26:07.812 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:26:07.812 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:24:29.884891+0000", 2024-09-10T19:26:07.812 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00091967700000000004 2024-09-10T19:26:07.812 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:26:07.812 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:26:07.812 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-10T19:26:07.812 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:24:29.884929+0000", 2024-09-10T19:26:07.812 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.8559999999999997e-05 2024-09-10T19:26:07.812 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:26:07.812 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:26:07.812 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-10T19:26:07.812 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:24:29.884970+0000", 2024-09-10T19:26:07.812 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.1069000000000003e-05 2024-09-10T19:26:07.813 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:26:07.813 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:26:07.813 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:26:07.813 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:26:07.813 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:26:07.813 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.25622.0:12657 10.5 10:a24207de:::benchmark_data_smithi183_176537_object12656:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e307)", 2024-09-10T19:26:07.813 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-10T19:24:28.956163+0000", 2024-09-10T19:26:07.813 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 98.789933950999995, 2024-09-10T19:26:07.813 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.92621870699999997, 2024-09-10T19:26:07.813 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-10T19:26:07.813 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T19:26:07.813 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-10T19:26:07.813 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.25622", 2024-09-10T19:26:07.814 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.183:0/3685189199", 2024-09-10T19:26:07.814 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 12657 2024-09-10T19:26:07.814 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:26:07.814 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-10T19:26:07.814 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:26:07.814 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-10T19:26:07.814 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:24:28.956163+0000", 2024-09-10T19:26:07.814 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:26:07.814 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:26:07.814 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:26:07.814 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-10T19:26:07.814 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:24:28.956163+0000", 2024-09-10T19:26:07.814 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:26:07.814 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:26:07.814 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:26:07.815 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-10T19:26:07.815 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:24:28.956162+0000", 2024-09-10T19:26:07.815 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4294967295.9999986 2024-09-10T19:26:07.815 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:26:07.815 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:26:07.815 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-10T19:26:07.815 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:24:28.956181+0000", 2024-09-10T19:26:07.815 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.9301999999999999e-05 2024-09-10T19:26:07.815 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:26:07.815 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:26:07.815 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-10T19:26:07.815 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:24:28.956193+0000", 2024-09-10T19:26:07.815 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.1528999999999999e-05 2024-09-10T19:26:07.816 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:26:07.816 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:26:07.816 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-10T19:26:07.816 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:24:28.956198+0000", 2024-09-10T19:26:07.816 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.7919999999999998e-06 2024-09-10T19:26:07.816 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:26:07.816 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:26:07.816 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-10T19:26:07.816 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:24:28.956267+0000", 2024-09-10T19:26:07.816 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.8324000000000005e-05 2024-09-10T19:26:07.816 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:26:07.816 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:26:07.816 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for peered", 2024-09-10T19:26:07.817 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:24:28.956275+0000", 2024-09-10T19:26:07.817 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.8019999999999992e-06 2024-09-10T19:26:07.817 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:26:07.817 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:26:07.817 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-10T19:26:07.817 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:24:29.879170+0000", 2024-09-10T19:26:07.817 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.92289558299999996 2024-09-10T19:26:07.817 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:26:07.817 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:26:07.817 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-10T19:26:07.817 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:24:29.879276+0000", 2024-09-10T19:26:07.817 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000105388 2024-09-10T19:26:07.817 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:26:07.818 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:26:07.818 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 3,7", 2024-09-10T19:26:07.818 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:24:29.879393+0000", 2024-09-10T19:26:07.818 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000117619 2024-09-10T19:26:07.818 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:26:07.818 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:26:07.818 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-10T19:26:07.818 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:24:29.881087+0000", 2024-09-10T19:26:07.818 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.001694306 2024-09-10T19:26:07.818 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:26:07.818 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:26:07.818 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:26:07.819 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:24:29.881248+0000", 2024-09-10T19:26:07.819 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000160688 2024-09-10T19:26:07.819 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:26:07.819 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:26:07.819 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:26:07.819 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:24:29.882288+0000", 2024-09-10T19:26:07.819 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.001039732 2024-09-10T19:26:07.819 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:26:07.819 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:26:07.819 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-10T19:26:07.819 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:24:29.882340+0000", 2024-09-10T19:26:07.819 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.2179999999999998e-05 2024-09-10T19:26:07.819 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:26:07.819 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:26:07.819 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-10T19:26:07.820 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:24:29.882382+0000", 2024-09-10T19:26:07.820 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.1662000000000002e-05 2024-09-10T19:26:07.820 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:26:07.820 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:26:07.820 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:26:07.820 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:26:07.820 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:26:07.820 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.25622.0:12658 10.5 10:a20d8f3f:::benchmark_data_smithi183_176537_object12657:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e307)", 2024-09-10T19:26:07.820 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-10T19:24:28.956260+0000", 2024-09-10T19:26:07.820 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 98.789836972000003, 2024-09-10T19:26:07.820 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.92623411, 2024-09-10T19:26:07.820 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-10T19:26:07.820 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T19:26:07.821 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-10T19:26:07.821 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.25622", 2024-09-10T19:26:07.821 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.183:0/3685189199", 2024-09-10T19:26:07.821 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 12658 2024-09-10T19:26:07.821 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:26:07.821 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-10T19:26:07.821 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:26:07.821 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-10T19:26:07.821 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:24:28.956260+0000", 2024-09-10T19:26:07.821 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:26:07.821 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:26:07.821 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:26:07.821 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-10T19:26:07.822 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:24:28.956260+0000", 2024-09-10T19:26:07.822 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:26:07.822 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:26:07.822 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:26:07.822 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-10T19:26:07.822 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:24:28.956259+0000", 2024-09-10T19:26:07.822 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4294967295.999999 2024-09-10T19:26:07.822 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:26:07.822 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:26:07.822 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-10T19:26:07.822 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:24:28.956340+0000", 2024-09-10T19:26:07.822 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.0567000000000003e-05 2024-09-10T19:26:07.823 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:26:07.823 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:26:07.823 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-10T19:26:07.823 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:24:28.956352+0000", 2024-09-10T19:26:07.823 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.2128e-05 2024-09-10T19:26:07.823 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:26:07.823 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:26:07.823 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-10T19:26:07.823 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:24:28.956357+0000", 2024-09-10T19:26:07.823 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.4700000000000001e-06 2024-09-10T19:26:07.823 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:26:07.823 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:26:07.824 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-10T19:26:07.824 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:24:28.956400+0000", 2024-09-10T19:26:07.824 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.2388000000000001e-05 2024-09-10T19:26:07.824 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:26:07.824 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:26:07.824 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for peered", 2024-09-10T19:26:07.824 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:24:28.956407+0000", 2024-09-10T19:26:07.824 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.0419999999999997e-06 2024-09-10T19:26:07.824 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:26:07.824 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:26:07.824 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-10T19:26:07.824 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:24:29.879909+0000", 2024-09-10T19:26:07.824 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.92350201099999996 2024-09-10T19:26:07.825 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:26:07.825 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:26:07.825 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-10T19:26:07.825 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:24:29.880016+0000", 2024-09-10T19:26:07.825 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000107054 2024-09-10T19:26:07.825 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:26:07.825 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:26:07.825 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 3,7", 2024-09-10T19:26:07.825 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:24:29.880187+0000", 2024-09-10T19:26:07.825 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00017112599999999999 2024-09-10T19:26:07.825 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:26:07.825 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:26:07.826 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-10T19:26:07.826 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:24:29.881260+0000", 2024-09-10T19:26:07.826 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0010734329999999999 2024-09-10T19:26:07.826 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:26:07.826 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:26:07.826 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:26:07.826 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:24:29.881489+0000", 2024-09-10T19:26:07.827 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000228753 2024-09-10T19:26:07.828 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:26:07.828 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:26:07.828 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:26:07.828 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:24:29.882422+0000", 2024-09-10T19:26:07.828 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000932868 2024-09-10T19:26:07.828 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:26:07.828 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:26:07.828 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-10T19:26:07.828 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:24:29.882460+0000", 2024-09-10T19:26:07.828 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.7697000000000002e-05 2024-09-10T19:26:07.828 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:26:07.829 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:26:07.829 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-10T19:26:07.829 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:24:29.882494+0000", 2024-09-10T19:26:07.829 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.4289999999999999e-05 2024-09-10T19:26:07.829 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:26:07.829 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:26:07.829 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:26:07.829 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:26:07.829 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:26:07.829 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.25622.0:12684 10.4 10:2e399445:::benchmark_data_smithi183_176537_object12683:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e308)", 2024-09-10T19:26:07.829 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-10T19:24:29.882761+0000", 2024-09-10T19:26:07.829 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 97.863335837999998, 2024-09-10T19:26:07.829 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.005803829, 2024-09-10T19:26:07.830 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-10T19:26:07.830 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T19:26:07.830 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-10T19:26:07.830 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.25622", 2024-09-10T19:26:07.830 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.183:0/3685189199", 2024-09-10T19:26:07.830 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 12684 2024-09-10T19:26:07.830 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:26:07.830 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-10T19:26:07.830 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:26:07.830 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-10T19:26:07.830 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:24:29.882761+0000", 2024-09-10T19:26:07.830 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:26:07.831 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:26:07.831 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:26:07.831 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-10T19:26:07.831 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:24:29.882761+0000", 2024-09-10T19:26:07.831 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:26:07.831 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:26:07.831 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:26:07.831 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-10T19:26:07.831 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:24:29.882760+0000", 2024-09-10T19:26:07.831 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4294967295.9999986 2024-09-10T19:26:07.831 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:26:07.831 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:26:07.832 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-10T19:26:07.832 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:24:29.883462+0000", 2024-09-10T19:26:07.832 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00070262500000000002 2024-09-10T19:26:07.832 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:26:07.832 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:26:07.832 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-10T19:26:07.832 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:24:29.883478+0000", 2024-09-10T19:26:07.832 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.5625e-05 2024-09-10T19:26:07.832 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:26:07.832 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:26:07.832 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-10T19:26:07.832 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:24:29.883484+0000", 2024-09-10T19:26:07.832 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.0440000000000001e-06 2024-09-10T19:26:07.833 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:26:07.833 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:26:07.833 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-10T19:26:07.833 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:24:29.883667+0000", 2024-09-10T19:26:07.833 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00018281300000000001 2024-09-10T19:26:07.833 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:26:07.833 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:26:07.833 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for peered", 2024-09-10T19:26:07.833 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:24:29.883676+0000", 2024-09-10T19:26:07.833 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.7099999999999996e-06 2024-09-10T19:26:07.833 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:26:07.833 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:26:07.834 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-10T19:26:07.834 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:24:30.885672+0000", 2024-09-10T19:26:07.834 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.001996911 2024-09-10T19:26:07.834 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:26:07.834 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:26:07.834 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-10T19:26:07.834 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:24:30.885788+0000", 2024-09-10T19:26:07.834 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00011582900000000001 2024-09-10T19:26:07.834 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:26:07.834 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:26:07.834 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 3,4", 2024-09-10T19:26:07.834 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:24:30.885946+0000", 2024-09-10T19:26:07.834 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00015771900000000001 2024-09-10T19:26:07.835 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:26:07.835 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:26:07.835 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-10T19:26:07.835 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:24:30.887379+0000", 2024-09-10T19:26:07.835 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0014330460000000001 2024-09-10T19:26:07.835 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:26:07.835 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:26:07.835 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:26:07.835 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:24:30.888331+0000", 2024-09-10T19:26:07.835 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00095185400000000002 2024-09-10T19:26:07.835 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:26:07.835 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:26:07.835 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:26:07.835 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:24:30.888409+0000", 2024-09-10T19:26:07.835 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.8170000000000005e-05 2024-09-10T19:26:07.836 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:26:07.836 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:26:07.836 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-10T19:26:07.836 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:24:30.888517+0000", 2024-09-10T19:26:07.836 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000108372 2024-09-10T19:26:07.836 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:26:07.836 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:26:07.836 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-10T19:26:07.836 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:24:30.888565+0000", 2024-09-10T19:26:07.836 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.7478999999999997e-05 2024-09-10T19:26:07.836 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:26:07.836 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:26:07.836 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:26:07.836 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:26:07.837 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:26:07.837 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.25622.0:12697 10.4 10:27da895c:::benchmark_data_smithi183_176537_object12696:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e308)", 2024-09-10T19:26:07.837 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-10T19:24:29.887985+0000", 2024-09-10T19:26:07.837 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 97.858112259999999, 2024-09-10T19:26:07.837 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.0007626949999999, 2024-09-10T19:26:07.837 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-10T19:26:07.837 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T19:26:07.837 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-10T19:26:07.837 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.25622", 2024-09-10T19:26:07.837 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.183:0/3685189199", 2024-09-10T19:26:07.837 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 12697 2024-09-10T19:26:07.837 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:26:07.838 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-10T19:26:07.838 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:26:07.838 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-10T19:26:07.838 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:24:29.887985+0000", 2024-09-10T19:26:07.838 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:26:07.838 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:26:07.838 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:26:07.838 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-10T19:26:07.838 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:24:29.887985+0000", 2024-09-10T19:26:07.838 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:26:07.838 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:26:07.838 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:26:07.838 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-10T19:26:07.839 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:24:29.887983+0000", 2024-09-10T19:26:07.839 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4294967295.9999986 2024-09-10T19:26:07.839 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:26:07.839 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:26:07.839 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-10T19:26:07.839 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:24:29.888009+0000", 2024-09-10T19:26:07.839 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.5593e-05 2024-09-10T19:26:07.839 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:26:07.839 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:26:07.839 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-10T19:26:07.839 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:24:29.888019+0000", 2024-09-10T19:26:07.839 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.0064000000000001e-05 2024-09-10T19:26:07.840 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:26:07.840 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:26:07.840 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-10T19:26:07.840 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:24:29.888025+0000", 2024-09-10T19:26:07.840 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.7989999999999999e-06 2024-09-10T19:26:07.840 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:26:07.840 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:26:07.840 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-10T19:26:07.840 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:24:29.888091+0000", 2024-09-10T19:26:07.840 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.6730999999999995e-05 2024-09-10T19:26:07.840 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:26:07.840 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:26:07.840 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for peered", 2024-09-10T19:26:07.840 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:24:29.888101+0000", 2024-09-10T19:26:07.841 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.5489999999999995e-06 2024-09-10T19:26:07.841 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:26:07.841 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:26:07.841 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-10T19:26:07.841 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:24:30.886701+0000", 2024-09-10T19:26:07.841 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.99860015000000002 2024-09-10T19:26:07.841 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:26:07.841 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:26:07.841 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-10T19:26:07.841 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:24:30.886808+0000", 2024-09-10T19:26:07.841 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00010702800000000001 2024-09-10T19:26:07.841 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:26:07.842 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:26:07.842 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 3,4", 2024-09-10T19:26:07.842 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:24:30.886959+0000", 2024-09-10T19:26:07.842 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000150464 2024-09-10T19:26:07.842 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:26:07.842 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:26:07.842 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:26:07.842 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:24:30.888617+0000", 2024-09-10T19:26:07.842 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.001658666 2024-09-10T19:26:07.842 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:26:07.842 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:26:07.842 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-10T19:26:07.842 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:24:30.888627+0000", 2024-09-10T19:26:07.843 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.5109999999999992e-06 2024-09-10T19:26:07.843 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:26:07.843 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:26:07.843 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:26:07.843 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:24:30.888666+0000", 2024-09-10T19:26:07.843 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.9038000000000001e-05 2024-09-10T19:26:07.843 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:26:07.843 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:26:07.843 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-10T19:26:07.843 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:24:30.888710+0000", 2024-09-10T19:26:07.843 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.3869000000000003e-05 2024-09-10T19:26:07.843 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:26:07.844 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:26:07.844 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-10T19:26:07.844 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:24:30.888747+0000", 2024-09-10T19:26:07.844 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.7759000000000002e-05 2024-09-10T19:26:07.844 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:26:07.844 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:26:07.844 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:26:07.844 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:26:07.844 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:26:07.844 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.25622.0:12699 10.4 10:2fc36a6a:::benchmark_data_smithi183_176537_object12698:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e308)", 2024-09-10T19:26:07.844 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-10T19:24:29.888800+0000", 2024-09-10T19:26:07.844 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 97.857297067999994, 2024-09-10T19:26:07.844 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.0001131809999999, 2024-09-10T19:26:07.845 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-10T19:26:07.845 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T19:26:07.845 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-10T19:26:07.845 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.25622", 2024-09-10T19:26:07.845 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.183:0/3685189199", 2024-09-10T19:26:07.845 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 12699 2024-09-10T19:26:07.845 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:26:07.845 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-10T19:26:07.845 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:26:07.845 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-10T19:26:07.845 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:24:29.888800+0000", 2024-09-10T19:26:07.845 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:26:07.845 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:26:07.846 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:26:07.846 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-10T19:26:07.846 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:24:29.888800+0000", 2024-09-10T19:26:07.846 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:26:07.846 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:26:07.846 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:26:07.846 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-10T19:26:07.846 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:24:29.888799+0000", 2024-09-10T19:26:07.846 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4294967295.999999 2024-09-10T19:26:07.846 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:26:07.846 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:26:07.846 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-10T19:26:07.846 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:24:29.888824+0000", 2024-09-10T19:26:07.846 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.5369000000000001e-05 2024-09-10T19:26:07.846 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:26:07.847 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:26:07.847 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-10T19:26:07.847 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:24:29.888835+0000", 2024-09-10T19:26:07.847 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.0900999999999999e-05 2024-09-10T19:26:07.847 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:26:07.847 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:26:07.847 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-10T19:26:07.847 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:24:29.888849+0000", 2024-09-10T19:26:07.847 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.4238e-05 2024-09-10T19:26:07.847 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:26:07.847 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:26:07.847 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-10T19:26:07.847 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:24:29.888898+0000", 2024-09-10T19:26:07.847 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.8343000000000001e-05 2024-09-10T19:26:07.847 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:26:07.848 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:26:07.848 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for peered", 2024-09-10T19:26:07.848 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:24:29.888906+0000", 2024-09-10T19:26:07.848 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.2490000000000001e-06 2024-09-10T19:26:07.848 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:26:07.848 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:26:07.848 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-10T19:26:07.848 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:24:30.887425+0000", 2024-09-10T19:26:07.848 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.99851880299999995 2024-09-10T19:26:07.848 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:26:07.848 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:26:07.848 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-10T19:26:07.848 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:24:30.887563+0000", 2024-09-10T19:26:07.848 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000138288 2024-09-10T19:26:07.849 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:26:07.849 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:26:07.849 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 3,4", 2024-09-10T19:26:07.849 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:24:30.887797+0000", 2024-09-10T19:26:07.849 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000233644 2024-09-10T19:26:07.849 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:26:07.849 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:26:07.849 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-10T19:26:07.849 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:24:30.888632+0000", 2024-09-10T19:26:07.849 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00083575200000000004 2024-09-10T19:26:07.849 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:26:07.849 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:26:07.849 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:26:07.849 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:24:30.888795+0000", 2024-09-10T19:26:07.850 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00016316400000000001 2024-09-10T19:26:07.850 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:26:07.850 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:26:07.850 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:26:07.850 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:24:30.888841+0000", 2024-09-10T19:26:07.850 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.5364000000000001e-05 2024-09-10T19:26:07.850 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:26:07.850 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:26:07.850 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-10T19:26:07.850 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:24:30.888882+0000", 2024-09-10T19:26:07.850 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.0689999999999998e-05 2024-09-10T19:26:07.850 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:26:07.850 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:26:07.850 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-10T19:26:07.850 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:24:30.888913+0000", 2024-09-10T19:26:07.851 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.1510000000000002e-05 2024-09-10T19:26:07.851 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:26:07.851 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:26:07.851 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:26:07.851 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:26:07.851 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:26:07.851 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.25622.0:12714 10.4 10:20ad68a0:::benchmark_data_smithi183_176537_object12713:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e308)", 2024-09-10T19:26:07.851 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-10T19:24:29.904233+0000", 2024-09-10T19:26:07.851 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 97.841863660000001, 2024-09-10T19:26:07.851 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.98803233300000004, 2024-09-10T19:26:07.851 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-10T19:26:07.851 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T19:26:07.851 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-10T19:26:07.851 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.25622", 2024-09-10T19:26:07.851 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.183:0/3685189199", 2024-09-10T19:26:07.852 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 12714 2024-09-10T19:26:07.852 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:26:07.852 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-10T19:26:07.852 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:26:07.852 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-10T19:26:07.852 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:24:29.904233+0000", 2024-09-10T19:26:07.852 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:26:07.852 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:26:07.852 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:26:07.852 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-10T19:26:07.852 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:24:29.904233+0000", 2024-09-10T19:26:07.852 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:26:07.852 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:26:07.852 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:26:07.853 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-10T19:26:07.853 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:24:29.904233+0000", 2024-09-10T19:26:07.853 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4294967295.9999995 2024-09-10T19:26:07.853 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:26:07.853 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:26:07.853 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-10T19:26:07.853 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:24:29.904253+0000", 2024-09-10T19:26:07.853 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.0465000000000001e-05 2024-09-10T19:26:07.853 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:26:07.853 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:26:07.853 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-10T19:26:07.853 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:24:29.904259+0000", 2024-09-10T19:26:07.853 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.1709999999999999e-06 2024-09-10T19:26:07.854 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:26:07.854 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:26:07.854 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-10T19:26:07.854 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:24:29.904263+0000", 2024-09-10T19:26:07.854 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.4419999999999998e-06 2024-09-10T19:26:07.854 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:26:07.854 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:26:07.854 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-10T19:26:07.854 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:24:29.904311+0000", 2024-09-10T19:26:07.854 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.8387999999999998e-05 2024-09-10T19:26:07.854 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:26:07.854 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:26:07.854 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for peered", 2024-09-10T19:26:07.855 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:24:29.904324+0000", 2024-09-10T19:26:07.855 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.2996999999999999e-05 2024-09-10T19:26:07.855 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:26:07.855 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:26:07.855 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-10T19:26:07.855 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:24:30.888954+0000", 2024-09-10T19:26:07.855 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.98463026600000003 2024-09-10T19:26:07.855 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:26:07.855 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:26:07.855 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-10T19:26:07.855 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:24:30.889061+0000", 2024-09-10T19:26:07.855 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000106817 2024-09-10T19:26:07.855 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:26:07.856 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:26:07.856 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 3,4", 2024-09-10T19:26:07.856 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:24:30.889193+0000", 2024-09-10T19:26:07.856 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000131679 2024-09-10T19:26:07.856 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:26:07.856 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:26:07.856 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:26:07.856 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:24:30.890913+0000", 2024-09-10T19:26:07.856 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.001719932 2024-09-10T19:26:07.856 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:26:07.856 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:26:07.856 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-10T19:26:07.856 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:24:30.890922+0000", 2024-09-10T19:26:07.856 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.8140000000000007e-06 2024-09-10T19:26:07.857 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:26:07.857 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:26:07.857 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:26:07.857 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:24:30.892184+0000", 2024-09-10T19:26:07.857 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.001261908 2024-09-10T19:26:07.857 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:26:07.857 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:26:07.857 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-10T19:26:07.857 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:24:30.892235+0000", 2024-09-10T19:26:07.857 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.1345999999999999e-05 2024-09-10T19:26:07.857 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:26:07.857 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:26:07.857 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-10T19:26:07.858 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:24:30.892266+0000", 2024-09-10T19:26:07.858 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.0732999999999999e-05 2024-09-10T19:26:07.858 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:26:07.858 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:26:07.858 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:26:07.858 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:26:07.858 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:26:07.858 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.25622.0:12735 10.4 10:28d6c39d:::benchmark_data_smithi183_176537_object12734:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e308)", 2024-09-10T19:26:07.858 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-10T19:24:29.913367+0000", 2024-09-10T19:26:07.858 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 97.832730217000005, 2024-09-10T19:26:07.858 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.97905510100000004, 2024-09-10T19:26:07.858 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-10T19:26:07.858 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T19:26:07.858 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-10T19:26:07.859 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.25622", 2024-09-10T19:26:07.859 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.183:0/3685189199", 2024-09-10T19:26:07.859 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 12735 2024-09-10T19:26:07.859 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:26:07.859 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-10T19:26:07.859 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:26:07.859 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-10T19:26:07.859 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:24:29.913367+0000", 2024-09-10T19:26:07.859 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:26:07.859 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:26:07.859 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:26:07.859 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-10T19:26:07.859 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:24:29.913367+0000", 2024-09-10T19:26:07.860 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:26:07.860 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:26:07.860 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:26:07.860 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-10T19:26:07.860 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:24:29.913366+0000", 2024-09-10T19:26:07.860 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4294967295.999999 2024-09-10T19:26:07.860 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:26:07.860 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:26:07.860 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-10T19:26:07.860 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:24:29.913387+0000", 2024-09-10T19:26:07.860 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.1467e-05 2024-09-10T19:26:07.861 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:26:07.861 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:26:07.861 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-10T19:26:07.861 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:24:29.913396+0000", 2024-09-10T19:26:07.861 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.6600000000000001e-06 2024-09-10T19:26:07.861 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:26:07.861 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:26:07.861 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-10T19:26:07.861 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:24:29.913400+0000", 2024-09-10T19:26:07.861 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.4089999999999999e-06 2024-09-10T19:26:07.861 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:26:07.861 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:26:07.861 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-10T19:26:07.862 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:24:29.913458+0000", 2024-09-10T19:26:07.862 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.8452999999999998e-05 2024-09-10T19:26:07.862 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:26:07.862 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:26:07.862 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for peered", 2024-09-10T19:26:07.862 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:24:29.913467+0000", 2024-09-10T19:26:07.862 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.551e-06 2024-09-10T19:26:07.862 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:26:07.862 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:26:07.862 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-10T19:26:07.862 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:24:30.889563+0000", 2024-09-10T19:26:07.862 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.97609653399999996 2024-09-10T19:26:07.863 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:26:07.863 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:26:07.863 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-10T19:26:07.863 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:24:30.889669+0000", 2024-09-10T19:26:07.863 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00010582300000000001 2024-09-10T19:26:07.863 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:26:07.863 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:26:07.863 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 3,4", 2024-09-10T19:26:07.863 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:24:30.889821+0000", 2024-09-10T19:26:07.863 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00015202299999999999 2024-09-10T19:26:07.863 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:26:07.863 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:26:07.863 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:26:07.863 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:24:30.890976+0000", 2024-09-10T19:26:07.864 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.001155449 2024-09-10T19:26:07.864 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:26:07.864 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:26:07.864 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-10T19:26:07.864 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:24:30.891480+0000", 2024-09-10T19:26:07.864 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00050338900000000003 2024-09-10T19:26:07.864 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:26:07.864 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:26:07.864 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:26:07.864 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:24:30.892331+0000", 2024-09-10T19:26:07.864 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00085081600000000001 2024-09-10T19:26:07.864 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:26:07.864 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:26:07.865 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-10T19:26:07.865 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:24:30.892378+0000", 2024-09-10T19:26:07.865 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.7392000000000003e-05 2024-09-10T19:26:07.865 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:26:07.865 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:26:07.865 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-10T19:26:07.865 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:24:30.892422+0000", 2024-09-10T19:26:07.865 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.3872999999999998e-05 2024-09-10T19:26:07.865 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:26:07.865 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:26:07.865 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:26:07.865 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:26:07.865 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:26:07.866 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.25622.0:12743 10.4 10:24f51a86:::benchmark_data_smithi183_176537_object12742:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e308)", 2024-09-10T19:26:07.866 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-10T19:24:29.916291+0000", 2024-09-10T19:26:07.866 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 97.829805516999997, 2024-09-10T19:26:07.866 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.97625881800000003, 2024-09-10T19:26:07.866 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-10T19:26:07.866 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T19:26:07.866 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-10T19:26:07.866 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.25622", 2024-09-10T19:26:07.866 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.183:0/3685189199", 2024-09-10T19:26:07.866 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 12743 2024-09-10T19:26:07.866 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:26:07.866 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-10T19:26:07.866 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:26:07.866 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-10T19:26:07.867 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:24:29.916291+0000", 2024-09-10T19:26:07.867 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:26:07.867 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:26:07.867 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:26:07.867 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-10T19:26:07.867 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:24:29.916291+0000", 2024-09-10T19:26:07.867 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:26:07.867 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:26:07.867 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:26:07.867 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-10T19:26:07.867 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:24:29.916291+0000", 2024-09-10T19:26:07.867 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4294967295.999999 2024-09-10T19:26:07.867 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:26:07.867 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:26:07.868 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-10T19:26:07.868 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:24:29.916331+0000", 2024-09-10T19:26:07.868 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.0561e-05 2024-09-10T19:26:07.868 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:26:07.868 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:26:07.868 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-10T19:26:07.868 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:24:29.916340+0000", 2024-09-10T19:26:07.868 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.6940000000000001e-06 2024-09-10T19:26:07.868 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:26:07.868 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:26:07.868 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-10T19:26:07.868 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:24:29.916343+0000", 2024-09-10T19:26:07.868 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.5769999999999998e-06 2024-09-10T19:26:07.869 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:26:07.869 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:26:07.869 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-10T19:26:07.869 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:24:29.916379+0000", 2024-09-10T19:26:07.869 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.5574999999999997e-05 2024-09-10T19:26:07.869 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:26:07.869 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:26:07.869 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for peered", 2024-09-10T19:26:07.869 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:24:29.916386+0000", 2024-09-10T19:26:07.869 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.4449999999999999e-06 2024-09-10T19:26:07.869 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:26:07.869 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:26:07.869 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-10T19:26:07.869 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:24:30.890251+0000", 2024-09-10T19:26:07.870 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.97386462699999998 2024-09-10T19:26:07.870 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:26:07.870 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:26:07.870 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-10T19:26:07.870 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:24:30.890352+0000", 2024-09-10T19:26:07.870 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000101092 2024-09-10T19:26:07.870 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:26:07.870 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:26:07.870 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 3,4", 2024-09-10T19:26:07.871 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:24:30.890497+0000", 2024-09-10T19:26:07.871 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000144976 2024-09-10T19:26:07.871 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:26:07.871 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:26:07.872 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:26:07.872 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:24:30.892108+0000", 2024-09-10T19:26:07.872 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0016106 2024-09-10T19:26:07.872 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:26:07.872 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:26:07.872 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-10T19:26:07.872 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:24:30.892114+0000", 2024-09-10T19:26:07.872 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.6320000000000002e-06 2024-09-10T19:26:07.872 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:26:07.872 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:26:07.872 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:26:07.872 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:24:30.892471+0000", 2024-09-10T19:26:07.872 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00035673500000000001 2024-09-10T19:26:07.873 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:26:07.873 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:26:07.873 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-10T19:26:07.873 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:24:30.892514+0000", 2024-09-10T19:26:07.873 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.2778999999999998e-05 2024-09-10T19:26:07.873 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:26:07.873 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:26:07.873 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-10T19:26:07.873 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:24:30.892550+0000", 2024-09-10T19:26:07.873 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.6390000000000002e-05 2024-09-10T19:26:07.874 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:26:07.874 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:26:07.874 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:26:07.874 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:26:07.874 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:26:07.874 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.25622.0:12764 10.4 10:27245bd5:::benchmark_data_smithi183_176537_object12763:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e308)", 2024-09-10T19:26:07.874 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-10T19:24:29.923188+0000", 2024-09-10T19:26:07.874 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 97.822908519999999, 2024-09-10T19:26:07.874 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.97315216999999998, 2024-09-10T19:26:07.874 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-10T19:26:07.874 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T19:26:07.874 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-10T19:26:07.875 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.25622", 2024-09-10T19:26:07.875 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.183:0/3685189199", 2024-09-10T19:26:07.875 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 12764 2024-09-10T19:26:07.875 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:26:07.875 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-10T19:26:07.875 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:26:07.875 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-10T19:26:07.875 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:24:29.923188+0000", 2024-09-10T19:26:07.875 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:26:07.875 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:26:07.875 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:26:07.875 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-10T19:26:07.875 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:24:29.923188+0000", 2024-09-10T19:26:07.876 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:26:07.876 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:26:07.876 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:26:07.876 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-10T19:26:07.876 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:24:29.923188+0000", 2024-09-10T19:26:07.876 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4294967295.999999 2024-09-10T19:26:07.876 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:26:07.876 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:26:07.876 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-10T19:26:07.876 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:24:29.923209+0000", 2024-09-10T19:26:07.876 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.1639999999999999e-05 2024-09-10T19:26:07.876 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:26:07.876 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:26:07.877 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-10T19:26:07.877 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:24:29.923222+0000", 2024-09-10T19:26:07.877 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.2829e-05 2024-09-10T19:26:07.877 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:26:07.877 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:26:07.877 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-10T19:26:07.877 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:24:29.923228+0000", 2024-09-10T19:26:07.877 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.7180000000000001e-06 2024-09-10T19:26:07.877 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:26:07.877 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:26:07.877 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-10T19:26:07.877 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:24:29.923269+0000", 2024-09-10T19:26:07.877 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.1003e-05 2024-09-10T19:26:07.878 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:26:07.878 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:26:07.878 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for peered", 2024-09-10T19:26:07.878 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:24:29.923275+0000", 2024-09-10T19:26:07.878 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.528e-06 2024-09-10T19:26:07.878 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:26:07.878 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:26:07.878 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-10T19:26:07.878 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:24:30.891015+0000", 2024-09-10T19:26:07.878 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.96773988899999996 2024-09-10T19:26:07.878 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:26:07.878 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:26:07.879 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-10T19:26:07.879 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:24:30.891097+0000", 2024-09-10T19:26:07.879 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.1909000000000005e-05 2024-09-10T19:26:07.879 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:26:07.879 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:26:07.879 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 3,4", 2024-09-10T19:26:07.879 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:24:30.891192+0000", 2024-09-10T19:26:07.879 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.4766999999999996e-05 2024-09-10T19:26:07.879 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:26:07.879 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:26:07.879 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:26:07.879 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:24:30.892143+0000", 2024-09-10T19:26:07.880 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00095131499999999999 2024-09-10T19:26:07.880 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:26:07.880 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:26:07.880 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-10T19:26:07.880 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:24:30.892274+0000", 2024-09-10T19:26:07.880 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000130786 2024-09-10T19:26:07.880 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:26:07.880 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:26:07.880 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:26:07.880 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:24:30.896260+0000", 2024-09-10T19:26:07.880 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0039863219999999996 2024-09-10T19:26:07.880 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:26:07.880 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:26:07.881 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-10T19:26:07.881 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:24:30.896305+0000", 2024-09-10T19:26:07.881 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.4517000000000001e-05 2024-09-10T19:26:07.881 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:26:07.881 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:26:07.881 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-10T19:26:07.881 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:24:30.896341+0000", 2024-09-10T19:26:07.881 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.5729999999999998e-05 2024-09-10T19:26:07.881 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:26:07.881 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:26:07.881 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:26:07.881 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:26:07.882 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:26:07.882 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.25622.0:12766 10.4 10:2eaf6d51:::benchmark_data_smithi183_176537_object12765:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e308)", 2024-09-10T19:26:07.882 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-10T19:24:29.923661+0000", 2024-09-10T19:26:07.882 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 97.822436046000007, 2024-09-10T19:26:07.882 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.97444871899999996, 2024-09-10T19:26:07.882 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-10T19:26:07.882 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T19:26:07.882 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-10T19:26:07.882 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.25622", 2024-09-10T19:26:07.882 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.183:0/3685189199", 2024-09-10T19:26:07.882 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 12766 2024-09-10T19:26:07.882 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:26:07.882 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-10T19:26:07.882 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:26:07.883 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-10T19:26:07.883 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:24:29.923661+0000", 2024-09-10T19:26:07.883 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:26:07.883 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:26:07.883 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:26:07.883 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-10T19:26:07.883 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:24:29.923661+0000", 2024-09-10T19:26:07.883 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:26:07.883 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:26:07.883 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:26:07.883 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-10T19:26:07.883 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:24:29.923660+0000", 2024-09-10T19:26:07.884 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4294967295.999999 2024-09-10T19:26:07.884 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:26:07.884 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:26:07.884 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-10T19:26:07.884 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:24:29.923683+0000", 2024-09-10T19:26:07.884 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.2634999999999999e-05 2024-09-10T19:26:07.884 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:26:07.884 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:26:07.884 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-10T19:26:07.884 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:24:29.923694+0000", 2024-09-10T19:26:07.884 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.1684e-05 2024-09-10T19:26:07.884 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:26:07.884 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:26:07.885 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-10T19:26:07.885 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:24:29.923700+0000", 2024-09-10T19:26:07.885 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.4249999999999997e-06 2024-09-10T19:26:07.885 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:26:07.885 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:26:07.885 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-10T19:26:07.885 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:24:29.923763+0000", 2024-09-10T19:26:07.885 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.3712000000000003e-05 2024-09-10T19:26:07.885 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:26:07.885 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:26:07.885 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for peered", 2024-09-10T19:26:07.885 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:24:29.923772+0000", 2024-09-10T19:26:07.885 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.7509999999999997e-06 2024-09-10T19:26:07.885 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:26:07.886 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:26:07.886 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-10T19:26:07.886 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:24:30.891518+0000", 2024-09-10T19:26:07.886 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.96774575200000001 2024-09-10T19:26:07.886 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:26:07.886 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:26:07.886 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-10T19:26:07.886 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:24:30.891609+0000", 2024-09-10T19:26:07.886 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.1219999999999997e-05 2024-09-10T19:26:07.886 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:26:07.886 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:26:07.886 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 3,4", 2024-09-10T19:26:07.886 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:24:30.891730+0000", 2024-09-10T19:26:07.886 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000120949 2024-09-10T19:26:07.887 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:26:07.887 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:26:07.887 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-10T19:26:07.887 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:24:30.894095+0000", 2024-09-10T19:26:07.887 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.002364985 2024-09-10T19:26:07.887 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:26:07.887 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:26:07.887 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:26:07.887 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:24:30.894150+0000", 2024-09-10T19:26:07.887 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.5216999999999997e-05 2024-09-10T19:26:07.887 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:26:07.887 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:26:07.887 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:26:07.887 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:24:30.898028+0000", 2024-09-10T19:26:07.888 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0038773660000000001 2024-09-10T19:26:07.888 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:26:07.888 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:26:07.888 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-10T19:26:07.888 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:24:30.898082+0000", 2024-09-10T19:26:07.888 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.4367000000000003e-05 2024-09-10T19:26:07.888 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:26:07.888 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:26:07.888 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-10T19:26:07.888 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:24:30.898110+0000", 2024-09-10T19:26:07.888 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.7551000000000001e-05 2024-09-10T19:26:07.888 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:26:07.888 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:26:07.888 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:26:07.889 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:26:07.889 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:26:07.889 INFO:teuthology.orchestra.run.smithi183.stdout:} 2024-09-10T19:26:07.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:26:07.891 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:26:07.891 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:26:07.992 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:26:08.033 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:26:08.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:26:08.034 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:26:08.134 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:26:08.172 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:26:08.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:26:08.173 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:26:08.274 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:26:08.303 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:26:08.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:26:08.304 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:26:08.381 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:26:07 smithi183 ceph-mon[29736]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T19:26:08.381 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:26:07 smithi183 ceph-mon[29736]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-10T19:26:08.381 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:26:07 smithi183 ceph-mon[29736]: osdmap e343: 8 total, 8 up, 7 in 2024-09-10T19:26:08.381 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:26:07 smithi183 ceph-mon[29736]: 6.18 scrub starts 2024-09-10T19:26:08.381 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:26:07 smithi183 ceph-mon[29736]: 6.18 scrub ok 2024-09-10T19:26:08.382 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:26:07 smithi183 ceph-mon[41562]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T19:26:08.382 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:26:07 smithi183 ceph-mon[41562]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-10T19:26:08.382 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:26:07 smithi183 ceph-mon[41562]: osdmap e343: 8 total, 8 up, 7 in 2024-09-10T19:26:08.382 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:26:07 smithi183 ceph-mon[41562]: 6.18 scrub starts 2024-09-10T19:26:08.382 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:26:07 smithi183 ceph-mon[41562]: 6.18 scrub ok 2024-09-10T19:26:08.404 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:26:08.445 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:26:08.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:26:08.446 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:26:08.465 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:26:07 smithi191 ceph-mon[32884]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T19:26:08.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:26:07 smithi191 ceph-mon[32884]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-10T19:26:08.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:26:07 smithi191 ceph-mon[32884]: osdmap e343: 8 total, 8 up, 7 in 2024-09-10T19:26:08.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:26:07 smithi191 ceph-mon[32884]: 6.18 scrub starts 2024-09-10T19:26:08.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:26:07 smithi191 ceph-mon[32884]: 6.18 scrub ok 2024-09-10T19:26:08.547 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:26:08.568 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:26:08.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:26:08.569 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:26:08.670 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:26:08.701 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:26:08.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:26:08.702 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:26:08.803 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:26:08.840 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:26:08.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:26:08.841 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:26:08.942 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:26:08.968 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:26:08.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:26:08.969 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:26:09.067 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph daemon osd.2 dump_ops_in_flight 2024-09-10T19:26:09.069 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:26:09.103 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:26:09.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:26:09.105 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:26:09.205 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:26:09.239 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:26:09.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:26:09.240 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:26:09.331 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:26:08 smithi183 ceph-mon[29736]: 5.18 scrub starts 2024-09-10T19:26:09.331 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:26:08 smithi183 ceph-mon[29736]: 5.18 scrub ok 2024-09-10T19:26:09.331 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:26:08 smithi183 ceph-mon[29736]: pgmap v735: 158 pgs: 158 active+clean; 6.3 GiB data, 26 GiB used, 600 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.37k op/s 2024-09-10T19:26:09.331 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:26:08 smithi183 ceph-mon[29736]: 2.b scrub starts 2024-09-10T19:26:09.331 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:26:08 smithi183 ceph-mon[29736]: 2.b scrub ok 2024-09-10T19:26:09.331 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:26:08 smithi183 ceph-mon[29736]: 2.9 scrub starts 2024-09-10T19:26:09.331 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:26:08 smithi183 ceph-mon[29736]: 2.17 scrub starts 2024-09-10T19:26:09.331 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:26:08 smithi183 ceph-mon[29736]: 2.9 scrub ok 2024-09-10T19:26:09.331 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:26:08 smithi183 ceph-mon[29736]: 2.17 scrub ok 2024-09-10T19:26:09.331 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:26:08 smithi183 ceph-mon[29736]: 5.3 scrub starts 2024-09-10T19:26:09.331 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:26:08 smithi183 ceph-mon[29736]: 5.3 scrub ok 2024-09-10T19:26:09.332 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:26:08 smithi183 ceph-mon[41562]: 5.18 scrub starts 2024-09-10T19:26:09.332 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:26:08 smithi183 ceph-mon[41562]: 5.18 scrub ok 2024-09-10T19:26:09.332 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:26:08 smithi183 ceph-mon[41562]: pgmap v735: 158 pgs: 158 active+clean; 6.3 GiB data, 26 GiB used, 600 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.37k op/s 2024-09-10T19:26:09.332 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:26:08 smithi183 ceph-mon[41562]: 2.b scrub starts 2024-09-10T19:26:09.332 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:26:08 smithi183 ceph-mon[41562]: 2.b scrub ok 2024-09-10T19:26:09.332 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:26:08 smithi183 ceph-mon[41562]: 2.9 scrub starts 2024-09-10T19:26:09.332 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:26:08 smithi183 ceph-mon[41562]: 2.17 scrub starts 2024-09-10T19:26:09.332 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:26:08 smithi183 ceph-mon[41562]: 2.9 scrub ok 2024-09-10T19:26:09.332 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:26:08 smithi183 ceph-mon[41562]: 2.17 scrub ok 2024-09-10T19:26:09.332 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:26:08 smithi183 ceph-mon[41562]: 5.3 scrub starts 2024-09-10T19:26:09.332 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:26:08 smithi183 ceph-mon[41562]: 5.3 scrub ok 2024-09-10T19:26:09.341 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:26:09.380 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:26:09.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:26:09.381 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:26:09.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:26:08 smithi191 ceph-mon[32884]: 5.18 scrub starts 2024-09-10T19:26:09.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:26:08 smithi191 ceph-mon[32884]: 5.18 scrub ok 2024-09-10T19:26:09.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:26:08 smithi191 ceph-mon[32884]: pgmap v735: 158 pgs: 158 active+clean; 6.3 GiB data, 26 GiB used, 600 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.37k op/s 2024-09-10T19:26:09.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:26:08 smithi191 ceph-mon[32884]: 2.b scrub starts 2024-09-10T19:26:09.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:26:08 smithi191 ceph-mon[32884]: 2.b scrub ok 2024-09-10T19:26:09.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:26:08 smithi191 ceph-mon[32884]: 2.9 scrub starts 2024-09-10T19:26:09.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:26:08 smithi191 ceph-mon[32884]: 2.17 scrub starts 2024-09-10T19:26:09.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:26:08 smithi191 ceph-mon[32884]: 2.9 scrub ok 2024-09-10T19:26:09.467 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:26:08 smithi191 ceph-mon[32884]: 2.17 scrub ok 2024-09-10T19:26:09.467 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:26:08 smithi191 ceph-mon[32884]: 5.3 scrub starts 2024-09-10T19:26:09.467 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:26:08 smithi191 ceph-mon[32884]: 5.3 scrub ok 2024-09-10T19:26:09.482 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:26:09.519 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:26:09.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:26:09.520 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:26:09.621 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:26:09.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:26:09.659 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:26:09.660 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:26:09.761 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:26:09.795 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:26:09.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:26:09.797 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:26:09.897 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:26:09.923 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:26:09.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:26:09.924 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:26:10.025 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:26:10.052 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:26:10.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:26:10.053 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:26:10.154 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:26:10.179 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:26:10.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:26:10.181 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:26:10.215 INFO:journalctl@ceph.iscsi.iscsi.a.smithi191.stdout:Sep 10 19:26:09 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-iscsi-iscsi-a[61816]: debug there is no tcmu-runner data available 2024-09-10T19:26:10.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:26:10 smithi191 ceph-mon[32884]: 6.f scrub starts 2024-09-10T19:26:10.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:26:10 smithi191 ceph-mon[32884]: 6.f scrub ok 2024-09-10T19:26:10.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:26:10 smithi191 ceph-mon[32884]: 2.3 scrub starts 2024-09-10T19:26:10.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:26:10 smithi191 ceph-mon[32884]: 2.3 scrub ok 2024-09-10T19:26:10.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:26:10 smithi191 ceph-mon[32884]: 4.6 scrub starts 2024-09-10T19:26:10.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:26:10 smithi191 ceph-mon[32884]: 4.6 scrub ok 2024-09-10T19:26:10.263 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:26:10.282 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:26:10.290 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:26:10.325 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:26:10.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:26:10.326 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:26:10.390 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:26:09 smithi183 ceph-mon[29736]: 6.f scrub starts 2024-09-10T19:26:10.390 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:26:09 smithi183 ceph-mon[29736]: 6.f scrub ok 2024-09-10T19:26:10.391 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:26:09 smithi183 ceph-mon[29736]: 2.3 scrub starts 2024-09-10T19:26:10.391 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:26:09 smithi183 ceph-mon[29736]: 2.3 scrub ok 2024-09-10T19:26:10.391 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:26:09 smithi183 ceph-mon[29736]: 4.6 scrub starts 2024-09-10T19:26:10.391 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:26:09 smithi183 ceph-mon[29736]: 4.6 scrub ok 2024-09-10T19:26:10.391 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:26:10 smithi183 ceph-mon[41562]: 6.f scrub starts 2024-09-10T19:26:10.391 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:26:10 smithi183 ceph-mon[41562]: 6.f scrub ok 2024-09-10T19:26:10.391 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:26:10 smithi183 ceph-mon[41562]: 2.3 scrub starts 2024-09-10T19:26:10.391 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:26:10 smithi183 ceph-mon[41562]: 2.3 scrub ok 2024-09-10T19:26:10.391 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:26:10 smithi183 ceph-mon[41562]: 4.6 scrub starts 2024-09-10T19:26:10.391 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:26:10 smithi183 ceph-mon[41562]: 4.6 scrub ok 2024-09-10T19:26:10.427 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:26:10.459 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:26:10.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:26:10.460 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:26:10.560 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:26:10.603 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:26:10.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:26:10.604 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:26:10.705 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:26:10.739 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:26:10.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:26:10.741 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:26:10.841 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:26:10.875 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:26:10.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:26:10.877 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:26:10.977 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:26:11.012 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:26:11.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:26:11.013 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:26:11.114 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:26:11.147 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:26:11.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:26:11.148 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:26:11.249 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:26:11.290 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:26:11.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:26:11.291 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:26:11.316 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:26:11 smithi183 ceph-mon[29736]: purged_snaps scrub starts 2024-09-10T19:26:11.316 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:26:11 smithi183 ceph-mon[29736]: purged_snaps scrub ok 2024-09-10T19:26:11.316 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:26:11 smithi183 ceph-mon[29736]: from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:26:11.316 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:26:11 smithi183 ceph-mon[29736]: 6.6 scrub starts 2024-09-10T19:26:11.316 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:26:11 smithi183 ceph-mon[29736]: 6.6 scrub ok 2024-09-10T19:26:11.316 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:26:11 smithi183 ceph-mon[29736]: pgmap v736: 158 pgs: 158 active+clean; 6.2 GiB data, 26 GiB used, 600 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.38k op/s 2024-09-10T19:26:11.316 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:26:11 smithi183 ceph-mon[29736]: osdmap e344: 8 total, 8 up, 7 in 2024-09-10T19:26:11.316 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:26:11 smithi183 ceph-mon[29736]: 4.4 scrub starts 2024-09-10T19:26:11.316 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:26:11 smithi183 ceph-mon[29736]: 4.4 scrub ok 2024-09-10T19:26:11.317 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:26:11 smithi183 ceph-mon[41562]: purged_snaps scrub starts 2024-09-10T19:26:11.317 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:26:11 smithi183 ceph-mon[41562]: purged_snaps scrub ok 2024-09-10T19:26:11.317 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:26:11 smithi183 ceph-mon[41562]: from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:26:11.317 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:26:11 smithi183 ceph-mon[41562]: 6.6 scrub starts 2024-09-10T19:26:11.317 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:26:11 smithi183 ceph-mon[41562]: 6.6 scrub ok 2024-09-10T19:26:11.317 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:26:11 smithi183 ceph-mon[41562]: pgmap v736: 158 pgs: 158 active+clean; 6.2 GiB data, 26 GiB used, 600 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.38k op/s 2024-09-10T19:26:11.317 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:26:11 smithi183 ceph-mon[41562]: osdmap e344: 8 total, 8 up, 7 in 2024-09-10T19:26:11.317 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:26:11 smithi183 ceph-mon[41562]: 4.4 scrub starts 2024-09-10T19:26:11.317 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:26:11 smithi183 ceph-mon[41562]: 4.4 scrub ok 2024-09-10T19:26:11.391 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:26:11.434 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:26:11.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:26:11.435 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:26:11.465 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:26:11 smithi191 ceph-mon[32884]: purged_snaps scrub starts 2024-09-10T19:26:11.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:26:11 smithi191 ceph-mon[32884]: purged_snaps scrub ok 2024-09-10T19:26:11.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:26:11 smithi191 ceph-mon[32884]: from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:26:11.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:26:11 smithi191 ceph-mon[32884]: 6.6 scrub starts 2024-09-10T19:26:11.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:26:11 smithi191 ceph-mon[32884]: 6.6 scrub ok 2024-09-10T19:26:11.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:26:11 smithi191 ceph-mon[32884]: pgmap v736: 158 pgs: 158 active+clean; 6.2 GiB data, 26 GiB used, 600 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.38k op/s 2024-09-10T19:26:11.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:26:11 smithi191 ceph-mon[32884]: osdmap e344: 8 total, 8 up, 7 in 2024-09-10T19:26:11.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:26:11 smithi191 ceph-mon[32884]: 4.4 scrub starts 2024-09-10T19:26:11.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:26:11 smithi191 ceph-mon[32884]: 4.4 scrub ok 2024-09-10T19:26:11.535 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:26:11.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:26:11.573 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:26:11.573 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:26:11.637 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Sep 10 19:26:11 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-y[29989]: ::ffff:172.21.15.191 - - [10/Sep/2024:19:26:11] "GET /metrics HTTP/1.1" 200 224797 "" "Prometheus/2.43.0" 2024-09-10T19:26:11.668 INFO:teuthology.orchestra.run.smithi183.stdout:osd.0: {} 2024-09-10T19:26:11.668 INFO:teuthology.orchestra.run.smithi183.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-10T19:26:11.674 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:26:11.692 INFO:teuthology.orchestra.run.smithi183.stdout:osd.1: {} 2024-09-10T19:26:11.693 INFO:teuthology.orchestra.run.smithi183.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-10T19:26:11.711 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:26:11.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:26:11.712 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:26:11.713 INFO:teuthology.orchestra.run.smithi183.stdout:osd.2: {} 2024-09-10T19:26:11.713 INFO:teuthology.orchestra.run.smithi183.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-10T19:26:11.731 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-10T19:26:11.732 INFO:teuthology.orchestra.run.smithi183.stdout:osd.3: {} 2024-09-10T19:26:11.733 INFO:teuthology.orchestra.run.smithi183.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-10T19:26:11.763 INFO:teuthology.orchestra.run.smithi183.stdout:osd.4: {} 2024-09-10T19:26:11.763 INFO:teuthology.orchestra.run.smithi183.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-10T19:26:11.801 INFO:teuthology.orchestra.run.smithi183.stdout:osd.5: {} 2024-09-10T19:26:11.802 INFO:teuthology.orchestra.run.smithi183.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-10T19:26:11.813 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:26:11.839 INFO:teuthology.orchestra.run.smithi183.stdout:osd.6: {} 2024-09-10T19:26:11.839 INFO:teuthology.orchestra.run.smithi183.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-10T19:26:11.852 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:26:11.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:26:11.853 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:26:11.877 INFO:teuthology.orchestra.run.smithi183.stdout:osd.7: {} 2024-09-10T19:26:11.877 INFO:teuthology.orchestra.run.smithi183.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-10T19:26:11.955 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:26:11.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:26:11.984 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:26:11.984 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:26:12.085 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:26:12.115 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:26:12.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:26:12.116 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:26:12.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:26:12 smithi183 ceph-mon[29736]: 5.8 scrub starts 2024-09-10T19:26:12.182 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:26:12 smithi183 ceph-mon[29736]: 5.8 scrub ok 2024-09-10T19:26:12.182 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:26:12 smithi183 ceph-mon[29736]: from='client.? 172.21.15.183:0/2409013104' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T19:26:12.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:26:12 smithi183 ceph-mon[41562]: 5.8 scrub starts 2024-09-10T19:26:12.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:26:12 smithi183 ceph-mon[41562]: 5.8 scrub ok 2024-09-10T19:26:12.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:26:12 smithi183 ceph-mon[41562]: from='client.? 172.21.15.183:0/2409013104' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T19:26:12.217 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:26:12.249 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:26:12.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:26:12.250 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:26:12.351 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:26:12.386 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:26:12.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:26:12.387 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:26:12.465 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:26:12 smithi191 ceph-mon[32884]: 5.8 scrub starts 2024-09-10T19:26:12.465 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:26:12 smithi191 ceph-mon[32884]: 5.8 scrub ok 2024-09-10T19:26:12.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:26:12 smithi191 ceph-mon[32884]: from='client.? 172.21.15.183:0/2409013104' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T19:26:12.488 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:26:12.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:26:12.525 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:26:12.525 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:26:12.626 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:26:12.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:26:12.666 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:26:12.666 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:26:12.767 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:26:12.799 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:26:12.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:26:12.801 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:26:12.901 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:26:12.941 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:26:12.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:26:12.942 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:26:13.015 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:26:13 smithi183 ceph-mon[29736]: 4.9 scrub starts 2024-09-10T19:26:13.015 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:26:13 smithi183 ceph-mon[29736]: 4.9 scrub ok 2024-09-10T19:26:13.015 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:26:13 smithi183 ceph-mon[29736]: 2.1 scrub starts 2024-09-10T19:26:13.015 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:26:13 smithi183 ceph-mon[29736]: 2.1 scrub ok 2024-09-10T19:26:13.015 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:26:13 smithi183 ceph-mon[29736]: pgmap v738: 158 pgs: 158 active+clean; 6.0 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 646 B/s rd, 0 B/s wr, 1.51k op/s 2024-09-10T19:26:13.043 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:26:13.079 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:26:13.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:26:13.080 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:26:13.181 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:26:13.214 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:26:13.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:26:13.215 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:26:13.309 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:26:13 smithi183 ceph-mon[41562]: 4.9 scrub starts 2024-09-10T19:26:13.309 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:26:13 smithi183 ceph-mon[41562]: 4.9 scrub ok 2024-09-10T19:26:13.309 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:26:13 smithi183 ceph-mon[41562]: 2.1 scrub starts 2024-09-10T19:26:13.309 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:26:13 smithi183 ceph-mon[41562]: 2.1 scrub ok 2024-09-10T19:26:13.309 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:26:13 smithi183 ceph-mon[41562]: pgmap v738: 158 pgs: 158 active+clean; 6.0 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 646 B/s rd, 0 B/s wr, 1.51k op/s 2024-09-10T19:26:13.317 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:26:13.343 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:26:13.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:26:13.344 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:26:13.445 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:26:13.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:26:13 smithi191 ceph-mon[32884]: 4.9 scrub starts 2024-09-10T19:26:13.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:26:13 smithi191 ceph-mon[32884]: 4.9 scrub ok 2024-09-10T19:26:13.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:26:13 smithi191 ceph-mon[32884]: 2.1 scrub starts 2024-09-10T19:26:13.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:26:13 smithi191 ceph-mon[32884]: 2.1 scrub ok 2024-09-10T19:26:13.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:26:13 smithi191 ceph-mon[32884]: pgmap v738: 158 pgs: 158 active+clean; 6.0 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 646 B/s rd, 0 B/s wr, 1.51k op/s 2024-09-10T19:26:13.485 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:26:13.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:26:13.486 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:26:13.587 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:26:13.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:26:13.623 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:26:13.623 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:26:13.724 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:26:13.757 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:26:13.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:26:13.758 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:26:13.859 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:26:13.886 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:26:13.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:26:13.887 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:26:13.988 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:26:14.021 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:26:14.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:26:14.022 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:26:14.022 INFO:teuthology.orchestra.run.smithi183.stderr:reweighted osd.6 to 1 (10000) 2024-09-10T19:26:14.123 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:26:14.151 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:26:14.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:26:14.152 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:26:14.253 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:26:14.291 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:26:14.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:26:14.292 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:26:14.393 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:26:14.421 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:26:14.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:26:14.422 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:26:14.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:26:14 smithi183 ceph-mon[29736]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:26:14.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:26:14 smithi183 ceph-mon[29736]: from='client.? 172.21.15.183:0/2821825270' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-09-10T19:26:14.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:26:14 smithi183 ceph-mon[29736]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-09-10T19:26:14.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:26:14 smithi183 ceph-mon[41562]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:26:14.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:26:14 smithi183 ceph-mon[41562]: from='client.? 172.21.15.183:0/2821825270' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-09-10T19:26:14.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:26:14 smithi183 ceph-mon[41562]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-09-10T19:26:14.465 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:26:14 smithi191 ceph-mon[32884]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:26:14.465 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:26:14 smithi191 ceph-mon[32884]: from='client.? 172.21.15.183:0/2821825270' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-09-10T19:26:14.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:26:14 smithi191 ceph-mon[32884]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-09-10T19:26:14.523 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:26:14.558 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:26:14.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:26:14.559 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:26:14.660 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:26:14.702 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:26:14.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:26:14.703 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:26:14.803 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:26:14.840 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:26:14.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:26:14.841 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:26:14.942 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:26:14.968 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:26:14.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:26:14.969 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:26:15.070 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:26:15.104 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:26:15.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:26:15.105 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:26:15.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:26:15 smithi183 ceph-mon[29736]: 6.e scrub starts 2024-09-10T19:26:15.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:26:15 smithi183 ceph-mon[29736]: 6.e scrub ok 2024-09-10T19:26:15.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:26:15 smithi183 ceph-mon[29736]: pgmap v739: 158 pgs: 158 active+clean; 5.9 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.06k op/s 2024-09-10T19:26:15.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:26:15 smithi183 ceph-mon[29736]: 2.1b scrub starts 2024-09-10T19:26:15.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:26:15 smithi183 ceph-mon[29736]: 2.1b scrub ok 2024-09-10T19:26:15.182 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:26:15 smithi183 ceph-mon[29736]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-09-10T19:26:15.182 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:26:15 smithi183 ceph-mon[29736]: osdmap e345: 8 total, 8 up, 7 in 2024-09-10T19:26:15.182 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:26:15 smithi183 ceph-mon[29736]: 10.c scrub starts 2024-09-10T19:26:15.182 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:26:15 smithi183 ceph-mon[29736]: 4.b scrub starts 2024-09-10T19:26:15.182 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:26:15 smithi183 ceph-mon[29736]: 4.b scrub ok 2024-09-10T19:26:15.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:26:15 smithi183 ceph-mon[41562]: 6.e scrub starts 2024-09-10T19:26:15.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:26:15 smithi183 ceph-mon[41562]: 6.e scrub ok 2024-09-10T19:26:15.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:26:15 smithi183 ceph-mon[41562]: pgmap v739: 158 pgs: 158 active+clean; 5.9 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.06k op/s 2024-09-10T19:26:15.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:26:15 smithi183 ceph-mon[41562]: 2.1b scrub starts 2024-09-10T19:26:15.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:26:15 smithi183 ceph-mon[41562]: 2.1b scrub ok 2024-09-10T19:26:15.183 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:26:15 smithi183 ceph-mon[41562]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-09-10T19:26:15.183 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:26:15 smithi183 ceph-mon[41562]: osdmap e345: 8 total, 8 up, 7 in 2024-09-10T19:26:15.183 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:26:15 smithi183 ceph-mon[41562]: 10.c scrub starts 2024-09-10T19:26:15.183 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:26:15 smithi183 ceph-mon[41562]: 4.b scrub starts 2024-09-10T19:26:15.183 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:26:15 smithi183 ceph-mon[41562]: 4.b scrub ok 2024-09-10T19:26:15.203 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-10T19:26:15.205 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:26:15.235 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:26:15.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:26:15.236 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:26:15.318 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:26:15.337 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:26:15.353 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph osd reweight 7 1 2024-09-10T19:26:15.392 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:26:15.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:26:15.393 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:26:15.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:26:15 smithi191 ceph-mon[32884]: 6.e scrub starts 2024-09-10T19:26:15.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:26:15 smithi191 ceph-mon[32884]: 6.e scrub ok 2024-09-10T19:26:15.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:26:15 smithi191 ceph-mon[32884]: pgmap v739: 158 pgs: 158 active+clean; 5.9 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.06k op/s 2024-09-10T19:26:15.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:26:15 smithi191 ceph-mon[32884]: 2.1b scrub starts 2024-09-10T19:26:15.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:26:15 smithi191 ceph-mon[32884]: 2.1b scrub ok 2024-09-10T19:26:15.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:26:15 smithi191 ceph-mon[32884]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-09-10T19:26:15.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:26:15 smithi191 ceph-mon[32884]: osdmap e345: 8 total, 8 up, 7 in 2024-09-10T19:26:15.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:26:15 smithi191 ceph-mon[32884]: 10.c scrub starts 2024-09-10T19:26:15.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:26:15 smithi191 ceph-mon[32884]: 4.b scrub starts 2024-09-10T19:26:15.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:26:15 smithi191 ceph-mon[32884]: 4.b scrub ok 2024-09-10T19:26:15.495 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:26:15.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:26:15.520 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:26:15.520 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:26:15.621 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:26:15.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:26:15.660 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:26:15.660 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:26:15.761 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:26:15.800 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:26:15.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:26:15.801 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:26:15.902 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:26:15.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:26:15.936 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:26:15.936 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:26:16.037 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:26:16.074 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:26:16.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:26:16.075 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:26:16.175 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:26:16.202 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:26:16.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:26:16.205 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:26:16.306 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:26:16.335 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:26:16.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:26:16.336 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:26:16.364 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:26:16 smithi183 ceph-mon[41562]: 5.1a scrub starts 2024-09-10T19:26:16.365 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:26:16 smithi183 ceph-mon[41562]: 5.1a scrub ok 2024-09-10T19:26:16.365 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:26:16 smithi183 ceph-mon[29736]: 5.1a scrub starts 2024-09-10T19:26:16.365 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:26:16 smithi183 ceph-mon[29736]: 5.1a scrub ok 2024-09-10T19:26:16.437 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:26:16.465 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:26:16 smithi191 ceph-mon[32884]: 5.1a scrub starts 2024-09-10T19:26:16.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:26:16 smithi191 ceph-mon[32884]: 5.1a scrub ok 2024-09-10T19:26:16.469 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:26:16.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:26:16.470 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:26:16.571 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:26:16.598 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:26:16.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:26:16.599 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:26:16.700 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:26:16.732 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:26:16.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:26:16.733 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:26:16.835 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:26:16.865 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:26:16.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:26:16.866 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:26:16.953 INFO:teuthology.orchestra.run.smithi183.stdout:ERROR: (22) Invalid argument 2024-09-10T19:26:16.953 INFO:teuthology.orchestra.run.smithi183.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-10T19:26:16.967 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:26:17.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:26:17.003 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:26:17.003 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:26:17.104 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:26:17.139 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:26:17.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:26:17.140 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:26:17.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:26:17 smithi183 ceph-mon[29736]: pgmap v741: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 5.6 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 1.49k op/s 2024-09-10T19:26:17.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:26:17 smithi183 ceph-mon[29736]: 6.1c scrub starts 2024-09-10T19:26:17.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:26:17 smithi183 ceph-mon[29736]: 6.1c scrub ok 2024-09-10T19:26:17.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:26:17 smithi183 ceph-mon[29736]: 10.5 scrub starts 2024-09-10T19:26:17.182 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:26:17 smithi183 ceph-mon[29736]: 5.12 scrub starts 2024-09-10T19:26:17.182 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:26:17 smithi183 ceph-mon[29736]: 5.12 scrub ok 2024-09-10T19:26:17.182 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:26:17 smithi183 ceph-mon[29736]: 10.c scrub ok 2024-09-10T19:26:17.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:26:17 smithi183 ceph-mon[41562]: pgmap v741: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 5.6 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 1.49k op/s 2024-09-10T19:26:17.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:26:17 smithi183 ceph-mon[41562]: 6.1c scrub starts 2024-09-10T19:26:17.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:26:17 smithi183 ceph-mon[41562]: 6.1c scrub ok 2024-09-10T19:26:17.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:26:17 smithi183 ceph-mon[41562]: 10.5 scrub starts 2024-09-10T19:26:17.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:26:17 smithi183 ceph-mon[41562]: 5.12 scrub starts 2024-09-10T19:26:17.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:26:17 smithi183 ceph-mon[41562]: 5.12 scrub ok 2024-09-10T19:26:17.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:26:17 smithi183 ceph-mon[41562]: 10.c scrub ok 2024-09-10T19:26:17.241 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:26:17.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:26:17.282 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:26:17.282 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:26:17.383 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:26:17.413 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:26:17.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:26:17.414 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:26:17.451 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:26:17 smithi191 ceph-mon[32884]: pgmap v741: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 5.6 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 1.49k op/s 2024-09-10T19:26:17.451 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:26:17 smithi191 ceph-mon[32884]: 6.1c scrub starts 2024-09-10T19:26:17.451 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:26:17 smithi191 ceph-mon[32884]: 6.1c scrub ok 2024-09-10T19:26:17.451 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:26:17 smithi191 ceph-mon[32884]: 10.5 scrub starts 2024-09-10T19:26:17.451 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:26:17 smithi191 ceph-mon[32884]: 5.12 scrub starts 2024-09-10T19:26:17.451 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:26:17 smithi191 ceph-mon[32884]: 5.12 scrub ok 2024-09-10T19:26:17.451 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:26:17 smithi191 ceph-mon[32884]: 10.c scrub ok 2024-09-10T19:26:17.515 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:26:17.546 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:26:17.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:26:17.547 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:26:17.648 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:26:17.682 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:26:17.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:26:17.683 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:26:17.716 INFO:journalctl@ceph.mgr.x.smithi191.stdout:Sep 10 19:26:17 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-x[34967]: ::ffff:172.21.15.191 - - [10/Sep/2024:19:26:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T19:26:17.785 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:26:17.815 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:26:17.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:26:17.817 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:26:17.917 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:26:17.945 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:26:17.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:26:17.946 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:26:18.046 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:26:18.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:26:18.085 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:26:18.085 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:26:18.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:26:18 smithi183 ceph-mon[29736]: 5.14 scrub starts 2024-09-10T19:26:18.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:26:18 smithi183 ceph-mon[29736]: 5.14 scrub ok 2024-09-10T19:26:18.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:26:18 smithi183 ceph-mon[29736]: 4.f scrub starts 2024-09-10T19:26:18.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:26:18 smithi183 ceph-mon[29736]: 4.f scrub ok 2024-09-10T19:26:18.182 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:26:18 smithi183 ceph-mon[29736]: 10.5 scrub ok 2024-09-10T19:26:18.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:26:18 smithi183 ceph-mon[41562]: 5.14 scrub starts 2024-09-10T19:26:18.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:26:18 smithi183 ceph-mon[41562]: 5.14 scrub ok 2024-09-10T19:26:18.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:26:18 smithi183 ceph-mon[41562]: 4.f scrub starts 2024-09-10T19:26:18.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:26:18 smithi183 ceph-mon[41562]: 4.f scrub ok 2024-09-10T19:26:18.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:26:18 smithi183 ceph-mon[41562]: 10.5 scrub ok 2024-09-10T19:26:18.186 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:26:18.226 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:26:18.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:26:18.227 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:26:18.327 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:26:18.355 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph daemon osd.2 dump_blocked_ops 2024-09-10T19:26:18.356 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:26:18.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:26:18.357 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:26:18.400 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:26:18.458 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:26:18.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:26:18 smithi191 ceph-mon[32884]: 5.14 scrub starts 2024-09-10T19:26:18.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:26:18 smithi191 ceph-mon[32884]: 5.14 scrub ok 2024-09-10T19:26:18.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:26:18 smithi191 ceph-mon[32884]: 4.f scrub starts 2024-09-10T19:26:18.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:26:18 smithi191 ceph-mon[32884]: 4.f scrub ok 2024-09-10T19:26:18.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:26:18 smithi191 ceph-mon[32884]: 10.5 scrub ok 2024-09-10T19:26:18.493 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:26:18.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:26:18.494 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:26:18.595 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:26:18.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:26:18.629 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:26:18.629 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:26:18.730 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:26:18.765 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:26:18.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:26:18.766 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:26:18.867 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:26:18.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:26:18.901 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:26:18.902 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:26:19.002 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:26:19.030 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:26:19.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:26:19.031 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:26:19.132 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:26:19.169 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:26:19.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:26:19.171 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:26:19.176 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:26:19 smithi183 ceph-mon[29736]: 5.16 scrub starts 2024-09-10T19:26:19.176 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:26:19 smithi183 ceph-mon[29736]: 5.16 scrub ok 2024-09-10T19:26:19.176 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:26:19 smithi183 ceph-mon[29736]: pgmap v742: 158 pgs: 2 active+clean+scrubbing, 156 active+clean; 5.5 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.38k op/s 2024-09-10T19:26:19.176 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:26:19 smithi183 ceph-mon[29736]: 10.d scrub starts 2024-09-10T19:26:19.176 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:26:19 smithi183 ceph-mon[41562]: 5.16 scrub starts 2024-09-10T19:26:19.177 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:26:19 smithi183 ceph-mon[41562]: 5.16 scrub ok 2024-09-10T19:26:19.177 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:26:19 smithi183 ceph-mon[41562]: pgmap v742: 158 pgs: 2 active+clean+scrubbing, 156 active+clean; 5.5 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.38k op/s 2024-09-10T19:26:19.177 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:26:19 smithi183 ceph-mon[41562]: 10.d scrub starts 2024-09-10T19:26:19.184 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:26:19.272 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:26:19.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:26:19.300 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:26:19.300 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:26:19.401 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:26:19.430 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:26:19.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:26:19.431 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:26:19.465 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:26:19 smithi191 ceph-mon[32884]: 5.16 scrub starts 2024-09-10T19:26:19.465 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:26:19 smithi191 ceph-mon[32884]: 5.16 scrub ok 2024-09-10T19:26:19.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:26:19 smithi191 ceph-mon[32884]: pgmap v742: 158 pgs: 2 active+clean+scrubbing, 156 active+clean; 5.5 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.38k op/s 2024-09-10T19:26:19.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:26:19 smithi191 ceph-mon[32884]: 10.d scrub starts 2024-09-10T19:26:19.532 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:26:19.560 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:26:19.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:26:19.561 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:26:19.661 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:26:19.702 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:26:19.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:26:19.702 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:26:19.803 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:26:19.830 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:26:19.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:26:19.831 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:26:19.932 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:26:19.965 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:26:19.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:26:19.966 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:26:20.061 INFO:journalctl@ceph.iscsi.iscsi.a.smithi191.stdout:Sep 10 19:26:19 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-iscsi-iscsi-a[61816]: debug there is no tcmu-runner data available 2024-09-10T19:26:20.067 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:26:20.067 INFO:teuthology.orchestra.run.smithi183.stderr:nodeep-scrub is unset 2024-09-10T19:26:20.104 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:26:20.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:26:20.105 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:26:20.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:26:20 smithi183 ceph-mon[29736]: 5.0 scrub starts 2024-09-10T19:26:20.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:26:20 smithi183 ceph-mon[29736]: 5.0 scrub ok 2024-09-10T19:26:20.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:26:20 smithi183 ceph-mon[29736]: from='client.? 172.21.15.183:0/2334438646' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-10T19:26:20.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:26:20 smithi183 ceph-mon[29736]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-10T19:26:20.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:26:20 smithi183 ceph-mon[29736]: 10.b scrub starts 2024-09-10T19:26:20.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:26:20 smithi183 ceph-mon[29736]: 5.1e scrub starts 2024-09-10T19:26:20.182 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:26:20 smithi183 ceph-mon[29736]: 5.1e scrub ok 2024-09-10T19:26:20.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:26:20 smithi183 ceph-mon[41562]: 5.0 scrub starts 2024-09-10T19:26:20.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:26:20 smithi183 ceph-mon[41562]: 5.0 scrub ok 2024-09-10T19:26:20.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:26:20 smithi183 ceph-mon[41562]: from='client.? 172.21.15.183:0/2334438646' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-10T19:26:20.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:26:20 smithi183 ceph-mon[41562]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-10T19:26:20.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:26:20 smithi183 ceph-mon[41562]: 10.b scrub starts 2024-09-10T19:26:20.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:26:20 smithi183 ceph-mon[41562]: 5.1e scrub starts 2024-09-10T19:26:20.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:26:20 smithi183 ceph-mon[41562]: 5.1e scrub ok 2024-09-10T19:26:20.205 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:26:20.232 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:26:20.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:26:20.233 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:26:20.334 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:26:20.360 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:26:20.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:26:20.361 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:26:20.462 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:26:20.465 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:26:20 smithi191 ceph-mon[32884]: 5.0 scrub starts 2024-09-10T19:26:20.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:26:20 smithi191 ceph-mon[32884]: 5.0 scrub ok 2024-09-10T19:26:20.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:26:20 smithi191 ceph-mon[32884]: from='client.? 172.21.15.183:0/2334438646' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-10T19:26:20.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:26:20 smithi191 ceph-mon[32884]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-10T19:26:20.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:26:20 smithi191 ceph-mon[32884]: 10.b scrub starts 2024-09-10T19:26:20.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:26:20 smithi191 ceph-mon[32884]: 5.1e scrub starts 2024-09-10T19:26:20.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:26:20 smithi191 ceph-mon[32884]: 5.1e scrub ok 2024-09-10T19:26:20.488 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:26:20.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:26:20.489 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:26:20.590 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:26:20.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:26:20.623 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:26:20.623 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:26:20.724 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:26:20.753 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:26:20.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:26:20.753 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:26:20.854 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:26:20.879 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:26:20.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:26:20.880 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:26:20.981 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:26:21.006 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:26:21.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:26:21.007 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:26:21.108 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:26:21.136 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:26:21.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:26:21.137 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:26:21.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:26:21 smithi183 ceph-mon[29736]: from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:26:21.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:26:21 smithi183 ceph-mon[29736]: pgmap v743: 158 pgs: 4 active+clean+scrubbing, 154 active+clean; 5.3 GiB data, 23 GiB used, 602 GiB / 626 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1.36k op/s 2024-09-10T19:26:21.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:26:21 smithi183 ceph-mon[29736]: 1.1 scrub starts 2024-09-10T19:26:21.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:26:21 smithi183 ceph-mon[29736]: 1.1 scrub ok 2024-09-10T19:26:21.182 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:26:21 smithi183 ceph-mon[29736]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-10T19:26:21.182 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:26:21 smithi183 ceph-mon[29736]: Cluster is now healthy 2024-09-10T19:26:21.182 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:26:21 smithi183 ceph-mon[29736]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-10T19:26:21.182 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:26:21 smithi183 ceph-mon[29736]: osdmap e346: 8 total, 8 up, 7 in 2024-09-10T19:26:21.182 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:26:21 smithi183 ceph-mon[29736]: 10.d scrub ok 2024-09-10T19:26:21.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:26:21 smithi183 ceph-mon[41562]: from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:26:21.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:26:21 smithi183 ceph-mon[41562]: pgmap v743: 158 pgs: 4 active+clean+scrubbing, 154 active+clean; 5.3 GiB data, 23 GiB used, 602 GiB / 626 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1.36k op/s 2024-09-10T19:26:21.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:26:21 smithi183 ceph-mon[41562]: 1.1 scrub starts 2024-09-10T19:26:21.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:26:21 smithi183 ceph-mon[41562]: 1.1 scrub ok 2024-09-10T19:26:21.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:26:21 smithi183 ceph-mon[41562]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-10T19:26:21.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:26:21 smithi183 ceph-mon[41562]: Cluster is now healthy 2024-09-10T19:26:21.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:26:21 smithi183 ceph-mon[41562]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-10T19:26:21.183 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:26:21 smithi183 ceph-mon[41562]: osdmap e346: 8 total, 8 up, 7 in 2024-09-10T19:26:21.183 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:26:21 smithi183 ceph-mon[41562]: 10.d scrub ok 2024-09-10T19:26:21.238 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:26:21.273 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:26:21.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:26:21.274 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:26:21.375 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:26:21.402 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:26:21.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:26:21.403 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:26:21.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:26:21 smithi191 ceph-mon[32884]: from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:26:21.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:26:21 smithi191 ceph-mon[32884]: pgmap v743: 158 pgs: 4 active+clean+scrubbing, 154 active+clean; 5.3 GiB data, 23 GiB used, 602 GiB / 626 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1.36k op/s 2024-09-10T19:26:21.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:26:21 smithi191 ceph-mon[32884]: 1.1 scrub starts 2024-09-10T19:26:21.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:26:21 smithi191 ceph-mon[32884]: 1.1 scrub ok 2024-09-10T19:26:21.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:26:21 smithi191 ceph-mon[32884]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-10T19:26:21.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:26:21 smithi191 ceph-mon[32884]: Cluster is now healthy 2024-09-10T19:26:21.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:26:21 smithi191 ceph-mon[32884]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-10T19:26:21.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:26:21 smithi191 ceph-mon[32884]: osdmap e346: 8 total, 8 up, 7 in 2024-09-10T19:26:21.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:26:21 smithi191 ceph-mon[32884]: 10.d scrub ok 2024-09-10T19:26:21.470 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Sep 10 19:26:21 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-y[29989]: ::ffff:172.21.15.191 - - [10/Sep/2024:19:26:21] "GET /metrics HTTP/1.1" 200 224768 "" "Prometheus/2.43.0" 2024-09-10T19:26:21.504 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:26:21.544 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:26:21.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:26:21.546 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:26:21.646 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:26:21.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:26:21.683 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:26:21.683 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:26:21.784 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:26:21.820 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:26:21.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:26:21.822 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:26:21.923 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:26:21.954 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:26:21.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:26:21.955 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:26:22.056 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:26:22.094 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:26:22.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:26:22.096 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:26:22.196 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:26:22.235 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:26:22.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:26:22.236 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:26:22.336 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:26:22.357 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:26:22.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:26:22.358 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:26:22.423 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:26:22 smithi183 ceph-mon[29736]: purged_snaps scrub starts 2024-09-10T19:26:22.424 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:26:22 smithi183 ceph-mon[29736]: purged_snaps scrub ok 2024-09-10T19:26:22.424 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:26:22 smithi183 ceph-mon[29736]: osdmap e347: 8 total, 8 up, 7 in 2024-09-10T19:26:22.424 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:26:22 smithi183 ceph-mon[29736]: 10.b scrub ok 2024-09-10T19:26:22.424 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:26:22 smithi183 ceph-mon[41562]: purged_snaps scrub starts 2024-09-10T19:26:22.424 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:26:22 smithi183 ceph-mon[41562]: purged_snaps scrub ok 2024-09-10T19:26:22.424 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:26:22 smithi183 ceph-mon[41562]: osdmap e347: 8 total, 8 up, 7 in 2024-09-10T19:26:22.424 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:26:22 smithi183 ceph-mon[41562]: 10.b scrub ok 2024-09-10T19:26:22.459 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:26:22.467 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:26:22 smithi191 ceph-mon[32884]: purged_snaps scrub starts 2024-09-10T19:26:22.467 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:26:22 smithi191 ceph-mon[32884]: purged_snaps scrub ok 2024-09-10T19:26:22.467 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:26:22 smithi191 ceph-mon[32884]: osdmap e347: 8 total, 8 up, 7 in 2024-09-10T19:26:22.467 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:26:22 smithi191 ceph-mon[32884]: 10.b scrub ok 2024-09-10T19:26:22.496 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:26:22.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:26:22.497 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:26:22.597 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:26:22.622 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:26:22.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:26:22.624 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:26:22.724 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:26:22.750 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:26:22.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:26:22.751 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:26:22.852 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:26:22.887 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:26:22.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:26:22.888 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:26:22.989 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:26:23.017 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:26:23.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:26:23.018 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:26:23.119 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:26:23.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:26:23.145 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:26:23.145 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:26:23.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:26:23 smithi183 ceph-mon[29736]: pgmap v746: 158 pgs: 3 active+clean+scrubbing, 155 active+clean; 5.2 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 0 B/s wr, 1.47k op/s 2024-09-10T19:26:23.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:26:23 smithi183 ceph-mon[29736]: 5.6 scrub starts 2024-09-10T19:26:23.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:26:23 smithi183 ceph-mon[29736]: 5.6 scrub ok 2024-09-10T19:26:23.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:26:23 smithi183 ceph-mon[29736]: 6.1b scrub starts 2024-09-10T19:26:23.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:26:23 smithi183 ceph-mon[29736]: 6.1b scrub ok 2024-09-10T19:26:23.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:26:23 smithi183 ceph-mon[41562]: pgmap v746: 158 pgs: 3 active+clean+scrubbing, 155 active+clean; 5.2 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 0 B/s wr, 1.47k op/s 2024-09-10T19:26:23.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:26:23 smithi183 ceph-mon[41562]: 5.6 scrub starts 2024-09-10T19:26:23.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:26:23 smithi183 ceph-mon[41562]: 5.6 scrub ok 2024-09-10T19:26:23.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:26:23 smithi183 ceph-mon[41562]: 6.1b scrub starts 2024-09-10T19:26:23.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:26:23 smithi183 ceph-mon[41562]: 6.1b scrub ok 2024-09-10T19:26:23.246 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:26:23.287 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:26:23.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:26:23.288 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:26:23.389 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:26:23.419 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:26:23.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:26:23.420 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:26:23.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:26:23 smithi191 ceph-mon[32884]: pgmap v746: 158 pgs: 3 active+clean+scrubbing, 155 active+clean; 5.2 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 0 B/s wr, 1.47k op/s 2024-09-10T19:26:23.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:26:23 smithi191 ceph-mon[32884]: 5.6 scrub starts 2024-09-10T19:26:23.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:26:23 smithi191 ceph-mon[32884]: 5.6 scrub ok 2024-09-10T19:26:23.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:26:23 smithi191 ceph-mon[32884]: 6.1b scrub starts 2024-09-10T19:26:23.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:26:23 smithi191 ceph-mon[32884]: 6.1b scrub ok 2024-09-10T19:26:23.521 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:26:23.552 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:26:23.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:26:23.553 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:26:23.654 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:26:23.698 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:26:23.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:26:23.699 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:26:23.800 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:26:23.835 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:26:23.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:26:23.836 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:26:23.937 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:26:23.959 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:26:23.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:26:23.960 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:26:24.060 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:26:24.097 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:26:24.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:26:24.098 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:26:24.200 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:26:24.229 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:26:24.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:26:24.230 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:26:24.331 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:26:24.362 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:26:24.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:26:24.363 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:26:24.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:26:24 smithi183 ceph-mon[29736]: 4.1b scrub starts 2024-09-10T19:26:24.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:26:24 smithi183 ceph-mon[29736]: 4.1b scrub ok 2024-09-10T19:26:24.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:26:24 smithi183 ceph-mon[29736]: purged_snaps scrub starts 2024-09-10T19:26:24.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:26:24 smithi183 ceph-mon[29736]: purged_snaps scrub ok 2024-09-10T19:26:24.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:26:24 smithi183 ceph-mon[29736]: 10.8 scrub starts 2024-09-10T19:26:24.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:26:24 smithi183 ceph-mon[41562]: 4.1b scrub starts 2024-09-10T19:26:24.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:26:24 smithi183 ceph-mon[41562]: 4.1b scrub ok 2024-09-10T19:26:24.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:26:24 smithi183 ceph-mon[41562]: purged_snaps scrub starts 2024-09-10T19:26:24.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:26:24 smithi183 ceph-mon[41562]: purged_snaps scrub ok 2024-09-10T19:26:24.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:26:24 smithi183 ceph-mon[41562]: 10.8 scrub starts 2024-09-10T19:26:24.464 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:26:24.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:26:24 smithi191 ceph-mon[32884]: 4.1b scrub starts 2024-09-10T19:26:24.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:26:24 smithi191 ceph-mon[32884]: 4.1b scrub ok 2024-09-10T19:26:24.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:26:24 smithi191 ceph-mon[32884]: purged_snaps scrub starts 2024-09-10T19:26:24.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:26:24 smithi191 ceph-mon[32884]: purged_snaps scrub ok 2024-09-10T19:26:24.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:26:24 smithi191 ceph-mon[32884]: 10.8 scrub starts 2024-09-10T19:26:24.492 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:26:24.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:26:24.493 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:26:24.594 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:26:24.636 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:26:24.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:26:24.637 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:26:24.737 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:26:24.759 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:26:24.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:26:24.760 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:26:24.861 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:26:24.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:26:24.894 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:26:24.895 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:26:24.996 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:26:25.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:26:25.029 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:26:25.029 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:26:25.130 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:26:25.161 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:26:25.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:26:25.162 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:26:25.263 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:26:25.283 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:26:25.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:26:25.284 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:26:25.385 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:26:25.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:26:25.421 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:26:25.421 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:26:25.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:26:25 smithi183 ceph-mon[29736]: 5.13 scrub starts 2024-09-10T19:26:25.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:26:25 smithi183 ceph-mon[29736]: 5.13 scrub ok 2024-09-10T19:26:25.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:26:25 smithi183 ceph-mon[29736]: pgmap v747: 158 pgs: 2 active+clean+scrubbing, 156 active+clean; 5.1 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 0 B/s wr, 1.01k op/s 2024-09-10T19:26:25.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:26:25 smithi183 ceph-mon[29736]: purged_snaps scrub starts 2024-09-10T19:26:25.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:26:25 smithi183 ceph-mon[29736]: purged_snaps scrub ok 2024-09-10T19:26:25.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:26:25 smithi183 ceph-mon[29736]: osdmap e348: 8 total, 8 up, 7 in 2024-09-10T19:26:25.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:26:25 smithi183 ceph-mon[29736]: 10.4 scrub starts 2024-09-10T19:26:25.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:26:25 smithi183 ceph-mon[41562]: 5.13 scrub starts 2024-09-10T19:26:25.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:26:25 smithi183 ceph-mon[41562]: 5.13 scrub ok 2024-09-10T19:26:25.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:26:25 smithi183 ceph-mon[41562]: pgmap v747: 158 pgs: 2 active+clean+scrubbing, 156 active+clean; 5.1 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 0 B/s wr, 1.01k op/s 2024-09-10T19:26:25.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:26:25 smithi183 ceph-mon[41562]: purged_snaps scrub starts 2024-09-10T19:26:25.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:26:25 smithi183 ceph-mon[41562]: purged_snaps scrub ok 2024-09-10T19:26:25.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:26:25 smithi183 ceph-mon[41562]: osdmap e348: 8 total, 8 up, 7 in 2024-09-10T19:26:25.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:26:25 smithi183 ceph-mon[41562]: 10.4 scrub starts 2024-09-10T19:26:25.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:26:25 smithi191 ceph-mon[32884]: 5.13 scrub starts 2024-09-10T19:26:25.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:26:25 smithi191 ceph-mon[32884]: 5.13 scrub ok 2024-09-10T19:26:25.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:26:25 smithi191 ceph-mon[32884]: pgmap v747: 158 pgs: 2 active+clean+scrubbing, 156 active+clean; 5.1 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 0 B/s wr, 1.01k op/s 2024-09-10T19:26:25.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:26:25 smithi191 ceph-mon[32884]: purged_snaps scrub starts 2024-09-10T19:26:25.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:26:25 smithi191 ceph-mon[32884]: purged_snaps scrub ok 2024-09-10T19:26:25.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:26:25 smithi191 ceph-mon[32884]: osdmap e348: 8 total, 8 up, 7 in 2024-09-10T19:26:25.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:26:25 smithi191 ceph-mon[32884]: 10.4 scrub starts 2024-09-10T19:26:25.523 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:26:25.549 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:26:25.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:26:25.550 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:26:25.651 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:26:25.679 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:26:25.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:26:25.680 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:26:25.781 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:26:25.821 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:26:25.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:26:25.822 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:26:25.923 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:26:25.963 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:26:25.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:26:25.964 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:26:26.065 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:26:26.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:26:26.096 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:26:26.097 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:26:26.197 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:26:26.226 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:26:26.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:26:26.227 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:26:26.328 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:26:26.355 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:26:26.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:26:26.356 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:26:26.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:26:26 smithi183 ceph-mon[29736]: 5.17 scrub starts 2024-09-10T19:26:26.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:26:26 smithi183 ceph-mon[29736]: 5.17 scrub ok 2024-09-10T19:26:26.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:26:26 smithi183 ceph-mon[29736]: 5.1 deep-scrub starts 2024-09-10T19:26:26.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:26:26 smithi183 ceph-mon[29736]: 5.1 deep-scrub ok 2024-09-10T19:26:26.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:26:26 smithi183 ceph-mon[29736]: osdmap e349: 8 total, 8 up, 7 in 2024-09-10T19:26:26.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:26:26 smithi183 ceph-mon[29736]: 6.3 scrub starts 2024-09-10T19:26:26.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:26:26 smithi183 ceph-mon[29736]: 6.3 scrub ok 2024-09-10T19:26:26.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:26:26 smithi183 ceph-mon[29736]: 10.8 scrub ok 2024-09-10T19:26:26.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:26:26 smithi183 ceph-mon[41562]: 5.17 scrub starts 2024-09-10T19:26:26.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:26:26 smithi183 ceph-mon[41562]: 5.17 scrub ok 2024-09-10T19:26:26.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:26:26 smithi183 ceph-mon[41562]: 5.1 deep-scrub starts 2024-09-10T19:26:26.433 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:26:26 smithi183 ceph-mon[41562]: 5.1 deep-scrub ok 2024-09-10T19:26:26.433 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:26:26 smithi183 ceph-mon[41562]: osdmap e349: 8 total, 8 up, 7 in 2024-09-10T19:26:26.433 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:26:26 smithi183 ceph-mon[41562]: 6.3 scrub starts 2024-09-10T19:26:26.433 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:26:26 smithi183 ceph-mon[41562]: 6.3 scrub ok 2024-09-10T19:26:26.433 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:26:26 smithi183 ceph-mon[41562]: 10.8 scrub ok 2024-09-10T19:26:26.457 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:26:26.467 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:26:26 smithi191 ceph-mon[32884]: 5.17 scrub starts 2024-09-10T19:26:26.467 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:26:26 smithi191 ceph-mon[32884]: 5.17 scrub ok 2024-09-10T19:26:26.467 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:26:26 smithi191 ceph-mon[32884]: 5.1 deep-scrub starts 2024-09-10T19:26:26.467 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:26:26 smithi191 ceph-mon[32884]: 5.1 deep-scrub ok 2024-09-10T19:26:26.467 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:26:26 smithi191 ceph-mon[32884]: osdmap e349: 8 total, 8 up, 7 in 2024-09-10T19:26:26.467 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:26:26 smithi191 ceph-mon[32884]: 6.3 scrub starts 2024-09-10T19:26:26.467 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:26:26 smithi191 ceph-mon[32884]: 6.3 scrub ok 2024-09-10T19:26:26.467 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:26:26 smithi191 ceph-mon[32884]: 10.8 scrub ok 2024-09-10T19:26:26.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:26:26.493 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:26:26.493 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:26:26.594 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:26:26.628 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:26:26.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:26:26.629 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:26:26.730 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:26:26.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:26:26.766 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:26:26.766 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:26:26.867 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:26:26.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:26:26.897 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:26:26.897 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:26:26.998 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:26:27.028 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:26:27.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:26:27.029 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:26:27.085 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:26:27.130 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:26:27.160 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:26:27.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:26:27.162 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:26:27.262 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:26:27.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:26:27.303 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:26:27.303 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:26:27.350 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:26:27 smithi183 ceph-mon[29736]: pgmap v750: 158 pgs: 158 active+clean; 4.8 GiB data, 21 GiB used, 604 GiB / 626 GiB avail; 0 B/s wr, 1.47k op/s 2024-09-10T19:26:27.351 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:26:27 smithi183 ceph-mon[29736]: 10.4 scrub ok 2024-09-10T19:26:27.351 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:26:27 smithi183 ceph-mon[41562]: pgmap v750: 158 pgs: 158 active+clean; 4.8 GiB data, 21 GiB used, 604 GiB / 626 GiB avail; 0 B/s wr, 1.47k op/s 2024-09-10T19:26:27.351 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:26:27 smithi183 ceph-mon[41562]: 10.4 scrub ok 2024-09-10T19:26:27.351 INFO:teuthology.orchestra.run.smithi183.stdout:osd.0: {} 2024-09-10T19:26:27.351 INFO:teuthology.orchestra.run.smithi183.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-10T19:26:27.367 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:26:27 smithi191 ceph-mon[32884]: pgmap v750: 158 pgs: 158 active+clean; 4.8 GiB data, 21 GiB used, 604 GiB / 626 GiB avail; 0 B/s wr, 1.47k op/s 2024-09-10T19:26:27.367 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:26:27 smithi191 ceph-mon[32884]: 10.4 scrub ok 2024-09-10T19:26:27.371 INFO:teuthology.orchestra.run.smithi183.stdout:osd.1: {} 2024-09-10T19:26:27.371 INFO:teuthology.orchestra.run.smithi183.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-10T19:26:27.391 INFO:teuthology.orchestra.run.smithi183.stdout:osd.2: {} 2024-09-10T19:26:27.391 INFO:teuthology.orchestra.run.smithi183.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-10T19:26:27.404 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:26:27.413 INFO:teuthology.orchestra.run.smithi183.stdout:osd.3: {} 2024-09-10T19:26:27.414 INFO:teuthology.orchestra.run.smithi183.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-10T19:26:27.434 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:26:27.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:26:27.435 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:26:27.442 INFO:teuthology.orchestra.run.smithi183.stdout:osd.4: {} 2024-09-10T19:26:27.443 INFO:teuthology.orchestra.run.smithi183.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-10T19:26:27.474 INFO:teuthology.orchestra.run.smithi183.stdout:osd.5: {} 2024-09-10T19:26:27.475 INFO:teuthology.orchestra.run.smithi183.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-10T19:26:27.501 INFO:teuthology.orchestra.run.smithi183.stdout:osd.6: {} 2024-09-10T19:26:27.501 INFO:teuthology.orchestra.run.smithi183.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-10T19:26:27.536 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:26:27.537 INFO:teuthology.orchestra.run.smithi183.stdout:osd.7: {} 2024-09-10T19:26:27.538 INFO:teuthology.orchestra.run.smithi183.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-10T19:26:27.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:26:27.568 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:26:27.568 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:26:27.669 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:26:27.696 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:26:27.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:26:27.697 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:26:27.716 INFO:journalctl@ceph.mgr.x.smithi191.stdout:Sep 10 19:26:27 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-x[34967]: ::ffff:172.21.15.191 - - [10/Sep/2024:19:26:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T19:26:27.797 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:26:27.833 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:26:27.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:26:27.834 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:26:27.934 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:26:27.972 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:26:27.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:26:27.973 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:26:28.074 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:26:28.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:26:28.117 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:26:28.117 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:26:28.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:26:28 smithi183 ceph-mon[29736]: 2.1e scrub starts 2024-09-10T19:26:28.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:26:28 smithi183 ceph-mon[29736]: 2.1e scrub ok 2024-09-10T19:26:28.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:26:28 smithi183 ceph-mon[29736]: from='client.? 172.21.15.183:0/1224869769' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T19:26:28.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:26:28 smithi183 ceph-mon[29736]: 2.1f scrub starts 2024-09-10T19:26:28.182 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:26:28 smithi183 ceph-mon[29736]: 2.1f scrub ok 2024-09-10T19:26:28.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:26:28 smithi183 ceph-mon[41562]: 2.1e scrub starts 2024-09-10T19:26:28.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:26:28 smithi183 ceph-mon[41562]: 2.1e scrub ok 2024-09-10T19:26:28.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:26:28 smithi183 ceph-mon[41562]: from='client.? 172.21.15.183:0/1224869769' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T19:26:28.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:26:28 smithi183 ceph-mon[41562]: 2.1f scrub starts 2024-09-10T19:26:28.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:26:28 smithi183 ceph-mon[41562]: 2.1f scrub ok 2024-09-10T19:26:28.217 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:26:28.245 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:26:28.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:26:28.246 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:26:28.347 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:26:28.384 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:26:28.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:26:28.386 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:26:28.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:26:28 smithi191 ceph-mon[32884]: 2.1e scrub starts 2024-09-10T19:26:28.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:26:28 smithi191 ceph-mon[32884]: 2.1e scrub ok 2024-09-10T19:26:28.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:26:28 smithi191 ceph-mon[32884]: from='client.? 172.21.15.183:0/1224869769' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T19:26:28.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:26:28 smithi191 ceph-mon[32884]: 2.1f scrub starts 2024-09-10T19:26:28.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:26:28 smithi191 ceph-mon[32884]: 2.1f scrub ok 2024-09-10T19:26:28.487 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:26:28.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:26:28.517 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:26:28.517 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:26:28.618 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:26:28.645 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:26:28.645 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph osd set noscrub 2024-09-10T19:26:28.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:26:28.647 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:26:28.748 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:26:28.785 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:26:28.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:26:28.786 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:26:28.887 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:26:28.920 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:26:28.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:26:28.922 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:26:28.966 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:26:28 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=sqlstore.transactions t=2024-09-10T19:26:28.675000736Z level=info msg="Database locked, sleeping then retrying" error="database is locked" retry=0 code="database is locked" 2024-09-10T19:26:29.022 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:26:29.058 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:26:29.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:26:29.059 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:26:29.159 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:26:29.196 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:26:29.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:26:29.197 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:26:29.297 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:26:29.333 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:26:29.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:26:29.334 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:26:29.402 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:26:29 smithi183 ceph-mon[29736]: 4.d scrub starts 2024-09-10T19:26:29.402 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:26:29 smithi183 ceph-mon[29736]: 4.d scrub ok 2024-09-10T19:26:29.402 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:26:29 smithi183 ceph-mon[29736]: pgmap v751: 158 pgs: 158 active+clean; 4.7 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 1.14k op/s 2024-09-10T19:26:29.402 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:26:29 smithi183 ceph-mon[29736]: 5.2 scrub starts 2024-09-10T19:26:29.402 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:26:29 smithi183 ceph-mon[29736]: 5.2 scrub ok 2024-09-10T19:26:29.402 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:26:29 smithi183 ceph-mon[29736]: from='mgr.14556 ' entity='mgr.y' 2024-09-10T19:26:29.402 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:26:29 smithi183 ceph-mon[29736]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:26:29.403 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:26:29 smithi183 ceph-mon[41562]: 4.d scrub starts 2024-09-10T19:26:29.403 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:26:29 smithi183 ceph-mon[41562]: 4.d scrub ok 2024-09-10T19:26:29.403 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:26:29 smithi183 ceph-mon[41562]: pgmap v751: 158 pgs: 158 active+clean; 4.7 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 1.14k op/s 2024-09-10T19:26:29.403 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:26:29 smithi183 ceph-mon[41562]: 5.2 scrub starts 2024-09-10T19:26:29.403 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:26:29 smithi183 ceph-mon[41562]: 5.2 scrub ok 2024-09-10T19:26:29.403 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:26:29 smithi183 ceph-mon[41562]: from='mgr.14556 ' entity='mgr.y' 2024-09-10T19:26:29.403 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:26:29 smithi183 ceph-mon[41562]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:26:29.435 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:26:29.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:26:29 smithi191 ceph-mon[32884]: 4.d scrub starts 2024-09-10T19:26:29.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:26:29 smithi191 ceph-mon[32884]: 4.d scrub ok 2024-09-10T19:26:29.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:26:29 smithi191 ceph-mon[32884]: pgmap v751: 158 pgs: 158 active+clean; 4.7 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 1.5 KiB/s rd, 0 B/s wr, 1.14k op/s 2024-09-10T19:26:29.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:26:29 smithi191 ceph-mon[32884]: 5.2 scrub starts 2024-09-10T19:26:29.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:26:29 smithi191 ceph-mon[32884]: 5.2 scrub ok 2024-09-10T19:26:29.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:26:29 smithi191 ceph-mon[32884]: from='mgr.14556 ' entity='mgr.y' 2024-09-10T19:26:29.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:26:29 smithi191 ceph-mon[32884]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:26:29.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:26:29.467 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:26:29.467 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:26:29.568 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:26:29.596 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:26:29.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:26:29.597 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:26:29.698 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:26:29.737 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:26:29.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:26:29.738 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:26:29.839 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:26:29.870 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:26:29.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:26:29.871 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:26:29.972 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:26:30.008 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:26:30.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:26:30.009 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:26:30.110 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:26:30.130 INFO:journalctl@ceph.iscsi.iscsi.a.smithi191.stdout:Sep 10 19:26:29 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-iscsi-iscsi-a[61816]: debug there is no tcmu-runner data available 2024-09-10T19:26:30.141 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:26:30.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:26:30.142 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:26:30.239 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:26:30 smithi183 ceph-mon[41562]: 2.5 scrub starts 2024-09-10T19:26:30.239 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:26:30 smithi183 ceph-mon[41562]: 2.5 scrub ok 2024-09-10T19:26:30.239 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:26:30 smithi183 ceph-mon[29736]: 2.5 scrub starts 2024-09-10T19:26:30.239 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:26:30 smithi183 ceph-mon[29736]: 2.5 scrub ok 2024-09-10T19:26:30.243 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:26:30.276 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:26:30.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:26:30.277 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:26:30.377 INFO:teuthology.orchestra.run.smithi183.stdout:{ 2024-09-10T19:26:30.377 INFO:teuthology.orchestra.run.smithi183.stdout: "ops": [], 2024-09-10T19:26:30.377 INFO:teuthology.orchestra.run.smithi183.stdout: "complaint_time": 30, 2024-09-10T19:26:30.377 INFO:teuthology.orchestra.run.smithi183.stdout: "num_blocked_ops": 0 2024-09-10T19:26:30.378 INFO:teuthology.orchestra.run.smithi183.stdout:} 2024-09-10T19:26:30.378 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:26:30.388 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-10T19:26:30.413 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:26:30.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:26:30.414 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:26:30.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:26:30 smithi191 ceph-mon[32884]: 2.5 scrub starts 2024-09-10T19:26:30.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:26:30 smithi191 ceph-mon[32884]: 2.5 scrub ok 2024-09-10T19:26:30.514 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:26:30.544 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:26:30.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:26:30.545 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:26:30.645 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:26:30.674 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:26:30.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:26:30.675 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:26:30.776 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:26:30.815 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:26:30.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:26:30.816 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:26:30.917 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:26:30.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:26:30.945 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:26:30.945 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:26:31.046 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:26:31.079 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:26:31.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:26:31.080 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:26:31.181 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:26:31.221 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:26:31.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:26:31.222 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:26:31.322 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:26:31.346 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:26:31.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:26:31.348 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:26:31.431 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Sep 10 19:26:31 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-y[29989]: ::ffff:172.21.15.191 - - [10/Sep/2024:19:26:31] "GET /metrics HTTP/1.1" 200 224797 "" "Prometheus/2.43.0" 2024-09-10T19:26:31.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:26:31 smithi183 ceph-mon[29736]: from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:26:31.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:26:31 smithi183 ceph-mon[29736]: pgmap v752: 158 pgs: 158 active+clean; 4.6 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.28k op/s 2024-09-10T19:26:31.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:26:31 smithi183 ceph-mon[29736]: 5.1c scrub starts 2024-09-10T19:26:31.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:26:31 smithi183 ceph-mon[29736]: 5.1c scrub ok 2024-09-10T19:26:31.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:26:31 smithi183 ceph-mon[29736]: 6.13 deep-scrub starts 2024-09-10T19:26:31.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:26:31 smithi183 ceph-mon[29736]: 6.13 deep-scrub ok 2024-09-10T19:26:31.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:26:31 smithi183 ceph-mon[29736]: 2.d scrub starts 2024-09-10T19:26:31.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:26:31 smithi183 ceph-mon[29736]: 2.d scrub ok 2024-09-10T19:26:31.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:26:31 smithi183 ceph-mon[29736]: 1.6 scrub starts 2024-09-10T19:26:31.433 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:26:31 smithi183 ceph-mon[29736]: 1.6 scrub ok 2024-09-10T19:26:31.433 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:26:31 smithi183 ceph-mon[29736]: 5.b scrub starts 2024-09-10T19:26:31.433 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:26:31 smithi183 ceph-mon[29736]: 5.b scrub ok 2024-09-10T19:26:31.433 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:26:31 smithi183 ceph-mon[41562]: from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:26:31.433 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:26:31 smithi183 ceph-mon[41562]: pgmap v752: 158 pgs: 158 active+clean; 4.6 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.28k op/s 2024-09-10T19:26:31.433 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:26:31 smithi183 ceph-mon[41562]: 5.1c scrub starts 2024-09-10T19:26:31.433 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:26:31 smithi183 ceph-mon[41562]: 5.1c scrub ok 2024-09-10T19:26:31.433 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:26:31 smithi183 ceph-mon[41562]: 6.13 deep-scrub starts 2024-09-10T19:26:31.433 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:26:31 smithi183 ceph-mon[41562]: 6.13 deep-scrub ok 2024-09-10T19:26:31.433 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:26:31 smithi183 ceph-mon[41562]: 2.d scrub starts 2024-09-10T19:26:31.433 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:26:31 smithi183 ceph-mon[41562]: 2.d scrub ok 2024-09-10T19:26:31.434 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:26:31 smithi183 ceph-mon[41562]: 1.6 scrub starts 2024-09-10T19:26:31.434 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:26:31 smithi183 ceph-mon[41562]: 1.6 scrub ok 2024-09-10T19:26:31.434 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:26:31 smithi183 ceph-mon[41562]: 5.b scrub starts 2024-09-10T19:26:31.434 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:26:31 smithi183 ceph-mon[41562]: 5.b scrub ok 2024-09-10T19:26:31.448 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:26:31.465 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:26:31 smithi191 ceph-mon[32884]: from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:26:31.465 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:26:31 smithi191 ceph-mon[32884]: pgmap v752: 158 pgs: 158 active+clean; 4.6 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.28k op/s 2024-09-10T19:26:31.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:26:31 smithi191 ceph-mon[32884]: 5.1c scrub starts 2024-09-10T19:26:31.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:26:31 smithi191 ceph-mon[32884]: 5.1c scrub ok 2024-09-10T19:26:31.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:26:31 smithi191 ceph-mon[32884]: 6.13 deep-scrub starts 2024-09-10T19:26:31.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:26:31 smithi191 ceph-mon[32884]: 6.13 deep-scrub ok 2024-09-10T19:26:31.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:26:31 smithi191 ceph-mon[32884]: 2.d scrub starts 2024-09-10T19:26:31.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:26:31 smithi191 ceph-mon[32884]: 2.d scrub ok 2024-09-10T19:26:31.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:26:31 smithi191 ceph-mon[32884]: 1.6 scrub starts 2024-09-10T19:26:31.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:26:31 smithi191 ceph-mon[32884]: 1.6 scrub ok 2024-09-10T19:26:31.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:26:31 smithi191 ceph-mon[32884]: 5.b scrub starts 2024-09-10T19:26:31.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:26:31 smithi191 ceph-mon[32884]: 5.b scrub ok 2024-09-10T19:26:31.470 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:26:31.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:26:31.470 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:26:31.571 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:26:31.599 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:26:31.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:26:31.600 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:26:31.700 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:26:31.731 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:26:31.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:26:31.732 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:26:31.832 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:26:31.869 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:26:31.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:26:31.870 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:26:31.971 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:26:32.007 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:26:32.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:26:32.008 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:26:32.109 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:26:32.137 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:26:32.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:26:32.138 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:26:32.142 INFO:teuthology.orchestra.run.smithi183.stderr:reweighted osd.7 to 1 (10000) 2024-09-10T19:26:32.239 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:26:32.267 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:26:32.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:26:32.268 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:26:32.369 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:26:32.410 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:26:32.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:26:32.411 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:26:32.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:26:32 smithi183 ceph-mon[29736]: 6.1e scrub starts 2024-09-10T19:26:32.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:26:32 smithi183 ceph-mon[29736]: 6.1e scrub ok 2024-09-10T19:26:32.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:26:32 smithi183 ceph-mon[29736]: 6.1f scrub starts 2024-09-10T19:26:32.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:26:32 smithi183 ceph-mon[29736]: 6.1f scrub ok 2024-09-10T19:26:32.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:26:32 smithi183 ceph-mon[29736]: 1.7 scrub starts 2024-09-10T19:26:32.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:26:32 smithi183 ceph-mon[29736]: 1.7 scrub ok 2024-09-10T19:26:32.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:26:32 smithi183 ceph-mon[29736]: 2.15 deep-scrub starts 2024-09-10T19:26:32.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:26:32 smithi183 ceph-mon[29736]: 5.4 scrub starts 2024-09-10T19:26:32.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:26:32 smithi183 ceph-mon[29736]: 2.15 deep-scrub ok 2024-09-10T19:26:32.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:26:32 smithi183 ceph-mon[29736]: 5.4 scrub ok 2024-09-10T19:26:32.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:26:32 smithi183 ceph-mon[29736]: 6.9 scrub starts 2024-09-10T19:26:32.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:26:32 smithi183 ceph-mon[29736]: 6.9 scrub ok 2024-09-10T19:26:32.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:26:32 smithi183 ceph-mon[29736]: from='client.? 172.21.15.183:0/119290472' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-09-10T19:26:32.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:26:32 smithi183 ceph-mon[29736]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-09-10T19:26:32.433 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:26:32 smithi183 ceph-mon[41562]: 6.1e scrub starts 2024-09-10T19:26:32.433 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:26:32 smithi183 ceph-mon[41562]: 6.1e scrub ok 2024-09-10T19:26:32.433 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:26:32 smithi183 ceph-mon[41562]: 6.1f scrub starts 2024-09-10T19:26:32.433 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:26:32 smithi183 ceph-mon[41562]: 6.1f scrub ok 2024-09-10T19:26:32.433 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:26:32 smithi183 ceph-mon[41562]: 1.7 scrub starts 2024-09-10T19:26:32.433 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:26:32 smithi183 ceph-mon[41562]: 1.7 scrub ok 2024-09-10T19:26:32.433 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:26:32 smithi183 ceph-mon[41562]: 2.15 deep-scrub starts 2024-09-10T19:26:32.433 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:26:32 smithi183 ceph-mon[41562]: 5.4 scrub starts 2024-09-10T19:26:32.433 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:26:32 smithi183 ceph-mon[41562]: 2.15 deep-scrub ok 2024-09-10T19:26:32.433 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:26:32 smithi183 ceph-mon[41562]: 5.4 scrub ok 2024-09-10T19:26:32.434 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:26:32 smithi183 ceph-mon[41562]: 6.9 scrub starts 2024-09-10T19:26:32.434 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:26:32 smithi183 ceph-mon[41562]: 6.9 scrub ok 2024-09-10T19:26:32.434 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:26:32 smithi183 ceph-mon[41562]: from='client.? 172.21.15.183:0/119290472' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-09-10T19:26:32.434 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:26:32 smithi183 ceph-mon[41562]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-09-10T19:26:32.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:26:32 smithi191 ceph-mon[32884]: 6.1e scrub starts 2024-09-10T19:26:32.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:26:32 smithi191 ceph-mon[32884]: 6.1e scrub ok 2024-09-10T19:26:32.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:26:32 smithi191 ceph-mon[32884]: 6.1f scrub starts 2024-09-10T19:26:32.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:26:32 smithi191 ceph-mon[32884]: 6.1f scrub ok 2024-09-10T19:26:32.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:26:32 smithi191 ceph-mon[32884]: 1.7 scrub starts 2024-09-10T19:26:32.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:26:32 smithi191 ceph-mon[32884]: 1.7 scrub ok 2024-09-10T19:26:32.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:26:32 smithi191 ceph-mon[32884]: 2.15 deep-scrub starts 2024-09-10T19:26:32.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:26:32 smithi191 ceph-mon[32884]: 5.4 scrub starts 2024-09-10T19:26:32.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:26:32 smithi191 ceph-mon[32884]: 2.15 deep-scrub ok 2024-09-10T19:26:32.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:26:32 smithi191 ceph-mon[32884]: 5.4 scrub ok 2024-09-10T19:26:32.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:26:32 smithi191 ceph-mon[32884]: 6.9 scrub starts 2024-09-10T19:26:32.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:26:32 smithi191 ceph-mon[32884]: 6.9 scrub ok 2024-09-10T19:26:32.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:26:32 smithi191 ceph-mon[32884]: from='client.? 172.21.15.183:0/119290472' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-09-10T19:26:32.467 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:26:32 smithi191 ceph-mon[32884]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-09-10T19:26:32.512 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:26:32.550 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:26:32.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:26:32.551 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:26:32.626 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph daemon osd.2 dump_historic_ops 2024-09-10T19:26:32.652 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:26:32.684 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:26:32.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:26:32.685 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:26:32.786 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:26:32.820 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:26:32.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:26:32.822 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:26:32.922 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:26:32.951 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:26:32.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:26:32.952 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:26:33.053 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:26:33.088 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:26:33.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:26:33.089 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:26:33.092 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph osd reweight 1 1 2024-09-10T19:26:33.158 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:26:33.189 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:26:33.227 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:26:33.232 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:26:33.240 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:26:33.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:26:33.241 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:26:33.311 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:26:33 smithi183 ceph-mon[29736]: pgmap v753: 158 pgs: 158 active+clean; 4.4 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 1.40k op/s 2024-09-10T19:26:33.368 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:26:33 smithi183 ceph-mon[29736]: 4.11 scrub starts 2024-09-10T19:26:33.369 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:26:33 smithi183 ceph-mon[29736]: 4.11 scrub ok 2024-09-10T19:26:33.369 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:26:33 smithi183 ceph-mon[29736]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-09-10T19:26:33.369 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:26:33 smithi183 ceph-mon[29736]: osdmap e350: 8 total, 8 up, 7 in 2024-09-10T19:26:33.369 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:26:33 smithi183 ceph-mon[29736]: 5.5 scrub starts 2024-09-10T19:26:33.369 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:26:33 smithi183 ceph-mon[29736]: 5.5 scrub ok 2024-09-10T19:26:33.369 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:26:33 smithi183 ceph-mon[41562]: pgmap v753: 158 pgs: 158 active+clean; 4.4 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 1.40k op/s 2024-09-10T19:26:33.369 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:26:33 smithi183 ceph-mon[41562]: 4.11 scrub starts 2024-09-10T19:26:33.369 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:26:33 smithi183 ceph-mon[41562]: 4.11 scrub ok 2024-09-10T19:26:33.369 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:26:33 smithi183 ceph-mon[41562]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-09-10T19:26:33.369 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:26:33 smithi183 ceph-mon[41562]: osdmap e350: 8 total, 8 up, 7 in 2024-09-10T19:26:33.370 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:26:33 smithi183 ceph-mon[41562]: 5.5 scrub starts 2024-09-10T19:26:33.370 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:26:33 smithi183 ceph-mon[41562]: 5.5 scrub ok 2024-09-10T19:26:33.370 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:26:33.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:26:33.414 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:26:33.414 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:26:33.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:26:33 smithi191 ceph-mon[32884]: pgmap v753: 158 pgs: 158 active+clean; 4.4 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 1.40k op/s 2024-09-10T19:26:33.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:26:33 smithi191 ceph-mon[32884]: 4.11 scrub starts 2024-09-10T19:26:33.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:26:33 smithi191 ceph-mon[32884]: 4.11 scrub ok 2024-09-10T19:26:33.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:26:33 smithi191 ceph-mon[32884]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-09-10T19:26:33.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:26:33 smithi191 ceph-mon[32884]: osdmap e350: 8 total, 8 up, 7 in 2024-09-10T19:26:33.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:26:33 smithi191 ceph-mon[32884]: 5.5 scrub starts 2024-09-10T19:26:33.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:26:33 smithi191 ceph-mon[32884]: 5.5 scrub ok 2024-09-10T19:26:33.515 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:26:33.550 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:26:33.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:26:33.551 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:26:33.652 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:26:33.691 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:26:33.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:26:33.693 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:26:33.793 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:26:33.829 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:26:33.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:26:33.830 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:26:33.930 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:26:33.973 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:26:33.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:26:33.974 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:26:34.075 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:26:34.109 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:26:34.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:26:34.110 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:26:34.211 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:26:34.247 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:26:34.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:26:34.248 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:26:34.349 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:26:34.390 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:26:34.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:26:34.391 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:26:34.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:26:34 smithi183 ceph-mon[29736]: 2.16 scrub starts 2024-09-10T19:26:34.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:26:34 smithi183 ceph-mon[29736]: 2.16 scrub ok 2024-09-10T19:26:34.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:26:34 smithi183 ceph-mon[29736]: 4.a scrub starts 2024-09-10T19:26:34.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:26:34 smithi183 ceph-mon[29736]: 4.a scrub ok 2024-09-10T19:26:34.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:26:34 smithi183 ceph-mon[29736]: 6.11 scrub starts 2024-09-10T19:26:34.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:26:34 smithi183 ceph-mon[29736]: 6.11 scrub ok 2024-09-10T19:26:34.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:26:34 smithi183 ceph-mon[41562]: 2.16 scrub starts 2024-09-10T19:26:34.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:26:34 smithi183 ceph-mon[41562]: 2.16 scrub ok 2024-09-10T19:26:34.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:26:34 smithi183 ceph-mon[41562]: 4.a scrub starts 2024-09-10T19:26:34.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:26:34 smithi183 ceph-mon[41562]: 4.a scrub ok 2024-09-10T19:26:34.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:26:34 smithi183 ceph-mon[41562]: 6.11 scrub starts 2024-09-10T19:26:34.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:26:34 smithi183 ceph-mon[41562]: 6.11 scrub ok 2024-09-10T19:26:34.465 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:26:34 smithi191 ceph-mon[32884]: 2.16 scrub starts 2024-09-10T19:26:34.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:26:34 smithi191 ceph-mon[32884]: 2.16 scrub ok 2024-09-10T19:26:34.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:26:34 smithi191 ceph-mon[32884]: 4.a scrub starts 2024-09-10T19:26:34.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:26:34 smithi191 ceph-mon[32884]: 4.a scrub ok 2024-09-10T19:26:34.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:26:34 smithi191 ceph-mon[32884]: 6.11 scrub starts 2024-09-10T19:26:34.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:26:34 smithi191 ceph-mon[32884]: 6.11 scrub ok 2024-09-10T19:26:34.491 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:26:34.517 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:26:34.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:26:34.518 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:26:34.619 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:26:34.655 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:26:34.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:26:34.656 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:26:34.757 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:26:34.788 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:26:34.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:26:34.789 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:26:34.890 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:26:34.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:26:34.922 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:26:34.922 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:26:35.023 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:26:35.050 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:26:35.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:26:35.051 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:26:35.152 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:26:35.183 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:26:35.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:26:35.184 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:26:35.233 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:26:35.285 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:26:35.315 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:26:35.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:26:35.316 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:26:35.417 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:26:35.431 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:26:35 smithi183 ceph-mon[41562]: 6.7 deep-scrub starts 2024-09-10T19:26:35.431 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:26:35 smithi183 ceph-mon[41562]: 6.7 deep-scrub ok 2024-09-10T19:26:35.431 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:26:35 smithi183 ceph-mon[41562]: pgmap v755: 158 pgs: 158 active+clean; 4.3 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.20k op/s 2024-09-10T19:26:35.431 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:26:35 smithi183 ceph-mon[41562]: 4.13 scrub starts 2024-09-10T19:26:35.431 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:26:35 smithi183 ceph-mon[41562]: 4.13 scrub ok 2024-09-10T19:26:35.431 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:26:35 smithi183 ceph-mon[41562]: 1.4 scrub starts 2024-09-10T19:26:35.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:26:35 smithi183 ceph-mon[41562]: 1.4 scrub ok 2024-09-10T19:26:35.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:26:35 smithi183 ceph-mon[29736]: 6.7 deep-scrub starts 2024-09-10T19:26:35.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:26:35 smithi183 ceph-mon[29736]: 6.7 deep-scrub ok 2024-09-10T19:26:35.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:26:35 smithi183 ceph-mon[29736]: pgmap v755: 158 pgs: 158 active+clean; 4.3 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.20k op/s 2024-09-10T19:26:35.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:26:35 smithi183 ceph-mon[29736]: 4.13 scrub starts 2024-09-10T19:26:35.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:26:35 smithi183 ceph-mon[29736]: 4.13 scrub ok 2024-09-10T19:26:35.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:26:35 smithi183 ceph-mon[29736]: 1.4 scrub starts 2024-09-10T19:26:35.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:26:35 smithi183 ceph-mon[29736]: 1.4 scrub ok 2024-09-10T19:26:35.459 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:26:35.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:26:35.460 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:26:35.465 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:26:35 smithi191 ceph-mon[32884]: 6.7 deep-scrub starts 2024-09-10T19:26:35.465 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:26:35 smithi191 ceph-mon[32884]: 6.7 deep-scrub ok 2024-09-10T19:26:35.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:26:35 smithi191 ceph-mon[32884]: pgmap v755: 158 pgs: 158 active+clean; 4.3 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.20k op/s 2024-09-10T19:26:35.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:26:35 smithi191 ceph-mon[32884]: 4.13 scrub starts 2024-09-10T19:26:35.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:26:35 smithi191 ceph-mon[32884]: 4.13 scrub ok 2024-09-10T19:26:35.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:26:35 smithi191 ceph-mon[32884]: 1.4 scrub starts 2024-09-10T19:26:35.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:26:35 smithi191 ceph-mon[32884]: 1.4 scrub ok 2024-09-10T19:26:35.561 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:26:35.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:26:35.593 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:26:35.593 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:26:35.694 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:26:35.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:26:35.734 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:26:35.735 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:26:35.835 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:26:35.864 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:26:35.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:26:35.866 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:26:35.967 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:26:36.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:26:36.008 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:26:36.008 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:26:36.109 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:26:36.136 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:26:36.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:26:36.137 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:26:36.238 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:26:36.265 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:26:36 smithi183 ceph-mon[29736]: 6.d scrub starts 2024-09-10T19:26:36.265 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:26:36 smithi183 ceph-mon[29736]: 6.d scrub ok 2024-09-10T19:26:36.265 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:26:36 smithi183 ceph-mon[29736]: 2.1d scrub starts 2024-09-10T19:26:36.265 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:26:36 smithi183 ceph-mon[29736]: 2.1d scrub ok 2024-09-10T19:26:36.265 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:26:36 smithi183 ceph-mon[41562]: 6.d scrub starts 2024-09-10T19:26:36.266 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:26:36 smithi183 ceph-mon[41562]: 6.d scrub ok 2024-09-10T19:26:36.266 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:26:36 smithi183 ceph-mon[41562]: 2.1d scrub starts 2024-09-10T19:26:36.266 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:26:36 smithi183 ceph-mon[41562]: 2.1d scrub ok 2024-09-10T19:26:36.268 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:26:36.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:26:36.269 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:26:36.370 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:26:36.397 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:26:36.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:26:36.398 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:26:36.465 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:26:36 smithi191 ceph-mon[32884]: 6.d scrub starts 2024-09-10T19:26:36.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:26:36 smithi191 ceph-mon[32884]: 6.d scrub ok 2024-09-10T19:26:36.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:26:36 smithi191 ceph-mon[32884]: 2.1d scrub starts 2024-09-10T19:26:36.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:26:36 smithi191 ceph-mon[32884]: 2.1d scrub ok 2024-09-10T19:26:36.499 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:26:36.527 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:26:36.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:26:36.528 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:26:36.629 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:26:36.673 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:26:36.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:26:36.674 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:26:36.775 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:26:36.806 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:26:36.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:26:36.807 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:26:36.908 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:26:36.945 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:26:36.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:26:36.945 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:26:37.047 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:26:37.084 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:26:37.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:26:37.085 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:26:37.186 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:26:37.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:26:37.230 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:26:37.230 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:26:37.331 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:26:37.358 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:26:37.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:26:37.359 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:26:37.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:26:37 smithi183 ceph-mon[29736]: 10.9 scrub starts 2024-09-10T19:26:37.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:26:37 smithi183 ceph-mon[29736]: pgmap v756: 158 pgs: 158 active+clean; 4.0 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.32k op/s 2024-09-10T19:26:37.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:26:37 smithi183 ceph-mon[29736]: 10.2 scrub starts 2024-09-10T19:26:37.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:26:37 smithi183 ceph-mon[29736]: 4.3 scrub starts 2024-09-10T19:26:37.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:26:37 smithi183 ceph-mon[29736]: 4.3 scrub ok 2024-09-10T19:26:37.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:26:37 smithi183 ceph-mon[29736]: 2.19 scrub starts 2024-09-10T19:26:37.433 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:26:37 smithi183 ceph-mon[29736]: 2.19 scrub ok 2024-09-10T19:26:37.433 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:26:37 smithi183 ceph-mon[41562]: 10.9 scrub starts 2024-09-10T19:26:37.433 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:26:37 smithi183 ceph-mon[41562]: pgmap v756: 158 pgs: 158 active+clean; 4.0 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.32k op/s 2024-09-10T19:26:37.433 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:26:37 smithi183 ceph-mon[41562]: 10.2 scrub starts 2024-09-10T19:26:37.434 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:26:37 smithi183 ceph-mon[41562]: 4.3 scrub starts 2024-09-10T19:26:37.434 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:26:37 smithi183 ceph-mon[41562]: 4.3 scrub ok 2024-09-10T19:26:37.434 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:26:37 smithi183 ceph-mon[41562]: 2.19 scrub starts 2024-09-10T19:26:37.434 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:26:37 smithi183 ceph-mon[41562]: 2.19 scrub ok 2024-09-10T19:26:37.450 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:26:37 smithi191 ceph-mon[32884]: 10.9 scrub starts 2024-09-10T19:26:37.450 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:26:37 smithi191 ceph-mon[32884]: pgmap v756: 158 pgs: 158 active+clean; 4.0 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.32k op/s 2024-09-10T19:26:37.450 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:26:37 smithi191 ceph-mon[32884]: 10.2 scrub starts 2024-09-10T19:26:37.450 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:26:37 smithi191 ceph-mon[32884]: 4.3 scrub starts 2024-09-10T19:26:37.451 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:26:37 smithi191 ceph-mon[32884]: 4.3 scrub ok 2024-09-10T19:26:37.451 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:26:37 smithi191 ceph-mon[32884]: 2.19 scrub starts 2024-09-10T19:26:37.451 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:26:37 smithi191 ceph-mon[32884]: 2.19 scrub ok 2024-09-10T19:26:37.460 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:26:37.482 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:26:37.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:26:37.483 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:26:37.584 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:26:37.612 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:26:37.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:26:37.613 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:26:37.714 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:26:37.715 INFO:journalctl@ceph.mgr.x.smithi191.stdout:Sep 10 19:26:37 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-x[34967]: ::ffff:172.21.15.191 - - [10/Sep/2024:19:26:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T19:26:37.748 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:26:37.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:26:37.749 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:26:37.850 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:26:37.878 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:26:37.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:26:37.879 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:26:37.980 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:26:38.008 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:26:38.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:26:38.009 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:26:38.109 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:26:38.149 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:26:38.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:26:38.150 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:26:38.189 INFO:teuthology.orchestra.run.smithi183.stderr:noscrub is set 2024-09-10T19:26:38.251 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:26:38.295 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:26:38.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:26:38.296 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:26:38.397 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:26:38.428 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:26:38.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:26:38.429 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:26:38.431 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:26:38 smithi183 ceph-mon[41562]: from='client.? 172.21.15.183:0/2851817523' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-10T19:26:38.431 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:26:38 smithi183 ceph-mon[41562]: 4.2 deep-scrub starts 2024-09-10T19:26:38.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:26:38 smithi183 ceph-mon[41562]: 4.2 deep-scrub ok 2024-09-10T19:26:38.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:26:38 smithi183 ceph-mon[41562]: 10.2 scrub ok 2024-09-10T19:26:38.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:26:38 smithi183 ceph-mon[41562]: 10.9 scrub ok 2024-09-10T19:26:38.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:26:38 smithi183 ceph-mon[41562]: 1.2 scrub starts 2024-09-10T19:26:38.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:26:38 smithi183 ceph-mon[41562]: 1.2 scrub ok 2024-09-10T19:26:38.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:26:38 smithi183 ceph-mon[29736]: from='client.? 172.21.15.183:0/2851817523' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-10T19:26:38.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:26:38 smithi183 ceph-mon[29736]: 4.2 deep-scrub starts 2024-09-10T19:26:38.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:26:38 smithi183 ceph-mon[29736]: 4.2 deep-scrub ok 2024-09-10T19:26:38.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:26:38 smithi183 ceph-mon[29736]: 10.2 scrub ok 2024-09-10T19:26:38.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:26:38 smithi183 ceph-mon[29736]: 10.9 scrub ok 2024-09-10T19:26:38.433 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:26:38 smithi183 ceph-mon[29736]: 1.2 scrub starts 2024-09-10T19:26:38.433 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:26:38 smithi183 ceph-mon[29736]: 1.2 scrub ok 2024-09-10T19:26:38.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:26:38 smithi191 ceph-mon[32884]: from='client.? 172.21.15.183:0/2851817523' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-10T19:26:38.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:26:38 smithi191 ceph-mon[32884]: 4.2 deep-scrub starts 2024-09-10T19:26:38.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:26:38 smithi191 ceph-mon[32884]: 4.2 deep-scrub ok 2024-09-10T19:26:38.467 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:26:38 smithi191 ceph-mon[32884]: 10.2 scrub ok 2024-09-10T19:26:38.467 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:26:38 smithi191 ceph-mon[32884]: 10.9 scrub ok 2024-09-10T19:26:38.467 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:26:38 smithi191 ceph-mon[32884]: 1.2 scrub starts 2024-09-10T19:26:38.467 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:26:38 smithi191 ceph-mon[32884]: 1.2 scrub ok 2024-09-10T19:26:38.530 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:26:38.560 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:26:38.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:26:38.561 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:26:38.662 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:26:38.702 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:26:38.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:26:38.703 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:26:38.711 INFO:teuthology.orchestra.run.smithi183.stdout:{ 2024-09-10T19:26:38.711 INFO:teuthology.orchestra.run.smithi183.stdout: "size": 20, 2024-09-10T19:26:38.711 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 600, 2024-09-10T19:26:38.711 INFO:teuthology.orchestra.run.smithi183.stdout: "ops": [] 2024-09-10T19:26:38.711 INFO:teuthology.orchestra.run.smithi183.stdout:} 2024-09-10T19:26:38.804 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:26:38.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:26:38.831 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:26:38.831 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:26:38.932 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:26:38.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:26:38.971 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:26:38.972 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:26:39.072 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:26:39.112 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:26:39.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:26:39.113 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:26:39.214 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:26:39.240 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:26:39.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:26:39.241 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:26:39.341 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:26:39.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:26:39.370 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:26:39.370 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:26:39.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:26:39 smithi183 ceph-mon[29736]: pgmap v757: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 3.9 GiB data, 18 GiB used, 607 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.33k op/s 2024-09-10T19:26:39.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:26:39 smithi183 ceph-mon[29736]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T19:26:39.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:26:39 smithi183 ceph-mon[29736]: from='client.? 172.21.15.183:0/2851817523' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-10T19:26:39.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:26:39 smithi183 ceph-mon[29736]: osdmap e351: 8 total, 8 up, 7 in 2024-09-10T19:26:39.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:26:39 smithi183 ceph-mon[29736]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T19:26:39.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:26:39 smithi183 ceph-mon[41562]: pgmap v757: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 3.9 GiB data, 18 GiB used, 607 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.33k op/s 2024-09-10T19:26:39.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:26:39 smithi183 ceph-mon[41562]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T19:26:39.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:26:39 smithi183 ceph-mon[41562]: from='client.? 172.21.15.183:0/2851817523' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-10T19:26:39.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:26:39 smithi183 ceph-mon[41562]: osdmap e351: 8 total, 8 up, 7 in 2024-09-10T19:26:39.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:26:39 smithi183 ceph-mon[41562]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T19:26:39.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:26:39 smithi191 ceph-mon[32884]: pgmap v757: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 3.9 GiB data, 18 GiB used, 607 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.33k op/s 2024-09-10T19:26:39.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:26:39 smithi191 ceph-mon[32884]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T19:26:39.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:26:39 smithi191 ceph-mon[32884]: from='client.? 172.21.15.183:0/2851817523' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-10T19:26:39.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:26:39 smithi191 ceph-mon[32884]: osdmap e351: 8 total, 8 up, 7 in 2024-09-10T19:26:39.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:26:39 smithi191 ceph-mon[32884]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T19:26:39.471 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:26:39.506 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:26:39.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:26:39.507 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:26:39.608 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:26:39.650 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:26:39.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:26:39.651 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:26:39.751 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:26:39.797 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:26:39.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:26:39.798 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:26:39.899 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:26:39.939 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:26:39.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:26:39.940 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:26:40.041 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:26:40.081 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:26:40.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:26:40.082 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:26:40.183 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:26:40.215 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:26:40.216 INFO:journalctl@ceph.iscsi.iscsi.a.smithi191.stdout:Sep 10 19:26:39 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-iscsi-iscsi-a[61816]: debug there is no tcmu-runner data available 2024-09-10T19:26:40.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:26:40.217 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:26:40.318 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:26:40.345 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:26:40.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:26:40.346 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:26:40.447 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:26:40.474 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:26:40.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:26:40.475 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:26:40.576 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:26:40.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:26:40.609 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:26:40.609 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:26:40.710 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:26:40.735 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:26:40.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:26:40.736 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:26:40.837 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:26:40.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:26:40.871 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:26:40.871 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:26:40.972 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:26:41.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:26:41.010 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:26:41.010 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:26:41.111 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:26:41.155 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:26:41.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:26:41.156 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:26:41.257 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:26:41.285 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:26:41.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:26:41.286 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:26:41.387 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:26:41.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:26:41.425 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:26:41.425 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:26:41.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:26:41 smithi183 ceph-mon[29736]: from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:26:41.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:26:41 smithi183 ceph-mon[29736]: pgmap v759: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 3.8 GiB data, 18 GiB used, 607 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.34k op/s 2024-09-10T19:26:41.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:26:41 smithi183 ceph-mon[29736]: from='mgr.14556 ' entity='mgr.y' 2024-09-10T19:26:41.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:26:41 smithi183 ceph-mon[29736]: from='mgr.14556 ' entity='mgr.y' 2024-09-10T19:26:41.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:26:41 smithi183 ceph-mon[41562]: from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:26:41.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:26:41 smithi183 ceph-mon[41562]: pgmap v759: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 3.8 GiB data, 18 GiB used, 607 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.34k op/s 2024-09-10T19:26:41.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:26:41 smithi183 ceph-mon[41562]: from='mgr.14556 ' entity='mgr.y' 2024-09-10T19:26:41.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:26:41 smithi183 ceph-mon[41562]: from='mgr.14556 ' entity='mgr.y' 2024-09-10T19:26:41.432 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Sep 10 19:26:41 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-y[29989]: ::ffff:172.21.15.191 - - [10/Sep/2024:19:26:41] "GET /metrics HTTP/1.1" 200 224797 "" "Prometheus/2.43.0" 2024-09-10T19:26:41.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:26:41 smithi191 ceph-mon[32884]: from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:26:41.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:26:41 smithi191 ceph-mon[32884]: pgmap v759: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 3.8 GiB data, 18 GiB used, 607 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.34k op/s 2024-09-10T19:26:41.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:26:41 smithi191 ceph-mon[32884]: from='mgr.14556 ' entity='mgr.y' 2024-09-10T19:26:41.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:26:41 smithi191 ceph-mon[32884]: from='mgr.14556 ' entity='mgr.y' 2024-09-10T19:26:41.527 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:26:41.569 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:26:41.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:26:41.570 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:26:41.671 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:26:41.692 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph daemon osd.3 dump_ops_in_flight 2024-09-10T19:26:41.702 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:26:41.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:26:41.704 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:26:41.805 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:26:41.841 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:26:41.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:26:41.843 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:26:41.943 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:26:41.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:26:41.979 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:26:41.979 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:26:42.080 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:26:42.120 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:26:42.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:26:42.121 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:26:42.222 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:26:42.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:26:42.255 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:26:42.255 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:26:42.356 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:26:42.385 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:26:42.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:26:42.386 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:26:42.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:26:42 smithi183 ceph-mon[29736]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T19:26:42.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:26:42 smithi183 ceph-mon[29736]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T19:26:42.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:26:42 smithi183 ceph-mon[29736]: from='mgr.14556 ' entity='mgr.y' 2024-09-10T19:26:42.431 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:26:42 smithi183 ceph-mon[41562]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T19:26:42.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:26:42 smithi183 ceph-mon[41562]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T19:26:42.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:26:42 smithi183 ceph-mon[41562]: from='mgr.14556 ' entity='mgr.y' 2024-09-10T19:26:42.465 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:26:42 smithi191 ceph-mon[32884]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T19:26:42.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:26:42 smithi191 ceph-mon[32884]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T19:26:42.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:26:42 smithi191 ceph-mon[32884]: from='mgr.14556 ' entity='mgr.y' 2024-09-10T19:26:42.487 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:26:42.514 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:26:42.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:26:42.515 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:26:42.616 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:26:42.642 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:26:42.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:26:42.643 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:26:42.744 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:26:42.781 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:26:42.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:26:42.783 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:26:42.883 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:26:42.922 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:26:42.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:26:42.923 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:26:43.024 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:26:43.058 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:26:43.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:26:43.059 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:26:43.068 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:26:43.108 INFO:teuthology.orchestra.run.smithi183.stdout:osd.0: {} 2024-09-10T19:26:43.108 INFO:teuthology.orchestra.run.smithi183.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-10T19:26:43.135 INFO:teuthology.orchestra.run.smithi183.stdout:osd.1: {} 2024-09-10T19:26:43.135 INFO:teuthology.orchestra.run.smithi183.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-10T19:26:43.156 INFO:teuthology.orchestra.run.smithi183.stdout:osd.2: {} 2024-09-10T19:26:43.156 INFO:teuthology.orchestra.run.smithi183.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-10T19:26:43.160 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:26:43.178 INFO:teuthology.orchestra.run.smithi183.stdout:osd.3: {} 2024-09-10T19:26:43.178 INFO:teuthology.orchestra.run.smithi183.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-10T19:26:43.195 INFO:teuthology.orchestra.run.smithi183.stderr:reweighted osd.1 to 1 (10000) 2024-09-10T19:26:43.195 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:26:43.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:26:43.196 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:26:43.211 INFO:teuthology.orchestra.run.smithi183.stdout:osd.4: {} 2024-09-10T19:26:43.211 INFO:teuthology.orchestra.run.smithi183.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-10T19:26:43.238 INFO:teuthology.orchestra.run.smithi183.stdout:osd.5: {} 2024-09-10T19:26:43.238 INFO:teuthology.orchestra.run.smithi183.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-10T19:26:43.259 INFO:teuthology.orchestra.run.smithi183.stdout:osd.6: {} 2024-09-10T19:26:43.259 INFO:teuthology.orchestra.run.smithi183.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-10T19:26:43.292 INFO:teuthology.orchestra.run.smithi183.stdout:osd.7: {} 2024-09-10T19:26:43.292 INFO:teuthology.orchestra.run.smithi183.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-10T19:26:43.297 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:26:43.335 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:26:43.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:26:43.336 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:26:43.378 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:26:43 smithi183 ceph-mon[29736]: pgmap v760: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 3.6 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1.22k op/s 2024-09-10T19:26:43.378 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:26:43 smithi183 ceph-mon[29736]: from='client.? 172.21.15.183:0/1136915234' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-09-10T19:26:43.378 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:26:43 smithi183 ceph-mon[29736]: from='client.? 172.21.15.183:0/3401381660' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T19:26:43.379 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:26:43 smithi183 ceph-mon[41562]: pgmap v760: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 3.6 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1.22k op/s 2024-09-10T19:26:43.379 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:26:43 smithi183 ceph-mon[41562]: from='client.? 172.21.15.183:0/1136915234' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-09-10T19:26:43.379 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:26:43 smithi183 ceph-mon[41562]: from='client.? 172.21.15.183:0/3401381660' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T19:26:43.437 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:26:43.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:26:43 smithi191 ceph-mon[32884]: pgmap v760: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 3.6 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1.22k op/s 2024-09-10T19:26:43.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:26:43 smithi191 ceph-mon[32884]: from='client.? 172.21.15.183:0/1136915234' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-09-10T19:26:43.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:26:43 smithi191 ceph-mon[32884]: from='client.? 172.21.15.183:0/3401381660' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T19:26:43.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:26:43.468 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:26:43.469 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:26:43.570 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:26:43.598 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:26:43.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:26:43.599 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:26:43.699 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:26:43.730 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:26:43.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:26:43.731 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:26:43.832 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:26:43.866 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:26:43.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:26:43.867 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:26:43.968 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:26:43.994 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:26:43.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:26:43.995 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:26:44.096 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:26:44.124 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:26:44.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:26:44.125 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:26:44.226 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:26:44.266 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:26:44.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:26:44.267 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:26:44.368 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:26:44.403 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:26:44.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:26:44.404 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:26:44.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:26:44 smithi183 ceph-mon[29736]: from='client.? 172.21.15.183:0/1136915234' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-09-10T19:26:44.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:26:44 smithi183 ceph-mon[29736]: osdmap e352: 8 total, 8 up, 7 in 2024-09-10T19:26:44.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:26:44 smithi183 ceph-mon[29736]: from='mgr.14556 ' entity='mgr.y' 2024-09-10T19:26:44.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:26:44 smithi183 ceph-mon[29736]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:26:44.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:26:44 smithi183 ceph-mon[41562]: from='client.? 172.21.15.183:0/1136915234' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-09-10T19:26:44.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:26:44 smithi183 ceph-mon[41562]: osdmap e352: 8 total, 8 up, 7 in 2024-09-10T19:26:44.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:26:44 smithi183 ceph-mon[41562]: from='mgr.14556 ' entity='mgr.y' 2024-09-10T19:26:44.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:26:44 smithi183 ceph-mon[41562]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:26:44.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:26:44 smithi191 ceph-mon[32884]: from='client.? 172.21.15.183:0/1136915234' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-09-10T19:26:44.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:26:44 smithi191 ceph-mon[32884]: osdmap e352: 8 total, 8 up, 7 in 2024-09-10T19:26:44.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:26:44 smithi191 ceph-mon[32884]: from='mgr.14556 ' entity='mgr.y' 2024-09-10T19:26:44.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:26:44 smithi191 ceph-mon[32884]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:26:44.505 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:26:44.539 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:26:44.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:26:44.540 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:26:44.565 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-10T19:26:44.641 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:26:44.672 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:26:44.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:26:44.673 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:26:44.774 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:26:44.806 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:26:44.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:26:44.807 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:26:44.908 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:26:44.945 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:26:44.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:26:44.946 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:26:45.046 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:26:45.080 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:26:45.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:26:45.081 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:26:45.182 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:26:45.213 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:26:45.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:26:45.214 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:26:45.315 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:26:45.346 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:26:45.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:26:45.347 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:26:45.448 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:26:45.465 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:26:45 smithi191 ceph-mon[32884]: pgmap v762: 158 pgs: 158 active+clean; 3.5 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.03k op/s 2024-09-10T19:26:45.472 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:26:45.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:26:45.473 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:26:45.574 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:26:45.600 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:26:45.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:26:45.601 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:26:45.681 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:26:45 smithi183 ceph-mon[29736]: pgmap v762: 158 pgs: 158 active+clean; 3.5 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.03k op/s 2024-09-10T19:26:45.681 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:26:45 smithi183 ceph-mon[41562]: pgmap v762: 158 pgs: 158 active+clean; 3.5 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.03k op/s 2024-09-10T19:26:45.702 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:26:45.722 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:26:45.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:26:45.723 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:26:45.824 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:26:45.852 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:26:45.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:26:45.853 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:26:45.954 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:26:45.995 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:26:45.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:26:45.996 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:26:46.097 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:26:46.131 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:26:46.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:26:46.132 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:26:46.233 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:26:46.270 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:26:46.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:26:46.271 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:26:46.372 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:26:46.401 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:26:46.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:26:46.402 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:26:46.503 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:26:46.515 INFO:tasks.cephadm.ceph_manager.ceph:waiting for recovery to complete 2024-09-10T19:26:46.515 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph pg dump --format=json 2024-09-10T19:26:46.543 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:26:46.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:26:46.544 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:26:46.645 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:26:46.679 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:26:46.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:26:46.680 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:26:46.781 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:26:46.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:26:46.814 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:26:46.815 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:26:46.916 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:26:46.952 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:26:46.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:26:46.953 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:26:47.028 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:26:47.054 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:26:47.086 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:26:47.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:26:47.087 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:26:47.188 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:26:47.219 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:26:47.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:26:47.220 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:26:47.321 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:26:47.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:26:47.369 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:26:47.369 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:26:47.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:26:47 smithi183 ceph-mon[29736]: pgmap v763: 158 pgs: 158 active+clean; 3.2 GiB data, 16 GiB used, 609 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 1.45k op/s 2024-09-10T19:26:47.431 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:26:47 smithi183 ceph-mon[41562]: pgmap v763: 158 pgs: 158 active+clean; 3.2 GiB data, 16 GiB used, 609 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 1.45k op/s 2024-09-10T19:26:47.450 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:26:47 smithi191 ceph-mon[32884]: pgmap v763: 158 pgs: 158 active+clean; 3.2 GiB data, 16 GiB used, 609 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 1.45k op/s 2024-09-10T19:26:47.470 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:26:47.511 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:26:47.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:26:47.512 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:26:47.613 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:26:47.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:26:47.650 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:26:47.651 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:26:47.716 INFO:journalctl@ceph.mgr.x.smithi191.stdout:Sep 10 19:26:47 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-x[34967]: ::ffff:172.21.15.191 - - [10/Sep/2024:19:26:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T19:26:47.751 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:26:47.781 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:26:47.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:26:47.783 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:26:47.883 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:26:47.914 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:26:47.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:26:47.915 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:26:48.016 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:26:48.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:26:48.053 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:26:48.053 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:26:48.154 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:26:48.188 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:26:48.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:26:48.189 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:26:48.289 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:26:48.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:26:48.329 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:26:48.329 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:26:48.395 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-10T19:26:48.430 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:26:48.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:26:48.467 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:26:48.467 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:26:48.567 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:26:48.605 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:26:48.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:26:48.606 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:26:48.614 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:26:48.707 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:26:48.741 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:26:48.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:26:48.745 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:26:48.845 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:26:48.876 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:26:48.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:26:48.877 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:26:48.978 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:26:49.017 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:26:49.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:26:49.018 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:26:49.119 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:26:49.160 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:26:49.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:26:49.161 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:26:49.261 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:26:49.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:26:49.298 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:26:49.299 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:26:49.375 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:26:49 smithi183 ceph-mon[29736]: pgmap v764: 158 pgs: 158 active+clean; 3.1 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1.36k op/s 2024-09-10T19:26:49.375 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:26:49 smithi183 ceph-mon[41562]: pgmap v764: 158 pgs: 158 active+clean; 3.1 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1.36k op/s 2024-09-10T19:26:49.400 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:26:49.427 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:26:49.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:26:49.428 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:26:49.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:26:49 smithi191 ceph-mon[32884]: pgmap v764: 158 pgs: 158 active+clean; 3.1 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1.36k op/s 2024-09-10T19:26:49.528 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:26:49.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:26:49.556 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:26:49.556 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:26:49.657 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:26:49.686 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:26:49.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:26:49.687 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:26:49.788 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:26:49.824 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:26:49.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:26:49.829 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:26:49.852 INFO:teuthology.orchestra.run.smithi183.stdout:ERROR: (22) Invalid argument 2024-09-10T19:26:49.852 INFO:teuthology.orchestra.run.smithi183.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-10T19:26:49.930 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:26:49.960 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:26:49.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:26:49.961 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:26:50.062 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:26:50.091 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:26:50.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:26:50.092 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:26:50.193 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:26:50.216 INFO:journalctl@ceph.iscsi.iscsi.a.smithi191.stdout:Sep 10 19:26:49 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-iscsi-iscsi-a[61816]: debug there is no tcmu-runner data available 2024-09-10T19:26:50.223 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:26:50.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:26:50.224 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:26:50.325 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:26:50.368 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:26:50.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:26:50.369 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:26:50.470 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:26:50.500 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:26:50.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:26:50.502 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:26:50.602 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:26:50.632 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:26:50.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:26:50.633 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:26:50.734 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:26:50.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:26:50.773 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:26:50.773 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:26:50.874 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:26:50.908 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:26:50.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:26:50.910 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:26:51.011 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:26:51.042 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:26:51.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:26:51.043 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:26:51.144 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:26:51.179 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:26:51.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:26:51.180 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:26:51.281 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:26:51.308 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:26:51.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:26:51.309 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:26:51.409 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:26:51.437 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:26:51 smithi183 ceph-mon[41562]: from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:26:51.437 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:26:51 smithi183 ceph-mon[41562]: pgmap v765: 158 pgs: 158 active+clean; 2.9 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.33k op/s 2024-09-10T19:26:51.437 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:26:51 smithi183 ceph-mon[29736]: from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:26:51.437 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:26:51 smithi183 ceph-mon[29736]: pgmap v765: 158 pgs: 158 active+clean; 2.9 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.33k op/s 2024-09-10T19:26:51.438 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Sep 10 19:26:51 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-y[29989]: ::ffff:172.21.15.191 - - [10/Sep/2024:19:26:51] "GET /metrics HTTP/1.1" 200 224792 "" "Prometheus/2.43.0" 2024-09-10T19:26:51.439 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph daemon osd.3 dump_blocked_ops 2024-09-10T19:26:51.441 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:26:51.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:26:51.442 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:26:51.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:26:51 smithi191 ceph-mon[32884]: from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:26:51.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:26:51 smithi191 ceph-mon[32884]: pgmap v765: 158 pgs: 158 active+clean; 2.9 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.33k op/s 2024-09-10T19:26:51.544 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:26:51.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:26:51.578 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:26:51.578 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:26:51.679 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:26:51.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:26:51.722 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:26:51.722 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:26:51.728 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:26:51.823 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:26:51.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:26:51.858 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:26:51.858 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:26:51.959 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:26:51.989 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:26:51.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:26:51.990 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:26:52.090 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:26:52.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:26:52.129 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:26:52.129 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:26:52.229 INFO:teuthology.orchestra.run.smithi183.stderr:nodeep-scrub is set 2024-09-10T19:26:52.230 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:26:52.267 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:26:52.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:26:52.268 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:26:52.369 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:26:52.405 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:26:52.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:26:52.406 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:26:52.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:26:52 smithi183 ceph-mon[29736]: from='client.? 172.21.15.183:0/1703233078' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-10T19:26:52.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:26:52 smithi183 ceph-mon[41562]: from='client.? 172.21.15.183:0/1703233078' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-10T19:26:52.465 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:26:52 smithi191 ceph-mon[32884]: from='client.? 172.21.15.183:0/1703233078' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-10T19:26:52.507 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:26:52.544 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:26:52.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:26:52.546 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:26:52.647 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:26:52.678 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:26:52.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:26:52.679 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:26:52.780 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:26:52.809 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:26:52.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:26:52.810 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:26:52.911 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:26:52.938 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:26:52.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:26:52.939 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:26:53.039 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:26:53.076 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:26:53.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:26:53.077 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:26:53.177 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:26:53.205 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:26:53.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:26:53.206 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:26:53.307 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:26:53.345 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:26:53.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:26:53.347 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:26:53.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:26:53 smithi183 ceph-mon[29736]: pgmap v766: 158 pgs: 158 active+clean; 2.8 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.34k op/s 2024-09-10T19:26:53.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:26:53 smithi183 ceph-mon[29736]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T19:26:53.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:26:53 smithi183 ceph-mon[29736]: from='client.? 172.21.15.183:0/1703233078' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-10T19:26:53.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:26:53 smithi183 ceph-mon[29736]: osdmap e353: 8 total, 8 up, 7 in 2024-09-10T19:26:53.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:26:53 smithi183 ceph-mon[41562]: pgmap v766: 158 pgs: 158 active+clean; 2.8 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.34k op/s 2024-09-10T19:26:53.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:26:53 smithi183 ceph-mon[41562]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T19:26:53.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:26:53 smithi183 ceph-mon[41562]: from='client.? 172.21.15.183:0/1703233078' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-10T19:26:53.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:26:53 smithi183 ceph-mon[41562]: osdmap e353: 8 total, 8 up, 7 in 2024-09-10T19:26:53.447 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:26:53.465 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:26:53 smithi191 ceph-mon[32884]: pgmap v766: 158 pgs: 158 active+clean; 2.8 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.34k op/s 2024-09-10T19:26:53.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:26:53 smithi191 ceph-mon[32884]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T19:26:53.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:26:53 smithi191 ceph-mon[32884]: from='client.? 172.21.15.183:0/1703233078' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-10T19:26:53.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:26:53 smithi191 ceph-mon[32884]: osdmap e353: 8 total, 8 up, 7 in 2024-09-10T19:26:53.475 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:26:53.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:26:53.476 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:26:53.576 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:26:53.602 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:26:53.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:26:53.604 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:26:53.704 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:26:53.739 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:26:53.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:26:53.740 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:26:53.841 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:26:53.867 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:26:53.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:26:53.869 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:26:53.969 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:26:54.004 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:26:54.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:26:54.005 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:26:54.106 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:26:54.134 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:26:54.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:26:54.135 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:26:54.235 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:26:54.272 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:26:54.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:26:54.273 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:26:54.374 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:26:54.411 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:26:54.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:26:54.412 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:26:54.513 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:26:54.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:26:54.547 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:26:54.548 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:26:54.648 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:26:54.681 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:26:54.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:26:54.683 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:26:54.783 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:26:54.809 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:26:54.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:26:54.810 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:26:54.911 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:26:54.953 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:26:54.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:26:54.954 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:26:55.055 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:26:55.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:26:55.094 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:26:55.094 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:26:55.195 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:26:55.229 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:26:55.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:26:55.230 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:26:55.331 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:26:55.364 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:26:55.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:26:55.365 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:26:55.466 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:26:55.506 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:26:55.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:26:55.508 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:26:55.568 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:26:55 smithi183 ceph-mon[29736]: pgmap v768: 158 pgs: 158 active+clean; 2.7 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.35k op/s 2024-09-10T19:26:55.569 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:26:55 smithi183 ceph-mon[41562]: pgmap v768: 158 pgs: 158 active+clean; 2.7 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.35k op/s 2024-09-10T19:26:55.609 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:26:55.637 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:26:55.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:26:55.638 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:26:55.715 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:26:55 smithi191 ceph-mon[32884]: pgmap v768: 158 pgs: 158 active+clean; 2.7 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.35k op/s 2024-09-10T19:26:55.738 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:26:55.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:26:55.773 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:26:55.773 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:26:55.874 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:26:55.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:26:55.914 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:26:55.914 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:26:56.015 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:26:56.047 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:26:56.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:26:56.048 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:26:56.149 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:26:56.182 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:26:56.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:26:56.183 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:26:56.284 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:26:56.297 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-09-10T19:26:56.300 INFO:teuthology.orchestra.run.smithi183.stderr:dumped all 2024-09-10T19:26:56.314 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:26:56.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:26:56.316 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:26:56.416 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:26:56.446 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:26:56.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:26:56.447 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:26:56.490 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:26:56 smithi183 ceph-mon[29736]: purged_snaps scrub starts 2024-09-10T19:26:56.490 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:26:56 smithi183 ceph-mon[29736]: purged_snaps scrub ok 2024-09-10T19:26:56.490 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:26:56 smithi183 ceph-mon[41562]: purged_snaps scrub starts 2024-09-10T19:26:56.490 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:26:56 smithi183 ceph-mon[41562]: purged_snaps scrub ok 2024-09-10T19:26:56.548 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:26:56.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:26:56.585 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:26:56.585 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:26:56.687 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:26:56.715 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:26:56 smithi191 ceph-mon[32884]: purged_snaps scrub starts 2024-09-10T19:26:56.715 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:26:56 smithi191 ceph-mon[32884]: purged_snaps scrub ok 2024-09-10T19:26:56.722 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:26:56.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:26:56.723 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:26:56.824 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:26:56.853 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:26:56.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:26:56.854 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:26:56.955 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:26:56.985 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:26:56.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:26:56.986 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:26:57.086 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:26:57.112 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:26:57.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:26:57.113 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:26:57.205 INFO:teuthology.orchestra.run.smithi183.stdout:{"pg_ready":true,"pg_map":{"version":770,"stamp":"2024-09-10T19:26:56.230346+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":2554927348,"num_objects":39173,"num_object_clones":0,"num_object_copies":117519,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":39173,"num_whiteouts":0,"num_read":1324,"num_read_kb":1076,"num_write":208151,"num_write_kb":7885164,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4984,"num_bytes_recovered":306715526,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":73173,"ondisk_log_size":73173,"up":474,"acting":474,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":482,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":656216064,"kb_used":14394496,"kb_used_data":9272580,"kb_used_omap":19,"kb_used_meta":5121196,"kb_avail":641821568,"statfs":{"total":671965249536,"available":657225285632,"internally_reserved":0,"allocated":9495121920,"data_stored":9492975881,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19457,"internal_metadata":5244105727},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-772014080,"num_objects":-11780,"num_object_clones":0,"num_object_copies":-35340,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-11780,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":11780,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.327671"},"pg_stats":[{"pgid":"6.1b","version":"0'0","reported_seq":57,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-09-10T19:26:38.186905+0000","last_change":"2024-09-10T19:26:22.493488+0000","last_active":"2024-09-10T19:26:38.186905+0000","last_peered":"2024-09-10T19:26:38.186905+0000","last_clean":"2024-09-10T19:26:38.186905+0000","last_became_active":"2024-09-10T19:24:44.895077+0000","last_became_peered":"2024-09-10T19:24:44.895077+0000","last_unstale":"2024-09-10T19:26:38.186905+0000","last_undegraded":"2024-09-10T19:26:38.186905+0000","last_fullsized":"2024-09-10T19:26:38.186905+0000","mapping_epoch":314,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":315,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:26:22.493431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:26:22.493431+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:27:46.054799+0000","scrub_duration":0.0052677039999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6],"acting":[0,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":325,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-09-10T19:26:38.186586+0000","last_change":"2024-09-10T19:26:27.556376+0000","last_active":"2024-09-10T19:26:38.186586+0000","last_peered":"2024-09-10T19:26:38.186586+0000","last_clean":"2024-09-10T19:26:38.186586+0000","last_became_active":"2024-09-10T19:24:23.892621+0000","last_became_peered":"2024-09-10T19:24:23.892621+0000","last_unstale":"2024-09-10T19:26:38.186586+0000","last_undegraded":"2024-09-10T19:26:38.186586+0000","last_fullsized":"2024-09-10T19:26:38.186586+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:26:27.556297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:16:38.194472+0000","last_clean_scrub_stamp":"2024-09-10T19:26:27.556297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:27:32.452754+0000","scrub_duration":0.0045129510000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.19","version":"237'27","reported_seq":427,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-09-10T19:26:43.197968+0000","last_change":"2024-09-10T19:24:40.424985+0000","last_active":"2024-09-10T19:26:43.197968+0000","last_peered":"2024-09-10T19:26:43.197968+0000","last_clean":"2024-09-10T19:26:43.197968+0000","last_became_active":"2024-09-10T19:24:23.906592+0000","last_became_peered":"2024-09-10T19:24:23.906592+0000","last_unstale":"2024-09-10T19:26:43.197968+0000","last_undegraded":"2024-09-10T19:26:43.197968+0000","last_fullsized":"2024-09-10T19:26:43.197968+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"237'27","last_scrub_stamp":"2024-09-10T19:24:40.424918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:24:40.424918+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0040964390000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":401,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-09-10T19:26:43.198223+0000","last_change":"2024-09-10T19:26:07.791959+0000","last_active":"2024-09-10T19:26:43.198223+0000","last_peered":"2024-09-10T19:26:43.198223+0000","last_clean":"2024-09-10T19:26:43.198223+0000","last_became_active":"2024-09-10T19:24:01.976841+0000","last_became_peered":"2024-09-10T19:24:01.976841+0000","last_unstale":"2024-09-10T19:26:43.198223+0000","last_undegraded":"2024-09-10T19:26:43.198223+0000","last_fullsized":"2024-09-10T19:26:43.198223+0000","mapping_epoch":292,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":293,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:26:07.791886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:19:28.498560+0000","last_clean_scrub_stamp":"2024-09-10T19:26:07.791886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:27:28.278006+0000","scrub_duration":0.0046830530000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"59'1","reported_seq":388,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-09-10T19:26:43.198886+0000","last_change":"2024-09-10T19:26:26.891622+0000","last_active":"2024-09-10T19:26:43.198886+0000","last_peered":"2024-09-10T19:26:43.198886+0000","last_clean":"2024-09-10T19:26:43.198886+0000","last_became_active":"2024-09-10T19:24:23.908614+0000","last_became_peered":"2024-09-10T19:24:23.908614+0000","last_unstale":"2024-09-10T19:26:43.198886+0000","last_undegraded":"2024-09-10T19:26:43.198886+0000","last_fullsized":"2024-09-10T19:26:43.198886+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'1","last_scrub_stamp":"2024-09-10T19:26:26.891431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:26:26.891431+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:27:30.232324+0000","scrub_duration":0.0037677470000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":874,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"237'15","reported_seq":435,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-09-10T19:26:43.198304+0000","last_change":"2024-09-10T19:25:20.871177+0000","last_active":"2024-09-10T19:26:43.198304+0000","last_peered":"2024-09-10T19:26:43.198304+0000","last_clean":"2024-09-10T19:26:43.198304+0000","last_became_active":"2024-09-10T19:24:01.976339+0000","last_became_peered":"2024-09-10T19:24:01.976339+0000","last_unstale":"2024-09-10T19:26:43.198304+0000","last_undegraded":"2024-09-10T19:26:43.198304+0000","last_fullsized":"2024-09-10T19:26:43.198304+0000","mapping_epoch":292,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":293,"parent":"0.0","parent_split_bits":0,"last_scrub":"237'15","last_scrub_stamp":"2024-09-10T19:25:20.871034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:25:20.871034+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.011611994,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":58,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-09-10T19:26:43.201379+0000","last_change":"2024-09-10T19:24:53.986685+0000","last_active":"2024-09-10T19:26:43.201379+0000","last_peered":"2024-09-10T19:26:43.201379+0000","last_clean":"2024-09-10T19:26:43.201379+0000","last_became_active":"2024-09-10T19:24:44.895513+0000","last_became_peered":"2024-09-10T19:24:44.895513+0000","last_unstale":"2024-09-10T19:26:43.201379+0000","last_undegraded":"2024-09-10T19:26:43.201379+0000","last_fullsized":"2024-09-10T19:26:43.201379+0000","mapping_epoch":314,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":315,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:24:53.986597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:24:53.986597+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.010150255,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,7],"acting":[6,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":389,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-09-10T19:26:43.198403+0000","last_change":"2024-09-10T19:25:31.919853+0000","last_active":"2024-09-10T19:26:43.198403+0000","last_peered":"2024-09-10T19:26:43.198403+0000","last_clean":"2024-09-10T19:26:43.198403+0000","last_became_active":"2024-09-10T19:24:01.971921+0000","last_became_peered":"2024-09-10T19:24:01.971921+0000","last_unstale":"2024-09-10T19:26:43.198403+0000","last_undegraded":"2024-09-10T19:26:43.198403+0000","last_fullsized":"2024-09-10T19:26:43.198403+0000","mapping_epoch":292,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":293,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:25:31.919782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:25:31.919782+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0099438370000000005,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":380,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-09-10T19:26:43.202486+0000","last_change":"2024-09-10T19:26:35.582648+0000","last_active":"2024-09-10T19:26:43.202486+0000","last_peered":"2024-09-10T19:26:43.202486+0000","last_clean":"2024-09-10T19:26:43.202486+0000","last_became_active":"2024-09-10T19:24:23.902270+0000","last_became_peered":"2024-09-10T19:24:23.902270+0000","last_unstale":"2024-09-10T19:26:43.202486+0000","last_undegraded":"2024-09-10T19:26:43.202486+0000","last_fullsized":"2024-09-10T19:26:43.202486+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:26:35.582561+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:26:35.582561+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:27:37.081154+0000","scrub_duration":0.013149586,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1b","version":"237'7","reported_seq":431,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-09-10T19:26:43.197905+0000","last_change":"2024-09-10T19:26:22.847139+0000","last_active":"2024-09-10T19:26:43.197905+0000","last_peered":"2024-09-10T19:26:43.197905+0000","last_clean":"2024-09-10T19:26:43.197905+0000","last_became_active":"2024-09-10T19:24:01.974813+0000","last_became_peered":"2024-09-10T19:24:01.974813+0000","last_unstale":"2024-09-10T19:26:43.197905+0000","last_undegraded":"2024-09-10T19:26:43.197905+0000","last_fullsized":"2024-09-10T19:26:43.197905+0000","mapping_epoch":292,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":293,"parent":"0.0","parent_split_bits":0,"last_scrub":"237'7","last_scrub_stamp":"2024-09-10T19:26:22.847061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:26:22.847061+0000","objects_scrubbed":2,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:27:33.920907+0000","scrub_duration":0.001667078,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":10,"num_write":8,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":383,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-09-10T19:26:43.201713+0000","last_change":"2024-09-10T19:26:15.175676+0000","last_active":"2024-09-10T19:26:43.201713+0000","last_peered":"2024-09-10T19:26:43.201713+0000","last_clean":"2024-09-10T19:26:43.201713+0000","last_became_active":"2024-09-10T19:24:01.976587+0000","last_became_peered":"2024-09-10T19:24:01.976587+0000","last_unstale":"2024-09-10T19:26:43.201713+0000","last_undegraded":"2024-09-10T19:26:43.201713+0000","last_fullsized":"2024-09-10T19:26:43.201713+0000","mapping_epoch":292,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":293,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:26:15.175601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:26:15.175601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:27:36.641175+0000","scrub_duration":0.013129844,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":381,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-09-10T19:26:43.210147+0000","last_change":"2024-09-10T19:24:07.659325+0000","last_active":"2024-09-10T19:26:43.210147+0000","last_peered":"2024-09-10T19:26:43.210147+0000","last_clean":"2024-09-10T19:26:43.210147+0000","last_became_active":"2024-09-10T19:24:01.970118+0000","last_became_peered":"2024-09-10T19:24:01.970118+0000","last_unstale":"2024-09-10T19:26:43.210147+0000","last_undegraded":"2024-09-10T19:26:43.210147+0000","last_fullsized":"2024-09-10T19:26:43.210147+0000","mapping_epoch":292,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":293,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:24:07.659224+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:24:07.659224+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001912307,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":373,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-09-10T19:26:43.203159+0000","last_change":"2024-09-10T19:24:53.823606+0000","last_active":"2024-09-10T19:26:43.203159+0000","last_peered":"2024-09-10T19:26:43.203159+0000","last_clean":"2024-09-10T19:26:43.203159+0000","last_became_active":"2024-09-10T19:16:24.858800+0000","last_became_peered":"2024-09-10T19:16:24.858800+0000","last_unstale":"2024-09-10T19:26:43.203159+0000","last_undegraded":"2024-09-10T19:26:43.203159+0000","last_fullsized":"2024-09-10T19:26:43.203159+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:24:53.823351+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:24:53.823351+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0022636959999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1a","version":"237'15","reported_seq":395,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-09-10T19:26:43.198572+0000","last_change":"2024-09-10T19:25:14.879733+0000","last_active":"2024-09-10T19:26:43.198572+0000","last_peered":"2024-09-10T19:26:43.198572+0000","last_clean":"2024-09-10T19:26:43.198572+0000","last_became_active":"2024-09-10T19:24:23.891213+0000","last_became_peered":"2024-09-10T19:24:23.891213+0000","last_unstale":"2024-09-10T19:26:43.198572+0000","last_undegraded":"2024-09-10T19:26:43.198572+0000","last_fullsized":"2024-09-10T19:26:43.198572+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"237'15","last_scrub_stamp":"2024-09-10T19:25:14.879649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:25:14.879649+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0026001190000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":366,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-09-10T19:26:43.213266+0000","last_change":"2024-09-10T19:25:13.751512+0000","last_active":"2024-09-10T19:26:43.213266+0000","last_peered":"2024-09-10T19:26:43.213266+0000","last_clean":"2024-09-10T19:26:43.213266+0000","last_became_active":"2024-09-10T19:24:23.907653+0000","last_became_peered":"2024-09-10T19:24:23.907653+0000","last_unstale":"2024-09-10T19:26:43.213266+0000","last_undegraded":"2024-09-10T19:26:43.213266+0000","last_fullsized":"2024-09-10T19:26:43.213266+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:25:13.751431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:25:13.751431+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0091990130000000007,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":338,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-09-10T19:26:38.186188+0000","last_change":"2024-09-10T19:26:07.493741+0000","last_active":"2024-09-10T19:26:38.186188+0000","last_peered":"2024-09-10T19:26:38.186188+0000","last_clean":"2024-09-10T19:26:38.186188+0000","last_became_active":"2024-09-10T19:24:24.907431+0000","last_became_peered":"2024-09-10T19:24:24.907431+0000","last_unstale":"2024-09-10T19:26:38.186188+0000","last_undegraded":"2024-09-10T19:26:38.186188+0000","last_fullsized":"2024-09-10T19:26:38.186188+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:26:07.493656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:26:07.493656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:27:10.117261+0000","scrub_duration":0.0071789510000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.3","version":"353'12935","reported_seq":13001,"reported_epoch":353,"state":"active+clean","last_fresh":"2024-09-10T19:26:55.506158+0000","last_change":"2024-09-10T19:24:09.045117+0000","last_active":"2024-09-10T19:26:55.506158+0000","last_peered":"2024-09-10T19:26:55.506158+0000","last_clean":"2024-09-10T19:26:55.506158+0000","last_became_active":"2024-09-10T19:24:09.043872+0000","last_became_peered":"2024-09-10T19:24:09.043872+0000","last_unstale":"2024-09-10T19:26:55.506158+0000","last_undegraded":"2024-09-10T19:26:55.506158+0000","last_fullsized":"2024-09-10T19:26:55.506158+0000","mapping_epoch":295,"log_start":"341'8500","ondisk_log_start":"341'8500","created":295,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:24:08.023080+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:24:08.023080+0000","last_clean_scrub_stamp":"2024-09-10T19:24:08.023080+0000","objects_scrubbed":0,"log_size":4435,"log_dups_size":0,"ondisk_log_size":4435,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":152764416,"num_objects":2331,"num_object_clones":0,"num_object_copies":6993,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2331,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12935,"num_write_kb":488512,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.8","version":"74'79","reported_seq":143,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-09-10T19:26:43.210466+0000","last_change":"2024-09-10T19:25:17.772519+0000","last_active":"2024-09-10T19:26:43.210466+0000","last_peered":"2024-09-10T19:26:43.210466+0000","last_clean":"2024-09-10T19:26:43.210466+0000","last_became_active":"2024-09-10T19:20:30.839050+0000","last_became_peered":"2024-09-10T19:20:30.839050+0000","last_unstale":"2024-09-10T19:26:43.210466+0000","last_undegraded":"2024-09-10T19:26:43.210466+0000","last_fullsized":"2024-09-10T19:26:43.210466+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":4,"last_scrub":"74'79","last_scrub_stamp":"2024-09-10T19:25:17.772431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:08:00.699315+0000","last_clean_scrub_stamp":"2024-09-10T19:25:17.772431+0000","objects_scrubbed":1,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.015350954,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.d","version":"237'31","reported_seq":507,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-09-10T19:26:43.199381+0000","last_change":"2024-09-10T19:26:27.880510+0000","last_active":"2024-09-10T19:26:43.199381+0000","last_peered":"2024-09-10T19:26:43.199381+0000","last_clean":"2024-09-10T19:26:43.199381+0000","last_became_active":"2024-09-10T19:24:01.975691+0000","last_became_peered":"2024-09-10T19:24:01.975691+0000","last_unstale":"2024-09-10T19:26:43.199381+0000","last_undegraded":"2024-09-10T19:26:43.199381+0000","last_fullsized":"2024-09-10T19:26:43.199381+0000","mapping_epoch":292,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":293,"parent":"0.0","parent_split_bits":0,"last_scrub":"237'31","last_scrub_stamp":"2024-09-10T19:26:27.880373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:26:27.880373+0000","objects_scrubbed":8,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:27:28.551207+0000","scrub_duration":0.0086552290000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":50,"num_read_kb":40,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,6],"acting":[4,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":373,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-09-10T19:26:43.202613+0000","last_change":"2024-09-10T19:26:08.089251+0000","last_active":"2024-09-10T19:26:43.202613+0000","last_peered":"2024-09-10T19:26:43.202613+0000","last_clean":"2024-09-10T19:26:43.202613+0000","last_became_active":"2024-09-10T19:10:07.691119+0000","last_became_peered":"2024-09-10T19:10:07.691119+0000","last_unstale":"2024-09-10T19:26:43.202613+0000","last_undegraded":"2024-09-10T19:26:43.202613+0000","last_fullsized":"2024-09-10T19:26:43.202613+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:26:08.089172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:16:39.716674+0000","last_clean_scrub_stamp":"2024-09-10T19:26:08.089172+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:27:16.557379+0000","scrub_duration":0.0047132800000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":94,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-09-10T19:26:43.192986+0000","last_change":"2024-09-10T19:24:46.462624+0000","last_active":"2024-09-10T19:26:43.192986+0000","last_peered":"2024-09-10T19:26:43.192986+0000","last_clean":"2024-09-10T19:26:43.192986+0000","last_became_active":"2024-09-10T19:24:24.886479+0000","last_became_peered":"2024-09-10T19:24:24.886479+0000","last_unstale":"2024-09-10T19:26:43.192986+0000","last_undegraded":"2024-09-10T19:26:43.192986+0000","last_fullsized":"2024-09-10T19:26:43.192986+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:24:46.462536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:24:46.462536+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.005382062,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":241,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-09-10T19:26:43.198897+0000","last_change":"2024-09-10T19:26:08.894794+0000","last_active":"2024-09-10T19:26:43.198897+0000","last_peered":"2024-09-10T19:26:43.198897+0000","last_clean":"2024-09-10T19:26:43.198897+0000","last_became_active":"2024-09-10T19:16:24.857843+0000","last_became_peered":"2024-09-10T19:16:24.857843+0000","last_unstale":"2024-09-10T19:26:43.198897+0000","last_undegraded":"2024-09-10T19:26:43.198897+0000","last_fullsized":"2024-09-10T19:26:43.198897+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:26:08.894702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:26:08.894702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:27:33.178000+0000","scrub_duration":0.010734235,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.2","version":"352'12986","reported_seq":14290,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-09-10T19:26:52.207610+0000","last_change":"2024-09-10T19:26:37.419041+0000","last_active":"2024-09-10T19:26:52.207610+0000","last_peered":"2024-09-10T19:26:52.207610+0000","last_clean":"2024-09-10T19:26:52.207610+0000","last_became_active":"2024-09-10T19:24:27.932777+0000","last_became_peered":"2024-09-10T19:24:27.932777+0000","last_unstale":"2024-09-10T19:26:52.207610+0000","last_undegraded":"2024-09-10T19:26:52.207610+0000","last_fullsized":"2024-09-10T19:26:52.207610+0000","mapping_epoch":305,"log_start":"341'8200","ondisk_log_start":"341'8200","created":295,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"350'11705","last_scrub_stamp":"2024-09-10T19:26:37.418937+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:24:08.023080+0000","last_clean_scrub_stamp":"2024-09-10T19:26:37.418937+0000","objects_scrubbed":3909,"log_size":4786,"log_dups_size":0,"ondisk_log_size":4786,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:28:07.315479+0000","scrub_duration":1.013252139,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":169345024,"num_objects":2584,"num_object_clones":0,"num_object_copies":7752,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2584,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12986,"num_write_kb":498240,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":526,"num_bytes_recovered":34471936,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"74'79","reported_seq":428,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-09-10T19:26:43.206172+0000","last_change":"2024-09-10T19:25:12.867805+0000","last_active":"2024-09-10T19:26:43.206172+0000","last_peered":"2024-09-10T19:26:43.206172+0000","last_clean":"2024-09-10T19:26:43.206172+0000","last_became_active":"2024-09-10T19:20:30.844108+0000","last_became_peered":"2024-09-10T19:20:30.844108+0000","last_unstale":"2024-09-10T19:26:43.206172+0000","last_undegraded":"2024-09-10T19:26:43.206172+0000","last_fullsized":"2024-09-10T19:26:43.206172+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":4,"last_scrub":"74'79","last_scrub_stamp":"2024-09-10T19:25:12.867565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:08:00.699315+0000","last_clean_scrub_stamp":"2024-09-10T19:25:12.867565+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.022682969000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.c","version":"237'20","reported_seq":403,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-09-10T19:26:43.199580+0000","last_change":"2024-09-10T19:25:16.888274+0000","last_active":"2024-09-10T19:26:43.199580+0000","last_peered":"2024-09-10T19:26:43.199580+0000","last_clean":"2024-09-10T19:26:43.199580+0000","last_became_active":"2024-09-10T19:10:09.688321+0000","last_became_peered":"2024-09-10T19:10:09.688321+0000","last_unstale":"2024-09-10T19:26:43.199580+0000","last_undegraded":"2024-09-10T19:26:43.199580+0000","last_fullsized":"2024-09-10T19:26:43.199580+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"237'20","last_scrub_stamp":"2024-09-10T19:25:16.887901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:25:16.887901+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.012411174000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":80,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-09-10T19:26:43.194999+0000","last_change":"2024-09-10T19:25:29.750938+0000","last_active":"2024-09-10T19:26:43.194999+0000","last_peered":"2024-09-10T19:26:43.194999+0000","last_clean":"2024-09-10T19:26:43.194999+0000","last_became_active":"2024-09-10T19:24:01.973848+0000","last_became_peered":"2024-09-10T19:24:01.973848+0000","last_unstale":"2024-09-10T19:26:43.194999+0000","last_undegraded":"2024-09-10T19:26:43.194999+0000","last_fullsized":"2024-09-10T19:26:43.194999+0000","mapping_epoch":292,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":293,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:25:29.750877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:25:29.750877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001313533,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.d","version":"303'12","reported_seq":446,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-09-10T19:26:50.928776+0000","last_change":"2024-09-10T19:24:36.790767+0000","last_active":"2024-09-10T19:26:50.928776+0000","last_peered":"2024-09-10T19:26:50.928776+0000","last_clean":"2024-09-10T19:26:50.928776+0000","last_became_active":"2024-09-10T19:24:23.906819+0000","last_became_peered":"2024-09-10T19:24:23.906819+0000","last_unstale":"2024-09-10T19:26:50.928776+0000","last_undegraded":"2024-09-10T19:26:50.928776+0000","last_fullsized":"2024-09-10T19:26:50.928776+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"303'12","last_scrub_stamp":"2024-09-10T19:24:36.790665+0000","last_deep_scrub":"80'9","last_deep_scrub_stamp":"2024-09-10T19:11:25.156465+0000","last_clean_scrub_stamp":"2024-09-10T19:24:36.790665+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0028602430000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,0],"acting":[7,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":399,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-09-10T19:26:43.199727+0000","last_change":"2024-09-10T19:26:13.829815+0000","last_active":"2024-09-10T19:26:43.199727+0000","last_peered":"2024-09-10T19:26:43.199727+0000","last_clean":"2024-09-10T19:26:43.199727+0000","last_became_active":"2024-09-10T19:24:01.972593+0000","last_became_peered":"2024-09-10T19:24:01.972593+0000","last_unstale":"2024-09-10T19:26:43.199727+0000","last_undegraded":"2024-09-10T19:26:43.199727+0000","last_fullsized":"2024-09-10T19:26:43.199727+0000","mapping_epoch":292,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":293,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:26:13.829736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:26:13.829736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:27:16.644048+0000","scrub_duration":0.025408242000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,5],"acting":[4,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.1","version":"353'13284","reported_seq":14258,"reported_epoch":353,"state":"active+clean","last_fresh":"2024-09-10T19:26:55.158222+0000","last_change":"2024-09-10T19:24:29.878172+0000","last_active":"2024-09-10T19:26:55.158222+0000","last_peered":"2024-09-10T19:26:55.158222+0000","last_clean":"2024-09-10T19:26:55.158222+0000","last_became_active":"2024-09-10T19:24:29.875287+0000","last_became_peered":"2024-09-10T19:24:29.875287+0000","last_unstale":"2024-09-10T19:26:55.158222+0000","last_undegraded":"2024-09-10T19:26:55.158222+0000","last_fullsized":"2024-09-10T19:26:55.158222+0000","mapping_epoch":307,"log_start":"342'9000","ondisk_log_start":"342'9000","created":295,"last_epoch_clean":308,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:24:08.023080+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:24:08.023080+0000","last_clean_scrub_stamp":"2024-09-10T19:24:08.023080+0000","objects_scrubbed":0,"log_size":4284,"log_dups_size":0,"ondisk_log_size":4284,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":157548544,"num_objects":2404,"num_object_clones":0,"num_object_copies":7212,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2404,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13284,"num_write_kb":502016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":446,"num_bytes_recovered":29229056,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.a","version":"74'79","reported_seq":138,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-09-10T19:26:43.209599+0000","last_change":"2024-09-10T19:25:18.795091+0000","last_active":"2024-09-10T19:26:43.209599+0000","last_peered":"2024-09-10T19:26:43.209599+0000","last_clean":"2024-09-10T19:26:43.209599+0000","last_became_active":"2024-09-10T19:20:30.844773+0000","last_became_peered":"2024-09-10T19:20:30.844773+0000","last_unstale":"2024-09-10T19:26:43.209599+0000","last_undegraded":"2024-09-10T19:26:43.209599+0000","last_fullsized":"2024-09-10T19:26:43.209599+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":4,"last_scrub":"74'79","last_scrub_stamp":"2024-09-10T19:25:18.795003+0000","last_deep_scrub":"74'79","last_deep_scrub_stamp":"2024-09-10T19:21:53.326433+0000","last_clean_scrub_stamp":"2024-09-10T19:25:18.795003+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0047392160000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,4],"acting":[5,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.f","version":"237'27","reported_seq":96,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-09-10T19:26:43.194970+0000","last_change":"2024-09-10T19:26:17.458249+0000","last_active":"2024-09-10T19:26:43.194970+0000","last_peered":"2024-09-10T19:26:43.194970+0000","last_clean":"2024-09-10T19:26:43.194970+0000","last_became_active":"2024-09-10T19:24:01.979187+0000","last_became_peered":"2024-09-10T19:24:01.979187+0000","last_unstale":"2024-09-10T19:26:43.194970+0000","last_undegraded":"2024-09-10T19:26:43.194970+0000","last_fullsized":"2024-09-10T19:26:43.194970+0000","mapping_epoch":292,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":293,"parent":"0.0","parent_split_bits":0,"last_scrub":"237'27","last_scrub_stamp":"2024-09-10T19:26:17.458177+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:26:17.458177+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:27:23.370611+0000","scrub_duration":0.0044892370000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":80,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-09-10T19:26:43.192919+0000","last_change":"2024-09-10T19:26:08.391803+0000","last_active":"2024-09-10T19:26:43.192919+0000","last_peered":"2024-09-10T19:26:43.192919+0000","last_clean":"2024-09-10T19:26:43.192919+0000","last_became_active":"2024-09-10T19:24:01.974773+0000","last_became_peered":"2024-09-10T19:24:01.974773+0000","last_unstale":"2024-09-10T19:26:43.192919+0000","last_undegraded":"2024-09-10T19:26:43.192919+0000","last_fullsized":"2024-09-10T19:26:43.192919+0000","mapping_epoch":292,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":293,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:26:08.391730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:26:08.391730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:27:18.740314+0000","scrub_duration":0.0020244629999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.e","version":"303'10","reported_seq":569,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-09-10T19:26:50.928690+0000","last_change":"2024-09-10T19:24:41.682443+0000","last_active":"2024-09-10T19:26:50.928690+0000","last_peered":"2024-09-10T19:26:50.928690+0000","last_clean":"2024-09-10T19:26:50.928690+0000","last_became_active":"2024-09-10T19:24:23.908760+0000","last_became_peered":"2024-09-10T19:24:23.908760+0000","last_unstale":"2024-09-10T19:26:50.928690+0000","last_undegraded":"2024-09-10T19:26:50.928690+0000","last_fullsized":"2024-09-10T19:26:50.928690+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"303'10","last_scrub_stamp":"2024-09-10T19:24:41.682347+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:24:41.682347+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0099881439999999991,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":405,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-09-10T19:26:43.212681+0000","last_change":"2024-09-10T19:26:34.847681+0000","last_active":"2024-09-10T19:26:43.212681+0000","last_peered":"2024-09-10T19:26:43.212681+0000","last_clean":"2024-09-10T19:26:43.212681+0000","last_became_active":"2024-09-10T19:24:24.917109+0000","last_became_peered":"2024-09-10T19:24:24.917109+0000","last_unstale":"2024-09-10T19:26:43.212681+0000","last_undegraded":"2024-09-10T19:26:43.212681+0000","last_fullsized":"2024-09-10T19:26:43.212681+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:26:34.847602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:26:34.847602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:27:54.686477+0000","scrub_duration":0.0034671599999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.0","version":"353'12761","reported_seq":13074,"reported_epoch":353,"state":"active+clean","last_fresh":"2024-09-10T19:26:54.670637+0000","last_change":"2024-09-10T19:25:31.237539+0000","last_active":"2024-09-10T19:26:54.670637+0000","last_peered":"2024-09-10T19:26:54.670637+0000","last_clean":"2024-09-10T19:26:54.670637+0000","last_became_active":"2024-09-10T19:24:09.046656+0000","last_became_peered":"2024-09-10T19:24:09.046656+0000","last_unstale":"2024-09-10T19:26:54.670637+0000","last_undegraded":"2024-09-10T19:26:54.670637+0000","last_fullsized":"2024-09-10T19:26:54.670637+0000","mapping_epoch":295,"log_start":"341'8000","ondisk_log_start":"341'8000","created":295,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"333'6034","last_scrub_stamp":"2024-09-10T19:25:31.237447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:24:08.023080+0000","last_clean_scrub_stamp":"2024-09-10T19:25:31.237447+0000","objects_scrubbed":5911,"log_size":4761,"log_dups_size":0,"ondisk_log_size":4761,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"queued for scrub","scrub_duration":2.954390187,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":157614080,"num_objects":2405,"num_object_clones":0,"num_object_copies":7215,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2405,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12761,"num_write_kb":485312,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.e","version":"237'24","reported_seq":472,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-09-10T19:26:43.199956+0000","last_change":"2024-09-10T19:25:22.898084+0000","last_active":"2024-09-10T19:26:43.199956+0000","last_peered":"2024-09-10T19:26:43.199956+0000","last_clean":"2024-09-10T19:26:43.199956+0000","last_became_active":"2024-09-10T19:24:44.894167+0000","last_became_peered":"2024-09-10T19:24:44.894167+0000","last_unstale":"2024-09-10T19:26:43.199956+0000","last_undegraded":"2024-09-10T19:26:43.199956+0000","last_fullsized":"2024-09-10T19:26:43.199956+0000","mapping_epoch":314,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":315,"parent":"0.0","parent_split_bits":0,"last_scrub":"237'24","last_scrub_stamp":"2024-09-10T19:25:22.897993+0000","last_deep_scrub":"84'16","last_deep_scrub_stamp":"2024-09-10T19:11:56.157975+0000","last_clean_scrub_stamp":"2024-09-10T19:25:22.897993+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.021319980999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":44,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-09-10T19:26:43.197979+0000","last_change":"2024-09-10T19:24:53.756744+0000","last_active":"2024-09-10T19:26:43.197979+0000","last_peered":"2024-09-10T19:26:43.197979+0000","last_clean":"2024-09-10T19:26:43.197979+0000","last_became_active":"2024-09-10T19:24:53.756220+0000","last_became_peered":"2024-09-10T19:24:53.756220+0000","last_unstale":"2024-09-10T19:26:43.197979+0000","last_undegraded":"2024-09-10T19:26:43.197979+0000","last_fullsized":"2024-09-10T19:26:43.197979+0000","mapping_epoch":318,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":319,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:24:13.636249+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:24:13.636249+0000","last_clean_scrub_stamp":"2024-09-10T19:24:13.636249+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001732363,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,3],"acting":[4,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":349,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-09-10T19:26:43.216194+0000","last_change":"2024-09-10T19:25:29.715973+0000","last_active":"2024-09-10T19:26:43.216194+0000","last_peered":"2024-09-10T19:26:43.216194+0000","last_clean":"2024-09-10T19:26:43.216194+0000","last_became_active":"2024-09-10T19:10:11.692215+0000","last_became_peered":"2024-09-10T19:10:11.692215+0000","last_unstale":"2024-09-10T19:26:43.216194+0000","last_undegraded":"2024-09-10T19:26:43.216194+0000","last_fullsized":"2024-09-10T19:26:43.216194+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:25:29.715888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:25:29.715888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:26:56.816598+0000","scrub_duration":0.024627837999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":347,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-09-10T19:26:43.199008+0000","last_change":"2024-09-10T19:25:13.722998+0000","last_active":"2024-09-10T19:26:43.199008+0000","last_peered":"2024-09-10T19:26:43.199008+0000","last_clean":"2024-09-10T19:26:43.199008+0000","last_became_active":"2024-09-10T19:10:14.053736+0000","last_became_peered":"2024-09-10T19:10:14.053736+0000","last_unstale":"2024-09-10T19:26:43.199008+0000","last_undegraded":"2024-09-10T19:26:43.199008+0000","last_fullsized":"2024-09-10T19:26:43.199008+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:25:13.722943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:25:13.722943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0024886520000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.f","version":"353'12929","reported_seq":13216,"reported_epoch":353,"state":"active+clean","last_fresh":"2024-09-10T19:26:54.686786+0000","last_change":"2024-09-10T19:25:21.021707+0000","last_active":"2024-09-10T19:26:54.686786+0000","last_peered":"2024-09-10T19:26:54.686786+0000","last_clean":"2024-09-10T19:26:54.686786+0000","last_became_active":"2024-09-10T19:24:09.047213+0000","last_became_peered":"2024-09-10T19:24:09.047213+0000","last_unstale":"2024-09-10T19:26:54.686786+0000","last_undegraded":"2024-09-10T19:26:54.686786+0000","last_fullsized":"2024-09-10T19:26:54.686786+0000","mapping_epoch":295,"log_start":"341'8200","ondisk_log_start":"341'8200","created":295,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"331'5382","last_scrub_stamp":"2024-09-10T19:25:21.021381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:24:08.023080+0000","last_clean_scrub_stamp":"2024-09-10T19:25:21.021381+0000","objects_scrubbed":5280,"log_size":4729,"log_dups_size":0,"ondisk_log_size":4729,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"queued for scrub","scrub_duration":2.7958712999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":158269440,"num_objects":2415,"num_object_clones":0,"num_object_copies":7245,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2415,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12929,"num_write_kb":491008,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,4],"acting":[6,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.4","version":"317'80","reported_seq":154,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-09-10T19:26:43.198390+0000","last_change":"2024-09-10T19:26:34.932583+0000","last_active":"2024-09-10T19:26:43.198390+0000","last_peered":"2024-09-10T19:26:43.198390+0000","last_clean":"2024-09-10T19:26:43.198390+0000","last_became_active":"2024-09-10T19:20:30.838441+0000","last_became_peered":"2024-09-10T19:20:30.838441+0000","last_unstale":"2024-09-10T19:26:43.198390+0000","last_undegraded":"2024-09-10T19:26:43.198390+0000","last_fullsized":"2024-09-10T19:26:43.198390+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":3,"last_scrub":"317'80","last_scrub_stamp":"2024-09-10T19:26:34.932490+0000","last_deep_scrub":"317'80","last_deep_scrub_stamp":"2024-09-10T19:24:51.540456+0000","last_clean_scrub_stamp":"2024-09-10T19:26:34.932490+0000","objects_scrubbed":1,"log_size":80,"log_dups_size":0,"ondisk_log_size":80,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:27:38.776542+0000","scrub_duration":0.010119784999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"265'35","reported_seq":295,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-09-10T19:26:43.198556+0000","last_change":"2024-09-10T19:25:29.871366+0000","last_active":"2024-09-10T19:26:43.198556+0000","last_peered":"2024-09-10T19:26:43.198556+0000","last_clean":"2024-09-10T19:26:43.198556+0000","last_became_active":"2024-09-10T19:16:24.852821+0000","last_became_peered":"2024-09-10T19:16:24.852821+0000","last_unstale":"2024-09-10T19:26:43.198556+0000","last_undegraded":"2024-09-10T19:26:43.198556+0000","last_fullsized":"2024-09-10T19:26:43.198556+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"265'35","last_scrub_stamp":"2024-09-10T19:25:29.871291+0000","last_deep_scrub":"265'35","last_deep_scrub_stamp":"2024-09-10T19:25:29.871291+0000","last_clean_scrub_stamp":"2024-09-10T19:25:29.871291+0000","objects_scrubbed":7,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0058405009999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":42,"num_read_kb":35,"num_write":28,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,7],"acting":[4,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":383,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-09-10T19:26:43.204184+0000","last_change":"2024-09-10T19:24:49.962031+0000","last_active":"2024-09-10T19:26:43.204184+0000","last_peered":"2024-09-10T19:26:43.204184+0000","last_clean":"2024-09-10T19:26:43.204184+0000","last_became_active":"2024-09-10T19:16:24.855861+0000","last_became_peered":"2024-09-10T19:16:24.855861+0000","last_unstale":"2024-09-10T19:26:43.204184+0000","last_undegraded":"2024-09-10T19:26:43.204184+0000","last_fullsized":"2024-09-10T19:26:43.204184+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:24:49.961960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:24:49.961960+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.015143467000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.0","version":"294'12","reported_seq":602,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-09-10T19:26:50.928393+0000","last_change":"2024-09-10T19:26:18.869349+0000","last_active":"2024-09-10T19:26:50.928393+0000","last_peered":"2024-09-10T19:26:50.928393+0000","last_clean":"2024-09-10T19:26:50.928393+0000","last_became_active":"2024-09-10T19:24:01.972645+0000","last_became_peered":"2024-09-10T19:24:01.972645+0000","last_unstale":"2024-09-10T19:26:50.928393+0000","last_undegraded":"2024-09-10T19:26:50.928393+0000","last_fullsized":"2024-09-10T19:26:50.928393+0000","mapping_epoch":292,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":293,"parent":"0.0","parent_split_bits":0,"last_scrub":"294'12","last_scrub_stamp":"2024-09-10T19:26:18.869259+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:26:18.869259+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:27:30.361030+0000","scrub_duration":0.0029728440000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":374,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-09-10T19:26:43.202319+0000","last_change":"2024-09-10T19:26:25.368855+0000","last_active":"2024-09-10T19:26:43.202319+0000","last_peered":"2024-09-10T19:26:43.202319+0000","last_clean":"2024-09-10T19:26:43.202319+0000","last_became_active":"2024-09-10T19:24:01.974621+0000","last_became_peered":"2024-09-10T19:24:01.974621+0000","last_unstale":"2024-09-10T19:26:43.202319+0000","last_undegraded":"2024-09-10T19:26:43.202319+0000","last_fullsized":"2024-09-10T19:26:43.202319+0000","mapping_epoch":292,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":293,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:26:25.368770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:11:41.073833+0000","last_clean_scrub_stamp":"2024-09-10T19:26:25.368770+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:27:50.731053+0000","scrub_duration":0.0030199950000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.e","version":"353'12757","reported_seq":12823,"reported_epoch":353,"state":"active+clean","last_fresh":"2024-09-10T19:26:55.157535+0000","last_change":"2024-09-10T19:24:09.046030+0000","last_active":"2024-09-10T19:26:55.157535+0000","last_peered":"2024-09-10T19:26:55.157535+0000","last_clean":"2024-09-10T19:26:55.157535+0000","last_became_active":"2024-09-10T19:24:09.045762+0000","last_became_peered":"2024-09-10T19:24:09.045762+0000","last_unstale":"2024-09-10T19:26:55.157535+0000","last_undegraded":"2024-09-10T19:26:55.157535+0000","last_fullsized":"2024-09-10T19:26:55.157535+0000","mapping_epoch":295,"log_start":"341'8500","ondisk_log_start":"341'8500","created":295,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:24:08.023080+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:24:08.023080+0000","last_clean_scrub_stamp":"2024-09-10T19:24:08.023080+0000","objects_scrubbed":0,"log_size":4257,"log_dups_size":0,"ondisk_log_size":4257,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":153812992,"num_objects":2347,"num_object_clones":0,"num_object_copies":7041,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2347,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12757,"num_write_kb":483328,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.5","version":"74'79","reported_seq":148,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-09-10T19:26:43.198689+0000","last_change":"2024-09-10T19:25:15.865396+0000","last_active":"2024-09-10T19:26:43.198689+0000","last_peered":"2024-09-10T19:26:43.198689+0000","last_clean":"2024-09-10T19:26:43.198689+0000","last_became_active":"2024-09-10T19:20:30.838556+0000","last_became_peered":"2024-09-10T19:20:30.838556+0000","last_unstale":"2024-09-10T19:26:43.198689+0000","last_undegraded":"2024-09-10T19:26:43.198689+0000","last_fullsized":"2024-09-10T19:26:43.198689+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":3,"last_scrub":"74'79","last_scrub_stamp":"2024-09-10T19:25:15.865263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:08:00.699315+0000","last_clean_scrub_stamp":"2024-09-10T19:25:15.865263+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0027887379999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"237'30","reported_seq":408,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-09-10T19:26:43.197633+0000","last_change":"2024-09-10T19:24:37.476248+0000","last_active":"2024-09-10T19:26:43.197633+0000","last_peered":"2024-09-10T19:26:43.197633+0000","last_clean":"2024-09-10T19:26:43.197633+0000","last_became_active":"2024-09-10T19:24:23.903784+0000","last_became_peered":"2024-09-10T19:24:23.903784+0000","last_unstale":"2024-09-10T19:26:43.197633+0000","last_undegraded":"2024-09-10T19:26:43.197633+0000","last_fullsized":"2024-09-10T19:26:43.197633+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"237'30","last_scrub_stamp":"2024-09-10T19:24:37.476174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:24:37.476174+0000","objects_scrubbed":8,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0032561880000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":52,"num_read_kb":40,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":80,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-09-10T19:26:43.195201+0000","last_change":"2024-09-10T19:25:14.621455+0000","last_active":"2024-09-10T19:26:43.195201+0000","last_peered":"2024-09-10T19:26:43.195201+0000","last_clean":"2024-09-10T19:26:43.195201+0000","last_became_active":"2024-09-10T19:24:01.978995+0000","last_became_peered":"2024-09-10T19:24:01.978995+0000","last_unstale":"2024-09-10T19:26:43.195201+0000","last_undegraded":"2024-09-10T19:26:43.195201+0000","last_fullsized":"2024-09-10T19:26:43.195201+0000","mapping_epoch":292,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":293,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:25:14.621399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:17:26.374376+0000","last_clean_scrub_stamp":"2024-09-10T19:25:14.621399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0050623270000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":368,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-09-10T19:26:43.198796+0000","last_change":"2024-09-10T19:26:24.897597+0000","last_active":"2024-09-10T19:26:43.198796+0000","last_peered":"2024-09-10T19:26:43.198796+0000","last_clean":"2024-09-10T19:26:43.198796+0000","last_became_active":"2024-09-10T19:10:11.698054+0000","last_became_peered":"2024-09-10T19:10:11.698054+0000","last_unstale":"2024-09-10T19:26:43.198796+0000","last_undegraded":"2024-09-10T19:26:43.198796+0000","last_fullsized":"2024-09-10T19:26:43.198796+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:26:24.897519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:26:24.897519+0000","last_clean_scrub_stamp":"2024-09-10T19:26:24.897519+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:27:42.219567+0000","scrub_duration":0.0035237760000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":366,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-09-10T19:26:43.198546+0000","last_change":"2024-09-10T19:23:55.452716+0000","last_active":"2024-09-10T19:26:43.198546+0000","last_peered":"2024-09-10T19:26:43.198546+0000","last_clean":"2024-09-10T19:26:43.198546+0000","last_became_active":"2024-09-10T19:16:24.848109+0000","last_became_peered":"2024-09-10T19:16:24.848109+0000","last_unstale":"2024-09-10T19:26:43.198546+0000","last_undegraded":"2024-09-10T19:26:43.198546+0000","last_fullsized":"2024-09-10T19:26:43.198546+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:23:55.452654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:23:55.452654+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0018215709999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.d","version":"353'12859","reported_seq":14084,"reported_epoch":353,"state":"active+clean","last_fresh":"2024-09-10T19:26:54.702252+0000","last_change":"2024-09-10T19:26:20.870487+0000","last_active":"2024-09-10T19:26:54.702252+0000","last_peered":"2024-09-10T19:26:54.702252+0000","last_clean":"2024-09-10T19:26:54.702252+0000","last_became_active":"2024-09-10T19:24:28.941711+0000","last_became_peered":"2024-09-10T19:24:28.941711+0000","last_unstale":"2024-09-10T19:26:54.702252+0000","last_undegraded":"2024-09-10T19:26:54.702252+0000","last_fullsized":"2024-09-10T19:26:54.702252+0000","mapping_epoch":306,"log_start":"341'8100","ondisk_log_start":"341'8100","created":295,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":0,"last_scrub":"346'10089","last_scrub_stamp":"2024-09-10T19:26:20.870323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:24:08.023080+0000","last_clean_scrub_stamp":"2024-09-10T19:26:20.870323+0000","objects_scrubbed":5326,"log_size":4759,"log_dups_size":0,"ondisk_log_size":4759,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:27:45.236018+0000","scrub_duration":2.477425953,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":160956440,"num_objects":2457,"num_object_clones":0,"num_object_copies":7371,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2457,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":12859,"num_write_kb":490049,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":456,"num_bytes_recovered":29884416,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.6","version":"74'79","reported_seq":235,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-09-10T19:26:43.202683+0000","last_change":"2024-09-10T19:26:30.421621+0000","last_active":"2024-09-10T19:26:43.202683+0000","last_peered":"2024-09-10T19:26:43.202683+0000","last_clean":"2024-09-10T19:26:43.202683+0000","last_became_active":"2024-09-10T19:24:23.902290+0000","last_became_peered":"2024-09-10T19:24:23.902290+0000","last_unstale":"2024-09-10T19:26:43.202683+0000","last_undegraded":"2024-09-10T19:26:43.202683+0000","last_fullsized":"2024-09-10T19:26:43.202683+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":3,"last_scrub":"74'79","last_scrub_stamp":"2024-09-10T19:26:30.421470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:08:00.699315+0000","last_clean_scrub_stamp":"2024-09-10T19:26:30.421470+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:27:44.391737+0000","scrub_duration":0.0045070300000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"237'37","reported_seq":367,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-09-10T19:26:38.188937+0000","last_change":"2024-09-10T19:26:36.464107+0000","last_active":"2024-09-10T19:26:38.188937+0000","last_peered":"2024-09-10T19:26:38.188937+0000","last_clean":"2024-09-10T19:26:38.188937+0000","last_became_active":"2024-09-10T19:24:23.901836+0000","last_became_peered":"2024-09-10T19:24:23.901836+0000","last_unstale":"2024-09-10T19:26:38.188937+0000","last_undegraded":"2024-09-10T19:26:38.188937+0000","last_fullsized":"2024-09-10T19:26:38.188937+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"237'37","last_scrub_stamp":"2024-09-10T19:26:36.464050+0000","last_deep_scrub":"104'27","last_deep_scrub_stamp":"2024-09-10T19:13:02.225944+0000","last_clean_scrub_stamp":"2024-09-10T19:26:36.464050+0000","objects_scrubbed":8,"log_size":37,"log_dups_size":0,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:27:56.488429+0000","scrub_duration":0.0033331939999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":53,"num_read_kb":39,"num_write":32,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":380,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-09-10T19:26:43.202547+0000","last_change":"2024-09-10T19:26:29.426781+0000","last_active":"2024-09-10T19:26:43.202547+0000","last_peered":"2024-09-10T19:26:43.202547+0000","last_clean":"2024-09-10T19:26:43.202547+0000","last_became_active":"2024-09-10T19:24:23.899556+0000","last_became_peered":"2024-09-10T19:24:23.899556+0000","last_unstale":"2024-09-10T19:26:43.202547+0000","last_undegraded":"2024-09-10T19:26:43.202547+0000","last_fullsized":"2024-09-10T19:26:43.202547+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:26:29.426689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:16:37.779967+0000","last_clean_scrub_stamp":"2024-09-10T19:26:29.426689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:27:52.170491+0000","scrub_duration":0.0054033600000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":379,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-09-10T19:26:43.203662+0000","last_change":"2024-09-10T19:26:28.366786+0000","last_active":"2024-09-10T19:26:43.203662+0000","last_peered":"2024-09-10T19:26:43.203662+0000","last_clean":"2024-09-10T19:26:43.203662+0000","last_became_active":"2024-09-10T19:24:23.901937+0000","last_became_peered":"2024-09-10T19:24:23.901937+0000","last_unstale":"2024-09-10T19:26:43.203662+0000","last_undegraded":"2024-09-10T19:26:43.203662+0000","last_fullsized":"2024-09-10T19:26:43.203662+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:26:28.366688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:26:28.366688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:27:41.195321+0000","scrub_duration":0.010474305,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":80,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-09-10T19:26:43.192847+0000","last_change":"2024-09-10T19:24:42.439826+0000","last_active":"2024-09-10T19:26:43.192847+0000","last_peered":"2024-09-10T19:26:43.192847+0000","last_clean":"2024-09-10T19:26:43.192847+0000","last_became_active":"2024-09-10T19:24:01.974076+0000","last_became_peered":"2024-09-10T19:24:01.974076+0000","last_unstale":"2024-09-10T19:26:43.192847+0000","last_undegraded":"2024-09-10T19:26:43.192847+0000","last_fullsized":"2024-09-10T19:26:43.192847+0000","mapping_epoch":292,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":293,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:24:42.439766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:24:42.439766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0031982790000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,7],"acting":[1,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.c","version":"353'13173","reported_seq":14500,"reported_epoch":353,"state":"active+clean","last_fresh":"2024-09-10T19:26:55.385775+0000","last_change":"2024-09-10T19:26:16.570194+0000","last_active":"2024-09-10T19:26:55.385775+0000","last_peered":"2024-09-10T19:26:55.385775+0000","last_clean":"2024-09-10T19:26:55.385775+0000","last_became_active":"2024-09-10T19:24:28.941128+0000","last_became_peered":"2024-09-10T19:24:28.941128+0000","last_unstale":"2024-09-10T19:26:55.385775+0000","last_undegraded":"2024-09-10T19:26:55.385775+0000","last_fullsized":"2024-09-10T19:26:55.385775+0000","mapping_epoch":306,"log_start":"341'8800","ondisk_log_start":"341'8800","created":295,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":0,"last_scrub":"345'9915","last_scrub_stamp":"2024-09-10T19:26:16.570088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:24:08.023080+0000","last_clean_scrub_stamp":"2024-09-10T19:26:16.570088+0000","objects_scrubbed":5727,"log_size":4373,"log_dups_size":0,"ondisk_log_size":4373,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:27:19.007947+0000","scrub_duration":2.4177897690000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":155516928,"num_objects":2373,"num_object_clones":0,"num_object_copies":7119,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2373,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13173,"num_write_kb":497472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":501,"num_bytes_recovered":32833536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.7","version":"74'79","reported_seq":228,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-09-10T19:26:43.202875+0000","last_change":"2024-09-10T19:26:31.437736+0000","last_active":"2024-09-10T19:26:43.202875+0000","last_peered":"2024-09-10T19:26:43.202875+0000","last_clean":"2024-09-10T19:26:43.202875+0000","last_became_active":"2024-09-10T19:20:30.842857+0000","last_became_peered":"2024-09-10T19:20:30.842857+0000","last_unstale":"2024-09-10T19:26:43.202875+0000","last_undegraded":"2024-09-10T19:26:43.202875+0000","last_fullsized":"2024-09-10T19:26:43.202875+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":3,"last_scrub":"74'79","last_scrub_stamp":"2024-09-10T19:26:31.437685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:08:00.699315+0000","last_clean_scrub_stamp":"2024-09-10T19:26:31.437685+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:27:42.941791+0000","scrub_duration":0.00135721,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.2","version":"350'25","reported_seq":107,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-09-10T19:26:43.195458+0000","last_change":"2024-09-10T19:26:37.415152+0000","last_active":"2024-09-10T19:26:43.195458+0000","last_peered":"2024-09-10T19:26:43.195458+0000","last_clean":"2024-09-10T19:26:43.195458+0000","last_became_active":"2024-09-10T19:24:01.976841+0000","last_became_peered":"2024-09-10T19:24:01.976841+0000","last_unstale":"2024-09-10T19:26:43.195458+0000","last_undegraded":"2024-09-10T19:26:43.195458+0000","last_fullsized":"2024-09-10T19:26:43.195458+0000","mapping_epoch":292,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":293,"parent":"0.0","parent_split_bits":0,"last_scrub":"350'25","last_scrub_stamp":"2024-09-10T19:26:37.415039+0000","last_deep_scrub":"350'25","last_deep_scrub_stamp":"2024-09-10T19:26:37.415039+0000","last_clean_scrub_stamp":"2024-09-10T19:26:37.415039+0000","objects_scrubbed":5,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:27:43.198721+0000","scrub_duration":0.022344089000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":28,"num_read_kb":23,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":94,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-09-10T19:26:43.193054+0000","last_change":"2024-09-10T19:24:45.452542+0000","last_active":"2024-09-10T19:26:43.193054+0000","last_peered":"2024-09-10T19:26:43.193054+0000","last_clean":"2024-09-10T19:26:43.193054+0000","last_became_active":"2024-09-10T19:24:24.910173+0000","last_became_peered":"2024-09-10T19:24:24.910173+0000","last_unstale":"2024-09-10T19:26:43.193054+0000","last_undegraded":"2024-09-10T19:26:43.193054+0000","last_fullsized":"2024-09-10T19:26:43.193054+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:24:45.452446+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:24:45.452446+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0063464669999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"303'11","reported_seq":485,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-09-10T19:26:45.927753+0000","last_change":"2024-09-10T19:26:08.469365+0000","last_active":"2024-09-10T19:26:45.927753+0000","last_peered":"2024-09-10T19:26:45.927753+0000","last_clean":"2024-09-10T19:26:45.927753+0000","last_became_active":"2024-09-10T19:24:23.905107+0000","last_became_peered":"2024-09-10T19:24:23.905107+0000","last_unstale":"2024-09-10T19:26:45.927753+0000","last_undegraded":"2024-09-10T19:26:45.927753+0000","last_fullsized":"2024-09-10T19:26:45.927753+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"303'11","last_scrub_stamp":"2024-09-10T19:26:08.469285+0000","last_deep_scrub":"84'9","last_deep_scrub_stamp":"2024-09-10T19:11:53.879272+0000","last_clean_scrub_stamp":"2024-09-10T19:26:08.469285+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:27:26.565539+0000","scrub_duration":0.010068145000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":322,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-09-10T19:26:38.187036+0000","last_change":"2024-09-10T19:25:21.356950+0000","last_active":"2024-09-10T19:26:38.187036+0000","last_peered":"2024-09-10T19:26:38.187036+0000","last_clean":"2024-09-10T19:26:38.187036+0000","last_became_active":"2024-09-10T19:24:23.905977+0000","last_became_peered":"2024-09-10T19:24:23.905977+0000","last_unstale":"2024-09-10T19:26:38.187036+0000","last_undegraded":"2024-09-10T19:26:38.187036+0000","last_fullsized":"2024-09-10T19:26:38.187036+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:25:21.356861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:25:21.356861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.00363751,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,5],"acting":[0,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.a","version":"353'12958","reported_seq":14152,"reported_epoch":353,"state":"active+clean","last_fresh":"2024-09-10T19:26:54.692863+0000","last_change":"2024-09-10T19:25:21.328117+0000","last_active":"2024-09-10T19:26:54.692863+0000","last_peered":"2024-09-10T19:26:54.692863+0000","last_clean":"2024-09-10T19:26:54.692863+0000","last_became_active":"2024-09-10T19:24:29.876670+0000","last_became_peered":"2024-09-10T19:24:29.876670+0000","last_unstale":"2024-09-10T19:26:54.692863+0000","last_undegraded":"2024-09-10T19:26:54.692863+0000","last_fullsized":"2024-09-10T19:26:54.692863+0000","mapping_epoch":307,"log_start":"341'8200","ondisk_log_start":"341'8200","created":295,"last_epoch_clean":308,"parent":"0.0","parent_split_bits":0,"last_scrub":"331'5451","last_scrub_stamp":"2024-09-10T19:25:21.327858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:24:08.023080+0000","last_clean_scrub_stamp":"2024-09-10T19:25:21.327858+0000","objects_scrubbed":5372,"log_size":4758,"log_dups_size":0,"ondisk_log_size":4758,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"queued for scrub","scrub_duration":2.071574102,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154664960,"num_objects":2360,"num_object_clones":0,"num_object_copies":7080,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2360,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12958,"num_write_kb":490176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":443,"num_bytes_recovered":29032448,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.1","version":"74'79","reported_seq":145,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-09-10T19:26:43.209770+0000","last_change":"2024-09-10T19:26:19.928630+0000","last_active":"2024-09-10T19:26:43.209770+0000","last_peered":"2024-09-10T19:26:43.209770+0000","last_clean":"2024-09-10T19:26:43.209770+0000","last_became_active":"2024-09-10T19:24:23.907264+0000","last_became_peered":"2024-09-10T19:24:23.907264+0000","last_unstale":"2024-09-10T19:26:43.209770+0000","last_undegraded":"2024-09-10T19:26:43.209770+0000","last_fullsized":"2024-09-10T19:26:43.209770+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":4,"last_scrub":"74'79","last_scrub_stamp":"2024-09-10T19:26:19.927731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:08:00.699315+0000","last_clean_scrub_stamp":"2024-09-10T19:26:19.927731+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:27:34.215011+0000","scrub_duration":0.025790303000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,4],"acting":[5,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"294'44","reported_seq":136,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-09-10T19:26:50.928397+0000","last_change":"2024-09-10T19:26:10.425078+0000","last_active":"2024-09-10T19:26:50.928397+0000","last_peered":"2024-09-10T19:26:50.928397+0000","last_clean":"2024-09-10T19:26:50.928397+0000","last_became_active":"2024-09-10T19:24:01.972831+0000","last_became_peered":"2024-09-10T19:24:01.972831+0000","last_unstale":"2024-09-10T19:26:50.928397+0000","last_undegraded":"2024-09-10T19:26:50.928397+0000","last_fullsized":"2024-09-10T19:26:50.928397+0000","mapping_epoch":292,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":293,"parent":"0.0","parent_split_bits":0,"last_scrub":"294'44","last_scrub_stamp":"2024-09-10T19:26:10.424990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:26:10.424990+0000","objects_scrubbed":10,"log_size":44,"log_dups_size":0,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:27:22.306370+0000","scrub_duration":0.0089341379999999995,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":65,"num_read_kb":50,"num_write":38,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":716,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":377,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-09-10T19:26:43.209728+0000","last_change":"2024-09-10T19:24:42.457503+0000","last_active":"2024-09-10T19:26:43.209728+0000","last_peered":"2024-09-10T19:26:43.209728+0000","last_clean":"2024-09-10T19:26:43.209728+0000","last_became_active":"2024-09-10T19:24:01.978187+0000","last_became_peered":"2024-09-10T19:24:01.978187+0000","last_unstale":"2024-09-10T19:26:43.209728+0000","last_undegraded":"2024-09-10T19:26:43.209728+0000","last_fullsized":"2024-09-10T19:26:43.209728+0000","mapping_epoch":292,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":293,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:24:42.457291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:24:42.457291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0080109410000000006,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":332,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-09-10T19:26:38.190220+0000","last_change":"2024-09-10T19:26:32.570336+0000","last_active":"2024-09-10T19:26:38.190220+0000","last_peered":"2024-09-10T19:26:38.190220+0000","last_clean":"2024-09-10T19:26:38.190220+0000","last_became_active":"2024-09-10T19:24:24.892692+0000","last_became_peered":"2024-09-10T19:24:24.892692+0000","last_unstale":"2024-09-10T19:26:38.190220+0000","last_undegraded":"2024-09-10T19:26:38.190220+0000","last_fullsized":"2024-09-10T19:26:38.190220+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:26:32.570260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:26:32.570260+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:27:53.412595+0000","scrub_duration":0.0043044880000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":347,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-09-10T19:26:43.198808+0000","last_change":"2024-09-10T19:26:09.890646+0000","last_active":"2024-09-10T19:26:43.198808+0000","last_peered":"2024-09-10T19:26:43.198808+0000","last_clean":"2024-09-10T19:26:43.198808+0000","last_became_active":"2024-09-10T19:10:13.721744+0000","last_became_peered":"2024-09-10T19:10:13.721744+0000","last_unstale":"2024-09-10T19:26:43.198808+0000","last_undegraded":"2024-09-10T19:26:43.198808+0000","last_fullsized":"2024-09-10T19:26:43.198808+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:26:09.890570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:26:09.890570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:27:30.451846+0000","scrub_duration":0.012478856,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.9","version":"353'13092","reported_seq":13324,"reported_epoch":353,"state":"active+clean","last_fresh":"2024-09-10T19:26:55.469599+0000","last_change":"2024-09-10T19:26:37.477311+0000","last_active":"2024-09-10T19:26:55.469599+0000","last_peered":"2024-09-10T19:26:55.469599+0000","last_clean":"2024-09-10T19:26:55.469599+0000","last_became_active":"2024-09-10T19:24:09.042606+0000","last_became_peered":"2024-09-10T19:24:09.042606+0000","last_unstale":"2024-09-10T19:26:55.469599+0000","last_undegraded":"2024-09-10T19:26:55.469599+0000","last_fullsized":"2024-09-10T19:26:55.469599+0000","mapping_epoch":295,"log_start":"341'8600","ondisk_log_start":"341'8600","created":295,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"350'11604","last_scrub_stamp":"2024-09-10T19:26:37.477228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:24:08.023080+0000","last_clean_scrub_stamp":"2024-09-10T19:26:37.477228+0000","objects_scrubbed":3893,"log_size":4492,"log_dups_size":0,"ondisk_log_size":4492,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:27:44.794432+0000","scrub_duration":1.5813720249999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154271744,"num_objects":2354,"num_object_clones":0,"num_object_copies":7062,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2354,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13092,"num_write_kb":494272,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,4],"acting":[5,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.2","version":"74'79","reported_seq":445,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-09-10T19:26:43.201848+0000","last_change":"2024-09-10T19:26:37.555224+0000","last_active":"2024-09-10T19:26:43.201848+0000","last_peered":"2024-09-10T19:26:43.201848+0000","last_clean":"2024-09-10T19:26:43.201848+0000","last_became_active":"2024-09-10T19:24:23.895247+0000","last_became_peered":"2024-09-10T19:24:23.895247+0000","last_unstale":"2024-09-10T19:26:43.201848+0000","last_undegraded":"2024-09-10T19:26:43.201848+0000","last_fullsized":"2024-09-10T19:26:43.201848+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":4,"last_scrub":"74'79","last_scrub_stamp":"2024-09-10T19:26:37.555133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:08:00.699315+0000","last_clean_scrub_stamp":"2024-09-10T19:26:37.555133+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:27:38.585160+0000","scrub_duration":0.0060663699999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.7","version":"237'22","reported_seq":106,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-09-10T19:26:43.195046+0000","last_change":"2024-09-10T19:25:13.591203+0000","last_active":"2024-09-10T19:26:43.195046+0000","last_peered":"2024-09-10T19:26:43.195046+0000","last_clean":"2024-09-10T19:26:43.195046+0000","last_became_active":"2024-09-10T19:24:24.894293+0000","last_became_peered":"2024-09-10T19:24:24.894293+0000","last_unstale":"2024-09-10T19:26:43.195046+0000","last_undegraded":"2024-09-10T19:26:43.195046+0000","last_fullsized":"2024-09-10T19:26:43.195046+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"237'22","last_scrub_stamp":"2024-09-10T19:25:13.591105+0000","last_deep_scrub":"82'18","last_deep_scrub_stamp":"2024-09-10T19:15:55.777361+0000","last_clean_scrub_stamp":"2024-09-10T19:25:13.591105+0000","objects_scrubbed":5,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.002392178,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":36,"num_read_kb":25,"num_write":20,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"159'6","reported_seq":266,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-09-10T19:26:52.119670+0000","last_change":"2024-09-10T19:24:43.417953+0000","last_active":"2024-09-10T19:26:52.119670+0000","last_peered":"2024-09-10T19:26:52.119670+0000","last_clean":"2024-09-10T19:26:52.119670+0000","last_became_active":"2024-09-10T19:24:24.887336+0000","last_became_peered":"2024-09-10T19:24:24.887336+0000","last_unstale":"2024-09-10T19:26:52.119670+0000","last_undegraded":"2024-09-10T19:26:52.119670+0000","last_fullsized":"2024-09-10T19:26:52.119670+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"159'6","last_scrub_stamp":"2024-09-10T19:24:43.417863+0000","last_deep_scrub":"159'6","last_deep_scrub_stamp":"2024-09-10T19:16:32.649914+0000","last_clean_scrub_stamp":"2024-09-10T19:24:43.417863+0000","objects_scrubbed":1,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0045847300000000004,"objects_trimmed":0,"snaptrim_duration":0.00034454499999999997,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":179,"num_read_kb":174,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":778,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":282,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-09-10T19:26:43.198024+0000","last_change":"2024-09-10T19:26:11.877736+0000","last_active":"2024-09-10T19:26:43.198024+0000","last_peered":"2024-09-10T19:26:43.198024+0000","last_clean":"2024-09-10T19:26:43.198024+0000","last_became_active":"2024-09-10T19:24:24.924859+0000","last_became_peered":"2024-09-10T19:24:24.924859+0000","last_unstale":"2024-09-10T19:26:43.198024+0000","last_undegraded":"2024-09-10T19:26:43.198024+0000","last_fullsized":"2024-09-10T19:26:43.198024+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:26:11.877648+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:26:11.877648+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:27:30.098099+0000","scrub_duration":0.0035177559999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":251,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-09-10T19:26:43.215297+0000","last_change":"2024-09-10T19:26:21.946019+0000","last_active":"2024-09-10T19:26:43.215297+0000","last_peered":"2024-09-10T19:26:43.215297+0000","last_clean":"2024-09-10T19:26:43.215297+0000","last_became_active":"2024-09-10T19:16:24.861481+0000","last_became_peered":"2024-09-10T19:16:24.861481+0000","last_unstale":"2024-09-10T19:26:43.215297+0000","last_undegraded":"2024-09-10T19:26:43.215297+0000","last_fullsized":"2024-09-10T19:26:43.215297+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:26:21.945941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:23:28.736502+0000","last_clean_scrub_stamp":"2024-09-10T19:26:21.945941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:27:22.081102+0000","scrub_duration":0.019402639999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,4],"acting":[5,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":360,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-09-10T19:26:43.202195+0000","last_change":"2024-09-10T19:25:21.790876+0000","last_active":"2024-09-10T19:26:43.202195+0000","last_peered":"2024-09-10T19:26:43.202195+0000","last_clean":"2024-09-10T19:26:43.202195+0000","last_became_active":"2024-09-10T19:10:14.053697+0000","last_became_peered":"2024-09-10T19:10:14.053697+0000","last_unstale":"2024-09-10T19:26:43.202195+0000","last_undegraded":"2024-09-10T19:26:43.202195+0000","last_fullsized":"2024-09-10T19:26:43.202195+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:25:21.790674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:22:01.208272+0000","last_clean_scrub_stamp":"2024-09-10T19:25:21.790674+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0098199150000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.8","version":"352'12755","reported_seq":13040,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-09-10T19:26:51.944638+0000","last_change":"2024-09-10T19:26:25.382432+0000","last_active":"2024-09-10T19:26:51.944638+0000","last_peered":"2024-09-10T19:26:51.944638+0000","last_clean":"2024-09-10T19:26:51.944638+0000","last_became_active":"2024-09-10T19:24:29.877601+0000","last_became_peered":"2024-09-10T19:24:29.877601+0000","last_unstale":"2024-09-10T19:26:51.944638+0000","last_undegraded":"2024-09-10T19:26:51.944638+0000","last_fullsized":"2024-09-10T19:26:51.944638+0000","mapping_epoch":307,"log_start":"342'8800","ondisk_log_start":"342'8800","created":295,"last_epoch_clean":308,"parent":"0.0","parent_split_bits":0,"last_scrub":"349'10531","last_scrub_stamp":"2024-09-10T19:26:25.382365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:24:08.023080+0000","last_clean_scrub_stamp":"2024-09-10T19:26:25.382365+0000","objects_scrubbed":4903,"log_size":3955,"log_dups_size":0,"ondisk_log_size":3955,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:27:26.462413+0000","scrub_duration":1.887493402,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":172294144,"num_objects":2629,"num_object_clones":0,"num_object_copies":7887,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2629,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12755,"num_write_kb":492288,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":455,"num_bytes_recovered":29818880,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,5],"acting":[0,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"74'79","reported_seq":80,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-09-10T19:26:43.196068+0000","last_change":"2024-09-10T19:25:17.733733+0000","last_active":"2024-09-10T19:26:43.196068+0000","last_peered":"2024-09-10T19:26:43.196068+0000","last_clean":"2024-09-10T19:26:43.196068+0000","last_became_active":"2024-09-10T19:24:01.977477+0000","last_became_peered":"2024-09-10T19:24:01.977477+0000","last_unstale":"2024-09-10T19:26:43.196068+0000","last_undegraded":"2024-09-10T19:26:43.196068+0000","last_fullsized":"2024-09-10T19:26:43.196068+0000","mapping_epoch":292,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":293,"parent":"0.0","parent_split_bits":3,"last_scrub":"74'79","last_scrub_stamp":"2024-09-10T19:25:17.733642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:08:00.699315+0000","last_clean_scrub_stamp":"2024-09-10T19:25:17.733642+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0056412739999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"237'30","reported_seq":412,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-09-10T19:26:38.189861+0000","last_change":"2024-09-10T19:26:09.481450+0000","last_active":"2024-09-10T19:26:38.189861+0000","last_peered":"2024-09-10T19:26:38.189861+0000","last_clean":"2024-09-10T19:26:38.189861+0000","last_became_active":"2024-09-10T19:24:24.899711+0000","last_became_peered":"2024-09-10T19:24:24.899711+0000","last_unstale":"2024-09-10T19:26:38.189861+0000","last_undegraded":"2024-09-10T19:26:38.189861+0000","last_fullsized":"2024-09-10T19:26:38.189861+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"237'30","last_scrub_stamp":"2024-09-10T19:26:09.481366+0000","last_deep_scrub":"221'18","last_deep_scrub_stamp":"2024-09-10T19:19:29.606748+0000","last_clean_scrub_stamp":"2024-09-10T19:26:09.481366+0000","objects_scrubbed":6,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:27:32.712334+0000","scrub_duration":0.0023429890000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"61'1","reported_seq":315,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-09-10T19:26:38.186486+0000","last_change":"2024-09-10T19:25:13.486963+0000","last_active":"2024-09-10T19:26:38.186486+0000","last_peered":"2024-09-10T19:26:38.186486+0000","last_clean":"2024-09-10T19:26:38.186486+0000","last_became_active":"2024-09-10T19:24:23.897412+0000","last_became_peered":"2024-09-10T19:24:23.897412+0000","last_unstale":"2024-09-10T19:26:38.186486+0000","last_undegraded":"2024-09-10T19:26:38.186486+0000","last_fullsized":"2024-09-10T19:26:38.186486+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'1","last_scrub_stamp":"2024-09-10T19:25:13.486865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:25:13.486865+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0054878949999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":401,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-09-10T19:26:43.203500+0000","last_change":"2024-09-10T19:25:28.762482+0000","last_active":"2024-09-10T19:26:43.203500+0000","last_peered":"2024-09-10T19:26:43.203500+0000","last_clean":"2024-09-10T19:26:43.203500+0000","last_became_active":"2024-09-10T19:24:24.896201+0000","last_became_peered":"2024-09-10T19:24:24.896201+0000","last_unstale":"2024-09-10T19:26:43.203500+0000","last_undegraded":"2024-09-10T19:26:43.203500+0000","last_fullsized":"2024-09-10T19:26:43.203500+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:25:28.762237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:25:28.762237+0000","last_clean_scrub_stamp":"2024-09-10T19:25:28.762237+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0099337430000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":397,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-09-10T19:26:43.216418+0000","last_change":"2024-09-10T19:24:49.491925+0000","last_active":"2024-09-10T19:26:43.216418+0000","last_peered":"2024-09-10T19:26:43.216418+0000","last_clean":"2024-09-10T19:26:43.216418+0000","last_became_active":"2024-09-10T19:24:24.914680+0000","last_became_peered":"2024-09-10T19:24:24.914680+0000","last_unstale":"2024-09-10T19:26:43.216418+0000","last_undegraded":"2024-09-10T19:26:43.216418+0000","last_fullsized":"2024-09-10T19:26:43.216418+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:24:49.491839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:24:49.491839+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0022950959999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":80,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-09-10T19:26:43.192910+0000","last_change":"2024-09-10T19:25:22.846256+0000","last_active":"2024-09-10T19:26:43.192910+0000","last_peered":"2024-09-10T19:26:43.192910+0000","last_clean":"2024-09-10T19:26:43.192910+0000","last_became_active":"2024-09-10T19:24:01.974039+0000","last_became_peered":"2024-09-10T19:24:01.974039+0000","last_unstale":"2024-09-10T19:26:43.192910+0000","last_undegraded":"2024-09-10T19:26:43.192910+0000","last_fullsized":"2024-09-10T19:26:43.192910+0000","mapping_epoch":292,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":293,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:25:22.846176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:25:22.846176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001764222,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.b","version":"353'13212","reported_seq":14475,"reported_epoch":353,"state":"active+clean","last_fresh":"2024-09-10T19:26:54.681881+0000","last_change":"2024-09-10T19:26:21.279920+0000","last_active":"2024-09-10T19:26:54.681881+0000","last_peered":"2024-09-10T19:26:54.681881+0000","last_clean":"2024-09-10T19:26:54.681881+0000","last_became_active":"2024-09-10T19:24:28.940329+0000","last_became_peered":"2024-09-10T19:24:28.940329+0000","last_unstale":"2024-09-10T19:26:54.681881+0000","last_undegraded":"2024-09-10T19:26:54.681881+0000","last_fullsized":"2024-09-10T19:26:54.681881+0000","mapping_epoch":306,"log_start":"341'8500","ondisk_log_start":"341'8500","created":295,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":0,"last_scrub":"347'10423","last_scrub_stamp":"2024-09-10T19:26:21.279841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:24:08.023080+0000","last_clean_scrub_stamp":"2024-09-10T19:26:21.279841+0000","objects_scrubbed":5302,"log_size":4712,"log_dups_size":0,"ondisk_log_size":4712,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:27:21.438796+0000","scrub_duration":1.8983051479999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":160301056,"num_objects":2446,"num_object_clones":0,"num_object_copies":7338,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2446,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13212,"num_write_kb":501056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":478,"num_bytes_recovered":31326208,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"237'33","reported_seq":428,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-09-10T19:26:43.202636+0000","last_change":"2024-09-10T19:24:44.886858+0000","last_active":"2024-09-10T19:26:43.202636+0000","last_peered":"2024-09-10T19:26:43.202636+0000","last_clean":"2024-09-10T19:26:43.202636+0000","last_became_active":"2024-09-10T19:10:09.822184+0000","last_became_peered":"2024-09-10T19:10:09.822184+0000","last_unstale":"2024-09-10T19:26:43.202636+0000","last_undegraded":"2024-09-10T19:26:43.202636+0000","last_fullsized":"2024-09-10T19:26:43.202636+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"237'33","last_scrub_stamp":"2024-09-10T19:24:44.884921+0000","last_deep_scrub":"80'21","last_deep_scrub_stamp":"2024-09-10T19:11:39.836144+0000","last_clean_scrub_stamp":"2024-09-10T19:24:44.884921+0000","objects_scrubbed":7,"log_size":33,"log_dups_size":0,"ondisk_log_size":33,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.015025566000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":33,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"63'2","reported_seq":370,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-09-10T19:26:43.198303+0000","last_change":"2024-09-10T19:25:23.821301+0000","last_active":"2024-09-10T19:26:43.198303+0000","last_peered":"2024-09-10T19:26:43.198303+0000","last_clean":"2024-09-10T19:26:43.198303+0000","last_became_active":"2024-09-10T19:10:07.684784+0000","last_became_peered":"2024-09-10T19:10:07.684784+0000","last_unstale":"2024-09-10T19:26:43.198303+0000","last_undegraded":"2024-09-10T19:26:43.198303+0000","last_fullsized":"2024-09-10T19:26:43.198303+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'2","last_scrub_stamp":"2024-09-10T19:25:23.821213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:25:23.821213+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0063659800000000002,"objects_trimmed":0,"snaptrim_duration":0.00075522999999999996,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":373,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-09-10T19:26:43.213573+0000","last_change":"2024-09-10T19:26:08.934930+0000","last_active":"2024-09-10T19:26:43.213573+0000","last_peered":"2024-09-10T19:26:43.213573+0000","last_clean":"2024-09-10T19:26:43.213573+0000","last_became_active":"2024-09-10T19:16:24.858662+0000","last_became_peered":"2024-09-10T19:16:24.858662+0000","last_unstale":"2024-09-10T19:26:43.213573+0000","last_undegraded":"2024-09-10T19:26:43.213573+0000","last_fullsized":"2024-09-10T19:26:43.213573+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:26:08.934746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:26:08.934746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:27:38.162812+0000","scrub_duration":0.0052906790000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"74'79","reported_seq":426,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-09-10T19:26:43.203638+0000","last_change":"2024-09-10T19:24:40.776904+0000","last_active":"2024-09-10T19:26:43.203638+0000","last_peered":"2024-09-10T19:26:43.203638+0000","last_clean":"2024-09-10T19:26:43.203638+0000","last_became_active":"2024-09-10T19:24:23.904267+0000","last_became_peered":"2024-09-10T19:24:23.904267+0000","last_unstale":"2024-09-10T19:26:43.203638+0000","last_undegraded":"2024-09-10T19:26:43.203638+0000","last_fullsized":"2024-09-10T19:26:43.203638+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'79","last_scrub_stamp":"2024-09-10T19:24:40.776821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:08:00.699315+0000","last_clean_scrub_stamp":"2024-09-10T19:24:40.776821+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0056971499999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":386,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-09-10T19:26:43.202871+0000","last_change":"2024-09-10T19:26:31.509806+0000","last_active":"2024-09-10T19:26:43.202871+0000","last_peered":"2024-09-10T19:26:43.202871+0000","last_clean":"2024-09-10T19:26:43.202871+0000","last_became_active":"2024-09-10T19:24:23.903868+0000","last_became_peered":"2024-09-10T19:24:23.903868+0000","last_unstale":"2024-09-10T19:26:43.202871+0000","last_undegraded":"2024-09-10T19:26:43.202871+0000","last_fullsized":"2024-09-10T19:26:43.202871+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:26:31.509498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:26:31.509498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:27:59.336304+0000","scrub_duration":0.029737573999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,0],"acting":[7,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":357,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-09-10T19:26:43.216530+0000","last_change":"2024-09-10T19:26:33.822326+0000","last_active":"2024-09-10T19:26:43.216530+0000","last_peered":"2024-09-10T19:26:43.216530+0000","last_clean":"2024-09-10T19:26:43.216530+0000","last_became_active":"2024-09-10T19:10:13.720647+0000","last_became_peered":"2024-09-10T19:10:13.720647+0000","last_unstale":"2024-09-10T19:26:43.216530+0000","last_undegraded":"2024-09-10T19:26:43.216530+0000","last_fullsized":"2024-09-10T19:26:43.216530+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:26:33.820918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:26:33.820918+0000","last_clean_scrub_stamp":"2024-09-10T19:26:33.820918+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:27:50.206519+0000","scrub_duration":0.0080461309999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.4","version":"352'12874","reported_seq":13441,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-09-10T19:26:51.949711+0000","last_change":"2024-09-10T19:26:26.177340+0000","last_active":"2024-09-10T19:26:51.949711+0000","last_peered":"2024-09-10T19:26:51.949711+0000","last_clean":"2024-09-10T19:26:51.949711+0000","last_became_active":"2024-09-10T19:24:30.884803+0000","last_became_peered":"2024-09-10T19:24:30.884803+0000","last_unstale":"2024-09-10T19:26:51.949711+0000","last_undegraded":"2024-09-10T19:26:51.949711+0000","last_fullsized":"2024-09-10T19:26:51.949711+0000","mapping_epoch":308,"log_start":"342'8900","ondisk_log_start":"342'8900","created":295,"last_epoch_clean":309,"parent":"0.0","parent_split_bits":0,"last_scrub":"349'10642","last_scrub_stamp":"2024-09-10T19:26:26.177254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:24:08.023080+0000","last_clean_scrub_stamp":"2024-09-10T19:26:26.177254+0000","objects_scrubbed":4897,"log_size":3974,"log_dups_size":0,"ondisk_log_size":3974,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:27:27.930007+0000","scrub_duration":1.705149934,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":169869312,"num_objects":2592,"num_object_clones":0,"num_object_copies":7776,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2592,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12874,"num_write_kb":494912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":441,"num_bytes_recovered":28901376,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"237'35","reported_seq":499,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-09-10T19:26:43.203937+0000","last_change":"2024-09-10T19:26:33.411895+0000","last_active":"2024-09-10T19:26:43.203937+0000","last_peered":"2024-09-10T19:26:43.203937+0000","last_clean":"2024-09-10T19:26:43.203937+0000","last_became_active":"2024-09-10T19:24:01.973201+0000","last_became_peered":"2024-09-10T19:24:01.973201+0000","last_unstale":"2024-09-10T19:26:43.203937+0000","last_undegraded":"2024-09-10T19:26:43.203937+0000","last_fullsized":"2024-09-10T19:26:43.203937+0000","mapping_epoch":292,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":293,"parent":"0.0","parent_split_bits":0,"last_scrub":"237'35","last_scrub_stamp":"2024-09-10T19:26:33.411803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:26:33.411803+0000","objects_scrubbed":9,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:27:55.317233+0000","scrub_duration":0.0050992160000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":56,"num_read_kb":45,"num_write":36,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":262,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-09-10T19:26:43.212505+0000","last_change":"2024-09-10T19:24:24.913980+0000","last_active":"2024-09-10T19:26:43.212505+0000","last_peered":"2024-09-10T19:26:43.212505+0000","last_clean":"2024-09-10T19:26:43.212505+0000","last_became_active":"2024-09-10T19:24:24.913709+0000","last_became_peered":"2024-09-10T19:24:24.913709+0000","last_unstale":"2024-09-10T19:26:43.212505+0000","last_undegraded":"2024-09-10T19:26:43.212505+0000","last_fullsized":"2024-09-10T19:26:43.212505+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:24:05.723155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:24:05.723155+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0015060060000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,1],"acting":[5,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":192,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-09-10T19:26:38.188734+0000","last_change":"2024-09-10T19:26:30.557805+0000","last_active":"2024-09-10T19:26:38.188734+0000","last_peered":"2024-09-10T19:26:38.188734+0000","last_clean":"2024-09-10T19:26:38.188734+0000","last_became_active":"2024-09-10T19:24:23.901415+0000","last_became_peered":"2024-09-10T19:24:23.901415+0000","last_unstale":"2024-09-10T19:26:38.188734+0000","last_undegraded":"2024-09-10T19:26:38.188734+0000","last_fullsized":"2024-09-10T19:26:38.188734+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:26:30.557724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:26:30.557724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:27:49.638525+0000","scrub_duration":0.015588159000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":377,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-09-10T19:26:43.203568+0000","last_change":"2024-09-10T19:25:30.845977+0000","last_active":"2024-09-10T19:26:43.203568+0000","last_peered":"2024-09-10T19:26:43.203568+0000","last_clean":"2024-09-10T19:26:43.203568+0000","last_became_active":"2024-09-10T19:16:24.858172+0000","last_became_peered":"2024-09-10T19:16:24.858172+0000","last_unstale":"2024-09-10T19:26:43.203568+0000","last_undegraded":"2024-09-10T19:26:43.203568+0000","last_fullsized":"2024-09-10T19:26:43.203568+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:25:30.845546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:21:03.910074+0000","last_clean_scrub_stamp":"2024-09-10T19:25:30.845546+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0066935149999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,6],"acting":[7,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.5","version":"352'12845","reported_seq":13245,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-09-10T19:26:51.942763+0000","last_change":"2024-09-10T19:26:17.719959+0000","last_active":"2024-09-10T19:26:51.942763+0000","last_peered":"2024-09-10T19:26:51.942763+0000","last_clean":"2024-09-10T19:26:51.942763+0000","last_became_active":"2024-09-10T19:24:29.877183+0000","last_became_peered":"2024-09-10T19:24:29.877183+0000","last_unstale":"2024-09-10T19:26:51.942763+0000","last_undegraded":"2024-09-10T19:26:51.942763+0000","last_fullsized":"2024-09-10T19:26:51.942763+0000","mapping_epoch":307,"log_start":"342'8900","ondisk_log_start":"342'8900","created":295,"last_epoch_clean":308,"parent":"0.0","parent_split_bits":0,"last_scrub":"345'10001","last_scrub_stamp":"2024-09-10T19:26:17.719855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:24:08.023080+0000","last_clean_scrub_stamp":"2024-09-10T19:26:17.719855+0000","objects_scrubbed":5476,"log_size":3945,"log_dups_size":0,"ondisk_log_size":3945,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:27:21.631854+0000","scrub_duration":1.304291364,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":169017344,"num_objects":2579,"num_object_clones":0,"num_object_copies":7737,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2579,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12845,"num_write_kb":493568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":446,"num_bytes_recovered":29229056,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.b","version":"237'15","reported_seq":396,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-09-10T19:26:38.189113+0000","last_change":"2024-09-10T19:26:14.505086+0000","last_active":"2024-09-10T19:26:38.189113+0000","last_peered":"2024-09-10T19:26:38.189113+0000","last_clean":"2024-09-10T19:26:38.189113+0000","last_became_active":"2024-09-10T19:24:44.892294+0000","last_became_peered":"2024-09-10T19:24:44.892294+0000","last_unstale":"2024-09-10T19:26:38.189113+0000","last_undegraded":"2024-09-10T19:26:38.189113+0000","last_fullsized":"2024-09-10T19:26:38.189113+0000","mapping_epoch":314,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":315,"parent":"0.0","parent_split_bits":0,"last_scrub":"237'15","last_scrub_stamp":"2024-09-10T19:26:14.504998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:26:14.504998+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:27:24.336321+0000","scrub_duration":0.016209647000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,4],"acting":[0,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":80,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-09-10T19:26:43.193505+0000","last_change":"2024-09-10T19:26:30.411444+0000","last_active":"2024-09-10T19:26:43.193505+0000","last_peered":"2024-09-10T19:26:43.193505+0000","last_clean":"2024-09-10T19:26:43.193505+0000","last_became_active":"2024-09-10T19:24:01.980348+0000","last_became_peered":"2024-09-10T19:24:01.980348+0000","last_unstale":"2024-09-10T19:26:43.193505+0000","last_undegraded":"2024-09-10T19:26:43.193505+0000","last_fullsized":"2024-09-10T19:26:43.193505+0000","mapping_epoch":292,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":293,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:26:30.411368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:11:44.254513+0000","last_clean_scrub_stamp":"2024-09-10T19:26:30.411368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:27:44.702203+0000","scrub_duration":0.0027079880000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":261,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-09-10T19:26:43.198475+0000","last_change":"2024-09-10T19:24:54.563313+0000","last_active":"2024-09-10T19:26:43.198475+0000","last_peered":"2024-09-10T19:26:43.198475+0000","last_clean":"2024-09-10T19:26:43.198475+0000","last_became_active":"2024-09-10T19:24:23.891161+0000","last_became_peered":"2024-09-10T19:24:23.891161+0000","last_unstale":"2024-09-10T19:26:43.198475+0000","last_undegraded":"2024-09-10T19:26:43.198475+0000","last_fullsized":"2024-09-10T19:26:43.198475+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:24:54.563235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:24:54.563235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0029717020000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":310,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-09-10T19:26:38.186561+0000","last_change":"2024-09-10T19:26:31.581355+0000","last_active":"2024-09-10T19:26:38.186561+0000","last_peered":"2024-09-10T19:26:38.186561+0000","last_clean":"2024-09-10T19:26:38.186561+0000","last_became_active":"2024-09-10T19:24:23.905112+0000","last_became_peered":"2024-09-10T19:24:23.905112+0000","last_unstale":"2024-09-10T19:26:38.186561+0000","last_undegraded":"2024-09-10T19:26:38.186561+0000","last_fullsized":"2024-09-10T19:26:38.186561+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:26:31.581300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:26:31.581300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:27:51.658432+0000","scrub_duration":0.0033456739999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.6","version":"353'13175","reported_seq":14236,"reported_epoch":353,"state":"active+clean","last_fresh":"2024-09-10T19:26:55.466952+0000","last_change":"2024-09-10T19:24:30.889054+0000","last_active":"2024-09-10T19:26:55.466952+0000","last_peered":"2024-09-10T19:26:55.466952+0000","last_clean":"2024-09-10T19:26:55.466952+0000","last_became_active":"2024-09-10T19:24:30.883926+0000","last_became_peered":"2024-09-10T19:24:30.883926+0000","last_unstale":"2024-09-10T19:26:55.466952+0000","last_undegraded":"2024-09-10T19:26:55.466952+0000","last_fullsized":"2024-09-10T19:26:55.466952+0000","mapping_epoch":308,"log_start":"341'8700","ondisk_log_start":"341'8700","created":295,"last_epoch_clean":309,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:24:08.023080+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:24:08.023080+0000","last_clean_scrub_stamp":"2024-09-10T19:24:08.023080+0000","objects_scrubbed":0,"log_size":4475,"log_dups_size":0,"ondisk_log_size":4475,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":153944064,"num_objects":2349,"num_object_clones":0,"num_object_copies":7047,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2349,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13175,"num_write_kb":496768,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":488,"num_bytes_recovered":31981568,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,4],"acting":[5,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.8","version":"237'34","reported_seq":401,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-09-10T19:26:43.210588+0000","last_change":"2024-09-10T19:25:27.750310+0000","last_active":"2024-09-10T19:26:43.210588+0000","last_peered":"2024-09-10T19:26:43.210588+0000","last_clean":"2024-09-10T19:26:43.210588+0000","last_became_active":"2024-09-10T19:10:09.820950+0000","last_became_peered":"2024-09-10T19:10:09.820950+0000","last_unstale":"2024-09-10T19:26:43.210588+0000","last_undegraded":"2024-09-10T19:26:43.210588+0000","last_fullsized":"2024-09-10T19:26:43.210588+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"237'34","last_scrub_stamp":"2024-09-10T19:25:27.750229+0000","last_deep_scrub":"220'22","last_deep_scrub_stamp":"2024-09-10T19:19:24.597866+0000","last_clean_scrub_stamp":"2024-09-10T19:25:27.750229+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.013590471,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":241,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-09-10T19:26:43.198832+0000","last_change":"2024-09-10T19:25:24.833996+0000","last_active":"2024-09-10T19:26:43.198832+0000","last_peered":"2024-09-10T19:26:43.198832+0000","last_clean":"2024-09-10T19:26:43.198832+0000","last_became_active":"2024-09-10T19:16:24.857253+0000","last_became_peered":"2024-09-10T19:16:24.857253+0000","last_unstale":"2024-09-10T19:26:43.198832+0000","last_undegraded":"2024-09-10T19:26:43.198832+0000","last_fullsized":"2024-09-10T19:26:43.198832+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:25:24.833939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:25:24.833939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0086811059999999992,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.9","version":"80'9","reported_seq":579,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-09-10T19:26:50.930973+0000","last_change":"2024-09-10T19:25:29.833358+0000","last_active":"2024-09-10T19:26:50.930973+0000","last_peered":"2024-09-10T19:26:50.930973+0000","last_clean":"2024-09-10T19:26:50.930973+0000","last_became_active":"2024-09-10T19:10:11.702878+0000","last_became_peered":"2024-09-10T19:10:11.702878+0000","last_unstale":"2024-09-10T19:26:50.930973+0000","last_undegraded":"2024-09-10T19:26:50.930973+0000","last_fullsized":"2024-09-10T19:26:50.930973+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"80'9","last_scrub_stamp":"2024-09-10T19:25:29.833271+0000","last_deep_scrub":"80'9","last_deep_scrub_stamp":"2024-09-10T19:25:29.833271+0000","last_clean_scrub_stamp":"2024-09-10T19:25:29.833271+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.027756563000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":354,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-09-10T19:26:43.212997+0000","last_change":"2024-09-10T19:24:38.483717+0000","last_active":"2024-09-10T19:26:43.212997+0000","last_peered":"2024-09-10T19:26:43.212997+0000","last_clean":"2024-09-10T19:26:43.212997+0000","last_became_active":"2024-09-10T19:24:23.905606+0000","last_became_peered":"2024-09-10T19:24:23.905606+0000","last_unstale":"2024-09-10T19:26:43.212997+0000","last_undegraded":"2024-09-10T19:26:43.212997+0000","last_fullsized":"2024-09-10T19:26:43.212997+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:24:38.483596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:24:38.483596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0049697939999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.7","version":"353'12840","reported_seq":13148,"reported_epoch":353,"state":"active+clean","last_fresh":"2024-09-10T19:26:54.683097+0000","last_change":"2024-09-10T19:25:27.604914+0000","last_active":"2024-09-10T19:26:54.683097+0000","last_peered":"2024-09-10T19:26:54.683097+0000","last_clean":"2024-09-10T19:26:54.683097+0000","last_became_active":"2024-09-10T19:24:09.045557+0000","last_became_peered":"2024-09-10T19:24:09.045557+0000","last_unstale":"2024-09-10T19:26:54.683097+0000","last_undegraded":"2024-09-10T19:26:54.683097+0000","last_fullsized":"2024-09-10T19:26:54.683097+0000","mapping_epoch":295,"log_start":"341'8100","ondisk_log_start":"341'8100","created":295,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"333'5834","last_scrub_stamp":"2024-09-10T19:25:27.604825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:24:08.023080+0000","last_clean_scrub_stamp":"2024-09-10T19:25:27.604825+0000","objects_scrubbed":5783,"log_size":4740,"log_dups_size":0,"ondisk_log_size":4740,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"queued for scrub","scrub_duration":2.3305890900000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154140672,"num_objects":2352,"num_object_clones":0,"num_object_copies":7056,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2352,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12840,"num_write_kb":486144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.9","version":"237'18","reported_seq":446,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-09-10T19:26:43.198921+0000","last_change":"2024-09-10T19:26:11.809995+0000","last_active":"2024-09-10T19:26:43.198921+0000","last_peered":"2024-09-10T19:26:43.198921+0000","last_clean":"2024-09-10T19:26:43.198921+0000","last_became_active":"2024-09-10T19:24:01.974878+0000","last_became_peered":"2024-09-10T19:24:01.974878+0000","last_unstale":"2024-09-10T19:26:43.198921+0000","last_undegraded":"2024-09-10T19:26:43.198921+0000","last_fullsized":"2024-09-10T19:26:43.198921+0000","mapping_epoch":292,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":293,"parent":"0.0","parent_split_bits":0,"last_scrub":"237'18","last_scrub_stamp":"2024-09-10T19:26:11.809915+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:26:11.809915+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:27:39.096722+0000","scrub_duration":0.014721300999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.f","version":"59'2","reported_seq":385,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-09-10T19:26:43.205231+0000","last_change":"2024-09-10T19:24:53.593207+0000","last_active":"2024-09-10T19:26:43.205231+0000","last_peered":"2024-09-10T19:26:43.205231+0000","last_clean":"2024-09-10T19:26:43.205231+0000","last_became_active":"2024-09-10T19:24:23.893796+0000","last_became_peered":"2024-09-10T19:24:23.893796+0000","last_unstale":"2024-09-10T19:26:43.205231+0000","last_undegraded":"2024-09-10T19:26:43.205231+0000","last_fullsized":"2024-09-10T19:26:43.205231+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'2","last_scrub_stamp":"2024-09-10T19:24:53.593127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:24:53.593127+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0062292040000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":14,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":294,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-09-10T19:26:38.186877+0000","last_change":"2024-09-10T19:26:11.459855+0000","last_active":"2024-09-10T19:26:38.186877+0000","last_peered":"2024-09-10T19:26:38.186877+0000","last_clean":"2024-09-10T19:26:38.186877+0000","last_became_active":"2024-09-10T19:24:24.914252+0000","last_became_peered":"2024-09-10T19:24:24.914252+0000","last_unstale":"2024-09-10T19:26:38.186877+0000","last_undegraded":"2024-09-10T19:26:38.186877+0000","last_fullsized":"2024-09-10T19:26:38.186877+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:26:11.459804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:11:46.587846+0000","last_clean_scrub_stamp":"2024-09-10T19:26:11.459804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:27:13.584075+0000","scrub_duration":0.0072695340000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,5],"acting":[0,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":374,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-09-10T19:26:43.198419+0000","last_change":"2024-09-10T19:25:21.864732+0000","last_active":"2024-09-10T19:26:43.198419+0000","last_peered":"2024-09-10T19:26:43.198419+0000","last_clean":"2024-09-10T19:26:43.198419+0000","last_became_active":"2024-09-10T19:24:01.971131+0000","last_became_peered":"2024-09-10T19:24:01.971131+0000","last_unstale":"2024-09-10T19:26:43.198419+0000","last_undegraded":"2024-09-10T19:26:43.198419+0000","last_fullsized":"2024-09-10T19:26:43.198419+0000","mapping_epoch":292,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":293,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:25:21.864515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:25:21.864515+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0042058019999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.16","version":"237'15","reported_seq":350,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-09-10T19:26:38.189553+0000","last_change":"2024-09-10T19:25:28.386031+0000","last_active":"2024-09-10T19:26:38.189553+0000","last_peered":"2024-09-10T19:26:38.189553+0000","last_clean":"2024-09-10T19:26:38.189553+0000","last_became_active":"2024-09-10T19:24:23.901610+0000","last_became_peered":"2024-09-10T19:24:23.901610+0000","last_unstale":"2024-09-10T19:26:38.189553+0000","last_undegraded":"2024-09-10T19:26:38.189553+0000","last_fullsized":"2024-09-10T19:26:38.189553+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"237'15","last_scrub_stamp":"2024-09-10T19:25:28.385945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:25:28.385945+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:26:57.346081+0000","scrub_duration":0.0031317430000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":94,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-09-10T19:26:43.193130+0000","last_change":"2024-09-10T19:24:49.534368+0000","last_active":"2024-09-10T19:26:43.193130+0000","last_peered":"2024-09-10T19:26:43.193130+0000","last_clean":"2024-09-10T19:26:43.193130+0000","last_became_active":"2024-09-10T19:24:24.894815+0000","last_became_peered":"2024-09-10T19:24:24.894815+0000","last_unstale":"2024-09-10T19:26:43.193130+0000","last_undegraded":"2024-09-10T19:26:43.193130+0000","last_fullsized":"2024-09-10T19:26:43.193130+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:24:49.534307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:11:41.458523+0000","last_clean_scrub_stamp":"2024-09-10T19:24:49.534307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0045336179999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,4],"acting":[1,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":383,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-09-10T19:26:43.199222+0000","last_change":"2024-09-10T19:26:24.891697+0000","last_active":"2024-09-10T19:26:43.199222+0000","last_peered":"2024-09-10T19:26:43.199222+0000","last_clean":"2024-09-10T19:26:43.199222+0000","last_became_active":"2024-09-10T19:24:01.970325+0000","last_became_peered":"2024-09-10T19:24:01.970325+0000","last_unstale":"2024-09-10T19:26:43.199222+0000","last_undegraded":"2024-09-10T19:26:43.199222+0000","last_fullsized":"2024-09-10T19:26:43.199222+0000","mapping_epoch":292,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":293,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:26:24.891605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:24:09.367654+0000","last_clean_scrub_stamp":"2024-09-10T19:26:24.891605+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:27:54.880956+0000","scrub_duration":0.0032178340000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"66'1","reported_seq":261,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-09-10T19:26:43.199381+0000","last_change":"2024-09-10T19:24:43.668696+0000","last_active":"2024-09-10T19:26:43.199381+0000","last_peered":"2024-09-10T19:26:43.199381+0000","last_clean":"2024-09-10T19:26:43.199381+0000","last_became_active":"2024-09-10T19:24:23.897858+0000","last_became_peered":"2024-09-10T19:24:23.897858+0000","last_unstale":"2024-09-10T19:26:43.199381+0000","last_undegraded":"2024-09-10T19:26:43.199381+0000","last_fullsized":"2024-09-10T19:26:43.199381+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'1","last_scrub_stamp":"2024-09-10T19:24:43.668620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:24:43.668620+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0040764670000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":13,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":396,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-09-10T19:26:43.201695+0000","last_change":"2024-09-10T19:25:14.241349+0000","last_active":"2024-09-10T19:26:43.201695+0000","last_peered":"2024-09-10T19:26:43.201695+0000","last_clean":"2024-09-10T19:26:43.201695+0000","last_became_active":"2024-09-10T19:24:01.975706+0000","last_became_peered":"2024-09-10T19:24:01.975706+0000","last_unstale":"2024-09-10T19:26:43.201695+0000","last_undegraded":"2024-09-10T19:26:43.201695+0000","last_fullsized":"2024-09-10T19:26:43.201695+0000","mapping_epoch":292,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":293,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:25:14.241294+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:25:14.241294+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0019729370000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"237'12","reported_seq":420,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-09-10T19:26:43.199368+0000","last_change":"2024-09-10T19:24:54.606498+0000","last_active":"2024-09-10T19:26:43.199368+0000","last_peered":"2024-09-10T19:26:43.199368+0000","last_clean":"2024-09-10T19:26:43.199368+0000","last_became_active":"2024-09-10T19:24:01.971492+0000","last_became_peered":"2024-09-10T19:24:01.971492+0000","last_unstale":"2024-09-10T19:26:43.199368+0000","last_undegraded":"2024-09-10T19:26:43.199368+0000","last_fullsized":"2024-09-10T19:26:43.199368+0000","mapping_epoch":292,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":293,"parent":"0.0","parent_split_bits":0,"last_scrub":"237'12","last_scrub_stamp":"2024-09-10T19:24:54.606424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:24:54.606424+0000","objects_scrubbed":3,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0049032219999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":18,"num_read_kb":15,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":376,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-09-10T19:26:43.215364+0000","last_change":"2024-09-10T19:26:17.869690+0000","last_active":"2024-09-10T19:26:43.215364+0000","last_peered":"2024-09-10T19:26:43.215364+0000","last_clean":"2024-09-10T19:26:43.215364+0000","last_became_active":"2024-09-10T19:24:01.978215+0000","last_became_peered":"2024-09-10T19:24:01.978215+0000","last_unstale":"2024-09-10T19:26:43.215364+0000","last_undegraded":"2024-09-10T19:26:43.215364+0000","last_fullsized":"2024-09-10T19:26:43.215364+0000","mapping_epoch":292,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":293,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:26:17.869622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:11:23.264793+0000","last_clean_scrub_stamp":"2024-09-10T19:26:17.869622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:27:23.953590+0000","scrub_duration":0.005157554,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":357,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-09-10T19:26:43.202126+0000","last_change":"2024-09-10T19:24:49.802835+0000","last_active":"2024-09-10T19:26:43.202126+0000","last_peered":"2024-09-10T19:26:43.202126+0000","last_clean":"2024-09-10T19:26:43.202126+0000","last_became_active":"2024-09-10T19:10:14.054360+0000","last_became_peered":"2024-09-10T19:10:14.054360+0000","last_unstale":"2024-09-10T19:26:43.202126+0000","last_undegraded":"2024-09-10T19:26:43.202126+0000","last_fullsized":"2024-09-10T19:26:43.202126+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:24:49.802743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:24:49.802743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.023450184999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"237'25","reported_seq":459,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-09-10T19:26:43.198566+0000","last_change":"2024-09-10T19:25:16.796941+0000","last_active":"2024-09-10T19:26:43.198566+0000","last_peered":"2024-09-10T19:26:43.198566+0000","last_clean":"2024-09-10T19:26:43.198566+0000","last_became_active":"2024-09-10T19:24:01.971173+0000","last_became_peered":"2024-09-10T19:24:01.971173+0000","last_unstale":"2024-09-10T19:26:43.198566+0000","last_undegraded":"2024-09-10T19:26:43.198566+0000","last_fullsized":"2024-09-10T19:26:43.198566+0000","mapping_epoch":292,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":293,"parent":"0.0","parent_split_bits":0,"last_scrub":"237'25","last_scrub_stamp":"2024-09-10T19:25:16.796821+0000","last_deep_scrub":"80'15","last_deep_scrub_stamp":"2024-09-10T19:11:41.719310+0000","last_clean_scrub_stamp":"2024-09-10T19:25:16.796821+0000","objects_scrubbed":5,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0062295650000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":353,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-09-10T19:26:43.209803+0000","last_change":"2024-09-10T19:25:14.758934+0000","last_active":"2024-09-10T19:26:43.209803+0000","last_peered":"2024-09-10T19:26:43.209803+0000","last_clean":"2024-09-10T19:26:43.209803+0000","last_became_active":"2024-09-10T19:10:07.685870+0000","last_became_peered":"2024-09-10T19:10:07.685870+0000","last_unstale":"2024-09-10T19:26:43.209803+0000","last_undegraded":"2024-09-10T19:26:43.209803+0000","last_fullsized":"2024-09-10T19:26:43.209803+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:25:14.758809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:25:14.758809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.002869814,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"303'15","reported_seq":609,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-09-10T19:26:50.928391+0000","last_change":"2024-09-10T19:24:47.487930+0000","last_active":"2024-09-10T19:26:50.928391+0000","last_peered":"2024-09-10T19:26:50.928391+0000","last_clean":"2024-09-10T19:26:50.928391+0000","last_became_active":"2024-09-10T19:24:24.917099+0000","last_became_peered":"2024-09-10T19:24:24.917099+0000","last_unstale":"2024-09-10T19:26:50.928391+0000","last_undegraded":"2024-09-10T19:26:50.928391+0000","last_fullsized":"2024-09-10T19:26:50.928391+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"303'15","last_scrub_stamp":"2024-09-10T19:24:47.487840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:24:47.487840+0000","objects_scrubbed":1,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0054748990000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":303,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-09-10T19:26:38.189735+0000","last_change":"2024-09-10T19:25:19.396237+0000","last_active":"2024-09-10T19:26:38.189735+0000","last_peered":"2024-09-10T19:26:38.189735+0000","last_clean":"2024-09-10T19:26:38.189735+0000","last_became_active":"2024-09-10T19:24:23.900513+0000","last_became_peered":"2024-09-10T19:24:23.900513+0000","last_unstale":"2024-09-10T19:26:38.189735+0000","last_undegraded":"2024-09-10T19:26:38.189735+0000","last_fullsized":"2024-09-10T19:26:38.189735+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:25:19.396159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:25:19.396159+0000","last_clean_scrub_stamp":"2024-09-10T19:25:19.396159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0046456329999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"237'15","reported_seq":416,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-09-10T19:26:43.212987+0000","last_change":"2024-09-10T19:25:16.712492+0000","last_active":"2024-09-10T19:26:43.212987+0000","last_peered":"2024-09-10T19:26:43.212987+0000","last_clean":"2024-09-10T19:26:43.212987+0000","last_became_active":"2024-09-10T19:16:24.859740+0000","last_became_peered":"2024-09-10T19:16:24.859740+0000","last_unstale":"2024-09-10T19:26:43.212987+0000","last_undegraded":"2024-09-10T19:26:43.212987+0000","last_fullsized":"2024-09-10T19:26:43.212987+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"237'15","last_scrub_stamp":"2024-09-10T19:25:16.712402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:25:16.712402+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0031776310000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":309,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-09-10T19:26:38.188809+0000","last_change":"2024-09-10T19:25:20.358819+0000","last_active":"2024-09-10T19:26:38.188809+0000","last_peered":"2024-09-10T19:26:38.188809+0000","last_clean":"2024-09-10T19:26:38.188809+0000","last_became_active":"2024-09-10T19:24:23.895163+0000","last_became_peered":"2024-09-10T19:24:23.895163+0000","last_unstale":"2024-09-10T19:26:38.188809+0000","last_undegraded":"2024-09-10T19:26:38.188809+0000","last_fullsized":"2024-09-10T19:26:38.188809+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:25:20.358737+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:25:20.358737+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.002668761,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"297'11","reported_seq":597,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-09-10T19:26:50.928438+0000","last_change":"2024-09-10T19:26:16.945209+0000","last_active":"2024-09-10T19:26:50.928438+0000","last_peered":"2024-09-10T19:26:50.928438+0000","last_clean":"2024-09-10T19:26:50.928438+0000","last_became_active":"2024-09-10T19:24:01.974816+0000","last_became_peered":"2024-09-10T19:24:01.974816+0000","last_unstale":"2024-09-10T19:26:50.928438+0000","last_undegraded":"2024-09-10T19:26:50.928438+0000","last_fullsized":"2024-09-10T19:26:50.928438+0000","mapping_epoch":292,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":293,"parent":"0.0","parent_split_bits":0,"last_scrub":"297'11","last_scrub_stamp":"2024-09-10T19:26:16.945115+0000","last_deep_scrub":"297'11","last_deep_scrub_stamp":"2024-09-10T19:24:13.318130+0000","last_clean_scrub_stamp":"2024-09-10T19:26:16.945115+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:27:37.714654+0000","scrub_duration":0.0051112889999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":366,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-09-10T19:26:43.204940+0000","last_change":"2024-09-10T19:24:37.671881+0000","last_active":"2024-09-10T19:26:43.204940+0000","last_peered":"2024-09-10T19:26:43.204940+0000","last_clean":"2024-09-10T19:26:43.204940+0000","last_became_active":"2024-09-10T19:24:23.903779+0000","last_became_peered":"2024-09-10T19:24:23.903779+0000","last_unstale":"2024-09-10T19:26:43.204940+0000","last_undegraded":"2024-09-10T19:26:43.204940+0000","last_fullsized":"2024-09-10T19:26:43.204940+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:24:37.671797+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:24:37.671797+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.002698401,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.12","version":"237'19","reported_seq":104,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-09-10T19:26:43.194836+0000","last_change":"2024-09-10T19:24:47.503877+0000","last_active":"2024-09-10T19:26:43.194836+0000","last_peered":"2024-09-10T19:26:43.194836+0000","last_clean":"2024-09-10T19:26:43.194836+0000","last_became_active":"2024-09-10T19:24:24.913130+0000","last_became_peered":"2024-09-10T19:24:24.913130+0000","last_unstale":"2024-09-10T19:26:43.194836+0000","last_undegraded":"2024-09-10T19:26:43.194836+0000","last_fullsized":"2024-09-10T19:26:43.194836+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"237'19","last_scrub_stamp":"2024-09-10T19:24:47.503819+0000","last_deep_scrub":"79'13","last_deep_scrub_stamp":"2024-09-10T19:11:22.072400+0000","last_clean_scrub_stamp":"2024-09-10T19:24:47.503819+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0097974459999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"59'1","reported_seq":380,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-09-10T19:26:43.202020+0000","last_change":"2024-09-10T19:25:17.225875+0000","last_active":"2024-09-10T19:26:43.202020+0000","last_peered":"2024-09-10T19:26:43.202020+0000","last_clean":"2024-09-10T19:26:43.202020+0000","last_became_active":"2024-09-10T19:10:07.683263+0000","last_became_peered":"2024-09-10T19:10:07.683263+0000","last_unstale":"2024-09-10T19:26:43.202020+0000","last_undegraded":"2024-09-10T19:26:43.202020+0000","last_fullsized":"2024-09-10T19:26:43.202020+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'1","last_scrub_stamp":"2024-09-10T19:25:17.225794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:25:17.225794+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0095600029999999992,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":362,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":397,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-09-10T19:26:43.198745+0000","last_change":"2024-09-10T19:26:23.842995+0000","last_active":"2024-09-10T19:26:43.198745+0000","last_peered":"2024-09-10T19:26:43.198745+0000","last_clean":"2024-09-10T19:26:43.198745+0000","last_became_active":"2024-09-10T19:24:24.923804+0000","last_became_peered":"2024-09-10T19:24:24.923804+0000","last_unstale":"2024-09-10T19:26:43.198745+0000","last_undegraded":"2024-09-10T19:26:43.198745+0000","last_fullsized":"2024-09-10T19:26:43.198745+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:26:23.842943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:26:23.842943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:27:48.719400+0000","scrub_duration":0.0015653100000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":322,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-09-10T19:26:38.185703+0000","last_change":"2024-09-10T19:25:27.412632+0000","last_active":"2024-09-10T19:26:38.185703+0000","last_peered":"2024-09-10T19:26:38.185703+0000","last_clean":"2024-09-10T19:26:38.185703+0000","last_became_active":"2024-09-10T19:24:24.916105+0000","last_became_peered":"2024-09-10T19:24:24.916105+0000","last_unstale":"2024-09-10T19:26:38.185703+0000","last_undegraded":"2024-09-10T19:26:38.185703+0000","last_fullsized":"2024-09-10T19:26:38.185703+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:25:27.412579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:25:27.412579+0000","last_clean_scrub_stamp":"2024-09-10T19:25:27.412579+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0084519790000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"237'19","reported_seq":445,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-09-10T19:26:43.197983+0000","last_change":"2024-09-10T19:26:34.043973+0000","last_active":"2024-09-10T19:26:43.197983+0000","last_peered":"2024-09-10T19:26:43.197983+0000","last_clean":"2024-09-10T19:26:43.197983+0000","last_became_active":"2024-09-10T19:24:01.975520+0000","last_became_peered":"2024-09-10T19:24:01.975520+0000","last_unstale":"2024-09-10T19:26:43.197983+0000","last_undegraded":"2024-09-10T19:26:43.197983+0000","last_fullsized":"2024-09-10T19:26:43.197983+0000","mapping_epoch":292,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":293,"parent":"0.0","parent_split_bits":0,"last_scrub":"237'19","last_scrub_stamp":"2024-09-10T19:26:34.043784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:26:34.043784+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:27:34.264062+0000","scrub_duration":0.0096839449999999994,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":104,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-09-10T19:26:43.193443+0000","last_change":"2024-09-10T19:26:31.508255+0000","last_active":"2024-09-10T19:26:43.193443+0000","last_peered":"2024-09-10T19:26:43.193443+0000","last_clean":"2024-09-10T19:26:43.193443+0000","last_became_active":"2024-09-10T19:24:24.909540+0000","last_became_peered":"2024-09-10T19:24:24.909540+0000","last_unstale":"2024-09-10T19:26:43.193443+0000","last_undegraded":"2024-09-10T19:26:43.193443+0000","last_fullsized":"2024-09-10T19:26:43.193443+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:26:31.508168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:26:31.508168+0000","last_clean_scrub_stamp":"2024-09-10T19:26:31.508168+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:27:44.367352+0000","scrub_duration":0.069232562999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":80,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-09-10T19:26:43.195064+0000","last_change":"2024-09-10T19:26:16.473660+0000","last_active":"2024-09-10T19:26:43.195064+0000","last_peered":"2024-09-10T19:26:43.195064+0000","last_clean":"2024-09-10T19:26:43.195064+0000","last_became_active":"2024-09-10T19:24:01.975520+0000","last_became_peered":"2024-09-10T19:24:01.975520+0000","last_unstale":"2024-09-10T19:26:43.195064+0000","last_undegraded":"2024-09-10T19:26:43.195064+0000","last_fullsized":"2024-09-10T19:26:43.195064+0000","mapping_epoch":292,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":293,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:26:16.473571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:26:16.473571+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:27:33.963609+0000","scrub_duration":0.0030308330000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":364,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-09-10T19:26:43.197652+0000","last_change":"2024-09-10T19:26:33.897629+0000","last_active":"2024-09-10T19:26:43.197652+0000","last_peered":"2024-09-10T19:26:43.197652+0000","last_clean":"2024-09-10T19:26:43.197652+0000","last_became_active":"2024-09-10T19:24:23.907136+0000","last_became_peered":"2024-09-10T19:24:23.907136+0000","last_unstale":"2024-09-10T19:26:43.197652+0000","last_undegraded":"2024-09-10T19:26:43.197652+0000","last_fullsized":"2024-09-10T19:26:43.197652+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:26:33.897552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:26:33.897552+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:27:59.638065+0000","scrub_duration":0.0025392180000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"237'10","reported_seq":404,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-09-10T19:26:43.197559+0000","last_change":"2024-09-10T19:24:53.568499+0000","last_active":"2024-09-10T19:26:43.197559+0000","last_peered":"2024-09-10T19:26:43.197559+0000","last_clean":"2024-09-10T19:26:43.197559+0000","last_became_active":"2024-09-10T19:24:01.970758+0000","last_became_peered":"2024-09-10T19:24:01.970758+0000","last_unstale":"2024-09-10T19:26:43.197559+0000","last_undegraded":"2024-09-10T19:26:43.197559+0000","last_fullsized":"2024-09-10T19:26:43.197559+0000","mapping_epoch":292,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":293,"parent":"0.0","parent_split_bits":0,"last_scrub":"237'10","last_scrub_stamp":"2024-09-10T19:24:53.568298+0000","last_deep_scrub":"222'6","last_deep_scrub_stamp":"2024-09-10T19:19:30.328841+0000","last_clean_scrub_stamp":"2024-09-10T19:24:53.568298+0000","objects_scrubbed":2,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.002398168,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":12,"num_read_kb":10,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":383,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-09-10T19:26:43.215481+0000","last_change":"2024-09-10T19:26:32.839813+0000","last_active":"2024-09-10T19:26:43.215481+0000","last_peered":"2024-09-10T19:26:43.215481+0000","last_clean":"2024-09-10T19:26:43.215481+0000","last_became_active":"2024-09-10T19:24:01.978534+0000","last_became_peered":"2024-09-10T19:24:01.978534+0000","last_unstale":"2024-09-10T19:26:43.215481+0000","last_undegraded":"2024-09-10T19:26:43.215481+0000","last_fullsized":"2024-09-10T19:26:43.215481+0000","mapping_epoch":292,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":293,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:26:32.839727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:26:32.839727+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:27:39.076381+0000","scrub_duration":0.0064300950000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,1],"acting":[5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":369,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-09-10T19:26:43.201547+0000","last_change":"2024-09-10T19:24:42.823431+0000","last_active":"2024-09-10T19:26:43.201547+0000","last_peered":"2024-09-10T19:26:43.201547+0000","last_clean":"2024-09-10T19:26:43.201547+0000","last_became_active":"2024-09-10T19:10:11.698536+0000","last_became_peered":"2024-09-10T19:10:11.698536+0000","last_unstale":"2024-09-10T19:26:43.201547+0000","last_undegraded":"2024-09-10T19:26:43.201547+0000","last_fullsized":"2024-09-10T19:26:43.201547+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:24:42.823297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:11:31.837944+0000","last_clean_scrub_stamp":"2024-09-10T19:24:42.823297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0052557389999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":377,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-09-10T19:26:43.201644+0000","last_change":"2024-09-10T19:25:16.770678+0000","last_active":"2024-09-10T19:26:43.201644+0000","last_peered":"2024-09-10T19:26:43.201644+0000","last_clean":"2024-09-10T19:26:43.201644+0000","last_became_active":"2024-09-10T19:16:24.852356+0000","last_became_peered":"2024-09-10T19:16:24.852356+0000","last_unstale":"2024-09-10T19:26:43.201644+0000","last_undegraded":"2024-09-10T19:26:43.201644+0000","last_fullsized":"2024-09-10T19:26:43.201644+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:25:16.770592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:25:16.770592+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001878469,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,3],"acting":[7,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.11","version":"237'19","reported_seq":153,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-09-10T19:26:43.197475+0000","last_change":"2024-09-10T19:26:31.962670+0000","last_active":"2024-09-10T19:26:43.197475+0000","last_peered":"2024-09-10T19:26:43.197475+0000","last_clean":"2024-09-10T19:26:43.197475+0000","last_became_active":"2024-09-10T19:21:04.169444+0000","last_became_peered":"2024-09-10T19:21:04.169444+0000","last_unstale":"2024-09-10T19:26:43.197475+0000","last_undegraded":"2024-09-10T19:26:43.197475+0000","last_fullsized":"2024-09-10T19:26:43.197475+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"237'19","last_scrub_stamp":"2024-09-10T19:26:31.962585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:26:31.962585+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:27:39.609732+0000","scrub_duration":0.0022806200000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":390,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-09-10T19:26:43.204021+0000","last_change":"2024-09-10T19:26:08.395741+0000","last_active":"2024-09-10T19:26:43.204021+0000","last_peered":"2024-09-10T19:26:43.204021+0000","last_clean":"2024-09-10T19:26:43.204021+0000","last_became_active":"2024-09-10T19:24:01.974565+0000","last_became_peered":"2024-09-10T19:24:01.974565+0000","last_unstale":"2024-09-10T19:26:43.204021+0000","last_undegraded":"2024-09-10T19:26:43.204021+0000","last_fullsized":"2024-09-10T19:26:43.204021+0000","mapping_epoch":292,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":293,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:26:08.395687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:13:05.876655+0000","last_clean_scrub_stamp":"2024-09-10T19:26:08.395687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:27:08.537061+0000","scrub_duration":0.0050263720000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":359,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-09-10T19:26:43.203163+0000","last_change":"2024-09-10T19:25:23.767085+0000","last_active":"2024-09-10T19:26:43.203163+0000","last_peered":"2024-09-10T19:26:43.203163+0000","last_clean":"2024-09-10T19:26:43.203163+0000","last_became_active":"2024-09-10T19:10:11.699013+0000","last_became_peered":"2024-09-10T19:10:11.699013+0000","last_unstale":"2024-09-10T19:26:43.203163+0000","last_undegraded":"2024-09-10T19:26:43.203163+0000","last_fullsized":"2024-09-10T19:26:43.203163+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:25:23.766985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:11:57.112092+0000","last_clean_scrub_stamp":"2024-09-10T19:25:23.766985+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0068928649999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":364,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-09-10T19:26:43.198392+0000","last_change":"2024-09-10T19:26:29.959285+0000","last_active":"2024-09-10T19:26:43.198392+0000","last_peered":"2024-09-10T19:26:43.198392+0000","last_clean":"2024-09-10T19:26:43.198392+0000","last_became_active":"2024-09-10T19:24:23.896454+0000","last_became_peered":"2024-09-10T19:24:23.896454+0000","last_unstale":"2024-09-10T19:26:43.198392+0000","last_undegraded":"2024-09-10T19:26:43.198392+0000","last_fullsized":"2024-09-10T19:26:43.198392+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:26:29.959202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:26:29.959202+0000","last_clean_scrub_stamp":"2024-09-10T19:26:29.959202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:27:48.266668+0000","scrub_duration":0.0059040539999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"80'2","reported_seq":383,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-09-10T19:26:43.202800+0000","last_change":"2024-09-10T19:26:16.224098+0000","last_active":"2024-09-10T19:26:43.202800+0000","last_peered":"2024-09-10T19:26:43.202800+0000","last_clean":"2024-09-10T19:26:43.202800+0000","last_became_active":"2024-09-10T19:16:24.851220+0000","last_became_peered":"2024-09-10T19:16:24.851220+0000","last_unstale":"2024-09-10T19:26:43.202800+0000","last_undegraded":"2024-09-10T19:26:43.202800+0000","last_fullsized":"2024-09-10T19:26:43.202800+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"80'2","last_scrub_stamp":"2024-09-10T19:26:16.223955+0000","last_deep_scrub":"80'2","last_deep_scrub_stamp":"2024-09-10T19:11:30.272043+0000","last_clean_scrub_stamp":"2024-09-10T19:26:16.223955+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:27:24.628267+0000","scrub_duration":0.012727367999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":369,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":369,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,3],"acting":[7,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":370,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-09-10T19:26:43.210940+0000","last_change":"2024-09-10T19:24:41.498889+0000","last_active":"2024-09-10T19:26:43.210940+0000","last_peered":"2024-09-10T19:26:43.210940+0000","last_clean":"2024-09-10T19:26:43.210940+0000","last_became_active":"2024-09-10T19:24:23.906812+0000","last_became_peered":"2024-09-10T19:24:23.906812+0000","last_unstale":"2024-09-10T19:26:43.210940+0000","last_undegraded":"2024-09-10T19:26:43.210940+0000","last_fullsized":"2024-09-10T19:26:43.210940+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:24:41.498807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:24:41.498807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.010867098,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,0],"acting":[5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1e","version":"237'25","reported_seq":342,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-09-10T19:26:38.187017+0000","last_change":"2024-09-10T19:25:15.471102+0000","last_active":"2024-09-10T19:26:38.187017+0000","last_peered":"2024-09-10T19:26:38.187017+0000","last_clean":"2024-09-10T19:26:38.187017+0000","last_became_active":"2024-09-10T19:24:23.899507+0000","last_became_peered":"2024-09-10T19:24:23.899507+0000","last_unstale":"2024-09-10T19:26:38.187017+0000","last_undegraded":"2024-09-10T19:26:38.187017+0000","last_fullsized":"2024-09-10T19:26:38.187017+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"237'25","last_scrub_stamp":"2024-09-10T19:25:15.471043+0000","last_deep_scrub":"80'15","last_deep_scrub_stamp":"2024-09-10T19:11:25.883919+0000","last_clean_scrub_stamp":"2024-09-10T19:25:15.471043+0000","objects_scrubbed":5,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0027617729999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":24,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1f","version":"66'8","reported_seq":584,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-09-10T19:26:50.928579+0000","last_change":"2024-09-10T19:24:46.897327+0000","last_active":"2024-09-10T19:26:50.928579+0000","last_peered":"2024-09-10T19:26:50.928579+0000","last_clean":"2024-09-10T19:26:50.928579+0000","last_became_active":"2024-09-10T19:10:11.695427+0000","last_became_peered":"2024-09-10T19:10:11.695427+0000","last_unstale":"2024-09-10T19:26:50.928579+0000","last_undegraded":"2024-09-10T19:26:50.928579+0000","last_fullsized":"2024-09-10T19:26:50.928579+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'8","last_scrub_stamp":"2024-09-10T19:24:46.897253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:24:46.897253+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0048746450000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":80,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-09-10T19:26:43.193568+0000","last_change":"2024-09-10T19:25:18.764190+0000","last_active":"2024-09-10T19:26:43.193568+0000","last_peered":"2024-09-10T19:26:43.193568+0000","last_clean":"2024-09-10T19:26:43.193568+0000","last_became_active":"2024-09-10T19:24:01.980391+0000","last_became_peered":"2024-09-10T19:24:01.980391+0000","last_unstale":"2024-09-10T19:26:43.193568+0000","last_undegraded":"2024-09-10T19:26:43.193568+0000","last_fullsized":"2024-09-10T19:26:43.193568+0000","mapping_epoch":292,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":293,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:25:18.764101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:25:18.764101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0082808180000000006,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":380,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-09-10T19:26:43.198035+0000","last_change":"2024-09-10T19:26:36.923605+0000","last_active":"2024-09-10T19:26:43.198035+0000","last_peered":"2024-09-10T19:26:43.198035+0000","last_clean":"2024-09-10T19:26:43.198035+0000","last_became_active":"2024-09-10T19:24:23.905869+0000","last_became_peered":"2024-09-10T19:24:23.905869+0000","last_unstale":"2024-09-10T19:26:43.198035+0000","last_undegraded":"2024-09-10T19:26:43.198035+0000","last_fullsized":"2024-09-10T19:26:43.198035+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:26:36.923520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:26:36.923520+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:28:05.745115+0000","scrub_duration":0.001294981,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"237'24","reported_seq":476,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-09-10T19:26:43.203948+0000","last_change":"2024-09-10T19:25:31.311744+0000","last_active":"2024-09-10T19:26:43.203948+0000","last_peered":"2024-09-10T19:26:43.203948+0000","last_clean":"2024-09-10T19:26:43.203948+0000","last_became_active":"2024-09-10T19:24:01.974988+0000","last_became_peered":"2024-09-10T19:24:01.974988+0000","last_unstale":"2024-09-10T19:26:43.203948+0000","last_undegraded":"2024-09-10T19:26:43.203948+0000","last_fullsized":"2024-09-10T19:26:43.203948+0000","mapping_epoch":292,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":293,"parent":"0.0","parent_split_bits":0,"last_scrub":"237'24","last_scrub_stamp":"2024-09-10T19:25:31.311493+0000","last_deep_scrub":"80'16","last_deep_scrub_stamp":"2024-09-10T19:11:40.853611+0000","last_clean_scrub_stamp":"2024-09-10T19:25:31.311493+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:26:53.477394+0000","scrub_duration":0.0054020810000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":297,"reported_epoch":351,"state":"active+clean","last_fresh":"2024-09-10T19:26:38.186936+0000","last_change":"2024-09-10T19:26:19.418628+0000","last_active":"2024-09-10T19:26:38.186936+0000","last_peered":"2024-09-10T19:26:38.186936+0000","last_clean":"2024-09-10T19:26:38.186936+0000","last_became_active":"2024-09-10T19:24:24.899153+0000","last_became_peered":"2024-09-10T19:24:24.899153+0000","last_unstale":"2024-09-10T19:26:38.186936+0000","last_undegraded":"2024-09-10T19:26:38.186936+0000","last_fullsized":"2024-09-10T19:26:38.186936+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:26:19.418556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:11:28.822327+0000","last_clean_scrub_stamp":"2024-09-10T19:26:19.418556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:27:48.413741+0000","scrub_duration":0.005516361,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":366,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-09-10T19:26:43.199809+0000","last_change":"2024-09-10T19:26:30.969045+0000","last_active":"2024-09-10T19:26:43.199809+0000","last_peered":"2024-09-10T19:26:43.199809+0000","last_clean":"2024-09-10T19:26:43.199809+0000","last_became_active":"2024-09-10T19:10:14.053619+0000","last_became_peered":"2024-09-10T19:10:14.053619+0000","last_unstale":"2024-09-10T19:26:43.199809+0000","last_undegraded":"2024-09-10T19:26:43.199809+0000","last_fullsized":"2024-09-10T19:26:43.199809+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:26:30.968972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:26:30.968972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:28:00.935626+0000","scrub_duration":0.0072260129999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":373,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-09-10T19:26:43.204105+0000","last_change":"2024-09-10T19:25:26.260593+0000","last_active":"2024-09-10T19:26:43.204105+0000","last_peered":"2024-09-10T19:26:43.204105+0000","last_clean":"2024-09-10T19:26:43.204105+0000","last_became_active":"2024-09-10T19:10:07.685997+0000","last_became_peered":"2024-09-10T19:10:07.685997+0000","last_unstale":"2024-09-10T19:26:43.204105+0000","last_undegraded":"2024-09-10T19:26:43.204105+0000","last_fullsized":"2024-09-10T19:26:43.204105+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:25:26.260513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:25:26.260513+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0073960600000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1c","version":"237'34","reported_seq":110,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-09-10T19:26:43.193224+0000","last_change":"2024-09-10T19:24:44.441810+0000","last_active":"2024-09-10T19:26:43.193224+0000","last_peered":"2024-09-10T19:26:43.193224+0000","last_clean":"2024-09-10T19:26:43.193224+0000","last_became_active":"2024-09-10T19:24:24.910577+0000","last_became_peered":"2024-09-10T19:24:24.910577+0000","last_unstale":"2024-09-10T19:26:43.193224+0000","last_undegraded":"2024-09-10T19:26:43.193224+0000","last_fullsized":"2024-09-10T19:26:43.193224+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"237'34","last_scrub_stamp":"2024-09-10T19:24:44.441712+0000","last_deep_scrub":"78'22","last_deep_scrub_stamp":"2024-09-10T19:11:21.118937+0000","last_clean_scrub_stamp":"2024-09-10T19:24:44.441712+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0042000179999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":94,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-09-10T19:26:43.194373+0000","last_change":"2024-09-10T19:24:35.438665+0000","last_active":"2024-09-10T19:26:43.194373+0000","last_peered":"2024-09-10T19:26:43.194373+0000","last_clean":"2024-09-10T19:26:43.194373+0000","last_became_active":"2024-09-10T19:24:24.887205+0000","last_became_peered":"2024-09-10T19:24:24.887205+0000","last_unstale":"2024-09-10T19:26:43.194373+0000","last_undegraded":"2024-09-10T19:26:43.194373+0000","last_fullsized":"2024-09-10T19:26:43.194373+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:24:35.438586+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:24:35.438586+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0036432579999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":360,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-09-10T19:26:43.199045+0000","last_change":"2024-09-10T19:26:30.990896+0000","last_active":"2024-09-10T19:26:43.199045+0000","last_peered":"2024-09-10T19:26:43.199045+0000","last_clean":"2024-09-10T19:26:43.199045+0000","last_became_active":"2024-09-10T19:10:14.053698+0000","last_became_peered":"2024-09-10T19:10:14.053698+0000","last_unstale":"2024-09-10T19:26:43.199045+0000","last_undegraded":"2024-09-10T19:26:43.199045+0000","last_fullsized":"2024-09-10T19:26:43.199045+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:26:30.990819+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:24:46.518816+0000","last_clean_scrub_stamp":"2024-09-10T19:26:30.990819+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:27:45.420255+0000","scrub_duration":0.0073104629999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":363,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-09-10T19:26:43.198626+0000","last_change":"2024-09-10T19:26:13.910830+0000","last_active":"2024-09-10T19:26:43.198626+0000","last_peered":"2024-09-10T19:26:43.198626+0000","last_clean":"2024-09-10T19:26:43.198626+0000","last_became_active":"2024-09-10T19:10:07.682811+0000","last_became_peered":"2024-09-10T19:10:07.682811+0000","last_unstale":"2024-09-10T19:26:43.198626+0000","last_undegraded":"2024-09-10T19:26:43.198626+0000","last_fullsized":"2024-09-10T19:26:43.198626+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:26:13.910732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:26:13.910732+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:27:32.512695+0000","scrub_duration":0.0067593929999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1d","version":"250'23","reported_seq":439,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-09-10T19:26:43.198594+0000","last_change":"2024-09-10T19:25:31.852135+0000","last_active":"2024-09-10T19:26:43.198594+0000","last_peered":"2024-09-10T19:26:43.198594+0000","last_clean":"2024-09-10T19:26:43.198594+0000","last_became_active":"2024-09-10T19:24:01.976664+0000","last_became_peered":"2024-09-10T19:24:01.976664+0000","last_unstale":"2024-09-10T19:26:43.198594+0000","last_undegraded":"2024-09-10T19:26:43.198594+0000","last_fullsized":"2024-09-10T19:26:43.198594+0000","mapping_epoch":292,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":293,"parent":"0.0","parent_split_bits":0,"last_scrub":"250'23","last_scrub_stamp":"2024-09-10T19:25:31.852016+0000","last_deep_scrub":"250'23","last_deep_scrub_stamp":"2024-09-10T19:20:55.582173+0000","last_clean_scrub_stamp":"2024-09-10T19:25:31.852016+0000","objects_scrubbed":5,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0046609609999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":378,"reported_epoch":352,"state":"active+clean","last_fresh":"2024-09-10T19:26:43.199509+0000","last_change":"2024-09-10T19:26:29.937585+0000","last_active":"2024-09-10T19:26:43.199509+0000","last_peered":"2024-09-10T19:26:43.199509+0000","last_clean":"2024-09-10T19:26:43.199509+0000","last_became_active":"2024-09-10T19:16:24.843013+0000","last_became_peered":"2024-09-10T19:16:24.843013+0000","last_unstale":"2024-09-10T19:26:43.199509+0000","last_undegraded":"2024-09-10T19:26:43.199509+0000","last_fullsized":"2024-09-10T19:26:43.199509+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:26:29.937532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:26:29.937532+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:27:55.254997+0000","scrub_duration":0.0024737750000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":10,"num_pg":16,"stat_sum":{"num_bytes":2554331160,"num_objects":38977,"num_object_clones":0,"num_object_copies":116931,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":38977,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":207435,"num_write_kb":7885121,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4680,"num_bytes_recovered":306708480,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":7654289408,"data_stored":7654277192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":71435,"ondisk_log_size":71435,"up":48,"acting":48,"num_store_stats":7},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":382,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":1528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":88,"ondisk_log_size":88,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":177,"num_object_clones":0,"num_object_copies":531,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":177,"num_whiteouts":0,"num_read":1117,"num_read_kb":874,"num_write":703,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":293,"num_bytes_recovered":5886,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":458752,"data_stored":12124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":764,"ondisk_log_size":764,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":179,"num_read_kb":174,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":778,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1613,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":9,"ondisk_log_size":9,"up":9,"acting":9,"num_store_stats":7},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1328,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":27,"num_read_kb":27,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":3984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4,"ondisk_log_size":4,"up":96,"acting":96,"num_store_stats":8},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":870,"ondisk_log_size":870,"up":33,"acting":33,"num_store_stats":7}],"osd_stats":[{"osd":7,"up_from":54,"seq":231928234190,"num_pgs":69,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1661976,"kb_used_data":773776,"kb_used_omap":1,"kb_used_meta":888126,"kb_avail":92083176,"statfs":{"total":95995035648,"available":94293172224,"internally_reserved":0,"allocated":792346624,"data_stored":792053687,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":909441368},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 19:26:26 2024","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.89200000000000002,"15min":0.94399999999999995},"min":{"1min":0.24399999999999999,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":2.6949999999999998,"5min":3,"15min":3},"last":0.73899999999999999},{"interface":"front","average":{"1min":1.036,"5min":0.75800000000000001,"15min":0.71199999999999997},"min":{"1min":0.33100000000000002,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":3.5409999999999999,"5min":3.8260000000000001,"15min":3.8260000000000001},"last":0.93200000000000005}]},{"osd":1,"last update":"Tue Sep 10 19:26:54 2024","interfaces":[{"interface":"back","average":{"1min":0.88,"5min":0.93000000000000005,"15min":1},"min":{"1min":0.24299999999999999,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":4.1139999999999999,"5min":5.8140000000000001,"15min":5.8140000000000001},"last":2.633},{"interface":"front","average":{"1min":1.2769999999999999,"5min":0.98099999999999998,"15min":0.88500000000000001},"min":{"1min":0.26700000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":4.8540000000000001,"5min":4.8540000000000001,"15min":4.8540000000000001},"last":1.147}]},{"osd":2,"last update":"Tue Sep 10 19:26:11 2024","interfaces":[{"interface":"back","average":{"1min":0.92400000000000004,"5min":0.89700000000000002,"15min":0.96699999999999997},"min":{"1min":0.38300000000000001,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":3.601,"5min":4.0190000000000001,"15min":31.940999999999999},"last":2.4750000000000001},{"interface":"front","average":{"1min":1.2649999999999999,"5min":0.95999999999999996,"15min":1.0780000000000001},"min":{"1min":0.32500000000000001,"5min":0.26300000000000001,"15min":0.24099999999999999},"max":{"1min":4.3150000000000004,"5min":4.4139999999999997,"15min":25.056000000000001},"last":1.2170000000000001}]},{"osd":3,"last update":"Tue Sep 10 19:26:11 2024","interfaces":[{"interface":"back","average":{"1min":0.90500000000000003,"5min":0.94199999999999995,"15min":1.331},"min":{"1min":0.38500000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":3.5790000000000002,"5min":4.6070000000000002,"15min":35.277999999999999},"last":2.5030000000000001},{"interface":"front","average":{"1min":1.073,"5min":0.92400000000000004,"15min":1.321},"min":{"1min":0.32500000000000001,"5min":0.27400000000000002,"15min":0.25700000000000001},"max":{"1min":4.1379999999999999,"5min":4.4500000000000002,"15min":36.512999999999998},"last":2.536}]},{"osd":4,"last update":"Tue Sep 10 19:26:11 2024","interfaces":[{"interface":"back","average":{"1min":1.8,"5min":4.5860000000000003,"15min":2.3650000000000002},"min":{"1min":0.49199999999999999,"5min":0.222,"15min":0.222},"max":{"1min":5.8600000000000003,"5min":231.923,"15min":231.923},"last":1.2470000000000001},{"interface":"front","average":{"1min":1.4239999999999999,"5min":4.4340000000000002,"15min":2.1000000000000001},"min":{"1min":0.34999999999999998,"5min":0.24399999999999999,"15min":0.23000000000000001},"max":{"1min":3.8959999999999999,"5min":231.322,"15min":231.322},"last":1.131}]},{"osd":5,"last update":"Tue Sep 10 19:26:11 2024","interfaces":[{"interface":"back","average":{"1min":1.409,"5min":2.0179999999999998,"15min":1.552},"min":{"1min":0.34999999999999998,"5min":0.24099999999999999,"15min":0.23699999999999999},"max":{"1min":4.3550000000000004,"5min":67.382999999999996,"15min":67.382999999999996},"last":1.1899999999999999},{"interface":"front","average":{"1min":1.8500000000000001,"5min":2.3679999999999999,"15min":1.7589999999999999},"min":{"1min":0.309,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":5.8789999999999996,"5min":67.75,"15min":67.75},"last":0.97699999999999998}]},{"osd":6,"last update":"Tue Sep 10 19:26:11 2024","interfaces":[{"interface":"back","average":{"1min":1.6830000000000001,"5min":3.8799999999999999,"15min":2.161},"min":{"1min":0.33000000000000002,"5min":0.28000000000000003,"15min":0.26200000000000001},"max":{"1min":6.7869999999999999,"5min":181.53800000000001,"15min":181.53800000000001},"last":1.3089999999999999},{"interface":"front","average":{"1min":1.171,"5min":3.4860000000000002,"15min":1.869},"min":{"1min":0.34499999999999997,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":3.9590000000000001,"5min":182.05600000000001,"15min":182.05600000000001},"last":1.2849999999999999}]}]},{"osd":6,"up_from":47,"seq":201863463123,"num_pgs":64,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1975724,"kb_used_data":1089248,"kb_used_omap":1,"kb_used_meta":886462,"kb_avail":91769428,"statfs":{"total":95995035648,"available":93971894272,"internally_reserved":0,"allocated":1115389952,"data_stored":1115081743,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":907737432},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 19:26:30 2024","interfaces":[{"interface":"back","average":{"1min":1.3049999999999999,"5min":1.76,"15min":1.8360000000000001},"min":{"1min":0.183,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":5.0410000000000004,"5min":18.545999999999999,"15min":18.545999999999999},"last":1.9279999999999999},{"interface":"front","average":{"1min":1.3879999999999999,"5min":1.8759999999999999,"15min":1.958},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":7.1020000000000003,"5min":19.818000000000001,"15min":19.818000000000001},"last":0.59399999999999997}]},{"osd":1,"last update":"Tue Sep 10 19:25:53 2024","interfaces":[{"interface":"back","average":{"1min":1.8380000000000001,"5min":0.88300000000000001,"15min":0.77600000000000002},"min":{"1min":0.23400000000000001,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":5.3540000000000001,"5min":5.3540000000000001,"15min":5.3540000000000001},"last":1.909},{"interface":"front","average":{"1min":0.75900000000000001,"5min":0.75600000000000001,"15min":0.754},"min":{"1min":0.21299999999999999,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":2.855,"5min":3.1040000000000001,"15min":3.1040000000000001},"last":0.624}]},{"osd":2,"last update":"Tue Sep 10 19:26:02 2024","interfaces":[{"interface":"back","average":{"1min":0.84499999999999997,"5min":0.81200000000000006,"15min":1.1819999999999999},"min":{"1min":0.29699999999999999,"5min":0.23799999999999999,"15min":0.20399999999999999},"max":{"1min":4.4459999999999997,"5min":5.9029999999999996,"15min":112.864},"last":0.64800000000000002},{"interface":"front","average":{"1min":1.409,"5min":0.90700000000000003,"15min":1.3},"min":{"1min":0.26200000000000001,"5min":0.25600000000000001,"15min":0.254},"max":{"1min":5.1920000000000002,"5min":5.1920000000000002,"15min":112.822},"last":0.33500000000000002}]},{"osd":3,"last update":"Tue Sep 10 19:26:02 2024","interfaces":[{"interface":"back","average":{"1min":2.044,"5min":0.96699999999999997,"15min":1.5189999999999999},"min":{"1min":0.32200000000000001,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":5.3230000000000004,"5min":5.3230000000000004,"15min":125.56399999999999},"last":1.8660000000000001},{"interface":"front","average":{"1min":1.0189999999999999,"5min":0.85799999999999998,"15min":1.4910000000000001},"min":{"1min":0.29199999999999998,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":4.3899999999999997,"5min":4.3899999999999997,"15min":125.47},"last":0.69999999999999996}]},{"osd":4,"last update":"Tue Sep 10 19:26:02 2024","interfaces":[{"interface":"back","average":{"1min":2.9180000000000001,"5min":1.5229999999999999,"15min":1.1259999999999999},"min":{"1min":0.32900000000000001,"5min":0.254,"15min":0.20699999999999999},"max":{"1min":7.7599999999999998,"5min":7.7599999999999998,"15min":7.7599999999999998},"last":0.875},{"interface":"front","average":{"1min":3.0070000000000001,"5min":1.4650000000000001,"15min":1.0569999999999999},"min":{"1min":0.41399999999999998,"5min":0.24099999999999999,"15min":0.22500000000000001},"max":{"1min":6.2549999999999999,"5min":8.6419999999999995,"15min":8.6419999999999995},"last":1.948}]},{"osd":5,"last update":"Tue Sep 10 19:26:02 2024","interfaces":[{"interface":"back","average":{"1min":1.8340000000000001,"5min":1.333,"15min":1.258},"min":{"1min":0.34999999999999998,"5min":0.28100000000000003,"15min":0.248},"max":{"1min":9.7270000000000003,"5min":9.7270000000000003,"15min":44.765000000000001},"last":0.45300000000000001},{"interface":"front","average":{"1min":2.605,"5min":1.5129999999999999,"15min":1.3720000000000001},"min":{"1min":0.36499999999999999,"5min":0.28399999999999997,"15min":0.27500000000000002},"max":{"1min":5.2309999999999999,"5min":6.9459999999999997,"15min":43.036000000000001},"last":1.8340000000000001}]},{"osd":7,"last update":"Tue Sep 10 19:26:30 2024","interfaces":[{"interface":"back","average":{"1min":2.3620000000000001,"5min":2.3479999999999999,"15min":1.3879999999999999},"min":{"1min":0.32000000000000001,"5min":0.28100000000000003,"15min":0.27500000000000002},"max":{"1min":5.6269999999999998,"5min":80.629999999999995,"15min":80.629999999999995},"last":0.67500000000000004},{"interface":"front","average":{"1min":2.7029999999999998,"5min":2.552,"15min":1.599},"min":{"1min":0.33800000000000002,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":16.654,"5min":81.088999999999999,"15min":81.088999999999999},"last":1.887}]}]},{"osd":5,"up_from":39,"seq":167503724760,"num_pgs":68,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1959456,"kb_used_data":1069848,"kb_used_omap":1,"kb_used_meta":889598,"kb_avail":91785696,"statfs":{"total":95995035648,"available":93988552704,"internally_reserved":0,"allocated":1095524352,"data_stored":1095224983,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":910948696},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 19:26:26 2024","interfaces":[{"interface":"back","average":{"1min":1.2709999999999999,"5min":1.151,"15min":1.131},"min":{"1min":0.253,"5min":0.187,"15min":0.187},"max":{"1min":3.5710000000000002,"5min":4.2220000000000004,"15min":4.2220000000000004},"last":0.44700000000000001},{"interface":"front","average":{"1min":0.93999999999999995,"5min":1.2190000000000001,"15min":1.2649999999999999},"min":{"1min":0.214,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":3.339,"5min":3.8580000000000001,"15min":3.8580000000000001},"last":2.1709999999999998}]},{"osd":1,"last update":"Tue Sep 10 19:26:46 2024","interfaces":[{"interface":"back","average":{"1min":0.84499999999999997,"5min":0.98199999999999998,"15min":0.98799999999999999},"min":{"1min":0.247,"5min":0.247,"15min":0.247},"max":{"1min":2.7250000000000001,"5min":8.2319999999999993,"15min":8.2319999999999993},"last":3.2450000000000001},{"interface":"front","average":{"1min":1.1399999999999999,"5min":1.0069999999999999,"15min":1.0449999999999999},"min":{"1min":0.37,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":3.605,"5min":3.9449999999999998,"15min":3.9449999999999998},"last":0.47699999999999998}]},{"osd":2,"last update":"Tue Sep 10 19:26:29 2024","interfaces":[{"interface":"back","average":{"1min":1.319,"5min":1.0880000000000001,"15min":1.046},"min":{"1min":0.47399999999999998,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":3.649,"5min":8.6920000000000002,"15min":9.9399999999999995},"last":0.52100000000000002},{"interface":"front","average":{"1min":0.90900000000000003,"5min":0.97899999999999998,"15min":1.0109999999999999},"min":{"1min":0.27700000000000002,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":2.8570000000000002,"5min":3.8319999999999999,"15min":6.3789999999999996},"last":2.1389999999999998}]},{"osd":3,"last update":"Tue Sep 10 19:26:29 2024","interfaces":[{"interface":"back","average":{"1min":1.2809999999999999,"5min":1.3999999999999999,"15min":1.131},"min":{"1min":0.442,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":3.629,"5min":26.355,"15min":26.355},"last":0.499},{"interface":"front","average":{"1min":0.97199999999999998,"5min":1.2949999999999999,"15min":1.091},"min":{"1min":0.30599999999999999,"5min":0.246,"15min":0.246},"max":{"1min":2.8359999999999999,"5min":26.327000000000002,"15min":26.327000000000002},"last":2.198}]},{"osd":4,"last update":"Tue Sep 10 19:26:29 2024","interfaces":[{"interface":"back","average":{"1min":1.718,"5min":2.9489999999999998,"15min":1.6879999999999999},"min":{"1min":0.55700000000000005,"5min":0.26100000000000001,"15min":0.23699999999999999},"max":{"1min":4.0039999999999996,"5min":128.08099999999999,"15min":128.08099999999999},"last":3.206},{"interface":"front","average":{"1min":1.8540000000000001,"5min":3.0859999999999999,"15min":1.792},"min":{"1min":0.54800000000000004,"5min":0.217,"15min":0.217},"max":{"1min":5.3109999999999999,"5min":126.321,"15min":126.321},"last":2.4590000000000001}]},{"osd":6,"last update":"Tue Sep 10 19:26:04 2024","interfaces":[{"interface":"back","average":{"1min":1.8520000000000001,"5min":2.6419999999999999,"15min":1.6719999999999999},"min":{"1min":0.33700000000000002,"5min":0.28399999999999997,"15min":0.27100000000000002},"max":{"1min":8.3339999999999996,"5min":111.328,"15min":111.328},"last":2.4870000000000001},{"interface":"front","average":{"1min":2.5489999999999999,"5min":2.7250000000000001,"15min":1.655},"min":{"1min":0.35299999999999998,"5min":0.27200000000000002,"15min":0.26800000000000002},"max":{"1min":7.2910000000000004,"5min":111.705,"15min":111.705},"last":2.0819999999999999}]},{"osd":7,"last update":"Tue Sep 10 19:26:22 2024","interfaces":[{"interface":"back","average":{"1min":1.405,"5min":1.4810000000000001,"15min":1.2410000000000001},"min":{"1min":0.29799999999999999,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":3.2799999999999998,"5min":6.7869999999999999,"15min":6.7869999999999999},"last":2.1070000000000002},{"interface":"front","average":{"1min":1.29,"5min":1.4390000000000001,"15min":1.2150000000000001},"min":{"1min":0.32800000000000001,"5min":0.29299999999999998,"15min":0.27100000000000002},"max":{"1min":4.0369999999999999,"5min":8.1679999999999993,"15min":9.4909999999999997},"last":3.2759999999999998}]}]},{"osd":4,"up_from":32,"seq":137438953692,"num_pgs":71,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2585824,"kb_used_data":1689112,"kb_used_omap":1,"kb_used_meta":896702,"kb_avail":91159328,"statfs":{"total":95995035648,"available":93347151872,"internally_reserved":0,"allocated":1729650688,"data_stored":1729349434,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":918223192},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 19:26:25 2024","interfaces":[{"interface":"back","average":{"1min":4.008,"5min":2.6360000000000001,"15min":2.407},"min":{"1min":0.221,"5min":0.221,"15min":0.221},"max":{"1min":39.097000000000001,"5min":39.097000000000001,"15min":39.097000000000001},"last":0.25900000000000001},{"interface":"front","average":{"1min":4.1109999999999998,"5min":2.9199999999999999,"15min":2.722},"min":{"1min":0.20499999999999999,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":39.112000000000002,"5min":39.112000000000002,"15min":39.112000000000002},"last":3.8239999999999998}]},{"osd":1,"last update":"Tue Sep 10 19:26:48 2024","interfaces":[{"interface":"back","average":{"1min":1.5229999999999999,"5min":1.0249999999999999,"15min":0.81100000000000005},"min":{"1min":0.20599999999999999,"5min":0.20599999999999999,"15min":0.20599999999999999},"max":{"1min":4.7960000000000003,"5min":4.7960000000000003,"15min":4.7960000000000003},"last":3.8690000000000002},{"interface":"front","average":{"1min":1.548,"5min":0.88100000000000001,"15min":0.69799999999999995},"min":{"1min":0.23400000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":5.7519999999999998,"5min":5.7519999999999998,"15min":5.7519999999999998},"last":3.8460000000000001}]},{"osd":2,"last update":"Tue Sep 10 19:26:21 2024","interfaces":[{"interface":"back","average":{"1min":1.6679999999999999,"5min":1.0189999999999999,"15min":0.91400000000000003},"min":{"1min":0.27000000000000002,"5min":0.25600000000000001,"15min":0.214},"max":{"1min":4.8410000000000002,"5min":4.8410000000000002,"15min":6.8869999999999996},"last":3.9100000000000001},{"interface":"front","average":{"1min":1.6839999999999999,"5min":1.032,"15min":0.90400000000000003},"min":{"1min":0.30199999999999999,"5min":0.25800000000000001,"15min":0.23400000000000001},"max":{"1min":5.0410000000000004,"5min":5.0410000000000004,"15min":7.085},"last":3.9700000000000002}]},{"osd":3,"last update":"Tue Sep 10 19:26:21 2024","interfaces":[{"interface":"back","average":{"1min":1.48,"5min":0.99299999999999999,"15min":1.1080000000000001},"min":{"1min":0.27800000000000002,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":5.0179999999999998,"5min":5.0179999999999998,"15min":35.451999999999998},"last":4.0069999999999997},{"interface":"front","average":{"1min":1.7509999999999999,"5min":0.96399999999999997,"15min":1.1659999999999999},"min":{"1min":0.28599999999999998,"5min":0.246,"15min":0.246},"max":{"1min":5.8949999999999996,"5min":5.8949999999999996,"15min":32.880000000000003},"last":0.496}]},{"osd":5,"last update":"Tue Sep 10 19:26:40 2024","interfaces":[{"interface":"back","average":{"1min":1.8129999999999999,"5min":4.5030000000000001,"15min":2.4700000000000002},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.224},"max":{"1min":5.9859999999999998,"5min":227.726,"15min":227.726},"last":0.52900000000000003},{"interface":"front","average":{"1min":2.254,"5min":4.6210000000000004,"15min":2.4340000000000002},"min":{"1min":0.27800000000000002,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":5.4859999999999998,"5min":227.70500000000001,"15min":227.70500000000001},"last":3.988}]},{"osd":6,"last update":"Tue Sep 10 19:26:17 2024","interfaces":[{"interface":"back","average":{"1min":2.718,"5min":1.472,"15min":1.0980000000000001},"min":{"1min":0.47099999999999997,"5min":0.27800000000000002,"15min":0.253},"max":{"1min":8.1010000000000009,"5min":8.1010000000000009,"15min":8.1010000000000009},"last":3.9319999999999999},{"interface":"front","average":{"1min":3.484,"5min":1.5329999999999999,"15min":1.1359999999999999},"min":{"1min":0.26400000000000001,"5min":0.26400000000000001,"15min":0.247},"max":{"1min":16.815000000000001,"5min":16.815000000000001,"15min":16.815000000000001},"last":0.55500000000000005}]},{"osd":7,"last update":"Tue Sep 10 19:26:25 2024","interfaces":[{"interface":"back","average":{"1min":2.2000000000000002,"5min":1.46,"15min":1.284},"min":{"1min":0.33100000000000002,"5min":0.311,"15min":0.27700000000000002},"max":{"1min":16.853999999999999,"5min":16.853999999999999,"15min":23.081},"last":0.59099999999999997},{"interface":"front","average":{"1min":1.913,"5min":1.504,"15min":1.278},"min":{"1min":0.32000000000000001,"5min":0.26100000000000001,"15min":0.23699999999999999},"max":{"1min":10.02,"5min":10.02,"15min":22.666},"last":4.0350000000000001}]}]},{"osd":3,"up_from":26,"seq":111669149921,"num_pgs":70,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1647160,"kb_used_data":754608,"kb_used_omap":1,"kb_used_meta":891966,"kb_avail":92097992,"statfs":{"total":95995035648,"available":94308343808,"internally_reserved":0,"allocated":772718592,"data_stored":772395225,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":913373528},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 19:26:25 2024","interfaces":[{"interface":"back","average":{"1min":0.52300000000000002,"5min":0.53100000000000003,"15min":0.53300000000000003},"min":{"1min":0.18099999999999999,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":1.802,"5min":1.802,"15min":1.802},"last":0.252},{"interface":"front","average":{"1min":0.56799999999999995,"5min":0.51200000000000001,"15min":0.503},"min":{"1min":0.221,"5min":0.20100000000000001,"15min":0.20100000000000001},"max":{"1min":1.7889999999999999,"5min":1.7889999999999999,"15min":1.7889999999999999},"last":0.28599999999999998}]},{"osd":1,"last update":"Tue Sep 10 19:25:51 2024","interfaces":[{"interface":"back","average":{"1min":0.44800000000000001,"5min":0.35099999999999998,"15min":0.313},"min":{"1min":0.192,"5min":0.192,"15min":0.192},"max":{"1min":1.768,"5min":2.5939999999999999,"15min":2.5939999999999999},"last":0.27900000000000003},{"interface":"front","average":{"1min":0.41499999999999998,"5min":0.34699999999999998,"15min":0.32500000000000001},"min":{"1min":0.215,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":0.75800000000000001,"5min":1.3720000000000001,"15min":1.3720000000000001},"last":0.32600000000000001}]},{"osd":2,"last update":"Tue Sep 10 19:26:03 2024","interfaces":[{"interface":"back","average":{"1min":0.41499999999999998,"5min":0.37,"15min":0.63600000000000001},"min":{"1min":0.25700000000000001,"5min":0.17599999999999999,"15min":0.17599999999999999},"max":{"1min":0.76500000000000001,"5min":2.2530000000000001,"15min":5.3029999999999999},"last":0.35299999999999998},{"interface":"front","average":{"1min":0.65500000000000003,"5min":0.42799999999999999,"15min":0.71799999999999997},"min":{"1min":0.29599999999999999,"5min":0.16400000000000001,"15min":0.16400000000000001},"max":{"1min":2.944,"5min":2.944,"15min":17.806000000000001},"last":0.47999999999999998}]},{"osd":4,"last update":"Tue Sep 10 19:26:18 2024","interfaces":[{"interface":"back","average":{"1min":0.71699999999999997,"5min":1.9650000000000001,"15min":1.3680000000000001},"min":{"1min":0.28799999999999998,"5min":0.26900000000000002,"15min":0.23599999999999999},"max":{"1min":1.649,"5min":81.334000000000003,"15min":81.334000000000003},"last":1.405},{"interface":"front","average":{"1min":1.536,"5min":2.2349999999999999,"15min":1.5820000000000001},"min":{"1min":0.27900000000000003,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":3.6789999999999998,"5min":81.381,"15min":81.381},"last":0.751}]},{"osd":5,"last update":"Tue Sep 10 19:26:41 2024","interfaces":[{"interface":"back","average":{"1min":1.5860000000000001,"5min":2.0659999999999998,"15min":1.724},"min":{"1min":0.35399999999999998,"5min":0.28100000000000003,"15min":0.27900000000000003},"max":{"1min":6.7240000000000002,"5min":104.672,"15min":104.672},"last":1.52},{"interface":"front","average":{"1min":1.137,"5min":1.9139999999999999,"15min":1.7849999999999999},"min":{"1min":0.32200000000000001,"5min":0.24299999999999999,"15min":0.22},"max":{"1min":4.1159999999999997,"5min":105.971,"15min":105.971},"last":0.57599999999999996}]},{"osd":6,"last update":"Tue Sep 10 19:26:03 2024","interfaces":[{"interface":"back","average":{"1min":1.282,"5min":2.5049999999999999,"15min":1.615},"min":{"1min":0.40400000000000003,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":5.2480000000000002,"5min":116.14,"15min":116.14},"last":0.59799999999999998},{"interface":"front","average":{"1min":1.1890000000000001,"5min":2.4900000000000002,"15min":1.4930000000000001},"min":{"1min":0.42099999999999999,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":4.008,"5min":116.095,"15min":116.095},"last":0.52000000000000002}]},{"osd":7,"last update":"Tue Sep 10 19:26:35 2024","interfaces":[{"interface":"back","average":{"1min":1.0880000000000001,"5min":1.9590000000000001,"15min":1.4259999999999999},"min":{"1min":0.34799999999999998,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":4.4640000000000004,"5min":104.749,"15min":104.749},"last":0.94599999999999995},{"interface":"front","average":{"1min":0.90500000000000003,"5min":1.8009999999999999,"15min":1.431},"min":{"1min":0.33100000000000002,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":2.5219999999999998,"5min":104.69499999999999,"15min":104.69499999999999},"last":0.89200000000000002}]}]},{"osd":2,"up_from":0,"seq":77309411558,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":276,"seq":1185410973747,"num_pgs":63,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":829156,"kb_used_data":514972,"kb_used_omap":6,"kb_used_meta":314169,"kb_avail":92915996,"statfs":{"total":95995035648,"available":95145979904,"internally_reserved":0,"allocated":527331328,"data_stored":527018616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6335,"internal_metadata":321709889},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 19:26:29 2024","interfaces":[{"interface":"back","average":{"1min":0.55000000000000004,"5min":0.499,"15min":0.49099999999999999},"min":{"1min":0.24199999999999999,"5min":0.182,"15min":0.182},"max":{"1min":1.4550000000000001,"5min":1.8160000000000001,"15min":1.8160000000000001},"last":0.247},{"interface":"front","average":{"1min":0.58299999999999996,"5min":0.443,"15min":0.41899999999999998},"min":{"1min":0.214,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":1.2450000000000001,"5min":1.387,"15min":1.387},"last":0.27300000000000002}]},{"osd":2,"last update":"Tue Sep 10 19:25:49 2024","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.35599999999999998,"15min":0.31},"min":{"1min":0.251,"5min":0.19,"15min":0.19},"max":{"1min":1.837,"5min":1.837,"15min":1.837},"last":0.30199999999999999},{"interface":"front","average":{"1min":0.53800000000000003,"5min":0.38700000000000001,"15min":0.35199999999999998},"min":{"1min":0.29199999999999998,"5min":0.222,"15min":0.222},"max":{"1min":1.7909999999999999,"5min":1.7909999999999999,"15min":1.7909999999999999},"last":0.377}]},{"osd":3,"last update":"Tue Sep 10 19:25:49 2024","interfaces":[{"interface":"back","average":{"1min":0.38100000000000001,"5min":0.33300000000000002,"15min":0.313},"min":{"1min":0.23400000000000001,"5min":0.221,"15min":0.221},"max":{"1min":0.64700000000000002,"5min":1.4570000000000001,"15min":1.4570000000000001},"last":0.36899999999999999},{"interface":"front","average":{"1min":0.372,"5min":0.32400000000000001,"15min":0.308},"min":{"1min":0.20899999999999999,"5min":0.17599999999999999,"15min":0.17599999999999999},"max":{"1min":1.181,"5min":1.2450000000000001,"15min":1.2450000000000001},"last":0.41299999999999998}]},{"osd":4,"last update":"Tue Sep 10 19:25:49 2024","interfaces":[{"interface":"back","average":{"1min":0.94199999999999995,"5min":0.65600000000000003,"15min":0.57299999999999995},"min":{"1min":0.26600000000000001,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":3.0640000000000001,"5min":3.0640000000000001,"15min":3.0640000000000001},"last":0.99399999999999999},{"interface":"front","average":{"1min":1.1359999999999999,"5min":0.72799999999999998,"15min":0.65300000000000002},"min":{"1min":0.32200000000000001,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":3.1680000000000001,"5min":3.1680000000000001,"15min":3.1680000000000001},"last":1.2450000000000001}]},{"osd":5,"last update":"Tue Sep 10 19:25:49 2024","interfaces":[{"interface":"back","average":{"1min":0.73099999999999998,"5min":0.86299999999999999,"15min":0.94999999999999996},"min":{"1min":0.307,"5min":0.307,"15min":0.307},"max":{"1min":3.5270000000000001,"5min":4.7519999999999998,"15min":4.7519999999999998},"last":0.876},{"interface":"front","average":{"1min":0.82999999999999996,"5min":1.151,"15min":1.339},"min":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":3.5179999999999998,"5min":5.617,"15min":5.617},"last":0.88600000000000001}]},{"osd":6,"last update":"Tue Sep 10 19:25:49 2024","interfaces":[{"interface":"back","average":{"1min":1.268,"5min":1.8720000000000001,"15min":2.1859999999999999},"min":{"1min":0.31,"5min":0.31,"15min":0.31},"max":{"1min":3.27,"5min":27.379999999999999,"15min":27.379999999999999},"last":0.79300000000000004},{"interface":"front","average":{"1min":1.3380000000000001,"5min":1.849,"15min":2.177},"min":{"1min":0.42999999999999999,"5min":0.311,"15min":0.311},"max":{"1min":4.4859999999999998,"5min":26.812999999999999,"15min":26.812999999999999},"last":0.39100000000000001}]},{"osd":7,"last update":"Tue Sep 10 19:25:49 2024","interfaces":[{"interface":"back","average":{"1min":1.1080000000000001,"5min":0.95499999999999996,"15min":1.008},"min":{"1min":0.40200000000000002,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":3.3340000000000001,"5min":5.6260000000000003,"15min":5.6260000000000003},"last":0.55900000000000005},{"interface":"front","average":{"1min":1.103,"5min":0.96099999999999997,"15min":1.0169999999999999},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":4.0199999999999996,"5min":5.6050000000000004,"15min":5.6050000000000004},"last":0.42399999999999999}]}]},{"osd":0,"up_from":301,"seq":1292785156127,"num_pgs":77,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3735200,"kb_used_data":3381016,"kb_used_omap":4,"kb_used_meta":354171,"kb_avail":90009952,"statfs":{"total":95995035648,"available":92170190848,"internally_reserved":0,"allocated":3462160384,"data_stored":3461852193,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":362671622},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Sep 10 19:26:29 2024","interfaces":[{"interface":"back","average":{"1min":0.38700000000000001,"5min":0.53600000000000003,"15min":0.56100000000000005},"min":{"1min":0.17499999999999999,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":0.72599999999999998,"5min":1.5269999999999999,"15min":1.5269999999999999},"last":0.36799999999999999},{"interface":"front","average":{"1min":0.34999999999999998,"5min":0.46500000000000002,"15min":0.48399999999999999},"min":{"1min":0.16900000000000001,"5min":0.16900000000000001,"15min":0.16900000000000001},"max":{"1min":0.81200000000000006,"5min":1.216,"15min":1.216},"last":0.33500000000000002}]},{"osd":2,"last update":"Tue Sep 10 19:26:29 2024","interfaces":[{"interface":"back","average":{"1min":0.46999999999999997,"5min":0.57599999999999996,"15min":0.59399999999999997},"min":{"1min":0.20499999999999999,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":1.244,"5min":1.734,"15min":1.734},"last":0.35299999999999998},{"interface":"front","average":{"1min":0.47399999999999998,"5min":0.60199999999999998,"15min":0.624},"min":{"1min":0.22600000000000001,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":0.82699999999999996,"5min":1.5580000000000001,"15min":1.5580000000000001},"last":0.40699999999999997}]},{"osd":3,"last update":"Tue Sep 10 19:26:29 2024","interfaces":[{"interface":"back","average":{"1min":0.42899999999999999,"5min":0.55700000000000005,"15min":0.57799999999999996},"min":{"1min":0.20599999999999999,"5min":0.20599999999999999,"15min":0.20599999999999999},"max":{"1min":0.77700000000000002,"5min":1.1910000000000001,"15min":1.1910000000000001},"last":0.38700000000000001},{"interface":"front","average":{"1min":0.58299999999999996,"5min":0.61499999999999999,"15min":0.621},"min":{"1min":0.23499999999999999,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":1.4950000000000001,"5min":1.677,"15min":1.677},"last":0.49299999999999999}]},{"osd":4,"last update":"Tue Sep 10 19:26:29 2024","interfaces":[{"interface":"back","average":{"1min":1.8260000000000001,"5min":3.968,"15min":4.3250000000000002},"min":{"1min":0.25800000000000001,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":6.5640000000000001,"5min":53.670000000000002,"15min":53.670000000000002},"last":2.5249999999999999},{"interface":"front","average":{"1min":1.2729999999999999,"5min":3.8639999999999999,"15min":4.2960000000000003},"min":{"1min":0.314,"5min":0.314,"15min":0.314},"max":{"1min":6.5019999999999998,"5min":53.935000000000002,"15min":53.935000000000002},"last":1.869}]},{"osd":5,"last update":"Tue Sep 10 19:26:29 2024","interfaces":[{"interface":"back","average":{"1min":0.90200000000000002,"5min":3.4449999999999998,"15min":3.8690000000000002},"min":{"1min":0.28599999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":2.9550000000000001,"5min":53.884999999999998,"15min":53.884999999999998},"last":2.2730000000000001},{"interface":"front","average":{"1min":1.0609999999999999,"5min":3.3610000000000002,"15min":3.7440000000000002},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":3.1200000000000001,"5min":53.875999999999998,"15min":53.875999999999998},"last":2.1829999999999998}]},{"osd":6,"last update":"Tue Sep 10 19:26:29 2024","interfaces":[{"interface":"back","average":{"1min":1.165,"5min":4.0960000000000001,"15min":4.5839999999999996},"min":{"1min":0.36799999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":2.8919999999999999,"5min":56.901000000000003,"15min":56.901000000000003},"last":2.149},{"interface":"front","average":{"1min":1.1200000000000001,"5min":4.024,"15min":4.508},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":3.1669999999999998,"5min":55.177,"15min":55.177},"last":2.3180000000000001}]},{"osd":7,"last update":"Tue Sep 10 19:26:29 2024","interfaces":[{"interface":"back","average":{"1min":1.288,"5min":1.2470000000000001,"15min":1.24},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":3.8260000000000001,"5min":3.8260000000000001,"15min":3.8260000000000001},"last":0.58099999999999996},{"interface":"front","average":{"1min":1.079,"5min":1.222,"15min":1.246},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":3.2069999999999999,"5min":3.2519999999999998,"15min":3.2519999999999998},"last":0.63900000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":966,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1236,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1236,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":362,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1540,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":2118,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1018,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1502,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1724911616,"data_stored":1724907544,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":513081344,"data_stored":513081344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":765984768,"data_stored":765984768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1713180672,"data_stored":1713176600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1066991616,"data_stored":1066991616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1101991936,"data_stored":1101987864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":768147456,"data_stored":768147456,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T19:26:57.214 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph pg dump --format=json 2024-09-10T19:26:57.215 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:26:57.218 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:26:57.249 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:26:57.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:26:57.251 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:26:57.351 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:26:57.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:26:57.388 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:26:57.389 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:26:57.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:26:57 smithi183 ceph-mon[29736]: pgmap v769: 158 pgs: 158 active+clean; 2.4 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.34k op/s 2024-09-10T19:26:57.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:26:57 smithi183 ceph-mon[29736]: osdmap e354: 8 total, 8 up, 7 in 2024-09-10T19:26:57.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:26:57 smithi183 ceph-mon[29736]: from='client.15933 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-10T19:26:57.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:26:57 smithi183 ceph-mon[41562]: pgmap v769: 158 pgs: 158 active+clean; 2.4 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.34k op/s 2024-09-10T19:26:57.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:26:57 smithi183 ceph-mon[41562]: osdmap e354: 8 total, 8 up, 7 in 2024-09-10T19:26:57.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:26:57 smithi183 ceph-mon[41562]: from='client.15933 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-10T19:26:57.489 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:26:57.534 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:26:57.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:26:57.535 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:26:57.617 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:26:57 smithi191 ceph-mon[32884]: pgmap v769: 158 pgs: 158 active+clean; 2.4 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.34k op/s 2024-09-10T19:26:57.617 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:26:57 smithi191 ceph-mon[32884]: osdmap e354: 8 total, 8 up, 7 in 2024-09-10T19:26:57.617 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:26:57 smithi191 ceph-mon[32884]: from='client.15933 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-10T19:26:57.617 INFO:journalctl@ceph.mgr.x.smithi191.stdout:Sep 10 19:26:57 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-x[34967]: ::ffff:172.21.15.191 - - [10/Sep/2024:19:26:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T19:26:57.636 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:26:57.664 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:26:57.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:26:57.665 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:26:57.761 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph osd unset noscrub 2024-09-10T19:26:57.766 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:26:57.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:26:57.834 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:26:57.834 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:26:57.935 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:26:57.973 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:26:57.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:26:57.974 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:26:58.038 INFO:teuthology.orchestra.run.smithi183.stdout:osd.0: {} 2024-09-10T19:26:58.038 INFO:teuthology.orchestra.run.smithi183.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-10T19:26:58.055 INFO:teuthology.orchestra.run.smithi183.stdout:osd.1: {} 2024-09-10T19:26:58.055 INFO:teuthology.orchestra.run.smithi183.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-10T19:26:58.074 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:26:58.088 INFO:teuthology.orchestra.run.smithi183.stdout:osd.2: {} 2024-09-10T19:26:58.088 INFO:teuthology.orchestra.run.smithi183.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-10T19:26:58.109 INFO:teuthology.orchestra.run.smithi183.stdout:osd.3: {} 2024-09-10T19:26:58.110 INFO:teuthology.orchestra.run.smithi183.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-10T19:26:58.111 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:26:58.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:26:58.112 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:26:58.147 INFO:teuthology.orchestra.run.smithi183.stdout:osd.4: {} 2024-09-10T19:26:58.147 INFO:teuthology.orchestra.run.smithi183.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-10T19:26:58.172 INFO:teuthology.orchestra.run.smithi183.stdout:osd.5: {} 2024-09-10T19:26:58.172 INFO:teuthology.orchestra.run.smithi183.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-10T19:26:58.192 INFO:teuthology.orchestra.run.smithi183.stdout:osd.6: {} 2024-09-10T19:26:58.192 INFO:teuthology.orchestra.run.smithi183.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-10T19:26:58.212 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:26:58.221 INFO:teuthology.orchestra.run.smithi183.stdout:osd.7: {} 2024-09-10T19:26:58.222 INFO:teuthology.orchestra.run.smithi183.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-10T19:26:58.245 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:26:58.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:26:58.246 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:26:58.347 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:26:58.384 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:26:58.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:26:58.385 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:26:58.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:26:58 smithi183 ceph-mon[29736]: from='client.? 172.21.15.183:0/3932253838' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T19:26:58.431 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:26:58 smithi183 ceph-mon[41562]: from='client.? 172.21.15.183:0/3932253838' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T19:26:58.486 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:26:58.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:26:58.528 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:26:58.528 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:26:58.629 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:26:58.654 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:26:58.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:26:58.655 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:26:58.715 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:26:58 smithi191 ceph-mon[32884]: from='client.? 172.21.15.183:0/3932253838' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T19:26:58.756 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:26:58.795 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:26:58.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:26:58.796 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:26:58.897 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:26:58.939 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:26:58.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:26:58.940 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:26:59.041 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:26:59.067 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:26:59.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:26:59.068 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:26:59.169 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:26:59.214 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:26:59.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:26:59.215 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:26:59.316 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:26:59.354 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:26:59.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:26:59.355 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:26:59.456 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:26:59.493 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:26:59.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:26:59.494 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:26:59.591 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:26:59 smithi183 ceph-mon[29736]: pgmap v771: 158 pgs: 158 active+clean; 2.3 GiB data, 13 GiB used, 612 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.37k op/s 2024-09-10T19:26:59.592 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:26:59 smithi183 ceph-mon[29736]: purged_snaps scrub starts 2024-09-10T19:26:59.592 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:26:59 smithi183 ceph-mon[29736]: purged_snaps scrub ok 2024-09-10T19:26:59.592 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:26:59 smithi183 ceph-mon[29736]: osdmap e355: 8 total, 8 up, 7 in 2024-09-10T19:26:59.592 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:26:59 smithi183 ceph-mon[29736]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:26:59.592 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:26:59 smithi183 ceph-mon[41562]: pgmap v771: 158 pgs: 158 active+clean; 2.3 GiB data, 13 GiB used, 612 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.37k op/s 2024-09-10T19:26:59.592 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:26:59 smithi183 ceph-mon[41562]: purged_snaps scrub starts 2024-09-10T19:26:59.592 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:26:59 smithi183 ceph-mon[41562]: purged_snaps scrub ok 2024-09-10T19:26:59.592 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:26:59 smithi183 ceph-mon[41562]: osdmap e355: 8 total, 8 up, 7 in 2024-09-10T19:26:59.593 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:26:59 smithi183 ceph-mon[41562]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:26:59.594 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:26:59.623 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:26:59.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:26:59.624 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:26:59.715 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:26:59 smithi191 ceph-mon[32884]: pgmap v771: 158 pgs: 158 active+clean; 2.3 GiB data, 13 GiB used, 612 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.37k op/s 2024-09-10T19:26:59.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:26:59 smithi191 ceph-mon[32884]: purged_snaps scrub starts 2024-09-10T19:26:59.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:26:59 smithi191 ceph-mon[32884]: purged_snaps scrub ok 2024-09-10T19:26:59.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:26:59 smithi191 ceph-mon[32884]: osdmap e355: 8 total, 8 up, 7 in 2024-09-10T19:26:59.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:26:59 smithi191 ceph-mon[32884]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:26:59.724 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:26:59.761 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:26:59.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:26:59.762 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:26:59.863 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:26:59.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:26:59.890 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:26:59.890 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:26:59.991 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:27:00.018 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:27:00.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:00.019 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:27:00.120 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:27:00.147 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:27:00.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:00.148 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:27:00.215 INFO:journalctl@ceph.iscsi.iscsi.a.smithi191.stdout:Sep 10 19:26:59 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-iscsi-iscsi-a[61816]: debug there is no tcmu-runner data available 2024-09-10T19:27:00.248 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:27:00.288 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:27:00.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:00.289 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:27:00.389 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:27:00.421 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:27:00.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:00.422 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:27:00.523 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:27:00.549 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:27:00.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:00.550 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:27:00.651 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:27:00.688 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:27:00.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:00.689 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:27:00.790 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:27:00.827 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:27:00.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:00.828 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:27:00.929 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:27:00.957 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:27:00.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:00.958 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:27:01.059 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:27:01.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:01.089 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:27:01.090 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:27:01.190 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:27:01.212 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:27:01.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:01.213 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:27:01.313 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:27:01.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:01.345 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:27:01.345 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:27:01.356 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:27:01 smithi183 ceph-mon[29736]: from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:27:01.356 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:27:01 smithi183 ceph-mon[29736]: pgmap v773: 158 pgs: 158 active+clean; 2.1 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.24k op/s 2024-09-10T19:27:01.356 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:27:01 smithi183 ceph-mon[29736]: purged_snaps scrub starts 2024-09-10T19:27:01.356 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:27:01 smithi183 ceph-mon[29736]: purged_snaps scrub ok 2024-09-10T19:27:01.357 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:27:01 smithi183 ceph-mon[41562]: from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:27:01.357 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:27:01 smithi183 ceph-mon[41562]: pgmap v773: 158 pgs: 158 active+clean; 2.1 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.24k op/s 2024-09-10T19:27:01.357 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:27:01 smithi183 ceph-mon[41562]: purged_snaps scrub starts 2024-09-10T19:27:01.357 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:27:01 smithi183 ceph-mon[41562]: purged_snaps scrub ok 2024-09-10T19:27:01.357 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Sep 10 19:27:01 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-y[29989]: ::ffff:172.21.15.191 - - [10/Sep/2024:19:27:01] "GET /metrics HTTP/1.1" 200 224779 "" "Prometheus/2.43.0" 2024-09-10T19:27:01.366 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:27:01.366 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:27:01.405 INFO:teuthology.orchestra.run.smithi183.stdout:{ 2024-09-10T19:27:01.405 INFO:teuthology.orchestra.run.smithi183.stdout: "ops": [], 2024-09-10T19:27:01.405 INFO:teuthology.orchestra.run.smithi183.stdout: "complaint_time": 30, 2024-09-10T19:27:01.406 INFO:teuthology.orchestra.run.smithi183.stdout: "num_blocked_ops": 0 2024-09-10T19:27:01.406 INFO:teuthology.orchestra.run.smithi183.stdout:} 2024-09-10T19:27:01.446 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:27:01.473 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:27:01.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:01.474 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:27:01.575 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:27:01.600 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:27:01.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:01.601 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:27:01.702 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:27:01.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:27:01 smithi191 ceph-mon[32884]: from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:27:01.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:27:01 smithi191 ceph-mon[32884]: pgmap v773: 158 pgs: 158 active+clean; 2.1 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.24k op/s 2024-09-10T19:27:01.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:27:01 smithi191 ceph-mon[32884]: purged_snaps scrub starts 2024-09-10T19:27:01.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:27:01 smithi191 ceph-mon[32884]: purged_snaps scrub ok 2024-09-10T19:27:01.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:01.745 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:27:01.745 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:27:01.846 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:27:01.884 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:27:01.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:01.886 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:27:01.987 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:27:02.021 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:27:02.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:02.023 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:27:02.123 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:27:02.128 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph daemon osd.3 dump_historic_ops 2024-09-10T19:27:02.161 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:27:02.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:02.162 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:27:02.263 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:27:02.293 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:27:02.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:02.294 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:27:02.395 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:27:02.414 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:27:02 smithi183 ceph-mon[29736]: osdmap e356: 8 total, 8 up, 7 in 2024-09-10T19:27:02.414 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:27:02 smithi183 ceph-mon[41562]: osdmap e356: 8 total, 8 up, 7 in 2024-09-10T19:27:02.432 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:27:02.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:02.433 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:27:02.533 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:27:02.560 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:27:02.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:02.561 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:27:02.662 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:27:02.692 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:27:02.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:02.693 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:27:02.715 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:27:02 smithi191 ceph-mon[32884]: osdmap e356: 8 total, 8 up, 7 in 2024-09-10T19:27:02.794 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:27:02.822 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:27:02.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:02.823 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:27:02.861 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-10T19:27:02.924 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:27:02.952 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:27:02.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:02.953 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:27:03.054 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:27:03.094 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:27:03.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:03.096 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:27:03.196 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:27:03.232 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:27:03.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:03.233 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:27:03.334 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:27:03.376 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:27:03.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:03.377 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:27:03.478 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:27:03.513 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:27:03.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:03.514 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:27:03.615 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:27:03.643 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:27:03.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:03.645 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:27:03.681 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:27:03 smithi183 ceph-mon[29736]: pgmap v775: 158 pgs: 158 active+clean; 2.0 GiB data, 12 GiB used, 613 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 1.10k op/s 2024-09-10T19:27:03.681 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:27:03 smithi183 ceph-mon[41562]: pgmap v775: 158 pgs: 158 active+clean; 2.0 GiB data, 12 GiB used, 613 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 1.10k op/s 2024-09-10T19:27:03.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:27:03 smithi191 ceph-mon[32884]: pgmap v775: 158 pgs: 158 active+clean; 2.0 GiB data, 12 GiB used, 613 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 1.10k op/s 2024-09-10T19:27:03.747 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:27:03.781 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:27:03.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:03.782 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:27:03.883 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:27:03.923 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:27:03.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:03.924 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:27:04.025 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:27:04.079 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:27:04.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:04.080 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:27:04.181 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:27:04.209 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:27:04.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:04.210 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:27:04.311 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:27:04.339 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:27:04.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:04.340 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:27:04.441 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:27:04.466 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:27:04.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:04.467 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:27:04.568 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:27:04.595 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:27:04.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:04.596 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:27:04.697 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:27:04.735 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:27:04.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:04.736 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:27:04.837 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:27:04.879 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:27:04.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:04.880 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:27:04.981 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:27:05.021 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:27:05.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:05.022 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:27:05.123 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:27:05.152 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:27:05.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:05.153 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:27:05.254 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:27:05.281 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:27:05.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:05.282 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:27:05.299 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:27:05 smithi183 ceph-mon[29736]: pgmap v776: 158 pgs: 158 active+clean; 1.9 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.07k op/s 2024-09-10T19:27:05.299 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:27:05 smithi183 ceph-mon[41562]: pgmap v776: 158 pgs: 158 active+clean; 1.9 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.07k op/s 2024-09-10T19:27:05.383 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:27:05.420 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:27:05.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:05.421 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:27:05.521 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:27:05.551 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:27:05.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:05.552 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:27:05.653 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:27:05.694 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:27:05.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:05.695 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:27:05.715 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:27:05 smithi191 ceph-mon[32884]: pgmap v776: 158 pgs: 158 active+clean; 1.9 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.07k op/s 2024-09-10T19:27:05.796 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:27:05.804 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:27:05.809 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:27:05.830 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:27:05.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:05.832 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:27:05.878 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-09-10T19:27:05.885 INFO:teuthology.orchestra.run.smithi183.stderr:dumped all 2024-09-10T19:27:05.933 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:27:05.971 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:27:05.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:05.972 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:27:06.073 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:27:06.109 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:27:06.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:06.111 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:27:06.212 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:27:06.239 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:27:06.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:06.240 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:27:06.293 INFO:teuthology.orchestra.run.smithi183.stderr:noscrub is unset 2024-09-10T19:27:06.341 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:27:06.369 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:27:06.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:06.370 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:27:06.470 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:27:06.508 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:27:06.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:06.509 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:27:06.610 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:27:06.642 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:27:06.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:06.643 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:27:06.658 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:27:06 smithi183 ceph-mon[29736]: from='client.? 172.21.15.183:0/3463444277' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-10T19:27:06.658 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:27:06 smithi183 ceph-mon[41562]: from='client.? 172.21.15.183:0/3463444277' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-10T19:27:06.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:27:06 smithi191 ceph-mon[32884]: from='client.? 172.21.15.183:0/3463444277' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-10T19:27:06.744 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:27:06.784 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:27:06.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:06.785 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:27:06.886 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:27:06.909 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:27:06.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:06.910 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:27:07.010 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:27:07.053 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:27:07.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:07.054 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:27:07.155 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:27:07.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:07.191 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:27:07.192 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:27:07.202 INFO:teuthology.orchestra.run.smithi183.stdout:{"pg_ready":true,"pg_map":{"version":776,"stamp":"2024-09-10T19:27:03.907342+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":2014320884,"num_objects":30924,"num_object_clones":0,"num_object_copies":92772,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":30924,"num_whiteouts":0,"num_read":1334,"num_read_kb":1086,"num_write":216400,"num_write_kb":7885164,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4984,"num_bytes_recovered":306715526,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":73322,"ondisk_log_size":73322,"up":474,"acting":474,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":482,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":656216064,"kb_used":12668004,"kb_used_data":7517148,"kb_used_omap":19,"kb_used_meta":5150316,"kb_avail":643548060,"statfs":{"total":671965249536,"available":658993213440,"internally_reserved":0,"allocated":7697559552,"data_stored":7695341917,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19457,"internal_metadata":5273924607},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,3,1,5,3,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-540606464,"num_objects":-8249,"num_object_clones":0,"num_object_copies":-24747,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-8249,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":8249,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.676995"},"pg_stats":[{"pgid":"6.1b","version":"0'0","reported_seq":61,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-09-10T19:26:58.263921+0000","last_change":"2024-09-10T19:26:22.493488+0000","last_active":"2024-09-10T19:26:58.263921+0000","last_peered":"2024-09-10T19:26:58.263921+0000","last_clean":"2024-09-10T19:26:58.263921+0000","last_became_active":"2024-09-10T19:24:44.895077+0000","last_became_peered":"2024-09-10T19:24:44.895077+0000","last_unstale":"2024-09-10T19:26:58.263921+0000","last_undegraded":"2024-09-10T19:26:58.263921+0000","last_fullsized":"2024-09-10T19:26:58.263921+0000","mapping_epoch":314,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":315,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:26:22.493431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:26:22.493431+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:27:46.054799+0000","scrub_duration":0.0052677039999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6],"acting":[0,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":329,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-09-10T19:26:58.288229+0000","last_change":"2024-09-10T19:26:27.556376+0000","last_active":"2024-09-10T19:26:58.288229+0000","last_peered":"2024-09-10T19:26:58.288229+0000","last_clean":"2024-09-10T19:26:58.288229+0000","last_became_active":"2024-09-10T19:24:23.892621+0000","last_became_peered":"2024-09-10T19:24:23.892621+0000","last_unstale":"2024-09-10T19:26:58.288229+0000","last_undegraded":"2024-09-10T19:26:58.288229+0000","last_fullsized":"2024-09-10T19:26:58.288229+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:26:27.556297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:16:38.194472+0000","last_clean_scrub_stamp":"2024-09-10T19:26:27.556297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:27:32.452754+0000","scrub_duration":0.0045129510000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.19","version":"237'27","reported_seq":429,"reported_epoch":354,"state":"active+clean","last_fresh":"2024-09-10T19:26:56.235524+0000","last_change":"2024-09-10T19:24:40.424985+0000","last_active":"2024-09-10T19:26:56.235524+0000","last_peered":"2024-09-10T19:26:56.235524+0000","last_clean":"2024-09-10T19:26:56.235524+0000","last_became_active":"2024-09-10T19:24:23.906592+0000","last_became_peered":"2024-09-10T19:24:23.906592+0000","last_unstale":"2024-09-10T19:26:56.235524+0000","last_undegraded":"2024-09-10T19:26:56.235524+0000","last_fullsized":"2024-09-10T19:26:56.235524+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"237'27","last_scrub_stamp":"2024-09-10T19:24:40.424918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:24:40.424918+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0040964390000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":403,"reported_epoch":354,"state":"active+clean","last_fresh":"2024-09-10T19:26:56.242378+0000","last_change":"2024-09-10T19:26:07.791959+0000","last_active":"2024-09-10T19:26:56.242378+0000","last_peered":"2024-09-10T19:26:56.242378+0000","last_clean":"2024-09-10T19:26:56.242378+0000","last_became_active":"2024-09-10T19:24:01.976841+0000","last_became_peered":"2024-09-10T19:24:01.976841+0000","last_unstale":"2024-09-10T19:26:56.242378+0000","last_undegraded":"2024-09-10T19:26:56.242378+0000","last_fullsized":"2024-09-10T19:26:56.242378+0000","mapping_epoch":292,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":293,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:26:07.791886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:19:28.498560+0000","last_clean_scrub_stamp":"2024-09-10T19:26:07.791886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:27:28.278006+0000","scrub_duration":0.0046830530000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"59'1","reported_seq":390,"reported_epoch":354,"state":"active+clean","last_fresh":"2024-09-10T19:26:56.236845+0000","last_change":"2024-09-10T19:26:26.891622+0000","last_active":"2024-09-10T19:26:56.236845+0000","last_peered":"2024-09-10T19:26:56.236845+0000","last_clean":"2024-09-10T19:26:56.236845+0000","last_became_active":"2024-09-10T19:24:23.908614+0000","last_became_peered":"2024-09-10T19:24:23.908614+0000","last_unstale":"2024-09-10T19:26:56.236845+0000","last_undegraded":"2024-09-10T19:26:56.236845+0000","last_fullsized":"2024-09-10T19:26:56.236845+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'1","last_scrub_stamp":"2024-09-10T19:26:26.891431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:26:26.891431+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:27:30.232324+0000","scrub_duration":0.0037677470000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":874,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"237'15","reported_seq":437,"reported_epoch":354,"state":"active+clean","last_fresh":"2024-09-10T19:26:56.242389+0000","last_change":"2024-09-10T19:25:20.871177+0000","last_active":"2024-09-10T19:26:56.242389+0000","last_peered":"2024-09-10T19:26:56.242389+0000","last_clean":"2024-09-10T19:26:56.242389+0000","last_became_active":"2024-09-10T19:24:01.976339+0000","last_became_peered":"2024-09-10T19:24:01.976339+0000","last_unstale":"2024-09-10T19:26:56.242389+0000","last_undegraded":"2024-09-10T19:26:56.242389+0000","last_fullsized":"2024-09-10T19:26:56.242389+0000","mapping_epoch":292,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":293,"parent":"0.0","parent_split_bits":0,"last_scrub":"237'15","last_scrub_stamp":"2024-09-10T19:25:20.871034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:25:20.871034+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.011611994,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":60,"reported_epoch":354,"state":"active+clean","last_fresh":"2024-09-10T19:26:56.244315+0000","last_change":"2024-09-10T19:24:53.986685+0000","last_active":"2024-09-10T19:26:56.244315+0000","last_peered":"2024-09-10T19:26:56.244315+0000","last_clean":"2024-09-10T19:26:56.244315+0000","last_became_active":"2024-09-10T19:24:44.895513+0000","last_became_peered":"2024-09-10T19:24:44.895513+0000","last_unstale":"2024-09-10T19:26:56.244315+0000","last_undegraded":"2024-09-10T19:26:56.244315+0000","last_fullsized":"2024-09-10T19:26:56.244315+0000","mapping_epoch":314,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":315,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:24:53.986597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:24:53.986597+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.010150255,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,7],"acting":[6,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":391,"reported_epoch":354,"state":"active+clean","last_fresh":"2024-09-10T19:26:56.245465+0000","last_change":"2024-09-10T19:25:31.919853+0000","last_active":"2024-09-10T19:26:56.245465+0000","last_peered":"2024-09-10T19:26:56.245465+0000","last_clean":"2024-09-10T19:26:56.245465+0000","last_became_active":"2024-09-10T19:24:01.971921+0000","last_became_peered":"2024-09-10T19:24:01.971921+0000","last_unstale":"2024-09-10T19:26:56.245465+0000","last_undegraded":"2024-09-10T19:26:56.245465+0000","last_fullsized":"2024-09-10T19:26:56.245465+0000","mapping_epoch":292,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":293,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:25:31.919782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:25:31.919782+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0099438370000000005,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":382,"reported_epoch":354,"state":"active+clean","last_fresh":"2024-09-10T19:26:56.237966+0000","last_change":"2024-09-10T19:26:35.582648+0000","last_active":"2024-09-10T19:26:56.237966+0000","last_peered":"2024-09-10T19:26:56.237966+0000","last_clean":"2024-09-10T19:26:56.237966+0000","last_became_active":"2024-09-10T19:24:23.902270+0000","last_became_peered":"2024-09-10T19:24:23.902270+0000","last_unstale":"2024-09-10T19:26:56.237966+0000","last_undegraded":"2024-09-10T19:26:56.237966+0000","last_fullsized":"2024-09-10T19:26:56.237966+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:26:35.582561+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:26:35.582561+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:27:37.081154+0000","scrub_duration":0.013149586,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1b","version":"237'7","reported_seq":433,"reported_epoch":354,"state":"active+clean","last_fresh":"2024-09-10T19:26:56.245216+0000","last_change":"2024-09-10T19:26:22.847139+0000","last_active":"2024-09-10T19:26:56.245216+0000","last_peered":"2024-09-10T19:26:56.245216+0000","last_clean":"2024-09-10T19:26:56.245216+0000","last_became_active":"2024-09-10T19:24:01.974813+0000","last_became_peered":"2024-09-10T19:24:01.974813+0000","last_unstale":"2024-09-10T19:26:56.245216+0000","last_undegraded":"2024-09-10T19:26:56.245216+0000","last_fullsized":"2024-09-10T19:26:56.245216+0000","mapping_epoch":292,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":293,"parent":"0.0","parent_split_bits":0,"last_scrub":"237'7","last_scrub_stamp":"2024-09-10T19:26:22.847061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:26:22.847061+0000","objects_scrubbed":2,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:27:33.920907+0000","scrub_duration":0.001667078,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":10,"num_write":8,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":385,"reported_epoch":354,"state":"active+clean","last_fresh":"2024-09-10T19:26:56.238908+0000","last_change":"2024-09-10T19:26:15.175676+0000","last_active":"2024-09-10T19:26:56.238908+0000","last_peered":"2024-09-10T19:26:56.238908+0000","last_clean":"2024-09-10T19:26:56.238908+0000","last_became_active":"2024-09-10T19:24:01.976587+0000","last_became_peered":"2024-09-10T19:24:01.976587+0000","last_unstale":"2024-09-10T19:26:56.238908+0000","last_undegraded":"2024-09-10T19:26:56.238908+0000","last_fullsized":"2024-09-10T19:26:56.238908+0000","mapping_epoch":292,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":293,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:26:15.175601+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:26:15.175601+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:27:36.641175+0000","scrub_duration":0.013129844,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":383,"reported_epoch":354,"state":"active+clean","last_fresh":"2024-09-10T19:26:56.237849+0000","last_change":"2024-09-10T19:24:07.659325+0000","last_active":"2024-09-10T19:26:56.237849+0000","last_peered":"2024-09-10T19:26:56.237849+0000","last_clean":"2024-09-10T19:26:56.237849+0000","last_became_active":"2024-09-10T19:24:01.970118+0000","last_became_peered":"2024-09-10T19:24:01.970118+0000","last_unstale":"2024-09-10T19:26:56.237849+0000","last_undegraded":"2024-09-10T19:26:56.237849+0000","last_fullsized":"2024-09-10T19:26:56.237849+0000","mapping_epoch":292,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":293,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:24:07.659224+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:24:07.659224+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001912307,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":375,"reported_epoch":354,"state":"active+clean","last_fresh":"2024-09-10T19:26:56.239284+0000","last_change":"2024-09-10T19:24:53.823606+0000","last_active":"2024-09-10T19:26:56.239284+0000","last_peered":"2024-09-10T19:26:56.239284+0000","last_clean":"2024-09-10T19:26:56.239284+0000","last_became_active":"2024-09-10T19:16:24.858800+0000","last_became_peered":"2024-09-10T19:16:24.858800+0000","last_unstale":"2024-09-10T19:26:56.239284+0000","last_undegraded":"2024-09-10T19:26:56.239284+0000","last_fullsized":"2024-09-10T19:26:56.239284+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:24:53.823351+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:24:53.823351+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0022636959999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1a","version":"237'15","reported_seq":397,"reported_epoch":354,"state":"active+clean","last_fresh":"2024-09-10T19:26:56.245747+0000","last_change":"2024-09-10T19:25:14.879733+0000","last_active":"2024-09-10T19:26:56.245747+0000","last_peered":"2024-09-10T19:26:56.245747+0000","last_clean":"2024-09-10T19:26:56.245747+0000","last_became_active":"2024-09-10T19:24:23.891213+0000","last_became_peered":"2024-09-10T19:24:23.891213+0000","last_unstale":"2024-09-10T19:26:56.245747+0000","last_undegraded":"2024-09-10T19:26:56.245747+0000","last_fullsized":"2024-09-10T19:26:56.245747+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"237'15","last_scrub_stamp":"2024-09-10T19:25:14.879649+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:25:14.879649+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0026001190000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":368,"reported_epoch":354,"state":"active+clean","last_fresh":"2024-09-10T19:26:56.247649+0000","last_change":"2024-09-10T19:25:13.751512+0000","last_active":"2024-09-10T19:26:56.247649+0000","last_peered":"2024-09-10T19:26:56.247649+0000","last_clean":"2024-09-10T19:26:56.247649+0000","last_became_active":"2024-09-10T19:24:23.907653+0000","last_became_peered":"2024-09-10T19:24:23.907653+0000","last_unstale":"2024-09-10T19:26:56.247649+0000","last_undegraded":"2024-09-10T19:26:56.247649+0000","last_fullsized":"2024-09-10T19:26:56.247649+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:25:13.751431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:25:13.751431+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0091990130000000007,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":342,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-09-10T19:26:58.263670+0000","last_change":"2024-09-10T19:26:07.493741+0000","last_active":"2024-09-10T19:26:58.263670+0000","last_peered":"2024-09-10T19:26:58.263670+0000","last_clean":"2024-09-10T19:26:58.263670+0000","last_became_active":"2024-09-10T19:24:24.907431+0000","last_became_peered":"2024-09-10T19:24:24.907431+0000","last_unstale":"2024-09-10T19:26:58.263670+0000","last_undegraded":"2024-09-10T19:26:58.263670+0000","last_fullsized":"2024-09-10T19:26:58.263670+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:26:07.493656+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:26:07.493656+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:27:10.117261+0000","scrub_duration":0.0071789510000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.3","version":"355'13357","reported_seq":13425,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-09-10T19:27:00.476724+0000","last_change":"2024-09-10T19:24:09.045117+0000","last_active":"2024-09-10T19:27:00.476724+0000","last_peered":"2024-09-10T19:27:00.476724+0000","last_clean":"2024-09-10T19:27:00.476724+0000","last_became_active":"2024-09-10T19:24:09.043872+0000","last_became_peered":"2024-09-10T19:24:09.043872+0000","last_unstale":"2024-09-10T19:27:00.476724+0000","last_undegraded":"2024-09-10T19:27:00.476724+0000","last_fullsized":"2024-09-10T19:27:00.476724+0000","mapping_epoch":295,"log_start":"342'8900","ondisk_log_start":"342'8900","created":295,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:24:08.023080+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:24:08.023080+0000","last_clean_scrub_stamp":"2024-09-10T19:24:08.023080+0000","objects_scrubbed":0,"log_size":4457,"log_dups_size":0,"ondisk_log_size":4457,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":125108224,"num_objects":1909,"num_object_clones":0,"num_object_copies":5727,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1909,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13357,"num_write_kb":488512,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.8","version":"74'79","reported_seq":145,"reported_epoch":354,"state":"active+clean","last_fresh":"2024-09-10T19:26:56.238821+0000","last_change":"2024-09-10T19:25:17.772519+0000","last_active":"2024-09-10T19:26:56.238821+0000","last_peered":"2024-09-10T19:26:56.238821+0000","last_clean":"2024-09-10T19:26:56.238821+0000","last_became_active":"2024-09-10T19:20:30.839050+0000","last_became_peered":"2024-09-10T19:20:30.839050+0000","last_unstale":"2024-09-10T19:26:56.238821+0000","last_undegraded":"2024-09-10T19:26:56.238821+0000","last_fullsized":"2024-09-10T19:26:56.238821+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":4,"last_scrub":"74'79","last_scrub_stamp":"2024-09-10T19:25:17.772431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:08:00.699315+0000","last_clean_scrub_stamp":"2024-09-10T19:25:17.772431+0000","objects_scrubbed":1,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.015350954,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.d","version":"237'31","reported_seq":509,"reported_epoch":354,"state":"active+clean","last_fresh":"2024-09-10T19:26:56.245500+0000","last_change":"2024-09-10T19:26:27.880510+0000","last_active":"2024-09-10T19:26:56.245500+0000","last_peered":"2024-09-10T19:26:56.245500+0000","last_clean":"2024-09-10T19:26:56.245500+0000","last_became_active":"2024-09-10T19:24:01.975691+0000","last_became_peered":"2024-09-10T19:24:01.975691+0000","last_unstale":"2024-09-10T19:26:56.245500+0000","last_undegraded":"2024-09-10T19:26:56.245500+0000","last_fullsized":"2024-09-10T19:26:56.245500+0000","mapping_epoch":292,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":293,"parent":"0.0","parent_split_bits":0,"last_scrub":"237'31","last_scrub_stamp":"2024-09-10T19:26:27.880373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:26:27.880373+0000","objects_scrubbed":8,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:27:28.551207+0000","scrub_duration":0.0086552290000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":50,"num_read_kb":40,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,6],"acting":[4,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":375,"reported_epoch":354,"state":"active+clean","last_fresh":"2024-09-10T19:26:56.239734+0000","last_change":"2024-09-10T19:26:08.089251+0000","last_active":"2024-09-10T19:26:56.239734+0000","last_peered":"2024-09-10T19:26:56.239734+0000","last_clean":"2024-09-10T19:26:56.239734+0000","last_became_active":"2024-09-10T19:10:07.691119+0000","last_became_peered":"2024-09-10T19:10:07.691119+0000","last_unstale":"2024-09-10T19:26:56.239734+0000","last_undegraded":"2024-09-10T19:26:56.239734+0000","last_fullsized":"2024-09-10T19:26:56.239734+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:26:08.089172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:16:39.716674+0000","last_clean_scrub_stamp":"2024-09-10T19:26:08.089172+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:27:16.557379+0000","scrub_duration":0.0047132800000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":97,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-09-10T19:26:58.262866+0000","last_change":"2024-09-10T19:24:46.462624+0000","last_active":"2024-09-10T19:26:58.262866+0000","last_peered":"2024-09-10T19:26:58.262866+0000","last_clean":"2024-09-10T19:26:58.262866+0000","last_became_active":"2024-09-10T19:24:24.886479+0000","last_became_peered":"2024-09-10T19:24:24.886479+0000","last_unstale":"2024-09-10T19:26:58.262866+0000","last_undegraded":"2024-09-10T19:26:58.262866+0000","last_fullsized":"2024-09-10T19:26:58.262866+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:24:46.462536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:24:46.462536+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.005382062,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":243,"reported_epoch":354,"state":"active+clean","last_fresh":"2024-09-10T19:26:56.237024+0000","last_change":"2024-09-10T19:26:08.894794+0000","last_active":"2024-09-10T19:26:56.237024+0000","last_peered":"2024-09-10T19:26:56.237024+0000","last_clean":"2024-09-10T19:26:56.237024+0000","last_became_active":"2024-09-10T19:16:24.857843+0000","last_became_peered":"2024-09-10T19:16:24.857843+0000","last_unstale":"2024-09-10T19:26:56.237024+0000","last_undegraded":"2024-09-10T19:26:56.237024+0000","last_fullsized":"2024-09-10T19:26:56.237024+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:26:08.894702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:26:08.894702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:27:33.178000+0000","scrub_duration":0.010734235,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.2","version":"356'13792","reported_seq":15099,"reported_epoch":356,"state":"active+clean","last_fresh":"2024-09-10T19:27:02.220023+0000","last_change":"2024-09-10T19:26:37.419041+0000","last_active":"2024-09-10T19:27:02.220023+0000","last_peered":"2024-09-10T19:27:02.220023+0000","last_clean":"2024-09-10T19:27:02.220023+0000","last_became_active":"2024-09-10T19:24:27.932777+0000","last_became_peered":"2024-09-10T19:24:27.932777+0000","last_unstale":"2024-09-10T19:27:02.220023+0000","last_undegraded":"2024-09-10T19:27:02.220023+0000","last_fullsized":"2024-09-10T19:27:02.220023+0000","mapping_epoch":305,"log_start":"342'9000","ondisk_log_start":"342'9000","created":295,"last_epoch_clean":306,"parent":"0.0","parent_split_bits":0,"last_scrub":"350'11705","last_scrub_stamp":"2024-09-10T19:26:37.418937+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:24:08.023080+0000","last_clean_scrub_stamp":"2024-09-10T19:26:37.418937+0000","objects_scrubbed":3909,"log_size":4792,"log_dups_size":0,"ondisk_log_size":4792,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:28:07.315479+0000","scrub_duration":1.013252139,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":116523008,"num_objects":1778,"num_object_clones":0,"num_object_copies":5334,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1778,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13792,"num_write_kb":498240,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":526,"num_bytes_recovered":34471936,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.9","version":"74'79","reported_seq":430,"reported_epoch":354,"state":"active+clean","last_fresh":"2024-09-10T19:26:56.240063+0000","last_change":"2024-09-10T19:25:12.867805+0000","last_active":"2024-09-10T19:26:56.240063+0000","last_peered":"2024-09-10T19:26:56.240063+0000","last_clean":"2024-09-10T19:26:56.240063+0000","last_became_active":"2024-09-10T19:20:30.844108+0000","last_became_peered":"2024-09-10T19:20:30.844108+0000","last_unstale":"2024-09-10T19:26:56.240063+0000","last_undegraded":"2024-09-10T19:26:56.240063+0000","last_fullsized":"2024-09-10T19:26:56.240063+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":4,"last_scrub":"74'79","last_scrub_stamp":"2024-09-10T19:25:12.867565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:08:00.699315+0000","last_clean_scrub_stamp":"2024-09-10T19:25:12.867565+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.022682969000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.c","version":"237'20","reported_seq":405,"reported_epoch":354,"state":"active+clean","last_fresh":"2024-09-10T19:26:56.246681+0000","last_change":"2024-09-10T19:25:16.888274+0000","last_active":"2024-09-10T19:26:56.246681+0000","last_peered":"2024-09-10T19:26:56.246681+0000","last_clean":"2024-09-10T19:26:56.246681+0000","last_became_active":"2024-09-10T19:10:09.688321+0000","last_became_peered":"2024-09-10T19:10:09.688321+0000","last_unstale":"2024-09-10T19:26:56.246681+0000","last_undegraded":"2024-09-10T19:26:56.246681+0000","last_fullsized":"2024-09-10T19:26:56.246681+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"237'20","last_scrub_stamp":"2024-09-10T19:25:16.887901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:25:16.887901+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.012411174000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":83,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-09-10T19:26:58.263020+0000","last_change":"2024-09-10T19:25:29.750938+0000","last_active":"2024-09-10T19:26:58.263020+0000","last_peered":"2024-09-10T19:26:58.263020+0000","last_clean":"2024-09-10T19:26:58.263020+0000","last_became_active":"2024-09-10T19:24:01.973848+0000","last_became_peered":"2024-09-10T19:24:01.973848+0000","last_unstale":"2024-09-10T19:26:58.263020+0000","last_undegraded":"2024-09-10T19:26:58.263020+0000","last_fullsized":"2024-09-10T19:26:58.263020+0000","mapping_epoch":292,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":293,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:25:29.750877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:25:29.750877+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001313533,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.d","version":"303'12","reported_seq":449,"reported_epoch":354,"state":"active+clean","last_fresh":"2024-09-10T19:26:56.237773+0000","last_change":"2024-09-10T19:24:36.790767+0000","last_active":"2024-09-10T19:26:56.237773+0000","last_peered":"2024-09-10T19:26:56.237773+0000","last_clean":"2024-09-10T19:26:56.237773+0000","last_became_active":"2024-09-10T19:24:23.906819+0000","last_became_peered":"2024-09-10T19:24:23.906819+0000","last_unstale":"2024-09-10T19:26:56.237773+0000","last_undegraded":"2024-09-10T19:26:56.237773+0000","last_fullsized":"2024-09-10T19:26:56.237773+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"303'12","last_scrub_stamp":"2024-09-10T19:24:36.790665+0000","last_deep_scrub":"80'9","last_deep_scrub_stamp":"2024-09-10T19:11:25.156465+0000","last_clean_scrub_stamp":"2024-09-10T19:24:36.790665+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0028602430000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,0],"acting":[7,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":401,"reported_epoch":354,"state":"active+clean","last_fresh":"2024-09-10T19:26:56.247545+0000","last_change":"2024-09-10T19:26:13.829815+0000","last_active":"2024-09-10T19:26:56.247545+0000","last_peered":"2024-09-10T19:26:56.247545+0000","last_clean":"2024-09-10T19:26:56.247545+0000","last_became_active":"2024-09-10T19:24:01.972593+0000","last_became_peered":"2024-09-10T19:24:01.972593+0000","last_unstale":"2024-09-10T19:26:56.247545+0000","last_undegraded":"2024-09-10T19:26:56.247545+0000","last_fullsized":"2024-09-10T19:26:56.247545+0000","mapping_epoch":292,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":293,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:26:13.829736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:26:13.829736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:27:16.644048+0000","scrub_duration":0.025408242000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,5],"acting":[4,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.1","version":"355'13676","reported_seq":14654,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-09-10T19:27:00.143330+0000","last_change":"2024-09-10T19:24:29.878172+0000","last_active":"2024-09-10T19:27:00.143330+0000","last_peered":"2024-09-10T19:27:00.143330+0000","last_clean":"2024-09-10T19:27:00.143330+0000","last_became_active":"2024-09-10T19:24:29.875287+0000","last_became_peered":"2024-09-10T19:24:29.875287+0000","last_unstale":"2024-09-10T19:27:00.143330+0000","last_undegraded":"2024-09-10T19:27:00.143330+0000","last_fullsized":"2024-09-10T19:27:00.143330+0000","mapping_epoch":307,"log_start":"343'9400","ondisk_log_start":"343'9400","created":295,"last_epoch_clean":308,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:24:08.023080+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:24:08.023080+0000","last_clean_scrub_stamp":"2024-09-10T19:24:08.023080+0000","objects_scrubbed":0,"log_size":4276,"log_dups_size":0,"ondisk_log_size":4276,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":131858432,"num_objects":2012,"num_object_clones":0,"num_object_copies":6036,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2012,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13676,"num_write_kb":502016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":446,"num_bytes_recovered":29229056,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.a","version":"74'79","reported_seq":140,"reported_epoch":354,"state":"active+clean","last_fresh":"2024-09-10T19:26:56.234355+0000","last_change":"2024-09-10T19:25:18.795091+0000","last_active":"2024-09-10T19:26:56.234355+0000","last_peered":"2024-09-10T19:26:56.234355+0000","last_clean":"2024-09-10T19:26:56.234355+0000","last_became_active":"2024-09-10T19:20:30.844773+0000","last_became_peered":"2024-09-10T19:20:30.844773+0000","last_unstale":"2024-09-10T19:26:56.234355+0000","last_undegraded":"2024-09-10T19:26:56.234355+0000","last_fullsized":"2024-09-10T19:26:56.234355+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":4,"last_scrub":"74'79","last_scrub_stamp":"2024-09-10T19:25:18.795003+0000","last_deep_scrub":"74'79","last_deep_scrub_stamp":"2024-09-10T19:21:53.326433+0000","last_clean_scrub_stamp":"2024-09-10T19:25:18.795003+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0047392160000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,4],"acting":[5,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.f","version":"237'27","reported_seq":99,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-09-10T19:26:58.287174+0000","last_change":"2024-09-10T19:26:17.458249+0000","last_active":"2024-09-10T19:26:58.287174+0000","last_peered":"2024-09-10T19:26:58.287174+0000","last_clean":"2024-09-10T19:26:58.287174+0000","last_became_active":"2024-09-10T19:24:01.979187+0000","last_became_peered":"2024-09-10T19:24:01.979187+0000","last_unstale":"2024-09-10T19:26:58.287174+0000","last_undegraded":"2024-09-10T19:26:58.287174+0000","last_fullsized":"2024-09-10T19:26:58.287174+0000","mapping_epoch":292,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":293,"parent":"0.0","parent_split_bits":0,"last_scrub":"237'27","last_scrub_stamp":"2024-09-10T19:26:17.458177+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:26:17.458177+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:27:23.370611+0000","scrub_duration":0.0044892370000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":83,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-09-10T19:26:58.262777+0000","last_change":"2024-09-10T19:26:08.391803+0000","last_active":"2024-09-10T19:26:58.262777+0000","last_peered":"2024-09-10T19:26:58.262777+0000","last_clean":"2024-09-10T19:26:58.262777+0000","last_became_active":"2024-09-10T19:24:01.974773+0000","last_became_peered":"2024-09-10T19:24:01.974773+0000","last_unstale":"2024-09-10T19:26:58.262777+0000","last_undegraded":"2024-09-10T19:26:58.262777+0000","last_fullsized":"2024-09-10T19:26:58.262777+0000","mapping_epoch":292,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":293,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:26:08.391730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:26:08.391730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:27:18.740314+0000","scrub_duration":0.0020244629999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.e","version":"303'10","reported_seq":572,"reported_epoch":354,"state":"active+clean","last_fresh":"2024-09-10T19:26:56.247676+0000","last_change":"2024-09-10T19:24:41.682443+0000","last_active":"2024-09-10T19:26:56.247676+0000","last_peered":"2024-09-10T19:26:56.247676+0000","last_clean":"2024-09-10T19:26:56.247676+0000","last_became_active":"2024-09-10T19:24:23.908760+0000","last_became_peered":"2024-09-10T19:24:23.908760+0000","last_unstale":"2024-09-10T19:26:56.247676+0000","last_undegraded":"2024-09-10T19:26:56.247676+0000","last_fullsized":"2024-09-10T19:26:56.247676+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"303'10","last_scrub_stamp":"2024-09-10T19:24:41.682347+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:24:41.682347+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0099881439999999991,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":407,"reported_epoch":354,"state":"active+clean","last_fresh":"2024-09-10T19:26:56.235063+0000","last_change":"2024-09-10T19:26:34.847681+0000","last_active":"2024-09-10T19:26:56.235063+0000","last_peered":"2024-09-10T19:26:56.235063+0000","last_clean":"2024-09-10T19:26:56.235063+0000","last_became_active":"2024-09-10T19:24:24.917109+0000","last_became_peered":"2024-09-10T19:24:24.917109+0000","last_unstale":"2024-09-10T19:26:56.235063+0000","last_undegraded":"2024-09-10T19:26:56.235063+0000","last_fullsized":"2024-09-10T19:26:56.235063+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:26:34.847602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:26:34.847602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:27:54.686477+0000","scrub_duration":0.0034671599999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.0","version":"355'13185","reported_seq":13500,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-09-10T19:26:59.687343+0000","last_change":"2024-09-10T19:25:31.237539+0000","last_active":"2024-09-10T19:26:59.687343+0000","last_peered":"2024-09-10T19:26:59.687343+0000","last_clean":"2024-09-10T19:26:59.687343+0000","last_became_active":"2024-09-10T19:24:09.046656+0000","last_became_peered":"2024-09-10T19:24:09.046656+0000","last_unstale":"2024-09-10T19:26:59.687343+0000","last_undegraded":"2024-09-10T19:26:59.687343+0000","last_fullsized":"2024-09-10T19:26:59.687343+0000","mapping_epoch":295,"log_start":"341'8400","ondisk_log_start":"341'8400","created":295,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"333'6034","last_scrub_stamp":"2024-09-10T19:25:31.237447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:24:08.023080+0000","last_clean_scrub_stamp":"2024-09-10T19:25:31.237447+0000","objects_scrubbed":5911,"log_size":4785,"log_dups_size":0,"ondisk_log_size":4785,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"queued for scrub","scrub_duration":2.954390187,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":129826816,"num_objects":1981,"num_object_clones":0,"num_object_copies":5943,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1981,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13185,"num_write_kb":485312,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.e","version":"237'24","reported_seq":474,"reported_epoch":354,"state":"active+clean","last_fresh":"2024-09-10T19:26:56.247746+0000","last_change":"2024-09-10T19:25:22.898084+0000","last_active":"2024-09-10T19:26:56.247746+0000","last_peered":"2024-09-10T19:26:56.247746+0000","last_clean":"2024-09-10T19:26:56.247746+0000","last_became_active":"2024-09-10T19:24:44.894167+0000","last_became_peered":"2024-09-10T19:24:44.894167+0000","last_unstale":"2024-09-10T19:26:56.247746+0000","last_undegraded":"2024-09-10T19:26:56.247746+0000","last_fullsized":"2024-09-10T19:26:56.247746+0000","mapping_epoch":314,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":315,"parent":"0.0","parent_split_bits":0,"last_scrub":"237'24","last_scrub_stamp":"2024-09-10T19:25:22.897993+0000","last_deep_scrub":"84'16","last_deep_scrub_stamp":"2024-09-10T19:11:56.157975+0000","last_clean_scrub_stamp":"2024-09-10T19:25:22.897993+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.021319980999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":46,"reported_epoch":354,"state":"active+clean","last_fresh":"2024-09-10T19:26:56.242248+0000","last_change":"2024-09-10T19:24:53.756744+0000","last_active":"2024-09-10T19:26:56.242248+0000","last_peered":"2024-09-10T19:26:56.242248+0000","last_clean":"2024-09-10T19:26:56.242248+0000","last_became_active":"2024-09-10T19:24:53.756220+0000","last_became_peered":"2024-09-10T19:24:53.756220+0000","last_unstale":"2024-09-10T19:26:56.242248+0000","last_undegraded":"2024-09-10T19:26:56.242248+0000","last_fullsized":"2024-09-10T19:26:56.242248+0000","mapping_epoch":318,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":319,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:24:13.636249+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:24:13.636249+0000","last_clean_scrub_stamp":"2024-09-10T19:24:13.636249+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001732363,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,3],"acting":[4,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":351,"reported_epoch":354,"state":"active+clean","last_fresh":"2024-09-10T19:26:56.234888+0000","last_change":"2024-09-10T19:25:29.715973+0000","last_active":"2024-09-10T19:26:56.234888+0000","last_peered":"2024-09-10T19:26:56.234888+0000","last_clean":"2024-09-10T19:26:56.234888+0000","last_became_active":"2024-09-10T19:10:11.692215+0000","last_became_peered":"2024-09-10T19:10:11.692215+0000","last_unstale":"2024-09-10T19:26:56.234888+0000","last_undegraded":"2024-09-10T19:26:56.234888+0000","last_fullsized":"2024-09-10T19:26:56.234888+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:25:29.715888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:25:29.715888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.024627837999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":349,"reported_epoch":354,"state":"active+clean","last_fresh":"2024-09-10T19:26:56.238812+0000","last_change":"2024-09-10T19:25:13.722998+0000","last_active":"2024-09-10T19:26:56.238812+0000","last_peered":"2024-09-10T19:26:56.238812+0000","last_clean":"2024-09-10T19:26:56.238812+0000","last_became_active":"2024-09-10T19:10:14.053736+0000","last_became_peered":"2024-09-10T19:10:14.053736+0000","last_unstale":"2024-09-10T19:26:56.238812+0000","last_undegraded":"2024-09-10T19:26:56.238812+0000","last_fullsized":"2024-09-10T19:26:56.238812+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:25:13.722943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:25:13.722943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0024886520000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.f","version":"355'13362","reported_seq":13648,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-09-10T19:26:59.721071+0000","last_change":"2024-09-10T19:25:21.021707+0000","last_active":"2024-09-10T19:26:59.721071+0000","last_peered":"2024-09-10T19:26:59.721071+0000","last_clean":"2024-09-10T19:26:59.721071+0000","last_became_active":"2024-09-10T19:24:09.047213+0000","last_became_peered":"2024-09-10T19:24:09.047213+0000","last_unstale":"2024-09-10T19:26:59.721071+0000","last_undegraded":"2024-09-10T19:26:59.721071+0000","last_fullsized":"2024-09-10T19:26:59.721071+0000","mapping_epoch":295,"log_start":"341'8600","ondisk_log_start":"341'8600","created":295,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"331'5382","last_scrub_stamp":"2024-09-10T19:25:21.021381+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:24:08.023080+0000","last_clean_scrub_stamp":"2024-09-10T19:25:21.021381+0000","objects_scrubbed":5280,"log_size":4762,"log_dups_size":0,"ondisk_log_size":4762,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"queued for scrub","scrub_duration":2.7958712999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":129892352,"num_objects":1982,"num_object_clones":0,"num_object_copies":5946,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1982,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13362,"num_write_kb":491008,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,4],"acting":[6,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.4","version":"317'80","reported_seq":156,"reported_epoch":354,"state":"active+clean","last_fresh":"2024-09-10T19:26:56.236150+0000","last_change":"2024-09-10T19:26:34.932583+0000","last_active":"2024-09-10T19:26:56.236150+0000","last_peered":"2024-09-10T19:26:56.236150+0000","last_clean":"2024-09-10T19:26:56.236150+0000","last_became_active":"2024-09-10T19:20:30.838441+0000","last_became_peered":"2024-09-10T19:20:30.838441+0000","last_unstale":"2024-09-10T19:26:56.236150+0000","last_undegraded":"2024-09-10T19:26:56.236150+0000","last_fullsized":"2024-09-10T19:26:56.236150+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":3,"last_scrub":"317'80","last_scrub_stamp":"2024-09-10T19:26:34.932490+0000","last_deep_scrub":"317'80","last_deep_scrub_stamp":"2024-09-10T19:24:51.540456+0000","last_clean_scrub_stamp":"2024-09-10T19:26:34.932490+0000","objects_scrubbed":1,"log_size":80,"log_dups_size":0,"ondisk_log_size":80,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:27:38.776542+0000","scrub_duration":0.010119784999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"265'35","reported_seq":297,"reported_epoch":354,"state":"active+clean","last_fresh":"2024-09-10T19:26:56.245678+0000","last_change":"2024-09-10T19:25:29.871366+0000","last_active":"2024-09-10T19:26:56.245678+0000","last_peered":"2024-09-10T19:26:56.245678+0000","last_clean":"2024-09-10T19:26:56.245678+0000","last_became_active":"2024-09-10T19:16:24.852821+0000","last_became_peered":"2024-09-10T19:16:24.852821+0000","last_unstale":"2024-09-10T19:26:56.245678+0000","last_undegraded":"2024-09-10T19:26:56.245678+0000","last_fullsized":"2024-09-10T19:26:56.245678+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"265'35","last_scrub_stamp":"2024-09-10T19:25:29.871291+0000","last_deep_scrub":"265'35","last_deep_scrub_stamp":"2024-09-10T19:25:29.871291+0000","last_clean_scrub_stamp":"2024-09-10T19:25:29.871291+0000","objects_scrubbed":7,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0058405009999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":42,"num_read_kb":35,"num_write":28,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,7],"acting":[4,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":385,"reported_epoch":354,"state":"active+clean","last_fresh":"2024-09-10T19:26:56.246035+0000","last_change":"2024-09-10T19:24:49.962031+0000","last_active":"2024-09-10T19:26:56.246035+0000","last_peered":"2024-09-10T19:26:56.246035+0000","last_clean":"2024-09-10T19:26:56.246035+0000","last_became_active":"2024-09-10T19:16:24.855861+0000","last_became_peered":"2024-09-10T19:16:24.855861+0000","last_unstale":"2024-09-10T19:26:56.246035+0000","last_undegraded":"2024-09-10T19:26:56.246035+0000","last_fullsized":"2024-09-10T19:26:56.246035+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:24:49.961960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:24:49.961960+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.015143467000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.0","version":"294'12","reported_seq":605,"reported_epoch":354,"state":"active+clean","last_fresh":"2024-09-10T19:26:56.234992+0000","last_change":"2024-09-10T19:26:18.869349+0000","last_active":"2024-09-10T19:26:56.234992+0000","last_peered":"2024-09-10T19:26:56.234992+0000","last_clean":"2024-09-10T19:26:56.234992+0000","last_became_active":"2024-09-10T19:24:01.972645+0000","last_became_peered":"2024-09-10T19:24:01.972645+0000","last_unstale":"2024-09-10T19:26:56.234992+0000","last_undegraded":"2024-09-10T19:26:56.234992+0000","last_fullsized":"2024-09-10T19:26:56.234992+0000","mapping_epoch":292,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":293,"parent":"0.0","parent_split_bits":0,"last_scrub":"294'12","last_scrub_stamp":"2024-09-10T19:26:18.869259+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:26:18.869259+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:27:30.361030+0000","scrub_duration":0.0029728440000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":376,"reported_epoch":354,"state":"active+clean","last_fresh":"2024-09-10T19:26:56.239404+0000","last_change":"2024-09-10T19:26:25.368855+0000","last_active":"2024-09-10T19:26:56.239404+0000","last_peered":"2024-09-10T19:26:56.239404+0000","last_clean":"2024-09-10T19:26:56.239404+0000","last_became_active":"2024-09-10T19:24:01.974621+0000","last_became_peered":"2024-09-10T19:24:01.974621+0000","last_unstale":"2024-09-10T19:26:56.239404+0000","last_undegraded":"2024-09-10T19:26:56.239404+0000","last_fullsized":"2024-09-10T19:26:56.239404+0000","mapping_epoch":292,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":293,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:26:25.368770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:11:41.073833+0000","last_clean_scrub_stamp":"2024-09-10T19:26:25.368770+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:27:50.731053+0000","scrub_duration":0.0030199950000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.e","version":"355'13155","reported_seq":13223,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-09-10T19:27:00.148946+0000","last_change":"2024-09-10T19:24:09.046030+0000","last_active":"2024-09-10T19:27:00.148946+0000","last_peered":"2024-09-10T19:27:00.148946+0000","last_clean":"2024-09-10T19:27:00.148946+0000","last_became_active":"2024-09-10T19:24:09.045762+0000","last_became_peered":"2024-09-10T19:24:09.045762+0000","last_unstale":"2024-09-10T19:27:00.148946+0000","last_undegraded":"2024-09-10T19:27:00.148946+0000","last_fullsized":"2024-09-10T19:27:00.148946+0000","mapping_epoch":295,"log_start":"342'8900","ondisk_log_start":"342'8900","created":295,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:24:08.023080+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:24:08.023080+0000","last_clean_scrub_stamp":"2024-09-10T19:24:08.023080+0000","objects_scrubbed":0,"log_size":4255,"log_dups_size":0,"ondisk_log_size":4255,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":127729664,"num_objects":1949,"num_object_clones":0,"num_object_copies":5847,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1949,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13155,"num_write_kb":483328,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.5","version":"74'79","reported_seq":150,"reported_epoch":354,"state":"active+clean","last_fresh":"2024-09-10T19:26:56.245172+0000","last_change":"2024-09-10T19:25:15.865396+0000","last_active":"2024-09-10T19:26:56.245172+0000","last_peered":"2024-09-10T19:26:56.245172+0000","last_clean":"2024-09-10T19:26:56.245172+0000","last_became_active":"2024-09-10T19:20:30.838556+0000","last_became_peered":"2024-09-10T19:20:30.838556+0000","last_unstale":"2024-09-10T19:26:56.245172+0000","last_undegraded":"2024-09-10T19:26:56.245172+0000","last_fullsized":"2024-09-10T19:26:56.245172+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":3,"last_scrub":"74'79","last_scrub_stamp":"2024-09-10T19:25:15.865263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:08:00.699315+0000","last_clean_scrub_stamp":"2024-09-10T19:25:15.865263+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0027887379999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"237'30","reported_seq":410,"reported_epoch":354,"state":"active+clean","last_fresh":"2024-09-10T19:26:56.235078+0000","last_change":"2024-09-10T19:24:37.476248+0000","last_active":"2024-09-10T19:26:56.235078+0000","last_peered":"2024-09-10T19:26:56.235078+0000","last_clean":"2024-09-10T19:26:56.235078+0000","last_became_active":"2024-09-10T19:24:23.903784+0000","last_became_peered":"2024-09-10T19:24:23.903784+0000","last_unstale":"2024-09-10T19:26:56.235078+0000","last_undegraded":"2024-09-10T19:26:56.235078+0000","last_fullsized":"2024-09-10T19:26:56.235078+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"237'30","last_scrub_stamp":"2024-09-10T19:24:37.476174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:24:37.476174+0000","objects_scrubbed":8,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0032561880000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":52,"num_read_kb":40,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":83,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-09-10T19:26:58.286824+0000","last_change":"2024-09-10T19:25:14.621455+0000","last_active":"2024-09-10T19:26:58.286824+0000","last_peered":"2024-09-10T19:26:58.286824+0000","last_clean":"2024-09-10T19:26:58.286824+0000","last_became_active":"2024-09-10T19:24:01.978995+0000","last_became_peered":"2024-09-10T19:24:01.978995+0000","last_unstale":"2024-09-10T19:26:58.286824+0000","last_undegraded":"2024-09-10T19:26:58.286824+0000","last_fullsized":"2024-09-10T19:26:58.286824+0000","mapping_epoch":292,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":293,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:25:14.621399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:17:26.374376+0000","last_clean_scrub_stamp":"2024-09-10T19:25:14.621399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0050623270000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":370,"reported_epoch":354,"state":"active+clean","last_fresh":"2024-09-10T19:26:56.245885+0000","last_change":"2024-09-10T19:26:24.897597+0000","last_active":"2024-09-10T19:26:56.245885+0000","last_peered":"2024-09-10T19:26:56.245885+0000","last_clean":"2024-09-10T19:26:56.245885+0000","last_became_active":"2024-09-10T19:10:11.698054+0000","last_became_peered":"2024-09-10T19:10:11.698054+0000","last_unstale":"2024-09-10T19:26:56.245885+0000","last_undegraded":"2024-09-10T19:26:56.245885+0000","last_fullsized":"2024-09-10T19:26:56.245885+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:26:24.897519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:26:24.897519+0000","last_clean_scrub_stamp":"2024-09-10T19:26:24.897519+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:27:42.219567+0000","scrub_duration":0.0035237760000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":368,"reported_epoch":354,"state":"active+clean","last_fresh":"2024-09-10T19:26:56.245677+0000","last_change":"2024-09-10T19:23:55.452716+0000","last_active":"2024-09-10T19:26:56.245677+0000","last_peered":"2024-09-10T19:26:56.245677+0000","last_clean":"2024-09-10T19:26:56.245677+0000","last_became_active":"2024-09-10T19:16:24.848109+0000","last_became_peered":"2024-09-10T19:16:24.848109+0000","last_unstale":"2024-09-10T19:26:56.245677+0000","last_undegraded":"2024-09-10T19:26:56.245677+0000","last_fullsized":"2024-09-10T19:26:56.245677+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:23:55.452654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:23:55.452654+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0018215709999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.d","version":"355'13264","reported_seq":14491,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-09-10T19:26:59.661361+0000","last_change":"2024-09-10T19:26:20.870487+0000","last_active":"2024-09-10T19:26:59.661361+0000","last_peered":"2024-09-10T19:26:59.661361+0000","last_clean":"2024-09-10T19:26:59.661361+0000","last_became_active":"2024-09-10T19:24:28.941711+0000","last_became_peered":"2024-09-10T19:24:28.941711+0000","last_unstale":"2024-09-10T19:26:59.661361+0000","last_undegraded":"2024-09-10T19:26:59.661361+0000","last_fullsized":"2024-09-10T19:26:59.661361+0000","mapping_epoch":306,"log_start":"341'8500","ondisk_log_start":"341'8500","created":295,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":0,"last_scrub":"346'10089","last_scrub_stamp":"2024-09-10T19:26:20.870323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:24:08.023080+0000","last_clean_scrub_stamp":"2024-09-10T19:26:20.870323+0000","objects_scrubbed":5326,"log_size":4764,"log_dups_size":0,"ondisk_log_size":4764,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:27:45.236018+0000","scrub_duration":2.477425953,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":134414360,"num_objects":2052,"num_object_clones":0,"num_object_copies":6156,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2052,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":13264,"num_write_kb":490049,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":456,"num_bytes_recovered":29884416,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.6","version":"74'79","reported_seq":237,"reported_epoch":354,"state":"active+clean","last_fresh":"2024-09-10T19:26:56.245074+0000","last_change":"2024-09-10T19:26:30.421621+0000","last_active":"2024-09-10T19:26:56.245074+0000","last_peered":"2024-09-10T19:26:56.245074+0000","last_clean":"2024-09-10T19:26:56.245074+0000","last_became_active":"2024-09-10T19:24:23.902290+0000","last_became_peered":"2024-09-10T19:24:23.902290+0000","last_unstale":"2024-09-10T19:26:56.245074+0000","last_undegraded":"2024-09-10T19:26:56.245074+0000","last_fullsized":"2024-09-10T19:26:56.245074+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":3,"last_scrub":"74'79","last_scrub_stamp":"2024-09-10T19:26:30.421470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:08:00.699315+0000","last_clean_scrub_stamp":"2024-09-10T19:26:30.421470+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:27:44.391737+0000","scrub_duration":0.0045070300000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"237'37","reported_seq":371,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-09-10T19:26:58.297654+0000","last_change":"2024-09-10T19:26:36.464107+0000","last_active":"2024-09-10T19:26:58.297654+0000","last_peered":"2024-09-10T19:26:58.297654+0000","last_clean":"2024-09-10T19:26:58.297654+0000","last_became_active":"2024-09-10T19:24:23.901836+0000","last_became_peered":"2024-09-10T19:24:23.901836+0000","last_unstale":"2024-09-10T19:26:58.297654+0000","last_undegraded":"2024-09-10T19:26:58.297654+0000","last_fullsized":"2024-09-10T19:26:58.297654+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"237'37","last_scrub_stamp":"2024-09-10T19:26:36.464050+0000","last_deep_scrub":"104'27","last_deep_scrub_stamp":"2024-09-10T19:13:02.225944+0000","last_clean_scrub_stamp":"2024-09-10T19:26:36.464050+0000","objects_scrubbed":8,"log_size":37,"log_dups_size":0,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:27:56.488429+0000","scrub_duration":0.0033331939999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":53,"num_read_kb":39,"num_write":32,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":382,"reported_epoch":354,"state":"active+clean","last_fresh":"2024-09-10T19:26:56.238038+0000","last_change":"2024-09-10T19:26:29.426781+0000","last_active":"2024-09-10T19:26:56.238038+0000","last_peered":"2024-09-10T19:26:56.238038+0000","last_clean":"2024-09-10T19:26:56.238038+0000","last_became_active":"2024-09-10T19:24:23.899556+0000","last_became_peered":"2024-09-10T19:24:23.899556+0000","last_unstale":"2024-09-10T19:26:56.238038+0000","last_undegraded":"2024-09-10T19:26:56.238038+0000","last_fullsized":"2024-09-10T19:26:56.238038+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:26:29.426689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:16:37.779967+0000","last_clean_scrub_stamp":"2024-09-10T19:26:29.426689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:27:52.170491+0000","scrub_duration":0.0054033600000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":381,"reported_epoch":354,"state":"active+clean","last_fresh":"2024-09-10T19:26:56.246827+0000","last_change":"2024-09-10T19:26:28.366786+0000","last_active":"2024-09-10T19:26:56.246827+0000","last_peered":"2024-09-10T19:26:56.246827+0000","last_clean":"2024-09-10T19:26:56.246827+0000","last_became_active":"2024-09-10T19:24:23.901937+0000","last_became_peered":"2024-09-10T19:24:23.901937+0000","last_unstale":"2024-09-10T19:26:56.246827+0000","last_undegraded":"2024-09-10T19:26:56.246827+0000","last_fullsized":"2024-09-10T19:26:56.246827+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:26:28.366688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:26:28.366688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:27:41.195321+0000","scrub_duration":0.010474305,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":83,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-09-10T19:26:58.262706+0000","last_change":"2024-09-10T19:24:42.439826+0000","last_active":"2024-09-10T19:26:58.262706+0000","last_peered":"2024-09-10T19:26:58.262706+0000","last_clean":"2024-09-10T19:26:58.262706+0000","last_became_active":"2024-09-10T19:24:01.974076+0000","last_became_peered":"2024-09-10T19:24:01.974076+0000","last_unstale":"2024-09-10T19:26:58.262706+0000","last_undegraded":"2024-09-10T19:26:58.262706+0000","last_fullsized":"2024-09-10T19:26:58.262706+0000","mapping_epoch":292,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":293,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:24:42.439766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:24:42.439766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0031982790000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,7],"acting":[1,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.c","version":"355'13611","reported_seq":14940,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-09-10T19:27:00.406674+0000","last_change":"2024-09-10T19:26:16.570194+0000","last_active":"2024-09-10T19:27:00.406674+0000","last_peered":"2024-09-10T19:27:00.406674+0000","last_clean":"2024-09-10T19:27:00.406674+0000","last_became_active":"2024-09-10T19:24:28.941128+0000","last_became_peered":"2024-09-10T19:24:28.941128+0000","last_unstale":"2024-09-10T19:27:00.406674+0000","last_undegraded":"2024-09-10T19:27:00.406674+0000","last_fullsized":"2024-09-10T19:27:00.406674+0000","mapping_epoch":306,"log_start":"343'9200","ondisk_log_start":"343'9200","created":295,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":0,"last_scrub":"345'9915","last_scrub_stamp":"2024-09-10T19:26:16.570088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:24:08.023080+0000","last_clean_scrub_stamp":"2024-09-10T19:26:16.570088+0000","objects_scrubbed":5727,"log_size":4411,"log_dups_size":0,"ondisk_log_size":4411,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:27:19.007947+0000","scrub_duration":2.4177897690000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":126812160,"num_objects":1935,"num_object_clones":0,"num_object_copies":5805,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1935,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13611,"num_write_kb":497472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":501,"num_bytes_recovered":32833536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.7","version":"74'79","reported_seq":230,"reported_epoch":354,"state":"active+clean","last_fresh":"2024-09-10T19:26:56.245450+0000","last_change":"2024-09-10T19:26:31.437736+0000","last_active":"2024-09-10T19:26:56.245450+0000","last_peered":"2024-09-10T19:26:56.245450+0000","last_clean":"2024-09-10T19:26:56.245450+0000","last_became_active":"2024-09-10T19:20:30.842857+0000","last_became_peered":"2024-09-10T19:20:30.842857+0000","last_unstale":"2024-09-10T19:26:56.245450+0000","last_undegraded":"2024-09-10T19:26:56.245450+0000","last_fullsized":"2024-09-10T19:26:56.245450+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":3,"last_scrub":"74'79","last_scrub_stamp":"2024-09-10T19:26:31.437685+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:08:00.699315+0000","last_clean_scrub_stamp":"2024-09-10T19:26:31.437685+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:27:42.941791+0000","scrub_duration":0.00135721,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.2","version":"350'25","reported_seq":110,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-09-10T19:26:58.263108+0000","last_change":"2024-09-10T19:26:37.415152+0000","last_active":"2024-09-10T19:26:58.263108+0000","last_peered":"2024-09-10T19:26:58.263108+0000","last_clean":"2024-09-10T19:26:58.263108+0000","last_became_active":"2024-09-10T19:24:01.976841+0000","last_became_peered":"2024-09-10T19:24:01.976841+0000","last_unstale":"2024-09-10T19:26:58.263108+0000","last_undegraded":"2024-09-10T19:26:58.263108+0000","last_fullsized":"2024-09-10T19:26:58.263108+0000","mapping_epoch":292,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":293,"parent":"0.0","parent_split_bits":0,"last_scrub":"350'25","last_scrub_stamp":"2024-09-10T19:26:37.415039+0000","last_deep_scrub":"350'25","last_deep_scrub_stamp":"2024-09-10T19:26:37.415039+0000","last_clean_scrub_stamp":"2024-09-10T19:26:37.415039+0000","objects_scrubbed":5,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:27:43.198721+0000","scrub_duration":0.022344089000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":28,"num_read_kb":23,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":97,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-09-10T19:26:58.287694+0000","last_change":"2024-09-10T19:24:45.452542+0000","last_active":"2024-09-10T19:26:58.287694+0000","last_peered":"2024-09-10T19:26:58.287694+0000","last_clean":"2024-09-10T19:26:58.287694+0000","last_became_active":"2024-09-10T19:24:24.910173+0000","last_became_peered":"2024-09-10T19:24:24.910173+0000","last_unstale":"2024-09-10T19:26:58.287694+0000","last_undegraded":"2024-09-10T19:26:58.287694+0000","last_fullsized":"2024-09-10T19:26:58.287694+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:24:45.452446+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:24:45.452446+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0063464669999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"303'11","reported_seq":491,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-09-10T19:27:00.930845+0000","last_change":"2024-09-10T19:26:08.469365+0000","last_active":"2024-09-10T19:27:00.930845+0000","last_peered":"2024-09-10T19:27:00.930845+0000","last_clean":"2024-09-10T19:27:00.930845+0000","last_became_active":"2024-09-10T19:24:23.905107+0000","last_became_peered":"2024-09-10T19:24:23.905107+0000","last_unstale":"2024-09-10T19:27:00.930845+0000","last_undegraded":"2024-09-10T19:27:00.930845+0000","last_fullsized":"2024-09-10T19:27:00.930845+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"303'11","last_scrub_stamp":"2024-09-10T19:26:08.469285+0000","last_deep_scrub":"84'9","last_deep_scrub_stamp":"2024-09-10T19:11:53.879272+0000","last_clean_scrub_stamp":"2024-09-10T19:26:08.469285+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:27:26.565539+0000","scrub_duration":0.010068145000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":326,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-09-10T19:26:58.263916+0000","last_change":"2024-09-10T19:25:21.356950+0000","last_active":"2024-09-10T19:26:58.263916+0000","last_peered":"2024-09-10T19:26:58.263916+0000","last_clean":"2024-09-10T19:26:58.263916+0000","last_became_active":"2024-09-10T19:24:23.905977+0000","last_became_peered":"2024-09-10T19:24:23.905977+0000","last_unstale":"2024-09-10T19:26:58.263916+0000","last_undegraded":"2024-09-10T19:26:58.263916+0000","last_fullsized":"2024-09-10T19:26:58.263916+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:25:21.356861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:25:21.356861+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.00363751,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,5],"acting":[0,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.a","version":"355'13376","reported_seq":14572,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-09-10T19:26:59.706778+0000","last_change":"2024-09-10T19:25:21.328117+0000","last_active":"2024-09-10T19:26:59.706778+0000","last_peered":"2024-09-10T19:26:59.706778+0000","last_clean":"2024-09-10T19:26:59.706778+0000","last_became_active":"2024-09-10T19:24:29.876670+0000","last_became_peered":"2024-09-10T19:24:29.876670+0000","last_unstale":"2024-09-10T19:26:59.706778+0000","last_undegraded":"2024-09-10T19:26:59.706778+0000","last_fullsized":"2024-09-10T19:26:59.706778+0000","mapping_epoch":307,"log_start":"341'8600","ondisk_log_start":"341'8600","created":295,"last_epoch_clean":308,"parent":"0.0","parent_split_bits":0,"last_scrub":"331'5451","last_scrub_stamp":"2024-09-10T19:25:21.327858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:24:08.023080+0000","last_clean_scrub_stamp":"2024-09-10T19:25:21.327858+0000","objects_scrubbed":5372,"log_size":4776,"log_dups_size":0,"ondisk_log_size":4776,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"queued for scrub","scrub_duration":2.071574102,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":127270912,"num_objects":1942,"num_object_clones":0,"num_object_copies":5826,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1942,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13376,"num_write_kb":490176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":443,"num_bytes_recovered":29032448,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.1","version":"74'79","reported_seq":147,"reported_epoch":354,"state":"active+clean","last_fresh":"2024-09-10T19:26:56.237463+0000","last_change":"2024-09-10T19:26:19.928630+0000","last_active":"2024-09-10T19:26:56.237463+0000","last_peered":"2024-09-10T19:26:56.237463+0000","last_clean":"2024-09-10T19:26:56.237463+0000","last_became_active":"2024-09-10T19:24:23.907264+0000","last_became_peered":"2024-09-10T19:24:23.907264+0000","last_unstale":"2024-09-10T19:26:56.237463+0000","last_undegraded":"2024-09-10T19:26:56.237463+0000","last_fullsized":"2024-09-10T19:26:56.237463+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":4,"last_scrub":"74'79","last_scrub_stamp":"2024-09-10T19:26:19.927731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:08:00.699315+0000","last_clean_scrub_stamp":"2024-09-10T19:26:19.927731+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:27:34.215011+0000","scrub_duration":0.025790303000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,4],"acting":[5,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"294'44","reported_seq":141,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-09-10T19:27:00.930495+0000","last_change":"2024-09-10T19:26:10.425078+0000","last_active":"2024-09-10T19:27:00.930495+0000","last_peered":"2024-09-10T19:27:00.930495+0000","last_clean":"2024-09-10T19:27:00.930495+0000","last_became_active":"2024-09-10T19:24:01.972831+0000","last_became_peered":"2024-09-10T19:24:01.972831+0000","last_unstale":"2024-09-10T19:27:00.930495+0000","last_undegraded":"2024-09-10T19:27:00.930495+0000","last_fullsized":"2024-09-10T19:27:00.930495+0000","mapping_epoch":292,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":293,"parent":"0.0","parent_split_bits":0,"last_scrub":"294'44","last_scrub_stamp":"2024-09-10T19:26:10.424990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:26:10.424990+0000","objects_scrubbed":10,"log_size":44,"log_dups_size":0,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:27:22.306370+0000","scrub_duration":0.0089341379999999995,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":65,"num_read_kb":50,"num_write":38,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":716,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":379,"reported_epoch":354,"state":"active+clean","last_fresh":"2024-09-10T19:26:56.234968+0000","last_change":"2024-09-10T19:24:42.457503+0000","last_active":"2024-09-10T19:26:56.234968+0000","last_peered":"2024-09-10T19:26:56.234968+0000","last_clean":"2024-09-10T19:26:56.234968+0000","last_became_active":"2024-09-10T19:24:01.978187+0000","last_became_peered":"2024-09-10T19:24:01.978187+0000","last_unstale":"2024-09-10T19:26:56.234968+0000","last_undegraded":"2024-09-10T19:26:56.234968+0000","last_fullsized":"2024-09-10T19:26:56.234968+0000","mapping_epoch":292,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":293,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:24:42.457291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:24:42.457291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0080109410000000006,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":336,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-09-10T19:26:58.289875+0000","last_change":"2024-09-10T19:26:32.570336+0000","last_active":"2024-09-10T19:26:58.289875+0000","last_peered":"2024-09-10T19:26:58.289875+0000","last_clean":"2024-09-10T19:26:58.289875+0000","last_became_active":"2024-09-10T19:24:24.892692+0000","last_became_peered":"2024-09-10T19:24:24.892692+0000","last_unstale":"2024-09-10T19:26:58.289875+0000","last_undegraded":"2024-09-10T19:26:58.289875+0000","last_fullsized":"2024-09-10T19:26:58.289875+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:26:32.570260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:26:32.570260+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:27:53.412595+0000","scrub_duration":0.0043044880000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":349,"reported_epoch":354,"state":"active+clean","last_fresh":"2024-09-10T19:26:56.236487+0000","last_change":"2024-09-10T19:26:09.890646+0000","last_active":"2024-09-10T19:26:56.236487+0000","last_peered":"2024-09-10T19:26:56.236487+0000","last_clean":"2024-09-10T19:26:56.236487+0000","last_became_active":"2024-09-10T19:10:13.721744+0000","last_became_peered":"2024-09-10T19:10:13.721744+0000","last_unstale":"2024-09-10T19:26:56.236487+0000","last_undegraded":"2024-09-10T19:26:56.236487+0000","last_fullsized":"2024-09-10T19:26:56.236487+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:26:09.890570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:26:09.890570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:27:30.451846+0000","scrub_duration":0.012478856,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.9","version":"355'13505","reported_seq":13739,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-09-10T19:27:00.493549+0000","last_change":"2024-09-10T19:26:37.477311+0000","last_active":"2024-09-10T19:27:00.493549+0000","last_peered":"2024-09-10T19:27:00.493549+0000","last_clean":"2024-09-10T19:27:00.493549+0000","last_became_active":"2024-09-10T19:24:09.042606+0000","last_became_peered":"2024-09-10T19:24:09.042606+0000","last_unstale":"2024-09-10T19:27:00.493549+0000","last_undegraded":"2024-09-10T19:27:00.493549+0000","last_fullsized":"2024-09-10T19:27:00.493549+0000","mapping_epoch":295,"log_start":"342'9000","ondisk_log_start":"342'9000","created":295,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"350'11604","last_scrub_stamp":"2024-09-10T19:26:37.477228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:24:08.023080+0000","last_clean_scrub_stamp":"2024-09-10T19:26:37.477228+0000","objects_scrubbed":3893,"log_size":4505,"log_dups_size":0,"ondisk_log_size":4505,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:27:44.794432+0000","scrub_duration":1.5813720249999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":127205376,"num_objects":1941,"num_object_clones":0,"num_object_copies":5823,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1941,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13505,"num_write_kb":494272,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,4],"acting":[5,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.2","version":"74'79","reported_seq":447,"reported_epoch":354,"state":"active+clean","last_fresh":"2024-09-10T19:26:56.239050+0000","last_change":"2024-09-10T19:26:37.555224+0000","last_active":"2024-09-10T19:26:56.239050+0000","last_peered":"2024-09-10T19:26:56.239050+0000","last_clean":"2024-09-10T19:26:56.239050+0000","last_became_active":"2024-09-10T19:24:23.895247+0000","last_became_peered":"2024-09-10T19:24:23.895247+0000","last_unstale":"2024-09-10T19:26:56.239050+0000","last_undegraded":"2024-09-10T19:26:56.239050+0000","last_fullsized":"2024-09-10T19:26:56.239050+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":4,"last_scrub":"74'79","last_scrub_stamp":"2024-09-10T19:26:37.555133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:08:00.699315+0000","last_clean_scrub_stamp":"2024-09-10T19:26:37.555133+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:27:38.585160+0000","scrub_duration":0.0060663699999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.7","version":"237'22","reported_seq":109,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-09-10T19:26:58.287254+0000","last_change":"2024-09-10T19:25:13.591203+0000","last_active":"2024-09-10T19:26:58.287254+0000","last_peered":"2024-09-10T19:26:58.287254+0000","last_clean":"2024-09-10T19:26:58.287254+0000","last_became_active":"2024-09-10T19:24:24.894293+0000","last_became_peered":"2024-09-10T19:24:24.894293+0000","last_unstale":"2024-09-10T19:26:58.287254+0000","last_undegraded":"2024-09-10T19:26:58.287254+0000","last_fullsized":"2024-09-10T19:26:58.287254+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"237'22","last_scrub_stamp":"2024-09-10T19:25:13.591105+0000","last_deep_scrub":"82'18","last_deep_scrub_stamp":"2024-09-10T19:15:55.777361+0000","last_clean_scrub_stamp":"2024-09-10T19:25:13.591105+0000","objects_scrubbed":5,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.002392178,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":36,"num_read_kb":25,"num_write":20,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"159'6","reported_seq":279,"reported_epoch":356,"state":"active+clean","last_fresh":"2024-09-10T19:27:01.253795+0000","last_change":"2024-09-10T19:24:43.417953+0000","last_active":"2024-09-10T19:27:01.253795+0000","last_peered":"2024-09-10T19:27:01.253795+0000","last_clean":"2024-09-10T19:27:01.253795+0000","last_became_active":"2024-09-10T19:24:24.887336+0000","last_became_peered":"2024-09-10T19:24:24.887336+0000","last_unstale":"2024-09-10T19:27:01.253795+0000","last_undegraded":"2024-09-10T19:27:01.253795+0000","last_fullsized":"2024-09-10T19:27:01.253795+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"159'6","last_scrub_stamp":"2024-09-10T19:24:43.417863+0000","last_deep_scrub":"159'6","last_deep_scrub_stamp":"2024-09-10T19:16:32.649914+0000","last_clean_scrub_stamp":"2024-09-10T19:24:43.417863+0000","objects_scrubbed":1,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0045847300000000004,"objects_trimmed":0,"snaptrim_duration":0.00034454499999999997,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":189,"num_read_kb":184,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":778,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":284,"reported_epoch":354,"state":"active+clean","last_fresh":"2024-09-10T19:26:56.235598+0000","last_change":"2024-09-10T19:26:11.877736+0000","last_active":"2024-09-10T19:26:56.235598+0000","last_peered":"2024-09-10T19:26:56.235598+0000","last_clean":"2024-09-10T19:26:56.235598+0000","last_became_active":"2024-09-10T19:24:24.924859+0000","last_became_peered":"2024-09-10T19:24:24.924859+0000","last_unstale":"2024-09-10T19:26:56.235598+0000","last_undegraded":"2024-09-10T19:26:56.235598+0000","last_fullsized":"2024-09-10T19:26:56.235598+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:26:11.877648+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:26:11.877648+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:27:30.098099+0000","scrub_duration":0.0035177559999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":253,"reported_epoch":354,"state":"active+clean","last_fresh":"2024-09-10T19:26:56.235834+0000","last_change":"2024-09-10T19:26:21.946019+0000","last_active":"2024-09-10T19:26:56.235834+0000","last_peered":"2024-09-10T19:26:56.235834+0000","last_clean":"2024-09-10T19:26:56.235834+0000","last_became_active":"2024-09-10T19:16:24.861481+0000","last_became_peered":"2024-09-10T19:16:24.861481+0000","last_unstale":"2024-09-10T19:26:56.235834+0000","last_undegraded":"2024-09-10T19:26:56.235834+0000","last_fullsized":"2024-09-10T19:26:56.235834+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:26:21.945941+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:23:28.736502+0000","last_clean_scrub_stamp":"2024-09-10T19:26:21.945941+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:27:22.081102+0000","scrub_duration":0.019402639999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,4],"acting":[5,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":362,"reported_epoch":354,"state":"active+clean","last_fresh":"2024-09-10T19:26:56.237688+0000","last_change":"2024-09-10T19:25:21.790876+0000","last_active":"2024-09-10T19:26:56.237688+0000","last_peered":"2024-09-10T19:26:56.237688+0000","last_clean":"2024-09-10T19:26:56.237688+0000","last_became_active":"2024-09-10T19:10:14.053697+0000","last_became_peered":"2024-09-10T19:10:14.053697+0000","last_unstale":"2024-09-10T19:26:56.237688+0000","last_undegraded":"2024-09-10T19:26:56.237688+0000","last_fullsized":"2024-09-10T19:26:56.237688+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:25:21.790674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:22:01.208272+0000","last_clean_scrub_stamp":"2024-09-10T19:25:21.790674+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0098199150000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.8","version":"356'13601","reported_seq":13890,"reported_epoch":356,"state":"active+clean","last_fresh":"2024-09-10T19:27:01.934363+0000","last_change":"2024-09-10T19:26:25.382432+0000","last_active":"2024-09-10T19:27:01.934363+0000","last_peered":"2024-09-10T19:27:01.934363+0000","last_clean":"2024-09-10T19:27:01.934363+0000","last_became_active":"2024-09-10T19:24:29.877601+0000","last_became_peered":"2024-09-10T19:24:29.877601+0000","last_unstale":"2024-09-10T19:27:01.934363+0000","last_undegraded":"2024-09-10T19:27:01.934363+0000","last_fullsized":"2024-09-10T19:27:01.934363+0000","mapping_epoch":307,"log_start":"345'9700","ondisk_log_start":"345'9700","created":295,"last_epoch_clean":308,"parent":"0.0","parent_split_bits":0,"last_scrub":"349'10531","last_scrub_stamp":"2024-09-10T19:26:25.382365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:24:08.023080+0000","last_clean_scrub_stamp":"2024-09-10T19:26:25.382365+0000","objects_scrubbed":4903,"log_size":3901,"log_dups_size":0,"ondisk_log_size":3901,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:27:26.462413+0000","scrub_duration":1.887493402,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":116850688,"num_objects":1783,"num_object_clones":0,"num_object_copies":5349,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1783,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13601,"num_write_kb":492288,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":455,"num_bytes_recovered":29818880,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,5],"acting":[0,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"74'79","reported_seq":83,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-09-10T19:26:58.287055+0000","last_change":"2024-09-10T19:25:17.733733+0000","last_active":"2024-09-10T19:26:58.287055+0000","last_peered":"2024-09-10T19:26:58.287055+0000","last_clean":"2024-09-10T19:26:58.287055+0000","last_became_active":"2024-09-10T19:24:01.977477+0000","last_became_peered":"2024-09-10T19:24:01.977477+0000","last_unstale":"2024-09-10T19:26:58.287055+0000","last_undegraded":"2024-09-10T19:26:58.287055+0000","last_fullsized":"2024-09-10T19:26:58.287055+0000","mapping_epoch":292,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":293,"parent":"0.0","parent_split_bits":3,"last_scrub":"74'79","last_scrub_stamp":"2024-09-10T19:25:17.733642+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:08:00.699315+0000","last_clean_scrub_stamp":"2024-09-10T19:25:17.733642+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0056412739999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"237'30","reported_seq":416,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-09-10T19:26:58.290242+0000","last_change":"2024-09-10T19:26:09.481450+0000","last_active":"2024-09-10T19:26:58.290242+0000","last_peered":"2024-09-10T19:26:58.290242+0000","last_clean":"2024-09-10T19:26:58.290242+0000","last_became_active":"2024-09-10T19:24:24.899711+0000","last_became_peered":"2024-09-10T19:24:24.899711+0000","last_unstale":"2024-09-10T19:26:58.290242+0000","last_undegraded":"2024-09-10T19:26:58.290242+0000","last_fullsized":"2024-09-10T19:26:58.290242+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"237'30","last_scrub_stamp":"2024-09-10T19:26:09.481366+0000","last_deep_scrub":"221'18","last_deep_scrub_stamp":"2024-09-10T19:19:29.606748+0000","last_clean_scrub_stamp":"2024-09-10T19:26:09.481366+0000","objects_scrubbed":6,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:27:32.712334+0000","scrub_duration":0.0023429890000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"61'1","reported_seq":319,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-09-10T19:26:58.263963+0000","last_change":"2024-09-10T19:25:13.486963+0000","last_active":"2024-09-10T19:26:58.263963+0000","last_peered":"2024-09-10T19:26:58.263963+0000","last_clean":"2024-09-10T19:26:58.263963+0000","last_became_active":"2024-09-10T19:24:23.897412+0000","last_became_peered":"2024-09-10T19:24:23.897412+0000","last_unstale":"2024-09-10T19:26:58.263963+0000","last_undegraded":"2024-09-10T19:26:58.263963+0000","last_fullsized":"2024-09-10T19:26:58.263963+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'1","last_scrub_stamp":"2024-09-10T19:25:13.486865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:25:13.486865+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0054878949999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":403,"reported_epoch":354,"state":"active+clean","last_fresh":"2024-09-10T19:26:56.240643+0000","last_change":"2024-09-10T19:25:28.762482+0000","last_active":"2024-09-10T19:26:56.240643+0000","last_peered":"2024-09-10T19:26:56.240643+0000","last_clean":"2024-09-10T19:26:56.240643+0000","last_became_active":"2024-09-10T19:24:24.896201+0000","last_became_peered":"2024-09-10T19:24:24.896201+0000","last_unstale":"2024-09-10T19:26:56.240643+0000","last_undegraded":"2024-09-10T19:26:56.240643+0000","last_fullsized":"2024-09-10T19:26:56.240643+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:25:28.762237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:25:28.762237+0000","last_clean_scrub_stamp":"2024-09-10T19:25:28.762237+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0099337430000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":399,"reported_epoch":354,"state":"active+clean","last_fresh":"2024-09-10T19:26:56.234951+0000","last_change":"2024-09-10T19:24:49.491925+0000","last_active":"2024-09-10T19:26:56.234951+0000","last_peered":"2024-09-10T19:26:56.234951+0000","last_clean":"2024-09-10T19:26:56.234951+0000","last_became_active":"2024-09-10T19:24:24.914680+0000","last_became_peered":"2024-09-10T19:24:24.914680+0000","last_unstale":"2024-09-10T19:26:56.234951+0000","last_undegraded":"2024-09-10T19:26:56.234951+0000","last_fullsized":"2024-09-10T19:26:56.234951+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:24:49.491839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:24:49.491839+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0022950959999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":83,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-09-10T19:26:58.262794+0000","last_change":"2024-09-10T19:25:22.846256+0000","last_active":"2024-09-10T19:26:58.262794+0000","last_peered":"2024-09-10T19:26:58.262794+0000","last_clean":"2024-09-10T19:26:58.262794+0000","last_became_active":"2024-09-10T19:24:01.974039+0000","last_became_peered":"2024-09-10T19:24:01.974039+0000","last_unstale":"2024-09-10T19:26:58.262794+0000","last_undegraded":"2024-09-10T19:26:58.262794+0000","last_fullsized":"2024-09-10T19:26:58.262794+0000","mapping_epoch":292,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":293,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:25:22.846176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:25:22.846176+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001764222,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.b","version":"355'13622","reported_seq":14887,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-09-10T19:26:59.701103+0000","last_change":"2024-09-10T19:26:21.279920+0000","last_active":"2024-09-10T19:26:59.701103+0000","last_peered":"2024-09-10T19:26:59.701103+0000","last_clean":"2024-09-10T19:26:59.701103+0000","last_became_active":"2024-09-10T19:24:28.940329+0000","last_became_peered":"2024-09-10T19:24:28.940329+0000","last_unstale":"2024-09-10T19:26:59.701103+0000","last_undegraded":"2024-09-10T19:26:59.701103+0000","last_fullsized":"2024-09-10T19:26:59.701103+0000","mapping_epoch":306,"log_start":"341'8900","ondisk_log_start":"341'8900","created":295,"last_epoch_clean":307,"parent":"0.0","parent_split_bits":0,"last_scrub":"347'10423","last_scrub_stamp":"2024-09-10T19:26:21.279841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:24:08.023080+0000","last_clean_scrub_stamp":"2024-09-10T19:26:21.279841+0000","objects_scrubbed":5302,"log_size":4722,"log_dups_size":0,"ondisk_log_size":4722,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:27:21.438796+0000","scrub_duration":1.8983051479999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":133431296,"num_objects":2036,"num_object_clones":0,"num_object_copies":6108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2036,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13622,"num_write_kb":501056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":478,"num_bytes_recovered":31326208,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"237'33","reported_seq":430,"reported_epoch":354,"state":"active+clean","last_fresh":"2024-09-10T19:26:56.246213+0000","last_change":"2024-09-10T19:24:44.886858+0000","last_active":"2024-09-10T19:26:56.246213+0000","last_peered":"2024-09-10T19:26:56.246213+0000","last_clean":"2024-09-10T19:26:56.246213+0000","last_became_active":"2024-09-10T19:10:09.822184+0000","last_became_peered":"2024-09-10T19:10:09.822184+0000","last_unstale":"2024-09-10T19:26:56.246213+0000","last_undegraded":"2024-09-10T19:26:56.246213+0000","last_fullsized":"2024-09-10T19:26:56.246213+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"237'33","last_scrub_stamp":"2024-09-10T19:24:44.884921+0000","last_deep_scrub":"80'21","last_deep_scrub_stamp":"2024-09-10T19:11:39.836144+0000","last_clean_scrub_stamp":"2024-09-10T19:24:44.884921+0000","objects_scrubbed":7,"log_size":33,"log_dups_size":0,"ondisk_log_size":33,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.015025566000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":33,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"63'2","reported_seq":372,"reported_epoch":354,"state":"active+clean","last_fresh":"2024-09-10T19:26:56.235918+0000","last_change":"2024-09-10T19:25:23.821301+0000","last_active":"2024-09-10T19:26:56.235918+0000","last_peered":"2024-09-10T19:26:56.235918+0000","last_clean":"2024-09-10T19:26:56.235918+0000","last_became_active":"2024-09-10T19:10:07.684784+0000","last_became_peered":"2024-09-10T19:10:07.684784+0000","last_unstale":"2024-09-10T19:26:56.235918+0000","last_undegraded":"2024-09-10T19:26:56.235918+0000","last_fullsized":"2024-09-10T19:26:56.235918+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'2","last_scrub_stamp":"2024-09-10T19:25:23.821213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:25:23.821213+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0063659800000000002,"objects_trimmed":0,"snaptrim_duration":0.00075522999999999996,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":375,"reported_epoch":354,"state":"active+clean","last_fresh":"2024-09-10T19:26:56.247898+0000","last_change":"2024-09-10T19:26:08.934930+0000","last_active":"2024-09-10T19:26:56.247898+0000","last_peered":"2024-09-10T19:26:56.247898+0000","last_clean":"2024-09-10T19:26:56.247898+0000","last_became_active":"2024-09-10T19:16:24.858662+0000","last_became_peered":"2024-09-10T19:16:24.858662+0000","last_unstale":"2024-09-10T19:26:56.247898+0000","last_undegraded":"2024-09-10T19:26:56.247898+0000","last_fullsized":"2024-09-10T19:26:56.247898+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:26:08.934746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:26:08.934746+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:27:38.162812+0000","scrub_duration":0.0052906790000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"74'79","reported_seq":428,"reported_epoch":354,"state":"active+clean","last_fresh":"2024-09-10T19:26:56.240798+0000","last_change":"2024-09-10T19:24:40.776904+0000","last_active":"2024-09-10T19:26:56.240798+0000","last_peered":"2024-09-10T19:26:56.240798+0000","last_clean":"2024-09-10T19:26:56.240798+0000","last_became_active":"2024-09-10T19:24:23.904267+0000","last_became_peered":"2024-09-10T19:24:23.904267+0000","last_unstale":"2024-09-10T19:26:56.240798+0000","last_undegraded":"2024-09-10T19:26:56.240798+0000","last_fullsized":"2024-09-10T19:26:56.240798+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'79","last_scrub_stamp":"2024-09-10T19:24:40.776821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:08:00.699315+0000","last_clean_scrub_stamp":"2024-09-10T19:24:40.776821+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0056971499999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":388,"reported_epoch":354,"state":"active+clean","last_fresh":"2024-09-10T19:26:56.238956+0000","last_change":"2024-09-10T19:26:31.509806+0000","last_active":"2024-09-10T19:26:56.238956+0000","last_peered":"2024-09-10T19:26:56.238956+0000","last_clean":"2024-09-10T19:26:56.238956+0000","last_became_active":"2024-09-10T19:24:23.903868+0000","last_became_peered":"2024-09-10T19:24:23.903868+0000","last_unstale":"2024-09-10T19:26:56.238956+0000","last_undegraded":"2024-09-10T19:26:56.238956+0000","last_fullsized":"2024-09-10T19:26:56.238956+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:26:31.509498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:26:31.509498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:27:59.336304+0000","scrub_duration":0.029737573999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,0],"acting":[7,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":359,"reported_epoch":354,"state":"active+clean","last_fresh":"2024-09-10T19:26:56.235063+0000","last_change":"2024-09-10T19:26:33.822326+0000","last_active":"2024-09-10T19:26:56.235063+0000","last_peered":"2024-09-10T19:26:56.235063+0000","last_clean":"2024-09-10T19:26:56.235063+0000","last_became_active":"2024-09-10T19:10:13.720647+0000","last_became_peered":"2024-09-10T19:10:13.720647+0000","last_unstale":"2024-09-10T19:26:56.235063+0000","last_undegraded":"2024-09-10T19:26:56.235063+0000","last_fullsized":"2024-09-10T19:26:56.235063+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:26:33.820918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:26:33.820918+0000","last_clean_scrub_stamp":"2024-09-10T19:26:33.820918+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:27:50.206519+0000","scrub_duration":0.0080461309999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.4","version":"356'13686","reported_seq":14259,"reported_epoch":356,"state":"active+clean","last_fresh":"2024-09-10T19:27:01.958715+0000","last_change":"2024-09-10T19:26:26.177340+0000","last_active":"2024-09-10T19:27:01.958715+0000","last_peered":"2024-09-10T19:27:01.958715+0000","last_clean":"2024-09-10T19:27:01.958715+0000","last_became_active":"2024-09-10T19:24:30.884803+0000","last_became_peered":"2024-09-10T19:24:30.884803+0000","last_unstale":"2024-09-10T19:27:01.958715+0000","last_undegraded":"2024-09-10T19:27:01.958715+0000","last_fullsized":"2024-09-10T19:27:01.958715+0000","mapping_epoch":308,"log_start":"344'9700","ondisk_log_start":"344'9700","created":295,"last_epoch_clean":309,"parent":"0.0","parent_split_bits":0,"last_scrub":"349'10642","last_scrub_stamp":"2024-09-10T19:26:26.177254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:24:08.023080+0000","last_clean_scrub_stamp":"2024-09-10T19:26:26.177254+0000","objects_scrubbed":4897,"log_size":3986,"log_dups_size":0,"ondisk_log_size":3986,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:27:27.930007+0000","scrub_duration":1.705149934,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":116654080,"num_objects":1780,"num_object_clones":0,"num_object_copies":5340,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1780,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13686,"num_write_kb":494912,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":441,"num_bytes_recovered":28901376,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"237'35","reported_seq":501,"reported_epoch":354,"state":"active+clean","last_fresh":"2024-09-10T19:26:56.247203+0000","last_change":"2024-09-10T19:26:33.411895+0000","last_active":"2024-09-10T19:26:56.247203+0000","last_peered":"2024-09-10T19:26:56.247203+0000","last_clean":"2024-09-10T19:26:56.247203+0000","last_became_active":"2024-09-10T19:24:01.973201+0000","last_became_peered":"2024-09-10T19:24:01.973201+0000","last_unstale":"2024-09-10T19:26:56.247203+0000","last_undegraded":"2024-09-10T19:26:56.247203+0000","last_fullsized":"2024-09-10T19:26:56.247203+0000","mapping_epoch":292,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":293,"parent":"0.0","parent_split_bits":0,"last_scrub":"237'35","last_scrub_stamp":"2024-09-10T19:26:33.411803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:26:33.411803+0000","objects_scrubbed":9,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:27:55.317233+0000","scrub_duration":0.0050992160000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":56,"num_read_kb":45,"num_write":36,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":264,"reported_epoch":354,"state":"active+clean","last_fresh":"2024-09-10T19:26:56.234798+0000","last_change":"2024-09-10T19:24:24.913980+0000","last_active":"2024-09-10T19:26:56.234798+0000","last_peered":"2024-09-10T19:26:56.234798+0000","last_clean":"2024-09-10T19:26:56.234798+0000","last_became_active":"2024-09-10T19:24:24.913709+0000","last_became_peered":"2024-09-10T19:24:24.913709+0000","last_unstale":"2024-09-10T19:26:56.234798+0000","last_undegraded":"2024-09-10T19:26:56.234798+0000","last_fullsized":"2024-09-10T19:26:56.234798+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:24:05.723155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:24:05.723155+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0015060060000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,1],"acting":[5,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":196,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-09-10T19:26:58.297411+0000","last_change":"2024-09-10T19:26:30.557805+0000","last_active":"2024-09-10T19:26:58.297411+0000","last_peered":"2024-09-10T19:26:58.297411+0000","last_clean":"2024-09-10T19:26:58.297411+0000","last_became_active":"2024-09-10T19:24:23.901415+0000","last_became_peered":"2024-09-10T19:24:23.901415+0000","last_unstale":"2024-09-10T19:26:58.297411+0000","last_undegraded":"2024-09-10T19:26:58.297411+0000","last_fullsized":"2024-09-10T19:26:58.297411+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:26:30.557724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:26:30.557724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:27:49.638525+0000","scrub_duration":0.015588159000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":379,"reported_epoch":354,"state":"active+clean","last_fresh":"2024-09-10T19:26:56.240715+0000","last_change":"2024-09-10T19:25:30.845977+0000","last_active":"2024-09-10T19:26:56.240715+0000","last_peered":"2024-09-10T19:26:56.240715+0000","last_clean":"2024-09-10T19:26:56.240715+0000","last_became_active":"2024-09-10T19:16:24.858172+0000","last_became_peered":"2024-09-10T19:16:24.858172+0000","last_unstale":"2024-09-10T19:26:56.240715+0000","last_undegraded":"2024-09-10T19:26:56.240715+0000","last_fullsized":"2024-09-10T19:26:56.240715+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:25:30.845546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:21:03.910074+0000","last_clean_scrub_stamp":"2024-09-10T19:25:30.845546+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0066935149999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,6],"acting":[7,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.5","version":"356'13661","reported_seq":14065,"reported_epoch":356,"state":"active+clean","last_fresh":"2024-09-10T19:27:01.937819+0000","last_change":"2024-09-10T19:26:17.719959+0000","last_active":"2024-09-10T19:27:01.937819+0000","last_peered":"2024-09-10T19:27:01.937819+0000","last_clean":"2024-09-10T19:27:01.937819+0000","last_became_active":"2024-09-10T19:24:29.877183+0000","last_became_peered":"2024-09-10T19:24:29.877183+0000","last_unstale":"2024-09-10T19:27:01.937819+0000","last_undegraded":"2024-09-10T19:27:01.937819+0000","last_fullsized":"2024-09-10T19:27:01.937819+0000","mapping_epoch":307,"log_start":"345'9700","ondisk_log_start":"345'9700","created":295,"last_epoch_clean":308,"parent":"0.0","parent_split_bits":0,"last_scrub":"345'10001","last_scrub_stamp":"2024-09-10T19:26:17.719855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:24:08.023080+0000","last_clean_scrub_stamp":"2024-09-10T19:26:17.719855+0000","objects_scrubbed":5476,"log_size":3961,"log_dups_size":0,"ondisk_log_size":3961,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:27:21.631854+0000","scrub_duration":1.304291364,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":115539968,"num_objects":1763,"num_object_clones":0,"num_object_copies":5289,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1763,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13661,"num_write_kb":493568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":446,"num_bytes_recovered":29229056,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.b","version":"237'15","reported_seq":400,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-09-10T19:26:58.297840+0000","last_change":"2024-09-10T19:26:14.505086+0000","last_active":"2024-09-10T19:26:58.297840+0000","last_peered":"2024-09-10T19:26:58.297840+0000","last_clean":"2024-09-10T19:26:58.297840+0000","last_became_active":"2024-09-10T19:24:44.892294+0000","last_became_peered":"2024-09-10T19:24:44.892294+0000","last_unstale":"2024-09-10T19:26:58.297840+0000","last_undegraded":"2024-09-10T19:26:58.297840+0000","last_fullsized":"2024-09-10T19:26:58.297840+0000","mapping_epoch":314,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":315,"parent":"0.0","parent_split_bits":0,"last_scrub":"237'15","last_scrub_stamp":"2024-09-10T19:26:14.504998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:26:14.504998+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:27:24.336321+0000","scrub_duration":0.016209647000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,4],"acting":[0,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":83,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-09-10T19:26:58.287436+0000","last_change":"2024-09-10T19:26:30.411444+0000","last_active":"2024-09-10T19:26:58.287436+0000","last_peered":"2024-09-10T19:26:58.287436+0000","last_clean":"2024-09-10T19:26:58.287436+0000","last_became_active":"2024-09-10T19:24:01.980348+0000","last_became_peered":"2024-09-10T19:24:01.980348+0000","last_unstale":"2024-09-10T19:26:58.287436+0000","last_undegraded":"2024-09-10T19:26:58.287436+0000","last_fullsized":"2024-09-10T19:26:58.287436+0000","mapping_epoch":292,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":293,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:26:30.411368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:11:44.254513+0000","last_clean_scrub_stamp":"2024-09-10T19:26:30.411368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:27:44.702203+0000","scrub_duration":0.0027079880000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":263,"reported_epoch":354,"state":"active+clean","last_fresh":"2024-09-10T19:26:56.245543+0000","last_change":"2024-09-10T19:24:54.563313+0000","last_active":"2024-09-10T19:26:56.245543+0000","last_peered":"2024-09-10T19:26:56.245543+0000","last_clean":"2024-09-10T19:26:56.245543+0000","last_became_active":"2024-09-10T19:24:23.891161+0000","last_became_peered":"2024-09-10T19:24:23.891161+0000","last_unstale":"2024-09-10T19:26:56.245543+0000","last_undegraded":"2024-09-10T19:26:56.245543+0000","last_fullsized":"2024-09-10T19:26:56.245543+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:24:54.563235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:24:54.563235+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0029717020000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":314,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-09-10T19:26:58.264047+0000","last_change":"2024-09-10T19:26:31.581355+0000","last_active":"2024-09-10T19:26:58.264047+0000","last_peered":"2024-09-10T19:26:58.264047+0000","last_clean":"2024-09-10T19:26:58.264047+0000","last_became_active":"2024-09-10T19:24:23.905112+0000","last_became_peered":"2024-09-10T19:24:23.905112+0000","last_unstale":"2024-09-10T19:26:58.264047+0000","last_undegraded":"2024-09-10T19:26:58.264047+0000","last_fullsized":"2024-09-10T19:26:58.264047+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:26:31.581300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:26:31.581300+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:27:51.658432+0000","scrub_duration":0.0033456739999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.6","version":"355'13571","reported_seq":14634,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-09-10T19:27:00.434029+0000","last_change":"2024-09-10T19:24:30.889054+0000","last_active":"2024-09-10T19:27:00.434029+0000","last_peered":"2024-09-10T19:27:00.434029+0000","last_clean":"2024-09-10T19:27:00.434029+0000","last_became_active":"2024-09-10T19:24:30.883926+0000","last_became_peered":"2024-09-10T19:24:30.883926+0000","last_unstale":"2024-09-10T19:27:00.434029+0000","last_undegraded":"2024-09-10T19:27:00.434029+0000","last_fullsized":"2024-09-10T19:27:00.434029+0000","mapping_epoch":308,"log_start":"342'9100","ondisk_log_start":"342'9100","created":295,"last_epoch_clean":309,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:24:08.023080+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:24:08.023080+0000","last_clean_scrub_stamp":"2024-09-10T19:24:08.023080+0000","objects_scrubbed":0,"log_size":4471,"log_dups_size":0,"ondisk_log_size":4471,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":127991808,"num_objects":1953,"num_object_clones":0,"num_object_copies":5859,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1953,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13571,"num_write_kb":496768,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":488,"num_bytes_recovered":31981568,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,4],"acting":[5,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.8","version":"237'34","reported_seq":403,"reported_epoch":354,"state":"active+clean","last_fresh":"2024-09-10T19:26:56.238958+0000","last_change":"2024-09-10T19:25:27.750310+0000","last_active":"2024-09-10T19:26:56.238958+0000","last_peered":"2024-09-10T19:26:56.238958+0000","last_clean":"2024-09-10T19:26:56.238958+0000","last_became_active":"2024-09-10T19:10:09.820950+0000","last_became_peered":"2024-09-10T19:10:09.820950+0000","last_unstale":"2024-09-10T19:26:56.238958+0000","last_undegraded":"2024-09-10T19:26:56.238958+0000","last_fullsized":"2024-09-10T19:26:56.238958+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"237'34","last_scrub_stamp":"2024-09-10T19:25:27.750229+0000","last_deep_scrub":"220'22","last_deep_scrub_stamp":"2024-09-10T19:19:24.597866+0000","last_clean_scrub_stamp":"2024-09-10T19:25:27.750229+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.013590471,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":243,"reported_epoch":354,"state":"active+clean","last_fresh":"2024-09-10T19:26:56.236767+0000","last_change":"2024-09-10T19:25:24.833996+0000","last_active":"2024-09-10T19:26:56.236767+0000","last_peered":"2024-09-10T19:26:56.236767+0000","last_clean":"2024-09-10T19:26:56.236767+0000","last_became_active":"2024-09-10T19:16:24.857253+0000","last_became_peered":"2024-09-10T19:16:24.857253+0000","last_unstale":"2024-09-10T19:26:56.236767+0000","last_undegraded":"2024-09-10T19:26:56.236767+0000","last_fullsized":"2024-09-10T19:26:56.236767+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:25:24.833939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:25:24.833939+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0086811059999999992,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.9","version":"80'9","reported_seq":582,"reported_epoch":354,"state":"active+clean","last_fresh":"2024-09-10T19:26:56.239981+0000","last_change":"2024-09-10T19:25:29.833358+0000","last_active":"2024-09-10T19:26:56.239981+0000","last_peered":"2024-09-10T19:26:56.239981+0000","last_clean":"2024-09-10T19:26:56.239981+0000","last_became_active":"2024-09-10T19:10:11.702878+0000","last_became_peered":"2024-09-10T19:10:11.702878+0000","last_unstale":"2024-09-10T19:26:56.239981+0000","last_undegraded":"2024-09-10T19:26:56.239981+0000","last_fullsized":"2024-09-10T19:26:56.239981+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"80'9","last_scrub_stamp":"2024-09-10T19:25:29.833271+0000","last_deep_scrub":"80'9","last_deep_scrub_stamp":"2024-09-10T19:25:29.833271+0000","last_clean_scrub_stamp":"2024-09-10T19:25:29.833271+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.027756563000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":356,"reported_epoch":354,"state":"active+clean","last_fresh":"2024-09-10T19:26:56.235460+0000","last_change":"2024-09-10T19:24:38.483717+0000","last_active":"2024-09-10T19:26:56.235460+0000","last_peered":"2024-09-10T19:26:56.235460+0000","last_clean":"2024-09-10T19:26:56.235460+0000","last_became_active":"2024-09-10T19:24:23.905606+0000","last_became_peered":"2024-09-10T19:24:23.905606+0000","last_unstale":"2024-09-10T19:26:56.235460+0000","last_undegraded":"2024-09-10T19:26:56.235460+0000","last_fullsized":"2024-09-10T19:26:56.235460+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:24:38.483596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:24:38.483596+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0049697939999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.7","version":"355'13260","reported_seq":13569,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-09-10T19:26:59.692785+0000","last_change":"2024-09-10T19:25:27.604914+0000","last_active":"2024-09-10T19:26:59.692785+0000","last_peered":"2024-09-10T19:26:59.692785+0000","last_clean":"2024-09-10T19:26:59.692785+0000","last_became_active":"2024-09-10T19:24:09.045557+0000","last_became_peered":"2024-09-10T19:24:09.045557+0000","last_unstale":"2024-09-10T19:26:59.692785+0000","last_undegraded":"2024-09-10T19:26:59.692785+0000","last_fullsized":"2024-09-10T19:26:59.692785+0000","mapping_epoch":295,"log_start":"341'8500","ondisk_log_start":"341'8500","created":295,"last_epoch_clean":296,"parent":"0.0","parent_split_bits":0,"last_scrub":"333'5834","last_scrub_stamp":"2024-09-10T19:25:27.604825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:24:08.023080+0000","last_clean_scrub_stamp":"2024-09-10T19:25:27.604825+0000","objects_scrubbed":5783,"log_size":4760,"log_dups_size":0,"ondisk_log_size":4760,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"queued for scrub","scrub_duration":2.3305890900000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":126615552,"num_objects":1932,"num_object_clones":0,"num_object_copies":5796,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1932,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13260,"num_write_kb":486144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.9","version":"237'18","reported_seq":448,"reported_epoch":354,"state":"active+clean","last_fresh":"2024-09-10T19:26:56.246201+0000","last_change":"2024-09-10T19:26:11.809995+0000","last_active":"2024-09-10T19:26:56.246201+0000","last_peered":"2024-09-10T19:26:56.246201+0000","last_clean":"2024-09-10T19:26:56.246201+0000","last_became_active":"2024-09-10T19:24:01.974878+0000","last_became_peered":"2024-09-10T19:24:01.974878+0000","last_unstale":"2024-09-10T19:26:56.246201+0000","last_undegraded":"2024-09-10T19:26:56.246201+0000","last_fullsized":"2024-09-10T19:26:56.246201+0000","mapping_epoch":292,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":293,"parent":"0.0","parent_split_bits":0,"last_scrub":"237'18","last_scrub_stamp":"2024-09-10T19:26:11.809915+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:26:11.809915+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:27:39.096722+0000","scrub_duration":0.014721300999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.f","version":"59'2","reported_seq":387,"reported_epoch":354,"state":"active+clean","last_fresh":"2024-09-10T19:26:56.247412+0000","last_change":"2024-09-10T19:24:53.593207+0000","last_active":"2024-09-10T19:26:56.247412+0000","last_peered":"2024-09-10T19:26:56.247412+0000","last_clean":"2024-09-10T19:26:56.247412+0000","last_became_active":"2024-09-10T19:24:23.893796+0000","last_became_peered":"2024-09-10T19:24:23.893796+0000","last_unstale":"2024-09-10T19:26:56.247412+0000","last_undegraded":"2024-09-10T19:26:56.247412+0000","last_fullsized":"2024-09-10T19:26:56.247412+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'2","last_scrub_stamp":"2024-09-10T19:24:53.593127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:24:53.593127+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0062292040000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":14,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":298,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-09-10T19:26:58.263835+0000","last_change":"2024-09-10T19:26:11.459855+0000","last_active":"2024-09-10T19:26:58.263835+0000","last_peered":"2024-09-10T19:26:58.263835+0000","last_clean":"2024-09-10T19:26:58.263835+0000","last_became_active":"2024-09-10T19:24:24.914252+0000","last_became_peered":"2024-09-10T19:24:24.914252+0000","last_unstale":"2024-09-10T19:26:58.263835+0000","last_undegraded":"2024-09-10T19:26:58.263835+0000","last_fullsized":"2024-09-10T19:26:58.263835+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:26:11.459804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:11:46.587846+0000","last_clean_scrub_stamp":"2024-09-10T19:26:11.459804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:27:13.584075+0000","scrub_duration":0.0072695340000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,5],"acting":[0,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":376,"reported_epoch":354,"state":"active+clean","last_fresh":"2024-09-10T19:26:56.236133+0000","last_change":"2024-09-10T19:25:21.864732+0000","last_active":"2024-09-10T19:26:56.236133+0000","last_peered":"2024-09-10T19:26:56.236133+0000","last_clean":"2024-09-10T19:26:56.236133+0000","last_became_active":"2024-09-10T19:24:01.971131+0000","last_became_peered":"2024-09-10T19:24:01.971131+0000","last_unstale":"2024-09-10T19:26:56.236133+0000","last_undegraded":"2024-09-10T19:26:56.236133+0000","last_fullsized":"2024-09-10T19:26:56.236133+0000","mapping_epoch":292,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":293,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:25:21.864515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:25:21.864515+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0042058019999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.16","version":"237'15","reported_seq":354,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-09-10T19:26:58.290049+0000","last_change":"2024-09-10T19:25:28.386031+0000","last_active":"2024-09-10T19:26:58.290049+0000","last_peered":"2024-09-10T19:26:58.290049+0000","last_clean":"2024-09-10T19:26:58.290049+0000","last_became_active":"2024-09-10T19:24:23.901610+0000","last_became_peered":"2024-09-10T19:24:23.901610+0000","last_unstale":"2024-09-10T19:26:58.290049+0000","last_undegraded":"2024-09-10T19:26:58.290049+0000","last_fullsized":"2024-09-10T19:26:58.290049+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"237'15","last_scrub_stamp":"2024-09-10T19:25:28.385945+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:25:28.385945+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0031317430000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":97,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-09-10T19:26:58.286779+0000","last_change":"2024-09-10T19:24:49.534368+0000","last_active":"2024-09-10T19:26:58.286779+0000","last_peered":"2024-09-10T19:26:58.286779+0000","last_clean":"2024-09-10T19:26:58.286779+0000","last_became_active":"2024-09-10T19:24:24.894815+0000","last_became_peered":"2024-09-10T19:24:24.894815+0000","last_unstale":"2024-09-10T19:26:58.286779+0000","last_undegraded":"2024-09-10T19:26:58.286779+0000","last_fullsized":"2024-09-10T19:26:58.286779+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:24:49.534307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:11:41.458523+0000","last_clean_scrub_stamp":"2024-09-10T19:24:49.534307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0045336179999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,4],"acting":[1,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":385,"reported_epoch":354,"state":"active+clean","last_fresh":"2024-09-10T19:26:56.237285+0000","last_change":"2024-09-10T19:26:24.891697+0000","last_active":"2024-09-10T19:26:56.237285+0000","last_peered":"2024-09-10T19:26:56.237285+0000","last_clean":"2024-09-10T19:26:56.237285+0000","last_became_active":"2024-09-10T19:24:01.970325+0000","last_became_peered":"2024-09-10T19:24:01.970325+0000","last_unstale":"2024-09-10T19:26:56.237285+0000","last_undegraded":"2024-09-10T19:26:56.237285+0000","last_fullsized":"2024-09-10T19:26:56.237285+0000","mapping_epoch":292,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":293,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:26:24.891605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:24:09.367654+0000","last_clean_scrub_stamp":"2024-09-10T19:26:24.891605+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:27:54.880956+0000","scrub_duration":0.0032178340000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"66'1","reported_seq":263,"reported_epoch":354,"state":"active+clean","last_fresh":"2024-09-10T19:26:56.246400+0000","last_change":"2024-09-10T19:24:43.668696+0000","last_active":"2024-09-10T19:26:56.246400+0000","last_peered":"2024-09-10T19:26:56.246400+0000","last_clean":"2024-09-10T19:26:56.246400+0000","last_became_active":"2024-09-10T19:24:23.897858+0000","last_became_peered":"2024-09-10T19:24:23.897858+0000","last_unstale":"2024-09-10T19:26:56.246400+0000","last_undegraded":"2024-09-10T19:26:56.246400+0000","last_fullsized":"2024-09-10T19:26:56.246400+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'1","last_scrub_stamp":"2024-09-10T19:24:43.668620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:24:43.668620+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0040764670000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":13,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":398,"reported_epoch":354,"state":"active+clean","last_fresh":"2024-09-10T19:26:56.244708+0000","last_change":"2024-09-10T19:25:14.241349+0000","last_active":"2024-09-10T19:26:56.244708+0000","last_peered":"2024-09-10T19:26:56.244708+0000","last_clean":"2024-09-10T19:26:56.244708+0000","last_became_active":"2024-09-10T19:24:01.975706+0000","last_became_peered":"2024-09-10T19:24:01.975706+0000","last_unstale":"2024-09-10T19:26:56.244708+0000","last_undegraded":"2024-09-10T19:26:56.244708+0000","last_fullsized":"2024-09-10T19:26:56.244708+0000","mapping_epoch":292,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":293,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:25:14.241294+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:25:14.241294+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0019729370000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"237'12","reported_seq":422,"reported_epoch":354,"state":"active+clean","last_fresh":"2024-09-10T19:26:56.237444+0000","last_change":"2024-09-10T19:24:54.606498+0000","last_active":"2024-09-10T19:26:56.237444+0000","last_peered":"2024-09-10T19:26:56.237444+0000","last_clean":"2024-09-10T19:26:56.237444+0000","last_became_active":"2024-09-10T19:24:01.971492+0000","last_became_peered":"2024-09-10T19:24:01.971492+0000","last_unstale":"2024-09-10T19:26:56.237444+0000","last_undegraded":"2024-09-10T19:26:56.237444+0000","last_fullsized":"2024-09-10T19:26:56.237444+0000","mapping_epoch":292,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":293,"parent":"0.0","parent_split_bits":0,"last_scrub":"237'12","last_scrub_stamp":"2024-09-10T19:24:54.606424+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:24:54.606424+0000","objects_scrubbed":3,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0049032219999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":18,"num_read_kb":15,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":378,"reported_epoch":354,"state":"active+clean","last_fresh":"2024-09-10T19:26:56.235867+0000","last_change":"2024-09-10T19:26:17.869690+0000","last_active":"2024-09-10T19:26:56.235867+0000","last_peered":"2024-09-10T19:26:56.235867+0000","last_clean":"2024-09-10T19:26:56.235867+0000","last_became_active":"2024-09-10T19:24:01.978215+0000","last_became_peered":"2024-09-10T19:24:01.978215+0000","last_unstale":"2024-09-10T19:26:56.235867+0000","last_undegraded":"2024-09-10T19:26:56.235867+0000","last_fullsized":"2024-09-10T19:26:56.235867+0000","mapping_epoch":292,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":293,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:26:17.869622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:11:23.264793+0000","last_clean_scrub_stamp":"2024-09-10T19:26:17.869622+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:27:23.953590+0000","scrub_duration":0.005157554,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":359,"reported_epoch":354,"state":"active+clean","last_fresh":"2024-09-10T19:26:56.237608+0000","last_change":"2024-09-10T19:24:49.802835+0000","last_active":"2024-09-10T19:26:56.237608+0000","last_peered":"2024-09-10T19:26:56.237608+0000","last_clean":"2024-09-10T19:26:56.237608+0000","last_became_active":"2024-09-10T19:10:14.054360+0000","last_became_peered":"2024-09-10T19:10:14.054360+0000","last_unstale":"2024-09-10T19:26:56.237608+0000","last_undegraded":"2024-09-10T19:26:56.237608+0000","last_fullsized":"2024-09-10T19:26:56.237608+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:24:49.802743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:24:49.802743+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.023450184999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"237'25","reported_seq":461,"reported_epoch":354,"state":"active+clean","last_fresh":"2024-09-10T19:26:56.236232+0000","last_change":"2024-09-10T19:25:16.796941+0000","last_active":"2024-09-10T19:26:56.236232+0000","last_peered":"2024-09-10T19:26:56.236232+0000","last_clean":"2024-09-10T19:26:56.236232+0000","last_became_active":"2024-09-10T19:24:01.971173+0000","last_became_peered":"2024-09-10T19:24:01.971173+0000","last_unstale":"2024-09-10T19:26:56.236232+0000","last_undegraded":"2024-09-10T19:26:56.236232+0000","last_fullsized":"2024-09-10T19:26:56.236232+0000","mapping_epoch":292,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":293,"parent":"0.0","parent_split_bits":0,"last_scrub":"237'25","last_scrub_stamp":"2024-09-10T19:25:16.796821+0000","last_deep_scrub":"80'15","last_deep_scrub_stamp":"2024-09-10T19:11:41.719310+0000","last_clean_scrub_stamp":"2024-09-10T19:25:16.796821+0000","objects_scrubbed":5,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0062295650000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":355,"reported_epoch":354,"state":"active+clean","last_fresh":"2024-09-10T19:26:56.235189+0000","last_change":"2024-09-10T19:25:14.758934+0000","last_active":"2024-09-10T19:26:56.235189+0000","last_peered":"2024-09-10T19:26:56.235189+0000","last_clean":"2024-09-10T19:26:56.235189+0000","last_became_active":"2024-09-10T19:10:07.685870+0000","last_became_peered":"2024-09-10T19:10:07.685870+0000","last_unstale":"2024-09-10T19:26:56.235189+0000","last_undegraded":"2024-09-10T19:26:56.235189+0000","last_fullsized":"2024-09-10T19:26:56.235189+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:25:14.758809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:25:14.758809+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.002869814,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"303'15","reported_seq":612,"reported_epoch":354,"state":"active+clean","last_fresh":"2024-09-10T19:26:56.235187+0000","last_change":"2024-09-10T19:24:47.487930+0000","last_active":"2024-09-10T19:26:56.235187+0000","last_peered":"2024-09-10T19:26:56.235187+0000","last_clean":"2024-09-10T19:26:56.235187+0000","last_became_active":"2024-09-10T19:24:24.917099+0000","last_became_peered":"2024-09-10T19:24:24.917099+0000","last_unstale":"2024-09-10T19:26:56.235187+0000","last_undegraded":"2024-09-10T19:26:56.235187+0000","last_fullsized":"2024-09-10T19:26:56.235187+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"303'15","last_scrub_stamp":"2024-09-10T19:24:47.487840+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:24:47.487840+0000","objects_scrubbed":1,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0054748990000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":307,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-09-10T19:26:58.290120+0000","last_change":"2024-09-10T19:25:19.396237+0000","last_active":"2024-09-10T19:26:58.290120+0000","last_peered":"2024-09-10T19:26:58.290120+0000","last_clean":"2024-09-10T19:26:58.290120+0000","last_became_active":"2024-09-10T19:24:23.900513+0000","last_became_peered":"2024-09-10T19:24:23.900513+0000","last_unstale":"2024-09-10T19:26:58.290120+0000","last_undegraded":"2024-09-10T19:26:58.290120+0000","last_fullsized":"2024-09-10T19:26:58.290120+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:25:19.396159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:25:19.396159+0000","last_clean_scrub_stamp":"2024-09-10T19:25:19.396159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0046456329999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"237'15","reported_seq":418,"reported_epoch":354,"state":"active+clean","last_fresh":"2024-09-10T19:26:56.235309+0000","last_change":"2024-09-10T19:25:16.712492+0000","last_active":"2024-09-10T19:26:56.235309+0000","last_peered":"2024-09-10T19:26:56.235309+0000","last_clean":"2024-09-10T19:26:56.235309+0000","last_became_active":"2024-09-10T19:16:24.859740+0000","last_became_peered":"2024-09-10T19:16:24.859740+0000","last_unstale":"2024-09-10T19:26:56.235309+0000","last_undegraded":"2024-09-10T19:26:56.235309+0000","last_fullsized":"2024-09-10T19:26:56.235309+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"237'15","last_scrub_stamp":"2024-09-10T19:25:16.712402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:25:16.712402+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0031776310000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":313,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-09-10T19:26:58.297527+0000","last_change":"2024-09-10T19:25:20.358819+0000","last_active":"2024-09-10T19:26:58.297527+0000","last_peered":"2024-09-10T19:26:58.297527+0000","last_clean":"2024-09-10T19:26:58.297527+0000","last_became_active":"2024-09-10T19:24:23.895163+0000","last_became_peered":"2024-09-10T19:24:23.895163+0000","last_unstale":"2024-09-10T19:26:58.297527+0000","last_undegraded":"2024-09-10T19:26:58.297527+0000","last_fullsized":"2024-09-10T19:26:58.297527+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:25:20.358737+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:25:20.358737+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.002668761,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"297'11","reported_seq":600,"reported_epoch":354,"state":"active+clean","last_fresh":"2024-09-10T19:26:56.236572+0000","last_change":"2024-09-10T19:26:16.945209+0000","last_active":"2024-09-10T19:26:56.236572+0000","last_peered":"2024-09-10T19:26:56.236572+0000","last_clean":"2024-09-10T19:26:56.236572+0000","last_became_active":"2024-09-10T19:24:01.974816+0000","last_became_peered":"2024-09-10T19:24:01.974816+0000","last_unstale":"2024-09-10T19:26:56.236572+0000","last_undegraded":"2024-09-10T19:26:56.236572+0000","last_fullsized":"2024-09-10T19:26:56.236572+0000","mapping_epoch":292,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":293,"parent":"0.0","parent_split_bits":0,"last_scrub":"297'11","last_scrub_stamp":"2024-09-10T19:26:16.945115+0000","last_deep_scrub":"297'11","last_deep_scrub_stamp":"2024-09-10T19:24:13.318130+0000","last_clean_scrub_stamp":"2024-09-10T19:26:16.945115+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:27:37.714654+0000","scrub_duration":0.0051112889999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":368,"reported_epoch":354,"state":"active+clean","last_fresh":"2024-09-10T19:26:56.247166+0000","last_change":"2024-09-10T19:24:37.671881+0000","last_active":"2024-09-10T19:26:56.247166+0000","last_peered":"2024-09-10T19:26:56.247166+0000","last_clean":"2024-09-10T19:26:56.247166+0000","last_became_active":"2024-09-10T19:24:23.903779+0000","last_became_peered":"2024-09-10T19:24:23.903779+0000","last_unstale":"2024-09-10T19:26:56.247166+0000","last_undegraded":"2024-09-10T19:26:56.247166+0000","last_fullsized":"2024-09-10T19:26:56.247166+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:24:37.671797+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:24:37.671797+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.002698401,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.12","version":"237'19","reported_seq":107,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-09-10T19:26:58.262853+0000","last_change":"2024-09-10T19:24:47.503877+0000","last_active":"2024-09-10T19:26:58.262853+0000","last_peered":"2024-09-10T19:26:58.262853+0000","last_clean":"2024-09-10T19:26:58.262853+0000","last_became_active":"2024-09-10T19:24:24.913130+0000","last_became_peered":"2024-09-10T19:24:24.913130+0000","last_unstale":"2024-09-10T19:26:58.262853+0000","last_undegraded":"2024-09-10T19:26:58.262853+0000","last_fullsized":"2024-09-10T19:26:58.262853+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"237'19","last_scrub_stamp":"2024-09-10T19:24:47.503819+0000","last_deep_scrub":"79'13","last_deep_scrub_stamp":"2024-09-10T19:11:22.072400+0000","last_clean_scrub_stamp":"2024-09-10T19:24:47.503819+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0097974459999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"59'1","reported_seq":382,"reported_epoch":354,"state":"active+clean","last_fresh":"2024-09-10T19:26:56.244484+0000","last_change":"2024-09-10T19:25:17.225875+0000","last_active":"2024-09-10T19:26:56.244484+0000","last_peered":"2024-09-10T19:26:56.244484+0000","last_clean":"2024-09-10T19:26:56.244484+0000","last_became_active":"2024-09-10T19:10:07.683263+0000","last_became_peered":"2024-09-10T19:10:07.683263+0000","last_unstale":"2024-09-10T19:26:56.244484+0000","last_undegraded":"2024-09-10T19:26:56.244484+0000","last_fullsized":"2024-09-10T19:26:56.244484+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'1","last_scrub_stamp":"2024-09-10T19:25:17.225794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:25:17.225794+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0095600029999999992,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":362,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":399,"reported_epoch":354,"state":"active+clean","last_fresh":"2024-09-10T19:26:56.236202+0000","last_change":"2024-09-10T19:26:23.842995+0000","last_active":"2024-09-10T19:26:56.236202+0000","last_peered":"2024-09-10T19:26:56.236202+0000","last_clean":"2024-09-10T19:26:56.236202+0000","last_became_active":"2024-09-10T19:24:24.923804+0000","last_became_peered":"2024-09-10T19:24:24.923804+0000","last_unstale":"2024-09-10T19:26:56.236202+0000","last_undegraded":"2024-09-10T19:26:56.236202+0000","last_fullsized":"2024-09-10T19:26:56.236202+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:26:23.842943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:26:23.842943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:27:48.719400+0000","scrub_duration":0.0015653100000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":326,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-09-10T19:26:58.263585+0000","last_change":"2024-09-10T19:25:27.412632+0000","last_active":"2024-09-10T19:26:58.263585+0000","last_peered":"2024-09-10T19:26:58.263585+0000","last_clean":"2024-09-10T19:26:58.263585+0000","last_became_active":"2024-09-10T19:24:24.916105+0000","last_became_peered":"2024-09-10T19:24:24.916105+0000","last_unstale":"2024-09-10T19:26:58.263585+0000","last_undegraded":"2024-09-10T19:26:58.263585+0000","last_fullsized":"2024-09-10T19:26:58.263585+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:25:27.412579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:25:27.412579+0000","last_clean_scrub_stamp":"2024-09-10T19:25:27.412579+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0084519790000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"237'19","reported_seq":447,"reported_epoch":354,"state":"active+clean","last_fresh":"2024-09-10T19:26:56.245292+0000","last_change":"2024-09-10T19:26:34.043973+0000","last_active":"2024-09-10T19:26:56.245292+0000","last_peered":"2024-09-10T19:26:56.245292+0000","last_clean":"2024-09-10T19:26:56.245292+0000","last_became_active":"2024-09-10T19:24:01.975520+0000","last_became_peered":"2024-09-10T19:24:01.975520+0000","last_unstale":"2024-09-10T19:26:56.245292+0000","last_undegraded":"2024-09-10T19:26:56.245292+0000","last_fullsized":"2024-09-10T19:26:56.245292+0000","mapping_epoch":292,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":293,"parent":"0.0","parent_split_bits":0,"last_scrub":"237'19","last_scrub_stamp":"2024-09-10T19:26:34.043784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:26:34.043784+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:27:34.264062+0000","scrub_duration":0.0096839449999999994,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":107,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-09-10T19:26:58.287374+0000","last_change":"2024-09-10T19:26:31.508255+0000","last_active":"2024-09-10T19:26:58.287374+0000","last_peered":"2024-09-10T19:26:58.287374+0000","last_clean":"2024-09-10T19:26:58.287374+0000","last_became_active":"2024-09-10T19:24:24.909540+0000","last_became_peered":"2024-09-10T19:24:24.909540+0000","last_unstale":"2024-09-10T19:26:58.287374+0000","last_undegraded":"2024-09-10T19:26:58.287374+0000","last_fullsized":"2024-09-10T19:26:58.287374+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:26:31.508168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:26:31.508168+0000","last_clean_scrub_stamp":"2024-09-10T19:26:31.508168+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:27:44.367352+0000","scrub_duration":0.069232562999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":83,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-09-10T19:26:58.287321+0000","last_change":"2024-09-10T19:26:16.473660+0000","last_active":"2024-09-10T19:26:58.287321+0000","last_peered":"2024-09-10T19:26:58.287321+0000","last_clean":"2024-09-10T19:26:58.287321+0000","last_became_active":"2024-09-10T19:24:01.975520+0000","last_became_peered":"2024-09-10T19:24:01.975520+0000","last_unstale":"2024-09-10T19:26:58.287321+0000","last_undegraded":"2024-09-10T19:26:58.287321+0000","last_fullsized":"2024-09-10T19:26:58.287321+0000","mapping_epoch":292,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":293,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:26:16.473571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:26:16.473571+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:27:33.963609+0000","scrub_duration":0.0030308330000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":366,"reported_epoch":354,"state":"active+clean","last_fresh":"2024-09-10T19:26:56.235109+0000","last_change":"2024-09-10T19:26:33.897629+0000","last_active":"2024-09-10T19:26:56.235109+0000","last_peered":"2024-09-10T19:26:56.235109+0000","last_clean":"2024-09-10T19:26:56.235109+0000","last_became_active":"2024-09-10T19:24:23.907136+0000","last_became_peered":"2024-09-10T19:24:23.907136+0000","last_unstale":"2024-09-10T19:26:56.235109+0000","last_undegraded":"2024-09-10T19:26:56.235109+0000","last_fullsized":"2024-09-10T19:26:56.235109+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:26:33.897552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:26:33.897552+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:27:59.638065+0000","scrub_duration":0.0025392180000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"237'10","reported_seq":406,"reported_epoch":354,"state":"active+clean","last_fresh":"2024-09-10T19:26:56.235008+0000","last_change":"2024-09-10T19:24:53.568499+0000","last_active":"2024-09-10T19:26:56.235008+0000","last_peered":"2024-09-10T19:26:56.235008+0000","last_clean":"2024-09-10T19:26:56.235008+0000","last_became_active":"2024-09-10T19:24:01.970758+0000","last_became_peered":"2024-09-10T19:24:01.970758+0000","last_unstale":"2024-09-10T19:26:56.235008+0000","last_undegraded":"2024-09-10T19:26:56.235008+0000","last_fullsized":"2024-09-10T19:26:56.235008+0000","mapping_epoch":292,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":293,"parent":"0.0","parent_split_bits":0,"last_scrub":"237'10","last_scrub_stamp":"2024-09-10T19:24:53.568298+0000","last_deep_scrub":"222'6","last_deep_scrub_stamp":"2024-09-10T19:19:30.328841+0000","last_clean_scrub_stamp":"2024-09-10T19:24:53.568298+0000","objects_scrubbed":2,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.002398168,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":12,"num_read_kb":10,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":385,"reported_epoch":354,"state":"active+clean","last_fresh":"2024-09-10T19:26:56.235942+0000","last_change":"2024-09-10T19:26:32.839813+0000","last_active":"2024-09-10T19:26:56.235942+0000","last_peered":"2024-09-10T19:26:56.235942+0000","last_clean":"2024-09-10T19:26:56.235942+0000","last_became_active":"2024-09-10T19:24:01.978534+0000","last_became_peered":"2024-09-10T19:24:01.978534+0000","last_unstale":"2024-09-10T19:26:56.235942+0000","last_undegraded":"2024-09-10T19:26:56.235942+0000","last_fullsized":"2024-09-10T19:26:56.235942+0000","mapping_epoch":292,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":293,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:26:32.839727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:26:32.839727+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:27:39.076381+0000","scrub_duration":0.0064300950000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,1],"acting":[5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":371,"reported_epoch":354,"state":"active+clean","last_fresh":"2024-09-10T19:26:56.244530+0000","last_change":"2024-09-10T19:24:42.823431+0000","last_active":"2024-09-10T19:26:56.244530+0000","last_peered":"2024-09-10T19:26:56.244530+0000","last_clean":"2024-09-10T19:26:56.244530+0000","last_became_active":"2024-09-10T19:10:11.698536+0000","last_became_peered":"2024-09-10T19:10:11.698536+0000","last_unstale":"2024-09-10T19:26:56.244530+0000","last_undegraded":"2024-09-10T19:26:56.244530+0000","last_fullsized":"2024-09-10T19:26:56.244530+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:24:42.823297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:11:31.837944+0000","last_clean_scrub_stamp":"2024-09-10T19:24:42.823297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0052557389999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":379,"reported_epoch":354,"state":"active+clean","last_fresh":"2024-09-10T19:26:56.238836+0000","last_change":"2024-09-10T19:25:16.770678+0000","last_active":"2024-09-10T19:26:56.238836+0000","last_peered":"2024-09-10T19:26:56.238836+0000","last_clean":"2024-09-10T19:26:56.238836+0000","last_became_active":"2024-09-10T19:16:24.852356+0000","last_became_peered":"2024-09-10T19:16:24.852356+0000","last_unstale":"2024-09-10T19:26:56.238836+0000","last_undegraded":"2024-09-10T19:26:56.238836+0000","last_fullsized":"2024-09-10T19:26:56.238836+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:25:16.770592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:25:16.770592+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001878469,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,3],"acting":[7,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.11","version":"237'19","reported_seq":155,"reported_epoch":354,"state":"active+clean","last_fresh":"2024-09-10T19:26:56.234894+0000","last_change":"2024-09-10T19:26:31.962670+0000","last_active":"2024-09-10T19:26:56.234894+0000","last_peered":"2024-09-10T19:26:56.234894+0000","last_clean":"2024-09-10T19:26:56.234894+0000","last_became_active":"2024-09-10T19:21:04.169444+0000","last_became_peered":"2024-09-10T19:21:04.169444+0000","last_unstale":"2024-09-10T19:26:56.234894+0000","last_undegraded":"2024-09-10T19:26:56.234894+0000","last_fullsized":"2024-09-10T19:26:56.234894+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"237'19","last_scrub_stamp":"2024-09-10T19:26:31.962585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:26:31.962585+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:27:39.609732+0000","scrub_duration":0.0022806200000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":392,"reported_epoch":354,"state":"active+clean","last_fresh":"2024-09-10T19:26:56.245902+0000","last_change":"2024-09-10T19:26:08.395741+0000","last_active":"2024-09-10T19:26:56.245902+0000","last_peered":"2024-09-10T19:26:56.245902+0000","last_clean":"2024-09-10T19:26:56.245902+0000","last_became_active":"2024-09-10T19:24:01.974565+0000","last_became_peered":"2024-09-10T19:24:01.974565+0000","last_unstale":"2024-09-10T19:26:56.245902+0000","last_undegraded":"2024-09-10T19:26:56.245902+0000","last_fullsized":"2024-09-10T19:26:56.245902+0000","mapping_epoch":292,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":293,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:26:08.395687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:13:05.876655+0000","last_clean_scrub_stamp":"2024-09-10T19:26:08.395687+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:27:08.537061+0000","scrub_duration":0.0050263720000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":361,"reported_epoch":354,"state":"active+clean","last_fresh":"2024-09-10T19:26:56.236260+0000","last_change":"2024-09-10T19:25:23.767085+0000","last_active":"2024-09-10T19:26:56.236260+0000","last_peered":"2024-09-10T19:26:56.236260+0000","last_clean":"2024-09-10T19:26:56.236260+0000","last_became_active":"2024-09-10T19:10:11.699013+0000","last_became_peered":"2024-09-10T19:10:11.699013+0000","last_unstale":"2024-09-10T19:26:56.236260+0000","last_undegraded":"2024-09-10T19:26:56.236260+0000","last_fullsized":"2024-09-10T19:26:56.236260+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:25:23.766985+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:11:57.112092+0000","last_clean_scrub_stamp":"2024-09-10T19:25:23.766985+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0068928649999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":366,"reported_epoch":354,"state":"active+clean","last_fresh":"2024-09-10T19:26:56.235684+0000","last_change":"2024-09-10T19:26:29.959285+0000","last_active":"2024-09-10T19:26:56.235684+0000","last_peered":"2024-09-10T19:26:56.235684+0000","last_clean":"2024-09-10T19:26:56.235684+0000","last_became_active":"2024-09-10T19:24:23.896454+0000","last_became_peered":"2024-09-10T19:24:23.896454+0000","last_unstale":"2024-09-10T19:26:56.235684+0000","last_undegraded":"2024-09-10T19:26:56.235684+0000","last_fullsized":"2024-09-10T19:26:56.235684+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:26:29.959202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:26:29.959202+0000","last_clean_scrub_stamp":"2024-09-10T19:26:29.959202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:27:48.266668+0000","scrub_duration":0.0059040539999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"80'2","reported_seq":385,"reported_epoch":354,"state":"active+clean","last_fresh":"2024-09-10T19:26:56.238877+0000","last_change":"2024-09-10T19:26:16.224098+0000","last_active":"2024-09-10T19:26:56.238877+0000","last_peered":"2024-09-10T19:26:56.238877+0000","last_clean":"2024-09-10T19:26:56.238877+0000","last_became_active":"2024-09-10T19:16:24.851220+0000","last_became_peered":"2024-09-10T19:16:24.851220+0000","last_unstale":"2024-09-10T19:26:56.238877+0000","last_undegraded":"2024-09-10T19:26:56.238877+0000","last_fullsized":"2024-09-10T19:26:56.238877+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"80'2","last_scrub_stamp":"2024-09-10T19:26:16.223955+0000","last_deep_scrub":"80'2","last_deep_scrub_stamp":"2024-09-10T19:11:30.272043+0000","last_clean_scrub_stamp":"2024-09-10T19:26:16.223955+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:27:24.628267+0000","scrub_duration":0.012727367999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":369,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":369,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,3],"acting":[7,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":372,"reported_epoch":354,"state":"active+clean","last_fresh":"2024-09-10T19:26:56.239079+0000","last_change":"2024-09-10T19:24:41.498889+0000","last_active":"2024-09-10T19:26:56.239079+0000","last_peered":"2024-09-10T19:26:56.239079+0000","last_clean":"2024-09-10T19:26:56.239079+0000","last_became_active":"2024-09-10T19:24:23.906812+0000","last_became_peered":"2024-09-10T19:24:23.906812+0000","last_unstale":"2024-09-10T19:26:56.239079+0000","last_undegraded":"2024-09-10T19:26:56.239079+0000","last_fullsized":"2024-09-10T19:26:56.239079+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:24:41.498807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:24:41.498807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.010867098,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,0],"acting":[5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1e","version":"237'25","reported_seq":346,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-09-10T19:26:58.289977+0000","last_change":"2024-09-10T19:25:15.471102+0000","last_active":"2024-09-10T19:26:58.289977+0000","last_peered":"2024-09-10T19:26:58.289977+0000","last_clean":"2024-09-10T19:26:58.289977+0000","last_became_active":"2024-09-10T19:24:23.899507+0000","last_became_peered":"2024-09-10T19:24:23.899507+0000","last_unstale":"2024-09-10T19:26:58.289977+0000","last_undegraded":"2024-09-10T19:26:58.289977+0000","last_fullsized":"2024-09-10T19:26:58.289977+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"237'25","last_scrub_stamp":"2024-09-10T19:25:15.471043+0000","last_deep_scrub":"80'15","last_deep_scrub_stamp":"2024-09-10T19:11:25.883919+0000","last_clean_scrub_stamp":"2024-09-10T19:25:15.471043+0000","objects_scrubbed":5,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0027617729999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":24,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1f","version":"66'8","reported_seq":587,"reported_epoch":354,"state":"active+clean","last_fresh":"2024-09-10T19:26:56.245758+0000","last_change":"2024-09-10T19:24:46.897327+0000","last_active":"2024-09-10T19:26:56.245758+0000","last_peered":"2024-09-10T19:26:56.245758+0000","last_clean":"2024-09-10T19:26:56.245758+0000","last_became_active":"2024-09-10T19:10:11.695427+0000","last_became_peered":"2024-09-10T19:10:11.695427+0000","last_unstale":"2024-09-10T19:26:56.245758+0000","last_undegraded":"2024-09-10T19:26:56.245758+0000","last_fullsized":"2024-09-10T19:26:56.245758+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'8","last_scrub_stamp":"2024-09-10T19:24:46.897253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:24:46.897253+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0048746450000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":83,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-09-10T19:26:58.287503+0000","last_change":"2024-09-10T19:25:18.764190+0000","last_active":"2024-09-10T19:26:58.287503+0000","last_peered":"2024-09-10T19:26:58.287503+0000","last_clean":"2024-09-10T19:26:58.287503+0000","last_became_active":"2024-09-10T19:24:01.980391+0000","last_became_peered":"2024-09-10T19:24:01.980391+0000","last_unstale":"2024-09-10T19:26:58.287503+0000","last_undegraded":"2024-09-10T19:26:58.287503+0000","last_fullsized":"2024-09-10T19:26:58.287503+0000","mapping_epoch":292,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":293,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:25:18.764101+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:25:18.764101+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0082808180000000006,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":382,"reported_epoch":354,"state":"active+clean","last_fresh":"2024-09-10T19:26:56.235679+0000","last_change":"2024-09-10T19:26:36.923605+0000","last_active":"2024-09-10T19:26:56.235679+0000","last_peered":"2024-09-10T19:26:56.235679+0000","last_clean":"2024-09-10T19:26:56.235679+0000","last_became_active":"2024-09-10T19:24:23.905869+0000","last_became_peered":"2024-09-10T19:24:23.905869+0000","last_unstale":"2024-09-10T19:26:56.235679+0000","last_undegraded":"2024-09-10T19:26:56.235679+0000","last_fullsized":"2024-09-10T19:26:56.235679+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:26:36.923520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:26:36.923520+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:28:05.745115+0000","scrub_duration":0.001294981,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"237'24","reported_seq":478,"reported_epoch":354,"state":"active+clean","last_fresh":"2024-09-10T19:26:56.245830+0000","last_change":"2024-09-10T19:25:31.311744+0000","last_active":"2024-09-10T19:26:56.245830+0000","last_peered":"2024-09-10T19:26:56.245830+0000","last_clean":"2024-09-10T19:26:56.245830+0000","last_became_active":"2024-09-10T19:24:01.974988+0000","last_became_peered":"2024-09-10T19:24:01.974988+0000","last_unstale":"2024-09-10T19:26:56.245830+0000","last_undegraded":"2024-09-10T19:26:56.245830+0000","last_fullsized":"2024-09-10T19:26:56.245830+0000","mapping_epoch":292,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":293,"parent":"0.0","parent_split_bits":0,"last_scrub":"237'24","last_scrub_stamp":"2024-09-10T19:25:31.311493+0000","last_deep_scrub":"80'16","last_deep_scrub_stamp":"2024-09-10T19:11:40.853611+0000","last_clean_scrub_stamp":"2024-09-10T19:25:31.311493+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0054020810000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":301,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-09-10T19:26:58.289900+0000","last_change":"2024-09-10T19:26:19.418628+0000","last_active":"2024-09-10T19:26:58.289900+0000","last_peered":"2024-09-10T19:26:58.289900+0000","last_clean":"2024-09-10T19:26:58.289900+0000","last_became_active":"2024-09-10T19:24:24.899153+0000","last_became_peered":"2024-09-10T19:24:24.899153+0000","last_unstale":"2024-09-10T19:26:58.289900+0000","last_undegraded":"2024-09-10T19:26:58.289900+0000","last_fullsized":"2024-09-10T19:26:58.289900+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:26:19.418556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:11:28.822327+0000","last_clean_scrub_stamp":"2024-09-10T19:26:19.418556+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:27:48.413741+0000","scrub_duration":0.005516361,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":368,"reported_epoch":354,"state":"active+clean","last_fresh":"2024-09-10T19:26:56.247609+0000","last_change":"2024-09-10T19:26:30.969045+0000","last_active":"2024-09-10T19:26:56.247609+0000","last_peered":"2024-09-10T19:26:56.247609+0000","last_clean":"2024-09-10T19:26:56.247609+0000","last_became_active":"2024-09-10T19:10:14.053619+0000","last_became_peered":"2024-09-10T19:10:14.053619+0000","last_unstale":"2024-09-10T19:26:56.247609+0000","last_undegraded":"2024-09-10T19:26:56.247609+0000","last_fullsized":"2024-09-10T19:26:56.247609+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:26:30.968972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:26:30.968972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:28:00.935626+0000","scrub_duration":0.0072260129999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":375,"reported_epoch":354,"state":"active+clean","last_fresh":"2024-09-10T19:26:56.247390+0000","last_change":"2024-09-10T19:25:26.260593+0000","last_active":"2024-09-10T19:26:56.247390+0000","last_peered":"2024-09-10T19:26:56.247390+0000","last_clean":"2024-09-10T19:26:56.247390+0000","last_became_active":"2024-09-10T19:10:07.685997+0000","last_became_peered":"2024-09-10T19:10:07.685997+0000","last_unstale":"2024-09-10T19:26:56.247390+0000","last_undegraded":"2024-09-10T19:26:56.247390+0000","last_fullsized":"2024-09-10T19:26:56.247390+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:25:26.260513+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:25:26.260513+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0073960600000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1c","version":"237'34","reported_seq":113,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-09-10T19:26:58.286409+0000","last_change":"2024-09-10T19:24:44.441810+0000","last_active":"2024-09-10T19:26:58.286409+0000","last_peered":"2024-09-10T19:26:58.286409+0000","last_clean":"2024-09-10T19:26:58.286409+0000","last_became_active":"2024-09-10T19:24:24.910577+0000","last_became_peered":"2024-09-10T19:24:24.910577+0000","last_unstale":"2024-09-10T19:26:58.286409+0000","last_undegraded":"2024-09-10T19:26:58.286409+0000","last_fullsized":"2024-09-10T19:26:58.286409+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"237'34","last_scrub_stamp":"2024-09-10T19:24:44.441712+0000","last_deep_scrub":"78'22","last_deep_scrub_stamp":"2024-09-10T19:11:21.118937+0000","last_clean_scrub_stamp":"2024-09-10T19:24:44.441712+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0042000179999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":97,"reported_epoch":355,"state":"active+clean","last_fresh":"2024-09-10T19:26:58.287611+0000","last_change":"2024-09-10T19:24:35.438665+0000","last_active":"2024-09-10T19:26:58.287611+0000","last_peered":"2024-09-10T19:26:58.287611+0000","last_clean":"2024-09-10T19:26:58.287611+0000","last_became_active":"2024-09-10T19:24:24.887205+0000","last_became_peered":"2024-09-10T19:24:24.887205+0000","last_unstale":"2024-09-10T19:26:58.287611+0000","last_undegraded":"2024-09-10T19:26:58.287611+0000","last_fullsized":"2024-09-10T19:26:58.287611+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:24:35.438586+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:24:35.438586+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0036432579999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":362,"reported_epoch":354,"state":"active+clean","last_fresh":"2024-09-10T19:26:56.237133+0000","last_change":"2024-09-10T19:26:30.990896+0000","last_active":"2024-09-10T19:26:56.237133+0000","last_peered":"2024-09-10T19:26:56.237133+0000","last_clean":"2024-09-10T19:26:56.237133+0000","last_became_active":"2024-09-10T19:10:14.053698+0000","last_became_peered":"2024-09-10T19:10:14.053698+0000","last_unstale":"2024-09-10T19:26:56.237133+0000","last_undegraded":"2024-09-10T19:26:56.237133+0000","last_fullsized":"2024-09-10T19:26:56.237133+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:26:30.990819+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:24:46.518816+0000","last_clean_scrub_stamp":"2024-09-10T19:26:30.990819+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:27:45.420255+0000","scrub_duration":0.0073104629999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":365,"reported_epoch":354,"state":"active+clean","last_fresh":"2024-09-10T19:26:56.236369+0000","last_change":"2024-09-10T19:26:13.910830+0000","last_active":"2024-09-10T19:26:56.236369+0000","last_peered":"2024-09-10T19:26:56.236369+0000","last_clean":"2024-09-10T19:26:56.236369+0000","last_became_active":"2024-09-10T19:10:07.682811+0000","last_became_peered":"2024-09-10T19:10:07.682811+0000","last_unstale":"2024-09-10T19:26:56.236369+0000","last_undegraded":"2024-09-10T19:26:56.236369+0000","last_fullsized":"2024-09-10T19:26:56.236369+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:26:13.910732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:26:13.910732+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:27:32.512695+0000","scrub_duration":0.0067593929999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1d","version":"250'23","reported_seq":441,"reported_epoch":354,"state":"active+clean","last_fresh":"2024-09-10T19:26:56.236291+0000","last_change":"2024-09-10T19:25:31.852135+0000","last_active":"2024-09-10T19:26:56.236291+0000","last_peered":"2024-09-10T19:26:56.236291+0000","last_clean":"2024-09-10T19:26:56.236291+0000","last_became_active":"2024-09-10T19:24:01.976664+0000","last_became_peered":"2024-09-10T19:24:01.976664+0000","last_unstale":"2024-09-10T19:26:56.236291+0000","last_undegraded":"2024-09-10T19:26:56.236291+0000","last_fullsized":"2024-09-10T19:26:56.236291+0000","mapping_epoch":292,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":293,"parent":"0.0","parent_split_bits":0,"last_scrub":"250'23","last_scrub_stamp":"2024-09-10T19:25:31.852016+0000","last_deep_scrub":"250'23","last_deep_scrub_stamp":"2024-09-10T19:20:55.582173+0000","last_clean_scrub_stamp":"2024-09-10T19:25:31.852016+0000","objects_scrubbed":5,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0046609609999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":380,"reported_epoch":354,"state":"active+clean","last_fresh":"2024-09-10T19:26:56.246576+0000","last_change":"2024-09-10T19:26:29.937585+0000","last_active":"2024-09-10T19:26:56.246576+0000","last_peered":"2024-09-10T19:26:56.246576+0000","last_clean":"2024-09-10T19:26:56.246576+0000","last_became_active":"2024-09-10T19:16:24.843013+0000","last_became_peered":"2024-09-10T19:16:24.843013+0000","last_unstale":"2024-09-10T19:26:56.246576+0000","last_undegraded":"2024-09-10T19:26:56.246576+0000","last_fullsized":"2024-09-10T19:26:56.246576+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:26:29.937532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:26:29.937532+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:27:55.254997+0000","scrub_duration":0.0024737750000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":10,"num_pg":16,"stat_sum":{"num_bytes":2013724696,"num_objects":30728,"num_object_clones":0,"num_object_copies":92184,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":30728,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":215684,"num_write_kb":7885121,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4680,"num_bytes_recovered":306708480,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":6007697408,"data_stored":6007685192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":71584,"ondisk_log_size":71584,"up":48,"acting":48,"num_store_stats":7},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":382,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":1528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":88,"ondisk_log_size":88,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":177,"num_object_clones":0,"num_object_copies":531,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":177,"num_whiteouts":0,"num_read":1117,"num_read_kb":874,"num_write":703,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":293,"num_bytes_recovered":5886,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":458752,"data_stored":12124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":764,"ondisk_log_size":764,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":189,"num_read_kb":184,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":778,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1613,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":9,"ondisk_log_size":9,"up":9,"acting":9,"num_store_stats":7},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1328,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":27,"num_read_kb":27,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":3984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4,"ondisk_log_size":4,"up":96,"acting":96,"num_store_stats":8},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":870,"ondisk_log_size":870,"up":33,"acting":33,"num_store_stats":7}],"osd_stats":[{"osd":7,"up_from":54,"seq":231928234191,"num_pgs":69,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1534960,"kb_used_data":642600,"kb_used_omap":1,"kb_used_meta":892286,"kb_avail":92210192,"statfs":{"total":95995035648,"available":94423236608,"internally_reserved":0,"allocated":658022400,"data_stored":657721519,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":913701208},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 19:26:26 2024","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.89200000000000002,"15min":0.94399999999999995},"min":{"1min":0.24399999999999999,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":2.6949999999999998,"5min":3,"15min":3},"last":0.76200000000000001},{"interface":"front","average":{"1min":1.036,"5min":0.75800000000000001,"15min":0.71199999999999997},"min":{"1min":0.33100000000000002,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":3.5409999999999999,"5min":3.8260000000000001,"15min":3.8260000000000001},"last":1.371}]},{"osd":1,"last update":"Tue Sep 10 19:26:54 2024","interfaces":[{"interface":"back","average":{"1min":0.88,"5min":0.93000000000000005,"15min":1},"min":{"1min":0.24299999999999999,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":4.1139999999999999,"5min":5.8140000000000001,"15min":5.8140000000000001},"last":0.79300000000000004},{"interface":"front","average":{"1min":1.2769999999999999,"5min":0.98099999999999998,"15min":0.88500000000000001},"min":{"1min":0.26700000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":4.8540000000000001,"5min":4.8540000000000001,"15min":4.8540000000000001},"last":1.1539999999999999}]},{"osd":2,"last update":"Tue Sep 10 19:26:11 2024","interfaces":[{"interface":"back","average":{"1min":0.92400000000000004,"5min":0.89700000000000002,"15min":0.96699999999999997},"min":{"1min":0.38300000000000001,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":3.601,"5min":4.0190000000000001,"15min":31.940999999999999},"last":1.403},{"interface":"front","average":{"1min":1.2649999999999999,"5min":0.95999999999999996,"15min":1.0780000000000001},"min":{"1min":0.32500000000000001,"5min":0.26300000000000001,"15min":0.24099999999999999},"max":{"1min":4.3150000000000004,"5min":4.4139999999999997,"15min":25.056000000000001},"last":1.2070000000000001}]},{"osd":3,"last update":"Tue Sep 10 19:26:11 2024","interfaces":[{"interface":"back","average":{"1min":0.90500000000000003,"5min":0.94199999999999995,"15min":1.331},"min":{"1min":0.38500000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":3.5790000000000002,"5min":4.6070000000000002,"15min":35.277999999999999},"last":1.446},{"interface":"front","average":{"1min":1.073,"5min":0.92400000000000004,"15min":1.321},"min":{"1min":0.32500000000000001,"5min":0.27400000000000002,"15min":0.25700000000000001},"max":{"1min":4.1379999999999999,"5min":4.4500000000000002,"15min":36.512999999999998},"last":1.1870000000000001}]},{"osd":4,"last update":"Tue Sep 10 19:26:11 2024","interfaces":[{"interface":"back","average":{"1min":1.8,"5min":4.5860000000000003,"15min":2.3650000000000002},"min":{"1min":0.49199999999999999,"5min":0.222,"15min":0.222},"max":{"1min":5.8600000000000003,"5min":231.923,"15min":231.923},"last":0.88300000000000001},{"interface":"front","average":{"1min":1.4239999999999999,"5min":4.4340000000000002,"15min":2.1000000000000001},"min":{"1min":0.34999999999999998,"5min":0.24399999999999999,"15min":0.23000000000000001},"max":{"1min":3.8959999999999999,"5min":231.322,"15min":231.322},"last":1.4259999999999999}]},{"osd":5,"last update":"Tue Sep 10 19:26:11 2024","interfaces":[{"interface":"back","average":{"1min":1.409,"5min":2.0179999999999998,"15min":1.552},"min":{"1min":0.34999999999999998,"5min":0.24099999999999999,"15min":0.23699999999999999},"max":{"1min":4.3550000000000004,"5min":67.382999999999996,"15min":67.382999999999996},"last":1.2649999999999999},{"interface":"front","average":{"1min":1.8500000000000001,"5min":2.3679999999999999,"15min":1.7589999999999999},"min":{"1min":0.309,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":5.8789999999999996,"5min":67.75,"15min":67.75},"last":1.716}]},{"osd":6,"last update":"Tue Sep 10 19:26:11 2024","interfaces":[{"interface":"back","average":{"1min":1.6830000000000001,"5min":3.8799999999999999,"15min":2.161},"min":{"1min":0.33000000000000002,"5min":0.28000000000000003,"15min":0.26200000000000001},"max":{"1min":6.7869999999999999,"5min":181.53800000000001,"15min":181.53800000000001},"last":0.81999999999999995},{"interface":"front","average":{"1min":1.171,"5min":3.4860000000000002,"15min":1.869},"min":{"1min":0.34499999999999997,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":3.9590000000000001,"5min":182.05600000000001,"15min":182.05600000000001},"last":1.466}]}]},{"osd":6,"up_from":47,"seq":201863463124,"num_pgs":64,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1783812,"kb_used_data":901368,"kb_used_omap":1,"kb_used_meta":882302,"kb_avail":91961340,"statfs":{"total":95995035648,"available":94168412160,"internally_reserved":0,"allocated":923000832,"data_stored":922684679,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":903477592},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 19:26:30 2024","interfaces":[{"interface":"back","average":{"1min":1.3049999999999999,"5min":1.76,"15min":1.8360000000000001},"min":{"1min":0.183,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":5.0410000000000004,"5min":18.545999999999999,"15min":18.545999999999999},"last":0.217},{"interface":"front","average":{"1min":1.3879999999999999,"5min":1.8759999999999999,"15min":1.958},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":7.1020000000000003,"5min":19.818000000000001,"15min":19.818000000000001},"last":5.8319999999999999}]},{"osd":1,"last update":"Tue Sep 10 19:26:56 2024","interfaces":[{"interface":"back","average":{"1min":1.7509999999999999,"5min":1.089,"15min":0.84499999999999997},"min":{"1min":0.21199999999999999,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":7.0019999999999998,"5min":7.0019999999999998,"15min":7.0019999999999998},"last":0.251},{"interface":"front","average":{"1min":1.212,"5min":0.84799999999999998,"15min":0.78400000000000003},"min":{"1min":0.23000000000000001,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":3.25,"5min":3.25,"15min":3.25},"last":5.7329999999999997}]},{"osd":2,"last update":"Tue Sep 10 19:26:02 2024","interfaces":[{"interface":"back","average":{"1min":0.84499999999999997,"5min":0.81200000000000006,"15min":1.1819999999999999},"min":{"1min":0.29699999999999999,"5min":0.23799999999999999,"15min":0.20399999999999999},"max":{"1min":4.4459999999999997,"5min":5.9029999999999996,"15min":112.864},"last":5.8010000000000002},{"interface":"front","average":{"1min":1.409,"5min":0.90700000000000003,"15min":1.3},"min":{"1min":0.26200000000000001,"5min":0.25600000000000001,"15min":0.254},"max":{"1min":5.1920000000000002,"5min":5.1920000000000002,"15min":112.822},"last":2.9399999999999999}]},{"osd":3,"last update":"Tue Sep 10 19:26:02 2024","interfaces":[{"interface":"back","average":{"1min":2.044,"5min":0.96699999999999997,"15min":1.5189999999999999},"min":{"1min":0.32200000000000001,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":5.3230000000000004,"5min":5.3230000000000004,"15min":125.56399999999999},"last":0.37},{"interface":"front","average":{"1min":1.0189999999999999,"5min":0.85799999999999998,"15min":1.4910000000000001},"min":{"1min":0.29199999999999998,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":4.3899999999999997,"5min":4.3899999999999997,"15min":125.47},"last":5.7720000000000002}]},{"osd":4,"last update":"Tue Sep 10 19:26:02 2024","interfaces":[{"interface":"back","average":{"1min":2.9180000000000001,"5min":1.5229999999999999,"15min":1.1259999999999999},"min":{"1min":0.32900000000000001,"5min":0.254,"15min":0.20699999999999999},"max":{"1min":7.7599999999999998,"5min":7.7599999999999998,"15min":7.7599999999999998},"last":3.98},{"interface":"front","average":{"1min":3.0070000000000001,"5min":1.4650000000000001,"15min":1.0569999999999999},"min":{"1min":0.41399999999999998,"5min":0.24099999999999999,"15min":0.22500000000000001},"max":{"1min":6.2549999999999999,"5min":8.6419999999999995,"15min":8.6419999999999995},"last":0.27600000000000002}]},{"osd":5,"last update":"Tue Sep 10 19:26:02 2024","interfaces":[{"interface":"back","average":{"1min":1.8340000000000001,"5min":1.333,"15min":1.258},"min":{"1min":0.34999999999999998,"5min":0.28100000000000003,"15min":0.248},"max":{"1min":9.7270000000000003,"5min":9.7270000000000003,"15min":44.765000000000001},"last":2.9780000000000002},{"interface":"front","average":{"1min":2.605,"5min":1.5129999999999999,"15min":1.3720000000000001},"min":{"1min":0.36499999999999999,"5min":0.28399999999999997,"15min":0.27500000000000002},"max":{"1min":5.2309999999999999,"5min":6.9459999999999997,"15min":43.036000000000001},"last":2.8340000000000001}]},{"osd":7,"last update":"Tue Sep 10 19:26:30 2024","interfaces":[{"interface":"back","average":{"1min":2.3620000000000001,"5min":2.3479999999999999,"15min":1.3879999999999999},"min":{"1min":0.32000000000000001,"5min":0.28100000000000003,"15min":0.27500000000000002},"max":{"1min":5.6269999999999998,"5min":80.629999999999995,"15min":80.629999999999995},"last":6.0490000000000004},{"interface":"front","average":{"1min":2.7029999999999998,"5min":2.552,"15min":1.599},"min":{"1min":0.33800000000000002,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":16.654,"5min":81.088999999999999,"15min":81.088999999999999},"last":0.29999999999999999}]}]},{"osd":5,"up_from":39,"seq":167503724761,"num_pgs":68,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1786488,"kb_used_data":884336,"kb_used_omap":1,"kb_used_meta":902078,"kb_avail":91958664,"statfs":{"total":95995035648,"available":94165671936,"internally_reserved":0,"allocated":905560064,"data_stored":905252751,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":923728216},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,1,3,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 19:26:26 2024","interfaces":[{"interface":"back","average":{"1min":1.2709999999999999,"5min":1.151,"15min":1.131},"min":{"1min":0.253,"5min":0.187,"15min":0.187},"max":{"1min":3.5710000000000002,"5min":4.2220000000000004,"15min":4.2220000000000004},"last":1.4059999999999999},{"interface":"front","average":{"1min":0.93999999999999995,"5min":1.2190000000000001,"15min":1.2649999999999999},"min":{"1min":0.214,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":3.339,"5min":3.8580000000000001,"15min":3.8580000000000001},"last":0.48699999999999999}]},{"osd":1,"last update":"Tue Sep 10 19:26:46 2024","interfaces":[{"interface":"back","average":{"1min":0.84499999999999997,"5min":0.98199999999999998,"15min":0.98799999999999999},"min":{"1min":0.247,"5min":0.247,"15min":0.247},"max":{"1min":2.7250000000000001,"5min":8.2319999999999993,"15min":8.2319999999999993},"last":1.2110000000000001},{"interface":"front","average":{"1min":1.1399999999999999,"5min":1.0069999999999999,"15min":1.0449999999999999},"min":{"1min":0.37,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":3.605,"5min":3.9449999999999998,"15min":3.9449999999999998},"last":1.4399999999999999}]},{"osd":2,"last update":"Tue Sep 10 19:26:29 2024","interfaces":[{"interface":"back","average":{"1min":1.319,"5min":1.0880000000000001,"15min":1.046},"min":{"1min":0.47399999999999998,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":3.649,"5min":8.6920000000000002,"15min":9.9399999999999995},"last":1.484},{"interface":"front","average":{"1min":0.90900000000000003,"5min":0.97899999999999998,"15min":1.0109999999999999},"min":{"1min":0.27700000000000002,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":2.8570000000000002,"5min":3.8319999999999999,"15min":6.3789999999999996},"last":0.54900000000000004}]},{"osd":3,"last update":"Tue Sep 10 19:26:29 2024","interfaces":[{"interface":"back","average":{"1min":1.2809999999999999,"5min":1.3999999999999999,"15min":1.131},"min":{"1min":0.442,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":3.629,"5min":26.355,"15min":26.355},"last":1.464},{"interface":"front","average":{"1min":0.97199999999999998,"5min":1.2949999999999999,"15min":1.091},"min":{"1min":0.30599999999999999,"5min":0.246,"15min":0.246},"max":{"1min":2.8359999999999999,"5min":26.327000000000002,"15min":26.327000000000002},"last":0.52600000000000002}]},{"osd":4,"last update":"Tue Sep 10 19:26:29 2024","interfaces":[{"interface":"back","average":{"1min":1.718,"5min":2.9489999999999998,"15min":1.6879999999999999},"min":{"1min":0.55700000000000005,"5min":0.26100000000000001,"15min":0.23699999999999999},"max":{"1min":4.0039999999999996,"5min":128.08099999999999,"15min":128.08099999999999},"last":1.153},{"interface":"front","average":{"1min":1.8540000000000001,"5min":3.0859999999999999,"15min":1.792},"min":{"1min":0.54800000000000004,"5min":0.217,"15min":0.217},"max":{"1min":5.3109999999999999,"5min":126.321,"15min":126.321},"last":6.4000000000000004}]},{"osd":6,"last update":"Tue Sep 10 19:26:04 2024","interfaces":[{"interface":"back","average":{"1min":1.8520000000000001,"5min":2.6419999999999999,"15min":1.6719999999999999},"min":{"1min":0.33700000000000002,"5min":0.28399999999999997,"15min":0.27100000000000002},"max":{"1min":8.3339999999999996,"5min":111.328,"15min":111.328},"last":6.3440000000000003},{"interface":"front","average":{"1min":2.5489999999999999,"5min":2.7250000000000001,"15min":1.655},"min":{"1min":0.35299999999999998,"5min":0.27200000000000002,"15min":0.26800000000000002},"max":{"1min":7.2910000000000004,"5min":111.705,"15min":111.705},"last":0.77400000000000002}]},{"osd":7,"last update":"Tue Sep 10 19:26:22 2024","interfaces":[{"interface":"back","average":{"1min":1.405,"5min":1.4810000000000001,"15min":1.2410000000000001},"min":{"1min":0.29799999999999999,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":3.2799999999999998,"5min":6.7869999999999999,"15min":6.7869999999999999},"last":0.57099999999999995},{"interface":"front","average":{"1min":1.29,"5min":1.4390000000000001,"15min":1.2150000000000001},"min":{"1min":0.32800000000000001,"5min":0.29299999999999998,"15min":0.27100000000000002},"max":{"1min":4.0369999999999999,"5min":8.1679999999999993,"15min":9.4909999999999997},"last":1.1819999999999999}]}]},{"osd":4,"up_from":32,"seq":137438953693,"num_pgs":71,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2296956,"kb_used_data":1404400,"kb_used_omap":1,"kb_used_meta":892542,"kb_avail":91448196,"statfs":{"total":95995035648,"available":93642952704,"internally_reserved":0,"allocated":1438105600,"data_stored":1437796402,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":913963352},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,3,0,3],"upper_bound":16},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 19:26:25 2024","interfaces":[{"interface":"back","average":{"1min":4.008,"5min":2.6360000000000001,"15min":2.407},"min":{"1min":0.221,"5min":0.221,"15min":0.221},"max":{"1min":39.097000000000001,"5min":39.097000000000001,"15min":39.097000000000001},"last":1.288},{"interface":"front","average":{"1min":4.1109999999999998,"5min":2.9199999999999999,"15min":2.722},"min":{"1min":0.20499999999999999,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":39.112000000000002,"5min":39.112000000000002,"15min":39.112000000000002},"last":4.3579999999999997}]},{"osd":1,"last update":"Tue Sep 10 19:26:48 2024","interfaces":[{"interface":"back","average":{"1min":1.5229999999999999,"5min":1.0249999999999999,"15min":0.81100000000000005},"min":{"1min":0.20599999999999999,"5min":0.20599999999999999,"15min":0.20599999999999999},"max":{"1min":4.7960000000000003,"5min":4.7960000000000003,"15min":4.7960000000000003},"last":4.3129999999999997},{"interface":"front","average":{"1min":1.548,"5min":0.88100000000000001,"15min":0.69799999999999995},"min":{"1min":0.23400000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":5.7519999999999998,"5min":5.7519999999999998,"15min":5.7519999999999998},"last":0.39600000000000002}]},{"osd":2,"last update":"Tue Sep 10 19:26:21 2024","interfaces":[{"interface":"back","average":{"1min":1.6679999999999999,"5min":1.0189999999999999,"15min":0.91400000000000003},"min":{"1min":0.27000000000000002,"5min":0.25600000000000001,"15min":0.214},"max":{"1min":4.8410000000000002,"5min":4.8410000000000002,"15min":6.8869999999999996},"last":4.3360000000000003},{"interface":"front","average":{"1min":1.6839999999999999,"5min":1.032,"15min":0.90400000000000003},"min":{"1min":0.30199999999999999,"5min":0.25800000000000001,"15min":0.23400000000000001},"max":{"1min":5.0410000000000004,"5min":5.0410000000000004,"15min":7.085},"last":0.42499999999999999}]},{"osd":3,"last update":"Tue Sep 10 19:26:21 2024","interfaces":[{"interface":"back","average":{"1min":1.48,"5min":0.99299999999999999,"15min":1.1080000000000001},"min":{"1min":0.27800000000000002,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":5.0179999999999998,"5min":5.0179999999999998,"15min":35.451999999999998},"last":1.018},{"interface":"front","average":{"1min":1.7509999999999999,"5min":0.96399999999999997,"15min":1.1659999999999999},"min":{"1min":0.28599999999999998,"5min":0.246,"15min":0.246},"max":{"1min":5.8949999999999996,"5min":5.8949999999999996,"15min":32.880000000000003},"last":1.256}]},{"osd":5,"last update":"Tue Sep 10 19:26:40 2024","interfaces":[{"interface":"back","average":{"1min":1.8129999999999999,"5min":4.5030000000000001,"15min":2.4700000000000002},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.224},"max":{"1min":5.9859999999999998,"5min":227.726,"15min":227.726},"last":1.355},{"interface":"front","average":{"1min":2.254,"5min":4.6210000000000004,"15min":2.4340000000000002},"min":{"1min":0.27800000000000002,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":5.4859999999999998,"5min":227.70500000000001,"15min":227.70500000000001},"last":4.4619999999999997}]},{"osd":6,"last update":"Tue Sep 10 19:26:17 2024","interfaces":[{"interface":"back","average":{"1min":2.718,"5min":1.472,"15min":1.0980000000000001},"min":{"1min":0.47099999999999997,"5min":0.27800000000000002,"15min":0.253},"max":{"1min":8.1010000000000009,"5min":8.1010000000000009,"15min":8.1010000000000009},"last":4.2720000000000002},{"interface":"front","average":{"1min":3.484,"5min":1.5329999999999999,"15min":1.1359999999999999},"min":{"1min":0.26400000000000001,"5min":0.26400000000000001,"15min":0.247},"max":{"1min":16.815000000000001,"5min":16.815000000000001,"15min":16.815000000000001},"last":1.3069999999999999}]},{"osd":7,"last update":"Tue Sep 10 19:26:25 2024","interfaces":[{"interface":"back","average":{"1min":2.2000000000000002,"5min":1.46,"15min":1.284},"min":{"1min":0.33100000000000002,"5min":0.311,"15min":0.27700000000000002},"max":{"1min":16.853999999999999,"5min":16.853999999999999,"15min":23.081},"last":1.909},{"interface":"front","average":{"1min":1.913,"5min":1.504,"15min":1.278},"min":{"1min":0.32000000000000001,"5min":0.26100000000000001,"15min":0.23699999999999999},"max":{"1min":10.02,"5min":10.02,"15min":22.666},"last":1.0449999999999999}]}]},{"osd":3,"up_from":26,"seq":111669149922,"num_pgs":70,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1524304,"kb_used_data":624008,"kb_used_omap":1,"kb_used_meta":900286,"kb_avail":92220848,"statfs":{"total":95995035648,"available":94434148352,"internally_reserved":0,"allocated":638984192,"data_stored":638652881,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":921893208},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 19:26:25 2024","interfaces":[{"interface":"back","average":{"1min":0.52300000000000002,"5min":0.53100000000000003,"15min":0.53300000000000003},"min":{"1min":0.18099999999999999,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":1.802,"5min":1.802,"15min":1.802},"last":0.48099999999999998},{"interface":"front","average":{"1min":0.56799999999999995,"5min":0.51200000000000001,"15min":0.503},"min":{"1min":0.221,"5min":0.20100000000000001,"15min":0.20100000000000001},"max":{"1min":1.7889999999999999,"5min":1.7889999999999999,"15min":1.7889999999999999},"last":0.56000000000000005}]},{"osd":1,"last update":"Tue Sep 10 19:26:55 2024","interfaces":[{"interface":"back","average":{"1min":0.41599999999999998,"5min":0.375,"15min":0.32100000000000001},"min":{"1min":0.185,"5min":0.185,"15min":0.185},"max":{"1min":1.1599999999999999,"5min":2.5939999999999999,"15min":2.5939999999999999},"last":0.307},{"interface":"front","average":{"1min":0.39000000000000001,"5min":0.36199999999999999,"15min":0.33000000000000002},"min":{"1min":0.19500000000000001,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":1.046,"5min":1.3720000000000001,"15min":1.3720000000000001},"last":0.25900000000000001}]},{"osd":2,"last update":"Tue Sep 10 19:26:03 2024","interfaces":[{"interface":"back","average":{"1min":0.41499999999999998,"5min":0.37,"15min":0.63600000000000001},"min":{"1min":0.25700000000000001,"5min":0.17599999999999999,"15min":0.17599999999999999},"max":{"1min":0.76500000000000001,"5min":2.2530000000000001,"15min":5.3029999999999999},"last":0.41099999999999998},{"interface":"front","average":{"1min":0.65500000000000003,"5min":0.42799999999999999,"15min":0.71799999999999997},"min":{"1min":0.29599999999999999,"5min":0.16400000000000001,"15min":0.16400000000000001},"max":{"1min":2.944,"5min":2.944,"15min":17.806000000000001},"last":0.32500000000000001}]},{"osd":4,"last update":"Tue Sep 10 19:26:18 2024","interfaces":[{"interface":"back","average":{"1min":0.71699999999999997,"5min":1.9650000000000001,"15min":1.3680000000000001},"min":{"1min":0.28799999999999998,"5min":0.26900000000000002,"15min":0.23599999999999999},"max":{"1min":1.649,"5min":81.334000000000003,"15min":81.334000000000003},"last":1.671},{"interface":"front","average":{"1min":1.536,"5min":2.2349999999999999,"15min":1.5820000000000001},"min":{"1min":0.27900000000000003,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":3.6789999999999998,"5min":81.381,"15min":81.381},"last":0.71799999999999997}]},{"osd":5,"last update":"Tue Sep 10 19:26:41 2024","interfaces":[{"interface":"back","average":{"1min":1.5860000000000001,"5min":2.0659999999999998,"15min":1.724},"min":{"1min":0.35399999999999998,"5min":0.28100000000000003,"15min":0.27900000000000003},"max":{"1min":6.7240000000000002,"5min":104.672,"15min":104.672},"last":1.4279999999999999},{"interface":"front","average":{"1min":1.137,"5min":1.9139999999999999,"15min":1.7849999999999999},"min":{"1min":0.32200000000000001,"5min":0.24299999999999999,"15min":0.22},"max":{"1min":4.1159999999999997,"5min":105.971,"15min":105.971},"last":2.3599999999999999}]},{"osd":6,"last update":"Tue Sep 10 19:26:03 2024","interfaces":[{"interface":"back","average":{"1min":1.282,"5min":2.5049999999999999,"15min":1.615},"min":{"1min":0.40400000000000003,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":5.2480000000000002,"5min":116.14,"15min":116.14},"last":0.503},{"interface":"front","average":{"1min":1.1890000000000001,"5min":2.4900000000000002,"15min":1.4930000000000001},"min":{"1min":0.42099999999999999,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":4.008,"5min":116.095,"15min":116.095},"last":4.4370000000000003}]},{"osd":7,"last update":"Tue Sep 10 19:26:35 2024","interfaces":[{"interface":"back","average":{"1min":1.0880000000000001,"5min":1.9590000000000001,"15min":1.4259999999999999},"min":{"1min":0.34799999999999998,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":4.4640000000000004,"5min":104.749,"15min":104.749},"last":1.0329999999999999},{"interface":"front","average":{"1min":0.90500000000000003,"5min":1.8009999999999999,"15min":1.431},"min":{"1min":0.33100000000000002,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":2.5219999999999998,"5min":104.69499999999999,"15min":104.69499999999999},"last":0.53000000000000003}]}]},{"osd":2,"up_from":0,"seq":77309411559,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":276,"seq":1185410973749,"num_pgs":63,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":669976,"kb_used_data":359884,"kb_used_omap":6,"kb_used_meta":310009,"kb_avail":93075176,"statfs":{"total":95995035648,"available":95308980224,"internally_reserved":0,"allocated":368521216,"data_stored":368192566,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6335,"internal_metadata":317450049},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 19:26:29 2024","interfaces":[{"interface":"back","average":{"1min":0.55000000000000004,"5min":0.499,"15min":0.49099999999999999},"min":{"1min":0.24199999999999999,"5min":0.182,"15min":0.182},"max":{"1min":1.4550000000000001,"5min":1.8160000000000001,"15min":1.8160000000000001},"last":0.32400000000000001},{"interface":"front","average":{"1min":0.58299999999999996,"5min":0.443,"15min":0.41899999999999998},"min":{"1min":0.214,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":1.2450000000000001,"5min":1.387,"15min":1.387},"last":0.27300000000000002}]},{"osd":2,"last update":"Tue Sep 10 19:26:53 2024","interfaces":[{"interface":"back","average":{"1min":0.51800000000000002,"5min":0.40200000000000002,"15min":0.32600000000000001},"min":{"1min":0.23499999999999999,"5min":0.19,"15min":0.19},"max":{"1min":1.298,"5min":1.837,"15min":1.837},"last":0.373},{"interface":"front","average":{"1min":0.49299999999999999,"5min":0.41899999999999998,"15min":0.36299999999999999},"min":{"1min":0.255,"5min":0.222,"15min":0.222},"max":{"1min":1.4730000000000001,"5min":1.7909999999999999,"15min":1.7909999999999999},"last":0.34000000000000002}]},{"osd":3,"last update":"Tue Sep 10 19:26:53 2024","interfaces":[{"interface":"back","average":{"1min":0.57499999999999996,"5min":0.38700000000000001,"15min":0.33100000000000002},"min":{"1min":0.23000000000000001,"5min":0.221,"15min":0.221},"max":{"1min":1.573,"5min":1.573,"15min":1.573},"last":0.23899999999999999},{"interface":"front","average":{"1min":0.47499999999999998,"5min":0.35899999999999999,"15min":0.32000000000000001},"min":{"1min":0.217,"5min":0.17599999999999999,"15min":0.17599999999999999},"max":{"1min":1.984,"5min":1.984,"15min":1.984},"last":0.311}]},{"osd":4,"last update":"Tue Sep 10 19:26:53 2024","interfaces":[{"interface":"back","average":{"1min":1.605,"5min":0.871,"15min":0.64500000000000002},"min":{"1min":0.27500000000000002,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":4.0190000000000001,"5min":4.0190000000000001,"15min":4.0190000000000001},"last":0.40100000000000002},{"interface":"front","average":{"1min":1.619,"5min":0.92900000000000005,"15min":0.71999999999999997},"min":{"1min":0.43099999999999999,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":5.7599999999999998,"5min":5.7599999999999998,"15min":5.7599999999999998},"last":0.41399999999999998}]},{"osd":5,"last update":"Tue Sep 10 19:26:53 2024","interfaces":[{"interface":"back","average":{"1min":1.464,"5min":0.95699999999999996,"15min":0.98099999999999998},"min":{"1min":0.32300000000000001,"5min":0.307,"15min":0.307},"max":{"1min":4.1529999999999996,"5min":4.7519999999999998,"15min":4.7519999999999998},"last":0.67100000000000004},{"interface":"front","average":{"1min":1.8280000000000001,"5min":1.23,"15min":1.365},"min":{"1min":0.39200000000000002,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":4.181,"5min":5.617,"15min":5.617},"last":0.65700000000000003}]},{"osd":6,"last update":"Tue Sep 10 19:26:53 2024","interfaces":[{"interface":"back","average":{"1min":1.339,"5min":1.671,"15min":2.1190000000000002},"min":{"1min":0.33000000000000002,"5min":0.31,"15min":0.31},"max":{"1min":3.1000000000000001,"5min":27.379999999999999,"15min":27.379999999999999},"last":0.433},{"interface":"front","average":{"1min":1.169,"5min":1.615,"15min":2.0990000000000002},"min":{"1min":0.30299999999999999,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":2.8180000000000001,"5min":26.812999999999999,"15min":26.812999999999999},"last":0.35599999999999998}]},{"osd":7,"last update":"Tue Sep 10 19:26:53 2024","interfaces":[{"interface":"back","average":{"1min":1.2370000000000001,"5min":0.996,"15min":1.022},"min":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":3.4249999999999998,"5min":5.6260000000000003,"15min":5.6260000000000003},"last":0.56399999999999995},{"interface":"front","average":{"1min":1.26,"5min":1.0029999999999999,"15min":1.0309999999999999},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":3.4430000000000001,"5min":5.6050000000000004,"15min":5.6050000000000004},"last":0.55300000000000005}]}]},{"osd":0,"up_from":301,"seq":1292785156129,"num_pgs":77,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3071508,"kb_used_data":2700552,"kb_used_omap":4,"kb_used_meta":370811,"kb_avail":90673644,"statfs":{"total":95995035648,"available":92849811456,"internally_reserved":0,"allocated":2765365248,"data_stored":2765041119,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":379710982},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Sep 10 19:26:29 2024","interfaces":[{"interface":"back","average":{"1min":0.38700000000000001,"5min":0.53600000000000003,"15min":0.56100000000000005},"min":{"1min":0.17499999999999999,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":0.72599999999999998,"5min":1.5269999999999999,"15min":1.5269999999999999},"last":0.78200000000000003},{"interface":"front","average":{"1min":0.34999999999999998,"5min":0.46500000000000002,"15min":0.48399999999999999},"min":{"1min":0.16900000000000001,"5min":0.16900000000000001,"15min":0.16900000000000001},"max":{"1min":0.81200000000000006,"5min":1.216,"15min":1.216},"last":0.47299999999999998}]},{"osd":2,"last update":"Tue Sep 10 19:26:29 2024","interfaces":[{"interface":"back","average":{"1min":0.46999999999999997,"5min":0.57599999999999996,"15min":0.59399999999999997},"min":{"1min":0.20499999999999999,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":1.244,"5min":1.734,"15min":1.734},"last":1.1759999999999999},{"interface":"front","average":{"1min":0.47399999999999998,"5min":0.60199999999999998,"15min":0.624},"min":{"1min":0.22600000000000001,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":0.82699999999999996,"5min":1.5580000000000001,"15min":1.5580000000000001},"last":0.81399999999999995}]},{"osd":3,"last update":"Tue Sep 10 19:26:29 2024","interfaces":[{"interface":"back","average":{"1min":0.42899999999999999,"5min":0.55700000000000005,"15min":0.57799999999999996},"min":{"1min":0.20599999999999999,"5min":0.20599999999999999,"15min":0.20599999999999999},"max":{"1min":0.77700000000000002,"5min":1.1910000000000001,"15min":1.1910000000000001},"last":0.58399999999999996},{"interface":"front","average":{"1min":0.58299999999999996,"5min":0.61499999999999999,"15min":0.621},"min":{"1min":0.23499999999999999,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":1.4950000000000001,"5min":1.677,"15min":1.677},"last":1.2090000000000001}]},{"osd":4,"last update":"Tue Sep 10 19:26:29 2024","interfaces":[{"interface":"back","average":{"1min":1.8260000000000001,"5min":3.968,"15min":4.3250000000000002},"min":{"1min":0.25800000000000001,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":6.5640000000000001,"5min":53.670000000000002,"15min":53.670000000000002},"last":1.8680000000000001},{"interface":"front","average":{"1min":1.2729999999999999,"5min":3.8639999999999999,"15min":4.2960000000000003},"min":{"1min":0.314,"5min":0.314,"15min":0.314},"max":{"1min":6.5019999999999998,"5min":53.935000000000002,"15min":53.935000000000002},"last":2.0009999999999999}]},{"osd":5,"last update":"Tue Sep 10 19:26:29 2024","interfaces":[{"interface":"back","average":{"1min":0.90200000000000002,"5min":3.4449999999999998,"15min":3.8690000000000002},"min":{"1min":0.28599999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":2.9550000000000001,"5min":53.884999999999998,"15min":53.884999999999998},"last":1.3100000000000001},{"interface":"front","average":{"1min":1.0609999999999999,"5min":3.3610000000000002,"15min":3.7440000000000002},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":3.1200000000000001,"5min":53.875999999999998,"15min":53.875999999999998},"last":1.905}]},{"osd":6,"last update":"Tue Sep 10 19:26:29 2024","interfaces":[{"interface":"back","average":{"1min":1.165,"5min":4.0960000000000001,"15min":4.5839999999999996},"min":{"1min":0.36799999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":2.8919999999999999,"5min":56.901000000000003,"15min":56.901000000000003},"last":0.61199999999999999},{"interface":"front","average":{"1min":1.1200000000000001,"5min":4.024,"15min":4.508},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":3.1669999999999998,"5min":55.177,"15min":55.177},"last":1.2849999999999999}]},{"osd":7,"last update":"Tue Sep 10 19:26:29 2024","interfaces":[{"interface":"back","average":{"1min":1.288,"5min":1.2470000000000001,"15min":1.24},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":3.8260000000000001,"5min":3.8260000000000001,"15min":3.8260000000000001},"last":1.3500000000000001},{"interface":"front","average":{"1min":1.079,"5min":1.222,"15min":1.246},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":3.2069999999999999,"5min":3.2519999999999998,"15min":3.2519999999999998},"last":1.0389999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":966,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1236,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1236,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":362,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1540,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":2118,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1018,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1502,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1185353728,"data_stored":1185349656,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":355663872,"data_stored":355663872,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":631308288,"data_stored":631308288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1415516160,"data_stored":1415512088,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":875495424,"data_stored":875495424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":911151104,"data_stored":911147032,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":633208832,"data_stored":633208832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T19:27:07.209 INFO:tasks.cephadm.ceph_manager.ceph:recovered! 2024-09-10T19:27:07.209 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-10T19:27:07.209 INFO:tasks.thrashosds.thrasher:Removing osd 6, in_osds are: [0, 3, 4, 5, 6, 7, 1] 2024-09-10T19:27:07.210 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph osd out 6 2024-09-10T19:27:07.293 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:27:07.328 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:27:07.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:07.329 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:27:07.430 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:27:07.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:27:07 smithi183 ceph-mon[29736]: from='client.15945 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-10T19:27:07.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:27:07 smithi183 ceph-mon[29736]: pgmap v777: 158 pgs: 158 active+clean; 1.6 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 1.46k op/s 2024-09-10T19:27:07.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:27:07 smithi183 ceph-mon[29736]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T19:27:07.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:27:07 smithi183 ceph-mon[29736]: from='client.? 172.21.15.183:0/3463444277' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-10T19:27:07.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:27:07 smithi183 ceph-mon[29736]: osdmap e357: 8 total, 8 up, 7 in 2024-09-10T19:27:07.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:27:07 smithi183 ceph-mon[41562]: from='client.15945 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-10T19:27:07.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:27:07 smithi183 ceph-mon[41562]: pgmap v777: 158 pgs: 158 active+clean; 1.6 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 1.46k op/s 2024-09-10T19:27:07.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:27:07 smithi183 ceph-mon[41562]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T19:27:07.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:27:07 smithi183 ceph-mon[41562]: from='client.? 172.21.15.183:0/3463444277' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-10T19:27:07.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:27:07 smithi183 ceph-mon[41562]: osdmap e357: 8 total, 8 up, 7 in 2024-09-10T19:27:07.464 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:27:07.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:07.466 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:27:07.566 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:27:07.603 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:27:07.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:07.604 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:27:07.705 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:27:07.716 INFO:journalctl@ceph.mgr.x.smithi191.stdout:Sep 10 19:27:07 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-x[34967]: ::ffff:172.21.15.191 - - [10/Sep/2024:19:27:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T19:27:07.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:27:07 smithi191 ceph-mon[32884]: from='client.15945 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-10T19:27:07.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:27:07 smithi191 ceph-mon[32884]: pgmap v777: 158 pgs: 158 active+clean; 1.6 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 1.46k op/s 2024-09-10T19:27:07.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:27:07 smithi191 ceph-mon[32884]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T19:27:07.717 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:27:07 smithi191 ceph-mon[32884]: from='client.? 172.21.15.183:0/3463444277' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-10T19:27:07.717 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:27:07 smithi191 ceph-mon[32884]: osdmap e357: 8 total, 8 up, 7 in 2024-09-10T19:27:07.746 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:27:07.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:07.747 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:27:07.848 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:27:07.876 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:27:07.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:07.877 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:27:07.978 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:27:08.015 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:27:08.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:08.016 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:27:08.117 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:27:08.157 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:27:08.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:08.158 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:27:08.259 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:27:08.287 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:27:08.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:08.288 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:27:08.389 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:27:08.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:08.427 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:27:08.428 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:27:08.528 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:27:08.538 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:27:08 smithi183 ceph-mon[29736]: 5.1d scrub starts 2024-09-10T19:27:08.538 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:27:08 smithi183 ceph-mon[29736]: 5.1d scrub ok 2024-09-10T19:27:08.538 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:27:08 smithi183 ceph-mon[29736]: 3.1 scrub starts 2024-09-10T19:27:08.538 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:27:08 smithi183 ceph-mon[29736]: 3.1 scrub ok 2024-09-10T19:27:08.538 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:27:08 smithi183 ceph-mon[29736]: 6.2 scrub starts 2024-09-10T19:27:08.538 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:27:08 smithi183 ceph-mon[29736]: 6.2 scrub ok 2024-09-10T19:27:08.538 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:27:08 smithi183 ceph-mon[41562]: 5.1d scrub starts 2024-09-10T19:27:08.538 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:27:08 smithi183 ceph-mon[41562]: 5.1d scrub ok 2024-09-10T19:27:08.538 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:27:08 smithi183 ceph-mon[41562]: 3.1 scrub starts 2024-09-10T19:27:08.539 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:27:08 smithi183 ceph-mon[41562]: 3.1 scrub ok 2024-09-10T19:27:08.539 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:27:08 smithi183 ceph-mon[41562]: 6.2 scrub starts 2024-09-10T19:27:08.539 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:27:08 smithi183 ceph-mon[41562]: 6.2 scrub ok 2024-09-10T19:27:08.568 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:27:08.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:08.569 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:27:08.670 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:27:08.696 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:27:08.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:08.697 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:27:08.715 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:27:08 smithi191 ceph-mon[32884]: 5.1d scrub starts 2024-09-10T19:27:08.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:27:08 smithi191 ceph-mon[32884]: 5.1d scrub ok 2024-09-10T19:27:08.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:27:08 smithi191 ceph-mon[32884]: 3.1 scrub starts 2024-09-10T19:27:08.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:27:08 smithi191 ceph-mon[32884]: 3.1 scrub ok 2024-09-10T19:27:08.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:27:08 smithi191 ceph-mon[32884]: 6.2 scrub starts 2024-09-10T19:27:08.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:27:08 smithi191 ceph-mon[32884]: 6.2 scrub ok 2024-09-10T19:27:08.798 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:27:08.824 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:27:08.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:08.825 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:27:08.926 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:27:08.960 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:27:08.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:08.961 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:27:08.973 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:27:09.062 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:27:09.098 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:27:09.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:09.099 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:27:09.200 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:27:09.237 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:27:09.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:09.238 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:27:09.291 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:27:09 smithi183 ceph-mon[41562]: pgmap v779: 158 pgs: 158 active+clean; 1.5 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.35k op/s 2024-09-10T19:27:09.291 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:27:09 smithi183 ceph-mon[29736]: pgmap v779: 158 pgs: 158 active+clean; 1.5 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.35k op/s 2024-09-10T19:27:09.291 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:27:09 smithi183 ceph-mon[29736]: 5.1f scrub starts 2024-09-10T19:27:09.291 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:27:09 smithi183 ceph-mon[29736]: 5.1f scrub ok 2024-09-10T19:27:09.291 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:27:09 smithi183 ceph-mon[29736]: 6.0 scrub starts 2024-09-10T19:27:09.291 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:27:09 smithi183 ceph-mon[29736]: 6.0 scrub ok 2024-09-10T19:27:09.294 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:27:09 smithi183 ceph-mon[41562]: 5.1f scrub starts 2024-09-10T19:27:09.294 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:27:09 smithi183 ceph-mon[41562]: 5.1f scrub ok 2024-09-10T19:27:09.294 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:27:09 smithi183 ceph-mon[41562]: 6.0 scrub starts 2024-09-10T19:27:09.295 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:27:09 smithi183 ceph-mon[41562]: 6.0 scrub ok 2024-09-10T19:27:09.339 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:27:09.377 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:27:09.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:09.378 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:27:09.480 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:27:09.515 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:27:09.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:09.516 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:27:09.617 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:27:09.656 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:27:09.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:09.657 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:27:09.710 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-10T19:27:09.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:27:09 smithi191 ceph-mon[32884]: pgmap v779: 158 pgs: 158 active+clean; 1.5 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.35k op/s 2024-09-10T19:27:09.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:27:09 smithi191 ceph-mon[32884]: 5.1f scrub starts 2024-09-10T19:27:09.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:27:09 smithi191 ceph-mon[32884]: 5.1f scrub ok 2024-09-10T19:27:09.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:27:09 smithi191 ceph-mon[32884]: 6.0 scrub starts 2024-09-10T19:27:09.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:27:09 smithi191 ceph-mon[32884]: 6.0 scrub ok 2024-09-10T19:27:09.758 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:27:09.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:09.786 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:27:09.787 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:27:09.887 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:27:09.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:09.940 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:27:09.940 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:27:10.041 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:27:10.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:10.075 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:27:10.075 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:27:10.176 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:27:10.203 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:27:10.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:10.205 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:27:10.215 INFO:journalctl@ceph.iscsi.iscsi.a.smithi191.stdout:Sep 10 19:27:09 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-iscsi-iscsi-a[61816]: debug there is no tcmu-runner data available 2024-09-10T19:27:10.305 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:27:10.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:10.347 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:27:10.347 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:27:10.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:27:10 smithi183 ceph-mon[29736]: 10.e scrub starts 2024-09-10T19:27:10.431 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:27:10 smithi183 ceph-mon[41562]: 10.e scrub starts 2024-09-10T19:27:10.448 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:27:10.474 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:27:10.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:10.475 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:27:10.576 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:27:10.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:10.614 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:27:10.614 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:27:10.715 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:27:10.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:27:10 smithi191 ceph-mon[32884]: 10.e scrub starts 2024-09-10T19:27:10.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:10.748 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:27:10.748 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:27:10.849 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:27:10.887 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:27:10.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:10.888 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:27:10.989 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:27:11.025 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:27:11.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:11.026 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:27:11.126 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:27:11.154 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:27:11.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:11.155 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:27:11.256 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:27:11.288 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:27:11.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:11.289 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:27:11.390 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:27:11.428 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:27:11.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:11.429 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:27:11.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:27:11 smithi183 ceph-mon[29736]: from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:27:11.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:27:11 smithi183 ceph-mon[29736]: pgmap v780: 158 pgs: 158 active+clean; 1.3 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.26k op/s 2024-09-10T19:27:11.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:27:11 smithi183 ceph-mon[29736]: 2.4 scrub starts 2024-09-10T19:27:11.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:27:11 smithi183 ceph-mon[29736]: 2.4 scrub ok 2024-09-10T19:27:11.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:27:11 smithi183 ceph-mon[29736]: 10.e scrub ok 2024-09-10T19:27:11.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:27:11 smithi183 ceph-mon[29736]: 5.d scrub starts 2024-09-10T19:27:11.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:27:11 smithi183 ceph-mon[29736]: 5.d scrub ok 2024-09-10T19:27:11.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:27:11 smithi183 ceph-mon[41562]: from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:27:11.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:27:11 smithi183 ceph-mon[41562]: pgmap v780: 158 pgs: 158 active+clean; 1.3 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.26k op/s 2024-09-10T19:27:11.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:27:11 smithi183 ceph-mon[41562]: 2.4 scrub starts 2024-09-10T19:27:11.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:27:11 smithi183 ceph-mon[41562]: 2.4 scrub ok 2024-09-10T19:27:11.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:27:11 smithi183 ceph-mon[41562]: 10.e scrub ok 2024-09-10T19:27:11.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:27:11 smithi183 ceph-mon[41562]: 5.d scrub starts 2024-09-10T19:27:11.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:27:11 smithi183 ceph-mon[41562]: 5.d scrub ok 2024-09-10T19:27:11.433 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Sep 10 19:27:11 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-y[29989]: ::ffff:172.21.15.191 - - [10/Sep/2024:19:27:11] "GET /metrics HTTP/1.1" 200 224779 "" "Prometheus/2.43.0" 2024-09-10T19:27:11.530 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:27:11.554 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:27:11.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:11.555 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:27:11.656 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:27:11.694 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:27:11.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:11.695 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:27:11.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:27:11 smithi191 ceph-mon[32884]: from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:27:11.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:27:11 smithi191 ceph-mon[32884]: pgmap v780: 158 pgs: 158 active+clean; 1.3 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.26k op/s 2024-09-10T19:27:11.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:27:11 smithi191 ceph-mon[32884]: 2.4 scrub starts 2024-09-10T19:27:11.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:27:11 smithi191 ceph-mon[32884]: 2.4 scrub ok 2024-09-10T19:27:11.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:27:11 smithi191 ceph-mon[32884]: 10.e scrub ok 2024-09-10T19:27:11.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:27:11 smithi191 ceph-mon[32884]: 5.d scrub starts 2024-09-10T19:27:11.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:27:11 smithi191 ceph-mon[32884]: 5.d scrub ok 2024-09-10T19:27:11.796 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:27:11.823 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:27:11.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:11.824 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:27:11.925 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:27:11.962 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:27:11.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:11.963 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:27:12.064 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:27:12.092 INFO:teuthology.orchestra.run.smithi183.stdout:osd.0: {} 2024-09-10T19:27:12.092 INFO:teuthology.orchestra.run.smithi183.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-10T19:27:12.100 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:27:12.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:12.101 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:27:12.116 INFO:teuthology.orchestra.run.smithi183.stdout:osd.1: {} 2024-09-10T19:27:12.117 INFO:teuthology.orchestra.run.smithi183.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-10T19:27:12.143 INFO:teuthology.orchestra.run.smithi183.stdout:osd.2: {} 2024-09-10T19:27:12.144 INFO:teuthology.orchestra.run.smithi183.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-10T19:27:12.166 INFO:teuthology.orchestra.run.smithi183.stdout:osd.3: {} 2024-09-10T19:27:12.166 INFO:teuthology.orchestra.run.smithi183.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-10T19:27:12.201 INFO:teuthology.orchestra.run.smithi183.stdout:osd.4: {} 2024-09-10T19:27:12.201 INFO:teuthology.orchestra.run.smithi183.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-10T19:27:12.202 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:27:12.229 INFO:teuthology.orchestra.run.smithi183.stdout:osd.5: {} 2024-09-10T19:27:12.229 INFO:teuthology.orchestra.run.smithi183.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-10T19:27:12.245 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:27:12.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:12.246 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:27:12.258 INFO:teuthology.orchestra.run.smithi183.stdout:osd.6: {} 2024-09-10T19:27:12.258 INFO:teuthology.orchestra.run.smithi183.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-10T19:27:12.286 INFO:teuthology.orchestra.run.smithi183.stdout:osd.7: {} 2024-09-10T19:27:12.286 INFO:teuthology.orchestra.run.smithi183.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-10T19:27:12.348 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:27:12.384 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:27:12.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:12.385 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:27:12.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:27:12 smithi183 ceph-mon[29736]: 3.0 scrub starts 2024-09-10T19:27:12.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:27:12 smithi183 ceph-mon[29736]: 3.0 scrub ok 2024-09-10T19:27:12.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:27:12 smithi183 ceph-mon[29736]: pgmap v781: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 1.2 GiB data, 9.9 GiB used, 616 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.34k op/s 2024-09-10T19:27:12.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:27:12 smithi183 ceph-mon[29736]: from='client.? 172.21.15.183:0/1444558770' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T19:27:12.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:27:12 smithi183 ceph-mon[41562]: 3.0 scrub starts 2024-09-10T19:27:12.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:27:12 smithi183 ceph-mon[41562]: 3.0 scrub ok 2024-09-10T19:27:12.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:27:12 smithi183 ceph-mon[41562]: pgmap v781: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 1.2 GiB data, 9.9 GiB used, 616 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.34k op/s 2024-09-10T19:27:12.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:27:12 smithi183 ceph-mon[41562]: from='client.? 172.21.15.183:0/1444558770' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T19:27:12.486 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:27:12.514 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:27:12.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:12.515 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:27:12.616 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:27:12.642 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:27:12.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:12.643 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:27:12.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:27:12 smithi191 ceph-mon[32884]: 3.0 scrub starts 2024-09-10T19:27:12.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:27:12 smithi191 ceph-mon[32884]: 3.0 scrub ok 2024-09-10T19:27:12.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:27:12 smithi191 ceph-mon[32884]: pgmap v781: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 1.2 GiB data, 9.9 GiB used, 616 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.34k op/s 2024-09-10T19:27:12.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:27:12 smithi191 ceph-mon[32884]: from='client.? 172.21.15.183:0/1444558770' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T19:27:12.745 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:27:12.780 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:27:12.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:12.781 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:27:12.840 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:27:12.881 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:27:12.920 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:27:12.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:12.921 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:27:13.022 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:27:13.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:13.057 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:27:13.057 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:27:13.158 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:27:13.186 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:27:13.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:13.187 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:27:13.234 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:27:13 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mon-a[29713]: 2024-09-10T19:27:13.043+0000 7f9f28e29640 -1 mon.a@0(leader).osd e357 definitely_dead 0 2024-09-10T19:27:13.288 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:27:13.330 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:27:13.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:13.332 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:27:13.335 INFO:teuthology.orchestra.run.smithi183.stderr:marked out osd.6. 2024-09-10T19:27:13.433 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:27:13.455 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:27:13.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:13.456 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:27:13.556 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:27:13.577 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:27:13.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:13.578 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:27:13.679 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:27:13.681 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:27:13 smithi183 ceph-mon[29736]: 4.0 scrub starts 2024-09-10T19:27:13.681 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:27:13 smithi183 ceph-mon[29736]: 4.0 scrub ok 2024-09-10T19:27:13.681 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:27:13 smithi183 ceph-mon[29736]: from='client.? 172.21.15.183:0/3515820347' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["6"]}]: dispatch 2024-09-10T19:27:13.681 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:27:13 smithi183 ceph-mon[29736]: Client client.admin marked osd.6 out, while it was still marked up 2024-09-10T19:27:13.681 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:27:13 smithi183 ceph-mon[41562]: 4.0 scrub starts 2024-09-10T19:27:13.682 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:27:13 smithi183 ceph-mon[41562]: 4.0 scrub ok 2024-09-10T19:27:13.682 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:27:13 smithi183 ceph-mon[41562]: from='client.? 172.21.15.183:0/3515820347' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["6"]}]: dispatch 2024-09-10T19:27:13.682 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:27:13 smithi183 ceph-mon[41562]: Client client.admin marked osd.6 out, while it was still marked up 2024-09-10T19:27:13.701 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:27:13.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:13.702 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:27:13.715 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:27:13 smithi191 ceph-mon[32884]: 4.0 scrub starts 2024-09-10T19:27:13.715 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:27:13 smithi191 ceph-mon[32884]: 4.0 scrub ok 2024-09-10T19:27:13.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:27:13 smithi191 ceph-mon[32884]: from='client.? 172.21.15.183:0/3515820347' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["6"]}]: dispatch 2024-09-10T19:27:13.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:27:13 smithi191 ceph-mon[32884]: Client client.admin marked osd.6 out, while it was still marked up 2024-09-10T19:27:13.802 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:27:13.823 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:27:13.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:13.824 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:27:13.925 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:27:13.947 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:27:13.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:13.948 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:27:14.048 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:27:14.070 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:27:14.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:14.071 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:27:14.172 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:27:14.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:14.194 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:27:14.194 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:27:14.295 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:27:14.316 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:27:14.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:14.317 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:27:14.418 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:27:14.439 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:27:14.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:14.440 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:27:14.500 INFO:teuthology.orchestra.run.smithi183.stdout:ERROR: (22) Invalid argument 2024-09-10T19:27:14.501 INFO:teuthology.orchestra.run.smithi183.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-10T19:27:14.501 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:27:14 smithi183 ceph-mon[41562]: purged_snaps scrub starts 2024-09-10T19:27:14.501 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:27:14 smithi183 ceph-mon[41562]: purged_snaps scrub ok 2024-09-10T19:27:14.501 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:27:14 smithi183 ceph-mon[41562]: from='client.? 172.21.15.183:0/3515820347' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["6"]}]': finished 2024-09-10T19:27:14.501 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:27:14 smithi183 ceph-mon[41562]: osdmap e358: 8 total, 8 up, 6 in 2024-09-10T19:27:14.501 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:27:14 smithi183 ceph-mon[41562]: 2.13 scrub starts 2024-09-10T19:27:14.502 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:27:14 smithi183 ceph-mon[41562]: 2.13 scrub ok 2024-09-10T19:27:14.502 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:27:14 smithi183 ceph-mon[41562]: 1.9 scrub starts 2024-09-10T19:27:14.502 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:27:14 smithi183 ceph-mon[41562]: 1.9 scrub ok 2024-09-10T19:27:14.502 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:27:14 smithi183 ceph-mon[41562]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:27:14.502 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:27:14 smithi183 ceph-mon[41562]: pgmap v783: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 1.1 GiB data, 8.2 GiB used, 528 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.04k op/s 2024-09-10T19:27:14.502 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:27:14 smithi183 ceph-mon[29736]: purged_snaps scrub starts 2024-09-10T19:27:14.502 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:27:14 smithi183 ceph-mon[29736]: purged_snaps scrub ok 2024-09-10T19:27:14.502 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:27:14 smithi183 ceph-mon[29736]: from='client.? 172.21.15.183:0/3515820347' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["6"]}]': finished 2024-09-10T19:27:14.502 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:27:14 smithi183 ceph-mon[29736]: osdmap e358: 8 total, 8 up, 6 in 2024-09-10T19:27:14.502 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:27:14 smithi183 ceph-mon[29736]: 2.13 scrub starts 2024-09-10T19:27:14.502 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:27:14 smithi183 ceph-mon[29736]: 2.13 scrub ok 2024-09-10T19:27:14.502 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:27:14 smithi183 ceph-mon[29736]: 1.9 scrub starts 2024-09-10T19:27:14.503 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:27:14 smithi183 ceph-mon[29736]: 1.9 scrub ok 2024-09-10T19:27:14.503 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:27:14 smithi183 ceph-mon[29736]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:27:14.503 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:27:14 smithi183 ceph-mon[29736]: pgmap v783: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 1.1 GiB data, 8.2 GiB used, 528 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.04k op/s 2024-09-10T19:27:14.540 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:27:14.561 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:27:14.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:14.562 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:27:14.663 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:27:14.685 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:27:14.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:14.686 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:27:14.715 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:27:14 smithi191 ceph-mon[32884]: purged_snaps scrub starts 2024-09-10T19:27:14.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:27:14 smithi191 ceph-mon[32884]: purged_snaps scrub ok 2024-09-10T19:27:14.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:27:14 smithi191 ceph-mon[32884]: from='client.? 172.21.15.183:0/3515820347' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["6"]}]': finished 2024-09-10T19:27:14.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:27:14 smithi191 ceph-mon[32884]: osdmap e358: 8 total, 8 up, 6 in 2024-09-10T19:27:14.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:27:14 smithi191 ceph-mon[32884]: 2.13 scrub starts 2024-09-10T19:27:14.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:27:14 smithi191 ceph-mon[32884]: 2.13 scrub ok 2024-09-10T19:27:14.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:27:14 smithi191 ceph-mon[32884]: 1.9 scrub starts 2024-09-10T19:27:14.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:27:14 smithi191 ceph-mon[32884]: 1.9 scrub ok 2024-09-10T19:27:14.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:27:14 smithi191 ceph-mon[32884]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:27:14.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:27:14 smithi191 ceph-mon[32884]: pgmap v783: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 1.1 GiB data, 8.2 GiB used, 528 GiB / 536 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.04k op/s 2024-09-10T19:27:14.788 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:27:14.810 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:27:14.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:14.811 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:27:14.912 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:27:14.935 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:27:14.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:14.936 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:27:15.037 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:27:15.059 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:27:15.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:15.060 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:27:15.161 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:27:15.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:15.184 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:27:15.185 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:27:15.285 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:27:15.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:15.310 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:27:15.310 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:27:15.411 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:27:15.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:15.447 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:27:15.447 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:27:15.548 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:27:15.569 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:27:15.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:15.570 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:27:15.671 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:27:15.681 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:27:15 smithi183 ceph-mon[29736]: 4.17 scrub starts 2024-09-10T19:27:15.681 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:27:15 smithi183 ceph-mon[29736]: 4.17 scrub ok 2024-09-10T19:27:15.681 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:27:15 smithi183 ceph-mon[29736]: osdmap e359: 8 total, 8 up, 6 in 2024-09-10T19:27:15.681 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:27:15 smithi183 ceph-mon[29736]: 2.8 scrub starts 2024-09-10T19:27:15.681 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:27:15 smithi183 ceph-mon[29736]: 2.8 scrub ok 2024-09-10T19:27:15.681 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:27:15 smithi183 ceph-mon[29736]: osdmap e360: 8 total, 8 up, 6 in 2024-09-10T19:27:15.682 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:27:15 smithi183 ceph-mon[41562]: 4.17 scrub starts 2024-09-10T19:27:15.682 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:27:15 smithi183 ceph-mon[41562]: 4.17 scrub ok 2024-09-10T19:27:15.682 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:27:15 smithi183 ceph-mon[41562]: osdmap e359: 8 total, 8 up, 6 in 2024-09-10T19:27:15.682 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:27:15 smithi183 ceph-mon[41562]: 2.8 scrub starts 2024-09-10T19:27:15.682 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:27:15 smithi183 ceph-mon[41562]: 2.8 scrub ok 2024-09-10T19:27:15.682 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:27:15 smithi183 ceph-mon[41562]: osdmap e360: 8 total, 8 up, 6 in 2024-09-10T19:27:15.697 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:27:15.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:15.698 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:27:15.715 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:27:15 smithi191 ceph-mon[32884]: 4.17 scrub starts 2024-09-10T19:27:15.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:27:15 smithi191 ceph-mon[32884]: 4.17 scrub ok 2024-09-10T19:27:15.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:27:15 smithi191 ceph-mon[32884]: osdmap e359: 8 total, 8 up, 6 in 2024-09-10T19:27:15.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:27:15 smithi191 ceph-mon[32884]: 2.8 scrub starts 2024-09-10T19:27:15.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:27:15 smithi191 ceph-mon[32884]: 2.8 scrub ok 2024-09-10T19:27:15.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:27:15 smithi191 ceph-mon[32884]: osdmap e360: 8 total, 8 up, 6 in 2024-09-10T19:27:15.798 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:27:15.828 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:27:15.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:15.829 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:27:15.930 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:27:15.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:15.962 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:27:15.962 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:27:16.063 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:27:16.098 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:27:16.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:16.099 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:27:16.200 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:27:16.211 DEBUG:teuthology.orchestra.run.smithi191:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph daemon osd.4 dump_ops_in_flight 2024-09-10T19:27:16.235 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:27:16.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:16.236 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:27:16.337 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:27:16.382 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:27:16.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:16.383 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:27:16.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:27:16 smithi183 ceph-mon[29736]: 10.d starting backfill to osd.3 from (0'0,0'0] MAX to 357'14438 2024-09-10T19:27:16.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:27:16 smithi183 ceph-mon[29736]: 10.e starting backfill to osd.3 from (0'0,0'0] MAX to 357'14277 2024-09-10T19:27:16.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:27:16 smithi183 ceph-mon[29736]: 10.0 starting backfill to osd.7 from (0'0,0'0] MAX to 357'14302 2024-09-10T19:27:16.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:27:16 smithi183 ceph-mon[29736]: 10.f starting backfill to osd.1 from (0'0,0'0] MAX to 357'14523 2024-09-10T19:27:16.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:27:16 smithi183 ceph-mon[29736]: 10.b starting backfill to osd.4 from (0'0,0'0] MAX to 357'14770 2024-09-10T19:27:16.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:27:16 smithi183 ceph-mon[29736]: 10.7 starting backfill to osd.4 from (0'0,0'0] MAX to 357'14357 2024-09-10T19:27:16.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:27:16 smithi183 ceph-mon[29736]: 10.a starting backfill to osd.3 from (0'0,0'0] MAX to 357'14502 2024-09-10T19:27:16.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:27:16 smithi183 ceph-mon[29736]: 2.c scrub starts 2024-09-10T19:27:16.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:27:16 smithi183 ceph-mon[29736]: pgmap v786: 158 pgs: 4 remapped+peering, 154 active+clean; 930 MiB data, 7.6 GiB used, 529 GiB / 536 GiB avail; 2.8 KiB/s rd, 25 KiB/s wr, 1.11k op/s; 52 KiB/s, 9 objects/s recovering 2024-09-10T19:27:16.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:27:16 smithi183 ceph-mon[41562]: 10.d starting backfill to osd.3 from (0'0,0'0] MAX to 357'14438 2024-09-10T19:27:16.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:27:16 smithi183 ceph-mon[41562]: 10.e starting backfill to osd.3 from (0'0,0'0] MAX to 357'14277 2024-09-10T19:27:16.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:27:16 smithi183 ceph-mon[41562]: 10.0 starting backfill to osd.7 from (0'0,0'0] MAX to 357'14302 2024-09-10T19:27:16.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:27:16 smithi183 ceph-mon[41562]: 10.f starting backfill to osd.1 from (0'0,0'0] MAX to 357'14523 2024-09-10T19:27:16.433 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:27:16 smithi183 ceph-mon[41562]: 10.b starting backfill to osd.4 from (0'0,0'0] MAX to 357'14770 2024-09-10T19:27:16.433 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:27:16 smithi183 ceph-mon[41562]: 10.7 starting backfill to osd.4 from (0'0,0'0] MAX to 357'14357 2024-09-10T19:27:16.433 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:27:16 smithi183 ceph-mon[41562]: 10.a starting backfill to osd.3 from (0'0,0'0] MAX to 357'14502 2024-09-10T19:27:16.433 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:27:16 smithi183 ceph-mon[41562]: 2.c scrub starts 2024-09-10T19:27:16.433 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:27:16 smithi183 ceph-mon[41562]: pgmap v786: 158 pgs: 4 remapped+peering, 154 active+clean; 930 MiB data, 7.6 GiB used, 529 GiB / 536 GiB avail; 2.8 KiB/s rd, 25 KiB/s wr, 1.11k op/s; 52 KiB/s, 9 objects/s recovering 2024-09-10T19:27:16.483 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:27:16.520 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:27:16.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:16.521 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:27:16.622 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:27:16.653 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:27:16.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:16.654 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:27:16.680 INFO:teuthology.orchestra.run.smithi191.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.b/config 2024-09-10T19:27:16.718 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:27:16 smithi191 ceph-mon[32884]: 10.d starting backfill to osd.3 from (0'0,0'0] MAX to 357'14438 2024-09-10T19:27:16.718 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:27:16 smithi191 ceph-mon[32884]: 10.e starting backfill to osd.3 from (0'0,0'0] MAX to 357'14277 2024-09-10T19:27:16.718 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:27:16 smithi191 ceph-mon[32884]: 10.0 starting backfill to osd.7 from (0'0,0'0] MAX to 357'14302 2024-09-10T19:27:16.718 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:27:16 smithi191 ceph-mon[32884]: 10.f starting backfill to osd.1 from (0'0,0'0] MAX to 357'14523 2024-09-10T19:27:16.718 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:27:16 smithi191 ceph-mon[32884]: 10.b starting backfill to osd.4 from (0'0,0'0] MAX to 357'14770 2024-09-10T19:27:16.718 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:27:16 smithi191 ceph-mon[32884]: 10.7 starting backfill to osd.4 from (0'0,0'0] MAX to 357'14357 2024-09-10T19:27:16.718 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:27:16 smithi191 ceph-mon[32884]: 10.a starting backfill to osd.3 from (0'0,0'0] MAX to 357'14502 2024-09-10T19:27:16.719 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:27:16 smithi191 ceph-mon[32884]: 2.c scrub starts 2024-09-10T19:27:16.719 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:27:16 smithi191 ceph-mon[32884]: pgmap v786: 158 pgs: 4 remapped+peering, 154 active+clean; 930 MiB data, 7.6 GiB used, 529 GiB / 536 GiB avail; 2.8 KiB/s rd, 25 KiB/s wr, 1.11k op/s; 52 KiB/s, 9 objects/s recovering 2024-09-10T19:27:16.754 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:27:16.795 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:27:16.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:16.796 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:27:16.897 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:27:16.935 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:27:16.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:16.936 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:27:17.037 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:27:17.069 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:27:17.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:17.071 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:27:17.172 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:27:17.207 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:27:17.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:17.209 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:27:17.309 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:27:17.347 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:27:17.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:17.349 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:27:17.449 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:27:17.499 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:27:17.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:17.500 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:27:17.601 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:27:17.615 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:27:17 smithi191 ceph-mon[32884]: 6.10 scrub starts 2024-09-10T19:27:17.615 INFO:journalctl@ceph.mgr.x.smithi191.stdout:Sep 10 19:27:17 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-x[34967]: ::ffff:172.21.15.191 - - [10/Sep/2024:19:27:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T19:27:17.630 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:27:17.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:17.631 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:27:17.681 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:27:17 smithi183 ceph-mon[29736]: 6.10 scrub starts 2024-09-10T19:27:17.681 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:27:17 smithi183 ceph-mon[41562]: 6.10 scrub starts 2024-09-10T19:27:17.732 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:27:17.763 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:27:17.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:17.764 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:27:17.865 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:27:17.906 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:27:17.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:17.907 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:27:18.008 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:27:18.043 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:27:18.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:18.044 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:27:18.145 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:27:18.187 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:27:18.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:18.188 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:27:18.289 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:27:18.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:18.311 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:27:18.311 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:27:18.395 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:27:18 smithi183 ceph-mon[29736]: 6.16 scrub starts 2024-09-10T19:27:18.395 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:27:18 smithi183 ceph-mon[29736]: 10.3 scrub starts 2024-09-10T19:27:18.395 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:27:18 smithi183 ceph-mon[29736]: pgmap v787: 158 pgs: 2 active+remapped+backfilling, 4 remapped+peering, 1 active+remapped+backfill_toofull, 151 active+clean; 799 MiB data, 7.4 GiB used, 529 GiB / 536 GiB avail; 6.7 KiB/s rd, 49 KiB/s wr, 1.04k op/s; 1798/38916 objects misplaced (4.620%); 4.9 MiB/s, 90 objects/s recovering 2024-09-10T19:27:18.396 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:27:18 smithi183 ceph-mon[41562]: 6.16 scrub starts 2024-09-10T19:27:18.396 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:27:18 smithi183 ceph-mon[41562]: 10.3 scrub starts 2024-09-10T19:27:18.396 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:27:18 smithi183 ceph-mon[41562]: pgmap v787: 158 pgs: 2 active+remapped+backfilling, 4 remapped+peering, 1 active+remapped+backfill_toofull, 151 active+clean; 799 MiB data, 7.4 GiB used, 529 GiB / 536 GiB avail; 6.7 KiB/s rd, 49 KiB/s wr, 1.04k op/s; 1798/38916 objects misplaced (4.620%); 4.9 MiB/s, 90 objects/s recovering 2024-09-10T19:27:18.412 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:27:18.452 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:27:18.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:18.453 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:27:18.554 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:27:18.588 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:27:18.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:18.589 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:27:18.690 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:27:18.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:27:18 smithi191 ceph-mon[32884]: 6.16 scrub starts 2024-09-10T19:27:18.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:27:18 smithi191 ceph-mon[32884]: 10.3 scrub starts 2024-09-10T19:27:18.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:27:18 smithi191 ceph-mon[32884]: pgmap v787: 158 pgs: 2 active+remapped+backfilling, 4 remapped+peering, 1 active+remapped+backfill_toofull, 151 active+clean; 799 MiB data, 7.4 GiB used, 529 GiB / 536 GiB avail; 6.7 KiB/s rd, 49 KiB/s wr, 1.04k op/s; 1798/38916 objects misplaced (4.620%); 4.9 MiB/s, 90 objects/s recovering 2024-09-10T19:27:18.723 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:27:18.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:18.724 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:27:18.764 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-10T19:27:18.825 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:27:18.848 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:27:18.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:18.849 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:27:18.950 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:27:18.980 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:27:18.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:18.982 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:27:19.083 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:27:19.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:19.121 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:27:19.121 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:27:19.141 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:27:19.222 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:27:19.247 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:27:19.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:19.248 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:27:19.349 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:27:19.373 INFO:teuthology.orchestra.run.smithi183.stderr:nodeep-scrub is unset 2024-09-10T19:27:19.388 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:27:19.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:19.389 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:27:19.490 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:27:19.521 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:27:19.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:19.522 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:27:19.623 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:27:19.655 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:27:19.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:19.656 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:27:19.681 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:27:19 smithi183 ceph-mon[29736]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-09-10T19:27:19.681 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:27:19 smithi183 ceph-mon[29736]: from='client.? 172.21.15.183:0/2853310592' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-10T19:27:19.681 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:27:19 smithi183 ceph-mon[41562]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-09-10T19:27:19.681 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:27:19 smithi183 ceph-mon[41562]: from='client.? 172.21.15.183:0/2853310592' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-10T19:27:19.715 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:27:19 smithi191 ceph-mon[32884]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2024-09-10T19:27:19.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:27:19 smithi191 ceph-mon[32884]: from='client.? 172.21.15.183:0/2853310592' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-10T19:27:19.757 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:27:19.791 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:27:19.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:19.792 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:27:19.893 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:27:19.914 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:27:19.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:19.915 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:27:20.015 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:27:20.039 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:27:20.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:20.041 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:27:20.141 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:27:20.177 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:27:20.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:20.178 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:27:20.215 INFO:journalctl@ceph.iscsi.iscsi.a.smithi191.stdout:Sep 10 19:27:19 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-iscsi-iscsi-a[61816]: debug there is no tcmu-runner data available 2024-09-10T19:27:20.279 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:27:20.320 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:27:20.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:20.321 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:27:20.422 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:27:20.459 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:27:20.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:20.460 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:27:20.561 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:27:20.595 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:27:20.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:20.596 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:27:20.681 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:27:20 smithi183 ceph-mon[29736]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-10T19:27:20.681 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:27:20 smithi183 ceph-mon[29736]: from='client.? 172.21.15.183:0/2853310592' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-10T19:27:20.681 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:27:20 smithi183 ceph-mon[29736]: osdmap e361: 8 total, 8 up, 6 in 2024-09-10T19:27:20.682 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:27:20 smithi183 ceph-mon[29736]: 4.1e scrub starts 2024-09-10T19:27:20.682 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:27:20 smithi183 ceph-mon[29736]: 6.19 scrub starts 2024-09-10T19:27:20.682 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:27:20 smithi183 ceph-mon[29736]: from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:27:20.682 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:27:20 smithi183 ceph-mon[29736]: pgmap v789: 158 pgs: 2 active+remapped+backfilling, 4 remapped+peering, 1 active+remapped+backfill_toofull, 151 active+clean; 799 MiB data, 7.4 GiB used, 529 GiB / 536 GiB avail; 5.8 KiB/s rd, 49 KiB/s wr, 767 op/s; 1798/38916 objects misplaced (4.620%); 4.9 MiB/s, 90 objects/s recovering 2024-09-10T19:27:20.682 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:27:20 smithi183 ceph-mon[41562]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-10T19:27:20.682 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:27:20 smithi183 ceph-mon[41562]: from='client.? 172.21.15.183:0/2853310592' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-10T19:27:20.682 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:27:20 smithi183 ceph-mon[41562]: osdmap e361: 8 total, 8 up, 6 in 2024-09-10T19:27:20.682 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:27:20 smithi183 ceph-mon[41562]: 4.1e scrub starts 2024-09-10T19:27:20.682 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:27:20 smithi183 ceph-mon[41562]: 6.19 scrub starts 2024-09-10T19:27:20.682 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:27:20 smithi183 ceph-mon[41562]: from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:27:20.683 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:27:20 smithi183 ceph-mon[41562]: pgmap v789: 158 pgs: 2 active+remapped+backfilling, 4 remapped+peering, 1 active+remapped+backfill_toofull, 151 active+clean; 799 MiB data, 7.4 GiB used, 529 GiB / 536 GiB avail; 5.8 KiB/s rd, 49 KiB/s wr, 767 op/s; 1798/38916 objects misplaced (4.620%); 4.9 MiB/s, 90 objects/s recovering 2024-09-10T19:27:20.697 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:27:20.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:27:20 smithi191 ceph-mon[32884]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-10T19:27:20.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:27:20 smithi191 ceph-mon[32884]: from='client.? 172.21.15.183:0/2853310592' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-10T19:27:20.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:27:20 smithi191 ceph-mon[32884]: osdmap e361: 8 total, 8 up, 6 in 2024-09-10T19:27:20.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:27:20 smithi191 ceph-mon[32884]: 4.1e scrub starts 2024-09-10T19:27:20.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:27:20 smithi191 ceph-mon[32884]: 6.19 scrub starts 2024-09-10T19:27:20.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:27:20 smithi191 ceph-mon[32884]: from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:27:20.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:27:20 smithi191 ceph-mon[32884]: pgmap v789: 158 pgs: 2 active+remapped+backfilling, 4 remapped+peering, 1 active+remapped+backfill_toofull, 151 active+clean; 799 MiB data, 7.4 GiB used, 529 GiB / 536 GiB avail; 5.8 KiB/s rd, 49 KiB/s wr, 767 op/s; 1798/38916 objects misplaced (4.620%); 4.9 MiB/s, 90 objects/s recovering 2024-09-10T19:27:20.725 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:27:20.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:20.726 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:27:20.827 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:27:20.857 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:27:20.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:20.858 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:27:20.958 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:27:20.985 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:27:20.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:20.986 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:27:21.087 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:27:21.109 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:27:21.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:21.110 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:27:21.211 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:27:21.232 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:27:21.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:21.233 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:27:21.298 INFO:teuthology.orchestra.run.smithi191.stdout:ERROR: (22) Invalid argument 2024-09-10T19:27:21.298 INFO:teuthology.orchestra.run.smithi191.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-10T19:27:21.334 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:27:21.355 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:27:21.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:21.356 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:27:21.457 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:27:21.481 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:27:21.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:21.482 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:27:21.583 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:27:21.604 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:27:21.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:21.605 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:27:21.681 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Sep 10 19:27:21 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-y[29989]: ::ffff:172.21.15.191 - - [10/Sep/2024:19:27:21] "GET /metrics HTTP/1.1" 200 224764 "" "Prometheus/2.43.0" 2024-09-10T19:27:21.706 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:27:21.728 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:27:21.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:21.729 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:27:21.830 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:27:21.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:21.857 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:27:21.857 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:27:21.958 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:27:21.975 DEBUG:teuthology.orchestra.run.smithi191:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph daemon osd.4 dump_blocked_ops 2024-09-10T19:27:21.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:21.985 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:27:21.985 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:27:22.085 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:27:22.114 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:27:22.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:22.115 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:27:22.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:27:21 smithi191 ceph-mon[32884]: 1.0 scrub starts 2024-09-10T19:27:22.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:27:21 smithi191 ceph-mon[32884]: osdmap e362: 8 total, 8 up, 6 in 2024-09-10T19:27:22.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:27:21 smithi191 ceph-mon[32884]: 6.1 deep-scrub starts 2024-09-10T19:27:22.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:27:21 smithi191 ceph-mon[32884]: 6.1 deep-scrub ok 2024-09-10T19:27:22.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:27:21 smithi191 ceph-mon[32884]: 10.6 scrub starts 2024-09-10T19:27:22.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:27:21 smithi191 ceph-mon[32884]: 10.6 scrub ok 2024-09-10T19:27:22.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:27:21 smithi191 ceph-mon[32884]: 10.1 scrub starts 2024-09-10T19:27:22.216 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:27:22.238 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:27:22.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:22.239 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:27:22.295 INFO:teuthology.orchestra.run.smithi191.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.b/config 2024-09-10T19:27:22.323 INFO:tasks.thrashosds.thrasher:in_osds: [0, 3, 4, 5, 7, 1] out_osds: [2, 6] dead_osds: [] live_osds: [2, 3, 4, 5, 6, 7, 1, 0] 2024-09-10T19:27:22.323 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-10T19:27:22.323 INFO:tasks.thrashosds.thrasher:inject_pause on osd.0 2024-09-10T19:27:22.323 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-09-10T19:27:22.324 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-09-10T19:27:22.324 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph daemon osd.0 config set filestore_inject_stall 3 2024-09-10T19:27:22.340 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:27:22.346 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:27:21 smithi183 ceph-mon[29736]: 1.0 scrub starts 2024-09-10T19:27:22.346 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:27:21 smithi183 ceph-mon[29736]: osdmap e362: 8 total, 8 up, 6 in 2024-09-10T19:27:22.346 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:27:21 smithi183 ceph-mon[29736]: 6.1 deep-scrub starts 2024-09-10T19:27:22.346 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:27:21 smithi183 ceph-mon[29736]: 6.1 deep-scrub ok 2024-09-10T19:27:22.346 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:27:21 smithi183 ceph-mon[29736]: 10.6 scrub starts 2024-09-10T19:27:22.346 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:27:21 smithi183 ceph-mon[29736]: 10.6 scrub ok 2024-09-10T19:27:22.346 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:27:21 smithi183 ceph-mon[29736]: 10.1 scrub starts 2024-09-10T19:27:22.346 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:27:21 smithi183 ceph-mon[41562]: 1.0 scrub starts 2024-09-10T19:27:22.346 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:27:21 smithi183 ceph-mon[41562]: osdmap e362: 8 total, 8 up, 6 in 2024-09-10T19:27:22.347 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:27:21 smithi183 ceph-mon[41562]: 6.1 deep-scrub starts 2024-09-10T19:27:22.347 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:27:21 smithi183 ceph-mon[41562]: 6.1 deep-scrub ok 2024-09-10T19:27:22.347 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:27:21 smithi183 ceph-mon[41562]: 10.6 scrub starts 2024-09-10T19:27:22.347 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:27:21 smithi183 ceph-mon[41562]: 10.6 scrub ok 2024-09-10T19:27:22.347 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:27:21 smithi183 ceph-mon[41562]: 10.1 scrub starts 2024-09-10T19:27:22.362 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:27:22.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:22.363 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:27:22.464 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:27:22.486 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:27:22.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:22.487 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:27:22.588 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:27:22.609 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:27:22.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:22.610 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:27:22.711 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:27:22.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:22.733 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:27:22.733 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:27:22.834 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:27:22.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:22.856 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:27:22.856 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:27:22.958 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:27:22.965 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:27:22 smithi191 ceph-mon[32884]: pgmap v791: 158 pgs: 3 active+remapped+backfilling, 4 active+remapped+backfill_toofull, 151 active+clean; 599 MiB data, 6.9 GiB used, 529 GiB / 536 GiB avail; 3.3 KiB/s rd, 23 KiB/s wr, 955 op/s; 3351/29316 objects misplaced (11.431%); 9.9 MiB/s, 161 objects/s recovering 2024-09-10T19:27:22.966 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:27:22 smithi191 ceph-mon[32884]: 10.1 scrub ok 2024-09-10T19:27:22.966 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:27:22 smithi191 ceph-mon[32884]: osdmap e363: 8 total, 8 up, 6 in 2024-09-10T19:27:22.966 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:27:22 smithi191 ceph-mon[32884]: 2.1c scrub starts 2024-09-10T19:27:22.966 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:27:22 smithi191 ceph-mon[32884]: 2.1c scrub ok 2024-09-10T19:27:22.966 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:27:22 smithi191 ceph-mon[32884]: 2.2 scrub starts 2024-09-10T19:27:22.966 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:27:22 smithi191 ceph-mon[32884]: 2.2 scrub ok 2024-09-10T19:27:22.980 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:27:22.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:22.981 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:27:23.081 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:27:23.109 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:27:23.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:23.111 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:27:23.212 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:27:23.244 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:27:23.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:23.245 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:27:23.346 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:27:23.367 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:27:23.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:23.368 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:27:23.431 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:27:22 smithi183 ceph-mon[41562]: pgmap v791: 158 pgs: 3 active+remapped+backfilling, 4 active+remapped+backfill_toofull, 151 active+clean; 599 MiB data, 6.9 GiB used, 529 GiB / 536 GiB avail; 3.3 KiB/s rd, 23 KiB/s wr, 955 op/s; 3351/29316 objects misplaced (11.431%); 9.9 MiB/s, 161 objects/s recovering 2024-09-10T19:27:23.431 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:27:22 smithi183 ceph-mon[41562]: 10.1 scrub ok 2024-09-10T19:27:23.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:27:22 smithi183 ceph-mon[41562]: osdmap e363: 8 total, 8 up, 6 in 2024-09-10T19:27:23.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:27:22 smithi183 ceph-mon[41562]: 2.1c scrub starts 2024-09-10T19:27:23.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:27:22 smithi183 ceph-mon[41562]: 2.1c scrub ok 2024-09-10T19:27:23.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:27:22 smithi183 ceph-mon[41562]: 2.2 scrub starts 2024-09-10T19:27:23.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:27:22 smithi183 ceph-mon[41562]: 2.2 scrub ok 2024-09-10T19:27:23.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:27:22 smithi183 ceph-mon[29736]: pgmap v791: 158 pgs: 3 active+remapped+backfilling, 4 active+remapped+backfill_toofull, 151 active+clean; 599 MiB data, 6.9 GiB used, 529 GiB / 536 GiB avail; 3.3 KiB/s rd, 23 KiB/s wr, 955 op/s; 3351/29316 objects misplaced (11.431%); 9.9 MiB/s, 161 objects/s recovering 2024-09-10T19:27:23.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:27:22 smithi183 ceph-mon[29736]: 10.1 scrub ok 2024-09-10T19:27:23.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:27:22 smithi183 ceph-mon[29736]: osdmap e363: 8 total, 8 up, 6 in 2024-09-10T19:27:23.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:27:22 smithi183 ceph-mon[29736]: 2.1c scrub starts 2024-09-10T19:27:23.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:27:22 smithi183 ceph-mon[29736]: 2.1c scrub ok 2024-09-10T19:27:23.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:27:22 smithi183 ceph-mon[29736]: 2.2 scrub starts 2024-09-10T19:27:23.433 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:27:22 smithi183 ceph-mon[29736]: 2.2 scrub ok 2024-09-10T19:27:23.469 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:27:23.490 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:27:23.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:23.491 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:27:23.592 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:27:23.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:23.613 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:27:23.614 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:27:23.714 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:27:23.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:23.749 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:27:23.750 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:27:23.851 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:27:23.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:23.891 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:27:23.891 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:27:23.992 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:27:24.019 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:27:24.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:24.020 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:27:24.121 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:27:24.151 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:27:24.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:24.153 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:27:24.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:27:23 smithi183 ceph-mon[29736]: osdmap e364: 8 total, 8 up, 6 in 2024-09-10T19:27:24.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:27:23 smithi183 ceph-mon[29736]: 6.15 scrub starts 2024-09-10T19:27:24.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:27:23 smithi183 ceph-mon[29736]: 6.15 scrub ok 2024-09-10T19:27:24.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:27:23 smithi183 ceph-mon[41562]: osdmap e364: 8 total, 8 up, 6 in 2024-09-10T19:27:24.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:27:23 smithi183 ceph-mon[41562]: 6.15 scrub starts 2024-09-10T19:27:24.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:27:23 smithi183 ceph-mon[41562]: 6.15 scrub ok 2024-09-10T19:27:24.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:27:23 smithi191 ceph-mon[32884]: osdmap e364: 8 total, 8 up, 6 in 2024-09-10T19:27:24.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:27:23 smithi191 ceph-mon[32884]: 6.15 scrub starts 2024-09-10T19:27:24.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:27:23 smithi191 ceph-mon[32884]: 6.15 scrub ok 2024-09-10T19:27:24.254 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:27:24.291 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:27:24.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:24.292 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:27:24.393 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:27:24.424 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:27:24.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:24.425 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:27:24.526 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:27:24.562 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:27:24.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:24.563 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:27:24.664 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:27:24.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:24.697 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:27:24.698 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:27:24.798 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:27:24.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:24.841 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:27:24.842 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:27:24.943 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:27:24.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:24.974 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:27:24.974 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:27:25.075 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:27:25.078 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:27:24 smithi183 ceph-mon[29736]: 4.16 scrub starts 2024-09-10T19:27:25.078 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:27:24 smithi183 ceph-mon[29736]: 4.16 scrub ok 2024-09-10T19:27:25.078 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:27:24 smithi183 ceph-mon[29736]: pgmap v794: 158 pgs: 1 peering, 1 active+remapped+backfilling, 4 active+remapped+backfill_toofull, 152 active+clean; 465 MiB data, 6.6 GiB used, 530 GiB / 536 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.18k op/s; 1984/22899 objects misplaced (8.664%); 20 MiB/s, 315 objects/s recovering 2024-09-10T19:27:25.078 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:27:24 smithi183 ceph-mon[29736]: 6.a scrub starts 2024-09-10T19:27:25.078 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:27:24 smithi183 ceph-mon[29736]: 6.a scrub ok 2024-09-10T19:27:25.078 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:27:24 smithi183 ceph-mon[29736]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 4 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-09-10T19:27:25.079 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:27:24 smithi183 ceph-mon[41562]: 4.16 scrub starts 2024-09-10T19:27:25.079 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:27:24 smithi183 ceph-mon[41562]: 4.16 scrub ok 2024-09-10T19:27:25.079 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:27:24 smithi183 ceph-mon[41562]: pgmap v794: 158 pgs: 1 peering, 1 active+remapped+backfilling, 4 active+remapped+backfill_toofull, 152 active+clean; 465 MiB data, 6.6 GiB used, 530 GiB / 536 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.18k op/s; 1984/22899 objects misplaced (8.664%); 20 MiB/s, 315 objects/s recovering 2024-09-10T19:27:25.079 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:27:24 smithi183 ceph-mon[41562]: 6.a scrub starts 2024-09-10T19:27:25.079 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:27:24 smithi183 ceph-mon[41562]: 6.a scrub ok 2024-09-10T19:27:25.079 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:27:24 smithi183 ceph-mon[41562]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 4 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-09-10T19:27:25.095 INFO:teuthology.orchestra.run.smithi191.stdout:ERROR: (22) Invalid argument 2024-09-10T19:27:25.096 INFO:teuthology.orchestra.run.smithi191.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-10T19:27:25.121 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:27:25.121 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:27:25.122 INFO:teuthology.orchestra.run.smithi183.stdout:osd.0: {} 2024-09-10T19:27:25.122 INFO:teuthology.orchestra.run.smithi183.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-10T19:27:25.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:25.123 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:27:25.160 INFO:teuthology.orchestra.run.smithi183.stdout:osd.1: {} 2024-09-10T19:27:25.160 INFO:teuthology.orchestra.run.smithi183.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-10T19:27:25.177 INFO:teuthology.orchestra.run.smithi183.stdout:osd.2: {} 2024-09-10T19:27:25.177 INFO:teuthology.orchestra.run.smithi183.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-10T19:27:25.192 INFO:teuthology.orchestra.run.smithi183.stdout:osd.3: {} 2024-09-10T19:27:25.192 INFO:teuthology.orchestra.run.smithi183.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-10T19:27:25.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:27:24 smithi191 ceph-mon[32884]: 4.16 scrub starts 2024-09-10T19:27:25.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:27:24 smithi191 ceph-mon[32884]: 4.16 scrub ok 2024-09-10T19:27:25.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:27:24 smithi191 ceph-mon[32884]: pgmap v794: 158 pgs: 1 peering, 1 active+remapped+backfilling, 4 active+remapped+backfill_toofull, 152 active+clean; 465 MiB data, 6.6 GiB used, 530 GiB / 536 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.18k op/s; 1984/22899 objects misplaced (8.664%); 20 MiB/s, 315 objects/s recovering 2024-09-10T19:27:25.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:27:24 smithi191 ceph-mon[32884]: 6.a scrub starts 2024-09-10T19:27:25.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:27:24 smithi191 ceph-mon[32884]: 6.a scrub ok 2024-09-10T19:27:25.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:27:24 smithi191 ceph-mon[32884]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 4 pgs backfill_toofull (PG_BACKFILL_FULL) 2024-09-10T19:27:25.224 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:27:25.246 INFO:teuthology.orchestra.run.smithi183.stdout:osd.4: {} 2024-09-10T19:27:25.246 INFO:teuthology.orchestra.run.smithi183.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-10T19:27:25.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:25.262 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:27:25.262 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:27:25.279 INFO:teuthology.orchestra.run.smithi183.stdout:osd.5: {} 2024-09-10T19:27:25.279 INFO:teuthology.orchestra.run.smithi183.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-10T19:27:25.307 INFO:teuthology.orchestra.run.smithi183.stdout:osd.6: {} 2024-09-10T19:27:25.307 INFO:teuthology.orchestra.run.smithi183.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-10T19:27:25.331 INFO:teuthology.orchestra.run.smithi183.stdout:osd.7: {} 2024-09-10T19:27:25.331 INFO:teuthology.orchestra.run.smithi183.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-10T19:27:25.363 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:27:25.400 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:27:25.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:25.401 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:27:25.502 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:27:25.539 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:27:25.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:25.540 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:27:25.640 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:27:25.669 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:27:25.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:25.670 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:27:25.771 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:27:25.783 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph osd set noscrub 2024-09-10T19:27:25.805 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:27:25.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:25.807 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:27:25.908 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:27:25.947 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:27:25.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:25.948 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:27:26.049 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:27:26.083 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:27:26.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:26.084 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:27:26.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:27:25 smithi183 ceph-mon[29736]: purged_snaps scrub starts 2024-09-10T19:27:26.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:27:25 smithi183 ceph-mon[29736]: purged_snaps scrub ok 2024-09-10T19:27:26.182 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:27:25 smithi183 ceph-mon[29736]: from='client.? 172.21.15.183:0/1997440225' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T19:27:26.182 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:27:25 smithi183 ceph-mon[29736]: 2.10 scrub starts 2024-09-10T19:27:26.182 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:27:25 smithi183 ceph-mon[29736]: 2.10 scrub ok 2024-09-10T19:27:26.182 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:27:25 smithi183 ceph-mon[29736]: 5.15 scrub starts 2024-09-10T19:27:26.182 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:27:25 smithi183 ceph-mon[29736]: 5.15 scrub ok 2024-09-10T19:27:26.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:27:25 smithi183 ceph-mon[41562]: purged_snaps scrub starts 2024-09-10T19:27:26.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:27:25 smithi183 ceph-mon[41562]: purged_snaps scrub ok 2024-09-10T19:27:26.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:27:25 smithi183 ceph-mon[41562]: from='client.? 172.21.15.183:0/1997440225' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T19:27:26.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:27:25 smithi183 ceph-mon[41562]: 2.10 scrub starts 2024-09-10T19:27:26.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:27:25 smithi183 ceph-mon[41562]: 2.10 scrub ok 2024-09-10T19:27:26.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:27:25 smithi183 ceph-mon[41562]: 5.15 scrub starts 2024-09-10T19:27:26.183 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:27:25 smithi183 ceph-mon[41562]: 5.15 scrub ok 2024-09-10T19:27:26.185 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:27:26.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:26.218 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:27:26.218 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:27:26.319 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:27:26.346 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:27:26.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:26.348 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:27:26.448 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:27:26.465 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:27:25 smithi191 ceph-mon[32884]: purged_snaps scrub starts 2024-09-10T19:27:26.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:27:25 smithi191 ceph-mon[32884]: purged_snaps scrub ok 2024-09-10T19:27:26.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:27:25 smithi191 ceph-mon[32884]: from='client.? 172.21.15.183:0/1997440225' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T19:27:26.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:27:25 smithi191 ceph-mon[32884]: 2.10 scrub starts 2024-09-10T19:27:26.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:27:25 smithi191 ceph-mon[32884]: 2.10 scrub ok 2024-09-10T19:27:26.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:27:25 smithi191 ceph-mon[32884]: 5.15 scrub starts 2024-09-10T19:27:26.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:27:25 smithi191 ceph-mon[32884]: 5.15 scrub ok 2024-09-10T19:27:26.479 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:27:26.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:26.479 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:27:26.513 DEBUG:teuthology.orchestra.run.smithi191:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph daemon osd.4 dump_historic_ops 2024-09-10T19:27:26.580 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:27:26.638 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:27:26.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:26.639 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:27:26.740 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:27:26.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:26.782 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:27:26.782 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:27:26.868 INFO:teuthology.orchestra.run.smithi191.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.b/config 2024-09-10T19:27:26.883 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:27:26.906 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:27:26.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:26.907 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:27:27.008 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:27:27.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:27.040 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:27:27.040 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:27:27.141 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:27:27.169 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:27:27.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:27.170 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:27:27.234 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:27:26 smithi183 ceph-mon[29736]: pgmap v795: 158 pgs: 1 peering, 4 active+remapped+backfill_toofull, 153 active+clean; 328 MiB data, 6.1 GiB used, 530 GiB / 536 GiB avail; 853 B/s rd, 0 B/s wr, 1.26k op/s; 1187/16302 objects misplaced (7.281%); 15 MiB/s, 247 objects/s recovering 2024-09-10T19:27:27.234 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:27:26 smithi183 ceph-mon[29736]: osdmap e365: 8 total, 8 up, 6 in 2024-09-10T19:27:27.234 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:27:26 smithi183 ceph-mon[29736]: 4.1c scrub starts 2024-09-10T19:27:27.234 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:27:26 smithi183 ceph-mon[29736]: 4.1c scrub ok 2024-09-10T19:27:27.234 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:27:26 smithi183 ceph-mon[29736]: 2.7 scrub starts 2024-09-10T19:27:27.234 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:27:26 smithi183 ceph-mon[29736]: 2.7 scrub ok 2024-09-10T19:27:27.235 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:27:26 smithi183 ceph-mon[41562]: pgmap v795: 158 pgs: 1 peering, 4 active+remapped+backfill_toofull, 153 active+clean; 328 MiB data, 6.1 GiB used, 530 GiB / 536 GiB avail; 853 B/s rd, 0 B/s wr, 1.26k op/s; 1187/16302 objects misplaced (7.281%); 15 MiB/s, 247 objects/s recovering 2024-09-10T19:27:27.235 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:27:26 smithi183 ceph-mon[41562]: osdmap e365: 8 total, 8 up, 6 in 2024-09-10T19:27:27.235 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:27:26 smithi183 ceph-mon[41562]: 4.1c scrub starts 2024-09-10T19:27:27.235 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:27:26 smithi183 ceph-mon[41562]: 4.1c scrub ok 2024-09-10T19:27:27.235 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:27:26 smithi183 ceph-mon[41562]: 2.7 scrub starts 2024-09-10T19:27:27.235 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:27:26 smithi183 ceph-mon[41562]: 2.7 scrub ok 2024-09-10T19:27:27.272 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:27:27.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:27.308 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:27:27.308 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:27:27.361 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:27:26 smithi191 ceph-mon[32884]: pgmap v795: 158 pgs: 1 peering, 4 active+remapped+backfill_toofull, 153 active+clean; 328 MiB data, 6.1 GiB used, 530 GiB / 536 GiB avail; 853 B/s rd, 0 B/s wr, 1.26k op/s; 1187/16302 objects misplaced (7.281%); 15 MiB/s, 247 objects/s recovering 2024-09-10T19:27:27.361 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:27:26 smithi191 ceph-mon[32884]: osdmap e365: 8 total, 8 up, 6 in 2024-09-10T19:27:27.361 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:27:26 smithi191 ceph-mon[32884]: 4.1c scrub starts 2024-09-10T19:27:27.361 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:27:26 smithi191 ceph-mon[32884]: 4.1c scrub ok 2024-09-10T19:27:27.361 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:27:26 smithi191 ceph-mon[32884]: 2.7 scrub starts 2024-09-10T19:27:27.361 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:27:26 smithi191 ceph-mon[32884]: 2.7 scrub ok 2024-09-10T19:27:27.410 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:27:27.451 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:27:27.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:27.451 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:27:27.552 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:27:27.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:27.585 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:27:27.585 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:27:27.686 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:27:27.715 INFO:journalctl@ceph.mgr.x.smithi191.stdout:Sep 10 19:27:27 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-x[34967]: ::ffff:172.21.15.191 - - [10/Sep/2024:19:27:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T19:27:27.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:27.719 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:27:27.719 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:27:27.820 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:27:27.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:27.855 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:27:27.855 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:27:27.956 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:27:27.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:27.989 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:27:27.989 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:27:28.090 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:27:28.130 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:27:28.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:28.131 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:27:28.199 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:27:27 smithi191 ceph-mon[32884]: 5.7 scrub starts 2024-09-10T19:27:28.199 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:27:27 smithi191 ceph-mon[32884]: 5.7 scrub ok 2024-09-10T19:27:28.233 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:27:28.263 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:27:28.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:28.265 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:27:28.287 INFO:tasks.radosbench.radosbench.0.smithi183.stdout:Removed 123205 objects 2024-09-10T19:27:28.287 INFO:tasks.radosbench.radosbench.0.smithi183.stdout:Clean up completed and total clean up time :97.4186 2024-09-10T19:27:28.324 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:27:27 smithi183 ceph-mon[29736]: 5.7 scrub starts 2024-09-10T19:27:28.324 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:27:27 smithi183 ceph-mon[29736]: 5.7 scrub ok 2024-09-10T19:27:28.325 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:27:27 smithi183 ceph-mon[41562]: 5.7 scrub starts 2024-09-10T19:27:28.325 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:27:27 smithi183 ceph-mon[41562]: 5.7 scrub ok 2024-09-10T19:27:28.366 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:27:28.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:28.389 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:27:28.389 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:27:28.490 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:27:28.512 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:27:28.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:28.513 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:27:28.614 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:27:28.637 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:27:28.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:28.637 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:27:28.691 INFO:teuthology.orchestra.run.smithi183.stdout:{ 2024-09-10T19:27:28.692 INFO:teuthology.orchestra.run.smithi183.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-09-10T19:27:28.692 INFO:teuthology.orchestra.run.smithi183.stdout:} 2024-09-10T19:27:28.738 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:27:28.760 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:27:28.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:28.761 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:27:28.804 INFO:teuthology.orchestra.run.smithi191.stdout:{ 2024-09-10T19:27:28.804 INFO:teuthology.orchestra.run.smithi191.stdout: "size": 20, 2024-09-10T19:27:28.804 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 600, 2024-09-10T19:27:28.805 INFO:teuthology.orchestra.run.smithi191.stdout: "ops": [ 2024-09-10T19:27:28.805 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:28.805 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.15051.0:109257 8.18 8:1f2c40d0:::benchmark_data_smithi183_114924_object109256:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e186)", 2024-09-10T19:27:28.805 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-10T19:17:44.707009+0000", 2024-09-10T19:27:28.805 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 584.09366383300005, 2024-09-10T19:27:28.805 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.98109888499999998, 2024-09-10T19:27:28.805 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-10T19:27:28.805 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T19:27:28.805 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-10T19:27:28.805 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.15051", 2024-09-10T19:27:28.805 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.183:0/3632028672", 2024-09-10T19:27:28.805 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 109257 2024-09-10T19:27:28.806 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:28.806 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-10T19:27:28.806 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:28.806 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-10T19:27:28.806 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:44.707009+0000", 2024-09-10T19:27:28.806 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:27:28.806 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:28.806 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:28.806 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-10T19:27:28.806 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:44.707009+0000", 2024-09-10T19:27:28.806 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:27:28.806 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:28.806 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:28.806 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-10T19:27:28.807 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:44.707008+0000", 2024-09-10T19:27:28.807 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999986 2024-09-10T19:27:28.807 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:28.807 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:28.807 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-10T19:27:28.807 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:44.707043+0000", 2024-09-10T19:27:28.807 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.5602999999999998e-05 2024-09-10T19:27:28.807 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:28.807 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:28.807 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-10T19:27:28.807 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:44.707057+0000", 2024-09-10T19:27:28.807 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.3131e-05 2024-09-10T19:27:28.807 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:28.808 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:28.808 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-10T19:27:28.808 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:44.707065+0000", 2024-09-10T19:27:28.808 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.2300000000000008e-06 2024-09-10T19:27:28.808 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:28.808 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:28.808 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:27:28.808 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:44.707123+0000", 2024-09-10T19:27:28.808 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.8063999999999998e-05 2024-09-10T19:27:28.808 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:28.808 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:28.808 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for peered", 2024-09-10T19:27:28.809 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:44.707131+0000", 2024-09-10T19:27:28.809 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.8199999999999997e-06 2024-09-10T19:27:28.809 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:28.809 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:28.809 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:27:28.809 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:45.679988+0000", 2024-09-10T19:27:28.809 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.97285749300000002 2024-09-10T19:27:28.809 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:28.809 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:28.809 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-10T19:27:28.809 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:45.680110+0000", 2024-09-10T19:27:28.809 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000122033 2024-09-10T19:27:28.809 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:28.810 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:28.810 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 3,5", 2024-09-10T19:27:28.810 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:45.680378+0000", 2024-09-10T19:27:28.810 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00026790800000000002 2024-09-10T19:27:28.810 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:28.810 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:28.810 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:27:28.810 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:45.685092+0000", 2024-09-10T19:27:28.810 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0047136210000000003 2024-09-10T19:27:28.810 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:28.810 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:28.810 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-09-10T19:27:28.811 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:45.685100+0000", 2024-09-10T19:27:28.811 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.96e-06 2024-09-10T19:27:28.811 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:28.811 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:28.811 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:27:28.811 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:45.688017+0000", 2024-09-10T19:27:28.811 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0029178479999999998 2024-09-10T19:27:28.811 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:28.811 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:28.811 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-09-10T19:27:28.811 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:45.688066+0000", 2024-09-10T19:27:28.811 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.8701e-05 2024-09-10T19:27:28.811 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:28.811 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:28.812 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-10T19:27:28.812 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:45.688108+0000", 2024-09-10T19:27:28.812 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.1934999999999997e-05 2024-09-10T19:27:28.812 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:27:28.812 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-10T19:27:28.812 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:27:28.812 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:28.812 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:28.812 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.15051.0:109277 8.18 8:1b4b775d:::benchmark_data_smithi183_114924_object109276:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e186)", 2024-09-10T19:27:28.812 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-10T19:17:44.723277+0000", 2024-09-10T19:27:28.812 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 584.07739621099995, 2024-09-10T19:27:28.812 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.964988281, 2024-09-10T19:27:28.813 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-10T19:27:28.813 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T19:27:28.813 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-10T19:27:28.813 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.15051", 2024-09-10T19:27:28.813 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.183:0/3632028672", 2024-09-10T19:27:28.813 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 109277 2024-09-10T19:27:28.813 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:28.813 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-10T19:27:28.813 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:28.814 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-10T19:27:28.814 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:44.723277+0000", 2024-09-10T19:27:28.814 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:27:28.814 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:28.814 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:28.814 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-10T19:27:28.814 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:44.723277+0000", 2024-09-10T19:27:28.814 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:27:28.814 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:28.814 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:28.814 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-10T19:27:28.814 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:44.723275+0000", 2024-09-10T19:27:28.814 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999981 2024-09-10T19:27:28.815 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:28.815 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:28.815 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-10T19:27:28.815 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:44.723306+0000", 2024-09-10T19:27:28.815 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.1040000000000001e-05 2024-09-10T19:27:28.815 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:28.815 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:28.815 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-10T19:27:28.815 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:44.723323+0000", 2024-09-10T19:27:28.815 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.7142e-05 2024-09-10T19:27:28.815 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:28.815 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:28.815 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-10T19:27:28.815 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:44.723330+0000", 2024-09-10T19:27:28.816 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.9489999999999999e-06 2024-09-10T19:27:28.816 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:28.816 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:28.816 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:27:28.816 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:44.723965+0000", 2024-09-10T19:27:28.816 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00063528900000000004 2024-09-10T19:27:28.816 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:28.816 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:28.816 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for peered", 2024-09-10T19:27:28.817 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:44.723973+0000", 2024-09-10T19:27:28.817 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.7929999999999998e-06 2024-09-10T19:27:28.817 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:28.817 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:28.817 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:27:28.817 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:45.681943+0000", 2024-09-10T19:27:28.817 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.957969819 2024-09-10T19:27:28.817 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:28.817 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:28.817 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-10T19:27:28.818 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:45.682064+0000", 2024-09-10T19:27:28.818 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00012128700000000001 2024-09-10T19:27:28.818 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:28.818 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:28.818 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 3,5", 2024-09-10T19:27:28.818 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:45.682284+0000", 2024-09-10T19:27:28.818 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00021983300000000001 2024-09-10T19:27:28.818 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:28.818 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:28.818 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:27:28.818 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:45.685212+0000", 2024-09-10T19:27:28.818 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.002928375 2024-09-10T19:27:28.819 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:28.819 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:28.819 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-09-10T19:27:28.819 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:45.685217+0000", 2024-09-10T19:27:28.819 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.0490000000000003e-06 2024-09-10T19:27:28.819 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:28.819 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:28.819 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:27:28.819 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:45.688185+0000", 2024-09-10T19:27:28.819 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0029672430000000001 2024-09-10T19:27:28.819 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:28.819 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:28.819 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-09-10T19:27:28.819 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:45.688229+0000", 2024-09-10T19:27:28.819 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.4601000000000002e-05 2024-09-10T19:27:28.820 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:28.820 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:28.820 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-10T19:27:28.820 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:45.688265+0000", 2024-09-10T19:27:28.820 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.5947e-05 2024-09-10T19:27:28.820 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:27:28.820 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-10T19:27:28.820 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:27:28.820 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:28.820 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:28.820 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.15051.0:109290 8.18 8:18d42a63:::benchmark_data_smithi183_114924_object109289:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e186)", 2024-09-10T19:27:28.820 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-10T19:17:44.735197+0000", 2024-09-10T19:27:28.821 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 584.06547618100001, 2024-09-10T19:27:28.821 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.95321090799999997, 2024-09-10T19:27:28.821 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-10T19:27:28.821 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T19:27:28.821 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-10T19:27:28.821 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.15051", 2024-09-10T19:27:28.821 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.183:0/3632028672", 2024-09-10T19:27:28.821 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 109290 2024-09-10T19:27:28.821 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:28.821 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-10T19:27:28.821 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:28.821 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-10T19:27:28.821 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:44.735197+0000", 2024-09-10T19:27:28.821 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:27:28.822 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:28.822 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:28.822 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-10T19:27:28.822 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:44.735197+0000", 2024-09-10T19:27:28.822 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:27:28.822 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:28.822 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:28.822 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-10T19:27:28.822 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:44.735195+0000", 2024-09-10T19:27:28.822 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999986 2024-09-10T19:27:28.822 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:28.822 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:28.822 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-10T19:27:28.823 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:44.735220+0000", 2024-09-10T19:27:28.823 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.4819000000000001e-05 2024-09-10T19:27:28.823 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:28.823 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:28.823 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-10T19:27:28.823 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:44.735231+0000", 2024-09-10T19:27:28.823 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.0446e-05 2024-09-10T19:27:28.823 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:28.823 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:28.823 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-10T19:27:28.823 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:44.735236+0000", 2024-09-10T19:27:28.824 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.0869999999999997e-06 2024-09-10T19:27:28.824 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:28.824 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:28.824 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:27:28.824 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:44.735275+0000", 2024-09-10T19:27:28.824 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.9081000000000001e-05 2024-09-10T19:27:28.824 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:28.824 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:28.824 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for peered", 2024-09-10T19:27:28.824 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:44.735282+0000", 2024-09-10T19:27:28.824 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.515e-06 2024-09-10T19:27:28.824 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:28.824 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:28.825 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:27:28.825 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:45.683581+0000", 2024-09-10T19:27:28.825 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.94829899299999998 2024-09-10T19:27:28.825 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:28.825 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:28.825 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-10T19:27:28.825 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:45.683705+0000", 2024-09-10T19:27:28.825 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000123891 2024-09-10T19:27:28.825 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:28.825 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:28.825 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 3,5", 2024-09-10T19:27:28.825 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:45.683959+0000", 2024-09-10T19:27:28.826 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00025346100000000002 2024-09-10T19:27:28.826 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:28.826 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:28.826 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-09-10T19:27:28.826 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:45.685221+0000", 2024-09-10T19:27:28.826 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0012626429999999999 2024-09-10T19:27:28.826 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:28.826 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:28.826 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:27:28.826 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:45.685248+0000", 2024-09-10T19:27:28.826 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.6767999999999999e-05 2024-09-10T19:27:28.826 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:28.827 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:28.827 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:27:28.827 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:45.688319+0000", 2024-09-10T19:27:28.827 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0030708910000000001 2024-09-10T19:27:28.827 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:28.827 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:28.827 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-09-10T19:27:28.827 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:45.688367+0000", 2024-09-10T19:27:28.827 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.7590999999999999e-05 2024-09-10T19:27:28.827 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:28.827 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:28.827 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-10T19:27:28.827 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:45.688408+0000", 2024-09-10T19:27:28.827 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.1227999999999999e-05 2024-09-10T19:27:28.828 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:27:28.828 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-10T19:27:28.828 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:27:28.828 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:28.828 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:28.828 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.15051.0:109305 8.18 8:1d65ec91:::benchmark_data_smithi183_114924_object109304:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e186)", 2024-09-10T19:27:28.828 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-10T19:17:44.749098+0000", 2024-09-10T19:27:28.828 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 584.05157532800001, 2024-09-10T19:27:28.828 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.93944762599999998, 2024-09-10T19:27:28.828 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-10T19:27:28.828 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T19:27:28.828 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-10T19:27:28.829 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.15051", 2024-09-10T19:27:28.829 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.183:0/3632028672", 2024-09-10T19:27:28.829 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 109305 2024-09-10T19:27:28.829 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:28.829 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-10T19:27:28.829 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:28.829 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-10T19:27:28.829 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:44.749098+0000", 2024-09-10T19:27:28.829 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:27:28.829 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:28.829 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:28.829 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-10T19:27:28.829 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:44.749098+0000", 2024-09-10T19:27:28.829 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:27:28.830 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:28.830 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:28.830 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-10T19:27:28.830 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:44.749097+0000", 2024-09-10T19:27:28.830 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.999999 2024-09-10T19:27:28.830 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:28.830 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:28.830 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-10T19:27:28.830 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:44.749161+0000", 2024-09-10T19:27:28.830 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.4610999999999995e-05 2024-09-10T19:27:28.830 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:28.830 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:28.830 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-10T19:27:28.830 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:44.749172+0000", 2024-09-10T19:27:28.830 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.0696999999999999e-05 2024-09-10T19:27:28.831 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:28.831 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:28.831 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-10T19:27:28.831 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:44.749176+0000", 2024-09-10T19:27:28.831 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.1520000000000002e-06 2024-09-10T19:27:28.831 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:28.831 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:28.831 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:27:28.831 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:44.749219+0000", 2024-09-10T19:27:28.831 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.2441999999999998e-05 2024-09-10T19:27:28.831 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:28.831 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:28.831 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for peered", 2024-09-10T19:27:28.832 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:44.749226+0000", 2024-09-10T19:27:28.832 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.2280000000000001e-06 2024-09-10T19:27:28.832 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:28.832 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:28.832 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:27:28.832 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:45.685267+0000", 2024-09-10T19:27:28.832 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.93604158599999998 2024-09-10T19:27:28.832 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:28.832 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:28.832 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-10T19:27:28.832 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:45.685374+0000", 2024-09-10T19:27:28.832 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00010631199999999999 2024-09-10T19:27:28.832 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:28.832 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:28.833 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 3,5", 2024-09-10T19:27:28.833 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:45.685579+0000", 2024-09-10T19:27:28.833 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000205448 2024-09-10T19:27:28.833 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:28.833 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:28.833 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:27:28.833 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:45.687962+0000", 2024-09-10T19:27:28.833 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0023832110000000001 2024-09-10T19:27:28.833 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:28.833 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:28.833 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-09-10T19:27:28.833 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:45.687970+0000", 2024-09-10T19:27:28.833 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.0099999999999995e-06 2024-09-10T19:27:28.834 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:28.834 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:28.834 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:27:28.834 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:45.688466+0000", 2024-09-10T19:27:28.834 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00049536100000000004 2024-09-10T19:27:28.835 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:28.835 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:28.835 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-09-10T19:27:28.835 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:45.688509+0000", 2024-09-10T19:27:28.835 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.3285999999999999e-05 2024-09-10T19:27:28.835 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:28.835 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:28.836 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-10T19:27:28.836 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:45.688545+0000", 2024-09-10T19:27:28.836 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.6424000000000002e-05 2024-09-10T19:27:28.836 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:27:28.836 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-10T19:27:28.836 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:27:28.836 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:28.836 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:28.836 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.15051.0:109329 8.18 8:19b17ae3:::benchmark_data_smithi183_114924_object109328:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e186)", 2024-09-10T19:27:28.836 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-10T19:17:44.775519+0000", 2024-09-10T19:27:28.836 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 584.02515402699999, 2024-09-10T19:27:28.837 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.91461550800000002, 2024-09-10T19:27:28.837 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-10T19:27:28.837 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T19:27:28.837 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-10T19:27:28.837 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.15051", 2024-09-10T19:27:28.837 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.183:0/3632028672", 2024-09-10T19:27:28.837 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 109329 2024-09-10T19:27:28.837 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:28.837 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-10T19:27:28.837 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:28.837 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-10T19:27:28.838 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:44.775519+0000", 2024-09-10T19:27:28.838 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:27:28.838 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:28.838 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:28.838 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-10T19:27:28.838 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:44.775519+0000", 2024-09-10T19:27:28.838 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:27:28.838 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:28.838 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:28.838 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-10T19:27:28.838 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:44.775517+0000", 2024-09-10T19:27:28.838 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999976 2024-09-10T19:27:28.838 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:28.838 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:28.839 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-10T19:27:28.839 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:44.775552+0000", 2024-09-10T19:27:28.839 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.4876000000000003e-05 2024-09-10T19:27:28.839 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:28.839 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:28.839 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-10T19:27:28.839 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:44.775567+0000", 2024-09-10T19:27:28.839 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.5048000000000001e-05 2024-09-10T19:27:28.839 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:28.839 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:28.839 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-10T19:27:28.839 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:44.775573+0000", 2024-09-10T19:27:28.839 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.1369999999999999e-06 2024-09-10T19:27:28.839 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:28.840 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:28.840 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:27:28.840 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:44.775614+0000", 2024-09-10T19:27:28.840 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.1650000000000003e-05 2024-09-10T19:27:28.840 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:28.840 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:28.840 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for peered", 2024-09-10T19:27:28.840 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:44.775622+0000", 2024-09-10T19:27:28.840 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.8310000000000001e-06 2024-09-10T19:27:28.840 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:28.840 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:28.840 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:27:28.840 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:45.686659+0000", 2024-09-10T19:27:28.841 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.91103709799999999 2024-09-10T19:27:28.841 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:28.841 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:28.841 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-10T19:27:28.841 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:45.686778+0000", 2024-09-10T19:27:28.841 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00011879600000000001 2024-09-10T19:27:28.841 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:28.841 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:28.841 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 3,5", 2024-09-10T19:27:28.841 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:45.686960+0000", 2024-09-10T19:27:28.841 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00018159599999999999 2024-09-10T19:27:28.842 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:28.842 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:28.842 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:27:28.842 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:45.688145+0000", 2024-09-10T19:27:28.842 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0011851640000000001 2024-09-10T19:27:28.842 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:28.842 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:28.842 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-09-10T19:27:28.842 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:45.688420+0000", 2024-09-10T19:27:28.842 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00027469400000000002 2024-09-10T19:27:28.842 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:28.842 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:28.842 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:27:28.842 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:45.690057+0000", 2024-09-10T19:27:28.842 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0016371039999999999 2024-09-10T19:27:28.843 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:28.843 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:28.843 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-09-10T19:27:28.843 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:45.690100+0000", 2024-09-10T19:27:28.843 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.2896999999999999e-05 2024-09-10T19:27:28.843 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:28.843 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:28.843 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-10T19:27:28.843 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:45.690135+0000", 2024-09-10T19:27:28.843 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.4944000000000003e-05 2024-09-10T19:27:28.844 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:27:28.844 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-10T19:27:28.844 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:27:28.844 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:28.844 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:28.844 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.15051.0:109354 8.18 8:1fca233c:::benchmark_data_smithi183_114924_object109353:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e186)", 2024-09-10T19:27:28.844 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-10T19:17:44.808085+0000", 2024-09-10T19:27:28.844 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 583.99258783000005, 2024-09-10T19:27:28.844 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.89144352100000002, 2024-09-10T19:27:28.844 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-10T19:27:28.845 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T19:27:28.845 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-10T19:27:28.845 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.15051", 2024-09-10T19:27:28.845 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.183:0/3632028672", 2024-09-10T19:27:28.845 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 109354 2024-09-10T19:27:28.845 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:28.845 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-10T19:27:28.845 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:28.845 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-10T19:27:28.845 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:44.808085+0000", 2024-09-10T19:27:28.845 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:27:28.846 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:28.846 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:28.846 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-10T19:27:28.846 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:44.808085+0000", 2024-09-10T19:27:28.846 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:27:28.846 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:28.846 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:28.846 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-10T19:27:28.846 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:44.808084+0000", 2024-09-10T19:27:28.846 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999986 2024-09-10T19:27:28.846 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:28.846 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:28.846 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-10T19:27:28.847 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:44.808185+0000", 2024-09-10T19:27:28.847 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000100951 2024-09-10T19:27:28.847 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:28.847 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:28.847 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-10T19:27:28.847 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:44.808199+0000", 2024-09-10T19:27:28.847 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.4737e-05 2024-09-10T19:27:28.847 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:28.847 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:28.847 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-10T19:27:28.847 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:44.808205+0000", 2024-09-10T19:27:28.847 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.6500000000000001e-06 2024-09-10T19:27:28.847 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:28.847 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:28.848 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:27:28.848 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:44.808257+0000", 2024-09-10T19:27:28.848 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.2340999999999999e-05 2024-09-10T19:27:28.848 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:28.848 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:28.848 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for peered", 2024-09-10T19:27:28.848 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:44.808268+0000", 2024-09-10T19:27:28.848 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.0567999999999999e-05 2024-09-10T19:27:28.848 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:28.848 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:28.848 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:27:28.848 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:45.688585+0000", 2024-09-10T19:27:28.848 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.88031695499999996 2024-09-10T19:27:28.848 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:28.849 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:28.849 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-10T19:27:28.849 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:45.688705+0000", 2024-09-10T19:27:28.849 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000119876 2024-09-10T19:27:28.849 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:28.849 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:28.849 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 3,5", 2024-09-10T19:27:28.849 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:45.688915+0000", 2024-09-10T19:27:28.849 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00021063899999999999 2024-09-10T19:27:28.849 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:28.849 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:28.850 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:27:28.850 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:45.693186+0000", 2024-09-10T19:27:28.850 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0042711149999999998 2024-09-10T19:27:28.850 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:28.850 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:28.850 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-09-10T19:27:28.850 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:45.699298+0000", 2024-09-10T19:27:28.850 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0061111129999999996 2024-09-10T19:27:28.850 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:28.850 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:28.850 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:27:28.850 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:45.699358+0000", 2024-09-10T19:27:28.850 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.0961999999999997e-05 2024-09-10T19:27:28.850 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:28.851 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:28.851 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-09-10T19:27:28.851 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:45.699397+0000", 2024-09-10T19:27:28.851 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.8618999999999997e-05 2024-09-10T19:27:28.851 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:28.851 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:28.851 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-10T19:27:28.851 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:45.699529+0000", 2024-09-10T19:27:28.851 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00013165900000000001 2024-09-10T19:27:28.851 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:27:28.851 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-10T19:27:28.851 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:27:28.851 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:28.852 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:28.852 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.15051.0:109357 8.18 8:1ae888de:::benchmark_data_smithi183_114924_object109356:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e186)", 2024-09-10T19:27:28.852 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-10T19:17:44.809153+0000", 2024-09-10T19:27:28.852 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 583.99151976200005, 2024-09-10T19:27:28.852 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.89176314000000001, 2024-09-10T19:27:28.852 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-10T19:27:28.852 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T19:27:28.852 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-10T19:27:28.852 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.15051", 2024-09-10T19:27:28.852 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.183:0/3632028672", 2024-09-10T19:27:28.852 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 109357 2024-09-10T19:27:28.852 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:28.852 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-10T19:27:28.853 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:28.853 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-10T19:27:28.853 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:44.809153+0000", 2024-09-10T19:27:28.853 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:27:28.853 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:28.853 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:28.853 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-10T19:27:28.853 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:44.809153+0000", 2024-09-10T19:27:28.853 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:27:28.853 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:28.853 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:28.853 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-10T19:27:28.853 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:44.809152+0000", 2024-09-10T19:27:28.853 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.999999 2024-09-10T19:27:28.854 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:28.854 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:28.854 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-10T19:27:28.854 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:44.809183+0000", 2024-09-10T19:27:28.854 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.0394000000000001e-05 2024-09-10T19:27:28.854 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:28.854 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:28.854 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-10T19:27:28.854 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:44.809193+0000", 2024-09-10T19:27:28.854 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.0380999999999999e-05 2024-09-10T19:27:28.854 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:28.854 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:28.854 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-10T19:27:28.854 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:44.809198+0000", 2024-09-10T19:27:28.855 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.3109999999999996e-06 2024-09-10T19:27:28.855 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:28.855 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:28.855 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:27:28.855 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:44.809243+0000", 2024-09-10T19:27:28.855 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.4291e-05 2024-09-10T19:27:28.855 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:28.855 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:28.855 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for peered", 2024-09-10T19:27:28.855 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:44.809251+0000", 2024-09-10T19:27:28.855 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.2819999999999996e-06 2024-09-10T19:27:28.855 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:28.855 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:28.855 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:27:28.856 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:45.690162+0000", 2024-09-10T19:27:28.856 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.88091123500000001 2024-09-10T19:27:28.856 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:28.856 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:28.856 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-10T19:27:28.856 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:45.690261+0000", 2024-09-10T19:27:28.856 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.8450999999999996e-05 2024-09-10T19:27:28.856 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:28.856 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:28.856 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 3,5", 2024-09-10T19:27:28.856 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:45.690799+0000", 2024-09-10T19:27:28.856 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00053819200000000005 2024-09-10T19:27:28.856 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:28.856 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:28.857 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:27:28.857 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:45.699286+0000", 2024-09-10T19:27:28.857 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0084870320000000003 2024-09-10T19:27:28.857 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:28.857 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:28.857 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-09-10T19:27:28.857 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:45.699303+0000", 2024-09-10T19:27:28.857 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.6541999999999998e-05 2024-09-10T19:27:28.857 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:28.857 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:28.857 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:27:28.857 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:45.699588+0000", 2024-09-10T19:27:28.857 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00028581800000000002 2024-09-10T19:27:28.858 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:28.858 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:28.858 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-09-10T19:27:28.858 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:45.699656+0000", 2024-09-10T19:27:28.858 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.7737000000000006e-05 2024-09-10T19:27:28.858 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:28.858 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:28.858 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-10T19:27:28.858 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:45.700916+0000", 2024-09-10T19:27:28.858 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0012603899999999999 2024-09-10T19:27:28.858 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:27:28.858 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-10T19:27:28.858 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:27:28.858 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:28.859 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:28.859 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.15051.0:109391 8.18 8:1f03c761:::benchmark_data_smithi183_114924_object109390:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e186)", 2024-09-10T19:27:28.859 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-10T19:17:44.863381+0000", 2024-09-10T19:27:28.859 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 583.93729193599995, 2024-09-10T19:27:28.859 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.83769398399999995, 2024-09-10T19:27:28.859 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-10T19:27:28.859 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T19:27:28.859 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-10T19:27:28.859 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.15051", 2024-09-10T19:27:28.859 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.183:0/3632028672", 2024-09-10T19:27:28.859 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 109391 2024-09-10T19:27:28.859 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:28.859 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-10T19:27:28.860 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:28.860 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-10T19:27:28.860 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:44.863381+0000", 2024-09-10T19:27:28.860 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:27:28.860 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:28.860 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:28.860 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-10T19:27:28.860 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:44.863381+0000", 2024-09-10T19:27:28.860 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:27:28.860 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:28.860 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:28.860 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-10T19:27:28.860 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:44.863380+0000", 2024-09-10T19:27:28.860 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999986 2024-09-10T19:27:28.861 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:28.861 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:28.861 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-10T19:27:28.861 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:44.863412+0000", 2024-09-10T19:27:28.861 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.2434999999999997e-05 2024-09-10T19:27:28.861 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:28.861 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:28.861 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-10T19:27:28.861 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:44.863422+0000", 2024-09-10T19:27:28.861 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.6830000000000003e-06 2024-09-10T19:27:28.861 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:28.861 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:28.861 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-10T19:27:28.862 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:44.863427+0000", 2024-09-10T19:27:28.862 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.0189999999999997e-06 2024-09-10T19:27:28.862 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:28.862 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:28.862 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:27:28.862 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:44.863472+0000", 2024-09-10T19:27:28.862 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.5565000000000002e-05 2024-09-10T19:27:28.862 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:28.862 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:28.862 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for peered", 2024-09-10T19:27:28.862 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:44.863480+0000", 2024-09-10T19:27:28.862 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.1659999999999997e-06 2024-09-10T19:27:28.862 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:28.863 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:28.863 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:27:28.863 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:45.691866+0000", 2024-09-10T19:27:28.863 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.82838657800000004 2024-09-10T19:27:28.863 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:28.863 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:28.863 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-10T19:27:28.863 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:45.691983+0000", 2024-09-10T19:27:28.863 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00011704700000000001 2024-09-10T19:27:28.863 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:28.863 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:28.863 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 3,5", 2024-09-10T19:27:28.863 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:45.692178+0000", 2024-09-10T19:27:28.864 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00019426400000000001 2024-09-10T19:27:28.864 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:28.864 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:28.864 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-09-10T19:27:28.864 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:45.699323+0000", 2024-09-10T19:27:28.864 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0071453899999999997 2024-09-10T19:27:28.864 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:28.864 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:28.864 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:27:28.864 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:45.699565+0000", 2024-09-10T19:27:28.864 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00024215200000000001 2024-09-10T19:27:28.864 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:28.864 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:28.865 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:27:28.865 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:45.701004+0000", 2024-09-10T19:27:28.865 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0014388910000000001 2024-09-10T19:27:28.865 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:28.865 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:28.865 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-09-10T19:27:28.865 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:45.701044+0000", 2024-09-10T19:27:28.865 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.9573999999999998e-05 2024-09-10T19:27:28.865 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:28.865 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:28.865 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-10T19:27:28.865 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:45.701075+0000", 2024-09-10T19:27:28.865 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.1590000000000001e-05 2024-09-10T19:27:28.866 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:27:28.866 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-10T19:27:28.866 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:27:28.866 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:28.866 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:28.866 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.15051.0:109419 8.18 8:1c6092b9:::benchmark_data_smithi183_114924_object109418:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e186)", 2024-09-10T19:27:28.866 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-10T19:17:44.898684+0000", 2024-09-10T19:27:28.866 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 583.90198918700003, 2024-09-10T19:27:28.866 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.80250911300000005, 2024-09-10T19:27:28.866 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-10T19:27:28.866 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T19:27:28.866 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-10T19:27:28.866 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.15051", 2024-09-10T19:27:28.866 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.183:0/3632028672", 2024-09-10T19:27:28.867 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 109419 2024-09-10T19:27:28.867 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:28.867 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-10T19:27:28.867 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:28.867 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-10T19:27:28.867 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:44.898684+0000", 2024-09-10T19:27:28.867 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:27:28.867 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:28.867 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:28.867 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-10T19:27:28.867 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:44.898684+0000", 2024-09-10T19:27:28.867 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:27:28.867 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:28.868 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:28.868 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-10T19:27:28.868 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:44.898682+0000", 2024-09-10T19:27:28.868 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999986 2024-09-10T19:27:28.868 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:28.868 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:28.868 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-10T19:27:28.868 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:44.898720+0000", 2024-09-10T19:27:28.868 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.7280000000000002e-05 2024-09-10T19:27:28.868 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:28.868 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:28.869 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-10T19:27:28.869 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:44.898731+0000", 2024-09-10T19:27:28.869 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.1309e-05 2024-09-10T19:27:28.869 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:28.869 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:28.869 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-10T19:27:28.869 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:44.898737+0000", 2024-09-10T19:27:28.869 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.8109999999999999e-06 2024-09-10T19:27:28.869 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:28.869 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:28.869 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:27:28.869 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:44.898792+0000", 2024-09-10T19:27:28.869 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.5478e-05 2024-09-10T19:27:28.870 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:28.870 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:28.870 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for peered", 2024-09-10T19:27:28.870 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:44.898820+0000", 2024-09-10T19:27:28.870 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.8115000000000001e-05 2024-09-10T19:27:28.870 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:28.870 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:28.870 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:27:28.870 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:45.693210+0000", 2024-09-10T19:27:28.870 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.794389656 2024-09-10T19:27:28.870 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:28.870 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:28.870 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-10T19:27:28.870 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:45.693275+0000", 2024-09-10T19:27:28.871 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.5479000000000001e-05 2024-09-10T19:27:28.871 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:28.871 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:28.871 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 3,5", 2024-09-10T19:27:28.871 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:45.693420+0000", 2024-09-10T19:27:28.871 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00014434699999999999 2024-09-10T19:27:28.871 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:28.871 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:28.871 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-09-10T19:27:28.871 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:45.699328+0000", 2024-09-10T19:27:28.871 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0059082379999999997 2024-09-10T19:27:28.871 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:28.871 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:28.871 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:27:28.872 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:45.700968+0000", 2024-09-10T19:27:28.872 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.001639869 2024-09-10T19:27:28.872 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:28.872 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:28.872 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:27:28.872 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:45.701133+0000", 2024-09-10T19:27:28.872 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00016527100000000001 2024-09-10T19:27:28.872 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:28.872 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:28.872 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-09-10T19:27:28.872 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:45.701166+0000", 2024-09-10T19:27:28.873 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.2764e-05 2024-09-10T19:27:28.873 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:28.873 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:28.873 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-10T19:27:28.873 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:45.701193+0000", 2024-09-10T19:27:28.873 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.7086000000000001e-05 2024-09-10T19:27:28.873 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:27:28.873 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-10T19:27:28.873 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:27:28.873 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:28.873 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:28.874 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.15051.0:109435 8.18 8:19ceba12:::benchmark_data_smithi183_114924_object109434:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e186)", 2024-09-10T19:27:28.874 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-10T19:17:44.932644+0000", 2024-09-10T19:27:28.874 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 583.86802919599995, 2024-09-10T19:27:28.874 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.78822556499999996, 2024-09-10T19:27:28.874 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-10T19:27:28.874 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T19:27:28.874 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-10T19:27:28.874 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.15051", 2024-09-10T19:27:28.874 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.183:0/3632028672", 2024-09-10T19:27:28.874 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 109435 2024-09-10T19:27:28.874 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:28.874 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-10T19:27:28.874 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:28.875 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-10T19:27:28.875 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:44.932644+0000", 2024-09-10T19:27:28.875 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:27:28.875 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:28.875 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:28.875 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-10T19:27:28.875 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:44.932644+0000", 2024-09-10T19:27:28.875 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:27:28.875 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:28.875 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:28.875 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-10T19:27:28.875 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:44.932642+0000", 2024-09-10T19:27:28.875 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999986 2024-09-10T19:27:28.876 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:28.876 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:28.876 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-10T19:27:28.876 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:44.932682+0000", 2024-09-10T19:27:28.876 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.9594000000000001e-05 2024-09-10T19:27:28.876 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:28.876 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:28.876 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-10T19:27:28.876 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:44.932693+0000", 2024-09-10T19:27:28.876 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.1331e-05 2024-09-10T19:27:28.876 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:28.876 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:28.876 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-10T19:27:28.876 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:44.932698+0000", 2024-09-10T19:27:28.877 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.9490000000000004e-06 2024-09-10T19:27:28.877 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:28.877 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:28.877 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:27:28.877 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:44.932741+0000", 2024-09-10T19:27:28.877 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.2579e-05 2024-09-10T19:27:28.877 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:28.877 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:28.877 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for peered", 2024-09-10T19:27:28.877 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:44.932748+0000", 2024-09-10T19:27:28.877 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.951e-06 2024-09-10T19:27:28.877 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:28.877 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:28.878 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:27:28.878 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:45.697245+0000", 2024-09-10T19:27:28.878 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.76449673299999998 2024-09-10T19:27:28.878 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:28.878 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:28.878 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-10T19:27:28.878 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:45.697364+0000", 2024-09-10T19:27:28.878 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000119673 2024-09-10T19:27:28.878 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:28.878 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:28.878 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 3,5", 2024-09-10T19:27:28.878 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:45.697634+0000", 2024-09-10T19:27:28.878 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000269776 2024-09-10T19:27:28.879 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:28.879 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:28.879 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:27:28.879 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:45.701106+0000", 2024-09-10T19:27:28.879 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0034719830000000001 2024-09-10T19:27:28.879 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:28.879 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:28.879 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:27:28.879 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:45.704757+0000", 2024-09-10T19:27:28.879 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0036509709999999998 2024-09-10T19:27:28.879 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:28.879 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:28.879 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-09-10T19:27:28.879 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:45.720792+0000", 2024-09-10T19:27:28.880 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.016035489 2024-09-10T19:27:28.880 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:28.880 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:28.880 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-09-10T19:27:28.880 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:45.720834+0000", 2024-09-10T19:27:28.880 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.1683000000000001e-05 2024-09-10T19:27:28.880 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:28.880 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:28.880 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-10T19:27:28.880 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:45.720869+0000", 2024-09-10T19:27:28.880 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.5293999999999998e-05 2024-09-10T19:27:28.880 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:27:28.881 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-10T19:27:28.881 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:27:28.881 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:28.881 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:28.881 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.15051.0:109452 8.18 8:1a60d4e7:::benchmark_data_smithi183_114924_object109451:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e186)", 2024-09-10T19:27:28.881 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-10T19:17:44.975994+0000", 2024-09-10T19:27:28.881 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 583.82467949600004, 2024-09-10T19:27:28.881 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.74494188400000005, 2024-09-10T19:27:28.881 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-10T19:27:28.882 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T19:27:28.882 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-10T19:27:28.882 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.15051", 2024-09-10T19:27:28.882 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.183:0/3632028672", 2024-09-10T19:27:28.882 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 109452 2024-09-10T19:27:28.882 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:28.882 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-10T19:27:28.882 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:28.882 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-10T19:27:28.882 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:44.975994+0000", 2024-09-10T19:27:28.882 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:27:28.882 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:28.883 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:28.883 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-10T19:27:28.883 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:44.975994+0000", 2024-09-10T19:27:28.883 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:27:28.883 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:28.883 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:28.883 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-10T19:27:28.883 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:44.975990+0000", 2024-09-10T19:27:28.883 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999962 2024-09-10T19:27:28.883 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:28.883 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:28.883 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-10T19:27:28.883 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:44.976043+0000", 2024-09-10T19:27:28.884 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.2958999999999999e-05 2024-09-10T19:27:28.884 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:28.884 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:28.884 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-10T19:27:28.884 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:44.976056+0000", 2024-09-10T19:27:28.884 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.3149999999999999e-05 2024-09-10T19:27:28.884 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:28.884 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:28.884 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-10T19:27:28.884 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:44.976062+0000", 2024-09-10T19:27:28.884 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.02e-06 2024-09-10T19:27:28.884 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:28.884 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:28.885 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:27:28.885 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:44.976098+0000", 2024-09-10T19:27:28.885 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.6434999999999999e-05 2024-09-10T19:27:28.885 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:28.885 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:28.885 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for peered", 2024-09-10T19:27:28.885 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:44.976103+0000", 2024-09-10T19:27:28.885 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.2290000000000002e-06 2024-09-10T19:27:28.885 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:28.885 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:28.885 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:27:28.885 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:45.701230+0000", 2024-09-10T19:27:28.885 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.72512634499999995 2024-09-10T19:27:28.886 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:28.886 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:28.886 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-10T19:27:28.886 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:45.701394+0000", 2024-09-10T19:27:28.886 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00016436399999999999 2024-09-10T19:27:28.886 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:28.886 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:28.886 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 3,5", 2024-09-10T19:27:28.886 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:45.701706+0000", 2024-09-10T19:27:28.886 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00031236099999999999 2024-09-10T19:27:28.886 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:28.886 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:28.886 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:27:28.886 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:45.712548+0000", 2024-09-10T19:27:28.886 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.010841368000000001 2024-09-10T19:27:28.887 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:28.887 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:28.887 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:27:28.887 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:45.712578+0000", 2024-09-10T19:27:28.887 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.0361999999999998e-05 2024-09-10T19:27:28.887 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:28.887 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:28.887 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-09-10T19:27:28.887 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:45.720877+0000", 2024-09-10T19:27:28.887 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0082986919999999999 2024-09-10T19:27:28.887 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:28.887 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:28.887 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-09-10T19:27:28.887 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:45.720909+0000", 2024-09-10T19:27:28.887 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.1770000000000002e-05 2024-09-10T19:27:28.888 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:28.888 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:28.888 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-10T19:27:28.888 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:45.720935+0000", 2024-09-10T19:27:28.888 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.6798999999999999e-05 2024-09-10T19:27:28.888 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:27:28.888 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-10T19:27:28.888 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:27:28.888 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:28.888 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:28.888 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.15051.0:109470 8.18 8:1ec8e40c:::benchmark_data_smithi183_114924_object109469:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e186)", 2024-09-10T19:27:28.888 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-10T19:17:45.022852+0000", 2024-09-10T19:27:28.888 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 583.77782062300003, 2024-09-10T19:27:28.889 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.69814480099999998, 2024-09-10T19:27:28.889 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-10T19:27:28.889 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T19:27:28.889 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-10T19:27:28.889 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.15051", 2024-09-10T19:27:28.889 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.183:0/3632028672", 2024-09-10T19:27:28.889 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 109470 2024-09-10T19:27:28.889 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:28.889 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-10T19:27:28.889 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:28.889 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-10T19:27:28.889 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:45.022852+0000", 2024-09-10T19:27:28.889 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:27:28.890 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:28.890 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:28.890 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-10T19:27:28.890 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:45.022852+0000", 2024-09-10T19:27:28.890 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:27:28.890 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:28.890 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:28.890 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-10T19:27:28.890 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:45.022851+0000", 2024-09-10T19:27:28.890 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999986 2024-09-10T19:27:28.890 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:28.890 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:28.891 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-10T19:27:28.891 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:45.022896+0000", 2024-09-10T19:27:28.891 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.4920999999999999e-05 2024-09-10T19:27:28.891 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:28.891 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:28.891 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-10T19:27:28.891 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:45.022908+0000", 2024-09-10T19:27:28.891 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.2568000000000001e-05 2024-09-10T19:27:28.891 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:28.891 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:28.891 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-10T19:27:28.891 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:45.022914+0000", 2024-09-10T19:27:28.891 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.0680000000000002e-06 2024-09-10T19:27:28.891 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:28.892 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:28.892 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:27:28.892 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:45.022964+0000", 2024-09-10T19:27:28.892 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.9564999999999997e-05 2024-09-10T19:27:28.892 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:28.892 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:28.892 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for peered", 2024-09-10T19:27:28.892 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:45.022971+0000", 2024-09-10T19:27:28.892 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.8750000000000002e-06 2024-09-10T19:27:28.892 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:28.892 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:28.892 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:27:28.892 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:45.703155+0000", 2024-09-10T19:27:28.892 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.68018371 2024-09-10T19:27:28.893 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:28.893 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:28.893 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-10T19:27:28.893 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:45.703266+0000", 2024-09-10T19:27:28.893 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00011097799999999999 2024-09-10T19:27:28.893 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:28.893 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:28.893 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 3,5", 2024-09-10T19:27:28.893 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:45.703506+0000", 2024-09-10T19:27:28.893 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00024089599999999999 2024-09-10T19:27:28.893 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:28.893 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:28.893 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:27:28.894 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:45.712612+0000", 2024-09-10T19:27:28.894 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0091058879999999995 2024-09-10T19:27:28.894 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:28.894 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:28.894 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:27:28.894 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:45.712640+0000", 2024-09-10T19:27:28.894 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.7667000000000001e-05 2024-09-10T19:27:28.894 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:28.894 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:28.894 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-09-10T19:27:28.894 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:45.720941+0000", 2024-09-10T19:27:28.894 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0083011660000000004 2024-09-10T19:27:28.894 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:28.894 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:28.895 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-09-10T19:27:28.895 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:45.720971+0000", 2024-09-10T19:27:28.895 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.0202e-05 2024-09-10T19:27:28.895 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:28.895 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:28.895 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-10T19:27:28.895 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:45.720997+0000", 2024-09-10T19:27:28.895 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.584e-05 2024-09-10T19:27:28.895 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:27:28.895 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-10T19:27:28.895 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:27:28.895 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:28.895 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:28.896 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.15360.0:169787 9.4 9:2e2a4fb2:::benchmark_data_smithi183_142499_object23298:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e263)", 2024-09-10T19:27:28.896 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-10T19:21:46.995886+0000", 2024-09-10T19:27:28.896 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 341.804786756, 2024-09-10T19:27:28.896 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.0085892139999999, 2024-09-10T19:27:28.896 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-10T19:27:28.896 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T19:27:28.896 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-10T19:27:28.896 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.15360", 2024-09-10T19:27:28.896 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.183:0/2364804131", 2024-09-10T19:27:28.896 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 169787 2024-09-10T19:27:28.896 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:28.896 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-10T19:27:28.897 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:28.897 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-10T19:27:28.897 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:21:46.995886+0000", 2024-09-10T19:27:28.897 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:27:28.897 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:28.897 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:28.897 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-10T19:27:28.897 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:21:46.995886+0000", 2024-09-10T19:27:28.897 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:27:28.898 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:28.898 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:28.898 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-10T19:27:28.898 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:21:46.995885+0000", 2024-09-10T19:27:28.898 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999986 2024-09-10T19:27:28.898 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:28.898 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:28.898 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-10T19:27:28.898 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:21:46.995892+0000", 2024-09-10T19:27:28.898 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.9489999999999999e-06 2024-09-10T19:27:28.898 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:28.898 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:28.898 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-10T19:27:28.899 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:21:46.995908+0000", 2024-09-10T19:27:28.899 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.5848000000000001e-05 2024-09-10T19:27:28.899 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:28.899 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:28.899 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-10T19:27:28.899 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:21:46.995913+0000", 2024-09-10T19:27:28.899 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.361e-06 2024-09-10T19:27:28.899 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:28.899 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:28.899 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:27:28.899 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:21:46.995963+0000", 2024-09-10T19:27:28.899 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.9731000000000002e-05 2024-09-10T19:27:28.899 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:28.899 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:28.900 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for peered", 2024-09-10T19:27:28.900 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:21:46.995969+0000", 2024-09-10T19:27:28.900 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.9789999999999998e-06 2024-09-10T19:27:28.900 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:28.900 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:28.900 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:27:28.900 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:21:48.000327+0000", 2024-09-10T19:27:28.900 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.0043578500000001 2024-09-10T19:27:28.900 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:28.900 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:28.900 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-10T19:27:28.900 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:21:48.000433+0000", 2024-09-10T19:27:28.900 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000106162 2024-09-10T19:27:28.900 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:28.901 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:28.901 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 3", 2024-09-10T19:27:28.901 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:21:48.000552+0000", 2024-09-10T19:27:28.901 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00011955099999999999 2024-09-10T19:27:28.901 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:28.901 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:28.901 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-09-10T19:27:28.901 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:21:48.003555+0000", 2024-09-10T19:27:28.901 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0030022439999999998 2024-09-10T19:27:28.901 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:28.901 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:28.901 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:27:28.901 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:21:48.004276+0000", 2024-09-10T19:27:28.901 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00072138699999999998 2024-09-10T19:27:28.902 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:28.902 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:28.902 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-09-10T19:27:28.902 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:21:48.004394+0000", 2024-09-10T19:27:28.902 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000118077 2024-09-10T19:27:28.902 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:28.902 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:28.902 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-10T19:27:28.902 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:21:48.004476+0000", 2024-09-10T19:27:28.902 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.1513000000000004e-05 2024-09-10T19:27:28.902 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:27:28.902 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-10T19:27:28.903 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:27:28.903 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:28.903 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:28.903 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.14427.0:1797 5.e 5:7759931f:::notify.3:head [watch reconnect cookie 94646863662448 gen 1] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e263)", 2024-09-10T19:27:28.903 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-10T19:21:47.374947+0000", 2024-09-10T19:27:28.903 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 341.42572572799997, 2024-09-10T19:27:28.903 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.62052324699999994, 2024-09-10T19:27:28.903 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-10T19:27:28.903 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "started", 2024-09-10T19:27:28.903 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-10T19:27:28.903 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.14427", 2024-09-10T19:27:28.903 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.183:0/3843093913", 2024-09-10T19:27:28.904 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 1797 2024-09-10T19:27:28.904 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:28.904 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-10T19:27:28.904 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:28.904 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-10T19:27:28.904 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:21:47.374947+0000", 2024-09-10T19:27:28.904 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:27:28.904 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:28.904 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:28.904 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-10T19:27:28.904 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:21:47.374947+0000", 2024-09-10T19:27:28.904 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:27:28.904 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:28.904 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:28.905 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-10T19:27:28.905 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:21:47.374946+0000", 2024-09-10T19:27:28.905 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999981 2024-09-10T19:27:28.905 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:28.905 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:28.905 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-10T19:27:28.905 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:21:47.374953+0000", 2024-09-10T19:27:28.905 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.968e-06 2024-09-10T19:27:28.905 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:28.905 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:28.905 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-10T19:27:28.905 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:21:47.374967+0000", 2024-09-10T19:27:28.906 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.4452e-05 2024-09-10T19:27:28.906 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:28.906 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:28.906 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-10T19:27:28.906 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:21:47.374974+0000", 2024-09-10T19:27:28.906 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.7719999999999997e-06 2024-09-10T19:27:28.906 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:28.906 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:28.906 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:27:28.906 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:21:47.375036+0000", 2024-09-10T19:27:28.906 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.1804999999999995e-05 2024-09-10T19:27:28.906 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:28.906 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:28.907 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for peered", 2024-09-10T19:27:28.907 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:21:47.375046+0000", 2024-09-10T19:27:28.907 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.9559999999999992e-06 2024-09-10T19:27:28.907 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:28.907 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:28.907 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:27:28.907 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:21:47.981476+0000", 2024-09-10T19:27:28.907 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.60643018599999998 2024-09-10T19:27:28.907 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:28.907 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:28.907 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-10T19:27:28.907 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:21:47.985671+0000", 2024-09-10T19:27:28.907 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0041950080000000001 2024-09-10T19:27:28.907 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:28.907 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:28.908 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-10T19:27:28.908 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:21:47.995471+0000", 2024-09-10T19:27:28.908 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0097998219999999997 2024-09-10T19:27:28.908 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:27:28.908 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-10T19:27:28.908 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:27:28.908 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:28.908 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:28.908 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.25622.0:12642 10.1 10:878e127a:::benchmark_data_smithi183_176537_object12641:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e307)", 2024-09-10T19:27:28.908 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-10T19:24:28.950229+0000", 2024-09-10T19:27:28.908 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 179.85044424700001, 2024-09-10T19:27:28.908 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.93090725299999999, 2024-09-10T19:27:28.908 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-10T19:27:28.909 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T19:27:28.909 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-10T19:27:28.909 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.25622", 2024-09-10T19:27:28.909 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.183:0/3685189199", 2024-09-10T19:27:28.909 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 12642 2024-09-10T19:27:28.909 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:28.909 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-10T19:27:28.909 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:28.909 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-10T19:27:28.909 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:24:28.950229+0000", 2024-09-10T19:27:28.909 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:27:28.909 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:28.909 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:28.910 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-10T19:27:28.910 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:24:28.950229+0000", 2024-09-10T19:27:28.910 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:27:28.910 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:28.910 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:28.910 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-10T19:27:28.910 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:24:28.950228+0000", 2024-09-10T19:27:28.910 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.999999 2024-09-10T19:27:28.910 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:28.910 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:28.910 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-10T19:27:28.910 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:24:28.950301+0000", 2024-09-10T19:27:28.910 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.3622999999999996e-05 2024-09-10T19:27:28.910 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:28.911 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:28.911 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-10T19:27:28.911 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:24:28.950311+0000", 2024-09-10T19:27:28.911 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.8099999999999992e-06 2024-09-10T19:27:28.911 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:28.911 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:28.911 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-10T19:27:28.911 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:24:28.950327+0000", 2024-09-10T19:27:28.911 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.5585000000000001e-05 2024-09-10T19:27:28.911 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:28.911 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:28.911 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:27:28.911 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:24:28.950380+0000", 2024-09-10T19:27:28.912 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.2664000000000003e-05 2024-09-10T19:27:28.912 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:28.912 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:28.912 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for peered", 2024-09-10T19:27:28.912 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:24:28.950389+0000", 2024-09-10T19:27:28.912 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.1279999999999997e-06 2024-09-10T19:27:28.912 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:28.912 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:28.912 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:27:28.912 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:24:29.876886+0000", 2024-09-10T19:27:28.912 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.92649711400000001 2024-09-10T19:27:28.912 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:28.912 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:28.913 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-10T19:27:28.913 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:24:29.877049+0000", 2024-09-10T19:27:28.913 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00016302499999999999 2024-09-10T19:27:28.913 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:28.913 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:28.913 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0,7", 2024-09-10T19:27:28.913 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:24:29.877195+0000", 2024-09-10T19:27:28.913 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00014664699999999999 2024-09-10T19:27:28.913 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:28.913 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:28.913 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-09-10T19:27:28.914 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:24:29.879270+0000", 2024-09-10T19:27:28.914 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0020743189999999998 2024-09-10T19:27:28.914 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:28.914 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:28.914 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:27:28.914 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:24:29.880080+0000", 2024-09-10T19:27:28.914 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00081002500000000002 2024-09-10T19:27:28.914 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:28.914 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:28.914 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:27:28.915 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:24:29.881044+0000", 2024-09-10T19:27:28.915 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00096470200000000001 2024-09-10T19:27:28.915 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:28.915 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:28.915 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-09-10T19:27:28.915 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:24:29.881096+0000", 2024-09-10T19:27:28.915 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.1672999999999998e-05 2024-09-10T19:27:28.915 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:28.915 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:28.916 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-10T19:27:28.916 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:24:29.881136+0000", 2024-09-10T19:27:28.916 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.9947000000000002e-05 2024-09-10T19:27:28.916 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:27:28.916 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-10T19:27:28.916 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:27:28.916 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:28.917 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:28.917 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.25622.0:12643 10.1 10:8a8984f9:::benchmark_data_smithi183_176537_object12642:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e307)", 2024-09-10T19:27:28.917 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-10T19:24:28.950343+0000", 2024-09-10T19:27:28.917 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 179.850330378, 2024-09-10T19:27:28.917 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.93261173900000005, 2024-09-10T19:27:28.917 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-10T19:27:28.918 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T19:27:28.918 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-10T19:27:28.918 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.25622", 2024-09-10T19:27:28.918 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.183:0/3685189199", 2024-09-10T19:27:28.918 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 12643 2024-09-10T19:27:28.918 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:28.918 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-10T19:27:28.919 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:28.919 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-10T19:27:28.919 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:24:28.950343+0000", 2024-09-10T19:27:28.919 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:27:28.919 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:28.919 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:28.920 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-10T19:27:28.920 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:24:28.950343+0000", 2024-09-10T19:27:28.920 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:27:28.920 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:28.920 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:28.920 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-10T19:27:28.920 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:24:28.950340+0000", 2024-09-10T19:27:28.920 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999976 2024-09-10T19:27:28.920 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:28.921 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:28.921 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-10T19:27:28.921 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:24:28.950372+0000", 2024-09-10T19:27:28.921 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.1443999999999999e-05 2024-09-10T19:27:28.921 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:28.921 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:28.921 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-10T19:27:28.921 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:24:28.950380+0000", 2024-09-10T19:27:28.921 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.1580000000000005e-06 2024-09-10T19:27:28.921 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:28.921 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:28.921 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-10T19:27:28.921 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:24:28.950385+0000", 2024-09-10T19:27:28.922 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.6299999999999997e-06 2024-09-10T19:27:28.922 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:28.922 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:28.922 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:27:28.922 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:24:28.950413+0000", 2024-09-10T19:27:28.922 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.8068999999999998e-05 2024-09-10T19:27:28.922 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:28.922 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:28.922 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for peered", 2024-09-10T19:27:28.922 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:24:28.950419+0000", 2024-09-10T19:27:28.922 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.4810000000000003e-06 2024-09-10T19:27:28.922 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:28.922 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:28.922 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:27:28.923 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:24:29.878269+0000", 2024-09-10T19:27:28.923 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.92784950099999997 2024-09-10T19:27:28.923 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:28.923 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:28.923 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-10T19:27:28.923 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:24:29.878370+0000", 2024-09-10T19:27:28.923 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000101264 2024-09-10T19:27:28.923 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:28.923 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:28.923 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0,7", 2024-09-10T19:27:28.924 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:24:29.878628+0000", 2024-09-10T19:27:28.924 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00025846199999999999 2024-09-10T19:27:28.924 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:28.924 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:28.924 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:27:28.924 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:24:29.880115+0000", 2024-09-10T19:27:28.924 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0014862079999999999 2024-09-10T19:27:28.924 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:28.924 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:28.924 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-09-10T19:27:28.924 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:24:29.880545+0000", 2024-09-10T19:27:28.924 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00043079300000000001 2024-09-10T19:27:28.924 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:28.925 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:28.925 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:27:28.925 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:24:29.882793+0000", 2024-09-10T19:27:28.925 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.002247911 2024-09-10T19:27:28.925 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:28.925 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:28.925 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:27:28.926 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:27:28.927 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-09-10T19:27:28.927 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:24:29.882902+0000", 2024-09-10T19:27:28.927 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000108947 2024-09-10T19:27:28.927 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:28.927 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:28.927 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-10T19:27:28.927 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:24:29.882954+0000", 2024-09-10T19:27:28.927 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.2252e-05 2024-09-10T19:27:28.927 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:27:28.927 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-10T19:27:28.927 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:27:28.927 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:28.928 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:28.928 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.25622.0:12673 10.1 10:80f8aab5:::benchmark_data_smithi183_176537_object12672:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e307)", 2024-09-10T19:27:28.928 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-10T19:24:28.964234+0000", 2024-09-10T19:27:28.928 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 179.836438609, 2024-09-10T19:27:28.928 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.91885601699999997, 2024-09-10T19:27:28.928 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-10T19:27:28.928 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T19:27:28.928 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-10T19:27:28.928 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.25622", 2024-09-10T19:27:28.928 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.183:0/3685189199", 2024-09-10T19:27:28.928 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 12673 2024-09-10T19:27:28.928 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:28.928 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-10T19:27:28.928 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:28.929 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-10T19:27:28.929 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:24:28.964234+0000", 2024-09-10T19:27:28.929 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:27:28.929 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:28.929 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:28.929 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-10T19:27:28.929 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:24:28.964234+0000", 2024-09-10T19:27:28.929 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:27:28.929 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:28.929 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:28.929 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-10T19:27:28.929 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:24:28.964234+0000", 2024-09-10T19:27:28.929 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.999999 2024-09-10T19:27:28.930 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:28.930 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:28.930 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-10T19:27:28.930 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:24:28.964333+0000", 2024-09-10T19:27:28.930 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.8970999999999998e-05 2024-09-10T19:27:28.930 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:28.930 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:28.930 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-10T19:27:28.930 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:24:28.964341+0000", 2024-09-10T19:27:28.930 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.9559999999999997e-06 2024-09-10T19:27:28.930 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:28.930 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:28.930 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-10T19:27:28.930 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:24:28.964345+0000", 2024-09-10T19:27:28.931 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.8870000000000002e-06 2024-09-10T19:27:28.931 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:28.931 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:28.931 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:27:28.931 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:24:28.964386+0000", 2024-09-10T19:27:28.931 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.1436000000000001e-05 2024-09-10T19:27:28.931 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:28.931 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:28.931 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for peered", 2024-09-10T19:27:28.931 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:24:28.964392+0000", 2024-09-10T19:27:28.931 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.998e-06 2024-09-10T19:27:28.931 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:28.931 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:28.931 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:27:28.932 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:24:29.879296+0000", 2024-09-10T19:27:28.932 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.91490435400000003 2024-09-10T19:27:28.932 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:28.932 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:28.932 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-10T19:27:28.932 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:24:29.879394+0000", 2024-09-10T19:27:28.932 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.8115999999999994e-05 2024-09-10T19:27:28.932 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:28.932 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:28.932 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0,7", 2024-09-10T19:27:28.932 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:24:29.879522+0000", 2024-09-10T19:27:28.932 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000127611 2024-09-10T19:27:28.932 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:28.932 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:28.933 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-09-10T19:27:28.933 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:24:29.880551+0000", 2024-09-10T19:27:28.933 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0010293609999999999 2024-09-10T19:27:28.933 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:28.933 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:28.933 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:27:28.933 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:24:29.881237+0000", 2024-09-10T19:27:28.933 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00068514699999999999 2024-09-10T19:27:28.933 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:28.933 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:28.933 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:27:28.933 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:24:29.883007+0000", 2024-09-10T19:27:28.933 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.001770562 2024-09-10T19:27:28.933 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:28.934 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:28.934 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-09-10T19:27:28.934 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:24:29.883055+0000", 2024-09-10T19:27:28.934 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.7886000000000002e-05 2024-09-10T19:27:28.934 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:28.934 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:28.934 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-10T19:27:28.934 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:24:29.883090+0000", 2024-09-10T19:27:28.934 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.5475999999999997e-05 2024-09-10T19:27:28.934 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:27:28.934 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-10T19:27:28.934 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:27:28.934 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:28.934 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:28.935 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.25622.0:18260 10.e 10:72e13d67:::benchmark_data_smithi183_176537_object18259:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e310)", 2024-09-10T19:27:28.935 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-10T19:24:33.282278+0000", 2024-09-10T19:27:28.935 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 175.51839491800001, 2024-09-10T19:27:28.935 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.32334280399999998, 2024-09-10T19:27:28.935 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-10T19:27:28.935 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T19:27:28.935 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-10T19:27:28.935 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.25622", 2024-09-10T19:27:28.935 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.183:0/3685189199", 2024-09-10T19:27:28.935 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 18260 2024-09-10T19:27:28.935 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:28.935 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-10T19:27:28.935 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:28.935 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-10T19:27:28.936 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:24:33.282278+0000", 2024-09-10T19:27:28.936 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:27:28.936 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:28.936 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:28.936 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-10T19:27:28.936 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:24:33.282278+0000", 2024-09-10T19:27:28.936 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:27:28.936 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:28.936 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:28.936 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-10T19:27:28.937 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:24:33.282276+0000", 2024-09-10T19:27:28.937 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999981 2024-09-10T19:27:28.937 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:28.937 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:28.937 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-10T19:27:28.937 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:24:33.282382+0000", 2024-09-10T19:27:28.937 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000105437 2024-09-10T19:27:28.937 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:28.937 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:28.937 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-10T19:27:28.937 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:24:33.282392+0000", 2024-09-10T19:27:28.938 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.058e-05 2024-09-10T19:27:28.938 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:28.938 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:28.938 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-10T19:27:28.938 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:24:33.282397+0000", 2024-09-10T19:27:28.938 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.9479999999999999e-06 2024-09-10T19:27:28.938 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:28.938 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:28.938 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:27:28.938 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:24:33.282947+0000", 2024-09-10T19:27:28.938 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00054977200000000002 2024-09-10T19:27:28.938 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:28.938 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:28.938 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-10T19:27:28.939 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:24:33.283080+0000", 2024-09-10T19:27:28.939 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000133122 2024-09-10T19:27:28.939 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:28.939 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:28.939 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 5,6", 2024-09-10T19:27:28.939 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:24:33.283207+0000", 2024-09-10T19:27:28.939 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000126589 2024-09-10T19:27:28.939 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:28.939 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:28.939 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-09-10T19:27:28.939 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:24:33.286624+0000", 2024-09-10T19:27:28.939 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.003417542 2024-09-10T19:27:28.939 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:28.939 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:28.940 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:27:28.940 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:24:33.287095+0000", 2024-09-10T19:27:28.940 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000470965 2024-09-10T19:27:28.940 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:28.940 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:28.940 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:27:28.940 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:24:33.605535+0000", 2024-09-10T19:27:28.940 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.31843990900000002 2024-09-10T19:27:28.940 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:28.940 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:28.940 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-09-10T19:27:28.940 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:24:33.605581+0000", 2024-09-10T19:27:28.940 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.5921999999999998e-05 2024-09-10T19:27:28.940 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:28.941 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:28.941 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-10T19:27:28.941 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:24:33.605621+0000", 2024-09-10T19:27:28.941 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.9728000000000003e-05 2024-09-10T19:27:28.941 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:27:28.941 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-10T19:27:28.941 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:27:28.941 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:28.941 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:28.941 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_repop(client.25622.0:18258 10.9 e310/295 10:97c589d4:::benchmark_data_smithi183_176537_object18257:head v 310'1121, mlcod=310'1119)", 2024-09-10T19:27:28.941 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-10T19:24:33.282950+0000", 2024-09-10T19:27:28.941 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 175.517723171, 2024-09-10T19:27:28.941 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.32312085099999999, 2024-09-10T19:27:28.942 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-10T19:27:28.942 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T19:27:28.942 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-10T19:27:28.942 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:28.942 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-10T19:27:28.942 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:24:33.282950+0000", 2024-09-10T19:27:28.942 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:27:28.942 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:28.942 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:28.942 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-10T19:27:28.942 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:24:33.282950+0000", 2024-09-10T19:27:28.942 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:27:28.942 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:28.942 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:28.942 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-10T19:27:28.943 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:24:33.282949+0000", 2024-09-10T19:27:28.943 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999995 2024-09-10T19:27:28.943 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:28.943 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:28.943 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-10T19:27:28.943 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:24:33.282973+0000", 2024-09-10T19:27:28.943 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.3444000000000001e-05 2024-09-10T19:27:28.943 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:28.943 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:28.943 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-10T19:27:28.943 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:24:33.282982+0000", 2024-09-10T19:27:28.943 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.9879999999999993e-06 2024-09-10T19:27:28.944 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:28.944 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:28.944 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-10T19:27:28.944 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:24:33.282987+0000", 2024-09-10T19:27:28.944 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.1200000000000001e-06 2024-09-10T19:27:28.944 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:28.944 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:28.944 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:27:28.944 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:24:33.283043+0000", 2024-09-10T19:27:28.944 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.6039999999999999e-05 2024-09-10T19:27:28.944 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:28.944 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:28.944 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-10T19:27:28.945 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:24:33.283062+0000", 2024-09-10T19:27:28.945 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.8569000000000001e-05 2024-09-10T19:27:28.945 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:28.945 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:28.945 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-09-10T19:27:28.945 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:24:33.606041+0000", 2024-09-10T19:27:28.945 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.32297986200000001 2024-09-10T19:27:28.945 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:28.945 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:28.945 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-10T19:27:28.945 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:24:33.606071+0000", 2024-09-10T19:27:28.945 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.9368000000000001e-05 2024-09-10T19:27:28.945 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:27:28.945 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-10T19:27:28.946 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:27:28.946 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:27:28.946 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-10T19:27:28.946 INFO:teuthology.orchestra.run.smithi191.stdout:} 2024-09-10T19:27:28.967 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:27:28.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:28.968 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:27:29.068 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:27:29.090 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:27:29.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:29.091 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:27:29.145 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:27:28 smithi191 ceph-mon[32884]: 2.18 scrub starts 2024-09-10T19:27:29.145 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:27:28 smithi191 ceph-mon[32884]: 2.18 scrub ok 2024-09-10T19:27:29.145 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:27:28 smithi191 ceph-mon[32884]: pgmap v797: 158 pgs: 4 active+remapped+backfill_toofull, 154 active+clean; 186 MiB data, 5.8 GiB used, 531 GiB / 536 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.10k op/s; 865/9480 objects misplaced (9.124%); 9.5 MiB/s, 152 objects/s recovering 2024-09-10T19:27:29.145 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:27:28 smithi191 ceph-mon[32884]: purged_snaps scrub starts 2024-09-10T19:27:29.145 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:27:28 smithi191 ceph-mon[32884]: purged_snaps scrub ok 2024-09-10T19:27:29.146 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:27:28 smithi191 ceph-mon[32884]: purged_snaps scrub starts 2024-09-10T19:27:29.146 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:27:28 smithi191 ceph-mon[32884]: purged_snaps scrub ok 2024-09-10T19:27:29.146 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:27:28 smithi191 ceph-mon[32884]: from='mgr.14556 ' entity='mgr.y' 2024-09-10T19:27:29.146 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:27:28 smithi191 ceph-mon[32884]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:27:29.191 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:27:29.213 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:27:29.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:29.214 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:27:29.315 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:27:29.327 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:27:28 smithi183 ceph-mon[29736]: 2.18 scrub starts 2024-09-10T19:27:29.327 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:27:28 smithi183 ceph-mon[29736]: 2.18 scrub ok 2024-09-10T19:27:29.327 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:27:28 smithi183 ceph-mon[29736]: pgmap v797: 158 pgs: 4 active+remapped+backfill_toofull, 154 active+clean; 186 MiB data, 5.8 GiB used, 531 GiB / 536 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.10k op/s; 865/9480 objects misplaced (9.124%); 9.5 MiB/s, 152 objects/s recovering 2024-09-10T19:27:29.327 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:27:28 smithi183 ceph-mon[29736]: purged_snaps scrub starts 2024-09-10T19:27:29.327 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:27:28 smithi183 ceph-mon[29736]: purged_snaps scrub ok 2024-09-10T19:27:29.327 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:27:28 smithi183 ceph-mon[29736]: purged_snaps scrub starts 2024-09-10T19:27:29.327 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:27:28 smithi183 ceph-mon[29736]: purged_snaps scrub ok 2024-09-10T19:27:29.327 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:27:28 smithi183 ceph-mon[29736]: from='mgr.14556 ' entity='mgr.y' 2024-09-10T19:27:29.327 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:27:28 smithi183 ceph-mon[29736]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:27:29.328 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:27:28 smithi183 ceph-mon[41562]: 2.18 scrub starts 2024-09-10T19:27:29.328 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:27:28 smithi183 ceph-mon[41562]: 2.18 scrub ok 2024-09-10T19:27:29.328 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:27:28 smithi183 ceph-mon[41562]: pgmap v797: 158 pgs: 4 active+remapped+backfill_toofull, 154 active+clean; 186 MiB data, 5.8 GiB used, 531 GiB / 536 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.10k op/s; 865/9480 objects misplaced (9.124%); 9.5 MiB/s, 152 objects/s recovering 2024-09-10T19:27:29.328 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:27:28 smithi183 ceph-mon[41562]: purged_snaps scrub starts 2024-09-10T19:27:29.328 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:27:28 smithi183 ceph-mon[41562]: purged_snaps scrub ok 2024-09-10T19:27:29.328 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:27:28 smithi183 ceph-mon[41562]: purged_snaps scrub starts 2024-09-10T19:27:29.328 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:27:28 smithi183 ceph-mon[41562]: purged_snaps scrub ok 2024-09-10T19:27:29.328 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:27:28 smithi183 ceph-mon[41562]: from='mgr.14556 ' entity='mgr.y' 2024-09-10T19:27:29.328 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:27:28 smithi183 ceph-mon[41562]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:27:29.338 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:27:29.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:29.339 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:27:29.440 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:27:29.462 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:27:29.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:29.463 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:27:29.563 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:27:29.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:29.585 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:27:29.585 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:27:29.645 DEBUG:teuthology.orchestra.run.smithi191:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph daemon osd.5 dump_ops_in_flight 2024-09-10T19:27:29.699 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:27:29.721 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:27:29.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:29.722 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:27:29.822 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:27:29.844 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:27:29.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:29.845 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:27:29.946 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:27:29.946 INFO:tasks.cephadm.ceph_manager.ceph:removing pool_name unique_pool_3 2024-09-10T19:27:29.947 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph osd pool rm unique_pool_3 unique_pool_3 --yes-i-really-really-mean-it 2024-09-10T19:27:29.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:29.969 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:27:29.970 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:27:29.976 INFO:teuthology.orchestra.run.smithi191.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.b/config 2024-09-10T19:27:29.993 INFO:journalctl@ceph.iscsi.iscsi.a.smithi191.stdout:Sep 10 19:27:29 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-iscsi-iscsi-a[61816]: debug there is no tcmu-runner data available 2024-09-10T19:27:30.070 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:27:30.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:30.093 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:27:30.093 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:27:30.194 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:27:30.212 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-10T19:27:30.215 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:27:30.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:30.216 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:27:30.234 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:27:29 smithi183 ceph-mon[29736]: 6.18 scrub starts 2024-09-10T19:27:30.234 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:27:29 smithi183 ceph-mon[29736]: 6.18 scrub ok 2024-09-10T19:27:30.234 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:27:29 smithi183 ceph-mon[29736]: 1.8 scrub starts 2024-09-10T19:27:30.234 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:27:29 smithi183 ceph-mon[29736]: 1.8 scrub ok 2024-09-10T19:27:30.234 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:27:29 smithi183 ceph-mon[29736]: osdmap e366: 8 total, 8 up, 6 in 2024-09-10T19:27:30.234 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:27:29 smithi183 ceph-mon[29736]: 4.10 scrub starts 2024-09-10T19:27:30.234 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:27:29 smithi183 ceph-mon[29736]: 4.10 scrub ok 2024-09-10T19:27:30.235 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:27:29 smithi183 ceph-mon[41562]: 6.18 scrub starts 2024-09-10T19:27:30.235 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:27:29 smithi183 ceph-mon[41562]: 6.18 scrub ok 2024-09-10T19:27:30.235 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:27:29 smithi183 ceph-mon[41562]: 1.8 scrub starts 2024-09-10T19:27:30.235 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:27:29 smithi183 ceph-mon[41562]: 1.8 scrub ok 2024-09-10T19:27:30.235 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:27:29 smithi183 ceph-mon[41562]: osdmap e366: 8 total, 8 up, 6 in 2024-09-10T19:27:30.235 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:27:29 smithi183 ceph-mon[41562]: 4.10 scrub starts 2024-09-10T19:27:30.235 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:27:29 smithi183 ceph-mon[41562]: 4.10 scrub ok 2024-09-10T19:27:30.251 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:27:29 smithi191 ceph-mon[32884]: 6.18 scrub starts 2024-09-10T19:27:30.251 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:27:29 smithi191 ceph-mon[32884]: 6.18 scrub ok 2024-09-10T19:27:30.251 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:27:29 smithi191 ceph-mon[32884]: 1.8 scrub starts 2024-09-10T19:27:30.251 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:27:29 smithi191 ceph-mon[32884]: 1.8 scrub ok 2024-09-10T19:27:30.251 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:27:29 smithi191 ceph-mon[32884]: osdmap e366: 8 total, 8 up, 6 in 2024-09-10T19:27:30.251 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:27:29 smithi191 ceph-mon[32884]: 4.10 scrub starts 2024-09-10T19:27:30.251 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:27:29 smithi191 ceph-mon[32884]: 4.10 scrub ok 2024-09-10T19:27:30.317 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:27:30.338 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:27:30.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:30.339 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:27:30.439 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:27:30.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:30.462 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:27:30.462 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:27:30.563 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:27:30.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:30.585 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:27:30.585 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:27:30.686 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:27:30.707 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:27:30.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:30.708 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:27:30.767 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:27:30.809 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:27:30.830 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:27:30.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:30.831 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:27:30.932 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:27:30.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:30.954 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:27:30.954 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:27:31.039 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:27:31.055 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:27:31.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:31.078 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:27:31.079 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:27:31.179 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:27:31.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:31.203 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:27:31.203 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:27:31.258 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:27:30 smithi183 ceph-mon[29736]: from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:27:31.258 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:27:30 smithi183 ceph-mon[29736]: pgmap v799: 158 pgs: 4 active+remapped+backfill_toofull, 154 active+clean; 186 MiB data, 5.8 GiB used, 531 GiB / 536 GiB avail; 735 B/s rd, 0 B/s wr, 642 op/s; 865/9480 objects misplaced (9.124%); 459 KiB/s, 7 objects/s recovering 2024-09-10T19:27:31.258 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:27:30 smithi183 ceph-mon[29736]: 6.14 scrub starts 2024-09-10T19:27:31.258 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:27:30 smithi183 ceph-mon[29736]: 6.14 scrub ok 2024-09-10T19:27:31.258 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:27:30 smithi183 ceph-mon[29736]: 5.c scrub starts 2024-09-10T19:27:31.258 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:27:30 smithi183 ceph-mon[29736]: 5.c scrub ok 2024-09-10T19:27:31.259 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:27:30 smithi183 ceph-mon[41562]: from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:27:31.259 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:27:30 smithi183 ceph-mon[41562]: pgmap v799: 158 pgs: 4 active+remapped+backfill_toofull, 154 active+clean; 186 MiB data, 5.8 GiB used, 531 GiB / 536 GiB avail; 735 B/s rd, 0 B/s wr, 642 op/s; 865/9480 objects misplaced (9.124%); 459 KiB/s, 7 objects/s recovering 2024-09-10T19:27:31.259 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:27:30 smithi183 ceph-mon[41562]: 6.14 scrub starts 2024-09-10T19:27:31.259 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:27:30 smithi183 ceph-mon[41562]: 6.14 scrub ok 2024-09-10T19:27:31.259 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:27:30 smithi183 ceph-mon[41562]: 5.c scrub starts 2024-09-10T19:27:31.259 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:27:30 smithi183 ceph-mon[41562]: 5.c scrub ok 2024-09-10T19:27:31.304 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:27:31.326 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:27:31.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:31.327 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:27:31.428 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:27:31.451 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:27:31.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:31.452 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:27:31.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:27:30 smithi191 ceph-mon[32884]: from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:27:31.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:27:30 smithi191 ceph-mon[32884]: pgmap v799: 158 pgs: 4 active+remapped+backfill_toofull, 154 active+clean; 186 MiB data, 5.8 GiB used, 531 GiB / 536 GiB avail; 735 B/s rd, 0 B/s wr, 642 op/s; 865/9480 objects misplaced (9.124%); 459 KiB/s, 7 objects/s recovering 2024-09-10T19:27:31.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:27:30 smithi191 ceph-mon[32884]: 6.14 scrub starts 2024-09-10T19:27:31.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:27:30 smithi191 ceph-mon[32884]: 6.14 scrub ok 2024-09-10T19:27:31.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:27:30 smithi191 ceph-mon[32884]: 5.c scrub starts 2024-09-10T19:27:31.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:27:30 smithi191 ceph-mon[32884]: 5.c scrub ok 2024-09-10T19:27:31.553 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:27:31.554 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Sep 10 19:27:31 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-y[29989]: ::ffff:172.21.15.191 - - [10/Sep/2024:19:27:31] "GET /metrics HTTP/1.1" 200 224841 "" "Prometheus/2.43.0" 2024-09-10T19:27:31.576 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:27:31.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:31.577 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:27:31.677 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:27:31.702 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:27:31.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:31.703 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:27:31.804 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:27:31.825 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:27:31.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:31.827 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:27:31.927 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:27:31.949 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:27:31.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:31.950 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:27:32.002 INFO:teuthology.orchestra.run.smithi183.stderr:noscrub is set 2024-09-10T19:27:32.051 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:27:32.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:32.073 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:27:32.073 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:27:32.174 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:27:32.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:27:31 smithi183 ceph-mon[29736]: from='client.? 172.21.15.183:0/719795194' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-10T19:27:32.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:27:31 smithi183 ceph-mon[29736]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-10T19:27:32.181 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:27:31 smithi183 ceph-mon[41562]: from='client.? 172.21.15.183:0/719795194' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-10T19:27:32.181 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:27:31 smithi183 ceph-mon[41562]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-10T19:27:32.195 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:27:32.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:32.196 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:27:32.297 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:27:32.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:32.320 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:27:32.320 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:27:32.421 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:27:32.435 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:27:31 smithi191 ceph-mon[32884]: from='client.? 172.21.15.183:0/719795194' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-10T19:27:32.435 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:27:31 smithi191 ceph-mon[32884]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-10T19:27:32.447 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:27:32.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:32.448 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:27:32.549 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:27:32.570 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:27:32.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:32.571 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:27:32.672 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:27:32.693 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:27:32.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:32.694 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:27:32.795 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:27:32.818 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:27:32.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:32.819 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:27:32.919 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:27:32.944 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:27:32.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:32.945 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:27:33.046 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:27:33.068 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:27:33.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:33.069 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:27:33.170 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:27:33.192 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:27:33.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:33.193 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:27:33.249 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:27:32 smithi191 ceph-mon[32884]: 1.6 scrub starts 2024-09-10T19:27:33.249 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:27:32 smithi191 ceph-mon[32884]: 1.6 scrub ok 2024-09-10T19:27:33.249 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:27:32 smithi191 ceph-mon[32884]: 5.1b scrub starts 2024-09-10T19:27:33.249 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:27:32 smithi191 ceph-mon[32884]: 5.1b scrub ok 2024-09-10T19:27:33.249 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:27:32 smithi191 ceph-mon[32884]: pgmap v800: 158 pgs: 4 active+remapped+backfill_toofull, 154 active+clean; 47 MiB data, 5.2 GiB used, 531 GiB / 536 GiB avail; 639 B/s rd, 0 B/s wr, 837 op/s; 109/2817 objects misplaced (3.869%); 400 KiB/s, 6 objects/s recovering 2024-09-10T19:27:33.249 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:27:32 smithi191 ceph-mon[32884]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T19:27:33.249 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:27:32 smithi191 ceph-mon[32884]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-10T19:27:33.249 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:27:32 smithi191 ceph-mon[32884]: osdmap e367: 8 total, 8 up, 6 in 2024-09-10T19:27:33.249 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:27:32 smithi191 ceph-mon[32884]: 6.5 scrub starts 2024-09-10T19:27:33.294 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:27:33.316 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:27:33.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:33.318 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:27:33.419 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:27:33.429 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:27:32 smithi183 ceph-mon[29736]: 1.6 scrub starts 2024-09-10T19:27:33.429 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:27:32 smithi183 ceph-mon[29736]: 1.6 scrub ok 2024-09-10T19:27:33.429 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:27:32 smithi183 ceph-mon[29736]: 5.1b scrub starts 2024-09-10T19:27:33.429 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:27:32 smithi183 ceph-mon[29736]: 5.1b scrub ok 2024-09-10T19:27:33.429 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:27:32 smithi183 ceph-mon[29736]: pgmap v800: 158 pgs: 4 active+remapped+backfill_toofull, 154 active+clean; 47 MiB data, 5.2 GiB used, 531 GiB / 536 GiB avail; 639 B/s rd, 0 B/s wr, 837 op/s; 109/2817 objects misplaced (3.869%); 400 KiB/s, 6 objects/s recovering 2024-09-10T19:27:33.429 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:27:32 smithi183 ceph-mon[29736]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T19:27:33.430 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:27:32 smithi183 ceph-mon[29736]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-10T19:27:33.430 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:27:32 smithi183 ceph-mon[29736]: osdmap e367: 8 total, 8 up, 6 in 2024-09-10T19:27:33.430 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:27:32 smithi183 ceph-mon[29736]: 6.5 scrub starts 2024-09-10T19:27:33.430 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:27:32 smithi183 ceph-mon[41562]: 1.6 scrub starts 2024-09-10T19:27:33.430 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:27:32 smithi183 ceph-mon[41562]: 1.6 scrub ok 2024-09-10T19:27:33.430 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:27:32 smithi183 ceph-mon[41562]: 5.1b scrub starts 2024-09-10T19:27:33.430 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:27:32 smithi183 ceph-mon[41562]: 5.1b scrub ok 2024-09-10T19:27:33.430 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:27:32 smithi183 ceph-mon[41562]: pgmap v800: 158 pgs: 4 active+remapped+backfill_toofull, 154 active+clean; 47 MiB data, 5.2 GiB used, 531 GiB / 536 GiB avail; 639 B/s rd, 0 B/s wr, 837 op/s; 109/2817 objects misplaced (3.869%); 400 KiB/s, 6 objects/s recovering 2024-09-10T19:27:33.430 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:27:32 smithi183 ceph-mon[41562]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T19:27:33.430 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:27:32 smithi183 ceph-mon[41562]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-10T19:27:33.430 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:27:32 smithi183 ceph-mon[41562]: osdmap e367: 8 total, 8 up, 6 in 2024-09-10T19:27:33.430 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:27:32 smithi183 ceph-mon[41562]: 6.5 scrub starts 2024-09-10T19:27:33.440 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:27:33.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:33.441 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:27:33.451 INFO:teuthology.orchestra.run.smithi191.stdout:{ 2024-09-10T19:27:33.451 INFO:teuthology.orchestra.run.smithi191.stdout: "ops": [], 2024-09-10T19:27:33.451 INFO:teuthology.orchestra.run.smithi191.stdout: "num_ops": 0 2024-09-10T19:27:33.451 INFO:teuthology.orchestra.run.smithi191.stdout:} 2024-09-10T19:27:33.542 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:27:33.566 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:27:33.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:33.567 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:27:33.668 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:27:33.692 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:27:33.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:33.693 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:27:33.794 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:27:33.818 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:27:33.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:33.819 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:27:33.920 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:27:33.942 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:27:33.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:33.943 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:27:34.044 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:27:34.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:34.069 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:27:34.069 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:27:34.160 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:27:33 smithi183 ceph-mon[29736]: 5.8 scrub starts 2024-09-10T19:27:34.160 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:27:33 smithi183 ceph-mon[29736]: 5.8 scrub ok 2024-09-10T19:27:34.160 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:27:33 smithi183 ceph-mon[41562]: 5.8 scrub starts 2024-09-10T19:27:34.160 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:27:33 smithi183 ceph-mon[41562]: 5.8 scrub ok 2024-09-10T19:27:34.170 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:27:34.191 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:27:34.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:34.192 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:27:34.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:27:33 smithi191 ceph-mon[32884]: 5.8 scrub starts 2024-09-10T19:27:34.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:27:33 smithi191 ceph-mon[32884]: 5.8 scrub ok 2024-09-10T19:27:34.269 DEBUG:teuthology.orchestra.run.smithi191:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph daemon osd.5 dump_blocked_ops 2024-09-10T19:27:34.293 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:27:34.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:34.334 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:27:34.334 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:27:34.435 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:27:34.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:34.457 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:27:34.458 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:27:34.556 INFO:teuthology.orchestra.run.smithi191.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.b/config 2024-09-10T19:27:34.558 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:27:34.579 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:27:34.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:34.580 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:27:34.681 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:27:34.702 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:27:34.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:34.704 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:27:34.715 INFO:tasks.thrashosds.thrasher:in_osds: [0, 3, 4, 5, 7, 1] out_osds: [2, 6] dead_osds: [] live_osds: [2, 3, 4, 5, 6, 7, 1, 0] 2024-09-10T19:27:34.715 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-10T19:27:34.716 INFO:tasks.thrashosds.thrasher:inject_pause on osd.2 2024-09-10T19:27:34.716 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-09-10T19:27:34.716 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-09-10T19:27:34.716 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph daemon osd.2 config set filestore_inject_stall 3 2024-09-10T19:27:34.805 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:27:34.826 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:27:34.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:34.827 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:27:34.928 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:27:34.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:34.951 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:27:34.952 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:27:35.053 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:27:35.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:35.075 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:27:35.075 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:27:35.176 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:27:35.198 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:27:35.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:35.199 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:27:35.265 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:27:34 smithi183 ceph-mon[29736]: pgmap v802: 158 pgs: 4 active+remapped+backfill_toofull, 154 active+clean; 635 KiB data, 5.1 GiB used, 531 GiB / 536 GiB avail; 0 B/s wr, 657 op/s 2024-09-10T19:27:35.265 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:27:34 smithi183 ceph-mon[41562]: pgmap v802: 158 pgs: 4 active+remapped+backfill_toofull, 154 active+clean; 635 KiB data, 5.1 GiB used, 531 GiB / 536 GiB avail; 0 B/s wr, 657 op/s 2024-09-10T19:27:35.300 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:27:35.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:35.322 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:27:35.322 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:27:35.423 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:27:35.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:35.446 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:27:35.446 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:27:35.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:27:34 smithi191 ceph-mon[32884]: pgmap v802: 158 pgs: 4 active+remapped+backfill_toofull, 154 active+clean; 635 KiB data, 5.1 GiB used, 531 GiB / 536 GiB avail; 0 B/s wr, 657 op/s 2024-09-10T19:27:35.547 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:27:35.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:35.570 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:27:35.570 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:27:35.671 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:27:35.673 INFO:teuthology.orchestra.run.smithi183.stdout:osd.0: {} 2024-09-10T19:27:35.673 INFO:teuthology.orchestra.run.smithi183.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-10T19:27:35.688 INFO:teuthology.orchestra.run.smithi183.stdout:osd.1: {} 2024-09-10T19:27:35.688 INFO:teuthology.orchestra.run.smithi183.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-10T19:27:35.692 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:27:35.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:35.693 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:27:35.703 INFO:teuthology.orchestra.run.smithi183.stdout:osd.2: {} 2024-09-10T19:27:35.703 INFO:teuthology.orchestra.run.smithi183.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-10T19:27:35.719 INFO:teuthology.orchestra.run.smithi183.stdout:osd.3: {} 2024-09-10T19:27:35.719 INFO:teuthology.orchestra.run.smithi183.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-10T19:27:35.734 INFO:teuthology.orchestra.run.smithi183.stdout:osd.4: {} 2024-09-10T19:27:35.734 INFO:teuthology.orchestra.run.smithi183.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-10T19:27:35.750 INFO:teuthology.orchestra.run.smithi183.stdout:osd.5: {} 2024-09-10T19:27:35.750 INFO:teuthology.orchestra.run.smithi183.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-10T19:27:35.766 INFO:teuthology.orchestra.run.smithi183.stdout:osd.6: {} 2024-09-10T19:27:35.766 INFO:teuthology.orchestra.run.smithi183.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-10T19:27:35.781 INFO:teuthology.orchestra.run.smithi183.stdout:osd.7: {} 2024-09-10T19:27:35.782 INFO:teuthology.orchestra.run.smithi183.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-10T19:27:35.794 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:27:35.817 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:27:35.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:35.819 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:27:35.919 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:27:35.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:35.942 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:27:35.943 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:27:36.044 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:27:36.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:36.066 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:27:36.066 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:27:36.138 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:27:36 smithi191 ceph-mon[32884]: from='client.? 172.21.15.183:0/4016989661' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T19:27:36.167 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:27:36.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:27:36 smithi183 ceph-mon[29736]: from='client.? 172.21.15.183:0/4016989661' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T19:27:36.181 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:27:36 smithi183 ceph-mon[41562]: from='client.? 172.21.15.183:0/4016989661' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T19:27:36.188 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:27:36.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:36.190 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:27:36.291 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:27:36.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:36.313 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:27:36.313 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:27:36.343 INFO:teuthology.orchestra.run.smithi191.stdout:ERROR: (22) Invalid argument 2024-09-10T19:27:36.343 INFO:teuthology.orchestra.run.smithi191.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-10T19:27:36.414 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:27:36.435 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:27:36.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:36.436 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:27:36.537 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:27:36.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:36.560 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:27:36.561 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:27:36.602 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-10T19:27:36.662 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:27:36.684 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:27:36.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:36.685 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:27:36.774 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:27:36.786 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:27:36.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:36.809 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:27:36.810 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:27:36.910 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:27:36.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:36.933 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:27:36.933 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:27:37.015 DEBUG:teuthology.orchestra.run.smithi191:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph daemon osd.5 dump_historic_ops 2024-09-10T19:27:37.015 INFO:teuthology.orchestra.run.smithi183.stderr:pool 'unique_pool_3' removed 2024-09-10T19:27:37.034 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:27:37.083 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:27:37.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:37.084 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:27:37.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:27:37 smithi183 ceph-mon[29736]: pgmap v803: 158 pgs: 4 active+remapped+backfill_toofull, 154 active+clean; 635 KiB data, 5.1 GiB used, 531 GiB / 536 GiB avail; 0 B/s wr, 369 op/s 2024-09-10T19:27:37.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:27:37 smithi183 ceph-mon[29736]: from='client.? 172.21.15.183:0/2804442244' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_3", "pool2": "unique_pool_3", "yes_i_really_really_mean_it": true}]: dispatch 2024-09-10T19:27:37.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:27:37 smithi183 ceph-mon[41562]: pgmap v803: 158 pgs: 4 active+remapped+backfill_toofull, 154 active+clean; 635 KiB data, 5.1 GiB used, 531 GiB / 536 GiB avail; 0 B/s wr, 369 op/s 2024-09-10T19:27:37.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:27:37 smithi183 ceph-mon[41562]: from='client.? 172.21.15.183:0/2804442244' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_3", "pool2": "unique_pool_3", "yes_i_really_really_mean_it": true}]: dispatch 2024-09-10T19:27:37.185 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:27:37.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:37.207 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:27:37.208 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:27:37.283 INFO:teuthology.orchestra.run.smithi191.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.b/config 2024-09-10T19:27:37.309 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:27:37.330 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:27:37.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:37.331 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:27:37.404 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:27:37 smithi191 ceph-mon[32884]: pgmap v803: 158 pgs: 4 active+remapped+backfill_toofull, 154 active+clean; 635 KiB data, 5.1 GiB used, 531 GiB / 536 GiB avail; 0 B/s wr, 369 op/s 2024-09-10T19:27:37.404 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:27:37 smithi191 ceph-mon[32884]: from='client.? 172.21.15.183:0/2804442244' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_3", "pool2": "unique_pool_3", "yes_i_really_really_mean_it": true}]: dispatch 2024-09-10T19:27:37.433 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:27:37.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:37.455 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:27:37.455 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:27:37.556 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:27:37.579 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:27:37.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:37.580 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:27:37.680 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:27:37.701 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:27:37.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:37.702 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:27:37.708 INFO:journalctl@ceph.mgr.x.smithi191.stdout:Sep 10 19:27:37 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-x[34967]: ::ffff:172.21.15.191 - - [10/Sep/2024:19:27:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T19:27:37.803 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:27:37.825 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:27:37.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:37.828 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:27:37.929 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:27:37.951 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:27:37.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:37.952 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:27:38.053 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:27:38.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:38.076 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:27:38.077 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:27:38.161 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:27:38 smithi183 ceph-mon[41562]: from='client.? 172.21.15.183:0/2804442244' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_3", "pool2": "unique_pool_3", "yes_i_really_really_mean_it": true}]': finished 2024-09-10T19:27:38.161 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:27:38 smithi183 ceph-mon[41562]: osdmap e368: 8 total, 8 up, 6 in 2024-09-10T19:27:38.161 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:27:38 smithi183 ceph-mon[29736]: from='client.? 172.21.15.183:0/2804442244' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_3", "pool2": "unique_pool_3", "yes_i_really_really_mean_it": true}]': finished 2024-09-10T19:27:38.161 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:27:38 smithi183 ceph-mon[29736]: osdmap e368: 8 total, 8 up, 6 in 2024-09-10T19:27:38.178 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:27:38.200 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:27:38.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:38.201 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:27:38.301 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:27:38.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:38.324 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:27:38.324 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:27:38.425 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:27:38.446 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:27:38.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:38.447 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:27:38.465 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:27:38 smithi191 ceph-mon[32884]: from='client.? 172.21.15.183:0/2804442244' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_3", "pool2": "unique_pool_3", "yes_i_really_really_mean_it": true}]': finished 2024-09-10T19:27:38.465 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:27:38 smithi191 ceph-mon[32884]: osdmap e368: 8 total, 8 up, 6 in 2024-09-10T19:27:38.547 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:27:38.568 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:27:38.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:38.570 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:27:38.670 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:27:38.679 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:27:38.692 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:27:38.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:38.693 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:27:38.703 INFO:teuthology.task.full_sequential:In full_sequential, running task radosbench... 2024-09-10T19:27:38.704 INFO:tasks.radosbench:Beginning radosbench... 2024-09-10T19:27:38.705 INFO:tasks.cephadm.ceph_manager.ceph:creating pool_name unique_pool_4 2024-09-10T19:27:38.705 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph osd pool create unique_pool_4 16 2024-09-10T19:27:38.776 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-10T19:27:38.794 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:27:38.841 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:27:38.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:38.842 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:27:38.943 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:27:38.964 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:27:38.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:38.965 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:27:39.046 INFO:teuthology.orchestra.run.smithi191.stdout:ERROR: (22) Invalid argument 2024-09-10T19:27:39.046 INFO:teuthology.orchestra.run.smithi191.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-10T19:27:39.066 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:27:39.089 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:27:39.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:39.090 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:27:39.191 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:27:39.201 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:27:39 smithi183 ceph-mon[29736]: pgmap v805: 142 pgs: 142 active+clean; 635 KiB data, 5.0 GiB used, 531 GiB / 536 GiB avail 2024-09-10T19:27:39.201 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:27:39 smithi183 ceph-mon[29736]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 4 pgs backfill_toofull) 2024-09-10T19:27:39.201 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:27:39 smithi183 ceph-mon[29736]: purged_snaps scrub starts 2024-09-10T19:27:39.201 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:27:39 smithi183 ceph-mon[29736]: purged_snaps scrub ok 2024-09-10T19:27:39.201 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:27:39 smithi183 ceph-mon[41562]: pgmap v805: 142 pgs: 142 active+clean; 635 KiB data, 5.0 GiB used, 531 GiB / 536 GiB avail 2024-09-10T19:27:39.202 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:27:39 smithi183 ceph-mon[41562]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 4 pgs backfill_toofull) 2024-09-10T19:27:39.202 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:27:39 smithi183 ceph-mon[41562]: purged_snaps scrub starts 2024-09-10T19:27:39.202 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:27:39 smithi183 ceph-mon[41562]: purged_snaps scrub ok 2024-09-10T19:27:39.215 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:27:39.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:27:39 smithi191 ceph-mon[32884]: pgmap v805: 142 pgs: 142 active+clean; 635 KiB data, 5.0 GiB used, 531 GiB / 536 GiB avail 2024-09-10T19:27:39.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:27:39 smithi191 ceph-mon[32884]: Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 4 pgs backfill_toofull) 2024-09-10T19:27:39.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:27:39 smithi191 ceph-mon[32884]: purged_snaps scrub starts 2024-09-10T19:27:39.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:27:39 smithi191 ceph-mon[32884]: purged_snaps scrub ok 2024-09-10T19:27:39.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:39.216 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:27:39.317 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:27:39.339 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:27:39.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:39.340 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:27:39.441 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:27:39.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:39.463 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:27:39.463 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:27:39.564 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:27:39.586 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:27:39.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:39.587 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:27:39.625 DEBUG:teuthology.orchestra.run.smithi191:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph daemon osd.6 dump_ops_in_flight 2024-09-10T19:27:39.687 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:27:39.709 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:27:39.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:39.710 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:27:39.811 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:27:39.833 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:27:39.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:39.834 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:27:39.865 INFO:teuthology.orchestra.run.smithi183.stdout:{ 2024-09-10T19:27:39.866 INFO:teuthology.orchestra.run.smithi183.stdout: "success": "" 2024-09-10T19:27:39.866 INFO:teuthology.orchestra.run.smithi183.stdout:} 2024-09-10T19:27:39.915 INFO:teuthology.orchestra.run.smithi191.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.b/config 2024-09-10T19:27:39.935 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:27:39.957 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:27:39.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:39.958 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:27:40.059 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:27:40.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:40.081 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:27:40.082 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:27:40.166 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:27:40 smithi191 ceph-mon[32884]: osdmap e369: 8 total, 8 up, 6 in 2024-09-10T19:27:40.166 INFO:journalctl@ceph.iscsi.iscsi.a.smithi191.stdout:Sep 10 19:27:39 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-iscsi-iscsi-a[61816]: debug there is no tcmu-runner data available 2024-09-10T19:27:40.183 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:27:40.205 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:27:40.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:40.209 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:27:40.310 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:27:40.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:40.332 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:27:40.332 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:27:40.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:27:40 smithi183 ceph-mon[29736]: osdmap e369: 8 total, 8 up, 6 in 2024-09-10T19:27:40.431 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:27:40 smithi183 ceph-mon[41562]: osdmap e369: 8 total, 8 up, 6 in 2024-09-10T19:27:40.433 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:27:40.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:40.456 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:27:40.456 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:27:40.557 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:27:40.578 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:27:40.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:40.579 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:27:40.680 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:27:40.701 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:27:40.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:40.702 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:27:40.803 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:27:40.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:40.825 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:27:40.825 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:27:40.926 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:27:40.948 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:27:40.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:40.950 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:27:41.051 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:27:41.072 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:27:41.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:41.073 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:27:41.174 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:27:41.196 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:27:41.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:41.197 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:27:41.298 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:27:41.304 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:27:41 smithi191 ceph-mon[32884]: from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:27:41.305 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:27:41 smithi191 ceph-mon[32884]: pgmap v807: 142 pgs: 142 active+clean; 635 KiB data, 5.0 GiB used, 531 GiB / 536 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-10T19:27:41.316 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:27:41 smithi183 ceph-mon[29736]: from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:27:41.316 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:27:41 smithi183 ceph-mon[29736]: pgmap v807: 142 pgs: 142 active+clean; 635 KiB data, 5.0 GiB used, 531 GiB / 536 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-10T19:27:41.316 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:27:41 smithi183 ceph-mon[41562]: from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:27:41.316 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:27:41 smithi183 ceph-mon[41562]: pgmap v807: 142 pgs: 142 active+clean; 635 KiB data, 5.0 GiB used, 531 GiB / 536 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-10T19:27:41.321 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:27:41.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:41.322 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:27:41.423 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:27:41.445 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:27:41.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:41.446 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:27:41.547 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:27:41.569 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:27:41.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:41.570 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:27:41.644 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Sep 10 19:27:41 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-y[29989]: ::ffff:172.21.15.191 - - [10/Sep/2024:19:27:41] "GET /metrics HTTP/1.1" 200 224841 "" "Prometheus/2.43.0" 2024-09-10T19:27:41.649 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:27:41.666 INFO:teuthology.orchestra.run.smithi191.stdout:ERROR: (22) Invalid argument 2024-09-10T19:27:41.666 INFO:teuthology.orchestra.run.smithi191.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-10T19:27:41.671 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:27:41.672 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:27:41.694 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:27:41.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:41.695 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:27:41.796 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:27:41.817 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:27:41.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:41.818 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:27:41.919 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:27:41.944 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:27:41.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:41.945 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:27:42.033 INFO:teuthology.orchestra.run.smithi183.stderr:nodeep-scrub is set 2024-09-10T19:27:42.046 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:27:42.070 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:27:42.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:42.071 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:27:42.172 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:27:42.187 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:27:42 smithi183 ceph-mon[29736]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T19:27:42.187 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:27:42 smithi183 ceph-mon[29736]: from='client.? 172.21.15.183:0/390570851' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-10T19:27:42.187 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:27:42 smithi183 ceph-mon[29736]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-10T19:27:42.187 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:27:42 smithi183 ceph-mon[29736]: from='mgr.14556 ' entity='mgr.y' 2024-09-10T19:27:42.187 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:27:42 smithi183 ceph-mon[29736]: from='mgr.14556 ' entity='mgr.y' 2024-09-10T19:27:42.187 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:27:42 smithi183 ceph-mon[41562]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T19:27:42.187 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:27:42 smithi183 ceph-mon[41562]: from='client.? 172.21.15.183:0/390570851' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-10T19:27:42.187 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:27:42 smithi183 ceph-mon[41562]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-10T19:27:42.187 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:27:42 smithi183 ceph-mon[41562]: from='mgr.14556 ' entity='mgr.y' 2024-09-10T19:27:42.187 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:27:42 smithi183 ceph-mon[41562]: from='mgr.14556 ' entity='mgr.y' 2024-09-10T19:27:42.193 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:27:42.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:42.194 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:27:42.295 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:27:42.316 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:27:42.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:42.317 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:27:42.418 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:27:42.440 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:27:42.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:42.441 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:27:42.465 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:27:42 smithi191 ceph-mon[32884]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T19:27:42.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:27:42 smithi191 ceph-mon[32884]: from='client.? 172.21.15.183:0/390570851' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-10T19:27:42.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:27:42 smithi191 ceph-mon[32884]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-10T19:27:42.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:27:42 smithi191 ceph-mon[32884]: from='mgr.14556 ' entity='mgr.y' 2024-09-10T19:27:42.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:27:42 smithi191 ceph-mon[32884]: from='mgr.14556 ' entity='mgr.y' 2024-09-10T19:27:42.537 DEBUG:teuthology.orchestra.run.smithi191:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph daemon osd.6 dump_blocked_ops 2024-09-10T19:27:42.541 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:27:42.563 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:27:42.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:42.564 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:27:42.664 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:27:42.686 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:27:42.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:42.687 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:27:42.788 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:27:42.809 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:27:42.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:42.810 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:27:42.834 INFO:teuthology.orchestra.run.smithi191.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.b/config 2024-09-10T19:27:42.911 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:27:42.932 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:27:42.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:42.933 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:27:43.034 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:27:43.054 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:27:43.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:43.055 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:27:43.102 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:27:43 smithi191 ceph-mon[32884]: pgmap v808: 142 pgs: 142 active+clean; 635 KiB data, 5.0 GiB used, 531 GiB / 536 GiB avail; 511 B/s rd, 0 op/s 2024-09-10T19:27:43.102 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:27:43 smithi191 ceph-mon[32884]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T19:27:43.102 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:27:43 smithi191 ceph-mon[32884]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-10T19:27:43.102 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:27:43 smithi191 ceph-mon[32884]: osdmap e370: 8 total, 8 up, 6 in 2024-09-10T19:27:43.102 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:27:43 smithi191 ceph-mon[32884]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T19:27:43.102 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:27:43 smithi191 ceph-mon[32884]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T19:27:43.102 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:27:43 smithi191 ceph-mon[32884]: from='mgr.14556 ' entity='mgr.y' 2024-09-10T19:27:43.156 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:27:43.177 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:27:43.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:43.178 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:27:43.280 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:27:43.302 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:27:43.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:43.303 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:27:43.388 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:27:43 smithi183 ceph-mon[29736]: pgmap v808: 142 pgs: 142 active+clean; 635 KiB data, 5.0 GiB used, 531 GiB / 536 GiB avail; 511 B/s rd, 0 op/s 2024-09-10T19:27:43.388 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:27:43 smithi183 ceph-mon[29736]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T19:27:43.389 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:27:43 smithi183 ceph-mon[29736]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-10T19:27:43.389 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:27:43 smithi183 ceph-mon[29736]: osdmap e370: 8 total, 8 up, 6 in 2024-09-10T19:27:43.389 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:27:43 smithi183 ceph-mon[29736]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T19:27:43.389 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:27:43 smithi183 ceph-mon[29736]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T19:27:43.389 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:27:43 smithi183 ceph-mon[29736]: from='mgr.14556 ' entity='mgr.y' 2024-09-10T19:27:43.389 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:27:43 smithi183 ceph-mon[41562]: pgmap v808: 142 pgs: 142 active+clean; 635 KiB data, 5.0 GiB used, 531 GiB / 536 GiB avail; 511 B/s rd, 0 op/s 2024-09-10T19:27:43.389 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:27:43 smithi183 ceph-mon[41562]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T19:27:43.389 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:27:43 smithi183 ceph-mon[41562]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-10T19:27:43.389 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:27:43 smithi183 ceph-mon[41562]: osdmap e370: 8 total, 8 up, 6 in 2024-09-10T19:27:43.389 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:27:43 smithi183 ceph-mon[41562]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T19:27:43.390 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:27:43 smithi183 ceph-mon[41562]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T19:27:43.390 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:27:43 smithi183 ceph-mon[41562]: from='mgr.14556 ' entity='mgr.y' 2024-09-10T19:27:43.403 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:27:43.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:43.426 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:27:43.426 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:27:43.527 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:27:43.549 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:27:43.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:43.550 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:27:43.651 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:27:43.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:43.674 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:27:43.674 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:27:43.775 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:27:43.797 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:27:43.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:43.798 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:27:43.899 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:27:43.920 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:27:43.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:43.921 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:27:44.022 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:27:44.044 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:27:44.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:44.045 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:27:44.146 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:27:44.167 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:27:44.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:44.168 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:27:44.268 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:27:44.290 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:27:44.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:44.291 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:27:44.391 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:27:44.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:44.413 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:27:44.413 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:27:44.514 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:27:44.515 INFO:teuthology.orchestra.run.smithi191.stdout:ERROR: (22) Invalid argument 2024-09-10T19:27:44.515 INFO:teuthology.orchestra.run.smithi191.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-10T19:27:44.536 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:27:44.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:44.538 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:27:44.640 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:27:44.661 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:27:44.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:44.662 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:27:44.763 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:27:44.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:44.786 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:27:44.786 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:27:44.887 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:27:44.892 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:27:44 smithi191 ceph-mon[32884]: from='mgr.14556 ' entity='mgr.y' 2024-09-10T19:27:44.893 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:27:44 smithi191 ceph-mon[32884]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:27:44.893 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:27:44 smithi191 ceph-mon[32884]: pgmap v810: 142 pgs: 142 active+clean; 635 KiB data, 4.9 GiB used, 531 GiB / 536 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-09-10T19:27:44.909 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:27:44.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:44.910 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:27:44.916 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:27:44 smithi183 ceph-mon[29736]: from='mgr.14556 ' entity='mgr.y' 2024-09-10T19:27:44.917 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:27:44 smithi183 ceph-mon[29736]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:27:44.917 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:27:44 smithi183 ceph-mon[29736]: pgmap v810: 142 pgs: 142 active+clean; 635 KiB data, 4.9 GiB used, 531 GiB / 536 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-09-10T19:27:44.917 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:27:44 smithi183 ceph-mon[41562]: from='mgr.14556 ' entity='mgr.y' 2024-09-10T19:27:44.917 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:27:44 smithi183 ceph-mon[41562]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:27:44.917 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:27:44 smithi183 ceph-mon[41562]: pgmap v810: 142 pgs: 142 active+clean; 635 KiB data, 4.9 GiB used, 531 GiB / 536 GiB avail; 1.4 KiB/s rd, 1 op/s 2024-09-10T19:27:45.010 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:27:45.031 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:27:45.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:45.032 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:27:45.134 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:27:45.155 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:27:45.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:45.156 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:27:45.257 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:27:45.269 DEBUG:teuthology.orchestra.run.smithi191:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph daemon osd.6 dump_historic_ops 2024-09-10T19:27:45.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:45.280 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:27:45.280 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:27:45.381 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:27:45.404 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:27:45.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:45.405 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:27:45.506 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:27:45.521 INFO:teuthology.orchestra.run.smithi191.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.b/config 2024-09-10T19:27:45.528 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:27:45.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:45.529 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:27:45.630 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:27:45.651 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:27:45.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:45.652 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:27:45.670 INFO:tasks.thrashosds.thrasher:in_osds: [0, 3, 4, 5, 7, 1] out_osds: [2, 6] dead_osds: [] live_osds: [2, 3, 4, 5, 6, 7, 1, 0] 2024-09-10T19:27:45.670 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-10T19:27:45.670 INFO:tasks.thrashosds.thrasher:Adding osd 6 2024-09-10T19:27:45.670 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph osd in 6 2024-09-10T19:27:45.681 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:27:45 smithi183 ceph-mon[29736]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.15", "id": [1, 3]}]: dispatch 2024-09-10T19:27:45.681 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:27:45 smithi183 ceph-mon[29736]: from='mgr.14556 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.15", "id": [1, 3]}]: dispatch 2024-09-10T19:27:45.682 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:27:45 smithi183 ceph-mon[29736]: from='client.? 172.21.15.183:0/468176387' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_4", "pg_num": 16}]: dispatch 2024-09-10T19:27:45.682 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:27:45 smithi183 ceph-mon[29736]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_4", "pg_num": 16}]: dispatch 2024-09-10T19:27:45.699 INFO:teuthology.orchestra.run.smithi183.stderr:pool 'unique_pool_4' created 2024-09-10T19:27:45.753 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:27:45.774 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:27:45.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:45.775 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:27:45.876 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:27:45.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:45.898 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:27:45.899 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:27:45.965 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:27:45 smithi191 ceph-mon[32884]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.15", "id": [1, 3]}]: dispatch 2024-09-10T19:27:45.966 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:27:45 smithi191 ceph-mon[32884]: from='mgr.14556 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.15", "id": [1, 3]}]: dispatch 2024-09-10T19:27:45.966 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:27:45 smithi191 ceph-mon[32884]: from='client.? 172.21.15.183:0/468176387' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_4", "pg_num": 16}]: dispatch 2024-09-10T19:27:45.966 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:27:45 smithi191 ceph-mon[32884]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_4", "pg_num": 16}]: dispatch 2024-09-10T19:27:45.999 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:27:46.021 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:27:46.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:46.022 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:27:46.039 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:27:45 smithi183 ceph-mon[41562]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.15", "id": [1, 3]}]: dispatch 2024-09-10T19:27:46.040 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:27:45 smithi183 ceph-mon[41562]: from='mgr.14556 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.15", "id": [1, 3]}]: dispatch 2024-09-10T19:27:46.040 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:27:45 smithi183 ceph-mon[41562]: from='client.? 172.21.15.183:0/468176387' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_4", "pg_num": 16}]: dispatch 2024-09-10T19:27:46.040 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:27:45 smithi183 ceph-mon[41562]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_4", "pg_num": 16}]: dispatch 2024-09-10T19:27:46.123 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:27:46.144 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:27:46.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:46.145 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:27:46.147 INFO:teuthology.orchestra.run.smithi183.stdout:osd.0: {} 2024-09-10T19:27:46.148 INFO:teuthology.orchestra.run.smithi183.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-10T19:27:46.150 INFO:teuthology.orchestra.run.smithi183.stdout:osd.1: {} 2024-09-10T19:27:46.151 INFO:teuthology.orchestra.run.smithi183.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-10T19:27:46.165 INFO:teuthology.orchestra.run.smithi183.stdout:osd.2: {} 2024-09-10T19:27:46.166 INFO:teuthology.orchestra.run.smithi183.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-10T19:27:46.182 INFO:teuthology.orchestra.run.smithi183.stdout:osd.3: {} 2024-09-10T19:27:46.182 INFO:teuthology.orchestra.run.smithi183.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-10T19:27:46.197 INFO:teuthology.orchestra.run.smithi183.stdout:osd.4: {} 2024-09-10T19:27:46.197 INFO:teuthology.orchestra.run.smithi183.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-10T19:27:46.212 INFO:teuthology.orchestra.run.smithi183.stdout:osd.5: {} 2024-09-10T19:27:46.213 INFO:teuthology.orchestra.run.smithi183.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-10T19:27:46.228 INFO:teuthology.orchestra.run.smithi183.stdout:osd.6: {} 2024-09-10T19:27:46.228 INFO:teuthology.orchestra.run.smithi183.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-10T19:27:46.244 INFO:teuthology.orchestra.run.smithi183.stdout:osd.7: {} 2024-09-10T19:27:46.244 INFO:teuthology.orchestra.run.smithi183.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-10T19:27:46.246 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:27:46.268 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:27:46.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:46.269 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:27:46.367 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph osd unset noscrub 2024-09-10T19:27:46.370 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:27:46.432 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:27:46.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:46.433 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:27:46.534 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:27:46.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:46.556 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:27:46.557 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:27:46.633 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph osd pool application enable unique_pool_4 rados --yes-i-really-mean-it || true 2024-09-10T19:27:46.658 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:27:46.680 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:27:46.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:46.681 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:27:46.782 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:27:46.788 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:27:46 smithi191 ceph-mon[32884]: from='mgr.14556 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.15", "id": [1, 3]}]': finished 2024-09-10T19:27:46.788 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:27:46 smithi191 ceph-mon[32884]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_4", "pg_num": 16}]': finished 2024-09-10T19:27:46.788 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:27:46 smithi191 ceph-mon[32884]: osdmap e371: 8 total, 8 up, 6 in 2024-09-10T19:27:46.788 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:27:46 smithi191 ceph-mon[32884]: pgmap v812: 158 pgs: 16 unknown, 142 active+clean; 635 KiB data, 4.9 GiB used, 531 GiB / 536 GiB avail; 741 B/s rd, 0 op/s 2024-09-10T19:27:46.788 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:27:46 smithi191 ceph-mon[32884]: from='client.? 172.21.15.183:0/3357002423' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T19:27:46.803 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:27:46.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:46.804 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:27:46.905 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:27:46.927 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:27:46.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:46.928 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:27:46.931 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:27:46 smithi183 ceph-mon[41562]: from='mgr.14556 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.15", "id": [1, 3]}]': finished 2024-09-10T19:27:46.931 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:27:46 smithi183 ceph-mon[41562]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_4", "pg_num": 16}]': finished 2024-09-10T19:27:46.931 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:27:46 smithi183 ceph-mon[41562]: osdmap e371: 8 total, 8 up, 6 in 2024-09-10T19:27:46.931 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:27:46 smithi183 ceph-mon[41562]: pgmap v812: 158 pgs: 16 unknown, 142 active+clean; 635 KiB data, 4.9 GiB used, 531 GiB / 536 GiB avail; 741 B/s rd, 0 op/s 2024-09-10T19:27:46.931 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:27:46 smithi183 ceph-mon[41562]: from='client.? 172.21.15.183:0/3357002423' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T19:27:46.932 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:27:46 smithi183 ceph-mon[29736]: from='mgr.14556 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.15", "id": [1, 3]}]': finished 2024-09-10T19:27:46.932 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:27:46 smithi183 ceph-mon[29736]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_4", "pg_num": 16}]': finished 2024-09-10T19:27:46.932 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:27:46 smithi183 ceph-mon[29736]: osdmap e371: 8 total, 8 up, 6 in 2024-09-10T19:27:46.932 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:27:46 smithi183 ceph-mon[29736]: pgmap v812: 158 pgs: 16 unknown, 142 active+clean; 635 KiB data, 4.9 GiB used, 531 GiB / 536 GiB avail; 741 B/s rd, 0 op/s 2024-09-10T19:27:46.932 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:27:46 smithi183 ceph-mon[29736]: from='client.? 172.21.15.183:0/3357002423' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T19:27:47.029 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:27:47.049 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:27:47.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:47.050 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:27:47.151 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:27:47.173 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:27:47.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:47.174 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:27:47.275 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:27:47.297 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:27:47.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:47.298 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:27:47.309 INFO:teuthology.orchestra.run.smithi191.stdout:{ 2024-09-10T19:27:47.309 INFO:teuthology.orchestra.run.smithi191.stdout: "size": 20, 2024-09-10T19:27:47.309 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 600, 2024-09-10T19:27:47.309 INFO:teuthology.orchestra.run.smithi191.stdout: "ops": [ 2024-09-10T19:27:47.310 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:47.310 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.15360.0:169776 9.0 9:09829b82:::benchmark_data_smithi183_142499_object23287:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e262)", 2024-09-10T19:27:47.310 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-10T19:21:46.960092+0000", 2024-09-10T19:27:47.310 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 360.33333691799999, 2024-09-10T19:27:47.310 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.0762074109999999, 2024-09-10T19:27:47.310 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-10T19:27:47.310 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "reached pg", 2024-09-10T19:27:47.310 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-10T19:27:47.311 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.15360", 2024-09-10T19:27:47.311 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.183:0/2364804131", 2024-09-10T19:27:47.311 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 169776 2024-09-10T19:27:47.311 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:47.311 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-10T19:27:47.311 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:47.311 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-10T19:27:47.311 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:21:46.960092+0000", 2024-09-10T19:27:47.311 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:27:47.311 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:47.311 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:47.311 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-10T19:27:47.311 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:21:46.960092+0000", 2024-09-10T19:27:47.311 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:27:47.312 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:47.312 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:47.312 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-10T19:27:47.312 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:21:46.960090+0000", 2024-09-10T19:27:47.312 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999976 2024-09-10T19:27:47.312 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:47.312 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:47.312 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-10T19:27:47.312 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:21:46.960098+0000", 2024-09-10T19:27:47.312 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.3909999999999998e-06 2024-09-10T19:27:47.312 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:47.312 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:47.312 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-10T19:27:47.313 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:21:46.960111+0000", 2024-09-10T19:27:47.313 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.2378000000000001e-05 2024-09-10T19:27:47.313 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:47.313 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:47.313 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-10T19:27:47.313 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:21:46.960117+0000", 2024-09-10T19:27:47.313 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.2489999999999998e-06 2024-09-10T19:27:47.313 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:47.313 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:47.313 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:27:47.313 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:21:46.960174+0000", 2024-09-10T19:27:47.313 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.6975000000000003e-05 2024-09-10T19:27:47.313 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:47.314 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:47.314 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-10T19:27:47.314 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:21:46.960329+0000", 2024-09-10T19:27:47.314 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000155067 2024-09-10T19:27:47.314 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:47.314 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:47.314 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0,5", 2024-09-10T19:27:47.314 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:21:46.960449+0000", 2024-09-10T19:27:47.314 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000120213 2024-09-10T19:27:47.314 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:47.315 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:47.315 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-09-10T19:27:47.315 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:21:46.978831+0000", 2024-09-10T19:27:47.315 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.018381511999999999 2024-09-10T19:27:47.315 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:47.315 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:47.315 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:27:47.315 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:21:46.996714+0000", 2024-09-10T19:27:47.315 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.017883928 2024-09-10T19:27:47.315 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:47.315 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:47.315 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for peered", 2024-09-10T19:27:47.315 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:21:46.996746+0000", 2024-09-10T19:27:47.316 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.1588999999999999e-05 2024-09-10T19:27:47.316 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:47.316 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:47.316 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:27:47.316 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:21:48.024253+0000", 2024-09-10T19:27:47.316 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.0275069779999999 2024-09-10T19:27:47.316 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:47.316 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:47.316 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for missing object", 2024-09-10T19:27:47.316 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:21:48.024346+0000", 2024-09-10T19:27:47.316 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.3387000000000006e-05 2024-09-10T19:27:47.316 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:47.316 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:47.317 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:27:47.317 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:21:48.036242+0000", 2024-09-10T19:27:47.317 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.011895773 2024-09-10T19:27:47.317 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:47.317 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:47.317 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-10T19:27:47.317 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:21:48.036300+0000", 2024-09-10T19:27:47.317 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.7324999999999997e-05 2024-09-10T19:27:47.317 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:27:47.317 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-10T19:27:47.317 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:27:47.317 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:47.318 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:47.318 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.15360.0:169777 9.0 9:0ac81f83:::benchmark_data_smithi183_142499_object23288:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e262)", 2024-09-10T19:27:47.318 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-10T19:21:46.960134+0000", 2024-09-10T19:27:47.318 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 360.333295405, 2024-09-10T19:27:47.318 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.079806681, 2024-09-10T19:27:47.318 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-10T19:27:47.318 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "reached pg", 2024-09-10T19:27:47.318 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-10T19:27:47.318 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.15360", 2024-09-10T19:27:47.318 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.183:0/2364804131", 2024-09-10T19:27:47.318 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 169777 2024-09-10T19:27:47.318 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:47.318 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-10T19:27:47.318 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:47.319 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-10T19:27:47.319 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:21:46.960134+0000", 2024-09-10T19:27:47.319 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:27:47.319 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:47.319 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:47.319 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-10T19:27:47.319 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:21:46.960134+0000", 2024-09-10T19:27:47.319 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:27:47.319 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:47.319 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:47.319 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-10T19:27:47.319 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:21:46.960133+0000", 2024-09-10T19:27:47.319 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999995 2024-09-10T19:27:47.319 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:47.320 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:47.320 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-10T19:27:47.320 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:21:46.960136+0000", 2024-09-10T19:27:47.320 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.2880000000000001e-06 2024-09-10T19:27:47.320 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:47.320 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:47.320 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-10T19:27:47.320 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:21:46.960143+0000", 2024-09-10T19:27:47.320 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.9630000000000001e-06 2024-09-10T19:27:47.320 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:47.320 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:47.320 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-10T19:27:47.320 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:21:46.960147+0000", 2024-09-10T19:27:47.321 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.9310000000000001e-06 2024-09-10T19:27:47.321 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:47.321 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:47.321 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:27:47.321 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:21:46.968247+0000", 2024-09-10T19:27:47.321 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0081001040000000003 2024-09-10T19:27:47.321 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:47.321 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:47.321 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-10T19:27:47.321 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:21:46.968594+0000", 2024-09-10T19:27:47.321 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00034688600000000003 2024-09-10T19:27:47.321 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:47.321 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:47.321 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0,5", 2024-09-10T19:27:47.322 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:21:46.968997+0000", 2024-09-10T19:27:47.322 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00040243000000000001 2024-09-10T19:27:47.322 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:47.322 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:47.322 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-09-10T19:27:47.322 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:21:46.978853+0000", 2024-09-10T19:27:47.322 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0098559110000000002 2024-09-10T19:27:47.322 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:47.322 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:47.322 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:27:47.322 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:21:46.996785+0000", 2024-09-10T19:27:47.322 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.017932496999999999 2024-09-10T19:27:47.322 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:47.323 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:47.323 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for peered", 2024-09-10T19:27:47.323 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:21:46.996790+0000", 2024-09-10T19:27:47.323 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.5109999999999997e-06 2024-09-10T19:27:47.323 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:47.323 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:47.323 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:27:47.323 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:21:48.024372+0000", 2024-09-10T19:27:47.323 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.027582099 2024-09-10T19:27:47.323 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:47.323 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:47.323 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for missing object", 2024-09-10T19:27:47.323 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:21:48.024457+0000", 2024-09-10T19:27:47.324 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.5382000000000003e-05 2024-09-10T19:27:47.324 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:47.324 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:47.324 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:27:47.324 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:21:48.039837+0000", 2024-09-10T19:27:47.324 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.015380072999999999 2024-09-10T19:27:47.324 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:47.324 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:47.324 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-10T19:27:47.324 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:21:48.039940+0000", 2024-09-10T19:27:47.324 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000103138 2024-09-10T19:27:47.324 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:27:47.324 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-10T19:27:47.325 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:27:47.325 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:47.325 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:47.325 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.15360.0:169758 9.b 9:d2ede469:::benchmark_data_smithi183_142499_object23269:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e263)", 2024-09-10T19:27:47.325 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-10T19:21:46.960378+0000", 2024-09-10T19:27:47.325 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 360.33305112900001, 2024-09-10T19:27:47.325 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.0602797209999999, 2024-09-10T19:27:47.325 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-10T19:27:47.325 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "reached pg", 2024-09-10T19:27:47.325 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-10T19:27:47.325 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.15360", 2024-09-10T19:27:47.325 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.183:0/2364804131", 2024-09-10T19:27:47.325 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 169758 2024-09-10T19:27:47.326 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:47.326 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-10T19:27:47.326 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:47.326 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-10T19:27:47.326 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:21:46.960378+0000", 2024-09-10T19:27:47.326 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:27:47.326 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:47.326 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:47.326 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-10T19:27:47.326 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:21:46.960378+0000", 2024-09-10T19:27:47.326 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:27:47.326 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:47.326 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:47.326 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-10T19:27:47.327 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:21:46.960377+0000", 2024-09-10T19:27:47.327 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.999999 2024-09-10T19:27:47.327 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:47.327 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:47.327 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-10T19:27:47.327 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:21:46.960382+0000", 2024-09-10T19:27:47.327 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.0969999999999996e-06 2024-09-10T19:27:47.327 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:47.327 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:47.327 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-10T19:27:47.327 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:21:46.960392+0000", 2024-09-10T19:27:47.327 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.5589999999999994e-06 2024-09-10T19:27:47.327 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:47.328 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:47.328 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-10T19:27:47.328 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:21:46.960397+0000", 2024-09-10T19:27:47.328 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.1710000000000001e-06 2024-09-10T19:27:47.328 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:47.328 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:47.328 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:27:47.328 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:21:46.963182+0000", 2024-09-10T19:27:47.328 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0027847459999999998 2024-09-10T19:27:47.328 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:47.328 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:47.328 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op must wait for map", 2024-09-10T19:27:47.328 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:21:46.963190+0000", 2024-09-10T19:27:47.329 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.0709999999999995e-06 2024-09-10T19:27:47.329 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:47.329 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:47.329 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:27:47.329 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:21:46.986739+0000", 2024-09-10T19:27:47.329 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.023549371999999999 2024-09-10T19:27:47.329 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:47.329 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:47.329 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for peered", 2024-09-10T19:27:47.329 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:21:46.986748+0000", 2024-09-10T19:27:47.329 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.6449999999999994e-06 2024-09-10T19:27:47.329 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:47.329 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:47.330 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:27:47.330 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:21:48.019246+0000", 2024-09-10T19:27:47.330 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.0324984290000001 2024-09-10T19:27:47.330 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:47.330 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:47.330 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-10T19:27:47.330 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:21:48.020658+0000", 2024-09-10T19:27:47.330 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.001411488 2024-09-10T19:27:47.330 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:27:47.330 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-10T19:27:47.330 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:27:47.330 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:47.330 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:47.331 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.15360.0:169773 9.b 9:d1791580:::benchmark_data_smithi183_142499_object23284:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e263)", 2024-09-10T19:27:47.331 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-10T19:21:46.960409+0000", 2024-09-10T19:27:47.331 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 360.33302002099998, 2024-09-10T19:27:47.331 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.0603718419999999, 2024-09-10T19:27:47.331 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-10T19:27:47.331 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "reached pg", 2024-09-10T19:27:47.331 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-10T19:27:47.331 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.15360", 2024-09-10T19:27:47.331 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.183:0/2364804131", 2024-09-10T19:27:47.331 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 169773 2024-09-10T19:27:47.331 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:47.331 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-10T19:27:47.331 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:47.332 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-10T19:27:47.332 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:21:46.960409+0000", 2024-09-10T19:27:47.332 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:27:47.332 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:47.332 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:47.332 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-10T19:27:47.332 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:21:46.960409+0000", 2024-09-10T19:27:47.332 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:27:47.332 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:47.332 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:47.332 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-10T19:27:47.332 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:21:46.960409+0000", 2024-09-10T19:27:47.333 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999995 2024-09-10T19:27:47.333 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:47.333 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:47.333 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-10T19:27:47.333 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:21:46.960412+0000", 2024-09-10T19:27:47.333 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.1659999999999998e-06 2024-09-10T19:27:47.333 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:47.333 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:47.333 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-10T19:27:47.333 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:21:46.960431+0000", 2024-09-10T19:27:47.333 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.9749999999999999e-05 2024-09-10T19:27:47.333 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:47.334 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:47.334 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-10T19:27:47.334 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:21:46.960435+0000", 2024-09-10T19:27:47.334 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.5839999999999999e-06 2024-09-10T19:27:47.334 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:47.334 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:47.334 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:27:47.334 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:21:46.968037+0000", 2024-09-10T19:27:47.334 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0076023000000000002 2024-09-10T19:27:47.334 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:47.334 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:47.334 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting_for_map not empty", 2024-09-10T19:27:47.334 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:21:46.968045+0000", 2024-09-10T19:27:47.334 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.6159999999999996e-06 2024-09-10T19:27:47.335 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:47.335 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:47.335 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:27:47.335 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:21:46.986775+0000", 2024-09-10T19:27:47.335 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.018730508999999999 2024-09-10T19:27:47.335 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:47.335 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:47.335 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for peered", 2024-09-10T19:27:47.335 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:21:46.986780+0000", 2024-09-10T19:27:47.335 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.9570000000000001e-06 2024-09-10T19:27:47.335 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:47.335 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:47.336 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:27:47.336 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:21:48.020682+0000", 2024-09-10T19:27:47.336 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.0339018090000001 2024-09-10T19:27:47.336 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:47.336 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:47.336 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-10T19:27:47.336 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:21:48.020781+0000", 2024-09-10T19:27:47.336 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.8597000000000005e-05 2024-09-10T19:27:47.336 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:27:47.336 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-10T19:27:47.336 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:27:47.336 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:47.336 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:47.337 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.15360.0:169778 9.0 9:0318be15:::benchmark_data_smithi183_142499_object23289:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e263)", 2024-09-10T19:27:47.337 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-10T19:21:46.973391+0000", 2024-09-10T19:27:47.337 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 360.32003842900002, 2024-09-10T19:27:47.337 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.0624239179999999, 2024-09-10T19:27:47.337 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-10T19:27:47.337 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T19:27:47.337 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-10T19:27:47.337 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.15360", 2024-09-10T19:27:47.337 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.183:0/2364804131", 2024-09-10T19:27:47.337 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 169778 2024-09-10T19:27:47.337 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:47.337 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-10T19:27:47.337 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:47.337 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-10T19:27:47.338 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:21:46.973391+0000", 2024-09-10T19:27:47.338 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:27:47.338 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:47.338 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:47.338 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-10T19:27:47.338 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:21:46.973391+0000", 2024-09-10T19:27:47.338 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:27:47.338 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:47.338 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:47.338 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-10T19:27:47.339 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:21:46.973388+0000", 2024-09-10T19:27:47.339 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999976 2024-09-10T19:27:47.339 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:47.339 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:47.339 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-10T19:27:47.339 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:21:46.973397+0000", 2024-09-10T19:27:47.339 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.3440000000000007e-06 2024-09-10T19:27:47.339 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:47.339 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:47.339 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-10T19:27:47.339 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:21:46.973416+0000", 2024-09-10T19:27:47.339 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.8856e-05 2024-09-10T19:27:47.339 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:47.340 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:47.340 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-10T19:27:47.340 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:21:46.973425+0000", 2024-09-10T19:27:47.340 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.7450000000000001e-06 2024-09-10T19:27:47.340 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:47.340 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:47.340 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:27:47.340 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:21:46.974228+0000", 2024-09-10T19:27:47.340 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00080324300000000003 2024-09-10T19:27:47.340 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:47.340 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:47.340 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op must wait for map", 2024-09-10T19:27:47.340 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:21:46.974253+0000", 2024-09-10T19:27:47.340 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.5154999999999999e-05 2024-09-10T19:27:47.341 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:47.341 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:47.341 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:27:47.341 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:21:46.996811+0000", 2024-09-10T19:27:47.341 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.022557480000000001 2024-09-10T19:27:47.341 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:47.341 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:47.341 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for peered", 2024-09-10T19:27:47.341 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:21:46.996815+0000", 2024-09-10T19:27:47.341 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.104e-06 2024-09-10T19:27:47.341 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:47.341 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:47.341 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:27:47.342 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:21:48.024477+0000", 2024-09-10T19:27:47.342 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.0276621420000001 2024-09-10T19:27:47.342 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:47.342 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:47.342 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-10T19:27:47.342 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:21:48.024564+0000", 2024-09-10T19:27:47.342 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.6817000000000006e-05 2024-09-10T19:27:47.342 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:47.342 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:47.342 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 5", 2024-09-10T19:27:47.342 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:21:48.024666+0000", 2024-09-10T19:27:47.342 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000101591 2024-09-10T19:27:47.342 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:47.342 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:47.343 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-09-10T19:27:47.343 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:21:48.034735+0000", 2024-09-10T19:27:47.343 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.010069052 2024-09-10T19:27:47.343 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:47.343 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:47.343 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:27:47.343 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:21:48.035677+0000", 2024-09-10T19:27:47.343 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000942811 2024-09-10T19:27:47.343 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:47.343 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:47.343 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-09-10T19:27:47.343 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:21:48.035722+0000", 2024-09-10T19:27:47.343 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.4885999999999997e-05 2024-09-10T19:27:47.344 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:47.344 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:47.344 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-10T19:27:47.344 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:21:48.035815+0000", 2024-09-10T19:27:47.344 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.2181000000000004e-05 2024-09-10T19:27:47.344 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:27:47.344 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-10T19:27:47.344 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:27:47.344 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:47.344 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:47.344 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.15360.0:169782 9.b 9:d3afc8c6:::benchmark_data_smithi183_142499_object23293:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e263)", 2024-09-10T19:27:47.344 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-10T19:21:46.990784+0000", 2024-09-10T19:27:47.344 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 360.30264517000001, 2024-09-10T19:27:47.345 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.0362314909999999, 2024-09-10T19:27:47.345 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-10T19:27:47.345 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T19:27:47.345 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-10T19:27:47.345 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.15360", 2024-09-10T19:27:47.345 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.183:0/2364804131", 2024-09-10T19:27:47.345 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 169782 2024-09-10T19:27:47.345 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:47.345 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-10T19:27:47.345 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:47.345 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-10T19:27:47.345 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:21:46.990784+0000", 2024-09-10T19:27:47.345 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:27:47.346 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:47.346 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:47.346 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-10T19:27:47.346 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:21:46.990784+0000", 2024-09-10T19:27:47.346 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:27:47.346 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:47.346 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:47.346 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-10T19:27:47.346 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:21:46.990782+0000", 2024-09-10T19:27:47.346 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999981 2024-09-10T19:27:47.346 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:47.346 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:47.346 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-10T19:27:47.346 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:21:46.990790+0000", 2024-09-10T19:27:47.347 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.4169999999999992e-06 2024-09-10T19:27:47.347 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:47.347 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:47.347 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-10T19:27:47.347 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:21:46.990803+0000", 2024-09-10T19:27:47.347 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.2993000000000001e-05 2024-09-10T19:27:47.347 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:47.347 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:47.347 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-10T19:27:47.347 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:21:46.990809+0000", 2024-09-10T19:27:47.347 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.0310000000000004e-06 2024-09-10T19:27:47.347 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:47.347 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:47.348 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:27:47.348 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:21:46.992349+0000", 2024-09-10T19:27:47.348 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.001539456 2024-09-10T19:27:47.348 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:47.348 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:47.348 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for peered", 2024-09-10T19:27:47.348 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:21:46.992355+0000", 2024-09-10T19:27:47.348 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.737e-06 2024-09-10T19:27:47.348 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:47.348 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:47.348 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:27:47.348 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:21:48.020802+0000", 2024-09-10T19:27:47.348 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.028446918 2024-09-10T19:27:47.349 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:47.349 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:47.349 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-10T19:27:47.349 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:21:48.021206+0000", 2024-09-10T19:27:47.349 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00040392599999999998 2024-09-10T19:27:47.349 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:47.349 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:47.349 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 5", 2024-09-10T19:27:47.349 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:21:48.021307+0000", 2024-09-10T19:27:47.349 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00010042100000000001 2024-09-10T19:27:47.349 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:47.349 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:47.349 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-09-10T19:27:47.350 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:21:48.026664+0000", 2024-09-10T19:27:47.350 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0053572689999999996 2024-09-10T19:27:47.350 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:47.350 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:47.350 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:27:47.350 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:21:48.026860+0000", 2024-09-10T19:27:47.350 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00019617899999999999 2024-09-10T19:27:47.350 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:47.350 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:47.350 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-09-10T19:27:47.350 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:21:48.026932+0000", 2024-09-10T19:27:47.350 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.2219999999999996e-05 2024-09-10T19:27:47.350 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:47.351 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:47.351 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-10T19:27:47.351 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:21:48.027015+0000", 2024-09-10T19:27:47.351 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.2974999999999998e-05 2024-09-10T19:27:47.351 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:27:47.351 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-10T19:27:47.351 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:27:47.351 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:47.351 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:47.351 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.15360.0:169784 9.b 9:d2cc5225:::benchmark_data_smithi183_142499_object23295:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e263)", 2024-09-10T19:27:47.351 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-10T19:21:46.992263+0000", 2024-09-10T19:27:47.351 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 360.30116607999997, 2024-09-10T19:27:47.351 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.04009342, 2024-09-10T19:27:47.351 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-10T19:27:47.352 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T19:27:47.352 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-10T19:27:47.352 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.15360", 2024-09-10T19:27:47.352 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.183:0/2364804131", 2024-09-10T19:27:47.352 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 169784 2024-09-10T19:27:47.352 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:47.352 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-10T19:27:47.352 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:47.352 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-10T19:27:47.352 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:21:46.992263+0000", 2024-09-10T19:27:47.352 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:27:47.352 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:47.352 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:47.353 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-10T19:27:47.353 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:21:46.992263+0000", 2024-09-10T19:27:47.353 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:27:47.353 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:47.353 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:47.353 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-10T19:27:47.353 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:21:46.992262+0000", 2024-09-10T19:27:47.353 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.999999 2024-09-10T19:27:47.353 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:47.353 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:47.353 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-10T19:27:47.353 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:21:46.992267+0000", 2024-09-10T19:27:47.353 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.5339999999999999e-06 2024-09-10T19:27:47.354 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:47.354 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:47.354 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-10T19:27:47.354 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:21:46.992279+0000", 2024-09-10T19:27:47.354 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.1938e-05 2024-09-10T19:27:47.354 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:47.354 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:47.354 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-10T19:27:47.354 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:21:46.992285+0000", 2024-09-10T19:27:47.354 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.677e-06 2024-09-10T19:27:47.354 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:47.354 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:47.354 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:27:47.354 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:21:46.992373+0000", 2024-09-10T19:27:47.355 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.7473999999999996e-05 2024-09-10T19:27:47.355 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:47.355 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:47.355 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for peered", 2024-09-10T19:27:47.355 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:21:46.992377+0000", 2024-09-10T19:27:47.355 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.5569999999999997e-06 2024-09-10T19:27:47.355 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:47.355 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:47.355 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:27:47.355 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:21:48.023395+0000", 2024-09-10T19:27:47.355 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.031017764 2024-09-10T19:27:47.355 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:47.355 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:47.355 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-10T19:27:47.356 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:21:48.023486+0000", 2024-09-10T19:27:47.356 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.1160000000000001e-05 2024-09-10T19:27:47.356 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:47.356 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:47.356 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 5", 2024-09-10T19:27:47.356 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:21:48.023593+0000", 2024-09-10T19:27:47.356 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000107265 2024-09-10T19:27:47.356 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:47.356 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:47.356 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-09-10T19:27:47.356 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:21:48.028806+0000", 2024-09-10T19:27:47.356 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0052124720000000001 2024-09-10T19:27:47.357 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:47.357 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:47.357 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:27:47.357 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:21:48.032228+0000", 2024-09-10T19:27:47.357 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0034222900000000001 2024-09-10T19:27:47.357 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:47.357 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:47.357 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-09-10T19:27:47.357 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:21:48.032276+0000", 2024-09-10T19:27:47.357 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.8106000000000003e-05 2024-09-10T19:27:47.357 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:47.357 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:47.357 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-10T19:27:47.357 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:21:48.032356+0000", 2024-09-10T19:27:47.358 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.0247999999999995e-05 2024-09-10T19:27:47.358 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:27:47.358 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-10T19:27:47.358 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:27:47.358 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:47.358 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:47.358 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.25622.0:12628 10.a 10:5dd0c111:::benchmark_data_smithi183_176537_object12627:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e307)", 2024-09-10T19:27:47.358 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-10T19:24:28.946449+0000", 2024-09-10T19:27:47.358 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 198.346980229, 2024-09-10T19:27:47.358 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.93712236599999998, 2024-09-10T19:27:47.358 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-10T19:27:47.358 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T19:27:47.358 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-10T19:27:47.358 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.25622", 2024-09-10T19:27:47.359 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.183:0/3685189199", 2024-09-10T19:27:47.359 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 12628 2024-09-10T19:27:47.359 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:47.359 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-10T19:27:47.359 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:47.359 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-10T19:27:47.359 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:24:28.946449+0000", 2024-09-10T19:27:47.359 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:27:47.359 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:47.359 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:47.359 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-10T19:27:47.359 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:24:28.946449+0000", 2024-09-10T19:27:47.359 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:27:47.360 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:47.360 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:47.360 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-10T19:27:47.360 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:24:28.946448+0000", 2024-09-10T19:27:47.360 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.999999 2024-09-10T19:27:47.360 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:47.360 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:47.360 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-10T19:27:47.360 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:24:28.946516+0000", 2024-09-10T19:27:47.360 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.7679e-05 2024-09-10T19:27:47.360 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:47.360 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:47.360 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-10T19:27:47.360 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:24:28.946525+0000", 2024-09-10T19:27:47.361 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.4979999999999995e-06 2024-09-10T19:27:47.361 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:47.361 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:47.361 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-10T19:27:47.361 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:24:28.946531+0000", 2024-09-10T19:27:47.361 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.6720000000000001e-06 2024-09-10T19:27:47.361 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:47.361 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:47.361 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:27:47.361 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:24:28.946794+0000", 2024-09-10T19:27:47.361 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00026344600000000003 2024-09-10T19:27:47.361 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:47.361 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:47.362 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for peered", 2024-09-10T19:27:47.362 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:24:28.946804+0000", 2024-09-10T19:27:47.362 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.9210000000000004e-06 2024-09-10T19:27:47.362 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:47.362 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:47.362 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:27:47.362 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:24:29.877833+0000", 2024-09-10T19:27:47.362 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.93102842699999999 2024-09-10T19:27:47.362 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:47.362 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:47.362 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-10T19:27:47.362 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:24:29.877997+0000", 2024-09-10T19:27:47.362 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00016494099999999999 2024-09-10T19:27:47.362 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:47.363 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:47.363 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0,5", 2024-09-10T19:27:47.363 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:24:29.878176+0000", 2024-09-10T19:27:47.363 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00017845900000000001 2024-09-10T19:27:47.363 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:47.363 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:47.363 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-09-10T19:27:47.363 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:24:29.883053+0000", 2024-09-10T19:27:47.363 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.004877363 2024-09-10T19:27:47.363 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:47.363 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:47.363 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:27:47.363 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:24:29.883448+0000", 2024-09-10T19:27:47.363 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00039474300000000003 2024-09-10T19:27:47.364 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:47.364 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:47.364 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:27:47.364 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:24:29.883484+0000", 2024-09-10T19:27:47.364 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.6440999999999999e-05 2024-09-10T19:27:47.364 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:47.364 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:47.364 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-09-10T19:27:47.364 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:24:29.883533+0000", 2024-09-10T19:27:47.364 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.8776000000000002e-05 2024-09-10T19:27:47.364 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:47.364 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:47.364 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-10T19:27:47.365 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:24:29.883571+0000", 2024-09-10T19:27:47.365 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.7910000000000001e-05 2024-09-10T19:27:47.365 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:27:47.365 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-10T19:27:47.365 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:27:47.365 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:47.365 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:47.365 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.25622.0:12630 10.a 10:5fb2a500:::benchmark_data_smithi183_176537_object12629:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e307)", 2024-09-10T19:27:47.365 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-10T19:24:28.948059+0000", 2024-09-10T19:27:47.365 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 198.345370422, 2024-09-10T19:27:47.365 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.935686668, 2024-09-10T19:27:47.365 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-10T19:27:47.365 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T19:27:47.365 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-10T19:27:47.366 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.25622", 2024-09-10T19:27:47.366 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.183:0/3685189199", 2024-09-10T19:27:47.366 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 12630 2024-09-10T19:27:47.366 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:47.366 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-10T19:27:47.366 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:47.366 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-10T19:27:47.366 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:24:28.948059+0000", 2024-09-10T19:27:47.366 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:27:47.366 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:47.366 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:47.366 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-10T19:27:47.366 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:24:28.948059+0000", 2024-09-10T19:27:47.366 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:27:47.367 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:47.367 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:47.367 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-10T19:27:47.367 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:24:28.948057+0000", 2024-09-10T19:27:47.367 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999986 2024-09-10T19:27:47.367 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:47.367 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:47.367 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-10T19:27:47.367 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:24:28.948135+0000", 2024-09-10T19:27:47.367 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.8008000000000003e-05 2024-09-10T19:27:47.368 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:47.369 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:47.369 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-10T19:27:47.369 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:24:28.948148+0000", 2024-09-10T19:27:47.369 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.3161e-05 2024-09-10T19:27:47.369 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:47.369 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:47.369 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-10T19:27:47.369 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:24:28.948155+0000", 2024-09-10T19:27:47.369 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.2890000000000003e-06 2024-09-10T19:27:47.369 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:47.369 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:47.369 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:27:47.369 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:24:28.948206+0000", 2024-09-10T19:27:47.370 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.1678000000000002e-05 2024-09-10T19:27:47.370 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:47.370 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:47.370 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for peered", 2024-09-10T19:27:47.370 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:24:28.948217+0000", 2024-09-10T19:27:47.370 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.0465000000000001e-05 2024-09-10T19:27:47.370 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:47.370 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:47.370 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:27:47.370 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:24:29.878966+0000", 2024-09-10T19:27:47.370 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.930749732 2024-09-10T19:27:47.370 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:47.370 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:47.371 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-10T19:27:47.371 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:24:29.879060+0000", 2024-09-10T19:27:47.371 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.3184999999999996e-05 2024-09-10T19:27:47.371 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:47.371 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:47.371 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0,5", 2024-09-10T19:27:47.371 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:24:29.879198+0000", 2024-09-10T19:27:47.371 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00013824700000000001 2024-09-10T19:27:47.371 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:47.371 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:47.371 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-09-10T19:27:47.371 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:24:29.883060+0000", 2024-09-10T19:27:47.371 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0038619230000000002 2024-09-10T19:27:47.372 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:47.372 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:47.372 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:27:47.372 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:24:29.883608+0000", 2024-09-10T19:27:47.372 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00054817799999999997 2024-09-10T19:27:47.372 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:47.372 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:47.372 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:27:47.372 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:24:29.883667+0000", 2024-09-10T19:27:47.372 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.9132999999999998e-05 2024-09-10T19:27:47.372 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:47.372 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:47.372 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-09-10T19:27:47.373 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:24:29.883713+0000", 2024-09-10T19:27:47.373 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.5405000000000003e-05 2024-09-10T19:27:47.373 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:47.373 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:47.373 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-10T19:27:47.373 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:24:29.883745+0000", 2024-09-10T19:27:47.373 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.2724e-05 2024-09-10T19:27:47.373 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:27:47.373 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-10T19:27:47.373 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:27:47.373 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:47.373 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:47.373 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.25622.0:12666 10.a 10:5cac3cb0:::benchmark_data_smithi183_176537_object12665:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e307)", 2024-09-10T19:27:47.373 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-10T19:24:28.959299+0000", 2024-09-10T19:27:47.374 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 198.334130139, 2024-09-10T19:27:47.374 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.93215646600000002, 2024-09-10T19:27:47.374 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-10T19:27:47.374 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T19:27:47.374 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-10T19:27:47.374 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.25622", 2024-09-10T19:27:47.374 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.183:0/3685189199", 2024-09-10T19:27:47.374 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 12666 2024-09-10T19:27:47.374 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:47.374 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-10T19:27:47.374 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:47.374 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-10T19:27:47.374 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:24:28.959299+0000", 2024-09-10T19:27:47.375 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:27:47.375 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:47.375 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:47.375 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-10T19:27:47.375 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:24:28.959299+0000", 2024-09-10T19:27:47.375 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:27:47.375 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:47.375 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:47.375 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-10T19:27:47.375 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:24:28.959298+0000", 2024-09-10T19:27:47.375 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.999999 2024-09-10T19:27:47.375 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:47.375 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:47.376 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-10T19:27:47.376 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:24:28.959353+0000", 2024-09-10T19:27:47.376 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.4951999999999998e-05 2024-09-10T19:27:47.376 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:47.376 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:47.376 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-10T19:27:47.376 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:24:28.959365+0000", 2024-09-10T19:27:47.376 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.2214e-05 2024-09-10T19:27:47.376 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:47.376 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:47.376 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-10T19:27:47.376 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:24:28.959371+0000", 2024-09-10T19:27:47.376 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.5609999999999998e-06 2024-09-10T19:27:47.376 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:47.377 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:47.377 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:27:47.377 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:24:28.959438+0000", 2024-09-10T19:27:47.377 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.6937e-05 2024-09-10T19:27:47.377 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:47.377 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:47.377 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for peered", 2024-09-10T19:27:47.377 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:24:28.959447+0000", 2024-09-10T19:27:47.377 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.1570000000000006e-06 2024-09-10T19:27:47.377 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:47.377 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:47.377 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:27:47.377 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:24:29.882013+0000", 2024-09-10T19:27:47.378 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.92256636599999997 2024-09-10T19:27:47.378 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:47.378 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:47.378 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-10T19:27:47.378 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:24:29.882151+0000", 2024-09-10T19:27:47.378 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000137701 2024-09-10T19:27:47.378 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:47.378 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:47.378 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0,5", 2024-09-10T19:27:47.378 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:24:29.882331+0000", 2024-09-10T19:27:47.378 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00017971299999999999 2024-09-10T19:27:47.378 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:47.378 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:47.378 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-09-10T19:27:47.379 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:24:29.883455+0000", 2024-09-10T19:27:47.379 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0011240169999999999 2024-09-10T19:27:47.379 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:47.379 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:47.379 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:27:47.379 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:24:29.884910+0000", 2024-09-10T19:27:47.379 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.001455795 2024-09-10T19:27:47.379 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:47.379 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:47.379 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:27:47.379 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:24:29.891379+0000", 2024-09-10T19:27:47.379 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0064681690000000002 2024-09-10T19:27:47.379 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:47.379 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:47.380 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-09-10T19:27:47.380 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:24:29.891420+0000", 2024-09-10T19:27:47.380 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.1023999999999999e-05 2024-09-10T19:27:47.380 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:47.380 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:47.380 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-10T19:27:47.380 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:24:29.891455+0000", 2024-09-10T19:27:47.380 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.5738999999999998e-05 2024-09-10T19:27:47.380 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:27:47.380 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-10T19:27:47.380 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:27:47.380 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:47.380 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:47.381 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.25622.0:73667 10.7 10:e2d404cb:::benchmark_data_smithi183_176537_object73666:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e326)", 2024-09-10T19:27:47.381 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-10T19:25:09.295868+0000", 2024-09-10T19:27:47.381 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 157.99756102000001, 2024-09-10T19:27:47.381 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.13609428200000001, 2024-09-10T19:27:47.381 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-10T19:27:47.381 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T19:27:47.381 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-10T19:27:47.381 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.25622", 2024-09-10T19:27:47.381 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.183:0/3685189199", 2024-09-10T19:27:47.381 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 73667 2024-09-10T19:27:47.381 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:47.381 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-10T19:27:47.381 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:47.381 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-10T19:27:47.382 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:25:09.295868+0000", 2024-09-10T19:27:47.382 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:27:47.382 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:47.382 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:47.382 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-10T19:27:47.382 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:25:09.295868+0000", 2024-09-10T19:27:47.382 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:27:47.382 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:47.382 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:47.382 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-10T19:27:47.382 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:25:09.295867+0000", 2024-09-10T19:27:47.382 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999995 2024-09-10T19:27:47.382 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:47.383 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:47.383 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-10T19:27:47.383 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:25:09.295900+0000", 2024-09-10T19:27:47.383 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.2632000000000002e-05 2024-09-10T19:27:47.383 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:47.383 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:47.383 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-10T19:27:47.383 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:25:09.295907+0000", 2024-09-10T19:27:47.383 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.6760000000000001e-06 2024-09-10T19:27:47.383 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:47.383 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:47.383 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-10T19:27:47.384 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:25:09.295912+0000", 2024-09-10T19:27:47.384 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.814e-06 2024-09-10T19:27:47.384 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:47.384 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:47.384 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:27:47.384 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:25:09.298112+0000", 2024-09-10T19:27:47.384 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0022006550000000001 2024-09-10T19:27:47.384 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:47.384 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:47.384 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-10T19:27:47.384 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:25:09.298226+0000", 2024-09-10T19:27:47.384 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000113621 2024-09-10T19:27:47.384 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:47.385 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:47.385 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 3,5", 2024-09-10T19:27:47.385 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:25:09.298348+0000", 2024-09-10T19:27:47.385 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00012250499999999999 2024-09-10T19:27:47.385 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:47.385 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:47.385 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-09-10T19:27:47.385 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:25:09.305144+0000", 2024-09-10T19:27:47.385 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0067952489999999997 2024-09-10T19:27:47.385 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:47.385 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:47.385 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:27:47.385 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:25:09.311783+0000", 2024-09-10T19:27:47.385 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0066396329999999998 2024-09-10T19:27:47.386 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:47.386 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:47.386 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:27:47.386 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:25:09.431541+0000", 2024-09-10T19:27:47.386 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.11975781000000001 2024-09-10T19:27:47.386 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:47.386 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:47.386 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-09-10T19:27:47.386 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:25:09.431663+0000", 2024-09-10T19:27:47.386 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000121944 2024-09-10T19:27:47.386 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:47.386 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:47.386 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-10T19:27:47.387 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:25:09.431962+0000", 2024-09-10T19:27:47.387 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00029935500000000001 2024-09-10T19:27:47.387 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:27:47.387 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-10T19:27:47.387 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:27:47.387 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:47.387 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:47.387 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.25622.0:75586 10.7 10:e79d0d88:::benchmark_data_smithi183_176537_object75585:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e327)", 2024-09-10T19:27:47.387 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-10T19:25:11.416205+0000", 2024-09-10T19:27:47.387 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 155.87722417399999, 2024-09-10T19:27:47.387 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.26102794400000001, 2024-09-10T19:27:47.387 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-10T19:27:47.387 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T19:27:47.387 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-10T19:27:47.388 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.25622", 2024-09-10T19:27:47.388 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.183:0/3685189199", 2024-09-10T19:27:47.388 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 75586 2024-09-10T19:27:47.388 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:47.388 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-10T19:27:47.388 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:47.388 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-10T19:27:47.388 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:25:11.416205+0000", 2024-09-10T19:27:47.388 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:27:47.388 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:47.388 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:47.388 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-10T19:27:47.388 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:25:11.416205+0000", 2024-09-10T19:27:47.389 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:27:47.389 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:47.389 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:47.389 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-10T19:27:47.389 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:25:11.416204+0000", 2024-09-10T19:27:47.389 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999995 2024-09-10T19:27:47.389 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:47.389 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:47.389 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-10T19:27:47.389 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:25:11.416268+0000", 2024-09-10T19:27:47.389 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.3509999999999993e-05 2024-09-10T19:27:47.389 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:47.389 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:47.389 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-10T19:27:47.390 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:25:11.416280+0000", 2024-09-10T19:27:47.390 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.2264e-05 2024-09-10T19:27:47.390 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:47.390 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:47.390 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-10T19:27:47.390 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:25:11.416286+0000", 2024-09-10T19:27:47.390 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.4670000000000003e-06 2024-09-10T19:27:47.390 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:47.390 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:47.390 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:27:47.390 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:25:11.420151+0000", 2024-09-10T19:27:47.390 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0038651900000000001 2024-09-10T19:27:47.391 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:47.391 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:47.391 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-10T19:27:47.391 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:25:11.420252+0000", 2024-09-10T19:27:47.391 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000100923 2024-09-10T19:27:47.391 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:47.391 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:47.391 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 3,5", 2024-09-10T19:27:47.391 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:25:11.420365+0000", 2024-09-10T19:27:47.391 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00011299200000000001 2024-09-10T19:27:47.391 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:47.391 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:47.391 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:27:47.391 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:25:11.428878+0000", 2024-09-10T19:27:47.392 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0085136380000000005 2024-09-10T19:27:47.392 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:47.392 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:47.392 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-09-10T19:27:47.392 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:25:11.430023+0000", 2024-09-10T19:27:47.392 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.001144915 2024-09-10T19:27:47.392 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:47.392 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:47.392 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:27:47.392 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:25:11.677088+0000", 2024-09-10T19:27:47.392 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.247064382 2024-09-10T19:27:47.392 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:47.392 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:47.392 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-09-10T19:27:47.393 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:25:11.677184+0000", 2024-09-10T19:27:47.393 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.6521000000000006e-05 2024-09-10T19:27:47.393 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:47.393 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:47.393 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-10T19:27:47.393 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:25:11.677233+0000", 2024-09-10T19:27:47.393 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.8662999999999998e-05 2024-09-10T19:27:47.393 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:27:47.393 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-10T19:27:47.393 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:27:47.393 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:47.393 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:47.393 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.25622.0:75587 10.7 10:ea69e0d6:::benchmark_data_smithi183_176537_object75586:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e327)", 2024-09-10T19:27:47.394 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-10T19:25:11.417779+0000", 2024-09-10T19:27:47.394 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 155.87565012900001, 2024-09-10T19:27:47.394 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.27165624599999999, 2024-09-10T19:27:47.394 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-10T19:27:47.394 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T19:27:47.394 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-10T19:27:47.394 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.25622", 2024-09-10T19:27:47.394 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.183:0/3685189199", 2024-09-10T19:27:47.394 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 75587 2024-09-10T19:27:47.394 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:47.394 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-10T19:27:47.394 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:47.394 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-10T19:27:47.395 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:25:11.417779+0000", 2024-09-10T19:27:47.395 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:27:47.395 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:47.395 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:47.395 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-10T19:27:47.395 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:25:11.417779+0000", 2024-09-10T19:27:47.395 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:27:47.395 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:47.395 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:47.395 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-10T19:27:47.395 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:25:11.417778+0000", 2024-09-10T19:27:47.395 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.999999 2024-09-10T19:27:47.395 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:47.396 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:47.396 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-10T19:27:47.396 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:25:11.417810+0000", 2024-09-10T19:27:47.396 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.2657000000000003e-05 2024-09-10T19:27:47.396 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:47.396 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:47.396 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-10T19:27:47.396 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:25:11.417823+0000", 2024-09-10T19:27:47.396 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.2299e-05 2024-09-10T19:27:47.396 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:47.396 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:47.396 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-10T19:27:47.396 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:25:11.417828+0000", 2024-09-10T19:27:47.397 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.6860000000000003e-06 2024-09-10T19:27:47.397 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:47.397 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:47.397 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:27:47.397 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:25:11.423297+0000", 2024-09-10T19:27:47.397 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0054688610000000002 2024-09-10T19:27:47.397 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:47.397 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:47.397 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-10T19:27:47.397 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:25:11.423415+0000", 2024-09-10T19:27:47.397 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00011759700000000001 2024-09-10T19:27:47.397 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:47.397 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:47.398 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 3,5", 2024-09-10T19:27:47.398 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:25:11.423561+0000", 2024-09-10T19:27:47.398 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00014665699999999999 2024-09-10T19:27:47.398 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:47.398 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:47.398 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:27:47.398 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:25:11.428902+0000", 2024-09-10T19:27:47.398 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0053408320000000002 2024-09-10T19:27:47.398 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:47.398 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:47.398 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-09-10T19:27:47.398 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:25:11.430028+0000", 2024-09-10T19:27:47.398 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.001125626 2024-09-10T19:27:47.398 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:47.399 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:47.399 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:27:47.399 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:25:11.689335+0000", 2024-09-10T19:27:47.399 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.259307171 2024-09-10T19:27:47.399 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:47.399 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:47.399 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-09-10T19:27:47.399 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:25:11.689385+0000", 2024-09-10T19:27:47.399 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.9749000000000001e-05 2024-09-10T19:27:47.399 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:47.399 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:47.399 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-10T19:27:47.399 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:25:11.689435+0000", 2024-09-10T19:27:47.400 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.0299000000000001e-05 2024-09-10T19:27:47.400 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:27:47.400 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-10T19:27:47.400 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:27:47.400 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:47.400 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:47.400 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.25622.0:75593 10.a 10:51a0c1e6:::benchmark_data_smithi183_176537_object75592:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e327)", 2024-09-10T19:27:47.400 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-10T19:25:11.421202+0000", 2024-09-10T19:27:47.400 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 155.87222738099999, 2024-09-10T19:27:47.400 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.26489980899999999, 2024-09-10T19:27:47.400 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-10T19:27:47.400 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T19:27:47.401 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-10T19:27:47.401 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.25622", 2024-09-10T19:27:47.401 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.183:0/3685189199", 2024-09-10T19:27:47.401 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 75593 2024-09-10T19:27:47.401 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:47.401 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-10T19:27:47.401 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:47.401 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-10T19:27:47.401 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:25:11.421202+0000", 2024-09-10T19:27:47.401 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:27:47.401 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:47.402 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:47.402 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-10T19:27:47.402 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:25:11.421202+0000", 2024-09-10T19:27:47.402 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:27:47.402 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:47.402 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:47.402 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-10T19:27:47.402 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:25:11.421201+0000", 2024-09-10T19:27:47.402 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999995 2024-09-10T19:27:47.402 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:47.402 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:47.402 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-10T19:27:47.403 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:25:11.421231+0000", 2024-09-10T19:27:47.403 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.9921e-05 2024-09-10T19:27:47.403 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:47.403 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:47.403 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-10T19:27:47.403 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:25:11.421238+0000", 2024-09-10T19:27:47.403 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.421e-06 2024-09-10T19:27:47.403 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:47.403 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:47.403 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-10T19:27:47.403 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:25:11.421241+0000", 2024-09-10T19:27:47.403 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.6739999999999999e-06 2024-09-10T19:27:47.403 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:47.404 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:47.404 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:27:47.404 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:25:11.421422+0000", 2024-09-10T19:27:47.404 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00018082300000000001 2024-09-10T19:27:47.404 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:47.404 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:47.404 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-10T19:27:47.404 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:25:11.421566+0000", 2024-09-10T19:27:47.404 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00014406099999999999 2024-09-10T19:27:47.404 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:47.404 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:47.404 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0,5", 2024-09-10T19:27:47.404 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:25:11.421701+0000", 2024-09-10T19:27:47.405 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00013509199999999999 2024-09-10T19:27:47.405 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:47.405 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:47.405 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:27:47.405 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:25:11.425060+0000", 2024-09-10T19:27:47.405 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0033590120000000002 2024-09-10T19:27:47.405 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:47.405 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:47.405 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-09-10T19:27:47.405 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:25:11.425069+0000", 2024-09-10T19:27:47.405 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.2399999999999996e-06 2024-09-10T19:27:47.405 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:47.405 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:47.406 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:27:47.406 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:25:11.686011+0000", 2024-09-10T19:27:47.406 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.260941861 2024-09-10T19:27:47.406 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:47.406 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:47.406 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-09-10T19:27:47.406 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:25:11.686062+0000", 2024-09-10T19:27:47.406 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.0568000000000001e-05 2024-09-10T19:27:47.406 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:47.406 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:47.406 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-10T19:27:47.406 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:25:11.686101+0000", 2024-09-10T19:27:47.406 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.9561000000000003e-05 2024-09-10T19:27:47.407 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:27:47.407 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-10T19:27:47.407 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:27:47.407 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:47.407 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:47.407 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.25622.0:75596 10.7 10:ed839bb7:::benchmark_data_smithi183_176537_object75595:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e327)", 2024-09-10T19:27:47.407 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-10T19:25:11.422174+0000", 2024-09-10T19:27:47.407 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 155.87125483200001, 2024-09-10T19:27:47.407 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.267390299, 2024-09-10T19:27:47.407 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-10T19:27:47.407 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T19:27:47.407 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-10T19:27:47.407 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.25622", 2024-09-10T19:27:47.408 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.183:0/3685189199", 2024-09-10T19:27:47.408 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 75596 2024-09-10T19:27:47.408 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:47.408 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-10T19:27:47.408 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:47.408 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-10T19:27:47.408 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:25:11.422174+0000", 2024-09-10T19:27:47.408 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:27:47.408 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:47.409 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:47.409 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-10T19:27:47.409 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:25:11.422174+0000", 2024-09-10T19:27:47.409 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:27:47.409 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:47.409 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:47.409 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-10T19:27:47.409 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:25:11.422173+0000", 2024-09-10T19:27:47.409 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.999999 2024-09-10T19:27:47.409 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:47.409 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:47.409 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-10T19:27:47.409 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:25:11.422237+0000", 2024-09-10T19:27:47.410 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.4103000000000006e-05 2024-09-10T19:27:47.410 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:47.410 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:47.410 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-10T19:27:47.410 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:25:11.422249+0000", 2024-09-10T19:27:47.410 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.1925e-05 2024-09-10T19:27:47.410 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:47.410 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:47.410 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-10T19:27:47.410 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:25:11.422254+0000", 2024-09-10T19:27:47.410 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.2689999999999999e-06 2024-09-10T19:27:47.410 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:47.410 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:47.410 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:27:47.411 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:25:11.426201+0000", 2024-09-10T19:27:47.411 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00394613 2024-09-10T19:27:47.411 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:47.411 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:47.411 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-10T19:27:47.411 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:25:11.426315+0000", 2024-09-10T19:27:47.411 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000114838 2024-09-10T19:27:47.411 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:47.411 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:47.411 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 3,5", 2024-09-10T19:27:47.411 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:25:11.426433+0000", 2024-09-10T19:27:47.411 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000118023 2024-09-10T19:27:47.411 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:47.412 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:47.412 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-09-10T19:27:47.412 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:25:11.430034+0000", 2024-09-10T19:27:47.412 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0036005450000000001 2024-09-10T19:27:47.412 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:47.412 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:47.412 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:27:47.412 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:25:11.430225+0000", 2024-09-10T19:27:47.412 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000190664 2024-09-10T19:27:47.412 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:47.412 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:47.412 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:27:47.413 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:25:11.689478+0000", 2024-09-10T19:27:47.413 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.25925326900000001 2024-09-10T19:27:47.413 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:47.413 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:47.413 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-09-10T19:27:47.413 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:25:11.689531+0000", 2024-09-10T19:27:47.413 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.2713999999999997e-05 2024-09-10T19:27:47.413 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:47.413 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:47.413 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-10T19:27:47.413 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:25:11.689565+0000", 2024-09-10T19:27:47.413 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.3917000000000001e-05 2024-09-10T19:27:47.413 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:27:47.413 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-10T19:27:47.414 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:27:47.414 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:47.414 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:47.414 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.25622.0:75604 10.a 10:5539316d:::benchmark_data_smithi183_176537_object75603:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e327)", 2024-09-10T19:27:47.414 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-10T19:25:11.428706+0000", 2024-09-10T19:27:47.414 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 155.86472343599999, 2024-09-10T19:27:47.414 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.26062794299999997, 2024-09-10T19:27:47.414 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-10T19:27:47.414 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T19:27:47.414 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-10T19:27:47.414 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.25622", 2024-09-10T19:27:47.414 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.183:0/3685189199", 2024-09-10T19:27:47.414 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 75604 2024-09-10T19:27:47.415 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:47.415 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-10T19:27:47.415 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:47.415 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-10T19:27:47.415 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:25:11.428706+0000", 2024-09-10T19:27:47.415 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:27:47.415 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:47.415 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:47.415 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-10T19:27:47.415 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:25:11.428706+0000", 2024-09-10T19:27:47.415 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:27:47.415 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:47.415 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:47.416 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-10T19:27:47.416 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:25:11.428704+0000", 2024-09-10T19:27:47.416 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999986 2024-09-10T19:27:47.416 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:47.416 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:47.416 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-10T19:27:47.416 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:25:11.428767+0000", 2024-09-10T19:27:47.416 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.2732999999999997e-05 2024-09-10T19:27:47.416 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:47.416 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:47.416 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-10T19:27:47.416 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:25:11.428781+0000", 2024-09-10T19:27:47.416 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.3672999999999999e-05 2024-09-10T19:27:47.417 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:47.417 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:47.417 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-10T19:27:47.417 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:25:11.428786+0000", 2024-09-10T19:27:47.417 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.5310000000000001e-06 2024-09-10T19:27:47.417 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:47.417 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:47.417 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:27:47.417 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:25:11.428828+0000", 2024-09-10T19:27:47.417 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.1505000000000003e-05 2024-09-10T19:27:47.417 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:47.417 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:47.417 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-10T19:27:47.417 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:25:11.429060+0000", 2024-09-10T19:27:47.418 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00023256000000000001 2024-09-10T19:27:47.418 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:47.418 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:47.418 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0,5", 2024-09-10T19:27:47.418 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:25:11.429276+0000", 2024-09-10T19:27:47.418 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000215603 2024-09-10T19:27:47.418 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:47.418 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:47.418 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-09-10T19:27:47.418 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:25:11.432225+0000", 2024-09-10T19:27:47.418 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.002949063 2024-09-10T19:27:47.418 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:47.418 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:47.419 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:27:47.419 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:25:11.434188+0000", 2024-09-10T19:27:47.419 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0019629059999999999 2024-09-10T19:27:47.419 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:47.419 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:47.419 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:27:47.419 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:25:11.689233+0000", 2024-09-10T19:27:47.419 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.25504526399999999 2024-09-10T19:27:47.419 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:47.419 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:47.419 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-09-10T19:27:47.419 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:25:11.689291+0000", 2024-09-10T19:27:47.419 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.8059000000000001e-05 2024-09-10T19:27:47.420 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:47.420 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:47.420 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-10T19:27:47.420 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:25:11.689334+0000", 2024-09-10T19:27:47.420 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.2419000000000002e-05 2024-09-10T19:27:47.420 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:27:47.420 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-10T19:27:47.420 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:27:47.420 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:47.420 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:47.420 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.25622.0:75606 10.7 10:e001ef56:::benchmark_data_smithi183_176537_object75605:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e327)", 2024-09-10T19:27:47.420 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-10T19:25:11.432040+0000", 2024-09-10T19:27:47.420 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 155.86138948000001, 2024-09-10T19:27:47.421 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.257691526, 2024-09-10T19:27:47.421 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-10T19:27:47.421 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T19:27:47.421 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-10T19:27:47.421 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.25622", 2024-09-10T19:27:47.421 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.183:0/3685189199", 2024-09-10T19:27:47.421 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 75606 2024-09-10T19:27:47.421 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:47.421 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-10T19:27:47.421 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:47.421 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-10T19:27:47.421 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:25:11.432040+0000", 2024-09-10T19:27:47.421 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:27:47.422 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:47.422 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:47.422 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-10T19:27:47.422 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:25:11.432040+0000", 2024-09-10T19:27:47.422 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:27:47.422 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:47.422 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:47.422 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-10T19:27:47.422 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:25:11.432038+0000", 2024-09-10T19:27:47.422 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999986 2024-09-10T19:27:47.422 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:47.422 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:47.422 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-10T19:27:47.423 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:25:11.432077+0000", 2024-09-10T19:27:47.423 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.9103999999999997e-05 2024-09-10T19:27:47.423 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:47.423 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:47.423 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-10T19:27:47.423 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:25:11.432091+0000", 2024-09-10T19:27:47.423 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.3196999999999999e-05 2024-09-10T19:27:47.423 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:47.423 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:47.423 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-10T19:27:47.423 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:25:11.432097+0000", 2024-09-10T19:27:47.423 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.1079999999999998e-06 2024-09-10T19:27:47.423 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:47.423 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:47.424 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:27:47.424 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:25:11.432244+0000", 2024-09-10T19:27:47.424 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000147417 2024-09-10T19:27:47.424 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:47.424 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:47.424 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-10T19:27:47.424 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:25:11.432361+0000", 2024-09-10T19:27:47.424 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000117199 2024-09-10T19:27:47.424 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:47.424 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:47.424 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 3,5", 2024-09-10T19:27:47.424 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:25:11.432493+0000", 2024-09-10T19:27:47.424 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000131259 2024-09-10T19:27:47.425 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:47.425 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:47.425 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-09-10T19:27:47.425 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:25:11.434716+0000", 2024-09-10T19:27:47.425 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.002223521 2024-09-10T19:27:47.425 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:47.425 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:47.425 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:27:47.425 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:25:11.435380+0000", 2024-09-10T19:27:47.425 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00066381899999999995 2024-09-10T19:27:47.425 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:47.425 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:47.426 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:27:47.426 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:25:11.689602+0000", 2024-09-10T19:27:47.426 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.25422249299999999 2024-09-10T19:27:47.426 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:47.426 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:47.426 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-09-10T19:27:47.426 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:25:11.689689+0000", 2024-09-10T19:27:47.426 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.7091999999999999e-05 2024-09-10T19:27:47.426 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:47.426 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:47.426 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-10T19:27:47.426 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:25:11.689731+0000", 2024-09-10T19:27:47.427 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.1650000000000003e-05 2024-09-10T19:27:47.427 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:27:47.427 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-10T19:27:47.427 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:27:47.427 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:47.427 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:47.427 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.25622.0:75613 10.a 10:5044ed86:::benchmark_data_smithi183_176537_object75612:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e327)", 2024-09-10T19:27:47.427 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-10T19:25:11.435976+0000", 2024-09-10T19:27:47.427 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 155.85745346300001, 2024-09-10T19:27:47.427 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.253482241, 2024-09-10T19:27:47.427 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-10T19:27:47.427 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T19:27:47.428 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-10T19:27:47.428 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.25622", 2024-09-10T19:27:47.428 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.183:0/3685189199", 2024-09-10T19:27:47.428 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 75613 2024-09-10T19:27:47.428 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:47.428 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-10T19:27:47.428 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:47.428 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-10T19:27:47.428 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:25:11.435976+0000", 2024-09-10T19:27:47.428 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:27:47.428 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:47.428 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:47.428 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-10T19:27:47.429 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:25:11.435976+0000", 2024-09-10T19:27:47.429 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:27:47.429 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:47.429 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:47.429 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-10T19:27:47.429 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:25:11.435974+0000", 2024-09-10T19:27:47.429 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999986 2024-09-10T19:27:47.429 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:47.429 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:47.429 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-10T19:27:47.429 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:25:11.436036+0000", 2024-09-10T19:27:47.429 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.1719999999999999e-05 2024-09-10T19:27:47.429 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:47.429 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:47.430 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-10T19:27:47.430 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:25:11.436047+0000", 2024-09-10T19:27:47.430 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.0744000000000001e-05 2024-09-10T19:27:47.430 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:47.430 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:47.430 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-10T19:27:47.430 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:25:11.436051+0000", 2024-09-10T19:27:47.430 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.1690000000000002e-06 2024-09-10T19:27:47.430 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:47.430 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:47.430 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:27:47.430 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:25:11.436089+0000", 2024-09-10T19:27:47.431 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.7768999999999997e-05 2024-09-10T19:27:47.431 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:47.431 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:47.431 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-10T19:27:47.431 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:25:11.436209+0000", 2024-09-10T19:27:47.431 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000120259 2024-09-10T19:27:47.431 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:47.431 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:47.431 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0,5", 2024-09-10T19:27:47.431 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:25:11.436349+0000", 2024-09-10T19:27:47.431 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00013955599999999999 2024-09-10T19:27:47.431 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:47.431 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:47.432 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-09-10T19:27:47.432 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:25:11.438701+0000", 2024-09-10T19:27:47.432 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0023522249999999999 2024-09-10T19:27:47.432 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:47.432 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:47.432 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:27:47.432 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:25:11.438942+0000", 2024-09-10T19:27:47.432 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00024158299999999999 2024-09-10T19:27:47.432 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:47.432 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:47.432 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:27:47.432 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:25:11.689377+0000", 2024-09-10T19:27:47.432 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.25043495900000001 2024-09-10T19:27:47.433 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:47.433 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:47.433 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-09-10T19:27:47.433 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:25:11.689427+0000", 2024-09-10T19:27:47.433 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.9978999999999997e-05 2024-09-10T19:27:47.433 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:47.433 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:47.433 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-10T19:27:47.433 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:25:11.689458+0000", 2024-09-10T19:27:47.433 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.0488e-05 2024-09-10T19:27:47.433 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:27:47.433 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-10T19:27:47.433 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:27:47.434 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:47.434 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:47.434 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.25622.0:75617 10.7 10:ecaf8376:::benchmark_data_smithi183_176537_object75616:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e327)", 2024-09-10T19:27:47.434 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-10T19:25:11.439011+0000", 2024-09-10T19:27:47.434 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 155.85441793699999, 2024-09-10T19:27:47.434 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.25604972199999998, 2024-09-10T19:27:47.434 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-10T19:27:47.434 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T19:27:47.434 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-10T19:27:47.434 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.25622", 2024-09-10T19:27:47.434 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.183:0/3685189199", 2024-09-10T19:27:47.434 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 75617 2024-09-10T19:27:47.434 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:47.435 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-10T19:27:47.435 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:47.435 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-10T19:27:47.435 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:25:11.439011+0000", 2024-09-10T19:27:47.435 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:27:47.435 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:47.435 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:47.435 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-10T19:27:47.435 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:25:11.439011+0000", 2024-09-10T19:27:47.435 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:27:47.435 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:47.435 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:47.435 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-10T19:27:47.436 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:25:11.439010+0000", 2024-09-10T19:27:47.436 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999986 2024-09-10T19:27:47.436 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:47.436 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:47.436 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-10T19:27:47.436 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:25:11.439059+0000", 2024-09-10T19:27:47.436 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.9817000000000001e-05 2024-09-10T19:27:47.436 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:47.436 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:47.436 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-10T19:27:47.436 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:25:11.439074+0000", 2024-09-10T19:27:47.436 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.4117999999999999e-05 2024-09-10T19:27:47.436 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:47.437 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:47.437 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-10T19:27:47.437 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:25:11.439079+0000", 2024-09-10T19:27:47.437 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.6529999999999999e-06 2024-09-10T19:27:47.437 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:47.437 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:47.437 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:27:47.437 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:25:11.439258+0000", 2024-09-10T19:27:47.437 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00017832899999999999 2024-09-10T19:27:47.437 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:47.437 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:47.437 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-10T19:27:47.437 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:25:11.439369+0000", 2024-09-10T19:27:47.438 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000111075 2024-09-10T19:27:47.438 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:47.438 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:47.438 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 3,5", 2024-09-10T19:27:47.438 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:25:11.439491+0000", 2024-09-10T19:27:47.438 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00012204100000000001 2024-09-10T19:27:47.438 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:47.438 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:47.438 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-09-10T19:27:47.438 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:25:11.442277+0000", 2024-09-10T19:27:47.438 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0027867460000000001 2024-09-10T19:27:47.438 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:47.438 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:47.439 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:27:47.439 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:25:11.443460+0000", 2024-09-10T19:27:47.439 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.001182759 2024-09-10T19:27:47.439 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:47.439 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:47.439 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:27:47.439 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:25:11.693503+0000", 2024-09-10T19:27:47.439 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.250042771 2024-09-10T19:27:47.439 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:47.439 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:47.439 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-09-10T19:27:47.439 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:25:11.693911+0000", 2024-09-10T19:27:47.439 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00040811800000000001 2024-09-10T19:27:47.440 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:47.440 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:47.440 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-10T19:27:47.440 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:25:11.695061+0000", 2024-09-10T19:27:47.440 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0011497230000000001 2024-09-10T19:27:47.440 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:27:47.440 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-10T19:27:47.440 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:27:47.440 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:47.440 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:47.440 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_repop(client.25622.0:244547 10.f e365/359 10:f9fe1760:::benchmark_data_smithi183_176537_object121339:head v 365'15233, mlcod=365'15230)", 2024-09-10T19:27:47.440 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-10T19:27:26.758325+0000", 2024-09-10T19:27:47.441 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 20.535104536999999, 2024-09-10T19:27:47.441 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.017058633, 2024-09-10T19:27:47.441 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-10T19:27:47.441 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T19:27:47.441 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-10T19:27:47.441 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:47.441 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-10T19:27:47.441 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:27:26.758325+0000", 2024-09-10T19:27:47.441 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:27:47.441 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:47.441 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:47.441 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-10T19:27:47.442 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:27:26.758325+0000", 2024-09-10T19:27:47.442 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:27:47.442 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:47.442 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:47.442 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-10T19:27:47.442 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:27:26.758324+0000", 2024-09-10T19:27:47.442 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.999999 2024-09-10T19:27:47.442 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:47.442 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:47.442 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-10T19:27:47.442 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:27:26.758332+0000", 2024-09-10T19:27:47.442 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.1710000000000002e-06 2024-09-10T19:27:47.442 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:47.442 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:47.443 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-10T19:27:47.443 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:27:26.758341+0000", 2024-09-10T19:27:47.443 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.5969999999999998e-06 2024-09-10T19:27:47.443 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:47.443 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:47.443 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-10T19:27:47.443 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:27:26.758347+0000", 2024-09-10T19:27:47.443 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.8690000000000001e-06 2024-09-10T19:27:47.443 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:47.443 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:47.443 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:27:47.443 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:27:26.761659+0000", 2024-09-10T19:27:47.444 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.003311469 2024-09-10T19:27:47.444 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:47.444 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:47.444 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-10T19:27:47.444 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:27:26.761729+0000", 2024-09-10T19:27:47.444 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.0782000000000001e-05 2024-09-10T19:27:47.444 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:47.444 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:47.444 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-09-10T19:27:47.444 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:27:26.775362+0000", 2024-09-10T19:27:47.444 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.013633098999999999 2024-09-10T19:27:47.444 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:47.444 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:47.444 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-10T19:27:47.445 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:27:26.775383+0000", 2024-09-10T19:27:47.445 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.0647000000000001e-05 2024-09-10T19:27:47.445 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:27:47.445 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-10T19:27:47.445 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:27:47.445 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:27:47.445 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-10T19:27:47.445 INFO:teuthology.orchestra.run.smithi191.stdout:} 2024-09-10T19:27:47.445 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:27:47.469 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:27:47.471 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:27:47.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:47.472 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:27:47.505 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:27:47.574 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:27:47.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:47.596 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:27:47.596 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:27:47.689 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:27:47.694 INFO:journalctl@ceph.mgr.x.smithi191.stdout:Sep 10 19:27:47 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-x[34967]: ::ffff:172.21.15.191 - - [10/Sep/2024:19:27:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T19:27:47.697 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:27:47.725 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:27:47.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:47.726 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:27:47.827 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:27:47.849 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:27:47.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:47.850 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:27:47.951 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:27:47.952 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:27:47 smithi183 ceph-mon[29736]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T19:27:47.952 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:27:47 smithi183 ceph-mon[29736]: osdmap e372: 8 total, 8 up, 6 in 2024-09-10T19:27:47.952 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:27:47 smithi183 ceph-mon[41562]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T19:27:47.952 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:27:47 smithi183 ceph-mon[41562]: osdmap e372: 8 total, 8 up, 6 in 2024-09-10T19:27:47.965 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:27:47 smithi191 ceph-mon[32884]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T19:27:47.965 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:27:47 smithi191 ceph-mon[32884]: osdmap e372: 8 total, 8 up, 6 in 2024-09-10T19:27:47.973 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:27:47.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:47.974 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:27:48.074 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:27:48.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:48.098 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:27:48.098 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:27:48.115 DEBUG:teuthology.orchestra.run.smithi191:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph daemon osd.7 dump_ops_in_flight 2024-09-10T19:27:48.199 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:27:48.222 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:27:48.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:48.223 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:27:48.324 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:27:48.346 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:27:48.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:48.347 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:27:48.398 INFO:teuthology.orchestra.run.smithi191.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.b/config 2024-09-10T19:27:48.448 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:27:48.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:48.470 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:27:48.470 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:27:48.571 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:27:48.592 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:27:48.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:48.593 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:27:48.694 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:27:48.715 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:27:48.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:48.716 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:27:48.817 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:27:48.821 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:27:48 smithi191 ceph-mon[32884]: osdmap e373: 8 total, 8 up, 6 in 2024-09-10T19:27:48.821 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:27:48 smithi191 ceph-mon[32884]: pgmap v815: 158 pgs: 13 unknown, 145 active+clean; 635 KiB data, 4.8 GiB used, 532 GiB / 536 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-09-10T19:27:48.839 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:27:48.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:48.840 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:27:48.940 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:27:48.962 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:27:48.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:48.963 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:27:49.064 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:27:49.086 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:27:49.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:49.087 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:27:49.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:27:48 smithi183 ceph-mon[29736]: osdmap e373: 8 total, 8 up, 6 in 2024-09-10T19:27:49.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:27:48 smithi183 ceph-mon[29736]: pgmap v815: 158 pgs: 13 unknown, 145 active+clean; 635 KiB data, 4.8 GiB used, 532 GiB / 536 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-09-10T19:27:49.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:27:48 smithi183 ceph-mon[41562]: osdmap e373: 8 total, 8 up, 6 in 2024-09-10T19:27:49.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:27:48 smithi183 ceph-mon[41562]: pgmap v815: 158 pgs: 13 unknown, 145 active+clean; 635 KiB data, 4.8 GiB used, 532 GiB / 536 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-09-10T19:27:49.188 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:27:49.209 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:27:49.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:49.211 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:27:49.311 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:27:49.333 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:27:49.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:49.334 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:27:49.434 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:27:49.456 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:27:49.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:49.457 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:27:49.459 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-10T19:27:49.558 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:27:49.579 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:27:49.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:49.580 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:27:49.681 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:27:49.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:49.703 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:27:49.703 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:27:49.804 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:27:49.826 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:27:49.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:49.827 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:27:49.928 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:27:49.949 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:27:49.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:49.951 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:27:50.051 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:27:50.073 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:27:50.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:50.074 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:27:50.110 INFO:journalctl@ceph.iscsi.iscsi.a.smithi191.stdout:Sep 10 19:27:49 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-iscsi-iscsi-a[61816]: debug there is no tcmu-runner data available 2024-09-10T19:27:50.114 INFO:teuthology.orchestra.run.smithi191.stdout:{ 2024-09-10T19:27:50.114 INFO:teuthology.orchestra.run.smithi191.stdout: "ops": [], 2024-09-10T19:27:50.114 INFO:teuthology.orchestra.run.smithi191.stdout: "num_ops": 0 2024-09-10T19:27:50.114 INFO:teuthology.orchestra.run.smithi191.stdout:} 2024-09-10T19:27:50.175 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:27:50.197 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:27:50.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:50.198 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:27:50.299 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:27:50.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:50.322 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:27:50.322 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:27:50.423 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:27:50.445 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:27:50.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:50.446 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:27:50.547 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:27:50.569 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:27:50.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:50.570 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:27:50.671 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:27:50.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:50.695 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:27:50.695 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:27:50.786 DEBUG:teuthology.orchestra.run.smithi191:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph daemon osd.7 dump_blocked_ops 2024-09-10T19:27:50.796 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:27:50.817 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:27:50.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:50.818 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:27:50.919 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:27:50.940 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:27:50.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:50.941 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:27:51.037 INFO:teuthology.orchestra.run.smithi191.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.b/config 2024-09-10T19:27:51.042 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:27:51.067 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:27:51.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:51.068 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:27:51.158 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:27:50 smithi191 ceph-mon[32884]: from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:27:51.158 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:27:50 smithi191 ceph-mon[32884]: pgmap v816: 158 pgs: 13 unknown, 145 active+clean; 635 KiB data, 4.8 GiB used, 532 GiB / 536 GiB avail; 853 B/s rd, 0 op/s 2024-09-10T19:27:51.169 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:27:51.190 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:27:51.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:51.191 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:27:51.231 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:27:50 smithi183 ceph-mon[29736]: from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:27:51.231 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:27:50 smithi183 ceph-mon[29736]: pgmap v816: 158 pgs: 13 unknown, 145 active+clean; 635 KiB data, 4.8 GiB used, 532 GiB / 536 GiB avail; 853 B/s rd, 0 op/s 2024-09-10T19:27:51.231 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:27:50 smithi183 ceph-mon[41562]: from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:27:51.232 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:27:50 smithi183 ceph-mon[41562]: pgmap v816: 158 pgs: 13 unknown, 145 active+clean; 635 KiB data, 4.8 GiB used, 532 GiB / 536 GiB avail; 853 B/s rd, 0 op/s 2024-09-10T19:27:51.292 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:27:51.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:51.316 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:27:51.316 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:27:51.417 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:27:51.439 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:27:51.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:51.440 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:27:51.541 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:27:51.562 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:27:51.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:51.563 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:27:51.565 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Sep 10 19:27:51 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-y[29989]: ::ffff:172.21.15.191 - - [10/Sep/2024:19:27:51] "GET /metrics HTTP/1.1" 200 222274 "" "Prometheus/2.43.0" 2024-09-10T19:27:51.664 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:27:51.685 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:27:51.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:51.686 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:27:51.737 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:27:51.787 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:27:51.809 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:27:51.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:51.810 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:27:51.844 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:27:51 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mon-a[29713]: 2024-09-10T19:27:51.602+0000 7f9f28e29640 -1 mon.a@0(leader).osd e373 definitely_dead 0 2024-09-10T19:27:51.910 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:27:51.931 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:27:51.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:51.932 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:27:51.986 INFO:teuthology.orchestra.run.smithi183.stderr:enabled application 'rados' on pool 'unique_pool_4' 2024-09-10T19:27:51.986 INFO:teuthology.orchestra.run.smithi183.stderr:marked in osd.6. 2024-09-10T19:27:52.033 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:27:52.055 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:27:52.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:52.056 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:27:52.127 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:27:51 smithi183 ceph-mon[29736]: from='client.? 172.21.15.183:0/3840202608' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["6"]}]: dispatch 2024-09-10T19:27:52.128 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:27:51 smithi183 ceph-mon[29736]: from='client.? 172.21.15.183:0/3523179754' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_4", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:27:52.128 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:27:51 smithi183 ceph-mon[29736]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_4", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:27:52.128 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:27:51 smithi183 ceph-mon[41562]: from='client.? 172.21.15.183:0/3840202608' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["6"]}]: dispatch 2024-09-10T19:27:52.128 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:27:51 smithi183 ceph-mon[41562]: from='client.? 172.21.15.183:0/3523179754' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_4", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:27:52.128 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:27:51 smithi183 ceph-mon[41562]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_4", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:27:52.157 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:27:52.178 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:27:52.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:52.179 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:27:52.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:27:51 smithi191 ceph-mon[32884]: from='client.? 172.21.15.183:0/3840202608' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["6"]}]: dispatch 2024-09-10T19:27:52.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:27:51 smithi191 ceph-mon[32884]: from='client.? 172.21.15.183:0/3523179754' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_4", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:27:52.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:27:51 smithi191 ceph-mon[32884]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_4", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-09-10T19:27:52.280 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:27:52.302 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:27:52.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:52.303 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:27:52.404 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:27:52.425 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:27:52.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:52.426 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:27:52.527 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:27:52.550 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:27:52.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:52.551 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:27:52.651 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:27:52.673 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:27:52.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:52.674 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:27:52.775 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:27:52.778 INFO:teuthology.orchestra.run.smithi191.stdout:{ 2024-09-10T19:27:52.779 INFO:teuthology.orchestra.run.smithi191.stdout: "ops": [], 2024-09-10T19:27:52.779 INFO:teuthology.orchestra.run.smithi191.stdout: "complaint_time": 30, 2024-09-10T19:27:52.779 INFO:teuthology.orchestra.run.smithi191.stdout: "num_blocked_ops": 0 2024-09-10T19:27:52.779 INFO:teuthology.orchestra.run.smithi191.stdout:} 2024-09-10T19:27:52.797 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:27:52.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:52.798 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:27:52.899 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:27:52.920 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:27:52.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:52.921 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:27:53.022 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:27:53.049 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:27:53.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:53.050 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:27:53.150 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:27:53.171 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:27:53.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:53.172 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:27:53.181 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:27:52 smithi183 ceph-mon[41562]: pgmap v817: 158 pgs: 158 active+clean; 635 KiB data, 4.8 GiB used, 532 GiB / 536 GiB avail; 821 B/s rd, 0 op/s; 0 B/s, 0 objects/s recovering 2024-09-10T19:27:53.181 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:27:52 smithi183 ceph-mon[41562]: from='client.? 172.21.15.183:0/3840202608' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["6"]}]': finished 2024-09-10T19:27:53.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:27:52 smithi183 ceph-mon[41562]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_4", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-09-10T19:27:53.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:27:52 smithi183 ceph-mon[41562]: osdmap e374: 8 total, 8 up, 7 in 2024-09-10T19:27:53.182 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:27:52 smithi183 ceph-mon[29736]: pgmap v817: 158 pgs: 158 active+clean; 635 KiB data, 4.8 GiB used, 532 GiB / 536 GiB avail; 821 B/s rd, 0 op/s; 0 B/s, 0 objects/s recovering 2024-09-10T19:27:53.182 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:27:52 smithi183 ceph-mon[29736]: from='client.? 172.21.15.183:0/3840202608' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["6"]}]': finished 2024-09-10T19:27:53.182 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:27:52 smithi183 ceph-mon[29736]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_4", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-09-10T19:27:53.182 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:27:52 smithi183 ceph-mon[29736]: osdmap e374: 8 total, 8 up, 7 in 2024-09-10T19:27:53.273 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:27:53.294 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:27:53.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:53.295 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:27:53.396 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:27:53.417 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:27:53.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:53.418 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:27:53.441 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:27:52 smithi191 ceph-mon[32884]: pgmap v817: 158 pgs: 158 active+clean; 635 KiB data, 4.8 GiB used, 532 GiB / 536 GiB avail; 821 B/s rd, 0 op/s; 0 B/s, 0 objects/s recovering 2024-09-10T19:27:53.441 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:27:52 smithi191 ceph-mon[32884]: from='client.? 172.21.15.183:0/3840202608' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["6"]}]': finished 2024-09-10T19:27:53.441 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:27:52 smithi191 ceph-mon[32884]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_4", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-09-10T19:27:53.441 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:27:52 smithi191 ceph-mon[32884]: osdmap e374: 8 total, 8 up, 7 in 2024-09-10T19:27:53.465 DEBUG:teuthology.orchestra.run.smithi191:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph daemon osd.7 dump_historic_ops 2024-09-10T19:27:53.519 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:27:53.542 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:27:53.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:53.543 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:27:53.644 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:27:53.665 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:27:53.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:53.666 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:27:53.747 INFO:teuthology.orchestra.run.smithi191.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.b/config 2024-09-10T19:27:53.767 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:27:53.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:53.789 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:27:53.789 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:27:53.890 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:27:53.891 DEBUG:teuthology.orchestra.run.smithi183:> /bin/sh -c 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --no-log-to-stderr --name client.0 -b 65536 --object-size 65536 -p unique_pool_4 bench 90 write' 2024-09-10T19:27:53.893 INFO:tasks.radosbench:joining radosbench (timing out after 3000s) 2024-09-10T19:27:53.894 INFO:teuthology.orchestra.run:waiting for 3000 2024-09-10T19:27:53.912 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:27:53.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:53.913 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:27:54.013 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:27:54.037 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:27:54.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:54.038 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:27:54.057 INFO:tasks.radosbench.radosbench.0.smithi183.stdout:hints = 1 2024-09-10T19:27:54.057 INFO:tasks.radosbench.radosbench.0.smithi183.stdout:Maintaining 16 concurrent writes of 65536 bytes to objects of size 65536 for up to 90 seconds or 0 objects 2024-09-10T19:27:54.057 INFO:tasks.radosbench.radosbench.0.smithi183.stdout:Object prefix: benchmark_data_smithi183_203944 2024-09-10T19:27:54.057 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-10T19:27:54.057 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: 0 0 0 0 0 0 - 0 2024-09-10T19:27:54.139 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:27:54.174 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:27:54.175 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:27:53 smithi183 ceph-mon[29736]: osdmap e375: 8 total, 8 up, 7 in 2024-09-10T19:27:54.175 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:27:53 smithi183 ceph-mon[41562]: osdmap e375: 8 total, 8 up, 7 in 2024-09-10T19:27:54.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:54.176 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:27:54.179 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:27:53 smithi191 ceph-mon[32884]: osdmap e375: 8 total, 8 up, 7 in 2024-09-10T19:27:54.277 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:27:54.310 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:27:54.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:54.311 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:27:54.412 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:27:54.450 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:27:54.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:54.451 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:27:54.552 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:27:54.586 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:27:54.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:54.587 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:27:54.688 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:27:54.722 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:27:54.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:54.723 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:27:54.824 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:27:54.865 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:27:54.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:54.866 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:27:54.967 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:27:55.003 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:27:55.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:55.004 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:27:55.020 INFO:teuthology.orchestra.run.smithi183.stderr:noscrub is unset 2024-09-10T19:27:55.045 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:27:54 smithi183 ceph-mon[29736]: pgmap v820: 158 pgs: 6 peering, 152 active+clean; 635 KiB data, 4.8 GiB used, 532 GiB / 536 GiB avail; 164 B/s rd, 0 op/s; 0 B/s, 0 objects/s recovering 2024-09-10T19:27:55.045 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:27:54 smithi183 ceph-mon[29736]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-10T19:27:55.045 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:27:54 smithi183 ceph-mon[29736]: from='client.? 172.21.15.183:0/3520079644' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-10T19:27:55.045 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:27:55 smithi183 ceph-mon[41562]: pgmap v820: 158 pgs: 6 peering, 152 active+clean; 635 KiB data, 4.8 GiB used, 532 GiB / 536 GiB avail; 164 B/s rd, 0 op/s; 0 B/s, 0 objects/s recovering 2024-09-10T19:27:55.045 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:27:55 smithi183 ceph-mon[41562]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-10T19:27:55.045 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:27:55 smithi183 ceph-mon[41562]: from='client.? 172.21.15.183:0/3520079644' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-10T19:27:55.057 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: 1 16 3839 3823 238.927 238.938 0.00551768 0.00417267 2024-09-10T19:27:55.105 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:27:55.152 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:27:55.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:55.153 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:27:55.254 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:27:55.293 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:27:55.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:55.294 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:27:55.364 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:27:55 smithi191 ceph-mon[32884]: pgmap v820: 158 pgs: 6 peering, 152 active+clean; 635 KiB data, 4.8 GiB used, 532 GiB / 536 GiB avail; 164 B/s rd, 0 op/s; 0 B/s, 0 objects/s recovering 2024-09-10T19:27:55.364 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:27:55 smithi191 ceph-mon[32884]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-10T19:27:55.364 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:27:55 smithi191 ceph-mon[32884]: from='client.? 172.21.15.183:0/3520079644' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-10T19:27:55.395 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:27:55.429 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:27:55.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:55.430 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:27:55.531 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:27:55.567 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:27:55.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:55.568 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:27:55.669 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:27:55.678 INFO:teuthology.orchestra.run.smithi191.stdout:{ 2024-09-10T19:27:55.678 INFO:teuthology.orchestra.run.smithi191.stdout: "size": 20, 2024-09-10T19:27:55.678 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 600, 2024-09-10T19:27:55.678 INFO:teuthology.orchestra.run.smithi191.stdout: "ops": [ 2024-09-10T19:27:55.678 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:55.678 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDPGPush(8.13 192/180 [PushOp(8:cf919698:::benchmark_data_smithi183_114924_object26566:head, version: 159'1771, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(8:cf919698:::benchmark_data_smithi183_114924_object26566:head@159'1771, size: 65536, copy_subset: [0~65536], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:65536, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-10T19:27:55.679 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-10T19:17:56.866594+0000", 2024-09-10T19:27:55.679 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 598.80440789700003, 2024-09-10T19:27:55.679 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.49036462, 2024-09-10T19:27:55.679 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-10T19:27:55.679 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "started", 2024-09-10T19:27:55.679 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-10T19:27:55.679 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:55.679 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-10T19:27:55.679 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:56.866594+0000", 2024-09-10T19:27:55.679 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:27:55.679 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:55.679 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:55.679 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-10T19:27:55.680 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:56.866594+0000", 2024-09-10T19:27:55.680 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:27:55.680 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:55.680 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:55.680 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-10T19:27:55.680 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:56.866593+0000", 2024-09-10T19:27:55.680 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999995 2024-09-10T19:27:55.680 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:55.680 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:55.680 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-10T19:27:55.680 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:56.866670+0000", 2024-09-10T19:27:55.680 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.6539999999999996e-05 2024-09-10T19:27:55.680 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:55.680 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:55.681 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-10T19:27:55.681 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:56.866685+0000", 2024-09-10T19:27:55.681 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.5319999999999999e-05 2024-09-10T19:27:55.681 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:55.681 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:55.681 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-10T19:27:55.681 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:56.866693+0000", 2024-09-10T19:27:55.681 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.4980000000000006e-06 2024-09-10T19:27:55.681 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:55.681 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:55.681 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:27:55.681 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:58.356122+0000", 2024-09-10T19:27:55.681 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.4894283750000001 2024-09-10T19:27:55.682 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:55.682 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:55.682 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-10T19:27:55.682 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:58.356126+0000", 2024-09-10T19:27:55.682 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.9260000000000002e-06 2024-09-10T19:27:55.682 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:55.682 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:55.682 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-10T19:27:55.682 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:58.356958+0000", 2024-09-10T19:27:55.682 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00083252900000000004 2024-09-10T19:27:55.682 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:27:55.682 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-10T19:27:55.682 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:27:55.683 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:55.683 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:55.683 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "MOSDPGPush(8.13 192/180 [PushOp(8:c8afa908:::benchmark_data_smithi183_114924_object26967:head, version: 159'1795, data_included: [0~65536], data_size: 65536, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(8:c8afa908:::benchmark_data_smithi183_114924_object26967:head@159'1795, size: 65536, copy_subset: [0~65536], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:65536, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-10T19:27:55.683 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-10T19:17:59.974190+0000", 2024-09-10T19:27:55.683 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 595.696811508, 2024-09-10T19:27:55.683 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.97933013000000002, 2024-09-10T19:27:55.683 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-10T19:27:55.683 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "started", 2024-09-10T19:27:55.683 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-10T19:27:55.683 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:55.683 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-10T19:27:55.683 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:59.974190+0000", 2024-09-10T19:27:55.683 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:27:55.683 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:55.684 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:55.684 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-10T19:27:55.684 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:59.974190+0000", 2024-09-10T19:27:55.684 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:27:55.684 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:55.684 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:55.684 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-10T19:27:55.684 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:59.974190+0000", 2024-09-10T19:27:55.684 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999995 2024-09-10T19:27:55.684 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:55.684 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:55.684 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-10T19:27:55.684 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:59.974226+0000", 2024-09-10T19:27:55.685 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.6442000000000001e-05 2024-09-10T19:27:55.685 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:55.685 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:55.685 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-10T19:27:55.685 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:59.974243+0000", 2024-09-10T19:27:55.685 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.6586e-05 2024-09-10T19:27:55.685 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:55.685 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:55.685 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-10T19:27:55.685 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:17:59.974252+0000", 2024-09-10T19:27:55.685 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.4550000000000001e-06 2024-09-10T19:27:55.685 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:55.685 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:55.686 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:27:55.686 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:18:00.951928+0000", 2024-09-10T19:27:55.686 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.97767566100000003 2024-09-10T19:27:55.686 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:55.686 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:55.686 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-10T19:27:55.686 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:18:00.951935+0000", 2024-09-10T19:27:55.686 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.7139999999999996e-06 2024-09-10T19:27:55.686 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:55.686 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:55.686 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-10T19:27:55.686 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:18:00.953520+0000", 2024-09-10T19:27:55.686 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0015855299999999999 2024-09-10T19:27:55.687 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:27:55.687 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-10T19:27:55.687 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:27:55.687 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:55.687 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:55.687 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.15051.0:172172 8.17 8:ec5a1e4c:::benchmark_data_smithi183_114924_object51889:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e205)", 2024-09-10T19:27:55.687 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-10T19:18:43.179473+0000", 2024-09-10T19:27:55.687 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 552.49152872499997, 2024-09-10T19:27:55.687 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.99473523600000002, 2024-09-10T19:27:55.687 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-10T19:27:55.687 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T19:27:55.687 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-10T19:27:55.687 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.15051", 2024-09-10T19:27:55.687 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.183:0/3632028672", 2024-09-10T19:27:55.688 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 172172 2024-09-10T19:27:55.688 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:55.688 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-10T19:27:55.688 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:55.688 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-10T19:27:55.688 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:18:43.179473+0000", 2024-09-10T19:27:55.688 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:27:55.688 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:55.688 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:55.688 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-10T19:27:55.688 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:18:43.179473+0000", 2024-09-10T19:27:55.688 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:27:55.688 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:55.689 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:55.689 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-10T19:27:55.689 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:18:43.179471+0000", 2024-09-10T19:27:55.689 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999986 2024-09-10T19:27:55.689 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:55.689 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:55.689 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-10T19:27:55.689 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:18:43.179478+0000", 2024-09-10T19:27:55.689 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.2910000000000004e-06 2024-09-10T19:27:55.689 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:55.689 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:55.689 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-10T19:27:55.689 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:18:43.179489+0000", 2024-09-10T19:27:55.689 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.1528e-05 2024-09-10T19:27:55.690 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:55.690 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:55.690 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-10T19:27:55.690 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:18:43.179495+0000", 2024-09-10T19:27:55.690 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.0480000000000004e-06 2024-09-10T19:27:55.690 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:55.690 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:55.690 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:27:55.690 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:18:43.179674+0000", 2024-09-10T19:27:55.690 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00017865900000000001 2024-09-10T19:27:55.690 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:55.690 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:55.690 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for peered", 2024-09-10T19:27:55.690 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:18:43.179687+0000", 2024-09-10T19:27:55.691 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.2853999999999999e-05 2024-09-10T19:27:55.691 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:55.691 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:55.691 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:27:55.691 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:18:44.165270+0000", 2024-09-10T19:27:55.691 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.98558308299999997 2024-09-10T19:27:55.691 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:55.691 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:55.691 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-10T19:27:55.691 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:18:44.165366+0000", 2024-09-10T19:27:55.691 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.6162000000000005e-05 2024-09-10T19:27:55.691 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:55.691 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:55.692 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0,6", 2024-09-10T19:27:55.692 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:18:44.165590+0000", 2024-09-10T19:27:55.692 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00022405299999999999 2024-09-10T19:27:55.692 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:55.692 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:55.692 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-09-10T19:27:55.692 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:18:44.172416+0000", 2024-09-10T19:27:55.692 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0068258320000000004 2024-09-10T19:27:55.692 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:55.692 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:55.692 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:27:55.692 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:18:44.174090+0000", 2024-09-10T19:27:55.692 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0016737659999999999 2024-09-10T19:27:55.692 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:55.693 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:55.693 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:27:55.693 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:18:44.174127+0000", 2024-09-10T19:27:55.693 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.7360999999999997e-05 2024-09-10T19:27:55.693 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:55.693 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:55.693 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-09-10T19:27:55.693 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:18:44.174171+0000", 2024-09-10T19:27:55.693 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.4270999999999997e-05 2024-09-10T19:27:55.693 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:55.693 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:55.693 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-10T19:27:55.693 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:18:44.174208+0000", 2024-09-10T19:27:55.694 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.6720000000000001e-05 2024-09-10T19:27:55.694 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:27:55.694 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-10T19:27:55.694 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:27:55.694 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:55.694 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:55.694 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.15051.0:172461 8.15 8:aff08a5b:::benchmark_data_smithi183_114924_object52178:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e206)", 2024-09-10T19:27:55.694 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-10T19:18:44.222397+0000", 2024-09-10T19:27:55.694 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 551.44860440900004, 2024-09-10T19:27:55.694 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.95883427200000004, 2024-09-10T19:27:55.694 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-10T19:27:55.694 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T19:27:55.694 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-10T19:27:55.695 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.15051", 2024-09-10T19:27:55.695 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.183:0/3632028672", 2024-09-10T19:27:55.695 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 172461 2024-09-10T19:27:55.695 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:55.695 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-10T19:27:55.695 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:55.695 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-10T19:27:55.695 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:18:44.222397+0000", 2024-09-10T19:27:55.695 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:27:55.695 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:55.695 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:55.695 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-10T19:27:55.695 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:18:44.222397+0000", 2024-09-10T19:27:55.695 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:27:55.696 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:55.696 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:55.696 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-10T19:27:55.696 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:18:44.222396+0000", 2024-09-10T19:27:55.696 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999986 2024-09-10T19:27:55.696 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:55.696 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:55.696 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-10T19:27:55.696 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:18:44.222403+0000", 2024-09-10T19:27:55.696 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.8820000000000003e-06 2024-09-10T19:27:55.696 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:55.696 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:55.696 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-10T19:27:55.696 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:18:44.222416+0000", 2024-09-10T19:27:55.697 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.3291e-05 2024-09-10T19:27:55.697 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:55.697 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:55.697 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-10T19:27:55.697 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:18:44.222422+0000", 2024-09-10T19:27:55.697 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.2580000000000001e-06 2024-09-10T19:27:55.697 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:55.697 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:55.697 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:27:55.697 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:18:44.222507+0000", 2024-09-10T19:27:55.697 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.5118999999999996e-05 2024-09-10T19:27:55.697 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:55.698 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:55.698 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for peered", 2024-09-10T19:27:55.698 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:18:44.222515+0000", 2024-09-10T19:27:55.698 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.6380000000000004e-06 2024-09-10T19:27:55.698 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:55.698 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:55.698 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:27:55.698 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:18:45.163741+0000", 2024-09-10T19:27:55.698 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.94122644200000005 2024-09-10T19:27:55.698 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:55.698 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:55.698 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-10T19:27:55.698 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:18:45.163835+0000", 2024-09-10T19:27:55.698 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.3490000000000001e-05 2024-09-10T19:27:55.698 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:55.699 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:55.699 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0,6", 2024-09-10T19:27:55.699 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:18:45.164047+0000", 2024-09-10T19:27:55.699 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00021215599999999999 2024-09-10T19:27:55.699 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:55.699 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:55.699 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-09-10T19:27:55.699 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:18:45.168946+0000", 2024-09-10T19:27:55.699 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0048991410000000001 2024-09-10T19:27:55.699 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:55.699 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:55.699 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:27:55.699 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:18:45.181058+0000", 2024-09-10T19:27:55.699 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.012112303 2024-09-10T19:27:55.700 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:55.700 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:55.700 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:27:55.700 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:18:45.181172+0000", 2024-09-10T19:27:55.700 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00011330100000000001 2024-09-10T19:27:55.700 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:55.700 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:55.700 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-09-10T19:27:55.700 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:18:45.181202+0000", 2024-09-10T19:27:55.700 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.0518999999999997e-05 2024-09-10T19:27:55.700 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:55.700 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:55.700 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-10T19:27:55.701 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:18:45.181231+0000", 2024-09-10T19:27:55.701 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.9150999999999999e-05 2024-09-10T19:27:55.701 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:27:55.701 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-10T19:27:55.701 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:27:55.701 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:55.701 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:55.701 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.15360.0:169775 9.9 9:9da34f67:::benchmark_data_smithi183_142499_object23286:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e263)", 2024-09-10T19:27:55.701 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-10T19:21:46.960468+0000", 2024-09-10T19:27:55.701 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 368.71053353999997, 2024-09-10T19:27:55.701 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.0671650109999999, 2024-09-10T19:27:55.701 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-10T19:27:55.701 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T19:27:55.702 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-10T19:27:55.702 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.15360", 2024-09-10T19:27:55.702 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.183:0/2364804131", 2024-09-10T19:27:55.702 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 169775 2024-09-10T19:27:55.702 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:55.702 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-10T19:27:55.702 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:55.702 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-10T19:27:55.702 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:21:46.960468+0000", 2024-09-10T19:27:55.702 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:27:55.702 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:55.702 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:55.702 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-10T19:27:55.703 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:21:46.960468+0000", 2024-09-10T19:27:55.703 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:27:55.703 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:55.703 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:55.703 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-10T19:27:55.703 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:21:46.960466+0000", 2024-09-10T19:27:55.703 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999981 2024-09-10T19:27:55.703 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:55.703 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:55.703 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-10T19:27:55.703 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:21:46.960473+0000", 2024-09-10T19:27:55.703 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.1659999999999997e-06 2024-09-10T19:27:55.703 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:55.703 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:55.704 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-10T19:27:55.704 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:21:46.960486+0000", 2024-09-10T19:27:55.704 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.2972e-05 2024-09-10T19:27:55.704 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:55.704 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:55.704 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-10T19:27:55.704 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:21:46.960493+0000", 2024-09-10T19:27:55.704 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.1910000000000003e-06 2024-09-10T19:27:55.704 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:55.704 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:55.704 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:27:55.704 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:21:46.991408+0000", 2024-09-10T19:27:55.704 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.030915074000000001 2024-09-10T19:27:55.704 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:55.705 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:55.705 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for peered", 2024-09-10T19:27:55.705 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:21:46.991416+0000", 2024-09-10T19:27:55.705 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.5140000000000004e-06 2024-09-10T19:27:55.705 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:55.705 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:55.705 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:27:55.705 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:21:48.013668+0000", 2024-09-10T19:27:55.705 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.022252231 2024-09-10T19:27:55.705 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:55.705 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:55.705 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-10T19:27:55.705 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:21:48.015474+0000", 2024-09-10T19:27:55.706 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0018054760000000001 2024-09-10T19:27:55.706 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:55.706 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:55.706 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 5", 2024-09-10T19:27:55.706 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:21:48.015713+0000", 2024-09-10T19:27:55.706 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00023921599999999999 2024-09-10T19:27:55.706 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:55.706 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:55.706 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-09-10T19:27:55.706 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:21:48.024289+0000", 2024-09-10T19:27:55.706 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0085763880000000008 2024-09-10T19:27:55.706 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:55.706 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:55.707 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:27:55.707 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:21:48.027515+0000", 2024-09-10T19:27:55.707 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0032260079999999998 2024-09-10T19:27:55.707 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:55.707 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:55.707 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-09-10T19:27:55.707 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:21:48.027559+0000", 2024-09-10T19:27:55.707 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.3996999999999999e-05 2024-09-10T19:27:55.707 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:55.707 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:55.707 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-10T19:27:55.707 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:21:48.027633+0000", 2024-09-10T19:27:55.707 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.3619000000000001e-05 2024-09-10T19:27:55.707 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:27:55.708 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-10T19:27:55.708 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:27:55.708 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:55.708 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:55.708 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.15360.0:169783 9.9 9:93db4d72:::benchmark_data_smithi183_142499_object23294:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e263)", 2024-09-10T19:27:55.708 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-10T19:21:46.991598+0000", 2024-09-10T19:27:55.708 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 368.67940368699999, 2024-09-10T19:27:55.708 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.0422353129999999, 2024-09-10T19:27:55.708 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-10T19:27:55.708 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T19:27:55.708 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-10T19:27:55.708 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.15360", 2024-09-10T19:27:55.708 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.183:0/2364804131", 2024-09-10T19:27:55.708 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 169783 2024-09-10T19:27:55.709 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:55.709 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-10T19:27:55.709 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:55.709 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-10T19:27:55.709 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:21:46.991598+0000", 2024-09-10T19:27:55.709 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:27:55.709 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:55.709 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:55.709 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-10T19:27:55.709 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:21:46.991598+0000", 2024-09-10T19:27:55.709 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:27:55.709 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:55.709 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:55.710 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-10T19:27:55.710 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:21:46.991597+0000", 2024-09-10T19:27:55.710 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999986 2024-09-10T19:27:55.710 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:55.710 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:55.710 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-10T19:27:55.710 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:21:46.991602+0000", 2024-09-10T19:27:55.710 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.5380000000000002e-06 2024-09-10T19:27:55.710 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:55.710 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:55.710 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-10T19:27:55.710 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:21:46.991612+0000", 2024-09-10T19:27:55.711 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.5289999999999997e-06 2024-09-10T19:27:55.711 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:55.711 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:55.711 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-10T19:27:55.711 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:21:46.991617+0000", 2024-09-10T19:27:55.711 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.5110000000000003e-06 2024-09-10T19:27:55.711 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:55.711 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:55.711 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:27:55.711 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:21:46.991868+0000", 2024-09-10T19:27:55.711 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000251339 2024-09-10T19:27:55.711 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:55.711 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:55.712 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for peered", 2024-09-10T19:27:55.712 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:21:46.991875+0000", 2024-09-10T19:27:55.712 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.2600000000000002e-06 2024-09-10T19:27:55.712 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:55.712 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:55.712 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:27:55.712 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:21:48.020111+0000", 2024-09-10T19:27:55.712 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.0282362519999999 2024-09-10T19:27:55.712 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:55.712 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:55.712 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-10T19:27:55.712 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:21:48.020206+0000", 2024-09-10T19:27:55.712 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.5179000000000005e-05 2024-09-10T19:27:55.713 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:55.713 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:55.713 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 5", 2024-09-10T19:27:55.713 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:21:48.020309+0000", 2024-09-10T19:27:55.713 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00010310999999999999 2024-09-10T19:27:55.713 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:55.713 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:55.713 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-09-10T19:27:55.713 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:21:48.027641+0000", 2024-09-10T19:27:55.713 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0073316939999999997 2024-09-10T19:27:55.713 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:55.713 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:55.713 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:27:55.714 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:21:48.033698+0000", 2024-09-10T19:27:55.714 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0060571660000000001 2024-09-10T19:27:55.714 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:55.714 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:55.714 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-09-10T19:27:55.714 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:21:48.033745+0000", 2024-09-10T19:27:55.714 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.6461000000000001e-05 2024-09-10T19:27:55.714 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:55.714 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:55.714 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-10T19:27:55.714 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:21:48.033833+0000", 2024-09-10T19:27:55.714 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.8530999999999997e-05 2024-09-10T19:27:55.714 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:27:55.714 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-10T19:27:55.715 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:27:55.715 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:55.715 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:55.715 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.15360.0:169785 9.9 9:9cc1dce6:::benchmark_data_smithi183_142499_object23296:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e263)", 2024-09-10T19:27:55.715 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-10T19:21:46.993227+0000", 2024-09-10T19:27:55.715 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 368.67777478099998, 2024-09-10T19:27:55.715 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.0407537469999999, 2024-09-10T19:27:55.715 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-10T19:27:55.715 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T19:27:55.715 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-10T19:27:55.715 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.15360", 2024-09-10T19:27:55.715 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.183:0/2364804131", 2024-09-10T19:27:55.715 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 169785 2024-09-10T19:27:55.716 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:55.716 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-10T19:27:55.716 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:55.716 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-10T19:27:55.716 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:21:46.993227+0000", 2024-09-10T19:27:55.716 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:27:55.716 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:55.716 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:55.716 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-10T19:27:55.716 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:21:46.993227+0000", 2024-09-10T19:27:55.716 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:27:55.716 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:55.716 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:55.716 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-10T19:27:55.717 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:21:46.993226+0000", 2024-09-10T19:27:55.717 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.999999 2024-09-10T19:27:55.717 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:55.717 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:55.717 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-10T19:27:55.717 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:21:46.993231+0000", 2024-09-10T19:27:55.717 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.5840000000000002e-06 2024-09-10T19:27:55.717 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:55.717 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:55.717 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-10T19:27:55.717 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:21:46.993243+0000", 2024-09-10T19:27:55.717 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.1686e-05 2024-09-10T19:27:55.717 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:55.718 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:55.718 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-10T19:27:55.718 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:21:46.993249+0000", 2024-09-10T19:27:55.718 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.0780000000000001e-06 2024-09-10T19:27:55.718 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:55.718 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:55.718 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:27:55.718 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:21:46.993347+0000", 2024-09-10T19:27:55.718 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.8405000000000004e-05 2024-09-10T19:27:55.718 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:55.718 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:55.718 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for peered", 2024-09-10T19:27:55.718 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:21:46.993354+0000", 2024-09-10T19:27:55.718 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.7390000000000002e-06 2024-09-10T19:27:55.719 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:55.719 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:55.719 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:27:55.719 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:21:48.022236+0000", 2024-09-10T19:27:55.719 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.028881905 2024-09-10T19:27:55.720 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:55.720 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:55.720 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-10T19:27:55.720 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:21:48.022331+0000", 2024-09-10T19:27:55.720 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.4734000000000005e-05 2024-09-10T19:27:55.720 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:55.720 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:55.720 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 5", 2024-09-10T19:27:55.720 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:21:48.022436+0000", 2024-09-10T19:27:55.720 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00010508799999999999 2024-09-10T19:27:55.720 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:55.720 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:55.720 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-09-10T19:27:55.721 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:21:48.027646+0000", 2024-09-10T19:27:55.721 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.005209984 2024-09-10T19:27:55.721 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:55.721 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:55.721 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:27:55.721 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:21:48.033863+0000", 2024-09-10T19:27:55.721 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0062175030000000001 2024-09-10T19:27:55.721 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:55.721 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:55.721 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-09-10T19:27:55.721 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:21:48.033907+0000", 2024-09-10T19:27:55.721 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.3958000000000002e-05 2024-09-10T19:27:55.721 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:55.721 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:55.722 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-10T19:27:55.722 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:21:48.033980+0000", 2024-09-10T19:27:55.722 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.3159999999999997e-05 2024-09-10T19:27:55.722 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:27:55.722 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-10T19:27:55.722 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:27:55.722 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:55.722 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:55.722 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.25622.0:216511 10.c 10:39bc3006:::benchmark_data_smithi183_176537_object93303:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e355)", 2024-09-10T19:27:55.722 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-10T19:27:01.151163+0000", 2024-09-10T19:27:55.722 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 54.519838381, 2024-09-10T19:27:55.722 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.12147474, 2024-09-10T19:27:55.722 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-10T19:27:55.723 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T19:27:55.723 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-10T19:27:55.723 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.25622", 2024-09-10T19:27:55.723 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.183:0/3685189199", 2024-09-10T19:27:55.723 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 216511 2024-09-10T19:27:55.723 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:55.723 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-10T19:27:55.723 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:55.723 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-10T19:27:55.723 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:27:01.151163+0000", 2024-09-10T19:27:55.723 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:27:55.723 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:55.723 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:55.724 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-10T19:27:55.724 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:27:01.151163+0000", 2024-09-10T19:27:55.724 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:27:55.724 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:55.724 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:55.724 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-10T19:27:55.724 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:27:01.151162+0000", 2024-09-10T19:27:55.724 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.999999 2024-09-10T19:27:55.724 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:55.724 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:55.724 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-10T19:27:55.724 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:27:01.151169+0000", 2024-09-10T19:27:55.724 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.6390000000000003e-06 2024-09-10T19:27:55.725 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:55.725 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:55.725 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-10T19:27:55.725 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:27:01.250667+0000", 2024-09-10T19:27:55.725 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.099497944000000005 2024-09-10T19:27:55.725 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:55.725 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:55.725 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-10T19:27:55.725 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:27:01.250692+0000", 2024-09-10T19:27:55.725 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.5034000000000001e-05 2024-09-10T19:27:55.725 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:55.725 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:55.725 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:27:55.725 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:27:01.256038+0000", 2024-09-10T19:27:55.726 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0053465759999999996 2024-09-10T19:27:55.726 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:55.726 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:55.726 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-10T19:27:55.726 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:27:01.256145+0000", 2024-09-10T19:27:55.726 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00010721399999999999 2024-09-10T19:27:55.726 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:55.726 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:55.726 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0,4", 2024-09-10T19:27:55.726 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:27:01.256272+0000", 2024-09-10T19:27:55.726 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000126673 2024-09-10T19:27:55.726 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:55.726 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:55.726 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:27:55.727 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:27:01.269436+0000", 2024-09-10T19:27:55.727 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.013163846 2024-09-10T19:27:55.727 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:55.727 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:55.727 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:27:55.727 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:27:01.269615+0000", 2024-09-10T19:27:55.727 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00017960200000000001 2024-09-10T19:27:55.727 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:55.727 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:55.727 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-09-10T19:27:55.727 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:27:01.272523+0000", 2024-09-10T19:27:55.727 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.002907621 2024-09-10T19:27:55.727 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:55.727 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:55.728 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-09-10T19:27:55.728 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:27:01.272602+0000", 2024-09-10T19:27:55.728 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.8578999999999994e-05 2024-09-10T19:27:55.728 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:55.728 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:55.728 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-10T19:27:55.728 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:27:01.272638+0000", 2024-09-10T19:27:55.728 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.6164999999999997e-05 2024-09-10T19:27:55.728 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:27:55.728 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-10T19:27:55.728 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:27:55.728 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:55.728 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:55.729 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.25622.0:244644 10.c 10:30e0a74b:::benchmark_data_smithi183_176537_object121436:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e365)", 2024-09-10T19:27:55.729 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-10T19:27:27.020310+0000", 2024-09-10T19:27:55.729 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 28.650691834, 2024-09-10T19:27:55.729 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.114709229, 2024-09-10T19:27:55.729 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-10T19:27:55.729 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T19:27:55.729 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-10T19:27:55.729 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.25622", 2024-09-10T19:27:55.729 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.183:0/3685189199", 2024-09-10T19:27:55.729 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 244644 2024-09-10T19:27:55.729 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:55.729 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-10T19:27:55.729 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:55.729 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-10T19:27:55.730 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:27:27.020310+0000", 2024-09-10T19:27:55.730 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:27:55.730 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:55.730 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:55.730 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-10T19:27:55.730 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:27:27.020310+0000", 2024-09-10T19:27:55.730 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:27:55.730 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:55.730 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:55.730 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-10T19:27:55.730 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:27:27.020308+0000", 2024-09-10T19:27:55.730 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999981 2024-09-10T19:27:55.730 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:55.731 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:55.731 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-10T19:27:55.731 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:27:27.020313+0000", 2024-09-10T19:27:55.731 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.4589999999999997e-06 2024-09-10T19:27:55.731 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:55.731 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:55.731 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-10T19:27:55.731 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:27:27.020326+0000", 2024-09-10T19:27:55.731 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.2719e-05 2024-09-10T19:27:55.731 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:55.731 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:55.731 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-10T19:27:55.731 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:27:27.020331+0000", 2024-09-10T19:27:55.731 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.9400000000000001e-06 2024-09-10T19:27:55.732 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:55.732 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:55.732 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:27:55.732 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:27:27.020479+0000", 2024-09-10T19:27:55.732 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00014794999999999999 2024-09-10T19:27:55.732 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:55.732 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:55.732 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-10T19:27:55.732 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:27:27.020580+0000", 2024-09-10T19:27:55.732 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00010162699999999999 2024-09-10T19:27:55.732 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:55.732 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:55.732 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0,4", 2024-09-10T19:27:55.732 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:27:27.020689+0000", 2024-09-10T19:27:55.733 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000108774 2024-09-10T19:27:55.733 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:55.733 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:55.733 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-09-10T19:27:55.733 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:27:27.025499+0000", 2024-09-10T19:27:55.733 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0048099989999999997 2024-09-10T19:27:55.733 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:55.733 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:55.733 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:27:55.733 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:27:27.029598+0000", 2024-09-10T19:27:55.733 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.004098979 2024-09-10T19:27:55.733 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:55.733 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:55.734 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:27:55.734 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:27:27.134638+0000", 2024-09-10T19:27:55.734 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.105039705 2024-09-10T19:27:55.734 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:55.734 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:55.734 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-09-10T19:27:55.734 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:27:27.134761+0000", 2024-09-10T19:27:55.734 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00012371100000000001 2024-09-10T19:27:55.734 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:55.734 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:55.734 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-10T19:27:55.734 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:27:27.135019+0000", 2024-09-10T19:27:55.734 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00025737200000000002 2024-09-10T19:27:55.735 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:27:55.735 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-10T19:27:55.735 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:27:55.735 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:55.735 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:55.735 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.25622.0:244652 10.c 10:3cd232ca:::benchmark_data_smithi183_176537_object121444:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e365)", 2024-09-10T19:27:55.735 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-10T19:27:27.026942+0000", 2024-09-10T19:27:55.735 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 28.644059055, 2024-09-10T19:27:55.735 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.11243450300000001, 2024-09-10T19:27:55.735 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-10T19:27:55.735 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T19:27:55.735 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-10T19:27:55.735 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.25622", 2024-09-10T19:27:55.735 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.183:0/3685189199", 2024-09-10T19:27:55.736 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 244652 2024-09-10T19:27:55.736 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:55.736 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-10T19:27:55.736 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:55.736 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-10T19:27:55.736 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:27:27.026942+0000", 2024-09-10T19:27:55.736 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:27:55.736 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:55.736 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:55.736 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-10T19:27:55.736 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:27:27.026942+0000", 2024-09-10T19:27:55.736 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:27:55.736 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:55.736 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:55.737 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-10T19:27:55.737 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:27:27.026941+0000", 2024-09-10T19:27:55.737 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999986 2024-09-10T19:27:55.737 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:55.737 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:55.737 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-10T19:27:55.737 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:27:27.026947+0000", 2024-09-10T19:27:55.737 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.1720000000000004e-06 2024-09-10T19:27:55.737 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:55.737 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:55.737 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-10T19:27:55.737 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:27:27.026957+0000", 2024-09-10T19:27:55.738 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.0317e-05 2024-09-10T19:27:55.738 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:55.738 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:55.738 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-10T19:27:55.738 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:27:27.026962+0000", 2024-09-10T19:27:55.738 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.0579999999999997e-06 2024-09-10T19:27:55.738 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:55.738 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:55.738 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:27:55.738 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:27:27.027224+0000", 2024-09-10T19:27:55.738 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000261978 2024-09-10T19:27:55.738 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:55.738 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:55.738 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-10T19:27:55.739 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:27:27.027339+0000", 2024-09-10T19:27:55.739 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000114918 2024-09-10T19:27:55.739 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:55.739 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:55.739 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0,4", 2024-09-10T19:27:55.739 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:27:27.027445+0000", 2024-09-10T19:27:55.739 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000105845 2024-09-10T19:27:55.739 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:55.739 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:55.739 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:27:55.739 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:27:27.029697+0000", 2024-09-10T19:27:55.739 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0022516300000000001 2024-09-10T19:27:55.739 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:55.739 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:55.740 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-09-10T19:27:55.740 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:27:27.029706+0000", 2024-09-10T19:27:55.740 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.4369999999999996e-06 2024-09-10T19:27:55.740 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:55.740 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:55.740 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:27:55.740 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:27:27.139291+0000", 2024-09-10T19:27:55.740 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.109584581 2024-09-10T19:27:55.740 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:55.740 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:55.740 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-09-10T19:27:55.740 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:27:27.139339+0000", 2024-09-10T19:27:55.740 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.8449000000000003e-05 2024-09-10T19:27:55.740 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:55.741 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:55.741 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-10T19:27:55.741 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:27:27.139377+0000", 2024-09-10T19:27:55.741 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.7596999999999999e-05 2024-09-10T19:27:55.741 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:27:55.741 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-10T19:27:55.741 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:27:55.741 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:55.741 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:55.741 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.25622.0:244767 10.c 10:3240c45b:::benchmark_data_smithi183_176537_object121559:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e365)", 2024-09-10T19:27:55.741 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-10T19:27:27.206940+0000", 2024-09-10T19:27:55.741 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 28.464061939, 2024-09-10T19:27:55.741 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.020594564999999999, 2024-09-10T19:27:55.742 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-10T19:27:55.742 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T19:27:55.742 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-10T19:27:55.742 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.25622", 2024-09-10T19:27:55.742 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.183:0/3685189199", 2024-09-10T19:27:55.742 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 244767 2024-09-10T19:27:55.742 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:55.742 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-10T19:27:55.742 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:55.742 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-10T19:27:55.742 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:27:27.206940+0000", 2024-09-10T19:27:55.742 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:27:55.742 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:55.742 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:55.743 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-10T19:27:55.743 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:27:27.206940+0000", 2024-09-10T19:27:55.743 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:27:55.743 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:55.743 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:55.743 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-10T19:27:55.743 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:27:27.206938+0000", 2024-09-10T19:27:55.743 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999986 2024-09-10T19:27:55.743 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:55.743 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:55.743 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-10T19:27:55.743 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:27:27.206944+0000", 2024-09-10T19:27:55.743 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.5890000000000002e-06 2024-09-10T19:27:55.743 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:55.744 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:55.744 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-10T19:27:55.744 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:27:27.206957+0000", 2024-09-10T19:27:55.744 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.3234999999999999e-05 2024-09-10T19:27:55.744 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:55.744 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:55.744 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-10T19:27:55.744 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:27:27.206963+0000", 2024-09-10T19:27:55.744 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.8200000000000002e-06 2024-09-10T19:27:55.744 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:55.744 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:55.744 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:27:55.744 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:27:27.207027+0000", 2024-09-10T19:27:55.744 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.4876999999999996e-05 2024-09-10T19:27:55.745 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:55.745 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:55.745 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-10T19:27:55.745 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:27:27.207159+0000", 2024-09-10T19:27:55.745 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00013101899999999999 2024-09-10T19:27:55.745 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:55.745 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:55.745 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0,4", 2024-09-10T19:27:55.745 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:27:27.212974+0000", 2024-09-10T19:27:55.745 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.005815672 2024-09-10T19:27:55.745 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:55.745 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:55.745 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:27:55.745 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:27:27.218378+0000", 2024-09-10T19:27:55.746 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0054042150000000004 2024-09-10T19:27:55.746 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:55.746 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:55.746 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-09-10T19:27:55.746 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:27:27.218416+0000", 2024-09-10T19:27:55.746 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.7537000000000003e-05 2024-09-10T19:27:55.746 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:55.746 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:55.746 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:27:55.746 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:27:27.227393+0000", 2024-09-10T19:27:55.746 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0089772819999999996 2024-09-10T19:27:55.747 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:55.747 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:55.747 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-09-10T19:27:55.747 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:27:27.227492+0000", 2024-09-10T19:27:55.747 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.8719000000000001e-05 2024-09-10T19:27:55.747 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:55.747 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:55.747 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-10T19:27:55.747 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:27:27.227534+0000", 2024-09-10T19:27:55.747 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.214e-05 2024-09-10T19:27:55.747 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:27:55.747 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-10T19:27:55.747 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:27:55.748 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:55.748 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:55.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:55.749 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:27:55.749 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:27:55.750 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.25622.0:244773 10.c 10:360570fb:::benchmark_data_smithi183_176537_object121565:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e365)", 2024-09-10T19:27:55.750 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-10T19:27:27.212427+0000", 2024-09-10T19:27:55.750 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 28.458574109000001, 2024-09-10T19:27:55.750 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.022918494000000001, 2024-09-10T19:27:55.750 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-10T19:27:55.750 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T19:27:55.750 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-10T19:27:55.750 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.25622", 2024-09-10T19:27:55.750 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.183:0/3685189199", 2024-09-10T19:27:55.750 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 244773 2024-09-10T19:27:55.750 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:55.751 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-10T19:27:55.751 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:55.751 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-10T19:27:55.751 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:27:27.212427+0000", 2024-09-10T19:27:55.751 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:27:55.751 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:55.751 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:55.751 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-10T19:27:55.751 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:27:27.212427+0000", 2024-09-10T19:27:55.751 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:27:55.751 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:55.751 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:55.751 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-10T19:27:55.752 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:27:27.212426+0000", 2024-09-10T19:27:55.752 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999986 2024-09-10T19:27:55.752 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:55.752 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:55.752 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-10T19:27:55.752 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:27:27.212433+0000", 2024-09-10T19:27:55.752 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.3549999999999999e-06 2024-09-10T19:27:55.752 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:55.752 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:55.752 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-10T19:27:55.752 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:27:27.212455+0000", 2024-09-10T19:27:55.752 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.2087e-05 2024-09-10T19:27:55.752 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:55.753 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:55.753 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-10T19:27:55.753 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:27:27.212461+0000", 2024-09-10T19:27:55.753 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.032e-06 2024-09-10T19:27:55.753 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:55.753 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:55.753 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:27:55.753 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:27:27.218497+0000", 2024-09-10T19:27:55.753 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0060356810000000002 2024-09-10T19:27:55.753 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:55.753 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:55.753 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-10T19:27:55.753 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:27:27.218784+0000", 2024-09-10T19:27:55.753 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000286932 2024-09-10T19:27:55.754 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:55.754 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:55.754 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0,4", 2024-09-10T19:27:55.754 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:27:27.219122+0000", 2024-09-10T19:27:55.754 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00033848999999999999 2024-09-10T19:27:55.754 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:55.754 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:55.754 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:27:55.754 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:27:27.223242+0000", 2024-09-10T19:27:55.754 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0041198809999999997 2024-09-10T19:27:55.754 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:55.754 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:55.754 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-09-10T19:27:55.754 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:27:27.227553+0000", 2024-09-10T19:27:55.755 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0043104229999999999 2024-09-10T19:27:55.755 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:55.755 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:55.755 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:27:55.755 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:27:27.235264+0000", 2024-09-10T19:27:55.755 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0077119170000000004 2024-09-10T19:27:55.755 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:55.755 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:55.755 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-09-10T19:27:55.755 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:27:27.235312+0000", 2024-09-10T19:27:55.755 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.7079000000000001e-05 2024-09-10T19:27:55.755 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:55.755 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:55.755 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-10T19:27:55.756 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:27:27.235346+0000", 2024-09-10T19:27:55.756 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.4261999999999998e-05 2024-09-10T19:27:55.756 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:27:55.756 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-10T19:27:55.756 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:27:55.756 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:55.756 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:55.756 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.25622.0:245297 10.c 10:3acd0a33:::benchmark_data_smithi183_176537_object122089:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e365)", 2024-09-10T19:27:55.756 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-10T19:27:27.557070+0000", 2024-09-10T19:27:55.756 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 28.113930953000001, 2024-09-10T19:27:55.756 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.020907083999999999, 2024-09-10T19:27:55.756 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-10T19:27:55.756 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T19:27:55.757 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-10T19:27:55.757 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.25622", 2024-09-10T19:27:55.757 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.183:0/3685189199", 2024-09-10T19:27:55.757 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 245297 2024-09-10T19:27:55.757 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:55.757 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-10T19:27:55.757 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:55.757 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-10T19:27:55.757 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:27:27.557070+0000", 2024-09-10T19:27:55.757 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:27:55.757 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:55.757 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:55.757 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-10T19:27:55.758 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:27:27.557070+0000", 2024-09-10T19:27:55.758 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:27:55.758 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:55.758 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:55.758 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-10T19:27:55.758 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:27:27.557069+0000", 2024-09-10T19:27:55.758 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.999999 2024-09-10T19:27:55.758 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:55.758 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:55.758 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-10T19:27:55.758 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:27:27.557082+0000", 2024-09-10T19:27:55.758 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.2378000000000001e-05 2024-09-10T19:27:55.759 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:55.759 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:55.759 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-10T19:27:55.759 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:27:27.557094+0000", 2024-09-10T19:27:55.759 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.225e-05 2024-09-10T19:27:55.759 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:55.759 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:55.759 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-10T19:27:55.759 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:27:27.557099+0000", 2024-09-10T19:27:55.759 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.9239999999999998e-06 2024-09-10T19:27:55.759 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:55.759 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:55.759 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:27:55.759 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:27:27.558569+0000", 2024-09-10T19:27:55.760 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0014696449999999999 2024-09-10T19:27:55.760 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:55.760 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:55.760 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-10T19:27:55.760 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:27:27.558847+0000", 2024-09-10T19:27:55.760 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00027831600000000002 2024-09-10T19:27:55.760 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:55.760 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:55.760 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0,4", 2024-09-10T19:27:55.760 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:27:27.559283+0000", 2024-09-10T19:27:55.760 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00043646199999999998 2024-09-10T19:27:55.760 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:55.760 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:55.761 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:27:55.761 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:27:27.567537+0000", 2024-09-10T19:27:55.761 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0082539910000000005 2024-09-10T19:27:55.761 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:55.761 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:55.761 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-09-10T19:27:55.761 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:27:27.567558+0000", 2024-09-10T19:27:55.761 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.0319e-05 2024-09-10T19:27:55.761 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:55.761 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:55.761 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:27:55.761 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:27:27.577849+0000", 2024-09-10T19:27:55.761 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.010291827 2024-09-10T19:27:55.762 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:55.762 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:55.762 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-09-10T19:27:55.762 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:27:27.577926+0000", 2024-09-10T19:27:55.762 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.6765000000000001e-05 2024-09-10T19:27:55.762 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:55.762 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:55.762 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-10T19:27:55.762 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:27:27.577978+0000", 2024-09-10T19:27:55.762 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.1350000000000001e-05 2024-09-10T19:27:55.762 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:27:55.762 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-10T19:27:55.762 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:27:55.762 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:55.762 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:55.763 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.25622.0:245502 10.c 10:3bfcdd2a:::benchmark_data_smithi183_176537_object122294:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e365)", 2024-09-10T19:27:55.763 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-10T19:27:27.684388+0000", 2024-09-10T19:27:55.763 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 27.986613008999999, 2024-09-10T19:27:55.763 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.021788482000000001, 2024-09-10T19:27:55.763 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-10T19:27:55.763 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T19:27:55.763 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-10T19:27:55.763 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.25622", 2024-09-10T19:27:55.763 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.183:0/3685189199", 2024-09-10T19:27:55.763 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 245502 2024-09-10T19:27:55.763 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:55.763 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-10T19:27:55.763 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:55.763 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-10T19:27:55.764 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:27:27.684388+0000", 2024-09-10T19:27:55.764 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:27:55.764 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:55.764 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:55.764 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-10T19:27:55.764 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:27:27.684388+0000", 2024-09-10T19:27:55.764 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:27:55.764 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:55.764 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:55.764 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-10T19:27:55.764 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:27:27.684387+0000", 2024-09-10T19:27:55.764 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999986 2024-09-10T19:27:55.764 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:55.765 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:55.765 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-10T19:27:55.765 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:27:27.684393+0000", 2024-09-10T19:27:55.765 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.0610000000000001e-06 2024-09-10T19:27:55.765 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:55.765 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:55.765 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-10T19:27:55.765 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:27:27.684413+0000", 2024-09-10T19:27:55.765 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.0021e-05 2024-09-10T19:27:55.765 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:55.765 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:55.765 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-10T19:27:55.765 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:27:27.684420+0000", 2024-09-10T19:27:55.766 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.3999999999999997e-06 2024-09-10T19:27:55.766 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:55.766 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:55.766 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:27:55.766 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:27:27.688884+0000", 2024-09-10T19:27:55.766 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0044643570000000004 2024-09-10T19:27:55.766 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:55.766 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:55.766 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-10T19:27:55.766 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:27:27.689185+0000", 2024-09-10T19:27:55.766 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00030074799999999999 2024-09-10T19:27:55.766 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:55.766 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:55.766 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0,4", 2024-09-10T19:27:55.767 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:27:27.689555+0000", 2024-09-10T19:27:55.767 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00037013699999999998 2024-09-10T19:27:55.767 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:55.767 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:55.767 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:27:55.767 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:27:27.699915+0000", 2024-09-10T19:27:55.767 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.010359996999999999 2024-09-10T19:27:55.767 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:55.767 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:55.767 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-09-10T19:27:55.767 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:27:27.700001+0000", 2024-09-10T19:27:55.767 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.6125999999999995e-05 2024-09-10T19:27:55.767 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:55.768 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:55.768 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:27:55.768 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:27:27.706055+0000", 2024-09-10T19:27:55.768 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0060535559999999999 2024-09-10T19:27:55.768 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:55.768 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:55.768 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-09-10T19:27:55.768 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:27:27.706137+0000", 2024-09-10T19:27:55.768 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.2063000000000004e-05 2024-09-10T19:27:55.768 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:55.768 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:55.768 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-10T19:27:55.768 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:27:27.706177+0000", 2024-09-10T19:27:55.769 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.0259000000000002e-05 2024-09-10T19:27:55.769 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:27:55.769 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-10T19:27:55.769 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:27:55.769 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:55.769 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:55.769 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.25622.0:245503 10.c 10:3db0731a:::benchmark_data_smithi183_176537_object122295:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e365)", 2024-09-10T19:27:55.769 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-10T19:27:27.684435+0000", 2024-09-10T19:27:55.769 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 27.986566398000001, 2024-09-10T19:27:55.769 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.021884677000000002, 2024-09-10T19:27:55.769 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-10T19:27:55.769 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T19:27:55.769 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-10T19:27:55.770 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.25622", 2024-09-10T19:27:55.770 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.183:0/3685189199", 2024-09-10T19:27:55.770 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 245503 2024-09-10T19:27:55.770 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:55.770 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-10T19:27:55.770 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:55.770 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-10T19:27:55.770 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:27:27.684435+0000", 2024-09-10T19:27:55.770 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:27:55.770 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:55.770 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:55.770 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-10T19:27:55.770 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:27:27.684435+0000", 2024-09-10T19:27:55.771 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:27:55.771 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:55.771 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:55.771 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-10T19:27:55.771 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:27:27.684435+0000", 2024-09-10T19:27:55.771 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999995 2024-09-10T19:27:55.771 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:55.771 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:55.771 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-10T19:27:55.771 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:27:27.684438+0000", 2024-09-10T19:27:55.771 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.0139999999999999e-06 2024-09-10T19:27:55.772 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:55.772 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:55.772 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-10T19:27:55.772 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:27:27.684444+0000", 2024-09-10T19:27:55.772 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.4999999999999996e-06 2024-09-10T19:27:55.772 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:55.772 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:55.772 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-10T19:27:55.772 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:27:27.684448+0000", 2024-09-10T19:27:55.772 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.5300000000000001e-06 2024-09-10T19:27:55.772 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:55.772 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:55.772 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:27:55.772 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:27:27.692603+0000", 2024-09-10T19:27:55.773 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0081554990000000001 2024-09-10T19:27:55.773 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:55.773 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:55.773 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-10T19:27:55.773 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:27:27.692715+0000", 2024-09-10T19:27:55.773 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000112093 2024-09-10T19:27:55.773 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:55.773 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:55.773 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0,4", 2024-09-10T19:27:55.773 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:27:27.692836+0000", 2024-09-10T19:27:55.774 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00012086 2024-09-10T19:27:55.774 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:55.774 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:55.774 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-09-10T19:27:55.774 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:27:27.700005+0000", 2024-09-10T19:27:55.774 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.007169239 2024-09-10T19:27:55.774 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:55.774 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:55.774 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:27:55.774 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:27:27.700035+0000", 2024-09-10T19:27:55.774 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.9328000000000001e-05 2024-09-10T19:27:55.774 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:55.774 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:55.775 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:27:55.775 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:27:27.706220+0000", 2024-09-10T19:27:55.775 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0061852579999999999 2024-09-10T19:27:55.775 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:55.775 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:55.775 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-09-10T19:27:55.775 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:27:27.706287+0000", 2024-09-10T19:27:55.775 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.7130000000000003e-05 2024-09-10T19:27:55.775 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:55.775 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:55.775 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-10T19:27:55.775 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:27:27.706320+0000", 2024-09-10T19:27:55.775 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.2635999999999997e-05 2024-09-10T19:27:55.776 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:27:55.776 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-10T19:27:55.776 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:27:55.776 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:55.776 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:55.776 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.25622.0:245978 10.c 10:3e744ee9:::benchmark_data_smithi183_176537_object122770:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e365)", 2024-09-10T19:27:55.776 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-10T19:27:27.996292+0000", 2024-09-10T19:27:55.776 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 27.674709209, 2024-09-10T19:27:55.776 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.024030996999999998, 2024-09-10T19:27:55.776 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-10T19:27:55.776 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T19:27:55.776 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-10T19:27:55.776 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.25622", 2024-09-10T19:27:55.777 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.183:0/3685189199", 2024-09-10T19:27:55.777 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 245978 2024-09-10T19:27:55.777 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:55.777 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-10T19:27:55.777 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:55.777 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-10T19:27:55.777 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:27:27.996292+0000", 2024-09-10T19:27:55.777 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:27:55.777 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:55.777 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:55.777 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-10T19:27:55.777 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:27:27.996292+0000", 2024-09-10T19:27:55.777 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:27:55.778 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:55.778 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:55.778 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-10T19:27:55.778 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:27:27.996291+0000", 2024-09-10T19:27:55.778 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999986 2024-09-10T19:27:55.778 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:55.778 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:55.778 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-10T19:27:55.778 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:27:27.996297+0000", 2024-09-10T19:27:55.778 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.1109999999999996e-06 2024-09-10T19:27:55.778 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:55.778 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:55.778 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-10T19:27:55.778 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:27:27.996309+0000", 2024-09-10T19:27:55.779 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.2241e-05 2024-09-10T19:27:55.779 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:55.779 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:55.779 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-10T19:27:55.779 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:27:27.996315+0000", 2024-09-10T19:27:55.779 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.5500000000000002e-06 2024-09-10T19:27:55.779 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:55.779 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:55.779 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:27:55.779 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:27:27.996367+0000", 2024-09-10T19:27:55.779 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.1768999999999999e-05 2024-09-10T19:27:55.779 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:55.779 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:55.779 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-10T19:27:55.780 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:27:27.996478+0000", 2024-09-10T19:27:55.780 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00011107099999999999 2024-09-10T19:27:55.780 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:55.780 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:55.780 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0,4", 2024-09-10T19:27:55.780 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:27:27.996592+0000", 2024-09-10T19:27:55.780 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000114107 2024-09-10T19:27:55.780 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:55.780 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:55.780 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-09-10T19:27:55.780 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:27:28.000305+0000", 2024-09-10T19:27:55.780 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0037131920000000001 2024-09-10T19:27:55.780 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:55.780 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:55.781 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:27:55.781 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:27:28.004204+0000", 2024-09-10T19:27:55.781 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0038986569999999998 2024-09-10T19:27:55.781 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:55.781 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:55.781 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:27:55.781 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:27:28.020211+0000", 2024-09-10T19:27:55.781 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.016007903 2024-09-10T19:27:55.781 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:55.781 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:55.781 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-09-10T19:27:55.781 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:27:28.020284+0000", 2024-09-10T19:27:55.781 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.2421000000000003e-05 2024-09-10T19:27:55.782 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:55.782 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:55.782 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-10T19:27:55.782 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:27:28.020323+0000", 2024-09-10T19:27:55.782 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.9373999999999999e-05 2024-09-10T19:27:55.782 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:27:55.782 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-10T19:27:55.783 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:27:55.783 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:55.784 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:55.784 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.25622.0:246383 10.c 10:3914734c:::benchmark_data_smithi183_176537_object123175:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e365)", 2024-09-10T19:27:55.784 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-10T19:27:28.258862+0000", 2024-09-10T19:27:55.784 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 27.412139315000001, 2024-09-10T19:27:55.784 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.024925362999999999, 2024-09-10T19:27:55.784 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-10T19:27:55.784 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T19:27:55.784 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-10T19:27:55.784 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.25622", 2024-09-10T19:27:55.784 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.183:0/3685189199", 2024-09-10T19:27:55.784 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 246383 2024-09-10T19:27:55.784 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:55.785 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-10T19:27:55.785 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:55.785 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-10T19:27:55.785 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:27:28.258862+0000", 2024-09-10T19:27:55.785 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:27:55.785 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:55.785 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:55.785 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-10T19:27:55.785 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:27:28.258862+0000", 2024-09-10T19:27:55.785 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:27:55.785 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:55.785 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:55.785 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-10T19:27:55.786 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:27:28.258861+0000", 2024-09-10T19:27:55.786 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999986 2024-09-10T19:27:55.786 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:55.786 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:55.786 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-10T19:27:55.786 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:27:28.258867+0000", 2024-09-10T19:27:55.786 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.1349999999999997e-06 2024-09-10T19:27:55.786 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:55.786 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:55.786 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-10T19:27:55.786 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:27:28.258891+0000", 2024-09-10T19:27:55.786 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.4207999999999999e-05 2024-09-10T19:27:55.786 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:55.786 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:55.786 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-10T19:27:55.787 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:27:28.258897+0000", 2024-09-10T19:27:55.787 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.0170000000000002e-06 2024-09-10T19:27:55.787 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:55.787 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:55.787 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:27:55.787 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:27:28.258986+0000", 2024-09-10T19:27:55.787 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.8523999999999995e-05 2024-09-10T19:27:55.787 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:55.787 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:55.787 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-10T19:27:55.787 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:27:28.259089+0000", 2024-09-10T19:27:55.787 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000103618 2024-09-10T19:27:55.787 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:55.787 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:55.788 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0,4", 2024-09-10T19:27:55.788 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:27:28.259198+0000", 2024-09-10T19:27:55.788 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00010846200000000001 2024-09-10T19:27:55.788 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:55.788 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:55.788 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-09-10T19:27:55.788 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:27:28.262375+0000", 2024-09-10T19:27:55.788 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0031771 2024-09-10T19:27:55.788 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:55.788 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:55.788 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:27:55.788 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:27:28.262403+0000", 2024-09-10T19:27:55.788 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.8459999999999999e-05 2024-09-10T19:27:55.788 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:55.789 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:55.789 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:27:55.789 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:27:28.283636+0000", 2024-09-10T19:27:55.789 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.021232320999999998 2024-09-10T19:27:55.789 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:55.789 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:55.789 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-09-10T19:27:55.789 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:27:28.283733+0000", 2024-09-10T19:27:55.789 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.7214999999999997e-05 2024-09-10T19:27:55.789 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:55.789 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:55.789 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-10T19:27:55.790 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:27:28.283788+0000", 2024-09-10T19:27:55.790 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.4592999999999997e-05 2024-09-10T19:27:55.790 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:27:55.790 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-10T19:27:55.790 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:27:55.790 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:55.790 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:55.790 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.25622.0:246384 10.c 10:361cc3a9:::benchmark_data_smithi183_176537_object123176:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e365)", 2024-09-10T19:27:55.790 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-10T19:27:28.258908+0000", 2024-09-10T19:27:55.790 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 27.412093037999998, 2024-09-10T19:27:55.790 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.025076522, 2024-09-10T19:27:55.790 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-10T19:27:55.790 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T19:27:55.790 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-10T19:27:55.791 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.25622", 2024-09-10T19:27:55.791 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.183:0/3685189199", 2024-09-10T19:27:55.791 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 246384 2024-09-10T19:27:55.791 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:55.791 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-10T19:27:55.791 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:55.791 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-10T19:27:55.791 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:27:28.258908+0000", 2024-09-10T19:27:55.791 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:27:55.791 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:55.791 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:55.791 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-10T19:27:55.791 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:27:28.258908+0000", 2024-09-10T19:27:55.791 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:27:55.792 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:55.792 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:55.792 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-10T19:27:55.792 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:27:28.258908+0000", 2024-09-10T19:27:55.792 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999995 2024-09-10T19:27:55.792 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:55.792 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:55.792 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-10T19:27:55.792 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:27:28.258911+0000", 2024-09-10T19:27:55.792 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.7870000000000002e-06 2024-09-10T19:27:55.792 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:55.792 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:55.792 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-10T19:27:55.792 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:27:28.258917+0000", 2024-09-10T19:27:55.793 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.0279999999999997e-06 2024-09-10T19:27:55.793 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:55.793 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:55.793 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-10T19:27:55.793 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:27:28.258921+0000", 2024-09-10T19:27:55.793 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.6990000000000001e-06 2024-09-10T19:27:55.793 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:55.793 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:55.793 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:27:55.793 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:27:28.262423+0000", 2024-09-10T19:27:55.793 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0035028250000000002 2024-09-10T19:27:55.793 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:55.793 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:55.794 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-10T19:27:55.794 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:27:28.262525+0000", 2024-09-10T19:27:55.794 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00010192999999999999 2024-09-10T19:27:55.794 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:55.794 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:55.794 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0,4", 2024-09-10T19:27:55.794 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:27:28.262631+0000", 2024-09-10T19:27:55.794 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000105456 2024-09-10T19:27:55.794 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:55.794 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:55.794 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:27:55.794 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:27:28.265408+0000", 2024-09-10T19:27:55.794 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0027769869999999999 2024-09-10T19:27:55.794 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:55.794 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:55.795 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-09-10T19:27:55.795 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:27:28.265525+0000", 2024-09-10T19:27:55.795 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00011738 2024-09-10T19:27:55.795 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:55.795 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:55.795 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:27:55.795 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:27:28.283825+0000", 2024-09-10T19:27:55.795 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.018299673999999998 2024-09-10T19:27:55.795 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:55.795 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:55.795 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-09-10T19:27:55.795 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:27:28.283891+0000", 2024-09-10T19:27:55.795 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.6076999999999998e-05 2024-09-10T19:27:55.795 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:55.796 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:55.796 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-10T19:27:55.796 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:27:28.283985+0000", 2024-09-10T19:27:55.796 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.4103000000000003e-05 2024-09-10T19:27:55.796 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:27:55.796 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-10T19:27:55.796 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:27:55.796 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:55.796 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:55.796 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.16026.0:3524 11.4 11:2372fc33:::benchmark_data_smithi183_203944_object3523:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e375)", 2024-09-10T19:27:55.796 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-10T19:27:54.950305+0000", 2024-09-10T19:27:55.796 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 0.72069623800000004, 2024-09-10T19:27:55.796 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.014944404999999999, 2024-09-10T19:27:55.797 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-10T19:27:55.797 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T19:27:55.797 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-10T19:27:55.797 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.16026", 2024-09-10T19:27:55.797 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.183:0/1450567484", 2024-09-10T19:27:55.797 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 3524 2024-09-10T19:27:55.797 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:55.797 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-10T19:27:55.797 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:55.797 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-10T19:27:55.797 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:27:54.950305+0000", 2024-09-10T19:27:55.797 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:27:55.797 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:55.798 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:55.798 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-10T19:27:55.798 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:27:54.950305+0000", 2024-09-10T19:27:55.798 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:27:55.798 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:55.798 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:55.798 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-10T19:27:55.798 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:27:54.950304+0000", 2024-09-10T19:27:55.798 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.999999 2024-09-10T19:27:55.798 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:55.798 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:55.798 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-10T19:27:55.798 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:27:54.950344+0000", 2024-09-10T19:27:55.798 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.9619000000000001e-05 2024-09-10T19:27:55.799 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:55.799 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:55.799 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-10T19:27:55.799 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:27:54.950354+0000", 2024-09-10T19:27:55.799 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.9899999999999992e-06 2024-09-10T19:27:55.799 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:55.799 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:55.799 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-10T19:27:55.799 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:27:54.950360+0000", 2024-09-10T19:27:55.799 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.5879999999999997e-06 2024-09-10T19:27:55.799 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:55.799 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:55.799 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:27:55.799 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:27:54.950399+0000", 2024-09-10T19:27:55.800 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.9199999999999997e-05 2024-09-10T19:27:55.800 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:55.800 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:55.800 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-10T19:27:55.800 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:27:54.950685+0000", 2024-09-10T19:27:55.800 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00028602400000000001 2024-09-10T19:27:55.800 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:55.800 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:55.800 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 1,6", 2024-09-10T19:27:55.800 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:27:54.950819+0000", 2024-09-10T19:27:55.800 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00013433100000000001 2024-09-10T19:27:55.800 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:55.800 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:55.800 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-09-10T19:27:55.801 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:27:54.952035+0000", 2024-09-10T19:27:55.801 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0012159499999999999 2024-09-10T19:27:55.801 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:55.801 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:55.801 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:27:55.801 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:27:54.952138+0000", 2024-09-10T19:27:55.801 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000102835 2024-09-10T19:27:55.801 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:55.801 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:55.801 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:27:55.801 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:27:54.965190+0000", 2024-09-10T19:27:55.801 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.013051959 2024-09-10T19:27:55.801 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:55.801 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:55.802 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-09-10T19:27:55.802 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:27:54.965223+0000", 2024-09-10T19:27:55.802 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.3244000000000002e-05 2024-09-10T19:27:55.802 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:55.802 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:55.802 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-10T19:27:55.802 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:27:54.965250+0000", 2024-09-10T19:27:55.802 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.6523e-05 2024-09-10T19:27:55.802 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:27:55.802 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-10T19:27:55.802 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:27:55.802 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:55.802 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:55.802 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.16026.0:3530 11.4 11:2bb346db:::benchmark_data_smithi183_203944_object3529:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e375)", 2024-09-10T19:27:55.803 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-10T19:27:54.952834+0000", 2024-09-10T19:27:55.803 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 0.71816763800000005, 2024-09-10T19:27:55.803 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.014909802, 2024-09-10T19:27:55.803 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-10T19:27:55.803 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T19:27:55.803 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-10T19:27:55.803 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.16026", 2024-09-10T19:27:55.803 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.183:0/1450567484", 2024-09-10T19:27:55.803 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 3530 2024-09-10T19:27:55.803 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:55.803 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-10T19:27:55.803 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:55.803 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-10T19:27:55.803 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:27:54.952834+0000", 2024-09-10T19:27:55.804 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:27:55.804 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:55.804 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:55.804 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-10T19:27:55.804 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:27:54.952834+0000", 2024-09-10T19:27:55.804 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:27:55.804 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:55.804 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:55.804 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-10T19:27:55.804 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:27:54.952833+0000", 2024-09-10T19:27:55.804 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999995 2024-09-10T19:27:55.804 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:55.804 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:55.805 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-10T19:27:55.805 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:27:54.952863+0000", 2024-09-10T19:27:55.805 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.9722000000000001e-05 2024-09-10T19:27:55.805 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:55.805 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:55.805 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-10T19:27:55.805 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:27:54.952873+0000", 2024-09-10T19:27:55.805 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.0077e-05 2024-09-10T19:27:55.805 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:55.805 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:55.805 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-10T19:27:55.805 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:27:54.952878+0000", 2024-09-10T19:27:55.805 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.5539999999999997e-06 2024-09-10T19:27:55.805 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:55.806 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:55.806 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:27:55.806 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:27:54.953458+0000", 2024-09-10T19:27:55.806 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00057918300000000004 2024-09-10T19:27:55.806 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:55.806 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:55.806 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-10T19:27:55.806 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:27:54.953566+0000", 2024-09-10T19:27:55.806 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00010865399999999999 2024-09-10T19:27:55.806 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:55.806 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:55.806 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 1,6", 2024-09-10T19:27:55.806 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:27:54.953691+0000", 2024-09-10T19:27:55.806 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00012519199999999999 2024-09-10T19:27:55.807 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:55.807 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:55.807 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-09-10T19:27:55.807 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:27:54.958878+0000", 2024-09-10T19:27:55.807 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.005186586 2024-09-10T19:27:55.807 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:55.807 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:55.807 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:27:55.807 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:27:54.961335+0000", 2024-09-10T19:27:55.807 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0024567500000000002 2024-09-10T19:27:55.807 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:55.807 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:55.808 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:27:55.808 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:27:54.967658+0000", 2024-09-10T19:27:55.808 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0063228420000000004 2024-09-10T19:27:55.808 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:55.809 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:55.809 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-09-10T19:27:55.809 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:27:54.967697+0000", 2024-09-10T19:27:55.809 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.9292999999999998e-05 2024-09-10T19:27:55.809 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:27:55.809 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:27:55.809 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-10T19:27:55.809 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:27:54.967744+0000", 2024-09-10T19:27:55.809 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.6650000000000002e-05 2024-09-10T19:27:55.809 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:27:55.809 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-10T19:27:55.809 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:27:55.809 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:27:55.809 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-10T19:27:55.810 INFO:teuthology.orchestra.run.smithi191.stdout:} 2024-09-10T19:27:55.850 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:27:55.886 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:27:55.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:55.887 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:27:55.988 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:27:56.023 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:27:56.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:56.024 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:27:56.057 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: 2 16 7489 7473 233.512 228.125 0.00540461 0.00427477 2024-09-10T19:27:56.125 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:27:56.159 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:27:56.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:56.161 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:27:56.226 INFO:tasks.thrashosds.thrasher:Added osd 6 2024-09-10T19:27:56.261 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:27:56.302 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:27:56.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:56.304 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:27:56.324 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:27:56 smithi183 ceph-mon[29736]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T19:27:56.324 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:27:56 smithi183 ceph-mon[29736]: from='client.? 172.21.15.183:0/3520079644' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-10T19:27:56.324 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:27:56 smithi183 ceph-mon[29736]: osdmap e376: 8 total, 8 up, 7 in 2024-09-10T19:27:56.324 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:27:56 smithi183 ceph-mon[29736]: 5.e scrub starts 2024-09-10T19:27:56.324 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:27:56 smithi183 ceph-mon[29736]: 5.e scrub ok 2024-09-10T19:27:56.324 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:27:56 smithi183 ceph-mon[41562]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T19:27:56.324 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:27:56 smithi183 ceph-mon[41562]: from='client.? 172.21.15.183:0/3520079644' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-10T19:27:56.324 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:27:56 smithi183 ceph-mon[41562]: osdmap e376: 8 total, 8 up, 7 in 2024-09-10T19:27:56.324 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:27:56 smithi183 ceph-mon[41562]: 5.e scrub starts 2024-09-10T19:27:56.324 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:27:56 smithi183 ceph-mon[41562]: 5.e scrub ok 2024-09-10T19:27:56.405 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:27:56.443 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:27:56.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:56.444 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:27:56.465 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:27:56 smithi191 ceph-mon[32884]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T19:27:56.465 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:27:56 smithi191 ceph-mon[32884]: from='client.? 172.21.15.183:0/3520079644' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-10T19:27:56.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:27:56 smithi191 ceph-mon[32884]: osdmap e376: 8 total, 8 up, 7 in 2024-09-10T19:27:56.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:27:56 smithi191 ceph-mon[32884]: 5.e scrub starts 2024-09-10T19:27:56.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:27:56 smithi191 ceph-mon[32884]: 5.e scrub ok 2024-09-10T19:27:56.544 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:27:56.584 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:27:56.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:56.585 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:27:56.686 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:27:56.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:56.722 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:27:56.722 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:27:56.743 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph daemon osd.1 dump_ops_in_flight 2024-09-10T19:27:56.823 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:27:56.855 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:27:56.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:56.856 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:27:56.907 INFO:teuthology.orchestra.run.smithi183.stdout:osd.0: {} 2024-09-10T19:27:56.907 INFO:teuthology.orchestra.run.smithi183.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-10T19:27:56.944 INFO:teuthology.orchestra.run.smithi183.stdout:osd.1: {} 2024-09-10T19:27:56.944 INFO:teuthology.orchestra.run.smithi183.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-10T19:27:56.957 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:27:56.974 INFO:teuthology.orchestra.run.smithi183.stdout:osd.2: {} 2024-09-10T19:27:56.974 INFO:teuthology.orchestra.run.smithi183.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-10T19:27:56.997 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:27:56.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:56.998 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:27:57.009 INFO:teuthology.orchestra.run.smithi183.stdout:osd.3: {} 2024-09-10T19:27:57.009 INFO:teuthology.orchestra.run.smithi183.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-10T19:27:57.039 INFO:teuthology.orchestra.run.smithi183.stdout:osd.4: {} 2024-09-10T19:27:57.040 INFO:teuthology.orchestra.run.smithi183.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-10T19:27:57.058 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: 3 15 10782 10767 224.259 205.875 0.00373462 0.00444964 2024-09-10T19:27:57.072 INFO:teuthology.orchestra.run.smithi183.stdout:osd.5: {} 2024-09-10T19:27:57.072 INFO:teuthology.orchestra.run.smithi183.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-10T19:27:57.099 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:27:57.103 INFO:teuthology.orchestra.run.smithi183.stdout:osd.6: {} 2024-09-10T19:27:57.104 INFO:teuthology.orchestra.run.smithi183.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-10T19:27:57.104 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:27:57 smithi183 ceph-mon[29736]: pgmap v822: 158 pgs: 6 peering, 152 active+clean; 224 MiB data, 6.0 GiB used, 620 GiB / 626 GiB avail; 170 B/s rd, 37 MiB/s wr, 595 op/s; 43 KiB/s, 10 objects/s recovering 2024-09-10T19:27:57.104 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:27:57 smithi183 ceph-mon[29736]: 4.19 scrub starts 2024-09-10T19:27:57.104 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:27:57 smithi183 ceph-mon[29736]: 4.19 scrub ok 2024-09-10T19:27:57.104 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:27:57 smithi183 ceph-mon[29736]: 6.10 scrub starts 2024-09-10T19:27:57.104 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:27:57 smithi183 ceph-mon[29736]: 6.10 scrub ok 2024-09-10T19:27:57.104 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:27:57 smithi183 ceph-mon[29736]: from='client.? 172.21.15.183:0/1535661795' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T19:27:57.105 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:27:57 smithi183 ceph-mon[29736]: 5.a scrub starts 2024-09-10T19:27:57.105 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:27:57 smithi183 ceph-mon[29736]: 5.a scrub ok 2024-09-10T19:27:57.105 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:27:57 smithi183 ceph-mon[41562]: pgmap v822: 158 pgs: 6 peering, 152 active+clean; 224 MiB data, 6.0 GiB used, 620 GiB / 626 GiB avail; 170 B/s rd, 37 MiB/s wr, 595 op/s; 43 KiB/s, 10 objects/s recovering 2024-09-10T19:27:57.105 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:27:57 smithi183 ceph-mon[41562]: 4.19 scrub starts 2024-09-10T19:27:57.105 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:27:57 smithi183 ceph-mon[41562]: 4.19 scrub ok 2024-09-10T19:27:57.105 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:27:57 smithi183 ceph-mon[41562]: 6.10 scrub starts 2024-09-10T19:27:57.105 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:27:57 smithi183 ceph-mon[41562]: 6.10 scrub ok 2024-09-10T19:27:57.105 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:27:57 smithi183 ceph-mon[41562]: from='client.? 172.21.15.183:0/1535661795' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T19:27:57.105 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:27:57 smithi183 ceph-mon[41562]: 5.a scrub starts 2024-09-10T19:27:57.105 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:27:57 smithi183 ceph-mon[41562]: 5.a scrub ok 2024-09-10T19:27:57.140 INFO:teuthology.orchestra.run.smithi183.stdout:osd.7: {} 2024-09-10T19:27:57.141 INFO:teuthology.orchestra.run.smithi183.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-10T19:27:57.147 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:27:57.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:57.148 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:27:57.249 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:27:57.286 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:27:57.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:57.287 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:27:57.388 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:27:57.423 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:27:57.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:57.424 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:27:57.452 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:27:57 smithi191 ceph-mon[32884]: pgmap v822: 158 pgs: 6 peering, 152 active+clean; 224 MiB data, 6.0 GiB used, 620 GiB / 626 GiB avail; 170 B/s rd, 37 MiB/s wr, 595 op/s; 43 KiB/s, 10 objects/s recovering 2024-09-10T19:27:57.452 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:27:57 smithi191 ceph-mon[32884]: 4.19 scrub starts 2024-09-10T19:27:57.453 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:27:57 smithi191 ceph-mon[32884]: 4.19 scrub ok 2024-09-10T19:27:57.453 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:27:57 smithi191 ceph-mon[32884]: 6.10 scrub starts 2024-09-10T19:27:57.453 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:27:57 smithi191 ceph-mon[32884]: 6.10 scrub ok 2024-09-10T19:27:57.453 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:27:57 smithi191 ceph-mon[32884]: from='client.? 172.21.15.183:0/1535661795' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T19:27:57.453 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:27:57 smithi191 ceph-mon[32884]: 5.a scrub starts 2024-09-10T19:27:57.453 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:27:57 smithi191 ceph-mon[32884]: 5.a scrub ok 2024-09-10T19:27:57.525 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:27:57.562 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:27:57.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:57.563 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:27:57.585 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:27:57.664 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:27:57.700 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:27:57.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:57.701 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:27:57.715 INFO:journalctl@ceph.mgr.x.smithi191.stdout:Sep 10 19:27:57 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-x[34967]: ::ffff:172.21.15.191 - - [10/Sep/2024:19:27:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T19:27:57.802 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:27:57.811 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-10T19:27:57.838 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:27:57.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:57.840 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:27:57.941 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:27:57.986 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:27:57.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:57.987 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:27:58.058 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: 4 16 13906 13890 216.988 195.188 0.00811169 0.00460079 2024-09-10T19:27:58.087 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:27:58.123 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:27:58.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:58.124 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:27:58.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:27:58 smithi191 ceph-mon[32884]: 5.11 scrub starts 2024-09-10T19:27:58.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:27:58 smithi191 ceph-mon[32884]: 5.11 scrub ok 2024-09-10T19:27:58.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:27:58 smithi191 ceph-mon[32884]: 6.16 scrub starts 2024-09-10T19:27:58.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:27:58 smithi191 ceph-mon[32884]: 6.16 scrub ok 2024-09-10T19:27:58.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:27:58 smithi191 ceph-mon[32884]: 4.12 scrub starts 2024-09-10T19:27:58.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:27:58 smithi191 ceph-mon[32884]: 4.12 scrub ok 2024-09-10T19:27:58.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:27:58 smithi191 ceph-mon[32884]: 1.0 scrub starts 2024-09-10T19:27:58.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:27:58 smithi191 ceph-mon[32884]: 1.0 scrub ok 2024-09-10T19:27:58.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:27:58 smithi191 ceph-mon[32884]: 6.17 scrub starts 2024-09-10T19:27:58.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:27:58 smithi191 ceph-mon[32884]: 6.17 scrub ok 2024-09-10T19:27:58.225 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:27:58.263 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:27:58.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:58.264 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:27:58.365 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:27:58.396 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:27:58.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:58.397 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:27:58.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:27:58 smithi183 ceph-mon[29736]: 5.11 scrub starts 2024-09-10T19:27:58.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:27:58 smithi183 ceph-mon[29736]: 5.11 scrub ok 2024-09-10T19:27:58.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:27:58 smithi183 ceph-mon[29736]: 6.16 scrub starts 2024-09-10T19:27:58.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:27:58 smithi183 ceph-mon[29736]: 6.16 scrub ok 2024-09-10T19:27:58.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:27:58 smithi183 ceph-mon[29736]: 4.12 scrub starts 2024-09-10T19:27:58.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:27:58 smithi183 ceph-mon[29736]: 4.12 scrub ok 2024-09-10T19:27:58.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:27:58 smithi183 ceph-mon[29736]: 1.0 scrub starts 2024-09-10T19:27:58.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:27:58 smithi183 ceph-mon[29736]: 1.0 scrub ok 2024-09-10T19:27:58.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:27:58 smithi183 ceph-mon[29736]: 6.17 scrub starts 2024-09-10T19:27:58.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:27:58 smithi183 ceph-mon[29736]: 6.17 scrub ok 2024-09-10T19:27:58.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:27:58 smithi183 ceph-mon[41562]: 5.11 scrub starts 2024-09-10T19:27:58.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:27:58 smithi183 ceph-mon[41562]: 5.11 scrub ok 2024-09-10T19:27:58.433 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:27:58 smithi183 ceph-mon[41562]: 6.16 scrub starts 2024-09-10T19:27:58.433 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:27:58 smithi183 ceph-mon[41562]: 6.16 scrub ok 2024-09-10T19:27:58.433 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:27:58 smithi183 ceph-mon[41562]: 4.12 scrub starts 2024-09-10T19:27:58.433 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:27:58 smithi183 ceph-mon[41562]: 4.12 scrub ok 2024-09-10T19:27:58.433 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:27:58 smithi183 ceph-mon[41562]: 1.0 scrub starts 2024-09-10T19:27:58.433 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:27:58 smithi183 ceph-mon[41562]: 1.0 scrub ok 2024-09-10T19:27:58.433 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:27:58 smithi183 ceph-mon[41562]: 6.17 scrub starts 2024-09-10T19:27:58.433 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:27:58 smithi183 ceph-mon[41562]: 6.17 scrub ok 2024-09-10T19:27:58.498 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:27:58.534 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:27:58.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:58.535 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:27:58.636 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:27:58.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:58.672 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:27:58.672 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:27:58.774 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:27:58.801 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:27:58.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:58.802 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:27:58.903 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:27:58.937 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:27:58.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:58.938 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:27:59.039 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:27:59.058 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: 5 16 16255 16239 202.952 146.812 0.00323997 0.00487318 2024-09-10T19:27:59.061 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:27:59.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:59.062 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:27:59.162 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:27:59.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:59.201 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:27:59.201 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:27:59.302 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:27:59.338 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:27:59.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:59.339 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:27:59.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:27:59 smithi183 ceph-mon[29736]: pgmap v823: 158 pgs: 158 active+clean; 356 MiB data, 6.5 GiB used, 619 GiB / 626 GiB avail; 1023 B/s rd, 59 MiB/s wr, 949 op/s; 43 KiB/s, 12 objects/s recovering 2024-09-10T19:27:59.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:27:59 smithi183 ceph-mon[29736]: 2.c scrub starts 2024-09-10T19:27:59.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:27:59 smithi183 ceph-mon[29736]: 2.c scrub ok 2024-09-10T19:27:59.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:27:59 smithi183 ceph-mon[29736]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:27:59.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:27:59 smithi183 ceph-mon[29736]: 6.12 scrub starts 2024-09-10T19:27:59.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:27:59 smithi183 ceph-mon[29736]: 6.12 scrub ok 2024-09-10T19:27:59.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:27:59 smithi183 ceph-mon[29736]: 2.f scrub starts 2024-09-10T19:27:59.433 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:27:59 smithi183 ceph-mon[29736]: 2.f scrub ok 2024-09-10T19:27:59.433 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:27:59 smithi183 ceph-mon[41562]: pgmap v823: 158 pgs: 158 active+clean; 356 MiB data, 6.5 GiB used, 619 GiB / 626 GiB avail; 1023 B/s rd, 59 MiB/s wr, 949 op/s; 43 KiB/s, 12 objects/s recovering 2024-09-10T19:27:59.433 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:27:59 smithi183 ceph-mon[41562]: 2.c scrub starts 2024-09-10T19:27:59.433 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:27:59 smithi183 ceph-mon[41562]: 2.c scrub ok 2024-09-10T19:27:59.434 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:27:59 smithi183 ceph-mon[41562]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:27:59.434 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:27:59 smithi183 ceph-mon[41562]: 6.12 scrub starts 2024-09-10T19:27:59.434 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:27:59 smithi183 ceph-mon[41562]: 6.12 scrub ok 2024-09-10T19:27:59.434 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:27:59 smithi183 ceph-mon[41562]: 2.f scrub starts 2024-09-10T19:27:59.434 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:27:59 smithi183 ceph-mon[41562]: 2.f scrub ok 2024-09-10T19:27:59.440 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:27:59.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:27:59 smithi191 ceph-mon[32884]: pgmap v823: 158 pgs: 158 active+clean; 356 MiB data, 6.5 GiB used, 619 GiB / 626 GiB avail; 1023 B/s rd, 59 MiB/s wr, 949 op/s; 43 KiB/s, 12 objects/s recovering 2024-09-10T19:27:59.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:27:59 smithi191 ceph-mon[32884]: 2.c scrub starts 2024-09-10T19:27:59.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:27:59 smithi191 ceph-mon[32884]: 2.c scrub ok 2024-09-10T19:27:59.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:27:59 smithi191 ceph-mon[32884]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:27:59.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:27:59 smithi191 ceph-mon[32884]: 6.12 scrub starts 2024-09-10T19:27:59.467 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:27:59 smithi191 ceph-mon[32884]: 6.12 scrub ok 2024-09-10T19:27:59.467 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:27:59 smithi191 ceph-mon[32884]: 2.f scrub starts 2024-09-10T19:27:59.467 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:27:59 smithi191 ceph-mon[32884]: 2.f scrub ok 2024-09-10T19:27:59.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:59.475 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:27:59.475 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:27:59.576 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:27:59.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:59.618 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:27:59.618 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:27:59.719 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:27:59.757 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:27:59.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:59.758 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:27:59.859 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:27:59.894 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:27:59.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:27:59.895 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:27:59.996 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:28:00.033 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:28:00.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:28:00.034 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:28:00.058 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: 6 16 18914 18898 196.823 166.188 0.00480343 0.00507593 2024-09-10T19:28:00.135 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:28:00.170 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:28:00.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:28:00.171 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:28:00.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:28:00 smithi183 ceph-mon[29736]: 6.19 scrub starts 2024-09-10T19:28:00.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:28:00 smithi183 ceph-mon[29736]: 6.19 scrub ok 2024-09-10T19:28:00.182 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:28:00 smithi183 ceph-mon[29736]: 5.10 scrub starts 2024-09-10T19:28:00.182 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:28:00 smithi183 ceph-mon[29736]: 5.10 scrub ok 2024-09-10T19:28:00.182 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:28:00 smithi183 ceph-mon[29736]: osdmap e377: 8 total, 8 up, 7 in 2024-09-10T19:28:00.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:28:00 smithi183 ceph-mon[41562]: 6.19 scrub starts 2024-09-10T19:28:00.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:28:00 smithi183 ceph-mon[41562]: 6.19 scrub ok 2024-09-10T19:28:00.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:28:00 smithi183 ceph-mon[41562]: 5.10 scrub starts 2024-09-10T19:28:00.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:28:00 smithi183 ceph-mon[41562]: 5.10 scrub ok 2024-09-10T19:28:00.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:28:00 smithi183 ceph-mon[41562]: osdmap e377: 8 total, 8 up, 7 in 2024-09-10T19:28:00.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:28:00 smithi191 ceph-mon[32884]: 6.19 scrub starts 2024-09-10T19:28:00.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:28:00 smithi191 ceph-mon[32884]: 6.19 scrub ok 2024-09-10T19:28:00.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:28:00 smithi191 ceph-mon[32884]: 5.10 scrub starts 2024-09-10T19:28:00.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:28:00 smithi191 ceph-mon[32884]: 5.10 scrub ok 2024-09-10T19:28:00.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:28:00 smithi191 ceph-mon[32884]: osdmap e377: 8 total, 8 up, 7 in 2024-09-10T19:28:00.217 INFO:journalctl@ceph.iscsi.iscsi.a.smithi191.stdout:Sep 10 19:27:59 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-iscsi-iscsi-a[61816]: debug there is no tcmu-runner data available 2024-09-10T19:28:00.272 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:28:00.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:28:00.312 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:28:00.313 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:28:00.413 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:28:00.449 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:28:00.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:28:00.450 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:28:00.551 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:28:00.583 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:28:00.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:28:00.585 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:28:00.674 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-10T19:28:00.686 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:28:00.764 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:28:00.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:28:00.765 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:28:00.772 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:28:00.866 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:28:00.898 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:28:00.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:28:00.900 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:28:01.001 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:28:01.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:28:01.038 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:28:01.038 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:28:01.040 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:28:01 smithi183 ceph-mon[29736]: from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:28:01.040 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:28:01 smithi183 ceph-mon[29736]: pgmap v824: 158 pgs: 158 active+clean; 471 MiB data, 6.9 GiB used, 619 GiB / 626 GiB avail; 773 B/s rd, 59 MiB/s wr, 948 op/s; 33 KiB/s, 9 objects/s recovering 2024-09-10T19:28:01.040 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:28:01 smithi183 ceph-mon[29736]: 4.1e scrub starts 2024-09-10T19:28:01.040 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:28:01 smithi183 ceph-mon[29736]: 4.1e scrub ok 2024-09-10T19:28:01.059 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: 7 16 21650 21634 193.086 171 0.00509822 0.00517323 2024-09-10T19:28:01.139 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:28:01.174 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:28:01.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:28:01.175 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:28:01.226 INFO:tasks.thrashosds.thrasher:in_osds: [0, 3, 4, 5, 7, 1, 6] out_osds: [2] dead_osds: [] live_osds: [2, 3, 4, 5, 6, 7, 1, 0] 2024-09-10T19:28:01.226 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-10T19:28:01.227 INFO:tasks.thrashosds.thrasher:inject_pause on osd.4 2024-09-10T19:28:01.227 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-09-10T19:28:01.227 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-09-10T19:28:01.227 DEBUG:teuthology.orchestra.run.smithi191:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph daemon osd.4 config set filestore_inject_stall 3 2024-09-10T19:28:01.275 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:28:01.313 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:28:01.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:28:01.314 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:28:01.316 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:28:01 smithi183 ceph-mon[41562]: from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:28:01.317 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:28:01 smithi183 ceph-mon[41562]: pgmap v824: 158 pgs: 158 active+clean; 471 MiB data, 6.9 GiB used, 619 GiB / 626 GiB avail; 773 B/s rd, 59 MiB/s wr, 948 op/s; 33 KiB/s, 9 objects/s recovering 2024-09-10T19:28:01.317 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:28:01 smithi183 ceph-mon[41562]: 4.1e scrub starts 2024-09-10T19:28:01.317 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:28:01 smithi183 ceph-mon[41562]: 4.1e scrub ok 2024-09-10T19:28:01.415 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:28:01.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:28:01.455 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:28:01.455 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:28:01.465 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:28:01 smithi191 ceph-mon[32884]: from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:28:01.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:28:01 smithi191 ceph-mon[32884]: pgmap v824: 158 pgs: 158 active+clean; 471 MiB data, 6.9 GiB used, 619 GiB / 626 GiB avail; 773 B/s rd, 59 MiB/s wr, 948 op/s; 33 KiB/s, 9 objects/s recovering 2024-09-10T19:28:01.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:28:01 smithi191 ceph-mon[32884]: 4.1e scrub starts 2024-09-10T19:28:01.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:28:01 smithi191 ceph-mon[32884]: 4.1e scrub ok 2024-09-10T19:28:01.557 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:28:01.589 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Sep 10 19:28:01 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-y[29989]: ::ffff:172.21.15.191 - - [10/Sep/2024:19:28:01] "GET /metrics HTTP/1.1" 200 224767 "" "Prometheus/2.43.0" 2024-09-10T19:28:01.594 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:28:01.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:28:01.595 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:28:01.657 INFO:teuthology.orchestra.run.smithi191.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.b/config 2024-09-10T19:28:01.697 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:28:01.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:28:01.738 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:28:01.738 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:28:01.839 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:28:01.872 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:28:01.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:28:01.873 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:28:01.974 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:28:02.022 INFO:teuthology.orchestra.run.smithi183.stdout:ERROR: (22) Invalid argument 2024-09-10T19:28:02.023 INFO:teuthology.orchestra.run.smithi183.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-10T19:28:02.025 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:28:02.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:28:02.026 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:28:02.060 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: 8 16 24180 24164 188.716 158.125 0.00493235 0.00529227 2024-09-10T19:28:02.128 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:28:02.163 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:28:02.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:28:02.164 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:28:02.265 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:28:02.304 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:28:02.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:28:02.306 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:28:02.311 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:28:02 smithi191 ceph-mon[32884]: 4.15 scrub starts 2024-09-10T19:28:02.311 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:28:02 smithi191 ceph-mon[32884]: 4.15 scrub ok 2024-09-10T19:28:02.311 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:28:02 smithi191 ceph-mon[32884]: 4.b scrub starts 2024-09-10T19:28:02.311 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:28:02 smithi191 ceph-mon[32884]: 4.b scrub ok 2024-09-10T19:28:02.312 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:28:02 smithi191 ceph-mon[32884]: 6.5 scrub starts 2024-09-10T19:28:02.312 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:28:02 smithi191 ceph-mon[32884]: 6.5 scrub ok 2024-09-10T19:28:02.406 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:28:02.433 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:28:02 smithi183 ceph-mon[29736]: 4.15 scrub starts 2024-09-10T19:28:02.433 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:28:02 smithi183 ceph-mon[29736]: 4.15 scrub ok 2024-09-10T19:28:02.433 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:28:02 smithi183 ceph-mon[29736]: 4.b scrub starts 2024-09-10T19:28:02.433 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:28:02 smithi183 ceph-mon[29736]: 4.b scrub ok 2024-09-10T19:28:02.434 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:28:02 smithi183 ceph-mon[29736]: 6.5 scrub starts 2024-09-10T19:28:02.434 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:28:02 smithi183 ceph-mon[29736]: 6.5 scrub ok 2024-09-10T19:28:02.434 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:28:02 smithi183 ceph-mon[41562]: 4.15 scrub starts 2024-09-10T19:28:02.434 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:28:02 smithi183 ceph-mon[41562]: 4.15 scrub ok 2024-09-10T19:28:02.434 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:28:02 smithi183 ceph-mon[41562]: 4.b scrub starts 2024-09-10T19:28:02.434 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:28:02 smithi183 ceph-mon[41562]: 4.b scrub ok 2024-09-10T19:28:02.434 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:28:02 smithi183 ceph-mon[41562]: 6.5 scrub starts 2024-09-10T19:28:02.434 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:28:02 smithi183 ceph-mon[41562]: 6.5 scrub ok 2024-09-10T19:28:02.440 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:28:02.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:28:02.441 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:28:02.542 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:28:02.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:28:02.564 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:28:02.565 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:28:02.665 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:28:02.700 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:28:02.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:28:02.702 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:28:02.803 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:28:02.838 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:28:02.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:28:02.839 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:28:02.940 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:28:02.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:28:02.977 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:28:02.977 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:28:03.060 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: 9 16 26642 26626 184.844 153.875 0.00436963 0.00540495 2024-09-10T19:28:03.078 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:28:03.113 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:28:03.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:28:03.115 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:28:03.182 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:28:03 smithi183 ceph-mon[29736]: pgmap v826: 158 pgs: 158 active+clean; 1.1 GiB data, 8.3 GiB used, 617 GiB / 626 GiB avail; 639 B/s rd, 140 MiB/s wr, 2.24k op/s; 32 KiB/s, 9 objects/s recovering 2024-09-10T19:28:03.182 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:28:03 smithi183 ceph-mon[29736]: 4.14 scrub starts 2024-09-10T19:28:03.182 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:28:03 smithi183 ceph-mon[29736]: 4.14 scrub ok 2024-09-10T19:28:03.182 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:28:03 smithi183 ceph-mon[29736]: 4.5 scrub starts 2024-09-10T19:28:03.182 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:28:03 smithi183 ceph-mon[29736]: 4.5 scrub ok 2024-09-10T19:28:03.182 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:28:03 smithi183 ceph-mon[29736]: 6.8 scrub starts 2024-09-10T19:28:03.182 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:28:03 smithi183 ceph-mon[29736]: 6.8 scrub ok 2024-09-10T19:28:03.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:28:03 smithi183 ceph-mon[41562]: pgmap v826: 158 pgs: 158 active+clean; 1.1 GiB data, 8.3 GiB used, 617 GiB / 626 GiB avail; 639 B/s rd, 140 MiB/s wr, 2.24k op/s; 32 KiB/s, 9 objects/s recovering 2024-09-10T19:28:03.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:28:03 smithi183 ceph-mon[41562]: 4.14 scrub starts 2024-09-10T19:28:03.183 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:28:03 smithi183 ceph-mon[41562]: 4.14 scrub ok 2024-09-10T19:28:03.183 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:28:03 smithi183 ceph-mon[41562]: 4.5 scrub starts 2024-09-10T19:28:03.183 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:28:03 smithi183 ceph-mon[41562]: 4.5 scrub ok 2024-09-10T19:28:03.183 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:28:03 smithi183 ceph-mon[41562]: 6.8 scrub starts 2024-09-10T19:28:03.183 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:28:03 smithi183 ceph-mon[41562]: 6.8 scrub ok 2024-09-10T19:28:03.215 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:28:03.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:28:03.251 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:28:03.251 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:28:03.298 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:28:03 smithi191 ceph-mon[32884]: pgmap v826: 158 pgs: 158 active+clean; 1.1 GiB data, 8.3 GiB used, 617 GiB / 626 GiB avail; 639 B/s rd, 140 MiB/s wr, 2.24k op/s; 32 KiB/s, 9 objects/s recovering 2024-09-10T19:28:03.298 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:28:03 smithi191 ceph-mon[32884]: 4.14 scrub starts 2024-09-10T19:28:03.298 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:28:03 smithi191 ceph-mon[32884]: 4.14 scrub ok 2024-09-10T19:28:03.298 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:28:03 smithi191 ceph-mon[32884]: 4.5 scrub starts 2024-09-10T19:28:03.298 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:28:03 smithi191 ceph-mon[32884]: 4.5 scrub ok 2024-09-10T19:28:03.298 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:28:03 smithi191 ceph-mon[32884]: 6.8 scrub starts 2024-09-10T19:28:03.299 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:28:03 smithi191 ceph-mon[32884]: 6.8 scrub ok 2024-09-10T19:28:03.352 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:28:03.399 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:28:03.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:28:03.400 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:28:03.501 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:28:03.526 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph daemon osd.1 dump_blocked_ops 2024-09-10T19:28:03.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:28:03.538 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:28:03.538 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:28:03.639 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:28:03.683 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:28:03.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:28:03.684 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:28:03.785 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:28:03.820 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:28:03.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:28:03.821 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:28:03.922 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:28:03.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:28:03.959 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:28:03.959 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:28:03.992 INFO:teuthology.orchestra.run.smithi191.stdout:{ 2024-09-10T19:28:03.993 INFO:teuthology.orchestra.run.smithi191.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-09-10T19:28:03.993 INFO:teuthology.orchestra.run.smithi191.stdout:} 2024-09-10T19:28:04.060 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:28:04.060 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: 10 16 29003 28987 181.113 147.562 0.00835003 0.00551755 2024-09-10T19:28:04.075 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:28:04.118 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:28:04.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:28:04.120 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:28:04.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:28:04 smithi191 ceph-mon[32884]: 6.b scrub starts 2024-09-10T19:28:04.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:28:04 smithi191 ceph-mon[32884]: 6.b scrub ok 2024-09-10T19:28:04.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:28:04 smithi191 ceph-mon[32884]: 2.14 scrub starts 2024-09-10T19:28:04.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:28:04 smithi191 ceph-mon[32884]: 2.14 scrub ok 2024-09-10T19:28:04.219 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:28:04 smithi183 ceph-mon[29736]: 6.b scrub starts 2024-09-10T19:28:04.220 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:28:04 smithi183 ceph-mon[29736]: 6.b scrub ok 2024-09-10T19:28:04.220 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:28:04 smithi183 ceph-mon[29736]: 2.14 scrub starts 2024-09-10T19:28:04.220 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:28:04 smithi183 ceph-mon[29736]: 2.14 scrub ok 2024-09-10T19:28:04.220 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:28:04 smithi183 ceph-mon[41562]: 6.b scrub starts 2024-09-10T19:28:04.220 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:28:04 smithi183 ceph-mon[41562]: 6.b scrub ok 2024-09-10T19:28:04.220 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:28:04 smithi183 ceph-mon[41562]: 2.14 scrub starts 2024-09-10T19:28:04.220 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:28:04 smithi183 ceph-mon[41562]: 2.14 scrub ok 2024-09-10T19:28:04.220 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:28:04.256 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:28:04.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:28:04.257 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:28:04.358 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:28:04.396 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:28:04.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:28:04.397 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:28:04.498 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:28:04.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:28:04.531 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:28:04.532 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:28:04.633 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:28:04.676 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:28:04.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:28:04.677 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:28:04.778 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:28:04.819 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:28:04.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:28:04.821 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:28:04.921 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:28:04.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:28:04.959 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:28:04.960 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:28:05.060 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:28:05.062 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: 11 16 31295 31279 177.671 143.25 0.00414313 0.00562343 2024-09-10T19:28:05.069 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:28:05.092 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:28:05.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:28:05.093 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:28:05.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:28:05 smithi183 ceph-mon[29736]: pgmap v827: 158 pgs: 158 active+clean; 1.2 GiB data, 9.1 GiB used, 617 GiB / 626 GiB avail; 1.1 KiB/s rd, 141 MiB/s wr, 2.26k op/s; 29 KiB/s, 7 objects/s recovering 2024-09-10T19:28:05.182 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:28:05 smithi183 ceph-mon[29736]: 2.11 scrub starts 2024-09-10T19:28:05.182 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:28:05 smithi183 ceph-mon[29736]: 2.11 scrub ok 2024-09-10T19:28:05.182 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:28:05 smithi183 ceph-mon[29736]: 5.19 scrub starts 2024-09-10T19:28:05.182 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:28:05 smithi183 ceph-mon[29736]: 5.19 scrub ok 2024-09-10T19:28:05.183 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:28:05 smithi183 ceph-mon[29736]: osdmap e378: 8 total, 8 up, 7 in 2024-09-10T19:28:05.183 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:28:05 smithi183 ceph-mon[41562]: pgmap v827: 158 pgs: 158 active+clean; 1.2 GiB data, 9.1 GiB used, 617 GiB / 626 GiB avail; 1.1 KiB/s rd, 141 MiB/s wr, 2.26k op/s; 29 KiB/s, 7 objects/s recovering 2024-09-10T19:28:05.183 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:28:05 smithi183 ceph-mon[41562]: 2.11 scrub starts 2024-09-10T19:28:05.183 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:28:05 smithi183 ceph-mon[41562]: 2.11 scrub ok 2024-09-10T19:28:05.184 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:28:05 smithi183 ceph-mon[41562]: 5.19 scrub starts 2024-09-10T19:28:05.184 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:28:05 smithi183 ceph-mon[41562]: 5.19 scrub ok 2024-09-10T19:28:05.184 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:28:05 smithi183 ceph-mon[41562]: osdmap e378: 8 total, 8 up, 7 in 2024-09-10T19:28:05.193 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:28:05.227 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:28:05.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:28:05.228 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:28:05.329 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:28:05.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:28:05.375 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:28:05.376 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:28:05.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:28:05 smithi191 ceph-mon[32884]: pgmap v827: 158 pgs: 158 active+clean; 1.2 GiB data, 9.1 GiB used, 617 GiB / 626 GiB avail; 1.1 KiB/s rd, 141 MiB/s wr, 2.26k op/s; 29 KiB/s, 7 objects/s recovering 2024-09-10T19:28:05.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:28:05 smithi191 ceph-mon[32884]: 2.11 scrub starts 2024-09-10T19:28:05.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:28:05 smithi191 ceph-mon[32884]: 2.11 scrub ok 2024-09-10T19:28:05.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:28:05 smithi191 ceph-mon[32884]: 5.19 scrub starts 2024-09-10T19:28:05.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:28:05 smithi191 ceph-mon[32884]: 5.19 scrub ok 2024-09-10T19:28:05.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:28:05 smithi191 ceph-mon[32884]: osdmap e378: 8 total, 8 up, 7 in 2024-09-10T19:28:05.476 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:28:05.509 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:28:05.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:28:05.510 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:28:05.611 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:28:05.643 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:28:05.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:28:05.644 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:28:05.745 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:28:05.780 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:28:05.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:28:05.781 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:28:05.882 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:28:05.919 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:28:05.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:28:05.920 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:28:06.021 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:28:06.056 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:28:06.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:28:06.057 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:28:06.060 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: 12 16 33560 33544 174.662 141.562 0.0100743 0.00571995 2024-09-10T19:28:06.089 INFO:teuthology.orchestra.run.smithi183.stderr:nodeep-scrub is unset 2024-09-10T19:28:06.158 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:28:06.192 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:28:06.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:28:06.193 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:28:06.295 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:28:06.333 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:28:06.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:28:06.334 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:28:06.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:28:06 smithi183 ceph-mon[29736]: purged_snaps scrub starts 2024-09-10T19:28:06.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:28:06 smithi183 ceph-mon[29736]: purged_snaps scrub ok 2024-09-10T19:28:06.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:28:06 smithi183 ceph-mon[29736]: 2.1f scrub starts 2024-09-10T19:28:06.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:28:06 smithi183 ceph-mon[29736]: 2.1f scrub ok 2024-09-10T19:28:06.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:28:06 smithi183 ceph-mon[29736]: 4.7 scrub starts 2024-09-10T19:28:06.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:28:06 smithi183 ceph-mon[29736]: 4.7 scrub ok 2024-09-10T19:28:06.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:28:06 smithi183 ceph-mon[29736]: from='client.? 172.21.15.183:0/1818215640' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-10T19:28:06.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:28:06 smithi183 ceph-mon[29736]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-10T19:28:06.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:28:06 smithi183 ceph-mon[29736]: 2.0 scrub starts 2024-09-10T19:28:06.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:28:06 smithi183 ceph-mon[29736]: 2.0 scrub ok 2024-09-10T19:28:06.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:28:06 smithi183 ceph-mon[29736]: 4.1a scrub starts 2024-09-10T19:28:06.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:28:06 smithi183 ceph-mon[29736]: 4.1a scrub ok 2024-09-10T19:28:06.433 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:28:06 smithi183 ceph-mon[41562]: purged_snaps scrub starts 2024-09-10T19:28:06.433 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:28:06 smithi183 ceph-mon[41562]: purged_snaps scrub ok 2024-09-10T19:28:06.433 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:28:06 smithi183 ceph-mon[41562]: 2.1f scrub starts 2024-09-10T19:28:06.433 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:28:06 smithi183 ceph-mon[41562]: 2.1f scrub ok 2024-09-10T19:28:06.433 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:28:06 smithi183 ceph-mon[41562]: 4.7 scrub starts 2024-09-10T19:28:06.433 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:28:06 smithi183 ceph-mon[41562]: 4.7 scrub ok 2024-09-10T19:28:06.433 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:28:06 smithi183 ceph-mon[41562]: from='client.? 172.21.15.183:0/1818215640' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-10T19:28:06.433 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:28:06 smithi183 ceph-mon[41562]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-10T19:28:06.433 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:28:06 smithi183 ceph-mon[41562]: 2.0 scrub starts 2024-09-10T19:28:06.433 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:28:06 smithi183 ceph-mon[41562]: 2.0 scrub ok 2024-09-10T19:28:06.433 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:28:06 smithi183 ceph-mon[41562]: 4.1a scrub starts 2024-09-10T19:28:06.433 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:28:06 smithi183 ceph-mon[41562]: 4.1a scrub ok 2024-09-10T19:28:06.435 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:28:06.468 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:28:06 smithi191 ceph-mon[32884]: purged_snaps scrub starts 2024-09-10T19:28:06.468 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:28:06 smithi191 ceph-mon[32884]: purged_snaps scrub ok 2024-09-10T19:28:06.468 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:28:06 smithi191 ceph-mon[32884]: 2.1f scrub starts 2024-09-10T19:28:06.468 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:28:06 smithi191 ceph-mon[32884]: 2.1f scrub ok 2024-09-10T19:28:06.468 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:28:06 smithi191 ceph-mon[32884]: 4.7 scrub starts 2024-09-10T19:28:06.469 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:28:06 smithi191 ceph-mon[32884]: 4.7 scrub ok 2024-09-10T19:28:06.469 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:28:06 smithi191 ceph-mon[32884]: from='client.? 172.21.15.183:0/1818215640' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-10T19:28:06.469 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:28:06 smithi191 ceph-mon[32884]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-10T19:28:06.469 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:28:06 smithi191 ceph-mon[32884]: 2.0 scrub starts 2024-09-10T19:28:06.469 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:28:06 smithi191 ceph-mon[32884]: 2.0 scrub ok 2024-09-10T19:28:06.469 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:28:06 smithi191 ceph-mon[32884]: 4.1a scrub starts 2024-09-10T19:28:06.469 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:28:06 smithi191 ceph-mon[32884]: 4.1a scrub ok 2024-09-10T19:28:06.471 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:28:06.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:28:06.472 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:28:06.573 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:28:06.609 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:28:06.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:28:06.610 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:28:06.711 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:28:06.747 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:28:06.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:28:06.749 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:28:06.852 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:28:06.886 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:28:06.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:28:06.888 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:28:06.989 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:28:07.023 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:28:07.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:28:07.024 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:28:07.060 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: 13 16 35810 35794 172.043 140.625 0.0041954 0.00580782 2024-09-10T19:28:07.125 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:28:07.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:28:07.163 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:28:07.163 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:28:07.264 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:28:07.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:28:07.306 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:28:07.306 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:28:07.407 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:28:07.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:28:07 smithi183 ceph-mon[29736]: pgmap v829: 158 pgs: 158 active+clean; 1.8 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 639 B/s rd, 192 MiB/s wr, 3.07k op/s 2024-09-10T19:28:07.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:28:07 smithi183 ceph-mon[29736]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-10T19:28:07.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:28:07 smithi183 ceph-mon[29736]: Cluster is now healthy 2024-09-10T19:28:07.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:28:07 smithi183 ceph-mon[29736]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-10T19:28:07.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:28:07 smithi183 ceph-mon[29736]: osdmap e379: 8 total, 8 up, 7 in 2024-09-10T19:28:07.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:28:07 smithi183 ceph-mon[29736]: 5.9 scrub starts 2024-09-10T19:28:07.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:28:07 smithi183 ceph-mon[29736]: 5.9 scrub ok 2024-09-10T19:28:07.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:28:07 smithi183 ceph-mon[41562]: pgmap v829: 158 pgs: 158 active+clean; 1.8 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 639 B/s rd, 192 MiB/s wr, 3.07k op/s 2024-09-10T19:28:07.433 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:28:07 smithi183 ceph-mon[41562]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-10T19:28:07.433 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:28:07 smithi183 ceph-mon[41562]: Cluster is now healthy 2024-09-10T19:28:07.433 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:28:07 smithi183 ceph-mon[41562]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-10T19:28:07.433 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:28:07 smithi183 ceph-mon[41562]: osdmap e379: 8 total, 8 up, 7 in 2024-09-10T19:28:07.433 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:28:07 smithi183 ceph-mon[41562]: 5.9 scrub starts 2024-09-10T19:28:07.433 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:28:07 smithi183 ceph-mon[41562]: 5.9 scrub ok 2024-09-10T19:28:07.442 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:28:07.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:28:07.444 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:28:07.451 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:28:07 smithi191 ceph-mon[32884]: pgmap v829: 158 pgs: 158 active+clean; 1.8 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 639 B/s rd, 192 MiB/s wr, 3.07k op/s 2024-09-10T19:28:07.452 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:28:07 smithi191 ceph-mon[32884]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-10T19:28:07.452 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:28:07 smithi191 ceph-mon[32884]: Cluster is now healthy 2024-09-10T19:28:07.452 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:28:07 smithi191 ceph-mon[32884]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-10T19:28:07.452 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:28:07 smithi191 ceph-mon[32884]: osdmap e379: 8 total, 8 up, 7 in 2024-09-10T19:28:07.452 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:28:07 smithi191 ceph-mon[32884]: 5.9 scrub starts 2024-09-10T19:28:07.452 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:28:07 smithi191 ceph-mon[32884]: 5.9 scrub ok 2024-09-10T19:28:07.545 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:28:07.581 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:28:07.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:28:07.583 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:28:07.630 INFO:teuthology.orchestra.run.smithi183.stdout:osd.0: {} 2024-09-10T19:28:07.631 INFO:teuthology.orchestra.run.smithi183.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-10T19:28:07.664 INFO:teuthology.orchestra.run.smithi183.stdout:osd.1: {} 2024-09-10T19:28:07.664 INFO:teuthology.orchestra.run.smithi183.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-10T19:28:07.684 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:28:07.694 INFO:teuthology.orchestra.run.smithi183.stdout:osd.2: {} 2024-09-10T19:28:07.694 INFO:teuthology.orchestra.run.smithi183.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-10T19:28:07.716 INFO:journalctl@ceph.mgr.x.smithi191.stdout:Sep 10 19:28:07 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-x[34967]: ::ffff:172.21.15.191 - - [10/Sep/2024:19:28:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T19:28:07.718 INFO:teuthology.orchestra.run.smithi183.stdout:osd.3: {} 2024-09-10T19:28:07.718 INFO:teuthology.orchestra.run.smithi183.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-10T19:28:07.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:28:07.719 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:28:07.719 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:28:07.746 INFO:teuthology.orchestra.run.smithi183.stdout:osd.4: {} 2024-09-10T19:28:07.746 INFO:teuthology.orchestra.run.smithi183.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-10T19:28:07.778 INFO:teuthology.orchestra.run.smithi183.stdout:osd.5: {} 2024-09-10T19:28:07.778 INFO:teuthology.orchestra.run.smithi183.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-10T19:28:07.808 INFO:teuthology.orchestra.run.smithi183.stdout:osd.6: {} 2024-09-10T19:28:07.808 INFO:teuthology.orchestra.run.smithi183.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-10T19:28:07.820 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:28:07.847 INFO:teuthology.orchestra.run.smithi183.stdout:osd.7: {} 2024-09-10T19:28:07.848 INFO:teuthology.orchestra.run.smithi183.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-10T19:28:07.858 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:28:07.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:28:07.859 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:28:07.960 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:28:07.994 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:28:07.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:28:07.995 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:28:08.060 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: 14 16 37936 37920 169.245 132.875 0.00349282 0.00590349 2024-09-10T19:28:08.096 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:28:08.118 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:28:08.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:28:08.119 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:28:08.220 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:28:08.241 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:28:08.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:28:08.242 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:28:08.343 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:28:08.370 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:28:08.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:28:08.372 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:28:08.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:28:08 smithi183 ceph-mon[29736]: purged_snaps scrub starts 2024-09-10T19:28:08.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:28:08 smithi183 ceph-mon[29736]: purged_snaps scrub ok 2024-09-10T19:28:08.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:28:08 smithi183 ceph-mon[29736]: from='client.? 172.21.15.183:0/527680397' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T19:28:08.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:28:08 smithi183 ceph-mon[41562]: purged_snaps scrub starts 2024-09-10T19:28:08.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:28:08 smithi183 ceph-mon[41562]: purged_snaps scrub ok 2024-09-10T19:28:08.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:28:08 smithi183 ceph-mon[41562]: from='client.? 172.21.15.183:0/527680397' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T19:28:08.465 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:28:08 smithi191 ceph-mon[32884]: purged_snaps scrub starts 2024-09-10T19:28:08.465 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:28:08 smithi191 ceph-mon[32884]: purged_snaps scrub ok 2024-09-10T19:28:08.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:28:08 smithi191 ceph-mon[32884]: from='client.? 172.21.15.183:0/527680397' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T19:28:08.472 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:28:08.508 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:28:08.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:28:08.509 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:28:08.610 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:28:08.645 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:28:08.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:28:08.646 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:28:08.747 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:28:08.782 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:28:08.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:28:08.784 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:28:08.885 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:28:08.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:28:08.924 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:28:08.924 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:28:09.025 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:28:09.060 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: 15 16 39481 39465 164.399 96.5625 0.0064356 0.00607653 2024-09-10T19:28:09.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:28:09.062 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:28:09.062 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:28:09.163 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:28:09.198 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:28:09.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:28:09.199 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:28:09.300 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:28:09.322 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:28:09.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:28:09.323 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:28:09.424 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:28:09.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:28:09 smithi183 ceph-mon[29736]: pgmap v831: 158 pgs: 158 active+clean; 2.0 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 1.3 KiB/s rd, 195 MiB/s wr, 3.12k op/s 2024-09-10T19:28:09.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:28:09 smithi183 ceph-mon[29736]: osdmap e380: 8 total, 8 up, 7 in 2024-09-10T19:28:09.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:28:09 smithi183 ceph-mon[29736]: 3.2 scrub starts 2024-09-10T19:28:09.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:28:09 smithi183 ceph-mon[29736]: 3.2 scrub ok 2024-09-10T19:28:09.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:28:09 smithi183 ceph-mon[41562]: pgmap v831: 158 pgs: 158 active+clean; 2.0 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 1.3 KiB/s rd, 195 MiB/s wr, 3.12k op/s 2024-09-10T19:28:09.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:28:09 smithi183 ceph-mon[41562]: osdmap e380: 8 total, 8 up, 7 in 2024-09-10T19:28:09.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:28:09 smithi183 ceph-mon[41562]: 3.2 scrub starts 2024-09-10T19:28:09.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:28:09 smithi183 ceph-mon[41562]: 3.2 scrub ok 2024-09-10T19:28:09.457 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:28:09.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:28:09.458 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:28:09.465 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:28:09 smithi191 ceph-mon[32884]: pgmap v831: 158 pgs: 158 active+clean; 2.0 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 1.3 KiB/s rd, 195 MiB/s wr, 3.12k op/s 2024-09-10T19:28:09.465 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:28:09 smithi191 ceph-mon[32884]: osdmap e380: 8 total, 8 up, 7 in 2024-09-10T19:28:09.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:28:09 smithi191 ceph-mon[32884]: 3.2 scrub starts 2024-09-10T19:28:09.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:28:09 smithi191 ceph-mon[32884]: 3.2 scrub ok 2024-09-10T19:28:09.558 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:28:09.583 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:28:09.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:28:09.584 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:28:09.685 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:28:09.716 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:28:09.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:28:09.717 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:28:09.818 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:28:09.838 INFO:tasks.thrashosds.thrasher:in_osds: [0, 3, 4, 5, 7, 1, 6] out_osds: [2] dead_osds: [] live_osds: [2, 3, 4, 5, 6, 7, 1, 0] 2024-09-10T19:28:09.838 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-10T19:28:09.838 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap_items 2024-09-10T19:28:09.838 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph osd dump -f json-pretty 2024-09-10T19:28:09.851 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:28:09.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:28:09.852 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:28:09.953 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:28:09.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:28:09.975 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:28:09.975 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:28:10.060 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: 16 16 40890 40874 159.628 88.0625 0.0112792 0.00625897 2024-09-10T19:28:10.076 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:28:10.112 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:28:10.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:28:10.113 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:28:10.181 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:28:10 smithi183 ceph-mon[41562]: 5.f scrub starts 2024-09-10T19:28:10.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:28:10 smithi183 ceph-mon[41562]: 5.f scrub ok 2024-09-10T19:28:10.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:28:10 smithi183 ceph-mon[41562]: 4.1d scrub starts 2024-09-10T19:28:10.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:28:10 smithi183 ceph-mon[41562]: 4.1d scrub ok 2024-09-10T19:28:10.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:28:10 smithi183 ceph-mon[41562]: 4.6 scrub starts 2024-09-10T19:28:10.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:28:10 smithi183 ceph-mon[41562]: 4.6 scrub ok 2024-09-10T19:28:10.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:28:10 smithi183 ceph-mon[41562]: osdmap e381: 8 total, 8 up, 7 in 2024-09-10T19:28:10.182 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:28:10 smithi183 ceph-mon[29736]: 5.f scrub starts 2024-09-10T19:28:10.182 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:28:10 smithi183 ceph-mon[29736]: 5.f scrub ok 2024-09-10T19:28:10.182 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:28:10 smithi183 ceph-mon[29736]: 4.1d scrub starts 2024-09-10T19:28:10.182 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:28:10 smithi183 ceph-mon[29736]: 4.1d scrub ok 2024-09-10T19:28:10.182 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:28:10 smithi183 ceph-mon[29736]: 4.6 scrub starts 2024-09-10T19:28:10.183 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:28:10 smithi183 ceph-mon[29736]: 4.6 scrub ok 2024-09-10T19:28:10.183 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:28:10 smithi183 ceph-mon[29736]: osdmap e381: 8 total, 8 up, 7 in 2024-09-10T19:28:10.213 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:28:10.215 INFO:journalctl@ceph.iscsi.iscsi.a.smithi191.stdout:Sep 10 19:28:09 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-iscsi-iscsi-a[61816]: debug there is no tcmu-runner data available 2024-09-10T19:28:10.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:28:10 smithi191 ceph-mon[32884]: 5.f scrub starts 2024-09-10T19:28:10.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:28:10 smithi191 ceph-mon[32884]: 5.f scrub ok 2024-09-10T19:28:10.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:28:10 smithi191 ceph-mon[32884]: 4.1d scrub starts 2024-09-10T19:28:10.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:28:10 smithi191 ceph-mon[32884]: 4.1d scrub ok 2024-09-10T19:28:10.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:28:10 smithi191 ceph-mon[32884]: 4.6 scrub starts 2024-09-10T19:28:10.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:28:10 smithi191 ceph-mon[32884]: 4.6 scrub ok 2024-09-10T19:28:10.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:28:10 smithi191 ceph-mon[32884]: osdmap e381: 8 total, 8 up, 7 in 2024-09-10T19:28:10.249 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:28:10.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:28:10.250 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:28:10.351 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:28:10.386 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:28:10.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:28:10.387 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:28:10.488 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:28:10.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:28:10.520 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:28:10.520 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:28:10.621 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:28:10.663 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:28:10.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:28:10.664 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:28:10.765 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:28:10.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:28:10.801 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:28:10.801 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:28:10.902 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:28:10.935 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:28:10.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:28:10.937 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:28:11.037 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:28:11.060 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: 17 16 42766 42750 157.135 117.25 0.00978059 0.00635884 2024-09-10T19:28:11.076 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:28:11.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:28:11.077 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:28:11.177 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:28:11.212 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:28:11.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:28:11.213 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:28:11.314 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:28:11.350 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:28:11.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:28:11.351 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:28:11.431 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Sep 10 19:28:11 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-y[29989]: ::ffff:172.21.15.191 - - [10/Sep/2024:19:28:11] "GET /metrics HTTP/1.1" 200 224767 "" "Prometheus/2.43.0" 2024-09-10T19:28:11.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:28:11 smithi183 ceph-mon[29736]: from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:28:11.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:28:11 smithi183 ceph-mon[29736]: pgmap v833: 158 pgs: 158 active+clean; 2.1 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 853 B/s rd, 139 MiB/s wr, 2.22k op/s 2024-09-10T19:28:11.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:28:11 smithi183 ceph-mon[29736]: 1.a scrub starts 2024-09-10T19:28:11.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:28:11 smithi183 ceph-mon[29736]: 1.a scrub ok 2024-09-10T19:28:11.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:28:11 smithi183 ceph-mon[29736]: 4.1f scrub starts 2024-09-10T19:28:11.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:28:11 smithi183 ceph-mon[29736]: 4.1f scrub ok 2024-09-10T19:28:11.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:28:11 smithi183 ceph-mon[29736]: 1.5 scrub starts 2024-09-10T19:28:11.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:28:11 smithi183 ceph-mon[29736]: 1.5 scrub ok 2024-09-10T19:28:11.433 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:28:11 smithi183 ceph-mon[41562]: from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:28:11.433 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:28:11 smithi183 ceph-mon[41562]: pgmap v833: 158 pgs: 158 active+clean; 2.1 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 853 B/s rd, 139 MiB/s wr, 2.22k op/s 2024-09-10T19:28:11.433 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:28:11 smithi183 ceph-mon[41562]: 1.a scrub starts 2024-09-10T19:28:11.433 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:28:11 smithi183 ceph-mon[41562]: 1.a scrub ok 2024-09-10T19:28:11.433 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:28:11 smithi183 ceph-mon[41562]: 4.1f scrub starts 2024-09-10T19:28:11.433 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:28:11 smithi183 ceph-mon[41562]: 4.1f scrub ok 2024-09-10T19:28:11.433 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:28:11 smithi183 ceph-mon[41562]: 1.5 scrub starts 2024-09-10T19:28:11.433 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:28:11 smithi183 ceph-mon[41562]: 1.5 scrub ok 2024-09-10T19:28:11.452 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:28:11.465 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:28:11 smithi191 ceph-mon[32884]: from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:28:11.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:28:11 smithi191 ceph-mon[32884]: pgmap v833: 158 pgs: 158 active+clean; 2.1 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 853 B/s rd, 139 MiB/s wr, 2.22k op/s 2024-09-10T19:28:11.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:28:11 smithi191 ceph-mon[32884]: 1.a scrub starts 2024-09-10T19:28:11.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:28:11 smithi191 ceph-mon[32884]: 1.a scrub ok 2024-09-10T19:28:11.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:28:11 smithi191 ceph-mon[32884]: 4.1f scrub starts 2024-09-10T19:28:11.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:28:11 smithi191 ceph-mon[32884]: 4.1f scrub ok 2024-09-10T19:28:11.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:28:11 smithi191 ceph-mon[32884]: 1.5 scrub starts 2024-09-10T19:28:11.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:28:11 smithi191 ceph-mon[32884]: 1.5 scrub ok 2024-09-10T19:28:11.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:28:11.476 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:28:11.476 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:28:11.577 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:28:11.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:28:11.617 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:28:11.618 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:28:11.719 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:28:11.751 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:28:11.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:28:11.752 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:28:11.853 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:28:11.881 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:28:11.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:28:11.882 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:28:11.983 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:28:12.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:28:12.017 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:28:12.017 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:28:12.061 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: 18 16 44374 44358 153.988 100.5 0.0102213 0.00648959 2024-09-10T19:28:12.118 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:28:12.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:28:12.148 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:28:12.148 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:28:12.249 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:28:12.285 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:28:12.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:28:12.286 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:28:12.387 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:28:12.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:28:12.419 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:28:12.419 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:28:12.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:28:12 smithi183 ceph-mon[29736]: 2.12 scrub starts 2024-09-10T19:28:12.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:28:12 smithi183 ceph-mon[29736]: 2.12 scrub ok 2024-09-10T19:28:12.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:28:12 smithi183 ceph-mon[29736]: 2.1a scrub starts 2024-09-10T19:28:12.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:28:12 smithi183 ceph-mon[29736]: 2.1a scrub ok 2024-09-10T19:28:12.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:28:12 smithi183 ceph-mon[29736]: 4.e scrub starts 2024-09-10T19:28:12.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:28:12 smithi183 ceph-mon[29736]: 4.e scrub ok 2024-09-10T19:28:12.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:28:12 smithi183 ceph-mon[41562]: 2.12 scrub starts 2024-09-10T19:28:12.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:28:12 smithi183 ceph-mon[41562]: 2.12 scrub ok 2024-09-10T19:28:12.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:28:12 smithi183 ceph-mon[41562]: 2.1a scrub starts 2024-09-10T19:28:12.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:28:12 smithi183 ceph-mon[41562]: 2.1a scrub ok 2024-09-10T19:28:12.433 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:28:12 smithi183 ceph-mon[41562]: 4.e scrub starts 2024-09-10T19:28:12.433 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:28:12 smithi183 ceph-mon[41562]: 4.e scrub ok 2024-09-10T19:28:12.465 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:28:12 smithi191 ceph-mon[32884]: 2.12 scrub starts 2024-09-10T19:28:12.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:28:12 smithi191 ceph-mon[32884]: 2.12 scrub ok 2024-09-10T19:28:12.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:28:12 smithi191 ceph-mon[32884]: 2.1a scrub starts 2024-09-10T19:28:12.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:28:12 smithi191 ceph-mon[32884]: 2.1a scrub ok 2024-09-10T19:28:12.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:28:12 smithi191 ceph-mon[32884]: 4.e scrub starts 2024-09-10T19:28:12.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:28:12 smithi191 ceph-mon[32884]: 4.e scrub ok 2024-09-10T19:28:12.520 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:28:12.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:28:12.554 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:28:12.554 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:28:12.655 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:28:12.691 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:28:12.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:28:12.692 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:28:12.793 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:28:12.824 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:28:12.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:28:12.825 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:28:12.926 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:28:12.959 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:28:12.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:28:12.960 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:28:13.060 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:28:13.062 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: 19 16 45936 45920 151.022 97.625 0.00847058 0.00661622 2024-09-10T19:28:13.094 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:28:13.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:28:13.095 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:28:13.196 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:28:13.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:28:13.238 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:28:13.238 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:28:13.339 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:28:13.371 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:28:13.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:28:13.372 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:28:13.431 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:28:13 smithi183 ceph-mon[41562]: pgmap v835: 158 pgs: 158 active+clean; 2.5 GiB data, 12 GiB used, 613 GiB / 626 GiB avail; 853 B/s rd, 104 MiB/s wr, 1.67k op/s 2024-09-10T19:28:13.431 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:28:13 smithi183 ceph-mon[41562]: 2.17 scrub starts 2024-09-10T19:28:13.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:28:13 smithi183 ceph-mon[41562]: 2.17 scrub ok 2024-09-10T19:28:13.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:28:13 smithi183 ceph-mon[41562]: 4.c scrub starts 2024-09-10T19:28:13.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:28:13 smithi183 ceph-mon[41562]: 4.c scrub ok 2024-09-10T19:28:13.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:28:13 smithi183 ceph-mon[29736]: pgmap v835: 158 pgs: 158 active+clean; 2.5 GiB data, 12 GiB used, 613 GiB / 626 GiB avail; 853 B/s rd, 104 MiB/s wr, 1.67k op/s 2024-09-10T19:28:13.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:28:13 smithi183 ceph-mon[29736]: 2.17 scrub starts 2024-09-10T19:28:13.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:28:13 smithi183 ceph-mon[29736]: 2.17 scrub ok 2024-09-10T19:28:13.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:28:13 smithi183 ceph-mon[29736]: 4.c scrub starts 2024-09-10T19:28:13.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:28:13 smithi183 ceph-mon[29736]: 4.c scrub ok 2024-09-10T19:28:13.465 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:28:13 smithi191 ceph-mon[32884]: pgmap v835: 158 pgs: 158 active+clean; 2.5 GiB data, 12 GiB used, 613 GiB / 626 GiB avail; 853 B/s rd, 104 MiB/s wr, 1.67k op/s 2024-09-10T19:28:13.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:28:13 smithi191 ceph-mon[32884]: 2.17 scrub starts 2024-09-10T19:28:13.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:28:13 smithi191 ceph-mon[32884]: 2.17 scrub ok 2024-09-10T19:28:13.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:28:13 smithi191 ceph-mon[32884]: 4.c scrub starts 2024-09-10T19:28:13.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:28:13 smithi191 ceph-mon[32884]: 4.c scrub ok 2024-09-10T19:28:13.473 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:28:13.507 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:28:13.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:28:13.508 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:28:13.612 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:28:13.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:28:13.648 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:28:13.649 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:28:13.749 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:28:13.792 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:28:13.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:28:13.793 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:28:13.894 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:28:13.936 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:28:13.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:28:13.937 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:28:14.038 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:28:14.061 INFO:tasks.radosbench.radosbench.0.smithi183.stdout:2024-09-10T19:28:14.060349+0000 min lat: 0.00134808 max lat: 0.145052 avg lat: 0.00671453 2024-09-10T19:28:14.061 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-10T19:28:14.061 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: 20 16 47615 47599 148.717 104.938 0.00425718 0.00671453 2024-09-10T19:28:14.064 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:28:14.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:28:14.065 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:28:14.165 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:28:14.202 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:28:14.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:28:14.203 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:28:14.304 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:28:14.336 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:28:14.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:28:14.337 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:28:14.438 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:28:14.474 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:28:14.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:28:14.475 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:28:14.576 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:28:14.613 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:28:14.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:28:14.614 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:28:14.715 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:28:14.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:28:14.750 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:28:14.751 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:28:14.851 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:28:14.880 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:28:14.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:28:14.881 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:28:14.966 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:28:14 smithi191 ceph-mon[32884]: 4.8 scrub starts 2024-09-10T19:28:14.966 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:28:14 smithi191 ceph-mon[32884]: 4.8 scrub ok 2024-09-10T19:28:14.966 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:28:14 smithi191 ceph-mon[32884]: 6.c scrub starts 2024-09-10T19:28:14.966 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:28:14 smithi191 ceph-mon[32884]: 6.c scrub ok 2024-09-10T19:28:14.966 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:28:14 smithi191 ceph-mon[32884]: from='mgr.14556 ' entity='mgr.y' 2024-09-10T19:28:14.966 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:28:14 smithi191 ceph-mon[32884]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:28:14.966 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:28:14 smithi191 ceph-mon[32884]: pgmap v836: 158 pgs: 158 active+clean; 2.6 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 1.3 KiB/s rd, 92 MiB/s wr, 1.47k op/s 2024-09-10T19:28:14.982 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:28:15.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:28:15.008 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:28:15.009 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:28:15.061 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: 21 15 49345 49330 146.787 108.188 0.00484555 0.00680816 2024-09-10T19:28:15.110 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:28:15.147 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:28:15.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:28:15.148 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:28:15.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:28:14 smithi183 ceph-mon[29736]: 4.8 scrub starts 2024-09-10T19:28:15.182 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:28:14 smithi183 ceph-mon[29736]: 4.8 scrub ok 2024-09-10T19:28:15.182 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:28:14 smithi183 ceph-mon[29736]: 6.c scrub starts 2024-09-10T19:28:15.182 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:28:14 smithi183 ceph-mon[29736]: 6.c scrub ok 2024-09-10T19:28:15.182 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:28:14 smithi183 ceph-mon[29736]: from='mgr.14556 ' entity='mgr.y' 2024-09-10T19:28:15.182 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:28:14 smithi183 ceph-mon[29736]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:28:15.182 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:28:14 smithi183 ceph-mon[29736]: pgmap v836: 158 pgs: 158 active+clean; 2.6 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 1.3 KiB/s rd, 92 MiB/s wr, 1.47k op/s 2024-09-10T19:28:15.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:28:14 smithi183 ceph-mon[41562]: 4.8 scrub starts 2024-09-10T19:28:15.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:28:14 smithi183 ceph-mon[41562]: 4.8 scrub ok 2024-09-10T19:28:15.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:28:14 smithi183 ceph-mon[41562]: 6.c scrub starts 2024-09-10T19:28:15.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:28:14 smithi183 ceph-mon[41562]: 6.c scrub ok 2024-09-10T19:28:15.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:28:14 smithi183 ceph-mon[41562]: from='mgr.14556 ' entity='mgr.y' 2024-09-10T19:28:15.183 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:28:14 smithi183 ceph-mon[41562]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:28:15.183 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:28:14 smithi183 ceph-mon[41562]: pgmap v836: 158 pgs: 158 active+clean; 2.6 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 1.3 KiB/s rd, 92 MiB/s wr, 1.47k op/s 2024-09-10T19:28:15.249 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:28:15.280 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:28:15.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:28:15.281 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:28:15.382 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:28:15.419 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:28:15.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:28:15.420 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:28:15.520 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:28:15.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:28:15.549 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:28:15.549 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:28:15.650 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:28:15.671 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:28:15.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:28:15.673 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:28:15.773 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:28:15.806 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:28:15.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:28:15.807 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:28:15.908 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:28:15.944 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:28:15.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:28:15.945 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:28:16.046 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:28:16.061 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: 22 16 50817 50801 144.294 91.9375 0.00440696 0.00692566 2024-09-10T19:28:16.082 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:28:16.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:28:16.083 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:28:16.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:28:15 smithi183 ceph-mon[29736]: osdmap e382: 8 total, 8 up, 7 in 2024-09-10T19:28:16.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:28:15 smithi183 ceph-mon[29736]: 2.e scrub starts 2024-09-10T19:28:16.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:28:15 smithi183 ceph-mon[29736]: 2.e scrub ok 2024-09-10T19:28:16.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:28:15 smithi183 ceph-mon[29736]: 2.6 deep-scrub starts 2024-09-10T19:28:16.182 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:28:15 smithi183 ceph-mon[29736]: 2.6 deep-scrub ok 2024-09-10T19:28:16.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:28:15 smithi183 ceph-mon[41562]: osdmap e382: 8 total, 8 up, 7 in 2024-09-10T19:28:16.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:28:15 smithi183 ceph-mon[41562]: 2.e scrub starts 2024-09-10T19:28:16.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:28:15 smithi183 ceph-mon[41562]: 2.e scrub ok 2024-09-10T19:28:16.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:28:15 smithi183 ceph-mon[41562]: 2.6 deep-scrub starts 2024-09-10T19:28:16.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:28:15 smithi183 ceph-mon[41562]: 2.6 deep-scrub ok 2024-09-10T19:28:16.183 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:28:16.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:28:15 smithi191 ceph-mon[32884]: osdmap e382: 8 total, 8 up, 7 in 2024-09-10T19:28:16.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:28:15 smithi191 ceph-mon[32884]: 2.e scrub starts 2024-09-10T19:28:16.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:28:15 smithi191 ceph-mon[32884]: 2.e scrub ok 2024-09-10T19:28:16.217 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:28:15 smithi191 ceph-mon[32884]: 2.6 deep-scrub starts 2024-09-10T19:28:16.217 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:28:15 smithi191 ceph-mon[32884]: 2.6 deep-scrub ok 2024-09-10T19:28:16.220 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:28:16.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:28:16.221 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:28:16.322 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:28:16.358 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:28:16.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:28:16.359 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:28:16.460 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:28:16.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:28:16.500 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:28:16.501 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:28:16.602 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:28:16.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:28:16.641 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:28:16.641 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:28:16.743 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:28:16.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:28:16.778 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:28:16.778 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:28:16.879 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:28:16.912 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:28:16.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:28:16.914 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:28:17.014 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:28:17.048 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:28:17.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:28:17.049 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:28:17.061 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: 23 16 52606 52590 142.881 111.812 0.00859872 0.00699404 2024-09-10T19:28:17.149 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:28:17.189 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:28:17.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:28:17.190 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:28:17.196 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:28:17.253 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:28:16 smithi183 ceph-mon[29736]: purged_snaps scrub starts 2024-09-10T19:28:17.253 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:28:16 smithi183 ceph-mon[29736]: purged_snaps scrub ok 2024-09-10T19:28:17.254 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:28:16 smithi183 ceph-mon[29736]: 5.6 scrub starts 2024-09-10T19:28:17.254 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:28:16 smithi183 ceph-mon[29736]: pgmap v838: 158 pgs: 158 active+clean; 3.0 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 653 B/s rd, 129 MiB/s wr, 2.07k op/s 2024-09-10T19:28:17.254 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:28:16 smithi183 ceph-mon[29736]: 5.6 scrub ok 2024-09-10T19:28:17.254 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:28:16 smithi183 ceph-mon[29736]: osdmap e383: 8 total, 8 up, 7 in 2024-09-10T19:28:17.254 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:28:16 smithi183 ceph-mon[29736]: 2.a scrub starts 2024-09-10T19:28:17.254 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:28:16 smithi183 ceph-mon[29736]: 2.a scrub ok 2024-09-10T19:28:17.254 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:28:16 smithi183 ceph-mon[29736]: 2.b scrub starts 2024-09-10T19:28:17.254 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:28:16 smithi183 ceph-mon[29736]: 2.b scrub ok 2024-09-10T19:28:17.254 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:28:16 smithi183 ceph-mon[41562]: purged_snaps scrub starts 2024-09-10T19:28:17.255 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:28:16 smithi183 ceph-mon[41562]: purged_snaps scrub ok 2024-09-10T19:28:17.255 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:28:16 smithi183 ceph-mon[41562]: 5.6 scrub starts 2024-09-10T19:28:17.255 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:28:16 smithi183 ceph-mon[41562]: pgmap v838: 158 pgs: 158 active+clean; 3.0 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 653 B/s rd, 129 MiB/s wr, 2.07k op/s 2024-09-10T19:28:17.255 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:28:16 smithi183 ceph-mon[41562]: 5.6 scrub ok 2024-09-10T19:28:17.255 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:28:16 smithi183 ceph-mon[41562]: osdmap e383: 8 total, 8 up, 7 in 2024-09-10T19:28:17.255 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:28:16 smithi183 ceph-mon[41562]: 2.a scrub starts 2024-09-10T19:28:17.255 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:28:16 smithi183 ceph-mon[41562]: 2.a scrub ok 2024-09-10T19:28:17.255 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:28:16 smithi183 ceph-mon[41562]: 2.b scrub starts 2024-09-10T19:28:17.255 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:28:16 smithi183 ceph-mon[41562]: 2.b scrub ok 2024-09-10T19:28:17.255 INFO:teuthology.orchestra.run.smithi183.stdout:{ 2024-09-10T19:28:17.256 INFO:teuthology.orchestra.run.smithi183.stdout: "ops": [], 2024-09-10T19:28:17.256 INFO:teuthology.orchestra.run.smithi183.stdout: "complaint_time": 30, 2024-09-10T19:28:17.256 INFO:teuthology.orchestra.run.smithi183.stdout: "num_blocked_ops": 0 2024-09-10T19:28:17.256 INFO:teuthology.orchestra.run.smithi183.stdout:} 2024-09-10T19:28:17.292 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:28:17.329 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:28:17.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:28:17.330 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:28:17.431 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:28:17.451 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:28:16 smithi191 ceph-mon[32884]: purged_snaps scrub starts 2024-09-10T19:28:17.452 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:28:16 smithi191 ceph-mon[32884]: purged_snaps scrub ok 2024-09-10T19:28:17.452 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:28:16 smithi191 ceph-mon[32884]: 5.6 scrub starts 2024-09-10T19:28:17.452 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:28:16 smithi191 ceph-mon[32884]: pgmap v838: 158 pgs: 158 active+clean; 3.0 GiB data, 14 GiB used, 612 GiB / 626 GiB avail; 653 B/s rd, 129 MiB/s wr, 2.07k op/s 2024-09-10T19:28:17.452 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:28:16 smithi191 ceph-mon[32884]: 5.6 scrub ok 2024-09-10T19:28:17.452 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:28:16 smithi191 ceph-mon[32884]: osdmap e383: 8 total, 8 up, 7 in 2024-09-10T19:28:17.452 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:28:16 smithi191 ceph-mon[32884]: 2.a scrub starts 2024-09-10T19:28:17.453 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:28:16 smithi191 ceph-mon[32884]: 2.a scrub ok 2024-09-10T19:28:17.453 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:28:16 smithi191 ceph-mon[32884]: 2.b scrub starts 2024-09-10T19:28:17.453 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:28:16 smithi191 ceph-mon[32884]: 2.b scrub ok 2024-09-10T19:28:17.470 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:28:17.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:28:17.471 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:28:17.572 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:28:17.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:28:17.608 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:28:17.609 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:28:17.709 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:28:17.715 INFO:journalctl@ceph.mgr.x.smithi191.stdout:Sep 10 19:28:17 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-x[34967]: ::ffff:172.21.15.191 - - [10/Sep/2024:19:28:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T19:28:17.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:28:17.746 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:28:17.747 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:28:17.802 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph osd set noscrub 2024-09-10T19:28:17.848 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:28:17.895 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:28:17.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:28:17.896 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:28:17.997 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:28:18.029 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:28:18.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:28:18.030 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:28:18.061 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: 24 16 54298 54282 141.334 105.75 0.00685914 0.0070708 2024-09-10T19:28:18.131 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:28:18.168 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:28:18.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:28:18.170 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:28:18.270 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:28:18.303 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:28:18.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:28:18.304 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:28:18.353 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph daemon osd.1 dump_historic_ops 2024-09-10T19:28:18.405 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:28:18.441 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:28:18.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:28:18.443 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:28:18.543 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:28:18.581 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:28:18.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:28:18.582 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:28:18.683 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:28:18.713 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-10T19:28:18.720 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:28:18.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:28:18.721 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:28:18.822 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:28:18.871 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:28:18.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:28:18.873 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:28:18.973 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:28:19.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:28:19.028 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:28:19.029 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:28:19.061 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: 25 16 55962 55946 139.84 104 0.00839597 0.00714634 2024-09-10T19:28:19.129 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:28:19.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:28:19.167 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:28:19.168 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:28:19.268 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:28:19.303 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:28:19.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:28:19.304 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:28:19.404 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:28:19.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:28:18 smithi183 ceph-mon[29736]: 5.3 deep-scrub starts 2024-09-10T19:28:19.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:28:18 smithi183 ceph-mon[29736]: 5.3 deep-scrub ok 2024-09-10T19:28:19.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:28:18 smithi183 ceph-mon[29736]: 5.16 scrub starts 2024-09-10T19:28:19.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:28:18 smithi183 ceph-mon[29736]: 5.16 scrub ok 2024-09-10T19:28:19.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:28:18 smithi183 ceph-mon[29736]: pgmap v840: 158 pgs: 158 active+clean; 3.1 GiB data, 14 GiB used, 611 GiB / 626 GiB avail; 1.3 KiB/s rd, 129 MiB/s wr, 2.07k op/s 2024-09-10T19:28:19.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:28:18 smithi183 ceph-mon[41562]: 5.3 deep-scrub starts 2024-09-10T19:28:19.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:28:18 smithi183 ceph-mon[41562]: 5.3 deep-scrub ok 2024-09-10T19:28:19.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:28:18 smithi183 ceph-mon[41562]: 5.16 scrub starts 2024-09-10T19:28:19.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:28:18 smithi183 ceph-mon[41562]: 5.16 scrub ok 2024-09-10T19:28:19.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:28:18 smithi183 ceph-mon[41562]: pgmap v840: 158 pgs: 158 active+clean; 3.1 GiB data, 14 GiB used, 611 GiB / 626 GiB avail; 1.3 KiB/s rd, 129 MiB/s wr, 2.07k op/s 2024-09-10T19:28:19.441 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:28:19.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:28:19.442 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:28:19.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:28:18 smithi191 ceph-mon[32884]: 5.3 deep-scrub starts 2024-09-10T19:28:19.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:28:18 smithi191 ceph-mon[32884]: 5.3 deep-scrub ok 2024-09-10T19:28:19.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:28:18 smithi191 ceph-mon[32884]: 5.16 scrub starts 2024-09-10T19:28:19.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:28:18 smithi191 ceph-mon[32884]: 5.16 scrub ok 2024-09-10T19:28:19.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:28:18 smithi191 ceph-mon[32884]: pgmap v840: 158 pgs: 158 active+clean; 3.1 GiB data, 14 GiB used, 611 GiB / 626 GiB avail; 1.3 KiB/s rd, 129 MiB/s wr, 2.07k op/s 2024-09-10T19:28:19.543 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:28:19.579 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:28:19.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:28:19.581 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:28:19.681 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:28:19.717 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:28:19.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:28:19.718 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:28:19.819 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:28:19.850 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:28:19.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:28:19.851 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:28:19.952 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:28:19.992 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:28:19.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:28:19.993 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:28:20.061 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: 26 16 57636 57620 138.485 104.625 0.00775495 0.00721609 2024-09-10T19:28:20.094 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:28:20.128 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:28:20.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:28:20.129 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:28:20.175 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:28:20.215 INFO:journalctl@ceph.iscsi.iscsi.a.smithi191.stdout:Sep 10 19:28:19 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-iscsi-iscsi-a[61816]: debug there is no tcmu-runner data available 2024-09-10T19:28:20.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:28:19 smithi191 ceph-mon[32884]: 6.1b deep-scrub starts 2024-09-10T19:28:20.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:28:19 smithi191 ceph-mon[32884]: 6.1b deep-scrub ok 2024-09-10T19:28:20.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:28:19 smithi191 ceph-mon[32884]: 1.3 scrub starts 2024-09-10T19:28:20.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:28:19 smithi191 ceph-mon[32884]: 1.3 scrub ok 2024-09-10T19:28:20.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:28:19 smithi191 ceph-mon[32884]: osdmap e384: 8 total, 8 up, 7 in 2024-09-10T19:28:20.230 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:28:20.232 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:28:20.265 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:28:20.289 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:28:20.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:28:20.290 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:28:20.335 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:28:19 smithi183 ceph-mon[29736]: 6.1b deep-scrub starts 2024-09-10T19:28:20.335 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:28:19 smithi183 ceph-mon[29736]: 6.1b deep-scrub ok 2024-09-10T19:28:20.335 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:28:19 smithi183 ceph-mon[29736]: 1.3 scrub starts 2024-09-10T19:28:20.335 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:28:19 smithi183 ceph-mon[29736]: 1.3 scrub ok 2024-09-10T19:28:20.335 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:28:19 smithi183 ceph-mon[29736]: osdmap e384: 8 total, 8 up, 7 in 2024-09-10T19:28:20.336 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:28:19 smithi183 ceph-mon[41562]: 6.1b deep-scrub starts 2024-09-10T19:28:20.336 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:28:19 smithi183 ceph-mon[41562]: 6.1b deep-scrub ok 2024-09-10T19:28:20.336 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:28:19 smithi183 ceph-mon[41562]: 1.3 scrub starts 2024-09-10T19:28:20.336 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:28:19 smithi183 ceph-mon[41562]: 1.3 scrub ok 2024-09-10T19:28:20.336 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:28:19 smithi183 ceph-mon[41562]: osdmap e384: 8 total, 8 up, 7 in 2024-09-10T19:28:20.390 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:28:20.425 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:28:20.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:28:20.426 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:28:20.527 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:28:20.564 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:28:20.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:28:20.565 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:28:20.666 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:28:20.699 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:28:20.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:28:20.700 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:28:20.801 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:28:20.833 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:28:20.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:28:20.834 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:28:20.935 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:28:20.973 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:28:20.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:28:20.974 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:28:21.061 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: 27 16 59188 59172 136.949 97 0.00977543 0.00729723 2024-09-10T19:28:21.075 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:28:21.110 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:28:21.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:28:21.111 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:28:21.143 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:28:20 smithi183 ceph-mon[29736]: 2.1b scrub starts 2024-09-10T19:28:21.143 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:28:20 smithi183 ceph-mon[29736]: 2.1b scrub ok 2024-09-10T19:28:21.143 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:28:20 smithi183 ceph-mon[29736]: 5.1e deep-scrub starts 2024-09-10T19:28:21.143 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:28:20 smithi183 ceph-mon[29736]: 5.1e deep-scrub ok 2024-09-10T19:28:21.144 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:28:20 smithi183 ceph-mon[29736]: pgmap v841: 158 pgs: 158 active+clean; 3.1 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 1.2 KiB/s rd, 86 MiB/s wr, 1.38k op/s 2024-09-10T19:28:21.144 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:28:20 smithi183 ceph-mon[29736]: from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:28:21.144 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:28:20 smithi183 ceph-mon[29736]: 6.1d deep-scrub starts 2024-09-10T19:28:21.144 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:28:20 smithi183 ceph-mon[29736]: 6.1d deep-scrub ok 2024-09-10T19:28:21.144 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:28:20 smithi183 ceph-mon[41562]: 2.1b scrub starts 2024-09-10T19:28:21.144 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:28:20 smithi183 ceph-mon[41562]: 2.1b scrub ok 2024-09-10T19:28:21.144 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:28:20 smithi183 ceph-mon[41562]: 5.1e deep-scrub starts 2024-09-10T19:28:21.144 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:28:20 smithi183 ceph-mon[41562]: 5.1e deep-scrub ok 2024-09-10T19:28:21.144 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:28:20 smithi183 ceph-mon[41562]: pgmap v841: 158 pgs: 158 active+clean; 3.1 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 1.2 KiB/s rd, 86 MiB/s wr, 1.38k op/s 2024-09-10T19:28:21.144 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:28:20 smithi183 ceph-mon[41562]: from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:28:21.144 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:28:20 smithi183 ceph-mon[41562]: 6.1d deep-scrub starts 2024-09-10T19:28:21.145 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:28:20 smithi183 ceph-mon[41562]: 6.1d deep-scrub ok 2024-09-10T19:28:21.212 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:28:21.251 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:28:21.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:28:21.252 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:28:21.353 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:28:21.383 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:28:21.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:28:21.384 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:28:21.431 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Sep 10 19:28:21 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-y[29989]: ::ffff:172.21.15.191 - - [10/Sep/2024:19:28:21] "GET /metrics HTTP/1.1" 200 224852 "" "Prometheus/2.43.0" 2024-09-10T19:28:21.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:28:20 smithi191 ceph-mon[32884]: 2.1b scrub starts 2024-09-10T19:28:21.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:28:20 smithi191 ceph-mon[32884]: 2.1b scrub ok 2024-09-10T19:28:21.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:28:20 smithi191 ceph-mon[32884]: 5.1e deep-scrub starts 2024-09-10T19:28:21.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:28:20 smithi191 ceph-mon[32884]: 5.1e deep-scrub ok 2024-09-10T19:28:21.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:28:20 smithi191 ceph-mon[32884]: pgmap v841: 158 pgs: 158 active+clean; 3.1 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 1.2 KiB/s rd, 86 MiB/s wr, 1.38k op/s 2024-09-10T19:28:21.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:28:20 smithi191 ceph-mon[32884]: from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:28:21.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:28:20 smithi191 ceph-mon[32884]: 6.1d deep-scrub starts 2024-09-10T19:28:21.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:28:20 smithi191 ceph-mon[32884]: 6.1d deep-scrub ok 2024-09-10T19:28:21.485 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:28:21.527 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:28:21.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:28:21.528 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:28:21.629 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:28:21.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:28:21.669 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:28:21.670 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:28:21.726 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-09-10T19:28:21.727 INFO:teuthology.orchestra.run.smithi183.stdout:{ 2024-09-10T19:28:21.727 INFO:teuthology.orchestra.run.smithi183.stdout: "epoch": 384, 2024-09-10T19:28:21.727 INFO:teuthology.orchestra.run.smithi183.stdout: "fsid": "320f6df8-6fa7-11ef-bcea-c7b262605968", 2024-09-10T19:28:21.727 INFO:teuthology.orchestra.run.smithi183.stdout: "created": "2024-09-10T19:05:03.688531+0000", 2024-09-10T19:28:21.727 INFO:teuthology.orchestra.run.smithi183.stdout: "modified": "2024-09-10T19:28:19.941070+0000", 2024-09-10T19:28:21.727 INFO:teuthology.orchestra.run.smithi183.stdout: "last_up_change": "2024-09-10T19:24:22.870873+0000", 2024-09-10T19:28:21.728 INFO:teuthology.orchestra.run.smithi183.stdout: "last_in_change": "2024-09-10T19:27:51.975891+0000", 2024-09-10T19:28:21.728 INFO:teuthology.orchestra.run.smithi183.stdout: "flags": "sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-09-10T19:28:21.728 INFO:teuthology.orchestra.run.smithi183.stdout: "flags_num": 5799936, 2024-09-10T19:28:21.728 INFO:teuthology.orchestra.run.smithi183.stdout: "flags_set": [ 2024-09-10T19:28:21.728 INFO:teuthology.orchestra.run.smithi183.stdout: "pglog_hardlimit", 2024-09-10T19:28:21.728 INFO:teuthology.orchestra.run.smithi183.stdout: "purged_snapdirs", 2024-09-10T19:28:21.728 INFO:teuthology.orchestra.run.smithi183.stdout: "recovery_deletes", 2024-09-10T19:28:21.728 INFO:teuthology.orchestra.run.smithi183.stdout: "sortbitwise" 2024-09-10T19:28:21.728 INFO:teuthology.orchestra.run.smithi183.stdout: ], 2024-09-10T19:28:21.728 INFO:teuthology.orchestra.run.smithi183.stdout: "crush_version": 21, 2024-09-10T19:28:21.728 INFO:teuthology.orchestra.run.smithi183.stdout: "full_ratio": 0.94999998807907104, 2024-09-10T19:28:21.728 INFO:teuthology.orchestra.run.smithi183.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-09-10T19:28:21.728 INFO:teuthology.orchestra.run.smithi183.stdout: "nearfull_ratio": 0.85000002384185791, 2024-09-10T19:28:21.729 INFO:teuthology.orchestra.run.smithi183.stdout: "cluster_snapshot": "", 2024-09-10T19:28:21.729 INFO:teuthology.orchestra.run.smithi183.stdout: "pool_max": 11, 2024-09-10T19:28:21.729 INFO:teuthology.orchestra.run.smithi183.stdout: "max_osd": 8, 2024-09-10T19:28:21.729 INFO:teuthology.orchestra.run.smithi183.stdout: "require_min_compat_client": "luminous", 2024-09-10T19:28:21.729 INFO:teuthology.orchestra.run.smithi183.stdout: "min_compat_client": "luminous", 2024-09-10T19:28:21.729 INFO:teuthology.orchestra.run.smithi183.stdout: "require_osd_release": "quincy", 2024-09-10T19:28:21.729 INFO:teuthology.orchestra.run.smithi183.stdout: "pools": [ 2024-09-10T19:28:21.729 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:28:21.729 INFO:teuthology.orchestra.run.smithi183.stdout: "pool": 1, 2024-09-10T19:28:21.729 INFO:teuthology.orchestra.run.smithi183.stdout: "pool_name": ".mgr", 2024-09-10T19:28:21.729 INFO:teuthology.orchestra.run.smithi183.stdout: "create_time": "2024-09-10T19:08:00.456786+0000", 2024-09-10T19:28:21.729 INFO:teuthology.orchestra.run.smithi183.stdout: "flags": 1, 2024-09-10T19:28:21.729 INFO:teuthology.orchestra.run.smithi183.stdout: "flags_names": "hashpspool", 2024-09-10T19:28:21.730 INFO:teuthology.orchestra.run.smithi183.stdout: "type": 1, 2024-09-10T19:28:21.730 INFO:teuthology.orchestra.run.smithi183.stdout: "size": 3, 2024-09-10T19:28:21.730 INFO:teuthology.orchestra.run.smithi183.stdout: "min_size": 2, 2024-09-10T19:28:21.730 INFO:teuthology.orchestra.run.smithi183.stdout: "crush_rule": 0, 2024-09-10T19:28:21.730 INFO:teuthology.orchestra.run.smithi183.stdout: "peering_crush_bucket_count": 0, 2024-09-10T19:28:21.730 INFO:teuthology.orchestra.run.smithi183.stdout: "peering_crush_bucket_target": 0, 2024-09-10T19:28:21.730 INFO:teuthology.orchestra.run.smithi183.stdout: "peering_crush_bucket_barrier": 0, 2024-09-10T19:28:21.730 INFO:teuthology.orchestra.run.smithi183.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-10T19:28:21.730 INFO:teuthology.orchestra.run.smithi183.stdout: "object_hash": 2, 2024-09-10T19:28:21.730 INFO:teuthology.orchestra.run.smithi183.stdout: "pg_autoscale_mode": "off", 2024-09-10T19:28:21.730 INFO:teuthology.orchestra.run.smithi183.stdout: "pg_num": 11, 2024-09-10T19:28:21.730 INFO:teuthology.orchestra.run.smithi183.stdout: "pg_placement_num": 11, 2024-09-10T19:28:21.730 INFO:teuthology.orchestra.run.smithi183.stdout: "pg_placement_num_target": 11, 2024-09-10T19:28:21.730 INFO:teuthology.orchestra.run.smithi183.stdout: "pg_num_target": 11, 2024-09-10T19:28:21.731 INFO:teuthology.orchestra.run.smithi183.stdout: "pg_num_pending": 11, 2024-09-10T19:28:21.731 INFO:teuthology.orchestra.run.smithi183.stdout: "last_pg_merge_meta": { 2024-09-10T19:28:21.731 INFO:teuthology.orchestra.run.smithi183.stdout: "source_pgid": "0.0", 2024-09-10T19:28:21.731 INFO:teuthology.orchestra.run.smithi183.stdout: "ready_epoch": 0, 2024-09-10T19:28:21.731 INFO:teuthology.orchestra.run.smithi183.stdout: "last_epoch_started": 0, 2024-09-10T19:28:21.731 INFO:teuthology.orchestra.run.smithi183.stdout: "last_epoch_clean": 0, 2024-09-10T19:28:21.731 INFO:teuthology.orchestra.run.smithi183.stdout: "source_version": "0'0", 2024-09-10T19:28:21.731 INFO:teuthology.orchestra.run.smithi183.stdout: "target_version": "0'0" 2024-09-10T19:28:21.731 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:28:21.731 INFO:teuthology.orchestra.run.smithi183.stdout: "last_change": "271", 2024-09-10T19:28:21.731 INFO:teuthology.orchestra.run.smithi183.stdout: "last_force_op_resend": "0", 2024-09-10T19:28:21.731 INFO:teuthology.orchestra.run.smithi183.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-10T19:28:21.731 INFO:teuthology.orchestra.run.smithi183.stdout: "last_force_op_resend_preluminous": "242", 2024-09-10T19:28:21.731 INFO:teuthology.orchestra.run.smithi183.stdout: "auid": 0, 2024-09-10T19:28:21.732 INFO:teuthology.orchestra.run.smithi183.stdout: "snap_mode": "selfmanaged", 2024-09-10T19:28:21.732 INFO:teuthology.orchestra.run.smithi183.stdout: "snap_seq": 0, 2024-09-10T19:28:21.732 INFO:teuthology.orchestra.run.smithi183.stdout: "snap_epoch": 0, 2024-09-10T19:28:21.732 INFO:teuthology.orchestra.run.smithi183.stdout: "pool_snaps": [], 2024-09-10T19:28:21.732 INFO:teuthology.orchestra.run.smithi183.stdout: "removed_snaps": "[]", 2024-09-10T19:28:21.732 INFO:teuthology.orchestra.run.smithi183.stdout: "quota_max_bytes": 0, 2024-09-10T19:28:21.732 INFO:teuthology.orchestra.run.smithi183.stdout: "quota_max_objects": 0, 2024-09-10T19:28:21.732 INFO:teuthology.orchestra.run.smithi183.stdout: "tiers": [], 2024-09-10T19:28:21.732 INFO:teuthology.orchestra.run.smithi183.stdout: "tier_of": -1, 2024-09-10T19:28:21.732 INFO:teuthology.orchestra.run.smithi183.stdout: "read_tier": -1, 2024-09-10T19:28:21.732 INFO:teuthology.orchestra.run.smithi183.stdout: "write_tier": -1, 2024-09-10T19:28:21.732 INFO:teuthology.orchestra.run.smithi183.stdout: "cache_mode": "none", 2024-09-10T19:28:21.732 INFO:teuthology.orchestra.run.smithi183.stdout: "target_max_bytes": 0, 2024-09-10T19:28:21.733 INFO:teuthology.orchestra.run.smithi183.stdout: "target_max_objects": 0, 2024-09-10T19:28:21.733 INFO:teuthology.orchestra.run.smithi183.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-10T19:28:21.733 INFO:teuthology.orchestra.run.smithi183.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-10T19:28:21.733 INFO:teuthology.orchestra.run.smithi183.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-10T19:28:21.733 INFO:teuthology.orchestra.run.smithi183.stdout: "cache_min_flush_age": 0, 2024-09-10T19:28:21.733 INFO:teuthology.orchestra.run.smithi183.stdout: "cache_min_evict_age": 0, 2024-09-10T19:28:21.733 INFO:teuthology.orchestra.run.smithi183.stdout: "erasure_code_profile": "", 2024-09-10T19:28:21.733 INFO:teuthology.orchestra.run.smithi183.stdout: "hit_set_params": { 2024-09-10T19:28:21.733 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "none" 2024-09-10T19:28:21.733 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:28:21.733 INFO:teuthology.orchestra.run.smithi183.stdout: "hit_set_period": 0, 2024-09-10T19:28:21.733 INFO:teuthology.orchestra.run.smithi183.stdout: "hit_set_count": 0, 2024-09-10T19:28:21.733 INFO:teuthology.orchestra.run.smithi183.stdout: "use_gmt_hitset": true, 2024-09-10T19:28:21.734 INFO:teuthology.orchestra.run.smithi183.stdout: "min_read_recency_for_promote": 0, 2024-09-10T19:28:21.734 INFO:teuthology.orchestra.run.smithi183.stdout: "min_write_recency_for_promote": 0, 2024-09-10T19:28:21.734 INFO:teuthology.orchestra.run.smithi183.stdout: "hit_set_grade_decay_rate": 0, 2024-09-10T19:28:21.734 INFO:teuthology.orchestra.run.smithi183.stdout: "hit_set_search_last_n": 0, 2024-09-10T19:28:21.734 INFO:teuthology.orchestra.run.smithi183.stdout: "grade_table": [], 2024-09-10T19:28:21.734 INFO:teuthology.orchestra.run.smithi183.stdout: "stripe_width": 0, 2024-09-10T19:28:21.734 INFO:teuthology.orchestra.run.smithi183.stdout: "expected_num_objects": 0, 2024-09-10T19:28:21.734 INFO:teuthology.orchestra.run.smithi183.stdout: "fast_read": false, 2024-09-10T19:28:21.734 INFO:teuthology.orchestra.run.smithi183.stdout: "options": { 2024-09-10T19:28:21.734 INFO:teuthology.orchestra.run.smithi183.stdout: "pg_num_max": 32, 2024-09-10T19:28:21.734 INFO:teuthology.orchestra.run.smithi183.stdout: "pg_num_min": 1 2024-09-10T19:28:21.734 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:28:21.734 INFO:teuthology.orchestra.run.smithi183.stdout: "application_metadata": { 2024-09-10T19:28:21.734 INFO:teuthology.orchestra.run.smithi183.stdout: "mgr": {} 2024-09-10T19:28:21.735 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:28:21.735 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:28:21.735 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:28:21.735 INFO:teuthology.orchestra.run.smithi183.stdout: "pool": 2, 2024-09-10T19:28:21.735 INFO:teuthology.orchestra.run.smithi183.stdout: "pool_name": ".rgw.root", 2024-09-10T19:28:21.735 INFO:teuthology.orchestra.run.smithi183.stdout: "create_time": "2024-09-10T19:10:05.993763+0000", 2024-09-10T19:28:21.735 INFO:teuthology.orchestra.run.smithi183.stdout: "flags": 1, 2024-09-10T19:28:21.735 INFO:teuthology.orchestra.run.smithi183.stdout: "flags_names": "hashpspool", 2024-09-10T19:28:21.735 INFO:teuthology.orchestra.run.smithi183.stdout: "type": 1, 2024-09-10T19:28:21.735 INFO:teuthology.orchestra.run.smithi183.stdout: "size": 3, 2024-09-10T19:28:21.735 INFO:teuthology.orchestra.run.smithi183.stdout: "min_size": 2, 2024-09-10T19:28:21.735 INFO:teuthology.orchestra.run.smithi183.stdout: "crush_rule": 0, 2024-09-10T19:28:21.735 INFO:teuthology.orchestra.run.smithi183.stdout: "peering_crush_bucket_count": 0, 2024-09-10T19:28:21.736 INFO:teuthology.orchestra.run.smithi183.stdout: "peering_crush_bucket_target": 0, 2024-09-10T19:28:21.736 INFO:teuthology.orchestra.run.smithi183.stdout: "peering_crush_bucket_barrier": 0, 2024-09-10T19:28:21.736 INFO:teuthology.orchestra.run.smithi183.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-10T19:28:21.736 INFO:teuthology.orchestra.run.smithi183.stdout: "object_hash": 2, 2024-09-10T19:28:21.736 INFO:teuthology.orchestra.run.smithi183.stdout: "pg_autoscale_mode": "off", 2024-09-10T19:28:21.736 INFO:teuthology.orchestra.run.smithi183.stdout: "pg_num": 32, 2024-09-10T19:28:21.736 INFO:teuthology.orchestra.run.smithi183.stdout: "pg_placement_num": 32, 2024-09-10T19:28:21.736 INFO:teuthology.orchestra.run.smithi183.stdout: "pg_placement_num_target": 32, 2024-09-10T19:28:21.736 INFO:teuthology.orchestra.run.smithi183.stdout: "pg_num_target": 32, 2024-09-10T19:28:21.736 INFO:teuthology.orchestra.run.smithi183.stdout: "pg_num_pending": 32, 2024-09-10T19:28:21.736 INFO:teuthology.orchestra.run.smithi183.stdout: "last_pg_merge_meta": { 2024-09-10T19:28:21.736 INFO:teuthology.orchestra.run.smithi183.stdout: "source_pgid": "0.0", 2024-09-10T19:28:21.736 INFO:teuthology.orchestra.run.smithi183.stdout: "ready_epoch": 0, 2024-09-10T19:28:21.737 INFO:teuthology.orchestra.run.smithi183.stdout: "last_epoch_started": 0, 2024-09-10T19:28:21.737 INFO:teuthology.orchestra.run.smithi183.stdout: "last_epoch_clean": 0, 2024-09-10T19:28:21.737 INFO:teuthology.orchestra.run.smithi183.stdout: "source_version": "0'0", 2024-09-10T19:28:21.737 INFO:teuthology.orchestra.run.smithi183.stdout: "target_version": "0'0" 2024-09-10T19:28:21.737 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:28:21.737 INFO:teuthology.orchestra.run.smithi183.stdout: "last_change": "60", 2024-09-10T19:28:21.737 INFO:teuthology.orchestra.run.smithi183.stdout: "last_force_op_resend": "0", 2024-09-10T19:28:21.737 INFO:teuthology.orchestra.run.smithi183.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-10T19:28:21.737 INFO:teuthology.orchestra.run.smithi183.stdout: "last_force_op_resend_preluminous": "0", 2024-09-10T19:28:21.737 INFO:teuthology.orchestra.run.smithi183.stdout: "auid": 0, 2024-09-10T19:28:21.737 INFO:teuthology.orchestra.run.smithi183.stdout: "snap_mode": "selfmanaged", 2024-09-10T19:28:21.738 INFO:teuthology.orchestra.run.smithi183.stdout: "snap_seq": 0, 2024-09-10T19:28:21.738 INFO:teuthology.orchestra.run.smithi183.stdout: "snap_epoch": 0, 2024-09-10T19:28:21.738 INFO:teuthology.orchestra.run.smithi183.stdout: "pool_snaps": [], 2024-09-10T19:28:21.738 INFO:teuthology.orchestra.run.smithi183.stdout: "removed_snaps": "[]", 2024-09-10T19:28:21.738 INFO:teuthology.orchestra.run.smithi183.stdout: "quota_max_bytes": 0, 2024-09-10T19:28:21.738 INFO:teuthology.orchestra.run.smithi183.stdout: "quota_max_objects": 0, 2024-09-10T19:28:21.738 INFO:teuthology.orchestra.run.smithi183.stdout: "tiers": [], 2024-09-10T19:28:21.738 INFO:teuthology.orchestra.run.smithi183.stdout: "tier_of": -1, 2024-09-10T19:28:21.738 INFO:teuthology.orchestra.run.smithi183.stdout: "read_tier": -1, 2024-09-10T19:28:21.738 INFO:teuthology.orchestra.run.smithi183.stdout: "write_tier": -1, 2024-09-10T19:28:21.738 INFO:teuthology.orchestra.run.smithi183.stdout: "cache_mode": "none", 2024-09-10T19:28:21.738 INFO:teuthology.orchestra.run.smithi183.stdout: "target_max_bytes": 0, 2024-09-10T19:28:21.738 INFO:teuthology.orchestra.run.smithi183.stdout: "target_max_objects": 0, 2024-09-10T19:28:21.738 INFO:teuthology.orchestra.run.smithi183.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-10T19:28:21.739 INFO:teuthology.orchestra.run.smithi183.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-10T19:28:21.739 INFO:teuthology.orchestra.run.smithi183.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-10T19:28:21.739 INFO:teuthology.orchestra.run.smithi183.stdout: "cache_min_flush_age": 0, 2024-09-10T19:28:21.739 INFO:teuthology.orchestra.run.smithi183.stdout: "cache_min_evict_age": 0, 2024-09-10T19:28:21.739 INFO:teuthology.orchestra.run.smithi183.stdout: "erasure_code_profile": "", 2024-09-10T19:28:21.739 INFO:teuthology.orchestra.run.smithi183.stdout: "hit_set_params": { 2024-09-10T19:28:21.739 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "none" 2024-09-10T19:28:21.739 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:28:21.739 INFO:teuthology.orchestra.run.smithi183.stdout: "hit_set_period": 0, 2024-09-10T19:28:21.739 INFO:teuthology.orchestra.run.smithi183.stdout: "hit_set_count": 0, 2024-09-10T19:28:21.740 INFO:teuthology.orchestra.run.smithi183.stdout: "use_gmt_hitset": true, 2024-09-10T19:28:21.740 INFO:teuthology.orchestra.run.smithi183.stdout: "min_read_recency_for_promote": 0, 2024-09-10T19:28:21.740 INFO:teuthology.orchestra.run.smithi183.stdout: "min_write_recency_for_promote": 0, 2024-09-10T19:28:21.740 INFO:teuthology.orchestra.run.smithi183.stdout: "hit_set_grade_decay_rate": 0, 2024-09-10T19:28:21.740 INFO:teuthology.orchestra.run.smithi183.stdout: "hit_set_search_last_n": 0, 2024-09-10T19:28:21.740 INFO:teuthology.orchestra.run.smithi183.stdout: "grade_table": [], 2024-09-10T19:28:21.740 INFO:teuthology.orchestra.run.smithi183.stdout: "stripe_width": 0, 2024-09-10T19:28:21.740 INFO:teuthology.orchestra.run.smithi183.stdout: "expected_num_objects": 0, 2024-09-10T19:28:21.740 INFO:teuthology.orchestra.run.smithi183.stdout: "fast_read": false, 2024-09-10T19:28:21.740 INFO:teuthology.orchestra.run.smithi183.stdout: "options": {}, 2024-09-10T19:28:21.740 INFO:teuthology.orchestra.run.smithi183.stdout: "application_metadata": { 2024-09-10T19:28:21.741 INFO:teuthology.orchestra.run.smithi183.stdout: "rgw": {} 2024-09-10T19:28:21.741 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:28:21.741 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:28:21.741 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:28:21.741 INFO:teuthology.orchestra.run.smithi183.stdout: "pool": 3, 2024-09-10T19:28:21.741 INFO:teuthology.orchestra.run.smithi183.stdout: "pool_name": "datapool", 2024-09-10T19:28:21.741 INFO:teuthology.orchestra.run.smithi183.stdout: "create_time": "2024-09-10T19:10:06.224267+0000", 2024-09-10T19:28:21.741 INFO:teuthology.orchestra.run.smithi183.stdout: "flags": 8193, 2024-09-10T19:28:21.741 INFO:teuthology.orchestra.run.smithi183.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-09-10T19:28:21.741 INFO:teuthology.orchestra.run.smithi183.stdout: "type": 1, 2024-09-10T19:28:21.741 INFO:teuthology.orchestra.run.smithi183.stdout: "size": 3, 2024-09-10T19:28:21.741 INFO:teuthology.orchestra.run.smithi183.stdout: "min_size": 2, 2024-09-10T19:28:21.741 INFO:teuthology.orchestra.run.smithi183.stdout: "crush_rule": 0, 2024-09-10T19:28:21.741 INFO:teuthology.orchestra.run.smithi183.stdout: "peering_crush_bucket_count": 0, 2024-09-10T19:28:21.742 INFO:teuthology.orchestra.run.smithi183.stdout: "peering_crush_bucket_target": 0, 2024-09-10T19:28:21.742 INFO:teuthology.orchestra.run.smithi183.stdout: "peering_crush_bucket_barrier": 0, 2024-09-10T19:28:21.742 INFO:teuthology.orchestra.run.smithi183.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-10T19:28:21.742 INFO:teuthology.orchestra.run.smithi183.stdout: "object_hash": 2, 2024-09-10T19:28:21.742 INFO:teuthology.orchestra.run.smithi183.stdout: "pg_autoscale_mode": "off", 2024-09-10T19:28:21.742 INFO:teuthology.orchestra.run.smithi183.stdout: "pg_num": 3, 2024-09-10T19:28:21.742 INFO:teuthology.orchestra.run.smithi183.stdout: "pg_placement_num": 3, 2024-09-10T19:28:21.742 INFO:teuthology.orchestra.run.smithi183.stdout: "pg_placement_num_target": 3, 2024-09-10T19:28:21.742 INFO:teuthology.orchestra.run.smithi183.stdout: "pg_num_target": 3, 2024-09-10T19:28:21.742 INFO:teuthology.orchestra.run.smithi183.stdout: "pg_num_pending": 3, 2024-09-10T19:28:21.742 INFO:teuthology.orchestra.run.smithi183.stdout: "last_pg_merge_meta": { 2024-09-10T19:28:21.742 INFO:teuthology.orchestra.run.smithi183.stdout: "source_pgid": "0.0", 2024-09-10T19:28:21.742 INFO:teuthology.orchestra.run.smithi183.stdout: "ready_epoch": 0, 2024-09-10T19:28:21.742 INFO:teuthology.orchestra.run.smithi183.stdout: "last_epoch_started": 0, 2024-09-10T19:28:21.742 INFO:teuthology.orchestra.run.smithi183.stdout: "last_epoch_clean": 0, 2024-09-10T19:28:21.743 INFO:teuthology.orchestra.run.smithi183.stdout: "source_version": "0'0", 2024-09-10T19:28:21.743 INFO:teuthology.orchestra.run.smithi183.stdout: "target_version": "0'0" 2024-09-10T19:28:21.743 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:28:21.743 INFO:teuthology.orchestra.run.smithi183.stdout: "last_change": "63", 2024-09-10T19:28:21.743 INFO:teuthology.orchestra.run.smithi183.stdout: "last_force_op_resend": "0", 2024-09-10T19:28:21.743 INFO:teuthology.orchestra.run.smithi183.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-10T19:28:21.743 INFO:teuthology.orchestra.run.smithi183.stdout: "last_force_op_resend_preluminous": "0", 2024-09-10T19:28:21.743 INFO:teuthology.orchestra.run.smithi183.stdout: "auid": 0, 2024-09-10T19:28:21.743 INFO:teuthology.orchestra.run.smithi183.stdout: "snap_mode": "selfmanaged", 2024-09-10T19:28:21.743 INFO:teuthology.orchestra.run.smithi183.stdout: "snap_seq": 3, 2024-09-10T19:28:21.743 INFO:teuthology.orchestra.run.smithi183.stdout: "snap_epoch": 63, 2024-09-10T19:28:21.743 INFO:teuthology.orchestra.run.smithi183.stdout: "pool_snaps": [], 2024-09-10T19:28:21.743 INFO:teuthology.orchestra.run.smithi183.stdout: "removed_snaps": "[]", 2024-09-10T19:28:21.743 INFO:teuthology.orchestra.run.smithi183.stdout: "quota_max_bytes": 0, 2024-09-10T19:28:21.744 INFO:teuthology.orchestra.run.smithi183.stdout: "quota_max_objects": 0, 2024-09-10T19:28:21.744 INFO:teuthology.orchestra.run.smithi183.stdout: "tiers": [], 2024-09-10T19:28:21.744 INFO:teuthology.orchestra.run.smithi183.stdout: "tier_of": -1, 2024-09-10T19:28:21.744 INFO:teuthology.orchestra.run.smithi183.stdout: "read_tier": -1, 2024-09-10T19:28:21.744 INFO:teuthology.orchestra.run.smithi183.stdout: "write_tier": -1, 2024-09-10T19:28:21.744 INFO:teuthology.orchestra.run.smithi183.stdout: "cache_mode": "none", 2024-09-10T19:28:21.744 INFO:teuthology.orchestra.run.smithi183.stdout: "target_max_bytes": 0, 2024-09-10T19:28:21.744 INFO:teuthology.orchestra.run.smithi183.stdout: "target_max_objects": 0, 2024-09-10T19:28:21.744 INFO:teuthology.orchestra.run.smithi183.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-10T19:28:21.744 INFO:teuthology.orchestra.run.smithi183.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-10T19:28:21.744 INFO:teuthology.orchestra.run.smithi183.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-10T19:28:21.744 INFO:teuthology.orchestra.run.smithi183.stdout: "cache_min_flush_age": 0, 2024-09-10T19:28:21.744 INFO:teuthology.orchestra.run.smithi183.stdout: "cache_min_evict_age": 0, 2024-09-10T19:28:21.744 INFO:teuthology.orchestra.run.smithi183.stdout: "erasure_code_profile": "", 2024-09-10T19:28:21.744 INFO:teuthology.orchestra.run.smithi183.stdout: "hit_set_params": { 2024-09-10T19:28:21.745 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "none" 2024-09-10T19:28:21.745 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:28:21.745 INFO:teuthology.orchestra.run.smithi183.stdout: "hit_set_period": 0, 2024-09-10T19:28:21.745 INFO:teuthology.orchestra.run.smithi183.stdout: "hit_set_count": 0, 2024-09-10T19:28:21.745 INFO:teuthology.orchestra.run.smithi183.stdout: "use_gmt_hitset": true, 2024-09-10T19:28:21.746 INFO:teuthology.orchestra.run.smithi183.stdout: "min_read_recency_for_promote": 0, 2024-09-10T19:28:21.746 INFO:teuthology.orchestra.run.smithi183.stdout: "min_write_recency_for_promote": 0, 2024-09-10T19:28:21.746 INFO:teuthology.orchestra.run.smithi183.stdout: "hit_set_grade_decay_rate": 0, 2024-09-10T19:28:21.746 INFO:teuthology.orchestra.run.smithi183.stdout: "hit_set_search_last_n": 0, 2024-09-10T19:28:21.746 INFO:teuthology.orchestra.run.smithi183.stdout: "grade_table": [], 2024-09-10T19:28:21.746 INFO:teuthology.orchestra.run.smithi183.stdout: "stripe_width": 0, 2024-09-10T19:28:21.746 INFO:teuthology.orchestra.run.smithi183.stdout: "expected_num_objects": 0, 2024-09-10T19:28:21.746 INFO:teuthology.orchestra.run.smithi183.stdout: "fast_read": false, 2024-09-10T19:28:21.746 INFO:teuthology.orchestra.run.smithi183.stdout: "options": {}, 2024-09-10T19:28:21.746 INFO:teuthology.orchestra.run.smithi183.stdout: "application_metadata": { 2024-09-10T19:28:21.746 INFO:teuthology.orchestra.run.smithi183.stdout: "rbd": {} 2024-09-10T19:28:21.746 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:28:21.746 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:28:21.747 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:28:21.747 INFO:teuthology.orchestra.run.smithi183.stdout: "pool": 4, 2024-09-10T19:28:21.747 INFO:teuthology.orchestra.run.smithi183.stdout: "pool_name": "default.rgw.log", 2024-09-10T19:28:21.747 INFO:teuthology.orchestra.run.smithi183.stdout: "create_time": "2024-09-10T19:10:07.958284+0000", 2024-09-10T19:28:21.747 INFO:teuthology.orchestra.run.smithi183.stdout: "flags": 1, 2024-09-10T19:28:21.747 INFO:teuthology.orchestra.run.smithi183.stdout: "flags_names": "hashpspool", 2024-09-10T19:28:21.747 INFO:teuthology.orchestra.run.smithi183.stdout: "type": 1, 2024-09-10T19:28:21.747 INFO:teuthology.orchestra.run.smithi183.stdout: "size": 3, 2024-09-10T19:28:21.747 INFO:teuthology.orchestra.run.smithi183.stdout: "min_size": 2, 2024-09-10T19:28:21.747 INFO:teuthology.orchestra.run.smithi183.stdout: "crush_rule": 0, 2024-09-10T19:28:21.747 INFO:teuthology.orchestra.run.smithi183.stdout: "peering_crush_bucket_count": 0, 2024-09-10T19:28:21.747 INFO:teuthology.orchestra.run.smithi183.stdout: "peering_crush_bucket_target": 0, 2024-09-10T19:28:21.747 INFO:teuthology.orchestra.run.smithi183.stdout: "peering_crush_bucket_barrier": 0, 2024-09-10T19:28:21.747 INFO:teuthology.orchestra.run.smithi183.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-10T19:28:21.748 INFO:teuthology.orchestra.run.smithi183.stdout: "object_hash": 2, 2024-09-10T19:28:21.748 INFO:teuthology.orchestra.run.smithi183.stdout: "pg_autoscale_mode": "off", 2024-09-10T19:28:21.748 INFO:teuthology.orchestra.run.smithi183.stdout: "pg_num": 32, 2024-09-10T19:28:21.748 INFO:teuthology.orchestra.run.smithi183.stdout: "pg_placement_num": 32, 2024-09-10T19:28:21.748 INFO:teuthology.orchestra.run.smithi183.stdout: "pg_placement_num_target": 32, 2024-09-10T19:28:21.748 INFO:teuthology.orchestra.run.smithi183.stdout: "pg_num_target": 32, 2024-09-10T19:28:21.748 INFO:teuthology.orchestra.run.smithi183.stdout: "pg_num_pending": 32, 2024-09-10T19:28:21.748 INFO:teuthology.orchestra.run.smithi183.stdout: "last_pg_merge_meta": { 2024-09-10T19:28:21.748 INFO:teuthology.orchestra.run.smithi183.stdout: "source_pgid": "0.0", 2024-09-10T19:28:21.748 INFO:teuthology.orchestra.run.smithi183.stdout: "ready_epoch": 0, 2024-09-10T19:28:21.748 INFO:teuthology.orchestra.run.smithi183.stdout: "last_epoch_started": 0, 2024-09-10T19:28:21.748 INFO:teuthology.orchestra.run.smithi183.stdout: "last_epoch_clean": 0, 2024-09-10T19:28:21.748 INFO:teuthology.orchestra.run.smithi183.stdout: "source_version": "0'0", 2024-09-10T19:28:21.749 INFO:teuthology.orchestra.run.smithi183.stdout: "target_version": "0'0" 2024-09-10T19:28:21.749 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:28:21.749 INFO:teuthology.orchestra.run.smithi183.stdout: "last_change": "62", 2024-09-10T19:28:21.749 INFO:teuthology.orchestra.run.smithi183.stdout: "last_force_op_resend": "0", 2024-09-10T19:28:21.749 INFO:teuthology.orchestra.run.smithi183.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-10T19:28:21.749 INFO:teuthology.orchestra.run.smithi183.stdout: "last_force_op_resend_preluminous": "0", 2024-09-10T19:28:21.749 INFO:teuthology.orchestra.run.smithi183.stdout: "auid": 0, 2024-09-10T19:28:21.749 INFO:teuthology.orchestra.run.smithi183.stdout: "snap_mode": "selfmanaged", 2024-09-10T19:28:21.749 INFO:teuthology.orchestra.run.smithi183.stdout: "snap_seq": 0, 2024-09-10T19:28:21.749 INFO:teuthology.orchestra.run.smithi183.stdout: "snap_epoch": 0, 2024-09-10T19:28:21.749 INFO:teuthology.orchestra.run.smithi183.stdout: "pool_snaps": [], 2024-09-10T19:28:21.749 INFO:teuthology.orchestra.run.smithi183.stdout: "removed_snaps": "[]", 2024-09-10T19:28:21.749 INFO:teuthology.orchestra.run.smithi183.stdout: "quota_max_bytes": 0, 2024-09-10T19:28:21.749 INFO:teuthology.orchestra.run.smithi183.stdout: "quota_max_objects": 0, 2024-09-10T19:28:21.750 INFO:teuthology.orchestra.run.smithi183.stdout: "tiers": [], 2024-09-10T19:28:21.750 INFO:teuthology.orchestra.run.smithi183.stdout: "tier_of": -1, 2024-09-10T19:28:21.750 INFO:teuthology.orchestra.run.smithi183.stdout: "read_tier": -1, 2024-09-10T19:28:21.750 INFO:teuthology.orchestra.run.smithi183.stdout: "write_tier": -1, 2024-09-10T19:28:21.750 INFO:teuthology.orchestra.run.smithi183.stdout: "cache_mode": "none", 2024-09-10T19:28:21.750 INFO:teuthology.orchestra.run.smithi183.stdout: "target_max_bytes": 0, 2024-09-10T19:28:21.750 INFO:teuthology.orchestra.run.smithi183.stdout: "target_max_objects": 0, 2024-09-10T19:28:21.750 INFO:teuthology.orchestra.run.smithi183.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-10T19:28:21.750 INFO:teuthology.orchestra.run.smithi183.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-10T19:28:21.750 INFO:teuthology.orchestra.run.smithi183.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-10T19:28:21.750 INFO:teuthology.orchestra.run.smithi183.stdout: "cache_min_flush_age": 0, 2024-09-10T19:28:21.750 INFO:teuthology.orchestra.run.smithi183.stdout: "cache_min_evict_age": 0, 2024-09-10T19:28:21.750 INFO:teuthology.orchestra.run.smithi183.stdout: "erasure_code_profile": "", 2024-09-10T19:28:21.750 INFO:teuthology.orchestra.run.smithi183.stdout: "hit_set_params": { 2024-09-10T19:28:21.751 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "none" 2024-09-10T19:28:21.751 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:28:21.751 INFO:teuthology.orchestra.run.smithi183.stdout: "hit_set_period": 0, 2024-09-10T19:28:21.751 INFO:teuthology.orchestra.run.smithi183.stdout: "hit_set_count": 0, 2024-09-10T19:28:21.751 INFO:teuthology.orchestra.run.smithi183.stdout: "use_gmt_hitset": true, 2024-09-10T19:28:21.751 INFO:teuthology.orchestra.run.smithi183.stdout: "min_read_recency_for_promote": 0, 2024-09-10T19:28:21.751 INFO:teuthology.orchestra.run.smithi183.stdout: "min_write_recency_for_promote": 0, 2024-09-10T19:28:21.751 INFO:teuthology.orchestra.run.smithi183.stdout: "hit_set_grade_decay_rate": 0, 2024-09-10T19:28:21.751 INFO:teuthology.orchestra.run.smithi183.stdout: "hit_set_search_last_n": 0, 2024-09-10T19:28:21.751 INFO:teuthology.orchestra.run.smithi183.stdout: "grade_table": [], 2024-09-10T19:28:21.751 INFO:teuthology.orchestra.run.smithi183.stdout: "stripe_width": 0, 2024-09-10T19:28:21.751 INFO:teuthology.orchestra.run.smithi183.stdout: "expected_num_objects": 0, 2024-09-10T19:28:21.751 INFO:teuthology.orchestra.run.smithi183.stdout: "fast_read": false, 2024-09-10T19:28:21.751 INFO:teuthology.orchestra.run.smithi183.stdout: "options": {}, 2024-09-10T19:28:21.752 INFO:teuthology.orchestra.run.smithi183.stdout: "application_metadata": { 2024-09-10T19:28:21.752 INFO:teuthology.orchestra.run.smithi183.stdout: "rgw": {} 2024-09-10T19:28:21.752 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:28:21.752 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:28:21.752 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:28:21.752 INFO:teuthology.orchestra.run.smithi183.stdout: "pool": 5, 2024-09-10T19:28:21.752 INFO:teuthology.orchestra.run.smithi183.stdout: "pool_name": "default.rgw.control", 2024-09-10T19:28:21.752 INFO:teuthology.orchestra.run.smithi183.stdout: "create_time": "2024-09-10T19:10:09.933596+0000", 2024-09-10T19:28:21.752 INFO:teuthology.orchestra.run.smithi183.stdout: "flags": 1, 2024-09-10T19:28:21.752 INFO:teuthology.orchestra.run.smithi183.stdout: "flags_names": "hashpspool", 2024-09-10T19:28:21.752 INFO:teuthology.orchestra.run.smithi183.stdout: "type": 1, 2024-09-10T19:28:21.752 INFO:teuthology.orchestra.run.smithi183.stdout: "size": 3, 2024-09-10T19:28:21.752 INFO:teuthology.orchestra.run.smithi183.stdout: "min_size": 2, 2024-09-10T19:28:21.752 INFO:teuthology.orchestra.run.smithi183.stdout: "crush_rule": 0, 2024-09-10T19:28:21.753 INFO:teuthology.orchestra.run.smithi183.stdout: "peering_crush_bucket_count": 0, 2024-09-10T19:28:21.753 INFO:teuthology.orchestra.run.smithi183.stdout: "peering_crush_bucket_target": 0, 2024-09-10T19:28:21.753 INFO:teuthology.orchestra.run.smithi183.stdout: "peering_crush_bucket_barrier": 0, 2024-09-10T19:28:21.753 INFO:teuthology.orchestra.run.smithi183.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-10T19:28:21.753 INFO:teuthology.orchestra.run.smithi183.stdout: "object_hash": 2, 2024-09-10T19:28:21.753 INFO:teuthology.orchestra.run.smithi183.stdout: "pg_autoscale_mode": "off", 2024-09-10T19:28:21.753 INFO:teuthology.orchestra.run.smithi183.stdout: "pg_num": 32, 2024-09-10T19:28:21.753 INFO:teuthology.orchestra.run.smithi183.stdout: "pg_placement_num": 32, 2024-09-10T19:28:21.753 INFO:teuthology.orchestra.run.smithi183.stdout: "pg_placement_num_target": 32, 2024-09-10T19:28:21.753 INFO:teuthology.orchestra.run.smithi183.stdout: "pg_num_target": 32, 2024-09-10T19:28:21.753 INFO:teuthology.orchestra.run.smithi183.stdout: "pg_num_pending": 32, 2024-09-10T19:28:21.753 INFO:teuthology.orchestra.run.smithi183.stdout: "last_pg_merge_meta": { 2024-09-10T19:28:21.753 INFO:teuthology.orchestra.run.smithi183.stdout: "source_pgid": "0.0", 2024-09-10T19:28:21.754 INFO:teuthology.orchestra.run.smithi183.stdout: "ready_epoch": 0, 2024-09-10T19:28:21.754 INFO:teuthology.orchestra.run.smithi183.stdout: "last_epoch_started": 0, 2024-09-10T19:28:21.754 INFO:teuthology.orchestra.run.smithi183.stdout: "last_epoch_clean": 0, 2024-09-10T19:28:21.754 INFO:teuthology.orchestra.run.smithi183.stdout: "source_version": "0'0", 2024-09-10T19:28:21.754 INFO:teuthology.orchestra.run.smithi183.stdout: "target_version": "0'0" 2024-09-10T19:28:21.754 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:28:21.754 INFO:teuthology.orchestra.run.smithi183.stdout: "last_change": "64", 2024-09-10T19:28:21.754 INFO:teuthology.orchestra.run.smithi183.stdout: "last_force_op_resend": "0", 2024-09-10T19:28:21.754 INFO:teuthology.orchestra.run.smithi183.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-10T19:28:21.754 INFO:teuthology.orchestra.run.smithi183.stdout: "last_force_op_resend_preluminous": "0", 2024-09-10T19:28:21.754 INFO:teuthology.orchestra.run.smithi183.stdout: "auid": 0, 2024-09-10T19:28:21.754 INFO:teuthology.orchestra.run.smithi183.stdout: "snap_mode": "selfmanaged", 2024-09-10T19:28:21.754 INFO:teuthology.orchestra.run.smithi183.stdout: "snap_seq": 0, 2024-09-10T19:28:21.754 INFO:teuthology.orchestra.run.smithi183.stdout: "snap_epoch": 0, 2024-09-10T19:28:21.754 INFO:teuthology.orchestra.run.smithi183.stdout: "pool_snaps": [], 2024-09-10T19:28:21.755 INFO:teuthology.orchestra.run.smithi183.stdout: "removed_snaps": "[]", 2024-09-10T19:28:21.755 INFO:teuthology.orchestra.run.smithi183.stdout: "quota_max_bytes": 0, 2024-09-10T19:28:21.755 INFO:teuthology.orchestra.run.smithi183.stdout: "quota_max_objects": 0, 2024-09-10T19:28:21.755 INFO:teuthology.orchestra.run.smithi183.stdout: "tiers": [], 2024-09-10T19:28:21.755 INFO:teuthology.orchestra.run.smithi183.stdout: "tier_of": -1, 2024-09-10T19:28:21.755 INFO:teuthology.orchestra.run.smithi183.stdout: "read_tier": -1, 2024-09-10T19:28:21.755 INFO:teuthology.orchestra.run.smithi183.stdout: "write_tier": -1, 2024-09-10T19:28:21.755 INFO:teuthology.orchestra.run.smithi183.stdout: "cache_mode": "none", 2024-09-10T19:28:21.755 INFO:teuthology.orchestra.run.smithi183.stdout: "target_max_bytes": 0, 2024-09-10T19:28:21.755 INFO:teuthology.orchestra.run.smithi183.stdout: "target_max_objects": 0, 2024-09-10T19:28:21.755 INFO:teuthology.orchestra.run.smithi183.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-10T19:28:21.755 INFO:teuthology.orchestra.run.smithi183.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-10T19:28:21.755 INFO:teuthology.orchestra.run.smithi183.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-10T19:28:21.756 INFO:teuthology.orchestra.run.smithi183.stdout: "cache_min_flush_age": 0, 2024-09-10T19:28:21.756 INFO:teuthology.orchestra.run.smithi183.stdout: "cache_min_evict_age": 0, 2024-09-10T19:28:21.756 INFO:teuthology.orchestra.run.smithi183.stdout: "erasure_code_profile": "", 2024-09-10T19:28:21.756 INFO:teuthology.orchestra.run.smithi183.stdout: "hit_set_params": { 2024-09-10T19:28:21.756 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "none" 2024-09-10T19:28:21.756 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:28:21.756 INFO:teuthology.orchestra.run.smithi183.stdout: "hit_set_period": 0, 2024-09-10T19:28:21.756 INFO:teuthology.orchestra.run.smithi183.stdout: "hit_set_count": 0, 2024-09-10T19:28:21.756 INFO:teuthology.orchestra.run.smithi183.stdout: "use_gmt_hitset": true, 2024-09-10T19:28:21.756 INFO:teuthology.orchestra.run.smithi183.stdout: "min_read_recency_for_promote": 0, 2024-09-10T19:28:21.756 INFO:teuthology.orchestra.run.smithi183.stdout: "min_write_recency_for_promote": 0, 2024-09-10T19:28:21.756 INFO:teuthology.orchestra.run.smithi183.stdout: "hit_set_grade_decay_rate": 0, 2024-09-10T19:28:21.756 INFO:teuthology.orchestra.run.smithi183.stdout: "hit_set_search_last_n": 0, 2024-09-10T19:28:21.756 INFO:teuthology.orchestra.run.smithi183.stdout: "grade_table": [], 2024-09-10T19:28:21.757 INFO:teuthology.orchestra.run.smithi183.stdout: "stripe_width": 0, 2024-09-10T19:28:21.757 INFO:teuthology.orchestra.run.smithi183.stdout: "expected_num_objects": 0, 2024-09-10T19:28:21.757 INFO:teuthology.orchestra.run.smithi183.stdout: "fast_read": false, 2024-09-10T19:28:21.757 INFO:teuthology.orchestra.run.smithi183.stdout: "options": {}, 2024-09-10T19:28:21.757 INFO:teuthology.orchestra.run.smithi183.stdout: "application_metadata": { 2024-09-10T19:28:21.757 INFO:teuthology.orchestra.run.smithi183.stdout: "rgw": {} 2024-09-10T19:28:21.757 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:28:21.757 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:28:21.757 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:28:21.757 INFO:teuthology.orchestra.run.smithi183.stdout: "pool": 6, 2024-09-10T19:28:21.757 INFO:teuthology.orchestra.run.smithi183.stdout: "pool_name": "default.rgw.meta", 2024-09-10T19:28:21.757 INFO:teuthology.orchestra.run.smithi183.stdout: "create_time": "2024-09-10T19:10:11.993712+0000", 2024-09-10T19:28:21.757 INFO:teuthology.orchestra.run.smithi183.stdout: "flags": 1, 2024-09-10T19:28:21.757 INFO:teuthology.orchestra.run.smithi183.stdout: "flags_names": "hashpspool", 2024-09-10T19:28:21.758 INFO:teuthology.orchestra.run.smithi183.stdout: "type": 1, 2024-09-10T19:28:21.758 INFO:teuthology.orchestra.run.smithi183.stdout: "size": 3, 2024-09-10T19:28:21.758 INFO:teuthology.orchestra.run.smithi183.stdout: "min_size": 2, 2024-09-10T19:28:21.758 INFO:teuthology.orchestra.run.smithi183.stdout: "crush_rule": 0, 2024-09-10T19:28:21.758 INFO:teuthology.orchestra.run.smithi183.stdout: "peering_crush_bucket_count": 0, 2024-09-10T19:28:21.758 INFO:teuthology.orchestra.run.smithi183.stdout: "peering_crush_bucket_target": 0, 2024-09-10T19:28:21.758 INFO:teuthology.orchestra.run.smithi183.stdout: "peering_crush_bucket_barrier": 0, 2024-09-10T19:28:21.758 INFO:teuthology.orchestra.run.smithi183.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-10T19:28:21.758 INFO:teuthology.orchestra.run.smithi183.stdout: "object_hash": 2, 2024-09-10T19:28:21.758 INFO:teuthology.orchestra.run.smithi183.stdout: "pg_autoscale_mode": "off", 2024-09-10T19:28:21.758 INFO:teuthology.orchestra.run.smithi183.stdout: "pg_num": 32, 2024-09-10T19:28:21.758 INFO:teuthology.orchestra.run.smithi183.stdout: "pg_placement_num": 32, 2024-09-10T19:28:21.758 INFO:teuthology.orchestra.run.smithi183.stdout: "pg_placement_num_target": 32, 2024-09-10T19:28:21.759 INFO:teuthology.orchestra.run.smithi183.stdout: "pg_num_target": 32, 2024-09-10T19:28:21.759 INFO:teuthology.orchestra.run.smithi183.stdout: "pg_num_pending": 32, 2024-09-10T19:28:21.759 INFO:teuthology.orchestra.run.smithi183.stdout: "last_pg_merge_meta": { 2024-09-10T19:28:21.759 INFO:teuthology.orchestra.run.smithi183.stdout: "source_pgid": "0.0", 2024-09-10T19:28:21.759 INFO:teuthology.orchestra.run.smithi183.stdout: "ready_epoch": 0, 2024-09-10T19:28:21.759 INFO:teuthology.orchestra.run.smithi183.stdout: "last_epoch_started": 0, 2024-09-10T19:28:21.759 INFO:teuthology.orchestra.run.smithi183.stdout: "last_epoch_clean": 0, 2024-09-10T19:28:21.759 INFO:teuthology.orchestra.run.smithi183.stdout: "source_version": "0'0", 2024-09-10T19:28:21.759 INFO:teuthology.orchestra.run.smithi183.stdout: "target_version": "0'0" 2024-09-10T19:28:21.759 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:28:21.759 INFO:teuthology.orchestra.run.smithi183.stdout: "last_change": "66", 2024-09-10T19:28:21.759 INFO:teuthology.orchestra.run.smithi183.stdout: "last_force_op_resend": "0", 2024-09-10T19:28:21.759 INFO:teuthology.orchestra.run.smithi183.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-10T19:28:21.760 INFO:teuthology.orchestra.run.smithi183.stdout: "last_force_op_resend_preluminous": "0", 2024-09-10T19:28:21.760 INFO:teuthology.orchestra.run.smithi183.stdout: "auid": 0, 2024-09-10T19:28:21.760 INFO:teuthology.orchestra.run.smithi183.stdout: "snap_mode": "selfmanaged", 2024-09-10T19:28:21.760 INFO:teuthology.orchestra.run.smithi183.stdout: "snap_seq": 0, 2024-09-10T19:28:21.760 INFO:teuthology.orchestra.run.smithi183.stdout: "snap_epoch": 0, 2024-09-10T19:28:21.760 INFO:teuthology.orchestra.run.smithi183.stdout: "pool_snaps": [], 2024-09-10T19:28:21.760 INFO:teuthology.orchestra.run.smithi183.stdout: "removed_snaps": "[]", 2024-09-10T19:28:21.760 INFO:teuthology.orchestra.run.smithi183.stdout: "quota_max_bytes": 0, 2024-09-10T19:28:21.760 INFO:teuthology.orchestra.run.smithi183.stdout: "quota_max_objects": 0, 2024-09-10T19:28:21.760 INFO:teuthology.orchestra.run.smithi183.stdout: "tiers": [], 2024-09-10T19:28:21.760 INFO:teuthology.orchestra.run.smithi183.stdout: "tier_of": -1, 2024-09-10T19:28:21.760 INFO:teuthology.orchestra.run.smithi183.stdout: "read_tier": -1, 2024-09-10T19:28:21.760 INFO:teuthology.orchestra.run.smithi183.stdout: "write_tier": -1, 2024-09-10T19:28:21.761 INFO:teuthology.orchestra.run.smithi183.stdout: "cache_mode": "none", 2024-09-10T19:28:21.761 INFO:teuthology.orchestra.run.smithi183.stdout: "target_max_bytes": 0, 2024-09-10T19:28:21.761 INFO:teuthology.orchestra.run.smithi183.stdout: "target_max_objects": 0, 2024-09-10T19:28:21.761 INFO:teuthology.orchestra.run.smithi183.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-10T19:28:21.761 INFO:teuthology.orchestra.run.smithi183.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-10T19:28:21.761 INFO:teuthology.orchestra.run.smithi183.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-10T19:28:21.761 INFO:teuthology.orchestra.run.smithi183.stdout: "cache_min_flush_age": 0, 2024-09-10T19:28:21.761 INFO:teuthology.orchestra.run.smithi183.stdout: "cache_min_evict_age": 0, 2024-09-10T19:28:21.761 INFO:teuthology.orchestra.run.smithi183.stdout: "erasure_code_profile": "", 2024-09-10T19:28:21.761 INFO:teuthology.orchestra.run.smithi183.stdout: "hit_set_params": { 2024-09-10T19:28:21.761 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "none" 2024-09-10T19:28:21.761 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:28:21.761 INFO:teuthology.orchestra.run.smithi183.stdout: "hit_set_period": 0, 2024-09-10T19:28:21.762 INFO:teuthology.orchestra.run.smithi183.stdout: "hit_set_count": 0, 2024-09-10T19:28:21.762 INFO:teuthology.orchestra.run.smithi183.stdout: "use_gmt_hitset": true, 2024-09-10T19:28:21.762 INFO:teuthology.orchestra.run.smithi183.stdout: "min_read_recency_for_promote": 0, 2024-09-10T19:28:21.762 INFO:teuthology.orchestra.run.smithi183.stdout: "min_write_recency_for_promote": 0, 2024-09-10T19:28:21.762 INFO:teuthology.orchestra.run.smithi183.stdout: "hit_set_grade_decay_rate": 0, 2024-09-10T19:28:21.762 INFO:teuthology.orchestra.run.smithi183.stdout: "hit_set_search_last_n": 0, 2024-09-10T19:28:21.762 INFO:teuthology.orchestra.run.smithi183.stdout: "grade_table": [], 2024-09-10T19:28:21.762 INFO:teuthology.orchestra.run.smithi183.stdout: "stripe_width": 0, 2024-09-10T19:28:21.762 INFO:teuthology.orchestra.run.smithi183.stdout: "expected_num_objects": 0, 2024-09-10T19:28:21.762 INFO:teuthology.orchestra.run.smithi183.stdout: "fast_read": false, 2024-09-10T19:28:21.762 INFO:teuthology.orchestra.run.smithi183.stdout: "options": { 2024-09-10T19:28:21.762 INFO:teuthology.orchestra.run.smithi183.stdout: "pg_autoscale_bias": 4 2024-09-10T19:28:21.762 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:28:21.762 INFO:teuthology.orchestra.run.smithi183.stdout: "application_metadata": { 2024-09-10T19:28:21.763 INFO:teuthology.orchestra.run.smithi183.stdout: "rgw": {} 2024-09-10T19:28:21.763 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:28:21.763 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:28:21.763 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:28:21.763 INFO:teuthology.orchestra.run.smithi183.stdout: "pool": 11, 2024-09-10T19:28:21.763 INFO:teuthology.orchestra.run.smithi183.stdout: "pool_name": "unique_pool_4", 2024-09-10T19:28:21.763 INFO:teuthology.orchestra.run.smithi183.stdout: "create_time": "2024-09-10T19:27:45.303826+0000", 2024-09-10T19:28:21.763 INFO:teuthology.orchestra.run.smithi183.stdout: "flags": 1, 2024-09-10T19:28:21.763 INFO:teuthology.orchestra.run.smithi183.stdout: "flags_names": "hashpspool", 2024-09-10T19:28:21.763 INFO:teuthology.orchestra.run.smithi183.stdout: "type": 1, 2024-09-10T19:28:21.763 INFO:teuthology.orchestra.run.smithi183.stdout: "size": 3, 2024-09-10T19:28:21.763 INFO:teuthology.orchestra.run.smithi183.stdout: "min_size": 2, 2024-09-10T19:28:21.764 INFO:teuthology.orchestra.run.smithi183.stdout: "crush_rule": 0, 2024-09-10T19:28:21.764 INFO:teuthology.orchestra.run.smithi183.stdout: "peering_crush_bucket_count": 0, 2024-09-10T19:28:21.764 INFO:teuthology.orchestra.run.smithi183.stdout: "peering_crush_bucket_target": 0, 2024-09-10T19:28:21.764 INFO:teuthology.orchestra.run.smithi183.stdout: "peering_crush_bucket_barrier": 0, 2024-09-10T19:28:21.764 INFO:teuthology.orchestra.run.smithi183.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-10T19:28:21.764 INFO:teuthology.orchestra.run.smithi183.stdout: "object_hash": 2, 2024-09-10T19:28:21.764 INFO:teuthology.orchestra.run.smithi183.stdout: "pg_autoscale_mode": "off", 2024-09-10T19:28:21.764 INFO:teuthology.orchestra.run.smithi183.stdout: "pg_num": 16, 2024-09-10T19:28:21.764 INFO:teuthology.orchestra.run.smithi183.stdout: "pg_placement_num": 16, 2024-09-10T19:28:21.764 INFO:teuthology.orchestra.run.smithi183.stdout: "pg_placement_num_target": 16, 2024-09-10T19:28:21.765 INFO:teuthology.orchestra.run.smithi183.stdout: "pg_num_target": 16, 2024-09-10T19:28:21.765 INFO:teuthology.orchestra.run.smithi183.stdout: "pg_num_pending": 16, 2024-09-10T19:28:21.765 INFO:teuthology.orchestra.run.smithi183.stdout: "last_pg_merge_meta": { 2024-09-10T19:28:21.765 INFO:teuthology.orchestra.run.smithi183.stdout: "source_pgid": "0.0", 2024-09-10T19:28:21.765 INFO:teuthology.orchestra.run.smithi183.stdout: "ready_epoch": 0, 2024-09-10T19:28:21.765 INFO:teuthology.orchestra.run.smithi183.stdout: "last_epoch_started": 0, 2024-09-10T19:28:21.765 INFO:teuthology.orchestra.run.smithi183.stdout: "last_epoch_clean": 0, 2024-09-10T19:28:21.765 INFO:teuthology.orchestra.run.smithi183.stdout: "source_version": "0'0", 2024-09-10T19:28:21.765 INFO:teuthology.orchestra.run.smithi183.stdout: "target_version": "0'0" 2024-09-10T19:28:21.765 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:28:21.765 INFO:teuthology.orchestra.run.smithi183.stdout: "last_change": "374", 2024-09-10T19:28:21.765 INFO:teuthology.orchestra.run.smithi183.stdout: "last_force_op_resend": "0", 2024-09-10T19:28:21.765 INFO:teuthology.orchestra.run.smithi183.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-10T19:28:21.765 INFO:teuthology.orchestra.run.smithi183.stdout: "last_force_op_resend_preluminous": "0", 2024-09-10T19:28:21.765 INFO:teuthology.orchestra.run.smithi183.stdout: "auid": 0, 2024-09-10T19:28:21.766 INFO:teuthology.orchestra.run.smithi183.stdout: "snap_mode": "selfmanaged", 2024-09-10T19:28:21.766 INFO:teuthology.orchestra.run.smithi183.stdout: "snap_seq": 0, 2024-09-10T19:28:21.766 INFO:teuthology.orchestra.run.smithi183.stdout: "snap_epoch": 0, 2024-09-10T19:28:21.766 INFO:teuthology.orchestra.run.smithi183.stdout: "pool_snaps": [], 2024-09-10T19:28:21.766 INFO:teuthology.orchestra.run.smithi183.stdout: "removed_snaps": "[]", 2024-09-10T19:28:21.766 INFO:teuthology.orchestra.run.smithi183.stdout: "quota_max_bytes": 0, 2024-09-10T19:28:21.766 INFO:teuthology.orchestra.run.smithi183.stdout: "quota_max_objects": 0, 2024-09-10T19:28:21.766 INFO:teuthology.orchestra.run.smithi183.stdout: "tiers": [], 2024-09-10T19:28:21.766 INFO:teuthology.orchestra.run.smithi183.stdout: "tier_of": -1, 2024-09-10T19:28:21.766 INFO:teuthology.orchestra.run.smithi183.stdout: "read_tier": -1, 2024-09-10T19:28:21.766 INFO:teuthology.orchestra.run.smithi183.stdout: "write_tier": -1, 2024-09-10T19:28:21.766 INFO:teuthology.orchestra.run.smithi183.stdout: "cache_mode": "none", 2024-09-10T19:28:21.766 INFO:teuthology.orchestra.run.smithi183.stdout: "target_max_bytes": 0, 2024-09-10T19:28:21.767 INFO:teuthology.orchestra.run.smithi183.stdout: "target_max_objects": 0, 2024-09-10T19:28:21.767 INFO:teuthology.orchestra.run.smithi183.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-10T19:28:21.767 INFO:teuthology.orchestra.run.smithi183.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-10T19:28:21.767 INFO:teuthology.orchestra.run.smithi183.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-10T19:28:21.767 INFO:teuthology.orchestra.run.smithi183.stdout: "cache_min_flush_age": 0, 2024-09-10T19:28:21.767 INFO:teuthology.orchestra.run.smithi183.stdout: "cache_min_evict_age": 0, 2024-09-10T19:28:21.767 INFO:teuthology.orchestra.run.smithi183.stdout: "erasure_code_profile": "", 2024-09-10T19:28:21.767 INFO:teuthology.orchestra.run.smithi183.stdout: "hit_set_params": { 2024-09-10T19:28:21.767 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "none" 2024-09-10T19:28:21.767 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:28:21.767 INFO:teuthology.orchestra.run.smithi183.stdout: "hit_set_period": 0, 2024-09-10T19:28:21.767 INFO:teuthology.orchestra.run.smithi183.stdout: "hit_set_count": 0, 2024-09-10T19:28:21.767 INFO:teuthology.orchestra.run.smithi183.stdout: "use_gmt_hitset": true, 2024-09-10T19:28:21.768 INFO:teuthology.orchestra.run.smithi183.stdout: "min_read_recency_for_promote": 0, 2024-09-10T19:28:21.768 INFO:teuthology.orchestra.run.smithi183.stdout: "min_write_recency_for_promote": 0, 2024-09-10T19:28:21.768 INFO:teuthology.orchestra.run.smithi183.stdout: "hit_set_grade_decay_rate": 0, 2024-09-10T19:28:21.768 INFO:teuthology.orchestra.run.smithi183.stdout: "hit_set_search_last_n": 0, 2024-09-10T19:28:21.768 INFO:teuthology.orchestra.run.smithi183.stdout: "grade_table": [], 2024-09-10T19:28:21.768 INFO:teuthology.orchestra.run.smithi183.stdout: "stripe_width": 0, 2024-09-10T19:28:21.768 INFO:teuthology.orchestra.run.smithi183.stdout: "expected_num_objects": 0, 2024-09-10T19:28:21.768 INFO:teuthology.orchestra.run.smithi183.stdout: "fast_read": false, 2024-09-10T19:28:21.768 INFO:teuthology.orchestra.run.smithi183.stdout: "options": {}, 2024-09-10T19:28:21.768 INFO:teuthology.orchestra.run.smithi183.stdout: "application_metadata": { 2024-09-10T19:28:21.768 INFO:teuthology.orchestra.run.smithi183.stdout: "rados": {} 2024-09-10T19:28:21.768 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:28:21.768 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:28:21.768 INFO:teuthology.orchestra.run.smithi183.stdout: ], 2024-09-10T19:28:21.769 INFO:teuthology.orchestra.run.smithi183.stdout: "osds": [ 2024-09-10T19:28:21.769 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:28:21.769 INFO:teuthology.orchestra.run.smithi183.stdout: "osd": 0, 2024-09-10T19:28:21.769 INFO:teuthology.orchestra.run.smithi183.stdout: "uuid": "9d323010-394a-49aa-ba9b-244d87faf36e", 2024-09-10T19:28:21.769 INFO:teuthology.orchestra.run.smithi183.stdout: "up": 1, 2024-09-10T19:28:21.769 INFO:teuthology.orchestra.run.smithi183.stdout: "in": 1, 2024-09-10T19:28:21.769 INFO:teuthology.orchestra.run.smithi183.stdout: "weight": 1, 2024-09-10T19:28:21.769 INFO:teuthology.orchestra.run.smithi183.stdout: "primary_affinity": 1, 2024-09-10T19:28:21.769 INFO:teuthology.orchestra.run.smithi183.stdout: "last_clean_begin": 8, 2024-09-10T19:28:21.769 INFO:teuthology.orchestra.run.smithi183.stdout: "last_clean_end": 262, 2024-09-10T19:28:21.769 INFO:teuthology.orchestra.run.smithi183.stdout: "up_from": 301, 2024-09-10T19:28:21.769 INFO:teuthology.orchestra.run.smithi183.stdout: "up_thru": 374, 2024-09-10T19:28:21.769 INFO:teuthology.orchestra.run.smithi183.stdout: "down_at": 263, 2024-09-10T19:28:21.770 INFO:teuthology.orchestra.run.smithi183.stdout: "lost_at": 0, 2024-09-10T19:28:21.770 INFO:teuthology.orchestra.run.smithi183.stdout: "public_addrs": { 2024-09-10T19:28:21.770 INFO:teuthology.orchestra.run.smithi183.stdout: "addrvec": [ 2024-09-10T19:28:21.770 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:28:21.770 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v2", 2024-09-10T19:28:21.770 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.183:6810", 2024-09-10T19:28:21.770 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 3346539414 2024-09-10T19:28:21.770 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:28:21.770 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:28:21.770 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v1", 2024-09-10T19:28:21.770 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.183:6811", 2024-09-10T19:28:21.770 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 3346539414 2024-09-10T19:28:21.770 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:28:21.770 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:28:21.771 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:28:21.771 INFO:teuthology.orchestra.run.smithi183.stdout: "cluster_addrs": { 2024-09-10T19:28:21.771 INFO:teuthology.orchestra.run.smithi183.stdout: "addrvec": [ 2024-09-10T19:28:21.771 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:28:21.771 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v2", 2024-09-10T19:28:21.771 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.183:6812", 2024-09-10T19:28:21.771 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 3346539414 2024-09-10T19:28:21.771 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:28:21.771 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:28:21.771 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v1", 2024-09-10T19:28:21.771 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.183:6813", 2024-09-10T19:28:21.771 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 3346539414 2024-09-10T19:28:21.771 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:28:21.772 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:28:21.772 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:28:21.772 INFO:teuthology.orchestra.run.smithi183.stdout: "heartbeat_back_addrs": { 2024-09-10T19:28:21.772 INFO:teuthology.orchestra.run.smithi183.stdout: "addrvec": [ 2024-09-10T19:28:21.772 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:28:21.772 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v2", 2024-09-10T19:28:21.772 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.183:6816", 2024-09-10T19:28:21.772 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 3346539414 2024-09-10T19:28:21.772 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:28:21.772 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:28:21.772 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v1", 2024-09-10T19:28:21.772 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.183:6817", 2024-09-10T19:28:21.772 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 3346539414 2024-09-10T19:28:21.773 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:28:21.773 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:28:21.773 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:28:21.773 INFO:teuthology.orchestra.run.smithi183.stdout: "heartbeat_front_addrs": { 2024-09-10T19:28:21.773 INFO:teuthology.orchestra.run.smithi183.stdout: "addrvec": [ 2024-09-10T19:28:21.773 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:28:21.773 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v2", 2024-09-10T19:28:21.773 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.183:6814", 2024-09-10T19:28:21.773 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 3346539414 2024-09-10T19:28:21.773 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:28:21.773 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:28:21.773 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v1", 2024-09-10T19:28:21.773 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.183:6815", 2024-09-10T19:28:21.774 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 3346539414 2024-09-10T19:28:21.774 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:28:21.774 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:28:21.774 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:28:21.774 INFO:teuthology.orchestra.run.smithi183.stdout: "public_addr": "172.21.15.183:6811/3346539414", 2024-09-10T19:28:21.774 INFO:teuthology.orchestra.run.smithi183.stdout: "cluster_addr": "172.21.15.183:6813/3346539414", 2024-09-10T19:28:21.774 INFO:teuthology.orchestra.run.smithi183.stdout: "heartbeat_back_addr": "172.21.15.183:6817/3346539414", 2024-09-10T19:28:21.774 INFO:teuthology.orchestra.run.smithi183.stdout: "heartbeat_front_addr": "172.21.15.183:6815/3346539414", 2024-09-10T19:28:21.774 INFO:teuthology.orchestra.run.smithi183.stdout: "state": [ 2024-09-10T19:28:21.774 INFO:teuthology.orchestra.run.smithi183.stdout: "exists", 2024-09-10T19:28:21.774 INFO:teuthology.orchestra.run.smithi183.stdout: "up" 2024-09-10T19:28:21.774 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:28:21.774 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:28:21.774 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:28:21.775 INFO:teuthology.orchestra.run.smithi183.stdout: "osd": 1, 2024-09-10T19:28:21.775 INFO:teuthology.orchestra.run.smithi183.stdout: "uuid": "71a6c5dc-5109-4284-8d20-187d8e7afc9e", 2024-09-10T19:28:21.775 INFO:teuthology.orchestra.run.smithi183.stdout: "up": 1, 2024-09-10T19:28:21.775 INFO:teuthology.orchestra.run.smithi183.stdout: "in": 1, 2024-09-10T19:28:21.775 INFO:teuthology.orchestra.run.smithi183.stdout: "weight": 1, 2024-09-10T19:28:21.775 INFO:teuthology.orchestra.run.smithi183.stdout: "primary_affinity": 1, 2024-09-10T19:28:21.775 INFO:teuthology.orchestra.run.smithi183.stdout: "last_clean_begin": 138, 2024-09-10T19:28:21.775 INFO:teuthology.orchestra.run.smithi183.stdout: "last_clean_end": 255, 2024-09-10T19:28:21.775 INFO:teuthology.orchestra.run.smithi183.stdout: "up_from": 276, 2024-09-10T19:28:21.775 INFO:teuthology.orchestra.run.smithi183.stdout: "up_thru": 374, 2024-09-10T19:28:21.775 INFO:teuthology.orchestra.run.smithi183.stdout: "down_at": 256, 2024-09-10T19:28:21.775 INFO:teuthology.orchestra.run.smithi183.stdout: "lost_at": 0, 2024-09-10T19:28:21.775 INFO:teuthology.orchestra.run.smithi183.stdout: "public_addrs": { 2024-09-10T19:28:21.775 INFO:teuthology.orchestra.run.smithi183.stdout: "addrvec": [ 2024-09-10T19:28:21.776 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:28:21.776 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v2", 2024-09-10T19:28:21.776 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.183:6802", 2024-09-10T19:28:21.776 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 2764394151 2024-09-10T19:28:21.776 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:28:21.776 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:28:21.776 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v1", 2024-09-10T19:28:21.776 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.183:6803", 2024-09-10T19:28:21.776 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 2764394151 2024-09-10T19:28:21.776 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:28:21.776 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:28:21.776 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:28:21.776 INFO:teuthology.orchestra.run.smithi183.stdout: "cluster_addrs": { 2024-09-10T19:28:21.776 INFO:teuthology.orchestra.run.smithi183.stdout: "addrvec": [ 2024-09-10T19:28:21.777 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:28:21.777 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v2", 2024-09-10T19:28:21.777 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.183:6804", 2024-09-10T19:28:21.777 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 2764394151 2024-09-10T19:28:21.777 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:28:21.777 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:28:21.777 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v1", 2024-09-10T19:28:21.777 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.183:6805", 2024-09-10T19:28:21.777 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 2764394151 2024-09-10T19:28:21.777 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:28:21.777 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:28:21.777 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:28:21.777 INFO:teuthology.orchestra.run.smithi183.stdout: "heartbeat_back_addrs": { 2024-09-10T19:28:21.777 INFO:teuthology.orchestra.run.smithi183.stdout: "addrvec": [ 2024-09-10T19:28:21.778 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:28:21.778 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v2", 2024-09-10T19:28:21.778 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.183:6808", 2024-09-10T19:28:21.778 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 2764394151 2024-09-10T19:28:21.778 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:28:21.778 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:28:21.778 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v1", 2024-09-10T19:28:21.778 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.183:6809", 2024-09-10T19:28:21.778 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 2764394151 2024-09-10T19:28:21.778 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:28:21.778 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:28:21.778 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:28:21.778 INFO:teuthology.orchestra.run.smithi183.stdout: "heartbeat_front_addrs": { 2024-09-10T19:28:21.779 INFO:teuthology.orchestra.run.smithi183.stdout: "addrvec": [ 2024-09-10T19:28:21.779 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:28:21.779 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v2", 2024-09-10T19:28:21.779 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.183:6806", 2024-09-10T19:28:21.779 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 2764394151 2024-09-10T19:28:21.779 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:28:21.779 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:28:21.779 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v1", 2024-09-10T19:28:21.779 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.183:6807", 2024-09-10T19:28:21.779 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 2764394151 2024-09-10T19:28:21.779 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:28:21.779 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:28:21.779 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:28:21.779 INFO:teuthology.orchestra.run.smithi183.stdout: "public_addr": "172.21.15.183:6803/2764394151", 2024-09-10T19:28:21.780 INFO:teuthology.orchestra.run.smithi183.stdout: "cluster_addr": "172.21.15.183:6805/2764394151", 2024-09-10T19:28:21.780 INFO:teuthology.orchestra.run.smithi183.stdout: "heartbeat_back_addr": "172.21.15.183:6809/2764394151", 2024-09-10T19:28:21.780 INFO:teuthology.orchestra.run.smithi183.stdout: "heartbeat_front_addr": "172.21.15.183:6807/2764394151", 2024-09-10T19:28:21.780 INFO:teuthology.orchestra.run.smithi183.stdout: "state": [ 2024-09-10T19:28:21.780 INFO:teuthology.orchestra.run.smithi183.stdout: "exists", 2024-09-10T19:28:21.780 INFO:teuthology.orchestra.run.smithi183.stdout: "up" 2024-09-10T19:28:21.780 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:28:21.780 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:28:21.780 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:28:21.780 INFO:teuthology.orchestra.run.smithi183.stdout: "osd": 2, 2024-09-10T19:28:21.780 INFO:teuthology.orchestra.run.smithi183.stdout: "uuid": "fc310d5b-b89b-4655-9989-7e10ba356509", 2024-09-10T19:28:21.780 INFO:teuthology.orchestra.run.smithi183.stdout: "up": 1, 2024-09-10T19:28:21.780 INFO:teuthology.orchestra.run.smithi183.stdout: "in": 0, 2024-09-10T19:28:21.781 INFO:teuthology.orchestra.run.smithi183.stdout: "weight": 0, 2024-09-10T19:28:21.781 INFO:teuthology.orchestra.run.smithi183.stdout: "primary_affinity": 1, 2024-09-10T19:28:21.781 INFO:teuthology.orchestra.run.smithi183.stdout: "last_clean_begin": 0, 2024-09-10T19:28:21.781 INFO:teuthology.orchestra.run.smithi183.stdout: "last_clean_end": 0, 2024-09-10T19:28:21.781 INFO:teuthology.orchestra.run.smithi183.stdout: "up_from": 18, 2024-09-10T19:28:21.781 INFO:teuthology.orchestra.run.smithi183.stdout: "up_thru": 148, 2024-09-10T19:28:21.781 INFO:teuthology.orchestra.run.smithi183.stdout: "down_at": 0, 2024-09-10T19:28:21.781 INFO:teuthology.orchestra.run.smithi183.stdout: "lost_at": 0, 2024-09-10T19:28:21.781 INFO:teuthology.orchestra.run.smithi183.stdout: "public_addrs": { 2024-09-10T19:28:21.781 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:28:21.783 INFO:teuthology.orchestra.run.smithi183.stdout: "addrvec": [ 2024-09-10T19:28:21.783 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:28:21.783 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v2", 2024-09-10T19:28:21.783 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.183:6818", 2024-09-10T19:28:21.783 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 3730013786 2024-09-10T19:28:21.783 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:28:21.783 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:28:21.783 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v1", 2024-09-10T19:28:21.783 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.183:6819", 2024-09-10T19:28:21.783 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 3730013786 2024-09-10T19:28:21.783 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:28:21.784 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:28:21.784 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:28:21.784 INFO:teuthology.orchestra.run.smithi183.stdout: "cluster_addrs": { 2024-09-10T19:28:21.784 INFO:teuthology.orchestra.run.smithi183.stdout: "addrvec": [ 2024-09-10T19:28:21.784 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:28:21.784 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v2", 2024-09-10T19:28:21.784 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.183:6820", 2024-09-10T19:28:21.784 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 3730013786 2024-09-10T19:28:21.784 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:28:21.784 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:28:21.784 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v1", 2024-09-10T19:28:21.784 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.183:6821", 2024-09-10T19:28:21.784 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 3730013786 2024-09-10T19:28:21.785 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:28:21.785 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:28:21.785 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:28:21.785 INFO:teuthology.orchestra.run.smithi183.stdout: "heartbeat_back_addrs": { 2024-09-10T19:28:21.785 INFO:teuthology.orchestra.run.smithi183.stdout: "addrvec": [ 2024-09-10T19:28:21.785 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:28:21.785 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v2", 2024-09-10T19:28:21.785 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.183:6824", 2024-09-10T19:28:21.785 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 3730013786 2024-09-10T19:28:21.785 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:28:21.785 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:28:21.785 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v1", 2024-09-10T19:28:21.785 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.183:6825", 2024-09-10T19:28:21.786 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 3730013786 2024-09-10T19:28:21.786 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:28:21.786 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:28:21.786 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:28:21.786 INFO:teuthology.orchestra.run.smithi183.stdout: "heartbeat_front_addrs": { 2024-09-10T19:28:21.786 INFO:teuthology.orchestra.run.smithi183.stdout: "addrvec": [ 2024-09-10T19:28:21.786 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:28:21.786 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v2", 2024-09-10T19:28:21.786 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.183:6822", 2024-09-10T19:28:21.786 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 3730013786 2024-09-10T19:28:21.786 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:28:21.786 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:28:21.786 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v1", 2024-09-10T19:28:21.786 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.183:6823", 2024-09-10T19:28:21.787 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 3730013786 2024-09-10T19:28:21.787 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:28:21.787 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:28:21.787 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:28:21.787 INFO:teuthology.orchestra.run.smithi183.stdout: "public_addr": "172.21.15.183:6819/3730013786", 2024-09-10T19:28:21.787 INFO:teuthology.orchestra.run.smithi183.stdout: "cluster_addr": "172.21.15.183:6821/3730013786", 2024-09-10T19:28:21.787 INFO:teuthology.orchestra.run.smithi183.stdout: "heartbeat_back_addr": "172.21.15.183:6825/3730013786", 2024-09-10T19:28:21.787 INFO:teuthology.orchestra.run.smithi183.stdout: "heartbeat_front_addr": "172.21.15.183:6823/3730013786", 2024-09-10T19:28:21.787 INFO:teuthology.orchestra.run.smithi183.stdout: "state": [ 2024-09-10T19:28:21.787 INFO:teuthology.orchestra.run.smithi183.stdout: "exists", 2024-09-10T19:28:21.787 INFO:teuthology.orchestra.run.smithi183.stdout: "up" 2024-09-10T19:28:21.788 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:28:21.788 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:28:21.788 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:28:21.788 INFO:teuthology.orchestra.run.smithi183.stdout: "osd": 3, 2024-09-10T19:28:21.788 INFO:teuthology.orchestra.run.smithi183.stdout: "uuid": "7fd07a95-ad22-43f5-a968-539552283e89", 2024-09-10T19:28:21.788 INFO:teuthology.orchestra.run.smithi183.stdout: "up": 1, 2024-09-10T19:28:21.788 INFO:teuthology.orchestra.run.smithi183.stdout: "in": 1, 2024-09-10T19:28:21.788 INFO:teuthology.orchestra.run.smithi183.stdout: "weight": 1, 2024-09-10T19:28:21.788 INFO:teuthology.orchestra.run.smithi183.stdout: "primary_affinity": 1, 2024-09-10T19:28:21.788 INFO:teuthology.orchestra.run.smithi183.stdout: "last_clean_begin": 0, 2024-09-10T19:28:21.788 INFO:teuthology.orchestra.run.smithi183.stdout: "last_clean_end": 0, 2024-09-10T19:28:21.788 INFO:teuthology.orchestra.run.smithi183.stdout: "up_from": 26, 2024-09-10T19:28:21.788 INFO:teuthology.orchestra.run.smithi183.stdout: "up_thru": 374, 2024-09-10T19:28:21.789 INFO:teuthology.orchestra.run.smithi183.stdout: "down_at": 0, 2024-09-10T19:28:21.789 INFO:teuthology.orchestra.run.smithi183.stdout: "lost_at": 0, 2024-09-10T19:28:21.789 INFO:teuthology.orchestra.run.smithi183.stdout: "public_addrs": { 2024-09-10T19:28:21.789 INFO:teuthology.orchestra.run.smithi183.stdout: "addrvec": [ 2024-09-10T19:28:21.789 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:28:21.789 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v2", 2024-09-10T19:28:21.789 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.183:6826", 2024-09-10T19:28:21.789 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 3522836984 2024-09-10T19:28:21.789 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:28:21.789 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:28:21.789 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v1", 2024-09-10T19:28:21.789 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.183:6827", 2024-09-10T19:28:21.789 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 3522836984 2024-09-10T19:28:21.790 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:28:21.790 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:28:21.790 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:28:21.790 INFO:teuthology.orchestra.run.smithi183.stdout: "cluster_addrs": { 2024-09-10T19:28:21.790 INFO:teuthology.orchestra.run.smithi183.stdout: "addrvec": [ 2024-09-10T19:28:21.790 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:28:21.790 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v2", 2024-09-10T19:28:21.790 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.183:6828", 2024-09-10T19:28:21.790 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 3522836984 2024-09-10T19:28:21.790 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:28:21.790 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:28:21.790 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v1", 2024-09-10T19:28:21.790 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.183:6829", 2024-09-10T19:28:21.790 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 3522836984 2024-09-10T19:28:21.791 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:28:21.791 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:28:21.791 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:28:21.791 INFO:teuthology.orchestra.run.smithi183.stdout: "heartbeat_back_addrs": { 2024-09-10T19:28:21.791 INFO:teuthology.orchestra.run.smithi183.stdout: "addrvec": [ 2024-09-10T19:28:21.791 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:28:21.791 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v2", 2024-09-10T19:28:21.791 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.183:6832", 2024-09-10T19:28:21.791 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 3522836984 2024-09-10T19:28:21.791 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:28:21.791 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:28:21.791 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v1", 2024-09-10T19:28:21.791 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.183:6833", 2024-09-10T19:28:21.792 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 3522836984 2024-09-10T19:28:21.792 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:28:21.792 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:28:21.792 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:28:21.792 INFO:teuthology.orchestra.run.smithi183.stdout: "heartbeat_front_addrs": { 2024-09-10T19:28:21.792 INFO:teuthology.orchestra.run.smithi183.stdout: "addrvec": [ 2024-09-10T19:28:21.792 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:28:21.792 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v2", 2024-09-10T19:28:21.792 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.183:6830", 2024-09-10T19:28:21.792 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 3522836984 2024-09-10T19:28:21.792 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:28:21.792 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:28:21.792 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v1", 2024-09-10T19:28:21.793 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.183:6831", 2024-09-10T19:28:21.793 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 3522836984 2024-09-10T19:28:21.793 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:28:21.793 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:28:21.793 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:28:21.793 INFO:teuthology.orchestra.run.smithi183.stdout: "public_addr": "172.21.15.183:6827/3522836984", 2024-09-10T19:28:21.793 INFO:teuthology.orchestra.run.smithi183.stdout: "cluster_addr": "172.21.15.183:6829/3522836984", 2024-09-10T19:28:21.793 INFO:teuthology.orchestra.run.smithi183.stdout: "heartbeat_back_addr": "172.21.15.183:6833/3522836984", 2024-09-10T19:28:21.793 INFO:teuthology.orchestra.run.smithi183.stdout: "heartbeat_front_addr": "172.21.15.183:6831/3522836984", 2024-09-10T19:28:21.793 INFO:teuthology.orchestra.run.smithi183.stdout: "state": [ 2024-09-10T19:28:21.793 INFO:teuthology.orchestra.run.smithi183.stdout: "exists", 2024-09-10T19:28:21.793 INFO:teuthology.orchestra.run.smithi183.stdout: "up" 2024-09-10T19:28:21.793 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:28:21.793 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:28:21.794 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:28:21.794 INFO:teuthology.orchestra.run.smithi183.stdout: "osd": 4, 2024-09-10T19:28:21.794 INFO:teuthology.orchestra.run.smithi183.stdout: "uuid": "631083fb-e503-44da-9b6b-154e88539ee7", 2024-09-10T19:28:21.794 INFO:teuthology.orchestra.run.smithi183.stdout: "up": 1, 2024-09-10T19:28:21.794 INFO:teuthology.orchestra.run.smithi183.stdout: "in": 1, 2024-09-10T19:28:21.794 INFO:teuthology.orchestra.run.smithi183.stdout: "weight": 1, 2024-09-10T19:28:21.794 INFO:teuthology.orchestra.run.smithi183.stdout: "primary_affinity": 1, 2024-09-10T19:28:21.794 INFO:teuthology.orchestra.run.smithi183.stdout: "last_clean_begin": 0, 2024-09-10T19:28:21.794 INFO:teuthology.orchestra.run.smithi183.stdout: "last_clean_end": 0, 2024-09-10T19:28:21.794 INFO:teuthology.orchestra.run.smithi183.stdout: "up_from": 32, 2024-09-10T19:28:21.794 INFO:teuthology.orchestra.run.smithi183.stdout: "up_thru": 374, 2024-09-10T19:28:21.794 INFO:teuthology.orchestra.run.smithi183.stdout: "down_at": 0, 2024-09-10T19:28:21.794 INFO:teuthology.orchestra.run.smithi183.stdout: "lost_at": 0, 2024-09-10T19:28:21.794 INFO:teuthology.orchestra.run.smithi183.stdout: "public_addrs": { 2024-09-10T19:28:21.795 INFO:teuthology.orchestra.run.smithi183.stdout: "addrvec": [ 2024-09-10T19:28:21.795 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:28:21.795 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v2", 2024-09-10T19:28:21.795 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.191:6800", 2024-09-10T19:28:21.795 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 2674828539 2024-09-10T19:28:21.795 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:28:21.795 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:28:21.795 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v1", 2024-09-10T19:28:21.795 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.191:6801", 2024-09-10T19:28:21.795 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 2674828539 2024-09-10T19:28:21.795 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:28:21.795 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:28:21.795 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:28:21.795 INFO:teuthology.orchestra.run.smithi183.stdout: "cluster_addrs": { 2024-09-10T19:28:21.796 INFO:teuthology.orchestra.run.smithi183.stdout: "addrvec": [ 2024-09-10T19:28:21.796 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:28:21.796 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v2", 2024-09-10T19:28:21.796 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.191:6802", 2024-09-10T19:28:21.796 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 2674828539 2024-09-10T19:28:21.796 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:28:21.796 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:28:21.796 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v1", 2024-09-10T19:28:21.796 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.191:6803", 2024-09-10T19:28:21.796 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 2674828539 2024-09-10T19:28:21.796 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:28:21.796 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:28:21.796 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:28:21.796 INFO:teuthology.orchestra.run.smithi183.stdout: "heartbeat_back_addrs": { 2024-09-10T19:28:21.797 INFO:teuthology.orchestra.run.smithi183.stdout: "addrvec": [ 2024-09-10T19:28:21.797 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:28:21.797 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v2", 2024-09-10T19:28:21.797 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.191:6806", 2024-09-10T19:28:21.797 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 2674828539 2024-09-10T19:28:21.797 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:28:21.797 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:28:21.797 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v1", 2024-09-10T19:28:21.797 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.191:6807", 2024-09-10T19:28:21.797 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 2674828539 2024-09-10T19:28:21.797 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:28:21.797 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:28:21.797 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:28:21.797 INFO:teuthology.orchestra.run.smithi183.stdout: "heartbeat_front_addrs": { 2024-09-10T19:28:21.798 INFO:teuthology.orchestra.run.smithi183.stdout: "addrvec": [ 2024-09-10T19:28:21.798 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:28:21.798 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v2", 2024-09-10T19:28:21.798 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.191:6804", 2024-09-10T19:28:21.798 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 2674828539 2024-09-10T19:28:21.798 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:28:21.798 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:28:21.798 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v1", 2024-09-10T19:28:21.798 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.191:6805", 2024-09-10T19:28:21.798 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 2674828539 2024-09-10T19:28:21.798 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:28:21.798 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:28:21.798 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:28:21.798 INFO:teuthology.orchestra.run.smithi183.stdout: "public_addr": "172.21.15.191:6801/2674828539", 2024-09-10T19:28:21.799 INFO:teuthology.orchestra.run.smithi183.stdout: "cluster_addr": "172.21.15.191:6803/2674828539", 2024-09-10T19:28:21.799 INFO:teuthology.orchestra.run.smithi183.stdout: "heartbeat_back_addr": "172.21.15.191:6807/2674828539", 2024-09-10T19:28:21.799 INFO:teuthology.orchestra.run.smithi183.stdout: "heartbeat_front_addr": "172.21.15.191:6805/2674828539", 2024-09-10T19:28:21.799 INFO:teuthology.orchestra.run.smithi183.stdout: "state": [ 2024-09-10T19:28:21.799 INFO:teuthology.orchestra.run.smithi183.stdout: "exists", 2024-09-10T19:28:21.799 INFO:teuthology.orchestra.run.smithi183.stdout: "up" 2024-09-10T19:28:21.799 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:28:21.799 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:28:21.799 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:28:21.799 INFO:teuthology.orchestra.run.smithi183.stdout: "osd": 5, 2024-09-10T19:28:21.799 INFO:teuthology.orchestra.run.smithi183.stdout: "uuid": "6b963695-3eca-4e6e-9207-05b8010ebff9", 2024-09-10T19:28:21.799 INFO:teuthology.orchestra.run.smithi183.stdout: "up": 1, 2024-09-10T19:28:21.799 INFO:teuthology.orchestra.run.smithi183.stdout: "in": 1, 2024-09-10T19:28:21.799 INFO:teuthology.orchestra.run.smithi183.stdout: "weight": 1, 2024-09-10T19:28:21.800 INFO:teuthology.orchestra.run.smithi183.stdout: "primary_affinity": 1, 2024-09-10T19:28:21.800 INFO:teuthology.orchestra.run.smithi183.stdout: "last_clean_begin": 0, 2024-09-10T19:28:21.800 INFO:teuthology.orchestra.run.smithi183.stdout: "last_clean_end": 0, 2024-09-10T19:28:21.800 INFO:teuthology.orchestra.run.smithi183.stdout: "up_from": 39, 2024-09-10T19:28:21.800 INFO:teuthology.orchestra.run.smithi183.stdout: "up_thru": 374, 2024-09-10T19:28:21.800 INFO:teuthology.orchestra.run.smithi183.stdout: "down_at": 0, 2024-09-10T19:28:21.800 INFO:teuthology.orchestra.run.smithi183.stdout: "lost_at": 0, 2024-09-10T19:28:21.800 INFO:teuthology.orchestra.run.smithi183.stdout: "public_addrs": { 2024-09-10T19:28:21.800 INFO:teuthology.orchestra.run.smithi183.stdout: "addrvec": [ 2024-09-10T19:28:21.800 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:28:21.800 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v2", 2024-09-10T19:28:21.800 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.191:6808", 2024-09-10T19:28:21.800 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 681733378 2024-09-10T19:28:21.800 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:28:21.801 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:28:21.801 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v1", 2024-09-10T19:28:21.801 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.191:6809", 2024-09-10T19:28:21.801 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 681733378 2024-09-10T19:28:21.801 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:28:21.801 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:28:21.801 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:28:21.801 INFO:teuthology.orchestra.run.smithi183.stdout: "cluster_addrs": { 2024-09-10T19:28:21.801 INFO:teuthology.orchestra.run.smithi183.stdout: "addrvec": [ 2024-09-10T19:28:21.801 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:28:21.801 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v2", 2024-09-10T19:28:21.801 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.191:6810", 2024-09-10T19:28:21.801 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 681733378 2024-09-10T19:28:21.802 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:28:21.802 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:28:21.802 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v1", 2024-09-10T19:28:21.802 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.191:6811", 2024-09-10T19:28:21.802 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 681733378 2024-09-10T19:28:21.802 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:28:21.802 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:28:21.802 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:28:21.802 INFO:teuthology.orchestra.run.smithi183.stdout: "heartbeat_back_addrs": { 2024-09-10T19:28:21.802 INFO:teuthology.orchestra.run.smithi183.stdout: "addrvec": [ 2024-09-10T19:28:21.802 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:28:21.802 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v2", 2024-09-10T19:28:21.802 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.191:6814", 2024-09-10T19:28:21.802 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 681733378 2024-09-10T19:28:21.803 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:28:21.803 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:28:21.803 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v1", 2024-09-10T19:28:21.803 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.191:6815", 2024-09-10T19:28:21.803 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 681733378 2024-09-10T19:28:21.803 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:28:21.803 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:28:21.803 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:28:21.803 INFO:teuthology.orchestra.run.smithi183.stdout: "heartbeat_front_addrs": { 2024-09-10T19:28:21.803 INFO:teuthology.orchestra.run.smithi183.stdout: "addrvec": [ 2024-09-10T19:28:21.803 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:28:21.803 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v2", 2024-09-10T19:28:21.803 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.191:6812", 2024-09-10T19:28:21.803 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 681733378 2024-09-10T19:28:21.804 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:28:21.804 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:28:21.804 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v1", 2024-09-10T19:28:21.804 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.191:6813", 2024-09-10T19:28:21.804 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 681733378 2024-09-10T19:28:21.804 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:28:21.804 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:28:21.804 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:28:21.804 INFO:teuthology.orchestra.run.smithi183.stdout: "public_addr": "172.21.15.191:6809/681733378", 2024-09-10T19:28:21.804 INFO:teuthology.orchestra.run.smithi183.stdout: "cluster_addr": "172.21.15.191:6811/681733378", 2024-09-10T19:28:21.804 INFO:teuthology.orchestra.run.smithi183.stdout: "heartbeat_back_addr": "172.21.15.191:6815/681733378", 2024-09-10T19:28:21.804 INFO:teuthology.orchestra.run.smithi183.stdout: "heartbeat_front_addr": "172.21.15.191:6813/681733378", 2024-09-10T19:28:21.805 INFO:teuthology.orchestra.run.smithi183.stdout: "state": [ 2024-09-10T19:28:21.805 INFO:teuthology.orchestra.run.smithi183.stdout: "exists", 2024-09-10T19:28:21.805 INFO:teuthology.orchestra.run.smithi183.stdout: "up" 2024-09-10T19:28:21.805 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:28:21.805 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:28:21.805 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:28:21.805 INFO:teuthology.orchestra.run.smithi183.stdout: "osd": 6, 2024-09-10T19:28:21.805 INFO:teuthology.orchestra.run.smithi183.stdout: "uuid": "2dc9754f-6aef-4e19-8605-2f1591921811", 2024-09-10T19:28:21.805 INFO:teuthology.orchestra.run.smithi183.stdout: "up": 1, 2024-09-10T19:28:21.805 INFO:teuthology.orchestra.run.smithi183.stdout: "in": 1, 2024-09-10T19:28:21.805 INFO:teuthology.orchestra.run.smithi183.stdout: "weight": 1, 2024-09-10T19:28:21.805 INFO:teuthology.orchestra.run.smithi183.stdout: "primary_affinity": 1, 2024-09-10T19:28:21.805 INFO:teuthology.orchestra.run.smithi183.stdout: "last_clean_begin": 0, 2024-09-10T19:28:21.805 INFO:teuthology.orchestra.run.smithi183.stdout: "last_clean_end": 0, 2024-09-10T19:28:21.805 INFO:teuthology.orchestra.run.smithi183.stdout: "up_from": 47, 2024-09-10T19:28:21.806 INFO:teuthology.orchestra.run.smithi183.stdout: "up_thru": 374, 2024-09-10T19:28:21.806 INFO:teuthology.orchestra.run.smithi183.stdout: "down_at": 0, 2024-09-10T19:28:21.806 INFO:teuthology.orchestra.run.smithi183.stdout: "lost_at": 0, 2024-09-10T19:28:21.806 INFO:teuthology.orchestra.run.smithi183.stdout: "public_addrs": { 2024-09-10T19:28:21.806 INFO:teuthology.orchestra.run.smithi183.stdout: "addrvec": [ 2024-09-10T19:28:21.806 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:28:21.806 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v2", 2024-09-10T19:28:21.806 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.191:6816", 2024-09-10T19:28:21.806 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 2825813477 2024-09-10T19:28:21.806 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:28:21.806 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:28:21.806 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v1", 2024-09-10T19:28:21.806 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.191:6817", 2024-09-10T19:28:21.806 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 2825813477 2024-09-10T19:28:21.807 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:28:21.807 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:28:21.807 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:28:21.807 INFO:teuthology.orchestra.run.smithi183.stdout: "cluster_addrs": { 2024-09-10T19:28:21.807 INFO:teuthology.orchestra.run.smithi183.stdout: "addrvec": [ 2024-09-10T19:28:21.807 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:28:21.807 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v2", 2024-09-10T19:28:21.807 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.191:6818", 2024-09-10T19:28:21.807 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 2825813477 2024-09-10T19:28:21.807 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:28:21.807 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:28:21.807 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v1", 2024-09-10T19:28:21.807 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.191:6819", 2024-09-10T19:28:21.807 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 2825813477 2024-09-10T19:28:21.808 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:28:21.808 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:28:21.808 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:28:21.808 INFO:teuthology.orchestra.run.smithi183.stdout: "heartbeat_back_addrs": { 2024-09-10T19:28:21.808 INFO:teuthology.orchestra.run.smithi183.stdout: "addrvec": [ 2024-09-10T19:28:21.808 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:28:21.808 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v2", 2024-09-10T19:28:21.808 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.191:6822", 2024-09-10T19:28:21.808 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 2825813477 2024-09-10T19:28:21.808 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:28:21.808 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:28:21.808 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v1", 2024-09-10T19:28:21.808 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.191:6823", 2024-09-10T19:28:21.809 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 2825813477 2024-09-10T19:28:21.809 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:28:21.809 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:28:21.809 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:28:21.809 INFO:teuthology.orchestra.run.smithi183.stdout: "heartbeat_front_addrs": { 2024-09-10T19:28:21.809 INFO:teuthology.orchestra.run.smithi183.stdout: "addrvec": [ 2024-09-10T19:28:21.809 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:28:21.809 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v2", 2024-09-10T19:28:21.809 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.191:6820", 2024-09-10T19:28:21.812 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 2825813477 2024-09-10T19:28:21.812 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:28:21.812 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:28:21.812 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v1", 2024-09-10T19:28:21.812 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.191:6821", 2024-09-10T19:28:21.812 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 2825813477 2024-09-10T19:28:21.812 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:28:21.812 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:28:21.812 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:28:21.813 INFO:teuthology.orchestra.run.smithi183.stdout: "public_addr": "172.21.15.191:6817/2825813477", 2024-09-10T19:28:21.813 INFO:teuthology.orchestra.run.smithi183.stdout: "cluster_addr": "172.21.15.191:6819/2825813477", 2024-09-10T19:28:21.813 INFO:teuthology.orchestra.run.smithi183.stdout: "heartbeat_back_addr": "172.21.15.191:6823/2825813477", 2024-09-10T19:28:21.813 INFO:teuthology.orchestra.run.smithi183.stdout: "heartbeat_front_addr": "172.21.15.191:6821/2825813477", 2024-09-10T19:28:21.813 INFO:teuthology.orchestra.run.smithi183.stdout: "state": [ 2024-09-10T19:28:21.813 INFO:teuthology.orchestra.run.smithi183.stdout: "exists", 2024-09-10T19:28:21.813 INFO:teuthology.orchestra.run.smithi183.stdout: "up" 2024-09-10T19:28:21.813 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:28:21.813 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:28:21.813 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:28:21.813 INFO:teuthology.orchestra.run.smithi183.stdout: "osd": 7, 2024-09-10T19:28:21.813 INFO:teuthology.orchestra.run.smithi183.stdout: "uuid": "1a9a299e-d5ba-4b9a-85d5-7f6a5bc754e1", 2024-09-10T19:28:21.813 INFO:teuthology.orchestra.run.smithi183.stdout: "up": 1, 2024-09-10T19:28:21.813 INFO:teuthology.orchestra.run.smithi183.stdout: "in": 1, 2024-09-10T19:28:21.814 INFO:teuthology.orchestra.run.smithi183.stdout: "weight": 1, 2024-09-10T19:28:21.814 INFO:teuthology.orchestra.run.smithi183.stdout: "primary_affinity": 1, 2024-09-10T19:28:21.814 INFO:teuthology.orchestra.run.smithi183.stdout: "last_clean_begin": 0, 2024-09-10T19:28:21.814 INFO:teuthology.orchestra.run.smithi183.stdout: "last_clean_end": 0, 2024-09-10T19:28:21.814 INFO:teuthology.orchestra.run.smithi183.stdout: "up_from": 54, 2024-09-10T19:28:21.814 INFO:teuthology.orchestra.run.smithi183.stdout: "up_thru": 374, 2024-09-10T19:28:21.814 INFO:teuthology.orchestra.run.smithi183.stdout: "down_at": 0, 2024-09-10T19:28:21.814 INFO:teuthology.orchestra.run.smithi183.stdout: "lost_at": 0, 2024-09-10T19:28:21.814 INFO:teuthology.orchestra.run.smithi183.stdout: "public_addrs": { 2024-09-10T19:28:21.814 INFO:teuthology.orchestra.run.smithi183.stdout: "addrvec": [ 2024-09-10T19:28:21.814 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:28:21.814 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v2", 2024-09-10T19:28:21.814 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.191:6824", 2024-09-10T19:28:21.814 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 1422610021 2024-09-10T19:28:21.814 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:28:21.815 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:28:21.815 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v1", 2024-09-10T19:28:21.815 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.191:6825", 2024-09-10T19:28:21.815 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 1422610021 2024-09-10T19:28:21.815 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:28:21.815 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:28:21.815 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:28:21.815 INFO:teuthology.orchestra.run.smithi183.stdout: "cluster_addrs": { 2024-09-10T19:28:21.815 INFO:teuthology.orchestra.run.smithi183.stdout: "addrvec": [ 2024-09-10T19:28:21.815 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:28:21.815 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v2", 2024-09-10T19:28:21.815 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.191:6826", 2024-09-10T19:28:21.815 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 1422610021 2024-09-10T19:28:21.815 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:28:21.816 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:28:21.816 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v1", 2024-09-10T19:28:21.816 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.191:6827", 2024-09-10T19:28:21.816 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 1422610021 2024-09-10T19:28:21.816 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:28:21.816 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:28:21.816 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:28:21.816 INFO:teuthology.orchestra.run.smithi183.stdout: "heartbeat_back_addrs": { 2024-09-10T19:28:21.816 INFO:teuthology.orchestra.run.smithi183.stdout: "addrvec": [ 2024-09-10T19:28:21.816 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:28:21.816 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v2", 2024-09-10T19:28:21.816 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.191:6830", 2024-09-10T19:28:21.816 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 1422610021 2024-09-10T19:28:21.816 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:28:21.817 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:28:21.817 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v1", 2024-09-10T19:28:21.817 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.191:6831", 2024-09-10T19:28:21.817 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 1422610021 2024-09-10T19:28:21.817 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:28:21.817 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:28:21.817 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:28:21.817 INFO:teuthology.orchestra.run.smithi183.stdout: "heartbeat_front_addrs": { 2024-09-10T19:28:21.817 INFO:teuthology.orchestra.run.smithi183.stdout: "addrvec": [ 2024-09-10T19:28:21.817 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:28:21.817 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v2", 2024-09-10T19:28:21.817 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.191:6828", 2024-09-10T19:28:21.817 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 1422610021 2024-09-10T19:28:21.817 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:28:21.817 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:28:21.818 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v1", 2024-09-10T19:28:21.818 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.191:6829", 2024-09-10T19:28:21.818 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 1422610021 2024-09-10T19:28:21.818 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:28:21.818 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:28:21.818 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:28:21.818 INFO:teuthology.orchestra.run.smithi183.stdout: "public_addr": "172.21.15.191:6825/1422610021", 2024-09-10T19:28:21.818 INFO:teuthology.orchestra.run.smithi183.stdout: "cluster_addr": "172.21.15.191:6827/1422610021", 2024-09-10T19:28:21.818 INFO:teuthology.orchestra.run.smithi183.stdout: "heartbeat_back_addr": "172.21.15.191:6831/1422610021", 2024-09-10T19:28:21.818 INFO:teuthology.orchestra.run.smithi183.stdout: "heartbeat_front_addr": "172.21.15.191:6829/1422610021", 2024-09-10T19:28:21.818 INFO:teuthology.orchestra.run.smithi183.stdout: "state": [ 2024-09-10T19:28:21.818 INFO:teuthology.orchestra.run.smithi183.stdout: "exists", 2024-09-10T19:28:21.818 INFO:teuthology.orchestra.run.smithi183.stdout: "up" 2024-09-10T19:28:21.818 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:28:21.818 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:28:21.819 INFO:teuthology.orchestra.run.smithi183.stdout: ], 2024-09-10T19:28:21.819 INFO:teuthology.orchestra.run.smithi183.stdout: "osd_xinfo": [ 2024-09-10T19:28:21.819 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:28:21.819 INFO:teuthology.orchestra.run.smithi183.stdout: "osd": 0, 2024-09-10T19:28:21.819 INFO:teuthology.orchestra.run.smithi183.stdout: "down_stamp": "0.000000", 2024-09-10T19:28:21.819 INFO:teuthology.orchestra.run.smithi183.stdout: "laggy_probability": 0, 2024-09-10T19:28:21.819 INFO:teuthology.orchestra.run.smithi183.stdout: "laggy_interval": 0, 2024-09-10T19:28:21.819 INFO:teuthology.orchestra.run.smithi183.stdout: "features": 4540138320759226367, 2024-09-10T19:28:21.819 INFO:teuthology.orchestra.run.smithi183.stdout: "old_weight": 0, 2024-09-10T19:28:21.819 INFO:teuthology.orchestra.run.smithi183.stdout: "last_purged_snaps_scrub": "2024-09-10T19:28:07.083034+0000", 2024-09-10T19:28:21.819 INFO:teuthology.orchestra.run.smithi183.stdout: "dead_epoch": 262 2024-09-10T19:28:21.819 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:28:21.819 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:28:21.819 INFO:teuthology.orchestra.run.smithi183.stdout: "osd": 1, 2024-09-10T19:28:21.819 INFO:teuthology.orchestra.run.smithi183.stdout: "down_stamp": "0.000000", 2024-09-10T19:28:21.820 INFO:teuthology.orchestra.run.smithi183.stdout: "laggy_probability": 0, 2024-09-10T19:28:21.820 INFO:teuthology.orchestra.run.smithi183.stdout: "laggy_interval": 0, 2024-09-10T19:28:21.820 INFO:teuthology.orchestra.run.smithi183.stdout: "features": 4540138320759226367, 2024-09-10T19:28:21.820 INFO:teuthology.orchestra.run.smithi183.stdout: "old_weight": 0, 2024-09-10T19:28:21.820 INFO:teuthology.orchestra.run.smithi183.stdout: "last_purged_snaps_scrub": "2024-09-10T19:27:28.059871+0000", 2024-09-10T19:28:21.820 INFO:teuthology.orchestra.run.smithi183.stdout: "dead_epoch": 255 2024-09-10T19:28:21.820 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:28:21.820 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:28:21.820 INFO:teuthology.orchestra.run.smithi183.stdout: "osd": 2, 2024-09-10T19:28:21.820 INFO:teuthology.orchestra.run.smithi183.stdout: "down_stamp": "0.000000", 2024-09-10T19:28:21.820 INFO:teuthology.orchestra.run.smithi183.stdout: "laggy_probability": 0, 2024-09-10T19:28:21.820 INFO:teuthology.orchestra.run.smithi183.stdout: "laggy_interval": 0, 2024-09-10T19:28:21.820 INFO:teuthology.orchestra.run.smithi183.stdout: "features": 4540138320759226367, 2024-09-10T19:28:21.821 INFO:teuthology.orchestra.run.smithi183.stdout: "old_weight": 65536, 2024-09-10T19:28:21.821 INFO:teuthology.orchestra.run.smithi183.stdout: "last_purged_snaps_scrub": "2024-09-10T19:28:04.528297+0000", 2024-09-10T19:28:21.821 INFO:teuthology.orchestra.run.smithi183.stdout: "dead_epoch": 0 2024-09-10T19:28:21.821 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:28:21.821 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:28:21.821 INFO:teuthology.orchestra.run.smithi183.stdout: "osd": 3, 2024-09-10T19:28:21.821 INFO:teuthology.orchestra.run.smithi183.stdout: "down_stamp": "0.000000", 2024-09-10T19:28:21.821 INFO:teuthology.orchestra.run.smithi183.stdout: "laggy_probability": 0, 2024-09-10T19:28:21.821 INFO:teuthology.orchestra.run.smithi183.stdout: "laggy_interval": 0, 2024-09-10T19:28:21.821 INFO:teuthology.orchestra.run.smithi183.stdout: "features": 4540138320759226367, 2024-09-10T19:28:21.821 INFO:teuthology.orchestra.run.smithi183.stdout: "old_weight": 0, 2024-09-10T19:28:21.821 INFO:teuthology.orchestra.run.smithi183.stdout: "last_purged_snaps_scrub": "2024-09-10T19:27:28.035138+0000", 2024-09-10T19:28:21.821 INFO:teuthology.orchestra.run.smithi183.stdout: "dead_epoch": 0 2024-09-10T19:28:21.821 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:28:21.821 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:28:21.822 INFO:teuthology.orchestra.run.smithi183.stdout: "osd": 4, 2024-09-10T19:28:21.822 INFO:teuthology.orchestra.run.smithi183.stdout: "down_stamp": "0.000000", 2024-09-10T19:28:21.822 INFO:teuthology.orchestra.run.smithi183.stdout: "laggy_probability": 0, 2024-09-10T19:28:21.822 INFO:teuthology.orchestra.run.smithi183.stdout: "laggy_interval": 0, 2024-09-10T19:28:21.823 INFO:teuthology.orchestra.run.smithi183.stdout: "features": 4540138320759226367, 2024-09-10T19:28:21.823 INFO:teuthology.orchestra.run.smithi183.stdout: "old_weight": 0, 2024-09-10T19:28:21.823 INFO:teuthology.orchestra.run.smithi183.stdout: "last_purged_snaps_scrub": "2024-09-10T19:28:15.901036+0000", 2024-09-10T19:28:21.823 INFO:teuthology.orchestra.run.smithi183.stdout: "dead_epoch": 0 2024-09-10T19:28:21.823 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:28:21.823 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:28:21.823 INFO:teuthology.orchestra.run.smithi183.stdout: "osd": 5, 2024-09-10T19:28:21.823 INFO:teuthology.orchestra.run.smithi183.stdout: "down_stamp": "0.000000", 2024-09-10T19:28:21.823 INFO:teuthology.orchestra.run.smithi183.stdout: "laggy_probability": 0, 2024-09-10T19:28:21.824 INFO:teuthology.orchestra.run.smithi183.stdout: "laggy_interval": 0, 2024-09-10T19:28:21.824 INFO:teuthology.orchestra.run.smithi183.stdout: "features": 4540138320759226367, 2024-09-10T19:28:21.824 INFO:teuthology.orchestra.run.smithi183.stdout: "old_weight": 0, 2024-09-10T19:28:21.824 INFO:teuthology.orchestra.run.smithi183.stdout: "last_purged_snaps_scrub": "2024-09-10T19:27:24.953283+0000", 2024-09-10T19:28:21.824 INFO:teuthology.orchestra.run.smithi183.stdout: "dead_epoch": 0 2024-09-10T19:28:21.824 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:28:21.824 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:28:21.824 INFO:teuthology.orchestra.run.smithi183.stdout: "osd": 6, 2024-09-10T19:28:21.824 INFO:teuthology.orchestra.run.smithi183.stdout: "down_stamp": "0.000000", 2024-09-10T19:28:21.824 INFO:teuthology.orchestra.run.smithi183.stdout: "laggy_probability": 0, 2024-09-10T19:28:21.824 INFO:teuthology.orchestra.run.smithi183.stdout: "laggy_interval": 0, 2024-09-10T19:28:21.824 INFO:teuthology.orchestra.run.smithi183.stdout: "features": 4540138320759226367, 2024-09-10T19:28:21.824 INFO:teuthology.orchestra.run.smithi183.stdout: "old_weight": 0, 2024-09-10T19:28:21.824 INFO:teuthology.orchestra.run.smithi183.stdout: "last_purged_snaps_scrub": "2024-09-10T19:27:13.126902+0000", 2024-09-10T19:28:21.825 INFO:teuthology.orchestra.run.smithi183.stdout: "dead_epoch": 0 2024-09-10T19:28:21.825 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:28:21.825 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:28:21.825 INFO:teuthology.orchestra.run.smithi183.stdout: "osd": 7, 2024-09-10T19:28:21.825 INFO:teuthology.orchestra.run.smithi183.stdout: "down_stamp": "0.000000", 2024-09-10T19:28:21.825 INFO:teuthology.orchestra.run.smithi183.stdout: "laggy_probability": 0, 2024-09-10T19:28:21.825 INFO:teuthology.orchestra.run.smithi183.stdout: "laggy_interval": 0, 2024-09-10T19:28:21.825 INFO:teuthology.orchestra.run.smithi183.stdout: "features": 4540138320759226367, 2024-09-10T19:28:21.825 INFO:teuthology.orchestra.run.smithi183.stdout: "old_weight": 0, 2024-09-10T19:28:21.825 INFO:teuthology.orchestra.run.smithi183.stdout: "last_purged_snaps_scrub": "2024-09-10T19:27:38.429368+0000", 2024-09-10T19:28:21.825 INFO:teuthology.orchestra.run.smithi183.stdout: "dead_epoch": 0 2024-09-10T19:28:21.825 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:28:21.825 INFO:teuthology.orchestra.run.smithi183.stdout: ], 2024-09-10T19:28:21.825 INFO:teuthology.orchestra.run.smithi183.stdout: "pg_upmap": [ 2024-09-10T19:28:21.826 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:28:21.826 INFO:teuthology.orchestra.run.smithi183.stdout: "pgid": "2.13", 2024-09-10T19:28:21.826 INFO:teuthology.orchestra.run.smithi183.stdout: "osds": [ 2024-09-10T19:28:21.826 INFO:teuthology.orchestra.run.smithi183.stdout: 2, 2024-09-10T19:28:21.826 INFO:teuthology.orchestra.run.smithi183.stdout: 0, 2024-09-10T19:28:21.826 INFO:teuthology.orchestra.run.smithi183.stdout: 5 2024-09-10T19:28:21.826 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:28:21.826 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:28:21.826 INFO:teuthology.orchestra.run.smithi183.stdout: ], 2024-09-10T19:28:21.826 INFO:teuthology.orchestra.run.smithi183.stdout: "pg_upmap_items": [ 2024-09-10T19:28:21.826 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:28:21.826 INFO:teuthology.orchestra.run.smithi183.stdout: "pgid": "2.8", 2024-09-10T19:28:21.826 INFO:teuthology.orchestra.run.smithi183.stdout: "mappings": [ 2024-09-10T19:28:21.826 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:28:21.827 INFO:teuthology.orchestra.run.smithi183.stdout: "from": 0, 2024-09-10T19:28:21.827 INFO:teuthology.orchestra.run.smithi183.stdout: "to": 5 2024-09-10T19:28:21.827 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:28:21.827 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:28:21.827 INFO:teuthology.orchestra.run.smithi183.stdout: "from": 1, 2024-09-10T19:28:21.827 INFO:teuthology.orchestra.run.smithi183.stdout: "to": 3 2024-09-10T19:28:21.827 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:28:21.827 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:28:21.827 INFO:teuthology.orchestra.run.smithi183.stdout: "from": 7, 2024-09-10T19:28:21.827 INFO:teuthology.orchestra.run.smithi183.stdout: "to": 4 2024-09-10T19:28:21.827 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:28:21.827 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:28:21.827 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:28:21.827 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:28:21.828 INFO:teuthology.orchestra.run.smithi183.stdout: "pgid": "2.18", 2024-09-10T19:28:21.828 INFO:teuthology.orchestra.run.smithi183.stdout: "mappings": [ 2024-09-10T19:28:21.828 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:28:21.828 INFO:teuthology.orchestra.run.smithi183.stdout: "from": 7, 2024-09-10T19:28:21.828 INFO:teuthology.orchestra.run.smithi183.stdout: "to": 0 2024-09-10T19:28:21.828 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:28:21.828 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:28:21.828 INFO:teuthology.orchestra.run.smithi183.stdout: "from": 5, 2024-09-10T19:28:21.828 INFO:teuthology.orchestra.run.smithi183.stdout: "to": 4 2024-09-10T19:28:21.828 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:28:21.828 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:28:21.828 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:28:21.828 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:28:21.828 INFO:teuthology.orchestra.run.smithi183.stdout: "pgid": "4.b", 2024-09-10T19:28:21.829 INFO:teuthology.orchestra.run.smithi183.stdout: "mappings": [ 2024-09-10T19:28:21.829 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:28:21.829 INFO:teuthology.orchestra.run.smithi183.stdout: "from": 1, 2024-09-10T19:28:21.829 INFO:teuthology.orchestra.run.smithi183.stdout: "to": 5 2024-09-10T19:28:21.829 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:28:21.829 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:28:21.829 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:28:21.829 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:28:21.829 INFO:teuthology.orchestra.run.smithi183.stdout: "pgid": "4.e", 2024-09-10T19:28:21.829 INFO:teuthology.orchestra.run.smithi183.stdout: "mappings": [ 2024-09-10T19:28:21.829 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:28:21.829 INFO:teuthology.orchestra.run.smithi183.stdout: "from": 1, 2024-09-10T19:28:21.829 INFO:teuthology.orchestra.run.smithi183.stdout: "to": 5 2024-09-10T19:28:21.829 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:28:21.829 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:28:21.830 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:28:21.830 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:28:21.830 INFO:teuthology.orchestra.run.smithi183.stdout: "pgid": "5.15", 2024-09-10T19:28:21.830 INFO:teuthology.orchestra.run.smithi183.stdout: "mappings": [ 2024-09-10T19:28:21.830 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:28:21.830 INFO:teuthology.orchestra.run.smithi183.stdout: "from": 1, 2024-09-10T19:28:21.830 INFO:teuthology.orchestra.run.smithi183.stdout: "to": 3 2024-09-10T19:28:21.830 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:28:21.830 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:28:21.830 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:28:21.830 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:28:21.830 INFO:teuthology.orchestra.run.smithi183.stdout: "pgid": "6.1b", 2024-09-10T19:28:21.830 INFO:teuthology.orchestra.run.smithi183.stdout: "mappings": [ 2024-09-10T19:28:21.830 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:28:21.831 INFO:teuthology.orchestra.run.smithi183.stdout: "from": 3, 2024-09-10T19:28:21.831 INFO:teuthology.orchestra.run.smithi183.stdout: "to": 0 2024-09-10T19:28:21.831 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:28:21.831 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:28:21.831 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:28:21.831 INFO:teuthology.orchestra.run.smithi183.stdout: ], 2024-09-10T19:28:21.831 INFO:teuthology.orchestra.run.smithi183.stdout: "pg_temp": [], 2024-09-10T19:28:21.831 INFO:teuthology.orchestra.run.smithi183.stdout: "primary_temp": [], 2024-09-10T19:28:21.831 INFO:teuthology.orchestra.run.smithi183.stdout: "blocklist": { 2024-09-10T19:28:21.831 INFO:teuthology.orchestra.run.smithi183.stdout: "172.21.15.183:6801/3311783362": "2024-09-11T19:10:43.576943+0000", 2024-09-10T19:28:21.831 INFO:teuthology.orchestra.run.smithi183.stdout: "172.21.15.183:0/4088553922": "2024-09-11T19:10:43.576943+0000", 2024-09-10T19:28:21.831 INFO:teuthology.orchestra.run.smithi183.stdout: "172.21.15.183:0/3528769414": "2024-09-11T19:05:48.153879+0000", 2024-09-10T19:28:21.831 INFO:teuthology.orchestra.run.smithi183.stdout: "172.21.15.183:0/1766036658": "2024-09-11T19:05:48.153879+0000", 2024-09-10T19:28:21.831 INFO:teuthology.orchestra.run.smithi183.stdout: "172.21.15.183:0/3044170088": "2024-09-11T19:05:25.702354+0000", 2024-09-10T19:28:21.832 INFO:teuthology.orchestra.run.smithi183.stdout: "172.21.15.183:0/3037333778": "2024-09-11T19:05:25.702354+0000", 2024-09-10T19:28:21.832 INFO:teuthology.orchestra.run.smithi183.stdout: "172.21.15.183:6801/3747320570": "2024-09-11T19:05:25.702354+0000", 2024-09-10T19:28:21.832 INFO:teuthology.orchestra.run.smithi183.stdout: "172.21.15.183:0/3982386906": "2024-09-11T19:10:43.576943+0000", 2024-09-10T19:28:21.832 INFO:teuthology.orchestra.run.smithi183.stdout: "172.21.15.183:0/2436014455": "2024-09-11T19:05:25.702354+0000", 2024-09-10T19:28:21.832 INFO:teuthology.orchestra.run.smithi183.stdout: "172.21.15.183:6801/659745765": "2024-09-11T19:05:48.153879+0000", 2024-09-10T19:28:21.832 INFO:teuthology.orchestra.run.smithi183.stdout: "172.21.15.183:6800/3747320570": "2024-09-11T19:05:25.702354+0000", 2024-09-10T19:28:21.832 INFO:teuthology.orchestra.run.smithi183.stdout: "172.21.15.183:0/1652576282": "2024-09-11T19:10:43.576943+0000", 2024-09-10T19:28:21.832 INFO:teuthology.orchestra.run.smithi183.stdout: "172.21.15.183:6800/3311783362": "2024-09-11T19:10:43.576943+0000", 2024-09-10T19:28:21.832 INFO:teuthology.orchestra.run.smithi183.stdout: "172.21.15.183:0/1012899549": "2024-09-11T19:05:48.153879+0000", 2024-09-10T19:28:21.832 INFO:teuthology.orchestra.run.smithi183.stdout: "172.21.15.183:6800/659745765": "2024-09-11T19:05:48.153879+0000", 2024-09-10T19:28:21.832 INFO:teuthology.orchestra.run.smithi183.stdout: "172.21.15.183:0/412044400": "2024-09-11T19:10:43.576943+0000", 2024-09-10T19:28:21.832 INFO:teuthology.orchestra.run.smithi183.stdout: "172.21.15.183:0/3636171401": "2024-09-11T19:10:43.576943+0000" 2024-09-10T19:28:21.832 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:28:21.832 INFO:teuthology.orchestra.run.smithi183.stdout: "range_blocklist": {}, 2024-09-10T19:28:21.833 INFO:teuthology.orchestra.run.smithi183.stdout: "erasure_code_profiles": { 2024-09-10T19:28:21.833 INFO:teuthology.orchestra.run.smithi183.stdout: "default": { 2024-09-10T19:28:21.833 INFO:teuthology.orchestra.run.smithi183.stdout: "crush-failure-domain": "osd", 2024-09-10T19:28:21.833 INFO:teuthology.orchestra.run.smithi183.stdout: "k": "2", 2024-09-10T19:28:21.833 INFO:teuthology.orchestra.run.smithi183.stdout: "m": "1", 2024-09-10T19:28:21.833 INFO:teuthology.orchestra.run.smithi183.stdout: "plugin": "jerasure", 2024-09-10T19:28:21.833 INFO:teuthology.orchestra.run.smithi183.stdout: "technique": "reed_sol_van" 2024-09-10T19:28:21.833 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:28:21.833 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:28:21.833 INFO:teuthology.orchestra.run.smithi183.stdout: "removed_snaps_queue": [], 2024-09-10T19:28:21.833 INFO:teuthology.orchestra.run.smithi183.stdout: "new_removed_snaps": [], 2024-09-10T19:28:21.833 INFO:teuthology.orchestra.run.smithi183.stdout: "new_purged_snaps": [], 2024-09-10T19:28:21.833 INFO:teuthology.orchestra.run.smithi183.stdout: "crush_node_flags": {}, 2024-09-10T19:28:21.833 INFO:teuthology.orchestra.run.smithi183.stdout: "device_class_flags": {}, 2024-09-10T19:28:21.834 INFO:teuthology.orchestra.run.smithi183.stdout: "stretch_mode": { 2024-09-10T19:28:21.834 INFO:teuthology.orchestra.run.smithi183.stdout: "stretch_mode_enabled": false, 2024-09-10T19:28:21.834 INFO:teuthology.orchestra.run.smithi183.stdout: "stretch_bucket_count": 0, 2024-09-10T19:28:21.834 INFO:teuthology.orchestra.run.smithi183.stdout: "degraded_stretch_mode": 0, 2024-09-10T19:28:21.834 INFO:teuthology.orchestra.run.smithi183.stdout: "recovering_stretch_mode": 0, 2024-09-10T19:28:21.834 INFO:teuthology.orchestra.run.smithi183.stdout: "stretch_mode_bucket": 0 2024-09-10T19:28:21.834 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:28:21.834 INFO:teuthology.orchestra.run.smithi183.stdout:} 2024-09-10T19:28:21.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:28:21.835 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:28:21.835 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:28:21.936 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:28:21.972 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:28:21.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:28:21.973 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:28:22.062 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: 28 16 60783 60767 135.618 99.6875 0.00623739 0.0073693 2024-09-10T19:28:22.074 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:28:22.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:28:22.111 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:28:22.111 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:28:22.212 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:28:22.249 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:28:22.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:28:22.250 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:28:22.267 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:28:21 smithi183 ceph-mon[29736]: 5.b scrub starts 2024-09-10T19:28:22.267 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:28:21 smithi183 ceph-mon[29736]: 5.b scrub ok 2024-09-10T19:28:22.267 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:28:21 smithi183 ceph-mon[29736]: 4.a scrub starts 2024-09-10T19:28:22.268 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:28:21 smithi183 ceph-mon[29736]: 4.a scrub ok 2024-09-10T19:28:22.268 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:28:21 smithi183 ceph-mon[29736]: from='client.? 172.21.15.183:0/4201735596' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-09-10T19:28:22.268 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:28:21 smithi183 ceph-mon[41562]: 5.b scrub starts 2024-09-10T19:28:22.268 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:28:21 smithi183 ceph-mon[41562]: 5.b scrub ok 2024-09-10T19:28:22.269 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:28:21 smithi183 ceph-mon[41562]: 4.a scrub starts 2024-09-10T19:28:22.269 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:28:21 smithi183 ceph-mon[41562]: 4.a scrub ok 2024-09-10T19:28:22.269 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:28:21 smithi183 ceph-mon[41562]: from='client.? 172.21.15.183:0/4201735596' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-09-10T19:28:22.351 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:28:22.381 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:28:22.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:28:22.382 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:28:22.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:28:21 smithi191 ceph-mon[32884]: 5.b scrub starts 2024-09-10T19:28:22.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:28:21 smithi191 ceph-mon[32884]: 5.b scrub ok 2024-09-10T19:28:22.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:28:21 smithi191 ceph-mon[32884]: 4.a scrub starts 2024-09-10T19:28:22.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:28:21 smithi191 ceph-mon[32884]: 4.a scrub ok 2024-09-10T19:28:22.467 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:28:21 smithi191 ceph-mon[32884]: from='client.? 172.21.15.183:0/4201735596' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-09-10T19:28:22.483 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:28:22.518 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:28:22.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:28:22.519 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:28:22.620 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:28:22.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:28:22.657 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:28:22.657 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:28:22.758 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:28:22.792 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:28:22.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:28:22.794 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:28:22.894 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:28:22.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:28:22.933 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:28:22.933 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:28:23.034 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:28:23.062 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: 29 16 62323 62307 134.26 96.25 0.0085933 0.00744364 2024-09-10T19:28:23.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:28:23.068 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:28:23.068 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:28:23.163 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:28:22 smithi183 ceph-mon[29736]: 2.1 deep-scrub starts 2024-09-10T19:28:23.164 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:28:22 smithi183 ceph-mon[29736]: 2.1 deep-scrub ok 2024-09-10T19:28:23.164 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:28:22 smithi183 ceph-mon[29736]: 4.18 deep-scrub starts 2024-09-10T19:28:23.164 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:28:22 smithi183 ceph-mon[29736]: 4.18 deep-scrub ok 2024-09-10T19:28:23.164 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:28:22 smithi183 ceph-mon[29736]: 1.1 scrub starts 2024-09-10T19:28:23.164 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:28:22 smithi183 ceph-mon[29736]: 1.1 scrub ok 2024-09-10T19:28:23.164 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:28:22 smithi183 ceph-mon[29736]: pgmap v843: 158 pgs: 158 active+clean; 3.5 GiB data, 15 GiB used, 610 GiB / 626 GiB avail; 733 B/s rd, 126 MiB/s wr, 2.02k op/s 2024-09-10T19:28:23.164 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:28:22 smithi183 ceph-mon[29736]: 6.4 scrub starts 2024-09-10T19:28:23.164 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:28:22 smithi183 ceph-mon[29736]: 6.4 scrub ok 2024-09-10T19:28:23.164 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:28:22 smithi183 ceph-mon[29736]: 6.1a scrub starts 2024-09-10T19:28:23.164 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:28:22 smithi183 ceph-mon[29736]: 6.1a scrub ok 2024-09-10T19:28:23.165 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:28:22 smithi183 ceph-mon[41562]: 2.1 deep-scrub starts 2024-09-10T19:28:23.165 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:28:22 smithi183 ceph-mon[41562]: 2.1 deep-scrub ok 2024-09-10T19:28:23.165 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:28:22 smithi183 ceph-mon[41562]: 4.18 deep-scrub starts 2024-09-10T19:28:23.165 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:28:22 smithi183 ceph-mon[41562]: 4.18 deep-scrub ok 2024-09-10T19:28:23.165 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:28:22 smithi183 ceph-mon[41562]: 1.1 scrub starts 2024-09-10T19:28:23.165 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:28:22 smithi183 ceph-mon[41562]: 1.1 scrub ok 2024-09-10T19:28:23.165 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:28:22 smithi183 ceph-mon[41562]: pgmap v843: 158 pgs: 158 active+clean; 3.5 GiB data, 15 GiB used, 610 GiB / 626 GiB avail; 733 B/s rd, 126 MiB/s wr, 2.02k op/s 2024-09-10T19:28:23.165 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:28:22 smithi183 ceph-mon[41562]: 6.4 scrub starts 2024-09-10T19:28:23.165 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:28:22 smithi183 ceph-mon[41562]: 6.4 scrub ok 2024-09-10T19:28:23.165 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:28:22 smithi183 ceph-mon[41562]: 6.1a scrub starts 2024-09-10T19:28:23.166 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:28:22 smithi183 ceph-mon[41562]: 6.1a scrub ok 2024-09-10T19:28:23.169 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:28:23.205 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:28:23.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:28:23.206 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:28:23.307 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:28:23.336 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:28:23.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:28:23.338 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:28:23.438 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:28:23.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:28:22 smithi191 ceph-mon[32884]: 2.1 deep-scrub starts 2024-09-10T19:28:23.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:28:22 smithi191 ceph-mon[32884]: 2.1 deep-scrub ok 2024-09-10T19:28:23.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:28:22 smithi191 ceph-mon[32884]: 4.18 deep-scrub starts 2024-09-10T19:28:23.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:28:22 smithi191 ceph-mon[32884]: 4.18 deep-scrub ok 2024-09-10T19:28:23.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:28:22 smithi191 ceph-mon[32884]: 1.1 scrub starts 2024-09-10T19:28:23.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:28:22 smithi191 ceph-mon[32884]: 1.1 scrub ok 2024-09-10T19:28:23.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:28:22 smithi191 ceph-mon[32884]: pgmap v843: 158 pgs: 158 active+clean; 3.5 GiB data, 15 GiB used, 610 GiB / 626 GiB avail; 733 B/s rd, 126 MiB/s wr, 2.02k op/s 2024-09-10T19:28:23.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:28:22 smithi191 ceph-mon[32884]: 6.4 scrub starts 2024-09-10T19:28:23.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:28:22 smithi191 ceph-mon[32884]: 6.4 scrub ok 2024-09-10T19:28:23.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:28:22 smithi191 ceph-mon[32884]: 6.1a scrub starts 2024-09-10T19:28:23.467 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:28:22 smithi191 ceph-mon[32884]: 6.1a scrub ok 2024-09-10T19:28:23.470 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:28:23.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:28:23.471 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:28:23.572 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:28:23.612 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:28:23.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:28:23.613 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:28:23.713 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:28:23.750 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:28:23.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:28:23.751 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:28:23.852 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:28:23.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:28:23.892 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:28:23.893 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:28:23.993 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:28:24.031 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:28:24.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:28:24.032 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:28:24.062 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: 30 16 63793 63777 132.847 91.875 0.00824036 0.00752095 2024-09-10T19:28:24.132 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:28:24.178 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:28:24.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:28:24.179 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:28:24.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:28:24 smithi183 ceph-mon[29736]: 4.4 deep-scrub starts 2024-09-10T19:28:24.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:28:24 smithi183 ceph-mon[29736]: 4.4 deep-scrub ok 2024-09-10T19:28:24.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:28:24 smithi183 ceph-mon[29736]: 6.1c scrub starts 2024-09-10T19:28:24.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:28:24 smithi183 ceph-mon[29736]: 6.1c scrub ok 2024-09-10T19:28:24.181 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:28:24 smithi183 ceph-mon[41562]: 4.4 deep-scrub starts 2024-09-10T19:28:24.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:28:24 smithi183 ceph-mon[41562]: 4.4 deep-scrub ok 2024-09-10T19:28:24.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:28:24 smithi183 ceph-mon[41562]: 6.1c scrub starts 2024-09-10T19:28:24.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:28:24 smithi183 ceph-mon[41562]: 6.1c scrub ok 2024-09-10T19:28:24.280 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:28:24.315 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:28:24.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:28:24.317 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:28:24.417 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:28:24.449 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:28:24.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:28:24.450 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:28:24.465 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:28:24 smithi191 ceph-mon[32884]: 4.4 deep-scrub starts 2024-09-10T19:28:24.465 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:28:24 smithi191 ceph-mon[32884]: 4.4 deep-scrub ok 2024-09-10T19:28:24.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:28:24 smithi191 ceph-mon[32884]: 6.1c scrub starts 2024-09-10T19:28:24.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:28:24 smithi191 ceph-mon[32884]: 6.1c scrub ok 2024-09-10T19:28:24.550 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:28:24.578 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:28:24.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:28:24.579 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:28:24.680 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:28:24.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:28:24.726 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:28:24.726 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:28:24.827 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:28:24.860 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:28:24.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:28:24.861 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:28:24.962 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:28:24.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:28:24.995 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:28:24.996 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:28:25.062 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: 31 16 65125 65109 131.247 83.25 0.00704543 0.00761423 2024-09-10T19:28:25.097 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:28:25.122 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:28:25.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:28:25.123 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:28:25.223 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:28:25.263 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:28:25.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:28:25.264 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:28:25.365 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:28:25.401 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:28:25.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:28:25.402 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:28:25.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:28:25 smithi183 ceph-mon[29736]: 2.1e deep-scrub starts 2024-09-10T19:28:25.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:28:25 smithi183 ceph-mon[29736]: 2.1e deep-scrub ok 2024-09-10T19:28:25.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:28:25 smithi183 ceph-mon[29736]: pgmap v844: 158 pgs: 158 active+clean; 3.6 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 1.2 KiB/s rd, 77 MiB/s wr, 1.23k op/s 2024-09-10T19:28:25.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:28:25 smithi183 ceph-mon[29736]: 2.9 deep-scrub starts 2024-09-10T19:28:25.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:28:25 smithi183 ceph-mon[29736]: 2.9 deep-scrub ok 2024-09-10T19:28:25.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:28:25 smithi183 ceph-mon[29736]: 4.1 deep-scrub starts 2024-09-10T19:28:25.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:28:25 smithi183 ceph-mon[29736]: 4.1 deep-scrub ok 2024-09-10T19:28:25.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:28:25 smithi183 ceph-mon[41562]: 2.1e deep-scrub starts 2024-09-10T19:28:25.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:28:25 smithi183 ceph-mon[41562]: 2.1e deep-scrub ok 2024-09-10T19:28:25.433 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:28:25 smithi183 ceph-mon[41562]: pgmap v844: 158 pgs: 158 active+clean; 3.6 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 1.2 KiB/s rd, 77 MiB/s wr, 1.23k op/s 2024-09-10T19:28:25.433 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:28:25 smithi183 ceph-mon[41562]: 2.9 deep-scrub starts 2024-09-10T19:28:25.433 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:28:25 smithi183 ceph-mon[41562]: 2.9 deep-scrub ok 2024-09-10T19:28:25.433 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:28:25 smithi183 ceph-mon[41562]: 4.1 deep-scrub starts 2024-09-10T19:28:25.433 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:28:25 smithi183 ceph-mon[41562]: 4.1 deep-scrub ok 2024-09-10T19:28:25.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:28:25 smithi191 ceph-mon[32884]: 2.1e deep-scrub starts 2024-09-10T19:28:25.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:28:25 smithi191 ceph-mon[32884]: 2.1e deep-scrub ok 2024-09-10T19:28:25.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:28:25 smithi191 ceph-mon[32884]: pgmap v844: 158 pgs: 158 active+clean; 3.6 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 1.2 KiB/s rd, 77 MiB/s wr, 1.23k op/s 2024-09-10T19:28:25.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:28:25 smithi191 ceph-mon[32884]: 2.9 deep-scrub starts 2024-09-10T19:28:25.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:28:25 smithi191 ceph-mon[32884]: 2.9 deep-scrub ok 2024-09-10T19:28:25.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:28:25 smithi191 ceph-mon[32884]: 4.1 deep-scrub starts 2024-09-10T19:28:25.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:28:25 smithi191 ceph-mon[32884]: 4.1 deep-scrub ok 2024-09-10T19:28:25.503 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:28:25.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:28:25.539 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:28:25.539 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:28:25.640 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:28:25.665 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:28:25.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:28:25.666 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:28:25.767 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:28:25.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:28:25.803 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:28:25.803 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:28:25.904 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:28:25.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:28:25.937 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:28:25.938 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:28:26.038 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:28:26.062 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: 32 15 66464 66449 129.762 83.75 0.0180903 0.00770148 2024-09-10T19:28:26.070 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:28:26.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:28:26.071 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:28:26.172 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:28:26.206 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:28:26.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:28:26.208 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:28:26.308 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:28:26.349 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:28:26.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:28:26.350 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:28:26.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:28:26 smithi183 ceph-mon[29736]: 6.9 scrub starts 2024-09-10T19:28:26.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:28:26 smithi183 ceph-mon[29736]: 6.9 scrub ok 2024-09-10T19:28:26.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:28:26 smithi183 ceph-mon[29736]: 6.3 deep-scrub starts 2024-09-10T19:28:26.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:28:26 smithi183 ceph-mon[29736]: 6.3 deep-scrub ok 2024-09-10T19:28:26.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:28:26 smithi183 ceph-mon[41562]: 6.9 scrub starts 2024-09-10T19:28:26.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:28:26 smithi183 ceph-mon[41562]: 6.9 scrub ok 2024-09-10T19:28:26.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:28:26 smithi183 ceph-mon[41562]: 6.3 deep-scrub starts 2024-09-10T19:28:26.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:28:26 smithi183 ceph-mon[41562]: 6.3 deep-scrub ok 2024-09-10T19:28:26.451 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:28:26.465 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:28:26 smithi191 ceph-mon[32884]: 6.9 scrub starts 2024-09-10T19:28:26.465 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:28:26 smithi191 ceph-mon[32884]: 6.9 scrub ok 2024-09-10T19:28:26.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:28:26 smithi191 ceph-mon[32884]: 6.3 deep-scrub starts 2024-09-10T19:28:26.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:28:26 smithi191 ceph-mon[32884]: 6.3 deep-scrub ok 2024-09-10T19:28:26.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:28:26.486 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:28:26.486 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:28:26.587 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:28:26.624 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:28:26.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:28:26.625 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:28:26.725 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:28:26.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:28:26.762 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:28:26.762 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:28:26.863 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:28:26.886 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:28:26.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:28:26.887 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:28:26.988 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:28:27.023 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:28:27.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:28:27.024 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:28:27.062 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: 33 16 67829 67813 128.413 85.25 0.00991328 0.00778263 2024-09-10T19:28:27.124 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:28:27.157 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:28:27.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:28:27.158 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:28:27.259 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:28:27.294 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:28:27.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:28:27.295 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:28:27.367 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:28:27 smithi191 ceph-mon[32884]: 2.3 scrub starts 2024-09-10T19:28:27.367 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:28:27 smithi191 ceph-mon[32884]: 2.3 scrub ok 2024-09-10T19:28:27.367 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:28:27 smithi191 ceph-mon[32884]: pgmap v845: 158 pgs: 158 active+clean; 3.9 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 1.0 KiB/s rd, 100 MiB/s wr, 1.59k op/s 2024-09-10T19:28:27.367 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:28:27 smithi191 ceph-mon[32884]: 5.5 scrub starts 2024-09-10T19:28:27.367 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:28:27 smithi191 ceph-mon[32884]: 5.5 scrub ok 2024-09-10T19:28:27.367 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:28:27 smithi191 ceph-mon[32884]: 4.f scrub starts 2024-09-10T19:28:27.368 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:28:27 smithi191 ceph-mon[32884]: 4.f scrub ok 2024-09-10T19:28:27.368 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:28:27 smithi191 ceph-mon[32884]: 1.7 scrub starts 2024-09-10T19:28:27.368 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:28:27 smithi191 ceph-mon[32884]: 1.7 scrub ok 2024-09-10T19:28:27.396 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:28:27.429 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:28:27.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:28:27.430 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:28:27.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:28:27 smithi183 ceph-mon[29736]: 2.3 scrub starts 2024-09-10T19:28:27.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:28:27 smithi183 ceph-mon[29736]: 2.3 scrub ok 2024-09-10T19:28:27.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:28:27 smithi183 ceph-mon[29736]: pgmap v845: 158 pgs: 158 active+clean; 3.9 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 1.0 KiB/s rd, 100 MiB/s wr, 1.59k op/s 2024-09-10T19:28:27.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:28:27 smithi183 ceph-mon[29736]: 5.5 scrub starts 2024-09-10T19:28:27.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:28:27 smithi183 ceph-mon[29736]: 5.5 scrub ok 2024-09-10T19:28:27.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:28:27 smithi183 ceph-mon[29736]: 4.f scrub starts 2024-09-10T19:28:27.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:28:27 smithi183 ceph-mon[29736]: 4.f scrub ok 2024-09-10T19:28:27.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:28:27 smithi183 ceph-mon[29736]: 1.7 scrub starts 2024-09-10T19:28:27.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:28:27 smithi183 ceph-mon[29736]: 1.7 scrub ok 2024-09-10T19:28:27.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:28:27 smithi183 ceph-mon[41562]: 2.3 scrub starts 2024-09-10T19:28:27.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:28:27 smithi183 ceph-mon[41562]: 2.3 scrub ok 2024-09-10T19:28:27.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:28:27 smithi183 ceph-mon[41562]: pgmap v845: 158 pgs: 158 active+clean; 3.9 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 1.0 KiB/s rd, 100 MiB/s wr, 1.59k op/s 2024-09-10T19:28:27.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:28:27 smithi183 ceph-mon[41562]: 5.5 scrub starts 2024-09-10T19:28:27.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:28:27 smithi183 ceph-mon[41562]: 5.5 scrub ok 2024-09-10T19:28:27.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:28:27 smithi183 ceph-mon[41562]: 4.f scrub starts 2024-09-10T19:28:27.433 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:28:27 smithi183 ceph-mon[41562]: 4.f scrub ok 2024-09-10T19:28:27.433 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:28:27 smithi183 ceph-mon[41562]: 1.7 scrub starts 2024-09-10T19:28:27.433 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:28:27 smithi183 ceph-mon[41562]: 1.7 scrub ok 2024-09-10T19:28:27.531 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:28:27.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:28:27.557 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:28:27.557 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:28:27.658 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:28:27.692 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:28:27.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:28:27.693 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:28:27.715 INFO:journalctl@ceph.mgr.x.smithi191.stdout:Sep 10 19:28:27 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-x[34967]: ::ffff:172.21.15.191 - - [10/Sep/2024:19:28:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T19:28:27.794 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:28:27.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:28:27.827 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:28:27.827 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:28:27.928 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:28:27.959 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:28:27.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:28:27.960 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:28:28.061 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:28:28.062 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: 34 16 69048 69032 126.877 76.1875 0.00794578 0.0078769 2024-09-10T19:28:28.100 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:28:28.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:28:28.101 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:28:28.202 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:28:28.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:28:28 smithi191 ceph-mon[32884]: 5.2 scrub starts 2024-09-10T19:28:28.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:28:28 smithi191 ceph-mon[32884]: 5.2 scrub ok 2024-09-10T19:28:28.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:28:28.246 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:28:28.246 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:28:28.268 INFO:tasks.thrashosds.thrasher:j is {'epoch': 384, 'fsid': '320f6df8-6fa7-11ef-bcea-c7b262605968', 'created': '2024-09-10T19:05:03.688531+0000', 'modified': '2024-09-10T19:28:19.941070+0000', 'last_up_change': '2024-09-10T19:24:22.870873+0000', 'last_in_change': '2024-09-10T19:27:51.975891+0000', 'flags': 'sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5799936, 'flags_set': ['pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 21, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 11, 'max_osd': 8, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'quincy', 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-10T19:08:00.456786+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 11, 'pg_placement_num': 11, 'pg_placement_num_target': 11, 'pg_num_target': 11, 'pg_num_pending': 11, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '271', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '242', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}, {'pool': 2, 'pool_name': '.rgw.root', 'create_time': '2024-09-10T19:10:05.993763+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '60', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}}, {'pool': 3, 'pool_name': 'datapool', 'create_time': '2024-09-10T19:10:06.224267+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 3, 'pg_placement_num': 3, 'pg_placement_num_target': 3, 'pg_num_target': 3, 'pg_num_pending': 3, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '63', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 3, 'snap_epoch': 63, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 4, 'pool_name': 'default.rgw.log', 'create_time': '2024-09-10T19:10:07.958284+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '62', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}}, {'pool': 5, 'pool_name': 'default.rgw.control', 'create_time': '2024-09-10T19:10:09.933596+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '64', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}}, {'pool': 6, 'pool_name': 'default.rgw.meta', 'create_time': '2024-09-10T19:10:11.993712+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '66', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4}, 'application_metadata': {'rgw': {}}}, {'pool': 11, 'pool_name': 'unique_pool_4', 'create_time': '2024-09-10T19:27:45.303826+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 16, 'pg_placement_num': 16, 'pg_placement_num_target': 16, 'pg_num_target': 16, 'pg_num_pending': 16, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '374', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}}], 'osds': [{'osd': 0, 'uuid': '9d323010-394a-49aa-ba9b-244d87faf36e', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 8, 'last_clean_end': 262, 'up_from': 301, 'up_thru': 374, 'down_at': 263, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.183:6810', 'nonce': 3346539414}, {'type': 'v1', 'addr': '172.21.15.183:6811', 'nonce': 3346539414}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.183:6812', 'nonce': 3346539414}, {'type': 'v1', 'addr': '172.21.15.183:6813', 'nonce': 3346539414}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.183:6816', 'nonce': 3346539414}, {'type': 'v1', 'addr': '172.21.15.183:6817', 'nonce': 3346539414}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.183:6814', 'nonce': 3346539414}, {'type': 'v1', 'addr': '172.21.15.183:6815', 'nonce': 3346539414}]}, 'public_addr': '172.21.15.183:6811/3346539414', 'cluster_addr': '172.21.15.183:6813/3346539414', 'heartbeat_back_addr': '172.21.15.183:6817/3346539414', 'heartbeat_front_addr': '172.21.15.183:6815/3346539414', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': '71a6c5dc-5109-4284-8d20-187d8e7afc9e', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 138, 'last_clean_end': 255, 'up_from': 276, 'up_thru': 374, 'down_at': 256, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.183:6802', 'nonce': 2764394151}, {'type': 'v1', 'addr': '172.21.15.183:6803', 'nonce': 2764394151}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.183:6804', 'nonce': 2764394151}, {'type': 'v1', 'addr': '172.21.15.183:6805', 'nonce': 2764394151}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.183:6808', 'nonce': 2764394151}, {'type': 'v1', 'addr': '172.21.15.183:6809', 'nonce': 2764394151}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.183:6806', 'nonce': 2764394151}, {'type': 'v1', 'addr': '172.21.15.183:6807', 'nonce': 2764394151}]}, 'public_addr': '172.21.15.183:6803/2764394151', 'cluster_addr': '172.21.15.183:6805/2764394151', 'heartbeat_back_addr': '172.21.15.183:6809/2764394151', 'heartbeat_front_addr': '172.21.15.183:6807/2764394151', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': 'fc310d5b-b89b-4655-9989-7e10ba356509', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 18, 'up_thru': 148, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.183:6818', 'nonce': 3730013786}, {'type': 'v1', 'addr': '172.21.15.183:6819', 'nonce': 3730013786}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.183:6820', 'nonce': 3730013786}, {'type': 'v1', 'addr': '172.21.15.183:6821', 'nonce': 3730013786}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.183:6824', 'nonce': 3730013786}, {'type': 'v1', 'addr': '172.21.15.183:6825', 'nonce': 3730013786}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.183:6822', 'nonce': 3730013786}, {'type': 'v1', 'addr': '172.21.15.183:6823', 'nonce': 3730013786}]}, 'public_addr': '172.21.15.183:6819/3730013786', 'cluster_addr': '172.21.15.183:6821/3730013786', 'heartbeat_back_addr': '172.21.15.183:6825/3730013786', 'heartbeat_front_addr': '172.21.15.183:6823/3730013786', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': '7fd07a95-ad22-43f5-a968-539552283e89', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 26, 'up_thru': 374, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.183:6826', 'nonce': 3522836984}, {'type': 'v1', 'addr': '172.21.15.183:6827', 'nonce': 3522836984}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.183:6828', 'nonce': 3522836984}, {'type': 'v1', 'addr': '172.21.15.183:6829', 'nonce': 3522836984}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.183:6832', 'nonce': 3522836984}, {'type': 'v1', 'addr': '172.21.15.183:6833', 'nonce': 3522836984}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.183:6830', 'nonce': 3522836984}, {'type': 'v1', 'addr': '172.21.15.183:6831', 'nonce': 3522836984}]}, 'public_addr': '172.21.15.183:6827/3522836984', 'cluster_addr': '172.21.15.183:6829/3522836984', 'heartbeat_back_addr': '172.21.15.183:6833/3522836984', 'heartbeat_front_addr': '172.21.15.183:6831/3522836984', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': '631083fb-e503-44da-9b6b-154e88539ee7', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 32, 'up_thru': 374, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.191:6800', 'nonce': 2674828539}, {'type': 'v1', 'addr': '172.21.15.191:6801', 'nonce': 2674828539}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.191:6802', 'nonce': 2674828539}, {'type': 'v1', 'addr': '172.21.15.191:6803', 'nonce': 2674828539}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.191:6806', 'nonce': 2674828539}, {'type': 'v1', 'addr': '172.21.15.191:6807', 'nonce': 2674828539}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.191:6804', 'nonce': 2674828539}, {'type': 'v1', 'addr': '172.21.15.191:6805', 'nonce': 2674828539}]}, 'public_addr': '172.21.15.191:6801/2674828539', 'cluster_addr': '172.21.15.191:6803/2674828539', 'heartbeat_back_addr': '172.21.15.191:6807/2674828539', 'heartbeat_front_addr': '172.21.15.191:6805/2674828539', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': '6b963695-3eca-4e6e-9207-05b8010ebff9', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 39, 'up_thru': 374, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.191:6808', 'nonce': 681733378}, {'type': 'v1', 'addr': '172.21.15.191:6809', 'nonce': 681733378}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.191:6810', 'nonce': 681733378}, {'type': 'v1', 'addr': '172.21.15.191:6811', 'nonce': 681733378}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.191:6814', 'nonce': 681733378}, {'type': 'v1', 'addr': '172.21.15.191:6815', 'nonce': 681733378}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.191:6812', 'nonce': 681733378}, {'type': 'v1', 'addr': '172.21.15.191:6813', 'nonce': 681733378}]}, 'public_addr': '172.21.15.191:6809/681733378', 'cluster_addr': '172.21.15.191:6811/681733378', 'heartbeat_back_addr': '172.21.15.191:6815/681733378', 'heartbeat_front_addr': '172.21.15.191:6813/681733378', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': '2dc9754f-6aef-4e19-8605-2f1591921811', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 47, 'up_thru': 374, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.191:6816', 'nonce': 2825813477}, {'type': 'v1', 'addr': '172.21.15.191:6817', 'nonce': 2825813477}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.191:6818', 'nonce': 2825813477}, {'type': 'v1', 'addr': '172.21.15.191:6819', 'nonce': 2825813477}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.191:6822', 'nonce': 2825813477}, {'type': 'v1', 'addr': '172.21.15.191:6823', 'nonce': 2825813477}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.191:6820', 'nonce': 2825813477}, {'type': 'v1', 'addr': '172.21.15.191:6821', 'nonce': 2825813477}]}, 'public_addr': '172.21.15.191:6817/2825813477', 'cluster_addr': '172.21.15.191:6819/2825813477', 'heartbeat_back_addr': '172.21.15.191:6823/2825813477', 'heartbeat_front_addr': '172.21.15.191:6821/2825813477', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': '1a9a299e-d5ba-4b9a-85d5-7f6a5bc754e1', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 54, 'up_thru': 374, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.191:6824', 'nonce': 1422610021}, {'type': 'v1', 'addr': '172.21.15.191:6825', 'nonce': 1422610021}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.191:6826', 'nonce': 1422610021}, {'type': 'v1', 'addr': '172.21.15.191:6827', 'nonce': 1422610021}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.191:6830', 'nonce': 1422610021}, {'type': 'v1', 'addr': '172.21.15.191:6831', 'nonce': 1422610021}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.191:6828', 'nonce': 1422610021}, {'type': 'v1', 'addr': '172.21.15.191:6829', 'nonce': 1422610021}]}, 'public_addr': '172.21.15.191:6825/1422610021', 'cluster_addr': '172.21.15.191:6827/1422610021', 'heartbeat_back_addr': '172.21.15.191:6831/1422610021', 'heartbeat_front_addr': '172.21.15.191:6829/1422610021', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138320759226367, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-10T19:28:07.083034+0000', 'dead_epoch': 262}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138320759226367, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-10T19:27:28.059871+0000', 'dead_epoch': 255}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138320759226367, 'old_weight': 65536, 'last_purged_snaps_scrub': '2024-09-10T19:28:04.528297+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138320759226367, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-10T19:27:28.035138+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138320759226367, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-10T19:28:15.901036+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138320759226367, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-10T19:27:24.953283+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138320759226367, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-10T19:27:13.126902+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138320759226367, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-10T19:27:38.429368+0000', 'dead_epoch': 0}], 'pg_upmap': [{'pgid': '2.13', 'osds': [2, 0, 5]}], 'pg_upmap_items': [{'pgid': '2.8', 'mappings': [{'from': 0, 'to': 5}, {'from': 1, 'to': 3}, {'from': 7, 'to': 4}]}, {'pgid': '2.18', 'mappings': [{'from': 7, 'to': 0}, {'from': 5, 'to': 4}]}, {'pgid': '4.b', 'mappings': [{'from': 1, 'to': 5}]}, {'pgid': '4.e', 'mappings': [{'from': 1, 'to': 5}]}, {'pgid': '5.15', 'mappings': [{'from': 1, 'to': 3}]}, {'pgid': '6.1b', 'mappings': [{'from': 3, 'to': 0}]}], 'pg_temp': [], 'primary_temp': [], 'blocklist': {'172.21.15.183:6801/3311783362': '2024-09-11T19:10:43.576943+0000', '172.21.15.183:0/4088553922': '2024-09-11T19:10:43.576943+0000', '172.21.15.183:0/3528769414': '2024-09-11T19:05:48.153879+0000', '172.21.15.183:0/1766036658': '2024-09-11T19:05:48.153879+0000', '172.21.15.183:0/3044170088': '2024-09-11T19:05:25.702354+0000', '172.21.15.183:0/3037333778': '2024-09-11T19:05:25.702354+0000', '172.21.15.183:6801/3747320570': '2024-09-11T19:05:25.702354+0000', '172.21.15.183:0/3982386906': '2024-09-11T19:10:43.576943+0000', '172.21.15.183:0/2436014455': '2024-09-11T19:05:25.702354+0000', '172.21.15.183:6801/659745765': '2024-09-11T19:05:48.153879+0000', '172.21.15.183:6800/3747320570': '2024-09-11T19:05:25.702354+0000', '172.21.15.183:0/1652576282': '2024-09-11T19:10:43.576943+0000', '172.21.15.183:6800/3311783362': '2024-09-11T19:10:43.576943+0000', '172.21.15.183:0/1012899549': '2024-09-11T19:05:48.153879+0000', '172.21.15.183:6800/659745765': '2024-09-11T19:05:48.153879+0000', '172.21.15.183:0/412044400': '2024-09-11T19:10:43.576943+0000', '172.21.15.183:0/3636171401': '2024-09-11T19:10:43.576943+0000'}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-09-10T19:28:28.268 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph pg dump --format=json 2024-09-10T19:28:28.338 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:28:28 smithi183 ceph-mon[29736]: 5.2 scrub starts 2024-09-10T19:28:28.338 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:28:28 smithi183 ceph-mon[29736]: 5.2 scrub ok 2024-09-10T19:28:28.338 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:28:28 smithi183 ceph-mon[41562]: 5.2 scrub starts 2024-09-10T19:28:28.338 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:28:28 smithi183 ceph-mon[41562]: 5.2 scrub ok 2024-09-10T19:28:28.347 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:28:28.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:28:28.393 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:28:28.393 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:28:28.494 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:28:28.527 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:28:28.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:28:28.528 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:28:28.629 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:28:28.662 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:28:28.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:28:28.663 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:28:28.764 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:28:28.807 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:28:28.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:28:28.808 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:28:28.909 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:28:28.941 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:28:28.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:28:28.942 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:28:29.043 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:28:29.062 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: 35 16 70343 70327 125.564 80.9375 0.0148393 0.0079558 2024-09-10T19:28:29.067 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:28:29.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:28:29.068 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:28:29.169 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:28:29.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:28:29.212 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:28:29.212 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:28:29.313 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:28:29.347 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:28:29.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:28:29.348 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:28:29.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:28:29 smithi183 ceph-mon[29736]: pgmap v846: 158 pgs: 158 active+clean; 4.0 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 1023 B/s rd, 97 MiB/s wr, 1.56k op/s 2024-09-10T19:28:29.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:28:29 smithi183 ceph-mon[29736]: 4.3 scrub starts 2024-09-10T19:28:29.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:28:29 smithi183 ceph-mon[29736]: 4.3 scrub ok 2024-09-10T19:28:29.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:28:29 smithi183 ceph-mon[29736]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:28:29.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:28:29 smithi183 ceph-mon[29736]: 5.1a scrub starts 2024-09-10T19:28:29.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:28:29 smithi183 ceph-mon[29736]: 5.1a scrub ok 2024-09-10T19:28:29.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:28:29 smithi183 ceph-mon[41562]: pgmap v846: 158 pgs: 158 active+clean; 4.0 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 1023 B/s rd, 97 MiB/s wr, 1.56k op/s 2024-09-10T19:28:29.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:28:29 smithi183 ceph-mon[41562]: 4.3 scrub starts 2024-09-10T19:28:29.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:28:29 smithi183 ceph-mon[41562]: 4.3 scrub ok 2024-09-10T19:28:29.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:28:29 smithi183 ceph-mon[41562]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:28:29.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:28:29 smithi183 ceph-mon[41562]: 5.1a scrub starts 2024-09-10T19:28:29.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:28:29 smithi183 ceph-mon[41562]: 5.1a scrub ok 2024-09-10T19:28:29.449 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:28:29.465 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:28:29 smithi191 ceph-mon[32884]: pgmap v846: 158 pgs: 158 active+clean; 4.0 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 1023 B/s rd, 97 MiB/s wr, 1.56k op/s 2024-09-10T19:28:29.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:28:29 smithi191 ceph-mon[32884]: 4.3 scrub starts 2024-09-10T19:28:29.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:28:29 smithi191 ceph-mon[32884]: 4.3 scrub ok 2024-09-10T19:28:29.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:28:29 smithi191 ceph-mon[32884]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:28:29.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:28:29 smithi191 ceph-mon[32884]: 5.1a scrub starts 2024-09-10T19:28:29.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:28:29 smithi191 ceph-mon[32884]: 5.1a scrub ok 2024-09-10T19:28:29.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:28:29.486 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:28:29.486 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:28:29.587 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:28:29.619 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:28:29.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:28:29.620 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:28:29.721 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:28:29.752 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:28:29.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:28:29.753 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:28:29.854 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:28:29.890 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:28:29.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:28:29.891 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:28:29.992 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:28:30.029 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:28:30.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:28:30.030 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:28:30.063 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: 36 16 71745 71729 124.51 87.625 0.00554212 0.0080257 2024-09-10T19:28:30.066 INFO:teuthology.orchestra.run.smithi183.stderr:noscrub is set 2024-09-10T19:28:30.130 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:28:30.170 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:28:30.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:28:30.171 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:28:30.216 INFO:journalctl@ceph.iscsi.iscsi.a.smithi191.stdout:Sep 10 19:28:29 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-iscsi-iscsi-a[61816]: debug there is no tcmu-runner data available 2024-09-10T19:28:30.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:28:30 smithi191 ceph-mon[32884]: 5.0 scrub starts 2024-09-10T19:28:30.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:28:30 smithi191 ceph-mon[32884]: 5.0 scrub ok 2024-09-10T19:28:30.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:28:30 smithi191 ceph-mon[32884]: 2.16 scrub starts 2024-09-10T19:28:30.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:28:30 smithi191 ceph-mon[32884]: 2.16 scrub ok 2024-09-10T19:28:30.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:28:30 smithi191 ceph-mon[32884]: from='client.? 172.21.15.183:0/3519473808' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-10T19:28:30.217 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:28:30 smithi191 ceph-mon[32884]: 5.12 scrub starts 2024-09-10T19:28:30.217 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:28:30 smithi191 ceph-mon[32884]: 5.12 scrub ok 2024-09-10T19:28:30.271 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:28:30.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:28:30.309 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:28:30.309 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:28:30.410 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:28:30.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:28:30 smithi183 ceph-mon[29736]: 5.0 scrub starts 2024-09-10T19:28:30.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:28:30 smithi183 ceph-mon[29736]: 5.0 scrub ok 2024-09-10T19:28:30.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:28:30 smithi183 ceph-mon[29736]: 2.16 scrub starts 2024-09-10T19:28:30.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:28:30 smithi183 ceph-mon[29736]: 2.16 scrub ok 2024-09-10T19:28:30.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:28:30 smithi183 ceph-mon[29736]: from='client.? 172.21.15.183:0/3519473808' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-10T19:28:30.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:28:30 smithi183 ceph-mon[29736]: 5.12 scrub starts 2024-09-10T19:28:30.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:28:30 smithi183 ceph-mon[29736]: 5.12 scrub ok 2024-09-10T19:28:30.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:28:30 smithi183 ceph-mon[41562]: 5.0 scrub starts 2024-09-10T19:28:30.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:28:30 smithi183 ceph-mon[41562]: 5.0 scrub ok 2024-09-10T19:28:30.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:28:30 smithi183 ceph-mon[41562]: 2.16 scrub starts 2024-09-10T19:28:30.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:28:30 smithi183 ceph-mon[41562]: 2.16 scrub ok 2024-09-10T19:28:30.433 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:28:30 smithi183 ceph-mon[41562]: from='client.? 172.21.15.183:0/3519473808' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-10T19:28:30.433 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:28:30 smithi183 ceph-mon[41562]: 5.12 scrub starts 2024-09-10T19:28:30.433 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:28:30 smithi183 ceph-mon[41562]: 5.12 scrub ok 2024-09-10T19:28:30.441 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:28:30.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:28:30.442 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:28:30.543 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:28:30.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:28:30.595 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:28:30.595 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:28:30.696 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:28:30.731 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:28:30.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:28:30.732 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:28:30.832 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:28:30.868 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:28:30.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:28:30.869 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:28:30.970 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:28:31.003 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:28:31.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:28:31.005 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:28:31.063 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: 37 16 73263 73247 123.709 94.875 0.00840269 0.00807944 2024-09-10T19:28:31.084 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:28:31.106 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:28:31.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:28:31.152 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:28:31.152 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:28:31.257 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:28:31.293 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:28:31.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:28:31.294 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:28:31.317 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:28:31 smithi183 ceph-mon[29736]: 6.7 deep-scrub starts 2024-09-10T19:28:31.317 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:28:31 smithi183 ceph-mon[29736]: 6.7 deep-scrub ok 2024-09-10T19:28:31.317 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:28:31 smithi183 ceph-mon[29736]: pgmap v847: 158 pgs: 158 active+clean; 4.1 GiB data, 17 GiB used, 608 GiB / 626 GiB avail; 96 MiB/s wr, 1.53k op/s 2024-09-10T19:28:31.317 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:28:31 smithi183 ceph-mon[29736]: from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:28:31.317 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:28:31 smithi183 ceph-mon[29736]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T19:28:31.317 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:28:31 smithi183 ceph-mon[29736]: from='client.? 172.21.15.183:0/3519473808' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-10T19:28:31.318 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:28:31 smithi183 ceph-mon[29736]: osdmap e385: 8 total, 8 up, 7 in 2024-09-10T19:28:31.318 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:28:31 smithi183 ceph-mon[41562]: 6.7 deep-scrub starts 2024-09-10T19:28:31.318 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:28:31 smithi183 ceph-mon[41562]: 6.7 deep-scrub ok 2024-09-10T19:28:31.318 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:28:31 smithi183 ceph-mon[41562]: pgmap v847: 158 pgs: 158 active+clean; 4.1 GiB data, 17 GiB used, 608 GiB / 626 GiB avail; 96 MiB/s wr, 1.53k op/s 2024-09-10T19:28:31.318 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:28:31 smithi183 ceph-mon[41562]: from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:28:31.318 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:28:31 smithi183 ceph-mon[41562]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T19:28:31.318 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:28:31 smithi183 ceph-mon[41562]: from='client.? 172.21.15.183:0/3519473808' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-10T19:28:31.318 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:28:31 smithi183 ceph-mon[41562]: osdmap e385: 8 total, 8 up, 7 in 2024-09-10T19:28:31.395 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:28:31.428 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:28:31.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:28:31.430 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:28:31.465 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:28:31 smithi191 ceph-mon[32884]: 6.7 deep-scrub starts 2024-09-10T19:28:31.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:28:31 smithi191 ceph-mon[32884]: 6.7 deep-scrub ok 2024-09-10T19:28:31.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:28:31 smithi191 ceph-mon[32884]: pgmap v847: 158 pgs: 158 active+clean; 4.1 GiB data, 17 GiB used, 608 GiB / 626 GiB avail; 96 MiB/s wr, 1.53k op/s 2024-09-10T19:28:31.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:28:31 smithi191 ceph-mon[32884]: from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:28:31.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:28:31 smithi191 ceph-mon[32884]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T19:28:31.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:28:31 smithi191 ceph-mon[32884]: from='client.? 172.21.15.183:0/3519473808' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-10T19:28:31.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:28:31 smithi191 ceph-mon[32884]: osdmap e385: 8 total, 8 up, 7 in 2024-09-10T19:28:31.530 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:28:31.567 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:28:31.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:28:31.569 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:28:31.596 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Sep 10 19:28:31 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-y[29989]: ::ffff:172.21.15.191 - - [10/Sep/2024:19:28:31] "GET /metrics HTTP/1.1" 200 224855 "" "Prometheus/2.43.0" 2024-09-10T19:28:31.670 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:28:31.704 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:28:31.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:28:31.705 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:28:31.806 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:28:31.840 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:28:31.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:28:31.841 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:28:31.942 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:28:31.976 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:28:31.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:28:31.978 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:28:32.062 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: 38 16 74752 74736 122.903 93.0625 0.0122812 0.00813197 2024-09-10T19:28:32.078 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:28:32.116 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:28:32.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:28:32.117 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:28:32.218 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:28:32.254 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:28:32.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:28:32.255 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:28:32.356 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:28:32.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:28:32.394 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:28:32.394 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:28:32.495 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:28:32.529 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:28:32.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:28:32.530 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:28:32.631 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:28:32.659 INFO:teuthology.orchestra.run.smithi183.stdout:osd.0: {} 2024-09-10T19:28:32.659 INFO:teuthology.orchestra.run.smithi183.stdout:osd.1: {} 2024-09-10T19:28:32.659 INFO:teuthology.orchestra.run.smithi183.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-10T19:28:32.659 INFO:teuthology.orchestra.run.smithi183.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-10T19:28:32.664 INFO:teuthology.orchestra.run.smithi183.stdout:osd.2: {} 2024-09-10T19:28:32.665 INFO:teuthology.orchestra.run.smithi183.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-10T19:28:32.694 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:28:32.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:28:32.695 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:28:32.703 INFO:teuthology.orchestra.run.smithi183.stdout:osd.3: {} 2024-09-10T19:28:32.703 INFO:teuthology.orchestra.run.smithi183.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-10T19:28:32.741 INFO:teuthology.orchestra.run.smithi183.stdout:osd.4: {} 2024-09-10T19:28:32.741 INFO:teuthology.orchestra.run.smithi183.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-10T19:28:32.764 INFO:teuthology.orchestra.run.smithi183.stdout:ERROR: (22) Invalid argument 2024-09-10T19:28:32.764 INFO:teuthology.orchestra.run.smithi183.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-10T19:28:32.771 INFO:teuthology.orchestra.run.smithi183.stdout:osd.5: {} 2024-09-10T19:28:32.772 INFO:teuthology.orchestra.run.smithi183.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-10T19:28:32.796 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:28:32.806 INFO:teuthology.orchestra.run.smithi183.stdout:osd.6: {} 2024-09-10T19:28:32.806 INFO:teuthology.orchestra.run.smithi183.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-10T19:28:32.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:28:32.828 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:28:32.829 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:28:32.840 INFO:teuthology.orchestra.run.smithi183.stdout:osd.7: {} 2024-09-10T19:28:32.840 INFO:teuthology.orchestra.run.smithi183.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-10T19:28:32.929 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:28:32.972 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:28:32.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:28:32.973 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:28:33.059 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:28:33 smithi183 ceph-mon[29736]: pgmap v849: 158 pgs: 158 active+clean; 4.4 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 89 MiB/s wr, 1.43k op/s 2024-09-10T19:28:33.059 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:28:33 smithi183 ceph-mon[41562]: pgmap v849: 158 pgs: 158 active+clean; 4.4 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 89 MiB/s wr, 1.43k op/s 2024-09-10T19:28:33.059 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:28:33 smithi183 ceph-mon[41562]: from='client.? 172.21.15.183:0/1121106006' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T19:28:33.062 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: 39 16 76176 76160 122.033 89 0.00937587 0.00818994 2024-09-10T19:28:33.074 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:28:33.106 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:28:33.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:28:33.107 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:28:33.125 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-10T19:28:33.208 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:28:33.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:28:33.257 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:28:33.257 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:28:33.358 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:28:33.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:28:33.406 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:28:33.406 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:28:33.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:28:33 smithi183 ceph-mon[29736]: from='client.? 172.21.15.183:0/1121106006' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T19:28:33.465 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:28:33 smithi191 ceph-mon[32884]: pgmap v849: 158 pgs: 158 active+clean; 4.4 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 89 MiB/s wr, 1.43k op/s 2024-09-10T19:28:33.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:28:33 smithi191 ceph-mon[32884]: from='client.? 172.21.15.183:0/1121106006' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T19:28:33.507 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:28:33.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:28:33.546 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:28:33.546 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:28:33.647 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:28:33.676 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:28:33.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:28:33.677 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:28:33.777 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:28:33.799 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph daemon osd.0 dump_ops_in_flight 2024-09-10T19:28:33.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:28:33.815 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:28:33.816 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:28:33.917 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:28:33.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:28:33.950 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:28:33.950 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:28:34.051 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:28:34.063 INFO:tasks.radosbench.radosbench.0.smithi183.stdout:2024-09-10T19:28:34.062299+0000 min lat: 0.00134808 max lat: 0.165025 avg lat: 0.00824499 2024-09-10T19:28:34.064 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-10T19:28:34.064 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: 40 16 77596 77580 121.201 88.75 0.0109409 0.00824499 2024-09-10T19:28:34.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:28:34.090 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:28:34.091 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:28:34.191 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:28:34.228 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:28:34.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:28:34.229 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:28:34.330 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:28:34.368 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:28:34.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:28:34.372 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:28:34.473 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:28:34.521 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:28:34.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:28:34.522 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:28:34.623 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:28:34.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:28:34.667 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:28:34.667 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:28:34.768 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:28:34.802 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:28:34.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:28:34.803 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:28:34.904 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:28:34.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:28:34.938 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:28:34.938 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:28:35.039 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:28:35.063 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: 41 16 78960 78944 120.324 85.25 0.0117208 0.00830614 2024-09-10T19:28:35.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:28:35.073 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:28:35.073 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:28:35.174 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:28:35.211 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:28:35.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:28:35.212 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:28:35.312 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:28:35.352 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:28:35.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:28:35.353 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:28:35.424 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:28:35 smithi183 ceph-mon[29736]: pgmap v850: 158 pgs: 158 active+clean; 4.4 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 88 MiB/s wr, 1.41k op/s 2024-09-10T19:28:35.424 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:28:35 smithi183 ceph-mon[29736]: osdmap e386: 8 total, 8 up, 7 in 2024-09-10T19:28:35.424 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:28:35 smithi183 ceph-mon[41562]: pgmap v850: 158 pgs: 158 active+clean; 4.4 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 88 MiB/s wr, 1.41k op/s 2024-09-10T19:28:35.424 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:28:35 smithi183 ceph-mon[41562]: osdmap e386: 8 total, 8 up, 7 in 2024-09-10T19:28:35.454 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:28:35.467 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:28:35 smithi191 ceph-mon[32884]: pgmap v850: 158 pgs: 158 active+clean; 4.4 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 88 MiB/s wr, 1.41k op/s 2024-09-10T19:28:35.467 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:28:35 smithi191 ceph-mon[32884]: osdmap e386: 8 total, 8 up, 7 in 2024-09-10T19:28:35.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:28:35.489 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:28:35.490 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:28:35.590 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:28:35.625 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:28:35.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:28:35.626 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:28:35.727 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:28:35.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:28:35.769 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:28:35.770 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:28:35.870 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:28:35.910 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:28:35.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:28:35.912 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:28:36.012 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:28:36.041 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:28:36.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:28:36.042 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:28:36.063 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: 42 16 80356 80340 119.536 87.25 0.00379407 0.00835808 2024-09-10T19:28:36.143 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:28:36.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:28:36.169 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:28:36.169 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:28:36.270 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:28:36.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:28:36.307 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:28:36.307 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:28:36.408 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:28:36.441 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:28:36.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:28:36.442 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:28:36.543 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:28:36.577 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:28:36.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:28:36.578 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:28:36.679 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:28:36.708 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:28:36.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:28:36.709 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:28:36.810 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:28:36.853 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:28:36.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:28:36.854 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:28:36.954 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:28:36.971 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:28:37.013 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:28:37.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:28:37.017 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:28:37.056 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:28:37.063 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: 43 16 81680 81664 118.68 82.75 0.0147413 0.00842115 2024-09-10T19:28:37.118 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:28:37.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:28:37.155 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:28:37.156 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:28:37.256 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:28:37.291 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:28:37.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:28:37.292 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:28:37.309 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:28:37 smithi183 ceph-mon[29736]: pgmap v852: 158 pgs: 158 active+clean; 4.8 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 98 MiB/s wr, 1.57k op/s 2024-09-10T19:28:37.311 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:28:37 smithi183 ceph-mon[41562]: pgmap v852: 158 pgs: 158 active+clean; 4.8 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 98 MiB/s wr, 1.57k op/s 2024-09-10T19:28:37.393 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:28:37.428 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:28:37.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:28:37.429 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:28:37.450 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:28:37 smithi191 ceph-mon[32884]: pgmap v852: 158 pgs: 158 active+clean; 4.8 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 98 MiB/s wr, 1.57k op/s 2024-09-10T19:28:37.530 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:28:37.566 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:28:37.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:28:37.567 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:28:37.668 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:28:37.707 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:28:37.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:28:37.708 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:28:37.715 INFO:journalctl@ceph.mgr.x.smithi191.stdout:Sep 10 19:28:37 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-x[34967]: ::ffff:172.21.15.191 - - [10/Sep/2024:19:28:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T19:28:37.809 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:28:37.852 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:28:37.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:28:37.853 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:28:37.954 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:28:37.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:28:37.989 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:28:37.989 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:28:38.063 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: 44 16 83105 83089 118.007 89.0625 0.0187837 0.00846967 2024-09-10T19:28:38.090 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:28:38.124 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:28:38.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:28:38.125 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:28:38.158 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-09-10T19:28:38.161 INFO:teuthology.orchestra.run.smithi183.stderr:dumped all 2024-09-10T19:28:38.227 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:28:38.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:28:38.263 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:28:38.263 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:28:38.364 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:28:38.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:28:38.402 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:28:38.402 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:28:38.503 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:28:38.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:28:38.536 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:28:38.536 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:28:38.637 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:28:38.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:28:38.670 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:28:38.670 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:28:38.771 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:28:38.803 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:28:38.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:28:38.804 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:28:38.905 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:28:38.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:28:38.940 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:28:38.940 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:28:38.992 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-10T19:28:39.041 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:28:39.067 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: 45 16 84572 84556 117.414 91.6875 0.0084213 0.00851214 2024-09-10T19:28:39.072 INFO:teuthology.orchestra.run.smithi183.stdout:{"pg_ready":true,"pg_map":{"version":853,"stamp":"2024-09-10T19:28:37.937715+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":5223487708,"num_objects":79891,"num_object_clones":0,"num_object_copies":239673,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":79891,"num_whiteouts":0,"num_read":1414,"num_read_kb":1166,"num_write":80411,"num_write_kb":5100523,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":458,"num_bytes_recovered":536482,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":71757,"ondisk_log_size":71757,"up":474,"acting":474,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":482,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":656216064,"kb_used":20419504,"kb_used_data":15273268,"kb_used_omap":19,"kb_used_meta":5146156,"kb_avail":635796560,"statfs":{"total":671965249536,"available":651055677440,"internally_reserved":0,"allocated":15639826432,"data_stored":15636983141,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19457,"internal_metadata":5269664767},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,3,2],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":852951040,"num_objects":13015,"num_object_clones":0,"num_object_copies":39045,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13015,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":13015,"num_write_kb":832960,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.003032"},"pg_stats":[{"pgid":"6.1b","version":"0'0","reported_seq":121,"reported_epoch":385,"state":"active+clean","last_fresh":"2024-09-10T19:28:30.062411+0000","last_change":"2024-09-10T19:28:18.455274+0000","last_active":"2024-09-10T19:28:30.062411+0000","last_peered":"2024-09-10T19:28:30.062411+0000","last_clean":"2024-09-10T19:28:30.062411+0000","last_became_active":"2024-09-10T19:27:52.994757+0000","last_became_peered":"2024-09-10T19:27:52.994757+0000","last_unstale":"2024-09-10T19:28:30.062411+0000","last_undegraded":"2024-09-10T19:28:30.062411+0000","last_fullsized":"2024-09-10T19:28:30.062411+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:28:18.455084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:28:18.455084+0000","last_clean_scrub_stamp":"2024-09-10T19:28:18.455084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:29:18.964456+0000","scrub_duration":0.0084988790000000008,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6],"acting":[0,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":369,"reported_epoch":385,"state":"active+clean","last_fresh":"2024-09-10T19:28:30.062707+0000","last_change":"2024-09-10T19:28:05.475068+0000","last_active":"2024-09-10T19:28:30.062707+0000","last_peered":"2024-09-10T19:28:30.062707+0000","last_clean":"2024-09-10T19:28:30.062707+0000","last_became_active":"2024-09-10T19:24:23.892621+0000","last_became_peered":"2024-09-10T19:24:23.892621+0000","last_unstale":"2024-09-10T19:28:30.062707+0000","last_undegraded":"2024-09-10T19:28:30.062707+0000","last_fullsized":"2024-09-10T19:28:30.062707+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:28:05.474843+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:16:38.194472+0000","last_clean_scrub_stamp":"2024-09-10T19:28:05.474843+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:29:11.103208+0000","scrub_duration":0.0055108800000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.19","version":"237'27","reported_seq":470,"reported_epoch":385,"state":"active+clean","last_fresh":"2024-09-10T19:28:30.067135+0000","last_change":"2024-09-10T19:27:56.015747+0000","last_active":"2024-09-10T19:28:30.067135+0000","last_peered":"2024-09-10T19:28:30.067135+0000","last_clean":"2024-09-10T19:28:30.067135+0000","last_became_active":"2024-09-10T19:24:23.906592+0000","last_became_peered":"2024-09-10T19:24:23.906592+0000","last_unstale":"2024-09-10T19:28:30.067135+0000","last_undegraded":"2024-09-10T19:28:30.067135+0000","last_fullsized":"2024-09-10T19:28:30.067135+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"237'27","last_scrub_stamp":"2024-09-10T19:27:56.015661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:27:56.015661+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:29:02.292655+0000","scrub_duration":0.0091554960000000008,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":454,"reported_epoch":385,"state":"active+clean","last_fresh":"2024-09-10T19:28:30.066664+0000","last_change":"2024-09-10T19:27:53.006502+0000","last_active":"2024-09-10T19:28:30.066664+0000","last_peered":"2024-09-10T19:28:30.066664+0000","last_clean":"2024-09-10T19:28:30.066664+0000","last_became_active":"2024-09-10T19:27:53.006175+0000","last_became_peered":"2024-09-10T19:27:53.006175+0000","last_unstale":"2024-09-10T19:28:30.066664+0000","last_undegraded":"2024-09-10T19:28:30.066664+0000","last_fullsized":"2024-09-10T19:28:30.066664+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:26:07.791886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:19:28.498560+0000","last_clean_scrub_stamp":"2024-09-10T19:26:07.791886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0046830530000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"384'2","reported_seq":432,"reported_epoch":385,"state":"active+clean","last_fresh":"2024-09-10T19:28:30.070046+0000","last_change":"2024-09-10T19:28:23.240900+0000","last_active":"2024-09-10T19:28:30.070046+0000","last_peered":"2024-09-10T19:28:30.070046+0000","last_clean":"2024-09-10T19:28:30.070046+0000","last_became_active":"2024-09-10T19:24:23.908614+0000","last_became_peered":"2024-09-10T19:24:23.908614+0000","last_unstale":"2024-09-10T19:28:30.070046+0000","last_undegraded":"2024-09-10T19:28:30.070046+0000","last_fullsized":"2024-09-10T19:28:30.070046+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"384'2","last_scrub_stamp":"2024-09-10T19:28:23.240824+0000","last_deep_scrub":"384'2","last_deep_scrub_stamp":"2024-09-10T19:28:23.240824+0000","last_clean_scrub_stamp":"2024-09-10T19:28:23.240824+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:29:47.336087+0000","scrub_duration":0.0060048749999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":874,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"384'19","reported_seq":522,"reported_epoch":385,"state":"active+clean","last_fresh":"2024-09-10T19:28:30.067310+0000","last_change":"2024-09-10T19:28:21.800015+0000","last_active":"2024-09-10T19:28:30.067310+0000","last_peered":"2024-09-10T19:28:30.067310+0000","last_clean":"2024-09-10T19:28:30.067310+0000","last_became_active":"2024-09-10T19:27:53.013509+0000","last_became_peered":"2024-09-10T19:27:53.013509+0000","last_unstale":"2024-09-10T19:28:30.067310+0000","last_undegraded":"2024-09-10T19:28:30.067310+0000","last_fullsized":"2024-09-10T19:28:30.067310+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"384'19","last_scrub_stamp":"2024-09-10T19:28:21.799921+0000","last_deep_scrub":"384'19","last_deep_scrub_stamp":"2024-09-10T19:28:21.799921+0000","last_clean_scrub_stamp":"2024-09-10T19:28:21.799921+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:29:44.757714+0000","scrub_duration":0.013765777,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":47,"reported_epoch":385,"state":"active+clean","last_fresh":"2024-09-10T19:28:30.068518+0000","last_change":"2024-09-10T19:28:04.548915+0000","last_active":"2024-09-10T19:28:30.068518+0000","last_peered":"2024-09-10T19:28:30.068518+0000","last_clean":"2024-09-10T19:28:30.068518+0000","last_became_active":"2024-09-10T19:27:14.338549+0000","last_became_peered":"2024-09-10T19:27:14.338549+0000","last_unstale":"2024-09-10T19:28:30.068518+0000","last_undegraded":"2024-09-10T19:28:30.068518+0000","last_fullsized":"2024-09-10T19:28:30.068518+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:28:04.548825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:28:04.548825+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:29:25.437974+0000","scrub_duration":0.0042672989999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":432,"reported_epoch":385,"state":"active+clean","last_fresh":"2024-09-10T19:28:30.066931+0000","last_change":"2024-09-10T19:28:22.780930+0000","last_active":"2024-09-10T19:28:30.066931+0000","last_peered":"2024-09-10T19:28:30.066931+0000","last_clean":"2024-09-10T19:28:30.066931+0000","last_became_active":"2024-09-10T19:24:01.971921+0000","last_became_peered":"2024-09-10T19:24:01.971921+0000","last_unstale":"2024-09-10T19:28:30.066931+0000","last_undegraded":"2024-09-10T19:28:30.066931+0000","last_fullsized":"2024-09-10T19:28:30.066931+0000","mapping_epoch":292,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":293,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:28:22.780871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:28:22.780871+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:29:24.186620+0000","scrub_duration":0.0031209279999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":433,"reported_epoch":385,"state":"active+clean","last_fresh":"2024-09-10T19:28:30.075822+0000","last_change":"2024-09-10T19:27:53.012211+0000","last_active":"2024-09-10T19:28:30.075822+0000","last_peered":"2024-09-10T19:28:30.075822+0000","last_clean":"2024-09-10T19:28:30.075822+0000","last_became_active":"2024-09-10T19:27:53.011202+0000","last_became_peered":"2024-09-10T19:27:53.011202+0000","last_unstale":"2024-09-10T19:28:30.075822+0000","last_undegraded":"2024-09-10T19:28:30.075822+0000","last_fullsized":"2024-09-10T19:28:30.075822+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:26:35.582561+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:26:35.582561+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.013149586,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1b","version":"237'7","reported_seq":464,"reported_epoch":385,"state":"active+clean","last_fresh":"2024-09-10T19:28:30.066925+0000","last_change":"2024-09-10T19:26:22.847139+0000","last_active":"2024-09-10T19:28:30.066925+0000","last_peered":"2024-09-10T19:28:30.066925+0000","last_clean":"2024-09-10T19:28:30.066925+0000","last_became_active":"2024-09-10T19:24:01.974813+0000","last_became_peered":"2024-09-10T19:24:01.974813+0000","last_unstale":"2024-09-10T19:28:30.066925+0000","last_undegraded":"2024-09-10T19:28:30.066925+0000","last_fullsized":"2024-09-10T19:28:30.066925+0000","mapping_epoch":292,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":293,"parent":"0.0","parent_split_bits":0,"last_scrub":"237'7","last_scrub_stamp":"2024-09-10T19:26:22.847061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:26:22.847061+0000","objects_scrubbed":2,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001667078,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":10,"num_write":8,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":426,"reported_epoch":385,"state":"active+clean","last_fresh":"2024-09-10T19:28:30.074462+0000","last_change":"2024-09-10T19:28:28.788461+0000","last_active":"2024-09-10T19:28:30.074462+0000","last_peered":"2024-09-10T19:28:30.074462+0000","last_clean":"2024-09-10T19:28:30.074462+0000","last_became_active":"2024-09-10T19:24:01.976587+0000","last_became_peered":"2024-09-10T19:24:01.976587+0000","last_unstale":"2024-09-10T19:28:30.074462+0000","last_undegraded":"2024-09-10T19:28:30.074462+0000","last_fullsized":"2024-09-10T19:28:30.074462+0000","mapping_epoch":292,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":293,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:28:28.788403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:28:28.788403+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:29:29.453764+0000","scrub_duration":0.0033852510000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":427,"reported_epoch":385,"state":"active+clean","last_fresh":"2024-09-10T19:28:30.078964+0000","last_change":"2024-09-10T19:27:58.990300+0000","last_active":"2024-09-10T19:28:30.078964+0000","last_peered":"2024-09-10T19:28:30.078964+0000","last_clean":"2024-09-10T19:28:30.078964+0000","last_became_active":"2024-09-10T19:24:01.970118+0000","last_became_peered":"2024-09-10T19:24:01.970118+0000","last_unstale":"2024-09-10T19:28:30.078964+0000","last_undegraded":"2024-09-10T19:28:30.078964+0000","last_fullsized":"2024-09-10T19:28:30.078964+0000","mapping_epoch":292,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":293,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:27:58.990222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:27:58.990222+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:29:28.464406+0000","scrub_duration":0.0025876580000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":436,"reported_epoch":385,"state":"active+clean","last_fresh":"2024-09-10T19:28:30.078304+0000","last_change":"2024-09-10T19:27:53.008856+0000","last_active":"2024-09-10T19:28:30.078304+0000","last_peered":"2024-09-10T19:28:30.078304+0000","last_clean":"2024-09-10T19:28:30.078304+0000","last_became_active":"2024-09-10T19:27:53.008555+0000","last_became_peered":"2024-09-10T19:27:53.008555+0000","last_unstale":"2024-09-10T19:28:30.078304+0000","last_undegraded":"2024-09-10T19:28:30.078304+0000","last_fullsized":"2024-09-10T19:28:30.078304+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:27:22.675093+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:27:22.675093+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:28:49.889807+0000","scrub_duration":0.0014582810000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1a","version":"237'15","reported_seq":438,"reported_epoch":385,"state":"active+clean","last_fresh":"2024-09-10T19:28:30.067207+0000","last_change":"2024-09-10T19:28:05.830531+0000","last_active":"2024-09-10T19:28:30.067207+0000","last_peered":"2024-09-10T19:28:30.067207+0000","last_clean":"2024-09-10T19:28:30.067207+0000","last_became_active":"2024-09-10T19:24:23.891213+0000","last_became_peered":"2024-09-10T19:24:23.891213+0000","last_unstale":"2024-09-10T19:28:30.067207+0000","last_undegraded":"2024-09-10T19:28:30.067207+0000","last_fullsized":"2024-09-10T19:28:30.067207+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"237'15","last_scrub_stamp":"2024-09-10T19:28:05.830456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:28:05.830456+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:29:24.221040+0000","scrub_duration":0.0046734919999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":409,"reported_epoch":385,"state":"active+clean","last_fresh":"2024-09-10T19:28:30.079152+0000","last_change":"2024-09-10T19:27:31.878487+0000","last_active":"2024-09-10T19:28:30.079152+0000","last_peered":"2024-09-10T19:28:30.079152+0000","last_clean":"2024-09-10T19:28:30.079152+0000","last_became_active":"2024-09-10T19:24:23.907653+0000","last_became_peered":"2024-09-10T19:24:23.907653+0000","last_unstale":"2024-09-10T19:28:30.079152+0000","last_undegraded":"2024-09-10T19:28:30.079152+0000","last_fullsized":"2024-09-10T19:28:30.079152+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:27:31.878435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:27:31.878435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:28:38.371996+0000","scrub_duration":0.00132883,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":382,"reported_epoch":385,"state":"active+clean","last_fresh":"2024-09-10T19:28:30.062502+0000","last_change":"2024-09-10T19:27:28.398388+0000","last_active":"2024-09-10T19:28:30.062502+0000","last_peered":"2024-09-10T19:28:30.062502+0000","last_clean":"2024-09-10T19:28:30.062502+0000","last_became_active":"2024-09-10T19:24:24.907431+0000","last_became_peered":"2024-09-10T19:24:24.907431+0000","last_unstale":"2024-09-10T19:28:30.062502+0000","last_undegraded":"2024-09-10T19:28:30.062502+0000","last_fullsized":"2024-09-10T19:28:30.062502+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:27:28.398334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:27:28.398334+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0012534989999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.2","version":"386'4995","reported_seq":5018,"reported_epoch":386,"state":"active+clean","last_fresh":"2024-09-10T19:28:35.206375+0000","last_change":"2024-09-10T19:27:46.703748+0000","last_active":"2024-09-10T19:28:35.206375+0000","last_peered":"2024-09-10T19:28:35.206375+0000","last_clean":"2024-09-10T19:28:35.206375+0000","last_became_active":"2024-09-10T19:27:46.703515+0000","last_became_peered":"2024-09-10T19:27:46.703515+0000","last_unstale":"2024-09-10T19:28:35.206375+0000","last_undegraded":"2024-09-10T19:28:35.206375+0000","last_fullsized":"2024-09-10T19:28:35.206375+0000","mapping_epoch":371,"log_start":"376'800","ondisk_log_start":"376'800","created":371,"last_epoch_clean":372,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:27:45.679464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:27:45.679464+0000","last_clean_scrub_stamp":"2024-09-10T19:27:45.679464+0000","objects_scrubbed":0,"log_size":4195,"log_dups_size":0,"ondisk_log_size":4195,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:29:14.665007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327352320,"num_objects":4995,"num_object_clones":0,"num_object_copies":14985,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4995,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4995,"num_write_kb":319680,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,7],"acting":[3,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"74'79","reported_seq":214,"reported_epoch":385,"state":"active+clean","last_fresh":"2024-09-10T19:28:30.078764+0000","last_change":"2024-09-10T19:27:53.019123+0000","last_active":"2024-09-10T19:28:30.078764+0000","last_peered":"2024-09-10T19:28:30.078764+0000","last_clean":"2024-09-10T19:28:30.078764+0000","last_became_active":"2024-09-10T19:27:53.004335+0000","last_became_peered":"2024-09-10T19:27:53.004335+0000","last_unstale":"2024-09-10T19:28:30.078764+0000","last_undegraded":"2024-09-10T19:28:30.078764+0000","last_fullsized":"2024-09-10T19:28:30.078764+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":4,"last_scrub":"74'79","last_scrub_stamp":"2024-09-10T19:27:28.911137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:08:00.699315+0000","last_clean_scrub_stamp":"2024-09-10T19:27:28.911137+0000","objects_scrubbed":1,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:28:39.777749+0000","scrub_duration":0.0013578080000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":525376,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.d","version":"237'31","reported_seq":596,"reported_epoch":385,"state":"active+clean","last_fresh":"2024-09-10T19:28:30.068199+0000","last_change":"2024-09-10T19:27:53.053251+0000","last_active":"2024-09-10T19:28:30.068199+0000","last_peered":"2024-09-10T19:28:30.068199+0000","last_clean":"2024-09-10T19:28:30.068199+0000","last_became_active":"2024-09-10T19:27:53.012536+0000","last_became_peered":"2024-09-10T19:27:53.012536+0000","last_unstale":"2024-09-10T19:28:30.068199+0000","last_undegraded":"2024-09-10T19:28:30.068199+0000","last_fullsized":"2024-09-10T19:28:30.068199+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"237'31","last_scrub_stamp":"2024-09-10T19:26:27.880373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:26:27.880373+0000","objects_scrubbed":8,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0086552290000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":50,"num_read_kb":40,"num_write":32,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":40,"num_bytes_recovered":550,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,6],"acting":[4,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":416,"reported_epoch":385,"state":"active+clean","last_fresh":"2024-09-10T19:28:30.074810+0000","last_change":"2024-09-10T19:28:16.806242+0000","last_active":"2024-09-10T19:28:30.074810+0000","last_peered":"2024-09-10T19:28:30.074810+0000","last_clean":"2024-09-10T19:28:30.074810+0000","last_became_active":"2024-09-10T19:10:07.691119+0000","last_became_peered":"2024-09-10T19:10:07.691119+0000","last_unstale":"2024-09-10T19:28:30.074810+0000","last_undegraded":"2024-09-10T19:28:30.074810+0000","last_fullsized":"2024-09-10T19:28:30.074810+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:28:16.806115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:16:39.716674+0000","last_clean_scrub_stamp":"2024-09-10T19:28:16.806115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:29:31.184036+0000","scrub_duration":0.0015054790000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":137,"reported_epoch":385,"state":"active+clean","last_fresh":"2024-09-10T19:28:30.072450+0000","last_change":"2024-09-10T19:27:30.389502+0000","last_active":"2024-09-10T19:28:30.072450+0000","last_peered":"2024-09-10T19:28:30.072450+0000","last_clean":"2024-09-10T19:28:30.072450+0000","last_became_active":"2024-09-10T19:24:24.886479+0000","last_became_peered":"2024-09-10T19:24:24.886479+0000","last_unstale":"2024-09-10T19:28:30.072450+0000","last_undegraded":"2024-09-10T19:28:30.072450+0000","last_fullsized":"2024-09-10T19:28:30.072450+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:27:30.389447+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:27:30.389447+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:28:50.847650+0000","scrub_duration":0.0014884690000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":294,"reported_epoch":385,"state":"active+clean","last_fresh":"2024-09-10T19:28:30.068192+0000","last_change":"2024-09-10T19:27:53.026930+0000","last_active":"2024-09-10T19:28:30.068192+0000","last_peered":"2024-09-10T19:28:30.068192+0000","last_clean":"2024-09-10T19:28:30.068192+0000","last_became_active":"2024-09-10T19:27:53.026452+0000","last_became_peered":"2024-09-10T19:27:53.026452+0000","last_unstale":"2024-09-10T19:28:30.068192+0000","last_undegraded":"2024-09-10T19:28:30.068192+0000","last_fullsized":"2024-09-10T19:28:30.068192+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:26:08.894702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:26:08.894702+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.010734235,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.3","version":"385'4900","reported_seq":4920,"reported_epoch":385,"state":"active+clean","last_fresh":"2024-09-10T19:28:34.774568+0000","last_change":"2024-09-10T19:27:53.013568+0000","last_active":"2024-09-10T19:28:34.774568+0000","last_peered":"2024-09-10T19:28:34.774568+0000","last_clean":"2024-09-10T19:28:34.774568+0000","last_became_active":"2024-09-10T19:27:53.010097+0000","last_became_peered":"2024-09-10T19:27:53.010097+0000","last_unstale":"2024-09-10T19:28:34.774568+0000","last_undegraded":"2024-09-10T19:28:34.774568+0000","last_fullsized":"2024-09-10T19:28:34.774568+0000","mapping_epoch":374,"log_start":"375'200","ondisk_log_start":"375'200","created":371,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:27:45.679464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:27:45.679464+0000","last_clean_scrub_stamp":"2024-09-10T19:27:45.679464+0000","objects_scrubbed":0,"log_size":4700,"log_dups_size":0,"ondisk_log_size":4700,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:29:01.854367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":321126400,"num_objects":4900,"num_object_clones":0,"num_object_copies":14700,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4900,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4900,"num_write_kb":313600,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1],"acting":[6,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.9","version":"74'79","reported_seq":471,"reported_epoch":385,"state":"active+clean","last_fresh":"2024-09-10T19:28:30.074295+0000","last_change":"2024-09-10T19:27:13.532461+0000","last_active":"2024-09-10T19:28:30.074295+0000","last_peered":"2024-09-10T19:28:30.074295+0000","last_clean":"2024-09-10T19:28:30.074295+0000","last_became_active":"2024-09-10T19:20:30.844108+0000","last_became_peered":"2024-09-10T19:20:30.844108+0000","last_unstale":"2024-09-10T19:28:30.074295+0000","last_undegraded":"2024-09-10T19:28:30.074295+0000","last_fullsized":"2024-09-10T19:28:30.074295+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":4,"last_scrub":"74'79","last_scrub_stamp":"2024-09-10T19:27:13.532411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:08:00.699315+0000","last_clean_scrub_stamp":"2024-09-10T19:27:13.532411+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:28:40.580879+0000","scrub_duration":0.001570758,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.c","version":"237'20","reported_seq":490,"reported_epoch":385,"state":"active+clean","last_fresh":"2024-09-10T19:28:30.069439+0000","last_change":"2024-09-10T19:28:12.871441+0000","last_active":"2024-09-10T19:28:30.069439+0000","last_peered":"2024-09-10T19:28:30.069439+0000","last_clean":"2024-09-10T19:28:30.069439+0000","last_became_active":"2024-09-10T19:27:53.012617+0000","last_became_peered":"2024-09-10T19:27:53.012617+0000","last_unstale":"2024-09-10T19:28:30.069439+0000","last_undegraded":"2024-09-10T19:28:30.069439+0000","last_fullsized":"2024-09-10T19:28:30.069439+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"237'20","last_scrub_stamp":"2024-09-10T19:28:12.871325+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:28:12.871325+0000","objects_scrubbed":5,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:29:18.028584+0000","scrub_duration":0.0069951600000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":123,"reported_epoch":385,"state":"active+clean","last_fresh":"2024-09-10T19:28:30.067868+0000","last_change":"2024-09-10T19:28:16.491452+0000","last_active":"2024-09-10T19:28:30.067868+0000","last_peered":"2024-09-10T19:28:30.067868+0000","last_clean":"2024-09-10T19:28:30.067868+0000","last_became_active":"2024-09-10T19:24:01.973848+0000","last_became_peered":"2024-09-10T19:24:01.973848+0000","last_unstale":"2024-09-10T19:28:30.067868+0000","last_undegraded":"2024-09-10T19:28:30.067868+0000","last_fullsized":"2024-09-10T19:28:30.067868+0000","mapping_epoch":292,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":293,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:28:16.491371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:28:16.491371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:29:40.642402+0000","scrub_duration":0.014455551000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.d","version":"303'12","reported_seq":509,"reported_epoch":385,"state":"active+clean","last_fresh":"2024-09-10T19:28:30.947748+0000","last_change":"2024-09-10T19:27:10.521498+0000","last_active":"2024-09-10T19:28:30.947748+0000","last_peered":"2024-09-10T19:28:30.947748+0000","last_clean":"2024-09-10T19:28:30.947748+0000","last_became_active":"2024-09-10T19:24:23.906819+0000","last_became_peered":"2024-09-10T19:24:23.906819+0000","last_unstale":"2024-09-10T19:28:30.947748+0000","last_undegraded":"2024-09-10T19:28:30.947748+0000","last_fullsized":"2024-09-10T19:28:30.947748+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"303'12","last_scrub_stamp":"2024-09-10T19:27:10.521358+0000","last_deep_scrub":"80'9","last_deep_scrub_stamp":"2024-09-10T19:11:25.156465+0000","last_clean_scrub_stamp":"2024-09-10T19:27:10.521358+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0040348709999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,0],"acting":[7,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":432,"reported_epoch":385,"state":"active+clean","last_fresh":"2024-09-10T19:28:30.069770+0000","last_change":"2024-09-10T19:26:13.829815+0000","last_active":"2024-09-10T19:28:30.069770+0000","last_peered":"2024-09-10T19:28:30.069770+0000","last_clean":"2024-09-10T19:28:30.069770+0000","last_became_active":"2024-09-10T19:24:01.972593+0000","last_became_peered":"2024-09-10T19:24:01.972593+0000","last_unstale":"2024-09-10T19:28:30.069770+0000","last_undegraded":"2024-09-10T19:28:30.069770+0000","last_fullsized":"2024-09-10T19:28:30.069770+0000","mapping_epoch":292,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":293,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:26:13.829736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:26:13.829736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.025408242000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,5],"acting":[4,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.0","version":"386'5113","reported_seq":5136,"reported_epoch":386,"state":"active+clean","last_fresh":"2024-09-10T19:28:37.258993+0000","last_change":"2024-09-10T19:27:46.705815+0000","last_active":"2024-09-10T19:28:37.258993+0000","last_peered":"2024-09-10T19:28:37.258993+0000","last_clean":"2024-09-10T19:28:37.258993+0000","last_became_active":"2024-09-10T19:27:46.705538+0000","last_became_peered":"2024-09-10T19:27:46.705538+0000","last_unstale":"2024-09-10T19:28:37.258993+0000","last_undegraded":"2024-09-10T19:28:37.258993+0000","last_fullsized":"2024-09-10T19:28:37.258993+0000","mapping_epoch":371,"log_start":"376'800","ondisk_log_start":"376'800","created":371,"last_epoch_clean":372,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:27:45.679464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:27:45.679464+0000","last_clean_scrub_stamp":"2024-09-10T19:27:45.679464+0000","objects_scrubbed":0,"log_size":4313,"log_dups_size":0,"ondisk_log_size":4313,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:28:56.976030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":335085568,"num_objects":5113,"num_object_clones":0,"num_object_copies":15339,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5113,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5113,"num_write_kb":327232,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"74'79","reported_seq":201,"reported_epoch":385,"state":"active+clean","last_fresh":"2024-09-10T19:28:30.079137+0000","last_change":"2024-09-10T19:28:10.021054+0000","last_active":"2024-09-10T19:28:30.079137+0000","last_peered":"2024-09-10T19:28:30.079137+0000","last_clean":"2024-09-10T19:28:30.079137+0000","last_became_active":"2024-09-10T19:27:53.004646+0000","last_became_peered":"2024-09-10T19:27:53.004646+0000","last_unstale":"2024-09-10T19:28:30.079137+0000","last_undegraded":"2024-09-10T19:28:30.079137+0000","last_fullsized":"2024-09-10T19:28:30.079137+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":4,"last_scrub":"74'79","last_scrub_stamp":"2024-09-10T19:28:10.020996+0000","last_deep_scrub":"74'79","last_deep_scrub_stamp":"2024-09-10T19:21:53.326433+0000","last_clean_scrub_stamp":"2024-09-10T19:28:10.020996+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:29:25.495279+0000","scrub_duration":0.0026290670000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,4],"acting":[5,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.f","version":"237'27","reported_seq":139,"reported_epoch":385,"state":"active+clean","last_fresh":"2024-09-10T19:28:30.068143+0000","last_change":"2024-09-10T19:28:26.434825+0000","last_active":"2024-09-10T19:28:30.068143+0000","last_peered":"2024-09-10T19:28:30.068143+0000","last_clean":"2024-09-10T19:28:30.068143+0000","last_became_active":"2024-09-10T19:24:01.979187+0000","last_became_peered":"2024-09-10T19:24:01.979187+0000","last_unstale":"2024-09-10T19:28:30.068143+0000","last_undegraded":"2024-09-10T19:28:30.068143+0000","last_fullsized":"2024-09-10T19:28:30.068143+0000","mapping_epoch":292,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":293,"parent":"0.0","parent_split_bits":0,"last_scrub":"237'27","last_scrub_stamp":"2024-09-10T19:28:26.434563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:28:26.434563+0000","objects_scrubbed":7,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:29:46.632200+0000","scrub_duration":0.0096637310000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":123,"reported_epoch":385,"state":"active+clean","last_fresh":"2024-09-10T19:28:30.069134+0000","last_change":"2024-09-10T19:28:24.420346+0000","last_active":"2024-09-10T19:28:30.069134+0000","last_peered":"2024-09-10T19:28:30.069134+0000","last_clean":"2024-09-10T19:28:30.069134+0000","last_became_active":"2024-09-10T19:24:01.974773+0000","last_became_peered":"2024-09-10T19:24:01.974773+0000","last_unstale":"2024-09-10T19:28:30.069134+0000","last_undegraded":"2024-09-10T19:28:30.069134+0000","last_fullsized":"2024-09-10T19:28:30.069134+0000","mapping_epoch":292,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":293,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:28:24.420270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:28:24.420270+0000","last_clean_scrub_stamp":"2024-09-10T19:28:24.420270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:29:54.313868+0000","scrub_duration":0.016717709000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.e","version":"303'10","reported_seq":632,"reported_epoch":385,"state":"active+clean","last_fresh":"2024-09-10T19:28:30.947467+0000","last_change":"2024-09-10T19:27:55.933279+0000","last_active":"2024-09-10T19:28:30.947467+0000","last_peered":"2024-09-10T19:28:30.947467+0000","last_clean":"2024-09-10T19:28:30.947467+0000","last_became_active":"2024-09-10T19:24:23.908760+0000","last_became_peered":"2024-09-10T19:24:23.908760+0000","last_unstale":"2024-09-10T19:28:30.947467+0000","last_undegraded":"2024-09-10T19:28:30.947467+0000","last_fullsized":"2024-09-10T19:28:30.947467+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"303'10","last_scrub_stamp":"2024-09-10T19:27:55.933189+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:27:55.933189+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:29:12.409131+0000","scrub_duration":0.0045590129999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":438,"reported_epoch":385,"state":"active+clean","last_fresh":"2024-09-10T19:28:30.079907+0000","last_change":"2024-09-10T19:26:34.847681+0000","last_active":"2024-09-10T19:28:30.079907+0000","last_peered":"2024-09-10T19:28:30.079907+0000","last_clean":"2024-09-10T19:28:30.079907+0000","last_became_active":"2024-09-10T19:24:24.917109+0000","last_became_peered":"2024-09-10T19:24:24.917109+0000","last_unstale":"2024-09-10T19:28:30.079907+0000","last_undegraded":"2024-09-10T19:28:30.079907+0000","last_fullsized":"2024-09-10T19:28:30.079907+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:26:34.847602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:26:34.847602+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0034671599999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.1","version":"385'4887","reported_seq":4907,"reported_epoch":385,"state":"active+clean","last_fresh":"2024-09-10T19:28:34.767504+0000","last_change":"2024-09-10T19:27:53.017680+0000","last_active":"2024-09-10T19:28:34.767504+0000","last_peered":"2024-09-10T19:28:34.767504+0000","last_clean":"2024-09-10T19:28:34.767504+0000","last_became_active":"2024-09-10T19:27:53.016565+0000","last_became_peered":"2024-09-10T19:27:53.016565+0000","last_unstale":"2024-09-10T19:28:34.767504+0000","last_undegraded":"2024-09-10T19:28:34.767504+0000","last_fullsized":"2024-09-10T19:28:34.767504+0000","mapping_epoch":374,"log_start":"375'200","ondisk_log_start":"375'200","created":371,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:27:45.679464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:27:45.679464+0000","last_clean_scrub_stamp":"2024-09-10T19:27:45.679464+0000","objects_scrubbed":0,"log_size":4687,"log_dups_size":0,"ondisk_log_size":4687,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:29:13.640153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":320274432,"num_objects":4887,"num_object_clones":0,"num_object_copies":14661,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4887,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4887,"num_write_kb":312768,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.e","version":"237'24","reported_seq":559,"reported_epoch":385,"state":"active+clean","last_fresh":"2024-09-10T19:28:30.070002+0000","last_change":"2024-09-10T19:28:11.887030+0000","last_active":"2024-09-10T19:28:30.070002+0000","last_peered":"2024-09-10T19:28:30.070002+0000","last_clean":"2024-09-10T19:28:30.070002+0000","last_became_active":"2024-09-10T19:27:53.012101+0000","last_became_peered":"2024-09-10T19:27:53.012101+0000","last_unstale":"2024-09-10T19:28:30.070002+0000","last_undegraded":"2024-09-10T19:28:30.070002+0000","last_fullsized":"2024-09-10T19:28:30.070002+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"237'24","last_scrub_stamp":"2024-09-10T19:28:11.886976+0000","last_deep_scrub":"84'16","last_deep_scrub_stamp":"2024-09-10T19:11:56.157975+0000","last_clean_scrub_stamp":"2024-09-10T19:28:11.886976+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:29:31.540270+0000","scrub_duration":0.001647307,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":550,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":87,"reported_epoch":385,"state":"active+clean","last_fresh":"2024-09-10T19:28:30.066393+0000","last_change":"2024-09-10T19:27:14.793074+0000","last_active":"2024-09-10T19:28:30.066393+0000","last_peered":"2024-09-10T19:28:30.066393+0000","last_clean":"2024-09-10T19:28:30.066393+0000","last_became_active":"2024-09-10T19:24:53.756220+0000","last_became_peered":"2024-09-10T19:24:53.756220+0000","last_unstale":"2024-09-10T19:28:30.066393+0000","last_undegraded":"2024-09-10T19:28:30.066393+0000","last_fullsized":"2024-09-10T19:28:30.066393+0000","mapping_epoch":318,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":319,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:27:14.793013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:24:13.636249+0000","last_clean_scrub_stamp":"2024-09-10T19:27:14.793013+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:28:39.183835+0000","scrub_duration":0.001277001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,3],"acting":[4,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":412,"reported_epoch":385,"state":"active+clean","last_fresh":"2024-09-10T19:28:30.079943+0000","last_change":"2024-09-10T19:28:09.029321+0000","last_active":"2024-09-10T19:28:30.079943+0000","last_peered":"2024-09-10T19:28:30.079943+0000","last_clean":"2024-09-10T19:28:30.079943+0000","last_became_active":"2024-09-10T19:27:53.007094+0000","last_became_peered":"2024-09-10T19:27:53.007094+0000","last_unstale":"2024-09-10T19:28:30.079943+0000","last_undegraded":"2024-09-10T19:28:30.079943+0000","last_fullsized":"2024-09-10T19:28:30.079943+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:28:09.028965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:28:09.028965+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:29:37.370546+0000","scrub_duration":0.0058230010000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":410,"reported_epoch":385,"state":"active+clean","last_fresh":"2024-09-10T19:28:30.070162+0000","last_change":"2024-09-10T19:28:13.177491+0000","last_active":"2024-09-10T19:28:30.070162+0000","last_peered":"2024-09-10T19:28:30.070162+0000","last_clean":"2024-09-10T19:28:30.070162+0000","last_became_active":"2024-09-10T19:27:53.025345+0000","last_became_peered":"2024-09-10T19:27:53.025345+0000","last_unstale":"2024-09-10T19:28:30.070162+0000","last_undegraded":"2024-09-10T19:28:30.070162+0000","last_fullsized":"2024-09-10T19:28:30.070162+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:28:13.177417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:28:13.177417+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:29:37.803353+0000","scrub_duration":0.0025953450000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.e","version":"386'4954","reported_seq":4977,"reported_epoch":386,"state":"active+clean","last_fresh":"2024-09-10T19:28:35.206632+0000","last_change":"2024-09-10T19:27:46.704421+0000","last_active":"2024-09-10T19:28:35.206632+0000","last_peered":"2024-09-10T19:28:35.206632+0000","last_clean":"2024-09-10T19:28:35.206632+0000","last_became_active":"2024-09-10T19:27:46.704103+0000","last_became_peered":"2024-09-10T19:27:46.704103+0000","last_unstale":"2024-09-10T19:28:35.206632+0000","last_undegraded":"2024-09-10T19:28:35.206632+0000","last_fullsized":"2024-09-10T19:28:35.206632+0000","mapping_epoch":371,"log_start":"376'300","ondisk_log_start":"376'300","created":371,"last_epoch_clean":372,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:27:45.679464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:27:45.679464+0000","last_clean_scrub_stamp":"2024-09-10T19:27:45.679464+0000","objects_scrubbed":0,"log_size":4654,"log_dups_size":0,"ondisk_log_size":4654,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:28:49.870446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":324665344,"num_objects":4954,"num_object_clones":0,"num_object_copies":14862,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4954,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4954,"num_write_kb":317056,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,7],"acting":[4,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.4","version":"317'80","reported_seq":187,"reported_epoch":385,"state":"active+clean","last_fresh":"2024-09-10T19:28:30.067525+0000","last_change":"2024-09-10T19:26:34.932583+0000","last_active":"2024-09-10T19:28:30.067525+0000","last_peered":"2024-09-10T19:28:30.067525+0000","last_clean":"2024-09-10T19:28:30.067525+0000","last_became_active":"2024-09-10T19:20:30.838441+0000","last_became_peered":"2024-09-10T19:20:30.838441+0000","last_unstale":"2024-09-10T19:28:30.067525+0000","last_undegraded":"2024-09-10T19:28:30.067525+0000","last_fullsized":"2024-09-10T19:28:30.067525+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":3,"last_scrub":"317'80","last_scrub_stamp":"2024-09-10T19:26:34.932490+0000","last_deep_scrub":"317'80","last_deep_scrub_stamp":"2024-09-10T19:24:51.540456+0000","last_clean_scrub_stamp":"2024-09-10T19:26:34.932490+0000","objects_scrubbed":1,"log_size":80,"log_dups_size":0,"ondisk_log_size":80,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.010119784999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"265'35","reported_seq":390,"reported_epoch":385,"state":"active+clean","last_fresh":"2024-09-10T19:28:30.066396+0000","last_change":"2024-09-10T19:28:24.824514+0000","last_active":"2024-09-10T19:28:30.066396+0000","last_peered":"2024-09-10T19:28:30.066396+0000","last_clean":"2024-09-10T19:28:30.066396+0000","last_became_active":"2024-09-10T19:27:53.011799+0000","last_became_peered":"2024-09-10T19:27:53.011799+0000","last_unstale":"2024-09-10T19:28:30.066396+0000","last_undegraded":"2024-09-10T19:28:30.066396+0000","last_fullsized":"2024-09-10T19:28:30.066396+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"265'35","last_scrub_stamp":"2024-09-10T19:28:24.824459+0000","last_deep_scrub":"265'35","last_deep_scrub_stamp":"2024-09-10T19:28:24.824459+0000","last_clean_scrub_stamp":"2024-09-10T19:28:24.824459+0000","objects_scrubbed":7,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:29:34.083971+0000","scrub_duration":0.00203419,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":42,"num_read_kb":35,"num_write":28,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":20,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-09-10T19:28:19.956163+0000","last_change":"2024-09-10T19:27:53.020775+0000","last_active":"2024-09-10T19:28:19.956163+0000","last_peered":"2024-09-10T19:28:19.956163+0000","last_clean":"2024-09-10T19:28:19.956163+0000","last_became_active":"2024-09-10T19:27:53.020498+0000","last_became_peered":"2024-09-10T19:27:53.020498+0000","last_unstale":"2024-09-10T19:28:19.956163+0000","last_undegraded":"2024-09-10T19:28:19.956163+0000","last_fullsized":"2024-09-10T19:28:19.956163+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:27:26.633920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:27:26.633920+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:28:42.542572+0000","scrub_duration":0.0035531249999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.0","version":"294'12","reported_seq":665,"reported_epoch":385,"state":"active+clean","last_fresh":"2024-09-10T19:28:30.950664+0000","last_change":"2024-09-10T19:28:28.202557+0000","last_active":"2024-09-10T19:28:30.950664+0000","last_peered":"2024-09-10T19:28:30.950664+0000","last_clean":"2024-09-10T19:28:30.950664+0000","last_became_active":"2024-09-10T19:24:01.972645+0000","last_became_peered":"2024-09-10T19:24:01.972645+0000","last_unstale":"2024-09-10T19:28:30.950664+0000","last_undegraded":"2024-09-10T19:28:30.950664+0000","last_fullsized":"2024-09-10T19:28:30.950664+0000","mapping_epoch":292,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":293,"parent":"0.0","parent_split_bits":0,"last_scrub":"294'12","last_scrub_stamp":"2024-09-10T19:28:28.202485+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:28:28.202485+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:29:55.536626+0000","scrub_duration":0.0051773380000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":437,"reported_epoch":385,"state":"active+clean","last_fresh":"2024-09-10T19:28:30.074074+0000","last_change":"2024-09-10T19:28:25.822999+0000","last_active":"2024-09-10T19:28:30.074074+0000","last_peered":"2024-09-10T19:28:30.074074+0000","last_clean":"2024-09-10T19:28:30.074074+0000","last_became_active":"2024-09-10T19:27:53.009392+0000","last_became_peered":"2024-09-10T19:27:53.009392+0000","last_unstale":"2024-09-10T19:28:30.074074+0000","last_undegraded":"2024-09-10T19:28:30.074074+0000","last_fullsized":"2024-09-10T19:28:30.074074+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:28:25.822918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:28:25.822918+0000","last_clean_scrub_stamp":"2024-09-10T19:28:25.822918+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:29:26.703747+0000","scrub_duration":0.0020554969999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.f","version":"386'4957","reported_seq":4980,"reported_epoch":386,"state":"active+clean","last_fresh":"2024-09-10T19:28:35.456469+0000","last_change":"2024-09-10T19:27:46.707916+0000","last_active":"2024-09-10T19:28:35.456469+0000","last_peered":"2024-09-10T19:28:35.456469+0000","last_clean":"2024-09-10T19:28:35.456469+0000","last_became_active":"2024-09-10T19:27:46.707637+0000","last_became_peered":"2024-09-10T19:27:46.707637+0000","last_unstale":"2024-09-10T19:28:35.456469+0000","last_undegraded":"2024-09-10T19:28:35.456469+0000","last_fullsized":"2024-09-10T19:28:35.456469+0000","mapping_epoch":371,"log_start":"376'700","ondisk_log_start":"376'700","created":371,"last_epoch_clean":372,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:27:45.679464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:27:45.679464+0000","last_clean_scrub_stamp":"2024-09-10T19:27:45.679464+0000","objects_scrubbed":0,"log_size":4257,"log_dups_size":0,"ondisk_log_size":4257,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:29:14.475828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":324861952,"num_objects":4957,"num_object_clones":0,"num_object_copies":14871,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4957,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4957,"num_write_kb":317248,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.5","version":"74'79","reported_seq":211,"reported_epoch":385,"state":"active+clean","last_fresh":"2024-09-10T19:28:30.067751+0000","last_change":"2024-09-10T19:28:10.903017+0000","last_active":"2024-09-10T19:28:30.067751+0000","last_peered":"2024-09-10T19:28:30.067751+0000","last_clean":"2024-09-10T19:28:30.067751+0000","last_became_active":"2024-09-10T19:27:53.013021+0000","last_became_peered":"2024-09-10T19:27:53.013021+0000","last_unstale":"2024-09-10T19:28:30.067751+0000","last_undegraded":"2024-09-10T19:28:30.067751+0000","last_fullsized":"2024-09-10T19:28:30.067751+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":3,"last_scrub":"74'79","last_scrub_stamp":"2024-09-10T19:28:10.902933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:08:00.699315+0000","last_clean_scrub_stamp":"2024-09-10T19:28:10.902933+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:29:18.031740+0000","scrub_duration":0.004069792,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"237'30","reported_seq":451,"reported_epoch":385,"state":"active+clean","last_fresh":"2024-09-10T19:28:30.066899+0000","last_change":"2024-09-10T19:27:12.218512+0000","last_active":"2024-09-10T19:28:30.066899+0000","last_peered":"2024-09-10T19:28:30.066899+0000","last_clean":"2024-09-10T19:28:30.066899+0000","last_became_active":"2024-09-10T19:24:23.903784+0000","last_became_peered":"2024-09-10T19:24:23.903784+0000","last_unstale":"2024-09-10T19:28:30.066899+0000","last_undegraded":"2024-09-10T19:28:30.066899+0000","last_fullsized":"2024-09-10T19:28:30.066899+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"237'30","last_scrub_stamp":"2024-09-10T19:27:12.218416+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:27:12.218416+0000","objects_scrubbed":8,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0048686429999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":52,"num_read_kb":40,"num_write":32,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":143,"reported_epoch":385,"state":"active+clean","last_fresh":"2024-09-10T19:28:30.069198+0000","last_change":"2024-09-10T19:28:15.534424+0000","last_active":"2024-09-10T19:28:30.069198+0000","last_peered":"2024-09-10T19:28:30.069198+0000","last_clean":"2024-09-10T19:28:30.069198+0000","last_became_active":"2024-09-10T19:27:53.008302+0000","last_became_peered":"2024-09-10T19:27:53.008302+0000","last_unstale":"2024-09-10T19:28:30.069198+0000","last_undegraded":"2024-09-10T19:28:30.069198+0000","last_fullsized":"2024-09-10T19:28:30.069198+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:28:15.534211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:28:15.534211+0000","last_clean_scrub_stamp":"2024-09-10T19:28:15.534211+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:29:17.016383+0000","scrub_duration":0.028349828000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":401,"reported_epoch":385,"state":"active+clean","last_fresh":"2024-09-10T19:28:30.067370+0000","last_change":"2024-09-10T19:26:24.897597+0000","last_active":"2024-09-10T19:28:30.067370+0000","last_peered":"2024-09-10T19:28:30.067370+0000","last_clean":"2024-09-10T19:28:30.067370+0000","last_became_active":"2024-09-10T19:10:11.698054+0000","last_became_peered":"2024-09-10T19:10:11.698054+0000","last_unstale":"2024-09-10T19:28:30.067370+0000","last_undegraded":"2024-09-10T19:28:30.067370+0000","last_fullsized":"2024-09-10T19:28:30.067370+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:26:24.897519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:26:24.897519+0000","last_clean_scrub_stamp":"2024-09-10T19:26:24.897519+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0035237760000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":429,"reported_epoch":385,"state":"active+clean","last_fresh":"2024-09-10T19:28:30.067131+0000","last_change":"2024-09-10T19:27:53.017450+0000","last_active":"2024-09-10T19:28:30.067131+0000","last_peered":"2024-09-10T19:28:30.067131+0000","last_clean":"2024-09-10T19:28:30.067131+0000","last_became_active":"2024-09-10T19:27:53.017161+0000","last_became_peered":"2024-09-10T19:27:53.017161+0000","last_unstale":"2024-09-10T19:28:30.067131+0000","last_undegraded":"2024-09-10T19:28:30.067131+0000","last_fullsized":"2024-09-10T19:28:30.067131+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:27:07.753784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:27:07.753784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0086359330000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.c","version":"386'5078","reported_seq":5101,"reported_epoch":386,"state":"active+clean","last_fresh":"2024-09-10T19:28:37.280898+0000","last_change":"2024-09-10T19:27:46.705826+0000","last_active":"2024-09-10T19:28:37.280898+0000","last_peered":"2024-09-10T19:28:37.280898+0000","last_clean":"2024-09-10T19:28:37.280898+0000","last_became_active":"2024-09-10T19:27:46.705563+0000","last_became_peered":"2024-09-10T19:27:46.705563+0000","last_unstale":"2024-09-10T19:28:37.280898+0000","last_undegraded":"2024-09-10T19:28:37.280898+0000","last_fullsized":"2024-09-10T19:28:37.280898+0000","mapping_epoch":371,"log_start":"376'700","ondisk_log_start":"376'700","created":371,"last_epoch_clean":372,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:27:45.679464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:27:45.679464+0000","last_clean_scrub_stamp":"2024-09-10T19:27:45.679464+0000","objects_scrubbed":0,"log_size":4378,"log_dups_size":0,"ondisk_log_size":4378,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:28:54.464452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":332791808,"num_objects":5078,"num_object_clones":0,"num_object_copies":15234,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5078,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5078,"num_write_kb":324992,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,7],"acting":[1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"74'79","reported_seq":20,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-09-10T19:28:19.955442+0000","last_change":"2024-09-10T19:27:53.018301+0000","last_active":"2024-09-10T19:28:19.955442+0000","last_peered":"2024-09-10T19:28:19.955442+0000","last_clean":"2024-09-10T19:28:19.955442+0000","last_became_active":"2024-09-10T19:27:53.017899+0000","last_became_peered":"2024-09-10T19:27:53.017899+0000","last_unstale":"2024-09-10T19:28:19.955442+0000","last_undegraded":"2024-09-10T19:28:19.955442+0000","last_fullsized":"2024-09-10T19:28:19.955442+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":3,"last_scrub":"74'79","last_scrub_stamp":"2024-09-10T19:27:31.381498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:08:00.699315+0000","last_clean_scrub_stamp":"2024-09-10T19:27:31.381498+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:28:39.434523+0000","scrub_duration":0.0015974209999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"237'37","reported_seq":411,"reported_epoch":385,"state":"active+clean","last_fresh":"2024-09-10T19:28:30.062658+0000","last_change":"2024-09-10T19:28:28.361531+0000","last_active":"2024-09-10T19:28:30.062658+0000","last_peered":"2024-09-10T19:28:30.062658+0000","last_clean":"2024-09-10T19:28:30.062658+0000","last_became_active":"2024-09-10T19:24:23.901836+0000","last_became_peered":"2024-09-10T19:24:23.901836+0000","last_unstale":"2024-09-10T19:28:30.062658+0000","last_undegraded":"2024-09-10T19:28:30.062658+0000","last_fullsized":"2024-09-10T19:28:30.062658+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"237'37","last_scrub_stamp":"2024-09-10T19:28:28.361309+0000","last_deep_scrub":"104'27","last_deep_scrub_stamp":"2024-09-10T19:13:02.225944+0000","last_clean_scrub_stamp":"2024-09-10T19:28:28.361309+0000","objects_scrubbed":8,"log_size":37,"log_dups_size":0,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:29:29.927784+0000","scrub_duration":0.0036049300000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":53,"num_read_kb":39,"num_write":32,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":413,"reported_epoch":385,"state":"active+clean","last_fresh":"2024-09-10T19:28:30.075883+0000","last_change":"2024-09-10T19:26:29.426781+0000","last_active":"2024-09-10T19:28:30.075883+0000","last_peered":"2024-09-10T19:28:30.075883+0000","last_clean":"2024-09-10T19:28:30.075883+0000","last_became_active":"2024-09-10T19:24:23.899556+0000","last_became_peered":"2024-09-10T19:24:23.899556+0000","last_unstale":"2024-09-10T19:28:30.075883+0000","last_undegraded":"2024-09-10T19:28:30.075883+0000","last_fullsized":"2024-09-10T19:28:30.075883+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:26:29.426689+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:16:37.779967+0000","last_clean_scrub_stamp":"2024-09-10T19:26:29.426689+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0054033600000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":30,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-09-10T19:28:27.479115+0000","last_change":"2024-09-10T19:28:27.478990+0000","last_active":"2024-09-10T19:28:27.479115+0000","last_peered":"2024-09-10T19:28:27.479115+0000","last_clean":"2024-09-10T19:28:27.479115+0000","last_became_active":"2024-09-10T19:27:53.003734+0000","last_became_peered":"2024-09-10T19:27:53.003734+0000","last_unstale":"2024-09-10T19:28:27.479115+0000","last_undegraded":"2024-09-10T19:28:27.479115+0000","last_fullsized":"2024-09-10T19:28:27.479115+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:28:27.478726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:28:27.478726+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:29:46.895125+0000","scrub_duration":0.0074560529999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":143,"reported_epoch":385,"state":"active+clean","last_fresh":"2024-09-10T19:28:30.069061+0000","last_change":"2024-09-10T19:27:53.017667+0000","last_active":"2024-09-10T19:28:30.069061+0000","last_peered":"2024-09-10T19:28:30.069061+0000","last_clean":"2024-09-10T19:28:30.069061+0000","last_became_active":"2024-09-10T19:27:53.016905+0000","last_became_peered":"2024-09-10T19:27:53.016905+0000","last_unstale":"2024-09-10T19:28:30.069061+0000","last_undegraded":"2024-09-10T19:28:30.069061+0000","last_fullsized":"2024-09-10T19:28:30.069061+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:27:21.382671+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:27:21.382671+0000","last_clean_scrub_stamp":"2024-09-10T19:27:21.382671+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:28:41.008523+0000","scrub_duration":0.001581863,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,7],"acting":[1,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.d","version":"386'4956","reported_seq":4989,"reported_epoch":386,"state":"active+clean","last_fresh":"2024-09-10T19:28:35.211612+0000","last_change":"2024-09-10T19:27:53.011704+0000","last_active":"2024-09-10T19:28:35.211612+0000","last_peered":"2024-09-10T19:28:35.211612+0000","last_clean":"2024-09-10T19:28:35.211612+0000","last_became_active":"2024-09-10T19:27:53.011209+0000","last_became_peered":"2024-09-10T19:27:53.011209+0000","last_unstale":"2024-09-10T19:28:35.211612+0000","last_undegraded":"2024-09-10T19:28:35.211612+0000","last_fullsized":"2024-09-10T19:28:35.211612+0000","mapping_epoch":374,"log_start":"376'300","ondisk_log_start":"376'300","created":371,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:27:45.679464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:27:45.679464+0000","last_clean_scrub_stamp":"2024-09-10T19:27:45.679464+0000","objects_scrubbed":0,"log_size":4656,"log_dups_size":0,"ondisk_log_size":4656,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:29:02.018676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":324796416,"num_objects":4956,"num_object_clones":0,"num_object_copies":14868,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4956,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4956,"num_write_kb":317184,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.7","version":"74'79","reported_seq":30,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-09-10T19:28:26.515983+0000","last_change":"2024-09-10T19:28:26.515921+0000","last_active":"2024-09-10T19:28:26.515983+0000","last_peered":"2024-09-10T19:28:26.515983+0000","last_clean":"2024-09-10T19:28:26.515983+0000","last_became_active":"2024-09-10T19:27:53.020085+0000","last_became_peered":"2024-09-10T19:27:53.020085+0000","last_unstale":"2024-09-10T19:28:26.515983+0000","last_undegraded":"2024-09-10T19:28:26.515983+0000","last_fullsized":"2024-09-10T19:28:26.515983+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":3,"last_scrub":"74'79","last_scrub_stamp":"2024-09-10T19:28:26.515845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:08:00.699315+0000","last_clean_scrub_stamp":"2024-09-10T19:28:26.515845+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:29:30.215115+0000","scrub_duration":0.001888685,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.2","version":"350'25","reported_seq":140,"reported_epoch":385,"state":"active+clean","last_fresh":"2024-09-10T19:28:30.068181+0000","last_change":"2024-09-10T19:26:37.415152+0000","last_active":"2024-09-10T19:28:30.068181+0000","last_peered":"2024-09-10T19:28:30.068181+0000","last_clean":"2024-09-10T19:28:30.068181+0000","last_became_active":"2024-09-10T19:24:01.976841+0000","last_became_peered":"2024-09-10T19:24:01.976841+0000","last_unstale":"2024-09-10T19:28:30.068181+0000","last_undegraded":"2024-09-10T19:28:30.068181+0000","last_fullsized":"2024-09-10T19:28:30.068181+0000","mapping_epoch":292,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":293,"parent":"0.0","parent_split_bits":0,"last_scrub":"350'25","last_scrub_stamp":"2024-09-10T19:26:37.415039+0000","last_deep_scrub":"350'25","last_deep_scrub_stamp":"2024-09-10T19:26:37.415039+0000","last_clean_scrub_stamp":"2024-09-10T19:26:37.415039+0000","objects_scrubbed":5,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.022344089000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":28,"num_read_kb":23,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":137,"reported_epoch":385,"state":"active+clean","last_fresh":"2024-09-10T19:28:30.072513+0000","last_change":"2024-09-10T19:27:10.344762+0000","last_active":"2024-09-10T19:28:30.072513+0000","last_peered":"2024-09-10T19:28:30.072513+0000","last_clean":"2024-09-10T19:28:30.072513+0000","last_became_active":"2024-09-10T19:24:24.910173+0000","last_became_peered":"2024-09-10T19:24:24.910173+0000","last_unstale":"2024-09-10T19:28:30.072513+0000","last_undegraded":"2024-09-10T19:28:30.072513+0000","last_fullsized":"2024-09-10T19:28:30.072513+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:27:10.344704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:27:10.344704+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0044944579999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"376'13","reported_seq":578,"reported_epoch":386,"state":"active+clean","last_fresh":"2024-09-10T19:28:34.967955+0000","last_change":"2024-09-10T19:28:17.490484+0000","last_active":"2024-09-10T19:28:34.967955+0000","last_peered":"2024-09-10T19:28:34.967955+0000","last_clean":"2024-09-10T19:28:34.967955+0000","last_became_active":"2024-09-10T19:27:53.000311+0000","last_became_peered":"2024-09-10T19:27:53.000311+0000","last_unstale":"2024-09-10T19:28:34.967955+0000","last_undegraded":"2024-09-10T19:28:34.967955+0000","last_fullsized":"2024-09-10T19:28:34.967955+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"376'13","last_scrub_stamp":"2024-09-10T19:28:17.490297+0000","last_deep_scrub":"376'13","last_deep_scrub_stamp":"2024-09-10T19:28:17.490297+0000","last_clean_scrub_stamp":"2024-09-10T19:28:17.490297+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:29:27.712220+0000","scrub_duration":0.0032574890000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":366,"reported_epoch":385,"state":"active+clean","last_fresh":"2024-09-10T19:28:30.062909+0000","last_change":"2024-09-10T19:27:08.514626+0000","last_active":"2024-09-10T19:28:30.062909+0000","last_peered":"2024-09-10T19:28:30.062909+0000","last_clean":"2024-09-10T19:28:30.062909+0000","last_became_active":"2024-09-10T19:24:23.905977+0000","last_became_peered":"2024-09-10T19:24:23.905977+0000","last_unstale":"2024-09-10T19:28:30.062909+0000","last_undegraded":"2024-09-10T19:28:30.062909+0000","last_fullsized":"2024-09-10T19:28:30.062909+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:27:08.514575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:27:08.514575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.004157361,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,5],"acting":[0,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.b","version":"386'4892","reported_seq":4913,"reported_epoch":386,"state":"active+clean","last_fresh":"2024-09-10T19:28:35.219520+0000","last_change":"2024-09-10T19:27:46.709806+0000","last_active":"2024-09-10T19:28:35.219520+0000","last_peered":"2024-09-10T19:28:35.219520+0000","last_clean":"2024-09-10T19:28:35.219520+0000","last_became_active":"2024-09-10T19:27:46.709536+0000","last_became_peered":"2024-09-10T19:27:46.709536+0000","last_unstale":"2024-09-10T19:28:35.219520+0000","last_undegraded":"2024-09-10T19:28:35.219520+0000","last_fullsized":"2024-09-10T19:28:35.219520+0000","mapping_epoch":371,"log_start":"376'700","ondisk_log_start":"376'700","created":371,"last_epoch_clean":372,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:27:45.679464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:27:45.679464+0000","last_clean_scrub_stamp":"2024-09-10T19:27:45.679464+0000","objects_scrubbed":0,"log_size":4192,"log_dups_size":0,"ondisk_log_size":4192,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:28:47.258194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":320602112,"num_objects":4892,"num_object_clones":0,"num_object_copies":14676,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4892,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4892,"num_write_kb":313088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"74'79","reported_seq":188,"reported_epoch":385,"state":"active+clean","last_fresh":"2024-09-10T19:28:30.078613+0000","last_change":"2024-09-10T19:28:21.898202+0000","last_active":"2024-09-10T19:28:30.078613+0000","last_peered":"2024-09-10T19:28:30.078613+0000","last_clean":"2024-09-10T19:28:30.078613+0000","last_became_active":"2024-09-10T19:24:23.907264+0000","last_became_peered":"2024-09-10T19:24:23.907264+0000","last_unstale":"2024-09-10T19:28:30.078613+0000","last_undegraded":"2024-09-10T19:28:30.078613+0000","last_fullsized":"2024-09-10T19:28:30.078613+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":4,"last_scrub":"74'79","last_scrub_stamp":"2024-09-10T19:28:21.898118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:08:00.699315+0000","last_clean_scrub_stamp":"2024-09-10T19:28:21.898118+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:29:35.403601+0000","scrub_duration":0.0024691890000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,4],"acting":[5,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"384'56","reported_seq":274,"reported_epoch":386,"state":"active+clean","last_fresh":"2024-09-10T19:28:34.963284+0000","last_change":"2024-09-10T19:28:23.457194+0000","last_active":"2024-09-10T19:28:34.963284+0000","last_peered":"2024-09-10T19:28:34.963284+0000","last_clean":"2024-09-10T19:28:34.963284+0000","last_became_active":"2024-09-10T19:27:53.008533+0000","last_became_peered":"2024-09-10T19:27:53.008533+0000","last_unstale":"2024-09-10T19:28:34.963284+0000","last_undegraded":"2024-09-10T19:28:34.963284+0000","last_fullsized":"2024-09-10T19:28:34.963284+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"384'56","last_scrub_stamp":"2024-09-10T19:28:23.457119+0000","last_deep_scrub":"384'56","last_deep_scrub_stamp":"2024-09-10T19:28:23.457119+0000","last_clean_scrub_stamp":"2024-09-10T19:28:23.457119+0000","objects_scrubbed":10,"log_size":56,"log_dups_size":0,"ondisk_log_size":56,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:29:32.825313+0000","scrub_duration":0.015902317999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":65,"num_read_kb":50,"num_write":38,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":40,"num_bytes_recovered":1432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":440,"reported_epoch":385,"state":"active+clean","last_fresh":"2024-09-10T19:28:30.079217+0000","last_change":"2024-09-10T19:27:53.005095+0000","last_active":"2024-09-10T19:28:30.079217+0000","last_peered":"2024-09-10T19:28:30.079217+0000","last_clean":"2024-09-10T19:28:30.079217+0000","last_became_active":"2024-09-10T19:27:53.004220+0000","last_became_peered":"2024-09-10T19:27:53.004220+0000","last_unstale":"2024-09-10T19:28:30.079217+0000","last_undegraded":"2024-09-10T19:28:30.079217+0000","last_fullsized":"2024-09-10T19:28:30.079217+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:27:22.758990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:27:22.758990+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:28:39.516295+0000","scrub_duration":0.0013854620000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":376,"reported_epoch":385,"state":"active+clean","last_fresh":"2024-09-10T19:28:30.063576+0000","last_change":"2024-09-10T19:28:26.358381+0000","last_active":"2024-09-10T19:28:30.063576+0000","last_peered":"2024-09-10T19:28:30.063576+0000","last_clean":"2024-09-10T19:28:30.063576+0000","last_became_active":"2024-09-10T19:24:24.892692+0000","last_became_peered":"2024-09-10T19:24:24.892692+0000","last_unstale":"2024-09-10T19:28:30.063576+0000","last_undegraded":"2024-09-10T19:28:30.063576+0000","last_fullsized":"2024-09-10T19:28:30.063576+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:28:26.358297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:28:26.358297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:29:44.782128+0000","scrub_duration":0.0046568440000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":380,"reported_epoch":385,"state":"active+clean","last_fresh":"2024-09-10T19:28:30.069782+0000","last_change":"2024-09-10T19:26:09.890646+0000","last_active":"2024-09-10T19:28:30.069782+0000","last_peered":"2024-09-10T19:28:30.069782+0000","last_clean":"2024-09-10T19:28:30.069782+0000","last_became_active":"2024-09-10T19:10:13.721744+0000","last_became_peered":"2024-09-10T19:10:13.721744+0000","last_unstale":"2024-09-10T19:28:30.069782+0000","last_undegraded":"2024-09-10T19:28:30.069782+0000","last_fullsized":"2024-09-10T19:28:30.069782+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:26:09.890570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:26:09.890570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.012478856,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.8","version":"386'4897","reported_seq":4930,"reported_epoch":386,"state":"active+clean","last_fresh":"2024-09-10T19:28:35.200674+0000","last_change":"2024-09-10T19:27:53.025153+0000","last_active":"2024-09-10T19:28:35.200674+0000","last_peered":"2024-09-10T19:28:35.200674+0000","last_clean":"2024-09-10T19:28:35.200674+0000","last_became_active":"2024-09-10T19:27:53.024817+0000","last_became_peered":"2024-09-10T19:27:53.024817+0000","last_unstale":"2024-09-10T19:28:35.200674+0000","last_undegraded":"2024-09-10T19:28:35.200674+0000","last_fullsized":"2024-09-10T19:28:35.200674+0000","mapping_epoch":374,"log_start":"376'700","ondisk_log_start":"376'700","created":371,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:27:45.679464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:27:45.679464+0000","last_clean_scrub_stamp":"2024-09-10T19:27:45.679464+0000","objects_scrubbed":0,"log_size":4197,"log_dups_size":0,"ondisk_log_size":4197,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:28:56.628117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":320929792,"num_objects":4897,"num_object_clones":0,"num_object_copies":14691,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4897,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4897,"num_write_kb":313408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"74'79","reported_seq":478,"reported_epoch":385,"state":"active+clean","last_fresh":"2024-09-10T19:28:30.074702+0000","last_change":"2024-09-10T19:26:37.555224+0000","last_active":"2024-09-10T19:28:30.074702+0000","last_peered":"2024-09-10T19:28:30.074702+0000","last_clean":"2024-09-10T19:28:30.074702+0000","last_became_active":"2024-09-10T19:24:23.895247+0000","last_became_peered":"2024-09-10T19:24:23.895247+0000","last_unstale":"2024-09-10T19:28:30.074702+0000","last_undegraded":"2024-09-10T19:28:30.074702+0000","last_fullsized":"2024-09-10T19:28:30.074702+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":4,"last_scrub":"74'79","last_scrub_stamp":"2024-09-10T19:26:37.555133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:08:00.699315+0000","last_clean_scrub_stamp":"2024-09-10T19:26:37.555133+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0060663699999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.7","version":"237'22","reported_seq":149,"reported_epoch":385,"state":"active+clean","last_fresh":"2024-09-10T19:28:30.068047+0000","last_change":"2024-09-10T19:28:05.521802+0000","last_active":"2024-09-10T19:28:30.068047+0000","last_peered":"2024-09-10T19:28:30.068047+0000","last_clean":"2024-09-10T19:28:30.068047+0000","last_became_active":"2024-09-10T19:24:24.894293+0000","last_became_peered":"2024-09-10T19:24:24.894293+0000","last_unstale":"2024-09-10T19:28:30.068047+0000","last_undegraded":"2024-09-10T19:28:30.068047+0000","last_fullsized":"2024-09-10T19:28:30.068047+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"237'22","last_scrub_stamp":"2024-09-10T19:28:05.521719+0000","last_deep_scrub":"82'18","last_deep_scrub_stamp":"2024-09-10T19:15:55.777361+0000","last_clean_scrub_stamp":"2024-09-10T19:28:05.521719+0000","objects_scrubbed":5,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:29:26.347313+0000","scrub_duration":0.0035457660000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":36,"num_read_kb":25,"num_write":20,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"159'6","reported_seq":441,"reported_epoch":386,"state":"active+clean","last_fresh":"2024-09-10T19:28:36.205184+0000","last_change":"2024-09-10T19:27:53.033132+0000","last_active":"2024-09-10T19:28:36.205184+0000","last_peered":"2024-09-10T19:28:36.205184+0000","last_clean":"2024-09-10T19:28:36.205184+0000","last_became_active":"2024-09-10T19:27:53.017381+0000","last_became_peered":"2024-09-10T19:27:53.017381+0000","last_unstale":"2024-09-10T19:28:36.205184+0000","last_undegraded":"2024-09-10T19:28:36.205184+0000","last_fullsized":"2024-09-10T19:28:36.205184+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"159'6","last_scrub_stamp":"2024-09-10T19:27:11.342570+0000","last_deep_scrub":"159'6","last_deep_scrub_stamp":"2024-09-10T19:16:32.649914+0000","last_clean_scrub_stamp":"2024-09-10T19:27:11.342570+0000","objects_scrubbed":1,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0038877080000000001,"objects_trimmed":0,"snaptrim_duration":0.00034454499999999997,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":278,"num_read_kb":273,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1556,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":325,"reported_epoch":385,"state":"active+clean","last_fresh":"2024-09-10T19:28:30.070308+0000","last_change":"2024-09-10T19:28:21.199391+0000","last_active":"2024-09-10T19:28:30.070308+0000","last_peered":"2024-09-10T19:28:30.070308+0000","last_clean":"2024-09-10T19:28:30.070308+0000","last_became_active":"2024-09-10T19:24:24.924859+0000","last_became_peered":"2024-09-10T19:24:24.924859+0000","last_unstale":"2024-09-10T19:28:30.070308+0000","last_undegraded":"2024-09-10T19:28:30.070308+0000","last_fullsized":"2024-09-10T19:28:30.070308+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:28:21.199065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:28:21.199065+0000","last_clean_scrub_stamp":"2024-09-10T19:28:21.199065+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:29:41.238427+0000","scrub_duration":0.0096488849999999994,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":294,"reported_epoch":385,"state":"active+clean","last_fresh":"2024-09-10T19:28:30.079648+0000","last_change":"2024-09-10T19:28:15.932318+0000","last_active":"2024-09-10T19:28:30.079648+0000","last_peered":"2024-09-10T19:28:30.079648+0000","last_clean":"2024-09-10T19:28:30.079648+0000","last_became_active":"2024-09-10T19:16:24.861481+0000","last_became_peered":"2024-09-10T19:16:24.861481+0000","last_unstale":"2024-09-10T19:28:30.079648+0000","last_undegraded":"2024-09-10T19:28:30.079648+0000","last_fullsized":"2024-09-10T19:28:30.079648+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:28:15.932184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:23:28.736502+0000","last_clean_scrub_stamp":"2024-09-10T19:28:15.932184+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:29:41.780270+0000","scrub_duration":0.0078893149999999992,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,4],"acting":[5,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":429,"reported_epoch":385,"state":"active+clean","last_fresh":"2024-09-10T19:28:30.075495+0000","last_change":"2024-09-10T19:28:01.784903+0000","last_active":"2024-09-10T19:28:30.075495+0000","last_peered":"2024-09-10T19:28:30.075495+0000","last_clean":"2024-09-10T19:28:30.075495+0000","last_became_active":"2024-09-10T19:27:53.011923+0000","last_became_peered":"2024-09-10T19:27:53.011923+0000","last_unstale":"2024-09-10T19:28:30.075495+0000","last_undegraded":"2024-09-10T19:28:30.075495+0000","last_fullsized":"2024-09-10T19:28:30.075495+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:28:01.784827+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:22:01.208272+0000","last_clean_scrub_stamp":"2024-09-10T19:28:01.784827+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:29:10.879150+0000","scrub_duration":0.0048401779999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.9","version":"386'4990","reported_seq":5023,"reported_epoch":386,"state":"active+clean","last_fresh":"2024-09-10T19:28:35.181555+0000","last_change":"2024-09-10T19:27:53.019298+0000","last_active":"2024-09-10T19:28:35.181555+0000","last_peered":"2024-09-10T19:28:35.181555+0000","last_clean":"2024-09-10T19:28:35.181555+0000","last_became_active":"2024-09-10T19:27:53.018973+0000","last_became_peered":"2024-09-10T19:27:53.018973+0000","last_unstale":"2024-09-10T19:28:35.181555+0000","last_undegraded":"2024-09-10T19:28:35.181555+0000","last_fullsized":"2024-09-10T19:28:35.181555+0000","mapping_epoch":374,"log_start":"376'800","ondisk_log_start":"376'800","created":371,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:27:45.679464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:27:45.679464+0000","last_clean_scrub_stamp":"2024-09-10T19:27:45.679464+0000","objects_scrubbed":0,"log_size":4190,"log_dups_size":0,"ondisk_log_size":4190,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:29:14.057963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327024640,"num_objects":4990,"num_object_clones":0,"num_object_copies":14970,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4990,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4990,"num_write_kb":319360,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"74'79","reported_seq":143,"reported_epoch":385,"state":"active+clean","last_fresh":"2024-09-10T19:28:30.069167+0000","last_change":"2024-09-10T19:28:19.495654+0000","last_active":"2024-09-10T19:28:30.069167+0000","last_peered":"2024-09-10T19:28:30.069167+0000","last_clean":"2024-09-10T19:28:30.069167+0000","last_became_active":"2024-09-10T19:27:53.016965+0000","last_became_peered":"2024-09-10T19:27:53.016965+0000","last_unstale":"2024-09-10T19:28:30.069167+0000","last_undegraded":"2024-09-10T19:28:30.069167+0000","last_fullsized":"2024-09-10T19:28:30.069167+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":3,"last_scrub":"74'79","last_scrub_stamp":"2024-09-10T19:28:19.495519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:08:00.699315+0000","last_clean_scrub_stamp":"2024-09-10T19:28:19.495519+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:29:38.604596+0000","scrub_duration":0.0042113080000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"237'30","reported_seq":456,"reported_epoch":385,"state":"active+clean","last_fresh":"2024-09-10T19:28:30.063997+0000","last_change":"2024-09-10T19:28:09.697081+0000","last_active":"2024-09-10T19:28:30.063997+0000","last_peered":"2024-09-10T19:28:30.063997+0000","last_clean":"2024-09-10T19:28:30.063997+0000","last_became_active":"2024-09-10T19:24:24.899711+0000","last_became_peered":"2024-09-10T19:24:24.899711+0000","last_unstale":"2024-09-10T19:28:30.063997+0000","last_undegraded":"2024-09-10T19:28:30.063997+0000","last_fullsized":"2024-09-10T19:28:30.063997+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"237'30","last_scrub_stamp":"2024-09-10T19:28:09.696933+0000","last_deep_scrub":"221'18","last_deep_scrub_stamp":"2024-09-10T19:19:29.606748+0000","last_clean_scrub_stamp":"2024-09-10T19:28:09.696933+0000","objects_scrubbed":6,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:29:27.118905+0000","scrub_duration":0.106209103,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":24,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"61'1","reported_seq":359,"reported_epoch":385,"state":"active+clean","last_fresh":"2024-09-10T19:28:30.063360+0000","last_change":"2024-09-10T19:27:07.480165+0000","last_active":"2024-09-10T19:28:30.063360+0000","last_peered":"2024-09-10T19:28:30.063360+0000","last_clean":"2024-09-10T19:28:30.063360+0000","last_became_active":"2024-09-10T19:24:23.897412+0000","last_became_peered":"2024-09-10T19:24:23.897412+0000","last_unstale":"2024-09-10T19:28:30.063360+0000","last_undegraded":"2024-09-10T19:28:30.063360+0000","last_fullsized":"2024-09-10T19:28:30.063360+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'1","last_scrub_stamp":"2024-09-10T19:27:07.480084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:27:07.480084+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001817936,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":444,"reported_epoch":385,"state":"active+clean","last_fresh":"2024-09-10T19:28:30.074262+0000","last_change":"2024-09-10T19:28:05.803811+0000","last_active":"2024-09-10T19:28:30.074262+0000","last_peered":"2024-09-10T19:28:30.074262+0000","last_clean":"2024-09-10T19:28:30.074262+0000","last_became_active":"2024-09-10T19:24:24.896201+0000","last_became_peered":"2024-09-10T19:24:24.896201+0000","last_unstale":"2024-09-10T19:28:30.074262+0000","last_undegraded":"2024-09-10T19:28:30.074262+0000","last_fullsized":"2024-09-10T19:28:30.074262+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:28:05.803723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:25:28.762237+0000","last_clean_scrub_stamp":"2024-09-10T19:28:05.803723+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:29:28.796074+0000","scrub_duration":0.0069439890000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":440,"reported_epoch":385,"state":"active+clean","last_fresh":"2024-09-10T19:28:30.080009+0000","last_change":"2024-09-10T19:27:27.008442+0000","last_active":"2024-09-10T19:28:30.080009+0000","last_peered":"2024-09-10T19:28:30.080009+0000","last_clean":"2024-09-10T19:28:30.080009+0000","last_became_active":"2024-09-10T19:24:24.914680+0000","last_became_peered":"2024-09-10T19:24:24.914680+0000","last_unstale":"2024-09-10T19:28:30.080009+0000","last_undegraded":"2024-09-10T19:28:30.080009+0000","last_fullsized":"2024-09-10T19:28:30.080009+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:27:27.008361+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:27:27.008361+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:28:56.381080+0000","scrub_duration":0.16799967800000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":123,"reported_epoch":385,"state":"active+clean","last_fresh":"2024-09-10T19:28:30.072381+0000","last_change":"2024-09-10T19:28:22.537849+0000","last_active":"2024-09-10T19:28:30.072381+0000","last_peered":"2024-09-10T19:28:30.072381+0000","last_clean":"2024-09-10T19:28:30.072381+0000","last_became_active":"2024-09-10T19:24:01.974039+0000","last_became_peered":"2024-09-10T19:24:01.974039+0000","last_unstale":"2024-09-10T19:28:30.072381+0000","last_undegraded":"2024-09-10T19:28:30.072381+0000","last_fullsized":"2024-09-10T19:28:30.072381+0000","mapping_epoch":292,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":293,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:28:22.537315+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:28:22.537315+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:29:25.004816+0000","scrub_duration":0.052250502999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.a","version":"386'4996","reported_seq":5029,"reported_epoch":386,"state":"active+clean","last_fresh":"2024-09-10T19:28:35.200447+0000","last_change":"2024-09-10T19:27:53.026334+0000","last_active":"2024-09-10T19:28:35.200447+0000","last_peered":"2024-09-10T19:28:35.200447+0000","last_clean":"2024-09-10T19:28:35.200447+0000","last_became_active":"2024-09-10T19:27:53.026021+0000","last_became_peered":"2024-09-10T19:27:53.026021+0000","last_unstale":"2024-09-10T19:28:35.200447+0000","last_undegraded":"2024-09-10T19:28:35.200447+0000","last_fullsized":"2024-09-10T19:28:35.200447+0000","mapping_epoch":374,"log_start":"376'800","ondisk_log_start":"376'800","created":371,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:27:45.679464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:27:45.679464+0000","last_clean_scrub_stamp":"2024-09-10T19:27:45.679464+0000","objects_scrubbed":0,"log_size":4196,"log_dups_size":0,"ondisk_log_size":4196,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:29:06.601000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327417856,"num_objects":4996,"num_object_clones":0,"num_object_copies":14988,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4996,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4996,"num_write_kb":319744,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"237'33","reported_seq":46,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-09-10T19:28:19.956590+0000","last_change":"2024-09-10T19:28:02.273264+0000","last_active":"2024-09-10T19:28:19.956590+0000","last_peered":"2024-09-10T19:28:19.956590+0000","last_clean":"2024-09-10T19:28:19.956590+0000","last_became_active":"2024-09-10T19:27:53.006290+0000","last_became_peered":"2024-09-10T19:27:53.006290+0000","last_unstale":"2024-09-10T19:28:19.956590+0000","last_undegraded":"2024-09-10T19:28:19.956590+0000","last_fullsized":"2024-09-10T19:28:19.956590+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"237'33","last_scrub_stamp":"2024-09-10T19:28:02.273137+0000","last_deep_scrub":"80'21","last_deep_scrub_stamp":"2024-09-10T19:11:39.836144+0000","last_clean_scrub_stamp":"2024-09-10T19:28:02.273137+0000","objects_scrubbed":7,"log_size":33,"log_dups_size":0,"ondisk_log_size":33,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:29:18.618152+0000","scrub_duration":0.0029425269999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":42,"num_read_kb":32,"num_write":25,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":154,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"63'2","reported_seq":441,"reported_epoch":385,"state":"active+clean","last_fresh":"2024-09-10T19:28:30.071183+0000","last_change":"2024-09-10T19:28:08.143382+0000","last_active":"2024-09-10T19:28:30.071183+0000","last_peered":"2024-09-10T19:28:30.071183+0000","last_clean":"2024-09-10T19:28:30.071183+0000","last_became_active":"2024-09-10T19:27:53.025802+0000","last_became_peered":"2024-09-10T19:27:53.025802+0000","last_unstale":"2024-09-10T19:28:30.071183+0000","last_undegraded":"2024-09-10T19:28:30.071183+0000","last_fullsized":"2024-09-10T19:28:30.071183+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'2","last_scrub_stamp":"2024-09-10T19:28:08.143309+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:28:08.143309+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:29:21.026886+0000","scrub_duration":0.021682264999999999,"objects_trimmed":0,"snaptrim_duration":0.00075522999999999996,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":416,"reported_epoch":385,"state":"active+clean","last_fresh":"2024-09-10T19:28:30.079372+0000","last_change":"2024-09-10T19:28:25.914938+0000","last_active":"2024-09-10T19:28:30.079372+0000","last_peered":"2024-09-10T19:28:30.079372+0000","last_clean":"2024-09-10T19:28:30.079372+0000","last_became_active":"2024-09-10T19:16:24.858662+0000","last_became_peered":"2024-09-10T19:16:24.858662+0000","last_unstale":"2024-09-10T19:28:30.079372+0000","last_undegraded":"2024-09-10T19:28:30.079372+0000","last_fullsized":"2024-09-10T19:28:30.079372+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:28:25.914847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:28:25.914847+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:29:44.608724+0000","scrub_duration":0.038511755000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"74'79","reported_seq":492,"reported_epoch":385,"state":"active+clean","last_fresh":"2024-09-10T19:28:30.074442+0000","last_change":"2024-09-10T19:27:57.727870+0000","last_active":"2024-09-10T19:28:30.074442+0000","last_peered":"2024-09-10T19:28:30.074442+0000","last_clean":"2024-09-10T19:28:30.074442+0000","last_became_active":"2024-09-10T19:27:53.009367+0000","last_became_peered":"2024-09-10T19:27:53.009367+0000","last_unstale":"2024-09-10T19:28:30.074442+0000","last_undegraded":"2024-09-10T19:28:30.074442+0000","last_fullsized":"2024-09-10T19:28:30.074442+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'79","last_scrub_stamp":"2024-09-10T19:27:57.727776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:08:00.699315+0000","last_clean_scrub_stamp":"2024-09-10T19:27:57.727776+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:29:01.958737+0000","scrub_duration":0.0057466210000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":419,"reported_epoch":385,"state":"active+clean","last_fresh":"2024-09-10T19:28:30.078393+0000","last_change":"2024-09-10T19:26:31.509806+0000","last_active":"2024-09-10T19:28:30.078393+0000","last_peered":"2024-09-10T19:28:30.078393+0000","last_clean":"2024-09-10T19:28:30.078393+0000","last_became_active":"2024-09-10T19:24:23.903868+0000","last_became_peered":"2024-09-10T19:24:23.903868+0000","last_unstale":"2024-09-10T19:28:30.078393+0000","last_undegraded":"2024-09-10T19:28:30.078393+0000","last_fullsized":"2024-09-10T19:28:30.078393+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:26:31.509498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:26:31.509498+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.029737573999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,0],"acting":[7,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":400,"reported_epoch":385,"state":"active+clean","last_fresh":"2024-09-10T19:28:30.080253+0000","last_change":"2024-09-10T19:28:29.899683+0000","last_active":"2024-09-10T19:28:30.080253+0000","last_peered":"2024-09-10T19:28:30.080253+0000","last_clean":"2024-09-10T19:28:30.080253+0000","last_became_active":"2024-09-10T19:10:13.720647+0000","last_became_peered":"2024-09-10T19:10:13.720647+0000","last_unstale":"2024-09-10T19:28:30.080253+0000","last_undegraded":"2024-09-10T19:28:30.080253+0000","last_fullsized":"2024-09-10T19:28:30.080253+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:28:29.899579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:28:29.899579+0000","last_clean_scrub_stamp":"2024-09-10T19:28:29.899579+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:29:40.248636+0000","scrub_duration":0.0094756419999999994,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.5","version":"386'5050","reported_seq":5082,"reported_epoch":386,"state":"active+clean","last_fresh":"2024-09-10T19:28:35.471537+0000","last_change":"2024-09-10T19:27:53.011775+0000","last_active":"2024-09-10T19:28:35.471537+0000","last_peered":"2024-09-10T19:28:35.471537+0000","last_clean":"2024-09-10T19:28:35.471537+0000","last_became_active":"2024-09-10T19:27:53.010838+0000","last_became_peered":"2024-09-10T19:27:53.010838+0000","last_unstale":"2024-09-10T19:28:35.471537+0000","last_undegraded":"2024-09-10T19:28:35.471537+0000","last_fullsized":"2024-09-10T19:28:35.471537+0000","mapping_epoch":374,"log_start":"376'800","ondisk_log_start":"376'800","created":371,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:27:45.679464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:27:45.679464+0000","last_clean_scrub_stamp":"2024-09-10T19:27:45.679464+0000","objects_scrubbed":0,"log_size":4250,"log_dups_size":0,"ondisk_log_size":4250,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:28:48.018920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330956800,"num_objects":5050,"num_object_clones":0,"num_object_copies":15150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5050,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5050,"num_write_kb":323200,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"237'35","reported_seq":50,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-09-10T19:28:20.571325+0000","last_change":"2024-09-10T19:28:20.571263+0000","last_active":"2024-09-10T19:28:20.571325+0000","last_peered":"2024-09-10T19:28:20.571325+0000","last_clean":"2024-09-10T19:28:20.571325+0000","last_became_active":"2024-09-10T19:27:53.014163+0000","last_became_peered":"2024-09-10T19:27:53.014163+0000","last_unstale":"2024-09-10T19:28:20.571325+0000","last_undegraded":"2024-09-10T19:28:20.571325+0000","last_fullsized":"2024-09-10T19:28:20.571325+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"237'35","last_scrub_stamp":"2024-09-10T19:28:20.571183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:28:20.571183+0000","objects_scrubbed":9,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:29:42.214809+0000","scrub_duration":0.0061966299999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":56,"num_read_kb":45,"num_write":36,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":308,"reported_epoch":385,"state":"active+clean","last_fresh":"2024-09-10T19:28:30.079565+0000","last_change":"2024-09-10T19:27:57.985665+0000","last_active":"2024-09-10T19:28:30.079565+0000","last_peered":"2024-09-10T19:28:30.079565+0000","last_clean":"2024-09-10T19:28:30.079565+0000","last_became_active":"2024-09-10T19:24:24.913709+0000","last_became_peered":"2024-09-10T19:24:24.913709+0000","last_unstale":"2024-09-10T19:28:30.079565+0000","last_undegraded":"2024-09-10T19:28:30.079565+0000","last_fullsized":"2024-09-10T19:28:30.079565+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:27:57.985573+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:27:57.985573+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:29:24.408046+0000","scrub_duration":0.002297257,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,1],"acting":[5,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":236,"reported_epoch":385,"state":"active+clean","last_fresh":"2024-09-10T19:28:30.062479+0000","last_change":"2024-09-10T19:28:20.429105+0000","last_active":"2024-09-10T19:28:30.062479+0000","last_peered":"2024-09-10T19:28:30.062479+0000","last_clean":"2024-09-10T19:28:30.062479+0000","last_became_active":"2024-09-10T19:24:23.901415+0000","last_became_peered":"2024-09-10T19:24:23.901415+0000","last_unstale":"2024-09-10T19:28:30.062479+0000","last_undegraded":"2024-09-10T19:28:30.062479+0000","last_fullsized":"2024-09-10T19:28:30.062479+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:28:20.428968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:28:20.428968+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:29:44.162343+0000","scrub_duration":0.0052837259999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":440,"reported_epoch":385,"state":"active+clean","last_fresh":"2024-09-10T19:28:30.074359+0000","last_change":"2024-09-10T19:28:02.832572+0000","last_active":"2024-09-10T19:28:30.074359+0000","last_peered":"2024-09-10T19:28:30.074359+0000","last_clean":"2024-09-10T19:28:30.074359+0000","last_became_active":"2024-09-10T19:27:53.008850+0000","last_became_peered":"2024-09-10T19:27:53.008850+0000","last_unstale":"2024-09-10T19:28:30.074359+0000","last_undegraded":"2024-09-10T19:28:30.074359+0000","last_fullsized":"2024-09-10T19:28:30.074359+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:28:02.832476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:21:03.910074+0000","last_clean_scrub_stamp":"2024-09-10T19:28:02.832476+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:29:29.472641+0000","scrub_duration":0.013473214000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,6],"acting":[7,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.4","version":"386'4910","reported_seq":4943,"reported_epoch":386,"state":"active+clean","last_fresh":"2024-09-10T19:28:35.467025+0000","last_change":"2024-09-10T19:27:53.012790+0000","last_active":"2024-09-10T19:28:35.467025+0000","last_peered":"2024-09-10T19:28:35.467025+0000","last_clean":"2024-09-10T19:28:35.467025+0000","last_became_active":"2024-09-10T19:27:53.012510+0000","last_became_peered":"2024-09-10T19:27:53.012510+0000","last_unstale":"2024-09-10T19:28:35.467025+0000","last_undegraded":"2024-09-10T19:28:35.467025+0000","last_fullsized":"2024-09-10T19:28:35.467025+0000","mapping_epoch":374,"log_start":"376'600","ondisk_log_start":"376'600","created":371,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:27:45.679464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:27:45.679464+0000","last_clean_scrub_stamp":"2024-09-10T19:27:45.679464+0000","objects_scrubbed":0,"log_size":4310,"log_dups_size":0,"ondisk_log_size":4310,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:28:48.631961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":321781760,"num_objects":4910,"num_object_clones":0,"num_object_copies":14730,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4910,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4910,"num_write_kb":314240,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"237'15","reported_seq":440,"reported_epoch":385,"state":"active+clean","last_fresh":"2024-09-10T19:28:30.062846+0000","last_change":"2024-09-10T19:28:01.368969+0000","last_active":"2024-09-10T19:28:30.062846+0000","last_peered":"2024-09-10T19:28:30.062846+0000","last_clean":"2024-09-10T19:28:30.062846+0000","last_became_active":"2024-09-10T19:24:44.892294+0000","last_became_peered":"2024-09-10T19:24:44.892294+0000","last_unstale":"2024-09-10T19:28:30.062846+0000","last_undegraded":"2024-09-10T19:28:30.062846+0000","last_fullsized":"2024-09-10T19:28:30.062846+0000","mapping_epoch":314,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":315,"parent":"0.0","parent_split_bits":0,"last_scrub":"237'15","last_scrub_stamp":"2024-09-10T19:28:01.368870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:28:01.368870+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:29:27.255179+0000","scrub_duration":0.003607327,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,4],"acting":[0,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":113,"reported_epoch":385,"state":"active+clean","last_fresh":"2024-09-10T19:28:30.070503+0000","last_change":"2024-09-10T19:26:30.411444+0000","last_active":"2024-09-10T19:28:30.070503+0000","last_peered":"2024-09-10T19:28:30.070503+0000","last_clean":"2024-09-10T19:28:30.070503+0000","last_became_active":"2024-09-10T19:24:01.980348+0000","last_became_peered":"2024-09-10T19:24:01.980348+0000","last_unstale":"2024-09-10T19:28:30.070503+0000","last_undegraded":"2024-09-10T19:28:30.070503+0000","last_fullsized":"2024-09-10T19:28:30.070503+0000","mapping_epoch":292,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":293,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:26:30.411368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:11:44.254513+0000","last_clean_scrub_stamp":"2024-09-10T19:26:30.411368+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0027079880000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":304,"reported_epoch":385,"state":"active+clean","last_fresh":"2024-09-10T19:28:30.067004+0000","last_change":"2024-09-10T19:27:56.924199+0000","last_active":"2024-09-10T19:28:30.067004+0000","last_peered":"2024-09-10T19:28:30.067004+0000","last_clean":"2024-09-10T19:28:30.067004+0000","last_became_active":"2024-09-10T19:24:23.891161+0000","last_became_peered":"2024-09-10T19:24:23.891161+0000","last_unstale":"2024-09-10T19:28:30.067004+0000","last_undegraded":"2024-09-10T19:28:30.067004+0000","last_fullsized":"2024-09-10T19:28:30.067004+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:27:56.924107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:27:56.924107+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:29:03.247483+0000","scrub_duration":0.0047147070000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":354,"reported_epoch":385,"state":"active+clean","last_fresh":"2024-09-10T19:28:30.063429+0000","last_change":"2024-09-10T19:28:25.385708+0000","last_active":"2024-09-10T19:28:30.063429+0000","last_peered":"2024-09-10T19:28:30.063429+0000","last_clean":"2024-09-10T19:28:30.063429+0000","last_became_active":"2024-09-10T19:24:23.905112+0000","last_became_peered":"2024-09-10T19:24:23.905112+0000","last_unstale":"2024-09-10T19:28:30.063429+0000","last_undegraded":"2024-09-10T19:28:30.063429+0000","last_fullsized":"2024-09-10T19:28:30.063429+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:28:25.385655+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:28:25.385655+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:29:54.020875+0000","scrub_duration":0.0048080329999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.7","version":"386'5016","reported_seq":5039,"reported_epoch":386,"state":"active+clean","last_fresh":"2024-09-10T19:28:35.535429+0000","last_change":"2024-09-10T19:27:46.707718+0000","last_active":"2024-09-10T19:28:35.535429+0000","last_peered":"2024-09-10T19:28:35.535429+0000","last_clean":"2024-09-10T19:28:35.535429+0000","last_became_active":"2024-09-10T19:27:46.707393+0000","last_became_peered":"2024-09-10T19:27:46.707393+0000","last_unstale":"2024-09-10T19:28:35.535429+0000","last_undegraded":"2024-09-10T19:28:35.535429+0000","last_fullsized":"2024-09-10T19:28:35.535429+0000","mapping_epoch":371,"log_start":"376'400","ondisk_log_start":"376'400","created":371,"last_epoch_clean":372,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:27:45.679464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:27:45.679464+0000","last_clean_scrub_stamp":"2024-09-10T19:27:45.679464+0000","objects_scrubbed":0,"log_size":4616,"log_dups_size":0,"ondisk_log_size":4616,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:29:04.469943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":328728576,"num_objects":5016,"num_object_clones":0,"num_object_copies":15048,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5016,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5016,"num_write_kb":321024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,4],"acting":[5,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.8","version":"237'34","reported_seq":496,"reported_epoch":385,"state":"active+clean","last_fresh":"2024-09-10T19:28:30.078881+0000","last_change":"2024-09-10T19:28:12.970031+0000","last_active":"2024-09-10T19:28:30.078881+0000","last_peered":"2024-09-10T19:28:30.078881+0000","last_clean":"2024-09-10T19:28:30.078881+0000","last_became_active":"2024-09-10T19:27:53.004858+0000","last_became_peered":"2024-09-10T19:27:53.004858+0000","last_unstale":"2024-09-10T19:28:30.078881+0000","last_undegraded":"2024-09-10T19:28:30.078881+0000","last_fullsized":"2024-09-10T19:28:30.078881+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"237'34","last_scrub_stamp":"2024-09-10T19:28:12.969975+0000","last_deep_scrub":"220'22","last_deep_scrub_stamp":"2024-09-10T19:19:24.597866+0000","last_clean_scrub_stamp":"2024-09-10T19:28:12.969975+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:29:26.338740+0000","scrub_duration":0.0023662679999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":304,"reported_epoch":385,"state":"active+clean","last_fresh":"2024-09-10T19:28:30.069978+0000","last_change":"2024-09-10T19:28:15.131232+0000","last_active":"2024-09-10T19:28:30.069978+0000","last_peered":"2024-09-10T19:28:30.069978+0000","last_clean":"2024-09-10T19:28:30.069978+0000","last_became_active":"2024-09-10T19:27:53.016898+0000","last_became_peered":"2024-09-10T19:27:53.016898+0000","last_unstale":"2024-09-10T19:28:30.069978+0000","last_undegraded":"2024-09-10T19:28:30.069978+0000","last_fullsized":"2024-09-10T19:28:30.069978+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:28:15.131140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:28:15.131140+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:29:29.946980+0000","scrub_duration":0.0062187090000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.9","version":"376'11","reported_seq":670,"reported_epoch":385,"state":"active+clean","last_fresh":"2024-09-10T19:28:30.947955+0000","last_change":"2024-09-10T19:28:06.800963+0000","last_active":"2024-09-10T19:28:30.947955+0000","last_peered":"2024-09-10T19:28:30.947955+0000","last_clean":"2024-09-10T19:28:30.947955+0000","last_became_active":"2024-09-10T19:27:53.009810+0000","last_became_peered":"2024-09-10T19:27:53.009810+0000","last_unstale":"2024-09-10T19:28:30.947955+0000","last_undegraded":"2024-09-10T19:28:30.947955+0000","last_fullsized":"2024-09-10T19:28:30.947955+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"376'11","last_scrub_stamp":"2024-09-10T19:28:06.800870+0000","last_deep_scrub":"80'9","last_deep_scrub_stamp":"2024-09-10T19:25:29.833271+0000","last_clean_scrub_stamp":"2024-09-10T19:28:06.800870+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:29:14.710226+0000","scrub_duration":0.014022766000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":417,"reported_epoch":385,"state":"active+clean","last_fresh":"2024-09-10T19:28:30.079531+0000","last_change":"2024-09-10T19:27:53.016793+0000","last_active":"2024-09-10T19:28:30.079531+0000","last_peered":"2024-09-10T19:28:30.079531+0000","last_clean":"2024-09-10T19:28:30.079531+0000","last_became_active":"2024-09-10T19:27:53.015957+0000","last_became_peered":"2024-09-10T19:27:53.015957+0000","last_unstale":"2024-09-10T19:28:30.079531+0000","last_undegraded":"2024-09-10T19:28:30.079531+0000","last_fullsized":"2024-09-10T19:28:30.079531+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:27:24.791128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:27:24.791128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:28:49.360924+0000","scrub_duration":0.0043852980000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.6","version":"386'5104","reported_seq":5127,"reported_epoch":386,"state":"active+clean","last_fresh":"2024-09-10T19:28:36.957537+0000","last_change":"2024-09-10T19:27:46.705126+0000","last_active":"2024-09-10T19:28:36.957537+0000","last_peered":"2024-09-10T19:28:36.957537+0000","last_clean":"2024-09-10T19:28:36.957537+0000","last_became_active":"2024-09-10T19:27:46.704823+0000","last_became_peered":"2024-09-10T19:27:46.704823+0000","last_unstale":"2024-09-10T19:28:36.957537+0000","last_undegraded":"2024-09-10T19:28:36.957537+0000","last_fullsized":"2024-09-10T19:28:36.957537+0000","mapping_epoch":371,"log_start":"376'900","ondisk_log_start":"376'900","created":371,"last_epoch_clean":372,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:27:45.679464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:27:45.679464+0000","last_clean_scrub_stamp":"2024-09-10T19:27:45.679464+0000","objects_scrubbed":0,"log_size":4204,"log_dups_size":0,"ondisk_log_size":4204,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:28:47.064715+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":334495744,"num_objects":5104,"num_object_clones":0,"num_object_copies":15312,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5104,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5104,"num_write_kb":326656,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.9","version":"237'18","reported_seq":479,"reported_epoch":385,"state":"active+clean","last_fresh":"2024-09-10T19:28:30.067456+0000","last_change":"2024-09-10T19:26:11.809995+0000","last_active":"2024-09-10T19:28:30.067456+0000","last_peered":"2024-09-10T19:28:30.067456+0000","last_clean":"2024-09-10T19:28:30.067456+0000","last_became_active":"2024-09-10T19:24:01.974878+0000","last_became_peered":"2024-09-10T19:24:01.974878+0000","last_unstale":"2024-09-10T19:28:30.067456+0000","last_undegraded":"2024-09-10T19:28:30.067456+0000","last_fullsized":"2024-09-10T19:28:30.067456+0000","mapping_epoch":292,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":293,"parent":"0.0","parent_split_bits":0,"last_scrub":"237'18","last_scrub_stamp":"2024-09-10T19:26:11.809915+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:26:11.809915+0000","objects_scrubbed":5,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.014721300999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.f","version":"59'2","reported_seq":428,"reported_epoch":385,"state":"active+clean","last_fresh":"2024-09-10T19:28:30.068019+0000","last_change":"2024-09-10T19:27:58.903675+0000","last_active":"2024-09-10T19:28:30.068019+0000","last_peered":"2024-09-10T19:28:30.068019+0000","last_clean":"2024-09-10T19:28:30.068019+0000","last_became_active":"2024-09-10T19:24:23.893796+0000","last_became_peered":"2024-09-10T19:24:23.893796+0000","last_unstale":"2024-09-10T19:28:30.068019+0000","last_undegraded":"2024-09-10T19:28:30.068019+0000","last_fullsized":"2024-09-10T19:28:30.068019+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'2","last_scrub_stamp":"2024-09-10T19:27:58.903580+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:27:58.903580+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:29:22.292017+0000","scrub_duration":0.0054793380000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":14,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":338,"reported_epoch":385,"state":"active+clean","last_fresh":"2024-09-10T19:28:30.062766+0000","last_change":"2024-09-10T19:27:32.372295+0000","last_active":"2024-09-10T19:28:30.062766+0000","last_peered":"2024-09-10T19:28:30.062766+0000","last_clean":"2024-09-10T19:28:30.062766+0000","last_became_active":"2024-09-10T19:24:24.914252+0000","last_became_peered":"2024-09-10T19:24:24.914252+0000","last_unstale":"2024-09-10T19:28:30.062766+0000","last_undegraded":"2024-09-10T19:28:30.062766+0000","last_fullsized":"2024-09-10T19:28:30.062766+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:27:32.372200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:11:46.587846+0000","last_clean_scrub_stamp":"2024-09-10T19:27:32.372200+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:28:56.603539+0000","scrub_duration":0.002128497,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,5],"acting":[0,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":417,"reported_epoch":385,"state":"active+clean","last_fresh":"2024-09-10T19:28:30.067867+0000","last_change":"2024-09-10T19:28:03.066155+0000","last_active":"2024-09-10T19:28:30.067867+0000","last_peered":"2024-09-10T19:28:30.067867+0000","last_clean":"2024-09-10T19:28:30.067867+0000","last_became_active":"2024-09-10T19:24:01.971131+0000","last_became_peered":"2024-09-10T19:24:01.971131+0000","last_unstale":"2024-09-10T19:28:30.067867+0000","last_undegraded":"2024-09-10T19:28:30.067867+0000","last_fullsized":"2024-09-10T19:28:30.067867+0000","mapping_epoch":292,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":293,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:28:03.066075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:28:03.066075+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:29:18.971018+0000","scrub_duration":0.0025642619999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.16","version":"237'15","reported_seq":394,"reported_epoch":385,"state":"active+clean","last_fresh":"2024-09-10T19:28:30.063850+0000","last_change":"2024-09-10T19:27:23.434149+0000","last_active":"2024-09-10T19:28:30.063850+0000","last_peered":"2024-09-10T19:28:30.063850+0000","last_clean":"2024-09-10T19:28:30.063850+0000","last_became_active":"2024-09-10T19:24:23.901610+0000","last_became_peered":"2024-09-10T19:24:23.901610+0000","last_unstale":"2024-09-10T19:28:30.063850+0000","last_undegraded":"2024-09-10T19:28:30.063850+0000","last_fullsized":"2024-09-10T19:28:30.063850+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"237'15","last_scrub_stamp":"2024-09-10T19:27:23.434086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:27:23.434086+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:28:44.995269+0000","scrub_duration":0.002067663,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":137,"reported_epoch":385,"state":"active+clean","last_fresh":"2024-09-10T19:28:30.068539+0000","last_change":"2024-09-10T19:27:25.450030+0000","last_active":"2024-09-10T19:28:30.068539+0000","last_peered":"2024-09-10T19:28:30.068539+0000","last_clean":"2024-09-10T19:28:30.068539+0000","last_became_active":"2024-09-10T19:24:24.894815+0000","last_became_peered":"2024-09-10T19:24:24.894815+0000","last_unstale":"2024-09-10T19:28:30.068539+0000","last_undegraded":"2024-09-10T19:28:30.068539+0000","last_fullsized":"2024-09-10T19:28:30.068539+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:27:25.449801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:11:41.458523+0000","last_clean_scrub_stamp":"2024-09-10T19:27:25.449801+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0094907180000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,4],"acting":[1,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":416,"reported_epoch":385,"state":"active+clean","last_fresh":"2024-09-10T19:28:30.068399+0000","last_change":"2024-09-10T19:26:24.891697+0000","last_active":"2024-09-10T19:28:30.068399+0000","last_peered":"2024-09-10T19:28:30.068399+0000","last_clean":"2024-09-10T19:28:30.068399+0000","last_became_active":"2024-09-10T19:24:01.970325+0000","last_became_peered":"2024-09-10T19:24:01.970325+0000","last_unstale":"2024-09-10T19:28:30.068399+0000","last_undegraded":"2024-09-10T19:28:30.068399+0000","last_fullsized":"2024-09-10T19:28:30.068399+0000","mapping_epoch":292,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":293,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:26:24.891605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:24:09.367654+0000","last_clean_scrub_stamp":"2024-09-10T19:26:24.891605+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0032178340000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"66'1","reported_seq":304,"reported_epoch":385,"state":"active+clean","last_fresh":"2024-09-10T19:28:30.069235+0000","last_change":"2024-09-10T19:27:29.948091+0000","last_active":"2024-09-10T19:28:30.069235+0000","last_peered":"2024-09-10T19:28:30.069235+0000","last_clean":"2024-09-10T19:28:30.069235+0000","last_became_active":"2024-09-10T19:24:23.897858+0000","last_became_peered":"2024-09-10T19:24:23.897858+0000","last_unstale":"2024-09-10T19:28:30.069235+0000","last_undegraded":"2024-09-10T19:28:30.069235+0000","last_fullsized":"2024-09-10T19:28:30.069235+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'1","last_scrub_stamp":"2024-09-10T19:27:29.947993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:27:29.947993+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:28:37.197250+0000","scrub_duration":0.00156365,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":13,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":30,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-09-10T19:28:19.957707+0000","last_change":"2024-09-10T19:28:04.300167+0000","last_active":"2024-09-10T19:28:19.957707+0000","last_peered":"2024-09-10T19:28:19.957707+0000","last_clean":"2024-09-10T19:28:19.957707+0000","last_became_active":"2024-09-10T19:27:53.007487+0000","last_became_peered":"2024-09-10T19:27:53.007487+0000","last_unstale":"2024-09-10T19:28:19.957707+0000","last_undegraded":"2024-09-10T19:28:19.957707+0000","last_fullsized":"2024-09-10T19:28:19.957707+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:28:04.300089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:28:04.300089+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:29:11.662211+0000","scrub_duration":0.0069551550000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"237'12","reported_seq":463,"reported_epoch":385,"state":"active+clean","last_fresh":"2024-09-10T19:28:30.068767+0000","last_change":"2024-09-10T19:27:14.261846+0000","last_active":"2024-09-10T19:28:30.068767+0000","last_peered":"2024-09-10T19:28:30.068767+0000","last_clean":"2024-09-10T19:28:30.068767+0000","last_became_active":"2024-09-10T19:24:01.971492+0000","last_became_peered":"2024-09-10T19:24:01.971492+0000","last_unstale":"2024-09-10T19:28:30.068767+0000","last_undegraded":"2024-09-10T19:28:30.068767+0000","last_fullsized":"2024-09-10T19:28:30.068767+0000","mapping_epoch":292,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":293,"parent":"0.0","parent_split_bits":0,"last_scrub":"237'12","last_scrub_stamp":"2024-09-10T19:27:14.261789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:27:14.261789+0000","objects_scrubbed":3,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:28:43.284384+0000","scrub_duration":0.001545739,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":18,"num_read_kb":15,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":419,"reported_epoch":385,"state":"active+clean","last_fresh":"2024-09-10T19:28:30.079686+0000","last_change":"2024-09-10T19:28:17.872220+0000","last_active":"2024-09-10T19:28:30.079686+0000","last_peered":"2024-09-10T19:28:30.079686+0000","last_clean":"2024-09-10T19:28:30.079686+0000","last_became_active":"2024-09-10T19:24:01.978215+0000","last_became_peered":"2024-09-10T19:24:01.978215+0000","last_unstale":"2024-09-10T19:28:30.079686+0000","last_undegraded":"2024-09-10T19:28:30.079686+0000","last_fullsized":"2024-09-10T19:28:30.079686+0000","mapping_epoch":292,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":293,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:28:17.872134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:11:23.264793+0000","last_clean_scrub_stamp":"2024-09-10T19:28:17.872134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:29:40.354294+0000","scrub_duration":0.0039908060000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":420,"reported_epoch":385,"state":"active+clean","last_fresh":"2024-09-10T19:28:30.075413+0000","last_change":"2024-09-10T19:27:53.011473+0000","last_active":"2024-09-10T19:28:30.075413+0000","last_peered":"2024-09-10T19:28:30.075413+0000","last_clean":"2024-09-10T19:28:30.075413+0000","last_became_active":"2024-09-10T19:27:53.010120+0000","last_became_peered":"2024-09-10T19:27:53.010120+0000","last_unstale":"2024-09-10T19:28:30.075413+0000","last_undegraded":"2024-09-10T19:28:30.075413+0000","last_fullsized":"2024-09-10T19:28:30.075413+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:27:23.645891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:27:23.645891+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0046670679999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"237'25","reported_seq":502,"reported_epoch":385,"state":"active+clean","last_fresh":"2024-09-10T19:28:30.067668+0000","last_change":"2024-09-10T19:28:02.094718+0000","last_active":"2024-09-10T19:28:30.067668+0000","last_peered":"2024-09-10T19:28:30.067668+0000","last_clean":"2024-09-10T19:28:30.067668+0000","last_became_active":"2024-09-10T19:24:01.971173+0000","last_became_peered":"2024-09-10T19:24:01.971173+0000","last_unstale":"2024-09-10T19:28:30.067668+0000","last_undegraded":"2024-09-10T19:28:30.067668+0000","last_fullsized":"2024-09-10T19:28:30.067668+0000","mapping_epoch":292,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":293,"parent":"0.0","parent_split_bits":0,"last_scrub":"237'25","last_scrub_stamp":"2024-09-10T19:28:02.094638+0000","last_deep_scrub":"80'15","last_deep_scrub_stamp":"2024-09-10T19:11:41.719310+0000","last_clean_scrub_stamp":"2024-09-10T19:28:02.094638+0000","objects_scrubbed":5,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:29:28.959997+0000","scrub_duration":0.0049922860000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":25,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":396,"reported_epoch":385,"state":"active+clean","last_fresh":"2024-09-10T19:28:30.079286+0000","last_change":"2024-09-10T19:28:10.987417+0000","last_active":"2024-09-10T19:28:30.079286+0000","last_peered":"2024-09-10T19:28:30.079286+0000","last_clean":"2024-09-10T19:28:30.079286+0000","last_became_active":"2024-09-10T19:10:07.685870+0000","last_became_peered":"2024-09-10T19:10:07.685870+0000","last_unstale":"2024-09-10T19:28:30.079286+0000","last_undegraded":"2024-09-10T19:28:30.079286+0000","last_fullsized":"2024-09-10T19:28:30.079286+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:28:10.987330+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:28:10.987330+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:29:24.636516+0000","scrub_duration":0.007045585,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"372'16","reported_seq":686,"reported_epoch":385,"state":"active+clean","last_fresh":"2024-09-10T19:28:30.947393+0000","last_change":"2024-09-10T19:27:46.707870+0000","last_active":"2024-09-10T19:28:30.947393+0000","last_peered":"2024-09-10T19:28:30.947393+0000","last_clean":"2024-09-10T19:28:30.947393+0000","last_became_active":"2024-09-10T19:27:46.703433+0000","last_became_peered":"2024-09-10T19:27:46.703433+0000","last_unstale":"2024-09-10T19:28:30.947393+0000","last_undegraded":"2024-09-10T19:28:30.947393+0000","last_fullsized":"2024-09-10T19:28:30.947393+0000","mapping_epoch":371,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":372,"parent":"0.0","parent_split_bits":0,"last_scrub":"303'15","last_scrub_stamp":"2024-09-10T19:27:25.799459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:27:25.799459+0000","objects_scrubbed":1,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:28:44.019099+0000","scrub_duration":0.0047920439999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,0],"acting":[5,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":350,"reported_epoch":385,"state":"active+clean","last_fresh":"2024-09-10T19:28:30.063944+0000","last_change":"2024-09-10T19:27:57.378094+0000","last_active":"2024-09-10T19:28:30.063944+0000","last_peered":"2024-09-10T19:28:30.063944+0000","last_clean":"2024-09-10T19:28:30.063944+0000","last_became_active":"2024-09-10T19:24:23.900513+0000","last_became_peered":"2024-09-10T19:24:23.900513+0000","last_unstale":"2024-09-10T19:28:30.063944+0000","last_undegraded":"2024-09-10T19:28:30.063944+0000","last_fullsized":"2024-09-10T19:28:30.063944+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:27:57.378012+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:25:19.396159+0000","last_clean_scrub_stamp":"2024-09-10T19:27:57.378012+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:29:11.974152+0000","scrub_duration":0.0029670780000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"237'15","reported_seq":459,"reported_epoch":385,"state":"active+clean","last_fresh":"2024-09-10T19:28:30.080074+0000","last_change":"2024-09-10T19:28:00.983992+0000","last_active":"2024-09-10T19:28:30.080074+0000","last_peered":"2024-09-10T19:28:30.080074+0000","last_clean":"2024-09-10T19:28:30.080074+0000","last_became_active":"2024-09-10T19:16:24.859740+0000","last_became_peered":"2024-09-10T19:16:24.859740+0000","last_unstale":"2024-09-10T19:28:30.080074+0000","last_undegraded":"2024-09-10T19:28:30.080074+0000","last_fullsized":"2024-09-10T19:28:30.080074+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"237'15","last_scrub_stamp":"2024-09-10T19:28:00.983937+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:28:00.983937+0000","objects_scrubbed":4,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:29:08.060287+0000","scrub_duration":0.005075876,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":20,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":353,"reported_epoch":385,"state":"active+clean","last_fresh":"2024-09-10T19:28:30.062541+0000","last_change":"2024-09-10T19:27:13.478575+0000","last_active":"2024-09-10T19:28:30.062541+0000","last_peered":"2024-09-10T19:28:30.062541+0000","last_clean":"2024-09-10T19:28:30.062541+0000","last_became_active":"2024-09-10T19:24:23.895163+0000","last_became_peered":"2024-09-10T19:24:23.895163+0000","last_unstale":"2024-09-10T19:28:30.062541+0000","last_undegraded":"2024-09-10T19:28:30.062541+0000","last_fullsized":"2024-09-10T19:28:30.062541+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:27:13.478519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:27:13.478519+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:28:38.794638+0000","scrub_duration":0.0016275479999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"297'11","reported_seq":650,"reported_epoch":385,"state":"active+clean","last_fresh":"2024-09-10T19:28:30.948145+0000","last_change":"2024-09-10T19:26:16.945209+0000","last_active":"2024-09-10T19:28:30.948145+0000","last_peered":"2024-09-10T19:28:30.948145+0000","last_clean":"2024-09-10T19:28:30.948145+0000","last_became_active":"2024-09-10T19:24:01.974816+0000","last_became_peered":"2024-09-10T19:24:01.974816+0000","last_unstale":"2024-09-10T19:28:30.948145+0000","last_undegraded":"2024-09-10T19:28:30.948145+0000","last_fullsized":"2024-09-10T19:28:30.948145+0000","mapping_epoch":292,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":293,"parent":"0.0","parent_split_bits":0,"last_scrub":"297'11","last_scrub_stamp":"2024-09-10T19:26:16.945115+0000","last_deep_scrub":"297'11","last_deep_scrub_stamp":"2024-09-10T19:24:13.318130+0000","last_clean_scrub_stamp":"2024-09-10T19:26:16.945115+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0051112889999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":409,"reported_epoch":385,"state":"active+clean","last_fresh":"2024-09-10T19:28:30.067739+0000","last_change":"2024-09-10T19:27:57.895873+0000","last_active":"2024-09-10T19:28:30.067739+0000","last_peered":"2024-09-10T19:28:30.067739+0000","last_clean":"2024-09-10T19:28:30.067739+0000","last_became_active":"2024-09-10T19:24:23.903779+0000","last_became_peered":"2024-09-10T19:24:23.903779+0000","last_unstale":"2024-09-10T19:28:30.067739+0000","last_undegraded":"2024-09-10T19:28:30.067739+0000","last_fullsized":"2024-09-10T19:28:30.067739+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:27:57.895799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:27:57.895799+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:29:00.756260+0000","scrub_duration":0.0021993009999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.12","version":"237'19","reported_seq":187,"reported_epoch":385,"state":"active+clean","last_fresh":"2024-09-10T19:28:30.068077+0000","last_change":"2024-09-10T19:27:57.553080+0000","last_active":"2024-09-10T19:28:30.068077+0000","last_peered":"2024-09-10T19:28:30.068077+0000","last_clean":"2024-09-10T19:28:30.068077+0000","last_became_active":"2024-09-10T19:27:53.017161+0000","last_became_peered":"2024-09-10T19:27:53.017161+0000","last_unstale":"2024-09-10T19:28:30.068077+0000","last_undegraded":"2024-09-10T19:28:30.068077+0000","last_fullsized":"2024-09-10T19:28:30.068077+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"237'19","last_scrub_stamp":"2024-09-10T19:27:57.552961+0000","last_deep_scrub":"79'13","last_deep_scrub_stamp":"2024-09-10T19:11:22.072400+0000","last_clean_scrub_stamp":"2024-09-10T19:27:57.552961+0000","objects_scrubbed":4,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:29:19.292651+0000","scrub_duration":0.0054571649999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":19,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":550,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"59'1","reported_seq":34,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-09-10T19:28:19.957168+0000","last_change":"2024-09-10T19:28:03.327487+0000","last_active":"2024-09-10T19:28:19.957168+0000","last_peered":"2024-09-10T19:28:19.957168+0000","last_clean":"2024-09-10T19:28:19.957168+0000","last_became_active":"2024-09-10T19:27:53.018470+0000","last_became_peered":"2024-09-10T19:27:53.018470+0000","last_unstale":"2024-09-10T19:28:19.957168+0000","last_undegraded":"2024-09-10T19:28:19.957168+0000","last_fullsized":"2024-09-10T19:28:19.957168+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'1","last_scrub_stamp":"2024-09-10T19:28:03.327375+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:28:03.327375+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:29:06.088498+0000","scrub_duration":0.0079842409999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":362,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":724,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":430,"reported_epoch":385,"state":"active+clean","last_fresh":"2024-09-10T19:28:30.068049+0000","last_change":"2024-09-10T19:26:23.842995+0000","last_active":"2024-09-10T19:28:30.068049+0000","last_peered":"2024-09-10T19:28:30.068049+0000","last_clean":"2024-09-10T19:28:30.068049+0000","last_became_active":"2024-09-10T19:24:24.923804+0000","last_became_peered":"2024-09-10T19:24:24.923804+0000","last_unstale":"2024-09-10T19:28:30.068049+0000","last_undegraded":"2024-09-10T19:28:30.068049+0000","last_fullsized":"2024-09-10T19:28:30.068049+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:26:23.842943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:26:23.842943+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0015653100000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":369,"reported_epoch":385,"state":"active+clean","last_fresh":"2024-09-10T19:28:30.062276+0000","last_change":"2024-09-10T19:27:56.409510+0000","last_active":"2024-09-10T19:28:30.062276+0000","last_peered":"2024-09-10T19:28:30.062276+0000","last_clean":"2024-09-10T19:28:30.062276+0000","last_became_active":"2024-09-10T19:24:24.916105+0000","last_became_peered":"2024-09-10T19:24:24.916105+0000","last_unstale":"2024-09-10T19:28:30.062276+0000","last_undegraded":"2024-09-10T19:28:30.062276+0000","last_fullsized":"2024-09-10T19:28:30.062276+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:27:56.409429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:25:27.412579+0000","last_clean_scrub_stamp":"2024-09-10T19:27:56.409429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:29:08.483281+0000","scrub_duration":0.0027316609999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"237'19","reported_seq":522,"reported_epoch":385,"state":"active+clean","last_fresh":"2024-09-10T19:28:30.066998+0000","last_change":"2024-09-10T19:27:53.052328+0000","last_active":"2024-09-10T19:28:30.066998+0000","last_peered":"2024-09-10T19:28:30.066998+0000","last_clean":"2024-09-10T19:28:30.066998+0000","last_became_active":"2024-09-10T19:27:53.016518+0000","last_became_peered":"2024-09-10T19:27:53.016518+0000","last_unstale":"2024-09-10T19:28:30.066998+0000","last_undegraded":"2024-09-10T19:28:30.066998+0000","last_fullsized":"2024-09-10T19:28:30.066998+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"237'19","last_scrub_stamp":"2024-09-10T19:26:34.043784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:26:34.043784+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0096839449999999994,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":137,"reported_epoch":385,"state":"active+clean","last_fresh":"2024-09-10T19:28:30.070489+0000","last_change":"2024-09-10T19:26:31.508255+0000","last_active":"2024-09-10T19:28:30.070489+0000","last_peered":"2024-09-10T19:28:30.070489+0000","last_clean":"2024-09-10T19:28:30.070489+0000","last_became_active":"2024-09-10T19:24:24.909540+0000","last_became_peered":"2024-09-10T19:24:24.909540+0000","last_unstale":"2024-09-10T19:28:30.070489+0000","last_undegraded":"2024-09-10T19:28:30.070489+0000","last_fullsized":"2024-09-10T19:28:30.070489+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:26:31.508168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:26:31.508168+0000","last_clean_scrub_stamp":"2024-09-10T19:26:31.508168+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.069232562999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":123,"reported_epoch":385,"state":"active+clean","last_fresh":"2024-09-10T19:28:30.067948+0000","last_change":"2024-09-10T19:28:29.495516+0000","last_active":"2024-09-10T19:28:30.067948+0000","last_peered":"2024-09-10T19:28:30.067948+0000","last_clean":"2024-09-10T19:28:30.067948+0000","last_became_active":"2024-09-10T19:24:01.975520+0000","last_became_peered":"2024-09-10T19:24:01.975520+0000","last_unstale":"2024-09-10T19:28:30.067948+0000","last_undegraded":"2024-09-10T19:28:30.067948+0000","last_fullsized":"2024-09-10T19:28:30.067948+0000","mapping_epoch":292,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":293,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:28:29.495437+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:28:29.495437+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:29:56.211784+0000","scrub_duration":0.0066786709999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":397,"reported_epoch":385,"state":"active+clean","last_fresh":"2024-09-10T19:28:30.070601+0000","last_change":"2024-09-10T19:26:33.897629+0000","last_active":"2024-09-10T19:28:30.070601+0000","last_peered":"2024-09-10T19:28:30.070601+0000","last_clean":"2024-09-10T19:28:30.070601+0000","last_became_active":"2024-09-10T19:24:23.907136+0000","last_became_peered":"2024-09-10T19:24:23.907136+0000","last_unstale":"2024-09-10T19:28:30.070601+0000","last_undegraded":"2024-09-10T19:28:30.070601+0000","last_fullsized":"2024-09-10T19:28:30.070601+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:26:33.897552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:26:33.897552+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0025392180000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"237'10","reported_seq":479,"reported_epoch":385,"state":"active+clean","last_fresh":"2024-09-10T19:28:30.066862+0000","last_change":"2024-09-10T19:27:53.042059+0000","last_active":"2024-09-10T19:28:30.066862+0000","last_peered":"2024-09-10T19:28:30.066862+0000","last_clean":"2024-09-10T19:28:30.066862+0000","last_became_active":"2024-09-10T19:27:53.024782+0000","last_became_peered":"2024-09-10T19:27:53.024782+0000","last_unstale":"2024-09-10T19:28:30.066862+0000","last_undegraded":"2024-09-10T19:28:30.066862+0000","last_fullsized":"2024-09-10T19:28:30.066862+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"237'10","last_scrub_stamp":"2024-09-10T19:27:29.122016+0000","last_deep_scrub":"222'6","last_deep_scrub_stamp":"2024-09-10T19:19:30.328841+0000","last_clean_scrub_stamp":"2024-09-10T19:27:29.122016+0000","objects_scrubbed":2,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:28:50.222590+0000","scrub_duration":0.0031041139999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":12,"num_read_kb":10,"num_write":8,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":446,"reported_epoch":385,"state":"active+clean","last_fresh":"2024-09-10T19:28:30.079763+0000","last_change":"2024-09-10T19:28:28.930460+0000","last_active":"2024-09-10T19:28:30.079763+0000","last_peered":"2024-09-10T19:28:30.079763+0000","last_clean":"2024-09-10T19:28:30.079763+0000","last_became_active":"2024-09-10T19:27:53.000789+0000","last_became_peered":"2024-09-10T19:27:53.000789+0000","last_unstale":"2024-09-10T19:28:30.079763+0000","last_undegraded":"2024-09-10T19:28:30.079763+0000","last_fullsized":"2024-09-10T19:28:30.079763+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:28:28.930372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:28:28.930372+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:29:38.711416+0000","scrub_duration":0.0094954489999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,1],"acting":[5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":30,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-09-10T19:28:19.957904+0000","last_change":"2024-09-10T19:27:57.214988+0000","last_active":"2024-09-10T19:28:19.957904+0000","last_peered":"2024-09-10T19:28:19.957904+0000","last_clean":"2024-09-10T19:28:19.957904+0000","last_became_active":"2024-09-10T19:27:53.003591+0000","last_became_peered":"2024-09-10T19:27:53.003591+0000","last_unstale":"2024-09-10T19:28:19.957904+0000","last_undegraded":"2024-09-10T19:28:19.957904+0000","last_fullsized":"2024-09-10T19:28:19.957904+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:27:57.214792+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:11:31.837944+0000","last_clean_scrub_stamp":"2024-09-10T19:27:57.214792+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:29:11.448959+0000","scrub_duration":0.0032014230000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":420,"reported_epoch":385,"state":"active+clean","last_fresh":"2024-09-10T19:28:30.074394+0000","last_change":"2024-09-10T19:27:58.771502+0000","last_active":"2024-09-10T19:28:30.074394+0000","last_peered":"2024-09-10T19:28:30.074394+0000","last_clean":"2024-09-10T19:28:30.074394+0000","last_became_active":"2024-09-10T19:16:24.852356+0000","last_became_peered":"2024-09-10T19:16:24.852356+0000","last_unstale":"2024-09-10T19:28:30.074394+0000","last_undegraded":"2024-09-10T19:28:30.074394+0000","last_fullsized":"2024-09-10T19:28:30.074394+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:27:58.771428+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:27:58.771428+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:29:01.702679+0000","scrub_duration":0.0067317310000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,3],"acting":[7,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.11","version":"237'19","reported_seq":230,"reported_epoch":385,"state":"active+clean","last_fresh":"2024-09-10T19:28:30.070919+0000","last_change":"2024-09-10T19:27:53.044633+0000","last_active":"2024-09-10T19:28:30.070919+0000","last_peered":"2024-09-10T19:28:30.070919+0000","last_clean":"2024-09-10T19:28:30.070919+0000","last_became_active":"2024-09-10T19:27:53.018213+0000","last_became_peered":"2024-09-10T19:27:53.018213+0000","last_unstale":"2024-09-10T19:28:30.070919+0000","last_undegraded":"2024-09-10T19:28:30.070919+0000","last_fullsized":"2024-09-10T19:28:30.070919+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"237'19","last_scrub_stamp":"2024-09-10T19:26:31.962585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:26:31.962585+0000","objects_scrubbed":5,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0022806200000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":32,"num_read_kb":25,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":30,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-09-10T19:28:19.956989+0000","last_change":"2024-09-10T19:28:12.336641+0000","last_active":"2024-09-10T19:28:19.956989+0000","last_peered":"2024-09-10T19:28:19.956989+0000","last_clean":"2024-09-10T19:28:19.956989+0000","last_became_active":"2024-09-10T19:27:53.008197+0000","last_became_peered":"2024-09-10T19:27:53.008197+0000","last_unstale":"2024-09-10T19:28:19.956989+0000","last_undegraded":"2024-09-10T19:28:19.956989+0000","last_fullsized":"2024-09-10T19:28:19.956989+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:28:12.336562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:13:05.876655+0000","last_clean_scrub_stamp":"2024-09-10T19:28:12.336562+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:29:31.495257+0000","scrub_duration":0.0063511399999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":422,"reported_epoch":385,"state":"active+clean","last_fresh":"2024-09-10T19:28:30.073973+0000","last_change":"2024-09-10T19:27:59.744732+0000","last_active":"2024-09-10T19:28:30.073973+0000","last_peered":"2024-09-10T19:28:30.073973+0000","last_clean":"2024-09-10T19:28:30.073973+0000","last_became_active":"2024-09-10T19:27:53.008444+0000","last_became_peered":"2024-09-10T19:27:53.008444+0000","last_unstale":"2024-09-10T19:28:30.073973+0000","last_undegraded":"2024-09-10T19:28:30.073973+0000","last_fullsized":"2024-09-10T19:28:30.073973+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:27:59.744651+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:11:57.112092+0000","last_clean_scrub_stamp":"2024-09-10T19:27:59.744651+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:29:17.632950+0000","scrub_duration":0.0070166489999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":417,"reported_epoch":385,"state":"active+clean","last_fresh":"2024-09-10T19:28:30.067636+0000","last_change":"2024-09-10T19:27:53.026128+0000","last_active":"2024-09-10T19:28:30.067636+0000","last_peered":"2024-09-10T19:28:30.067636+0000","last_clean":"2024-09-10T19:28:30.067636+0000","last_became_active":"2024-09-10T19:27:53.025650+0000","last_became_peered":"2024-09-10T19:27:53.025650+0000","last_unstale":"2024-09-10T19:28:30.067636+0000","last_undegraded":"2024-09-10T19:28:30.067636+0000","last_fullsized":"2024-09-10T19:28:30.067636+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:26:29.959202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:26:29.959202+0000","last_clean_scrub_stamp":"2024-09-10T19:26:29.959202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0059040539999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"80'2","reported_seq":426,"reported_epoch":385,"state":"active+clean","last_fresh":"2024-09-10T19:28:30.078492+0000","last_change":"2024-09-10T19:28:23.749141+0000","last_active":"2024-09-10T19:28:30.078492+0000","last_peered":"2024-09-10T19:28:30.078492+0000","last_clean":"2024-09-10T19:28:30.078492+0000","last_became_active":"2024-09-10T19:16:24.851220+0000","last_became_peered":"2024-09-10T19:16:24.851220+0000","last_unstale":"2024-09-10T19:28:30.078492+0000","last_undegraded":"2024-09-10T19:28:30.078492+0000","last_fullsized":"2024-09-10T19:28:30.078492+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"80'2","last_scrub_stamp":"2024-09-10T19:28:23.749046+0000","last_deep_scrub":"80'2","last_deep_scrub_stamp":"2024-09-10T19:11:30.272043+0000","last_clean_scrub_stamp":"2024-09-10T19:28:23.749046+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:29:50.660289+0000","scrub_duration":0.005759307,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":369,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":369,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,3],"acting":[7,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":413,"reported_epoch":385,"state":"active+clean","last_fresh":"2024-09-10T19:28:30.078994+0000","last_change":"2024-09-10T19:27:27.890736+0000","last_active":"2024-09-10T19:28:30.078994+0000","last_peered":"2024-09-10T19:28:30.078994+0000","last_clean":"2024-09-10T19:28:30.078994+0000","last_became_active":"2024-09-10T19:24:23.906812+0000","last_became_peered":"2024-09-10T19:24:23.906812+0000","last_unstale":"2024-09-10T19:28:30.078994+0000","last_undegraded":"2024-09-10T19:28:30.078994+0000","last_fullsized":"2024-09-10T19:28:30.078994+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:27:27.890412+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:27:27.890412+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:28:37.948238+0000","scrub_duration":0.008899265,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,0],"acting":[5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1e","version":"237'25","reported_seq":389,"reported_epoch":385,"state":"active+clean","last_fresh":"2024-09-10T19:28:30.063849+0000","last_change":"2024-09-10T19:28:00.387263+0000","last_active":"2024-09-10T19:28:30.063849+0000","last_peered":"2024-09-10T19:28:30.063849+0000","last_clean":"2024-09-10T19:28:30.063849+0000","last_became_active":"2024-09-10T19:24:23.899507+0000","last_became_peered":"2024-09-10T19:24:23.899507+0000","last_unstale":"2024-09-10T19:28:30.063849+0000","last_undegraded":"2024-09-10T19:28:30.063849+0000","last_fullsized":"2024-09-10T19:28:30.063849+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"237'25","last_scrub_stamp":"2024-09-10T19:28:00.387191+0000","last_deep_scrub":"80'15","last_deep_scrub_stamp":"2024-09-10T19:11:25.883919+0000","last_clean_scrub_stamp":"2024-09-10T19:28:00.387191+0000","objects_scrubbed":5,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:29:02.957263+0000","scrub_duration":0.0049959890000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":24,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1f","version":"376'10","reported_seq":32,"reported_epoch":385,"state":"active+clean","last_fresh":"2024-09-10T19:28:30.072707+0000","last_change":"2024-09-10T19:27:53.026209+0000","last_active":"2024-09-10T19:28:30.072707+0000","last_peered":"2024-09-10T19:28:30.072707+0000","last_clean":"2024-09-10T19:28:30.072707+0000","last_became_active":"2024-09-10T19:27:53.003676+0000","last_became_peered":"2024-09-10T19:27:53.003676+0000","last_unstale":"2024-09-10T19:28:30.072707+0000","last_undegraded":"2024-09-10T19:28:30.072707+0000","last_fullsized":"2024-09-10T19:28:30.072707+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'8","last_scrub_stamp":"2024-09-10T19:27:08.400635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:27:08.400635+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":123,"reported_epoch":385,"state":"active+clean","last_fresh":"2024-09-10T19:28:30.070573+0000","last_change":"2024-09-10T19:28:20.514190+0000","last_active":"2024-09-10T19:28:30.070573+0000","last_peered":"2024-09-10T19:28:30.070573+0000","last_clean":"2024-09-10T19:28:30.070573+0000","last_became_active":"2024-09-10T19:24:01.980391+0000","last_became_peered":"2024-09-10T19:24:01.980391+0000","last_unstale":"2024-09-10T19:28:30.070573+0000","last_undegraded":"2024-09-10T19:28:30.070573+0000","last_fullsized":"2024-09-10T19:28:30.070573+0000","mapping_epoch":292,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":293,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:28:20.514115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:28:20.514115+0000","last_clean_scrub_stamp":"2024-09-10T19:28:20.514115+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:29:24.632679+0000","scrub_duration":0.004668223,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":433,"reported_epoch":385,"state":"active+clean","last_fresh":"2024-09-10T19:28:30.066622+0000","last_change":"2024-09-10T19:27:53.018605+0000","last_active":"2024-09-10T19:28:30.066622+0000","last_peered":"2024-09-10T19:28:30.066622+0000","last_clean":"2024-09-10T19:28:30.066622+0000","last_became_active":"2024-09-10T19:27:53.017153+0000","last_became_peered":"2024-09-10T19:27:53.017153+0000","last_unstale":"2024-09-10T19:28:30.066622+0000","last_undegraded":"2024-09-10T19:28:30.066622+0000","last_fullsized":"2024-09-10T19:28:30.066622+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:26:36.923520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:26:36.923520+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001294981,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"237'24","reported_seq":42,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-09-10T19:28:19.955733+0000","last_change":"2024-09-10T19:28:10.314164+0000","last_active":"2024-09-10T19:28:19.955733+0000","last_peered":"2024-09-10T19:28:19.955733+0000","last_clean":"2024-09-10T19:28:19.955733+0000","last_became_active":"2024-09-10T19:27:53.008233+0000","last_became_peered":"2024-09-10T19:27:53.008233+0000","last_unstale":"2024-09-10T19:28:19.955733+0000","last_undegraded":"2024-09-10T19:28:19.955733+0000","last_fullsized":"2024-09-10T19:28:19.955733+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"237'24","last_scrub_stamp":"2024-09-10T19:28:10.314087+0000","last_deep_scrub":"80'16","last_deep_scrub_stamp":"2024-09-10T19:11:40.853611+0000","last_clean_scrub_stamp":"2024-09-10T19:28:10.314087+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:29:16.033278+0000","scrub_duration":0.0077151600000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":24,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":341,"reported_epoch":385,"state":"active+clean","last_fresh":"2024-09-10T19:28:30.063705+0000","last_change":"2024-09-10T19:28:19.464489+0000","last_active":"2024-09-10T19:28:30.063705+0000","last_peered":"2024-09-10T19:28:30.063705+0000","last_clean":"2024-09-10T19:28:30.063705+0000","last_became_active":"2024-09-10T19:24:24.899153+0000","last_became_peered":"2024-09-10T19:24:24.899153+0000","last_unstale":"2024-09-10T19:28:30.063705+0000","last_undegraded":"2024-09-10T19:28:30.063705+0000","last_fullsized":"2024-09-10T19:28:30.063705+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:28:19.464426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:28:19.464426+0000","last_clean_scrub_stamp":"2024-09-10T19:28:19.464426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:29:41.533757+0000","scrub_duration":0.005180703,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":419,"reported_epoch":385,"state":"active+clean","last_fresh":"2024-09-10T19:28:30.069864+0000","last_change":"2024-09-10T19:27:53.011903+0000","last_active":"2024-09-10T19:28:30.069864+0000","last_peered":"2024-09-10T19:28:30.069864+0000","last_clean":"2024-09-10T19:28:30.069864+0000","last_became_active":"2024-09-10T19:27:53.011336+0000","last_became_peered":"2024-09-10T19:27:53.011336+0000","last_unstale":"2024-09-10T19:28:30.069864+0000","last_undegraded":"2024-09-10T19:28:30.069864+0000","last_fullsized":"2024-09-10T19:28:30.069864+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:26:30.968972+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:26:30.968972+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0072260129999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":30,"reported_epoch":384,"state":"active+clean","last_fresh":"2024-09-10T19:28:19.955197+0000","last_change":"2024-09-10T19:28:11.358765+0000","last_active":"2024-09-10T19:28:19.955197+0000","last_peered":"2024-09-10T19:28:19.955197+0000","last_clean":"2024-09-10T19:28:19.955197+0000","last_became_active":"2024-09-10T19:27:53.002313+0000","last_became_peered":"2024-09-10T19:27:53.002313+0000","last_unstale":"2024-09-10T19:28:19.955197+0000","last_undegraded":"2024-09-10T19:28:19.955197+0000","last_fullsized":"2024-09-10T19:28:19.955197+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:28:11.358680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:28:11.358680+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:29:17.177067+0000","scrub_duration":0.0022050009999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1c","version":"237'34","reported_seq":153,"reported_epoch":385,"state":"active+clean","last_fresh":"2024-09-10T19:28:30.072659+0000","last_change":"2024-09-10T19:27:26.470498+0000","last_active":"2024-09-10T19:28:30.072659+0000","last_peered":"2024-09-10T19:28:30.072659+0000","last_clean":"2024-09-10T19:28:30.072659+0000","last_became_active":"2024-09-10T19:24:24.910577+0000","last_became_peered":"2024-09-10T19:24:24.910577+0000","last_unstale":"2024-09-10T19:28:30.072659+0000","last_undegraded":"2024-09-10T19:28:30.072659+0000","last_fullsized":"2024-09-10T19:28:30.072659+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"237'34","last_scrub_stamp":"2024-09-10T19:27:26.470398+0000","last_deep_scrub":"78'22","last_deep_scrub_stamp":"2024-09-10T19:11:21.118937+0000","last_clean_scrub_stamp":"2024-09-10T19:27:26.470398+0000","objects_scrubbed":7,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:28:53.169245+0000","scrub_duration":0.002418529,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":44,"num_read_kb":35,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":137,"reported_epoch":385,"state":"active+clean","last_fresh":"2024-09-10T19:28:30.070659+0000","last_change":"2024-09-10T19:27:07.380501+0000","last_active":"2024-09-10T19:28:30.070659+0000","last_peered":"2024-09-10T19:28:30.070659+0000","last_clean":"2024-09-10T19:28:30.070659+0000","last_became_active":"2024-09-10T19:24:24.887205+0000","last_became_peered":"2024-09-10T19:24:24.887205+0000","last_unstale":"2024-09-10T19:28:30.070659+0000","last_undegraded":"2024-09-10T19:28:30.070659+0000","last_fullsized":"2024-09-10T19:28:30.070659+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:27:07.380435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:27:07.380435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:28:35.544652+0000","scrub_duration":0.0036522960000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":413,"reported_epoch":385,"state":"active+clean","last_fresh":"2024-09-10T19:28:30.068271+0000","last_change":"2024-09-10T19:27:53.026958+0000","last_active":"2024-09-10T19:28:30.068271+0000","last_peered":"2024-09-10T19:28:30.068271+0000","last_clean":"2024-09-10T19:28:30.068271+0000","last_became_active":"2024-09-10T19:27:53.026591+0000","last_became_peered":"2024-09-10T19:27:53.026591+0000","last_unstale":"2024-09-10T19:28:30.068271+0000","last_undegraded":"2024-09-10T19:28:30.068271+0000","last_fullsized":"2024-09-10T19:28:30.068271+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:26:30.990819+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:24:46.518816+0000","last_clean_scrub_stamp":"2024-09-10T19:26:30.990819+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0073104629999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":426,"reported_epoch":385,"state":"active+clean","last_fresh":"2024-09-10T19:28:30.068126+0000","last_change":"2024-09-10T19:28:19.170262+0000","last_active":"2024-09-10T19:28:30.068126+0000","last_peered":"2024-09-10T19:28:30.068126+0000","last_clean":"2024-09-10T19:28:30.068126+0000","last_became_active":"2024-09-10T19:27:53.018726+0000","last_became_peered":"2024-09-10T19:27:53.018726+0000","last_unstale":"2024-09-10T19:28:30.068126+0000","last_undegraded":"2024-09-10T19:28:30.068126+0000","last_fullsized":"2024-09-10T19:28:30.068126+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:28:19.170191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:28:19.170191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:29:29.039423+0000","scrub_duration":0.0055786129999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1d","version":"250'23","reported_seq":482,"reported_epoch":385,"state":"active+clean","last_fresh":"2024-09-10T19:28:30.069389+0000","last_change":"2024-09-10T19:28:09.103555+0000","last_active":"2024-09-10T19:28:30.069389+0000","last_peered":"2024-09-10T19:28:30.069389+0000","last_clean":"2024-09-10T19:28:30.069389+0000","last_became_active":"2024-09-10T19:24:01.976664+0000","last_became_peered":"2024-09-10T19:24:01.976664+0000","last_unstale":"2024-09-10T19:28:30.069389+0000","last_undegraded":"2024-09-10T19:28:30.069389+0000","last_fullsized":"2024-09-10T19:28:30.069389+0000","mapping_epoch":292,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":293,"parent":"0.0","parent_split_bits":0,"last_scrub":"250'23","last_scrub_stamp":"2024-09-10T19:28:09.103469+0000","last_deep_scrub":"250'23","last_deep_scrub_stamp":"2024-09-10T19:20:55.582173+0000","last_clean_scrub_stamp":"2024-09-10T19:28:09.103469+0000","objects_scrubbed":5,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:29:36.986330+0000","scrub_duration":0.0078562079999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":34,"num_read_kb":25,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":411,"reported_epoch":385,"state":"active+clean","last_fresh":"2024-09-10T19:28:30.069364+0000","last_change":"2024-09-10T19:26:29.937585+0000","last_active":"2024-09-10T19:28:30.069364+0000","last_peered":"2024-09-10T19:28:30.069364+0000","last_clean":"2024-09-10T19:28:30.069364+0000","last_became_active":"2024-09-10T19:16:24.843013+0000","last_became_peered":"2024-09-10T19:16:24.843013+0000","last_unstale":"2024-09-10T19:28:30.069364+0000","last_undegraded":"2024-09-10T19:28:30.069364+0000","last_fullsized":"2024-09-10T19:28:30.069364+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:26:29.937532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:26:29.937532+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0024737750000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":11,"num_pg":16,"stat_sum":{"num_bytes":5222891520,"num_objects":79695,"num_object_clones":0,"num_object_copies":239085,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":79695,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":79695,"num_write_kb":5100480,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":15742337024,"data_stored":15742337024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":69995,"ondisk_log_size":69995,"up":48,"acting":48,"num_store_stats":7},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":382,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":1528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":95,"ondisk_log_size":95,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":177,"num_object_clones":0,"num_object_copies":531,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":177,"num_whiteouts":0,"num_read":1115,"num_read_kb":872,"num_write":703,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":433,"num_bytes_recovered":8406,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":458752,"data_stored":12124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":780,"ondisk_log_size":780,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":278,"num_read_kb":273,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1594,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1613,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":9,"ondisk_log_size":9,"up":9,"acting":9,"num_store_stats":7},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1328,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":21,"num_read_kb":21,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":724,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":3984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5,"ondisk_log_size":5,"up":96,"acting":96,"num_store_stats":8},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":525376,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":870,"ondisk_log_size":870,"up":33,"acting":33,"num_store_stats":7}],"osd_stats":[{"osd":7,"up_from":54,"seq":231928234210,"num_pgs":71,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3409516,"kb_used_data":2517220,"kb_used_omap":1,"kb_used_meta":892286,"kb_avail":90335636,"statfs":{"total":95995035648,"available":92503691264,"internally_reserved":0,"allocated":2577633280,"data_stored":2577241297,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":913701208},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,2,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 19:28:31 2024","interfaces":[{"interface":"back","average":{"1min":0.82999999999999996,"5min":0.90200000000000002,"15min":0.94699999999999995},"min":{"1min":0.21199999999999999,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":1.766,"5min":5.0510000000000002,"15min":5.0510000000000002},"last":1.157},{"interface":"front","average":{"1min":0.86099999999999999,"5min":0.873,"15min":0.75},"min":{"1min":0.27000000000000002,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":2.9940000000000002,"5min":4.7190000000000003,"15min":4.7190000000000003},"last":1.028}]},{"osd":1,"last update":"Tue Sep 10 19:27:57 2024","interfaces":[{"interface":"back","average":{"1min":1.1359999999999999,"5min":0.94999999999999996,"15min":1.0069999999999999},"min":{"1min":0.27400000000000002,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":5.0140000000000002,"5min":5.8140000000000001,"15min":5.8140000000000001},"last":1.228},{"interface":"front","average":{"1min":0.77600000000000002,"5min":0.96799999999999997,"15min":0.88},"min":{"1min":0.26400000000000001,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":4.585,"5min":4.8540000000000001,"15min":4.8540000000000001},"last":0.82199999999999995}]},{"osd":2,"last update":"Tue Sep 10 19:28:12 2024","interfaces":[{"interface":"back","average":{"1min":0.70399999999999996,"5min":0.81499999999999995,"15min":1.03},"min":{"1min":0.308,"5min":0.29799999999999999,"15min":0.23000000000000001},"max":{"1min":3.0579999999999998,"5min":4.7400000000000002,"15min":31.940999999999999},"last":1.0940000000000001},{"interface":"front","average":{"1min":0.64900000000000002,"5min":1.012,"15min":1.1319999999999999},"min":{"1min":0.34599999999999997,"5min":0.30299999999999999,"15min":0.24099999999999999},"max":{"1min":1.458,"5min":4.9630000000000001,"15min":25.056000000000001},"last":0.57499999999999996}]},{"osd":3,"last update":"Tue Sep 10 19:28:12 2024","interfaces":[{"interface":"back","average":{"1min":0.71999999999999997,"5min":0.82199999999999995,"15min":1.1910000000000001},"min":{"1min":0.30399999999999999,"5min":0.25900000000000001,"15min":0.20200000000000001},"max":{"1min":3.641,"5min":4.6909999999999998,"15min":25.443999999999999},"last":1.117},{"interface":"front","average":{"1min":0.72199999999999998,"5min":1.018,"15min":1.2030000000000001},"min":{"1min":0.33300000000000002,"5min":0.27700000000000002,"15min":0.25700000000000001},"max":{"1min":1.8839999999999999,"5min":4.8819999999999997,"15min":25.530000000000001},"last":0.60599999999999998}]},{"osd":4,"last update":"Tue Sep 10 19:28:12 2024","interfaces":[{"interface":"back","average":{"1min":2.3290000000000002,"5min":4.7290000000000001,"15min":2.5840000000000001},"min":{"1min":0.30099999999999999,"5min":0.249,"15min":0.222},"max":{"1min":18.664999999999999,"5min":231.923,"15min":231.923},"last":1.206},{"interface":"front","average":{"1min":1.7569999999999999,"5min":4.548,"15min":2.282},"min":{"1min":0.28999999999999998,"5min":0.28000000000000003,"15min":0.23000000000000001},"max":{"1min":14.837,"5min":231.322,"15min":231.322},"last":1.056}]},{"osd":5,"last update":"Tue Sep 10 19:28:12 2024","interfaces":[{"interface":"back","average":{"1min":0.70099999999999996,"5min":1.1739999999999999,"15min":1.6759999999999999},"min":{"1min":0.35099999999999998,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":2.794,"5min":8.8300000000000001,"15min":67.382999999999996},"last":0.63},{"interface":"front","average":{"1min":1.1759999999999999,"5min":1.3440000000000001,"15min":1.903},"min":{"1min":0.34000000000000002,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":3.6560000000000001,"5min":8.7040000000000006,"15min":67.75},"last":1.1859999999999999}]},{"osd":6,"last update":"Tue Sep 10 19:28:12 2024","interfaces":[{"interface":"back","average":{"1min":1.212,"5min":2.0430000000000001,"15min":2.2959999999999998},"min":{"1min":0.38100000000000001,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":3.5609999999999999,"5min":42.851999999999997,"15min":181.53800000000001},"last":1.2490000000000001},{"interface":"front","average":{"1min":0.77300000000000002,"5min":1.758,"15min":2.0059999999999998},"min":{"1min":0.318,"5min":0.30299999999999999,"15min":0.27000000000000002},"max":{"1min":3.032,"5min":41.731999999999999,"15min":182.05600000000001},"last":1.0760000000000001}]}]},{"osd":6,"up_from":47,"seq":201863463143,"num_pgs":65,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3397752,"kb_used_data":2498800,"kb_used_omap":1,"kb_used_meta":898942,"kb_avail":90347400,"statfs":{"total":95995035648,"available":92515737600,"internally_reserved":0,"allocated":2558771200,"data_stored":2558368017,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":920516952},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 19:28:34 2024","interfaces":[{"interface":"back","average":{"1min":0.83699999999999997,"5min":1.569,"15min":1.772},"min":{"1min":0.33700000000000002,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":2.8399999999999999,"5min":18.545999999999999,"15min":18.545999999999999},"last":0.875},{"interface":"front","average":{"1min":0.70099999999999996,"5min":1.4790000000000001,"15min":1.825},"min":{"1min":0.32800000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":2.7890000000000001,"5min":19.818000000000001,"15min":19.818000000000001},"last":1.1339999999999999}]},{"osd":1,"last update":"Tue Sep 10 19:27:57 2024","interfaces":[{"interface":"back","average":{"1min":0.96799999999999997,"5min":1.1379999999999999,"15min":0.86099999999999999},"min":{"1min":0.251,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":3.4820000000000002,"5min":7.0019999999999998,"15min":7.0019999999999998},"last":0.38800000000000001},{"interface":"front","average":{"1min":1.1060000000000001,"5min":0.91800000000000004,"15min":0.80800000000000005},"min":{"1min":0.27300000000000002,"5min":0.21299999999999999,"15min":0.21299999999999999},"max":{"1min":5.7329999999999997,"5min":5.7329999999999997,"15min":5.7329999999999997},"last":0.36799999999999999}]},{"osd":2,"last update":"Tue Sep 10 19:28:08 2024","interfaces":[{"interface":"back","average":{"1min":0.64700000000000002,"5min":0.97499999999999998,"15min":1.2749999999999999},"min":{"1min":0.31900000000000001,"5min":0.25700000000000001,"15min":0.20399999999999999},"max":{"1min":3.4129999999999998,"5min":5.9029999999999996,"15min":112.864},"last":0.40300000000000002},{"interface":"front","average":{"1min":0.65600000000000003,"5min":0.88900000000000001,"15min":1.359},"min":{"1min":0.33700000000000002,"5min":0.25600000000000001,"15min":0.254},"max":{"1min":2.4990000000000001,"5min":6.0709999999999997,"15min":112.822},"last":0.48699999999999999}]},{"osd":3,"last update":"Tue Sep 10 19:28:08 2024","interfaces":[{"interface":"back","average":{"1min":1.395,"5min":1.2549999999999999,"15min":1.6499999999999999},"min":{"1min":0.38900000000000001,"5min":0.26900000000000002,"15min":0.20399999999999999},"max":{"1min":9.2390000000000008,"5min":9.2390000000000008,"15min":125.56399999999999},"last":1.6220000000000001},{"interface":"front","average":{"1min":1.169,"5min":1.1100000000000001,"15min":1.6040000000000001},"min":{"1min":0.29899999999999999,"5min":0.247,"15min":0.20499999999999999},"max":{"1min":9.0250000000000004,"5min":9.0250000000000004,"15min":125.47},"last":0.47399999999999998}]},{"osd":4,"last update":"Tue Sep 10 19:28:08 2024","interfaces":[{"interface":"back","average":{"1min":2.8570000000000002,"5min":1.736,"15min":1.365},"min":{"1min":0.32800000000000001,"5min":0.22,"15min":0.20699999999999999},"max":{"1min":35.552,"5min":35.552,"15min":35.552},"last":0.26900000000000002},{"interface":"front","average":{"1min":2.984,"5min":1.98,"15min":1.343},"min":{"1min":0.32300000000000001,"5min":0.26400000000000001,"15min":0.22500000000000001},"max":{"1min":35.192999999999998,"5min":35.192999999999998,"15min":35.192999999999998},"last":0.52700000000000002}]},{"osd":5,"last update":"Tue Sep 10 19:28:08 2024","interfaces":[{"interface":"back","average":{"1min":3.6469999999999998,"5min":1.7789999999999999,"15min":1.5569999999999999},"min":{"1min":0.34599999999999997,"5min":0.26500000000000001,"15min":0.248},"max":{"1min":48.216000000000001,"5min":48.216000000000001,"15min":48.216000000000001},"last":0.51600000000000001},{"interface":"front","average":{"1min":3.79,"5min":2.2130000000000001,"15min":1.75},"min":{"1min":0.379,"5min":0.254,"15min":0.254},"max":{"1min":50.372999999999998,"5min":50.372999999999998,"15min":50.372999999999998},"last":0.35499999999999998}]},{"osd":7,"last update":"Tue Sep 10 19:28:34 2024","interfaces":[{"interface":"back","average":{"1min":0.78200000000000003,"5min":2.4260000000000002,"15min":1.5009999999999999},"min":{"1min":0.28299999999999997,"5min":0.28100000000000003,"15min":0.27500000000000002},"max":{"1min":4.1269999999999998,"5min":80.629999999999995,"15min":80.629999999999995},"last":0.437},{"interface":"front","average":{"1min":1.5289999999999999,"5min":2.6579999999999999,"15min":1.7709999999999999},"min":{"1min":0.29399999999999998,"5min":0.28299999999999997,"15min":0.25700000000000001},"max":{"1min":6.5659999999999998,"5min":81.088999999999999,"15min":81.088999999999999},"last":0.41699999999999998}]}]},{"osd":5,"up_from":39,"seq":167503724780,"num_pgs":65,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2163636,"kb_used_data":1269868,"kb_used_omap":1,"kb_used_meta":893758,"kb_avail":91581516,"statfs":{"total":95995035648,"available":93779472384,"internally_reserved":0,"allocated":1300344832,"data_stored":1299946417,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":915208536},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1,1,1],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 19:28:32 2024","interfaces":[{"interface":"back","average":{"1min":0.94699999999999995,"5min":1.1200000000000001,"15min":1.1200000000000001},"min":{"1min":0.22700000000000001,"5min":0.187,"15min":0.187},"max":{"1min":4.3529999999999998,"5min":5.0369999999999999,"15min":5.0369999999999999},"last":1.6120000000000001},{"interface":"front","average":{"1min":0.83799999999999997,"5min":1.089,"15min":1.222},"min":{"1min":0.26100000000000001,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":2.657,"5min":3.8580000000000001,"15min":3.8580000000000001},"last":1.734}]},{"osd":1,"last update":"Tue Sep 10 19:27:48 2024","interfaces":[{"interface":"back","average":{"1min":0.83499999999999996,"5min":0.95099999999999996,"15min":0.97699999999999998},"min":{"1min":0.214,"5min":0.214,"15min":0.214},"max":{"1min":3.2450000000000001,"5min":8.2319999999999993,"15min":8.2319999999999993},"last":0.65900000000000003},{"interface":"front","average":{"1min":1.081,"5min":1.0109999999999999,"15min":1.046},"min":{"1min":0.221,"5min":0.221,"15min":0.221},"max":{"1min":4.9569999999999999,"5min":4.9569999999999999,"15min":4.9569999999999999},"last":1.6619999999999999}]},{"osd":2,"last update":"Tue Sep 10 19:27:32 2024","interfaces":[{"interface":"back","average":{"1min":1.2290000000000001,"5min":1.0249999999999999,"15min":1.095},"min":{"1min":0.27700000000000002,"5min":0.245,"15min":0.22900000000000001},"max":{"1min":4.992,"5min":4.992,"15min":9.9399999999999995},"last":0.60499999999999998},{"interface":"front","average":{"1min":1.1319999999999999,"5min":0.95299999999999996,"15min":1.052},"min":{"1min":0.34799999999999998,"5min":0.22500000000000001,"15min":0.21199999999999999},"max":{"1min":3.6629999999999998,"5min":3.8319999999999999,"15min":6.3789999999999996},"last":0.72699999999999998}]},{"osd":3,"last update":"Tue Sep 10 19:27:32 2024","interfaces":[{"interface":"back","average":{"1min":1.252,"5min":1.3420000000000001,"15min":1.181},"min":{"1min":0.254,"5min":0.222,"15min":0.19900000000000001},"max":{"1min":5.0149999999999997,"5min":26.355,"15min":26.355},"last":1.641},{"interface":"front","average":{"1min":1.1579999999999999,"5min":1.27,"15min":1.1319999999999999},"min":{"1min":0.35199999999999998,"5min":0.246,"15min":0.246},"max":{"1min":3.6829999999999998,"5min":26.327000000000002,"15min":26.327000000000002},"last":1.7669999999999999}]},{"osd":4,"last update":"Tue Sep 10 19:27:32 2024","interfaces":[{"interface":"back","average":{"1min":1.3939999999999999,"5min":2.8740000000000001,"15min":1.754},"min":{"1min":0.33100000000000002,"5min":0.26100000000000001,"15min":0.23699999999999999},"max":{"1min":4.3819999999999997,"5min":128.08099999999999,"15min":128.08099999999999},"last":2.1749999999999998},{"interface":"front","average":{"1min":1.98,"5min":3.093,"15min":1.895},"min":{"1min":0.28699999999999998,"5min":0.217,"15min":0.217},"max":{"1min":6.4000000000000004,"5min":126.321,"15min":126.321},"last":0.56499999999999995}]},{"osd":6,"last update":"Tue Sep 10 19:28:09 2024","interfaces":[{"interface":"back","average":{"1min":0.92800000000000005,"5min":1.4450000000000001,"15min":1.8129999999999999},"min":{"1min":0.28599999999999998,"5min":0.28599999999999998,"15min":0.27100000000000002},"max":{"1min":5.202,"5min":8.3339999999999996,"15min":111.328},"last":2.4100000000000001},{"interface":"front","average":{"1min":0.95999999999999996,"5min":1.4590000000000001,"15min":1.758},"min":{"1min":0.27400000000000002,"5min":0.27200000000000002,"15min":0.26900000000000002},"max":{"1min":4.7039999999999997,"5min":7.2910000000000004,"15min":111.705},"last":0.68899999999999995}]},{"osd":7,"last update":"Tue Sep 10 19:28:26 2024","interfaces":[{"interface":"back","average":{"1min":0.75,"5min":1.1910000000000001,"15min":1.335},"min":{"1min":0.34899999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":3.7029999999999998,"5min":4.101,"15min":6.7869999999999999},"last":0.88700000000000001},{"interface":"front","average":{"1min":0.66700000000000004,"5min":1.149,"15min":1.29},"min":{"1min":0.35299999999999998,"5min":0.32600000000000001,"15min":0.27100000000000002},"max":{"1min":2.0920000000000001,"5min":8.1679999999999993,"15min":9.4909999999999997},"last":0.63700000000000001}]}]},{"osd":4,"up_from":32,"seq":137438953712,"num_pgs":65,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2475184,"kb_used_data":1578472,"kb_used_omap":1,"kb_used_meta":896702,"kb_avail":91269968,"statfs":{"total":95995035648,"available":93460447232,"internally_reserved":0,"allocated":1616355328,"data_stored":1615959100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":918223192},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 19:28:29 2024","interfaces":[{"interface":"back","average":{"1min":0.68300000000000005,"5min":2.081,"15min":2.222},"min":{"1min":0.252,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":1.77,"5min":39.097000000000001,"15min":39.097000000000001},"last":0.38700000000000001},{"interface":"front","average":{"1min":0.60399999999999998,"5min":2.3399999999999999,"15min":2.528},"min":{"1min":0.25800000000000001,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":1.927,"5min":39.112000000000002,"15min":39.112000000000002},"last":0.41699999999999998}]},{"osd":1,"last update":"Tue Sep 10 19:27:52 2024","interfaces":[{"interface":"back","average":{"1min":1.1539999999999999,"5min":1.115,"15min":0.84099999999999997},"min":{"1min":0.376,"5min":0.20599999999999999,"15min":0.20599999999999999},"max":{"1min":4.3129999999999997,"5min":4.7960000000000003,"15min":4.7960000000000003},"last":1.6459999999999999},{"interface":"front","average":{"1min":1.179,"5min":0.996,"15min":0.73599999999999999},"min":{"1min":0.254,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":5.8680000000000003,"5min":5.8680000000000003,"15min":5.8680000000000003},"last":1.198}]},{"osd":2,"last update":"Tue Sep 10 19:28:24 2024","interfaces":[{"interface":"back","average":{"1min":0.66700000000000004,"5min":1.1819999999999999,"15min":1.006},"min":{"1min":0.33800000000000002,"5min":0.27000000000000002,"15min":0.214},"max":{"1min":2.0649999999999999,"5min":4.8410000000000002,"15min":6.8869999999999996},"last":0.55900000000000005},{"interface":"front","average":{"1min":0.66000000000000003,"5min":1.101,"15min":0.98199999999999998},"min":{"1min":0.28599999999999998,"5min":0.28299999999999997,"15min":0.23400000000000001},"max":{"1min":2.214,"5min":6.7690000000000001,"15min":7.085},"last":2.3839999999999999}]},{"osd":3,"last update":"Tue Sep 10 19:28:24 2024","interfaces":[{"interface":"back","average":{"1min":0.72499999999999998,"5min":1.085,"15min":1.1819999999999999},"min":{"1min":0.34699999999999998,"5min":0.27800000000000002,"15min":0.20999999999999999},"max":{"1min":2.5350000000000001,"5min":6.8019999999999996,"15min":35.451999999999998},"last":0.29099999999999998},{"interface":"front","average":{"1min":0.746,"5min":0.97399999999999998,"15min":1.208},"min":{"1min":0.32400000000000001,"5min":0.28599999999999998,"15min":0.246},"max":{"1min":1.8100000000000001,"5min":5.8949999999999996,"15min":32.880000000000003},"last":0.35899999999999999}]},{"osd":5,"last update":"Tue Sep 10 19:27:42 2024","interfaces":[{"interface":"back","average":{"1min":1.4259999999999999,"5min":2.0299999999999998,"15min":2.5369999999999999},"min":{"1min":0.32900000000000001,"5min":0.26800000000000002,"15min":0.224},"max":{"1min":3.536,"5min":58.009999999999998,"15min":227.726},"last":0.64900000000000002},{"interface":"front","average":{"1min":1.4850000000000001,"5min":2.1539999999999999,"15min":2.5049999999999999},"min":{"1min":0.32800000000000001,"5min":0.24299999999999999,"15min":0.20699999999999999},"max":{"1min":5.3860000000000001,"5min":56.027000000000001,"15min":227.70500000000001},"last":0.61399999999999999}]},{"osd":6,"last update":"Tue Sep 10 19:28:24 2024","interfaces":[{"interface":"back","average":{"1min":0.79600000000000004,"5min":1.6399999999999999,"15min":1.2410000000000001},"min":{"1min":0.32100000000000001,"5min":0.32100000000000001,"15min":0.253},"max":{"1min":2.073,"5min":8.1010000000000009,"15min":8.1010000000000009},"last":0.53200000000000003},{"interface":"front","average":{"1min":0.88900000000000001,"5min":1.621,"15min":1.262},"min":{"1min":0.29499999999999998,"5min":0.26400000000000001,"15min":0.247},"max":{"1min":3.3690000000000002,"5min":16.815000000000001,"15min":16.815000000000001},"last":1.222}]},{"osd":7,"last update":"Tue Sep 10 19:28:29 2024","interfaces":[{"interface":"back","average":{"1min":0.90900000000000003,"5min":1.2549999999999999,"15min":1.3779999999999999},"min":{"1min":0.34499999999999997,"5min":0.33100000000000002,"15min":0.28100000000000003},"max":{"1min":2.7480000000000002,"5min":16.853999999999999,"15min":23.081},"last":1.468},{"interface":"front","average":{"1min":0.72699999999999998,"5min":1.48,"15min":1.4059999999999999},"min":{"1min":0.30499999999999999,"5min":0.26100000000000001,"15min":0.23699999999999999},"max":{"1min":2.1509999999999998,"5min":10.02,"15min":22.666},"last":0.57399999999999995}]}]},{"osd":3,"up_from":26,"seq":111669149941,"num_pgs":73,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3068876,"kb_used_data":2181060,"kb_used_omap":1,"kb_used_meta":887806,"kb_avail":90676276,"statfs":{"total":95995035648,"available":92852506624,"internally_reserved":0,"allocated":2233405440,"data_stored":2232983027,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":909113688},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 19:28:31 2024","interfaces":[{"interface":"back","average":{"1min":0.76300000000000001,"5min":0.68600000000000005,"15min":0.58399999999999996},"min":{"1min":0.253,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":2.2559999999999998,"5min":11.942,"15min":11.942},"last":1.24},{"interface":"front","average":{"1min":0.78200000000000003,"5min":0.68100000000000005,"15min":0.55900000000000005},"min":{"1min":0.25,"5min":0.184,"15min":0.184},"max":{"1min":2.9220000000000002,"5min":12.225,"15min":12.225},"last":0.33800000000000002}]},{"osd":1,"last update":"Tue Sep 10 19:27:59 2024","interfaces":[{"interface":"back","average":{"1min":0.52800000000000002,"5min":0.42199999999999999,"15min":0.33600000000000002},"min":{"1min":0.19500000000000001,"5min":0.185,"15min":0.185},"max":{"1min":2.7759999999999998,"5min":2.7759999999999998,"15min":2.7759999999999998},"last":1.212},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.42899999999999999,"15min":0.35199999999999998},"min":{"1min":0.216,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":4.4100000000000001,"5min":4.4100000000000001,"15min":4.4100000000000001},"last":1.583}]},{"osd":2,"last update":"Tue Sep 10 19:28:06 2024","interfaces":[{"interface":"back","average":{"1min":0.71199999999999997,"5min":0.45800000000000002,"15min":0.65700000000000003},"min":{"1min":0.309,"5min":0.23000000000000001,"15min":0.17599999999999999},"max":{"1min":4.3769999999999998,"5min":4.3769999999999998,"15min":5.3029999999999999},"last":0.40699999999999997},{"interface":"front","average":{"1min":0.46200000000000002,"5min":0.47499999999999998,"15min":0.72399999999999998},"min":{"1min":0.26400000000000001,"5min":0.20100000000000001,"15min":0.16400000000000001},"max":{"1min":1.234,"5min":2.944,"15min":17.806000000000001},"last":0.45400000000000001}]},{"osd":4,"last update":"Tue Sep 10 19:28:23 2024","interfaces":[{"interface":"back","average":{"1min":1.0549999999999999,"5min":2.101,"15min":1.391},"min":{"1min":0.373,"5min":0.26900000000000002,"15min":0.23599999999999999},"max":{"1min":4.4169999999999998,"5min":81.334000000000003,"15min":81.334000000000003},"last":0.435},{"interface":"front","average":{"1min":0.97899999999999998,"5min":2.194,"15min":1.5489999999999999},"min":{"1min":0.40200000000000002,"5min":0.27200000000000002,"15min":0.22700000000000001},"max":{"1min":4.343,"5min":81.381,"15min":81.381},"last":1.167}]},{"osd":5,"last update":"Tue Sep 10 19:27:43 2024","interfaces":[{"interface":"back","average":{"1min":1.012,"5min":1.96,"15min":1.756},"min":{"1min":0.36299999999999999,"5min":0.28100000000000003,"15min":0.27900000000000003},"max":{"1min":4.7830000000000004,"5min":104.672,"15min":104.672},"last":0.51900000000000002},{"interface":"front","average":{"1min":1.1439999999999999,"5min":1.871,"15min":1.825},"min":{"1min":0.28799999999999998,"5min":0.28799999999999998,"15min":0.22},"max":{"1min":4.6550000000000002,"5min":105.971,"15min":105.971},"last":1.2629999999999999}]},{"osd":6,"last update":"Tue Sep 10 19:28:06 2024","interfaces":[{"interface":"back","average":{"1min":1.0509999999999999,"5min":0.999,"15min":1.6890000000000001},"min":{"1min":0.33600000000000002,"5min":0.28000000000000003,"15min":0.23699999999999999},"max":{"1min":3.4550000000000001,"5min":5.2480000000000002,"15min":116.14},"last":0.47599999999999998},{"interface":"front","average":{"1min":0.96699999999999997,"5min":0.82999999999999996,"15min":1.532},"min":{"1min":0.32700000000000001,"5min":0.29299999999999998,"15min":0.26800000000000002},"max":{"1min":3.7029999999999998,"5min":4.4370000000000003,"15min":116.095},"last":0.37}]},{"osd":7,"last update":"Tue Sep 10 19:27:38 2024","interfaces":[{"interface":"back","average":{"1min":0.91900000000000004,"5min":1.867,"15min":1.45},"min":{"1min":0.371,"5min":0.314,"15min":0.26100000000000001},"max":{"1min":4.46,"5min":104.749,"15min":104.749},"last":0.59099999999999997},{"interface":"front","average":{"1min":0.68100000000000005,"5min":1.79,"15min":1.4379999999999999},"min":{"1min":0.33500000000000002,"5min":0.317,"15min":0.26100000000000001},"max":{"1min":1.4750000000000001,"5min":104.69499999999999,"15min":104.69499999999999},"last":0.5}]}]},{"osd":2,"up_from":0,"seq":77309411578,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":276,"seq":1185410973768,"num_pgs":70,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":3541584,"kb_used_data":3219080,"kb_used_omap":6,"kb_used_meta":322489,"kb_avail":90203568,"statfs":{"total":95995035648,"available":92368453632,"internally_reserved":0,"allocated":3296337920,"data_stored":3295918089,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6335,"internal_metadata":330229569},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 19:28:35 2024","interfaces":[{"interface":"back","average":{"1min":0.85799999999999998,"5min":0.57399999999999995,"15min":0.51600000000000001},"min":{"1min":0.19800000000000001,"5min":0.16700000000000001,"15min":0.16700000000000001},"max":{"1min":3.5979999999999999,"5min":3.5979999999999999,"15min":3.5979999999999999},"last":0.19800000000000001},{"interface":"front","average":{"1min":0.86699999999999999,"5min":0.55000000000000004,"15min":0.45500000000000002},"min":{"1min":0.19400000000000001,"5min":0.19400000000000001,"15min":0.19400000000000001},"max":{"1min":3.8570000000000002,"5min":3.8570000000000002,"15min":3.8570000000000002},"last":0.216}]},{"osd":2,"last update":"Tue Sep 10 19:27:55 2024","interfaces":[{"interface":"back","average":{"1min":0.45400000000000001,"5min":0.435,"15min":0.33700000000000002},"min":{"1min":0.254,"5min":0.19,"15min":0.19},"max":{"1min":1.0109999999999999,"5min":1.837,"15min":1.837},"last":0.25600000000000001},{"interface":"front","average":{"1min":0.40400000000000003,"5min":0.433,"15min":0.36699999999999999},"min":{"1min":0.27900000000000003,"5min":0.222,"15min":0.222},"max":{"1min":0.629,"5min":1.7909999999999999,"15min":1.7909999999999999},"last":0.27000000000000002}]},{"osd":3,"last update":"Tue Sep 10 19:27:55 2024","interfaces":[{"interface":"back","average":{"1min":0.47399999999999998,"5min":0.42099999999999999,"15min":0.34300000000000003},"min":{"1min":0.22800000000000001,"5min":0.221,"15min":0.221},"max":{"1min":1.2310000000000001,"5min":1.573,"15min":1.573},"last":5.0999999999999996},{"interface":"front","average":{"1min":0.42999999999999999,"5min":0.38500000000000001,"15min":0.32900000000000001},"min":{"1min":0.23200000000000001,"5min":0.17599999999999999,"15min":0.17599999999999999},"max":{"1min":1.5980000000000001,"5min":1.984,"15min":1.984},"last":5.141}]},{"osd":4,"last update":"Tue Sep 10 19:27:55 2024","interfaces":[{"interface":"back","average":{"1min":0.96699999999999997,"5min":0.95799999999999996,"15min":0.67400000000000004},"min":{"1min":0.28199999999999997,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":5.0099999999999998,"5min":5.0099999999999998,"15min":5.0099999999999998},"last":0.52200000000000002},{"interface":"front","average":{"1min":0.60099999999999998,"5min":0.92600000000000005,"15min":0.71899999999999997},"min":{"1min":0.31,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":1.0660000000000001,"5min":5.7599999999999998,"15min":5.7599999999999998},"last":0.64800000000000002}]},{"osd":5,"last update":"Tue Sep 10 19:27:55 2024","interfaces":[{"interface":"back","average":{"1min":0.66000000000000003,"5min":0.89000000000000001,"15min":0.95899999999999996},"min":{"1min":0.33000000000000002,"5min":0.307,"15min":0.307},"max":{"1min":1.915,"5min":4.7519999999999998,"15min":4.7519999999999998},"last":0.66800000000000004},{"interface":"front","average":{"1min":0.67900000000000005,"5min":1.079,"15min":1.3149999999999999},"min":{"1min":0.313,"5min":0.313,"15min":0.313},"max":{"1min":2.2410000000000001,"5min":5.617,"15min":5.617},"last":0.628}]},{"osd":6,"last update":"Tue Sep 10 19:27:55 2024","interfaces":[{"interface":"back","average":{"1min":0.70199999999999996,"5min":1.3420000000000001,"15min":2.0099999999999998},"min":{"1min":0.40799999999999997,"5min":0.31,"15min":0.31},"max":{"1min":2.0179999999999998,"5min":27.379999999999999,"15min":27.379999999999999},"last":0.65700000000000003},{"interface":"front","average":{"1min":0.79400000000000004,"5min":1.3049999999999999,"15min":1.9950000000000001},"min":{"1min":0.313,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":1.978,"5min":26.812999999999999,"15min":26.812999999999999},"last":0.54400000000000004}]},{"osd":7,"last update":"Tue Sep 10 19:27:55 2024","interfaces":[{"interface":"back","average":{"1min":0.95099999999999996,"5min":0.97899999999999998,"15min":1.016},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":8.6039999999999992,"5min":8.6039999999999992,"15min":8.6039999999999992},"last":0.60799999999999998},{"interface":"front","average":{"1min":0.91400000000000003,"5min":0.97699999999999998,"15min":1.0229999999999999},"min":{"1min":0.33600000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":8.5899999999999999,"5min":8.5899999999999999,"15min":8.5899999999999999},"last":0.63900000000000001}]}]},{"osd":0,"up_from":301,"seq":1292785156148,"num_pgs":73,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2362956,"kb_used_data":2008768,"kb_used_omap":4,"kb_used_meta":354171,"kb_avail":91382196,"statfs":{"total":95995035648,"available":93575368704,"internally_reserved":0,"allocated":2056978432,"data_stored":2056567194,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":362671622},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Sep 10 19:28:29 2024","interfaces":[{"interface":"back","average":{"1min":0.82099999999999995,"5min":0.53600000000000003,"15min":0.56100000000000005},"min":{"1min":0.223,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":5.3659999999999997,"5min":5.3659999999999997,"15min":5.3659999999999997},"last":0.88700000000000001},{"interface":"front","average":{"1min":0.55300000000000005,"5min":0.45000000000000001,"15min":0.47899999999999998},"min":{"1min":0.23400000000000001,"5min":0.16900000000000001,"15min":0.16900000000000001},"max":{"1min":1.6930000000000001,"5min":1.6930000000000001,"15min":1.6930000000000001},"last":0.79900000000000004}]},{"osd":2,"last update":"Tue Sep 10 19:28:29 2024","interfaces":[{"interface":"back","average":{"1min":0.48399999999999999,"5min":0.51100000000000001,"15min":0.57199999999999995},"min":{"1min":0.22600000000000001,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":1.242,"5min":1.734,"15min":1.734},"last":0.56799999999999995},{"interface":"front","average":{"1min":0.46500000000000002,"5min":0.52800000000000002,"15min":0.59899999999999998},"min":{"1min":0.25700000000000001,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":1.1140000000000001,"5min":1.5580000000000001,"15min":1.5580000000000001},"last":0.61799999999999999}]},{"osd":3,"last update":"Tue Sep 10 19:28:29 2024","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.52000000000000002,"15min":0.56599999999999995},"min":{"1min":0.219,"5min":0.20599999999999999,"15min":0.20599999999999999},"max":{"1min":1.242,"5min":1.242,"15min":1.242},"last":0.77900000000000003},{"interface":"front","average":{"1min":0.55100000000000005,"5min":0.57599999999999996,"15min":0.60799999999999998},"min":{"1min":0.28999999999999998,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":1.21,"5min":1.677,"15min":1.677},"last":1.0760000000000001}]},{"osd":4,"last update":"Tue Sep 10 19:28:29 2024","interfaces":[{"interface":"back","average":{"1min":0.63400000000000001,"5min":2.4950000000000001,"15min":3.8340000000000001},"min":{"1min":0.32700000000000001,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":1.454,"5min":53.670000000000002,"15min":53.670000000000002},"last":1.76},{"interface":"front","average":{"1min":0.66900000000000004,"5min":2.468,"15min":3.8300000000000001},"min":{"1min":0.311,"5min":0.311,"15min":0.311},"max":{"1min":1.3149999999999999,"5min":53.935000000000002,"15min":53.935000000000002},"last":0.63800000000000001}]},{"osd":5,"last update":"Tue Sep 10 19:28:29 2024","interfaces":[{"interface":"back","average":{"1min":0.70499999999999996,"5min":2.1720000000000002,"15min":3.444},"min":{"1min":0.25700000000000001,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":2.4039999999999999,"5min":53.884999999999998,"15min":53.884999999999998},"last":0.42799999999999999},{"interface":"front","average":{"1min":0.68600000000000005,"5min":2.157,"15min":3.343},"min":{"1min":0.34899999999999998,"5min":0.309,"15min":0.309},"max":{"1min":2.2909999999999999,"5min":53.875999999999998,"15min":53.875999999999998},"last":0.68700000000000006}]},{"osd":6,"last update":"Tue Sep 10 19:28:29 2024","interfaces":[{"interface":"back","average":{"1min":0.873,"5min":2.5619999999999998,"15min":4.0730000000000004},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":4.0250000000000004,"5min":56.901000000000003,"15min":56.901000000000003},"last":0.66100000000000003},{"interface":"front","average":{"1min":0.77400000000000002,"5min":2.5209999999999999,"15min":4.0069999999999997},"min":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":1.7010000000000001,"5min":55.177,"15min":55.177},"last":0.59699999999999998}]},{"osd":7,"last update":"Tue Sep 10 19:28:29 2024","interfaces":[{"interface":"back","average":{"1min":0.78900000000000003,"5min":1.139,"15min":1.204},"min":{"1min":0.29499999999999998,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":2.0459999999999998,"5min":4.0670000000000002,"15min":4.0670000000000002},"last":0.878},{"interface":"front","average":{"1min":0.72199999999999998,"5min":1.0640000000000001,"15min":1.1930000000000001},"min":{"1min":0.39100000000000001,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":1.792,"5min":3.2519999999999998,"15min":3.2519999999999998},"last":1.0600000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":966,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1236,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1236,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":362,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1540,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":2118,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1018,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1502,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":2004615168,"data_stored":2004615168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":3350265856,"data_stored":3350265856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":2269904896,"data_stored":2269904896,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":1623457792,"data_stored":1623457792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":1312686080,"data_stored":1312686080,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":2575564800,"data_stored":2575564800,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":2605842432,"data_stored":2605842432,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T19:28:39.081 INFO:tasks.thrashosds.thrasher:Setting 2.14 to [0, 3, 7, 2, 5, 6] 2024-09-10T19:28:39.081 INFO:tasks.thrashosds.thrasher:cmd ['osd', 'pg-upmap-items', '2.14', '0', '3', '7', '2', '5', '6'] 2024-09-10T19:28:39.081 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph osd pg-upmap-items 2.14 0 3 7 2 5 6 2024-09-10T19:28:39.087 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:28:39.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:28:39.090 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:28:39.190 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:28:39.220 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:28:39.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:28:39.221 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:28:39.319 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:28:39 smithi183 ceph-mon[29736]: pgmap v853: 158 pgs: 158 active+clean; 4.9 GiB data, 19 GiB used, 606 GiB / 626 GiB avail; 1.2 KiB/s rd, 102 MiB/s wr, 1.63k op/s 2024-09-10T19:28:39.319 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:28:39 smithi183 ceph-mon[29736]: from='client.25726 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-10T19:28:39.319 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:28:39 smithi183 ceph-mon[29736]: purged_snaps scrub starts 2024-09-10T19:28:39.319 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:28:39 smithi183 ceph-mon[29736]: purged_snaps scrub ok 2024-09-10T19:28:39.320 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:28:39 smithi183 ceph-mon[41562]: pgmap v853: 158 pgs: 158 active+clean; 4.9 GiB data, 19 GiB used, 606 GiB / 626 GiB avail; 1.2 KiB/s rd, 102 MiB/s wr, 1.63k op/s 2024-09-10T19:28:39.320 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:28:39 smithi183 ceph-mon[41562]: from='client.25726 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-10T19:28:39.321 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:28:39 smithi183 ceph-mon[41562]: purged_snaps scrub starts 2024-09-10T19:28:39.321 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:28:39 smithi183 ceph-mon[41562]: purged_snaps scrub ok 2024-09-10T19:28:39.322 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:28:39.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:28:39.349 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:28:39.349 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:28:39.450 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:28:39.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:28:39 smithi191 ceph-mon[32884]: pgmap v853: 158 pgs: 158 active+clean; 4.9 GiB data, 19 GiB used, 606 GiB / 626 GiB avail; 1.2 KiB/s rd, 102 MiB/s wr, 1.63k op/s 2024-09-10T19:28:39.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:28:39 smithi191 ceph-mon[32884]: from='client.25726 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-10T19:28:39.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:28:39 smithi191 ceph-mon[32884]: purged_snaps scrub starts 2024-09-10T19:28:39.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:28:39 smithi191 ceph-mon[32884]: purged_snaps scrub ok 2024-09-10T19:28:39.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:28:39.485 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:28:39.485 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:28:39.586 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:28:39.625 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:28:39.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:28:39.626 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:28:39.726 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:28:39.765 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:28:39.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:28:39.766 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:28:39.867 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:28:39.898 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:28:39.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:28:39.899 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:28:40.000 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:28:40.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:28:40.039 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:28:40.039 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:28:40.066 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: 46 16 85900 85884 116.666 83 0.00444982 0.0085669 2024-09-10T19:28:40.092 INFO:journalctl@ceph.iscsi.iscsi.a.smithi191.stdout:Sep 10 19:28:39 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-iscsi-iscsi-a[61816]: debug there is no tcmu-runner data available 2024-09-10T19:28:40.140 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:28:40.170 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:28:40.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:28:40.171 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:28:40.272 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:28:40.304 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:28:40.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:28:40.305 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:28:40.406 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:28:40.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:28:40 smithi183 ceph-mon[29736]: osdmap e387: 8 total, 8 up, 7 in 2024-09-10T19:28:40.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:28:40 smithi183 ceph-mon[29736]: osdmap e388: 8 total, 8 up, 7 in 2024-09-10T19:28:40.431 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:28:40 smithi183 ceph-mon[41562]: osdmap e387: 8 total, 8 up, 7 in 2024-09-10T19:28:40.431 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:28:40 smithi183 ceph-mon[41562]: osdmap e388: 8 total, 8 up, 7 in 2024-09-10T19:28:40.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:28:40.443 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:28:40.443 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:28:40.465 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:28:40 smithi191 ceph-mon[32884]: osdmap e387: 8 total, 8 up, 7 in 2024-09-10T19:28:40.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:28:40 smithi191 ceph-mon[32884]: osdmap e388: 8 total, 8 up, 7 in 2024-09-10T19:28:40.544 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:28:40.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:28:40.583 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:28:40.583 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:28:40.684 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:28:40.714 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:28:40.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:28:40.715 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:28:40.815 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:28:40.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:28:40.856 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:28:40.856 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:28:40.957 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:28:40.997 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:28:40.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:28:40.998 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:28:41.067 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: 47 15 87385 87370 116.159 92.875 0.00825091 0.00860436 2024-09-10T19:28:41.099 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:28:41.134 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:28:41.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:28:41.135 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:28:41.236 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:28:41.267 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:28:41 smithi183 ceph-mon[41562]: pgmap v855: 158 pgs: 158 active+clean; 4.9 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 1.2 KiB/s rd, 73 MiB/s wr, 1.16k op/s 2024-09-10T19:28:41.267 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:28:41 smithi183 ceph-mon[41562]: from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:28:41.267 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:28:41 smithi183 ceph-mon[29736]: pgmap v855: 158 pgs: 158 active+clean; 4.9 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 1.2 KiB/s rd, 73 MiB/s wr, 1.16k op/s 2024-09-10T19:28:41.267 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:28:41 smithi183 ceph-mon[29736]: from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:28:41.269 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:28:41.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:28:41.271 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:28:41.371 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:28:41.411 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:28:41.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:28:41.414 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:28:41.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:28:41 smithi191 ceph-mon[32884]: pgmap v855: 158 pgs: 158 active+clean; 4.9 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 1.2 KiB/s rd, 73 MiB/s wr, 1.16k op/s 2024-09-10T19:28:41.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:28:41 smithi191 ceph-mon[32884]: from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:28:41.514 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:28:41.525 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Sep 10 19:28:41 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-y[29989]: ::ffff:172.21.15.191 - - [10/Sep/2024:19:28:41] "GET /metrics HTTP/1.1" 200 224855 "" "Prometheus/2.43.0" 2024-09-10T19:28:41.548 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:28:41.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:28:41.549 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:28:41.650 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:28:41.683 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:28:41.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:28:41.684 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:28:41.785 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:28:41.819 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:28:41.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:28:41.820 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:28:41.921 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:28:41.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:28:41.965 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:28:41.965 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:28:42.022 INFO:teuthology.orchestra.run.smithi183.stdout:ERROR: (22) Invalid argument 2024-09-10T19:28:42.022 INFO:teuthology.orchestra.run.smithi183.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-10T19:28:42.036 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:28:42.066 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:28:42.067 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: 48 16 88751 88735 115.516 85.3125 0.0240653 0.00865222 2024-09-10T19:28:42.101 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:28:42.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:28:42.102 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:28:42.202 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:28:42.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:28:42.239 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:28:42.239 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:28:42.318 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:28:42.340 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:28:42.368 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:28:42.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:28:42.369 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:28:42.470 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:28:42.504 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:28:42.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:28:42.505 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:28:42.606 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:28:42.642 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:28:42.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:28:42.643 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:28:42.744 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:28:42.776 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:28:42.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:28:42.777 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:28:42.878 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:28:42.922 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:28:42.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:28:42.923 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:28:42.950 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph daemon osd.0 dump_blocked_ops 2024-09-10T19:28:43.024 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:28:43.068 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: 49 16 90160 90144 114.955 88.0625 0.0112934 0.00869409 2024-09-10T19:28:43.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:28:43.077 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:28:43.078 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:28:43.140 INFO:teuthology.orchestra.run.smithi183.stderr:nodeep-scrub is set 2024-09-10T19:28:43.165 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:28:43 smithi183 ceph-mon[29736]: pgmap v857: 158 pgs: 158 active+clean; 5.2 GiB data, 20 GiB used, 605 GiB / 626 GiB avail; 733 B/s rd, 106 MiB/s wr, 1.70k op/s 2024-09-10T19:28:43.166 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:28:43 smithi183 ceph-mon[29736]: from='client.? 172.21.15.183:0/3600257260' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-10T19:28:43.166 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:28:43 smithi183 ceph-mon[29736]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-10T19:28:43.166 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:28:43 smithi183 ceph-mon[29736]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T19:28:43.167 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:28:43 smithi183 ceph-mon[41562]: pgmap v857: 158 pgs: 158 active+clean; 5.2 GiB data, 20 GiB used, 605 GiB / 626 GiB avail; 733 B/s rd, 106 MiB/s wr, 1.70k op/s 2024-09-10T19:28:43.167 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:28:43 smithi183 ceph-mon[41562]: from='client.? 172.21.15.183:0/3600257260' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-10T19:28:43.167 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:28:43 smithi183 ceph-mon[41562]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-10T19:28:43.167 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:28:43 smithi183 ceph-mon[41562]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T19:28:43.178 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:28:43.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:28:43.228 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:28:43.228 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:28:43.329 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:28:43.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:28:43.370 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:28:43.370 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:28:43.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:28:43 smithi191 ceph-mon[32884]: pgmap v857: 158 pgs: 158 active+clean; 5.2 GiB data, 20 GiB used, 605 GiB / 626 GiB avail; 733 B/s rd, 106 MiB/s wr, 1.70k op/s 2024-09-10T19:28:43.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:28:43 smithi191 ceph-mon[32884]: from='client.? 172.21.15.183:0/3600257260' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-10T19:28:43.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:28:43 smithi191 ceph-mon[32884]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-10T19:28:43.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:28:43 smithi191 ceph-mon[32884]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T19:28:43.472 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:28:43.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:28:43.519 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:28:43.519 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:28:43.620 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:28:43.661 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:28:43.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:28:43.662 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:28:43.763 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:28:43.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:28:43.800 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:28:43.800 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:28:43.901 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:28:43.935 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:28:43.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:28:43.936 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:28:44.037 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:28:44.067 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: 50 16 91480 91464 114.306 82.5 0.00441694 0.00874419 2024-09-10T19:28:44.072 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:28:44.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:28:44.073 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:28:44.174 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:28:44.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:28:44.212 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:28:44.212 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:28:44.313 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:28:44.351 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:28:44.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:28:44.352 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:28:44.396 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:28:44 smithi183 ceph-mon[29736]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T19:28:44.396 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:28:44 smithi183 ceph-mon[29736]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-10T19:28:44.396 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:28:44 smithi183 ceph-mon[29736]: osdmap e389: 8 total, 8 up, 7 in 2024-09-10T19:28:44.396 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:28:44 smithi183 ceph-mon[29736]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T19:28:44.396 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:28:44 smithi183 ceph-mon[29736]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T19:28:44.396 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:28:44 smithi183 ceph-mon[29736]: from='mgr.14556 ' entity='mgr.y' 2024-09-10T19:28:44.396 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:28:44 smithi183 ceph-mon[29736]: from='mgr.14556 ' entity='mgr.y' 2024-09-10T19:28:44.397 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:28:44 smithi183 ceph-mon[29736]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:28:44.397 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:28:44 smithi183 ceph-mon[41562]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T19:28:44.397 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:28:44 smithi183 ceph-mon[41562]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-10T19:28:44.397 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:28:44 smithi183 ceph-mon[41562]: osdmap e389: 8 total, 8 up, 7 in 2024-09-10T19:28:44.397 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:28:44 smithi183 ceph-mon[41562]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T19:28:44.397 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:28:44 smithi183 ceph-mon[41562]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T19:28:44.397 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:28:44 smithi183 ceph-mon[41562]: from='mgr.14556 ' entity='mgr.y' 2024-09-10T19:28:44.397 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:28:44 smithi183 ceph-mon[41562]: from='mgr.14556 ' entity='mgr.y' 2024-09-10T19:28:44.397 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:28:44 smithi183 ceph-mon[41562]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:28:44.453 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:28:44.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:28:44 smithi191 ceph-mon[32884]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T19:28:44.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:28:44 smithi191 ceph-mon[32884]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-10T19:28:44.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:28:44 smithi191 ceph-mon[32884]: osdmap e389: 8 total, 8 up, 7 in 2024-09-10T19:28:44.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:28:44 smithi191 ceph-mon[32884]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T19:28:44.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:28:44 smithi191 ceph-mon[32884]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T19:28:44.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:28:44 smithi191 ceph-mon[32884]: from='mgr.14556 ' entity='mgr.y' 2024-09-10T19:28:44.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:28:44 smithi191 ceph-mon[32884]: from='mgr.14556 ' entity='mgr.y' 2024-09-10T19:28:44.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:28:44 smithi191 ceph-mon[32884]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:28:44.488 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:28:44.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:28:44.489 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:28:44.590 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:28:44.630 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:28:44.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:28:44.631 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:28:44.732 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:28:44.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:28:44.767 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:28:44.767 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:28:44.868 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:28:44.901 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:28:44.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:28:44.902 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:28:45.002 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:28:45.037 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:28:45.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:28:45.038 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:28:45.068 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: 51 16 92938 92922 113.851 91.125 0.0119369 0.008779 2024-09-10T19:28:45.139 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:28:45.201 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:28:45.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:28:45.202 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:28:45.303 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:28:45.338 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:28:45.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:28:45.339 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:28:45.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:28:45 smithi183 ceph-mon[29736]: pgmap v859: 158 pgs: 158 active+clean; 5.3 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 853 B/s rd, 73 MiB/s wr, 1.16k op/s 2024-09-10T19:28:45.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:28:45 smithi183 ceph-mon[29736]: osdmap e390: 8 total, 8 up, 7 in 2024-09-10T19:28:45.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:28:45 smithi183 ceph-mon[41562]: pgmap v859: 158 pgs: 158 active+clean; 5.3 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 853 B/s rd, 73 MiB/s wr, 1.16k op/s 2024-09-10T19:28:45.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:28:45 smithi183 ceph-mon[41562]: osdmap e390: 8 total, 8 up, 7 in 2024-09-10T19:28:45.439 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:28:45.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:28:45 smithi191 ceph-mon[32884]: pgmap v859: 158 pgs: 158 active+clean; 5.3 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 853 B/s rd, 73 MiB/s wr, 1.16k op/s 2024-09-10T19:28:45.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:28:45 smithi191 ceph-mon[32884]: osdmap e390: 8 total, 8 up, 7 in 2024-09-10T19:28:45.476 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:28:45.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:28:45.477 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:28:45.578 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:28:45.612 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:28:45.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:28:45.613 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:28:45.714 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:28:45.749 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:28:45.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:28:45.750 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:28:45.851 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:28:45.884 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:28:45.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:28:45.885 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:28:45.986 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:28:46.023 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:28:46.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:28:46.024 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:28:46.068 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: 52 16 94420 94404 113.443 92.625 0.0100049 0.00881046 2024-09-10T19:28:46.125 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:28:46.162 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:28:46.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:28:46.163 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:28:46.264 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:28:46.306 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:28:46.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:28:46.307 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:28:46.379 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:28:46 smithi183 ceph-mon[29736]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.19", "id": [1, 6]}]: dispatch 2024-09-10T19:28:46.379 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:28:46 smithi183 ceph-mon[29736]: from='mgr.14556 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.19", "id": [1, 6]}]: dispatch 2024-09-10T19:28:46.379 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:28:46 smithi183 ceph-mon[41562]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.19", "id": [1, 6]}]: dispatch 2024-09-10T19:28:46.379 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:28:46 smithi183 ceph-mon[41562]: from='mgr.14556 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.19", "id": [1, 6]}]: dispatch 2024-09-10T19:28:46.408 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:28:46.439 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:28:46.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:28:46.440 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:28:46.447 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:28:46.465 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:28:46 smithi191 ceph-mon[32884]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.19", "id": [1, 6]}]: dispatch 2024-09-10T19:28:46.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:28:46 smithi191 ceph-mon[32884]: from='mgr.14556 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.19", "id": [1, 6]}]: dispatch 2024-09-10T19:28:46.541 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:28:46.572 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:28:46.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:28:46.573 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:28:46.675 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:28:46.731 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:28:46.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:28:46.733 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:28:46.833 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:28:46.872 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:28:46.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:28:46.873 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:28:46.974 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:28:47.012 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:28:47.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:28:47.013 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:28:47.068 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: 53 15 95881 95866 113.026 91.375 0.0060392 0.00884292 2024-09-10T19:28:47.115 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:28:47.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:28:47.154 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:28:47.155 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:28:47.181 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph osd unset noscrub 2024-09-10T19:28:47.228 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:28:47 smithi183 ceph-mon[29736]: pgmap v861: 158 pgs: 158 active+clean; 5.6 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 853 B/s rd, 123 MiB/s wr, 1.98k op/s 2024-09-10T19:28:47.228 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:28:47 smithi183 ceph-mon[29736]: from='mgr.14556 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.19", "id": [1, 6]}]': finished 2024-09-10T19:28:47.228 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:28:47 smithi183 ceph-mon[29736]: osdmap e391: 8 total, 8 up, 7 in 2024-09-10T19:28:47.229 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:28:47 smithi183 ceph-mon[41562]: pgmap v861: 158 pgs: 158 active+clean; 5.6 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 853 B/s rd, 123 MiB/s wr, 1.98k op/s 2024-09-10T19:28:47.229 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:28:47 smithi183 ceph-mon[41562]: from='mgr.14556 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.19", "id": [1, 6]}]': finished 2024-09-10T19:28:47.229 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:28:47 smithi183 ceph-mon[41562]: osdmap e391: 8 total, 8 up, 7 in 2024-09-10T19:28:47.255 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:28:47.301 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:28:47.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:28:47.302 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:28:47.303 INFO:teuthology.orchestra.run.smithi183.stdout:osd.0: {} 2024-09-10T19:28:47.304 INFO:teuthology.orchestra.run.smithi183.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-10T19:28:47.342 INFO:teuthology.orchestra.run.smithi183.stdout:osd.1: {} 2024-09-10T19:28:47.342 INFO:teuthology.orchestra.run.smithi183.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-10T19:28:47.375 INFO:teuthology.orchestra.run.smithi183.stdout:osd.2: {} 2024-09-10T19:28:47.375 INFO:teuthology.orchestra.run.smithi183.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-10T19:28:47.403 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:28:47.408 INFO:teuthology.orchestra.run.smithi183.stdout:osd.3: {} 2024-09-10T19:28:47.408 INFO:teuthology.orchestra.run.smithi183.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-10T19:28:47.440 INFO:teuthology.orchestra.run.smithi183.stdout:osd.4: {} 2024-09-10T19:28:47.440 INFO:teuthology.orchestra.run.smithi183.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-10T19:28:47.450 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:28:47 smithi191 ceph-mon[32884]: pgmap v861: 158 pgs: 158 active+clean; 5.6 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 853 B/s rd, 123 MiB/s wr, 1.98k op/s 2024-09-10T19:28:47.451 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:28:47 smithi191 ceph-mon[32884]: from='mgr.14556 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.19", "id": [1, 6]}]': finished 2024-09-10T19:28:47.451 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:28:47 smithi191 ceph-mon[32884]: osdmap e391: 8 total, 8 up, 7 in 2024-09-10T19:28:47.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:28:47.460 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:28:47.460 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:28:47.475 INFO:teuthology.orchestra.run.smithi183.stdout:osd.5: {} 2024-09-10T19:28:47.476 INFO:teuthology.orchestra.run.smithi183.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-10T19:28:47.522 INFO:teuthology.orchestra.run.smithi183.stdout:osd.6: {} 2024-09-10T19:28:47.523 INFO:teuthology.orchestra.run.smithi183.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-10T19:28:47.561 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:28:47.565 INFO:teuthology.orchestra.run.smithi183.stdout:osd.7: {} 2024-09-10T19:28:47.565 INFO:teuthology.orchestra.run.smithi183.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-10T19:28:47.594 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:28:47.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:28:47.595 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:28:47.696 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:28:47.716 INFO:journalctl@ceph.mgr.x.smithi191.stdout:Sep 10 19:28:47 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-x[34967]: ::ffff:172.21.15.191 - - [10/Sep/2024:19:28:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T19:28:47.733 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:28:47.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:28:47.736 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:28:47.837 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:28:47.870 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:28:47.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:28:47.872 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:28:47.973 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:28:48.000 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:28:48.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:28:48.001 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:28:48.068 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: 54 15 97132 97117 112.381 78.1875 0.0066593 0.00889424 2024-09-10T19:28:48.101 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:28:48.135 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:28:48.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:28:48.137 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:28:48.237 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:28:48.272 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:28:48.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:28:48.273 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:28:48.374 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:28:48.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:28:48.423 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:28:48.423 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:28:48.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:28:48 smithi183 ceph-mon[29736]: osdmap e392: 8 total, 8 up, 7 in 2024-09-10T19:28:48.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:28:48 smithi183 ceph-mon[29736]: from='client.? 172.21.15.183:0/4058160007' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T19:28:48.431 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:28:48 smithi183 ceph-mon[41562]: osdmap e392: 8 total, 8 up, 7 in 2024-09-10T19:28:48.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:28:48 smithi183 ceph-mon[41562]: from='client.? 172.21.15.183:0/4058160007' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T19:28:48.465 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:28:48 smithi191 ceph-mon[32884]: osdmap e392: 8 total, 8 up, 7 in 2024-09-10T19:28:48.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:28:48 smithi191 ceph-mon[32884]: from='client.? 172.21.15.183:0/4058160007' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T19:28:48.524 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:28:48.553 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:28:48.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:28:48.554 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:28:48.655 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:28:48.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:28:48.691 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:28:48.692 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:28:48.793 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:28:48.833 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:28:48.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:28:48.834 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:28:48.935 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:28:48.973 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:28:48.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:28:48.974 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:28:49.068 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: 55 16 98564 98548 111.964 89.4375 0.00844852 0.00892753 2024-09-10T19:28:49.075 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:28:49.117 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:28:49.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:28:49.118 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:28:49.175 INFO:teuthology.orchestra.run.smithi183.stderr:set 2.14 pg_upmap_items mapping to [0->3,7->2,5->6] 2024-09-10T19:28:49.218 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:28:49.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:28:49.252 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:28:49.252 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:28:49.353 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:28:49.388 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:28:49.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:28:49.389 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:28:49.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:28:49 smithi183 ceph-mon[29736]: pgmap v864: 158 pgs: 158 active+clean; 5.7 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 1.0 KiB/s rd, 91 MiB/s wr, 1.46k op/s 2024-09-10T19:28:49.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:28:49 smithi183 ceph-mon[29736]: from='client.? 172.21.15.183:0/605554742' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "2.14", "id": [0, 3, 7, 2, 5, 6]}]: dispatch 2024-09-10T19:28:49.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:28:49 smithi183 ceph-mon[29736]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "2.14", "id": [0, 3, 7, 2, 5, 6]}]: dispatch 2024-09-10T19:28:49.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:28:49 smithi183 ceph-mon[41562]: pgmap v864: 158 pgs: 158 active+clean; 5.7 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 1.0 KiB/s rd, 91 MiB/s wr, 1.46k op/s 2024-09-10T19:28:49.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:28:49 smithi183 ceph-mon[41562]: from='client.? 172.21.15.183:0/605554742' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "2.14", "id": [0, 3, 7, 2, 5, 6]}]: dispatch 2024-09-10T19:28:49.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:28:49 smithi183 ceph-mon[41562]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "2.14", "id": [0, 3, 7, 2, 5, 6]}]: dispatch 2024-09-10T19:28:49.465 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:28:49 smithi191 ceph-mon[32884]: pgmap v864: 158 pgs: 158 active+clean; 5.7 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 1.0 KiB/s rd, 91 MiB/s wr, 1.46k op/s 2024-09-10T19:28:49.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:28:49 smithi191 ceph-mon[32884]: from='client.? 172.21.15.183:0/605554742' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "2.14", "id": [0, 3, 7, 2, 5, 6]}]: dispatch 2024-09-10T19:28:49.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:28:49 smithi191 ceph-mon[32884]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "2.14", "id": [0, 3, 7, 2, 5, 6]}]: dispatch 2024-09-10T19:28:49.490 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:28:49.527 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:28:49.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:28:49.528 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:28:49.629 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:28:49.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:28:49.673 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:28:49.674 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:28:49.774 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:28:49.808 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:28:49.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:28:49.809 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:28:49.910 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:28:49.951 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:28:49.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:28:49.952 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:28:50.053 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:28:50.068 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: 56 16 100001 99985 111.568 89.8125 0.00680433 0.00895936 2024-09-10T19:28:50.085 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:28:50.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:28:50.086 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:28:50.188 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:28:50.228 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:28:50.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:28:50.229 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:28:50.330 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:28:50.335 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:28:50 smithi183 ceph-mon[41562]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "2.14", "id": [0, 3, 7, 2, 5, 6]}]': finished 2024-09-10T19:28:50.335 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:28:50 smithi183 ceph-mon[41562]: osdmap e393: 8 total, 8 up, 7 in 2024-09-10T19:28:50.335 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:28:50 smithi183 ceph-mon[41562]: osdmap e394: 8 total, 8 up, 7 in 2024-09-10T19:28:50.335 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:28:50 smithi183 ceph-mon[29736]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "2.14", "id": [0, 3, 7, 2, 5, 6]}]': finished 2024-09-10T19:28:50.335 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:28:50 smithi183 ceph-mon[29736]: osdmap e393: 8 total, 8 up, 7 in 2024-09-10T19:28:50.335 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:28:50 smithi183 ceph-mon[29736]: osdmap e394: 8 total, 8 up, 7 in 2024-09-10T19:28:50.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:28:50.368 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:28:50.368 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:28:50.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:28:50 smithi191 ceph-mon[32884]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "2.14", "id": [0, 3, 7, 2, 5, 6]}]': finished 2024-09-10T19:28:50.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:28:50 smithi191 ceph-mon[32884]: osdmap e393: 8 total, 8 up, 7 in 2024-09-10T19:28:50.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:28:50 smithi191 ceph-mon[32884]: osdmap e394: 8 total, 8 up, 7 in 2024-09-10T19:28:50.466 INFO:journalctl@ceph.iscsi.iscsi.a.smithi191.stdout:Sep 10 19:28:49 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-iscsi-iscsi-a[61816]: debug there is no tcmu-runner data available 2024-09-10T19:28:50.469 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:28:50.504 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:28:50.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:28:50.505 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:28:50.606 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:28:50.643 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:28:50.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:28:50.644 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:28:50.745 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:28:50.783 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:28:50.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:28:50.784 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:28:50.885 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:28:50.916 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:28:50.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:28:50.917 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:28:51.018 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:28:51.050 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:28:51.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:28:51.051 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:28:51.068 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: 57 16 101468 101452 111.219 91.6875 0.00783271 0.00898695 2024-09-10T19:28:51.152 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:28:51.191 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:28:51.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:28:51.192 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:28:51.240 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:28:51 smithi183 ceph-mon[29736]: pgmap v866: 158 pgs: 158 active+clean; 5.8 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 1.0 KiB/s rd, 88 MiB/s wr, 1.40k op/s 2024-09-10T19:28:51.240 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:28:51 smithi183 ceph-mon[29736]: from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:28:51.240 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:28:51 smithi183 ceph-mon[41562]: pgmap v866: 158 pgs: 158 active+clean; 5.8 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 1.0 KiB/s rd, 88 MiB/s wr, 1.40k op/s 2024-09-10T19:28:51.240 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:28:51 smithi183 ceph-mon[41562]: from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:28:51.293 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:28:51.299 INFO:teuthology.orchestra.run.smithi183.stdout:{ 2024-09-10T19:28:51.299 INFO:teuthology.orchestra.run.smithi183.stdout: "ops": [], 2024-09-10T19:28:51.299 INFO:teuthology.orchestra.run.smithi183.stdout: "complaint_time": 30, 2024-09-10T19:28:51.299 INFO:teuthology.orchestra.run.smithi183.stdout: "num_blocked_ops": 0 2024-09-10T19:28:51.299 INFO:teuthology.orchestra.run.smithi183.stdout:} 2024-09-10T19:28:51.351 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:28:51.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:28:51.352 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:28:51.453 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:28:51.465 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:28:51 smithi191 ceph-mon[32884]: pgmap v866: 158 pgs: 158 active+clean; 5.8 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 1.0 KiB/s rd, 88 MiB/s wr, 1.40k op/s 2024-09-10T19:28:51.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:28:51 smithi191 ceph-mon[32884]: from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:28:51.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:28:51.490 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:28:51.490 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:28:51.591 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:28:51.623 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:28:51.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:28:51.624 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:28:51.662 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Sep 10 19:28:51 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-y[29989]: ::ffff:172.21.15.191 - - [10/Sep/2024:19:28:51] "GET /metrics HTTP/1.1" 200 224873 "" "Prometheus/2.43.0" 2024-09-10T19:28:51.725 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:28:51.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:28:51.763 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:28:51.764 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:28:51.864 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:28:51.892 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:28:51.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:28:51.893 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:28:51.994 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:28:52.035 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:28:52.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:28:52.036 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:28:52.057 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-10T19:28:52.069 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: 58 16 102813 102797 110.75 84.0625 0.00564846 0.00902497 2024-09-10T19:28:52.137 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:28:52.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:28:52.174 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:28:52.174 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:28:52.275 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:28:52.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:28:52.317 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:28:52.317 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:28:52.418 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:28:52.453 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:28:52.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:28:52.454 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:28:52.556 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:28:52.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:28:52.597 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:28:52.597 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:28:52.698 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:28:52.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:28:52.723 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:28:52.724 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:28:52.825 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:28:52.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:28:52.861 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:28:52.861 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:28:52.964 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:28:52.993 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:28:52.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:28:52.994 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:28:53.069 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: 59 16 104023 104007 110.155 75.625 0.0113226 0.00907325 2024-09-10T19:28:53.095 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:28:53.130 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:28:53.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:28:53.131 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:28:53.232 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:28:53.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:28:53.266 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:28:53.267 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:28:53.367 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:28:53.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:28:53.400 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:28:53.400 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:28:53.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:28:53 smithi183 ceph-mon[29736]: purged_snaps scrub starts 2024-09-10T19:28:53.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:28:53 smithi183 ceph-mon[29736]: purged_snaps scrub ok 2024-09-10T19:28:53.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:28:53 smithi183 ceph-mon[29736]: pgmap v868: 158 pgs: 158 active+clean; 6.1 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 880 B/s rd, 76 MiB/s wr, 1.22k op/s 2024-09-10T19:28:53.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:28:53 smithi183 ceph-mon[29736]: osdmap e395: 8 total, 8 up, 7 in 2024-09-10T19:28:53.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:28:53 smithi183 ceph-mon[29736]: purged_snaps scrub starts 2024-09-10T19:28:53.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:28:53 smithi183 ceph-mon[29736]: purged_snaps scrub ok 2024-09-10T19:28:53.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:28:53 smithi183 ceph-mon[41562]: purged_snaps scrub starts 2024-09-10T19:28:53.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:28:53 smithi183 ceph-mon[41562]: purged_snaps scrub ok 2024-09-10T19:28:53.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:28:53 smithi183 ceph-mon[41562]: pgmap v868: 158 pgs: 158 active+clean; 6.1 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 880 B/s rd, 76 MiB/s wr, 1.22k op/s 2024-09-10T19:28:53.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:28:53 smithi183 ceph-mon[41562]: osdmap e395: 8 total, 8 up, 7 in 2024-09-10T19:28:53.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:28:53 smithi183 ceph-mon[41562]: purged_snaps scrub starts 2024-09-10T19:28:53.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:28:53 smithi183 ceph-mon[41562]: purged_snaps scrub ok 2024-09-10T19:28:53.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:28:53 smithi191 ceph-mon[32884]: purged_snaps scrub starts 2024-09-10T19:28:53.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:28:53 smithi191 ceph-mon[32884]: purged_snaps scrub ok 2024-09-10T19:28:53.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:28:53 smithi191 ceph-mon[32884]: pgmap v868: 158 pgs: 158 active+clean; 6.1 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 880 B/s rd, 76 MiB/s wr, 1.22k op/s 2024-09-10T19:28:53.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:28:53 smithi191 ceph-mon[32884]: osdmap e395: 8 total, 8 up, 7 in 2024-09-10T19:28:53.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:28:53 smithi191 ceph-mon[32884]: purged_snaps scrub starts 2024-09-10T19:28:53.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:28:53 smithi191 ceph-mon[32884]: purged_snaps scrub ok 2024-09-10T19:28:53.502 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:28:53.539 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:28:53.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:28:53.540 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:28:53.641 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:28:53.674 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:28:53.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:28:53.675 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:28:53.776 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:28:53.810 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:28:53.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:28:53.811 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:28:53.911 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:28:53.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:28:53.938 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:28:53.938 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:28:54.039 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:28:54.069 INFO:tasks.radosbench.radosbench.0.smithi183.stdout:2024-09-10T19:28:54.068467+0000 min lat: 0.00134808 max lat: 0.165025 avg lat: 0.00911778 2024-09-10T19:28:54.069 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-10T19:28:54.069 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: 60 16 105279 105263 109.627 78.5 0.0140728 0.00911778 2024-09-10T19:28:54.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:28:54.073 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:28:54.074 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:28:54.175 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:28:54.215 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:28:54.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:28:54.216 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:28:54.317 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:28:54.352 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:28:54.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:28:54.353 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:28:54.454 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:28:54.486 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:28:54.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:28:54.487 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:28:54.587 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:28:54.609 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:28:54.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:28:54.610 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:28:54.614 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:28:54 smithi183 ceph-mon[29736]: osdmap e396: 8 total, 8 up, 7 in 2024-09-10T19:28:54.615 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:28:54 smithi183 ceph-mon[41562]: osdmap e396: 8 total, 8 up, 7 in 2024-09-10T19:28:54.621 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:28:54.642 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph daemon osd.0 dump_historic_ops 2024-09-10T19:28:54.710 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:28:54.715 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:28:54 smithi191 ceph-mon[32884]: osdmap e396: 8 total, 8 up, 7 in 2024-09-10T19:28:54.745 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:28:54.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:28:54.747 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:28:54.848 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:28:54.891 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:28:54.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:28:54.892 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:28:54.993 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:28:55.018 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:28:55.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:28:55.019 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:28:55.069 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: 61 16 106387 106371 108.965 69.25 0.00320621 0.00916878 2024-09-10T19:28:55.121 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:28:55.154 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:28:55.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:28:55.155 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:28:55.256 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:28:55.284 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:28:55.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:28:55.286 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:28:55.386 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:28:55.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:28:55.424 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:28:55.424 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:28:55.525 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:28:55.558 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:28:55.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:28:55.559 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:28:55.660 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:28:55.681 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:28:55 smithi183 ceph-mon[29736]: pgmap v871: 158 pgs: 158 active+clean; 6.2 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 1.0 KiB/s rd, 93 MiB/s wr, 1.49k op/s 2024-09-10T19:28:55.681 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:28:55 smithi183 ceph-mon[41562]: pgmap v871: 158 pgs: 158 active+clean; 6.2 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 1.0 KiB/s rd, 93 MiB/s wr, 1.49k op/s 2024-09-10T19:28:55.693 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:28:55.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:28:55.694 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:28:55.715 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:28:55 smithi191 ceph-mon[32884]: pgmap v871: 158 pgs: 158 active+clean; 6.2 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 1.0 KiB/s rd, 93 MiB/s wr, 1.49k op/s 2024-09-10T19:28:55.720 INFO:tasks.thrashosds.thrasher:in_osds: [0, 3, 4, 5, 7, 1, 6] out_osds: [2] dead_osds: [] live_osds: [2, 3, 4, 5, 6, 7, 1, 0] 2024-09-10T19:28:55.720 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph osd reweight 0 1 2024-09-10T19:28:55.796 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:28:55.831 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:28:55.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:28:55.832 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:28:55.933 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:28:55.974 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:28:55.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:28:55.976 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:28:56.070 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: 62 16 107555 107539 108.382 73 0.0112899 0.00922254 2024-09-10T19:28:56.077 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:28:56.113 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:28:56.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:28:56.114 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:28:56.215 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:28:56.243 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:28:56.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:28:56.244 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:28:56.345 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:28:56.383 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:28:56.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:28:56.384 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:28:56.485 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:28:56.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:28:56.514 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:28:56.515 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:28:56.615 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:28:56.658 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:28:56.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:28:56.659 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:28:56.681 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:28:56 smithi183 ceph-mon[29736]: purged_snaps scrub starts 2024-09-10T19:28:56.681 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:28:56 smithi183 ceph-mon[29736]: purged_snaps scrub ok 2024-09-10T19:28:56.682 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:28:56 smithi183 ceph-mon[41562]: purged_snaps scrub starts 2024-09-10T19:28:56.682 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:28:56 smithi183 ceph-mon[41562]: purged_snaps scrub ok 2024-09-10T19:28:56.715 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:28:56 smithi191 ceph-mon[32884]: purged_snaps scrub starts 2024-09-10T19:28:56.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:28:56 smithi191 ceph-mon[32884]: purged_snaps scrub ok 2024-09-10T19:28:56.760 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:28:56.799 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:28:56.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:28:56.800 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:28:56.901 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:28:56.940 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:28:56.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:28:56.941 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:28:57.041 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:28:57.071 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: 63 16 108837 108821 107.934 80.125 0.00479815 0.00926087 2024-09-10T19:28:57.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:28:57.080 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:28:57.081 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:28:57.181 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:28:57.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:28:57.202 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:28:57.203 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:28:57.303 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:28:57.326 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:28:57.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:28:57.327 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:28:57.428 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:28:57.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:28:57.470 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:28:57.470 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:28:57.571 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:28:57.611 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:28:57.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:28:57.612 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:28:57.617 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:28:57 smithi191 ceph-mon[32884]: pgmap v872: 158 pgs: 158 active+clean; 6.5 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 852 B/s rd, 118 MiB/s wr, 1.89k op/s 2024-09-10T19:28:57.617 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:28:57 smithi191 ceph-mon[32884]: osdmap e397: 8 total, 8 up, 7 in 2024-09-10T19:28:57.617 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:28:57 smithi191 ceph-mon[32884]: purged_snaps scrub starts 2024-09-10T19:28:57.617 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:28:57 smithi191 ceph-mon[32884]: purged_snaps scrub ok 2024-09-10T19:28:57.617 INFO:journalctl@ceph.mgr.x.smithi191.stdout:Sep 10 19:28:57 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-x[34967]: ::ffff:172.21.15.191 - - [10/Sep/2024:19:28:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T19:28:57.681 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:28:57 smithi183 ceph-mon[29736]: pgmap v872: 158 pgs: 158 active+clean; 6.5 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 852 B/s rd, 118 MiB/s wr, 1.89k op/s 2024-09-10T19:28:57.682 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:28:57 smithi183 ceph-mon[29736]: osdmap e397: 8 total, 8 up, 7 in 2024-09-10T19:28:57.682 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:28:57 smithi183 ceph-mon[29736]: purged_snaps scrub starts 2024-09-10T19:28:57.682 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:28:57 smithi183 ceph-mon[29736]: purged_snaps scrub ok 2024-09-10T19:28:57.682 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:28:57 smithi183 ceph-mon[41562]: pgmap v872: 158 pgs: 158 active+clean; 6.5 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 852 B/s rd, 118 MiB/s wr, 1.89k op/s 2024-09-10T19:28:57.682 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:28:57 smithi183 ceph-mon[41562]: osdmap e397: 8 total, 8 up, 7 in 2024-09-10T19:28:57.682 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:28:57 smithi183 ceph-mon[41562]: purged_snaps scrub starts 2024-09-10T19:28:57.682 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:28:57 smithi183 ceph-mon[41562]: purged_snaps scrub ok 2024-09-10T19:28:57.713 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:28:57.736 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:28:57.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:28:57.737 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:28:57.838 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:28:57.858 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:28:57.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:28:57.859 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:28:57.960 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:28:57.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:28:57.982 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:28:57.982 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:28:58.071 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: 64 16 109598 109582 106.991 47.5625 0.00343683 0.00930628 2024-09-10T19:28:58.083 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:28:58.104 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:28:58.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:28:58.105 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:28:58.206 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:28:58.228 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:28:58.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:28:58.229 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:28:58.329 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:28:58.356 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:28:58.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:28:58.386 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:28:58.387 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:28:58.400 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:28:58.465 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:28:58 smithi191 ceph-mon[32884]: osdmap e398: 8 total, 8 up, 7 in 2024-09-10T19:28:58.488 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:28:58.503 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:28:58 smithi183 ceph-mon[29736]: osdmap e398: 8 total, 8 up, 7 in 2024-09-10T19:28:58.503 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:28:58 smithi183 ceph-mon[41562]: osdmap e398: 8 total, 8 up, 7 in 2024-09-10T19:28:58.523 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:28:58.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:28:58.524 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:28:58.625 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:28:58.667 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:28:58.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:28:58.669 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:28:58.770 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:28:58.803 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:28:58.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:28:58.805 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:28:58.905 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:28:58.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:28:58.943 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:28:58.943 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:28:59.044 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:28:59.071 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: 65 16 110599 110583 106.307 62.5625 0.0106443 0.00940198 2024-09-10T19:28:59.080 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:28:59.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:28:59.081 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:28:59.182 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:28:59.220 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:28:59.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:28:59.221 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:28:59.322 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:28:59.358 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:28:59.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:28:59.359 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:28:59.460 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:28:59.465 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:28:59 smithi191 ceph-mon[32884]: pgmap v875: 158 pgs: 158 active+clean; 6.5 GiB data, 24 GiB used, 601 GiB / 626 GiB avail; 1.7 KiB/s rd, 77 MiB/s wr, 1.23k op/s 2024-09-10T19:28:59.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:28:59 smithi191 ceph-mon[32884]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:28:59.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:28:59.495 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:28:59.495 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:28:59.596 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:28:59.630 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:28:59.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:28:59.631 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:28:59.681 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:28:59 smithi183 ceph-mon[29736]: pgmap v875: 158 pgs: 158 active+clean; 6.5 GiB data, 24 GiB used, 601 GiB / 626 GiB avail; 1.7 KiB/s rd, 77 MiB/s wr, 1.23k op/s 2024-09-10T19:28:59.681 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:28:59 smithi183 ceph-mon[29736]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:28:59.682 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:28:59 smithi183 ceph-mon[41562]: pgmap v875: 158 pgs: 158 active+clean; 6.5 GiB data, 24 GiB used, 601 GiB / 626 GiB avail; 1.7 KiB/s rd, 77 MiB/s wr, 1.23k op/s 2024-09-10T19:28:59.682 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:28:59 smithi183 ceph-mon[41562]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:28:59.731 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:28:59.767 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:28:59.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:28:59.767 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:28:59.868 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:28:59.907 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:28:59.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:28:59.908 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:29:00.009 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:29:00.047 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:29:00.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:29:00.048 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:29:00.071 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: 66 16 111908 111892 105.936 81.8125 0.0100492 0.00943467 2024-09-10T19:29:00.085 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:29:00.148 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:29:00.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:29:00.184 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:29:00.184 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:29:00.285 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:29:00.310 INFO:journalctl@ceph.iscsi.iscsi.a.smithi191.stdout:Sep 10 19:28:59 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-iscsi-iscsi-a[61816]: debug there is no tcmu-runner data available 2024-09-10T19:29:00.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:29:00.325 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:29:00.325 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:29:00.426 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:29:00.467 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:29:00.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:29:00.468 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:29:00.569 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:29:00.610 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:29:00.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:29:00.611 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:29:00.712 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:29:00.746 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:29:00.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:29:00.747 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:29:00.847 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:29:00.869 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:29:00.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:29:00.871 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:29:00.972 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:29:01.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:29:01.009 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:29:01.010 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:29:01.071 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: 67 16 113287 113271 105.641 86.1875 0.0181452 0.00946207 2024-09-10T19:29:01.110 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:29:01.150 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:29:01.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:29:01.152 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:29:01.252 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:29:01.289 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:29:01.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:29:01.290 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:29:01.379 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:29:01 smithi183 ceph-mon[29736]: pgmap v876: 158 pgs: 158 active+clean; 6.6 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 760 B/s rd, 64 MiB/s wr, 1.02k op/s 2024-09-10T19:29:01.380 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:29:01 smithi183 ceph-mon[29736]: from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:29:01.380 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:29:01 smithi183 ceph-mon[41562]: pgmap v876: 158 pgs: 158 active+clean; 6.6 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 760 B/s rd, 64 MiB/s wr, 1.02k op/s 2024-09-10T19:29:01.380 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:29:01 smithi183 ceph-mon[41562]: from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:29:01.380 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Sep 10 19:29:01 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-y[29989]: ::ffff:172.21.15.191 - - [10/Sep/2024:19:29:01] "GET /metrics HTTP/1.1" 200 224905 "" "Prometheus/2.43.0" 2024-09-10T19:29:01.391 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:29:01.427 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:29:01.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:29:01.428 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:29:01.529 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:29:01.561 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:29:01.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:29:01.563 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:29:01.657 INFO:teuthology.orchestra.run.smithi183.stdout:osd.0: {} 2024-09-10T19:29:01.658 INFO:teuthology.orchestra.run.smithi183.stdout:osd.1: {} 2024-09-10T19:29:01.658 INFO:teuthology.orchestra.run.smithi183.stdout:osd.2: {} 2024-09-10T19:29:01.658 INFO:teuthology.orchestra.run.smithi183.stdout:osd.3: {} 2024-09-10T19:29:01.658 INFO:teuthology.orchestra.run.smithi183.stdout:osd.4: {} 2024-09-10T19:29:01.658 INFO:teuthology.orchestra.run.smithi183.stdout:osd.5: {} 2024-09-10T19:29:01.658 INFO:teuthology.orchestra.run.smithi183.stdout:osd.6: {} 2024-09-10T19:29:01.658 INFO:teuthology.orchestra.run.smithi183.stdout:osd.7: {} 2024-09-10T19:29:01.658 INFO:teuthology.orchestra.run.smithi183.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-10T19:29:01.659 INFO:teuthology.orchestra.run.smithi183.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-10T19:29:01.659 INFO:teuthology.orchestra.run.smithi183.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-10T19:29:01.659 INFO:teuthology.orchestra.run.smithi183.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-10T19:29:01.659 INFO:teuthology.orchestra.run.smithi183.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-10T19:29:01.659 INFO:teuthology.orchestra.run.smithi183.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-10T19:29:01.659 INFO:teuthology.orchestra.run.smithi183.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-10T19:29:01.659 INFO:teuthology.orchestra.run.smithi183.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-10T19:29:01.663 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:29:01.700 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:29:01.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:29:01.701 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:29:01.715 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:29:01 smithi191 ceph-mon[32884]: pgmap v876: 158 pgs: 158 active+clean; 6.6 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 760 B/s rd, 64 MiB/s wr, 1.02k op/s 2024-09-10T19:29:01.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:29:01 smithi191 ceph-mon[32884]: from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:29:01.802 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:29:01.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:29:01.844 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:29:01.845 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:29:01.945 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:29:01.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:29:01.983 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:29:01.983 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:29:02.071 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: 68 16 114731 114715 105.415 90.25 0.00437912 0.00948195 2024-09-10T19:29:02.084 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:29:02.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:29:02.120 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:29:02.120 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:29:02.221 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:29:02.259 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:29:02.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:29:02.260 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:29:02.361 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:29:02.397 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:29:02.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:29:02.398 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:29:02.499 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:29:02.532 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:29:02.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:29:02.533 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:29:02.634 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:29:02.640 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:29:02 smithi183 ceph-mon[41562]: from='client.? 172.21.15.183:0/839059331' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T19:29:02.640 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:29:02 smithi183 ceph-mon[29736]: from='client.? 172.21.15.183:0/839059331' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T19:29:02.665 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:29:02.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:29:02.666 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:29:02.715 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:29:02 smithi191 ceph-mon[32884]: from='client.? 172.21.15.183:0/839059331' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T19:29:02.768 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:29:02.806 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:29:02.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:29:02.807 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:29:02.908 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:29:02.945 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:29:02.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:29:02.946 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:29:03.047 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:29:03.072 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: 69 16 116206 116190 105.222 92.1875 0.0130946 0.00949948 2024-09-10T19:29:03.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:29:03.090 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:29:03.090 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:29:03.191 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:29:03.226 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:29:03.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:29:03.227 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:29:03.328 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:29:03.380 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:29:03.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:29:03.381 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:29:03.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:29:03 smithi183 ceph-mon[29736]: pgmap v877: 158 pgs: 158 active+clean; 6.8 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 639 B/s rd, 84 MiB/s wr, 1.34k op/s 2024-09-10T19:29:03.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:29:03 smithi183 ceph-mon[41562]: pgmap v877: 158 pgs: 158 active+clean; 6.8 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 639 B/s rd, 84 MiB/s wr, 1.34k op/s 2024-09-10T19:29:03.482 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:29:03.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:29:03.519 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:29:03.519 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:29:03.620 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:29:03.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:29:03.657 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:29:03.657 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:29:03.715 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:29:03 smithi191 ceph-mon[32884]: pgmap v877: 158 pgs: 158 active+clean; 6.8 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 639 B/s rd, 84 MiB/s wr, 1.34k op/s 2024-09-10T19:29:03.758 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:29:03.781 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:29:03.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:29:03.782 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:29:03.883 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:29:03.904 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:29:03.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:29:03.906 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:29:04.006 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:29:04.041 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:29:04.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:29:04.042 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:29:04.072 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: 70 16 117415 117399 104.798 75.5625 0.00574419 0.00953772 2024-09-10T19:29:04.143 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:29:04.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:29:04.183 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:29:04.183 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:29:04.284 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:29:04.327 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:29:04.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:29:04.328 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:29:04.428 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:29:04.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:29:04.473 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:29:04.473 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:29:04.515 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-10T19:29:04.574 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:29:04.602 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:29:04.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:29:04.603 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:29:04.704 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:29:04.736 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:29:04.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:29:04.737 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:29:04.838 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:29:04.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:29:04.878 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:29:04.878 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:29:04.979 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:29:05.013 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:29:05.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:29:05.014 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:29:05.072 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: 71 16 118768 118752 104.513 84.5625 0.0154822 0.00956417 2024-09-10T19:29:05.115 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:29:05.152 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:29:05.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:29:05.153 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:29:05.254 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:29:05.293 INFO:teuthology.orchestra.run.smithi183.stderr:noscrub is unset 2024-09-10T19:29:05.295 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:29:05.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:29:05.296 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:29:05.397 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:29:05.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:29:05.436 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:29:05.436 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:29:05.537 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:29:05.569 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:29:05.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:29:05.570 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:29:05.663 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:29:05 smithi183 ceph-mon[29736]: pgmap v878: 158 pgs: 158 active+clean; 6.9 GiB data, 26 GiB used, 600 GiB / 626 GiB avail; 1.2 KiB/s rd, 53 MiB/s wr, 855 op/s 2024-09-10T19:29:05.663 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:29:05 smithi183 ceph-mon[29736]: from='client.? 172.21.15.183:0/4005320773' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-10T19:29:05.664 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:29:05 smithi183 ceph-mon[29736]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-10T19:29:05.664 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:29:05 smithi183 ceph-mon[41562]: pgmap v878: 158 pgs: 158 active+clean; 6.9 GiB data, 26 GiB used, 600 GiB / 626 GiB avail; 1.2 KiB/s rd, 53 MiB/s wr, 855 op/s 2024-09-10T19:29:05.664 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:29:05 smithi183 ceph-mon[41562]: from='client.? 172.21.15.183:0/4005320773' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-10T19:29:05.664 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:29:05 smithi183 ceph-mon[41562]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-10T19:29:05.671 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:29:05.705 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:29:05.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:29:05.707 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:29:05.715 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:29:05 smithi191 ceph-mon[32884]: pgmap v878: 158 pgs: 158 active+clean; 6.9 GiB data, 26 GiB used, 600 GiB / 626 GiB avail; 1.2 KiB/s rd, 53 MiB/s wr, 855 op/s 2024-09-10T19:29:05.715 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:29:05 smithi191 ceph-mon[32884]: from='client.? 172.21.15.183:0/4005320773' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-10T19:29:05.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:29:05 smithi191 ceph-mon[32884]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-10T19:29:05.807 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:29:05.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:29:05.843 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:29:05.843 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:29:05.944 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:29:05.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:29:05.978 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:29:05.978 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:29:06.072 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: 72 16 120200 120184 104.304 89.5 0.0151993 0.00958303 2024-09-10T19:29:06.079 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:29:06.113 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:29:06.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:29:06.114 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:29:06.214 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:29:06.248 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:29:06.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:29:06.252 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:29:06.353 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:29:06.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:29:06.389 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:29:06.389 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:29:06.490 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:29:06.524 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:29:06.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:29:06.525 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:29:06.626 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:29:06.664 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:29:06.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:29:06.666 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:29:06.681 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:29:06 smithi183 ceph-mon[29736]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T19:29:06.681 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:29:06 smithi183 ceph-mon[29736]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-10T19:29:06.681 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:29:06 smithi183 ceph-mon[29736]: osdmap e399: 8 total, 8 up, 7 in 2024-09-10T19:29:06.682 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:29:06 smithi183 ceph-mon[41562]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T19:29:06.682 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:29:06 smithi183 ceph-mon[41562]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-10T19:29:06.682 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:29:06 smithi183 ceph-mon[41562]: osdmap e399: 8 total, 8 up, 7 in 2024-09-10T19:29:06.715 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:29:06 smithi191 ceph-mon[32884]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T19:29:06.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:29:06 smithi191 ceph-mon[32884]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-10T19:29:06.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:29:06 smithi191 ceph-mon[32884]: osdmap e399: 8 total, 8 up, 7 in 2024-09-10T19:29:06.766 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:29:06.802 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:29:06.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:29:06.803 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:29:06.904 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:29:06.942 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:29:06.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:29:06.943 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:29:07.044 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:29:07.073 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: 73 16 121669 121653 104.133 91.8125 0.0140734 0.00959872 2024-09-10T19:29:07.089 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:29:07.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:29:07.090 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:29:07.191 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:29:07.231 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:29:07.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:29:07.232 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:29:07.333 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:29:07.377 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:29:07.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:29:07.378 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:29:07.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:29:07 smithi183 ceph-mon[29736]: pgmap v880: 158 pgs: 158 active+clean; 7.2 GiB data, 26 GiB used, 599 GiB / 626 GiB avail; 1.1 KiB/s rd, 86 MiB/s wr, 1.38k op/s 2024-09-10T19:29:07.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:29:07 smithi183 ceph-mon[29736]: 4.2 scrub starts 2024-09-10T19:29:07.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:29:07 smithi183 ceph-mon[29736]: 4.2 scrub ok 2024-09-10T19:29:07.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:29:07 smithi183 ceph-mon[29736]: 6.0 scrub starts 2024-09-10T19:29:07.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:29:07 smithi183 ceph-mon[29736]: 6.0 scrub ok 2024-09-10T19:29:07.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:29:07 smithi183 ceph-mon[41562]: pgmap v880: 158 pgs: 158 active+clean; 7.2 GiB data, 26 GiB used, 599 GiB / 626 GiB avail; 1.1 KiB/s rd, 86 MiB/s wr, 1.38k op/s 2024-09-10T19:29:07.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:29:07 smithi183 ceph-mon[41562]: 4.2 scrub starts 2024-09-10T19:29:07.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:29:07 smithi183 ceph-mon[41562]: 4.2 scrub ok 2024-09-10T19:29:07.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:29:07 smithi183 ceph-mon[41562]: 6.0 scrub starts 2024-09-10T19:29:07.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:29:07 smithi183 ceph-mon[41562]: 6.0 scrub ok 2024-09-10T19:29:07.437 INFO:teuthology.orchestra.run.smithi183.stdout:ERROR: (22) Invalid argument 2024-09-10T19:29:07.437 INFO:teuthology.orchestra.run.smithi183.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-10T19:29:07.453 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:29:07.479 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:29:07.527 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:29:07.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:29:07.528 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:29:07.629 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:29:07.671 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:29:07.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:29:07.672 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:29:07.716 INFO:journalctl@ceph.mgr.x.smithi191.stdout:Sep 10 19:29:07 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-x[34967]: ::ffff:172.21.15.191 - - [10/Sep/2024:19:29:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T19:29:07.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:29:07 smithi191 ceph-mon[32884]: pgmap v880: 158 pgs: 158 active+clean; 7.2 GiB data, 26 GiB used, 599 GiB / 626 GiB avail; 1.1 KiB/s rd, 86 MiB/s wr, 1.38k op/s 2024-09-10T19:29:07.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:29:07 smithi191 ceph-mon[32884]: 4.2 scrub starts 2024-09-10T19:29:07.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:29:07 smithi191 ceph-mon[32884]: 4.2 scrub ok 2024-09-10T19:29:07.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:29:07 smithi191 ceph-mon[32884]: 6.0 scrub starts 2024-09-10T19:29:07.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:29:07 smithi191 ceph-mon[32884]: 6.0 scrub ok 2024-09-10T19:29:07.772 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:29:07.810 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:29:07.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:29:07.811 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:29:07.912 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:29:07.946 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:29:07.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:29:07.947 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:29:08.048 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:29:08.072 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: 74 16 123021 123005 103.867 84.5 0.0181425 0.00962312 2024-09-10T19:29:08.086 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:29:08.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:29:08.087 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:29:08.188 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:29:08.225 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:29:08.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:29:08.226 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:29:08.322 INFO:teuthology.orchestra.run.smithi183.stderr:reweighted osd.0 to 1 (10000) 2024-09-10T19:29:08.328 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:29:08.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:29:08.373 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:29:08.373 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:29:08.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:29:08 smithi183 ceph-mon[29736]: 5.1f scrub starts 2024-09-10T19:29:08.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:29:08 smithi183 ceph-mon[29736]: 5.1f scrub ok 2024-09-10T19:29:08.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:29:08 smithi183 ceph-mon[29736]: 5.18 scrub starts 2024-09-10T19:29:08.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:29:08 smithi183 ceph-mon[29736]: 5.18 scrub ok 2024-09-10T19:29:08.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:29:08 smithi183 ceph-mon[29736]: from='client.? 172.21.15.183:0/2748679654' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-09-10T19:29:08.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:29:08 smithi183 ceph-mon[29736]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-09-10T19:29:08.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:29:08 smithi183 ceph-mon[29736]: 1.2 scrub starts 2024-09-10T19:29:08.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:29:08 smithi183 ceph-mon[29736]: 1.2 scrub ok 2024-09-10T19:29:08.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:29:08 smithi183 ceph-mon[29736]: 6.6 scrub starts 2024-09-10T19:29:08.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:29:08 smithi183 ceph-mon[29736]: 6.6 scrub ok 2024-09-10T19:29:08.433 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:29:08 smithi183 ceph-mon[41562]: 5.1f scrub starts 2024-09-10T19:29:08.433 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:29:08 smithi183 ceph-mon[41562]: 5.1f scrub ok 2024-09-10T19:29:08.433 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:29:08 smithi183 ceph-mon[41562]: 5.18 scrub starts 2024-09-10T19:29:08.433 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:29:08 smithi183 ceph-mon[41562]: 5.18 scrub ok 2024-09-10T19:29:08.433 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:29:08 smithi183 ceph-mon[41562]: from='client.? 172.21.15.183:0/2748679654' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-09-10T19:29:08.433 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:29:08 smithi183 ceph-mon[41562]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-09-10T19:29:08.433 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:29:08 smithi183 ceph-mon[41562]: 1.2 scrub starts 2024-09-10T19:29:08.433 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:29:08 smithi183 ceph-mon[41562]: 1.2 scrub ok 2024-09-10T19:29:08.433 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:29:08 smithi183 ceph-mon[41562]: 6.6 scrub starts 2024-09-10T19:29:08.433 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:29:08 smithi183 ceph-mon[41562]: 6.6 scrub ok 2024-09-10T19:29:08.474 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:29:08.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:29:08.505 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:29:08.505 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:29:08.606 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:29:08.645 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:29:08.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:29:08.646 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:29:08.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:29:08 smithi191 ceph-mon[32884]: 5.1f scrub starts 2024-09-10T19:29:08.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:29:08 smithi191 ceph-mon[32884]: 5.1f scrub ok 2024-09-10T19:29:08.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:29:08 smithi191 ceph-mon[32884]: 5.18 scrub starts 2024-09-10T19:29:08.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:29:08 smithi191 ceph-mon[32884]: 5.18 scrub ok 2024-09-10T19:29:08.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:29:08 smithi191 ceph-mon[32884]: from='client.? 172.21.15.183:0/2748679654' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-09-10T19:29:08.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:29:08 smithi191 ceph-mon[32884]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-09-10T19:29:08.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:29:08 smithi191 ceph-mon[32884]: 1.2 scrub starts 2024-09-10T19:29:08.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:29:08 smithi191 ceph-mon[32884]: 1.2 scrub ok 2024-09-10T19:29:08.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:29:08 smithi191 ceph-mon[32884]: 6.6 scrub starts 2024-09-10T19:29:08.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:29:08 smithi191 ceph-mon[32884]: 6.6 scrub ok 2024-09-10T19:29:08.747 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:29:08.783 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:29:08.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:29:08.784 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:29:08.885 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:29:08.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:29:08.918 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:29:08.918 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:29:08.921 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-10T19:29:09.019 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:29:09.061 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:29:09.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:29:09.062 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:29:09.072 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: 75 16 124472 124456 103.692 90.6875 0.0092352 0.00963979 2024-09-10T19:29:09.163 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:29:09.195 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:29:09.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:29:09.196 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:29:09.296 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:29:09.323 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:29:09.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:29:09.324 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:29:09.340 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph daemon osd.2 dump_ops_in_flight 2024-09-10T19:29:09.425 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:29:09.465 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:29:09.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:29:09.467 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:29:09.567 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:29:09.598 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:29:09.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:29:09.599 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:29:09.624 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:29:09 smithi183 ceph-mon[29736]: pgmap v881: 158 pgs: 158 active+clean; 7.3 GiB data, 27 GiB used, 599 GiB / 626 GiB avail; 1023 B/s rd, 79 MiB/s wr, 1.26k op/s 2024-09-10T19:29:09.624 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:29:09 smithi183 ceph-mon[29736]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-09-10T19:29:09.624 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:29:09 smithi183 ceph-mon[29736]: osdmap e400: 8 total, 8 up, 7 in 2024-09-10T19:29:09.624 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:29:09 smithi183 ceph-mon[29736]: 3.1 scrub starts 2024-09-10T19:29:09.624 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:29:09 smithi183 ceph-mon[29736]: 3.1 scrub ok 2024-09-10T19:29:09.624 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:29:09 smithi183 ceph-mon[41562]: pgmap v881: 158 pgs: 158 active+clean; 7.3 GiB data, 27 GiB used, 599 GiB / 626 GiB avail; 1023 B/s rd, 79 MiB/s wr, 1.26k op/s 2024-09-10T19:29:09.624 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:29:09 smithi183 ceph-mon[41562]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-09-10T19:29:09.624 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:29:09 smithi183 ceph-mon[41562]: osdmap e400: 8 total, 8 up, 7 in 2024-09-10T19:29:09.625 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:29:09 smithi183 ceph-mon[41562]: 3.1 scrub starts 2024-09-10T19:29:09.625 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:29:09 smithi183 ceph-mon[41562]: 3.1 scrub ok 2024-09-10T19:29:09.700 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:29:09.715 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:29:09 smithi191 ceph-mon[32884]: pgmap v881: 158 pgs: 158 active+clean; 7.3 GiB data, 27 GiB used, 599 GiB / 626 GiB avail; 1023 B/s rd, 79 MiB/s wr, 1.26k op/s 2024-09-10T19:29:09.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:29:09 smithi191 ceph-mon[32884]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-09-10T19:29:09.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:29:09 smithi191 ceph-mon[32884]: osdmap e400: 8 total, 8 up, 7 in 2024-09-10T19:29:09.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:29:09 smithi191 ceph-mon[32884]: 3.1 scrub starts 2024-09-10T19:29:09.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:29:09 smithi191 ceph-mon[32884]: 3.1 scrub ok 2024-09-10T19:29:09.739 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:29:09.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:29:09.740 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:29:09.841 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:29:09.881 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:29:09.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:29:09.883 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:29:09.983 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:29:10.017 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:29:10.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:29:10.018 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:29:10.073 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: 76 16 125785 125769 103.407 82.0625 0.00886206 0.00966649 2024-09-10T19:29:10.118 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:29:10.150 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:29:10.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:29:10.152 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:29:10.252 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:29:10.290 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:29:10.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:29:10.291 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:29:10.336 INFO:journalctl@ceph.iscsi.iscsi.a.smithi191.stdout:Sep 10 19:29:09 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-iscsi-iscsi-a[61816]: debug there is no tcmu-runner data available 2024-09-10T19:29:10.392 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:29:10.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:29:10.430 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:29:10.430 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:29:10.530 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:29:10.571 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:29:10.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:29:10.572 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:29:10.594 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:29:10 smithi183 ceph-mon[29736]: 2.15 scrub starts 2024-09-10T19:29:10.595 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:29:10 smithi183 ceph-mon[29736]: 2.15 scrub ok 2024-09-10T19:29:10.595 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:29:10 smithi183 ceph-mon[29736]: 6.f scrub starts 2024-09-10T19:29:10.595 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:29:10 smithi183 ceph-mon[29736]: 6.f scrub ok 2024-09-10T19:29:10.595 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:29:10 smithi183 ceph-mon[41562]: 2.15 scrub starts 2024-09-10T19:29:10.595 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:29:10 smithi183 ceph-mon[41562]: 2.15 scrub ok 2024-09-10T19:29:10.596 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:29:10 smithi183 ceph-mon[41562]: 6.f scrub starts 2024-09-10T19:29:10.596 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:29:10 smithi183 ceph-mon[41562]: 6.f scrub ok 2024-09-10T19:29:10.597 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph osd reweight 3 1 2024-09-10T19:29:10.673 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:29:10.715 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:29:10 smithi191 ceph-mon[32884]: 2.15 scrub starts 2024-09-10T19:29:10.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:29:10 smithi191 ceph-mon[32884]: 2.15 scrub ok 2024-09-10T19:29:10.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:29:10 smithi191 ceph-mon[32884]: 6.f scrub starts 2024-09-10T19:29:10.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:29:10 smithi191 ceph-mon[32884]: 6.f scrub ok 2024-09-10T19:29:10.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:29:10.723 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:29:10.724 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:29:10.824 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:29:10.862 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:29:10.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:29:10.863 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:29:10.964 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:29:10.998 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:29:10.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:29:10.999 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:29:11.073 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: 77 16 127205 127189 103.217 88.75 0.00878242 0.00968407 2024-09-10T19:29:11.099 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:29:11.165 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:29:11.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:29:11.167 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:29:11.185 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:29:11.205 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:29:11.213 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:29:11.268 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:29:11.311 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:29:11.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:29:11.312 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:29:11.324 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Sep 10 19:29:11 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-y[29989]: ::ffff:172.21.15.191 - - [10/Sep/2024:19:29:11] "GET /metrics HTTP/1.1" 200 224905 "" "Prometheus/2.43.0" 2024-09-10T19:29:11.413 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:29:11.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:29:11.453 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:29:11.453 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:29:11.554 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:29:11.589 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:29:11 smithi183 ceph-mon[29736]: pgmap v883: 158 pgs: 158 active+clean; 7.4 GiB data, 27 GiB used, 599 GiB / 626 GiB avail; 1.2 KiB/s rd, 70 MiB/s wr, 1.12k op/s 2024-09-10T19:29:11.589 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:29:11 smithi183 ceph-mon[29736]: from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:29:11.589 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:29:11 smithi183 ceph-mon[29736]: 6.e scrub starts 2024-09-10T19:29:11.590 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:29:11 smithi183 ceph-mon[29736]: 6.e scrub ok 2024-09-10T19:29:11.590 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:29:11 smithi183 ceph-mon[29736]: 1.6 scrub starts 2024-09-10T19:29:11.590 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:29:11 smithi183 ceph-mon[29736]: 1.6 scrub ok 2024-09-10T19:29:11.590 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:29:11 smithi183 ceph-mon[29736]: 6.18 scrub starts 2024-09-10T19:29:11.590 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:29:11 smithi183 ceph-mon[29736]: 6.18 scrub ok 2024-09-10T19:29:11.590 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:29:11 smithi183 ceph-mon[29736]: 5.14 scrub starts 2024-09-10T19:29:11.590 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:29:11 smithi183 ceph-mon[29736]: 5.14 scrub ok 2024-09-10T19:29:11.590 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:29:11 smithi183 ceph-mon[41562]: pgmap v883: 158 pgs: 158 active+clean; 7.4 GiB data, 27 GiB used, 599 GiB / 626 GiB avail; 1.2 KiB/s rd, 70 MiB/s wr, 1.12k op/s 2024-09-10T19:29:11.590 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:29:11 smithi183 ceph-mon[41562]: from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:29:11.591 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:29:11 smithi183 ceph-mon[41562]: 6.e scrub starts 2024-09-10T19:29:11.591 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:29:11 smithi183 ceph-mon[41562]: 6.e scrub ok 2024-09-10T19:29:11.591 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:29:11 smithi183 ceph-mon[41562]: 1.6 scrub starts 2024-09-10T19:29:11.591 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:29:11 smithi183 ceph-mon[41562]: 1.6 scrub ok 2024-09-10T19:29:11.591 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:29:11 smithi183 ceph-mon[41562]: 6.18 scrub starts 2024-09-10T19:29:11.591 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:29:11 smithi183 ceph-mon[41562]: 6.18 scrub ok 2024-09-10T19:29:11.591 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:29:11 smithi183 ceph-mon[41562]: 5.14 scrub starts 2024-09-10T19:29:11.591 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:29:11 smithi183 ceph-mon[41562]: 5.14 scrub ok 2024-09-10T19:29:11.595 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:29:11.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:29:11.596 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:29:11.697 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:29:11.715 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:29:11 smithi191 ceph-mon[32884]: pgmap v883: 158 pgs: 158 active+clean; 7.4 GiB data, 27 GiB used, 599 GiB / 626 GiB avail; 1.2 KiB/s rd, 70 MiB/s wr, 1.12k op/s 2024-09-10T19:29:11.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:29:11 smithi191 ceph-mon[32884]: from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:29:11.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:29:11 smithi191 ceph-mon[32884]: 6.e scrub starts 2024-09-10T19:29:11.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:29:11 smithi191 ceph-mon[32884]: 6.e scrub ok 2024-09-10T19:29:11.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:29:11 smithi191 ceph-mon[32884]: 1.6 scrub starts 2024-09-10T19:29:11.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:29:11 smithi191 ceph-mon[32884]: 1.6 scrub ok 2024-09-10T19:29:11.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:29:11 smithi191 ceph-mon[32884]: 6.18 scrub starts 2024-09-10T19:29:11.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:29:11 smithi191 ceph-mon[32884]: 6.18 scrub ok 2024-09-10T19:29:11.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:29:11 smithi191 ceph-mon[32884]: 5.14 scrub starts 2024-09-10T19:29:11.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:29:11 smithi191 ceph-mon[32884]: 5.14 scrub ok 2024-09-10T19:29:11.727 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:29:11.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:29:11.728 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:29:11.829 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:29:11.865 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:29:11.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:29:11.866 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:29:11.967 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:29:12.005 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:29:12.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:29:12.006 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:29:12.073 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: 78 16 128577 128561 102.993 85.75 0.0175367 0.00970527 2024-09-10T19:29:12.106 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:29:12.143 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:29:12.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:29:12.144 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:29:12.244 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:29:12.279 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:29:12.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:29:12.280 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:29:12.381 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:29:12.397 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:29:12 smithi183 ceph-mon[29736]: 2.13 scrub starts 2024-09-10T19:29:12.397 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:29:12 smithi183 ceph-mon[29736]: 2.13 scrub ok 2024-09-10T19:29:12.397 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:29:12 smithi183 ceph-mon[29736]: 2.1d scrub starts 2024-09-10T19:29:12.398 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:29:12 smithi183 ceph-mon[29736]: 2.1d scrub ok 2024-09-10T19:29:12.398 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:29:12 smithi183 ceph-mon[29736]: pgmap v884: 158 pgs: 158 active+clean; 7.7 GiB data, 28 GiB used, 598 GiB / 626 GiB avail; 639 B/s rd, 99 MiB/s wr, 1.58k op/s 2024-09-10T19:29:12.398 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:29:12 smithi183 ceph-mon[41562]: 2.13 scrub starts 2024-09-10T19:29:12.398 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:29:12 smithi183 ceph-mon[41562]: 2.13 scrub ok 2024-09-10T19:29:12.398 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:29:12 smithi183 ceph-mon[41562]: 2.1d scrub starts 2024-09-10T19:29:12.398 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:29:12 smithi183 ceph-mon[41562]: 2.1d scrub ok 2024-09-10T19:29:12.398 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:29:12 smithi183 ceph-mon[41562]: pgmap v884: 158 pgs: 158 active+clean; 7.7 GiB data, 28 GiB used, 598 GiB / 626 GiB avail; 639 B/s rd, 99 MiB/s wr, 1.58k op/s 2024-09-10T19:29:12.409 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:29:12.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:29:12.410 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:29:12.511 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:29:12.550 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:29:12.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:29:12.551 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:29:12.651 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:29:12.687 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:29:12.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:29:12.688 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:29:12.715 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:29:12 smithi191 ceph-mon[32884]: 2.13 scrub starts 2024-09-10T19:29:12.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:29:12 smithi191 ceph-mon[32884]: 2.13 scrub ok 2024-09-10T19:29:12.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:29:12 smithi191 ceph-mon[32884]: 2.1d scrub starts 2024-09-10T19:29:12.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:29:12 smithi191 ceph-mon[32884]: 2.1d scrub ok 2024-09-10T19:29:12.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:29:12 smithi191 ceph-mon[32884]: pgmap v884: 158 pgs: 158 active+clean; 7.7 GiB data, 28 GiB used, 598 GiB / 626 GiB avail; 639 B/s rd, 99 MiB/s wr, 1.58k op/s 2024-09-10T19:29:12.789 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:29:12.823 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:29:12.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:29:12.824 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:29:12.925 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:29:12.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:29:12.964 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:29:12.964 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:29:13.065 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:29:13.073 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: 79 16 130036 130020 102.843 91.1875 0.00897747 0.00971909 2024-09-10T19:29:13.102 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:29:13.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:29:13.103 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:29:13.204 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:29:13.211 INFO:teuthology.orchestra.run.smithi183.stdout:osd.0: {} 2024-09-10T19:29:13.211 INFO:teuthology.orchestra.run.smithi183.stdout:osd.1: {} 2024-09-10T19:29:13.211 INFO:teuthology.orchestra.run.smithi183.stdout:osd.2: {} 2024-09-10T19:29:13.211 INFO:teuthology.orchestra.run.smithi183.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-10T19:29:13.211 INFO:teuthology.orchestra.run.smithi183.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-10T19:29:13.211 INFO:teuthology.orchestra.run.smithi183.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-10T19:29:13.213 INFO:teuthology.orchestra.run.smithi183.stdout:osd.3: {} 2024-09-10T19:29:13.213 INFO:teuthology.orchestra.run.smithi183.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-10T19:29:13.233 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:29:13.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:29:13.234 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:29:13.244 INFO:teuthology.orchestra.run.smithi183.stdout:osd.4: {} 2024-09-10T19:29:13.244 INFO:teuthology.orchestra.run.smithi183.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-10T19:29:13.276 INFO:teuthology.orchestra.run.smithi183.stdout:osd.5: {} 2024-09-10T19:29:13.276 INFO:teuthology.orchestra.run.smithi183.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-10T19:29:13.308 INFO:teuthology.orchestra.run.smithi183.stdout:osd.6: {} 2024-09-10T19:29:13.308 INFO:teuthology.orchestra.run.smithi183.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-10T19:29:13.336 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:29:13.340 INFO:teuthology.orchestra.run.smithi183.stdout:osd.7: {} 2024-09-10T19:29:13.340 INFO:teuthology.orchestra.run.smithi183.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-10T19:29:13.367 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:29:13.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:29:13.368 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:29:13.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:29:13 smithi183 ceph-mon[29736]: 6.1e scrub starts 2024-09-10T19:29:13.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:29:13 smithi183 ceph-mon[29736]: 6.1e scrub ok 2024-09-10T19:29:13.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:29:13 smithi183 ceph-mon[29736]: 2.5 scrub starts 2024-09-10T19:29:13.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:29:13 smithi183 ceph-mon[29736]: 2.5 scrub ok 2024-09-10T19:29:13.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:29:13 smithi183 ceph-mon[29736]: from='client.? 172.21.15.183:0/3986131780' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T19:29:13.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:29:13 smithi183 ceph-mon[41562]: 6.1e scrub starts 2024-09-10T19:29:13.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:29:13 smithi183 ceph-mon[41562]: 6.1e scrub ok 2024-09-10T19:29:13.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:29:13 smithi183 ceph-mon[41562]: 2.5 scrub starts 2024-09-10T19:29:13.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:29:13 smithi183 ceph-mon[41562]: 2.5 scrub ok 2024-09-10T19:29:13.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:29:13 smithi183 ceph-mon[41562]: from='client.? 172.21.15.183:0/3986131780' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T19:29:13.468 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:29:13.508 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:29:13.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:29:13.509 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:29:13.610 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:29:13.643 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:29:13.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:29:13.644 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:29:13.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:29:13 smithi191 ceph-mon[32884]: 6.1e scrub starts 2024-09-10T19:29:13.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:29:13 smithi191 ceph-mon[32884]: 6.1e scrub ok 2024-09-10T19:29:13.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:29:13 smithi191 ceph-mon[32884]: 2.5 scrub starts 2024-09-10T19:29:13.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:29:13 smithi191 ceph-mon[32884]: 2.5 scrub ok 2024-09-10T19:29:13.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:29:13 smithi191 ceph-mon[32884]: from='client.? 172.21.15.183:0/3986131780' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T19:29:13.745 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:29:13.778 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:29:13.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:29:13.779 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:29:13.880 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:29:13.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:29:13.922 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:29:13.922 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:29:14.023 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:29:14.070 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:29:14.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:29:14.071 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:29:14.073 INFO:tasks.radosbench.radosbench.0.smithi183.stdout:2024-09-10T19:29:14.072513+0000 min lat: 0.00134808 max lat: 0.573472 avg lat: 0.00973571 2024-09-10T19:29:14.073 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2024-09-10T19:29:14.073 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: 80 16 131465 131449 102.674 89.3125 0.00664519 0.00973571 2024-09-10T19:29:14.171 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:29:14.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:29:14.217 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:29:14.217 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:29:14.318 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:29:14.353 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:29:14.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:29:14.354 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:29:14.455 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:29:14.496 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:29:14.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:29:14.497 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:29:14.598 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:29:14.632 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:29:14.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:29:14.633 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:29:14.681 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:29:14 smithi183 ceph-mon[29736]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:29:14.681 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:29:14 smithi183 ceph-mon[29736]: 4.16 scrub starts 2024-09-10T19:29:14.681 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:29:14 smithi183 ceph-mon[29736]: 4.16 scrub ok 2024-09-10T19:29:14.682 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:29:14 smithi183 ceph-mon[29736]: 5.4 scrub starts 2024-09-10T19:29:14.682 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:29:14 smithi183 ceph-mon[29736]: 5.4 scrub ok 2024-09-10T19:29:14.682 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:29:14 smithi183 ceph-mon[29736]: pgmap v885: 158 pgs: 158 active+clean; 7.7 GiB data, 28 GiB used, 597 GiB / 626 GiB avail; 1.2 KiB/s rd, 72 MiB/s wr, 1.16k op/s 2024-09-10T19:29:14.682 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:29:14 smithi183 ceph-mon[41562]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:29:14.682 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:29:14 smithi183 ceph-mon[41562]: 4.16 scrub starts 2024-09-10T19:29:14.682 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:29:14 smithi183 ceph-mon[41562]: 4.16 scrub ok 2024-09-10T19:29:14.682 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:29:14 smithi183 ceph-mon[41562]: 5.4 scrub starts 2024-09-10T19:29:14.682 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:29:14 smithi183 ceph-mon[41562]: 5.4 scrub ok 2024-09-10T19:29:14.682 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:29:14 smithi183 ceph-mon[41562]: pgmap v885: 158 pgs: 158 active+clean; 7.7 GiB data, 28 GiB used, 597 GiB / 626 GiB avail; 1.2 KiB/s rd, 72 MiB/s wr, 1.16k op/s 2024-09-10T19:29:14.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:29:14 smithi191 ceph-mon[32884]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:29:14.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:29:14 smithi191 ceph-mon[32884]: 4.16 scrub starts 2024-09-10T19:29:14.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:29:14 smithi191 ceph-mon[32884]: 4.16 scrub ok 2024-09-10T19:29:14.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:29:14 smithi191 ceph-mon[32884]: 5.4 scrub starts 2024-09-10T19:29:14.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:29:14 smithi191 ceph-mon[32884]: 5.4 scrub ok 2024-09-10T19:29:14.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:29:14 smithi191 ceph-mon[32884]: pgmap v885: 158 pgs: 158 active+clean; 7.7 GiB data, 28 GiB used, 597 GiB / 626 GiB avail; 1.2 KiB/s rd, 72 MiB/s wr, 1.16k op/s 2024-09-10T19:29:14.735 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:29:14.771 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:29:14.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:29:14.772 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:29:14.873 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:29:14.906 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:29:14.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:29:14.907 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:29:15.008 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:29:15.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:29:15.042 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:29:15.042 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:29:15.073 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: 81 15 132888 132873 102.505 89 0.0109682 0.00975147 2024-09-10T19:29:15.143 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:29:15.183 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:29:15.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:29:15.184 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:29:15.285 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:29:15.315 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:29:15.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:29:15.317 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:29:15.417 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:29:15.454 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:29:15.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:29:15.455 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:29:15.556 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:29:15.590 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:29:15.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:29:15.591 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:29:15.681 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:29:15 smithi183 ceph-mon[41562]: 4.1b scrub starts 2024-09-10T19:29:15.682 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:29:15 smithi183 ceph-mon[41562]: 4.1b scrub ok 2024-09-10T19:29:15.682 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:29:15 smithi183 ceph-mon[41562]: 1.4 scrub starts 2024-09-10T19:29:15.682 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:29:15 smithi183 ceph-mon[41562]: 1.4 scrub ok 2024-09-10T19:29:15.682 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:29:15 smithi183 ceph-mon[41562]: 2.d scrub starts 2024-09-10T19:29:15.682 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:29:15 smithi183 ceph-mon[41562]: 2.d scrub ok 2024-09-10T19:29:15.683 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:29:15 smithi183 ceph-mon[41562]: 2.7 scrub starts 2024-09-10T19:29:15.683 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:29:15 smithi183 ceph-mon[41562]: 2.7 scrub ok 2024-09-10T19:29:15.683 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:29:15 smithi183 ceph-mon[41562]: 11.6 scrub starts 2024-09-10T19:29:15.683 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:29:15 smithi183 ceph-mon[41562]: 4.11 scrub starts 2024-09-10T19:29:15.683 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:29:15 smithi183 ceph-mon[29736]: 4.1b scrub starts 2024-09-10T19:29:15.683 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:29:15 smithi183 ceph-mon[29736]: 4.1b scrub ok 2024-09-10T19:29:15.684 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:29:15 smithi183 ceph-mon[29736]: 1.4 scrub starts 2024-09-10T19:29:15.684 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:29:15 smithi183 ceph-mon[29736]: 1.4 scrub ok 2024-09-10T19:29:15.684 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:29:15 smithi183 ceph-mon[29736]: 2.d scrub starts 2024-09-10T19:29:15.684 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:29:15 smithi183 ceph-mon[29736]: 2.d scrub ok 2024-09-10T19:29:15.684 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:29:15 smithi183 ceph-mon[29736]: 2.7 scrub starts 2024-09-10T19:29:15.684 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:29:15 smithi183 ceph-mon[29736]: 2.7 scrub ok 2024-09-10T19:29:15.685 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:29:15 smithi183 ceph-mon[29736]: 11.6 scrub starts 2024-09-10T19:29:15.685 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:29:15 smithi183 ceph-mon[29736]: 4.11 scrub starts 2024-09-10T19:29:15.691 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:29:15.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:29:15 smithi191 ceph-mon[32884]: 4.1b scrub starts 2024-09-10T19:29:15.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:29:15 smithi191 ceph-mon[32884]: 4.1b scrub ok 2024-09-10T19:29:15.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:29:15 smithi191 ceph-mon[32884]: 1.4 scrub starts 2024-09-10T19:29:15.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:29:15 smithi191 ceph-mon[32884]: 1.4 scrub ok 2024-09-10T19:29:15.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:29:15 smithi191 ceph-mon[32884]: 2.d scrub starts 2024-09-10T19:29:15.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:29:15 smithi191 ceph-mon[32884]: 2.d scrub ok 2024-09-10T19:29:15.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:29:15 smithi191 ceph-mon[32884]: 2.7 scrub starts 2024-09-10T19:29:15.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:29:15 smithi191 ceph-mon[32884]: 2.7 scrub ok 2024-09-10T19:29:15.717 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:29:15 smithi191 ceph-mon[32884]: 11.6 scrub starts 2024-09-10T19:29:15.717 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:29:15 smithi191 ceph-mon[32884]: 4.11 scrub starts 2024-09-10T19:29:15.723 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:29:15.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:29:15.725 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:29:15.825 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:29:15.862 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:29:15.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:29:15.863 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:29:15.964 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:29:16.003 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:29:16.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:29:16.003 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:29:16.073 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: 82 16 134259 134243 102.299 85.625 0.0153676 0.00977129 2024-09-10T19:29:16.105 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:29:16.140 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:29:16.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:29:16.142 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:29:16.242 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:29:16.287 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:29:16.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:29:16.288 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:29:16.388 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:29:16.392 INFO:teuthology.orchestra.run.smithi183.stdout:{ 2024-09-10T19:29:16.392 INFO:teuthology.orchestra.run.smithi183.stdout: "ops": [], 2024-09-10T19:29:16.392 INFO:teuthology.orchestra.run.smithi183.stdout: "num_ops": 0 2024-09-10T19:29:16.392 INFO:teuthology.orchestra.run.smithi183.stdout:} 2024-09-10T19:29:16.431 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:29:16.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:29:16.432 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:29:16.533 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:29:16.568 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:29:16.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:29:16.569 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:29:16.670 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:29:16.681 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:29:16 smithi183 ceph-mon[29736]: 4.11 scrub ok 2024-09-10T19:29:16.681 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:29:16 smithi183 ceph-mon[29736]: 6.d scrub starts 2024-09-10T19:29:16.681 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:29:16 smithi183 ceph-mon[29736]: 6.d scrub ok 2024-09-10T19:29:16.681 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:29:16 smithi183 ceph-mon[29736]: 11.3 scrub starts 2024-09-10T19:29:16.682 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:29:16 smithi183 ceph-mon[29736]: 5.d scrub starts 2024-09-10T19:29:16.682 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:29:16 smithi183 ceph-mon[29736]: 5.d scrub ok 2024-09-10T19:29:16.682 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:29:16 smithi183 ceph-mon[29736]: pgmap v886: 158 pgs: 158 active+clean; 8.1 GiB data, 29 GiB used, 597 GiB / 626 GiB avail; 1023 B/s rd, 88 MiB/s wr, 1.41k op/s 2024-09-10T19:29:16.682 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:29:16 smithi183 ceph-mon[41562]: 4.11 scrub ok 2024-09-10T19:29:16.682 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:29:16 smithi183 ceph-mon[41562]: 6.d scrub starts 2024-09-10T19:29:16.682 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:29:16 smithi183 ceph-mon[41562]: 6.d scrub ok 2024-09-10T19:29:16.682 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:29:16 smithi183 ceph-mon[41562]: 11.3 scrub starts 2024-09-10T19:29:16.682 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:29:16 smithi183 ceph-mon[41562]: 5.d scrub starts 2024-09-10T19:29:16.682 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:29:16 smithi183 ceph-mon[41562]: 5.d scrub ok 2024-09-10T19:29:16.682 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:29:16 smithi183 ceph-mon[41562]: pgmap v886: 158 pgs: 158 active+clean; 8.1 GiB data, 29 GiB used, 597 GiB / 626 GiB avail; 1023 B/s rd, 88 MiB/s wr, 1.41k op/s 2024-09-10T19:29:16.703 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:29:16.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:29:16.704 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:29:16.715 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:29:16 smithi191 ceph-mon[32884]: 4.11 scrub ok 2024-09-10T19:29:16.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:29:16 smithi191 ceph-mon[32884]: 6.d scrub starts 2024-09-10T19:29:16.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:29:16 smithi191 ceph-mon[32884]: 6.d scrub ok 2024-09-10T19:29:16.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:29:16 smithi191 ceph-mon[32884]: 11.3 scrub starts 2024-09-10T19:29:16.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:29:16 smithi191 ceph-mon[32884]: 5.d scrub starts 2024-09-10T19:29:16.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:29:16 smithi191 ceph-mon[32884]: 5.d scrub ok 2024-09-10T19:29:16.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:29:16 smithi191 ceph-mon[32884]: pgmap v886: 158 pgs: 158 active+clean; 8.1 GiB data, 29 GiB used, 597 GiB / 626 GiB avail; 1023 B/s rd, 88 MiB/s wr, 1.41k op/s 2024-09-10T19:29:16.804 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:29:16.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:29:16.839 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:29:16.839 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:29:16.940 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:29:16.981 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:29:16.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:29:16.982 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:29:17.074 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: 83 16 135578 135562 102.058 82.4375 0.00892285 0.00979426 2024-09-10T19:29:17.083 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:29:17.122 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:29:17.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:29:17.124 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:29:17.224 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:29:17.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:29:17.261 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:29:17.262 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:29:17.362 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:29:17.401 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:29:17.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:29:17.403 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:29:17.503 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:29:17.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:29:17.551 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:29:17.551 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:29:17.647 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:29:17 smithi183 ceph-mon[29736]: 3.0 scrub starts 2024-09-10T19:29:17.648 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:29:17 smithi183 ceph-mon[29736]: 3.0 scrub ok 2024-09-10T19:29:17.648 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:29:17 smithi183 ceph-mon[29736]: 2.14 scrub starts 2024-09-10T19:29:17.648 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:29:17 smithi183 ceph-mon[29736]: 2.14 scrub ok 2024-09-10T19:29:17.648 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:29:17 smithi183 ceph-mon[29736]: 5.8 scrub starts 2024-09-10T19:29:17.648 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:29:17 smithi183 ceph-mon[29736]: 5.8 scrub ok 2024-09-10T19:29:17.648 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:29:17 smithi183 ceph-mon[41562]: 3.0 scrub starts 2024-09-10T19:29:17.648 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:29:17 smithi183 ceph-mon[41562]: 3.0 scrub ok 2024-09-10T19:29:17.648 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:29:17 smithi183 ceph-mon[41562]: 2.14 scrub starts 2024-09-10T19:29:17.648 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:29:17 smithi183 ceph-mon[41562]: 2.14 scrub ok 2024-09-10T19:29:17.649 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:29:17 smithi183 ceph-mon[41562]: 5.8 scrub starts 2024-09-10T19:29:17.649 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:29:17 smithi183 ceph-mon[41562]: 5.8 scrub ok 2024-09-10T19:29:17.652 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:29:17.694 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:29:17.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:29:17.695 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:29:17.715 INFO:journalctl@ceph.mgr.x.smithi191.stdout:Sep 10 19:29:17 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-x[34967]: ::ffff:172.21.15.191 - - [10/Sep/2024:19:29:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T19:29:17.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:29:17 smithi191 ceph-mon[32884]: 3.0 scrub starts 2024-09-10T19:29:17.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:29:17 smithi191 ceph-mon[32884]: 3.0 scrub ok 2024-09-10T19:29:17.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:29:17 smithi191 ceph-mon[32884]: 2.14 scrub starts 2024-09-10T19:29:17.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:29:17 smithi191 ceph-mon[32884]: 2.14 scrub ok 2024-09-10T19:29:17.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:29:17 smithi191 ceph-mon[32884]: 5.8 scrub starts 2024-09-10T19:29:17.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:29:17 smithi191 ceph-mon[32884]: 5.8 scrub ok 2024-09-10T19:29:17.796 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:29:17.830 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:29:17.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:29:17.831 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:29:17.932 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:29:17.979 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:29:17.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:29:17.980 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:29:18.075 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: 84 16 136908 136892 101.833 83.125 0.0232261 0.00981583 2024-09-10T19:29:18.081 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:29:18.122 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:29:18.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:29:18.123 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:29:18.155 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-10T19:29:18.224 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:29:18.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:29:18.293 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:29:18.293 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:29:18.394 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:29:18.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:29:18.420 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:29:18.420 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:29:18.521 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:29:18.554 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:29:18.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:29:18.555 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:29:18.656 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:29:18.681 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:29:18 smithi183 ceph-mon[29736]: 4.d scrub starts 2024-09-10T19:29:18.681 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:29:18 smithi183 ceph-mon[29736]: 4.d scrub ok 2024-09-10T19:29:18.682 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:29:18 smithi183 ceph-mon[29736]: 5.11 scrub starts 2024-09-10T19:29:18.682 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:29:18 smithi183 ceph-mon[29736]: 5.11 scrub ok 2024-09-10T19:29:18.682 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:29:18 smithi183 ceph-mon[29736]: 11.6 scrub ok 2024-09-10T19:29:18.682 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:29:18 smithi183 ceph-mon[29736]: 6.15 scrub starts 2024-09-10T19:29:18.682 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:29:18 smithi183 ceph-mon[29736]: 6.15 scrub ok 2024-09-10T19:29:18.682 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:29:18 smithi183 ceph-mon[29736]: pgmap v887: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 8.2 GiB data, 30 GiB used, 596 GiB / 626 GiB avail; 1023 B/s rd, 88 MiB/s wr, 1.41k op/s 2024-09-10T19:29:18.682 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:29:18 smithi183 ceph-mon[41562]: 4.d scrub starts 2024-09-10T19:29:18.682 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:29:18 smithi183 ceph-mon[41562]: 4.d scrub ok 2024-09-10T19:29:18.682 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:29:18 smithi183 ceph-mon[41562]: 5.11 scrub starts 2024-09-10T19:29:18.682 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:29:18 smithi183 ceph-mon[41562]: 5.11 scrub ok 2024-09-10T19:29:18.682 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:29:18 smithi183 ceph-mon[41562]: 11.6 scrub ok 2024-09-10T19:29:18.683 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:29:18 smithi183 ceph-mon[41562]: 6.15 scrub starts 2024-09-10T19:29:18.683 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:29:18 smithi183 ceph-mon[41562]: 6.15 scrub ok 2024-09-10T19:29:18.683 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:29:18 smithi183 ceph-mon[41562]: pgmap v887: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 8.2 GiB data, 30 GiB used, 596 GiB / 626 GiB avail; 1023 B/s rd, 88 MiB/s wr, 1.41k op/s 2024-09-10T19:29:18.691 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:29:18.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:29:18.692 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:29:18.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:29:18 smithi191 ceph-mon[32884]: 4.d scrub starts 2024-09-10T19:29:18.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:29:18 smithi191 ceph-mon[32884]: 4.d scrub ok 2024-09-10T19:29:18.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:29:18 smithi191 ceph-mon[32884]: 5.11 scrub starts 2024-09-10T19:29:18.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:29:18 smithi191 ceph-mon[32884]: 5.11 scrub ok 2024-09-10T19:29:18.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:29:18 smithi191 ceph-mon[32884]: 11.6 scrub ok 2024-09-10T19:29:18.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:29:18 smithi191 ceph-mon[32884]: 6.15 scrub starts 2024-09-10T19:29:18.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:29:18 smithi191 ceph-mon[32884]: 6.15 scrub ok 2024-09-10T19:29:18.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:29:18 smithi191 ceph-mon[32884]: pgmap v887: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 8.2 GiB data, 30 GiB used, 596 GiB / 626 GiB avail; 1023 B/s rd, 88 MiB/s wr, 1.41k op/s 2024-09-10T19:29:18.793 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:29:18.828 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:29:18.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:29:18.829 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:29:18.929 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:29:18.966 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:29:18.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:29:18.967 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:29:19.068 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:29:19.075 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: 85 16 138097 138081 101.508 74.3125 0.00982583 0.00984707 2024-09-10T19:29:19.104 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:29:19.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:29:19.106 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:29:19.206 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:29:19.239 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:29:19.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:29:19.240 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:29:19.341 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:29:19.378 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:29:19.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:29:19.380 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:29:19.419 INFO:teuthology.orchestra.run.smithi183.stderr:nodeep-scrub is unset 2024-09-10T19:29:19.424 INFO:teuthology.orchestra.run.smithi183.stderr:reweighted osd.3 to 1 (10000) 2024-09-10T19:29:19.481 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:29:19.521 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:29:19.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:29:19.522 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:29:19.623 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:29:19.658 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:29:19.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:29:19.659 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:29:19.681 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:29:19 smithi183 ceph-mon[29736]: 4.9 scrub starts 2024-09-10T19:29:19.681 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:29:19 smithi183 ceph-mon[29736]: 4.9 scrub ok 2024-09-10T19:29:19.682 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:29:19 smithi183 ceph-mon[29736]: 2.18 scrub starts 2024-09-10T19:29:19.682 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:29:19 smithi183 ceph-mon[29736]: 2.18 scrub ok 2024-09-10T19:29:19.682 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:29:19 smithi183 ceph-mon[29736]: from='client.? 172.21.15.183:0/3618934900' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-09-10T19:29:19.682 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:29:19 smithi183 ceph-mon[29736]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-09-10T19:29:19.682 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:29:19 smithi183 ceph-mon[29736]: 2.10 scrub starts 2024-09-10T19:29:19.682 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:29:19 smithi183 ceph-mon[29736]: 2.10 scrub ok 2024-09-10T19:29:19.682 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:29:19 smithi183 ceph-mon[29736]: from='client.? 172.21.15.183:0/296175456' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-10T19:29:19.682 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:29:19 smithi183 ceph-mon[29736]: 11.3 scrub ok 2024-09-10T19:29:19.682 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:29:19 smithi183 ceph-mon[29736]: 5.13 scrub starts 2024-09-10T19:29:19.682 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:29:19 smithi183 ceph-mon[29736]: 5.13 scrub ok 2024-09-10T19:29:19.682 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:29:19 smithi183 ceph-mon[41562]: 4.9 scrub starts 2024-09-10T19:29:19.683 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:29:19 smithi183 ceph-mon[41562]: 4.9 scrub ok 2024-09-10T19:29:19.683 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:29:19 smithi183 ceph-mon[41562]: 2.18 scrub starts 2024-09-10T19:29:19.683 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:29:19 smithi183 ceph-mon[41562]: 2.18 scrub ok 2024-09-10T19:29:19.683 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:29:19 smithi183 ceph-mon[41562]: from='client.? 172.21.15.183:0/3618934900' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-09-10T19:29:19.683 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:29:19 smithi183 ceph-mon[41562]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-09-10T19:29:19.683 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:29:19 smithi183 ceph-mon[41562]: 2.10 scrub starts 2024-09-10T19:29:19.683 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:29:19 smithi183 ceph-mon[41562]: 2.10 scrub ok 2024-09-10T19:29:19.683 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:29:19 smithi183 ceph-mon[41562]: from='client.? 172.21.15.183:0/296175456' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-10T19:29:19.683 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:29:19 smithi183 ceph-mon[41562]: 11.3 scrub ok 2024-09-10T19:29:19.683 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:29:19 smithi183 ceph-mon[41562]: 5.13 scrub starts 2024-09-10T19:29:19.683 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:29:19 smithi183 ceph-mon[41562]: 5.13 scrub ok 2024-09-10T19:29:19.715 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:29:19 smithi191 ceph-mon[32884]: 4.9 scrub starts 2024-09-10T19:29:19.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:29:19 smithi191 ceph-mon[32884]: 4.9 scrub ok 2024-09-10T19:29:19.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:29:19 smithi191 ceph-mon[32884]: 2.18 scrub starts 2024-09-10T19:29:19.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:29:19 smithi191 ceph-mon[32884]: 2.18 scrub ok 2024-09-10T19:29:19.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:29:19 smithi191 ceph-mon[32884]: from='client.? 172.21.15.183:0/3618934900' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-09-10T19:29:19.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:29:19 smithi191 ceph-mon[32884]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-09-10T19:29:19.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:29:19 smithi191 ceph-mon[32884]: 2.10 scrub starts 2024-09-10T19:29:19.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:29:19 smithi191 ceph-mon[32884]: 2.10 scrub ok 2024-09-10T19:29:19.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:29:19 smithi191 ceph-mon[32884]: from='client.? 172.21.15.183:0/296175456' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-10T19:29:19.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:29:19 smithi191 ceph-mon[32884]: 11.3 scrub ok 2024-09-10T19:29:19.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:29:19 smithi191 ceph-mon[32884]: 5.13 scrub starts 2024-09-10T19:29:19.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:29:19 smithi191 ceph-mon[32884]: 5.13 scrub ok 2024-09-10T19:29:19.761 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:29:19.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:29:19.796 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:29:19.797 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:29:19.897 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:29:19.939 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:29:19.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:29:19.940 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:29:20.041 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:29:20.075 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: 86 16 139522 139506 101.364 89.0625 0.00637435 0.00986118 2024-09-10T19:29:20.083 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:29:20.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:29:20.084 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:29:20.185 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:29:20.218 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:29:20.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:29:20.219 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:29:20.320 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:29:20.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:29:20.356 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:29:20.356 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:29:20.446 INFO:journalctl@ceph.iscsi.iscsi.a.smithi191.stdout:Sep 10 19:29:20 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-iscsi-iscsi-a[61816]: debug there is no tcmu-runner data available 2024-09-10T19:29:20.458 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:29:20.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:29:20.491 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:29:20.491 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:29:20.592 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:29:20.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:29:20.626 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:29:20.626 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:29:20.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:29:20 smithi191 ceph-mon[32884]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-10T19:29:20.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:29:20 smithi191 ceph-mon[32884]: Cluster is now healthy 2024-09-10T19:29:20.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:29:20 smithi191 ceph-mon[32884]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-09-10T19:29:20.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:29:20 smithi191 ceph-mon[32884]: from='client.? 172.21.15.183:0/296175456' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-10T19:29:20.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:29:20 smithi191 ceph-mon[32884]: osdmap e401: 8 total, 8 up, 7 in 2024-09-10T19:29:20.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:29:20 smithi191 ceph-mon[32884]: pgmap v889: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 8.2 GiB data, 30 GiB used, 596 GiB / 626 GiB avail; 1023 B/s rd, 88 MiB/s wr, 1.41k op/s 2024-09-10T19:29:20.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:29:20 smithi191 ceph-mon[32884]: from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:29:20.727 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:29:20.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:29:20.768 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:29:20.768 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:29:20.869 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:29:20.908 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:29:20.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:29:20.909 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:29:20.931 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:29:20 smithi183 ceph-mon[29736]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-10T19:29:20.932 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:29:20 smithi183 ceph-mon[29736]: Cluster is now healthy 2024-09-10T19:29:20.932 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:29:20 smithi183 ceph-mon[29736]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-09-10T19:29:20.932 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:29:20 smithi183 ceph-mon[29736]: from='client.? 172.21.15.183:0/296175456' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-10T19:29:20.932 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:29:20 smithi183 ceph-mon[29736]: osdmap e401: 8 total, 8 up, 7 in 2024-09-10T19:29:20.932 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:29:20 smithi183 ceph-mon[29736]: pgmap v889: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 8.2 GiB data, 30 GiB used, 596 GiB / 626 GiB avail; 1023 B/s rd, 88 MiB/s wr, 1.41k op/s 2024-09-10T19:29:20.933 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:29:20 smithi183 ceph-mon[29736]: from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:29:20.933 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:29:20 smithi183 ceph-mon[41562]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-10T19:29:20.933 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:29:20 smithi183 ceph-mon[41562]: Cluster is now healthy 2024-09-10T19:29:20.933 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:29:20 smithi183 ceph-mon[41562]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-09-10T19:29:20.933 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:29:20 smithi183 ceph-mon[41562]: from='client.? 172.21.15.183:0/296175456' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-10T19:29:20.934 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:29:20 smithi183 ceph-mon[41562]: osdmap e401: 8 total, 8 up, 7 in 2024-09-10T19:29:20.934 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:29:20 smithi183 ceph-mon[41562]: pgmap v889: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 8.2 GiB data, 30 GiB used, 596 GiB / 626 GiB avail; 1023 B/s rd, 88 MiB/s wr, 1.41k op/s 2024-09-10T19:29:20.934 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:29:20 smithi183 ceph-mon[41562]: from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:29:21.011 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:29:21.054 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:29:21.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:29:21.055 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:29:21.075 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: 87 16 140998 140982 101.259 92.25 0.00687513 0.00987149 2024-09-10T19:29:21.156 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:29:21.191 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:29:21.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:29:21.192 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:29:21.293 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:29:21.319 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph daemon osd.2 dump_blocked_ops 2024-09-10T19:29:21.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:29:21.327 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:29:21.327 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:29:21.428 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:29:21.466 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:29:21.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:29:21.467 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:29:21.568 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:29:21.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:29:21.606 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:29:21.606 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:29:21.607 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:29:21 smithi183 ceph-mon[29736]: 2.4 scrub starts 2024-09-10T19:29:21.607 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:29:21 smithi183 ceph-mon[29736]: 2.4 scrub ok 2024-09-10T19:29:21.607 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:29:21 smithi183 ceph-mon[29736]: 4.1e scrub starts 2024-09-10T19:29:21.607 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:29:21 smithi183 ceph-mon[29736]: 4.1e scrub ok 2024-09-10T19:29:21.608 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:29:21 smithi183 ceph-mon[41562]: 2.4 scrub starts 2024-09-10T19:29:21.608 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:29:21 smithi183 ceph-mon[41562]: 2.4 scrub ok 2024-09-10T19:29:21.608 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:29:21 smithi183 ceph-mon[41562]: 4.1e scrub starts 2024-09-10T19:29:21.608 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:29:21 smithi183 ceph-mon[41562]: 4.1e scrub ok 2024-09-10T19:29:21.608 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Sep 10 19:29:21 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-y[29989]: ::ffff:172.21.15.191 - - [10/Sep/2024:19:29:21] "GET /metrics HTTP/1.1" 200 224897 "" "Prometheus/2.43.0" 2024-09-10T19:29:21.707 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:29:21.715 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:29:21 smithi191 ceph-mon[32884]: 2.4 scrub starts 2024-09-10T19:29:21.715 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:29:21 smithi191 ceph-mon[32884]: 2.4 scrub ok 2024-09-10T19:29:21.715 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:29:21 smithi191 ceph-mon[32884]: 4.1e scrub starts 2024-09-10T19:29:21.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:29:21 smithi191 ceph-mon[32884]: 4.1e scrub ok 2024-09-10T19:29:21.750 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:29:21.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:29:21.751 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:29:21.852 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:29:21.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:29:21.890 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:29:21.890 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:29:21.991 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:29:22.025 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:29:22.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:29:22.026 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:29:22.075 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: 88 16 142345 142329 101.065 84.1875 0.00492351 0.00989061 2024-09-10T19:29:22.127 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:29:22.172 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:29:22.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:29:22.173 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:29:22.273 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:29:22.309 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:29:22.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:29:22.310 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:29:22.411 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:29:22.449 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:29:22.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:29:22.450 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:29:22.551 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:29:22.593 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:29:22.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:29:22.594 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:29:22.695 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:29:22.707 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph osd reweight 4 1 2024-09-10T19:29:22.710 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:29:22 smithi183 ceph-mon[29736]: 1.9 scrub starts 2024-09-10T19:29:22.710 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:29:22 smithi183 ceph-mon[29736]: 1.9 scrub ok 2024-09-10T19:29:22.710 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:29:22 smithi183 ceph-mon[29736]: 5.1b scrub starts 2024-09-10T19:29:22.711 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:29:22 smithi183 ceph-mon[29736]: 5.1b scrub ok 2024-09-10T19:29:22.711 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:29:22 smithi183 ceph-mon[29736]: 5.1d scrub starts 2024-09-10T19:29:22.711 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:29:22 smithi183 ceph-mon[29736]: 5.1d scrub ok 2024-09-10T19:29:22.711 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:29:22 smithi183 ceph-mon[29736]: pgmap v890: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 8.5 GiB data, 30 GiB used, 595 GiB / 626 GiB avail; 1023 B/s rd, 86 MiB/s wr, 1.38k op/s 2024-09-10T19:29:22.713 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:29:22 smithi183 ceph-mon[41562]: 1.9 scrub starts 2024-09-10T19:29:22.713 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:29:22 smithi183 ceph-mon[41562]: 1.9 scrub ok 2024-09-10T19:29:22.713 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:29:22 smithi183 ceph-mon[41562]: 5.1b scrub starts 2024-09-10T19:29:22.713 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:29:22 smithi183 ceph-mon[41562]: 5.1b scrub ok 2024-09-10T19:29:22.713 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:29:22 smithi183 ceph-mon[41562]: 5.1d scrub starts 2024-09-10T19:29:22.713 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:29:22 smithi183 ceph-mon[41562]: 5.1d scrub ok 2024-09-10T19:29:22.713 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:29:22 smithi183 ceph-mon[41562]: pgmap v890: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 8.5 GiB data, 30 GiB used, 595 GiB / 626 GiB avail; 1023 B/s rd, 86 MiB/s wr, 1.38k op/s 2024-09-10T19:29:22.715 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:29:22 smithi191 ceph-mon[32884]: 1.9 scrub starts 2024-09-10T19:29:22.715 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:29:22 smithi191 ceph-mon[32884]: 1.9 scrub ok 2024-09-10T19:29:22.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:29:22 smithi191 ceph-mon[32884]: 5.1b scrub starts 2024-09-10T19:29:22.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:29:22 smithi191 ceph-mon[32884]: 5.1b scrub ok 2024-09-10T19:29:22.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:29:22 smithi191 ceph-mon[32884]: 5.1d scrub starts 2024-09-10T19:29:22.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:29:22 smithi191 ceph-mon[32884]: 5.1d scrub ok 2024-09-10T19:29:22.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:29:22 smithi191 ceph-mon[32884]: pgmap v890: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 8.5 GiB data, 30 GiB used, 595 GiB / 626 GiB avail; 1023 B/s rd, 86 MiB/s wr, 1.38k op/s 2024-09-10T19:29:22.723 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:29:22.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:29:22.724 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:29:22.745 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:29:22.784 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:29:22.825 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:29:22.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:29:22.877 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:29:22.877 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:29:22.978 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:29:23.012 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:29:23.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:29:23.014 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:29:23.076 INFO:tasks.radosbench.radosbench.0.smithi183.stdout: 89 16 143583 143567 100.798 77.375 0.0154537 0.00991684 2024-09-10T19:29:23.115 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:29:23.161 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:29:23.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:29:23.162 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:29:23.263 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:29:23.307 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:29:23.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:29:23.308 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:29:23.409 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:29:23.449 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:29:23.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:29:23.450 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:29:23.551 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:29:23.585 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:29:23.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:29:23.586 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:29:23.681 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:29:23 smithi183 ceph-mon[41562]: 2.2 scrub starts 2024-09-10T19:29:23.682 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:29:23 smithi183 ceph-mon[41562]: 2.2 scrub ok 2024-09-10T19:29:23.682 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:29:23 smithi183 ceph-mon[41562]: 2.11 scrub starts 2024-09-10T19:29:23.682 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:29:23 smithi183 ceph-mon[41562]: 2.11 scrub ok 2024-09-10T19:29:23.682 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:29:23 smithi183 ceph-mon[41562]: 6.1 deep-scrub starts 2024-09-10T19:29:23.682 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:29:23 smithi183 ceph-mon[41562]: 6.1 deep-scrub ok 2024-09-10T19:29:23.682 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:29:23 smithi183 ceph-mon[29736]: 2.2 scrub starts 2024-09-10T19:29:23.682 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:29:23 smithi183 ceph-mon[29736]: 2.2 scrub ok 2024-09-10T19:29:23.682 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:29:23 smithi183 ceph-mon[29736]: 2.11 scrub starts 2024-09-10T19:29:23.682 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:29:23 smithi183 ceph-mon[29736]: 2.11 scrub ok 2024-09-10T19:29:23.682 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:29:23 smithi183 ceph-mon[29736]: 6.1 deep-scrub starts 2024-09-10T19:29:23.682 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:29:23 smithi183 ceph-mon[29736]: 6.1 deep-scrub ok 2024-09-10T19:29:23.687 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:29:23.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:29:23 smithi191 ceph-mon[32884]: 2.2 scrub starts 2024-09-10T19:29:23.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:29:23 smithi191 ceph-mon[32884]: 2.2 scrub ok 2024-09-10T19:29:23.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:29:23 smithi191 ceph-mon[32884]: 2.11 scrub starts 2024-09-10T19:29:23.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:29:23 smithi191 ceph-mon[32884]: 2.11 scrub ok 2024-09-10T19:29:23.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:29:23 smithi191 ceph-mon[32884]: 6.1 deep-scrub starts 2024-09-10T19:29:23.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:29:23 smithi191 ceph-mon[32884]: 6.1 deep-scrub ok 2024-09-10T19:29:23.726 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:29:23.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:29:23.727 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:29:23.828 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:29:23.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:29:23.867 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:29:23.867 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:29:23.968 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:29:24.001 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:29:24.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:29:24.002 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:29:24.076 INFO:tasks.radosbench.radosbench.0.smithi183.stdout:Total time run: 90.0106 2024-09-10T19:29:24.076 INFO:tasks.radosbench.radosbench.0.smithi183.stdout:Total writes made: 144887 2024-09-10T19:29:24.076 INFO:tasks.radosbench.radosbench.0.smithi183.stdout:Write size: 65536 2024-09-10T19:29:24.077 INFO:tasks.radosbench.radosbench.0.smithi183.stdout:Object size: 65536 2024-09-10T19:29:24.077 INFO:tasks.radosbench.radosbench.0.smithi183.stdout:Bandwidth (MB/sec): 100.604 2024-09-10T19:29:24.077 INFO:tasks.radosbench.radosbench.0.smithi183.stdout:Stddev Bandwidth: 33.9199 2024-09-10T19:29:24.077 INFO:tasks.radosbench.radosbench.0.smithi183.stdout:Max bandwidth (MB/sec): 238.938 2024-09-10T19:29:24.077 INFO:tasks.radosbench.radosbench.0.smithi183.stdout:Min bandwidth (MB/sec): 47.5625 2024-09-10T19:29:24.077 INFO:tasks.radosbench.radosbench.0.smithi183.stdout:Average IOPS: 1609 2024-09-10T19:29:24.077 INFO:tasks.radosbench.radosbench.0.smithi183.stdout:Stddev IOPS: 542.719 2024-09-10T19:29:24.077 INFO:tasks.radosbench.radosbench.0.smithi183.stdout:Max IOPS: 3823 2024-09-10T19:29:24.077 INFO:tasks.radosbench.radosbench.0.smithi183.stdout:Min IOPS: 761 2024-09-10T19:29:24.077 INFO:tasks.radosbench.radosbench.0.smithi183.stdout:Average Latency(s): 0.00993596 2024-09-10T19:29:24.078 INFO:tasks.radosbench.radosbench.0.smithi183.stdout:Stddev Latency(s): 0.00922723 2024-09-10T19:29:24.078 INFO:tasks.radosbench.radosbench.0.smithi183.stdout:Max latency(s): 0.573472 2024-09-10T19:29:24.078 INFO:tasks.radosbench.radosbench.0.smithi183.stdout:Min latency(s): 0.00134808 2024-09-10T19:29:24.081 INFO:tasks.radosbench.radosbench.0.smithi183.stdout:Cleaning up (deleting benchmark objects) 2024-09-10T19:29:24.103 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:29:24.121 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph osd set noscrub 2024-09-10T19:29:24.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:29:24.137 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:29:24.138 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:29:24.239 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:29:24.276 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:29:24.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:29:24.277 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:29:24.377 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:29:24.414 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:29:24.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:29:24.415 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:29:24.516 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:29:24.558 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:29:24.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:29:24.559 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:29:24.660 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:29:24.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:29:24.693 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:29:24.693 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:29:24.794 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:29:24.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:29:24.828 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:29:24.828 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:29:24.929 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:29:24.931 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:29:24 smithi183 ceph-mon[29736]: 11.5 scrub starts 2024-09-10T19:29:24.931 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:29:24 smithi183 ceph-mon[29736]: 5.1 scrub starts 2024-09-10T19:29:24.931 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:29:24 smithi183 ceph-mon[29736]: 5.1 scrub ok 2024-09-10T19:29:24.931 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:29:24 smithi183 ceph-mon[29736]: pgmap v891: 158 pgs: 158 active+clean; 8.6 GiB data, 31 GiB used, 595 GiB / 626 GiB avail; 1023 B/s rd, 86 MiB/s wr, 1.38k op/s 2024-09-10T19:29:24.932 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:29:24 smithi183 ceph-mon[41562]: 11.5 scrub starts 2024-09-10T19:29:24.932 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:29:24 smithi183 ceph-mon[41562]: 5.1 scrub starts 2024-09-10T19:29:24.932 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:29:24 smithi183 ceph-mon[41562]: 5.1 scrub ok 2024-09-10T19:29:24.932 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:29:24 smithi183 ceph-mon[41562]: pgmap v891: 158 pgs: 158 active+clean; 8.6 GiB data, 31 GiB used, 595 GiB / 626 GiB avail; 1023 B/s rd, 86 MiB/s wr, 1.38k op/s 2024-09-10T19:29:24.966 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:29:24.966 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:29:24 smithi191 ceph-mon[32884]: 11.5 scrub starts 2024-09-10T19:29:24.966 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:29:24 smithi191 ceph-mon[32884]: 5.1 scrub starts 2024-09-10T19:29:24.966 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:29:24 smithi191 ceph-mon[32884]: 5.1 scrub ok 2024-09-10T19:29:24.966 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:29:24 smithi191 ceph-mon[32884]: pgmap v891: 158 pgs: 158 active+clean; 8.6 GiB data, 31 GiB used, 595 GiB / 626 GiB avail; 1023 B/s rd, 86 MiB/s wr, 1.38k op/s 2024-09-10T19:29:24.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:29:24.967 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:29:25.068 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:29:25.096 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:29:25.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:29:25.097 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:29:25.198 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:29:25.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:29:25.237 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:29:25.237 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:29:25.338 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:29:25.378 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:29:25.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:29:25.379 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:29:25.480 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:29:25.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:29:25.514 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:29:25.514 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:29:25.615 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:29:25.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:29:25.648 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:29:25.648 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:29:25.749 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:29:25.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:29:25.782 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:29:25.783 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:29:25.883 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:29:25.915 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:29:25.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:29:25.916 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:29:25.931 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:29:25 smithi183 ceph-mon[29736]: 5.1c scrub starts 2024-09-10T19:29:25.931 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:29:25 smithi183 ceph-mon[29736]: 5.1c scrub ok 2024-09-10T19:29:25.931 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:29:25 smithi183 ceph-mon[29736]: 6.1f scrub starts 2024-09-10T19:29:25.931 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:29:25 smithi183 ceph-mon[29736]: 6.1f scrub ok 2024-09-10T19:29:25.932 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:29:25 smithi183 ceph-mon[41562]: 5.1c scrub starts 2024-09-10T19:29:25.932 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:29:25 smithi183 ceph-mon[41562]: 5.1c scrub ok 2024-09-10T19:29:25.932 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:29:25 smithi183 ceph-mon[41562]: 6.1f scrub starts 2024-09-10T19:29:25.932 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:29:25 smithi183 ceph-mon[41562]: 6.1f scrub ok 2024-09-10T19:29:25.966 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:29:25 smithi191 ceph-mon[32884]: 5.1c scrub starts 2024-09-10T19:29:25.966 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:29:25 smithi191 ceph-mon[32884]: 5.1c scrub ok 2024-09-10T19:29:25.966 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:29:25 smithi191 ceph-mon[32884]: 6.1f scrub starts 2024-09-10T19:29:25.966 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:29:25 smithi191 ceph-mon[32884]: 6.1f scrub ok 2024-09-10T19:29:26.017 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:29:26.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:29:26.049 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:29:26.049 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:29:26.150 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:29:26.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:29:26.180 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:29:26.180 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:29:26.281 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:29:26.314 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:29:26.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:29:26.315 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:29:26.416 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:29:26.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:29:26.451 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:29:26.451 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:29:26.552 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:29:26.585 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:29:26.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:29:26.586 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:29:26.687 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:29:26.719 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:29:26.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:29:26.720 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:29:26.821 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:29:26.851 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:29:26.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:29:26.853 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:29:26.931 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:29:26 smithi183 ceph-mon[29736]: 11.4 scrub starts 2024-09-10T19:29:26.932 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:29:26 smithi183 ceph-mon[29736]: 1.8 scrub starts 2024-09-10T19:29:26.932 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:29:26 smithi183 ceph-mon[29736]: 1.8 scrub ok 2024-09-10T19:29:26.932 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:29:26 smithi183 ceph-mon[29736]: 11.5 scrub ok 2024-09-10T19:29:26.932 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:29:26 smithi183 ceph-mon[29736]: pgmap v892: 158 pgs: 2 active+clean+scrubbing, 156 active+clean; 8.7 GiB data, 31 GiB used, 594 GiB / 626 GiB avail; 1.1 KiB/s rd, 76 MiB/s wr, 1.34k op/s 2024-09-10T19:29:26.932 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:29:26 smithi183 ceph-mon[41562]: 11.4 scrub starts 2024-09-10T19:29:26.932 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:29:26 smithi183 ceph-mon[41562]: 1.8 scrub starts 2024-09-10T19:29:26.932 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:29:26 smithi183 ceph-mon[41562]: 1.8 scrub ok 2024-09-10T19:29:26.932 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:29:26 smithi183 ceph-mon[41562]: 11.5 scrub ok 2024-09-10T19:29:26.932 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:29:26 smithi183 ceph-mon[41562]: pgmap v892: 158 pgs: 2 active+clean+scrubbing, 156 active+clean; 8.7 GiB data, 31 GiB used, 594 GiB / 626 GiB avail; 1.1 KiB/s rd, 76 MiB/s wr, 1.34k op/s 2024-09-10T19:29:26.954 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:29:26.968 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:29:26 smithi191 ceph-mon[32884]: 11.4 scrub starts 2024-09-10T19:29:26.968 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:29:26 smithi191 ceph-mon[32884]: 1.8 scrub starts 2024-09-10T19:29:26.968 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:29:26 smithi191 ceph-mon[32884]: 1.8 scrub ok 2024-09-10T19:29:26.968 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:29:26 smithi191 ceph-mon[32884]: 11.5 scrub ok 2024-09-10T19:29:26.968 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:29:26 smithi191 ceph-mon[32884]: pgmap v892: 158 pgs: 2 active+clean+scrubbing, 156 active+clean; 8.7 GiB data, 31 GiB used, 594 GiB / 626 GiB avail; 1.1 KiB/s rd, 76 MiB/s wr, 1.34k op/s 2024-09-10T19:29:26.989 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:29:26.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:29:26.990 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:29:27.090 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:29:27.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:29:27.130 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:29:27.131 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:29:27.231 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:29:27.268 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:29:27.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:29:27.269 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:29:27.370 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:29:27.405 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:29:27.408 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:29:27.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:29:27.409 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:29:27.510 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:29:27.543 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:29:27.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:29:27.544 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:29:27.645 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:29:27.668 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:29:27.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:29:27.669 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:29:27.715 INFO:journalctl@ceph.mgr.x.smithi191.stdout:Sep 10 19:29:27 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-x[34967]: ::ffff:172.21.15.191 - - [10/Sep/2024:19:29:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T19:29:27.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:29:27 smithi191 ceph-mon[32884]: 4.13 scrub starts 2024-09-10T19:29:27.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:29:27 smithi191 ceph-mon[32884]: 4.13 scrub ok 2024-09-10T19:29:27.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:29:27 smithi191 ceph-mon[32884]: 5.15 scrub starts 2024-09-10T19:29:27.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:29:27 smithi191 ceph-mon[32884]: 5.15 scrub ok 2024-09-10T19:29:27.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:29:27 smithi191 ceph-mon[32884]: 11.1 scrub starts 2024-09-10T19:29:27.770 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:29:27.799 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:29:27 smithi183 ceph-mon[29736]: 4.13 scrub starts 2024-09-10T19:29:27.799 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:29:27 smithi183 ceph-mon[29736]: 4.13 scrub ok 2024-09-10T19:29:27.800 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:29:27 smithi183 ceph-mon[29736]: 5.15 scrub starts 2024-09-10T19:29:27.800 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:29:27 smithi183 ceph-mon[29736]: 5.15 scrub ok 2024-09-10T19:29:27.800 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:29:27 smithi183 ceph-mon[29736]: 11.1 scrub starts 2024-09-10T19:29:27.800 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:29:27 smithi183 ceph-mon[41562]: 4.13 scrub starts 2024-09-10T19:29:27.800 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:29:27 smithi183 ceph-mon[41562]: 4.13 scrub ok 2024-09-10T19:29:27.800 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:29:27 smithi183 ceph-mon[41562]: 5.15 scrub starts 2024-09-10T19:29:27.800 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:29:27 smithi183 ceph-mon[41562]: 5.15 scrub ok 2024-09-10T19:29:27.800 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:29:27 smithi183 ceph-mon[41562]: 11.1 scrub starts 2024-09-10T19:29:27.802 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:29:27.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:29:27.803 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:29:27.903 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:29:27.941 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:29:27.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:29:27.942 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:29:28.042 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:29:28.077 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:29:28.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:29:28.078 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:29:28.179 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:29:28.214 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:29:28.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:29:28.215 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:29:28.306 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:29:28.317 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:29:28.346 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:29:28.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:29:28.347 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:29:28.448 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:29:28.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:29:28.485 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:29:28.485 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:29:28.586 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:29:28.624 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:29:28.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:29:28.625 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:29:28.726 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:29:28.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:29:28.758 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:29:28.759 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:29:28.772 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:29:28 smithi183 ceph-mon[41562]: 6.2 scrub starts 2024-09-10T19:29:28.772 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:29:28 smithi183 ceph-mon[41562]: 6.2 scrub ok 2024-09-10T19:29:28.772 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:29:28 smithi183 ceph-mon[41562]: 6.a deep-scrub starts 2024-09-10T19:29:28.772 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:29:28 smithi183 ceph-mon[41562]: 6.a deep-scrub ok 2024-09-10T19:29:28.773 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:29:28 smithi183 ceph-mon[41562]: pgmap v893: 158 pgs: 2 active+clean+scrubbing, 156 active+clean; 8.7 GiB data, 31 GiB used, 594 GiB / 626 GiB avail; 1.1 KiB/s rd, 70 MiB/s wr, 1.34k op/s 2024-09-10T19:29:28.773 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:29:28 smithi183 ceph-mon[29736]: 6.2 scrub starts 2024-09-10T19:29:28.773 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:29:28 smithi183 ceph-mon[29736]: 6.2 scrub ok 2024-09-10T19:29:28.773 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:29:28 smithi183 ceph-mon[29736]: 6.a deep-scrub starts 2024-09-10T19:29:28.773 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:29:28 smithi183 ceph-mon[29736]: 6.a deep-scrub ok 2024-09-10T19:29:28.773 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:29:28 smithi183 ceph-mon[29736]: pgmap v893: 158 pgs: 2 active+clean+scrubbing, 156 active+clean; 8.7 GiB data, 31 GiB used, 594 GiB / 626 GiB avail; 1.1 KiB/s rd, 70 MiB/s wr, 1.34k op/s 2024-09-10T19:29:28.860 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:29:28.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:29:28.899 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:29:28.899 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:29:28.965 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:29:28 smithi191 ceph-mon[32884]: 6.2 scrub starts 2024-09-10T19:29:28.966 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:29:28 smithi191 ceph-mon[32884]: 6.2 scrub ok 2024-09-10T19:29:28.966 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:29:28 smithi191 ceph-mon[32884]: 6.a deep-scrub starts 2024-09-10T19:29:28.966 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:29:28 smithi191 ceph-mon[32884]: 6.a deep-scrub ok 2024-09-10T19:29:28.966 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:29:28 smithi191 ceph-mon[32884]: pgmap v893: 158 pgs: 2 active+clean+scrubbing, 156 active+clean; 8.7 GiB data, 31 GiB used, 594 GiB / 626 GiB avail; 1.1 KiB/s rd, 70 MiB/s wr, 1.34k op/s 2024-09-10T19:29:29.001 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:29:29.037 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:29:29.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:29:29.038 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:29:29.139 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:29:29.172 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:29:29.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:29:29.173 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:29:29.274 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:29:29.297 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:29:29.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:29:29.298 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:29:29.398 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:29:29.439 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:29:29.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:29:29.440 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:29:29.541 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:29:29.578 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:29:29.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:29:29.579 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:29:29.680 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:29:29.715 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:29:29.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:29:29.716 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:29:29.795 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:29:29 smithi183 ceph-mon[29736]: 6.14 scrub starts 2024-09-10T19:29:29.795 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:29:29 smithi183 ceph-mon[29736]: 4.1f scrub starts 2024-09-10T19:29:29.795 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:29:29 smithi183 ceph-mon[29736]: 4.1f scrub ok 2024-09-10T19:29:29.795 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:29:29 smithi183 ceph-mon[29736]: 6.14 scrub ok 2024-09-10T19:29:29.795 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:29:29 smithi183 ceph-mon[29736]: 6.10 scrub starts 2024-09-10T19:29:29.796 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:29:29 smithi183 ceph-mon[29736]: 6.10 scrub ok 2024-09-10T19:29:29.796 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:29:29 smithi183 ceph-mon[29736]: from='mgr.14556 ' entity='mgr.y' 2024-09-10T19:29:29.796 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:29:29 smithi183 ceph-mon[29736]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:29:29.796 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:29:29 smithi183 ceph-mon[29736]: 11.4 scrub ok 2024-09-10T19:29:29.796 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:29:29 smithi183 ceph-mon[29736]: 5.17 scrub starts 2024-09-10T19:29:29.796 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:29:29 smithi183 ceph-mon[29736]: 5.17 scrub ok 2024-09-10T19:29:29.796 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:29:29 smithi183 ceph-mon[41562]: 6.14 scrub starts 2024-09-10T19:29:29.796 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:29:29 smithi183 ceph-mon[41562]: 4.1f scrub starts 2024-09-10T19:29:29.796 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:29:29 smithi183 ceph-mon[41562]: 4.1f scrub ok 2024-09-10T19:29:29.796 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:29:29 smithi183 ceph-mon[41562]: 6.14 scrub ok 2024-09-10T19:29:29.797 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:29:29 smithi183 ceph-mon[41562]: 6.10 scrub starts 2024-09-10T19:29:29.797 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:29:29 smithi183 ceph-mon[41562]: 6.10 scrub ok 2024-09-10T19:29:29.797 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:29:29 smithi183 ceph-mon[41562]: from='mgr.14556 ' entity='mgr.y' 2024-09-10T19:29:29.797 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:29:29 smithi183 ceph-mon[41562]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:29:29.797 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:29:29 smithi183 ceph-mon[41562]: 11.4 scrub ok 2024-09-10T19:29:29.797 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:29:29 smithi183 ceph-mon[41562]: 5.17 scrub starts 2024-09-10T19:29:29.797 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:29:29 smithi183 ceph-mon[41562]: 5.17 scrub ok 2024-09-10T19:29:29.816 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:29:29.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:29:29.855 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:29:29.855 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:29:29.956 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:29:29.965 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:29:29 smithi191 ceph-mon[32884]: 6.14 scrub starts 2024-09-10T19:29:29.966 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:29:29 smithi191 ceph-mon[32884]: 4.1f scrub starts 2024-09-10T19:29:29.966 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:29:29 smithi191 ceph-mon[32884]: 4.1f scrub ok 2024-09-10T19:29:29.966 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:29:29 smithi191 ceph-mon[32884]: 6.14 scrub ok 2024-09-10T19:29:29.966 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:29:29 smithi191 ceph-mon[32884]: 6.10 scrub starts 2024-09-10T19:29:29.966 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:29:29 smithi191 ceph-mon[32884]: 6.10 scrub ok 2024-09-10T19:29:29.966 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:29:29 smithi191 ceph-mon[32884]: from='mgr.14556 ' entity='mgr.y' 2024-09-10T19:29:29.966 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:29:29 smithi191 ceph-mon[32884]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:29:29.966 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:29:29 smithi191 ceph-mon[32884]: 11.4 scrub ok 2024-09-10T19:29:29.966 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:29:29 smithi191 ceph-mon[32884]: 5.17 scrub starts 2024-09-10T19:29:29.966 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:29:29 smithi191 ceph-mon[32884]: 5.17 scrub ok 2024-09-10T19:29:29.985 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:29:29.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:29:29.986 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:29:30.086 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:29:30.118 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:29:30.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:29:30.119 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:29:30.220 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:29:30.259 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:29:30.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:29:30.260 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:29:30.360 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:29:30.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:29:30.397 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:29:30.397 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:29:30.465 INFO:journalctl@ceph.iscsi.iscsi.a.smithi191.stdout:Sep 10 19:29:30 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-iscsi-iscsi-a[61816]: debug there is no tcmu-runner data available 2024-09-10T19:29:30.498 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:29:30.535 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:29:30.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:29:30.536 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:29:30.637 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:29:30.670 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:29:30.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:29:30.671 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:29:30.695 INFO:teuthology.orchestra.run.smithi183.stdout:{ 2024-09-10T19:29:30.695 INFO:teuthology.orchestra.run.smithi183.stdout: "ops": [], 2024-09-10T19:29:30.695 INFO:teuthology.orchestra.run.smithi183.stdout: "complaint_time": 30, 2024-09-10T19:29:30.695 INFO:teuthology.orchestra.run.smithi183.stdout: "num_blocked_ops": 0 2024-09-10T19:29:30.695 INFO:teuthology.orchestra.run.smithi183.stdout:} 2024-09-10T19:29:30.772 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:29:30.808 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:29:30.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:29:30.810 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:29:30.910 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:29:30.945 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:29:30.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:29:30.946 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:29:31.047 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:29:31.080 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:29:31.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:29:31.081 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:29:31.182 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:29:31.218 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:29:31.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:29:31.219 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:29:31.316 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:29:30 smithi183 ceph-mon[29736]: purged_snaps scrub starts 2024-09-10T19:29:31.317 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:29:30 smithi183 ceph-mon[29736]: purged_snaps scrub ok 2024-09-10T19:29:31.317 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:29:30 smithi183 ceph-mon[29736]: pgmap v894: 158 pgs: 3 active+clean+scrubbing, 155 active+clean; 8.7 GiB data, 31 GiB used, 595 GiB / 626 GiB avail; 1.0 KiB/s rd, 67 MiB/s wr, 1.35k op/s 2024-09-10T19:29:31.317 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:29:30 smithi183 ceph-mon[29736]: 11.1 scrub ok 2024-09-10T19:29:31.317 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:29:30 smithi183 ceph-mon[29736]: from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:29:31.317 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:29:30 smithi183 ceph-mon[29736]: 6.13 scrub starts 2024-09-10T19:29:31.317 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:29:30 smithi183 ceph-mon[29736]: 6.13 scrub ok 2024-09-10T19:29:31.317 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:29:30 smithi183 ceph-mon[29736]: 5.7 scrub starts 2024-09-10T19:29:31.317 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:29:30 smithi183 ceph-mon[29736]: 5.7 scrub ok 2024-09-10T19:29:31.317 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:29:30 smithi183 ceph-mon[29736]: 5.c scrub starts 2024-09-10T19:29:31.317 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:29:30 smithi183 ceph-mon[29736]: 5.c scrub ok 2024-09-10T19:29:31.317 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:29:30 smithi183 ceph-mon[29736]: osdmap e402: 8 total, 8 up, 7 in 2024-09-10T19:29:31.318 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:29:30 smithi183 ceph-mon[41562]: purged_snaps scrub starts 2024-09-10T19:29:31.318 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:29:30 smithi183 ceph-mon[41562]: purged_snaps scrub ok 2024-09-10T19:29:31.318 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:29:30 smithi183 ceph-mon[41562]: pgmap v894: 158 pgs: 3 active+clean+scrubbing, 155 active+clean; 8.7 GiB data, 31 GiB used, 595 GiB / 626 GiB avail; 1.0 KiB/s rd, 67 MiB/s wr, 1.35k op/s 2024-09-10T19:29:31.318 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:29:30 smithi183 ceph-mon[41562]: 11.1 scrub ok 2024-09-10T19:29:31.318 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:29:30 smithi183 ceph-mon[41562]: from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:29:31.318 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:29:30 smithi183 ceph-mon[41562]: 6.13 scrub starts 2024-09-10T19:29:31.318 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:29:30 smithi183 ceph-mon[41562]: 6.13 scrub ok 2024-09-10T19:29:31.318 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:29:30 smithi183 ceph-mon[41562]: 5.7 scrub starts 2024-09-10T19:29:31.318 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:29:30 smithi183 ceph-mon[41562]: 5.7 scrub ok 2024-09-10T19:29:31.318 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:29:30 smithi183 ceph-mon[41562]: 5.c scrub starts 2024-09-10T19:29:31.318 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:29:30 smithi183 ceph-mon[41562]: 5.c scrub ok 2024-09-10T19:29:31.318 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:29:30 smithi183 ceph-mon[41562]: osdmap e402: 8 total, 8 up, 7 in 2024-09-10T19:29:31.320 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:29:31.356 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:29:31.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:29:31.357 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:29:31.457 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:29:31.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:29:30 smithi191 ceph-mon[32884]: purged_snaps scrub starts 2024-09-10T19:29:31.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:29:30 smithi191 ceph-mon[32884]: purged_snaps scrub ok 2024-09-10T19:29:31.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:29:30 smithi191 ceph-mon[32884]: pgmap v894: 158 pgs: 3 active+clean+scrubbing, 155 active+clean; 8.7 GiB data, 31 GiB used, 595 GiB / 626 GiB avail; 1.0 KiB/s rd, 67 MiB/s wr, 1.35k op/s 2024-09-10T19:29:31.467 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:29:30 smithi191 ceph-mon[32884]: 11.1 scrub ok 2024-09-10T19:29:31.467 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:29:30 smithi191 ceph-mon[32884]: from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:29:31.467 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:29:30 smithi191 ceph-mon[32884]: 6.13 scrub starts 2024-09-10T19:29:31.467 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:29:30 smithi191 ceph-mon[32884]: 6.13 scrub ok 2024-09-10T19:29:31.467 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:29:30 smithi191 ceph-mon[32884]: 5.7 scrub starts 2024-09-10T19:29:31.467 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:29:30 smithi191 ceph-mon[32884]: 5.7 scrub ok 2024-09-10T19:29:31.467 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:29:30 smithi191 ceph-mon[32884]: 5.c scrub starts 2024-09-10T19:29:31.467 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:29:30 smithi191 ceph-mon[32884]: 5.c scrub ok 2024-09-10T19:29:31.467 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:29:30 smithi191 ceph-mon[32884]: osdmap e402: 8 total, 8 up, 7 in 2024-09-10T19:29:31.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:29:31.496 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:29:31.496 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:29:31.585 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph daemon osd.2 dump_historic_ops 2024-09-10T19:29:31.586 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Sep 10 19:29:31 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-y[29989]: ::ffff:172.21.15.191 - - [10/Sep/2024:19:29:31] "GET /metrics HTTP/1.1" 200 224878 "" "Prometheus/2.43.0" 2024-09-10T19:29:31.597 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:29:31.633 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:29:31.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:29:31.634 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:29:31.735 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:29:31.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:29:31.772 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:29:31.772 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:29:31.873 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:29:31.916 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:29:31.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:29:31.917 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:29:32.017 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:29:32.051 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:29:32.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:29:32.052 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:29:32.153 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:29:32.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:29:31 smithi183 ceph-mon[29736]: 2.8 scrub starts 2024-09-10T19:29:32.182 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:29:31 smithi183 ceph-mon[29736]: 2.8 scrub ok 2024-09-10T19:29:32.182 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:29:31 smithi183 ceph-mon[29736]: 11.7 scrub starts 2024-09-10T19:29:32.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:29:31 smithi183 ceph-mon[41562]: 2.8 scrub starts 2024-09-10T19:29:32.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:29:31 smithi183 ceph-mon[41562]: 2.8 scrub ok 2024-09-10T19:29:32.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:29:31 smithi183 ceph-mon[41562]: 11.7 scrub starts 2024-09-10T19:29:32.184 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:29:32.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:29:32.185 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:29:32.286 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:29:32.321 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:29:32.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:29:32.322 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:29:32.423 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:29:32.456 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:29:32.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:29:32.457 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:29:32.465 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:29:31 smithi191 ceph-mon[32884]: 2.8 scrub starts 2024-09-10T19:29:32.465 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:29:31 smithi191 ceph-mon[32884]: 2.8 scrub ok 2024-09-10T19:29:32.465 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:29:31 smithi191 ceph-mon[32884]: 11.7 scrub starts 2024-09-10T19:29:32.558 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:29:32.592 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:29:32.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:29:32.593 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:29:32.694 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:29:32.729 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:29:32.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:29:32.730 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:29:32.831 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:29:32.863 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:29:32.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:29:32.864 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:29:32.965 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:29:32.997 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:29:32.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:29:32.998 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:29:33.098 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:29:33.134 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:29:33.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:29:33.135 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:29:33.236 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:29:33.277 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:29:33.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:29:33.278 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:29:33.379 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:29:33.425 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:29:33.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:29:33.426 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:29:33.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:29:32 smithi183 ceph-mon[29736]: 2.1c scrub starts 2024-09-10T19:29:33.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:29:32 smithi183 ceph-mon[29736]: 2.1c scrub ok 2024-09-10T19:29:33.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:29:32 smithi183 ceph-mon[29736]: 11.e scrub starts 2024-09-10T19:29:33.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:29:32 smithi183 ceph-mon[29736]: purged_snaps scrub starts 2024-09-10T19:29:33.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:29:32 smithi183 ceph-mon[29736]: purged_snaps scrub ok 2024-09-10T19:29:33.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:29:32 smithi183 ceph-mon[29736]: pgmap v896: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 8.4 GiB data, 31 GiB used, 595 GiB / 626 GiB avail; 1.1 KiB/s rd, 36 MiB/s wr, 1.35k op/s 2024-09-10T19:29:33.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:29:32 smithi183 ceph-mon[29736]: osdmap e403: 8 total, 8 up, 7 in 2024-09-10T19:29:33.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:29:32 smithi183 ceph-mon[29736]: 4.15 scrub starts 2024-09-10T19:29:33.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:29:32 smithi183 ceph-mon[29736]: 4.15 scrub ok 2024-09-10T19:29:33.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:29:32 smithi183 ceph-mon[41562]: 2.1c scrub starts 2024-09-10T19:29:33.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:29:32 smithi183 ceph-mon[41562]: 2.1c scrub ok 2024-09-10T19:29:33.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:29:32 smithi183 ceph-mon[41562]: 11.e scrub starts 2024-09-10T19:29:33.433 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:29:32 smithi183 ceph-mon[41562]: purged_snaps scrub starts 2024-09-10T19:29:33.433 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:29:32 smithi183 ceph-mon[41562]: purged_snaps scrub ok 2024-09-10T19:29:33.433 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:29:32 smithi183 ceph-mon[41562]: pgmap v896: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 8.4 GiB data, 31 GiB used, 595 GiB / 626 GiB avail; 1.1 KiB/s rd, 36 MiB/s wr, 1.35k op/s 2024-09-10T19:29:33.433 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:29:32 smithi183 ceph-mon[41562]: osdmap e403: 8 total, 8 up, 7 in 2024-09-10T19:29:33.433 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:29:32 smithi183 ceph-mon[41562]: 4.15 scrub starts 2024-09-10T19:29:33.433 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:29:32 smithi183 ceph-mon[41562]: 4.15 scrub ok 2024-09-10T19:29:33.465 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:29:32 smithi191 ceph-mon[32884]: 2.1c scrub starts 2024-09-10T19:29:33.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:29:32 smithi191 ceph-mon[32884]: 2.1c scrub ok 2024-09-10T19:29:33.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:29:32 smithi191 ceph-mon[32884]: 11.e scrub starts 2024-09-10T19:29:33.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:29:32 smithi191 ceph-mon[32884]: purged_snaps scrub starts 2024-09-10T19:29:33.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:29:32 smithi191 ceph-mon[32884]: purged_snaps scrub ok 2024-09-10T19:29:33.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:29:32 smithi191 ceph-mon[32884]: pgmap v896: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 8.4 GiB data, 31 GiB used, 595 GiB / 626 GiB avail; 1.1 KiB/s rd, 36 MiB/s wr, 1.35k op/s 2024-09-10T19:29:33.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:29:32 smithi191 ceph-mon[32884]: osdmap e403: 8 total, 8 up, 7 in 2024-09-10T19:29:33.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:29:32 smithi191 ceph-mon[32884]: 4.15 scrub starts 2024-09-10T19:29:33.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:29:32 smithi191 ceph-mon[32884]: 4.15 scrub ok 2024-09-10T19:29:33.527 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:29:33.566 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:29:33.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:29:33.567 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:29:33.643 INFO:teuthology.orchestra.run.smithi183.stdout:osd.0: {} 2024-09-10T19:29:33.643 INFO:teuthology.orchestra.run.smithi183.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-10T19:29:33.667 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:29:33.674 INFO:teuthology.orchestra.run.smithi183.stdout:osd.1: {} 2024-09-10T19:29:33.675 INFO:teuthology.orchestra.run.smithi183.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-10T19:29:33.698 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:29:33.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:29:33.699 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:29:33.703 INFO:teuthology.orchestra.run.smithi183.stdout:osd.2: {} 2024-09-10T19:29:33.703 INFO:teuthology.orchestra.run.smithi183.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-10T19:29:33.729 INFO:teuthology.orchestra.run.smithi183.stdout:osd.3: {} 2024-09-10T19:29:33.730 INFO:teuthology.orchestra.run.smithi183.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-10T19:29:33.762 INFO:teuthology.orchestra.run.smithi183.stdout:osd.4: {} 2024-09-10T19:29:33.762 INFO:teuthology.orchestra.run.smithi183.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-10T19:29:33.792 INFO:teuthology.orchestra.run.smithi183.stdout:osd.5: {} 2024-09-10T19:29:33.792 INFO:teuthology.orchestra.run.smithi183.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-10T19:29:33.800 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:29:33.823 INFO:teuthology.orchestra.run.smithi183.stdout:osd.6: {} 2024-09-10T19:29:33.824 INFO:teuthology.orchestra.run.smithi183.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-10T19:29:33.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:29:33.846 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:29:33.846 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:29:33.856 INFO:teuthology.orchestra.run.smithi183.stdout:osd.7: {} 2024-09-10T19:29:33.856 INFO:teuthology.orchestra.run.smithi183.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-10T19:29:33.946 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:29:33.980 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:29:33.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:29:33.981 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:29:34.000 INFO:teuthology.orchestra.run.smithi183.stderr:reweighted osd.4 to 1 (10000) 2024-09-10T19:29:34.081 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:29:34.405 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:29:34.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:29:34.406 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:29:34.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:29:33 smithi183 ceph-mon[29736]: 6.17 scrub starts 2024-09-10T19:29:34.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:29:33 smithi183 ceph-mon[29736]: 6.17 scrub ok 2024-09-10T19:29:34.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:29:33 smithi183 ceph-mon[29736]: 2.1f scrub starts 2024-09-10T19:29:34.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:29:33 smithi183 ceph-mon[29736]: 2.1f scrub ok 2024-09-10T19:29:34.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:29:33 smithi183 ceph-mon[29736]: 6.12 scrub starts 2024-09-10T19:29:34.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:29:33 smithi183 ceph-mon[29736]: 6.11 scrub starts 2024-09-10T19:29:34.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:29:33 smithi183 ceph-mon[29736]: 6.11 scrub ok 2024-09-10T19:29:34.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:29:33 smithi183 ceph-mon[29736]: 6.12 scrub ok 2024-09-10T19:29:34.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:29:33 smithi183 ceph-mon[29736]: from='client.? 172.21.15.183:0/1601231705' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-09-10T19:29:34.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:29:33 smithi183 ceph-mon[29736]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-09-10T19:29:34.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:29:33 smithi183 ceph-mon[29736]: from='client.? 172.21.15.183:0/1388088372' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T19:29:34.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:29:34 smithi183 ceph-mon[41562]: 6.17 scrub starts 2024-09-10T19:29:34.433 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:29:34 smithi183 ceph-mon[41562]: 6.17 scrub ok 2024-09-10T19:29:34.433 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:29:34 smithi183 ceph-mon[41562]: 2.1f scrub starts 2024-09-10T19:29:34.433 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:29:34 smithi183 ceph-mon[41562]: 2.1f scrub ok 2024-09-10T19:29:34.433 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:29:34 smithi183 ceph-mon[41562]: 6.12 scrub starts 2024-09-10T19:29:34.433 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:29:34 smithi183 ceph-mon[41562]: 6.11 scrub starts 2024-09-10T19:29:34.433 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:29:34 smithi183 ceph-mon[41562]: 6.11 scrub ok 2024-09-10T19:29:34.433 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:29:34 smithi183 ceph-mon[41562]: 6.12 scrub ok 2024-09-10T19:29:34.433 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:29:34 smithi183 ceph-mon[41562]: from='client.? 172.21.15.183:0/1601231705' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-09-10T19:29:34.433 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:29:34 smithi183 ceph-mon[41562]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-09-10T19:29:34.433 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:29:34 smithi183 ceph-mon[41562]: from='client.? 172.21.15.183:0/1388088372' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T19:29:34.465 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:29:33 smithi191 ceph-mon[32884]: 6.17 scrub starts 2024-09-10T19:29:34.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:29:33 smithi191 ceph-mon[32884]: 6.17 scrub ok 2024-09-10T19:29:34.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:29:34 smithi191 ceph-mon[32884]: 2.1f scrub starts 2024-09-10T19:29:34.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:29:34 smithi191 ceph-mon[32884]: 2.1f scrub ok 2024-09-10T19:29:34.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:29:34 smithi191 ceph-mon[32884]: 6.12 scrub starts 2024-09-10T19:29:34.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:29:34 smithi191 ceph-mon[32884]: 6.11 scrub starts 2024-09-10T19:29:34.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:29:34 smithi191 ceph-mon[32884]: 6.11 scrub ok 2024-09-10T19:29:34.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:29:34 smithi191 ceph-mon[32884]: 6.12 scrub ok 2024-09-10T19:29:34.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:29:34 smithi191 ceph-mon[32884]: from='client.? 172.21.15.183:0/1601231705' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-09-10T19:29:34.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:29:34 smithi191 ceph-mon[32884]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-09-10T19:29:34.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:29:34 smithi191 ceph-mon[32884]: from='client.? 172.21.15.183:0/1388088372' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T19:29:34.507 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:29:34.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:29:34.543 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:29:34.544 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:29:34.645 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:29:34.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:29:34.678 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:29:34.678 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:29:34.779 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:29:34.809 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:29:34.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:29:34.810 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:29:34.911 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:29:34.942 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:29:34.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:29:34.943 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:29:35.044 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:29:35.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:29:35.077 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:29:35.077 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:29:35.178 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:29:35.209 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:29:35.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:29:35.210 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:29:35.311 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:29:35.343 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:29:35.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:29:35.344 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:29:35.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:29:35 smithi183 ceph-mon[29736]: 2.1a scrub starts 2024-09-10T19:29:35.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:29:35 smithi183 ceph-mon[29736]: 2.1a scrub ok 2024-09-10T19:29:35.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:29:35 smithi183 ceph-mon[29736]: pgmap v898: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 8.3 GiB data, 30 GiB used, 596 GiB / 626 GiB avail; 1.2 KiB/s rd, 3.5 MiB/s wr, 1.03k op/s 2024-09-10T19:29:35.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:29:35 smithi183 ceph-mon[29736]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-09-10T19:29:35.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:29:35 smithi183 ceph-mon[29736]: osdmap e404: 8 total, 8 up, 7 in 2024-09-10T19:29:35.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:29:35 smithi183 ceph-mon[29736]: 1.0 deep-scrub starts 2024-09-10T19:29:35.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:29:35 smithi183 ceph-mon[29736]: 1.0 deep-scrub ok 2024-09-10T19:29:35.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:29:35 smithi183 ceph-mon[29736]: 2.19 scrub starts 2024-09-10T19:29:35.433 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:29:35 smithi183 ceph-mon[29736]: 2.19 scrub ok 2024-09-10T19:29:35.433 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:29:35 smithi183 ceph-mon[29736]: 11.7 scrub ok 2024-09-10T19:29:35.433 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:29:35 smithi183 ceph-mon[29736]: 2.c scrub starts 2024-09-10T19:29:35.433 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:29:35 smithi183 ceph-mon[29736]: 2.c scrub ok 2024-09-10T19:29:35.433 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:29:35 smithi183 ceph-mon[29736]: 4.1c scrub starts 2024-09-10T19:29:35.433 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:29:35 smithi183 ceph-mon[29736]: 4.1c scrub ok 2024-09-10T19:29:35.433 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:29:35 smithi183 ceph-mon[41562]: 2.1a scrub starts 2024-09-10T19:29:35.433 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:29:35 smithi183 ceph-mon[41562]: 2.1a scrub ok 2024-09-10T19:29:35.433 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:29:35 smithi183 ceph-mon[41562]: pgmap v898: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 8.3 GiB data, 30 GiB used, 596 GiB / 626 GiB avail; 1.2 KiB/s rd, 3.5 MiB/s wr, 1.03k op/s 2024-09-10T19:29:35.433 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:29:35 smithi183 ceph-mon[41562]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-09-10T19:29:35.433 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:29:35 smithi183 ceph-mon[41562]: osdmap e404: 8 total, 8 up, 7 in 2024-09-10T19:29:35.433 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:29:35 smithi183 ceph-mon[41562]: 1.0 deep-scrub starts 2024-09-10T19:29:35.434 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:29:35 smithi183 ceph-mon[41562]: 1.0 deep-scrub ok 2024-09-10T19:29:35.434 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:29:35 smithi183 ceph-mon[41562]: 2.19 scrub starts 2024-09-10T19:29:35.434 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:29:35 smithi183 ceph-mon[41562]: 2.19 scrub ok 2024-09-10T19:29:35.434 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:29:35 smithi183 ceph-mon[41562]: 11.7 scrub ok 2024-09-10T19:29:35.434 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:29:35 smithi183 ceph-mon[41562]: 2.c scrub starts 2024-09-10T19:29:35.434 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:29:35 smithi183 ceph-mon[41562]: 2.c scrub ok 2024-09-10T19:29:35.434 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:29:35 smithi183 ceph-mon[41562]: 4.1c scrub starts 2024-09-10T19:29:35.434 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:29:35 smithi183 ceph-mon[41562]: 4.1c scrub ok 2024-09-10T19:29:35.445 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:29:35.465 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:29:35 smithi191 ceph-mon[32884]: 2.1a scrub starts 2024-09-10T19:29:35.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:29:35 smithi191 ceph-mon[32884]: 2.1a scrub ok 2024-09-10T19:29:35.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:29:35 smithi191 ceph-mon[32884]: pgmap v898: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 8.3 GiB data, 30 GiB used, 596 GiB / 626 GiB avail; 1.2 KiB/s rd, 3.5 MiB/s wr, 1.03k op/s 2024-09-10T19:29:35.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:29:35 smithi191 ceph-mon[32884]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-09-10T19:29:35.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:29:35 smithi191 ceph-mon[32884]: osdmap e404: 8 total, 8 up, 7 in 2024-09-10T19:29:35.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:29:35 smithi191 ceph-mon[32884]: 1.0 deep-scrub starts 2024-09-10T19:29:35.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:29:35 smithi191 ceph-mon[32884]: 1.0 deep-scrub ok 2024-09-10T19:29:35.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:29:35 smithi191 ceph-mon[32884]: 2.19 scrub starts 2024-09-10T19:29:35.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:29:35 smithi191 ceph-mon[32884]: 2.19 scrub ok 2024-09-10T19:29:35.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:29:35 smithi191 ceph-mon[32884]: 11.7 scrub ok 2024-09-10T19:29:35.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:29:35 smithi191 ceph-mon[32884]: 2.c scrub starts 2024-09-10T19:29:35.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:29:35 smithi191 ceph-mon[32884]: 2.c scrub ok 2024-09-10T19:29:35.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:29:35 smithi191 ceph-mon[32884]: 4.1c scrub starts 2024-09-10T19:29:35.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:29:35 smithi191 ceph-mon[32884]: 4.1c scrub ok 2024-09-10T19:29:35.475 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:29:35.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:29:35.476 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:29:35.576 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:29:35.608 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:29:35.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:29:35.609 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:29:35.614 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph osd reweight 5 1 2024-09-10T19:29:35.710 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:29:35.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:29:35.743 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:29:35.743 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:29:35.844 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:29:35.885 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:29:35.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:29:35.886 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:29:35.987 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:29:36.021 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:29:36.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:29:36.022 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:29:36.123 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:29:36.156 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:29:36.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:29:36.157 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:29:36.258 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:29:36.266 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:29:36 smithi183 ceph-mon[29736]: 6.16 deep-scrub starts 2024-09-10T19:29:36.266 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:29:36 smithi183 ceph-mon[29736]: 6.16 deep-scrub ok 2024-09-10T19:29:36.266 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:29:36 smithi183 ceph-mon[29736]: 11.e scrub ok 2024-09-10T19:29:36.266 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:29:36 smithi183 ceph-mon[29736]: 4.0 scrub starts 2024-09-10T19:29:36.266 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:29:36 smithi183 ceph-mon[29736]: 4.0 scrub ok 2024-09-10T19:29:36.267 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:29:36 smithi183 ceph-mon[41562]: 6.16 deep-scrub starts 2024-09-10T19:29:36.267 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:29:36 smithi183 ceph-mon[41562]: 6.16 deep-scrub ok 2024-09-10T19:29:36.267 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:29:36 smithi183 ceph-mon[41562]: 11.e scrub ok 2024-09-10T19:29:36.267 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:29:36 smithi183 ceph-mon[41562]: 4.0 scrub starts 2024-09-10T19:29:36.267 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:29:36 smithi183 ceph-mon[41562]: 4.0 scrub ok 2024-09-10T19:29:36.289 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:29:36.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:29:36.290 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:29:36.326 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:29:36.391 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:29:36.439 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:29:36.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:29:36.440 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:29:36.465 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:29:36 smithi191 ceph-mon[32884]: 6.16 deep-scrub starts 2024-09-10T19:29:36.465 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:29:36 smithi191 ceph-mon[32884]: 6.16 deep-scrub ok 2024-09-10T19:29:36.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:29:36 smithi191 ceph-mon[32884]: 11.e scrub ok 2024-09-10T19:29:36.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:29:36 smithi191 ceph-mon[32884]: 4.0 scrub starts 2024-09-10T19:29:36.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:29:36 smithi191 ceph-mon[32884]: 4.0 scrub ok 2024-09-10T19:29:36.541 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:29:36.572 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:29:36.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:29:36.573 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:29:36.674 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:29:36.708 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:29:36.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:29:36.709 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:29:36.810 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:29:36.846 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:29:36.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:29:36.847 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:29:36.947 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:29:36.980 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:29:36.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:29:36.981 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:29:37.032 INFO:teuthology.orchestra.run.smithi183.stderr:noscrub is set 2024-09-10T19:29:37.082 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:29:37.117 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:29:37.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:29:37.118 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:29:37.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:29:37 smithi183 ceph-mon[29736]: 6.1b scrub starts 2024-09-10T19:29:37.182 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:29:37 smithi183 ceph-mon[29736]: 6.1b scrub ok 2024-09-10T19:29:37.182 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:29:37 smithi183 ceph-mon[29736]: pgmap v900: 158 pgs: 158 active+clean; 7.9 GiB data, 29 GiB used, 597 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 2.01k op/s 2024-09-10T19:29:37.182 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:29:37 smithi183 ceph-mon[29736]: 2.12 scrub starts 2024-09-10T19:29:37.182 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:29:37 smithi183 ceph-mon[29736]: 2.12 scrub ok 2024-09-10T19:29:37.182 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:29:37 smithi183 ceph-mon[29736]: from='client.? 172.21.15.183:0/1919703268' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-10T19:29:37.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:29:37 smithi183 ceph-mon[41562]: 6.1b scrub starts 2024-09-10T19:29:37.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:29:37 smithi183 ceph-mon[41562]: 6.1b scrub ok 2024-09-10T19:29:37.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:29:37 smithi183 ceph-mon[41562]: pgmap v900: 158 pgs: 158 active+clean; 7.9 GiB data, 29 GiB used, 597 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 2.01k op/s 2024-09-10T19:29:37.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:29:37 smithi183 ceph-mon[41562]: 2.12 scrub starts 2024-09-10T19:29:37.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:29:37 smithi183 ceph-mon[41562]: 2.12 scrub ok 2024-09-10T19:29:37.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:29:37 smithi183 ceph-mon[41562]: from='client.? 172.21.15.183:0/1919703268' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-10T19:29:37.219 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:29:37.251 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:29:37.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:29:37.252 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:29:37.353 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:29:37.386 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:29:37.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:29:37.387 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:29:37.450 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:29:37 smithi191 ceph-mon[32884]: 6.1b scrub starts 2024-09-10T19:29:37.450 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:29:37 smithi191 ceph-mon[32884]: 6.1b scrub ok 2024-09-10T19:29:37.450 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:29:37 smithi191 ceph-mon[32884]: pgmap v900: 158 pgs: 158 active+clean; 7.9 GiB data, 29 GiB used, 597 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 2.01k op/s 2024-09-10T19:29:37.450 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:29:37 smithi191 ceph-mon[32884]: 2.12 scrub starts 2024-09-10T19:29:37.450 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:29:37 smithi191 ceph-mon[32884]: 2.12 scrub ok 2024-09-10T19:29:37.450 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:29:37 smithi191 ceph-mon[32884]: from='client.? 172.21.15.183:0/1919703268' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-10T19:29:37.488 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:29:37.519 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:29:37.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:29:37.520 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:29:37.621 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:29:37.660 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:29:37.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:29:37.661 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:29:37.716 INFO:journalctl@ceph.mgr.x.smithi191.stdout:Sep 10 19:29:37 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-x[34967]: ::ffff:172.21.15.191 - - [10/Sep/2024:19:29:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T19:29:37.761 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:29:37.795 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:29:37.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:29:37.796 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:29:37.897 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:29:37.937 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:29:37.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:29:37.937 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:29:38.038 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:29:38.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:29:38.075 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:29:38.075 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:29:38.102 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-10T19:29:38.176 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:29:38.212 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:29:38.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:29:38.213 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:29:38.313 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:29:38.332 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:29:38 smithi183 ceph-mon[29736]: 11.d scrub starts 2024-09-10T19:29:38.333 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:29:38 smithi183 ceph-mon[29736]: 4.6 scrub starts 2024-09-10T19:29:38.333 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:29:38 smithi183 ceph-mon[29736]: 4.6 scrub ok 2024-09-10T19:29:38.333 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:29:38 smithi183 ceph-mon[29736]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T19:29:38.333 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:29:38 smithi183 ceph-mon[29736]: from='client.? 172.21.15.183:0/1919703268' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-10T19:29:38.333 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:29:38 smithi183 ceph-mon[29736]: osdmap e405: 8 total, 8 up, 7 in 2024-09-10T19:29:38.333 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:29:38 smithi183 ceph-mon[41562]: 11.d scrub starts 2024-09-10T19:29:38.333 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:29:38 smithi183 ceph-mon[41562]: 4.6 scrub starts 2024-09-10T19:29:38.333 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:29:38 smithi183 ceph-mon[41562]: 4.6 scrub ok 2024-09-10T19:29:38.333 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:29:38 smithi183 ceph-mon[41562]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T19:29:38.333 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:29:38 smithi183 ceph-mon[41562]: from='client.? 172.21.15.183:0/1919703268' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-10T19:29:38.333 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:29:38 smithi183 ceph-mon[41562]: osdmap e405: 8 total, 8 up, 7 in 2024-09-10T19:29:38.344 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:29:38.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:29:38.345 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:29:38.446 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:29:38.465 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:29:38 smithi191 ceph-mon[32884]: 11.d scrub starts 2024-09-10T19:29:38.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:29:38 smithi191 ceph-mon[32884]: 4.6 scrub starts 2024-09-10T19:29:38.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:29:38 smithi191 ceph-mon[32884]: 4.6 scrub ok 2024-09-10T19:29:38.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:29:38 smithi191 ceph-mon[32884]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T19:29:38.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:29:38 smithi191 ceph-mon[32884]: from='client.? 172.21.15.183:0/1919703268' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-10T19:29:38.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:29:38 smithi191 ceph-mon[32884]: osdmap e405: 8 total, 8 up, 7 in 2024-09-10T19:29:38.480 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:29:38.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:29:38.481 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:29:38.582 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:29:38.613 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:29:38.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:29:38.614 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:29:38.715 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:29:38.751 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:29:38.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:29:38.752 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:29:38.852 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:29:38.884 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:29:38.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:29:38.885 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:29:38.986 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:29:39.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:29:39.020 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:29:39.020 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:29:39.121 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:29:39.159 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:29:39.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:29:39.161 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:29:39.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:29:39 smithi183 ceph-mon[29736]: pgmap v902: 158 pgs: 158 active+clean; 7.8 GiB data, 29 GiB used, 597 GiB / 626 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.44k op/s 2024-09-10T19:29:39.181 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:29:39 smithi183 ceph-mon[41562]: pgmap v902: 158 pgs: 158 active+clean; 7.8 GiB data, 29 GiB used, 597 GiB / 626 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.44k op/s 2024-09-10T19:29:39.261 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:29:39.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:29:39.309 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:29:39.310 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:29:39.343 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:29:39.368 INFO:teuthology.orchestra.run.smithi183.stdout:ERROR: (22) Invalid argument 2024-09-10T19:29:39.368 INFO:teuthology.orchestra.run.smithi183.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-10T19:29:39.411 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:29:39.444 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:29:39.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:29:39.446 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:29:39.465 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:29:39 smithi191 ceph-mon[32884]: pgmap v902: 158 pgs: 158 active+clean; 7.8 GiB data, 29 GiB used, 597 GiB / 626 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.44k op/s 2024-09-10T19:29:39.547 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:29:39.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:29:39.583 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:29:39.583 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:29:39.684 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:29:39.720 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:29:39.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:29:39.722 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:29:39.822 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:29:39.857 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:29:39.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:29:39.858 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:29:39.959 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:29:39.996 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:29:39.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:29:39.997 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:29:40.098 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:29:40.106 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph daemon osd.3 dump_ops_in_flight 2024-09-10T19:29:40.127 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:29:40.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:29:40.128 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:29:40.135 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:29:40.229 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:29:40.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:29:40.261 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:29:40.261 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:29:40.362 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:29:40.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:29:40.414 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:29:40.414 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:29:40.465 INFO:journalctl@ceph.iscsi.iscsi.a.smithi191.stdout:Sep 10 19:29:40 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-iscsi-iscsi-a[61816]: debug there is no tcmu-runner data available 2024-09-10T19:29:40.515 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:29:40.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:29:40.559 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:29:40.559 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:29:40.660 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:29:40.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:29:40.695 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:29:40.696 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:29:40.796 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:29:40.803 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-10T19:29:40.828 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:29:40.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:29:40.829 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:29:40.930 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:29:40.971 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:29:40.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:29:40.972 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:29:41.072 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:29:41.104 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:29:41.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:29:41.105 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:29:41.160 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:29:41 smithi183 ceph-mon[29736]: pgmap v903: 158 pgs: 158 active+clean; 7.8 GiB data, 29 GiB used, 597 GiB / 626 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.19k op/s 2024-09-10T19:29:41.160 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:29:41 smithi183 ceph-mon[29736]: from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:29:41.160 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:29:41 smithi183 ceph-mon[41562]: pgmap v903: 158 pgs: 158 active+clean; 7.8 GiB data, 29 GiB used, 597 GiB / 626 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.19k op/s 2024-09-10T19:29:41.160 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:29:41 smithi183 ceph-mon[41562]: from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:29:41.206 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:29:41.240 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:29:41.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:29:41.240 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:29:41.341 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:29:41.376 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:29:41.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:29:41.377 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:29:41.423 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Sep 10 19:29:41 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-y[29989]: ::ffff:172.21.15.191 - - [10/Sep/2024:19:29:41] "GET /metrics HTTP/1.1" 200 224878 "" "Prometheus/2.43.0" 2024-09-10T19:29:41.465 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:29:41 smithi191 ceph-mon[32884]: pgmap v903: 158 pgs: 158 active+clean; 7.8 GiB data, 29 GiB used, 597 GiB / 626 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.19k op/s 2024-09-10T19:29:41.465 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:29:41 smithi191 ceph-mon[32884]: from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:29:41.478 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:29:41.512 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:29:41.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:29:41.513 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:29:41.614 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:29:41.646 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:29:41.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:29:41.647 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:29:41.747 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:29:41.787 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:29:41.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:29:41.788 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:29:41.889 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:29:41.923 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:29:41.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:29:41.925 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:29:42.025 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:29:42.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:29:42.059 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:29:42.059 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:29:42.160 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:29:42.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:29:42.192 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:29:42.192 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:29:42.293 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:29:42.326 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:29:42.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:29:42.327 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:29:42.428 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:29:42.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:29:42 smithi183 ceph-mon[29736]: purged_snaps scrub starts 2024-09-10T19:29:42.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:29:42 smithi183 ceph-mon[29736]: purged_snaps scrub ok 2024-09-10T19:29:42.431 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:29:42 smithi183 ceph-mon[41562]: purged_snaps scrub starts 2024-09-10T19:29:42.431 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:29:42 smithi183 ceph-mon[41562]: purged_snaps scrub ok 2024-09-10T19:29:42.448 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:29:42.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:29:42.449 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:29:42.465 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:29:42 smithi191 ceph-mon[32884]: purged_snaps scrub starts 2024-09-10T19:29:42.465 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:29:42 smithi191 ceph-mon[32884]: purged_snaps scrub ok 2024-09-10T19:29:42.550 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:29:42.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:29:42.582 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:29:42.582 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:29:42.682 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:29:42.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:29:42.715 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:29:42.715 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:29:42.816 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:29:42.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:29:42.851 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:29:42.851 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:29:42.952 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:29:42.980 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:29:42.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:29:42.981 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:29:43.082 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:29:43.114 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:29:43.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:29:43.116 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:29:43.216 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:29:43.251 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:29:43.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:29:43.252 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:29:43.353 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:29:43.384 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:29:43.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:29:43.385 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:29:43.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:29:43 smithi183 ceph-mon[29736]: pgmap v904: 158 pgs: 158 active+clean; 7.5 GiB data, 28 GiB used, 598 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 1.67k op/s 2024-09-10T19:29:43.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:29:43 smithi183 ceph-mon[29736]: osdmap e406: 8 total, 8 up, 7 in 2024-09-10T19:29:43.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:29:43 smithi183 ceph-mon[41562]: pgmap v904: 158 pgs: 158 active+clean; 7.5 GiB data, 28 GiB used, 598 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 1.67k op/s 2024-09-10T19:29:43.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:29:43 smithi183 ceph-mon[41562]: osdmap e406: 8 total, 8 up, 7 in 2024-09-10T19:29:43.465 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:29:43 smithi191 ceph-mon[32884]: pgmap v904: 158 pgs: 158 active+clean; 7.5 GiB data, 28 GiB used, 598 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 1.67k op/s 2024-09-10T19:29:43.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:29:43 smithi191 ceph-mon[32884]: osdmap e406: 8 total, 8 up, 7 in 2024-09-10T19:29:43.486 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:29:43.519 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:29:43.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:29:43.520 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:29:43.620 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:29:43.656 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:29:43.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:29:43.657 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:29:43.758 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:29:43.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:29:43.797 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:29:43.797 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:29:43.898 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:29:43.918 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:29:43.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:29:43.919 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:29:44.021 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:29:44.051 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:29:44.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:29:44.052 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:29:44.153 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:29:44.175 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:29:44.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:29:44.176 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:29:44.277 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:29:44.309 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:29:44.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:29:44.310 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:29:44.411 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:29:44.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:29:44 smithi183 ceph-mon[29736]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T19:29:44.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:29:44 smithi183 ceph-mon[29736]: from='mgr.14556 ' entity='mgr.y' 2024-09-10T19:29:44.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:29:44 smithi183 ceph-mon[29736]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:29:44.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:29:44 smithi183 ceph-mon[41562]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T19:29:44.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:29:44 smithi183 ceph-mon[41562]: from='mgr.14556 ' entity='mgr.y' 2024-09-10T19:29:44.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:29:44 smithi183 ceph-mon[41562]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:29:44.433 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:29:44.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:29:44.434 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:29:44.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:29:44 smithi191 ceph-mon[32884]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T19:29:44.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:29:44 smithi191 ceph-mon[32884]: from='mgr.14556 ' entity='mgr.y' 2024-09-10T19:29:44.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:29:44 smithi191 ceph-mon[32884]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:29:44.536 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:29:44.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:29:44.567 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:29:44.567 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:29:44.668 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:29:44.694 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:29:44.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:29:44.695 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:29:44.796 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:29:44.824 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:29:44.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:29:44.825 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:29:44.926 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:29:44.955 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:29:44.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:29:44.956 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:29:45.057 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:29:45.093 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:29:45.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:29:45.094 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:29:45.194 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:29:45.224 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:29:45.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:29:45.225 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:29:45.326 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:29:45.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:29:45.359 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:29:45.359 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:29:45.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:29:45 smithi183 ceph-mon[29736]: pgmap v906: 158 pgs: 158 active+clean; 7.4 GiB data, 27 GiB used, 598 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.11k op/s 2024-09-10T19:29:45.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:29:45 smithi183 ceph-mon[41562]: pgmap v906: 158 pgs: 158 active+clean; 7.4 GiB data, 27 GiB used, 598 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.11k op/s 2024-09-10T19:29:45.460 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:29:45.465 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:29:45 smithi191 ceph-mon[32884]: pgmap v906: 158 pgs: 158 active+clean; 7.4 GiB data, 27 GiB used, 598 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.11k op/s 2024-09-10T19:29:45.491 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:29:45.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:29:45.492 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:29:45.592 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:29:45.619 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:29:45.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:29:45.620 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:29:45.721 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:29:45.751 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:29:45.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:29:45.752 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:29:45.853 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:29:45.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:29:45.890 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:29:45.891 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:29:45.991 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:29:46.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:29:46.025 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:29:46.025 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:29:46.126 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:29:46.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:29:46.157 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:29:46.157 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:29:46.258 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:29:46.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:29:46.280 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:29:46.280 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:29:46.381 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:29:46.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:29:46.414 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:29:46.414 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:29:46.515 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:29:46.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:29:46.546 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:29:46.547 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:29:46.647 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:29:46.681 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:29:46.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:29:46.682 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:29:46.783 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:29:46.811 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:29:46.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:29:46.812 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:29:46.912 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:29:46.944 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:29:46.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:29:46.945 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:29:47.046 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:29:47.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:29:47.081 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:29:47.081 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:29:47.182 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:29:47.194 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:29:47.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:29:47.253 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:29:47.253 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:29:47.315 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:29:47.354 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:29:47.385 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:29:47.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:29:47.387 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:29:47.388 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:29:47 smithi183 ceph-mon[29736]: pgmap v907: 158 pgs: 158 active+clean; 7.0 GiB data, 27 GiB used, 599 GiB / 626 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.65k op/s 2024-09-10T19:29:47.388 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:29:47 smithi183 ceph-mon[29736]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T19:29:47.388 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:29:47 smithi183 ceph-mon[29736]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T19:29:47.388 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:29:47 smithi183 ceph-mon[29736]: from='mgr.14556 ' entity='mgr.y' 2024-09-10T19:29:47.389 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:29:47 smithi183 ceph-mon[41562]: pgmap v907: 158 pgs: 158 active+clean; 7.0 GiB data, 27 GiB used, 599 GiB / 626 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.65k op/s 2024-09-10T19:29:47.389 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:29:47 smithi183 ceph-mon[41562]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T19:29:47.389 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:29:47 smithi183 ceph-mon[41562]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T19:29:47.389 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:29:47 smithi183 ceph-mon[41562]: from='mgr.14556 ' entity='mgr.y' 2024-09-10T19:29:47.451 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:29:47 smithi191 ceph-mon[32884]: pgmap v907: 158 pgs: 158 active+clean; 7.0 GiB data, 27 GiB used, 599 GiB / 626 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.65k op/s 2024-09-10T19:29:47.451 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:29:47 smithi191 ceph-mon[32884]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T19:29:47.452 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:29:47 smithi191 ceph-mon[32884]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T19:29:47.452 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:29:47 smithi191 ceph-mon[32884]: from='mgr.14556 ' entity='mgr.y' 2024-09-10T19:29:47.487 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:29:47.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:29:47.522 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:29:47.523 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:29:47.623 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:29:47.658 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:29:47.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:29:47.659 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:29:47.715 INFO:journalctl@ceph.mgr.x.smithi191.stdout:Sep 10 19:29:47 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-x[34967]: ::ffff:172.21.15.191 - - [10/Sep/2024:19:29:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T19:29:47.760 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:29:47.797 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:29:47.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:29:47.800 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:29:47.901 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:29:47.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:29:47.937 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:29:47.938 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:29:48.038 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:29:48.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:29:48.095 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:29:48.095 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:29:48.196 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:29:48.201 INFO:teuthology.orchestra.run.smithi183.stdout:osd.0: {} 2024-09-10T19:29:48.202 INFO:teuthology.orchestra.run.smithi183.stdout:osd.1: {} 2024-09-10T19:29:48.202 INFO:teuthology.orchestra.run.smithi183.stdout:osd.2: {} 2024-09-10T19:29:48.202 INFO:teuthology.orchestra.run.smithi183.stdout:osd.3: {} 2024-09-10T19:29:48.202 INFO:teuthology.orchestra.run.smithi183.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-10T19:29:48.202 INFO:teuthology.orchestra.run.smithi183.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-10T19:29:48.202 INFO:teuthology.orchestra.run.smithi183.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-10T19:29:48.202 INFO:teuthology.orchestra.run.smithi183.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-10T19:29:48.224 INFO:teuthology.orchestra.run.smithi183.stdout:osd.4: {} 2024-09-10T19:29:48.225 INFO:teuthology.orchestra.run.smithi183.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-10T19:29:48.238 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:29:48.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:29:48.239 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:29:48.253 INFO:teuthology.orchestra.run.smithi183.stdout:osd.5: {} 2024-09-10T19:29:48.254 INFO:teuthology.orchestra.run.smithi183.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-10T19:29:48.285 INFO:teuthology.orchestra.run.smithi183.stdout:osd.6: {} 2024-09-10T19:29:48.286 INFO:teuthology.orchestra.run.smithi183.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-10T19:29:48.313 INFO:teuthology.orchestra.run.smithi183.stdout:osd.7: {} 2024-09-10T19:29:48.314 INFO:teuthology.orchestra.run.smithi183.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-10T19:29:48.314 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:29:48 smithi183 ceph-mon[29736]: from='client.? 172.21.15.183:0/2120293780' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T19:29:48.314 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:29:48 smithi183 ceph-mon[41562]: from='client.? 172.21.15.183:0/2120293780' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T19:29:48.341 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:29:48.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:29:48.373 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:29:48.374 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:29:48.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:29:48 smithi191 ceph-mon[32884]: from='client.? 172.21.15.183:0/2120293780' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T19:29:48.475 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:29:48.511 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:29:48.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:29:48.512 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:29:48.613 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:29:48.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:29:48.647 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:29:48.647 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:29:48.748 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:29:48.782 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:29:48.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:29:48.783 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:29:48.884 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:29:48.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:29:48.923 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:29:48.924 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:29:49.025 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:29:49.060 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:29:49.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:29:49.061 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:29:49.162 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:29:49.197 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:29:49.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:29:49.198 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:29:49.299 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:29:49.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:29:49.336 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:29:49.336 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:29:49.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:29:49 smithi183 ceph-mon[29736]: pgmap v908: 158 pgs: 158 active+clean; 6.9 GiB data, 26 GiB used, 600 GiB / 626 GiB avail; 921 B/s rd, 0 B/s wr, 1.47k op/s 2024-09-10T19:29:49.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:29:49 smithi183 ceph-mon[41562]: pgmap v908: 158 pgs: 158 active+clean; 6.9 GiB data, 26 GiB used, 600 GiB / 626 GiB avail; 921 B/s rd, 0 B/s wr, 1.47k op/s 2024-09-10T19:29:49.437 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:29:49.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:29:49 smithi191 ceph-mon[32884]: pgmap v908: 158 pgs: 158 active+clean; 6.9 GiB data, 26 GiB used, 600 GiB / 626 GiB avail; 921 B/s rd, 0 B/s wr, 1.47k op/s 2024-09-10T19:29:49.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:29:49.476 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:29:49.476 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:29:49.577 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:29:49.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:29:49.611 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:29:49.611 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:29:49.711 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:29:49.749 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:29:49.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:29:49.750 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:29:49.851 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:29:49.886 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:29:49.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:29:49.887 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:29:49.988 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:29:50.020 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:29:50.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:29:50.021 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:29:50.122 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:29:50.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:29:50.160 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:29:50.160 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:29:50.261 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:29:50.292 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:29:50.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:29:50.293 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:29:50.394 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:29:50.429 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:29:50.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:29:50.430 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:29:50.465 INFO:journalctl@ceph.iscsi.iscsi.a.smithi191.stdout:Sep 10 19:29:50 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-iscsi-iscsi-a[61816]: debug there is no tcmu-runner data available 2024-09-10T19:29:50.531 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:29:50.568 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:29:50.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:29:50.569 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:29:50.670 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:29:50.706 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:29:50.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:29:50.707 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:29:50.808 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:29:50.840 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:29:50.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:29:50.841 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:29:50.941 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:29:50.977 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:29:50.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:29:50.978 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:29:51.079 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:29:51.113 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:29:51.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:29:51.114 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:29:51.215 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:29:51.246 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:29:51.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:29:51.247 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:29:51.313 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:29:51 smithi183 ceph-mon[29736]: pgmap v909: 158 pgs: 158 active+clean; 6.9 GiB data, 26 GiB used, 600 GiB / 626 GiB avail; 921 B/s rd, 0 B/s wr, 1.47k op/s 2024-09-10T19:29:51.314 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:29:51 smithi183 ceph-mon[29736]: from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:29:51.314 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:29:51 smithi183 ceph-mon[41562]: pgmap v909: 158 pgs: 158 active+clean; 6.9 GiB data, 26 GiB used, 600 GiB / 626 GiB avail; 921 B/s rd, 0 B/s wr, 1.47k op/s 2024-09-10T19:29:51.314 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:29:51 smithi183 ceph-mon[41562]: from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:29:51.347 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:29:51.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:29:51.387 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:29:51.387 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:29:51.465 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:29:51 smithi191 ceph-mon[32884]: pgmap v909: 158 pgs: 158 active+clean; 6.9 GiB data, 26 GiB used, 600 GiB / 626 GiB avail; 921 B/s rd, 0 B/s wr, 1.47k op/s 2024-09-10T19:29:51.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:29:51 smithi191 ceph-mon[32884]: from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:29:51.488 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:29:51.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:29:51.524 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:29:51.524 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:29:51.625 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:29:51.656 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:29:51.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:29:51.657 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:29:51.668 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Sep 10 19:29:51 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-y[29989]: ::ffff:172.21.15.191 - - [10/Sep/2024:19:29:51] "GET /metrics HTTP/1.1" 200 224908 "" "Prometheus/2.43.0" 2024-09-10T19:29:51.758 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:29:51.791 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:29:51.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:29:51.792 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:29:51.893 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:29:51.923 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:29:51.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:29:51.924 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:29:52.025 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:29:52.060 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:29:52.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:29:52.061 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:29:52.104 INFO:teuthology.orchestra.run.smithi183.stderr:nodeep-scrub is set 2024-09-10T19:29:52.162 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:29:52.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:29:52 smithi183 ceph-mon[29736]: from='client.? 172.21.15.183:0/380315979' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-10T19:29:52.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:29:52 smithi183 ceph-mon[29736]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-10T19:29:52.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:29:52 smithi183 ceph-mon[41562]: from='client.? 172.21.15.183:0/380315979' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-10T19:29:52.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:29:52 smithi183 ceph-mon[41562]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-10T19:29:52.189 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:29:52.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:29:52.190 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:29:52.291 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:29:52.321 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:29:52.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:29:52.322 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:29:52.423 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:29:52.458 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:29:52.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:29:52.460 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:29:52.465 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:29:52 smithi191 ceph-mon[32884]: from='client.? 172.21.15.183:0/380315979' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-10T19:29:52.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:29:52 smithi191 ceph-mon[32884]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-10T19:29:52.562 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:29:52.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:29:52.600 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:29:52.601 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:29:52.701 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:29:52.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:29:52.739 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:29:52.739 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:29:52.840 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:29:52.877 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:29:52.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:29:52.879 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:29:52.979 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:29:53.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:29:53.019 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:29:53.020 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:29:53.112 INFO:teuthology.orchestra.run.smithi183.stderr:reweighted osd.5 to 1 (10000) 2024-09-10T19:29:53.121 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:29:53.158 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:29:53.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:29:53.159 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:29:53.260 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:29:53.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:29:53.303 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:29:53.303 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:29:53.404 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:29:53.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:29:53 smithi183 ceph-mon[29736]: pgmap v910: 158 pgs: 158 active+clean; 6.6 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 921 B/s rd, 0 B/s wr, 1.45k op/s 2024-09-10T19:29:53.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:29:53 smithi183 ceph-mon[29736]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T19:29:53.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:29:53 smithi183 ceph-mon[29736]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-10T19:29:53.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:29:53 smithi183 ceph-mon[29736]: osdmap e407: 8 total, 8 up, 7 in 2024-09-10T19:29:53.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:29:53 smithi183 ceph-mon[29736]: from='client.? 172.21.15.183:0/73876548' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-09-10T19:29:53.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:29:53 smithi183 ceph-mon[41562]: pgmap v910: 158 pgs: 158 active+clean; 6.6 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 921 B/s rd, 0 B/s wr, 1.45k op/s 2024-09-10T19:29:53.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:29:53 smithi183 ceph-mon[41562]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T19:29:53.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:29:53 smithi183 ceph-mon[41562]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-10T19:29:53.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:29:53 smithi183 ceph-mon[41562]: osdmap e407: 8 total, 8 up, 7 in 2024-09-10T19:29:53.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:29:53 smithi183 ceph-mon[41562]: from='client.? 172.21.15.183:0/73876548' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-09-10T19:29:53.442 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:29:53.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:29:53.443 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:29:53.465 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:29:53 smithi191 ceph-mon[32884]: pgmap v910: 158 pgs: 158 active+clean; 6.6 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 921 B/s rd, 0 B/s wr, 1.45k op/s 2024-09-10T19:29:53.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:29:53 smithi191 ceph-mon[32884]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T19:29:53.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:29:53 smithi191 ceph-mon[32884]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-10T19:29:53.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:29:53 smithi191 ceph-mon[32884]: osdmap e407: 8 total, 8 up, 7 in 2024-09-10T19:29:53.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:29:53 smithi191 ceph-mon[32884]: from='client.? 172.21.15.183:0/73876548' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-09-10T19:29:53.543 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:29:53.576 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:29:53.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:29:53.577 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:29:53.678 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:29:53.707 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:29:53.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:29:53.708 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:29:53.809 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:29:53.843 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:29:53.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:29:53.844 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:29:53.945 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:29:53.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:29:53.982 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:29:53.982 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:29:54.083 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:29:54.119 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:29:54.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:29:54.121 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:29:54.221 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:29:54.258 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:29:54.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:29:54.260 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:29:54.360 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:29:54.392 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:29:54.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:29:54.393 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:29:54.413 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:29:54 smithi183 ceph-mon[41562]: from='client.? 172.21.15.183:0/73876548' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-09-10T19:29:54.413 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:29:54 smithi183 ceph-mon[41562]: osdmap e408: 8 total, 8 up, 7 in 2024-09-10T19:29:54.418 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:29:54 smithi183 ceph-mon[29736]: from='client.? 172.21.15.183:0/73876548' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-09-10T19:29:54.418 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:29:54 smithi183 ceph-mon[29736]: osdmap e408: 8 total, 8 up, 7 in 2024-09-10T19:29:54.452 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph osd reweight 7 1 2024-09-10T19:29:54.465 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:29:54 smithi191 ceph-mon[32884]: from='client.? 172.21.15.183:0/73876548' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-09-10T19:29:54.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:29:54 smithi191 ceph-mon[32884]: osdmap e408: 8 total, 8 up, 7 in 2024-09-10T19:29:54.494 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:29:54.498 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-10T19:29:54.529 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:29:54.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:29:54.533 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:29:54.633 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:29:54.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:29:54.694 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:29:54.694 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:29:54.795 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:29:54.827 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:29:54.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:29:54.828 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:29:54.929 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:29:54.971 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:29:54.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:29:54.972 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:29:55.073 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:29:55.083 INFO:teuthology.orchestra.run.smithi183.stdout:{ 2024-09-10T19:29:55.084 INFO:teuthology.orchestra.run.smithi183.stdout: "ops": [ 2024-09-10T19:29:55.084 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:29:55.084 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.16026.0:189522 11.2 11:4fb75b4a:::benchmark_data_smithi183_203944_object44632:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e409)", 2024-09-10T19:29:55.084 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-10T19:29:55.068214+0000", 2024-09-10T19:29:55.084 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 0.0095023650000000005, 2024-09-10T19:29:55.084 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0095362220000000004, 2024-09-10T19:29:55.084 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-10T19:29:55.084 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "waiting for sub ops", 2024-09-10T19:29:55.084 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-10T19:29:55.084 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.16026", 2024-09-10T19:29:55.085 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.183:0/1450567484", 2024-09-10T19:29:55.085 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 189522 2024-09-10T19:29:55.085 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:29:55.085 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-10T19:29:55.085 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:29:55.085 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-10T19:29:55.085 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:29:55.068214+0000", 2024-09-10T19:29:55.085 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:29:55.085 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:29:55.085 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:29:55.086 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-10T19:29:55.086 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:29:55.068214+0000", 2024-09-10T19:29:55.086 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:29:55.086 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:29:55.086 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:29:55.086 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-10T19:29:55.086 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:29:55.068213+0000", 2024-09-10T19:29:55.086 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4294967295.9999986 2024-09-10T19:29:55.086 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:29:55.087 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:29:55.087 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-10T19:29:55.087 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:29:55.068220+0000", 2024-09-10T19:29:55.087 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.5639999999999999e-06 2024-09-10T19:29:55.087 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:29:55.087 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:29:55.087 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-10T19:29:55.087 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:29:55.068234+0000", 2024-09-10T19:29:55.087 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.3920000000000001e-05 2024-09-10T19:29:55.088 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:29:55.088 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:29:55.088 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-10T19:29:55.088 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:29:55.068242+0000", 2024-09-10T19:29:55.088 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.6699999999999994e-06 2024-09-10T19:29:55.088 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:29:55.088 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:29:55.088 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-10T19:29:55.088 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:29:55.072446+0000", 2024-09-10T19:29:55.088 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0042038759999999996 2024-09-10T19:29:55.089 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:29:55.089 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:29:55.089 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-10T19:29:55.089 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:29:55.072591+0000", 2024-09-10T19:29:55.089 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000145076 2024-09-10T19:29:55.089 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:29:55.089 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:29:55.089 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 0,7", 2024-09-10T19:29:55.089 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:29:55.072720+0000", 2024-09-10T19:29:55.089 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000129746 2024-09-10T19:29:55.089 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:29:55.089 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:29:55.089 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:29:55.089 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:29:55.090 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:29:55.090 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_repop_reply(client.16026.0:189522 11.2 e409/371 ondisk, result = 0)", 2024-09-10T19:29:55.090 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-10T19:29:55.077683+0000", 2024-09-10T19:29:55.090 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 3.3157999999999997e-05, 2024-09-10T19:29:55.090 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00015040299999999999, 2024-09-10T19:29:55.090 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-10T19:29:55.090 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "started", 2024-09-10T19:29:55.090 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-10T19:29:55.090 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:29:55.090 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-10T19:29:55.090 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:29:55.077683+0000", 2024-09-10T19:29:55.090 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:29:55.091 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:29:55.091 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:29:55.091 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-10T19:29:55.091 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:29:55.077683+0000", 2024-09-10T19:29:55.091 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:29:55.091 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:29:55.091 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:29:55.091 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-10T19:29:55.091 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:29:55.077682+0000", 2024-09-10T19:29:55.091 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4294967295.999999 2024-09-10T19:29:55.091 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:29:55.091 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:29:55.091 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-10T19:29:55.091 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:29:55.077690+0000", 2024-09-10T19:29:55.092 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.1400000000000002e-06 2024-09-10T19:29:55.092 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:29:55.092 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:29:55.092 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-10T19:29:55.092 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:29:55.077707+0000", 2024-09-10T19:29:55.092 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.7007000000000002e-05 2024-09-10T19:29:55.092 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:29:55.092 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:29:55.092 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-10T19:29:55.092 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:29:55.077713+0000", 2024-09-10T19:29:55.092 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.4919999999999998e-06 2024-09-10T19:29:55.092 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:29:55.092 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:29:55.092 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-10T19:29:55.093 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:29:55.077783+0000", 2024-09-10T19:29:55.093 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.9467000000000004e-05 2024-09-10T19:29:55.093 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:29:55.093 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:29:55.093 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-10T19:29:55.093 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:29:55.077790+0000", 2024-09-10T19:29:55.093 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.7789999999999996e-06 2024-09-10T19:29:55.093 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:29:55.093 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:29:55.093 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-10T19:29:55.093 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:29:55.077834+0000", 2024-09-10T19:29:55.093 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.3775e-05 2024-09-10T19:29:55.093 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:29:55.093 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:29:55.094 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:29:55.094 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:29:55.094 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:29:55.094 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_repop(client.16026.0:189523 11.0 e409/371)", 2024-09-10T19:29:55.094 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-10T19:29:55.077862+0000", 2024-09-10T19:29:55.094 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 4294967295.9998546, 2024-09-10T19:29:55.094 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.3414999999999998e-05, 2024-09-10T19:29:55.094 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-10T19:29:55.094 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "queued for pg", 2024-09-10T19:29:55.094 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-10T19:29:55.094 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:29:55.094 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-10T19:29:55.094 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:29:55.077862+0000", 2024-09-10T19:29:55.094 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:29:55.095 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:29:55.095 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:29:55.095 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-10T19:29:55.095 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:29:55.077862+0000", 2024-09-10T19:29:55.095 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:29:55.095 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:29:55.095 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:29:55.095 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-10T19:29:55.095 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:29:55.077861+0000", 2024-09-10T19:29:55.095 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4294967295.999999 2024-09-10T19:29:55.095 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:29:55.095 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:29:55.095 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-10T19:29:55.095 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:29:55.077870+0000", 2024-09-10T19:29:55.096 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.6260000000000001e-06 2024-09-10T19:29:55.096 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:29:55.096 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:29:55.096 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-10T19:29:55.096 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:29:55.077882+0000", 2024-09-10T19:29:55.096 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.1678000000000001e-05 2024-09-10T19:29:55.096 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:29:55.096 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:29:55.096 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-10T19:29:55.096 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:29:55.077886+0000", 2024-09-10T19:29:55.096 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.8740000000000003e-06 2024-09-10T19:29:55.096 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:29:55.096 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:29:55.096 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:29:55.097 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:29:55.097 INFO:teuthology.orchestra.run.smithi183.stdout: ], 2024-09-10T19:29:55.097 INFO:teuthology.orchestra.run.smithi183.stdout: "num_ops": 3 2024-09-10T19:29:55.097 INFO:teuthology.orchestra.run.smithi183.stdout:} 2024-09-10T19:29:55.123 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:29:55.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:29:55.124 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:29:55.210 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph osd unset noscrub 2024-09-10T19:29:55.225 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:29:55.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:29:55.254 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:29:55.254 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:29:55.355 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:29:55.389 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:29:55.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:29:55.390 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:29:55.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:29:55 smithi183 ceph-mon[41562]: purged_snaps scrub starts 2024-09-10T19:29:55.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:29:55 smithi183 ceph-mon[41562]: purged_snaps scrub ok 2024-09-10T19:29:55.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:29:55 smithi183 ceph-mon[41562]: pgmap v913: 158 pgs: 158 active+clean; 6.5 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.06k op/s 2024-09-10T19:29:55.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:29:55 smithi183 ceph-mon[41562]: osdmap e409: 8 total, 8 up, 7 in 2024-09-10T19:29:55.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:29:55 smithi183 ceph-mon[29736]: purged_snaps scrub starts 2024-09-10T19:29:55.433 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:29:55 smithi183 ceph-mon[29736]: purged_snaps scrub ok 2024-09-10T19:29:55.433 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:29:55 smithi183 ceph-mon[29736]: pgmap v913: 158 pgs: 158 active+clean; 6.5 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.06k op/s 2024-09-10T19:29:55.433 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:29:55 smithi183 ceph-mon[29736]: osdmap e409: 8 total, 8 up, 7 in 2024-09-10T19:29:55.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:29:55 smithi191 ceph-mon[32884]: purged_snaps scrub starts 2024-09-10T19:29:55.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:29:55 smithi191 ceph-mon[32884]: purged_snaps scrub ok 2024-09-10T19:29:55.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:29:55 smithi191 ceph-mon[32884]: pgmap v913: 158 pgs: 158 active+clean; 6.5 GiB data, 25 GiB used, 601 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.06k op/s 2024-09-10T19:29:55.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:29:55 smithi191 ceph-mon[32884]: osdmap e409: 8 total, 8 up, 7 in 2024-09-10T19:29:55.491 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:29:55.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:29:55.521 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:29:55.522 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:29:55.622 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:29:55.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:29:55.659 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:29:55.659 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:29:55.760 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:29:55.793 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:29:55.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:29:55.795 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:29:55.895 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:29:55.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:29:55.930 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:29:55.931 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:29:55.976 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph daemon osd.3 dump_blocked_ops 2024-09-10T19:29:56.004 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:29:56.032 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:29:56.064 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:29:56.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:29:56.065 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:29:56.068 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:29:56.166 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:29:56.194 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:29:56.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:29:56.195 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:29:56.296 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:29:56.329 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:29:56.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:29:56.330 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:29:56.431 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:29:56.469 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:29:56.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:29:56.470 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:29:56.571 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:29:56.609 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:29:56.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:29:56.610 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:29:56.711 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:29:56.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:29:56.754 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:29:56.754 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:29:56.855 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:29:56.889 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:29:56.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:29:56.890 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:29:56.991 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:29:57.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:29:57.027 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:29:57.027 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:29:57.128 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:29:57.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:29:57.168 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:29:57.168 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:29:57.226 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:29:57.249 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:29:57 smithi183 ceph-mon[29736]: pgmap v915: 158 pgs: 158 active+clean; 6.1 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 2.04k op/s 2024-09-10T19:29:57.249 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:29:57 smithi183 ceph-mon[41562]: pgmap v915: 158 pgs: 158 active+clean; 6.1 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 2.04k op/s 2024-09-10T19:29:57.269 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:29:57.295 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:29:57.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:29:57.296 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:29:57.397 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:29:57.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:29:57.430 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:29:57.430 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:29:57.453 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:29:57 smithi191 ceph-mon[32884]: pgmap v915: 158 pgs: 158 active+clean; 6.1 GiB data, 24 GiB used, 602 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 2.04k op/s 2024-09-10T19:29:57.531 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:29:57.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:29:57.568 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:29:57.568 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:29:57.669 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:29:57.701 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:29:57.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:29:57.703 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:29:57.715 INFO:journalctl@ceph.mgr.x.smithi191.stdout:Sep 10 19:29:57 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-x[34967]: ::ffff:172.21.15.191 - - [10/Sep/2024:19:29:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T19:29:57.803 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:29:57.852 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:29:57.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:29:57.853 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:29:57.953 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:29:57.987 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:29:57.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:29:57.988 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:29:58.089 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:29:58.122 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:29:58.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:29:58.123 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:29:58.223 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:29:58.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:29:58.258 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:29:58.258 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:29:58.359 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:29:58.399 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:29:58.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:29:58.400 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:29:58.501 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:29:58.538 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:29:58.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:29:58.539 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:29:58.639 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:29:58.671 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:29:58.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:29:58.672 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:29:58.773 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:29:58.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:29:58.812 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:29:58.812 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:29:58.913 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:29:58.943 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:29:58.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:29:58.944 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:29:59.045 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:29:59.076 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:29:59.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:29:59.077 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:29:59.178 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:29:59.214 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:29:59.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:29:59.215 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:29:59.316 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:29:59.348 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:29:59.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:29:59.349 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:29:59.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:29:59 smithi183 ceph-mon[29736]: pgmap v916: 158 pgs: 158 active+clean; 6.1 GiB data, 23 GiB used, 602 GiB / 626 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.44k op/s 2024-09-10T19:29:59.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:29:59 smithi183 ceph-mon[29736]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:29:59.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:29:59 smithi183 ceph-mon[41562]: pgmap v916: 158 pgs: 158 active+clean; 6.1 GiB data, 23 GiB used, 602 GiB / 626 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.44k op/s 2024-09-10T19:29:59.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:29:59 smithi183 ceph-mon[41562]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:29:59.449 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:29:59.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:29:59 smithi191 ceph-mon[32884]: pgmap v916: 158 pgs: 158 active+clean; 6.1 GiB data, 23 GiB used, 602 GiB / 626 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.44k op/s 2024-09-10T19:29:59.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:29:59 smithi191 ceph-mon[32884]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:29:59.488 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:29:59.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:29:59.489 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:29:59.590 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:29:59.625 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:29:59.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:29:59.626 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:29:59.727 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:29:59.761 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:29:59.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:29:59.762 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:29:59.863 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:29:59.896 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:29:59.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:29:59.897 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:29:59.998 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:30:00.030 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:30:00.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:30:00.031 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:30:00.132 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:30:00.163 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:30:00.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:30:00.164 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:30:00.265 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:30:00.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:30:00.307 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:30:00.307 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:30:00.408 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:30:00.440 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:30:00.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:30:00.441 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:30:00.466 INFO:journalctl@ceph.iscsi.iscsi.a.smithi191.stdout:Sep 10 19:30:00 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-iscsi-iscsi-a[61816]: debug there is no tcmu-runner data available 2024-09-10T19:30:00.541 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:30:00.573 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:30:00.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:30:00.574 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:30:00.641 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:30:00.675 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:30:00.710 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:30:00.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:30:00.711 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:30:00.812 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:30:00.845 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:30:00.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:30:00.846 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:30:00.934 INFO:teuthology.orchestra.run.smithi183.stdout:osd.0: {} 2024-09-10T19:30:00.935 INFO:teuthology.orchestra.run.smithi183.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-10T19:30:00.947 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:30:00.968 INFO:teuthology.orchestra.run.smithi183.stdout:osd.1: {} 2024-09-10T19:30:00.968 INFO:teuthology.orchestra.run.smithi183.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-10T19:30:00.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:30:00.993 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:30:00.993 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:30:01.001 INFO:teuthology.orchestra.run.smithi183.stdout:osd.2: {} 2024-09-10T19:30:01.002 INFO:teuthology.orchestra.run.smithi183.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-10T19:30:01.009 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:30:00 smithi183 ceph-mon[29736]: pgmap v917: 158 pgs: 158 active+clean; 6.0 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.15k op/s 2024-09-10T19:30:01.009 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:30:00 smithi183 ceph-mon[29736]: Health detail: HEALTH_WARN noscrub,nodeep-scrub flag(s) set 2024-09-10T19:30:01.009 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:30:00 smithi183 ceph-mon[29736]: [WRN] OSDMAP_FLAGS: noscrub,nodeep-scrub flag(s) set 2024-09-10T19:30:01.009 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:30:00 smithi183 ceph-mon[29736]: from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:30:01.009 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:30:00 smithi183 ceph-mon[29736]: from='client.? 172.21.15.183:0/1879957744' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-09-10T19:30:01.009 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:30:00 smithi183 ceph-mon[29736]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-09-10T19:30:01.009 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:30:00 smithi183 ceph-mon[29736]: from='client.? 172.21.15.183:0/4006232693' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T19:30:01.009 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:30:00 smithi183 ceph-mon[41562]: pgmap v917: 158 pgs: 158 active+clean; 6.0 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.15k op/s 2024-09-10T19:30:01.009 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:30:00 smithi183 ceph-mon[41562]: Health detail: HEALTH_WARN noscrub,nodeep-scrub flag(s) set 2024-09-10T19:30:01.009 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:30:00 smithi183 ceph-mon[41562]: [WRN] OSDMAP_FLAGS: noscrub,nodeep-scrub flag(s) set 2024-09-10T19:30:01.010 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:30:00 smithi183 ceph-mon[41562]: from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:30:01.010 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:30:00 smithi183 ceph-mon[41562]: from='client.? 172.21.15.183:0/1879957744' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-09-10T19:30:01.010 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:30:00 smithi183 ceph-mon[41562]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-09-10T19:30:01.010 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:30:00 smithi183 ceph-mon[41562]: from='client.? 172.21.15.183:0/4006232693' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T19:30:01.033 INFO:teuthology.orchestra.run.smithi183.stdout:osd.3: {} 2024-09-10T19:30:01.033 INFO:teuthology.orchestra.run.smithi183.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-10T19:30:01.060 INFO:teuthology.orchestra.run.smithi183.stdout:osd.4: {} 2024-09-10T19:30:01.060 INFO:teuthology.orchestra.run.smithi183.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-10T19:30:01.090 INFO:teuthology.orchestra.run.smithi183.stdout:osd.5: {} 2024-09-10T19:30:01.090 INFO:teuthology.orchestra.run.smithi183.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-10T19:30:01.094 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:30:01.125 INFO:teuthology.orchestra.run.smithi183.stdout:osd.6: {} 2024-09-10T19:30:01.125 INFO:teuthology.orchestra.run.smithi183.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-10T19:30:01.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:30:01.137 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:30:01.138 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:30:01.159 INFO:teuthology.orchestra.run.smithi183.stdout:osd.7: {} 2024-09-10T19:30:01.159 INFO:teuthology.orchestra.run.smithi183.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-10T19:30:01.162 INFO:teuthology.orchestra.run.smithi183.stderr:reweighted osd.7 to 1 (10000) 2024-09-10T19:30:01.238 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:30:01.267 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:30:01.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:30:01.268 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:30:01.369 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:30:01.402 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:30:01.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:30:01.403 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:30:01.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:30:00 smithi191 ceph-mon[32884]: pgmap v917: 158 pgs: 158 active+clean; 6.0 GiB data, 23 GiB used, 603 GiB / 626 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.15k op/s 2024-09-10T19:30:01.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:30:00 smithi191 ceph-mon[32884]: Health detail: HEALTH_WARN noscrub,nodeep-scrub flag(s) set 2024-09-10T19:30:01.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:30:00 smithi191 ceph-mon[32884]: [WRN] OSDMAP_FLAGS: noscrub,nodeep-scrub flag(s) set 2024-09-10T19:30:01.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:30:00 smithi191 ceph-mon[32884]: from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:30:01.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:30:00 smithi191 ceph-mon[32884]: from='client.? 172.21.15.183:0/1879957744' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-09-10T19:30:01.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:30:00 smithi191 ceph-mon[32884]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-09-10T19:30:01.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:30:00 smithi191 ceph-mon[32884]: from='client.? 172.21.15.183:0/4006232693' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T19:30:01.505 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:30:01.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:30:01.537 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:30:01.537 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:30:01.638 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:30:01.671 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:30:01.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:30:01.672 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:30:01.681 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Sep 10 19:30:01 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-y[29989]: ::ffff:172.21.15.191 - - [10/Sep/2024:19:30:01] "GET /metrics HTTP/1.1" 200 224907 "" "Prometheus/2.43.0" 2024-09-10T19:30:01.773 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:30:01.807 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:30:01.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:30:01.808 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:30:01.909 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:30:01.946 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:30:01.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:30:01.948 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:30:02.049 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:30:02.085 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:30:02.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:30:02.086 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:30:02.187 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:30:02.219 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:30:02.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:30:02.220 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:30:02.321 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:30:02.360 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:30:02.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:30:02.361 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:30:02.416 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:30:02 smithi183 ceph-mon[29736]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-09-10T19:30:02.416 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:30:02 smithi183 ceph-mon[29736]: osdmap e410: 8 total, 8 up, 7 in 2024-09-10T19:30:02.416 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:30:02 smithi183 ceph-mon[29736]: purged_snaps scrub starts 2024-09-10T19:30:02.417 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:30:02 smithi183 ceph-mon[29736]: purged_snaps scrub ok 2024-09-10T19:30:02.417 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:30:02 smithi183 ceph-mon[41562]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-09-10T19:30:02.417 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:30:02 smithi183 ceph-mon[41562]: osdmap e410: 8 total, 8 up, 7 in 2024-09-10T19:30:02.417 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:30:02 smithi183 ceph-mon[41562]: purged_snaps scrub starts 2024-09-10T19:30:02.417 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:30:02 smithi183 ceph-mon[41562]: purged_snaps scrub ok 2024-09-10T19:30:02.462 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:30:02.465 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:30:02 smithi191 ceph-mon[32884]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-09-10T19:30:02.465 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:30:02 smithi191 ceph-mon[32884]: osdmap e410: 8 total, 8 up, 7 in 2024-09-10T19:30:02.465 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:30:02 smithi191 ceph-mon[32884]: purged_snaps scrub starts 2024-09-10T19:30:02.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:30:02 smithi191 ceph-mon[32884]: purged_snaps scrub ok 2024-09-10T19:30:02.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:30:02.496 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:30:02.496 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:30:02.597 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:30:02.633 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:30:02.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:30:02.635 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:30:02.735 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:30:02.767 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:30:02.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:30:02.768 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:30:02.869 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:30:02.906 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:30:02.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:30:02.907 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:30:03.008 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:30:03.039 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:30:03.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:30:03.040 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:30:03.141 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:30:03.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:30:03.182 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:30:03.183 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:30:03.284 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:30:03.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:30:03.331 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:30:03.331 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:30:03.358 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:30:03 smithi183 ceph-mon[29736]: pgmap v919: 158 pgs: 158 active+clean; 5.7 GiB data, 22 GiB used, 603 GiB / 626 GiB avail; 511 B/s rd, 0 B/s wr, 1.67k op/s 2024-09-10T19:30:03.358 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:30:03 smithi183 ceph-mon[29736]: osdmap e411: 8 total, 8 up, 7 in 2024-09-10T19:30:03.358 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:30:03 smithi183 ceph-mon[41562]: pgmap v919: 158 pgs: 158 active+clean; 5.7 GiB data, 22 GiB used, 603 GiB / 626 GiB avail; 511 B/s rd, 0 B/s wr, 1.67k op/s 2024-09-10T19:30:03.358 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:30:03 smithi183 ceph-mon[41562]: osdmap e411: 8 total, 8 up, 7 in 2024-09-10T19:30:03.433 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:30:03.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:30:03.461 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:30:03.462 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:30:03.465 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:30:03 smithi191 ceph-mon[32884]: pgmap v919: 158 pgs: 158 active+clean; 5.7 GiB data, 22 GiB used, 603 GiB / 626 GiB avail; 511 B/s rd, 0 B/s wr, 1.67k op/s 2024-09-10T19:30:03.465 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:30:03 smithi191 ceph-mon[32884]: osdmap e411: 8 total, 8 up, 7 in 2024-09-10T19:30:03.562 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:30:03.597 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:30:03.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:30:03.598 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:30:03.615 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph osd reweight 1 1 2024-09-10T19:30:03.699 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:30:03.740 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:30:03.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:30:03.741 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:30:03.842 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:30:03.877 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:30:03.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:30:03.878 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:30:03.979 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:30:04.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:30:04.013 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:30:04.014 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:30:04.114 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:30:04.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:30:04.154 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:30:04.155 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:30:04.182 INFO:teuthology.orchestra.run.smithi183.stderr:noscrub is unset 2024-09-10T19:30:04.255 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:30:04.285 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:30:04.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:30:04.286 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:30:04.387 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:30:04.425 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:30:04.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:30:04.426 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:30:04.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:30:04 smithi183 ceph-mon[29736]: from='client.? 172.21.15.183:0/2941273503' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-10T19:30:04.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:30:04 smithi183 ceph-mon[41562]: from='client.? 172.21.15.183:0/2941273503' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-10T19:30:04.465 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:30:04 smithi191 ceph-mon[32884]: from='client.? 172.21.15.183:0/2941273503' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-10T19:30:04.527 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:30:04.566 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:30:04.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:30:04.568 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:30:04.668 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:30:04.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:30:04.710 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:30:04.710 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:30:04.812 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:30:04.850 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:30:04.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:30:04.851 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:30:04.877 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-10T19:30:04.952 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:30:04.987 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:30:04.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:30:04.988 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:30:05.089 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:30:05.120 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:30:05.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:30:05.121 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:30:05.222 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:30:05.263 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:30:05.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:30:05.264 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:30:05.365 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:30:05.400 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:30:05.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:30:05.401 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:30:05.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:30:05 smithi183 ceph-mon[29736]: pgmap v921: 158 pgs: 158 active+clean; 5.6 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.11k op/s 2024-09-10T19:30:05.433 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:30:05 smithi183 ceph-mon[29736]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T19:30:05.433 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:30:05 smithi183 ceph-mon[29736]: from='client.? 172.21.15.183:0/2941273503' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-10T19:30:05.433 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:30:05 smithi183 ceph-mon[29736]: osdmap e412: 8 total, 8 up, 7 in 2024-09-10T19:30:05.433 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:30:05 smithi183 ceph-mon[29736]: 1.a scrub starts 2024-09-10T19:30:05.433 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:30:05 smithi183 ceph-mon[29736]: 1.a scrub ok 2024-09-10T19:30:05.433 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:30:05 smithi183 ceph-mon[29736]: 11.d scrub starts 2024-09-10T19:30:05.433 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:30:05 smithi183 ceph-mon[41562]: pgmap v921: 158 pgs: 158 active+clean; 5.6 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.11k op/s 2024-09-10T19:30:05.433 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:30:05 smithi183 ceph-mon[41562]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T19:30:05.433 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:30:05 smithi183 ceph-mon[41562]: from='client.? 172.21.15.183:0/2941273503' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-10T19:30:05.433 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:30:05 smithi183 ceph-mon[41562]: osdmap e412: 8 total, 8 up, 7 in 2024-09-10T19:30:05.434 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:30:05 smithi183 ceph-mon[41562]: 1.a scrub starts 2024-09-10T19:30:05.434 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:30:05 smithi183 ceph-mon[41562]: 1.a scrub ok 2024-09-10T19:30:05.434 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:30:05 smithi183 ceph-mon[41562]: 11.d scrub starts 2024-09-10T19:30:05.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:30:05 smithi191 ceph-mon[32884]: pgmap v921: 158 pgs: 158 active+clean; 5.6 GiB data, 22 GiB used, 604 GiB / 626 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.11k op/s 2024-09-10T19:30:05.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:30:05 smithi191 ceph-mon[32884]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T19:30:05.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:30:05 smithi191 ceph-mon[32884]: from='client.? 172.21.15.183:0/2941273503' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-10T19:30:05.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:30:05 smithi191 ceph-mon[32884]: osdmap e412: 8 total, 8 up, 7 in 2024-09-10T19:30:05.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:30:05 smithi191 ceph-mon[32884]: 1.a scrub starts 2024-09-10T19:30:05.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:30:05 smithi191 ceph-mon[32884]: 1.a scrub ok 2024-09-10T19:30:05.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:30:05 smithi191 ceph-mon[32884]: 11.d scrub starts 2024-09-10T19:30:05.503 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:30:05.505 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:30:05.532 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:30:05.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:30:05.533 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:30:05.544 INFO:teuthology.orchestra.run.smithi183.stdout:ERROR: (22) Invalid argument 2024-09-10T19:30:05.544 INFO:teuthology.orchestra.run.smithi183.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-10T19:30:05.634 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:30:05.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:30:05.677 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:30:05.677 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:30:05.778 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:30:05.812 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:30:05.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:30:05.813 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:30:05.914 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:30:05.948 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:30:05.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:30:05.949 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:30:06.050 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:30:06.082 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:30:06.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:30:06.083 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:30:06.184 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:30:06.219 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:30:06.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:30:06.220 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:30:06.321 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:30:06.360 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:30:06.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:30:06.361 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:30:06.424 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:30:06 smithi183 ceph-mon[29736]: 4.17 scrub starts 2024-09-10T19:30:06.424 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:30:06 smithi183 ceph-mon[29736]: 4.17 scrub ok 2024-09-10T19:30:06.424 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:30:06 smithi183 ceph-mon[29736]: 5.a scrub starts 2024-09-10T19:30:06.424 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:30:06 smithi183 ceph-mon[29736]: 5.a scrub ok 2024-09-10T19:30:06.424 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:30:06 smithi183 ceph-mon[29736]: purged_snaps scrub starts 2024-09-10T19:30:06.424 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:30:06 smithi183 ceph-mon[29736]: purged_snaps scrub ok 2024-09-10T19:30:06.424 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:30:06 smithi183 ceph-mon[29736]: 6.5 scrub starts 2024-09-10T19:30:06.424 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:30:06 smithi183 ceph-mon[29736]: 6.5 scrub ok 2024-09-10T19:30:06.424 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:30:06 smithi183 ceph-mon[41562]: 4.17 scrub starts 2024-09-10T19:30:06.425 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:30:06 smithi183 ceph-mon[41562]: 4.17 scrub ok 2024-09-10T19:30:06.425 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:30:06 smithi183 ceph-mon[41562]: 5.a scrub starts 2024-09-10T19:30:06.425 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:30:06 smithi183 ceph-mon[41562]: 5.a scrub ok 2024-09-10T19:30:06.425 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:30:06 smithi183 ceph-mon[41562]: purged_snaps scrub starts 2024-09-10T19:30:06.425 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:30:06 smithi183 ceph-mon[41562]: purged_snaps scrub ok 2024-09-10T19:30:06.425 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:30:06 smithi183 ceph-mon[41562]: 6.5 scrub starts 2024-09-10T19:30:06.425 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:30:06 smithi183 ceph-mon[41562]: 6.5 scrub ok 2024-09-10T19:30:06.435 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:30:06.462 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:30:06.465 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:30:06 smithi191 ceph-mon[32884]: 4.17 scrub starts 2024-09-10T19:30:06.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:30:06 smithi191 ceph-mon[32884]: 4.17 scrub ok 2024-09-10T19:30:06.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:30:06 smithi191 ceph-mon[32884]: 5.a scrub starts 2024-09-10T19:30:06.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:30:06 smithi191 ceph-mon[32884]: 5.a scrub ok 2024-09-10T19:30:06.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:30:06 smithi191 ceph-mon[32884]: purged_snaps scrub starts 2024-09-10T19:30:06.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:30:06 smithi191 ceph-mon[32884]: purged_snaps scrub ok 2024-09-10T19:30:06.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:30:06 smithi191 ceph-mon[32884]: 6.5 scrub starts 2024-09-10T19:30:06.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:30:06 smithi191 ceph-mon[32884]: 6.5 scrub ok 2024-09-10T19:30:06.469 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph daemon osd.3 dump_historic_ops 2024-09-10T19:30:06.492 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:30:06.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:30:06.493 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:30:06.594 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:30:06.645 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:30:06.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:30:06.646 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:30:06.747 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:30:06.778 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:30:06.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:30:06.780 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:30:06.880 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:30:06.920 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:30:06.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:30:06.921 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:30:07.022 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:30:07.060 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:30:07.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:30:07.061 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:30:07.162 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:30:07.202 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:30:07.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:30:07.203 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:30:07.301 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-10T19:30:07.303 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:30:07.338 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:30:07 smithi183 ceph-mon[29736]: 11.b scrub starts 2024-09-10T19:30:07.339 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:30:07 smithi183 ceph-mon[29736]: 4.5 scrub starts 2024-09-10T19:30:07.339 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:30:07 smithi183 ceph-mon[29736]: 4.5 scrub ok 2024-09-10T19:30:07.339 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:30:07 smithi183 ceph-mon[29736]: pgmap v923: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 5.2 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 2.13k op/s 2024-09-10T19:30:07.339 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:30:07 smithi183 ceph-mon[29736]: osdmap e413: 8 total, 8 up, 7 in 2024-09-10T19:30:07.339 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:30:07 smithi183 ceph-mon[29736]: 11.d scrub ok 2024-09-10T19:30:07.339 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:30:07 smithi183 ceph-mon[29736]: 4.8 scrub starts 2024-09-10T19:30:07.339 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:30:07 smithi183 ceph-mon[29736]: 4.8 scrub ok 2024-09-10T19:30:07.339 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:30:07 smithi183 ceph-mon[29736]: 11.f scrub starts 2024-09-10T19:30:07.339 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:30:07 smithi183 ceph-mon[41562]: 11.b scrub starts 2024-09-10T19:30:07.339 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:30:07 smithi183 ceph-mon[41562]: 4.5 scrub starts 2024-09-10T19:30:07.340 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:30:07 smithi183 ceph-mon[41562]: 4.5 scrub ok 2024-09-10T19:30:07.340 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:30:07 smithi183 ceph-mon[41562]: pgmap v923: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 5.2 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 2.13k op/s 2024-09-10T19:30:07.340 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:30:07 smithi183 ceph-mon[41562]: osdmap e413: 8 total, 8 up, 7 in 2024-09-10T19:30:07.340 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:30:07 smithi183 ceph-mon[41562]: 11.d scrub ok 2024-09-10T19:30:07.340 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:30:07 smithi183 ceph-mon[41562]: 4.8 scrub starts 2024-09-10T19:30:07.340 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:30:07 smithi183 ceph-mon[41562]: 4.8 scrub ok 2024-09-10T19:30:07.340 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:30:07 smithi183 ceph-mon[41562]: 11.f scrub starts 2024-09-10T19:30:07.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:30:07.343 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:30:07.343 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:30:07.443 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:30:07.453 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:30:07 smithi191 ceph-mon[32884]: 11.b scrub starts 2024-09-10T19:30:07.453 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:30:07 smithi191 ceph-mon[32884]: 4.5 scrub starts 2024-09-10T19:30:07.453 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:30:07 smithi191 ceph-mon[32884]: 4.5 scrub ok 2024-09-10T19:30:07.453 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:30:07 smithi191 ceph-mon[32884]: pgmap v923: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 5.2 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 2.13k op/s 2024-09-10T19:30:07.454 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:30:07 smithi191 ceph-mon[32884]: osdmap e413: 8 total, 8 up, 7 in 2024-09-10T19:30:07.454 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:30:07 smithi191 ceph-mon[32884]: 11.d scrub ok 2024-09-10T19:30:07.454 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:30:07 smithi191 ceph-mon[32884]: 4.8 scrub starts 2024-09-10T19:30:07.454 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:30:07 smithi191 ceph-mon[32884]: 4.8 scrub ok 2024-09-10T19:30:07.454 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:30:07 smithi191 ceph-mon[32884]: 11.f scrub starts 2024-09-10T19:30:07.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:30:07.472 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:30:07.472 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:30:07.573 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:30:07.605 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:30:07.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:30:07.606 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:30:07.707 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:30:07.716 INFO:journalctl@ceph.mgr.x.smithi191.stdout:Sep 10 19:30:07 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-x[34967]: ::ffff:172.21.15.191 - - [10/Sep/2024:19:30:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T19:30:07.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:30:07.745 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:30:07.745 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:30:07.846 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:30:07.880 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:30:07.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:30:07.882 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:30:07.982 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:30:08.005 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:30:08.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:30:08.006 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:30:08.106 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:30:08.143 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:30:08.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:30:08.144 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:30:08.245 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:30:08.276 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:30:08.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:30:08.277 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:30:08.378 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:30:08.418 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:30:08.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:30:08.419 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:30:08.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:30:08 smithi183 ceph-mon[29736]: 4.10 scrub starts 2024-09-10T19:30:08.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:30:08 smithi183 ceph-mon[29736]: 4.10 scrub ok 2024-09-10T19:30:08.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:30:08 smithi183 ceph-mon[29736]: 5.19 scrub starts 2024-09-10T19:30:08.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:30:08 smithi183 ceph-mon[29736]: 5.19 scrub ok 2024-09-10T19:30:08.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:30:08 smithi183 ceph-mon[29736]: 11.c scrub starts 2024-09-10T19:30:08.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:30:08 smithi183 ceph-mon[29736]: 5.e scrub starts 2024-09-10T19:30:08.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:30:08 smithi183 ceph-mon[29736]: 5.e scrub ok 2024-09-10T19:30:08.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:30:08 smithi183 ceph-mon[41562]: 4.10 scrub starts 2024-09-10T19:30:08.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:30:08 smithi183 ceph-mon[41562]: 4.10 scrub ok 2024-09-10T19:30:08.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:30:08 smithi183 ceph-mon[41562]: 5.19 scrub starts 2024-09-10T19:30:08.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:30:08 smithi183 ceph-mon[41562]: 5.19 scrub ok 2024-09-10T19:30:08.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:30:08 smithi183 ceph-mon[41562]: 11.c scrub starts 2024-09-10T19:30:08.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:30:08 smithi183 ceph-mon[41562]: 5.e scrub starts 2024-09-10T19:30:08.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:30:08 smithi183 ceph-mon[41562]: 5.e scrub ok 2024-09-10T19:30:08.465 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:30:08 smithi191 ceph-mon[32884]: 4.10 scrub starts 2024-09-10T19:30:08.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:30:08 smithi191 ceph-mon[32884]: 4.10 scrub ok 2024-09-10T19:30:08.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:30:08 smithi191 ceph-mon[32884]: 5.19 scrub starts 2024-09-10T19:30:08.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:30:08 smithi191 ceph-mon[32884]: 5.19 scrub ok 2024-09-10T19:30:08.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:30:08 smithi191 ceph-mon[32884]: 11.c scrub starts 2024-09-10T19:30:08.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:30:08 smithi191 ceph-mon[32884]: 5.e scrub starts 2024-09-10T19:30:08.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:30:08 smithi191 ceph-mon[32884]: 5.e scrub ok 2024-09-10T19:30:08.519 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:30:08.555 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:30:08.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:30:08.556 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:30:08.656 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:30:08.694 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:30:08.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:30:08.695 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:30:08.796 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:30:08.827 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:30:08.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:30:08.829 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:30:08.929 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:30:08.964 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:30:08.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:30:08.965 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:30:09.066 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:30:09.088 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:30:09.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:30:09.089 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:30:09.190 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:30:09.223 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:30:09.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:30:09.225 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:30:09.325 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:30:09.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:30:09.361 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:30:09.361 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:30:09.462 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:30:09.465 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:30:09 smithi191 ceph-mon[32884]: 11.b scrub ok 2024-09-10T19:30:09.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:30:09 smithi191 ceph-mon[32884]: 11.8 scrub starts 2024-09-10T19:30:09.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:30:09 smithi191 ceph-mon[32884]: pgmap v925: 158 pgs: 2 active+clean+scrubbing, 156 active+clean; 5.1 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.48k op/s 2024-09-10T19:30:09.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:30:09 smithi191 ceph-mon[32884]: 6.19 scrub starts 2024-09-10T19:30:09.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:30:09 smithi191 ceph-mon[32884]: 6.19 scrub ok 2024-09-10T19:30:09.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:30:09 smithi191 ceph-mon[32884]: 4.c scrub starts 2024-09-10T19:30:09.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:30:09 smithi191 ceph-mon[32884]: 4.c scrub ok 2024-09-10T19:30:09.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:30:09 smithi191 ceph-mon[32884]: 11.f scrub ok 2024-09-10T19:30:09.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:30:09.490 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:30:09.491 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:30:09.591 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:30:09.624 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:30:09.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:30:09.626 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:30:09.681 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:30:09 smithi183 ceph-mon[29736]: 11.b scrub ok 2024-09-10T19:30:09.681 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:30:09 smithi183 ceph-mon[29736]: 11.8 scrub starts 2024-09-10T19:30:09.681 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:30:09 smithi183 ceph-mon[29736]: pgmap v925: 158 pgs: 2 active+clean+scrubbing, 156 active+clean; 5.1 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.48k op/s 2024-09-10T19:30:09.682 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:30:09 smithi183 ceph-mon[29736]: 6.19 scrub starts 2024-09-10T19:30:09.682 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:30:09 smithi183 ceph-mon[29736]: 6.19 scrub ok 2024-09-10T19:30:09.682 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:30:09 smithi183 ceph-mon[29736]: 4.c scrub starts 2024-09-10T19:30:09.682 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:30:09 smithi183 ceph-mon[29736]: 4.c scrub ok 2024-09-10T19:30:09.682 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:30:09 smithi183 ceph-mon[29736]: 11.f scrub ok 2024-09-10T19:30:09.682 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:30:09 smithi183 ceph-mon[41562]: 11.b scrub ok 2024-09-10T19:30:09.682 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:30:09 smithi183 ceph-mon[41562]: 11.8 scrub starts 2024-09-10T19:30:09.682 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:30:09 smithi183 ceph-mon[41562]: pgmap v925: 158 pgs: 2 active+clean+scrubbing, 156 active+clean; 5.1 GiB data, 21 GiB used, 605 GiB / 626 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.48k op/s 2024-09-10T19:30:09.682 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:30:09 smithi183 ceph-mon[41562]: 6.19 scrub starts 2024-09-10T19:30:09.682 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:30:09 smithi183 ceph-mon[41562]: 6.19 scrub ok 2024-09-10T19:30:09.682 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:30:09 smithi183 ceph-mon[41562]: 4.c scrub starts 2024-09-10T19:30:09.682 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:30:09 smithi183 ceph-mon[41562]: 4.c scrub ok 2024-09-10T19:30:09.683 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:30:09 smithi183 ceph-mon[41562]: 11.f scrub ok 2024-09-10T19:30:09.726 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:30:09.759 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:30:09.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:30:09.760 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:30:09.861 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:30:09.895 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:30:09.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:30:09.896 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:30:09.997 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:30:10.032 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:30:10.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:30:10.033 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:30:10.136 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:30:10.168 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:30:10.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:30:10.169 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:30:10.272 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:30:10.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:30:10.305 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:30:10.305 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:30:10.406 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:30:10.435 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:30:10.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:30:10.436 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:30:10.466 INFO:journalctl@ceph.iscsi.iscsi.a.smithi191.stdout:Sep 10 19:30:10 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-iscsi-iscsi-a[61816]: debug there is no tcmu-runner data available 2024-09-10T19:30:10.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:30:10 smithi191 ceph-mon[32884]: 11.8 scrub ok 2024-09-10T19:30:10.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:30:10 smithi191 ceph-mon[32884]: 1.1 scrub starts 2024-09-10T19:30:10.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:30:10 smithi191 ceph-mon[32884]: 1.1 scrub ok 2024-09-10T19:30:10.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:30:10 smithi191 ceph-mon[32884]: 11.c scrub ok 2024-09-10T19:30:10.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:30:10 smithi191 ceph-mon[32884]: 5.9 scrub starts 2024-09-10T19:30:10.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:30:10 smithi191 ceph-mon[32884]: 5.9 scrub ok 2024-09-10T19:30:10.501 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:30:10 smithi183 ceph-mon[29736]: 11.8 scrub ok 2024-09-10T19:30:10.501 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:30:10 smithi183 ceph-mon[29736]: 1.1 scrub starts 2024-09-10T19:30:10.501 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:30:10 smithi183 ceph-mon[29736]: 1.1 scrub ok 2024-09-10T19:30:10.501 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:30:10 smithi183 ceph-mon[29736]: 11.c scrub ok 2024-09-10T19:30:10.501 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:30:10 smithi183 ceph-mon[29736]: 5.9 scrub starts 2024-09-10T19:30:10.501 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:30:10 smithi183 ceph-mon[29736]: 5.9 scrub ok 2024-09-10T19:30:10.502 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:30:10 smithi183 ceph-mon[41562]: 11.8 scrub ok 2024-09-10T19:30:10.502 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:30:10 smithi183 ceph-mon[41562]: 1.1 scrub starts 2024-09-10T19:30:10.502 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:30:10 smithi183 ceph-mon[41562]: 1.1 scrub ok 2024-09-10T19:30:10.502 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:30:10 smithi183 ceph-mon[41562]: 11.c scrub ok 2024-09-10T19:30:10.502 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:30:10 smithi183 ceph-mon[41562]: 5.9 scrub starts 2024-09-10T19:30:10.502 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:30:10 smithi183 ceph-mon[41562]: 5.9 scrub ok 2024-09-10T19:30:10.537 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:30:10.573 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:30:10.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:30:10.575 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:30:10.676 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:30:10.704 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:30:10.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:30:10.705 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:30:10.806 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:30:10.835 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:30:10.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:30:10.837 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:30:10.937 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:30:10.972 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:30:10.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:30:10.973 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:30:11.074 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:30:11.108 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:30:11.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:30:11.110 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:30:11.210 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:30:11.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:30:11.248 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:30:11.248 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:30:11.317 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:30:11 smithi183 ceph-mon[29736]: 4.19 scrub starts 2024-09-10T19:30:11.317 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:30:11 smithi183 ceph-mon[29736]: 4.19 scrub ok 2024-09-10T19:30:11.317 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:30:11 smithi183 ceph-mon[29736]: 1.7 scrub starts 2024-09-10T19:30:11.317 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:30:11 smithi183 ceph-mon[29736]: 1.7 scrub ok 2024-09-10T19:30:11.317 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:30:11 smithi183 ceph-mon[29736]: pgmap v926: 158 pgs: 2 active+clean+scrubbing, 156 active+clean; 5.1 GiB data, 20 GiB used, 605 GiB / 626 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.19k op/s 2024-09-10T19:30:11.317 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:30:11 smithi183 ceph-mon[29736]: from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:30:11.317 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:30:11 smithi183 ceph-mon[29736]: 1.5 scrub starts 2024-09-10T19:30:11.317 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:30:11 smithi183 ceph-mon[29736]: 1.5 scrub ok 2024-09-10T19:30:11.318 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:30:11 smithi183 ceph-mon[41562]: 4.19 scrub starts 2024-09-10T19:30:11.318 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:30:11 smithi183 ceph-mon[41562]: 4.19 scrub ok 2024-09-10T19:30:11.318 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:30:11 smithi183 ceph-mon[41562]: 1.7 scrub starts 2024-09-10T19:30:11.318 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:30:11 smithi183 ceph-mon[41562]: 1.7 scrub ok 2024-09-10T19:30:11.318 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:30:11 smithi183 ceph-mon[41562]: pgmap v926: 158 pgs: 2 active+clean+scrubbing, 156 active+clean; 5.1 GiB data, 20 GiB used, 605 GiB / 626 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.19k op/s 2024-09-10T19:30:11.318 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:30:11 smithi183 ceph-mon[41562]: from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:30:11.318 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:30:11 smithi183 ceph-mon[41562]: 1.5 scrub starts 2024-09-10T19:30:11.318 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:30:11 smithi183 ceph-mon[41562]: 1.5 scrub ok 2024-09-10T19:30:11.349 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:30:11.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:30:11.381 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:30:11.381 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:30:11.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:30:11 smithi191 ceph-mon[32884]: 4.19 scrub starts 2024-09-10T19:30:11.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:30:11 smithi191 ceph-mon[32884]: 4.19 scrub ok 2024-09-10T19:30:11.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:30:11 smithi191 ceph-mon[32884]: 1.7 scrub starts 2024-09-10T19:30:11.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:30:11 smithi191 ceph-mon[32884]: 1.7 scrub ok 2024-09-10T19:30:11.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:30:11 smithi191 ceph-mon[32884]: pgmap v926: 158 pgs: 2 active+clean+scrubbing, 156 active+clean; 5.1 GiB data, 20 GiB used, 605 GiB / 626 GiB avail; 1.3 KiB/s rd, 0 B/s wr, 1.19k op/s 2024-09-10T19:30:11.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:30:11 smithi191 ceph-mon[32884]: from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:30:11.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:30:11 smithi191 ceph-mon[32884]: 1.5 scrub starts 2024-09-10T19:30:11.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:30:11 smithi191 ceph-mon[32884]: 1.5 scrub ok 2024-09-10T19:30:11.483 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:30:11.505 INFO:teuthology.orchestra.run.smithi183.stdout:osd.0: {} 2024-09-10T19:30:11.505 INFO:teuthology.orchestra.run.smithi183.stdout:osd.1: {} 2024-09-10T19:30:11.505 INFO:teuthology.orchestra.run.smithi183.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-10T19:30:11.505 INFO:teuthology.orchestra.run.smithi183.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-10T19:30:11.521 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:30:11.521 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:30:11.525 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:30:11.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:30:11.526 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:30:11.531 INFO:teuthology.orchestra.run.smithi183.stdout:osd.2: {} 2024-09-10T19:30:11.531 INFO:teuthology.orchestra.run.smithi183.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-10T19:30:11.569 INFO:teuthology.orchestra.run.smithi183.stdout:osd.3: {} 2024-09-10T19:30:11.569 INFO:teuthology.orchestra.run.smithi183.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-10T19:30:11.570 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Sep 10 19:30:11 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-y[29989]: ::ffff:172.21.15.191 - - [10/Sep/2024:19:30:11] "GET /metrics HTTP/1.1" 200 224907 "" "Prometheus/2.43.0" 2024-09-10T19:30:11.598 INFO:teuthology.orchestra.run.smithi183.stdout:osd.4: {} 2024-09-10T19:30:11.598 INFO:teuthology.orchestra.run.smithi183.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-10T19:30:11.628 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:30:11.635 INFO:teuthology.orchestra.run.smithi183.stdout:osd.5: {} 2024-09-10T19:30:11.635 INFO:teuthology.orchestra.run.smithi183.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-10T19:30:11.668 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:30:11.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:30:11.669 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:30:11.672 INFO:teuthology.orchestra.run.smithi183.stdout:osd.6: {} 2024-09-10T19:30:11.672 INFO:teuthology.orchestra.run.smithi183.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-10T19:30:11.701 INFO:teuthology.orchestra.run.smithi183.stdout:osd.7: {} 2024-09-10T19:30:11.702 INFO:teuthology.orchestra.run.smithi183.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-10T19:30:11.770 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:30:11.805 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:30:11.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:30:11.807 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:30:11.907 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:30:11.948 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:30:11.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:30:11.949 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:30:12.050 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:30:12.082 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:30:12.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:30:12.083 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:30:12.184 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:30:12.217 INFO:teuthology.orchestra.run.smithi183.stderr:reweighted osd.1 to 1 (10000) 2024-09-10T19:30:12.219 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:30:12.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:30:12.220 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:30:12.321 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:30:12.355 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:30:12.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:30:12.356 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:30:12.457 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:30:12.465 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:30:12 smithi183 ceph-mon[29736]: 2.17 scrub starts 2024-09-10T19:30:12.465 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:30:12 smithi183 ceph-mon[29736]: 2.17 scrub ok 2024-09-10T19:30:12.465 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:30:12 smithi183 ceph-mon[29736]: from='client.? 172.21.15.183:0/3203676695' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T19:30:12.466 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:30:12 smithi183 ceph-mon[29736]: from='client.? 172.21.15.183:0/166614471' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-09-10T19:30:12.466 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:30:12 smithi183 ceph-mon[29736]: 5.10 scrub starts 2024-09-10T19:30:12.466 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:30:12 smithi183 ceph-mon[29736]: 5.10 scrub ok 2024-09-10T19:30:12.466 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:30:12 smithi183 ceph-mon[29736]: purged_snaps scrub starts 2024-09-10T19:30:12.466 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:30:12 smithi183 ceph-mon[29736]: purged_snaps scrub ok 2024-09-10T19:30:12.466 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:30:12 smithi183 ceph-mon[41562]: 2.17 scrub starts 2024-09-10T19:30:12.466 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:30:12 smithi183 ceph-mon[41562]: 2.17 scrub ok 2024-09-10T19:30:12.466 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:30:12 smithi183 ceph-mon[41562]: from='client.? 172.21.15.183:0/3203676695' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T19:30:12.466 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:30:12 smithi183 ceph-mon[41562]: from='client.? 172.21.15.183:0/166614471' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-09-10T19:30:12.466 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:30:12 smithi183 ceph-mon[41562]: 5.10 scrub starts 2024-09-10T19:30:12.467 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:30:12 smithi183 ceph-mon[41562]: 5.10 scrub ok 2024-09-10T19:30:12.467 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:30:12 smithi183 ceph-mon[41562]: purged_snaps scrub starts 2024-09-10T19:30:12.467 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:30:12 smithi183 ceph-mon[41562]: purged_snaps scrub ok 2024-09-10T19:30:12.467 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:30:12 smithi191 ceph-mon[32884]: 2.17 scrub starts 2024-09-10T19:30:12.467 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:30:12 smithi191 ceph-mon[32884]: 2.17 scrub ok 2024-09-10T19:30:12.467 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:30:12 smithi191 ceph-mon[32884]: from='client.? 172.21.15.183:0/3203676695' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T19:30:12.467 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:30:12 smithi191 ceph-mon[32884]: from='client.? 172.21.15.183:0/166614471' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-09-10T19:30:12.467 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:30:12 smithi191 ceph-mon[32884]: 5.10 scrub starts 2024-09-10T19:30:12.467 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:30:12 smithi191 ceph-mon[32884]: 5.10 scrub ok 2024-09-10T19:30:12.468 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:30:12 smithi191 ceph-mon[32884]: purged_snaps scrub starts 2024-09-10T19:30:12.468 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:30:12 smithi191 ceph-mon[32884]: purged_snaps scrub ok 2024-09-10T19:30:12.490 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:30:12.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:30:12.491 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:30:12.591 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:30:12.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:30:12.630 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:30:12.630 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:30:12.731 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:30:12.767 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:30:12.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:30:12.768 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:30:12.869 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:30:12.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:30:12.910 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:30:12.910 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:30:13.011 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:30:13.043 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:30:13.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:30:13.044 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:30:13.145 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:30:13.183 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:30:13.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:30:13.185 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:30:13.286 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:30:13.321 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:30:13.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:30:13.322 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:30:13.423 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:30:13.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:30:13.458 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:30:13.458 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:30:13.465 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:30:13 smithi191 ceph-mon[32884]: pgmap v927: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 4.8 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 1.64k op/s 2024-09-10T19:30:13.465 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:30:13 smithi191 ceph-mon[32884]: from='client.? 172.21.15.183:0/166614471' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-09-10T19:30:13.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:30:13 smithi191 ceph-mon[32884]: osdmap e414: 8 total, 8 up, 7 in 2024-09-10T19:30:13.559 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:30:13.590 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:30:13.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:30:13.591 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:30:13.639 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:30:13 smithi183 ceph-mon[29736]: pgmap v927: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 4.8 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 1.64k op/s 2024-09-10T19:30:13.640 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:30:13 smithi183 ceph-mon[29736]: from='client.? 172.21.15.183:0/166614471' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-09-10T19:30:13.640 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:30:13 smithi183 ceph-mon[29736]: osdmap e414: 8 total, 8 up, 7 in 2024-09-10T19:30:13.640 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:30:13 smithi183 ceph-mon[41562]: pgmap v927: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 4.8 GiB data, 20 GiB used, 606 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 1.64k op/s 2024-09-10T19:30:13.640 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:30:13 smithi183 ceph-mon[41562]: from='client.? 172.21.15.183:0/166614471' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-09-10T19:30:13.640 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:30:13 smithi183 ceph-mon[41562]: osdmap e414: 8 total, 8 up, 7 in 2024-09-10T19:30:13.677 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph osd reweight 6 1 2024-09-10T19:30:13.691 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:30:13.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:30:13.770 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:30:13.770 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:30:13.871 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:30:13.903 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:30:13.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:30:13.904 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:30:14.005 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:30:14.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:30:14.039 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:30:14.039 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:30:14.140 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:30:14.171 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:30:14.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:30:14.174 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:30:14.275 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:30:14.313 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:30:14.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:30:14.314 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:30:14.415 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:30:14.449 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:30:14.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:30:14.450 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:30:14.536 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:30:14 smithi183 ceph-mon[29736]: 4.a scrub starts 2024-09-10T19:30:14.536 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:30:14 smithi183 ceph-mon[29736]: 4.a scrub ok 2024-09-10T19:30:14.536 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:30:14 smithi183 ceph-mon[29736]: 11.0 scrub starts 2024-09-10T19:30:14.536 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:30:14 smithi183 ceph-mon[29736]: 4.b scrub starts 2024-09-10T19:30:14.536 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:30:14 smithi183 ceph-mon[29736]: 4.b scrub ok 2024-09-10T19:30:14.536 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:30:14 smithi183 ceph-mon[29736]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:30:14.536 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:30:14 smithi183 ceph-mon[41562]: 4.a scrub starts 2024-09-10T19:30:14.536 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:30:14 smithi183 ceph-mon[41562]: 4.a scrub ok 2024-09-10T19:30:14.537 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:30:14 smithi183 ceph-mon[41562]: 11.0 scrub starts 2024-09-10T19:30:14.537 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:30:14 smithi183 ceph-mon[41562]: 4.b scrub starts 2024-09-10T19:30:14.537 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:30:14 smithi183 ceph-mon[41562]: 4.b scrub ok 2024-09-10T19:30:14.537 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:30:14 smithi183 ceph-mon[41562]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:30:14.551 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:30:14.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:30:14.604 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:30:14.604 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:30:14.705 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:30:14.715 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:30:14 smithi191 ceph-mon[32884]: 4.a scrub starts 2024-09-10T19:30:14.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:30:14 smithi191 ceph-mon[32884]: 4.a scrub ok 2024-09-10T19:30:14.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:30:14 smithi191 ceph-mon[32884]: 11.0 scrub starts 2024-09-10T19:30:14.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:30:14 smithi191 ceph-mon[32884]: 4.b scrub starts 2024-09-10T19:30:14.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:30:14 smithi191 ceph-mon[32884]: 4.b scrub ok 2024-09-10T19:30:14.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:30:14 smithi191 ceph-mon[32884]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:30:14.743 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:30:14.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:30:14.744 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:30:14.845 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:30:14.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:30:14.882 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:30:14.882 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:30:14.983 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:30:15.026 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:30:15.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:30:15.027 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:30:15.128 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:30:15.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:30:15.163 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:30:15.163 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:30:15.264 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:30:15.301 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:30:15.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:30:15.302 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:30:15.403 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:30:15.430 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:30:15.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:30:15.431 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:30:15.532 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:30:15.565 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:30:15.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:30:15.566 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:30:15.666 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:30:15.681 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:30:15 smithi183 ceph-mon[29736]: 11.a scrub starts 2024-09-10T19:30:15.681 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:30:15 smithi183 ceph-mon[29736]: pgmap v929: 158 pgs: 158 active+clean; 4.7 GiB data, 19 GiB used, 606 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.04k op/s 2024-09-10T19:30:15.681 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:30:15 smithi183 ceph-mon[29736]: 5.3 scrub starts 2024-09-10T19:30:15.682 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:30:15 smithi183 ceph-mon[29736]: 5.3 scrub ok 2024-09-10T19:30:15.682 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:30:15 smithi183 ceph-mon[29736]: 6.3 scrub starts 2024-09-10T19:30:15.682 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:30:15 smithi183 ceph-mon[29736]: 6.3 scrub ok 2024-09-10T19:30:15.682 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:30:15 smithi183 ceph-mon[41562]: 11.a scrub starts 2024-09-10T19:30:15.682 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:30:15 smithi183 ceph-mon[41562]: pgmap v929: 158 pgs: 158 active+clean; 4.7 GiB data, 19 GiB used, 606 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.04k op/s 2024-09-10T19:30:15.682 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:30:15 smithi183 ceph-mon[41562]: 5.3 scrub starts 2024-09-10T19:30:15.682 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:30:15 smithi183 ceph-mon[41562]: 5.3 scrub ok 2024-09-10T19:30:15.682 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:30:15 smithi183 ceph-mon[41562]: 6.3 scrub starts 2024-09-10T19:30:15.682 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:30:15 smithi183 ceph-mon[41562]: 6.3 scrub ok 2024-09-10T19:30:15.705 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:30:15.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:30:15.706 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:30:15.715 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:30:15 smithi191 ceph-mon[32884]: 11.a scrub starts 2024-09-10T19:30:15.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:30:15 smithi191 ceph-mon[32884]: pgmap v929: 158 pgs: 158 active+clean; 4.7 GiB data, 19 GiB used, 606 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.04k op/s 2024-09-10T19:30:15.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:30:15 smithi191 ceph-mon[32884]: 5.3 scrub starts 2024-09-10T19:30:15.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:30:15 smithi191 ceph-mon[32884]: 5.3 scrub ok 2024-09-10T19:30:15.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:30:15 smithi191 ceph-mon[32884]: 6.3 scrub starts 2024-09-10T19:30:15.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:30:15 smithi191 ceph-mon[32884]: 6.3 scrub ok 2024-09-10T19:30:15.807 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:30:15.847 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:30:15.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:30:15.848 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:30:15.948 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:30:15.970 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:30:15.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:30:15.971 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:30:16.072 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:30:16.101 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:30:16.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:30:16.102 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:30:16.203 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:30:16.232 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:30:16.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:30:16.233 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:30:16.334 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:30:16.370 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:30:16.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:30:16.371 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:30:16.472 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:30:16.505 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:30:16.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:30:16.507 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:30:16.607 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:30:16.645 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:30:16.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:30:16.646 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:30:16.681 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:30:16 smithi183 ceph-mon[29736]: 11.9 scrub starts 2024-09-10T19:30:16.681 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:30:16 smithi183 ceph-mon[29736]: 5.f scrub starts 2024-09-10T19:30:16.682 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:30:16 smithi183 ceph-mon[29736]: 5.f scrub ok 2024-09-10T19:30:16.682 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:30:16 smithi183 ceph-mon[29736]: 11.0 scrub ok 2024-09-10T19:30:16.682 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:30:16 smithi183 ceph-mon[29736]: 2.f scrub starts 2024-09-10T19:30:16.682 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:30:16 smithi183 ceph-mon[29736]: 2.f scrub ok 2024-09-10T19:30:16.682 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:30:16 smithi183 ceph-mon[41562]: 11.9 scrub starts 2024-09-10T19:30:16.682 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:30:16 smithi183 ceph-mon[41562]: 5.f scrub starts 2024-09-10T19:30:16.682 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:30:16 smithi183 ceph-mon[41562]: 5.f scrub ok 2024-09-10T19:30:16.682 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:30:16 smithi183 ceph-mon[41562]: 11.0 scrub ok 2024-09-10T19:30:16.682 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:30:16 smithi183 ceph-mon[41562]: 2.f scrub starts 2024-09-10T19:30:16.682 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:30:16 smithi183 ceph-mon[41562]: 2.f scrub ok 2024-09-10T19:30:16.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:30:16 smithi191 ceph-mon[32884]: 11.9 scrub starts 2024-09-10T19:30:16.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:30:16 smithi191 ceph-mon[32884]: 5.f scrub starts 2024-09-10T19:30:16.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:30:16 smithi191 ceph-mon[32884]: 5.f scrub ok 2024-09-10T19:30:16.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:30:16 smithi191 ceph-mon[32884]: 11.0 scrub ok 2024-09-10T19:30:16.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:30:16 smithi191 ceph-mon[32884]: 2.f scrub starts 2024-09-10T19:30:16.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:30:16 smithi191 ceph-mon[32884]: 2.f scrub ok 2024-09-10T19:30:16.747 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:30:16.787 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:30:16.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:30:16.789 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:30:16.890 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:30:16.927 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:30:16.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:30:16.928 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:30:17.029 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:30:17.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:30:17.067 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:30:17.067 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:30:17.168 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:30:17.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:30:17.202 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:30:17.202 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:30:17.303 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:30:17.341 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:30:17.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:30:17.342 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:30:17.443 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:30:17.465 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:30:17.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:30:17.466 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:30:17.567 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:30:17.602 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:30:17.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:30:17.603 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:30:17.681 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:30:17 smithi183 ceph-mon[29736]: 11.a scrub ok 2024-09-10T19:30:17.681 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:30:17 smithi183 ceph-mon[29736]: pgmap v930: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 4.4 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 3.8 KiB/s rd, 0 B/s wr, 1.45k op/s 2024-09-10T19:30:17.681 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:30:17 smithi183 ceph-mon[29736]: 4.3 scrub starts 2024-09-10T19:30:17.682 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:30:17 smithi183 ceph-mon[29736]: 4.3 scrub ok 2024-09-10T19:30:17.682 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:30:17 smithi183 ceph-mon[29736]: 6.1a scrub starts 2024-09-10T19:30:17.682 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:30:17 smithi183 ceph-mon[29736]: 6.1a scrub ok 2024-09-10T19:30:17.682 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:30:17 smithi183 ceph-mon[41562]: 11.a scrub ok 2024-09-10T19:30:17.682 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:30:17 smithi183 ceph-mon[41562]: pgmap v930: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 4.4 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 3.8 KiB/s rd, 0 B/s wr, 1.45k op/s 2024-09-10T19:30:17.682 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:30:17 smithi183 ceph-mon[41562]: 4.3 scrub starts 2024-09-10T19:30:17.682 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:30:17 smithi183 ceph-mon[41562]: 4.3 scrub ok 2024-09-10T19:30:17.682 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:30:17 smithi183 ceph-mon[41562]: 6.1a scrub starts 2024-09-10T19:30:17.682 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:30:17 smithi183 ceph-mon[41562]: 6.1a scrub ok 2024-09-10T19:30:17.704 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:30:17.716 INFO:journalctl@ceph.mgr.x.smithi191.stdout:Sep 10 19:30:17 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-x[34967]: ::ffff:172.21.15.191 - - [10/Sep/2024:19:30:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T19:30:17.717 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:30:17 smithi191 ceph-mon[32884]: 11.a scrub ok 2024-09-10T19:30:17.717 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:30:17 smithi191 ceph-mon[32884]: pgmap v930: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 4.4 GiB data, 19 GiB used, 607 GiB / 626 GiB avail; 3.8 KiB/s rd, 0 B/s wr, 1.45k op/s 2024-09-10T19:30:17.717 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:30:17 smithi191 ceph-mon[32884]: 4.3 scrub starts 2024-09-10T19:30:17.717 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:30:17 smithi191 ceph-mon[32884]: 4.3 scrub ok 2024-09-10T19:30:17.717 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:30:17 smithi191 ceph-mon[32884]: 6.1a scrub starts 2024-09-10T19:30:17.717 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:30:17 smithi191 ceph-mon[32884]: 6.1a scrub ok 2024-09-10T19:30:17.737 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:30:17.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:30:17.738 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:30:17.839 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:30:17.878 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:30:17.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:30:17.879 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:30:17.980 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:30:18.020 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:30:18.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:30:18.021 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:30:18.124 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:30:18.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:30:18.159 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:30:18.159 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:30:18.260 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:30:18.297 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:30:18.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:30:18.298 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:30:18.399 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:30:18.434 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:30:18.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:30:18.435 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:30:18.536 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:30:18.573 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:30:18.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:30:18.574 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:30:18.675 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:30:18.681 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:30:18 smithi183 ceph-mon[29736]: purged_snaps scrub starts 2024-09-10T19:30:18.681 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:30:18 smithi183 ceph-mon[29736]: purged_snaps scrub ok 2024-09-10T19:30:18.682 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:30:18 smithi183 ceph-mon[29736]: 11.2 scrub starts 2024-09-10T19:30:18.682 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:30:18 smithi183 ceph-mon[29736]: 11.9 scrub ok 2024-09-10T19:30:18.682 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:30:18 smithi183 ceph-mon[29736]: osdmap e415: 8 total, 8 up, 7 in 2024-09-10T19:30:18.682 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:30:18 smithi183 ceph-mon[29736]: 4.1a scrub starts 2024-09-10T19:30:18.682 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:30:18 smithi183 ceph-mon[29736]: 4.1a scrub ok 2024-09-10T19:30:18.682 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:30:18 smithi183 ceph-mon[29736]: 2.0 scrub starts 2024-09-10T19:30:18.682 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:30:18 smithi183 ceph-mon[29736]: 2.0 scrub ok 2024-09-10T19:30:18.682 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:30:18 smithi183 ceph-mon[41562]: purged_snaps scrub starts 2024-09-10T19:30:18.682 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:30:18 smithi183 ceph-mon[41562]: purged_snaps scrub ok 2024-09-10T19:30:18.682 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:30:18 smithi183 ceph-mon[41562]: 11.2 scrub starts 2024-09-10T19:30:18.682 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:30:18 smithi183 ceph-mon[41562]: 11.9 scrub ok 2024-09-10T19:30:18.683 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:30:18 smithi183 ceph-mon[41562]: osdmap e415: 8 total, 8 up, 7 in 2024-09-10T19:30:18.683 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:30:18 smithi183 ceph-mon[41562]: 4.1a scrub starts 2024-09-10T19:30:18.683 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:30:18 smithi183 ceph-mon[41562]: 4.1a scrub ok 2024-09-10T19:30:18.683 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:30:18 smithi183 ceph-mon[41562]: 2.0 scrub starts 2024-09-10T19:30:18.683 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:30:18 smithi183 ceph-mon[41562]: 2.0 scrub ok 2024-09-10T19:30:18.707 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:30:18.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:30:18.708 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:30:18.715 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:30:18 smithi191 ceph-mon[32884]: purged_snaps scrub starts 2024-09-10T19:30:18.715 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:30:18 smithi191 ceph-mon[32884]: purged_snaps scrub ok 2024-09-10T19:30:18.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:30:18 smithi191 ceph-mon[32884]: 11.2 scrub starts 2024-09-10T19:30:18.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:30:18 smithi191 ceph-mon[32884]: 11.9 scrub ok 2024-09-10T19:30:18.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:30:18 smithi191 ceph-mon[32884]: osdmap e415: 8 total, 8 up, 7 in 2024-09-10T19:30:18.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:30:18 smithi191 ceph-mon[32884]: 4.1a scrub starts 2024-09-10T19:30:18.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:30:18 smithi191 ceph-mon[32884]: 4.1a scrub ok 2024-09-10T19:30:18.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:30:18 smithi191 ceph-mon[32884]: 2.0 scrub starts 2024-09-10T19:30:18.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:30:18 smithi191 ceph-mon[32884]: 2.0 scrub ok 2024-09-10T19:30:18.809 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:30:18.847 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:30:18.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:30:18.848 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:30:18.949 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:30:18.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:30:18.992 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:30:18.992 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:30:19.093 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:30:19.128 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:30:19.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:30:19.129 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:30:19.230 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:30:19.266 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:30:19.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:30:19.267 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:30:19.367 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:30:19.402 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:30:19.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:30:19.403 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:30:19.503 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:30:19.534 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:30:19.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:30:19.535 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:30:19.636 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:30:19.672 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:30:19.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:30:19.673 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:30:19.681 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:30:19 smithi183 ceph-mon[29736]: 6.b scrub starts 2024-09-10T19:30:19.681 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:30:19 smithi183 ceph-mon[29736]: 6.b scrub ok 2024-09-10T19:30:19.681 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:30:19 smithi183 ceph-mon[29736]: pgmap v932: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 4.3 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 23 KiB/s rd, 0 B/s wr, 1.70k op/s 2024-09-10T19:30:19.681 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:30:19 smithi183 ceph-mon[29736]: 11.2 scrub ok 2024-09-10T19:30:19.681 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:30:19 smithi183 ceph-mon[29736]: 4.e scrub starts 2024-09-10T19:30:19.682 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:30:19 smithi183 ceph-mon[29736]: 4.e scrub ok 2024-09-10T19:30:19.682 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:30:19 smithi183 ceph-mon[41562]: 6.b scrub starts 2024-09-10T19:30:19.682 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:30:19 smithi183 ceph-mon[41562]: 6.b scrub ok 2024-09-10T19:30:19.682 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:30:19 smithi183 ceph-mon[41562]: pgmap v932: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 4.3 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 23 KiB/s rd, 0 B/s wr, 1.70k op/s 2024-09-10T19:30:19.682 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:30:19 smithi183 ceph-mon[41562]: 11.2 scrub ok 2024-09-10T19:30:19.682 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:30:19 smithi183 ceph-mon[41562]: 4.e scrub starts 2024-09-10T19:30:19.682 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:30:19 smithi183 ceph-mon[41562]: 4.e scrub ok 2024-09-10T19:30:19.715 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:30:19 smithi191 ceph-mon[32884]: 6.b scrub starts 2024-09-10T19:30:19.715 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:30:19 smithi191 ceph-mon[32884]: 6.b scrub ok 2024-09-10T19:30:19.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:30:19 smithi191 ceph-mon[32884]: pgmap v932: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 4.3 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 23 KiB/s rd, 0 B/s wr, 1.70k op/s 2024-09-10T19:30:19.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:30:19 smithi191 ceph-mon[32884]: 11.2 scrub ok 2024-09-10T19:30:19.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:30:19 smithi191 ceph-mon[32884]: 4.e scrub starts 2024-09-10T19:30:19.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:30:19 smithi191 ceph-mon[32884]: 4.e scrub ok 2024-09-10T19:30:19.774 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:30:19.807 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:30:19.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:30:19.809 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:30:19.909 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:30:19.941 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:30:19.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:30:19.943 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:30:20.044 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:30:20.080 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:30:20.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:30:20.081 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:30:20.182 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:30:20.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:30:20.215 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:30:20.215 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:30:20.316 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:30:20.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:30:20.356 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:30:20.357 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:30:20.458 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:30:20.465 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:30:20 smithi191 ceph-mon[32884]: 2.6 scrub starts 2024-09-10T19:30:20.465 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:30:20 smithi191 ceph-mon[32884]: 2.6 scrub ok 2024-09-10T19:30:20.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:30:20 smithi191 ceph-mon[32884]: 4.1 scrub starts 2024-09-10T19:30:20.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:30:20 smithi191 ceph-mon[32884]: 4.1 scrub ok 2024-09-10T19:30:20.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:30:20 smithi191 ceph-mon[32884]: 5.1a scrub starts 2024-09-10T19:30:20.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:30:20 smithi191 ceph-mon[32884]: 5.1a scrub ok 2024-09-10T19:30:20.466 INFO:journalctl@ceph.iscsi.iscsi.a.smithi191.stdout:Sep 10 19:30:20 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-iscsi-iscsi-a[61816]: debug there is no tcmu-runner data available 2024-09-10T19:30:20.492 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:30:20.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:30:20.493 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:30:20.594 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:30:20.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:30:20.630 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:30:20.630 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:30:20.681 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:30:20 smithi183 ceph-mon[29736]: 2.6 scrub starts 2024-09-10T19:30:20.681 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:30:20 smithi183 ceph-mon[29736]: 2.6 scrub ok 2024-09-10T19:30:20.681 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:30:20 smithi183 ceph-mon[29736]: 4.1 scrub starts 2024-09-10T19:30:20.682 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:30:20 smithi183 ceph-mon[29736]: 4.1 scrub ok 2024-09-10T19:30:20.682 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:30:20 smithi183 ceph-mon[29736]: 5.1a scrub starts 2024-09-10T19:30:20.682 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:30:20 smithi183 ceph-mon[29736]: 5.1a scrub ok 2024-09-10T19:30:20.682 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:30:20 smithi183 ceph-mon[41562]: 2.6 scrub starts 2024-09-10T19:30:20.682 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:30:20 smithi183 ceph-mon[41562]: 2.6 scrub ok 2024-09-10T19:30:20.682 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:30:20 smithi183 ceph-mon[41562]: 4.1 scrub starts 2024-09-10T19:30:20.682 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:30:20 smithi183 ceph-mon[41562]: 4.1 scrub ok 2024-09-10T19:30:20.682 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:30:20 smithi183 ceph-mon[41562]: 5.1a scrub starts 2024-09-10T19:30:20.682 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:30:20 smithi183 ceph-mon[41562]: 5.1a scrub ok 2024-09-10T19:30:20.731 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:30:20.763 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:30:20.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:30:20.764 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:30:20.864 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:30:20.900 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:30:20.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:30:20.901 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:30:21.002 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:30:21.026 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:30:21.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:30:21.027 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:30:21.128 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:30:21.158 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:30:21.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:30:21.159 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:30:21.260 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:30:21.298 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:30:21.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:30:21.299 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:30:21.400 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:30:21.430 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:30:21.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:30:21.431 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:30:21.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:30:21 smithi183 ceph-mon[29736]: 5.2 scrub starts 2024-09-10T19:30:21.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:30:21 smithi183 ceph-mon[29736]: 5.2 scrub ok 2024-09-10T19:30:21.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:30:21 smithi183 ceph-mon[29736]: pgmap v933: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 4.2 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 30 KiB/s rd, 0 B/s wr, 1.22k op/s 2024-09-10T19:30:21.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:30:21 smithi183 ceph-mon[29736]: from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:30:21.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:30:21 smithi183 ceph-mon[29736]: 2.16 scrub starts 2024-09-10T19:30:21.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:30:21 smithi183 ceph-mon[29736]: 2.16 scrub ok 2024-09-10T19:30:21.433 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:30:21 smithi183 ceph-mon[41562]: 5.2 scrub starts 2024-09-10T19:30:21.433 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:30:21 smithi183 ceph-mon[41562]: 5.2 scrub ok 2024-09-10T19:30:21.433 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:30:21 smithi183 ceph-mon[41562]: pgmap v933: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 4.2 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 30 KiB/s rd, 0 B/s wr, 1.22k op/s 2024-09-10T19:30:21.433 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:30:21 smithi183 ceph-mon[41562]: from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:30:21.433 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:30:21 smithi183 ceph-mon[41562]: 2.16 scrub starts 2024-09-10T19:30:21.433 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:30:21 smithi183 ceph-mon[41562]: 2.16 scrub ok 2024-09-10T19:30:21.433 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Sep 10 19:30:21 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-y[29989]: ::ffff:172.21.15.191 - - [10/Sep/2024:19:30:21] "GET /metrics HTTP/1.1" 200 224900 "" "Prometheus/2.43.0" 2024-09-10T19:30:21.532 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:30:21.555 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:30:21.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:30:21.556 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:30:21.657 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:30:21.678 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:30:21.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:30:21.679 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:30:21.715 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:30:21 smithi191 ceph-mon[32884]: 5.2 scrub starts 2024-09-10T19:30:21.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:30:21 smithi191 ceph-mon[32884]: 5.2 scrub ok 2024-09-10T19:30:21.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:30:21 smithi191 ceph-mon[32884]: pgmap v933: 158 pgs: 1 active+clean+scrubbing, 157 active+clean; 4.2 GiB data, 18 GiB used, 608 GiB / 626 GiB avail; 30 KiB/s rd, 0 B/s wr, 1.22k op/s 2024-09-10T19:30:21.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:30:21 smithi191 ceph-mon[32884]: from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:30:21.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:30:21 smithi191 ceph-mon[32884]: 2.16 scrub starts 2024-09-10T19:30:21.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:30:21 smithi191 ceph-mon[32884]: 2.16 scrub ok 2024-09-10T19:30:21.779 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:30:21.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:30:21.811 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:30:21.811 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:30:21.912 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:30:21.933 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:30:21.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:30:21.934 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:30:22.034 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:30:22.063 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:30:22.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:30:22.064 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:30:22.165 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:30:22.196 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:30:22.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:30:22.197 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:30:22.298 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:30:22.331 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:30:22.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:30:22.331 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:30:22.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:30:22 smithi183 ceph-mon[29736]: 6.7 scrub starts 2024-09-10T19:30:22.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:30:22 smithi183 ceph-mon[29736]: 6.7 scrub ok 2024-09-10T19:30:22.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:30:22 smithi183 ceph-mon[29736]: 3.2 scrub starts 2024-09-10T19:30:22.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:30:22 smithi183 ceph-mon[29736]: 3.2 scrub ok 2024-09-10T19:30:22.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:30:22 smithi183 ceph-mon[29736]: 6.8 scrub starts 2024-09-10T19:30:22.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:30:22 smithi183 ceph-mon[29736]: 6.8 scrub ok 2024-09-10T19:30:22.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:30:22 smithi183 ceph-mon[41562]: 6.7 scrub starts 2024-09-10T19:30:22.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:30:22 smithi183 ceph-mon[41562]: 6.7 scrub ok 2024-09-10T19:30:22.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:30:22 smithi183 ceph-mon[41562]: 3.2 scrub starts 2024-09-10T19:30:22.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:30:22 smithi183 ceph-mon[41562]: 3.2 scrub ok 2024-09-10T19:30:22.433 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:30:22 smithi183 ceph-mon[41562]: 6.8 scrub starts 2024-09-10T19:30:22.433 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:30:22 smithi183 ceph-mon[41562]: 6.8 scrub ok 2024-09-10T19:30:22.433 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:30:22.463 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:30:22.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:30:22.464 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:30:22.565 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:30:22.603 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:30:22.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:30:22.604 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:30:22.705 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:30:22.715 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:30:22 smithi191 ceph-mon[32884]: 6.7 scrub starts 2024-09-10T19:30:22.715 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:30:22 smithi191 ceph-mon[32884]: 6.7 scrub ok 2024-09-10T19:30:22.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:30:22 smithi191 ceph-mon[32884]: 3.2 scrub starts 2024-09-10T19:30:22.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:30:22 smithi191 ceph-mon[32884]: 3.2 scrub ok 2024-09-10T19:30:22.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:30:22 smithi191 ceph-mon[32884]: 6.8 scrub starts 2024-09-10T19:30:22.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:30:22 smithi191 ceph-mon[32884]: 6.8 scrub ok 2024-09-10T19:30:22.742 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:30:22.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:30:22.743 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:30:22.844 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:30:22.869 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:30:22.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:30:22.870 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:30:22.971 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:30:22.992 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:30:22.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:30:22.993 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:30:23.094 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:30:23.115 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:30:23.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:30:23.117 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:30:23.217 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:30:23.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:30:23.255 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:30:23.255 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:30:23.356 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:30:23.388 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:30:23.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:30:23.390 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:30:23.491 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:30:23.528 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:30:23.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:30:23.529 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:30:23.630 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:30:23.667 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:30:23.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:30:23.667 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:30:23.681 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:30:23 smithi183 ceph-mon[29736]: pgmap v934: 158 pgs: 158 active+clean; 3.9 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 45 KiB/s rd, 0 B/s wr, 1.48k op/s 2024-09-10T19:30:23.681 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:30:23 smithi183 ceph-mon[29736]: 5.16 scrub starts 2024-09-10T19:30:23.681 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:30:23 smithi183 ceph-mon[29736]: 5.16 scrub ok 2024-09-10T19:30:23.682 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:30:23 smithi183 ceph-mon[41562]: pgmap v934: 158 pgs: 158 active+clean; 3.9 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 45 KiB/s rd, 0 B/s wr, 1.48k op/s 2024-09-10T19:30:23.682 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:30:23 smithi183 ceph-mon[41562]: 5.16 scrub starts 2024-09-10T19:30:23.682 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:30:23 smithi183 ceph-mon[41562]: 5.16 scrub ok 2024-09-10T19:30:23.715 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:30:23 smithi191 ceph-mon[32884]: pgmap v934: 158 pgs: 158 active+clean; 3.9 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 45 KiB/s rd, 0 B/s wr, 1.48k op/s 2024-09-10T19:30:23.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:30:23 smithi191 ceph-mon[32884]: 5.16 scrub starts 2024-09-10T19:30:23.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:30:23 smithi191 ceph-mon[32884]: 5.16 scrub ok 2024-09-10T19:30:23.768 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:30:23.801 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:30:23.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:30:23.802 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:30:23.903 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:30:23.939 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:30:23.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:30:23.940 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:30:24.041 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:30:24.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:30:24.074 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:30:24.074 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:30:24.175 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:30:24.213 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:30:24.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:30:24.214 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:30:24.316 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:30:24.340 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:30:24.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:30:24.342 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:30:24.442 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:30:24.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:30:24.479 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:30:24.479 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:30:24.580 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:30:24.611 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:30:24.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:30:24.613 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:30:24.681 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:30:24 smithi183 ceph-mon[29736]: 5.1f scrub starts 2024-09-10T19:30:24.681 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:30:24 smithi183 ceph-mon[29736]: 5.1f scrub ok 2024-09-10T19:30:24.682 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:30:24 smithi183 ceph-mon[29736]: 5.6 scrub starts 2024-09-10T19:30:24.682 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:30:24 smithi183 ceph-mon[29736]: 5.6 scrub ok 2024-09-10T19:30:24.682 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:30:24 smithi183 ceph-mon[29736]: 4.12 scrub starts 2024-09-10T19:30:24.682 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:30:24 smithi183 ceph-mon[29736]: 4.12 scrub ok 2024-09-10T19:30:24.682 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:30:24 smithi183 ceph-mon[29736]: 4.14 scrub starts 2024-09-10T19:30:24.682 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:30:24 smithi183 ceph-mon[29736]: 4.14 scrub ok 2024-09-10T19:30:24.682 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:30:24 smithi183 ceph-mon[41562]: 5.1f scrub starts 2024-09-10T19:30:24.682 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:30:24 smithi183 ceph-mon[41562]: 5.1f scrub ok 2024-09-10T19:30:24.682 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:30:24 smithi183 ceph-mon[41562]: 5.6 scrub starts 2024-09-10T19:30:24.682 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:30:24 smithi183 ceph-mon[41562]: 5.6 scrub ok 2024-09-10T19:30:24.682 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:30:24 smithi183 ceph-mon[41562]: 4.12 scrub starts 2024-09-10T19:30:24.683 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:30:24 smithi183 ceph-mon[41562]: 4.12 scrub ok 2024-09-10T19:30:24.683 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:30:24 smithi183 ceph-mon[41562]: 4.14 scrub starts 2024-09-10T19:30:24.683 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:30:24 smithi183 ceph-mon[41562]: 4.14 scrub ok 2024-09-10T19:30:24.713 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:30:24.715 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:30:24 smithi191 ceph-mon[32884]: 5.1f scrub starts 2024-09-10T19:30:24.715 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:30:24 smithi191 ceph-mon[32884]: 5.1f scrub ok 2024-09-10T19:30:24.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:30:24 smithi191 ceph-mon[32884]: 5.6 scrub starts 2024-09-10T19:30:24.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:30:24 smithi191 ceph-mon[32884]: 5.6 scrub ok 2024-09-10T19:30:24.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:30:24 smithi191 ceph-mon[32884]: 4.12 scrub starts 2024-09-10T19:30:24.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:30:24 smithi191 ceph-mon[32884]: 4.12 scrub ok 2024-09-10T19:30:24.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:30:24 smithi191 ceph-mon[32884]: 4.14 scrub starts 2024-09-10T19:30:24.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:30:24 smithi191 ceph-mon[32884]: 4.14 scrub ok 2024-09-10T19:30:24.748 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:30:24.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:30:24.750 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:30:24.850 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:30:24.879 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:30:24.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:30:24.880 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:30:24.981 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:30:25.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:30:25.015 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:30:25.015 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:30:25.115 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:30:25.143 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:30:25.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:30:25.145 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:30:25.245 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:30:25.266 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:30:25.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:30:25.267 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:30:25.367 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:30:25.400 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:30:25.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:30:25.401 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:30:25.502 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:30:25.555 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:30:25.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:30:25.556 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:30:25.657 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:30:25.681 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:30:25 smithi183 ceph-mon[29736]: 1.6 scrub starts 2024-09-10T19:30:25.681 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:30:25 smithi183 ceph-mon[29736]: 1.6 scrub ok 2024-09-10T19:30:25.681 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:30:25 smithi183 ceph-mon[29736]: pgmap v935: 158 pgs: 158 active+clean; 3.9 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 44 KiB/s rd, 0 B/s wr, 1.47k op/s 2024-09-10T19:30:25.682 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:30:25 smithi183 ceph-mon[29736]: 4.18 scrub starts 2024-09-10T19:30:25.682 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:30:25 smithi183 ceph-mon[29736]: 4.18 scrub ok 2024-09-10T19:30:25.682 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:30:25 smithi183 ceph-mon[29736]: 2.1b scrub starts 2024-09-10T19:30:25.682 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:30:25 smithi183 ceph-mon[29736]: 2.1b scrub ok 2024-09-10T19:30:25.682 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:30:25 smithi183 ceph-mon[29736]: 5.1e scrub starts 2024-09-10T19:30:25.682 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:30:25 smithi183 ceph-mon[29736]: 5.1e scrub ok 2024-09-10T19:30:25.682 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:30:25 smithi183 ceph-mon[41562]: 1.6 scrub starts 2024-09-10T19:30:25.682 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:30:25 smithi183 ceph-mon[41562]: 1.6 scrub ok 2024-09-10T19:30:25.682 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:30:25 smithi183 ceph-mon[41562]: pgmap v935: 158 pgs: 158 active+clean; 3.9 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 44 KiB/s rd, 0 B/s wr, 1.47k op/s 2024-09-10T19:30:25.682 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:30:25 smithi183 ceph-mon[41562]: 4.18 scrub starts 2024-09-10T19:30:25.682 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:30:25 smithi183 ceph-mon[41562]: 4.18 scrub ok 2024-09-10T19:30:25.683 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:30:25 smithi183 ceph-mon[41562]: 2.1b scrub starts 2024-09-10T19:30:25.683 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:30:25 smithi183 ceph-mon[41562]: 2.1b scrub ok 2024-09-10T19:30:25.683 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:30:25 smithi183 ceph-mon[41562]: 5.1e scrub starts 2024-09-10T19:30:25.683 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:30:25 smithi183 ceph-mon[41562]: 5.1e scrub ok 2024-09-10T19:30:25.692 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:30:25.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:30:25.693 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:30:25.715 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:30:25 smithi191 ceph-mon[32884]: 1.6 scrub starts 2024-09-10T19:30:25.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:30:25 smithi191 ceph-mon[32884]: 1.6 scrub ok 2024-09-10T19:30:25.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:30:25 smithi191 ceph-mon[32884]: pgmap v935: 158 pgs: 158 active+clean; 3.9 GiB data, 17 GiB used, 609 GiB / 626 GiB avail; 44 KiB/s rd, 0 B/s wr, 1.47k op/s 2024-09-10T19:30:25.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:30:25 smithi191 ceph-mon[32884]: 4.18 scrub starts 2024-09-10T19:30:25.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:30:25 smithi191 ceph-mon[32884]: 4.18 scrub ok 2024-09-10T19:30:25.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:30:25 smithi191 ceph-mon[32884]: 2.1b scrub starts 2024-09-10T19:30:25.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:30:25 smithi191 ceph-mon[32884]: 2.1b scrub ok 2024-09-10T19:30:25.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:30:25 smithi191 ceph-mon[32884]: 5.1e scrub starts 2024-09-10T19:30:25.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:30:25 smithi191 ceph-mon[32884]: 5.1e scrub ok 2024-09-10T19:30:25.794 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:30:25.817 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:30:25.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:30:25.817 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:30:25.918 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:30:25.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:30:25.954 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:30:25.955 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:30:26.055 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:30:26.094 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:30:26.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:30:26.095 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:30:26.196 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:30:26.230 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:30:26.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:30:26.231 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:30:26.332 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:30:26.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:30:26.364 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:30:26.365 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:30:26.431 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:30:26 smithi183 ceph-mon[41562]: 2.3 scrub starts 2024-09-10T19:30:26.431 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:30:26 smithi183 ceph-mon[41562]: 2.3 scrub ok 2024-09-10T19:30:26.431 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:30:26 smithi183 ceph-mon[41562]: 6.1d scrub starts 2024-09-10T19:30:26.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:30:26 smithi183 ceph-mon[41562]: 6.1d scrub ok 2024-09-10T19:30:26.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:30:26 smithi183 ceph-mon[41562]: 2.e scrub starts 2024-09-10T19:30:26.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:30:26 smithi183 ceph-mon[41562]: 2.e scrub ok 2024-09-10T19:30:26.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:30:26 smithi183 ceph-mon[29736]: 2.3 scrub starts 2024-09-10T19:30:26.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:30:26 smithi183 ceph-mon[29736]: 2.3 scrub ok 2024-09-10T19:30:26.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:30:26 smithi183 ceph-mon[29736]: 6.1d scrub starts 2024-09-10T19:30:26.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:30:26 smithi183 ceph-mon[29736]: 6.1d scrub ok 2024-09-10T19:30:26.433 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:30:26 smithi183 ceph-mon[29736]: 2.e scrub starts 2024-09-10T19:30:26.433 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:30:26 smithi183 ceph-mon[29736]: 2.e scrub ok 2024-09-10T19:30:26.465 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:30:26.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:30:26.502 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:30:26.502 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:30:26.603 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:30:26.635 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:30:26.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:30:26.637 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:30:26.715 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:30:26 smithi191 ceph-mon[32884]: 2.3 scrub starts 2024-09-10T19:30:26.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:30:26 smithi191 ceph-mon[32884]: 2.3 scrub ok 2024-09-10T19:30:26.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:30:26 smithi191 ceph-mon[32884]: 6.1d scrub starts 2024-09-10T19:30:26.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:30:26 smithi191 ceph-mon[32884]: 6.1d scrub ok 2024-09-10T19:30:26.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:30:26 smithi191 ceph-mon[32884]: 2.e scrub starts 2024-09-10T19:30:26.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:30:26 smithi191 ceph-mon[32884]: 2.e scrub ok 2024-09-10T19:30:26.737 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:30:26.770 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:30:26.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:30:26.771 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:30:26.872 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:30:26.915 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:30:26.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:30:26.916 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:30:27.017 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:30:27.054 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:30:27.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:30:27.055 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:30:27.156 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:30:27.192 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:30:27.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:30:27.193 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:30:27.294 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:30:27.314 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:30:27.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:30:27.315 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:30:27.416 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:30:27.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:30:27.452 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:30:27.453 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:30:27.541 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-10T19:30:27.552 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:30:27 smithi183 ceph-mon[29736]: pgmap v936: 158 pgs: 158 active+clean; 3.6 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 41 KiB/s rd, 0 B/s wr, 1.37k op/s 2024-09-10T19:30:27.552 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:30:27 smithi183 ceph-mon[29736]: 2.2 scrub starts 2024-09-10T19:30:27.552 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:30:27 smithi183 ceph-mon[29736]: 2.2 scrub ok 2024-09-10T19:30:27.552 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:30:27 smithi183 ceph-mon[29736]: 4.d scrub starts 2024-09-10T19:30:27.552 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:30:27 smithi183 ceph-mon[29736]: 4.d scrub ok 2024-09-10T19:30:27.552 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:30:27 smithi183 ceph-mon[41562]: pgmap v936: 158 pgs: 158 active+clean; 3.6 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 41 KiB/s rd, 0 B/s wr, 1.37k op/s 2024-09-10T19:30:27.553 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:30:27 smithi183 ceph-mon[41562]: 2.2 scrub starts 2024-09-10T19:30:27.553 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:30:27 smithi183 ceph-mon[41562]: 2.2 scrub ok 2024-09-10T19:30:27.553 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:30:27 smithi183 ceph-mon[41562]: 4.d scrub starts 2024-09-10T19:30:27.553 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:30:27 smithi183 ceph-mon[41562]: 4.d scrub ok 2024-09-10T19:30:27.553 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:30:27.592 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:30:27.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:30:27.593 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:30:27.617 INFO:journalctl@ceph.mgr.x.smithi191.stdout:Sep 10 19:30:27 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-x[34967]: ::ffff:172.21.15.191 - - [10/Sep/2024:19:30:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T19:30:27.617 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:30:27 smithi191 ceph-mon[32884]: pgmap v936: 158 pgs: 158 active+clean; 3.6 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 41 KiB/s rd, 0 B/s wr, 1.37k op/s 2024-09-10T19:30:27.617 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:30:27 smithi191 ceph-mon[32884]: 2.2 scrub starts 2024-09-10T19:30:27.617 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:30:27 smithi191 ceph-mon[32884]: 2.2 scrub ok 2024-09-10T19:30:27.617 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:30:27 smithi191 ceph-mon[32884]: 4.d scrub starts 2024-09-10T19:30:27.617 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:30:27 smithi191 ceph-mon[32884]: 4.d scrub ok 2024-09-10T19:30:27.694 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:30:27.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:30:27.730 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:30:27.730 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:30:27.831 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:30:27.862 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:30:27.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:30:27.863 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:30:27.964 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:30:28.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:30:28.002 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:30:28.002 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:30:28.042 INFO:teuthology.orchestra.run.smithi183.stdout:{ 2024-09-10T19:30:28.042 INFO:teuthology.orchestra.run.smithi183.stdout: "size": 20, 2024-09-10T19:30:28.042 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 600, 2024-09-10T19:30:28.042 INFO:teuthology.orchestra.run.smithi183.stdout: "ops": [ 2024-09-10T19:30:28.042 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:30:28.043 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.25622.0:244547 10.f 10:f9fe1760:::benchmark_data_smithi183_176537_object121339:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e365)", 2024-09-10T19:30:28.043 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-10T19:27:26.756116+0000", 2024-09-10T19:30:28.043 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 180.70170256399999, 2024-09-10T19:30:28.043 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.25318963100000003, 2024-09-10T19:30:28.043 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-10T19:30:28.043 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T19:30:28.043 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-10T19:30:28.043 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.25622", 2024-09-10T19:30:28.043 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.183:0/3685189199", 2024-09-10T19:30:28.043 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 244547 2024-09-10T19:30:28.043 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:30:28.043 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-10T19:30:28.043 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:30:28.044 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-10T19:30:28.044 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:27:26.756116+0000", 2024-09-10T19:30:28.044 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:30:28.044 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:30:28.044 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:30:28.044 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-10T19:30:28.044 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:27:26.756116+0000", 2024-09-10T19:30:28.044 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:30:28.044 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:30:28.044 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:30:28.044 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-10T19:30:28.044 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:27:26.756115+0000", 2024-09-10T19:30:28.044 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4294967295.9999995 2024-09-10T19:30:28.045 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:30:28.045 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:30:28.045 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-10T19:30:28.045 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:27:26.756119+0000", 2024-09-10T19:30:28.045 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.1309999999999999e-06 2024-09-10T19:30:28.045 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:30:28.045 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:30:28.045 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-10T19:30:28.045 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:27:26.756128+0000", 2024-09-10T19:30:28.045 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.1090000000000004e-06 2024-09-10T19:30:28.045 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:30:28.045 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:30:28.046 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-10T19:30:28.046 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:27:26.756136+0000", 2024-09-10T19:30:28.046 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.9759999999999995e-06 2024-09-10T19:30:28.046 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:30:28.046 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:30:28.046 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-10T19:30:28.046 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:27:26.757961+0000", 2024-09-10T19:30:28.046 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0018246720000000001 2024-09-10T19:30:28.046 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:30:28.047 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:30:28.047 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-10T19:30:28.047 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:27:26.758077+0000", 2024-09-10T19:30:28.047 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000116078 2024-09-10T19:30:28.047 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:30:28.047 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:30:28.047 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 1,4,6", 2024-09-10T19:30:28.047 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:27:26.758204+0000", 2024-09-10T19:30:28.047 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00012736799999999999 2024-09-10T19:30:28.047 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:30:28.047 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:30:28.047 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-10T19:30:28.047 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:27:26.759857+0000", 2024-09-10T19:30:28.048 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0016528490000000001 2024-09-10T19:30:28.048 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:30:28.048 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:30:28.048 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:30:28.048 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:27:26.762715+0000", 2024-09-10T19:30:28.048 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0028582529999999998 2024-09-10T19:30:28.048 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:30:28.048 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:30:28.048 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:30:28.048 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:27:26.775548+0000", 2024-09-10T19:30:28.048 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.012832417 2024-09-10T19:30:28.048 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:30:28.048 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:30:28.049 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:30:28.049 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:27:27.009134+0000", 2024-09-10T19:30:28.049 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.23358630699999999 2024-09-10T19:30:28.049 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:30:28.049 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:30:28.049 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-10T19:30:28.049 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:27:27.009189+0000", 2024-09-10T19:30:28.049 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.5278000000000002e-05 2024-09-10T19:30:28.049 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:30:28.049 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:30:28.049 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-10T19:30:28.049 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:27:27.009305+0000", 2024-09-10T19:30:28.049 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000115878 2024-09-10T19:30:28.049 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:30:28.050 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:30:28.050 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:30:28.050 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:30:28.050 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:30:28.050 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.16026.0:109548 11.9 11:9b7e1a0e:::benchmark_data_smithi183_203944_object109547:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e398)", 2024-09-10T19:30:28.050 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-10T19:28:57.729937+0000", 2024-09-10T19:30:28.050 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 89.727881160999999, 2024-09-10T19:30:28.050 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.57268561699999998, 2024-09-10T19:30:28.050 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-10T19:30:28.050 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T19:30:28.050 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-10T19:30:28.050 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.16026", 2024-09-10T19:30:28.050 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.183:0/1450567484", 2024-09-10T19:30:28.050 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 109548 2024-09-10T19:30:28.051 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:30:28.051 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-10T19:30:28.051 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:30:28.051 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-10T19:30:28.051 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:28:57.729937+0000", 2024-09-10T19:30:28.051 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:30:28.051 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:30:28.051 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:30:28.051 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-10T19:30:28.051 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:28:57.729937+0000", 2024-09-10T19:30:28.051 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:30:28.051 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:30:28.051 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:30:28.051 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-10T19:30:28.051 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:28:57.729936+0000", 2024-09-10T19:30:28.051 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4294967295.9999986 2024-09-10T19:30:28.052 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:30:28.052 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:30:28.052 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-10T19:30:28.052 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:28:57.729957+0000", 2024-09-10T19:30:28.052 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.0908999999999999e-05 2024-09-10T19:30:28.052 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:30:28.052 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:30:28.052 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-10T19:30:28.052 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:28:57.729968+0000", 2024-09-10T19:30:28.052 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.0934999999999999e-05 2024-09-10T19:30:28.052 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:30:28.052 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:30:28.052 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-10T19:30:28.052 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:28:57.729972+0000", 2024-09-10T19:30:28.052 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.3710000000000002e-06 2024-09-10T19:30:28.052 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:30:28.053 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:30:28.053 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-10T19:30:28.053 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:28:57.730015+0000", 2024-09-10T19:30:28.053 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.2622e-05 2024-09-10T19:30:28.053 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:30:28.053 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:30:28.053 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-10T19:30:28.053 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:28:57.730104+0000", 2024-09-10T19:30:28.053 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.9147999999999994e-05 2024-09-10T19:30:28.053 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:30:28.053 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:30:28.053 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 0,6", 2024-09-10T19:30:28.053 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:28:57.730197+0000", 2024-09-10T19:30:28.053 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.3484000000000001e-05 2024-09-10T19:30:28.054 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:30:28.054 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:30:28.054 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-10T19:30:28.054 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:28:57.732011+0000", 2024-09-10T19:30:28.054 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.001814136 2024-09-10T19:30:28.054 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:30:28.054 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:30:28.054 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:30:28.054 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:28:57.732179+0000", 2024-09-10T19:30:28.054 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000167353 2024-09-10T19:30:28.054 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:30:28.054 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:30:28.054 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:30:28.054 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:28:58.302518+0000", 2024-09-10T19:30:28.055 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.57033974700000001 2024-09-10T19:30:28.055 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:30:28.055 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:30:28.055 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-10T19:30:28.055 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:28:58.302579+0000", 2024-09-10T19:30:28.055 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.0390000000000003e-05 2024-09-10T19:30:28.055 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:30:28.055 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:30:28.055 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-10T19:30:28.055 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:28:58.302623+0000", 2024-09-10T19:30:28.055 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.3982e-05 2024-09-10T19:30:28.055 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:30:28.055 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:30:28.055 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:30:28.056 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:30:28.056 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:30:28.056 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.16026.0:109568 11.9 11:91957b42:::benchmark_data_smithi183_203944_object109567:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e398)", 2024-09-10T19:30:28.056 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-10T19:28:57.812016+0000", 2024-09-10T19:30:28.056 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 89.645802001000007, 2024-09-10T19:30:28.056 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.490735015, 2024-09-10T19:30:28.056 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-10T19:30:28.056 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T19:30:28.056 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-10T19:30:28.056 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.16026", 2024-09-10T19:30:28.056 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.183:0/1450567484", 2024-09-10T19:30:28.056 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 109568 2024-09-10T19:30:28.056 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:30:28.057 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-10T19:30:28.057 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:30:28.057 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-10T19:30:28.057 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:28:57.812016+0000", 2024-09-10T19:30:28.057 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:30:28.057 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:30:28.057 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:30:28.057 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-10T19:30:28.057 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:28:57.812016+0000", 2024-09-10T19:30:28.057 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:30:28.057 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:30:28.058 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:30:28.058 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-10T19:30:28.058 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:28:57.812015+0000", 2024-09-10T19:30:28.058 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4294967295.9999986 2024-09-10T19:30:28.058 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:30:28.058 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:30:28.058 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-10T19:30:28.058 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:28:57.812038+0000", 2024-09-10T19:30:28.058 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.3166000000000001e-05 2024-09-10T19:30:28.058 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:30:28.058 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:30:28.058 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-10T19:30:28.058 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:28:57.812052+0000", 2024-09-10T19:30:28.059 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.3514e-05 2024-09-10T19:30:28.059 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:30:28.059 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:30:28.059 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-10T19:30:28.059 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:28:57.812057+0000", 2024-09-10T19:30:28.059 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.558e-06 2024-09-10T19:30:28.059 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:30:28.059 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:30:28.059 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-10T19:30:28.059 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:28:57.813367+0000", 2024-09-10T19:30:28.059 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0013095229999999999 2024-09-10T19:30:28.059 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:30:28.059 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:30:28.059 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-10T19:30:28.060 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:28:57.813498+0000", 2024-09-10T19:30:28.060 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00013126300000000001 2024-09-10T19:30:28.060 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:30:28.060 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:30:28.060 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 0,6", 2024-09-10T19:30:28.060 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:28:57.813626+0000", 2024-09-10T19:30:28.060 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00012840200000000001 2024-09-10T19:30:28.060 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:30:28.060 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:30:28.060 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-10T19:30:28.060 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:28:57.816512+0000", 2024-09-10T19:30:28.060 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0028858989999999999 2024-09-10T19:30:28.061 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:30:28.061 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:30:28.061 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:30:28.061 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:28:57.819984+0000", 2024-09-10T19:30:28.061 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0034715190000000002 2024-09-10T19:30:28.061 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:30:28.061 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:30:28.061 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:30:28.061 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:28:58.302669+0000", 2024-09-10T19:30:28.061 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.48268537099999997 2024-09-10T19:30:28.061 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:30:28.061 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:30:28.062 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-10T19:30:28.062 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:28:58.302715+0000", 2024-09-10T19:30:28.062 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.5306000000000003e-05 2024-09-10T19:30:28.062 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:30:28.062 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:30:28.062 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-10T19:30:28.062 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:28:58.302751+0000", 2024-09-10T19:30:28.062 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.6783999999999999e-05 2024-09-10T19:30:28.062 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:30:28.062 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:30:28.062 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:30:28.062 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:30:28.062 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:30:28.062 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.16026.0:109574 11.8 11:1a081587:::benchmark_data_smithi183_203944_object109573:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e398)", 2024-09-10T19:30:28.063 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-10T19:28:57.817898+0000", 2024-09-10T19:30:28.063 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 89.639920270999994, 2024-09-10T19:30:28.063 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.48492027799999998, 2024-09-10T19:30:28.063 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-10T19:30:28.063 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T19:30:28.063 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-10T19:30:28.063 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.16026", 2024-09-10T19:30:28.063 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.183:0/1450567484", 2024-09-10T19:30:28.063 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 109574 2024-09-10T19:30:28.063 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:30:28.063 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-10T19:30:28.063 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:30:28.063 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-10T19:30:28.063 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:28:57.817898+0000", 2024-09-10T19:30:28.064 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:30:28.064 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:30:28.064 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:30:28.064 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-10T19:30:28.064 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:28:57.817898+0000", 2024-09-10T19:30:28.064 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:30:28.064 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:30:28.064 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:30:28.064 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-10T19:30:28.064 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:28:57.817897+0000", 2024-09-10T19:30:28.064 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4294967295.999999 2024-09-10T19:30:28.064 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:30:28.065 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:30:28.065 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-10T19:30:28.065 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:28:57.818184+0000", 2024-09-10T19:30:28.065 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000287242 2024-09-10T19:30:28.065 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:30:28.065 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:30:28.065 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-10T19:30:28.065 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:28:57.818201+0000", 2024-09-10T19:30:28.065 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.6356000000000001e-05 2024-09-10T19:30:28.065 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:30:28.065 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:30:28.065 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-10T19:30:28.065 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:28:57.818206+0000", 2024-09-10T19:30:28.065 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.7690000000000002e-06 2024-09-10T19:30:28.066 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:30:28.066 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:30:28.066 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-10T19:30:28.066 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:28:57.818672+0000", 2024-09-10T19:30:28.066 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00046584699999999997 2024-09-10T19:30:28.066 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:30:28.066 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:30:28.066 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-10T19:30:28.066 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:28:57.818783+0000", 2024-09-10T19:30:28.066 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000110969 2024-09-10T19:30:28.066 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:30:28.066 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:30:28.066 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 5,6", 2024-09-10T19:30:28.066 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:28:57.818915+0000", 2024-09-10T19:30:28.066 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000131397 2024-09-10T19:30:28.067 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:30:28.067 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:30:28.067 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-10T19:30:28.067 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:28:57.821529+0000", 2024-09-10T19:30:28.067 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0026148909999999998 2024-09-10T19:30:28.067 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:30:28.067 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:30:28.067 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:30:28.067 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:28:57.825443+0000", 2024-09-10T19:30:28.067 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.003913761 2024-09-10T19:30:28.067 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:30:28.067 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:30:28.067 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:30:28.067 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:28:58.302597+0000", 2024-09-10T19:30:28.068 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.47715353700000002 2024-09-10T19:30:28.068 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:30:28.068 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:30:28.068 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-10T19:30:28.068 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:28:58.302636+0000", 2024-09-10T19:30:28.068 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.8795999999999999e-05 2024-09-10T19:30:28.068 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:30:28.068 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:30:28.068 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-10T19:30:28.068 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:28:58.302818+0000", 2024-09-10T19:30:28.068 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00018279599999999999 2024-09-10T19:30:28.068 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:30:28.068 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:30:28.068 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:30:28.069 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:30:28.069 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:30:28.069 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.16026.0:109586 11.8 11:187f4a71:::benchmark_data_smithi183_203944_object109585:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e398)", 2024-09-10T19:30:28.069 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-10T19:28:57.825987+0000", 2024-09-10T19:30:28.069 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 89.631831453999993, 2024-09-10T19:30:28.069 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.48428602500000001, 2024-09-10T19:30:28.069 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-10T19:30:28.069 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T19:30:28.069 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-10T19:30:28.069 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.16026", 2024-09-10T19:30:28.069 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.183:0/1450567484", 2024-09-10T19:30:28.069 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 109586 2024-09-10T19:30:28.069 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:30:28.069 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-10T19:30:28.070 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:30:28.070 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-10T19:30:28.070 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:28:57.825987+0000", 2024-09-10T19:30:28.070 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:30:28.070 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:30:28.070 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:30:28.070 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-10T19:30:28.070 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:28:57.825987+0000", 2024-09-10T19:30:28.070 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:30:28.070 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:30:28.070 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:30:28.070 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-10T19:30:28.070 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:28:57.825986+0000", 2024-09-10T19:30:28.070 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4294967295.9999995 2024-09-10T19:30:28.071 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:30:28.071 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:30:28.071 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-10T19:30:28.071 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:28:57.826005+0000", 2024-09-10T19:30:28.071 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.8329999999999999e-05 2024-09-10T19:30:28.071 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:30:28.071 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:30:28.071 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-10T19:30:28.071 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:28:57.826011+0000", 2024-09-10T19:30:28.071 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.8810000000000001e-06 2024-09-10T19:30:28.071 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:30:28.071 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:30:28.071 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-10T19:30:28.071 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:28:57.826013+0000", 2024-09-10T19:30:28.072 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.7709999999999998e-06 2024-09-10T19:30:28.072 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:30:28.072 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:30:28.072 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-10T19:30:28.072 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:28:57.826050+0000", 2024-09-10T19:30:28.072 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.6208000000000003e-05 2024-09-10T19:30:28.072 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:30:28.072 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:30:28.072 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-10T19:30:28.072 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:28:57.826182+0000", 2024-09-10T19:30:28.072 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000131977 2024-09-10T19:30:28.072 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:30:28.072 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:30:28.072 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 5,6", 2024-09-10T19:30:28.073 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:28:57.826317+0000", 2024-09-10T19:30:28.073 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00013548100000000001 2024-09-10T19:30:28.073 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:30:28.073 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:30:28.073 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:30:28.073 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:28:57.830138+0000", 2024-09-10T19:30:28.073 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.003820468 2024-09-10T19:30:28.073 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:30:28.073 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:30:28.073 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-10T19:30:28.073 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:28:57.830173+0000", 2024-09-10T19:30:28.073 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.5802e-05 2024-09-10T19:30:28.073 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:30:28.074 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:30:28.074 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:30:28.074 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:28:58.310120+0000", 2024-09-10T19:30:28.074 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.47994628900000003 2024-09-10T19:30:28.074 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:30:28.074 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:30:28.074 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-10T19:30:28.074 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:28:58.310230+0000", 2024-09-10T19:30:28.074 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000110812 2024-09-10T19:30:28.074 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:30:28.074 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:30:28.074 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-10T19:30:28.074 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:28:58.310273+0000", 2024-09-10T19:30:28.074 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.2419999999999997e-05 2024-09-10T19:30:28.075 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:30:28.075 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:30:28.075 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:30:28.075 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:30:28.075 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:30:28.075 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.16026.0:109588 11.a 11:5eb71bf9:::benchmark_data_smithi183_203944_object109587:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e398)", 2024-09-10T19:30:28.075 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-10T19:28:57.829373+0000", 2024-09-10T19:30:28.075 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 89.628444962000003, 2024-09-10T19:30:28.075 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.48298683799999997, 2024-09-10T19:30:28.075 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-10T19:30:28.075 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T19:30:28.075 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-10T19:30:28.075 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.16026", 2024-09-10T19:30:28.075 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.183:0/1450567484", 2024-09-10T19:30:28.075 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 109588 2024-09-10T19:30:28.076 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:30:28.076 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-10T19:30:28.076 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:30:28.076 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-10T19:30:28.076 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:28:57.829373+0000", 2024-09-10T19:30:28.076 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:30:28.076 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:30:28.076 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:30:28.076 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-10T19:30:28.076 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:28:57.829373+0000", 2024-09-10T19:30:28.076 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:30:28.076 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:30:28.076 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:30:28.077 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-10T19:30:28.077 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:28:57.829372+0000", 2024-09-10T19:30:28.077 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4294967295.9999986 2024-09-10T19:30:28.077 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:30:28.077 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:30:28.077 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-10T19:30:28.077 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:28:57.829398+0000", 2024-09-10T19:30:28.077 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.6191e-05 2024-09-10T19:30:28.077 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:30:28.077 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:30:28.077 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-10T19:30:28.077 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:28:57.829414+0000", 2024-09-10T19:30:28.077 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.5716e-05 2024-09-10T19:30:28.078 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:30:28.078 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:30:28.078 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-10T19:30:28.078 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:28:57.829420+0000", 2024-09-10T19:30:28.078 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.1550000000000004e-06 2024-09-10T19:30:28.078 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:30:28.078 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:30:28.078 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-10T19:30:28.078 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:28:57.829467+0000", 2024-09-10T19:30:28.078 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.7256000000000003e-05 2024-09-10T19:30:28.078 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:30:28.078 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:30:28.078 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-10T19:30:28.079 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:28:57.829597+0000", 2024-09-10T19:30:28.079 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00012990399999999999 2024-09-10T19:30:28.079 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:30:28.079 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:30:28.079 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 5,6", 2024-09-10T19:30:28.079 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:28:57.829787+0000", 2024-09-10T19:30:28.079 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000189808 2024-09-10T19:30:28.079 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:30:28.079 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:30:28.079 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:30:28.079 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:28:57.834387+0000", 2024-09-10T19:30:28.079 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0046003870000000001 2024-09-10T19:30:28.079 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:30:28.080 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:30:28.080 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-10T19:30:28.080 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:28:57.834403+0000", 2024-09-10T19:30:28.080 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.6147999999999999e-05 2024-09-10T19:30:28.080 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:30:28.080 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:30:28.080 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:30:28.080 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:28:58.312211+0000", 2024-09-10T19:30:28.080 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.47780803199999999 2024-09-10T19:30:28.080 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:30:28.080 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:30:28.080 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-10T19:30:28.080 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:28:58.312320+0000", 2024-09-10T19:30:28.081 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00010822199999999999 2024-09-10T19:30:28.081 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:30:28.081 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:30:28.081 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-10T19:30:28.081 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:28:58.312360+0000", 2024-09-10T19:30:28.081 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.0568000000000002e-05 2024-09-10T19:30:28.081 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:30:28.081 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:30:28.081 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:30:28.081 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:30:28.081 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:30:28.081 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.16026.0:109590 11.9 11:9e973a10:::benchmark_data_smithi183_203944_object109589:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e398)", 2024-09-10T19:30:28.081 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-10T19:28:57.829738+0000", 2024-09-10T19:30:28.082 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 89.628080569000005, 2024-09-10T19:30:28.082 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.475843662, 2024-09-10T19:30:28.082 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-10T19:30:28.082 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T19:30:28.082 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-10T19:30:28.082 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.16026", 2024-09-10T19:30:28.082 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.183:0/1450567484", 2024-09-10T19:30:28.082 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 109590 2024-09-10T19:30:28.082 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:30:28.082 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-10T19:30:28.082 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:30:28.082 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-10T19:30:28.082 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:28:57.829738+0000", 2024-09-10T19:30:28.082 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:30:28.083 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:30:28.083 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:30:28.083 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-10T19:30:28.083 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:28:57.829738+0000", 2024-09-10T19:30:28.083 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:30:28.083 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:30:28.083 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:30:28.083 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-10T19:30:28.083 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:28:57.829737+0000", 2024-09-10T19:30:28.083 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4294967295.999999 2024-09-10T19:30:28.083 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:30:28.083 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:30:28.083 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-10T19:30:28.083 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:28:57.829796+0000", 2024-09-10T19:30:28.083 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.8872000000000001e-05 2024-09-10T19:30:28.084 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:30:28.084 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:30:28.084 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-10T19:30:28.084 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:28:57.829808+0000", 2024-09-10T19:30:28.084 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.2531000000000001e-05 2024-09-10T19:30:28.084 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:30:28.084 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:30:28.084 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-10T19:30:28.084 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:28:57.829815+0000", 2024-09-10T19:30:28.084 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.0099999999999998e-06 2024-09-10T19:30:28.084 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:30:28.084 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:30:28.084 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-10T19:30:28.085 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:28:57.829857+0000", 2024-09-10T19:30:28.085 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.1656999999999998e-05 2024-09-10T19:30:28.085 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:30:28.085 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:30:28.085 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-10T19:30:28.085 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:28:57.829988+0000", 2024-09-10T19:30:28.085 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00013115199999999999 2024-09-10T19:30:28.085 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:30:28.085 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:30:28.085 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 0,6", 2024-09-10T19:30:28.085 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:28:57.830121+0000", 2024-09-10T19:30:28.085 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00013268700000000001 2024-09-10T19:30:28.085 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:30:28.085 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:30:28.086 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-10T19:30:28.086 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:28:57.832523+0000", 2024-09-10T19:30:28.086 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0024022959999999999 2024-09-10T19:30:28.086 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:30:28.086 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:30:28.086 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:30:28.086 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:28:57.832841+0000", 2024-09-10T19:30:28.086 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00031806599999999999 2024-09-10T19:30:28.086 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:30:28.086 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:30:28.086 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:30:28.086 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:28:58.305481+0000", 2024-09-10T19:30:28.086 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.47264019899999998 2024-09-10T19:30:28.087 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:30:28.087 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:30:28.087 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-10T19:30:28.087 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:28:58.305543+0000", 2024-09-10T19:30:28.087 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.1641000000000002e-05 2024-09-10T19:30:28.087 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:30:28.087 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:30:28.087 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-10T19:30:28.087 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:28:58.305581+0000", 2024-09-10T19:30:28.087 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.8503e-05 2024-09-10T19:30:28.087 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:30:28.087 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:30:28.087 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:30:28.088 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:30:28.088 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:30:28.088 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.16026.0:109591 11.a 11:5dbffd77:::benchmark_data_smithi183_203944_object109590:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e398)", 2024-09-10T19:30:28.088 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-10T19:28:57.830108+0000", 2024-09-10T19:30:28.088 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 89.627710702000002, 2024-09-10T19:30:28.088 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.48242876699999998, 2024-09-10T19:30:28.088 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-10T19:30:28.088 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T19:30:28.088 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-10T19:30:28.088 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.16026", 2024-09-10T19:30:28.088 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.183:0/1450567484", 2024-09-10T19:30:28.088 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 109591 2024-09-10T19:30:28.089 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:30:28.089 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-10T19:30:28.089 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:30:28.089 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-10T19:30:28.089 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:28:57.830108+0000", 2024-09-10T19:30:28.089 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:30:28.089 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:30:28.089 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:30:28.089 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-10T19:30:28.089 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:28:57.830108+0000", 2024-09-10T19:30:28.089 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:30:28.089 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:30:28.089 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:30:28.089 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-10T19:30:28.090 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:28:57.830107+0000", 2024-09-10T19:30:28.090 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4294967295.999999 2024-09-10T19:30:28.090 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:30:28.090 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:30:28.090 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-10T19:30:28.090 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:28:57.830129+0000", 2024-09-10T19:30:28.090 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.2412000000000002e-05 2024-09-10T19:30:28.090 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:30:28.090 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:30:28.090 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-10T19:30:28.090 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:28:57.830140+0000", 2024-09-10T19:30:28.090 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.1056999999999999e-05 2024-09-10T19:30:28.090 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:30:28.090 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:30:28.091 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-10T19:30:28.091 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:28:57.830146+0000", 2024-09-10T19:30:28.091 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.4e-06 2024-09-10T19:30:28.091 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:30:28.091 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:30:28.091 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-10T19:30:28.091 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:28:57.832057+0000", 2024-09-10T19:30:28.091 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.001911339 2024-09-10T19:30:28.091 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:30:28.091 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:30:28.091 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-10T19:30:28.091 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:28:57.832166+0000", 2024-09-10T19:30:28.091 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00010852400000000001 2024-09-10T19:30:28.091 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:30:28.092 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:30:28.092 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 5,6", 2024-09-10T19:30:28.092 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:28:57.832307+0000", 2024-09-10T19:30:28.092 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00014175200000000001 2024-09-10T19:30:28.092 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:30:28.092 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:30:28.092 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-10T19:30:28.092 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:28:57.834533+0000", 2024-09-10T19:30:28.092 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.002225318 2024-09-10T19:30:28.092 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:30:28.092 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:30:28.092 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:30:28.092 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:28:57.834660+0000", 2024-09-10T19:30:28.093 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00012756399999999999 2024-09-10T19:30:28.093 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:30:28.093 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:30:28.093 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:30:28.093 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:28:58.312391+0000", 2024-09-10T19:30:28.093 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.47773068299999999 2024-09-10T19:30:28.093 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:30:28.093 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:30:28.093 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-10T19:30:28.093 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:28:58.312503+0000", 2024-09-10T19:30:28.093 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00011225700000000001 2024-09-10T19:30:28.093 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:30:28.093 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:30:28.093 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-10T19:30:28.094 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:28:58.312536+0000", 2024-09-10T19:30:28.094 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.3269000000000003e-05 2024-09-10T19:30:28.094 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:30:28.094 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:30:28.094 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:30:28.094 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:30:28.094 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:30:28.094 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.16026.0:109597 11.8 11:1c9118ed:::benchmark_data_smithi183_203944_object109596:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e398)", 2024-09-10T19:30:28.094 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-10T19:28:57.837841+0000", 2024-09-10T19:30:28.095 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 89.619977426000005, 2024-09-10T19:30:28.095 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.47342097999999999, 2024-09-10T19:30:28.095 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-10T19:30:28.095 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T19:30:28.095 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-10T19:30:28.095 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.16026", 2024-09-10T19:30:28.095 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.183:0/1450567484", 2024-09-10T19:30:28.095 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 109597 2024-09-10T19:30:28.095 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:30:28.095 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-10T19:30:28.095 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:30:28.095 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-10T19:30:28.095 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:28:57.837841+0000", 2024-09-10T19:30:28.096 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:30:28.096 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:30:28.096 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:30:28.096 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-10T19:30:28.096 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:28:57.837841+0000", 2024-09-10T19:30:28.096 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:30:28.096 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:30:28.096 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:30:28.096 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-10T19:30:28.096 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:28:57.837840+0000", 2024-09-10T19:30:28.096 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4294967295.999999 2024-09-10T19:30:28.096 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:30:28.096 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:30:28.096 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-10T19:30:28.097 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:28:57.837861+0000", 2024-09-10T19:30:28.097 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.0747999999999999e-05 2024-09-10T19:30:28.097 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:30:28.097 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:30:28.097 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-10T19:30:28.097 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:28:57.837872+0000", 2024-09-10T19:30:28.097 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.0886999999999999e-05 2024-09-10T19:30:28.097 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:30:28.097 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:30:28.097 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-10T19:30:28.097 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:28:57.837877+0000", 2024-09-10T19:30:28.097 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.5570000000000003e-06 2024-09-10T19:30:28.097 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:30:28.098 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:30:28.098 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-10T19:30:28.098 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:28:57.837920+0000", 2024-09-10T19:30:28.098 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.2919e-05 2024-09-10T19:30:28.098 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:30:28.098 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:30:28.098 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-10T19:30:28.098 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:28:57.838023+0000", 2024-09-10T19:30:28.098 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000102515 2024-09-10T19:30:28.098 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:30:28.098 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:30:28.098 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 5,6", 2024-09-10T19:30:28.098 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:28:57.838117+0000", 2024-09-10T19:30:28.099 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.4178999999999994e-05 2024-09-10T19:30:28.099 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:30:28.099 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:30:28.099 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-10T19:30:28.099 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:28:57.839950+0000", 2024-09-10T19:30:28.099 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00183311 2024-09-10T19:30:28.099 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:30:28.099 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:30:28.099 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:30:28.099 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:28:57.840515+0000", 2024-09-10T19:30:28.099 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00056553099999999998 2024-09-10T19:30:28.099 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:30:28.099 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:30:28.100 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:30:28.100 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:28:58.311179+0000", 2024-09-10T19:30:28.100 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.47066333100000002 2024-09-10T19:30:28.100 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:30:28.100 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:30:28.100 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-10T19:30:28.100 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:28:58.311225+0000", 2024-09-10T19:30:28.100 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.6783999999999998e-05 2024-09-10T19:30:28.100 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:30:28.100 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:30:28.100 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-10T19:30:28.100 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:28:58.311262+0000", 2024-09-10T19:30:28.100 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.6378999999999998e-05 2024-09-10T19:30:28.101 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:30:28.101 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:30:28.101 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:30:28.101 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:30:28.101 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:30:28.101 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.16026.0:109598 11.a 11:53657317:::benchmark_data_smithi183_203944_object109597:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e398)", 2024-09-10T19:30:28.101 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-10T19:28:57.839742+0000", 2024-09-10T19:30:28.101 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 89.618075924999999, 2024-09-10T19:30:28.101 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.47296167700000002, 2024-09-10T19:30:28.101 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-10T19:30:28.101 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T19:30:28.101 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-10T19:30:28.101 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.16026", 2024-09-10T19:30:28.101 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.183:0/1450567484", 2024-09-10T19:30:28.102 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 109598 2024-09-10T19:30:28.102 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:30:28.102 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-10T19:30:28.102 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:30:28.102 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-10T19:30:28.102 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:28:57.839742+0000", 2024-09-10T19:30:28.102 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:30:28.102 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:30:28.102 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:30:28.102 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-10T19:30:28.102 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:28:57.839742+0000", 2024-09-10T19:30:28.102 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:30:28.102 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:30:28.102 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:30:28.103 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-10T19:30:28.103 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:28:57.839741+0000", 2024-09-10T19:30:28.103 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4294967295.999999 2024-09-10T19:30:28.103 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:30:28.103 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:30:28.103 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-10T19:30:28.103 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:28:57.839762+0000", 2024-09-10T19:30:28.103 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.056e-05 2024-09-10T19:30:28.103 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:30:28.103 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:30:28.103 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-10T19:30:28.103 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:28:57.839772+0000", 2024-09-10T19:30:28.103 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.0053e-05 2024-09-10T19:30:28.103 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:30:28.104 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:30:28.104 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-10T19:30:28.104 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:28:57.839778+0000", 2024-09-10T19:30:28.104 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.7359999999999998e-06 2024-09-10T19:30:28.104 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:30:28.104 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:30:28.104 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-10T19:30:28.104 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:28:57.839819+0000", 2024-09-10T19:30:28.104 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.0708999999999999e-05 2024-09-10T19:30:28.104 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:30:28.104 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:30:28.104 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-10T19:30:28.104 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:28:57.839902+0000", 2024-09-10T19:30:28.105 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.3411000000000005e-05 2024-09-10T19:30:28.105 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:30:28.105 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:30:28.105 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 5,6", 2024-09-10T19:30:28.105 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:28:57.839999+0000", 2024-09-10T19:30:28.105 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.7516999999999995e-05 2024-09-10T19:30:28.105 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:30:28.105 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:30:28.105 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-10T19:30:28.105 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:28:57.841798+0000", 2024-09-10T19:30:28.105 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.001798736 2024-09-10T19:30:28.106 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:30:28.106 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:30:28.106 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:30:28.106 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:28:57.842420+0000", 2024-09-10T19:30:28.106 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00062174500000000005 2024-09-10T19:30:28.106 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:30:28.106 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:30:28.106 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:30:28.106 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:28:58.312566+0000", 2024-09-10T19:30:28.106 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.47014572900000001 2024-09-10T19:30:28.106 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:30:28.106 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:30:28.106 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-10T19:30:28.107 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:28:58.312672+0000", 2024-09-10T19:30:28.107 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000106057 2024-09-10T19:30:28.107 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:30:28.107 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:30:28.107 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-10T19:30:28.107 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:28:58.312704+0000", 2024-09-10T19:30:28.107 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.2311999999999998e-05 2024-09-10T19:30:28.107 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:30:28.107 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:30:28.107 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:30:28.107 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:30:28.107 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:30:28.107 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.16026.0:227634 11.b 11:d30fd0c8:::benchmark_data_smithi183_203944_object82744:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e415)", 2024-09-10T19:30:28.108 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-10T19:30:21.513775+0000", 2024-09-10T19:30:28.108 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 5.9440428919999997, 2024-09-10T19:30:28.108 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.249436089, 2024-09-10T19:30:28.108 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-10T19:30:28.108 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T19:30:28.108 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-10T19:30:28.108 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.16026", 2024-09-10T19:30:28.108 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.183:0/1450567484", 2024-09-10T19:30:28.108 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 227634 2024-09-10T19:30:28.108 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:30:28.108 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-10T19:30:28.108 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:30:28.109 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-10T19:30:28.109 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:30:21.513775+0000", 2024-09-10T19:30:28.109 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:30:28.109 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:30:28.109 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:30:28.109 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-10T19:30:28.109 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:30:21.513775+0000", 2024-09-10T19:30:28.109 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:30:28.109 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:30:28.109 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:30:28.109 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-10T19:30:28.109 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:30:21.513775+0000", 2024-09-10T19:30:28.109 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4294967295.9999995 2024-09-10T19:30:28.109 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:30:28.110 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:30:28.110 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-10T19:30:28.110 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:30:21.513777+0000", 2024-09-10T19:30:28.110 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.4569999999999999e-06 2024-09-10T19:30:28.110 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:30:28.110 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:30:28.110 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-10T19:30:28.110 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:30:21.513783+0000", 2024-09-10T19:30:28.110 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.7710000000000003e-06 2024-09-10T19:30:28.110 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:30:28.110 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:30:28.110 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-10T19:30:28.110 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:30:21.513787+0000", 2024-09-10T19:30:28.111 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.585e-06 2024-09-10T19:30:28.111 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:30:28.111 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:30:28.111 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-10T19:30:28.111 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:30:21.516235+0000", 2024-09-10T19:30:28.111 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0024476760000000002 2024-09-10T19:30:28.111 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:30:28.111 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:30:28.111 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-10T19:30:28.111 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:30:21.516373+0000", 2024-09-10T19:30:28.111 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00013869500000000001 2024-09-10T19:30:28.111 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:30:28.111 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:30:28.111 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 0,1", 2024-09-10T19:30:28.112 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:30:21.516485+0000", 2024-09-10T19:30:28.112 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00011158900000000001 2024-09-10T19:30:28.112 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:30:28.112 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:30:28.112 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-10T19:30:28.112 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:30:21.525899+0000", 2024-09-10T19:30:28.112 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0094142129999999994 2024-09-10T19:30:28.112 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:30:28.112 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:30:28.112 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:30:28.112 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:30:21.528295+0000", 2024-09-10T19:30:28.112 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0023963790000000001 2024-09-10T19:30:28.112 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:30:28.113 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:30:28.113 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:30:28.113 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:30:21.762934+0000", 2024-09-10T19:30:28.113 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.234638762 2024-09-10T19:30:28.113 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:30:28.113 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:30:28.113 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-10T19:30:28.113 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:30:21.763126+0000", 2024-09-10T19:30:28.113 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00019138 2024-09-10T19:30:28.113 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:30:28.113 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:30:28.113 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-10T19:30:28.113 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:30:21.763211+0000", 2024-09-10T19:30:28.113 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.5955e-05 2024-09-10T19:30:28.114 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:30:28.114 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:30:28.114 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:30:28.114 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:30:28.114 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:30:28.114 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.16026.0:227639 11.b 11:d7a7b515:::benchmark_data_smithi183_203944_object82749:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e415)", 2024-09-10T19:30:28.114 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-10T19:30:21.517088+0000", 2024-09-10T19:30:28.114 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 5.9407302299999998, 2024-09-10T19:30:28.114 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.27640787100000003, 2024-09-10T19:30:28.114 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-10T19:30:28.114 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T19:30:28.114 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-10T19:30:28.115 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.16026", 2024-09-10T19:30:28.115 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.183:0/1450567484", 2024-09-10T19:30:28.115 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 227639 2024-09-10T19:30:28.115 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:30:28.115 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-10T19:30:28.115 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:30:28.115 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-10T19:30:28.115 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:30:21.517088+0000", 2024-09-10T19:30:28.115 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:30:28.115 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:30:28.115 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:30:28.115 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-10T19:30:28.115 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:30:21.517088+0000", 2024-09-10T19:30:28.116 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:30:28.116 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:30:28.116 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:30:28.116 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-10T19:30:28.116 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:30:21.517087+0000", 2024-09-10T19:30:28.116 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4294967295.999999 2024-09-10T19:30:28.116 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:30:28.116 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:30:28.116 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-10T19:30:28.116 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:30:21.517092+0000", 2024-09-10T19:30:28.116 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.9080000000000003e-06 2024-09-10T19:30:28.116 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:30:28.116 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:30:28.116 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-10T19:30:28.117 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:30:21.517102+0000", 2024-09-10T19:30:28.117 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.4760000000000004e-06 2024-09-10T19:30:28.117 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:30:28.117 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:30:28.117 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-10T19:30:28.117 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:30:21.517107+0000", 2024-09-10T19:30:28.117 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.989e-06 2024-09-10T19:30:28.117 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:30:28.117 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:30:28.117 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-10T19:30:28.117 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:30:21.521596+0000", 2024-09-10T19:30:28.117 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0044891009999999997 2024-09-10T19:30:28.117 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:30:28.118 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:30:28.118 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-10T19:30:28.118 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:30:21.521657+0000", 2024-09-10T19:30:28.118 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.1383999999999994e-05 2024-09-10T19:30:28.118 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:30:28.118 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:30:28.118 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 0,1", 2024-09-10T19:30:28.118 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:30:21.521717+0000", 2024-09-10T19:30:28.118 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.9956e-05 2024-09-10T19:30:28.118 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:30:28.118 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:30:28.118 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-10T19:30:28.118 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:30:21.528301+0000", 2024-09-10T19:30:28.118 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0065842640000000003 2024-09-10T19:30:28.119 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:30:28.119 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:30:28.119 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:30:28.119 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:30:21.528343+0000", 2024-09-10T19:30:28.119 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.1844999999999997e-05 2024-09-10T19:30:28.119 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:30:28.119 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:30:28.119 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:30:28.119 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:30:21.792686+0000", 2024-09-10T19:30:28.119 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.26434327400000002 2024-09-10T19:30:28.119 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:30:28.119 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:30:28.119 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-10T19:30:28.119 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:30:21.793451+0000", 2024-09-10T19:30:28.119 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00076462399999999999 2024-09-10T19:30:28.120 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:30:28.120 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:30:28.120 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-10T19:30:28.120 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:30:21.793496+0000", 2024-09-10T19:30:28.120 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.4854000000000001e-05 2024-09-10T19:30:28.120 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:30:28.120 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:30:28.120 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:30:28.120 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:30:28.120 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:30:28.120 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.16026.0:227643 11.b 11:d91fd5d3:::benchmark_data_smithi183_203944_object82753:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e415)", 2024-09-10T19:30:28.120 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-10T19:30:21.520570+0000", 2024-09-10T19:30:28.120 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 5.937248318, 2024-09-10T19:30:28.121 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.27303066999999998, 2024-09-10T19:30:28.121 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-10T19:30:28.121 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T19:30:28.121 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-10T19:30:28.121 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.16026", 2024-09-10T19:30:28.121 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.183:0/1450567484", 2024-09-10T19:30:28.121 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 227643 2024-09-10T19:30:28.121 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:30:28.121 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-10T19:30:28.121 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:30:28.121 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-10T19:30:28.121 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:30:21.520570+0000", 2024-09-10T19:30:28.121 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:30:28.121 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:30:28.121 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:30:28.122 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-10T19:30:28.122 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:30:21.520570+0000", 2024-09-10T19:30:28.122 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:30:28.122 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:30:28.122 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:30:28.122 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-10T19:30:28.122 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:30:21.520569+0000", 2024-09-10T19:30:28.122 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4294967295.999999 2024-09-10T19:30:28.122 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:30:28.122 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:30:28.122 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-10T19:30:28.122 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:30:21.520574+0000", 2024-09-10T19:30:28.122 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.7639999999999996e-06 2024-09-10T19:30:28.122 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:30:28.123 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:30:28.123 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-10T19:30:28.123 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:30:21.520605+0000", 2024-09-10T19:30:28.123 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.1485000000000001e-05 2024-09-10T19:30:28.123 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:30:28.123 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:30:28.123 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-10T19:30:28.123 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:30:21.520612+0000", 2024-09-10T19:30:28.123 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.071e-06 2024-09-10T19:30:28.123 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:30:28.123 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:30:28.123 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-10T19:30:28.123 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:30:21.523719+0000", 2024-09-10T19:30:28.123 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0031077209999999999 2024-09-10T19:30:28.123 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:30:28.123 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:30:28.124 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-10T19:30:28.124 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:30:21.523823+0000", 2024-09-10T19:30:28.124 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000103281 2024-09-10T19:30:28.124 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:30:28.124 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:30:28.124 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 0,1", 2024-09-10T19:30:28.124 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:30:21.523938+0000", 2024-09-10T19:30:28.124 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000115536 2024-09-10T19:30:28.124 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:30:28.124 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:30:28.124 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-10T19:30:28.124 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:30:21.528324+0000", 2024-09-10T19:30:28.124 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0043861660000000004 2024-09-10T19:30:28.124 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:30:28.125 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:30:28.125 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:30:28.125 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:30:21.528357+0000", 2024-09-10T19:30:28.125 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.3000999999999998e-05 2024-09-10T19:30:28.125 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:30:28.125 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:30:28.125 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:30:28.125 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:30:21.793534+0000", 2024-09-10T19:30:28.125 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.26517657100000003 2024-09-10T19:30:28.125 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:30:28.125 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:30:28.125 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-10T19:30:28.125 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:30:21.793571+0000", 2024-09-10T19:30:28.126 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.7375e-05 2024-09-10T19:30:28.126 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:30:28.126 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:30:28.126 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-10T19:30:28.126 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:30:21.793601+0000", 2024-09-10T19:30:28.126 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.9456999999999999e-05 2024-09-10T19:30:28.126 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:30:28.126 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:30:28.126 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:30:28.126 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:30:28.126 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:30:28.126 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.16026.0:227644 11.b 11:da7be35d:::benchmark_data_smithi183_203944_object82754:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e415)", 2024-09-10T19:30:28.126 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-10T19:30:21.520625+0000", 2024-09-10T19:30:28.126 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 5.937193035, 2024-09-10T19:30:28.126 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.27308842, 2024-09-10T19:30:28.126 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-10T19:30:28.127 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T19:30:28.127 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-10T19:30:28.127 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.16026", 2024-09-10T19:30:28.127 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.183:0/1450567484", 2024-09-10T19:30:28.127 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 227644 2024-09-10T19:30:28.127 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:30:28.127 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-10T19:30:28.127 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:30:28.127 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-10T19:30:28.127 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:30:21.520625+0000", 2024-09-10T19:30:28.127 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:30:28.127 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:30:28.127 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:30:28.127 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-10T19:30:28.128 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:30:21.520625+0000", 2024-09-10T19:30:28.128 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:30:28.128 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:30:28.128 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:30:28.128 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-10T19:30:28.128 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:30:21.520625+0000", 2024-09-10T19:30:28.128 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4294967295.9999995 2024-09-10T19:30:28.128 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:30:28.128 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:30:28.128 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-10T19:30:28.128 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:30:21.520628+0000", 2024-09-10T19:30:28.128 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.9270000000000001e-06 2024-09-10T19:30:28.128 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:30:28.128 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:30:28.129 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-10T19:30:28.129 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:30:21.520634+0000", 2024-09-10T19:30:28.129 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.4529999999999999e-06 2024-09-10T19:30:28.129 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:30:28.129 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:30:28.129 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-10T19:30:28.129 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:30:21.520638+0000", 2024-09-10T19:30:28.129 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.3450000000000002e-06 2024-09-10T19:30:28.129 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:30:28.129 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:30:28.129 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-10T19:30:28.129 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:30:21.525948+0000", 2024-09-10T19:30:28.129 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0053100350000000003 2024-09-10T19:30:28.130 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:30:28.130 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:30:28.130 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-10T19:30:28.130 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:30:21.526058+0000", 2024-09-10T19:30:28.130 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000110012 2024-09-10T19:30:28.130 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:30:28.130 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:30:28.130 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 0,1", 2024-09-10T19:30:28.130 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:30:21.526181+0000", 2024-09-10T19:30:28.130 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00012385599999999999 2024-09-10T19:30:28.130 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:30:28.130 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:30:28.130 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-10T19:30:28.130 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:30:21.528327+0000", 2024-09-10T19:30:28.131 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0021455139999999998 2024-09-10T19:30:28.131 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:30:28.131 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:30:28.131 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:30:28.131 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:30:21.528377+0000", 2024-09-10T19:30:28.131 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.9922000000000001e-05 2024-09-10T19:30:28.131 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:30:28.131 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:30:28.131 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:30:28.131 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:30:21.793651+0000", 2024-09-10T19:30:28.131 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.26527367200000002 2024-09-10T19:30:28.131 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:30:28.131 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:30:28.132 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-10T19:30:28.132 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:30:21.793685+0000", 2024-09-10T19:30:28.132 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.4981999999999998e-05 2024-09-10T19:30:28.132 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:30:28.132 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:30:28.132 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-10T19:30:28.132 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:30:21.793714+0000", 2024-09-10T19:30:28.132 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.8180000000000001e-05 2024-09-10T19:30:28.132 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:30:28.132 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:30:28.132 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:30:28.132 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:30:28.133 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:30:28.133 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.16026.0:227647 11.b 11:d046dfb9:::benchmark_data_smithi183_203944_object82757:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e415)", 2024-09-10T19:30:28.133 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-10T19:30:21.524556+0000", 2024-09-10T19:30:28.133 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 5.9332618080000001, 2024-09-10T19:30:28.133 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.26924948500000001, 2024-09-10T19:30:28.133 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-10T19:30:28.133 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T19:30:28.133 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-10T19:30:28.133 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.16026", 2024-09-10T19:30:28.133 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.183:0/1450567484", 2024-09-10T19:30:28.133 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 227647 2024-09-10T19:30:28.133 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:30:28.133 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-10T19:30:28.134 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:30:28.134 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-10T19:30:28.134 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:30:21.524556+0000", 2024-09-10T19:30:28.134 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:30:28.134 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:30:28.134 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:30:28.134 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-10T19:30:28.134 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:30:21.524556+0000", 2024-09-10T19:30:28.134 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:30:28.134 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:30:28.134 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:30:28.134 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-10T19:30:28.135 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:30:21.524556+0000", 2024-09-10T19:30:28.135 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4294967295.999999 2024-09-10T19:30:28.135 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:30:28.135 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:30:28.135 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-10T19:30:28.135 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:30:21.524561+0000", 2024-09-10T19:30:28.135 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.6799999999999998e-06 2024-09-10T19:30:28.135 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:30:28.135 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:30:28.135 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-10T19:30:28.135 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:30:21.524571+0000", 2024-09-10T19:30:28.135 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 9.4229999999999994e-06 2024-09-10T19:30:28.135 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:30:28.136 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:30:28.136 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-10T19:30:28.136 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:30:21.524576+0000", 2024-09-10T19:30:28.136 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.5339999999999999e-06 2024-09-10T19:30:28.136 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:30:28.136 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:30:28.136 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-10T19:30:28.136 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:30:21.528405+0000", 2024-09-10T19:30:28.136 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0038286290000000001 2024-09-10T19:30:28.136 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:30:28.136 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:30:28.136 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-10T19:30:28.136 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:30:21.528517+0000", 2024-09-10T19:30:28.137 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000112002 2024-09-10T19:30:28.137 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:30:28.137 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:30:28.137 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 0,1", 2024-09-10T19:30:28.137 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:30:21.528630+0000", 2024-09-10T19:30:28.137 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000113081 2024-09-10T19:30:28.137 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:30:28.137 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:30:28.137 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-10T19:30:28.137 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:30:21.532649+0000", 2024-09-10T19:30:28.137 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0040194460000000003 2024-09-10T19:30:28.137 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:30:28.137 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:30:28.138 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:30:28.138 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:30:21.532700+0000", 2024-09-10T19:30:28.138 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.079e-05 2024-09-10T19:30:28.138 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:30:28.138 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:30:28.138 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:30:28.138 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:30:21.793746+0000", 2024-09-10T19:30:28.138 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.26104580700000002 2024-09-10T19:30:28.138 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:30:28.138 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:30:28.138 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-10T19:30:28.138 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:30:21.793779+0000", 2024-09-10T19:30:28.139 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.2617999999999998e-05 2024-09-10T19:30:28.139 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:30:28.139 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:30:28.139 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-10T19:30:28.139 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:30:21.793806+0000", 2024-09-10T19:30:28.139 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 2.7432000000000001e-05 2024-09-10T19:30:28.139 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:30:28.139 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:30:28.139 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:30:28.139 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:30:28.139 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:30:28.140 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.16026.0:227650 11.b 11:d2d8694b:::benchmark_data_smithi183_203944_object82760:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e415)", 2024-09-10T19:30:28.140 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-10T19:30:21.528477+0000", 2024-09-10T19:30:28.140 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 5.9293409170000002, 2024-09-10T19:30:28.140 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.265418973, 2024-09-10T19:30:28.140 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-10T19:30:28.140 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T19:30:28.140 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-10T19:30:28.140 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.16026", 2024-09-10T19:30:28.140 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.183:0/1450567484", 2024-09-10T19:30:28.140 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 227650 2024-09-10T19:30:28.140 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:30:28.140 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-10T19:30:28.140 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:30:28.140 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-10T19:30:28.141 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:30:21.528477+0000", 2024-09-10T19:30:28.141 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:30:28.141 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:30:28.141 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:30:28.141 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-10T19:30:28.141 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:30:21.528477+0000", 2024-09-10T19:30:28.141 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:30:28.141 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:30:28.141 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:30:28.141 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-10T19:30:28.141 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:30:21.528476+0000", 2024-09-10T19:30:28.141 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4294967295.999999 2024-09-10T19:30:28.141 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:30:28.141 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:30:28.142 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-10T19:30:28.142 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:30:21.528482+0000", 2024-09-10T19:30:28.142 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.0939999999999996e-06 2024-09-10T19:30:28.142 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:30:28.142 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:30:28.142 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-10T19:30:28.142 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:30:21.528499+0000", 2024-09-10T19:30:28.142 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.6546e-05 2024-09-10T19:30:28.142 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:30:28.142 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:30:28.142 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-10T19:30:28.142 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:30:21.528505+0000", 2024-09-10T19:30:28.143 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.3160000000000002e-06 2024-09-10T19:30:28.143 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:30:28.143 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:30:28.143 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-10T19:30:28.143 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:30:21.530526+0000", 2024-09-10T19:30:28.143 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0020209999999999998 2024-09-10T19:30:28.143 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:30:28.143 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:30:28.143 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-10T19:30:28.143 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:30:21.530633+0000", 2024-09-10T19:30:28.143 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000106414 2024-09-10T19:30:28.143 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:30:28.143 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:30:28.144 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 0,1", 2024-09-10T19:30:28.144 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:30:21.530746+0000", 2024-09-10T19:30:28.144 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000113378 2024-09-10T19:30:28.144 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:30:28.144 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:30:28.144 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-10T19:30:28.144 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:30:21.532772+0000", 2024-09-10T19:30:28.144 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0020264329999999998 2024-09-10T19:30:28.144 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:30:28.144 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:30:28.145 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:30:28.145 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:30:21.533079+0000", 2024-09-10T19:30:28.145 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000306161 2024-09-10T19:30:28.145 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:30:28.145 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:30:28.145 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:30:28.145 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:30:21.793834+0000", 2024-09-10T19:30:28.145 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.26075545999999999 2024-09-10T19:30:28.145 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:30:28.145 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:30:28.145 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-10T19:30:28.145 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:30:21.793865+0000", 2024-09-10T19:30:28.145 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.1390999999999998e-05 2024-09-10T19:30:28.145 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:30:28.146 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:30:28.146 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-10T19:30:28.146 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:30:21.793896+0000", 2024-09-10T19:30:28.146 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.0933999999999999e-05 2024-09-10T19:30:28.146 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:30:28.146 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:30:28.146 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:30:28.146 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:30:28.146 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:30:28.146 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.16026.0:228872 11.9 11:9e5bc215:::benchmark_data_smithi183_203944_object83982:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e415)", 2024-09-10T19:30:28.146 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-10T19:30:22.871216+0000", 2024-09-10T19:30:28.146 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 4.5866027540000003, 2024-09-10T19:30:28.146 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.32780975099999998, 2024-09-10T19:30:28.146 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-10T19:30:28.147 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T19:30:28.147 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-10T19:30:28.147 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.16026", 2024-09-10T19:30:28.147 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.183:0/1450567484", 2024-09-10T19:30:28.147 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 228872 2024-09-10T19:30:28.147 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:30:28.147 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-10T19:30:28.147 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:30:28.147 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-10T19:30:28.147 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:30:22.871216+0000", 2024-09-10T19:30:28.147 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:30:28.147 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:30:28.147 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:30:28.147 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-10T19:30:28.148 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:30:22.871216+0000", 2024-09-10T19:30:28.148 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:30:28.148 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:30:28.148 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:30:28.148 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-10T19:30:28.148 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:30:22.871214+0000", 2024-09-10T19:30:28.148 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4294967295.999999 2024-09-10T19:30:28.148 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:30:28.148 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:30:28.148 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-10T19:30:28.148 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:30:22.871220+0000", 2024-09-10T19:30:28.148 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.7620000000000001e-06 2024-09-10T19:30:28.148 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:30:28.149 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:30:28.149 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-10T19:30:28.149 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:30:22.871231+0000", 2024-09-10T19:30:28.149 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.1012000000000001e-05 2024-09-10T19:30:28.149 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:30:28.149 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:30:28.149 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-10T19:30:28.149 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:30:22.871236+0000", 2024-09-10T19:30:28.149 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.9679999999999997e-06 2024-09-10T19:30:28.149 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:30:28.149 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:30:28.149 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-10T19:30:28.149 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:30:22.871282+0000", 2024-09-10T19:30:28.150 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.5697e-05 2024-09-10T19:30:28.150 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:30:28.150 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:30:28.150 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-10T19:30:28.150 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:30:22.871387+0000", 2024-09-10T19:30:28.150 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00010466799999999999 2024-09-10T19:30:28.150 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:30:28.150 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:30:28.150 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 0,6", 2024-09-10T19:30:28.150 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:30:22.871476+0000", 2024-09-10T19:30:28.150 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.9586999999999995e-05 2024-09-10T19:30:28.150 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:30:28.151 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:30:28.151 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:30:28.151 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:30:22.882800+0000", 2024-09-10T19:30:28.151 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.011323478999999999 2024-09-10T19:30:28.151 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:30:28.151 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:30:28.151 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-10T19:30:28.151 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:30:22.886685+0000", 2024-09-10T19:30:28.151 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0038854670000000001 2024-09-10T19:30:28.151 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:30:28.151 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:30:28.151 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:30:28.151 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:30:23.198942+0000", 2024-09-10T19:30:28.151 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.31225700000000001 2024-09-10T19:30:28.152 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:30:28.152 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:30:28.152 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-10T19:30:28.152 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:30:23.198988+0000", 2024-09-10T19:30:28.152 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.5646000000000003e-05 2024-09-10T19:30:28.152 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:30:28.152 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:30:28.152 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-10T19:30:28.152 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:30:23.199025+0000", 2024-09-10T19:30:28.152 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.7595999999999997e-05 2024-09-10T19:30:28.152 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:30:28.152 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:30:28.152 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:30:28.153 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:30:28.153 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:30:28.153 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.16026.0:228879 11.9 11:9c060528:::benchmark_data_smithi183_203944_object83989:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e415)", 2024-09-10T19:30:28.153 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-10T19:30:22.872822+0000", 2024-09-10T19:30:28.153 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 4.5849964270000001, 2024-09-10T19:30:28.153 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.33573559400000003, 2024-09-10T19:30:28.153 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-10T19:30:28.153 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T19:30:28.153 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-10T19:30:28.153 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.16026", 2024-09-10T19:30:28.153 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.183:0/1450567484", 2024-09-10T19:30:28.153 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 228879 2024-09-10T19:30:28.153 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:30:28.154 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-10T19:30:28.154 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:30:28.154 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-10T19:30:28.154 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:30:22.872822+0000", 2024-09-10T19:30:28.154 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:30:28.154 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:30:28.154 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:30:28.154 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-10T19:30:28.154 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:30:22.872822+0000", 2024-09-10T19:30:28.154 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:30:28.154 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:30:28.154 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:30:28.154 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-10T19:30:28.155 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:30:22.872821+0000", 2024-09-10T19:30:28.155 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4294967295.999999 2024-09-10T19:30:28.155 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:30:28.155 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:30:28.155 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-10T19:30:28.155 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:30:22.872826+0000", 2024-09-10T19:30:28.155 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.882e-06 2024-09-10T19:30:28.155 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:30:28.155 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:30:28.155 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-10T19:30:28.155 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:30:22.872837+0000", 2024-09-10T19:30:28.155 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.1085e-05 2024-09-10T19:30:28.155 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:30:28.155 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:30:28.156 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-10T19:30:28.156 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:30:22.872842+0000", 2024-09-10T19:30:28.156 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.2630000000000003e-06 2024-09-10T19:30:28.156 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:30:28.156 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:30:28.156 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-10T19:30:28.156 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:30:22.877105+0000", 2024-09-10T19:30:28.156 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0042628750000000002 2024-09-10T19:30:28.156 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:30:28.156 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:30:28.156 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-10T19:30:28.156 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:30:22.877264+0000", 2024-09-10T19:30:28.156 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0001585 2024-09-10T19:30:28.157 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:30:28.157 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:30:28.157 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 0,6", 2024-09-10T19:30:28.157 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:30:22.877404+0000", 2024-09-10T19:30:28.157 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00014040100000000001 2024-09-10T19:30:28.157 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:30:28.157 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:30:28.157 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:30:28.157 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:30:22.882943+0000", 2024-09-10T19:30:28.157 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0055385069999999998 2024-09-10T19:30:28.157 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:30:28.157 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:30:28.158 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-10T19:30:28.158 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:30:22.886691+0000", 2024-09-10T19:30:28.158 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.003748787 2024-09-10T19:30:28.158 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:30:28.158 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:30:28.158 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:30:28.158 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:30:23.208219+0000", 2024-09-10T19:30:28.158 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.32152761200000002 2024-09-10T19:30:28.158 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:30:28.158 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:30:28.158 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-10T19:30:28.158 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:30:23.208431+0000", 2024-09-10T19:30:28.159 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00021175 2024-09-10T19:30:28.159 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:30:28.159 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:30:28.159 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-10T19:30:28.159 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:30:23.208557+0000", 2024-09-10T19:30:28.159 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00012666899999999999 2024-09-10T19:30:28.159 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:30:28.159 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:30:28.159 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:30:28.159 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:30:28.159 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:30:28.159 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.16026.0:228897 11.9 11:92ef8df4:::benchmark_data_smithi183_203944_object84007:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e415)", 2024-09-10T19:30:28.159 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-10T19:30:22.887699+0000", 2024-09-10T19:30:28.160 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 4.570119654, 2024-09-10T19:30:28.160 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.32589949200000001, 2024-09-10T19:30:28.160 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-10T19:30:28.160 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T19:30:28.160 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-10T19:30:28.160 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.16026", 2024-09-10T19:30:28.160 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.183:0/1450567484", 2024-09-10T19:30:28.160 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 228897 2024-09-10T19:30:28.160 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:30:28.160 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-10T19:30:28.160 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:30:28.160 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-10T19:30:28.161 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:30:22.887699+0000", 2024-09-10T19:30:28.161 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:30:28.161 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:30:28.161 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:30:28.161 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-10T19:30:28.161 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:30:22.887699+0000", 2024-09-10T19:30:28.161 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:30:28.161 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:30:28.161 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:30:28.161 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-10T19:30:28.161 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:30:22.887697+0000", 2024-09-10T19:30:28.161 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4294967295.9999986 2024-09-10T19:30:28.161 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:30:28.162 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:30:28.162 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-10T19:30:28.162 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:30:22.887704+0000", 2024-09-10T19:30:28.162 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.6019999999999996e-06 2024-09-10T19:30:28.162 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:30:28.162 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:30:28.162 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-10T19:30:28.162 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:30:22.887720+0000", 2024-09-10T19:30:28.162 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 1.5588000000000001e-05 2024-09-10T19:30:28.162 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:30:28.162 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:30:28.162 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-10T19:30:28.162 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:30:22.887725+0000", 2024-09-10T19:30:28.162 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 5.7389999999999996e-06 2024-09-10T19:30:28.163 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:30:28.163 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:30:28.163 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-10T19:30:28.163 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:30:22.889238+0000", 2024-09-10T19:30:28.163 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.001513084 2024-09-10T19:30:28.163 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:30:28.163 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:30:28.163 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-10T19:30:28.163 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:30:22.889345+0000", 2024-09-10T19:30:28.163 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000106229 2024-09-10T19:30:28.163 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:30:28.163 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:30:28.163 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 0,6", 2024-09-10T19:30:28.163 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:30:22.889451+0000", 2024-09-10T19:30:28.164 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000106149 2024-09-10T19:30:28.164 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:30:28.164 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:30:28.164 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:30:28.164 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:30:22.892988+0000", 2024-09-10T19:30:28.164 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0035372580000000002 2024-09-10T19:30:28.164 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:30:28.164 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:30:28.164 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-10T19:30:28.164 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:30:22.893211+0000", 2024-09-10T19:30:28.164 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.00022293399999999999 2024-09-10T19:30:28.164 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:30:28.164 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:30:28.165 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:30:28.165 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:30:23.213514+0000", 2024-09-10T19:30:28.165 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.32030341200000001 2024-09-10T19:30:28.165 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:30:28.165 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:30:28.165 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-10T19:30:28.165 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:30:23.213562+0000", 2024-09-10T19:30:28.165 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4.7796000000000001e-05 2024-09-10T19:30:28.165 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:30:28.165 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:30:28.165 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-10T19:30:28.165 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:30:23.213598+0000", 2024-09-10T19:30:28.166 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.5908000000000002e-05 2024-09-10T19:30:28.166 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:30:28.166 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:30:28.166 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:30:28.166 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:30:28.166 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:30:28.166 INFO:teuthology.orchestra.run.smithi183.stdout: "description": "osd_op(client.16026.0:228912 11.b 11:d9a8f57e:::benchmark_data_smithi183_203944_object84022:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e415)", 2024-09-10T19:30:28.166 INFO:teuthology.orchestra.run.smithi183.stdout: "initiated_at": "2024-09-10T19:30:22.905213+0000", 2024-09-10T19:30:28.166 INFO:teuthology.orchestra.run.smithi183.stdout: "age": 4.5526051460000003, 2024-09-10T19:30:28.166 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.29651384200000003, 2024-09-10T19:30:28.166 INFO:teuthology.orchestra.run.smithi183.stdout: "type_data": { 2024-09-10T19:30:28.166 INFO:teuthology.orchestra.run.smithi183.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T19:30:28.166 INFO:teuthology.orchestra.run.smithi183.stdout: "client_info": { 2024-09-10T19:30:28.166 INFO:teuthology.orchestra.run.smithi183.stdout: "client": "client.16026", 2024-09-10T19:30:28.166 INFO:teuthology.orchestra.run.smithi183.stdout: "client_addr": "172.21.15.183:0/1450567484", 2024-09-10T19:30:28.167 INFO:teuthology.orchestra.run.smithi183.stdout: "tid": 228912 2024-09-10T19:30:28.167 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:30:28.167 INFO:teuthology.orchestra.run.smithi183.stdout: "events": [ 2024-09-10T19:30:28.167 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:30:28.167 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "initiated", 2024-09-10T19:30:28.167 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:30:22.905213+0000", 2024-09-10T19:30:28.167 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:30:28.167 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:30:28.167 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:30:28.167 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "throttled", 2024-09-10T19:30:28.167 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:30:22.905213+0000", 2024-09-10T19:30:28.167 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0 2024-09-10T19:30:28.167 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:30:28.167 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:30:28.168 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "header_read", 2024-09-10T19:30:28.168 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:30:22.905212+0000", 2024-09-10T19:30:28.168 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 4294967295.9999995 2024-09-10T19:30:28.168 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:30:28.168 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:30:28.168 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "all_read", 2024-09-10T19:30:28.168 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:30:22.905216+0000", 2024-09-10T19:30:28.168 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.8720000000000004e-06 2024-09-10T19:30:28.168 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:30:28.168 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:30:28.168 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "dispatched", 2024-09-10T19:30:28.168 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:30:22.905225+0000", 2024-09-10T19:30:28.168 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 8.4579999999999993e-06 2024-09-10T19:30:28.168 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:30:28.168 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:30:28.169 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "queued_for_pg", 2024-09-10T19:30:28.169 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:30:22.905228+0000", 2024-09-10T19:30:28.169 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.4199999999999999e-06 2024-09-10T19:30:28.169 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:30:28.169 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:30:28.169 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "reached_pg", 2024-09-10T19:30:28.169 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:30:22.905259+0000", 2024-09-10T19:30:28.169 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.048e-05 2024-09-10T19:30:28.169 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:30:28.169 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:30:28.169 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "started", 2024-09-10T19:30:28.169 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:30:22.905330+0000", 2024-09-10T19:30:28.169 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 7.1627999999999999e-05 2024-09-10T19:30:28.169 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:30:28.170 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:30:28.170 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "waiting for subops from 0,1", 2024-09-10T19:30:28.170 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:30:22.905394+0000", 2024-09-10T19:30:28.170 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.3480999999999997e-05 2024-09-10T19:30:28.170 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:30:28.170 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:30:28.170 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "op_commit", 2024-09-10T19:30:28.170 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:30:22.907150+0000", 2024-09-10T19:30:28.170 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.0017563909999999999 2024-09-10T19:30:28.170 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:30:28.170 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:30:28.170 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:30:28.170 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:30:22.907386+0000", 2024-09-10T19:30:28.170 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.000236133 2024-09-10T19:30:28.171 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:30:28.171 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:30:28.171 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:30:28.171 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:30:23.201627+0000", 2024-09-10T19:30:28.171 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 0.29424099999999997 2024-09-10T19:30:28.171 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:30:28.171 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:30:28.171 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "commit_sent", 2024-09-10T19:30:28.171 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:30:23.201690+0000", 2024-09-10T19:30:28.171 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 6.2706000000000006e-05 2024-09-10T19:30:28.171 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:30:28.171 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:30:28.171 INFO:teuthology.orchestra.run.smithi183.stdout: "event": "done", 2024-09-10T19:30:28.172 INFO:teuthology.orchestra.run.smithi183.stdout: "time": "2024-09-10T19:30:23.201727+0000", 2024-09-10T19:30:28.172 INFO:teuthology.orchestra.run.smithi183.stdout: "duration": 3.6934999999999998e-05 2024-09-10T19:30:28.172 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:30:28.172 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:30:28.172 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:30:28.172 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:30:28.172 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:30:28.172 INFO:teuthology.orchestra.run.smithi183.stdout:} 2024-09-10T19:30:28.172 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:30:28.211 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:30:28.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:30:28.212 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:30:28.313 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:30:28.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:30:28.348 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:30:28.348 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:30:28.449 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:30:28.481 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:30:28.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:30:28.482 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:30:28.583 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:30:28.617 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:30:28.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:30:28.618 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:30:28.681 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:30:28 smithi183 ceph-mon[29736]: 2.b scrub starts 2024-09-10T19:30:28.681 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:30:28 smithi183 ceph-mon[29736]: 2.b scrub ok 2024-09-10T19:30:28.681 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:30:28 smithi183 ceph-mon[29736]: 5.b scrub starts 2024-09-10T19:30:28.681 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:30:28 smithi183 ceph-mon[29736]: 5.b scrub ok 2024-09-10T19:30:28.682 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:30:28 smithi183 ceph-mon[41562]: 2.b scrub starts 2024-09-10T19:30:28.682 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:30:28 smithi183 ceph-mon[41562]: 2.b scrub ok 2024-09-10T19:30:28.682 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:30:28 smithi183 ceph-mon[41562]: 5.b scrub starts 2024-09-10T19:30:28.682 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:30:28 smithi183 ceph-mon[41562]: 5.b scrub ok 2024-09-10T19:30:28.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:30:28 smithi191 ceph-mon[32884]: 2.b scrub starts 2024-09-10T19:30:28.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:30:28 smithi191 ceph-mon[32884]: 2.b scrub ok 2024-09-10T19:30:28.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:30:28 smithi191 ceph-mon[32884]: 5.b scrub starts 2024-09-10T19:30:28.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:30:28 smithi191 ceph-mon[32884]: 5.b scrub ok 2024-09-10T19:30:28.719 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:30:28.760 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:30:28.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:30:28.761 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:30:28.862 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:30:28.899 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:30:28.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:30:28.900 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:30:29.001 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:30:29.037 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:30:29.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:30:29.038 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:30:29.139 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:30:29.177 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:30:29.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:30:29.178 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:30:29.279 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:30:29.315 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:30:29.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:30:29.316 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:30:29.416 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:30:29.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:30:29.452 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:30:29.452 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:30:29.553 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:30:29.587 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:30:29.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:30:29.588 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:30:29.621 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:30:29 smithi183 ceph-mon[29736]: pgmap v937: 158 pgs: 158 active+clean; 3.5 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 33 KiB/s rd, 0 B/s wr, 1.34k op/s 2024-09-10T19:30:29.621 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:30:29 smithi183 ceph-mon[29736]: 1.8 scrub starts 2024-09-10T19:30:29.621 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:30:29 smithi183 ceph-mon[29736]: 1.8 scrub ok 2024-09-10T19:30:29.621 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:30:29 smithi183 ceph-mon[29736]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:30:29.621 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:30:29 smithi183 ceph-mon[29736]: 4.1d scrub starts 2024-09-10T19:30:29.621 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:30:29 smithi183 ceph-mon[29736]: 4.1d scrub ok 2024-09-10T19:30:29.622 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:30:29 smithi183 ceph-mon[41562]: pgmap v937: 158 pgs: 158 active+clean; 3.5 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 33 KiB/s rd, 0 B/s wr, 1.34k op/s 2024-09-10T19:30:29.622 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:30:29 smithi183 ceph-mon[41562]: 1.8 scrub starts 2024-09-10T19:30:29.622 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:30:29 smithi183 ceph-mon[41562]: 1.8 scrub ok 2024-09-10T19:30:29.622 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:30:29 smithi183 ceph-mon[41562]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:30:29.622 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:30:29 smithi183 ceph-mon[41562]: 4.1d scrub starts 2024-09-10T19:30:29.622 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:30:29 smithi183 ceph-mon[41562]: 4.1d scrub ok 2024-09-10T19:30:29.689 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:30:29.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:30:29 smithi191 ceph-mon[32884]: pgmap v937: 158 pgs: 158 active+clean; 3.5 GiB data, 16 GiB used, 610 GiB / 626 GiB avail; 33 KiB/s rd, 0 B/s wr, 1.34k op/s 2024-09-10T19:30:29.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:30:29 smithi191 ceph-mon[32884]: 1.8 scrub starts 2024-09-10T19:30:29.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:30:29 smithi191 ceph-mon[32884]: 1.8 scrub ok 2024-09-10T19:30:29.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:30:29 smithi191 ceph-mon[32884]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:30:29.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:30:29 smithi191 ceph-mon[32884]: 4.1d scrub starts 2024-09-10T19:30:29.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:30:29 smithi191 ceph-mon[32884]: 4.1d scrub ok 2024-09-10T19:30:29.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:30:29.724 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:30:29.724 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:30:29.825 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:30:29.845 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:30:29.846 DEBUG:teuthology.orchestra.run.smithi191:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph daemon osd.4 dump_ops_in_flight 2024-09-10T19:30:29.852 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:30:29.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:30:29.853 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:30:29.955 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:30:29.997 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:30:29.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:30:29.998 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:30:30.099 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:30:30.134 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:30:30.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:30:30.135 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:30:30.215 INFO:journalctl@ceph.iscsi.iscsi.a.smithi191.stdout:Sep 10 19:30:30 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-iscsi-iscsi-a[61816]: debug there is no tcmu-runner data available 2024-09-10T19:30:30.236 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:30:30.249 INFO:teuthology.orchestra.run.smithi191.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.b/config 2024-09-10T19:30:30.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:30:30.269 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:30:30.269 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:30:30.370 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:30:30.434 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:30:30.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:30:30.435 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:30:30.534 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:30:30 smithi183 ceph-mon[29736]: 5.1b scrub starts 2024-09-10T19:30:30.534 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:30:30 smithi183 ceph-mon[29736]: 5.1b scrub ok 2024-09-10T19:30:30.534 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:30:30 smithi183 ceph-mon[29736]: 6.c scrub starts 2024-09-10T19:30:30.534 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:30:30 smithi183 ceph-mon[29736]: 6.c scrub ok 2024-09-10T19:30:30.534 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:30:30 smithi183 ceph-mon[41562]: 5.1b scrub starts 2024-09-10T19:30:30.534 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:30:30 smithi183 ceph-mon[41562]: 5.1b scrub ok 2024-09-10T19:30:30.534 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:30:30 smithi183 ceph-mon[41562]: 6.c scrub starts 2024-09-10T19:30:30.534 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:30:30 smithi183 ceph-mon[41562]: 6.c scrub ok 2024-09-10T19:30:30.536 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:30:30.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:30:30.571 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:30:30.571 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:30:30.577 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:30:30 smithi191 ceph-mon[32884]: 5.1b scrub starts 2024-09-10T19:30:30.577 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:30:30 smithi191 ceph-mon[32884]: 5.1b scrub ok 2024-09-10T19:30:30.577 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:30:30 smithi191 ceph-mon[32884]: 6.c scrub starts 2024-09-10T19:30:30.577 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:30:30 smithi191 ceph-mon[32884]: 6.c scrub ok 2024-09-10T19:30:30.672 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:30:30.708 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:30:30.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:30:30.709 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:30:30.809 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:30:30.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:30:30.850 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:30:30.850 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:30:30.864 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:30:30.951 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:30:30.987 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:30:30.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:30:30.988 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:30:31.089 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:30:31.126 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:30:31.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:30:31.127 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:30:31.228 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:30:31.263 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:30:31.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:30:31.264 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:30:31.334 INFO:teuthology.orchestra.run.smithi183.stderr:nodeep-scrub is unset 2024-09-10T19:30:31.365 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:30:31.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:30:31.399 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:30:31.399 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:30:31.500 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:30:31.535 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:30:31.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:30:31.536 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:30:31.637 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:30:31.676 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:30:31.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:30:31.677 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:30:31.681 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:30:31 smithi183 ceph-mon[29736]: pgmap v938: 158 pgs: 158 active+clean; 3.4 GiB data, 15 GiB used, 610 GiB / 626 GiB avail; 22 KiB/s rd, 0 B/s wr, 1.18k op/s 2024-09-10T19:30:31.681 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:30:31 smithi183 ceph-mon[29736]: from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:30:31.681 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:30:31 smithi183 ceph-mon[29736]: from='client.? 172.21.15.183:0/1099749513' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-10T19:30:31.682 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:30:31 smithi183 ceph-mon[29736]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-10T19:30:31.682 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:30:31 smithi183 ceph-mon[29736]: 6.1e scrub starts 2024-09-10T19:30:31.682 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:30:31 smithi183 ceph-mon[29736]: 6.1e scrub ok 2024-09-10T19:30:31.682 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:30:31 smithi183 ceph-mon[29736]: 5.5 scrub starts 2024-09-10T19:30:31.682 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:30:31 smithi183 ceph-mon[29736]: 5.5 scrub ok 2024-09-10T19:30:31.682 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Sep 10 19:30:31 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-y[29989]: ::ffff:172.21.15.191 - - [10/Sep/2024:19:30:31] "GET /metrics HTTP/1.1" 200 224934 "" "Prometheus/2.43.0" 2024-09-10T19:30:31.682 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:30:31 smithi183 ceph-mon[41562]: pgmap v938: 158 pgs: 158 active+clean; 3.4 GiB data, 15 GiB used, 610 GiB / 626 GiB avail; 22 KiB/s rd, 0 B/s wr, 1.18k op/s 2024-09-10T19:30:31.682 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:30:31 smithi183 ceph-mon[41562]: from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:30:31.683 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:30:31 smithi183 ceph-mon[41562]: from='client.? 172.21.15.183:0/1099749513' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-10T19:30:31.683 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:30:31 smithi183 ceph-mon[41562]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-10T19:30:31.683 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:30:31 smithi183 ceph-mon[41562]: 6.1e scrub starts 2024-09-10T19:30:31.683 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:30:31 smithi183 ceph-mon[41562]: 6.1e scrub ok 2024-09-10T19:30:31.683 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:30:31 smithi183 ceph-mon[41562]: 5.5 scrub starts 2024-09-10T19:30:31.683 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:30:31 smithi183 ceph-mon[41562]: 5.5 scrub ok 2024-09-10T19:30:31.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:30:31 smithi191 ceph-mon[32884]: pgmap v938: 158 pgs: 158 active+clean; 3.4 GiB data, 15 GiB used, 610 GiB / 626 GiB avail; 22 KiB/s rd, 0 B/s wr, 1.18k op/s 2024-09-10T19:30:31.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:30:31 smithi191 ceph-mon[32884]: from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:30:31.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:30:31 smithi191 ceph-mon[32884]: from='client.? 172.21.15.183:0/1099749513' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-10T19:30:31.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:30:31 smithi191 ceph-mon[32884]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-10T19:30:31.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:30:31 smithi191 ceph-mon[32884]: 6.1e scrub starts 2024-09-10T19:30:31.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:30:31 smithi191 ceph-mon[32884]: 6.1e scrub ok 2024-09-10T19:30:31.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:30:31 smithi191 ceph-mon[32884]: 5.5 scrub starts 2024-09-10T19:30:31.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:30:31 smithi191 ceph-mon[32884]: 5.5 scrub ok 2024-09-10T19:30:31.778 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:30:31.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:30:31.813 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:30:31.813 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:30:31.914 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:30:31.953 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:30:31.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:30:31.954 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:30:32.054 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:30:32.091 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:30:32.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:30:32.092 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:30:32.193 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:30:32.230 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:30:32.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:30:32.231 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:30:32.332 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:30:32.370 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:30:32.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:30:32.371 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:30:32.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:30:32 smithi191 ceph-mon[32884]: 6.1c scrub starts 2024-09-10T19:30:32.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:30:32 smithi191 ceph-mon[32884]: 6.1c scrub ok 2024-09-10T19:30:32.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:30:32 smithi191 ceph-mon[32884]: 6.4 scrub starts 2024-09-10T19:30:32.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:30:32 smithi191 ceph-mon[32884]: 6.4 scrub ok 2024-09-10T19:30:32.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:30:32 smithi191 ceph-mon[32884]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-10T19:30:32.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:30:32 smithi191 ceph-mon[32884]: Cluster is now healthy 2024-09-10T19:30:32.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:30:32 smithi191 ceph-mon[32884]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-10T19:30:32.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:30:32 smithi191 ceph-mon[32884]: osdmap e416: 8 total, 8 up, 7 in 2024-09-10T19:30:32.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:30:32 smithi191 ceph-mon[32884]: 2.1 scrub starts 2024-09-10T19:30:32.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:30:32 smithi191 ceph-mon[32884]: 2.1 scrub ok 2024-09-10T19:30:32.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:30:32 smithi191 ceph-mon[32884]: 2.5 scrub starts 2024-09-10T19:30:32.472 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:30:32.502 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:30:32.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:30:32.503 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:30:32.560 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:30:32 smithi183 ceph-mon[29736]: 6.1c scrub starts 2024-09-10T19:30:32.561 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:30:32 smithi183 ceph-mon[29736]: 6.1c scrub ok 2024-09-10T19:30:32.561 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:30:32 smithi183 ceph-mon[29736]: 6.4 scrub starts 2024-09-10T19:30:32.561 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:30:32 smithi183 ceph-mon[29736]: 6.4 scrub ok 2024-09-10T19:30:32.561 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:30:32 smithi183 ceph-mon[29736]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-10T19:30:32.561 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:30:32 smithi183 ceph-mon[29736]: Cluster is now healthy 2024-09-10T19:30:32.561 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:30:32 smithi183 ceph-mon[29736]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-10T19:30:32.561 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:30:32 smithi183 ceph-mon[29736]: osdmap e416: 8 total, 8 up, 7 in 2024-09-10T19:30:32.561 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:30:32 smithi183 ceph-mon[29736]: 2.1 scrub starts 2024-09-10T19:30:32.561 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:30:32 smithi183 ceph-mon[29736]: 2.1 scrub ok 2024-09-10T19:30:32.561 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:30:32 smithi183 ceph-mon[29736]: 2.5 scrub starts 2024-09-10T19:30:32.561 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:30:32 smithi183 ceph-mon[41562]: 6.1c scrub starts 2024-09-10T19:30:32.562 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:30:32 smithi183 ceph-mon[41562]: 6.1c scrub ok 2024-09-10T19:30:32.562 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:30:32 smithi183 ceph-mon[41562]: 6.4 scrub starts 2024-09-10T19:30:32.562 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:30:32 smithi183 ceph-mon[41562]: 6.4 scrub ok 2024-09-10T19:30:32.562 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:30:32 smithi183 ceph-mon[41562]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-10T19:30:32.562 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:30:32 smithi183 ceph-mon[41562]: Cluster is now healthy 2024-09-10T19:30:32.562 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:30:32 smithi183 ceph-mon[41562]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-10T19:30:32.562 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:30:32 smithi183 ceph-mon[41562]: osdmap e416: 8 total, 8 up, 7 in 2024-09-10T19:30:32.562 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:30:32 smithi183 ceph-mon[41562]: 2.1 scrub starts 2024-09-10T19:30:32.562 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:30:32 smithi183 ceph-mon[41562]: 2.1 scrub ok 2024-09-10T19:30:32.562 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:30:32 smithi183 ceph-mon[41562]: 2.5 scrub starts 2024-09-10T19:30:32.604 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:30:32.642 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:30:32.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:30:32.643 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:30:32.744 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:30:32.782 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:30:32.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:30:32.783 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:30:32.883 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:30:32.917 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:30:32.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:30:32.918 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:30:33.019 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:30:33.060 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:30:33.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:30:33.061 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:30:33.162 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:30:33.198 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:30:33.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:30:33.199 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:30:33.300 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:30:33.334 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:30:33.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:30:33.335 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:30:33.350 INFO:teuthology.orchestra.run.smithi191.stdout:{ 2024-09-10T19:30:33.350 INFO:teuthology.orchestra.run.smithi191.stdout: "ops": [ 2024-09-10T19:30:33.350 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:30:33.350 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.16026.0:243688 11.e 11:74dbed7d:::benchmark_data_smithi183_203944_object98798:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e416)", 2024-09-10T19:30:33.350 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-10T19:30:33.344653+0000", 2024-09-10T19:30:33.350 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 0.00325406, 2024-09-10T19:30:33.350 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0032819339999999998, 2024-09-10T19:30:33.351 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-10T19:30:33.351 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "waiting for sub ops", 2024-09-10T19:30:33.351 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-10T19:30:33.351 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.16026", 2024-09-10T19:30:33.351 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.183:0/1450567484", 2024-09-10T19:30:33.351 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 243688 2024-09-10T19:30:33.351 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:30:33.351 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-10T19:30:33.351 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:30:33.351 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-10T19:30:33.351 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:30:33.344653+0000", 2024-09-10T19:30:33.351 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:30:33.351 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:30:33.352 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:30:33.352 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-10T19:30:33.352 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:30:33.344653+0000", 2024-09-10T19:30:33.352 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:30:33.352 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:30:33.352 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:30:33.352 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-10T19:30:33.352 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:30:33.344651+0000", 2024-09-10T19:30:33.352 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999986 2024-09-10T19:30:33.352 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:30:33.352 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:30:33.352 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-10T19:30:33.352 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:30:33.344659+0000", 2024-09-10T19:30:33.353 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.7449999999999995e-06 2024-09-10T19:30:33.353 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:30:33.353 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:30:33.353 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-10T19:30:33.353 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:30:33.344674+0000", 2024-09-10T19:30:33.353 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.5058000000000001e-05 2024-09-10T19:30:33.353 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:30:33.353 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:30:33.353 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-10T19:30:33.353 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:30:33.344681+0000", 2024-09-10T19:30:33.353 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.3620000000000002e-06 2024-09-10T19:30:33.353 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:30:33.353 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:30:33.354 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:30:33.354 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:30:33.344941+0000", 2024-09-10T19:30:33.354 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00025998800000000001 2024-09-10T19:30:33.354 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:30:33.354 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:30:33.354 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-10T19:30:33.354 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:30:33.345199+0000", 2024-09-10T19:30:33.354 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00025844600000000001 2024-09-10T19:30:33.354 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:30:33.354 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:30:33.354 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 1,7", 2024-09-10T19:30:33.354 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:30:33.345526+0000", 2024-09-10T19:30:33.354 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00032686400000000001 2024-09-10T19:30:33.354 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:30:33.355 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-10T19:30:33.355 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:30:33.355 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:30:33.355 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:30:33.355 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.16026.0:243679 11.e 11:7d85ec45:::benchmark_data_smithi183_203944_object98789:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e416)", 2024-09-10T19:30:33.355 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-10T19:30:33.341136+0000", 2024-09-10T19:30:33.355 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 0.0067710360000000002, 2024-09-10T19:30:33.355 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0068933600000000003, 2024-09-10T19:30:33.355 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-10T19:30:33.355 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "waiting for sub ops", 2024-09-10T19:30:33.355 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-10T19:30:33.355 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.16026", 2024-09-10T19:30:33.355 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.183:0/1450567484", 2024-09-10T19:30:33.356 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 243679 2024-09-10T19:30:33.356 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:30:33.356 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-10T19:30:33.356 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:30:33.356 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-10T19:30:33.356 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:30:33.341136+0000", 2024-09-10T19:30:33.356 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:30:33.356 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:30:33.356 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:30:33.356 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-10T19:30:33.356 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:30:33.341136+0000", 2024-09-10T19:30:33.356 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:30:33.356 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:30:33.357 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:30:33.357 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-10T19:30:33.357 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:30:33.341134+0000", 2024-09-10T19:30:33.357 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999986 2024-09-10T19:30:33.357 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:30:33.357 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:30:33.357 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-10T19:30:33.357 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:30:33.341140+0000", 2024-09-10T19:30:33.357 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.3679999999999998e-06 2024-09-10T19:30:33.357 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:30:33.358 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:30:33.358 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-10T19:30:33.358 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:30:33.341158+0000", 2024-09-10T19:30:33.358 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.719e-05 2024-09-10T19:30:33.358 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:30:33.358 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:30:33.358 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-10T19:30:33.358 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:30:33.341165+0000", 2024-09-10T19:30:33.358 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.3069999999999997e-06 2024-09-10T19:30:33.358 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:30:33.358 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:30:33.358 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:30:33.358 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:30:33.341218+0000", 2024-09-10T19:30:33.359 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.3198000000000001e-05 2024-09-10T19:30:33.359 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:30:33.359 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:30:33.359 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-10T19:30:33.359 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:30:33.341347+0000", 2024-09-10T19:30:33.359 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00012846099999999999 2024-09-10T19:30:33.359 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:30:33.359 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:30:33.359 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 1,7", 2024-09-10T19:30:33.359 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:30:33.341472+0000", 2024-09-10T19:30:33.359 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00012574400000000001 2024-09-10T19:30:33.359 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:30:33.359 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:30:33.360 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:30:33.360 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:30:33.344451+0000", 2024-09-10T19:30:33.360 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.002978156 2024-09-10T19:30:33.360 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:30:33.360 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:30:33.360 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-09-10T19:30:33.360 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:30:33.344467+0000", 2024-09-10T19:30:33.360 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.6384000000000001e-05 2024-09-10T19:30:33.360 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:30:33.360 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-10T19:30:33.360 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:30:33.360 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:30:33.360 INFO:teuthology.orchestra.run.smithi191.stdout: ], 2024-09-10T19:30:33.361 INFO:teuthology.orchestra.run.smithi191.stdout: "num_ops": 2 2024-09-10T19:30:33.361 INFO:teuthology.orchestra.run.smithi191.stdout:} 2024-09-10T19:30:33.436 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:30:33.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:30:33.473 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:30:33.473 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:30:33.574 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:30:33.605 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:30:33.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:30:33.606 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:30:33.681 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:30:33 smithi183 ceph-mon[29736]: 2.14 scrub starts 2024-09-10T19:30:33.682 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:30:33 smithi183 ceph-mon[29736]: 2.14 scrub ok 2024-09-10T19:30:33.682 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:30:33 smithi183 ceph-mon[29736]: pgmap v940: 158 pgs: 158 active+clean; 3.1 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.31k op/s 2024-09-10T19:30:33.682 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:30:33 smithi183 ceph-mon[29736]: 2.5 scrub ok 2024-09-10T19:30:33.682 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:30:33 smithi183 ceph-mon[29736]: 5.4 scrub starts 2024-09-10T19:30:33.682 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:30:33 smithi183 ceph-mon[29736]: 5.4 scrub ok 2024-09-10T19:30:33.682 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:30:33 smithi183 ceph-mon[41562]: 2.14 scrub starts 2024-09-10T19:30:33.682 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:30:33 smithi183 ceph-mon[41562]: 2.14 scrub ok 2024-09-10T19:30:33.682 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:30:33 smithi183 ceph-mon[41562]: pgmap v940: 158 pgs: 158 active+clean; 3.1 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.31k op/s 2024-09-10T19:30:33.682 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:30:33 smithi183 ceph-mon[41562]: 2.5 scrub ok 2024-09-10T19:30:33.682 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:30:33 smithi183 ceph-mon[41562]: 5.4 scrub starts 2024-09-10T19:30:33.683 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:30:33 smithi183 ceph-mon[41562]: 5.4 scrub ok 2024-09-10T19:30:33.707 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:30:33.708 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:30:33 smithi191 ceph-mon[32884]: 2.14 scrub starts 2024-09-10T19:30:33.708 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:30:33 smithi191 ceph-mon[32884]: 2.14 scrub ok 2024-09-10T19:30:33.708 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:30:33 smithi191 ceph-mon[32884]: pgmap v940: 158 pgs: 158 active+clean; 3.1 GiB data, 15 GiB used, 611 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.31k op/s 2024-09-10T19:30:33.708 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:30:33 smithi191 ceph-mon[32884]: 2.5 scrub ok 2024-09-10T19:30:33.708 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:30:33 smithi191 ceph-mon[32884]: 5.4 scrub starts 2024-09-10T19:30:33.708 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:30:33 smithi191 ceph-mon[32884]: 5.4 scrub ok 2024-09-10T19:30:33.732 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:30:33.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:30:33.733 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:30:33.834 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:30:33.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:30:33.873 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:30:33.873 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:30:33.974 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:30:34.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:30:34.010 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:30:34.010 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:30:34.111 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:30:34.152 DEBUG:teuthology.orchestra.run.smithi191:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph daemon osd.4 dump_blocked_ops 2024-09-10T19:30:34.156 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:30:34.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:30:34.157 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:30:34.258 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:30:34.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:30:34.299 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:30:34.300 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:30:34.400 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:30:34.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:30:34.440 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:30:34.440 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:30:34.456 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:30:34 smithi183 ceph-mon[29736]: 5.11 scrub starts 2024-09-10T19:30:34.457 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:30:34 smithi183 ceph-mon[29736]: 5.11 scrub ok 2024-09-10T19:30:34.457 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:30:34 smithi183 ceph-mon[29736]: 6.9 deep-scrub starts 2024-09-10T19:30:34.457 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:30:34 smithi183 ceph-mon[29736]: 6.9 deep-scrub ok 2024-09-10T19:30:34.457 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:30:34 smithi183 ceph-mon[41562]: 5.11 scrub starts 2024-09-10T19:30:34.457 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:30:34 smithi183 ceph-mon[41562]: 5.11 scrub ok 2024-09-10T19:30:34.457 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:30:34 smithi183 ceph-mon[41562]: 6.9 deep-scrub starts 2024-09-10T19:30:34.457 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:30:34 smithi183 ceph-mon[41562]: 6.9 deep-scrub ok 2024-09-10T19:30:34.541 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:30:34.582 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:30:34.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:30:34.583 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:30:34.608 INFO:teuthology.orchestra.run.smithi191.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.b/config 2024-09-10T19:30:34.685 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:30:34.715 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:30:34 smithi191 ceph-mon[32884]: 5.11 scrub starts 2024-09-10T19:30:34.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:30:34 smithi191 ceph-mon[32884]: 5.11 scrub ok 2024-09-10T19:30:34.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:30:34 smithi191 ceph-mon[32884]: 6.9 deep-scrub starts 2024-09-10T19:30:34.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:30:34 smithi191 ceph-mon[32884]: 6.9 deep-scrub ok 2024-09-10T19:30:34.718 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:30:34.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:30:34.719 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:30:34.820 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:30:34.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:30:34.858 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:30:34.858 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:30:34.959 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:30:34.998 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:30:34.998 INFO:teuthology.orchestra.run.smithi183.stdout:osd.0: {} 2024-09-10T19:30:34.999 INFO:teuthology.orchestra.run.smithi183.stdout:osd.1: {} 2024-09-10T19:30:34.999 INFO:teuthology.orchestra.run.smithi183.stdout:osd.2: {} 2024-09-10T19:30:34.999 INFO:teuthology.orchestra.run.smithi183.stdout:osd.3: {} 2024-09-10T19:30:34.999 INFO:teuthology.orchestra.run.smithi183.stdout:osd.4: {} 2024-09-10T19:30:34.999 INFO:teuthology.orchestra.run.smithi183.stdout:osd.5: {} 2024-09-10T19:30:34.999 INFO:teuthology.orchestra.run.smithi183.stdout:osd.6: {} 2024-09-10T19:30:34.999 INFO:teuthology.orchestra.run.smithi183.stdout:osd.7: {} 2024-09-10T19:30:34.999 INFO:teuthology.orchestra.run.smithi183.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-10T19:30:35.000 INFO:teuthology.orchestra.run.smithi183.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-10T19:30:35.000 INFO:teuthology.orchestra.run.smithi183.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-10T19:30:35.000 INFO:teuthology.orchestra.run.smithi183.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-10T19:30:35.000 INFO:teuthology.orchestra.run.smithi183.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-10T19:30:35.000 INFO:teuthology.orchestra.run.smithi183.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-10T19:30:35.000 INFO:teuthology.orchestra.run.smithi183.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-10T19:30:35.000 INFO:teuthology.orchestra.run.smithi183.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-10T19:30:35.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:30:35.001 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:30:35.102 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:30:35.129 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:30:35.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:30:35.130 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:30:35.231 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:30:35.264 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:30:35.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:30:35.265 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:30:35.366 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:30:35.398 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:30:35.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:30:35.399 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:30:35.500 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:30:35.532 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:30:35.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:30:35.533 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:30:35.634 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:30:35.666 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:30:35.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:30:35.668 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:30:35.680 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:30:35 smithi183 ceph-mon[29736]: pgmap v941: 158 pgs: 158 active+clean; 3.1 GiB data, 14 GiB used, 611 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.33k op/s 2024-09-10T19:30:35.680 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:30:35 smithi183 ceph-mon[29736]: 4.7 scrub starts 2024-09-10T19:30:35.680 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:30:35 smithi183 ceph-mon[29736]: 4.7 scrub ok 2024-09-10T19:30:35.680 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:30:35 smithi183 ceph-mon[29736]: from='client.? 172.21.15.183:0/1478686982' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T19:30:35.680 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:30:35 smithi183 ceph-mon[29736]: 2.11 deep-scrub starts 2024-09-10T19:30:35.680 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:30:35 smithi183 ceph-mon[29736]: 2.11 deep-scrub ok 2024-09-10T19:30:35.680 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:30:35 smithi183 ceph-mon[29736]: 6.e scrub starts 2024-09-10T19:30:35.680 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:30:35 smithi183 ceph-mon[29736]: 6.e scrub ok 2024-09-10T19:30:35.680 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:30:35 smithi183 ceph-mon[29736]: 2.1e scrub starts 2024-09-10T19:30:35.680 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:30:35 smithi183 ceph-mon[29736]: 2.1e scrub ok 2024-09-10T19:30:35.681 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:30:35 smithi183 ceph-mon[41562]: pgmap v941: 158 pgs: 158 active+clean; 3.1 GiB data, 14 GiB used, 611 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.33k op/s 2024-09-10T19:30:35.681 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:30:35 smithi183 ceph-mon[41562]: 4.7 scrub starts 2024-09-10T19:30:35.681 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:30:35 smithi183 ceph-mon[41562]: 4.7 scrub ok 2024-09-10T19:30:35.681 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:30:35 smithi183 ceph-mon[41562]: from='client.? 172.21.15.183:0/1478686982' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T19:30:35.681 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:30:35 smithi183 ceph-mon[41562]: 2.11 deep-scrub starts 2024-09-10T19:30:35.681 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:30:35 smithi183 ceph-mon[41562]: 2.11 deep-scrub ok 2024-09-10T19:30:35.681 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:30:35 smithi183 ceph-mon[41562]: 6.e scrub starts 2024-09-10T19:30:35.681 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:30:35 smithi183 ceph-mon[41562]: 6.e scrub ok 2024-09-10T19:30:35.681 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:30:35 smithi183 ceph-mon[41562]: 2.1e scrub starts 2024-09-10T19:30:35.681 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:30:35 smithi183 ceph-mon[41562]: 2.1e scrub ok 2024-09-10T19:30:35.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:30:35 smithi191 ceph-mon[32884]: pgmap v941: 158 pgs: 158 active+clean; 3.1 GiB data, 14 GiB used, 611 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.33k op/s 2024-09-10T19:30:35.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:30:35 smithi191 ceph-mon[32884]: 4.7 scrub starts 2024-09-10T19:30:35.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:30:35 smithi191 ceph-mon[32884]: 4.7 scrub ok 2024-09-10T19:30:35.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:30:35 smithi191 ceph-mon[32884]: from='client.? 172.21.15.183:0/1478686982' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T19:30:35.717 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:30:35 smithi191 ceph-mon[32884]: 2.11 deep-scrub starts 2024-09-10T19:30:35.717 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:30:35 smithi191 ceph-mon[32884]: 2.11 deep-scrub ok 2024-09-10T19:30:35.717 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:30:35 smithi191 ceph-mon[32884]: 6.e scrub starts 2024-09-10T19:30:35.717 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:30:35 smithi191 ceph-mon[32884]: 6.e scrub ok 2024-09-10T19:30:35.717 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:30:35 smithi191 ceph-mon[32884]: 2.1e scrub starts 2024-09-10T19:30:35.717 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:30:35 smithi191 ceph-mon[32884]: 2.1e scrub ok 2024-09-10T19:30:35.768 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:30:35.807 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:30:35.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:30:35.808 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:30:35.909 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:30:35.946 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:30:35.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:30:35.947 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:30:36.047 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:30:36.085 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:30:36.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:30:36.086 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:30:36.187 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:30:36.220 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:30:36.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:30:36.221 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:30:36.321 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:30:36.354 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:30:36.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:30:36.355 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:30:36.381 INFO:teuthology.orchestra.run.smithi183.stderr:reweighted osd.6 to 1 (10000) 2024-09-10T19:30:36.456 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:30:36.504 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:30:36.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:30:36.506 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:30:36.567 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:30:36 smithi191 ceph-mon[32884]: 5.7 scrub starts 2024-09-10T19:30:36.568 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:30:36 smithi191 ceph-mon[32884]: 5.7 scrub ok 2024-09-10T19:30:36.568 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:30:36 smithi191 ceph-mon[32884]: pgmap v942: 158 pgs: 158 active+clean; 2.7 GiB data, 13 GiB used, 612 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.45k op/s 2024-09-10T19:30:36.568 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:30:36 smithi191 ceph-mon[32884]: from='client.? 172.21.15.183:0/1313748957' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-09-10T19:30:36.568 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:30:36 smithi191 ceph-mon[32884]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-09-10T19:30:36.596 INFO:teuthology.orchestra.run.smithi191.stdout:ERROR: (22) Invalid argument 2024-09-10T19:30:36.596 INFO:teuthology.orchestra.run.smithi191.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-10T19:30:36.607 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:30:36.645 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:30:36.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:30:36.646 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:30:36.681 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:30:36 smithi183 ceph-mon[29736]: 5.7 scrub starts 2024-09-10T19:30:36.681 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:30:36 smithi183 ceph-mon[29736]: 5.7 scrub ok 2024-09-10T19:30:36.681 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:30:36 smithi183 ceph-mon[29736]: pgmap v942: 158 pgs: 158 active+clean; 2.7 GiB data, 13 GiB used, 612 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.45k op/s 2024-09-10T19:30:36.682 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:30:36 smithi183 ceph-mon[29736]: from='client.? 172.21.15.183:0/1313748957' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-09-10T19:30:36.682 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:30:36 smithi183 ceph-mon[29736]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-09-10T19:30:36.682 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:30:36 smithi183 ceph-mon[41562]: 5.7 scrub starts 2024-09-10T19:30:36.682 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:30:36 smithi183 ceph-mon[41562]: 5.7 scrub ok 2024-09-10T19:30:36.682 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:30:36 smithi183 ceph-mon[41562]: pgmap v942: 158 pgs: 158 active+clean; 2.7 GiB data, 13 GiB used, 612 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.45k op/s 2024-09-10T19:30:36.682 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:30:36 smithi183 ceph-mon[41562]: from='client.? 172.21.15.183:0/1313748957' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-09-10T19:30:36.682 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:30:36 smithi183 ceph-mon[41562]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-09-10T19:30:36.747 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:30:36.781 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:30:36.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:30:36.782 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:30:36.883 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:30:36.915 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph osd set noscrub 2024-09-10T19:30:36.919 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:30:36.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:30:36.920 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:30:37.021 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:30:37.055 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:30:37.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:30:37.056 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:30:37.117 INFO:tasks.cephadm.ceph_manager.ceph:waiting for recovery to complete 2024-09-10T19:30:37.117 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph pg dump --format=json 2024-09-10T19:30:37.158 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:30:37.188 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:30:37.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:30:37.189 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:30:37.290 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:30:37.321 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:30:37.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:30:37.323 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:30:37.363 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:30:37.423 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:30:37.462 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:30:37.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:30:37.463 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:30:37.564 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:30:37.599 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:30:37.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:30:37.600 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:30:37.669 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:30:37 smithi183 ceph-mon[29736]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-09-10T19:30:37.669 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:30:37 smithi183 ceph-mon[29736]: osdmap e417: 8 total, 8 up, 7 in 2024-09-10T19:30:37.669 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:30:37 smithi183 ceph-mon[41562]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-09-10T19:30:37.670 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:30:37 smithi183 ceph-mon[41562]: osdmap e417: 8 total, 8 up, 7 in 2024-09-10T19:30:37.701 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:30:37.714 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-10T19:30:37.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:30:37 smithi191 ceph-mon[32884]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-09-10T19:30:37.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:30:37 smithi191 ceph-mon[32884]: osdmap e417: 8 total, 8 up, 7 in 2024-09-10T19:30:37.717 INFO:journalctl@ceph.mgr.x.smithi191.stdout:Sep 10 19:30:37 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-x[34967]: ::ffff:172.21.15.191 - - [10/Sep/2024:19:30:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T19:30:37.731 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:30:37.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:30:37.732 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:30:37.833 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:30:37.859 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:30:37.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:30:37.860 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:30:37.961 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:30:37.992 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:30:37.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:30:37.993 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:30:38.094 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:30:38.128 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:30:38.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:30:38.129 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:30:38.230 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:30:38.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:30:38.267 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:30:38.267 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:30:38.368 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:30:38.418 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:30:38.422 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:30:38.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:30:38.423 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:30:38.524 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:30:38.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:30:38.565 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:30:38.565 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:30:38.666 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:30:38.681 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:30:38 smithi183 ceph-mon[29736]: 4.4 scrub starts 2024-09-10T19:30:38.681 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:30:38 smithi183 ceph-mon[29736]: 4.4 scrub ok 2024-09-10T19:30:38.682 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:30:38 smithi183 ceph-mon[29736]: 6.0 scrub starts 2024-09-10T19:30:38.682 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:30:38 smithi183 ceph-mon[29736]: 6.0 scrub ok 2024-09-10T19:30:38.682 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:30:38 smithi183 ceph-mon[29736]: pgmap v944: 158 pgs: 158 active+clean; 2.6 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.72k op/s 2024-09-10T19:30:38.682 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:30:38 smithi183 ceph-mon[41562]: 4.4 scrub starts 2024-09-10T19:30:38.682 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:30:38 smithi183 ceph-mon[41562]: 4.4 scrub ok 2024-09-10T19:30:38.682 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:30:38 smithi183 ceph-mon[41562]: 6.0 scrub starts 2024-09-10T19:30:38.682 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:30:38 smithi183 ceph-mon[41562]: 6.0 scrub ok 2024-09-10T19:30:38.682 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:30:38 smithi183 ceph-mon[41562]: pgmap v944: 158 pgs: 158 active+clean; 2.6 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.72k op/s 2024-09-10T19:30:38.708 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:30:38.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:30:38.710 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:30:38.715 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:30:38 smithi191 ceph-mon[32884]: 4.4 scrub starts 2024-09-10T19:30:38.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:30:38 smithi191 ceph-mon[32884]: 4.4 scrub ok 2024-09-10T19:30:38.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:30:38 smithi191 ceph-mon[32884]: 6.0 scrub starts 2024-09-10T19:30:38.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:30:38 smithi191 ceph-mon[32884]: 6.0 scrub ok 2024-09-10T19:30:38.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:30:38 smithi191 ceph-mon[32884]: pgmap v944: 158 pgs: 158 active+clean; 2.6 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.72k op/s 2024-09-10T19:30:38.811 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:30:38.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:30:38.843 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:30:38.843 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:30:38.944 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:30:38.987 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:30:38.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:30:38.988 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:30:38.990 DEBUG:teuthology.orchestra.run.smithi191:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph daemon osd.4 dump_historic_ops 2024-09-10T19:30:39.018 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:30:39.090 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:30:39.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:30:39.138 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:30:39.139 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:30:39.239 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:30:39.261 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:30:39.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:30:39.262 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:30:39.363 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:30:39.394 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:30:39.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:30:39.396 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:30:39.426 INFO:teuthology.orchestra.run.smithi191.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.b/config 2024-09-10T19:30:39.497 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:30:39.536 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:30:39.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:30:39.537 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:30:39.638 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:30:39.678 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:30:39.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:30:39.679 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:30:39.780 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:30:39.815 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:30:39.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:30:39.816 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:30:39.917 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:30:39.948 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:30:39.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:30:39.949 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:30:40.050 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:30:40.074 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:30:40.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:30:40.075 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:30:40.176 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:30:40.212 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:30:40.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:30:40.213 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:30:40.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:30:40 smithi191 ceph-mon[32884]: 5.1c deep-scrub starts 2024-09-10T19:30:40.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:30:40 smithi191 ceph-mon[32884]: 5.1c deep-scrub ok 2024-09-10T19:30:40.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:30:40 smithi191 ceph-mon[32884]: 1.9 scrub starts 2024-09-10T19:30:40.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:30:40 smithi191 ceph-mon[32884]: 1.9 scrub ok 2024-09-10T19:30:40.216 INFO:journalctl@ceph.iscsi.iscsi.a.smithi191.stdout:Sep 10 19:30:40 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-iscsi-iscsi-a[61816]: debug there is no tcmu-runner data available 2024-09-10T19:30:40.313 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:30:40.353 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:30:40.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:30:40.354 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:30:40.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:30:40 smithi183 ceph-mon[29736]: 5.1c deep-scrub starts 2024-09-10T19:30:40.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:30:40 smithi183 ceph-mon[29736]: 5.1c deep-scrub ok 2024-09-10T19:30:40.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:30:40 smithi183 ceph-mon[29736]: 1.9 scrub starts 2024-09-10T19:30:40.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:30:40 smithi183 ceph-mon[29736]: 1.9 scrub ok 2024-09-10T19:30:40.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:30:40 smithi183 ceph-mon[41562]: 5.1c deep-scrub starts 2024-09-10T19:30:40.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:30:40 smithi183 ceph-mon[41562]: 5.1c deep-scrub ok 2024-09-10T19:30:40.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:30:40 smithi183 ceph-mon[41562]: 1.9 scrub starts 2024-09-10T19:30:40.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:30:40 smithi183 ceph-mon[41562]: 1.9 scrub ok 2024-09-10T19:30:40.455 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:30:40.488 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:30:40.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:30:40.489 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:30:40.590 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:30:40.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:30:40.630 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:30:40.630 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:30:40.731 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:30:40.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:30:40.764 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:30:40.764 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:30:40.865 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:30:40.899 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:30:40.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:30:40.900 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:30:41.001 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:30:41.031 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:30:41.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:30:41.032 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:30:41.133 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:30:41.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:30:41.181 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:30:41.181 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:30:41.283 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:30:41.317 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:30:41 smithi183 ceph-mon[29736]: 3.1 deep-scrub starts 2024-09-10T19:30:41.317 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:30:41 smithi183 ceph-mon[29736]: 3.1 deep-scrub ok 2024-09-10T19:30:41.317 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:30:41 smithi183 ceph-mon[29736]: 5.0 deep-scrub starts 2024-09-10T19:30:41.317 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:30:41 smithi183 ceph-mon[29736]: 5.0 deep-scrub ok 2024-09-10T19:30:41.317 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:30:41 smithi183 ceph-mon[29736]: pgmap v945: 158 pgs: 158 active+clean; 2.5 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.12k op/s 2024-09-10T19:30:41.317 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:30:41 smithi183 ceph-mon[29736]: from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:30:41.317 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:30:41 smithi183 ceph-mon[29736]: 1.3 scrub starts 2024-09-10T19:30:41.317 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:30:41 smithi183 ceph-mon[29736]: 1.3 scrub ok 2024-09-10T19:30:41.317 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:30:41 smithi183 ceph-mon[29736]: 4.1b scrub starts 2024-09-10T19:30:41.317 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:30:41 smithi183 ceph-mon[29736]: 4.1b scrub ok 2024-09-10T19:30:41.318 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:30:41 smithi183 ceph-mon[41562]: 3.1 deep-scrub starts 2024-09-10T19:30:41.318 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:30:41 smithi183 ceph-mon[41562]: 3.1 deep-scrub ok 2024-09-10T19:30:41.318 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:30:41 smithi183 ceph-mon[41562]: 5.0 deep-scrub starts 2024-09-10T19:30:41.318 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:30:41 smithi183 ceph-mon[41562]: 5.0 deep-scrub ok 2024-09-10T19:30:41.318 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:30:41 smithi183 ceph-mon[41562]: pgmap v945: 158 pgs: 158 active+clean; 2.5 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.12k op/s 2024-09-10T19:30:41.318 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:30:41 smithi183 ceph-mon[41562]: from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:30:41.318 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:30:41 smithi183 ceph-mon[41562]: 1.3 scrub starts 2024-09-10T19:30:41.318 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:30:41 smithi183 ceph-mon[41562]: 1.3 scrub ok 2024-09-10T19:30:41.318 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:30:41 smithi183 ceph-mon[41562]: 4.1b scrub starts 2024-09-10T19:30:41.318 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:30:41 smithi183 ceph-mon[41562]: 4.1b scrub ok 2024-09-10T19:30:41.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:30:41.320 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:30:41.321 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:30:41.345 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:30:41 smithi191 ceph-mon[32884]: 3.1 deep-scrub starts 2024-09-10T19:30:41.345 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:30:41 smithi191 ceph-mon[32884]: 3.1 deep-scrub ok 2024-09-10T19:30:41.346 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:30:41 smithi191 ceph-mon[32884]: 5.0 deep-scrub starts 2024-09-10T19:30:41.346 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:30:41 smithi191 ceph-mon[32884]: 5.0 deep-scrub ok 2024-09-10T19:30:41.346 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:30:41 smithi191 ceph-mon[32884]: pgmap v945: 158 pgs: 158 active+clean; 2.5 GiB data, 13 GiB used, 613 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.12k op/s 2024-09-10T19:30:41.346 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:30:41 smithi191 ceph-mon[32884]: from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:30:41.346 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:30:41 smithi191 ceph-mon[32884]: 1.3 scrub starts 2024-09-10T19:30:41.346 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:30:41 smithi191 ceph-mon[32884]: 1.3 scrub ok 2024-09-10T19:30:41.346 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:30:41 smithi191 ceph-mon[32884]: 4.1b scrub starts 2024-09-10T19:30:41.346 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:30:41 smithi191 ceph-mon[32884]: 4.1b scrub ok 2024-09-10T19:30:41.422 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:30:41.460 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:30:41.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:30:41.462 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:30:41.562 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:30:41.602 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:30:41.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:30:41.603 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:30:41.682 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Sep 10 19:30:41 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-y[29989]: ::ffff:172.21.15.191 - - [10/Sep/2024:19:30:41] "GET /metrics HTTP/1.1" 200 224934 "" "Prometheus/2.43.0" 2024-09-10T19:30:41.704 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:30:41.736 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:30:41.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:30:41.737 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:30:41.838 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:30:41.873 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:30:41.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:30:41.874 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:30:41.891 INFO:teuthology.orchestra.run.smithi191.stdout:ERROR: (22) Invalid argument 2024-09-10T19:30:41.891 INFO:teuthology.orchestra.run.smithi191.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-10T19:30:41.975 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:30:42.003 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:30:42.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:30:42.004 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:30:42.076 INFO:teuthology.orchestra.run.smithi183.stderr:noscrub is set 2024-09-10T19:30:42.105 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:30:42.140 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:30:42.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:30:42.141 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:30:42.242 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:30:42.278 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:30:42.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:30:42.279 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:30:42.379 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:30:42.410 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:30:42.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:30:42.411 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:30:42.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:30:42 smithi183 ceph-mon[29736]: 11.3 scrub starts 2024-09-10T19:30:42.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:30:42 smithi183 ceph-mon[29736]: from='client.? 172.21.15.183:0/1553583749' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-10T19:30:42.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:30:42 smithi183 ceph-mon[29736]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-10T19:30:42.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:30:42 smithi183 ceph-mon[29736]: 11.3 scrub ok 2024-09-10T19:30:42.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:30:42 smithi183 ceph-mon[29736]: 5.18 scrub starts 2024-09-10T19:30:42.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:30:42 smithi183 ceph-mon[29736]: 5.18 scrub ok 2024-09-10T19:30:42.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:30:42 smithi183 ceph-mon[41562]: 11.3 scrub starts 2024-09-10T19:30:42.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:30:42 smithi183 ceph-mon[41562]: from='client.? 172.21.15.183:0/1553583749' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-10T19:30:42.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:30:42 smithi183 ceph-mon[41562]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-10T19:30:42.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:30:42 smithi183 ceph-mon[41562]: 11.3 scrub ok 2024-09-10T19:30:42.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:30:42 smithi183 ceph-mon[41562]: 5.18 scrub starts 2024-09-10T19:30:42.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:30:42 smithi183 ceph-mon[41562]: 5.18 scrub ok 2024-09-10T19:30:42.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:30:42 smithi191 ceph-mon[32884]: 11.3 scrub starts 2024-09-10T19:30:42.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:30:42 smithi191 ceph-mon[32884]: from='client.? 172.21.15.183:0/1553583749' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-10T19:30:42.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:30:42 smithi191 ceph-mon[32884]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-10T19:30:42.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:30:42 smithi191 ceph-mon[32884]: 11.3 scrub ok 2024-09-10T19:30:42.467 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:30:42 smithi191 ceph-mon[32884]: 5.18 scrub starts 2024-09-10T19:30:42.467 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:30:42 smithi191 ceph-mon[32884]: 5.18 scrub ok 2024-09-10T19:30:42.512 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:30:42.544 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:30:42.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:30:42.545 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:30:42.604 DEBUG:teuthology.orchestra.run.smithi191:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph daemon osd.5 dump_ops_in_flight 2024-09-10T19:30:42.646 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:30:42.697 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:30:42.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:30:42.698 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:30:42.799 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:30:42.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:30:42.841 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:30:42.841 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:30:42.942 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:30:42.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:30:42.976 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:30:42.976 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:30:43.041 INFO:teuthology.orchestra.run.smithi191.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.b/config 2024-09-10T19:30:43.078 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:30:43.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:30:43.119 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:30:43.119 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:30:43.220 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:30:43.241 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:30:43.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:30:43.242 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:30:43.343 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:30:43.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:30:43.377 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:30:43.377 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:30:43.428 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:30:43 smithi191 ceph-mon[32884]: 11.1 scrub starts 2024-09-10T19:30:43.428 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:30:43 smithi191 ceph-mon[32884]: 6.18 deep-scrub starts 2024-09-10T19:30:43.428 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:30:43 smithi191 ceph-mon[32884]: 6.18 deep-scrub ok 2024-09-10T19:30:43.428 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:30:43 smithi191 ceph-mon[32884]: pgmap v946: 158 pgs: 158 active+clean; 2.2 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.47k op/s 2024-09-10T19:30:43.428 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:30:43 smithi191 ceph-mon[32884]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T19:30:43.428 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:30:43 smithi191 ceph-mon[32884]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-10T19:30:43.428 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:30:43 smithi191 ceph-mon[32884]: osdmap e418: 8 total, 8 up, 7 in 2024-09-10T19:30:43.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:30:43 smithi183 ceph-mon[29736]: 11.1 scrub starts 2024-09-10T19:30:43.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:30:43 smithi183 ceph-mon[29736]: 6.18 deep-scrub starts 2024-09-10T19:30:43.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:30:43 smithi183 ceph-mon[29736]: 6.18 deep-scrub ok 2024-09-10T19:30:43.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:30:43 smithi183 ceph-mon[29736]: pgmap v946: 158 pgs: 158 active+clean; 2.2 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.47k op/s 2024-09-10T19:30:43.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:30:43 smithi183 ceph-mon[29736]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T19:30:43.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:30:43 smithi183 ceph-mon[29736]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-10T19:30:43.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:30:43 smithi183 ceph-mon[29736]: osdmap e418: 8 total, 8 up, 7 in 2024-09-10T19:30:43.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:30:43 smithi183 ceph-mon[41562]: 11.1 scrub starts 2024-09-10T19:30:43.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:30:43 smithi183 ceph-mon[41562]: 6.18 deep-scrub starts 2024-09-10T19:30:43.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:30:43 smithi183 ceph-mon[41562]: 6.18 deep-scrub ok 2024-09-10T19:30:43.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:30:43 smithi183 ceph-mon[41562]: pgmap v946: 158 pgs: 158 active+clean; 2.2 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.47k op/s 2024-09-10T19:30:43.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:30:43 smithi183 ceph-mon[41562]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T19:30:43.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:30:43 smithi183 ceph-mon[41562]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-10T19:30:43.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:30:43 smithi183 ceph-mon[41562]: osdmap e418: 8 total, 8 up, 7 in 2024-09-10T19:30:43.478 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:30:43.514 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:30:43.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:30:43.515 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:30:43.616 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:30:43.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:30:43.648 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:30:43.648 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:30:43.749 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:30:43.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:30:43.784 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:30:43.784 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:30:43.885 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:30:43.921 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:30:43.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:30:43.922 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:30:44.023 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:30:44.061 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:30:44.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:30:44.063 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:30:44.109 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-09-10T19:30:44.112 INFO:teuthology.orchestra.run.smithi183.stderr:dumped all 2024-09-10T19:30:44.164 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:30:44.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:30:44.199 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:30:44.200 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:30:44.300 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:30:44.324 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:30:44.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:30:44.325 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:30:44.425 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:30:44.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:30:44 smithi183 ceph-mon[29736]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:30:44.431 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:30:44 smithi183 ceph-mon[41562]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:30:44.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:30:44.457 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:30:44.457 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:30:44.465 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:30:44 smithi191 ceph-mon[32884]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:30:44.558 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:30:44.591 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:30:44.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:30:44.592 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:30:44.693 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:30:44.727 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:30:44.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:30:44.728 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:30:44.829 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:30:44.859 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:30:44.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:30:44.860 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:30:44.960 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:30:44.983 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:30:44.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:30:44.984 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:30:45.006 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph osd set nodeep-scrub 2024-09-10T19:30:45.085 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:30:45.126 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:30:45.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:30:45.127 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:30:45.184 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:30:45 smithi183 ceph-mon[29736]: pgmap v948: 158 pgs: 158 active+clean; 2.2 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.09k op/s 2024-09-10T19:30:45.184 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:30:45 smithi183 ceph-mon[29736]: from='client.25795 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-10T19:30:45.184 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:30:45 smithi183 ceph-mon[41562]: pgmap v948: 158 pgs: 158 active+clean; 2.2 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.09k op/s 2024-09-10T19:30:45.184 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:30:45 smithi183 ceph-mon[41562]: from='client.25795 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-10T19:30:45.228 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:30:45.247 INFO:teuthology.orchestra.run.smithi183.stdout:{"pg_ready":true,"pg_map":{"version":948,"stamp":"2024-09-10T19:30:43.985365+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":2314803444,"num_objects":35509,"num_object_clones":0,"num_object_copies":106527,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35509,"num_whiteouts":0,"num_read":1968,"num_read_kb":1720,"num_write":255465,"num_write_kb":9272812,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":458,"num_bytes_recovered":536482,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72339,"ondisk_log_size":72339,"up":474,"acting":474,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":474,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":656216064,"kb_used":12260080,"kb_used_data":6829044,"kb_used_omap":19,"kb_used_meta":5430380,"kb_avail":643955984,"statfs":{"total":671965249536,"available":659410927616,"internally_reserved":0,"allocated":6992941056,"data_stored":6990540045,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19457,"internal_metadata":5560710143},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,2,2],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-569376768,"num_objects":-8688,"num_object_clones":0,"num_object_copies":-26064,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-8688,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":8688,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002518"},"pg_stats":[{"pgid":"6.1b","version":"0'0","reported_seq":163,"reported_epoch":417,"state":"active+clean","last_fresh":"2024-09-10T19:30:36.370569+0000","last_change":"2024-09-10T19:29:35.538355+0000","last_active":"2024-09-10T19:30:36.370569+0000","last_peered":"2024-09-10T19:30:36.370569+0000","last_clean":"2024-09-10T19:30:36.370569+0000","last_became_active":"2024-09-10T19:27:52.994757+0000","last_became_peered":"2024-09-10T19:27:52.994757+0000","last_unstale":"2024-09-10T19:30:36.370569+0000","last_undegraded":"2024-09-10T19:30:36.370569+0000","last_fullsized":"2024-09-10T19:30:36.370569+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:29:35.538284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:28:18.455084+0000","last_clean_scrub_stamp":"2024-09-10T19:29:35.538284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:30:57.495675+0000","scrub_duration":0.0025697599999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6],"acting":[0,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":411,"reported_epoch":417,"state":"active+clean","last_fresh":"2024-09-10T19:30:36.373072+0000","last_change":"2024-09-10T19:29:32.579218+0000","last_active":"2024-09-10T19:30:36.373072+0000","last_peered":"2024-09-10T19:30:36.373072+0000","last_clean":"2024-09-10T19:30:36.373072+0000","last_became_active":"2024-09-10T19:24:23.892621+0000","last_became_peered":"2024-09-10T19:24:23.892621+0000","last_unstale":"2024-09-10T19:30:36.373072+0000","last_undegraded":"2024-09-10T19:30:36.373072+0000","last_fullsized":"2024-09-10T19:30:36.373072+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:29:32.579092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:16:38.194472+0000","last_clean_scrub_stamp":"2024-09-10T19:29:32.579092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:30:51.229113+0000","scrub_duration":0.010689732,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.19","version":"414'39","reported_seq":529,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-09-10T19:30:31.328843+0000","last_change":"2024-09-10T19:30:09.389815+0000","last_active":"2024-09-10T19:30:31.328843+0000","last_peered":"2024-09-10T19:30:31.328843+0000","last_clean":"2024-09-10T19:30:31.328843+0000","last_became_active":"2024-09-10T19:24:23.906592+0000","last_became_peered":"2024-09-10T19:24:23.906592+0000","last_unstale":"2024-09-10T19:30:31.328843+0000","last_undegraded":"2024-09-10T19:30:31.328843+0000","last_fullsized":"2024-09-10T19:30:31.328843+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"237'27","last_scrub_stamp":"2024-09-10T19:30:09.389670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:30:09.389670+0000","objects_scrubbed":7,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:31:38.619896+0000","scrub_duration":0.003161433,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":62,"num_read_kb":53,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":495,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-09-10T19:30:31.332314+0000","last_change":"2024-09-10T19:29:07.181969+0000","last_active":"2024-09-10T19:30:31.332314+0000","last_peered":"2024-09-10T19:30:31.332314+0000","last_clean":"2024-09-10T19:30:31.332314+0000","last_became_active":"2024-09-10T19:27:53.006175+0000","last_became_peered":"2024-09-10T19:27:53.006175+0000","last_unstale":"2024-09-10T19:30:31.332314+0000","last_undegraded":"2024-09-10T19:30:31.332314+0000","last_fullsized":"2024-09-10T19:30:31.332314+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:29:07.181888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:19:28.498560+0000","last_clean_scrub_stamp":"2024-09-10T19:29:07.181888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0065255809999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"384'2","reported_seq":473,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-09-10T19:30:34.918386+0000","last_change":"2024-09-10T19:30:34.918319+0000","last_active":"2024-09-10T19:30:34.918386+0000","last_peered":"2024-09-10T19:30:34.918386+0000","last_clean":"2024-09-10T19:30:34.918386+0000","last_became_active":"2024-09-10T19:24:23.908614+0000","last_became_peered":"2024-09-10T19:24:23.908614+0000","last_unstale":"2024-09-10T19:30:34.918386+0000","last_undegraded":"2024-09-10T19:30:34.918386+0000","last_fullsized":"2024-09-10T19:30:34.918386+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"384'2","last_scrub_stamp":"2024-09-10T19:30:34.918244+0000","last_deep_scrub":"384'2","last_deep_scrub_stamp":"2024-09-10T19:28:23.240824+0000","last_clean_scrub_stamp":"2024-09-10T19:30:34.918244+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:31:35.083745+0000","scrub_duration":0.0051432630000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":874,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"414'25","reported_seq":572,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-09-10T19:30:31.332435+0000","last_change":"2024-09-10T19:30:24.549021+0000","last_active":"2024-09-10T19:30:31.332435+0000","last_peered":"2024-09-10T19:30:31.332435+0000","last_clean":"2024-09-10T19:30:31.332435+0000","last_became_active":"2024-09-10T19:27:53.013509+0000","last_became_peered":"2024-09-10T19:27:53.013509+0000","last_unstale":"2024-09-10T19:30:31.332435+0000","last_undegraded":"2024-09-10T19:30:31.332435+0000","last_fullsized":"2024-09-10T19:30:31.332435+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"414'25","last_scrub_stamp":"2024-09-10T19:30:24.548931+0000","last_deep_scrub":"384'19","last_deep_scrub_stamp":"2024-09-10T19:28:21.799921+0000","last_clean_scrub_stamp":"2024-09-10T19:30:24.548931+0000","objects_scrubbed":4,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:31:44.970028+0000","scrub_duration":0.0074165239999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":45,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-09-10T19:30:31.335429+0000","last_change":"2024-09-10T19:30:06.545197+0000","last_active":"2024-09-10T19:30:31.335429+0000","last_peered":"2024-09-10T19:30:31.335429+0000","last_clean":"2024-09-10T19:30:31.335429+0000","last_became_active":"2024-09-10T19:28:47.151481+0000","last_became_peered":"2024-09-10T19:28:47.151481+0000","last_unstale":"2024-09-10T19:30:31.335429+0000","last_undegraded":"2024-09-10T19:30:31.335429+0000","last_fullsized":"2024-09-10T19:30:31.335429+0000","mapping_epoch":391,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":392,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:30:06.545123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:30:06.545123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:31:19.928036+0000","scrub_duration":0.0025621239999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,7],"acting":[6,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":473,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-09-10T19:30:31.332815+0000","last_change":"2024-09-10T19:30:16.469108+0000","last_active":"2024-09-10T19:30:31.332815+0000","last_peered":"2024-09-10T19:30:31.332815+0000","last_clean":"2024-09-10T19:30:31.332815+0000","last_became_active":"2024-09-10T19:24:01.971921+0000","last_became_peered":"2024-09-10T19:24:01.971921+0000","last_unstale":"2024-09-10T19:30:31.332815+0000","last_undegraded":"2024-09-10T19:30:31.332815+0000","last_fullsized":"2024-09-10T19:30:31.332815+0000","mapping_epoch":292,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":293,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:30:16.469056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:30:16.469056+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:31:23.825602+0000","scrub_duration":0.001491469,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":474,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-09-10T19:30:31.345993+0000","last_change":"2024-09-10T19:29:11.890872+0000","last_active":"2024-09-10T19:30:31.345993+0000","last_peered":"2024-09-10T19:30:31.345993+0000","last_clean":"2024-09-10T19:30:31.345993+0000","last_became_active":"2024-09-10T19:27:53.011202+0000","last_became_peered":"2024-09-10T19:27:53.011202+0000","last_unstale":"2024-09-10T19:30:31.345993+0000","last_undegraded":"2024-09-10T19:30:31.345993+0000","last_fullsized":"2024-09-10T19:30:31.345993+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:29:11.890794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:29:11.890794+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:30:38.401815+0000","scrub_duration":0.002093588,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1b","version":"414'9","reported_seq":508,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-09-10T19:30:31.332880+0000","last_change":"2024-09-10T19:29:14.194751+0000","last_active":"2024-09-10T19:30:31.332880+0000","last_peered":"2024-09-10T19:30:31.332880+0000","last_clean":"2024-09-10T19:30:31.332880+0000","last_became_active":"2024-09-10T19:24:01.974813+0000","last_became_peered":"2024-09-10T19:24:01.974813+0000","last_unstale":"2024-09-10T19:30:31.332880+0000","last_undegraded":"2024-09-10T19:30:31.332880+0000","last_fullsized":"2024-09-10T19:30:31.332880+0000","mapping_epoch":292,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":293,"parent":"0.0","parent_split_bits":0,"last_scrub":"237'7","last_scrub_stamp":"2024-09-10T19:29:14.194644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:29:14.194644+0000","objects_scrubbed":2,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.005958596,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":17,"num_read_kb":13,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":467,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-09-10T19:30:31.341004+0000","last_change":"2024-09-10T19:30:19.910559+0000","last_active":"2024-09-10T19:30:31.341004+0000","last_peered":"2024-09-10T19:30:31.341004+0000","last_clean":"2024-09-10T19:30:31.341004+0000","last_became_active":"2024-09-10T19:24:01.976587+0000","last_became_peered":"2024-09-10T19:24:01.976587+0000","last_unstale":"2024-09-10T19:30:31.341004+0000","last_undegraded":"2024-09-10T19:30:31.341004+0000","last_fullsized":"2024-09-10T19:30:31.341004+0000","mapping_epoch":292,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":293,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:30:19.910433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:30:19.910433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:31:47.412483+0000","scrub_duration":0.0056012320000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":468,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-09-10T19:30:31.331733+0000","last_change":"2024-09-10T19:30:08.321635+0000","last_active":"2024-09-10T19:30:31.331733+0000","last_peered":"2024-09-10T19:30:31.331733+0000","last_clean":"2024-09-10T19:30:31.331733+0000","last_became_active":"2024-09-10T19:24:01.970118+0000","last_became_peered":"2024-09-10T19:24:01.970118+0000","last_unstale":"2024-09-10T19:30:31.331733+0000","last_undegraded":"2024-09-10T19:30:31.331733+0000","last_fullsized":"2024-09-10T19:30:31.331733+0000","mapping_epoch":292,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":293,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:30:08.321561+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:30:08.321561+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:31:28.385750+0000","scrub_duration":0.0054793990000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":477,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-09-10T19:30:31.342079+0000","last_change":"2024-09-10T19:29:31.166120+0000","last_active":"2024-09-10T19:30:31.342079+0000","last_peered":"2024-09-10T19:30:31.342079+0000","last_clean":"2024-09-10T19:30:31.342079+0000","last_became_active":"2024-09-10T19:27:53.008555+0000","last_became_peered":"2024-09-10T19:27:53.008555+0000","last_unstale":"2024-09-10T19:30:31.342079+0000","last_undegraded":"2024-09-10T19:30:31.342079+0000","last_fullsized":"2024-09-10T19:30:31.342079+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:29:31.166021+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:29:31.166021+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:30:52.918293+0000","scrub_duration":0.012493738000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1a","version":"414'21","reported_seq":488,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-09-10T19:30:31.333078+0000","last_change":"2024-09-10T19:30:17.531183+0000","last_active":"2024-09-10T19:30:31.333078+0000","last_peered":"2024-09-10T19:30:31.333078+0000","last_clean":"2024-09-10T19:30:31.333078+0000","last_became_active":"2024-09-10T19:24:23.891213+0000","last_became_peered":"2024-09-10T19:24:23.891213+0000","last_unstale":"2024-09-10T19:30:31.333078+0000","last_undegraded":"2024-09-10T19:30:31.333078+0000","last_fullsized":"2024-09-10T19:30:31.333078+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"414'21","last_scrub_stamp":"2024-09-10T19:30:17.531099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:30:17.531099+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:31:44.519993+0000","scrub_duration":0.097305297999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":460,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-09-10T19:30:31.334225+0000","last_change":"2024-09-10T19:30:29.198545+0000","last_active":"2024-09-10T19:30:31.334225+0000","last_peered":"2024-09-10T19:30:31.334225+0000","last_clean":"2024-09-10T19:30:31.334225+0000","last_became_active":"2024-09-10T19:24:23.907653+0000","last_became_peered":"2024-09-10T19:24:23.907653+0000","last_unstale":"2024-09-10T19:30:31.334225+0000","last_undegraded":"2024-09-10T19:30:31.334225+0000","last_fullsized":"2024-09-10T19:30:31.334225+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:30:29.198451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:30:29.198451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:31:31.812451+0000","scrub_duration":0.0023167890000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":434,"reported_epoch":417,"state":"active+clean","last_fresh":"2024-09-10T19:30:41.905647+0000","last_change":"2024-09-10T19:30:41.905595+0000","last_active":"2024-09-10T19:30:41.905647+0000","last_peered":"2024-09-10T19:30:41.905647+0000","last_clean":"2024-09-10T19:30:41.905647+0000","last_became_active":"2024-09-10T19:24:24.907431+0000","last_became_peered":"2024-09-10T19:24:24.907431+0000","last_unstale":"2024-09-10T19:30:41.905647+0000","last_undegraded":"2024-09-10T19:30:41.905647+0000","last_fullsized":"2024-09-10T19:30:41.905647+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:30:41.905516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:30:41.905516+0000","last_clean_scrub_stamp":"2024-09-10T19:30:41.905516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:32:11.300289+0000","scrub_duration":0.002437481,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.2","version":"417'15756","reported_seq":15984,"reported_epoch":417,"state":"active+clean","last_fresh":"2024-09-10T19:30:40.332109+0000","last_change":"2024-09-10T19:30:18.096594+0000","last_active":"2024-09-10T19:30:40.332109+0000","last_peered":"2024-09-10T19:30:40.332109+0000","last_clean":"2024-09-10T19:30:40.332109+0000","last_became_active":"2024-09-10T19:27:46.703515+0000","last_became_peered":"2024-09-10T19:27:46.703515+0000","last_unstale":"2024-09-10T19:30:40.332109+0000","last_undegraded":"2024-09-10T19:30:40.332109+0000","last_fullsized":"2024-09-10T19:30:40.332109+0000","mapping_epoch":371,"log_start":"407'11600","ondisk_log_start":"407'11600","created":371,"last_epoch_clean":372,"parent":"0.0","parent_split_bits":0,"last_scrub":"415'13895","last_scrub_stamp":"2024-09-10T19:30:18.096518+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:27:45.679464+0000","last_clean_scrub_stamp":"2024-09-10T19:30:18.096518+0000","objects_scrubbed":4109,"log_size":4156,"log_dups_size":0,"ondisk_log_size":4156,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:31:35.525537+0000","scrub_duration":1.59077157,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":142999552,"num_objects":2182,"num_object_clones":0,"num_object_copies":6546,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2182,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15756,"num_write_kb":574016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,7],"acting":[3,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"74'79","reported_seq":265,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-09-10T19:30:31.331321+0000","last_change":"2024-09-10T19:30:28.196155+0000","last_active":"2024-09-10T19:30:31.331321+0000","last_peered":"2024-09-10T19:30:31.331321+0000","last_clean":"2024-09-10T19:30:31.331321+0000","last_became_active":"2024-09-10T19:27:53.004335+0000","last_became_peered":"2024-09-10T19:27:53.004335+0000","last_unstale":"2024-09-10T19:30:31.331321+0000","last_undegraded":"2024-09-10T19:30:31.331321+0000","last_fullsized":"2024-09-10T19:30:31.331321+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":4,"last_scrub":"74'79","last_scrub_stamp":"2024-09-10T19:30:28.196058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:08:00.699315+0000","last_clean_scrub_stamp":"2024-09-10T19:30:28.196058+0000","objects_scrubbed":1,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:31:45.244731+0000","scrub_duration":0.012192326,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":525376,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.d","version":"414'45","reported_seq":668,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-09-10T19:30:31.333413+0000","last_change":"2024-09-10T19:30:26.584510+0000","last_active":"2024-09-10T19:30:31.333413+0000","last_peered":"2024-09-10T19:30:31.333413+0000","last_clean":"2024-09-10T19:30:31.333413+0000","last_became_active":"2024-09-10T19:27:53.012536+0000","last_became_peered":"2024-09-10T19:27:53.012536+0000","last_unstale":"2024-09-10T19:30:31.333413+0000","last_undegraded":"2024-09-10T19:30:31.333413+0000","last_fullsized":"2024-09-10T19:30:31.333413+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"414'45","last_scrub_stamp":"2024-09-10T19:30:26.584371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:30:26.584371+0000","objects_scrubbed":8,"log_size":45,"log_dups_size":0,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:31:48.878559+0000","scrub_duration":0.0072232800000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":71,"num_read_kb":61,"num_write":46,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":40,"num_bytes_recovered":550,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,6],"acting":[4,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":457,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-09-10T19:30:31.342058+0000","last_change":"2024-09-10T19:30:27.008379+0000","last_active":"2024-09-10T19:30:31.342058+0000","last_peered":"2024-09-10T19:30:31.342058+0000","last_clean":"2024-09-10T19:30:31.342058+0000","last_became_active":"2024-09-10T19:10:07.691119+0000","last_became_peered":"2024-09-10T19:10:07.691119+0000","last_unstale":"2024-09-10T19:30:31.342058+0000","last_undegraded":"2024-09-10T19:30:31.342058+0000","last_fullsized":"2024-09-10T19:30:31.342058+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:30:27.008299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:16:39.716674+0000","last_clean_scrub_stamp":"2024-09-10T19:30:27.008299+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:31:49.390872+0000","scrub_duration":0.0014108809999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":179,"reported_epoch":417,"state":"active+clean","last_fresh":"2024-09-10T19:30:36.377492+0000","last_change":"2024-09-10T19:29:30.513955+0000","last_active":"2024-09-10T19:30:36.377492+0000","last_peered":"2024-09-10T19:30:36.377492+0000","last_clean":"2024-09-10T19:30:36.377492+0000","last_became_active":"2024-09-10T19:24:24.886479+0000","last_became_peered":"2024-09-10T19:24:24.886479+0000","last_unstale":"2024-09-10T19:30:36.377492+0000","last_undegraded":"2024-09-10T19:30:36.377492+0000","last_fullsized":"2024-09-10T19:30:36.377492+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:29:30.513872+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:29:30.513872+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0073816690000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":335,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-09-10T19:30:31.331625+0000","last_change":"2024-09-10T19:29:10.112087+0000","last_active":"2024-09-10T19:30:31.331625+0000","last_peered":"2024-09-10T19:30:31.331625+0000","last_clean":"2024-09-10T19:30:31.331625+0000","last_became_active":"2024-09-10T19:27:53.026452+0000","last_became_peered":"2024-09-10T19:27:53.026452+0000","last_unstale":"2024-09-10T19:30:31.331625+0000","last_undegraded":"2024-09-10T19:30:31.331625+0000","last_fullsized":"2024-09-10T19:30:31.331625+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:29:10.111999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:29:10.111999+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:30:37.481977+0000","scrub_duration":0.0051523979999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.3","version":"417'15720","reported_seq":16119,"reported_epoch":417,"state":"active+clean","last_fresh":"2024-09-10T19:30:39.901690+0000","last_change":"2024-09-10T19:29:18.890954+0000","last_active":"2024-09-10T19:30:39.901690+0000","last_peered":"2024-09-10T19:30:39.901690+0000","last_clean":"2024-09-10T19:30:39.901690+0000","last_became_active":"2024-09-10T19:27:53.010097+0000","last_became_peered":"2024-09-10T19:27:53.010097+0000","last_unstale":"2024-09-10T19:30:39.901690+0000","last_undegraded":"2024-09-10T19:30:39.901690+0000","last_fullsized":"2024-09-10T19:30:39.901690+0000","mapping_epoch":374,"log_start":"406'11151","ondisk_log_start":"406'11151","created":371,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"400'8567","last_scrub_stamp":"2024-09-10T19:29:18.890871+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:27:45.679464+0000","last_clean_scrub_stamp":"2024-09-10T19:29:18.890871+0000","objects_scrubbed":8434,"log_size":4569,"log_dups_size":0,"ondisk_log_size":4569,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"queued for scrub","scrub_duration":3.2533210819999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":151388160,"num_objects":2310,"num_object_clones":0,"num_object_copies":6930,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2310,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15720,"num_write_kb":576960,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1],"acting":[6,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.9","version":"74'79","reported_seq":522,"reported_epoch":417,"state":"active+clean","last_fresh":"2024-09-10T19:30:39.992999+0000","last_change":"2024-09-10T19:30:39.992958+0000","last_active":"2024-09-10T19:30:39.992999+0000","last_peered":"2024-09-10T19:30:39.992999+0000","last_clean":"2024-09-10T19:30:39.992999+0000","last_became_active":"2024-09-10T19:20:30.844108+0000","last_became_peered":"2024-09-10T19:20:30.844108+0000","last_unstale":"2024-09-10T19:30:39.992999+0000","last_undegraded":"2024-09-10T19:30:39.992999+0000","last_fullsized":"2024-09-10T19:30:39.992999+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":4,"last_scrub":"74'79","last_scrub_stamp":"2024-09-10T19:30:39.992877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:08:00.699315+0000","last_clean_scrub_stamp":"2024-09-10T19:30:39.992877+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:31:53.125566+0000","scrub_duration":0.0080860159999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.c","version":"414'30","reported_seq":546,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-09-10T19:30:31.333599+0000","last_change":"2024-09-10T19:30:08.475937+0000","last_active":"2024-09-10T19:30:31.333599+0000","last_peered":"2024-09-10T19:30:31.333599+0000","last_clean":"2024-09-10T19:30:31.333599+0000","last_became_active":"2024-09-10T19:27:53.012617+0000","last_became_peered":"2024-09-10T19:27:53.012617+0000","last_unstale":"2024-09-10T19:30:31.333599+0000","last_undegraded":"2024-09-10T19:30:31.333599+0000","last_fullsized":"2024-09-10T19:30:31.333599+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"237'20","last_scrub_stamp":"2024-09-10T19:30:08.475832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:30:08.475832+0000","objects_scrubbed":5,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:31:23.767469+0000","scrub_duration":0.0094002249999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":45,"num_read_kb":40,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":155,"reported_epoch":417,"state":"active+clean","last_fresh":"2024-09-10T19:30:36.376340+0000","last_change":"2024-09-10T19:28:16.491452+0000","last_active":"2024-09-10T19:30:36.376340+0000","last_peered":"2024-09-10T19:30:36.376340+0000","last_clean":"2024-09-10T19:30:36.376340+0000","last_became_active":"2024-09-10T19:24:01.973848+0000","last_became_peered":"2024-09-10T19:24:01.973848+0000","last_unstale":"2024-09-10T19:30:36.376340+0000","last_undegraded":"2024-09-10T19:30:36.376340+0000","last_fullsized":"2024-09-10T19:30:36.376340+0000","mapping_epoch":292,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":293,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:28:16.491371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:28:16.491371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.014455551000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.d","version":"303'12","reported_seq":575,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-09-10T19:30:35.968924+0000","last_change":"2024-09-10T19:29:15.947794+0000","last_active":"2024-09-10T19:30:35.968924+0000","last_peered":"2024-09-10T19:30:35.968924+0000","last_clean":"2024-09-10T19:30:35.968924+0000","last_became_active":"2024-09-10T19:24:23.906819+0000","last_became_peered":"2024-09-10T19:24:23.906819+0000","last_unstale":"2024-09-10T19:30:35.968924+0000","last_undegraded":"2024-09-10T19:30:35.968924+0000","last_fullsized":"2024-09-10T19:30:35.968924+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"303'12","last_scrub_stamp":"2024-09-10T19:29:15.947693+0000","last_deep_scrub":"80'9","last_deep_scrub_stamp":"2024-09-10T19:11:25.156465+0000","last_clean_scrub_stamp":"2024-09-10T19:29:15.947693+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:30:41.293004+0000","scrub_duration":0.012440023,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,0],"acting":[7,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":483,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-09-10T19:30:34.534552+0000","last_change":"2024-09-10T19:30:34.534444+0000","last_active":"2024-09-10T19:30:34.534552+0000","last_peered":"2024-09-10T19:30:34.534552+0000","last_clean":"2024-09-10T19:30:34.534552+0000","last_became_active":"2024-09-10T19:24:01.972593+0000","last_became_peered":"2024-09-10T19:24:01.972593+0000","last_unstale":"2024-09-10T19:30:34.534552+0000","last_undegraded":"2024-09-10T19:30:34.534552+0000","last_fullsized":"2024-09-10T19:30:34.534552+0000","mapping_epoch":292,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":293,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:30:34.534128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:30:34.534128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:31:59.004309+0000","scrub_duration":0.029869611000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,5],"acting":[4,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.0","version":"418'15991","reported_seq":16236,"reported_epoch":418,"state":"active+clean","last_fresh":"2024-09-10T19:30:42.549783+0000","last_change":"2024-09-10T19:30:15.362614+0000","last_active":"2024-09-10T19:30:42.549783+0000","last_peered":"2024-09-10T19:30:42.549783+0000","last_clean":"2024-09-10T19:30:42.549783+0000","last_became_active":"2024-09-10T19:27:46.705538+0000","last_became_peered":"2024-09-10T19:27:46.705538+0000","last_unstale":"2024-09-10T19:30:42.549783+0000","last_undegraded":"2024-09-10T19:30:42.549783+0000","last_fullsized":"2024-09-10T19:30:42.549783+0000","mapping_epoch":371,"log_start":"407'11600","ondisk_log_start":"407'11600","created":371,"last_epoch_clean":372,"parent":"0.0","parent_split_bits":0,"last_scrub":"414'13664","last_scrub_stamp":"2024-09-10T19:30:15.362539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:27:45.679464+0000","last_clean_scrub_stamp":"2024-09-10T19:30:15.362539+0000","objects_scrubbed":4480,"log_size":4391,"log_dups_size":0,"ondisk_log_size":4391,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:31:16.651953+0000","scrub_duration":1.8958626860000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":136380416,"num_objects":2081,"num_object_clones":0,"num_object_copies":6243,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2081,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15991,"num_write_kb":578304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"74'79","reported_seq":242,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-09-10T19:30:31.331549+0000","last_change":"2024-09-10T19:30:04.397199+0000","last_active":"2024-09-10T19:30:31.331549+0000","last_peered":"2024-09-10T19:30:31.331549+0000","last_clean":"2024-09-10T19:30:31.331549+0000","last_became_active":"2024-09-10T19:27:53.004646+0000","last_became_peered":"2024-09-10T19:27:53.004646+0000","last_unstale":"2024-09-10T19:30:31.331549+0000","last_undegraded":"2024-09-10T19:30:31.331549+0000","last_fullsized":"2024-09-10T19:30:31.331549+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":4,"last_scrub":"74'79","last_scrub_stamp":"2024-09-10T19:30:04.397114+0000","last_deep_scrub":"74'79","last_deep_scrub_stamp":"2024-09-10T19:21:53.326433+0000","last_clean_scrub_stamp":"2024-09-10T19:30:04.397114+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:31:34.304105+0000","scrub_duration":0.0040705209999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,4],"acting":[5,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.f","version":"414'39","reported_seq":189,"reported_epoch":417,"state":"active+clean","last_fresh":"2024-09-10T19:30:36.376582+0000","last_change":"2024-09-10T19:28:26.434825+0000","last_active":"2024-09-10T19:30:36.376582+0000","last_peered":"2024-09-10T19:30:36.376582+0000","last_clean":"2024-09-10T19:30:36.376582+0000","last_became_active":"2024-09-10T19:24:01.979187+0000","last_became_peered":"2024-09-10T19:24:01.979187+0000","last_unstale":"2024-09-10T19:30:36.376582+0000","last_undegraded":"2024-09-10T19:30:36.376582+0000","last_fullsized":"2024-09-10T19:30:36.376582+0000","mapping_epoch":292,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":293,"parent":"0.0","parent_split_bits":0,"last_scrub":"237'27","last_scrub_stamp":"2024-09-10T19:28:26.434563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:28:26.434563+0000","objects_scrubbed":7,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0096637310000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":61,"num_read_kb":52,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":155,"reported_epoch":417,"state":"active+clean","last_fresh":"2024-09-10T19:30:36.375975+0000","last_change":"2024-09-10T19:28:24.420346+0000","last_active":"2024-09-10T19:30:36.375975+0000","last_peered":"2024-09-10T19:30:36.375975+0000","last_clean":"2024-09-10T19:30:36.375975+0000","last_became_active":"2024-09-10T19:24:01.974773+0000","last_became_peered":"2024-09-10T19:24:01.974773+0000","last_unstale":"2024-09-10T19:30:36.375975+0000","last_undegraded":"2024-09-10T19:30:36.375975+0000","last_fullsized":"2024-09-10T19:30:36.375975+0000","mapping_epoch":292,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":293,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:28:24.420270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:28:24.420270+0000","last_clean_scrub_stamp":"2024-09-10T19:28:24.420270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.016717709000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.e","version":"303'10","reported_seq":698,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-09-10T19:30:35.968537+0000","last_change":"2024-09-10T19:30:07.517646+0000","last_active":"2024-09-10T19:30:35.968537+0000","last_peered":"2024-09-10T19:30:35.968537+0000","last_clean":"2024-09-10T19:30:35.968537+0000","last_became_active":"2024-09-10T19:24:23.908760+0000","last_became_peered":"2024-09-10T19:24:23.908760+0000","last_unstale":"2024-09-10T19:30:35.968537+0000","last_undegraded":"2024-09-10T19:30:35.968537+0000","last_fullsized":"2024-09-10T19:30:35.968537+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"303'10","last_scrub_stamp":"2024-09-10T19:30:07.517566+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:30:07.517566+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:31:36.012505+0000","scrub_duration":0.0052487269999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":479,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-09-10T19:30:31.331954+0000","last_change":"2024-09-10T19:29:15.298967+0000","last_active":"2024-09-10T19:30:31.331954+0000","last_peered":"2024-09-10T19:30:31.331954+0000","last_clean":"2024-09-10T19:30:31.331954+0000","last_became_active":"2024-09-10T19:24:24.917109+0000","last_became_peered":"2024-09-10T19:24:24.917109+0000","last_unstale":"2024-09-10T19:30:31.331954+0000","last_undegraded":"2024-09-10T19:30:31.331954+0000","last_fullsized":"2024-09-10T19:30:31.331954+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:29:15.298879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:29:15.298879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.003007774,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.1","version":"417'15694","reported_seq":16101,"reported_epoch":417,"state":"active+clean","last_fresh":"2024-09-10T19:30:39.886057+0000","last_change":"2024-09-10T19:29:29.986893+0000","last_active":"2024-09-10T19:30:39.886057+0000","last_peered":"2024-09-10T19:30:39.886057+0000","last_clean":"2024-09-10T19:30:39.886057+0000","last_became_active":"2024-09-10T19:27:53.016565+0000","last_became_peered":"2024-09-10T19:27:53.016565+0000","last_unstale":"2024-09-10T19:30:39.886057+0000","last_undegraded":"2024-09-10T19:30:39.886057+0000","last_fullsized":"2024-09-10T19:30:39.886057+0000","mapping_epoch":374,"log_start":"406'11123","ondisk_log_start":"406'11123","created":371,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"401'9457","last_scrub_stamp":"2024-09-10T19:29:29.986778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:27:45.679464+0000","last_clean_scrub_stamp":"2024-09-10T19:29:29.986778+0000","objects_scrubbed":8636,"log_size":4571,"log_dups_size":0,"ondisk_log_size":4571,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"queued for scrub","scrub_duration":3.5527387689999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":147587072,"num_objects":2252,"num_object_clones":0,"num_object_copies":6756,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2252,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15694,"num_write_kb":574272,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.e","version":"414'32","reported_seq":612,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-09-10T19:30:31.333830+0000","last_change":"2024-09-10T19:30:18.472016+0000","last_active":"2024-09-10T19:30:31.333830+0000","last_peered":"2024-09-10T19:30:31.333830+0000","last_clean":"2024-09-10T19:30:31.333830+0000","last_became_active":"2024-09-10T19:27:53.012101+0000","last_became_peered":"2024-09-10T19:27:53.012101+0000","last_unstale":"2024-09-10T19:30:31.333830+0000","last_undegraded":"2024-09-10T19:30:31.333830+0000","last_fullsized":"2024-09-10T19:30:31.333830+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"414'32","last_scrub_stamp":"2024-09-10T19:30:18.471961+0000","last_deep_scrub":"84'16","last_deep_scrub_stamp":"2024-09-10T19:11:56.157975+0000","last_clean_scrub_stamp":"2024-09-10T19:30:18.471961+0000","objects_scrubbed":5,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:31:40.658421+0000","scrub_duration":0.001699491,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":37,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":550,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":128,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-09-10T19:30:31.331259+0000","last_change":"2024-09-10T19:29:30.393193+0000","last_active":"2024-09-10T19:30:31.331259+0000","last_peered":"2024-09-10T19:30:31.331259+0000","last_clean":"2024-09-10T19:30:31.331259+0000","last_became_active":"2024-09-10T19:24:53.756220+0000","last_became_peered":"2024-09-10T19:24:53.756220+0000","last_unstale":"2024-09-10T19:30:31.331259+0000","last_undegraded":"2024-09-10T19:30:31.331259+0000","last_fullsized":"2024-09-10T19:30:31.331259+0000","mapping_epoch":318,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":319,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:29:30.393109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:24:13.636249+0000","last_clean_scrub_stamp":"2024-09-10T19:29:30.393109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:30:50.348247+0000","scrub_duration":0.0027000890000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,3],"acting":[4,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":453,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-09-10T19:30:31.332066+0000","last_change":"2024-09-10T19:30:15.118053+0000","last_active":"2024-09-10T19:30:31.332066+0000","last_peered":"2024-09-10T19:30:31.332066+0000","last_clean":"2024-09-10T19:30:31.332066+0000","last_became_active":"2024-09-10T19:27:53.007094+0000","last_became_peered":"2024-09-10T19:27:53.007094+0000","last_unstale":"2024-09-10T19:30:31.332066+0000","last_undegraded":"2024-09-10T19:30:31.332066+0000","last_fullsized":"2024-09-10T19:30:31.332066+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:30:15.117979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:30:15.117979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:31:33.405690+0000","scrub_duration":0.0051218720000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":451,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-09-10T19:30:31.330782+0000","last_change":"2024-09-10T19:30:29.757452+0000","last_active":"2024-09-10T19:30:31.330782+0000","last_peered":"2024-09-10T19:30:31.330782+0000","last_clean":"2024-09-10T19:30:31.330782+0000","last_became_active":"2024-09-10T19:27:53.025345+0000","last_became_peered":"2024-09-10T19:27:53.025345+0000","last_unstale":"2024-09-10T19:30:31.330782+0000","last_undegraded":"2024-09-10T19:30:31.330782+0000","last_fullsized":"2024-09-10T19:30:31.330782+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:30:29.757372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:30:29.757372+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:31:37.286874+0000","scrub_duration":0.0047515539999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.e","version":"417'15874","reported_seq":16268,"reported_epoch":417,"state":"active+clean","last_fresh":"2024-09-10T19:30:40.284743+0000","last_change":"2024-09-10T19:29:34.726363+0000","last_active":"2024-09-10T19:30:40.284743+0000","last_peered":"2024-09-10T19:30:40.284743+0000","last_clean":"2024-09-10T19:30:40.284743+0000","last_became_active":"2024-09-10T19:27:46.704103+0000","last_became_peered":"2024-09-10T19:27:46.704103+0000","last_unstale":"2024-09-10T19:30:40.284743+0000","last_undegraded":"2024-09-10T19:30:40.284743+0000","last_fullsized":"2024-09-10T19:30:40.284743+0000","mapping_epoch":371,"log_start":"406'11200","ondisk_log_start":"406'11200","created":371,"last_epoch_clean":372,"parent":"0.0","parent_split_bits":0,"last_scrub":"404'9978","last_scrub_stamp":"2024-09-10T19:29:34.726102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:27:45.679464+0000","last_clean_scrub_stamp":"2024-09-10T19:29:34.726102+0000","objects_scrubbed":8262,"log_size":4674,"log_dups_size":0,"ondisk_log_size":4674,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:30:55.165671+0000","scrub_duration":3.3425435960000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":146276352,"num_objects":2232,"num_object_clones":0,"num_object_copies":6696,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2232,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15874,"num_write_kb":579392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,7],"acting":[4,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.4","version":"317'80","reported_seq":228,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-09-10T19:30:31.329762+0000","last_change":"2024-09-10T19:29:14.237503+0000","last_active":"2024-09-10T19:30:31.329762+0000","last_peered":"2024-09-10T19:30:31.329762+0000","last_clean":"2024-09-10T19:30:31.329762+0000","last_became_active":"2024-09-10T19:20:30.838441+0000","last_became_peered":"2024-09-10T19:20:30.838441+0000","last_unstale":"2024-09-10T19:30:31.329762+0000","last_undegraded":"2024-09-10T19:30:31.329762+0000","last_fullsized":"2024-09-10T19:30:31.329762+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":3,"last_scrub":"317'80","last_scrub_stamp":"2024-09-10T19:29:14.237292+0000","last_deep_scrub":"317'80","last_deep_scrub_stamp":"2024-09-10T19:24:51.540456+0000","last_clean_scrub_stamp":"2024-09-10T19:29:14.237292+0000","objects_scrubbed":1,"log_size":80,"log_dups_size":0,"ondisk_log_size":80,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0022248250000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"414'49","reported_seq":452,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-09-10T19:30:31.332407+0000","last_change":"2024-09-10T19:30:19.518616+0000","last_active":"2024-09-10T19:30:31.332407+0000","last_peered":"2024-09-10T19:30:31.332407+0000","last_clean":"2024-09-10T19:30:31.332407+0000","last_became_active":"2024-09-10T19:27:53.011799+0000","last_became_peered":"2024-09-10T19:27:53.011799+0000","last_unstale":"2024-09-10T19:30:31.332407+0000","last_undegraded":"2024-09-10T19:30:31.332407+0000","last_fullsized":"2024-09-10T19:30:31.332407+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"414'49","last_scrub_stamp":"2024-09-10T19:30:19.518533+0000","last_deep_scrub":"265'35","last_deep_scrub_stamp":"2024-09-10T19:28:24.824459+0000","last_clean_scrub_stamp":"2024-09-10T19:30:19.518533+0000","objects_scrubbed":7,"log_size":49,"log_dups_size":0,"ondisk_log_size":49,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:31:45.050810+0000","scrub_duration":0.0080552469999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":63,"num_read_kb":56,"num_write":42,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":62,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-09-10T19:30:31.336699+0000","last_change":"2024-09-10T19:29:14.668212+0000","last_active":"2024-09-10T19:30:31.336699+0000","last_peered":"2024-09-10T19:30:31.336699+0000","last_clean":"2024-09-10T19:30:31.336699+0000","last_became_active":"2024-09-10T19:27:53.020498+0000","last_became_peered":"2024-09-10T19:27:53.020498+0000","last_unstale":"2024-09-10T19:30:31.336699+0000","last_undegraded":"2024-09-10T19:30:31.336699+0000","last_fullsized":"2024-09-10T19:30:31.336699+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:29:14.668130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:29:14.668130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:30:41.489859+0000","scrub_duration":0.0062469530000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.0","version":"417'13","reported_seq":732,"reported_epoch":417,"state":"active+clean","last_fresh":"2024-09-10T19:30:39.965054+0000","last_change":"2024-09-10T19:30:39.965009+0000","last_active":"2024-09-10T19:30:39.965054+0000","last_peered":"2024-09-10T19:30:39.965054+0000","last_clean":"2024-09-10T19:30:39.965054+0000","last_became_active":"2024-09-10T19:24:01.972645+0000","last_became_peered":"2024-09-10T19:24:01.972645+0000","last_unstale":"2024-09-10T19:30:39.965054+0000","last_undegraded":"2024-09-10T19:30:39.965054+0000","last_fullsized":"2024-09-10T19:30:39.965054+0000","mapping_epoch":292,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":293,"parent":"0.0","parent_split_bits":0,"last_scrub":"417'13","last_scrub_stamp":"2024-09-10T19:30:39.964914+0000","last_deep_scrub":"417'13","last_deep_scrub_stamp":"2024-09-10T19:30:39.964914+0000","last_clean_scrub_stamp":"2024-09-10T19:30:39.964914+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:31:49.349883+0000","scrub_duration":0.0087623219999999995,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":478,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-09-10T19:30:31.341092+0000","last_change":"2024-09-10T19:30:14.879292+0000","last_active":"2024-09-10T19:30:31.341092+0000","last_peered":"2024-09-10T19:30:31.341092+0000","last_clean":"2024-09-10T19:30:31.341092+0000","last_became_active":"2024-09-10T19:27:53.009392+0000","last_became_peered":"2024-09-10T19:27:53.009392+0000","last_unstale":"2024-09-10T19:30:31.341092+0000","last_undegraded":"2024-09-10T19:30:31.341092+0000","last_fullsized":"2024-09-10T19:30:31.341092+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:30:14.879002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:28:25.822918+0000","last_clean_scrub_stamp":"2024-09-10T19:30:14.879002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:31:25.289697+0000","scrub_duration":0.011117784,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.f","version":"417'15794","reported_seq":16061,"reported_epoch":417,"state":"active+clean","last_fresh":"2024-09-10T19:30:40.555962+0000","last_change":"2024-09-10T19:30:08.691714+0000","last_active":"2024-09-10T19:30:40.555962+0000","last_peered":"2024-09-10T19:30:40.555962+0000","last_clean":"2024-09-10T19:30:40.555962+0000","last_became_active":"2024-09-10T19:27:46.707637+0000","last_became_peered":"2024-09-10T19:27:46.707637+0000","last_unstale":"2024-09-10T19:30:40.555962+0000","last_undegraded":"2024-09-10T19:30:40.555962+0000","last_fullsized":"2024-09-10T19:30:40.555962+0000","mapping_epoch":371,"log_start":"406'11500","ondisk_log_start":"406'11500","created":371,"last_epoch_clean":372,"parent":"0.0","parent_split_bits":0,"last_scrub":"413'13046","last_scrub_stamp":"2024-09-10T19:30:08.691638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:27:45.679464+0000","last_clean_scrub_stamp":"2024-09-10T19:30:08.691638+0000","objects_scrubbed":5075,"log_size":4294,"log_dups_size":0,"ondisk_log_size":4294,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:31:16.854516+0000","scrub_duration":1.8581280769999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":146669568,"num_objects":2238,"num_object_clones":0,"num_object_copies":6714,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2238,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15794,"num_write_kb":577024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.5","version":"74'79","reported_seq":252,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-09-10T19:30:31.332913+0000","last_change":"2024-09-10T19:30:10.470076+0000","last_active":"2024-09-10T19:30:31.332913+0000","last_peered":"2024-09-10T19:30:31.332913+0000","last_clean":"2024-09-10T19:30:31.332913+0000","last_became_active":"2024-09-10T19:27:53.013021+0000","last_became_peered":"2024-09-10T19:27:53.013021+0000","last_unstale":"2024-09-10T19:30:31.332913+0000","last_undegraded":"2024-09-10T19:30:31.332913+0000","last_fullsized":"2024-09-10T19:30:31.332913+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":3,"last_scrub":"74'79","last_scrub_stamp":"2024-09-10T19:30:10.470024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:08:00.699315+0000","last_clean_scrub_stamp":"2024-09-10T19:30:10.470024+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:31:19.660974+0000","scrub_duration":0.0083031130000000009,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"414'42","reported_seq":510,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-09-10T19:30:31.331940+0000","last_change":"2024-09-10T19:29:35.143247+0000","last_active":"2024-09-10T19:30:31.331940+0000","last_peered":"2024-09-10T19:30:31.331940+0000","last_clean":"2024-09-10T19:30:31.331940+0000","last_became_active":"2024-09-10T19:24:23.903784+0000","last_became_peered":"2024-09-10T19:24:23.903784+0000","last_unstale":"2024-09-10T19:30:31.331940+0000","last_undegraded":"2024-09-10T19:30:31.331940+0000","last_fullsized":"2024-09-10T19:30:31.331940+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"237'30","last_scrub_stamp":"2024-09-10T19:29:35.143157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:29:35.143157+0000","objects_scrubbed":8,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:30:58.866237+0000","scrub_duration":0.016979355000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":70,"num_read_kb":58,"num_write":44,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":185,"reported_epoch":417,"state":"active+clean","last_fresh":"2024-09-10T19:30:36.376584+0000","last_change":"2024-09-10T19:30:19.275603+0000","last_active":"2024-09-10T19:30:36.376584+0000","last_peered":"2024-09-10T19:30:36.376584+0000","last_clean":"2024-09-10T19:30:36.376584+0000","last_became_active":"2024-09-10T19:27:53.008302+0000","last_became_peered":"2024-09-10T19:27:53.008302+0000","last_unstale":"2024-09-10T19:30:36.376584+0000","last_undegraded":"2024-09-10T19:30:36.376584+0000","last_fullsized":"2024-09-10T19:30:36.376584+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:30:19.275465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:28:15.534211+0000","last_clean_scrub_stamp":"2024-09-10T19:30:19.275465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:31:42.840763+0000","scrub_duration":0.0057158039999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":442,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-09-10T19:30:31.332564+0000","last_change":"2024-09-10T19:29:23.262835+0000","last_active":"2024-09-10T19:30:31.332564+0000","last_peered":"2024-09-10T19:30:31.332564+0000","last_clean":"2024-09-10T19:30:31.332564+0000","last_became_active":"2024-09-10T19:10:11.698054+0000","last_became_peered":"2024-09-10T19:10:11.698054+0000","last_unstale":"2024-09-10T19:30:31.332564+0000","last_undegraded":"2024-09-10T19:30:31.332564+0000","last_fullsized":"2024-09-10T19:30:31.332564+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:29:23.262748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:26:24.897519+0000","last_clean_scrub_stamp":"2024-09-10T19:29:23.262748+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:30:39.031012+0000","scrub_duration":0.0056390679999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":470,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-09-10T19:30:31.333004+0000","last_change":"2024-09-10T19:29:27.360561+0000","last_active":"2024-09-10T19:30:31.333004+0000","last_peered":"2024-09-10T19:30:31.333004+0000","last_clean":"2024-09-10T19:30:31.333004+0000","last_became_active":"2024-09-10T19:27:53.017161+0000","last_became_peered":"2024-09-10T19:27:53.017161+0000","last_unstale":"2024-09-10T19:30:31.333004+0000","last_undegraded":"2024-09-10T19:30:31.333004+0000","last_fullsized":"2024-09-10T19:30:31.333004+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:29:27.360479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:29:27.360479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:30:37.680274+0000","scrub_duration":0.0086931189999999992,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.c","version":"418'15963","reported_seq":16228,"reported_epoch":418,"state":"active+clean","last_fresh":"2024-09-10T19:30:42.523242+0000","last_change":"2024-09-10T19:30:09.592766+0000","last_active":"2024-09-10T19:30:42.523242+0000","last_peered":"2024-09-10T19:30:42.523242+0000","last_clean":"2024-09-10T19:30:42.523242+0000","last_became_active":"2024-09-10T19:27:46.705563+0000","last_became_peered":"2024-09-10T19:27:46.705563+0000","last_unstale":"2024-09-10T19:30:42.523242+0000","last_undegraded":"2024-09-10T19:30:42.523242+0000","last_fullsized":"2024-09-10T19:30:42.523242+0000","mapping_epoch":371,"log_start":"408'11600","ondisk_log_start":"408'11600","created":371,"last_epoch_clean":372,"parent":"0.0","parent_split_bits":0,"last_scrub":"413'13084","last_scrub_stamp":"2024-09-10T19:30:09.592622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:27:45.679464+0000","last_clean_scrub_stamp":"2024-09-10T19:30:09.592622+0000","objects_scrubbed":4998,"log_size":4363,"log_dups_size":0,"ondisk_log_size":4363,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:31:23.730993+0000","scrub_duration":2.154546888,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":131923968,"num_objects":2013,"num_object_clones":0,"num_object_copies":6039,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2013,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15963,"num_write_kb":575232,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,7],"acting":[1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"74'79","reported_seq":72,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-09-10T19:30:31.336246+0000","last_change":"2024-09-10T19:30:23.520229+0000","last_active":"2024-09-10T19:30:31.336246+0000","last_peered":"2024-09-10T19:30:31.336246+0000","last_clean":"2024-09-10T19:30:31.336246+0000","last_became_active":"2024-09-10T19:27:53.017899+0000","last_became_peered":"2024-09-10T19:27:53.017899+0000","last_unstale":"2024-09-10T19:30:31.336246+0000","last_undegraded":"2024-09-10T19:30:31.336246+0000","last_fullsized":"2024-09-10T19:30:31.336246+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":3,"last_scrub":"74'79","last_scrub_stamp":"2024-09-10T19:30:23.520135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:08:00.699315+0000","last_clean_scrub_stamp":"2024-09-10T19:30:23.520135+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:31:47.744700+0000","scrub_duration":0.0041853660000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"414'47","reported_seq":468,"reported_epoch":417,"state":"active+clean","last_fresh":"2024-09-10T19:30:36.370875+0000","last_change":"2024-09-10T19:30:16.465094+0000","last_active":"2024-09-10T19:30:36.370875+0000","last_peered":"2024-09-10T19:30:36.370875+0000","last_clean":"2024-09-10T19:30:36.370875+0000","last_became_active":"2024-09-10T19:24:23.901836+0000","last_became_peered":"2024-09-10T19:24:23.901836+0000","last_unstale":"2024-09-10T19:30:36.370875+0000","last_undegraded":"2024-09-10T19:30:36.370875+0000","last_fullsized":"2024-09-10T19:30:36.370875+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"414'43","last_scrub_stamp":"2024-09-10T19:30:16.464894+0000","last_deep_scrub":"104'27","last_deep_scrub_stamp":"2024-09-10T19:13:02.225944+0000","last_clean_scrub_stamp":"2024-09-10T19:30:16.464894+0000","objects_scrubbed":8,"log_size":47,"log_dups_size":0,"ondisk_log_size":47,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:31:35.284993+0000","scrub_duration":0.0076287560000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":68,"num_read_kb":54,"num_write":42,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":464,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-09-10T19:30:32.017001+0000","last_change":"2024-09-10T19:30:32.016942+0000","last_active":"2024-09-10T19:30:32.017001+0000","last_peered":"2024-09-10T19:30:32.017001+0000","last_clean":"2024-09-10T19:30:32.017001+0000","last_became_active":"2024-09-10T19:24:23.899556+0000","last_became_peered":"2024-09-10T19:24:23.899556+0000","last_unstale":"2024-09-10T19:30:32.017001+0000","last_undegraded":"2024-09-10T19:30:32.017001+0000","last_fullsized":"2024-09-10T19:30:32.017001+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:30:32.016858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:16:37.779967+0000","last_clean_scrub_stamp":"2024-09-10T19:30:32.016858+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:31:37.792761+0000","scrub_duration":0.0083493209999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":72,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-09-10T19:30:31.336449+0000","last_change":"2024-09-10T19:30:19.498917+0000","last_active":"2024-09-10T19:30:31.336449+0000","last_peered":"2024-09-10T19:30:31.336449+0000","last_clean":"2024-09-10T19:30:31.336449+0000","last_became_active":"2024-09-10T19:27:53.003734+0000","last_became_peered":"2024-09-10T19:27:53.003734+0000","last_unstale":"2024-09-10T19:30:31.336449+0000","last_undegraded":"2024-09-10T19:30:31.336449+0000","last_fullsized":"2024-09-10T19:30:31.336449+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:30:19.498837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:30:19.498837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:31:47.588619+0000","scrub_duration":0.004490628,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":185,"reported_epoch":417,"state":"active+clean","last_fresh":"2024-09-10T19:30:36.375897+0000","last_change":"2024-09-10T19:29:22.538192+0000","last_active":"2024-09-10T19:30:36.375897+0000","last_peered":"2024-09-10T19:30:36.375897+0000","last_clean":"2024-09-10T19:30:36.375897+0000","last_became_active":"2024-09-10T19:27:53.016905+0000","last_became_peered":"2024-09-10T19:27:53.016905+0000","last_unstale":"2024-09-10T19:30:36.375897+0000","last_undegraded":"2024-09-10T19:30:36.375897+0000","last_fullsized":"2024-09-10T19:30:36.375897+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:29:22.538092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:29:22.538092+0000","last_clean_scrub_stamp":"2024-09-10T19:29:22.538092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:30:48.324749+0000","scrub_duration":0.0051848659999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,7],"acting":[1,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.d","version":"417'15906","reported_seq":16281,"reported_epoch":417,"state":"active+clean","last_fresh":"2024-09-10T19:30:40.320521+0000","last_change":"2024-09-10T19:30:06.229050+0000","last_active":"2024-09-10T19:30:40.320521+0000","last_peered":"2024-09-10T19:30:40.320521+0000","last_clean":"2024-09-10T19:30:40.320521+0000","last_became_active":"2024-09-10T19:27:53.011209+0000","last_became_peered":"2024-09-10T19:27:53.011209+0000","last_unstale":"2024-09-10T19:30:40.320521+0000","last_undegraded":"2024-09-10T19:30:40.320521+0000","last_fullsized":"2024-09-10T19:30:40.320521+0000","mapping_epoch":374,"log_start":"406'11200","ondisk_log_start":"406'11200","created":371,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"413'12851","last_scrub_stamp":"2024-09-10T19:30:06.228898+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:27:45.679464+0000","last_clean_scrub_stamp":"2024-09-10T19:30:06.228898+0000","objects_scrubbed":5320,"log_size":4706,"log_dups_size":0,"ondisk_log_size":4706,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:31:22.959068+0000","scrub_duration":1.7693102619999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":143589400,"num_objects":2192,"num_object_clones":0,"num_object_copies":6576,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2192,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":15906,"num_write_kb":579073,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.7","version":"74'79","reported_seq":72,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-09-10T19:30:31.336628+0000","last_change":"2024-09-10T19:30:09.567679+0000","last_active":"2024-09-10T19:30:31.336628+0000","last_peered":"2024-09-10T19:30:31.336628+0000","last_clean":"2024-09-10T19:30:31.336628+0000","last_became_active":"2024-09-10T19:27:53.020085+0000","last_became_peered":"2024-09-10T19:27:53.020085+0000","last_unstale":"2024-09-10T19:30:31.336628+0000","last_undegraded":"2024-09-10T19:30:31.336628+0000","last_fullsized":"2024-09-10T19:30:31.336628+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":3,"last_scrub":"74'79","last_scrub_stamp":"2024-09-10T19:30:09.567596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:08:00.699315+0000","last_clean_scrub_stamp":"2024-09-10T19:30:09.567596+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:31:38.788437+0000","scrub_duration":0.0064510310000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.2","version":"414'35","reported_seq":197,"reported_epoch":417,"state":"active+clean","last_fresh":"2024-09-10T19:30:36.376471+0000","last_change":"2024-09-10T19:29:06.293570+0000","last_active":"2024-09-10T19:30:36.376471+0000","last_peered":"2024-09-10T19:30:36.376471+0000","last_clean":"2024-09-10T19:30:36.376471+0000","last_became_active":"2024-09-10T19:24:01.976841+0000","last_became_peered":"2024-09-10T19:24:01.976841+0000","last_unstale":"2024-09-10T19:30:36.376471+0000","last_undegraded":"2024-09-10T19:30:36.376471+0000","last_fullsized":"2024-09-10T19:30:36.376471+0000","mapping_epoch":292,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":293,"parent":"0.0","parent_split_bits":0,"last_scrub":"350'25","last_scrub_stamp":"2024-09-10T19:29:06.293356+0000","last_deep_scrub":"350'25","last_deep_scrub_stamp":"2024-09-10T19:26:37.415039+0000","last_clean_scrub_stamp":"2024-09-10T19:29:06.293356+0000","objects_scrubbed":5,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0079160140000000007,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":38,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":179,"reported_epoch":417,"state":"active+clean","last_fresh":"2024-09-10T19:30:36.377560+0000","last_change":"2024-09-10T19:29:20.468976+0000","last_active":"2024-09-10T19:30:36.377560+0000","last_peered":"2024-09-10T19:30:36.377560+0000","last_clean":"2024-09-10T19:30:36.377560+0000","last_became_active":"2024-09-10T19:24:24.910173+0000","last_became_peered":"2024-09-10T19:24:24.910173+0000","last_unstale":"2024-09-10T19:30:36.377560+0000","last_undegraded":"2024-09-10T19:30:36.377560+0000","last_fullsized":"2024-09-10T19:30:36.377560+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:29:20.468827+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:29:20.468827+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0064360270000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"376'13","reported_seq":645,"reported_epoch":417,"state":"active+clean","last_fresh":"2024-09-10T19:30:40.969312+0000","last_change":"2024-09-10T19:30:14.496516+0000","last_active":"2024-09-10T19:30:40.969312+0000","last_peered":"2024-09-10T19:30:40.969312+0000","last_clean":"2024-09-10T19:30:40.969312+0000","last_became_active":"2024-09-10T19:27:53.000311+0000","last_became_peered":"2024-09-10T19:27:53.000311+0000","last_unstale":"2024-09-10T19:30:40.969312+0000","last_undegraded":"2024-09-10T19:30:40.969312+0000","last_fullsized":"2024-09-10T19:30:40.969312+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"376'13","last_scrub_stamp":"2024-09-10T19:30:14.496433+0000","last_deep_scrub":"376'13","last_deep_scrub_stamp":"2024-09-10T19:28:17.490297+0000","last_clean_scrub_stamp":"2024-09-10T19:30:14.496433+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:31:34.498533+0000","scrub_duration":0.0060631859999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":418,"reported_epoch":417,"state":"active+clean","last_fresh":"2024-09-10T19:30:37.927711+0000","last_change":"2024-09-10T19:30:37.927650+0000","last_active":"2024-09-10T19:30:37.927711+0000","last_peered":"2024-09-10T19:30:37.927711+0000","last_clean":"2024-09-10T19:30:37.927711+0000","last_became_active":"2024-09-10T19:24:23.905977+0000","last_became_peered":"2024-09-10T19:24:23.905977+0000","last_unstale":"2024-09-10T19:30:37.927711+0000","last_undegraded":"2024-09-10T19:30:37.927711+0000","last_fullsized":"2024-09-10T19:30:37.927711+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:30:37.927553+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:30:37.927553+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:31:49.883711+0000","scrub_duration":0.013815849,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,5],"acting":[0,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.b","version":"417'15856","reported_seq":16133,"reported_epoch":417,"state":"active+clean","last_fresh":"2024-09-10T19:30:40.353792+0000","last_change":"2024-09-10T19:30:07.316594+0000","last_active":"2024-09-10T19:30:40.353792+0000","last_peered":"2024-09-10T19:30:40.353792+0000","last_clean":"2024-09-10T19:30:40.353792+0000","last_became_active":"2024-09-10T19:27:46.709536+0000","last_became_peered":"2024-09-10T19:27:46.709536+0000","last_unstale":"2024-09-10T19:30:40.353792+0000","last_undegraded":"2024-09-10T19:30:40.353792+0000","last_fullsized":"2024-09-10T19:30:40.353792+0000","mapping_epoch":371,"log_start":"408'11700","ondisk_log_start":"408'11700","created":371,"last_epoch_clean":372,"parent":"0.0","parent_split_bits":0,"last_scrub":"413'12913","last_scrub_stamp":"2024-09-10T19:30:07.316444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:27:45.679464+0000","last_clean_scrub_stamp":"2024-09-10T19:30:07.316444+0000","objects_scrubbed":5296,"log_size":4156,"log_dups_size":0,"ondisk_log_size":4156,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:31:23.548541+0000","scrub_duration":1.9481164020000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":148373504,"num_objects":2264,"num_object_clones":0,"num_object_copies":6792,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2264,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15856,"num_write_kb":579840,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"74'79","reported_seq":229,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-09-10T19:30:31.331459+0000","last_change":"2024-09-10T19:30:09.313860+0000","last_active":"2024-09-10T19:30:31.331459+0000","last_peered":"2024-09-10T19:30:31.331459+0000","last_clean":"2024-09-10T19:30:31.331459+0000","last_became_active":"2024-09-10T19:24:23.907264+0000","last_became_peered":"2024-09-10T19:24:23.907264+0000","last_unstale":"2024-09-10T19:30:31.331459+0000","last_undegraded":"2024-09-10T19:30:31.331459+0000","last_fullsized":"2024-09-10T19:30:31.331459+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":4,"last_scrub":"74'79","last_scrub_stamp":"2024-09-10T19:30:09.313780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:08:00.699315+0000","last_clean_scrub_stamp":"2024-09-10T19:30:09.313780+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:31:22.971677+0000","scrub_duration":0.0047975530000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,4],"acting":[5,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"414'68","reported_seq":359,"reported_epoch":417,"state":"active+clean","last_fresh":"2024-09-10T19:30:40.971629+0000","last_change":"2024-09-10T19:30:37.347615+0000","last_active":"2024-09-10T19:30:40.971629+0000","last_peered":"2024-09-10T19:30:40.971629+0000","last_clean":"2024-09-10T19:30:40.971629+0000","last_became_active":"2024-09-10T19:27:53.008533+0000","last_became_peered":"2024-09-10T19:27:53.008533+0000","last_unstale":"2024-09-10T19:30:40.971629+0000","last_undegraded":"2024-09-10T19:30:40.971629+0000","last_fullsized":"2024-09-10T19:30:40.971629+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"414'68","last_scrub_stamp":"2024-09-10T19:30:37.347412+0000","last_deep_scrub":"384'56","last_deep_scrub_stamp":"2024-09-10T19:28:23.457119+0000","last_clean_scrub_stamp":"2024-09-10T19:30:37.347412+0000","objects_scrubbed":10,"log_size":68,"log_dups_size":0,"ondisk_log_size":68,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:31:59.524500+0000","scrub_duration":0.012024986,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":83,"num_read_kb":68,"num_write":50,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":40,"num_bytes_recovered":1432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":491,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-09-10T19:30:31.331615+0000","last_change":"2024-09-10T19:30:26.182093+0000","last_active":"2024-09-10T19:30:31.331615+0000","last_peered":"2024-09-10T19:30:31.331615+0000","last_clean":"2024-09-10T19:30:31.331615+0000","last_became_active":"2024-09-10T19:27:53.004220+0000","last_became_peered":"2024-09-10T19:27:53.004220+0000","last_unstale":"2024-09-10T19:30:31.331615+0000","last_undegraded":"2024-09-10T19:30:31.331615+0000","last_fullsized":"2024-09-10T19:30:31.331615+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:30:26.182025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:30:26.182025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:31:49.676633+0000","scrub_duration":0.0084229500000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":418,"reported_epoch":417,"state":"active+clean","last_fresh":"2024-09-10T19:30:36.370665+0000","last_change":"2024-09-10T19:30:30.855859+0000","last_active":"2024-09-10T19:30:36.370665+0000","last_peered":"2024-09-10T19:30:36.370665+0000","last_clean":"2024-09-10T19:30:36.370665+0000","last_became_active":"2024-09-10T19:24:24.892692+0000","last_became_peered":"2024-09-10T19:24:24.892692+0000","last_unstale":"2024-09-10T19:30:36.370665+0000","last_undegraded":"2024-09-10T19:30:36.370665+0000","last_fullsized":"2024-09-10T19:30:36.370665+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:30:30.855783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:30:30.855783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:31:57.710760+0000","scrub_duration":0.0039706239999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":421,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-09-10T19:30:31.332101+0000","last_change":"2024-09-10T19:29:08.124587+0000","last_active":"2024-09-10T19:30:31.332101+0000","last_peered":"2024-09-10T19:30:31.332101+0000","last_clean":"2024-09-10T19:30:31.332101+0000","last_became_active":"2024-09-10T19:10:13.721744+0000","last_became_peered":"2024-09-10T19:10:13.721744+0000","last_unstale":"2024-09-10T19:30:31.332101+0000","last_undegraded":"2024-09-10T19:30:31.332101+0000","last_fullsized":"2024-09-10T19:30:31.332101+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:29:08.124503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:29:08.124503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.010126721999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.8","version":"417'15800","reported_seq":16075,"reported_epoch":417,"state":"active+clean","last_fresh":"2024-09-10T19:30:40.347332+0000","last_change":"2024-09-10T19:30:09.096902+0000","last_active":"2024-09-10T19:30:40.347332+0000","last_peered":"2024-09-10T19:30:40.347332+0000","last_clean":"2024-09-10T19:30:40.347332+0000","last_became_active":"2024-09-10T19:27:53.024817+0000","last_became_peered":"2024-09-10T19:27:53.024817+0000","last_unstale":"2024-09-10T19:30:40.347332+0000","last_undegraded":"2024-09-10T19:30:40.347332+0000","last_fullsized":"2024-09-10T19:30:40.347332+0000","mapping_epoch":374,"log_start":"407'11600","ondisk_log_start":"407'11600","created":371,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"413'13050","last_scrub_stamp":"2024-09-10T19:30:09.096759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:27:45.679464+0000","last_clean_scrub_stamp":"2024-09-10T19:30:09.096759+0000","objects_scrubbed":5028,"log_size":4200,"log_dups_size":0,"ondisk_log_size":4200,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:31:09.617290+0000","scrub_duration":1.7211459499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":144965632,"num_objects":2212,"num_object_clones":0,"num_object_copies":6636,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2212,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15800,"num_write_kb":576384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"74'79","reported_seq":519,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-09-10T19:30:31.341212+0000","last_change":"2024-09-10T19:29:07.906129+0000","last_active":"2024-09-10T19:30:31.341212+0000","last_peered":"2024-09-10T19:30:31.341212+0000","last_clean":"2024-09-10T19:30:31.341212+0000","last_became_active":"2024-09-10T19:24:23.895247+0000","last_became_peered":"2024-09-10T19:24:23.895247+0000","last_unstale":"2024-09-10T19:30:31.341212+0000","last_undegraded":"2024-09-10T19:30:31.341212+0000","last_fullsized":"2024-09-10T19:30:31.341212+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":4,"last_scrub":"74'79","last_scrub_stamp":"2024-09-10T19:29:07.906054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:08:00.699315+0000","last_clean_scrub_stamp":"2024-09-10T19:29:07.906054+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.006366633,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.7","version":"414'26","reported_seq":197,"reported_epoch":417,"state":"active+clean","last_fresh":"2024-09-10T19:30:36.376510+0000","last_change":"2024-09-10T19:30:34.330565+0000","last_active":"2024-09-10T19:30:36.376510+0000","last_peered":"2024-09-10T19:30:36.376510+0000","last_clean":"2024-09-10T19:30:36.376510+0000","last_became_active":"2024-09-10T19:24:24.894293+0000","last_became_peered":"2024-09-10T19:24:24.894293+0000","last_unstale":"2024-09-10T19:30:36.376510+0000","last_undegraded":"2024-09-10T19:30:36.376510+0000","last_fullsized":"2024-09-10T19:30:36.376510+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"414'26","last_scrub_stamp":"2024-09-10T19:30:34.330411+0000","last_deep_scrub":"82'18","last_deep_scrub_stamp":"2024-09-10T19:15:55.777361+0000","last_clean_scrub_stamp":"2024-09-10T19:30:34.330411+0000","objects_scrubbed":5,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:31:42.777447+0000","scrub_duration":0.010661754000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":42,"num_read_kb":31,"num_write":24,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"159'6","reported_seq":607,"reported_epoch":417,"state":"active+clean","last_fresh":"2024-09-10T19:30:41.581085+0000","last_change":"2024-09-10T19:29:16.485160+0000","last_active":"2024-09-10T19:30:41.581085+0000","last_peered":"2024-09-10T19:30:41.581085+0000","last_clean":"2024-09-10T19:30:41.581085+0000","last_became_active":"2024-09-10T19:27:53.017381+0000","last_became_peered":"2024-09-10T19:27:53.017381+0000","last_unstale":"2024-09-10T19:30:41.581085+0000","last_undegraded":"2024-09-10T19:30:41.581085+0000","last_fullsized":"2024-09-10T19:30:41.581085+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"159'6","last_scrub_stamp":"2024-09-10T19:29:16.485081+0000","last_deep_scrub":"159'6","last_deep_scrub_stamp":"2024-09-10T19:16:32.649914+0000","last_clean_scrub_stamp":"2024-09-10T19:29:16.485081+0000","objects_scrubbed":1,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0054402080000000002,"objects_trimmed":0,"snaptrim_duration":0.00034454499999999997,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":402,"num_read_kb":397,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1556,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":366,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-09-10T19:30:31.835455+0000","last_change":"2024-09-10T19:30:31.835352+0000","last_active":"2024-09-10T19:30:31.835455+0000","last_peered":"2024-09-10T19:30:31.835455+0000","last_clean":"2024-09-10T19:30:31.835455+0000","last_became_active":"2024-09-10T19:24:24.924859+0000","last_became_peered":"2024-09-10T19:24:24.924859+0000","last_unstale":"2024-09-10T19:30:31.835455+0000","last_undegraded":"2024-09-10T19:30:31.835455+0000","last_fullsized":"2024-09-10T19:30:31.835455+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:30:31.835272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:28:21.199065+0000","last_clean_scrub_stamp":"2024-09-10T19:30:31.835272+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:31:56.356268+0000","scrub_duration":0.0079787279999999992,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":335,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-09-10T19:30:31.334708+0000","last_change":"2024-09-10T19:30:23.179974+0000","last_active":"2024-09-10T19:30:31.334708+0000","last_peered":"2024-09-10T19:30:31.334708+0000","last_clean":"2024-09-10T19:30:31.334708+0000","last_became_active":"2024-09-10T19:16:24.861481+0000","last_became_peered":"2024-09-10T19:16:24.861481+0000","last_unstale":"2024-09-10T19:30:31.334708+0000","last_undegraded":"2024-09-10T19:30:31.334708+0000","last_fullsized":"2024-09-10T19:30:31.334708+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:30:23.179923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:23:28.736502+0000","last_clean_scrub_stamp":"2024-09-10T19:30:23.179923+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:31:53.153842+0000","scrub_duration":0.0011139259999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,4],"acting":[5,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":470,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-09-10T19:30:31.341340+0000","last_change":"2024-09-10T19:30:05.863333+0000","last_active":"2024-09-10T19:30:31.341340+0000","last_peered":"2024-09-10T19:30:31.341340+0000","last_clean":"2024-09-10T19:30:31.341340+0000","last_became_active":"2024-09-10T19:27:53.011923+0000","last_became_peered":"2024-09-10T19:27:53.011923+0000","last_unstale":"2024-09-10T19:30:31.341340+0000","last_undegraded":"2024-09-10T19:30:31.341340+0000","last_fullsized":"2024-09-10T19:30:31.341340+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:30:05.863186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:22:01.208272+0000","last_clean_scrub_stamp":"2024-09-10T19:30:05.863186+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:31:07.939735+0000","scrub_duration":0.0076258740000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.9","version":"417'16059","reported_seq":16313,"reported_epoch":417,"state":"active+clean","last_fresh":"2024-09-10T19:30:40.354405+0000","last_change":"2024-09-10T19:30:16.747438+0000","last_active":"2024-09-10T19:30:40.354405+0000","last_peered":"2024-09-10T19:30:40.354405+0000","last_clean":"2024-09-10T19:30:40.354405+0000","last_became_active":"2024-09-10T19:27:53.018973+0000","last_became_peered":"2024-09-10T19:27:53.018973+0000","last_unstale":"2024-09-10T19:30:40.354405+0000","last_undegraded":"2024-09-10T19:30:40.354405+0000","last_fullsized":"2024-09-10T19:30:40.354405+0000","mapping_epoch":374,"log_start":"408'11900","ondisk_log_start":"408'11900","created":371,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"414'13961","last_scrub_stamp":"2024-09-10T19:30:16.747355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:27:45.679464+0000","last_clean_scrub_stamp":"2024-09-10T19:30:16.747355+0000","objects_scrubbed":4475,"log_size":4159,"log_dups_size":0,"ondisk_log_size":4159,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:31:28.383270+0000","scrub_duration":2.2859866719999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":148176896,"num_objects":2261,"num_object_clones":0,"num_object_copies":6783,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2261,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16059,"num_write_kb":586240,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"74'79","reported_seq":185,"reported_epoch":417,"state":"active+clean","last_fresh":"2024-09-10T19:30:40.383373+0000","last_change":"2024-09-10T19:30:40.383314+0000","last_active":"2024-09-10T19:30:40.383373+0000","last_peered":"2024-09-10T19:30:40.383373+0000","last_clean":"2024-09-10T19:30:40.383373+0000","last_became_active":"2024-09-10T19:27:53.016965+0000","last_became_peered":"2024-09-10T19:27:53.016965+0000","last_unstale":"2024-09-10T19:30:40.383373+0000","last_undegraded":"2024-09-10T19:30:40.383373+0000","last_fullsized":"2024-09-10T19:30:40.383373+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":3,"last_scrub":"74'79","last_scrub_stamp":"2024-09-10T19:30:40.383237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:08:00.699315+0000","last_clean_scrub_stamp":"2024-09-10T19:30:40.383237+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:31:57.251947+0000","scrub_duration":0.0084886000000000007,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"414'42","reported_seq":516,"reported_epoch":417,"state":"active+clean","last_fresh":"2024-09-10T19:30:36.373365+0000","last_change":"2024-09-10T19:29:36.586040+0000","last_active":"2024-09-10T19:30:36.373365+0000","last_peered":"2024-09-10T19:30:36.373365+0000","last_clean":"2024-09-10T19:30:36.373365+0000","last_became_active":"2024-09-10T19:24:24.899711+0000","last_became_peered":"2024-09-10T19:24:24.899711+0000","last_unstale":"2024-09-10T19:30:36.373365+0000","last_undegraded":"2024-09-10T19:30:36.373365+0000","last_fullsized":"2024-09-10T19:30:36.373365+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"237'30","last_scrub_stamp":"2024-09-10T19:29:36.584508+0000","last_deep_scrub":"221'18","last_deep_scrub_stamp":"2024-09-10T19:19:29.606748+0000","last_clean_scrub_stamp":"2024-09-10T19:29:36.584508+0000","objects_scrubbed":6,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:30:50.767622+0000","scrub_duration":0.0052445089999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":53,"num_read_kb":47,"num_write":36,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"417'2","reported_seq":412,"reported_epoch":417,"state":"active+clean","last_fresh":"2024-09-10T19:30:39.904673+0000","last_change":"2024-09-10T19:30:39.904486+0000","last_active":"2024-09-10T19:30:39.904673+0000","last_peered":"2024-09-10T19:30:39.904673+0000","last_clean":"2024-09-10T19:30:39.904673+0000","last_became_active":"2024-09-10T19:24:23.897412+0000","last_became_peered":"2024-09-10T19:24:23.897412+0000","last_unstale":"2024-09-10T19:30:39.904673+0000","last_undegraded":"2024-09-10T19:30:39.904673+0000","last_fullsized":"2024-09-10T19:30:39.904673+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"417'2","last_scrub_stamp":"2024-09-10T19:30:39.904339+0000","last_deep_scrub":"417'2","last_deep_scrub_stamp":"2024-09-10T19:30:39.904339+0000","last_clean_scrub_stamp":"2024-09-10T19:30:39.904339+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:32:02.472678+0000","scrub_duration":0.0076308299999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":485,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-09-10T19:30:31.340912+0000","last_change":"2024-09-10T19:30:17.882408+0000","last_active":"2024-09-10T19:30:31.340912+0000","last_peered":"2024-09-10T19:30:31.340912+0000","last_clean":"2024-09-10T19:30:31.340912+0000","last_became_active":"2024-09-10T19:24:24.896201+0000","last_became_peered":"2024-09-10T19:24:24.896201+0000","last_unstale":"2024-09-10T19:30:31.340912+0000","last_undegraded":"2024-09-10T19:30:31.340912+0000","last_fullsized":"2024-09-10T19:30:31.340912+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:30:17.882327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:25:28.762237+0000","last_clean_scrub_stamp":"2024-09-10T19:30:17.882327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:31:19.706681+0000","scrub_duration":0.010437434000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":491,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-09-10T19:30:35.213781+0000","last_change":"2024-09-10T19:30:35.213643+0000","last_active":"2024-09-10T19:30:35.213781+0000","last_peered":"2024-09-10T19:30:35.213781+0000","last_clean":"2024-09-10T19:30:35.213781+0000","last_became_active":"2024-09-10T19:24:24.914680+0000","last_became_peered":"2024-09-10T19:24:24.914680+0000","last_unstale":"2024-09-10T19:30:35.213781+0000","last_undegraded":"2024-09-10T19:30:35.213781+0000","last_fullsized":"2024-09-10T19:30:35.213781+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:30:35.213557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:30:35.213557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:31:44.941744+0000","scrub_duration":0.0069688670000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":165,"reported_epoch":417,"state":"active+clean","last_fresh":"2024-09-10T19:30:36.377423+0000","last_change":"2024-09-10T19:30:31.245345+0000","last_active":"2024-09-10T19:30:36.377423+0000","last_peered":"2024-09-10T19:30:36.377423+0000","last_clean":"2024-09-10T19:30:36.377423+0000","last_became_active":"2024-09-10T19:24:01.974039+0000","last_became_peered":"2024-09-10T19:24:01.974039+0000","last_unstale":"2024-09-10T19:30:36.377423+0000","last_undegraded":"2024-09-10T19:30:36.377423+0000","last_fullsized":"2024-09-10T19:30:36.377423+0000","mapping_epoch":292,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":293,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:30:31.245191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:30:31.245191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:31:45.846908+0000","scrub_duration":0.0041648700000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.a","version":"417'16108","reported_seq":16364,"reported_epoch":417,"state":"active+clean","last_fresh":"2024-09-10T19:30:40.358704+0000","last_change":"2024-09-10T19:30:15.315333+0000","last_active":"2024-09-10T19:30:40.358704+0000","last_peered":"2024-09-10T19:30:40.358704+0000","last_clean":"2024-09-10T19:30:40.358704+0000","last_became_active":"2024-09-10T19:27:53.026021+0000","last_became_peered":"2024-09-10T19:27:53.026021+0000","last_unstale":"2024-09-10T19:30:40.358704+0000","last_undegraded":"2024-09-10T19:30:40.358704+0000","last_fullsized":"2024-09-10T19:30:40.358704+0000","mapping_epoch":374,"log_start":"408'11900","ondisk_log_start":"408'11900","created":371,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"414'13886","last_scrub_stamp":"2024-09-10T19:30:15.315183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:27:45.679464+0000","last_clean_scrub_stamp":"2024-09-10T19:30:15.315183+0000","objects_scrubbed":4548,"log_size":4208,"log_dups_size":0,"ondisk_log_size":4208,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:31:19.467846+0000","scrub_duration":1.898975455,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":148242432,"num_objects":2262,"num_object_clones":0,"num_object_copies":6786,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2262,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16108,"num_write_kb":587840,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"414'45","reported_seq":106,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-09-10T19:30:31.336551+0000","last_change":"2024-09-10T19:30:05.568635+0000","last_active":"2024-09-10T19:30:31.336551+0000","last_peered":"2024-09-10T19:30:31.336551+0000","last_clean":"2024-09-10T19:30:31.336551+0000","last_became_active":"2024-09-10T19:27:53.006290+0000","last_became_peered":"2024-09-10T19:27:53.006290+0000","last_unstale":"2024-09-10T19:30:31.336551+0000","last_undegraded":"2024-09-10T19:30:31.336551+0000","last_fullsized":"2024-09-10T19:30:31.336551+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"237'33","last_scrub_stamp":"2024-09-10T19:30:05.568555+0000","last_deep_scrub":"80'21","last_deep_scrub_stamp":"2024-09-10T19:11:39.836144+0000","last_clean_scrub_stamp":"2024-09-10T19:30:05.568555+0000","objects_scrubbed":7,"log_size":45,"log_dups_size":0,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:31:29.632123+0000","scrub_duration":0.0023682590000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":60,"num_read_kb":50,"num_write":37,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":154,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"63'2","reported_seq":482,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-09-10T19:30:31.330372+0000","last_change":"2024-09-10T19:30:21.647345+0000","last_active":"2024-09-10T19:30:31.330372+0000","last_peered":"2024-09-10T19:30:31.330372+0000","last_clean":"2024-09-10T19:30:31.330372+0000","last_became_active":"2024-09-10T19:27:53.025802+0000","last_became_peered":"2024-09-10T19:27:53.025802+0000","last_unstale":"2024-09-10T19:30:31.330372+0000","last_undegraded":"2024-09-10T19:30:31.330372+0000","last_fullsized":"2024-09-10T19:30:31.330372+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'2","last_scrub_stamp":"2024-09-10T19:30:21.647282+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:30:21.647282+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:31:27.555961+0000","scrub_duration":0.0017519429999999999,"objects_trimmed":0,"snaptrim_duration":0.00075522999999999996,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":457,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-09-10T19:30:31.334448+0000","last_change":"2024-09-10T19:30:25.207929+0000","last_active":"2024-09-10T19:30:31.334448+0000","last_peered":"2024-09-10T19:30:31.334448+0000","last_clean":"2024-09-10T19:30:31.334448+0000","last_became_active":"2024-09-10T19:16:24.858662+0000","last_became_peered":"2024-09-10T19:16:24.858662+0000","last_unstale":"2024-09-10T19:30:31.334448+0000","last_undegraded":"2024-09-10T19:30:31.334448+0000","last_fullsized":"2024-09-10T19:30:31.334448+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:30:25.207849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:30:25.207849+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:31:37.377965+0000","scrub_duration":0.0026356230000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"74'79","reported_seq":533,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-09-10T19:30:31.341070+0000","last_change":"2024-09-10T19:29:34.083194+0000","last_active":"2024-09-10T19:30:31.341070+0000","last_peered":"2024-09-10T19:30:31.341070+0000","last_clean":"2024-09-10T19:30:31.341070+0000","last_became_active":"2024-09-10T19:27:53.009367+0000","last_became_peered":"2024-09-10T19:27:53.009367+0000","last_unstale":"2024-09-10T19:30:31.341070+0000","last_undegraded":"2024-09-10T19:30:31.341070+0000","last_fullsized":"2024-09-10T19:30:31.341070+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'79","last_scrub_stamp":"2024-09-10T19:29:34.080904+0000","last_deep_scrub":"74'79","last_deep_scrub_stamp":"2024-09-10T19:29:34.080904+0000","last_clean_scrub_stamp":"2024-09-10T19:29:34.080904+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:31:01.369559+0000","scrub_duration":0.0067365319999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":470,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-09-10T19:30:32.974430+0000","last_change":"2024-09-10T19:30:32.974376+0000","last_active":"2024-09-10T19:30:32.974430+0000","last_peered":"2024-09-10T19:30:32.974430+0000","last_clean":"2024-09-10T19:30:32.974430+0000","last_became_active":"2024-09-10T19:24:23.903868+0000","last_became_peered":"2024-09-10T19:24:23.903868+0000","last_unstale":"2024-09-10T19:30:32.974430+0000","last_undegraded":"2024-09-10T19:30:32.974430+0000","last_fullsized":"2024-09-10T19:30:32.974430+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:30:32.973996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:30:32.973996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:31:46.944272+0000","scrub_duration":0.0038225009999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,0],"acting":[7,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":441,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-09-10T19:30:31.334591+0000","last_change":"2024-09-10T19:30:21.186215+0000","last_active":"2024-09-10T19:30:31.334591+0000","last_peered":"2024-09-10T19:30:31.334591+0000","last_clean":"2024-09-10T19:30:31.334591+0000","last_became_active":"2024-09-10T19:10:13.720647+0000","last_became_peered":"2024-09-10T19:10:13.720647+0000","last_unstale":"2024-09-10T19:30:31.334591+0000","last_undegraded":"2024-09-10T19:30:31.334591+0000","last_fullsized":"2024-09-10T19:30:31.334591+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:30:21.186015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:28:29.899579+0000","last_clean_scrub_stamp":"2024-09-10T19:30:21.186015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:31:41.388908+0000","scrub_duration":0.0046533069999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.5","version":"417'16192","reported_seq":16634,"reported_epoch":417,"state":"active+clean","last_fresh":"2024-09-10T19:30:40.559400+0000","last_change":"2024-09-10T19:29:25.827058+0000","last_active":"2024-09-10T19:30:40.559400+0000","last_peered":"2024-09-10T19:30:40.559400+0000","last_clean":"2024-09-10T19:30:40.559400+0000","last_became_active":"2024-09-10T19:27:53.010838+0000","last_became_peered":"2024-09-10T19:27:53.010838+0000","last_unstale":"2024-09-10T19:30:40.559400+0000","last_undegraded":"2024-09-10T19:30:40.559400+0000","last_fullsized":"2024-09-10T19:30:40.559400+0000","mapping_epoch":374,"log_start":"407'11900","ondisk_log_start":"407'11900","created":371,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"401'9408","last_scrub_stamp":"2024-09-10T19:29:25.826994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:27:45.679464+0000","last_clean_scrub_stamp":"2024-09-10T19:29:25.826994+0000","objects_scrubbed":9189,"log_size":4292,"log_dups_size":0,"ondisk_log_size":4292,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:30:50.869426+0000","scrub_duration":2.7029503350000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":151388160,"num_objects":2310,"num_object_clones":0,"num_object_copies":6930,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2310,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16192,"num_write_kb":592064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"414'51","reported_seq":116,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-09-10T19:30:31.337423+0000","last_change":"2024-09-10T19:30:12.541122+0000","last_active":"2024-09-10T19:30:31.337423+0000","last_peered":"2024-09-10T19:30:31.337423+0000","last_clean":"2024-09-10T19:30:31.337423+0000","last_became_active":"2024-09-10T19:27:53.014163+0000","last_became_peered":"2024-09-10T19:27:53.014163+0000","last_unstale":"2024-09-10T19:30:31.337423+0000","last_undegraded":"2024-09-10T19:30:31.337423+0000","last_fullsized":"2024-09-10T19:30:31.337423+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"237'35","last_scrub_stamp":"2024-09-10T19:30:12.541044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:30:12.541044+0000","objects_scrubbed":9,"log_size":51,"log_dups_size":0,"ondisk_log_size":51,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:31:28.477210+0000","scrub_duration":0.003693792,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":80,"num_read_kb":69,"num_write":52,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":349,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-09-10T19:30:31.334136+0000","last_change":"2024-09-10T19:29:34.448268+0000","last_active":"2024-09-10T19:30:31.334136+0000","last_peered":"2024-09-10T19:30:31.334136+0000","last_clean":"2024-09-10T19:30:31.334136+0000","last_became_active":"2024-09-10T19:24:24.913709+0000","last_became_peered":"2024-09-10T19:24:24.913709+0000","last_unstale":"2024-09-10T19:30:31.334136+0000","last_undegraded":"2024-09-10T19:30:31.334136+0000","last_fullsized":"2024-09-10T19:30:31.334136+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:29:34.448187+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:29:34.448187+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:30:58.993915+0000","scrub_duration":0.0038381330000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,1],"acting":[5,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":278,"reported_epoch":417,"state":"active+clean","last_fresh":"2024-09-10T19:30:36.370644+0000","last_change":"2024-09-10T19:30:27.831433+0000","last_active":"2024-09-10T19:30:36.370644+0000","last_peered":"2024-09-10T19:30:36.370644+0000","last_clean":"2024-09-10T19:30:36.370644+0000","last_became_active":"2024-09-10T19:24:23.901415+0000","last_became_peered":"2024-09-10T19:24:23.901415+0000","last_unstale":"2024-09-10T19:30:36.370644+0000","last_undegraded":"2024-09-10T19:30:36.370644+0000","last_fullsized":"2024-09-10T19:30:36.370644+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:30:27.831297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:30:27.831297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:31:47.099106+0000","scrub_duration":0.0028653020000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":481,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-09-10T19:30:31.340988+0000","last_change":"2024-09-10T19:30:21.912161+0000","last_active":"2024-09-10T19:30:31.340988+0000","last_peered":"2024-09-10T19:30:31.340988+0000","last_clean":"2024-09-10T19:30:31.340988+0000","last_became_active":"2024-09-10T19:27:53.008850+0000","last_became_peered":"2024-09-10T19:27:53.008850+0000","last_unstale":"2024-09-10T19:30:31.340988+0000","last_undegraded":"2024-09-10T19:30:31.340988+0000","last_fullsized":"2024-09-10T19:30:31.340988+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:30:21.912110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:21:03.910074+0000","last_clean_scrub_stamp":"2024-09-10T19:30:21.912110+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:31:37.278159+0000","scrub_duration":0.001113969,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,6],"acting":[7,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.4","version":"417'15751","reported_seq":16174,"reported_epoch":417,"state":"active+clean","last_fresh":"2024-09-10T19:30:40.571261+0000","last_change":"2024-09-10T19:29:28.791421+0000","last_active":"2024-09-10T19:30:40.571261+0000","last_peered":"2024-09-10T19:30:40.571261+0000","last_clean":"2024-09-10T19:30:40.571261+0000","last_became_active":"2024-09-10T19:27:53.012510+0000","last_became_peered":"2024-09-10T19:27:53.012510+0000","last_unstale":"2024-09-10T19:30:40.571261+0000","last_undegraded":"2024-09-10T19:30:40.571261+0000","last_fullsized":"2024-09-10T19:30:40.571261+0000","mapping_epoch":374,"log_start":"406'11500","ondisk_log_start":"406'11500","created":371,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"401'9402","last_scrub_stamp":"2024-09-10T19:29:28.791340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:27:45.679464+0000","last_clean_scrub_stamp":"2024-09-10T19:29:28.791340+0000","objects_scrubbed":8715,"log_size":4251,"log_dups_size":0,"ondisk_log_size":4251,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"queued for scrub","scrub_duration":3.694930034,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":145031168,"num_objects":2213,"num_object_clones":0,"num_object_copies":6639,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2213,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15751,"num_write_kb":574848,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"414'21","reported_seq":491,"reported_epoch":417,"state":"active+clean","last_fresh":"2024-09-10T19:30:36.371230+0000","last_change":"2024-09-10T19:30:13.503181+0000","last_active":"2024-09-10T19:30:36.371230+0000","last_peered":"2024-09-10T19:30:36.371230+0000","last_clean":"2024-09-10T19:30:36.371230+0000","last_became_active":"2024-09-10T19:24:44.892294+0000","last_became_peered":"2024-09-10T19:24:44.892294+0000","last_unstale":"2024-09-10T19:30:36.371230+0000","last_undegraded":"2024-09-10T19:30:36.371230+0000","last_fullsized":"2024-09-10T19:30:36.371230+0000","mapping_epoch":314,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":315,"parent":"0.0","parent_split_bits":0,"last_scrub":"237'15","last_scrub_stamp":"2024-09-10T19:30:13.502912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:30:13.502912+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:31:38.767513+0000","scrub_duration":0.0086509359999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":34,"num_read_kb":28,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,4],"acting":[0,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":155,"reported_epoch":417,"state":"active+clean","last_fresh":"2024-09-10T19:30:36.376046+0000","last_change":"2024-09-10T19:29:14.403288+0000","last_active":"2024-09-10T19:30:36.376046+0000","last_peered":"2024-09-10T19:30:36.376046+0000","last_clean":"2024-09-10T19:30:36.376046+0000","last_became_active":"2024-09-10T19:24:01.980348+0000","last_became_peered":"2024-09-10T19:24:01.980348+0000","last_unstale":"2024-09-10T19:30:36.376046+0000","last_undegraded":"2024-09-10T19:30:36.376046+0000","last_fullsized":"2024-09-10T19:30:36.376046+0000","mapping_epoch":292,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":293,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:29:14.403209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:11:44.254513+0000","last_clean_scrub_stamp":"2024-09-10T19:29:14.403209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0046595619999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":345,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-09-10T19:30:31.332881+0000","last_change":"2024-09-10T19:30:05.467928+0000","last_active":"2024-09-10T19:30:31.332881+0000","last_peered":"2024-09-10T19:30:31.332881+0000","last_clean":"2024-09-10T19:30:31.332881+0000","last_became_active":"2024-09-10T19:24:23.891161+0000","last_became_peered":"2024-09-10T19:24:23.891161+0000","last_unstale":"2024-09-10T19:30:31.332881+0000","last_undegraded":"2024-09-10T19:30:31.332881+0000","last_fullsized":"2024-09-10T19:30:31.332881+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:30:05.467845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:30:05.467845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:31:32.032620+0000","scrub_duration":0.0056686540000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":396,"reported_epoch":417,"state":"active+clean","last_fresh":"2024-09-10T19:30:36.371858+0000","last_change":"2024-09-10T19:30:33.974218+0000","last_active":"2024-09-10T19:30:36.371858+0000","last_peered":"2024-09-10T19:30:36.371858+0000","last_clean":"2024-09-10T19:30:36.371858+0000","last_became_active":"2024-09-10T19:24:23.905112+0000","last_became_peered":"2024-09-10T19:24:23.905112+0000","last_unstale":"2024-09-10T19:30:36.371858+0000","last_undegraded":"2024-09-10T19:30:36.371858+0000","last_fullsized":"2024-09-10T19:30:36.371858+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:30:33.974134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:30:33.974134+0000","last_clean_scrub_stamp":"2024-09-10T19:30:33.974134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:31:46.881624+0000","scrub_duration":0.0063507879999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.7","version":"417'15879","reported_seq":16275,"reported_epoch":417,"state":"active+clean","last_fresh":"2024-09-10T19:30:40.576930+0000","last_change":"2024-09-10T19:29:34.192828+0000","last_active":"2024-09-10T19:30:40.576930+0000","last_peered":"2024-09-10T19:30:40.576930+0000","last_clean":"2024-09-10T19:30:40.576930+0000","last_became_active":"2024-09-10T19:27:46.707393+0000","last_became_peered":"2024-09-10T19:27:46.707393+0000","last_unstale":"2024-09-10T19:30:40.576930+0000","last_undegraded":"2024-09-10T19:30:40.576930+0000","last_fullsized":"2024-09-10T19:30:40.576930+0000","mapping_epoch":371,"log_start":"406'11200","ondisk_log_start":"406'11200","created":371,"last_epoch_clean":372,"parent":"0.0","parent_split_bits":0,"last_scrub":"404'9864","last_scrub_stamp":"2024-09-10T19:29:34.192737+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:27:45.679464+0000","last_clean_scrub_stamp":"2024-09-10T19:29:34.192737+0000","objects_scrubbed":8292,"log_size":4679,"log_dups_size":0,"ondisk_log_size":4679,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:30:49.858203+0000","scrub_duration":2.7792294690000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":141492224,"num_objects":2159,"num_object_clones":0,"num_object_copies":6477,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2159,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15879,"num_write_kb":577216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,4],"acting":[5,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.8","version":"414'46","reported_seq":555,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-09-10T19:30:31.331401+0000","last_change":"2024-09-10T19:30:06.380329+0000","last_active":"2024-09-10T19:30:31.331401+0000","last_peered":"2024-09-10T19:30:31.331401+0000","last_clean":"2024-09-10T19:30:31.331401+0000","last_became_active":"2024-09-10T19:27:53.004858+0000","last_became_peered":"2024-09-10T19:27:53.004858+0000","last_unstale":"2024-09-10T19:30:31.331401+0000","last_undegraded":"2024-09-10T19:30:31.331401+0000","last_fullsized":"2024-09-10T19:30:31.331401+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"237'34","last_scrub_stamp":"2024-09-10T19:30:06.380238+0000","last_deep_scrub":"220'22","last_deep_scrub_stamp":"2024-09-10T19:19:24.597866+0000","last_clean_scrub_stamp":"2024-09-10T19:30:06.380238+0000","objects_scrubbed":7,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:31:27.961508+0000","scrub_duration":0.0065648299999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":62,"num_read_kb":53,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":345,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-09-10T19:30:31.332237+0000","last_change":"2024-09-10T19:30:25.698629+0000","last_active":"2024-09-10T19:30:31.332237+0000","last_peered":"2024-09-10T19:30:31.332237+0000","last_clean":"2024-09-10T19:30:31.332237+0000","last_became_active":"2024-09-10T19:27:53.016898+0000","last_became_peered":"2024-09-10T19:27:53.016898+0000","last_unstale":"2024-09-10T19:30:31.332237+0000","last_undegraded":"2024-09-10T19:30:31.332237+0000","last_fullsized":"2024-09-10T19:30:31.332237+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:30:25.698575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:30:25.698575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:31:48.430610+0000","scrub_duration":0.008668004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.9","version":"376'11","reported_seq":736,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-09-10T19:30:35.968915+0000","last_change":"2024-09-10T19:30:09.879798+0000","last_active":"2024-09-10T19:30:35.968915+0000","last_peered":"2024-09-10T19:30:35.968915+0000","last_clean":"2024-09-10T19:30:35.968915+0000","last_became_active":"2024-09-10T19:27:53.009810+0000","last_became_peered":"2024-09-10T19:27:53.009810+0000","last_unstale":"2024-09-10T19:30:35.968915+0000","last_undegraded":"2024-09-10T19:30:35.968915+0000","last_fullsized":"2024-09-10T19:30:35.968915+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"376'11","last_scrub_stamp":"2024-09-10T19:30:09.879695+0000","last_deep_scrub":"80'9","last_deep_scrub_stamp":"2024-09-10T19:25:29.833271+0000","last_clean_scrub_stamp":"2024-09-10T19:30:09.879695+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:31:37.107588+0000","scrub_duration":0.0040800790000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":458,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-09-10T19:30:31.334727+0000","last_change":"2024-09-10T19:29:27.408515+0000","last_active":"2024-09-10T19:30:31.334727+0000","last_peered":"2024-09-10T19:30:31.334727+0000","last_clean":"2024-09-10T19:30:31.334727+0000","last_became_active":"2024-09-10T19:27:53.015957+0000","last_became_peered":"2024-09-10T19:27:53.015957+0000","last_unstale":"2024-09-10T19:30:31.334727+0000","last_undegraded":"2024-09-10T19:30:31.334727+0000","last_fullsized":"2024-09-10T19:30:31.334727+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:29:27.408374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:29:27.408374+0000","last_clean_scrub_stamp":"2024-09-10T19:29:27.408374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:30:50.070590+0000","scrub_duration":0.0088258940000000008,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.6","version":"417'16120","reported_seq":16523,"reported_epoch":417,"state":"active+clean","last_fresh":"2024-09-10T19:30:42.064403+0000","last_change":"2024-09-10T19:29:17.664714+0000","last_active":"2024-09-10T19:30:42.064403+0000","last_peered":"2024-09-10T19:30:42.064403+0000","last_clean":"2024-09-10T19:30:42.064403+0000","last_became_active":"2024-09-10T19:27:46.704823+0000","last_became_peered":"2024-09-10T19:27:46.704823+0000","last_unstale":"2024-09-10T19:30:42.064403+0000","last_undegraded":"2024-09-10T19:30:42.064403+0000","last_fullsized":"2024-09-10T19:30:42.064403+0000","mapping_epoch":371,"log_start":"406'11500","ondisk_log_start":"406'11500","created":371,"last_epoch_clean":372,"parent":"0.0","parent_split_bits":0,"last_scrub":"400'8581","last_scrub_stamp":"2024-09-10T19:29:17.664631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:27:45.679464+0000","last_clean_scrub_stamp":"2024-09-10T19:29:17.664631+0000","objects_scrubbed":8456,"log_size":4620,"log_dups_size":0,"ondisk_log_size":4620,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"queued for scrub","scrub_duration":2.9535849939999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":139722752,"num_objects":2132,"num_object_clones":0,"num_object_copies":6396,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2132,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16120,"num_write_kb":584064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.9","version":"414'24","reported_seq":529,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-09-10T19:30:31.333143+0000","last_change":"2024-09-10T19:29:18.241176+0000","last_active":"2024-09-10T19:30:31.333143+0000","last_peered":"2024-09-10T19:30:31.333143+0000","last_clean":"2024-09-10T19:30:31.333143+0000","last_became_active":"2024-09-10T19:24:01.974878+0000","last_became_peered":"2024-09-10T19:24:01.974878+0000","last_unstale":"2024-09-10T19:30:31.333143+0000","last_undegraded":"2024-09-10T19:30:31.333143+0000","last_fullsized":"2024-09-10T19:30:31.333143+0000","mapping_epoch":292,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":293,"parent":"0.0","parent_split_bits":0,"last_scrub":"237'18","last_scrub_stamp":"2024-09-10T19:29:18.241077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:29:18.241077+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:30:43.947723+0000","scrub_duration":0.0095846869999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.f","version":"59'2","reported_seq":469,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-09-10T19:30:31.333707+0000","last_change":"2024-09-10T19:30:15.444681+0000","last_active":"2024-09-10T19:30:31.333707+0000","last_peered":"2024-09-10T19:30:31.333707+0000","last_clean":"2024-09-10T19:30:31.333707+0000","last_became_active":"2024-09-10T19:24:23.893796+0000","last_became_peered":"2024-09-10T19:24:23.893796+0000","last_unstale":"2024-09-10T19:30:31.333707+0000","last_undegraded":"2024-09-10T19:30:31.333707+0000","last_fullsized":"2024-09-10T19:30:31.333707+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'2","last_scrub_stamp":"2024-09-10T19:30:15.444556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:30:15.444556+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:31:42.533307+0000","scrub_duration":0.003937945,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":14,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":380,"reported_epoch":417,"state":"active+clean","last_fresh":"2024-09-10T19:30:36.374481+0000","last_change":"2024-09-10T19:29:16.702372+0000","last_active":"2024-09-10T19:30:36.374481+0000","last_peered":"2024-09-10T19:30:36.374481+0000","last_clean":"2024-09-10T19:30:36.374481+0000","last_became_active":"2024-09-10T19:24:24.914252+0000","last_became_peered":"2024-09-10T19:24:24.914252+0000","last_unstale":"2024-09-10T19:30:36.374481+0000","last_undegraded":"2024-09-10T19:30:36.374481+0000","last_fullsized":"2024-09-10T19:30:36.374481+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:29:16.702283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:11:46.587846+0000","last_clean_scrub_stamp":"2024-09-10T19:29:16.702283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0047568310000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,5],"acting":[0,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":458,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-09-10T19:30:31.329740+0000","last_change":"2024-09-10T19:30:17.554426+0000","last_active":"2024-09-10T19:30:31.329740+0000","last_peered":"2024-09-10T19:30:31.329740+0000","last_clean":"2024-09-10T19:30:31.329740+0000","last_became_active":"2024-09-10T19:24:01.971131+0000","last_became_peered":"2024-09-10T19:24:01.971131+0000","last_unstale":"2024-09-10T19:30:31.329740+0000","last_undegraded":"2024-09-10T19:30:31.329740+0000","last_fullsized":"2024-09-10T19:30:31.329740+0000","mapping_epoch":292,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":293,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:30:17.554352+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:30:17.554352+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:31:26.930983+0000","scrub_duration":0.001698869,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.16","version":"414'21","reported_seq":445,"reported_epoch":417,"state":"active+clean","last_fresh":"2024-09-10T19:30:36.373194+0000","last_change":"2024-09-10T19:29:13.724314+0000","last_active":"2024-09-10T19:30:36.373194+0000","last_peered":"2024-09-10T19:30:36.373194+0000","last_clean":"2024-09-10T19:30:36.373194+0000","last_became_active":"2024-09-10T19:24:23.901610+0000","last_became_peered":"2024-09-10T19:24:23.901610+0000","last_unstale":"2024-09-10T19:30:36.373194+0000","last_undegraded":"2024-09-10T19:30:36.373194+0000","last_fullsized":"2024-09-10T19:30:36.373194+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"237'15","last_scrub_stamp":"2024-09-10T19:29:13.724222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:29:13.724222+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0074413179999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":34,"num_read_kb":28,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":179,"reported_epoch":417,"state":"active+clean","last_fresh":"2024-09-10T19:30:36.376135+0000","last_change":"2024-09-10T19:29:18.504177+0000","last_active":"2024-09-10T19:30:36.376135+0000","last_peered":"2024-09-10T19:30:36.376135+0000","last_clean":"2024-09-10T19:30:36.376135+0000","last_became_active":"2024-09-10T19:24:24.894815+0000","last_became_peered":"2024-09-10T19:24:24.894815+0000","last_unstale":"2024-09-10T19:30:36.376135+0000","last_undegraded":"2024-09-10T19:30:36.376135+0000","last_fullsized":"2024-09-10T19:30:36.376135+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:29:18.504104+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:11:41.458523+0000","last_clean_scrub_stamp":"2024-09-10T19:29:18.504104+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:30:42.921317+0000","scrub_duration":0.0057679100000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,4],"acting":[1,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":457,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-09-10T19:30:31.331850+0000","last_change":"2024-09-10T19:29:29.071593+0000","last_active":"2024-09-10T19:30:31.331850+0000","last_peered":"2024-09-10T19:30:31.331850+0000","last_clean":"2024-09-10T19:30:31.331850+0000","last_became_active":"2024-09-10T19:24:01.970325+0000","last_became_peered":"2024-09-10T19:24:01.970325+0000","last_unstale":"2024-09-10T19:30:31.331850+0000","last_undegraded":"2024-09-10T19:30:31.331850+0000","last_fullsized":"2024-09-10T19:30:31.331850+0000","mapping_epoch":292,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":293,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:29:29.071516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:24:09.367654+0000","last_clean_scrub_stamp":"2024-09-10T19:29:29.071516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:30:54.608866+0000","scrub_duration":0.0048760050000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"66'1","reported_seq":345,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-09-10T19:30:31.333387+0000","last_change":"2024-09-10T19:29:28.371890+0000","last_active":"2024-09-10T19:30:31.333387+0000","last_peered":"2024-09-10T19:30:31.333387+0000","last_clean":"2024-09-10T19:30:31.333387+0000","last_became_active":"2024-09-10T19:24:23.897858+0000","last_became_peered":"2024-09-10T19:24:23.897858+0000","last_unstale":"2024-09-10T19:30:31.333387+0000","last_undegraded":"2024-09-10T19:30:31.333387+0000","last_fullsized":"2024-09-10T19:30:31.333387+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'1","last_scrub_stamp":"2024-09-10T19:29:28.371731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:29:28.371731+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:30:51.061816+0000","scrub_duration":0.028433127999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":13,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":82,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-09-10T19:30:34.487404+0000","last_change":"2024-09-10T19:30:34.487341+0000","last_active":"2024-09-10T19:30:34.487404+0000","last_peered":"2024-09-10T19:30:34.487404+0000","last_clean":"2024-09-10T19:30:34.487404+0000","last_became_active":"2024-09-10T19:27:53.007487+0000","last_became_peered":"2024-09-10T19:27:53.007487+0000","last_unstale":"2024-09-10T19:30:34.487404+0000","last_undegraded":"2024-09-10T19:30:34.487404+0000","last_fullsized":"2024-09-10T19:30:34.487404+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:30:34.487271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:30:34.487271+0000","last_clean_scrub_stamp":"2024-09-10T19:30:34.487271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:31:58.958373+0000","scrub_duration":0.007547064,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"414'18","reported_seq":513,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-09-10T19:30:31.332040+0000","last_change":"2024-09-10T19:30:04.390756+0000","last_active":"2024-09-10T19:30:31.332040+0000","last_peered":"2024-09-10T19:30:31.332040+0000","last_clean":"2024-09-10T19:30:31.332040+0000","last_became_active":"2024-09-10T19:24:01.971492+0000","last_became_peered":"2024-09-10T19:24:01.971492+0000","last_unstale":"2024-09-10T19:30:31.332040+0000","last_undegraded":"2024-09-10T19:30:31.332040+0000","last_fullsized":"2024-09-10T19:30:31.332040+0000","mapping_epoch":292,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":293,"parent":"0.0","parent_split_bits":0,"last_scrub":"237'12","last_scrub_stamp":"2024-09-10T19:30:04.390680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:30:04.390680+0000","objects_scrubbed":3,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:31:06.735608+0000","scrub_duration":0.0029448790000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":27,"num_read_kb":24,"num_write":18,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":460,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-09-10T19:30:31.334792+0000","last_change":"2024-09-10T19:30:22.162341+0000","last_active":"2024-09-10T19:30:31.334792+0000","last_peered":"2024-09-10T19:30:31.334792+0000","last_clean":"2024-09-10T19:30:31.334792+0000","last_became_active":"2024-09-10T19:24:01.978215+0000","last_became_peered":"2024-09-10T19:24:01.978215+0000","last_unstale":"2024-09-10T19:30:31.334792+0000","last_undegraded":"2024-09-10T19:30:31.334792+0000","last_fullsized":"2024-09-10T19:30:31.334792+0000","mapping_epoch":292,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":293,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:30:22.162288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:11:23.264793+0000","last_clean_scrub_stamp":"2024-09-10T19:30:22.162288+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:31:34.322359+0000","scrub_duration":0.0055022539999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":461,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-09-10T19:30:31.341264+0000","last_change":"2024-09-10T19:29:17.920965+0000","last_active":"2024-09-10T19:30:31.341264+0000","last_peered":"2024-09-10T19:30:31.341264+0000","last_clean":"2024-09-10T19:30:31.341264+0000","last_became_active":"2024-09-10T19:27:53.010120+0000","last_became_peered":"2024-09-10T19:27:53.010120+0000","last_unstale":"2024-09-10T19:30:31.341264+0000","last_undegraded":"2024-09-10T19:30:31.341264+0000","last_fullsized":"2024-09-10T19:30:31.341264+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:29:17.920880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:29:17.920880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0025791519999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"414'35","reported_seq":558,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-09-10T19:30:31.329913+0000","last_change":"2024-09-10T19:30:23.678519+0000","last_active":"2024-09-10T19:30:31.329913+0000","last_peered":"2024-09-10T19:30:31.329913+0000","last_clean":"2024-09-10T19:30:31.329913+0000","last_became_active":"2024-09-10T19:24:01.971173+0000","last_became_peered":"2024-09-10T19:24:01.971173+0000","last_unstale":"2024-09-10T19:30:31.329913+0000","last_undegraded":"2024-09-10T19:30:31.329913+0000","last_fullsized":"2024-09-10T19:30:31.329913+0000","mapping_epoch":292,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":293,"parent":"0.0","parent_split_bits":0,"last_scrub":"414'35","last_scrub_stamp":"2024-09-10T19:30:23.678420+0000","last_deep_scrub":"80'15","last_deep_scrub_stamp":"2024-09-10T19:11:41.719310+0000","last_clean_scrub_stamp":"2024-09-10T19:30:23.678420+0000","objects_scrubbed":5,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:31:41.464166+0000","scrub_duration":0.0043333590000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":45,"num_read_kb":40,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":437,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-09-10T19:30:31.331678+0000","last_change":"2024-09-10T19:29:36.523640+0000","last_active":"2024-09-10T19:30:31.331678+0000","last_peered":"2024-09-10T19:30:31.331678+0000","last_clean":"2024-09-10T19:30:31.331678+0000","last_became_active":"2024-09-10T19:10:07.685870+0000","last_became_peered":"2024-09-10T19:10:07.685870+0000","last_unstale":"2024-09-10T19:30:31.331678+0000","last_undegraded":"2024-09-10T19:30:31.331678+0000","last_fullsized":"2024-09-10T19:30:31.331678+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:29:36.523540+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:29:36.523540+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:30:36.529392+0000","scrub_duration":0.0026209499999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"372'16","reported_seq":752,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-09-10T19:30:35.968529+0000","last_change":"2024-09-10T19:29:26.417697+0000","last_active":"2024-09-10T19:30:35.968529+0000","last_peered":"2024-09-10T19:30:35.968529+0000","last_clean":"2024-09-10T19:30:35.968529+0000","last_became_active":"2024-09-10T19:27:46.703433+0000","last_became_peered":"2024-09-10T19:27:46.703433+0000","last_unstale":"2024-09-10T19:30:35.968529+0000","last_undegraded":"2024-09-10T19:30:35.968529+0000","last_fullsized":"2024-09-10T19:30:35.968529+0000","mapping_epoch":371,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":372,"parent":"0.0","parent_split_bits":0,"last_scrub":"372'16","last_scrub_stamp":"2024-09-10T19:29:26.415252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:29:26.415252+0000","objects_scrubbed":1,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:30:40.542191+0000","scrub_duration":0.007524256,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,0],"acting":[5,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":392,"reported_epoch":417,"state":"active+clean","last_fresh":"2024-09-10T19:30:36.373266+0000","last_change":"2024-09-10T19:29:34.578620+0000","last_active":"2024-09-10T19:30:36.373266+0000","last_peered":"2024-09-10T19:30:36.373266+0000","last_clean":"2024-09-10T19:30:36.373266+0000","last_became_active":"2024-09-10T19:24:23.900513+0000","last_became_peered":"2024-09-10T19:24:23.900513+0000","last_unstale":"2024-09-10T19:30:36.373266+0000","last_undegraded":"2024-09-10T19:30:36.373266+0000","last_fullsized":"2024-09-10T19:30:36.373266+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:29:34.578469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:29:34.578469+0000","last_clean_scrub_stamp":"2024-09-10T19:29:34.578469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0058680599999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"414'21","reported_seq":509,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-09-10T19:30:31.334186+0000","last_change":"2024-09-10T19:29:32.382258+0000","last_active":"2024-09-10T19:30:31.334186+0000","last_peered":"2024-09-10T19:30:31.334186+0000","last_clean":"2024-09-10T19:30:31.334186+0000","last_became_active":"2024-09-10T19:16:24.859740+0000","last_became_peered":"2024-09-10T19:16:24.859740+0000","last_unstale":"2024-09-10T19:30:31.334186+0000","last_undegraded":"2024-09-10T19:30:31.334186+0000","last_fullsized":"2024-09-10T19:30:31.334186+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"237'15","last_scrub_stamp":"2024-09-10T19:29:32.382176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:29:32.382176+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:30:39.846865+0000","scrub_duration":0.0057339599999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":395,"reported_epoch":417,"state":"active+clean","last_fresh":"2024-09-10T19:30:36.370714+0000","last_change":"2024-09-10T19:29:11.750820+0000","last_active":"2024-09-10T19:30:36.370714+0000","last_peered":"2024-09-10T19:30:36.370714+0000","last_clean":"2024-09-10T19:30:36.370714+0000","last_became_active":"2024-09-10T19:24:23.895163+0000","last_became_peered":"2024-09-10T19:24:23.895163+0000","last_unstale":"2024-09-10T19:30:36.370714+0000","last_undegraded":"2024-09-10T19:30:36.370714+0000","last_fullsized":"2024-09-10T19:30:36.370714+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:29:11.750709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:29:11.750709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0029571649999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"297'11","reported_seq":716,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-09-10T19:30:35.968520+0000","last_change":"2024-09-10T19:29:11.159023+0000","last_active":"2024-09-10T19:30:35.968520+0000","last_peered":"2024-09-10T19:30:35.968520+0000","last_clean":"2024-09-10T19:30:35.968520+0000","last_became_active":"2024-09-10T19:24:01.974816+0000","last_became_peered":"2024-09-10T19:24:01.974816+0000","last_unstale":"2024-09-10T19:30:35.968520+0000","last_undegraded":"2024-09-10T19:30:35.968520+0000","last_fullsized":"2024-09-10T19:30:35.968520+0000","mapping_epoch":292,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":293,"parent":"0.0","parent_split_bits":0,"last_scrub":"297'11","last_scrub_stamp":"2024-09-10T19:29:11.158902+0000","last_deep_scrub":"297'11","last_deep_scrub_stamp":"2024-09-10T19:24:13.318130+0000","last_clean_scrub_stamp":"2024-09-10T19:29:11.158902+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.014823852,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":450,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-09-10T19:30:31.333515+0000","last_change":"2024-09-10T19:29:32.361240+0000","last_active":"2024-09-10T19:30:31.333515+0000","last_peered":"2024-09-10T19:30:31.333515+0000","last_clean":"2024-09-10T19:30:31.333515+0000","last_became_active":"2024-09-10T19:24:23.903779+0000","last_became_peered":"2024-09-10T19:24:23.903779+0000","last_unstale":"2024-09-10T19:30:31.333515+0000","last_undegraded":"2024-09-10T19:30:31.333515+0000","last_fullsized":"2024-09-10T19:30:31.333515+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:29:32.361029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:29:32.361029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:30:58.523472+0000","scrub_duration":0.010203366,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.12","version":"414'25","reported_seq":238,"reported_epoch":417,"state":"active+clean","last_fresh":"2024-09-10T19:30:36.376274+0000","last_change":"2024-09-10T19:30:23.274575+0000","last_active":"2024-09-10T19:30:36.376274+0000","last_peered":"2024-09-10T19:30:36.376274+0000","last_clean":"2024-09-10T19:30:36.376274+0000","last_became_active":"2024-09-10T19:27:53.017161+0000","last_became_peered":"2024-09-10T19:27:53.017161+0000","last_unstale":"2024-09-10T19:30:36.376274+0000","last_undegraded":"2024-09-10T19:30:36.376274+0000","last_fullsized":"2024-09-10T19:30:36.376274+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"414'25","last_scrub_stamp":"2024-09-10T19:30:23.274493+0000","last_deep_scrub":"79'13","last_deep_scrub_stamp":"2024-09-10T19:11:22.072400+0000","last_clean_scrub_stamp":"2024-09-10T19:30:23.274493+0000","objects_scrubbed":4,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:31:42.647014+0000","scrub_duration":0.0027214610000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":34,"num_read_kb":28,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":550,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"59'1","reported_seq":86,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-09-10T19:30:31.402402+0000","last_change":"2024-09-10T19:30:31.402122+0000","last_active":"2024-09-10T19:30:31.402402+0000","last_peered":"2024-09-10T19:30:31.402402+0000","last_clean":"2024-09-10T19:30:31.402402+0000","last_became_active":"2024-09-10T19:27:53.018470+0000","last_became_peered":"2024-09-10T19:27:53.018470+0000","last_unstale":"2024-09-10T19:30:31.402402+0000","last_undegraded":"2024-09-10T19:30:31.402402+0000","last_fullsized":"2024-09-10T19:30:31.402402+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'1","last_scrub_stamp":"2024-09-10T19:30:31.401664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:30:31.401664+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:31:59.702734+0000","scrub_duration":0.0059519009999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":362,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":724,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":471,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-09-10T19:30:31.330020+0000","last_change":"2024-09-10T19:29:19.145340+0000","last_active":"2024-09-10T19:30:31.330020+0000","last_peered":"2024-09-10T19:30:31.330020+0000","last_clean":"2024-09-10T19:30:31.330020+0000","last_became_active":"2024-09-10T19:24:24.923804+0000","last_became_peered":"2024-09-10T19:24:24.923804+0000","last_unstale":"2024-09-10T19:30:31.330020+0000","last_undegraded":"2024-09-10T19:30:31.330020+0000","last_fullsized":"2024-09-10T19:30:31.330020+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:29:19.145160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:29:19.145160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:30:43.891130+0000","scrub_duration":0.0019117959999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":411,"reported_epoch":417,"state":"active+clean","last_fresh":"2024-09-10T19:30:36.374173+0000","last_change":"2024-09-10T19:29:28.581248+0000","last_active":"2024-09-10T19:30:36.374173+0000","last_peered":"2024-09-10T19:30:36.374173+0000","last_clean":"2024-09-10T19:30:36.374173+0000","last_became_active":"2024-09-10T19:24:24.916105+0000","last_became_peered":"2024-09-10T19:24:24.916105+0000","last_unstale":"2024-09-10T19:30:36.374173+0000","last_undegraded":"2024-09-10T19:30:36.374173+0000","last_fullsized":"2024-09-10T19:30:36.374173+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:29:28.581177+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:25:27.412579+0000","last_clean_scrub_stamp":"2024-09-10T19:29:28.581177+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:30:47.919026+0000","scrub_duration":0.0049057299999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"414'27","reported_seq":575,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-09-10T19:30:31.332957+0000","last_change":"2024-09-10T19:29:26.356083+0000","last_active":"2024-09-10T19:30:31.332957+0000","last_peered":"2024-09-10T19:30:31.332957+0000","last_clean":"2024-09-10T19:30:31.332957+0000","last_became_active":"2024-09-10T19:27:53.016518+0000","last_became_peered":"2024-09-10T19:27:53.016518+0000","last_unstale":"2024-09-10T19:30:31.332957+0000","last_undegraded":"2024-09-10T19:30:31.332957+0000","last_fullsized":"2024-09-10T19:30:31.332957+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"237'19","last_scrub_stamp":"2024-09-10T19:29:26.355998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:29:26.355998+0000","objects_scrubbed":5,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:30:39.698620+0000","scrub_duration":0.030792693999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":37,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":179,"reported_epoch":417,"state":"active+clean","last_fresh":"2024-09-10T19:30:36.375972+0000","last_change":"2024-09-10T19:29:09.345677+0000","last_active":"2024-09-10T19:30:36.375972+0000","last_peered":"2024-09-10T19:30:36.375972+0000","last_clean":"2024-09-10T19:30:36.375972+0000","last_became_active":"2024-09-10T19:24:24.909540+0000","last_became_peered":"2024-09-10T19:24:24.909540+0000","last_unstale":"2024-09-10T19:30:36.375972+0000","last_undegraded":"2024-09-10T19:30:36.375972+0000","last_fullsized":"2024-09-10T19:30:36.375972+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:29:09.345588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:26:31.508168+0000","last_clean_scrub_stamp":"2024-09-10T19:29:09.345588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.008652847,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":155,"reported_epoch":417,"state":"active+clean","last_fresh":"2024-09-10T19:30:36.376403+0000","last_change":"2024-09-10T19:28:29.495516+0000","last_active":"2024-09-10T19:30:36.376403+0000","last_peered":"2024-09-10T19:30:36.376403+0000","last_clean":"2024-09-10T19:30:36.376403+0000","last_became_active":"2024-09-10T19:24:01.975520+0000","last_became_peered":"2024-09-10T19:24:01.975520+0000","last_unstale":"2024-09-10T19:30:36.376403+0000","last_undegraded":"2024-09-10T19:30:36.376403+0000","last_fullsized":"2024-09-10T19:30:36.376403+0000","mapping_epoch":292,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":293,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:28:29.495437+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:28:29.495437+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0066786709999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":438,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-09-10T19:30:31.329197+0000","last_change":"2024-09-10T19:29:33.122658+0000","last_active":"2024-09-10T19:30:31.329197+0000","last_peered":"2024-09-10T19:30:31.329197+0000","last_clean":"2024-09-10T19:30:31.329197+0000","last_became_active":"2024-09-10T19:24:23.907136+0000","last_became_peered":"2024-09-10T19:24:23.907136+0000","last_unstale":"2024-09-10T19:30:31.329197+0000","last_undegraded":"2024-09-10T19:30:31.329197+0000","last_fullsized":"2024-09-10T19:30:31.329197+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:29:33.121149+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:29:33.121149+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:31:00.857785+0000","scrub_duration":0.0077727279999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"414'14","reported_seq":526,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-09-10T19:30:31.331855+0000","last_change":"2024-09-10T19:30:06.362408+0000","last_active":"2024-09-10T19:30:31.331855+0000","last_peered":"2024-09-10T19:30:31.331855+0000","last_clean":"2024-09-10T19:30:31.331855+0000","last_became_active":"2024-09-10T19:27:53.024782+0000","last_became_peered":"2024-09-10T19:27:53.024782+0000","last_unstale":"2024-09-10T19:30:31.331855+0000","last_undegraded":"2024-09-10T19:30:31.331855+0000","last_fullsized":"2024-09-10T19:30:31.331855+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"237'10","last_scrub_stamp":"2024-09-10T19:30:06.361528+0000","last_deep_scrub":"222'6","last_deep_scrub_stamp":"2024-09-10T19:19:30.328841+0000","last_clean_scrub_stamp":"2024-09-10T19:30:06.361528+0000","objects_scrubbed":2,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:31:24.920334+0000","scrub_duration":0.0072268020000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":18,"num_read_kb":16,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":487,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-09-10T19:30:31.334923+0000","last_change":"2024-09-10T19:30:20.146538+0000","last_active":"2024-09-10T19:30:31.334923+0000","last_peered":"2024-09-10T19:30:31.334923+0000","last_clean":"2024-09-10T19:30:31.334923+0000","last_became_active":"2024-09-10T19:27:53.000789+0000","last_became_peered":"2024-09-10T19:27:53.000789+0000","last_unstale":"2024-09-10T19:30:31.334923+0000","last_undegraded":"2024-09-10T19:30:31.334923+0000","last_fullsized":"2024-09-10T19:30:31.334923+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:30:20.146470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:30:20.146470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:31:21.242258+0000","scrub_duration":0.0032897199999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,1],"acting":[5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":82,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-09-10T19:30:32.432911+0000","last_change":"2024-09-10T19:30:32.432843+0000","last_active":"2024-09-10T19:30:32.432911+0000","last_peered":"2024-09-10T19:30:32.432911+0000","last_clean":"2024-09-10T19:30:32.432911+0000","last_became_active":"2024-09-10T19:27:53.003591+0000","last_became_peered":"2024-09-10T19:27:53.003591+0000","last_unstale":"2024-09-10T19:30:32.432911+0000","last_undegraded":"2024-09-10T19:30:32.432911+0000","last_fullsized":"2024-09-10T19:30:32.432911+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:30:32.432739+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:11:31.837944+0000","last_clean_scrub_stamp":"2024-09-10T19:30:32.432739+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:31:40.637363+0000","scrub_duration":0.010234006,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":461,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-09-10T19:30:31.340930+0000","last_change":"2024-09-10T19:29:33.126317+0000","last_active":"2024-09-10T19:30:31.340930+0000","last_peered":"2024-09-10T19:30:31.340930+0000","last_clean":"2024-09-10T19:30:31.340930+0000","last_became_active":"2024-09-10T19:16:24.852356+0000","last_became_peered":"2024-09-10T19:16:24.852356+0000","last_unstale":"2024-09-10T19:30:31.340930+0000","last_undegraded":"2024-09-10T19:30:31.340930+0000","last_fullsized":"2024-09-10T19:30:31.340930+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:29:33.125081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:29:33.125081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.019182421000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,3],"acting":[7,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.11","version":"414'27","reported_seq":283,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-09-10T19:30:31.329870+0000","last_change":"2024-09-10T19:29:15.225773+0000","last_active":"2024-09-10T19:30:31.329870+0000","last_peered":"2024-09-10T19:30:31.329870+0000","last_clean":"2024-09-10T19:30:31.329870+0000","last_became_active":"2024-09-10T19:27:53.018213+0000","last_became_peered":"2024-09-10T19:27:53.018213+0000","last_unstale":"2024-09-10T19:30:31.329870+0000","last_undegraded":"2024-09-10T19:30:31.329870+0000","last_fullsized":"2024-09-10T19:30:31.329870+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"237'19","last_scrub_stamp":"2024-09-10T19:29:15.225687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:29:15.225687+0000","objects_scrubbed":5,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:30:36.896555+0000","scrub_duration":0.015395733,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":37,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":72,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-09-10T19:30:31.336609+0000","last_change":"2024-09-10T19:30:10.528365+0000","last_active":"2024-09-10T19:30:31.336609+0000","last_peered":"2024-09-10T19:30:31.336609+0000","last_clean":"2024-09-10T19:30:31.336609+0000","last_became_active":"2024-09-10T19:27:53.008197+0000","last_became_peered":"2024-09-10T19:27:53.008197+0000","last_unstale":"2024-09-10T19:30:31.336609+0000","last_undegraded":"2024-09-10T19:30:31.336609+0000","last_fullsized":"2024-09-10T19:30:31.336609+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:30:10.528284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:13:05.876655+0000","last_clean_scrub_stamp":"2024-09-10T19:30:10.528284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:31:10.861466+0000","scrub_duration":0.002704536,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":463,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-09-10T19:30:31.340738+0000","last_change":"2024-09-10T19:30:11.797823+0000","last_active":"2024-09-10T19:30:31.340738+0000","last_peered":"2024-09-10T19:30:31.340738+0000","last_clean":"2024-09-10T19:30:31.340738+0000","last_became_active":"2024-09-10T19:27:53.008444+0000","last_became_peered":"2024-09-10T19:27:53.008444+0000","last_unstale":"2024-09-10T19:30:31.340738+0000","last_undegraded":"2024-09-10T19:30:31.340738+0000","last_fullsized":"2024-09-10T19:30:31.340738+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:30:11.797669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:11:57.112092+0000","last_clean_scrub_stamp":"2024-09-10T19:30:11.797669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:31:28.693832+0000","scrub_duration":0.002187483,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":458,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-09-10T19:30:31.329664+0000","last_change":"2024-09-10T19:29:30.096377+0000","last_active":"2024-09-10T19:30:31.329664+0000","last_peered":"2024-09-10T19:30:31.329664+0000","last_clean":"2024-09-10T19:30:31.329664+0000","last_became_active":"2024-09-10T19:27:53.025650+0000","last_became_peered":"2024-09-10T19:27:53.025650+0000","last_unstale":"2024-09-10T19:30:31.329664+0000","last_undegraded":"2024-09-10T19:30:31.329664+0000","last_fullsized":"2024-09-10T19:30:31.329664+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:29:30.096297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:26:29.959202+0000","last_clean_scrub_stamp":"2024-09-10T19:29:30.096297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:30:44.930869+0000","scrub_duration":0.012023613000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"80'2","reported_seq":467,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-09-10T19:30:31.343164+0000","last_change":"2024-09-10T19:30:31.056345+0000","last_active":"2024-09-10T19:30:31.343164+0000","last_peered":"2024-09-10T19:30:31.343164+0000","last_clean":"2024-09-10T19:30:31.343164+0000","last_became_active":"2024-09-10T19:16:24.851220+0000","last_became_peered":"2024-09-10T19:16:24.851220+0000","last_unstale":"2024-09-10T19:30:31.343164+0000","last_undegraded":"2024-09-10T19:30:31.343164+0000","last_fullsized":"2024-09-10T19:30:31.343164+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"80'2","last_scrub_stamp":"2024-09-10T19:30:31.056271+0000","last_deep_scrub":"80'2","last_deep_scrub_stamp":"2024-09-10T19:11:30.272043+0000","last_clean_scrub_stamp":"2024-09-10T19:30:31.056271+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:31:41.406250+0000","scrub_duration":0.0025698990000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":369,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":369,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,3],"acting":[7,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":454,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-09-10T19:30:31.331455+0000","last_change":"2024-09-10T19:29:18.277593+0000","last_active":"2024-09-10T19:30:31.331455+0000","last_peered":"2024-09-10T19:30:31.331455+0000","last_clean":"2024-09-10T19:30:31.331455+0000","last_became_active":"2024-09-10T19:24:23.906812+0000","last_became_peered":"2024-09-10T19:24:23.906812+0000","last_unstale":"2024-09-10T19:30:31.331455+0000","last_undegraded":"2024-09-10T19:30:31.331455+0000","last_fullsized":"2024-09-10T19:30:31.331455+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:29:18.277508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:29:18.277508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:30:46.445119+0000","scrub_duration":0.007405455,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,0],"acting":[5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1e","version":"414'35","reported_seq":446,"reported_epoch":417,"state":"active+clean","last_fresh":"2024-09-10T19:30:36.373117+0000","last_change":"2024-09-10T19:29:20.656771+0000","last_active":"2024-09-10T19:30:36.373117+0000","last_peered":"2024-09-10T19:30:36.373117+0000","last_clean":"2024-09-10T19:30:36.373117+0000","last_became_active":"2024-09-10T19:24:23.899507+0000","last_became_peered":"2024-09-10T19:24:23.899507+0000","last_unstale":"2024-09-10T19:30:36.373117+0000","last_undegraded":"2024-09-10T19:30:36.373117+0000","last_fullsized":"2024-09-10T19:30:36.373117+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"237'25","last_scrub_stamp":"2024-09-10T19:29:20.656690+0000","last_deep_scrub":"80'15","last_deep_scrub_stamp":"2024-09-10T19:11:25.883919+0000","last_clean_scrub_stamp":"2024-09-10T19:29:20.656690+0000","objects_scrubbed":5,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0034048949999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":39,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1f","version":"376'10","reported_seq":109,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-09-10T19:30:35.968674+0000","last_change":"2024-09-10T19:30:22.536949+0000","last_active":"2024-09-10T19:30:35.968674+0000","last_peered":"2024-09-10T19:30:35.968674+0000","last_clean":"2024-09-10T19:30:35.968674+0000","last_became_active":"2024-09-10T19:27:53.003676+0000","last_became_peered":"2024-09-10T19:27:53.003676+0000","last_unstale":"2024-09-10T19:30:35.968674+0000","last_undegraded":"2024-09-10T19:30:35.968674+0000","last_fullsized":"2024-09-10T19:30:35.968674+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"376'10","last_scrub_stamp":"2024-09-10T19:30:22.536865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:30:22.536865+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:31:49.099743+0000","scrub_duration":0.0053077080000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":165,"reported_epoch":417,"state":"active+clean","last_fresh":"2024-09-10T19:30:36.376126+0000","last_change":"2024-09-10T19:30:25.316849+0000","last_active":"2024-09-10T19:30:36.376126+0000","last_peered":"2024-09-10T19:30:36.376126+0000","last_clean":"2024-09-10T19:30:36.376126+0000","last_became_active":"2024-09-10T19:24:01.980391+0000","last_became_peered":"2024-09-10T19:24:01.980391+0000","last_unstale":"2024-09-10T19:30:36.376126+0000","last_undegraded":"2024-09-10T19:30:36.376126+0000","last_fullsized":"2024-09-10T19:30:36.376126+0000","mapping_epoch":292,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":293,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:30:25.316711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:28:20.514115+0000","last_clean_scrub_stamp":"2024-09-10T19:30:25.316711+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:31:48.741913+0000","scrub_duration":0.0018401960000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":474,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-09-10T19:30:31.328648+0000","last_change":"2024-09-10T19:29:34.161440+0000","last_active":"2024-09-10T19:30:31.328648+0000","last_peered":"2024-09-10T19:30:31.328648+0000","last_clean":"2024-09-10T19:30:31.328648+0000","last_became_active":"2024-09-10T19:27:53.017153+0000","last_became_peered":"2024-09-10T19:27:53.017153+0000","last_unstale":"2024-09-10T19:30:31.328648+0000","last_undegraded":"2024-09-10T19:30:31.328648+0000","last_fullsized":"2024-09-10T19:30:31.328648+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:29:34.161296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:29:34.161296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:30:50.439084+0000","scrub_duration":0.005093261,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"414'32","reported_seq":96,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-09-10T19:30:31.336552+0000","last_change":"2024-09-10T19:29:28.357694+0000","last_active":"2024-09-10T19:30:31.336552+0000","last_peered":"2024-09-10T19:30:31.336552+0000","last_clean":"2024-09-10T19:30:31.336552+0000","last_became_active":"2024-09-10T19:27:53.008233+0000","last_became_peered":"2024-09-10T19:27:53.008233+0000","last_unstale":"2024-09-10T19:30:31.336552+0000","last_undegraded":"2024-09-10T19:30:31.336552+0000","last_fullsized":"2024-09-10T19:30:31.336552+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"237'24","last_scrub_stamp":"2024-09-10T19:29:28.357607+0000","last_deep_scrub":"80'16","last_deep_scrub_stamp":"2024-09-10T19:11:40.853611+0000","last_clean_scrub_stamp":"2024-09-10T19:29:28.357607+0000","objects_scrubbed":5,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:30:40.574883+0000","scrub_duration":0.0067797259999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":36,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":383,"reported_epoch":417,"state":"active+clean","last_fresh":"2024-09-10T19:30:36.373290+0000","last_change":"2024-09-10T19:30:24.834659+0000","last_active":"2024-09-10T19:30:36.373290+0000","last_peered":"2024-09-10T19:30:36.373290+0000","last_clean":"2024-09-10T19:30:36.373290+0000","last_became_active":"2024-09-10T19:24:24.899153+0000","last_became_peered":"2024-09-10T19:24:24.899153+0000","last_unstale":"2024-09-10T19:30:36.373290+0000","last_undegraded":"2024-09-10T19:30:36.373290+0000","last_fullsized":"2024-09-10T19:30:36.373290+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:30:24.834494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:28:19.464426+0000","last_clean_scrub_stamp":"2024-09-10T19:30:24.834494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:31:26.567702+0000","scrub_duration":0.0076205279999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":470,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-09-10T19:30:31.333681+0000","last_change":"2024-09-10T19:30:30.520734+0000","last_active":"2024-09-10T19:30:31.333681+0000","last_peered":"2024-09-10T19:30:31.333681+0000","last_clean":"2024-09-10T19:30:31.333681+0000","last_became_active":"2024-09-10T19:27:53.011336+0000","last_became_peered":"2024-09-10T19:27:53.011336+0000","last_unstale":"2024-09-10T19:30:31.333681+0000","last_undegraded":"2024-09-10T19:30:31.333681+0000","last_fullsized":"2024-09-10T19:30:31.333681+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:30:30.520643+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:30:30.520643+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:31:57.292887+0000","scrub_duration":0.018058100000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":72,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-09-10T19:30:31.336378+0000","last_change":"2024-09-10T19:29:33.401343+0000","last_active":"2024-09-10T19:30:31.336378+0000","last_peered":"2024-09-10T19:30:31.336378+0000","last_clean":"2024-09-10T19:30:31.336378+0000","last_became_active":"2024-09-10T19:27:53.002313+0000","last_became_peered":"2024-09-10T19:27:53.002313+0000","last_unstale":"2024-09-10T19:30:31.336378+0000","last_undegraded":"2024-09-10T19:30:31.336378+0000","last_fullsized":"2024-09-10T19:30:31.336378+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:29:33.401164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:29:33.401164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:31:02.399310+0000","scrub_duration":0.016867561,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1c","version":"414'46","reported_seq":213,"reported_epoch":417,"state":"active+clean","last_fresh":"2024-09-10T19:30:36.377708+0000","last_change":"2024-09-10T19:29:34.532076+0000","last_active":"2024-09-10T19:30:36.377708+0000","last_peered":"2024-09-10T19:30:36.377708+0000","last_clean":"2024-09-10T19:30:36.377708+0000","last_became_active":"2024-09-10T19:24:24.910577+0000","last_became_peered":"2024-09-10T19:24:24.910577+0000","last_unstale":"2024-09-10T19:30:36.377708+0000","last_undegraded":"2024-09-10T19:30:36.377708+0000","last_fullsized":"2024-09-10T19:30:36.377708+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"237'34","last_scrub_stamp":"2024-09-10T19:29:34.531993+0000","last_deep_scrub":"78'22","last_deep_scrub_stamp":"2024-09-10T19:11:21.118937+0000","last_clean_scrub_stamp":"2024-09-10T19:29:34.531993+0000","objects_scrubbed":7,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:30:47.686023+0000","scrub_duration":0.0018029159999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":62,"num_read_kb":53,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":179,"reported_epoch":417,"state":"active+clean","last_fresh":"2024-09-10T19:30:36.376245+0000","last_change":"2024-09-10T19:29:21.494875+0000","last_active":"2024-09-10T19:30:36.376245+0000","last_peered":"2024-09-10T19:30:36.376245+0000","last_clean":"2024-09-10T19:30:36.376245+0000","last_became_active":"2024-09-10T19:24:24.887205+0000","last_became_peered":"2024-09-10T19:24:24.887205+0000","last_unstale":"2024-09-10T19:30:36.376245+0000","last_undegraded":"2024-09-10T19:30:36.376245+0000","last_fullsized":"2024-09-10T19:30:36.376245+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:29:21.494801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:29:21.494801+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:30:42.597169+0000","scrub_duration":0.011816722,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":454,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-09-10T19:30:31.331739+0000","last_change":"2024-09-10T19:29:25.182733+0000","last_active":"2024-09-10T19:30:31.331739+0000","last_peered":"2024-09-10T19:30:31.331739+0000","last_clean":"2024-09-10T19:30:31.331739+0000","last_became_active":"2024-09-10T19:27:53.026591+0000","last_became_peered":"2024-09-10T19:27:53.026591+0000","last_unstale":"2024-09-10T19:30:31.331739+0000","last_undegraded":"2024-09-10T19:30:31.331739+0000","last_fullsized":"2024-09-10T19:30:31.331739+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:29:25.182654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:24:46.518816+0000","last_clean_scrub_stamp":"2024-09-10T19:29:25.182654+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0060637180000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":467,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-09-10T19:30:31.329957+0000","last_change":"2024-09-10T19:30:24.721276+0000","last_active":"2024-09-10T19:30:31.329957+0000","last_peered":"2024-09-10T19:30:31.329957+0000","last_clean":"2024-09-10T19:30:31.329957+0000","last_became_active":"2024-09-10T19:27:53.018726+0000","last_became_peered":"2024-09-10T19:27:53.018726+0000","last_unstale":"2024-09-10T19:30:31.329957+0000","last_undegraded":"2024-09-10T19:30:31.329957+0000","last_fullsized":"2024-09-10T19:30:31.329957+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:30:24.721140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:30:24.721140+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:31:33.177185+0000","scrub_duration":0.0073255109999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1d","version":"414'29","reported_seq":532,"reported_epoch":416,"state":"active+clean","last_fresh":"2024-09-10T19:30:31.330953+0000","last_change":"2024-09-10T19:30:28.765125+0000","last_active":"2024-09-10T19:30:31.330953+0000","last_peered":"2024-09-10T19:30:31.330953+0000","last_clean":"2024-09-10T19:30:31.330953+0000","last_became_active":"2024-09-10T19:24:01.976664+0000","last_became_peered":"2024-09-10T19:24:01.976664+0000","last_unstale":"2024-09-10T19:30:31.330953+0000","last_undegraded":"2024-09-10T19:30:31.330953+0000","last_fullsized":"2024-09-10T19:30:31.330953+0000","mapping_epoch":292,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":293,"parent":"0.0","parent_split_bits":0,"last_scrub":"414'29","last_scrub_stamp":"2024-09-10T19:30:28.765025+0000","last_deep_scrub":"250'23","last_deep_scrub_stamp":"2024-09-10T19:20:55.582173+0000","last_clean_scrub_stamp":"2024-09-10T19:30:28.765025+0000","objects_scrubbed":5,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:31:57.061451+0000","scrub_duration":0.012181088,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":462,"reported_epoch":417,"state":"active+clean","last_fresh":"2024-09-10T19:30:39.493770+0000","last_change":"2024-09-10T19:30:39.493327+0000","last_active":"2024-09-10T19:30:39.493770+0000","last_peered":"2024-09-10T19:30:39.493770+0000","last_clean":"2024-09-10T19:30:39.493770+0000","last_became_active":"2024-09-10T19:16:24.843013+0000","last_became_peered":"2024-09-10T19:16:24.843013+0000","last_unstale":"2024-09-10T19:30:39.493770+0000","last_undegraded":"2024-09-10T19:30:39.493770+0000","last_fullsized":"2024-09-10T19:30:39.493770+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:30:39.493257+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:30:39.493257+0000","last_clean_scrub_stamp":"2024-09-10T19:30:39.493257+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:32:02.390742+0000","scrub_duration":0.0032639090000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":11,"num_pg":16,"stat_sum":{"num_bytes":2314207256,"num_objects":35313,"num_object_clones":0,"num_object_copies":105939,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35313,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":254463,"num_write_kb":9272769,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":6871265280,"data_stored":6871253064,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":70289,"ondisk_log_size":70289,"up":48,"acting":48,"num_store_stats":7},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":382,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":1528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":96,"ondisk_log_size":96,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":177,"num_object_clones":0,"num_object_copies":531,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":177,"num_whiteouts":0,"num_read":1544,"num_read_kb":1301,"num_write":989,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":433,"num_bytes_recovered":8406,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":458752,"data_stored":12124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1066,"ondisk_log_size":1066,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":402,"num_read_kb":397,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1594,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1613,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":10,"ondisk_log_size":10,"up":9,"acting":9,"num_store_stats":7},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1328,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":21,"num_read_kb":21,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":724,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":3984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5,"ondisk_log_size":5,"up":96,"acting":96,"num_store_stats":8},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":525376,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":870,"ondisk_log_size":870,"up":33,"acting":33,"num_store_stats":7}],"osd_stats":[{"osd":7,"up_from":54,"seq":231928234235,"num_pgs":71,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2346284,"kb_used_data":1169700,"kb_used_omap":1,"kb_used_meta":1176510,"kb_avail":91398868,"statfs":{"total":95995035648,"available":93592440832,"internally_reserved":0,"allocated":1197772800,"data_stored":1197445158,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":1204746584},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 19:30:32 2024","interfaces":[{"interface":"back","average":{"1min":1.2030000000000001,"5min":0.98599999999999999,"15min":0.97499999999999998},"min":{"1min":0.30499999999999999,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":3.27,"5min":5.0510000000000002,"15min":5.0510000000000002},"last":0.56599999999999995},{"interface":"front","average":{"1min":1.304,"5min":1.103,"15min":0.82699999999999996},"min":{"1min":0.23400000000000001,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":3.2570000000000001,"5min":4.7190000000000003,"15min":4.7190000000000003},"last":1.3300000000000001}]},{"osd":1,"last update":"Tue Sep 10 19:30:01 2024","interfaces":[{"interface":"back","average":{"1min":1.4950000000000001,"5min":1.2729999999999999,"15min":1.0880000000000001},"min":{"1min":0.35599999999999998,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":4.6050000000000004,"5min":12.773999999999999,"15min":12.773999999999999},"last":0.59599999999999997},{"interface":"front","average":{"1min":1.155,"5min":1.2430000000000001,"15min":0.96699999999999997},"min":{"1min":0.38600000000000001,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":3.6019999999999999,"5min":12.837999999999999,"15min":12.837999999999999},"last":0.65600000000000003}]},{"osd":2,"last update":"Tue Sep 10 19:30:19 2024","interfaces":[{"interface":"back","average":{"1min":1.3879999999999999,"5min":0.996,"15min":0.92300000000000004},"min":{"1min":0.26600000000000001,"5min":0.26600000000000001,"15min":0.23000000000000001},"max":{"1min":3.3100000000000001,"5min":4.7400000000000002,"15min":4.7400000000000002},"last":1.3799999999999999},{"interface":"front","average":{"1min":1.327,"5min":1.1060000000000001,"15min":1.04},"min":{"1min":0.37,"5min":0.29099999999999998,"15min":0.26300000000000001},"max":{"1min":6.1059999999999999,"5min":6.1059999999999999,"15min":8.0839999999999996},"last":0.73199999999999998}]},{"osd":3,"last update":"Tue Sep 10 19:30:19 2024","interfaces":[{"interface":"back","average":{"1min":2.0640000000000001,"5min":1.3720000000000001,"15min":1.204},"min":{"1min":0.32800000000000001,"5min":0.25900000000000001,"15min":0.20200000000000001},"max":{"1min":14.515000000000001,"5min":14.515000000000001,"15min":21.074999999999999},"last":1.3580000000000001},{"interface":"front","average":{"1min":1.915,"5min":1.3560000000000001,"15min":1.1839999999999999},"min":{"1min":0.374,"5min":0.27700000000000002,"15min":0.25700000000000001},"max":{"1min":14.227,"5min":14.227,"15min":18.82},"last":0.68600000000000005}]},{"osd":4,"last update":"Tue Sep 10 19:30:19 2024","interfaces":[{"interface":"back","average":{"1min":1.496,"5min":1.716,"15min":2.677},"min":{"1min":0.35399999999999998,"5min":0.24399999999999999,"15min":0.222},"max":{"1min":5.194,"5min":18.664999999999999,"15min":231.923},"last":0.501},{"interface":"front","average":{"1min":1.488,"5min":1.466,"15min":2.3479999999999999},"min":{"1min":0.29599999999999999,"5min":0.25600000000000001,"15min":0.23100000000000001},"max":{"1min":6.6029999999999998,"5min":14.837,"15min":231.322},"last":1.6659999999999999}]},{"osd":5,"last update":"Tue Sep 10 19:30:19 2024","interfaces":[{"interface":"back","average":{"1min":1.1639999999999999,"5min":1.1850000000000001,"15min":1.6339999999999999},"min":{"1min":0.39000000000000001,"5min":0.25800000000000001,"15min":0.24099999999999999},"max":{"1min":3.5750000000000002,"5min":8.8300000000000001,"15min":67.382999999999996},"last":0.70999999999999996},{"interface":"front","average":{"1min":1.77,"5min":1.5549999999999999,"15min":1.99},"min":{"1min":0.33700000000000002,"5min":0.309,"15min":0.22800000000000001},"max":{"1min":6.7939999999999996,"5min":8.7040000000000006,"15min":67.75},"last":2.0190000000000001}]},{"osd":6,"last update":"Tue Sep 10 19:30:19 2024","interfaces":[{"interface":"back","average":{"1min":2.0139999999999998,"5min":1.5980000000000001,"15min":2.4079999999999999},"min":{"1min":0.29799999999999999,"5min":0.29799999999999999,"15min":0.28000000000000003},"max":{"1min":6.7530000000000001,"5min":6.7869999999999999,"15min":181.53800000000001},"last":0.623},{"interface":"front","average":{"1min":1.591,"5min":1.393,"15min":2.101},"min":{"1min":0.34399999999999997,"5min":0.318,"15min":0.27000000000000002},"max":{"1min":6.6349999999999998,"5min":9.7110000000000003,"15min":182.05600000000001},"last":1.6399999999999999}]}]},{"osd":6,"up_from":47,"seq":201863463168,"num_pgs":66,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":2088700,"kb_used_data":1193844,"kb_used_omap":1,"kb_used_meta":894782,"kb_avail":91656452,"statfs":{"total":95995035648,"available":93856206848,"internally_reserved":0,"allocated":1222496256,"data_stored":1222153342,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":916257112},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1,2,2],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 19:29:34 2024","interfaces":[{"interface":"back","average":{"1min":1.357,"5min":1.4650000000000001,"15min":1.7370000000000001},"min":{"1min":0.21299999999999999,"5min":0.17899999999999999,"15min":0.17899999999999999},"max":{"1min":4.8029999999999999,"5min":18.545999999999999,"15min":18.545999999999999},"last":0.57699999999999996},{"interface":"front","average":{"1min":1.6799999999999999,"5min":1.415,"15min":1.804},"min":{"1min":0.28599999999999998,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":4.5700000000000003,"5min":19.818000000000001,"15min":19.818000000000001},"last":0.20599999999999999}]},{"osd":1,"last update":"Tue Sep 10 19:30:02 2024","interfaces":[{"interface":"back","average":{"1min":1.8859999999999999,"5min":1.6200000000000001,"15min":1.0009999999999999},"min":{"1min":0.315,"5min":0.20999999999999999,"15min":0.19800000000000001},"max":{"1min":5.7320000000000002,"5min":7.9039999999999999,"15min":7.9039999999999999},"last":0.55500000000000005},{"interface":"front","average":{"1min":1.5129999999999999,"5min":1.29,"15min":0.93200000000000005},"min":{"1min":0.21199999999999999,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":5.6799999999999997,"5min":7.9189999999999996,"15min":7.9189999999999996},"last":0.246}]},{"osd":2,"last update":"Tue Sep 10 19:30:10 2024","interfaces":[{"interface":"back","average":{"1min":1.258,"5min":1.1459999999999999,"15min":0.91400000000000003},"min":{"1min":0.24299999999999999,"5min":0.24299999999999999,"15min":0.23799999999999999},"max":{"1min":4.8330000000000002,"5min":5.8010000000000002,"15min":5.9029999999999996},"last":0.26900000000000002},{"interface":"front","average":{"1min":1.218,"5min":1.1259999999999999,"15min":0.95999999999999996},"min":{"1min":0.36799999999999999,"5min":0.25600000000000001,"15min":0.254},"max":{"1min":5.6929999999999996,"5min":6.4560000000000004,"15min":6.4560000000000004},"last":0.29799999999999999}]},{"osd":3,"last update":"Tue Sep 10 19:30:10 2024","interfaces":[{"interface":"back","average":{"1min":1.4390000000000001,"5min":1.819,"15min":1.321},"min":{"1min":0.316,"5min":0.27900000000000003,"15min":0.20399999999999999},"max":{"1min":4.7290000000000001,"5min":35.847000000000001,"15min":35.847000000000001},"last":0.59799999999999998},{"interface":"front","average":{"1min":1.647,"5min":1.643,"15min":1.27},"min":{"1min":0.29899999999999999,"5min":0.29199999999999998,"15min":0.20499999999999999},"max":{"1min":5.6420000000000003,"5min":35.862000000000002,"15min":35.862000000000002},"last":0.28999999999999998}]},{"osd":4,"last update":"Tue Sep 10 19:30:10 2024","interfaces":[{"interface":"back","average":{"1min":1.429,"5min":1.9990000000000001,"15min":1.47},"min":{"1min":0.35499999999999998,"5min":0.22,"15min":0.22},"max":{"1min":5.6150000000000002,"5min":35.552,"15min":35.552},"last":1.52},{"interface":"front","average":{"1min":1.371,"5min":2.117,"15min":1.4239999999999999},"min":{"1min":0.30299999999999999,"5min":0.24399999999999999,"15min":0.22500000000000001},"max":{"1min":4.5069999999999997,"5min":35.192999999999998,"15min":35.192999999999998},"last":0.90500000000000003}]},{"osd":5,"last update":"Tue Sep 10 19:30:10 2024","interfaces":[{"interface":"back","average":{"1min":1.282,"5min":1.909,"15min":1.484},"min":{"1min":0.40799999999999997,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":7.4050000000000002,"5min":48.216000000000001,"15min":48.216000000000001},"last":1.488},{"interface":"front","average":{"1min":1.2529999999999999,"5min":2.3130000000000002,"15min":1.659},"min":{"1min":0.40100000000000002,"5min":0.254,"15min":0.254},"max":{"1min":4.3719999999999999,"5min":50.372999999999998,"15min":50.372999999999998},"last":1.0600000000000001}]},{"osd":7,"last update":"Tue Sep 10 19:29:34 2024","interfaces":[{"interface":"back","average":{"1min":1.617,"5min":2.5979999999999999,"15min":1.577},"min":{"1min":0.38100000000000001,"5min":0.28299999999999997,"15min":0.28100000000000003},"max":{"1min":7.609,"5min":80.629999999999995,"15min":80.629999999999995},"last":0.91700000000000004},{"interface":"front","average":{"1min":1.5880000000000001,"5min":2.8559999999999999,"15min":1.8340000000000001},"min":{"1min":0.55200000000000005,"5min":0.28299999999999997,"15min":0.25700000000000001},"max":{"1min":4.758,"5min":81.088999999999999,"15min":81.088999999999999},"last":0.52900000000000003}]}]},{"osd":5,"up_from":39,"seq":167503724805,"num_pgs":65,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1476148,"kb_used_data":578092,"kb_used_omap":1,"kb_used_meta":897918,"kb_avail":92269004,"statfs":{"total":95995035648,"available":94483460096,"internally_reserved":0,"allocated":591966208,"data_stored":591632134,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":919468376},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 19:30:33 2024","interfaces":[{"interface":"back","average":{"1min":1.278,"5min":1.0580000000000001,"15min":1.1000000000000001},"min":{"1min":0.223,"5min":0.19,"15min":0.187},"max":{"1min":7.5759999999999996,"5min":7.5759999999999996,"15min":7.5759999999999996},"last":0.747},{"interface":"front","average":{"1min":1.081,"5min":0.94899999999999995,"15min":1.175},"min":{"1min":0.20599999999999999,"5min":0.20599999999999999,"15min":0.20599999999999999},"max":{"1min":5.819,"5min":5.819,"15min":5.819},"last":0.26300000000000001}]},{"osd":1,"last update":"Tue Sep 10 19:29:50 2024","interfaces":[{"interface":"back","average":{"1min":3.0459999999999998,"5min":2.5699999999999998,"15min":1.4930000000000001},"min":{"1min":0.247,"5min":0.214,"15min":0.214},"max":{"1min":35.107999999999997,"5min":54.097999999999999,"15min":54.097999999999999},"last":1.2030000000000001},{"interface":"front","average":{"1min":2.5659999999999998,"5min":2.5089999999999999,"15min":1.5129999999999999},"min":{"1min":0.26300000000000001,"5min":0.221,"15min":0.221},"max":{"1min":35.173000000000002,"5min":54.337000000000003,"15min":54.337000000000003},"last":1.274}]},{"osd":2,"last update":"Tue Sep 10 19:30:39 2024","interfaces":[{"interface":"back","average":{"1min":1.0629999999999999,"5min":1.0089999999999999,"15min":1.1060000000000001},"min":{"1min":0.32800000000000001,"5min":0.26500000000000001,"15min":0.22900000000000001},"max":{"1min":7.4489999999999998,"5min":7.4489999999999998,"15min":9.9399999999999995},"last":0.80400000000000005},{"interface":"front","average":{"1min":0.754,"5min":0.98699999999999999,"15min":1.052},"min":{"1min":0.28599999999999998,"5min":0.27700000000000002,"15min":0.21199999999999999},"max":{"1min":2.8889999999999998,"5min":10.045999999999999,"15min":10.045999999999999},"last":0.61199999999999999}]},{"osd":3,"last update":"Tue Sep 10 19:30:39 2024","interfaces":[{"interface":"back","average":{"1min":1.2190000000000001,"5min":2.5089999999999999,"15min":1.696},"min":{"1min":0.26600000000000001,"5min":0.254,"15min":0.19900000000000001},"max":{"1min":7.5979999999999999,"5min":54.832000000000001,"15min":54.832000000000001},"last":0.88},{"interface":"front","average":{"1min":1.2150000000000001,"5min":2.484,"15min":1.659},"min":{"1min":0.28100000000000003,"5min":0.28100000000000003,"15min":0.246},"max":{"1min":7.3520000000000003,"5min":54.850999999999999,"15min":54.850999999999999},"last":0.33400000000000002}]},{"osd":4,"last update":"Tue Sep 10 19:30:39 2024","interfaces":[{"interface":"back","average":{"1min":1.0780000000000001,"5min":2.3799999999999999,"15min":2.202},"min":{"1min":0.26100000000000001,"5min":0.252,"15min":0.24399999999999999},"max":{"1min":5.6929999999999996,"5min":132.41900000000001,"15min":132.41900000000001},"last":0.67200000000000004},{"interface":"front","average":{"1min":1.1599999999999999,"5min":2.48,"15min":2.3119999999999998},"min":{"1min":0.29099999999999998,"5min":0.28699999999999998,"15min":0.217},"max":{"1min":7.5519999999999996,"5min":132.46700000000001,"15min":132.46700000000001},"last":0.84199999999999997}]},{"osd":6,"last update":"Tue Sep 10 19:30:13 2024","interfaces":[{"interface":"back","average":{"1min":0.83599999999999997,"5min":1.4019999999999999,"15min":1.8120000000000001},"min":{"1min":0.28699999999999998,"5min":0.28599999999999998,"15min":0.28399999999999997},"max":{"1min":2.4199999999999999,"5min":8.3339999999999996,"15min":111.328},"last":0.90600000000000003},{"interface":"front","average":{"1min":1.135,"5min":1.399,"15min":1.762},"min":{"1min":0.28299999999999997,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":4.1219999999999999,"5min":7.2910000000000004,"15min":111.705},"last":0.78100000000000003}]},{"osd":7,"last update":"Tue Sep 10 19:30:30 2024","interfaces":[{"interface":"back","average":{"1min":1.619,"5min":1.27,"15min":1.383},"min":{"1min":0.35099999999999998,"5min":0.29099999999999998,"15min":0.28599999999999998},"max":{"1min":4.5140000000000002,"5min":4.5140000000000002,"15min":6.7869999999999999},"last":2.6749999999999998},{"interface":"front","average":{"1min":1.2609999999999999,"5min":1.1739999999999999,"15min":1.357},"min":{"1min":0.30099999999999999,"5min":0.29899999999999999,"15min":0.27100000000000002},"max":{"1min":3.9100000000000001,"5min":5.6349999999999998,"15min":9.4909999999999997},"last":0.70199999999999996}]}]},{"osd":4,"up_from":32,"seq":137438953737,"num_pgs":65,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1625716,"kb_used_data":737324,"kb_used_omap":1,"kb_used_meta":888382,"kb_avail":92119436,"statfs":{"total":95995035648,"available":94330302464,"internally_reserved":0,"allocated":755019776,"data_stored":754683817,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":909703512},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 19:30:31 2024","interfaces":[{"interface":"back","average":{"1min":0.90400000000000003,"5min":1.506,"15min":2.0299999999999998},"min":{"1min":0.21199999999999999,"5min":0.20100000000000001,"15min":0.20100000000000001},"max":{"1min":3.6110000000000002,"5min":39.097000000000001,"15min":39.097000000000001},"last":0.79000000000000004},{"interface":"front","average":{"1min":0.50600000000000001,"5min":1.6299999999999999,"15min":2.2919999999999998},"min":{"1min":0.24099999999999999,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":1.1599999999999999,"5min":39.112000000000002,"15min":39.112000000000002},"last":1.0089999999999999}]},{"osd":1,"last update":"Tue Sep 10 19:29:52 2024","interfaces":[{"interface":"back","average":{"1min":1.5389999999999999,"5min":1.462,"15min":0.94899999999999995},"min":{"1min":0.29899999999999999,"5min":0.20599999999999999,"15min":0.20599999999999999},"max":{"1min":6.1870000000000003,"5min":6.1870000000000003,"15min":6.1870000000000003},"last":1.081},{"interface":"front","average":{"1min":1.212,"5min":1.153,"15min":0.78600000000000003},"min":{"1min":0.316,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":3.1539999999999999,"5min":5.8680000000000003,"15min":5.8680000000000003},"last":0.80300000000000005}]},{"osd":2,"last update":"Tue Sep 10 19:30:25 2024","interfaces":[{"interface":"back","average":{"1min":0.622,"5min":1.1479999999999999,"15min":1.016},"min":{"1min":0.29399999999999998,"5min":0.27000000000000002,"15min":0.254},"max":{"1min":2.6040000000000001,"5min":5.5609999999999999,"15min":6.8869999999999996},"last":1.056},{"interface":"front","average":{"1min":1.085,"5min":1.1619999999999999,"15min":1.002},"min":{"1min":0.29299999999999998,"5min":0.28299999999999997,"15min":0.253},"max":{"1min":3.2120000000000002,"5min":6.7690000000000001,"15min":7.085},"last":0.68500000000000005}]},{"osd":3,"last update":"Tue Sep 10 19:30:25 2024","interfaces":[{"interface":"back","average":{"1min":1.161,"5min":1.1910000000000001,"15min":1.2150000000000001},"min":{"1min":0.28899999999999998,"5min":0.27800000000000002,"15min":0.20999999999999999},"max":{"1min":3.1880000000000002,"5min":6.8019999999999996,"15min":35.451999999999998},"last":0.65600000000000003},{"interface":"front","average":{"1min":1.157,"5min":1.1619999999999999,"15min":1.216},"min":{"1min":0.28499999999999998,"5min":0.28499999999999998,"15min":0.246},"max":{"1min":3.5329999999999999,"5min":5.8949999999999996,"15min":32.880000000000003},"last":1.4339999999999999}]},{"osd":5,"last update":"Tue Sep 10 19:29:44 2024","interfaces":[{"interface":"back","average":{"1min":0.66400000000000003,"5min":1.97,"15min":2.5630000000000002},"min":{"1min":0.317,"5min":0.26800000000000002,"15min":0.224},"max":{"1min":2.1659999999999999,"5min":58.009999999999998,"15min":227.726},"last":1.181},{"interface":"front","average":{"1min":1.4470000000000001,"5min":2.2309999999999999,"15min":2.5670000000000002},"min":{"1min":0.28199999999999997,"5min":0.27400000000000002,"15min":0.20699999999999999},"max":{"1min":5.6699999999999999,"5min":56.027000000000001,"15min":227.70500000000001},"last":1.036}]},{"osd":6,"last update":"Tue Sep 10 19:30:25 2024","interfaces":[{"interface":"back","average":{"1min":1.0129999999999999,"5min":1.554,"15min":1.2629999999999999},"min":{"1min":0.34200000000000003,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":3.6259999999999999,"5min":8.1010000000000009,"15min":8.1010000000000009},"last":1.121},{"interface":"front","average":{"1min":1.6599999999999999,"5min":1.78,"15min":1.3420000000000001},"min":{"1min":0.254,"5min":0.254,"15min":0.247},"max":{"1min":6.0099999999999998,"5min":16.815000000000001,"15min":16.815000000000001},"last":0.76700000000000002}]},{"osd":7,"last update":"Tue Sep 10 19:30:31 2024","interfaces":[{"interface":"back","average":{"1min":1.1839999999999999,"5min":1.3660000000000001,"15min":1.3240000000000001},"min":{"1min":0.307,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":3.8319999999999999,"5min":16.853999999999999,"15min":16.853999999999999},"last":0.71799999999999997},{"interface":"front","average":{"1min":1.4590000000000001,"5min":1.4730000000000001,"15min":1.3899999999999999},"min":{"1min":0.35299999999999998,"5min":0.25700000000000001,"15min":0.23699999999999999},"max":{"1min":4.6020000000000003,"5min":10.02,"15min":10.02},"last":0.70499999999999996}]}]},{"osd":3,"up_from":26,"seq":111669149966,"num_pgs":73,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1921996,"kb_used_data":1021700,"kb_used_omap":1,"kb_used_meta":900286,"kb_avail":91823156,"statfs":{"total":95995035648,"available":94026911744,"internally_reserved":0,"allocated":1046220800,"data_stored":1045862728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":921893208},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 19:30:38 2024","interfaces":[{"interface":"back","average":{"1min":8.7970000000000006,"5min":3.673,"15min":1.5800000000000001},"min":{"1min":0.16800000000000001,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":117.655,"5min":117.655,"15min":117.655},"last":0.52500000000000002},{"interface":"front","average":{"1min":8.6489999999999991,"5min":3.6440000000000001,"15min":1.5469999999999999},"min":{"1min":0.21199999999999999,"5min":0.184,"15min":0.184},"max":{"1min":117.81399999999999,"5min":117.81399999999999,"15min":117.81399999999999},"last":0.21199999999999999}]},{"osd":1,"last update":"Tue Sep 10 19:30:02 2024","interfaces":[{"interface":"back","average":{"1min":0.94099999999999995,"5min":0.72299999999999998,"15min":0.44500000000000001},"min":{"1min":0.20100000000000001,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":2.5979999999999999,"5min":5.1779999999999999,"15min":5.1779999999999999},"last":1.405},{"interface":"front","average":{"1min":1.4370000000000001,"5min":0.77600000000000002,"15min":0.47199999999999998},"min":{"1min":0.245,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":4.0830000000000002,"5min":4.4100000000000001,"15min":4.4100000000000001},"last":1.276}]},{"osd":2,"last update":"Tue Sep 10 19:30:08 2024","interfaces":[{"interface":"back","average":{"1min":0.66900000000000004,"5min":0.58499999999999996,"15min":0.58399999999999996},"min":{"1min":0.35599999999999998,"5min":0.246,"15min":0.17599999999999999},"max":{"1min":1.827,"5min":4.3769999999999998,"15min":5.3029999999999999},"last":0.41699999999999998},{"interface":"front","average":{"1min":0.75700000000000001,"5min":0.64100000000000001,"15min":0.626},"min":{"1min":0.248,"5min":0.23699999999999999,"15min":0.16400000000000001},"max":{"1min":2.3980000000000001,"5min":4.2990000000000004,"15min":17.806000000000001},"last":0.42799999999999999}]},{"osd":4,"last update":"Tue Sep 10 19:30:28 2024","interfaces":[{"interface":"back","average":{"1min":1.2949999999999999,"5min":1.01,"15min":1.3360000000000001},"min":{"1min":0.30099999999999999,"5min":0.26900000000000002,"15min":0.254},"max":{"1min":4.1159999999999997,"5min":4.4169999999999998,"15min":81.334000000000003},"last":0.83599999999999997},{"interface":"front","average":{"1min":1.0589999999999999,"5min":1.0740000000000001,"15min":1.429},"min":{"1min":0.318,"5min":0.27900000000000003,"15min":0.22700000000000001},"max":{"1min":3.1720000000000002,"5min":4.343,"15min":81.381},"last":1.0009999999999999}]},{"osd":5,"last update":"Tue Sep 10 19:29:46 2024","interfaces":[{"interface":"back","average":{"1min":0.88200000000000001,"5min":1.1619999999999999,"15min":1.7110000000000001},"min":{"1min":0.28000000000000003,"5min":0.28000000000000003,"15min":0.27900000000000003},"max":{"1min":2.4710000000000001,"5min":6.7240000000000002,"15min":104.672},"last":0.45800000000000002},{"interface":"front","average":{"1min":0.90200000000000002,"5min":1.087,"15min":1.7330000000000001},"min":{"1min":0.36799999999999999,"5min":0.28799999999999998,"15min":0.22},"max":{"1min":2.5059999999999998,"5min":4.6550000000000002,"15min":105.971},"last":0.96399999999999997}]},{"osd":6,"last update":"Tue Sep 10 19:30:08 2024","interfaces":[{"interface":"back","average":{"1min":1.2470000000000001,"5min":2.8380000000000001,"15min":2.1709999999999998},"min":{"1min":0.34100000000000003,"5min":0.33600000000000002,"15min":0.23699999999999999},"max":{"1min":2.4340000000000002,"5min":148.09299999999999,"15min":148.09299999999999},"last":0.47599999999999998},{"interface":"front","average":{"1min":1.137,"5min":2.6659999999999999,"15min":2.0819999999999999},"min":{"1min":0.40200000000000002,"5min":0.32700000000000001,"15min":0.26800000000000002},"max":{"1min":3.6659999999999999,"5min":148.31999999999999,"15min":148.31999999999999},"last":0.65200000000000002}]},{"osd":7,"last update":"Tue Sep 10 19:29:43 2024","interfaces":[{"interface":"back","average":{"1min":1.306,"5min":1.069,"15min":1.4199999999999999},"min":{"1min":0.46300000000000002,"5min":0.33600000000000002,"15min":0.26100000000000001},"max":{"1min":4.8289999999999997,"5min":4.8289999999999997,"15min":104.749},"last":1.129},{"interface":"front","average":{"1min":1.2110000000000001,"5min":1.0509999999999999,"15min":1.3839999999999999},"min":{"1min":0.44700000000000001,"5min":0.33100000000000002,"15min":0.26100000000000001},"max":{"1min":3.198,"5min":4.6849999999999996,"15min":104.69499999999999},"last":1.2150000000000001}]}]},{"osd":2,"up_from":0,"seq":77309411603,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":276,"seq":1185410973793,"num_pgs":69,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1636824,"kb_used_data":1318284,"kb_used_omap":6,"kb_used_meta":318329,"kb_avail":92108328,"statfs":{"total":95995035648,"available":94318927872,"internally_reserved":0,"allocated":1349922816,"data_stored":1349563312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6335,"internal_metadata":325969729},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 19:30:39 2024","interfaces":[{"interface":"back","average":{"1min":1.2210000000000001,"5min":0.79000000000000004,"15min":0.58799999999999997},"min":{"1min":0.35999999999999999,"5min":0.16700000000000001,"15min":0.16700000000000001},"max":{"1min":3.5990000000000002,"5min":3.5990000000000002,"15min":3.5990000000000002},"last":0.55000000000000004},{"interface":"front","average":{"1min":1.1699999999999999,"5min":0.90700000000000003,"15min":0.57399999999999995},"min":{"1min":0.192,"5min":0.192,"15min":0.192},"max":{"1min":3.9449999999999998,"5min":4.5739999999999998,"15min":4.5739999999999998},"last":1.633}]},{"osd":2,"last update":"Tue Sep 10 19:29:59 2024","interfaces":[{"interface":"back","average":{"1min":0.997,"5min":0.68799999999999994,"15min":0.42799999999999999},"min":{"1min":0.313,"5min":0.23499999999999999,"15min":0.19},"max":{"1min":2.407,"5min":4.1779999999999999,"15min":4.1779999999999999},"last":0.48299999999999998},{"interface":"front","average":{"1min":1.1459999999999999,"5min":0.71399999999999997,"15min":0.46500000000000002},"min":{"1min":0.32100000000000001,"5min":0.22800000000000001,"15min":0.222},"max":{"1min":3.8580000000000001,"5min":4.1989999999999998,"15min":4.1989999999999998},"last":0.61899999999999999}]},{"osd":3,"last update":"Tue Sep 10 19:29:59 2024","interfaces":[{"interface":"back","average":{"1min":1.276,"5min":0.80500000000000005,"15min":0.47499999999999998},"min":{"1min":0.35799999999999998,"5min":0.22800000000000001,"15min":0.221},"max":{"1min":2.4969999999999999,"5min":5.7290000000000001,"15min":5.7290000000000001},"last":0.68500000000000005},{"interface":"front","average":{"1min":1.512,"5min":1.034,"15min":0.54800000000000004},"min":{"1min":0.26800000000000002,"5min":0.20899999999999999,"15min":0.17599999999999999},"max":{"1min":4.2439999999999998,"5min":21.545000000000002,"15min":21.545000000000002},"last":0.64700000000000002}]},{"osd":4,"last update":"Tue Sep 10 19:29:59 2024","interfaces":[{"interface":"back","average":{"1min":1.4299999999999999,"5min":1.244,"15min":0.78300000000000003},"min":{"1min":0.29199999999999998,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":5.8499999999999996,"5min":5.8499999999999996,"15min":5.8499999999999996},"last":1.331},{"interface":"front","average":{"1min":1.2869999999999999,"5min":1.163,"15min":0.80100000000000005},"min":{"1min":0.38700000000000001,"5min":0.30199999999999999,"15min":0.27600000000000002},"max":{"1min":2.3530000000000002,"5min":5.7599999999999998,"15min":5.7599999999999998},"last":0.73599999999999999}]},{"osd":5,"last update":"Tue Sep 10 19:29:59 2024","interfaces":[{"interface":"back","average":{"1min":1.421,"5min":1.1419999999999999,"15min":1.0169999999999999},"min":{"1min":0.32300000000000001,"5min":0.252,"15min":0.252},"max":{"1min":3.536,"5min":4.1529999999999996,"15min":4.7519999999999998},"last":1.3009999999999999},{"interface":"front","average":{"1min":1.4490000000000001,"5min":1.218,"15min":1.3069999999999999},"min":{"1min":0.47499999999999998,"5min":0.313,"15min":0.313},"max":{"1min":2.3860000000000001,"5min":4.181,"15min":5.617},"last":0.71499999999999997}]},{"osd":6,"last update":"Tue Sep 10 19:29:59 2024","interfaces":[{"interface":"back","average":{"1min":2.0489999999999999,"5min":1.363,"15min":1.931},"min":{"1min":0.30599999999999999,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":4.6319999999999997,"5min":7.5270000000000001,"15min":27.379999999999999},"last":1.9510000000000001},{"interface":"front","average":{"1min":1.7210000000000001,"5min":1.27,"15min":1.887},"min":{"1min":0.36199999999999999,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":5.8849999999999998,"5min":5.8849999999999998,"15min":26.812999999999999},"last":1.274}]},{"osd":7,"last update":"Tue Sep 10 19:29:59 2024","interfaces":[{"interface":"back","average":{"1min":1.702,"5min":1.3460000000000001,"15min":1.107},"min":{"1min":0.373,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":4.1849999999999996,"5min":8.6039999999999992,"15min":8.6039999999999992},"last":2.27},{"interface":"front","average":{"1min":1.5,"5min":1.296,"15min":1.097},"min":{"1min":0.41199999999999998,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":3.512,"5min":8.5899999999999999,"15min":8.5899999999999999},"last":3.5739999999999998}]}]},{"osd":0,"up_from":301,"seq":1292785156173,"num_pgs":65,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1164412,"kb_used_data":810100,"kb_used_omap":4,"kb_used_meta":354171,"kb_avail":92580740,"statfs":{"total":95995035648,"available":94802677760,"internally_reserved":0,"allocated":829542400,"data_stored":829199554,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":362671622},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Sep 10 19:30:34 2024","interfaces":[{"interface":"back","average":{"1min":1.7929999999999999,"5min":0.91600000000000004,"15min":0.68799999999999994},"min":{"1min":0.217,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":6.0369999999999999,"5min":6.0369999999999999,"15min":6.0369999999999999},"last":2.0019999999999998},{"interface":"front","average":{"1min":1.4430000000000001,"5min":0.75800000000000001,"15min":0.58199999999999996},"min":{"1min":0.19600000000000001,"5min":0.16900000000000001,"15min":0.16900000000000001},"max":{"1min":3.7909999999999999,"5min":3.7909999999999999,"15min":3.7909999999999999},"last":1.958}]},{"osd":2,"last update":"Tue Sep 10 19:30:34 2024","interfaces":[{"interface":"back","average":{"1min":0.84799999999999998,"5min":0.59199999999999997,"15min":0.59899999999999998},"min":{"1min":0.21299999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":4.3230000000000004,"5min":4.3230000000000004,"15min":4.3230000000000004},"last":0.247},{"interface":"front","average":{"1min":1.569,"5min":0.81799999999999995,"15min":0.69599999999999995},"min":{"1min":0.249,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":5.1079999999999997,"5min":5.1079999999999997,"15min":5.1079999999999997},"last":1.7390000000000001}]},{"osd":3,"last update":"Tue Sep 10 19:30:34 2024","interfaces":[{"interface":"back","average":{"1min":5.9119999999999999,"5min":1.6950000000000001,"15min":0.95699999999999996},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":89.468999999999994,"5min":89.468999999999994,"15min":89.468999999999994},"last":2.8839999999999999},{"interface":"front","average":{"1min":6.141,"5min":1.8200000000000001,"15min":1.022},"min":{"1min":0.216,"5min":0.216,"15min":0.216},"max":{"1min":89.272999999999996,"5min":89.272999999999996,"15min":89.272999999999996},"last":0.34399999999999997}]},{"osd":4,"last update":"Tue Sep 10 19:30:34 2024","interfaces":[{"interface":"back","average":{"1min":1.802,"5min":1.306,"15min":3.4380000000000002},"min":{"1min":0.38900000000000001,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":4.9729999999999999,"5min":6.5640000000000001,"15min":53.670000000000002},"last":1.8220000000000001},{"interface":"front","average":{"1min":1.103,"5min":1.087,"15min":3.3700000000000001},"min":{"1min":0.28000000000000003,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":3.6520000000000001,"5min":6.5019999999999998,"15min":53.935000000000002},"last":3.0209999999999999}]},{"osd":5,"last update":"Tue Sep 10 19:30:34 2024","interfaces":[{"interface":"back","average":{"1min":1.232,"5min":0.999,"15min":3.0529999999999999},"min":{"1min":0.30399999999999999,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":4.2779999999999996,"5min":4.2779999999999996,"15min":53.884999999999998},"last":2.306},{"interface":"front","average":{"1min":1.2769999999999999,"5min":1.093,"15min":2.988},"min":{"1min":0.38800000000000001,"5min":0.309,"15min":0.309},"max":{"1min":5.0519999999999996,"5min":5.0519999999999996,"15min":53.875999999999998},"last":1.8420000000000001}]},{"osd":6,"last update":"Tue Sep 10 19:30:34 2024","interfaces":[{"interface":"back","average":{"1min":1.6120000000000001,"5min":1.216,"15min":3.6240000000000001},"min":{"1min":0.34799999999999998,"5min":0.311,"15min":0.311},"max":{"1min":3.7069999999999999,"5min":5.3769999999999998,"15min":56.901000000000003},"last":5.1449999999999996},{"interface":"front","average":{"1min":1.296,"5min":1.1819999999999999,"15min":3.5609999999999999},"min":{"1min":0.29499999999999998,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":4.766,"5min":6.6840000000000002,"15min":55.177},"last":4.7560000000000002}]},{"osd":7,"last update":"Tue Sep 10 19:30:34 2024","interfaces":[{"interface":"back","average":{"1min":1.0860000000000001,"5min":1.1000000000000001,"15min":1.1910000000000001},"min":{"1min":0.32900000000000001,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":4.2149999999999999,"5min":4.2149999999999999,"15min":4.2149999999999999},"last":1.0780000000000001},{"interface":"front","average":{"1min":1.385,"5min":1.125,"15min":1.2130000000000001},"min":{"1min":0.375,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":5.0810000000000004,"5min":5.0810000000000004,"15min":5.0810000000000004},"last":1.7969999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":966,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1236,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1236,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":362,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1540,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":2118,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1018,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1502,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":822345728,"data_stored":822345728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":1340477440,"data_stored":1340473368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1027473408,"data_stored":1027473408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":730206208,"data_stored":730202136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":583401472,"data_stored":583401472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":1198788608,"data_stored":1198784536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":1168572416,"data_stored":1168572416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T19:30:45.256 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph pg dump --format=json 2024-09-10T19:30:45.258 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:30:45.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:30:45.259 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:30:45.360 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:30:45.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:30:45.396 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:30:45.396 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:30:45.446 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:30:45 smithi191 ceph-mon[32884]: pgmap v948: 158 pgs: 158 active+clean; 2.2 GiB data, 12 GiB used, 614 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.09k op/s 2024-09-10T19:30:45.446 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:30:45 smithi191 ceph-mon[32884]: from='client.25795 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-10T19:30:45.497 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:30:45.529 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:30:45.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:30:45.530 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:30:45.631 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:30:45.664 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:30:45.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:30:45.665 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:30:45.766 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:30:45.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:30:45.804 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:30:45.804 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:30:45.905 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:30:45.930 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:30:45.936 INFO:teuthology.orchestra.run.smithi183.stdout:osd.0: {} 2024-09-10T19:30:45.936 INFO:teuthology.orchestra.run.smithi183.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-10T19:30:45.939 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:30:45.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:30:45.976 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:30:45.977 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:30:45.977 INFO:teuthology.orchestra.run.smithi183.stdout:osd.1: {} 2024-09-10T19:30:45.978 INFO:teuthology.orchestra.run.smithi183.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-10T19:30:46.007 INFO:teuthology.orchestra.run.smithi183.stdout:osd.2: {} 2024-09-10T19:30:46.007 INFO:teuthology.orchestra.run.smithi183.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-10T19:30:46.037 INFO:teuthology.orchestra.run.smithi183.stdout:osd.3: {} 2024-09-10T19:30:46.037 INFO:teuthology.orchestra.run.smithi183.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-10T19:30:46.066 INFO:teuthology.orchestra.run.smithi183.stdout:osd.4: {} 2024-09-10T19:30:46.066 INFO:teuthology.orchestra.run.smithi183.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-10T19:30:46.077 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:30:46.098 INFO:teuthology.orchestra.run.smithi183.stdout:osd.5: {} 2024-09-10T19:30:46.099 INFO:teuthology.orchestra.run.smithi183.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-10T19:30:46.112 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:30:46.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:30:46.113 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:30:46.130 INFO:teuthology.orchestra.run.smithi183.stdout:osd.6: {} 2024-09-10T19:30:46.130 INFO:teuthology.orchestra.run.smithi183.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-10T19:30:46.161 INFO:teuthology.orchestra.run.smithi183.stdout:osd.7: {} 2024-09-10T19:30:46.161 INFO:teuthology.orchestra.run.smithi183.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-10T19:30:46.214 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:30:46.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:30:46 smithi191 ceph-mon[32884]: purged_snaps scrub starts 2024-09-10T19:30:46.217 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:30:46 smithi191 ceph-mon[32884]: purged_snaps scrub ok 2024-09-10T19:30:46.217 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:30:46 smithi191 ceph-mon[32884]: from='client.? 172.21.15.183:0/1979372271' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T19:30:46.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:30:46.250 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:30:46.251 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:30:46.351 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:30:46.355 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:30:46 smithi183 ceph-mon[29736]: purged_snaps scrub starts 2024-09-10T19:30:46.355 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:30:46 smithi183 ceph-mon[29736]: purged_snaps scrub ok 2024-09-10T19:30:46.356 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:30:46 smithi183 ceph-mon[29736]: from='client.? 172.21.15.183:0/1979372271' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T19:30:46.356 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:30:46 smithi183 ceph-mon[41562]: purged_snaps scrub starts 2024-09-10T19:30:46.356 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:30:46 smithi183 ceph-mon[41562]: purged_snaps scrub ok 2024-09-10T19:30:46.356 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:30:46 smithi183 ceph-mon[41562]: from='client.? 172.21.15.183:0/1979372271' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T19:30:46.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:30:46.392 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:30:46.392 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:30:46.493 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:30:46.530 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:30:46.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:30:46.531 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:30:46.570 INFO:teuthology.orchestra.run.smithi191.stdout:{ 2024-09-10T19:30:46.570 INFO:teuthology.orchestra.run.smithi191.stdout: "ops": [ 2024-09-10T19:30:46.570 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:30:46.570 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_repop(client.16026.0:262535 11.5 e419/374 11:a3094a1a:::benchmark_data_smithi183_203944_object117645:head v 419'16710, mlcod=419'16705)", 2024-09-10T19:30:46.570 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-10T19:30:46.565186+0000", 2024-09-10T19:30:46.570 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 0.002224831, 2024-09-10T19:30:46.570 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0022533359999999999, 2024-09-10T19:30:46.570 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-10T19:30:46.570 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "started", 2024-09-10T19:30:46.570 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-10T19:30:46.571 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:30:46.571 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-10T19:30:46.571 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:30:46.565186+0000", 2024-09-10T19:30:46.571 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:30:46.571 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:30:46.571 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:30:46.571 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-10T19:30:46.571 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:30:46.565186+0000", 2024-09-10T19:30:46.571 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:30:46.571 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:30:46.571 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:30:46.571 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-10T19:30:46.571 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:30:46.565185+0000", 2024-09-10T19:30:46.572 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.999999 2024-09-10T19:30:46.572 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:30:46.572 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:30:46.572 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-10T19:30:46.572 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:30:46.565193+0000", 2024-09-10T19:30:46.572 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.52e-06 2024-09-10T19:30:46.572 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:30:46.572 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:30:46.572 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-10T19:30:46.572 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:30:46.565207+0000", 2024-09-10T19:30:46.572 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.4595e-05 2024-09-10T19:30:46.572 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:30:46.572 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:30:46.573 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-10T19:30:46.573 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:30:46.565213+0000", 2024-09-10T19:30:46.573 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.0589999999999999e-06 2024-09-10T19:30:46.573 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:30:46.573 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:30:46.573 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:30:46.573 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:30:46.565807+0000", 2024-09-10T19:30:46.573 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00059348099999999998 2024-09-10T19:30:46.573 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:30:46.573 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:30:46.573 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-10T19:30:46.573 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:30:46.565824+0000", 2024-09-10T19:30:46.573 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.7312999999999998e-05 2024-09-10T19:30:46.574 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:30:46.574 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-10T19:30:46.574 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:30:46.574 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:30:46.574 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:30:46.574 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_repop(client.16026.0:262518 11.5 e419/374 11:a8fd60ee:::benchmark_data_smithi183_203944_object117628:head v 419'16707, mlcod=419'16704)", 2024-09-10T19:30:46.574 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-10T19:30:46.554329+0000", 2024-09-10T19:30:46.574 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 0.013082204, 2024-09-10T19:30:46.574 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.013179606, 2024-09-10T19:30:46.574 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-10T19:30:46.574 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "started", 2024-09-10T19:30:46.574 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-10T19:30:46.575 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:30:46.575 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-10T19:30:46.575 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:30:46.554329+0000", 2024-09-10T19:30:46.575 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:30:46.575 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:30:46.575 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:30:46.575 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-10T19:30:46.575 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:30:46.554329+0000", 2024-09-10T19:30:46.575 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:30:46.575 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:30:46.575 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:30:46.575 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-10T19:30:46.575 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:30:46.554328+0000", 2024-09-10T19:30:46.576 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.999999 2024-09-10T19:30:46.576 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:30:46.576 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:30:46.576 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-10T19:30:46.576 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:30:46.554336+0000", 2024-09-10T19:30:46.576 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.8159999999999994e-06 2024-09-10T19:30:46.576 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:30:46.576 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:30:46.576 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-10T19:30:46.576 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:30:46.554353+0000", 2024-09-10T19:30:46.576 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.7093000000000001e-05 2024-09-10T19:30:46.576 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:30:46.576 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:30:46.577 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-10T19:30:46.577 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:30:46.554359+0000", 2024-09-10T19:30:46.577 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.5849999999999996e-06 2024-09-10T19:30:46.577 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:30:46.577 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:30:46.577 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:30:46.577 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:30:46.554497+0000", 2024-09-10T19:30:46.577 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00013758299999999999 2024-09-10T19:30:46.577 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:30:46.577 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:30:46.577 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-10T19:30:46.577 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:30:46.554549+0000", 2024-09-10T19:30:46.577 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.2104999999999997e-05 2024-09-10T19:30:46.578 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:30:46.578 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-10T19:30:46.578 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:30:46.578 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:30:46.578 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:30:46.578 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_repop(client.16026.0:262525 11.5 e419/374 11:a85aa9d9:::benchmark_data_smithi183_203944_object117635:head v 419'16708, mlcod=419'16704)", 2024-09-10T19:30:46.578 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-10T19:30:46.557101+0000", 2024-09-10T19:30:46.578 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 0.010310211, 2024-09-10T19:30:46.578 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.010468740000000001, 2024-09-10T19:30:46.578 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-10T19:30:46.578 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "started", 2024-09-10T19:30:46.578 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-10T19:30:46.578 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:30:46.579 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-10T19:30:46.579 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:30:46.557101+0000", 2024-09-10T19:30:46.579 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:30:46.579 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:30:46.579 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:30:46.579 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-10T19:30:46.579 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:30:46.557101+0000", 2024-09-10T19:30:46.579 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:30:46.580 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:30:46.580 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:30:46.580 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-10T19:30:46.580 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:30:46.557100+0000", 2024-09-10T19:30:46.580 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.999999 2024-09-10T19:30:46.580 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:30:46.580 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:30:46.580 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-10T19:30:46.580 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:30:46.557107+0000", 2024-09-10T19:30:46.580 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.9920000000000001e-06 2024-09-10T19:30:46.580 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:30:46.580 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:30:46.580 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-10T19:30:46.581 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:30:46.557121+0000", 2024-09-10T19:30:46.581 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.3918e-05 2024-09-10T19:30:46.581 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:30:46.581 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:30:46.581 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-10T19:30:46.581 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:30:46.557126+0000", 2024-09-10T19:30:46.581 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.2859999999999999e-06 2024-09-10T19:30:46.581 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:30:46.581 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:30:46.581 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:30:46.581 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:30:46.561230+0000", 2024-09-10T19:30:46.581 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0041037000000000001 2024-09-10T19:30:46.581 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:30:46.581 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:30:46.581 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-10T19:30:46.582 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:30:46.561274+0000", 2024-09-10T19:30:46.582 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.405e-05 2024-09-10T19:30:46.582 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:30:46.582 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-10T19:30:46.582 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:30:46.582 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:30:46.582 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:30:46.582 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_repop(client.16026.0:262527 11.5 e419/374 11:aaa82c97:::benchmark_data_smithi183_203944_object117637:head v 419'16709, mlcod=419'16705)", 2024-09-10T19:30:46.582 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-10T19:30:46.559596+0000", 2024-09-10T19:30:46.582 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 0.0078146429999999996, 2024-09-10T19:30:46.582 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0080270629999999992, 2024-09-10T19:30:46.582 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-10T19:30:46.582 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "started", 2024-09-10T19:30:46.583 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-10T19:30:46.583 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:30:46.583 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-10T19:30:46.583 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:30:46.559596+0000", 2024-09-10T19:30:46.583 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:30:46.583 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:30:46.583 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:30:46.583 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-10T19:30:46.583 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:30:46.559596+0000", 2024-09-10T19:30:46.583 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:30:46.583 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:30:46.583 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:30:46.583 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-10T19:30:46.583 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:30:46.559596+0000", 2024-09-10T19:30:46.583 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.999999 2024-09-10T19:30:46.584 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:30:46.584 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:30:46.584 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-10T19:30:46.584 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:30:46.559603+0000", 2024-09-10T19:30:46.584 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.1219999999999997e-06 2024-09-10T19:30:46.584 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:30:46.584 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:30:46.584 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-10T19:30:46.584 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:30:46.559616+0000", 2024-09-10T19:30:46.584 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.3242e-05 2024-09-10T19:30:46.584 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:30:46.584 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:30:46.584 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-10T19:30:46.584 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:30:46.559622+0000", 2024-09-10T19:30:46.585 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.6200000000000004e-06 2024-09-10T19:30:46.585 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:30:46.585 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:30:46.585 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:30:46.585 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:30:46.563677+0000", 2024-09-10T19:30:46.585 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0040555269999999997 2024-09-10T19:30:46.585 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:30:46.585 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:30:46.585 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-10T19:30:46.585 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:30:46.563695+0000", 2024-09-10T19:30:46.585 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.8045000000000001e-05 2024-09-10T19:30:46.585 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:30:46.585 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-10T19:30:46.585 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:30:46.586 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:30:46.586 INFO:teuthology.orchestra.run.smithi191.stdout: ], 2024-09-10T19:30:46.586 INFO:teuthology.orchestra.run.smithi191.stdout: "num_ops": 4 2024-09-10T19:30:46.586 INFO:teuthology.orchestra.run.smithi191.stdout:} 2024-09-10T19:30:46.632 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:30:46.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:30:46.670 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:30:46.670 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:30:46.771 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:30:46.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:30:46.809 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:30:46.809 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:30:46.910 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:30:46.952 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:30:46.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:30:46.955 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:30:47.056 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:30:47.084 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:30:47.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:30:47.085 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:30:47.186 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:30:47.225 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:30:47.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:30:47.227 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:30:47.327 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:30:47.348 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:30:47 smithi183 ceph-mon[29736]: pgmap v949: 158 pgs: 158 active+clean; 1.8 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1.49k op/s 2024-09-10T19:30:47.348 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:30:47 smithi183 ceph-mon[29736]: osdmap e419: 8 total, 8 up, 7 in 2024-09-10T19:30:47.348 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:30:47 smithi183 ceph-mon[29736]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T19:30:47.348 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:30:47 smithi183 ceph-mon[41562]: pgmap v949: 158 pgs: 158 active+clean; 1.8 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1.49k op/s 2024-09-10T19:30:47.349 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:30:47 smithi183 ceph-mon[41562]: osdmap e419: 8 total, 8 up, 7 in 2024-09-10T19:30:47.349 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:30:47 smithi183 ceph-mon[41562]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T19:30:47.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:30:47.350 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:30:47.350 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:30:47.451 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:30:47.451 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:30:47 smithi191 ceph-mon[32884]: pgmap v949: 158 pgs: 158 active+clean; 1.8 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1.49k op/s 2024-09-10T19:30:47.451 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:30:47 smithi191 ceph-mon[32884]: osdmap e419: 8 total, 8 up, 7 in 2024-09-10T19:30:47.452 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:30:47 smithi191 ceph-mon[32884]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T19:30:47.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:30:47.473 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:30:47.473 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:30:47.574 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:30:47.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:30:47.596 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:30:47.597 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:30:47.697 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:30:47.716 INFO:journalctl@ceph.mgr.x.smithi191.stdout:Sep 10 19:30:47 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-x[34967]: ::ffff:172.21.15.191 - - [10/Sep/2024:19:30:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T19:30:47.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:30:47.731 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:30:47.731 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:30:47.832 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:30:47.866 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:30:47.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:30:47.867 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:30:47.968 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:30:48.010 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:30:48.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:30:48.011 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:30:48.096 DEBUG:teuthology.orchestra.run.smithi191:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph daemon osd.5 dump_blocked_ops 2024-09-10T19:30:48.112 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:30:48.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:30:48.214 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:30:48.214 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:30:48.315 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:30:48.356 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:30:48.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:30:48.357 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:30:48.458 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:30:48.499 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:30:48.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:30:48.500 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:30:48.524 INFO:teuthology.orchestra.run.smithi191.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.b/config 2024-09-10T19:30:48.600 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:30:48.629 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:30:48.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:30:48.630 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:30:48.731 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:30:48.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:30:48.763 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:30:48.763 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:30:48.864 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:30:48.908 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:30:48.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:30:48.909 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:30:49.010 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:30:49.044 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:30:49.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:30:49.045 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:30:49.146 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:30:49.184 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:30:49.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:30:49.185 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:30:49.286 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:30:49.322 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:30:49.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:30:49.323 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:30:49.424 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:30:49.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:30:49 smithi183 ceph-mon[29736]: pgmap v951: 158 pgs: 158 active+clean; 1.7 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.66k op/s 2024-09-10T19:30:49.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:30:49 smithi183 ceph-mon[29736]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T19:30:49.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:30:49 smithi183 ceph-mon[29736]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T19:30:49.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:30:49 smithi183 ceph-mon[29736]: from='mgr.14556 ' entity='mgr.y' 2024-09-10T19:30:49.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:30:49 smithi183 ceph-mon[41562]: pgmap v951: 158 pgs: 158 active+clean; 1.7 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.66k op/s 2024-09-10T19:30:49.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:30:49 smithi183 ceph-mon[41562]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T19:30:49.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:30:49 smithi183 ceph-mon[41562]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T19:30:49.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:30:49 smithi183 ceph-mon[41562]: from='mgr.14556 ' entity='mgr.y' 2024-09-10T19:30:49.458 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:30:49.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:30:49.459 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:30:49.465 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:30:49 smithi191 ceph-mon[32884]: pgmap v951: 158 pgs: 158 active+clean; 1.7 GiB data, 11 GiB used, 615 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.66k op/s 2024-09-10T19:30:49.465 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:30:49 smithi191 ceph-mon[32884]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T19:30:49.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:30:49 smithi191 ceph-mon[32884]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T19:30:49.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:30:49 smithi191 ceph-mon[32884]: from='mgr.14556 ' entity='mgr.y' 2024-09-10T19:30:49.560 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:30:49.593 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:30:49.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:30:49.594 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:30:49.695 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:30:49.727 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:30:49.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:30:49.728 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:30:49.829 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:30:49.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:30:49.857 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:30:49.858 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:30:49.958 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:30:50.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:30:50.007 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:30:50.008 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:30:50.108 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:30:50.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:30:50.143 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:30:50.143 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:30:50.216 INFO:journalctl@ceph.iscsi.iscsi.a.smithi191.stdout:Sep 10 19:30:50 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-iscsi-iscsi-a[61816]: debug there is no tcmu-runner data available 2024-09-10T19:30:50.245 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:30:50.273 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:30:50.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:30:50.274 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:30:50.375 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:30:50.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:30:50.410 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:30:50.410 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:30:50.483 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-10T19:30:50.512 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:30:50.563 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:30:50.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:30:50.564 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:30:50.587 INFO:teuthology.orchestra.run.smithi191.stdout:{ 2024-09-10T19:30:50.587 INFO:teuthology.orchestra.run.smithi191.stdout: "ops": [], 2024-09-10T19:30:50.587 INFO:teuthology.orchestra.run.smithi191.stdout: "complaint_time": 30, 2024-09-10T19:30:50.587 INFO:teuthology.orchestra.run.smithi191.stdout: "num_blocked_ops": 0 2024-09-10T19:30:50.588 INFO:teuthology.orchestra.run.smithi191.stdout:} 2024-09-10T19:30:50.665 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:30:50.703 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:30:50.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:30:50.704 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:30:50.804 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:30:50.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:30:50.886 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:30:50.886 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:30:50.987 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:30:51.018 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:30:51.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:30:51.019 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:30:51.120 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:30:51.155 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:30:51.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:30:51.156 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:30:51.257 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:30:51.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:30:51.286 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:30:51.287 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:30:51.326 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:30:51 smithi191 ceph-mon[32884]: pgmap v952: 158 pgs: 158 active+clean; 1.7 GiB data, 10 GiB used, 615 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.14k op/s 2024-09-10T19:30:51.326 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:30:51 smithi191 ceph-mon[32884]: from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:30:51.367 DEBUG:teuthology.orchestra.run.smithi191:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph daemon osd.5 dump_historic_ops 2024-09-10T19:30:51.387 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:30:51.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:30:51 smithi183 ceph-mon[29736]: pgmap v952: 158 pgs: 158 active+clean; 1.7 GiB data, 10 GiB used, 615 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.14k op/s 2024-09-10T19:30:51.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:30:51 smithi183 ceph-mon[29736]: from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:30:51.432 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Sep 10 19:30:51 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-y[29989]: ::ffff:172.21.15.191 - - [10/Sep/2024:19:30:51] "GET /metrics HTTP/1.1" 200 224944 "" "Prometheus/2.43.0" 2024-09-10T19:30:51.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:30:51 smithi183 ceph-mon[41562]: pgmap v952: 158 pgs: 158 active+clean; 1.7 GiB data, 10 GiB used, 615 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1.14k op/s 2024-09-10T19:30:51.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:30:51 smithi183 ceph-mon[41562]: from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:30:51.447 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:30:51.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:30:51.448 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:30:51.549 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:30:51.586 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:30:51.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:30:51.587 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:30:51.688 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:30:51.721 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:30:51.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:30:51.722 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:30:51.819 INFO:teuthology.orchestra.run.smithi191.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.b/config 2024-09-10T19:30:51.823 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:30:51.861 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:30:51.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:30:51.862 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:30:51.963 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:30:51.997 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:30:51.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:30:51.998 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:30:52.099 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:30:52.137 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:30:52.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:30:52.138 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:30:52.239 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:30:52.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:30:52.278 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:30:52.278 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:30:52.379 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:30:52.415 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:30:52.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:30:52.416 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:30:52.517 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:30:52.556 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:30:52.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:30:52.557 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:30:52.658 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:30:52.690 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:30:52.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:30:52.691 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:30:52.791 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:30:52.827 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:30:52.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:30:52.828 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:30:52.928 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:30:52.959 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:30:52.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:30:52.960 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:30:53.061 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:30:53.085 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:30:53.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:30:53.086 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:30:53.189 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:30:53.221 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:30:53.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:30:53.222 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:30:53.323 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:30:53.349 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:30:53.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:30:53.350 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:30:53.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:30:53 smithi183 ceph-mon[29736]: pgmap v953: 158 pgs: 158 active+clean; 1.4 GiB data, 9.9 GiB used, 616 GiB / 626 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1.36k op/s 2024-09-10T19:30:53.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:30:53 smithi183 ceph-mon[41562]: pgmap v953: 158 pgs: 158 active+clean; 1.4 GiB data, 9.9 GiB used, 616 GiB / 626 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1.36k op/s 2024-09-10T19:30:53.451 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:30:53.465 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:30:53 smithi191 ceph-mon[32884]: pgmap v953: 158 pgs: 158 active+clean; 1.4 GiB data, 9.9 GiB used, 616 GiB / 626 GiB avail; 1.0 KiB/s rd, 0 B/s wr, 1.36k op/s 2024-09-10T19:30:53.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:30:53.495 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:30:53.495 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:30:53.597 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:30:53.629 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:30:53.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:30:53.631 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:30:53.731 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:30:53.764 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:30:53.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:30:53.765 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:30:53.866 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:30:53.897 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:30:53.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:30:53.898 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:30:53.999 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:30:54.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:30:54.044 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:30:54.044 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:30:54.145 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:30:54.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:30:54.185 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:30:54.186 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:30:54.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:30:54 smithi191 ceph-mon[32884]: purged_snaps scrub starts 2024-09-10T19:30:54.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:30:54 smithi191 ceph-mon[32884]: purged_snaps scrub ok 2024-09-10T19:30:54.224 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:30:54 smithi183 ceph-mon[29736]: purged_snaps scrub starts 2024-09-10T19:30:54.224 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:30:54 smithi183 ceph-mon[29736]: purged_snaps scrub ok 2024-09-10T19:30:54.224 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:30:54 smithi183 ceph-mon[41562]: purged_snaps scrub starts 2024-09-10T19:30:54.225 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:30:54 smithi183 ceph-mon[41562]: purged_snaps scrub ok 2024-09-10T19:30:54.286 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:30:54.315 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:30:54.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:30:54.317 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:30:54.418 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:30:54.448 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:30:54.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:30:54.450 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:30:54.529 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:30:54.550 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:30:54.599 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:30:54.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:30:54.600 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:30:54.607 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-09-10T19:30:54.616 INFO:teuthology.orchestra.run.smithi183.stderr:dumped all 2024-09-10T19:30:54.701 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:30:54.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:30:54.737 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:30:54.737 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:30:54.838 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:30:54.859 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:30:54.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:30:54.860 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:30:54.961 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:30:54.997 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:30:54.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:30:54.998 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:30:55.099 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:30:55.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:30:55.131 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:30:55.131 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:30:55.158 INFO:teuthology.orchestra.run.smithi183.stderr:nodeep-scrub is set 2024-09-10T19:30:55.172 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:30:55 smithi183 ceph-mon[29736]: pgmap v954: 158 pgs: 158 active+clean; 1.3 GiB data, 9.5 GiB used, 616 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.33k op/s 2024-09-10T19:30:55.172 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:30:55 smithi183 ceph-mon[29736]: osdmap e420: 8 total, 8 up, 7 in 2024-09-10T19:30:55.172 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:30:55 smithi183 ceph-mon[29736]: from='client.? 172.21.15.183:0/996412043' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-10T19:30:55.172 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:30:55 smithi183 ceph-mon[29736]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-10T19:30:55.172 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:30:55 smithi183 ceph-mon[41562]: pgmap v954: 158 pgs: 158 active+clean; 1.3 GiB data, 9.5 GiB used, 616 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.33k op/s 2024-09-10T19:30:55.173 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:30:55 smithi183 ceph-mon[41562]: osdmap e420: 8 total, 8 up, 7 in 2024-09-10T19:30:55.173 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:30:55 smithi183 ceph-mon[41562]: from='client.? 172.21.15.183:0/996412043' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-10T19:30:55.173 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:30:55 smithi183 ceph-mon[41562]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-10T19:30:55.232 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:30:55.267 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:30:55.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:30:55.269 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:30:55.369 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:30:55.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:30:55.403 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:30:55.403 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:30:55.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:30:55 smithi191 ceph-mon[32884]: pgmap v954: 158 pgs: 158 active+clean; 1.3 GiB data, 9.5 GiB used, 616 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.33k op/s 2024-09-10T19:30:55.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:30:55 smithi191 ceph-mon[32884]: osdmap e420: 8 total, 8 up, 7 in 2024-09-10T19:30:55.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:30:55 smithi191 ceph-mon[32884]: from='client.? 172.21.15.183:0/996412043' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-10T19:30:55.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:30:55 smithi191 ceph-mon[32884]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-10T19:30:55.504 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:30:55.541 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:30:55.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:30:55.542 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:30:55.643 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:30:55.681 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:30:55.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:30:55.682 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:30:55.783 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:30:55.817 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:30:55.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:30:55.818 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:30:55.919 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:30:55.955 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:30:55.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:30:55.956 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:30:56.056 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:30:56.096 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:30:56.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:30:56.097 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:30:56.198 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:30:56.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:30:56.237 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:30:56.237 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:30:56.248 INFO:teuthology.orchestra.run.smithi183.stdout:{"pg_ready":true,"pg_map":{"version":955,"stamp":"2024-09-10T19:30:54.141945+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1441798388,"num_objects":22188,"num_object_clones":0,"num_object_copies":66564,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22188,"num_whiteouts":0,"num_read":1978,"num_read_kb":1730,"num_write":268786,"num_write_kb":9272812,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":458,"num_bytes_recovered":536482,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72360,"ondisk_log_size":72360,"up":474,"acting":474,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":474,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":656216064,"kb_used":9943112,"kb_used_data":4240912,"kb_used_omap":19,"kb_used_meta":5702124,"kb_avail":646272952,"statfs":{"total":671965249536,"available":661783502848,"internally_reserved":0,"allocated":4342693888,"data_stored":4340241318,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19457,"internal_metadata":5838975999},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[7,2,11,8,7,0,0,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-502857728,"num_objects":-7673,"num_object_clones":0,"num_object_copies":-23019,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-7673,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":7673,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.156135"},"pg_stats":[{"pgid":"6.1b","version":"0'0","reported_seq":164,"reported_epoch":418,"state":"active+clean","last_fresh":"2024-09-10T19:30:42.072342+0000","last_change":"2024-09-10T19:29:35.538355+0000","last_active":"2024-09-10T19:30:42.072342+0000","last_peered":"2024-09-10T19:30:42.072342+0000","last_clean":"2024-09-10T19:30:42.072342+0000","last_became_active":"2024-09-10T19:27:52.994757+0000","last_became_peered":"2024-09-10T19:27:52.994757+0000","last_unstale":"2024-09-10T19:30:42.072342+0000","last_undegraded":"2024-09-10T19:30:42.072342+0000","last_fullsized":"2024-09-10T19:30:42.072342+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:29:35.538284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:28:18.455084+0000","last_clean_scrub_stamp":"2024-09-10T19:29:35.538284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:30:57.495675+0000","scrub_duration":0.0025697599999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6],"acting":[0,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":412,"reported_epoch":418,"state":"active+clean","last_fresh":"2024-09-10T19:30:42.071333+0000","last_change":"2024-09-10T19:29:32.579218+0000","last_active":"2024-09-10T19:30:42.071333+0000","last_peered":"2024-09-10T19:30:42.071333+0000","last_clean":"2024-09-10T19:30:42.071333+0000","last_became_active":"2024-09-10T19:24:23.892621+0000","last_became_peered":"2024-09-10T19:24:23.892621+0000","last_unstale":"2024-09-10T19:30:42.071333+0000","last_undegraded":"2024-09-10T19:30:42.071333+0000","last_fullsized":"2024-09-10T19:30:42.071333+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:29:32.579092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:16:38.194472+0000","last_clean_scrub_stamp":"2024-09-10T19:29:32.579092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:30:51.229113+0000","scrub_duration":0.010689732,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.19","version":"414'39","reported_seq":531,"reported_epoch":418,"state":"active+clean","last_fresh":"2024-09-10T19:30:42.076254+0000","last_change":"2024-09-10T19:30:09.389815+0000","last_active":"2024-09-10T19:30:42.076254+0000","last_peered":"2024-09-10T19:30:42.076254+0000","last_clean":"2024-09-10T19:30:42.076254+0000","last_became_active":"2024-09-10T19:24:23.906592+0000","last_became_peered":"2024-09-10T19:24:23.906592+0000","last_unstale":"2024-09-10T19:30:42.076254+0000","last_undegraded":"2024-09-10T19:30:42.076254+0000","last_fullsized":"2024-09-10T19:30:42.076254+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"237'27","last_scrub_stamp":"2024-09-10T19:30:09.389670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:30:09.389670+0000","objects_scrubbed":7,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:31:38.619896+0000","scrub_duration":0.003161433,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":62,"num_read_kb":53,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":507,"reported_epoch":418,"state":"active+clean","last_fresh":"2024-09-10T19:30:42.073672+0000","last_change":"2024-09-10T19:30:41.457105+0000","last_active":"2024-09-10T19:30:42.073672+0000","last_peered":"2024-09-10T19:30:42.073672+0000","last_clean":"2024-09-10T19:30:42.073672+0000","last_became_active":"2024-09-10T19:27:53.006175+0000","last_became_peered":"2024-09-10T19:27:53.006175+0000","last_unstale":"2024-09-10T19:30:42.073672+0000","last_undegraded":"2024-09-10T19:30:42.073672+0000","last_fullsized":"2024-09-10T19:30:42.073672+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:30:41.457019+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:19:28.498560+0000","last_clean_scrub_stamp":"2024-09-10T19:30:41.457019+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:32:07.134757+0000","scrub_duration":0.0038471270000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"384'2","reported_seq":475,"reported_epoch":418,"state":"active+clean","last_fresh":"2024-09-10T19:30:42.076932+0000","last_change":"2024-09-10T19:30:34.918319+0000","last_active":"2024-09-10T19:30:42.076932+0000","last_peered":"2024-09-10T19:30:42.076932+0000","last_clean":"2024-09-10T19:30:42.076932+0000","last_became_active":"2024-09-10T19:24:23.908614+0000","last_became_peered":"2024-09-10T19:24:23.908614+0000","last_unstale":"2024-09-10T19:30:42.076932+0000","last_undegraded":"2024-09-10T19:30:42.076932+0000","last_fullsized":"2024-09-10T19:30:42.076932+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"384'2","last_scrub_stamp":"2024-09-10T19:30:34.918244+0000","last_deep_scrub":"384'2","last_deep_scrub_stamp":"2024-09-10T19:28:23.240824+0000","last_clean_scrub_stamp":"2024-09-10T19:30:34.918244+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:31:35.083745+0000","scrub_duration":0.0051432630000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":874,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"414'25","reported_seq":574,"reported_epoch":418,"state":"active+clean","last_fresh":"2024-09-10T19:30:42.073750+0000","last_change":"2024-09-10T19:30:24.549021+0000","last_active":"2024-09-10T19:30:42.073750+0000","last_peered":"2024-09-10T19:30:42.073750+0000","last_clean":"2024-09-10T19:30:42.073750+0000","last_became_active":"2024-09-10T19:27:53.013509+0000","last_became_peered":"2024-09-10T19:27:53.013509+0000","last_unstale":"2024-09-10T19:30:42.073750+0000","last_undegraded":"2024-09-10T19:30:42.073750+0000","last_fullsized":"2024-09-10T19:30:42.073750+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"414'25","last_scrub_stamp":"2024-09-10T19:30:24.548931+0000","last_deep_scrub":"384'19","last_deep_scrub_stamp":"2024-09-10T19:28:21.799921+0000","last_clean_scrub_stamp":"2024-09-10T19:30:24.548931+0000","objects_scrubbed":4,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:31:44.970028+0000","scrub_duration":0.0074165239999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":47,"reported_epoch":418,"state":"active+clean","last_fresh":"2024-09-10T19:30:42.078216+0000","last_change":"2024-09-10T19:30:06.545197+0000","last_active":"2024-09-10T19:30:42.078216+0000","last_peered":"2024-09-10T19:30:42.078216+0000","last_clean":"2024-09-10T19:30:42.078216+0000","last_became_active":"2024-09-10T19:28:47.151481+0000","last_became_peered":"2024-09-10T19:28:47.151481+0000","last_unstale":"2024-09-10T19:30:42.078216+0000","last_undegraded":"2024-09-10T19:30:42.078216+0000","last_fullsized":"2024-09-10T19:30:42.078216+0000","mapping_epoch":391,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":392,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:30:06.545123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:30:06.545123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:31:19.928036+0000","scrub_duration":0.0025621239999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,7],"acting":[6,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":475,"reported_epoch":418,"state":"active+clean","last_fresh":"2024-09-10T19:30:42.074122+0000","last_change":"2024-09-10T19:30:16.469108+0000","last_active":"2024-09-10T19:30:42.074122+0000","last_peered":"2024-09-10T19:30:42.074122+0000","last_clean":"2024-09-10T19:30:42.074122+0000","last_became_active":"2024-09-10T19:24:01.971921+0000","last_became_peered":"2024-09-10T19:24:01.971921+0000","last_unstale":"2024-09-10T19:30:42.074122+0000","last_undegraded":"2024-09-10T19:30:42.074122+0000","last_fullsized":"2024-09-10T19:30:42.074122+0000","mapping_epoch":292,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":293,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:30:16.469056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:30:16.469056+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:31:23.825602+0000","scrub_duration":0.001491469,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":476,"reported_epoch":418,"state":"active+clean","last_fresh":"2024-09-10T19:30:42.087490+0000","last_change":"2024-09-10T19:29:11.890872+0000","last_active":"2024-09-10T19:30:42.087490+0000","last_peered":"2024-09-10T19:30:42.087490+0000","last_clean":"2024-09-10T19:30:42.087490+0000","last_became_active":"2024-09-10T19:27:53.011202+0000","last_became_peered":"2024-09-10T19:27:53.011202+0000","last_unstale":"2024-09-10T19:30:42.087490+0000","last_undegraded":"2024-09-10T19:30:42.087490+0000","last_fullsized":"2024-09-10T19:30:42.087490+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:29:11.890794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:29:11.890794+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.002093588,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1b","version":"414'9","reported_seq":520,"reported_epoch":418,"state":"active+clean","last_fresh":"2024-09-10T19:30:42.074493+0000","last_change":"2024-09-10T19:30:40.489631+0000","last_active":"2024-09-10T19:30:42.074493+0000","last_peered":"2024-09-10T19:30:42.074493+0000","last_clean":"2024-09-10T19:30:42.074493+0000","last_became_active":"2024-09-10T19:24:01.974813+0000","last_became_peered":"2024-09-10T19:24:01.974813+0000","last_unstale":"2024-09-10T19:30:42.074493+0000","last_undegraded":"2024-09-10T19:30:42.074493+0000","last_fullsized":"2024-09-10T19:30:42.074493+0000","mapping_epoch":292,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":293,"parent":"0.0","parent_split_bits":0,"last_scrub":"414'9","last_scrub_stamp":"2024-09-10T19:30:40.489534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:30:40.489534+0000","objects_scrubbed":2,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:31:41.703879+0000","scrub_duration":0.0037854310000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":17,"num_read_kb":13,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":469,"reported_epoch":418,"state":"active+clean","last_fresh":"2024-09-10T19:30:42.084367+0000","last_change":"2024-09-10T19:30:19.910559+0000","last_active":"2024-09-10T19:30:42.084367+0000","last_peered":"2024-09-10T19:30:42.084367+0000","last_clean":"2024-09-10T19:30:42.084367+0000","last_became_active":"2024-09-10T19:24:01.976587+0000","last_became_peered":"2024-09-10T19:24:01.976587+0000","last_unstale":"2024-09-10T19:30:42.084367+0000","last_undegraded":"2024-09-10T19:30:42.084367+0000","last_fullsized":"2024-09-10T19:30:42.084367+0000","mapping_epoch":292,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":293,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:30:19.910433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:30:19.910433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:31:47.412483+0000","scrub_duration":0.0056012320000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":470,"reported_epoch":418,"state":"active+clean","last_fresh":"2024-09-10T19:30:42.075518+0000","last_change":"2024-09-10T19:30:08.321635+0000","last_active":"2024-09-10T19:30:42.075518+0000","last_peered":"2024-09-10T19:30:42.075518+0000","last_clean":"2024-09-10T19:30:42.075518+0000","last_became_active":"2024-09-10T19:24:01.970118+0000","last_became_peered":"2024-09-10T19:24:01.970118+0000","last_unstale":"2024-09-10T19:30:42.075518+0000","last_undegraded":"2024-09-10T19:30:42.075518+0000","last_fullsized":"2024-09-10T19:30:42.075518+0000","mapping_epoch":292,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":293,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:30:08.321561+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:30:08.321561+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:31:28.385750+0000","scrub_duration":0.0054793990000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":479,"reported_epoch":418,"state":"active+clean","last_fresh":"2024-09-10T19:30:42.084096+0000","last_change":"2024-09-10T19:29:31.166120+0000","last_active":"2024-09-10T19:30:42.084096+0000","last_peered":"2024-09-10T19:30:42.084096+0000","last_clean":"2024-09-10T19:30:42.084096+0000","last_became_active":"2024-09-10T19:27:53.008555+0000","last_became_peered":"2024-09-10T19:27:53.008555+0000","last_unstale":"2024-09-10T19:30:42.084096+0000","last_undegraded":"2024-09-10T19:30:42.084096+0000","last_fullsized":"2024-09-10T19:30:42.084096+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:29:31.166021+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:29:31.166021+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:30:52.918293+0000","scrub_duration":0.012493738000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1a","version":"414'21","reported_seq":490,"reported_epoch":418,"state":"active+clean","last_fresh":"2024-09-10T19:30:42.074318+0000","last_change":"2024-09-10T19:30:17.531183+0000","last_active":"2024-09-10T19:30:42.074318+0000","last_peered":"2024-09-10T19:30:42.074318+0000","last_clean":"2024-09-10T19:30:42.074318+0000","last_became_active":"2024-09-10T19:24:23.891213+0000","last_became_peered":"2024-09-10T19:24:23.891213+0000","last_unstale":"2024-09-10T19:30:42.074318+0000","last_undegraded":"2024-09-10T19:30:42.074318+0000","last_fullsized":"2024-09-10T19:30:42.074318+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"414'21","last_scrub_stamp":"2024-09-10T19:30:17.531099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:30:17.531099+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:31:44.519993+0000","scrub_duration":0.097305297999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":462,"reported_epoch":418,"state":"active+clean","last_fresh":"2024-09-10T19:30:42.075540+0000","last_change":"2024-09-10T19:30:29.198545+0000","last_active":"2024-09-10T19:30:42.075540+0000","last_peered":"2024-09-10T19:30:42.075540+0000","last_clean":"2024-09-10T19:30:42.075540+0000","last_became_active":"2024-09-10T19:24:23.907653+0000","last_became_peered":"2024-09-10T19:24:23.907653+0000","last_unstale":"2024-09-10T19:30:42.075540+0000","last_undegraded":"2024-09-10T19:30:42.075540+0000","last_fullsized":"2024-09-10T19:30:42.075540+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:30:29.198451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:30:29.198451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:31:31.812451+0000","scrub_duration":0.0023167890000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":435,"reported_epoch":418,"state":"active+clean","last_fresh":"2024-09-10T19:30:42.070233+0000","last_change":"2024-09-10T19:30:41.905595+0000","last_active":"2024-09-10T19:30:42.070233+0000","last_peered":"2024-09-10T19:30:42.070233+0000","last_clean":"2024-09-10T19:30:42.070233+0000","last_became_active":"2024-09-10T19:24:24.907431+0000","last_became_peered":"2024-09-10T19:24:24.907431+0000","last_unstale":"2024-09-10T19:30:42.070233+0000","last_undegraded":"2024-09-10T19:30:42.070233+0000","last_fullsized":"2024-09-10T19:30:42.070233+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:30:41.905516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:30:41.905516+0000","last_clean_scrub_stamp":"2024-09-10T19:30:41.905516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:32:11.300289+0000","scrub_duration":0.002437481,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.2","version":"419'16573","reported_seq":16803,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-09-10T19:30:50.346206+0000","last_change":"2024-09-10T19:30:18.096594+0000","last_active":"2024-09-10T19:30:50.346206+0000","last_peered":"2024-09-10T19:30:50.346206+0000","last_clean":"2024-09-10T19:30:50.346206+0000","last_became_active":"2024-09-10T19:27:46.703515+0000","last_became_peered":"2024-09-10T19:27:46.703515+0000","last_unstale":"2024-09-10T19:30:50.346206+0000","last_undegraded":"2024-09-10T19:30:50.346206+0000","last_fullsized":"2024-09-10T19:30:50.346206+0000","mapping_epoch":371,"log_start":"409'12400","ondisk_log_start":"409'12400","created":371,"last_epoch_clean":372,"parent":"0.0","parent_split_bits":0,"last_scrub":"415'13895","last_scrub_stamp":"2024-09-10T19:30:18.096518+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:27:45.679464+0000","last_clean_scrub_stamp":"2024-09-10T19:30:18.096518+0000","objects_scrubbed":4109,"log_size":4173,"log_dups_size":0,"ondisk_log_size":4173,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:31:35.525537+0000","scrub_duration":1.59077157,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":89456640,"num_objects":1365,"num_object_clones":0,"num_object_copies":4095,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1365,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16573,"num_write_kb":574016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,7],"acting":[3,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"74'79","reported_seq":267,"reported_epoch":418,"state":"active+clean","last_fresh":"2024-09-10T19:30:42.075360+0000","last_change":"2024-09-10T19:30:28.196155+0000","last_active":"2024-09-10T19:30:42.075360+0000","last_peered":"2024-09-10T19:30:42.075360+0000","last_clean":"2024-09-10T19:30:42.075360+0000","last_became_active":"2024-09-10T19:27:53.004335+0000","last_became_peered":"2024-09-10T19:27:53.004335+0000","last_unstale":"2024-09-10T19:30:42.075360+0000","last_undegraded":"2024-09-10T19:30:42.075360+0000","last_fullsized":"2024-09-10T19:30:42.075360+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":4,"last_scrub":"74'79","last_scrub_stamp":"2024-09-10T19:30:28.196058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:08:00.699315+0000","last_clean_scrub_stamp":"2024-09-10T19:30:28.196058+0000","objects_scrubbed":1,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:31:45.244731+0000","scrub_duration":0.012192326,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":525376,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.d","version":"414'45","reported_seq":670,"reported_epoch":418,"state":"active+clean","last_fresh":"2024-09-10T19:30:42.076027+0000","last_change":"2024-09-10T19:30:26.584510+0000","last_active":"2024-09-10T19:30:42.076027+0000","last_peered":"2024-09-10T19:30:42.076027+0000","last_clean":"2024-09-10T19:30:42.076027+0000","last_became_active":"2024-09-10T19:27:53.012536+0000","last_became_peered":"2024-09-10T19:27:53.012536+0000","last_unstale":"2024-09-10T19:30:42.076027+0000","last_undegraded":"2024-09-10T19:30:42.076027+0000","last_fullsized":"2024-09-10T19:30:42.076027+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"414'45","last_scrub_stamp":"2024-09-10T19:30:26.584371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:30:26.584371+0000","objects_scrubbed":8,"log_size":45,"log_dups_size":0,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:31:48.878559+0000","scrub_duration":0.0072232800000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":71,"num_read_kb":61,"num_write":46,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":40,"num_bytes_recovered":550,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,6],"acting":[4,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":459,"reported_epoch":418,"state":"active+clean","last_fresh":"2024-09-10T19:30:42.084610+0000","last_change":"2024-09-10T19:30:27.008379+0000","last_active":"2024-09-10T19:30:42.084610+0000","last_peered":"2024-09-10T19:30:42.084610+0000","last_clean":"2024-09-10T19:30:42.084610+0000","last_became_active":"2024-09-10T19:10:07.691119+0000","last_became_peered":"2024-09-10T19:10:07.691119+0000","last_unstale":"2024-09-10T19:30:42.084610+0000","last_undegraded":"2024-09-10T19:30:42.084610+0000","last_fullsized":"2024-09-10T19:30:42.084610+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:30:27.008299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:16:39.716674+0000","last_clean_scrub_stamp":"2024-09-10T19:30:27.008299+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:31:49.390872+0000","scrub_duration":0.0014108809999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":180,"reported_epoch":418,"state":"active+clean","last_fresh":"2024-09-10T19:30:42.083718+0000","last_change":"2024-09-10T19:29:30.513955+0000","last_active":"2024-09-10T19:30:42.083718+0000","last_peered":"2024-09-10T19:30:42.083718+0000","last_clean":"2024-09-10T19:30:42.083718+0000","last_became_active":"2024-09-10T19:24:24.886479+0000","last_became_peered":"2024-09-10T19:24:24.886479+0000","last_unstale":"2024-09-10T19:30:42.083718+0000","last_undegraded":"2024-09-10T19:30:42.083718+0000","last_fullsized":"2024-09-10T19:30:42.083718+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:29:30.513872+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:29:30.513872+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0073816690000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":337,"reported_epoch":418,"state":"active+clean","last_fresh":"2024-09-10T19:30:42.077021+0000","last_change":"2024-09-10T19:29:10.112087+0000","last_active":"2024-09-10T19:30:42.077021+0000","last_peered":"2024-09-10T19:30:42.077021+0000","last_clean":"2024-09-10T19:30:42.077021+0000","last_became_active":"2024-09-10T19:27:53.026452+0000","last_became_peered":"2024-09-10T19:27:53.026452+0000","last_unstale":"2024-09-10T19:30:42.077021+0000","last_undegraded":"2024-09-10T19:30:42.077021+0000","last_fullsized":"2024-09-10T19:30:42.077021+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:29:10.111999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:29:10.111999+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0051523979999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.3","version":"419'16537","reported_seq":17038,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-09-10T19:30:49.900761+0000","last_change":"2024-09-10T19:30:41.261593+0000","last_active":"2024-09-10T19:30:49.900761+0000","last_peered":"2024-09-10T19:30:49.900761+0000","last_clean":"2024-09-10T19:30:49.900761+0000","last_became_active":"2024-09-10T19:27:53.010097+0000","last_became_peered":"2024-09-10T19:27:53.010097+0000","last_unstale":"2024-09-10T19:30:49.900761+0000","last_undegraded":"2024-09-10T19:30:49.900761+0000","last_fullsized":"2024-09-10T19:30:49.900761+0000","mapping_epoch":374,"log_start":"409'11951","ondisk_log_start":"409'11951","created":371,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"417'15813","last_scrub_stamp":"2024-09-10T19:30:41.261503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:27:45.679464+0000","last_clean_scrub_stamp":"2024-09-10T19:30:41.261503+0000","objects_scrubbed":2246,"log_size":4586,"log_dups_size":0,"ondisk_log_size":4586,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:31:54.086031+0000","scrub_duration":0.79268019199999995,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":97845248,"num_objects":1493,"num_object_clones":0,"num_object_copies":4479,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1493,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16537,"num_write_kb":576960,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1],"acting":[6,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.9","version":"74'79","reported_seq":524,"reported_epoch":418,"state":"active+clean","last_fresh":"2024-09-10T19:30:42.082086+0000","last_change":"2024-09-10T19:30:39.992958+0000","last_active":"2024-09-10T19:30:42.082086+0000","last_peered":"2024-09-10T19:30:42.082086+0000","last_clean":"2024-09-10T19:30:42.082086+0000","last_became_active":"2024-09-10T19:20:30.844108+0000","last_became_peered":"2024-09-10T19:20:30.844108+0000","last_unstale":"2024-09-10T19:30:42.082086+0000","last_undegraded":"2024-09-10T19:30:42.082086+0000","last_fullsized":"2024-09-10T19:30:42.082086+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":4,"last_scrub":"74'79","last_scrub_stamp":"2024-09-10T19:30:39.992877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:08:00.699315+0000","last_clean_scrub_stamp":"2024-09-10T19:30:39.992877+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:31:53.125566+0000","scrub_duration":0.0080860159999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.c","version":"414'30","reported_seq":548,"reported_epoch":418,"state":"active+clean","last_fresh":"2024-09-10T19:30:42.074851+0000","last_change":"2024-09-10T19:30:08.475937+0000","last_active":"2024-09-10T19:30:42.074851+0000","last_peered":"2024-09-10T19:30:42.074851+0000","last_clean":"2024-09-10T19:30:42.074851+0000","last_became_active":"2024-09-10T19:27:53.012617+0000","last_became_peered":"2024-09-10T19:27:53.012617+0000","last_unstale":"2024-09-10T19:30:42.074851+0000","last_undegraded":"2024-09-10T19:30:42.074851+0000","last_fullsized":"2024-09-10T19:30:42.074851+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"237'20","last_scrub_stamp":"2024-09-10T19:30:08.475832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:30:08.475832+0000","objects_scrubbed":5,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:31:23.767469+0000","scrub_duration":0.0094002249999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":45,"num_read_kb":40,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":156,"reported_epoch":418,"state":"active+clean","last_fresh":"2024-09-10T19:30:42.083407+0000","last_change":"2024-09-10T19:28:16.491452+0000","last_active":"2024-09-10T19:30:42.083407+0000","last_peered":"2024-09-10T19:30:42.083407+0000","last_clean":"2024-09-10T19:30:42.083407+0000","last_became_active":"2024-09-10T19:24:01.973848+0000","last_became_peered":"2024-09-10T19:24:01.973848+0000","last_unstale":"2024-09-10T19:30:42.083407+0000","last_undegraded":"2024-09-10T19:30:42.083407+0000","last_fullsized":"2024-09-10T19:30:42.083407+0000","mapping_epoch":292,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":293,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:28:16.491371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:28:16.491371+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.014455551000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.d","version":"303'12","reported_seq":579,"reported_epoch":418,"state":"active+clean","last_fresh":"2024-09-10T19:30:45.976285+0000","last_change":"2024-09-10T19:29:15.947794+0000","last_active":"2024-09-10T19:30:45.976285+0000","last_peered":"2024-09-10T19:30:45.976285+0000","last_clean":"2024-09-10T19:30:45.976285+0000","last_became_active":"2024-09-10T19:24:23.906819+0000","last_became_peered":"2024-09-10T19:24:23.906819+0000","last_unstale":"2024-09-10T19:30:45.976285+0000","last_undegraded":"2024-09-10T19:30:45.976285+0000","last_fullsized":"2024-09-10T19:30:45.976285+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"303'12","last_scrub_stamp":"2024-09-10T19:29:15.947693+0000","last_deep_scrub":"80'9","last_deep_scrub_stamp":"2024-09-10T19:11:25.156465+0000","last_clean_scrub_stamp":"2024-09-10T19:29:15.947693+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.012440023,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,0],"acting":[7,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":485,"reported_epoch":418,"state":"active+clean","last_fresh":"2024-09-10T19:30:42.075830+0000","last_change":"2024-09-10T19:30:34.534444+0000","last_active":"2024-09-10T19:30:42.075830+0000","last_peered":"2024-09-10T19:30:42.075830+0000","last_clean":"2024-09-10T19:30:42.075830+0000","last_became_active":"2024-09-10T19:24:01.972593+0000","last_became_peered":"2024-09-10T19:24:01.972593+0000","last_unstale":"2024-09-10T19:30:42.075830+0000","last_undegraded":"2024-09-10T19:30:42.075830+0000","last_fullsized":"2024-09-10T19:30:42.075830+0000","mapping_epoch":292,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":293,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:30:34.534128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:30:34.534128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:31:59.004309+0000","scrub_duration":0.029869611000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,5],"acting":[4,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.0","version":"419'16837","reported_seq":17083,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-09-10T19:30:52.552501+0000","last_change":"2024-09-10T19:30:15.362614+0000","last_active":"2024-09-10T19:30:52.552501+0000","last_peered":"2024-09-10T19:30:52.552501+0000","last_clean":"2024-09-10T19:30:52.552501+0000","last_became_active":"2024-09-10T19:27:46.705538+0000","last_became_peered":"2024-09-10T19:27:46.705538+0000","last_unstale":"2024-09-10T19:30:52.552501+0000","last_undegraded":"2024-09-10T19:30:52.552501+0000","last_fullsized":"2024-09-10T19:30:52.552501+0000","mapping_epoch":371,"log_start":"409'12400","ondisk_log_start":"409'12400","created":371,"last_epoch_clean":372,"parent":"0.0","parent_split_bits":0,"last_scrub":"414'13664","last_scrub_stamp":"2024-09-10T19:30:15.362539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:27:45.679464+0000","last_clean_scrub_stamp":"2024-09-10T19:30:15.362539+0000","objects_scrubbed":4480,"log_size":4437,"log_dups_size":0,"ondisk_log_size":4437,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:31:16.651953+0000","scrub_duration":1.8958626860000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":80936960,"num_objects":1235,"num_object_clones":0,"num_object_copies":3705,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1235,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16837,"num_write_kb":578304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"74'79","reported_seq":244,"reported_epoch":418,"state":"active+clean","last_fresh":"2024-09-10T19:30:42.075425+0000","last_change":"2024-09-10T19:30:04.397199+0000","last_active":"2024-09-10T19:30:42.075425+0000","last_peered":"2024-09-10T19:30:42.075425+0000","last_clean":"2024-09-10T19:30:42.075425+0000","last_became_active":"2024-09-10T19:27:53.004646+0000","last_became_peered":"2024-09-10T19:27:53.004646+0000","last_unstale":"2024-09-10T19:30:42.075425+0000","last_undegraded":"2024-09-10T19:30:42.075425+0000","last_fullsized":"2024-09-10T19:30:42.075425+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":4,"last_scrub":"74'79","last_scrub_stamp":"2024-09-10T19:30:04.397114+0000","last_deep_scrub":"74'79","last_deep_scrub_stamp":"2024-09-10T19:21:53.326433+0000","last_clean_scrub_stamp":"2024-09-10T19:30:04.397114+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:31:34.304105+0000","scrub_duration":0.0040705209999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,4],"acting":[5,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.f","version":"414'39","reported_seq":190,"reported_epoch":418,"state":"active+clean","last_fresh":"2024-09-10T19:30:42.084059+0000","last_change":"2024-09-10T19:28:26.434825+0000","last_active":"2024-09-10T19:30:42.084059+0000","last_peered":"2024-09-10T19:30:42.084059+0000","last_clean":"2024-09-10T19:30:42.084059+0000","last_became_active":"2024-09-10T19:24:01.979187+0000","last_became_peered":"2024-09-10T19:24:01.979187+0000","last_unstale":"2024-09-10T19:30:42.084059+0000","last_undegraded":"2024-09-10T19:30:42.084059+0000","last_fullsized":"2024-09-10T19:30:42.084059+0000","mapping_epoch":292,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":293,"parent":"0.0","parent_split_bits":0,"last_scrub":"237'27","last_scrub_stamp":"2024-09-10T19:28:26.434563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:28:26.434563+0000","objects_scrubbed":7,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0096637310000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":61,"num_read_kb":52,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":156,"reported_epoch":418,"state":"active+clean","last_fresh":"2024-09-10T19:30:42.083178+0000","last_change":"2024-09-10T19:28:24.420346+0000","last_active":"2024-09-10T19:30:42.083178+0000","last_peered":"2024-09-10T19:30:42.083178+0000","last_clean":"2024-09-10T19:30:42.083178+0000","last_became_active":"2024-09-10T19:24:01.974773+0000","last_became_peered":"2024-09-10T19:24:01.974773+0000","last_unstale":"2024-09-10T19:30:42.083178+0000","last_undegraded":"2024-09-10T19:30:42.083178+0000","last_fullsized":"2024-09-10T19:30:42.083178+0000","mapping_epoch":292,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":293,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:28:24.420270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:28:24.420270+0000","last_clean_scrub_stamp":"2024-09-10T19:28:24.420270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.016717709000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.e","version":"303'10","reported_seq":702,"reported_epoch":418,"state":"active+clean","last_fresh":"2024-09-10T19:30:45.976302+0000","last_change":"2024-09-10T19:30:07.517646+0000","last_active":"2024-09-10T19:30:45.976302+0000","last_peered":"2024-09-10T19:30:45.976302+0000","last_clean":"2024-09-10T19:30:45.976302+0000","last_became_active":"2024-09-10T19:24:23.908760+0000","last_became_peered":"2024-09-10T19:24:23.908760+0000","last_unstale":"2024-09-10T19:30:45.976302+0000","last_undegraded":"2024-09-10T19:30:45.976302+0000","last_fullsized":"2024-09-10T19:30:45.976302+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"303'10","last_scrub_stamp":"2024-09-10T19:30:07.517566+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:30:07.517566+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:31:36.012505+0000","scrub_duration":0.0052487269999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":481,"reported_epoch":418,"state":"active+clean","last_fresh":"2024-09-10T19:30:42.076651+0000","last_change":"2024-09-10T19:29:15.298967+0000","last_active":"2024-09-10T19:30:42.076651+0000","last_peered":"2024-09-10T19:30:42.076651+0000","last_clean":"2024-09-10T19:30:42.076651+0000","last_became_active":"2024-09-10T19:24:24.917109+0000","last_became_peered":"2024-09-10T19:24:24.917109+0000","last_unstale":"2024-09-10T19:30:42.076651+0000","last_undegraded":"2024-09-10T19:30:42.076651+0000","last_fullsized":"2024-09-10T19:30:42.076651+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:29:15.298879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:29:15.298879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.003007774,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.1","version":"419'16550","reported_seq":17027,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-09-10T19:30:49.908612+0000","last_change":"2024-09-10T19:30:42.079899+0000","last_active":"2024-09-10T19:30:49.908612+0000","last_peered":"2024-09-10T19:30:49.908612+0000","last_clean":"2024-09-10T19:30:49.908612+0000","last_became_active":"2024-09-10T19:27:53.016565+0000","last_became_peered":"2024-09-10T19:27:53.016565+0000","last_unstale":"2024-09-10T19:30:49.908612+0000","last_undegraded":"2024-09-10T19:30:49.908612+0000","last_fullsized":"2024-09-10T19:30:49.908612+0000","mapping_epoch":374,"log_start":"409'11923","ondisk_log_start":"409'11923","created":371,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"401'9457","last_scrub_stamp":"2024-09-10T19:29:29.986778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:27:45.679464+0000","last_clean_scrub_stamp":"2024-09-10T19:29:29.986778+0000","objects_scrubbed":1549,"log_size":4627,"log_dups_size":0,"ondisk_log_size":4627,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"queued for scrub","scrub_duration":3.5527387689999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":91488256,"num_objects":1396,"num_object_clones":0,"num_object_copies":4188,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1396,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16550,"num_write_kb":574272,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.e","version":"414'32","reported_seq":614,"reported_epoch":418,"state":"active+clean","last_fresh":"2024-09-10T19:30:42.076511+0000","last_change":"2024-09-10T19:30:18.472016+0000","last_active":"2024-09-10T19:30:42.076511+0000","last_peered":"2024-09-10T19:30:42.076511+0000","last_clean":"2024-09-10T19:30:42.076511+0000","last_became_active":"2024-09-10T19:27:53.012101+0000","last_became_peered":"2024-09-10T19:27:53.012101+0000","last_unstale":"2024-09-10T19:30:42.076511+0000","last_undegraded":"2024-09-10T19:30:42.076511+0000","last_fullsized":"2024-09-10T19:30:42.076511+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"414'32","last_scrub_stamp":"2024-09-10T19:30:18.471961+0000","last_deep_scrub":"84'16","last_deep_scrub_stamp":"2024-09-10T19:11:56.157975+0000","last_clean_scrub_stamp":"2024-09-10T19:30:18.471961+0000","objects_scrubbed":5,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:31:40.658421+0000","scrub_duration":0.001699491,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":37,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":550,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":130,"reported_epoch":418,"state":"active+clean","last_fresh":"2024-09-10T19:30:42.073439+0000","last_change":"2024-09-10T19:29:30.393193+0000","last_active":"2024-09-10T19:30:42.073439+0000","last_peered":"2024-09-10T19:30:42.073439+0000","last_clean":"2024-09-10T19:30:42.073439+0000","last_became_active":"2024-09-10T19:24:53.756220+0000","last_became_peered":"2024-09-10T19:24:53.756220+0000","last_unstale":"2024-09-10T19:30:42.073439+0000","last_undegraded":"2024-09-10T19:30:42.073439+0000","last_fullsized":"2024-09-10T19:30:42.073439+0000","mapping_epoch":318,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":319,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:29:30.393109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:24:13.636249+0000","last_clean_scrub_stamp":"2024-09-10T19:29:30.393109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:30:50.348247+0000","scrub_duration":0.0027000890000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,3],"acting":[4,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":455,"reported_epoch":418,"state":"active+clean","last_fresh":"2024-09-10T19:30:42.076349+0000","last_change":"2024-09-10T19:30:15.118053+0000","last_active":"2024-09-10T19:30:42.076349+0000","last_peered":"2024-09-10T19:30:42.076349+0000","last_clean":"2024-09-10T19:30:42.076349+0000","last_became_active":"2024-09-10T19:27:53.007094+0000","last_became_peered":"2024-09-10T19:27:53.007094+0000","last_unstale":"2024-09-10T19:30:42.076349+0000","last_undegraded":"2024-09-10T19:30:42.076349+0000","last_fullsized":"2024-09-10T19:30:42.076349+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:30:15.117979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:30:15.117979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:31:33.405690+0000","scrub_duration":0.0051218720000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":453,"reported_epoch":418,"state":"active+clean","last_fresh":"2024-09-10T19:30:42.077766+0000","last_change":"2024-09-10T19:30:29.757452+0000","last_active":"2024-09-10T19:30:42.077766+0000","last_peered":"2024-09-10T19:30:42.077766+0000","last_clean":"2024-09-10T19:30:42.077766+0000","last_became_active":"2024-09-10T19:27:53.025345+0000","last_became_peered":"2024-09-10T19:27:53.025345+0000","last_unstale":"2024-09-10T19:30:42.077766+0000","last_undegraded":"2024-09-10T19:30:42.077766+0000","last_fullsized":"2024-09-10T19:30:42.077766+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:30:29.757372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:30:29.757372+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:31:37.286874+0000","scrub_duration":0.0047515539999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.e","version":"419'16676","reported_seq":17072,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-09-10T19:30:50.305963+0000","last_change":"2024-09-10T19:29:34.726363+0000","last_active":"2024-09-10T19:30:50.305963+0000","last_peered":"2024-09-10T19:30:50.305963+0000","last_clean":"2024-09-10T19:30:50.305963+0000","last_became_active":"2024-09-10T19:27:46.704103+0000","last_became_peered":"2024-09-10T19:27:46.704103+0000","last_unstale":"2024-09-10T19:30:50.305963+0000","last_undegraded":"2024-09-10T19:30:50.305963+0000","last_fullsized":"2024-09-10T19:30:50.305963+0000","mapping_epoch":371,"log_start":"409'12000","ondisk_log_start":"409'12000","created":371,"last_epoch_clean":372,"parent":"0.0","parent_split_bits":0,"last_scrub":"404'9978","last_scrub_stamp":"2024-09-10T19:29:34.726102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:27:45.679464+0000","last_clean_scrub_stamp":"2024-09-10T19:29:34.726102+0000","objects_scrubbed":8262,"log_size":4676,"log_dups_size":0,"ondisk_log_size":4676,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:30:55.165671+0000","scrub_duration":3.3425435960000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":93716480,"num_objects":1430,"num_object_clones":0,"num_object_copies":4290,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1430,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16676,"num_write_kb":579392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,7],"acting":[4,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.4","version":"317'80","reported_seq":230,"reported_epoch":418,"state":"active+clean","last_fresh":"2024-09-10T19:30:42.076009+0000","last_change":"2024-09-10T19:29:14.237503+0000","last_active":"2024-09-10T19:30:42.076009+0000","last_peered":"2024-09-10T19:30:42.076009+0000","last_clean":"2024-09-10T19:30:42.076009+0000","last_became_active":"2024-09-10T19:20:30.838441+0000","last_became_peered":"2024-09-10T19:20:30.838441+0000","last_unstale":"2024-09-10T19:30:42.076009+0000","last_undegraded":"2024-09-10T19:30:42.076009+0000","last_fullsized":"2024-09-10T19:30:42.076009+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":3,"last_scrub":"317'80","last_scrub_stamp":"2024-09-10T19:29:14.237292+0000","last_deep_scrub":"317'80","last_deep_scrub_stamp":"2024-09-10T19:24:51.540456+0000","last_clean_scrub_stamp":"2024-09-10T19:29:14.237292+0000","objects_scrubbed":1,"log_size":80,"log_dups_size":0,"ondisk_log_size":80,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0022248250000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"414'49","reported_seq":454,"reported_epoch":418,"state":"active+clean","last_fresh":"2024-09-10T19:30:42.073534+0000","last_change":"2024-09-10T19:30:19.518616+0000","last_active":"2024-09-10T19:30:42.073534+0000","last_peered":"2024-09-10T19:30:42.073534+0000","last_clean":"2024-09-10T19:30:42.073534+0000","last_became_active":"2024-09-10T19:27:53.011799+0000","last_became_peered":"2024-09-10T19:27:53.011799+0000","last_unstale":"2024-09-10T19:30:42.073534+0000","last_undegraded":"2024-09-10T19:30:42.073534+0000","last_fullsized":"2024-09-10T19:30:42.073534+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"414'49","last_scrub_stamp":"2024-09-10T19:30:19.518533+0000","last_deep_scrub":"265'35","last_deep_scrub_stamp":"2024-09-10T19:28:24.824459+0000","last_clean_scrub_stamp":"2024-09-10T19:30:19.518533+0000","objects_scrubbed":7,"log_size":49,"log_dups_size":0,"ondisk_log_size":49,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:31:45.050810+0000","scrub_duration":0.0080552469999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":63,"num_read_kb":56,"num_write":42,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":64,"reported_epoch":418,"state":"active+clean","last_fresh":"2024-09-10T19:30:42.079901+0000","last_change":"2024-09-10T19:29:14.668212+0000","last_active":"2024-09-10T19:30:42.079901+0000","last_peered":"2024-09-10T19:30:42.079901+0000","last_clean":"2024-09-10T19:30:42.079901+0000","last_became_active":"2024-09-10T19:27:53.020498+0000","last_became_peered":"2024-09-10T19:27:53.020498+0000","last_unstale":"2024-09-10T19:30:42.079901+0000","last_undegraded":"2024-09-10T19:30:42.079901+0000","last_fullsized":"2024-09-10T19:30:42.079901+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:29:14.668130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:29:14.668130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0062469530000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.0","version":"417'13","reported_seq":736,"reported_epoch":418,"state":"active+clean","last_fresh":"2024-09-10T19:30:45.976856+0000","last_change":"2024-09-10T19:30:39.965009+0000","last_active":"2024-09-10T19:30:45.976856+0000","last_peered":"2024-09-10T19:30:45.976856+0000","last_clean":"2024-09-10T19:30:45.976856+0000","last_became_active":"2024-09-10T19:24:01.972645+0000","last_became_peered":"2024-09-10T19:24:01.972645+0000","last_unstale":"2024-09-10T19:30:45.976856+0000","last_undegraded":"2024-09-10T19:30:45.976856+0000","last_fullsized":"2024-09-10T19:30:45.976856+0000","mapping_epoch":292,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":293,"parent":"0.0","parent_split_bits":0,"last_scrub":"417'13","last_scrub_stamp":"2024-09-10T19:30:39.964914+0000","last_deep_scrub":"417'13","last_deep_scrub_stamp":"2024-09-10T19:30:39.964914+0000","last_clean_scrub_stamp":"2024-09-10T19:30:39.964914+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:31:49.349883+0000","scrub_duration":0.0087623219999999995,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":480,"reported_epoch":418,"state":"active+clean","last_fresh":"2024-09-10T19:30:42.084257+0000","last_change":"2024-09-10T19:30:14.879292+0000","last_active":"2024-09-10T19:30:42.084257+0000","last_peered":"2024-09-10T19:30:42.084257+0000","last_clean":"2024-09-10T19:30:42.084257+0000","last_became_active":"2024-09-10T19:27:53.009392+0000","last_became_peered":"2024-09-10T19:27:53.009392+0000","last_unstale":"2024-09-10T19:30:42.084257+0000","last_undegraded":"2024-09-10T19:30:42.084257+0000","last_fullsized":"2024-09-10T19:30:42.084257+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:30:14.879002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:28:25.822918+0000","last_clean_scrub_stamp":"2024-09-10T19:30:14.879002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:31:25.289697+0000","scrub_duration":0.011117784,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.f","version":"419'16622","reported_seq":16891,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-09-10T19:30:50.512687+0000","last_change":"2024-09-10T19:30:08.691714+0000","last_active":"2024-09-10T19:30:50.512687+0000","last_peered":"2024-09-10T19:30:50.512687+0000","last_clean":"2024-09-10T19:30:50.512687+0000","last_became_active":"2024-09-10T19:27:46.707637+0000","last_became_peered":"2024-09-10T19:27:46.707637+0000","last_unstale":"2024-09-10T19:30:50.512687+0000","last_undegraded":"2024-09-10T19:30:50.512687+0000","last_fullsized":"2024-09-10T19:30:50.512687+0000","mapping_epoch":371,"log_start":"409'12300","ondisk_log_start":"409'12300","created":371,"last_epoch_clean":372,"parent":"0.0","parent_split_bits":0,"last_scrub":"413'13046","last_scrub_stamp":"2024-09-10T19:30:08.691638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:27:45.679464+0000","last_clean_scrub_stamp":"2024-09-10T19:30:08.691638+0000","objects_scrubbed":5075,"log_size":4322,"log_dups_size":0,"ondisk_log_size":4322,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:31:16.854516+0000","scrub_duration":1.8581280769999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92405760,"num_objects":1410,"num_object_clones":0,"num_object_copies":4230,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1410,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16622,"num_write_kb":577024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.5","version":"74'79","reported_seq":254,"reported_epoch":418,"state":"active+clean","last_fresh":"2024-09-10T19:30:42.074560+0000","last_change":"2024-09-10T19:30:10.470076+0000","last_active":"2024-09-10T19:30:42.074560+0000","last_peered":"2024-09-10T19:30:42.074560+0000","last_clean":"2024-09-10T19:30:42.074560+0000","last_became_active":"2024-09-10T19:27:53.013021+0000","last_became_peered":"2024-09-10T19:27:53.013021+0000","last_unstale":"2024-09-10T19:30:42.074560+0000","last_undegraded":"2024-09-10T19:30:42.074560+0000","last_fullsized":"2024-09-10T19:30:42.074560+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":3,"last_scrub":"74'79","last_scrub_stamp":"2024-09-10T19:30:10.470024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:08:00.699315+0000","last_clean_scrub_stamp":"2024-09-10T19:30:10.470024+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:31:19.660974+0000","scrub_duration":0.0083031130000000009,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"414'42","reported_seq":512,"reported_epoch":418,"state":"active+clean","last_fresh":"2024-09-10T19:30:42.077134+0000","last_change":"2024-09-10T19:29:35.143247+0000","last_active":"2024-09-10T19:30:42.077134+0000","last_peered":"2024-09-10T19:30:42.077134+0000","last_clean":"2024-09-10T19:30:42.077134+0000","last_became_active":"2024-09-10T19:24:23.903784+0000","last_became_peered":"2024-09-10T19:24:23.903784+0000","last_unstale":"2024-09-10T19:30:42.077134+0000","last_undegraded":"2024-09-10T19:30:42.077134+0000","last_fullsized":"2024-09-10T19:30:42.077134+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"237'30","last_scrub_stamp":"2024-09-10T19:29:35.143157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:29:35.143157+0000","objects_scrubbed":8,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:30:58.866237+0000","scrub_duration":0.016979355000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":70,"num_read_kb":58,"num_write":44,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":186,"reported_epoch":418,"state":"active+clean","last_fresh":"2024-09-10T19:30:42.087100+0000","last_change":"2024-09-10T19:30:19.275603+0000","last_active":"2024-09-10T19:30:42.087100+0000","last_peered":"2024-09-10T19:30:42.087100+0000","last_clean":"2024-09-10T19:30:42.087100+0000","last_became_active":"2024-09-10T19:27:53.008302+0000","last_became_peered":"2024-09-10T19:27:53.008302+0000","last_unstale":"2024-09-10T19:30:42.087100+0000","last_undegraded":"2024-09-10T19:30:42.087100+0000","last_fullsized":"2024-09-10T19:30:42.087100+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:30:19.275465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:28:15.534211+0000","last_clean_scrub_stamp":"2024-09-10T19:30:19.275465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:31:42.840763+0000","scrub_duration":0.0057158039999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":444,"reported_epoch":418,"state":"active+clean","last_fresh":"2024-09-10T19:30:42.073718+0000","last_change":"2024-09-10T19:29:23.262835+0000","last_active":"2024-09-10T19:30:42.073718+0000","last_peered":"2024-09-10T19:30:42.073718+0000","last_clean":"2024-09-10T19:30:42.073718+0000","last_became_active":"2024-09-10T19:10:11.698054+0000","last_became_peered":"2024-09-10T19:10:11.698054+0000","last_unstale":"2024-09-10T19:30:42.073718+0000","last_undegraded":"2024-09-10T19:30:42.073718+0000","last_fullsized":"2024-09-10T19:30:42.073718+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:29:23.262748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:26:24.897519+0000","last_clean_scrub_stamp":"2024-09-10T19:29:23.262748+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0056390679999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":472,"reported_epoch":418,"state":"active+clean","last_fresh":"2024-09-10T19:30:42.074308+0000","last_change":"2024-09-10T19:29:27.360561+0000","last_active":"2024-09-10T19:30:42.074308+0000","last_peered":"2024-09-10T19:30:42.074308+0000","last_clean":"2024-09-10T19:30:42.074308+0000","last_became_active":"2024-09-10T19:27:53.017161+0000","last_became_peered":"2024-09-10T19:27:53.017161+0000","last_unstale":"2024-09-10T19:30:42.074308+0000","last_undegraded":"2024-09-10T19:30:42.074308+0000","last_fullsized":"2024-09-10T19:30:42.074308+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:29:27.360479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:29:27.360479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0086931189999999992,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.c","version":"419'16766","reported_seq":17032,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-09-10T19:30:52.465283+0000","last_change":"2024-09-10T19:30:09.592766+0000","last_active":"2024-09-10T19:30:52.465283+0000","last_peered":"2024-09-10T19:30:52.465283+0000","last_clean":"2024-09-10T19:30:52.465283+0000","last_became_active":"2024-09-10T19:27:46.705563+0000","last_became_peered":"2024-09-10T19:27:46.705563+0000","last_unstale":"2024-09-10T19:30:52.465283+0000","last_undegraded":"2024-09-10T19:30:52.465283+0000","last_fullsized":"2024-09-10T19:30:52.465283+0000","mapping_epoch":371,"log_start":"411'12400","ondisk_log_start":"411'12400","created":371,"last_epoch_clean":372,"parent":"0.0","parent_split_bits":0,"last_scrub":"413'13084","last_scrub_stamp":"2024-09-10T19:30:09.592622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:27:45.679464+0000","last_clean_scrub_stamp":"2024-09-10T19:30:09.592622+0000","objects_scrubbed":4998,"log_size":4366,"log_dups_size":0,"ondisk_log_size":4366,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:31:23.730993+0000","scrub_duration":2.154546888,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":79298560,"num_objects":1210,"num_object_clones":0,"num_object_copies":3630,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1210,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16766,"num_write_kb":575232,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,7],"acting":[1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"74'79","reported_seq":74,"reported_epoch":418,"state":"active+clean","last_fresh":"2024-09-10T19:30:42.079285+0000","last_change":"2024-09-10T19:30:23.520229+0000","last_active":"2024-09-10T19:30:42.079285+0000","last_peered":"2024-09-10T19:30:42.079285+0000","last_clean":"2024-09-10T19:30:42.079285+0000","last_became_active":"2024-09-10T19:27:53.017899+0000","last_became_peered":"2024-09-10T19:27:53.017899+0000","last_unstale":"2024-09-10T19:30:42.079285+0000","last_undegraded":"2024-09-10T19:30:42.079285+0000","last_fullsized":"2024-09-10T19:30:42.079285+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":3,"last_scrub":"74'79","last_scrub_stamp":"2024-09-10T19:30:23.520135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:08:00.699315+0000","last_clean_scrub_stamp":"2024-09-10T19:30:23.520135+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:31:47.744700+0000","scrub_duration":0.0041853660000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"414'47","reported_seq":469,"reported_epoch":418,"state":"active+clean","last_fresh":"2024-09-10T19:30:42.072611+0000","last_change":"2024-09-10T19:30:16.465094+0000","last_active":"2024-09-10T19:30:42.072611+0000","last_peered":"2024-09-10T19:30:42.072611+0000","last_clean":"2024-09-10T19:30:42.072611+0000","last_became_active":"2024-09-10T19:24:23.901836+0000","last_became_peered":"2024-09-10T19:24:23.901836+0000","last_unstale":"2024-09-10T19:30:42.072611+0000","last_undegraded":"2024-09-10T19:30:42.072611+0000","last_fullsized":"2024-09-10T19:30:42.072611+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"414'43","last_scrub_stamp":"2024-09-10T19:30:16.464894+0000","last_deep_scrub":"104'27","last_deep_scrub_stamp":"2024-09-10T19:13:02.225944+0000","last_clean_scrub_stamp":"2024-09-10T19:30:16.464894+0000","objects_scrubbed":8,"log_size":47,"log_dups_size":0,"ondisk_log_size":47,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:31:35.284993+0000","scrub_duration":0.0076287560000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":68,"num_read_kb":54,"num_write":42,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":466,"reported_epoch":418,"state":"active+clean","last_fresh":"2024-09-10T19:30:42.087562+0000","last_change":"2024-09-10T19:30:32.016942+0000","last_active":"2024-09-10T19:30:42.087562+0000","last_peered":"2024-09-10T19:30:42.087562+0000","last_clean":"2024-09-10T19:30:42.087562+0000","last_became_active":"2024-09-10T19:24:23.899556+0000","last_became_peered":"2024-09-10T19:24:23.899556+0000","last_unstale":"2024-09-10T19:30:42.087562+0000","last_undegraded":"2024-09-10T19:30:42.087562+0000","last_fullsized":"2024-09-10T19:30:42.087562+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:30:32.016858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:16:37.779967+0000","last_clean_scrub_stamp":"2024-09-10T19:30:32.016858+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:31:37.792761+0000","scrub_duration":0.0083493209999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":74,"reported_epoch":418,"state":"active+clean","last_fresh":"2024-09-10T19:30:42.078638+0000","last_change":"2024-09-10T19:30:19.498917+0000","last_active":"2024-09-10T19:30:42.078638+0000","last_peered":"2024-09-10T19:30:42.078638+0000","last_clean":"2024-09-10T19:30:42.078638+0000","last_became_active":"2024-09-10T19:27:53.003734+0000","last_became_peered":"2024-09-10T19:27:53.003734+0000","last_unstale":"2024-09-10T19:30:42.078638+0000","last_undegraded":"2024-09-10T19:30:42.078638+0000","last_fullsized":"2024-09-10T19:30:42.078638+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:30:19.498837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:30:19.498837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:31:47.588619+0000","scrub_duration":0.004490628,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":186,"reported_epoch":418,"state":"active+clean","last_fresh":"2024-09-10T19:30:42.083248+0000","last_change":"2024-09-10T19:29:22.538192+0000","last_active":"2024-09-10T19:30:42.083248+0000","last_peered":"2024-09-10T19:30:42.083248+0000","last_clean":"2024-09-10T19:30:42.083248+0000","last_became_active":"2024-09-10T19:27:53.016905+0000","last_became_peered":"2024-09-10T19:27:53.016905+0000","last_unstale":"2024-09-10T19:30:42.083248+0000","last_undegraded":"2024-09-10T19:30:42.083248+0000","last_fullsized":"2024-09-10T19:30:42.083248+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:29:22.538092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:29:22.538092+0000","last_clean_scrub_stamp":"2024-09-10T19:29:22.538092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:30:48.324749+0000","scrub_duration":0.0051848659999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,7],"acting":[1,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.d","version":"419'16744","reported_seq":17121,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-09-10T19:30:50.296411+0000","last_change":"2024-09-10T19:30:06.229050+0000","last_active":"2024-09-10T19:30:50.296411+0000","last_peered":"2024-09-10T19:30:50.296411+0000","last_clean":"2024-09-10T19:30:50.296411+0000","last_became_active":"2024-09-10T19:27:53.011209+0000","last_became_peered":"2024-09-10T19:27:53.011209+0000","last_unstale":"2024-09-10T19:30:50.296411+0000","last_undegraded":"2024-09-10T19:30:50.296411+0000","last_fullsized":"2024-09-10T19:30:50.296411+0000","mapping_epoch":374,"log_start":"409'12100","ondisk_log_start":"409'12100","created":371,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"413'12851","last_scrub_stamp":"2024-09-10T19:30:06.228898+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:27:45.679464+0000","last_clean_scrub_stamp":"2024-09-10T19:30:06.228898+0000","objects_scrubbed":5320,"log_size":4644,"log_dups_size":0,"ondisk_log_size":4644,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:31:22.959068+0000","scrub_duration":1.7693102619999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":88670232,"num_objects":1354,"num_object_clones":0,"num_object_copies":4062,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1354,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":16744,"num_write_kb":579073,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.7","version":"74'79","reported_seq":74,"reported_epoch":418,"state":"active+clean","last_fresh":"2024-09-10T19:30:42.079833+0000","last_change":"2024-09-10T19:30:09.567679+0000","last_active":"2024-09-10T19:30:42.079833+0000","last_peered":"2024-09-10T19:30:42.079833+0000","last_clean":"2024-09-10T19:30:42.079833+0000","last_became_active":"2024-09-10T19:27:53.020085+0000","last_became_peered":"2024-09-10T19:27:53.020085+0000","last_unstale":"2024-09-10T19:30:42.079833+0000","last_undegraded":"2024-09-10T19:30:42.079833+0000","last_fullsized":"2024-09-10T19:30:42.079833+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":3,"last_scrub":"74'79","last_scrub_stamp":"2024-09-10T19:30:09.567596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:08:00.699315+0000","last_clean_scrub_stamp":"2024-09-10T19:30:09.567596+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:31:38.788437+0000","scrub_duration":0.0064510310000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.2","version":"414'35","reported_seq":198,"reported_epoch":418,"state":"active+clean","last_fresh":"2024-09-10T19:30:42.083611+0000","last_change":"2024-09-10T19:29:06.293570+0000","last_active":"2024-09-10T19:30:42.083611+0000","last_peered":"2024-09-10T19:30:42.083611+0000","last_clean":"2024-09-10T19:30:42.083611+0000","last_became_active":"2024-09-10T19:24:01.976841+0000","last_became_peered":"2024-09-10T19:24:01.976841+0000","last_unstale":"2024-09-10T19:30:42.083611+0000","last_undegraded":"2024-09-10T19:30:42.083611+0000","last_fullsized":"2024-09-10T19:30:42.083611+0000","mapping_epoch":292,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":293,"parent":"0.0","parent_split_bits":0,"last_scrub":"350'25","last_scrub_stamp":"2024-09-10T19:29:06.293356+0000","last_deep_scrub":"350'25","last_deep_scrub_stamp":"2024-09-10T19:26:37.415039+0000","last_clean_scrub_stamp":"2024-09-10T19:29:06.293356+0000","objects_scrubbed":5,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0079160140000000007,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":38,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":180,"reported_epoch":418,"state":"active+clean","last_fresh":"2024-09-10T19:30:42.083782+0000","last_change":"2024-09-10T19:29:20.468976+0000","last_active":"2024-09-10T19:30:42.083782+0000","last_peered":"2024-09-10T19:30:42.083782+0000","last_clean":"2024-09-10T19:30:42.083782+0000","last_became_active":"2024-09-10T19:24:24.910173+0000","last_became_peered":"2024-09-10T19:24:24.910173+0000","last_unstale":"2024-09-10T19:30:42.083782+0000","last_undegraded":"2024-09-10T19:30:42.083782+0000","last_fullsized":"2024-09-10T19:30:42.083782+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:29:20.468827+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:29:20.468827+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0064360270000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"376'13","reported_seq":648,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-09-10T19:30:46.115966+0000","last_change":"2024-09-10T19:30:14.496516+0000","last_active":"2024-09-10T19:30:46.115966+0000","last_peered":"2024-09-10T19:30:46.115966+0000","last_clean":"2024-09-10T19:30:46.115966+0000","last_became_active":"2024-09-10T19:27:53.000311+0000","last_became_peered":"2024-09-10T19:27:53.000311+0000","last_unstale":"2024-09-10T19:30:46.115966+0000","last_undegraded":"2024-09-10T19:30:46.115966+0000","last_fullsized":"2024-09-10T19:30:46.115966+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"376'13","last_scrub_stamp":"2024-09-10T19:30:14.496433+0000","last_deep_scrub":"376'13","last_deep_scrub_stamp":"2024-09-10T19:28:17.490297+0000","last_clean_scrub_stamp":"2024-09-10T19:30:14.496433+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:31:34.498533+0000","scrub_duration":0.0060631859999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":419,"reported_epoch":418,"state":"active+clean","last_fresh":"2024-09-10T19:30:42.070412+0000","last_change":"2024-09-10T19:30:37.927650+0000","last_active":"2024-09-10T19:30:42.070412+0000","last_peered":"2024-09-10T19:30:42.070412+0000","last_clean":"2024-09-10T19:30:42.070412+0000","last_became_active":"2024-09-10T19:24:23.905977+0000","last_became_peered":"2024-09-10T19:24:23.905977+0000","last_unstale":"2024-09-10T19:30:42.070412+0000","last_undegraded":"2024-09-10T19:30:42.070412+0000","last_fullsized":"2024-09-10T19:30:42.070412+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:30:37.927553+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:30:37.927553+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:31:49.883711+0000","scrub_duration":0.013815849,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,5],"acting":[0,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.b","version":"419'16730","reported_seq":17009,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-09-10T19:30:50.360430+0000","last_change":"2024-09-10T19:30:07.316594+0000","last_active":"2024-09-10T19:30:50.360430+0000","last_peered":"2024-09-10T19:30:50.360430+0000","last_clean":"2024-09-10T19:30:50.360430+0000","last_became_active":"2024-09-10T19:27:46.709536+0000","last_became_peered":"2024-09-10T19:27:46.709536+0000","last_unstale":"2024-09-10T19:30:50.360430+0000","last_undegraded":"2024-09-10T19:30:50.360430+0000","last_fullsized":"2024-09-10T19:30:50.360430+0000","mapping_epoch":371,"log_start":"411'12600","ondisk_log_start":"411'12600","created":371,"last_epoch_clean":372,"parent":"0.0","parent_split_bits":0,"last_scrub":"413'12913","last_scrub_stamp":"2024-09-10T19:30:07.316444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:27:45.679464+0000","last_clean_scrub_stamp":"2024-09-10T19:30:07.316444+0000","objects_scrubbed":5296,"log_size":4130,"log_dups_size":0,"ondisk_log_size":4130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:31:23.548541+0000","scrub_duration":1.9481164020000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":91095040,"num_objects":1390,"num_object_clones":0,"num_object_copies":4170,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1390,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16730,"num_write_kb":579840,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"74'79","reported_seq":231,"reported_epoch":418,"state":"active+clean","last_fresh":"2024-09-10T19:30:42.075322+0000","last_change":"2024-09-10T19:30:09.313860+0000","last_active":"2024-09-10T19:30:42.075322+0000","last_peered":"2024-09-10T19:30:42.075322+0000","last_clean":"2024-09-10T19:30:42.075322+0000","last_became_active":"2024-09-10T19:24:23.907264+0000","last_became_peered":"2024-09-10T19:24:23.907264+0000","last_unstale":"2024-09-10T19:30:42.075322+0000","last_undegraded":"2024-09-10T19:30:42.075322+0000","last_fullsized":"2024-09-10T19:30:42.075322+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":4,"last_scrub":"74'79","last_scrub_stamp":"2024-09-10T19:30:09.313780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:08:00.699315+0000","last_clean_scrub_stamp":"2024-09-10T19:30:09.313780+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:31:22.971677+0000","scrub_duration":0.0047975530000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,4],"acting":[5,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"414'68","reported_seq":362,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-09-10T19:30:46.114536+0000","last_change":"2024-09-10T19:30:37.347615+0000","last_active":"2024-09-10T19:30:46.114536+0000","last_peered":"2024-09-10T19:30:46.114536+0000","last_clean":"2024-09-10T19:30:46.114536+0000","last_became_active":"2024-09-10T19:27:53.008533+0000","last_became_peered":"2024-09-10T19:27:53.008533+0000","last_unstale":"2024-09-10T19:30:46.114536+0000","last_undegraded":"2024-09-10T19:30:46.114536+0000","last_fullsized":"2024-09-10T19:30:46.114536+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"414'68","last_scrub_stamp":"2024-09-10T19:30:37.347412+0000","last_deep_scrub":"384'56","last_deep_scrub_stamp":"2024-09-10T19:28:23.457119+0000","last_clean_scrub_stamp":"2024-09-10T19:30:37.347412+0000","objects_scrubbed":10,"log_size":68,"log_dups_size":0,"ondisk_log_size":68,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:31:59.524500+0000","scrub_duration":0.012024986,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":83,"num_read_kb":68,"num_write":50,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":40,"num_bytes_recovered":1432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":493,"reported_epoch":418,"state":"active+clean","last_fresh":"2024-09-10T19:30:42.076247+0000","last_change":"2024-09-10T19:30:26.182093+0000","last_active":"2024-09-10T19:30:42.076247+0000","last_peered":"2024-09-10T19:30:42.076247+0000","last_clean":"2024-09-10T19:30:42.076247+0000","last_became_active":"2024-09-10T19:27:53.004220+0000","last_became_peered":"2024-09-10T19:27:53.004220+0000","last_unstale":"2024-09-10T19:30:42.076247+0000","last_undegraded":"2024-09-10T19:30:42.076247+0000","last_fullsized":"2024-09-10T19:30:42.076247+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:30:26.182025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:30:26.182025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:31:49.676633+0000","scrub_duration":0.0084229500000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":419,"reported_epoch":418,"state":"active+clean","last_fresh":"2024-09-10T19:30:42.070618+0000","last_change":"2024-09-10T19:30:30.855859+0000","last_active":"2024-09-10T19:30:42.070618+0000","last_peered":"2024-09-10T19:30:42.070618+0000","last_clean":"2024-09-10T19:30:42.070618+0000","last_became_active":"2024-09-10T19:24:24.892692+0000","last_became_peered":"2024-09-10T19:24:24.892692+0000","last_unstale":"2024-09-10T19:30:42.070618+0000","last_undegraded":"2024-09-10T19:30:42.070618+0000","last_fullsized":"2024-09-10T19:30:42.070618+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:30:30.855783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:30:30.855783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:31:57.710760+0000","scrub_duration":0.0039706239999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":423,"reported_epoch":418,"state":"active+clean","last_fresh":"2024-09-10T19:30:42.076687+0000","last_change":"2024-09-10T19:29:08.124587+0000","last_active":"2024-09-10T19:30:42.076687+0000","last_peered":"2024-09-10T19:30:42.076687+0000","last_clean":"2024-09-10T19:30:42.076687+0000","last_became_active":"2024-09-10T19:10:13.721744+0000","last_became_peered":"2024-09-10T19:10:13.721744+0000","last_unstale":"2024-09-10T19:30:42.076687+0000","last_undegraded":"2024-09-10T19:30:42.076687+0000","last_fullsized":"2024-09-10T19:30:42.076687+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:29:08.124503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:29:08.124503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.010126721999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.8","version":"419'16643","reported_seq":16920,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-09-10T19:30:50.353806+0000","last_change":"2024-09-10T19:30:09.096902+0000","last_active":"2024-09-10T19:30:50.353806+0000","last_peered":"2024-09-10T19:30:50.353806+0000","last_clean":"2024-09-10T19:30:50.353806+0000","last_became_active":"2024-09-10T19:27:53.024817+0000","last_became_peered":"2024-09-10T19:27:53.024817+0000","last_unstale":"2024-09-10T19:30:50.353806+0000","last_undegraded":"2024-09-10T19:30:50.353806+0000","last_fullsized":"2024-09-10T19:30:50.353806+0000","mapping_epoch":374,"log_start":"411'12500","ondisk_log_start":"411'12500","created":371,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"413'13050","last_scrub_stamp":"2024-09-10T19:30:09.096759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:27:45.679464+0000","last_clean_scrub_stamp":"2024-09-10T19:30:09.096759+0000","objects_scrubbed":5028,"log_size":4143,"log_dups_size":0,"ondisk_log_size":4143,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:31:09.617290+0000","scrub_duration":1.7211459499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":89718784,"num_objects":1369,"num_object_clones":0,"num_object_copies":4107,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1369,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16643,"num_write_kb":576384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"74'79","reported_seq":521,"reported_epoch":418,"state":"active+clean","last_fresh":"2024-09-10T19:30:42.084580+0000","last_change":"2024-09-10T19:29:07.906129+0000","last_active":"2024-09-10T19:30:42.084580+0000","last_peered":"2024-09-10T19:30:42.084580+0000","last_clean":"2024-09-10T19:30:42.084580+0000","last_became_active":"2024-09-10T19:24:23.895247+0000","last_became_peered":"2024-09-10T19:24:23.895247+0000","last_unstale":"2024-09-10T19:30:42.084580+0000","last_undegraded":"2024-09-10T19:30:42.084580+0000","last_fullsized":"2024-09-10T19:30:42.084580+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":4,"last_scrub":"74'79","last_scrub_stamp":"2024-09-10T19:29:07.906054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:08:00.699315+0000","last_clean_scrub_stamp":"2024-09-10T19:29:07.906054+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.006366633,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.7","version":"414'26","reported_seq":198,"reported_epoch":418,"state":"active+clean","last_fresh":"2024-09-10T19:30:42.084036+0000","last_change":"2024-09-10T19:30:34.330565+0000","last_active":"2024-09-10T19:30:42.084036+0000","last_peered":"2024-09-10T19:30:42.084036+0000","last_clean":"2024-09-10T19:30:42.084036+0000","last_became_active":"2024-09-10T19:24:24.894293+0000","last_became_peered":"2024-09-10T19:24:24.894293+0000","last_unstale":"2024-09-10T19:30:42.084036+0000","last_undegraded":"2024-09-10T19:30:42.084036+0000","last_fullsized":"2024-09-10T19:30:42.084036+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"414'26","last_scrub_stamp":"2024-09-10T19:30:34.330411+0000","last_deep_scrub":"82'18","last_deep_scrub_stamp":"2024-09-10T19:15:55.777361+0000","last_clean_scrub_stamp":"2024-09-10T19:30:34.330411+0000","objects_scrubbed":5,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:31:42.777447+0000","scrub_duration":0.010661754000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":42,"num_read_kb":31,"num_write":24,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"159'6","reported_seq":618,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-09-10T19:30:50.597095+0000","last_change":"2024-09-10T19:29:16.485160+0000","last_active":"2024-09-10T19:30:50.597095+0000","last_peered":"2024-09-10T19:30:50.597095+0000","last_clean":"2024-09-10T19:30:50.597095+0000","last_became_active":"2024-09-10T19:27:53.017381+0000","last_became_peered":"2024-09-10T19:27:53.017381+0000","last_unstale":"2024-09-10T19:30:50.597095+0000","last_undegraded":"2024-09-10T19:30:50.597095+0000","last_fullsized":"2024-09-10T19:30:50.597095+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"159'6","last_scrub_stamp":"2024-09-10T19:29:16.485081+0000","last_deep_scrub":"159'6","last_deep_scrub_stamp":"2024-09-10T19:16:32.649914+0000","last_clean_scrub_stamp":"2024-09-10T19:29:16.485081+0000","objects_scrubbed":1,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0054402080000000002,"objects_trimmed":0,"snaptrim_duration":0.00034454499999999997,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":412,"num_read_kb":407,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1556,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":368,"reported_epoch":418,"state":"active+clean","last_fresh":"2024-09-10T19:30:42.076166+0000","last_change":"2024-09-10T19:30:31.835352+0000","last_active":"2024-09-10T19:30:42.076166+0000","last_peered":"2024-09-10T19:30:42.076166+0000","last_clean":"2024-09-10T19:30:42.076166+0000","last_became_active":"2024-09-10T19:24:24.924859+0000","last_became_peered":"2024-09-10T19:24:24.924859+0000","last_unstale":"2024-09-10T19:30:42.076166+0000","last_undegraded":"2024-09-10T19:30:42.076166+0000","last_fullsized":"2024-09-10T19:30:42.076166+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:30:31.835272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:28:21.199065+0000","last_clean_scrub_stamp":"2024-09-10T19:30:31.835272+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:31:56.356268+0000","scrub_duration":0.0079787279999999992,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":337,"reported_epoch":418,"state":"active+clean","last_fresh":"2024-09-10T19:30:42.076277+0000","last_change":"2024-09-10T19:30:23.179974+0000","last_active":"2024-09-10T19:30:42.076277+0000","last_peered":"2024-09-10T19:30:42.076277+0000","last_clean":"2024-09-10T19:30:42.076277+0000","last_became_active":"2024-09-10T19:16:24.861481+0000","last_became_peered":"2024-09-10T19:16:24.861481+0000","last_unstale":"2024-09-10T19:30:42.076277+0000","last_undegraded":"2024-09-10T19:30:42.076277+0000","last_fullsized":"2024-09-10T19:30:42.076277+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:30:23.179923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:23:28.736502+0000","last_clean_scrub_stamp":"2024-09-10T19:30:23.179923+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:31:53.153842+0000","scrub_duration":0.0011139259999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,4],"acting":[5,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":472,"reported_epoch":418,"state":"active+clean","last_fresh":"2024-09-10T19:30:42.082931+0000","last_change":"2024-09-10T19:30:05.863333+0000","last_active":"2024-09-10T19:30:42.082931+0000","last_peered":"2024-09-10T19:30:42.082931+0000","last_clean":"2024-09-10T19:30:42.082931+0000","last_became_active":"2024-09-10T19:27:53.011923+0000","last_became_peered":"2024-09-10T19:27:53.011923+0000","last_unstale":"2024-09-10T19:30:42.082931+0000","last_undegraded":"2024-09-10T19:30:42.082931+0000","last_fullsized":"2024-09-10T19:30:42.082931+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:30:05.863186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:22:01.208272+0000","last_clean_scrub_stamp":"2024-09-10T19:30:05.863186+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:31:07.939735+0000","scrub_duration":0.0076258740000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.9","version":"419'16867","reported_seq":17123,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-09-10T19:30:50.354164+0000","last_change":"2024-09-10T19:30:16.747438+0000","last_active":"2024-09-10T19:30:50.354164+0000","last_peered":"2024-09-10T19:30:50.354164+0000","last_clean":"2024-09-10T19:30:50.354164+0000","last_became_active":"2024-09-10T19:27:53.018973+0000","last_became_peered":"2024-09-10T19:27:53.018973+0000","last_unstale":"2024-09-10T19:30:50.354164+0000","last_undegraded":"2024-09-10T19:30:50.354164+0000","last_fullsized":"2024-09-10T19:30:50.354164+0000","mapping_epoch":374,"log_start":"411'12700","ondisk_log_start":"411'12700","created":371,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"414'13961","last_scrub_stamp":"2024-09-10T19:30:16.747355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:27:45.679464+0000","last_clean_scrub_stamp":"2024-09-10T19:30:16.747355+0000","objects_scrubbed":4475,"log_size":4167,"log_dups_size":0,"ondisk_log_size":4167,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:31:28.383270+0000","scrub_duration":2.2859866719999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":95223808,"num_objects":1453,"num_object_clones":0,"num_object_copies":4359,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1453,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16867,"num_write_kb":586240,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"74'79","reported_seq":186,"reported_epoch":418,"state":"active+clean","last_fresh":"2024-09-10T19:30:42.087893+0000","last_change":"2024-09-10T19:30:40.383314+0000","last_active":"2024-09-10T19:30:42.087893+0000","last_peered":"2024-09-10T19:30:42.087893+0000","last_clean":"2024-09-10T19:30:42.087893+0000","last_became_active":"2024-09-10T19:27:53.016965+0000","last_became_peered":"2024-09-10T19:27:53.016965+0000","last_unstale":"2024-09-10T19:30:42.087893+0000","last_undegraded":"2024-09-10T19:30:42.087893+0000","last_fullsized":"2024-09-10T19:30:42.087893+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":3,"last_scrub":"74'79","last_scrub_stamp":"2024-09-10T19:30:40.383237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:08:00.699315+0000","last_clean_scrub_stamp":"2024-09-10T19:30:40.383237+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:31:57.251947+0000","scrub_duration":0.0084886000000000007,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"414'42","reported_seq":517,"reported_epoch":418,"state":"active+clean","last_fresh":"2024-09-10T19:30:42.071259+0000","last_change":"2024-09-10T19:29:36.586040+0000","last_active":"2024-09-10T19:30:42.071259+0000","last_peered":"2024-09-10T19:30:42.071259+0000","last_clean":"2024-09-10T19:30:42.071259+0000","last_became_active":"2024-09-10T19:24:24.899711+0000","last_became_peered":"2024-09-10T19:24:24.899711+0000","last_unstale":"2024-09-10T19:30:42.071259+0000","last_undegraded":"2024-09-10T19:30:42.071259+0000","last_fullsized":"2024-09-10T19:30:42.071259+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"237'30","last_scrub_stamp":"2024-09-10T19:29:36.584508+0000","last_deep_scrub":"221'18","last_deep_scrub_stamp":"2024-09-10T19:19:29.606748+0000","last_clean_scrub_stamp":"2024-09-10T19:29:36.584508+0000","objects_scrubbed":6,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:30:50.767622+0000","scrub_duration":0.0052445089999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":53,"num_read_kb":47,"num_write":36,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"417'2","reported_seq":413,"reported_epoch":418,"state":"active+clean","last_fresh":"2024-09-10T19:30:42.074344+0000","last_change":"2024-09-10T19:30:39.904486+0000","last_active":"2024-09-10T19:30:42.074344+0000","last_peered":"2024-09-10T19:30:42.074344+0000","last_clean":"2024-09-10T19:30:42.074344+0000","last_became_active":"2024-09-10T19:24:23.897412+0000","last_became_peered":"2024-09-10T19:24:23.897412+0000","last_unstale":"2024-09-10T19:30:42.074344+0000","last_undegraded":"2024-09-10T19:30:42.074344+0000","last_fullsized":"2024-09-10T19:30:42.074344+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"417'2","last_scrub_stamp":"2024-09-10T19:30:39.904339+0000","last_deep_scrub":"417'2","last_deep_scrub_stamp":"2024-09-10T19:30:39.904339+0000","last_clean_scrub_stamp":"2024-09-10T19:30:39.904339+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:32:02.472678+0000","scrub_duration":0.0076308299999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":487,"reported_epoch":418,"state":"active+clean","last_fresh":"2024-09-10T19:30:42.083178+0000","last_change":"2024-09-10T19:30:17.882408+0000","last_active":"2024-09-10T19:30:42.083178+0000","last_peered":"2024-09-10T19:30:42.083178+0000","last_clean":"2024-09-10T19:30:42.083178+0000","last_became_active":"2024-09-10T19:24:24.896201+0000","last_became_peered":"2024-09-10T19:24:24.896201+0000","last_unstale":"2024-09-10T19:30:42.083178+0000","last_undegraded":"2024-09-10T19:30:42.083178+0000","last_fullsized":"2024-09-10T19:30:42.083178+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:30:17.882327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:25:28.762237+0000","last_clean_scrub_stamp":"2024-09-10T19:30:17.882327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:31:19.706681+0000","scrub_duration":0.010437434000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":493,"reported_epoch":418,"state":"active+clean","last_fresh":"2024-09-10T19:30:42.076418+0000","last_change":"2024-09-10T19:30:35.213643+0000","last_active":"2024-09-10T19:30:42.076418+0000","last_peered":"2024-09-10T19:30:42.076418+0000","last_clean":"2024-09-10T19:30:42.076418+0000","last_became_active":"2024-09-10T19:24:24.914680+0000","last_became_peered":"2024-09-10T19:24:24.914680+0000","last_unstale":"2024-09-10T19:30:42.076418+0000","last_undegraded":"2024-09-10T19:30:42.076418+0000","last_fullsized":"2024-09-10T19:30:42.076418+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:30:35.213557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:30:35.213557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:31:44.941744+0000","scrub_duration":0.0069688670000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":166,"reported_epoch":418,"state":"active+clean","last_fresh":"2024-09-10T19:30:42.083651+0000","last_change":"2024-09-10T19:30:31.245345+0000","last_active":"2024-09-10T19:30:42.083651+0000","last_peered":"2024-09-10T19:30:42.083651+0000","last_clean":"2024-09-10T19:30:42.083651+0000","last_became_active":"2024-09-10T19:24:01.974039+0000","last_became_peered":"2024-09-10T19:24:01.974039+0000","last_unstale":"2024-09-10T19:30:42.083651+0000","last_undegraded":"2024-09-10T19:30:42.083651+0000","last_fullsized":"2024-09-10T19:30:42.083651+0000","mapping_epoch":292,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":293,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:30:31.245191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:30:31.245191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:31:45.846908+0000","scrub_duration":0.0041648700000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.a","version":"419'16933","reported_seq":17191,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-09-10T19:30:50.362791+0000","last_change":"2024-09-10T19:30:15.315333+0000","last_active":"2024-09-10T19:30:50.362791+0000","last_peered":"2024-09-10T19:30:50.362791+0000","last_clean":"2024-09-10T19:30:50.362791+0000","last_became_active":"2024-09-10T19:27:53.026021+0000","last_became_peered":"2024-09-10T19:27:53.026021+0000","last_unstale":"2024-09-10T19:30:50.362791+0000","last_undegraded":"2024-09-10T19:30:50.362791+0000","last_fullsized":"2024-09-10T19:30:50.362791+0000","mapping_epoch":374,"log_start":"411'12800","ondisk_log_start":"411'12800","created":371,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"414'13886","last_scrub_stamp":"2024-09-10T19:30:15.315183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:27:45.679464+0000","last_clean_scrub_stamp":"2024-09-10T19:30:15.315183+0000","objects_scrubbed":4548,"log_size":4133,"log_dups_size":0,"ondisk_log_size":4133,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:31:19.467846+0000","scrub_duration":1.898975455,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":94175232,"num_objects":1437,"num_object_clones":0,"num_object_copies":4311,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1437,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16933,"num_write_kb":587840,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"414'45","reported_seq":108,"reported_epoch":418,"state":"active+clean","last_fresh":"2024-09-10T19:30:42.079865+0000","last_change":"2024-09-10T19:30:05.568635+0000","last_active":"2024-09-10T19:30:42.079865+0000","last_peered":"2024-09-10T19:30:42.079865+0000","last_clean":"2024-09-10T19:30:42.079865+0000","last_became_active":"2024-09-10T19:27:53.006290+0000","last_became_peered":"2024-09-10T19:27:53.006290+0000","last_unstale":"2024-09-10T19:30:42.079865+0000","last_undegraded":"2024-09-10T19:30:42.079865+0000","last_fullsized":"2024-09-10T19:30:42.079865+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"237'33","last_scrub_stamp":"2024-09-10T19:30:05.568555+0000","last_deep_scrub":"80'21","last_deep_scrub_stamp":"2024-09-10T19:11:39.836144+0000","last_clean_scrub_stamp":"2024-09-10T19:30:05.568555+0000","objects_scrubbed":7,"log_size":45,"log_dups_size":0,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:31:29.632123+0000","scrub_duration":0.0023682590000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":60,"num_read_kb":50,"num_write":37,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":154,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"63'2","reported_seq":484,"reported_epoch":418,"state":"active+clean","last_fresh":"2024-09-10T19:30:42.077247+0000","last_change":"2024-09-10T19:30:21.647345+0000","last_active":"2024-09-10T19:30:42.077247+0000","last_peered":"2024-09-10T19:30:42.077247+0000","last_clean":"2024-09-10T19:30:42.077247+0000","last_became_active":"2024-09-10T19:27:53.025802+0000","last_became_peered":"2024-09-10T19:27:53.025802+0000","last_unstale":"2024-09-10T19:30:42.077247+0000","last_undegraded":"2024-09-10T19:30:42.077247+0000","last_fullsized":"2024-09-10T19:30:42.077247+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'2","last_scrub_stamp":"2024-09-10T19:30:21.647282+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:30:21.647282+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:31:27.555961+0000","scrub_duration":0.0017519429999999999,"objects_trimmed":0,"snaptrim_duration":0.00075522999999999996,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":459,"reported_epoch":418,"state":"active+clean","last_fresh":"2024-09-10T19:30:42.075704+0000","last_change":"2024-09-10T19:30:25.207929+0000","last_active":"2024-09-10T19:30:42.075704+0000","last_peered":"2024-09-10T19:30:42.075704+0000","last_clean":"2024-09-10T19:30:42.075704+0000","last_became_active":"2024-09-10T19:16:24.858662+0000","last_became_peered":"2024-09-10T19:16:24.858662+0000","last_unstale":"2024-09-10T19:30:42.075704+0000","last_undegraded":"2024-09-10T19:30:42.075704+0000","last_fullsized":"2024-09-10T19:30:42.075704+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:30:25.207849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:30:25.207849+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:31:37.377965+0000","scrub_duration":0.0026356230000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"74'79","reported_seq":535,"reported_epoch":418,"state":"active+clean","last_fresh":"2024-09-10T19:30:42.083250+0000","last_change":"2024-09-10T19:29:34.083194+0000","last_active":"2024-09-10T19:30:42.083250+0000","last_peered":"2024-09-10T19:30:42.083250+0000","last_clean":"2024-09-10T19:30:42.083250+0000","last_became_active":"2024-09-10T19:27:53.009367+0000","last_became_peered":"2024-09-10T19:27:53.009367+0000","last_unstale":"2024-09-10T19:30:42.083250+0000","last_undegraded":"2024-09-10T19:30:42.083250+0000","last_fullsized":"2024-09-10T19:30:42.083250+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'79","last_scrub_stamp":"2024-09-10T19:29:34.080904+0000","last_deep_scrub":"74'79","last_deep_scrub_stamp":"2024-09-10T19:29:34.080904+0000","last_clean_scrub_stamp":"2024-09-10T19:29:34.080904+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:31:01.369559+0000","scrub_duration":0.0067365319999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":472,"reported_epoch":418,"state":"active+clean","last_fresh":"2024-09-10T19:30:42.084159+0000","last_change":"2024-09-10T19:30:32.974376+0000","last_active":"2024-09-10T19:30:42.084159+0000","last_peered":"2024-09-10T19:30:42.084159+0000","last_clean":"2024-09-10T19:30:42.084159+0000","last_became_active":"2024-09-10T19:24:23.903868+0000","last_became_peered":"2024-09-10T19:24:23.903868+0000","last_unstale":"2024-09-10T19:30:42.084159+0000","last_undegraded":"2024-09-10T19:30:42.084159+0000","last_fullsized":"2024-09-10T19:30:42.084159+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:30:32.973996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:30:32.973996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:31:46.944272+0000","scrub_duration":0.0038225009999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,0],"acting":[7,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":443,"reported_epoch":418,"state":"active+clean","last_fresh":"2024-09-10T19:30:42.076540+0000","last_change":"2024-09-10T19:30:21.186215+0000","last_active":"2024-09-10T19:30:42.076540+0000","last_peered":"2024-09-10T19:30:42.076540+0000","last_clean":"2024-09-10T19:30:42.076540+0000","last_became_active":"2024-09-10T19:10:13.720647+0000","last_became_peered":"2024-09-10T19:10:13.720647+0000","last_unstale":"2024-09-10T19:30:42.076540+0000","last_undegraded":"2024-09-10T19:30:42.076540+0000","last_fullsized":"2024-09-10T19:30:42.076540+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:30:21.186015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:28:29.899579+0000","last_clean_scrub_stamp":"2024-09-10T19:30:21.186015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:31:41.388908+0000","scrub_duration":0.0046533069999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.5","version":"419'17019","reported_seq":17463,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-09-10T19:30:50.549012+0000","last_change":"2024-09-10T19:29:25.827058+0000","last_active":"2024-09-10T19:30:50.549012+0000","last_peered":"2024-09-10T19:30:50.549012+0000","last_clean":"2024-09-10T19:30:50.549012+0000","last_became_active":"2024-09-10T19:27:53.010838+0000","last_became_peered":"2024-09-10T19:27:53.010838+0000","last_unstale":"2024-09-10T19:30:50.549012+0000","last_undegraded":"2024-09-10T19:30:50.549012+0000","last_fullsized":"2024-09-10T19:30:50.549012+0000","mapping_epoch":374,"log_start":"410'12700","ondisk_log_start":"410'12700","created":371,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"401'9408","last_scrub_stamp":"2024-09-10T19:29:25.826994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:27:45.679464+0000","last_clean_scrub_stamp":"2024-09-10T19:29:25.826994+0000","objects_scrubbed":9189,"log_size":4319,"log_dups_size":0,"ondisk_log_size":4319,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:30:50.869426+0000","scrub_duration":2.7029503350000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":97189888,"num_objects":1483,"num_object_clones":0,"num_object_copies":4449,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1483,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17019,"num_write_kb":592064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"414'51","reported_seq":118,"reported_epoch":418,"state":"active+clean","last_fresh":"2024-09-10T19:30:42.078843+0000","last_change":"2024-09-10T19:30:12.541122+0000","last_active":"2024-09-10T19:30:42.078843+0000","last_peered":"2024-09-10T19:30:42.078843+0000","last_clean":"2024-09-10T19:30:42.078843+0000","last_became_active":"2024-09-10T19:27:53.014163+0000","last_became_peered":"2024-09-10T19:27:53.014163+0000","last_unstale":"2024-09-10T19:30:42.078843+0000","last_undegraded":"2024-09-10T19:30:42.078843+0000","last_fullsized":"2024-09-10T19:30:42.078843+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"237'35","last_scrub_stamp":"2024-09-10T19:30:12.541044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:30:12.541044+0000","objects_scrubbed":9,"log_size":51,"log_dups_size":0,"ondisk_log_size":51,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:31:28.477210+0000","scrub_duration":0.003693792,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":80,"num_read_kb":69,"num_write":52,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":351,"reported_epoch":418,"state":"active+clean","last_fresh":"2024-09-10T19:30:42.076358+0000","last_change":"2024-09-10T19:29:34.448268+0000","last_active":"2024-09-10T19:30:42.076358+0000","last_peered":"2024-09-10T19:30:42.076358+0000","last_clean":"2024-09-10T19:30:42.076358+0000","last_became_active":"2024-09-10T19:24:24.913709+0000","last_became_peered":"2024-09-10T19:24:24.913709+0000","last_unstale":"2024-09-10T19:30:42.076358+0000","last_undegraded":"2024-09-10T19:30:42.076358+0000","last_fullsized":"2024-09-10T19:30:42.076358+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:29:34.448187+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:29:34.448187+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:30:58.993915+0000","scrub_duration":0.0038381330000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,1],"acting":[5,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":279,"reported_epoch":418,"state":"active+clean","last_fresh":"2024-09-10T19:30:42.072443+0000","last_change":"2024-09-10T19:30:27.831433+0000","last_active":"2024-09-10T19:30:42.072443+0000","last_peered":"2024-09-10T19:30:42.072443+0000","last_clean":"2024-09-10T19:30:42.072443+0000","last_became_active":"2024-09-10T19:24:23.901415+0000","last_became_peered":"2024-09-10T19:24:23.901415+0000","last_unstale":"2024-09-10T19:30:42.072443+0000","last_undegraded":"2024-09-10T19:30:42.072443+0000","last_fullsized":"2024-09-10T19:30:42.072443+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:30:27.831297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:30:27.831297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:31:47.099106+0000","scrub_duration":0.0028653020000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":483,"reported_epoch":418,"state":"active+clean","last_fresh":"2024-09-10T19:30:42.083233+0000","last_change":"2024-09-10T19:30:21.912161+0000","last_active":"2024-09-10T19:30:42.083233+0000","last_peered":"2024-09-10T19:30:42.083233+0000","last_clean":"2024-09-10T19:30:42.083233+0000","last_became_active":"2024-09-10T19:27:53.008850+0000","last_became_peered":"2024-09-10T19:27:53.008850+0000","last_unstale":"2024-09-10T19:30:42.083233+0000","last_undegraded":"2024-09-10T19:30:42.083233+0000","last_fullsized":"2024-09-10T19:30:42.083233+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:30:21.912110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:21:03.910074+0000","last_clean_scrub_stamp":"2024-09-10T19:30:21.912110+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:31:37.278159+0000","scrub_duration":0.001113969,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,6],"acting":[7,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.4","version":"419'16599","reported_seq":17023,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-09-10T19:30:50.586619+0000","last_change":"2024-09-10T19:29:28.791421+0000","last_active":"2024-09-10T19:30:50.586619+0000","last_peered":"2024-09-10T19:30:50.586619+0000","last_clean":"2024-09-10T19:30:50.586619+0000","last_became_active":"2024-09-10T19:27:53.012510+0000","last_became_peered":"2024-09-10T19:27:53.012510+0000","last_unstale":"2024-09-10T19:30:50.586619+0000","last_undegraded":"2024-09-10T19:30:50.586619+0000","last_fullsized":"2024-09-10T19:30:50.586619+0000","mapping_epoch":374,"log_start":"409'12300","ondisk_log_start":"409'12300","created":371,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"401'9402","last_scrub_stamp":"2024-09-10T19:29:28.791340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:27:45.679464+0000","last_clean_scrub_stamp":"2024-09-10T19:29:28.791340+0000","objects_scrubbed":8715,"log_size":4299,"log_dups_size":0,"ondisk_log_size":4299,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"queued for scrub","scrub_duration":3.694930034,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":89456640,"num_objects":1365,"num_object_clones":0,"num_object_copies":4095,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1365,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16599,"num_write_kb":574848,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"414'21","reported_seq":492,"reported_epoch":418,"state":"active+clean","last_fresh":"2024-09-10T19:30:42.072749+0000","last_change":"2024-09-10T19:30:13.503181+0000","last_active":"2024-09-10T19:30:42.072749+0000","last_peered":"2024-09-10T19:30:42.072749+0000","last_clean":"2024-09-10T19:30:42.072749+0000","last_became_active":"2024-09-10T19:24:44.892294+0000","last_became_peered":"2024-09-10T19:24:44.892294+0000","last_unstale":"2024-09-10T19:30:42.072749+0000","last_undegraded":"2024-09-10T19:30:42.072749+0000","last_fullsized":"2024-09-10T19:30:42.072749+0000","mapping_epoch":314,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":315,"parent":"0.0","parent_split_bits":0,"last_scrub":"237'15","last_scrub_stamp":"2024-09-10T19:30:13.502912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:30:13.502912+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:31:38.767513+0000","scrub_duration":0.0086509359999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":34,"num_read_kb":28,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,4],"acting":[0,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":156,"reported_epoch":418,"state":"active+clean","last_fresh":"2024-09-10T19:30:42.088866+0000","last_change":"2024-09-10T19:29:14.403288+0000","last_active":"2024-09-10T19:30:42.088866+0000","last_peered":"2024-09-10T19:30:42.088866+0000","last_clean":"2024-09-10T19:30:42.088866+0000","last_became_active":"2024-09-10T19:24:01.980348+0000","last_became_peered":"2024-09-10T19:24:01.980348+0000","last_unstale":"2024-09-10T19:30:42.088866+0000","last_undegraded":"2024-09-10T19:30:42.088866+0000","last_fullsized":"2024-09-10T19:30:42.088866+0000","mapping_epoch":292,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":293,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:29:14.403209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:11:44.254513+0000","last_clean_scrub_stamp":"2024-09-10T19:29:14.403209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0046595619999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":347,"reported_epoch":418,"state":"active+clean","last_fresh":"2024-09-10T19:30:42.074197+0000","last_change":"2024-09-10T19:30:05.467928+0000","last_active":"2024-09-10T19:30:42.074197+0000","last_peered":"2024-09-10T19:30:42.074197+0000","last_clean":"2024-09-10T19:30:42.074197+0000","last_became_active":"2024-09-10T19:24:23.891161+0000","last_became_peered":"2024-09-10T19:24:23.891161+0000","last_unstale":"2024-09-10T19:30:42.074197+0000","last_undegraded":"2024-09-10T19:30:42.074197+0000","last_fullsized":"2024-09-10T19:30:42.074197+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:30:05.467845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:30:05.467845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:31:32.032620+0000","scrub_duration":0.0056686540000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":397,"reported_epoch":418,"state":"active+clean","last_fresh":"2024-09-10T19:30:42.074412+0000","last_change":"2024-09-10T19:30:33.974218+0000","last_active":"2024-09-10T19:30:42.074412+0000","last_peered":"2024-09-10T19:30:42.074412+0000","last_clean":"2024-09-10T19:30:42.074412+0000","last_became_active":"2024-09-10T19:24:23.905112+0000","last_became_peered":"2024-09-10T19:24:23.905112+0000","last_unstale":"2024-09-10T19:30:42.074412+0000","last_undegraded":"2024-09-10T19:30:42.074412+0000","last_fullsized":"2024-09-10T19:30:42.074412+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:30:33.974134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:30:33.974134+0000","last_clean_scrub_stamp":"2024-09-10T19:30:33.974134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:31:46.881624+0000","scrub_duration":0.0063507879999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.7","version":"419'16719","reported_seq":17116,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-09-10T19:30:50.587429+0000","last_change":"2024-09-10T19:29:34.192828+0000","last_active":"2024-09-10T19:30:50.587429+0000","last_peered":"2024-09-10T19:30:50.587429+0000","last_clean":"2024-09-10T19:30:50.587429+0000","last_became_active":"2024-09-10T19:27:46.707393+0000","last_became_peered":"2024-09-10T19:27:46.707393+0000","last_unstale":"2024-09-10T19:30:50.587429+0000","last_undegraded":"2024-09-10T19:30:50.587429+0000","last_fullsized":"2024-09-10T19:30:50.587429+0000","mapping_epoch":371,"log_start":"409'12100","ondisk_log_start":"409'12100","created":371,"last_epoch_clean":372,"parent":"0.0","parent_split_bits":0,"last_scrub":"404'9864","last_scrub_stamp":"2024-09-10T19:29:34.192737+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:27:45.679464+0000","last_clean_scrub_stamp":"2024-09-10T19:29:34.192737+0000","objects_scrubbed":8292,"log_size":4619,"log_dups_size":0,"ondisk_log_size":4619,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"queued for scrub","scrub_duration":2.7792294690000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":86441984,"num_objects":1319,"num_object_clones":0,"num_object_copies":3957,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1319,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16719,"num_write_kb":577216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,4],"acting":[5,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.8","version":"414'46","reported_seq":557,"reported_epoch":418,"state":"active+clean","last_fresh":"2024-09-10T19:30:42.075489+0000","last_change":"2024-09-10T19:30:06.380329+0000","last_active":"2024-09-10T19:30:42.075489+0000","last_peered":"2024-09-10T19:30:42.075489+0000","last_clean":"2024-09-10T19:30:42.075489+0000","last_became_active":"2024-09-10T19:27:53.004858+0000","last_became_peered":"2024-09-10T19:27:53.004858+0000","last_unstale":"2024-09-10T19:30:42.075489+0000","last_undegraded":"2024-09-10T19:30:42.075489+0000","last_fullsized":"2024-09-10T19:30:42.075489+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"237'34","last_scrub_stamp":"2024-09-10T19:30:06.380238+0000","last_deep_scrub":"220'22","last_deep_scrub_stamp":"2024-09-10T19:19:24.597866+0000","last_clean_scrub_stamp":"2024-09-10T19:30:06.380238+0000","objects_scrubbed":7,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:31:27.961508+0000","scrub_duration":0.0065648299999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":62,"num_read_kb":53,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":347,"reported_epoch":418,"state":"active+clean","last_fresh":"2024-09-10T19:30:42.076855+0000","last_change":"2024-09-10T19:30:25.698629+0000","last_active":"2024-09-10T19:30:42.076855+0000","last_peered":"2024-09-10T19:30:42.076855+0000","last_clean":"2024-09-10T19:30:42.076855+0000","last_became_active":"2024-09-10T19:27:53.016898+0000","last_became_peered":"2024-09-10T19:27:53.016898+0000","last_unstale":"2024-09-10T19:30:42.076855+0000","last_undegraded":"2024-09-10T19:30:42.076855+0000","last_fullsized":"2024-09-10T19:30:42.076855+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:30:25.698575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:30:25.698575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:31:48.430610+0000","scrub_duration":0.008668004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.9","version":"376'11","reported_seq":740,"reported_epoch":418,"state":"active+clean","last_fresh":"2024-09-10T19:30:45.976347+0000","last_change":"2024-09-10T19:30:09.879798+0000","last_active":"2024-09-10T19:30:45.976347+0000","last_peered":"2024-09-10T19:30:45.976347+0000","last_clean":"2024-09-10T19:30:45.976347+0000","last_became_active":"2024-09-10T19:27:53.009810+0000","last_became_peered":"2024-09-10T19:27:53.009810+0000","last_unstale":"2024-09-10T19:30:45.976347+0000","last_undegraded":"2024-09-10T19:30:45.976347+0000","last_fullsized":"2024-09-10T19:30:45.976347+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"376'11","last_scrub_stamp":"2024-09-10T19:30:09.879695+0000","last_deep_scrub":"80'9","last_deep_scrub_stamp":"2024-09-10T19:25:29.833271+0000","last_clean_scrub_stamp":"2024-09-10T19:30:09.879695+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:31:37.107588+0000","scrub_duration":0.0040800790000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":460,"reported_epoch":418,"state":"active+clean","last_fresh":"2024-09-10T19:30:42.076115+0000","last_change":"2024-09-10T19:29:27.408515+0000","last_active":"2024-09-10T19:30:42.076115+0000","last_peered":"2024-09-10T19:30:42.076115+0000","last_clean":"2024-09-10T19:30:42.076115+0000","last_became_active":"2024-09-10T19:27:53.015957+0000","last_became_peered":"2024-09-10T19:27:53.015957+0000","last_unstale":"2024-09-10T19:30:42.076115+0000","last_undegraded":"2024-09-10T19:30:42.076115+0000","last_fullsized":"2024-09-10T19:30:42.076115+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:29:27.408374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:29:27.408374+0000","last_clean_scrub_stamp":"2024-09-10T19:29:27.408374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:30:50.070590+0000","scrub_duration":0.0088258940000000008,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.6","version":"419'16969","reported_seq":17373,"reported_epoch":419,"state":"active+clean","last_fresh":"2024-09-10T19:30:52.060872+0000","last_change":"2024-09-10T19:29:17.664714+0000","last_active":"2024-09-10T19:30:52.060872+0000","last_peered":"2024-09-10T19:30:52.060872+0000","last_clean":"2024-09-10T19:30:52.060872+0000","last_became_active":"2024-09-10T19:27:46.704823+0000","last_became_peered":"2024-09-10T19:27:46.704823+0000","last_unstale":"2024-09-10T19:30:52.060872+0000","last_undegraded":"2024-09-10T19:30:52.060872+0000","last_fullsized":"2024-09-10T19:30:52.060872+0000","mapping_epoch":371,"log_start":"409'12300","ondisk_log_start":"409'12300","created":371,"last_epoch_clean":372,"parent":"0.0","parent_split_bits":0,"last_scrub":"400'8581","last_scrub_stamp":"2024-09-10T19:29:17.664631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:27:45.679464+0000","last_clean_scrub_stamp":"2024-09-10T19:29:17.664631+0000","objects_scrubbed":8456,"log_size":4669,"log_dups_size":0,"ondisk_log_size":4669,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"queued for scrub","scrub_duration":2.9535849939999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":84082688,"num_objects":1283,"num_object_clones":0,"num_object_copies":3849,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1283,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16969,"num_write_kb":584064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.9","version":"414'24","reported_seq":531,"reported_epoch":418,"state":"active+clean","last_fresh":"2024-09-10T19:30:42.073862+0000","last_change":"2024-09-10T19:29:18.241176+0000","last_active":"2024-09-10T19:30:42.073862+0000","last_peered":"2024-09-10T19:30:42.073862+0000","last_clean":"2024-09-10T19:30:42.073862+0000","last_became_active":"2024-09-10T19:24:01.974878+0000","last_became_peered":"2024-09-10T19:24:01.974878+0000","last_unstale":"2024-09-10T19:30:42.073862+0000","last_undegraded":"2024-09-10T19:30:42.073862+0000","last_fullsized":"2024-09-10T19:30:42.073862+0000","mapping_epoch":292,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":293,"parent":"0.0","parent_split_bits":0,"last_scrub":"237'18","last_scrub_stamp":"2024-09-10T19:29:18.241077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:29:18.241077+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0095846869999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.f","version":"59'2","reported_seq":471,"reported_epoch":418,"state":"active+clean","last_fresh":"2024-09-10T19:30:42.075321+0000","last_change":"2024-09-10T19:30:15.444681+0000","last_active":"2024-09-10T19:30:42.075321+0000","last_peered":"2024-09-10T19:30:42.075321+0000","last_clean":"2024-09-10T19:30:42.075321+0000","last_became_active":"2024-09-10T19:24:23.893796+0000","last_became_peered":"2024-09-10T19:24:23.893796+0000","last_unstale":"2024-09-10T19:30:42.075321+0000","last_undegraded":"2024-09-10T19:30:42.075321+0000","last_fullsized":"2024-09-10T19:30:42.075321+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'2","last_scrub_stamp":"2024-09-10T19:30:15.444556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:30:15.444556+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:31:42.533307+0000","scrub_duration":0.003937945,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":14,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":381,"reported_epoch":418,"state":"active+clean","last_fresh":"2024-09-10T19:30:42.070343+0000","last_change":"2024-09-10T19:29:16.702372+0000","last_active":"2024-09-10T19:30:42.070343+0000","last_peered":"2024-09-10T19:30:42.070343+0000","last_clean":"2024-09-10T19:30:42.070343+0000","last_became_active":"2024-09-10T19:24:24.914252+0000","last_became_peered":"2024-09-10T19:24:24.914252+0000","last_unstale":"2024-09-10T19:30:42.070343+0000","last_undegraded":"2024-09-10T19:30:42.070343+0000","last_fullsized":"2024-09-10T19:30:42.070343+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:29:16.702283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:11:46.587846+0000","last_clean_scrub_stamp":"2024-09-10T19:29:16.702283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0047568310000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,5],"acting":[0,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":460,"reported_epoch":418,"state":"active+clean","last_fresh":"2024-09-10T19:30:42.076496+0000","last_change":"2024-09-10T19:30:17.554426+0000","last_active":"2024-09-10T19:30:42.076496+0000","last_peered":"2024-09-10T19:30:42.076496+0000","last_clean":"2024-09-10T19:30:42.076496+0000","last_became_active":"2024-09-10T19:24:01.971131+0000","last_became_peered":"2024-09-10T19:24:01.971131+0000","last_unstale":"2024-09-10T19:30:42.076496+0000","last_undegraded":"2024-09-10T19:30:42.076496+0000","last_fullsized":"2024-09-10T19:30:42.076496+0000","mapping_epoch":292,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":293,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:30:17.554352+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:30:17.554352+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:31:26.930983+0000","scrub_duration":0.001698869,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.16","version":"414'21","reported_seq":446,"reported_epoch":418,"state":"active+clean","last_fresh":"2024-09-10T19:30:42.071031+0000","last_change":"2024-09-10T19:29:13.724314+0000","last_active":"2024-09-10T19:30:42.071031+0000","last_peered":"2024-09-10T19:30:42.071031+0000","last_clean":"2024-09-10T19:30:42.071031+0000","last_became_active":"2024-09-10T19:24:23.901610+0000","last_became_peered":"2024-09-10T19:24:23.901610+0000","last_unstale":"2024-09-10T19:30:42.071031+0000","last_undegraded":"2024-09-10T19:30:42.071031+0000","last_fullsized":"2024-09-10T19:30:42.071031+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"237'15","last_scrub_stamp":"2024-09-10T19:29:13.724222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:29:13.724222+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0074413179999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":34,"num_read_kb":28,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":180,"reported_epoch":418,"state":"active+clean","last_fresh":"2024-09-10T19:30:42.086305+0000","last_change":"2024-09-10T19:29:18.504177+0000","last_active":"2024-09-10T19:30:42.086305+0000","last_peered":"2024-09-10T19:30:42.086305+0000","last_clean":"2024-09-10T19:30:42.086305+0000","last_became_active":"2024-09-10T19:24:24.894815+0000","last_became_peered":"2024-09-10T19:24:24.894815+0000","last_unstale":"2024-09-10T19:30:42.086305+0000","last_undegraded":"2024-09-10T19:30:42.086305+0000","last_fullsized":"2024-09-10T19:30:42.086305+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:29:18.504104+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:11:41.458523+0000","last_clean_scrub_stamp":"2024-09-10T19:29:18.504104+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0057679100000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,4],"acting":[1,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":459,"reported_epoch":418,"state":"active+clean","last_fresh":"2024-09-10T19:30:42.077247+0000","last_change":"2024-09-10T19:29:29.071593+0000","last_active":"2024-09-10T19:30:42.077247+0000","last_peered":"2024-09-10T19:30:42.077247+0000","last_clean":"2024-09-10T19:30:42.077247+0000","last_became_active":"2024-09-10T19:24:01.970325+0000","last_became_peered":"2024-09-10T19:24:01.970325+0000","last_unstale":"2024-09-10T19:30:42.077247+0000","last_undegraded":"2024-09-10T19:30:42.077247+0000","last_fullsized":"2024-09-10T19:30:42.077247+0000","mapping_epoch":292,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":293,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:29:29.071516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:24:09.367654+0000","last_clean_scrub_stamp":"2024-09-10T19:29:29.071516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:30:54.608866+0000","scrub_duration":0.0048760050000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"66'1","reported_seq":347,"reported_epoch":418,"state":"active+clean","last_fresh":"2024-09-10T19:30:42.075207+0000","last_change":"2024-09-10T19:29:28.371890+0000","last_active":"2024-09-10T19:30:42.075207+0000","last_peered":"2024-09-10T19:30:42.075207+0000","last_clean":"2024-09-10T19:30:42.075207+0000","last_became_active":"2024-09-10T19:24:23.897858+0000","last_became_peered":"2024-09-10T19:24:23.897858+0000","last_unstale":"2024-09-10T19:30:42.075207+0000","last_undegraded":"2024-09-10T19:30:42.075207+0000","last_fullsized":"2024-09-10T19:30:42.075207+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'1","last_scrub_stamp":"2024-09-10T19:29:28.371731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:29:28.371731+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:30:51.061816+0000","scrub_duration":0.028433127999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":13,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":84,"reported_epoch":418,"state":"active+clean","last_fresh":"2024-09-10T19:30:42.078506+0000","last_change":"2024-09-10T19:30:34.487341+0000","last_active":"2024-09-10T19:30:42.078506+0000","last_peered":"2024-09-10T19:30:42.078506+0000","last_clean":"2024-09-10T19:30:42.078506+0000","last_became_active":"2024-09-10T19:27:53.007487+0000","last_became_peered":"2024-09-10T19:27:53.007487+0000","last_unstale":"2024-09-10T19:30:42.078506+0000","last_undegraded":"2024-09-10T19:30:42.078506+0000","last_fullsized":"2024-09-10T19:30:42.078506+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:30:34.487271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:30:34.487271+0000","last_clean_scrub_stamp":"2024-09-10T19:30:34.487271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:31:58.958373+0000","scrub_duration":0.007547064,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"414'18","reported_seq":515,"reported_epoch":418,"state":"active+clean","last_fresh":"2024-09-10T19:30:42.077422+0000","last_change":"2024-09-10T19:30:04.390756+0000","last_active":"2024-09-10T19:30:42.077422+0000","last_peered":"2024-09-10T19:30:42.077422+0000","last_clean":"2024-09-10T19:30:42.077422+0000","last_became_active":"2024-09-10T19:24:01.971492+0000","last_became_peered":"2024-09-10T19:24:01.971492+0000","last_unstale":"2024-09-10T19:30:42.077422+0000","last_undegraded":"2024-09-10T19:30:42.077422+0000","last_fullsized":"2024-09-10T19:30:42.077422+0000","mapping_epoch":292,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":293,"parent":"0.0","parent_split_bits":0,"last_scrub":"237'12","last_scrub_stamp":"2024-09-10T19:30:04.390680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:30:04.390680+0000","objects_scrubbed":3,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:31:06.735608+0000","scrub_duration":0.0029448790000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":27,"num_read_kb":24,"num_write":18,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":462,"reported_epoch":418,"state":"active+clean","last_fresh":"2024-09-10T19:30:42.076349+0000","last_change":"2024-09-10T19:30:22.162341+0000","last_active":"2024-09-10T19:30:42.076349+0000","last_peered":"2024-09-10T19:30:42.076349+0000","last_clean":"2024-09-10T19:30:42.076349+0000","last_became_active":"2024-09-10T19:24:01.978215+0000","last_became_peered":"2024-09-10T19:24:01.978215+0000","last_unstale":"2024-09-10T19:30:42.076349+0000","last_undegraded":"2024-09-10T19:30:42.076349+0000","last_fullsized":"2024-09-10T19:30:42.076349+0000","mapping_epoch":292,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":293,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:30:22.162288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:11:23.264793+0000","last_clean_scrub_stamp":"2024-09-10T19:30:22.162288+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:31:34.322359+0000","scrub_duration":0.0055022539999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":463,"reported_epoch":418,"state":"active+clean","last_fresh":"2024-09-10T19:30:42.082861+0000","last_change":"2024-09-10T19:29:17.920965+0000","last_active":"2024-09-10T19:30:42.082861+0000","last_peered":"2024-09-10T19:30:42.082861+0000","last_clean":"2024-09-10T19:30:42.082861+0000","last_became_active":"2024-09-10T19:27:53.010120+0000","last_became_peered":"2024-09-10T19:27:53.010120+0000","last_unstale":"2024-09-10T19:30:42.082861+0000","last_undegraded":"2024-09-10T19:30:42.082861+0000","last_fullsized":"2024-09-10T19:30:42.082861+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:29:17.920880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:29:17.920880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0025791519999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"414'35","reported_seq":560,"reported_epoch":418,"state":"active+clean","last_fresh":"2024-09-10T19:30:42.076151+0000","last_change":"2024-09-10T19:30:23.678519+0000","last_active":"2024-09-10T19:30:42.076151+0000","last_peered":"2024-09-10T19:30:42.076151+0000","last_clean":"2024-09-10T19:30:42.076151+0000","last_became_active":"2024-09-10T19:24:01.971173+0000","last_became_peered":"2024-09-10T19:24:01.971173+0000","last_unstale":"2024-09-10T19:30:42.076151+0000","last_undegraded":"2024-09-10T19:30:42.076151+0000","last_fullsized":"2024-09-10T19:30:42.076151+0000","mapping_epoch":292,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":293,"parent":"0.0","parent_split_bits":0,"last_scrub":"414'35","last_scrub_stamp":"2024-09-10T19:30:23.678420+0000","last_deep_scrub":"80'15","last_deep_scrub_stamp":"2024-09-10T19:11:41.719310+0000","last_clean_scrub_stamp":"2024-09-10T19:30:23.678420+0000","objects_scrubbed":5,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:31:41.464166+0000","scrub_duration":0.0043333590000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":45,"num_read_kb":40,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":439,"reported_epoch":418,"state":"active+clean","last_fresh":"2024-09-10T19:30:42.075786+0000","last_change":"2024-09-10T19:29:36.523640+0000","last_active":"2024-09-10T19:30:42.075786+0000","last_peered":"2024-09-10T19:30:42.075786+0000","last_clean":"2024-09-10T19:30:42.075786+0000","last_became_active":"2024-09-10T19:10:07.685870+0000","last_became_peered":"2024-09-10T19:10:07.685870+0000","last_unstale":"2024-09-10T19:30:42.075786+0000","last_undegraded":"2024-09-10T19:30:42.075786+0000","last_fullsized":"2024-09-10T19:30:42.075786+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:29:36.523540+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:29:36.523540+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0026209499999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"372'16","reported_seq":756,"reported_epoch":418,"state":"active+clean","last_fresh":"2024-09-10T19:30:45.976350+0000","last_change":"2024-09-10T19:29:26.417697+0000","last_active":"2024-09-10T19:30:45.976350+0000","last_peered":"2024-09-10T19:30:45.976350+0000","last_clean":"2024-09-10T19:30:45.976350+0000","last_became_active":"2024-09-10T19:27:46.703433+0000","last_became_peered":"2024-09-10T19:27:46.703433+0000","last_unstale":"2024-09-10T19:30:45.976350+0000","last_undegraded":"2024-09-10T19:30:45.976350+0000","last_fullsized":"2024-09-10T19:30:45.976350+0000","mapping_epoch":371,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":372,"parent":"0.0","parent_split_bits":0,"last_scrub":"372'16","last_scrub_stamp":"2024-09-10T19:29:26.415252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:29:26.415252+0000","objects_scrubbed":1,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.007524256,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,0],"acting":[5,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":393,"reported_epoch":418,"state":"active+clean","last_fresh":"2024-09-10T19:30:42.071112+0000","last_change":"2024-09-10T19:29:34.578620+0000","last_active":"2024-09-10T19:30:42.071112+0000","last_peered":"2024-09-10T19:30:42.071112+0000","last_clean":"2024-09-10T19:30:42.071112+0000","last_became_active":"2024-09-10T19:24:23.900513+0000","last_became_peered":"2024-09-10T19:24:23.900513+0000","last_unstale":"2024-09-10T19:30:42.071112+0000","last_undegraded":"2024-09-10T19:30:42.071112+0000","last_fullsized":"2024-09-10T19:30:42.071112+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:29:34.578469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:29:34.578469+0000","last_clean_scrub_stamp":"2024-09-10T19:29:34.578469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0058680599999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"414'21","reported_seq":511,"reported_epoch":418,"state":"active+clean","last_fresh":"2024-09-10T19:30:42.076934+0000","last_change":"2024-09-10T19:29:32.382258+0000","last_active":"2024-09-10T19:30:42.076934+0000","last_peered":"2024-09-10T19:30:42.076934+0000","last_clean":"2024-09-10T19:30:42.076934+0000","last_became_active":"2024-09-10T19:16:24.859740+0000","last_became_peered":"2024-09-10T19:16:24.859740+0000","last_unstale":"2024-09-10T19:30:42.076934+0000","last_undegraded":"2024-09-10T19:30:42.076934+0000","last_fullsized":"2024-09-10T19:30:42.076934+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"237'15","last_scrub_stamp":"2024-09-10T19:29:32.382176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:29:32.382176+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0057339599999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":396,"reported_epoch":418,"state":"active+clean","last_fresh":"2024-09-10T19:30:42.072504+0000","last_change":"2024-09-10T19:29:11.750820+0000","last_active":"2024-09-10T19:30:42.072504+0000","last_peered":"2024-09-10T19:30:42.072504+0000","last_clean":"2024-09-10T19:30:42.072504+0000","last_became_active":"2024-09-10T19:24:23.895163+0000","last_became_peered":"2024-09-10T19:24:23.895163+0000","last_unstale":"2024-09-10T19:30:42.072504+0000","last_undegraded":"2024-09-10T19:30:42.072504+0000","last_fullsized":"2024-09-10T19:30:42.072504+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:29:11.750709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:29:11.750709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0029571649999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"297'11","reported_seq":720,"reported_epoch":418,"state":"active+clean","last_fresh":"2024-09-10T19:30:45.977141+0000","last_change":"2024-09-10T19:29:11.159023+0000","last_active":"2024-09-10T19:30:45.977141+0000","last_peered":"2024-09-10T19:30:45.977141+0000","last_clean":"2024-09-10T19:30:45.977141+0000","last_became_active":"2024-09-10T19:24:01.974816+0000","last_became_peered":"2024-09-10T19:24:01.974816+0000","last_unstale":"2024-09-10T19:30:45.977141+0000","last_undegraded":"2024-09-10T19:30:45.977141+0000","last_fullsized":"2024-09-10T19:30:45.977141+0000","mapping_epoch":292,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":293,"parent":"0.0","parent_split_bits":0,"last_scrub":"297'11","last_scrub_stamp":"2024-09-10T19:29:11.158902+0000","last_deep_scrub":"297'11","last_deep_scrub_stamp":"2024-09-10T19:24:13.318130+0000","last_clean_scrub_stamp":"2024-09-10T19:29:11.158902+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.014823852,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":452,"reported_epoch":418,"state":"active+clean","last_fresh":"2024-09-10T19:30:42.075136+0000","last_change":"2024-09-10T19:29:32.361240+0000","last_active":"2024-09-10T19:30:42.075136+0000","last_peered":"2024-09-10T19:30:42.075136+0000","last_clean":"2024-09-10T19:30:42.075136+0000","last_became_active":"2024-09-10T19:24:23.903779+0000","last_became_peered":"2024-09-10T19:24:23.903779+0000","last_unstale":"2024-09-10T19:30:42.075136+0000","last_undegraded":"2024-09-10T19:30:42.075136+0000","last_fullsized":"2024-09-10T19:30:42.075136+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:29:32.361029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:29:32.361029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:30:58.523472+0000","scrub_duration":0.010203366,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.12","version":"414'25","reported_seq":239,"reported_epoch":418,"state":"active+clean","last_fresh":"2024-09-10T19:30:42.083318+0000","last_change":"2024-09-10T19:30:23.274575+0000","last_active":"2024-09-10T19:30:42.083318+0000","last_peered":"2024-09-10T19:30:42.083318+0000","last_clean":"2024-09-10T19:30:42.083318+0000","last_became_active":"2024-09-10T19:27:53.017161+0000","last_became_peered":"2024-09-10T19:27:53.017161+0000","last_unstale":"2024-09-10T19:30:42.083318+0000","last_undegraded":"2024-09-10T19:30:42.083318+0000","last_fullsized":"2024-09-10T19:30:42.083318+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"414'25","last_scrub_stamp":"2024-09-10T19:30:23.274493+0000","last_deep_scrub":"79'13","last_deep_scrub_stamp":"2024-09-10T19:11:22.072400+0000","last_clean_scrub_stamp":"2024-09-10T19:30:23.274493+0000","objects_scrubbed":4,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:31:42.647014+0000","scrub_duration":0.0027214610000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":34,"num_read_kb":28,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":550,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"59'1","reported_seq":88,"reported_epoch":418,"state":"active+clean","last_fresh":"2024-09-10T19:30:42.078707+0000","last_change":"2024-09-10T19:30:31.402122+0000","last_active":"2024-09-10T19:30:42.078707+0000","last_peered":"2024-09-10T19:30:42.078707+0000","last_clean":"2024-09-10T19:30:42.078707+0000","last_became_active":"2024-09-10T19:27:53.018470+0000","last_became_peered":"2024-09-10T19:27:53.018470+0000","last_unstale":"2024-09-10T19:30:42.078707+0000","last_undegraded":"2024-09-10T19:30:42.078707+0000","last_fullsized":"2024-09-10T19:30:42.078707+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'1","last_scrub_stamp":"2024-09-10T19:30:31.401664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:30:31.401664+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:31:59.702734+0000","scrub_duration":0.0059519009999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":362,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":724,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":473,"reported_epoch":418,"state":"active+clean","last_fresh":"2024-09-10T19:30:42.076562+0000","last_change":"2024-09-10T19:29:19.145340+0000","last_active":"2024-09-10T19:30:42.076562+0000","last_peered":"2024-09-10T19:30:42.076562+0000","last_clean":"2024-09-10T19:30:42.076562+0000","last_became_active":"2024-09-10T19:24:24.923804+0000","last_became_peered":"2024-09-10T19:24:24.923804+0000","last_unstale":"2024-09-10T19:30:42.076562+0000","last_undegraded":"2024-09-10T19:30:42.076562+0000","last_fullsized":"2024-09-10T19:30:42.076562+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:29:19.145160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:29:19.145160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0019117959999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":412,"reported_epoch":418,"state":"active+clean","last_fresh":"2024-09-10T19:30:42.070085+0000","last_change":"2024-09-10T19:29:28.581248+0000","last_active":"2024-09-10T19:30:42.070085+0000","last_peered":"2024-09-10T19:30:42.070085+0000","last_clean":"2024-09-10T19:30:42.070085+0000","last_became_active":"2024-09-10T19:24:24.916105+0000","last_became_peered":"2024-09-10T19:24:24.916105+0000","last_unstale":"2024-09-10T19:30:42.070085+0000","last_undegraded":"2024-09-10T19:30:42.070085+0000","last_fullsized":"2024-09-10T19:30:42.070085+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:29:28.581177+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:25:27.412579+0000","last_clean_scrub_stamp":"2024-09-10T19:29:28.581177+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:30:47.919026+0000","scrub_duration":0.0049057299999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"414'27","reported_seq":577,"reported_epoch":418,"state":"active+clean","last_fresh":"2024-09-10T19:30:42.074598+0000","last_change":"2024-09-10T19:29:26.356083+0000","last_active":"2024-09-10T19:30:42.074598+0000","last_peered":"2024-09-10T19:30:42.074598+0000","last_clean":"2024-09-10T19:30:42.074598+0000","last_became_active":"2024-09-10T19:27:53.016518+0000","last_became_peered":"2024-09-10T19:27:53.016518+0000","last_unstale":"2024-09-10T19:30:42.074598+0000","last_undegraded":"2024-09-10T19:30:42.074598+0000","last_fullsized":"2024-09-10T19:30:42.074598+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"237'19","last_scrub_stamp":"2024-09-10T19:29:26.355998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:29:26.355998+0000","objects_scrubbed":5,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.030792693999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":37,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":180,"reported_epoch":418,"state":"active+clean","last_fresh":"2024-09-10T19:30:42.088813+0000","last_change":"2024-09-10T19:29:09.345677+0000","last_active":"2024-09-10T19:30:42.088813+0000","last_peered":"2024-09-10T19:30:42.088813+0000","last_clean":"2024-09-10T19:30:42.088813+0000","last_became_active":"2024-09-10T19:24:24.909540+0000","last_became_peered":"2024-09-10T19:24:24.909540+0000","last_unstale":"2024-09-10T19:30:42.088813+0000","last_undegraded":"2024-09-10T19:30:42.088813+0000","last_fullsized":"2024-09-10T19:30:42.088813+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:29:09.345588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:26:31.508168+0000","last_clean_scrub_stamp":"2024-09-10T19:29:09.345588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.008652847,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":156,"reported_epoch":418,"state":"active+clean","last_fresh":"2024-09-10T19:30:42.083482+0000","last_change":"2024-09-10T19:28:29.495516+0000","last_active":"2024-09-10T19:30:42.083482+0000","last_peered":"2024-09-10T19:30:42.083482+0000","last_clean":"2024-09-10T19:30:42.083482+0000","last_became_active":"2024-09-10T19:24:01.975520+0000","last_became_peered":"2024-09-10T19:24:01.975520+0000","last_unstale":"2024-09-10T19:30:42.083482+0000","last_undegraded":"2024-09-10T19:30:42.083482+0000","last_fullsized":"2024-09-10T19:30:42.083482+0000","mapping_epoch":292,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":293,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:28:29.495437+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:28:29.495437+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0066786709999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":440,"reported_epoch":418,"state":"active+clean","last_fresh":"2024-09-10T19:30:42.076541+0000","last_change":"2024-09-10T19:29:33.122658+0000","last_active":"2024-09-10T19:30:42.076541+0000","last_peered":"2024-09-10T19:30:42.076541+0000","last_clean":"2024-09-10T19:30:42.076541+0000","last_became_active":"2024-09-10T19:24:23.907136+0000","last_became_peered":"2024-09-10T19:24:23.907136+0000","last_unstale":"2024-09-10T19:30:42.076541+0000","last_undegraded":"2024-09-10T19:30:42.076541+0000","last_fullsized":"2024-09-10T19:30:42.076541+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:29:33.121149+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:29:33.121149+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:31:00.857785+0000","scrub_duration":0.0077727279999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"414'14","reported_seq":528,"reported_epoch":418,"state":"active+clean","last_fresh":"2024-09-10T19:30:42.077184+0000","last_change":"2024-09-10T19:30:06.362408+0000","last_active":"2024-09-10T19:30:42.077184+0000","last_peered":"2024-09-10T19:30:42.077184+0000","last_clean":"2024-09-10T19:30:42.077184+0000","last_became_active":"2024-09-10T19:27:53.024782+0000","last_became_peered":"2024-09-10T19:27:53.024782+0000","last_unstale":"2024-09-10T19:30:42.077184+0000","last_undegraded":"2024-09-10T19:30:42.077184+0000","last_fullsized":"2024-09-10T19:30:42.077184+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"237'10","last_scrub_stamp":"2024-09-10T19:30:06.361528+0000","last_deep_scrub":"222'6","last_deep_scrub_stamp":"2024-09-10T19:19:30.328841+0000","last_clean_scrub_stamp":"2024-09-10T19:30:06.361528+0000","objects_scrubbed":2,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:31:24.920334+0000","scrub_duration":0.0072268020000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":18,"num_read_kb":16,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":489,"reported_epoch":418,"state":"active+clean","last_fresh":"2024-09-10T19:30:42.076471+0000","last_change":"2024-09-10T19:30:20.146538+0000","last_active":"2024-09-10T19:30:42.076471+0000","last_peered":"2024-09-10T19:30:42.076471+0000","last_clean":"2024-09-10T19:30:42.076471+0000","last_became_active":"2024-09-10T19:27:53.000789+0000","last_became_peered":"2024-09-10T19:27:53.000789+0000","last_unstale":"2024-09-10T19:30:42.076471+0000","last_undegraded":"2024-09-10T19:30:42.076471+0000","last_fullsized":"2024-09-10T19:30:42.076471+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:30:20.146470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:30:20.146470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:31:21.242258+0000","scrub_duration":0.0032897199999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,1],"acting":[5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":84,"reported_epoch":418,"state":"active+clean","last_fresh":"2024-09-10T19:30:42.078627+0000","last_change":"2024-09-10T19:30:32.432843+0000","last_active":"2024-09-10T19:30:42.078627+0000","last_peered":"2024-09-10T19:30:42.078627+0000","last_clean":"2024-09-10T19:30:42.078627+0000","last_became_active":"2024-09-10T19:27:53.003591+0000","last_became_peered":"2024-09-10T19:27:53.003591+0000","last_unstale":"2024-09-10T19:30:42.078627+0000","last_undegraded":"2024-09-10T19:30:42.078627+0000","last_fullsized":"2024-09-10T19:30:42.078627+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:30:32.432739+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:11:31.837944+0000","last_clean_scrub_stamp":"2024-09-10T19:30:32.432739+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:31:40.637363+0000","scrub_duration":0.010234006,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":463,"reported_epoch":418,"state":"active+clean","last_fresh":"2024-09-10T19:30:42.084783+0000","last_change":"2024-09-10T19:29:33.126317+0000","last_active":"2024-09-10T19:30:42.084783+0000","last_peered":"2024-09-10T19:30:42.084783+0000","last_clean":"2024-09-10T19:30:42.084783+0000","last_became_active":"2024-09-10T19:16:24.852356+0000","last_became_peered":"2024-09-10T19:16:24.852356+0000","last_unstale":"2024-09-10T19:30:42.084783+0000","last_undegraded":"2024-09-10T19:30:42.084783+0000","last_fullsized":"2024-09-10T19:30:42.084783+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:29:33.125081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:29:33.125081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.019182421000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,3],"acting":[7,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.11","version":"414'27","reported_seq":285,"reported_epoch":418,"state":"active+clean","last_fresh":"2024-09-10T19:30:42.077274+0000","last_change":"2024-09-10T19:29:15.225773+0000","last_active":"2024-09-10T19:30:42.077274+0000","last_peered":"2024-09-10T19:30:42.077274+0000","last_clean":"2024-09-10T19:30:42.077274+0000","last_became_active":"2024-09-10T19:27:53.018213+0000","last_became_peered":"2024-09-10T19:27:53.018213+0000","last_unstale":"2024-09-10T19:30:42.077274+0000","last_undegraded":"2024-09-10T19:30:42.077274+0000","last_fullsized":"2024-09-10T19:30:42.077274+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"237'19","last_scrub_stamp":"2024-09-10T19:29:15.225687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:29:15.225687+0000","objects_scrubbed":5,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.015395733,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":37,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":74,"reported_epoch":418,"state":"active+clean","last_fresh":"2024-09-10T19:30:42.079743+0000","last_change":"2024-09-10T19:30:10.528365+0000","last_active":"2024-09-10T19:30:42.079743+0000","last_peered":"2024-09-10T19:30:42.079743+0000","last_clean":"2024-09-10T19:30:42.079743+0000","last_became_active":"2024-09-10T19:27:53.008197+0000","last_became_peered":"2024-09-10T19:27:53.008197+0000","last_unstale":"2024-09-10T19:30:42.079743+0000","last_undegraded":"2024-09-10T19:30:42.079743+0000","last_fullsized":"2024-09-10T19:30:42.079743+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:30:10.528284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:13:05.876655+0000","last_clean_scrub_stamp":"2024-09-10T19:30:10.528284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:31:10.861466+0000","scrub_duration":0.002704536,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":465,"reported_epoch":418,"state":"active+clean","last_fresh":"2024-09-10T19:30:42.083044+0000","last_change":"2024-09-10T19:30:11.797823+0000","last_active":"2024-09-10T19:30:42.083044+0000","last_peered":"2024-09-10T19:30:42.083044+0000","last_clean":"2024-09-10T19:30:42.083044+0000","last_became_active":"2024-09-10T19:27:53.008444+0000","last_became_peered":"2024-09-10T19:27:53.008444+0000","last_unstale":"2024-09-10T19:30:42.083044+0000","last_undegraded":"2024-09-10T19:30:42.083044+0000","last_fullsized":"2024-09-10T19:30:42.083044+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:30:11.797669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:11:57.112092+0000","last_clean_scrub_stamp":"2024-09-10T19:30:11.797669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:31:28.693832+0000","scrub_duration":0.002187483,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":460,"reported_epoch":418,"state":"active+clean","last_fresh":"2024-09-10T19:30:42.076339+0000","last_change":"2024-09-10T19:29:30.096377+0000","last_active":"2024-09-10T19:30:42.076339+0000","last_peered":"2024-09-10T19:30:42.076339+0000","last_clean":"2024-09-10T19:30:42.076339+0000","last_became_active":"2024-09-10T19:27:53.025650+0000","last_became_peered":"2024-09-10T19:27:53.025650+0000","last_unstale":"2024-09-10T19:30:42.076339+0000","last_undegraded":"2024-09-10T19:30:42.076339+0000","last_fullsized":"2024-09-10T19:30:42.076339+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:29:30.096297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:26:29.959202+0000","last_clean_scrub_stamp":"2024-09-10T19:29:30.096297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.012023613000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"80'2","reported_seq":469,"reported_epoch":418,"state":"active+clean","last_fresh":"2024-09-10T19:30:42.084237+0000","last_change":"2024-09-10T19:30:31.056345+0000","last_active":"2024-09-10T19:30:42.084237+0000","last_peered":"2024-09-10T19:30:42.084237+0000","last_clean":"2024-09-10T19:30:42.084237+0000","last_became_active":"2024-09-10T19:16:24.851220+0000","last_became_peered":"2024-09-10T19:16:24.851220+0000","last_unstale":"2024-09-10T19:30:42.084237+0000","last_undegraded":"2024-09-10T19:30:42.084237+0000","last_fullsized":"2024-09-10T19:30:42.084237+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"80'2","last_scrub_stamp":"2024-09-10T19:30:31.056271+0000","last_deep_scrub":"80'2","last_deep_scrub_stamp":"2024-09-10T19:11:30.272043+0000","last_clean_scrub_stamp":"2024-09-10T19:30:31.056271+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:31:41.406250+0000","scrub_duration":0.0025698990000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":369,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":369,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,3],"acting":[7,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":456,"reported_epoch":418,"state":"active+clean","last_fresh":"2024-09-10T19:30:42.075724+0000","last_change":"2024-09-10T19:29:18.277593+0000","last_active":"2024-09-10T19:30:42.075724+0000","last_peered":"2024-09-10T19:30:42.075724+0000","last_clean":"2024-09-10T19:30:42.075724+0000","last_became_active":"2024-09-10T19:24:23.906812+0000","last_became_peered":"2024-09-10T19:24:23.906812+0000","last_unstale":"2024-09-10T19:30:42.075724+0000","last_undegraded":"2024-09-10T19:30:42.075724+0000","last_fullsized":"2024-09-10T19:30:42.075724+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:29:18.277508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:29:18.277508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:30:46.445119+0000","scrub_duration":0.007405455,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,0],"acting":[5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1e","version":"414'35","reported_seq":447,"reported_epoch":418,"state":"active+clean","last_fresh":"2024-09-10T19:30:42.070948+0000","last_change":"2024-09-10T19:29:20.656771+0000","last_active":"2024-09-10T19:30:42.070948+0000","last_peered":"2024-09-10T19:30:42.070948+0000","last_clean":"2024-09-10T19:30:42.070948+0000","last_became_active":"2024-09-10T19:24:23.899507+0000","last_became_peered":"2024-09-10T19:24:23.899507+0000","last_unstale":"2024-09-10T19:30:42.070948+0000","last_undegraded":"2024-09-10T19:30:42.070948+0000","last_fullsized":"2024-09-10T19:30:42.070948+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"237'25","last_scrub_stamp":"2024-09-10T19:29:20.656690+0000","last_deep_scrub":"80'15","last_deep_scrub_stamp":"2024-09-10T19:11:25.883919+0000","last_clean_scrub_stamp":"2024-09-10T19:29:20.656690+0000","objects_scrubbed":5,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0034048949999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":39,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1f","version":"376'10","reported_seq":113,"reported_epoch":418,"state":"active+clean","last_fresh":"2024-09-10T19:30:45.976265+0000","last_change":"2024-09-10T19:30:22.536949+0000","last_active":"2024-09-10T19:30:45.976265+0000","last_peered":"2024-09-10T19:30:45.976265+0000","last_clean":"2024-09-10T19:30:45.976265+0000","last_became_active":"2024-09-10T19:27:53.003676+0000","last_became_peered":"2024-09-10T19:27:53.003676+0000","last_unstale":"2024-09-10T19:30:45.976265+0000","last_undegraded":"2024-09-10T19:30:45.976265+0000","last_fullsized":"2024-09-10T19:30:45.976265+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"376'10","last_scrub_stamp":"2024-09-10T19:30:22.536865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:30:22.536865+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:31:49.099743+0000","scrub_duration":0.0053077080000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":166,"reported_epoch":418,"state":"active+clean","last_fresh":"2024-09-10T19:30:42.088931+0000","last_change":"2024-09-10T19:30:25.316849+0000","last_active":"2024-09-10T19:30:42.088931+0000","last_peered":"2024-09-10T19:30:42.088931+0000","last_clean":"2024-09-10T19:30:42.088931+0000","last_became_active":"2024-09-10T19:24:01.980391+0000","last_became_peered":"2024-09-10T19:24:01.980391+0000","last_unstale":"2024-09-10T19:30:42.088931+0000","last_undegraded":"2024-09-10T19:30:42.088931+0000","last_fullsized":"2024-09-10T19:30:42.088931+0000","mapping_epoch":292,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":293,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:30:25.316711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:28:20.514115+0000","last_clean_scrub_stamp":"2024-09-10T19:30:25.316711+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:31:48.741913+0000","scrub_duration":0.0018401960000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":476,"reported_epoch":418,"state":"active+clean","last_fresh":"2024-09-10T19:30:42.076088+0000","last_change":"2024-09-10T19:29:34.161440+0000","last_active":"2024-09-10T19:30:42.076088+0000","last_peered":"2024-09-10T19:30:42.076088+0000","last_clean":"2024-09-10T19:30:42.076088+0000","last_became_active":"2024-09-10T19:27:53.017153+0000","last_became_peered":"2024-09-10T19:27:53.017153+0000","last_unstale":"2024-09-10T19:30:42.076088+0000","last_undegraded":"2024-09-10T19:30:42.076088+0000","last_fullsized":"2024-09-10T19:30:42.076088+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:29:34.161296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:29:34.161296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:30:50.439084+0000","scrub_duration":0.005093261,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"414'32","reported_seq":98,"reported_epoch":418,"state":"active+clean","last_fresh":"2024-09-10T19:30:42.079666+0000","last_change":"2024-09-10T19:29:28.357694+0000","last_active":"2024-09-10T19:30:42.079666+0000","last_peered":"2024-09-10T19:30:42.079666+0000","last_clean":"2024-09-10T19:30:42.079666+0000","last_became_active":"2024-09-10T19:27:53.008233+0000","last_became_peered":"2024-09-10T19:27:53.008233+0000","last_unstale":"2024-09-10T19:30:42.079666+0000","last_undegraded":"2024-09-10T19:30:42.079666+0000","last_fullsized":"2024-09-10T19:30:42.079666+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"237'24","last_scrub_stamp":"2024-09-10T19:29:28.357607+0000","last_deep_scrub":"80'16","last_deep_scrub_stamp":"2024-09-10T19:11:40.853611+0000","last_clean_scrub_stamp":"2024-09-10T19:29:28.357607+0000","objects_scrubbed":5,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0067797259999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":36,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":384,"reported_epoch":418,"state":"active+clean","last_fresh":"2024-09-10T19:30:42.070374+0000","last_change":"2024-09-10T19:30:24.834659+0000","last_active":"2024-09-10T19:30:42.070374+0000","last_peered":"2024-09-10T19:30:42.070374+0000","last_clean":"2024-09-10T19:30:42.070374+0000","last_became_active":"2024-09-10T19:24:24.899153+0000","last_became_peered":"2024-09-10T19:24:24.899153+0000","last_unstale":"2024-09-10T19:30:42.070374+0000","last_undegraded":"2024-09-10T19:30:42.070374+0000","last_fullsized":"2024-09-10T19:30:42.070374+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:30:24.834494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:28:19.464426+0000","last_clean_scrub_stamp":"2024-09-10T19:30:24.834494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:31:26.567702+0000","scrub_duration":0.0076205279999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":472,"reported_epoch":418,"state":"active+clean","last_fresh":"2024-09-10T19:30:42.075900+0000","last_change":"2024-09-10T19:30:30.520734+0000","last_active":"2024-09-10T19:30:42.075900+0000","last_peered":"2024-09-10T19:30:42.075900+0000","last_clean":"2024-09-10T19:30:42.075900+0000","last_became_active":"2024-09-10T19:27:53.011336+0000","last_became_peered":"2024-09-10T19:27:53.011336+0000","last_unstale":"2024-09-10T19:30:42.075900+0000","last_undegraded":"2024-09-10T19:30:42.075900+0000","last_fullsized":"2024-09-10T19:30:42.075900+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:30:30.520643+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:30:30.520643+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:31:57.292887+0000","scrub_duration":0.018058100000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":74,"reported_epoch":418,"state":"active+clean","last_fresh":"2024-09-10T19:30:42.078549+0000","last_change":"2024-09-10T19:29:33.401343+0000","last_active":"2024-09-10T19:30:42.078549+0000","last_peered":"2024-09-10T19:30:42.078549+0000","last_clean":"2024-09-10T19:30:42.078549+0000","last_became_active":"2024-09-10T19:27:53.002313+0000","last_became_peered":"2024-09-10T19:27:53.002313+0000","last_unstale":"2024-09-10T19:30:42.078549+0000","last_undegraded":"2024-09-10T19:30:42.078549+0000","last_fullsized":"2024-09-10T19:30:42.078549+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:29:33.401164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:29:33.401164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:31:02.399310+0000","scrub_duration":0.016867561,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1c","version":"414'46","reported_seq":214,"reported_epoch":418,"state":"active+clean","last_fresh":"2024-09-10T19:30:42.083923+0000","last_change":"2024-09-10T19:29:34.532076+0000","last_active":"2024-09-10T19:30:42.083923+0000","last_peered":"2024-09-10T19:30:42.083923+0000","last_clean":"2024-09-10T19:30:42.083923+0000","last_became_active":"2024-09-10T19:24:24.910577+0000","last_became_peered":"2024-09-10T19:24:24.910577+0000","last_unstale":"2024-09-10T19:30:42.083923+0000","last_undegraded":"2024-09-10T19:30:42.083923+0000","last_fullsized":"2024-09-10T19:30:42.083923+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"237'34","last_scrub_stamp":"2024-09-10T19:29:34.531993+0000","last_deep_scrub":"78'22","last_deep_scrub_stamp":"2024-09-10T19:11:21.118937+0000","last_clean_scrub_stamp":"2024-09-10T19:29:34.531993+0000","objects_scrubbed":7,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:30:47.686023+0000","scrub_duration":0.0018029159999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":62,"num_read_kb":53,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":180,"reported_epoch":418,"state":"active+clean","last_fresh":"2024-09-10T19:30:42.089040+0000","last_change":"2024-09-10T19:29:21.494875+0000","last_active":"2024-09-10T19:30:42.089040+0000","last_peered":"2024-09-10T19:30:42.089040+0000","last_clean":"2024-09-10T19:30:42.089040+0000","last_became_active":"2024-09-10T19:24:24.887205+0000","last_became_peered":"2024-09-10T19:24:24.887205+0000","last_unstale":"2024-09-10T19:30:42.089040+0000","last_undegraded":"2024-09-10T19:30:42.089040+0000","last_fullsized":"2024-09-10T19:30:42.089040+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:29:21.494801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:29:21.494801+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.011816722,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":456,"reported_epoch":418,"state":"active+clean","last_fresh":"2024-09-10T19:30:42.077136+0000","last_change":"2024-09-10T19:29:25.182733+0000","last_active":"2024-09-10T19:30:42.077136+0000","last_peered":"2024-09-10T19:30:42.077136+0000","last_clean":"2024-09-10T19:30:42.077136+0000","last_became_active":"2024-09-10T19:27:53.026591+0000","last_became_peered":"2024-09-10T19:27:53.026591+0000","last_unstale":"2024-09-10T19:30:42.077136+0000","last_undegraded":"2024-09-10T19:30:42.077136+0000","last_fullsized":"2024-09-10T19:30:42.077136+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:29:25.182654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:24:46.518816+0000","last_clean_scrub_stamp":"2024-09-10T19:29:25.182654+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0060637180000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":469,"reported_epoch":418,"state":"active+clean","last_fresh":"2024-09-10T19:30:42.076750+0000","last_change":"2024-09-10T19:30:24.721276+0000","last_active":"2024-09-10T19:30:42.076750+0000","last_peered":"2024-09-10T19:30:42.076750+0000","last_clean":"2024-09-10T19:30:42.076750+0000","last_became_active":"2024-09-10T19:27:53.018726+0000","last_became_peered":"2024-09-10T19:27:53.018726+0000","last_unstale":"2024-09-10T19:30:42.076750+0000","last_undegraded":"2024-09-10T19:30:42.076750+0000","last_fullsized":"2024-09-10T19:30:42.076750+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:30:24.721140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:30:24.721140+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:31:33.177185+0000","scrub_duration":0.0073255109999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1d","version":"414'29","reported_seq":534,"reported_epoch":418,"state":"active+clean","last_fresh":"2024-09-10T19:30:42.076412+0000","last_change":"2024-09-10T19:30:28.765125+0000","last_active":"2024-09-10T19:30:42.076412+0000","last_peered":"2024-09-10T19:30:42.076412+0000","last_clean":"2024-09-10T19:30:42.076412+0000","last_became_active":"2024-09-10T19:24:01.976664+0000","last_became_peered":"2024-09-10T19:24:01.976664+0000","last_unstale":"2024-09-10T19:30:42.076412+0000","last_undegraded":"2024-09-10T19:30:42.076412+0000","last_fullsized":"2024-09-10T19:30:42.076412+0000","mapping_epoch":292,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":293,"parent":"0.0","parent_split_bits":0,"last_scrub":"414'29","last_scrub_stamp":"2024-09-10T19:30:28.765025+0000","last_deep_scrub":"250'23","last_deep_scrub_stamp":"2024-09-10T19:20:55.582173+0000","last_clean_scrub_stamp":"2024-09-10T19:30:28.765025+0000","objects_scrubbed":5,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:31:57.061451+0000","scrub_duration":0.012181088,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":464,"reported_epoch":418,"state":"active+clean","last_fresh":"2024-09-10T19:30:42.074772+0000","last_change":"2024-09-10T19:30:39.493327+0000","last_active":"2024-09-10T19:30:42.074772+0000","last_peered":"2024-09-10T19:30:42.074772+0000","last_clean":"2024-09-10T19:30:42.074772+0000","last_became_active":"2024-09-10T19:16:24.843013+0000","last_became_peered":"2024-09-10T19:16:24.843013+0000","last_unstale":"2024-09-10T19:30:42.074772+0000","last_undegraded":"2024-09-10T19:30:42.074772+0000","last_fullsized":"2024-09-10T19:30:42.074772+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:30:39.493257+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:30:39.493257+0000","last_clean_scrub_stamp":"2024-09-10T19:30:39.493257+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:32:02.390742+0000","scrub_duration":0.0032639090000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":11,"num_pg":16,"stat_sum":{"num_bytes":1441202200,"num_objects":21992,"num_object_clones":0,"num_object_copies":65976,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21992,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":267784,"num_write_kb":9272769,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4242747392,"data_stored":4242735176,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":70310,"ondisk_log_size":70310,"up":48,"acting":48,"num_store_stats":7},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":382,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":1528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":96,"ondisk_log_size":96,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":177,"num_object_clones":0,"num_object_copies":531,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":177,"num_whiteouts":0,"num_read":1544,"num_read_kb":1301,"num_write":989,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":433,"num_bytes_recovered":8406,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":458752,"data_stored":12124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1066,"ondisk_log_size":1066,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":412,"num_read_kb":407,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1594,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1613,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":10,"ondisk_log_size":10,"up":9,"acting":9,"num_store_stats":7},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1328,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":21,"num_read_kb":21,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":724,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":3984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5,"ondisk_log_size":5,"up":96,"acting":96,"num_store_stats":8},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":525376,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":870,"ondisk_log_size":870,"up":33,"acting":33,"num_store_stats":7}],"osd_stats":[{"osd":7,"up_from":54,"seq":231928234237,"num_pgs":71,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1923204,"kb_used_data":738364,"kb_used_omap":1,"kb_used_meta":1184830,"kb_avail":91821948,"statfs":{"total":95995035648,"available":94025674752,"internally_reserved":0,"allocated":756084736,"data_stored":755749158,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":1213266264},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[2,0,1,4,4],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 19:30:32 2024","interfaces":[{"interface":"back","average":{"1min":1.2030000000000001,"5min":0.98599999999999999,"15min":0.97499999999999998},"min":{"1min":0.30499999999999999,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":3.27,"5min":5.0510000000000002,"15min":5.0510000000000002},"last":1.8240000000000001},{"interface":"front","average":{"1min":1.304,"5min":1.103,"15min":0.82699999999999996},"min":{"1min":0.23400000000000001,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":3.2570000000000001,"5min":4.7190000000000003,"15min":4.7190000000000003},"last":1.121}]},{"osd":1,"last update":"Tue Sep 10 19:30:01 2024","interfaces":[{"interface":"back","average":{"1min":1.4950000000000001,"5min":1.2729999999999999,"15min":1.0880000000000001},"min":{"1min":0.35599999999999998,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":4.6050000000000004,"5min":12.773999999999999,"15min":12.773999999999999},"last":1.8819999999999999},{"interface":"front","average":{"1min":1.155,"5min":1.2430000000000001,"15min":0.96699999999999997},"min":{"1min":0.38600000000000001,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":3.6019999999999999,"5min":12.837999999999999,"15min":12.837999999999999},"last":1.544}]},{"osd":2,"last update":"Tue Sep 10 19:30:19 2024","interfaces":[{"interface":"back","average":{"1min":1.3879999999999999,"5min":0.996,"15min":0.92300000000000004},"min":{"1min":0.26600000000000001,"5min":0.26600000000000001,"15min":0.23000000000000001},"max":{"1min":3.3100000000000001,"5min":4.7400000000000002,"15min":4.7400000000000002},"last":1.1000000000000001},{"interface":"front","average":{"1min":1.327,"5min":1.1060000000000001,"15min":1.04},"min":{"1min":0.37,"5min":0.29099999999999998,"15min":0.26300000000000001},"max":{"1min":6.1059999999999999,"5min":6.1059999999999999,"15min":8.0839999999999996},"last":1.5960000000000001}]},{"osd":3,"last update":"Tue Sep 10 19:30:19 2024","interfaces":[{"interface":"back","average":{"1min":2.0640000000000001,"5min":1.3720000000000001,"15min":1.204},"min":{"1min":0.32800000000000001,"5min":0.25900000000000001,"15min":0.20200000000000001},"max":{"1min":14.515000000000001,"5min":14.515000000000001,"15min":21.074999999999999},"last":1.073},{"interface":"front","average":{"1min":1.915,"5min":1.3560000000000001,"15min":1.1839999999999999},"min":{"1min":0.374,"5min":0.27700000000000002,"15min":0.25700000000000001},"max":{"1min":14.227,"5min":14.227,"15min":18.82},"last":1.5680000000000001}]},{"osd":4,"last update":"Tue Sep 10 19:30:19 2024","interfaces":[{"interface":"back","average":{"1min":1.496,"5min":1.716,"15min":2.677},"min":{"1min":0.35399999999999998,"5min":0.24399999999999999,"15min":0.222},"max":{"1min":5.194,"5min":18.664999999999999,"15min":231.923},"last":1.855},{"interface":"front","average":{"1min":1.488,"5min":1.466,"15min":2.3479999999999999},"min":{"1min":0.29599999999999999,"5min":0.25600000000000001,"15min":0.23100000000000001},"max":{"1min":6.6029999999999998,"5min":14.837,"15min":231.322},"last":3.161}]},{"osd":5,"last update":"Tue Sep 10 19:30:19 2024","interfaces":[{"interface":"back","average":{"1min":1.1639999999999999,"5min":1.1850000000000001,"15min":1.6339999999999999},"min":{"1min":0.39000000000000001,"5min":0.25800000000000001,"15min":0.24099999999999999},"max":{"1min":3.5750000000000002,"5min":8.8300000000000001,"15min":67.382999999999996},"last":2.2069999999999999},{"interface":"front","average":{"1min":1.77,"5min":1.5549999999999999,"15min":1.99},"min":{"1min":0.33700000000000002,"5min":0.309,"15min":0.22800000000000001},"max":{"1min":6.7939999999999996,"5min":8.7040000000000006,"15min":67.75},"last":6.1559999999999997}]},{"osd":6,"last update":"Tue Sep 10 19:30:19 2024","interfaces":[{"interface":"back","average":{"1min":2.0139999999999998,"5min":1.5980000000000001,"15min":2.4079999999999999},"min":{"1min":0.29799999999999999,"5min":0.29799999999999999,"15min":0.28000000000000003},"max":{"1min":6.7530000000000001,"5min":6.7869999999999999,"15min":181.53800000000001},"last":1.907},{"interface":"front","average":{"1min":1.591,"5min":1.393,"15min":2.101},"min":{"1min":0.34399999999999997,"5min":0.318,"15min":0.27000000000000002},"max":{"1min":6.6349999999999998,"5min":9.7110000000000003,"15min":182.05600000000001},"last":1.484}]}]},{"osd":6,"up_from":47,"seq":201863463170,"num_pgs":66,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1648980,"kb_used_data":762508,"kb_used_omap":1,"kb_used_meta":886462,"kb_avail":92096172,"statfs":{"total":95995035648,"available":94306480128,"internally_reserved":0,"allocated":780808192,"data_stored":780457342,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":907737432},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 19:30:39 2024","interfaces":[{"interface":"back","average":{"1min":7.5650000000000004,"5min":2.6040000000000001,"15min":2.117},"min":{"1min":0.26800000000000002,"5min":0.183,"15min":0.17899999999999999},"max":{"1min":127.395,"5min":127.395,"15min":127.395},"last":0.51400000000000001},{"interface":"front","average":{"1min":7.6219999999999999,"5min":2.5390000000000001,"15min":2.1789999999999998},"min":{"1min":0.20599999999999999,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":127.547,"5min":127.547,"15min":127.547},"last":0.26600000000000001}]},{"osd":1,"last update":"Tue Sep 10 19:30:02 2024","interfaces":[{"interface":"back","average":{"1min":1.8859999999999999,"5min":1.6200000000000001,"15min":1.0009999999999999},"min":{"1min":0.315,"5min":0.20999999999999999,"15min":0.19800000000000001},"max":{"1min":5.7320000000000002,"5min":7.9039999999999999,"15min":7.9039999999999999},"last":0.318},{"interface":"front","average":{"1min":1.5129999999999999,"5min":1.29,"15min":0.93200000000000005},"min":{"1min":0.21199999999999999,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":5.6799999999999997,"5min":7.9189999999999996,"15min":7.9189999999999996},"last":0.33400000000000002}]},{"osd":2,"last update":"Tue Sep 10 19:30:10 2024","interfaces":[{"interface":"back","average":{"1min":1.258,"5min":1.1459999999999999,"15min":0.91400000000000003},"min":{"1min":0.24299999999999999,"5min":0.24299999999999999,"15min":0.23799999999999999},"max":{"1min":4.8330000000000002,"5min":5.8010000000000002,"15min":5.9029999999999996},"last":0.378},{"interface":"front","average":{"1min":1.218,"5min":1.1259999999999999,"15min":0.95999999999999996},"min":{"1min":0.36799999999999999,"5min":0.25600000000000001,"15min":0.254},"max":{"1min":5.6929999999999996,"5min":6.4560000000000004,"15min":6.4560000000000004},"last":0.28899999999999998}]},{"osd":3,"last update":"Tue Sep 10 19:30:10 2024","interfaces":[{"interface":"back","average":{"1min":1.4390000000000001,"5min":1.819,"15min":1.321},"min":{"1min":0.316,"5min":0.27900000000000003,"15min":0.20399999999999999},"max":{"1min":4.7290000000000001,"5min":35.847000000000001,"15min":35.847000000000001},"last":1.2350000000000001},{"interface":"front","average":{"1min":1.647,"5min":1.643,"15min":1.27},"min":{"1min":0.29899999999999999,"5min":0.29199999999999998,"15min":0.20499999999999999},"max":{"1min":5.6420000000000003,"5min":35.862000000000002,"15min":35.862000000000002},"last":0.61299999999999999}]},{"osd":4,"last update":"Tue Sep 10 19:30:10 2024","interfaces":[{"interface":"back","average":{"1min":1.429,"5min":1.9990000000000001,"15min":1.47},"min":{"1min":0.35499999999999998,"5min":0.22,"15min":0.22},"max":{"1min":5.6150000000000002,"5min":35.552,"15min":35.552},"last":0.34999999999999998},{"interface":"front","average":{"1min":1.371,"5min":2.117,"15min":1.4239999999999999},"min":{"1min":0.30299999999999999,"5min":0.24399999999999999,"15min":0.22500000000000001},"max":{"1min":4.5069999999999997,"5min":35.192999999999998,"15min":35.192999999999998},"last":0.42999999999999999}]},{"osd":5,"last update":"Tue Sep 10 19:30:10 2024","interfaces":[{"interface":"back","average":{"1min":1.282,"5min":1.909,"15min":1.484},"min":{"1min":0.40799999999999997,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":7.4050000000000002,"5min":48.216000000000001,"15min":48.216000000000001},"last":0.40200000000000002},{"interface":"front","average":{"1min":1.2529999999999999,"5min":2.3130000000000002,"15min":1.659},"min":{"1min":0.40100000000000002,"5min":0.254,"15min":0.254},"max":{"1min":4.3719999999999999,"5min":50.372999999999998,"15min":50.372999999999998},"last":0.46700000000000003}]},{"osd":7,"last update":"Tue Sep 10 19:30:39 2024","interfaces":[{"interface":"back","average":{"1min":0.94899999999999995,"5min":1.5209999999999999,"15min":1.5940000000000001},"min":{"1min":0.376,"5min":0.28299999999999997,"15min":0.28100000000000003},"max":{"1min":1.986,"5min":7.609,"15min":80.629999999999995},"last":0.36299999999999999},{"interface":"front","average":{"1min":1.2749999999999999,"5min":1.855,"15min":1.8680000000000001},"min":{"1min":0.35299999999999998,"5min":0.28299999999999997,"15min":0.25700000000000001},"max":{"1min":3.9710000000000001,"5min":16.654,"15min":81.088999999999999},"last":0.72699999999999998}]}]},{"osd":5,"up_from":39,"seq":167503724807,"num_pgs":65,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1259148,"kb_used_data":365380,"kb_used_omap":1,"kb_used_meta":893758,"kb_avail":92486004,"statfs":{"total":95995035648,"available":94705668096,"internally_reserved":0,"allocated":374149120,"data_stored":373807110,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":915208536},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,1,3],"upper_bound":16},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 19:30:33 2024","interfaces":[{"interface":"back","average":{"1min":1.278,"5min":1.0580000000000001,"15min":1.1000000000000001},"min":{"1min":0.223,"5min":0.19,"15min":0.187},"max":{"1min":7.5759999999999996,"5min":7.5759999999999996,"15min":7.5759999999999996},"last":0.41599999999999998},{"interface":"front","average":{"1min":1.081,"5min":0.94899999999999995,"15min":1.175},"min":{"1min":0.20599999999999999,"5min":0.20599999999999999,"15min":0.20599999999999999},"max":{"1min":5.819,"5min":5.819,"15min":5.819},"last":0.253}]},{"osd":1,"last update":"Tue Sep 10 19:29:50 2024","interfaces":[{"interface":"back","average":{"1min":3.0459999999999998,"5min":2.5699999999999998,"15min":1.4930000000000001},"min":{"1min":0.247,"5min":0.214,"15min":0.214},"max":{"1min":35.107999999999997,"5min":54.097999999999999,"15min":54.097999999999999},"last":0.90000000000000002},{"interface":"front","average":{"1min":2.5659999999999998,"5min":2.5089999999999999,"15min":1.5129999999999999},"min":{"1min":0.26300000000000001,"5min":0.221,"15min":0.221},"max":{"1min":35.173000000000002,"5min":54.337000000000003,"15min":54.337000000000003},"last":0.499}]},{"osd":2,"last update":"Tue Sep 10 19:30:39 2024","interfaces":[{"interface":"back","average":{"1min":1.0629999999999999,"5min":1.0089999999999999,"15min":1.1060000000000001},"min":{"1min":0.32800000000000001,"5min":0.26500000000000001,"15min":0.22900000000000001},"max":{"1min":7.4489999999999998,"5min":7.4489999999999998,"15min":9.9399999999999995},"last":0.46700000000000003},{"interface":"front","average":{"1min":0.754,"5min":0.98699999999999999,"15min":1.052},"min":{"1min":0.28599999999999998,"5min":0.27700000000000002,"15min":0.21199999999999999},"max":{"1min":2.8889999999999998,"5min":10.045999999999999,"15min":10.045999999999999},"last":0.314}]},{"osd":3,"last update":"Tue Sep 10 19:30:39 2024","interfaces":[{"interface":"back","average":{"1min":1.2190000000000001,"5min":2.5089999999999999,"15min":1.696},"min":{"1min":0.26600000000000001,"5min":0.254,"15min":0.19900000000000001},"max":{"1min":7.5979999999999999,"5min":54.832000000000001,"15min":54.832000000000001},"last":0.52400000000000002},{"interface":"front","average":{"1min":1.2150000000000001,"5min":2.484,"15min":1.659},"min":{"1min":0.28100000000000003,"5min":0.28100000000000003,"15min":0.246},"max":{"1min":7.3520000000000003,"5min":54.850999999999999,"15min":54.850999999999999},"last":0.34499999999999997}]},{"osd":4,"last update":"Tue Sep 10 19:30:39 2024","interfaces":[{"interface":"back","average":{"1min":1.0780000000000001,"5min":2.3799999999999999,"15min":2.202},"min":{"1min":0.26100000000000001,"5min":0.252,"15min":0.24399999999999999},"max":{"1min":5.6929999999999996,"5min":132.41900000000001,"15min":132.41900000000001},"last":0.83399999999999996},{"interface":"front","average":{"1min":1.1599999999999999,"5min":2.48,"15min":2.3119999999999998},"min":{"1min":0.29099999999999998,"5min":0.28699999999999998,"15min":0.217},"max":{"1min":7.5519999999999996,"5min":132.46700000000001,"15min":132.46700000000001},"last":0.56399999999999995}]},{"osd":6,"last update":"Tue Sep 10 19:30:13 2024","interfaces":[{"interface":"back","average":{"1min":0.83599999999999997,"5min":1.4019999999999999,"15min":1.8120000000000001},"min":{"1min":0.28699999999999998,"5min":0.28599999999999998,"15min":0.28399999999999997},"max":{"1min":2.4199999999999999,"5min":8.3339999999999996,"15min":111.328},"last":0.54600000000000004},{"interface":"front","average":{"1min":1.135,"5min":1.399,"15min":1.762},"min":{"1min":0.28299999999999997,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":4.1219999999999999,"5min":7.2910000000000004,"15min":111.705},"last":0.57499999999999996}]},{"osd":7,"last update":"Tue Sep 10 19:30:30 2024","interfaces":[{"interface":"back","average":{"1min":1.619,"5min":1.27,"15min":1.383},"min":{"1min":0.35099999999999998,"5min":0.29099999999999998,"15min":0.28599999999999998},"max":{"1min":4.5140000000000002,"5min":4.5140000000000002,"15min":6.7869999999999999},"last":0.59899999999999998},{"interface":"front","average":{"1min":1.2609999999999999,"5min":1.1739999999999999,"15min":1.357},"min":{"1min":0.30099999999999999,"5min":0.29899999999999999,"15min":0.27100000000000002},"max":{"1min":3.9100000000000001,"5min":5.6349999999999998,"15min":9.4909999999999997},"last":4.8620000000000001}]}]},{"osd":4,"up_from":32,"seq":137438953739,"num_pgs":65,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1351372,"kb_used_data":462980,"kb_used_omap":1,"kb_used_meta":888382,"kb_avail":92393780,"statfs":{"total":95995035648,"available":94611230720,"internally_reserved":0,"allocated":474091520,"data_stored":473747625,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":909703512},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[4,1,3,1,1],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 19:30:31 2024","interfaces":[{"interface":"back","average":{"1min":0.90400000000000003,"5min":1.506,"15min":2.0299999999999998},"min":{"1min":0.21199999999999999,"5min":0.20100000000000001,"15min":0.20100000000000001},"max":{"1min":3.6110000000000002,"5min":39.097000000000001,"15min":39.097000000000001},"last":2.1600000000000001},{"interface":"front","average":{"1min":0.50600000000000001,"5min":1.6299999999999999,"15min":2.2919999999999998},"min":{"1min":0.24099999999999999,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":1.1599999999999999,"5min":39.112000000000002,"15min":39.112000000000002},"last":0.47799999999999998}]},{"osd":1,"last update":"Tue Sep 10 19:29:52 2024","interfaces":[{"interface":"back","average":{"1min":1.5389999999999999,"5min":1.462,"15min":0.94899999999999995},"min":{"1min":0.29899999999999999,"5min":0.20599999999999999,"15min":0.20599999999999999},"max":{"1min":6.1870000000000003,"5min":6.1870000000000003,"15min":6.1870000000000003},"last":0.504},{"interface":"front","average":{"1min":1.212,"5min":1.153,"15min":0.78600000000000003},"min":{"1min":0.316,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":3.1539999999999999,"5min":5.8680000000000003,"15min":5.8680000000000003},"last":0.308}]},{"osd":2,"last update":"Tue Sep 10 19:30:25 2024","interfaces":[{"interface":"back","average":{"1min":0.622,"5min":1.1479999999999999,"15min":1.016},"min":{"1min":0.29399999999999998,"5min":0.27000000000000002,"15min":0.254},"max":{"1min":2.6040000000000001,"5min":5.5609999999999999,"15min":6.8869999999999996},"last":0.54700000000000004},{"interface":"front","average":{"1min":1.085,"5min":1.1619999999999999,"15min":1.002},"min":{"1min":0.29299999999999998,"5min":0.28299999999999997,"15min":0.253},"max":{"1min":3.2120000000000002,"5min":6.7690000000000001,"15min":7.085},"last":0.36199999999999999}]},{"osd":3,"last update":"Tue Sep 10 19:30:25 2024","interfaces":[{"interface":"back","average":{"1min":1.161,"5min":1.1910000000000001,"15min":1.2150000000000001},"min":{"1min":0.28899999999999998,"5min":0.27800000000000002,"15min":0.20999999999999999},"max":{"1min":3.1880000000000002,"5min":6.8019999999999996,"15min":35.451999999999998},"last":0.33500000000000002},{"interface":"front","average":{"1min":1.157,"5min":1.1619999999999999,"15min":1.216},"min":{"1min":0.28499999999999998,"5min":0.28499999999999998,"15min":0.246},"max":{"1min":3.5329999999999999,"5min":5.8949999999999996,"15min":32.880000000000003},"last":2.2610000000000001}]},{"osd":5,"last update":"Tue Sep 10 19:30:45 2024","interfaces":[{"interface":"back","average":{"1min":1.2769999999999999,"5min":1.177,"15min":2.6030000000000002},"min":{"1min":0.30399999999999999,"5min":0.26800000000000002,"15min":0.224},"max":{"1min":3.5840000000000001,"5min":5.9859999999999998,"15min":227.726},"last":2.238},{"interface":"front","average":{"1min":0.80700000000000005,"5min":1.3180000000000001,"15min":2.589},"min":{"1min":0.28799999999999998,"5min":0.27400000000000002,"15min":0.20699999999999999},"max":{"1min":3.7040000000000002,"5min":5.6699999999999999,"15min":227.70500000000001},"last":0.52700000000000002}]},{"osd":6,"last update":"Tue Sep 10 19:30:25 2024","interfaces":[{"interface":"back","average":{"1min":1.0129999999999999,"5min":1.554,"15min":1.2629999999999999},"min":{"1min":0.34200000000000003,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":3.6259999999999999,"5min":8.1010000000000009,"15min":8.1010000000000009},"last":0.59299999999999997},{"interface":"front","average":{"1min":1.6599999999999999,"5min":1.78,"15min":1.3420000000000001},"min":{"1min":0.254,"5min":0.254,"15min":0.247},"max":{"1min":6.0099999999999998,"5min":16.815000000000001,"15min":16.815000000000001},"last":2.218}]},{"osd":7,"last update":"Tue Sep 10 19:30:31 2024","interfaces":[{"interface":"back","average":{"1min":1.1839999999999999,"5min":1.3660000000000001,"15min":1.3240000000000001},"min":{"1min":0.307,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":3.8319999999999999,"5min":16.853999999999999,"15min":16.853999999999999},"last":2.198},{"interface":"front","average":{"1min":1.4590000000000001,"5min":1.4730000000000001,"15min":1.3899999999999999},"min":{"1min":0.35299999999999998,"5min":0.25700000000000001,"15min":0.23699999999999999},"max":{"1min":4.6020000000000003,"5min":10.02,"15min":10.02},"last":2.8399999999999999}]}]},{"osd":3,"up_from":26,"seq":111669149968,"num_pgs":73,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1815652,"kb_used_data":639452,"kb_used_omap":1,"kb_used_meta":1176190,"kb_avail":91929500,"statfs":{"total":95995035648,"available":94135808000,"internally_reserved":0,"allocated":654798848,"data_stored":654432840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":1204418904},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,2,0,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 19:30:38 2024","interfaces":[{"interface":"back","average":{"1min":8.7970000000000006,"5min":3.673,"15min":1.5800000000000001},"min":{"1min":0.16800000000000001,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":117.655,"5min":117.655,"15min":117.655},"last":0.26000000000000001},{"interface":"front","average":{"1min":8.6489999999999991,"5min":3.6440000000000001,"15min":1.5469999999999999},"min":{"1min":0.21199999999999999,"5min":0.184,"15min":0.184},"max":{"1min":117.81399999999999,"5min":117.81399999999999,"15min":117.81399999999999},"last":0.28100000000000003}]},{"osd":1,"last update":"Tue Sep 10 19:30:02 2024","interfaces":[{"interface":"back","average":{"1min":0.94099999999999995,"5min":0.72299999999999998,"15min":0.44500000000000001},"min":{"1min":0.20100000000000001,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":2.5979999999999999,"5min":5.1779999999999999,"15min":5.1779999999999999},"last":0.53100000000000003},{"interface":"front","average":{"1min":1.4370000000000001,"5min":0.77600000000000002,"15min":0.47199999999999998},"min":{"1min":0.245,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":4.0830000000000002,"5min":4.4100000000000001,"15min":4.4100000000000001},"last":0.48199999999999998}]},{"osd":2,"last update":"Tue Sep 10 19:30:08 2024","interfaces":[{"interface":"back","average":{"1min":0.66900000000000004,"5min":0.58499999999999996,"15min":0.58399999999999996},"min":{"1min":0.35599999999999998,"5min":0.246,"15min":0.17599999999999999},"max":{"1min":1.827,"5min":4.3769999999999998,"15min":5.3029999999999999},"last":0.34699999999999998},{"interface":"front","average":{"1min":0.75700000000000001,"5min":0.64100000000000001,"15min":0.626},"min":{"1min":0.248,"5min":0.23699999999999999,"15min":0.16400000000000001},"max":{"1min":2.3980000000000001,"5min":4.2990000000000004,"15min":17.806000000000001},"last":0.51000000000000001}]},{"osd":4,"last update":"Tue Sep 10 19:30:28 2024","interfaces":[{"interface":"back","average":{"1min":1.2949999999999999,"5min":1.01,"15min":1.3360000000000001},"min":{"1min":0.30099999999999999,"5min":0.26900000000000002,"15min":0.254},"max":{"1min":4.1159999999999997,"5min":4.4169999999999998,"15min":81.334000000000003},"last":0.61599999999999999},{"interface":"front","average":{"1min":1.0589999999999999,"5min":1.0740000000000001,"15min":1.429},"min":{"1min":0.318,"5min":0.27900000000000003,"15min":0.22700000000000001},"max":{"1min":3.1720000000000002,"5min":4.343,"15min":81.381},"last":2.21}]},{"osd":5,"last update":"Tue Sep 10 19:30:48 2024","interfaces":[{"interface":"back","average":{"1min":1.153,"5min":1.1559999999999999,"15min":1.7330000000000001},"min":{"1min":0.40400000000000003,"5min":0.28000000000000003,"15min":0.27900000000000003},"max":{"1min":3.6600000000000001,"5min":6.7240000000000002,"15min":104.672},"last":1.0920000000000001},{"interface":"front","average":{"1min":1.4039999999999999,"5min":1.159,"15min":1.722},"min":{"1min":0.314,"5min":0.28799999999999998,"15min":0.24299999999999999},"max":{"1min":3.4900000000000002,"5min":4.6550000000000002,"15min":105.971},"last":1.8939999999999999}]},{"osd":6,"last update":"Tue Sep 10 19:30:08 2024","interfaces":[{"interface":"back","average":{"1min":1.2470000000000001,"5min":2.8380000000000001,"15min":2.1709999999999998},"min":{"1min":0.34100000000000003,"5min":0.33600000000000002,"15min":0.23699999999999999},"max":{"1min":2.4340000000000002,"5min":148.09299999999999,"15min":148.09299999999999},"last":0.83099999999999996},{"interface":"front","average":{"1min":1.137,"5min":2.6659999999999999,"15min":2.0819999999999999},"min":{"1min":0.40200000000000002,"5min":0.32700000000000001,"15min":0.26800000000000002},"max":{"1min":3.6659999999999999,"5min":148.31999999999999,"15min":148.31999999999999},"last":0.996}]},{"osd":7,"last update":"Tue Sep 10 19:30:48 2024","interfaces":[{"interface":"back","average":{"1min":1.716,"5min":1.175,"15min":1.4530000000000001},"min":{"1min":0.65300000000000002,"5min":0.33600000000000002,"15min":0.26100000000000001},"max":{"1min":6.8639999999999999,"5min":6.8639999999999999,"15min":104.749},"last":1.7749999999999999},{"interface":"front","average":{"1min":1.1830000000000001,"5min":1.056,"15min":1.3859999999999999},"min":{"1min":0.55900000000000005,"5min":0.33100000000000002,"15min":0.26100000000000001},"max":{"1min":2.3300000000000001,"5min":4.6849999999999996,"15min":104.69499999999999},"last":1.673}]}]},{"osd":2,"up_from":0,"seq":77309411605,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":276,"seq":1185410973795,"num_pgs":69,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1094304,"kb_used_data":780120,"kb_used_omap":6,"kb_used_meta":314169,"kb_avail":92650848,"statfs":{"total":95995035648,"available":94874468352,"internally_reserved":0,"allocated":798842880,"data_stored":798479433,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6335,"internal_metadata":321709889},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,4],"upper_bound":256},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 19:30:39 2024","interfaces":[{"interface":"back","average":{"1min":1.2210000000000001,"5min":0.79000000000000004,"15min":0.58799999999999997},"min":{"1min":0.35999999999999999,"5min":0.16700000000000001,"15min":0.16700000000000001},"max":{"1min":3.5990000000000002,"5min":3.5990000000000002,"15min":3.5990000000000002},"last":0.64300000000000002},{"interface":"front","average":{"1min":1.1699999999999999,"5min":0.90700000000000003,"15min":0.57399999999999995},"min":{"1min":0.192,"5min":0.192,"15min":0.192},"max":{"1min":3.9449999999999998,"5min":4.5739999999999998,"15min":4.5739999999999998},"last":3.532}]},{"osd":2,"last update":"Tue Sep 10 19:29:59 2024","interfaces":[{"interface":"back","average":{"1min":0.997,"5min":0.68799999999999994,"15min":0.42799999999999999},"min":{"1min":0.313,"5min":0.23499999999999999,"15min":0.19},"max":{"1min":2.407,"5min":4.1779999999999999,"15min":4.1779999999999999},"last":0.66800000000000004},{"interface":"front","average":{"1min":1.1459999999999999,"5min":0.71399999999999997,"15min":0.46500000000000002},"min":{"1min":0.32100000000000001,"5min":0.22800000000000001,"15min":0.222},"max":{"1min":3.8580000000000001,"5min":4.1989999999999998,"15min":4.1989999999999998},"last":1.236}]},{"osd":3,"last update":"Tue Sep 10 19:29:59 2024","interfaces":[{"interface":"back","average":{"1min":1.276,"5min":0.80500000000000005,"15min":0.47499999999999998},"min":{"1min":0.35799999999999998,"5min":0.22800000000000001,"15min":0.221},"max":{"1min":2.4969999999999999,"5min":5.7290000000000001,"15min":5.7290000000000001},"last":6.282},{"interface":"front","average":{"1min":1.512,"5min":1.034,"15min":0.54800000000000004},"min":{"1min":0.26800000000000002,"5min":0.20899999999999999,"15min":0.17599999999999999},"max":{"1min":4.2439999999999998,"5min":21.545000000000002,"15min":21.545000000000002},"last":5.8049999999999997}]},{"osd":4,"last update":"Tue Sep 10 19:29:59 2024","interfaces":[{"interface":"back","average":{"1min":1.4299999999999999,"5min":1.244,"15min":0.78300000000000003},"min":{"1min":0.29199999999999998,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":5.8499999999999996,"5min":5.8499999999999996,"15min":5.8499999999999996},"last":1.1000000000000001},{"interface":"front","average":{"1min":1.2869999999999999,"5min":1.163,"15min":0.80100000000000005},"min":{"1min":0.38700000000000001,"5min":0.30199999999999999,"15min":0.27600000000000002},"max":{"1min":2.3530000000000002,"5min":5.7599999999999998,"15min":5.7599999999999998},"last":0.68799999999999994}]},{"osd":5,"last update":"Tue Sep 10 19:29:59 2024","interfaces":[{"interface":"back","average":{"1min":1.421,"5min":1.1419999999999999,"15min":1.0169999999999999},"min":{"1min":0.32300000000000001,"5min":0.252,"15min":0.252},"max":{"1min":3.536,"5min":4.1529999999999996,"15min":4.7519999999999998},"last":1.1679999999999999},{"interface":"front","average":{"1min":1.4490000000000001,"5min":1.218,"15min":1.3069999999999999},"min":{"1min":0.47499999999999998,"5min":0.313,"15min":0.313},"max":{"1min":2.3860000000000001,"5min":4.181,"15min":5.617},"last":0.70899999999999996}]},{"osd":6,"last update":"Tue Sep 10 19:29:59 2024","interfaces":[{"interface":"back","average":{"1min":2.0489999999999999,"5min":1.363,"15min":1.931},"min":{"1min":0.30599999999999999,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":4.6319999999999997,"5min":7.5270000000000001,"15min":27.379999999999999},"last":1.27},{"interface":"front","average":{"1min":1.7210000000000001,"5min":1.27,"15min":1.887},"min":{"1min":0.36199999999999999,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":5.8849999999999998,"5min":5.8849999999999998,"15min":26.812999999999999},"last":1.1919999999999999}]},{"osd":7,"last update":"Tue Sep 10 19:29:59 2024","interfaces":[{"interface":"back","average":{"1min":1.702,"5min":1.3460000000000001,"15min":1.107},"min":{"1min":0.373,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":4.1849999999999996,"5min":8.6039999999999992,"15min":8.6039999999999992},"last":1.29},{"interface":"front","average":{"1min":1.5,"5min":1.296,"15min":1.097},"min":{"1min":0.41199999999999998,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":3.512,"5min":8.5899999999999999,"15min":8.5899999999999999},"last":1.137}]}]},{"osd":0,"up_from":301,"seq":1292785156175,"num_pgs":65,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":850452,"kb_used_data":492108,"kb_used_omap":4,"kb_used_meta":358331,"kb_avail":92894700,"statfs":{"total":95995035648,"available":95124172800,"internally_reserved":0,"allocated":503918592,"data_stored":503567810,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":366931462},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,4],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Sep 10 19:30:34 2024","interfaces":[{"interface":"back","average":{"1min":1.7929999999999999,"5min":0.91600000000000004,"15min":0.68799999999999994},"min":{"1min":0.217,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":6.0369999999999999,"5min":6.0369999999999999,"15min":6.0369999999999999},"last":0.54000000000000004},{"interface":"front","average":{"1min":1.4430000000000001,"5min":0.75800000000000001,"15min":0.58199999999999996},"min":{"1min":0.19600000000000001,"5min":0.16900000000000001,"15min":0.16900000000000001},"max":{"1min":3.7909999999999999,"5min":3.7909999999999999,"15min":3.7909999999999999},"last":0.40300000000000002}]},{"osd":2,"last update":"Tue Sep 10 19:30:34 2024","interfaces":[{"interface":"back","average":{"1min":0.84799999999999998,"5min":0.59199999999999997,"15min":0.59899999999999998},"min":{"1min":0.21299999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":4.3230000000000004,"5min":4.3230000000000004,"15min":4.3230000000000004},"last":0.55500000000000005},{"interface":"front","average":{"1min":1.569,"5min":0.81799999999999995,"15min":0.69599999999999995},"min":{"1min":0.249,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":5.1079999999999997,"5min":5.1079999999999997,"15min":5.1079999999999997},"last":0.51400000000000001}]},{"osd":3,"last update":"Tue Sep 10 19:30:34 2024","interfaces":[{"interface":"back","average":{"1min":5.9119999999999999,"5min":1.6950000000000001,"15min":0.95699999999999996},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":89.468999999999994,"5min":89.468999999999994,"15min":89.468999999999994},"last":0.35299999999999998},{"interface":"front","average":{"1min":6.141,"5min":1.8200000000000001,"15min":1.022},"min":{"1min":0.216,"5min":0.216,"15min":0.216},"max":{"1min":89.272999999999996,"5min":89.272999999999996,"15min":89.272999999999996},"last":0.90600000000000003}]},{"osd":4,"last update":"Tue Sep 10 19:30:34 2024","interfaces":[{"interface":"back","average":{"1min":1.802,"5min":1.306,"15min":3.4380000000000002},"min":{"1min":0.38900000000000001,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":4.9729999999999999,"5min":6.5640000000000001,"15min":53.670000000000002},"last":0.58899999999999997},{"interface":"front","average":{"1min":1.103,"5min":1.087,"15min":3.3700000000000001},"min":{"1min":0.28000000000000003,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":3.6520000000000001,"5min":6.5019999999999998,"15min":53.935000000000002},"last":0.57399999999999995}]},{"osd":5,"last update":"Tue Sep 10 19:30:34 2024","interfaces":[{"interface":"back","average":{"1min":1.232,"5min":0.999,"15min":3.0529999999999999},"min":{"1min":0.30399999999999999,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":4.2779999999999996,"5min":4.2779999999999996,"15min":53.884999999999998},"last":0.97499999999999998},{"interface":"front","average":{"1min":1.2769999999999999,"5min":1.093,"15min":2.988},"min":{"1min":0.38800000000000001,"5min":0.309,"15min":0.309},"max":{"1min":5.0519999999999996,"5min":5.0519999999999996,"15min":53.875999999999998},"last":0.61499999999999999}]},{"osd":6,"last update":"Tue Sep 10 19:30:34 2024","interfaces":[{"interface":"back","average":{"1min":1.6120000000000001,"5min":1.216,"15min":3.6240000000000001},"min":{"1min":0.34799999999999998,"5min":0.311,"15min":0.311},"max":{"1min":3.7069999999999999,"5min":5.3769999999999998,"15min":56.901000000000003},"last":0.99199999999999999},{"interface":"front","average":{"1min":1.296,"5min":1.1819999999999999,"15min":3.5609999999999999},"min":{"1min":0.29499999999999998,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":4.766,"5min":6.6840000000000002,"15min":55.177},"last":1.0629999999999999}]},{"osd":7,"last update":"Tue Sep 10 19:30:34 2024","interfaces":[{"interface":"back","average":{"1min":1.0860000000000001,"5min":1.1000000000000001,"15min":1.1910000000000001},"min":{"1min":0.32900000000000001,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":4.2149999999999999,"5min":4.2149999999999999,"15min":4.2149999999999999},"last":6.3959999999999999},{"interface":"front","average":{"1min":1.385,"5min":1.125,"15min":1.2130000000000001},"min":{"1min":0.375,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":5.0810000000000004,"5min":5.0810000000000004,"15min":5.0810000000000004},"last":0.745}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":966,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1236,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1236,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":362,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1540,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":2118,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1018,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1502,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":494731264,"data_stored":494731264,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":792072192,"data_stored":792068120,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":641466368,"data_stored":641466368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":453578752,"data_stored":453574680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":364642304,"data_stored":364642304,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":761860096,"data_stored":761856024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":734396416,"data_stored":734396416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T19:30:56.257 INFO:tasks.cephadm.ceph_manager.ceph:recovered! 2024-09-10T19:30:56.257 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-10T19:30:56.257 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap 2024-09-10T19:30:56.257 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph osd dump -f json-pretty 2024-09-10T19:30:56.338 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:30:56.359 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:30:56.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:30:56.360 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:30:56.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:30:56 smithi183 ceph-mon[29736]: from='client.16191 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-10T19:30:56.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:30:56 smithi183 ceph-mon[29736]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T19:30:56.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:30:56 smithi183 ceph-mon[29736]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-10T19:30:56.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:30:56 smithi183 ceph-mon[29736]: osdmap e421: 8 total, 8 up, 7 in 2024-09-10T19:30:56.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:30:56 smithi183 ceph-mon[41562]: from='client.16191 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-10T19:30:56.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:30:56 smithi183 ceph-mon[41562]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T19:30:56.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:30:56 smithi183 ceph-mon[41562]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-10T19:30:56.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:30:56 smithi183 ceph-mon[41562]: osdmap e421: 8 total, 8 up, 7 in 2024-09-10T19:30:56.460 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:30:56.465 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:30:56 smithi191 ceph-mon[32884]: from='client.16191 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-10T19:30:56.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:30:56 smithi191 ceph-mon[32884]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T19:30:56.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:30:56 smithi191 ceph-mon[32884]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-10T19:30:56.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:30:56 smithi191 ceph-mon[32884]: osdmap e421: 8 total, 8 up, 7 in 2024-09-10T19:30:56.481 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:30:56.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:30:56.482 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:30:56.583 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:30:56.603 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:30:56.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:30:56.604 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:30:56.705 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:30:56.727 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:30:56.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:30:56.728 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:30:56.829 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:30:56.851 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:30:56.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:30:56.852 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:30:56.953 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:30:56.980 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:30:56.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:30:56.981 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:30:57.082 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:30:57.114 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:30:57.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:30:57.115 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:30:57.215 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:30:57.246 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:30:57.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:30:57.247 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:30:57.348 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:30:57.380 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:30:57.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:30:57.381 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:30:57.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:30:57 smithi183 ceph-mon[29736]: purged_snaps scrub starts 2024-09-10T19:30:57.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:30:57 smithi183 ceph-mon[29736]: purged_snaps scrub ok 2024-09-10T19:30:57.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:30:57 smithi183 ceph-mon[29736]: pgmap v957: 158 pgs: 158 active+clean; 1.0 GiB data, 8.7 GiB used, 617 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 1.50k op/s 2024-09-10T19:30:57.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:30:57 smithi183 ceph-mon[29736]: osdmap e422: 8 total, 8 up, 7 in 2024-09-10T19:30:57.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:30:57 smithi183 ceph-mon[41562]: purged_snaps scrub starts 2024-09-10T19:30:57.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:30:57 smithi183 ceph-mon[41562]: purged_snaps scrub ok 2024-09-10T19:30:57.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:30:57 smithi183 ceph-mon[41562]: pgmap v957: 158 pgs: 158 active+clean; 1.0 GiB data, 8.7 GiB used, 617 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 1.50k op/s 2024-09-10T19:30:57.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:30:57 smithi183 ceph-mon[41562]: osdmap e422: 8 total, 8 up, 7 in 2024-09-10T19:30:57.451 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:30:57 smithi191 ceph-mon[32884]: purged_snaps scrub starts 2024-09-10T19:30:57.451 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:30:57 smithi191 ceph-mon[32884]: purged_snaps scrub ok 2024-09-10T19:30:57.451 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:30:57 smithi191 ceph-mon[32884]: pgmap v957: 158 pgs: 158 active+clean; 1.0 GiB data, 8.7 GiB used, 617 GiB / 626 GiB avail; 639 B/s rd, 0 B/s wr, 1.50k op/s 2024-09-10T19:30:57.451 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:30:57 smithi191 ceph-mon[32884]: osdmap e422: 8 total, 8 up, 7 in 2024-09-10T19:30:57.482 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:30:57.504 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:30:57.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:30:57.505 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:30:57.606 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:30:57.631 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:30:57.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:30:57.632 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:30:57.716 INFO:journalctl@ceph.mgr.x.smithi191.stdout:Sep 10 19:30:57 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-x[34967]: ::ffff:172.21.15.191 - - [10/Sep/2024:19:30:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T19:30:57.732 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:30:57.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:30:57.757 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:30:57.757 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:30:57.858 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:30:57.884 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:30:57.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:30:57.885 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:30:57.967 INFO:teuthology.orchestra.run.smithi191.stdout:{ 2024-09-10T19:30:57.967 INFO:teuthology.orchestra.run.smithi191.stdout: "size": 20, 2024-09-10T19:30:57.967 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 600, 2024-09-10T19:30:57.967 INFO:teuthology.orchestra.run.smithi191.stdout: "ops": [ 2024-09-10T19:30:57.968 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:30:57.968 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.15360.0:169771 9.6 9:6a78d762:::benchmark_data_smithi183_142499_object23282:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e262)", 2024-09-10T19:30:57.968 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-10T19:21:46.953119+0000", 2024-09-10T19:30:57.968 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 551.00940109999999, 2024-09-10T19:30:57.968 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.06359206, 2024-09-10T19:30:57.968 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-10T19:30:57.968 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "reached pg", 2024-09-10T19:30:57.968 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-10T19:30:57.968 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.15360", 2024-09-10T19:30:57.968 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.183:0/2364804131", 2024-09-10T19:30:57.968 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 169771 2024-09-10T19:30:57.968 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:30:57.969 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-10T19:30:57.969 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:30:57.969 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-10T19:30:57.969 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:21:46.953119+0000", 2024-09-10T19:30:57.969 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:30:57.969 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:30:57.969 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:30:57.969 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-10T19:30:57.969 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:21:46.953119+0000", 2024-09-10T19:30:57.969 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:30:57.969 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:30:57.969 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:30:57.969 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-10T19:30:57.969 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:21:46.953119+0000", 2024-09-10T19:30:57.970 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999995 2024-09-10T19:30:57.970 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:30:57.970 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:30:57.970 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-10T19:30:57.970 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:21:46.953125+0000", 2024-09-10T19:30:57.970 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.9950000000000002e-06 2024-09-10T19:30:57.970 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:30:57.970 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:30:57.970 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-10T19:30:57.970 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:21:46.953133+0000", 2024-09-10T19:30:57.971 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.8390000000000007e-06 2024-09-10T19:30:57.971 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:30:57.971 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:30:57.971 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-10T19:30:57.971 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:21:46.953137+0000", 2024-09-10T19:30:57.971 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.87e-06 2024-09-10T19:30:57.971 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:30:57.971 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:30:57.971 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:30:57.971 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:21:46.954452+0000", 2024-09-10T19:30:57.971 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.001314787 2024-09-10T19:30:57.971 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:30:57.971 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:30:57.971 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-10T19:30:57.972 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:21:46.954562+0000", 2024-09-10T19:30:57.972 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00011018000000000001 2024-09-10T19:30:57.972 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:30:57.972 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:30:57.972 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0,6", 2024-09-10T19:30:57.972 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:21:46.954675+0000", 2024-09-10T19:30:57.972 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000112895 2024-09-10T19:30:57.972 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:30:57.972 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:30:57.972 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-09-10T19:30:57.972 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:21:46.959358+0000", 2024-09-10T19:30:57.972 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0046832080000000003 2024-09-10T19:30:57.972 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:30:57.973 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:30:57.973 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:30:57.973 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:21:46.971046+0000", 2024-09-10T19:30:57.973 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.011687362 2024-09-10T19:30:57.973 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:30:57.973 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:30:57.973 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:30:57.973 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:21:46.986841+0000", 2024-09-10T19:30:57.973 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.015795317 2024-09-10T19:30:57.973 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:30:57.973 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:30:57.973 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for peered", 2024-09-10T19:30:57.973 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:21:46.986849+0000", 2024-09-10T19:30:57.973 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.0059999999999992e-06 2024-09-10T19:30:57.974 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:30:57.974 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:30:57.974 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:30:57.974 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:21:48.016648+0000", 2024-09-10T19:30:57.974 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.0297984440000001 2024-09-10T19:30:57.974 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:30:57.974 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:30:57.974 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-10T19:30:57.974 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:21:48.016711+0000", 2024-09-10T19:30:57.974 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.3819999999999995e-05 2024-09-10T19:30:57.974 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:30:57.974 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-10T19:30:57.974 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:30:57.975 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:30:57.975 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:30:57.975 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.15360.0:169759 9.5 9:a00f2629:::benchmark_data_smithi183_142499_object23270:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e263)", 2024-09-10T19:30:57.975 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-10T19:21:46.960422+0000", 2024-09-10T19:30:57.975 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 551.00209879700003, 2024-09-10T19:30:57.975 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.057974347, 2024-09-10T19:30:57.975 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-10T19:30:57.975 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "reached pg", 2024-09-10T19:30:57.975 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-10T19:30:57.975 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.15360", 2024-09-10T19:30:57.975 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.183:0/2364804131", 2024-09-10T19:30:57.975 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 169759 2024-09-10T19:30:57.975 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:30:57.976 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-10T19:30:57.976 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:30:57.976 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-10T19:30:57.976 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:21:46.960422+0000", 2024-09-10T19:30:57.976 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:30:57.976 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:30:57.976 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:30:57.976 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-10T19:30:57.976 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:21:46.960422+0000", 2024-09-10T19:30:57.976 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:30:57.976 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:30:57.976 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:30:57.976 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-10T19:30:57.977 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:21:46.960420+0000", 2024-09-10T19:30:57.977 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999986 2024-09-10T19:30:57.977 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:30:57.977 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:30:57.977 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-10T19:30:57.977 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:21:46.960426+0000", 2024-09-10T19:30:57.977 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.9159999999999998e-06 2024-09-10T19:30:57.977 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:30:57.977 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:30:57.977 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-10T19:30:57.977 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:21:46.960436+0000", 2024-09-10T19:30:57.977 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.5380000000000008e-06 2024-09-10T19:30:57.977 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:30:57.977 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:30:57.978 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-10T19:30:57.978 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:21:46.960441+0000", 2024-09-10T19:30:57.978 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.271e-06 2024-09-10T19:30:57.978 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:30:57.978 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:30:57.978 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:30:57.978 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:21:46.987106+0000", 2024-09-10T19:30:57.978 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.026665287999999999 2024-09-10T19:30:57.978 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:30:57.978 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:30:57.978 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for peered", 2024-09-10T19:30:57.978 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:21:46.987113+0000", 2024-09-10T19:30:57.979 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.2129999999999996e-06 2024-09-10T19:30:57.979 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:30:57.979 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:30:57.979 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:30:57.979 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:21:48.014573+0000", 2024-09-10T19:30:57.979 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.027460866 2024-09-10T19:30:57.979 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:30:57.979 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:30:57.979 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-10T19:30:57.979 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:21:48.018396+0000", 2024-09-10T19:30:57.979 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0038226290000000001 2024-09-10T19:30:57.979 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:30:57.979 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-10T19:30:57.980 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:30:57.980 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:30:57.980 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:30:57.980 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.15360.0:169769 9.5 9:aaef8d5a:::benchmark_data_smithi183_142499_object23280:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e263)", 2024-09-10T19:30:57.980 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-10T19:21:46.960447+0000", 2024-09-10T19:30:57.980 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 551.00207355099997, 2024-09-10T19:30:57.980 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.059507046, 2024-09-10T19:30:57.980 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-10T19:30:57.980 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "reached pg", 2024-09-10T19:30:57.980 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-10T19:30:57.980 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.15360", 2024-09-10T19:30:57.980 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.183:0/2364804131", 2024-09-10T19:30:57.980 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 169769 2024-09-10T19:30:57.980 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:30:57.981 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-10T19:30:57.981 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:30:57.981 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-10T19:30:57.981 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:21:46.960447+0000", 2024-09-10T19:30:57.981 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:30:57.981 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:30:57.981 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:30:57.981 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-10T19:30:57.981 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:21:46.960447+0000", 2024-09-10T19:30:57.981 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:30:57.981 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:30:57.981 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:30:57.981 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-10T19:30:57.982 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:21:46.960446+0000", 2024-09-10T19:30:57.982 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999995 2024-09-10T19:30:57.982 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:30:57.982 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:30:57.982 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-10T19:30:57.982 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:21:46.960449+0000", 2024-09-10T19:30:57.983 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.6359999999999998e-06 2024-09-10T19:30:57.983 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:30:57.983 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:30:57.983 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-10T19:30:57.983 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:21:46.960455+0000", 2024-09-10T19:30:57.983 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.0839999999999997e-06 2024-09-10T19:30:57.983 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:30:57.983 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:30:57.983 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-10T19:30:57.983 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:21:46.960459+0000", 2024-09-10T19:30:57.983 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.4419999999999998e-06 2024-09-10T19:30:57.984 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:30:57.984 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:30:57.984 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:30:57.984 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:21:46.987136+0000", 2024-09-10T19:30:57.984 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.026677101000000002 2024-09-10T19:30:57.984 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:30:57.984 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:30:57.984 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for peered", 2024-09-10T19:30:57.984 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:21:46.987140+0000", 2024-09-10T19:30:57.984 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.3170000000000003e-06 2024-09-10T19:30:57.984 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:30:57.984 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:30:57.984 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:30:57.985 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:21:48.019902+0000", 2024-09-10T19:30:57.985 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.03276155 2024-09-10T19:30:57.985 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:30:57.985 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:30:57.985 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-10T19:30:57.985 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:21:48.019954+0000", 2024-09-10T19:30:57.985 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.2389000000000003e-05 2024-09-10T19:30:57.985 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:30:57.985 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-10T19:30:57.985 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:30:57.985 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:30:57.985 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:30:57.985 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.25622.0:12696 10.6 10:6d4ba2dc:::benchmark_data_smithi183_176537_object12695:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e308)", 2024-09-10T19:30:57.986 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-10T19:24:29.887487+0000", 2024-09-10T19:30:57.986 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 388.07503330700001, 2024-09-10T19:30:57.986 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.00185239, 2024-09-10T19:30:57.986 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-10T19:30:57.986 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T19:30:57.986 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-10T19:30:57.986 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.25622", 2024-09-10T19:30:57.986 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.183:0/3685189199", 2024-09-10T19:30:57.986 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 12696 2024-09-10T19:30:57.986 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:30:57.986 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-10T19:30:57.986 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:30:57.986 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-10T19:30:57.986 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:24:29.887487+0000", 2024-09-10T19:30:57.987 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:30:57.987 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:30:57.987 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:30:57.987 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-10T19:30:57.987 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:24:29.887487+0000", 2024-09-10T19:30:57.987 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:30:57.987 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:30:57.987 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:30:57.987 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-10T19:30:57.987 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:24:29.887485+0000", 2024-09-10T19:30:57.987 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999981 2024-09-10T19:30:57.987 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:30:57.987 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:30:57.988 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-10T19:30:57.988 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:24:29.887594+0000", 2024-09-10T19:30:57.988 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000108457 2024-09-10T19:30:57.988 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:30:57.988 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:30:57.988 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-10T19:30:57.988 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:24:29.887615+0000", 2024-09-10T19:30:57.988 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.1115e-05 2024-09-10T19:30:57.988 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:30:57.988 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:30:57.988 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-10T19:30:57.988 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:24:29.887621+0000", 2024-09-10T19:30:57.988 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.2099999999999998e-06 2024-09-10T19:30:57.988 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:30:57.989 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:30:57.989 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:30:57.989 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:24:29.887782+0000", 2024-09-10T19:30:57.989 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000160519 2024-09-10T19:30:57.989 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:30:57.989 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:30:57.989 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for peered", 2024-09-10T19:30:57.989 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:24:29.887791+0000", 2024-09-10T19:30:57.989 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.8740000000000007e-06 2024-09-10T19:30:57.989 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:30:57.989 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:30:57.989 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:30:57.989 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:24:30.884650+0000", 2024-09-10T19:30:57.990 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.99685872399999997 2024-09-10T19:30:57.990 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:30:57.990 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:30:57.990 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-10T19:30:57.990 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:24:30.884780+0000", 2024-09-10T19:30:57.990 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000129565 2024-09-10T19:30:57.990 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:30:57.990 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:30:57.990 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0,4", 2024-09-10T19:30:57.990 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:24:30.884882+0000", 2024-09-10T19:30:57.990 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000102049 2024-09-10T19:30:57.990 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:30:57.990 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:30:57.990 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-09-10T19:30:57.990 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:24:30.887536+0000", 2024-09-10T19:30:57.991 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.002654413 2024-09-10T19:30:57.991 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:30:57.991 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:30:57.991 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:30:57.991 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:24:30.889151+0000", 2024-09-10T19:30:57.991 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.001615219 2024-09-10T19:30:57.991 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:30:57.991 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:30:57.991 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:30:57.991 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:24:30.889184+0000", 2024-09-10T19:30:57.991 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.2679999999999999e-05 2024-09-10T19:30:57.991 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:30:57.991 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:30:57.991 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-09-10T19:30:57.992 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:24:30.889250+0000", 2024-09-10T19:30:57.992 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.5979e-05 2024-09-10T19:30:57.992 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:30:57.992 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:30:57.992 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-10T19:30:57.992 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:24:30.889340+0000", 2024-09-10T19:30:57.992 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.9493999999999994e-05 2024-09-10T19:30:57.992 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:30:57.992 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-10T19:30:57.992 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:30:57.992 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:30:57.992 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:30:57.992 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.25622.0:74177 10.9 10:965cfef6:::benchmark_data_smithi183_176537_object74176:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e327)", 2024-09-10T19:30:57.993 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-10T19:25:09.901416+0000", 2024-09-10T19:30:57.993 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 348.06110450300002, 2024-09-10T19:30:57.993 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.30973668100000001, 2024-09-10T19:30:57.993 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-10T19:30:57.993 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T19:30:57.993 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-10T19:30:57.993 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.25622", 2024-09-10T19:30:57.993 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.183:0/3685189199", 2024-09-10T19:30:57.993 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 74177 2024-09-10T19:30:57.993 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:30:57.993 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-10T19:30:57.993 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:30:57.993 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-10T19:30:57.993 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:25:09.901416+0000", 2024-09-10T19:30:57.994 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:30:57.994 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:30:57.994 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:30:57.994 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-10T19:30:57.994 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:25:09.901416+0000", 2024-09-10T19:30:57.994 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:30:57.994 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:30:57.994 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:30:57.994 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-10T19:30:57.994 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:25:09.901405+0000", 2024-09-10T19:30:57.994 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999886 2024-09-10T19:30:57.994 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:30:57.995 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:30:57.995 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-10T19:30:57.995 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:25:09.901446+0000", 2024-09-10T19:30:57.995 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.1315e-05 2024-09-10T19:30:57.995 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:30:57.995 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:30:57.995 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-10T19:30:57.995 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:25:09.901461+0000", 2024-09-10T19:30:57.995 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.5225000000000001e-05 2024-09-10T19:30:57.995 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:30:57.995 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:30:57.995 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-10T19:30:57.995 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:25:09.901467+0000", 2024-09-10T19:30:57.995 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.1840000000000004e-06 2024-09-10T19:30:57.995 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:30:57.996 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:30:57.996 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:30:57.996 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:25:09.901551+0000", 2024-09-10T19:30:57.996 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.3354000000000002e-05 2024-09-10T19:30:57.996 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:30:57.996 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:30:57.996 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-10T19:30:57.996 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:25:09.901835+0000", 2024-09-10T19:30:57.996 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00028411000000000002 2024-09-10T19:30:57.996 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:30:57.996 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:30:57.996 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 4,7", 2024-09-10T19:30:57.996 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:25:09.901971+0000", 2024-09-10T19:30:57.996 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000136083 2024-09-10T19:30:57.997 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:30:57.997 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:30:57.997 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:30:57.997 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:25:09.907343+0000", 2024-09-10T19:30:57.997 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0053719400000000004 2024-09-10T19:30:57.997 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:30:57.997 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:30:57.997 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-09-10T19:30:57.997 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:25:09.907351+0000", 2024-09-10T19:30:57.997 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.0669999999999992e-06 2024-09-10T19:30:57.997 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:30:57.997 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:30:57.997 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:30:57.998 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:25:10.211056+0000", 2024-09-10T19:30:57.998 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.303704796 2024-09-10T19:30:57.998 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:30:57.998 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:30:57.998 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-09-10T19:30:57.998 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:25:10.211113+0000", 2024-09-10T19:30:57.998 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.6962000000000001e-05 2024-09-10T19:30:57.998 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:30:57.998 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:30:57.998 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-10T19:30:57.998 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:25:10.211153+0000", 2024-09-10T19:30:57.998 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.0009000000000003e-05 2024-09-10T19:30:57.998 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:30:57.998 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-10T19:30:57.999 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:30:57.999 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:30:57.999 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:30:57.999 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.25622.0:74179 10.9 10:9955a35e:::benchmark_data_smithi183_176537_object74178:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e327)", 2024-09-10T19:30:57.999 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-10T19:25:09.901623+0000", 2024-09-10T19:30:57.999 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 348.06089700500002, 2024-09-10T19:30:57.999 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.30965538999999997, 2024-09-10T19:30:57.999 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-10T19:30:57.999 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T19:30:57.999 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-10T19:30:57.999 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.25622", 2024-09-10T19:30:57.999 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.183:0/3685189199", 2024-09-10T19:30:57.999 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 74179 2024-09-10T19:30:58.000 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:30:58.000 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-10T19:30:58.000 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:30:58.000 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-10T19:30:58.000 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:25:09.901623+0000", 2024-09-10T19:30:58.000 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:30:58.000 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:30:58.000 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:30:58.000 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-10T19:30:58.000 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:25:09.901623+0000", 2024-09-10T19:30:58.000 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:30:58.001 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:30:58.001 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:30:58.001 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-10T19:30:58.001 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:25:09.901623+0000", 2024-09-10T19:30:58.001 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999995 2024-09-10T19:30:58.001 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:30:58.001 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:30:58.001 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-10T19:30:58.001 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:25:09.901678+0000", 2024-09-10T19:30:58.001 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.4846000000000003e-05 2024-09-10T19:30:58.001 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:30:58.001 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:30:58.001 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-10T19:30:58.002 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:25:09.901685+0000", 2024-09-10T19:30:58.002 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.4019999999999996e-06 2024-09-10T19:30:58.002 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:30:58.002 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:30:58.002 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-10T19:30:58.002 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:25:09.901689+0000", 2024-09-10T19:30:58.002 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.0080000000000004e-06 2024-09-10T19:30:58.002 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:30:58.002 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:30:58.002 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:30:58.002 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:25:09.904575+0000", 2024-09-10T19:30:58.002 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0028858949999999999 2024-09-10T19:30:58.003 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:30:58.003 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:30:58.003 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-10T19:30:58.003 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:25:09.904686+0000", 2024-09-10T19:30:58.003 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000111109 2024-09-10T19:30:58.003 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:30:58.003 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:30:58.003 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 4,7", 2024-09-10T19:30:58.003 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:25:09.904827+0000", 2024-09-10T19:30:58.003 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00014105899999999999 2024-09-10T19:30:58.003 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:30:58.003 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:30:58.003 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-09-10T19:30:58.003 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:25:09.911040+0000", 2024-09-10T19:30:58.004 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0062127340000000001 2024-09-10T19:30:58.004 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:30:58.004 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:30:58.004 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:30:58.004 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:25:09.914428+0000", 2024-09-10T19:30:58.004 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0033878269999999999 2024-09-10T19:30:58.004 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:30:58.004 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:30:58.004 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:30:58.004 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:25:10.211200+0000", 2024-09-10T19:30:58.004 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.296771866 2024-09-10T19:30:58.004 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:30:58.004 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:30:58.004 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-09-10T19:30:58.005 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:25:10.211245+0000", 2024-09-10T19:30:58.005 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.5105999999999998e-05 2024-09-10T19:30:58.005 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:30:58.005 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:30:58.005 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-10T19:30:58.005 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:25:10.211279+0000", 2024-09-10T19:30:58.005 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.4110999999999999e-05 2024-09-10T19:30:58.005 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:30:58.005 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-10T19:30:58.005 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:30:58.005 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:30:58.005 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:30:58.005 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.25622.0:74184 10.9 10:99282d95:::benchmark_data_smithi183_176537_object74183:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e327)", 2024-09-10T19:30:58.005 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-10T19:25:09.907805+0000", 2024-09-10T19:30:58.006 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 348.05471566099999, 2024-09-10T19:30:58.006 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.313228269, 2024-09-10T19:30:58.006 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-10T19:30:58.006 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T19:30:58.006 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-10T19:30:58.006 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.25622", 2024-09-10T19:30:58.006 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.183:0/3685189199", 2024-09-10T19:30:58.006 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 74184 2024-09-10T19:30:58.006 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:30:58.006 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-10T19:30:58.006 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:30:58.006 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-10T19:30:58.006 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:25:09.907805+0000", 2024-09-10T19:30:58.006 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:30:58.007 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:30:58.007 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:30:58.007 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-10T19:30:58.007 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:25:09.907805+0000", 2024-09-10T19:30:58.007 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:30:58.007 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:30:58.007 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:30:58.007 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-10T19:30:58.007 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:25:09.907803+0000", 2024-09-10T19:30:58.007 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999986 2024-09-10T19:30:58.007 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:30:58.007 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:30:58.007 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-10T19:30:58.007 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:25:09.907839+0000", 2024-09-10T19:30:58.008 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.6143999999999998e-05 2024-09-10T19:30:58.008 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:30:58.008 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:30:58.008 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-10T19:30:58.008 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:25:09.907850+0000", 2024-09-10T19:30:58.008 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.0973e-05 2024-09-10T19:30:58.008 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:30:58.008 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:30:58.008 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-10T19:30:58.008 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:25:09.907856+0000", 2024-09-10T19:30:58.008 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.3970000000000002e-06 2024-09-10T19:30:58.008 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:30:58.008 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:30:58.008 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:30:58.009 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:25:09.907933+0000", 2024-09-10T19:30:58.009 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.6839000000000001e-05 2024-09-10T19:30:58.009 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:30:58.009 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:30:58.009 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-10T19:30:58.009 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:25:09.908103+0000", 2024-09-10T19:30:58.009 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000170342 2024-09-10T19:30:58.009 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:30:58.009 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:30:58.009 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 4,7", 2024-09-10T19:30:58.009 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:25:09.908269+0000", 2024-09-10T19:30:58.009 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00016650700000000001 2024-09-10T19:30:58.009 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:30:58.009 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:30:58.010 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-09-10T19:30:58.010 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:25:09.914609+0000", 2024-09-10T19:30:58.010 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0063397749999999998 2024-09-10T19:30:58.010 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:30:58.010 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:30:58.010 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:30:58.010 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:25:09.915151+0000", 2024-09-10T19:30:58.010 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00054191700000000005 2024-09-10T19:30:58.010 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:30:58.010 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:30:58.010 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:30:58.010 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:25:10.220929+0000", 2024-09-10T19:30:58.010 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.30577790399999999 2024-09-10T19:30:58.011 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:30:58.011 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:30:58.011 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-09-10T19:30:58.011 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:25:10.220987+0000", 2024-09-10T19:30:58.011 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.8396999999999997e-05 2024-09-10T19:30:58.011 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:30:58.011 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:30:58.011 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-10T19:30:58.011 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:25:10.221033+0000", 2024-09-10T19:30:58.011 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.5714000000000003e-05 2024-09-10T19:30:58.011 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:30:58.011 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-10T19:30:58.012 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:30:58.012 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:30:58.012 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:30:58.012 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.25622.0:74216 10.9 10:99cba0a9:::benchmark_data_smithi183_176537_object74215:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e327)", 2024-09-10T19:30:58.012 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-10T19:25:09.942318+0000", 2024-09-10T19:30:58.012 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 348.02020276600001, 2024-09-10T19:30:58.012 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.28289799999999998, 2024-09-10T19:30:58.012 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-10T19:30:58.012 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T19:30:58.012 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-10T19:30:58.012 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.25622", 2024-09-10T19:30:58.012 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.183:0/3685189199", 2024-09-10T19:30:58.012 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 74216 2024-09-10T19:30:58.013 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:30:58.013 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-10T19:30:58.013 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:30:58.013 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-10T19:30:58.013 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:25:09.942318+0000", 2024-09-10T19:30:58.013 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:30:58.013 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:30:58.013 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:30:58.013 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-10T19:30:58.013 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:25:09.942318+0000", 2024-09-10T19:30:58.013 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:30:58.013 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:30:58.013 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:30:58.014 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-10T19:30:58.014 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:25:09.942317+0000", 2024-09-10T19:30:58.014 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.999999 2024-09-10T19:30:58.014 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:30:58.014 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:30:58.014 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-10T19:30:58.014 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:25:09.942374+0000", 2024-09-10T19:30:58.014 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.7215e-05 2024-09-10T19:30:58.014 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:30:58.014 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:30:58.014 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-10T19:30:58.014 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:25:09.942383+0000", 2024-09-10T19:30:58.014 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.6279999999999994e-06 2024-09-10T19:30:58.015 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:30:58.015 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:30:58.015 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-10T19:30:58.015 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:25:09.942386+0000", 2024-09-10T19:30:58.015 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.591e-06 2024-09-10T19:30:58.015 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:30:58.015 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:30:58.015 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:30:58.015 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:25:09.942432+0000", 2024-09-10T19:30:58.015 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.5766000000000002e-05 2024-09-10T19:30:58.015 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:30:58.015 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:30:58.015 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-10T19:30:58.015 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:25:09.942579+0000", 2024-09-10T19:30:58.016 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00014672099999999999 2024-09-10T19:30:58.016 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:30:58.016 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:30:58.016 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 4,7", 2024-09-10T19:30:58.016 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:25:09.942743+0000", 2024-09-10T19:30:58.016 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00016406400000000001 2024-09-10T19:30:58.016 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:30:58.016 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:30:58.016 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:30:58.016 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:25:09.945782+0000", 2024-09-10T19:30:58.016 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0030390690000000001 2024-09-10T19:30:58.016 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:30:58.017 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:30:58.017 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-09-10T19:30:58.017 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:25:09.945880+0000", 2024-09-10T19:30:58.017 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.8105999999999999e-05 2024-09-10T19:30:58.017 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:30:58.017 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:30:58.017 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:30:58.017 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:25:10.225129+0000", 2024-09-10T19:30:58.017 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.27924855799999998 2024-09-10T19:30:58.017 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:30:58.017 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:30:58.017 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-09-10T19:30:58.017 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:25:10.225178+0000", 2024-09-10T19:30:58.018 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.9429999999999999e-05 2024-09-10T19:30:58.018 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:30:58.018 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:30:58.018 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-10T19:30:58.018 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:25:10.225216+0000", 2024-09-10T19:30:58.018 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.7703999999999997e-05 2024-09-10T19:30:58.018 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:30:58.018 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-10T19:30:58.018 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:30:58.018 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:30:58.018 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:30:58.018 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.25622.0:75598 10.3 10:cdb716c5:::benchmark_data_smithi183_176537_object75597:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e327)", 2024-09-10T19:30:58.018 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-10T19:25:11.423897+0000", 2024-09-10T19:30:58.018 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 346.53862333900003, 2024-09-10T19:30:58.019 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.26821467599999999, 2024-09-10T19:30:58.019 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-10T19:30:58.019 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T19:30:58.019 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-10T19:30:58.019 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.25622", 2024-09-10T19:30:58.019 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.183:0/3685189199", 2024-09-10T19:30:58.019 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 75598 2024-09-10T19:30:58.019 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:30:58.019 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-10T19:30:58.019 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:30:58.019 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-10T19:30:58.019 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:25:11.423897+0000", 2024-09-10T19:30:58.019 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:30:58.019 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:30:58.020 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:30:58.020 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-10T19:30:58.020 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:25:11.423897+0000", 2024-09-10T19:30:58.020 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:30:58.020 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:30:58.020 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:30:58.020 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-10T19:30:58.020 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:25:11.423896+0000", 2024-09-10T19:30:58.020 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999986 2024-09-10T19:30:58.020 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:30:58.020 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:30:58.020 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-10T19:30:58.020 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:25:11.423928+0000", 2024-09-10T19:30:58.020 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.2422999999999998e-05 2024-09-10T19:30:58.021 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:30:58.021 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:30:58.021 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-10T19:30:58.021 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:25:11.423941+0000", 2024-09-10T19:30:58.021 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.2906e-05 2024-09-10T19:30:58.021 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:30:58.021 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:30:58.021 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-10T19:30:58.021 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:25:11.423947+0000", 2024-09-10T19:30:58.021 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.2140000000000001e-06 2024-09-10T19:30:58.021 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:30:58.021 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:30:58.021 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:30:58.022 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:25:11.424001+0000", 2024-09-10T19:30:58.022 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.3566999999999997e-05 2024-09-10T19:30:58.022 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:30:58.022 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:30:58.022 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-10T19:30:58.022 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:25:11.676257+0000", 2024-09-10T19:30:58.022 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.25225620999999998 2024-09-10T19:30:58.022 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:30:58.022 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:30:58.022 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 3,4", 2024-09-10T19:30:58.022 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:25:11.676412+0000", 2024-09-10T19:30:58.022 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000155181 2024-09-10T19:30:58.022 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:30:58.023 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:30:58.023 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:30:58.023 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:25:11.688285+0000", 2024-09-10T19:30:58.023 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.011873091000000001 2024-09-10T19:30:58.023 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:30:58.023 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:30:58.023 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-09-10T19:30:58.023 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:25:11.688292+0000", 2024-09-10T19:30:58.023 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.7700000000000004e-06 2024-09-10T19:30:58.023 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:30:58.023 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:30:58.023 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:30:58.023 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:25:11.692027+0000", 2024-09-10T19:30:58.023 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0037353360000000001 2024-09-10T19:30:58.024 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:30:58.024 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:30:58.024 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-09-10T19:30:58.024 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:25:11.692075+0000", 2024-09-10T19:30:58.024 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.7719999999999997e-05 2024-09-10T19:30:58.024 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:30:58.024 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:30:58.024 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-10T19:30:58.024 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:25:11.692112+0000", 2024-09-10T19:30:58.024 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.6721000000000003e-05 2024-09-10T19:30:58.024 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:30:58.024 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-10T19:30:58.024 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:30:58.024 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:30:58.025 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:30:58.025 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:30:58.026 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.16026.0:105908 11.7 11:e2facd4d:::benchmark_data_smithi183_203944_object105907:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e396)", 2024-09-10T19:30:58.026 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-10T19:28:54.512030+0000", 2024-09-10T19:30:58.026 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 123.450490355, 2024-09-10T19:30:58.026 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.143914124, 2024-09-10T19:30:58.026 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-10T19:30:58.026 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T19:30:58.027 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-10T19:30:58.027 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.16026", 2024-09-10T19:30:58.027 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.183:0/1450567484", 2024-09-10T19:30:58.027 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 105908 2024-09-10T19:30:58.027 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:30:58.027 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-10T19:30:58.027 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:30:58.027 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-10T19:30:58.027 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:28:54.512030+0000", 2024-09-10T19:30:58.027 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:30:58.027 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:30:58.027 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:30:58.027 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-10T19:30:58.027 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:28:54.512030+0000", 2024-09-10T19:30:58.028 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:30:58.028 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:30:58.028 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:30:58.028 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-10T19:30:58.028 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:28:54.512028+0000", 2024-09-10T19:30:58.028 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999981 2024-09-10T19:30:58.028 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:30:58.028 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:30:58.028 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-10T19:30:58.028 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:28:54.512102+0000", 2024-09-10T19:30:58.028 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.3497999999999993e-05 2024-09-10T19:30:58.028 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:30:58.028 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:30:58.029 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-10T19:30:58.029 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:28:54.512117+0000", 2024-09-10T19:30:58.029 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.5104e-05 2024-09-10T19:30:58.029 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:30:58.029 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:30:58.029 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-10T19:30:58.029 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:28:54.512123+0000", 2024-09-10T19:30:58.029 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.3060000000000003e-06 2024-09-10T19:30:58.029 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:30:58.029 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:30:58.029 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:30:58.029 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:28:54.512402+0000", 2024-09-10T19:30:58.029 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00027886600000000001 2024-09-10T19:30:58.029 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:30:58.030 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:30:58.030 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-10T19:30:58.030 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:28:54.512517+0000", 2024-09-10T19:30:58.030 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000114758 2024-09-10T19:30:58.030 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:30:58.030 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:30:58.030 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 1,4", 2024-09-10T19:30:58.030 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:28:54.512656+0000", 2024-09-10T19:30:58.030 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00013939900000000001 2024-09-10T19:30:58.030 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:30:58.030 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:30:58.030 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-09-10T19:30:58.030 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:28:54.516364+0000", 2024-09-10T19:30:58.031 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0037082349999999998 2024-09-10T19:30:58.031 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:30:58.031 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:30:58.031 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:30:58.031 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:28:54.517489+0000", 2024-09-10T19:30:58.031 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.001125069 2024-09-10T19:30:58.031 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:30:58.031 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:30:58.031 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:30:58.031 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:28:54.655848+0000", 2024-09-10T19:30:58.031 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.13835856299999999 2024-09-10T19:30:58.031 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:30:58.032 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:30:58.032 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-09-10T19:30:58.032 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:28:54.655900+0000", 2024-09-10T19:30:58.032 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.1687999999999997e-05 2024-09-10T19:30:58.032 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:30:58.032 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:30:58.032 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-10T19:30:58.032 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:28:54.655944+0000", 2024-09-10T19:30:58.032 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.4517000000000001e-05 2024-09-10T19:30:58.032 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:30:58.032 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-10T19:30:58.032 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:30:58.032 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:30:58.033 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:30:58.033 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.16026.0:108981 11.7 11:e75738ab:::benchmark_data_smithi183_203944_object108980:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e397)", 2024-09-10T19:30:58.033 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-10T19:28:57.168351+0000", 2024-09-10T19:30:58.033 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 120.794169457, 2024-09-10T19:30:58.033 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.19295467999999999, 2024-09-10T19:30:58.033 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-10T19:30:58.033 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T19:30:58.033 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-10T19:30:58.033 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.16026", 2024-09-10T19:30:58.033 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.183:0/1450567484", 2024-09-10T19:30:58.033 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 108981 2024-09-10T19:30:58.033 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:30:58.033 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-10T19:30:58.034 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:30:58.034 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-10T19:30:58.034 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:28:57.168351+0000", 2024-09-10T19:30:58.034 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:30:58.034 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:30:58.034 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:30:58.034 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-10T19:30:58.034 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:28:57.168351+0000", 2024-09-10T19:30:58.034 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:30:58.034 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:30:58.034 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:30:58.034 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-10T19:30:58.034 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:28:57.168349+0000", 2024-09-10T19:30:58.034 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999981 2024-09-10T19:30:58.034 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:30:58.034 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:30:58.035 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-10T19:30:58.035 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:28:57.168382+0000", 2024-09-10T19:30:58.035 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.2669000000000002e-05 2024-09-10T19:30:58.035 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:30:58.035 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:30:58.035 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-10T19:30:58.035 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:28:57.168398+0000", 2024-09-10T19:30:58.035 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.5739999999999998e-05 2024-09-10T19:30:58.035 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:30:58.035 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:30:58.035 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-10T19:30:58.035 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:28:57.168404+0000", 2024-09-10T19:30:58.035 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.3910000000000003e-06 2024-09-10T19:30:58.035 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:30:58.035 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:30:58.036 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:30:58.036 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:28:57.168613+0000", 2024-09-10T19:30:58.036 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00020932899999999999 2024-09-10T19:30:58.036 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:30:58.036 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:30:58.036 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-10T19:30:58.036 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:28:57.168943+0000", 2024-09-10T19:30:58.036 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00032922000000000002 2024-09-10T19:30:58.036 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:30:58.036 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:30:58.036 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 1,4", 2024-09-10T19:30:58.036 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:28:57.169358+0000", 2024-09-10T19:30:58.036 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00041506099999999999 2024-09-10T19:30:58.036 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:30:58.037 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:30:58.037 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-09-10T19:30:58.037 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:28:57.172580+0000", 2024-09-10T19:30:58.037 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.003222197 2024-09-10T19:30:58.037 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:30:58.037 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:30:58.037 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:30:58.037 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:28:57.175337+0000", 2024-09-10T19:30:58.037 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0027574499999999998 2024-09-10T19:30:58.037 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:30:58.037 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:30:58.037 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:30:58.037 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:28:57.361176+0000", 2024-09-10T19:30:58.037 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.18583892199999999 2024-09-10T19:30:58.038 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:30:58.038 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:30:58.038 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-09-10T19:30:58.038 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:28:57.361235+0000", 2024-09-10T19:30:58.038 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.8368000000000001e-05 2024-09-10T19:30:58.038 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:30:58.038 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:30:58.038 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-10T19:30:58.038 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:28:57.361306+0000", 2024-09-10T19:30:58.038 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.1056000000000005e-05 2024-09-10T19:30:58.038 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:30:58.038 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-10T19:30:58.038 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:30:58.038 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:30:58.039 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:30:58.039 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.16026.0:263535 11.7 11:ea38cdbd:::benchmark_data_smithi183_203944_object118645:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e419)", 2024-09-10T19:30:58.039 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-10T19:30:47.295255+0000", 2024-09-10T19:30:58.039 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 10.667265069000001, 2024-09-10T19:30:58.039 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.37229807700000001, 2024-09-10T19:30:58.039 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-10T19:30:58.039 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T19:30:58.039 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-10T19:30:58.039 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.16026", 2024-09-10T19:30:58.039 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.183:0/1450567484", 2024-09-10T19:30:58.039 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 263535 2024-09-10T19:30:58.039 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:30:58.039 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-10T19:30:58.039 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:30:58.040 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-10T19:30:58.040 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:30:47.295255+0000", 2024-09-10T19:30:58.040 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:30:58.040 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:30:58.040 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:30:58.040 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-10T19:30:58.040 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:30:47.295255+0000", 2024-09-10T19:30:58.040 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:30:58.040 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:30:58.040 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:30:58.040 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-10T19:30:58.040 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:30:47.295254+0000", 2024-09-10T19:30:58.040 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999981 2024-09-10T19:30:58.040 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:30:58.041 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:30:58.041 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-10T19:30:58.041 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:30:47.295262+0000", 2024-09-10T19:30:58.041 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.8010000000000004e-06 2024-09-10T19:30:58.041 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:30:58.041 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:30:58.041 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-10T19:30:58.041 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:30:47.295279+0000", 2024-09-10T19:30:58.041 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.7530999999999999e-05 2024-09-10T19:30:58.041 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:30:58.041 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:30:58.041 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-10T19:30:58.041 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:30:47.295286+0000", 2024-09-10T19:30:58.041 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.3849999999999996e-06 2024-09-10T19:30:58.042 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:30:58.042 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:30:58.042 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:30:58.042 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:30:47.295323+0000", 2024-09-10T19:30:58.042 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.6464000000000002e-05 2024-09-10T19:30:58.042 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:30:58.042 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:30:58.042 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-10T19:30:58.042 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:30:47.295490+0000", 2024-09-10T19:30:58.042 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00016752600000000001 2024-09-10T19:30:58.042 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:30:58.042 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:30:58.043 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 1,4", 2024-09-10T19:30:58.043 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:30:47.295611+0000", 2024-09-10T19:30:58.043 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000120246 2024-09-10T19:30:58.043 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:30:58.043 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:30:58.043 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-09-10T19:30:58.043 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:30:47.298031+0000", 2024-09-10T19:30:58.043 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0024200720000000001 2024-09-10T19:30:58.043 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:30:58.043 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:30:58.043 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:30:58.043 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:30:47.301367+0000", 2024-09-10T19:30:58.043 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0033367200000000001 2024-09-10T19:30:58.044 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:30:58.044 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:30:58.044 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:30:58.044 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:30:47.667403+0000", 2024-09-10T19:30:58.044 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.36603518099999999 2024-09-10T19:30:58.044 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:30:58.044 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:30:58.044 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-09-10T19:30:58.044 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:30:47.667509+0000", 2024-09-10T19:30:58.044 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000106495 2024-09-10T19:30:58.044 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:30:58.044 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:30:58.044 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-10T19:30:58.045 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:30:47.667553+0000", 2024-09-10T19:30:58.045 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.4332999999999997e-05 2024-09-10T19:30:58.045 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:30:58.045 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-10T19:30:58.045 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:30:58.045 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:30:58.045 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:30:58.045 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.16026.0:263551 11.7 11:e39cef4f:::benchmark_data_smithi183_203944_object118661:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e419)", 2024-09-10T19:30:58.045 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-10T19:30:47.309582+0000", 2024-09-10T19:30:58.045 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 10.652938731000001, 2024-09-10T19:30:58.045 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.358108488, 2024-09-10T19:30:58.045 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-10T19:30:58.045 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T19:30:58.045 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-10T19:30:58.046 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.16026", 2024-09-10T19:30:58.046 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.183:0/1450567484", 2024-09-10T19:30:58.046 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 263551 2024-09-10T19:30:58.046 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:30:58.046 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-10T19:30:58.046 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:30:58.046 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-10T19:30:58.046 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:30:47.309582+0000", 2024-09-10T19:30:58.046 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:30:58.046 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:30:58.046 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:30:58.046 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-10T19:30:58.046 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:30:47.309582+0000", 2024-09-10T19:30:58.047 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:30:58.047 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:30:58.047 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:30:58.047 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-10T19:30:58.047 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:30:47.309580+0000", 2024-09-10T19:30:58.047 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999981 2024-09-10T19:30:58.047 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:30:58.047 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:30:58.047 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-10T19:30:58.047 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:30:47.309587+0000", 2024-09-10T19:30:58.047 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.9149999999999999e-06 2024-09-10T19:30:58.047 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:30:58.047 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:30:58.048 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-10T19:30:58.048 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:30:47.309600+0000", 2024-09-10T19:30:58.048 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.3243e-05 2024-09-10T19:30:58.048 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:30:58.048 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:30:58.048 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-10T19:30:58.048 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:30:47.309610+0000", 2024-09-10T19:30:58.048 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.927e-06 2024-09-10T19:30:58.048 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:30:58.048 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:30:58.048 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:30:58.048 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:30:47.309653+0000", 2024-09-10T19:30:58.048 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.3189999999999998e-05 2024-09-10T19:30:58.049 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:30:58.049 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:30:58.049 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-10T19:30:58.049 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:30:47.309735+0000", 2024-09-10T19:30:58.049 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.1345000000000001e-05 2024-09-10T19:30:58.049 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:30:58.049 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:30:58.049 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 1,4", 2024-09-10T19:30:58.049 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:30:47.309827+0000", 2024-09-10T19:30:58.049 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.1906999999999999e-05 2024-09-10T19:30:58.049 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:30:58.049 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:30:58.049 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-09-10T19:30:58.050 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:30:47.313218+0000", 2024-09-10T19:30:58.050 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0033914079999999998 2024-09-10T19:30:58.050 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:30:58.050 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:30:58.050 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:30:58.050 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:30:47.315018+0000", 2024-09-10T19:30:58.050 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.001800002 2024-09-10T19:30:58.050 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:30:58.050 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:30:58.050 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:30:58.050 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:30:47.667604+0000", 2024-09-10T19:30:58.050 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.35258634300000002 2024-09-10T19:30:58.050 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:30:58.050 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:30:58.051 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-09-10T19:30:58.051 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:30:47.667655+0000", 2024-09-10T19:30:58.051 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.0448999999999998e-05 2024-09-10T19:30:58.051 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:30:58.051 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:30:58.051 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-10T19:30:58.051 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:30:47.667690+0000", 2024-09-10T19:30:58.051 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.5515999999999997e-05 2024-09-10T19:30:58.051 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:30:58.051 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-10T19:30:58.051 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:30:58.051 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:30:58.051 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:30:58.052 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.16026.0:263552 11.7 11:e702782e:::benchmark_data_smithi183_203944_object118662:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e419)", 2024-09-10T19:30:58.052 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-10T19:30:47.309623+0000", 2024-09-10T19:30:58.052 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 10.652897167000001, 2024-09-10T19:30:58.052 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.36473556600000001, 2024-09-10T19:30:58.052 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-10T19:30:58.052 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T19:30:58.052 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-10T19:30:58.052 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.16026", 2024-09-10T19:30:58.052 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.183:0/1450567484", 2024-09-10T19:30:58.052 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 263552 2024-09-10T19:30:58.052 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:30:58.052 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-10T19:30:58.052 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:30:58.053 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-10T19:30:58.053 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:30:47.309623+0000", 2024-09-10T19:30:58.053 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:30:58.053 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:30:58.053 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:30:58.053 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-10T19:30:58.053 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:30:47.309623+0000", 2024-09-10T19:30:58.053 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:30:58.053 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:30:58.053 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:30:58.053 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-10T19:30:58.053 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:30:47.309623+0000", 2024-09-10T19:30:58.053 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999995 2024-09-10T19:30:58.053 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:30:58.054 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:30:58.054 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-10T19:30:58.054 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:30:47.309626+0000", 2024-09-10T19:30:58.054 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.0690000000000002e-06 2024-09-10T19:30:58.054 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:30:58.054 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:30:58.054 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-10T19:30:58.054 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:30:47.309632+0000", 2024-09-10T19:30:58.054 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.4960000000000001e-06 2024-09-10T19:30:58.054 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:30:58.054 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:30:58.054 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-10T19:30:58.054 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:30:47.309637+0000", 2024-09-10T19:30:58.055 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.352e-06 2024-09-10T19:30:58.055 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:30:58.055 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:30:58.055 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:30:58.055 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:30:47.311840+0000", 2024-09-10T19:30:58.055 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0022034770000000001 2024-09-10T19:30:58.055 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:30:58.055 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:30:58.055 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-10T19:30:58.055 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:30:47.311914+0000", 2024-09-10T19:30:58.055 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.3708999999999995e-05 2024-09-10T19:30:58.055 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:30:58.055 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:30:58.056 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 1,4", 2024-09-10T19:30:58.056 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:30:47.311978+0000", 2024-09-10T19:30:58.056 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.4152999999999994e-05 2024-09-10T19:30:58.056 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:30:58.056 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:30:58.056 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-09-10T19:30:58.056 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:30:47.313307+0000", 2024-09-10T19:30:58.056 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0013284460000000001 2024-09-10T19:30:58.056 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:30:58.056 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:30:58.056 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:30:58.056 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:30:47.315817+0000", 2024-09-10T19:30:58.056 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.002510474 2024-09-10T19:30:58.057 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:30:58.057 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:30:58.057 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:30:58.057 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:30:47.674218+0000", 2024-09-10T19:30:58.057 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.35840088399999998 2024-09-10T19:30:58.057 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:30:58.057 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:30:58.057 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-09-10T19:30:58.057 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:30:47.674318+0000", 2024-09-10T19:30:58.057 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000100622 2024-09-10T19:30:58.057 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:30:58.057 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:30:58.057 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-10T19:30:58.057 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:30:47.674359+0000", 2024-09-10T19:30:58.058 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.0386000000000003e-05 2024-09-10T19:30:58.058 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:30:58.058 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-10T19:30:58.058 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:30:58.058 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:30:58.058 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:30:58.058 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.16026.0:263563 11.7 11:e3738614:::benchmark_data_smithi183_203944_object118673:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e419)", 2024-09-10T19:30:58.058 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-10T19:30:47.320335+0000", 2024-09-10T19:30:58.058 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 10.642185358999999, 2024-09-10T19:30:58.058 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.35416735700000002, 2024-09-10T19:30:58.058 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-10T19:30:58.058 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T19:30:58.058 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-10T19:30:58.059 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.16026", 2024-09-10T19:30:58.059 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.183:0/1450567484", 2024-09-10T19:30:58.059 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 263563 2024-09-10T19:30:58.059 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:30:58.059 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-10T19:30:58.059 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:30:58.059 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-10T19:30:58.059 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:30:47.320335+0000", 2024-09-10T19:30:58.059 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:30:58.059 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:30:58.059 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:30:58.059 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-10T19:30:58.059 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:30:47.320335+0000", 2024-09-10T19:30:58.059 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:30:58.060 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:30:58.060 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:30:58.060 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-10T19:30:58.060 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:30:47.320334+0000", 2024-09-10T19:30:58.060 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.999999 2024-09-10T19:30:58.060 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:30:58.060 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:30:58.060 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-10T19:30:58.060 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:30:47.320340+0000", 2024-09-10T19:30:58.060 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.224e-06 2024-09-10T19:30:58.060 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:30:58.060 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:30:58.060 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-10T19:30:58.061 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:30:47.320352+0000", 2024-09-10T19:30:58.061 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.1497e-05 2024-09-10T19:30:58.061 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:30:58.061 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:30:58.061 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-10T19:30:58.061 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:30:47.320358+0000", 2024-09-10T19:30:58.061 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.8000000000000004e-06 2024-09-10T19:30:58.061 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:30:58.061 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:30:58.061 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:30:58.061 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:30:47.320392+0000", 2024-09-10T19:30:58.061 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.4737999999999998e-05 2024-09-10T19:30:58.061 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:30:58.061 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:30:58.062 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-10T19:30:58.062 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:30:47.320467+0000", 2024-09-10T19:30:58.062 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.4859999999999998e-05 2024-09-10T19:30:58.062 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:30:58.062 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:30:58.062 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 1,4", 2024-09-10T19:30:58.062 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:30:47.320537+0000", 2024-09-10T19:30:58.062 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.9611999999999997e-05 2024-09-10T19:30:58.062 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:30:58.062 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:30:58.062 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-09-10T19:30:58.063 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:30:47.322352+0000", 2024-09-10T19:30:58.063 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0018150810000000001 2024-09-10T19:30:58.063 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:30:58.063 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:30:58.063 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:30:58.063 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:30:47.322776+0000", 2024-09-10T19:30:58.063 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000423886 2024-09-10T19:30:58.063 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:30:58.063 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:30:58.063 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:30:58.063 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:30:47.674402+0000", 2024-09-10T19:30:58.063 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.35162591500000001 2024-09-10T19:30:58.063 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:30:58.064 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:30:58.064 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-09-10T19:30:58.064 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:30:47.674469+0000", 2024-09-10T19:30:58.064 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.7657999999999995e-05 2024-09-10T19:30:58.064 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:30:58.064 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:30:58.064 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-10T19:30:58.064 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:30:47.674502+0000", 2024-09-10T19:30:58.064 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.3176999999999998e-05 2024-09-10T19:30:58.064 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:30:58.064 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-10T19:30:58.064 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:30:58.064 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:30:58.064 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:30:58.065 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.16026.0:275565 11.7 11:eb0ecfa5:::benchmark_data_smithi183_203944_object130675:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e422)", 2024-09-10T19:30:58.065 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-10T19:30:56.401299+0000", 2024-09-10T19:30:58.065 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 1.5612210069999999, 2024-09-10T19:30:58.065 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.52892819499999999, 2024-09-10T19:30:58.065 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-10T19:30:58.065 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T19:30:58.065 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-10T19:30:58.065 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.16026", 2024-09-10T19:30:58.065 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.183:0/1450567484", 2024-09-10T19:30:58.065 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 275565 2024-09-10T19:30:58.065 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:30:58.065 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-10T19:30:58.065 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:30:58.066 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-10T19:30:58.066 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:30:56.401299+0000", 2024-09-10T19:30:58.066 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:30:58.066 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:30:58.066 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:30:58.066 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-10T19:30:58.066 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:30:56.401299+0000", 2024-09-10T19:30:58.066 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:30:58.066 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:30:58.066 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:30:58.066 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-10T19:30:58.066 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:30:56.401297+0000", 2024-09-10T19:30:58.066 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999981 2024-09-10T19:30:58.066 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:30:58.067 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:30:58.067 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-10T19:30:58.067 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:30:56.401304+0000", 2024-09-10T19:30:58.067 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.9999999999999999e-06 2024-09-10T19:30:58.067 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:30:58.067 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:30:58.067 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-10T19:30:58.067 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:30:56.401318+0000", 2024-09-10T19:30:58.067 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.3614e-05 2024-09-10T19:30:58.067 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:30:58.067 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:30:58.067 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-10T19:30:58.067 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:30:56.401325+0000", 2024-09-10T19:30:58.067 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.443e-06 2024-09-10T19:30:58.068 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:30:58.068 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:30:58.068 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:30:58.068 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:30:56.401375+0000", 2024-09-10T19:30:58.068 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.0831999999999997e-05 2024-09-10T19:30:58.068 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:30:58.068 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:30:58.068 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-10T19:30:58.068 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:30:56.401477+0000", 2024-09-10T19:30:58.068 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000101856 2024-09-10T19:30:58.068 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:30:58.068 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:30:58.069 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 1,4", 2024-09-10T19:30:58.069 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:30:56.401577+0000", 2024-09-10T19:30:58.069 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.9815000000000006e-05 2024-09-10T19:30:58.069 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:30:58.070 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:30:58.070 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-09-10T19:30:58.070 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:30:56.404387+0000", 2024-09-10T19:30:58.070 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0028094679999999999 2024-09-10T19:30:58.070 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:30:58.070 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:30:58.070 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:30:58.070 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:30:56.404423+0000", 2024-09-10T19:30:58.070 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.6040000000000001e-05 2024-09-10T19:30:58.070 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:30:58.070 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:30:58.070 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:30:58.071 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:30:56.930156+0000", 2024-09-10T19:30:58.071 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.52573330799999995 2024-09-10T19:30:58.071 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:30:58.071 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:30:58.071 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-09-10T19:30:58.071 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:30:56.930195+0000", 2024-09-10T19:30:58.071 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.9580999999999999e-05 2024-09-10T19:30:58.071 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:30:58.071 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:30:58.071 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-10T19:30:58.071 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:30:56.930228+0000", 2024-09-10T19:30:58.071 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.2228999999999999e-05 2024-09-10T19:30:58.071 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:30:58.072 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-10T19:30:58.072 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:30:58.072 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:30:58.072 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:30:58.072 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_repop(client.16026.0:275562 11.a e422/374 11:5712aefe:::benchmark_data_smithi183_203944_object130672:head v 422'17479, mlcod=422'17473)", 2024-09-10T19:30:58.072 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-10T19:30:56.404864+0000", 2024-09-10T19:30:58.072 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 1.557656036, 2024-09-10T19:30:58.072 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.52734995200000001, 2024-09-10T19:30:58.072 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-10T19:30:58.072 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T19:30:58.072 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-10T19:30:58.072 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:30:58.072 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-10T19:30:58.073 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:30:56.404864+0000", 2024-09-10T19:30:58.073 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:30:58.073 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:30:58.073 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:30:58.073 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-10T19:30:58.073 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:30:56.404864+0000", 2024-09-10T19:30:58.073 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:30:58.073 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:30:58.073 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:30:58.073 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-10T19:30:58.073 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:30:56.404864+0000", 2024-09-10T19:30:58.073 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999995 2024-09-10T19:30:58.073 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:30:58.074 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:30:58.074 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-10T19:30:58.074 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:30:56.404870+0000", 2024-09-10T19:30:58.074 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.6900000000000003e-06 2024-09-10T19:30:58.074 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:30:58.074 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:30:58.074 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-10T19:30:58.074 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:30:56.404881+0000", 2024-09-10T19:30:58.074 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.0829000000000001e-05 2024-09-10T19:30:58.074 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:30:58.074 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:30:58.074 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-10T19:30:58.074 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:30:56.405155+0000", 2024-09-10T19:30:58.074 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00027342499999999999 2024-09-10T19:30:58.075 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:30:58.075 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:30:58.075 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:30:58.075 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:30:56.927445+0000", 2024-09-10T19:30:58.075 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.52228988200000004 2024-09-10T19:30:58.075 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:30:58.075 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:30:58.075 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-10T19:30:58.075 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:30:56.927470+0000", 2024-09-10T19:30:58.075 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.5046e-05 2024-09-10T19:30:58.075 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:30:58.075 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:30:58.075 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-09-10T19:30:58.075 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:30:56.932162+0000", 2024-09-10T19:30:58.075 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0046926670000000002 2024-09-10T19:30:58.075 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:30:58.076 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:30:58.076 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-10T19:30:58.076 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:30:56.932214+0000", 2024-09-10T19:30:58.076 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.2125e-05 2024-09-10T19:30:58.076 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:30:58.076 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-10T19:30:58.076 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:30:58.076 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:30:58.076 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:30:58.076 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.16026.0:275571 11.7 11:e3729467:::benchmark_data_smithi183_203944_object130681:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e422)", 2024-09-10T19:30:58.076 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-10T19:30:56.404879+0000", 2024-09-10T19:30:58.076 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 1.557641082, 2024-09-10T19:30:58.076 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.528586793, 2024-09-10T19:30:58.076 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-10T19:30:58.077 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T19:30:58.077 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-10T19:30:58.077 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.16026", 2024-09-10T19:30:58.077 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.183:0/1450567484", 2024-09-10T19:30:58.077 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 275571 2024-09-10T19:30:58.077 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:30:58.077 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-10T19:30:58.077 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:30:58.077 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-10T19:30:58.077 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:30:56.404879+0000", 2024-09-10T19:30:58.077 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:30:58.077 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:30:58.077 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:30:58.077 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-10T19:30:58.078 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:30:56.404879+0000", 2024-09-10T19:30:58.078 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:30:58.078 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:30:58.078 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:30:58.078 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-10T19:30:58.078 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:30:56.404878+0000", 2024-09-10T19:30:58.078 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999981 2024-09-10T19:30:58.078 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:30:58.078 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:30:58.078 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-10T19:30:58.078 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:30:56.404887+0000", 2024-09-10T19:30:58.078 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.6690000000000001e-06 2024-09-10T19:30:58.078 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:30:58.078 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:30:58.078 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-10T19:30:58.078 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:30:56.404900+0000", 2024-09-10T19:30:58.079 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.2707e-05 2024-09-10T19:30:58.079 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:30:58.079 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:30:58.079 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-10T19:30:58.079 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:30:56.404906+0000", 2024-09-10T19:30:58.079 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.0220000000000001e-06 2024-09-10T19:30:58.079 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:30:58.079 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:30:58.079 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:30:58.079 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:30:56.404949+0000", 2024-09-10T19:30:58.079 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.3034999999999997e-05 2024-09-10T19:30:58.079 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:30:58.079 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:30:58.079 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-10T19:30:58.079 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:30:56.405051+0000", 2024-09-10T19:30:58.080 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000102074 2024-09-10T19:30:58.080 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:30:58.080 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:30:58.080 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 1,4", 2024-09-10T19:30:58.080 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:30:56.405157+0000", 2024-09-10T19:30:58.080 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00010591599999999999 2024-09-10T19:30:58.080 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:30:58.080 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:30:58.080 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:30:58.080 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:30:56.932970+0000", 2024-09-10T19:30:58.080 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.52781254399999999 2024-09-10T19:30:58.080 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:30:58.080 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:30:58.080 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-09-10T19:30:58.080 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:30:56.932981+0000", 2024-09-10T19:30:58.081 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.1596e-05 2024-09-10T19:30:58.081 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:30:58.081 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:30:58.081 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:30:58.081 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:30:56.933377+0000", 2024-09-10T19:30:58.081 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00039562899999999999 2024-09-10T19:30:58.081 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:30:58.081 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:30:58.081 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-09-10T19:30:58.081 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:30:56.933423+0000", 2024-09-10T19:30:58.081 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.6097000000000003e-05 2024-09-10T19:30:58.081 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:30:58.081 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:30:58.081 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-10T19:30:58.082 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:30:56.933466+0000", 2024-09-10T19:30:58.082 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.3324000000000001e-05 2024-09-10T19:30:58.082 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:30:58.082 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-10T19:30:58.082 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:30:58.082 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:30:58.082 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:30:58.082 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_repop_reply(client.16026.0:275565 11.7 e422/371 ondisk, result = 0)", 2024-09-10T19:30:58.082 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-10T19:30:56.405226+0000", 2024-09-10T19:30:58.082 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 1.5572944710000001, 2024-09-10T19:30:58.082 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.52500909699999998, 2024-09-10T19:30:58.082 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-10T19:30:58.082 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "started", 2024-09-10T19:30:58.083 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-10T19:30:58.083 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:30:58.083 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-10T19:30:58.083 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:30:56.405226+0000", 2024-09-10T19:30:58.083 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:30:58.083 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:30:58.083 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:30:58.083 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-10T19:30:58.083 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:30:56.405226+0000", 2024-09-10T19:30:58.083 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:30:58.083 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:30:58.083 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:30:58.083 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-10T19:30:58.084 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:30:56.405226+0000", 2024-09-10T19:30:58.084 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999995 2024-09-10T19:30:58.084 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:30:58.084 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:30:58.084 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-10T19:30:58.084 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:30:56.405229+0000", 2024-09-10T19:30:58.084 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.941e-06 2024-09-10T19:30:58.084 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:30:58.084 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:30:58.084 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-10T19:30:58.084 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:30:56.927386+0000", 2024-09-10T19:30:58.084 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.52215652999999995 2024-09-10T19:30:58.085 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:30:58.085 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:30:58.085 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-10T19:30:58.085 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:30:56.927401+0000", 2024-09-10T19:30:58.085 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.5007e-05 2024-09-10T19:30:58.085 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:30:58.085 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:30:58.085 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:30:58.085 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:30:56.930144+0000", 2024-09-10T19:30:58.085 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0027431199999999999 2024-09-10T19:30:58.085 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:30:58.085 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:30:58.085 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-10T19:30:58.085 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:30:56.930151+0000", 2024-09-10T19:30:58.085 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.7000000000000002e-06 2024-09-10T19:30:58.086 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:30:58.086 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:30:58.086 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-10T19:30:58.086 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:30:56.930235+0000", 2024-09-10T19:30:58.086 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.4246999999999995e-05 2024-09-10T19:30:58.086 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:30:58.086 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-10T19:30:58.086 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:30:58.086 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:30:58.086 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:30:58.086 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.16026.0:275575 11.7 11:e304d8ff:::benchmark_data_smithi183_203944_object130685:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e422)", 2024-09-10T19:30:58.086 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-10T19:30:56.407436+0000", 2024-09-10T19:30:58.086 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 1.5550846919999999, 2024-09-10T19:30:58.086 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.52623960800000003, 2024-09-10T19:30:58.087 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-10T19:30:58.087 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T19:30:58.087 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-10T19:30:58.087 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.16026", 2024-09-10T19:30:58.087 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.183:0/1450567484", 2024-09-10T19:30:58.087 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 275575 2024-09-10T19:30:58.087 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:30:58.087 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-10T19:30:58.087 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:30:58.087 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-10T19:30:58.087 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:30:56.407436+0000", 2024-09-10T19:30:58.087 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:30:58.087 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:30:58.087 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:30:58.088 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-10T19:30:58.088 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:30:56.407436+0000", 2024-09-10T19:30:58.088 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:30:58.088 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:30:58.088 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:30:58.088 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-10T19:30:58.088 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:30:56.407435+0000", 2024-09-10T19:30:58.088 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.999999 2024-09-10T19:30:58.088 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:30:58.088 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:30:58.088 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-10T19:30:58.088 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:30:56.407440+0000", 2024-09-10T19:30:58.088 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.2680000000000002e-06 2024-09-10T19:30:58.088 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:30:58.089 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:30:58.089 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-10T19:30:58.089 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:30:56.927378+0000", 2024-09-10T19:30:58.089 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.51993773200000004 2024-09-10T19:30:58.089 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:30:58.089 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:30:58.089 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-10T19:30:58.089 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:30:56.927399+0000", 2024-09-10T19:30:58.089 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.1401000000000001e-05 2024-09-10T19:30:58.089 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:30:58.089 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:30:58.089 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:30:58.089 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:30:56.930272+0000", 2024-09-10T19:30:58.089 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0028732639999999999 2024-09-10T19:30:58.090 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:30:58.090 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:30:58.090 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-10T19:30:58.090 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:30:56.930376+0000", 2024-09-10T19:30:58.090 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000103231 2024-09-10T19:30:58.090 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:30:58.090 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:30:58.090 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 1,4", 2024-09-10T19:30:58.090 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:30:56.930483+0000", 2024-09-10T19:30:58.090 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000107611 2024-09-10T19:30:58.090 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:30:58.090 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:30:58.090 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:30:58.090 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:30:56.933013+0000", 2024-09-10T19:30:58.090 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0025298669999999999 2024-09-10T19:30:58.091 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:30:58.091 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:30:58.091 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-09-10T19:30:58.091 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:30:56.933096+0000", 2024-09-10T19:30:58.091 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.2755000000000003e-05 2024-09-10T19:30:58.091 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:30:58.091 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:30:58.091 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:30:58.091 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:30:56.933627+0000", 2024-09-10T19:30:58.091 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00053062900000000002 2024-09-10T19:30:58.091 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:30:58.091 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:30:58.091 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-09-10T19:30:58.091 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:30:56.933655+0000", 2024-09-10T19:30:58.091 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.8272e-05 2024-09-10T19:30:58.092 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:30:58.092 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:30:58.092 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-10T19:30:58.092 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:30:56.933675+0000", 2024-09-10T19:30:58.092 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.0562999999999999e-05 2024-09-10T19:30:58.092 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:30:58.092 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-10T19:30:58.092 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:30:58.092 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:30:58.092 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-10T19:30:58.092 INFO:teuthology.orchestra.run.smithi191.stdout:} 2024-09-10T19:30:58.103 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:30:58.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:30:58.104 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:30:58.205 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:30:58.239 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:30:58.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:30:58.240 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:30:58.341 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:30:58.376 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:30:58.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:30:58.377 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:30:58.478 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:30:58.501 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:30:58.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:30:58.502 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:30:58.603 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:30:58.635 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:30:58.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:30:58.636 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:30:58.737 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:30:58.774 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:30:58.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:30:58.775 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:30:58.875 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:30:58.911 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:30:58.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:30:58.912 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:30:59.013 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:30:59.049 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:30:59.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:30:59.050 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:30:59.151 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:30:59.181 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:30:59.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:30:59.183 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:30:59.283 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:30:59.317 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:30:59.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:30:59.318 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:30:59.419 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:30:59.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:30:59 smithi183 ceph-mon[29736]: pgmap v959: 158 pgs: 158 active+clean; 955 MiB data, 8.3 GiB used, 618 GiB / 626 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.33k op/s 2024-09-10T19:30:59.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:30:59 smithi183 ceph-mon[29736]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:30:59.431 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:30:59 smithi183 ceph-mon[41562]: pgmap v959: 158 pgs: 158 active+clean; 955 MiB data, 8.3 GiB used, 618 GiB / 626 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.33k op/s 2024-09-10T19:30:59.431 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:30:59 smithi183 ceph-mon[41562]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:30:59.447 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:30:59.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:30:59.448 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:30:59.465 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:30:59 smithi191 ceph-mon[32884]: pgmap v959: 158 pgs: 158 active+clean; 955 MiB data, 8.3 GiB used, 618 GiB / 626 GiB avail; 1.7 KiB/s rd, 0 B/s wr, 1.33k op/s 2024-09-10T19:30:59.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:30:59 smithi191 ceph-mon[32884]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:30:59.549 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:30:59.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:30:59.584 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:30:59.584 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:30:59.685 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:30:59.724 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:30:59.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:30:59.725 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:30:59.826 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:30:59.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:30:59.880 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:30:59.880 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:30:59.981 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:31:00.017 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:31:00.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:31:00.018 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:31:00.063 DEBUG:teuthology.orchestra.run.smithi191:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph daemon osd.6 dump_ops_in_flight 2024-09-10T19:31:00.119 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:31:00.155 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:31:00.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:31:00.156 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:31:00.257 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:31:00.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:31:00.292 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:31:00.292 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:31:00.393 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:31:00.403 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:31:00.416 INFO:teuthology.orchestra.run.smithi183.stdout:osd.0: {} 2024-09-10T19:31:00.416 INFO:teuthology.orchestra.run.smithi183.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-10T19:31:00.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:31:00.437 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:31:00.437 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:31:00.451 INFO:teuthology.orchestra.run.smithi183.stdout:osd.1: {} 2024-09-10T19:31:00.451 INFO:teuthology.orchestra.run.smithi183.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-10T19:31:00.465 INFO:journalctl@ceph.iscsi.iscsi.a.smithi191.stdout:Sep 10 19:31:00 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-iscsi-iscsi-a[61816]: debug there is no tcmu-runner data available 2024-09-10T19:31:00.478 INFO:teuthology.orchestra.run.smithi183.stdout:osd.2: {} 2024-09-10T19:31:00.478 INFO:teuthology.orchestra.run.smithi183.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-10T19:31:00.499 INFO:teuthology.orchestra.run.smithi183.stdout:osd.3: {} 2024-09-10T19:31:00.499 INFO:teuthology.orchestra.run.smithi183.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-10T19:31:00.513 INFO:teuthology.orchestra.run.smithi191.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.b/config 2024-09-10T19:31:00.531 INFO:teuthology.orchestra.run.smithi183.stdout:osd.4: {} 2024-09-10T19:31:00.532 INFO:teuthology.orchestra.run.smithi183.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-10T19:31:00.538 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:31:00.561 INFO:teuthology.orchestra.run.smithi183.stdout:osd.5: {} 2024-09-10T19:31:00.561 INFO:teuthology.orchestra.run.smithi183.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-10T19:31:00.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:31:00.590 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:31:00.590 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:31:00.604 INFO:teuthology.orchestra.run.smithi183.stdout:osd.6: {} 2024-09-10T19:31:00.604 INFO:teuthology.orchestra.run.smithi183.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-10T19:31:00.630 INFO:teuthology.orchestra.run.smithi183.stdout:osd.7: {} 2024-09-10T19:31:00.631 INFO:teuthology.orchestra.run.smithi183.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-10T19:31:00.690 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:31:00.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:31:00.728 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:31:00.728 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:31:00.829 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:31:00.856 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:31:00.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:31:00.857 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:31:00.958 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:31:00.995 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:31:00.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:31:00.996 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:31:01.096 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:31:01.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:31:01.136 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:31:01.136 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:31:01.237 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:31:01.245 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph osd unset noscrub 2024-09-10T19:31:01.268 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:31:01.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:31:01.269 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:31:01.298 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:31:01 smithi183 ceph-mon[29736]: pgmap v960: 158 pgs: 158 active+clean; 908 MiB data, 8.1 GiB used, 618 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 1.25k op/s 2024-09-10T19:31:01.299 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:31:01 smithi183 ceph-mon[29736]: from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:31:01.299 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:31:01 smithi183 ceph-mon[29736]: from='client.? 172.21.15.183:0/706459357' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T19:31:01.299 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:31:01 smithi183 ceph-mon[41562]: pgmap v960: 158 pgs: 158 active+clean; 908 MiB data, 8.1 GiB used, 618 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 1.25k op/s 2024-09-10T19:31:01.299 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:31:01 smithi183 ceph-mon[41562]: from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:31:01.299 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:31:01 smithi183 ceph-mon[41562]: from='client.? 172.21.15.183:0/706459357' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T19:31:01.370 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:31:01.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:31:01.401 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:31:01.401 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:31:01.465 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:31:01 smithi191 ceph-mon[32884]: pgmap v960: 158 pgs: 158 active+clean; 908 MiB data, 8.1 GiB used, 618 GiB / 626 GiB avail; 853 B/s rd, 0 B/s wr, 1.25k op/s 2024-09-10T19:31:01.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:31:01 smithi191 ceph-mon[32884]: from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:31:01.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:31:01 smithi191 ceph-mon[32884]: from='client.? 172.21.15.183:0/706459357' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T19:31:01.502 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:31:01.557 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:31:01.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:31:01.558 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:31:01.625 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Sep 10 19:31:01 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-y[29989]: ::ffff:172.21.15.191 - - [10/Sep/2024:19:31:01] "GET /metrics HTTP/1.1" 200 224921 "" "Prometheus/2.43.0" 2024-09-10T19:31:01.659 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:31:01.689 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:31:01.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:31:01.690 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:31:01.791 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:31:01.824 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:31:01.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:31:01.826 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:31:01.926 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:31:01.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:31:01.963 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:31:01.963 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:31:02.064 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:31:02.097 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:31:02.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:31:02.098 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:31:02.199 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:31:02.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:31:02.233 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:31:02.233 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:31:02.334 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:31:02.374 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:31:02.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:31:02.375 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:31:02.476 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:31:02.529 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:31:02.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:31:02.530 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:31:02.627 INFO:teuthology.orchestra.run.smithi191.stdout:ERROR: (22) Invalid argument 2024-09-10T19:31:02.627 INFO:teuthology.orchestra.run.smithi191.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-10T19:31:02.631 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:31:02.658 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:31:02.692 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:31:02.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:31:02.693 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:31:02.794 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:31:02.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:31:02.834 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:31:02.834 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:31:02.935 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:31:02.970 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:31:02.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:31:02.971 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:31:03.072 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:31:03.109 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:31:03.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:31:03.110 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:31:03.211 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:31:03.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:31:03.248 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:31:03.249 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:31:03.288 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:31:03 smithi191 ceph-mon[32884]: pgmap v961: 158 pgs: 158 active+clean; 662 MiB data, 7.5 GiB used, 618 GiB / 626 GiB avail; 652 B/s rd, 0 B/s wr, 1.45k op/s 2024-09-10T19:31:03.327 DEBUG:teuthology.orchestra.run.smithi191:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph daemon osd.6 dump_blocked_ops 2024-09-10T19:31:03.345 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:31:03 smithi183 ceph-mon[29736]: pgmap v961: 158 pgs: 158 active+clean; 662 MiB data, 7.5 GiB used, 618 GiB / 626 GiB avail; 652 B/s rd, 0 B/s wr, 1.45k op/s 2024-09-10T19:31:03.346 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:31:03 smithi183 ceph-mon[41562]: pgmap v961: 158 pgs: 158 active+clean; 662 MiB data, 7.5 GiB used, 618 GiB / 626 GiB avail; 652 B/s rd, 0 B/s wr, 1.45k op/s 2024-09-10T19:31:03.349 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:31:03.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:31:03.411 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:31:03.411 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:31:03.512 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:31:03.519 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-10T19:31:03.547 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:31:03.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:31:03.548 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:31:03.649 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:31:03.690 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:31:03.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:31:03.691 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:31:03.713 INFO:teuthology.orchestra.run.smithi191.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.b/config 2024-09-10T19:31:03.792 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:31:03.810 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-09-10T19:31:03.810 INFO:teuthology.orchestra.run.smithi183.stdout:{ 2024-09-10T19:31:03.810 INFO:teuthology.orchestra.run.smithi183.stdout: "epoch": 422, 2024-09-10T19:31:03.810 INFO:teuthology.orchestra.run.smithi183.stdout: "fsid": "320f6df8-6fa7-11ef-bcea-c7b262605968", 2024-09-10T19:31:03.810 INFO:teuthology.orchestra.run.smithi183.stdout: "created": "2024-09-10T19:05:03.688531+0000", 2024-09-10T19:31:03.810 INFO:teuthology.orchestra.run.smithi183.stdout: "modified": "2024-09-10T19:30:56.151400+0000", 2024-09-10T19:31:03.810 INFO:teuthology.orchestra.run.smithi183.stdout: "last_up_change": "2024-09-10T19:24:22.870873+0000", 2024-09-10T19:31:03.810 INFO:teuthology.orchestra.run.smithi183.stdout: "last_in_change": "2024-09-10T19:27:51.975891+0000", 2024-09-10T19:31:03.810 INFO:teuthology.orchestra.run.smithi183.stdout: "flags": "noscrub,nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-09-10T19:31:03.810 INFO:teuthology.orchestra.run.smithi183.stdout: "flags_num": 5806080, 2024-09-10T19:31:03.811 INFO:teuthology.orchestra.run.smithi183.stdout: "flags_set": [ 2024-09-10T19:31:03.811 INFO:teuthology.orchestra.run.smithi183.stdout: "nodeep-scrub", 2024-09-10T19:31:03.811 INFO:teuthology.orchestra.run.smithi183.stdout: "noscrub", 2024-09-10T19:31:03.811 INFO:teuthology.orchestra.run.smithi183.stdout: "pglog_hardlimit", 2024-09-10T19:31:03.811 INFO:teuthology.orchestra.run.smithi183.stdout: "purged_snapdirs", 2024-09-10T19:31:03.811 INFO:teuthology.orchestra.run.smithi183.stdout: "recovery_deletes", 2024-09-10T19:31:03.811 INFO:teuthology.orchestra.run.smithi183.stdout: "sortbitwise" 2024-09-10T19:31:03.811 INFO:teuthology.orchestra.run.smithi183.stdout: ], 2024-09-10T19:31:03.811 INFO:teuthology.orchestra.run.smithi183.stdout: "crush_version": 21, 2024-09-10T19:31:03.811 INFO:teuthology.orchestra.run.smithi183.stdout: "full_ratio": 0.94999998807907104, 2024-09-10T19:31:03.811 INFO:teuthology.orchestra.run.smithi183.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-09-10T19:31:03.811 INFO:teuthology.orchestra.run.smithi183.stdout: "nearfull_ratio": 0.85000002384185791, 2024-09-10T19:31:03.811 INFO:teuthology.orchestra.run.smithi183.stdout: "cluster_snapshot": "", 2024-09-10T19:31:03.811 INFO:teuthology.orchestra.run.smithi183.stdout: "pool_max": 11, 2024-09-10T19:31:03.812 INFO:teuthology.orchestra.run.smithi183.stdout: "max_osd": 8, 2024-09-10T19:31:03.812 INFO:teuthology.orchestra.run.smithi183.stdout: "require_min_compat_client": "luminous", 2024-09-10T19:31:03.812 INFO:teuthology.orchestra.run.smithi183.stdout: "min_compat_client": "luminous", 2024-09-10T19:31:03.812 INFO:teuthology.orchestra.run.smithi183.stdout: "require_osd_release": "quincy", 2024-09-10T19:31:03.812 INFO:teuthology.orchestra.run.smithi183.stdout: "pools": [ 2024-09-10T19:31:03.812 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:31:03.812 INFO:teuthology.orchestra.run.smithi183.stdout: "pool": 1, 2024-09-10T19:31:03.812 INFO:teuthology.orchestra.run.smithi183.stdout: "pool_name": ".mgr", 2024-09-10T19:31:03.812 INFO:teuthology.orchestra.run.smithi183.stdout: "create_time": "2024-09-10T19:08:00.456786+0000", 2024-09-10T19:31:03.812 INFO:teuthology.orchestra.run.smithi183.stdout: "flags": 1, 2024-09-10T19:31:03.812 INFO:teuthology.orchestra.run.smithi183.stdout: "flags_names": "hashpspool", 2024-09-10T19:31:03.812 INFO:teuthology.orchestra.run.smithi183.stdout: "type": 1, 2024-09-10T19:31:03.812 INFO:teuthology.orchestra.run.smithi183.stdout: "size": 3, 2024-09-10T19:31:03.813 INFO:teuthology.orchestra.run.smithi183.stdout: "min_size": 2, 2024-09-10T19:31:03.813 INFO:teuthology.orchestra.run.smithi183.stdout: "crush_rule": 0, 2024-09-10T19:31:03.813 INFO:teuthology.orchestra.run.smithi183.stdout: "peering_crush_bucket_count": 0, 2024-09-10T19:31:03.813 INFO:teuthology.orchestra.run.smithi183.stdout: "peering_crush_bucket_target": 0, 2024-09-10T19:31:03.813 INFO:teuthology.orchestra.run.smithi183.stdout: "peering_crush_bucket_barrier": 0, 2024-09-10T19:31:03.813 INFO:teuthology.orchestra.run.smithi183.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-10T19:31:03.813 INFO:teuthology.orchestra.run.smithi183.stdout: "object_hash": 2, 2024-09-10T19:31:03.814 INFO:teuthology.orchestra.run.smithi183.stdout: "pg_autoscale_mode": "off", 2024-09-10T19:31:03.814 INFO:teuthology.orchestra.run.smithi183.stdout: "pg_num": 11, 2024-09-10T19:31:03.814 INFO:teuthology.orchestra.run.smithi183.stdout: "pg_placement_num": 11, 2024-09-10T19:31:03.814 INFO:teuthology.orchestra.run.smithi183.stdout: "pg_placement_num_target": 11, 2024-09-10T19:31:03.814 INFO:teuthology.orchestra.run.smithi183.stdout: "pg_num_target": 11, 2024-09-10T19:31:03.814 INFO:teuthology.orchestra.run.smithi183.stdout: "pg_num_pending": 11, 2024-09-10T19:31:03.814 INFO:teuthology.orchestra.run.smithi183.stdout: "last_pg_merge_meta": { 2024-09-10T19:31:03.814 INFO:teuthology.orchestra.run.smithi183.stdout: "source_pgid": "0.0", 2024-09-10T19:31:03.814 INFO:teuthology.orchestra.run.smithi183.stdout: "ready_epoch": 0, 2024-09-10T19:31:03.814 INFO:teuthology.orchestra.run.smithi183.stdout: "last_epoch_started": 0, 2024-09-10T19:31:03.815 INFO:teuthology.orchestra.run.smithi183.stdout: "last_epoch_clean": 0, 2024-09-10T19:31:03.815 INFO:teuthology.orchestra.run.smithi183.stdout: "source_version": "0'0", 2024-09-10T19:31:03.815 INFO:teuthology.orchestra.run.smithi183.stdout: "target_version": "0'0" 2024-09-10T19:31:03.815 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:31:03.815 INFO:teuthology.orchestra.run.smithi183.stdout: "last_change": "271", 2024-09-10T19:31:03.816 INFO:teuthology.orchestra.run.smithi183.stdout: "last_force_op_resend": "0", 2024-09-10T19:31:03.816 INFO:teuthology.orchestra.run.smithi183.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-10T19:31:03.816 INFO:teuthology.orchestra.run.smithi183.stdout: "last_force_op_resend_preluminous": "242", 2024-09-10T19:31:03.816 INFO:teuthology.orchestra.run.smithi183.stdout: "auid": 0, 2024-09-10T19:31:03.816 INFO:teuthology.orchestra.run.smithi183.stdout: "snap_mode": "selfmanaged", 2024-09-10T19:31:03.817 INFO:teuthology.orchestra.run.smithi183.stdout: "snap_seq": 0, 2024-09-10T19:31:03.817 INFO:teuthology.orchestra.run.smithi183.stdout: "snap_epoch": 0, 2024-09-10T19:31:03.817 INFO:teuthology.orchestra.run.smithi183.stdout: "pool_snaps": [], 2024-09-10T19:31:03.817 INFO:teuthology.orchestra.run.smithi183.stdout: "removed_snaps": "[]", 2024-09-10T19:31:03.817 INFO:teuthology.orchestra.run.smithi183.stdout: "quota_max_bytes": 0, 2024-09-10T19:31:03.818 INFO:teuthology.orchestra.run.smithi183.stdout: "quota_max_objects": 0, 2024-09-10T19:31:03.818 INFO:teuthology.orchestra.run.smithi183.stdout: "tiers": [], 2024-09-10T19:31:03.818 INFO:teuthology.orchestra.run.smithi183.stdout: "tier_of": -1, 2024-09-10T19:31:03.818 INFO:teuthology.orchestra.run.smithi183.stdout: "read_tier": -1, 2024-09-10T19:31:03.818 INFO:teuthology.orchestra.run.smithi183.stdout: "write_tier": -1, 2024-09-10T19:31:03.818 INFO:teuthology.orchestra.run.smithi183.stdout: "cache_mode": "none", 2024-09-10T19:31:03.818 INFO:teuthology.orchestra.run.smithi183.stdout: "target_max_bytes": 0, 2024-09-10T19:31:03.818 INFO:teuthology.orchestra.run.smithi183.stdout: "target_max_objects": 0, 2024-09-10T19:31:03.818 INFO:teuthology.orchestra.run.smithi183.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-10T19:31:03.819 INFO:teuthology.orchestra.run.smithi183.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-10T19:31:03.819 INFO:teuthology.orchestra.run.smithi183.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-10T19:31:03.819 INFO:teuthology.orchestra.run.smithi183.stdout: "cache_min_flush_age": 0, 2024-09-10T19:31:03.819 INFO:teuthology.orchestra.run.smithi183.stdout: "cache_min_evict_age": 0, 2024-09-10T19:31:03.819 INFO:teuthology.orchestra.run.smithi183.stdout: "erasure_code_profile": "", 2024-09-10T19:31:03.819 INFO:teuthology.orchestra.run.smithi183.stdout: "hit_set_params": { 2024-09-10T19:31:03.819 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "none" 2024-09-10T19:31:03.819 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:31:03.819 INFO:teuthology.orchestra.run.smithi183.stdout: "hit_set_period": 0, 2024-09-10T19:31:03.819 INFO:teuthology.orchestra.run.smithi183.stdout: "hit_set_count": 0, 2024-09-10T19:31:03.820 INFO:teuthology.orchestra.run.smithi183.stdout: "use_gmt_hitset": true, 2024-09-10T19:31:03.820 INFO:teuthology.orchestra.run.smithi183.stdout: "min_read_recency_for_promote": 0, 2024-09-10T19:31:03.820 INFO:teuthology.orchestra.run.smithi183.stdout: "min_write_recency_for_promote": 0, 2024-09-10T19:31:03.820 INFO:teuthology.orchestra.run.smithi183.stdout: "hit_set_grade_decay_rate": 0, 2024-09-10T19:31:03.820 INFO:teuthology.orchestra.run.smithi183.stdout: "hit_set_search_last_n": 0, 2024-09-10T19:31:03.820 INFO:teuthology.orchestra.run.smithi183.stdout: "grade_table": [], 2024-09-10T19:31:03.821 INFO:teuthology.orchestra.run.smithi183.stdout: "stripe_width": 0, 2024-09-10T19:31:03.821 INFO:teuthology.orchestra.run.smithi183.stdout: "expected_num_objects": 0, 2024-09-10T19:31:03.821 INFO:teuthology.orchestra.run.smithi183.stdout: "fast_read": false, 2024-09-10T19:31:03.821 INFO:teuthology.orchestra.run.smithi183.stdout: "options": { 2024-09-10T19:31:03.821 INFO:teuthology.orchestra.run.smithi183.stdout: "pg_num_max": 32, 2024-09-10T19:31:03.821 INFO:teuthology.orchestra.run.smithi183.stdout: "pg_num_min": 1 2024-09-10T19:31:03.821 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:31:03.821 INFO:teuthology.orchestra.run.smithi183.stdout: "application_metadata": { 2024-09-10T19:31:03.821 INFO:teuthology.orchestra.run.smithi183.stdout: "mgr": {} 2024-09-10T19:31:03.821 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:31:03.821 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:31:03.822 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:31:03.822 INFO:teuthology.orchestra.run.smithi183.stdout: "pool": 2, 2024-09-10T19:31:03.822 INFO:teuthology.orchestra.run.smithi183.stdout: "pool_name": ".rgw.root", 2024-09-10T19:31:03.822 INFO:teuthology.orchestra.run.smithi183.stdout: "create_time": "2024-09-10T19:10:05.993763+0000", 2024-09-10T19:31:03.822 INFO:teuthology.orchestra.run.smithi183.stdout: "flags": 1, 2024-09-10T19:31:03.822 INFO:teuthology.orchestra.run.smithi183.stdout: "flags_names": "hashpspool", 2024-09-10T19:31:03.822 INFO:teuthology.orchestra.run.smithi183.stdout: "type": 1, 2024-09-10T19:31:03.822 INFO:teuthology.orchestra.run.smithi183.stdout: "size": 3, 2024-09-10T19:31:03.822 INFO:teuthology.orchestra.run.smithi183.stdout: "min_size": 2, 2024-09-10T19:31:03.822 INFO:teuthology.orchestra.run.smithi183.stdout: "crush_rule": 0, 2024-09-10T19:31:03.822 INFO:teuthology.orchestra.run.smithi183.stdout: "peering_crush_bucket_count": 0, 2024-09-10T19:31:03.823 INFO:teuthology.orchestra.run.smithi183.stdout: "peering_crush_bucket_target": 0, 2024-09-10T19:31:03.823 INFO:teuthology.orchestra.run.smithi183.stdout: "peering_crush_bucket_barrier": 0, 2024-09-10T19:31:03.823 INFO:teuthology.orchestra.run.smithi183.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-10T19:31:03.823 INFO:teuthology.orchestra.run.smithi183.stdout: "object_hash": 2, 2024-09-10T19:31:03.823 INFO:teuthology.orchestra.run.smithi183.stdout: "pg_autoscale_mode": "off", 2024-09-10T19:31:03.823 INFO:teuthology.orchestra.run.smithi183.stdout: "pg_num": 32, 2024-09-10T19:31:03.823 INFO:teuthology.orchestra.run.smithi183.stdout: "pg_placement_num": 32, 2024-09-10T19:31:03.823 INFO:teuthology.orchestra.run.smithi183.stdout: "pg_placement_num_target": 32, 2024-09-10T19:31:03.823 INFO:teuthology.orchestra.run.smithi183.stdout: "pg_num_target": 32, 2024-09-10T19:31:03.823 INFO:teuthology.orchestra.run.smithi183.stdout: "pg_num_pending": 32, 2024-09-10T19:31:03.823 INFO:teuthology.orchestra.run.smithi183.stdout: "last_pg_merge_meta": { 2024-09-10T19:31:03.823 INFO:teuthology.orchestra.run.smithi183.stdout: "source_pgid": "0.0", 2024-09-10T19:31:03.824 INFO:teuthology.orchestra.run.smithi183.stdout: "ready_epoch": 0, 2024-09-10T19:31:03.824 INFO:teuthology.orchestra.run.smithi183.stdout: "last_epoch_started": 0, 2024-09-10T19:31:03.824 INFO:teuthology.orchestra.run.smithi183.stdout: "last_epoch_clean": 0, 2024-09-10T19:31:03.824 INFO:teuthology.orchestra.run.smithi183.stdout: "source_version": "0'0", 2024-09-10T19:31:03.824 INFO:teuthology.orchestra.run.smithi183.stdout: "target_version": "0'0" 2024-09-10T19:31:03.824 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:31:03.824 INFO:teuthology.orchestra.run.smithi183.stdout: "last_change": "60", 2024-09-10T19:31:03.824 INFO:teuthology.orchestra.run.smithi183.stdout: "last_force_op_resend": "0", 2024-09-10T19:31:03.825 INFO:teuthology.orchestra.run.smithi183.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-10T19:31:03.825 INFO:teuthology.orchestra.run.smithi183.stdout: "last_force_op_resend_preluminous": "0", 2024-09-10T19:31:03.825 INFO:teuthology.orchestra.run.smithi183.stdout: "auid": 0, 2024-09-10T19:31:03.825 INFO:teuthology.orchestra.run.smithi183.stdout: "snap_mode": "selfmanaged", 2024-09-10T19:31:03.825 INFO:teuthology.orchestra.run.smithi183.stdout: "snap_seq": 0, 2024-09-10T19:31:03.825 INFO:teuthology.orchestra.run.smithi183.stdout: "snap_epoch": 0, 2024-09-10T19:31:03.825 INFO:teuthology.orchestra.run.smithi183.stdout: "pool_snaps": [], 2024-09-10T19:31:03.825 INFO:teuthology.orchestra.run.smithi183.stdout: "removed_snaps": "[]", 2024-09-10T19:31:03.825 INFO:teuthology.orchestra.run.smithi183.stdout: "quota_max_bytes": 0, 2024-09-10T19:31:03.825 INFO:teuthology.orchestra.run.smithi183.stdout: "quota_max_objects": 0, 2024-09-10T19:31:03.825 INFO:teuthology.orchestra.run.smithi183.stdout: "tiers": [], 2024-09-10T19:31:03.826 INFO:teuthology.orchestra.run.smithi183.stdout: "tier_of": -1, 2024-09-10T19:31:03.826 INFO:teuthology.orchestra.run.smithi183.stdout: "read_tier": -1, 2024-09-10T19:31:03.826 INFO:teuthology.orchestra.run.smithi183.stdout: "write_tier": -1, 2024-09-10T19:31:03.826 INFO:teuthology.orchestra.run.smithi183.stdout: "cache_mode": "none", 2024-09-10T19:31:03.826 INFO:teuthology.orchestra.run.smithi183.stdout: "target_max_bytes": 0, 2024-09-10T19:31:03.826 INFO:teuthology.orchestra.run.smithi183.stdout: "target_max_objects": 0, 2024-09-10T19:31:03.826 INFO:teuthology.orchestra.run.smithi183.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-10T19:31:03.826 INFO:teuthology.orchestra.run.smithi183.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-10T19:31:03.826 INFO:teuthology.orchestra.run.smithi183.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-10T19:31:03.826 INFO:teuthology.orchestra.run.smithi183.stdout: "cache_min_flush_age": 0, 2024-09-10T19:31:03.826 INFO:teuthology.orchestra.run.smithi183.stdout: "cache_min_evict_age": 0, 2024-09-10T19:31:03.826 INFO:teuthology.orchestra.run.smithi183.stdout: "erasure_code_profile": "", 2024-09-10T19:31:03.827 INFO:teuthology.orchestra.run.smithi183.stdout: "hit_set_params": { 2024-09-10T19:31:03.827 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "none" 2024-09-10T19:31:03.827 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:31:03.827 INFO:teuthology.orchestra.run.smithi183.stdout: "hit_set_period": 0, 2024-09-10T19:31:03.827 INFO:teuthology.orchestra.run.smithi183.stdout: "hit_set_count": 0, 2024-09-10T19:31:03.827 INFO:teuthology.orchestra.run.smithi183.stdout: "use_gmt_hitset": true, 2024-09-10T19:31:03.827 INFO:teuthology.orchestra.run.smithi183.stdout: "min_read_recency_for_promote": 0, 2024-09-10T19:31:03.827 INFO:teuthology.orchestra.run.smithi183.stdout: "min_write_recency_for_promote": 0, 2024-09-10T19:31:03.827 INFO:teuthology.orchestra.run.smithi183.stdout: "hit_set_grade_decay_rate": 0, 2024-09-10T19:31:03.827 INFO:teuthology.orchestra.run.smithi183.stdout: "hit_set_search_last_n": 0, 2024-09-10T19:31:03.827 INFO:teuthology.orchestra.run.smithi183.stdout: "grade_table": [], 2024-09-10T19:31:03.827 INFO:teuthology.orchestra.run.smithi183.stdout: "stripe_width": 0, 2024-09-10T19:31:03.827 INFO:teuthology.orchestra.run.smithi183.stdout: "expected_num_objects": 0, 2024-09-10T19:31:03.828 INFO:teuthology.orchestra.run.smithi183.stdout: "fast_read": false, 2024-09-10T19:31:03.828 INFO:teuthology.orchestra.run.smithi183.stdout: "options": {}, 2024-09-10T19:31:03.828 INFO:teuthology.orchestra.run.smithi183.stdout: "application_metadata": { 2024-09-10T19:31:03.828 INFO:teuthology.orchestra.run.smithi183.stdout: "rgw": {} 2024-09-10T19:31:03.828 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:31:03.828 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:31:03.828 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:31:03.828 INFO:teuthology.orchestra.run.smithi183.stdout: "pool": 3, 2024-09-10T19:31:03.828 INFO:teuthology.orchestra.run.smithi183.stdout: "pool_name": "datapool", 2024-09-10T19:31:03.828 INFO:teuthology.orchestra.run.smithi183.stdout: "create_time": "2024-09-10T19:10:06.224267+0000", 2024-09-10T19:31:03.828 INFO:teuthology.orchestra.run.smithi183.stdout: "flags": 8193, 2024-09-10T19:31:03.829 INFO:teuthology.orchestra.run.smithi183.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-09-10T19:31:03.829 INFO:teuthology.orchestra.run.smithi183.stdout: "type": 1, 2024-09-10T19:31:03.829 INFO:teuthology.orchestra.run.smithi183.stdout: "size": 3, 2024-09-10T19:31:03.829 INFO:teuthology.orchestra.run.smithi183.stdout: "min_size": 2, 2024-09-10T19:31:03.829 INFO:teuthology.orchestra.run.smithi183.stdout: "crush_rule": 0, 2024-09-10T19:31:03.829 INFO:teuthology.orchestra.run.smithi183.stdout: "peering_crush_bucket_count": 0, 2024-09-10T19:31:03.829 INFO:teuthology.orchestra.run.smithi183.stdout: "peering_crush_bucket_target": 0, 2024-09-10T19:31:03.829 INFO:teuthology.orchestra.run.smithi183.stdout: "peering_crush_bucket_barrier": 0, 2024-09-10T19:31:03.829 INFO:teuthology.orchestra.run.smithi183.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-10T19:31:03.829 INFO:teuthology.orchestra.run.smithi183.stdout: "object_hash": 2, 2024-09-10T19:31:03.829 INFO:teuthology.orchestra.run.smithi183.stdout: "pg_autoscale_mode": "off", 2024-09-10T19:31:03.830 INFO:teuthology.orchestra.run.smithi183.stdout: "pg_num": 3, 2024-09-10T19:31:03.830 INFO:teuthology.orchestra.run.smithi183.stdout: "pg_placement_num": 3, 2024-09-10T19:31:03.830 INFO:teuthology.orchestra.run.smithi183.stdout: "pg_placement_num_target": 3, 2024-09-10T19:31:03.830 INFO:teuthology.orchestra.run.smithi183.stdout: "pg_num_target": 3, 2024-09-10T19:31:03.830 INFO:teuthology.orchestra.run.smithi183.stdout: "pg_num_pending": 3, 2024-09-10T19:31:03.830 INFO:teuthology.orchestra.run.smithi183.stdout: "last_pg_merge_meta": { 2024-09-10T19:31:03.830 INFO:teuthology.orchestra.run.smithi183.stdout: "source_pgid": "0.0", 2024-09-10T19:31:03.830 INFO:teuthology.orchestra.run.smithi183.stdout: "ready_epoch": 0, 2024-09-10T19:31:03.831 INFO:teuthology.orchestra.run.smithi183.stdout: "last_epoch_started": 0, 2024-09-10T19:31:03.831 INFO:teuthology.orchestra.run.smithi183.stdout: "last_epoch_clean": 0, 2024-09-10T19:31:03.831 INFO:teuthology.orchestra.run.smithi183.stdout: "source_version": "0'0", 2024-09-10T19:31:03.831 INFO:teuthology.orchestra.run.smithi183.stdout: "target_version": "0'0" 2024-09-10T19:31:03.831 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:31:03.831 INFO:teuthology.orchestra.run.smithi183.stdout: "last_change": "63", 2024-09-10T19:31:03.831 INFO:teuthology.orchestra.run.smithi183.stdout: "last_force_op_resend": "0", 2024-09-10T19:31:03.831 INFO:teuthology.orchestra.run.smithi183.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-10T19:31:03.831 INFO:teuthology.orchestra.run.smithi183.stdout: "last_force_op_resend_preluminous": "0", 2024-09-10T19:31:03.831 INFO:teuthology.orchestra.run.smithi183.stdout: "auid": 0, 2024-09-10T19:31:03.831 INFO:teuthology.orchestra.run.smithi183.stdout: "snap_mode": "selfmanaged", 2024-09-10T19:31:03.831 INFO:teuthology.orchestra.run.smithi183.stdout: "snap_seq": 3, 2024-09-10T19:31:03.832 INFO:teuthology.orchestra.run.smithi183.stdout: "snap_epoch": 63, 2024-09-10T19:31:03.832 INFO:teuthology.orchestra.run.smithi183.stdout: "pool_snaps": [], 2024-09-10T19:31:03.832 INFO:teuthology.orchestra.run.smithi183.stdout: "removed_snaps": "[]", 2024-09-10T19:31:03.832 INFO:teuthology.orchestra.run.smithi183.stdout: "quota_max_bytes": 0, 2024-09-10T19:31:03.832 INFO:teuthology.orchestra.run.smithi183.stdout: "quota_max_objects": 0, 2024-09-10T19:31:03.832 INFO:teuthology.orchestra.run.smithi183.stdout: "tiers": [], 2024-09-10T19:31:03.832 INFO:teuthology.orchestra.run.smithi183.stdout: "tier_of": -1, 2024-09-10T19:31:03.832 INFO:teuthology.orchestra.run.smithi183.stdout: "read_tier": -1, 2024-09-10T19:31:03.832 INFO:teuthology.orchestra.run.smithi183.stdout: "write_tier": -1, 2024-09-10T19:31:03.832 INFO:teuthology.orchestra.run.smithi183.stdout: "cache_mode": "none", 2024-09-10T19:31:03.832 INFO:teuthology.orchestra.run.smithi183.stdout: "target_max_bytes": 0, 2024-09-10T19:31:03.832 INFO:teuthology.orchestra.run.smithi183.stdout: "target_max_objects": 0, 2024-09-10T19:31:03.833 INFO:teuthology.orchestra.run.smithi183.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-10T19:31:03.833 INFO:teuthology.orchestra.run.smithi183.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-10T19:31:03.833 INFO:teuthology.orchestra.run.smithi183.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-10T19:31:03.833 INFO:teuthology.orchestra.run.smithi183.stdout: "cache_min_flush_age": 0, 2024-09-10T19:31:03.833 INFO:teuthology.orchestra.run.smithi183.stdout: "cache_min_evict_age": 0, 2024-09-10T19:31:03.833 INFO:teuthology.orchestra.run.smithi183.stdout: "erasure_code_profile": "", 2024-09-10T19:31:03.833 INFO:teuthology.orchestra.run.smithi183.stdout: "hit_set_params": { 2024-09-10T19:31:03.833 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "none" 2024-09-10T19:31:03.833 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:31:03.833 INFO:teuthology.orchestra.run.smithi183.stdout: "hit_set_period": 0, 2024-09-10T19:31:03.833 INFO:teuthology.orchestra.run.smithi183.stdout: "hit_set_count": 0, 2024-09-10T19:31:03.833 INFO:teuthology.orchestra.run.smithi183.stdout: "use_gmt_hitset": true, 2024-09-10T19:31:03.833 INFO:teuthology.orchestra.run.smithi183.stdout: "min_read_recency_for_promote": 0, 2024-09-10T19:31:03.834 INFO:teuthology.orchestra.run.smithi183.stdout: "min_write_recency_for_promote": 0, 2024-09-10T19:31:03.834 INFO:teuthology.orchestra.run.smithi183.stdout: "hit_set_grade_decay_rate": 0, 2024-09-10T19:31:03.834 INFO:teuthology.orchestra.run.smithi183.stdout: "hit_set_search_last_n": 0, 2024-09-10T19:31:03.834 INFO:teuthology.orchestra.run.smithi183.stdout: "grade_table": [], 2024-09-10T19:31:03.834 INFO:teuthology.orchestra.run.smithi183.stdout: "stripe_width": 0, 2024-09-10T19:31:03.834 INFO:teuthology.orchestra.run.smithi183.stdout: "expected_num_objects": 0, 2024-09-10T19:31:03.835 INFO:teuthology.orchestra.run.smithi183.stdout: "fast_read": false, 2024-09-10T19:31:03.835 INFO:teuthology.orchestra.run.smithi183.stdout: "options": {}, 2024-09-10T19:31:03.835 INFO:teuthology.orchestra.run.smithi183.stdout: "application_metadata": { 2024-09-10T19:31:03.835 INFO:teuthology.orchestra.run.smithi183.stdout: "rbd": {} 2024-09-10T19:31:03.835 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:31:03.835 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:31:03.835 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:31:03.835 INFO:teuthology.orchestra.run.smithi183.stdout: "pool": 4, 2024-09-10T19:31:03.835 INFO:teuthology.orchestra.run.smithi183.stdout: "pool_name": "default.rgw.log", 2024-09-10T19:31:03.835 INFO:teuthology.orchestra.run.smithi183.stdout: "create_time": "2024-09-10T19:10:07.958284+0000", 2024-09-10T19:31:03.835 INFO:teuthology.orchestra.run.smithi183.stdout: "flags": 1, 2024-09-10T19:31:03.836 INFO:teuthology.orchestra.run.smithi183.stdout: "flags_names": "hashpspool", 2024-09-10T19:31:03.836 INFO:teuthology.orchestra.run.smithi183.stdout: "type": 1, 2024-09-10T19:31:03.836 INFO:teuthology.orchestra.run.smithi183.stdout: "size": 3, 2024-09-10T19:31:03.836 INFO:teuthology.orchestra.run.smithi183.stdout: "min_size": 2, 2024-09-10T19:31:03.836 INFO:teuthology.orchestra.run.smithi183.stdout: "crush_rule": 0, 2024-09-10T19:31:03.836 INFO:teuthology.orchestra.run.smithi183.stdout: "peering_crush_bucket_count": 0, 2024-09-10T19:31:03.836 INFO:teuthology.orchestra.run.smithi183.stdout: "peering_crush_bucket_target": 0, 2024-09-10T19:31:03.836 INFO:teuthology.orchestra.run.smithi183.stdout: "peering_crush_bucket_barrier": 0, 2024-09-10T19:31:03.836 INFO:teuthology.orchestra.run.smithi183.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-10T19:31:03.837 INFO:teuthology.orchestra.run.smithi183.stdout: "object_hash": 2, 2024-09-10T19:31:03.837 INFO:teuthology.orchestra.run.smithi183.stdout: "pg_autoscale_mode": "off", 2024-09-10T19:31:03.837 INFO:teuthology.orchestra.run.smithi183.stdout: "pg_num": 32, 2024-09-10T19:31:03.837 INFO:teuthology.orchestra.run.smithi183.stdout: "pg_placement_num": 32, 2024-09-10T19:31:03.837 INFO:teuthology.orchestra.run.smithi183.stdout: "pg_placement_num_target": 32, 2024-09-10T19:31:03.838 INFO:teuthology.orchestra.run.smithi183.stdout: "pg_num_target": 32, 2024-09-10T19:31:03.838 INFO:teuthology.orchestra.run.smithi183.stdout: "pg_num_pending": 32, 2024-09-10T19:31:03.838 INFO:teuthology.orchestra.run.smithi183.stdout: "last_pg_merge_meta": { 2024-09-10T19:31:03.838 INFO:teuthology.orchestra.run.smithi183.stdout: "source_pgid": "0.0", 2024-09-10T19:31:03.838 INFO:teuthology.orchestra.run.smithi183.stdout: "ready_epoch": 0, 2024-09-10T19:31:03.838 INFO:teuthology.orchestra.run.smithi183.stdout: "last_epoch_started": 0, 2024-09-10T19:31:03.838 INFO:teuthology.orchestra.run.smithi183.stdout: "last_epoch_clean": 0, 2024-09-10T19:31:03.838 INFO:teuthology.orchestra.run.smithi183.stdout: "source_version": "0'0", 2024-09-10T19:31:03.839 INFO:teuthology.orchestra.run.smithi183.stdout: "target_version": "0'0" 2024-09-10T19:31:03.839 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:31:03.839 INFO:teuthology.orchestra.run.smithi183.stdout: "last_change": "62", 2024-09-10T19:31:03.839 INFO:teuthology.orchestra.run.smithi183.stdout: "last_force_op_resend": "0", 2024-09-10T19:31:03.839 INFO:teuthology.orchestra.run.smithi183.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-10T19:31:03.839 INFO:teuthology.orchestra.run.smithi183.stdout: "last_force_op_resend_preluminous": "0", 2024-09-10T19:31:03.839 INFO:teuthology.orchestra.run.smithi183.stdout: "auid": 0, 2024-09-10T19:31:03.839 INFO:teuthology.orchestra.run.smithi183.stdout: "snap_mode": "selfmanaged", 2024-09-10T19:31:03.839 INFO:teuthology.orchestra.run.smithi183.stdout: "snap_seq": 0, 2024-09-10T19:31:03.839 INFO:teuthology.orchestra.run.smithi183.stdout: "snap_epoch": 0, 2024-09-10T19:31:03.839 INFO:teuthology.orchestra.run.smithi183.stdout: "pool_snaps": [], 2024-09-10T19:31:03.839 INFO:teuthology.orchestra.run.smithi183.stdout: "removed_snaps": "[]", 2024-09-10T19:31:03.839 INFO:teuthology.orchestra.run.smithi183.stdout: "quota_max_bytes": 0, 2024-09-10T19:31:03.839 INFO:teuthology.orchestra.run.smithi183.stdout: "quota_max_objects": 0, 2024-09-10T19:31:03.840 INFO:teuthology.orchestra.run.smithi183.stdout: "tiers": [], 2024-09-10T19:31:03.840 INFO:teuthology.orchestra.run.smithi183.stdout: "tier_of": -1, 2024-09-10T19:31:03.840 INFO:teuthology.orchestra.run.smithi183.stdout: "read_tier": -1, 2024-09-10T19:31:03.840 INFO:teuthology.orchestra.run.smithi183.stdout: "write_tier": -1, 2024-09-10T19:31:03.840 INFO:teuthology.orchestra.run.smithi183.stdout: "cache_mode": "none", 2024-09-10T19:31:03.840 INFO:teuthology.orchestra.run.smithi183.stdout: "target_max_bytes": 0, 2024-09-10T19:31:03.840 INFO:teuthology.orchestra.run.smithi183.stdout: "target_max_objects": 0, 2024-09-10T19:31:03.840 INFO:teuthology.orchestra.run.smithi183.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-10T19:31:03.840 INFO:teuthology.orchestra.run.smithi183.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-10T19:31:03.840 INFO:teuthology.orchestra.run.smithi183.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-10T19:31:03.840 INFO:teuthology.orchestra.run.smithi183.stdout: "cache_min_flush_age": 0, 2024-09-10T19:31:03.840 INFO:teuthology.orchestra.run.smithi183.stdout: "cache_min_evict_age": 0, 2024-09-10T19:31:03.841 INFO:teuthology.orchestra.run.smithi183.stdout: "erasure_code_profile": "", 2024-09-10T19:31:03.841 INFO:teuthology.orchestra.run.smithi183.stdout: "hit_set_params": { 2024-09-10T19:31:03.841 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "none" 2024-09-10T19:31:03.841 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:31:03.841 INFO:teuthology.orchestra.run.smithi183.stdout: "hit_set_period": 0, 2024-09-10T19:31:03.841 INFO:teuthology.orchestra.run.smithi183.stdout: "hit_set_count": 0, 2024-09-10T19:31:03.841 INFO:teuthology.orchestra.run.smithi183.stdout: "use_gmt_hitset": true, 2024-09-10T19:31:03.841 INFO:teuthology.orchestra.run.smithi183.stdout: "min_read_recency_for_promote": 0, 2024-09-10T19:31:03.841 INFO:teuthology.orchestra.run.smithi183.stdout: "min_write_recency_for_promote": 0, 2024-09-10T19:31:03.842 INFO:teuthology.orchestra.run.smithi183.stdout: "hit_set_grade_decay_rate": 0, 2024-09-10T19:31:03.842 INFO:teuthology.orchestra.run.smithi183.stdout: "hit_set_search_last_n": 0, 2024-09-10T19:31:03.845 INFO:teuthology.orchestra.run.smithi183.stdout: "grade_table": [], 2024-09-10T19:31:03.845 INFO:teuthology.orchestra.run.smithi183.stdout: "stripe_width": 0, 2024-09-10T19:31:03.845 INFO:teuthology.orchestra.run.smithi183.stdout: "expected_num_objects": 0, 2024-09-10T19:31:03.845 INFO:teuthology.orchestra.run.smithi183.stdout: "fast_read": false, 2024-09-10T19:31:03.845 INFO:teuthology.orchestra.run.smithi183.stdout: "options": {}, 2024-09-10T19:31:03.845 INFO:teuthology.orchestra.run.smithi183.stdout: "application_metadata": { 2024-09-10T19:31:03.845 INFO:teuthology.orchestra.run.smithi183.stdout: "rgw": {} 2024-09-10T19:31:03.846 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:31:03.846 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:31:03.846 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:31:03.846 INFO:teuthology.orchestra.run.smithi183.stdout: "pool": 5, 2024-09-10T19:31:03.846 INFO:teuthology.orchestra.run.smithi183.stdout: "pool_name": "default.rgw.control", 2024-09-10T19:31:03.846 INFO:teuthology.orchestra.run.smithi183.stdout: "create_time": "2024-09-10T19:10:09.933596+0000", 2024-09-10T19:31:03.846 INFO:teuthology.orchestra.run.smithi183.stdout: "flags": 1, 2024-09-10T19:31:03.846 INFO:teuthology.orchestra.run.smithi183.stdout: "flags_names": "hashpspool", 2024-09-10T19:31:03.846 INFO:teuthology.orchestra.run.smithi183.stdout: "type": 1, 2024-09-10T19:31:03.846 INFO:teuthology.orchestra.run.smithi183.stdout: "size": 3, 2024-09-10T19:31:03.846 INFO:teuthology.orchestra.run.smithi183.stdout: "min_size": 2, 2024-09-10T19:31:03.846 INFO:teuthology.orchestra.run.smithi183.stdout: "crush_rule": 0, 2024-09-10T19:31:03.847 INFO:teuthology.orchestra.run.smithi183.stdout: "peering_crush_bucket_count": 0, 2024-09-10T19:31:03.847 INFO:teuthology.orchestra.run.smithi183.stdout: "peering_crush_bucket_target": 0, 2024-09-10T19:31:03.847 INFO:teuthology.orchestra.run.smithi183.stdout: "peering_crush_bucket_barrier": 0, 2024-09-10T19:31:03.847 INFO:teuthology.orchestra.run.smithi183.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-10T19:31:03.847 INFO:teuthology.orchestra.run.smithi183.stdout: "object_hash": 2, 2024-09-10T19:31:03.847 INFO:teuthology.orchestra.run.smithi183.stdout: "pg_autoscale_mode": "off", 2024-09-10T19:31:03.847 INFO:teuthology.orchestra.run.smithi183.stdout: "pg_num": 32, 2024-09-10T19:31:03.847 INFO:teuthology.orchestra.run.smithi183.stdout: "pg_placement_num": 32, 2024-09-10T19:31:03.847 INFO:teuthology.orchestra.run.smithi183.stdout: "pg_placement_num_target": 32, 2024-09-10T19:31:03.847 INFO:teuthology.orchestra.run.smithi183.stdout: "pg_num_target": 32, 2024-09-10T19:31:03.847 INFO:teuthology.orchestra.run.smithi183.stdout: "pg_num_pending": 32, 2024-09-10T19:31:03.847 INFO:teuthology.orchestra.run.smithi183.stdout: "last_pg_merge_meta": { 2024-09-10T19:31:03.847 INFO:teuthology.orchestra.run.smithi183.stdout: "source_pgid": "0.0", 2024-09-10T19:31:03.848 INFO:teuthology.orchestra.run.smithi183.stdout: "ready_epoch": 0, 2024-09-10T19:31:03.848 INFO:teuthology.orchestra.run.smithi183.stdout: "last_epoch_started": 0, 2024-09-10T19:31:03.848 INFO:teuthology.orchestra.run.smithi183.stdout: "last_epoch_clean": 0, 2024-09-10T19:31:03.848 INFO:teuthology.orchestra.run.smithi183.stdout: "source_version": "0'0", 2024-09-10T19:31:03.848 INFO:teuthology.orchestra.run.smithi183.stdout: "target_version": "0'0" 2024-09-10T19:31:03.848 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:31:03.848 INFO:teuthology.orchestra.run.smithi183.stdout: "last_change": "64", 2024-09-10T19:31:03.848 INFO:teuthology.orchestra.run.smithi183.stdout: "last_force_op_resend": "0", 2024-09-10T19:31:03.848 INFO:teuthology.orchestra.run.smithi183.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-10T19:31:03.848 INFO:teuthology.orchestra.run.smithi183.stdout: "last_force_op_resend_preluminous": "0", 2024-09-10T19:31:03.848 INFO:teuthology.orchestra.run.smithi183.stdout: "auid": 0, 2024-09-10T19:31:03.848 INFO:teuthology.orchestra.run.smithi183.stdout: "snap_mode": "selfmanaged", 2024-09-10T19:31:03.848 INFO:teuthology.orchestra.run.smithi183.stdout: "snap_seq": 0, 2024-09-10T19:31:03.849 INFO:teuthology.orchestra.run.smithi183.stdout: "snap_epoch": 0, 2024-09-10T19:31:03.849 INFO:teuthology.orchestra.run.smithi183.stdout: "pool_snaps": [], 2024-09-10T19:31:03.849 INFO:teuthology.orchestra.run.smithi183.stdout: "removed_snaps": "[]", 2024-09-10T19:31:03.849 INFO:teuthology.orchestra.run.smithi183.stdout: "quota_max_bytes": 0, 2024-09-10T19:31:03.849 INFO:teuthology.orchestra.run.smithi183.stdout: "quota_max_objects": 0, 2024-09-10T19:31:03.849 INFO:teuthology.orchestra.run.smithi183.stdout: "tiers": [], 2024-09-10T19:31:03.849 INFO:teuthology.orchestra.run.smithi183.stdout: "tier_of": -1, 2024-09-10T19:31:03.849 INFO:teuthology.orchestra.run.smithi183.stdout: "read_tier": -1, 2024-09-10T19:31:03.849 INFO:teuthology.orchestra.run.smithi183.stdout: "write_tier": -1, 2024-09-10T19:31:03.849 INFO:teuthology.orchestra.run.smithi183.stdout: "cache_mode": "none", 2024-09-10T19:31:03.849 INFO:teuthology.orchestra.run.smithi183.stdout: "target_max_bytes": 0, 2024-09-10T19:31:03.849 INFO:teuthology.orchestra.run.smithi183.stdout: "target_max_objects": 0, 2024-09-10T19:31:03.849 INFO:teuthology.orchestra.run.smithi183.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-10T19:31:03.850 INFO:teuthology.orchestra.run.smithi183.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-10T19:31:03.850 INFO:teuthology.orchestra.run.smithi183.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-10T19:31:03.850 INFO:teuthology.orchestra.run.smithi183.stdout: "cache_min_flush_age": 0, 2024-09-10T19:31:03.850 INFO:teuthology.orchestra.run.smithi183.stdout: "cache_min_evict_age": 0, 2024-09-10T19:31:03.850 INFO:teuthology.orchestra.run.smithi183.stdout: "erasure_code_profile": "", 2024-09-10T19:31:03.850 INFO:teuthology.orchestra.run.smithi183.stdout: "hit_set_params": { 2024-09-10T19:31:03.850 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "none" 2024-09-10T19:31:03.850 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:31:03.850 INFO:teuthology.orchestra.run.smithi183.stdout: "hit_set_period": 0, 2024-09-10T19:31:03.850 INFO:teuthology.orchestra.run.smithi183.stdout: "hit_set_count": 0, 2024-09-10T19:31:03.850 INFO:teuthology.orchestra.run.smithi183.stdout: "use_gmt_hitset": true, 2024-09-10T19:31:03.850 INFO:teuthology.orchestra.run.smithi183.stdout: "min_read_recency_for_promote": 0, 2024-09-10T19:31:03.850 INFO:teuthology.orchestra.run.smithi183.stdout: "min_write_recency_for_promote": 0, 2024-09-10T19:31:03.851 INFO:teuthology.orchestra.run.smithi183.stdout: "hit_set_grade_decay_rate": 0, 2024-09-10T19:31:03.851 INFO:teuthology.orchestra.run.smithi183.stdout: "hit_set_search_last_n": 0, 2024-09-10T19:31:03.851 INFO:teuthology.orchestra.run.smithi183.stdout: "grade_table": [], 2024-09-10T19:31:03.851 INFO:teuthology.orchestra.run.smithi183.stdout: "stripe_width": 0, 2024-09-10T19:31:03.851 INFO:teuthology.orchestra.run.smithi183.stdout: "expected_num_objects": 0, 2024-09-10T19:31:03.851 INFO:teuthology.orchestra.run.smithi183.stdout: "fast_read": false, 2024-09-10T19:31:03.851 INFO:teuthology.orchestra.run.smithi183.stdout: "options": {}, 2024-09-10T19:31:03.851 INFO:teuthology.orchestra.run.smithi183.stdout: "application_metadata": { 2024-09-10T19:31:03.851 INFO:teuthology.orchestra.run.smithi183.stdout: "rgw": {} 2024-09-10T19:31:03.851 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:31:03.851 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:31:03.851 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:31:03.851 INFO:teuthology.orchestra.run.smithi183.stdout: "pool": 6, 2024-09-10T19:31:03.852 INFO:teuthology.orchestra.run.smithi183.stdout: "pool_name": "default.rgw.meta", 2024-09-10T19:31:03.852 INFO:teuthology.orchestra.run.smithi183.stdout: "create_time": "2024-09-10T19:10:11.993712+0000", 2024-09-10T19:31:03.852 INFO:teuthology.orchestra.run.smithi183.stdout: "flags": 1, 2024-09-10T19:31:03.852 INFO:teuthology.orchestra.run.smithi183.stdout: "flags_names": "hashpspool", 2024-09-10T19:31:03.852 INFO:teuthology.orchestra.run.smithi183.stdout: "type": 1, 2024-09-10T19:31:03.852 INFO:teuthology.orchestra.run.smithi183.stdout: "size": 3, 2024-09-10T19:31:03.852 INFO:teuthology.orchestra.run.smithi183.stdout: "min_size": 2, 2024-09-10T19:31:03.852 INFO:teuthology.orchestra.run.smithi183.stdout: "crush_rule": 0, 2024-09-10T19:31:03.852 INFO:teuthology.orchestra.run.smithi183.stdout: "peering_crush_bucket_count": 0, 2024-09-10T19:31:03.852 INFO:teuthology.orchestra.run.smithi183.stdout: "peering_crush_bucket_target": 0, 2024-09-10T19:31:03.852 INFO:teuthology.orchestra.run.smithi183.stdout: "peering_crush_bucket_barrier": 0, 2024-09-10T19:31:03.852 INFO:teuthology.orchestra.run.smithi183.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-10T19:31:03.853 INFO:teuthology.orchestra.run.smithi183.stdout: "object_hash": 2, 2024-09-10T19:31:03.853 INFO:teuthology.orchestra.run.smithi183.stdout: "pg_autoscale_mode": "off", 2024-09-10T19:31:03.853 INFO:teuthology.orchestra.run.smithi183.stdout: "pg_num": 32, 2024-09-10T19:31:03.853 INFO:teuthology.orchestra.run.smithi183.stdout: "pg_placement_num": 32, 2024-09-10T19:31:03.853 INFO:teuthology.orchestra.run.smithi183.stdout: "pg_placement_num_target": 32, 2024-09-10T19:31:03.853 INFO:teuthology.orchestra.run.smithi183.stdout: "pg_num_target": 32, 2024-09-10T19:31:03.853 INFO:teuthology.orchestra.run.smithi183.stdout: "pg_num_pending": 32, 2024-09-10T19:31:03.853 INFO:teuthology.orchestra.run.smithi183.stdout: "last_pg_merge_meta": { 2024-09-10T19:31:03.853 INFO:teuthology.orchestra.run.smithi183.stdout: "source_pgid": "0.0", 2024-09-10T19:31:03.853 INFO:teuthology.orchestra.run.smithi183.stdout: "ready_epoch": 0, 2024-09-10T19:31:03.853 INFO:teuthology.orchestra.run.smithi183.stdout: "last_epoch_started": 0, 2024-09-10T19:31:03.853 INFO:teuthology.orchestra.run.smithi183.stdout: "last_epoch_clean": 0, 2024-09-10T19:31:03.853 INFO:teuthology.orchestra.run.smithi183.stdout: "source_version": "0'0", 2024-09-10T19:31:03.854 INFO:teuthology.orchestra.run.smithi183.stdout: "target_version": "0'0" 2024-09-10T19:31:03.854 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:31:03.854 INFO:teuthology.orchestra.run.smithi183.stdout: "last_change": "66", 2024-09-10T19:31:03.854 INFO:teuthology.orchestra.run.smithi183.stdout: "last_force_op_resend": "0", 2024-09-10T19:31:03.854 INFO:teuthology.orchestra.run.smithi183.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-10T19:31:03.854 INFO:teuthology.orchestra.run.smithi183.stdout: "last_force_op_resend_preluminous": "0", 2024-09-10T19:31:03.854 INFO:teuthology.orchestra.run.smithi183.stdout: "auid": 0, 2024-09-10T19:31:03.854 INFO:teuthology.orchestra.run.smithi183.stdout: "snap_mode": "selfmanaged", 2024-09-10T19:31:03.854 INFO:teuthology.orchestra.run.smithi183.stdout: "snap_seq": 0, 2024-09-10T19:31:03.854 INFO:teuthology.orchestra.run.smithi183.stdout: "snap_epoch": 0, 2024-09-10T19:31:03.854 INFO:teuthology.orchestra.run.smithi183.stdout: "pool_snaps": [], 2024-09-10T19:31:03.855 INFO:teuthology.orchestra.run.smithi183.stdout: "removed_snaps": "[]", 2024-09-10T19:31:03.855 INFO:teuthology.orchestra.run.smithi183.stdout: "quota_max_bytes": 0, 2024-09-10T19:31:03.855 INFO:teuthology.orchestra.run.smithi183.stdout: "quota_max_objects": 0, 2024-09-10T19:31:03.855 INFO:teuthology.orchestra.run.smithi183.stdout: "tiers": [], 2024-09-10T19:31:03.855 INFO:teuthology.orchestra.run.smithi183.stdout: "tier_of": -1, 2024-09-10T19:31:03.855 INFO:teuthology.orchestra.run.smithi183.stdout: "read_tier": -1, 2024-09-10T19:31:03.855 INFO:teuthology.orchestra.run.smithi183.stdout: "write_tier": -1, 2024-09-10T19:31:03.855 INFO:teuthology.orchestra.run.smithi183.stdout: "cache_mode": "none", 2024-09-10T19:31:03.855 INFO:teuthology.orchestra.run.smithi183.stdout: "target_max_bytes": 0, 2024-09-10T19:31:03.855 INFO:teuthology.orchestra.run.smithi183.stdout: "target_max_objects": 0, 2024-09-10T19:31:03.855 INFO:teuthology.orchestra.run.smithi183.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-10T19:31:03.855 INFO:teuthology.orchestra.run.smithi183.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-10T19:31:03.855 INFO:teuthology.orchestra.run.smithi183.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-10T19:31:03.856 INFO:teuthology.orchestra.run.smithi183.stdout: "cache_min_flush_age": 0, 2024-09-10T19:31:03.856 INFO:teuthology.orchestra.run.smithi183.stdout: "cache_min_evict_age": 0, 2024-09-10T19:31:03.856 INFO:teuthology.orchestra.run.smithi183.stdout: "erasure_code_profile": "", 2024-09-10T19:31:03.856 INFO:teuthology.orchestra.run.smithi183.stdout: "hit_set_params": { 2024-09-10T19:31:03.856 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "none" 2024-09-10T19:31:03.856 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:31:03.856 INFO:teuthology.orchestra.run.smithi183.stdout: "hit_set_period": 0, 2024-09-10T19:31:03.856 INFO:teuthology.orchestra.run.smithi183.stdout: "hit_set_count": 0, 2024-09-10T19:31:03.856 INFO:teuthology.orchestra.run.smithi183.stdout: "use_gmt_hitset": true, 2024-09-10T19:31:03.856 INFO:teuthology.orchestra.run.smithi183.stdout: "min_read_recency_for_promote": 0, 2024-09-10T19:31:03.856 INFO:teuthology.orchestra.run.smithi183.stdout: "min_write_recency_for_promote": 0, 2024-09-10T19:31:03.856 INFO:teuthology.orchestra.run.smithi183.stdout: "hit_set_grade_decay_rate": 0, 2024-09-10T19:31:03.857 INFO:teuthology.orchestra.run.smithi183.stdout: "hit_set_search_last_n": 0, 2024-09-10T19:31:03.857 INFO:teuthology.orchestra.run.smithi183.stdout: "grade_table": [], 2024-09-10T19:31:03.857 INFO:teuthology.orchestra.run.smithi183.stdout: "stripe_width": 0, 2024-09-10T19:31:03.857 INFO:teuthology.orchestra.run.smithi183.stdout: "expected_num_objects": 0, 2024-09-10T19:31:03.857 INFO:teuthology.orchestra.run.smithi183.stdout: "fast_read": false, 2024-09-10T19:31:03.857 INFO:teuthology.orchestra.run.smithi183.stdout: "options": { 2024-09-10T19:31:03.858 INFO:teuthology.orchestra.run.smithi183.stdout: "pg_autoscale_bias": 4 2024-09-10T19:31:03.858 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:31:03.858 INFO:teuthology.orchestra.run.smithi183.stdout: "application_metadata": { 2024-09-10T19:31:03.858 INFO:teuthology.orchestra.run.smithi183.stdout: "rgw": {} 2024-09-10T19:31:03.858 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:31:03.858 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:31:03.858 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:31:03.858 INFO:teuthology.orchestra.run.smithi183.stdout: "pool": 11, 2024-09-10T19:31:03.858 INFO:teuthology.orchestra.run.smithi183.stdout: "pool_name": "unique_pool_4", 2024-09-10T19:31:03.859 INFO:teuthology.orchestra.run.smithi183.stdout: "create_time": "2024-09-10T19:27:45.303826+0000", 2024-09-10T19:31:03.859 INFO:teuthology.orchestra.run.smithi183.stdout: "flags": 1, 2024-09-10T19:31:03.859 INFO:teuthology.orchestra.run.smithi183.stdout: "flags_names": "hashpspool", 2024-09-10T19:31:03.859 INFO:teuthology.orchestra.run.smithi183.stdout: "type": 1, 2024-09-10T19:31:03.859 INFO:teuthology.orchestra.run.smithi183.stdout: "size": 3, 2024-09-10T19:31:03.859 INFO:teuthology.orchestra.run.smithi183.stdout: "min_size": 2, 2024-09-10T19:31:03.859 INFO:teuthology.orchestra.run.smithi183.stdout: "crush_rule": 0, 2024-09-10T19:31:03.859 INFO:teuthology.orchestra.run.smithi183.stdout: "peering_crush_bucket_count": 0, 2024-09-10T19:31:03.859 INFO:teuthology.orchestra.run.smithi183.stdout: "peering_crush_bucket_target": 0, 2024-09-10T19:31:03.859 INFO:teuthology.orchestra.run.smithi183.stdout: "peering_crush_bucket_barrier": 0, 2024-09-10T19:31:03.859 INFO:teuthology.orchestra.run.smithi183.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-10T19:31:03.859 INFO:teuthology.orchestra.run.smithi183.stdout: "object_hash": 2, 2024-09-10T19:31:03.859 INFO:teuthology.orchestra.run.smithi183.stdout: "pg_autoscale_mode": "off", 2024-09-10T19:31:03.860 INFO:teuthology.orchestra.run.smithi183.stdout: "pg_num": 16, 2024-09-10T19:31:03.860 INFO:teuthology.orchestra.run.smithi183.stdout: "pg_placement_num": 16, 2024-09-10T19:31:03.860 INFO:teuthology.orchestra.run.smithi183.stdout: "pg_placement_num_target": 16, 2024-09-10T19:31:03.860 INFO:teuthology.orchestra.run.smithi183.stdout: "pg_num_target": 16, 2024-09-10T19:31:03.860 INFO:teuthology.orchestra.run.smithi183.stdout: "pg_num_pending": 16, 2024-09-10T19:31:03.860 INFO:teuthology.orchestra.run.smithi183.stdout: "last_pg_merge_meta": { 2024-09-10T19:31:03.860 INFO:teuthology.orchestra.run.smithi183.stdout: "source_pgid": "0.0", 2024-09-10T19:31:03.860 INFO:teuthology.orchestra.run.smithi183.stdout: "ready_epoch": 0, 2024-09-10T19:31:03.860 INFO:teuthology.orchestra.run.smithi183.stdout: "last_epoch_started": 0, 2024-09-10T19:31:03.860 INFO:teuthology.orchestra.run.smithi183.stdout: "last_epoch_clean": 0, 2024-09-10T19:31:03.860 INFO:teuthology.orchestra.run.smithi183.stdout: "source_version": "0'0", 2024-09-10T19:31:03.860 INFO:teuthology.orchestra.run.smithi183.stdout: "target_version": "0'0" 2024-09-10T19:31:03.860 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:31:03.860 INFO:teuthology.orchestra.run.smithi183.stdout: "last_change": "374", 2024-09-10T19:31:03.861 INFO:teuthology.orchestra.run.smithi183.stdout: "last_force_op_resend": "0", 2024-09-10T19:31:03.861 INFO:teuthology.orchestra.run.smithi183.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-10T19:31:03.861 INFO:teuthology.orchestra.run.smithi183.stdout: "last_force_op_resend_preluminous": "0", 2024-09-10T19:31:03.861 INFO:teuthology.orchestra.run.smithi183.stdout: "auid": 0, 2024-09-10T19:31:03.861 INFO:teuthology.orchestra.run.smithi183.stdout: "snap_mode": "selfmanaged", 2024-09-10T19:31:03.861 INFO:teuthology.orchestra.run.smithi183.stdout: "snap_seq": 0, 2024-09-10T19:31:03.861 INFO:teuthology.orchestra.run.smithi183.stdout: "snap_epoch": 0, 2024-09-10T19:31:03.861 INFO:teuthology.orchestra.run.smithi183.stdout: "pool_snaps": [], 2024-09-10T19:31:03.861 INFO:teuthology.orchestra.run.smithi183.stdout: "removed_snaps": "[]", 2024-09-10T19:31:03.861 INFO:teuthology.orchestra.run.smithi183.stdout: "quota_max_bytes": 0, 2024-09-10T19:31:03.861 INFO:teuthology.orchestra.run.smithi183.stdout: "quota_max_objects": 0, 2024-09-10T19:31:03.861 INFO:teuthology.orchestra.run.smithi183.stdout: "tiers": [], 2024-09-10T19:31:03.861 INFO:teuthology.orchestra.run.smithi183.stdout: "tier_of": -1, 2024-09-10T19:31:03.862 INFO:teuthology.orchestra.run.smithi183.stdout: "read_tier": -1, 2024-09-10T19:31:03.862 INFO:teuthology.orchestra.run.smithi183.stdout: "write_tier": -1, 2024-09-10T19:31:03.862 INFO:teuthology.orchestra.run.smithi183.stdout: "cache_mode": "none", 2024-09-10T19:31:03.862 INFO:teuthology.orchestra.run.smithi183.stdout: "target_max_bytes": 0, 2024-09-10T19:31:03.862 INFO:teuthology.orchestra.run.smithi183.stdout: "target_max_objects": 0, 2024-09-10T19:31:03.862 INFO:teuthology.orchestra.run.smithi183.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-10T19:31:03.862 INFO:teuthology.orchestra.run.smithi183.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-10T19:31:03.862 INFO:teuthology.orchestra.run.smithi183.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-10T19:31:03.862 INFO:teuthology.orchestra.run.smithi183.stdout: "cache_min_flush_age": 0, 2024-09-10T19:31:03.862 INFO:teuthology.orchestra.run.smithi183.stdout: "cache_min_evict_age": 0, 2024-09-10T19:31:03.862 INFO:teuthology.orchestra.run.smithi183.stdout: "erasure_code_profile": "", 2024-09-10T19:31:03.862 INFO:teuthology.orchestra.run.smithi183.stdout: "hit_set_params": { 2024-09-10T19:31:03.862 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "none" 2024-09-10T19:31:03.863 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:31:03.863 INFO:teuthology.orchestra.run.smithi183.stdout: "hit_set_period": 0, 2024-09-10T19:31:03.863 INFO:teuthology.orchestra.run.smithi183.stdout: "hit_set_count": 0, 2024-09-10T19:31:03.863 INFO:teuthology.orchestra.run.smithi183.stdout: "use_gmt_hitset": true, 2024-09-10T19:31:03.863 INFO:teuthology.orchestra.run.smithi183.stdout: "min_read_recency_for_promote": 0, 2024-09-10T19:31:03.863 INFO:teuthology.orchestra.run.smithi183.stdout: "min_write_recency_for_promote": 0, 2024-09-10T19:31:03.863 INFO:teuthology.orchestra.run.smithi183.stdout: "hit_set_grade_decay_rate": 0, 2024-09-10T19:31:03.863 INFO:teuthology.orchestra.run.smithi183.stdout: "hit_set_search_last_n": 0, 2024-09-10T19:31:03.863 INFO:teuthology.orchestra.run.smithi183.stdout: "grade_table": [], 2024-09-10T19:31:03.863 INFO:teuthology.orchestra.run.smithi183.stdout: "stripe_width": 0, 2024-09-10T19:31:03.863 INFO:teuthology.orchestra.run.smithi183.stdout: "expected_num_objects": 0, 2024-09-10T19:31:03.863 INFO:teuthology.orchestra.run.smithi183.stdout: "fast_read": false, 2024-09-10T19:31:03.863 INFO:teuthology.orchestra.run.smithi183.stdout: "options": {}, 2024-09-10T19:31:03.864 INFO:teuthology.orchestra.run.smithi183.stdout: "application_metadata": { 2024-09-10T19:31:03.864 INFO:teuthology.orchestra.run.smithi183.stdout: "rados": {} 2024-09-10T19:31:03.864 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:31:03.864 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:31:03.864 INFO:teuthology.orchestra.run.smithi183.stdout: ], 2024-09-10T19:31:03.864 INFO:teuthology.orchestra.run.smithi183.stdout: "osds": [ 2024-09-10T19:31:03.864 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:31:03.864 INFO:teuthology.orchestra.run.smithi183.stdout: "osd": 0, 2024-09-10T19:31:03.864 INFO:teuthology.orchestra.run.smithi183.stdout: "uuid": "9d323010-394a-49aa-ba9b-244d87faf36e", 2024-09-10T19:31:03.864 INFO:teuthology.orchestra.run.smithi183.stdout: "up": 1, 2024-09-10T19:31:03.864 INFO:teuthology.orchestra.run.smithi183.stdout: "in": 1, 2024-09-10T19:31:03.864 INFO:teuthology.orchestra.run.smithi183.stdout: "weight": 1, 2024-09-10T19:31:03.864 INFO:teuthology.orchestra.run.smithi183.stdout: "primary_affinity": 1, 2024-09-10T19:31:03.864 INFO:teuthology.orchestra.run.smithi183.stdout: "last_clean_begin": 8, 2024-09-10T19:31:03.865 INFO:teuthology.orchestra.run.smithi183.stdout: "last_clean_end": 262, 2024-09-10T19:31:03.865 INFO:teuthology.orchestra.run.smithi183.stdout: "up_from": 301, 2024-09-10T19:31:03.865 INFO:teuthology.orchestra.run.smithi183.stdout: "up_thru": 374, 2024-09-10T19:31:03.865 INFO:teuthology.orchestra.run.smithi183.stdout: "down_at": 263, 2024-09-10T19:31:03.865 INFO:teuthology.orchestra.run.smithi183.stdout: "lost_at": 0, 2024-09-10T19:31:03.865 INFO:teuthology.orchestra.run.smithi183.stdout: "public_addrs": { 2024-09-10T19:31:03.865 INFO:teuthology.orchestra.run.smithi183.stdout: "addrvec": [ 2024-09-10T19:31:03.865 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:31:03.865 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v2", 2024-09-10T19:31:03.865 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.183:6810", 2024-09-10T19:31:03.865 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 3346539414 2024-09-10T19:31:03.865 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:31:03.865 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:31:03.866 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v1", 2024-09-10T19:31:03.866 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.183:6811", 2024-09-10T19:31:03.866 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 3346539414 2024-09-10T19:31:03.866 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:31:03.866 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:31:03.866 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:31:03.866 INFO:teuthology.orchestra.run.smithi183.stdout: "cluster_addrs": { 2024-09-10T19:31:03.866 INFO:teuthology.orchestra.run.smithi183.stdout: "addrvec": [ 2024-09-10T19:31:03.866 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:31:03.866 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v2", 2024-09-10T19:31:03.866 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.183:6812", 2024-09-10T19:31:03.866 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 3346539414 2024-09-10T19:31:03.866 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:31:03.866 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:31:03.867 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v1", 2024-09-10T19:31:03.867 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.183:6813", 2024-09-10T19:31:03.867 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 3346539414 2024-09-10T19:31:03.867 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:31:03.867 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:31:03.867 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:31:03.867 INFO:teuthology.orchestra.run.smithi183.stdout: "heartbeat_back_addrs": { 2024-09-10T19:31:03.867 INFO:teuthology.orchestra.run.smithi183.stdout: "addrvec": [ 2024-09-10T19:31:03.867 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:31:03.867 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v2", 2024-09-10T19:31:03.867 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.183:6816", 2024-09-10T19:31:03.867 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 3346539414 2024-09-10T19:31:03.867 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:31:03.868 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:31:03.868 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v1", 2024-09-10T19:31:03.868 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.183:6817", 2024-09-10T19:31:03.868 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 3346539414 2024-09-10T19:31:03.868 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:31:03.868 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:31:03.868 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:31:03.868 INFO:teuthology.orchestra.run.smithi183.stdout: "heartbeat_front_addrs": { 2024-09-10T19:31:03.868 INFO:teuthology.orchestra.run.smithi183.stdout: "addrvec": [ 2024-09-10T19:31:03.868 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:31:03.868 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v2", 2024-09-10T19:31:03.868 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.183:6814", 2024-09-10T19:31:03.868 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 3346539414 2024-09-10T19:31:03.868 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:31:03.869 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:31:03.869 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v1", 2024-09-10T19:31:03.869 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.183:6815", 2024-09-10T19:31:03.869 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 3346539414 2024-09-10T19:31:03.869 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:31:03.869 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:31:03.869 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:31:03.869 INFO:teuthology.orchestra.run.smithi183.stdout: "public_addr": "172.21.15.183:6811/3346539414", 2024-09-10T19:31:03.869 INFO:teuthology.orchestra.run.smithi183.stdout: "cluster_addr": "172.21.15.183:6813/3346539414", 2024-09-10T19:31:03.869 INFO:teuthology.orchestra.run.smithi183.stdout: "heartbeat_back_addr": "172.21.15.183:6817/3346539414", 2024-09-10T19:31:03.869 INFO:teuthology.orchestra.run.smithi183.stdout: "heartbeat_front_addr": "172.21.15.183:6815/3346539414", 2024-09-10T19:31:03.869 INFO:teuthology.orchestra.run.smithi183.stdout: "state": [ 2024-09-10T19:31:03.869 INFO:teuthology.orchestra.run.smithi183.stdout: "exists", 2024-09-10T19:31:03.870 INFO:teuthology.orchestra.run.smithi183.stdout: "up" 2024-09-10T19:31:03.870 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:31:03.870 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:31:03.870 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:31:03.870 INFO:teuthology.orchestra.run.smithi183.stdout: "osd": 1, 2024-09-10T19:31:03.870 INFO:teuthology.orchestra.run.smithi183.stdout: "uuid": "71a6c5dc-5109-4284-8d20-187d8e7afc9e", 2024-09-10T19:31:03.870 INFO:teuthology.orchestra.run.smithi183.stdout: "up": 1, 2024-09-10T19:31:03.870 INFO:teuthology.orchestra.run.smithi183.stdout: "in": 1, 2024-09-10T19:31:03.870 INFO:teuthology.orchestra.run.smithi183.stdout: "weight": 1, 2024-09-10T19:31:03.870 INFO:teuthology.orchestra.run.smithi183.stdout: "primary_affinity": 1, 2024-09-10T19:31:03.870 INFO:teuthology.orchestra.run.smithi183.stdout: "last_clean_begin": 138, 2024-09-10T19:31:03.870 INFO:teuthology.orchestra.run.smithi183.stdout: "last_clean_end": 255, 2024-09-10T19:31:03.870 INFO:teuthology.orchestra.run.smithi183.stdout: "up_from": 276, 2024-09-10T19:31:03.870 INFO:teuthology.orchestra.run.smithi183.stdout: "up_thru": 374, 2024-09-10T19:31:03.871 INFO:teuthology.orchestra.run.smithi183.stdout: "down_at": 256, 2024-09-10T19:31:03.871 INFO:teuthology.orchestra.run.smithi183.stdout: "lost_at": 0, 2024-09-10T19:31:03.871 INFO:teuthology.orchestra.run.smithi183.stdout: "public_addrs": { 2024-09-10T19:31:03.871 INFO:teuthology.orchestra.run.smithi183.stdout: "addrvec": [ 2024-09-10T19:31:03.871 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:31:03.871 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v2", 2024-09-10T19:31:03.871 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.183:6802", 2024-09-10T19:31:03.871 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 2764394151 2024-09-10T19:31:03.871 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:31:03.871 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:31:03.871 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v1", 2024-09-10T19:31:03.871 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.183:6803", 2024-09-10T19:31:03.871 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 2764394151 2024-09-10T19:31:03.872 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:31:03.872 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:31:03.872 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:31:03.872 INFO:teuthology.orchestra.run.smithi183.stdout: "cluster_addrs": { 2024-09-10T19:31:03.872 INFO:teuthology.orchestra.run.smithi183.stdout: "addrvec": [ 2024-09-10T19:31:03.872 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:31:03.872 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v2", 2024-09-10T19:31:03.872 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.183:6804", 2024-09-10T19:31:03.872 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 2764394151 2024-09-10T19:31:03.872 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:31:03.872 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:31:03.872 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v1", 2024-09-10T19:31:03.872 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.183:6805", 2024-09-10T19:31:03.873 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 2764394151 2024-09-10T19:31:03.873 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:31:03.873 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:31:03.873 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:31:03.873 INFO:teuthology.orchestra.run.smithi183.stdout: "heartbeat_back_addrs": { 2024-09-10T19:31:03.873 INFO:teuthology.orchestra.run.smithi183.stdout: "addrvec": [ 2024-09-10T19:31:03.873 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:31:03.873 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v2", 2024-09-10T19:31:03.873 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.183:6808", 2024-09-10T19:31:03.873 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 2764394151 2024-09-10T19:31:03.873 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:31:03.873 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:31:03.873 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v1", 2024-09-10T19:31:03.874 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.183:6809", 2024-09-10T19:31:03.874 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 2764394151 2024-09-10T19:31:03.874 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:31:03.874 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:31:03.874 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:31:03.874 INFO:teuthology.orchestra.run.smithi183.stdout: "heartbeat_front_addrs": { 2024-09-10T19:31:03.874 INFO:teuthology.orchestra.run.smithi183.stdout: "addrvec": [ 2024-09-10T19:31:03.874 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:31:03.874 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v2", 2024-09-10T19:31:03.874 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.183:6806", 2024-09-10T19:31:03.874 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 2764394151 2024-09-10T19:31:03.874 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:31:03.874 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:31:03.875 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v1", 2024-09-10T19:31:03.875 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.183:6807", 2024-09-10T19:31:03.875 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 2764394151 2024-09-10T19:31:03.875 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:31:03.875 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:31:03.875 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:31:03.875 INFO:teuthology.orchestra.run.smithi183.stdout: "public_addr": "172.21.15.183:6803/2764394151", 2024-09-10T19:31:03.875 INFO:teuthology.orchestra.run.smithi183.stdout: "cluster_addr": "172.21.15.183:6805/2764394151", 2024-09-10T19:31:03.875 INFO:teuthology.orchestra.run.smithi183.stdout: "heartbeat_back_addr": "172.21.15.183:6809/2764394151", 2024-09-10T19:31:03.875 INFO:teuthology.orchestra.run.smithi183.stdout: "heartbeat_front_addr": "172.21.15.183:6807/2764394151", 2024-09-10T19:31:03.875 INFO:teuthology.orchestra.run.smithi183.stdout: "state": [ 2024-09-10T19:31:03.875 INFO:teuthology.orchestra.run.smithi183.stdout: "exists", 2024-09-10T19:31:03.875 INFO:teuthology.orchestra.run.smithi183.stdout: "up" 2024-09-10T19:31:03.875 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:31:03.875 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:31:03.876 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:31:03.876 INFO:teuthology.orchestra.run.smithi183.stdout: "osd": 2, 2024-09-10T19:31:03.876 INFO:teuthology.orchestra.run.smithi183.stdout: "uuid": "fc310d5b-b89b-4655-9989-7e10ba356509", 2024-09-10T19:31:03.876 INFO:teuthology.orchestra.run.smithi183.stdout: "up": 1, 2024-09-10T19:31:03.876 INFO:teuthology.orchestra.run.smithi183.stdout: "in": 0, 2024-09-10T19:31:03.876 INFO:teuthology.orchestra.run.smithi183.stdout: "weight": 0, 2024-09-10T19:31:03.876 INFO:teuthology.orchestra.run.smithi183.stdout: "primary_affinity": 1, 2024-09-10T19:31:03.876 INFO:teuthology.orchestra.run.smithi183.stdout: "last_clean_begin": 0, 2024-09-10T19:31:03.876 INFO:teuthology.orchestra.run.smithi183.stdout: "last_clean_end": 0, 2024-09-10T19:31:03.876 INFO:teuthology.orchestra.run.smithi183.stdout: "up_from": 18, 2024-09-10T19:31:03.876 INFO:teuthology.orchestra.run.smithi183.stdout: "up_thru": 148, 2024-09-10T19:31:03.876 INFO:teuthology.orchestra.run.smithi183.stdout: "down_at": 0, 2024-09-10T19:31:03.876 INFO:teuthology.orchestra.run.smithi183.stdout: "lost_at": 0, 2024-09-10T19:31:03.877 INFO:teuthology.orchestra.run.smithi183.stdout: "public_addrs": { 2024-09-10T19:31:03.877 INFO:teuthology.orchestra.run.smithi183.stdout: "addrvec": [ 2024-09-10T19:31:03.877 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:31:03.877 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v2", 2024-09-10T19:31:03.877 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.183:6818", 2024-09-10T19:31:03.877 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 3730013786 2024-09-10T19:31:03.877 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:31:03.877 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:31:03.877 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v1", 2024-09-10T19:31:03.877 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.183:6819", 2024-09-10T19:31:03.878 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 3730013786 2024-09-10T19:31:03.878 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:31:03.878 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:31:03.878 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:31:03.878 INFO:teuthology.orchestra.run.smithi183.stdout: "cluster_addrs": { 2024-09-10T19:31:03.878 INFO:teuthology.orchestra.run.smithi183.stdout: "addrvec": [ 2024-09-10T19:31:03.878 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:31:03.878 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v2", 2024-09-10T19:31:03.878 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.183:6820", 2024-09-10T19:31:03.878 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 3730013786 2024-09-10T19:31:03.879 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:31:03.879 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:31:03.879 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v1", 2024-09-10T19:31:03.879 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.183:6821", 2024-09-10T19:31:03.879 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 3730013786 2024-09-10T19:31:03.879 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:31:03.879 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:31:03.879 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:31:03.879 INFO:teuthology.orchestra.run.smithi183.stdout: "heartbeat_back_addrs": { 2024-09-10T19:31:03.879 INFO:teuthology.orchestra.run.smithi183.stdout: "addrvec": [ 2024-09-10T19:31:03.879 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:31:03.880 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v2", 2024-09-10T19:31:03.880 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.183:6824", 2024-09-10T19:31:03.880 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 3730013786 2024-09-10T19:31:03.880 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:31:03.880 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:31:03.880 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v1", 2024-09-10T19:31:03.880 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.183:6825", 2024-09-10T19:31:03.880 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 3730013786 2024-09-10T19:31:03.880 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:31:03.880 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:31:03.880 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:31:03.880 INFO:teuthology.orchestra.run.smithi183.stdout: "heartbeat_front_addrs": { 2024-09-10T19:31:03.881 INFO:teuthology.orchestra.run.smithi183.stdout: "addrvec": [ 2024-09-10T19:31:03.881 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:31:03.881 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v2", 2024-09-10T19:31:03.881 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.183:6822", 2024-09-10T19:31:03.881 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 3730013786 2024-09-10T19:31:03.881 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:31:03.881 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:31:03.881 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v1", 2024-09-10T19:31:03.881 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.183:6823", 2024-09-10T19:31:03.881 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 3730013786 2024-09-10T19:31:03.882 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:31:03.882 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:31:03.882 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:31:03.882 INFO:teuthology.orchestra.run.smithi183.stdout: "public_addr": "172.21.15.183:6819/3730013786", 2024-09-10T19:31:03.882 INFO:teuthology.orchestra.run.smithi183.stdout: "cluster_addr": "172.21.15.183:6821/3730013786", 2024-09-10T19:31:03.882 INFO:teuthology.orchestra.run.smithi183.stdout: "heartbeat_back_addr": "172.21.15.183:6825/3730013786", 2024-09-10T19:31:03.882 INFO:teuthology.orchestra.run.smithi183.stdout: "heartbeat_front_addr": "172.21.15.183:6823/3730013786", 2024-09-10T19:31:03.882 INFO:teuthology.orchestra.run.smithi183.stdout: "state": [ 2024-09-10T19:31:03.882 INFO:teuthology.orchestra.run.smithi183.stdout: "exists", 2024-09-10T19:31:03.882 INFO:teuthology.orchestra.run.smithi183.stdout: "up" 2024-09-10T19:31:03.883 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:31:03.883 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:31:03.883 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:31:03.883 INFO:teuthology.orchestra.run.smithi183.stdout: "osd": 3, 2024-09-10T19:31:03.883 INFO:teuthology.orchestra.run.smithi183.stdout: "uuid": "7fd07a95-ad22-43f5-a968-539552283e89", 2024-09-10T19:31:03.883 INFO:teuthology.orchestra.run.smithi183.stdout: "up": 1, 2024-09-10T19:31:03.883 INFO:teuthology.orchestra.run.smithi183.stdout: "in": 1, 2024-09-10T19:31:03.883 INFO:teuthology.orchestra.run.smithi183.stdout: "weight": 1, 2024-09-10T19:31:03.883 INFO:teuthology.orchestra.run.smithi183.stdout: "primary_affinity": 1, 2024-09-10T19:31:03.883 INFO:teuthology.orchestra.run.smithi183.stdout: "last_clean_begin": 0, 2024-09-10T19:31:03.883 INFO:teuthology.orchestra.run.smithi183.stdout: "last_clean_end": 0, 2024-09-10T19:31:03.883 INFO:teuthology.orchestra.run.smithi183.stdout: "up_from": 26, 2024-09-10T19:31:03.883 INFO:teuthology.orchestra.run.smithi183.stdout: "up_thru": 374, 2024-09-10T19:31:03.884 INFO:teuthology.orchestra.run.smithi183.stdout: "down_at": 0, 2024-09-10T19:31:03.884 INFO:teuthology.orchestra.run.smithi183.stdout: "lost_at": 0, 2024-09-10T19:31:03.884 INFO:teuthology.orchestra.run.smithi183.stdout: "public_addrs": { 2024-09-10T19:31:03.884 INFO:teuthology.orchestra.run.smithi183.stdout: "addrvec": [ 2024-09-10T19:31:03.884 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:31:03.884 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v2", 2024-09-10T19:31:03.884 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.183:6826", 2024-09-10T19:31:03.884 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 3522836984 2024-09-10T19:31:03.884 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:31:03.884 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:31:03.884 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v1", 2024-09-10T19:31:03.884 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.183:6827", 2024-09-10T19:31:03.884 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 3522836984 2024-09-10T19:31:03.884 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:31:03.885 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:31:03.885 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:31:03.885 INFO:teuthology.orchestra.run.smithi183.stdout: "cluster_addrs": { 2024-09-10T19:31:03.885 INFO:teuthology.orchestra.run.smithi183.stdout: "addrvec": [ 2024-09-10T19:31:03.885 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:31:03.885 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v2", 2024-09-10T19:31:03.885 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.183:6828", 2024-09-10T19:31:03.885 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 3522836984 2024-09-10T19:31:03.885 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:31:03.885 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:31:03.885 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v1", 2024-09-10T19:31:03.885 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.183:6829", 2024-09-10T19:31:03.885 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 3522836984 2024-09-10T19:31:03.886 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:31:03.886 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:31:03.886 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:31:03.886 INFO:teuthology.orchestra.run.smithi183.stdout: "heartbeat_back_addrs": { 2024-09-10T19:31:03.886 INFO:teuthology.orchestra.run.smithi183.stdout: "addrvec": [ 2024-09-10T19:31:03.886 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:31:03.886 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v2", 2024-09-10T19:31:03.886 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.183:6832", 2024-09-10T19:31:03.886 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 3522836984 2024-09-10T19:31:03.886 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:31:03.886 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:31:03.886 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v1", 2024-09-10T19:31:03.886 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.183:6833", 2024-09-10T19:31:03.887 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 3522836984 2024-09-10T19:31:03.887 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:31:03.887 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:31:03.887 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:31:03.887 INFO:teuthology.orchestra.run.smithi183.stdout: "heartbeat_front_addrs": { 2024-09-10T19:31:03.887 INFO:teuthology.orchestra.run.smithi183.stdout: "addrvec": [ 2024-09-10T19:31:03.887 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:31:03.887 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v2", 2024-09-10T19:31:03.887 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.183:6830", 2024-09-10T19:31:03.887 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 3522836984 2024-09-10T19:31:03.887 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:31:03.887 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:31:03.887 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v1", 2024-09-10T19:31:03.887 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.183:6831", 2024-09-10T19:31:03.888 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 3522836984 2024-09-10T19:31:03.888 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:31:03.888 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:31:03.888 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:31:03.888 INFO:teuthology.orchestra.run.smithi183.stdout: "public_addr": "172.21.15.183:6827/3522836984", 2024-09-10T19:31:03.888 INFO:teuthology.orchestra.run.smithi183.stdout: "cluster_addr": "172.21.15.183:6829/3522836984", 2024-09-10T19:31:03.888 INFO:teuthology.orchestra.run.smithi183.stdout: "heartbeat_back_addr": "172.21.15.183:6833/3522836984", 2024-09-10T19:31:03.888 INFO:teuthology.orchestra.run.smithi183.stdout: "heartbeat_front_addr": "172.21.15.183:6831/3522836984", 2024-09-10T19:31:03.888 INFO:teuthology.orchestra.run.smithi183.stdout: "state": [ 2024-09-10T19:31:03.888 INFO:teuthology.orchestra.run.smithi183.stdout: "exists", 2024-09-10T19:31:03.888 INFO:teuthology.orchestra.run.smithi183.stdout: "up" 2024-09-10T19:31:03.888 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:31:03.889 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:31:03.889 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:31:03.889 INFO:teuthology.orchestra.run.smithi183.stdout: "osd": 4, 2024-09-10T19:31:03.889 INFO:teuthology.orchestra.run.smithi183.stdout: "uuid": "631083fb-e503-44da-9b6b-154e88539ee7", 2024-09-10T19:31:03.889 INFO:teuthology.orchestra.run.smithi183.stdout: "up": 1, 2024-09-10T19:31:03.889 INFO:teuthology.orchestra.run.smithi183.stdout: "in": 1, 2024-09-10T19:31:03.889 INFO:teuthology.orchestra.run.smithi183.stdout: "weight": 1, 2024-09-10T19:31:03.889 INFO:teuthology.orchestra.run.smithi183.stdout: "primary_affinity": 1, 2024-09-10T19:31:03.889 INFO:teuthology.orchestra.run.smithi183.stdout: "last_clean_begin": 0, 2024-09-10T19:31:03.889 INFO:teuthology.orchestra.run.smithi183.stdout: "last_clean_end": 0, 2024-09-10T19:31:03.889 INFO:teuthology.orchestra.run.smithi183.stdout: "up_from": 32, 2024-09-10T19:31:03.889 INFO:teuthology.orchestra.run.smithi183.stdout: "up_thru": 374, 2024-09-10T19:31:03.889 INFO:teuthology.orchestra.run.smithi183.stdout: "down_at": 0, 2024-09-10T19:31:03.889 INFO:teuthology.orchestra.run.smithi183.stdout: "lost_at": 0, 2024-09-10T19:31:03.890 INFO:teuthology.orchestra.run.smithi183.stdout: "public_addrs": { 2024-09-10T19:31:03.890 INFO:teuthology.orchestra.run.smithi183.stdout: "addrvec": [ 2024-09-10T19:31:03.890 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:31:03.890 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v2", 2024-09-10T19:31:03.890 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.191:6800", 2024-09-10T19:31:03.890 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 2674828539 2024-09-10T19:31:03.890 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:31:03.890 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:31:03.890 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v1", 2024-09-10T19:31:03.890 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.191:6801", 2024-09-10T19:31:03.890 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 2674828539 2024-09-10T19:31:03.890 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:31:03.890 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:31:03.891 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:31:03.891 INFO:teuthology.orchestra.run.smithi183.stdout: "cluster_addrs": { 2024-09-10T19:31:03.891 INFO:teuthology.orchestra.run.smithi183.stdout: "addrvec": [ 2024-09-10T19:31:03.891 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:31:03.891 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v2", 2024-09-10T19:31:03.891 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.191:6802", 2024-09-10T19:31:03.891 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 2674828539 2024-09-10T19:31:03.891 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:31:03.891 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:31:03.891 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v1", 2024-09-10T19:31:03.891 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.191:6803", 2024-09-10T19:31:03.891 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 2674828539 2024-09-10T19:31:03.891 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:31:03.891 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:31:03.892 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:31:03.892 INFO:teuthology.orchestra.run.smithi183.stdout: "heartbeat_back_addrs": { 2024-09-10T19:31:03.892 INFO:teuthology.orchestra.run.smithi183.stdout: "addrvec": [ 2024-09-10T19:31:03.892 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:31:03.892 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v2", 2024-09-10T19:31:03.892 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.191:6806", 2024-09-10T19:31:03.892 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 2674828539 2024-09-10T19:31:03.892 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:31:03.892 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:31:03.892 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v1", 2024-09-10T19:31:03.892 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.191:6807", 2024-09-10T19:31:03.892 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 2674828539 2024-09-10T19:31:03.892 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:31:03.893 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:31:03.893 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:31:03.893 INFO:teuthology.orchestra.run.smithi183.stdout: "heartbeat_front_addrs": { 2024-09-10T19:31:03.893 INFO:teuthology.orchestra.run.smithi183.stdout: "addrvec": [ 2024-09-10T19:31:03.893 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:31:03.893 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v2", 2024-09-10T19:31:03.893 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.191:6804", 2024-09-10T19:31:03.893 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 2674828539 2024-09-10T19:31:03.893 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:31:03.893 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:31:03.893 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v1", 2024-09-10T19:31:03.893 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.191:6805", 2024-09-10T19:31:03.893 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 2674828539 2024-09-10T19:31:03.894 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:31:03.894 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:31:03.894 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:31:03.894 INFO:teuthology.orchestra.run.smithi183.stdout: "public_addr": "172.21.15.191:6801/2674828539", 2024-09-10T19:31:03.894 INFO:teuthology.orchestra.run.smithi183.stdout: "cluster_addr": "172.21.15.191:6803/2674828539", 2024-09-10T19:31:03.894 INFO:teuthology.orchestra.run.smithi183.stdout: "heartbeat_back_addr": "172.21.15.191:6807/2674828539", 2024-09-10T19:31:03.894 INFO:teuthology.orchestra.run.smithi183.stdout: "heartbeat_front_addr": "172.21.15.191:6805/2674828539", 2024-09-10T19:31:03.894 INFO:teuthology.orchestra.run.smithi183.stdout: "state": [ 2024-09-10T19:31:03.894 INFO:teuthology.orchestra.run.smithi183.stdout: "exists", 2024-09-10T19:31:03.894 INFO:teuthology.orchestra.run.smithi183.stdout: "up" 2024-09-10T19:31:03.894 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:31:03.894 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:31:03.894 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:31:03.895 INFO:teuthology.orchestra.run.smithi183.stdout: "osd": 5, 2024-09-10T19:31:03.895 INFO:teuthology.orchestra.run.smithi183.stdout: "uuid": "6b963695-3eca-4e6e-9207-05b8010ebff9", 2024-09-10T19:31:03.895 INFO:teuthology.orchestra.run.smithi183.stdout: "up": 1, 2024-09-10T19:31:03.895 INFO:teuthology.orchestra.run.smithi183.stdout: "in": 1, 2024-09-10T19:31:03.895 INFO:teuthology.orchestra.run.smithi183.stdout: "weight": 1, 2024-09-10T19:31:03.895 INFO:teuthology.orchestra.run.smithi183.stdout: "primary_affinity": 1, 2024-09-10T19:31:03.895 INFO:teuthology.orchestra.run.smithi183.stdout: "last_clean_begin": 0, 2024-09-10T19:31:03.895 INFO:teuthology.orchestra.run.smithi183.stdout: "last_clean_end": 0, 2024-09-10T19:31:03.895 INFO:teuthology.orchestra.run.smithi183.stdout: "up_from": 39, 2024-09-10T19:31:03.895 INFO:teuthology.orchestra.run.smithi183.stdout: "up_thru": 374, 2024-09-10T19:31:03.895 INFO:teuthology.orchestra.run.smithi183.stdout: "down_at": 0, 2024-09-10T19:31:03.895 INFO:teuthology.orchestra.run.smithi183.stdout: "lost_at": 0, 2024-09-10T19:31:03.895 INFO:teuthology.orchestra.run.smithi183.stdout: "public_addrs": { 2024-09-10T19:31:03.896 INFO:teuthology.orchestra.run.smithi183.stdout: "addrvec": [ 2024-09-10T19:31:03.896 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:31:03.896 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v2", 2024-09-10T19:31:03.896 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.191:6808", 2024-09-10T19:31:03.896 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 681733378 2024-09-10T19:31:03.896 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:31:03.896 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:31:03.896 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v1", 2024-09-10T19:31:03.896 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.191:6809", 2024-09-10T19:31:03.896 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 681733378 2024-09-10T19:31:03.896 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:31:03.896 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:31:03.896 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:31:03.896 INFO:teuthology.orchestra.run.smithi183.stdout: "cluster_addrs": { 2024-09-10T19:31:03.897 INFO:teuthology.orchestra.run.smithi183.stdout: "addrvec": [ 2024-09-10T19:31:03.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:31:03.898 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:31:03.898 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:31:03.899 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:31:03.899 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v2", 2024-09-10T19:31:03.899 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.191:6810", 2024-09-10T19:31:03.899 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 681733378 2024-09-10T19:31:03.899 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:31:03.899 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:31:03.899 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v1", 2024-09-10T19:31:03.899 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.191:6811", 2024-09-10T19:31:03.899 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 681733378 2024-09-10T19:31:03.899 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:31:03.899 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:31:03.900 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:31:03.900 INFO:teuthology.orchestra.run.smithi183.stdout: "heartbeat_back_addrs": { 2024-09-10T19:31:03.900 INFO:teuthology.orchestra.run.smithi183.stdout: "addrvec": [ 2024-09-10T19:31:03.900 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:31:03.900 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v2", 2024-09-10T19:31:03.900 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.191:6814", 2024-09-10T19:31:03.900 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 681733378 2024-09-10T19:31:03.900 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:31:03.900 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:31:03.900 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v1", 2024-09-10T19:31:03.900 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.191:6815", 2024-09-10T19:31:03.900 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 681733378 2024-09-10T19:31:03.900 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:31:03.901 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:31:03.901 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:31:03.901 INFO:teuthology.orchestra.run.smithi183.stdout: "heartbeat_front_addrs": { 2024-09-10T19:31:03.901 INFO:teuthology.orchestra.run.smithi183.stdout: "addrvec": [ 2024-09-10T19:31:03.901 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:31:03.901 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v2", 2024-09-10T19:31:03.901 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.191:6812", 2024-09-10T19:31:03.901 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 681733378 2024-09-10T19:31:03.901 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:31:03.901 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:31:03.902 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v1", 2024-09-10T19:31:03.902 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.191:6813", 2024-09-10T19:31:03.902 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 681733378 2024-09-10T19:31:03.902 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:31:03.902 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:31:03.902 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:31:03.902 INFO:teuthology.orchestra.run.smithi183.stdout: "public_addr": "172.21.15.191:6809/681733378", 2024-09-10T19:31:03.902 INFO:teuthology.orchestra.run.smithi183.stdout: "cluster_addr": "172.21.15.191:6811/681733378", 2024-09-10T19:31:03.902 INFO:teuthology.orchestra.run.smithi183.stdout: "heartbeat_back_addr": "172.21.15.191:6815/681733378", 2024-09-10T19:31:03.902 INFO:teuthology.orchestra.run.smithi183.stdout: "heartbeat_front_addr": "172.21.15.191:6813/681733378", 2024-09-10T19:31:03.902 INFO:teuthology.orchestra.run.smithi183.stdout: "state": [ 2024-09-10T19:31:03.902 INFO:teuthology.orchestra.run.smithi183.stdout: "exists", 2024-09-10T19:31:03.902 INFO:teuthology.orchestra.run.smithi183.stdout: "up" 2024-09-10T19:31:03.903 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:31:03.903 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:31:03.903 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:31:03.903 INFO:teuthology.orchestra.run.smithi183.stdout: "osd": 6, 2024-09-10T19:31:03.903 INFO:teuthology.orchestra.run.smithi183.stdout: "uuid": "2dc9754f-6aef-4e19-8605-2f1591921811", 2024-09-10T19:31:03.903 INFO:teuthology.orchestra.run.smithi183.stdout: "up": 1, 2024-09-10T19:31:03.903 INFO:teuthology.orchestra.run.smithi183.stdout: "in": 1, 2024-09-10T19:31:03.903 INFO:teuthology.orchestra.run.smithi183.stdout: "weight": 1, 2024-09-10T19:31:03.903 INFO:teuthology.orchestra.run.smithi183.stdout: "primary_affinity": 1, 2024-09-10T19:31:03.903 INFO:teuthology.orchestra.run.smithi183.stdout: "last_clean_begin": 0, 2024-09-10T19:31:03.903 INFO:teuthology.orchestra.run.smithi183.stdout: "last_clean_end": 0, 2024-09-10T19:31:03.904 INFO:teuthology.orchestra.run.smithi183.stdout: "up_from": 47, 2024-09-10T19:31:03.904 INFO:teuthology.orchestra.run.smithi183.stdout: "up_thru": 391, 2024-09-10T19:31:03.904 INFO:teuthology.orchestra.run.smithi183.stdout: "down_at": 0, 2024-09-10T19:31:03.904 INFO:teuthology.orchestra.run.smithi183.stdout: "lost_at": 0, 2024-09-10T19:31:03.904 INFO:teuthology.orchestra.run.smithi183.stdout: "public_addrs": { 2024-09-10T19:31:03.904 INFO:teuthology.orchestra.run.smithi183.stdout: "addrvec": [ 2024-09-10T19:31:03.904 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:31:03.904 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v2", 2024-09-10T19:31:03.904 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.191:6816", 2024-09-10T19:31:03.904 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 2825813477 2024-09-10T19:31:03.904 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:31:03.905 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:31:03.905 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v1", 2024-09-10T19:31:03.905 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.191:6817", 2024-09-10T19:31:03.905 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 2825813477 2024-09-10T19:31:03.905 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:31:03.905 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:31:03.905 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:31:03.906 INFO:teuthology.orchestra.run.smithi183.stdout: "cluster_addrs": { 2024-09-10T19:31:03.906 INFO:teuthology.orchestra.run.smithi183.stdout: "addrvec": [ 2024-09-10T19:31:03.906 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:31:03.906 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v2", 2024-09-10T19:31:03.906 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.191:6818", 2024-09-10T19:31:03.906 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 2825813477 2024-09-10T19:31:03.906 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:31:03.906 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:31:03.906 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v1", 2024-09-10T19:31:03.906 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.191:6819", 2024-09-10T19:31:03.906 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 2825813477 2024-09-10T19:31:03.906 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:31:03.906 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:31:03.906 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:31:03.907 INFO:teuthology.orchestra.run.smithi183.stdout: "heartbeat_back_addrs": { 2024-09-10T19:31:03.907 INFO:teuthology.orchestra.run.smithi183.stdout: "addrvec": [ 2024-09-10T19:31:03.907 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:31:03.907 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v2", 2024-09-10T19:31:03.907 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.191:6822", 2024-09-10T19:31:03.907 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 2825813477 2024-09-10T19:31:03.907 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:31:03.907 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:31:03.907 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v1", 2024-09-10T19:31:03.907 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.191:6823", 2024-09-10T19:31:03.907 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 2825813477 2024-09-10T19:31:03.907 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:31:03.907 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:31:03.907 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:31:03.908 INFO:teuthology.orchestra.run.smithi183.stdout: "heartbeat_front_addrs": { 2024-09-10T19:31:03.908 INFO:teuthology.orchestra.run.smithi183.stdout: "addrvec": [ 2024-09-10T19:31:03.908 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:31:03.908 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v2", 2024-09-10T19:31:03.908 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.191:6820", 2024-09-10T19:31:03.908 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 2825813477 2024-09-10T19:31:03.908 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:31:03.908 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:31:03.908 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v1", 2024-09-10T19:31:03.908 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.191:6821", 2024-09-10T19:31:03.908 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 2825813477 2024-09-10T19:31:03.908 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:31:03.908 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:31:03.909 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:31:03.909 INFO:teuthology.orchestra.run.smithi183.stdout: "public_addr": "172.21.15.191:6817/2825813477", 2024-09-10T19:31:03.909 INFO:teuthology.orchestra.run.smithi183.stdout: "cluster_addr": "172.21.15.191:6819/2825813477", 2024-09-10T19:31:03.909 INFO:teuthology.orchestra.run.smithi183.stdout: "heartbeat_back_addr": "172.21.15.191:6823/2825813477", 2024-09-10T19:31:03.909 INFO:teuthology.orchestra.run.smithi183.stdout: "heartbeat_front_addr": "172.21.15.191:6821/2825813477", 2024-09-10T19:31:03.909 INFO:teuthology.orchestra.run.smithi183.stdout: "state": [ 2024-09-10T19:31:03.909 INFO:teuthology.orchestra.run.smithi183.stdout: "exists", 2024-09-10T19:31:03.909 INFO:teuthology.orchestra.run.smithi183.stdout: "up" 2024-09-10T19:31:03.909 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:31:03.909 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:31:03.909 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:31:03.909 INFO:teuthology.orchestra.run.smithi183.stdout: "osd": 7, 2024-09-10T19:31:03.910 INFO:teuthology.orchestra.run.smithi183.stdout: "uuid": "1a9a299e-d5ba-4b9a-85d5-7f6a5bc754e1", 2024-09-10T19:31:03.910 INFO:teuthology.orchestra.run.smithi183.stdout: "up": 1, 2024-09-10T19:31:03.910 INFO:teuthology.orchestra.run.smithi183.stdout: "in": 1, 2024-09-10T19:31:03.910 INFO:teuthology.orchestra.run.smithi183.stdout: "weight": 1, 2024-09-10T19:31:03.910 INFO:teuthology.orchestra.run.smithi183.stdout: "primary_affinity": 1, 2024-09-10T19:31:03.910 INFO:teuthology.orchestra.run.smithi183.stdout: "last_clean_begin": 0, 2024-09-10T19:31:03.910 INFO:teuthology.orchestra.run.smithi183.stdout: "last_clean_end": 0, 2024-09-10T19:31:03.910 INFO:teuthology.orchestra.run.smithi183.stdout: "up_from": 54, 2024-09-10T19:31:03.910 INFO:teuthology.orchestra.run.smithi183.stdout: "up_thru": 374, 2024-09-10T19:31:03.910 INFO:teuthology.orchestra.run.smithi183.stdout: "down_at": 0, 2024-09-10T19:31:03.910 INFO:teuthology.orchestra.run.smithi183.stdout: "lost_at": 0, 2024-09-10T19:31:03.910 INFO:teuthology.orchestra.run.smithi183.stdout: "public_addrs": { 2024-09-10T19:31:03.910 INFO:teuthology.orchestra.run.smithi183.stdout: "addrvec": [ 2024-09-10T19:31:03.910 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:31:03.911 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v2", 2024-09-10T19:31:03.911 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.191:6824", 2024-09-10T19:31:03.911 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 1422610021 2024-09-10T19:31:03.911 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:31:03.911 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:31:03.911 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v1", 2024-09-10T19:31:03.911 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.191:6825", 2024-09-10T19:31:03.911 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 1422610021 2024-09-10T19:31:03.911 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:31:03.911 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:31:03.911 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:31:03.911 INFO:teuthology.orchestra.run.smithi183.stdout: "cluster_addrs": { 2024-09-10T19:31:03.911 INFO:teuthology.orchestra.run.smithi183.stdout: "addrvec": [ 2024-09-10T19:31:03.911 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:31:03.912 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v2", 2024-09-10T19:31:03.912 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.191:6826", 2024-09-10T19:31:03.912 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 1422610021 2024-09-10T19:31:03.912 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:31:03.912 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:31:03.912 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v1", 2024-09-10T19:31:03.912 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.191:6827", 2024-09-10T19:31:03.912 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 1422610021 2024-09-10T19:31:03.912 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:31:03.912 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:31:03.912 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:31:03.912 INFO:teuthology.orchestra.run.smithi183.stdout: "heartbeat_back_addrs": { 2024-09-10T19:31:03.912 INFO:teuthology.orchestra.run.smithi183.stdout: "addrvec": [ 2024-09-10T19:31:03.913 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:31:03.913 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v2", 2024-09-10T19:31:03.913 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.191:6830", 2024-09-10T19:31:03.913 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 1422610021 2024-09-10T19:31:03.913 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:31:03.913 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:31:03.913 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v1", 2024-09-10T19:31:03.913 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.191:6831", 2024-09-10T19:31:03.913 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 1422610021 2024-09-10T19:31:03.913 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:31:03.913 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:31:03.913 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:31:03.914 INFO:teuthology.orchestra.run.smithi183.stdout: "heartbeat_front_addrs": { 2024-09-10T19:31:03.914 INFO:teuthology.orchestra.run.smithi183.stdout: "addrvec": [ 2024-09-10T19:31:03.914 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:31:03.914 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v2", 2024-09-10T19:31:03.914 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.191:6828", 2024-09-10T19:31:03.914 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 1422610021 2024-09-10T19:31:03.914 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:31:03.914 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:31:03.914 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v1", 2024-09-10T19:31:03.914 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "172.21.15.191:6829", 2024-09-10T19:31:03.914 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 1422610021 2024-09-10T19:31:03.914 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:31:03.914 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:31:03.915 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:31:03.915 INFO:teuthology.orchestra.run.smithi183.stdout: "public_addr": "172.21.15.191:6825/1422610021", 2024-09-10T19:31:03.915 INFO:teuthology.orchestra.run.smithi183.stdout: "cluster_addr": "172.21.15.191:6827/1422610021", 2024-09-10T19:31:03.915 INFO:teuthology.orchestra.run.smithi183.stdout: "heartbeat_back_addr": "172.21.15.191:6831/1422610021", 2024-09-10T19:31:03.915 INFO:teuthology.orchestra.run.smithi183.stdout: "heartbeat_front_addr": "172.21.15.191:6829/1422610021", 2024-09-10T19:31:03.915 INFO:teuthology.orchestra.run.smithi183.stdout: "state": [ 2024-09-10T19:31:03.915 INFO:teuthology.orchestra.run.smithi183.stdout: "exists", 2024-09-10T19:31:03.915 INFO:teuthology.orchestra.run.smithi183.stdout: "up" 2024-09-10T19:31:03.915 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:31:03.915 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:31:03.915 INFO:teuthology.orchestra.run.smithi183.stdout: ], 2024-09-10T19:31:03.915 INFO:teuthology.orchestra.run.smithi183.stdout: "osd_xinfo": [ 2024-09-10T19:31:03.915 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:31:03.915 INFO:teuthology.orchestra.run.smithi183.stdout: "osd": 0, 2024-09-10T19:31:03.916 INFO:teuthology.orchestra.run.smithi183.stdout: "down_stamp": "0.000000", 2024-09-10T19:31:03.916 INFO:teuthology.orchestra.run.smithi183.stdout: "laggy_probability": 0, 2024-09-10T19:31:03.916 INFO:teuthology.orchestra.run.smithi183.stdout: "laggy_interval": 0, 2024-09-10T19:31:03.916 INFO:teuthology.orchestra.run.smithi183.stdout: "features": 4540138320759226367, 2024-09-10T19:31:03.916 INFO:teuthology.orchestra.run.smithi183.stdout: "old_weight": 0, 2024-09-10T19:31:03.916 INFO:teuthology.orchestra.run.smithi183.stdout: "last_purged_snaps_scrub": "2024-09-10T19:30:55.624776+0000", 2024-09-10T19:31:03.916 INFO:teuthology.orchestra.run.smithi183.stdout: "dead_epoch": 262 2024-09-10T19:31:03.916 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:31:03.916 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:31:03.916 INFO:teuthology.orchestra.run.smithi183.stdout: "osd": 1, 2024-09-10T19:31:03.916 INFO:teuthology.orchestra.run.smithi183.stdout: "down_stamp": "0.000000", 2024-09-10T19:31:03.916 INFO:teuthology.orchestra.run.smithi183.stdout: "laggy_probability": 0, 2024-09-10T19:31:03.917 INFO:teuthology.orchestra.run.smithi183.stdout: "laggy_interval": 0, 2024-09-10T19:31:03.917 INFO:teuthology.orchestra.run.smithi183.stdout: "features": 4540138320759226367, 2024-09-10T19:31:03.917 INFO:teuthology.orchestra.run.smithi183.stdout: "old_weight": 0, 2024-09-10T19:31:03.917 INFO:teuthology.orchestra.run.smithi183.stdout: "last_purged_snaps_scrub": "2024-09-10T19:30:05.535777+0000", 2024-09-10T19:31:03.917 INFO:teuthology.orchestra.run.smithi183.stdout: "dead_epoch": 255 2024-09-10T19:31:03.917 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:31:03.917 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:31:03.917 INFO:teuthology.orchestra.run.smithi183.stdout: "osd": 2, 2024-09-10T19:31:03.917 INFO:teuthology.orchestra.run.smithi183.stdout: "down_stamp": "0.000000", 2024-09-10T19:31:03.917 INFO:teuthology.orchestra.run.smithi183.stdout: "laggy_probability": 0, 2024-09-10T19:31:03.917 INFO:teuthology.orchestra.run.smithi183.stdout: "laggy_interval": 0, 2024-09-10T19:31:03.917 INFO:teuthology.orchestra.run.smithi183.stdout: "features": 4540138320759226367, 2024-09-10T19:31:03.917 INFO:teuthology.orchestra.run.smithi183.stdout: "old_weight": 65536, 2024-09-10T19:31:03.918 INFO:teuthology.orchestra.run.smithi183.stdout: "last_purged_snaps_scrub": "2024-09-10T19:30:53.240003+0000", 2024-09-10T19:31:03.918 INFO:teuthology.orchestra.run.smithi183.stdout: "dead_epoch": 0 2024-09-10T19:31:03.918 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:31:03.918 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:31:03.918 INFO:teuthology.orchestra.run.smithi183.stdout: "osd": 3, 2024-09-10T19:31:03.918 INFO:teuthology.orchestra.run.smithi183.stdout: "down_stamp": "0.000000", 2024-09-10T19:31:03.918 INFO:teuthology.orchestra.run.smithi183.stdout: "laggy_probability": 0, 2024-09-10T19:31:03.918 INFO:teuthology.orchestra.run.smithi183.stdout: "laggy_interval": 0, 2024-09-10T19:31:03.918 INFO:teuthology.orchestra.run.smithi183.stdout: "features": 4540138320759226367, 2024-09-10T19:31:03.918 INFO:teuthology.orchestra.run.smithi183.stdout: "old_weight": 0, 2024-09-10T19:31:03.918 INFO:teuthology.orchestra.run.smithi183.stdout: "last_purged_snaps_scrub": "2024-09-10T19:30:16.451748+0000", 2024-09-10T19:31:03.918 INFO:teuthology.orchestra.run.smithi183.stdout: "dead_epoch": 0 2024-09-10T19:31:03.918 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:31:03.919 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:31:03.919 INFO:teuthology.orchestra.run.smithi183.stdout: "osd": 4, 2024-09-10T19:31:03.919 INFO:teuthology.orchestra.run.smithi183.stdout: "down_stamp": "0.000000", 2024-09-10T19:31:03.919 INFO:teuthology.orchestra.run.smithi183.stdout: "laggy_probability": 0, 2024-09-10T19:31:03.919 INFO:teuthology.orchestra.run.smithi183.stdout: "laggy_interval": 0, 2024-09-10T19:31:03.919 INFO:teuthology.orchestra.run.smithi183.stdout: "features": 4540138320759226367, 2024-09-10T19:31:03.919 INFO:teuthology.orchestra.run.smithi183.stdout: "old_weight": 0, 2024-09-10T19:31:03.919 INFO:teuthology.orchestra.run.smithi183.stdout: "last_purged_snaps_scrub": "2024-09-10T19:30:45.814254+0000", 2024-09-10T19:31:03.919 INFO:teuthology.orchestra.run.smithi183.stdout: "dead_epoch": 0 2024-09-10T19:31:03.919 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:31:03.919 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:31:03.919 INFO:teuthology.orchestra.run.smithi183.stdout: "osd": 5, 2024-09-10T19:31:03.919 INFO:teuthology.orchestra.run.smithi183.stdout: "down_stamp": "0.000000", 2024-09-10T19:31:03.920 INFO:teuthology.orchestra.run.smithi183.stdout: "laggy_probability": 0, 2024-09-10T19:31:03.920 INFO:teuthology.orchestra.run.smithi183.stdout: "laggy_interval": 0, 2024-09-10T19:31:03.920 INFO:teuthology.orchestra.run.smithi183.stdout: "features": 4540138320759226367, 2024-09-10T19:31:03.920 INFO:teuthology.orchestra.run.smithi183.stdout: "old_weight": 0, 2024-09-10T19:31:03.920 INFO:teuthology.orchestra.run.smithi183.stdout: "last_purged_snaps_scrub": "2024-09-10T19:30:01.762348+0000", 2024-09-10T19:31:03.920 INFO:teuthology.orchestra.run.smithi183.stdout: "dead_epoch": 0 2024-09-10T19:31:03.920 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:31:03.920 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:31:03.920 INFO:teuthology.orchestra.run.smithi183.stdout: "osd": 6, 2024-09-10T19:31:03.920 INFO:teuthology.orchestra.run.smithi183.stdout: "down_stamp": "0.000000", 2024-09-10T19:31:03.920 INFO:teuthology.orchestra.run.smithi183.stdout: "laggy_probability": 0, 2024-09-10T19:31:03.920 INFO:teuthology.orchestra.run.smithi183.stdout: "laggy_interval": 0, 2024-09-10T19:31:03.920 INFO:teuthology.orchestra.run.smithi183.stdout: "features": 4540138320759226367, 2024-09-10T19:31:03.920 INFO:teuthology.orchestra.run.smithi183.stdout: "old_weight": 0, 2024-09-10T19:31:03.921 INFO:teuthology.orchestra.run.smithi183.stdout: "last_purged_snaps_scrub": "2024-09-10T19:29:53.533700+0000", 2024-09-10T19:31:03.921 INFO:teuthology.orchestra.run.smithi183.stdout: "dead_epoch": 0 2024-09-10T19:31:03.921 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:31:03.921 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:31:03.921 INFO:teuthology.orchestra.run.smithi183.stdout: "osd": 7, 2024-09-10T19:31:03.921 INFO:teuthology.orchestra.run.smithi183.stdout: "down_stamp": "0.000000", 2024-09-10T19:31:03.921 INFO:teuthology.orchestra.run.smithi183.stdout: "laggy_probability": 0, 2024-09-10T19:31:03.921 INFO:teuthology.orchestra.run.smithi183.stdout: "laggy_interval": 0, 2024-09-10T19:31:03.921 INFO:teuthology.orchestra.run.smithi183.stdout: "features": 4540138320759226367, 2024-09-10T19:31:03.921 INFO:teuthology.orchestra.run.smithi183.stdout: "old_weight": 0, 2024-09-10T19:31:03.921 INFO:teuthology.orchestra.run.smithi183.stdout: "last_purged_snaps_scrub": "2024-09-10T19:30:11.918745+0000", 2024-09-10T19:31:03.921 INFO:teuthology.orchestra.run.smithi183.stdout: "dead_epoch": 0 2024-09-10T19:31:03.921 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:31:03.922 INFO:teuthology.orchestra.run.smithi183.stdout: ], 2024-09-10T19:31:03.922 INFO:teuthology.orchestra.run.smithi183.stdout: "pg_upmap": [ 2024-09-10T19:31:03.922 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:31:03.922 INFO:teuthology.orchestra.run.smithi183.stdout: "pgid": "2.13", 2024-09-10T19:31:03.922 INFO:teuthology.orchestra.run.smithi183.stdout: "osds": [ 2024-09-10T19:31:03.922 INFO:teuthology.orchestra.run.smithi183.stdout: 2, 2024-09-10T19:31:03.922 INFO:teuthology.orchestra.run.smithi183.stdout: 0, 2024-09-10T19:31:03.922 INFO:teuthology.orchestra.run.smithi183.stdout: 5 2024-09-10T19:31:03.922 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:31:03.922 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:31:03.922 INFO:teuthology.orchestra.run.smithi183.stdout: ], 2024-09-10T19:31:03.922 INFO:teuthology.orchestra.run.smithi183.stdout: "pg_upmap_items": [ 2024-09-10T19:31:03.922 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:31:03.922 INFO:teuthology.orchestra.run.smithi183.stdout: "pgid": "2.8", 2024-09-10T19:31:03.923 INFO:teuthology.orchestra.run.smithi183.stdout: "mappings": [ 2024-09-10T19:31:03.923 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:31:03.923 INFO:teuthology.orchestra.run.smithi183.stdout: "from": 0, 2024-09-10T19:31:03.923 INFO:teuthology.orchestra.run.smithi183.stdout: "to": 5 2024-09-10T19:31:03.923 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:31:03.923 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:31:03.923 INFO:teuthology.orchestra.run.smithi183.stdout: "from": 1, 2024-09-10T19:31:03.923 INFO:teuthology.orchestra.run.smithi183.stdout: "to": 3 2024-09-10T19:31:03.923 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:31:03.923 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:31:03.923 INFO:teuthology.orchestra.run.smithi183.stdout: "from": 7, 2024-09-10T19:31:03.923 INFO:teuthology.orchestra.run.smithi183.stdout: "to": 4 2024-09-10T19:31:03.923 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:31:03.923 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:31:03.924 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:31:03.924 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:31:03.924 INFO:teuthology.orchestra.run.smithi183.stdout: "pgid": "2.14", 2024-09-10T19:31:03.924 INFO:teuthology.orchestra.run.smithi183.stdout: "mappings": [ 2024-09-10T19:31:03.924 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:31:03.924 INFO:teuthology.orchestra.run.smithi183.stdout: "from": 5, 2024-09-10T19:31:03.924 INFO:teuthology.orchestra.run.smithi183.stdout: "to": 6 2024-09-10T19:31:03.924 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:31:03.924 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:31:03.924 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:31:03.924 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:31:03.924 INFO:teuthology.orchestra.run.smithi183.stdout: "pgid": "2.18", 2024-09-10T19:31:03.925 INFO:teuthology.orchestra.run.smithi183.stdout: "mappings": [ 2024-09-10T19:31:03.925 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:31:03.925 INFO:teuthology.orchestra.run.smithi183.stdout: "from": 7, 2024-09-10T19:31:03.925 INFO:teuthology.orchestra.run.smithi183.stdout: "to": 0 2024-09-10T19:31:03.925 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:31:03.925 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:31:03.925 INFO:teuthology.orchestra.run.smithi183.stdout: "from": 5, 2024-09-10T19:31:03.925 INFO:teuthology.orchestra.run.smithi183.stdout: "to": 4 2024-09-10T19:31:03.925 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:31:03.925 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:31:03.925 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:31:03.925 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:31:03.926 INFO:teuthology.orchestra.run.smithi183.stdout: "pgid": "4.b", 2024-09-10T19:31:03.926 INFO:teuthology.orchestra.run.smithi183.stdout: "mappings": [ 2024-09-10T19:31:03.926 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:31:03.926 INFO:teuthology.orchestra.run.smithi183.stdout: "from": 1, 2024-09-10T19:31:03.926 INFO:teuthology.orchestra.run.smithi183.stdout: "to": 5 2024-09-10T19:31:03.926 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:31:03.926 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:31:03.926 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:31:03.926 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:31:03.926 INFO:teuthology.orchestra.run.smithi183.stdout: "pgid": "4.e", 2024-09-10T19:31:03.926 INFO:teuthology.orchestra.run.smithi183.stdout: "mappings": [ 2024-09-10T19:31:03.926 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:31:03.926 INFO:teuthology.orchestra.run.smithi183.stdout: "from": 1, 2024-09-10T19:31:03.926 INFO:teuthology.orchestra.run.smithi183.stdout: "to": 5 2024-09-10T19:31:03.927 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:31:03.927 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:31:03.927 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:31:03.927 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:31:03.927 INFO:teuthology.orchestra.run.smithi183.stdout: "pgid": "5.15", 2024-09-10T19:31:03.927 INFO:teuthology.orchestra.run.smithi183.stdout: "mappings": [ 2024-09-10T19:31:03.927 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:31:03.927 INFO:teuthology.orchestra.run.smithi183.stdout: "from": 1, 2024-09-10T19:31:03.927 INFO:teuthology.orchestra.run.smithi183.stdout: "to": 3 2024-09-10T19:31:03.927 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:31:03.927 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:31:03.927 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:31:03.928 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:31:03.928 INFO:teuthology.orchestra.run.smithi183.stdout: "pgid": "5.19", 2024-09-10T19:31:03.928 INFO:teuthology.orchestra.run.smithi183.stdout: "mappings": [ 2024-09-10T19:31:03.928 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:31:03.928 INFO:teuthology.orchestra.run.smithi183.stdout: "from": 1, 2024-09-10T19:31:03.928 INFO:teuthology.orchestra.run.smithi183.stdout: "to": 6 2024-09-10T19:31:03.928 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:31:03.928 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:31:03.928 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:31:03.928 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:31:03.928 INFO:teuthology.orchestra.run.smithi183.stdout: "pgid": "6.1b", 2024-09-10T19:31:03.928 INFO:teuthology.orchestra.run.smithi183.stdout: "mappings": [ 2024-09-10T19:31:03.928 INFO:teuthology.orchestra.run.smithi183.stdout: { 2024-09-10T19:31:03.929 INFO:teuthology.orchestra.run.smithi183.stdout: "from": 3, 2024-09-10T19:31:03.929 INFO:teuthology.orchestra.run.smithi183.stdout: "to": 0 2024-09-10T19:31:03.929 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:31:03.929 INFO:teuthology.orchestra.run.smithi183.stdout: ] 2024-09-10T19:31:03.929 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:31:03.929 INFO:teuthology.orchestra.run.smithi183.stdout: ], 2024-09-10T19:31:03.929 INFO:teuthology.orchestra.run.smithi183.stdout: "pg_temp": [], 2024-09-10T19:31:03.929 INFO:teuthology.orchestra.run.smithi183.stdout: "primary_temp": [], 2024-09-10T19:31:03.929 INFO:teuthology.orchestra.run.smithi183.stdout: "blocklist": { 2024-09-10T19:31:03.929 INFO:teuthology.orchestra.run.smithi183.stdout: "172.21.15.183:6801/3311783362": "2024-09-11T19:10:43.576943+0000", 2024-09-10T19:31:03.929 INFO:teuthology.orchestra.run.smithi183.stdout: "172.21.15.183:0/4088553922": "2024-09-11T19:10:43.576943+0000", 2024-09-10T19:31:03.929 INFO:teuthology.orchestra.run.smithi183.stdout: "172.21.15.183:0/3528769414": "2024-09-11T19:05:48.153879+0000", 2024-09-10T19:31:03.929 INFO:teuthology.orchestra.run.smithi183.stdout: "172.21.15.183:0/1766036658": "2024-09-11T19:05:48.153879+0000", 2024-09-10T19:31:03.929 INFO:teuthology.orchestra.run.smithi183.stdout: "172.21.15.183:0/3044170088": "2024-09-11T19:05:25.702354+0000", 2024-09-10T19:31:03.930 INFO:teuthology.orchestra.run.smithi183.stdout: "172.21.15.183:0/3037333778": "2024-09-11T19:05:25.702354+0000", 2024-09-10T19:31:03.930 INFO:teuthology.orchestra.run.smithi183.stdout: "172.21.15.183:6801/3747320570": "2024-09-11T19:05:25.702354+0000", 2024-09-10T19:31:03.930 INFO:teuthology.orchestra.run.smithi183.stdout: "172.21.15.183:0/3982386906": "2024-09-11T19:10:43.576943+0000", 2024-09-10T19:31:03.930 INFO:teuthology.orchestra.run.smithi183.stdout: "172.21.15.183:0/2436014455": "2024-09-11T19:05:25.702354+0000", 2024-09-10T19:31:03.930 INFO:teuthology.orchestra.run.smithi183.stdout: "172.21.15.183:6801/659745765": "2024-09-11T19:05:48.153879+0000", 2024-09-10T19:31:03.930 INFO:teuthology.orchestra.run.smithi183.stdout: "172.21.15.183:6800/3747320570": "2024-09-11T19:05:25.702354+0000", 2024-09-10T19:31:03.930 INFO:teuthology.orchestra.run.smithi183.stdout: "172.21.15.183:0/1652576282": "2024-09-11T19:10:43.576943+0000", 2024-09-10T19:31:03.930 INFO:teuthology.orchestra.run.smithi183.stdout: "172.21.15.183:6800/3311783362": "2024-09-11T19:10:43.576943+0000", 2024-09-10T19:31:03.930 INFO:teuthology.orchestra.run.smithi183.stdout: "172.21.15.183:0/1012899549": "2024-09-11T19:05:48.153879+0000", 2024-09-10T19:31:03.930 INFO:teuthology.orchestra.run.smithi183.stdout: "172.21.15.183:6800/659745765": "2024-09-11T19:05:48.153879+0000", 2024-09-10T19:31:03.930 INFO:teuthology.orchestra.run.smithi183.stdout: "172.21.15.183:0/412044400": "2024-09-11T19:10:43.576943+0000", 2024-09-10T19:31:03.930 INFO:teuthology.orchestra.run.smithi183.stdout: "172.21.15.183:0/3636171401": "2024-09-11T19:10:43.576943+0000" 2024-09-10T19:31:03.931 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:31:03.931 INFO:teuthology.orchestra.run.smithi183.stdout: "range_blocklist": {}, 2024-09-10T19:31:03.931 INFO:teuthology.orchestra.run.smithi183.stdout: "erasure_code_profiles": { 2024-09-10T19:31:03.931 INFO:teuthology.orchestra.run.smithi183.stdout: "default": { 2024-09-10T19:31:03.931 INFO:teuthology.orchestra.run.smithi183.stdout: "crush-failure-domain": "osd", 2024-09-10T19:31:03.931 INFO:teuthology.orchestra.run.smithi183.stdout: "k": "2", 2024-09-10T19:31:03.931 INFO:teuthology.orchestra.run.smithi183.stdout: "m": "1", 2024-09-10T19:31:03.931 INFO:teuthology.orchestra.run.smithi183.stdout: "plugin": "jerasure", 2024-09-10T19:31:03.931 INFO:teuthology.orchestra.run.smithi183.stdout: "technique": "reed_sol_van" 2024-09-10T19:31:03.931 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:31:03.931 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-09-10T19:31:03.932 INFO:teuthology.orchestra.run.smithi183.stdout: "removed_snaps_queue": [], 2024-09-10T19:31:03.932 INFO:teuthology.orchestra.run.smithi183.stdout: "new_removed_snaps": [], 2024-09-10T19:31:03.932 INFO:teuthology.orchestra.run.smithi183.stdout: "new_purged_snaps": [], 2024-09-10T19:31:03.932 INFO:teuthology.orchestra.run.smithi183.stdout: "crush_node_flags": {}, 2024-09-10T19:31:03.932 INFO:teuthology.orchestra.run.smithi183.stdout: "device_class_flags": {}, 2024-09-10T19:31:03.932 INFO:teuthology.orchestra.run.smithi183.stdout: "stretch_mode": { 2024-09-10T19:31:03.932 INFO:teuthology.orchestra.run.smithi183.stdout: "stretch_mode_enabled": false, 2024-09-10T19:31:03.932 INFO:teuthology.orchestra.run.smithi183.stdout: "stretch_bucket_count": 0, 2024-09-10T19:31:03.932 INFO:teuthology.orchestra.run.smithi183.stdout: "degraded_stretch_mode": 0, 2024-09-10T19:31:03.932 INFO:teuthology.orchestra.run.smithi183.stdout: "recovering_stretch_mode": 0, 2024-09-10T19:31:03.932 INFO:teuthology.orchestra.run.smithi183.stdout: "stretch_mode_bucket": 0 2024-09-10T19:31:03.932 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-09-10T19:31:03.932 INFO:teuthology.orchestra.run.smithi183.stdout:} 2024-09-10T19:31:03.999 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:31:04.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:31:04.035 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:31:04.035 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:31:04.136 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:31:04.171 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:31:04.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:31:04.172 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:31:04.273 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:31:04.306 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:31:04.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:31:04.307 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:31:04.408 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:31:04.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:31:04 smithi183 ceph-mon[29736]: from='client.? 172.21.15.183:0/2669074434' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-09-10T19:31:04.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:31:04 smithi183 ceph-mon[41562]: from='client.? 172.21.15.183:0/2669074434' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-09-10T19:31:04.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:31:04.433 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:31:04.433 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:31:04.465 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:31:04 smithi191 ceph-mon[32884]: from='client.? 172.21.15.183:0/2669074434' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-09-10T19:31:04.533 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:31:04.570 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:31:04.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:31:04.572 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:31:04.673 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:31:04.707 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:31:04.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:31:04.708 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:31:04.809 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:31:04.834 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:31:04.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:31:04.835 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:31:04.936 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:31:04.970 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:31:04.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:31:04.971 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:31:05.072 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:31:05.103 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:31:05.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:31:05.104 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:31:05.205 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:31:05.240 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:31:05.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:31:05.241 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:31:05.341 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:31:05.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:31:05.375 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:31:05.375 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:31:05.452 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:31:05 smithi183 ceph-mon[41562]: pgmap v962: 158 pgs: 158 active+clean; 583 MiB data, 7.1 GiB used, 619 GiB / 626 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.33k op/s 2024-09-10T19:31:05.454 INFO:tasks.thrashosds.thrasher:j is {'epoch': 422, 'fsid': '320f6df8-6fa7-11ef-bcea-c7b262605968', 'created': '2024-09-10T19:05:03.688531+0000', 'modified': '2024-09-10T19:30:56.151400+0000', 'last_up_change': '2024-09-10T19:24:22.870873+0000', 'last_in_change': '2024-09-10T19:27:51.975891+0000', 'flags': 'noscrub,nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5806080, 'flags_set': ['nodeep-scrub', 'noscrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 21, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 11, 'max_osd': 8, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'quincy', 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-10T19:08:00.456786+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 11, 'pg_placement_num': 11, 'pg_placement_num_target': 11, 'pg_num_target': 11, 'pg_num_pending': 11, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '271', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '242', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}, {'pool': 2, 'pool_name': '.rgw.root', 'create_time': '2024-09-10T19:10:05.993763+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '60', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}}, {'pool': 3, 'pool_name': 'datapool', 'create_time': '2024-09-10T19:10:06.224267+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 3, 'pg_placement_num': 3, 'pg_placement_num_target': 3, 'pg_num_target': 3, 'pg_num_pending': 3, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '63', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 3, 'snap_epoch': 63, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 4, 'pool_name': 'default.rgw.log', 'create_time': '2024-09-10T19:10:07.958284+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '62', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}}, {'pool': 5, 'pool_name': 'default.rgw.control', 'create_time': '2024-09-10T19:10:09.933596+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '64', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}}, {'pool': 6, 'pool_name': 'default.rgw.meta', 'create_time': '2024-09-10T19:10:11.993712+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '66', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4}, 'application_metadata': {'rgw': {}}}, {'pool': 11, 'pool_name': 'unique_pool_4', 'create_time': '2024-09-10T19:27:45.303826+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 16, 'pg_placement_num': 16, 'pg_placement_num_target': 16, 'pg_num_target': 16, 'pg_num_pending': 16, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '374', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}}], 'osds': [{'osd': 0, 'uuid': '9d323010-394a-49aa-ba9b-244d87faf36e', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 8, 'last_clean_end': 262, 'up_from': 301, 'up_thru': 374, 'down_at': 263, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.183:6810', 'nonce': 3346539414}, {'type': 'v1', 'addr': '172.21.15.183:6811', 'nonce': 3346539414}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.183:6812', 'nonce': 3346539414}, {'type': 'v1', 'addr': '172.21.15.183:6813', 'nonce': 3346539414}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.183:6816', 'nonce': 3346539414}, {'type': 'v1', 'addr': '172.21.15.183:6817', 'nonce': 3346539414}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.183:6814', 'nonce': 3346539414}, {'type': 'v1', 'addr': '172.21.15.183:6815', 'nonce': 3346539414}]}, 'public_addr': '172.21.15.183:6811/3346539414', 'cluster_addr': '172.21.15.183:6813/3346539414', 'heartbeat_back_addr': '172.21.15.183:6817/3346539414', 'heartbeat_front_addr': '172.21.15.183:6815/3346539414', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': '71a6c5dc-5109-4284-8d20-187d8e7afc9e', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 138, 'last_clean_end': 255, 'up_from': 276, 'up_thru': 374, 'down_at': 256, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.183:6802', 'nonce': 2764394151}, {'type': 'v1', 'addr': '172.21.15.183:6803', 'nonce': 2764394151}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.183:6804', 'nonce': 2764394151}, {'type': 'v1', 'addr': '172.21.15.183:6805', 'nonce': 2764394151}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.183:6808', 'nonce': 2764394151}, {'type': 'v1', 'addr': '172.21.15.183:6809', 'nonce': 2764394151}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.183:6806', 'nonce': 2764394151}, {'type': 'v1', 'addr': '172.21.15.183:6807', 'nonce': 2764394151}]}, 'public_addr': '172.21.15.183:6803/2764394151', 'cluster_addr': '172.21.15.183:6805/2764394151', 'heartbeat_back_addr': '172.21.15.183:6809/2764394151', 'heartbeat_front_addr': '172.21.15.183:6807/2764394151', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': 'fc310d5b-b89b-4655-9989-7e10ba356509', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 18, 'up_thru': 148, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.183:6818', 'nonce': 3730013786}, {'type': 'v1', 'addr': '172.21.15.183:6819', 'nonce': 3730013786}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.183:6820', 'nonce': 3730013786}, {'type': 'v1', 'addr': '172.21.15.183:6821', 'nonce': 3730013786}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.183:6824', 'nonce': 3730013786}, {'type': 'v1', 'addr': '172.21.15.183:6825', 'nonce': 3730013786}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.183:6822', 'nonce': 3730013786}, {'type': 'v1', 'addr': '172.21.15.183:6823', 'nonce': 3730013786}]}, 'public_addr': '172.21.15.183:6819/3730013786', 'cluster_addr': '172.21.15.183:6821/3730013786', 'heartbeat_back_addr': '172.21.15.183:6825/3730013786', 'heartbeat_front_addr': '172.21.15.183:6823/3730013786', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': '7fd07a95-ad22-43f5-a968-539552283e89', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 26, 'up_thru': 374, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.183:6826', 'nonce': 3522836984}, {'type': 'v1', 'addr': '172.21.15.183:6827', 'nonce': 3522836984}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.183:6828', 'nonce': 3522836984}, {'type': 'v1', 'addr': '172.21.15.183:6829', 'nonce': 3522836984}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.183:6832', 'nonce': 3522836984}, {'type': 'v1', 'addr': '172.21.15.183:6833', 'nonce': 3522836984}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.183:6830', 'nonce': 3522836984}, {'type': 'v1', 'addr': '172.21.15.183:6831', 'nonce': 3522836984}]}, 'public_addr': '172.21.15.183:6827/3522836984', 'cluster_addr': '172.21.15.183:6829/3522836984', 'heartbeat_back_addr': '172.21.15.183:6833/3522836984', 'heartbeat_front_addr': '172.21.15.183:6831/3522836984', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': '631083fb-e503-44da-9b6b-154e88539ee7', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 32, 'up_thru': 374, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.191:6800', 'nonce': 2674828539}, {'type': 'v1', 'addr': '172.21.15.191:6801', 'nonce': 2674828539}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.191:6802', 'nonce': 2674828539}, {'type': 'v1', 'addr': '172.21.15.191:6803', 'nonce': 2674828539}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.191:6806', 'nonce': 2674828539}, {'type': 'v1', 'addr': '172.21.15.191:6807', 'nonce': 2674828539}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.191:6804', 'nonce': 2674828539}, {'type': 'v1', 'addr': '172.21.15.191:6805', 'nonce': 2674828539}]}, 'public_addr': '172.21.15.191:6801/2674828539', 'cluster_addr': '172.21.15.191:6803/2674828539', 'heartbeat_back_addr': '172.21.15.191:6807/2674828539', 'heartbeat_front_addr': '172.21.15.191:6805/2674828539', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': '6b963695-3eca-4e6e-9207-05b8010ebff9', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 39, 'up_thru': 374, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.191:6808', 'nonce': 681733378}, {'type': 'v1', 'addr': '172.21.15.191:6809', 'nonce': 681733378}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.191:6810', 'nonce': 681733378}, {'type': 'v1', 'addr': '172.21.15.191:6811', 'nonce': 681733378}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.191:6814', 'nonce': 681733378}, {'type': 'v1', 'addr': '172.21.15.191:6815', 'nonce': 681733378}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.191:6812', 'nonce': 681733378}, {'type': 'v1', 'addr': '172.21.15.191:6813', 'nonce': 681733378}]}, 'public_addr': '172.21.15.191:6809/681733378', 'cluster_addr': '172.21.15.191:6811/681733378', 'heartbeat_back_addr': '172.21.15.191:6815/681733378', 'heartbeat_front_addr': '172.21.15.191:6813/681733378', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': '2dc9754f-6aef-4e19-8605-2f1591921811', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 47, 'up_thru': 391, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.191:6816', 'nonce': 2825813477}, {'type': 'v1', 'addr': '172.21.15.191:6817', 'nonce': 2825813477}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.191:6818', 'nonce': 2825813477}, {'type': 'v1', 'addr': '172.21.15.191:6819', 'nonce': 2825813477}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.191:6822', 'nonce': 2825813477}, {'type': 'v1', 'addr': '172.21.15.191:6823', 'nonce': 2825813477}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.191:6820', 'nonce': 2825813477}, {'type': 'v1', 'addr': '172.21.15.191:6821', 'nonce': 2825813477}]}, 'public_addr': '172.21.15.191:6817/2825813477', 'cluster_addr': '172.21.15.191:6819/2825813477', 'heartbeat_back_addr': '172.21.15.191:6823/2825813477', 'heartbeat_front_addr': '172.21.15.191:6821/2825813477', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': '1a9a299e-d5ba-4b9a-85d5-7f6a5bc754e1', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 54, 'up_thru': 374, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.191:6824', 'nonce': 1422610021}, {'type': 'v1', 'addr': '172.21.15.191:6825', 'nonce': 1422610021}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.191:6826', 'nonce': 1422610021}, {'type': 'v1', 'addr': '172.21.15.191:6827', 'nonce': 1422610021}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.191:6830', 'nonce': 1422610021}, {'type': 'v1', 'addr': '172.21.15.191:6831', 'nonce': 1422610021}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.191:6828', 'nonce': 1422610021}, {'type': 'v1', 'addr': '172.21.15.191:6829', 'nonce': 1422610021}]}, 'public_addr': '172.21.15.191:6825/1422610021', 'cluster_addr': '172.21.15.191:6827/1422610021', 'heartbeat_back_addr': '172.21.15.191:6831/1422610021', 'heartbeat_front_addr': '172.21.15.191:6829/1422610021', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138320759226367, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-10T19:30:55.624776+0000', 'dead_epoch': 262}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138320759226367, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-10T19:30:05.535777+0000', 'dead_epoch': 255}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138320759226367, 'old_weight': 65536, 'last_purged_snaps_scrub': '2024-09-10T19:30:53.240003+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138320759226367, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-10T19:30:16.451748+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138320759226367, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-10T19:30:45.814254+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138320759226367, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-10T19:30:01.762348+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138320759226367, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-10T19:29:53.533700+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138320759226367, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-10T19:30:11.918745+0000', 'dead_epoch': 0}], 'pg_upmap': [{'pgid': '2.13', 'osds': [2, 0, 5]}], 'pg_upmap_items': [{'pgid': '2.8', 'mappings': [{'from': 0, 'to': 5}, {'from': 1, 'to': 3}, {'from': 7, 'to': 4}]}, {'pgid': '2.14', 'mappings': [{'from': 5, 'to': 6}]}, {'pgid': '2.18', 'mappings': [{'from': 7, 'to': 0}, {'from': 5, 'to': 4}]}, {'pgid': '4.b', 'mappings': [{'from': 1, 'to': 5}]}, {'pgid': '4.e', 'mappings': [{'from': 1, 'to': 5}]}, {'pgid': '5.15', 'mappings': [{'from': 1, 'to': 3}]}, {'pgid': '5.19', 'mappings': [{'from': 1, 'to': 6}]}, {'pgid': '6.1b', 'mappings': [{'from': 3, 'to': 0}]}], 'pg_temp': [], 'primary_temp': [], 'blocklist': {'172.21.15.183:6801/3311783362': '2024-09-11T19:10:43.576943+0000', '172.21.15.183:0/4088553922': '2024-09-11T19:10:43.576943+0000', '172.21.15.183:0/3528769414': '2024-09-11T19:05:48.153879+0000', '172.21.15.183:0/1766036658': '2024-09-11T19:05:48.153879+0000', '172.21.15.183:0/3044170088': '2024-09-11T19:05:25.702354+0000', '172.21.15.183:0/3037333778': '2024-09-11T19:05:25.702354+0000', '172.21.15.183:6801/3747320570': '2024-09-11T19:05:25.702354+0000', '172.21.15.183:0/3982386906': '2024-09-11T19:10:43.576943+0000', '172.21.15.183:0/2436014455': '2024-09-11T19:05:25.702354+0000', '172.21.15.183:6801/659745765': '2024-09-11T19:05:48.153879+0000', '172.21.15.183:6800/3747320570': '2024-09-11T19:05:25.702354+0000', '172.21.15.183:0/1652576282': '2024-09-11T19:10:43.576943+0000', '172.21.15.183:6800/3311783362': '2024-09-11T19:10:43.576943+0000', '172.21.15.183:0/1012899549': '2024-09-11T19:05:48.153879+0000', '172.21.15.183:6800/659745765': '2024-09-11T19:05:48.153879+0000', '172.21.15.183:0/412044400': '2024-09-11T19:10:43.576943+0000', '172.21.15.183:0/3636171401': '2024-09-11T19:10:43.576943+0000'}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-09-10T19:31:05.454 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph pg dump --format=json 2024-09-10T19:31:05.455 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:31:05 smithi183 ceph-mon[29736]: pgmap v962: 158 pgs: 158 active+clean; 583 MiB data, 7.1 GiB used, 619 GiB / 626 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.33k op/s 2024-09-10T19:31:05.465 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:31:05 smithi191 ceph-mon[32884]: pgmap v962: 158 pgs: 158 active+clean; 583 MiB data, 7.1 GiB used, 619 GiB / 626 GiB avail; 1.1 KiB/s rd, 0 B/s wr, 1.33k op/s 2024-09-10T19:31:05.477 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:31:05.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:31:05.537 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:31:05.537 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:31:05.638 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:31:05.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:31:05.676 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:31:05.676 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:31:05.777 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:31:05.823 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:31:05.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:31:05.837 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:31:05.837 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:31:05.938 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:31:05.980 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:31:05.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:31:05.981 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:31:06.026 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:31:06.082 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:31:06.120 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:31:06.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:31:06.121 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:31:06.222 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:31:06.252 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:31:06.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:31:06.253 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:31:06.335 INFO:teuthology.orchestra.run.smithi191.stdout:ERROR: (22) Invalid argument 2024-09-10T19:31:06.335 INFO:teuthology.orchestra.run.smithi191.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-10T19:31:06.354 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:31:06.385 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:31:06.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:31:06.386 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:31:06.487 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:31:06.522 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:31:06.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:31:06.523 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:31:06.623 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:31:06.663 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:31:06.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:31:06.664 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:31:06.765 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:31:06.802 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:31:06.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:31:06.803 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:31:06.904 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:31:06.939 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:31:06.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:31:06.940 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:31:07.041 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:31:07.068 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:31:07.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:31:07.069 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:31:07.077 DEBUG:teuthology.orchestra.run.smithi191:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph daemon osd.6 dump_historic_ops 2024-09-10T19:31:07.171 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:31:07.208 INFO:teuthology.orchestra.run.smithi183.stderr:noscrub is unset 2024-09-10T19:31:07.208 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:31:07.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:31:07.209 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:31:07.283 INFO:tasks.radosbench.radosbench.0.smithi183.stdout:Removed 144887 objects 2024-09-10T19:31:07.284 INFO:tasks.radosbench.radosbench.0.smithi183.stdout:Clean up completed and total clean up time :103.202 2024-09-10T19:31:07.310 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:31:07.331 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:31:07.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:31:07.332 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:31:07.401 INFO:teuthology.orchestra.run.smithi191.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.b/config 2024-09-10T19:31:07.433 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:31:07.451 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:31:07 smithi191 ceph-mon[32884]: pgmap v963: 158 pgs: 158 active+clean; 221 MiB data, 6.3 GiB used, 620 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.28k op/s 2024-09-10T19:31:07.451 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:31:07 smithi191 ceph-mon[32884]: from='client.? 172.21.15.183:0/1484196480' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-10T19:31:07.451 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:31:07 smithi191 ceph-mon[32884]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-10T19:31:07.454 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:31:07.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:31:07.455 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:31:07.556 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:31:07.578 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:31:07.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:31:07.579 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:31:07.680 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:31:07.681 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:31:07 smithi183 ceph-mon[41562]: pgmap v963: 158 pgs: 158 active+clean; 221 MiB data, 6.3 GiB used, 620 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.28k op/s 2024-09-10T19:31:07.681 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:31:07 smithi183 ceph-mon[41562]: from='client.? 172.21.15.183:0/1484196480' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-10T19:31:07.681 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:31:07 smithi183 ceph-mon[41562]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-10T19:31:07.682 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:31:07 smithi183 ceph-mon[29736]: pgmap v963: 158 pgs: 158 active+clean; 221 MiB data, 6.3 GiB used, 620 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.28k op/s 2024-09-10T19:31:07.682 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:31:07 smithi183 ceph-mon[29736]: from='client.? 172.21.15.183:0/1484196480' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-10T19:31:07.682 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:31:07 smithi183 ceph-mon[29736]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-10T19:31:07.703 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:31:07.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:31:07.704 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:31:07.707 INFO:journalctl@ceph.mgr.x.smithi191.stdout:Sep 10 19:31:07 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-x[34967]: ::ffff:172.21.15.191 - - [10/Sep/2024:19:31:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T19:31:07.805 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:31:07.827 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:31:07.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:31:07.828 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:31:07.929 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:31:07.950 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:31:07.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:31:07.951 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:31:08.052 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:31:08.074 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:31:08.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:31:08.075 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:31:08.176 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:31:08.198 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:31:08.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:31:08.199 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:31:08.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:31:08 smithi191 ceph-mon[32884]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T19:31:08.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:31:08 smithi191 ceph-mon[32884]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-10T19:31:08.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:31:08 smithi191 ceph-mon[32884]: osdmap e423: 8 total, 8 up, 7 in 2024-09-10T19:31:08.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:31:08 smithi191 ceph-mon[32884]: 2.a scrub starts 2024-09-10T19:31:08.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:31:08 smithi191 ceph-mon[32884]: 2.a scrub ok 2024-09-10T19:31:08.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:31:08 smithi191 ceph-mon[32884]: 6.d scrub starts 2024-09-10T19:31:08.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:31:08 smithi191 ceph-mon[32884]: 6.d scrub ok 2024-09-10T19:31:08.301 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:31:08.322 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:31:08.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:31:08.323 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:31:08.423 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:31:08.445 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:31:08.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:31:08.446 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:31:08.543 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:31:08 smithi183 ceph-mon[29736]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T19:31:08.543 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:31:08 smithi183 ceph-mon[29736]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-10T19:31:08.543 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:31:08 smithi183 ceph-mon[29736]: osdmap e423: 8 total, 8 up, 7 in 2024-09-10T19:31:08.543 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:31:08 smithi183 ceph-mon[29736]: 2.a scrub starts 2024-09-10T19:31:08.543 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:31:08 smithi183 ceph-mon[29736]: 2.a scrub ok 2024-09-10T19:31:08.543 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:31:08 smithi183 ceph-mon[29736]: 6.d scrub starts 2024-09-10T19:31:08.543 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:31:08 smithi183 ceph-mon[29736]: 6.d scrub ok 2024-09-10T19:31:08.544 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:31:08 smithi183 ceph-mon[41562]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T19:31:08.544 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:31:08 smithi183 ceph-mon[41562]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-10T19:31:08.544 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:31:08 smithi183 ceph-mon[41562]: osdmap e423: 8 total, 8 up, 7 in 2024-09-10T19:31:08.544 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:31:08 smithi183 ceph-mon[41562]: 2.a scrub starts 2024-09-10T19:31:08.544 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:31:08 smithi183 ceph-mon[41562]: 2.a scrub ok 2024-09-10T19:31:08.544 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:31:08 smithi183 ceph-mon[41562]: 6.d scrub starts 2024-09-10T19:31:08.544 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:31:08 smithi183 ceph-mon[41562]: 6.d scrub ok 2024-09-10T19:31:08.546 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:31:08.571 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:31:08.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:31:08.572 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:31:08.673 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:31:08.695 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:31:08.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:31:08.696 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:31:08.797 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:31:08.819 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:31:08.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:31:08.820 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:31:08.921 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:31:08.943 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:31:08.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:31:08.944 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:31:09.045 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:31:09.067 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:31:09.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:31:09.068 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:31:09.156 INFO:teuthology.orchestra.run.smithi191.stdout:ERROR: (22) Invalid argument 2024-09-10T19:31:09.156 INFO:teuthology.orchestra.run.smithi191.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-10T19:31:09.172 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:31:09.191 INFO:tasks.cephadm.ceph_manager.ceph:removing pool_name unique_pool_4 2024-09-10T19:31:09.192 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph osd pool rm unique_pool_4 unique_pool_4 --yes-i-really-really-mean-it 2024-09-10T19:31:09.194 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:31:09.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:31:09.195 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:31:09.296 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:31:09.317 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:31:09.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:31:09.318 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:31:09.419 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:31:09.431 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:31:09 smithi183 ceph-mon[41562]: pgmap v965: 158 pgs: 158 active+clean; 138 MiB data, 5.9 GiB used, 620 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.31k op/s 2024-09-10T19:31:09.431 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:31:09 smithi183 ceph-mon[41562]: 1.2 scrub starts 2024-09-10T19:31:09.431 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:31:09 smithi183 ceph-mon[41562]: 1.2 scrub ok 2024-09-10T19:31:09.431 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:31:09 smithi183 ceph-mon[41562]: 1.4 scrub starts 2024-09-10T19:31:09.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:31:09 smithi183 ceph-mon[41562]: 1.4 scrub ok 2024-09-10T19:31:09.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:31:09 smithi183 ceph-mon[41562]: 11.4 scrub starts 2024-09-10T19:31:09.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:31:09 smithi183 ceph-mon[29736]: pgmap v965: 158 pgs: 158 active+clean; 138 MiB data, 5.9 GiB used, 620 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.31k op/s 2024-09-10T19:31:09.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:31:09 smithi183 ceph-mon[29736]: 1.2 scrub starts 2024-09-10T19:31:09.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:31:09 smithi183 ceph-mon[29736]: 1.2 scrub ok 2024-09-10T19:31:09.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:31:09 smithi183 ceph-mon[29736]: 1.4 scrub starts 2024-09-10T19:31:09.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:31:09 smithi183 ceph-mon[29736]: 1.4 scrub ok 2024-09-10T19:31:09.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:31:09 smithi183 ceph-mon[29736]: 11.4 scrub starts 2024-09-10T19:31:09.442 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:31:09.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:31:09.443 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:31:09.465 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:31:09 smithi191 ceph-mon[32884]: pgmap v965: 158 pgs: 158 active+clean; 138 MiB data, 5.9 GiB used, 620 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.31k op/s 2024-09-10T19:31:09.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:31:09 smithi191 ceph-mon[32884]: 1.2 scrub starts 2024-09-10T19:31:09.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:31:09 smithi191 ceph-mon[32884]: 1.2 scrub ok 2024-09-10T19:31:09.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:31:09 smithi191 ceph-mon[32884]: 1.4 scrub starts 2024-09-10T19:31:09.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:31:09 smithi191 ceph-mon[32884]: 1.4 scrub ok 2024-09-10T19:31:09.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:31:09 smithi191 ceph-mon[32884]: 11.4 scrub starts 2024-09-10T19:31:09.544 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:31:09.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:31:09.567 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:31:09.567 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:31:09.659 INFO:teuthology.orchestra.run.smithi183.stdout:osd.0: {} 2024-09-10T19:31:09.659 INFO:teuthology.orchestra.run.smithi183.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-10T19:31:09.668 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:31:09.669 INFO:teuthology.orchestra.run.smithi183.stdout:osd.1: {} 2024-09-10T19:31:09.670 INFO:teuthology.orchestra.run.smithi183.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-10T19:31:09.686 INFO:teuthology.orchestra.run.smithi183.stdout:osd.2: {} 2024-09-10T19:31:09.687 INFO:teuthology.orchestra.run.smithi183.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-10T19:31:09.691 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:31:09.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:31:09.692 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:31:09.702 INFO:teuthology.orchestra.run.smithi183.stdout:osd.3: {} 2024-09-10T19:31:09.702 INFO:teuthology.orchestra.run.smithi183.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-10T19:31:09.717 INFO:teuthology.orchestra.run.smithi183.stdout:osd.4: {} 2024-09-10T19:31:09.717 INFO:teuthology.orchestra.run.smithi183.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-10T19:31:09.732 INFO:teuthology.orchestra.run.smithi183.stdout:osd.5: {} 2024-09-10T19:31:09.733 INFO:teuthology.orchestra.run.smithi183.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-10T19:31:09.748 INFO:teuthology.orchestra.run.smithi183.stdout:osd.6: {} 2024-09-10T19:31:09.748 INFO:teuthology.orchestra.run.smithi183.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-10T19:31:09.764 INFO:teuthology.orchestra.run.smithi183.stdout:osd.7: {} 2024-09-10T19:31:09.764 INFO:teuthology.orchestra.run.smithi183.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-10T19:31:09.792 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:31:09.815 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:31:09.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:31:09.816 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:31:09.849 DEBUG:teuthology.orchestra.run.smithi191:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph daemon osd.7 dump_ops_in_flight 2024-09-10T19:31:09.917 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:31:09.953 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:31:09.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:31:09.954 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:31:10.055 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:31:10.076 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:31:10.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:31:10.077 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:31:10.140 INFO:teuthology.orchestra.run.smithi191.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.b/config 2024-09-10T19:31:10.178 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:31:10.199 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:31:10.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:31:10.200 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:31:10.300 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-10T19:31:10.300 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:31:10.322 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:31:10.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:31:10.323 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:31:10.423 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:31:10.446 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:31:10.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:31:10.447 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:31:10.465 INFO:journalctl@ceph.iscsi.iscsi.a.smithi191.stdout:Sep 10 19:31:10 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-iscsi-iscsi-a[61816]: debug there is no tcmu-runner data available 2024-09-10T19:31:10.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:31:10 smithi191 ceph-mon[32884]: 11.4 scrub ok 2024-09-10T19:31:10.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:31:10 smithi191 ceph-mon[32884]: from='client.? 172.21.15.183:0/302133418' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T19:31:10.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:31:10 smithi191 ceph-mon[32884]: 2.12 scrub starts 2024-09-10T19:31:10.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:31:10 smithi191 ceph-mon[32884]: 2.12 scrub ok 2024-09-10T19:31:10.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:31:10 smithi191 ceph-mon[32884]: 6.2 scrub starts 2024-09-10T19:31:10.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:31:10 smithi191 ceph-mon[32884]: 6.2 scrub ok 2024-09-10T19:31:10.547 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:31:10.569 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:31:10.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:31:10.571 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:31:10.573 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:31:10 smithi183 ceph-mon[29736]: 11.4 scrub ok 2024-09-10T19:31:10.573 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:31:10 smithi183 ceph-mon[29736]: from='client.? 172.21.15.183:0/302133418' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T19:31:10.573 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:31:10 smithi183 ceph-mon[29736]: 2.12 scrub starts 2024-09-10T19:31:10.574 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:31:10 smithi183 ceph-mon[29736]: 2.12 scrub ok 2024-09-10T19:31:10.574 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:31:10 smithi183 ceph-mon[29736]: 6.2 scrub starts 2024-09-10T19:31:10.574 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:31:10 smithi183 ceph-mon[29736]: 6.2 scrub ok 2024-09-10T19:31:10.574 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:31:10 smithi183 ceph-mon[41562]: 11.4 scrub ok 2024-09-10T19:31:10.574 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:31:10 smithi183 ceph-mon[41562]: from='client.? 172.21.15.183:0/302133418' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T19:31:10.574 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:31:10 smithi183 ceph-mon[41562]: 2.12 scrub starts 2024-09-10T19:31:10.574 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:31:10 smithi183 ceph-mon[41562]: 2.12 scrub ok 2024-09-10T19:31:10.574 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:31:10 smithi183 ceph-mon[41562]: 6.2 scrub starts 2024-09-10T19:31:10.574 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:31:10 smithi183 ceph-mon[41562]: 6.2 scrub ok 2024-09-10T19:31:10.671 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:31:10.694 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:31:10.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:31:10.696 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:31:10.796 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:31:10.817 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:31:10.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:31:10.818 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:31:10.919 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:31:10.930 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:31:10.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:31:10.948 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:31:10.948 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:31:11.049 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:31:11.071 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-09-10T19:31:11.073 INFO:teuthology.orchestra.run.smithi183.stderr:dumped all 2024-09-10T19:31:11.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:31:11.076 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:31:11.077 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:31:11.178 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:31:11.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:31:11.200 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:31:11.200 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:31:11.301 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:31:11.358 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:31:11.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:31:11.359 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:31:11.460 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:31:11.465 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:31:11 smithi191 ceph-mon[32884]: 4.1e scrub starts 2024-09-10T19:31:11.465 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:31:11 smithi191 ceph-mon[32884]: 4.1e scrub ok 2024-09-10T19:31:11.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:31:11 smithi191 ceph-mon[32884]: pgmap v966: 158 pgs: 158 active+clean; 113 MiB data, 5.8 GiB used, 620 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.27k op/s 2024-09-10T19:31:11.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:31:11 smithi191 ceph-mon[32884]: from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:31:11.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:31:11 smithi191 ceph-mon[32884]: 5.14 scrub starts 2024-09-10T19:31:11.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:31:11 smithi191 ceph-mon[32884]: 5.14 scrub ok 2024-09-10T19:31:11.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:31:11 smithi191 ceph-mon[32884]: 4.15 scrub starts 2024-09-10T19:31:11.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:31:11 smithi191 ceph-mon[32884]: 4.15 scrub ok 2024-09-10T19:31:11.483 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:31:11.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:31:11.484 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:31:11.498 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:31:11 smithi183 ceph-mon[41562]: 4.1e scrub starts 2024-09-10T19:31:11.498 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:31:11 smithi183 ceph-mon[41562]: 4.1e scrub ok 2024-09-10T19:31:11.498 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:31:11 smithi183 ceph-mon[41562]: pgmap v966: 158 pgs: 158 active+clean; 113 MiB data, 5.8 GiB used, 620 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.27k op/s 2024-09-10T19:31:11.498 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:31:11 smithi183 ceph-mon[41562]: from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:31:11.498 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:31:11 smithi183 ceph-mon[41562]: 5.14 scrub starts 2024-09-10T19:31:11.498 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:31:11 smithi183 ceph-mon[41562]: 5.14 scrub ok 2024-09-10T19:31:11.498 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:31:11 smithi183 ceph-mon[41562]: 4.15 scrub starts 2024-09-10T19:31:11.498 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:31:11 smithi183 ceph-mon[41562]: 4.15 scrub ok 2024-09-10T19:31:11.498 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:31:11 smithi183 ceph-mon[29736]: 4.1e scrub starts 2024-09-10T19:31:11.498 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:31:11 smithi183 ceph-mon[29736]: 4.1e scrub ok 2024-09-10T19:31:11.499 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:31:11 smithi183 ceph-mon[29736]: pgmap v966: 158 pgs: 158 active+clean; 113 MiB data, 5.8 GiB used, 620 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.27k op/s 2024-09-10T19:31:11.499 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:31:11 smithi183 ceph-mon[29736]: from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:31:11.499 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:31:11 smithi183 ceph-mon[29736]: 5.14 scrub starts 2024-09-10T19:31:11.499 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:31:11 smithi183 ceph-mon[29736]: 5.14 scrub ok 2024-09-10T19:31:11.499 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:31:11 smithi183 ceph-mon[29736]: 4.15 scrub starts 2024-09-10T19:31:11.499 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:31:11 smithi183 ceph-mon[29736]: 4.15 scrub ok 2024-09-10T19:31:11.499 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Sep 10 19:31:11 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-y[29989]: ::ffff:172.21.15.191 - - [10/Sep/2024:19:31:11] "GET /metrics HTTP/1.1" 200 224921 "" "Prometheus/2.43.0" 2024-09-10T19:31:11.585 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:31:11.610 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:31:11.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:31:11.611 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:31:11.711 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:31:11.713 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:31:11.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:31:11.739 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:31:11.739 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:31:11.840 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:31:11.861 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:31:11.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:31:11.863 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:31:11.963 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:31:11.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:31:11.990 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:31:11.990 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:31:12.091 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:31:12.114 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:31:12.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:31:12.115 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:31:12.166 INFO:teuthology.orchestra.run.smithi183.stdout:{"pg_ready":true,"pg_map":{"version":966,"stamp":"2024-09-10T19:31:09.993281+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":118298868,"num_objects":1993,"num_object_clones":0,"num_object_copies":5979,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1993,"num_whiteouts":0,"num_read":1993,"num_read_kb":1745,"num_write":288981,"num_write_kb":9272812,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":458,"num_bytes_recovered":536482,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72255,"ondisk_log_size":72255,"up":474,"acting":474,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":474,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":656216064,"kb_used":6048192,"kb_used_data":320576,"kb_used_omap":19,"kb_used_meta":5727084,"kb_avail":650167872,"statfs":{"total":671965249536,"available":665771900928,"internally_reserved":0,"allocated":328269824,"data_stored":326092988,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19457,"internal_metadata":5864535039},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-833945600,"num_objects":-12725,"num_object_clones":0,"num_object_copies":-38175,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-12725,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":12725,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.003018"},"pg_stats":[{"pgid":"6.1b","version":"0'0","reported_seq":167,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-09-10T19:30:55.156186+0000","last_change":"2024-09-10T19:29:35.538355+0000","last_active":"2024-09-10T19:30:55.156186+0000","last_peered":"2024-09-10T19:30:55.156186+0000","last_clean":"2024-09-10T19:30:55.156186+0000","last_became_active":"2024-09-10T19:27:52.994757+0000","last_became_peered":"2024-09-10T19:27:52.994757+0000","last_unstale":"2024-09-10T19:30:55.156186+0000","last_undegraded":"2024-09-10T19:30:55.156186+0000","last_fullsized":"2024-09-10T19:30:55.156186+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:29:35.538284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:28:18.455084+0000","last_clean_scrub_stamp":"2024-09-10T19:29:35.538284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:30:57.495675+0000","scrub_duration":0.0025697599999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6],"acting":[0,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":415,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-09-10T19:30:55.156377+0000","last_change":"2024-09-10T19:29:32.579218+0000","last_active":"2024-09-10T19:30:55.156377+0000","last_peered":"2024-09-10T19:30:55.156377+0000","last_clean":"2024-09-10T19:30:55.156377+0000","last_became_active":"2024-09-10T19:24:23.892621+0000","last_became_peered":"2024-09-10T19:24:23.892621+0000","last_unstale":"2024-09-10T19:30:55.156377+0000","last_undegraded":"2024-09-10T19:30:55.156377+0000","last_fullsized":"2024-09-10T19:30:55.156377+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:29:32.579092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:16:38.194472+0000","last_clean_scrub_stamp":"2024-09-10T19:29:32.579092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.010689732,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.19","version":"414'39","reported_seq":534,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-09-10T19:30:55.154206+0000","last_change":"2024-09-10T19:30:09.389815+0000","last_active":"2024-09-10T19:30:55.154206+0000","last_peered":"2024-09-10T19:30:55.154206+0000","last_clean":"2024-09-10T19:30:55.154206+0000","last_became_active":"2024-09-10T19:24:23.906592+0000","last_became_peered":"2024-09-10T19:24:23.906592+0000","last_unstale":"2024-09-10T19:30:55.154206+0000","last_undegraded":"2024-09-10T19:30:55.154206+0000","last_fullsized":"2024-09-10T19:30:55.154206+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"237'27","last_scrub_stamp":"2024-09-10T19:30:09.389670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:30:09.389670+0000","objects_scrubbed":7,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:31:38.619896+0000","scrub_duration":0.003161433,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":62,"num_read_kb":53,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":510,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-09-10T19:30:55.160390+0000","last_change":"2024-09-10T19:30:41.457105+0000","last_active":"2024-09-10T19:30:55.160390+0000","last_peered":"2024-09-10T19:30:55.160390+0000","last_clean":"2024-09-10T19:30:55.160390+0000","last_became_active":"2024-09-10T19:27:53.006175+0000","last_became_peered":"2024-09-10T19:27:53.006175+0000","last_unstale":"2024-09-10T19:30:55.160390+0000","last_undegraded":"2024-09-10T19:30:55.160390+0000","last_fullsized":"2024-09-10T19:30:55.160390+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:30:41.457019+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:19:28.498560+0000","last_clean_scrub_stamp":"2024-09-10T19:30:41.457019+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:32:07.134757+0000","scrub_duration":0.0038471270000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"384'2","reported_seq":478,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-09-10T19:30:55.154778+0000","last_change":"2024-09-10T19:30:34.918319+0000","last_active":"2024-09-10T19:30:55.154778+0000","last_peered":"2024-09-10T19:30:55.154778+0000","last_clean":"2024-09-10T19:30:55.154778+0000","last_became_active":"2024-09-10T19:24:23.908614+0000","last_became_peered":"2024-09-10T19:24:23.908614+0000","last_unstale":"2024-09-10T19:30:55.154778+0000","last_undegraded":"2024-09-10T19:30:55.154778+0000","last_fullsized":"2024-09-10T19:30:55.154778+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"384'2","last_scrub_stamp":"2024-09-10T19:30:34.918244+0000","last_deep_scrub":"384'2","last_deep_scrub_stamp":"2024-09-10T19:28:23.240824+0000","last_clean_scrub_stamp":"2024-09-10T19:30:34.918244+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:31:35.083745+0000","scrub_duration":0.0051432630000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":874,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"414'25","reported_seq":577,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-09-10T19:30:55.160467+0000","last_change":"2024-09-10T19:30:24.549021+0000","last_active":"2024-09-10T19:30:55.160467+0000","last_peered":"2024-09-10T19:30:55.160467+0000","last_clean":"2024-09-10T19:30:55.160467+0000","last_became_active":"2024-09-10T19:27:53.013509+0000","last_became_peered":"2024-09-10T19:27:53.013509+0000","last_unstale":"2024-09-10T19:30:55.160467+0000","last_undegraded":"2024-09-10T19:30:55.160467+0000","last_fullsized":"2024-09-10T19:30:55.160467+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"414'25","last_scrub_stamp":"2024-09-10T19:30:24.548931+0000","last_deep_scrub":"384'19","last_deep_scrub_stamp":"2024-09-10T19:28:21.799921+0000","last_clean_scrub_stamp":"2024-09-10T19:30:24.548931+0000","objects_scrubbed":4,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:31:44.970028+0000","scrub_duration":0.0074165239999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":51,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-09-10T19:30:56.163878+0000","last_change":"2024-09-10T19:30:06.545197+0000","last_active":"2024-09-10T19:30:56.163878+0000","last_peered":"2024-09-10T19:30:56.163878+0000","last_clean":"2024-09-10T19:30:56.163878+0000","last_became_active":"2024-09-10T19:28:47.151481+0000","last_became_peered":"2024-09-10T19:28:47.151481+0000","last_unstale":"2024-09-10T19:30:56.163878+0000","last_undegraded":"2024-09-10T19:30:56.163878+0000","last_fullsized":"2024-09-10T19:30:56.163878+0000","mapping_epoch":391,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":392,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:30:06.545123+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:30:06.545123+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:31:19.928036+0000","scrub_duration":0.0025621239999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,7],"acting":[6,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":478,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-09-10T19:30:55.160445+0000","last_change":"2024-09-10T19:30:16.469108+0000","last_active":"2024-09-10T19:30:55.160445+0000","last_peered":"2024-09-10T19:30:55.160445+0000","last_clean":"2024-09-10T19:30:55.160445+0000","last_became_active":"2024-09-10T19:24:01.971921+0000","last_became_peered":"2024-09-10T19:24:01.971921+0000","last_unstale":"2024-09-10T19:30:55.160445+0000","last_undegraded":"2024-09-10T19:30:55.160445+0000","last_fullsized":"2024-09-10T19:30:55.160445+0000","mapping_epoch":292,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":293,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:30:16.469056+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:30:16.469056+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:31:23.825602+0000","scrub_duration":0.001491469,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":479,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-09-10T19:30:55.179593+0000","last_change":"2024-09-10T19:29:11.890872+0000","last_active":"2024-09-10T19:30:55.179593+0000","last_peered":"2024-09-10T19:30:55.179593+0000","last_clean":"2024-09-10T19:30:55.179593+0000","last_became_active":"2024-09-10T19:27:53.011202+0000","last_became_peered":"2024-09-10T19:27:53.011202+0000","last_unstale":"2024-09-10T19:30:55.179593+0000","last_undegraded":"2024-09-10T19:30:55.179593+0000","last_fullsized":"2024-09-10T19:30:55.179593+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:29:11.890794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:29:11.890794+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.002093588,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1b","version":"414'9","reported_seq":523,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-09-10T19:30:55.160547+0000","last_change":"2024-09-10T19:30:40.489631+0000","last_active":"2024-09-10T19:30:55.160547+0000","last_peered":"2024-09-10T19:30:55.160547+0000","last_clean":"2024-09-10T19:30:55.160547+0000","last_became_active":"2024-09-10T19:24:01.974813+0000","last_became_peered":"2024-09-10T19:24:01.974813+0000","last_unstale":"2024-09-10T19:30:55.160547+0000","last_undegraded":"2024-09-10T19:30:55.160547+0000","last_fullsized":"2024-09-10T19:30:55.160547+0000","mapping_epoch":292,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":293,"parent":"0.0","parent_split_bits":0,"last_scrub":"414'9","last_scrub_stamp":"2024-09-10T19:30:40.489534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:30:40.489534+0000","objects_scrubbed":2,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:31:41.703879+0000","scrub_duration":0.0037854310000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":17,"num_read_kb":13,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":472,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-09-10T19:30:55.161741+0000","last_change":"2024-09-10T19:30:19.910559+0000","last_active":"2024-09-10T19:30:55.161741+0000","last_peered":"2024-09-10T19:30:55.161741+0000","last_clean":"2024-09-10T19:30:55.161741+0000","last_became_active":"2024-09-10T19:24:01.976587+0000","last_became_peered":"2024-09-10T19:24:01.976587+0000","last_unstale":"2024-09-10T19:30:55.161741+0000","last_undegraded":"2024-09-10T19:30:55.161741+0000","last_fullsized":"2024-09-10T19:30:55.161741+0000","mapping_epoch":292,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":293,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:30:19.910433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:30:19.910433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:31:47.412483+0000","scrub_duration":0.0056012320000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":473,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-09-10T19:30:55.173113+0000","last_change":"2024-09-10T19:30:08.321635+0000","last_active":"2024-09-10T19:30:55.173113+0000","last_peered":"2024-09-10T19:30:55.173113+0000","last_clean":"2024-09-10T19:30:55.173113+0000","last_became_active":"2024-09-10T19:24:01.970118+0000","last_became_peered":"2024-09-10T19:24:01.970118+0000","last_unstale":"2024-09-10T19:30:55.173113+0000","last_undegraded":"2024-09-10T19:30:55.173113+0000","last_fullsized":"2024-09-10T19:30:55.173113+0000","mapping_epoch":292,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":293,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:30:08.321561+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:30:08.321561+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:31:28.385750+0000","scrub_duration":0.0054793990000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":482,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-09-10T19:30:55.177724+0000","last_change":"2024-09-10T19:29:31.166120+0000","last_active":"2024-09-10T19:30:55.177724+0000","last_peered":"2024-09-10T19:30:55.177724+0000","last_clean":"2024-09-10T19:30:55.177724+0000","last_became_active":"2024-09-10T19:27:53.008555+0000","last_became_peered":"2024-09-10T19:27:53.008555+0000","last_unstale":"2024-09-10T19:30:55.177724+0000","last_undegraded":"2024-09-10T19:30:55.177724+0000","last_fullsized":"2024-09-10T19:30:55.177724+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:29:31.166021+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:29:31.166021+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.012493738000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1a","version":"414'21","reported_seq":493,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-09-10T19:30:55.160721+0000","last_change":"2024-09-10T19:30:17.531183+0000","last_active":"2024-09-10T19:30:55.160721+0000","last_peered":"2024-09-10T19:30:55.160721+0000","last_clean":"2024-09-10T19:30:55.160721+0000","last_became_active":"2024-09-10T19:24:23.891213+0000","last_became_peered":"2024-09-10T19:24:23.891213+0000","last_unstale":"2024-09-10T19:30:55.160721+0000","last_undegraded":"2024-09-10T19:30:55.160721+0000","last_fullsized":"2024-09-10T19:30:55.160721+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"414'21","last_scrub_stamp":"2024-09-10T19:30:17.531099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:30:17.531099+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:31:44.519993+0000","scrub_duration":0.097305297999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":465,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-09-10T19:30:55.173262+0000","last_change":"2024-09-10T19:30:29.198545+0000","last_active":"2024-09-10T19:30:55.173262+0000","last_peered":"2024-09-10T19:30:55.173262+0000","last_clean":"2024-09-10T19:30:55.173262+0000","last_became_active":"2024-09-10T19:24:23.907653+0000","last_became_peered":"2024-09-10T19:24:23.907653+0000","last_unstale":"2024-09-10T19:30:55.173262+0000","last_undegraded":"2024-09-10T19:30:55.173262+0000","last_fullsized":"2024-09-10T19:30:55.173262+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:30:29.198451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:30:29.198451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:31:31.812451+0000","scrub_duration":0.0023167890000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":438,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-09-10T19:30:55.153718+0000","last_change":"2024-09-10T19:30:41.905595+0000","last_active":"2024-09-10T19:30:55.153718+0000","last_peered":"2024-09-10T19:30:55.153718+0000","last_clean":"2024-09-10T19:30:55.153718+0000","last_became_active":"2024-09-10T19:24:24.907431+0000","last_became_peered":"2024-09-10T19:24:24.907431+0000","last_unstale":"2024-09-10T19:30:55.153718+0000","last_undegraded":"2024-09-10T19:30:55.153718+0000","last_fullsized":"2024-09-10T19:30:55.153718+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:30:41.905516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:30:41.905516+0000","last_clean_scrub_stamp":"2024-09-10T19:30:41.905516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:32:11.300289+0000","scrub_duration":0.002437481,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.2","version":"422'17764","reported_seq":17997,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-09-10T19:31:05.352127+0000","last_change":"2024-09-10T19:30:18.096594+0000","last_active":"2024-09-10T19:31:05.352127+0000","last_peered":"2024-09-10T19:31:05.352127+0000","last_clean":"2024-09-10T19:31:05.352127+0000","last_became_active":"2024-09-10T19:27:46.703515+0000","last_became_peered":"2024-09-10T19:27:46.703515+0000","last_unstale":"2024-09-10T19:31:05.352127+0000","last_undegraded":"2024-09-10T19:31:05.352127+0000","last_fullsized":"2024-09-10T19:31:05.352127+0000","mapping_epoch":371,"log_start":"414'13600","ondisk_log_start":"414'13600","created":371,"last_epoch_clean":372,"parent":"0.0","parent_split_bits":0,"last_scrub":"415'13895","last_scrub_stamp":"2024-09-10T19:30:18.096518+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:27:45.679464+0000","last_clean_scrub_stamp":"2024-09-10T19:30:18.096518+0000","objects_scrubbed":4109,"log_size":4164,"log_dups_size":0,"ondisk_log_size":4164,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:31:35.525537+0000","scrub_duration":1.59077157,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11403264,"num_objects":174,"num_object_clones":0,"num_object_copies":522,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":174,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17764,"num_write_kb":574016,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,7],"acting":[3,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.8","version":"74'79","reported_seq":270,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-09-10T19:30:55.176830+0000","last_change":"2024-09-10T19:30:28.196155+0000","last_active":"2024-09-10T19:30:55.176830+0000","last_peered":"2024-09-10T19:30:55.176830+0000","last_clean":"2024-09-10T19:30:55.176830+0000","last_became_active":"2024-09-10T19:27:53.004335+0000","last_became_peered":"2024-09-10T19:27:53.004335+0000","last_unstale":"2024-09-10T19:30:55.176830+0000","last_undegraded":"2024-09-10T19:30:55.176830+0000","last_fullsized":"2024-09-10T19:30:55.176830+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":4,"last_scrub":"74'79","last_scrub_stamp":"2024-09-10T19:30:28.196058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:08:00.699315+0000","last_clean_scrub_stamp":"2024-09-10T19:30:28.196058+0000","objects_scrubbed":1,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:31:45.244731+0000","scrub_duration":0.012192326,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":525376,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.d","version":"414'45","reported_seq":673,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-09-10T19:30:55.161186+0000","last_change":"2024-09-10T19:30:26.584510+0000","last_active":"2024-09-10T19:30:55.161186+0000","last_peered":"2024-09-10T19:30:55.161186+0000","last_clean":"2024-09-10T19:30:55.161186+0000","last_became_active":"2024-09-10T19:27:53.012536+0000","last_became_peered":"2024-09-10T19:27:53.012536+0000","last_unstale":"2024-09-10T19:30:55.161186+0000","last_undegraded":"2024-09-10T19:30:55.161186+0000","last_fullsized":"2024-09-10T19:30:55.161186+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"414'45","last_scrub_stamp":"2024-09-10T19:30:26.584371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:30:26.584371+0000","objects_scrubbed":8,"log_size":45,"log_dups_size":0,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:31:48.878559+0000","scrub_duration":0.0072232800000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":71,"num_read_kb":61,"num_write":46,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":40,"num_bytes_recovered":550,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,6],"acting":[4,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":462,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-09-10T19:30:55.175429+0000","last_change":"2024-09-10T19:30:27.008379+0000","last_active":"2024-09-10T19:30:55.175429+0000","last_peered":"2024-09-10T19:30:55.175429+0000","last_clean":"2024-09-10T19:30:55.175429+0000","last_became_active":"2024-09-10T19:10:07.691119+0000","last_became_peered":"2024-09-10T19:10:07.691119+0000","last_unstale":"2024-09-10T19:30:55.175429+0000","last_undegraded":"2024-09-10T19:30:55.175429+0000","last_fullsized":"2024-09-10T19:30:55.175429+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:30:27.008299+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:16:39.716674+0000","last_clean_scrub_stamp":"2024-09-10T19:30:27.008299+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:31:49.390872+0000","scrub_duration":0.0014108809999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":184,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-09-10T19:30:56.161407+0000","last_change":"2024-09-10T19:29:30.513955+0000","last_active":"2024-09-10T19:30:56.161407+0000","last_peered":"2024-09-10T19:30:56.161407+0000","last_clean":"2024-09-10T19:30:56.161407+0000","last_became_active":"2024-09-10T19:24:24.886479+0000","last_became_peered":"2024-09-10T19:24:24.886479+0000","last_unstale":"2024-09-10T19:30:56.161407+0000","last_undegraded":"2024-09-10T19:30:56.161407+0000","last_fullsized":"2024-09-10T19:30:56.161407+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:29:30.513872+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:29:30.513872+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0073816690000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":340,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-09-10T19:30:55.154678+0000","last_change":"2024-09-10T19:29:10.112087+0000","last_active":"2024-09-10T19:30:55.154678+0000","last_peered":"2024-09-10T19:30:55.154678+0000","last_clean":"2024-09-10T19:30:55.154678+0000","last_became_active":"2024-09-10T19:27:53.026452+0000","last_became_peered":"2024-09-10T19:27:53.026452+0000","last_unstale":"2024-09-10T19:30:55.154678+0000","last_undegraded":"2024-09-10T19:30:55.154678+0000","last_fullsized":"2024-09-10T19:30:55.154678+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:29:10.111999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:29:10.111999+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0051523979999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.3","version":"423'18030","reported_seq":18535,"reported_epoch":423,"state":"active+clean","last_fresh":"2024-09-10T19:31:07.274235+0000","last_change":"2024-09-10T19:30:41.261593+0000","last_active":"2024-09-10T19:31:07.274235+0000","last_peered":"2024-09-10T19:31:07.274235+0000","last_clean":"2024-09-10T19:31:07.274235+0000","last_became_active":"2024-09-10T19:27:53.010097+0000","last_became_peered":"2024-09-10T19:27:53.010097+0000","last_unstale":"2024-09-10T19:31:07.274235+0000","last_undegraded":"2024-09-10T19:31:07.274235+0000","last_fullsized":"2024-09-10T19:31:07.274235+0000","mapping_epoch":374,"log_start":"414'13451","ondisk_log_start":"414'13451","created":371,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"417'15813","last_scrub_stamp":"2024-09-10T19:30:41.261503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:27:45.679464+0000","last_clean_scrub_stamp":"2024-09-10T19:30:41.261503+0000","objects_scrubbed":2246,"log_size":4579,"log_dups_size":0,"ondisk_log_size":4579,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:31:54.086031+0000","scrub_duration":0.79268019199999995,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18030,"num_write_kb":576960,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1],"acting":[6,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.9","version":"74'79","reported_seq":527,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-09-10T19:30:55.161800+0000","last_change":"2024-09-10T19:30:39.992958+0000","last_active":"2024-09-10T19:30:55.161800+0000","last_peered":"2024-09-10T19:30:55.161800+0000","last_clean":"2024-09-10T19:30:55.161800+0000","last_became_active":"2024-09-10T19:20:30.844108+0000","last_became_peered":"2024-09-10T19:20:30.844108+0000","last_unstale":"2024-09-10T19:30:55.161800+0000","last_undegraded":"2024-09-10T19:30:55.161800+0000","last_fullsized":"2024-09-10T19:30:55.161800+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":4,"last_scrub":"74'79","last_scrub_stamp":"2024-09-10T19:30:39.992877+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:08:00.699315+0000","last_clean_scrub_stamp":"2024-09-10T19:30:39.992877+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:31:53.125566+0000","scrub_duration":0.0080860159999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.c","version":"414'30","reported_seq":551,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-09-10T19:30:55.160873+0000","last_change":"2024-09-10T19:30:08.475937+0000","last_active":"2024-09-10T19:30:55.160873+0000","last_peered":"2024-09-10T19:30:55.160873+0000","last_clean":"2024-09-10T19:30:55.160873+0000","last_became_active":"2024-09-10T19:27:53.012617+0000","last_became_peered":"2024-09-10T19:27:53.012617+0000","last_unstale":"2024-09-10T19:30:55.160873+0000","last_undegraded":"2024-09-10T19:30:55.160873+0000","last_fullsized":"2024-09-10T19:30:55.160873+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"237'20","last_scrub_stamp":"2024-09-10T19:30:08.475832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:30:08.475832+0000","objects_scrubbed":5,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:31:23.767469+0000","scrub_duration":0.0094002249999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":45,"num_read_kb":40,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":170,"reported_epoch":423,"state":"active+clean","last_fresh":"2024-09-10T19:31:07.460505+0000","last_change":"2024-09-10T19:31:07.460481+0000","last_active":"2024-09-10T19:31:07.460505+0000","last_peered":"2024-09-10T19:31:07.460505+0000","last_clean":"2024-09-10T19:31:07.460505+0000","last_became_active":"2024-09-10T19:24:01.973848+0000","last_became_peered":"2024-09-10T19:24:01.973848+0000","last_unstale":"2024-09-10T19:31:07.460505+0000","last_undegraded":"2024-09-10T19:31:07.460505+0000","last_fullsized":"2024-09-10T19:31:07.460505+0000","mapping_epoch":292,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":293,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:31:07.460426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:31:07.460426+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:32:15.828800+0000","scrub_duration":0.001669605,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.d","version":"303'12","reported_seq":585,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-09-10T19:30:56.161300+0000","last_change":"2024-09-10T19:29:15.947794+0000","last_active":"2024-09-10T19:30:56.161300+0000","last_peered":"2024-09-10T19:30:56.161300+0000","last_clean":"2024-09-10T19:30:56.161300+0000","last_became_active":"2024-09-10T19:24:23.906819+0000","last_became_peered":"2024-09-10T19:24:23.906819+0000","last_unstale":"2024-09-10T19:30:56.161300+0000","last_undegraded":"2024-09-10T19:30:56.161300+0000","last_fullsized":"2024-09-10T19:30:56.161300+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"303'12","last_scrub_stamp":"2024-09-10T19:29:15.947693+0000","last_deep_scrub":"80'9","last_deep_scrub_stamp":"2024-09-10T19:11:25.156465+0000","last_clean_scrub_stamp":"2024-09-10T19:29:15.947693+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.012440023,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,0],"acting":[7,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":488,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-09-10T19:30:55.163882+0000","last_change":"2024-09-10T19:30:34.534444+0000","last_active":"2024-09-10T19:30:55.163882+0000","last_peered":"2024-09-10T19:30:55.163882+0000","last_clean":"2024-09-10T19:30:55.163882+0000","last_became_active":"2024-09-10T19:24:01.972593+0000","last_became_peered":"2024-09-10T19:24:01.972593+0000","last_unstale":"2024-09-10T19:30:55.163882+0000","last_undegraded":"2024-09-10T19:30:55.163882+0000","last_fullsized":"2024-09-10T19:30:55.163882+0000","mapping_epoch":292,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":293,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:30:34.534128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:30:34.534128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:31:59.004309+0000","scrub_duration":0.029869611000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,5],"acting":[4,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.0","version":"423'18072","reported_seq":18322,"reported_epoch":423,"state":"active+clean","last_fresh":"2024-09-10T19:31:07.269719+0000","last_change":"2024-09-10T19:30:15.362614+0000","last_active":"2024-09-10T19:31:07.269719+0000","last_peered":"2024-09-10T19:31:07.269719+0000","last_clean":"2024-09-10T19:31:07.269719+0000","last_became_active":"2024-09-10T19:27:46.705538+0000","last_became_peered":"2024-09-10T19:27:46.705538+0000","last_unstale":"2024-09-10T19:31:07.269719+0000","last_undegraded":"2024-09-10T19:31:07.269719+0000","last_fullsized":"2024-09-10T19:31:07.269719+0000","mapping_epoch":371,"log_start":"414'13700","ondisk_log_start":"414'13700","created":371,"last_epoch_clean":372,"parent":"0.0","parent_split_bits":0,"last_scrub":"414'13664","last_scrub_stamp":"2024-09-10T19:30:15.362539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:27:45.679464+0000","last_clean_scrub_stamp":"2024-09-10T19:30:15.362539+0000","objects_scrubbed":4480,"log_size":4372,"log_dups_size":0,"ondisk_log_size":4372,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:31:16.651953+0000","scrub_duration":1.8958626860000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18072,"num_write_kb":578304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.a","version":"74'79","reported_seq":247,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-09-10T19:30:55.173025+0000","last_change":"2024-09-10T19:30:04.397199+0000","last_active":"2024-09-10T19:30:55.173025+0000","last_peered":"2024-09-10T19:30:55.173025+0000","last_clean":"2024-09-10T19:30:55.173025+0000","last_became_active":"2024-09-10T19:27:53.004646+0000","last_became_peered":"2024-09-10T19:27:53.004646+0000","last_unstale":"2024-09-10T19:30:55.173025+0000","last_undegraded":"2024-09-10T19:30:55.173025+0000","last_fullsized":"2024-09-10T19:30:55.173025+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":4,"last_scrub":"74'79","last_scrub_stamp":"2024-09-10T19:30:04.397114+0000","last_deep_scrub":"74'79","last_deep_scrub_stamp":"2024-09-10T19:21:53.326433+0000","last_clean_scrub_stamp":"2024-09-10T19:30:04.397114+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:31:34.304105+0000","scrub_duration":0.0040705209999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,4],"acting":[5,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.f","version":"414'39","reported_seq":194,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-09-10T19:30:56.161898+0000","last_change":"2024-09-10T19:28:26.434825+0000","last_active":"2024-09-10T19:30:56.161898+0000","last_peered":"2024-09-10T19:30:56.161898+0000","last_clean":"2024-09-10T19:30:56.161898+0000","last_became_active":"2024-09-10T19:24:01.979187+0000","last_became_peered":"2024-09-10T19:24:01.979187+0000","last_unstale":"2024-09-10T19:30:56.161898+0000","last_undegraded":"2024-09-10T19:30:56.161898+0000","last_fullsized":"2024-09-10T19:30:56.161898+0000","mapping_epoch":292,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":293,"parent":"0.0","parent_split_bits":0,"last_scrub":"237'27","last_scrub_stamp":"2024-09-10T19:28:26.434563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:28:26.434563+0000","objects_scrubbed":7,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0096637310000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":61,"num_read_kb":52,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":160,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-09-10T19:30:56.160459+0000","last_change":"2024-09-10T19:28:24.420346+0000","last_active":"2024-09-10T19:30:56.160459+0000","last_peered":"2024-09-10T19:30:56.160459+0000","last_clean":"2024-09-10T19:30:56.160459+0000","last_became_active":"2024-09-10T19:24:01.974773+0000","last_became_peered":"2024-09-10T19:24:01.974773+0000","last_unstale":"2024-09-10T19:30:56.160459+0000","last_undegraded":"2024-09-10T19:30:56.160459+0000","last_fullsized":"2024-09-10T19:30:56.160459+0000","mapping_epoch":292,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":293,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:28:24.420270+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:28:24.420270+0000","last_clean_scrub_stamp":"2024-09-10T19:28:24.420270+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.016717709000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.e","version":"303'10","reported_seq":708,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-09-10T19:30:56.158916+0000","last_change":"2024-09-10T19:30:07.517646+0000","last_active":"2024-09-10T19:30:56.158916+0000","last_peered":"2024-09-10T19:30:56.158916+0000","last_clean":"2024-09-10T19:30:56.158916+0000","last_became_active":"2024-09-10T19:24:23.908760+0000","last_became_peered":"2024-09-10T19:24:23.908760+0000","last_unstale":"2024-09-10T19:30:56.158916+0000","last_undegraded":"2024-09-10T19:30:56.158916+0000","last_fullsized":"2024-09-10T19:30:56.158916+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"303'10","last_scrub_stamp":"2024-09-10T19:30:07.517566+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:30:07.517566+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:31:36.012505+0000","scrub_duration":0.0052487269999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":484,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-09-10T19:30:55.176838+0000","last_change":"2024-09-10T19:29:15.298967+0000","last_active":"2024-09-10T19:30:55.176838+0000","last_peered":"2024-09-10T19:30:55.176838+0000","last_clean":"2024-09-10T19:30:55.176838+0000","last_became_active":"2024-09-10T19:24:24.917109+0000","last_became_peered":"2024-09-10T19:24:24.917109+0000","last_unstale":"2024-09-10T19:30:55.176838+0000","last_undegraded":"2024-09-10T19:30:55.176838+0000","last_fullsized":"2024-09-10T19:30:55.176838+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:29:15.298879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:29:15.298879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.003007774,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.1","version":"423'17946","reported_seq":18427,"reported_epoch":423,"state":"active+clean","last_fresh":"2024-09-10T19:31:07.259143+0000","last_change":"2024-09-10T19:30:42.079899+0000","last_active":"2024-09-10T19:31:07.259143+0000","last_peered":"2024-09-10T19:31:07.259143+0000","last_clean":"2024-09-10T19:31:07.259143+0000","last_became_active":"2024-09-10T19:27:53.016565+0000","last_became_peered":"2024-09-10T19:27:53.016565+0000","last_unstale":"2024-09-10T19:31:07.259143+0000","last_undegraded":"2024-09-10T19:31:07.259143+0000","last_fullsized":"2024-09-10T19:31:07.259143+0000","mapping_epoch":374,"log_start":"414'13323","ondisk_log_start":"414'13323","created":371,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"401'9457","last_scrub_stamp":"2024-09-10T19:29:29.986778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:27:45.679464+0000","last_clean_scrub_stamp":"2024-09-10T19:29:29.986778+0000","objects_scrubbed":1549,"log_size":4623,"log_dups_size":0,"ondisk_log_size":4623,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"queued for scrub","scrub_duration":3.5527387689999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17946,"num_write_kb":574272,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.e","version":"414'32","reported_seq":617,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-09-10T19:30:55.163944+0000","last_change":"2024-09-10T19:30:18.472016+0000","last_active":"2024-09-10T19:30:55.163944+0000","last_peered":"2024-09-10T19:30:55.163944+0000","last_clean":"2024-09-10T19:30:55.163944+0000","last_became_active":"2024-09-10T19:27:53.012101+0000","last_became_peered":"2024-09-10T19:27:53.012101+0000","last_unstale":"2024-09-10T19:30:55.163944+0000","last_undegraded":"2024-09-10T19:30:55.163944+0000","last_fullsized":"2024-09-10T19:30:55.163944+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"414'32","last_scrub_stamp":"2024-09-10T19:30:18.471961+0000","last_deep_scrub":"84'16","last_deep_scrub_stamp":"2024-09-10T19:11:56.157975+0000","last_clean_scrub_stamp":"2024-09-10T19:30:18.471961+0000","objects_scrubbed":5,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:31:40.658421+0000","scrub_duration":0.001699491,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":37,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":550,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":133,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-09-10T19:30:55.160140+0000","last_change":"2024-09-10T19:29:30.393193+0000","last_active":"2024-09-10T19:30:55.160140+0000","last_peered":"2024-09-10T19:30:55.160140+0000","last_clean":"2024-09-10T19:30:55.160140+0000","last_became_active":"2024-09-10T19:24:53.756220+0000","last_became_peered":"2024-09-10T19:24:53.756220+0000","last_unstale":"2024-09-10T19:30:55.160140+0000","last_undegraded":"2024-09-10T19:30:55.160140+0000","last_fullsized":"2024-09-10T19:30:55.160140+0000","mapping_epoch":318,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":319,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:29:30.393109+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:24:13.636249+0000","last_clean_scrub_stamp":"2024-09-10T19:29:30.393109+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0027000890000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,3],"acting":[4,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":458,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-09-10T19:30:55.174676+0000","last_change":"2024-09-10T19:30:15.118053+0000","last_active":"2024-09-10T19:30:55.174676+0000","last_peered":"2024-09-10T19:30:55.174676+0000","last_clean":"2024-09-10T19:30:55.174676+0000","last_became_active":"2024-09-10T19:27:53.007094+0000","last_became_peered":"2024-09-10T19:27:53.007094+0000","last_unstale":"2024-09-10T19:30:55.174676+0000","last_undegraded":"2024-09-10T19:30:55.174676+0000","last_fullsized":"2024-09-10T19:30:55.174676+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:30:15.117979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:30:15.117979+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:31:33.405690+0000","scrub_duration":0.0051218720000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":456,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-09-10T19:30:55.154535+0000","last_change":"2024-09-10T19:30:29.757452+0000","last_active":"2024-09-10T19:30:55.154535+0000","last_peered":"2024-09-10T19:30:55.154535+0000","last_clean":"2024-09-10T19:30:55.154535+0000","last_became_active":"2024-09-10T19:27:53.025345+0000","last_became_peered":"2024-09-10T19:27:53.025345+0000","last_unstale":"2024-09-10T19:30:55.154535+0000","last_undegraded":"2024-09-10T19:30:55.154535+0000","last_fullsized":"2024-09-10T19:30:55.154535+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:30:29.757372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:30:29.757372+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:31:37.286874+0000","scrub_duration":0.0047515539999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.e","version":"422'17939","reported_seq":18338,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-09-10T19:31:05.328236+0000","last_change":"2024-09-10T19:29:34.726363+0000","last_active":"2024-09-10T19:31:05.328236+0000","last_peered":"2024-09-10T19:31:05.328236+0000","last_clean":"2024-09-10T19:31:05.328236+0000","last_became_active":"2024-09-10T19:27:46.704103+0000","last_became_peered":"2024-09-10T19:27:46.704103+0000","last_unstale":"2024-09-10T19:31:05.328236+0000","last_undegraded":"2024-09-10T19:31:05.328236+0000","last_fullsized":"2024-09-10T19:31:05.328236+0000","mapping_epoch":371,"log_start":"413'13300","ondisk_log_start":"413'13300","created":371,"last_epoch_clean":372,"parent":"0.0","parent_split_bits":0,"last_scrub":"404'9978","last_scrub_stamp":"2024-09-10T19:29:34.726102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:27:45.679464+0000","last_clean_scrub_stamp":"2024-09-10T19:29:34.726102+0000","objects_scrubbed":8262,"log_size":4639,"log_dups_size":0,"ondisk_log_size":4639,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"queued for scrub","scrub_duration":3.3425435960000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10944512,"num_objects":167,"num_object_clones":0,"num_object_copies":501,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":167,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17939,"num_write_kb":579392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,7],"acting":[4,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.4","version":"317'80","reported_seq":233,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-09-10T19:30:55.154039+0000","last_change":"2024-09-10T19:29:14.237503+0000","last_active":"2024-09-10T19:30:55.154039+0000","last_peered":"2024-09-10T19:30:55.154039+0000","last_clean":"2024-09-10T19:30:55.154039+0000","last_became_active":"2024-09-10T19:20:30.838441+0000","last_became_peered":"2024-09-10T19:20:30.838441+0000","last_unstale":"2024-09-10T19:30:55.154039+0000","last_undegraded":"2024-09-10T19:30:55.154039+0000","last_fullsized":"2024-09-10T19:30:55.154039+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":3,"last_scrub":"317'80","last_scrub_stamp":"2024-09-10T19:29:14.237292+0000","last_deep_scrub":"317'80","last_deep_scrub_stamp":"2024-09-10T19:24:51.540456+0000","last_clean_scrub_stamp":"2024-09-10T19:29:14.237292+0000","objects_scrubbed":1,"log_size":80,"log_dups_size":0,"ondisk_log_size":80,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0022248250000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"414'49","reported_seq":457,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-09-10T19:30:55.160199+0000","last_change":"2024-09-10T19:30:19.518616+0000","last_active":"2024-09-10T19:30:55.160199+0000","last_peered":"2024-09-10T19:30:55.160199+0000","last_clean":"2024-09-10T19:30:55.160199+0000","last_became_active":"2024-09-10T19:27:53.011799+0000","last_became_peered":"2024-09-10T19:27:53.011799+0000","last_unstale":"2024-09-10T19:30:55.160199+0000","last_undegraded":"2024-09-10T19:30:55.160199+0000","last_fullsized":"2024-09-10T19:30:55.160199+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"414'49","last_scrub_stamp":"2024-09-10T19:30:19.518533+0000","last_deep_scrub":"265'35","last_deep_scrub_stamp":"2024-09-10T19:28:24.824459+0000","last_clean_scrub_stamp":"2024-09-10T19:30:19.518533+0000","objects_scrubbed":7,"log_size":49,"log_dups_size":0,"ondisk_log_size":49,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:31:45.050810+0000","scrub_duration":0.0080552469999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":63,"num_read_kb":56,"num_write":42,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":68,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-09-10T19:30:56.165000+0000","last_change":"2024-09-10T19:29:14.668212+0000","last_active":"2024-09-10T19:30:56.165000+0000","last_peered":"2024-09-10T19:30:56.165000+0000","last_clean":"2024-09-10T19:30:56.165000+0000","last_became_active":"2024-09-10T19:27:53.020498+0000","last_became_peered":"2024-09-10T19:27:53.020498+0000","last_unstale":"2024-09-10T19:30:56.165000+0000","last_undegraded":"2024-09-10T19:30:56.165000+0000","last_fullsized":"2024-09-10T19:30:56.165000+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:29:14.668130+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:29:14.668130+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0062469530000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.0","version":"417'13","reported_seq":742,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-09-10T19:30:56.163448+0000","last_change":"2024-09-10T19:30:39.965009+0000","last_active":"2024-09-10T19:30:56.163448+0000","last_peered":"2024-09-10T19:30:56.163448+0000","last_clean":"2024-09-10T19:30:56.163448+0000","last_became_active":"2024-09-10T19:24:01.972645+0000","last_became_peered":"2024-09-10T19:24:01.972645+0000","last_unstale":"2024-09-10T19:30:56.163448+0000","last_undegraded":"2024-09-10T19:30:56.163448+0000","last_fullsized":"2024-09-10T19:30:56.163448+0000","mapping_epoch":292,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":293,"parent":"0.0","parent_split_bits":0,"last_scrub":"417'13","last_scrub_stamp":"2024-09-10T19:30:39.964914+0000","last_deep_scrub":"417'13","last_deep_scrub_stamp":"2024-09-10T19:30:39.964914+0000","last_clean_scrub_stamp":"2024-09-10T19:30:39.964914+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:31:49.349883+0000","scrub_duration":0.0087623219999999995,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":483,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-09-10T19:30:55.175083+0000","last_change":"2024-09-10T19:30:14.879292+0000","last_active":"2024-09-10T19:30:55.175083+0000","last_peered":"2024-09-10T19:30:55.175083+0000","last_clean":"2024-09-10T19:30:55.175083+0000","last_became_active":"2024-09-10T19:27:53.009392+0000","last_became_peered":"2024-09-10T19:27:53.009392+0000","last_unstale":"2024-09-10T19:30:55.175083+0000","last_undegraded":"2024-09-10T19:30:55.175083+0000","last_fullsized":"2024-09-10T19:30:55.175083+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:30:14.879002+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:28:25.822918+0000","last_clean_scrub_stamp":"2024-09-10T19:30:14.879002+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:31:25.289697+0000","scrub_duration":0.011117784,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.f","version":"422'17870","reported_seq":18142,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-09-10T19:31:05.587169+0000","last_change":"2024-09-10T19:30:08.691714+0000","last_active":"2024-09-10T19:31:05.587169+0000","last_peered":"2024-09-10T19:31:05.587169+0000","last_clean":"2024-09-10T19:31:05.587169+0000","last_became_active":"2024-09-10T19:27:46.707637+0000","last_became_peered":"2024-09-10T19:27:46.707637+0000","last_unstale":"2024-09-10T19:31:05.587169+0000","last_undegraded":"2024-09-10T19:31:05.587169+0000","last_fullsized":"2024-09-10T19:31:05.587169+0000","mapping_epoch":371,"log_start":"414'13600","ondisk_log_start":"414'13600","created":371,"last_epoch_clean":372,"parent":"0.0","parent_split_bits":0,"last_scrub":"413'13046","last_scrub_stamp":"2024-09-10T19:30:08.691638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:27:45.679464+0000","last_clean_scrub_stamp":"2024-09-10T19:30:08.691638+0000","objects_scrubbed":5075,"log_size":4270,"log_dups_size":0,"ondisk_log_size":4270,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:31:16.854516+0000","scrub_duration":1.8581280769999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10616832,"num_objects":162,"num_object_clones":0,"num_object_copies":486,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":162,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17870,"num_write_kb":577024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.5","version":"74'79","reported_seq":257,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-09-10T19:30:55.160691+0000","last_change":"2024-09-10T19:30:10.470076+0000","last_active":"2024-09-10T19:30:55.160691+0000","last_peered":"2024-09-10T19:30:55.160691+0000","last_clean":"2024-09-10T19:30:55.160691+0000","last_became_active":"2024-09-10T19:27:53.013021+0000","last_became_peered":"2024-09-10T19:27:53.013021+0000","last_unstale":"2024-09-10T19:30:55.160691+0000","last_undegraded":"2024-09-10T19:30:55.160691+0000","last_fullsized":"2024-09-10T19:30:55.160691+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":3,"last_scrub":"74'79","last_scrub_stamp":"2024-09-10T19:30:10.470024+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:08:00.699315+0000","last_clean_scrub_stamp":"2024-09-10T19:30:10.470024+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:31:19.660974+0000","scrub_duration":0.0083031130000000009,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"414'42","reported_seq":515,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-09-10T19:30:55.154339+0000","last_change":"2024-09-10T19:29:35.143247+0000","last_active":"2024-09-10T19:30:55.154339+0000","last_peered":"2024-09-10T19:30:55.154339+0000","last_clean":"2024-09-10T19:30:55.154339+0000","last_became_active":"2024-09-10T19:24:23.903784+0000","last_became_peered":"2024-09-10T19:24:23.903784+0000","last_unstale":"2024-09-10T19:30:55.154339+0000","last_undegraded":"2024-09-10T19:30:55.154339+0000","last_fullsized":"2024-09-10T19:30:55.154339+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"237'30","last_scrub_stamp":"2024-09-10T19:29:35.143157+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:29:35.143157+0000","objects_scrubbed":8,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:30:58.866237+0000","scrub_duration":0.016979355000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":70,"num_read_kb":58,"num_write":44,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":190,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-09-10T19:30:56.162640+0000","last_change":"2024-09-10T19:30:19.275603+0000","last_active":"2024-09-10T19:30:56.162640+0000","last_peered":"2024-09-10T19:30:56.162640+0000","last_clean":"2024-09-10T19:30:56.162640+0000","last_became_active":"2024-09-10T19:27:53.008302+0000","last_became_peered":"2024-09-10T19:27:53.008302+0000","last_unstale":"2024-09-10T19:30:56.162640+0000","last_undegraded":"2024-09-10T19:30:56.162640+0000","last_fullsized":"2024-09-10T19:30:56.162640+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:30:19.275465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:28:15.534211+0000","last_clean_scrub_stamp":"2024-09-10T19:30:19.275465+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:31:42.840763+0000","scrub_duration":0.0057158039999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":447,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-09-10T19:30:55.160332+0000","last_change":"2024-09-10T19:29:23.262835+0000","last_active":"2024-09-10T19:30:55.160332+0000","last_peered":"2024-09-10T19:30:55.160332+0000","last_clean":"2024-09-10T19:30:55.160332+0000","last_became_active":"2024-09-10T19:10:11.698054+0000","last_became_peered":"2024-09-10T19:10:11.698054+0000","last_unstale":"2024-09-10T19:30:55.160332+0000","last_undegraded":"2024-09-10T19:30:55.160332+0000","last_fullsized":"2024-09-10T19:30:55.160332+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:29:23.262748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:26:24.897519+0000","last_clean_scrub_stamp":"2024-09-10T19:29:23.262748+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0056390679999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":475,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-09-10T19:30:55.160644+0000","last_change":"2024-09-10T19:29:27.360561+0000","last_active":"2024-09-10T19:30:55.160644+0000","last_peered":"2024-09-10T19:30:55.160644+0000","last_clean":"2024-09-10T19:30:55.160644+0000","last_became_active":"2024-09-10T19:27:53.017161+0000","last_became_peered":"2024-09-10T19:27:53.017161+0000","last_unstale":"2024-09-10T19:30:55.160644+0000","last_undegraded":"2024-09-10T19:30:55.160644+0000","last_fullsized":"2024-09-10T19:30:55.160644+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:29:27.360479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:29:27.360479+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0086931189999999992,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.c","version":"423'17976","reported_seq":18246,"reported_epoch":423,"state":"active+clean","last_fresh":"2024-09-10T19:31:07.249481+0000","last_change":"2024-09-10T19:30:09.592766+0000","last_active":"2024-09-10T19:31:07.249481+0000","last_peered":"2024-09-10T19:31:07.249481+0000","last_clean":"2024-09-10T19:31:07.249481+0000","last_became_active":"2024-09-10T19:27:46.705563+0000","last_became_peered":"2024-09-10T19:27:46.705563+0000","last_unstale":"2024-09-10T19:31:07.249481+0000","last_undegraded":"2024-09-10T19:31:07.249481+0000","last_fullsized":"2024-09-10T19:31:07.249481+0000","mapping_epoch":371,"log_start":"414'13600","ondisk_log_start":"414'13600","created":371,"last_epoch_clean":372,"parent":"0.0","parent_split_bits":0,"last_scrub":"413'13084","last_scrub_stamp":"2024-09-10T19:30:09.592622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:27:45.679464+0000","last_clean_scrub_stamp":"2024-09-10T19:30:09.592622+0000","objects_scrubbed":4998,"log_size":4376,"log_dups_size":0,"ondisk_log_size":4376,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:31:23.730993+0000","scrub_duration":2.154546888,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17976,"num_write_kb":575232,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,7],"acting":[1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"74'79","reported_seq":78,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-09-10T19:30:56.165333+0000","last_change":"2024-09-10T19:30:23.520229+0000","last_active":"2024-09-10T19:30:56.165333+0000","last_peered":"2024-09-10T19:30:56.165333+0000","last_clean":"2024-09-10T19:30:56.165333+0000","last_became_active":"2024-09-10T19:27:53.017899+0000","last_became_peered":"2024-09-10T19:27:53.017899+0000","last_unstale":"2024-09-10T19:30:56.165333+0000","last_undegraded":"2024-09-10T19:30:56.165333+0000","last_fullsized":"2024-09-10T19:30:56.165333+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":3,"last_scrub":"74'79","last_scrub_stamp":"2024-09-10T19:30:23.520135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:08:00.699315+0000","last_clean_scrub_stamp":"2024-09-10T19:30:23.520135+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:31:47.744700+0000","scrub_duration":0.0041853660000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"414'47","reported_seq":472,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-09-10T19:30:55.156496+0000","last_change":"2024-09-10T19:30:16.465094+0000","last_active":"2024-09-10T19:30:55.156496+0000","last_peered":"2024-09-10T19:30:55.156496+0000","last_clean":"2024-09-10T19:30:55.156496+0000","last_became_active":"2024-09-10T19:24:23.901836+0000","last_became_peered":"2024-09-10T19:24:23.901836+0000","last_unstale":"2024-09-10T19:30:55.156496+0000","last_undegraded":"2024-09-10T19:30:55.156496+0000","last_fullsized":"2024-09-10T19:30:55.156496+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"414'43","last_scrub_stamp":"2024-09-10T19:30:16.464894+0000","last_deep_scrub":"104'27","last_deep_scrub_stamp":"2024-09-10T19:13:02.225944+0000","last_clean_scrub_stamp":"2024-09-10T19:30:16.464894+0000","objects_scrubbed":8,"log_size":47,"log_dups_size":0,"ondisk_log_size":47,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:31:35.284993+0000","scrub_duration":0.0076287560000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":68,"num_read_kb":54,"num_write":42,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":469,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-09-10T19:30:55.179658+0000","last_change":"2024-09-10T19:30:32.016942+0000","last_active":"2024-09-10T19:30:55.179658+0000","last_peered":"2024-09-10T19:30:55.179658+0000","last_clean":"2024-09-10T19:30:55.179658+0000","last_became_active":"2024-09-10T19:24:23.899556+0000","last_became_peered":"2024-09-10T19:24:23.899556+0000","last_unstale":"2024-09-10T19:30:55.179658+0000","last_undegraded":"2024-09-10T19:30:55.179658+0000","last_fullsized":"2024-09-10T19:30:55.179658+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:30:32.016858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:16:37.779967+0000","last_clean_scrub_stamp":"2024-09-10T19:30:32.016858+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:31:37.792761+0000","scrub_duration":0.0083493209999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":78,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-09-10T19:30:56.164917+0000","last_change":"2024-09-10T19:30:19.498917+0000","last_active":"2024-09-10T19:30:56.164917+0000","last_peered":"2024-09-10T19:30:56.164917+0000","last_clean":"2024-09-10T19:30:56.164917+0000","last_became_active":"2024-09-10T19:27:53.003734+0000","last_became_peered":"2024-09-10T19:27:53.003734+0000","last_unstale":"2024-09-10T19:30:56.164917+0000","last_undegraded":"2024-09-10T19:30:56.164917+0000","last_fullsized":"2024-09-10T19:30:56.164917+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:30:19.498837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:30:19.498837+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:31:47.588619+0000","scrub_duration":0.004490628,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":190,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-09-10T19:30:56.160462+0000","last_change":"2024-09-10T19:29:22.538192+0000","last_active":"2024-09-10T19:30:56.160462+0000","last_peered":"2024-09-10T19:30:56.160462+0000","last_clean":"2024-09-10T19:30:56.160462+0000","last_became_active":"2024-09-10T19:27:53.016905+0000","last_became_peered":"2024-09-10T19:27:53.016905+0000","last_unstale":"2024-09-10T19:30:56.160462+0000","last_undegraded":"2024-09-10T19:30:56.160462+0000","last_fullsized":"2024-09-10T19:30:56.160462+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:29:22.538092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:29:22.538092+0000","last_clean_scrub_stamp":"2024-09-10T19:29:22.538092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0051848659999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,7],"acting":[1,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.d","version":"422'17920","reported_seq":18300,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-09-10T19:31:05.317613+0000","last_change":"2024-09-10T19:30:06.229050+0000","last_active":"2024-09-10T19:31:05.317613+0000","last_peered":"2024-09-10T19:31:05.317613+0000","last_clean":"2024-09-10T19:31:05.317613+0000","last_became_active":"2024-09-10T19:27:53.011209+0000","last_became_peered":"2024-09-10T19:27:53.011209+0000","last_unstale":"2024-09-10T19:31:05.317613+0000","last_undegraded":"2024-09-10T19:31:05.317613+0000","last_fullsized":"2024-09-10T19:31:05.317613+0000","mapping_epoch":374,"log_start":"413'13300","ondisk_log_start":"413'13300","created":371,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"413'12851","last_scrub_stamp":"2024-09-10T19:30:06.228898+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:27:45.679464+0000","last_clean_scrub_stamp":"2024-09-10T19:30:06.228898+0000","objects_scrubbed":5320,"log_size":4620,"log_dups_size":0,"ondisk_log_size":4620,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:31:22.959068+0000","scrub_duration":1.7693102619999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11599896,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":17920,"num_write_kb":579073,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.7","version":"74'79","reported_seq":78,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-09-10T19:30:56.164920+0000","last_change":"2024-09-10T19:30:09.567679+0000","last_active":"2024-09-10T19:30:56.164920+0000","last_peered":"2024-09-10T19:30:56.164920+0000","last_clean":"2024-09-10T19:30:56.164920+0000","last_became_active":"2024-09-10T19:27:53.020085+0000","last_became_peered":"2024-09-10T19:27:53.020085+0000","last_unstale":"2024-09-10T19:30:56.164920+0000","last_undegraded":"2024-09-10T19:30:56.164920+0000","last_fullsized":"2024-09-10T19:30:56.164920+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":3,"last_scrub":"74'79","last_scrub_stamp":"2024-09-10T19:30:09.567596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:08:00.699315+0000","last_clean_scrub_stamp":"2024-09-10T19:30:09.567596+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:31:38.788437+0000","scrub_duration":0.0064510310000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.2","version":"414'35","reported_seq":202,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-09-10T19:30:56.160909+0000","last_change":"2024-09-10T19:29:06.293570+0000","last_active":"2024-09-10T19:30:56.160909+0000","last_peered":"2024-09-10T19:30:56.160909+0000","last_clean":"2024-09-10T19:30:56.160909+0000","last_became_active":"2024-09-10T19:24:01.976841+0000","last_became_peered":"2024-09-10T19:24:01.976841+0000","last_unstale":"2024-09-10T19:30:56.160909+0000","last_undegraded":"2024-09-10T19:30:56.160909+0000","last_fullsized":"2024-09-10T19:30:56.160909+0000","mapping_epoch":292,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":293,"parent":"0.0","parent_split_bits":0,"last_scrub":"350'25","last_scrub_stamp":"2024-09-10T19:29:06.293356+0000","last_deep_scrub":"350'25","last_deep_scrub_stamp":"2024-09-10T19:26:37.415039+0000","last_clean_scrub_stamp":"2024-09-10T19:29:06.293356+0000","objects_scrubbed":5,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0079160140000000007,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":38,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":184,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-09-10T19:30:56.161478+0000","last_change":"2024-09-10T19:29:20.468976+0000","last_active":"2024-09-10T19:30:56.161478+0000","last_peered":"2024-09-10T19:30:56.161478+0000","last_clean":"2024-09-10T19:30:56.161478+0000","last_became_active":"2024-09-10T19:24:24.910173+0000","last_became_peered":"2024-09-10T19:24:24.910173+0000","last_unstale":"2024-09-10T19:30:56.161478+0000","last_undegraded":"2024-09-10T19:30:56.161478+0000","last_fullsized":"2024-09-10T19:30:56.161478+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:29:20.468827+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:29:20.468827+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0064360270000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"376'13","reported_seq":654,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-09-10T19:31:00.979392+0000","last_change":"2024-09-10T19:30:14.496516+0000","last_active":"2024-09-10T19:31:00.979392+0000","last_peered":"2024-09-10T19:31:00.979392+0000","last_clean":"2024-09-10T19:31:00.979392+0000","last_became_active":"2024-09-10T19:27:53.000311+0000","last_became_peered":"2024-09-10T19:27:53.000311+0000","last_unstale":"2024-09-10T19:31:00.979392+0000","last_undegraded":"2024-09-10T19:31:00.979392+0000","last_fullsized":"2024-09-10T19:31:00.979392+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"376'13","last_scrub_stamp":"2024-09-10T19:30:14.496433+0000","last_deep_scrub":"376'13","last_deep_scrub_stamp":"2024-09-10T19:28:17.490297+0000","last_clean_scrub_stamp":"2024-09-10T19:30:14.496433+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:31:34.498533+0000","scrub_duration":0.0060631859999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":422,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-09-10T19:30:55.155352+0000","last_change":"2024-09-10T19:30:37.927650+0000","last_active":"2024-09-10T19:30:55.155352+0000","last_peered":"2024-09-10T19:30:55.155352+0000","last_clean":"2024-09-10T19:30:55.155352+0000","last_became_active":"2024-09-10T19:24:23.905977+0000","last_became_peered":"2024-09-10T19:24:23.905977+0000","last_unstale":"2024-09-10T19:30:55.155352+0000","last_undegraded":"2024-09-10T19:30:55.155352+0000","last_fullsized":"2024-09-10T19:30:55.155352+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:30:37.927553+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:30:37.927553+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:31:49.883711+0000","scrub_duration":0.013815849,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,5],"acting":[0,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.b","version":"422'17967","reported_seq":18248,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-09-10T19:31:05.363405+0000","last_change":"2024-09-10T19:30:07.316594+0000","last_active":"2024-09-10T19:31:05.363405+0000","last_peered":"2024-09-10T19:31:05.363405+0000","last_clean":"2024-09-10T19:31:05.363405+0000","last_became_active":"2024-09-10T19:27:46.709536+0000","last_became_peered":"2024-09-10T19:27:46.709536+0000","last_unstale":"2024-09-10T19:31:05.363405+0000","last_undegraded":"2024-09-10T19:31:05.363405+0000","last_fullsized":"2024-09-10T19:31:05.363405+0000","mapping_epoch":371,"log_start":"415'13800","ondisk_log_start":"415'13800","created":371,"last_epoch_clean":372,"parent":"0.0","parent_split_bits":0,"last_scrub":"413'12913","last_scrub_stamp":"2024-09-10T19:30:07.316444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:27:45.679464+0000","last_clean_scrub_stamp":"2024-09-10T19:30:07.316444+0000","objects_scrubbed":5296,"log_size":4167,"log_dups_size":0,"ondisk_log_size":4167,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:31:23.548541+0000","scrub_duration":1.9481164020000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10027008,"num_objects":153,"num_object_clones":0,"num_object_copies":459,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":153,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17967,"num_write_kb":579840,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,0],"acting":[3,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"74'79","reported_seq":234,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-09-10T19:30:55.174217+0000","last_change":"2024-09-10T19:30:09.313860+0000","last_active":"2024-09-10T19:30:55.174217+0000","last_peered":"2024-09-10T19:30:55.174217+0000","last_clean":"2024-09-10T19:30:55.174217+0000","last_became_active":"2024-09-10T19:24:23.907264+0000","last_became_peered":"2024-09-10T19:24:23.907264+0000","last_unstale":"2024-09-10T19:30:55.174217+0000","last_undegraded":"2024-09-10T19:30:55.174217+0000","last_fullsized":"2024-09-10T19:30:55.174217+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":4,"last_scrub":"74'79","last_scrub_stamp":"2024-09-10T19:30:09.313780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:08:00.699315+0000","last_clean_scrub_stamp":"2024-09-10T19:30:09.313780+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:31:22.971677+0000","scrub_duration":0.0047975530000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,4],"acting":[5,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"414'68","reported_seq":369,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-09-10T19:31:05.981351+0000","last_change":"2024-09-10T19:30:37.347615+0000","last_active":"2024-09-10T19:31:05.981351+0000","last_peered":"2024-09-10T19:31:05.981351+0000","last_clean":"2024-09-10T19:31:05.981351+0000","last_became_active":"2024-09-10T19:27:53.008533+0000","last_became_peered":"2024-09-10T19:27:53.008533+0000","last_unstale":"2024-09-10T19:31:05.981351+0000","last_undegraded":"2024-09-10T19:31:05.981351+0000","last_fullsized":"2024-09-10T19:31:05.981351+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"414'68","last_scrub_stamp":"2024-09-10T19:30:37.347412+0000","last_deep_scrub":"384'56","last_deep_scrub_stamp":"2024-09-10T19:28:23.457119+0000","last_clean_scrub_stamp":"2024-09-10T19:30:37.347412+0000","objects_scrubbed":10,"log_size":68,"log_dups_size":0,"ondisk_log_size":68,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:31:59.524500+0000","scrub_duration":0.012024986,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":83,"num_read_kb":68,"num_write":50,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":40,"num_bytes_recovered":1432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,6],"acting":[1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":496,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-09-10T19:30:55.173099+0000","last_change":"2024-09-10T19:30:26.182093+0000","last_active":"2024-09-10T19:30:55.173099+0000","last_peered":"2024-09-10T19:30:55.173099+0000","last_clean":"2024-09-10T19:30:55.173099+0000","last_became_active":"2024-09-10T19:27:53.004220+0000","last_became_peered":"2024-09-10T19:27:53.004220+0000","last_unstale":"2024-09-10T19:30:55.173099+0000","last_undegraded":"2024-09-10T19:30:55.173099+0000","last_fullsized":"2024-09-10T19:30:55.173099+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:30:26.182025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:30:26.182025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:31:49.676633+0000","scrub_duration":0.0084229500000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":422,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-09-10T19:30:55.156127+0000","last_change":"2024-09-10T19:30:30.855859+0000","last_active":"2024-09-10T19:30:55.156127+0000","last_peered":"2024-09-10T19:30:55.156127+0000","last_clean":"2024-09-10T19:30:55.156127+0000","last_became_active":"2024-09-10T19:24:24.892692+0000","last_became_peered":"2024-09-10T19:24:24.892692+0000","last_unstale":"2024-09-10T19:30:55.156127+0000","last_undegraded":"2024-09-10T19:30:55.156127+0000","last_fullsized":"2024-09-10T19:30:55.156127+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:30:30.855783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:30:30.855783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:31:57.710760+0000","scrub_duration":0.0039706239999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":426,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-09-10T19:30:55.154528+0000","last_change":"2024-09-10T19:29:08.124587+0000","last_active":"2024-09-10T19:30:55.154528+0000","last_peered":"2024-09-10T19:30:55.154528+0000","last_clean":"2024-09-10T19:30:55.154528+0000","last_became_active":"2024-09-10T19:10:13.721744+0000","last_became_peered":"2024-09-10T19:10:13.721744+0000","last_unstale":"2024-09-10T19:30:55.154528+0000","last_undegraded":"2024-09-10T19:30:55.154528+0000","last_fullsized":"2024-09-10T19:30:55.154528+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:29:08.124503+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:29:08.124503+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.010126721999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.8","version":"422'17874","reported_seq":18154,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-09-10T19:31:05.350248+0000","last_change":"2024-09-10T19:30:09.096902+0000","last_active":"2024-09-10T19:31:05.350248+0000","last_peered":"2024-09-10T19:31:05.350248+0000","last_clean":"2024-09-10T19:31:05.350248+0000","last_became_active":"2024-09-10T19:27:53.024817+0000","last_became_peered":"2024-09-10T19:27:53.024817+0000","last_unstale":"2024-09-10T19:31:05.350248+0000","last_undegraded":"2024-09-10T19:31:05.350248+0000","last_fullsized":"2024-09-10T19:31:05.350248+0000","mapping_epoch":374,"log_start":"414'13700","ondisk_log_start":"414'13700","created":371,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"413'13050","last_scrub_stamp":"2024-09-10T19:30:09.096759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:27:45.679464+0000","last_clean_scrub_stamp":"2024-09-10T19:30:09.096759+0000","objects_scrubbed":5028,"log_size":4174,"log_dups_size":0,"ondisk_log_size":4174,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:31:09.617290+0000","scrub_duration":1.7211459499999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9043968,"num_objects":138,"num_object_clones":0,"num_object_copies":414,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":138,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17874,"num_write_kb":576384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"74'79","reported_seq":524,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-09-10T19:30:55.161852+0000","last_change":"2024-09-10T19:29:07.906129+0000","last_active":"2024-09-10T19:30:55.161852+0000","last_peered":"2024-09-10T19:30:55.161852+0000","last_clean":"2024-09-10T19:30:55.161852+0000","last_became_active":"2024-09-10T19:24:23.895247+0000","last_became_peered":"2024-09-10T19:24:23.895247+0000","last_unstale":"2024-09-10T19:30:55.161852+0000","last_undegraded":"2024-09-10T19:30:55.161852+0000","last_fullsized":"2024-09-10T19:30:55.161852+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":4,"last_scrub":"74'79","last_scrub_stamp":"2024-09-10T19:29:07.906054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:08:00.699315+0000","last_clean_scrub_stamp":"2024-09-10T19:29:07.906054+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.006366633,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.7","version":"414'26","reported_seq":202,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-09-10T19:30:56.161198+0000","last_change":"2024-09-10T19:30:34.330565+0000","last_active":"2024-09-10T19:30:56.161198+0000","last_peered":"2024-09-10T19:30:56.161198+0000","last_clean":"2024-09-10T19:30:56.161198+0000","last_became_active":"2024-09-10T19:24:24.894293+0000","last_became_peered":"2024-09-10T19:24:24.894293+0000","last_unstale":"2024-09-10T19:30:56.161198+0000","last_undegraded":"2024-09-10T19:30:56.161198+0000","last_fullsized":"2024-09-10T19:30:56.161198+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"414'26","last_scrub_stamp":"2024-09-10T19:30:34.330411+0000","last_deep_scrub":"82'18","last_deep_scrub_stamp":"2024-09-10T19:15:55.777361+0000","last_clean_scrub_stamp":"2024-09-10T19:30:34.330411+0000","objects_scrubbed":5,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:31:42.777447+0000","scrub_duration":0.010661754000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":42,"num_read_kb":31,"num_write":24,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"159'6","reported_seq":637,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-09-10T19:31:06.628057+0000","last_change":"2024-09-10T19:29:16.485160+0000","last_active":"2024-09-10T19:31:06.628057+0000","last_peered":"2024-09-10T19:31:06.628057+0000","last_clean":"2024-09-10T19:31:06.628057+0000","last_became_active":"2024-09-10T19:27:53.017381+0000","last_became_peered":"2024-09-10T19:27:53.017381+0000","last_unstale":"2024-09-10T19:31:06.628057+0000","last_undegraded":"2024-09-10T19:31:06.628057+0000","last_fullsized":"2024-09-10T19:31:06.628057+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"159'6","last_scrub_stamp":"2024-09-10T19:29:16.485081+0000","last_deep_scrub":"159'6","last_deep_scrub_stamp":"2024-09-10T19:16:32.649914+0000","last_clean_scrub_stamp":"2024-09-10T19:29:16.485081+0000","objects_scrubbed":1,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0054402080000000002,"objects_trimmed":0,"snaptrim_duration":0.00034454499999999997,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":427,"num_read_kb":422,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1556,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":371,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-09-10T19:30:55.154688+0000","last_change":"2024-09-10T19:30:31.835352+0000","last_active":"2024-09-10T19:30:55.154688+0000","last_peered":"2024-09-10T19:30:55.154688+0000","last_clean":"2024-09-10T19:30:55.154688+0000","last_became_active":"2024-09-10T19:24:24.924859+0000","last_became_peered":"2024-09-10T19:24:24.924859+0000","last_unstale":"2024-09-10T19:30:55.154688+0000","last_undegraded":"2024-09-10T19:30:55.154688+0000","last_fullsized":"2024-09-10T19:30:55.154688+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:30:31.835272+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:28:21.199065+0000","last_clean_scrub_stamp":"2024-09-10T19:30:31.835272+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:31:56.356268+0000","scrub_duration":0.0079787279999999992,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":340,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-09-10T19:30:55.173496+0000","last_change":"2024-09-10T19:30:23.179974+0000","last_active":"2024-09-10T19:30:55.173496+0000","last_peered":"2024-09-10T19:30:55.173496+0000","last_clean":"2024-09-10T19:30:55.173496+0000","last_became_active":"2024-09-10T19:16:24.861481+0000","last_became_peered":"2024-09-10T19:16:24.861481+0000","last_unstale":"2024-09-10T19:30:55.173496+0000","last_undegraded":"2024-09-10T19:30:55.173496+0000","last_fullsized":"2024-09-10T19:30:55.173496+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:30:23.179923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:23:28.736502+0000","last_clean_scrub_stamp":"2024-09-10T19:30:23.179923+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:31:53.153842+0000","scrub_duration":0.0011139259999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,4],"acting":[5,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":475,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-09-10T19:30:55.179283+0000","last_change":"2024-09-10T19:30:05.863333+0000","last_active":"2024-09-10T19:30:55.179283+0000","last_peered":"2024-09-10T19:30:55.179283+0000","last_clean":"2024-09-10T19:30:55.179283+0000","last_became_active":"2024-09-10T19:27:53.011923+0000","last_became_peered":"2024-09-10T19:27:53.011923+0000","last_unstale":"2024-09-10T19:30:55.179283+0000","last_undegraded":"2024-09-10T19:30:55.179283+0000","last_fullsized":"2024-09-10T19:30:55.179283+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:30:05.863186+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:22:01.208272+0000","last_clean_scrub_stamp":"2024-09-10T19:30:05.863186+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:31:07.939735+0000","scrub_duration":0.0076258740000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.9","version":"422'18150","reported_seq":18408,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-09-10T19:31:05.375244+0000","last_change":"2024-09-10T19:30:16.747438+0000","last_active":"2024-09-10T19:31:05.375244+0000","last_peered":"2024-09-10T19:31:05.375244+0000","last_clean":"2024-09-10T19:31:05.375244+0000","last_became_active":"2024-09-10T19:27:53.018973+0000","last_became_peered":"2024-09-10T19:27:53.018973+0000","last_unstale":"2024-09-10T19:31:05.375244+0000","last_undegraded":"2024-09-10T19:31:05.375244+0000","last_fullsized":"2024-09-10T19:31:05.375244+0000","mapping_epoch":374,"log_start":"415'14000","ondisk_log_start":"415'14000","created":371,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"414'13961","last_scrub_stamp":"2024-09-10T19:30:16.747355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:27:45.679464+0000","last_clean_scrub_stamp":"2024-09-10T19:30:16.747355+0000","objects_scrubbed":4475,"log_size":4150,"log_dups_size":0,"ondisk_log_size":4150,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:31:28.383270+0000","scrub_duration":2.2859866719999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":11141120,"num_objects":170,"num_object_clones":0,"num_object_copies":510,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":170,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18150,"num_write_kb":586240,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.3","version":"74'79","reported_seq":190,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-09-10T19:30:56.163225+0000","last_change":"2024-09-10T19:30:40.383314+0000","last_active":"2024-09-10T19:30:56.163225+0000","last_peered":"2024-09-10T19:30:56.163225+0000","last_clean":"2024-09-10T19:30:56.163225+0000","last_became_active":"2024-09-10T19:27:53.016965+0000","last_became_peered":"2024-09-10T19:27:53.016965+0000","last_unstale":"2024-09-10T19:30:56.163225+0000","last_undegraded":"2024-09-10T19:30:56.163225+0000","last_fullsized":"2024-09-10T19:30:56.163225+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":3,"last_scrub":"74'79","last_scrub_stamp":"2024-09-10T19:30:40.383237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:08:00.699315+0000","last_clean_scrub_stamp":"2024-09-10T19:30:40.383237+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:31:57.251947+0000","scrub_duration":0.0084886000000000007,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"414'42","reported_seq":520,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-09-10T19:30:55.155975+0000","last_change":"2024-09-10T19:29:36.586040+0000","last_active":"2024-09-10T19:30:55.155975+0000","last_peered":"2024-09-10T19:30:55.155975+0000","last_clean":"2024-09-10T19:30:55.155975+0000","last_became_active":"2024-09-10T19:24:24.899711+0000","last_became_peered":"2024-09-10T19:24:24.899711+0000","last_unstale":"2024-09-10T19:30:55.155975+0000","last_undegraded":"2024-09-10T19:30:55.155975+0000","last_fullsized":"2024-09-10T19:30:55.155975+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"237'30","last_scrub_stamp":"2024-09-10T19:29:36.584508+0000","last_deep_scrub":"221'18","last_deep_scrub_stamp":"2024-09-10T19:19:29.606748+0000","last_clean_scrub_stamp":"2024-09-10T19:29:36.584508+0000","objects_scrubbed":6,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0052445089999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":53,"num_read_kb":47,"num_write":36,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"417'2","reported_seq":416,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-09-10T19:30:55.155492+0000","last_change":"2024-09-10T19:30:39.904486+0000","last_active":"2024-09-10T19:30:55.155492+0000","last_peered":"2024-09-10T19:30:55.155492+0000","last_clean":"2024-09-10T19:30:55.155492+0000","last_became_active":"2024-09-10T19:24:23.897412+0000","last_became_peered":"2024-09-10T19:24:23.897412+0000","last_unstale":"2024-09-10T19:30:55.155492+0000","last_undegraded":"2024-09-10T19:30:55.155492+0000","last_fullsized":"2024-09-10T19:30:55.155492+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"417'2","last_scrub_stamp":"2024-09-10T19:30:39.904339+0000","last_deep_scrub":"417'2","last_deep_scrub_stamp":"2024-09-10T19:30:39.904339+0000","last_clean_scrub_stamp":"2024-09-10T19:30:39.904339+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:32:02.472678+0000","scrub_duration":0.0076308299999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":490,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-09-10T19:30:55.177603+0000","last_change":"2024-09-10T19:30:17.882408+0000","last_active":"2024-09-10T19:30:55.177603+0000","last_peered":"2024-09-10T19:30:55.177603+0000","last_clean":"2024-09-10T19:30:55.177603+0000","last_became_active":"2024-09-10T19:24:24.896201+0000","last_became_peered":"2024-09-10T19:24:24.896201+0000","last_unstale":"2024-09-10T19:30:55.177603+0000","last_undegraded":"2024-09-10T19:30:55.177603+0000","last_fullsized":"2024-09-10T19:30:55.177603+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:30:17.882327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:25:28.762237+0000","last_clean_scrub_stamp":"2024-09-10T19:30:17.882327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:31:19.706681+0000","scrub_duration":0.010437434000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":496,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-09-10T19:30:55.174739+0000","last_change":"2024-09-10T19:30:35.213643+0000","last_active":"2024-09-10T19:30:55.174739+0000","last_peered":"2024-09-10T19:30:55.174739+0000","last_clean":"2024-09-10T19:30:55.174739+0000","last_became_active":"2024-09-10T19:24:24.914680+0000","last_became_peered":"2024-09-10T19:24:24.914680+0000","last_unstale":"2024-09-10T19:30:55.174739+0000","last_undegraded":"2024-09-10T19:30:55.174739+0000","last_fullsized":"2024-09-10T19:30:55.174739+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:30:35.213557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:30:35.213557+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:31:44.941744+0000","scrub_duration":0.0069688670000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":170,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-09-10T19:30:56.161334+0000","last_change":"2024-09-10T19:30:31.245345+0000","last_active":"2024-09-10T19:30:56.161334+0000","last_peered":"2024-09-10T19:30:56.161334+0000","last_clean":"2024-09-10T19:30:56.161334+0000","last_became_active":"2024-09-10T19:24:01.974039+0000","last_became_peered":"2024-09-10T19:24:01.974039+0000","last_unstale":"2024-09-10T19:30:56.161334+0000","last_undegraded":"2024-09-10T19:30:56.161334+0000","last_fullsized":"2024-09-10T19:30:56.161334+0000","mapping_epoch":292,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":293,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:30:31.245191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:30:31.245191+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:31:45.846908+0000","scrub_duration":0.0041648700000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.a","version":"422'18183","reported_seq":18444,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-09-10T19:31:05.362701+0000","last_change":"2024-09-10T19:30:15.315333+0000","last_active":"2024-09-10T19:31:05.362701+0000","last_peered":"2024-09-10T19:31:05.362701+0000","last_clean":"2024-09-10T19:31:05.362701+0000","last_became_active":"2024-09-10T19:27:53.026021+0000","last_became_peered":"2024-09-10T19:27:53.026021+0000","last_unstale":"2024-09-10T19:31:05.362701+0000","last_undegraded":"2024-09-10T19:31:05.362701+0000","last_fullsized":"2024-09-10T19:31:05.362701+0000","mapping_epoch":374,"log_start":"414'14000","ondisk_log_start":"414'14000","created":371,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"414'13886","last_scrub_stamp":"2024-09-10T19:30:15.315183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:27:45.679464+0000","last_clean_scrub_stamp":"2024-09-10T19:30:15.315183+0000","objects_scrubbed":4548,"log_size":4183,"log_dups_size":0,"ondisk_log_size":4183,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":2,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:31:19.467846+0000","scrub_duration":1.898975455,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":12255232,"num_objects":187,"num_object_clones":0,"num_object_copies":561,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":187,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18183,"num_write_kb":587840,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"414'45","reported_seq":112,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-09-10T19:30:56.171172+0000","last_change":"2024-09-10T19:30:05.568635+0000","last_active":"2024-09-10T19:30:56.171172+0000","last_peered":"2024-09-10T19:30:56.171172+0000","last_clean":"2024-09-10T19:30:56.171172+0000","last_became_active":"2024-09-10T19:27:53.006290+0000","last_became_peered":"2024-09-10T19:27:53.006290+0000","last_unstale":"2024-09-10T19:30:56.171172+0000","last_undegraded":"2024-09-10T19:30:56.171172+0000","last_fullsized":"2024-09-10T19:30:56.171172+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"237'33","last_scrub_stamp":"2024-09-10T19:30:05.568555+0000","last_deep_scrub":"80'21","last_deep_scrub_stamp":"2024-09-10T19:11:39.836144+0000","last_clean_scrub_stamp":"2024-09-10T19:30:05.568555+0000","objects_scrubbed":7,"log_size":45,"log_dups_size":0,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:31:29.632123+0000","scrub_duration":0.0023682590000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":60,"num_read_kb":50,"num_write":37,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":154,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"63'2","reported_seq":487,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-09-10T19:30:55.155012+0000","last_change":"2024-09-10T19:30:21.647345+0000","last_active":"2024-09-10T19:30:55.155012+0000","last_peered":"2024-09-10T19:30:55.155012+0000","last_clean":"2024-09-10T19:30:55.155012+0000","last_became_active":"2024-09-10T19:27:53.025802+0000","last_became_peered":"2024-09-10T19:27:53.025802+0000","last_unstale":"2024-09-10T19:30:55.155012+0000","last_undegraded":"2024-09-10T19:30:55.155012+0000","last_fullsized":"2024-09-10T19:30:55.155012+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"63'2","last_scrub_stamp":"2024-09-10T19:30:21.647282+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:30:21.647282+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:31:27.555961+0000","scrub_duration":0.0017519429999999999,"objects_trimmed":0,"snaptrim_duration":0.00075522999999999996,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":462,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-09-10T19:30:55.174562+0000","last_change":"2024-09-10T19:30:25.207929+0000","last_active":"2024-09-10T19:30:55.174562+0000","last_peered":"2024-09-10T19:30:55.174562+0000","last_clean":"2024-09-10T19:30:55.174562+0000","last_became_active":"2024-09-10T19:16:24.858662+0000","last_became_peered":"2024-09-10T19:16:24.858662+0000","last_unstale":"2024-09-10T19:30:55.174562+0000","last_undegraded":"2024-09-10T19:30:55.174562+0000","last_fullsized":"2024-09-10T19:30:55.174562+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:30:25.207849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:30:25.207849+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:31:37.377965+0000","scrub_duration":0.0026356230000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"74'79","reported_seq":538,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-09-10T19:30:55.177739+0000","last_change":"2024-09-10T19:29:34.083194+0000","last_active":"2024-09-10T19:30:55.177739+0000","last_peered":"2024-09-10T19:30:55.177739+0000","last_clean":"2024-09-10T19:30:55.177739+0000","last_became_active":"2024-09-10T19:27:53.009367+0000","last_became_peered":"2024-09-10T19:27:53.009367+0000","last_unstale":"2024-09-10T19:30:55.177739+0000","last_undegraded":"2024-09-10T19:30:55.177739+0000","last_fullsized":"2024-09-10T19:30:55.177739+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'79","last_scrub_stamp":"2024-09-10T19:29:34.080904+0000","last_deep_scrub":"74'79","last_deep_scrub_stamp":"2024-09-10T19:29:34.080904+0000","last_clean_scrub_stamp":"2024-09-10T19:29:34.080904+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:31:01.369559+0000","scrub_duration":0.0067365319999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":475,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-09-10T19:30:55.177802+0000","last_change":"2024-09-10T19:30:32.974376+0000","last_active":"2024-09-10T19:30:55.177802+0000","last_peered":"2024-09-10T19:30:55.177802+0000","last_clean":"2024-09-10T19:30:55.177802+0000","last_became_active":"2024-09-10T19:24:23.903868+0000","last_became_peered":"2024-09-10T19:24:23.903868+0000","last_unstale":"2024-09-10T19:30:55.177802+0000","last_undegraded":"2024-09-10T19:30:55.177802+0000","last_fullsized":"2024-09-10T19:30:55.177802+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:30:32.973996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:30:32.973996+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:31:46.944272+0000","scrub_duration":0.0038225009999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,0],"acting":[7,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":446,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-09-10T19:30:55.174868+0000","last_change":"2024-09-10T19:30:21.186215+0000","last_active":"2024-09-10T19:30:55.174868+0000","last_peered":"2024-09-10T19:30:55.174868+0000","last_clean":"2024-09-10T19:30:55.174868+0000","last_became_active":"2024-09-10T19:10:13.720647+0000","last_became_peered":"2024-09-10T19:10:13.720647+0000","last_unstale":"2024-09-10T19:30:55.174868+0000","last_undegraded":"2024-09-10T19:30:55.174868+0000","last_fullsized":"2024-09-10T19:30:55.174868+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:30:21.186015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:28:29.899579+0000","last_clean_scrub_stamp":"2024-09-10T19:30:21.186015+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:31:41.388908+0000","scrub_duration":0.0046533069999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.5","version":"422'18354","reported_seq":18801,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-09-10T19:31:05.591712+0000","last_change":"2024-09-10T19:29:25.827058+0000","last_active":"2024-09-10T19:31:05.591712+0000","last_peered":"2024-09-10T19:31:05.591712+0000","last_clean":"2024-09-10T19:31:05.591712+0000","last_became_active":"2024-09-10T19:27:53.010838+0000","last_became_peered":"2024-09-10T19:27:53.010838+0000","last_unstale":"2024-09-10T19:31:05.591712+0000","last_undegraded":"2024-09-10T19:31:05.591712+0000","last_fullsized":"2024-09-10T19:31:05.591712+0000","mapping_epoch":374,"log_start":"414'14100","ondisk_log_start":"414'14100","created":371,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"401'9408","last_scrub_stamp":"2024-09-10T19:29:25.826994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:27:45.679464+0000","last_clean_scrub_stamp":"2024-09-10T19:29:25.826994+0000","objects_scrubbed":9189,"log_size":4254,"log_dups_size":0,"ondisk_log_size":4254,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"queued for scrub","scrub_duration":2.7029503350000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":9699328,"num_objects":148,"num_object_clones":0,"num_object_copies":444,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":148,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18354,"num_write_kb":592064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"414'51","reported_seq":122,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-09-10T19:30:56.165152+0000","last_change":"2024-09-10T19:30:12.541122+0000","last_active":"2024-09-10T19:30:56.165152+0000","last_peered":"2024-09-10T19:30:56.165152+0000","last_clean":"2024-09-10T19:30:56.165152+0000","last_became_active":"2024-09-10T19:27:53.014163+0000","last_became_peered":"2024-09-10T19:27:53.014163+0000","last_unstale":"2024-09-10T19:30:56.165152+0000","last_undegraded":"2024-09-10T19:30:56.165152+0000","last_fullsized":"2024-09-10T19:30:56.165152+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"237'35","last_scrub_stamp":"2024-09-10T19:30:12.541044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:30:12.541044+0000","objects_scrubbed":9,"log_size":51,"log_dups_size":0,"ondisk_log_size":51,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:31:28.477210+0000","scrub_duration":0.003693792,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":80,"num_read_kb":69,"num_write":52,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":354,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-09-10T19:30:55.174376+0000","last_change":"2024-09-10T19:29:34.448268+0000","last_active":"2024-09-10T19:30:55.174376+0000","last_peered":"2024-09-10T19:30:55.174376+0000","last_clean":"2024-09-10T19:30:55.174376+0000","last_became_active":"2024-09-10T19:24:24.913709+0000","last_became_peered":"2024-09-10T19:24:24.913709+0000","last_unstale":"2024-09-10T19:30:55.174376+0000","last_undegraded":"2024-09-10T19:30:55.174376+0000","last_fullsized":"2024-09-10T19:30:55.174376+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:29:34.448187+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:29:34.448187+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:30:58.993915+0000","scrub_duration":0.0038381330000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,1],"acting":[5,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":282,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-09-10T19:30:55.156794+0000","last_change":"2024-09-10T19:30:27.831433+0000","last_active":"2024-09-10T19:30:55.156794+0000","last_peered":"2024-09-10T19:30:55.156794+0000","last_clean":"2024-09-10T19:30:55.156794+0000","last_became_active":"2024-09-10T19:24:23.901415+0000","last_became_peered":"2024-09-10T19:24:23.901415+0000","last_unstale":"2024-09-10T19:30:55.156794+0000","last_undegraded":"2024-09-10T19:30:55.156794+0000","last_fullsized":"2024-09-10T19:30:55.156794+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:30:27.831297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:30:27.831297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:31:47.099106+0000","scrub_duration":0.0028653020000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":486,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-09-10T19:30:55.177668+0000","last_change":"2024-09-10T19:30:21.912161+0000","last_active":"2024-09-10T19:30:55.177668+0000","last_peered":"2024-09-10T19:30:55.177668+0000","last_clean":"2024-09-10T19:30:55.177668+0000","last_became_active":"2024-09-10T19:27:53.008850+0000","last_became_peered":"2024-09-10T19:27:53.008850+0000","last_unstale":"2024-09-10T19:30:55.177668+0000","last_undegraded":"2024-09-10T19:30:55.177668+0000","last_fullsized":"2024-09-10T19:30:55.177668+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:30:21.912110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:21:03.910074+0000","last_clean_scrub_stamp":"2024-09-10T19:30:21.912110+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:31:37.278159+0000","scrub_duration":0.001113969,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,6],"acting":[7,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.4","version":"422'17803","reported_seq":18231,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-09-10T19:31:05.573558+0000","last_change":"2024-09-10T19:29:28.791421+0000","last_active":"2024-09-10T19:31:05.573558+0000","last_peered":"2024-09-10T19:31:05.573558+0000","last_clean":"2024-09-10T19:31:05.573558+0000","last_became_active":"2024-09-10T19:27:53.012510+0000","last_became_peered":"2024-09-10T19:27:53.012510+0000","last_unstale":"2024-09-10T19:31:05.573558+0000","last_undegraded":"2024-09-10T19:31:05.573558+0000","last_fullsized":"2024-09-10T19:31:05.573558+0000","mapping_epoch":374,"log_start":"414'13500","ondisk_log_start":"414'13500","created":371,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"401'9402","last_scrub_stamp":"2024-09-10T19:29:28.791340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:27:45.679464+0000","last_clean_scrub_stamp":"2024-09-10T19:29:28.791340+0000","objects_scrubbed":8715,"log_size":4303,"log_dups_size":0,"ondisk_log_size":4303,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":4,"scrub_schedule":"queued for scrub","scrub_duration":3.694930034,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":10551296,"num_objects":161,"num_object_clones":0,"num_object_copies":483,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":161,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17803,"num_write_kb":574848,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"414'21","reported_seq":495,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-09-10T19:30:55.156647+0000","last_change":"2024-09-10T19:30:13.503181+0000","last_active":"2024-09-10T19:30:55.156647+0000","last_peered":"2024-09-10T19:30:55.156647+0000","last_clean":"2024-09-10T19:30:55.156647+0000","last_became_active":"2024-09-10T19:24:44.892294+0000","last_became_peered":"2024-09-10T19:24:44.892294+0000","last_unstale":"2024-09-10T19:30:55.156647+0000","last_undegraded":"2024-09-10T19:30:55.156647+0000","last_fullsized":"2024-09-10T19:30:55.156647+0000","mapping_epoch":314,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":315,"parent":"0.0","parent_split_bits":0,"last_scrub":"237'15","last_scrub_stamp":"2024-09-10T19:30:13.502912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:30:13.502912+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:31:38.767513+0000","scrub_duration":0.0086509359999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":34,"num_read_kb":28,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,4],"acting":[0,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":160,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-09-10T19:30:56.161271+0000","last_change":"2024-09-10T19:29:14.403288+0000","last_active":"2024-09-10T19:30:56.161271+0000","last_peered":"2024-09-10T19:30:56.161271+0000","last_clean":"2024-09-10T19:30:56.161271+0000","last_became_active":"2024-09-10T19:24:01.980348+0000","last_became_peered":"2024-09-10T19:24:01.980348+0000","last_unstale":"2024-09-10T19:30:56.161271+0000","last_undegraded":"2024-09-10T19:30:56.161271+0000","last_fullsized":"2024-09-10T19:30:56.161271+0000","mapping_epoch":292,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":293,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:29:14.403209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:11:44.254513+0000","last_clean_scrub_stamp":"2024-09-10T19:29:14.403209+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0046595619999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":350,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-09-10T19:30:55.160517+0000","last_change":"2024-09-10T19:30:05.467928+0000","last_active":"2024-09-10T19:30:55.160517+0000","last_peered":"2024-09-10T19:30:55.160517+0000","last_clean":"2024-09-10T19:30:55.160517+0000","last_became_active":"2024-09-10T19:24:23.891161+0000","last_became_peered":"2024-09-10T19:24:23.891161+0000","last_unstale":"2024-09-10T19:30:55.160517+0000","last_undegraded":"2024-09-10T19:30:55.160517+0000","last_fullsized":"2024-09-10T19:30:55.160517+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:30:05.467845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:30:05.467845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:31:32.032620+0000","scrub_duration":0.0056686540000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":400,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-09-10T19:30:55.155627+0000","last_change":"2024-09-10T19:30:33.974218+0000","last_active":"2024-09-10T19:30:55.155627+0000","last_peered":"2024-09-10T19:30:55.155627+0000","last_clean":"2024-09-10T19:30:55.155627+0000","last_became_active":"2024-09-10T19:24:23.905112+0000","last_became_peered":"2024-09-10T19:24:23.905112+0000","last_unstale":"2024-09-10T19:30:55.155627+0000","last_undegraded":"2024-09-10T19:30:55.155627+0000","last_fullsized":"2024-09-10T19:30:55.155627+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:30:33.974134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:30:33.974134+0000","last_clean_scrub_stamp":"2024-09-10T19:30:33.974134+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:31:46.881624+0000","scrub_duration":0.0063507879999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.7","version":"422'17902","reported_seq":18303,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-09-10T19:31:05.587284+0000","last_change":"2024-09-10T19:29:34.192828+0000","last_active":"2024-09-10T19:31:05.587284+0000","last_peered":"2024-09-10T19:31:05.587284+0000","last_clean":"2024-09-10T19:31:05.587284+0000","last_became_active":"2024-09-10T19:27:46.707393+0000","last_became_peered":"2024-09-10T19:27:46.707393+0000","last_unstale":"2024-09-10T19:31:05.587284+0000","last_undegraded":"2024-09-10T19:31:05.587284+0000","last_fullsized":"2024-09-10T19:31:05.587284+0000","mapping_epoch":371,"log_start":"413'13200","ondisk_log_start":"413'13200","created":371,"last_epoch_clean":372,"parent":"0.0","parent_split_bits":0,"last_scrub":"404'9864","last_scrub_stamp":"2024-09-10T19:29:34.192737+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:27:45.679464+0000","last_clean_scrub_stamp":"2024-09-10T19:29:34.192737+0000","objects_scrubbed":8292,"log_size":4702,"log_dups_size":0,"ondisk_log_size":4702,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"queued for scrub","scrub_duration":2.7792294690000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":8912896,"num_objects":136,"num_object_clones":0,"num_object_copies":408,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":136,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17902,"num_write_kb":577216,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,4],"acting":[5,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.8","version":"414'46","reported_seq":560,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-09-10T19:30:55.173632+0000","last_change":"2024-09-10T19:30:06.380329+0000","last_active":"2024-09-10T19:30:55.173632+0000","last_peered":"2024-09-10T19:30:55.173632+0000","last_clean":"2024-09-10T19:30:55.173632+0000","last_became_active":"2024-09-10T19:27:53.004858+0000","last_became_peered":"2024-09-10T19:27:53.004858+0000","last_unstale":"2024-09-10T19:30:55.173632+0000","last_undegraded":"2024-09-10T19:30:55.173632+0000","last_fullsized":"2024-09-10T19:30:55.173632+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"237'34","last_scrub_stamp":"2024-09-10T19:30:06.380238+0000","last_deep_scrub":"220'22","last_deep_scrub_stamp":"2024-09-10T19:19:24.597866+0000","last_clean_scrub_stamp":"2024-09-10T19:30:06.380238+0000","objects_scrubbed":7,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:31:27.961508+0000","scrub_duration":0.0065648299999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":62,"num_read_kb":53,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":350,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-09-10T19:30:55.154695+0000","last_change":"2024-09-10T19:30:25.698629+0000","last_active":"2024-09-10T19:30:55.154695+0000","last_peered":"2024-09-10T19:30:55.154695+0000","last_clean":"2024-09-10T19:30:55.154695+0000","last_became_active":"2024-09-10T19:27:53.016898+0000","last_became_peered":"2024-09-10T19:27:53.016898+0000","last_unstale":"2024-09-10T19:30:55.154695+0000","last_undegraded":"2024-09-10T19:30:55.154695+0000","last_fullsized":"2024-09-10T19:30:55.154695+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:30:25.698575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:30:25.698575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:31:48.430610+0000","scrub_duration":0.008668004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.9","version":"376'11","reported_seq":746,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-09-10T19:30:56.160888+0000","last_change":"2024-09-10T19:30:09.879798+0000","last_active":"2024-09-10T19:30:56.160888+0000","last_peered":"2024-09-10T19:30:56.160888+0000","last_clean":"2024-09-10T19:30:56.160888+0000","last_became_active":"2024-09-10T19:27:53.009810+0000","last_became_peered":"2024-09-10T19:27:53.009810+0000","last_unstale":"2024-09-10T19:30:56.160888+0000","last_undegraded":"2024-09-10T19:30:56.160888+0000","last_fullsized":"2024-09-10T19:30:56.160888+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"376'11","last_scrub_stamp":"2024-09-10T19:30:09.879695+0000","last_deep_scrub":"80'9","last_deep_scrub_stamp":"2024-09-10T19:25:29.833271+0000","last_clean_scrub_stamp":"2024-09-10T19:30:09.879695+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:31:37.107588+0000","scrub_duration":0.0040800790000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":463,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-09-10T19:30:55.173497+0000","last_change":"2024-09-10T19:29:27.408515+0000","last_active":"2024-09-10T19:30:55.173497+0000","last_peered":"2024-09-10T19:30:55.173497+0000","last_clean":"2024-09-10T19:30:55.173497+0000","last_became_active":"2024-09-10T19:27:53.015957+0000","last_became_peered":"2024-09-10T19:27:53.015957+0000","last_unstale":"2024-09-10T19:30:55.173497+0000","last_undegraded":"2024-09-10T19:30:55.173497+0000","last_fullsized":"2024-09-10T19:30:55.173497+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:29:27.408374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:29:27.408374+0000","last_clean_scrub_stamp":"2024-09-10T19:29:27.408374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0088258940000000008,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.6","version":"422'18229","reported_seq":18636,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-09-10T19:31:07.076129+0000","last_change":"2024-09-10T19:29:17.664714+0000","last_active":"2024-09-10T19:31:07.076129+0000","last_peered":"2024-09-10T19:31:07.076129+0000","last_clean":"2024-09-10T19:31:07.076129+0000","last_became_active":"2024-09-10T19:27:46.704823+0000","last_became_peered":"2024-09-10T19:27:46.704823+0000","last_unstale":"2024-09-10T19:31:07.076129+0000","last_undegraded":"2024-09-10T19:31:07.076129+0000","last_fullsized":"2024-09-10T19:31:07.076129+0000","mapping_epoch":371,"log_start":"414'13600","ondisk_log_start":"414'13600","created":371,"last_epoch_clean":372,"parent":"0.0","parent_split_bits":0,"last_scrub":"400'8581","last_scrub_stamp":"2024-09-10T19:29:17.664631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:27:45.679464+0000","last_clean_scrub_stamp":"2024-09-10T19:29:17.664631+0000","objects_scrubbed":8456,"log_size":4629,"log_dups_size":0,"ondisk_log_size":4629,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":3,"scrub_schedule":"queued for scrub","scrub_duration":2.9535849939999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1507328,"num_objects":23,"num_object_clones":0,"num_object_copies":69,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18229,"num_write_kb":584064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.9","version":"414'24","reported_seq":534,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-09-10T19:30:55.160486+0000","last_change":"2024-09-10T19:29:18.241176+0000","last_active":"2024-09-10T19:30:55.160486+0000","last_peered":"2024-09-10T19:30:55.160486+0000","last_clean":"2024-09-10T19:30:55.160486+0000","last_became_active":"2024-09-10T19:24:01.974878+0000","last_became_peered":"2024-09-10T19:24:01.974878+0000","last_unstale":"2024-09-10T19:30:55.160486+0000","last_undegraded":"2024-09-10T19:30:55.160486+0000","last_fullsized":"2024-09-10T19:30:55.160486+0000","mapping_epoch":292,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":293,"parent":"0.0","parent_split_bits":0,"last_scrub":"237'18","last_scrub_stamp":"2024-09-10T19:29:18.241077+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:29:18.241077+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0095846869999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.f","version":"59'2","reported_seq":474,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-09-10T19:30:55.161338+0000","last_change":"2024-09-10T19:30:15.444681+0000","last_active":"2024-09-10T19:30:55.161338+0000","last_peered":"2024-09-10T19:30:55.161338+0000","last_clean":"2024-09-10T19:30:55.161338+0000","last_became_active":"2024-09-10T19:24:23.893796+0000","last_became_peered":"2024-09-10T19:24:23.893796+0000","last_unstale":"2024-09-10T19:30:55.161338+0000","last_undegraded":"2024-09-10T19:30:55.161338+0000","last_fullsized":"2024-09-10T19:30:55.161338+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'2","last_scrub_stamp":"2024-09-10T19:30:15.444556+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:30:15.444556+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:31:42.533307+0000","scrub_duration":0.003937945,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":14,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":384,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-09-10T19:30:55.155275+0000","last_change":"2024-09-10T19:29:16.702372+0000","last_active":"2024-09-10T19:30:55.155275+0000","last_peered":"2024-09-10T19:30:55.155275+0000","last_clean":"2024-09-10T19:30:55.155275+0000","last_became_active":"2024-09-10T19:24:24.914252+0000","last_became_peered":"2024-09-10T19:24:24.914252+0000","last_unstale":"2024-09-10T19:30:55.155275+0000","last_undegraded":"2024-09-10T19:30:55.155275+0000","last_fullsized":"2024-09-10T19:30:55.155275+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:29:16.702283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:11:46.587846+0000","last_clean_scrub_stamp":"2024-09-10T19:29:16.702283+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0047568310000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,5],"acting":[0,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":463,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-09-10T19:30:55.154412+0000","last_change":"2024-09-10T19:30:17.554426+0000","last_active":"2024-09-10T19:30:55.154412+0000","last_peered":"2024-09-10T19:30:55.154412+0000","last_clean":"2024-09-10T19:30:55.154412+0000","last_became_active":"2024-09-10T19:24:01.971131+0000","last_became_peered":"2024-09-10T19:24:01.971131+0000","last_unstale":"2024-09-10T19:30:55.154412+0000","last_undegraded":"2024-09-10T19:30:55.154412+0000","last_fullsized":"2024-09-10T19:30:55.154412+0000","mapping_epoch":292,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":293,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:30:17.554352+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:30:17.554352+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:31:26.930983+0000","scrub_duration":0.001698869,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.16","version":"414'21","reported_seq":449,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-09-10T19:30:55.155854+0000","last_change":"2024-09-10T19:29:13.724314+0000","last_active":"2024-09-10T19:30:55.155854+0000","last_peered":"2024-09-10T19:30:55.155854+0000","last_clean":"2024-09-10T19:30:55.155854+0000","last_became_active":"2024-09-10T19:24:23.901610+0000","last_became_peered":"2024-09-10T19:24:23.901610+0000","last_unstale":"2024-09-10T19:30:55.155854+0000","last_undegraded":"2024-09-10T19:30:55.155854+0000","last_fullsized":"2024-09-10T19:30:55.155854+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"237'15","last_scrub_stamp":"2024-09-10T19:29:13.724222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:29:13.724222+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0074413179999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":34,"num_read_kb":28,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":184,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-09-10T19:30:56.163813+0000","last_change":"2024-09-10T19:29:18.504177+0000","last_active":"2024-09-10T19:30:56.163813+0000","last_peered":"2024-09-10T19:30:56.163813+0000","last_clean":"2024-09-10T19:30:56.163813+0000","last_became_active":"2024-09-10T19:24:24.894815+0000","last_became_peered":"2024-09-10T19:24:24.894815+0000","last_unstale":"2024-09-10T19:30:56.163813+0000","last_undegraded":"2024-09-10T19:30:56.163813+0000","last_fullsized":"2024-09-10T19:30:56.163813+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:29:18.504104+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:11:41.458523+0000","last_clean_scrub_stamp":"2024-09-10T19:29:18.504104+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0057679100000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,4],"acting":[1,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":462,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-09-10T19:30:55.154914+0000","last_change":"2024-09-10T19:29:29.071593+0000","last_active":"2024-09-10T19:30:55.154914+0000","last_peered":"2024-09-10T19:30:55.154914+0000","last_clean":"2024-09-10T19:30:55.154914+0000","last_became_active":"2024-09-10T19:24:01.970325+0000","last_became_peered":"2024-09-10T19:24:01.970325+0000","last_unstale":"2024-09-10T19:30:55.154914+0000","last_undegraded":"2024-09-10T19:30:55.154914+0000","last_fullsized":"2024-09-10T19:30:55.154914+0000","mapping_epoch":292,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":293,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:29:29.071516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:24:09.367654+0000","last_clean_scrub_stamp":"2024-09-10T19:29:29.071516+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0048760050000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"66'1","reported_seq":350,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-09-10T19:30:55.160664+0000","last_change":"2024-09-10T19:29:28.371890+0000","last_active":"2024-09-10T19:30:55.160664+0000","last_peered":"2024-09-10T19:30:55.160664+0000","last_clean":"2024-09-10T19:30:55.160664+0000","last_became_active":"2024-09-10T19:24:23.897858+0000","last_became_peered":"2024-09-10T19:24:23.897858+0000","last_unstale":"2024-09-10T19:30:55.160664+0000","last_undegraded":"2024-09-10T19:30:55.160664+0000","last_fullsized":"2024-09-10T19:30:55.160664+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'1","last_scrub_stamp":"2024-09-10T19:29:28.371731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:29:28.371731+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.028433127999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":13,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":88,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-09-10T19:30:56.163977+0000","last_change":"2024-09-10T19:30:34.487341+0000","last_active":"2024-09-10T19:30:56.163977+0000","last_peered":"2024-09-10T19:30:56.163977+0000","last_clean":"2024-09-10T19:30:56.163977+0000","last_became_active":"2024-09-10T19:27:53.007487+0000","last_became_peered":"2024-09-10T19:27:53.007487+0000","last_unstale":"2024-09-10T19:30:56.163977+0000","last_undegraded":"2024-09-10T19:30:56.163977+0000","last_fullsized":"2024-09-10T19:30:56.163977+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:30:34.487271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:30:34.487271+0000","last_clean_scrub_stamp":"2024-09-10T19:30:34.487271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:31:58.958373+0000","scrub_duration":0.007547064,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"414'18","reported_seq":518,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-09-10T19:30:55.155136+0000","last_change":"2024-09-10T19:30:04.390756+0000","last_active":"2024-09-10T19:30:55.155136+0000","last_peered":"2024-09-10T19:30:55.155136+0000","last_clean":"2024-09-10T19:30:55.155136+0000","last_became_active":"2024-09-10T19:24:01.971492+0000","last_became_peered":"2024-09-10T19:24:01.971492+0000","last_unstale":"2024-09-10T19:30:55.155136+0000","last_undegraded":"2024-09-10T19:30:55.155136+0000","last_fullsized":"2024-09-10T19:30:55.155136+0000","mapping_epoch":292,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":293,"parent":"0.0","parent_split_bits":0,"last_scrub":"237'12","last_scrub_stamp":"2024-09-10T19:30:04.390680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:30:04.390680+0000","objects_scrubbed":3,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:31:06.735608+0000","scrub_duration":0.0029448790000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":27,"num_read_kb":24,"num_write":18,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":465,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-09-10T19:30:55.173570+0000","last_change":"2024-09-10T19:30:22.162341+0000","last_active":"2024-09-10T19:30:55.173570+0000","last_peered":"2024-09-10T19:30:55.173570+0000","last_clean":"2024-09-10T19:30:55.173570+0000","last_became_active":"2024-09-10T19:24:01.978215+0000","last_became_peered":"2024-09-10T19:24:01.978215+0000","last_unstale":"2024-09-10T19:30:55.173570+0000","last_undegraded":"2024-09-10T19:30:55.173570+0000","last_fullsized":"2024-09-10T19:30:55.173570+0000","mapping_epoch":292,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":293,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:30:22.162288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:11:23.264793+0000","last_clean_scrub_stamp":"2024-09-10T19:30:22.162288+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:31:34.322359+0000","scrub_duration":0.0055022539999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":466,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-09-10T19:30:55.179207+0000","last_change":"2024-09-10T19:29:17.920965+0000","last_active":"2024-09-10T19:30:55.179207+0000","last_peered":"2024-09-10T19:30:55.179207+0000","last_clean":"2024-09-10T19:30:55.179207+0000","last_became_active":"2024-09-10T19:27:53.010120+0000","last_became_peered":"2024-09-10T19:27:53.010120+0000","last_unstale":"2024-09-10T19:30:55.179207+0000","last_undegraded":"2024-09-10T19:30:55.179207+0000","last_fullsized":"2024-09-10T19:30:55.179207+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:29:17.920880+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:29:17.920880+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0025791519999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"414'35","reported_seq":563,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-09-10T19:30:55.154235+0000","last_change":"2024-09-10T19:30:23.678519+0000","last_active":"2024-09-10T19:30:55.154235+0000","last_peered":"2024-09-10T19:30:55.154235+0000","last_clean":"2024-09-10T19:30:55.154235+0000","last_became_active":"2024-09-10T19:24:01.971173+0000","last_became_peered":"2024-09-10T19:24:01.971173+0000","last_unstale":"2024-09-10T19:30:55.154235+0000","last_undegraded":"2024-09-10T19:30:55.154235+0000","last_fullsized":"2024-09-10T19:30:55.154235+0000","mapping_epoch":292,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":293,"parent":"0.0","parent_split_bits":0,"last_scrub":"414'35","last_scrub_stamp":"2024-09-10T19:30:23.678420+0000","last_deep_scrub":"80'15","last_deep_scrub_stamp":"2024-09-10T19:11:41.719310+0000","last_clean_scrub_stamp":"2024-09-10T19:30:23.678420+0000","objects_scrubbed":5,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:31:41.464166+0000","scrub_duration":0.0043333590000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":45,"num_read_kb":40,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":442,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-09-10T19:30:55.173167+0000","last_change":"2024-09-10T19:29:36.523640+0000","last_active":"2024-09-10T19:30:55.173167+0000","last_peered":"2024-09-10T19:30:55.173167+0000","last_clean":"2024-09-10T19:30:55.173167+0000","last_became_active":"2024-09-10T19:10:07.685870+0000","last_became_peered":"2024-09-10T19:10:07.685870+0000","last_unstale":"2024-09-10T19:30:55.173167+0000","last_undegraded":"2024-09-10T19:30:55.173167+0000","last_fullsized":"2024-09-10T19:30:55.173167+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:29:36.523540+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:29:36.523540+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0026209499999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"372'16","reported_seq":762,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-09-10T19:30:56.167192+0000","last_change":"2024-09-10T19:29:26.417697+0000","last_active":"2024-09-10T19:30:56.167192+0000","last_peered":"2024-09-10T19:30:56.167192+0000","last_clean":"2024-09-10T19:30:56.167192+0000","last_became_active":"2024-09-10T19:27:46.703433+0000","last_became_peered":"2024-09-10T19:27:46.703433+0000","last_unstale":"2024-09-10T19:30:56.167192+0000","last_undegraded":"2024-09-10T19:30:56.167192+0000","last_fullsized":"2024-09-10T19:30:56.167192+0000","mapping_epoch":371,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":372,"parent":"0.0","parent_split_bits":0,"last_scrub":"372'16","last_scrub_stamp":"2024-09-10T19:29:26.415252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:29:26.415252+0000","objects_scrubbed":1,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.007524256,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,0],"acting":[5,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":396,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-09-10T19:30:55.155891+0000","last_change":"2024-09-10T19:29:34.578620+0000","last_active":"2024-09-10T19:30:55.155891+0000","last_peered":"2024-09-10T19:30:55.155891+0000","last_clean":"2024-09-10T19:30:55.155891+0000","last_became_active":"2024-09-10T19:24:23.900513+0000","last_became_peered":"2024-09-10T19:24:23.900513+0000","last_unstale":"2024-09-10T19:30:55.155891+0000","last_undegraded":"2024-09-10T19:30:55.155891+0000","last_fullsized":"2024-09-10T19:30:55.155891+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:29:34.578469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:29:34.578469+0000","last_clean_scrub_stamp":"2024-09-10T19:29:34.578469+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0058680599999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"414'21","reported_seq":514,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-09-10T19:30:55.177077+0000","last_change":"2024-09-10T19:29:32.382258+0000","last_active":"2024-09-10T19:30:55.177077+0000","last_peered":"2024-09-10T19:30:55.177077+0000","last_clean":"2024-09-10T19:30:55.177077+0000","last_became_active":"2024-09-10T19:16:24.859740+0000","last_became_peered":"2024-09-10T19:16:24.859740+0000","last_unstale":"2024-09-10T19:30:55.177077+0000","last_undegraded":"2024-09-10T19:30:55.177077+0000","last_fullsized":"2024-09-10T19:30:55.177077+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"237'15","last_scrub_stamp":"2024-09-10T19:29:32.382176+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:29:32.382176+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0057339599999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":399,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-09-10T19:30:55.156369+0000","last_change":"2024-09-10T19:29:11.750820+0000","last_active":"2024-09-10T19:30:55.156369+0000","last_peered":"2024-09-10T19:30:55.156369+0000","last_clean":"2024-09-10T19:30:55.156369+0000","last_became_active":"2024-09-10T19:24:23.895163+0000","last_became_peered":"2024-09-10T19:24:23.895163+0000","last_unstale":"2024-09-10T19:30:55.156369+0000","last_undegraded":"2024-09-10T19:30:55.156369+0000","last_fullsized":"2024-09-10T19:30:55.156369+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:29:11.750709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:29:11.750709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0029571649999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"297'11","reported_seq":726,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-09-10T19:30:56.170806+0000","last_change":"2024-09-10T19:29:11.159023+0000","last_active":"2024-09-10T19:30:56.170806+0000","last_peered":"2024-09-10T19:30:56.170806+0000","last_clean":"2024-09-10T19:30:56.170806+0000","last_became_active":"2024-09-10T19:24:01.974816+0000","last_became_peered":"2024-09-10T19:24:01.974816+0000","last_unstale":"2024-09-10T19:30:56.170806+0000","last_undegraded":"2024-09-10T19:30:56.170806+0000","last_fullsized":"2024-09-10T19:30:56.170806+0000","mapping_epoch":292,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":293,"parent":"0.0","parent_split_bits":0,"last_scrub":"297'11","last_scrub_stamp":"2024-09-10T19:29:11.158902+0000","last_deep_scrub":"297'11","last_deep_scrub_stamp":"2024-09-10T19:24:13.318130+0000","last_clean_scrub_stamp":"2024-09-10T19:29:11.158902+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.014823852,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":455,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-09-10T19:30:55.161109+0000","last_change":"2024-09-10T19:29:32.361240+0000","last_active":"2024-09-10T19:30:55.161109+0000","last_peered":"2024-09-10T19:30:55.161109+0000","last_clean":"2024-09-10T19:30:55.161109+0000","last_became_active":"2024-09-10T19:24:23.903779+0000","last_became_peered":"2024-09-10T19:24:23.903779+0000","last_unstale":"2024-09-10T19:30:55.161109+0000","last_undegraded":"2024-09-10T19:30:55.161109+0000","last_fullsized":"2024-09-10T19:30:55.161109+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:29:32.361029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:29:32.361029+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:30:58.523472+0000","scrub_duration":0.010203366,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.12","version":"414'25","reported_seq":243,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-09-10T19:30:56.160756+0000","last_change":"2024-09-10T19:30:23.274575+0000","last_active":"2024-09-10T19:30:56.160756+0000","last_peered":"2024-09-10T19:30:56.160756+0000","last_clean":"2024-09-10T19:30:56.160756+0000","last_became_active":"2024-09-10T19:27:53.017161+0000","last_became_peered":"2024-09-10T19:27:53.017161+0000","last_unstale":"2024-09-10T19:30:56.160756+0000","last_undegraded":"2024-09-10T19:30:56.160756+0000","last_fullsized":"2024-09-10T19:30:56.160756+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"414'25","last_scrub_stamp":"2024-09-10T19:30:23.274493+0000","last_deep_scrub":"79'13","last_deep_scrub_stamp":"2024-09-10T19:11:22.072400+0000","last_clean_scrub_stamp":"2024-09-10T19:30:23.274493+0000","objects_scrubbed":4,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:31:42.647014+0000","scrub_duration":0.0027214610000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":34,"num_read_kb":28,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":550,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"59'1","reported_seq":92,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-09-10T19:30:56.165107+0000","last_change":"2024-09-10T19:30:31.402122+0000","last_active":"2024-09-10T19:30:56.165107+0000","last_peered":"2024-09-10T19:30:56.165107+0000","last_clean":"2024-09-10T19:30:56.165107+0000","last_became_active":"2024-09-10T19:27:53.018470+0000","last_became_peered":"2024-09-10T19:27:53.018470+0000","last_unstale":"2024-09-10T19:30:56.165107+0000","last_undegraded":"2024-09-10T19:30:56.165107+0000","last_fullsized":"2024-09-10T19:30:56.165107+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'1","last_scrub_stamp":"2024-09-10T19:30:31.401664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:30:31.401664+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:31:59.702734+0000","scrub_duration":0.0059519009999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":362,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":724,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":476,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-09-10T19:30:55.154396+0000","last_change":"2024-09-10T19:29:19.145340+0000","last_active":"2024-09-10T19:30:55.154396+0000","last_peered":"2024-09-10T19:30:55.154396+0000","last_clean":"2024-09-10T19:30:55.154396+0000","last_became_active":"2024-09-10T19:24:24.923804+0000","last_became_peered":"2024-09-10T19:24:24.923804+0000","last_unstale":"2024-09-10T19:30:55.154396+0000","last_undegraded":"2024-09-10T19:30:55.154396+0000","last_fullsized":"2024-09-10T19:30:55.154396+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:29:19.145160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:29:19.145160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0019117959999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":415,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-09-10T19:30:55.153457+0000","last_change":"2024-09-10T19:29:28.581248+0000","last_active":"2024-09-10T19:30:55.153457+0000","last_peered":"2024-09-10T19:30:55.153457+0000","last_clean":"2024-09-10T19:30:55.153457+0000","last_became_active":"2024-09-10T19:24:24.916105+0000","last_became_peered":"2024-09-10T19:24:24.916105+0000","last_unstale":"2024-09-10T19:30:55.153457+0000","last_undegraded":"2024-09-10T19:30:55.153457+0000","last_fullsized":"2024-09-10T19:30:55.153457+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:29:28.581177+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:25:27.412579+0000","last_clean_scrub_stamp":"2024-09-10T19:29:28.581177+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0049057299999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"414'27","reported_seq":580,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-09-10T19:30:55.160626+0000","last_change":"2024-09-10T19:29:26.356083+0000","last_active":"2024-09-10T19:30:55.160626+0000","last_peered":"2024-09-10T19:30:55.160626+0000","last_clean":"2024-09-10T19:30:55.160626+0000","last_became_active":"2024-09-10T19:27:53.016518+0000","last_became_peered":"2024-09-10T19:27:53.016518+0000","last_unstale":"2024-09-10T19:30:55.160626+0000","last_undegraded":"2024-09-10T19:30:55.160626+0000","last_fullsized":"2024-09-10T19:30:55.160626+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"237'19","last_scrub_stamp":"2024-09-10T19:29:26.355998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:29:26.355998+0000","objects_scrubbed":5,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.030792693999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":37,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":184,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-09-10T19:30:56.161200+0000","last_change":"2024-09-10T19:29:09.345677+0000","last_active":"2024-09-10T19:30:56.161200+0000","last_peered":"2024-09-10T19:30:56.161200+0000","last_clean":"2024-09-10T19:30:56.161200+0000","last_became_active":"2024-09-10T19:24:24.909540+0000","last_became_peered":"2024-09-10T19:24:24.909540+0000","last_unstale":"2024-09-10T19:30:56.161200+0000","last_undegraded":"2024-09-10T19:30:56.161200+0000","last_fullsized":"2024-09-10T19:30:56.161200+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:29:09.345588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:26:31.508168+0000","last_clean_scrub_stamp":"2024-09-10T19:29:09.345588+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.008652847,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":160,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-09-10T19:30:56.160830+0000","last_change":"2024-09-10T19:28:29.495516+0000","last_active":"2024-09-10T19:30:56.160830+0000","last_peered":"2024-09-10T19:30:56.160830+0000","last_clean":"2024-09-10T19:30:56.160830+0000","last_became_active":"2024-09-10T19:24:01.975520+0000","last_became_peered":"2024-09-10T19:24:01.975520+0000","last_unstale":"2024-09-10T19:30:56.160830+0000","last_undegraded":"2024-09-10T19:30:56.160830+0000","last_fullsized":"2024-09-10T19:30:56.160830+0000","mapping_epoch":292,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":293,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:28:29.495437+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:28:29.495437+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0066786709999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":443,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-09-10T19:30:55.154679+0000","last_change":"2024-09-10T19:29:33.122658+0000","last_active":"2024-09-10T19:30:55.154679+0000","last_peered":"2024-09-10T19:30:55.154679+0000","last_clean":"2024-09-10T19:30:55.154679+0000","last_became_active":"2024-09-10T19:24:23.907136+0000","last_became_peered":"2024-09-10T19:24:23.907136+0000","last_unstale":"2024-09-10T19:30:55.154679+0000","last_undegraded":"2024-09-10T19:30:55.154679+0000","last_fullsized":"2024-09-10T19:30:55.154679+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:29:33.121149+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:29:33.121149+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:31:00.857785+0000","scrub_duration":0.0077727279999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"414'14","reported_seq":531,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-09-10T19:30:55.154255+0000","last_change":"2024-09-10T19:30:06.362408+0000","last_active":"2024-09-10T19:30:55.154255+0000","last_peered":"2024-09-10T19:30:55.154255+0000","last_clean":"2024-09-10T19:30:55.154255+0000","last_became_active":"2024-09-10T19:27:53.024782+0000","last_became_peered":"2024-09-10T19:27:53.024782+0000","last_unstale":"2024-09-10T19:30:55.154255+0000","last_undegraded":"2024-09-10T19:30:55.154255+0000","last_fullsized":"2024-09-10T19:30:55.154255+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"237'10","last_scrub_stamp":"2024-09-10T19:30:06.361528+0000","last_deep_scrub":"222'6","last_deep_scrub_stamp":"2024-09-10T19:19:30.328841+0000","last_clean_scrub_stamp":"2024-09-10T19:30:06.361528+0000","objects_scrubbed":2,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:31:24.920334+0000","scrub_duration":0.0072268020000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":18,"num_read_kb":16,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":492,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-09-10T19:30:55.174621+0000","last_change":"2024-09-10T19:30:20.146538+0000","last_active":"2024-09-10T19:30:55.174621+0000","last_peered":"2024-09-10T19:30:55.174621+0000","last_clean":"2024-09-10T19:30:55.174621+0000","last_became_active":"2024-09-10T19:27:53.000789+0000","last_became_peered":"2024-09-10T19:27:53.000789+0000","last_unstale":"2024-09-10T19:30:55.174621+0000","last_undegraded":"2024-09-10T19:30:55.174621+0000","last_fullsized":"2024-09-10T19:30:55.174621+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:30:20.146470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:30:20.146470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:31:21.242258+0000","scrub_duration":0.0032897199999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,1],"acting":[5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":88,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-09-10T19:30:56.164057+0000","last_change":"2024-09-10T19:30:32.432843+0000","last_active":"2024-09-10T19:30:56.164057+0000","last_peered":"2024-09-10T19:30:56.164057+0000","last_clean":"2024-09-10T19:30:56.164057+0000","last_became_active":"2024-09-10T19:27:53.003591+0000","last_became_peered":"2024-09-10T19:27:53.003591+0000","last_unstale":"2024-09-10T19:30:56.164057+0000","last_undegraded":"2024-09-10T19:30:56.164057+0000","last_fullsized":"2024-09-10T19:30:56.164057+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:30:32.432739+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:11:31.837944+0000","last_clean_scrub_stamp":"2024-09-10T19:30:32.432739+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:31:40.637363+0000","scrub_duration":0.010234006,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":466,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-09-10T19:30:55.161742+0000","last_change":"2024-09-10T19:29:33.126317+0000","last_active":"2024-09-10T19:30:55.161742+0000","last_peered":"2024-09-10T19:30:55.161742+0000","last_clean":"2024-09-10T19:30:55.161742+0000","last_became_active":"2024-09-10T19:16:24.852356+0000","last_became_peered":"2024-09-10T19:16:24.852356+0000","last_unstale":"2024-09-10T19:30:55.161742+0000","last_undegraded":"2024-09-10T19:30:55.161742+0000","last_fullsized":"2024-09-10T19:30:55.161742+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:29:33.125081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:29:33.125081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.019182421000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,3],"acting":[7,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.11","version":"414'27","reported_seq":288,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-09-10T19:30:55.154879+0000","last_change":"2024-09-10T19:29:15.225773+0000","last_active":"2024-09-10T19:30:55.154879+0000","last_peered":"2024-09-10T19:30:55.154879+0000","last_clean":"2024-09-10T19:30:55.154879+0000","last_became_active":"2024-09-10T19:27:53.018213+0000","last_became_peered":"2024-09-10T19:27:53.018213+0000","last_unstale":"2024-09-10T19:30:55.154879+0000","last_undegraded":"2024-09-10T19:30:55.154879+0000","last_fullsized":"2024-09-10T19:30:55.154879+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"237'19","last_scrub_stamp":"2024-09-10T19:29:15.225687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:29:15.225687+0000","objects_scrubbed":5,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.015395733,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":37,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":78,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-09-10T19:30:56.164847+0000","last_change":"2024-09-10T19:30:10.528365+0000","last_active":"2024-09-10T19:30:56.164847+0000","last_peered":"2024-09-10T19:30:56.164847+0000","last_clean":"2024-09-10T19:30:56.164847+0000","last_became_active":"2024-09-10T19:27:53.008197+0000","last_became_peered":"2024-09-10T19:27:53.008197+0000","last_unstale":"2024-09-10T19:30:56.164847+0000","last_undegraded":"2024-09-10T19:30:56.164847+0000","last_fullsized":"2024-09-10T19:30:56.164847+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:30:10.528284+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:13:05.876655+0000","last_clean_scrub_stamp":"2024-09-10T19:30:10.528284+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:31:10.861466+0000","scrub_duration":0.002704536,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":468,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-09-10T19:30:55.177393+0000","last_change":"2024-09-10T19:30:11.797823+0000","last_active":"2024-09-10T19:30:55.177393+0000","last_peered":"2024-09-10T19:30:55.177393+0000","last_clean":"2024-09-10T19:30:55.177393+0000","last_became_active":"2024-09-10T19:27:53.008444+0000","last_became_peered":"2024-09-10T19:27:53.008444+0000","last_unstale":"2024-09-10T19:30:55.177393+0000","last_undegraded":"2024-09-10T19:30:55.177393+0000","last_fullsized":"2024-09-10T19:30:55.177393+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:30:11.797669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:11:57.112092+0000","last_clean_scrub_stamp":"2024-09-10T19:30:11.797669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:31:28.693832+0000","scrub_duration":0.002187483,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":463,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-09-10T19:30:55.153990+0000","last_change":"2024-09-10T19:29:30.096377+0000","last_active":"2024-09-10T19:30:55.153990+0000","last_peered":"2024-09-10T19:30:55.153990+0000","last_clean":"2024-09-10T19:30:55.153990+0000","last_became_active":"2024-09-10T19:27:53.025650+0000","last_became_peered":"2024-09-10T19:27:53.025650+0000","last_unstale":"2024-09-10T19:30:55.153990+0000","last_undegraded":"2024-09-10T19:30:55.153990+0000","last_fullsized":"2024-09-10T19:30:55.153990+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:29:30.096297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:26:29.959202+0000","last_clean_scrub_stamp":"2024-09-10T19:29:30.096297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.012023613000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"80'2","reported_seq":472,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-09-10T19:30:55.178239+0000","last_change":"2024-09-10T19:30:31.056345+0000","last_active":"2024-09-10T19:30:55.178239+0000","last_peered":"2024-09-10T19:30:55.178239+0000","last_clean":"2024-09-10T19:30:55.178239+0000","last_became_active":"2024-09-10T19:16:24.851220+0000","last_became_peered":"2024-09-10T19:16:24.851220+0000","last_unstale":"2024-09-10T19:30:55.178239+0000","last_undegraded":"2024-09-10T19:30:55.178239+0000","last_fullsized":"2024-09-10T19:30:55.178239+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"80'2","last_scrub_stamp":"2024-09-10T19:30:31.056271+0000","last_deep_scrub":"80'2","last_deep_scrub_stamp":"2024-09-10T19:11:30.272043+0000","last_clean_scrub_stamp":"2024-09-10T19:30:31.056271+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:31:41.406250+0000","scrub_duration":0.0025698990000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":369,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":369,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,3],"acting":[7,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":459,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-09-10T19:30:55.174322+0000","last_change":"2024-09-10T19:29:18.277593+0000","last_active":"2024-09-10T19:30:55.174322+0000","last_peered":"2024-09-10T19:30:55.174322+0000","last_clean":"2024-09-10T19:30:55.174322+0000","last_became_active":"2024-09-10T19:24:23.906812+0000","last_became_peered":"2024-09-10T19:24:23.906812+0000","last_unstale":"2024-09-10T19:30:55.174322+0000","last_undegraded":"2024-09-10T19:30:55.174322+0000","last_fullsized":"2024-09-10T19:30:55.174322+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:29:18.277508+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:29:18.277508+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.007405455,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,0],"acting":[5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1e","version":"414'35","reported_seq":450,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-09-10T19:30:55.155804+0000","last_change":"2024-09-10T19:29:20.656771+0000","last_active":"2024-09-10T19:30:55.155804+0000","last_peered":"2024-09-10T19:30:55.155804+0000","last_clean":"2024-09-10T19:30:55.155804+0000","last_became_active":"2024-09-10T19:24:23.899507+0000","last_became_peered":"2024-09-10T19:24:23.899507+0000","last_unstale":"2024-09-10T19:30:55.155804+0000","last_undegraded":"2024-09-10T19:30:55.155804+0000","last_fullsized":"2024-09-10T19:30:55.155804+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"237'25","last_scrub_stamp":"2024-09-10T19:29:20.656690+0000","last_deep_scrub":"80'15","last_deep_scrub_stamp":"2024-09-10T19:11:25.883919+0000","last_clean_scrub_stamp":"2024-09-10T19:29:20.656690+0000","objects_scrubbed":5,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0034048949999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":39,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1f","version":"376'10","reported_seq":121,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-09-10T19:31:05.980153+0000","last_change":"2024-09-10T19:30:22.536949+0000","last_active":"2024-09-10T19:31:05.980153+0000","last_peered":"2024-09-10T19:31:05.980153+0000","last_clean":"2024-09-10T19:31:05.980153+0000","last_became_active":"2024-09-10T19:27:53.003676+0000","last_became_peered":"2024-09-10T19:27:53.003676+0000","last_unstale":"2024-09-10T19:31:05.980153+0000","last_undegraded":"2024-09-10T19:31:05.980153+0000","last_fullsized":"2024-09-10T19:31:05.980153+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"376'10","last_scrub_stamp":"2024-09-10T19:30:22.536865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:30:22.536865+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:31:49.099743+0000","scrub_duration":0.0053077080000000004,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":170,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-09-10T19:30:56.161347+0000","last_change":"2024-09-10T19:30:25.316849+0000","last_active":"2024-09-10T19:30:56.161347+0000","last_peered":"2024-09-10T19:30:56.161347+0000","last_clean":"2024-09-10T19:30:56.161347+0000","last_became_active":"2024-09-10T19:24:01.980391+0000","last_became_peered":"2024-09-10T19:24:01.980391+0000","last_unstale":"2024-09-10T19:30:56.161347+0000","last_undegraded":"2024-09-10T19:30:56.161347+0000","last_fullsized":"2024-09-10T19:30:56.161347+0000","mapping_epoch":292,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":293,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:30:25.316711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:28:20.514115+0000","last_clean_scrub_stamp":"2024-09-10T19:30:25.316711+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:31:48.741913+0000","scrub_duration":0.0018401960000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":479,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-09-10T19:30:55.153840+0000","last_change":"2024-09-10T19:29:34.161440+0000","last_active":"2024-09-10T19:30:55.153840+0000","last_peered":"2024-09-10T19:30:55.153840+0000","last_clean":"2024-09-10T19:30:55.153840+0000","last_became_active":"2024-09-10T19:27:53.017153+0000","last_became_peered":"2024-09-10T19:27:53.017153+0000","last_unstale":"2024-09-10T19:30:55.153840+0000","last_undegraded":"2024-09-10T19:30:55.153840+0000","last_fullsized":"2024-09-10T19:30:55.153840+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:29:34.161296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:29:34.161296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.005093261,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"414'32","reported_seq":102,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-09-10T19:30:56.164775+0000","last_change":"2024-09-10T19:29:28.357694+0000","last_active":"2024-09-10T19:30:56.164775+0000","last_peered":"2024-09-10T19:30:56.164775+0000","last_clean":"2024-09-10T19:30:56.164775+0000","last_became_active":"2024-09-10T19:27:53.008233+0000","last_became_peered":"2024-09-10T19:27:53.008233+0000","last_unstale":"2024-09-10T19:30:56.164775+0000","last_undegraded":"2024-09-10T19:30:56.164775+0000","last_fullsized":"2024-09-10T19:30:56.164775+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"237'24","last_scrub_stamp":"2024-09-10T19:29:28.357607+0000","last_deep_scrub":"80'16","last_deep_scrub_stamp":"2024-09-10T19:11:40.853611+0000","last_clean_scrub_stamp":"2024-09-10T19:29:28.357607+0000","objects_scrubbed":5,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0067797259999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":36,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":387,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-09-10T19:30:55.155725+0000","last_change":"2024-09-10T19:30:24.834659+0000","last_active":"2024-09-10T19:30:55.155725+0000","last_peered":"2024-09-10T19:30:55.155725+0000","last_clean":"2024-09-10T19:30:55.155725+0000","last_became_active":"2024-09-10T19:24:24.899153+0000","last_became_peered":"2024-09-10T19:24:24.899153+0000","last_unstale":"2024-09-10T19:30:55.155725+0000","last_undegraded":"2024-09-10T19:30:55.155725+0000","last_fullsized":"2024-09-10T19:30:55.155725+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:30:24.834494+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:28:19.464426+0000","last_clean_scrub_stamp":"2024-09-10T19:30:24.834494+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:31:26.567702+0000","scrub_duration":0.0076205279999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":475,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-09-10T19:30:55.163875+0000","last_change":"2024-09-10T19:30:30.520734+0000","last_active":"2024-09-10T19:30:55.163875+0000","last_peered":"2024-09-10T19:30:55.163875+0000","last_clean":"2024-09-10T19:30:55.163875+0000","last_became_active":"2024-09-10T19:27:53.011336+0000","last_became_peered":"2024-09-10T19:27:53.011336+0000","last_unstale":"2024-09-10T19:30:55.163875+0000","last_undegraded":"2024-09-10T19:30:55.163875+0000","last_fullsized":"2024-09-10T19:30:55.163875+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:30:30.520643+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:30:30.520643+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:31:57.292887+0000","scrub_duration":0.018058100000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":78,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-09-10T19:30:56.164851+0000","last_change":"2024-09-10T19:29:33.401343+0000","last_active":"2024-09-10T19:30:56.164851+0000","last_peered":"2024-09-10T19:30:56.164851+0000","last_clean":"2024-09-10T19:30:56.164851+0000","last_became_active":"2024-09-10T19:27:53.002313+0000","last_became_peered":"2024-09-10T19:27:53.002313+0000","last_unstale":"2024-09-10T19:30:56.164851+0000","last_undegraded":"2024-09-10T19:30:56.164851+0000","last_fullsized":"2024-09-10T19:30:56.164851+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:29:33.401164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:29:33.401164+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.016867561,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1c","version":"414'46","reported_seq":218,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-09-10T19:30:56.161644+0000","last_change":"2024-09-10T19:29:34.532076+0000","last_active":"2024-09-10T19:30:56.161644+0000","last_peered":"2024-09-10T19:30:56.161644+0000","last_clean":"2024-09-10T19:30:56.161644+0000","last_became_active":"2024-09-10T19:24:24.910577+0000","last_became_peered":"2024-09-10T19:24:24.910577+0000","last_unstale":"2024-09-10T19:30:56.161644+0000","last_undegraded":"2024-09-10T19:30:56.161644+0000","last_fullsized":"2024-09-10T19:30:56.161644+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"237'34","last_scrub_stamp":"2024-09-10T19:29:34.531993+0000","last_deep_scrub":"78'22","last_deep_scrub_stamp":"2024-09-10T19:11:21.118937+0000","last_clean_scrub_stamp":"2024-09-10T19:29:34.531993+0000","objects_scrubbed":7,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0018029159999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":62,"num_read_kb":53,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,0],"acting":[1,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":184,"reported_epoch":422,"state":"active+clean","last_fresh":"2024-09-10T19:30:56.161470+0000","last_change":"2024-09-10T19:29:21.494875+0000","last_active":"2024-09-10T19:30:56.161470+0000","last_peered":"2024-09-10T19:30:56.161470+0000","last_clean":"2024-09-10T19:30:56.161470+0000","last_became_active":"2024-09-10T19:24:24.887205+0000","last_became_peered":"2024-09-10T19:24:24.887205+0000","last_unstale":"2024-09-10T19:30:56.161470+0000","last_undegraded":"2024-09-10T19:30:56.161470+0000","last_fullsized":"2024-09-10T19:30:56.161470+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:29:21.494801+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:29:21.494801+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.011816722,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":459,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-09-10T19:30:55.154800+0000","last_change":"2024-09-10T19:29:25.182733+0000","last_active":"2024-09-10T19:30:55.154800+0000","last_peered":"2024-09-10T19:30:55.154800+0000","last_clean":"2024-09-10T19:30:55.154800+0000","last_became_active":"2024-09-10T19:27:53.026591+0000","last_became_peered":"2024-09-10T19:27:53.026591+0000","last_unstale":"2024-09-10T19:30:55.154800+0000","last_undegraded":"2024-09-10T19:30:55.154800+0000","last_fullsized":"2024-09-10T19:30:55.154800+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:29:25.182654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:24:46.518816+0000","last_clean_scrub_stamp":"2024-09-10T19:29:25.182654+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0060637180000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":472,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-09-10T19:30:55.154639+0000","last_change":"2024-09-10T19:30:24.721276+0000","last_active":"2024-09-10T19:30:55.154639+0000","last_peered":"2024-09-10T19:30:55.154639+0000","last_clean":"2024-09-10T19:30:55.154639+0000","last_became_active":"2024-09-10T19:27:53.018726+0000","last_became_peered":"2024-09-10T19:27:53.018726+0000","last_unstale":"2024-09-10T19:30:55.154639+0000","last_undegraded":"2024-09-10T19:30:55.154639+0000","last_fullsized":"2024-09-10T19:30:55.154639+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:30:24.721140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:30:24.721140+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:31:33.177185+0000","scrub_duration":0.0073255109999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1d","version":"414'29","reported_seq":537,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-09-10T19:30:55.154579+0000","last_change":"2024-09-10T19:30:28.765125+0000","last_active":"2024-09-10T19:30:55.154579+0000","last_peered":"2024-09-10T19:30:55.154579+0000","last_clean":"2024-09-10T19:30:55.154579+0000","last_became_active":"2024-09-10T19:24:01.976664+0000","last_became_peered":"2024-09-10T19:24:01.976664+0000","last_unstale":"2024-09-10T19:30:55.154579+0000","last_undegraded":"2024-09-10T19:30:55.154579+0000","last_fullsized":"2024-09-10T19:30:55.154579+0000","mapping_epoch":292,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":293,"parent":"0.0","parent_split_bits":0,"last_scrub":"414'29","last_scrub_stamp":"2024-09-10T19:30:28.765025+0000","last_deep_scrub":"250'23","last_deep_scrub_stamp":"2024-09-10T19:20:55.582173+0000","last_clean_scrub_stamp":"2024-09-10T19:30:28.765025+0000","objects_scrubbed":5,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:31:57.061451+0000","scrub_duration":0.012181088,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":467,"reported_epoch":421,"state":"active+clean","last_fresh":"2024-09-10T19:30:55.160789+0000","last_change":"2024-09-10T19:30:39.493327+0000","last_active":"2024-09-10T19:30:55.160789+0000","last_peered":"2024-09-10T19:30:55.160789+0000","last_clean":"2024-09-10T19:30:55.160789+0000","last_became_active":"2024-09-10T19:16:24.843013+0000","last_became_peered":"2024-09-10T19:16:24.843013+0000","last_unstale":"2024-09-10T19:30:55.160789+0000","last_undegraded":"2024-09-10T19:30:55.160789+0000","last_fullsized":"2024-09-10T19:30:55.160789+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:30:39.493257+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:30:39.493257+0000","last_clean_scrub_stamp":"2024-09-10T19:30:39.493257+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:32:02.390742+0000","scrub_duration":0.0032639090000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":11,"num_pg":16,"stat_sum":{"num_bytes":117702680,"num_objects":1797,"num_object_clones":0,"num_object_copies":5391,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1797,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":287979,"num_write_kb":9272769,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":257101824,"data_stored":257097752,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":70205,"ondisk_log_size":70205,"up":48,"acting":48,"num_store_stats":7},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":382,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":1528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":96,"ondisk_log_size":96,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":177,"num_object_clones":0,"num_object_copies":531,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":177,"num_whiteouts":0,"num_read":1544,"num_read_kb":1301,"num_write":989,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":433,"num_bytes_recovered":8406,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":458752,"data_stored":12124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1066,"ondisk_log_size":1066,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":427,"num_read_kb":422,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1594,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1613,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":10,"ondisk_log_size":10,"up":9,"acting":9,"num_store_stats":7},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1328,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":21,"num_read_kb":21,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":724,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":3984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5,"ondisk_log_size":5,"up":96,"acting":96,"num_store_stats":8},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":525376,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":870,"ondisk_log_size":870,"up":33,"acting":33,"num_store_stats":7}],"osd_stats":[{"osd":7,"up_from":54,"seq":231928234240,"num_pgs":71,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1292452,"kb_used_data":107420,"kb_used_omap":1,"kb_used_meta":1184830,"kb_avail":92452700,"statfs":{"total":95995035648,"available":94671564800,"internally_reserved":0,"allocated":109998080,"data_stored":109701870,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":1213266264},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 19:30:32 2024","interfaces":[{"interface":"back","average":{"1min":1.2030000000000001,"5min":0.98599999999999999,"15min":0.97499999999999998},"min":{"1min":0.30499999999999999,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":3.27,"5min":5.0510000000000002,"15min":5.0510000000000002},"last":1.5780000000000001},{"interface":"front","average":{"1min":1.304,"5min":1.103,"15min":0.82699999999999996},"min":{"1min":0.23400000000000001,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":3.2570000000000001,"5min":4.7190000000000003,"15min":4.7190000000000003},"last":0.33800000000000002}]},{"osd":1,"last update":"Tue Sep 10 19:31:02 2024","interfaces":[{"interface":"back","average":{"1min":0.80500000000000005,"5min":1.22,"15min":1.0720000000000001},"min":{"1min":0.224,"5min":0.224,"15min":0.224},"max":{"1min":1.8819999999999999,"5min":12.773999999999999,"15min":12.773999999999999},"last":1.6639999999999999},{"interface":"front","average":{"1min":1.143,"5min":1.2350000000000001,"15min":0.98799999999999999},"min":{"1min":0.29599999999999999,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":2.952,"5min":12.837999999999999,"15min":12.837999999999999},"last":2.952}]},{"osd":2,"last update":"Tue Sep 10 19:30:19 2024","interfaces":[{"interface":"back","average":{"1min":1.3879999999999999,"5min":0.996,"15min":0.92300000000000004},"min":{"1min":0.26600000000000001,"5min":0.26600000000000001,"15min":0.23000000000000001},"max":{"1min":3.3100000000000001,"5min":4.7400000000000002,"15min":4.7400000000000002},"last":0.36299999999999999},{"interface":"front","average":{"1min":1.327,"5min":1.1060000000000001,"15min":1.04},"min":{"1min":0.37,"5min":0.29099999999999998,"15min":0.26300000000000001},"max":{"1min":6.1059999999999999,"5min":6.1059999999999999,"15min":8.0839999999999996},"last":0.69399999999999995}]},{"osd":3,"last update":"Tue Sep 10 19:30:19 2024","interfaces":[{"interface":"back","average":{"1min":2.0640000000000001,"5min":1.3720000000000001,"15min":1.204},"min":{"1min":0.32800000000000001,"5min":0.25900000000000001,"15min":0.20200000000000001},"max":{"1min":14.515000000000001,"5min":14.515000000000001,"15min":21.074999999999999},"last":0.42199999999999999},{"interface":"front","average":{"1min":1.915,"5min":1.3560000000000001,"15min":1.1839999999999999},"min":{"1min":0.374,"5min":0.27700000000000002,"15min":0.25700000000000001},"max":{"1min":14.227,"5min":14.227,"15min":18.82},"last":0.67200000000000004}]},{"osd":4,"last update":"Tue Sep 10 19:30:19 2024","interfaces":[{"interface":"back","average":{"1min":1.496,"5min":1.716,"15min":2.677},"min":{"1min":0.35399999999999998,"5min":0.24399999999999999,"15min":0.222},"max":{"1min":5.194,"5min":18.664999999999999,"15min":231.923},"last":1.6180000000000001},{"interface":"front","average":{"1min":1.488,"5min":1.466,"15min":2.3479999999999999},"min":{"1min":0.29599999999999999,"5min":0.25600000000000001,"15min":0.23100000000000001},"max":{"1min":6.6029999999999998,"5min":14.837,"15min":231.322},"last":0.309}]},{"osd":5,"last update":"Tue Sep 10 19:30:19 2024","interfaces":[{"interface":"back","average":{"1min":1.1639999999999999,"5min":1.1850000000000001,"15min":1.6339999999999999},"min":{"1min":0.39000000000000001,"5min":0.25800000000000001,"15min":0.24099999999999999},"max":{"1min":3.5750000000000002,"5min":8.8300000000000001,"15min":67.382999999999996},"last":0.64800000000000002},{"interface":"front","average":{"1min":1.77,"5min":1.5549999999999999,"15min":1.99},"min":{"1min":0.33700000000000002,"5min":0.309,"15min":0.22800000000000001},"max":{"1min":6.7939999999999996,"5min":8.7040000000000006,"15min":67.75},"last":1.6419999999999999}]},{"osd":6,"last update":"Tue Sep 10 19:30:19 2024","interfaces":[{"interface":"back","average":{"1min":2.0139999999999998,"5min":1.5980000000000001,"15min":2.4079999999999999},"min":{"1min":0.29799999999999999,"5min":0.29799999999999999,"15min":0.28000000000000003},"max":{"1min":6.7530000000000001,"5min":6.7869999999999999,"15min":181.53800000000001},"last":1.706},{"interface":"front","average":{"1min":1.591,"5min":1.393,"15min":2.101},"min":{"1min":0.34399999999999997,"5min":0.318,"15min":0.27000000000000002},"max":{"1min":6.6349999999999998,"5min":9.7110000000000003,"15min":182.05600000000001},"last":0.38700000000000001}]}]},{"osd":6,"up_from":47,"seq":201863463174,"num_pgs":66,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":895808,"kb_used_data":1012,"kb_used_omap":1,"kb_used_meta":894782,"kb_avail":92849344,"statfs":{"total":95995035648,"available":95077728256,"internally_reserved":0,"allocated":1036288,"data_stored":724885,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":916257112},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 19:30:39 2024","interfaces":[{"interface":"back","average":{"1min":7.5650000000000004,"5min":2.6040000000000001,"15min":2.117},"min":{"1min":0.26800000000000002,"5min":0.183,"15min":0.17899999999999999},"max":{"1min":127.395,"5min":127.395,"15min":127.395},"last":0.61099999999999999},{"interface":"front","average":{"1min":7.6219999999999999,"5min":2.5390000000000001,"15min":2.1789999999999998},"min":{"1min":0.20599999999999999,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":127.547,"5min":127.547,"15min":127.547},"last":0.41299999999999998}]},{"osd":1,"last update":"Tue Sep 10 19:31:04 2024","interfaces":[{"interface":"back","average":{"1min":4.4269999999999996,"5min":2.1379999999999999,"15min":1.248},"min":{"1min":0.23499999999999999,"5min":0.20999999999999999,"15min":0.19800000000000001},"max":{"1min":39.484999999999999,"5min":39.484999999999999,"15min":39.484999999999999},"last":0.51300000000000001},{"interface":"front","average":{"1min":4.6909999999999998,"5min":2.0760000000000001,"15min":1.1950000000000001},"min":{"1min":0.246,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":39.472999999999999,"5min":39.472999999999999,"15min":39.472999999999999},"last":0.52800000000000002}]},{"osd":2,"last update":"Tue Sep 10 19:30:10 2024","interfaces":[{"interface":"back","average":{"1min":1.258,"5min":1.1459999999999999,"15min":0.91400000000000003},"min":{"1min":0.24299999999999999,"5min":0.24299999999999999,"15min":0.23799999999999999},"max":{"1min":4.8330000000000002,"5min":5.8010000000000002,"15min":5.9029999999999996},"last":0.45300000000000001},{"interface":"front","average":{"1min":1.218,"5min":1.1259999999999999,"15min":0.95999999999999996},"min":{"1min":0.36799999999999999,"5min":0.25600000000000001,"15min":0.254},"max":{"1min":5.6929999999999996,"5min":6.4560000000000004,"15min":6.4560000000000004},"last":0.54700000000000004}]},{"osd":3,"last update":"Tue Sep 10 19:30:10 2024","interfaces":[{"interface":"back","average":{"1min":1.4390000000000001,"5min":1.819,"15min":1.321},"min":{"1min":0.316,"5min":0.27900000000000003,"15min":0.20399999999999999},"max":{"1min":4.7290000000000001,"5min":35.847000000000001,"15min":35.847000000000001},"last":0.56100000000000005},{"interface":"front","average":{"1min":1.647,"5min":1.643,"15min":1.27},"min":{"1min":0.29899999999999999,"5min":0.29199999999999998,"15min":0.20499999999999999},"max":{"1min":5.6420000000000003,"5min":35.862000000000002,"15min":35.862000000000002},"last":0.57099999999999995}]},{"osd":4,"last update":"Tue Sep 10 19:30:10 2024","interfaces":[{"interface":"back","average":{"1min":1.429,"5min":1.9990000000000001,"15min":1.47},"min":{"1min":0.35499999999999998,"5min":0.22,"15min":0.22},"max":{"1min":5.6150000000000002,"5min":35.552,"15min":35.552},"last":0.495},{"interface":"front","average":{"1min":1.371,"5min":2.117,"15min":1.4239999999999999},"min":{"1min":0.30299999999999999,"5min":0.24399999999999999,"15min":0.22500000000000001},"max":{"1min":4.5069999999999997,"5min":35.192999999999998,"15min":35.192999999999998},"last":0.58199999999999996}]},{"osd":5,"last update":"Tue Sep 10 19:30:10 2024","interfaces":[{"interface":"back","average":{"1min":1.282,"5min":1.909,"15min":1.484},"min":{"1min":0.40799999999999997,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":7.4050000000000002,"5min":48.216000000000001,"15min":48.216000000000001},"last":0.59299999999999997},{"interface":"front","average":{"1min":1.2529999999999999,"5min":2.3130000000000002,"15min":1.659},"min":{"1min":0.40100000000000002,"5min":0.254,"15min":0.254},"max":{"1min":4.3719999999999999,"5min":50.372999999999998,"15min":50.372999999999998},"last":0.63200000000000001}]},{"osd":7,"last update":"Tue Sep 10 19:30:39 2024","interfaces":[{"interface":"back","average":{"1min":0.94899999999999995,"5min":1.5209999999999999,"15min":1.5940000000000001},"min":{"1min":0.376,"5min":0.28299999999999997,"15min":0.28100000000000003},"max":{"1min":1.986,"5min":7.609,"15min":80.629999999999995},"last":0.60099999999999998},{"interface":"front","average":{"1min":1.2749999999999999,"5min":1.855,"15min":1.8680000000000001},"min":{"1min":0.35299999999999998,"5min":0.28299999999999997,"15min":0.25700000000000001},"max":{"1min":3.9710000000000001,"5min":16.654,"15min":81.088999999999999},"last":0.65000000000000002}]}]},{"osd":5,"up_from":39,"seq":167503724810,"num_pgs":65,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":936044,"kb_used_data":37924,"kb_used_omap":1,"kb_used_meta":897918,"kb_avail":92809108,"statfs":{"total":95995035648,"available":95036526592,"internally_reserved":0,"allocated":38834176,"data_stored":38531534,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":919468376},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 19:30:33 2024","interfaces":[{"interface":"back","average":{"1min":1.278,"5min":1.0580000000000001,"15min":1.1000000000000001},"min":{"1min":0.223,"5min":0.19,"15min":0.187},"max":{"1min":7.5759999999999996,"5min":7.5759999999999996,"15min":7.5759999999999996},"last":0.36199999999999999},{"interface":"front","average":{"1min":1.081,"5min":0.94899999999999995,"15min":1.175},"min":{"1min":0.20599999999999999,"5min":0.20599999999999999,"15min":0.20599999999999999},"max":{"1min":5.819,"5min":5.819,"15min":5.819},"last":0.98199999999999998}]},{"osd":1,"last update":"Tue Sep 10 19:30:51 2024","interfaces":[{"interface":"back","average":{"1min":1.504,"5min":2.5790000000000002,"15min":1.5269999999999999},"min":{"1min":0.28999999999999998,"5min":0.214,"15min":0.214},"max":{"1min":5.1289999999999996,"5min":54.097999999999999,"15min":54.097999999999999},"last":1.667},{"interface":"front","average":{"1min":1.214,"5min":2.5129999999999999,"15min":1.5229999999999999},"min":{"1min":0.29099999999999998,"5min":0.221,"15min":0.221},"max":{"1min":7.4130000000000003,"5min":54.337000000000003,"15min":54.337000000000003},"last":0.42699999999999999}]},{"osd":2,"last update":"Tue Sep 10 19:30:39 2024","interfaces":[{"interface":"back","average":{"1min":1.0629999999999999,"5min":1.0089999999999999,"15min":1.1060000000000001},"min":{"1min":0.32800000000000001,"5min":0.26500000000000001,"15min":0.22900000000000001},"max":{"1min":7.4489999999999998,"5min":7.4489999999999998,"15min":9.9399999999999995},"last":0.502},{"interface":"front","average":{"1min":0.754,"5min":0.98699999999999999,"15min":1.052},"min":{"1min":0.28599999999999998,"5min":0.27700000000000002,"15min":0.21199999999999999},"max":{"1min":2.8889999999999998,"5min":10.045999999999999,"15min":10.045999999999999},"last":1.0149999999999999}]},{"osd":3,"last update":"Tue Sep 10 19:30:39 2024","interfaces":[{"interface":"back","average":{"1min":1.2190000000000001,"5min":2.5089999999999999,"15min":1.696},"min":{"1min":0.26600000000000001,"5min":0.254,"15min":0.19900000000000001},"max":{"1min":7.5979999999999999,"5min":54.832000000000001,"15min":54.832000000000001},"last":0.48099999999999998},{"interface":"front","average":{"1min":1.2150000000000001,"5min":2.484,"15min":1.659},"min":{"1min":0.28100000000000003,"5min":0.28100000000000003,"15min":0.246},"max":{"1min":7.3520000000000003,"5min":54.850999999999999,"15min":54.850999999999999},"last":1.052}]},{"osd":4,"last update":"Tue Sep 10 19:30:39 2024","interfaces":[{"interface":"back","average":{"1min":1.0780000000000001,"5min":2.3799999999999999,"15min":2.202},"min":{"1min":0.26100000000000001,"5min":0.252,"15min":0.24399999999999999},"max":{"1min":5.6929999999999996,"5min":132.41900000000001,"15min":132.41900000000001},"last":0.90100000000000002},{"interface":"front","average":{"1min":1.1599999999999999,"5min":2.48,"15min":2.3119999999999998},"min":{"1min":0.29099999999999998,"5min":0.28699999999999998,"15min":0.217},"max":{"1min":7.5519999999999996,"5min":132.46700000000001,"15min":132.46700000000001},"last":0.45800000000000002}]},{"osd":6,"last update":"Tue Sep 10 19:30:13 2024","interfaces":[{"interface":"back","average":{"1min":0.83599999999999997,"5min":1.4019999999999999,"15min":1.8120000000000001},"min":{"1min":0.28699999999999998,"5min":0.28599999999999998,"15min":0.28399999999999997},"max":{"1min":2.4199999999999999,"5min":8.3339999999999996,"15min":111.328},"last":0.56699999999999995},{"interface":"front","average":{"1min":1.135,"5min":1.399,"15min":1.762},"min":{"1min":0.28299999999999997,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":4.1219999999999999,"5min":7.2910000000000004,"15min":111.705},"last":1.2010000000000001}]},{"osd":7,"last update":"Tue Sep 10 19:30:30 2024","interfaces":[{"interface":"back","average":{"1min":1.619,"5min":1.27,"15min":1.383},"min":{"1min":0.35099999999999998,"5min":0.29099999999999998,"15min":0.28599999999999998},"max":{"1min":4.5140000000000002,"5min":4.5140000000000002,"15min":6.7869999999999999},"last":1.2909999999999999},{"interface":"front","average":{"1min":1.2609999999999999,"5min":1.1739999999999999,"15min":1.357},"min":{"1min":0.30099999999999999,"5min":0.29899999999999999,"15min":0.27100000000000002},"max":{"1min":3.9100000000000001,"5min":5.6349999999999998,"15min":9.4909999999999997},"last":1.127}]}]},{"osd":4,"up_from":32,"seq":137438953742,"num_pgs":65,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":965100,"kb_used_data":72548,"kb_used_omap":1,"kb_used_meta":892542,"kb_avail":92780052,"statfs":{"total":95995035648,"available":95006773248,"internally_reserved":0,"allocated":74289152,"data_stored":73984625,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":913963352},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 19:30:31 2024","interfaces":[{"interface":"back","average":{"1min":0.90400000000000003,"5min":1.506,"15min":2.0299999999999998},"min":{"1min":0.21199999999999999,"5min":0.20100000000000001,"15min":0.20100000000000001},"max":{"1min":3.6110000000000002,"5min":39.097000000000001,"15min":39.097000000000001},"last":0.27100000000000002},{"interface":"front","average":{"1min":0.50600000000000001,"5min":1.6299999999999999,"15min":2.2919999999999998},"min":{"1min":0.24099999999999999,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":1.1599999999999999,"5min":39.112000000000002,"15min":39.112000000000002},"last":0.35399999999999998}]},{"osd":1,"last update":"Tue Sep 10 19:30:53 2024","interfaces":[{"interface":"back","average":{"1min":4.3979999999999997,"5min":2.0179999999999998,"15min":1.1950000000000001},"min":{"1min":0.27400000000000002,"5min":0.20599999999999999,"15min":0.20599999999999999},"max":{"1min":75.620000000000005,"5min":75.620000000000005,"15min":75.620000000000005},"last":0.39800000000000002},{"interface":"front","average":{"1min":4.4580000000000002,"5min":1.829,"15min":1.0429999999999999},"min":{"1min":0.23499999999999999,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":75.230999999999995,"5min":75.230999999999995,"15min":75.230999999999995},"last":2.9689999999999999}]},{"osd":2,"last update":"Tue Sep 10 19:30:25 2024","interfaces":[{"interface":"back","average":{"1min":0.622,"5min":1.1479999999999999,"15min":1.016},"min":{"1min":0.29399999999999998,"5min":0.27000000000000002,"15min":0.254},"max":{"1min":2.6040000000000001,"5min":5.5609999999999999,"15min":6.8869999999999996},"last":0.436},{"interface":"front","average":{"1min":1.085,"5min":1.1619999999999999,"15min":1.002},"min":{"1min":0.29299999999999998,"5min":0.28299999999999997,"15min":0.253},"max":{"1min":3.2120000000000002,"5min":6.7690000000000001,"15min":7.085},"last":2.8780000000000001}]},{"osd":3,"last update":"Tue Sep 10 19:30:25 2024","interfaces":[{"interface":"back","average":{"1min":1.161,"5min":1.1910000000000001,"15min":1.2150000000000001},"min":{"1min":0.28899999999999998,"5min":0.27800000000000002,"15min":0.20999999999999999},"max":{"1min":3.1880000000000002,"5min":6.8019999999999996,"15min":35.451999999999998},"last":2.8460000000000001},{"interface":"front","average":{"1min":1.157,"5min":1.1619999999999999,"15min":1.216},"min":{"1min":0.28499999999999998,"5min":0.28499999999999998,"15min":0.246},"max":{"1min":3.5329999999999999,"5min":5.8949999999999996,"15min":32.880000000000003},"last":0.316}]},{"osd":5,"last update":"Tue Sep 10 19:30:45 2024","interfaces":[{"interface":"back","average":{"1min":1.2769999999999999,"5min":1.177,"15min":2.6030000000000002},"min":{"1min":0.30399999999999999,"5min":0.26800000000000002,"15min":0.224},"max":{"1min":3.5840000000000001,"5min":5.9859999999999998,"15min":227.726},"last":4.7190000000000003},{"interface":"front","average":{"1min":0.80700000000000005,"5min":1.3180000000000001,"15min":2.589},"min":{"1min":0.28799999999999998,"5min":0.27400000000000002,"15min":0.20699999999999999},"max":{"1min":3.7040000000000002,"5min":5.6699999999999999,"15min":227.70500000000001},"last":0.41899999999999998}]},{"osd":6,"last update":"Tue Sep 10 19:30:25 2024","interfaces":[{"interface":"back","average":{"1min":1.0129999999999999,"5min":1.554,"15min":1.2629999999999999},"min":{"1min":0.34200000000000003,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":3.6259999999999999,"5min":8.1010000000000009,"15min":8.1010000000000009},"last":0.379},{"interface":"front","average":{"1min":1.6599999999999999,"5min":1.78,"15min":1.3420000000000001},"min":{"1min":0.254,"5min":0.254,"15min":0.247},"max":{"1min":6.0099999999999998,"5min":16.815000000000001,"15min":16.815000000000001},"last":0.81999999999999995}]},{"osd":7,"last update":"Tue Sep 10 19:30:31 2024","interfaces":[{"interface":"back","average":{"1min":1.1839999999999999,"5min":1.3660000000000001,"15min":1.3240000000000001},"min":{"1min":0.307,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":3.8319999999999999,"5min":16.853999999999999,"15min":16.853999999999999},"last":4.7629999999999999},{"interface":"front","average":{"1min":1.4590000000000001,"5min":1.4730000000000001,"15min":1.3899999999999999},"min":{"1min":0.35299999999999998,"5min":0.25700000000000001,"15min":0.23699999999999999},"max":{"1min":4.6020000000000003,"5min":10.02,"15min":10.02},"last":2.8999999999999999}]}]},{"osd":3,"up_from":26,"seq":111669149971,"num_pgs":73,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1262148,"kb_used_data":85884,"kb_used_omap":1,"kb_used_meta":1176190,"kb_avail":92483004,"statfs":{"total":95995035648,"available":94702596096,"internally_reserved":0,"allocated":87945216,"data_stored":87618576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":1204418904},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 19:30:38 2024","interfaces":[{"interface":"back","average":{"1min":8.7970000000000006,"5min":3.673,"15min":1.5800000000000001},"min":{"1min":0.16800000000000001,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":117.655,"5min":117.655,"15min":117.655},"last":3.3460000000000001},{"interface":"front","average":{"1min":8.6489999999999991,"5min":3.6440000000000001,"15min":1.5469999999999999},"min":{"1min":0.21199999999999999,"5min":0.184,"15min":0.184},"max":{"1min":117.81399999999999,"5min":117.81399999999999,"15min":117.81399999999999},"last":4.0620000000000003}]},{"osd":1,"last update":"Tue Sep 10 19:31:03 2024","interfaces":[{"interface":"back","average":{"1min":1.429,"5min":0.91900000000000004,"15min":0.52100000000000002},"min":{"1min":0.33800000000000002,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":3.54,"5min":5.1779999999999999,"15min":5.1779999999999999},"last":3.323},{"interface":"front","average":{"1min":1.266,"5min":0.94699999999999995,"15min":0.53600000000000003},"min":{"1min":0.376,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":4.0919999999999996,"5min":4.4100000000000001,"15min":4.4100000000000001},"last":4.0919999999999996}]},{"osd":2,"last update":"Tue Sep 10 19:30:08 2024","interfaces":[{"interface":"back","average":{"1min":0.66900000000000004,"5min":0.58499999999999996,"15min":0.58399999999999996},"min":{"1min":0.35599999999999998,"5min":0.246,"15min":0.17599999999999999},"max":{"1min":1.827,"5min":4.3769999999999998,"15min":5.3029999999999999},"last":0.57299999999999995},{"interface":"front","average":{"1min":0.75700000000000001,"5min":0.64100000000000001,"15min":0.626},"min":{"1min":0.248,"5min":0.23699999999999999,"15min":0.16400000000000001},"max":{"1min":2.3980000000000001,"5min":4.2990000000000004,"15min":17.806000000000001},"last":1.3029999999999999}]},{"osd":4,"last update":"Tue Sep 10 19:30:28 2024","interfaces":[{"interface":"back","average":{"1min":1.2949999999999999,"5min":1.01,"15min":1.3360000000000001},"min":{"1min":0.30099999999999999,"5min":0.26900000000000002,"15min":0.254},"max":{"1min":4.1159999999999997,"5min":4.4169999999999998,"15min":81.334000000000003},"last":1.2629999999999999},{"interface":"front","average":{"1min":1.0589999999999999,"5min":1.0740000000000001,"15min":1.429},"min":{"1min":0.318,"5min":0.27900000000000003,"15min":0.22700000000000001},"max":{"1min":3.1720000000000002,"5min":4.343,"15min":81.381},"last":3.2629999999999999}]},{"osd":5,"last update":"Tue Sep 10 19:30:48 2024","interfaces":[{"interface":"back","average":{"1min":1.153,"5min":1.1559999999999999,"15min":1.7330000000000001},"min":{"1min":0.40400000000000003,"5min":0.28000000000000003,"15min":0.27900000000000003},"max":{"1min":3.6600000000000001,"5min":6.7240000000000002,"15min":104.672},"last":1.4930000000000001},{"interface":"front","average":{"1min":1.4039999999999999,"5min":1.159,"15min":1.722},"min":{"1min":0.314,"5min":0.28799999999999998,"15min":0.24299999999999999},"max":{"1min":3.4900000000000002,"5min":4.6550000000000002,"15min":105.971},"last":3.298}]},{"osd":6,"last update":"Tue Sep 10 19:30:08 2024","interfaces":[{"interface":"back","average":{"1min":1.2470000000000001,"5min":2.8380000000000001,"15min":2.1709999999999998},"min":{"1min":0.34100000000000003,"5min":0.33600000000000002,"15min":0.23699999999999999},"max":{"1min":2.4340000000000002,"5min":148.09299999999999,"15min":148.09299999999999},"last":1.6830000000000001},{"interface":"front","average":{"1min":1.137,"5min":2.6659999999999999,"15min":2.0819999999999999},"min":{"1min":0.40200000000000002,"5min":0.32700000000000001,"15min":0.26800000000000002},"max":{"1min":3.6659999999999999,"5min":148.31999999999999,"15min":148.31999999999999},"last":0.60599999999999998}]},{"osd":7,"last update":"Tue Sep 10 19:30:48 2024","interfaces":[{"interface":"back","average":{"1min":1.716,"5min":1.175,"15min":1.4530000000000001},"min":{"1min":0.65300000000000002,"5min":0.33600000000000002,"15min":0.26100000000000001},"max":{"1min":6.8639999999999999,"5min":6.8639999999999999,"15min":104.749},"last":4.1749999999999998},{"interface":"front","average":{"1min":1.1830000000000001,"5min":1.056,"15min":1.3859999999999999},"min":{"1min":0.55900000000000005,"5min":0.33100000000000002,"15min":0.26100000000000001},"max":{"1min":2.3300000000000001,"5min":4.6849999999999996,"15min":104.69499999999999},"last":1.712}]}]},{"osd":2,"up_from":0,"seq":77309411609,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":276,"seq":1185410973798,"num_pgs":69,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":310796,"kb_used_data":768,"kb_used_omap":6,"kb_used_meta":310009,"kb_avail":93434356,"statfs":{"total":95995035648,"available":95676780544,"internally_reserved":0,"allocated":786432,"data_stored":462432,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6335,"internal_metadata":317450049},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 19:30:39 2024","interfaces":[{"interface":"back","average":{"1min":1.2210000000000001,"5min":0.79000000000000004,"15min":0.58799999999999997},"min":{"1min":0.35999999999999999,"5min":0.16700000000000001,"15min":0.16700000000000001},"max":{"1min":3.5990000000000002,"5min":3.5990000000000002,"15min":3.5990000000000002},"last":0.878},{"interface":"front","average":{"1min":1.1699999999999999,"5min":0.90700000000000003,"15min":0.57399999999999995},"min":{"1min":0.192,"5min":0.192,"15min":0.192},"max":{"1min":3.9449999999999998,"5min":4.5739999999999998,"15min":4.5739999999999998},"last":0.66400000000000003}]},{"osd":2,"last update":"Tue Sep 10 19:31:02 2024","interfaces":[{"interface":"back","average":{"1min":1.0580000000000001,"5min":0.79600000000000004,"15min":0.47999999999999998},"min":{"1min":0.24099999999999999,"5min":0.23499999999999999,"15min":0.19},"max":{"1min":2.9780000000000002,"5min":4.1779999999999999,"15min":4.1779999999999999},"last":0.92500000000000004},{"interface":"front","average":{"1min":1.044,"5min":0.81499999999999995,"15min":0.51200000000000001},"min":{"1min":0.26500000000000001,"5min":0.22800000000000001,"15min":0.222},"max":{"1min":4.5810000000000004,"5min":4.5810000000000004,"15min":4.5810000000000004},"last":0.69899999999999995}]},{"osd":3,"last update":"Tue Sep 10 19:31:02 2024","interfaces":[{"interface":"back","average":{"1min":2.04,"5min":1.137,"15min":0.59099999999999997},"min":{"1min":0.27700000000000002,"5min":0.22800000000000001,"15min":0.221},"max":{"1min":11.494999999999999,"5min":11.494999999999999,"15min":11.494999999999999},"last":0.90400000000000003},{"interface":"front","average":{"1min":1.7250000000000001,"5min":1.3049999999999999,"15min":0.64300000000000002},"min":{"1min":0.28100000000000003,"5min":0.217,"15min":0.17599999999999999},"max":{"1min":5.8049999999999997,"5min":21.545000000000002,"15min":21.545000000000002},"last":0.83199999999999996}]},{"osd":4,"last update":"Tue Sep 10 19:31:02 2024","interfaces":[{"interface":"back","average":{"1min":14.108000000000001,"5min":3.8769999999999998,"15min":1.6879999999999999},"min":{"1min":0.28599999999999998,"5min":0.27500000000000002,"15min":0.26600000000000001},"max":{"1min":271.53100000000001,"5min":271.53100000000001,"15min":271.53100000000001},"last":1.373},{"interface":"front","average":{"1min":14.308999999999999,"5min":3.7970000000000002,"15min":1.714},"min":{"1min":0.435,"5min":0.30199999999999999,"15min":0.27600000000000002},"max":{"1min":271.483,"5min":271.483,"15min":271.483},"last":0.97499999999999998}]},{"osd":5,"last update":"Tue Sep 10 19:31:02 2024","interfaces":[{"interface":"back","average":{"1min":14.177,"5min":3.831,"15min":1.8959999999999999},"min":{"1min":0.38900000000000001,"5min":0.252,"15min":0.252},"max":{"1min":212.727,"5min":212.727,"15min":212.727},"last":1.304},{"interface":"front","average":{"1min":14.183,"5min":3.8879999999999999,"15min":2.157},"min":{"1min":0.42799999999999999,"5min":0.313,"15min":0.313},"max":{"1min":212.74799999999999,"5min":212.74799999999999,"15min":212.74799999999999},"last":0.95599999999999996}]},{"osd":6,"last update":"Tue Sep 10 19:31:02 2024","interfaces":[{"interface":"back","average":{"1min":1.379,"5min":1.385,"15min":1.867},"min":{"1min":0.33100000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":4.633,"5min":7.5270000000000001,"15min":27.379999999999999},"last":0.72199999999999998},{"interface":"front","average":{"1min":1.4399999999999999,"5min":1.29,"15min":1.827},"min":{"1min":0.33500000000000002,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":3.6680000000000001,"5min":5.8849999999999998,"15min":26.812999999999999},"last":1.393}]},{"osd":7,"last update":"Tue Sep 10 19:31:02 2024","interfaces":[{"interface":"back","average":{"1min":12.827999999999999,"5min":3.6899999999999999,"15min":1.893},"min":{"1min":0.29999999999999999,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":242.33099999999999,"5min":242.33099999999999,"15min":242.33099999999999},"last":0.73999999999999999},{"interface":"front","average":{"1min":13.044,"5min":3.6840000000000002,"15min":1.897},"min":{"1min":0.40200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":242.31,"5min":242.31,"15min":242.31},"last":0.39100000000000001}]}]},{"osd":0,"up_from":301,"seq":1292785156178,"num_pgs":65,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":385844,"kb_used_data":15020,"kb_used_omap":4,"kb_used_meta":370811,"kb_avail":93359308,"statfs":{"total":95995035648,"available":95599931392,"internally_reserved":0,"allocated":15380480,"data_stored":15069066,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":379710982},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Sep 10 19:30:34 2024","interfaces":[{"interface":"back","average":{"1min":1.7929999999999999,"5min":0.91600000000000004,"15min":0.68799999999999994},"min":{"1min":0.217,"5min":0.17499999999999999,"15min":0.17499999999999999},"max":{"1min":6.0369999999999999,"5min":6.0369999999999999,"15min":6.0369999999999999},"last":2.3580000000000001},{"interface":"front","average":{"1min":1.4430000000000001,"5min":0.75800000000000001,"15min":0.58199999999999996},"min":{"1min":0.19600000000000001,"5min":0.16900000000000001,"15min":0.16900000000000001},"max":{"1min":3.7909999999999999,"5min":3.7909999999999999,"15min":3.7909999999999999},"last":2.9359999999999999}]},{"osd":2,"last update":"Tue Sep 10 19:30:34 2024","interfaces":[{"interface":"back","average":{"1min":0.84799999999999998,"5min":0.59199999999999997,"15min":0.59899999999999998},"min":{"1min":0.21299999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":4.3230000000000004,"5min":4.3230000000000004,"15min":4.3230000000000004},"last":1.96},{"interface":"front","average":{"1min":1.569,"5min":0.81799999999999995,"15min":0.69599999999999995},"min":{"1min":0.249,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":5.1079999999999997,"5min":5.1079999999999997,"15min":5.1079999999999997},"last":1.026}]},{"osd":3,"last update":"Tue Sep 10 19:30:34 2024","interfaces":[{"interface":"back","average":{"1min":5.9119999999999999,"5min":1.6950000000000001,"15min":0.95699999999999996},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":89.468999999999994,"5min":89.468999999999994,"15min":89.468999999999994},"last":1.4970000000000001},{"interface":"front","average":{"1min":6.141,"5min":1.8200000000000001,"15min":1.022},"min":{"1min":0.216,"5min":0.216,"15min":0.216},"max":{"1min":89.272999999999996,"5min":89.272999999999996,"15min":89.272999999999996},"last":0.35999999999999999}]},{"osd":4,"last update":"Tue Sep 10 19:30:34 2024","interfaces":[{"interface":"back","average":{"1min":1.802,"5min":1.306,"15min":3.4380000000000002},"min":{"1min":0.38900000000000001,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":4.9729999999999999,"5min":6.5640000000000001,"15min":53.670000000000002},"last":2.117},{"interface":"front","average":{"1min":1.103,"5min":1.087,"15min":3.3700000000000001},"min":{"1min":0.28000000000000003,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":3.6520000000000001,"5min":6.5019999999999998,"15min":53.935000000000002},"last":0.90300000000000002}]},{"osd":5,"last update":"Tue Sep 10 19:30:34 2024","interfaces":[{"interface":"back","average":{"1min":1.232,"5min":0.999,"15min":3.0529999999999999},"min":{"1min":0.30399999999999999,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":4.2779999999999996,"5min":4.2779999999999996,"15min":53.884999999999998},"last":0.39100000000000001},{"interface":"front","average":{"1min":1.2769999999999999,"5min":1.093,"15min":2.988},"min":{"1min":0.38800000000000001,"5min":0.309,"15min":0.309},"max":{"1min":5.0519999999999996,"5min":5.0519999999999996,"15min":53.875999999999998},"last":2.1539999999999999}]},{"osd":6,"last update":"Tue Sep 10 19:30:34 2024","interfaces":[{"interface":"back","average":{"1min":1.6120000000000001,"5min":1.216,"15min":3.6240000000000001},"min":{"1min":0.34799999999999998,"5min":0.311,"15min":0.311},"max":{"1min":3.7069999999999999,"5min":5.3769999999999998,"15min":56.901000000000003},"last":0.86899999999999999},{"interface":"front","average":{"1min":1.296,"5min":1.1819999999999999,"15min":3.5609999999999999},"min":{"1min":0.29499999999999998,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":4.766,"5min":6.6840000000000002,"15min":55.177},"last":0.40999999999999998}]},{"osd":7,"last update":"Tue Sep 10 19:30:34 2024","interfaces":[{"interface":"back","average":{"1min":1.0860000000000001,"5min":1.1000000000000001,"15min":1.1910000000000001},"min":{"1min":0.32900000000000001,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":4.2149999999999999,"5min":4.2149999999999999,"15min":4.2149999999999999},"last":1.9359999999999999},{"interface":"front","average":{"1min":1.385,"5min":1.125,"15min":1.2130000000000001},"min":{"1min":0.375,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":5.0810000000000004,"5min":5.0810000000000004,"15min":5.0810000000000004},"last":2.2970000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":966,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1236,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1236,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":362,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1540,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":2118,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1018,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1502,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":6619136,"data_stored":6619136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":75694080,"data_stored":75694080,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":56954880,"data_stored":56950808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":37158912,"data_stored":37158912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":80674816,"data_stored":80674816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T19:31:12.175 INFO:tasks.thrashosds.thrasher:Setting 4.a to [3, 2, 1] 2024-09-10T19:31:12.175 INFO:tasks.thrashosds.thrasher:cmd ['osd', 'pg-upmap', '4.a', '3', '2', '1'] 2024-09-10T19:31:12.175 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph osd pg-upmap 4.a 3 2 1 2024-09-10T19:31:12.216 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:31:12.238 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:31:12.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:31:12.239 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:31:12.340 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:31:12.363 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:31:12.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:31:12.365 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:31:12.421 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:31:12 smithi183 ceph-mon[29736]: from='client.26111 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-10T19:31:12.421 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:31:12 smithi183 ceph-mon[41562]: from='client.26111 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-10T19:31:12.465 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:31:12.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:31:12 smithi191 ceph-mon[32884]: from='client.26111 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-10T19:31:12.487 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:31:12.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:31:12.489 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:31:12.589 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:31:12.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:31:12.613 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:31:12.613 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:31:12.714 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:31:12.736 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:31:12.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:31:12.737 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:31:12.838 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:31:12.860 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:31:12.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:31:12.861 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:31:12.882 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:31:12.962 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:31:12.988 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:31:12.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:31:12.989 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:31:13.090 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:31:13.113 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:31:13.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:31:13.114 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:31:13.215 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:31:13.238 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:31:13.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:31:13.239 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:31:13.251 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:31:13 smithi191 ceph-mon[32884]: 11.1 scrub starts 2024-09-10T19:31:13.252 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:31:13 smithi191 ceph-mon[32884]: 11.1 scrub ok 2024-09-10T19:31:13.252 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:31:13 smithi191 ceph-mon[32884]: pgmap v967: 158 pgs: 158 active+clean; 2.0 MiB data, 5.5 GiB used, 620 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.06k op/s 2024-09-10T19:31:13.252 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:31:13 smithi191 ceph-mon[32884]: 6.15 scrub starts 2024-09-10T19:31:13.252 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:31:13 smithi191 ceph-mon[32884]: 6.15 scrub ok 2024-09-10T19:31:13.252 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:31:13 smithi191 ceph-mon[32884]: 6.6 scrub starts 2024-09-10T19:31:13.252 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:31:13 smithi191 ceph-mon[32884]: 6.6 scrub ok 2024-09-10T19:31:13.252 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:31:13 smithi191 ceph-mon[32884]: 5.15 scrub starts 2024-09-10T19:31:13.252 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:31:13 smithi191 ceph-mon[32884]: 5.15 scrub ok 2024-09-10T19:31:13.311 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:31:13 smithi183 ceph-mon[29736]: 11.1 scrub starts 2024-09-10T19:31:13.312 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:31:13 smithi183 ceph-mon[29736]: 11.1 scrub ok 2024-09-10T19:31:13.312 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:31:13 smithi183 ceph-mon[29736]: pgmap v967: 158 pgs: 158 active+clean; 2.0 MiB data, 5.5 GiB used, 620 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.06k op/s 2024-09-10T19:31:13.312 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:31:13 smithi183 ceph-mon[29736]: 6.15 scrub starts 2024-09-10T19:31:13.312 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:31:13 smithi183 ceph-mon[29736]: 6.15 scrub ok 2024-09-10T19:31:13.312 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:31:13 smithi183 ceph-mon[29736]: 6.6 scrub starts 2024-09-10T19:31:13.312 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:31:13 smithi183 ceph-mon[29736]: 6.6 scrub ok 2024-09-10T19:31:13.312 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:31:13 smithi183 ceph-mon[29736]: 5.15 scrub starts 2024-09-10T19:31:13.312 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:31:13 smithi183 ceph-mon[29736]: 5.15 scrub ok 2024-09-10T19:31:13.312 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:31:13 smithi183 ceph-mon[41562]: 11.1 scrub starts 2024-09-10T19:31:13.313 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:31:13 smithi183 ceph-mon[41562]: 11.1 scrub ok 2024-09-10T19:31:13.313 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:31:13 smithi183 ceph-mon[41562]: pgmap v967: 158 pgs: 158 active+clean; 2.0 MiB data, 5.5 GiB used, 620 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 1.06k op/s 2024-09-10T19:31:13.313 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:31:13 smithi183 ceph-mon[41562]: 6.15 scrub starts 2024-09-10T19:31:13.313 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:31:13 smithi183 ceph-mon[41562]: 6.15 scrub ok 2024-09-10T19:31:13.313 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:31:13 smithi183 ceph-mon[41562]: 6.6 scrub starts 2024-09-10T19:31:13.313 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:31:13 smithi183 ceph-mon[41562]: 6.6 scrub ok 2024-09-10T19:31:13.313 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:31:13 smithi183 ceph-mon[41562]: 5.15 scrub starts 2024-09-10T19:31:13.314 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:31:13 smithi183 ceph-mon[41562]: 5.15 scrub ok 2024-09-10T19:31:13.340 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:31:13.362 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:31:13.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:31:13.363 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:31:13.463 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:31:13.485 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:31:13.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:31:13.486 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:31:13.523 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-10T19:31:13.587 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:31:13.611 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:31:13.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:31:13.612 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:31:13.712 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:31:13.734 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:31:13.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:31:13.735 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:31:13.836 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:31:13.843 INFO:teuthology.orchestra.run.smithi191.stdout:{ 2024-09-10T19:31:13.843 INFO:teuthology.orchestra.run.smithi191.stdout: "ops": [], 2024-09-10T19:31:13.843 INFO:teuthology.orchestra.run.smithi191.stdout: "num_ops": 0 2024-09-10T19:31:13.843 INFO:teuthology.orchestra.run.smithi191.stdout:} 2024-09-10T19:31:13.858 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:31:13.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:31:13.859 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:31:13.960 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:31:13.981 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:31:13.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:31:13.982 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:31:14.083 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:31:14.106 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:31:14.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:31:14.107 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:31:14.207 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:31:14.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:31:14.230 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:31:14.231 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:31:14.331 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:31:14.353 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:31:14.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:31:14.354 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:31:14.431 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:31:14 smithi183 ceph-mon[41562]: 4.1f scrub starts 2024-09-10T19:31:14.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:31:14 smithi183 ceph-mon[41562]: 4.1f scrub ok 2024-09-10T19:31:14.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:31:14 smithi183 ceph-mon[41562]: 4.f scrub starts 2024-09-10T19:31:14.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:31:14 smithi183 ceph-mon[41562]: 4.f scrub ok 2024-09-10T19:31:14.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:31:14 smithi183 ceph-mon[41562]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:31:14.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:31:14 smithi183 ceph-mon[41562]: 5.1 scrub starts 2024-09-10T19:31:14.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:31:14 smithi183 ceph-mon[41562]: 5.1 scrub ok 2024-09-10T19:31:14.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:31:14 smithi183 ceph-mon[29736]: 4.1f scrub starts 2024-09-10T19:31:14.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:31:14 smithi183 ceph-mon[29736]: 4.1f scrub ok 2024-09-10T19:31:14.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:31:14 smithi183 ceph-mon[29736]: 4.f scrub starts 2024-09-10T19:31:14.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:31:14 smithi183 ceph-mon[29736]: 4.f scrub ok 2024-09-10T19:31:14.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:31:14 smithi183 ceph-mon[29736]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:31:14.433 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:31:14 smithi183 ceph-mon[29736]: 5.1 scrub starts 2024-09-10T19:31:14.433 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:31:14 smithi183 ceph-mon[29736]: 5.1 scrub ok 2024-09-10T19:31:14.455 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:31:14.465 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:31:14 smithi191 ceph-mon[32884]: 4.1f scrub starts 2024-09-10T19:31:14.465 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:31:14 smithi191 ceph-mon[32884]: 4.1f scrub ok 2024-09-10T19:31:14.465 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:31:14 smithi191 ceph-mon[32884]: 4.f scrub starts 2024-09-10T19:31:14.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:31:14 smithi191 ceph-mon[32884]: 4.f scrub ok 2024-09-10T19:31:14.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:31:14 smithi191 ceph-mon[32884]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:31:14.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:31:14 smithi191 ceph-mon[32884]: 5.1 scrub starts 2024-09-10T19:31:14.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:31:14 smithi191 ceph-mon[32884]: 5.1 scrub ok 2024-09-10T19:31:14.476 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:31:14.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:31:14.477 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:31:14.527 DEBUG:teuthology.orchestra.run.smithi191:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph daemon osd.7 dump_blocked_ops 2024-09-10T19:31:14.578 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:31:14.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:31:14.602 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:31:14.602 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:31:14.703 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:31:14.727 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:31:14.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:31:14.728 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:31:14.770 INFO:teuthology.orchestra.run.smithi191.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.b/config 2024-09-10T19:31:14.829 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:31:14.850 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:31:14.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:31:14.851 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:31:14.952 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:31:14.977 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:31:14.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:31:14.978 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:31:15.079 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:31:15.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:31:15.103 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:31:15.103 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:31:15.204 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:31:15.227 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:31:15.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:31:15.228 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:31:15.318 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:31:15 smithi183 ceph-mon[29736]: pgmap v968: 158 pgs: 158 active+clean; 582 KiB data, 5.5 GiB used, 620 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 933 op/s 2024-09-10T19:31:15.318 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:31:15 smithi183 ceph-mon[29736]: 2.18 scrub starts 2024-09-10T19:31:15.318 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:31:15 smithi183 ceph-mon[29736]: 2.18 scrub ok 2024-09-10T19:31:15.318 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:31:15 smithi183 ceph-mon[29736]: 4.13 scrub starts 2024-09-10T19:31:15.318 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:31:15 smithi183 ceph-mon[29736]: 4.13 scrub ok 2024-09-10T19:31:15.319 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:31:15 smithi183 ceph-mon[41562]: pgmap v968: 158 pgs: 158 active+clean; 582 KiB data, 5.5 GiB used, 620 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 933 op/s 2024-09-10T19:31:15.319 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:31:15 smithi183 ceph-mon[41562]: 2.18 scrub starts 2024-09-10T19:31:15.319 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:31:15 smithi183 ceph-mon[41562]: 2.18 scrub ok 2024-09-10T19:31:15.319 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:31:15 smithi183 ceph-mon[41562]: 4.13 scrub starts 2024-09-10T19:31:15.319 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:31:15 smithi183 ceph-mon[41562]: 4.13 scrub ok 2024-09-10T19:31:15.329 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:31:15.351 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:31:15.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:31:15.352 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:31:15.452 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:31:15.465 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:31:15 smithi191 ceph-mon[32884]: pgmap v968: 158 pgs: 158 active+clean; 582 KiB data, 5.5 GiB used, 620 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 933 op/s 2024-09-10T19:31:15.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:31:15 smithi191 ceph-mon[32884]: 2.18 scrub starts 2024-09-10T19:31:15.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:31:15 smithi191 ceph-mon[32884]: 2.18 scrub ok 2024-09-10T19:31:15.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:31:15 smithi191 ceph-mon[32884]: 4.13 scrub starts 2024-09-10T19:31:15.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:31:15 smithi191 ceph-mon[32884]: 4.13 scrub ok 2024-09-10T19:31:15.474 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:31:15.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:31:15.475 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:31:15.576 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:31:15.597 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:31:15.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:31:15.598 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:31:15.699 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:31:15.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:31:15.722 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:31:15.722 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:31:15.823 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:31:15.846 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:31:15.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:31:15.847 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:31:15.948 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:31:15.969 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:31:15.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:31:15.970 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:31:16.071 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:31:16.092 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:31:16.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:31:16.093 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:31:16.194 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:31:16.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:31:16.218 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:31:16.218 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:31:16.232 INFO:teuthology.orchestra.run.smithi183.stderr:nodeep-scrub is unset 2024-09-10T19:31:16.319 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:31:16.340 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:31:16.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:31:16.341 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:31:16.442 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:31:16.450 INFO:teuthology.orchestra.run.smithi191.stdout:{ 2024-09-10T19:31:16.451 INFO:teuthology.orchestra.run.smithi191.stdout: "ops": [], 2024-09-10T19:31:16.451 INFO:teuthology.orchestra.run.smithi191.stdout: "complaint_time": 30, 2024-09-10T19:31:16.451 INFO:teuthology.orchestra.run.smithi191.stdout: "num_blocked_ops": 0 2024-09-10T19:31:16.451 INFO:teuthology.orchestra.run.smithi191.stdout:} 2024-09-10T19:31:16.464 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:31:16.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:31:16.465 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:31:16.565 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:31:16.587 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:31:16.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:31:16.588 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:31:16.681 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:31:16 smithi183 ceph-mon[29736]: 6.1f scrub starts 2024-09-10T19:31:16.682 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:31:16 smithi183 ceph-mon[29736]: 6.1f scrub ok 2024-09-10T19:31:16.682 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:31:16 smithi183 ceph-mon[29736]: 2.9 scrub starts 2024-09-10T19:31:16.682 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:31:16 smithi183 ceph-mon[29736]: 2.9 scrub ok 2024-09-10T19:31:16.682 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:31:16 smithi183 ceph-mon[29736]: 11.7 scrub starts 2024-09-10T19:31:16.682 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:31:16 smithi183 ceph-mon[29736]: 11.7 scrub ok 2024-09-10T19:31:16.682 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:31:16 smithi183 ceph-mon[29736]: 5.8 scrub starts 2024-09-10T19:31:16.682 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:31:16 smithi183 ceph-mon[29736]: 5.8 scrub ok 2024-09-10T19:31:16.682 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:31:16 smithi183 ceph-mon[29736]: from='client.? 172.21.15.183:0/880712792' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-10T19:31:16.683 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:31:16 smithi183 ceph-mon[41562]: 6.1f scrub starts 2024-09-10T19:31:16.683 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:31:16 smithi183 ceph-mon[41562]: 6.1f scrub ok 2024-09-10T19:31:16.683 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:31:16 smithi183 ceph-mon[41562]: 2.9 scrub starts 2024-09-10T19:31:16.683 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:31:16 smithi183 ceph-mon[41562]: 2.9 scrub ok 2024-09-10T19:31:16.683 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:31:16 smithi183 ceph-mon[41562]: 11.7 scrub starts 2024-09-10T19:31:16.683 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:31:16 smithi183 ceph-mon[41562]: 11.7 scrub ok 2024-09-10T19:31:16.683 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:31:16 smithi183 ceph-mon[41562]: 5.8 scrub starts 2024-09-10T19:31:16.683 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:31:16 smithi183 ceph-mon[41562]: 5.8 scrub ok 2024-09-10T19:31:16.683 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:31:16 smithi183 ceph-mon[41562]: from='client.? 172.21.15.183:0/880712792' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-10T19:31:16.690 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:31:16.710 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:31:16.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:31:16.711 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:31:16.715 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:31:16 smithi191 ceph-mon[32884]: 6.1f scrub starts 2024-09-10T19:31:16.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:31:16 smithi191 ceph-mon[32884]: 6.1f scrub ok 2024-09-10T19:31:16.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:31:16 smithi191 ceph-mon[32884]: 2.9 scrub starts 2024-09-10T19:31:16.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:31:16 smithi191 ceph-mon[32884]: 2.9 scrub ok 2024-09-10T19:31:16.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:31:16 smithi191 ceph-mon[32884]: 11.7 scrub starts 2024-09-10T19:31:16.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:31:16 smithi191 ceph-mon[32884]: 11.7 scrub ok 2024-09-10T19:31:16.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:31:16 smithi191 ceph-mon[32884]: 5.8 scrub starts 2024-09-10T19:31:16.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:31:16 smithi191 ceph-mon[32884]: 5.8 scrub ok 2024-09-10T19:31:16.717 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:31:16 smithi191 ceph-mon[32884]: from='client.? 172.21.15.183:0/880712792' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-10T19:31:16.812 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:31:16.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:31:16.835 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:31:16.835 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:31:16.937 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:31:16.958 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:31:16.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:31:16.959 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:31:17.061 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:31:17.063 DEBUG:teuthology.orchestra.run.smithi191:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph daemon osd.7 dump_historic_ops 2024-09-10T19:31:17.084 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:31:17.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:31:17.085 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:31:17.186 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:31:17.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:31:17.208 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:31:17.209 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:31:17.305 INFO:teuthology.orchestra.run.smithi191.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.b/config 2024-09-10T19:31:17.309 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:31:17.330 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:31:17.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:31:17.332 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:31:17.426 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:31:17 smithi191 ceph-mon[32884]: pgmap v969: 158 pgs: 158 active+clean; 582 KiB data, 5.5 GiB used, 620 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 353 op/s 2024-09-10T19:31:17.426 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:31:17 smithi191 ceph-mon[32884]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-10T19:31:17.426 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:31:17 smithi191 ceph-mon[32884]: Cluster is now healthy 2024-09-10T19:31:17.426 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:31:17 smithi191 ceph-mon[32884]: from='client.? 172.21.15.183:0/880712792' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-10T19:31:17.426 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:31:17 smithi191 ceph-mon[32884]: osdmap e424: 8 total, 8 up, 7 in 2024-09-10T19:31:17.427 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:31:17 smithi191 ceph-mon[32884]: 5.12 scrub starts 2024-09-10T19:31:17.427 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:31:17 smithi191 ceph-mon[32884]: 5.12 scrub ok 2024-09-10T19:31:17.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:31:17 smithi183 ceph-mon[41562]: pgmap v969: 158 pgs: 158 active+clean; 582 KiB data, 5.5 GiB used, 620 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 353 op/s 2024-09-10T19:31:17.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:31:17 smithi183 ceph-mon[41562]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-10T19:31:17.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:31:17 smithi183 ceph-mon[41562]: Cluster is now healthy 2024-09-10T19:31:17.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:31:17 smithi183 ceph-mon[41562]: from='client.? 172.21.15.183:0/880712792' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-10T19:31:17.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:31:17 smithi183 ceph-mon[41562]: osdmap e424: 8 total, 8 up, 7 in 2024-09-10T19:31:17.433 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:31:17 smithi183 ceph-mon[41562]: 5.12 scrub starts 2024-09-10T19:31:17.433 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:31:17 smithi183 ceph-mon[41562]: 5.12 scrub ok 2024-09-10T19:31:17.433 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:31:17 smithi183 ceph-mon[29736]: pgmap v969: 158 pgs: 158 active+clean; 582 KiB data, 5.5 GiB used, 620 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 353 op/s 2024-09-10T19:31:17.433 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:31:17 smithi183 ceph-mon[29736]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-10T19:31:17.433 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:31:17 smithi183 ceph-mon[29736]: Cluster is now healthy 2024-09-10T19:31:17.434 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:31:17 smithi183 ceph-mon[29736]: from='client.? 172.21.15.183:0/880712792' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-10T19:31:17.434 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:31:17 smithi183 ceph-mon[29736]: osdmap e424: 8 total, 8 up, 7 in 2024-09-10T19:31:17.434 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:31:17 smithi183 ceph-mon[29736]: 5.12 scrub starts 2024-09-10T19:31:17.434 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:31:17 smithi183 ceph-mon[29736]: 5.12 scrub ok 2024-09-10T19:31:17.435 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:31:17.456 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:31:17.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:31:17.457 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:31:17.558 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:31:17.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:31:17.580 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:31:17.581 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:31:17.682 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:31:17.703 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:31:17.704 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:31:17.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:31:17.705 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:31:17.715 INFO:journalctl@ceph.mgr.x.smithi191.stdout:Sep 10 19:31:17 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-x[34967]: ::ffff:172.21.15.191 - - [10/Sep/2024:19:31:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T19:31:17.806 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:31:17.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:31:17.830 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:31:17.830 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:31:17.931 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:31:17.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:31:17.953 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:31:17.953 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:31:18.054 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:31:18.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:31:18.077 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:31:18.078 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:31:18.178 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:31:18.200 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:31:18.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:31:18.201 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:31:18.302 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:31:18.316 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:31:18 smithi183 ceph-mon[29736]: 2.7 scrub starts 2024-09-10T19:31:18.316 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:31:18 smithi183 ceph-mon[29736]: 2.7 scrub ok 2024-09-10T19:31:18.316 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:31:18 smithi183 ceph-mon[29736]: 4.9 scrub starts 2024-09-10T19:31:18.316 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:31:18 smithi183 ceph-mon[29736]: 4.9 scrub ok 2024-09-10T19:31:18.316 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:31:18 smithi183 ceph-mon[41562]: 2.7 scrub starts 2024-09-10T19:31:18.316 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:31:18 smithi183 ceph-mon[41562]: 2.7 scrub ok 2024-09-10T19:31:18.316 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:31:18 smithi183 ceph-mon[41562]: 4.9 scrub starts 2024-09-10T19:31:18.316 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:31:18 smithi183 ceph-mon[41562]: 4.9 scrub ok 2024-09-10T19:31:18.324 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:31:18.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:31:18.325 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:31:18.426 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:31:18.447 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:31:18.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:31:18.448 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:31:18.549 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:31:18.571 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:31:18.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:31:18.572 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:31:18.602 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:31:18 smithi191 ceph-mon[32884]: 2.7 scrub starts 2024-09-10T19:31:18.602 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:31:18 smithi191 ceph-mon[32884]: 2.7 scrub ok 2024-09-10T19:31:18.602 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:31:18 smithi191 ceph-mon[32884]: 4.9 scrub starts 2024-09-10T19:31:18.602 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:31:18 smithi191 ceph-mon[32884]: 4.9 scrub ok 2024-09-10T19:31:18.673 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:31:18.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:31:18.696 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:31:18.696 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:31:18.797 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:31:18.820 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:31:18.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:31:18.821 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:31:18.922 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:31:18.943 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:31:18.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:31:18.944 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:31:19.045 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:31:19.066 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:31:19.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:31:19.067 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:31:19.140 INFO:teuthology.orchestra.run.smithi191.stdout:{ 2024-09-10T19:31:19.140 INFO:teuthology.orchestra.run.smithi191.stdout: "size": 20, 2024-09-10T19:31:19.140 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 600, 2024-09-10T19:31:19.140 INFO:teuthology.orchestra.run.smithi191.stdout: "ops": [ 2024-09-10T19:31:19.140 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:31:19.140 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.15360.0:169775 9.9 9:9da34f67:::benchmark_data_smithi183_142499_object23286:head [delete] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected+supports_pool_eio e263)", 2024-09-10T19:31:19.140 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-10T19:21:46.960468+0000", 2024-09-10T19:31:19.141 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 572.16569391600001, 2024-09-10T19:31:19.141 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.0671650109999999, 2024-09-10T19:31:19.141 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-10T19:31:19.141 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T19:31:19.141 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-10T19:31:19.141 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.15360", 2024-09-10T19:31:19.141 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.183:0/2364804131", 2024-09-10T19:31:19.141 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 169775 2024-09-10T19:31:19.141 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:31:19.141 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-10T19:31:19.141 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:31:19.141 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-10T19:31:19.141 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:21:46.960468+0000", 2024-09-10T19:31:19.141 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:31:19.142 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:31:19.142 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:31:19.142 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-10T19:31:19.142 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:21:46.960468+0000", 2024-09-10T19:31:19.142 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:31:19.142 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:31:19.142 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:31:19.142 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-10T19:31:19.142 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:21:46.960466+0000", 2024-09-10T19:31:19.142 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999981 2024-09-10T19:31:19.142 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:31:19.142 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:31:19.142 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-10T19:31:19.142 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:21:46.960473+0000", 2024-09-10T19:31:19.143 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.1659999999999997e-06 2024-09-10T19:31:19.143 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:31:19.143 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:31:19.143 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-10T19:31:19.143 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:21:46.960486+0000", 2024-09-10T19:31:19.143 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.2972e-05 2024-09-10T19:31:19.143 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:31:19.143 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:31:19.143 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-10T19:31:19.144 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:21:46.960493+0000", 2024-09-10T19:31:19.144 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.1910000000000003e-06 2024-09-10T19:31:19.144 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:31:19.144 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:31:19.144 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:31:19.144 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:21:46.991408+0000", 2024-09-10T19:31:19.144 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.030915074000000001 2024-09-10T19:31:19.144 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:31:19.144 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:31:19.144 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for peered", 2024-09-10T19:31:19.144 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:21:46.991416+0000", 2024-09-10T19:31:19.144 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.5140000000000004e-06 2024-09-10T19:31:19.144 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:31:19.144 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:31:19.145 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:31:19.145 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:21:48.013668+0000", 2024-09-10T19:31:19.145 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.022252231 2024-09-10T19:31:19.145 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:31:19.145 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:31:19.145 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-10T19:31:19.145 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:21:48.015474+0000", 2024-09-10T19:31:19.145 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0018054760000000001 2024-09-10T19:31:19.145 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:31:19.145 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:31:19.145 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 5", 2024-09-10T19:31:19.145 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:21:48.015713+0000", 2024-09-10T19:31:19.145 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00023921599999999999 2024-09-10T19:31:19.146 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:31:19.146 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:31:19.146 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-09-10T19:31:19.146 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:21:48.024289+0000", 2024-09-10T19:31:19.146 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0085763880000000008 2024-09-10T19:31:19.146 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:31:19.146 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:31:19.146 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:31:19.146 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:21:48.027515+0000", 2024-09-10T19:31:19.146 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0032260079999999998 2024-09-10T19:31:19.146 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:31:19.146 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:31:19.146 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-09-10T19:31:19.146 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:21:48.027559+0000", 2024-09-10T19:31:19.147 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.3996999999999999e-05 2024-09-10T19:31:19.147 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:31:19.147 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:31:19.147 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-10T19:31:19.147 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:21:48.027633+0000", 2024-09-10T19:31:19.147 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.3619000000000001e-05 2024-09-10T19:31:19.147 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:31:19.147 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-10T19:31:19.147 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:31:19.147 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:31:19.147 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:31:19.147 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.15360.0:169783 9.9 9:93db4d72:::benchmark_data_smithi183_142499_object23294:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e263)", 2024-09-10T19:31:19.147 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-10T19:21:46.991598+0000", 2024-09-10T19:31:19.147 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 572.13456406299997, 2024-09-10T19:31:19.148 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.0422353129999999, 2024-09-10T19:31:19.148 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-10T19:31:19.148 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T19:31:19.148 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-10T19:31:19.148 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.15360", 2024-09-10T19:31:19.148 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.183:0/2364804131", 2024-09-10T19:31:19.148 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 169783 2024-09-10T19:31:19.148 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:31:19.148 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-10T19:31:19.148 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:31:19.148 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-10T19:31:19.148 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:21:46.991598+0000", 2024-09-10T19:31:19.148 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:31:19.148 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:31:19.148 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:31:19.149 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-10T19:31:19.149 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:21:46.991598+0000", 2024-09-10T19:31:19.149 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:31:19.149 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:31:19.149 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:31:19.149 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-10T19:31:19.149 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:21:46.991597+0000", 2024-09-10T19:31:19.149 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999986 2024-09-10T19:31:19.149 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:31:19.150 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:31:19.150 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-10T19:31:19.150 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:21:46.991602+0000", 2024-09-10T19:31:19.150 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.5380000000000002e-06 2024-09-10T19:31:19.150 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:31:19.150 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:31:19.150 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-10T19:31:19.150 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:21:46.991612+0000", 2024-09-10T19:31:19.150 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.5289999999999997e-06 2024-09-10T19:31:19.150 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:31:19.151 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:31:19.151 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-10T19:31:19.151 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:21:46.991617+0000", 2024-09-10T19:31:19.151 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.5110000000000003e-06 2024-09-10T19:31:19.151 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:31:19.151 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:31:19.151 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:31:19.151 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:21:46.991868+0000", 2024-09-10T19:31:19.151 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000251339 2024-09-10T19:31:19.151 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:31:19.151 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:31:19.151 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for peered", 2024-09-10T19:31:19.151 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:21:46.991875+0000", 2024-09-10T19:31:19.151 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.2600000000000002e-06 2024-09-10T19:31:19.152 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:31:19.152 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:31:19.152 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:31:19.152 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:21:48.020111+0000", 2024-09-10T19:31:19.152 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.0282362519999999 2024-09-10T19:31:19.152 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:31:19.152 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:31:19.152 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-10T19:31:19.152 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:21:48.020206+0000", 2024-09-10T19:31:19.152 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.5179000000000005e-05 2024-09-10T19:31:19.152 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:31:19.152 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:31:19.152 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 5", 2024-09-10T19:31:19.153 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:21:48.020309+0000", 2024-09-10T19:31:19.153 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00010310999999999999 2024-09-10T19:31:19.153 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:31:19.153 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:31:19.153 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-09-10T19:31:19.153 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:21:48.027641+0000", 2024-09-10T19:31:19.153 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0073316939999999997 2024-09-10T19:31:19.153 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:31:19.153 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:31:19.153 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:31:19.153 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:21:48.033698+0000", 2024-09-10T19:31:19.154 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0060571660000000001 2024-09-10T19:31:19.154 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:31:19.154 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:31:19.154 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-09-10T19:31:19.154 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:21:48.033745+0000", 2024-09-10T19:31:19.154 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.6461000000000001e-05 2024-09-10T19:31:19.154 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:31:19.154 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:31:19.154 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-10T19:31:19.154 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:21:48.033833+0000", 2024-09-10T19:31:19.154 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.8530999999999997e-05 2024-09-10T19:31:19.154 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:31:19.154 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-10T19:31:19.154 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:31:19.154 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:31:19.155 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:31:19.155 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.15360.0:169785 9.9 9:9cc1dce6:::benchmark_data_smithi183_142499_object23296:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e263)", 2024-09-10T19:31:19.155 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-10T19:21:46.993227+0000", 2024-09-10T19:31:19.155 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 572.13293515700002, 2024-09-10T19:31:19.155 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.0407537469999999, 2024-09-10T19:31:19.155 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-10T19:31:19.155 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T19:31:19.155 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-10T19:31:19.155 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.15360", 2024-09-10T19:31:19.155 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.183:0/2364804131", 2024-09-10T19:31:19.155 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 169785 2024-09-10T19:31:19.155 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:31:19.155 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-10T19:31:19.156 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:31:19.156 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-10T19:31:19.156 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:21:46.993227+0000", 2024-09-10T19:31:19.156 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:31:19.156 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:31:19.156 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:31:19.156 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-10T19:31:19.156 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:21:46.993227+0000", 2024-09-10T19:31:19.156 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:31:19.156 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:31:19.156 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:31:19.156 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-10T19:31:19.156 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:21:46.993226+0000", 2024-09-10T19:31:19.156 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.999999 2024-09-10T19:31:19.157 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:31:19.157 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:31:19.157 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-10T19:31:19.157 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:21:46.993231+0000", 2024-09-10T19:31:19.157 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.5840000000000002e-06 2024-09-10T19:31:19.157 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:31:19.157 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:31:19.157 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-10T19:31:19.157 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:21:46.993243+0000", 2024-09-10T19:31:19.157 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.1686e-05 2024-09-10T19:31:19.157 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:31:19.157 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:31:19.157 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-10T19:31:19.157 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:21:46.993249+0000", 2024-09-10T19:31:19.157 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.0780000000000001e-06 2024-09-10T19:31:19.158 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:31:19.158 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:31:19.158 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:31:19.158 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:21:46.993347+0000", 2024-09-10T19:31:19.158 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.8405000000000004e-05 2024-09-10T19:31:19.158 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:31:19.158 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:31:19.158 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for peered", 2024-09-10T19:31:19.158 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:21:46.993354+0000", 2024-09-10T19:31:19.158 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.7390000000000002e-06 2024-09-10T19:31:19.158 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:31:19.158 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:31:19.158 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:31:19.159 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:21:48.022236+0000", 2024-09-10T19:31:19.159 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.028881905 2024-09-10T19:31:19.159 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:31:19.159 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:31:19.159 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-10T19:31:19.159 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:21:48.022331+0000", 2024-09-10T19:31:19.159 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.4734000000000005e-05 2024-09-10T19:31:19.159 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:31:19.159 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:31:19.159 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 5", 2024-09-10T19:31:19.159 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:21:48.022436+0000", 2024-09-10T19:31:19.159 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00010508799999999999 2024-09-10T19:31:19.159 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:31:19.160 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:31:19.160 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-09-10T19:31:19.160 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:21:48.027646+0000", 2024-09-10T19:31:19.160 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.005209984 2024-09-10T19:31:19.160 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:31:19.160 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:31:19.160 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:31:19.160 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:21:48.033863+0000", 2024-09-10T19:31:19.160 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0062175030000000001 2024-09-10T19:31:19.160 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:31:19.160 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:31:19.160 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-09-10T19:31:19.161 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:21:48.033907+0000", 2024-09-10T19:31:19.161 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.3958000000000002e-05 2024-09-10T19:31:19.161 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:31:19.161 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:31:19.161 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-10T19:31:19.161 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:21:48.033980+0000", 2024-09-10T19:31:19.161 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.3159999999999997e-05 2024-09-10T19:31:19.161 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:31:19.161 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-10T19:31:19.161 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:31:19.161 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:31:19.161 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:31:19.161 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.25622.0:216511 10.c 10:39bc3006:::benchmark_data_smithi183_176537_object93303:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e355)", 2024-09-10T19:31:19.161 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-10T19:27:01.151163+0000", 2024-09-10T19:31:19.162 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 257.97499875699998, 2024-09-10T19:31:19.162 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.12147474, 2024-09-10T19:31:19.162 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-10T19:31:19.162 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T19:31:19.162 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-10T19:31:19.162 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.25622", 2024-09-10T19:31:19.162 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.183:0/3685189199", 2024-09-10T19:31:19.162 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 216511 2024-09-10T19:31:19.162 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:31:19.162 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-10T19:31:19.162 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:31:19.162 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-10T19:31:19.162 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:27:01.151163+0000", 2024-09-10T19:31:19.162 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:31:19.162 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:31:19.163 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:31:19.163 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-10T19:31:19.163 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:27:01.151163+0000", 2024-09-10T19:31:19.163 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:31:19.163 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:31:19.163 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:31:19.163 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-10T19:31:19.163 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:27:01.151162+0000", 2024-09-10T19:31:19.163 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.999999 2024-09-10T19:31:19.163 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:31:19.163 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:31:19.163 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-10T19:31:19.163 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:27:01.151169+0000", 2024-09-10T19:31:19.163 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.6390000000000003e-06 2024-09-10T19:31:19.164 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:31:19.164 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:31:19.164 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-10T19:31:19.164 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:27:01.250667+0000", 2024-09-10T19:31:19.164 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.099497944000000005 2024-09-10T19:31:19.164 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:31:19.164 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:31:19.164 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-10T19:31:19.164 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:27:01.250692+0000", 2024-09-10T19:31:19.164 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.5034000000000001e-05 2024-09-10T19:31:19.164 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:31:19.164 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:31:19.164 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:31:19.164 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:27:01.256038+0000", 2024-09-10T19:31:19.165 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0053465759999999996 2024-09-10T19:31:19.165 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:31:19.165 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:31:19.165 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-10T19:31:19.165 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:27:01.256145+0000", 2024-09-10T19:31:19.165 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00010721399999999999 2024-09-10T19:31:19.165 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:31:19.165 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:31:19.165 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0,4", 2024-09-10T19:31:19.165 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:27:01.256272+0000", 2024-09-10T19:31:19.165 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000126673 2024-09-10T19:31:19.165 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:31:19.165 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:31:19.165 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:31:19.166 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:27:01.269436+0000", 2024-09-10T19:31:19.166 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.013163846 2024-09-10T19:31:19.166 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:31:19.166 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:31:19.166 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:31:19.166 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:27:01.269615+0000", 2024-09-10T19:31:19.166 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00017960200000000001 2024-09-10T19:31:19.166 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:31:19.166 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:31:19.166 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-09-10T19:31:19.166 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:27:01.272523+0000", 2024-09-10T19:31:19.166 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.002907621 2024-09-10T19:31:19.166 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:31:19.166 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:31:19.167 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-09-10T19:31:19.167 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:27:01.272602+0000", 2024-09-10T19:31:19.167 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.8578999999999994e-05 2024-09-10T19:31:19.167 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:31:19.167 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:31:19.167 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-10T19:31:19.167 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:27:01.272638+0000", 2024-09-10T19:31:19.167 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.6164999999999997e-05 2024-09-10T19:31:19.167 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:31:19.167 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-10T19:31:19.167 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:31:19.167 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:31:19.167 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:31:19.167 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.16026.0:39810 11.4 11:2b810bb5:::benchmark_data_smithi183_203944_object39809:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e380)", 2024-09-10T19:31:19.167 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-10T19:28:09.213968+0000", 2024-09-10T19:31:19.168 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 189.912194113, 2024-09-10T19:31:19.168 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.124633437, 2024-09-10T19:31:19.168 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-10T19:31:19.168 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T19:31:19.168 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-10T19:31:19.168 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.16026", 2024-09-10T19:31:19.168 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.183:0/1450567484", 2024-09-10T19:31:19.168 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 39810 2024-09-10T19:31:19.168 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:31:19.168 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-10T19:31:19.168 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:31:19.168 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-10T19:31:19.168 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:28:09.213968+0000", 2024-09-10T19:31:19.169 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:31:19.169 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:31:19.169 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:31:19.169 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-10T19:31:19.169 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:28:09.213968+0000", 2024-09-10T19:31:19.169 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:31:19.169 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:31:19.169 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:31:19.169 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-10T19:31:19.169 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:28:09.213966+0000", 2024-09-10T19:31:19.170 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999986 2024-09-10T19:31:19.170 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:31:19.170 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:31:19.170 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-10T19:31:19.170 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:28:09.214021+0000", 2024-09-10T19:31:19.170 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.4175999999999998e-05 2024-09-10T19:31:19.170 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:31:19.170 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:31:19.170 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-10T19:31:19.170 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:28:09.214030+0000", 2024-09-10T19:31:19.170 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.5810000000000003e-06 2024-09-10T19:31:19.170 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:31:19.170 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:31:19.170 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-10T19:31:19.170 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:28:09.214034+0000", 2024-09-10T19:31:19.171 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.121e-06 2024-09-10T19:31:19.171 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:31:19.171 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:31:19.171 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:31:19.171 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:28:09.214073+0000", 2024-09-10T19:31:19.171 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.8257999999999999e-05 2024-09-10T19:31:19.171 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:31:19.171 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:31:19.171 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-10T19:31:19.171 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:28:09.214185+0000", 2024-09-10T19:31:19.171 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00011234300000000001 2024-09-10T19:31:19.171 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:31:19.171 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:31:19.172 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 1,6", 2024-09-10T19:31:19.172 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:28:09.214272+0000", 2024-09-10T19:31:19.172 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.6638999999999995e-05 2024-09-10T19:31:19.172 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:31:19.172 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:31:19.172 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-09-10T19:31:19.172 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:28:09.215507+0000", 2024-09-10T19:31:19.172 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0012353469999999999 2024-09-10T19:31:19.172 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:31:19.172 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:31:19.172 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:31:19.172 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:28:09.215833+0000", 2024-09-10T19:31:19.172 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00032615200000000002 2024-09-10T19:31:19.172 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:31:19.173 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:31:19.173 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:31:19.173 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:28:09.338504+0000", 2024-09-10T19:31:19.173 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.122671185 2024-09-10T19:31:19.173 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:31:19.173 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:31:19.173 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-09-10T19:31:19.173 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:28:09.338558+0000", 2024-09-10T19:31:19.173 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.3859000000000001e-05 2024-09-10T19:31:19.173 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:31:19.173 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:31:19.173 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-10T19:31:19.173 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:28:09.338601+0000", 2024-09-10T19:31:19.173 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.3031999999999997e-05 2024-09-10T19:31:19.174 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:31:19.174 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-10T19:31:19.174 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:31:19.174 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:31:19.174 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:31:19.174 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.16026.0:40312 11.f 11:fc5fdb0b:::benchmark_data_smithi183_203944_object40311:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e380)", 2024-09-10T19:31:19.174 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-10T19:28:09.566104+0000", 2024-09-10T19:31:19.174 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 189.560057519, 2024-09-10T19:31:19.174 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.131990037, 2024-09-10T19:31:19.174 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-10T19:31:19.174 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T19:31:19.174 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-10T19:31:19.174 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.16026", 2024-09-10T19:31:19.174 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.183:0/1450567484", 2024-09-10T19:31:19.174 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 40312 2024-09-10T19:31:19.175 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:31:19.175 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-10T19:31:19.175 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:31:19.175 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-10T19:31:19.175 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:28:09.566104+0000", 2024-09-10T19:31:19.175 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:31:19.175 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:31:19.175 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:31:19.175 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-10T19:31:19.175 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:28:09.566104+0000", 2024-09-10T19:31:19.175 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:31:19.175 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:31:19.175 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:31:19.175 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-10T19:31:19.176 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:28:09.566103+0000", 2024-09-10T19:31:19.176 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999986 2024-09-10T19:31:19.176 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:31:19.176 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:31:19.176 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-10T19:31:19.176 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:28:09.566150+0000", 2024-09-10T19:31:19.176 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.7122e-05 2024-09-10T19:31:19.176 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:31:19.176 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:31:19.176 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-10T19:31:19.176 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:28:09.566167+0000", 2024-09-10T19:31:19.176 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.6764000000000001e-05 2024-09-10T19:31:19.176 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:31:19.176 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:31:19.177 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-10T19:31:19.177 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:28:09.566172+0000", 2024-09-10T19:31:19.177 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.49e-06 2024-09-10T19:31:19.177 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:31:19.177 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:31:19.177 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:31:19.177 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:28:09.566218+0000", 2024-09-10T19:31:19.177 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.5982999999999997e-05 2024-09-10T19:31:19.177 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:31:19.177 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:31:19.177 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-10T19:31:19.177 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:28:09.566328+0000", 2024-09-10T19:31:19.177 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000109429 2024-09-10T19:31:19.178 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:31:19.178 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:31:19.178 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0,1", 2024-09-10T19:31:19.178 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:28:09.566455+0000", 2024-09-10T19:31:19.178 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00012701300000000001 2024-09-10T19:31:19.178 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:31:19.178 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:31:19.178 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-09-10T19:31:19.178 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:28:09.568380+0000", 2024-09-10T19:31:19.178 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0019255489999999999 2024-09-10T19:31:19.178 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:31:19.178 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:31:19.178 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:31:19.178 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:28:09.568409+0000", 2024-09-10T19:31:19.179 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.8918e-05 2024-09-10T19:31:19.179 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:31:19.179 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:31:19.179 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:31:19.179 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:28:09.698008+0000", 2024-09-10T19:31:19.179 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.129598926 2024-09-10T19:31:19.179 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:31:19.179 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:31:19.179 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-09-10T19:31:19.179 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:28:09.698058+0000", 2024-09-10T19:31:19.179 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.9369000000000001e-05 2024-09-10T19:31:19.179 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:31:19.179 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:31:19.179 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-10T19:31:19.179 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:28:09.698094+0000", 2024-09-10T19:31:19.180 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.6767000000000002e-05 2024-09-10T19:31:19.180 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:31:19.180 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-10T19:31:19.180 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:31:19.180 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:31:19.180 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:31:19.180 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.16026.0:40317 11.4 11:2bceac6c:::benchmark_data_smithi183_203944_object40316:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e380)", 2024-09-10T19:31:19.180 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-10T19:28:09.567844+0000", 2024-09-10T19:31:19.180 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 189.55831823299999, 2024-09-10T19:31:19.180 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.13031211100000001, 2024-09-10T19:31:19.180 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-10T19:31:19.180 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T19:31:19.180 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-10T19:31:19.180 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.16026", 2024-09-10T19:31:19.181 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.183:0/1450567484", 2024-09-10T19:31:19.181 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 40317 2024-09-10T19:31:19.181 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:31:19.181 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-10T19:31:19.181 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:31:19.181 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-10T19:31:19.181 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:28:09.567844+0000", 2024-09-10T19:31:19.181 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:31:19.181 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:31:19.181 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:31:19.181 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-10T19:31:19.181 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:28:09.567844+0000", 2024-09-10T19:31:19.181 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:31:19.181 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:31:19.182 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:31:19.182 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-10T19:31:19.182 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:28:09.567843+0000", 2024-09-10T19:31:19.182 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999995 2024-09-10T19:31:19.182 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:31:19.182 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:31:19.182 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-10T19:31:19.182 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:28:09.567867+0000", 2024-09-10T19:31:19.182 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.4513999999999999e-05 2024-09-10T19:31:19.182 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:31:19.182 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:31:19.182 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-10T19:31:19.182 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:28:09.567878+0000", 2024-09-10T19:31:19.182 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.0774000000000001e-05 2024-09-10T19:31:19.183 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:31:19.183 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:31:19.183 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-10T19:31:19.183 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:28:09.567884+0000", 2024-09-10T19:31:19.183 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.7060000000000001e-06 2024-09-10T19:31:19.183 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:31:19.183 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:31:19.183 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:31:19.183 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:28:09.567931+0000", 2024-09-10T19:31:19.183 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.7444000000000002e-05 2024-09-10T19:31:19.183 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:31:19.183 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:31:19.183 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-10T19:31:19.183 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:28:09.568044+0000", 2024-09-10T19:31:19.184 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000112256 2024-09-10T19:31:19.184 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:31:19.184 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:31:19.184 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 1,6", 2024-09-10T19:31:19.184 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:28:09.568172+0000", 2024-09-10T19:31:19.184 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00012878499999999999 2024-09-10T19:31:19.184 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:31:19.184 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:31:19.184 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:31:19.184 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:28:09.570661+0000", 2024-09-10T19:31:19.184 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0024890350000000001 2024-09-10T19:31:19.184 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:31:19.184 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:31:19.184 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-09-10T19:31:19.185 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:28:09.570773+0000", 2024-09-10T19:31:19.185 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000111778 2024-09-10T19:31:19.185 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:31:19.185 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:31:19.185 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:31:19.185 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:28:09.698038+0000", 2024-09-10T19:31:19.185 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.12726463299999999 2024-09-10T19:31:19.185 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:31:19.185 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:31:19.185 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-09-10T19:31:19.186 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:28:09.698113+0000", 2024-09-10T19:31:19.186 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.4751000000000003e-05 2024-09-10T19:31:19.186 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:31:19.186 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:31:19.186 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-10T19:31:19.186 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:28:09.698156+0000", 2024-09-10T19:31:19.186 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.3145000000000001e-05 2024-09-10T19:31:19.186 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:31:19.186 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-10T19:31:19.186 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:31:19.186 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:31:19.186 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:31:19.186 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.16026.0:109593 11.5 11:ad5391cd:::benchmark_data_smithi183_203944_object109592:head [set-alloc-hint object_size 65536 write_size 65536,write 0~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e398)", 2024-09-10T19:31:19.186 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-10T19:28:57.830784+0000", 2024-09-10T19:31:19.187 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 141.295377333, 2024-09-10T19:31:19.187 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.475219161, 2024-09-10T19:31:19.187 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-10T19:31:19.187 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T19:31:19.187 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-10T19:31:19.187 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.16026", 2024-09-10T19:31:19.187 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.183:0/1450567484", 2024-09-10T19:31:19.187 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 109593 2024-09-10T19:31:19.187 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:31:19.187 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-10T19:31:19.187 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:31:19.187 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-10T19:31:19.187 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:28:57.830784+0000", 2024-09-10T19:31:19.188 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:31:19.188 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:31:19.188 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:31:19.188 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-10T19:31:19.188 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:28:57.830784+0000", 2024-09-10T19:31:19.188 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:31:19.188 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:31:19.188 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:31:19.188 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-10T19:31:19.188 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:28:57.830783+0000", 2024-09-10T19:31:19.188 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999986 2024-09-10T19:31:19.188 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:31:19.188 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:31:19.189 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-10T19:31:19.189 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:28:57.830830+0000", 2024-09-10T19:31:19.189 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.7148999999999999e-05 2024-09-10T19:31:19.189 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:31:19.189 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:31:19.189 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-10T19:31:19.189 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:28:57.830842+0000", 2024-09-10T19:31:19.189 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.1421e-05 2024-09-10T19:31:19.189 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:31:19.189 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:31:19.189 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-10T19:31:19.189 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:28:57.830848+0000", 2024-09-10T19:31:19.189 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.0650000000000004e-06 2024-09-10T19:31:19.190 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:31:19.190 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:31:19.190 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:31:19.190 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:28:57.830897+0000", 2024-09-10T19:31:19.190 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.9422000000000002e-05 2024-09-10T19:31:19.190 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:31:19.190 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:31:19.190 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-10T19:31:19.190 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:28:57.831041+0000", 2024-09-10T19:31:19.190 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00014343900000000001 2024-09-10T19:31:19.190 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:31:19.190 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:31:19.190 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 5,6", 2024-09-10T19:31:19.190 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:28:57.831196+0000", 2024-09-10T19:31:19.191 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00015483599999999999 2024-09-10T19:31:19.191 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:31:19.191 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:31:19.191 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-09-10T19:31:19.191 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:28:57.833437+0000", 2024-09-10T19:31:19.191 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0022415540000000002 2024-09-10T19:31:19.191 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:31:19.191 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:31:19.191 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:31:19.191 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:28:57.833897+0000", 2024-09-10T19:31:19.191 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00045953200000000002 2024-09-10T19:31:19.191 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:31:19.191 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:31:19.192 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:31:19.192 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:28:58.305917+0000", 2024-09-10T19:31:19.192 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.472020261 2024-09-10T19:31:19.192 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:31:19.192 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:31:19.192 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-09-10T19:31:19.192 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:28:58.305964+0000", 2024-09-10T19:31:19.192 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.7456000000000001e-05 2024-09-10T19:31:19.192 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:31:19.192 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:31:19.192 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-10T19:31:19.192 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:28:58.306004+0000", 2024-09-10T19:31:19.192 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.9280999999999999e-05 2024-09-10T19:31:19.192 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:31:19.193 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-10T19:31:19.193 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:31:19.193 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:31:19.193 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:31:19.193 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.16026.0:227638 11.f 11:f4a90410:::benchmark_data_smithi183_203944_object82748:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e415)", 2024-09-10T19:31:19.193 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-10T19:30:21.516881+0000", 2024-09-10T19:31:19.193 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 57.609280345000002, 2024-09-10T19:31:19.193 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.23818558200000001, 2024-09-10T19:31:19.193 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-10T19:31:19.193 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T19:31:19.193 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-10T19:31:19.193 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.16026", 2024-09-10T19:31:19.193 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.183:0/1450567484", 2024-09-10T19:31:19.193 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 227638 2024-09-10T19:31:19.194 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:31:19.194 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-10T19:31:19.194 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:31:19.194 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-10T19:31:19.194 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:30:21.516881+0000", 2024-09-10T19:31:19.194 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:31:19.194 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:31:19.194 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:31:19.194 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-10T19:31:19.194 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:30:21.516881+0000", 2024-09-10T19:31:19.194 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:31:19.194 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:31:19.194 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:31:19.194 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-10T19:31:19.195 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:30:21.516880+0000", 2024-09-10T19:31:19.195 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999986 2024-09-10T19:31:19.195 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:31:19.195 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:31:19.195 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-10T19:31:19.195 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:30:21.516886+0000", 2024-09-10T19:31:19.195 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.7309999999999998e-06 2024-09-10T19:31:19.195 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:31:19.195 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:31:19.195 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-10T19:31:19.195 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:30:21.516898+0000", 2024-09-10T19:31:19.195 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.1846e-05 2024-09-10T19:31:19.195 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:31:19.195 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:31:19.196 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-10T19:31:19.196 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:30:21.516902+0000", 2024-09-10T19:31:19.196 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.6e-06 2024-09-10T19:31:19.196 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:31:19.196 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:31:19.196 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:31:19.196 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:30:21.516951+0000", 2024-09-10T19:31:19.196 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.8479000000000001e-05 2024-09-10T19:31:19.196 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:31:19.196 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:31:19.197 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:31:19.197 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph osd set noscrub 2024-09-10T19:31:19.198 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-10T19:31:19.198 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:30:21.517064+0000", 2024-09-10T19:31:19.198 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00011352 2024-09-10T19:31:19.198 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:31:19.198 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:31:19.199 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0,1", 2024-09-10T19:31:19.199 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:30:21.517236+0000", 2024-09-10T19:31:19.199 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00017193199999999999 2024-09-10T19:31:19.199 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:31:19.199 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:31:19.199 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-09-10T19:31:19.199 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:30:21.519760+0000", 2024-09-10T19:31:19.199 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0025239469999999999 2024-09-10T19:31:19.199 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:31:19.199 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:31:19.199 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:31:19.199 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:30:21.521983+0000", 2024-09-10T19:31:19.199 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0022231579999999998 2024-09-10T19:31:19.200 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:31:19.200 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:31:19.200 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:31:19.200 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:30:21.754969+0000", 2024-09-10T19:31:19.200 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.232985579 2024-09-10T19:31:19.200 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:31:19.200 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:31:19.200 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-09-10T19:31:19.200 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:30:21.755025+0000", 2024-09-10T19:31:19.200 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.6023999999999998e-05 2024-09-10T19:31:19.200 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:31:19.200 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:31:19.200 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-10T19:31:19.200 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:30:21.755067+0000", 2024-09-10T19:31:19.201 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.1974999999999997e-05 2024-09-10T19:31:19.201 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:31:19.201 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-10T19:31:19.201 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:31:19.201 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:31:19.201 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:31:19.201 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.16026.0:227726 11.f 11:f4200024:::benchmark_data_smithi183_203944_object82836:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e415)", 2024-09-10T19:31:19.201 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-10T19:30:21.819058+0000", 2024-09-10T19:31:19.201 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 57.307103951999999, 2024-09-10T19:31:19.201 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.14132866199999999, 2024-09-10T19:31:19.201 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-10T19:31:19.201 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T19:31:19.201 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-10T19:31:19.201 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.16026", 2024-09-10T19:31:19.202 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.183:0/1450567484", 2024-09-10T19:31:19.202 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 227726 2024-09-10T19:31:19.202 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:31:19.202 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-10T19:31:19.202 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:31:19.202 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-10T19:31:19.202 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:30:21.819058+0000", 2024-09-10T19:31:19.202 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:31:19.202 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:31:19.202 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:31:19.202 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-10T19:31:19.202 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:30:21.819058+0000", 2024-09-10T19:31:19.202 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:31:19.202 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:31:19.203 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:31:19.203 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-10T19:31:19.203 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:30:21.819057+0000", 2024-09-10T19:31:19.203 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.999999 2024-09-10T19:31:19.203 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:31:19.203 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:31:19.203 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-10T19:31:19.203 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:30:21.819061+0000", 2024-09-10T19:31:19.203 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.031e-06 2024-09-10T19:31:19.203 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:31:19.203 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:31:19.203 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-10T19:31:19.203 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:30:21.819072+0000", 2024-09-10T19:31:19.203 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.1072e-05 2024-09-10T19:31:19.204 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:31:19.204 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:31:19.204 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-10T19:31:19.204 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:30:21.819078+0000", 2024-09-10T19:31:19.204 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.2859999999999996e-06 2024-09-10T19:31:19.204 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:31:19.204 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:31:19.204 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:31:19.204 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:30:21.819117+0000", 2024-09-10T19:31:19.204 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.8936000000000001e-05 2024-09-10T19:31:19.204 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:31:19.204 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:31:19.204 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-10T19:31:19.204 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:30:21.819195+0000", 2024-09-10T19:31:19.205 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.7972999999999994e-05 2024-09-10T19:31:19.205 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:31:19.205 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:31:19.205 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0,1", 2024-09-10T19:31:19.205 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:30:21.819262+0000", 2024-09-10T19:31:19.205 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.6563000000000006e-05 2024-09-10T19:31:19.205 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:31:19.205 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:31:19.205 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-09-10T19:31:19.205 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:30:21.821020+0000", 2024-09-10T19:31:19.205 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0017582839999999999 2024-09-10T19:31:19.205 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:31:19.205 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:31:19.205 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:31:19.206 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:30:21.821163+0000", 2024-09-10T19:31:19.206 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000143282 2024-09-10T19:31:19.206 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:31:19.206 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:31:19.206 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:31:19.206 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:30:21.960313+0000", 2024-09-10T19:31:19.206 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.13915002300000001 2024-09-10T19:31:19.206 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:31:19.206 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:31:19.206 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-09-10T19:31:19.206 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:30:21.960359+0000", 2024-09-10T19:31:19.206 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.5503999999999997e-05 2024-09-10T19:31:19.206 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:31:19.206 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:31:19.207 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-10T19:31:19.207 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:30:21.960387+0000", 2024-09-10T19:31:19.207 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.7579000000000001e-05 2024-09-10T19:31:19.207 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:31:19.207 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-10T19:31:19.207 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:31:19.207 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:31:19.207 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:31:19.207 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.16026.0:228891 11.f 11:fb224b67:::benchmark_data_smithi183_203944_object84001:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e415)", 2024-09-10T19:31:19.207 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-10T19:30:22.882926+0000", 2024-09-10T19:31:19.207 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 56.243235415000001, 2024-09-10T19:31:19.207 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.31831812199999998, 2024-09-10T19:31:19.207 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-10T19:31:19.207 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T19:31:19.208 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-10T19:31:19.208 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.16026", 2024-09-10T19:31:19.208 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.183:0/1450567484", 2024-09-10T19:31:19.208 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 228891 2024-09-10T19:31:19.208 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:31:19.208 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-10T19:31:19.208 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:31:19.208 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-10T19:31:19.208 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:30:22.882926+0000", 2024-09-10T19:31:19.208 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:31:19.208 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:31:19.208 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:31:19.208 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-10T19:31:19.208 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:30:22.882926+0000", 2024-09-10T19:31:19.209 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:31:19.209 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:31:19.209 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:31:19.209 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-10T19:31:19.209 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:30:22.882925+0000", 2024-09-10T19:31:19.209 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.999999 2024-09-10T19:31:19.209 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:31:19.209 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:31:19.209 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-10T19:31:19.209 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:30:22.882931+0000", 2024-09-10T19:31:19.209 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.3519999999999997e-06 2024-09-10T19:31:19.209 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:31:19.209 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:31:19.209 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-10T19:31:19.210 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:30:22.882941+0000", 2024-09-10T19:31:19.210 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.0661e-05 2024-09-10T19:31:19.210 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:31:19.210 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:31:19.210 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-10T19:31:19.210 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:30:22.882947+0000", 2024-09-10T19:31:19.210 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.8880000000000002e-06 2024-09-10T19:31:19.210 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:31:19.210 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:31:19.210 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:31:19.210 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:30:22.882983+0000", 2024-09-10T19:31:19.210 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.5698999999999998e-05 2024-09-10T19:31:19.210 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:31:19.210 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:31:19.211 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-10T19:31:19.211 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:30:22.883116+0000", 2024-09-10T19:31:19.211 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000133292 2024-09-10T19:31:19.211 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:31:19.211 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:31:19.211 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0,1", 2024-09-10T19:31:19.211 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:30:22.883186+0000", 2024-09-10T19:31:19.211 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.0128000000000003e-05 2024-09-10T19:31:19.211 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:31:19.211 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:31:19.211 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-09-10T19:31:19.211 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:30:22.887398+0000", 2024-09-10T19:31:19.211 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0042117830000000002 2024-09-10T19:31:19.211 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:31:19.212 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:31:19.212 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:31:19.212 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:30:22.890370+0000", 2024-09-10T19:31:19.212 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0029720749999999998 2024-09-10T19:31:19.212 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:31:19.212 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:31:19.212 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:31:19.212 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:30:23.201150+0000", 2024-09-10T19:31:19.212 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.31078027899999999 2024-09-10T19:31:19.212 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:31:19.212 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:31:19.212 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-09-10T19:31:19.212 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:30:23.201202+0000", 2024-09-10T19:31:19.213 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.1545000000000002e-05 2024-09-10T19:31:19.213 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:31:19.213 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:31:19.213 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-10T19:31:19.213 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:30:23.201245+0000", 2024-09-10T19:31:19.216 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.2524999999999997e-05 2024-09-10T19:31:19.216 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:31:19.216 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-10T19:31:19.216 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:31:19.216 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:31:19.216 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:31:19.217 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.16026.0:228896 11.f 11:f9d2ea2e:::benchmark_data_smithi183_203944_object84006:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e415)", 2024-09-10T19:31:19.217 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-10T19:30:22.883337+0000", 2024-09-10T19:31:19.217 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 56.242824390999999, 2024-09-10T19:31:19.217 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.32019146100000001, 2024-09-10T19:31:19.217 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-10T19:31:19.217 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T19:31:19.217 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-10T19:31:19.217 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.16026", 2024-09-10T19:31:19.217 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.183:0/1450567484", 2024-09-10T19:31:19.217 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 228896 2024-09-10T19:31:19.217 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:31:19.217 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-10T19:31:19.217 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:31:19.218 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-10T19:31:19.218 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:30:22.883337+0000", 2024-09-10T19:31:19.218 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:31:19.218 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:31:19.218 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:31:19.218 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-10T19:31:19.218 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:30:22.883337+0000", 2024-09-10T19:31:19.218 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:31:19.218 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:31:19.218 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:31:19.218 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-10T19:31:19.218 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:30:22.883337+0000", 2024-09-10T19:31:19.218 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999995 2024-09-10T19:31:19.218 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:31:19.218 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:31:19.219 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-10T19:31:19.219 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:30:22.883341+0000", 2024-09-10T19:31:19.219 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.1489999999999996e-06 2024-09-10T19:31:19.219 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:31:19.219 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:31:19.219 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-10T19:31:19.219 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:30:22.883351+0000", 2024-09-10T19:31:19.219 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.0104e-05 2024-09-10T19:31:19.219 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:31:19.219 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:31:19.219 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-10T19:31:19.219 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:30:22.883355+0000", 2024-09-10T19:31:19.219 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.9929999999999997e-06 2024-09-10T19:31:19.219 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:31:19.220 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:31:19.220 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:31:19.220 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:30:22.885122+0000", 2024-09-10T19:31:19.220 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0017674570000000001 2024-09-10T19:31:19.220 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:31:19.220 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:31:19.220 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-10T19:31:19.220 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:30:22.885231+0000", 2024-09-10T19:31:19.220 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00010851799999999999 2024-09-10T19:31:19.220 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:31:19.220 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:31:19.220 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0,1", 2024-09-10T19:31:19.220 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:30:22.885313+0000", 2024-09-10T19:31:19.221 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.1934999999999994e-05 2024-09-10T19:31:19.221 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:31:19.221 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:31:19.221 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-09-10T19:31:19.221 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:30:22.887535+0000", 2024-09-10T19:31:19.221 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0022217920000000002 2024-09-10T19:31:19.221 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:31:19.221 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:31:19.221 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:31:19.221 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:30:22.890409+0000", 2024-09-10T19:31:19.221 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.002874241 2024-09-10T19:31:19.221 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:31:19.221 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:31:19.221 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:31:19.222 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:30:23.203443+0000", 2024-09-10T19:31:19.222 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.31303446499999998 2024-09-10T19:31:19.222 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:31:19.222 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:31:19.222 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-09-10T19:31:19.222 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:30:23.203488+0000", 2024-09-10T19:31:19.222 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.4626999999999998e-05 2024-09-10T19:31:19.222 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:31:19.222 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:31:19.222 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-10T19:31:19.222 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:30:23.203529+0000", 2024-09-10T19:31:19.222 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.0893000000000003e-05 2024-09-10T19:31:19.222 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:31:19.222 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-10T19:31:19.223 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:31:19.223 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:31:19.223 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:31:19.223 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.16026.0:228904 11.f 11:f81405ba:::benchmark_data_smithi183_203944_object84014:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e415)", 2024-09-10T19:31:19.223 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-10T19:30:22.891888+0000", 2024-09-10T19:31:19.223 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 56.234273864000002, 2024-09-10T19:31:19.223 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.31332881800000001, 2024-09-10T19:31:19.223 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-10T19:31:19.223 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T19:31:19.223 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-10T19:31:19.223 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.16026", 2024-09-10T19:31:19.223 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.183:0/1450567484", 2024-09-10T19:31:19.223 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 228904 2024-09-10T19:31:19.223 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:31:19.224 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-10T19:31:19.224 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:31:19.224 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-10T19:31:19.224 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:30:22.891888+0000", 2024-09-10T19:31:19.224 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:31:19.224 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:31:19.224 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:31:19.224 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-10T19:31:19.224 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:30:22.891888+0000", 2024-09-10T19:31:19.224 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:31:19.224 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:31:19.224 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:31:19.224 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-10T19:31:19.224 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:30:22.891886+0000", 2024-09-10T19:31:19.225 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999986 2024-09-10T19:31:19.225 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:31:19.225 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:31:19.225 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-10T19:31:19.225 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:30:22.891892+0000", 2024-09-10T19:31:19.225 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.0059999999999998e-06 2024-09-10T19:31:19.225 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:31:19.225 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:31:19.225 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-10T19:31:19.225 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:30:22.891907+0000", 2024-09-10T19:31:19.225 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.49e-05 2024-09-10T19:31:19.225 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:31:19.225 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:31:19.225 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-10T19:31:19.226 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:30:22.891913+0000", 2024-09-10T19:31:19.226 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.4229999999999996e-06 2024-09-10T19:31:19.226 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:31:19.226 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:31:19.226 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:31:19.226 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:30:22.891954+0000", 2024-09-10T19:31:19.226 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.1554999999999998e-05 2024-09-10T19:31:19.226 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:31:19.226 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:31:19.226 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-10T19:31:19.226 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:30:22.892056+0000", 2024-09-10T19:31:19.226 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00010134899999999999 2024-09-10T19:31:19.226 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:31:19.226 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:31:19.227 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0,1", 2024-09-10T19:31:19.227 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:30:22.892167+0000", 2024-09-10T19:31:19.227 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000111734 2024-09-10T19:31:19.227 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:31:19.227 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:31:19.227 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-09-10T19:31:19.227 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:30:22.894418+0000", 2024-09-10T19:31:19.227 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0022503520000000002 2024-09-10T19:31:19.227 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:31:19.227 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:31:19.227 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:31:19.227 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:30:22.894604+0000", 2024-09-10T19:31:19.227 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00018644899999999999 2024-09-10T19:31:19.227 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:31:19.228 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:31:19.228 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:31:19.228 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:30:23.205165+0000", 2024-09-10T19:31:19.228 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.31056067799999998 2024-09-10T19:31:19.228 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:31:19.228 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:31:19.228 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-09-10T19:31:19.228 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:30:23.205192+0000", 2024-09-10T19:31:19.228 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.7112000000000001e-05 2024-09-10T19:31:19.228 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:31:19.228 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:31:19.228 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-10T19:31:19.228 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:30:23.205217+0000", 2024-09-10T19:31:19.228 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.4791e-05 2024-09-10T19:31:19.229 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:31:19.229 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-10T19:31:19.229 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:31:19.229 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:31:19.229 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:31:19.229 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_repop(client.16026.0:275574 11.2 e422/371 11:4b64dc8e:::benchmark_data_smithi183_203944_object130684:head v 422'17064, mlcod=422'17060)", 2024-09-10T19:31:19.229 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-10T19:30:56.414001+0000", 2024-09-10T19:31:19.229 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 22.712160690000001, 2024-09-10T19:31:19.229 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.544578059, 2024-09-10T19:31:19.229 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-10T19:31:19.230 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T19:31:19.230 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-10T19:31:19.230 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:31:19.230 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-10T19:31:19.230 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:30:56.414001+0000", 2024-09-10T19:31:19.230 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:31:19.230 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:31:19.230 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:31:19.230 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-10T19:31:19.230 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:30:56.414001+0000", 2024-09-10T19:31:19.230 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:31:19.230 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:31:19.230 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:31:19.230 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-10T19:31:19.231 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:30:56.414000+0000", 2024-09-10T19:31:19.231 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.999999 2024-09-10T19:31:19.231 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:31:19.231 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:31:19.231 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-10T19:31:19.231 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:30:56.414009+0000", 2024-09-10T19:31:19.231 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.2390000000000002e-06 2024-09-10T19:31:19.231 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:31:19.231 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:31:19.231 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-10T19:31:19.231 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:30:56.414017+0000", 2024-09-10T19:31:19.231 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.3690000000000007e-06 2024-09-10T19:31:19.231 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:31:19.231 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:31:19.232 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-10T19:31:19.232 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:30:56.414021+0000", 2024-09-10T19:31:19.232 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.2409999999999997e-06 2024-09-10T19:31:19.232 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:31:19.232 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:31:19.232 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:31:19.232 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:30:56.414047+0000", 2024-09-10T19:31:19.232 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.5763999999999999e-05 2024-09-10T19:31:19.232 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:31:19.232 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:31:19.232 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-10T19:31:19.232 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:30:56.414058+0000", 2024-09-10T19:31:19.232 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.1326e-05 2024-09-10T19:31:19.233 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:31:19.233 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:31:19.233 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-09-10T19:31:19.233 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:30:56.958543+0000", 2024-09-10T19:31:19.233 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.54448502600000004 2024-09-10T19:31:19.233 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:31:19.233 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:31:19.233 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-10T19:31:19.233 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:30:56.958579+0000", 2024-09-10T19:31:19.233 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.5855999999999997e-05 2024-09-10T19:31:19.233 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:31:19.233 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-10T19:31:19.233 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:31:19.233 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:31:19.234 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:31:19.234 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.16026.0:275584 11.f 11:f7f5a3d5:::benchmark_data_smithi183_203944_object130694:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e422)", 2024-09-10T19:31:19.234 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-10T19:30:56.414473+0000", 2024-09-10T19:31:19.234 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 22.711688402, 2024-09-10T19:31:19.234 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.54858507400000001, 2024-09-10T19:31:19.234 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-10T19:31:19.234 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T19:31:19.234 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-10T19:31:19.234 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.16026", 2024-09-10T19:31:19.234 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.183:0/1450567484", 2024-09-10T19:31:19.234 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 275584 2024-09-10T19:31:19.234 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:31:19.234 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-10T19:31:19.234 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:31:19.235 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-10T19:31:19.235 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:30:56.414473+0000", 2024-09-10T19:31:19.235 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:31:19.235 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:31:19.235 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:31:19.235 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-10T19:31:19.235 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:30:56.414473+0000", 2024-09-10T19:31:19.235 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:31:19.235 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:31:19.235 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:31:19.235 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-10T19:31:19.235 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:30:56.414472+0000", 2024-09-10T19:31:19.235 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999981 2024-09-10T19:31:19.235 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:31:19.236 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:31:19.236 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-10T19:31:19.236 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:30:56.414479+0000", 2024-09-10T19:31:19.236 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.0450000000000003e-06 2024-09-10T19:31:19.236 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:31:19.236 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:31:19.236 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-10T19:31:19.236 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:30:56.414491+0000", 2024-09-10T19:31:19.236 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.2668e-05 2024-09-10T19:31:19.236 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:31:19.236 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:31:19.236 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-10T19:31:19.236 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:30:56.414497+0000", 2024-09-10T19:31:19.236 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.6620000000000002e-06 2024-09-10T19:31:19.237 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:31:19.237 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:31:19.237 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:31:19.237 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:30:56.414544+0000", 2024-09-10T19:31:19.237 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.6489000000000002e-05 2024-09-10T19:31:19.237 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:31:19.237 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:31:19.237 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-10T19:31:19.237 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:30:56.414659+0000", 2024-09-10T19:31:19.238 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00011563800000000001 2024-09-10T19:31:19.238 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:31:19.238 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:31:19.238 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0,1", 2024-09-10T19:31:19.238 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:30:56.414776+0000", 2024-09-10T19:31:19.238 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00011692300000000001 2024-09-10T19:31:19.238 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:31:19.238 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:31:19.238 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:31:19.238 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:30:56.960411+0000", 2024-09-10T19:31:19.238 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.54563435000000005 2024-09-10T19:31:19.238 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:31:19.238 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:31:19.238 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:31:19.239 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:30:56.960455+0000", 2024-09-10T19:31:19.239 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.4712000000000002e-05 2024-09-10T19:31:19.239 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:31:19.239 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:31:19.239 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-09-10T19:31:19.239 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:30:56.962956+0000", 2024-09-10T19:31:19.239 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0025005219999999998 2024-09-10T19:31:19.239 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:31:19.239 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:31:19.239 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-09-10T19:31:19.239 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:30:56.963009+0000", 2024-09-10T19:31:19.239 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.3427000000000002e-05 2024-09-10T19:31:19.239 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:31:19.239 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:31:19.240 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-10T19:31:19.240 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:30:56.963059+0000", 2024-09-10T19:31:19.240 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.9326999999999997e-05 2024-09-10T19:31:19.240 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:31:19.240 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-10T19:31:19.240 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:31:19.240 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:31:19.240 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:31:19.240 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.16026.0:275588 11.f 11:fb0ccc66:::benchmark_data_smithi183_203944_object130698:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e422)", 2024-09-10T19:31:19.240 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-10T19:30:56.418230+0000", 2024-09-10T19:31:19.240 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 22.707932129, 2024-09-10T19:31:19.240 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.54502345600000002, 2024-09-10T19:31:19.240 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-10T19:31:19.240 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T19:31:19.241 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-10T19:31:19.241 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.16026", 2024-09-10T19:31:19.241 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.183:0/1450567484", 2024-09-10T19:31:19.241 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 275588 2024-09-10T19:31:19.241 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:31:19.241 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-10T19:31:19.241 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:31:19.241 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-10T19:31:19.241 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:30:56.418230+0000", 2024-09-10T19:31:19.241 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:31:19.241 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:31:19.241 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:31:19.241 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-10T19:31:19.241 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:30:56.418230+0000", 2024-09-10T19:31:19.242 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:31:19.242 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:31:19.242 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:31:19.242 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-10T19:31:19.242 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:30:56.418229+0000", 2024-09-10T19:31:19.242 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.999999 2024-09-10T19:31:19.242 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:31:19.242 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:31:19.242 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-10T19:31:19.242 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:30:56.418234+0000", 2024-09-10T19:31:19.242 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.186e-06 2024-09-10T19:31:19.242 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:31:19.242 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:31:19.242 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-10T19:31:19.243 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:30:56.957871+0000", 2024-09-10T19:31:19.243 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.53963676599999999 2024-09-10T19:31:19.243 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:31:19.243 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:31:19.243 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-10T19:31:19.243 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:30:56.957890+0000", 2024-09-10T19:31:19.243 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.8970999999999998e-05 2024-09-10T19:31:19.243 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:31:19.243 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:31:19.243 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:31:19.243 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:30:56.960479+0000", 2024-09-10T19:31:19.243 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0025896140000000001 2024-09-10T19:31:19.243 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:31:19.244 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:31:19.244 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-10T19:31:19.244 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:30:56.960616+0000", 2024-09-10T19:31:19.244 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00013660500000000001 2024-09-10T19:31:19.244 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:31:19.244 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:31:19.244 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 0,1", 2024-09-10T19:31:19.244 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:30:56.960732+0000", 2024-09-10T19:31:19.244 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000116105 2024-09-10T19:31:19.244 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:31:19.244 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:31:19.244 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:31:19.244 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:30:56.963102+0000", 2024-09-10T19:31:19.244 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.002370016 2024-09-10T19:31:19.245 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:31:19.245 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:31:19.245 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:31:19.245 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:30:56.963135+0000", 2024-09-10T19:31:19.245 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.2901999999999998e-05 2024-09-10T19:31:19.245 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:31:19.245 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:31:19.245 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-09-10T19:31:19.245 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:30:56.963178+0000", 2024-09-10T19:31:19.245 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.3037000000000001e-05 2024-09-10T19:31:19.245 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:31:19.245 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:31:19.245 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-09-10T19:31:19.245 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:30:56.963220+0000", 2024-09-10T19:31:19.246 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.1789000000000003e-05 2024-09-10T19:31:19.246 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:31:19.246 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:31:19.246 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-10T19:31:19.246 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:30:56.963253+0000", 2024-09-10T19:31:19.246 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.3246e-05 2024-09-10T19:31:19.246 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:31:19.246 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-10T19:31:19.246 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:31:19.246 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:31:19.246 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:31:19.246 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_repop_reply(client.16026.0:275584 11.f e422/371 ondisk, result = 0)", 2024-09-10T19:31:19.246 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-10T19:30:56.418460+0000", 2024-09-10T19:31:19.246 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 22.707701599, 2024-09-10T19:31:19.247 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.54199613599999996, 2024-09-10T19:31:19.247 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-10T19:31:19.247 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "started", 2024-09-10T19:31:19.247 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-10T19:31:19.247 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:31:19.247 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-10T19:31:19.247 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:30:56.418460+0000", 2024-09-10T19:31:19.247 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:31:19.247 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:31:19.247 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:31:19.247 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-10T19:31:19.247 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:30:56.418460+0000", 2024-09-10T19:31:19.247 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:31:19.247 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:31:19.248 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:31:19.248 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-10T19:31:19.248 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:30:56.418460+0000", 2024-09-10T19:31:19.248 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999995 2024-09-10T19:31:19.248 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:31:19.248 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:31:19.248 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-10T19:31:19.248 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:30:56.418467+0000", 2024-09-10T19:31:19.248 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 7.0740000000000004e-06 2024-09-10T19:31:19.248 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:31:19.248 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:31:19.248 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-10T19:31:19.248 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:30:56.958237+0000", 2024-09-10T19:31:19.248 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.53977006500000002 2024-09-10T19:31:19.248 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:31:19.249 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:31:19.249 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-10T19:31:19.249 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:30:56.958243+0000", 2024-09-10T19:31:19.249 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.5599999999999999e-06 2024-09-10T19:31:19.249 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:31:19.249 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:31:19.249 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:31:19.249 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:30:56.960440+0000", 2024-09-10T19:31:19.249 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0021967610000000002 2024-09-10T19:31:19.249 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:31:19.249 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:31:19.249 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-10T19:31:19.249 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:30:56.960451+0000", 2024-09-10T19:31:19.250 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.0703000000000001e-05 2024-09-10T19:31:19.250 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:31:19.250 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:31:19.250 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-10T19:31:19.250 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:30:56.960456+0000", 2024-09-10T19:31:19.250 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.6269999999999997e-06 2024-09-10T19:31:19.250 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:31:19.250 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-10T19:31:19.250 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:31:19.250 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:31:19.250 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:31:19.250 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_repop(client.16026.0:275576 11.2 e422/371 11:48c27438:::benchmark_data_smithi183_203944_object130686:head v 422'17065, mlcod=422'17061)", 2024-09-10T19:31:19.250 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-10T19:30:56.418543+0000", 2024-09-10T19:31:19.250 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 22.707618804999999, 2024-09-10T19:31:19.250 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.55227596400000001, 2024-09-10T19:31:19.251 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-10T19:31:19.251 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T19:31:19.251 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-10T19:31:19.251 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:31:19.251 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-10T19:31:19.251 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:30:56.418543+0000", 2024-09-10T19:31:19.251 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:31:19.251 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:31:19.251 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:31:19.251 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-10T19:31:19.251 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:30:56.418543+0000", 2024-09-10T19:31:19.251 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:31:19.251 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:31:19.251 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:31:19.252 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-10T19:31:19.252 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:30:56.418542+0000", 2024-09-10T19:31:19.252 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.9999995 2024-09-10T19:31:19.252 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:31:19.252 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:31:19.252 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-10T19:31:19.252 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:30:56.418551+0000", 2024-09-10T19:31:19.252 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.4509999999999992e-06 2024-09-10T19:31:19.252 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:31:19.252 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:31:19.252 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-10T19:31:19.252 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:30:56.957921+0000", 2024-09-10T19:31:19.252 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.53937025000000005 2024-09-10T19:31:19.252 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:31:19.253 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:31:19.253 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-10T19:31:19.253 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:30:56.957930+0000", 2024-09-10T19:31:19.253 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.0149999999999992e-06 2024-09-10T19:31:19.253 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:31:19.253 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:31:19.253 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:31:19.253 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:30:56.958606+0000", 2024-09-10T19:31:19.253 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00067611500000000003 2024-09-10T19:31:19.253 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:31:19.253 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:31:19.253 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-10T19:31:19.253 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:30:56.958625+0000", 2024-09-10T19:31:19.253 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.8516e-05 2024-09-10T19:31:19.254 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:31:19.254 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:31:19.254 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-09-10T19:31:19.254 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:30:56.970788+0000", 2024-09-10T19:31:19.254 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.012163489 2024-09-10T19:31:19.254 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:31:19.254 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:31:19.254 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-10T19:31:19.254 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:30:56.970819+0000", 2024-09-10T19:31:19.254 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 3.0678e-05 2024-09-10T19:31:19.254 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:31:19.254 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-10T19:31:19.254 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:31:19.254 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:31:19.254 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:31:19.255 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.16026.0:277305 11.5 11:a5c34c70:::benchmark_data_smithi183_203944_object132415:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e422)", 2024-09-10T19:31:19.255 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-10T19:30:58.382143+0000", 2024-09-10T19:31:19.255 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 20.744018442000002, 2024-09-10T19:31:19.255 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.124010492, 2024-09-10T19:31:19.255 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-10T19:31:19.255 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T19:31:19.255 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-10T19:31:19.255 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.16026", 2024-09-10T19:31:19.255 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.183:0/1450567484", 2024-09-10T19:31:19.255 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 277305 2024-09-10T19:31:19.255 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:31:19.255 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-10T19:31:19.255 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:31:19.255 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-10T19:31:19.256 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:30:58.382143+0000", 2024-09-10T19:31:19.256 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:31:19.256 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:31:19.256 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:31:19.256 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-10T19:31:19.256 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:30:58.382143+0000", 2024-09-10T19:31:19.256 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:31:19.256 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:31:19.256 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:31:19.256 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-10T19:31:19.256 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:30:58.382142+0000", 2024-09-10T19:31:19.256 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.999999 2024-09-10T19:31:19.256 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:31:19.256 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:31:19.257 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-10T19:31:19.257 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:30:58.382148+0000", 2024-09-10T19:31:19.257 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.6509999999999998e-06 2024-09-10T19:31:19.257 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:31:19.257 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:31:19.257 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-10T19:31:19.257 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:30:58.382169+0000", 2024-09-10T19:31:19.257 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 2.0732000000000001e-05 2024-09-10T19:31:19.257 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:31:19.257 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:31:19.257 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-10T19:31:19.257 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:30:58.382175+0000", 2024-09-10T19:31:19.257 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.071e-06 2024-09-10T19:31:19.258 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:31:19.258 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:31:19.258 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:31:19.258 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:30:58.382218+0000", 2024-09-10T19:31:19.258 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4.3049000000000001e-05 2024-09-10T19:31:19.258 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:31:19.258 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:31:19.258 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-10T19:31:19.258 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:30:58.382328+0000", 2024-09-10T19:31:19.258 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000110089 2024-09-10T19:31:19.258 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:31:19.258 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:31:19.258 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 5,6", 2024-09-10T19:31:19.258 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:30:58.382447+0000", 2024-09-10T19:31:19.259 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000119444 2024-09-10T19:31:19.259 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:31:19.259 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:31:19.259 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-09-10T19:31:19.259 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:30:58.385934+0000", 2024-09-10T19:31:19.259 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0034871979999999999 2024-09-10T19:31:19.259 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:31:19.259 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:31:19.259 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:31:19.259 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:30:58.387929+0000", 2024-09-10T19:31:19.259 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0019949669999999998 2024-09-10T19:31:19.259 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:31:19.259 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:31:19.259 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:31:19.260 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:30:58.506008+0000", 2024-09-10T19:31:19.260 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.11807851599999999 2024-09-10T19:31:19.260 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:31:19.260 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:31:19.260 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-09-10T19:31:19.260 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:30:58.506102+0000", 2024-09-10T19:31:19.260 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 9.4290999999999996e-05 2024-09-10T19:31:19.260 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:31:19.260 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:31:19.260 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-10T19:31:19.260 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:30:58.506154+0000", 2024-09-10T19:31:19.260 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.1668000000000001e-05 2024-09-10T19:31:19.260 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:31:19.260 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-10T19:31:19.261 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:31:19.261 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:31:19.261 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:31:19.261 INFO:teuthology.orchestra.run.smithi191.stdout: "description": "osd_op(client.16026.0:277316 11.5 11:aadd03c5:::benchmark_data_smithi183_203944_object132426:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e422)", 2024-09-10T19:31:19.261 INFO:teuthology.orchestra.run.smithi191.stdout: "initiated_at": "2024-09-10T19:30:58.389986+0000", 2024-09-10T19:31:19.261 INFO:teuthology.orchestra.run.smithi191.stdout: "age": 20.736176176000001, 2024-09-10T19:31:19.261 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.119979321, 2024-09-10T19:31:19.261 INFO:teuthology.orchestra.run.smithi191.stdout: "type_data": { 2024-09-10T19:31:19.261 INFO:teuthology.orchestra.run.smithi191.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-10T19:31:19.261 INFO:teuthology.orchestra.run.smithi191.stdout: "client_info": { 2024-09-10T19:31:19.261 INFO:teuthology.orchestra.run.smithi191.stdout: "client": "client.16026", 2024-09-10T19:31:19.261 INFO:teuthology.orchestra.run.smithi191.stdout: "client_addr": "172.21.15.183:0/1450567484", 2024-09-10T19:31:19.261 INFO:teuthology.orchestra.run.smithi191.stdout: "tid": 277316 2024-09-10T19:31:19.262 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:31:19.262 INFO:teuthology.orchestra.run.smithi191.stdout: "events": [ 2024-09-10T19:31:19.262 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:31:19.262 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "initiated", 2024-09-10T19:31:19.262 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:30:58.389986+0000", 2024-09-10T19:31:19.262 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:31:19.262 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:31:19.262 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:31:19.262 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "throttled", 2024-09-10T19:31:19.262 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:30:58.389986+0000", 2024-09-10T19:31:19.262 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0 2024-09-10T19:31:19.262 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:31:19.262 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:31:19.262 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "header_read", 2024-09-10T19:31:19.263 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:30:58.389985+0000", 2024-09-10T19:31:19.263 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 4294967295.999999 2024-09-10T19:31:19.263 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:31:19.263 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:31:19.263 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "all_read", 2024-09-10T19:31:19.263 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:30:58.389990+0000", 2024-09-10T19:31:19.263 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.4500000000000003e-06 2024-09-10T19:31:19.263 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:31:19.263 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:31:19.263 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "dispatched", 2024-09-10T19:31:19.263 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:30:58.390002+0000", 2024-09-10T19:31:19.263 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 1.1865000000000001e-05 2024-09-10T19:31:19.263 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:31:19.263 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:31:19.264 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "queued_for_pg", 2024-09-10T19:31:19.264 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:30:58.390008+0000", 2024-09-10T19:31:19.264 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 6.2879999999999998e-06 2024-09-10T19:31:19.264 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:31:19.264 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:31:19.264 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "reached_pg", 2024-09-10T19:31:19.264 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:30:58.392435+0000", 2024-09-10T19:31:19.264 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0024268919999999999 2024-09-10T19:31:19.264 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:31:19.264 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:31:19.264 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "started", 2024-09-10T19:31:19.264 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:30:58.392538+0000", 2024-09-10T19:31:19.264 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.000103238 2024-09-10T19:31:19.265 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:31:19.265 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:31:19.265 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "waiting for subops from 5,6", 2024-09-10T19:31:19.265 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:30:58.392643+0000", 2024-09-10T19:31:19.265 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.00010464200000000001 2024-09-10T19:31:19.265 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:31:19.265 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:31:19.265 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:31:19.265 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:30:58.395512+0000", 2024-09-10T19:31:19.265 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.0028690090000000001 2024-09-10T19:31:19.265 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:31:19.265 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:31:19.265 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "op_commit", 2024-09-10T19:31:19.265 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:30:58.395571+0000", 2024-09-10T19:31:19.266 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.8773000000000001e-05 2024-09-10T19:31:19.266 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:31:19.266 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:31:19.266 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "sub_op_commit_rec", 2024-09-10T19:31:19.266 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:30:58.509833+0000", 2024-09-10T19:31:19.266 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 0.11426204600000001 2024-09-10T19:31:19.266 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:31:19.266 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:31:19.266 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "commit_sent", 2024-09-10T19:31:19.266 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:30:58.509914+0000", 2024-09-10T19:31:19.266 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 8.1253000000000003e-05 2024-09-10T19:31:19.266 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2024-09-10T19:31:19.266 INFO:teuthology.orchestra.run.smithi191.stdout: { 2024-09-10T19:31:19.266 INFO:teuthology.orchestra.run.smithi191.stdout: "event": "done", 2024-09-10T19:31:19.267 INFO:teuthology.orchestra.run.smithi191.stdout: "time": "2024-09-10T19:30:58.509965+0000", 2024-09-10T19:31:19.267 INFO:teuthology.orchestra.run.smithi191.stdout: "duration": 5.0828999999999997e-05 2024-09-10T19:31:19.267 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:31:19.267 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-10T19:31:19.267 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:31:19.267 INFO:teuthology.orchestra.run.smithi191.stdout: } 2024-09-10T19:31:19.267 INFO:teuthology.orchestra.run.smithi191.stdout: ] 2024-09-10T19:31:19.267 INFO:teuthology.orchestra.run.smithi191.stdout:} 2024-09-10T19:31:19.289 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:31:19.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:31:19.290 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:31:19.377 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:31:19 smithi183 ceph-mon[29736]: 2.1a scrub starts 2024-09-10T19:31:19.377 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:31:19 smithi183 ceph-mon[29736]: 2.1a scrub ok 2024-09-10T19:31:19.377 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:31:19 smithi183 ceph-mon[29736]: pgmap v971: 158 pgs: 158 active+clean; 582 KiB data, 5.5 GiB used, 620 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 220 op/s 2024-09-10T19:31:19.377 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:31:19 smithi183 ceph-mon[29736]: 6.12 scrub starts 2024-09-10T19:31:19.377 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:31:19 smithi183 ceph-mon[29736]: 6.12 scrub ok 2024-09-10T19:31:19.377 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:31:19 smithi183 ceph-mon[29736]: 4.11 scrub starts 2024-09-10T19:31:19.377 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:31:19 smithi183 ceph-mon[29736]: 4.11 scrub ok 2024-09-10T19:31:19.377 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:31:19 smithi183 ceph-mon[29736]: 2.15 scrub starts 2024-09-10T19:31:19.377 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:31:19 smithi183 ceph-mon[29736]: 2.15 scrub ok 2024-09-10T19:31:19.377 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:31:19 smithi183 ceph-mon[29736]: 2.13 scrub starts 2024-09-10T19:31:19.377 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:31:19 smithi183 ceph-mon[29736]: 2.13 scrub ok 2024-09-10T19:31:19.378 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:31:19 smithi183 ceph-mon[41562]: 2.1a scrub starts 2024-09-10T19:31:19.378 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:31:19 smithi183 ceph-mon[41562]: 2.1a scrub ok 2024-09-10T19:31:19.378 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:31:19 smithi183 ceph-mon[41562]: pgmap v971: 158 pgs: 158 active+clean; 582 KiB data, 5.5 GiB used, 620 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 220 op/s 2024-09-10T19:31:19.378 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:31:19 smithi183 ceph-mon[41562]: 6.12 scrub starts 2024-09-10T19:31:19.378 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:31:19 smithi183 ceph-mon[41562]: 6.12 scrub ok 2024-09-10T19:31:19.378 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:31:19 smithi183 ceph-mon[41562]: 4.11 scrub starts 2024-09-10T19:31:19.378 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:31:19 smithi183 ceph-mon[41562]: 4.11 scrub ok 2024-09-10T19:31:19.378 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:31:19 smithi183 ceph-mon[41562]: 2.15 scrub starts 2024-09-10T19:31:19.378 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:31:19 smithi183 ceph-mon[41562]: 2.15 scrub ok 2024-09-10T19:31:19.378 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:31:19 smithi183 ceph-mon[41562]: 2.13 scrub starts 2024-09-10T19:31:19.378 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:31:19 smithi183 ceph-mon[41562]: 2.13 scrub ok 2024-09-10T19:31:19.392 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:31:19.414 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:31:19.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:31:19.415 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:31:19.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:31:19 smithi191 ceph-mon[32884]: 2.1a scrub starts 2024-09-10T19:31:19.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:31:19 smithi191 ceph-mon[32884]: 2.1a scrub ok 2024-09-10T19:31:19.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:31:19 smithi191 ceph-mon[32884]: pgmap v971: 158 pgs: 158 active+clean; 582 KiB data, 5.5 GiB used, 620 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 220 op/s 2024-09-10T19:31:19.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:31:19 smithi191 ceph-mon[32884]: 6.12 scrub starts 2024-09-10T19:31:19.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:31:19 smithi191 ceph-mon[32884]: 6.12 scrub ok 2024-09-10T19:31:19.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:31:19 smithi191 ceph-mon[32884]: 4.11 scrub starts 2024-09-10T19:31:19.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:31:19 smithi191 ceph-mon[32884]: 4.11 scrub ok 2024-09-10T19:31:19.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:31:19 smithi191 ceph-mon[32884]: 2.15 scrub starts 2024-09-10T19:31:19.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:31:19 smithi191 ceph-mon[32884]: 2.15 scrub ok 2024-09-10T19:31:19.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:31:19 smithi191 ceph-mon[32884]: 2.13 scrub starts 2024-09-10T19:31:19.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:31:19 smithi191 ceph-mon[32884]: 2.13 scrub ok 2024-09-10T19:31:19.516 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:31:19.537 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:31:19.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:31:19.538 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:31:19.639 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:31:19.660 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:31:19.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:31:19.661 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:31:19.762 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:31:19.784 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:31:19.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:31:19.785 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:31:19.886 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:31:19.907 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:31:19.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:31:19.909 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:31:20.010 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5 2024-09-10T19:31:20.033 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph daemon osd.1 dump_ops_in_flight 2024-09-10T19:31:20.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:31:20.035 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:31:20.036 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.5' 2024-09-10T19:31:20.137 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:31:20.158 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:31:20.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:31:20.159 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:31:20.241 INFO:teuthology.orchestra.run.smithi183.stderr:set 4.a pg_upmap mapping to [3,2,1] 2024-09-10T19:31:20.259 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:31:20.281 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:31:20.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:31:20.282 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:31:20.383 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:31:20.404 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:31:20.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:31:20.405 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:31:20.465 INFO:journalctl@ceph.iscsi.iscsi.a.smithi191.stdout:Sep 10 19:31:20 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-iscsi-iscsi-a[61816]: debug there is no tcmu-runner data available 2024-09-10T19:31:20.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:31:20 smithi191 ceph-mon[32884]: 2.17 scrub starts 2024-09-10T19:31:20.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:31:20 smithi191 ceph-mon[32884]: 2.17 scrub ok 2024-09-10T19:31:20.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:31:20 smithi191 ceph-mon[32884]: 2.1d scrub starts 2024-09-10T19:31:20.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:31:20 smithi191 ceph-mon[32884]: 2.1d scrub ok 2024-09-10T19:31:20.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:31:20 smithi191 ceph-mon[32884]: 3.0 scrub starts 2024-09-10T19:31:20.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:31:20 smithi191 ceph-mon[32884]: 3.0 scrub ok 2024-09-10T19:31:20.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:31:20 smithi191 ceph-mon[32884]: from='client.? 172.21.15.183:0/3016592711' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "4.a", "id": [3, 2, 1]}]: dispatch 2024-09-10T19:31:20.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:31:20 smithi191 ceph-mon[32884]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "4.a", "id": [3, 2, 1]}]: dispatch 2024-09-10T19:31:20.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:31:20 smithi191 ceph-mon[32884]: 6.a scrub starts 2024-09-10T19:31:20.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:31:20 smithi191 ceph-mon[32884]: 6.a scrub ok 2024-09-10T19:31:20.506 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:31:20.530 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:31:20.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:31:20.531 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:31:20.632 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:31:20.654 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:31:20.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:31:20.656 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:31:20.681 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:31:20 smithi183 ceph-mon[41562]: 2.17 scrub starts 2024-09-10T19:31:20.682 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:31:20 smithi183 ceph-mon[41562]: 2.17 scrub ok 2024-09-10T19:31:20.682 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:31:20 smithi183 ceph-mon[41562]: 2.1d scrub starts 2024-09-10T19:31:20.682 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:31:20 smithi183 ceph-mon[41562]: 2.1d scrub ok 2024-09-10T19:31:20.682 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:31:20 smithi183 ceph-mon[41562]: 3.0 scrub starts 2024-09-10T19:31:20.682 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:31:20 smithi183 ceph-mon[41562]: 3.0 scrub ok 2024-09-10T19:31:20.682 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:31:20 smithi183 ceph-mon[41562]: from='client.? 172.21.15.183:0/3016592711' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "4.a", "id": [3, 2, 1]}]: dispatch 2024-09-10T19:31:20.682 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:31:20 smithi183 ceph-mon[41562]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "4.a", "id": [3, 2, 1]}]: dispatch 2024-09-10T19:31:20.682 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:31:20 smithi183 ceph-mon[41562]: 6.a scrub starts 2024-09-10T19:31:20.682 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:31:20 smithi183 ceph-mon[41562]: 6.a scrub ok 2024-09-10T19:31:20.682 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:31:20 smithi183 ceph-mon[29736]: 2.17 scrub starts 2024-09-10T19:31:20.683 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:31:20 smithi183 ceph-mon[29736]: 2.17 scrub ok 2024-09-10T19:31:20.683 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:31:20 smithi183 ceph-mon[29736]: 2.1d scrub starts 2024-09-10T19:31:20.683 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:31:20 smithi183 ceph-mon[29736]: 2.1d scrub ok 2024-09-10T19:31:20.683 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:31:20 smithi183 ceph-mon[29736]: 3.0 scrub starts 2024-09-10T19:31:20.683 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:31:20 smithi183 ceph-mon[29736]: 3.0 scrub ok 2024-09-10T19:31:20.683 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:31:20 smithi183 ceph-mon[29736]: from='client.? 172.21.15.183:0/3016592711' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "4.a", "id": [3, 2, 1]}]: dispatch 2024-09-10T19:31:20.683 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:31:20 smithi183 ceph-mon[29736]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "4.a", "id": [3, 2, 1]}]: dispatch 2024-09-10T19:31:20.683 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:31:20 smithi183 ceph-mon[29736]: 6.a scrub starts 2024-09-10T19:31:20.683 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:31:20 smithi183 ceph-mon[29736]: 6.a scrub ok 2024-09-10T19:31:20.757 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:31:20.778 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:31:20.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:31:20.779 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:31:20.880 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:31:20.903 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:31:20.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:31:20.904 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:31:21.005 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:31:21.028 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:31:21.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:31:21.029 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:31:21.130 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:31:21.151 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:31:21.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:31:21.152 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:31:21.253 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:31:21.260 INFO:teuthology.orchestra.run.smithi183.stderr:pool 'unique_pool_4' removed 2024-09-10T19:31:21.292 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:31:21.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:31:21.293 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:31:21.394 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:31:21.416 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:31:21.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:31:21.417 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:31:21.431 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Sep 10 19:31:21 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-y[29989]: ::ffff:172.21.15.191 - - [10/Sep/2024:19:31:21] "GET /metrics HTTP/1.1" 200 224844 "" "Prometheus/2.43.0" 2024-09-10T19:31:21.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:31:21 smithi183 ceph-mon[29736]: pgmap v972: 158 pgs: 158 active+clean; 582 KiB data, 5.5 GiB used, 620 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 180 op/s 2024-09-10T19:31:21.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:31:21 smithi183 ceph-mon[29736]: 5.d deep-scrub starts 2024-09-10T19:31:21.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:31:21 smithi183 ceph-mon[29736]: 5.d deep-scrub ok 2024-09-10T19:31:21.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:31:21 smithi183 ceph-mon[29736]: 6.f scrub starts 2024-09-10T19:31:21.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:31:21 smithi183 ceph-mon[29736]: 6.f scrub ok 2024-09-10T19:31:21.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:31:21 smithi183 ceph-mon[29736]: from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:31:21.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:31:21 smithi183 ceph-mon[29736]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "4.a", "id": [3, 2, 1]}]': finished 2024-09-10T19:31:21.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:31:21 smithi183 ceph-mon[29736]: osdmap e425: 8 total, 8 up, 7 in 2024-09-10T19:31:21.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:31:21 smithi183 ceph-mon[29736]: 4.2 deep-scrub starts 2024-09-10T19:31:21.433 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:31:21 smithi183 ceph-mon[29736]: 4.2 deep-scrub ok 2024-09-10T19:31:21.433 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:31:21 smithi183 ceph-mon[29736]: from='client.? 172.21.15.183:0/3639968893' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_4", "pool2": "unique_pool_4", "yes_i_really_really_mean_it": true}]: dispatch 2024-09-10T19:31:21.433 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:31:21 smithi183 ceph-mon[41562]: pgmap v972: 158 pgs: 158 active+clean; 582 KiB data, 5.5 GiB used, 620 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 180 op/s 2024-09-10T19:31:21.433 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:31:21 smithi183 ceph-mon[41562]: 5.d deep-scrub starts 2024-09-10T19:31:21.433 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:31:21 smithi183 ceph-mon[41562]: 5.d deep-scrub ok 2024-09-10T19:31:21.433 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:31:21 smithi183 ceph-mon[41562]: 6.f scrub starts 2024-09-10T19:31:21.433 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:31:21 smithi183 ceph-mon[41562]: 6.f scrub ok 2024-09-10T19:31:21.433 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:31:21 smithi183 ceph-mon[41562]: from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:31:21.433 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:31:21 smithi183 ceph-mon[41562]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "4.a", "id": [3, 2, 1]}]': finished 2024-09-10T19:31:21.434 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:31:21 smithi183 ceph-mon[41562]: osdmap e425: 8 total, 8 up, 7 in 2024-09-10T19:31:21.434 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:31:21 smithi183 ceph-mon[41562]: 4.2 deep-scrub starts 2024-09-10T19:31:21.434 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:31:21 smithi183 ceph-mon[41562]: 4.2 deep-scrub ok 2024-09-10T19:31:21.434 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:31:21 smithi183 ceph-mon[41562]: from='client.? 172.21.15.183:0/3639968893' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_4", "pool2": "unique_pool_4", "yes_i_really_really_mean_it": true}]: dispatch 2024-09-10T19:31:21.519 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:31:21.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:31:21.542 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:31:21.542 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:31:21.643 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:31:21.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:31:21.674 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:31:21.674 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:31:21.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:31:21 smithi191 ceph-mon[32884]: pgmap v972: 158 pgs: 158 active+clean; 582 KiB data, 5.5 GiB used, 620 GiB / 626 GiB avail; 1023 B/s rd, 0 B/s wr, 180 op/s 2024-09-10T19:31:21.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:31:21 smithi191 ceph-mon[32884]: 5.d deep-scrub starts 2024-09-10T19:31:21.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:31:21 smithi191 ceph-mon[32884]: 5.d deep-scrub ok 2024-09-10T19:31:21.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:31:21 smithi191 ceph-mon[32884]: 6.f scrub starts 2024-09-10T19:31:21.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:31:21 smithi191 ceph-mon[32884]: 6.f scrub ok 2024-09-10T19:31:21.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:31:21 smithi191 ceph-mon[32884]: from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:31:21.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:31:21 smithi191 ceph-mon[32884]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "4.a", "id": [3, 2, 1]}]': finished 2024-09-10T19:31:21.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:31:21 smithi191 ceph-mon[32884]: osdmap e425: 8 total, 8 up, 7 in 2024-09-10T19:31:21.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:31:21 smithi191 ceph-mon[32884]: 4.2 deep-scrub starts 2024-09-10T19:31:21.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:31:21 smithi191 ceph-mon[32884]: 4.2 deep-scrub ok 2024-09-10T19:31:21.717 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:31:21 smithi191 ceph-mon[32884]: from='client.? 172.21.15.183:0/3639968893' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_4", "pool2": "unique_pool_4", "yes_i_really_really_mean_it": true}]: dispatch 2024-09-10T19:31:21.774 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:31:21.795 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:31:21.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:31:21.796 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:31:21.897 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:31:21.919 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:31:21.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:31:21.920 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:31:22.021 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:31:22.042 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:31:22.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:31:22.043 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:31:22.144 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:31:22.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:31:22.167 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:31:22.168 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:31:22.168 INFO:teuthology.orchestra.run.smithi183.stdout:osd.0: {} 2024-09-10T19:31:22.169 INFO:teuthology.orchestra.run.smithi183.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-10T19:31:22.172 INFO:teuthology.orchestra.run.smithi183.stdout:osd.1: {} 2024-09-10T19:31:22.173 INFO:teuthology.orchestra.run.smithi183.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-10T19:31:22.187 INFO:teuthology.orchestra.run.smithi183.stdout:osd.2: {} 2024-09-10T19:31:22.187 INFO:teuthology.orchestra.run.smithi183.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-10T19:31:22.201 INFO:teuthology.orchestra.run.smithi183.stdout:osd.3: {} 2024-09-10T19:31:22.202 INFO:teuthology.orchestra.run.smithi183.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-10T19:31:22.217 INFO:teuthology.orchestra.run.smithi183.stdout:osd.4: {} 2024-09-10T19:31:22.217 INFO:teuthology.orchestra.run.smithi183.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-10T19:31:22.233 INFO:teuthology.orchestra.run.smithi183.stdout:osd.5: {} 2024-09-10T19:31:22.233 INFO:teuthology.orchestra.run.smithi183.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-10T19:31:22.248 INFO:teuthology.orchestra.run.smithi183.stdout:osd.6: {} 2024-09-10T19:31:22.248 INFO:teuthology.orchestra.run.smithi183.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-10T19:31:22.264 INFO:teuthology.orchestra.run.smithi183.stdout:osd.7: {} 2024-09-10T19:31:22.265 INFO:teuthology.orchestra.run.smithi183.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-10T19:31:22.269 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1 2024-09-10T19:31:22.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:31:22.292 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:31:22.293 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.1' 2024-09-10T19:31:22.354 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:31:22 smithi183 ceph-mon[29736]: purged_snaps scrub starts 2024-09-10T19:31:22.354 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:31:22 smithi183 ceph-mon[29736]: purged_snaps scrub ok 2024-09-10T19:31:22.355 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:31:22 smithi183 ceph-mon[29736]: 11.5 scrub starts 2024-09-10T19:31:22.355 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:31:22 smithi183 ceph-mon[29736]: 11.5 scrub ok 2024-09-10T19:31:22.355 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:31:22 smithi183 ceph-mon[29736]: 5.13 scrub starts 2024-09-10T19:31:22.355 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:31:22 smithi183 ceph-mon[29736]: 5.13 scrub ok 2024-09-10T19:31:22.355 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:31:22 smithi183 ceph-mon[29736]: from='client.? 172.21.15.183:0/3639968893' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_4", "pool2": "unique_pool_4", "yes_i_really_really_mean_it": true}]': finished 2024-09-10T19:31:22.355 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:31:22 smithi183 ceph-mon[29736]: osdmap e426: 8 total, 8 up, 7 in 2024-09-10T19:31:22.355 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:31:22 smithi183 ceph-mon[29736]: 2.d scrub starts 2024-09-10T19:31:22.355 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:31:22 smithi183 ceph-mon[29736]: 2.d scrub ok 2024-09-10T19:31:22.355 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:31:22 smithi183 ceph-mon[29736]: 2.8 deep-scrub starts 2024-09-10T19:31:22.355 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:31:22 smithi183 ceph-mon[29736]: 2.8 deep-scrub ok 2024-09-10T19:31:22.356 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:31:22 smithi183 ceph-mon[29736]: 6.16 scrub starts 2024-09-10T19:31:22.356 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:31:22 smithi183 ceph-mon[29736]: 6.16 scrub ok 2024-09-10T19:31:22.356 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:31:22 smithi183 ceph-mon[29736]: from='client.? 172.21.15.183:0/3677334220' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T19:31:22.356 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:31:22 smithi183 ceph-mon[41562]: purged_snaps scrub starts 2024-09-10T19:31:22.356 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:31:22 smithi183 ceph-mon[41562]: purged_snaps scrub ok 2024-09-10T19:31:22.356 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:31:22 smithi183 ceph-mon[41562]: 11.5 scrub starts 2024-09-10T19:31:22.356 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:31:22 smithi183 ceph-mon[41562]: 11.5 scrub ok 2024-09-10T19:31:22.356 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:31:22 smithi183 ceph-mon[41562]: 5.13 scrub starts 2024-09-10T19:31:22.357 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:31:22 smithi183 ceph-mon[41562]: 5.13 scrub ok 2024-09-10T19:31:22.357 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:31:22 smithi183 ceph-mon[41562]: from='client.? 172.21.15.183:0/3639968893' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_4", "pool2": "unique_pool_4", "yes_i_really_really_mean_it": true}]': finished 2024-09-10T19:31:22.357 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:31:22 smithi183 ceph-mon[41562]: osdmap e426: 8 total, 8 up, 7 in 2024-09-10T19:31:22.357 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:31:22 smithi183 ceph-mon[41562]: 2.d scrub starts 2024-09-10T19:31:22.357 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:31:22 smithi183 ceph-mon[41562]: 2.d scrub ok 2024-09-10T19:31:22.357 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:31:22 smithi183 ceph-mon[41562]: 2.8 deep-scrub starts 2024-09-10T19:31:22.357 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:31:22 smithi183 ceph-mon[41562]: 2.8 deep-scrub ok 2024-09-10T19:31:22.357 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:31:22 smithi183 ceph-mon[41562]: 6.16 scrub starts 2024-09-10T19:31:22.357 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:31:22 smithi183 ceph-mon[41562]: 6.16 scrub ok 2024-09-10T19:31:22.357 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:31:22 smithi183 ceph-mon[41562]: from='client.? 172.21.15.183:0/3677334220' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T19:31:22.393 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:31:22.415 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:31:22.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:31:22.416 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:31:22.517 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:31:22.538 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:31:22.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:31:22.539 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:31:22.640 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2 2024-09-10T19:31:22.662 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:31:22.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:31:22.663 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.2' 2024-09-10T19:31:22.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:31:22 smithi191 ceph-mon[32884]: purged_snaps scrub starts 2024-09-10T19:31:22.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:31:22 smithi191 ceph-mon[32884]: purged_snaps scrub ok 2024-09-10T19:31:22.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:31:22 smithi191 ceph-mon[32884]: 11.5 scrub starts 2024-09-10T19:31:22.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:31:22 smithi191 ceph-mon[32884]: 11.5 scrub ok 2024-09-10T19:31:22.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:31:22 smithi191 ceph-mon[32884]: 5.13 scrub starts 2024-09-10T19:31:22.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:31:22 smithi191 ceph-mon[32884]: 5.13 scrub ok 2024-09-10T19:31:22.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:31:22 smithi191 ceph-mon[32884]: from='client.? 172.21.15.183:0/3639968893' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "unique_pool_4", "pool2": "unique_pool_4", "yes_i_really_really_mean_it": true}]': finished 2024-09-10T19:31:22.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:31:22 smithi191 ceph-mon[32884]: osdmap e426: 8 total, 8 up, 7 in 2024-09-10T19:31:22.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:31:22 smithi191 ceph-mon[32884]: 2.d scrub starts 2024-09-10T19:31:22.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:31:22 smithi191 ceph-mon[32884]: 2.d scrub ok 2024-09-10T19:31:22.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:31:22 smithi191 ceph-mon[32884]: 2.8 deep-scrub starts 2024-09-10T19:31:22.717 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:31:22 smithi191 ceph-mon[32884]: 2.8 deep-scrub ok 2024-09-10T19:31:22.717 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:31:22 smithi191 ceph-mon[32884]: 6.16 scrub starts 2024-09-10T19:31:22.717 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:31:22 smithi191 ceph-mon[32884]: 6.16 scrub ok 2024-09-10T19:31:22.717 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:31:22 smithi191 ceph-mon[32884]: from='client.? 172.21.15.183:0/3677334220' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-10T19:31:22.763 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:31:22.785 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:31:22.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:31:22.786 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:31:22.887 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:31:22.910 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:31:22.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:31:22.911 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:31:23.012 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:31:23.035 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:31:23.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:31:23.037 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:31:23.138 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:31:23.160 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:31:23.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:31:23.161 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:31:23.262 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7 2024-09-10T19:31:23.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:31:23.285 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:31:23.285 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.7' 2024-09-10T19:31:23.386 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:31:23.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:31:23.408 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:31:23.409 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:31:23.431 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:31:23 smithi183 ceph-mon[41562]: 5.19 scrub starts 2024-09-10T19:31:23.431 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:31:23 smithi183 ceph-mon[41562]: 5.19 scrub ok 2024-09-10T19:31:23.431 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:31:23 smithi183 ceph-mon[41562]: pgmap v975: 142 pgs: 142 active+clean; 582 KiB data, 5.5 GiB used, 620 GiB / 626 GiB avail; 682 B/s rd, 0 op/s 2024-09-10T19:31:23.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:31:23 smithi183 ceph-mon[41562]: 2.1c scrub starts 2024-09-10T19:31:23.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:31:23 smithi183 ceph-mon[41562]: 2.1c scrub ok 2024-09-10T19:31:23.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:31:23 smithi183 ceph-mon[41562]: purged_snaps scrub starts 2024-09-10T19:31:23.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:31:23 smithi183 ceph-mon[41562]: purged_snaps scrub ok 2024-09-10T19:31:23.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:31:23 smithi183 ceph-mon[41562]: 4.16 scrub starts 2024-09-10T19:31:23.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:31:23 smithi183 ceph-mon[41562]: 4.16 scrub ok 2024-09-10T19:31:23.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:31:23 smithi183 ceph-mon[29736]: 5.19 scrub starts 2024-09-10T19:31:23.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:31:23 smithi183 ceph-mon[29736]: 5.19 scrub ok 2024-09-10T19:31:23.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:31:23 smithi183 ceph-mon[29736]: pgmap v975: 142 pgs: 142 active+clean; 582 KiB data, 5.5 GiB used, 620 GiB / 626 GiB avail; 682 B/s rd, 0 op/s 2024-09-10T19:31:23.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:31:23 smithi183 ceph-mon[29736]: 2.1c scrub starts 2024-09-10T19:31:23.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:31:23 smithi183 ceph-mon[29736]: 2.1c scrub ok 2024-09-10T19:31:23.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:31:23 smithi183 ceph-mon[29736]: purged_snaps scrub starts 2024-09-10T19:31:23.433 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:31:23 smithi183 ceph-mon[29736]: purged_snaps scrub ok 2024-09-10T19:31:23.433 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:31:23 smithi183 ceph-mon[29736]: 4.16 scrub starts 2024-09-10T19:31:23.433 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:31:23 smithi183 ceph-mon[29736]: 4.16 scrub ok 2024-09-10T19:31:23.509 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6 2024-09-10T19:31:23.531 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:31:23.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:31:23.532 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.6' 2024-09-10T19:31:23.596 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:31:23.633 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3 2024-09-10T19:31:23.654 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:31:23.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:31:23.655 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.3' 2024-09-10T19:31:23.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:31:23 smithi191 ceph-mon[32884]: 5.19 scrub starts 2024-09-10T19:31:23.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:31:23 smithi191 ceph-mon[32884]: 5.19 scrub ok 2024-09-10T19:31:23.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:31:23 smithi191 ceph-mon[32884]: pgmap v975: 142 pgs: 142 active+clean; 582 KiB data, 5.5 GiB used, 620 GiB / 626 GiB avail; 682 B/s rd, 0 op/s 2024-09-10T19:31:23.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:31:23 smithi191 ceph-mon[32884]: 2.1c scrub starts 2024-09-10T19:31:23.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:31:23 smithi191 ceph-mon[32884]: 2.1c scrub ok 2024-09-10T19:31:23.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:31:23 smithi191 ceph-mon[32884]: purged_snaps scrub starts 2024-09-10T19:31:23.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:31:23 smithi191 ceph-mon[32884]: purged_snaps scrub ok 2024-09-10T19:31:23.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:31:23 smithi191 ceph-mon[32884]: 4.16 scrub starts 2024-09-10T19:31:23.717 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:31:23 smithi191 ceph-mon[32884]: 4.16 scrub ok 2024-09-10T19:31:23.756 DEBUG:teuthology.orchestra.run.smithi191:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4 2024-09-10T19:31:23.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:31:23.778 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: docker: command not found 2024-09-10T19:31:23.778 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi191 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.4' 2024-09-10T19:31:23.879 DEBUG:teuthology.orchestra.run.smithi183:> sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0 2024-09-10T19:31:23.901 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: docker: command not found 2024-09-10T19:31:23.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:31:23.902 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi183 with status 1: 'sudo docker kill -s 1 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd.0' 2024-09-10T19:31:23.995 DEBUG:teuthology.run_tasks:Unwinding manager thrashosds 2024-09-10T19:31:24.005 INFO:tasks.thrashosds:joining thrashosds 2024-09-10T19:31:24.058 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:31:24.609 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:31:24 smithi183 ceph-mon[29736]: osdmap e427: 8 total, 8 up, 7 in 2024-09-10T19:31:24.609 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:31:24 smithi183 ceph-mon[41562]: osdmap e427: 8 total, 8 up, 7 in 2024-09-10T19:31:24.715 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:31:24 smithi191 ceph-mon[32884]: osdmap e427: 8 total, 8 up, 7 in 2024-09-10T19:31:25.681 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:31:25 smithi183 ceph-mon[41562]: pgmap v977: 142 pgs: 142 active+clean; 582 KiB data, 5.5 GiB used, 620 GiB / 626 GiB avail; 682 B/s rd, 0 op/s 2024-09-10T19:31:25.682 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:31:25 smithi183 ceph-mon[41562]: 5.c scrub starts 2024-09-10T19:31:25.682 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:31:25 smithi183 ceph-mon[41562]: 5.c scrub ok 2024-09-10T19:31:25.682 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:31:25 smithi183 ceph-mon[41562]: 6.10 scrub starts 2024-09-10T19:31:25.682 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:31:25 smithi183 ceph-mon[41562]: 6.10 scrub ok 2024-09-10T19:31:25.682 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:31:25 smithi183 ceph-mon[41562]: 1.0 deep-scrub starts 2024-09-10T19:31:25.682 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:31:25 smithi183 ceph-mon[41562]: 1.0 deep-scrub ok 2024-09-10T19:31:25.682 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:31:25 smithi183 ceph-mon[29736]: pgmap v977: 142 pgs: 142 active+clean; 582 KiB data, 5.5 GiB used, 620 GiB / 626 GiB avail; 682 B/s rd, 0 op/s 2024-09-10T19:31:25.682 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:31:25 smithi183 ceph-mon[29736]: 5.c scrub starts 2024-09-10T19:31:25.682 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:31:25 smithi183 ceph-mon[29736]: 5.c scrub ok 2024-09-10T19:31:25.682 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:31:25 smithi183 ceph-mon[29736]: 6.10 scrub starts 2024-09-10T19:31:25.683 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:31:25 smithi183 ceph-mon[29736]: 6.10 scrub ok 2024-09-10T19:31:25.683 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:31:25 smithi183 ceph-mon[29736]: 1.0 deep-scrub starts 2024-09-10T19:31:25.683 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:31:25 smithi183 ceph-mon[29736]: 1.0 deep-scrub ok 2024-09-10T19:31:25.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:31:25 smithi191 ceph-mon[32884]: pgmap v977: 142 pgs: 142 active+clean; 582 KiB data, 5.5 GiB used, 620 GiB / 626 GiB avail; 682 B/s rd, 0 op/s 2024-09-10T19:31:25.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:31:25 smithi191 ceph-mon[32884]: 5.c scrub starts 2024-09-10T19:31:25.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:31:25 smithi191 ceph-mon[32884]: 5.c scrub ok 2024-09-10T19:31:25.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:31:25 smithi191 ceph-mon[32884]: 6.10 scrub starts 2024-09-10T19:31:25.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:31:25 smithi191 ceph-mon[32884]: 6.10 scrub ok 2024-09-10T19:31:25.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:31:25 smithi191 ceph-mon[32884]: 1.0 deep-scrub starts 2024-09-10T19:31:25.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:31:25 smithi191 ceph-mon[32884]: 1.0 deep-scrub ok 2024-09-10T19:31:26.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:31:26 smithi183 ceph-mon[29736]: purged_snaps scrub starts 2024-09-10T19:31:26.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:31:26 smithi183 ceph-mon[29736]: purged_snaps scrub ok 2024-09-10T19:31:26.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:31:26 smithi183 ceph-mon[29736]: osdmap e428: 8 total, 8 up, 7 in 2024-09-10T19:31:26.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:31:26 smithi183 ceph-mon[29736]: 2.c deep-scrub starts 2024-09-10T19:31:26.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:31:26 smithi183 ceph-mon[29736]: 2.c deep-scrub ok 2024-09-10T19:31:26.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:31:26 smithi183 ceph-mon[29736]: 4.6 scrub starts 2024-09-10T19:31:26.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:31:26 smithi183 ceph-mon[29736]: 4.6 scrub ok 2024-09-10T19:31:26.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:31:26 smithi183 ceph-mon[41562]: purged_snaps scrub starts 2024-09-10T19:31:26.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:31:26 smithi183 ceph-mon[41562]: purged_snaps scrub ok 2024-09-10T19:31:26.433 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:31:26 smithi183 ceph-mon[41562]: osdmap e428: 8 total, 8 up, 7 in 2024-09-10T19:31:26.433 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:31:26 smithi183 ceph-mon[41562]: 2.c deep-scrub starts 2024-09-10T19:31:26.433 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:31:26 smithi183 ceph-mon[41562]: 2.c deep-scrub ok 2024-09-10T19:31:26.433 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:31:26 smithi183 ceph-mon[41562]: 4.6 scrub starts 2024-09-10T19:31:26.433 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:31:26 smithi183 ceph-mon[41562]: 4.6 scrub ok 2024-09-10T19:31:26.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:31:26 smithi191 ceph-mon[32884]: purged_snaps scrub starts 2024-09-10T19:31:26.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:31:26 smithi191 ceph-mon[32884]: purged_snaps scrub ok 2024-09-10T19:31:26.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:31:26 smithi191 ceph-mon[32884]: osdmap e428: 8 total, 8 up, 7 in 2024-09-10T19:31:26.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:31:26 smithi191 ceph-mon[32884]: 2.c deep-scrub starts 2024-09-10T19:31:26.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:31:26 smithi191 ceph-mon[32884]: 2.c deep-scrub ok 2024-09-10T19:31:26.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:31:26 smithi191 ceph-mon[32884]: 4.6 scrub starts 2024-09-10T19:31:26.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:31:26 smithi191 ceph-mon[32884]: 4.6 scrub ok 2024-09-10T19:31:27.229 INFO:teuthology.orchestra.run.smithi183.stdout:ERROR: (22) Invalid argument 2024-09-10T19:31:27.229 INFO:teuthology.orchestra.run.smithi183.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-10T19:31:27.617 INFO:journalctl@ceph.mgr.x.smithi191.stdout:Sep 10 19:31:27 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-x[34967]: ::ffff:172.21.15.191 - - [10/Sep/2024:19:31:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T19:31:27.618 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:31:27 smithi191 ceph-mon[32884]: pgmap v979: 142 pgs: 142 active+clean; 582 KiB data, 5.5 GiB used, 620 GiB / 626 GiB avail; 711 B/s rd, 0 op/s 2024-09-10T19:31:27.618 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:31:27 smithi191 ceph-mon[32884]: 2.16 scrub starts 2024-09-10T19:31:27.618 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:31:27 smithi191 ceph-mon[32884]: 6.14 scrub starts 2024-09-10T19:31:27.618 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:31:27 smithi191 ceph-mon[32884]: 2.16 scrub ok 2024-09-10T19:31:27.618 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:31:27 smithi191 ceph-mon[32884]: 6.14 scrub ok 2024-09-10T19:31:27.618 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:31:27 smithi191 ceph-mon[32884]: 2.1f scrub starts 2024-09-10T19:31:27.618 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:31:27 smithi191 ceph-mon[32884]: 2.1f scrub ok 2024-09-10T19:31:27.681 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:31:27 smithi183 ceph-mon[29736]: pgmap v979: 142 pgs: 142 active+clean; 582 KiB data, 5.5 GiB used, 620 GiB / 626 GiB avail; 711 B/s rd, 0 op/s 2024-09-10T19:31:27.681 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:31:27 smithi183 ceph-mon[29736]: 2.16 scrub starts 2024-09-10T19:31:27.681 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:31:27 smithi183 ceph-mon[29736]: 6.14 scrub starts 2024-09-10T19:31:27.682 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:31:27 smithi183 ceph-mon[29736]: 2.16 scrub ok 2024-09-10T19:31:27.682 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:31:27 smithi183 ceph-mon[29736]: 6.14 scrub ok 2024-09-10T19:31:27.682 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:31:27 smithi183 ceph-mon[29736]: 2.1f scrub starts 2024-09-10T19:31:27.682 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:31:27 smithi183 ceph-mon[29736]: 2.1f scrub ok 2024-09-10T19:31:27.682 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:31:27 smithi183 ceph-mon[41562]: pgmap v979: 142 pgs: 142 active+clean; 582 KiB data, 5.5 GiB used, 620 GiB / 626 GiB avail; 711 B/s rd, 0 op/s 2024-09-10T19:31:27.682 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:31:27 smithi183 ceph-mon[41562]: 2.16 scrub starts 2024-09-10T19:31:27.682 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:31:27 smithi183 ceph-mon[41562]: 6.14 scrub starts 2024-09-10T19:31:27.682 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:31:27 smithi183 ceph-mon[41562]: 2.16 scrub ok 2024-09-10T19:31:27.682 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:31:27 smithi183 ceph-mon[41562]: 6.14 scrub ok 2024-09-10T19:31:27.682 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:31:27 smithi183 ceph-mon[41562]: 2.1f scrub starts 2024-09-10T19:31:27.682 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:31:27 smithi183 ceph-mon[41562]: 2.1f scrub ok 2024-09-10T19:31:27.946 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph daemon osd.1 dump_blocked_ops 2024-09-10T19:31:28.304 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:31:28.519 INFO:tasks.thrashosds.thrasher:inning osd 2024-09-10T19:31:28.520 INFO:tasks.thrashosds.thrasher:Adding osd 2 2024-09-10T19:31:28.520 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph osd in 2 2024-09-10T19:31:28.543 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:31:28 smithi183 ceph-mon[29736]: 1.1 scrub starts 2024-09-10T19:31:28.543 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:31:28 smithi183 ceph-mon[29736]: 1.1 scrub ok 2024-09-10T19:31:28.543 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:31:28 smithi183 ceph-mon[29736]: 6.17 scrub starts 2024-09-10T19:31:28.543 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:31:28 smithi183 ceph-mon[29736]: 6.17 scrub ok 2024-09-10T19:31:28.543 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:31:28 smithi183 ceph-mon[41562]: 1.1 scrub starts 2024-09-10T19:31:28.543 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:31:28 smithi183 ceph-mon[41562]: 1.1 scrub ok 2024-09-10T19:31:28.543 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:31:28 smithi183 ceph-mon[41562]: 6.17 scrub starts 2024-09-10T19:31:28.543 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:31:28 smithi183 ceph-mon[41562]: 6.17 scrub ok 2024-09-10T19:31:28.637 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:31:28 smithi191 ceph-mon[32884]: 1.1 scrub starts 2024-09-10T19:31:28.637 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:31:28 smithi191 ceph-mon[32884]: 1.1 scrub ok 2024-09-10T19:31:28.637 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:31:28 smithi191 ceph-mon[32884]: 6.17 scrub starts 2024-09-10T19:31:28.637 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:31:28 smithi191 ceph-mon[32884]: 6.17 scrub ok 2024-09-10T19:31:29.108 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:31:29.298 INFO:teuthology.orchestra.run.smithi183.stderr:noscrub is set 2024-09-10T19:31:29.648 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:31:29 smithi183 ceph-mon[29736]: pgmap v980: 142 pgs: 142 active+clean; 582 KiB data, 5.5 GiB used, 620 GiB / 626 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-09-10T19:31:29.649 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:31:29 smithi183 ceph-mon[29736]: from='client.? 172.21.15.183:0/3989222556' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-10T19:31:29.649 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:31:29 smithi183 ceph-mon[29736]: from='mgr.14556 ' entity='mgr.y' 2024-09-10T19:31:29.649 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:31:29 smithi183 ceph-mon[29736]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:31:29.649 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:31:29 smithi183 ceph-mon[29736]: 1.5 scrub starts 2024-09-10T19:31:29.649 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:31:29 smithi183 ceph-mon[29736]: 1.5 scrub ok 2024-09-10T19:31:29.649 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:31:29 smithi183 ceph-mon[41562]: pgmap v980: 142 pgs: 142 active+clean; 582 KiB data, 5.5 GiB used, 620 GiB / 626 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-09-10T19:31:29.649 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:31:29 smithi183 ceph-mon[41562]: from='client.? 172.21.15.183:0/3989222556' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-10T19:31:29.649 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:31:29 smithi183 ceph-mon[41562]: from='mgr.14556 ' entity='mgr.y' 2024-09-10T19:31:29.649 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:31:29 smithi183 ceph-mon[41562]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:31:29.650 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:31:29 smithi183 ceph-mon[41562]: 1.5 scrub starts 2024-09-10T19:31:29.650 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:31:29 smithi183 ceph-mon[41562]: 1.5 scrub ok 2024-09-10T19:31:29.716 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:31:29 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a[65767]: logger=cleanup t=2024-09-10T19:31:29.425023121Z level=info msg="Completed cleanup jobs" duration=12.889896ms 2024-09-10T19:31:29.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:31:29 smithi191 ceph-mon[32884]: pgmap v980: 142 pgs: 142 active+clean; 582 KiB data, 5.5 GiB used, 620 GiB / 626 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-09-10T19:31:29.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:31:29 smithi191 ceph-mon[32884]: from='client.? 172.21.15.183:0/3989222556' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-10T19:31:29.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:31:29 smithi191 ceph-mon[32884]: from='mgr.14556 ' entity='mgr.y' 2024-09-10T19:31:29.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:31:29 smithi191 ceph-mon[32884]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:31:29.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:31:29 smithi191 ceph-mon[32884]: 1.5 scrub starts 2024-09-10T19:31:29.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:31:29 smithi191 ceph-mon[32884]: 1.5 scrub ok 2024-09-10T19:31:30.216 INFO:journalctl@ceph.iscsi.iscsi.a.smithi191.stdout:Sep 10 19:31:30 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-iscsi-iscsi-a[61816]: debug there is no tcmu-runner data available 2024-09-10T19:31:30.614 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:31:30 smithi183 ceph-mon[29736]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T19:31:30.614 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:31:30 smithi183 ceph-mon[29736]: from='client.? 172.21.15.183:0/3989222556' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-10T19:31:30.614 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:31:30 smithi183 ceph-mon[29736]: osdmap e429: 8 total, 8 up, 7 in 2024-09-10T19:31:30.614 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:31:30 smithi183 ceph-mon[29736]: purged_snaps scrub starts 2024-09-10T19:31:30.615 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:31:30 smithi183 ceph-mon[29736]: purged_snaps scrub ok 2024-09-10T19:31:30.615 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:31:30 smithi183 ceph-mon[41562]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T19:31:30.615 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:31:30 smithi183 ceph-mon[41562]: from='client.? 172.21.15.183:0/3989222556' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-10T19:31:30.615 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:31:30 smithi183 ceph-mon[41562]: osdmap e429: 8 total, 8 up, 7 in 2024-09-10T19:31:30.615 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:31:30 smithi183 ceph-mon[41562]: purged_snaps scrub starts 2024-09-10T19:31:30.615 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:31:30 smithi183 ceph-mon[41562]: purged_snaps scrub ok 2024-09-10T19:31:30.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:31:30 smithi191 ceph-mon[32884]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-10T19:31:30.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:31:30 smithi191 ceph-mon[32884]: from='client.? 172.21.15.183:0/3989222556' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-10T19:31:30.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:31:30 smithi191 ceph-mon[32884]: osdmap e429: 8 total, 8 up, 7 in 2024-09-10T19:31:30.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:31:30 smithi191 ceph-mon[32884]: purged_snaps scrub starts 2024-09-10T19:31:30.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:31:30 smithi191 ceph-mon[32884]: purged_snaps scrub ok 2024-09-10T19:31:31.613 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:31:31 smithi183 ceph-mon[29736]: pgmap v982: 142 pgs: 142 active+clean; 582 KiB data, 5.5 GiB used, 620 GiB / 626 GiB avail; 760 B/s rd, 0 op/s 2024-09-10T19:31:31.613 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:31:31 smithi183 ceph-mon[29736]: from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:31:31.613 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:31:31 smithi183 ceph-mon[29736]: osdmap e430: 8 total, 8 up, 7 in 2024-09-10T19:31:31.613 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:31:31 smithi183 ceph-mon[41562]: pgmap v982: 142 pgs: 142 active+clean; 582 KiB data, 5.5 GiB used, 620 GiB / 626 GiB avail; 760 B/s rd, 0 op/s 2024-09-10T19:31:31.613 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:31:31 smithi183 ceph-mon[41562]: from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:31:31.614 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:31:31 smithi183 ceph-mon[41562]: osdmap e430: 8 total, 8 up, 7 in 2024-09-10T19:31:31.614 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Sep 10 19:31:31 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-y[29989]: ::ffff:172.21.15.191 - - [10/Sep/2024:19:31:31] "GET /metrics HTTP/1.1" 200 222422 "" "Prometheus/2.43.0" 2024-09-10T19:31:31.715 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:31:31 smithi191 ceph-mon[32884]: pgmap v982: 142 pgs: 142 active+clean; 582 KiB data, 5.5 GiB used, 620 GiB / 626 GiB avail; 760 B/s rd, 0 op/s 2024-09-10T19:31:31.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:31:31 smithi191 ceph-mon[32884]: from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:31:31.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:31:31 smithi191 ceph-mon[32884]: osdmap e430: 8 total, 8 up, 7 in 2024-09-10T19:31:31.825 INFO:teuthology.orchestra.run.smithi183.stdout:ERROR: (22) Invalid argument 2024-09-10T19:31:31.826 INFO:teuthology.orchestra.run.smithi183.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-10T19:31:32.323 INFO:teuthology.orchestra.run.smithi183.stderr:marked in osd.2. 2024-09-10T19:31:32.431 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:31:32 smithi183 ceph-mon[41562]: from='client.? 172.21.15.183:0/788928949' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["2"]}]: dispatch 2024-09-10T19:31:32.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:31:32 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mon-a[29713]: 2024-09-10T19:31:32.174+0000 7f9f28e29640 -1 mon.a@0(leader).osd e430 definitely_dead 0 2024-09-10T19:31:32.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:31:32 smithi183 ceph-mon[29736]: from='client.? 172.21.15.183:0/788928949' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["2"]}]: dispatch 2024-09-10T19:31:32.715 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:31:32 smithi191 ceph-mon[32884]: from='client.? 172.21.15.183:0/788928949' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["2"]}]: dispatch 2024-09-10T19:31:33.602 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:31:33 smithi183 ceph-mon[29736]: pgmap v984: 142 pgs: 142 active+clean; 582 KiB data, 5.5 GiB used, 620 GiB / 626 GiB avail; 761 B/s rd, 0 op/s 2024-09-10T19:31:33.603 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:31:33 smithi183 ceph-mon[29736]: from='client.? 172.21.15.183:0/788928949' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["2"]}]': finished 2024-09-10T19:31:33.603 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:31:33 smithi183 ceph-mon[29736]: osdmap e431: 8 total, 8 up, 8 in 2024-09-10T19:31:33.603 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:31:33 smithi183 ceph-mon[41562]: pgmap v984: 142 pgs: 142 active+clean; 582 KiB data, 5.5 GiB used, 620 GiB / 626 GiB avail; 761 B/s rd, 0 op/s 2024-09-10T19:31:33.603 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:31:33 smithi183 ceph-mon[41562]: from='client.? 172.21.15.183:0/788928949' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["2"]}]': finished 2024-09-10T19:31:33.603 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:31:33 smithi183 ceph-mon[41562]: osdmap e431: 8 total, 8 up, 8 in 2024-09-10T19:31:33.604 INFO:tasks.thrashosds.thrasher:Added osd 2 2024-09-10T19:31:33.604 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph -- tell 'mon.*' injectargs --mon_osd_down_out_interval=600 2024-09-10T19:31:33.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:31:33 smithi191 ceph-mon[32884]: pgmap v984: 142 pgs: 142 active+clean; 582 KiB data, 5.5 GiB used, 620 GiB / 626 GiB avail; 761 B/s rd, 0 op/s 2024-09-10T19:31:33.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:31:33 smithi191 ceph-mon[32884]: from='client.? 172.21.15.183:0/788928949' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["2"]}]': finished 2024-09-10T19:31:33.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:31:33 smithi191 ceph-mon[32884]: osdmap e431: 8 total, 8 up, 8 in 2024-09-10T19:31:34.036 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph daemon osd.1 dump_historic_ops 2024-09-10T19:31:34.090 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:31:34.680 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:31:34 smithi183 ceph-mon[29736]: osdmap e432: 8 total, 8 up, 8 in 2024-09-10T19:31:34.681 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:31:34 smithi183 ceph-mon[41562]: osdmap e432: 8 total, 8 up, 8 in 2024-09-10T19:31:34.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:31:34 smithi191 ceph-mon[32884]: osdmap e432: 8 total, 8 up, 8 in 2024-09-10T19:31:34.916 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:31:35.117 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph osd unset noscrub 2024-09-10T19:31:35.461 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:31:35.635 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:31:35 smithi183 ceph-mon[29736]: pgmap v987: 142 pgs: 4 peering, 138 active+clean; 582 KiB data, 5.5 GiB used, 620 GiB / 626 GiB avail; 652 B/s rd, 0 op/s 2024-09-10T19:31:35.636 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:31:35 smithi183 ceph-mon[41562]: pgmap v987: 142 pgs: 4 peering, 138 active+clean; 582 KiB data, 5.5 GiB used, 620 GiB / 626 GiB avail; 652 B/s rd, 0 op/s 2024-09-10T19:31:35.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:31:35 smithi191 ceph-mon[32884]: pgmap v987: 142 pgs: 4 peering, 138 active+clean; 582 KiB data, 5.5 GiB used, 620 GiB / 626 GiB avail; 652 B/s rd, 0 op/s 2024-09-10T19:31:37.617 INFO:teuthology.orchestra.run.smithi183.stdout:ERROR: (22) Invalid argument 2024-09-10T19:31:37.617 INFO:teuthology.orchestra.run.smithi183.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-10T19:31:37.617 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:31:37 smithi183 ceph-mon[29736]: pgmap v988: 142 pgs: 4 peering, 138 active+clean; 582 KiB data, 5.7 GiB used, 709 GiB / 715 GiB avail; 170 B/s rd, 0 op/s; 43 KiB/s, 5 objects/s recovering 2024-09-10T19:31:37.617 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:31:37 smithi183 ceph-mon[41562]: pgmap v988: 142 pgs: 4 peering, 138 active+clean; 582 KiB data, 5.7 GiB used, 709 GiB / 715 GiB avail; 170 B/s rd, 0 op/s; 43 KiB/s, 5 objects/s recovering 2024-09-10T19:31:37.715 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:31:37 smithi191 ceph-mon[32884]: pgmap v988: 142 pgs: 4 peering, 138 active+clean; 582 KiB data, 5.7 GiB used, 709 GiB / 715 GiB avail; 170 B/s rd, 0 op/s; 43 KiB/s, 5 objects/s recovering 2024-09-10T19:31:37.716 INFO:journalctl@ceph.mgr.x.smithi191.stdout:Sep 10 19:31:37 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-x[34967]: ::ffff:172.21.15.191 - - [10/Sep/2024:19:31:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T19:31:38.362 INFO:teuthology.orchestra.run.smithi183.stdout:mon.a: {} 2024-09-10T19:31:38.362 INFO:teuthology.orchestra.run.smithi183.stderr:mon.a: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-09-10T19:31:38.370 INFO:teuthology.orchestra.run.smithi183.stdout:mon.b: {} 2024-09-10T19:31:38.370 INFO:teuthology.orchestra.run.smithi183.stderr:mon.b: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-09-10T19:31:38.378 INFO:teuthology.orchestra.run.smithi183.stdout:mon.c: {} 2024-09-10T19:31:38.378 INFO:teuthology.orchestra.run.smithi183.stderr:mon.c: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-09-10T19:31:39.187 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph daemon osd.0 dump_ops_in_flight 2024-09-10T19:31:39.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:31:39 smithi183 ceph-mon[29736]: pgmap v989: 142 pgs: 142 active+clean; 582 KiB data, 5.7 GiB used, 709 GiB / 715 GiB avail; 798 B/s rd, 0 op/s; 34 KiB/s, 7 objects/s recovering 2024-09-10T19:31:39.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:31:39 smithi183 ceph-mon[29736]: from='client.? 172.21.15.183:0/2620904815' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-10T19:31:39.431 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:31:39 smithi183 ceph-mon[41562]: pgmap v989: 142 pgs: 142 active+clean; 582 KiB data, 5.7 GiB used, 709 GiB / 715 GiB avail; 798 B/s rd, 0 op/s; 34 KiB/s, 7 objects/s recovering 2024-09-10T19:31:39.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:31:39 smithi183 ceph-mon[41562]: from='client.? 172.21.15.183:0/2620904815' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-10T19:31:39.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:31:39 smithi191 ceph-mon[32884]: pgmap v989: 142 pgs: 142 active+clean; 582 KiB data, 5.7 GiB used, 709 GiB / 715 GiB avail; 798 B/s rd, 0 op/s; 34 KiB/s, 7 objects/s recovering 2024-09-10T19:31:39.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:31:39 smithi191 ceph-mon[32884]: from='client.? 172.21.15.183:0/2620904815' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-10T19:31:40.465 INFO:journalctl@ceph.iscsi.iscsi.a.smithi191.stdout:Sep 10 19:31:40 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-iscsi-iscsi-a[61816]: debug there is no tcmu-runner data available 2024-09-10T19:31:40.531 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:31:40.959 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph osd reweight 2 1 2024-09-10T19:31:41.351 INFO:teuthology.orchestra.run.smithi183.stderr:noscrub is unset 2024-09-10T19:31:41.681 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:31:41 smithi183 ceph-mon[29736]: pgmap v990: 142 pgs: 142 active+clean; 582 KiB data, 5.7 GiB used, 709 GiB / 715 GiB avail; 767 B/s rd, 0 op/s; 32 KiB/s, 6 objects/s recovering 2024-09-10T19:31:41.682 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:31:41 smithi183 ceph-mon[29736]: from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:31:41.682 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:31:41 smithi183 ceph-mon[29736]: from='client.? 172.21.15.183:0/2037076119' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-10T19:31:41.682 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:31:41 smithi183 ceph-mon[29736]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-10T19:31:41.682 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:31:41 smithi183 ceph-mon[41562]: pgmap v990: 142 pgs: 142 active+clean; 582 KiB data, 5.7 GiB used, 709 GiB / 715 GiB avail; 767 B/s rd, 0 op/s; 32 KiB/s, 6 objects/s recovering 2024-09-10T19:31:41.682 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:31:41 smithi183 ceph-mon[41562]: from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:31:41.682 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:31:41 smithi183 ceph-mon[41562]: from='client.? 172.21.15.183:0/2037076119' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-10T19:31:41.682 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:31:41 smithi183 ceph-mon[41562]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-10T19:31:41.683 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Sep 10 19:31:41 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-y[29989]: ::ffff:172.21.15.191 - - [10/Sep/2024:19:31:41] "GET /metrics HTTP/1.1" 200 222422 "" "Prometheus/2.43.0" 2024-09-10T19:31:41.715 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:31:41 smithi191 ceph-mon[32884]: pgmap v990: 142 pgs: 142 active+clean; 582 KiB data, 5.7 GiB used, 709 GiB / 715 GiB avail; 767 B/s rd, 0 op/s; 32 KiB/s, 6 objects/s recovering 2024-09-10T19:31:41.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:31:41 smithi191 ceph-mon[32884]: from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:31:41.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:31:41 smithi191 ceph-mon[32884]: from='client.? 172.21.15.183:0/2037076119' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-10T19:31:41.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:31:41 smithi191 ceph-mon[32884]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-10T19:31:42.681 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:31:42 smithi183 ceph-mon[29736]: Health check cleared: OSDMAP_FLAGS (was: noscrub flag(s) set) 2024-09-10T19:31:42.682 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:31:42 smithi183 ceph-mon[29736]: Cluster is now healthy 2024-09-10T19:31:42.682 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:31:42 smithi183 ceph-mon[29736]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-10T19:31:42.682 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:31:42 smithi183 ceph-mon[29736]: osdmap e433: 8 total, 8 up, 8 in 2024-09-10T19:31:42.682 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:31:42 smithi183 ceph-mon[29736]: 2.10 scrub starts 2024-09-10T19:31:42.682 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:31:42 smithi183 ceph-mon[29736]: 2.10 scrub ok 2024-09-10T19:31:42.682 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:31:42 smithi183 ceph-mon[29736]: 6.1b deep-scrub starts 2024-09-10T19:31:42.682 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:31:42 smithi183 ceph-mon[29736]: 6.1b deep-scrub ok 2024-09-10T19:31:42.682 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:31:42 smithi183 ceph-mon[41562]: Health check cleared: OSDMAP_FLAGS (was: noscrub flag(s) set) 2024-09-10T19:31:42.683 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:31:42 smithi183 ceph-mon[41562]: Cluster is now healthy 2024-09-10T19:31:42.683 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:31:42 smithi183 ceph-mon[41562]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-10T19:31:42.683 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:31:42 smithi183 ceph-mon[41562]: osdmap e433: 8 total, 8 up, 8 in 2024-09-10T19:31:42.683 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:31:42 smithi183 ceph-mon[41562]: 2.10 scrub starts 2024-09-10T19:31:42.683 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:31:42 smithi183 ceph-mon[41562]: 2.10 scrub ok 2024-09-10T19:31:42.683 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:31:42 smithi183 ceph-mon[41562]: 6.1b deep-scrub starts 2024-09-10T19:31:42.683 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:31:42 smithi183 ceph-mon[41562]: 6.1b deep-scrub ok 2024-09-10T19:31:42.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:31:42 smithi191 ceph-mon[32884]: Health check cleared: OSDMAP_FLAGS (was: noscrub flag(s) set) 2024-09-10T19:31:42.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:31:42 smithi191 ceph-mon[32884]: Cluster is now healthy 2024-09-10T19:31:42.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:31:42 smithi191 ceph-mon[32884]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-10T19:31:42.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:31:42 smithi191 ceph-mon[32884]: osdmap e433: 8 total, 8 up, 8 in 2024-09-10T19:31:42.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:31:42 smithi191 ceph-mon[32884]: 2.10 scrub starts 2024-09-10T19:31:42.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:31:42 smithi191 ceph-mon[32884]: 2.10 scrub ok 2024-09-10T19:31:42.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:31:42 smithi191 ceph-mon[32884]: 6.1b deep-scrub starts 2024-09-10T19:31:42.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:31:42 smithi191 ceph-mon[32884]: 6.1b deep-scrub ok 2024-09-10T19:31:42.775 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph osd unset nodeep-scrub 2024-09-10T19:31:42.822 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:31:43.569 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:31:43 smithi183 ceph-mon[29736]: pgmap v992: 142 pgs: 142 active+clean; 582 KiB data, 5.7 GiB used, 709 GiB / 715 GiB avail; 353 B/s rd, 0 op/s; 30 KiB/s, 6 objects/s recovering 2024-09-10T19:31:43.569 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:31:43 smithi183 ceph-mon[29736]: 6.1 deep-scrub starts 2024-09-10T19:31:43.569 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:31:43 smithi183 ceph-mon[29736]: 6.1 deep-scrub ok 2024-09-10T19:31:43.569 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:31:43 smithi183 ceph-mon[29736]: 4.1c scrub starts 2024-09-10T19:31:43.569 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:31:43 smithi183 ceph-mon[29736]: 4.1c scrub ok 2024-09-10T19:31:43.570 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:31:43 smithi183 ceph-mon[29736]: 4.c scrub starts 2024-09-10T19:31:43.570 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:31:43 smithi183 ceph-mon[29736]: 4.c scrub ok 2024-09-10T19:31:43.570 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:31:43 smithi183 ceph-mon[41562]: pgmap v992: 142 pgs: 142 active+clean; 582 KiB data, 5.7 GiB used, 709 GiB / 715 GiB avail; 353 B/s rd, 0 op/s; 30 KiB/s, 6 objects/s recovering 2024-09-10T19:31:43.570 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:31:43 smithi183 ceph-mon[41562]: 6.1 deep-scrub starts 2024-09-10T19:31:43.570 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:31:43 smithi183 ceph-mon[41562]: 6.1 deep-scrub ok 2024-09-10T19:31:43.570 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:31:43 smithi183 ceph-mon[41562]: 4.1c scrub starts 2024-09-10T19:31:43.570 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:31:43 smithi183 ceph-mon[41562]: 4.1c scrub ok 2024-09-10T19:31:43.570 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:31:43 smithi183 ceph-mon[41562]: 4.c scrub starts 2024-09-10T19:31:43.570 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:31:43 smithi183 ceph-mon[41562]: 4.c scrub ok 2024-09-10T19:31:43.574 INFO:teuthology.orchestra.run.smithi183.stdout:ERROR: (22) Invalid argument 2024-09-10T19:31:43.574 INFO:teuthology.orchestra.run.smithi183.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-10T19:31:43.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:31:43 smithi191 ceph-mon[32884]: pgmap v992: 142 pgs: 142 active+clean; 582 KiB data, 5.7 GiB used, 709 GiB / 715 GiB avail; 353 B/s rd, 0 op/s; 30 KiB/s, 6 objects/s recovering 2024-09-10T19:31:43.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:31:43 smithi191 ceph-mon[32884]: 6.1 deep-scrub starts 2024-09-10T19:31:43.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:31:43 smithi191 ceph-mon[32884]: 6.1 deep-scrub ok 2024-09-10T19:31:43.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:31:43 smithi191 ceph-mon[32884]: 4.1c scrub starts 2024-09-10T19:31:43.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:31:43 smithi191 ceph-mon[32884]: 4.1c scrub ok 2024-09-10T19:31:43.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:31:43 smithi191 ceph-mon[32884]: 4.c scrub starts 2024-09-10T19:31:43.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:31:43 smithi191 ceph-mon[32884]: 4.c scrub ok 2024-09-10T19:31:44.431 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:31:44 smithi183 ceph-mon[41562]: 6.1a scrub starts 2024-09-10T19:31:44.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:31:44 smithi183 ceph-mon[41562]: 6.1a scrub ok 2024-09-10T19:31:44.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:31:44 smithi183 ceph-mon[41562]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:31:44.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:31:44 smithi183 ceph-mon[41562]: 6.13 scrub starts 2024-09-10T19:31:44.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:31:44 smithi183 ceph-mon[41562]: 6.13 scrub ok 2024-09-10T19:31:44.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:31:44 smithi183 ceph-mon[29736]: 6.1a scrub starts 2024-09-10T19:31:44.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:31:44 smithi183 ceph-mon[29736]: 6.1a scrub ok 2024-09-10T19:31:44.433 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:31:44 smithi183 ceph-mon[29736]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:31:44.433 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:31:44 smithi183 ceph-mon[29736]: 6.13 scrub starts 2024-09-10T19:31:44.433 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:31:44 smithi183 ceph-mon[29736]: 6.13 scrub ok 2024-09-10T19:31:44.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:31:44 smithi191 ceph-mon[32884]: 6.1a scrub starts 2024-09-10T19:31:44.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:31:44 smithi191 ceph-mon[32884]: 6.1a scrub ok 2024-09-10T19:31:44.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:31:44 smithi191 ceph-mon[32884]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:31:44.717 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:31:44 smithi191 ceph-mon[32884]: 6.13 scrub starts 2024-09-10T19:31:44.717 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:31:44 smithi191 ceph-mon[32884]: 6.13 scrub ok 2024-09-10T19:31:45.375 INFO:teuthology.orchestra.run.smithi183.stderr:reweighted osd.2 to 1 (10000) 2024-09-10T19:31:45.389 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:31:45 smithi183 ceph-mon[29736]: pgmap v993: 142 pgs: 142 active+clean; 582 KiB data, 5.7 GiB used, 709 GiB / 715 GiB avail; 819 B/s rd, 0 op/s; 26 KiB/s, 5 objects/s recovering 2024-09-10T19:31:45.389 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:31:45 smithi183 ceph-mon[29736]: 2.4 scrub starts 2024-09-10T19:31:45.389 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:31:45 smithi183 ceph-mon[29736]: 2.4 scrub ok 2024-09-10T19:31:45.390 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:31:45 smithi183 ceph-mon[29736]: 5.e deep-scrub starts 2024-09-10T19:31:45.390 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:31:45 smithi183 ceph-mon[29736]: 5.e deep-scrub ok 2024-09-10T19:31:45.390 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:31:45 smithi183 ceph-mon[29736]: purged_snaps scrub starts 2024-09-10T19:31:45.390 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:31:45 smithi183 ceph-mon[29736]: purged_snaps scrub ok 2024-09-10T19:31:45.390 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:31:45 smithi183 ceph-mon[29736]: 2.19 scrub starts 2024-09-10T19:31:45.390 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:31:45 smithi183 ceph-mon[29736]: 2.19 scrub ok 2024-09-10T19:31:45.391 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:31:45 smithi183 ceph-mon[29736]: from='client.? 172.21.15.183:0/3585045471' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-09-10T19:31:45.391 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:31:45 smithi183 ceph-mon[41562]: pgmap v993: 142 pgs: 142 active+clean; 582 KiB data, 5.7 GiB used, 709 GiB / 715 GiB avail; 819 B/s rd, 0 op/s; 26 KiB/s, 5 objects/s recovering 2024-09-10T19:31:45.392 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:31:45 smithi183 ceph-mon[41562]: 2.4 scrub starts 2024-09-10T19:31:45.392 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:31:45 smithi183 ceph-mon[41562]: 2.4 scrub ok 2024-09-10T19:31:45.392 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:31:45 smithi183 ceph-mon[41562]: 5.e deep-scrub starts 2024-09-10T19:31:45.392 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:31:45 smithi183 ceph-mon[41562]: 5.e deep-scrub ok 2024-09-10T19:31:45.392 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:31:45 smithi183 ceph-mon[41562]: purged_snaps scrub starts 2024-09-10T19:31:45.393 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:31:45 smithi183 ceph-mon[41562]: purged_snaps scrub ok 2024-09-10T19:31:45.393 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:31:45 smithi183 ceph-mon[41562]: 2.19 scrub starts 2024-09-10T19:31:45.393 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:31:45 smithi183 ceph-mon[41562]: 2.19 scrub ok 2024-09-10T19:31:45.393 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:31:45 smithi183 ceph-mon[41562]: from='client.? 172.21.15.183:0/3585045471' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-09-10T19:31:45.671 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:31:45.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:31:45 smithi191 ceph-mon[32884]: pgmap v993: 142 pgs: 142 active+clean; 582 KiB data, 5.7 GiB used, 709 GiB / 715 GiB avail; 819 B/s rd, 0 op/s; 26 KiB/s, 5 objects/s recovering 2024-09-10T19:31:45.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:31:45 smithi191 ceph-mon[32884]: 2.4 scrub starts 2024-09-10T19:31:45.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:31:45 smithi191 ceph-mon[32884]: 2.4 scrub ok 2024-09-10T19:31:45.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:31:45 smithi191 ceph-mon[32884]: 5.e deep-scrub starts 2024-09-10T19:31:45.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:31:45 smithi191 ceph-mon[32884]: 5.e deep-scrub ok 2024-09-10T19:31:45.717 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:31:45 smithi191 ceph-mon[32884]: purged_snaps scrub starts 2024-09-10T19:31:45.717 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:31:45 smithi191 ceph-mon[32884]: purged_snaps scrub ok 2024-09-10T19:31:45.717 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:31:45 smithi191 ceph-mon[32884]: 2.19 scrub starts 2024-09-10T19:31:45.717 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:31:45 smithi191 ceph-mon[32884]: 2.19 scrub ok 2024-09-10T19:31:45.717 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:31:45 smithi191 ceph-mon[32884]: from='client.? 172.21.15.183:0/3585045471' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-09-10T19:31:46.093 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph osd primary-affinity 2 1 2024-09-10T19:31:46.682 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:31:46 smithi183 ceph-mon[29736]: 4.5 scrub starts 2024-09-10T19:31:46.682 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:31:46 smithi183 ceph-mon[29736]: 4.5 scrub ok 2024-09-10T19:31:46.682 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:31:46 smithi183 ceph-mon[29736]: from='client.? 172.21.15.183:0/3585045471' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-09-10T19:31:46.682 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:31:46 smithi183 ceph-mon[29736]: osdmap e434: 8 total, 8 up, 8 in 2024-09-10T19:31:46.682 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:31:46 smithi183 ceph-mon[29736]: 1.a scrub starts 2024-09-10T19:31:46.682 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:31:46 smithi183 ceph-mon[29736]: 1.a scrub ok 2024-09-10T19:31:46.682 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:31:46 smithi183 ceph-mon[29736]: 5.3 scrub starts 2024-09-10T19:31:46.682 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:31:46 smithi183 ceph-mon[29736]: 5.3 scrub ok 2024-09-10T19:31:46.682 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:31:46 smithi183 ceph-mon[29736]: pgmap v995: 142 pgs: 142 active+clean; 582 KiB data, 5.7 GiB used, 709 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-10T19:31:46.683 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:31:46 smithi183 ceph-mon[29736]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.0", "id": [3, 2]}]: dispatch 2024-09-10T19:31:46.683 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:31:46 smithi183 ceph-mon[29736]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.18", "id": [1, 2]}]: dispatch 2024-09-10T19:31:46.683 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:31:46 smithi183 ceph-mon[29736]: from='mgr.14556 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.0", "id": [3, 2]}]: dispatch 2024-09-10T19:31:46.683 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:31:46 smithi183 ceph-mon[29736]: from='mgr.14556 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.18", "id": [1, 2]}]: dispatch 2024-09-10T19:31:46.683 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:31:46 smithi183 ceph-mon[41562]: 4.5 scrub starts 2024-09-10T19:31:46.683 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:31:46 smithi183 ceph-mon[41562]: 4.5 scrub ok 2024-09-10T19:31:46.683 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:31:46 smithi183 ceph-mon[41562]: from='client.? 172.21.15.183:0/3585045471' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-09-10T19:31:46.683 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:31:46 smithi183 ceph-mon[41562]: osdmap e434: 8 total, 8 up, 8 in 2024-09-10T19:31:46.683 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:31:46 smithi183 ceph-mon[41562]: 1.a scrub starts 2024-09-10T19:31:46.683 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:31:46 smithi183 ceph-mon[41562]: 1.a scrub ok 2024-09-10T19:31:46.683 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:31:46 smithi183 ceph-mon[41562]: 5.3 scrub starts 2024-09-10T19:31:46.684 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:31:46 smithi183 ceph-mon[41562]: 5.3 scrub ok 2024-09-10T19:31:46.684 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:31:46 smithi183 ceph-mon[41562]: pgmap v995: 142 pgs: 142 active+clean; 582 KiB data, 5.7 GiB used, 709 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-10T19:31:46.684 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:31:46 smithi183 ceph-mon[41562]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.0", "id": [3, 2]}]: dispatch 2024-09-10T19:31:46.684 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:31:46 smithi183 ceph-mon[41562]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.18", "id": [1, 2]}]: dispatch 2024-09-10T19:31:46.684 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:31:46 smithi183 ceph-mon[41562]: from='mgr.14556 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.0", "id": [3, 2]}]: dispatch 2024-09-10T19:31:46.684 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:31:46 smithi183 ceph-mon[41562]: from='mgr.14556 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.18", "id": [1, 2]}]: dispatch 2024-09-10T19:31:46.717 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:31:46 smithi191 ceph-mon[32884]: 4.5 scrub starts 2024-09-10T19:31:46.717 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:31:46 smithi191 ceph-mon[32884]: 4.5 scrub ok 2024-09-10T19:31:46.717 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:31:46 smithi191 ceph-mon[32884]: from='client.? 172.21.15.183:0/3585045471' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-09-10T19:31:46.717 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:31:46 smithi191 ceph-mon[32884]: osdmap e434: 8 total, 8 up, 8 in 2024-09-10T19:31:46.717 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:31:46 smithi191 ceph-mon[32884]: 1.a scrub starts 2024-09-10T19:31:46.717 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:31:46 smithi191 ceph-mon[32884]: 1.a scrub ok 2024-09-10T19:31:46.717 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:31:46 smithi191 ceph-mon[32884]: 5.3 scrub starts 2024-09-10T19:31:46.717 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:31:46 smithi191 ceph-mon[32884]: 5.3 scrub ok 2024-09-10T19:31:46.717 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:31:46 smithi191 ceph-mon[32884]: pgmap v995: 142 pgs: 142 active+clean; 582 KiB data, 5.7 GiB used, 709 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-10T19:31:46.717 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:31:46 smithi191 ceph-mon[32884]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.0", "id": [3, 2]}]: dispatch 2024-09-10T19:31:46.718 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:31:46 smithi191 ceph-mon[32884]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.18", "id": [1, 2]}]: dispatch 2024-09-10T19:31:46.718 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:31:46 smithi191 ceph-mon[32884]: from='mgr.14556 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.0", "id": [3, 2]}]: dispatch 2024-09-10T19:31:46.718 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:31:46 smithi191 ceph-mon[32884]: from='mgr.14556 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.18", "id": [1, 2]}]: dispatch 2024-09-10T19:31:46.754 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph daemon osd.0 dump_blocked_ops 2024-09-10T19:31:47.390 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:31:47 smithi183 ceph-mon[41562]: 6.5 scrub starts 2024-09-10T19:31:47.390 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:31:47 smithi183 ceph-mon[41562]: 6.5 scrub ok 2024-09-10T19:31:47.390 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:31:47 smithi183 ceph-mon[41562]: from='mgr.14556 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.0", "id": [3, 2]}]': finished 2024-09-10T19:31:47.390 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:31:47 smithi183 ceph-mon[41562]: from='mgr.14556 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.18", "id": [1, 2]}]': finished 2024-09-10T19:31:47.390 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:31:47 smithi183 ceph-mon[41562]: osdmap e435: 8 total, 8 up, 8 in 2024-09-10T19:31:47.391 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:31:47 smithi183 ceph-mon[41562]: 4.e scrub starts 2024-09-10T19:31:47.391 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:31:47 smithi183 ceph-mon[41562]: 4.e scrub ok 2024-09-10T19:31:47.391 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:31:47 smithi183 ceph-mon[41562]: 4.3 scrub starts 2024-09-10T19:31:47.391 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:31:47 smithi183 ceph-mon[41562]: 4.3 scrub ok 2024-09-10T19:31:47.681 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:31:47 smithi183 ceph-mon[29736]: 6.5 scrub starts 2024-09-10T19:31:47.682 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:31:47 smithi183 ceph-mon[29736]: 6.5 scrub ok 2024-09-10T19:31:47.682 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:31:47 smithi183 ceph-mon[29736]: from='mgr.14556 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.0", "id": [3, 2]}]': finished 2024-09-10T19:31:47.682 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:31:47 smithi183 ceph-mon[29736]: from='mgr.14556 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.18", "id": [1, 2]}]': finished 2024-09-10T19:31:47.682 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:31:47 smithi183 ceph-mon[29736]: osdmap e435: 8 total, 8 up, 8 in 2024-09-10T19:31:47.682 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:31:47 smithi183 ceph-mon[29736]: 4.e scrub starts 2024-09-10T19:31:47.682 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:31:47 smithi183 ceph-mon[29736]: 4.e scrub ok 2024-09-10T19:31:47.682 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:31:47 smithi183 ceph-mon[29736]: 4.3 scrub starts 2024-09-10T19:31:47.682 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:31:47 smithi183 ceph-mon[29736]: 4.3 scrub ok 2024-09-10T19:31:47.716 INFO:journalctl@ceph.mgr.x.smithi191.stdout:Sep 10 19:31:47 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-x[34967]: ::ffff:172.21.15.191 - - [10/Sep/2024:19:31:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T19:31:47.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:31:47 smithi191 ceph-mon[32884]: 6.5 scrub starts 2024-09-10T19:31:47.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:31:47 smithi191 ceph-mon[32884]: 6.5 scrub ok 2024-09-10T19:31:47.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:31:47 smithi191 ceph-mon[32884]: from='mgr.14556 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.0", "id": [3, 2]}]': finished 2024-09-10T19:31:47.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:31:47 smithi191 ceph-mon[32884]: from='mgr.14556 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.18", "id": [1, 2]}]': finished 2024-09-10T19:31:47.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:31:47 smithi191 ceph-mon[32884]: osdmap e435: 8 total, 8 up, 8 in 2024-09-10T19:31:47.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:31:47 smithi191 ceph-mon[32884]: 4.e scrub starts 2024-09-10T19:31:47.717 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:31:47 smithi191 ceph-mon[32884]: 4.e scrub ok 2024-09-10T19:31:47.717 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:31:47 smithi191 ceph-mon[32884]: 4.3 scrub starts 2024-09-10T19:31:47.717 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:31:47 smithi191 ceph-mon[32884]: 4.3 scrub ok 2024-09-10T19:31:47.850 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:31:47.888 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:31:48.656 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:31:48 smithi183 ceph-mon[29736]: 2.0 deep-scrub starts 2024-09-10T19:31:48.656 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:31:48 smithi183 ceph-mon[29736]: 2.0 deep-scrub ok 2024-09-10T19:31:48.656 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:31:48 smithi183 ceph-mon[29736]: 5.1d scrub starts 2024-09-10T19:31:48.656 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:31:48 smithi183 ceph-mon[29736]: 5.1d scrub ok 2024-09-10T19:31:48.657 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:31:48 smithi183 ceph-mon[29736]: osdmap e436: 8 total, 8 up, 8 in 2024-09-10T19:31:48.657 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:31:48 smithi183 ceph-mon[29736]: 4.0 deep-scrub starts 2024-09-10T19:31:48.657 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:31:48 smithi183 ceph-mon[29736]: 4.0 deep-scrub ok 2024-09-10T19:31:48.657 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:31:48 smithi183 ceph-mon[29736]: pgmap v998: 142 pgs: 142 active+clean; 582 KiB data, 5.7 GiB used, 709 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-09-10T19:31:48.657 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:31:48 smithi183 ceph-mon[41562]: 2.0 deep-scrub starts 2024-09-10T19:31:48.657 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:31:48 smithi183 ceph-mon[41562]: 2.0 deep-scrub ok 2024-09-10T19:31:48.657 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:31:48 smithi183 ceph-mon[41562]: 5.1d scrub starts 2024-09-10T19:31:48.657 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:31:48 smithi183 ceph-mon[41562]: 5.1d scrub ok 2024-09-10T19:31:48.657 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:31:48 smithi183 ceph-mon[41562]: osdmap e436: 8 total, 8 up, 8 in 2024-09-10T19:31:48.657 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:31:48 smithi183 ceph-mon[41562]: 4.0 deep-scrub starts 2024-09-10T19:31:48.657 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:31:48 smithi183 ceph-mon[41562]: 4.0 deep-scrub ok 2024-09-10T19:31:48.658 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:31:48 smithi183 ceph-mon[41562]: pgmap v998: 142 pgs: 142 active+clean; 582 KiB data, 5.7 GiB used, 709 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-09-10T19:31:48.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:31:48 smithi191 ceph-mon[32884]: 2.0 deep-scrub starts 2024-09-10T19:31:48.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:31:48 smithi191 ceph-mon[32884]: 2.0 deep-scrub ok 2024-09-10T19:31:48.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:31:48 smithi191 ceph-mon[32884]: 5.1d scrub starts 2024-09-10T19:31:48.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:31:48 smithi191 ceph-mon[32884]: 5.1d scrub ok 2024-09-10T19:31:48.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:31:48 smithi191 ceph-mon[32884]: osdmap e436: 8 total, 8 up, 8 in 2024-09-10T19:31:48.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:31:48 smithi191 ceph-mon[32884]: 4.0 deep-scrub starts 2024-09-10T19:31:48.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:31:48 smithi191 ceph-mon[32884]: 4.0 deep-scrub ok 2024-09-10T19:31:48.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:31:48 smithi191 ceph-mon[32884]: pgmap v998: 142 pgs: 142 active+clean; 582 KiB data, 5.7 GiB used, 709 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s 2024-09-10T19:31:49.681 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:31:49 smithi183 ceph-mon[29736]: 1.7 scrub starts 2024-09-10T19:31:49.682 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:31:49 smithi183 ceph-mon[29736]: 1.7 scrub ok 2024-09-10T19:31:49.682 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:31:49 smithi183 ceph-mon[29736]: 5.17 scrub starts 2024-09-10T19:31:49.682 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:31:49 smithi183 ceph-mon[29736]: 5.17 scrub ok 2024-09-10T19:31:49.682 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:31:49 smithi183 ceph-mon[29736]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T19:31:49.682 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:31:49 smithi183 ceph-mon[29736]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T19:31:49.682 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:31:49 smithi183 ceph-mon[29736]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T19:31:49.682 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:31:49 smithi183 ceph-mon[29736]: from='mgr.14556 ' entity='mgr.y' 2024-09-10T19:31:49.682 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:31:49 smithi183 ceph-mon[29736]: from='client.? 172.21.15.183:0/1869621417' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-10T19:31:49.683 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:31:49 smithi183 ceph-mon[41562]: 1.7 scrub starts 2024-09-10T19:31:49.683 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:31:49 smithi183 ceph-mon[41562]: 1.7 scrub ok 2024-09-10T19:31:49.683 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:31:49 smithi183 ceph-mon[41562]: 5.17 scrub starts 2024-09-10T19:31:49.683 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:31:49 smithi183 ceph-mon[41562]: 5.17 scrub ok 2024-09-10T19:31:49.683 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:31:49 smithi183 ceph-mon[41562]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T19:31:49.683 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:31:49 smithi183 ceph-mon[41562]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T19:31:49.683 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:31:49 smithi183 ceph-mon[41562]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T19:31:49.683 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:31:49 smithi183 ceph-mon[41562]: from='mgr.14556 ' entity='mgr.y' 2024-09-10T19:31:49.683 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:31:49 smithi183 ceph-mon[41562]: from='client.? 172.21.15.183:0/1869621417' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-10T19:31:49.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:31:49 smithi191 ceph-mon[32884]: 1.7 scrub starts 2024-09-10T19:31:49.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:31:49 smithi191 ceph-mon[32884]: 1.7 scrub ok 2024-09-10T19:31:49.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:31:49 smithi191 ceph-mon[32884]: 5.17 scrub starts 2024-09-10T19:31:49.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:31:49 smithi191 ceph-mon[32884]: 5.17 scrub ok 2024-09-10T19:31:49.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:31:49 smithi191 ceph-mon[32884]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T19:31:49.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:31:49 smithi191 ceph-mon[32884]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T19:31:49.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:31:49 smithi191 ceph-mon[32884]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T19:31:49.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:31:49 smithi191 ceph-mon[32884]: from='mgr.14556 ' entity='mgr.y' 2024-09-10T19:31:49.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:31:49 smithi191 ceph-mon[32884]: from='client.? 172.21.15.183:0/1869621417' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-10T19:31:49.835 INFO:teuthology.orchestra.run.smithi183.stderr:nodeep-scrub is unset 2024-09-10T19:31:50.425 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:31:50 smithi183 ceph-mon[29736]: 5.11 scrub starts 2024-09-10T19:31:50.425 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:31:50 smithi183 ceph-mon[29736]: 5.11 scrub ok 2024-09-10T19:31:50.425 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:31:50 smithi183 ceph-mon[29736]: from='client.? 172.21.15.183:0/1869621417' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-10T19:31:50.425 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:31:50 smithi183 ceph-mon[29736]: osdmap e437: 8 total, 8 up, 8 in 2024-09-10T19:31:50.425 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:31:50 smithi183 ceph-mon[29736]: pgmap v1000: 142 pgs: 142 active+clean; 582 KiB data, 5.7 GiB used, 709 GiB / 715 GiB avail; 884 B/s rd, 0 op/s; 47 B/s, 1 objects/s recovering 2024-09-10T19:31:50.425 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:31:50 smithi183 ceph-mon[29736]: from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:31:50.425 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:31:50 smithi183 ceph-mon[41562]: 5.11 scrub starts 2024-09-10T19:31:50.425 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:31:50 smithi183 ceph-mon[41562]: 5.11 scrub ok 2024-09-10T19:31:50.426 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:31:50 smithi183 ceph-mon[41562]: from='client.? 172.21.15.183:0/1869621417' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-10T19:31:50.426 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:31:50 smithi183 ceph-mon[41562]: osdmap e437: 8 total, 8 up, 8 in 2024-09-10T19:31:50.426 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:31:50 smithi183 ceph-mon[41562]: pgmap v1000: 142 pgs: 142 active+clean; 582 KiB data, 5.7 GiB used, 709 GiB / 715 GiB avail; 884 B/s rd, 0 op/s; 47 B/s, 1 objects/s recovering 2024-09-10T19:31:50.426 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:31:50 smithi183 ceph-mon[41562]: from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:31:50.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:31:50 smithi191 ceph-mon[32884]: 5.11 scrub starts 2024-09-10T19:31:50.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:31:50 smithi191 ceph-mon[32884]: 5.11 scrub ok 2024-09-10T19:31:50.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:31:50 smithi191 ceph-mon[32884]: from='client.? 172.21.15.183:0/1869621417' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-10T19:31:50.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:31:50 smithi191 ceph-mon[32884]: osdmap e437: 8 total, 8 up, 8 in 2024-09-10T19:31:50.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:31:50 smithi191 ceph-mon[32884]: pgmap v1000: 142 pgs: 142 active+clean; 582 KiB data, 5.7 GiB used, 709 GiB / 715 GiB avail; 884 B/s rd, 0 op/s; 47 B/s, 1 objects/s recovering 2024-09-10T19:31:50.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:31:50 smithi191 ceph-mon[32884]: from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:31:50.466 INFO:journalctl@ceph.iscsi.iscsi.a.smithi191.stdout:Sep 10 19:31:50 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-iscsi-iscsi-a[61816]: debug there is no tcmu-runner data available 2024-09-10T19:31:51.505 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:31:51 smithi183 ceph-mon[29736]: 5.2 scrub starts 2024-09-10T19:31:51.505 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:31:51 smithi183 ceph-mon[29736]: 5.2 scrub ok 2024-09-10T19:31:51.505 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:31:51 smithi183 ceph-mon[29736]: 4.8 scrub starts 2024-09-10T19:31:51.505 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:31:51 smithi183 ceph-mon[29736]: 4.8 scrub ok 2024-09-10T19:31:51.505 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:31:51 smithi183 ceph-mon[29736]: 4.18 scrub starts 2024-09-10T19:31:51.505 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:31:51 smithi183 ceph-mon[29736]: 4.18 scrub ok 2024-09-10T19:31:51.505 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:31:51 smithi183 ceph-mon[29736]: from='client.? 172.21.15.183:0/2253164186' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]: dispatch 2024-09-10T19:31:51.505 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Sep 10 19:31:51 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-y[29989]: ::ffff:172.21.15.191 - - [10/Sep/2024:19:31:51] "GET /metrics HTTP/1.1" 200 222430 "" "Prometheus/2.43.0" 2024-09-10T19:31:51.506 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:31:51 smithi183 ceph-mon[41562]: 5.2 scrub starts 2024-09-10T19:31:51.506 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:31:51 smithi183 ceph-mon[41562]: 5.2 scrub ok 2024-09-10T19:31:51.506 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:31:51 smithi183 ceph-mon[41562]: 4.8 scrub starts 2024-09-10T19:31:51.506 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:31:51 smithi183 ceph-mon[41562]: 4.8 scrub ok 2024-09-10T19:31:51.506 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:31:51 smithi183 ceph-mon[41562]: 4.18 scrub starts 2024-09-10T19:31:51.506 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:31:51 smithi183 ceph-mon[41562]: 4.18 scrub ok 2024-09-10T19:31:51.506 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:31:51 smithi183 ceph-mon[41562]: from='client.? 172.21.15.183:0/2253164186' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]: dispatch 2024-09-10T19:31:51.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:31:51 smithi191 ceph-mon[32884]: 5.2 scrub starts 2024-09-10T19:31:51.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:31:51 smithi191 ceph-mon[32884]: 5.2 scrub ok 2024-09-10T19:31:51.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:31:51 smithi191 ceph-mon[32884]: 4.8 scrub starts 2024-09-10T19:31:51.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:31:51 smithi191 ceph-mon[32884]: 4.8 scrub ok 2024-09-10T19:31:51.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:31:51 smithi191 ceph-mon[32884]: 4.18 scrub starts 2024-09-10T19:31:51.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:31:51 smithi191 ceph-mon[32884]: 4.18 scrub ok 2024-09-10T19:31:51.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:31:51 smithi191 ceph-mon[32884]: from='client.? 172.21.15.183:0/2253164186' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]: dispatch 2024-09-10T19:31:51.839 INFO:teuthology.orchestra.run.smithi183.stderr:set osd.2 primary-affinity to 1 (8655362) 2024-09-10T19:31:52.495 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph osd reweight 3 1 2024-09-10T19:31:53.058 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:31:52 smithi183 ceph-mon[29736]: 6.11 scrub starts 2024-09-10T19:31:53.058 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:31:52 smithi183 ceph-mon[29736]: 6.11 scrub ok 2024-09-10T19:31:53.058 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:31:52 smithi183 ceph-mon[29736]: 4.d deep-scrub starts 2024-09-10T19:31:53.059 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:31:52 smithi183 ceph-mon[29736]: 4.d deep-scrub ok 2024-09-10T19:31:53.059 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:31:52 smithi183 ceph-mon[29736]: from='client.? 172.21.15.183:0/2253164186' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]': finished 2024-09-10T19:31:53.059 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:31:52 smithi183 ceph-mon[29736]: osdmap e438: 8 total, 8 up, 8 in 2024-09-10T19:31:53.059 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:31:52 smithi183 ceph-mon[29736]: pgmap v1002: 142 pgs: 142 active+clean; 582 KiB data, 5.7 GiB used, 709 GiB / 715 GiB avail; 728 B/s rd, 0 op/s; 58 B/s, 2 objects/s recovering 2024-09-10T19:31:53.059 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:31:52 smithi183 ceph-mon[41562]: 6.11 scrub starts 2024-09-10T19:31:53.059 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:31:52 smithi183 ceph-mon[41562]: 6.11 scrub ok 2024-09-10T19:31:53.059 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:31:52 smithi183 ceph-mon[41562]: 4.d deep-scrub starts 2024-09-10T19:31:53.059 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:31:52 smithi183 ceph-mon[41562]: 4.d deep-scrub ok 2024-09-10T19:31:53.059 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:31:52 smithi183 ceph-mon[41562]: from='client.? 172.21.15.183:0/2253164186' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]': finished 2024-09-10T19:31:53.059 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:31:52 smithi183 ceph-mon[41562]: osdmap e438: 8 total, 8 up, 8 in 2024-09-10T19:31:53.059 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:31:52 smithi183 ceph-mon[41562]: pgmap v1002: 142 pgs: 142 active+clean; 582 KiB data, 5.7 GiB used, 709 GiB / 715 GiB avail; 728 B/s rd, 0 op/s; 58 B/s, 2 objects/s recovering 2024-09-10T19:31:53.157 INFO:teuthology.orchestra.run.smithi183.stdout:ERROR: (22) Invalid argument 2024-09-10T19:31:53.158 INFO:teuthology.orchestra.run.smithi183.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-10T19:31:53.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:31:52 smithi191 ceph-mon[32884]: 6.11 scrub starts 2024-09-10T19:31:53.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:31:52 smithi191 ceph-mon[32884]: 6.11 scrub ok 2024-09-10T19:31:53.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:31:52 smithi191 ceph-mon[32884]: 4.d deep-scrub starts 2024-09-10T19:31:53.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:31:52 smithi191 ceph-mon[32884]: 4.d deep-scrub ok 2024-09-10T19:31:53.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:31:52 smithi191 ceph-mon[32884]: from='client.? 172.21.15.183:0/2253164186' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]': finished 2024-09-10T19:31:53.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:31:52 smithi191 ceph-mon[32884]: osdmap e438: 8 total, 8 up, 8 in 2024-09-10T19:31:53.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:31:52 smithi191 ceph-mon[32884]: pgmap v1002: 142 pgs: 142 active+clean; 582 KiB data, 5.7 GiB used, 709 GiB / 715 GiB avail; 728 B/s rd, 0 op/s; 58 B/s, 2 objects/s recovering 2024-09-10T19:31:53.989 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph daemon osd.0 dump_historic_ops 2024-09-10T19:31:54.014 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:31:54.137 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:31:53 smithi183 ceph-mon[29736]: 6.8 scrub starts 2024-09-10T19:31:54.137 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:31:53 smithi183 ceph-mon[29736]: 6.8 scrub ok 2024-09-10T19:31:54.137 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:31:53 smithi183 ceph-mon[29736]: 1.6 scrub starts 2024-09-10T19:31:54.137 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:31:53 smithi183 ceph-mon[29736]: 1.6 scrub ok 2024-09-10T19:31:54.137 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:31:53 smithi183 ceph-mon[29736]: 4.b scrub starts 2024-09-10T19:31:54.137 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:31:53 smithi183 ceph-mon[29736]: 4.b scrub ok 2024-09-10T19:31:54.138 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:31:53 smithi183 ceph-mon[41562]: 6.8 scrub starts 2024-09-10T19:31:54.138 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:31:53 smithi183 ceph-mon[41562]: 6.8 scrub ok 2024-09-10T19:31:54.138 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:31:53 smithi183 ceph-mon[41562]: 1.6 scrub starts 2024-09-10T19:31:54.138 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:31:53 smithi183 ceph-mon[41562]: 1.6 scrub ok 2024-09-10T19:31:54.138 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:31:53 smithi183 ceph-mon[41562]: 4.b scrub starts 2024-09-10T19:31:54.138 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:31:53 smithi183 ceph-mon[41562]: 4.b scrub ok 2024-09-10T19:31:54.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:31:53 smithi191 ceph-mon[32884]: 6.8 scrub starts 2024-09-10T19:31:54.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:31:53 smithi191 ceph-mon[32884]: 6.8 scrub ok 2024-09-10T19:31:54.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:31:53 smithi191 ceph-mon[32884]: 1.6 scrub starts 2024-09-10T19:31:54.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:31:53 smithi191 ceph-mon[32884]: 1.6 scrub ok 2024-09-10T19:31:54.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:31:53 smithi191 ceph-mon[32884]: 4.b scrub starts 2024-09-10T19:31:54.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:31:53 smithi191 ceph-mon[32884]: 4.b scrub ok 2024-09-10T19:31:54.784 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:31:55.098 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:31:54 smithi183 ceph-mon[41562]: 5.10 scrub starts 2024-09-10T19:31:55.098 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:31:54 smithi183 ceph-mon[41562]: 5.10 scrub ok 2024-09-10T19:31:55.098 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:31:54 smithi183 ceph-mon[41562]: 6.19 deep-scrub starts 2024-09-10T19:31:55.098 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:31:54 smithi183 ceph-mon[41562]: 6.19 deep-scrub ok 2024-09-10T19:31:55.098 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:31:54 smithi183 ceph-mon[41562]: pgmap v1003: 142 pgs: 142 active+clean; 582 KiB data, 5.7 GiB used, 709 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s; 49 B/s, 1 objects/s recovering 2024-09-10T19:31:55.098 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:31:54 smithi183 ceph-mon[29736]: 5.10 scrub starts 2024-09-10T19:31:55.098 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:31:54 smithi183 ceph-mon[29736]: 5.10 scrub ok 2024-09-10T19:31:55.098 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:31:54 smithi183 ceph-mon[29736]: 6.19 deep-scrub starts 2024-09-10T19:31:55.099 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:31:54 smithi183 ceph-mon[29736]: 6.19 deep-scrub ok 2024-09-10T19:31:55.099 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:31:54 smithi183 ceph-mon[29736]: pgmap v1003: 142 pgs: 142 active+clean; 582 KiB data, 5.7 GiB used, 709 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s; 49 B/s, 1 objects/s recovering 2024-09-10T19:31:55.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:31:54 smithi191 ceph-mon[32884]: 5.10 scrub starts 2024-09-10T19:31:55.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:31:54 smithi191 ceph-mon[32884]: 5.10 scrub ok 2024-09-10T19:31:55.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:31:54 smithi191 ceph-mon[32884]: 6.19 deep-scrub starts 2024-09-10T19:31:55.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:31:54 smithi191 ceph-mon[32884]: 6.19 deep-scrub ok 2024-09-10T19:31:55.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:31:54 smithi191 ceph-mon[32884]: pgmap v1003: 142 pgs: 142 active+clean; 582 KiB data, 5.7 GiB used, 709 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s; 49 B/s, 1 objects/s recovering 2024-09-10T19:31:55.848 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:31:55 smithi183 ceph-mon[29736]: 5.4 scrub starts 2024-09-10T19:31:55.848 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:31:55 smithi183 ceph-mon[29736]: 5.4 scrub ok 2024-09-10T19:31:55.848 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:31:55 smithi183 ceph-mon[29736]: 4.7 scrub starts 2024-09-10T19:31:56.181 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:31:55 smithi183 ceph-mon[41562]: 5.4 scrub starts 2024-09-10T19:31:56.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:31:55 smithi183 ceph-mon[41562]: 5.4 scrub ok 2024-09-10T19:31:56.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:31:55 smithi183 ceph-mon[41562]: 4.7 scrub starts 2024-09-10T19:31:56.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:31:55 smithi183 ceph-mon[41562]: 4.7 scrub ok 2024-09-10T19:31:56.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:31:55 smithi183 ceph-mon[41562]: 4.17 scrub starts 2024-09-10T19:31:56.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:31:55 smithi183 ceph-mon[41562]: 4.17 scrub ok 2024-09-10T19:31:56.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:31:55 smithi183 ceph-mon[41562]: 2.e scrub starts 2024-09-10T19:31:56.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:31:55 smithi183 ceph-mon[41562]: 2.e scrub ok 2024-09-10T19:31:56.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:31:55 smithi183 ceph-mon[41562]: 5.1b scrub starts 2024-09-10T19:31:56.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:31:55 smithi183 ceph-mon[41562]: 5.1b scrub ok 2024-09-10T19:31:56.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:31:55 smithi183 ceph-mon[41562]: 4.1b scrub starts 2024-09-10T19:31:56.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:31:55 smithi183 ceph-mon[41562]: 4.1b scrub ok 2024-09-10T19:31:56.183 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:31:55 smithi183 ceph-mon[29736]: 4.7 scrub ok 2024-09-10T19:31:56.183 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:31:55 smithi183 ceph-mon[29736]: 4.17 scrub starts 2024-09-10T19:31:56.183 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:31:55 smithi183 ceph-mon[29736]: 4.17 scrub ok 2024-09-10T19:31:56.183 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:31:55 smithi183 ceph-mon[29736]: 2.e scrub starts 2024-09-10T19:31:56.183 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:31:55 smithi183 ceph-mon[29736]: 2.e scrub ok 2024-09-10T19:31:56.183 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:31:55 smithi183 ceph-mon[29736]: 5.1b scrub starts 2024-09-10T19:31:56.183 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:31:55 smithi183 ceph-mon[29736]: 5.1b scrub ok 2024-09-10T19:31:56.183 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:31:55 smithi183 ceph-mon[29736]: 4.1b scrub starts 2024-09-10T19:31:56.183 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:31:55 smithi183 ceph-mon[29736]: 4.1b scrub ok 2024-09-10T19:31:56.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:31:55 smithi191 ceph-mon[32884]: 5.4 scrub starts 2024-09-10T19:31:56.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:31:55 smithi191 ceph-mon[32884]: 5.4 scrub ok 2024-09-10T19:31:56.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:31:55 smithi191 ceph-mon[32884]: 4.7 scrub starts 2024-09-10T19:31:56.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:31:55 smithi191 ceph-mon[32884]: 4.7 scrub ok 2024-09-10T19:31:56.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:31:55 smithi191 ceph-mon[32884]: 4.17 scrub starts 2024-09-10T19:31:56.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:31:55 smithi191 ceph-mon[32884]: 4.17 scrub ok 2024-09-10T19:31:56.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:31:55 smithi191 ceph-mon[32884]: 2.e scrub starts 2024-09-10T19:31:56.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:31:55 smithi191 ceph-mon[32884]: 2.e scrub ok 2024-09-10T19:31:56.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:31:55 smithi191 ceph-mon[32884]: 5.1b scrub starts 2024-09-10T19:31:56.217 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:31:55 smithi191 ceph-mon[32884]: 5.1b scrub ok 2024-09-10T19:31:56.217 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:31:55 smithi191 ceph-mon[32884]: 4.1b scrub starts 2024-09-10T19:31:56.217 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:31:55 smithi191 ceph-mon[32884]: 4.1b scrub ok 2024-09-10T19:31:56.853 INFO:teuthology.orchestra.run.smithi183.stderr:reweighted osd.3 to 1 (10000) 2024-09-10T19:31:57.120 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:31:56 smithi191 ceph-mon[32884]: 5.9 scrub starts 2024-09-10T19:31:57.121 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:31:56 smithi191 ceph-mon[32884]: 5.9 scrub ok 2024-09-10T19:31:57.121 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:31:56 smithi191 ceph-mon[32884]: 5.a scrub starts 2024-09-10T19:31:57.121 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:31:56 smithi191 ceph-mon[32884]: 5.a scrub ok 2024-09-10T19:31:57.121 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:31:56 smithi191 ceph-mon[32884]: 5.f scrub starts 2024-09-10T19:31:57.121 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:31:56 smithi191 ceph-mon[32884]: 5.f scrub ok 2024-09-10T19:31:57.121 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:31:56 smithi191 ceph-mon[32884]: 2.f scrub starts 2024-09-10T19:31:57.121 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:31:56 smithi191 ceph-mon[32884]: 2.f scrub ok 2024-09-10T19:31:57.121 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:31:56 smithi191 ceph-mon[32884]: 5.1f scrub starts 2024-09-10T19:31:57.121 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:31:56 smithi191 ceph-mon[32884]: 5.1f scrub ok 2024-09-10T19:31:57.121 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:31:56 smithi191 ceph-mon[32884]: 5.1e scrub starts 2024-09-10T19:31:57.121 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:31:56 smithi191 ceph-mon[32884]: 5.1e scrub ok 2024-09-10T19:31:57.122 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:31:56 smithi191 ceph-mon[32884]: pgmap v1004: 142 pgs: 142 active+clean; 582 KiB data, 5.7 GiB used, 709 GiB / 715 GiB avail; 511 B/s rd, 0 op/s; 41 B/s, 1 objects/s recovering 2024-09-10T19:31:57.122 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:31:56 smithi191 ceph-mon[32884]: from='client.? 172.21.15.183:0/2836884971' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-09-10T19:31:57.122 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:31:56 smithi191 ceph-mon[32884]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-09-10T19:31:57.181 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:31:56 smithi183 ceph-mon[41562]: 5.9 scrub starts 2024-09-10T19:31:57.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:31:56 smithi183 ceph-mon[41562]: 5.9 scrub ok 2024-09-10T19:31:57.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:31:56 smithi183 ceph-mon[41562]: 5.a scrub starts 2024-09-10T19:31:57.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:31:56 smithi183 ceph-mon[41562]: 5.a scrub ok 2024-09-10T19:31:57.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:31:56 smithi183 ceph-mon[41562]: 5.f scrub starts 2024-09-10T19:31:57.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:31:56 smithi183 ceph-mon[41562]: 5.f scrub ok 2024-09-10T19:31:57.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:31:56 smithi183 ceph-mon[41562]: 2.f scrub starts 2024-09-10T19:31:57.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:31:56 smithi183 ceph-mon[41562]: 2.f scrub ok 2024-09-10T19:31:57.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:31:56 smithi183 ceph-mon[41562]: 5.1f scrub starts 2024-09-10T19:31:57.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:31:56 smithi183 ceph-mon[41562]: 5.1f scrub ok 2024-09-10T19:31:57.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:31:56 smithi183 ceph-mon[41562]: 5.1e scrub starts 2024-09-10T19:31:57.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:31:56 smithi183 ceph-mon[41562]: 5.1e scrub ok 2024-09-10T19:31:57.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:31:56 smithi183 ceph-mon[41562]: pgmap v1004: 142 pgs: 142 active+clean; 582 KiB data, 5.7 GiB used, 709 GiB / 715 GiB avail; 511 B/s rd, 0 op/s; 41 B/s, 1 objects/s recovering 2024-09-10T19:31:57.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:31:56 smithi183 ceph-mon[41562]: from='client.? 172.21.15.183:0/2836884971' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-09-10T19:31:57.183 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:31:56 smithi183 ceph-mon[41562]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-09-10T19:31:57.183 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:31:56 smithi183 ceph-mon[29736]: 5.9 scrub starts 2024-09-10T19:31:57.183 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:31:56 smithi183 ceph-mon[29736]: 5.9 scrub ok 2024-09-10T19:31:57.183 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:31:56 smithi183 ceph-mon[29736]: 5.a scrub starts 2024-09-10T19:31:57.183 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:31:56 smithi183 ceph-mon[29736]: 5.a scrub ok 2024-09-10T19:31:57.183 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:31:56 smithi183 ceph-mon[29736]: 5.f scrub starts 2024-09-10T19:31:57.183 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:31:56 smithi183 ceph-mon[29736]: 5.f scrub ok 2024-09-10T19:31:57.183 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:31:56 smithi183 ceph-mon[29736]: 2.f scrub starts 2024-09-10T19:31:57.183 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:31:56 smithi183 ceph-mon[29736]: 2.f scrub ok 2024-09-10T19:31:57.183 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:31:56 smithi183 ceph-mon[29736]: 5.1f scrub starts 2024-09-10T19:31:57.183 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:31:56 smithi183 ceph-mon[29736]: 5.1f scrub ok 2024-09-10T19:31:57.184 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:31:56 smithi183 ceph-mon[29736]: 5.1e scrub starts 2024-09-10T19:31:57.184 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:31:56 smithi183 ceph-mon[29736]: 5.1e scrub ok 2024-09-10T19:31:57.184 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:31:56 smithi183 ceph-mon[29736]: pgmap v1004: 142 pgs: 142 active+clean; 582 KiB data, 5.7 GiB used, 709 GiB / 715 GiB avail; 511 B/s rd, 0 op/s; 41 B/s, 1 objects/s recovering 2024-09-10T19:31:57.184 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:31:56 smithi183 ceph-mon[29736]: from='client.? 172.21.15.183:0/2836884971' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-09-10T19:31:57.184 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:31:56 smithi183 ceph-mon[29736]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2024-09-10T19:31:57.716 INFO:journalctl@ceph.mgr.x.smithi191.stdout:Sep 10 19:31:57 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-x[34967]: ::ffff:172.21.15.191 - - [10/Sep/2024:19:31:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T19:31:57.931 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:31:57 smithi183 ceph-mon[29736]: 6.e scrub starts 2024-09-10T19:31:57.931 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:31:57 smithi183 ceph-mon[29736]: 6.e scrub ok 2024-09-10T19:31:57.932 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:31:57 smithi183 ceph-mon[29736]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-09-10T19:31:57.932 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:31:57 smithi183 ceph-mon[29736]: osdmap e439: 8 total, 8 up, 8 in 2024-09-10T19:31:57.932 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:31:57 smithi183 ceph-mon[41562]: 6.e scrub starts 2024-09-10T19:31:57.932 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:31:57 smithi183 ceph-mon[41562]: 6.e scrub ok 2024-09-10T19:31:57.932 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:31:57 smithi183 ceph-mon[41562]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-09-10T19:31:57.932 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:31:57 smithi183 ceph-mon[41562]: osdmap e439: 8 total, 8 up, 8 in 2024-09-10T19:31:57.986 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph osd primary-affinity 3 1 2024-09-10T19:31:58.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:31:57 smithi191 ceph-mon[32884]: 6.e scrub starts 2024-09-10T19:31:58.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:31:57 smithi191 ceph-mon[32884]: 6.e scrub ok 2024-09-10T19:31:58.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:31:57 smithi191 ceph-mon[32884]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2024-09-10T19:31:58.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:31:57 smithi191 ceph-mon[32884]: osdmap e439: 8 total, 8 up, 8 in 2024-09-10T19:31:58.489 INFO:teuthology.orchestra.run.smithi183.stdout:ERROR: (22) Invalid argument 2024-09-10T19:31:58.490 INFO:teuthology.orchestra.run.smithi183.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-10T19:31:58.733 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:31:59.181 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:31:58 smithi183 ceph-mon[41562]: 5.b scrub starts 2024-09-10T19:31:59.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:31:58 smithi183 ceph-mon[41562]: 5.b scrub ok 2024-09-10T19:31:59.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:31:58 smithi183 ceph-mon[41562]: 5.16 scrub starts 2024-09-10T19:31:59.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:31:58 smithi183 ceph-mon[41562]: 5.16 scrub ok 2024-09-10T19:31:59.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:31:58 smithi183 ceph-mon[41562]: 4.1a scrub starts 2024-09-10T19:31:59.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:31:58 smithi183 ceph-mon[41562]: 4.1a scrub ok 2024-09-10T19:31:59.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:31:58 smithi183 ceph-mon[41562]: 6.9 scrub starts 2024-09-10T19:31:59.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:31:58 smithi183 ceph-mon[41562]: 6.9 scrub ok 2024-09-10T19:31:59.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:31:58 smithi183 ceph-mon[41562]: pgmap v1006: 142 pgs: 142 active+clean; 582 KiB data, 5.7 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 13 B/s, 0 objects/s recovering 2024-09-10T19:31:59.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:31:58 smithi183 ceph-mon[41562]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:31:59.183 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:31:58 smithi183 ceph-mon[29736]: 5.b scrub starts 2024-09-10T19:31:59.183 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:31:58 smithi183 ceph-mon[29736]: 5.b scrub ok 2024-09-10T19:31:59.183 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:31:58 smithi183 ceph-mon[29736]: 5.16 scrub starts 2024-09-10T19:31:59.183 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:31:58 smithi183 ceph-mon[29736]: 5.16 scrub ok 2024-09-10T19:31:59.183 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:31:58 smithi183 ceph-mon[29736]: 4.1a scrub starts 2024-09-10T19:31:59.183 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:31:58 smithi183 ceph-mon[29736]: 4.1a scrub ok 2024-09-10T19:31:59.183 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:31:58 smithi183 ceph-mon[29736]: 6.9 scrub starts 2024-09-10T19:31:59.183 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:31:58 smithi183 ceph-mon[29736]: 6.9 scrub ok 2024-09-10T19:31:59.183 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:31:58 smithi183 ceph-mon[29736]: pgmap v1006: 142 pgs: 142 active+clean; 582 KiB data, 5.7 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 13 B/s, 0 objects/s recovering 2024-09-10T19:31:59.183 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:31:58 smithi183 ceph-mon[29736]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:31:59.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:31:58 smithi191 ceph-mon[32884]: 5.b scrub starts 2024-09-10T19:31:59.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:31:58 smithi191 ceph-mon[32884]: 5.b scrub ok 2024-09-10T19:31:59.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:31:58 smithi191 ceph-mon[32884]: 5.16 scrub starts 2024-09-10T19:31:59.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:31:58 smithi191 ceph-mon[32884]: 5.16 scrub ok 2024-09-10T19:31:59.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:31:58 smithi191 ceph-mon[32884]: 4.1a scrub starts 2024-09-10T19:31:59.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:31:58 smithi191 ceph-mon[32884]: 4.1a scrub ok 2024-09-10T19:31:59.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:31:58 smithi191 ceph-mon[32884]: 6.9 scrub starts 2024-09-10T19:31:59.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:31:58 smithi191 ceph-mon[32884]: 6.9 scrub ok 2024-09-10T19:31:59.217 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:31:58 smithi191 ceph-mon[32884]: pgmap v1006: 142 pgs: 142 active+clean; 582 KiB data, 5.7 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s; 13 B/s, 0 objects/s recovering 2024-09-10T19:31:59.217 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:31:58 smithi191 ceph-mon[32884]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:31:59.931 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:31:59 smithi183 ceph-mon[29736]: 2.6 deep-scrub starts 2024-09-10T19:31:59.932 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:31:59 smithi183 ceph-mon[29736]: 2.6 deep-scrub ok 2024-09-10T19:31:59.932 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:31:59 smithi183 ceph-mon[29736]: 4.1 scrub starts 2024-09-10T19:31:59.932 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:31:59 smithi183 ceph-mon[29736]: 4.1 scrub ok 2024-09-10T19:31:59.932 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:31:59 smithi183 ceph-mon[41562]: 2.6 deep-scrub starts 2024-09-10T19:31:59.932 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:31:59 smithi183 ceph-mon[41562]: 2.6 deep-scrub ok 2024-09-10T19:31:59.932 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:31:59 smithi183 ceph-mon[41562]: 4.1 scrub starts 2024-09-10T19:31:59.932 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:31:59 smithi183 ceph-mon[41562]: 4.1 scrub ok 2024-09-10T19:32:00.174 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:31:59 smithi191 ceph-mon[32884]: 2.6 deep-scrub starts 2024-09-10T19:32:00.175 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:31:59 smithi191 ceph-mon[32884]: 2.6 deep-scrub ok 2024-09-10T19:32:00.175 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:31:59 smithi191 ceph-mon[32884]: 4.1 scrub starts 2024-09-10T19:32:00.175 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:31:59 smithi191 ceph-mon[32884]: 4.1 scrub ok 2024-09-10T19:32:00.468 INFO:journalctl@ceph.iscsi.iscsi.a.smithi191.stdout:Sep 10 19:32:00 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-iscsi-iscsi-a[61816]: debug there is no tcmu-runner data available 2024-09-10T19:32:01.104 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:32:00 smithi183 ceph-mon[29736]: 6.3 scrub starts 2024-09-10T19:32:01.104 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:32:00 smithi183 ceph-mon[29736]: 6.3 scrub ok 2024-09-10T19:32:01.104 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:32:00 smithi183 ceph-mon[29736]: 6.4 scrub starts 2024-09-10T19:32:01.105 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:32:00 smithi183 ceph-mon[29736]: 6.4 scrub ok 2024-09-10T19:32:01.105 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:32:00 smithi183 ceph-mon[29736]: 5.6 scrub starts 2024-09-10T19:32:01.105 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:32:00 smithi183 ceph-mon[29736]: 5.6 scrub ok 2024-09-10T19:32:01.105 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:32:00 smithi183 ceph-mon[29736]: 2.11 deep-scrub starts 2024-09-10T19:32:01.105 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:32:00 smithi183 ceph-mon[29736]: 2.11 deep-scrub ok 2024-09-10T19:32:01.105 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:32:00 smithi183 ceph-mon[29736]: 6.1e scrub starts 2024-09-10T19:32:01.105 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:32:00 smithi183 ceph-mon[29736]: 6.1e scrub ok 2024-09-10T19:32:01.105 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:32:00 smithi183 ceph-mon[29736]: pgmap v1007: 142 pgs: 142 active+clean; 582 KiB data, 5.7 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:32:01.105 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:32:00 smithi183 ceph-mon[29736]: 6.0 scrub starts 2024-09-10T19:32:01.105 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:32:00 smithi183 ceph-mon[29736]: 6.0 scrub ok 2024-09-10T19:32:01.106 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:32:00 smithi183 ceph-mon[29736]: from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:32:01.106 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:32:00 smithi183 ceph-mon[41562]: 6.3 scrub starts 2024-09-10T19:32:01.106 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:32:00 smithi183 ceph-mon[41562]: 6.3 scrub ok 2024-09-10T19:32:01.106 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:32:00 smithi183 ceph-mon[41562]: 6.4 scrub starts 2024-09-10T19:32:01.106 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:32:00 smithi183 ceph-mon[41562]: 6.4 scrub ok 2024-09-10T19:32:01.106 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:32:00 smithi183 ceph-mon[41562]: 5.6 scrub starts 2024-09-10T19:32:01.106 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:32:00 smithi183 ceph-mon[41562]: 5.6 scrub ok 2024-09-10T19:32:01.106 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:32:00 smithi183 ceph-mon[41562]: 2.11 deep-scrub starts 2024-09-10T19:32:01.106 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:32:00 smithi183 ceph-mon[41562]: 2.11 deep-scrub ok 2024-09-10T19:32:01.107 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:32:00 smithi183 ceph-mon[41562]: 6.1e scrub starts 2024-09-10T19:32:01.107 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:32:00 smithi183 ceph-mon[41562]: 6.1e scrub ok 2024-09-10T19:32:01.107 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:32:00 smithi183 ceph-mon[41562]: pgmap v1007: 142 pgs: 142 active+clean; 582 KiB data, 5.7 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:32:01.107 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:32:00 smithi183 ceph-mon[41562]: 6.0 scrub starts 2024-09-10T19:32:01.107 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:32:00 smithi183 ceph-mon[41562]: 6.0 scrub ok 2024-09-10T19:32:01.107 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:32:00 smithi183 ceph-mon[41562]: from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:32:01.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:32:00 smithi191 ceph-mon[32884]: 6.3 scrub starts 2024-09-10T19:32:01.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:32:00 smithi191 ceph-mon[32884]: 6.3 scrub ok 2024-09-10T19:32:01.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:32:00 smithi191 ceph-mon[32884]: 6.4 scrub starts 2024-09-10T19:32:01.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:32:00 smithi191 ceph-mon[32884]: 6.4 scrub ok 2024-09-10T19:32:01.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:32:00 smithi191 ceph-mon[32884]: 5.6 scrub starts 2024-09-10T19:32:01.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:32:00 smithi191 ceph-mon[32884]: 5.6 scrub ok 2024-09-10T19:32:01.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:32:00 smithi191 ceph-mon[32884]: 2.11 deep-scrub starts 2024-09-10T19:32:01.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:32:00 smithi191 ceph-mon[32884]: 2.11 deep-scrub ok 2024-09-10T19:32:01.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:32:00 smithi191 ceph-mon[32884]: 6.1e scrub starts 2024-09-10T19:32:01.217 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:32:00 smithi191 ceph-mon[32884]: 6.1e scrub ok 2024-09-10T19:32:01.217 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:32:00 smithi191 ceph-mon[32884]: pgmap v1007: 142 pgs: 142 active+clean; 582 KiB data, 5.7 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:32:01.217 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:32:00 smithi191 ceph-mon[32884]: 6.0 scrub starts 2024-09-10T19:32:01.217 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:32:00 smithi191 ceph-mon[32884]: 6.0 scrub ok 2024-09-10T19:32:01.217 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:32:00 smithi191 ceph-mon[32884]: from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:32:01.373 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Sep 10 19:32:01 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-y[29989]: ::ffff:172.21.15.191 - - [10/Sep/2024:19:32:01] "GET /metrics HTTP/1.1" 200 222448 "" "Prometheus/2.43.0" 2024-09-10T19:32:02.014 INFO:teuthology.orchestra.run.smithi183.stderr:set osd.3 primary-affinity to 1 (8655362) 2024-09-10T19:32:02.181 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:32:01 smithi183 ceph-mon[41562]: 2.1 scrub starts 2024-09-10T19:32:02.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:32:01 smithi183 ceph-mon[41562]: 1.8 deep-scrub starts 2024-09-10T19:32:02.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:32:01 smithi183 ceph-mon[41562]: 2.1 scrub ok 2024-09-10T19:32:02.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:32:01 smithi183 ceph-mon[41562]: 1.8 deep-scrub ok 2024-09-10T19:32:02.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:32:01 smithi183 ceph-mon[41562]: 4.a scrub starts 2024-09-10T19:32:02.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:32:01 smithi183 ceph-mon[41562]: 4.a scrub ok 2024-09-10T19:32:02.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:32:01 smithi183 ceph-mon[41562]: 5.1c scrub starts 2024-09-10T19:32:02.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:32:01 smithi183 ceph-mon[41562]: 5.1c scrub ok 2024-09-10T19:32:02.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:32:01 smithi183 ceph-mon[41562]: 5.5 scrub starts 2024-09-10T19:32:02.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:32:01 smithi183 ceph-mon[41562]: 5.5 scrub ok 2024-09-10T19:32:02.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:32:01 smithi183 ceph-mon[41562]: from='client.? 172.21.15.183:0/2412621756' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]: dispatch 2024-09-10T19:32:02.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:32:01 smithi183 ceph-mon[41562]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]: dispatch 2024-09-10T19:32:02.183 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:32:01 smithi183 ceph-mon[29736]: 2.1 scrub starts 2024-09-10T19:32:02.183 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:32:01 smithi183 ceph-mon[29736]: 1.8 deep-scrub starts 2024-09-10T19:32:02.183 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:32:01 smithi183 ceph-mon[29736]: 2.1 scrub ok 2024-09-10T19:32:02.183 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:32:01 smithi183 ceph-mon[29736]: 1.8 deep-scrub ok 2024-09-10T19:32:02.183 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:32:01 smithi183 ceph-mon[29736]: 4.a scrub starts 2024-09-10T19:32:02.183 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:32:01 smithi183 ceph-mon[29736]: 4.a scrub ok 2024-09-10T19:32:02.183 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:32:01 smithi183 ceph-mon[29736]: 5.1c scrub starts 2024-09-10T19:32:02.183 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:32:01 smithi183 ceph-mon[29736]: 5.1c scrub ok 2024-09-10T19:32:02.183 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:32:01 smithi183 ceph-mon[29736]: 5.5 scrub starts 2024-09-10T19:32:02.183 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:32:01 smithi183 ceph-mon[29736]: 5.5 scrub ok 2024-09-10T19:32:02.183 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:32:01 smithi183 ceph-mon[29736]: from='client.? 172.21.15.183:0/2412621756' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]: dispatch 2024-09-10T19:32:02.183 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:32:01 smithi183 ceph-mon[29736]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]: dispatch 2024-09-10T19:32:02.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:32:01 smithi191 ceph-mon[32884]: 2.1 scrub starts 2024-09-10T19:32:02.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:32:01 smithi191 ceph-mon[32884]: 1.8 deep-scrub starts 2024-09-10T19:32:02.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:32:01 smithi191 ceph-mon[32884]: 2.1 scrub ok 2024-09-10T19:32:02.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:32:01 smithi191 ceph-mon[32884]: 1.8 deep-scrub ok 2024-09-10T19:32:02.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:32:01 smithi191 ceph-mon[32884]: 4.a scrub starts 2024-09-10T19:32:02.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:32:01 smithi191 ceph-mon[32884]: 4.a scrub ok 2024-09-10T19:32:02.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:32:01 smithi191 ceph-mon[32884]: 5.1c scrub starts 2024-09-10T19:32:02.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:32:01 smithi191 ceph-mon[32884]: 5.1c scrub ok 2024-09-10T19:32:02.217 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:32:01 smithi191 ceph-mon[32884]: 5.5 scrub starts 2024-09-10T19:32:02.217 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:32:01 smithi191 ceph-mon[32884]: 5.5 scrub ok 2024-09-10T19:32:02.217 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:32:01 smithi191 ceph-mon[32884]: from='client.? 172.21.15.183:0/2412621756' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]: dispatch 2024-09-10T19:32:02.217 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:32:01 smithi191 ceph-mon[32884]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]: dispatch 2024-09-10T19:32:02.840 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph osd reweight 4 1 2024-09-10T19:32:03.090 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:32:03.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:32:02 smithi183 ceph-mon[41562]: 2.3 scrub starts 2024-09-10T19:32:03.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:32:02 smithi183 ceph-mon[41562]: 2.3 scrub ok 2024-09-10T19:32:03.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:32:02 smithi183 ceph-mon[41562]: 2.14 scrub starts 2024-09-10T19:32:03.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:32:02 smithi183 ceph-mon[41562]: 2.14 scrub ok 2024-09-10T19:32:03.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:32:02 smithi183 ceph-mon[41562]: pgmap v1008: 142 pgs: 142 active+clean; 582 KiB data, 5.7 GiB used, 709 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-10T19:32:03.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:32:02 smithi183 ceph-mon[41562]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]': finished 2024-09-10T19:32:03.183 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:32:02 smithi183 ceph-mon[41562]: osdmap e440: 8 total, 8 up, 8 in 2024-09-10T19:32:03.183 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:32:02 smithi183 ceph-mon[29736]: 2.3 scrub starts 2024-09-10T19:32:03.183 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:32:02 smithi183 ceph-mon[29736]: 2.3 scrub ok 2024-09-10T19:32:03.183 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:32:02 smithi183 ceph-mon[29736]: 2.14 scrub starts 2024-09-10T19:32:03.183 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:32:02 smithi183 ceph-mon[29736]: 2.14 scrub ok 2024-09-10T19:32:03.183 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:32:02 smithi183 ceph-mon[29736]: pgmap v1008: 142 pgs: 142 active+clean; 582 KiB data, 5.7 GiB used, 709 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-10T19:32:03.183 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:32:02 smithi183 ceph-mon[29736]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]': finished 2024-09-10T19:32:03.183 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:32:02 smithi183 ceph-mon[29736]: osdmap e440: 8 total, 8 up, 8 in 2024-09-10T19:32:03.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:32:02 smithi191 ceph-mon[32884]: 2.3 scrub starts 2024-09-10T19:32:03.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:32:02 smithi191 ceph-mon[32884]: 2.3 scrub ok 2024-09-10T19:32:03.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:32:02 smithi191 ceph-mon[32884]: 2.14 scrub starts 2024-09-10T19:32:03.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:32:02 smithi191 ceph-mon[32884]: 2.14 scrub ok 2024-09-10T19:32:03.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:32:02 smithi191 ceph-mon[32884]: pgmap v1008: 142 pgs: 142 active+clean; 582 KiB data, 5.7 GiB used, 709 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-10T19:32:03.217 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:32:02 smithi191 ceph-mon[32884]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]': finished 2024-09-10T19:32:03.217 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:32:02 smithi191 ceph-mon[32884]: osdmap e440: 8 total, 8 up, 8 in 2024-09-10T19:32:04.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:32:03 smithi183 ceph-mon[29736]: 6.1d scrub starts 2024-09-10T19:32:04.182 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:32:03 smithi183 ceph-mon[29736]: 6.1d scrub ok 2024-09-10T19:32:04.182 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:32:03 smithi183 ceph-mon[29736]: 6.7 scrub starts 2024-09-10T19:32:04.182 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:32:03 smithi183 ceph-mon[29736]: 6.7 scrub ok 2024-09-10T19:32:04.182 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:32:03 smithi183 ceph-mon[29736]: 4.10 scrub starts 2024-09-10T19:32:04.182 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:32:03 smithi183 ceph-mon[29736]: 4.10 scrub ok 2024-09-10T19:32:04.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:32:03 smithi183 ceph-mon[41562]: 6.1d scrub starts 2024-09-10T19:32:04.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:32:03 smithi183 ceph-mon[41562]: 6.1d scrub ok 2024-09-10T19:32:04.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:32:03 smithi183 ceph-mon[41562]: 6.7 scrub starts 2024-09-10T19:32:04.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:32:03 smithi183 ceph-mon[41562]: 6.7 scrub ok 2024-09-10T19:32:04.183 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:32:03 smithi183 ceph-mon[41562]: 4.10 scrub starts 2024-09-10T19:32:04.183 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:32:03 smithi183 ceph-mon[41562]: 4.10 scrub ok 2024-09-10T19:32:04.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:32:03 smithi191 ceph-mon[32884]: 6.1d scrub starts 2024-09-10T19:32:04.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:32:03 smithi191 ceph-mon[32884]: 6.1d scrub ok 2024-09-10T19:32:04.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:32:03 smithi191 ceph-mon[32884]: 6.7 scrub starts 2024-09-10T19:32:04.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:32:03 smithi191 ceph-mon[32884]: 6.7 scrub ok 2024-09-10T19:32:04.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:32:03 smithi191 ceph-mon[32884]: 4.10 scrub starts 2024-09-10T19:32:04.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:32:03 smithi191 ceph-mon[32884]: 4.10 scrub ok 2024-09-10T19:32:05.185 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:32:04 smithi183 ceph-mon[29736]: pgmap v1010: 142 pgs: 142 active+clean; 582 KiB data, 5.7 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:32:05.186 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:32:04 smithi183 ceph-mon[41562]: pgmap v1010: 142 pgs: 142 active+clean; 582 KiB data, 5.7 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:32:05.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:32:04 smithi191 ceph-mon[32884]: pgmap v1010: 142 pgs: 142 active+clean; 582 KiB data, 5.7 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:32:06.018 INFO:teuthology.orchestra.run.smithi183.stderr:reweighted osd.4 to 1 (10000) 2024-09-10T19:32:06.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:32:05 smithi183 ceph-mon[29736]: 4.12 scrub starts 2024-09-10T19:32:06.182 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:32:05 smithi183 ceph-mon[29736]: 4.12 scrub ok 2024-09-10T19:32:06.182 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:32:05 smithi183 ceph-mon[29736]: from='client.? 172.21.15.183:0/1768663682' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-09-10T19:32:06.182 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:32:05 smithi183 ceph-mon[29736]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-09-10T19:32:06.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:32:05 smithi183 ceph-mon[41562]: 4.12 scrub starts 2024-09-10T19:32:06.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:32:05 smithi183 ceph-mon[41562]: 4.12 scrub ok 2024-09-10T19:32:06.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:32:05 smithi183 ceph-mon[41562]: from='client.? 172.21.15.183:0/1768663682' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-09-10T19:32:06.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:32:05 smithi183 ceph-mon[41562]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-09-10T19:32:06.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:32:05 smithi191 ceph-mon[32884]: 4.12 scrub starts 2024-09-10T19:32:06.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:32:05 smithi191 ceph-mon[32884]: 4.12 scrub ok 2024-09-10T19:32:06.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:32:05 smithi191 ceph-mon[32884]: from='client.? 172.21.15.183:0/1768663682' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-09-10T19:32:06.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:32:05 smithi191 ceph-mon[32884]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-09-10T19:32:06.955 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph osd primary-affinity 4 1 2024-09-10T19:32:07.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:32:06 smithi183 ceph-mon[29736]: 6.1c scrub starts 2024-09-10T19:32:07.182 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:32:06 smithi183 ceph-mon[29736]: 6.1c scrub ok 2024-09-10T19:32:07.182 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:32:06 smithi183 ceph-mon[29736]: 1.3 scrub starts 2024-09-10T19:32:07.182 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:32:06 smithi183 ceph-mon[29736]: 1.3 scrub ok 2024-09-10T19:32:07.182 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:32:06 smithi183 ceph-mon[29736]: 4.19 deep-scrub starts 2024-09-10T19:32:07.182 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:32:06 smithi183 ceph-mon[29736]: 4.19 deep-scrub ok 2024-09-10T19:32:07.183 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:32:06 smithi183 ceph-mon[29736]: pgmap v1011: 142 pgs: 142 active+clean; 582 KiB data, 5.7 GiB used, 709 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-10T19:32:07.183 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:32:06 smithi183 ceph-mon[29736]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-09-10T19:32:07.183 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:32:06 smithi183 ceph-mon[29736]: osdmap e441: 8 total, 8 up, 8 in 2024-09-10T19:32:07.183 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:32:06 smithi183 ceph-mon[41562]: 6.1c scrub starts 2024-09-10T19:32:07.183 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:32:06 smithi183 ceph-mon[41562]: 6.1c scrub ok 2024-09-10T19:32:07.183 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:32:06 smithi183 ceph-mon[41562]: 1.3 scrub starts 2024-09-10T19:32:07.184 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:32:06 smithi183 ceph-mon[41562]: 1.3 scrub ok 2024-09-10T19:32:07.184 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:32:06 smithi183 ceph-mon[41562]: 4.19 deep-scrub starts 2024-09-10T19:32:07.184 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:32:06 smithi183 ceph-mon[41562]: 4.19 deep-scrub ok 2024-09-10T19:32:07.184 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:32:06 smithi183 ceph-mon[41562]: pgmap v1011: 142 pgs: 142 active+clean; 582 KiB data, 5.7 GiB used, 709 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-10T19:32:07.184 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:32:06 smithi183 ceph-mon[41562]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-09-10T19:32:07.184 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:32:06 smithi183 ceph-mon[41562]: osdmap e441: 8 total, 8 up, 8 in 2024-09-10T19:32:07.205 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:32:07.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:32:06 smithi191 ceph-mon[32884]: 6.1c scrub starts 2024-09-10T19:32:07.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:32:06 smithi191 ceph-mon[32884]: 6.1c scrub ok 2024-09-10T19:32:07.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:32:06 smithi191 ceph-mon[32884]: 1.3 scrub starts 2024-09-10T19:32:07.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:32:06 smithi191 ceph-mon[32884]: 1.3 scrub ok 2024-09-10T19:32:07.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:32:06 smithi191 ceph-mon[32884]: 4.19 deep-scrub starts 2024-09-10T19:32:07.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:32:06 smithi191 ceph-mon[32884]: 4.19 deep-scrub ok 2024-09-10T19:32:07.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:32:06 smithi191 ceph-mon[32884]: pgmap v1011: 142 pgs: 142 active+clean; 582 KiB data, 5.7 GiB used, 709 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-10T19:32:07.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:32:06 smithi191 ceph-mon[32884]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-09-10T19:32:07.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:32:06 smithi191 ceph-mon[32884]: osdmap e441: 8 total, 8 up, 8 in 2024-09-10T19:32:07.716 INFO:journalctl@ceph.mgr.x.smithi191.stdout:Sep 10 19:32:07 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-x[34967]: ::ffff:172.21.15.191 - - [10/Sep/2024:19:32:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T19:32:07.931 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:32:07 smithi183 ceph-mon[41562]: 2.5 scrub starts 2024-09-10T19:32:07.932 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:32:07 smithi183 ceph-mon[41562]: 2.5 scrub ok 2024-09-10T19:32:07.932 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:32:07 smithi183 ceph-mon[41562]: 5.7 scrub starts 2024-09-10T19:32:07.932 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:32:07 smithi183 ceph-mon[41562]: 5.7 scrub ok 2024-09-10T19:32:07.932 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:32:07 smithi183 ceph-mon[29736]: 2.5 scrub starts 2024-09-10T19:32:07.932 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:32:07 smithi183 ceph-mon[29736]: 2.5 scrub ok 2024-09-10T19:32:07.932 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:32:07 smithi183 ceph-mon[29736]: 5.7 scrub starts 2024-09-10T19:32:07.932 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:32:07 smithi183 ceph-mon[29736]: 5.7 scrub ok 2024-09-10T19:32:08.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:32:07 smithi191 ceph-mon[32884]: 2.5 scrub starts 2024-09-10T19:32:08.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:32:07 smithi191 ceph-mon[32884]: 2.5 scrub ok 2024-09-10T19:32:08.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:32:07 smithi191 ceph-mon[32884]: 5.7 scrub starts 2024-09-10T19:32:08.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:32:07 smithi191 ceph-mon[32884]: 5.7 scrub ok 2024-09-10T19:32:09.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:32:08 smithi183 ceph-mon[29736]: pgmap v1013: 142 pgs: 142 active+clean; 582 KiB data, 5.7 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:32:09.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:32:08 smithi183 ceph-mon[41562]: pgmap v1013: 142 pgs: 142 active+clean; 582 KiB data, 5.7 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:32:09.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:32:08 smithi191 ceph-mon[32884]: pgmap v1013: 142 pgs: 142 active+clean; 582 KiB data, 5.7 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:32:09.894 INFO:teuthology.orchestra.run.smithi183.stderr:set osd.4 primary-affinity to 1 (8655362) 2024-09-10T19:32:10.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:32:09 smithi183 ceph-mon[29736]: 5.1a scrub starts 2024-09-10T19:32:10.182 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:32:09 smithi183 ceph-mon[29736]: 5.1a scrub ok 2024-09-10T19:32:10.182 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:32:09 smithi183 ceph-mon[29736]: 5.18 scrub starts 2024-09-10T19:32:10.182 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:32:09 smithi183 ceph-mon[29736]: 5.18 scrub ok 2024-09-10T19:32:10.182 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:32:09 smithi183 ceph-mon[29736]: from='client.? 172.21.15.183:0/123710938' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]: dispatch 2024-09-10T19:32:10.182 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:32:09 smithi183 ceph-mon[29736]: 4.4 scrub starts 2024-09-10T19:32:10.182 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:32:09 smithi183 ceph-mon[29736]: 4.4 scrub ok 2024-09-10T19:32:10.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:32:09 smithi183 ceph-mon[41562]: 5.1a scrub starts 2024-09-10T19:32:10.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:32:09 smithi183 ceph-mon[41562]: 5.1a scrub ok 2024-09-10T19:32:10.183 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:32:09 smithi183 ceph-mon[41562]: 5.18 scrub starts 2024-09-10T19:32:10.183 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:32:09 smithi183 ceph-mon[41562]: 5.18 scrub ok 2024-09-10T19:32:10.183 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:32:09 smithi183 ceph-mon[41562]: from='client.? 172.21.15.183:0/123710938' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]: dispatch 2024-09-10T19:32:10.183 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:32:09 smithi183 ceph-mon[41562]: 4.4 scrub starts 2024-09-10T19:32:10.183 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:32:09 smithi183 ceph-mon[41562]: 4.4 scrub ok 2024-09-10T19:32:10.185 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:32:09 smithi191 ceph-mon[32884]: 5.1a scrub starts 2024-09-10T19:32:10.186 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:32:09 smithi191 ceph-mon[32884]: 5.1a scrub ok 2024-09-10T19:32:10.186 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:32:09 smithi191 ceph-mon[32884]: 5.18 scrub starts 2024-09-10T19:32:10.186 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:32:09 smithi191 ceph-mon[32884]: 5.18 scrub ok 2024-09-10T19:32:10.186 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:32:09 smithi191 ceph-mon[32884]: from='client.? 172.21.15.183:0/123710938' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]: dispatch 2024-09-10T19:32:10.186 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:32:09 smithi191 ceph-mon[32884]: 4.4 scrub starts 2024-09-10T19:32:10.186 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:32:09 smithi191 ceph-mon[32884]: 4.4 scrub ok 2024-09-10T19:32:10.466 INFO:journalctl@ceph.iscsi.iscsi.a.smithi191.stdout:Sep 10 19:32:10 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-iscsi-iscsi-a[61816]: debug there is no tcmu-runner data available 2024-09-10T19:32:10.693 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph osd reweight 5 1 2024-09-10T19:32:10.931 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:32:10 smithi183 ceph-mon[41562]: 2.2 deep-scrub starts 2024-09-10T19:32:10.931 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:32:10 smithi183 ceph-mon[41562]: 2.2 deep-scrub ok 2024-09-10T19:32:10.932 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:32:10 smithi183 ceph-mon[41562]: from='client.? 172.21.15.183:0/123710938' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]': finished 2024-09-10T19:32:10.932 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:32:10 smithi183 ceph-mon[41562]: osdmap e442: 8 total, 8 up, 8 in 2024-09-10T19:32:10.932 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:32:10 smithi183 ceph-mon[41562]: pgmap v1015: 142 pgs: 142 active+clean; 582 KiB data, 5.7 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:32:10.932 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:32:10 smithi183 ceph-mon[41562]: 3.1 scrub starts 2024-09-10T19:32:10.932 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:32:10 smithi183 ceph-mon[41562]: 3.1 scrub ok 2024-09-10T19:32:10.932 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:32:10 smithi183 ceph-mon[41562]: from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:32:10.932 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:32:10 smithi183 ceph-mon[29736]: 2.2 deep-scrub starts 2024-09-10T19:32:10.932 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:32:10 smithi183 ceph-mon[29736]: 2.2 deep-scrub ok 2024-09-10T19:32:10.932 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:32:10 smithi183 ceph-mon[29736]: from='client.? 172.21.15.183:0/123710938' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]': finished 2024-09-10T19:32:10.932 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:32:10 smithi183 ceph-mon[29736]: osdmap e442: 8 total, 8 up, 8 in 2024-09-10T19:32:10.933 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:32:10 smithi183 ceph-mon[29736]: pgmap v1015: 142 pgs: 142 active+clean; 582 KiB data, 5.7 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:32:10.933 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:32:10 smithi183 ceph-mon[29736]: 3.1 scrub starts 2024-09-10T19:32:10.933 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:32:10 smithi183 ceph-mon[29736]: 3.1 scrub ok 2024-09-10T19:32:10.933 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:32:10 smithi183 ceph-mon[29736]: from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:32:10.945 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:32:11.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:32:10 smithi191 ceph-mon[32884]: 2.2 deep-scrub starts 2024-09-10T19:32:11.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:32:10 smithi191 ceph-mon[32884]: 2.2 deep-scrub ok 2024-09-10T19:32:11.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:32:10 smithi191 ceph-mon[32884]: from='client.? 172.21.15.183:0/123710938' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 4, "weight": 1.0}]': finished 2024-09-10T19:32:11.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:32:10 smithi191 ceph-mon[32884]: osdmap e442: 8 total, 8 up, 8 in 2024-09-10T19:32:11.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:32:10 smithi191 ceph-mon[32884]: pgmap v1015: 142 pgs: 142 active+clean; 582 KiB data, 5.7 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:32:11.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:32:10 smithi191 ceph-mon[32884]: 3.1 scrub starts 2024-09-10T19:32:11.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:32:10 smithi191 ceph-mon[32884]: 3.1 scrub ok 2024-09-10T19:32:11.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:32:10 smithi191 ceph-mon[32884]: from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:32:11.317 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Sep 10 19:32:11 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-y[29989]: ::ffff:172.21.15.191 - - [10/Sep/2024:19:32:11] "GET /metrics HTTP/1.1" 200 222448 "" "Prometheus/2.43.0" 2024-09-10T19:32:12.141 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:32:11 smithi183 ceph-mon[29736]: 2.12 scrub starts 2024-09-10T19:32:12.141 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:32:11 smithi183 ceph-mon[29736]: 2.12 scrub ok 2024-09-10T19:32:12.141 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:32:11 smithi183 ceph-mon[41562]: 2.12 scrub starts 2024-09-10T19:32:12.141 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:32:11 smithi183 ceph-mon[41562]: 2.12 scrub ok 2024-09-10T19:32:12.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:32:11 smithi191 ceph-mon[32884]: 2.12 scrub starts 2024-09-10T19:32:12.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:32:11 smithi191 ceph-mon[32884]: 2.12 scrub ok 2024-09-10T19:32:12.910 INFO:teuthology.orchestra.run.smithi183.stderr:reweighted osd.5 to 1 (10000) 2024-09-10T19:32:13.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:32:12 smithi183 ceph-mon[29736]: pgmap v1016: 142 pgs: 142 active+clean; 582 KiB data, 5.7 GiB used, 709 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-10T19:32:13.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:32:12 smithi183 ceph-mon[29736]: from='client.? 172.21.15.183:0/17709104' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-09-10T19:32:13.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:32:12 smithi183 ceph-mon[41562]: pgmap v1016: 142 pgs: 142 active+clean; 582 KiB data, 5.7 GiB used, 709 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-10T19:32:13.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:32:12 smithi183 ceph-mon[41562]: from='client.? 172.21.15.183:0/17709104' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-09-10T19:32:13.215 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:32:12 smithi191 ceph-mon[32884]: pgmap v1016: 142 pgs: 142 active+clean; 582 KiB data, 5.7 GiB used, 709 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-10T19:32:13.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:32:12 smithi191 ceph-mon[32884]: from='client.? 172.21.15.183:0/17709104' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-09-10T19:32:13.760 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph osd primary-affinity 5 1 2024-09-10T19:32:14.012 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:32:14.131 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:32:13 smithi183 ceph-mon[29736]: 6.2 scrub starts 2024-09-10T19:32:14.132 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:32:13 smithi183 ceph-mon[29736]: 6.2 scrub ok 2024-09-10T19:32:14.132 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:32:13 smithi183 ceph-mon[29736]: from='client.? 172.21.15.183:0/17709104' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-09-10T19:32:14.132 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:32:13 smithi183 ceph-mon[29736]: osdmap e443: 8 total, 8 up, 8 in 2024-09-10T19:32:14.132 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:32:13 smithi183 ceph-mon[29736]: 6.18 scrub starts 2024-09-10T19:32:14.132 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:32:13 smithi183 ceph-mon[29736]: 6.18 scrub ok 2024-09-10T19:32:14.132 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:32:13 smithi183 ceph-mon[29736]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:32:14.132 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:32:13 smithi183 ceph-mon[41562]: 6.2 scrub starts 2024-09-10T19:32:14.132 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:32:13 smithi183 ceph-mon[41562]: 6.2 scrub ok 2024-09-10T19:32:14.132 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:32:13 smithi183 ceph-mon[41562]: from='client.? 172.21.15.183:0/17709104' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-09-10T19:32:14.133 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:32:13 smithi183 ceph-mon[41562]: osdmap e443: 8 total, 8 up, 8 in 2024-09-10T19:32:14.133 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:32:13 smithi183 ceph-mon[41562]: 6.18 scrub starts 2024-09-10T19:32:14.133 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:32:13 smithi183 ceph-mon[41562]: 6.18 scrub ok 2024-09-10T19:32:14.133 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:32:13 smithi183 ceph-mon[41562]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:32:14.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:32:13 smithi191 ceph-mon[32884]: 6.2 scrub starts 2024-09-10T19:32:14.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:32:13 smithi191 ceph-mon[32884]: 6.2 scrub ok 2024-09-10T19:32:14.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:32:13 smithi191 ceph-mon[32884]: from='client.? 172.21.15.183:0/17709104' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-09-10T19:32:14.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:32:13 smithi191 ceph-mon[32884]: osdmap e443: 8 total, 8 up, 8 in 2024-09-10T19:32:14.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:32:13 smithi191 ceph-mon[32884]: 6.18 scrub starts 2024-09-10T19:32:14.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:32:13 smithi191 ceph-mon[32884]: 6.18 scrub ok 2024-09-10T19:32:14.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:32:13 smithi191 ceph-mon[32884]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:32:15.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:32:14 smithi183 ceph-mon[41562]: 6.b scrub starts 2024-09-10T19:32:15.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:32:14 smithi183 ceph-mon[41562]: 6.b scrub ok 2024-09-10T19:32:15.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:32:14 smithi183 ceph-mon[41562]: purged_snaps scrub starts 2024-09-10T19:32:15.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:32:14 smithi183 ceph-mon[41562]: purged_snaps scrub ok 2024-09-10T19:32:15.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:32:14 smithi183 ceph-mon[41562]: osdmap e444: 8 total, 8 up, 8 in 2024-09-10T19:32:15.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:32:14 smithi183 ceph-mon[41562]: pgmap v1019: 142 pgs: 142 active+clean; 582 KiB data, 5.7 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-10T19:32:15.182 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:32:14 smithi183 ceph-mon[29736]: 6.b scrub starts 2024-09-10T19:32:15.182 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:32:14 smithi183 ceph-mon[29736]: 6.b scrub ok 2024-09-10T19:32:15.182 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:32:14 smithi183 ceph-mon[29736]: purged_snaps scrub starts 2024-09-10T19:32:15.183 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:32:14 smithi183 ceph-mon[29736]: purged_snaps scrub ok 2024-09-10T19:32:15.183 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:32:14 smithi183 ceph-mon[29736]: osdmap e444: 8 total, 8 up, 8 in 2024-09-10T19:32:15.183 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:32:14 smithi183 ceph-mon[29736]: pgmap v1019: 142 pgs: 142 active+clean; 582 KiB data, 5.7 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-10T19:32:15.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:32:14 smithi191 ceph-mon[32884]: 6.b scrub starts 2024-09-10T19:32:15.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:32:14 smithi191 ceph-mon[32884]: 6.b scrub ok 2024-09-10T19:32:15.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:32:14 smithi191 ceph-mon[32884]: purged_snaps scrub starts 2024-09-10T19:32:15.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:32:14 smithi191 ceph-mon[32884]: purged_snaps scrub ok 2024-09-10T19:32:15.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:32:14 smithi191 ceph-mon[32884]: osdmap e444: 8 total, 8 up, 8 in 2024-09-10T19:32:15.216 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:32:14 smithi191 ceph-mon[32884]: pgmap v1019: 142 pgs: 142 active+clean; 582 KiB data, 5.7 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-10T19:32:16.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:32:16 smithi183 ceph-mon[29736]: 6.d scrub starts 2024-09-10T19:32:16.182 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:32:16 smithi183 ceph-mon[29736]: 6.d scrub ok 2024-09-10T19:32:16.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:32:16 smithi183 ceph-mon[41562]: 6.d scrub starts 2024-09-10T19:32:16.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:32:16 smithi183 ceph-mon[41562]: 6.d scrub ok 2024-09-10T19:32:16.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:32:16 smithi191 ceph-mon[32884]: 6.d scrub starts 2024-09-10T19:32:16.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:32:16 smithi191 ceph-mon[32884]: 6.d scrub ok 2024-09-10T19:32:17.022 INFO:teuthology.orchestra.run.smithi183.stderr:set osd.5 primary-affinity to 1 (8655362) 2024-09-10T19:32:17.330 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:32:17 smithi183 ceph-mon[29736]: purged_snaps scrub starts 2024-09-10T19:32:17.330 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:32:17 smithi183 ceph-mon[29736]: purged_snaps scrub ok 2024-09-10T19:32:17.330 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:32:17 smithi183 ceph-mon[29736]: pgmap v1020: 142 pgs: 142 active+clean; 582 KiB data, 5.7 GiB used, 709 GiB / 715 GiB avail; 836 B/s rd, 0 op/s 2024-09-10T19:32:17.330 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:32:17 smithi183 ceph-mon[29736]: osdmap e445: 8 total, 8 up, 8 in 2024-09-10T19:32:17.331 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:32:17 smithi183 ceph-mon[29736]: from='client.? 172.21.15.183:0/2488021777' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]: dispatch 2024-09-10T19:32:17.331 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:32:17 smithi183 ceph-mon[29736]: 2.a scrub starts 2024-09-10T19:32:17.331 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:32:17 smithi183 ceph-mon[29736]: 2.a scrub ok 2024-09-10T19:32:17.331 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:32:17 smithi183 ceph-mon[29736]: from='client.? 172.21.15.183:0/2488021777' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]': finished 2024-09-10T19:32:17.331 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:32:17 smithi183 ceph-mon[29736]: osdmap e446: 8 total, 8 up, 8 in 2024-09-10T19:32:17.331 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:32:17 smithi183 ceph-mon[41562]: purged_snaps scrub starts 2024-09-10T19:32:17.331 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:32:17 smithi183 ceph-mon[41562]: purged_snaps scrub ok 2024-09-10T19:32:17.331 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:32:17 smithi183 ceph-mon[41562]: pgmap v1020: 142 pgs: 142 active+clean; 582 KiB data, 5.7 GiB used, 709 GiB / 715 GiB avail; 836 B/s rd, 0 op/s 2024-09-10T19:32:17.331 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:32:17 smithi183 ceph-mon[41562]: osdmap e445: 8 total, 8 up, 8 in 2024-09-10T19:32:17.331 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:32:17 smithi183 ceph-mon[41562]: from='client.? 172.21.15.183:0/2488021777' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]: dispatch 2024-09-10T19:32:17.332 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:32:17 smithi183 ceph-mon[41562]: 2.a scrub starts 2024-09-10T19:32:17.332 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:32:17 smithi183 ceph-mon[41562]: 2.a scrub ok 2024-09-10T19:32:17.332 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:32:17 smithi183 ceph-mon[41562]: from='client.? 172.21.15.183:0/2488021777' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]': finished 2024-09-10T19:32:17.332 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:32:17 smithi183 ceph-mon[41562]: osdmap e446: 8 total, 8 up, 8 in 2024-09-10T19:32:17.451 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:32:17 smithi191 ceph-mon[32884]: purged_snaps scrub starts 2024-09-10T19:32:17.451 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:32:17 smithi191 ceph-mon[32884]: purged_snaps scrub ok 2024-09-10T19:32:17.451 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:32:17 smithi191 ceph-mon[32884]: pgmap v1020: 142 pgs: 142 active+clean; 582 KiB data, 5.7 GiB used, 709 GiB / 715 GiB avail; 836 B/s rd, 0 op/s 2024-09-10T19:32:17.451 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:32:17 smithi191 ceph-mon[32884]: osdmap e445: 8 total, 8 up, 8 in 2024-09-10T19:32:17.451 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:32:17 smithi191 ceph-mon[32884]: from='client.? 172.21.15.183:0/2488021777' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]: dispatch 2024-09-10T19:32:17.451 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:32:17 smithi191 ceph-mon[32884]: 2.a scrub starts 2024-09-10T19:32:17.451 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:32:17 smithi191 ceph-mon[32884]: 2.a scrub ok 2024-09-10T19:32:17.452 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:32:17 smithi191 ceph-mon[32884]: from='client.? 172.21.15.183:0/2488021777' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 5, "weight": 1.0}]': finished 2024-09-10T19:32:17.452 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:32:17 smithi191 ceph-mon[32884]: osdmap e446: 8 total, 8 up, 8 in 2024-09-10T19:32:17.695 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph osd reweight 6 1 2024-09-10T19:32:17.715 INFO:journalctl@ceph.mgr.x.smithi191.stdout:Sep 10 19:32:17 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-x[34967]: ::ffff:172.21.15.191 - - [10/Sep/2024:19:32:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T19:32:17.944 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:32:19.354 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:32:19 smithi183 ceph-mon[29736]: 3.2 deep-scrub starts 2024-09-10T19:32:19.354 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:32:19 smithi183 ceph-mon[29736]: 3.2 deep-scrub ok 2024-09-10T19:32:19.354 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:32:19 smithi183 ceph-mon[29736]: pgmap v1023: 142 pgs: 142 active+clean; 582 KiB data, 5.7 GiB used, 709 GiB / 715 GiB avail; 1003 B/s rd, 0 op/s 2024-09-10T19:32:19.354 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:32:19 smithi183 ceph-mon[29736]: 4.1e deep-scrub starts 2024-09-10T19:32:19.354 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:32:19 smithi183 ceph-mon[29736]: 4.1e deep-scrub ok 2024-09-10T19:32:19.355 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:32:19 smithi183 ceph-mon[41562]: 3.2 deep-scrub starts 2024-09-10T19:32:19.355 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:32:19 smithi183 ceph-mon[41562]: 3.2 deep-scrub ok 2024-09-10T19:32:19.355 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:32:19 smithi183 ceph-mon[41562]: pgmap v1023: 142 pgs: 142 active+clean; 582 KiB data, 5.7 GiB used, 709 GiB / 715 GiB avail; 1003 B/s rd, 0 op/s 2024-09-10T19:32:19.355 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:32:19 smithi183 ceph-mon[41562]: 4.1e deep-scrub starts 2024-09-10T19:32:19.355 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:32:19 smithi183 ceph-mon[41562]: 4.1e deep-scrub ok 2024-09-10T19:32:19.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:32:19 smithi191 ceph-mon[32884]: 3.2 deep-scrub starts 2024-09-10T19:32:19.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:32:19 smithi191 ceph-mon[32884]: 3.2 deep-scrub ok 2024-09-10T19:32:19.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:32:19 smithi191 ceph-mon[32884]: pgmap v1023: 142 pgs: 142 active+clean; 582 KiB data, 5.7 GiB used, 709 GiB / 715 GiB avail; 1003 B/s rd, 0 op/s 2024-09-10T19:32:19.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:32:19 smithi191 ceph-mon[32884]: 4.1e deep-scrub starts 2024-09-10T19:32:19.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:32:19 smithi191 ceph-mon[32884]: 4.1e deep-scrub ok 2024-09-10T19:32:20.036 INFO:teuthology.orchestra.run.smithi183.stderr:reweighted osd.6 to 1 (10000) 2024-09-10T19:32:20.386 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:32:20 smithi183 ceph-mon[29736]: 2.1b scrub starts 2024-09-10T19:32:20.387 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:32:20 smithi183 ceph-mon[29736]: 2.1b scrub ok 2024-09-10T19:32:20.387 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:32:20 smithi183 ceph-mon[29736]: purged_snaps scrub starts 2024-09-10T19:32:20.387 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:32:20 smithi183 ceph-mon[29736]: purged_snaps scrub ok 2024-09-10T19:32:20.387 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:32:20 smithi183 ceph-mon[29736]: osdmap e447: 8 total, 8 up, 8 in 2024-09-10T19:32:20.387 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:32:20 smithi183 ceph-mon[29736]: from='client.? 172.21.15.183:0/3764915285' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-09-10T19:32:20.387 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:32:20 smithi183 ceph-mon[41562]: 2.1b scrub starts 2024-09-10T19:32:20.387 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:32:20 smithi183 ceph-mon[41562]: 2.1b scrub ok 2024-09-10T19:32:20.387 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:32:20 smithi183 ceph-mon[41562]: purged_snaps scrub starts 2024-09-10T19:32:20.387 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:32:20 smithi183 ceph-mon[41562]: purged_snaps scrub ok 2024-09-10T19:32:20.387 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:32:20 smithi183 ceph-mon[41562]: osdmap e447: 8 total, 8 up, 8 in 2024-09-10T19:32:20.387 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:32:20 smithi183 ceph-mon[41562]: from='client.? 172.21.15.183:0/3764915285' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-09-10T19:32:20.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:32:20 smithi191 ceph-mon[32884]: 2.1b scrub starts 2024-09-10T19:32:20.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:32:20 smithi191 ceph-mon[32884]: 2.1b scrub ok 2024-09-10T19:32:20.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:32:20 smithi191 ceph-mon[32884]: purged_snaps scrub starts 2024-09-10T19:32:20.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:32:20 smithi191 ceph-mon[32884]: purged_snaps scrub ok 2024-09-10T19:32:20.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:32:20 smithi191 ceph-mon[32884]: osdmap e447: 8 total, 8 up, 8 in 2024-09-10T19:32:20.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:32:20 smithi191 ceph-mon[32884]: from='client.? 172.21.15.183:0/3764915285' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2024-09-10T19:32:20.466 INFO:journalctl@ceph.iscsi.iscsi.a.smithi191.stdout:Sep 10 19:32:20 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-iscsi-iscsi-a[61816]: debug there is no tcmu-runner data available 2024-09-10T19:32:20.794 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph osd primary-affinity 6 1 2024-09-10T19:32:21.039 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:32:21.300 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:32:21 smithi183 ceph-mon[29736]: 4.13 scrub starts 2024-09-10T19:32:21.300 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:32:21 smithi183 ceph-mon[29736]: 4.13 scrub ok 2024-09-10T19:32:21.300 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:32:21 smithi183 ceph-mon[29736]: pgmap v1025: 142 pgs: 142 active+clean; 582 KiB data, 5.7 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-10T19:32:21.300 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:32:21 smithi183 ceph-mon[29736]: 2.b scrub starts 2024-09-10T19:32:21.300 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:32:21 smithi183 ceph-mon[29736]: 2.b scrub ok 2024-09-10T19:32:21.300 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:32:21 smithi183 ceph-mon[29736]: from='client.? 172.21.15.183:0/3764915285' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-09-10T19:32:21.300 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:32:21 smithi183 ceph-mon[29736]: osdmap e448: 8 total, 8 up, 8 in 2024-09-10T19:32:21.300 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:32:21 smithi183 ceph-mon[29736]: from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:32:21.300 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:32:21 smithi183 ceph-mon[29736]: 2.1e scrub starts 2024-09-10T19:32:21.300 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:32:21 smithi183 ceph-mon[29736]: 2.1e scrub ok 2024-09-10T19:32:21.301 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:32:21 smithi183 ceph-mon[29736]: 4.f scrub starts 2024-09-10T19:32:21.301 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:32:21 smithi183 ceph-mon[29736]: 4.f scrub ok 2024-09-10T19:32:21.301 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:32:21 smithi183 ceph-mon[41562]: 4.13 scrub starts 2024-09-10T19:32:21.301 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:32:21 smithi183 ceph-mon[41562]: 4.13 scrub ok 2024-09-10T19:32:21.301 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:32:21 smithi183 ceph-mon[41562]: pgmap v1025: 142 pgs: 142 active+clean; 582 KiB data, 5.7 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-10T19:32:21.301 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:32:21 smithi183 ceph-mon[41562]: 2.b scrub starts 2024-09-10T19:32:21.301 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:32:21 smithi183 ceph-mon[41562]: 2.b scrub ok 2024-09-10T19:32:21.301 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:32:21 smithi183 ceph-mon[41562]: from='client.? 172.21.15.183:0/3764915285' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-09-10T19:32:21.301 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:32:21 smithi183 ceph-mon[41562]: osdmap e448: 8 total, 8 up, 8 in 2024-09-10T19:32:21.301 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:32:21 smithi183 ceph-mon[41562]: from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:32:21.301 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:32:21 smithi183 ceph-mon[41562]: 2.1e scrub starts 2024-09-10T19:32:21.301 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:32:21 smithi183 ceph-mon[41562]: 2.1e scrub ok 2024-09-10T19:32:21.302 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:32:21 smithi183 ceph-mon[41562]: 4.f scrub starts 2024-09-10T19:32:21.302 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:32:21 smithi183 ceph-mon[41562]: 4.f scrub ok 2024-09-10T19:32:21.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:32:21 smithi191 ceph-mon[32884]: 4.13 scrub starts 2024-09-10T19:32:21.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:32:21 smithi191 ceph-mon[32884]: 4.13 scrub ok 2024-09-10T19:32:21.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:32:21 smithi191 ceph-mon[32884]: pgmap v1025: 142 pgs: 142 active+clean; 582 KiB data, 5.7 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-10T19:32:21.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:32:21 smithi191 ceph-mon[32884]: 2.b scrub starts 2024-09-10T19:32:21.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:32:21 smithi191 ceph-mon[32884]: 2.b scrub ok 2024-09-10T19:32:21.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:32:21 smithi191 ceph-mon[32884]: from='client.? 172.21.15.183:0/3764915285' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-09-10T19:32:21.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:32:21 smithi191 ceph-mon[32884]: osdmap e448: 8 total, 8 up, 8 in 2024-09-10T19:32:21.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:32:21 smithi191 ceph-mon[32884]: from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:32:21.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:32:21 smithi191 ceph-mon[32884]: 2.1e scrub starts 2024-09-10T19:32:21.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:32:21 smithi191 ceph-mon[32884]: 2.1e scrub ok 2024-09-10T19:32:21.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:32:21 smithi191 ceph-mon[32884]: 4.f scrub starts 2024-09-10T19:32:21.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:32:21 smithi191 ceph-mon[32884]: 4.f scrub ok 2024-09-10T19:32:21.681 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Sep 10 19:32:21 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-y[29989]: ::ffff:172.21.15.191 - - [10/Sep/2024:19:32:21] "GET /metrics HTTP/1.1" 200 222450 "" "Prometheus/2.43.0" 2024-09-10T19:32:22.161 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:32:22 smithi183 ceph-mon[29736]: 2.7 scrub starts 2024-09-10T19:32:22.161 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:32:22 smithi183 ceph-mon[29736]: 2.7 scrub ok 2024-09-10T19:32:22.161 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:32:22 smithi183 ceph-mon[29736]: 3.0 scrub starts 2024-09-10T19:32:22.161 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:32:22 smithi183 ceph-mon[29736]: 3.0 scrub ok 2024-09-10T19:32:22.161 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:32:22 smithi183 ceph-mon[41562]: 2.7 scrub starts 2024-09-10T19:32:22.161 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:32:22 smithi183 ceph-mon[41562]: 2.7 scrub ok 2024-09-10T19:32:22.161 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:32:22 smithi183 ceph-mon[41562]: 3.0 scrub starts 2024-09-10T19:32:22.162 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:32:22 smithi183 ceph-mon[41562]: 3.0 scrub ok 2024-09-10T19:32:22.465 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:32:22 smithi191 ceph-mon[32884]: 2.7 scrub starts 2024-09-10T19:32:22.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:32:22 smithi191 ceph-mon[32884]: 2.7 scrub ok 2024-09-10T19:32:22.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:32:22 smithi191 ceph-mon[32884]: 3.0 scrub starts 2024-09-10T19:32:22.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:32:22 smithi191 ceph-mon[32884]: 3.0 scrub ok 2024-09-10T19:32:23.053 INFO:teuthology.orchestra.run.smithi183.stderr:set osd.6 primary-affinity to 1 (8655362) 2024-09-10T19:32:23.376 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:32:23 smithi183 ceph-mon[29736]: pgmap v1027: 142 pgs: 142 active+clean; 582 KiB data, 5.7 GiB used, 709 GiB / 715 GiB avail; 854 B/s rd, 0 op/s 2024-09-10T19:32:23.376 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:32:23 smithi183 ceph-mon[29736]: 6.c scrub starts 2024-09-10T19:32:23.377 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:32:23 smithi183 ceph-mon[29736]: 6.c scrub ok 2024-09-10T19:32:23.377 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:32:23 smithi183 ceph-mon[29736]: from='client.? 172.21.15.183:0/3652001829' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]: dispatch 2024-09-10T19:32:23.377 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:32:23 smithi183 ceph-mon[29736]: 1.9 scrub starts 2024-09-10T19:32:23.377 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:32:23 smithi183 ceph-mon[29736]: 1.9 scrub ok 2024-09-10T19:32:23.377 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:32:23 smithi183 ceph-mon[41562]: pgmap v1027: 142 pgs: 142 active+clean; 582 KiB data, 5.7 GiB used, 709 GiB / 715 GiB avail; 854 B/s rd, 0 op/s 2024-09-10T19:32:23.377 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:32:23 smithi183 ceph-mon[41562]: 6.c scrub starts 2024-09-10T19:32:23.377 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:32:23 smithi183 ceph-mon[41562]: 6.c scrub ok 2024-09-10T19:32:23.377 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:32:23 smithi183 ceph-mon[41562]: from='client.? 172.21.15.183:0/3652001829' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]: dispatch 2024-09-10T19:32:23.377 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:32:23 smithi183 ceph-mon[41562]: 1.9 scrub starts 2024-09-10T19:32:23.377 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:32:23 smithi183 ceph-mon[41562]: 1.9 scrub ok 2024-09-10T19:32:23.465 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:32:23 smithi191 ceph-mon[32884]: pgmap v1027: 142 pgs: 142 active+clean; 582 KiB data, 5.7 GiB used, 709 GiB / 715 GiB avail; 854 B/s rd, 0 op/s 2024-09-10T19:32:23.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:32:23 smithi191 ceph-mon[32884]: 6.c scrub starts 2024-09-10T19:32:23.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:32:23 smithi191 ceph-mon[32884]: 6.c scrub ok 2024-09-10T19:32:23.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:32:23 smithi191 ceph-mon[32884]: from='client.? 172.21.15.183:0/3652001829' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]: dispatch 2024-09-10T19:32:23.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:32:23 smithi191 ceph-mon[32884]: 1.9 scrub starts 2024-09-10T19:32:23.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:32:23 smithi191 ceph-mon[32884]: 1.9 scrub ok 2024-09-10T19:32:23.685 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph osd reweight 7 1 2024-09-10T19:32:23.931 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:32:24.304 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:32:24 smithi183 ceph-mon[41562]: 5.1 scrub starts 2024-09-10T19:32:24.304 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:32:24 smithi183 ceph-mon[41562]: 5.1 scrub ok 2024-09-10T19:32:24.304 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:32:24 smithi183 ceph-mon[41562]: from='client.? 172.21.15.183:0/3652001829' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]': finished 2024-09-10T19:32:24.304 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:32:24 smithi183 ceph-mon[41562]: osdmap e449: 8 total, 8 up, 8 in 2024-09-10T19:32:24.304 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:32:24 smithi183 ceph-mon[41562]: 6.a scrub starts 2024-09-10T19:32:24.305 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:32:24 smithi183 ceph-mon[41562]: 6.a scrub ok 2024-09-10T19:32:24.305 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:32:24 smithi183 ceph-mon[41562]: 1.2 scrub starts 2024-09-10T19:32:24.305 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:32:24 smithi183 ceph-mon[41562]: 1.2 scrub ok 2024-09-10T19:32:24.305 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:32:24 smithi183 ceph-mon[29736]: 5.1 scrub starts 2024-09-10T19:32:24.305 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:32:24 smithi183 ceph-mon[29736]: 5.1 scrub ok 2024-09-10T19:32:24.305 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:32:24 smithi183 ceph-mon[29736]: from='client.? 172.21.15.183:0/3652001829' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]': finished 2024-09-10T19:32:24.305 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:32:24 smithi183 ceph-mon[29736]: osdmap e449: 8 total, 8 up, 8 in 2024-09-10T19:32:24.305 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:32:24 smithi183 ceph-mon[29736]: 6.a scrub starts 2024-09-10T19:32:24.305 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:32:24 smithi183 ceph-mon[29736]: 6.a scrub ok 2024-09-10T19:32:24.305 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:32:24 smithi183 ceph-mon[29736]: 1.2 scrub starts 2024-09-10T19:32:24.305 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:32:24 smithi183 ceph-mon[29736]: 1.2 scrub ok 2024-09-10T19:32:24.468 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:32:24 smithi191 ceph-mon[32884]: 5.1 scrub starts 2024-09-10T19:32:24.468 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:32:24 smithi191 ceph-mon[32884]: 5.1 scrub ok 2024-09-10T19:32:24.469 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:32:24 smithi191 ceph-mon[32884]: from='client.? 172.21.15.183:0/3652001829' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 6, "weight": 1.0}]': finished 2024-09-10T19:32:24.469 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:32:24 smithi191 ceph-mon[32884]: osdmap e449: 8 total, 8 up, 8 in 2024-09-10T19:32:24.469 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:32:24 smithi191 ceph-mon[32884]: 6.a scrub starts 2024-09-10T19:32:24.469 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:32:24 smithi191 ceph-mon[32884]: 6.a scrub ok 2024-09-10T19:32:24.469 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:32:24 smithi191 ceph-mon[32884]: 1.2 scrub starts 2024-09-10T19:32:24.469 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:32:24 smithi191 ceph-mon[32884]: 1.2 scrub ok 2024-09-10T19:32:25.420 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:32:25 smithi183 ceph-mon[29736]: 4.14 scrub starts 2024-09-10T19:32:25.420 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:32:25 smithi183 ceph-mon[29736]: 4.14 scrub ok 2024-09-10T19:32:25.420 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:32:25 smithi183 ceph-mon[29736]: pgmap v1029: 142 pgs: 142 active+clean; 582 KiB data, 5.7 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-10T19:32:25.420 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:32:25 smithi183 ceph-mon[41562]: 4.14 scrub starts 2024-09-10T19:32:25.420 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:32:25 smithi183 ceph-mon[41562]: 4.14 scrub ok 2024-09-10T19:32:25.421 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:32:25 smithi183 ceph-mon[41562]: pgmap v1029: 142 pgs: 142 active+clean; 582 KiB data, 5.7 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-10T19:32:25.465 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:32:25 smithi191 ceph-mon[32884]: 4.14 scrub starts 2024-09-10T19:32:25.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:32:25 smithi191 ceph-mon[32884]: 4.14 scrub ok 2024-09-10T19:32:25.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:32:25 smithi191 ceph-mon[32884]: pgmap v1029: 142 pgs: 142 active+clean; 582 KiB data, 5.7 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-10T19:32:26.075 INFO:teuthology.orchestra.run.smithi183.stderr:reweighted osd.7 to 1 (10000) 2024-09-10T19:32:26.391 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:32:26 smithi183 ceph-mon[29736]: 5.0 scrub starts 2024-09-10T19:32:26.391 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:32:26 smithi183 ceph-mon[29736]: 5.0 scrub ok 2024-09-10T19:32:26.391 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:32:26 smithi183 ceph-mon[29736]: from='client.? 172.21.15.183:0/2262906922' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-09-10T19:32:26.391 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:32:26 smithi183 ceph-mon[41562]: 5.0 scrub starts 2024-09-10T19:32:26.391 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:32:26 smithi183 ceph-mon[41562]: 5.0 scrub ok 2024-09-10T19:32:26.391 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:32:26 smithi183 ceph-mon[41562]: from='client.? 172.21.15.183:0/2262906922' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-09-10T19:32:26.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:32:26 smithi191 ceph-mon[32884]: 5.0 scrub starts 2024-09-10T19:32:26.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:32:26 smithi191 ceph-mon[32884]: 5.0 scrub ok 2024-09-10T19:32:26.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:32:26 smithi191 ceph-mon[32884]: from='client.? 172.21.15.183:0/2262906922' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-09-10T19:32:26.780 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph osd primary-affinity 7 1 2024-09-10T19:32:27.028 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:32:27.341 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:32:27 smithi183 ceph-mon[41562]: pgmap v1030: 142 pgs: 142 active+clean; 582 KiB data, 5.8 GiB used, 709 GiB / 715 GiB avail; 732 B/s rd, 0 op/s 2024-09-10T19:32:27.341 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:32:27 smithi183 ceph-mon[41562]: from='client.? 172.21.15.183:0/2262906922' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-09-10T19:32:27.342 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:32:27 smithi183 ceph-mon[41562]: osdmap e450: 8 total, 8 up, 8 in 2024-09-10T19:32:27.342 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:32:27 smithi183 ceph-mon[41562]: 6.15 scrub starts 2024-09-10T19:32:27.342 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:32:27 smithi183 ceph-mon[41562]: 6.15 scrub ok 2024-09-10T19:32:27.342 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:32:27 smithi183 ceph-mon[29736]: pgmap v1030: 142 pgs: 142 active+clean; 582 KiB data, 5.8 GiB used, 709 GiB / 715 GiB avail; 732 B/s rd, 0 op/s 2024-09-10T19:32:27.342 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:32:27 smithi183 ceph-mon[29736]: from='client.? 172.21.15.183:0/2262906922' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-09-10T19:32:27.342 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:32:27 smithi183 ceph-mon[29736]: osdmap e450: 8 total, 8 up, 8 in 2024-09-10T19:32:27.342 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:32:27 smithi183 ceph-mon[29736]: 6.15 scrub starts 2024-09-10T19:32:27.343 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:32:27 smithi183 ceph-mon[29736]: 6.15 scrub ok 2024-09-10T19:32:27.367 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:32:27 smithi191 ceph-mon[32884]: pgmap v1030: 142 pgs: 142 active+clean; 582 KiB data, 5.8 GiB used, 709 GiB / 715 GiB avail; 732 B/s rd, 0 op/s 2024-09-10T19:32:27.367 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:32:27 smithi191 ceph-mon[32884]: from='client.? 172.21.15.183:0/2262906922' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-09-10T19:32:27.367 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:32:27 smithi191 ceph-mon[32884]: osdmap e450: 8 total, 8 up, 8 in 2024-09-10T19:32:27.367 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:32:27 smithi191 ceph-mon[32884]: 6.15 scrub starts 2024-09-10T19:32:27.367 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:32:27 smithi191 ceph-mon[32884]: 6.15 scrub ok 2024-09-10T19:32:27.716 INFO:journalctl@ceph.mgr.x.smithi191.stdout:Sep 10 19:32:27 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-x[34967]: ::ffff:172.21.15.191 - - [10/Sep/2024:19:32:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T19:32:28.400 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:32:28 smithi183 ceph-mon[29736]: 4.1d scrub starts 2024-09-10T19:32:28.400 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:32:28 smithi183 ceph-mon[29736]: 4.1d scrub ok 2024-09-10T19:32:28.400 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:32:28 smithi183 ceph-mon[29736]: 2.18 scrub starts 2024-09-10T19:32:28.400 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:32:28 smithi183 ceph-mon[29736]: 2.18 scrub ok 2024-09-10T19:32:28.400 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:32:28 smithi183 ceph-mon[41562]: 4.1d scrub starts 2024-09-10T19:32:28.400 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:32:28 smithi183 ceph-mon[41562]: 4.1d scrub ok 2024-09-10T19:32:28.401 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:32:28 smithi183 ceph-mon[41562]: 2.18 scrub starts 2024-09-10T19:32:28.401 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:32:28 smithi183 ceph-mon[41562]: 2.18 scrub ok 2024-09-10T19:32:28.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:32:28 smithi191 ceph-mon[32884]: 4.1d scrub starts 2024-09-10T19:32:28.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:32:28 smithi191 ceph-mon[32884]: 4.1d scrub ok 2024-09-10T19:32:28.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:32:28 smithi191 ceph-mon[32884]: 2.18 scrub starts 2024-09-10T19:32:28.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:32:28 smithi191 ceph-mon[32884]: 2.18 scrub ok 2024-09-10T19:32:29.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:32:29 smithi183 ceph-mon[29736]: pgmap v1032: 142 pgs: 142 active+clean; 582 KiB data, 5.8 GiB used, 709 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-09-10T19:32:29.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:32:29 smithi183 ceph-mon[29736]: 1.4 scrub starts 2024-09-10T19:32:29.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:32:29 smithi183 ceph-mon[29736]: 1.4 scrub ok 2024-09-10T19:32:29.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:32:29 smithi183 ceph-mon[29736]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:32:29.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:32:29 smithi183 ceph-mon[41562]: pgmap v1032: 142 pgs: 142 active+clean; 582 KiB data, 5.8 GiB used, 709 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-09-10T19:32:29.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:32:29 smithi183 ceph-mon[41562]: 1.4 scrub starts 2024-09-10T19:32:29.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:32:29 smithi183 ceph-mon[41562]: 1.4 scrub ok 2024-09-10T19:32:29.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:32:29 smithi183 ceph-mon[41562]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:32:29.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:32:29 smithi191 ceph-mon[32884]: pgmap v1032: 142 pgs: 142 active+clean; 582 KiB data, 5.8 GiB used, 709 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-09-10T19:32:29.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:32:29 smithi191 ceph-mon[32884]: 1.4 scrub starts 2024-09-10T19:32:29.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:32:29 smithi191 ceph-mon[32884]: 1.4 scrub ok 2024-09-10T19:32:29.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:32:29 smithi191 ceph-mon[32884]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:32:30.096 INFO:teuthology.orchestra.run.smithi183.stderr:set osd.7 primary-affinity to 1 (8655362) 2024-09-10T19:32:30.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:32:30 smithi183 ceph-mon[29736]: from='client.? 172.21.15.183:0/3224124356' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]: dispatch 2024-09-10T19:32:30.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:32:30 smithi183 ceph-mon[29736]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]: dispatch 2024-09-10T19:32:30.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:32:30 smithi183 ceph-mon[41562]: from='client.? 172.21.15.183:0/3224124356' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]: dispatch 2024-09-10T19:32:30.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:32:30 smithi183 ceph-mon[41562]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]: dispatch 2024-09-10T19:32:30.466 INFO:journalctl@ceph.iscsi.iscsi.a.smithi191.stdout:Sep 10 19:32:30 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-iscsi-iscsi-a[61816]: debug there is no tcmu-runner data available 2024-09-10T19:32:30.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:32:30 smithi191 ceph-mon[32884]: from='client.? 172.21.15.183:0/3224124356' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]: dispatch 2024-09-10T19:32:30.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:32:30 smithi191 ceph-mon[32884]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]: dispatch 2024-09-10T19:32:30.862 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph osd reweight 1 1 2024-09-10T19:32:31.083 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:32:31 smithi183 ceph-mon[29736]: 6.17 scrub starts 2024-09-10T19:32:31.083 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:32:31 smithi183 ceph-mon[29736]: 6.17 scrub ok 2024-09-10T19:32:31.083 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:32:31 smithi183 ceph-mon[29736]: pgmap v1033: 142 pgs: 142 active+clean; 582 KiB data, 5.8 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:32:31.113 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:32:31.337 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Sep 10 19:32:31 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-y[29989]: ::ffff:172.21.15.191 - - [10/Sep/2024:19:32:31] "GET /metrics HTTP/1.1" 200 222431 "" "Prometheus/2.43.0" 2024-09-10T19:32:31.338 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:32:31 smithi183 ceph-mon[29736]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]': finished 2024-09-10T19:32:31.338 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:32:31 smithi183 ceph-mon[29736]: osdmap e451: 8 total, 8 up, 8 in 2024-09-10T19:32:31.338 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:32:31 smithi183 ceph-mon[29736]: 5.14 scrub starts 2024-09-10T19:32:31.338 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:32:31 smithi183 ceph-mon[29736]: from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:32:31.338 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:32:31 smithi183 ceph-mon[29736]: 5.14 scrub ok 2024-09-10T19:32:31.338 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:32:31 smithi183 ceph-mon[29736]: 4.15 scrub starts 2024-09-10T19:32:31.338 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:32:31 smithi183 ceph-mon[29736]: 4.15 scrub ok 2024-09-10T19:32:31.338 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:32:31 smithi183 ceph-mon[41562]: 6.17 scrub starts 2024-09-10T19:32:31.338 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:32:31 smithi183 ceph-mon[41562]: 6.17 scrub ok 2024-09-10T19:32:31.338 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:32:31 smithi183 ceph-mon[41562]: pgmap v1033: 142 pgs: 142 active+clean; 582 KiB data, 5.8 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:32:31.339 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:32:31 smithi183 ceph-mon[41562]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]': finished 2024-09-10T19:32:31.339 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:32:31 smithi183 ceph-mon[41562]: osdmap e451: 8 total, 8 up, 8 in 2024-09-10T19:32:31.339 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:32:31 smithi183 ceph-mon[41562]: 5.14 scrub starts 2024-09-10T19:32:31.339 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:32:31 smithi183 ceph-mon[41562]: from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:32:31.339 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:32:31 smithi183 ceph-mon[41562]: 5.14 scrub ok 2024-09-10T19:32:31.339 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:32:31 smithi183 ceph-mon[41562]: 4.15 scrub starts 2024-09-10T19:32:31.339 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:32:31 smithi183 ceph-mon[41562]: 4.15 scrub ok 2024-09-10T19:32:31.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:32:31 smithi191 ceph-mon[32884]: 6.17 scrub starts 2024-09-10T19:32:31.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:32:31 smithi191 ceph-mon[32884]: 6.17 scrub ok 2024-09-10T19:32:31.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:32:31 smithi191 ceph-mon[32884]: pgmap v1033: 142 pgs: 142 active+clean; 582 KiB data, 5.8 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:32:31.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:32:31 smithi191 ceph-mon[32884]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]': finished 2024-09-10T19:32:31.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:32:31 smithi191 ceph-mon[32884]: osdmap e451: 8 total, 8 up, 8 in 2024-09-10T19:32:31.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:32:31 smithi191 ceph-mon[32884]: 5.14 scrub starts 2024-09-10T19:32:31.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:32:31 smithi191 ceph-mon[32884]: from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:32:31.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:32:31 smithi191 ceph-mon[32884]: 5.14 scrub ok 2024-09-10T19:32:31.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:32:31 smithi191 ceph-mon[32884]: 4.15 scrub starts 2024-09-10T19:32:31.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:32:31 smithi191 ceph-mon[32884]: 4.15 scrub ok 2024-09-10T19:32:32.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:32:32 smithi183 ceph-mon[29736]: 5.13 scrub starts 2024-09-10T19:32:32.182 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:32:32 smithi183 ceph-mon[29736]: 5.13 scrub ok 2024-09-10T19:32:32.182 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:32:32 smithi183 ceph-mon[29736]: 5.12 scrub starts 2024-09-10T19:32:32.182 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:32:32 smithi183 ceph-mon[29736]: 5.12 scrub ok 2024-09-10T19:32:32.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:32:32 smithi183 ceph-mon[41562]: 5.13 scrub starts 2024-09-10T19:32:32.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:32:32 smithi183 ceph-mon[41562]: 5.13 scrub ok 2024-09-10T19:32:32.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:32:32 smithi183 ceph-mon[41562]: 5.12 scrub starts 2024-09-10T19:32:32.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:32:32 smithi183 ceph-mon[41562]: 5.12 scrub ok 2024-09-10T19:32:32.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:32:32 smithi191 ceph-mon[32884]: 5.13 scrub starts 2024-09-10T19:32:32.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:32:32 smithi191 ceph-mon[32884]: 5.13 scrub ok 2024-09-10T19:32:32.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:32:32 smithi191 ceph-mon[32884]: 5.12 scrub starts 2024-09-10T19:32:32.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:32:32 smithi191 ceph-mon[32884]: 5.12 scrub ok 2024-09-10T19:32:33.099 INFO:teuthology.orchestra.run.smithi183.stderr:reweighted osd.1 to 1 (10000) 2024-09-10T19:32:33.409 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:32:33 smithi183 ceph-mon[29736]: pgmap v1035: 142 pgs: 142 active+clean; 582 KiB data, 5.8 GiB used, 709 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-10T19:32:33.409 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:32:33 smithi183 ceph-mon[29736]: 5.c scrub starts 2024-09-10T19:32:33.409 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:32:33 smithi183 ceph-mon[29736]: 5.c scrub ok 2024-09-10T19:32:33.409 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:32:33 smithi183 ceph-mon[29736]: from='client.? 172.21.15.183:0/912254153' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-09-10T19:32:33.410 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:32:33 smithi183 ceph-mon[41562]: pgmap v1035: 142 pgs: 142 active+clean; 582 KiB data, 5.8 GiB used, 709 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-10T19:32:33.410 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:32:33 smithi183 ceph-mon[41562]: 5.c scrub starts 2024-09-10T19:32:33.410 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:32:33 smithi183 ceph-mon[41562]: 5.c scrub ok 2024-09-10T19:32:33.410 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:32:33 smithi183 ceph-mon[41562]: from='client.? 172.21.15.183:0/912254153' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-09-10T19:32:33.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:32:33 smithi191 ceph-mon[32884]: pgmap v1035: 142 pgs: 142 active+clean; 582 KiB data, 5.8 GiB used, 709 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-10T19:32:33.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:32:33 smithi191 ceph-mon[32884]: 5.c scrub starts 2024-09-10T19:32:33.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:32:33 smithi191 ceph-mon[32884]: 5.c scrub ok 2024-09-10T19:32:33.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:32:33 smithi191 ceph-mon[32884]: from='client.? 172.21.15.183:0/912254153' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-09-10T19:32:33.838 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph osd primary-affinity 1 1 2024-09-10T19:32:34.082 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:32:34.364 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:32:34 smithi183 ceph-mon[29736]: from='client.? 172.21.15.183:0/912254153' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-09-10T19:32:34.364 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:32:34 smithi183 ceph-mon[29736]: osdmap e452: 8 total, 8 up, 8 in 2024-09-10T19:32:34.365 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:32:34 smithi183 ceph-mon[29736]: 6.12 scrub starts 2024-09-10T19:32:34.365 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:32:34 smithi183 ceph-mon[29736]: 6.12 scrub ok 2024-09-10T19:32:34.365 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:32:34 smithi183 ceph-mon[41562]: from='client.? 172.21.15.183:0/912254153' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-09-10T19:32:34.365 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:32:34 smithi183 ceph-mon[41562]: osdmap e452: 8 total, 8 up, 8 in 2024-09-10T19:32:34.365 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:32:34 smithi183 ceph-mon[41562]: 6.12 scrub starts 2024-09-10T19:32:34.365 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:32:34 smithi183 ceph-mon[41562]: 6.12 scrub ok 2024-09-10T19:32:34.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:32:34 smithi191 ceph-mon[32884]: from='client.? 172.21.15.183:0/912254153' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-09-10T19:32:34.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:32:34 smithi191 ceph-mon[32884]: osdmap e452: 8 total, 8 up, 8 in 2024-09-10T19:32:34.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:32:34 smithi191 ceph-mon[32884]: 6.12 scrub starts 2024-09-10T19:32:34.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:32:34 smithi191 ceph-mon[32884]: 6.12 scrub ok 2024-09-10T19:32:35.389 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:32:35 smithi183 ceph-mon[29736]: 5.8 scrub starts 2024-09-10T19:32:35.389 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:32:35 smithi183 ceph-mon[29736]: 5.8 scrub ok 2024-09-10T19:32:35.389 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:32:35 smithi183 ceph-mon[29736]: pgmap v1037: 142 pgs: 142 active+clean; 582 KiB data, 5.8 GiB used, 709 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-09-10T19:32:35.389 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:32:35 smithi183 ceph-mon[41562]: 5.8 scrub starts 2024-09-10T19:32:35.389 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:32:35 smithi183 ceph-mon[41562]: 5.8 scrub ok 2024-09-10T19:32:35.389 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:32:35 smithi183 ceph-mon[41562]: pgmap v1037: 142 pgs: 142 active+clean; 582 KiB data, 5.8 GiB used, 709 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-09-10T19:32:35.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:32:35 smithi191 ceph-mon[32884]: 5.8 scrub starts 2024-09-10T19:32:35.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:32:35 smithi191 ceph-mon[32884]: 5.8 scrub ok 2024-09-10T19:32:35.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:32:35 smithi191 ceph-mon[32884]: pgmap v1037: 142 pgs: 142 active+clean; 582 KiB data, 5.8 GiB used, 709 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-09-10T19:32:36.118 INFO:teuthology.orchestra.run.smithi183.stderr:set osd.1 primary-affinity to 1 (8655362) 2024-09-10T19:32:36.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:32:36 smithi183 ceph-mon[29736]: 2.9 scrub starts 2024-09-10T19:32:36.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:32:36 smithi183 ceph-mon[29736]: 2.9 scrub ok 2024-09-10T19:32:36.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:32:36 smithi183 ceph-mon[29736]: from='client.? 172.21.15.183:0/2921251477' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]: dispatch 2024-09-10T19:32:36.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:32:36 smithi183 ceph-mon[29736]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]: dispatch 2024-09-10T19:32:36.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:32:36 smithi183 ceph-mon[41562]: 2.9 scrub starts 2024-09-10T19:32:36.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:32:36 smithi183 ceph-mon[41562]: 2.9 scrub ok 2024-09-10T19:32:36.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:32:36 smithi183 ceph-mon[41562]: from='client.? 172.21.15.183:0/2921251477' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]: dispatch 2024-09-10T19:32:36.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:32:36 smithi183 ceph-mon[41562]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]: dispatch 2024-09-10T19:32:36.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:32:36 smithi191 ceph-mon[32884]: 2.9 scrub starts 2024-09-10T19:32:36.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:32:36 smithi191 ceph-mon[32884]: 2.9 scrub ok 2024-09-10T19:32:36.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:32:36 smithi191 ceph-mon[32884]: from='client.? 172.21.15.183:0/2921251477' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]: dispatch 2024-09-10T19:32:36.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:32:36 smithi191 ceph-mon[32884]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]: dispatch 2024-09-10T19:32:36.796 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph osd reweight 0 1 2024-09-10T19:32:37.039 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:32:37.431 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:32:37 smithi183 ceph-mon[41562]: 2.17 scrub starts 2024-09-10T19:32:37.431 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:32:37 smithi183 ceph-mon[41562]: 2.17 scrub ok 2024-09-10T19:32:37.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:32:37 smithi183 ceph-mon[41562]: pgmap v1038: 142 pgs: 142 active+clean; 582 KiB data, 5.8 GiB used, 709 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-10T19:32:37.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:32:37 smithi183 ceph-mon[41562]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]': finished 2024-09-10T19:32:37.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:32:37 smithi183 ceph-mon[41562]: osdmap e453: 8 total, 8 up, 8 in 2024-09-10T19:32:37.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:32:37 smithi183 ceph-mon[41562]: 6.f scrub starts 2024-09-10T19:32:37.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:32:37 smithi183 ceph-mon[41562]: 6.f scrub ok 2024-09-10T19:32:37.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:32:37 smithi183 ceph-mon[29736]: 2.17 scrub starts 2024-09-10T19:32:37.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:32:37 smithi183 ceph-mon[29736]: 2.17 scrub ok 2024-09-10T19:32:37.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:32:37 smithi183 ceph-mon[29736]: pgmap v1038: 142 pgs: 142 active+clean; 582 KiB data, 5.8 GiB used, 709 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-10T19:32:37.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:32:37 smithi183 ceph-mon[29736]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]': finished 2024-09-10T19:32:37.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:32:37 smithi183 ceph-mon[29736]: osdmap e453: 8 total, 8 up, 8 in 2024-09-10T19:32:37.433 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:32:37 smithi183 ceph-mon[29736]: 6.f scrub starts 2024-09-10T19:32:37.433 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:32:37 smithi183 ceph-mon[29736]: 6.f scrub ok 2024-09-10T19:32:37.451 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:32:37 smithi191 ceph-mon[32884]: 2.17 scrub starts 2024-09-10T19:32:37.451 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:32:37 smithi191 ceph-mon[32884]: 2.17 scrub ok 2024-09-10T19:32:37.451 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:32:37 smithi191 ceph-mon[32884]: pgmap v1038: 142 pgs: 142 active+clean; 582 KiB data, 5.8 GiB used, 709 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-10T19:32:37.451 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:32:37 smithi191 ceph-mon[32884]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]': finished 2024-09-10T19:32:37.451 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:32:37 smithi191 ceph-mon[32884]: osdmap e453: 8 total, 8 up, 8 in 2024-09-10T19:32:37.451 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:32:37 smithi191 ceph-mon[32884]: 6.f scrub starts 2024-09-10T19:32:37.451 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:32:37 smithi191 ceph-mon[32884]: 6.f scrub ok 2024-09-10T19:32:37.716 INFO:journalctl@ceph.mgr.x.smithi191.stdout:Sep 10 19:32:37 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-x[34967]: ::ffff:172.21.15.191 - - [10/Sep/2024:19:32:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T19:32:39.128 INFO:teuthology.orchestra.run.smithi183.stderr:reweighted osd.0 to 1 (10000) 2024-09-10T19:32:39.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:32:39 smithi183 ceph-mon[29736]: pgmap v1040: 142 pgs: 142 active+clean; 582 KiB data, 5.8 GiB used, 709 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-09-10T19:32:39.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:32:39 smithi183 ceph-mon[29736]: 2.1f deep-scrub starts 2024-09-10T19:32:39.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:32:39 smithi183 ceph-mon[29736]: 2.1f deep-scrub ok 2024-09-10T19:32:39.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:32:39 smithi183 ceph-mon[29736]: 2.15 deep-scrub starts 2024-09-10T19:32:39.433 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:32:39 smithi183 ceph-mon[29736]: 2.15 deep-scrub ok 2024-09-10T19:32:39.433 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:32:39 smithi183 ceph-mon[29736]: from='client.? 172.21.15.183:0/3156304399' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-09-10T19:32:39.433 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:32:39 smithi183 ceph-mon[29736]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-09-10T19:32:39.433 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:32:39 smithi183 ceph-mon[41562]: pgmap v1040: 142 pgs: 142 active+clean; 582 KiB data, 5.8 GiB used, 709 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-09-10T19:32:39.433 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:32:39 smithi183 ceph-mon[41562]: 2.1f deep-scrub starts 2024-09-10T19:32:39.434 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:32:39 smithi183 ceph-mon[41562]: 2.1f deep-scrub ok 2024-09-10T19:32:39.434 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:32:39 smithi183 ceph-mon[41562]: 2.15 deep-scrub starts 2024-09-10T19:32:39.434 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:32:39 smithi183 ceph-mon[41562]: 2.15 deep-scrub ok 2024-09-10T19:32:39.434 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:32:39 smithi183 ceph-mon[41562]: from='client.? 172.21.15.183:0/3156304399' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-09-10T19:32:39.434 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:32:39 smithi183 ceph-mon[41562]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-09-10T19:32:39.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:32:39 smithi191 ceph-mon[32884]: pgmap v1040: 142 pgs: 142 active+clean; 582 KiB data, 5.8 GiB used, 709 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-09-10T19:32:39.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:32:39 smithi191 ceph-mon[32884]: 2.1f deep-scrub starts 2024-09-10T19:32:39.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:32:39 smithi191 ceph-mon[32884]: 2.1f deep-scrub ok 2024-09-10T19:32:39.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:32:39 smithi191 ceph-mon[32884]: 2.15 deep-scrub starts 2024-09-10T19:32:39.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:32:39 smithi191 ceph-mon[32884]: 2.15 deep-scrub ok 2024-09-10T19:32:39.467 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:32:39 smithi191 ceph-mon[32884]: from='client.? 172.21.15.183:0/3156304399' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-09-10T19:32:39.467 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:32:39 smithi191 ceph-mon[32884]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2024-09-10T19:32:39.882 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph osd primary-affinity 0 1 2024-09-10T19:32:40.131 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:32:40.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:32:40 smithi183 ceph-mon[29736]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-09-10T19:32:40.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:32:40 smithi183 ceph-mon[29736]: osdmap e454: 8 total, 8 up, 8 in 2024-09-10T19:32:40.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:32:40 smithi183 ceph-mon[29736]: 6.14 scrub starts 2024-09-10T19:32:40.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:32:40 smithi183 ceph-mon[29736]: 6.14 scrub ok 2024-09-10T19:32:40.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:32:40 smithi183 ceph-mon[41562]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-09-10T19:32:40.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:32:40 smithi183 ceph-mon[41562]: osdmap e454: 8 total, 8 up, 8 in 2024-09-10T19:32:40.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:32:40 smithi183 ceph-mon[41562]: 6.14 scrub starts 2024-09-10T19:32:40.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:32:40 smithi183 ceph-mon[41562]: 6.14 scrub ok 2024-09-10T19:32:40.466 INFO:journalctl@ceph.iscsi.iscsi.a.smithi191.stdout:Sep 10 19:32:40 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-iscsi-iscsi-a[61816]: debug there is no tcmu-runner data available 2024-09-10T19:32:40.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:32:40 smithi191 ceph-mon[32884]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-09-10T19:32:40.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:32:40 smithi191 ceph-mon[32884]: osdmap e454: 8 total, 8 up, 8 in 2024-09-10T19:32:40.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:32:40 smithi191 ceph-mon[32884]: 6.14 scrub starts 2024-09-10T19:32:40.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:32:40 smithi191 ceph-mon[32884]: 6.14 scrub ok 2024-09-10T19:32:41.313 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:32:41 smithi183 ceph-mon[29736]: 4.1f deep-scrub starts 2024-09-10T19:32:41.313 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:32:41 smithi183 ceph-mon[29736]: 4.1f deep-scrub ok 2024-09-10T19:32:41.313 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:32:41 smithi183 ceph-mon[29736]: pgmap v1042: 142 pgs: 142 active+clean; 582 KiB data, 5.8 GiB used, 709 GiB / 715 GiB avail; 740 B/s rd, 0 op/s 2024-09-10T19:32:41.313 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:32:41 smithi183 ceph-mon[29736]: from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:32:41.313 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:32:41 smithi183 ceph-mon[29736]: purged_snaps scrub starts 2024-09-10T19:32:41.313 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:32:41 smithi183 ceph-mon[29736]: purged_snaps scrub ok 2024-09-10T19:32:41.314 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:32:41 smithi183 ceph-mon[41562]: 4.1f deep-scrub starts 2024-09-10T19:32:41.314 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:32:41 smithi183 ceph-mon[41562]: 4.1f deep-scrub ok 2024-09-10T19:32:41.314 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:32:41 smithi183 ceph-mon[41562]: pgmap v1042: 142 pgs: 142 active+clean; 582 KiB data, 5.8 GiB used, 709 GiB / 715 GiB avail; 740 B/s rd, 0 op/s 2024-09-10T19:32:41.314 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:32:41 smithi183 ceph-mon[41562]: from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:32:41.314 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:32:41 smithi183 ceph-mon[41562]: purged_snaps scrub starts 2024-09-10T19:32:41.314 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:32:41 smithi183 ceph-mon[41562]: purged_snaps scrub ok 2024-09-10T19:32:41.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:32:41 smithi191 ceph-mon[32884]: 4.1f deep-scrub starts 2024-09-10T19:32:41.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:32:41 smithi191 ceph-mon[32884]: 4.1f deep-scrub ok 2024-09-10T19:32:41.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:32:41 smithi191 ceph-mon[32884]: pgmap v1042: 142 pgs: 142 active+clean; 582 KiB data, 5.8 GiB used, 709 GiB / 715 GiB avail; 740 B/s rd, 0 op/s 2024-09-10T19:32:41.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:32:41 smithi191 ceph-mon[32884]: from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:32:41.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:32:41 smithi191 ceph-mon[32884]: purged_snaps scrub starts 2024-09-10T19:32:41.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:32:41 smithi191 ceph-mon[32884]: purged_snaps scrub ok 2024-09-10T19:32:41.622 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Sep 10 19:32:41 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-y[29989]: ::ffff:172.21.15.191 - - [10/Sep/2024:19:32:41] "GET /metrics HTTP/1.1" 200 222431 "" "Prometheus/2.43.0" 2024-09-10T19:32:42.151 INFO:teuthology.orchestra.run.smithi183.stderr:set osd.0 primary-affinity to 1 (8655362) 2024-09-10T19:32:42.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:32:42 smithi183 ceph-mon[29736]: 6.16 deep-scrub starts 2024-09-10T19:32:42.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:32:42 smithi183 ceph-mon[29736]: 6.16 deep-scrub ok 2024-09-10T19:32:42.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:32:42 smithi183 ceph-mon[29736]: 6.6 scrub starts 2024-09-10T19:32:42.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:32:42 smithi183 ceph-mon[29736]: 6.6 scrub ok 2024-09-10T19:32:42.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:32:42 smithi183 ceph-mon[29736]: osdmap e455: 8 total, 8 up, 8 in 2024-09-10T19:32:42.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:32:42 smithi183 ceph-mon[29736]: 5.d scrub starts 2024-09-10T19:32:42.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:32:42 smithi183 ceph-mon[29736]: 5.d scrub ok 2024-09-10T19:32:42.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:32:42 smithi183 ceph-mon[29736]: 5.15 scrub starts 2024-09-10T19:32:42.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:32:42 smithi183 ceph-mon[29736]: 5.15 scrub ok 2024-09-10T19:32:42.433 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:32:42 smithi183 ceph-mon[29736]: 2.1d scrub starts 2024-09-10T19:32:42.433 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:32:42 smithi183 ceph-mon[29736]: 2.1d scrub ok 2024-09-10T19:32:42.433 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:32:42 smithi183 ceph-mon[29736]: from='client.? 172.21.15.183:0/1859875340' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]: dispatch 2024-09-10T19:32:42.433 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:32:42 smithi183 ceph-mon[29736]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]: dispatch 2024-09-10T19:32:42.433 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:32:42 smithi183 ceph-mon[41562]: 6.16 deep-scrub starts 2024-09-10T19:32:42.433 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:32:42 smithi183 ceph-mon[41562]: 6.16 deep-scrub ok 2024-09-10T19:32:42.433 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:32:42 smithi183 ceph-mon[41562]: 6.6 scrub starts 2024-09-10T19:32:42.433 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:32:42 smithi183 ceph-mon[41562]: 6.6 scrub ok 2024-09-10T19:32:42.433 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:32:42 smithi183 ceph-mon[41562]: osdmap e455: 8 total, 8 up, 8 in 2024-09-10T19:32:42.433 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:32:42 smithi183 ceph-mon[41562]: 5.d scrub starts 2024-09-10T19:32:42.433 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:32:42 smithi183 ceph-mon[41562]: 5.d scrub ok 2024-09-10T19:32:42.434 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:32:42 smithi183 ceph-mon[41562]: 5.15 scrub starts 2024-09-10T19:32:42.434 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:32:42 smithi183 ceph-mon[41562]: 5.15 scrub ok 2024-09-10T19:32:42.434 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:32:42 smithi183 ceph-mon[41562]: 2.1d scrub starts 2024-09-10T19:32:42.434 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:32:42 smithi183 ceph-mon[41562]: 2.1d scrub ok 2024-09-10T19:32:42.434 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:32:42 smithi183 ceph-mon[41562]: from='client.? 172.21.15.183:0/1859875340' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]: dispatch 2024-09-10T19:32:42.434 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:32:42 smithi183 ceph-mon[41562]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]: dispatch 2024-09-10T19:32:42.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:32:42 smithi191 ceph-mon[32884]: 6.16 deep-scrub starts 2024-09-10T19:32:42.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:32:42 smithi191 ceph-mon[32884]: 6.16 deep-scrub ok 2024-09-10T19:32:42.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:32:42 smithi191 ceph-mon[32884]: 6.6 scrub starts 2024-09-10T19:32:42.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:32:42 smithi191 ceph-mon[32884]: 6.6 scrub ok 2024-09-10T19:32:42.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:32:42 smithi191 ceph-mon[32884]: osdmap e455: 8 total, 8 up, 8 in 2024-09-10T19:32:42.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:32:42 smithi191 ceph-mon[32884]: 5.d scrub starts 2024-09-10T19:32:42.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:32:42 smithi191 ceph-mon[32884]: 5.d scrub ok 2024-09-10T19:32:42.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:32:42 smithi191 ceph-mon[32884]: 5.15 scrub starts 2024-09-10T19:32:42.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:32:42 smithi191 ceph-mon[32884]: 5.15 scrub ok 2024-09-10T19:32:42.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:32:42 smithi191 ceph-mon[32884]: 2.1d scrub starts 2024-09-10T19:32:42.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:32:42 smithi191 ceph-mon[32884]: 2.1d scrub ok 2024-09-10T19:32:42.467 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:32:42 smithi191 ceph-mon[32884]: from='client.? 172.21.15.183:0/1859875340' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]: dispatch 2024-09-10T19:32:42.467 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:32:42 smithi191 ceph-mon[32884]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]: dispatch 2024-09-10T19:32:42.887 INFO:tasks.thrashosds.thrasher:joining the do_sighup greenlet 2024-09-10T19:32:42.888 INFO:tasks.thrashosds.thrasher:joining the do_optrack_toggle greenlet 2024-09-10T19:32:42.888 INFO:tasks.thrashosds.thrasher:joining the do_dump_ops greenlet 2024-09-10T19:32:42.888 INFO:tasks.thrashosds.thrasher:joining the do_noscrub_toggle greenlet 2024-09-10T19:32:42.888 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-09-10T19:32:42.888 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph osd dump --format=json 2024-09-10T19:32:43.132 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:32:43.405 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:32:43 smithi183 ceph-mon[29736]: purged_snaps scrub starts 2024-09-10T19:32:43.405 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:32:43 smithi183 ceph-mon[29736]: purged_snaps scrub ok 2024-09-10T19:32:43.405 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:32:43 smithi183 ceph-mon[29736]: pgmap v1044: 142 pgs: 142 active+clean; 582 KiB data, 5.8 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-10T19:32:43.405 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:32:43 smithi183 ceph-mon[29736]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]': finished 2024-09-10T19:32:43.405 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:32:43 smithi183 ceph-mon[29736]: osdmap e456: 8 total, 8 up, 8 in 2024-09-10T19:32:43.405 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:32:43 smithi183 ceph-mon[29736]: 4.9 scrub starts 2024-09-10T19:32:43.405 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:32:43 smithi183 ceph-mon[29736]: 4.9 scrub ok 2024-09-10T19:32:43.405 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:32:43 smithi183 ceph-mon[41562]: purged_snaps scrub starts 2024-09-10T19:32:43.406 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:32:43 smithi183 ceph-mon[41562]: purged_snaps scrub ok 2024-09-10T19:32:43.406 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:32:43 smithi183 ceph-mon[41562]: pgmap v1044: 142 pgs: 142 active+clean; 582 KiB data, 5.8 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-10T19:32:43.406 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:32:43 smithi183 ceph-mon[41562]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]': finished 2024-09-10T19:32:43.406 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:32:43 smithi183 ceph-mon[41562]: osdmap e456: 8 total, 8 up, 8 in 2024-09-10T19:32:43.406 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:32:43 smithi183 ceph-mon[41562]: 4.9 scrub starts 2024-09-10T19:32:43.406 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:32:43 smithi183 ceph-mon[41562]: 4.9 scrub ok 2024-09-10T19:32:43.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:32:43 smithi191 ceph-mon[32884]: purged_snaps scrub starts 2024-09-10T19:32:43.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:32:43 smithi191 ceph-mon[32884]: purged_snaps scrub ok 2024-09-10T19:32:43.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:32:43 smithi191 ceph-mon[32884]: pgmap v1044: 142 pgs: 142 active+clean; 582 KiB data, 5.8 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-10T19:32:43.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:32:43 smithi191 ceph-mon[32884]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 0, "weight": 1.0}]': finished 2024-09-10T19:32:43.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:32:43 smithi191 ceph-mon[32884]: osdmap e456: 8 total, 8 up, 8 in 2024-09-10T19:32:43.467 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:32:43 smithi191 ceph-mon[32884]: 4.9 scrub starts 2024-09-10T19:32:43.467 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:32:43 smithi191 ceph-mon[32884]: 4.9 scrub ok 2024-09-10T19:32:44.379 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:32:44 smithi183 ceph-mon[41562]: 2.d scrub starts 2024-09-10T19:32:44.379 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:32:44 smithi183 ceph-mon[41562]: 2.d scrub ok 2024-09-10T19:32:44.379 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:32:44 smithi183 ceph-mon[41562]: 1.1 scrub starts 2024-09-10T19:32:44.379 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:32:44 smithi183 ceph-mon[41562]: 1.1 scrub ok 2024-09-10T19:32:44.379 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:32:44 smithi183 ceph-mon[41562]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:32:44.379 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:32:44 smithi183 ceph-mon[29736]: 2.d scrub starts 2024-09-10T19:32:44.380 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:32:44 smithi183 ceph-mon[29736]: 2.d scrub ok 2024-09-10T19:32:44.380 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:32:44 smithi183 ceph-mon[29736]: 1.1 scrub starts 2024-09-10T19:32:44.380 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:32:44 smithi183 ceph-mon[29736]: 1.1 scrub ok 2024-09-10T19:32:44.380 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:32:44 smithi183 ceph-mon[29736]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:32:44.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:32:44 smithi191 ceph-mon[32884]: 2.d scrub starts 2024-09-10T19:32:44.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:32:44 smithi191 ceph-mon[32884]: 2.d scrub ok 2024-09-10T19:32:44.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:32:44 smithi191 ceph-mon[32884]: 1.1 scrub starts 2024-09-10T19:32:44.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:32:44 smithi191 ceph-mon[32884]: 1.1 scrub ok 2024-09-10T19:32:44.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:32:44 smithi191 ceph-mon[32884]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:32:45.053 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-09-10T19:32:45.053 INFO:teuthology.orchestra.run.smithi183.stdout:{"epoch":456,"fsid":"320f6df8-6fa7-11ef-bcea-c7b262605968","created":"2024-09-10T19:05:03.688531+0000","modified":"2024-09-10T19:32:42.141578+0000","last_up_change":"2024-09-10T19:24:22.870873+0000","last_in_change":"2024-09-10T19:31:32.311543+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":21,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-10T19:08:00.456786+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":11,"pg_placement_num":11,"pg_placement_num_target":11,"pg_num_target":11,"pg_num_pending":11,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"271","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"242","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":".rgw.root","create_time":"2024-09-10T19:10:05.993763+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"60","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}},{"pool":3,"pool_name":"datapool","create_time":"2024-09-10T19:10:06.224267+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":3,"pg_placement_num":3,"pg_placement_num_target":3,"pg_num_target":3,"pg_num_pending":3,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"63","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":3,"snap_epoch":63,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":4,"pool_name":"default.rgw.log","create_time":"2024-09-10T19:10:07.958284+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"62","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}},{"pool":5,"pool_name":"default.rgw.control","create_time":"2024-09-10T19:10:09.933596+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"64","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}},{"pool":6,"pool_name":"default.rgw.meta","create_time":"2024-09-10T19:10:11.993712+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"66","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4},"application_metadata":{"rgw":{}}}],"osds":[{"osd":0,"uuid":"9d323010-394a-49aa-ba9b-244d87faf36e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":8,"last_clean_end":262,"up_from":301,"up_thru":431,"down_at":263,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6810","nonce":3346539414},{"type":"v1","addr":"172.21.15.183:6811","nonce":3346539414}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6812","nonce":3346539414},{"type":"v1","addr":"172.21.15.183:6813","nonce":3346539414}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6816","nonce":3346539414},{"type":"v1","addr":"172.21.15.183:6817","nonce":3346539414}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6814","nonce":3346539414},{"type":"v1","addr":"172.21.15.183:6815","nonce":3346539414}]},"public_addr":"172.21.15.183:6811/3346539414","cluster_addr":"172.21.15.183:6813/3346539414","heartbeat_back_addr":"172.21.15.183:6817/3346539414","heartbeat_front_addr":"172.21.15.183:6815/3346539414","state":["exists","up"]},{"osd":1,"uuid":"71a6c5dc-5109-4284-8d20-187d8e7afc9e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":138,"last_clean_end":255,"up_from":276,"up_thru":431,"down_at":256,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6802","nonce":2764394151},{"type":"v1","addr":"172.21.15.183:6803","nonce":2764394151}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6804","nonce":2764394151},{"type":"v1","addr":"172.21.15.183:6805","nonce":2764394151}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6808","nonce":2764394151},{"type":"v1","addr":"172.21.15.183:6809","nonce":2764394151}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6806","nonce":2764394151},{"type":"v1","addr":"172.21.15.183:6807","nonce":2764394151}]},"public_addr":"172.21.15.183:6803/2764394151","cluster_addr":"172.21.15.183:6805/2764394151","heartbeat_back_addr":"172.21.15.183:6809/2764394151","heartbeat_front_addr":"172.21.15.183:6807/2764394151","state":["exists","up"]},{"osd":2,"uuid":"fc310d5b-b89b-4655-9989-7e10ba356509","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":435,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6818","nonce":3730013786},{"type":"v1","addr":"172.21.15.183:6819","nonce":3730013786}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6820","nonce":3730013786},{"type":"v1","addr":"172.21.15.183:6821","nonce":3730013786}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6824","nonce":3730013786},{"type":"v1","addr":"172.21.15.183:6825","nonce":3730013786}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6822","nonce":3730013786},{"type":"v1","addr":"172.21.15.183:6823","nonce":3730013786}]},"public_addr":"172.21.15.183:6819/3730013786","cluster_addr":"172.21.15.183:6821/3730013786","heartbeat_back_addr":"172.21.15.183:6825/3730013786","heartbeat_front_addr":"172.21.15.183:6823/3730013786","state":["exists","up"]},{"osd":3,"uuid":"7fd07a95-ad22-43f5-a968-539552283e89","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":431,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6826","nonce":3522836984},{"type":"v1","addr":"172.21.15.183:6827","nonce":3522836984}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6828","nonce":3522836984},{"type":"v1","addr":"172.21.15.183:6829","nonce":3522836984}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6832","nonce":3522836984},{"type":"v1","addr":"172.21.15.183:6833","nonce":3522836984}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6830","nonce":3522836984},{"type":"v1","addr":"172.21.15.183:6831","nonce":3522836984}]},"public_addr":"172.21.15.183:6827/3522836984","cluster_addr":"172.21.15.183:6829/3522836984","heartbeat_back_addr":"172.21.15.183:6833/3522836984","heartbeat_front_addr":"172.21.15.183:6831/3522836984","state":["exists","up"]},{"osd":4,"uuid":"631083fb-e503-44da-9b6b-154e88539ee7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":435,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6800","nonce":2674828539},{"type":"v1","addr":"172.21.15.191:6801","nonce":2674828539}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6802","nonce":2674828539},{"type":"v1","addr":"172.21.15.191:6803","nonce":2674828539}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6806","nonce":2674828539},{"type":"v1","addr":"172.21.15.191:6807","nonce":2674828539}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6804","nonce":2674828539},{"type":"v1","addr":"172.21.15.191:6805","nonce":2674828539}]},"public_addr":"172.21.15.191:6801/2674828539","cluster_addr":"172.21.15.191:6803/2674828539","heartbeat_back_addr":"172.21.15.191:6807/2674828539","heartbeat_front_addr":"172.21.15.191:6805/2674828539","state":["exists","up"]},{"osd":5,"uuid":"6b963695-3eca-4e6e-9207-05b8010ebff9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":431,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6808","nonce":681733378},{"type":"v1","addr":"172.21.15.191:6809","nonce":681733378}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6810","nonce":681733378},{"type":"v1","addr":"172.21.15.191:6811","nonce":681733378}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6814","nonce":681733378},{"type":"v1","addr":"172.21.15.191:6815","nonce":681733378}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6812","nonce":681733378},{"type":"v1","addr":"172.21.15.191:6813","nonce":681733378}]},"public_addr":"172.21.15.191:6809/681733378","cluster_addr":"172.21.15.191:6811/681733378","heartbeat_back_addr":"172.21.15.191:6815/681733378","heartbeat_front_addr":"172.21.15.191:6813/681733378","state":["exists","up"]},{"osd":6,"uuid":"2dc9754f-6aef-4e19-8605-2f1591921811","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":431,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6816","nonce":2825813477},{"type":"v1","addr":"172.21.15.191:6817","nonce":2825813477}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6818","nonce":2825813477},{"type":"v1","addr":"172.21.15.191:6819","nonce":2825813477}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6822","nonce":2825813477},{"type":"v1","addr":"172.21.15.191:6823","nonce":2825813477}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6820","nonce":2825813477},{"type":"v1","addr":"172.21.15.191:6821","nonce":2825813477}]},"public_addr":"172.21.15.191:6817/2825813477","cluster_addr":"172.21.15.191:6819/2825813477","heartbeat_back_addr":"172.21.15.191:6823/2825813477","heartbeat_front_addr":"172.21.15.191:6821/2825813477","state":["exists","up"]},{"osd":7,"uuid":"1a9a299e-d5ba-4b9a-85d5-7f6a5bc754e1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":431,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6824","nonce":1422610021},{"type":"v1","addr":"172.21.15.191:6825","nonce":1422610021}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6826","nonce":1422610021},{"type":"v1","addr":"172.21.15.191:6827","nonce":1422610021}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6830","nonce":1422610021},{"type":"v1","addr":"172.21.15.191:6831","nonce":1422610021}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6828","nonce":1422610021},{"type":"v1","addr":"172.21.15.191:6829","nonce":1422610021}]},"public_addr":"172.21.15.191:6825/1422610021","cluster_addr":"172.21.15.191:6827/1422610021","heartbeat_back_addr":"172.21.15.191:6831/1422610021","heartbeat_front_addr":"172.21.15.191:6829/1422610021","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T19:32:18.748197+0000","dead_epoch":262},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T19:31:29.319415+0000","dead_epoch":255},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T19:32:15.944103+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T19:31:44.805134+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T19:32:13.817432+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T19:31:25.054746+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T19:32:41.720207+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T19:32:40.876679+0000","dead_epoch":0}],"pg_upmap":[{"pgid":"2.13","osds":[2,0,5]},{"pgid":"4.a","osds":[3,2,1]}],"pg_upmap_items":[{"pgid":"2.8","mappings":[{"from":0,"to":5},{"from":1,"to":3},{"from":7,"to":4}]},{"pgid":"2.14","mappings":[{"from":5,"to":6}]},{"pgid":"2.18","mappings":[{"from":7,"to":0},{"from":5,"to":4}]},{"pgid":"4.0","mappings":[{"from":3,"to":2}]},{"pgid":"4.b","mappings":[{"from":1,"to":5}]},{"pgid":"4.e","mappings":[{"from":1,"to":5}]},{"pgid":"4.18","mappings":[{"from":1,"to":2}]},{"pgid":"5.15","mappings":[{"from":1,"to":3}]},{"pgid":"5.19","mappings":[{"from":1,"to":6}]},{"pgid":"6.1b","mappings":[{"from":3,"to":0}]}],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.183:6801/3311783362":"2024-09-11T19:10:43.576943+0000","172.21.15.183:0/4088553922":"2024-09-11T19:10:43.576943+0000","172.21.15.183:0/3528769414":"2024-09-11T19:05:48.153879+0000","172.21.15.183:0/1766036658":"2024-09-11T19:05:48.153879+0000","172.21.15.183:0/3044170088":"2024-09-11T19:05:25.702354+0000","172.21.15.183:0/3037333778":"2024-09-11T19:05:25.702354+0000","172.21.15.183:6801/3747320570":"2024-09-11T19:05:25.702354+0000","172.21.15.183:0/3982386906":"2024-09-11T19:10:43.576943+0000","172.21.15.183:0/2436014455":"2024-09-11T19:05:25.702354+0000","172.21.15.183:6801/659745765":"2024-09-11T19:05:48.153879+0000","172.21.15.183:6800/3747320570":"2024-09-11T19:05:25.702354+0000","172.21.15.183:0/1652576282":"2024-09-11T19:10:43.576943+0000","172.21.15.183:6800/3311783362":"2024-09-11T19:10:43.576943+0000","172.21.15.183:0/1012899549":"2024-09-11T19:05:48.153879+0000","172.21.15.183:6800/659745765":"2024-09-11T19:05:48.153879+0000","172.21.15.183:0/412044400":"2024-09-11T19:10:43.576943+0000","172.21.15.183:0/3636171401":"2024-09-11T19:10:43.576943+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-10T19:32:45.378 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:32:45 smithi183 ceph-mon[29736]: 6.1f deep-scrub starts 2024-09-10T19:32:45.378 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:32:45 smithi183 ceph-mon[29736]: 6.1f deep-scrub ok 2024-09-10T19:32:45.379 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:32:45 smithi183 ceph-mon[29736]: pgmap v1046: 142 pgs: 142 active+clean; 582 KiB data, 5.8 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-10T19:32:45.379 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:32:45 smithi183 ceph-mon[29736]: 4.6 scrub starts 2024-09-10T19:32:45.379 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:32:45 smithi183 ceph-mon[29736]: 2.8 scrub starts 2024-09-10T19:32:45.379 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:32:45 smithi183 ceph-mon[29736]: 2.8 scrub ok 2024-09-10T19:32:45.379 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:32:45 smithi183 ceph-mon[29736]: from='client.? 172.21.15.183:0/1705598174' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-10T19:32:45.379 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:32:45 smithi183 ceph-mon[41562]: 6.1f deep-scrub starts 2024-09-10T19:32:45.379 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:32:45 smithi183 ceph-mon[41562]: 6.1f deep-scrub ok 2024-09-10T19:32:45.379 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:32:45 smithi183 ceph-mon[41562]: pgmap v1046: 142 pgs: 142 active+clean; 582 KiB data, 5.8 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-10T19:32:45.379 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:32:45 smithi183 ceph-mon[41562]: 4.6 scrub starts 2024-09-10T19:32:45.379 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:32:45 smithi183 ceph-mon[41562]: 2.8 scrub starts 2024-09-10T19:32:45.379 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:32:45 smithi183 ceph-mon[41562]: 2.8 scrub ok 2024-09-10T19:32:45.380 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:32:45 smithi183 ceph-mon[41562]: from='client.? 172.21.15.183:0/1705598174' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-10T19:32:45.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:32:45 smithi191 ceph-mon[32884]: 6.1f deep-scrub starts 2024-09-10T19:32:45.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:32:45 smithi191 ceph-mon[32884]: 6.1f deep-scrub ok 2024-09-10T19:32:45.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:32:45 smithi191 ceph-mon[32884]: pgmap v1046: 142 pgs: 142 active+clean; 582 KiB data, 5.8 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-10T19:32:45.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:32:45 smithi191 ceph-mon[32884]: 4.6 scrub starts 2024-09-10T19:32:45.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:32:45 smithi191 ceph-mon[32884]: 2.8 scrub starts 2024-09-10T19:32:45.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:32:45 smithi191 ceph-mon[32884]: 2.8 scrub ok 2024-09-10T19:32:45.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:32:45 smithi191 ceph-mon[32884]: from='client.? 172.21.15.183:0/1705598174' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-10T19:32:45.771 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-09-10T19:32:45.771 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph osd dump --format=json 2024-09-10T19:32:46.016 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:32:46.410 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:32:46 smithi183 ceph-mon[29736]: 4.6 scrub ok 2024-09-10T19:32:46.410 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:32:46 smithi183 ceph-mon[41562]: 4.6 scrub ok 2024-09-10T19:32:46.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:32:46 smithi191 ceph-mon[32884]: 4.6 scrub ok 2024-09-10T19:32:47.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:32:47 smithi183 ceph-mon[29736]: pgmap v1047: 142 pgs: 142 active+clean; 582 KiB data, 5.8 GiB used, 709 GiB / 715 GiB avail; 742 B/s rd, 0 op/s 2024-09-10T19:32:47.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:32:47 smithi183 ceph-mon[29736]: 4.2 scrub starts 2024-09-10T19:32:47.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:32:47 smithi183 ceph-mon[29736]: 4.2 scrub ok 2024-09-10T19:32:47.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:32:47 smithi183 ceph-mon[41562]: pgmap v1047: 142 pgs: 142 active+clean; 582 KiB data, 5.8 GiB used, 709 GiB / 715 GiB avail; 742 B/s rd, 0 op/s 2024-09-10T19:32:47.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:32:47 smithi183 ceph-mon[41562]: 4.2 scrub starts 2024-09-10T19:32:47.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:32:47 smithi183 ceph-mon[41562]: 4.2 scrub ok 2024-09-10T19:32:47.451 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:32:47 smithi191 ceph-mon[32884]: pgmap v1047: 142 pgs: 142 active+clean; 582 KiB data, 5.8 GiB used, 709 GiB / 715 GiB avail; 742 B/s rd, 0 op/s 2024-09-10T19:32:47.451 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:32:47 smithi191 ceph-mon[32884]: 4.2 scrub starts 2024-09-10T19:32:47.451 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:32:47 smithi191 ceph-mon[32884]: 4.2 scrub ok 2024-09-10T19:32:47.715 INFO:journalctl@ceph.mgr.x.smithi191.stdout:Sep 10 19:32:47 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-x[34967]: ::ffff:172.21.15.191 - - [10/Sep/2024:19:32:47] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T19:32:48.060 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-09-10T19:32:48.060 INFO:teuthology.orchestra.run.smithi183.stdout:{"epoch":456,"fsid":"320f6df8-6fa7-11ef-bcea-c7b262605968","created":"2024-09-10T19:05:03.688531+0000","modified":"2024-09-10T19:32:42.141578+0000","last_up_change":"2024-09-10T19:24:22.870873+0000","last_in_change":"2024-09-10T19:31:32.311543+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":21,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-10T19:08:00.456786+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":11,"pg_placement_num":11,"pg_placement_num_target":11,"pg_num_target":11,"pg_num_pending":11,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"271","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"242","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":".rgw.root","create_time":"2024-09-10T19:10:05.993763+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"60","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}},{"pool":3,"pool_name":"datapool","create_time":"2024-09-10T19:10:06.224267+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":3,"pg_placement_num":3,"pg_placement_num_target":3,"pg_num_target":3,"pg_num_pending":3,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"63","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":3,"snap_epoch":63,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":4,"pool_name":"default.rgw.log","create_time":"2024-09-10T19:10:07.958284+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"62","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}},{"pool":5,"pool_name":"default.rgw.control","create_time":"2024-09-10T19:10:09.933596+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"64","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}},{"pool":6,"pool_name":"default.rgw.meta","create_time":"2024-09-10T19:10:11.993712+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"66","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4},"application_metadata":{"rgw":{}}}],"osds":[{"osd":0,"uuid":"9d323010-394a-49aa-ba9b-244d87faf36e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":8,"last_clean_end":262,"up_from":301,"up_thru":431,"down_at":263,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6810","nonce":3346539414},{"type":"v1","addr":"172.21.15.183:6811","nonce":3346539414}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6812","nonce":3346539414},{"type":"v1","addr":"172.21.15.183:6813","nonce":3346539414}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6816","nonce":3346539414},{"type":"v1","addr":"172.21.15.183:6817","nonce":3346539414}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6814","nonce":3346539414},{"type":"v1","addr":"172.21.15.183:6815","nonce":3346539414}]},"public_addr":"172.21.15.183:6811/3346539414","cluster_addr":"172.21.15.183:6813/3346539414","heartbeat_back_addr":"172.21.15.183:6817/3346539414","heartbeat_front_addr":"172.21.15.183:6815/3346539414","state":["exists","up"]},{"osd":1,"uuid":"71a6c5dc-5109-4284-8d20-187d8e7afc9e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":138,"last_clean_end":255,"up_from":276,"up_thru":431,"down_at":256,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6802","nonce":2764394151},{"type":"v1","addr":"172.21.15.183:6803","nonce":2764394151}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6804","nonce":2764394151},{"type":"v1","addr":"172.21.15.183:6805","nonce":2764394151}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6808","nonce":2764394151},{"type":"v1","addr":"172.21.15.183:6809","nonce":2764394151}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6806","nonce":2764394151},{"type":"v1","addr":"172.21.15.183:6807","nonce":2764394151}]},"public_addr":"172.21.15.183:6803/2764394151","cluster_addr":"172.21.15.183:6805/2764394151","heartbeat_back_addr":"172.21.15.183:6809/2764394151","heartbeat_front_addr":"172.21.15.183:6807/2764394151","state":["exists","up"]},{"osd":2,"uuid":"fc310d5b-b89b-4655-9989-7e10ba356509","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":435,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6818","nonce":3730013786},{"type":"v1","addr":"172.21.15.183:6819","nonce":3730013786}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6820","nonce":3730013786},{"type":"v1","addr":"172.21.15.183:6821","nonce":3730013786}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6824","nonce":3730013786},{"type":"v1","addr":"172.21.15.183:6825","nonce":3730013786}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6822","nonce":3730013786},{"type":"v1","addr":"172.21.15.183:6823","nonce":3730013786}]},"public_addr":"172.21.15.183:6819/3730013786","cluster_addr":"172.21.15.183:6821/3730013786","heartbeat_back_addr":"172.21.15.183:6825/3730013786","heartbeat_front_addr":"172.21.15.183:6823/3730013786","state":["exists","up"]},{"osd":3,"uuid":"7fd07a95-ad22-43f5-a968-539552283e89","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":431,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6826","nonce":3522836984},{"type":"v1","addr":"172.21.15.183:6827","nonce":3522836984}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6828","nonce":3522836984},{"type":"v1","addr":"172.21.15.183:6829","nonce":3522836984}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6832","nonce":3522836984},{"type":"v1","addr":"172.21.15.183:6833","nonce":3522836984}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6830","nonce":3522836984},{"type":"v1","addr":"172.21.15.183:6831","nonce":3522836984}]},"public_addr":"172.21.15.183:6827/3522836984","cluster_addr":"172.21.15.183:6829/3522836984","heartbeat_back_addr":"172.21.15.183:6833/3522836984","heartbeat_front_addr":"172.21.15.183:6831/3522836984","state":["exists","up"]},{"osd":4,"uuid":"631083fb-e503-44da-9b6b-154e88539ee7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":435,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6800","nonce":2674828539},{"type":"v1","addr":"172.21.15.191:6801","nonce":2674828539}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6802","nonce":2674828539},{"type":"v1","addr":"172.21.15.191:6803","nonce":2674828539}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6806","nonce":2674828539},{"type":"v1","addr":"172.21.15.191:6807","nonce":2674828539}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6804","nonce":2674828539},{"type":"v1","addr":"172.21.15.191:6805","nonce":2674828539}]},"public_addr":"172.21.15.191:6801/2674828539","cluster_addr":"172.21.15.191:6803/2674828539","heartbeat_back_addr":"172.21.15.191:6807/2674828539","heartbeat_front_addr":"172.21.15.191:6805/2674828539","state":["exists","up"]},{"osd":5,"uuid":"6b963695-3eca-4e6e-9207-05b8010ebff9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":431,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6808","nonce":681733378},{"type":"v1","addr":"172.21.15.191:6809","nonce":681733378}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6810","nonce":681733378},{"type":"v1","addr":"172.21.15.191:6811","nonce":681733378}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6814","nonce":681733378},{"type":"v1","addr":"172.21.15.191:6815","nonce":681733378}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6812","nonce":681733378},{"type":"v1","addr":"172.21.15.191:6813","nonce":681733378}]},"public_addr":"172.21.15.191:6809/681733378","cluster_addr":"172.21.15.191:6811/681733378","heartbeat_back_addr":"172.21.15.191:6815/681733378","heartbeat_front_addr":"172.21.15.191:6813/681733378","state":["exists","up"]},{"osd":6,"uuid":"2dc9754f-6aef-4e19-8605-2f1591921811","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":431,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6816","nonce":2825813477},{"type":"v1","addr":"172.21.15.191:6817","nonce":2825813477}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6818","nonce":2825813477},{"type":"v1","addr":"172.21.15.191:6819","nonce":2825813477}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6822","nonce":2825813477},{"type":"v1","addr":"172.21.15.191:6823","nonce":2825813477}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6820","nonce":2825813477},{"type":"v1","addr":"172.21.15.191:6821","nonce":2825813477}]},"public_addr":"172.21.15.191:6817/2825813477","cluster_addr":"172.21.15.191:6819/2825813477","heartbeat_back_addr":"172.21.15.191:6823/2825813477","heartbeat_front_addr":"172.21.15.191:6821/2825813477","state":["exists","up"]},{"osd":7,"uuid":"1a9a299e-d5ba-4b9a-85d5-7f6a5bc754e1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":431,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6824","nonce":1422610021},{"type":"v1","addr":"172.21.15.191:6825","nonce":1422610021}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6826","nonce":1422610021},{"type":"v1","addr":"172.21.15.191:6827","nonce":1422610021}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6830","nonce":1422610021},{"type":"v1","addr":"172.21.15.191:6831","nonce":1422610021}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6828","nonce":1422610021},{"type":"v1","addr":"172.21.15.191:6829","nonce":1422610021}]},"public_addr":"172.21.15.191:6825/1422610021","cluster_addr":"172.21.15.191:6827/1422610021","heartbeat_back_addr":"172.21.15.191:6831/1422610021","heartbeat_front_addr":"172.21.15.191:6829/1422610021","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T19:32:18.748197+0000","dead_epoch":262},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T19:31:29.319415+0000","dead_epoch":255},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T19:32:15.944103+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T19:31:44.805134+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T19:32:13.817432+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T19:31:25.054746+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T19:32:41.720207+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T19:32:40.876679+0000","dead_epoch":0}],"pg_upmap":[{"pgid":"2.13","osds":[2,0,5]},{"pgid":"4.a","osds":[3,2,1]}],"pg_upmap_items":[{"pgid":"2.8","mappings":[{"from":0,"to":5},{"from":1,"to":3},{"from":7,"to":4}]},{"pgid":"2.14","mappings":[{"from":5,"to":6}]},{"pgid":"2.18","mappings":[{"from":7,"to":0},{"from":5,"to":4}]},{"pgid":"4.0","mappings":[{"from":3,"to":2}]},{"pgid":"4.b","mappings":[{"from":1,"to":5}]},{"pgid":"4.e","mappings":[{"from":1,"to":5}]},{"pgid":"4.18","mappings":[{"from":1,"to":2}]},{"pgid":"5.15","mappings":[{"from":1,"to":3}]},{"pgid":"5.19","mappings":[{"from":1,"to":6}]},{"pgid":"6.1b","mappings":[{"from":3,"to":0}]}],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.183:6801/3311783362":"2024-09-11T19:10:43.576943+0000","172.21.15.183:0/4088553922":"2024-09-11T19:10:43.576943+0000","172.21.15.183:0/3528769414":"2024-09-11T19:05:48.153879+0000","172.21.15.183:0/1766036658":"2024-09-11T19:05:48.153879+0000","172.21.15.183:0/3044170088":"2024-09-11T19:05:25.702354+0000","172.21.15.183:0/3037333778":"2024-09-11T19:05:25.702354+0000","172.21.15.183:6801/3747320570":"2024-09-11T19:05:25.702354+0000","172.21.15.183:0/3982386906":"2024-09-11T19:10:43.576943+0000","172.21.15.183:0/2436014455":"2024-09-11T19:05:25.702354+0000","172.21.15.183:6801/659745765":"2024-09-11T19:05:48.153879+0000","172.21.15.183:6800/3747320570":"2024-09-11T19:05:25.702354+0000","172.21.15.183:0/1652576282":"2024-09-11T19:10:43.576943+0000","172.21.15.183:6800/3311783362":"2024-09-11T19:10:43.576943+0000","172.21.15.183:0/1012899549":"2024-09-11T19:05:48.153879+0000","172.21.15.183:6800/659745765":"2024-09-11T19:05:48.153879+0000","172.21.15.183:0/412044400":"2024-09-11T19:10:43.576943+0000","172.21.15.183:0/3636171401":"2024-09-11T19:10:43.576943+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-10T19:32:48.419 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:32:48 smithi183 ceph-mon[29736]: from='client.? 172.21.15.183:0/1533059861' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-10T19:32:48.419 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:32:48 smithi183 ceph-mon[41562]: from='client.? 172.21.15.183:0/1533059861' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-10T19:32:48.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:32:48 smithi191 ceph-mon[32884]: from='client.? 172.21.15.183:0/1533059861' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-10T19:32:48.859 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph tell osd.0 flush_pg_stats 2024-09-10T19:32:48.860 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph tell osd.1 flush_pg_stats 2024-09-10T19:32:48.860 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph tell osd.2 flush_pg_stats 2024-09-10T19:32:48.860 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph tell osd.3 flush_pg_stats 2024-09-10T19:32:48.861 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph tell osd.4 flush_pg_stats 2024-09-10T19:32:48.861 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph tell osd.5 flush_pg_stats 2024-09-10T19:32:48.862 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph tell osd.6 flush_pg_stats 2024-09-10T19:32:48.862 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph tell osd.7 flush_pg_stats 2024-09-10T19:32:49.388 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:32:49.391 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:32:49.392 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:32:49.395 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:32:49.404 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:32:49.408 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:32:49.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:32:49 smithi183 ceph-mon[29736]: pgmap v1048: 142 pgs: 142 active+clean; 582 KiB data, 5.8 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:32:49.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:32:49 smithi183 ceph-mon[29736]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T19:32:49.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:32:49 smithi183 ceph-mon[41562]: pgmap v1048: 142 pgs: 142 active+clean; 582 KiB data, 5.8 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:32:49.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:32:49 smithi183 ceph-mon[41562]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T19:32:49.433 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:32:49.434 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:32:49.469 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:32:49 smithi191 ceph-mon[32884]: pgmap v1048: 142 pgs: 142 active+clean; 582 KiB data, 5.8 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:32:49.469 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:32:49 smithi191 ceph-mon[32884]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T19:32:50.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:32:50 smithi183 ceph-mon[29736]: 4.11 scrub starts 2024-09-10T19:32:50.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:32:50 smithi183 ceph-mon[29736]: 4.11 scrub ok 2024-09-10T19:32:50.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:32:50 smithi183 ceph-mon[29736]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T19:32:50.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:32:50 smithi183 ceph-mon[29736]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T19:32:50.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:32:50 smithi183 ceph-mon[29736]: from='mgr.14556 ' entity='mgr.y' 2024-09-10T19:32:50.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:32:50 smithi183 ceph-mon[29736]: 2.1c deep-scrub starts 2024-09-10T19:32:50.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:32:50 smithi183 ceph-mon[29736]: 2.1c deep-scrub ok 2024-09-10T19:32:50.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:32:50 smithi183 ceph-mon[41562]: 4.11 scrub starts 2024-09-10T19:32:50.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:32:50 smithi183 ceph-mon[41562]: 4.11 scrub ok 2024-09-10T19:32:50.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:32:50 smithi183 ceph-mon[41562]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T19:32:50.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:32:50 smithi183 ceph-mon[41562]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T19:32:50.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:32:50 smithi183 ceph-mon[41562]: from='mgr.14556 ' entity='mgr.y' 2024-09-10T19:32:50.433 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:32:50 smithi183 ceph-mon[41562]: 2.1c deep-scrub starts 2024-09-10T19:32:50.433 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:32:50 smithi183 ceph-mon[41562]: 2.1c deep-scrub ok 2024-09-10T19:32:50.466 INFO:journalctl@ceph.iscsi.iscsi.a.smithi191.stdout:Sep 10 19:32:50 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-iscsi-iscsi-a[61816]: debug there is no tcmu-runner data available 2024-09-10T19:32:50.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:32:50 smithi191 ceph-mon[32884]: 4.11 scrub starts 2024-09-10T19:32:50.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:32:50 smithi191 ceph-mon[32884]: 4.11 scrub ok 2024-09-10T19:32:50.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:32:50 smithi191 ceph-mon[32884]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-10T19:32:50.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:32:50 smithi191 ceph-mon[32884]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-10T19:32:50.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:32:50 smithi191 ceph-mon[32884]: from='mgr.14556 ' entity='mgr.y' 2024-09-10T19:32:50.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:32:50 smithi191 ceph-mon[32884]: 2.1c deep-scrub starts 2024-09-10T19:32:50.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:32:50 smithi191 ceph-mon[32884]: 2.1c deep-scrub ok 2024-09-10T19:32:51.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:32:51 smithi183 ceph-mon[41562]: pgmap v1049: 142 pgs: 142 active+clean; 582 KiB data, 5.8 GiB used, 709 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-10T19:32:51.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:32:51 smithi183 ceph-mon[41562]: from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:32:51.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:32:51 smithi183 ceph-mon[41562]: 2.16 scrub starts 2024-09-10T19:32:51.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:32:51 smithi183 ceph-mon[41562]: 2.16 scrub ok 2024-09-10T19:32:51.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:32:51 smithi183 ceph-mon[41562]: 6.1a deep-scrub starts 2024-09-10T19:32:51.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:32:51 smithi183 ceph-mon[41562]: 6.1a deep-scrub ok 2024-09-10T19:32:51.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:32:51 smithi183 ceph-mon[29736]: pgmap v1049: 142 pgs: 142 active+clean; 582 KiB data, 5.8 GiB used, 709 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-10T19:32:51.433 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:32:51 smithi183 ceph-mon[29736]: from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:32:51.433 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:32:51 smithi183 ceph-mon[29736]: 2.16 scrub starts 2024-09-10T19:32:51.433 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:32:51 smithi183 ceph-mon[29736]: 2.16 scrub ok 2024-09-10T19:32:51.433 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:32:51 smithi183 ceph-mon[29736]: 6.1a deep-scrub starts 2024-09-10T19:32:51.433 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:32:51 smithi183 ceph-mon[29736]: 6.1a deep-scrub ok 2024-09-10T19:32:51.433 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Sep 10 19:32:51 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-y[29989]: ::ffff:172.21.15.191 - - [10/Sep/2024:19:32:51] "GET /metrics HTTP/1.1" 200 222404 "" "Prometheus/2.43.0" 2024-09-10T19:32:51.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:32:51 smithi191 ceph-mon[32884]: pgmap v1049: 142 pgs: 142 active+clean; 582 KiB data, 5.8 GiB used, 709 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-10T19:32:51.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:32:51 smithi191 ceph-mon[32884]: from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:32:51.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:32:51 smithi191 ceph-mon[32884]: 2.16 scrub starts 2024-09-10T19:32:51.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:32:51 smithi191 ceph-mon[32884]: 2.16 scrub ok 2024-09-10T19:32:51.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:32:51 smithi191 ceph-mon[32884]: 6.1a deep-scrub starts 2024-09-10T19:32:51.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:32:51 smithi191 ceph-mon[32884]: 6.1a deep-scrub ok 2024-09-10T19:32:52.175 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:32:52 smithi183 ceph-mon[41562]: 6.10 scrub starts 2024-09-10T19:32:52.175 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:32:52 smithi183 ceph-mon[41562]: 6.10 scrub ok 2024-09-10T19:32:52.175 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:32:52 smithi183 ceph-mon[41562]: 2.1a scrub starts 2024-09-10T19:32:52.175 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:32:52 smithi183 ceph-mon[41562]: 2.1a scrub ok 2024-09-10T19:32:52.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:32:52 smithi183 ceph-mon[29736]: 6.10 scrub starts 2024-09-10T19:32:52.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:32:52 smithi183 ceph-mon[29736]: 6.10 scrub ok 2024-09-10T19:32:52.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:32:52 smithi183 ceph-mon[29736]: 2.1a scrub starts 2024-09-10T19:32:52.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:32:52 smithi183 ceph-mon[29736]: 2.1a scrub ok 2024-09-10T19:32:52.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:32:52 smithi191 ceph-mon[32884]: 6.10 scrub starts 2024-09-10T19:32:52.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:32:52 smithi191 ceph-mon[32884]: 6.10 scrub ok 2024-09-10T19:32:52.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:32:52 smithi191 ceph-mon[32884]: 2.1a scrub starts 2024-09-10T19:32:52.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:32:52 smithi191 ceph-mon[32884]: 2.1a scrub ok 2024-09-10T19:32:53.249 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:32:53 smithi183 ceph-mon[41562]: 4.16 scrub starts 2024-09-10T19:32:53.249 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:32:53 smithi183 ceph-mon[41562]: 4.16 scrub ok 2024-09-10T19:32:53.249 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:32:53 smithi183 ceph-mon[41562]: 5.19 scrub starts 2024-09-10T19:32:53.249 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:32:53 smithi183 ceph-mon[41562]: 5.19 scrub ok 2024-09-10T19:32:53.250 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:32:53 smithi183 ceph-mon[41562]: pgmap v1050: 142 pgs: 142 active+clean; 582 KiB data, 5.8 GiB used, 709 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-10T19:32:53.250 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:32:53 smithi183 ceph-mon[29736]: 4.16 scrub starts 2024-09-10T19:32:53.250 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:32:53 smithi183 ceph-mon[29736]: 4.16 scrub ok 2024-09-10T19:32:53.250 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:32:53 smithi183 ceph-mon[29736]: 5.19 scrub starts 2024-09-10T19:32:53.250 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:32:53 smithi183 ceph-mon[29736]: 5.19 scrub ok 2024-09-10T19:32:53.250 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:32:53 smithi183 ceph-mon[29736]: pgmap v1050: 142 pgs: 142 active+clean; 582 KiB data, 5.8 GiB used, 709 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-10T19:32:53.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:32:53 smithi191 ceph-mon[32884]: 4.16 scrub starts 2024-09-10T19:32:53.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:32:53 smithi191 ceph-mon[32884]: 4.16 scrub ok 2024-09-10T19:32:53.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:32:53 smithi191 ceph-mon[32884]: 5.19 scrub starts 2024-09-10T19:32:53.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:32:53 smithi191 ceph-mon[32884]: 5.19 scrub ok 2024-09-10T19:32:53.467 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:32:53 smithi191 ceph-mon[32884]: pgmap v1050: 142 pgs: 142 active+clean; 582 KiB data, 5.8 GiB used, 709 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-10T19:32:54.465 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:32:54 smithi191 ceph-mon[32884]: purged_snaps scrub starts 2024-09-10T19:32:54.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:32:54 smithi191 ceph-mon[32884]: purged_snaps scrub ok 2024-09-10T19:32:54.591 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:32:54 smithi183 ceph-mon[29736]: purged_snaps scrub starts 2024-09-10T19:32:54.592 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:32:54 smithi183 ceph-mon[29736]: purged_snaps scrub ok 2024-09-10T19:32:54.592 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:32:54 smithi183 ceph-mon[41562]: purged_snaps scrub starts 2024-09-10T19:32:54.592 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:32:54 smithi183 ceph-mon[41562]: purged_snaps scrub ok 2024-09-10T19:32:55.263 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:32:55 smithi183 ceph-mon[29736]: pgmap v1051: 142 pgs: 142 active+clean; 582 KiB data, 5.8 GiB used, 709 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-09-10T19:32:55.264 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:32:55 smithi183 ceph-mon[29736]: osdmap e457: 8 total, 8 up, 8 in 2024-09-10T19:32:55.264 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:32:55 smithi183 ceph-mon[29736]: 1.a scrub starts 2024-09-10T19:32:55.264 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:32:55 smithi183 ceph-mon[29736]: 1.a scrub ok 2024-09-10T19:32:55.264 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:32:55 smithi183 ceph-mon[41562]: pgmap v1051: 142 pgs: 142 active+clean; 582 KiB data, 5.8 GiB used, 709 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-09-10T19:32:55.264 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:32:55 smithi183 ceph-mon[41562]: osdmap e457: 8 total, 8 up, 8 in 2024-09-10T19:32:55.264 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:32:55 smithi183 ceph-mon[41562]: 1.a scrub starts 2024-09-10T19:32:55.264 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:32:55 smithi183 ceph-mon[41562]: 1.a scrub ok 2024-09-10T19:32:55.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:32:55 smithi191 ceph-mon[32884]: pgmap v1051: 142 pgs: 142 active+clean; 582 KiB data, 5.8 GiB used, 709 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-09-10T19:32:55.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:32:55 smithi191 ceph-mon[32884]: osdmap e457: 8 total, 8 up, 8 in 2024-09-10T19:32:55.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:32:55 smithi191 ceph-mon[32884]: 1.a scrub starts 2024-09-10T19:32:55.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:32:55 smithi191 ceph-mon[32884]: 1.a scrub ok 2024-09-10T19:32:56.398 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:32:56 smithi183 ceph-mon[29736]: 5.3 scrub starts 2024-09-10T19:32:56.398 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:32:56 smithi183 ceph-mon[29736]: 5.3 scrub ok 2024-09-10T19:32:56.398 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:32:56 smithi183 ceph-mon[29736]: 2.13 deep-scrub starts 2024-09-10T19:32:56.398 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:32:56 smithi183 ceph-mon[29736]: 2.13 deep-scrub ok 2024-09-10T19:32:56.399 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:32:56 smithi183 ceph-mon[41562]: 5.3 scrub starts 2024-09-10T19:32:56.399 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:32:56 smithi183 ceph-mon[41562]: 5.3 scrub ok 2024-09-10T19:32:56.399 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:32:56 smithi183 ceph-mon[41562]: 2.13 deep-scrub starts 2024-09-10T19:32:56.399 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:32:56 smithi183 ceph-mon[41562]: 2.13 deep-scrub ok 2024-09-10T19:32:56.465 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:32:56 smithi191 ceph-mon[32884]: 5.3 scrub starts 2024-09-10T19:32:56.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:32:56 smithi191 ceph-mon[32884]: 5.3 scrub ok 2024-09-10T19:32:56.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:32:56 smithi191 ceph-mon[32884]: 2.13 deep-scrub starts 2024-09-10T19:32:56.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:32:56 smithi191 ceph-mon[32884]: 2.13 deep-scrub ok 2024-09-10T19:32:57.431 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:32:57 smithi183 ceph-mon[41562]: pgmap v1053: 142 pgs: 142 active+clean; 582 KiB data, 5.8 GiB used, 709 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-10T19:32:57.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:32:57 smithi183 ceph-mon[29736]: pgmap v1053: 142 pgs: 142 active+clean; 582 KiB data, 5.8 GiB used, 709 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-10T19:32:57.451 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:32:57 smithi191 ceph-mon[32884]: pgmap v1053: 142 pgs: 142 active+clean; 582 KiB data, 5.8 GiB used, 709 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-10T19:32:57.716 INFO:journalctl@ceph.mgr.x.smithi191.stdout:Sep 10 19:32:57 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-x[34967]: ::ffff:172.21.15.191 - - [10/Sep/2024:19:32:57] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T19:32:58.431 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:32:58 smithi183 ceph-mon[41562]: 5.1b scrub starts 2024-09-10T19:32:58.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:32:58 smithi183 ceph-mon[41562]: 5.1b scrub ok 2024-09-10T19:32:58.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:32:58 smithi183 ceph-mon[41562]: 1.5 scrub starts 2024-09-10T19:32:58.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:32:58 smithi183 ceph-mon[41562]: 1.5 scrub ok 2024-09-10T19:32:58.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:32:58 smithi183 ceph-mon[29736]: 5.1b scrub starts 2024-09-10T19:32:58.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:32:58 smithi183 ceph-mon[29736]: 5.1b scrub ok 2024-09-10T19:32:58.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:32:58 smithi183 ceph-mon[29736]: 1.5 scrub starts 2024-09-10T19:32:58.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:32:58 smithi183 ceph-mon[29736]: 1.5 scrub ok 2024-09-10T19:32:58.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:32:58 smithi191 ceph-mon[32884]: 5.1b scrub starts 2024-09-10T19:32:58.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:32:58 smithi191 ceph-mon[32884]: 5.1b scrub ok 2024-09-10T19:32:58.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:32:58 smithi191 ceph-mon[32884]: 1.5 scrub starts 2024-09-10T19:32:58.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:32:58 smithi191 ceph-mon[32884]: 1.5 scrub ok 2024-09-10T19:32:59.238 INFO:teuthology.orchestra.run.smithi183.stdout:167503724832 2024-09-10T19:32:59.239 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph osd last-stat-seq osd.5 2024-09-10T19:32:59.465 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:32:59 smithi183 ceph-mon[29736]: pgmap v1054: 142 pgs: 142 active+clean; 582 KiB data, 5.8 GiB used, 709 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-10T19:32:59.465 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:32:59 smithi183 ceph-mon[29736]: purged_snaps scrub starts 2024-09-10T19:32:59.465 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:32:59 smithi183 ceph-mon[29736]: purged_snaps scrub ok 2024-09-10T19:32:59.465 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:32:59 smithi183 ceph-mon[29736]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:32:59.466 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:32:59 smithi183 ceph-mon[41562]: pgmap v1054: 142 pgs: 142 active+clean; 582 KiB data, 5.8 GiB used, 709 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-10T19:32:59.466 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:32:59 smithi183 ceph-mon[41562]: purged_snaps scrub starts 2024-09-10T19:32:59.466 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:32:59 smithi183 ceph-mon[41562]: purged_snaps scrub ok 2024-09-10T19:32:59.466 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:32:59 smithi183 ceph-mon[41562]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:32:59.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:32:59 smithi191 ceph-mon[32884]: pgmap v1054: 142 pgs: 142 active+clean; 582 KiB data, 5.8 GiB used, 709 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-10T19:32:59.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:32:59 smithi191 ceph-mon[32884]: purged_snaps scrub starts 2024-09-10T19:32:59.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:32:59 smithi191 ceph-mon[32884]: purged_snaps scrub ok 2024-09-10T19:32:59.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:32:59 smithi191 ceph-mon[32884]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:33:00.682 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:33:00 smithi183 ceph-mon[29736]: osdmap e458: 8 total, 8 up, 8 in 2024-09-10T19:33:00.682 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:33:00 smithi183 ceph-mon[29736]: 4.7 scrub starts 2024-09-10T19:33:00.682 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:33:00 smithi183 ceph-mon[29736]: 4.7 scrub ok 2024-09-10T19:33:00.682 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:33:00 smithi183 ceph-mon[41562]: osdmap e458: 8 total, 8 up, 8 in 2024-09-10T19:33:00.682 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:33:00 smithi183 ceph-mon[41562]: 4.7 scrub starts 2024-09-10T19:33:00.682 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:33:00 smithi183 ceph-mon[41562]: 4.7 scrub ok 2024-09-10T19:33:00.715 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:33:00 smithi191 ceph-mon[32884]: osdmap e458: 8 total, 8 up, 8 in 2024-09-10T19:33:00.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:33:00 smithi191 ceph-mon[32884]: 4.7 scrub starts 2024-09-10T19:33:00.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:33:00 smithi191 ceph-mon[32884]: 4.7 scrub ok 2024-09-10T19:33:00.716 INFO:journalctl@ceph.iscsi.iscsi.a.smithi191.stdout:Sep 10 19:33:00 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-iscsi-iscsi-a[61816]: debug there is no tcmu-runner data available 2024-09-10T19:33:01.316 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:33:01 smithi183 ceph-mon[29736]: pgmap v1056: 142 pgs: 142 active+clean; 582 KiB data, 5.8 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:33:01.316 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:33:01 smithi183 ceph-mon[29736]: from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:33:01.316 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:33:01 smithi183 ceph-mon[41562]: pgmap v1056: 142 pgs: 142 active+clean; 582 KiB data, 5.8 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:33:01.316 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:33:01 smithi183 ceph-mon[41562]: from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:33:01.572 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Sep 10 19:33:01 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-y[29989]: ::ffff:172.21.15.191 - - [10/Sep/2024:19:33:01] "GET /metrics HTTP/1.1" 200 222384 "" "Prometheus/2.43.0" 2024-09-10T19:33:01.715 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:33:01 smithi191 ceph-mon[32884]: pgmap v1056: 142 pgs: 142 active+clean; 582 KiB data, 5.8 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:33:01.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:33:01 smithi191 ceph-mon[32884]: from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:33:02.310 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:33:02 smithi183 ceph-mon[29736]: 6.13 scrub starts 2024-09-10T19:33:02.310 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:33:02 smithi183 ceph-mon[29736]: 6.13 scrub ok 2024-09-10T19:33:02.310 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:33:02 smithi183 ceph-mon[29736]: purged_snaps scrub starts 2024-09-10T19:33:02.310 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:33:02 smithi183 ceph-mon[29736]: purged_snaps scrub ok 2024-09-10T19:33:02.310 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:33:02 smithi183 ceph-mon[29736]: 4.0 scrub starts 2024-09-10T19:33:02.310 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:33:02 smithi183 ceph-mon[29736]: 4.0 scrub ok 2024-09-10T19:33:02.311 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:33:02 smithi183 ceph-mon[29736]: 1.0 scrub starts 2024-09-10T19:33:02.311 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:33:02 smithi183 ceph-mon[29736]: 1.0 scrub ok 2024-09-10T19:33:02.311 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:33:02 smithi183 ceph-mon[41562]: 6.13 scrub starts 2024-09-10T19:33:02.311 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:33:02 smithi183 ceph-mon[41562]: 6.13 scrub ok 2024-09-10T19:33:02.311 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:33:02 smithi183 ceph-mon[41562]: purged_snaps scrub starts 2024-09-10T19:33:02.311 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:33:02 smithi183 ceph-mon[41562]: purged_snaps scrub ok 2024-09-10T19:33:02.312 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:33:02 smithi183 ceph-mon[41562]: 4.0 scrub starts 2024-09-10T19:33:02.312 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:33:02 smithi183 ceph-mon[41562]: 4.0 scrub ok 2024-09-10T19:33:02.312 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:33:02 smithi183 ceph-mon[41562]: 1.0 scrub starts 2024-09-10T19:33:02.312 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:33:02 smithi183 ceph-mon[41562]: 1.0 scrub ok 2024-09-10T19:33:02.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:33:02 smithi191 ceph-mon[32884]: 6.13 scrub starts 2024-09-10T19:33:02.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:33:02 smithi191 ceph-mon[32884]: 6.13 scrub ok 2024-09-10T19:33:02.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:33:02 smithi191 ceph-mon[32884]: purged_snaps scrub starts 2024-09-10T19:33:02.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:33:02 smithi191 ceph-mon[32884]: purged_snaps scrub ok 2024-09-10T19:33:02.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:33:02 smithi191 ceph-mon[32884]: 4.0 scrub starts 2024-09-10T19:33:02.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:33:02 smithi191 ceph-mon[32884]: 4.0 scrub ok 2024-09-10T19:33:02.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:33:02 smithi191 ceph-mon[32884]: 1.0 scrub starts 2024-09-10T19:33:02.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:33:02 smithi191 ceph-mon[32884]: 1.0 scrub ok 2024-09-10T19:33:03.681 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:33:03 smithi183 ceph-mon[41562]: 4.3 scrub starts 2024-09-10T19:33:03.682 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:33:03 smithi183 ceph-mon[41562]: 4.3 scrub ok 2024-09-10T19:33:03.682 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:33:03 smithi183 ceph-mon[41562]: 1.6 scrub starts 2024-09-10T19:33:03.682 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:33:03 smithi183 ceph-mon[41562]: 1.6 scrub ok 2024-09-10T19:33:03.682 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:33:03 smithi183 ceph-mon[41562]: pgmap v1057: 142 pgs: 142 active+clean; 582 KiB data, 5.8 GiB used, 709 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-10T19:33:03.682 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:33:03 smithi183 ceph-mon[41562]: osdmap e459: 8 total, 8 up, 8 in 2024-09-10T19:33:03.682 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:33:03 smithi183 ceph-mon[29736]: 4.3 scrub starts 2024-09-10T19:33:03.682 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:33:03 smithi183 ceph-mon[29736]: 4.3 scrub ok 2024-09-10T19:33:03.682 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:33:03 smithi183 ceph-mon[29736]: 1.6 scrub starts 2024-09-10T19:33:03.683 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:33:03 smithi183 ceph-mon[29736]: 1.6 scrub ok 2024-09-10T19:33:03.683 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:33:03 smithi183 ceph-mon[29736]: pgmap v1057: 142 pgs: 142 active+clean; 582 KiB data, 5.8 GiB used, 709 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-10T19:33:03.683 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:33:03 smithi183 ceph-mon[29736]: osdmap e459: 8 total, 8 up, 8 in 2024-09-10T19:33:03.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:33:03 smithi191 ceph-mon[32884]: 4.3 scrub starts 2024-09-10T19:33:03.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:33:03 smithi191 ceph-mon[32884]: 4.3 scrub ok 2024-09-10T19:33:03.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:33:03 smithi191 ceph-mon[32884]: 1.6 scrub starts 2024-09-10T19:33:03.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:33:03 smithi191 ceph-mon[32884]: 1.6 scrub ok 2024-09-10T19:33:03.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:33:03 smithi191 ceph-mon[32884]: pgmap v1057: 142 pgs: 142 active+clean; 582 KiB data, 5.8 GiB used, 709 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-10T19:33:03.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:33:03 smithi191 ceph-mon[32884]: osdmap e459: 8 total, 8 up, 8 in 2024-09-10T19:33:03.840 INFO:teuthology.orchestra.run.smithi183.stdout:1185410973821 2024-09-10T19:33:03.841 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph osd last-stat-seq osd.1 2024-09-10T19:33:04.431 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:33:04 smithi183 ceph-mon[41562]: 5.1d scrub starts 2024-09-10T19:33:04.431 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:33:04 smithi183 ceph-mon[41562]: 5.1d scrub ok 2024-09-10T19:33:04.431 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:33:04 smithi183 ceph-mon[41562]: 2.10 scrub starts 2024-09-10T19:33:04.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:33:04 smithi183 ceph-mon[41562]: 2.10 scrub ok 2024-09-10T19:33:04.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:33:04 smithi183 ceph-mon[29736]: 5.1d scrub starts 2024-09-10T19:33:04.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:33:04 smithi183 ceph-mon[29736]: 5.1d scrub ok 2024-09-10T19:33:04.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:33:04 smithi183 ceph-mon[29736]: 2.10 scrub starts 2024-09-10T19:33:04.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:33:04 smithi183 ceph-mon[29736]: 2.10 scrub ok 2024-09-10T19:33:04.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:33:04 smithi191 ceph-mon[32884]: 5.1d scrub starts 2024-09-10T19:33:04.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:33:04 smithi191 ceph-mon[32884]: 5.1d scrub ok 2024-09-10T19:33:04.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:33:04 smithi191 ceph-mon[32884]: 2.10 scrub starts 2024-09-10T19:33:04.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:33:04 smithi191 ceph-mon[32884]: 2.10 scrub ok 2024-09-10T19:33:05.322 INFO:teuthology.orchestra.run.smithi183.stdout:77309411632 2024-09-10T19:33:05.322 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph osd last-stat-seq osd.2 2024-09-10T19:33:05.681 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:33:05 smithi183 ceph-mon[41562]: 4.b scrub starts 2024-09-10T19:33:05.682 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:33:05 smithi183 ceph-mon[41562]: 4.b scrub ok 2024-09-10T19:33:05.682 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:33:05 smithi183 ceph-mon[41562]: pgmap v1059: 142 pgs: 142 active+clean; 582 KiB data, 5.8 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:33:05.682 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:33:05 smithi183 ceph-mon[41562]: 6.1 scrub starts 2024-09-10T19:33:05.682 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:33:05 smithi183 ceph-mon[41562]: 6.1 scrub ok 2024-09-10T19:33:05.682 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:33:05 smithi183 ceph-mon[41562]: 5.e scrub starts 2024-09-10T19:33:05.682 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:33:05 smithi183 ceph-mon[41562]: 5.e scrub ok 2024-09-10T19:33:05.682 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:33:05 smithi183 ceph-mon[29736]: 4.b scrub starts 2024-09-10T19:33:05.682 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:33:05 smithi183 ceph-mon[29736]: 4.b scrub ok 2024-09-10T19:33:05.682 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:33:05 smithi183 ceph-mon[29736]: pgmap v1059: 142 pgs: 142 active+clean; 582 KiB data, 5.8 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:33:05.682 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:33:05 smithi183 ceph-mon[29736]: 6.1 scrub starts 2024-09-10T19:33:05.683 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:33:05 smithi183 ceph-mon[29736]: 6.1 scrub ok 2024-09-10T19:33:05.683 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:33:05 smithi183 ceph-mon[29736]: 5.e scrub starts 2024-09-10T19:33:05.683 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:33:05 smithi183 ceph-mon[29736]: 5.e scrub ok 2024-09-10T19:33:05.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:33:05 smithi191 ceph-mon[32884]: 4.b scrub starts 2024-09-10T19:33:05.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:33:05 smithi191 ceph-mon[32884]: 4.b scrub ok 2024-09-10T19:33:05.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:33:05 smithi191 ceph-mon[32884]: pgmap v1059: 142 pgs: 142 active+clean; 582 KiB data, 5.8 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:33:05.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:33:05 smithi191 ceph-mon[32884]: 6.1 scrub starts 2024-09-10T19:33:05.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:33:05 smithi191 ceph-mon[32884]: 6.1 scrub ok 2024-09-10T19:33:05.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:33:05 smithi191 ceph-mon[32884]: 5.e scrub starts 2024-09-10T19:33:05.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:33:05 smithi191 ceph-mon[32884]: 5.e scrub ok 2024-09-10T19:33:05.823 INFO:teuthology.orchestra.run.smithi183.stdout:137438953766 2024-09-10T19:33:05.823 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph osd last-stat-seq osd.4 2024-09-10T19:33:06.655 INFO:teuthology.orchestra.run.smithi183.stdout:1292785156202 2024-09-10T19:33:06.656 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph osd last-stat-seq osd.0 2024-09-10T19:33:07.124 INFO:teuthology.orchestra.run.smithi183.stdout:111669149995 2024-09-10T19:33:07.125 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph osd last-stat-seq osd.3 2024-09-10T19:33:07.351 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:33:07 smithi183 ceph-mon[29736]: pgmap v1060: 142 pgs: 142 active+clean; 582 KiB data, 5.8 GiB used, 709 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-10T19:33:07.351 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:33:07 smithi183 ceph-mon[29736]: 6.11 deep-scrub starts 2024-09-10T19:33:07.351 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:33:07 smithi183 ceph-mon[29736]: 6.11 deep-scrub ok 2024-09-10T19:33:07.352 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:33:07 smithi183 ceph-mon[29736]: 2.4 deep-scrub starts 2024-09-10T19:33:07.352 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:33:07 smithi183 ceph-mon[29736]: 2.4 deep-scrub ok 2024-09-10T19:33:07.352 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:33:07 smithi183 ceph-mon[29736]: 6.8 scrub starts 2024-09-10T19:33:07.352 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:33:07 smithi183 ceph-mon[29736]: 6.8 scrub ok 2024-09-10T19:33:07.352 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:33:07 smithi183 ceph-mon[29736]: 4.e scrub starts 2024-09-10T19:33:07.352 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:33:07 smithi183 ceph-mon[29736]: 4.e scrub ok 2024-09-10T19:33:07.352 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:33:07 smithi183 ceph-mon[41562]: pgmap v1060: 142 pgs: 142 active+clean; 582 KiB data, 5.8 GiB used, 709 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-10T19:33:07.352 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:33:07 smithi183 ceph-mon[41562]: 6.11 deep-scrub starts 2024-09-10T19:33:07.352 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:33:07 smithi183 ceph-mon[41562]: 6.11 deep-scrub ok 2024-09-10T19:33:07.353 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:33:07 smithi183 ceph-mon[41562]: 2.4 deep-scrub starts 2024-09-10T19:33:07.353 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:33:07 smithi183 ceph-mon[41562]: 2.4 deep-scrub ok 2024-09-10T19:33:07.353 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:33:07 smithi183 ceph-mon[41562]: 6.8 scrub starts 2024-09-10T19:33:07.353 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:33:07 smithi183 ceph-mon[41562]: 6.8 scrub ok 2024-09-10T19:33:07.353 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:33:07 smithi183 ceph-mon[41562]: 4.e scrub starts 2024-09-10T19:33:07.353 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:33:07 smithi183 ceph-mon[41562]: 4.e scrub ok 2024-09-10T19:33:07.716 INFO:journalctl@ceph.mgr.x.smithi191.stdout:Sep 10 19:33:07 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-x[34967]: ::ffff:172.21.15.191 - - [10/Sep/2024:19:33:07] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T19:33:07.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:33:07 smithi191 ceph-mon[32884]: pgmap v1060: 142 pgs: 142 active+clean; 582 KiB data, 5.8 GiB used, 709 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2024-09-10T19:33:07.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:33:07 smithi191 ceph-mon[32884]: 6.11 deep-scrub starts 2024-09-10T19:33:07.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:33:07 smithi191 ceph-mon[32884]: 6.11 deep-scrub ok 2024-09-10T19:33:07.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:33:07 smithi191 ceph-mon[32884]: 2.4 deep-scrub starts 2024-09-10T19:33:07.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:33:07 smithi191 ceph-mon[32884]: 2.4 deep-scrub ok 2024-09-10T19:33:07.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:33:07 smithi191 ceph-mon[32884]: 6.8 scrub starts 2024-09-10T19:33:07.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:33:07 smithi191 ceph-mon[32884]: 6.8 scrub ok 2024-09-10T19:33:07.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:33:07 smithi191 ceph-mon[32884]: 4.e scrub starts 2024-09-10T19:33:07.717 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:33:07 smithi191 ceph-mon[32884]: 4.e scrub ok 2024-09-10T19:33:08.279 INFO:teuthology.orchestra.run.smithi183.stdout:201863463197 2024-09-10T19:33:08.279 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph osd last-stat-seq osd.6 2024-09-10T19:33:08.360 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:33:08.360 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:33:08.682 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:33:08 smithi183 ceph-mon[41562]: 5.17 scrub starts 2024-09-10T19:33:08.682 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:33:08 smithi183 ceph-mon[41562]: 5.17 scrub ok 2024-09-10T19:33:08.682 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:33:08 smithi183 ceph-mon[41562]: 2.c scrub starts 2024-09-10T19:33:08.682 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:33:08 smithi183 ceph-mon[41562]: 2.c scrub ok 2024-09-10T19:33:08.682 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:33:08 smithi183 ceph-mon[41562]: 4.1b scrub starts 2024-09-10T19:33:08.682 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:33:08 smithi183 ceph-mon[41562]: 4.1b scrub ok 2024-09-10T19:33:08.683 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:33:08 smithi183 ceph-mon[29736]: 5.17 scrub starts 2024-09-10T19:33:08.683 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:33:08 smithi183 ceph-mon[29736]: 5.17 scrub ok 2024-09-10T19:33:08.683 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:33:08 smithi183 ceph-mon[29736]: 2.c scrub starts 2024-09-10T19:33:08.684 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:33:08 smithi183 ceph-mon[29736]: 2.c scrub ok 2024-09-10T19:33:08.684 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:33:08 smithi183 ceph-mon[29736]: 4.1b scrub starts 2024-09-10T19:33:08.684 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:33:08 smithi183 ceph-mon[29736]: 4.1b scrub ok 2024-09-10T19:33:08.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:33:08 smithi191 ceph-mon[32884]: 5.17 scrub starts 2024-09-10T19:33:08.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:33:08 smithi191 ceph-mon[32884]: 5.17 scrub ok 2024-09-10T19:33:08.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:33:08 smithi191 ceph-mon[32884]: 2.c scrub starts 2024-09-10T19:33:08.717 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:33:08 smithi191 ceph-mon[32884]: 2.c scrub ok 2024-09-10T19:33:08.717 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:33:08 smithi191 ceph-mon[32884]: 4.1b scrub starts 2024-09-10T19:33:08.717 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:33:08 smithi191 ceph-mon[32884]: 4.1b scrub ok 2024-09-10T19:33:08.773 INFO:teuthology.orchestra.run.smithi183.stdout:231928234265 2024-09-10T19:33:08.773 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph osd last-stat-seq osd.7 2024-09-10T19:33:08.855 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:33:09.076 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:33:09.083 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:33:09.085 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:33:09.352 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:33:09 smithi183 ceph-mon[29736]: pgmap v1061: 142 pgs: 142 active+clean; 582 KiB data, 5.8 GiB used, 709 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-10T19:33:09.352 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:33:09 smithi183 ceph-mon[29736]: 2.6 scrub starts 2024-09-10T19:33:09.352 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:33:09 smithi183 ceph-mon[29736]: 2.6 scrub ok 2024-09-10T19:33:09.352 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:33:09 smithi183 ceph-mon[29736]: 4.1 deep-scrub starts 2024-09-10T19:33:09.352 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:33:09 smithi183 ceph-mon[29736]: 4.1 deep-scrub ok 2024-09-10T19:33:09.352 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:33:09 smithi183 ceph-mon[29736]: 5.10 scrub starts 2024-09-10T19:33:09.353 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:33:09 smithi183 ceph-mon[29736]: 5.10 scrub ok 2024-09-10T19:33:09.353 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:33:09 smithi183 ceph-mon[41562]: pgmap v1061: 142 pgs: 142 active+clean; 582 KiB data, 5.8 GiB used, 709 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-10T19:33:09.353 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:33:09 smithi183 ceph-mon[41562]: 2.6 scrub starts 2024-09-10T19:33:09.353 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:33:09 smithi183 ceph-mon[41562]: 2.6 scrub ok 2024-09-10T19:33:09.353 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:33:09 smithi183 ceph-mon[41562]: 4.1 deep-scrub starts 2024-09-10T19:33:09.353 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:33:09 smithi183 ceph-mon[41562]: 4.1 deep-scrub ok 2024-09-10T19:33:09.353 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:33:09 smithi183 ceph-mon[41562]: 5.10 scrub starts 2024-09-10T19:33:09.353 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:33:09 smithi183 ceph-mon[41562]: 5.10 scrub ok 2024-09-10T19:33:09.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:33:09 smithi191 ceph-mon[32884]: pgmap v1061: 142 pgs: 142 active+clean; 582 KiB data, 5.8 GiB used, 709 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-09-10T19:33:09.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:33:09 smithi191 ceph-mon[32884]: 2.6 scrub starts 2024-09-10T19:33:09.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:33:09 smithi191 ceph-mon[32884]: 2.6 scrub ok 2024-09-10T19:33:09.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:33:09 smithi191 ceph-mon[32884]: 4.1 deep-scrub starts 2024-09-10T19:33:09.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:33:09 smithi191 ceph-mon[32884]: 4.1 deep-scrub ok 2024-09-10T19:33:09.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:33:09 smithi191 ceph-mon[32884]: 5.10 scrub starts 2024-09-10T19:33:09.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:33:09 smithi191 ceph-mon[32884]: 5.10 scrub ok 2024-09-10T19:33:10.536 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:33:10 smithi183 ceph-mon[29736]: 2.3 scrub starts 2024-09-10T19:33:10.536 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:33:10 smithi183 ceph-mon[29736]: 2.3 scrub ok 2024-09-10T19:33:10.536 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:33:10 smithi183 ceph-mon[29736]: 2.e scrub starts 2024-09-10T19:33:10.536 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:33:10 smithi183 ceph-mon[29736]: 2.e scrub ok 2024-09-10T19:33:10.536 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:33:10 smithi183 ceph-mon[29736]: 2.0 scrub starts 2024-09-10T19:33:10.536 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:33:10 smithi183 ceph-mon[29736]: 2.0 scrub ok 2024-09-10T19:33:10.537 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:33:10 smithi183 ceph-mon[41562]: 2.3 scrub starts 2024-09-10T19:33:10.537 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:33:10 smithi183 ceph-mon[41562]: 2.3 scrub ok 2024-09-10T19:33:10.537 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:33:10 smithi183 ceph-mon[41562]: 2.e scrub starts 2024-09-10T19:33:10.537 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:33:10 smithi183 ceph-mon[41562]: 2.e scrub ok 2024-09-10T19:33:10.537 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:33:10 smithi183 ceph-mon[41562]: 2.0 scrub starts 2024-09-10T19:33:10.537 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:33:10 smithi183 ceph-mon[41562]: 2.0 scrub ok 2024-09-10T19:33:10.716 INFO:journalctl@ceph.iscsi.iscsi.a.smithi191.stdout:Sep 10 19:33:10 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-iscsi-iscsi-a[61816]: debug there is no tcmu-runner data available 2024-09-10T19:33:10.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:33:10 smithi191 ceph-mon[32884]: 2.3 scrub starts 2024-09-10T19:33:10.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:33:10 smithi191 ceph-mon[32884]: 2.3 scrub ok 2024-09-10T19:33:10.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:33:10 smithi191 ceph-mon[32884]: 2.e scrub starts 2024-09-10T19:33:10.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:33:10 smithi191 ceph-mon[32884]: 2.e scrub ok 2024-09-10T19:33:10.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:33:10 smithi191 ceph-mon[32884]: 2.0 scrub starts 2024-09-10T19:33:10.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:33:10 smithi191 ceph-mon[32884]: 2.0 scrub ok 2024-09-10T19:33:11.681 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:33:11 smithi183 ceph-mon[29736]: pgmap v1062: 142 pgs: 142 active+clean; 582 KiB data, 5.8 GiB used, 709 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-10T19:33:11.681 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:33:11 smithi183 ceph-mon[29736]: from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:33:11.681 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:33:11 smithi183 ceph-mon[29736]: 6.1b scrub starts 2024-09-10T19:33:11.682 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:33:11 smithi183 ceph-mon[29736]: 6.1b scrub ok 2024-09-10T19:33:11.682 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:33:11 smithi183 ceph-mon[29736]: 4.1c scrub starts 2024-09-10T19:33:11.682 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:33:11 smithi183 ceph-mon[29736]: 4.1c scrub ok 2024-09-10T19:33:11.682 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:33:11 smithi183 ceph-mon[41562]: pgmap v1062: 142 pgs: 142 active+clean; 582 KiB data, 5.8 GiB used, 709 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-10T19:33:11.682 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:33:11 smithi183 ceph-mon[41562]: from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:33:11.682 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:33:11 smithi183 ceph-mon[41562]: 6.1b scrub starts 2024-09-10T19:33:11.682 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:33:11 smithi183 ceph-mon[41562]: 6.1b scrub ok 2024-09-10T19:33:11.682 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:33:11 smithi183 ceph-mon[41562]: 4.1c scrub starts 2024-09-10T19:33:11.683 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:33:11 smithi183 ceph-mon[41562]: 4.1c scrub ok 2024-09-10T19:33:11.683 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Sep 10 19:33:11 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-y[29989]: ::ffff:172.21.15.191 - - [10/Sep/2024:19:33:11] "GET /metrics HTTP/1.1" 200 222384 "" "Prometheus/2.43.0" 2024-09-10T19:33:11.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:33:11 smithi191 ceph-mon[32884]: pgmap v1062: 142 pgs: 142 active+clean; 582 KiB data, 5.8 GiB used, 709 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-10T19:33:11.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:33:11 smithi191 ceph-mon[32884]: from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:33:11.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:33:11 smithi191 ceph-mon[32884]: 6.1b scrub starts 2024-09-10T19:33:11.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:33:11 smithi191 ceph-mon[32884]: 6.1b scrub ok 2024-09-10T19:33:11.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:33:11 smithi191 ceph-mon[32884]: 4.1c scrub starts 2024-09-10T19:33:11.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:33:11 smithi191 ceph-mon[32884]: 4.1c scrub ok 2024-09-10T19:33:12.431 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:33:12 smithi183 ceph-mon[41562]: 2.5 scrub starts 2024-09-10T19:33:12.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:33:12 smithi183 ceph-mon[41562]: 2.5 scrub ok 2024-09-10T19:33:12.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:33:12 smithi183 ceph-mon[29736]: 2.5 scrub starts 2024-09-10T19:33:12.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:33:12 smithi183 ceph-mon[29736]: 2.5 scrub ok 2024-09-10T19:33:12.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:33:12 smithi191 ceph-mon[32884]: 2.5 scrub starts 2024-09-10T19:33:12.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:33:12 smithi191 ceph-mon[32884]: 2.5 scrub ok 2024-09-10T19:33:13.536 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:33:13 smithi183 ceph-mon[29736]: pgmap v1063: 142 pgs: 142 active+clean; 582 KiB data, 5.8 GiB used, 709 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-10T19:33:13.536 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:33:13 smithi183 ceph-mon[41562]: pgmap v1063: 142 pgs: 142 active+clean; 582 KiB data, 5.8 GiB used, 709 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-10T19:33:13.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:33:13 smithi191 ceph-mon[32884]: pgmap v1063: 142 pgs: 142 active+clean; 582 KiB data, 5.8 GiB used, 709 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-10T19:33:14.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:33:14 smithi183 ceph-mon[29736]: 4.17 scrub starts 2024-09-10T19:33:14.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:33:14 smithi183 ceph-mon[29736]: 4.17 scrub ok 2024-09-10T19:33:14.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:33:14 smithi183 ceph-mon[29736]: 5.f scrub starts 2024-09-10T19:33:14.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:33:14 smithi183 ceph-mon[29736]: 5.f scrub ok 2024-09-10T19:33:14.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:33:14 smithi183 ceph-mon[29736]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:33:14.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:33:14 smithi183 ceph-mon[29736]: 4.c scrub starts 2024-09-10T19:33:14.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:33:14 smithi183 ceph-mon[29736]: 4.c scrub ok 2024-09-10T19:33:14.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:33:14 smithi183 ceph-mon[41562]: 4.17 scrub starts 2024-09-10T19:33:14.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:33:14 smithi183 ceph-mon[41562]: 4.17 scrub ok 2024-09-10T19:33:14.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:33:14 smithi183 ceph-mon[41562]: 5.f scrub starts 2024-09-10T19:33:14.433 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:33:14 smithi183 ceph-mon[41562]: 5.f scrub ok 2024-09-10T19:33:14.433 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:33:14 smithi183 ceph-mon[41562]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:33:14.433 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:33:14 smithi183 ceph-mon[41562]: 4.c scrub starts 2024-09-10T19:33:14.433 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:33:14 smithi183 ceph-mon[41562]: 4.c scrub ok 2024-09-10T19:33:14.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:33:14 smithi191 ceph-mon[32884]: 4.17 scrub starts 2024-09-10T19:33:14.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:33:14 smithi191 ceph-mon[32884]: 4.17 scrub ok 2024-09-10T19:33:14.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:33:14 smithi191 ceph-mon[32884]: 5.f scrub starts 2024-09-10T19:33:14.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:33:14 smithi191 ceph-mon[32884]: 5.f scrub ok 2024-09-10T19:33:14.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:33:14 smithi191 ceph-mon[32884]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:33:14.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:33:14 smithi191 ceph-mon[32884]: 4.c scrub starts 2024-09-10T19:33:14.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:33:14 smithi191 ceph-mon[32884]: 4.c scrub ok 2024-09-10T19:33:15.388 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:33:15 smithi183 ceph-mon[29736]: 5.11 scrub starts 2024-09-10T19:33:15.388 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:33:15 smithi183 ceph-mon[29736]: 5.11 scrub ok 2024-09-10T19:33:15.388 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:33:15 smithi183 ceph-mon[29736]: pgmap v1064: 142 pgs: 142 active+clean; 582 KiB data, 5.8 GiB used, 709 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-09-10T19:33:15.388 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:33:15 smithi183 ceph-mon[29736]: 5.b scrub starts 2024-09-10T19:33:15.389 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:33:15 smithi183 ceph-mon[29736]: 5.b scrub ok 2024-09-10T19:33:15.389 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:33:15 smithi183 ceph-mon[29736]: 6.5 scrub starts 2024-09-10T19:33:15.389 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:33:15 smithi183 ceph-mon[29736]: 6.5 scrub ok 2024-09-10T19:33:15.389 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:33:15 smithi183 ceph-mon[29736]: from='client.? 172.21.15.183:0/1951954133' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-09-10T19:33:15.389 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:33:15 smithi183 ceph-mon[41562]: 5.11 scrub starts 2024-09-10T19:33:15.389 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:33:15 smithi183 ceph-mon[41562]: 5.11 scrub ok 2024-09-10T19:33:15.389 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:33:15 smithi183 ceph-mon[41562]: pgmap v1064: 142 pgs: 142 active+clean; 582 KiB data, 5.8 GiB used, 709 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-09-10T19:33:15.389 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:33:15 smithi183 ceph-mon[41562]: 5.b scrub starts 2024-09-10T19:33:15.389 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:33:15 smithi183 ceph-mon[41562]: 5.b scrub ok 2024-09-10T19:33:15.389 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:33:15 smithi183 ceph-mon[41562]: 6.5 scrub starts 2024-09-10T19:33:15.389 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:33:15 smithi183 ceph-mon[41562]: 6.5 scrub ok 2024-09-10T19:33:15.390 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:33:15 smithi183 ceph-mon[41562]: from='client.? 172.21.15.183:0/1951954133' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-09-10T19:33:15.536 INFO:teuthology.orchestra.run.smithi183.stdout:137438953768 2024-09-10T19:33:15.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:33:15 smithi191 ceph-mon[32884]: 5.11 scrub starts 2024-09-10T19:33:15.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:33:15 smithi191 ceph-mon[32884]: 5.11 scrub ok 2024-09-10T19:33:15.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:33:15 smithi191 ceph-mon[32884]: pgmap v1064: 142 pgs: 142 active+clean; 582 KiB data, 5.8 GiB used, 709 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2024-09-10T19:33:15.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:33:15 smithi191 ceph-mon[32884]: 5.b scrub starts 2024-09-10T19:33:15.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:33:15 smithi191 ceph-mon[32884]: 5.b scrub ok 2024-09-10T19:33:15.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:33:15 smithi191 ceph-mon[32884]: 6.5 scrub starts 2024-09-10T19:33:15.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:33:15 smithi191 ceph-mon[32884]: 6.5 scrub ok 2024-09-10T19:33:15.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:33:15 smithi191 ceph-mon[32884]: from='client.? 172.21.15.183:0/1951954133' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-09-10T19:33:15.731 INFO:teuthology.orchestra.run.smithi183.stdout:167503724836 2024-09-10T19:33:16.361 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:33:16 smithi183 ceph-mon[29736]: from='client.? 172.21.15.183:0/1164198146' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-09-10T19:33:16.361 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:33:16 smithi183 ceph-mon[29736]: from='client.? 172.21.15.183:0/4043205875' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-09-10T19:33:16.362 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:33:16 smithi183 ceph-mon[41562]: from='client.? 172.21.15.183:0/1164198146' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-09-10T19:33:16.362 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:33:16 smithi183 ceph-mon[41562]: from='client.? 172.21.15.183:0/4043205875' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-09-10T19:33:16.387 INFO:tasks.cephadm.ceph_manager.ceph:need seq 137438953766 got 137438953768 for osd.4 2024-09-10T19:33:16.388 DEBUG:teuthology.parallel:result is None 2024-09-10T19:33:16.486 INFO:teuthology.orchestra.run.smithi183.stdout:1292785156204 2024-09-10T19:33:16.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:33:16 smithi191 ceph-mon[32884]: from='client.? 172.21.15.183:0/1164198146' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-09-10T19:33:16.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:33:16 smithi191 ceph-mon[32884]: from='client.? 172.21.15.183:0/4043205875' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-09-10T19:33:17.076 INFO:tasks.cephadm.ceph_manager.ceph:need seq 1292785156202 got 1292785156204 for osd.0 2024-09-10T19:33:17.077 DEBUG:teuthology.parallel:result is None 2024-09-10T19:33:17.110 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:33:17.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:33:17 smithi183 ceph-mon[29736]: pgmap v1065: 142 pgs: 142 active+clean; 582 KiB data, 5.8 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-10T19:33:17.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:33:17 smithi183 ceph-mon[41562]: pgmap v1065: 142 pgs: 142 active+clean; 582 KiB data, 5.8 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-10T19:33:17.640 INFO:tasks.cephadm.ceph_manager.ceph:need seq 167503724832 got 167503724836 for osd.5 2024-09-10T19:33:17.640 DEBUG:teuthology.parallel:result is None 2024-09-10T19:33:17.715 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:33:17 smithi191 ceph-mon[32884]: pgmap v1065: 142 pgs: 142 active+clean; 582 KiB data, 5.8 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-10T19:33:17.716 INFO:journalctl@ceph.mgr.x.smithi191.stdout:Sep 10 19:33:17 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-x[34967]: ::ffff:172.21.15.191 - - [10/Sep/2024:19:33:17] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T19:33:18.152 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:33:18.667 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:33:18 smithi183 ceph-mon[29736]: 2.19 deep-scrub starts 2024-09-10T19:33:18.667 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:33:18 smithi183 ceph-mon[29736]: 2.19 deep-scrub ok 2024-09-10T19:33:18.667 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:33:18 smithi183 ceph-mon[29736]: 1.7 scrub starts 2024-09-10T19:33:18.667 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:33:18 smithi183 ceph-mon[29736]: 1.7 scrub ok 2024-09-10T19:33:18.667 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:33:18 smithi183 ceph-mon[29736]: 6.19 scrub starts 2024-09-10T19:33:18.667 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:33:18 smithi183 ceph-mon[29736]: 6.19 scrub ok 2024-09-10T19:33:18.667 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:33:18 smithi183 ceph-mon[41562]: 2.19 deep-scrub starts 2024-09-10T19:33:18.667 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:33:18 smithi183 ceph-mon[41562]: 2.19 deep-scrub ok 2024-09-10T19:33:18.668 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:33:18 smithi183 ceph-mon[41562]: 1.7 scrub starts 2024-09-10T19:33:18.668 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:33:18 smithi183 ceph-mon[41562]: 1.7 scrub ok 2024-09-10T19:33:18.668 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:33:18 smithi183 ceph-mon[41562]: 6.19 scrub starts 2024-09-10T19:33:18.668 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:33:18 smithi183 ceph-mon[41562]: 6.19 scrub ok 2024-09-10T19:33:18.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:33:18 smithi191 ceph-mon[32884]: 2.19 deep-scrub starts 2024-09-10T19:33:18.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:33:18 smithi191 ceph-mon[32884]: 2.19 deep-scrub ok 2024-09-10T19:33:18.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:33:18 smithi191 ceph-mon[32884]: 1.7 scrub starts 2024-09-10T19:33:18.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:33:18 smithi191 ceph-mon[32884]: 1.7 scrub ok 2024-09-10T19:33:18.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:33:18 smithi191 ceph-mon[32884]: 6.19 scrub starts 2024-09-10T19:33:18.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:33:18 smithi191 ceph-mon[32884]: 6.19 scrub ok 2024-09-10T19:33:19.400 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:33:19 smithi183 ceph-mon[29736]: 4.10 scrub starts 2024-09-10T19:33:19.400 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:33:19 smithi183 ceph-mon[29736]: 4.10 scrub ok 2024-09-10T19:33:19.400 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:33:19 smithi183 ceph-mon[29736]: pgmap v1066: 142 pgs: 142 active+clean; 582 KiB data, 5.8 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:33:19.400 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:33:19 smithi183 ceph-mon[29736]: 6.1d scrub starts 2024-09-10T19:33:19.401 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:33:19 smithi183 ceph-mon[29736]: 6.1d scrub ok 2024-09-10T19:33:19.401 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:33:19 smithi183 ceph-mon[29736]: 5.1e scrub starts 2024-09-10T19:33:19.401 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:33:19 smithi183 ceph-mon[29736]: 5.1e scrub ok 2024-09-10T19:33:19.401 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:33:19 smithi183 ceph-mon[41562]: 4.10 scrub starts 2024-09-10T19:33:19.401 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:33:19 smithi183 ceph-mon[41562]: 4.10 scrub ok 2024-09-10T19:33:19.401 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:33:19 smithi183 ceph-mon[41562]: pgmap v1066: 142 pgs: 142 active+clean; 582 KiB data, 5.8 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:33:19.401 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:33:19 smithi183 ceph-mon[41562]: 6.1d scrub starts 2024-09-10T19:33:19.401 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:33:19 smithi183 ceph-mon[41562]: 6.1d scrub ok 2024-09-10T19:33:19.401 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:33:19 smithi183 ceph-mon[41562]: 5.1e scrub starts 2024-09-10T19:33:19.401 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:33:19 smithi183 ceph-mon[41562]: 5.1e scrub ok 2024-09-10T19:33:19.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:33:19 smithi191 ceph-mon[32884]: 4.10 scrub starts 2024-09-10T19:33:19.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:33:19 smithi191 ceph-mon[32884]: 4.10 scrub ok 2024-09-10T19:33:19.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:33:19 smithi191 ceph-mon[32884]: pgmap v1066: 142 pgs: 142 active+clean; 582 KiB data, 5.8 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:33:19.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:33:19 smithi191 ceph-mon[32884]: 6.1d scrub starts 2024-09-10T19:33:19.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:33:19 smithi191 ceph-mon[32884]: 6.1d scrub ok 2024-09-10T19:33:19.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:33:19 smithi191 ceph-mon[32884]: 5.1e scrub starts 2024-09-10T19:33:19.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:33:19 smithi191 ceph-mon[32884]: 5.1e scrub ok 2024-09-10T19:33:20.521 INFO:teuthology.orchestra.run.smithi183.stdout:111669149998 2024-09-10T19:33:20.521 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:33:20 smithi183 ceph-mon[29736]: 4.12 scrub starts 2024-09-10T19:33:20.521 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:33:20 smithi183 ceph-mon[29736]: 4.12 scrub ok 2024-09-10T19:33:20.522 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:33:20 smithi183 ceph-mon[29736]: 4.8 scrub starts 2024-09-10T19:33:20.522 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:33:20 smithi183 ceph-mon[29736]: 4.8 scrub ok 2024-09-10T19:33:20.522 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:33:20 smithi183 ceph-mon[41562]: 4.12 scrub starts 2024-09-10T19:33:20.522 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:33:20 smithi183 ceph-mon[41562]: 4.12 scrub ok 2024-09-10T19:33:20.522 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:33:20 smithi183 ceph-mon[41562]: 4.8 scrub starts 2024-09-10T19:33:20.522 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:33:20 smithi183 ceph-mon[41562]: 4.8 scrub ok 2024-09-10T19:33:20.715 INFO:journalctl@ceph.iscsi.iscsi.a.smithi191.stdout:Sep 10 19:33:20 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-iscsi-iscsi-a[61816]: debug there is no tcmu-runner data available 2024-09-10T19:33:20.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:33:20 smithi191 ceph-mon[32884]: 4.12 scrub starts 2024-09-10T19:33:20.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:33:20 smithi191 ceph-mon[32884]: 4.12 scrub ok 2024-09-10T19:33:20.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:33:20 smithi191 ceph-mon[32884]: 4.8 scrub starts 2024-09-10T19:33:20.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:33:20 smithi191 ceph-mon[32884]: 4.8 scrub ok 2024-09-10T19:33:20.804 INFO:teuthology.orchestra.run.smithi183.stdout:1185410973825 2024-09-10T19:33:21.432 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Sep 10 19:33:21 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-y[29989]: ::ffff:172.21.15.191 - - [10/Sep/2024:19:33:21] "GET /metrics HTTP/1.1" 200 222422 "" "Prometheus/2.43.0" 2024-09-10T19:33:21.433 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:33:21 smithi183 ceph-mon[29736]: pgmap v1067: 142 pgs: 142 active+clean; 582 KiB data, 5.8 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-10T19:33:21.433 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:33:21 smithi183 ceph-mon[29736]: 6.9 scrub starts 2024-09-10T19:33:21.433 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:33:21 smithi183 ceph-mon[29736]: from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:33:21.433 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:33:21 smithi183 ceph-mon[29736]: 6.9 scrub ok 2024-09-10T19:33:21.433 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:33:21 smithi183 ceph-mon[29736]: 2.11 deep-scrub starts 2024-09-10T19:33:21.433 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:33:21 smithi183 ceph-mon[29736]: 2.11 deep-scrub ok 2024-09-10T19:33:21.433 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:33:21 smithi183 ceph-mon[29736]: from='client.? 172.21.15.183:0/3108151281' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-09-10T19:33:21.433 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:33:21 smithi183 ceph-mon[29736]: 5.7 scrub starts 2024-09-10T19:33:21.433 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:33:21 smithi183 ceph-mon[29736]: 5.7 scrub ok 2024-09-10T19:33:21.434 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:33:21 smithi183 ceph-mon[29736]: 5.6 deep-scrub starts 2024-09-10T19:33:21.434 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:33:21 smithi183 ceph-mon[29736]: 5.6 deep-scrub ok 2024-09-10T19:33:21.434 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:33:21 smithi183 ceph-mon[29736]: from='client.? 172.21.15.183:0/149886046' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-09-10T19:33:21.434 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:33:21 smithi183 ceph-mon[41562]: pgmap v1067: 142 pgs: 142 active+clean; 582 KiB data, 5.8 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-10T19:33:21.434 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:33:21 smithi183 ceph-mon[41562]: 6.9 scrub starts 2024-09-10T19:33:21.434 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:33:21 smithi183 ceph-mon[41562]: from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:33:21.434 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:33:21 smithi183 ceph-mon[41562]: 6.9 scrub ok 2024-09-10T19:33:21.434 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:33:21 smithi183 ceph-mon[41562]: 2.11 deep-scrub starts 2024-09-10T19:33:21.434 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:33:21 smithi183 ceph-mon[41562]: 2.11 deep-scrub ok 2024-09-10T19:33:21.434 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:33:21 smithi183 ceph-mon[41562]: from='client.? 172.21.15.183:0/3108151281' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-09-10T19:33:21.435 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:33:21 smithi183 ceph-mon[41562]: 5.7 scrub starts 2024-09-10T19:33:21.435 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:33:21 smithi183 ceph-mon[41562]: 5.7 scrub ok 2024-09-10T19:33:21.435 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:33:21 smithi183 ceph-mon[41562]: 5.6 deep-scrub starts 2024-09-10T19:33:21.435 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:33:21 smithi183 ceph-mon[41562]: 5.6 deep-scrub ok 2024-09-10T19:33:21.435 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:33:21 smithi183 ceph-mon[41562]: from='client.? 172.21.15.183:0/149886046' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-09-10T19:33:21.667 INFO:tasks.cephadm.ceph_manager.ceph:need seq 111669149995 got 111669149998 for osd.3 2024-09-10T19:33:21.667 DEBUG:teuthology.parallel:result is None 2024-09-10T19:33:21.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:33:21 smithi191 ceph-mon[32884]: pgmap v1067: 142 pgs: 142 active+clean; 582 KiB data, 5.8 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-10T19:33:21.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:33:21 smithi191 ceph-mon[32884]: 6.9 scrub starts 2024-09-10T19:33:21.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:33:21 smithi191 ceph-mon[32884]: from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:33:21.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:33:21 smithi191 ceph-mon[32884]: 6.9 scrub ok 2024-09-10T19:33:21.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:33:21 smithi191 ceph-mon[32884]: 2.11 deep-scrub starts 2024-09-10T19:33:21.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:33:21 smithi191 ceph-mon[32884]: 2.11 deep-scrub ok 2024-09-10T19:33:21.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:33:21 smithi191 ceph-mon[32884]: from='client.? 172.21.15.183:0/3108151281' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-09-10T19:33:21.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:33:21 smithi191 ceph-mon[32884]: 5.7 scrub starts 2024-09-10T19:33:21.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:33:21 smithi191 ceph-mon[32884]: 5.7 scrub ok 2024-09-10T19:33:21.717 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:33:21 smithi191 ceph-mon[32884]: 5.6 deep-scrub starts 2024-09-10T19:33:21.717 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:33:21 smithi191 ceph-mon[32884]: 5.6 deep-scrub ok 2024-09-10T19:33:21.717 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:33:21 smithi191 ceph-mon[32884]: from='client.? 172.21.15.183:0/149886046' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-09-10T19:33:22.371 INFO:tasks.cephadm.ceph_manager.ceph:need seq 1185410973821 got 1185410973825 for osd.1 2024-09-10T19:33:22.371 DEBUG:teuthology.parallel:result is None 2024-09-10T19:33:22.547 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:33:22 smithi183 ceph-mon[29736]: 4.5 scrub starts 2024-09-10T19:33:22.547 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:33:22 smithi183 ceph-mon[29736]: 4.5 scrub ok 2024-09-10T19:33:22.547 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:33:22 smithi183 ceph-mon[29736]: 5.16 deep-scrub starts 2024-09-10T19:33:22.547 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:33:22 smithi183 ceph-mon[29736]: 5.16 deep-scrub ok 2024-09-10T19:33:22.547 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:33:22 smithi183 ceph-mon[29736]: 6.e scrub starts 2024-09-10T19:33:22.547 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:33:22 smithi183 ceph-mon[29736]: 6.e scrub ok 2024-09-10T19:33:22.547 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:33:22 smithi183 ceph-mon[41562]: 4.5 scrub starts 2024-09-10T19:33:22.547 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:33:22 smithi183 ceph-mon[41562]: 4.5 scrub ok 2024-09-10T19:33:22.547 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:33:22 smithi183 ceph-mon[41562]: 5.16 deep-scrub starts 2024-09-10T19:33:22.548 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:33:22 smithi183 ceph-mon[41562]: 5.16 deep-scrub ok 2024-09-10T19:33:22.548 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:33:22 smithi183 ceph-mon[41562]: 6.e scrub starts 2024-09-10T19:33:22.548 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:33:22 smithi183 ceph-mon[41562]: 6.e scrub ok 2024-09-10T19:33:22.715 INFO:teuthology.orchestra.run.smithi183.stdout:77309411636 2024-09-10T19:33:22.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:33:22 smithi191 ceph-mon[32884]: 4.5 scrub starts 2024-09-10T19:33:22.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:33:22 smithi191 ceph-mon[32884]: 4.5 scrub ok 2024-09-10T19:33:22.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:33:22 smithi191 ceph-mon[32884]: 5.16 deep-scrub starts 2024-09-10T19:33:22.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:33:22 smithi191 ceph-mon[32884]: 5.16 deep-scrub ok 2024-09-10T19:33:22.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:33:22 smithi191 ceph-mon[32884]: 6.e scrub starts 2024-09-10T19:33:22.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:33:22 smithi191 ceph-mon[32884]: 6.e scrub ok 2024-09-10T19:33:23.618 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:33:23 smithi183 ceph-mon[41562]: pgmap v1068: 142 pgs: 142 active+clean; 582 KiB data, 5.8 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-10T19:33:23.618 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:33:23 smithi183 ceph-mon[41562]: from='client.? 172.21.15.183:0/216087964' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-09-10T19:33:23.618 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:33:23 smithi183 ceph-mon[41562]: purged_snaps scrub starts 2024-09-10T19:33:23.618 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:33:23 smithi183 ceph-mon[41562]: purged_snaps scrub ok 2024-09-10T19:33:23.618 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:33:23 smithi183 ceph-mon[41562]: 6.3 scrub starts 2024-09-10T19:33:23.618 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:33:23 smithi183 ceph-mon[41562]: 6.3 scrub ok 2024-09-10T19:33:23.619 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:33:23 smithi183 ceph-mon[29736]: pgmap v1068: 142 pgs: 142 active+clean; 582 KiB data, 5.8 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-10T19:33:23.619 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:33:23 smithi183 ceph-mon[29736]: from='client.? 172.21.15.183:0/216087964' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-09-10T19:33:23.619 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:33:23 smithi183 ceph-mon[29736]: purged_snaps scrub starts 2024-09-10T19:33:23.619 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:33:23 smithi183 ceph-mon[29736]: purged_snaps scrub ok 2024-09-10T19:33:23.619 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:33:23 smithi183 ceph-mon[29736]: 6.3 scrub starts 2024-09-10T19:33:23.619 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:33:23 smithi183 ceph-mon[29736]: 6.3 scrub ok 2024-09-10T19:33:23.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:33:23 smithi191 ceph-mon[32884]: pgmap v1068: 142 pgs: 142 active+clean; 582 KiB data, 5.8 GiB used, 709 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-10T19:33:23.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:33:23 smithi191 ceph-mon[32884]: from='client.? 172.21.15.183:0/216087964' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-09-10T19:33:23.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:33:23 smithi191 ceph-mon[32884]: purged_snaps scrub starts 2024-09-10T19:33:23.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:33:23 smithi191 ceph-mon[32884]: purged_snaps scrub ok 2024-09-10T19:33:23.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:33:23 smithi191 ceph-mon[32884]: 6.3 scrub starts 2024-09-10T19:33:23.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:33:23 smithi191 ceph-mon[32884]: 6.3 scrub ok 2024-09-10T19:33:24.567 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:33:24 smithi183 ceph-mon[29736]: osdmap e460: 8 total, 8 up, 8 in 2024-09-10T19:33:24.567 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:33:24 smithi183 ceph-mon[41562]: osdmap e460: 8 total, 8 up, 8 in 2024-09-10T19:33:24.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:33:24 smithi191 ceph-mon[32884]: osdmap e460: 8 total, 8 up, 8 in 2024-09-10T19:33:25.385 INFO:tasks.cephadm.ceph_manager.ceph:need seq 77309411632 got 77309411636 for osd.2 2024-09-10T19:33:25.386 DEBUG:teuthology.parallel:result is None 2024-09-10T19:33:25.602 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:33:25 smithi183 ceph-mon[41562]: pgmap v1070: 142 pgs: 142 active+clean; 582 KiB data, 5.8 GiB used, 709 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-10T19:33:25.602 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:33:25 smithi183 ceph-mon[41562]: 6.b scrub starts 2024-09-10T19:33:25.602 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:33:25 smithi183 ceph-mon[41562]: 6.b scrub ok 2024-09-10T19:33:25.602 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:33:25 smithi183 ceph-mon[41562]: 5.2 deep-scrub starts 2024-09-10T19:33:25.602 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:33:25 smithi183 ceph-mon[41562]: 5.2 deep-scrub ok 2024-09-10T19:33:25.603 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:33:25 smithi183 ceph-mon[41562]: 5.a scrub starts 2024-09-10T19:33:25.603 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:33:25 smithi183 ceph-mon[41562]: 5.a scrub ok 2024-09-10T19:33:25.603 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:33:25 smithi183 ceph-mon[41562]: 4.18 scrub starts 2024-09-10T19:33:25.603 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:33:25 smithi183 ceph-mon[41562]: 4.18 scrub ok 2024-09-10T19:33:25.603 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:33:25 smithi183 ceph-mon[29736]: pgmap v1070: 142 pgs: 142 active+clean; 582 KiB data, 5.8 GiB used, 709 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-10T19:33:25.603 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:33:25 smithi183 ceph-mon[29736]: 6.b scrub starts 2024-09-10T19:33:25.603 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:33:25 smithi183 ceph-mon[29736]: 6.b scrub ok 2024-09-10T19:33:25.603 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:33:25 smithi183 ceph-mon[29736]: 5.2 deep-scrub starts 2024-09-10T19:33:25.603 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:33:25 smithi183 ceph-mon[29736]: 5.2 deep-scrub ok 2024-09-10T19:33:25.603 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:33:25 smithi183 ceph-mon[29736]: 5.a scrub starts 2024-09-10T19:33:25.603 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:33:25 smithi183 ceph-mon[29736]: 5.a scrub ok 2024-09-10T19:33:25.604 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:33:25 smithi183 ceph-mon[29736]: 4.18 scrub starts 2024-09-10T19:33:25.604 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:33:25 smithi183 ceph-mon[29736]: 4.18 scrub ok 2024-09-10T19:33:25.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:33:25 smithi191 ceph-mon[32884]: pgmap v1070: 142 pgs: 142 active+clean; 582 KiB data, 5.8 GiB used, 709 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-10T19:33:25.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:33:25 smithi191 ceph-mon[32884]: 6.b scrub starts 2024-09-10T19:33:25.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:33:25 smithi191 ceph-mon[32884]: 6.b scrub ok 2024-09-10T19:33:25.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:33:25 smithi191 ceph-mon[32884]: 5.2 deep-scrub starts 2024-09-10T19:33:25.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:33:25 smithi191 ceph-mon[32884]: 5.2 deep-scrub ok 2024-09-10T19:33:25.717 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:33:25 smithi191 ceph-mon[32884]: 5.a scrub starts 2024-09-10T19:33:25.717 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:33:25 smithi191 ceph-mon[32884]: 5.a scrub ok 2024-09-10T19:33:25.717 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:33:25 smithi191 ceph-mon[32884]: 4.18 scrub starts 2024-09-10T19:33:25.717 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:33:25 smithi191 ceph-mon[32884]: 4.18 scrub ok 2024-09-10T19:33:26.052 INFO:teuthology.orchestra.run.smithi183.stdout:201863463201 2024-09-10T19:33:26.106 INFO:teuthology.orchestra.run.smithi183.stdout:231928234268 2024-09-10T19:33:26.365 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:33:26 smithi183 ceph-mon[29736]: 6.0 scrub starts 2024-09-10T19:33:26.365 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:33:26 smithi183 ceph-mon[29736]: 6.0 scrub ok 2024-09-10T19:33:26.365 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:33:26 smithi183 ceph-mon[29736]: purged_snaps scrub starts 2024-09-10T19:33:26.365 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:33:26 smithi183 ceph-mon[29736]: purged_snaps scrub ok 2024-09-10T19:33:26.365 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:33:26 smithi183 ceph-mon[29736]: from='client.? 172.21.15.183:0/2920651416' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-09-10T19:33:26.365 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:33:26 smithi183 ceph-mon[29736]: 4.d scrub starts 2024-09-10T19:33:26.366 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:33:26 smithi183 ceph-mon[29736]: 4.d scrub ok 2024-09-10T19:33:26.366 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:33:26 smithi183 ceph-mon[29736]: from='client.? 172.21.15.183:0/3279904689' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-09-10T19:33:26.366 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:33:26 smithi183 ceph-mon[41562]: 6.0 scrub starts 2024-09-10T19:33:26.366 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:33:26 smithi183 ceph-mon[41562]: 6.0 scrub ok 2024-09-10T19:33:26.366 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:33:26 smithi183 ceph-mon[41562]: purged_snaps scrub starts 2024-09-10T19:33:26.366 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:33:26 smithi183 ceph-mon[41562]: purged_snaps scrub ok 2024-09-10T19:33:26.366 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:33:26 smithi183 ceph-mon[41562]: from='client.? 172.21.15.183:0/2920651416' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-09-10T19:33:26.366 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:33:26 smithi183 ceph-mon[41562]: 4.d scrub starts 2024-09-10T19:33:26.366 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:33:26 smithi183 ceph-mon[41562]: 4.d scrub ok 2024-09-10T19:33:26.366 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:33:26 smithi183 ceph-mon[41562]: from='client.? 172.21.15.183:0/3279904689' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-09-10T19:33:26.715 INFO:tasks.cephadm.ceph_manager.ceph:need seq 201863463197 got 201863463201 for osd.6 2024-09-10T19:33:26.716 DEBUG:teuthology.parallel:result is None 2024-09-10T19:33:26.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:33:26 smithi191 ceph-mon[32884]: 6.0 scrub starts 2024-09-10T19:33:26.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:33:26 smithi191 ceph-mon[32884]: 6.0 scrub ok 2024-09-10T19:33:26.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:33:26 smithi191 ceph-mon[32884]: purged_snaps scrub starts 2024-09-10T19:33:26.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:33:26 smithi191 ceph-mon[32884]: purged_snaps scrub ok 2024-09-10T19:33:26.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:33:26 smithi191 ceph-mon[32884]: from='client.? 172.21.15.183:0/2920651416' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-09-10T19:33:26.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:33:26 smithi191 ceph-mon[32884]: 4.d scrub starts 2024-09-10T19:33:26.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:33:26 smithi191 ceph-mon[32884]: 4.d scrub ok 2024-09-10T19:33:26.717 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:33:26 smithi191 ceph-mon[32884]: from='client.? 172.21.15.183:0/3279904689' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-09-10T19:33:27.389 INFO:tasks.cephadm.ceph_manager.ceph:need seq 231928234265 got 231928234268 for osd.7 2024-09-10T19:33:27.389 DEBUG:teuthology.parallel:result is None 2024-09-10T19:33:27.389 INFO:tasks.cephadm.ceph_manager.ceph:waiting for recovery to complete 2024-09-10T19:33:27.390 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph pg dump --format=json 2024-09-10T19:33:27.617 INFO:journalctl@ceph.mgr.x.smithi191.stdout:Sep 10 19:33:27 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-x[34967]: ::ffff:172.21.15.191 - - [10/Sep/2024:19:33:27] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T19:33:27.617 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:33:27 smithi191 ceph-mon[32884]: pgmap v1071: 142 pgs: 142 active+clean; 582 KiB data, 5.8 GiB used, 709 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-10T19:33:27.617 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:33:27 smithi191 ceph-mon[32884]: osdmap e461: 8 total, 8 up, 8 in 2024-09-10T19:33:27.618 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:33:27 smithi191 ceph-mon[32884]: 5.1f scrub starts 2024-09-10T19:33:27.618 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:33:27 smithi191 ceph-mon[32884]: 5.1f scrub ok 2024-09-10T19:33:27.618 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:33:27 smithi191 ceph-mon[32884]: 2.2 scrub starts 2024-09-10T19:33:27.618 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:33:27 smithi191 ceph-mon[32884]: 2.2 scrub ok 2024-09-10T19:33:27.618 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:33:27 smithi191 ceph-mon[32884]: 2.1 scrub starts 2024-09-10T19:33:27.618 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:33:27 smithi191 ceph-mon[32884]: 2.1 scrub ok 2024-09-10T19:33:27.618 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:33:27 smithi191 ceph-mon[32884]: 2.f scrub starts 2024-09-10T19:33:27.618 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:33:27 smithi191 ceph-mon[32884]: 2.f scrub ok 2024-09-10T19:33:27.639 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:33:27.681 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:33:27 smithi183 ceph-mon[29736]: pgmap v1071: 142 pgs: 142 active+clean; 582 KiB data, 5.8 GiB used, 709 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-10T19:33:27.682 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:33:27 smithi183 ceph-mon[29736]: osdmap e461: 8 total, 8 up, 8 in 2024-09-10T19:33:27.682 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:33:27 smithi183 ceph-mon[29736]: 5.1f scrub starts 2024-09-10T19:33:27.682 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:33:27 smithi183 ceph-mon[29736]: 5.1f scrub ok 2024-09-10T19:33:27.682 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:33:27 smithi183 ceph-mon[29736]: 2.2 scrub starts 2024-09-10T19:33:27.682 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:33:27 smithi183 ceph-mon[29736]: 2.2 scrub ok 2024-09-10T19:33:27.682 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:33:27 smithi183 ceph-mon[29736]: 2.1 scrub starts 2024-09-10T19:33:27.682 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:33:27 smithi183 ceph-mon[29736]: 2.1 scrub ok 2024-09-10T19:33:27.682 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:33:27 smithi183 ceph-mon[29736]: 2.f scrub starts 2024-09-10T19:33:27.682 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:33:27 smithi183 ceph-mon[29736]: 2.f scrub ok 2024-09-10T19:33:27.682 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:33:27 smithi183 ceph-mon[41562]: pgmap v1071: 142 pgs: 142 active+clean; 582 KiB data, 5.8 GiB used, 709 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-10T19:33:27.683 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:33:27 smithi183 ceph-mon[41562]: osdmap e461: 8 total, 8 up, 8 in 2024-09-10T19:33:27.683 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:33:27 smithi183 ceph-mon[41562]: 5.1f scrub starts 2024-09-10T19:33:27.683 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:33:27 smithi183 ceph-mon[41562]: 5.1f scrub ok 2024-09-10T19:33:27.683 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:33:27 smithi183 ceph-mon[41562]: 2.2 scrub starts 2024-09-10T19:33:27.683 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:33:27 smithi183 ceph-mon[41562]: 2.2 scrub ok 2024-09-10T19:33:27.683 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:33:27 smithi183 ceph-mon[41562]: 2.1 scrub starts 2024-09-10T19:33:27.683 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:33:27 smithi183 ceph-mon[41562]: 2.1 scrub ok 2024-09-10T19:33:27.683 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:33:27 smithi183 ceph-mon[41562]: 2.f scrub starts 2024-09-10T19:33:27.683 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:33:27 smithi183 ceph-mon[41562]: 2.f scrub ok 2024-09-10T19:33:29.596 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-09-10T19:33:29.596 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:33:29 smithi183 ceph-mon[29736]: pgmap v1073: 142 pgs: 142 active+clean; 582 KiB data, 5.8 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:33:29.596 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:33:29 smithi183 ceph-mon[29736]: 6.4 scrub starts 2024-09-10T19:33:29.596 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:33:29 smithi183 ceph-mon[29736]: 6.4 scrub ok 2024-09-10T19:33:29.597 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:33:29 smithi183 ceph-mon[29736]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:33:29.597 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:33:29 smithi183 ceph-mon[41562]: pgmap v1073: 142 pgs: 142 active+clean; 582 KiB data, 5.8 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:33:29.597 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:33:29 smithi183 ceph-mon[41562]: 6.4 scrub starts 2024-09-10T19:33:29.597 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:33:29 smithi183 ceph-mon[41562]: 6.4 scrub ok 2024-09-10T19:33:29.597 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:33:29 smithi183 ceph-mon[41562]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:33:29.603 INFO:teuthology.orchestra.run.smithi183.stderr:dumped all 2024-09-10T19:33:29.715 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:33:29 smithi191 ceph-mon[32884]: pgmap v1073: 142 pgs: 142 active+clean; 582 KiB data, 5.8 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:33:29.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:33:29 smithi191 ceph-mon[32884]: 6.4 scrub starts 2024-09-10T19:33:29.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:33:29 smithi191 ceph-mon[32884]: 6.4 scrub ok 2024-09-10T19:33:29.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:33:29 smithi191 ceph-mon[32884]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-10T19:33:30.237 INFO:teuthology.orchestra.run.smithi183.stdout:{"pg_ready":true,"pg_map":{"version":1073,"stamp":"2024-09-10T19:33:28.030012+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596188,"num_objects":196,"num_object_clones":0,"num_object_copies":588,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":196,"num_whiteouts":0,"num_read":2127,"num_read_kb":1879,"num_write":1002,"num_write_kb":43,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":526,"num_bytes_recovered":801166,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2079,"ondisk_log_size":2079,"up":426,"acting":426,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":426,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":6037016,"kb_used_data":7472,"kb_used_omap":20,"kb_used_meta":6028587,"kb_avail":743924200,"statfs":{"total":767960285184,"available":761778380800,"internally_reserved":0,"allocated":7651328,"data_stored":5206259,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":21161,"internal_metadata":6173273431},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001825"},"pg_stats":[{"pgid":"6.1b","version":"0'0","reported_seq":226,"reported_epoch":460,"state":"active+clean","last_fresh":"2024-09-10T19:33:24.309179+0000","last_change":"2024-09-10T19:33:10.306914+0000","last_active":"2024-09-10T19:33:24.309179+0000","last_peered":"2024-09-10T19:33:24.309179+0000","last_clean":"2024-09-10T19:33:24.309179+0000","last_became_active":"2024-09-10T19:27:52.994757+0000","last_became_peered":"2024-09-10T19:27:52.994757+0000","last_unstale":"2024-09-10T19:33:24.309179+0000","last_undegraded":"2024-09-10T19:33:24.309179+0000","last_fullsized":"2024-09-10T19:33:24.309179+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:33:10.306857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:31:41.824279+0000","last_clean_scrub_stamp":"2024-09-10T19:33:10.306857+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:34:35.755721+0000","scrub_duration":0.0020253770000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6],"acting":[0,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":474,"reported_epoch":460,"state":"active+clean","last_fresh":"2024-09-10T19:33:24.310199+0000","last_change":"2024-09-10T19:32:38.216788+0000","last_active":"2024-09-10T19:33:24.310199+0000","last_peered":"2024-09-10T19:33:24.310199+0000","last_clean":"2024-09-10T19:33:24.310199+0000","last_became_active":"2024-09-10T19:24:23.892621+0000","last_became_peered":"2024-09-10T19:24:23.892621+0000","last_unstale":"2024-09-10T19:33:24.310199+0000","last_undegraded":"2024-09-10T19:33:24.310199+0000","last_fullsized":"2024-09-10T19:33:24.310199+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:32:38.216717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:32:38.216717+0000","last_clean_scrub_stamp":"2024-09-10T19:32:38.216717+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:33:41.435276+0000","scrub_duration":0.002118296,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.19","version":"440'46","reported_seq":615,"reported_epoch":459,"state":"active+clean","last_fresh":"2024-09-10T19:33:02.226521+0000","last_change":"2024-09-10T19:32:05.493764+0000","last_active":"2024-09-10T19:33:02.226521+0000","last_peered":"2024-09-10T19:33:02.226521+0000","last_clean":"2024-09-10T19:33:02.226521+0000","last_became_active":"2024-09-10T19:31:33.338277+0000","last_became_peered":"2024-09-10T19:31:33.338277+0000","last_unstale":"2024-09-10T19:33:02.226521+0000","last_undegraded":"2024-09-10T19:33:02.226521+0000","last_fullsized":"2024-09-10T19:33:02.226521+0000","mapping_epoch":431,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":432,"parent":"0.0","parent_split_bits":0,"last_scrub":"440'46","last_scrub_stamp":"2024-09-10T19:32:05.493712+0000","last_deep_scrub":"440'46","last_deep_scrub_stamp":"2024-09-10T19:32:05.493712+0000","last_clean_scrub_stamp":"2024-09-10T19:32:05.493712+0000","objects_scrubbed":7,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:33:35.039694+0000","scrub_duration":0.0044865089999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":62,"num_read_kb":53,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":558,"reported_epoch":459,"state":"active+clean","last_fresh":"2024-09-10T19:33:02.907057+0000","last_change":"2024-09-10T19:32:08.698175+0000","last_active":"2024-09-10T19:33:02.907057+0000","last_peered":"2024-09-10T19:33:02.907057+0000","last_clean":"2024-09-10T19:33:02.907057+0000","last_became_active":"2024-09-10T19:27:53.006175+0000","last_became_peered":"2024-09-10T19:27:53.006175+0000","last_unstale":"2024-09-10T19:33:02.907057+0000","last_undegraded":"2024-09-10T19:33:02.907057+0000","last_fullsized":"2024-09-10T19:33:02.907057+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:32:08.698092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:19:28.498560+0000","last_clean_scrub_stamp":"2024-09-10T19:32:08.698092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:33:28.005117+0000","scrub_duration":0.001424399,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"384'2","reported_seq":526,"reported_epoch":459,"state":"active+clean","last_fresh":"2024-09-10T19:33:02.227533+0000","last_change":"2024-09-10T19:32:20.254031+0000","last_active":"2024-09-10T19:33:02.227533+0000","last_peered":"2024-09-10T19:33:02.227533+0000","last_clean":"2024-09-10T19:33:02.227533+0000","last_became_active":"2024-09-10T19:24:23.908614+0000","last_became_peered":"2024-09-10T19:24:23.908614+0000","last_unstale":"2024-09-10T19:33:02.227533+0000","last_undegraded":"2024-09-10T19:33:02.227533+0000","last_fullsized":"2024-09-10T19:33:02.227533+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"384'2","last_scrub_stamp":"2024-09-10T19:32:20.253978+0000","last_deep_scrub":"384'2","last_deep_scrub_stamp":"2024-09-10T19:28:23.240824+0000","last_clean_scrub_stamp":"2024-09-10T19:32:20.253978+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0017100640000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":874,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"414'25","reported_seq":655,"reported_epoch":460,"state":"active+clean","last_fresh":"2024-09-10T19:33:25.034739+0000","last_change":"2024-09-10T19:33:25.034702+0000","last_active":"2024-09-10T19:33:25.034739+0000","last_peered":"2024-09-10T19:33:25.034739+0000","last_clean":"2024-09-10T19:33:25.034739+0000","last_became_active":"2024-09-10T19:31:47.389313+0000","last_became_peered":"2024-09-10T19:31:47.389313+0000","last_unstale":"2024-09-10T19:33:25.034739+0000","last_undegraded":"2024-09-10T19:33:25.034739+0000","last_fullsized":"2024-09-10T19:33:25.034739+0000","mapping_epoch":435,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":436,"parent":"0.0","parent_split_bits":0,"last_scrub":"414'25","last_scrub_stamp":"2024-09-10T19:33:25.034634+0000","last_deep_scrub":"384'19","last_deep_scrub_stamp":"2024-09-10T19:28:21.799921+0000","last_clean_scrub_stamp":"2024-09-10T19:33:25.034634+0000","objects_scrubbed":4,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:34:32.409274+0000","scrub_duration":0.0021945530000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":550,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":108,"reported_epoch":459,"state":"active+clean","last_fresh":"2024-09-10T19:33:02.857485+0000","last_change":"2024-09-10T19:32:51.579358+0000","last_active":"2024-09-10T19:33:02.857485+0000","last_peered":"2024-09-10T19:33:02.857485+0000","last_clean":"2024-09-10T19:33:02.857485+0000","last_became_active":"2024-09-10T19:28:47.151481+0000","last_became_peered":"2024-09-10T19:28:47.151481+0000","last_unstale":"2024-09-10T19:33:02.857485+0000","last_undegraded":"2024-09-10T19:33:02.857485+0000","last_fullsized":"2024-09-10T19:33:02.857485+0000","mapping_epoch":391,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":392,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:32:51.579297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:32:51.579297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:34:19.568803+0000","scrub_duration":0.001353557,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,7],"acting":[6,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":536,"reported_epoch":459,"state":"active+clean","last_fresh":"2024-09-10T19:33:02.907315+0000","last_change":"2024-09-10T19:32:50.757063+0000","last_active":"2024-09-10T19:33:02.907315+0000","last_peered":"2024-09-10T19:33:02.907315+0000","last_clean":"2024-09-10T19:33:02.907315+0000","last_became_active":"2024-09-10T19:24:01.971921+0000","last_became_peered":"2024-09-10T19:24:01.971921+0000","last_unstale":"2024-09-10T19:33:02.907315+0000","last_undegraded":"2024-09-10T19:33:02.907315+0000","last_fullsized":"2024-09-10T19:33:02.907315+0000","mapping_epoch":292,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":293,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:32:50.757000+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:32:50.757000+0000","last_clean_scrub_stamp":"2024-09-10T19:32:50.757000+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:33:54.139512+0000","scrub_duration":0.0013316630000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":537,"reported_epoch":459,"state":"active+clean","last_fresh":"2024-09-10T19:33:02.908692+0000","last_change":"2024-09-10T19:32:42.015279+0000","last_active":"2024-09-10T19:33:02.908692+0000","last_peered":"2024-09-10T19:33:02.908692+0000","last_clean":"2024-09-10T19:33:02.908692+0000","last_became_active":"2024-09-10T19:27:53.011202+0000","last_became_peered":"2024-09-10T19:27:53.011202+0000","last_unstale":"2024-09-10T19:33:02.908692+0000","last_undegraded":"2024-09-10T19:33:02.908692+0000","last_fullsized":"2024-09-10T19:33:02.908692+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:32:42.015213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:32:42.015213+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:33:48.046949+0000","scrub_duration":0.001419172,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1b","version":"414'9","reported_seq":581,"reported_epoch":459,"state":"active+clean","last_fresh":"2024-09-10T19:33:07.825084+0000","last_change":"2024-09-10T19:33:07.825011+0000","last_active":"2024-09-10T19:33:07.825084+0000","last_peered":"2024-09-10T19:33:07.825084+0000","last_clean":"2024-09-10T19:33:07.825084+0000","last_became_active":"2024-09-10T19:24:01.974813+0000","last_became_peered":"2024-09-10T19:24:01.974813+0000","last_unstale":"2024-09-10T19:33:07.825084+0000","last_undegraded":"2024-09-10T19:33:07.825084+0000","last_fullsized":"2024-09-10T19:33:07.825084+0000","mapping_epoch":292,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":293,"parent":"0.0","parent_split_bits":0,"last_scrub":"414'9","last_scrub_stamp":"2024-09-10T19:33:07.824943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:33:07.824943+0000","objects_scrubbed":2,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:34:30.077554+0000","scrub_duration":0.0019824869999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":17,"num_read_kb":13,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":520,"reported_epoch":459,"state":"active+clean","last_fresh":"2024-09-10T19:33:02.907557+0000","last_change":"2024-09-10T19:32:08.177548+0000","last_active":"2024-09-10T19:33:02.907557+0000","last_peered":"2024-09-10T19:33:02.907557+0000","last_clean":"2024-09-10T19:33:02.907557+0000","last_became_active":"2024-09-10T19:24:01.976587+0000","last_became_peered":"2024-09-10T19:24:01.976587+0000","last_unstale":"2024-09-10T19:33:02.907557+0000","last_undegraded":"2024-09-10T19:33:02.907557+0000","last_fullsized":"2024-09-10T19:33:02.907557+0000","mapping_epoch":292,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":293,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:32:08.177472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:32:08.177472+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:33:28.282859+0000","scrub_duration":0.001797674,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":531,"reported_epoch":459,"state":"active+clean","last_fresh":"2024-09-10T19:33:17.587171+0000","last_change":"2024-09-10T19:33:17.587123+0000","last_active":"2024-09-10T19:33:17.587171+0000","last_peered":"2024-09-10T19:33:17.587171+0000","last_clean":"2024-09-10T19:33:17.587171+0000","last_became_active":"2024-09-10T19:24:01.970118+0000","last_became_peered":"2024-09-10T19:24:01.970118+0000","last_unstale":"2024-09-10T19:33:17.587171+0000","last_undegraded":"2024-09-10T19:33:17.587171+0000","last_fullsized":"2024-09-10T19:33:17.587171+0000","mapping_epoch":292,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":293,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:33:17.587025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:31:53.538320+0000","last_clean_scrub_stamp":"2024-09-10T19:33:17.587025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:34:34.145375+0000","scrub_duration":0.002082534,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":550,"reported_epoch":459,"state":"active+clean","last_fresh":"2024-09-10T19:33:02.908283+0000","last_change":"2024-09-10T19:32:49.927969+0000","last_active":"2024-09-10T19:33:02.908283+0000","last_peered":"2024-09-10T19:33:02.908283+0000","last_clean":"2024-09-10T19:33:02.908283+0000","last_became_active":"2024-09-10T19:31:33.342560+0000","last_became_peered":"2024-09-10T19:31:33.342560+0000","last_unstale":"2024-09-10T19:33:02.908283+0000","last_undegraded":"2024-09-10T19:33:02.908283+0000","last_fullsized":"2024-09-10T19:33:02.908283+0000","mapping_epoch":431,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":432,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:32:49.927910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:32:49.927910+0000","last_clean_scrub_stamp":"2024-09-10T19:32:49.927910+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:33:57.987752+0000","scrub_duration":0.0012803409999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1a","version":"414'21","reported_seq":541,"reported_epoch":459,"state":"active+clean","last_fresh":"2024-09-10T19:33:02.907590+0000","last_change":"2024-09-10T19:31:57.777057+0000","last_active":"2024-09-10T19:33:02.907590+0000","last_peered":"2024-09-10T19:33:02.907590+0000","last_clean":"2024-09-10T19:33:02.907590+0000","last_became_active":"2024-09-10T19:24:23.891213+0000","last_became_peered":"2024-09-10T19:24:23.891213+0000","last_unstale":"2024-09-10T19:33:02.907590+0000","last_undegraded":"2024-09-10T19:33:02.907590+0000","last_fullsized":"2024-09-10T19:33:02.907590+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"414'21","last_scrub_stamp":"2024-09-10T19:31:57.776990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:31:57.776990+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:33:23.363866+0000","scrub_duration":0.0020528489999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":523,"reported_epoch":459,"state":"active+clean","last_fresh":"2024-09-10T19:33:02.227533+0000","last_change":"2024-09-10T19:32:57.594733+0000","last_active":"2024-09-10T19:33:02.227533+0000","last_peered":"2024-09-10T19:33:02.227533+0000","last_clean":"2024-09-10T19:33:02.227533+0000","last_became_active":"2024-09-10T19:24:23.907653+0000","last_became_peered":"2024-09-10T19:24:23.907653+0000","last_unstale":"2024-09-10T19:33:02.227533+0000","last_undegraded":"2024-09-10T19:33:02.227533+0000","last_fullsized":"2024-09-10T19:33:02.227533+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:32:57.594665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:32:57.594665+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:34:27.488641+0000","scrub_duration":0.0014914939999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":487,"reported_epoch":460,"state":"active+clean","last_fresh":"2024-09-10T19:33:24.309011+0000","last_change":"2024-09-10T19:32:13.105849+0000","last_active":"2024-09-10T19:33:24.309011+0000","last_peered":"2024-09-10T19:33:24.309011+0000","last_clean":"2024-09-10T19:33:24.309011+0000","last_became_active":"2024-09-10T19:24:24.907431+0000","last_became_peered":"2024-09-10T19:24:24.907431+0000","last_unstale":"2024-09-10T19:33:24.309011+0000","last_undegraded":"2024-09-10T19:33:24.309011+0000","last_fullsized":"2024-09-10T19:33:24.309011+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:32:13.105794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:30:41.905516+0000","last_clean_scrub_stamp":"2024-09-10T19:32:13.105794+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:33:42.643961+0000","scrub_duration":0.0027163510000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.8","version":"439'80","reported_seq":333,"reported_epoch":459,"state":"active+clean","last_fresh":"2024-09-10T19:33:02.226803+0000","last_change":"2024-09-10T19:32:00.542682+0000","last_active":"2024-09-10T19:33:02.226803+0000","last_peered":"2024-09-10T19:33:02.226803+0000","last_clean":"2024-09-10T19:33:02.226803+0000","last_became_active":"2024-09-10T19:31:33.348819+0000","last_became_peered":"2024-09-10T19:31:33.348819+0000","last_unstale":"2024-09-10T19:33:02.226803+0000","last_undegraded":"2024-09-10T19:33:02.226803+0000","last_fullsized":"2024-09-10T19:33:02.226803+0000","mapping_epoch":431,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":432,"parent":"0.0","parent_split_bits":4,"last_scrub":"439'80","last_scrub_stamp":"2024-09-10T19:32:00.542611+0000","last_deep_scrub":"439'80","last_deep_scrub_stamp":"2024-09-10T19:32:00.542611+0000","last_clean_scrub_stamp":"2024-09-10T19:32:00.542611+0000","objects_scrubbed":1,"log_size":80,"log_dups_size":0,"ondisk_log_size":80,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0044182830000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":788064,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.d","version":"437'53","reported_seq":757,"reported_epoch":459,"state":"active+clean","last_fresh":"2024-09-10T19:33:02.908237+0000","last_change":"2024-09-10T19:31:51.712213+0000","last_active":"2024-09-10T19:33:02.908237+0000","last_peered":"2024-09-10T19:33:02.908237+0000","last_clean":"2024-09-10T19:33:02.908237+0000","last_became_active":"2024-09-10T19:31:33.349405+0000","last_became_peered":"2024-09-10T19:31:33.349405+0000","last_unstale":"2024-09-10T19:33:02.908237+0000","last_undegraded":"2024-09-10T19:33:02.908237+0000","last_fullsized":"2024-09-10T19:33:02.908237+0000","mapping_epoch":431,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":432,"parent":"0.0","parent_split_bits":0,"last_scrub":"437'53","last_scrub_stamp":"2024-09-10T19:31:51.712146+0000","last_deep_scrub":"437'53","last_deep_scrub_stamp":"2024-09-10T19:31:51.712146+0000","last_clean_scrub_stamp":"2024-09-10T19:31:51.712146+0000","objects_scrubbed":8,"log_size":53,"log_dups_size":0,"ondisk_log_size":53,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0049495770000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":71,"num_read_kb":61,"num_write":46,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":48,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":510,"reported_epoch":459,"state":"active+clean","last_fresh":"2024-09-10T19:33:02.908120+0000","last_change":"2024-09-10T19:32:20.013848+0000","last_active":"2024-09-10T19:33:02.908120+0000","last_peered":"2024-09-10T19:33:02.908120+0000","last_clean":"2024-09-10T19:33:02.908120+0000","last_became_active":"2024-09-10T19:10:07.691119+0000","last_became_peered":"2024-09-10T19:10:07.691119+0000","last_unstale":"2024-09-10T19:33:02.908120+0000","last_undegraded":"2024-09-10T19:33:02.908120+0000","last_fullsized":"2024-09-10T19:33:02.908120+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:32:20.013777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:16:39.716674+0000","last_clean_scrub_stamp":"2024-09-10T19:32:20.013777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:33:43.355910+0000","scrub_duration":0.001598063,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":242,"reported_epoch":460,"state":"active+clean","last_fresh":"2024-09-10T19:33:24.312110+0000","last_change":"2024-09-10T19:32:32.353573+0000","last_active":"2024-09-10T19:33:24.312110+0000","last_peered":"2024-09-10T19:33:24.312110+0000","last_clean":"2024-09-10T19:33:24.312110+0000","last_became_active":"2024-09-10T19:24:24.886479+0000","last_became_peered":"2024-09-10T19:24:24.886479+0000","last_unstale":"2024-09-10T19:33:24.312110+0000","last_undegraded":"2024-09-10T19:33:24.312110+0000","last_fullsized":"2024-09-10T19:33:24.312110+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:32:32.353470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:32:32.353470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:33:52.060075+0000","scrub_duration":0.0019860799999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":48,"reported_epoch":459,"state":"active+clean","last_fresh":"2024-09-10T19:33:02.226221+0000","last_change":"2024-09-10T19:32:36.635397+0000","last_active":"2024-09-10T19:33:02.226221+0000","last_peered":"2024-09-10T19:33:02.226221+0000","last_clean":"2024-09-10T19:33:02.226221+0000","last_became_active":"2024-09-10T19:31:33.345895+0000","last_became_peered":"2024-09-10T19:31:33.345895+0000","last_unstale":"2024-09-10T19:33:02.226221+0000","last_undegraded":"2024-09-10T19:33:02.226221+0000","last_fullsized":"2024-09-10T19:33:02.226221+0000","mapping_epoch":431,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":432,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:32:36.635341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:32:36.635341+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:34:06.621717+0000","scrub_duration":0.0013187209999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"74'79","reported_seq":585,"reported_epoch":459,"state":"active+clean","last_fresh":"2024-09-10T19:33:02.907496+0000","last_change":"2024-09-10T19:32:22.957091+0000","last_active":"2024-09-10T19:33:02.907496+0000","last_peered":"2024-09-10T19:33:02.907496+0000","last_clean":"2024-09-10T19:33:02.907496+0000","last_became_active":"2024-09-10T19:31:33.337526+0000","last_became_peered":"2024-09-10T19:31:33.337526+0000","last_unstale":"2024-09-10T19:33:02.907496+0000","last_undegraded":"2024-09-10T19:33:02.907496+0000","last_fullsized":"2024-09-10T19:33:02.907496+0000","mapping_epoch":431,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":432,"parent":"0.0","parent_split_bits":4,"last_scrub":"74'79","last_scrub_stamp":"2024-09-10T19:32:22.957027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:08:00.699315+0000","last_clean_scrub_stamp":"2024-09-10T19:32:22.957027+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:33:35.149675+0000","scrub_duration":0.0018635259999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.c","version":"414'30","reported_seq":609,"reported_epoch":459,"state":"active+clean","last_fresh":"2024-09-10T19:33:13.850044+0000","last_change":"2024-09-10T19:33:13.849981+0000","last_active":"2024-09-10T19:33:13.850044+0000","last_peered":"2024-09-10T19:33:13.850044+0000","last_clean":"2024-09-10T19:33:13.850044+0000","last_became_active":"2024-09-10T19:27:53.012617+0000","last_became_peered":"2024-09-10T19:27:53.012617+0000","last_unstale":"2024-09-10T19:33:13.850044+0000","last_undegraded":"2024-09-10T19:33:13.850044+0000","last_fullsized":"2024-09-10T19:33:13.850044+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"414'30","last_scrub_stamp":"2024-09-10T19:33:13.849914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:33:13.849914+0000","objects_scrubbed":5,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:34:28.542502+0000","scrub_duration":0.001756968,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":45,"num_read_kb":40,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":218,"reported_epoch":460,"state":"active+clean","last_fresh":"2024-09-10T19:33:24.312070+0000","last_change":"2024-09-10T19:32:16.387044+0000","last_active":"2024-09-10T19:33:24.312070+0000","last_peered":"2024-09-10T19:33:24.312070+0000","last_clean":"2024-09-10T19:33:24.312070+0000","last_became_active":"2024-09-10T19:24:01.973848+0000","last_became_peered":"2024-09-10T19:24:01.973848+0000","last_unstale":"2024-09-10T19:33:24.312070+0000","last_undegraded":"2024-09-10T19:33:24.312070+0000","last_fullsized":"2024-09-10T19:33:24.312070+0000","mapping_epoch":292,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":293,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:32:16.386988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:32:16.386988+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:33:26.421807+0000","scrub_duration":0.0019624949999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.d","version":"432'13","reported_seq":74,"reported_epoch":459,"state":"active+clean","last_fresh":"2024-09-10T19:33:21.003595+0000","last_change":"2024-09-10T19:32:41.500567+0000","last_active":"2024-09-10T19:33:21.003595+0000","last_peered":"2024-09-10T19:33:21.003595+0000","last_clean":"2024-09-10T19:33:21.003595+0000","last_became_active":"2024-09-10T19:31:33.347510+0000","last_became_peered":"2024-09-10T19:31:33.347510+0000","last_unstale":"2024-09-10T19:33:21.003595+0000","last_undegraded":"2024-09-10T19:33:21.003595+0000","last_fullsized":"2024-09-10T19:33:21.003595+0000","mapping_epoch":431,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":432,"parent":"0.0","parent_split_bits":0,"last_scrub":"432'13","last_scrub_stamp":"2024-09-10T19:32:41.500514+0000","last_deep_scrub":"303'12","last_deep_scrub_stamp":"2024-09-10T19:31:20.099105+0000","last_clean_scrub_stamp":"2024-09-10T19:32:41.500514+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:34:02.582845+0000","scrub_duration":0.0024028790000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":556,"reported_epoch":459,"state":"active+clean","last_fresh":"2024-09-10T19:33:21.940021+0000","last_change":"2024-09-10T19:33:21.939966+0000","last_active":"2024-09-10T19:33:21.940021+0000","last_peered":"2024-09-10T19:33:21.940021+0000","last_clean":"2024-09-10T19:33:21.940021+0000","last_became_active":"2024-09-10T19:31:33.342272+0000","last_became_peered":"2024-09-10T19:31:33.342272+0000","last_unstale":"2024-09-10T19:33:21.940021+0000","last_undegraded":"2024-09-10T19:33:21.940021+0000","last_fullsized":"2024-09-10T19:33:21.940021+0000","mapping_epoch":431,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":432,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:33:21.939898+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:33:21.939898+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:34:51.337037+0000","scrub_duration":0.0016606660000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.a","version":"74'79","reported_seq":315,"reported_epoch":459,"state":"active+clean","last_fresh":"2024-09-10T19:33:02.226988+0000","last_change":"2024-09-10T19:32:54.538022+0000","last_active":"2024-09-10T19:33:02.226988+0000","last_peered":"2024-09-10T19:33:02.226988+0000","last_clean":"2024-09-10T19:33:02.226988+0000","last_became_active":"2024-09-10T19:31:33.349140+0000","last_became_peered":"2024-09-10T19:31:33.349140+0000","last_unstale":"2024-09-10T19:33:02.226988+0000","last_undegraded":"2024-09-10T19:33:02.226988+0000","last_fullsized":"2024-09-10T19:33:02.226988+0000","mapping_epoch":431,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":432,"parent":"0.0","parent_split_bits":4,"last_scrub":"74'79","last_scrub_stamp":"2024-09-10T19:32:54.537950+0000","last_deep_scrub":"74'79","last_deep_scrub_stamp":"2024-09-10T19:21:53.326433+0000","last_clean_scrub_stamp":"2024-09-10T19:32:54.537950+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:34:17.449988+0000","scrub_duration":0.0018094540000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.f","version":"414'39","reported_seq":252,"reported_epoch":460,"state":"active+clean","last_fresh":"2024-09-10T19:33:24.312249+0000","last_change":"2024-09-10T19:32:20.405218+0000","last_active":"2024-09-10T19:33:24.312249+0000","last_peered":"2024-09-10T19:33:24.312249+0000","last_clean":"2024-09-10T19:33:24.312249+0000","last_became_active":"2024-09-10T19:24:01.979187+0000","last_became_peered":"2024-09-10T19:24:01.979187+0000","last_unstale":"2024-09-10T19:33:24.312249+0000","last_undegraded":"2024-09-10T19:33:24.312249+0000","last_fullsized":"2024-09-10T19:33:24.312249+0000","mapping_epoch":292,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":293,"parent":"0.0","parent_split_bits":0,"last_scrub":"414'39","last_scrub_stamp":"2024-09-10T19:32:20.405155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:32:20.405155+0000","objects_scrubbed":7,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:33:34.076742+0000","scrub_duration":0.0020631650000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":61,"num_read_kb":52,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":218,"reported_epoch":460,"state":"active+clean","last_fresh":"2024-09-10T19:33:24.311854+0000","last_change":"2024-09-10T19:32:35.290354+0000","last_active":"2024-09-10T19:33:24.311854+0000","last_peered":"2024-09-10T19:33:24.311854+0000","last_clean":"2024-09-10T19:33:24.311854+0000","last_became_active":"2024-09-10T19:24:01.974773+0000","last_became_peered":"2024-09-10T19:24:01.974773+0000","last_unstale":"2024-09-10T19:33:24.311854+0000","last_undegraded":"2024-09-10T19:33:24.311854+0000","last_fullsized":"2024-09-10T19:33:24.311854+0000","mapping_epoch":292,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":293,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:32:35.290295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:28:24.420270+0000","last_clean_scrub_stamp":"2024-09-10T19:32:35.290295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:33:48.825101+0000","scrub_duration":0.0017394489999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.e","version":"433'11","reported_seq":795,"reported_epoch":459,"state":"active+clean","last_fresh":"2024-09-10T19:33:21.003873+0000","last_change":"2024-09-10T19:33:04.856855+0000","last_active":"2024-09-10T19:33:21.003873+0000","last_peered":"2024-09-10T19:33:21.003873+0000","last_clean":"2024-09-10T19:33:21.003873+0000","last_became_active":"2024-09-10T19:24:23.908760+0000","last_became_peered":"2024-09-10T19:24:23.908760+0000","last_unstale":"2024-09-10T19:33:21.003873+0000","last_undegraded":"2024-09-10T19:33:21.003873+0000","last_fullsized":"2024-09-10T19:33:21.003873+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"433'11","last_scrub_stamp":"2024-09-10T19:33:04.856780+0000","last_deep_scrub":"433'11","last_deep_scrub_stamp":"2024-09-10T19:31:44.714059+0000","last_clean_scrub_stamp":"2024-09-10T19:33:04.856780+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:34:26.622245+0000","scrub_duration":0.0019989629999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":542,"reported_epoch":459,"state":"active+clean","last_fresh":"2024-09-10T19:33:02.227326+0000","last_change":"2024-09-10T19:32:15.466055+0000","last_active":"2024-09-10T19:33:02.227326+0000","last_peered":"2024-09-10T19:33:02.227326+0000","last_clean":"2024-09-10T19:33:02.227326+0000","last_became_active":"2024-09-10T19:24:24.917109+0000","last_became_peered":"2024-09-10T19:24:24.917109+0000","last_unstale":"2024-09-10T19:33:02.227326+0000","last_undegraded":"2024-09-10T19:33:02.227326+0000","last_fullsized":"2024-09-10T19:33:02.227326+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:32:15.465990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:32:15.465990+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:33:40.503288+0000","scrub_duration":0.001701709,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.e","version":"414'32","reported_seq":675,"reported_epoch":459,"state":"active+clean","last_fresh":"2024-09-10T19:33:06.855806+0000","last_change":"2024-09-10T19:33:06.855729+0000","last_active":"2024-09-10T19:33:06.855806+0000","last_peered":"2024-09-10T19:33:06.855806+0000","last_clean":"2024-09-10T19:33:06.855806+0000","last_became_active":"2024-09-10T19:27:53.012101+0000","last_became_peered":"2024-09-10T19:27:53.012101+0000","last_unstale":"2024-09-10T19:33:06.855806+0000","last_undegraded":"2024-09-10T19:33:06.855806+0000","last_fullsized":"2024-09-10T19:33:06.855806+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"414'32","last_scrub_stamp":"2024-09-10T19:33:06.855661+0000","last_deep_scrub":"84'16","last_deep_scrub_stamp":"2024-09-10T19:11:56.157975+0000","last_clean_scrub_stamp":"2024-09-10T19:33:06.855661+0000","objects_scrubbed":5,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:34:33.242588+0000","scrub_duration":0.0020371830000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":37,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":550,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":191,"reported_epoch":459,"state":"active+clean","last_fresh":"2024-09-10T19:33:02.906962+0000","last_change":"2024-09-10T19:32:44.842610+0000","last_active":"2024-09-10T19:33:02.906962+0000","last_peered":"2024-09-10T19:33:02.906962+0000","last_clean":"2024-09-10T19:33:02.906962+0000","last_became_active":"2024-09-10T19:24:53.756220+0000","last_became_peered":"2024-09-10T19:24:53.756220+0000","last_unstale":"2024-09-10T19:33:02.906962+0000","last_undegraded":"2024-09-10T19:33:02.906962+0000","last_fullsized":"2024-09-10T19:33:02.906962+0000","mapping_epoch":318,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":319,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:32:44.842540+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:31:21.864910+0000","last_clean_scrub_stamp":"2024-09-10T19:32:44.842540+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:34:07.852289+0000","scrub_duration":0.001384515,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,3],"acting":[4,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":516,"reported_epoch":459,"state":"active+clean","last_fresh":"2024-09-10T19:33:13.545985+0000","last_change":"2024-09-10T19:33:13.545919+0000","last_active":"2024-09-10T19:33:13.545985+0000","last_peered":"2024-09-10T19:33:13.545985+0000","last_clean":"2024-09-10T19:33:13.545985+0000","last_became_active":"2024-09-10T19:27:53.007094+0000","last_became_peered":"2024-09-10T19:27:53.007094+0000","last_unstale":"2024-09-10T19:33:13.545985+0000","last_undegraded":"2024-09-10T19:33:13.545985+0000","last_fullsized":"2024-09-10T19:33:13.545985+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:33:13.545851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:33:13.545851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:34:17.009934+0000","scrub_duration":0.0014377470000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":504,"reported_epoch":459,"state":"active+clean","last_fresh":"2024-09-10T19:33:02.227329+0000","last_change":"2024-09-10T19:32:22.285635+0000","last_active":"2024-09-10T19:33:02.227329+0000","last_peered":"2024-09-10T19:33:02.227329+0000","last_clean":"2024-09-10T19:33:02.227329+0000","last_became_active":"2024-09-10T19:27:53.025345+0000","last_became_peered":"2024-09-10T19:27:53.025345+0000","last_unstale":"2024-09-10T19:33:02.227329+0000","last_undegraded":"2024-09-10T19:33:02.227329+0000","last_fullsized":"2024-09-10T19:33:02.227329+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:32:22.285577+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:32:22.285577+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:33:29.669292+0000","scrub_duration":0.0021926939999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.4","version":"317'80","reported_seq":291,"reported_epoch":459,"state":"active+clean","last_fresh":"2024-09-10T19:33:02.225773+0000","last_change":"2024-09-10T19:32:28.216883+0000","last_active":"2024-09-10T19:33:02.225773+0000","last_peered":"2024-09-10T19:33:02.225773+0000","last_clean":"2024-09-10T19:33:02.225773+0000","last_became_active":"2024-09-10T19:20:30.838441+0000","last_became_peered":"2024-09-10T19:20:30.838441+0000","last_unstale":"2024-09-10T19:33:02.225773+0000","last_undegraded":"2024-09-10T19:33:02.225773+0000","last_fullsized":"2024-09-10T19:33:02.225773+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":3,"last_scrub":"317'80","last_scrub_stamp":"2024-09-10T19:32:28.216823+0000","last_deep_scrub":"317'80","last_deep_scrub_stamp":"2024-09-10T19:24:51.540456+0000","last_clean_scrub_stamp":"2024-09-10T19:32:28.216823+0000","objects_scrubbed":1,"log_size":80,"log_dups_size":0,"ondisk_log_size":80,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:33:30.767417+0000","scrub_duration":0.0018724830000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"414'49","reported_seq":515,"reported_epoch":459,"state":"active+clean","last_fresh":"2024-09-10T19:33:08.800902+0000","last_change":"2024-09-10T19:33:08.800809+0000","last_active":"2024-09-10T19:33:08.800902+0000","last_peered":"2024-09-10T19:33:08.800902+0000","last_clean":"2024-09-10T19:33:08.800902+0000","last_became_active":"2024-09-10T19:27:53.011799+0000","last_became_peered":"2024-09-10T19:27:53.011799+0000","last_unstale":"2024-09-10T19:33:08.800902+0000","last_undegraded":"2024-09-10T19:33:08.800902+0000","last_fullsized":"2024-09-10T19:33:08.800902+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"414'49","last_scrub_stamp":"2024-09-10T19:33:08.800721+0000","last_deep_scrub":"414'49","last_deep_scrub_stamp":"2024-09-10T19:33:08.800721+0000","last_clean_scrub_stamp":"2024-09-10T19:33:08.800721+0000","objects_scrubbed":7,"log_size":49,"log_dups_size":0,"ondisk_log_size":49,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:34:11.843623+0000","scrub_duration":0.0024563380000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":63,"num_read_kb":56,"num_write":42,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":135,"reported_epoch":459,"state":"active+clean","last_fresh":"2024-09-10T19:33:02.858344+0000","last_change":"2024-09-10T19:32:20.731636+0000","last_active":"2024-09-10T19:33:02.858344+0000","last_peered":"2024-09-10T19:33:02.858344+0000","last_clean":"2024-09-10T19:33:02.858344+0000","last_became_active":"2024-09-10T19:31:33.338027+0000","last_became_peered":"2024-09-10T19:31:33.338027+0000","last_unstale":"2024-09-10T19:33:02.858344+0000","last_undegraded":"2024-09-10T19:33:02.858344+0000","last_fullsized":"2024-09-10T19:33:02.858344+0000","mapping_epoch":431,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":432,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:32:20.731566+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:32:20.731566+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:33:25.551017+0000","scrub_duration":0.0013631559999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.0","version":"417'13","reported_seq":818,"reported_epoch":459,"state":"active+clean","last_fresh":"2024-09-10T19:33:21.003612+0000","last_change":"2024-09-10T19:32:24.300782+0000","last_active":"2024-09-10T19:33:21.003612+0000","last_peered":"2024-09-10T19:33:21.003612+0000","last_clean":"2024-09-10T19:33:21.003612+0000","last_became_active":"2024-09-10T19:24:01.972645+0000","last_became_peered":"2024-09-10T19:24:01.972645+0000","last_unstale":"2024-09-10T19:33:21.003612+0000","last_undegraded":"2024-09-10T19:33:21.003612+0000","last_fullsized":"2024-09-10T19:33:21.003612+0000","mapping_epoch":292,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":293,"parent":"0.0","parent_split_bits":0,"last_scrub":"417'13","last_scrub_stamp":"2024-09-10T19:32:24.300723+0000","last_deep_scrub":"417'13","last_deep_scrub_stamp":"2024-09-10T19:30:39.964914+0000","last_clean_scrub_stamp":"2024-09-10T19:32:24.300723+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:33:29.321991+0000","scrub_duration":0.0017016399999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":551,"reported_epoch":459,"state":"active+clean","last_fresh":"2024-09-10T19:33:22.883151+0000","last_change":"2024-09-10T19:33:22.883103+0000","last_active":"2024-09-10T19:33:22.883151+0000","last_peered":"2024-09-10T19:33:22.883151+0000","last_clean":"2024-09-10T19:33:22.883151+0000","last_became_active":"2024-09-10T19:31:33.338017+0000","last_became_peered":"2024-09-10T19:31:33.338017+0000","last_unstale":"2024-09-10T19:33:22.883151+0000","last_undegraded":"2024-09-10T19:33:22.883151+0000","last_fullsized":"2024-09-10T19:33:22.883151+0000","mapping_epoch":431,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":432,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:33:22.883036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:28:25.822918+0000","last_clean_scrub_stamp":"2024-09-10T19:33:22.883036+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:34:31.415810+0000","scrub_duration":0.0014541980000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.5","version":"74'79","reported_seq":325,"reported_epoch":459,"state":"active+clean","last_fresh":"2024-09-10T19:33:02.907814+0000","last_change":"2024-09-10T19:32:57.763971+0000","last_active":"2024-09-10T19:33:02.907814+0000","last_peered":"2024-09-10T19:33:02.907814+0000","last_clean":"2024-09-10T19:33:02.907814+0000","last_became_active":"2024-09-10T19:31:33.339260+0000","last_became_peered":"2024-09-10T19:31:33.339260+0000","last_unstale":"2024-09-10T19:33:02.907814+0000","last_undegraded":"2024-09-10T19:33:02.907814+0000","last_fullsized":"2024-09-10T19:33:02.907814+0000","mapping_epoch":431,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":432,"parent":"0.0","parent_split_bits":3,"last_scrub":"74'79","last_scrub_stamp":"2024-09-10T19:32:57.763901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:08:00.699315+0000","last_clean_scrub_stamp":"2024-09-10T19:32:57.763901+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:34:26.852048+0000","scrub_duration":0.001528176,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"436'50","reported_seq":80,"reported_epoch":459,"state":"active+clean","last_fresh":"2024-09-10T19:33:02.227274+0000","last_change":"2024-09-10T19:33:01.502333+0000","last_active":"2024-09-10T19:33:02.227274+0000","last_peered":"2024-09-10T19:33:02.227274+0000","last_clean":"2024-09-10T19:33:02.227274+0000","last_became_active":"2024-09-10T19:31:47.391918+0000","last_became_peered":"2024-09-10T19:31:47.391918+0000","last_unstale":"2024-09-10T19:33:02.227274+0000","last_undegraded":"2024-09-10T19:33:02.227274+0000","last_fullsized":"2024-09-10T19:33:02.227274+0000","mapping_epoch":435,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":436,"parent":"0.0","parent_split_bits":0,"last_scrub":"436'50","last_scrub_stamp":"2024-09-10T19:33:01.502277+0000","last_deep_scrub":"436'50","last_deep_scrub_stamp":"2024-09-10T19:31:47.660982+0000","last_clean_scrub_stamp":"2024-09-10T19:33:01.502277+0000","objects_scrubbed":8,"log_size":50,"log_dups_size":0,"ondisk_log_size":50,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:34:15.160825+0000","scrub_duration":0.0019199830000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":69,"num_read_kb":57,"num_write":44,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,0],"acting":[2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":248,"reported_epoch":460,"state":"active+clean","last_fresh":"2024-09-10T19:33:24.312250+0000","last_change":"2024-09-10T19:33:08.232525+0000","last_active":"2024-09-10T19:33:24.312250+0000","last_peered":"2024-09-10T19:33:24.312250+0000","last_clean":"2024-09-10T19:33:24.312250+0000","last_became_active":"2024-09-10T19:27:53.008302+0000","last_became_peered":"2024-09-10T19:27:53.008302+0000","last_unstale":"2024-09-10T19:33:24.312250+0000","last_undegraded":"2024-09-10T19:33:24.312250+0000","last_fullsized":"2024-09-10T19:33:24.312250+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:33:08.232440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:31:58.291225+0000","last_clean_scrub_stamp":"2024-09-10T19:33:08.232440+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:34:38.130785+0000","scrub_duration":0.0016740069999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":505,"reported_epoch":459,"state":"active+clean","last_fresh":"2024-09-10T19:33:02.907267+0000","last_change":"2024-09-10T19:32:22.780163+0000","last_active":"2024-09-10T19:33:02.907267+0000","last_peered":"2024-09-10T19:33:02.907267+0000","last_clean":"2024-09-10T19:33:02.907267+0000","last_became_active":"2024-09-10T19:10:11.698054+0000","last_became_peered":"2024-09-10T19:10:11.698054+0000","last_unstale":"2024-09-10T19:33:02.907267+0000","last_undegraded":"2024-09-10T19:33:02.907267+0000","last_fullsized":"2024-09-10T19:33:02.907267+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:32:22.780098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:26:24.897519+0000","last_clean_scrub_stamp":"2024-09-10T19:32:22.780098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:33:23.483098+0000","scrub_duration":0.0013754450000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":543,"reported_epoch":459,"state":"active+clean","last_fresh":"2024-09-10T19:33:02.907501+0000","last_change":"2024-09-10T19:32:12.780125+0000","last_active":"2024-09-10T19:33:02.907501+0000","last_peered":"2024-09-10T19:33:02.907501+0000","last_clean":"2024-09-10T19:33:02.907501+0000","last_became_active":"2024-09-10T19:31:33.337265+0000","last_became_peered":"2024-09-10T19:31:33.337265+0000","last_unstale":"2024-09-10T19:33:02.907501+0000","last_undegraded":"2024-09-10T19:33:02.907501+0000","last_fullsized":"2024-09-10T19:33:02.907501+0000","mapping_epoch":431,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":432,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:32:12.780057+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:32:12.780057+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0014965810000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.6","version":"74'79","reported_seq":135,"reported_epoch":459,"state":"active+clean","last_fresh":"2024-09-10T19:33:02.858248+0000","last_change":"2024-09-10T19:33:01.489808+0000","last_active":"2024-09-10T19:33:02.858248+0000","last_peered":"2024-09-10T19:33:02.858248+0000","last_clean":"2024-09-10T19:33:02.858248+0000","last_became_active":"2024-09-10T19:27:53.017899+0000","last_became_peered":"2024-09-10T19:27:53.017899+0000","last_unstale":"2024-09-10T19:33:02.858248+0000","last_undegraded":"2024-09-10T19:33:02.858248+0000","last_fullsized":"2024-09-10T19:33:02.858248+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":3,"last_scrub":"74'79","last_scrub_stamp":"2024-09-10T19:33:01.489718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:08:00.699315+0000","last_clean_scrub_stamp":"2024-09-10T19:33:01.489718+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:34:27.307483+0000","scrub_duration":0.001384648,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"414'47","reported_seq":531,"reported_epoch":460,"state":"active+clean","last_fresh":"2024-09-10T19:33:24.309559+0000","last_change":"2024-09-10T19:33:01.332727+0000","last_active":"2024-09-10T19:33:24.309559+0000","last_peered":"2024-09-10T19:33:24.309559+0000","last_clean":"2024-09-10T19:33:24.309559+0000","last_became_active":"2024-09-10T19:24:23.901836+0000","last_became_peered":"2024-09-10T19:24:23.901836+0000","last_unstale":"2024-09-10T19:33:24.309559+0000","last_undegraded":"2024-09-10T19:33:24.309559+0000","last_fullsized":"2024-09-10T19:33:24.309559+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"414'47","last_scrub_stamp":"2024-09-10T19:33:01.332621+0000","last_deep_scrub":"104'27","last_deep_scrub_stamp":"2024-09-10T19:13:02.225944+0000","last_clean_scrub_stamp":"2024-09-10T19:33:01.332621+0000","objects_scrubbed":8,"log_size":47,"log_dups_size":0,"ondisk_log_size":47,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:34:10.981100+0000","scrub_duration":0.0023253050000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":68,"num_read_kb":54,"num_write":42,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":527,"reported_epoch":459,"state":"active+clean","last_fresh":"2024-09-10T19:33:11.812677+0000","last_change":"2024-09-10T19:33:11.812623+0000","last_active":"2024-09-10T19:33:11.812677+0000","last_peered":"2024-09-10T19:33:11.812677+0000","last_clean":"2024-09-10T19:33:11.812677+0000","last_became_active":"2024-09-10T19:24:23.899556+0000","last_became_peered":"2024-09-10T19:24:23.899556+0000","last_unstale":"2024-09-10T19:33:11.812677+0000","last_undegraded":"2024-09-10T19:33:11.812677+0000","last_fullsized":"2024-09-10T19:33:11.812677+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:33:11.812552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:16:37.779967+0000","last_clean_scrub_stamp":"2024-09-10T19:33:11.812552+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:34:41.688836+0000","scrub_duration":0.0016932640000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":135,"reported_epoch":460,"state":"active+clean","last_fresh":"2024-09-10T19:33:24.345737+0000","last_change":"2024-09-10T19:33:24.345706+0000","last_active":"2024-09-10T19:33:24.345737+0000","last_peered":"2024-09-10T19:33:24.345737+0000","last_clean":"2024-09-10T19:33:24.345737+0000","last_became_active":"2024-09-10T19:27:53.003734+0000","last_became_peered":"2024-09-10T19:27:53.003734+0000","last_unstale":"2024-09-10T19:33:24.345737+0000","last_undegraded":"2024-09-10T19:33:24.345737+0000","last_fullsized":"2024-09-10T19:33:24.345737+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:33:24.345648+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:33:24.345648+0000","last_clean_scrub_stamp":"2024-09-10T19:33:24.345648+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:34:51.577248+0000","scrub_duration":0.001489612,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":258,"reported_epoch":460,"state":"active+clean","last_fresh":"2024-09-10T19:33:24.311837+0000","last_change":"2024-09-10T19:33:04.243676+0000","last_active":"2024-09-10T19:33:24.311837+0000","last_peered":"2024-09-10T19:33:24.311837+0000","last_clean":"2024-09-10T19:33:24.311837+0000","last_became_active":"2024-09-10T19:31:33.342013+0000","last_became_peered":"2024-09-10T19:31:33.342013+0000","last_unstale":"2024-09-10T19:33:24.311837+0000","last_undegraded":"2024-09-10T19:33:24.311837+0000","last_fullsized":"2024-09-10T19:33:24.311837+0000","mapping_epoch":431,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":432,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:33:04.243576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:31:42.343083+0000","last_clean_scrub_stamp":"2024-09-10T19:33:04.243576+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:34:33.905888+0000","scrub_duration":0.0014906159999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"74'79","reported_seq":135,"reported_epoch":459,"state":"active+clean","last_fresh":"2024-09-10T19:33:17.381031+0000","last_change":"2024-09-10T19:33:17.380952+0000","last_active":"2024-09-10T19:33:17.381031+0000","last_peered":"2024-09-10T19:33:17.381031+0000","last_clean":"2024-09-10T19:33:17.381031+0000","last_became_active":"2024-09-10T19:27:53.020085+0000","last_became_peered":"2024-09-10T19:27:53.020085+0000","last_unstale":"2024-09-10T19:33:17.381031+0000","last_undegraded":"2024-09-10T19:33:17.381031+0000","last_fullsized":"2024-09-10T19:33:17.381031+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":3,"last_scrub":"74'79","last_scrub_stamp":"2024-09-10T19:33:17.380886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:08:00.699315+0000","last_clean_scrub_stamp":"2024-09-10T19:33:17.380886+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:34:33.357173+0000","scrub_duration":0.0014656389999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.2","version":"414'35","reported_seq":260,"reported_epoch":460,"state":"active+clean","last_fresh":"2024-09-10T19:33:24.312166+0000","last_change":"2024-09-10T19:32:46.261518+0000","last_active":"2024-09-10T19:33:24.312166+0000","last_peered":"2024-09-10T19:33:24.312166+0000","last_clean":"2024-09-10T19:33:24.312166+0000","last_became_active":"2024-09-10T19:24:01.976841+0000","last_became_peered":"2024-09-10T19:24:01.976841+0000","last_unstale":"2024-09-10T19:33:24.312166+0000","last_undegraded":"2024-09-10T19:33:24.312166+0000","last_fullsized":"2024-09-10T19:33:24.312166+0000","mapping_epoch":292,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":293,"parent":"0.0","parent_split_bits":0,"last_scrub":"414'35","last_scrub_stamp":"2024-09-10T19:32:46.261462+0000","last_deep_scrub":"414'35","last_deep_scrub_stamp":"2024-09-10T19:31:20.479343+0000","last_clean_scrub_stamp":"2024-09-10T19:32:46.261462+0000","objects_scrubbed":5,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:34:02.032280+0000","scrub_duration":0.0021662119999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":38,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":242,"reported_epoch":460,"state":"active+clean","last_fresh":"2024-09-10T19:33:24.312111+0000","last_change":"2024-09-10T19:33:06.213005+0000","last_active":"2024-09-10T19:33:24.312111+0000","last_peered":"2024-09-10T19:33:24.312111+0000","last_clean":"2024-09-10T19:33:24.312111+0000","last_became_active":"2024-09-10T19:24:24.910173+0000","last_became_peered":"2024-09-10T19:24:24.910173+0000","last_unstale":"2024-09-10T19:33:24.312111+0000","last_undegraded":"2024-09-10T19:33:24.312111+0000","last_fullsized":"2024-09-10T19:33:24.312111+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:33:06.212952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:33:06.212952+0000","last_clean_scrub_stamp":"2024-09-10T19:33:06.212952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:34:24.192124+0000","scrub_duration":0.0014953430000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"376'13","reported_seq":741,"reported_epoch":460,"state":"active+clean","last_fresh":"2024-09-10T19:33:26.004493+0000","last_change":"2024-09-10T19:32:54.273571+0000","last_active":"2024-09-10T19:33:26.004493+0000","last_peered":"2024-09-10T19:33:26.004493+0000","last_clean":"2024-09-10T19:33:26.004493+0000","last_became_active":"2024-09-10T19:27:53.000311+0000","last_became_peered":"2024-09-10T19:27:53.000311+0000","last_unstale":"2024-09-10T19:33:26.004493+0000","last_undegraded":"2024-09-10T19:33:26.004493+0000","last_fullsized":"2024-09-10T19:33:26.004493+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"376'13","last_scrub_stamp":"2024-09-10T19:32:54.273510+0000","last_deep_scrub":"376'13","last_deep_scrub_stamp":"2024-09-10T19:28:17.490297+0000","last_clean_scrub_stamp":"2024-09-10T19:32:54.273510+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:34:18.709551+0000","scrub_duration":0.0020515640000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":491,"reported_epoch":460,"state":"active+clean","last_fresh":"2024-09-10T19:33:25.321194+0000","last_change":"2024-09-10T19:33:25.321140+0000","last_active":"2024-09-10T19:33:25.321194+0000","last_peered":"2024-09-10T19:33:25.321194+0000","last_clean":"2024-09-10T19:33:25.321194+0000","last_became_active":"2024-09-10T19:31:33.335255+0000","last_became_peered":"2024-09-10T19:31:33.335255+0000","last_unstale":"2024-09-10T19:33:25.321194+0000","last_undegraded":"2024-09-10T19:33:25.321194+0000","last_fullsized":"2024-09-10T19:33:25.321194+0000","mapping_epoch":431,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":432,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:33:25.321069+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:33:25.321069+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:34:53.518637+0000","scrub_duration":0.0016760519999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"74'79","reported_seq":302,"reported_epoch":459,"state":"active+clean","last_fresh":"2024-09-10T19:33:02.226827+0000","last_change":"2024-09-10T19:32:43.488435+0000","last_active":"2024-09-10T19:33:02.226827+0000","last_peered":"2024-09-10T19:33:02.226827+0000","last_clean":"2024-09-10T19:33:02.226827+0000","last_became_active":"2024-09-10T19:31:33.338083+0000","last_became_peered":"2024-09-10T19:31:33.338083+0000","last_unstale":"2024-09-10T19:33:02.226827+0000","last_undegraded":"2024-09-10T19:33:02.226827+0000","last_fullsized":"2024-09-10T19:33:02.226827+0000","mapping_epoch":431,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":432,"parent":"0.0","parent_split_bits":4,"last_scrub":"74'79","last_scrub_stamp":"2024-09-10T19:32:43.488362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:08:00.699315+0000","last_clean_scrub_stamp":"2024-09-10T19:32:43.488362+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:34:06.539920+0000","scrub_duration":0.001671467,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"432'69","reported_seq":477,"reported_epoch":460,"state":"active+clean","last_fresh":"2024-09-10T19:33:26.004632+0000","last_change":"2024-09-10T19:32:09.310933+0000","last_active":"2024-09-10T19:33:26.004632+0000","last_peered":"2024-09-10T19:33:26.004632+0000","last_clean":"2024-09-10T19:33:26.004632+0000","last_became_active":"2024-09-10T19:31:33.348744+0000","last_became_peered":"2024-09-10T19:31:33.348744+0000","last_unstale":"2024-09-10T19:33:26.004632+0000","last_undegraded":"2024-09-10T19:33:26.004632+0000","last_fullsized":"2024-09-10T19:33:26.004632+0000","mapping_epoch":431,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":432,"parent":"0.0","parent_split_bits":0,"last_scrub":"432'69","last_scrub_stamp":"2024-09-10T19:32:09.310870+0000","last_deep_scrub":"384'56","last_deep_scrub_stamp":"2024-09-10T19:28:23.457119+0000","last_clean_scrub_stamp":"2024-09-10T19:32:09.310870+0000","objects_scrubbed":10,"log_size":69,"log_dups_size":0,"ondisk_log_size":69,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:33:37.960731+0000","scrub_duration":0.0021013970000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":83,"num_read_kb":68,"num_write":50,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":50,"num_bytes_recovered":1790,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":544,"reported_epoch":459,"state":"active+clean","last_fresh":"2024-09-10T19:33:02.226933+0000","last_change":"2024-09-10T19:32:09.420611+0000","last_active":"2024-09-10T19:33:02.226933+0000","last_peered":"2024-09-10T19:33:02.226933+0000","last_clean":"2024-09-10T19:33:02.226933+0000","last_became_active":"2024-09-10T19:27:53.004220+0000","last_became_peered":"2024-09-10T19:27:53.004220+0000","last_unstale":"2024-09-10T19:33:02.226933+0000","last_undegraded":"2024-09-10T19:33:02.226933+0000","last_fullsized":"2024-09-10T19:33:02.226933+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:32:09.420540+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:32:09.420540+0000","last_clean_scrub_stamp":"2024-09-10T19:32:09.420540+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001628226,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":471,"reported_epoch":460,"state":"active+clean","last_fresh":"2024-09-10T19:33:24.310083+0000","last_change":"2024-09-10T19:32:01.057536+0000","last_active":"2024-09-10T19:33:24.310083+0000","last_peered":"2024-09-10T19:33:24.310083+0000","last_clean":"2024-09-10T19:33:24.310083+0000","last_became_active":"2024-09-10T19:24:24.892692+0000","last_became_peered":"2024-09-10T19:24:24.892692+0000","last_unstale":"2024-09-10T19:33:24.310083+0000","last_undegraded":"2024-09-10T19:33:24.310083+0000","last_fullsized":"2024-09-10T19:33:24.310083+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:32:01.057482+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:32:01.057482+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:33:28.458880+0000","scrub_duration":0.001292009,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":484,"reported_epoch":459,"state":"active+clean","last_fresh":"2024-09-10T19:33:02.227381+0000","last_change":"2024-09-10T19:32:41.052328+0000","last_active":"2024-09-10T19:33:02.227381+0000","last_peered":"2024-09-10T19:33:02.227381+0000","last_clean":"2024-09-10T19:33:02.227381+0000","last_became_active":"2024-09-10T19:10:13.721744+0000","last_became_peered":"2024-09-10T19:10:13.721744+0000","last_unstale":"2024-09-10T19:33:02.227381+0000","last_undegraded":"2024-09-10T19:33:02.227381+0000","last_fullsized":"2024-09-10T19:33:02.227381+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:32:41.052274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:32:41.052274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:33:51.209533+0000","scrub_duration":0.0012557759999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"74'79","reported_seq":582,"reported_epoch":459,"state":"active+clean","last_fresh":"2024-09-10T19:33:02.907317+0000","last_change":"2024-09-10T19:32:23.941209+0000","last_active":"2024-09-10T19:33:02.907317+0000","last_peered":"2024-09-10T19:33:02.907317+0000","last_clean":"2024-09-10T19:33:02.907317+0000","last_became_active":"2024-09-10T19:24:23.895247+0000","last_became_peered":"2024-09-10T19:24:23.895247+0000","last_unstale":"2024-09-10T19:33:02.907317+0000","last_undegraded":"2024-09-10T19:33:02.907317+0000","last_fullsized":"2024-09-10T19:33:02.907317+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":4,"last_scrub":"74'79","last_scrub_stamp":"2024-09-10T19:32:23.941133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:08:00.699315+0000","last_clean_scrub_stamp":"2024-09-10T19:32:23.941133+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:33:52.738059+0000","scrub_duration":0.0015017470000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.7","version":"414'26","reported_seq":260,"reported_epoch":460,"state":"active+clean","last_fresh":"2024-09-10T19:33:24.312153+0000","last_change":"2024-09-10T19:32:59.306088+0000","last_active":"2024-09-10T19:33:24.312153+0000","last_peered":"2024-09-10T19:33:24.312153+0000","last_clean":"2024-09-10T19:33:24.312153+0000","last_became_active":"2024-09-10T19:24:24.894293+0000","last_became_peered":"2024-09-10T19:24:24.894293+0000","last_unstale":"2024-09-10T19:33:24.312153+0000","last_undegraded":"2024-09-10T19:33:24.312153+0000","last_fullsized":"2024-09-10T19:33:24.312153+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"414'26","last_scrub_stamp":"2024-09-10T19:32:59.306018+0000","last_deep_scrub":"82'18","last_deep_scrub_stamp":"2024-09-10T19:15:55.777361+0000","last_clean_scrub_stamp":"2024-09-10T19:32:59.306018+0000","objects_scrubbed":5,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:34:29.066767+0000","scrub_duration":0.0052662430000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":42,"num_read_kb":31,"num_write":24,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"159'6","reported_seq":849,"reported_epoch":461,"state":"active+clean","last_fresh":"2024-09-10T19:33:26.535668+0000","last_change":"2024-09-10T19:32:21.431132+0000","last_active":"2024-09-10T19:33:26.535668+0000","last_peered":"2024-09-10T19:33:26.535668+0000","last_clean":"2024-09-10T19:33:26.535668+0000","last_became_active":"2024-09-10T19:31:33.337696+0000","last_became_peered":"2024-09-10T19:31:33.337696+0000","last_unstale":"2024-09-10T19:33:26.535668+0000","last_undegraded":"2024-09-10T19:33:26.535668+0000","last_fullsized":"2024-09-10T19:33:26.535668+0000","mapping_epoch":431,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":432,"parent":"0.0","parent_split_bits":0,"last_scrub":"159'6","last_scrub_stamp":"2024-09-10T19:32:21.431038+0000","last_deep_scrub":"159'6","last_deep_scrub_stamp":"2024-09-10T19:16:32.649914+0000","last_clean_scrub_stamp":"2024-09-10T19:32:21.431038+0000","objects_scrubbed":1,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:33:30.035389+0000","scrub_duration":0.0019491409999999999,"objects_trimmed":0,"snaptrim_duration":0.00034454499999999997,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":565,"num_read_kb":560,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":1945,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":48,"reported_epoch":459,"state":"active+clean","last_fresh":"2024-09-10T19:33:02.225871+0000","last_change":"2024-09-10T19:32:00.538868+0000","last_active":"2024-09-10T19:33:02.225871+0000","last_peered":"2024-09-10T19:33:02.225871+0000","last_clean":"2024-09-10T19:33:02.225871+0000","last_became_active":"2024-09-10T19:31:33.339737+0000","last_became_peered":"2024-09-10T19:31:33.339737+0000","last_unstale":"2024-09-10T19:33:02.225871+0000","last_undegraded":"2024-09-10T19:33:02.225871+0000","last_fullsized":"2024-09-10T19:33:02.225871+0000","mapping_epoch":431,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":432,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:32:00.538807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:28:21.199065+0000","last_clean_scrub_stamp":"2024-09-10T19:32:00.538807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0012599849999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":58,"reported_epoch":459,"state":"active+clean","last_fresh":"2024-09-10T19:33:20.712455+0000","last_change":"2024-09-10T19:33:20.712421+0000","last_active":"2024-09-10T19:33:20.712455+0000","last_peered":"2024-09-10T19:33:20.712455+0000","last_clean":"2024-09-10T19:33:20.712455+0000","last_became_active":"2024-09-10T19:31:33.347689+0000","last_became_peered":"2024-09-10T19:31:33.347689+0000","last_unstale":"2024-09-10T19:33:20.712455+0000","last_undegraded":"2024-09-10T19:33:20.712455+0000","last_fullsized":"2024-09-10T19:33:20.712455+0000","mapping_epoch":431,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":432,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:33:20.712370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:33:20.712370+0000","last_clean_scrub_stamp":"2024-09-10T19:33:20.712370+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:34:35.235170+0000","scrub_duration":0.0013864369999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":533,"reported_epoch":459,"state":"active+clean","last_fresh":"2024-09-10T19:33:14.818773+0000","last_change":"2024-09-10T19:33:14.818692+0000","last_active":"2024-09-10T19:33:14.818773+0000","last_peered":"2024-09-10T19:33:14.818773+0000","last_clean":"2024-09-10T19:33:14.818773+0000","last_became_active":"2024-09-10T19:27:53.011923+0000","last_became_peered":"2024-09-10T19:27:53.011923+0000","last_unstale":"2024-09-10T19:33:14.818773+0000","last_undegraded":"2024-09-10T19:33:14.818773+0000","last_fullsized":"2024-09-10T19:33:14.818773+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:33:14.818620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:22:01.208272+0000","last_clean_scrub_stamp":"2024-09-10T19:33:14.818620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:34:36.076319+0000","scrub_duration":0.0016019960000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.3","version":"74'79","reported_seq":238,"reported_epoch":460,"state":"active+clean","last_fresh":"2024-09-10T19:33:24.312173+0000","last_change":"2024-09-10T19:32:05.334159+0000","last_active":"2024-09-10T19:33:24.312173+0000","last_peered":"2024-09-10T19:33:24.312173+0000","last_clean":"2024-09-10T19:33:24.312173+0000","last_became_active":"2024-09-10T19:27:53.016965+0000","last_became_peered":"2024-09-10T19:27:53.016965+0000","last_unstale":"2024-09-10T19:33:24.312173+0000","last_undegraded":"2024-09-10T19:33:24.312173+0000","last_fullsized":"2024-09-10T19:33:24.312173+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":3,"last_scrub":"74'79","last_scrub_stamp":"2024-09-10T19:32:05.334106+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:08:00.699315+0000","last_clean_scrub_stamp":"2024-09-10T19:32:05.334106+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001553776,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"414'42","reported_seq":603,"reported_epoch":460,"state":"active+clean","last_fresh":"2024-09-10T19:33:24.310216+0000","last_change":"2024-09-10T19:32:44.224279+0000","last_active":"2024-09-10T19:33:24.310216+0000","last_peered":"2024-09-10T19:33:24.310216+0000","last_clean":"2024-09-10T19:33:24.310216+0000","last_became_active":"2024-09-10T19:31:33.334775+0000","last_became_peered":"2024-09-10T19:31:33.334775+0000","last_unstale":"2024-09-10T19:33:24.310216+0000","last_undegraded":"2024-09-10T19:33:24.310216+0000","last_fullsized":"2024-09-10T19:33:24.310216+0000","mapping_epoch":431,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":432,"parent":"0.0","parent_split_bits":0,"last_scrub":"414'42","last_scrub_stamp":"2024-09-10T19:32:44.224189+0000","last_deep_scrub":"221'18","last_deep_scrub_stamp":"2024-09-10T19:19:29.606748+0000","last_clean_scrub_stamp":"2024-09-10T19:32:44.224189+0000","objects_scrubbed":6,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:34:04.064176+0000","scrub_duration":0.0016931520000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":53,"num_read_kb":47,"num_write":36,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"417'2","reported_seq":465,"reported_epoch":460,"state":"active+clean","last_fresh":"2024-09-10T19:33:24.309783+0000","last_change":"2024-09-10T19:32:10.020662+0000","last_active":"2024-09-10T19:33:24.309783+0000","last_peered":"2024-09-10T19:33:24.309783+0000","last_clean":"2024-09-10T19:33:24.309783+0000","last_became_active":"2024-09-10T19:24:23.897412+0000","last_became_peered":"2024-09-10T19:24:23.897412+0000","last_unstale":"2024-09-10T19:33:24.309783+0000","last_undegraded":"2024-09-10T19:33:24.309783+0000","last_fullsized":"2024-09-10T19:33:24.309783+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"417'2","last_scrub_stamp":"2024-09-10T19:32:10.020604+0000","last_deep_scrub":"417'2","last_deep_scrub_stamp":"2024-09-10T19:30:39.904339+0000","last_clean_scrub_stamp":"2024-09-10T19:32:10.020604+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:33:38.047182+0000","scrub_duration":0.0023223800000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":548,"reported_epoch":459,"state":"active+clean","last_fresh":"2024-09-10T19:33:09.836697+0000","last_change":"2024-09-10T19:33:09.836643+0000","last_active":"2024-09-10T19:33:09.836697+0000","last_peered":"2024-09-10T19:33:09.836697+0000","last_clean":"2024-09-10T19:33:09.836697+0000","last_became_active":"2024-09-10T19:24:24.896201+0000","last_became_peered":"2024-09-10T19:24:24.896201+0000","last_unstale":"2024-09-10T19:33:09.836697+0000","last_undegraded":"2024-09-10T19:33:09.836697+0000","last_fullsized":"2024-09-10T19:33:09.836697+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:33:09.836575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:31:47.151810+0000","last_clean_scrub_stamp":"2024-09-10T19:33:09.836575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:34:12.039746+0000","scrub_duration":0.0014646150000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":554,"reported_epoch":459,"state":"active+clean","last_fresh":"2024-09-10T19:33:20.544423+0000","last_change":"2024-09-10T19:33:20.544352+0000","last_active":"2024-09-10T19:33:20.544423+0000","last_peered":"2024-09-10T19:33:20.544423+0000","last_clean":"2024-09-10T19:33:20.544423+0000","last_became_active":"2024-09-10T19:24:24.914680+0000","last_became_peered":"2024-09-10T19:24:24.914680+0000","last_unstale":"2024-09-10T19:33:20.544423+0000","last_undegraded":"2024-09-10T19:33:20.544423+0000","last_fullsized":"2024-09-10T19:33:20.544423+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:33:20.544240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:33:20.544240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:34:32.234540+0000","scrub_duration":0.001710488,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":218,"reported_epoch":460,"state":"active+clean","last_fresh":"2024-09-10T19:33:24.311979+0000","last_change":"2024-09-10T19:31:59.247873+0000","last_active":"2024-09-10T19:33:24.311979+0000","last_peered":"2024-09-10T19:33:24.311979+0000","last_clean":"2024-09-10T19:33:24.311979+0000","last_became_active":"2024-09-10T19:24:01.974039+0000","last_became_peered":"2024-09-10T19:24:01.974039+0000","last_unstale":"2024-09-10T19:33:24.311979+0000","last_undegraded":"2024-09-10T19:33:24.311979+0000","last_fullsized":"2024-09-10T19:33:24.311979+0000","mapping_epoch":292,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":293,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:31:59.247819+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:31:59.247819+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001632283,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"414'45","reported_seq":169,"reported_epoch":459,"state":"active+clean","last_fresh":"2024-09-10T19:33:21.338419+0000","last_change":"2024-09-10T19:33:21.338362+0000","last_active":"2024-09-10T19:33:21.338419+0000","last_peered":"2024-09-10T19:33:21.338419+0000","last_clean":"2024-09-10T19:33:21.338419+0000","last_became_active":"2024-09-10T19:27:53.006290+0000","last_became_peered":"2024-09-10T19:27:53.006290+0000","last_unstale":"2024-09-10T19:33:21.338419+0000","last_undegraded":"2024-09-10T19:33:21.338419+0000","last_fullsized":"2024-09-10T19:33:21.338419+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"414'45","last_scrub_stamp":"2024-09-10T19:33:21.338288+0000","last_deep_scrub":"80'21","last_deep_scrub_stamp":"2024-09-10T19:11:39.836144+0000","last_clean_scrub_stamp":"2024-09-10T19:33:21.338288+0000","objects_scrubbed":7,"log_size":45,"log_dups_size":0,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:34:35.531323+0000","scrub_duration":0.0022233510000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":60,"num_read_kb":50,"num_write":37,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":154,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"446'3","reported_seq":536,"reported_epoch":459,"state":"active+clean","last_fresh":"2024-09-10T19:33:02.226169+0000","last_change":"2024-09-10T19:32:17.358949+0000","last_active":"2024-09-10T19:33:02.226169+0000","last_peered":"2024-09-10T19:33:02.226169+0000","last_clean":"2024-09-10T19:33:02.226169+0000","last_became_active":"2024-09-10T19:27:53.025802+0000","last_became_peered":"2024-09-10T19:27:53.025802+0000","last_unstale":"2024-09-10T19:33:02.226169+0000","last_undegraded":"2024-09-10T19:33:02.226169+0000","last_fullsized":"2024-09-10T19:33:02.226169+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"446'3","last_scrub_stamp":"2024-09-10T19:32:17.358880+0000","last_deep_scrub":"446'3","last_deep_scrub_stamp":"2024-09-10T19:32:17.358880+0000","last_clean_scrub_stamp":"2024-09-10T19:32:17.358880+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:33:33.552976+0000","scrub_duration":0.005695248,"objects_trimmed":0,"snaptrim_duration":0.00075522999999999996,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":530,"reported_epoch":459,"state":"active+clean","last_fresh":"2024-09-10T19:33:09.529208+0000","last_change":"2024-09-10T19:33:09.529149+0000","last_active":"2024-09-10T19:33:09.529208+0000","last_peered":"2024-09-10T19:33:09.529208+0000","last_clean":"2024-09-10T19:33:09.529208+0000","last_became_active":"2024-09-10T19:31:33.349216+0000","last_became_peered":"2024-09-10T19:31:33.349216+0000","last_unstale":"2024-09-10T19:33:09.529208+0000","last_undegraded":"2024-09-10T19:33:09.529208+0000","last_fullsized":"2024-09-10T19:33:09.529208+0000","mapping_epoch":431,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":432,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:33:09.529079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:33:09.529079+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:34:18.773549+0000","scrub_duration":0.0015652789999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"74'79","reported_seq":596,"reported_epoch":459,"state":"active+clean","last_fresh":"2024-09-10T19:33:02.907679+0000","last_change":"2024-09-10T19:33:01.852337+0000","last_active":"2024-09-10T19:33:02.907679+0000","last_peered":"2024-09-10T19:33:02.907679+0000","last_clean":"2024-09-10T19:33:02.907679+0000","last_became_active":"2024-09-10T19:27:53.009367+0000","last_became_peered":"2024-09-10T19:27:53.009367+0000","last_unstale":"2024-09-10T19:33:02.907679+0000","last_undegraded":"2024-09-10T19:33:02.907679+0000","last_fullsized":"2024-09-10T19:33:02.907679+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'79","last_scrub_stamp":"2024-09-10T19:33:01.852266+0000","last_deep_scrub":"74'79","last_deep_scrub_stamp":"2024-09-10T19:31:25.002382+0000","last_clean_scrub_stamp":"2024-09-10T19:33:01.852266+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:34:04.920975+0000","scrub_duration":0.0014929170000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":533,"reported_epoch":459,"state":"active+clean","last_fresh":"2024-09-10T19:33:02.908476+0000","last_change":"2024-09-10T19:31:54.175652+0000","last_active":"2024-09-10T19:33:02.908476+0000","last_peered":"2024-09-10T19:33:02.908476+0000","last_clean":"2024-09-10T19:33:02.908476+0000","last_became_active":"2024-09-10T19:31:33.339073+0000","last_became_peered":"2024-09-10T19:31:33.339073+0000","last_unstale":"2024-09-10T19:33:02.908476+0000","last_undegraded":"2024-09-10T19:33:02.908476+0000","last_fullsized":"2024-09-10T19:33:02.908476+0000","mapping_epoch":431,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":432,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:31:54.175551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:31:54.175551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0015182080000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":494,"reported_epoch":459,"state":"active+clean","last_fresh":"2024-09-10T19:33:02.227862+0000","last_change":"2024-09-10T19:32:02.509629+0000","last_active":"2024-09-10T19:33:02.227862+0000","last_peered":"2024-09-10T19:33:02.227862+0000","last_clean":"2024-09-10T19:33:02.227862+0000","last_became_active":"2024-09-10T19:10:13.720647+0000","last_became_peered":"2024-09-10T19:10:13.720647+0000","last_unstale":"2024-09-10T19:33:02.227862+0000","last_undegraded":"2024-09-10T19:33:02.227862+0000","last_fullsized":"2024-09-10T19:33:02.227862+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:32:02.509565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:28:29.899579+0000","last_clean_scrub_stamp":"2024-09-10T19:32:02.509565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:33:26.850660+0000","scrub_duration":0.0017406069999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.a","version":"414'51","reported_seq":77,"reported_epoch":459,"state":"active+clean","last_fresh":"2024-09-10T19:33:02.226089+0000","last_change":"2024-09-10T19:32:00.568977+0000","last_active":"2024-09-10T19:33:02.226089+0000","last_peered":"2024-09-10T19:33:02.226089+0000","last_clean":"2024-09-10T19:33:02.226089+0000","last_became_active":"2024-09-10T19:31:33.337763+0000","last_became_peered":"2024-09-10T19:31:33.337763+0000","last_unstale":"2024-09-10T19:33:02.226089+0000","last_undegraded":"2024-09-10T19:33:02.226089+0000","last_fullsized":"2024-09-10T19:33:02.226089+0000","mapping_epoch":431,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":432,"parent":"0.0","parent_split_bits":0,"last_scrub":"414'51","last_scrub_stamp":"2024-09-10T19:32:00.568913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:32:00.568913+0000","objects_scrubbed":9,"log_size":51,"log_dups_size":0,"ondisk_log_size":51,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:33:27.261721+0000","scrub_duration":0.0018776369999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":79,"num_read_kb":68,"num_write":52,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":36,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":48,"reported_epoch":459,"state":"active+clean","last_fresh":"2024-09-10T19:33:07.614786+0000","last_change":"2024-09-10T19:33:07.614745+0000","last_active":"2024-09-10T19:33:07.614786+0000","last_peered":"2024-09-10T19:33:07.614786+0000","last_clean":"2024-09-10T19:33:07.614786+0000","last_became_active":"2024-09-10T19:31:33.343515+0000","last_became_peered":"2024-09-10T19:31:33.343515+0000","last_unstale":"2024-09-10T19:33:07.614786+0000","last_undegraded":"2024-09-10T19:33:07.614786+0000","last_fullsized":"2024-09-10T19:33:07.614786+0000","mapping_epoch":431,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":432,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:33:07.614692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:31:25.719972+0000","last_clean_scrub_stamp":"2024-09-10T19:33:07.614692+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:34:25.529285+0000","scrub_duration":0.001709704,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":58,"reported_epoch":459,"state":"active+clean","last_fresh":"2024-09-10T19:33:14.709282+0000","last_change":"2024-09-10T19:33:14.709245+0000","last_active":"2024-09-10T19:33:14.709282+0000","last_peered":"2024-09-10T19:33:14.709282+0000","last_clean":"2024-09-10T19:33:14.709282+0000","last_became_active":"2024-09-10T19:31:33.340592+0000","last_became_peered":"2024-09-10T19:31:33.340592+0000","last_unstale":"2024-09-10T19:33:14.709282+0000","last_undegraded":"2024-09-10T19:33:14.709282+0000","last_fullsized":"2024-09-10T19:33:14.709282+0000","mapping_epoch":431,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":432,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:33:14.709190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:33:14.709190+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:34:39.599665+0000","scrub_duration":0.0013793360000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":554,"reported_epoch":459,"state":"active+clean","last_fresh":"2024-09-10T19:33:06.844051+0000","last_change":"2024-09-10T19:33:06.843999+0000","last_active":"2024-09-10T19:33:06.844051+0000","last_peered":"2024-09-10T19:33:06.844051+0000","last_clean":"2024-09-10T19:33:06.844051+0000","last_became_active":"2024-09-10T19:31:33.337545+0000","last_became_peered":"2024-09-10T19:31:33.337545+0000","last_unstale":"2024-09-10T19:33:06.844051+0000","last_undegraded":"2024-09-10T19:33:06.844051+0000","last_fullsized":"2024-09-10T19:33:06.844051+0000","mapping_epoch":431,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":432,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:33:06.843928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:21:03.910074+0000","last_clean_scrub_stamp":"2024-09-10T19:33:06.843928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:34:32.780150+0000","scrub_duration":0.001549398,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"414'21","reported_seq":554,"reported_epoch":460,"state":"active+clean","last_fresh":"2024-09-10T19:33:24.309655+0000","last_change":"2024-09-10T19:33:03.305872+0000","last_active":"2024-09-10T19:33:24.309655+0000","last_peered":"2024-09-10T19:33:24.309655+0000","last_clean":"2024-09-10T19:33:24.309655+0000","last_became_active":"2024-09-10T19:24:44.892294+0000","last_became_peered":"2024-09-10T19:24:44.892294+0000","last_unstale":"2024-09-10T19:33:24.309655+0000","last_undegraded":"2024-09-10T19:33:24.309655+0000","last_fullsized":"2024-09-10T19:33:24.309655+0000","mapping_epoch":314,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":315,"parent":"0.0","parent_split_bits":0,"last_scrub":"414'21","last_scrub_stamp":"2024-09-10T19:33:03.305818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:33:03.305818+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:34:14.068212+0000","scrub_duration":0.001846271,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":34,"num_read_kb":28,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,4],"acting":[0,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":218,"reported_epoch":460,"state":"active+clean","last_fresh":"2024-09-10T19:33:24.312362+0000","last_change":"2024-09-10T19:32:43.317430+0000","last_active":"2024-09-10T19:33:24.312362+0000","last_peered":"2024-09-10T19:33:24.312362+0000","last_clean":"2024-09-10T19:33:24.312362+0000","last_became_active":"2024-09-10T19:24:01.980348+0000","last_became_peered":"2024-09-10T19:24:01.980348+0000","last_unstale":"2024-09-10T19:33:24.312362+0000","last_undegraded":"2024-09-10T19:33:24.312362+0000","last_fullsized":"2024-09-10T19:33:24.312362+0000","mapping_epoch":292,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":293,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:32:43.317372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:11:44.254513+0000","last_clean_scrub_stamp":"2024-09-10T19:32:43.317372+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:34:07.570635+0000","scrub_duration":0.001876363,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":58,"reported_epoch":460,"state":"active+clean","last_fresh":"2024-09-10T19:33:24.681855+0000","last_change":"2024-09-10T19:33:24.681822+0000","last_active":"2024-09-10T19:33:24.681855+0000","last_peered":"2024-09-10T19:33:24.681855+0000","last_clean":"2024-09-10T19:33:24.681855+0000","last_became_active":"2024-09-10T19:31:33.345296+0000","last_became_peered":"2024-09-10T19:31:33.345296+0000","last_unstale":"2024-09-10T19:33:24.681855+0000","last_undegraded":"2024-09-10T19:33:24.681855+0000","last_fullsized":"2024-09-10T19:33:24.681855+0000","mapping_epoch":431,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":432,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:33:24.681769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:33:24.681769+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:34:50.503060+0000","scrub_duration":0.0016144099999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":469,"reported_epoch":460,"state":"active+clean","last_fresh":"2024-09-10T19:33:24.309312+0000","last_change":"2024-09-10T19:33:20.250155+0000","last_active":"2024-09-10T19:33:24.309312+0000","last_peered":"2024-09-10T19:33:24.309312+0000","last_clean":"2024-09-10T19:33:24.309312+0000","last_became_active":"2024-09-10T19:31:33.339964+0000","last_became_peered":"2024-09-10T19:31:33.339964+0000","last_unstale":"2024-09-10T19:33:24.309312+0000","last_undegraded":"2024-09-10T19:33:24.309312+0000","last_fullsized":"2024-09-10T19:33:24.309312+0000","mapping_epoch":431,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":432,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:33:20.250078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:30:33.974134+0000","last_clean_scrub_stamp":"2024-09-10T19:33:20.250078+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:34:43.565662+0000","scrub_duration":0.0022003880000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.8","version":"414'46","reported_seq":618,"reported_epoch":459,"state":"active+clean","last_fresh":"2024-09-10T19:33:19.539397+0000","last_change":"2024-09-10T19:33:19.539310+0000","last_active":"2024-09-10T19:33:19.539397+0000","last_peered":"2024-09-10T19:33:19.539397+0000","last_clean":"2024-09-10T19:33:19.539397+0000","last_became_active":"2024-09-10T19:27:53.004858+0000","last_became_peered":"2024-09-10T19:27:53.004858+0000","last_unstale":"2024-09-10T19:33:19.539397+0000","last_undegraded":"2024-09-10T19:33:19.539397+0000","last_fullsized":"2024-09-10T19:33:19.539397+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"414'46","last_scrub_stamp":"2024-09-10T19:33:19.539238+0000","last_deep_scrub":"220'22","last_deep_scrub_stamp":"2024-09-10T19:19:24.597866+0000","last_clean_scrub_stamp":"2024-09-10T19:33:19.539238+0000","objects_scrubbed":7,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:34:19.828559+0000","scrub_duration":0.0021546320000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":62,"num_read_kb":53,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":58,"reported_epoch":459,"state":"active+clean","last_fresh":"2024-09-10T19:33:09.660469+0000","last_change":"2024-09-10T19:33:09.660425+0000","last_active":"2024-09-10T19:33:09.660469+0000","last_peered":"2024-09-10T19:33:09.660469+0000","last_clean":"2024-09-10T19:33:09.660469+0000","last_became_active":"2024-09-10T19:31:33.347734+0000","last_became_peered":"2024-09-10T19:31:33.347734+0000","last_unstale":"2024-09-10T19:33:09.660469+0000","last_undegraded":"2024-09-10T19:33:09.660469+0000","last_fullsized":"2024-09-10T19:33:09.660469+0000","mapping_epoch":431,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":432,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:33:09.660372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:33:09.660372+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:34:36.542479+0000","scrub_duration":0.0015034549999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.9","version":"376'11","reported_seq":822,"reported_epoch":459,"state":"active+clean","last_fresh":"2024-09-10T19:33:21.003861+0000","last_change":"2024-09-10T19:31:55.163970+0000","last_active":"2024-09-10T19:33:21.003861+0000","last_peered":"2024-09-10T19:33:21.003861+0000","last_clean":"2024-09-10T19:33:21.003861+0000","last_became_active":"2024-09-10T19:27:53.009810+0000","last_became_peered":"2024-09-10T19:27:53.009810+0000","last_unstale":"2024-09-10T19:33:21.003861+0000","last_undegraded":"2024-09-10T19:33:21.003861+0000","last_fullsized":"2024-09-10T19:33:21.003861+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"376'11","last_scrub_stamp":"2024-09-10T19:31:55.163902+0000","last_deep_scrub":"80'9","last_deep_scrub_stamp":"2024-09-10T19:25:29.833271+0000","last_clean_scrub_stamp":"2024-09-10T19:31:55.163902+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0017787429999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":521,"reported_epoch":459,"state":"active+clean","last_fresh":"2024-09-10T19:33:02.227291+0000","last_change":"2024-09-10T19:32:23.388747+0000","last_active":"2024-09-10T19:33:02.227291+0000","last_peered":"2024-09-10T19:33:02.227291+0000","last_clean":"2024-09-10T19:33:02.227291+0000","last_became_active":"2024-09-10T19:27:53.015957+0000","last_became_peered":"2024-09-10T19:27:53.015957+0000","last_unstale":"2024-09-10T19:33:02.227291+0000","last_undegraded":"2024-09-10T19:33:02.227291+0000","last_fullsized":"2024-09-10T19:33:02.227291+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:32:23.388634+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:29:27.408374+0000","last_clean_scrub_stamp":"2024-09-10T19:32:23.388634+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:33:25.277720+0000","scrub_duration":0.0023981499999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"414'24","reported_seq":592,"reported_epoch":459,"state":"active+clean","last_fresh":"2024-09-10T19:33:02.907360+0000","last_change":"2024-09-10T19:32:42.833945+0000","last_active":"2024-09-10T19:33:02.907360+0000","last_peered":"2024-09-10T19:33:02.907360+0000","last_clean":"2024-09-10T19:33:02.907360+0000","last_became_active":"2024-09-10T19:24:01.974878+0000","last_became_peered":"2024-09-10T19:24:01.974878+0000","last_unstale":"2024-09-10T19:33:02.907360+0000","last_undegraded":"2024-09-10T19:33:02.907360+0000","last_fullsized":"2024-09-10T19:33:02.907360+0000","mapping_epoch":292,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":293,"parent":"0.0","parent_split_bits":0,"last_scrub":"414'24","last_scrub_stamp":"2024-09-10T19:32:42.833876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:32:42.833876+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:33:45.734397+0000","scrub_duration":0.0035356340000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.f","version":"59'2","reported_seq":522,"reported_epoch":459,"state":"active+clean","last_fresh":"2024-09-10T19:33:02.908583+0000","last_change":"2024-09-10T19:31:55.754140+0000","last_active":"2024-09-10T19:33:02.908583+0000","last_peered":"2024-09-10T19:33:02.908583+0000","last_clean":"2024-09-10T19:33:02.908583+0000","last_became_active":"2024-09-10T19:24:23.893796+0000","last_became_peered":"2024-09-10T19:24:23.893796+0000","last_unstale":"2024-09-10T19:33:02.908583+0000","last_undegraded":"2024-09-10T19:33:02.908583+0000","last_fullsized":"2024-09-10T19:33:02.908583+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'2","last_scrub_stamp":"2024-09-10T19:31:55.754073+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:31:55.754073+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.00167223,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":14,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":48,"reported_epoch":459,"state":"active+clean","last_fresh":"2024-09-10T19:33:02.225827+0000","last_change":"2024-09-10T19:32:33.664513+0000","last_active":"2024-09-10T19:33:02.225827+0000","last_peered":"2024-09-10T19:33:02.225827+0000","last_clean":"2024-09-10T19:33:02.225827+0000","last_became_active":"2024-09-10T19:31:33.344222+0000","last_became_peered":"2024-09-10T19:31:33.344222+0000","last_unstale":"2024-09-10T19:33:02.225827+0000","last_undegraded":"2024-09-10T19:33:02.225827+0000","last_fullsized":"2024-09-10T19:33:02.225827+0000","mapping_epoch":431,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":432,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:32:33.664429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:11:46.587846+0000","last_clean_scrub_stamp":"2024-09-10T19:32:33.664429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:33:45.166431+0000","scrub_duration":0.001339238,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":521,"reported_epoch":460,"state":"active+clean","last_fresh":"2024-09-10T19:33:24.334456+0000","last_change":"2024-09-10T19:33:24.334412+0000","last_active":"2024-09-10T19:33:24.334456+0000","last_peered":"2024-09-10T19:33:24.334456+0000","last_clean":"2024-09-10T19:33:24.334456+0000","last_became_active":"2024-09-10T19:24:01.971131+0000","last_became_peered":"2024-09-10T19:24:01.971131+0000","last_unstale":"2024-09-10T19:33:24.334456+0000","last_undegraded":"2024-09-10T19:33:24.334456+0000","last_fullsized":"2024-09-10T19:33:24.334456+0000","mapping_epoch":292,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":293,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:33:24.334329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:33:24.334329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:34:31.938960+0000","scrub_duration":0.00155292,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.16","version":"414'21","reported_seq":508,"reported_epoch":460,"state":"active+clean","last_fresh":"2024-09-10T19:33:24.310135+0000","last_change":"2024-09-10T19:32:51.292762+0000","last_active":"2024-09-10T19:33:24.310135+0000","last_peered":"2024-09-10T19:33:24.310135+0000","last_clean":"2024-09-10T19:33:24.310135+0000","last_became_active":"2024-09-10T19:24:23.901610+0000","last_became_peered":"2024-09-10T19:24:23.901610+0000","last_unstale":"2024-09-10T19:33:24.310135+0000","last_undegraded":"2024-09-10T19:33:24.310135+0000","last_fullsized":"2024-09-10T19:33:24.310135+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"414'21","last_scrub_stamp":"2024-09-10T19:32:51.292702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:32:51.292702+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:33:56.128089+0000","scrub_duration":0.0019199130000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":34,"num_read_kb":28,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":58,"reported_epoch":459,"state":"active+clean","last_fresh":"2024-09-10T19:33:03.574310+0000","last_change":"2024-09-10T19:33:03.574265+0000","last_active":"2024-09-10T19:33:03.574310+0000","last_peered":"2024-09-10T19:33:03.574310+0000","last_clean":"2024-09-10T19:33:03.574310+0000","last_became_active":"2024-09-10T19:31:33.343673+0000","last_became_peered":"2024-09-10T19:31:33.343673+0000","last_unstale":"2024-09-10T19:33:03.574310+0000","last_undegraded":"2024-09-10T19:33:03.574310+0000","last_fullsized":"2024-09-10T19:33:03.574310+0000","mapping_epoch":431,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":432,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:33:03.574206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:11:41.458523+0000","last_clean_scrub_stamp":"2024-09-10T19:33:03.574206+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:34:06.554523+0000","scrub_duration":0.001247939,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":520,"reported_epoch":459,"state":"active+clean","last_fresh":"2024-09-10T19:33:07.199569+0000","last_change":"2024-09-10T19:33:07.199530+0000","last_active":"2024-09-10T19:33:07.199569+0000","last_peered":"2024-09-10T19:33:07.199569+0000","last_clean":"2024-09-10T19:33:07.199569+0000","last_became_active":"2024-09-10T19:24:01.970325+0000","last_became_peered":"2024-09-10T19:24:01.970325+0000","last_unstale":"2024-09-10T19:33:07.199569+0000","last_undegraded":"2024-09-10T19:33:07.199569+0000","last_fullsized":"2024-09-10T19:33:07.199569+0000","mapping_epoch":292,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":293,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:33:07.199475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:24:09.367654+0000","last_clean_scrub_stamp":"2024-09-10T19:33:07.199475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:34:31.714857+0000","scrub_duration":0.0015561520000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"66'1","reported_seq":52,"reported_epoch":459,"state":"active+clean","last_fresh":"2024-09-10T19:33:02.226952+0000","last_change":"2024-09-10T19:32:39.551636+0000","last_active":"2024-09-10T19:33:02.226952+0000","last_peered":"2024-09-10T19:33:02.226952+0000","last_clean":"2024-09-10T19:33:02.226952+0000","last_became_active":"2024-09-10T19:31:33.347883+0000","last_became_peered":"2024-09-10T19:31:33.347883+0000","last_unstale":"2024-09-10T19:33:02.226952+0000","last_undegraded":"2024-09-10T19:33:02.226952+0000","last_fullsized":"2024-09-10T19:33:02.226952+0000","mapping_epoch":431,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":432,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'1","last_scrub_stamp":"2024-09-10T19:32:39.551576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:32:39.551576+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:33:45.602034+0000","scrub_duration":0.0021501459999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":13,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":145,"reported_epoch":459,"state":"active+clean","last_fresh":"2024-09-10T19:33:20.376880+0000","last_change":"2024-09-10T19:33:20.376833+0000","last_active":"2024-09-10T19:33:20.376880+0000","last_peered":"2024-09-10T19:33:20.376880+0000","last_clean":"2024-09-10T19:33:20.376880+0000","last_became_active":"2024-09-10T19:27:53.007487+0000","last_became_peered":"2024-09-10T19:27:53.007487+0000","last_unstale":"2024-09-10T19:33:20.376880+0000","last_undegraded":"2024-09-10T19:33:20.376880+0000","last_fullsized":"2024-09-10T19:33:20.376880+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:33:20.376739+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:33:20.376739+0000","last_clean_scrub_stamp":"2024-09-10T19:33:20.376739+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:34:23.186510+0000","scrub_duration":0.001408587,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"414'18","reported_seq":576,"reported_epoch":459,"state":"active+clean","last_fresh":"2024-09-10T19:33:13.285118+0000","last_change":"2024-09-10T19:33:13.285069+0000","last_active":"2024-09-10T19:33:13.285118+0000","last_peered":"2024-09-10T19:33:13.285118+0000","last_clean":"2024-09-10T19:33:13.285118+0000","last_became_active":"2024-09-10T19:24:01.971492+0000","last_became_peered":"2024-09-10T19:24:01.971492+0000","last_unstale":"2024-09-10T19:33:13.285118+0000","last_undegraded":"2024-09-10T19:33:13.285118+0000","last_fullsized":"2024-09-10T19:33:13.285118+0000","mapping_epoch":292,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":293,"parent":"0.0","parent_split_bits":0,"last_scrub":"414'18","last_scrub_stamp":"2024-09-10T19:33:13.285004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:33:13.285004+0000","objects_scrubbed":3,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:34:24.424743+0000","scrub_duration":0.0021640850000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":27,"num_read_kb":24,"num_write":18,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":523,"reported_epoch":459,"state":"active+clean","last_fresh":"2024-09-10T19:33:21.508968+0000","last_change":"2024-09-10T19:33:21.508895+0000","last_active":"2024-09-10T19:33:21.508968+0000","last_peered":"2024-09-10T19:33:21.508968+0000","last_clean":"2024-09-10T19:33:21.508968+0000","last_became_active":"2024-09-10T19:24:01.978215+0000","last_became_peered":"2024-09-10T19:24:01.978215+0000","last_unstale":"2024-09-10T19:33:21.508968+0000","last_undegraded":"2024-09-10T19:33:21.508968+0000","last_fullsized":"2024-09-10T19:33:21.508968+0000","mapping_epoch":292,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":293,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:33:21.508825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:33:21.508825+0000","last_clean_scrub_stamp":"2024-09-10T19:33:21.508825+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:34:47.274336+0000","scrub_duration":0.0019394900000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":524,"reported_epoch":459,"state":"active+clean","last_fresh":"2024-09-10T19:33:02.908203+0000","last_change":"2024-09-10T19:32:26.991373+0000","last_active":"2024-09-10T19:33:02.908203+0000","last_peered":"2024-09-10T19:33:02.908203+0000","last_clean":"2024-09-10T19:33:02.908203+0000","last_became_active":"2024-09-10T19:27:53.010120+0000","last_became_peered":"2024-09-10T19:27:53.010120+0000","last_unstale":"2024-09-10T19:33:02.908203+0000","last_undegraded":"2024-09-10T19:33:02.908203+0000","last_fullsized":"2024-09-10T19:33:02.908203+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:32:26.991302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:32:26.991302+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:33:54.243229+0000","scrub_duration":0.001732419,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"414'35","reported_seq":611,"reported_epoch":459,"state":"active+clean","last_fresh":"2024-09-10T19:33:02.226946+0000","last_change":"2024-09-10T19:32:23.292438+0000","last_active":"2024-09-10T19:33:02.226946+0000","last_peered":"2024-09-10T19:33:02.226946+0000","last_clean":"2024-09-10T19:33:02.226946+0000","last_became_active":"2024-09-10T19:24:01.971173+0000","last_became_peered":"2024-09-10T19:24:01.971173+0000","last_unstale":"2024-09-10T19:33:02.226946+0000","last_undegraded":"2024-09-10T19:33:02.226946+0000","last_fullsized":"2024-09-10T19:33:02.226946+0000","mapping_epoch":292,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":293,"parent":"0.0","parent_split_bits":0,"last_scrub":"414'35","last_scrub_stamp":"2024-09-10T19:32:23.292377+0000","last_deep_scrub":"80'15","last_deep_scrub_stamp":"2024-09-10T19:11:41.719310+0000","last_clean_scrub_stamp":"2024-09-10T19:32:23.292377+0000","objects_scrubbed":5,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:33:39.993723+0000","scrub_duration":0.0040992260000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":45,"num_read_kb":40,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":500,"reported_epoch":459,"state":"active+clean","last_fresh":"2024-09-10T19:33:02.227062+0000","last_change":"2024-09-10T19:32:10.391679+0000","last_active":"2024-09-10T19:33:02.227062+0000","last_peered":"2024-09-10T19:33:02.227062+0000","last_clean":"2024-09-10T19:33:02.227062+0000","last_became_active":"2024-09-10T19:10:07.685870+0000","last_became_peered":"2024-09-10T19:10:07.685870+0000","last_unstale":"2024-09-10T19:33:02.227062+0000","last_undegraded":"2024-09-10T19:33:02.227062+0000","last_fullsized":"2024-09-10T19:33:02.227062+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:32:10.391525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:32:10.391525+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0022317169999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"372'16","reported_seq":848,"reported_epoch":459,"state":"active+clean","last_fresh":"2024-09-10T19:33:21.003856+0000","last_change":"2024-09-10T19:32:41.509286+0000","last_active":"2024-09-10T19:33:21.003856+0000","last_peered":"2024-09-10T19:33:21.003856+0000","last_clean":"2024-09-10T19:33:21.003856+0000","last_became_active":"2024-09-10T19:27:46.703433+0000","last_became_peered":"2024-09-10T19:27:46.703433+0000","last_unstale":"2024-09-10T19:33:21.003856+0000","last_undegraded":"2024-09-10T19:33:21.003856+0000","last_fullsized":"2024-09-10T19:33:21.003856+0000","mapping_epoch":371,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":372,"parent":"0.0","parent_split_bits":0,"last_scrub":"372'16","last_scrub_stamp":"2024-09-10T19:32:41.509222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:32:41.509222+0000","objects_scrubbed":1,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:33:51.664560+0000","scrub_duration":0.001616609,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,0],"acting":[5,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":455,"reported_epoch":460,"state":"active+clean","last_fresh":"2024-09-10T19:33:24.310155+0000","last_change":"2024-09-10T19:32:40.230900+0000","last_active":"2024-09-10T19:33:24.310155+0000","last_peered":"2024-09-10T19:33:24.310155+0000","last_clean":"2024-09-10T19:33:24.310155+0000","last_became_active":"2024-09-10T19:24:23.900513+0000","last_became_peered":"2024-09-10T19:24:23.900513+0000","last_unstale":"2024-09-10T19:33:24.310155+0000","last_undegraded":"2024-09-10T19:33:24.310155+0000","last_fullsized":"2024-09-10T19:33:24.310155+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:32:40.230850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:32:40.230850+0000","last_clean_scrub_stamp":"2024-09-10T19:32:40.230850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:33:56.253229+0000","scrub_duration":0.0017157940000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"414'21","reported_seq":572,"reported_epoch":459,"state":"active+clean","last_fresh":"2024-09-10T19:33:02.227743+0000","last_change":"2024-09-10T19:32:30.493699+0000","last_active":"2024-09-10T19:33:02.227743+0000","last_peered":"2024-09-10T19:33:02.227743+0000","last_clean":"2024-09-10T19:33:02.227743+0000","last_became_active":"2024-09-10T19:16:24.859740+0000","last_became_peered":"2024-09-10T19:16:24.859740+0000","last_unstale":"2024-09-10T19:33:02.227743+0000","last_undegraded":"2024-09-10T19:33:02.227743+0000","last_fullsized":"2024-09-10T19:33:02.227743+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"414'21","last_scrub_stamp":"2024-09-10T19:32:30.493636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:32:30.493636+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:33:49.084909+0000","scrub_duration":0.0021584120000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":48,"reported_epoch":459,"state":"active+clean","last_fresh":"2024-09-10T19:33:02.225951+0000","last_change":"2024-09-10T19:32:55.441239+0000","last_active":"2024-09-10T19:33:02.225951+0000","last_peered":"2024-09-10T19:33:02.225951+0000","last_clean":"2024-09-10T19:33:02.225951+0000","last_became_active":"2024-09-10T19:31:33.341283+0000","last_became_peered":"2024-09-10T19:31:33.341283+0000","last_unstale":"2024-09-10T19:33:02.225951+0000","last_undegraded":"2024-09-10T19:33:02.225951+0000","last_fullsized":"2024-09-10T19:33:02.225951+0000","mapping_epoch":431,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":432,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:32:55.441184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:32:55.441184+0000","last_clean_scrub_stamp":"2024-09-10T19:32:55.441184+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:34:11.325146+0000","scrub_duration":0.001586613,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.14","version":"432'12","reported_seq":826,"reported_epoch":459,"state":"active+clean","last_fresh":"2024-09-10T19:33:21.003680+0000","last_change":"2024-09-10T19:32:30.209134+0000","last_active":"2024-09-10T19:33:21.003680+0000","last_peered":"2024-09-10T19:33:21.003680+0000","last_clean":"2024-09-10T19:33:21.003680+0000","last_became_active":"2024-09-10T19:31:33.341212+0000","last_became_peered":"2024-09-10T19:31:33.341212+0000","last_unstale":"2024-09-10T19:33:21.003680+0000","last_undegraded":"2024-09-10T19:33:21.003680+0000","last_fullsized":"2024-09-10T19:33:21.003680+0000","mapping_epoch":431,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":432,"parent":"0.0","parent_split_bits":0,"last_scrub":"432'12","last_scrub_stamp":"2024-09-10T19:32:30.209076+0000","last_deep_scrub":"297'11","last_deep_scrub_stamp":"2024-09-10T19:24:13.318130+0000","last_clean_scrub_stamp":"2024-09-10T19:32:30.209076+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:33:58.844449+0000","scrub_duration":0.0033330090000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":523,"reported_epoch":459,"state":"active+clean","last_fresh":"2024-09-10T19:33:02.908406+0000","last_change":"2024-09-10T19:32:29.848499+0000","last_active":"2024-09-10T19:33:02.908406+0000","last_peered":"2024-09-10T19:33:02.908406+0000","last_clean":"2024-09-10T19:33:02.908406+0000","last_became_active":"2024-09-10T19:31:33.337440+0000","last_became_peered":"2024-09-10T19:31:33.337440+0000","last_unstale":"2024-09-10T19:33:02.908406+0000","last_undegraded":"2024-09-10T19:33:02.908406+0000","last_fullsized":"2024-09-10T19:33:02.908406+0000","mapping_epoch":431,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":432,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:32:29.848435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:32:29.848435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:33:51.895728+0000","scrub_duration":0.001811006,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.12","version":"414'25","reported_seq":321,"reported_epoch":460,"state":"active+clean","last_fresh":"2024-09-10T19:33:24.312008+0000","last_change":"2024-09-10T19:33:19.255892+0000","last_active":"2024-09-10T19:33:24.312008+0000","last_peered":"2024-09-10T19:33:24.312008+0000","last_clean":"2024-09-10T19:33:24.312008+0000","last_became_active":"2024-09-10T19:31:33.342056+0000","last_became_peered":"2024-09-10T19:31:33.342056+0000","last_unstale":"2024-09-10T19:33:24.312008+0000","last_undegraded":"2024-09-10T19:33:24.312008+0000","last_fullsized":"2024-09-10T19:33:24.312008+0000","mapping_epoch":431,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":432,"parent":"0.0","parent_split_bits":0,"last_scrub":"414'25","last_scrub_stamp":"2024-09-10T19:33:19.255826+0000","last_deep_scrub":"79'13","last_deep_scrub_stamp":"2024-09-10T19:11:22.072400+0000","last_clean_scrub_stamp":"2024-09-10T19:33:19.255826+0000","objects_scrubbed":4,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:34:20.132738+0000","scrub_duration":0.001917565,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":34,"num_read_kb":28,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"59'1","reported_seq":139,"reported_epoch":459,"state":"active+clean","last_fresh":"2024-09-10T19:33:02.857923+0000","last_change":"2024-09-10T19:32:01.709050+0000","last_active":"2024-09-10T19:33:02.857923+0000","last_peered":"2024-09-10T19:33:02.857923+0000","last_clean":"2024-09-10T19:33:02.857923+0000","last_became_active":"2024-09-10T19:27:53.018470+0000","last_became_peered":"2024-09-10T19:27:53.018470+0000","last_unstale":"2024-09-10T19:33:02.857923+0000","last_undegraded":"2024-09-10T19:33:02.857923+0000","last_fullsized":"2024-09-10T19:33:02.857923+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'1","last_scrub_stamp":"2024-09-10T19:32:01.708979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:32:01.708979+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0018177480000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":362,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":724,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":534,"reported_epoch":459,"state":"active+clean","last_fresh":"2024-09-10T19:33:02.226502+0000","last_change":"2024-09-10T19:32:31.177171+0000","last_active":"2024-09-10T19:33:02.226502+0000","last_peered":"2024-09-10T19:33:02.226502+0000","last_clean":"2024-09-10T19:33:02.226502+0000","last_became_active":"2024-09-10T19:24:24.923804+0000","last_became_peered":"2024-09-10T19:24:24.923804+0000","last_unstale":"2024-09-10T19:33:02.226502+0000","last_undegraded":"2024-09-10T19:33:02.226502+0000","last_fullsized":"2024-09-10T19:33:02.226502+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:32:31.177118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:32:31.177118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:33:44.531588+0000","scrub_duration":0.001184212,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":474,"reported_epoch":460,"state":"active+clean","last_fresh":"2024-09-10T19:33:24.308905+0000","last_change":"2024-09-10T19:32:50.268020+0000","last_active":"2024-09-10T19:33:24.308905+0000","last_peered":"2024-09-10T19:33:24.308905+0000","last_clean":"2024-09-10T19:33:24.308905+0000","last_became_active":"2024-09-10T19:24:24.916105+0000","last_became_peered":"2024-09-10T19:24:24.916105+0000","last_unstale":"2024-09-10T19:33:24.308905+0000","last_undegraded":"2024-09-10T19:33:24.308905+0000","last_fullsized":"2024-09-10T19:33:24.308905+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:32:50.267968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:25:27.412579+0000","last_clean_scrub_stamp":"2024-09-10T19:32:50.267968+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:34:00.943563+0000","scrub_duration":0.0020030769999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"414'27","reported_seq":638,"reported_epoch":459,"state":"active+clean","last_fresh":"2024-09-10T19:33:02.907809+0000","last_change":"2024-09-10T19:32:19.738922+0000","last_active":"2024-09-10T19:33:02.907809+0000","last_peered":"2024-09-10T19:33:02.907809+0000","last_clean":"2024-09-10T19:33:02.907809+0000","last_became_active":"2024-09-10T19:27:53.016518+0000","last_became_peered":"2024-09-10T19:27:53.016518+0000","last_unstale":"2024-09-10T19:33:02.907809+0000","last_undegraded":"2024-09-10T19:33:02.907809+0000","last_fullsized":"2024-09-10T19:33:02.907809+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"414'27","last_scrub_stamp":"2024-09-10T19:32:19.738852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:32:19.738852+0000","objects_scrubbed":5,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:33:33.863400+0000","scrub_duration":0.0018747530000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":37,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":242,"reported_epoch":460,"state":"active+clean","last_fresh":"2024-09-10T19:33:24.312331+0000","last_change":"2024-09-10T19:32:38.297715+0000","last_active":"2024-09-10T19:33:24.312331+0000","last_peered":"2024-09-10T19:33:24.312331+0000","last_clean":"2024-09-10T19:33:24.312331+0000","last_became_active":"2024-09-10T19:24:24.909540+0000","last_became_peered":"2024-09-10T19:24:24.909540+0000","last_unstale":"2024-09-10T19:33:24.312331+0000","last_undegraded":"2024-09-10T19:33:24.312331+0000","last_fullsized":"2024-09-10T19:33:24.312331+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:32:38.297639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:32:38.297639+0000","last_clean_scrub_stamp":"2024-09-10T19:32:38.297639+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:33:56.775419+0000","scrub_duration":0.0020595330000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":218,"reported_epoch":460,"state":"active+clean","last_fresh":"2024-09-10T19:33:24.312084+0000","last_change":"2024-09-10T19:32:31.379520+0000","last_active":"2024-09-10T19:33:24.312084+0000","last_peered":"2024-09-10T19:33:24.312084+0000","last_clean":"2024-09-10T19:33:24.312084+0000","last_became_active":"2024-09-10T19:24:01.975520+0000","last_became_peered":"2024-09-10T19:24:01.975520+0000","last_unstale":"2024-09-10T19:33:24.312084+0000","last_undegraded":"2024-09-10T19:33:24.312084+0000","last_fullsized":"2024-09-10T19:33:24.312084+0000","mapping_epoch":292,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":293,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:32:31.379466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:32:31.379466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:33:39.186340+0000","scrub_duration":0.001424673,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":501,"reported_epoch":459,"state":"active+clean","last_fresh":"2024-09-10T19:33:06.160609+0000","last_change":"2024-09-10T19:33:06.160552+0000","last_active":"2024-09-10T19:33:06.160609+0000","last_peered":"2024-09-10T19:33:06.160609+0000","last_clean":"2024-09-10T19:33:06.160609+0000","last_became_active":"2024-09-10T19:24:23.907136+0000","last_became_peered":"2024-09-10T19:24:23.907136+0000","last_unstale":"2024-09-10T19:33:06.160609+0000","last_undegraded":"2024-09-10T19:33:06.160609+0000","last_fullsized":"2024-09-10T19:33:06.160609+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:33:06.160473+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:33:06.160473+0000","last_clean_scrub_stamp":"2024-09-10T19:33:06.160473+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:34:13.264507+0000","scrub_duration":0.0015304590000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"414'14","reported_seq":589,"reported_epoch":459,"state":"active+clean","last_fresh":"2024-09-10T19:33:17.354282+0000","last_change":"2024-09-10T19:33:17.354210+0000","last_active":"2024-09-10T19:33:17.354282+0000","last_peered":"2024-09-10T19:33:17.354282+0000","last_clean":"2024-09-10T19:33:17.354282+0000","last_became_active":"2024-09-10T19:27:53.024782+0000","last_became_peered":"2024-09-10T19:27:53.024782+0000","last_unstale":"2024-09-10T19:33:17.354282+0000","last_undegraded":"2024-09-10T19:33:17.354282+0000","last_fullsized":"2024-09-10T19:33:17.354282+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"414'14","last_scrub_stamp":"2024-09-10T19:33:17.354110+0000","last_deep_scrub":"222'6","last_deep_scrub_stamp":"2024-09-10T19:19:30.328841+0000","last_clean_scrub_stamp":"2024-09-10T19:33:17.354110+0000","objects_scrubbed":2,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:34:27.169214+0000","scrub_duration":0.0015824319999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":18,"num_read_kb":16,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":560,"reported_epoch":459,"state":"active+clean","last_fresh":"2024-09-10T19:33:02.227772+0000","last_change":"2024-09-10T19:32:50.571952+0000","last_active":"2024-09-10T19:33:02.227772+0000","last_peered":"2024-09-10T19:33:02.227772+0000","last_clean":"2024-09-10T19:33:02.227772+0000","last_became_active":"2024-09-10T19:31:33.338139+0000","last_became_peered":"2024-09-10T19:31:33.338139+0000","last_unstale":"2024-09-10T19:33:02.227772+0000","last_undegraded":"2024-09-10T19:33:02.227772+0000","last_fullsized":"2024-09-10T19:33:02.227772+0000","mapping_epoch":431,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":432,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:32:50.571844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:32:50.571844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:34:15.273985+0000","scrub_duration":0.00161389,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":145,"reported_epoch":459,"state":"active+clean","last_fresh":"2024-09-10T19:33:13.533568+0000","last_change":"2024-09-10T19:33:13.533511+0000","last_active":"2024-09-10T19:33:13.533568+0000","last_peered":"2024-09-10T19:33:13.533568+0000","last_clean":"2024-09-10T19:33:13.533568+0000","last_became_active":"2024-09-10T19:27:53.003591+0000","last_became_peered":"2024-09-10T19:27:53.003591+0000","last_unstale":"2024-09-10T19:33:13.533568+0000","last_undegraded":"2024-09-10T19:33:13.533568+0000","last_fullsized":"2024-09-10T19:33:13.533568+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:33:13.533438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:11:31.837944+0000","last_clean_scrub_stamp":"2024-09-10T19:33:13.533438+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:34:36.324746+0000","scrub_duration":0.001641984,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":534,"reported_epoch":459,"state":"active+clean","last_fresh":"2024-09-10T19:33:02.907196+0000","last_change":"2024-09-10T19:32:34.017145+0000","last_active":"2024-09-10T19:33:02.907196+0000","last_peered":"2024-09-10T19:33:02.907196+0000","last_clean":"2024-09-10T19:33:02.907196+0000","last_became_active":"2024-09-10T19:31:33.342711+0000","last_became_peered":"2024-09-10T19:31:33.342711+0000","last_unstale":"2024-09-10T19:33:02.907196+0000","last_undegraded":"2024-09-10T19:33:02.907196+0000","last_fullsized":"2024-09-10T19:33:02.907196+0000","mapping_epoch":431,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":432,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:32:34.017080+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:32:34.017080+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:33:44.281178+0000","scrub_duration":0.0015294970000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.11","version":"414'27","reported_seq":346,"reported_epoch":459,"state":"active+clean","last_fresh":"2024-09-10T19:33:02.226777+0000","last_change":"2024-09-10T19:32:49.022254+0000","last_active":"2024-09-10T19:33:02.226777+0000","last_peered":"2024-09-10T19:33:02.226777+0000","last_clean":"2024-09-10T19:33:02.226777+0000","last_became_active":"2024-09-10T19:27:53.018213+0000","last_became_peered":"2024-09-10T19:27:53.018213+0000","last_unstale":"2024-09-10T19:33:02.226777+0000","last_undegraded":"2024-09-10T19:33:02.226777+0000","last_fullsized":"2024-09-10T19:33:02.226777+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"414'27","last_scrub_stamp":"2024-09-10T19:32:49.022134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:32:49.022134+0000","objects_scrubbed":5,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:34:14.333132+0000","scrub_duration":0.0026081020000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":37,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":145,"reported_epoch":459,"state":"active+clean","last_fresh":"2024-09-10T19:33:02.858296+0000","last_change":"2024-09-10T19:32:35.608043+0000","last_active":"2024-09-10T19:33:02.858296+0000","last_peered":"2024-09-10T19:33:02.858296+0000","last_clean":"2024-09-10T19:33:02.858296+0000","last_became_active":"2024-09-10T19:31:33.338443+0000","last_became_peered":"2024-09-10T19:31:33.338443+0000","last_unstale":"2024-09-10T19:33:02.858296+0000","last_undegraded":"2024-09-10T19:33:02.858296+0000","last_fullsized":"2024-09-10T19:33:02.858296+0000","mapping_epoch":431,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":432,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:32:35.607968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:13:05.876655+0000","last_clean_scrub_stamp":"2024-09-10T19:32:35.607968+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:33:46.380646+0000","scrub_duration":0.0015078699999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":526,"reported_epoch":459,"state":"active+clean","last_fresh":"2024-09-10T19:33:08.807370+0000","last_change":"2024-09-10T19:33:08.807323+0000","last_active":"2024-09-10T19:33:08.807370+0000","last_peered":"2024-09-10T19:33:08.807370+0000","last_clean":"2024-09-10T19:33:08.807370+0000","last_became_active":"2024-09-10T19:27:53.008444+0000","last_became_peered":"2024-09-10T19:27:53.008444+0000","last_unstale":"2024-09-10T19:33:08.807370+0000","last_undegraded":"2024-09-10T19:33:08.807370+0000","last_fullsized":"2024-09-10T19:33:08.807370+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:33:08.807262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:11:57.112092+0000","last_clean_scrub_stamp":"2024-09-10T19:33:08.807262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:34:30.759357+0000","scrub_duration":0.001404708,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":521,"reported_epoch":459,"state":"active+clean","last_fresh":"2024-09-10T19:33:02.226352+0000","last_change":"2024-09-10T19:33:01.174031+0000","last_active":"2024-09-10T19:33:02.226352+0000","last_peered":"2024-09-10T19:33:02.226352+0000","last_clean":"2024-09-10T19:33:02.226352+0000","last_became_active":"2024-09-10T19:27:53.025650+0000","last_became_peered":"2024-09-10T19:27:53.025650+0000","last_unstale":"2024-09-10T19:33:02.226352+0000","last_undegraded":"2024-09-10T19:33:02.226352+0000","last_fullsized":"2024-09-10T19:33:02.226352+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:33:01.173976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:26:29.959202+0000","last_clean_scrub_stamp":"2024-09-10T19:33:01.173976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:34:27.056442+0000","scrub_duration":0.0015841900000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"80'2","reported_seq":534,"reported_epoch":459,"state":"active+clean","last_fresh":"2024-09-10T19:33:02.908438+0000","last_change":"2024-09-10T19:32:05.196679+0000","last_active":"2024-09-10T19:33:02.908438+0000","last_peered":"2024-09-10T19:33:02.908438+0000","last_clean":"2024-09-10T19:33:02.908438+0000","last_became_active":"2024-09-10T19:31:33.343250+0000","last_became_peered":"2024-09-10T19:31:33.343250+0000","last_unstale":"2024-09-10T19:33:02.908438+0000","last_undegraded":"2024-09-10T19:33:02.908438+0000","last_fullsized":"2024-09-10T19:33:02.908438+0000","mapping_epoch":431,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":432,"parent":"0.0","parent_split_bits":0,"last_scrub":"80'2","last_scrub_stamp":"2024-09-10T19:32:05.196606+0000","last_deep_scrub":"80'2","last_deep_scrub_stamp":"2024-09-10T19:11:30.272043+0000","last_clean_scrub_stamp":"2024-09-10T19:32:05.196606+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0020235969999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":369,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":738,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":517,"reported_epoch":459,"state":"active+clean","last_fresh":"2024-09-10T19:33:02.226949+0000","last_change":"2024-09-10T19:32:27.449151+0000","last_active":"2024-09-10T19:33:02.226949+0000","last_peered":"2024-09-10T19:33:02.226949+0000","last_clean":"2024-09-10T19:33:02.226949+0000","last_became_active":"2024-09-10T19:24:23.906812+0000","last_became_peered":"2024-09-10T19:24:23.906812+0000","last_unstale":"2024-09-10T19:33:02.226949+0000","last_undegraded":"2024-09-10T19:33:02.226949+0000","last_fullsized":"2024-09-10T19:33:02.226949+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:32:27.449085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:32:27.449085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:33:51.226533+0000","scrub_duration":0.001431234,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,0],"acting":[5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1e","version":"414'35","reported_seq":509,"reported_epoch":460,"state":"active+clean","last_fresh":"2024-09-10T19:33:24.310059+0000","last_change":"2024-09-10T19:32:18.102686+0000","last_active":"2024-09-10T19:33:24.310059+0000","last_peered":"2024-09-10T19:33:24.310059+0000","last_clean":"2024-09-10T19:33:24.310059+0000","last_became_active":"2024-09-10T19:24:23.899507+0000","last_became_peered":"2024-09-10T19:24:23.899507+0000","last_unstale":"2024-09-10T19:33:24.310059+0000","last_undegraded":"2024-09-10T19:33:24.310059+0000","last_fullsized":"2024-09-10T19:33:24.310059+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"414'35","last_scrub_stamp":"2024-09-10T19:32:18.102631+0000","last_deep_scrub":"414'35","last_deep_scrub_stamp":"2024-09-10T19:32:18.102631+0000","last_clean_scrub_stamp":"2024-09-10T19:32:18.102631+0000","objects_scrubbed":5,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:33:44.786784+0000","scrub_duration":0.001961431,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":39,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1f","version":"376'10","reported_seq":195,"reported_epoch":459,"state":"active+clean","last_fresh":"2024-09-10T19:33:21.003856+0000","last_change":"2024-09-10T19:31:55.778552+0000","last_active":"2024-09-10T19:33:21.003856+0000","last_peered":"2024-09-10T19:33:21.003856+0000","last_clean":"2024-09-10T19:33:21.003856+0000","last_became_active":"2024-09-10T19:27:53.003676+0000","last_became_peered":"2024-09-10T19:27:53.003676+0000","last_unstale":"2024-09-10T19:33:21.003856+0000","last_undegraded":"2024-09-10T19:33:21.003856+0000","last_fullsized":"2024-09-10T19:33:21.003856+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"376'10","last_scrub_stamp":"2024-09-10T19:31:55.778482+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:31:55.778482+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001552313,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":228,"reported_epoch":460,"state":"active+clean","last_fresh":"2024-09-10T19:33:24.312406+0000","last_change":"2024-09-10T19:33:18.239503+0000","last_active":"2024-09-10T19:33:24.312406+0000","last_peered":"2024-09-10T19:33:24.312406+0000","last_clean":"2024-09-10T19:33:24.312406+0000","last_became_active":"2024-09-10T19:24:01.980391+0000","last_became_peered":"2024-09-10T19:24:01.980391+0000","last_unstale":"2024-09-10T19:33:24.312406+0000","last_undegraded":"2024-09-10T19:33:24.312406+0000","last_fullsized":"2024-09-10T19:33:24.312406+0000","mapping_epoch":292,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":293,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:33:18.239445+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:28:20.514115+0000","last_clean_scrub_stamp":"2024-09-10T19:33:18.239445+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:34:32.561716+0000","scrub_duration":0.0017389860000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":537,"reported_epoch":459,"state":"active+clean","last_fresh":"2024-09-10T19:33:16.328271+0000","last_change":"2024-09-10T19:33:16.328234+0000","last_active":"2024-09-10T19:33:16.328271+0000","last_peered":"2024-09-10T19:33:16.328271+0000","last_clean":"2024-09-10T19:33:16.328271+0000","last_became_active":"2024-09-10T19:27:53.017153+0000","last_became_peered":"2024-09-10T19:27:53.017153+0000","last_unstale":"2024-09-10T19:33:16.328271+0000","last_undegraded":"2024-09-10T19:33:16.328271+0000","last_fullsized":"2024-09-10T19:33:16.328271+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:33:16.328178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:33:16.328178+0000","last_clean_scrub_stamp":"2024-09-10T19:33:16.328178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:34:29.545042+0000","scrub_duration":0.001686089,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"414'32","reported_seq":159,"reported_epoch":459,"state":"active+clean","last_fresh":"2024-09-10T19:33:02.858242+0000","last_change":"2024-09-10T19:32:39.649686+0000","last_active":"2024-09-10T19:33:02.858242+0000","last_peered":"2024-09-10T19:33:02.858242+0000","last_clean":"2024-09-10T19:33:02.858242+0000","last_became_active":"2024-09-10T19:27:53.008233+0000","last_became_peered":"2024-09-10T19:27:53.008233+0000","last_unstale":"2024-09-10T19:33:02.858242+0000","last_undegraded":"2024-09-10T19:33:02.858242+0000","last_fullsized":"2024-09-10T19:33:02.858242+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"414'32","last_scrub_stamp":"2024-09-10T19:32:39.649618+0000","last_deep_scrub":"414'32","last_deep_scrub_stamp":"2024-09-10T19:32:39.649618+0000","last_clean_scrub_stamp":"2024-09-10T19:32:39.649618+0000","objects_scrubbed":5,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:33:54.043975+0000","scrub_duration":0.0042512590000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":36,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":456,"reported_epoch":460,"state":"active+clean","last_fresh":"2024-09-10T19:33:24.310012+0000","last_change":"2024-09-10T19:33:18.273197+0000","last_active":"2024-09-10T19:33:24.310012+0000","last_peered":"2024-09-10T19:33:24.310012+0000","last_clean":"2024-09-10T19:33:24.310012+0000","last_became_active":"2024-09-10T19:31:33.334922+0000","last_became_peered":"2024-09-10T19:31:33.334922+0000","last_unstale":"2024-09-10T19:33:24.310012+0000","last_undegraded":"2024-09-10T19:33:24.310012+0000","last_fullsized":"2024-09-10T19:33:24.310012+0000","mapping_epoch":431,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":432,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:33:18.273126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:28:19.464426+0000","last_clean_scrub_stamp":"2024-09-10T19:33:18.273126+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:34:32.331325+0000","scrub_duration":0.001575543,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":523,"reported_epoch":459,"state":"active+clean","last_fresh":"2024-09-10T19:33:02.907861+0000","last_change":"2024-09-10T19:31:59.739548+0000","last_active":"2024-09-10T19:33:02.907861+0000","last_peered":"2024-09-10T19:33:02.907861+0000","last_clean":"2024-09-10T19:33:02.907861+0000","last_became_active":"2024-09-10T19:27:53.011336+0000","last_became_peered":"2024-09-10T19:27:53.011336+0000","last_unstale":"2024-09-10T19:33:02.907861+0000","last_undegraded":"2024-09-10T19:33:02.907861+0000","last_fullsized":"2024-09-10T19:33:02.907861+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:31:59.739484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:31:59.739484+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0013729689999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":135,"reported_epoch":459,"state":"active+clean","last_fresh":"2024-09-10T19:33:02.857771+0000","last_change":"2024-09-10T19:32:50.572647+0000","last_active":"2024-09-10T19:33:02.857771+0000","last_peered":"2024-09-10T19:33:02.857771+0000","last_clean":"2024-09-10T19:33:02.857771+0000","last_became_active":"2024-09-10T19:27:53.002313+0000","last_became_peered":"2024-09-10T19:27:53.002313+0000","last_unstale":"2024-09-10T19:33:02.857771+0000","last_undegraded":"2024-09-10T19:33:02.857771+0000","last_fullsized":"2024-09-10T19:33:02.857771+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:32:50.572581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:32:50.572581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:34:08.318842+0000","scrub_duration":0.0016561519999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1c","version":"414'46","reported_seq":74,"reported_epoch":459,"state":"active+clean","last_fresh":"2024-09-10T19:33:10.707911+0000","last_change":"2024-09-10T19:33:10.707829+0000","last_active":"2024-09-10T19:33:10.707911+0000","last_peered":"2024-09-10T19:33:10.707911+0000","last_clean":"2024-09-10T19:33:10.707911+0000","last_became_active":"2024-09-10T19:31:33.344217+0000","last_became_peered":"2024-09-10T19:31:33.344217+0000","last_unstale":"2024-09-10T19:33:10.707911+0000","last_undegraded":"2024-09-10T19:33:10.707911+0000","last_fullsized":"2024-09-10T19:33:10.707911+0000","mapping_epoch":431,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":432,"parent":"0.0","parent_split_bits":0,"last_scrub":"414'46","last_scrub_stamp":"2024-09-10T19:33:10.707736+0000","last_deep_scrub":"78'22","last_deep_scrub_stamp":"2024-09-10T19:11:21.118937+0000","last_clean_scrub_stamp":"2024-09-10T19:33:10.707736+0000","objects_scrubbed":7,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:34:31.395614+0000","scrub_duration":0.0018651609999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":61,"num_read_kb":52,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":242,"reported_epoch":460,"state":"active+clean","last_fresh":"2024-09-10T19:33:24.312443+0000","last_change":"2024-09-10T19:33:03.280294+0000","last_active":"2024-09-10T19:33:24.312443+0000","last_peered":"2024-09-10T19:33:24.312443+0000","last_clean":"2024-09-10T19:33:24.312443+0000","last_became_active":"2024-09-10T19:24:24.887205+0000","last_became_peered":"2024-09-10T19:24:24.887205+0000","last_unstale":"2024-09-10T19:33:24.312443+0000","last_undegraded":"2024-09-10T19:33:24.312443+0000","last_fullsized":"2024-09-10T19:33:24.312443+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:33:03.280233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:33:03.280233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:34:22.114440+0000","scrub_duration":0.0016609179999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":517,"reported_epoch":459,"state":"active+clean","last_fresh":"2024-09-10T19:33:02.227183+0000","last_change":"2024-09-10T19:32:43.974114+0000","last_active":"2024-09-10T19:33:02.227183+0000","last_peered":"2024-09-10T19:33:02.227183+0000","last_clean":"2024-09-10T19:33:02.227183+0000","last_became_active":"2024-09-10T19:27:53.026591+0000","last_became_peered":"2024-09-10T19:27:53.026591+0000","last_unstale":"2024-09-10T19:33:02.227183+0000","last_undegraded":"2024-09-10T19:33:02.227183+0000","last_fullsized":"2024-09-10T19:33:02.227183+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:32:43.974059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:32:43.974059+0000","last_clean_scrub_stamp":"2024-09-10T19:32:43.974059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:33:59.177332+0000","scrub_duration":0.0021937829999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":520,"reported_epoch":459,"state":"active+clean","last_fresh":"2024-09-10T19:33:02.226639+0000","last_change":"2024-09-10T19:32:18.334973+0000","last_active":"2024-09-10T19:33:02.226639+0000","last_peered":"2024-09-10T19:33:02.226639+0000","last_clean":"2024-09-10T19:33:02.226639+0000","last_became_active":"2024-09-10T19:27:53.018726+0000","last_became_peered":"2024-09-10T19:27:53.018726+0000","last_unstale":"2024-09-10T19:33:02.226639+0000","last_undegraded":"2024-09-10T19:33:02.226639+0000","last_fullsized":"2024-09-10T19:33:02.226639+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:32:18.334866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:32:18.334866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:33:34.896293+0000","scrub_duration":0.001806394,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1d","version":"414'29","reported_seq":585,"reported_epoch":459,"state":"active+clean","last_fresh":"2024-09-10T19:33:02.227058+0000","last_change":"2024-09-10T19:32:27.254562+0000","last_active":"2024-09-10T19:33:02.227058+0000","last_peered":"2024-09-10T19:33:02.227058+0000","last_clean":"2024-09-10T19:33:02.227058+0000","last_became_active":"2024-09-10T19:24:01.976664+0000","last_became_peered":"2024-09-10T19:24:01.976664+0000","last_unstale":"2024-09-10T19:33:02.227058+0000","last_undegraded":"2024-09-10T19:33:02.227058+0000","last_fullsized":"2024-09-10T19:33:02.227058+0000","mapping_epoch":292,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":293,"parent":"0.0","parent_split_bits":0,"last_scrub":"414'29","last_scrub_stamp":"2024-09-10T19:32:27.254505+0000","last_deep_scrub":"250'23","last_deep_scrub_stamp":"2024-09-10T19:20:55.582173+0000","last_clean_scrub_stamp":"2024-09-10T19:32:27.254505+0000","objects_scrubbed":5,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:33:41.635783+0000","scrub_duration":0.001812255,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":525,"reported_epoch":459,"state":"active+clean","last_fresh":"2024-09-10T19:33:02.907919+0000","last_change":"2024-09-10T19:32:00.743872+0000","last_active":"2024-09-10T19:33:02.907919+0000","last_peered":"2024-09-10T19:33:02.907919+0000","last_clean":"2024-09-10T19:33:02.907919+0000","last_became_active":"2024-09-10T19:31:33.337386+0000","last_became_peered":"2024-09-10T19:31:33.337386+0000","last_unstale":"2024-09-10T19:33:02.907919+0000","last_undegraded":"2024-09-10T19:33:02.907919+0000","last_fullsized":"2024-09-10T19:33:02.907919+0000","mapping_epoch":431,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":432,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:32:00.743803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:30:39.493257+0000","last_clean_scrub_stamp":"2024-09-10T19:32:00.743803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:33:30.335014+0000","scrub_duration":0.001558076,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":751,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":177,"num_object_clones":0,"num_object_copies":531,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":177,"num_whiteouts":0,"num_read":1541,"num_read_kb":1298,"num_write":989,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":496,"num_bytes_recovered":9644,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1090,"ondisk_log_size":1090,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":565,"num_read_kb":560,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":1983,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":11,"ondisk_log_size":11,"up":9,"acting":9,"num_store_stats":7},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1328,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":21,"num_read_kb":21,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":724,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":3984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5,"ondisk_log_size":5,"up":96,"acting":96,"num_store_stats":8},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":788064,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":871,"ondisk_log_size":871,"up":33,"acting":33,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":54,"seq":231928234269,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1177672,"kb_used_data":1028,"kb_used_omap":1,"kb_used_meta":1176510,"kb_avail":92567480,"statfs":{"total":95995035648,"available":94789099520,"internally_reserved":0,"allocated":1052672,"data_stored":755501,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":1204746584},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 19:32:36 2024","interfaces":[{"interface":"back","average":{"1min":0.54600000000000004,"5min":0.875,"15min":0.92500000000000004},"min":{"1min":0.24199999999999999,"5min":0.19600000000000001,"15min":0.19600000000000001},"max":{"1min":0.71299999999999997,"5min":4.5220000000000002,"15min":5.0510000000000002},"last":0.58899999999999997},{"interface":"front","average":{"1min":0.56799999999999995,"5min":0.94399999999999995,"15min":0.82399999999999995},"min":{"1min":0.23499999999999999,"5min":0.23400000000000001,"15min":0.21099999999999999},"max":{"1min":0.747,"5min":3.5089999999999999,"15min":4.7190000000000003},"last":0.625}]},{"osd":1,"last update":"Tue Sep 10 19:33:03 2024","interfaces":[{"interface":"back","average":{"1min":0.53900000000000003,"5min":1.0269999999999999,"15min":1.004},"min":{"1min":0.34000000000000002,"5min":0.224,"15min":0.224},"max":{"1min":0.70599999999999996,"5min":12.773999999999999,"15min":12.773999999999999},"last":0.65600000000000003},{"interface":"front","average":{"1min":0.56200000000000006,"5min":1.04,"15min":0.94799999999999995},"min":{"1min":0.38400000000000001,"5min":0.29599999999999999,"15min":0.26400000000000001},"max":{"1min":0.76500000000000001,"5min":12.837999999999999,"15min":12.837999999999999},"last":0.5}]},{"osd":2,"last update":"Tue Sep 10 19:33:21 2024","interfaces":[{"interface":"back","average":{"1min":0.53100000000000003,"5min":0.81399999999999995,"15min":0.90800000000000003},"min":{"1min":0.29599999999999999,"5min":0.26600000000000001,"15min":0.23000000000000001},"max":{"1min":0.80400000000000005,"5min":3.3100000000000001,"15min":4.7400000000000002},"last":0.55100000000000005},{"interface":"front","average":{"1min":0.55300000000000005,"5min":0.81499999999999995,"15min":0.90400000000000003},"min":{"1min":0.38800000000000001,"5min":0.29099999999999998,"15min":0.26300000000000001},"max":{"1min":0.754,"5min":6.1059999999999999,"15min":6.1059999999999999},"last":0.61199999999999999}]},{"osd":3,"last update":"Tue Sep 10 19:33:21 2024","interfaces":[{"interface":"back","average":{"1min":0.56299999999999994,"5min":1.2070000000000001,"15min":1.0529999999999999},"min":{"1min":0.34200000000000003,"5min":0.26300000000000001,"15min":0.20200000000000001},"max":{"1min":0.749,"5min":14.515000000000001,"15min":14.515000000000001},"last":0.52000000000000002},{"interface":"front","average":{"1min":0.57399999999999995,"5min":1.097,"15min":1.026},"min":{"1min":0.32500000000000001,"5min":0.32400000000000001,"15min":0.25700000000000001},"max":{"1min":0.80000000000000004,"5min":14.227,"15min":14.227},"last":0.57199999999999995}]},{"osd":4,"last update":"Tue Sep 10 19:33:21 2024","interfaces":[{"interface":"back","average":{"1min":0.437,"5min":0.89000000000000001,"15min":2.528},"min":{"1min":0.311,"5min":0.24299999999999999,"15min":0.222},"max":{"1min":0.56499999999999995,"5min":5.194,"15min":231.923},"last":0.52900000000000003},{"interface":"front","average":{"1min":0.502,"5min":0.85499999999999998,"15min":2.3130000000000002},"min":{"1min":0.34499999999999997,"5min":0.25600000000000001,"15min":0.23100000000000001},"max":{"1min":0.71999999999999997,"5min":6.6029999999999998,"15min":231.322},"last":0.39800000000000002}]},{"osd":5,"last update":"Tue Sep 10 19:33:21 2024","interfaces":[{"interface":"back","average":{"1min":0.5,"5min":1.341,"15min":1.6930000000000001},"min":{"1min":0.375,"5min":0.25800000000000001,"15min":0.24099999999999999},"max":{"1min":0.65000000000000002,"5min":54.441000000000003,"15min":67.382999999999996},"last":0.41899999999999998},{"interface":"front","average":{"1min":0.48999999999999999,"5min":1.633,"15min":2.0739999999999998},"min":{"1min":0.35399999999999998,"5min":0.316,"15min":0.22800000000000001},"max":{"1min":0.78400000000000003,"5min":54.756,"15min":67.75},"last":0.35399999999999998}]},{"osd":6,"last update":"Tue Sep 10 19:33:21 2024","interfaces":[{"interface":"back","average":{"1min":0.54700000000000004,"5min":1.704,"15min":2.4660000000000002},"min":{"1min":0.437,"5min":0.29799999999999999,"15min":0.28000000000000003},"max":{"1min":0.73599999999999999,"5min":47.933999999999997,"15min":181.53800000000001},"last":0.44500000000000001},{"interface":"front","average":{"1min":0.55500000000000005,"5min":1.5409999999999999,"15min":2.2040000000000002},"min":{"1min":0.42099999999999999,"5min":0.34399999999999997,"15min":0.27000000000000002},"max":{"1min":0.76900000000000002,"5min":48.021999999999998,"15min":182.05600000000001},"last":0.47599999999999998}]}]},{"osd":6,"up_from":47,"seq":201863463202,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":899772,"kb_used_data":696,"kb_used_omap":1,"kb_used_meta":898942,"kb_avail":92845380,"statfs":{"total":95995035648,"available":95073669120,"internally_reserved":0,"allocated":712704,"data_stored":427897,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":920516952},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 19:32:48 2024","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":2.2210000000000001,"15min":1.9570000000000001},"min":{"1min":0.33300000000000002,"5min":0.186,"15min":0.17899999999999999},"max":{"1min":0.80400000000000005,"5min":127.395,"15min":127.395},"last":0.66800000000000004},{"interface":"front","average":{"1min":0.51300000000000001,"5min":2.254,"15min":1.9970000000000001},"min":{"1min":0.33500000000000002,"5min":0.20599999999999999,"15min":0.20200000000000001},"max":{"1min":0.68000000000000005,"5min":127.547,"15min":127.547},"last":0.65800000000000003}]},{"osd":1,"last update":"Tue Sep 10 19:33:09 2024","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":1.8540000000000001,"15min":1.238},"min":{"1min":0.46200000000000002,"5min":0.20999999999999999,"15min":0.19800000000000001},"max":{"1min":0.77900000000000003,"5min":39.484999999999999,"15min":39.484999999999999},"last":0.72599999999999998},{"interface":"front","average":{"1min":0.58999999999999997,"5min":1.869,"15min":1.1799999999999999},"min":{"1min":0.41299999999999998,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":0.80300000000000005,"5min":39.472999999999999,"15min":39.472999999999999},"last":0.70899999999999996}]},{"osd":2,"last update":"Tue Sep 10 19:33:20 2024","interfaces":[{"interface":"back","average":{"1min":0.54100000000000004,"5min":0.873,"15min":0.89300000000000002},"min":{"1min":0.35799999999999998,"5min":0.24299999999999999,"15min":0.23799999999999999},"max":{"1min":0.66000000000000003,"5min":4.8330000000000002,"15min":5.9029999999999996},"last":0.56699999999999995},{"interface":"front","average":{"1min":0.53200000000000003,"5min":0.77600000000000002,"15min":0.88100000000000001},"min":{"1min":0.38500000000000001,"5min":0.28899999999999998,"15min":0.25600000000000001},"max":{"1min":0.65200000000000002,"5min":6.4560000000000004,"15min":6.4560000000000004},"last":0.64800000000000002}]},{"osd":3,"last update":"Tue Sep 10 19:33:20 2024","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":1.6859999999999999,"15min":1.3009999999999999},"min":{"1min":0.38400000000000001,"5min":0.30399999999999999,"15min":0.20399999999999999},"max":{"1min":0.749,"5min":42.929000000000002,"15min":42.929000000000002},"last":0.69599999999999995},{"interface":"front","average":{"1min":0.56799999999999995,"5min":1.742,"15min":1.2549999999999999},"min":{"1min":0.41599999999999998,"5min":0.28999999999999998,"15min":0.20499999999999999},"max":{"1min":0.76400000000000001,"5min":43.692999999999998,"15min":43.692999999999998},"last":0.628}]},{"osd":4,"last update":"Tue Sep 10 19:33:20 2024","interfaces":[{"interface":"back","average":{"1min":0.46600000000000003,"5min":0.85899999999999999,"15min":1.3700000000000001},"min":{"1min":0.35299999999999998,"5min":0.25700000000000001,"15min":0.22},"max":{"1min":0.60399999999999998,"5min":6.9859999999999998,"15min":35.552},"last":0.501},{"interface":"front","average":{"1min":0.52600000000000002,"5min":0.85499999999999998,"15min":1.369},"min":{"1min":0.35599999999999998,"5min":0.24399999999999999,"15min":0.24099999999999999},"max":{"1min":0.70499999999999996,"5min":4.5069999999999997,"15min":35.192999999999998},"last":0.59199999999999997}]},{"osd":5,"last update":"Tue Sep 10 19:33:20 2024","interfaces":[{"interface":"back","average":{"1min":0.49399999999999999,"5min":0.79900000000000004,"15min":1.3700000000000001},"min":{"1min":0.38300000000000001,"5min":0.29699999999999999,"15min":0.26500000000000001},"max":{"1min":0.63900000000000001,"5min":7.4050000000000002,"15min":48.216000000000001},"last":0.61099999999999999},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.84999999999999998,"15min":1.5369999999999999},"min":{"1min":0.34399999999999997,"5min":0.30399999999999999,"15min":0.254},"max":{"1min":0.72799999999999998,"5min":4.3719999999999999,"15min":50.372999999999998},"last":0.53300000000000003}]},{"osd":7,"last update":"Tue Sep 10 19:32:48 2024","interfaces":[{"interface":"back","average":{"1min":0.59799999999999998,"5min":0.97599999999999998,"15min":1.542},"min":{"1min":0.42299999999999999,"5min":0.28299999999999997,"15min":0.28100000000000003},"max":{"1min":0.78500000000000003,"5min":7.609,"15min":80.629999999999995},"last":0.68000000000000005},{"interface":"front","average":{"1min":0.63400000000000001,"5min":1.206,"15min":1.7989999999999999},"min":{"1min":0.439,"5min":0.29399999999999998,"15min":0.25700000000000001},"max":{"1min":0.90700000000000003,"5min":6.5659999999999998,"15min":81.088999999999999},"last":0.63700000000000001}]}]},{"osd":5,"up_from":39,"seq":167503724839,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":903504,"kb_used_data":1292,"kb_used_omap":1,"kb_used_meta":902078,"kb_avail":92841648,"statfs":{"total":95995035648,"available":95069847552,"internally_reserved":0,"allocated":1323008,"data_stored":1019405,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":923728216},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 19:32:38 2024","interfaces":[{"interface":"back","average":{"1min":0.502,"5min":0.79800000000000004,"15min":1.024},"min":{"1min":0.30299999999999999,"5min":0.223,"15min":0.187},"max":{"1min":0.75700000000000001,"5min":7.5759999999999996,"15min":7.5759999999999996},"last":0.56299999999999994},{"interface":"front","average":{"1min":0.497,"5min":0.80000000000000004,"15min":1.089},"min":{"1min":0.34100000000000003,"5min":0.20599999999999999,"15min":0.20599999999999999},"max":{"1min":0.65800000000000003,"5min":5.819,"15min":5.819},"last":0.60999999999999999}]},{"osd":1,"last update":"Tue Sep 10 19:32:54 2024","interfaces":[{"interface":"back","average":{"1min":0.51500000000000001,"5min":2.4649999999999999,"15min":1.4690000000000001},"min":{"1min":0.38100000000000001,"5min":0.247,"15min":0.214},"max":{"1min":0.69799999999999995,"5min":54.097999999999999,"15min":54.097999999999999},"last":0.54900000000000004},{"interface":"front","average":{"1min":0.57099999999999995,"5min":2.29,"15min":1.4550000000000001},"min":{"1min":0.39500000000000002,"5min":0.253,"15min":0.221},"max":{"1min":0.73999999999999999,"5min":54.337000000000003,"15min":54.337000000000003},"last":0.66100000000000003}]},{"osd":2,"last update":"Tue Sep 10 19:32:41 2024","interfaces":[{"interface":"back","average":{"1min":0.54300000000000004,"5min":0.73599999999999999,"15min":0.98599999999999999},"min":{"1min":0.34999999999999998,"5min":0.26500000000000001,"15min":0.22900000000000001},"max":{"1min":0.71699999999999997,"5min":7.4489999999999998,"15min":8.6920000000000002},"last":0.46500000000000002},{"interface":"front","average":{"1min":0.54200000000000004,"5min":0.83199999999999996,"15min":0.98499999999999999},"min":{"1min":0.38800000000000001,"5min":0.28399999999999997,"15min":0.21199999999999999},"max":{"1min":0.69599999999999995,"5min":10.045999999999999,"15min":10.045999999999999},"last":0.53300000000000003}]},{"osd":3,"last update":"Tue Sep 10 19:32:41 2024","interfaces":[{"interface":"back","average":{"1min":0.59699999999999998,"5min":2.286,"15min":1.599},"min":{"1min":0.44500000000000001,"5min":0.26600000000000001,"15min":0.19900000000000001},"max":{"1min":0.79100000000000004,"5min":54.832000000000001,"15min":54.832000000000001},"last":0.621},{"interface":"front","average":{"1min":0.57899999999999996,"5min":2.347,"15min":1.613},"min":{"1min":0.46700000000000003,"5min":0.28100000000000003,"15min":0.246},"max":{"1min":0.80600000000000005,"5min":54.850999999999999,"15min":54.850999999999999},"last":0.58299999999999996}]},{"osd":4,"last update":"Tue Sep 10 19:32:41 2024","interfaces":[{"interface":"back","average":{"1min":0.48299999999999998,"5min":1.9970000000000001,"15min":2.101},"min":{"1min":0.33400000000000002,"5min":0.252,"15min":0.252},"max":{"1min":0.65400000000000003,"5min":132.41900000000001,"15min":132.41900000000001},"last":0.42299999999999999},{"interface":"front","average":{"1min":0.57399999999999995,"5min":1.99,"15min":2.2090000000000001},"min":{"1min":0.35099999999999998,"5min":0.29099999999999998,"15min":0.217},"max":{"1min":0.70699999999999996,"5min":132.46700000000001,"15min":132.46700000000001},"last":0.32900000000000001}]},{"osd":6,"last update":"Tue Sep 10 19:33:23 2024","interfaces":[{"interface":"back","average":{"1min":0.58999999999999997,"5min":0.874,"15min":1.6599999999999999},"min":{"1min":0.36699999999999999,"5min":0.28699999999999998,"15min":0.28399999999999997},"max":{"1min":0.80200000000000005,"5min":7.6269999999999998,"15min":111.328},"last":0.39100000000000001},{"interface":"front","average":{"1min":0.56699999999999995,"5min":0.876,"15min":1.6679999999999999},"min":{"1min":0.35399999999999998,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.79000000000000004,"5min":4.1219999999999999,"15min":111.705},"last":0.373}]},{"osd":7,"last update":"Tue Sep 10 19:32:35 2024","interfaces":[{"interface":"back","average":{"1min":0.59899999999999998,"5min":1.028,"15min":1.369},"min":{"1min":0.39200000000000002,"5min":0.29099999999999998,"15min":0.28599999999999998},"max":{"1min":0.77300000000000002,"5min":5.0640000000000001,"15min":6.7869999999999999},"last":0.44700000000000001},{"interface":"front","average":{"1min":0.55000000000000004,"5min":0.98799999999999999,"15min":1.3580000000000001},"min":{"1min":0.371,"5min":0.29899999999999999,"15min":0.27100000000000002},"max":{"1min":0.68400000000000005,"5min":5.6349999999999998,"15min":9.4909999999999997},"last":0.48399999999999999}]}]},{"osd":4,"up_from":32,"seq":137438953771,"num_pgs":55,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":901712,"kb_used_data":716,"kb_used_omap":1,"kb_used_meta":900862,"kb_avail":92843440,"statfs":{"total":95995035648,"available":95071682560,"internally_reserved":0,"allocated":733184,"data_stored":427782,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":922483032},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 19:32:33 2024","interfaces":[{"interface":"back","average":{"1min":0.57099999999999995,"5min":0.76900000000000002,"15min":1.821},"min":{"1min":0.25,"5min":0.20100000000000001,"15min":0.20100000000000001},"max":{"1min":0.85499999999999998,"5min":3.6110000000000002,"15min":39.097000000000001},"last":0.39000000000000001},{"interface":"front","average":{"1min":0.60699999999999998,"5min":0.70199999999999996,"15min":2.0230000000000001},"min":{"1min":0.39400000000000002,"5min":0.24099999999999999,"15min":0.20499999999999999},"max":{"1min":0.93200000000000005,"5min":5.6379999999999999,"15min":39.112000000000002},"last":0.52000000000000002}]},{"osd":1,"last update":"Tue Sep 10 19:32:59 2024","interfaces":[{"interface":"back","average":{"1min":0.56100000000000005,"5min":1.712,"15min":1.177},"min":{"1min":0.36599999999999999,"5min":0.23699999999999999,"15min":0.20599999999999999},"max":{"1min":0.88900000000000001,"5min":75.620000000000005,"15min":75.620000000000005},"last":0.34699999999999998},{"interface":"front","average":{"1min":0.56599999999999995,"5min":1.5249999999999999,"15min":1.042},"min":{"1min":0.42499999999999999,"5min":0.23499999999999999,"15min":0.23400000000000001},"max":{"1min":0.74099999999999999,"5min":75.230999999999995,"15min":75.230999999999995},"last":0.42199999999999999}]},{"osd":2,"last update":"Tue Sep 10 19:32:26 2024","interfaces":[{"interface":"back","average":{"1min":0.54300000000000004,"5min":0.70299999999999996,"15min":0.93200000000000005},"min":{"1min":0.314,"5min":0.25600000000000001,"15min":0.254},"max":{"1min":0.83599999999999997,"5min":5.5609999999999999,"15min":5.5609999999999999},"last":0.46100000000000002},{"interface":"front","average":{"1min":0.53700000000000003,"5min":0.752,"15min":0.94899999999999995},"min":{"1min":0.32900000000000001,"5min":0.28599999999999998,"15min":0.253},"max":{"1min":0.71399999999999997,"5min":3.2120000000000002,"15min":6.7690000000000001},"last":0.56499999999999995}]},{"osd":3,"last update":"Tue Sep 10 19:32:26 2024","interfaces":[{"interface":"back","average":{"1min":0.56699999999999995,"5min":0.85499999999999998,"15min":1.0569999999999999},"min":{"1min":0.40899999999999997,"5min":0.28599999999999998,"15min":0.20999999999999999},"max":{"1min":0.73899999999999999,"5min":3.1880000000000002,"15min":15.436},"last":0.503},{"interface":"front","average":{"1min":0.53000000000000003,"5min":0.93400000000000005,"15min":1.0820000000000001},"min":{"1min":0.30199999999999999,"5min":0.26600000000000001,"15min":0.246},"max":{"1min":0.79900000000000004,"5min":5.3140000000000001,"15min":14.954000000000001},"last":0.48299999999999998}]},{"osd":5,"last update":"Tue Sep 10 19:32:48 2024","interfaces":[{"interface":"back","average":{"1min":0.48399999999999999,"5min":0.81100000000000005,"15min":2.2810000000000001},"min":{"1min":0.33200000000000002,"5min":0.29899999999999999,"15min":0.224},"max":{"1min":0.65700000000000003,"5min":4.7190000000000003,"15min":227.726},"last":0.53300000000000003},{"interface":"front","average":{"1min":0.51300000000000001,"5min":0.76800000000000002,"15min":2.2909999999999999},"min":{"1min":0.35299999999999998,"5min":0.27400000000000002,"15min":0.20699999999999999},"max":{"1min":0.70599999999999996,"5min":5.6699999999999999,"15min":227.70500000000001},"last":0.55200000000000005}]},{"osd":6,"last update":"Tue Sep 10 19:32:26 2024","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.77900000000000003,"15min":1.2470000000000001},"min":{"1min":0.39600000000000002,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":0.69699999999999995,"5min":5.8250000000000002,"15min":8.1010000000000009},"last":0.58299999999999996},{"interface":"front","average":{"1min":0.54400000000000004,"5min":1.016,"15min":1.3600000000000001},"min":{"1min":0.34100000000000003,"5min":0.254,"15min":0.247},"max":{"1min":0.76100000000000001,"5min":6.0099999999999998,"15min":16.815000000000001},"last":0.59199999999999997}]},{"osd":7,"last update":"Tue Sep 10 19:32:33 2024","interfaces":[{"interface":"back","average":{"1min":0.54600000000000004,"5min":1.2410000000000001,"15min":1.4099999999999999},"min":{"1min":0.38400000000000001,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.71199999999999997,"5min":14.166,"15min":16.853999999999999},"last":0.63200000000000001},{"interface":"front","average":{"1min":0.52200000000000002,"5min":1.214,"15min":1.4350000000000001},"min":{"1min":0.376,"5min":0.25700000000000001,"15min":0.23699999999999999},"max":{"1min":0.67800000000000005,"5min":14.380000000000001,"15min":14.380000000000001},"last":0.61899999999999999}]}]},{"osd":3,"up_from":26,"seq":111669150000,"num_pgs":60,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1181796,"kb_used_data":1312,"kb_used_omap":1,"kb_used_meta":1180350,"kb_avail":92563356,"statfs":{"total":95995035648,"available":94784876544,"internally_reserved":0,"allocated":1343488,"data_stored":1019614,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":1208678744},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 19:32:43 2024","interfaces":[{"interface":"back","average":{"1min":0.33000000000000002,"5min":3.5640000000000001,"15min":1.579},"min":{"1min":0.19500000000000001,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":0.52300000000000002,"5min":117.655,"15min":117.655},"last":0.32200000000000001},{"interface":"front","average":{"1min":0.34100000000000003,"5min":3.5409999999999999,"15min":1.554},"min":{"1min":0.219,"5min":0.185,"15min":0.184},"max":{"1min":0.745,"5min":117.81399999999999,"15min":117.81399999999999},"last":0.26700000000000002}]},{"osd":1,"last update":"Tue Sep 10 19:33:08 2024","interfaces":[{"interface":"back","average":{"1min":0.39200000000000002,"5min":0.876,"15min":0.53100000000000003},"min":{"1min":0.224,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":1.0960000000000001,"5min":5.1779999999999999,"15min":5.1779999999999999},"last":0.46100000000000002},{"interface":"front","average":{"1min":0.48799999999999999,"5min":0.90800000000000003,"15min":0.55000000000000004},"min":{"1min":0.221,"5min":0.19700000000000001,"15min":0.19500000000000001},"max":{"1min":1.873,"5min":4.0919999999999996,"15min":4.4100000000000001},"last":0.35199999999999998}]},{"osd":2,"last update":"Tue Sep 10 19:33:18 2024","interfaces":[{"interface":"back","average":{"1min":0.48899999999999999,"5min":0.60899999999999999,"15min":0.501},"min":{"1min":0.32200000000000001,"5min":0.25800000000000001,"15min":0.17599999999999999},"max":{"1min":1.3300000000000001,"5min":3.7730000000000001,"15min":4.3769999999999998},"last":0.51000000000000001},{"interface":"front","average":{"1min":0.45000000000000001,"5min":0.61799999999999999,"15min":0.495},"min":{"1min":0.27800000000000002,"5min":0.23100000000000001,"15min":0.16400000000000001},"max":{"1min":1.6579999999999999,"5min":4.2990000000000004,"15min":4.2990000000000004},"last":0.29499999999999998}]},{"osd":4,"last update":"Tue Sep 10 19:32:33 2024","interfaces":[{"interface":"back","average":{"1min":0.56200000000000006,"5min":0.88,"15min":1.262},"min":{"1min":0.39800000000000002,"5min":0.30099999999999999,"15min":0.254},"max":{"1min":0.81399999999999995,"5min":4.4169999999999998,"15min":81.334000000000003},"last":0.63700000000000001},{"interface":"front","average":{"1min":0.58199999999999996,"5min":0.92400000000000004,"15min":1.3560000000000001},"min":{"1min":0.40400000000000003,"5min":0.318,"15min":0.22700000000000001},"max":{"1min":0.81399999999999995,"5min":4.343,"15min":81.381},"last":0.85299999999999998}]},{"osd":5,"last update":"Tue Sep 10 19:32:48 2024","interfaces":[{"interface":"back","average":{"1min":0.58899999999999997,"5min":0.89100000000000001,"15min":1.3740000000000001},"min":{"1min":0.41899999999999998,"5min":0.28000000000000003,"15min":0.27900000000000003},"max":{"1min":0.95999999999999996,"5min":4.4560000000000004,"15min":104.672},"last":0.71899999999999997},{"interface":"front","average":{"1min":0.61399999999999999,"5min":0.98699999999999999,"15min":1.325},"min":{"1min":0.41799999999999998,"5min":0.314,"15min":0.24299999999999999},"max":{"1min":1.159,"5min":3.4900000000000002,"15min":105.971},"last":0.81399999999999995}]},{"osd":6,"last update":"Tue Sep 10 19:33:18 2024","interfaces":[{"interface":"back","average":{"1min":0.59399999999999997,"5min":2.5739999999999998,"15min":2.0590000000000002},"min":{"1min":0.46500000000000002,"5min":0.33400000000000002,"15min":0.23699999999999999},"max":{"1min":1.056,"5min":148.09299999999999,"15min":148.09299999999999},"last":0.67200000000000004},{"interface":"front","average":{"1min":0.65900000000000003,"5min":2.5499999999999998,"15min":1.9790000000000001},"min":{"1min":0.439,"5min":0.37,"15min":0.26800000000000002},"max":{"1min":1.393,"5min":148.31999999999999,"15min":148.31999999999999},"last":0.746}]},{"osd":7,"last update":"Tue Sep 10 19:32:48 2024","interfaces":[{"interface":"back","average":{"1min":0.65600000000000003,"5min":1.151,"15min":1.3979999999999999},"min":{"1min":0.443,"5min":0.33600000000000002,"15min":0.26100000000000001},"max":{"1min":1.361,"5min":6.8639999999999999,"15min":104.749},"last":0.65300000000000002},{"interface":"front","average":{"1min":0.60199999999999998,"5min":1.0589999999999999,"15min":1.319},"min":{"1min":0.48099999999999998,"5min":0.39200000000000002,"15min":0.26100000000000001},"max":{"1min":1.028,"5min":5.8929999999999998,"15min":104.69499999999999},"last":0.77500000000000002}]}]},{"osd":2,"up_from":18,"seq":77309411637,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":306768,"kb_used_data":972,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":93438384,"statfs":{"total":95995035648,"available":95680905216,"internally_reserved":0,"allocated":995328,"data_stored":690900,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 19:32:45 2024","interfaces":[{"interface":"back","average":{"1min":0.34699999999999998,"5min":0.495,"15min":0.46400000000000002},"min":{"1min":0.20899999999999999,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":0.63200000000000001,"5min":2.9129999999999998,"15min":3.6299999999999999},"last":0.46899999999999997},{"interface":"front","average":{"1min":0.36299999999999999,"5min":0.54700000000000004,"15min":0.51600000000000001},"min":{"1min":0.26500000000000001,"5min":0.249,"15min":0.19600000000000001},"max":{"1min":0.57199999999999995,"5min":3.2010000000000001,"15min":3.9300000000000002},"last":0.41799999999999998}]},{"osd":1,"last update":"Tue Sep 10 19:33:00 2024","interfaces":[{"interface":"back","average":{"1min":0.35099999999999998,"5min":3.3700000000000001,"15min":1.3480000000000001},"min":{"1min":0.20000000000000001,"5min":0.193,"15min":0.193},"max":{"1min":0.67600000000000005,"5min":111.398,"15min":111.398},"last":0.39800000000000002},{"interface":"front","average":{"1min":0.40300000000000002,"5min":3.4289999999999998,"15min":1.4059999999999999},"min":{"1min":0.311,"5min":0.245,"15min":0.22500000000000001},"max":{"1min":0.70899999999999996,"5min":111.36,"15min":111.36},"last":0.442}]},{"osd":3,"last update":"Tue Sep 10 19:32:23 2024","interfaces":[{"interface":"back","average":{"1min":0.38800000000000001,"5min":0.95799999999999996,"15min":0.71099999999999997},"min":{"1min":0.251,"5min":0.23100000000000001,"15min":0.156},"max":{"1min":0.61799999999999999,"5min":25.033000000000001,"15min":25.033000000000001},"last":0.48499999999999999},{"interface":"front","average":{"1min":0.36699999999999999,"5min":0.95399999999999996,"15min":0.70299999999999996},"min":{"1min":0.26200000000000001,"5min":0.21099999999999999,"15min":0.17299999999999999},"max":{"1min":0.60299999999999998,"5min":24.890999999999998,"15min":24.890999999999998},"last":0.499}]},{"osd":4,"last update":"Tue Sep 10 19:32:30 2024","interfaces":[{"interface":"back","average":{"1min":0.63800000000000001,"5min":0.78500000000000003,"15min":0.99099999999999999},"min":{"1min":0.48099999999999998,"5min":0.316,"15min":0.23200000000000001},"max":{"1min":0.87,"5min":4.6399999999999997,"15min":17.137},"last":0.76600000000000001},{"interface":"front","average":{"1min":0.63100000000000001,"5min":0.73799999999999999,"15min":0.96399999999999997},"min":{"1min":0.49399999999999999,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.84999999999999998,"5min":4.6289999999999996,"15min":17.041},"last":0.75700000000000001}]},{"osd":5,"last update":"Tue Sep 10 19:32:54 2024","interfaces":[{"interface":"back","average":{"1min":0.56499999999999995,"5min":0.84599999999999997,"15min":1.248},"min":{"1min":0.41099999999999998,"5min":0.28699999999999998,"15min":0.24099999999999999},"max":{"1min":0.77500000000000002,"5min":5.3029999999999999,"15min":50.848999999999997},"last":0.66900000000000004},{"interface":"front","average":{"1min":0.63800000000000001,"5min":0.77400000000000002,"15min":1.274},"min":{"1min":0.44800000000000001,"5min":0.307,"15min":0.23899999999999999},"max":{"1min":0.874,"5min":5.0179999999999998,"15min":50.817999999999998},"last":0.72999999999999998}]},{"osd":6,"last update":"Tue Sep 10 19:32:23 2024","interfaces":[{"interface":"back","average":{"1min":0.58799999999999997,"5min":0.85199999999999998,"15min":1.05},"min":{"1min":0.40100000000000002,"5min":0.34000000000000002,"15min":0.23300000000000001},"max":{"1min":0.83999999999999997,"5min":4.8650000000000002,"15min":17.859999999999999},"last":0.75},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.76400000000000001,"15min":0.95299999999999996},"min":{"1min":0.38400000000000001,"5min":0.308,"15min":0.25900000000000001},"max":{"1min":0.72499999999999998,"5min":3.819,"15min":17.274000000000001},"last":0.68799999999999994}]},{"osd":7,"last update":"Tue Sep 10 19:32:30 2024","interfaces":[{"interface":"back","average":{"1min":0.623,"5min":0.77200000000000002,"15min":1.022},"min":{"1min":0.42999999999999999,"5min":0.34000000000000002,"15min":0.32400000000000001},"max":{"1min":0.82299999999999995,"5min":5.1900000000000004,"15min":10.413},"last":0.67700000000000005},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.81399999999999995,"15min":1.0009999999999999},"min":{"1min":0.442,"5min":0.29999999999999999,"15min":0.27900000000000003},"max":{"1min":0.754,"5min":5.2039999999999997,"15min":11.047000000000001},"last":0.77400000000000002}]}]},{"osd":1,"up_from":276,"seq":1185410973827,"num_pgs":51,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":310772,"kb_used_data":744,"kb_used_omap":6,"kb_used_meta":310009,"kb_avail":93434380,"statfs":{"total":95995035648,"available":95676805120,"internally_reserved":0,"allocated":761856,"data_stored":436671,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6335,"internal_metadata":317450049},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 19:32:45 2024","interfaces":[{"interface":"back","average":{"1min":0.28699999999999998,"5min":0.75800000000000001,"15min":0.58199999999999996},"min":{"1min":0.16400000000000001,"5min":0.16400000000000001,"15min":0.16400000000000001},"max":{"1min":0.40899999999999997,"5min":3.5990000000000002,"15min":3.5990000000000002},"last":0.245},{"interface":"front","average":{"1min":0.308,"5min":0.93200000000000005,"15min":0.59899999999999998},"min":{"1min":0.20100000000000001,"5min":0.192,"15min":0.192},"max":{"1min":0.46700000000000003,"5min":4.6079999999999997,"15min":4.6079999999999997},"last":0.374}]},{"osd":2,"last update":"Tue Sep 10 19:33:10 2024","interfaces":[{"interface":"back","average":{"1min":0.36199999999999999,"5min":0.747,"15min":0.48999999999999999},"min":{"1min":0.245,"5min":0.24099999999999999,"15min":0.19},"max":{"1min":0.60099999999999998,"5min":4.1779999999999999,"15min":4.1779999999999999},"last":0.312},{"interface":"front","average":{"1min":0.374,"5min":0.78800000000000003,"15min":0.51800000000000002},"min":{"1min":0.27100000000000002,"5min":0.22800000000000001,"15min":0.222},"max":{"1min":0.53200000000000003,"5min":4.5810000000000004,"15min":4.5810000000000004},"last":0.32700000000000001}]},{"osd":3,"last update":"Tue Sep 10 19:33:10 2024","interfaces":[{"interface":"back","average":{"1min":0.371,"5min":1.077,"15min":0.60099999999999998},"min":{"1min":0.27800000000000002,"5min":0.23799999999999999,"15min":0.221},"max":{"1min":0.54100000000000004,"5min":11.494999999999999,"15min":11.494999999999999},"last":0.28199999999999997},{"interface":"front","average":{"1min":0.38800000000000001,"5min":1.276,"15min":0.65400000000000003},"min":{"1min":0.26200000000000001,"5min":0.26200000000000001,"15min":0.17599999999999999},"max":{"1min":0.57099999999999995,"5min":21.545000000000002,"15min":21.545000000000002},"last":0.35799999999999998}]},{"osd":4,"last update":"Tue Sep 10 19:33:10 2024","interfaces":[{"interface":"back","average":{"1min":0.50900000000000001,"5min":3.569,"15min":1.6859999999999999},"min":{"1min":0.35599999999999998,"5min":0.28599999999999998,"15min":0.26600000000000001},"max":{"1min":0.65600000000000003,"5min":271.53100000000001,"15min":271.53100000000001},"last":0.60899999999999999},{"interface":"front","average":{"1min":0.53900000000000003,"5min":3.5630000000000002,"15min":1.702},"min":{"1min":0.378,"5min":0.30199999999999999,"15min":0.27600000000000002},"max":{"1min":0.73299999999999998,"5min":271.483,"15min":271.483},"last":0.54600000000000004}]},{"osd":5,"last update":"Tue Sep 10 19:33:10 2024","interfaces":[{"interface":"back","average":{"1min":0.55400000000000005,"5min":3.621,"15min":1.835},"min":{"1min":0.44800000000000001,"5min":0.252,"15min":0.252},"max":{"1min":0.65200000000000002,"5min":212.727,"15min":212.727},"last":0.67900000000000005},{"interface":"front","average":{"1min":0.58599999999999997,"5min":3.613,"15min":2.0409999999999999},"min":{"1min":0.46000000000000002,"5min":0.32100000000000001,"15min":0.313},"max":{"1min":0.71399999999999997,"5min":212.74799999999999,"15min":212.74799999999999},"last":0.64700000000000002}]},{"osd":6,"last update":"Tue Sep 10 19:33:10 2024","interfaces":[{"interface":"back","average":{"1min":0.55500000000000005,"5min":1.1899999999999999,"15min":1.625},"min":{"1min":0.41499999999999998,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.75800000000000001,"5min":7.5270000000000001,"15min":27.379999999999999},"last":0.69999999999999996},{"interface":"front","average":{"1min":0.54500000000000004,"5min":1.117,"15min":1.587},"min":{"1min":0.40699999999999997,"5min":0.33500000000000002,"15min":0.30299999999999999},"max":{"1min":0.73999999999999999,"5min":5.8849999999999998,"15min":26.812999999999999},"last":0.58699999999999997}]},{"osd":7,"last update":"Tue Sep 10 19:33:10 2024","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":3.4729999999999999,"15min":1.829},"min":{"1min":0.46800000000000003,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.75,"5min":242.33099999999999,"15min":242.33099999999999},"last":0.65300000000000002},{"interface":"front","average":{"1min":0.58599999999999997,"5min":3.4689999999999999,"15min":1.831},"min":{"1min":0.436,"5min":0.39000000000000001,"15min":0.33200000000000002},"max":{"1min":0.72399999999999998,"5min":242.31,"15min":242.31},"last":0.71099999999999997}]}]},{"osd":0,"up_from":301,"seq":1292785156207,"num_pgs":51,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":355020,"kb_used_data":712,"kb_used_omap":4,"kb_used_meta":354171,"kb_avail":93390132,"statfs":{"total":95995035648,"available":95631495168,"internally_reserved":0,"allocated":729088,"data_stored":428489,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":362671622},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Sep 10 19:32:43 2024","interfaces":[{"interface":"back","average":{"1min":0.34599999999999997,"5min":1.004,"15min":0.68799999999999994},"min":{"1min":0.19,"5min":0.19,"15min":0.17499999999999999},"max":{"1min":0.55700000000000005,"5min":6.0369999999999999,"15min":6.0369999999999999},"last":0.45700000000000002},{"interface":"front","average":{"1min":0.372,"5min":0.84799999999999998,"15min":0.59299999999999997},"min":{"1min":0.23200000000000001,"5min":0.188,"15min":0.16900000000000001},"max":{"1min":0.57199999999999995,"5min":3.7909999999999999,"15min":3.7909999999999999},"last":0.41399999999999998}]},{"osd":2,"last update":"Tue Sep 10 19:32:43 2024","interfaces":[{"interface":"back","average":{"1min":0.35399999999999998,"5min":0.58999999999999997,"15min":0.57599999999999996},"min":{"1min":0.20499999999999999,"5min":0.20499999999999999,"15min":0.20399999999999999},"max":{"1min":0.50700000000000001,"5min":4.3230000000000004,"15min":4.3230000000000004},"last":0.47199999999999998},{"interface":"front","average":{"1min":0.38800000000000001,"5min":0.81899999999999995,"15min":0.67200000000000004},"min":{"1min":0.27200000000000002,"5min":0.249,"15min":0.22600000000000001},"max":{"1min":0.62,"5min":5.1079999999999997,"15min":5.1079999999999997},"last":0.376}]},{"osd":3,"last update":"Tue Sep 10 19:32:43 2024","interfaces":[{"interface":"back","average":{"1min":0.38900000000000001,"5min":1.7509999999999999,"15min":0.95699999999999996},"min":{"1min":0.27100000000000002,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.54200000000000004,"5min":89.468999999999994,"15min":89.468999999999994},"last":0.44600000000000001},{"interface":"front","average":{"1min":0.41599999999999998,"5min":1.8480000000000001,"15min":1.0209999999999999},"min":{"1min":0.28399999999999997,"5min":0.216,"15min":0.216},"max":{"1min":0.55300000000000005,"5min":89.272999999999996,"15min":89.272999999999996},"last":0.5}]},{"osd":4,"last update":"Tue Sep 10 19:32:43 2024","interfaces":[{"interface":"back","average":{"1min":0.57299999999999995,"5min":1.0509999999999999,"15min":2.9409999999999998},"min":{"1min":0.375,"5min":0.32700000000000001,"15min":0.25800000000000001},"max":{"1min":0.89700000000000002,"5min":4.9729999999999999,"15min":53.670000000000002},"last":0.60499999999999998},{"interface":"front","average":{"1min":0.60699999999999998,"5min":0.871,"15min":2.8730000000000002},"min":{"1min":0.39200000000000002,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.82499999999999996,"5min":4.0369999999999999,"15min":53.935000000000002},"last":0.68100000000000005}]},{"osd":5,"last update":"Tue Sep 10 19:32:43 2024","interfaces":[{"interface":"back","average":{"1min":0.60999999999999999,"5min":0.88200000000000001,"15min":2.6030000000000002},"min":{"1min":0.374,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":0.82299999999999995,"5min":4.2779999999999996,"15min":53.884999999999998},"last":0.79100000000000004},{"interface":"front","average":{"1min":0.622,"5min":0.95999999999999996,"15min":2.5670000000000002},"min":{"1min":0.40500000000000003,"5min":0.34899999999999998,"15min":0.309},"max":{"1min":0.874,"5min":5.0519999999999996,"15min":53.875999999999998},"last":0.67200000000000004}]},{"osd":6,"last update":"Tue Sep 10 19:32:43 2024","interfaces":[{"interface":"back","average":{"1min":0.625,"5min":1.073,"15min":3.085},"min":{"1min":0.432,"5min":0.311,"15min":0.311},"max":{"1min":0.85999999999999999,"5min":5.1449999999999996,"15min":56.901000000000003},"last":0.63800000000000001},{"interface":"front","average":{"1min":0.625,"5min":1.0649999999999999,"15min":3.044},"min":{"1min":0.39400000000000002,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.89600000000000002,"5min":6.6840000000000002,"15min":55.177},"last":0.80600000000000005}]},{"osd":7,"last update":"Tue Sep 10 19:32:43 2024","interfaces":[{"interface":"back","average":{"1min":0.625,"5min":1,"15min":1.155},"min":{"1min":0.40000000000000002,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.91800000000000004,"5min":6.3959999999999999,"15min":6.3959999999999999},"last":0.76900000000000002},{"interface":"front","average":{"1min":0.64800000000000002,"5min":1.0189999999999999,"15min":1.149},"min":{"1min":0.47399999999999998,"5min":0.375,"15min":0.34000000000000002},"max":{"1min":0.93000000000000005,"5min":5.0810000000000004,"15min":5.0810000000000004},"last":0.64800000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":966,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1236,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1236,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":362,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":2008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1238,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":1788,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":924,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T19:33:30.246 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph pg dump --format=json 2024-09-10T19:33:30.495 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:33:30.616 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:33:30 smithi183 ceph-mon[29736]: 2.14 deep-scrub starts 2024-09-10T19:33:30.616 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:33:30 smithi183 ceph-mon[29736]: 2.14 deep-scrub ok 2024-09-10T19:33:30.616 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:33:30 smithi183 ceph-mon[29736]: 6.1e scrub starts 2024-09-10T19:33:30.616 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:33:30 smithi183 ceph-mon[29736]: 6.1e scrub ok 2024-09-10T19:33:30.617 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:33:30 smithi183 ceph-mon[41562]: 2.14 deep-scrub starts 2024-09-10T19:33:30.617 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:33:30 smithi183 ceph-mon[41562]: 2.14 deep-scrub ok 2024-09-10T19:33:30.617 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:33:30 smithi183 ceph-mon[41562]: 6.1e scrub starts 2024-09-10T19:33:30.617 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:33:30 smithi183 ceph-mon[41562]: 6.1e scrub ok 2024-09-10T19:33:30.715 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:33:30 smithi191 ceph-mon[32884]: 2.14 deep-scrub starts 2024-09-10T19:33:30.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:33:30 smithi191 ceph-mon[32884]: 2.14 deep-scrub ok 2024-09-10T19:33:30.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:33:30 smithi191 ceph-mon[32884]: 6.1e scrub starts 2024-09-10T19:33:30.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:33:30 smithi191 ceph-mon[32884]: 6.1e scrub ok 2024-09-10T19:33:30.716 INFO:journalctl@ceph.iscsi.iscsi.a.smithi191.stdout:Sep 10 19:33:30 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-iscsi-iscsi-a[61816]: debug there is no tcmu-runner data available 2024-09-10T19:33:31.682 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Sep 10 19:33:31 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-y[29989]: ::ffff:172.21.15.191 - - [10/Sep/2024:19:33:31] "GET /metrics HTTP/1.1" 200 222377 "" "Prometheus/2.43.0" 2024-09-10T19:33:31.682 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:33:31 smithi183 ceph-mon[29736]: from='client.26381 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-10T19:33:31.682 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:33:31 smithi183 ceph-mon[29736]: pgmap v1074: 142 pgs: 142 active+clean; 582 KiB data, 5.8 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:33:31.683 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:33:31 smithi183 ceph-mon[29736]: from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:33:31.683 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:33:31 smithi183 ceph-mon[29736]: 2.1e scrub starts 2024-09-10T19:33:31.683 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:33:31 smithi183 ceph-mon[29736]: 2.1e scrub ok 2024-09-10T19:33:31.683 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:33:31 smithi183 ceph-mon[29736]: 1.3 deep-scrub starts 2024-09-10T19:33:31.683 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:33:31 smithi183 ceph-mon[29736]: 1.3 deep-scrub ok 2024-09-10T19:33:31.683 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:33:31 smithi183 ceph-mon[29736]: 2.12 scrub starts 2024-09-10T19:33:31.683 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:33:31 smithi183 ceph-mon[29736]: 2.12 scrub ok 2024-09-10T19:33:31.683 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:33:31 smithi183 ceph-mon[41562]: from='client.26381 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-10T19:33:31.683 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:33:31 smithi183 ceph-mon[41562]: pgmap v1074: 142 pgs: 142 active+clean; 582 KiB data, 5.8 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:33:31.683 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:33:31 smithi183 ceph-mon[41562]: from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:33:31.683 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:33:31 smithi183 ceph-mon[41562]: 2.1e scrub starts 2024-09-10T19:33:31.684 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:33:31 smithi183 ceph-mon[41562]: 2.1e scrub ok 2024-09-10T19:33:31.684 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:33:31 smithi183 ceph-mon[41562]: 1.3 deep-scrub starts 2024-09-10T19:33:31.684 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:33:31 smithi183 ceph-mon[41562]: 1.3 deep-scrub ok 2024-09-10T19:33:31.684 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:33:31 smithi183 ceph-mon[41562]: 2.12 scrub starts 2024-09-10T19:33:31.684 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:33:31 smithi183 ceph-mon[41562]: 2.12 scrub ok 2024-09-10T19:33:31.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:33:31 smithi191 ceph-mon[32884]: from='client.26381 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-10T19:33:31.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:33:31 smithi191 ceph-mon[32884]: pgmap v1074: 142 pgs: 142 active+clean; 582 KiB data, 5.8 GiB used, 709 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-10T19:33:31.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:33:31 smithi191 ceph-mon[32884]: from='client.24409 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-10T19:33:31.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:33:31 smithi191 ceph-mon[32884]: 2.1e scrub starts 2024-09-10T19:33:31.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:33:31 smithi191 ceph-mon[32884]: 2.1e scrub ok 2024-09-10T19:33:31.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:33:31 smithi191 ceph-mon[32884]: 1.3 deep-scrub starts 2024-09-10T19:33:31.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:33:31 smithi191 ceph-mon[32884]: 1.3 deep-scrub ok 2024-09-10T19:33:31.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:33:31 smithi191 ceph-mon[32884]: 2.12 scrub starts 2024-09-10T19:33:31.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:33:31 smithi191 ceph-mon[32884]: 2.12 scrub ok 2024-09-10T19:33:32.403 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-09-10T19:33:32.405 INFO:teuthology.orchestra.run.smithi183.stderr:dumped all 2024-09-10T19:33:32.684 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:33:32 smithi183 ceph-mon[41562]: 1.8 scrub starts 2024-09-10T19:33:32.685 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:33:32 smithi183 ceph-mon[41562]: 1.8 scrub ok 2024-09-10T19:33:32.685 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:33:32 smithi183 ceph-mon[41562]: 4.a scrub starts 2024-09-10T19:33:32.685 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:33:32 smithi183 ceph-mon[41562]: 4.a scrub ok 2024-09-10T19:33:32.685 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:33:32 smithi183 ceph-mon[29736]: 1.8 scrub starts 2024-09-10T19:33:32.685 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:33:32 smithi183 ceph-mon[29736]: 1.8 scrub ok 2024-09-10T19:33:32.685 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:33:32 smithi183 ceph-mon[29736]: 4.a scrub starts 2024-09-10T19:33:32.685 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:33:32 smithi183 ceph-mon[29736]: 4.a scrub ok 2024-09-10T19:33:32.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:33:32 smithi191 ceph-mon[32884]: 1.8 scrub starts 2024-09-10T19:33:32.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:33:32 smithi191 ceph-mon[32884]: 1.8 scrub ok 2024-09-10T19:33:32.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:33:32 smithi191 ceph-mon[32884]: 4.a scrub starts 2024-09-10T19:33:32.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:33:32 smithi191 ceph-mon[32884]: 4.a scrub ok 2024-09-10T19:33:33.250 INFO:teuthology.orchestra.run.smithi183.stdout:{"pg_ready":true,"pg_map":{"version":1075,"stamp":"2024-09-10T19:33:32.030790+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596188,"num_objects":196,"num_object_clones":0,"num_object_copies":588,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":196,"num_whiteouts":0,"num_read":2127,"num_read_kb":1879,"num_write":1002,"num_write_kb":43,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":526,"num_bytes_recovered":801166,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2080,"ondisk_log_size":2080,"up":426,"acting":426,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":426,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":6036360,"kb_used_data":7544,"kb_used_omap":20,"kb_used_meta":6028587,"kb_avail":743924856,"statfs":{"total":767960285184,"available":761779052544,"internally_reserved":0,"allocated":7725056,"data_stored":5254685,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":21161,"internal_metadata":6173273431},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.739650"},"pg_stats":[{"pgid":"6.1b","version":"0'0","reported_seq":226,"reported_epoch":460,"state":"active+clean","last_fresh":"2024-09-10T19:33:24.309179+0000","last_change":"2024-09-10T19:33:10.306914+0000","last_active":"2024-09-10T19:33:24.309179+0000","last_peered":"2024-09-10T19:33:24.309179+0000","last_clean":"2024-09-10T19:33:24.309179+0000","last_became_active":"2024-09-10T19:27:52.994757+0000","last_became_peered":"2024-09-10T19:27:52.994757+0000","last_unstale":"2024-09-10T19:33:24.309179+0000","last_undegraded":"2024-09-10T19:33:24.309179+0000","last_fullsized":"2024-09-10T19:33:24.309179+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:33:10.306857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:31:41.824279+0000","last_clean_scrub_stamp":"2024-09-10T19:33:10.306857+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:34:35.755721+0000","scrub_duration":0.0020253770000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6],"acting":[0,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":474,"reported_epoch":460,"state":"active+clean","last_fresh":"2024-09-10T19:33:24.310199+0000","last_change":"2024-09-10T19:32:38.216788+0000","last_active":"2024-09-10T19:33:24.310199+0000","last_peered":"2024-09-10T19:33:24.310199+0000","last_clean":"2024-09-10T19:33:24.310199+0000","last_became_active":"2024-09-10T19:24:23.892621+0000","last_became_peered":"2024-09-10T19:24:23.892621+0000","last_unstale":"2024-09-10T19:33:24.310199+0000","last_undegraded":"2024-09-10T19:33:24.310199+0000","last_fullsized":"2024-09-10T19:33:24.310199+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:32:38.216717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:32:38.216717+0000","last_clean_scrub_stamp":"2024-09-10T19:32:38.216717+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:33:41.435276+0000","scrub_duration":0.002118296,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.19","version":"440'46","reported_seq":616,"reported_epoch":460,"state":"active+clean","last_fresh":"2024-09-10T19:33:23.294353+0000","last_change":"2024-09-10T19:32:05.493764+0000","last_active":"2024-09-10T19:33:23.294353+0000","last_peered":"2024-09-10T19:33:23.294353+0000","last_clean":"2024-09-10T19:33:23.294353+0000","last_became_active":"2024-09-10T19:31:33.338277+0000","last_became_peered":"2024-09-10T19:31:33.338277+0000","last_unstale":"2024-09-10T19:33:23.294353+0000","last_undegraded":"2024-09-10T19:33:23.294353+0000","last_fullsized":"2024-09-10T19:33:23.294353+0000","mapping_epoch":431,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":432,"parent":"0.0","parent_split_bits":0,"last_scrub":"440'46","last_scrub_stamp":"2024-09-10T19:32:05.493712+0000","last_deep_scrub":"440'46","last_deep_scrub_stamp":"2024-09-10T19:32:05.493712+0000","last_clean_scrub_stamp":"2024-09-10T19:32:05.493712+0000","objects_scrubbed":7,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:33:35.039694+0000","scrub_duration":0.0044865089999999996,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":62,"num_read_kb":53,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":559,"reported_epoch":460,"state":"active+clean","last_fresh":"2024-09-10T19:33:23.292566+0000","last_change":"2024-09-10T19:32:08.698175+0000","last_active":"2024-09-10T19:33:23.292566+0000","last_peered":"2024-09-10T19:33:23.292566+0000","last_clean":"2024-09-10T19:33:23.292566+0000","last_became_active":"2024-09-10T19:27:53.006175+0000","last_became_peered":"2024-09-10T19:27:53.006175+0000","last_unstale":"2024-09-10T19:33:23.292566+0000","last_undegraded":"2024-09-10T19:33:23.292566+0000","last_fullsized":"2024-09-10T19:33:23.292566+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:32:08.698092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:19:28.498560+0000","last_clean_scrub_stamp":"2024-09-10T19:32:08.698092+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:33:28.005117+0000","scrub_duration":0.001424399,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"384'2","reported_seq":537,"reported_epoch":461,"state":"active+clean","last_fresh":"2024-09-10T19:33:30.352485+0000","last_change":"2024-09-10T19:33:30.352454+0000","last_active":"2024-09-10T19:33:30.352485+0000","last_peered":"2024-09-10T19:33:30.352485+0000","last_clean":"2024-09-10T19:33:30.352485+0000","last_became_active":"2024-09-10T19:24:23.908614+0000","last_became_peered":"2024-09-10T19:24:23.908614+0000","last_unstale":"2024-09-10T19:33:30.352485+0000","last_undegraded":"2024-09-10T19:33:30.352485+0000","last_fullsized":"2024-09-10T19:33:30.352485+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"384'2","last_scrub_stamp":"2024-09-10T19:33:30.352399+0000","last_deep_scrub":"384'2","last_deep_scrub_stamp":"2024-09-10T19:28:23.240824+0000","last_clean_scrub_stamp":"2024-09-10T19:33:30.352399+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:34:40.810067+0000","scrub_duration":0.00191665,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":874,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"414'25","reported_seq":656,"reported_epoch":460,"state":"active+clean","last_fresh":"2024-09-10T19:33:25.034771+0000","last_change":"2024-09-10T19:33:25.034702+0000","last_active":"2024-09-10T19:33:25.034771+0000","last_peered":"2024-09-10T19:33:25.034771+0000","last_clean":"2024-09-10T19:33:25.034771+0000","last_became_active":"2024-09-10T19:31:47.389313+0000","last_became_peered":"2024-09-10T19:31:47.389313+0000","last_unstale":"2024-09-10T19:33:25.034771+0000","last_undegraded":"2024-09-10T19:33:25.034771+0000","last_fullsized":"2024-09-10T19:33:25.034771+0000","mapping_epoch":435,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":436,"parent":"0.0","parent_split_bits":0,"last_scrub":"414'25","last_scrub_stamp":"2024-09-10T19:33:25.034634+0000","last_deep_scrub":"384'19","last_deep_scrub_stamp":"2024-09-10T19:28:21.799921+0000","last_clean_scrub_stamp":"2024-09-10T19:33:25.034634+0000","objects_scrubbed":4,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:34:32.409274+0000","scrub_duration":0.0021945530000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":550,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,2],"acting":[4,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":109,"reported_epoch":460,"state":"active+clean","last_fresh":"2024-09-10T19:33:24.310941+0000","last_change":"2024-09-10T19:32:51.579358+0000","last_active":"2024-09-10T19:33:24.310941+0000","last_peered":"2024-09-10T19:33:24.310941+0000","last_clean":"2024-09-10T19:33:24.310941+0000","last_became_active":"2024-09-10T19:28:47.151481+0000","last_became_peered":"2024-09-10T19:28:47.151481+0000","last_unstale":"2024-09-10T19:33:24.310941+0000","last_undegraded":"2024-09-10T19:33:24.310941+0000","last_fullsized":"2024-09-10T19:33:24.310941+0000","mapping_epoch":391,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":392,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:32:51.579297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:32:51.579297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:34:19.568803+0000","scrub_duration":0.001353557,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,7],"acting":[6,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":537,"reported_epoch":460,"state":"active+clean","last_fresh":"2024-09-10T19:33:23.292801+0000","last_change":"2024-09-10T19:32:50.757063+0000","last_active":"2024-09-10T19:33:23.292801+0000","last_peered":"2024-09-10T19:33:23.292801+0000","last_clean":"2024-09-10T19:33:23.292801+0000","last_became_active":"2024-09-10T19:24:01.971921+0000","last_became_peered":"2024-09-10T19:24:01.971921+0000","last_unstale":"2024-09-10T19:33:23.292801+0000","last_undegraded":"2024-09-10T19:33:23.292801+0000","last_fullsized":"2024-09-10T19:33:23.292801+0000","mapping_epoch":292,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":293,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:32:50.757000+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:32:50.757000+0000","last_clean_scrub_stamp":"2024-09-10T19:32:50.757000+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:33:54.139512+0000","scrub_duration":0.0013316630000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":538,"reported_epoch":460,"state":"active+clean","last_fresh":"2024-09-10T19:33:23.916722+0000","last_change":"2024-09-10T19:32:42.015279+0000","last_active":"2024-09-10T19:33:23.916722+0000","last_peered":"2024-09-10T19:33:23.916722+0000","last_clean":"2024-09-10T19:33:23.916722+0000","last_became_active":"2024-09-10T19:27:53.011202+0000","last_became_peered":"2024-09-10T19:27:53.011202+0000","last_unstale":"2024-09-10T19:33:23.916722+0000","last_undegraded":"2024-09-10T19:33:23.916722+0000","last_fullsized":"2024-09-10T19:33:23.916722+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:32:42.015213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:32:42.015213+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:33:48.046949+0000","scrub_duration":0.001419172,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,0],"acting":[7,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1b","version":"414'9","reported_seq":582,"reported_epoch":460,"state":"active+clean","last_fresh":"2024-09-10T19:33:23.292951+0000","last_change":"2024-09-10T19:33:07.825011+0000","last_active":"2024-09-10T19:33:23.292951+0000","last_peered":"2024-09-10T19:33:23.292951+0000","last_clean":"2024-09-10T19:33:23.292951+0000","last_became_active":"2024-09-10T19:24:01.974813+0000","last_became_peered":"2024-09-10T19:24:01.974813+0000","last_unstale":"2024-09-10T19:33:23.292951+0000","last_undegraded":"2024-09-10T19:33:23.292951+0000","last_fullsized":"2024-09-10T19:33:23.292951+0000","mapping_epoch":292,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":293,"parent":"0.0","parent_split_bits":0,"last_scrub":"414'9","last_scrub_stamp":"2024-09-10T19:33:07.824943+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:33:07.824943+0000","objects_scrubbed":2,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:34:30.077554+0000","scrub_duration":0.0019824869999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":17,"num_read_kb":13,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":521,"reported_epoch":460,"state":"active+clean","last_fresh":"2024-09-10T19:33:23.915998+0000","last_change":"2024-09-10T19:32:08.177548+0000","last_active":"2024-09-10T19:33:23.915998+0000","last_peered":"2024-09-10T19:33:23.915998+0000","last_clean":"2024-09-10T19:33:23.915998+0000","last_became_active":"2024-09-10T19:24:01.976587+0000","last_became_peered":"2024-09-10T19:24:01.976587+0000","last_unstale":"2024-09-10T19:33:23.915998+0000","last_undegraded":"2024-09-10T19:33:23.915998+0000","last_fullsized":"2024-09-10T19:33:23.915998+0000","mapping_epoch":292,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":293,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:32:08.177472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:32:08.177472+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:33:28.282859+0000","scrub_duration":0.001797674,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":532,"reported_epoch":460,"state":"active+clean","last_fresh":"2024-09-10T19:33:23.468877+0000","last_change":"2024-09-10T19:33:17.587123+0000","last_active":"2024-09-10T19:33:23.468877+0000","last_peered":"2024-09-10T19:33:23.468877+0000","last_clean":"2024-09-10T19:33:23.468877+0000","last_became_active":"2024-09-10T19:24:01.970118+0000","last_became_peered":"2024-09-10T19:24:01.970118+0000","last_unstale":"2024-09-10T19:33:23.468877+0000","last_undegraded":"2024-09-10T19:33:23.468877+0000","last_fullsized":"2024-09-10T19:33:23.468877+0000","mapping_epoch":292,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":293,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:33:17.587025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:31:53.538320+0000","last_clean_scrub_stamp":"2024-09-10T19:33:17.587025+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:34:34.145375+0000","scrub_duration":0.002082534,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":551,"reported_epoch":460,"state":"active+clean","last_fresh":"2024-09-10T19:33:23.916403+0000","last_change":"2024-09-10T19:32:49.927969+0000","last_active":"2024-09-10T19:33:23.916403+0000","last_peered":"2024-09-10T19:33:23.916403+0000","last_clean":"2024-09-10T19:33:23.916403+0000","last_became_active":"2024-09-10T19:31:33.342560+0000","last_became_peered":"2024-09-10T19:31:33.342560+0000","last_unstale":"2024-09-10T19:33:23.916403+0000","last_undegraded":"2024-09-10T19:33:23.916403+0000","last_fullsized":"2024-09-10T19:33:23.916403+0000","mapping_epoch":431,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":432,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:32:49.927910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:32:49.927910+0000","last_clean_scrub_stamp":"2024-09-10T19:32:49.927910+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:33:57.987752+0000","scrub_duration":0.0012803409999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1a","version":"414'21","reported_seq":542,"reported_epoch":460,"state":"active+clean","last_fresh":"2024-09-10T19:33:23.292985+0000","last_change":"2024-09-10T19:31:57.777057+0000","last_active":"2024-09-10T19:33:23.292985+0000","last_peered":"2024-09-10T19:33:23.292985+0000","last_clean":"2024-09-10T19:33:23.292985+0000","last_became_active":"2024-09-10T19:24:23.891213+0000","last_became_peered":"2024-09-10T19:24:23.891213+0000","last_unstale":"2024-09-10T19:33:23.292985+0000","last_undegraded":"2024-09-10T19:33:23.292985+0000","last_fullsized":"2024-09-10T19:33:23.292985+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"414'21","last_scrub_stamp":"2024-09-10T19:31:57.776990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:31:57.776990+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0020528489999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":524,"reported_epoch":460,"state":"active+clean","last_fresh":"2024-09-10T19:33:23.469032+0000","last_change":"2024-09-10T19:32:57.594733+0000","last_active":"2024-09-10T19:33:23.469032+0000","last_peered":"2024-09-10T19:33:23.469032+0000","last_clean":"2024-09-10T19:33:23.469032+0000","last_became_active":"2024-09-10T19:24:23.907653+0000","last_became_peered":"2024-09-10T19:24:23.907653+0000","last_unstale":"2024-09-10T19:33:23.469032+0000","last_undegraded":"2024-09-10T19:33:23.469032+0000","last_fullsized":"2024-09-10T19:33:23.469032+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:32:57.594665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:32:57.594665+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:34:27.488641+0000","scrub_duration":0.0014914939999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":487,"reported_epoch":460,"state":"active+clean","last_fresh":"2024-09-10T19:33:24.309011+0000","last_change":"2024-09-10T19:32:13.105849+0000","last_active":"2024-09-10T19:33:24.309011+0000","last_peered":"2024-09-10T19:33:24.309011+0000","last_clean":"2024-09-10T19:33:24.309011+0000","last_became_active":"2024-09-10T19:24:24.907431+0000","last_became_peered":"2024-09-10T19:24:24.907431+0000","last_unstale":"2024-09-10T19:33:24.309011+0000","last_undegraded":"2024-09-10T19:33:24.309011+0000","last_fullsized":"2024-09-10T19:33:24.309011+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:32:13.105794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:30:41.905516+0000","last_clean_scrub_stamp":"2024-09-10T19:32:13.105794+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:33:42.643961+0000","scrub_duration":0.0027163510000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.8","version":"439'80","reported_seq":334,"reported_epoch":460,"state":"active+clean","last_fresh":"2024-09-10T19:33:23.468668+0000","last_change":"2024-09-10T19:32:00.542682+0000","last_active":"2024-09-10T19:33:23.468668+0000","last_peered":"2024-09-10T19:33:23.468668+0000","last_clean":"2024-09-10T19:33:23.468668+0000","last_became_active":"2024-09-10T19:31:33.348819+0000","last_became_peered":"2024-09-10T19:31:33.348819+0000","last_unstale":"2024-09-10T19:33:23.468668+0000","last_undegraded":"2024-09-10T19:33:23.468668+0000","last_fullsized":"2024-09-10T19:33:23.468668+0000","mapping_epoch":431,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":432,"parent":"0.0","parent_split_bits":4,"last_scrub":"439'80","last_scrub_stamp":"2024-09-10T19:32:00.542611+0000","last_deep_scrub":"439'80","last_deep_scrub_stamp":"2024-09-10T19:32:00.542611+0000","last_clean_scrub_stamp":"2024-09-10T19:32:00.542611+0000","objects_scrubbed":1,"log_size":80,"log_dups_size":0,"ondisk_log_size":80,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0044182830000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":788064,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.d","version":"437'53","reported_seq":768,"reported_epoch":460,"state":"active+clean","last_fresh":"2024-09-10T19:33:26.076599+0000","last_change":"2024-09-10T19:33:26.076537+0000","last_active":"2024-09-10T19:33:26.076599+0000","last_peered":"2024-09-10T19:33:26.076599+0000","last_clean":"2024-09-10T19:33:26.076599+0000","last_became_active":"2024-09-10T19:31:33.349405+0000","last_became_peered":"2024-09-10T19:31:33.349405+0000","last_unstale":"2024-09-10T19:33:26.076599+0000","last_undegraded":"2024-09-10T19:33:26.076599+0000","last_fullsized":"2024-09-10T19:33:26.076599+0000","mapping_epoch":431,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":432,"parent":"0.0","parent_split_bits":0,"last_scrub":"437'53","last_scrub_stamp":"2024-09-10T19:33:26.076467+0000","last_deep_scrub":"437'53","last_deep_scrub_stamp":"2024-09-10T19:31:51.712146+0000","last_clean_scrub_stamp":"2024-09-10T19:33:26.076467+0000","objects_scrubbed":8,"log_size":53,"log_dups_size":0,"ondisk_log_size":53,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:34:43.827055+0000","scrub_duration":0.002194232,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":71,"num_read_kb":61,"num_write":46,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":48,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":511,"reported_epoch":460,"state":"active+clean","last_fresh":"2024-09-10T19:33:23.916580+0000","last_change":"2024-09-10T19:32:20.013848+0000","last_active":"2024-09-10T19:33:23.916580+0000","last_peered":"2024-09-10T19:33:23.916580+0000","last_clean":"2024-09-10T19:33:23.916580+0000","last_became_active":"2024-09-10T19:10:07.691119+0000","last_became_peered":"2024-09-10T19:10:07.691119+0000","last_unstale":"2024-09-10T19:33:23.916580+0000","last_undegraded":"2024-09-10T19:33:23.916580+0000","last_fullsized":"2024-09-10T19:33:23.916580+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:32:20.013777+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:16:39.716674+0000","last_clean_scrub_stamp":"2024-09-10T19:32:20.013777+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:33:43.355910+0000","scrub_duration":0.001598063,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":242,"reported_epoch":460,"state":"active+clean","last_fresh":"2024-09-10T19:33:24.312110+0000","last_change":"2024-09-10T19:32:32.353573+0000","last_active":"2024-09-10T19:33:24.312110+0000","last_peered":"2024-09-10T19:33:24.312110+0000","last_clean":"2024-09-10T19:33:24.312110+0000","last_became_active":"2024-09-10T19:24:24.886479+0000","last_became_peered":"2024-09-10T19:24:24.886479+0000","last_unstale":"2024-09-10T19:33:24.312110+0000","last_undegraded":"2024-09-10T19:33:24.312110+0000","last_fullsized":"2024-09-10T19:33:24.312110+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:32:32.353470+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:32:32.353470+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:33:52.060075+0000","scrub_duration":0.0019860799999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":49,"reported_epoch":460,"state":"active+clean","last_fresh":"2024-09-10T19:33:23.294013+0000","last_change":"2024-09-10T19:32:36.635397+0000","last_active":"2024-09-10T19:33:23.294013+0000","last_peered":"2024-09-10T19:33:23.294013+0000","last_clean":"2024-09-10T19:33:23.294013+0000","last_became_active":"2024-09-10T19:31:33.345895+0000","last_became_peered":"2024-09-10T19:31:33.345895+0000","last_unstale":"2024-09-10T19:33:23.294013+0000","last_undegraded":"2024-09-10T19:33:23.294013+0000","last_fullsized":"2024-09-10T19:33:23.294013+0000","mapping_epoch":431,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":432,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:32:36.635341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:32:36.635341+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:34:06.621717+0000","scrub_duration":0.0013187209999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.9","version":"74'79","reported_seq":586,"reported_epoch":460,"state":"active+clean","last_fresh":"2024-09-10T19:33:23.916059+0000","last_change":"2024-09-10T19:32:22.957091+0000","last_active":"2024-09-10T19:33:23.916059+0000","last_peered":"2024-09-10T19:33:23.916059+0000","last_clean":"2024-09-10T19:33:23.916059+0000","last_became_active":"2024-09-10T19:31:33.337526+0000","last_became_peered":"2024-09-10T19:31:33.337526+0000","last_unstale":"2024-09-10T19:33:23.916059+0000","last_undegraded":"2024-09-10T19:33:23.916059+0000","last_fullsized":"2024-09-10T19:33:23.916059+0000","mapping_epoch":431,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":432,"parent":"0.0","parent_split_bits":4,"last_scrub":"74'79","last_scrub_stamp":"2024-09-10T19:32:22.957027+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:08:00.699315+0000","last_clean_scrub_stamp":"2024-09-10T19:32:22.957027+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:33:35.149675+0000","scrub_duration":0.0018635259999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.c","version":"414'30","reported_seq":610,"reported_epoch":460,"state":"active+clean","last_fresh":"2024-09-10T19:33:23.293172+0000","last_change":"2024-09-10T19:33:13.849981+0000","last_active":"2024-09-10T19:33:23.293172+0000","last_peered":"2024-09-10T19:33:23.293172+0000","last_clean":"2024-09-10T19:33:23.293172+0000","last_became_active":"2024-09-10T19:27:53.012617+0000","last_became_peered":"2024-09-10T19:27:53.012617+0000","last_unstale":"2024-09-10T19:33:23.293172+0000","last_undegraded":"2024-09-10T19:33:23.293172+0000","last_fullsized":"2024-09-10T19:33:23.293172+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"414'30","last_scrub_stamp":"2024-09-10T19:33:13.849914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:33:13.849914+0000","objects_scrubbed":5,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:34:28.542502+0000","scrub_duration":0.001756968,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":45,"num_read_kb":40,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":218,"reported_epoch":460,"state":"active+clean","last_fresh":"2024-09-10T19:33:24.312070+0000","last_change":"2024-09-10T19:32:16.387044+0000","last_active":"2024-09-10T19:33:24.312070+0000","last_peered":"2024-09-10T19:33:24.312070+0000","last_clean":"2024-09-10T19:33:24.312070+0000","last_became_active":"2024-09-10T19:24:01.973848+0000","last_became_peered":"2024-09-10T19:24:01.973848+0000","last_unstale":"2024-09-10T19:33:24.312070+0000","last_undegraded":"2024-09-10T19:33:24.312070+0000","last_fullsized":"2024-09-10T19:33:24.312070+0000","mapping_epoch":292,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":293,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:32:16.386988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:32:16.386988+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:33:26.421807+0000","scrub_duration":0.0019624949999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,7],"acting":[1,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.d","version":"432'13","reported_seq":76,"reported_epoch":460,"state":"active+clean","last_fresh":"2024-09-10T19:33:26.004493+0000","last_change":"2024-09-10T19:32:41.500567+0000","last_active":"2024-09-10T19:33:26.004493+0000","last_peered":"2024-09-10T19:33:26.004493+0000","last_clean":"2024-09-10T19:33:26.004493+0000","last_became_active":"2024-09-10T19:31:33.347510+0000","last_became_peered":"2024-09-10T19:31:33.347510+0000","last_unstale":"2024-09-10T19:33:26.004493+0000","last_undegraded":"2024-09-10T19:33:26.004493+0000","last_fullsized":"2024-09-10T19:33:26.004493+0000","mapping_epoch":431,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":432,"parent":"0.0","parent_split_bits":0,"last_scrub":"432'13","last_scrub_stamp":"2024-09-10T19:32:41.500514+0000","last_deep_scrub":"303'12","last_deep_scrub_stamp":"2024-09-10T19:31:20.099105+0000","last_clean_scrub_stamp":"2024-09-10T19:32:41.500514+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:34:02.582845+0000","scrub_duration":0.0024028790000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":557,"reported_epoch":460,"state":"active+clean","last_fresh":"2024-09-10T19:33:23.293298+0000","last_change":"2024-09-10T19:33:21.939966+0000","last_active":"2024-09-10T19:33:23.293298+0000","last_peered":"2024-09-10T19:33:23.293298+0000","last_clean":"2024-09-10T19:33:23.293298+0000","last_became_active":"2024-09-10T19:31:33.342272+0000","last_became_peered":"2024-09-10T19:31:33.342272+0000","last_unstale":"2024-09-10T19:33:23.293298+0000","last_undegraded":"2024-09-10T19:33:23.293298+0000","last_fullsized":"2024-09-10T19:33:23.293298+0000","mapping_epoch":431,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":432,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:33:21.939898+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:33:21.939898+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:34:51.337037+0000","scrub_duration":0.0016606660000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.a","version":"74'79","reported_seq":316,"reported_epoch":460,"state":"active+clean","last_fresh":"2024-09-10T19:33:23.468848+0000","last_change":"2024-09-10T19:32:54.538022+0000","last_active":"2024-09-10T19:33:23.468848+0000","last_peered":"2024-09-10T19:33:23.468848+0000","last_clean":"2024-09-10T19:33:23.468848+0000","last_became_active":"2024-09-10T19:31:33.349140+0000","last_became_peered":"2024-09-10T19:31:33.349140+0000","last_unstale":"2024-09-10T19:33:23.468848+0000","last_undegraded":"2024-09-10T19:33:23.468848+0000","last_fullsized":"2024-09-10T19:33:23.468848+0000","mapping_epoch":431,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":432,"parent":"0.0","parent_split_bits":4,"last_scrub":"74'79","last_scrub_stamp":"2024-09-10T19:32:54.537950+0000","last_deep_scrub":"74'79","last_deep_scrub_stamp":"2024-09-10T19:21:53.326433+0000","last_clean_scrub_stamp":"2024-09-10T19:32:54.537950+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:34:17.449988+0000","scrub_duration":0.0018094540000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.f","version":"414'39","reported_seq":252,"reported_epoch":460,"state":"active+clean","last_fresh":"2024-09-10T19:33:24.312249+0000","last_change":"2024-09-10T19:32:20.405218+0000","last_active":"2024-09-10T19:33:24.312249+0000","last_peered":"2024-09-10T19:33:24.312249+0000","last_clean":"2024-09-10T19:33:24.312249+0000","last_became_active":"2024-09-10T19:24:01.979187+0000","last_became_peered":"2024-09-10T19:24:01.979187+0000","last_unstale":"2024-09-10T19:33:24.312249+0000","last_undegraded":"2024-09-10T19:33:24.312249+0000","last_fullsized":"2024-09-10T19:33:24.312249+0000","mapping_epoch":292,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":293,"parent":"0.0","parent_split_bits":0,"last_scrub":"414'39","last_scrub_stamp":"2024-09-10T19:32:20.405155+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:32:20.405155+0000","objects_scrubbed":7,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:33:34.076742+0000","scrub_duration":0.0020631650000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":61,"num_read_kb":52,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":218,"reported_epoch":460,"state":"active+clean","last_fresh":"2024-09-10T19:33:24.311854+0000","last_change":"2024-09-10T19:32:35.290354+0000","last_active":"2024-09-10T19:33:24.311854+0000","last_peered":"2024-09-10T19:33:24.311854+0000","last_clean":"2024-09-10T19:33:24.311854+0000","last_became_active":"2024-09-10T19:24:01.974773+0000","last_became_peered":"2024-09-10T19:24:01.974773+0000","last_unstale":"2024-09-10T19:33:24.311854+0000","last_undegraded":"2024-09-10T19:33:24.311854+0000","last_fullsized":"2024-09-10T19:33:24.311854+0000","mapping_epoch":292,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":293,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:32:35.290295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:28:24.420270+0000","last_clean_scrub_stamp":"2024-09-10T19:32:35.290295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:33:48.825101+0000","scrub_duration":0.0017394489999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.e","version":"433'11","reported_seq":797,"reported_epoch":460,"state":"active+clean","last_fresh":"2024-09-10T19:33:26.004866+0000","last_change":"2024-09-10T19:33:04.856855+0000","last_active":"2024-09-10T19:33:26.004866+0000","last_peered":"2024-09-10T19:33:26.004866+0000","last_clean":"2024-09-10T19:33:26.004866+0000","last_became_active":"2024-09-10T19:24:23.908760+0000","last_became_peered":"2024-09-10T19:24:23.908760+0000","last_unstale":"2024-09-10T19:33:26.004866+0000","last_undegraded":"2024-09-10T19:33:26.004866+0000","last_fullsized":"2024-09-10T19:33:26.004866+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"433'11","last_scrub_stamp":"2024-09-10T19:33:04.856780+0000","last_deep_scrub":"433'11","last_deep_scrub_stamp":"2024-09-10T19:31:44.714059+0000","last_clean_scrub_stamp":"2024-09-10T19:33:04.856780+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:34:26.622245+0000","scrub_duration":0.0019989629999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":543,"reported_epoch":460,"state":"active+clean","last_fresh":"2024-09-10T19:33:23.469299+0000","last_change":"2024-09-10T19:32:15.466055+0000","last_active":"2024-09-10T19:33:23.469299+0000","last_peered":"2024-09-10T19:33:23.469299+0000","last_clean":"2024-09-10T19:33:23.469299+0000","last_became_active":"2024-09-10T19:24:24.917109+0000","last_became_peered":"2024-09-10T19:24:24.917109+0000","last_unstale":"2024-09-10T19:33:23.469299+0000","last_undegraded":"2024-09-10T19:33:23.469299+0000","last_fullsized":"2024-09-10T19:33:23.469299+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:32:15.465990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:32:15.465990+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:33:40.503288+0000","scrub_duration":0.001701709,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.e","version":"414'32","reported_seq":676,"reported_epoch":460,"state":"active+clean","last_fresh":"2024-09-10T19:33:23.293451+0000","last_change":"2024-09-10T19:33:06.855729+0000","last_active":"2024-09-10T19:33:23.293451+0000","last_peered":"2024-09-10T19:33:23.293451+0000","last_clean":"2024-09-10T19:33:23.293451+0000","last_became_active":"2024-09-10T19:27:53.012101+0000","last_became_peered":"2024-09-10T19:27:53.012101+0000","last_unstale":"2024-09-10T19:33:23.293451+0000","last_undegraded":"2024-09-10T19:33:23.293451+0000","last_fullsized":"2024-09-10T19:33:23.293451+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"414'32","last_scrub_stamp":"2024-09-10T19:33:06.855661+0000","last_deep_scrub":"84'16","last_deep_scrub_stamp":"2024-09-10T19:11:56.157975+0000","last_clean_scrub_stamp":"2024-09-10T19:33:06.855661+0000","objects_scrubbed":5,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:34:33.242588+0000","scrub_duration":0.0020371830000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":37,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":550,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":192,"reported_epoch":460,"state":"active+clean","last_fresh":"2024-09-10T19:33:23.292443+0000","last_change":"2024-09-10T19:32:44.842610+0000","last_active":"2024-09-10T19:33:23.292443+0000","last_peered":"2024-09-10T19:33:23.292443+0000","last_clean":"2024-09-10T19:33:23.292443+0000","last_became_active":"2024-09-10T19:24:53.756220+0000","last_became_peered":"2024-09-10T19:24:53.756220+0000","last_unstale":"2024-09-10T19:33:23.292443+0000","last_undegraded":"2024-09-10T19:33:23.292443+0000","last_fullsized":"2024-09-10T19:33:23.292443+0000","mapping_epoch":318,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":319,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:32:44.842540+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:31:21.864910+0000","last_clean_scrub_stamp":"2024-09-10T19:32:44.842540+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:34:07.852289+0000","scrub_duration":0.001384515,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,3],"acting":[4,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":517,"reported_epoch":460,"state":"active+clean","last_fresh":"2024-09-10T19:33:23.469649+0000","last_change":"2024-09-10T19:33:13.545919+0000","last_active":"2024-09-10T19:33:23.469649+0000","last_peered":"2024-09-10T19:33:23.469649+0000","last_clean":"2024-09-10T19:33:23.469649+0000","last_became_active":"2024-09-10T19:27:53.007094+0000","last_became_peered":"2024-09-10T19:27:53.007094+0000","last_unstale":"2024-09-10T19:33:23.469649+0000","last_undegraded":"2024-09-10T19:33:23.469649+0000","last_fullsized":"2024-09-10T19:33:23.469649+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:33:13.545851+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:33:13.545851+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:34:17.009934+0000","scrub_duration":0.0014377470000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":505,"reported_epoch":460,"state":"active+clean","last_fresh":"2024-09-10T19:33:23.294230+0000","last_change":"2024-09-10T19:32:22.285635+0000","last_active":"2024-09-10T19:33:23.294230+0000","last_peered":"2024-09-10T19:33:23.294230+0000","last_clean":"2024-09-10T19:33:23.294230+0000","last_became_active":"2024-09-10T19:27:53.025345+0000","last_became_peered":"2024-09-10T19:27:53.025345+0000","last_unstale":"2024-09-10T19:33:23.294230+0000","last_undegraded":"2024-09-10T19:33:23.294230+0000","last_fullsized":"2024-09-10T19:33:23.294230+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:32:22.285577+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:32:22.285577+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:33:29.669292+0000","scrub_duration":0.0021926939999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.4","version":"317'80","reported_seq":292,"reported_epoch":460,"state":"active+clean","last_fresh":"2024-09-10T19:33:23.293012+0000","last_change":"2024-09-10T19:32:28.216883+0000","last_active":"2024-09-10T19:33:23.293012+0000","last_peered":"2024-09-10T19:33:23.293012+0000","last_clean":"2024-09-10T19:33:23.293012+0000","last_became_active":"2024-09-10T19:20:30.838441+0000","last_became_peered":"2024-09-10T19:20:30.838441+0000","last_unstale":"2024-09-10T19:33:23.293012+0000","last_undegraded":"2024-09-10T19:33:23.293012+0000","last_fullsized":"2024-09-10T19:33:23.293012+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":3,"last_scrub":"317'80","last_scrub_stamp":"2024-09-10T19:32:28.216823+0000","last_deep_scrub":"317'80","last_deep_scrub_stamp":"2024-09-10T19:24:51.540456+0000","last_clean_scrub_stamp":"2024-09-10T19:32:28.216823+0000","objects_scrubbed":1,"log_size":80,"log_dups_size":0,"ondisk_log_size":80,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:33:30.767417+0000","scrub_duration":0.0018724830000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"414'49","reported_seq":516,"reported_epoch":460,"state":"active+clean","last_fresh":"2024-09-10T19:33:23.292539+0000","last_change":"2024-09-10T19:33:08.800809+0000","last_active":"2024-09-10T19:33:23.292539+0000","last_peered":"2024-09-10T19:33:23.292539+0000","last_clean":"2024-09-10T19:33:23.292539+0000","last_became_active":"2024-09-10T19:27:53.011799+0000","last_became_peered":"2024-09-10T19:27:53.011799+0000","last_unstale":"2024-09-10T19:33:23.292539+0000","last_undegraded":"2024-09-10T19:33:23.292539+0000","last_fullsized":"2024-09-10T19:33:23.292539+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"414'49","last_scrub_stamp":"2024-09-10T19:33:08.800721+0000","last_deep_scrub":"414'49","last_deep_scrub_stamp":"2024-09-10T19:33:08.800721+0000","last_clean_scrub_stamp":"2024-09-10T19:33:08.800721+0000","objects_scrubbed":7,"log_size":49,"log_dups_size":0,"ondisk_log_size":49,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:34:11.843623+0000","scrub_duration":0.0024563380000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":63,"num_read_kb":56,"num_write":42,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":136,"reported_epoch":460,"state":"active+clean","last_fresh":"2024-09-10T19:33:24.311842+0000","last_change":"2024-09-10T19:32:20.731636+0000","last_active":"2024-09-10T19:33:24.311842+0000","last_peered":"2024-09-10T19:33:24.311842+0000","last_clean":"2024-09-10T19:33:24.311842+0000","last_became_active":"2024-09-10T19:31:33.338027+0000","last_became_peered":"2024-09-10T19:31:33.338027+0000","last_unstale":"2024-09-10T19:33:24.311842+0000","last_undegraded":"2024-09-10T19:33:24.311842+0000","last_fullsized":"2024-09-10T19:33:24.311842+0000","mapping_epoch":431,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":432,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:32:20.731566+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:32:20.731566+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0013631559999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.0","version":"417'13","reported_seq":820,"reported_epoch":460,"state":"active+clean","last_fresh":"2024-09-10T19:33:26.004480+0000","last_change":"2024-09-10T19:32:24.300782+0000","last_active":"2024-09-10T19:33:26.004480+0000","last_peered":"2024-09-10T19:33:26.004480+0000","last_clean":"2024-09-10T19:33:26.004480+0000","last_became_active":"2024-09-10T19:24:01.972645+0000","last_became_peered":"2024-09-10T19:24:01.972645+0000","last_unstale":"2024-09-10T19:33:26.004480+0000","last_undegraded":"2024-09-10T19:33:26.004480+0000","last_fullsized":"2024-09-10T19:33:26.004480+0000","mapping_epoch":292,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":293,"parent":"0.0","parent_split_bits":0,"last_scrub":"417'13","last_scrub_stamp":"2024-09-10T19:32:24.300723+0000","last_deep_scrub":"417'13","last_deep_scrub_stamp":"2024-09-10T19:30:39.964914+0000","last_clean_scrub_stamp":"2024-09-10T19:32:24.300723+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:33:29.321991+0000","scrub_duration":0.0017016399999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":552,"reported_epoch":460,"state":"active+clean","last_fresh":"2024-09-10T19:33:23.916206+0000","last_change":"2024-09-10T19:33:22.883103+0000","last_active":"2024-09-10T19:33:23.916206+0000","last_peered":"2024-09-10T19:33:23.916206+0000","last_clean":"2024-09-10T19:33:23.916206+0000","last_became_active":"2024-09-10T19:31:33.338017+0000","last_became_peered":"2024-09-10T19:31:33.338017+0000","last_unstale":"2024-09-10T19:33:23.916206+0000","last_undegraded":"2024-09-10T19:33:23.916206+0000","last_fullsized":"2024-09-10T19:33:23.916206+0000","mapping_epoch":431,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":432,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:33:22.883036+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:28:25.822918+0000","last_clean_scrub_stamp":"2024-09-10T19:33:22.883036+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:34:31.415810+0000","scrub_duration":0.0014541980000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.5","version":"74'79","reported_seq":326,"reported_epoch":460,"state":"active+clean","last_fresh":"2024-09-10T19:33:23.293203+0000","last_change":"2024-09-10T19:32:57.763971+0000","last_active":"2024-09-10T19:33:23.293203+0000","last_peered":"2024-09-10T19:33:23.293203+0000","last_clean":"2024-09-10T19:33:23.293203+0000","last_became_active":"2024-09-10T19:31:33.339260+0000","last_became_peered":"2024-09-10T19:31:33.339260+0000","last_unstale":"2024-09-10T19:33:23.293203+0000","last_undegraded":"2024-09-10T19:33:23.293203+0000","last_fullsized":"2024-09-10T19:33:23.293203+0000","mapping_epoch":431,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":432,"parent":"0.0","parent_split_bits":3,"last_scrub":"74'79","last_scrub_stamp":"2024-09-10T19:32:57.763901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:08:00.699315+0000","last_clean_scrub_stamp":"2024-09-10T19:32:57.763901+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:34:26.852048+0000","scrub_duration":0.001528176,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"436'50","reported_seq":81,"reported_epoch":460,"state":"active+clean","last_fresh":"2024-09-10T19:33:23.292608+0000","last_change":"2024-09-10T19:33:01.502333+0000","last_active":"2024-09-10T19:33:23.292608+0000","last_peered":"2024-09-10T19:33:23.292608+0000","last_clean":"2024-09-10T19:33:23.292608+0000","last_became_active":"2024-09-10T19:31:47.391918+0000","last_became_peered":"2024-09-10T19:31:47.391918+0000","last_unstale":"2024-09-10T19:33:23.292608+0000","last_undegraded":"2024-09-10T19:33:23.292608+0000","last_fullsized":"2024-09-10T19:33:23.292608+0000","mapping_epoch":435,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":436,"parent":"0.0","parent_split_bits":0,"last_scrub":"436'50","last_scrub_stamp":"2024-09-10T19:33:01.502277+0000","last_deep_scrub":"436'50","last_deep_scrub_stamp":"2024-09-10T19:31:47.660982+0000","last_clean_scrub_stamp":"2024-09-10T19:33:01.502277+0000","objects_scrubbed":8,"log_size":50,"log_dups_size":0,"ondisk_log_size":50,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:34:15.160825+0000","scrub_duration":0.0019199830000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":69,"num_read_kb":57,"num_write":44,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,0],"acting":[2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":248,"reported_epoch":460,"state":"active+clean","last_fresh":"2024-09-10T19:33:24.312250+0000","last_change":"2024-09-10T19:33:08.232525+0000","last_active":"2024-09-10T19:33:24.312250+0000","last_peered":"2024-09-10T19:33:24.312250+0000","last_clean":"2024-09-10T19:33:24.312250+0000","last_became_active":"2024-09-10T19:27:53.008302+0000","last_became_peered":"2024-09-10T19:27:53.008302+0000","last_unstale":"2024-09-10T19:33:24.312250+0000","last_undegraded":"2024-09-10T19:33:24.312250+0000","last_fullsized":"2024-09-10T19:33:24.312250+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:33:08.232440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:31:58.291225+0000","last_clean_scrub_stamp":"2024-09-10T19:33:08.232440+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:34:38.130785+0000","scrub_duration":0.0016740069999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":506,"reported_epoch":460,"state":"active+clean","last_fresh":"2024-09-10T19:33:23.292662+0000","last_change":"2024-09-10T19:32:22.780163+0000","last_active":"2024-09-10T19:33:23.292662+0000","last_peered":"2024-09-10T19:33:23.292662+0000","last_clean":"2024-09-10T19:33:23.292662+0000","last_became_active":"2024-09-10T19:10:11.698054+0000","last_became_peered":"2024-09-10T19:10:11.698054+0000","last_unstale":"2024-09-10T19:33:23.292662+0000","last_undegraded":"2024-09-10T19:33:23.292662+0000","last_fullsized":"2024-09-10T19:33:23.292662+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:32:22.780098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:26:24.897519+0000","last_clean_scrub_stamp":"2024-09-10T19:32:22.780098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0013754450000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":544,"reported_epoch":460,"state":"active+clean","last_fresh":"2024-09-10T19:33:23.292953+0000","last_change":"2024-09-10T19:32:12.780125+0000","last_active":"2024-09-10T19:33:23.292953+0000","last_peered":"2024-09-10T19:33:23.292953+0000","last_clean":"2024-09-10T19:33:23.292953+0000","last_became_active":"2024-09-10T19:31:33.337265+0000","last_became_peered":"2024-09-10T19:31:33.337265+0000","last_unstale":"2024-09-10T19:33:23.292953+0000","last_undegraded":"2024-09-10T19:33:23.292953+0000","last_fullsized":"2024-09-10T19:33:23.292953+0000","mapping_epoch":431,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":432,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:32:12.780057+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:32:12.780057+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0014965810000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.6","version":"74'79","reported_seq":136,"reported_epoch":460,"state":"active+clean","last_fresh":"2024-09-10T19:33:24.311689+0000","last_change":"2024-09-10T19:33:01.489808+0000","last_active":"2024-09-10T19:33:24.311689+0000","last_peered":"2024-09-10T19:33:24.311689+0000","last_clean":"2024-09-10T19:33:24.311689+0000","last_became_active":"2024-09-10T19:27:53.017899+0000","last_became_peered":"2024-09-10T19:27:53.017899+0000","last_unstale":"2024-09-10T19:33:24.311689+0000","last_undegraded":"2024-09-10T19:33:24.311689+0000","last_fullsized":"2024-09-10T19:33:24.311689+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":3,"last_scrub":"74'79","last_scrub_stamp":"2024-09-10T19:33:01.489718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:08:00.699315+0000","last_clean_scrub_stamp":"2024-09-10T19:33:01.489718+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:34:27.307483+0000","scrub_duration":0.001384648,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"414'47","reported_seq":531,"reported_epoch":460,"state":"active+clean","last_fresh":"2024-09-10T19:33:24.309559+0000","last_change":"2024-09-10T19:33:01.332727+0000","last_active":"2024-09-10T19:33:24.309559+0000","last_peered":"2024-09-10T19:33:24.309559+0000","last_clean":"2024-09-10T19:33:24.309559+0000","last_became_active":"2024-09-10T19:24:23.901836+0000","last_became_peered":"2024-09-10T19:24:23.901836+0000","last_unstale":"2024-09-10T19:33:24.309559+0000","last_undegraded":"2024-09-10T19:33:24.309559+0000","last_fullsized":"2024-09-10T19:33:24.309559+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"414'47","last_scrub_stamp":"2024-09-10T19:33:01.332621+0000","last_deep_scrub":"104'27","last_deep_scrub_stamp":"2024-09-10T19:13:02.225944+0000","last_clean_scrub_stamp":"2024-09-10T19:33:01.332621+0000","objects_scrubbed":8,"log_size":47,"log_dups_size":0,"ondisk_log_size":47,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:34:10.981100+0000","scrub_duration":0.0023253050000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":68,"num_read_kb":54,"num_write":42,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":528,"reported_epoch":460,"state":"active+clean","last_fresh":"2024-09-10T19:33:23.916801+0000","last_change":"2024-09-10T19:33:11.812623+0000","last_active":"2024-09-10T19:33:23.916801+0000","last_peered":"2024-09-10T19:33:23.916801+0000","last_clean":"2024-09-10T19:33:23.916801+0000","last_became_active":"2024-09-10T19:24:23.899556+0000","last_became_peered":"2024-09-10T19:24:23.899556+0000","last_unstale":"2024-09-10T19:33:23.916801+0000","last_undegraded":"2024-09-10T19:33:23.916801+0000","last_fullsized":"2024-09-10T19:33:23.916801+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:33:11.812552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:16:37.779967+0000","last_clean_scrub_stamp":"2024-09-10T19:33:11.812552+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:34:41.688836+0000","scrub_duration":0.0016932640000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":136,"reported_epoch":460,"state":"active+clean","last_fresh":"2024-09-10T19:33:24.345748+0000","last_change":"2024-09-10T19:33:24.345706+0000","last_active":"2024-09-10T19:33:24.345748+0000","last_peered":"2024-09-10T19:33:24.345748+0000","last_clean":"2024-09-10T19:33:24.345748+0000","last_became_active":"2024-09-10T19:27:53.003734+0000","last_became_peered":"2024-09-10T19:27:53.003734+0000","last_unstale":"2024-09-10T19:33:24.345748+0000","last_undegraded":"2024-09-10T19:33:24.345748+0000","last_fullsized":"2024-09-10T19:33:24.345748+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:33:24.345648+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:33:24.345648+0000","last_clean_scrub_stamp":"2024-09-10T19:33:24.345648+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:34:51.577248+0000","scrub_duration":0.001489612,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":258,"reported_epoch":460,"state":"active+clean","last_fresh":"2024-09-10T19:33:24.311837+0000","last_change":"2024-09-10T19:33:04.243676+0000","last_active":"2024-09-10T19:33:24.311837+0000","last_peered":"2024-09-10T19:33:24.311837+0000","last_clean":"2024-09-10T19:33:24.311837+0000","last_became_active":"2024-09-10T19:31:33.342013+0000","last_became_peered":"2024-09-10T19:31:33.342013+0000","last_unstale":"2024-09-10T19:33:24.311837+0000","last_undegraded":"2024-09-10T19:33:24.311837+0000","last_fullsized":"2024-09-10T19:33:24.311837+0000","mapping_epoch":431,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":432,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:33:04.243576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:31:42.343083+0000","last_clean_scrub_stamp":"2024-09-10T19:33:04.243576+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:34:33.905888+0000","scrub_duration":0.0014906159999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.7","version":"74'79","reported_seq":136,"reported_epoch":460,"state":"active+clean","last_fresh":"2024-09-10T19:33:24.311792+0000","last_change":"2024-09-10T19:33:17.380952+0000","last_active":"2024-09-10T19:33:24.311792+0000","last_peered":"2024-09-10T19:33:24.311792+0000","last_clean":"2024-09-10T19:33:24.311792+0000","last_became_active":"2024-09-10T19:27:53.020085+0000","last_became_peered":"2024-09-10T19:27:53.020085+0000","last_unstale":"2024-09-10T19:33:24.311792+0000","last_undegraded":"2024-09-10T19:33:24.311792+0000","last_fullsized":"2024-09-10T19:33:24.311792+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":3,"last_scrub":"74'79","last_scrub_stamp":"2024-09-10T19:33:17.380886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:08:00.699315+0000","last_clean_scrub_stamp":"2024-09-10T19:33:17.380886+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:34:33.357173+0000","scrub_duration":0.0014656389999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.2","version":"414'35","reported_seq":260,"reported_epoch":460,"state":"active+clean","last_fresh":"2024-09-10T19:33:24.312166+0000","last_change":"2024-09-10T19:32:46.261518+0000","last_active":"2024-09-10T19:33:24.312166+0000","last_peered":"2024-09-10T19:33:24.312166+0000","last_clean":"2024-09-10T19:33:24.312166+0000","last_became_active":"2024-09-10T19:24:01.976841+0000","last_became_peered":"2024-09-10T19:24:01.976841+0000","last_unstale":"2024-09-10T19:33:24.312166+0000","last_undegraded":"2024-09-10T19:33:24.312166+0000","last_fullsized":"2024-09-10T19:33:24.312166+0000","mapping_epoch":292,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":293,"parent":"0.0","parent_split_bits":0,"last_scrub":"414'35","last_scrub_stamp":"2024-09-10T19:32:46.261462+0000","last_deep_scrub":"414'35","last_deep_scrub_stamp":"2024-09-10T19:31:20.479343+0000","last_clean_scrub_stamp":"2024-09-10T19:32:46.261462+0000","objects_scrubbed":5,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:34:02.032280+0000","scrub_duration":0.0021662119999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":38,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":242,"reported_epoch":460,"state":"active+clean","last_fresh":"2024-09-10T19:33:24.312111+0000","last_change":"2024-09-10T19:33:06.213005+0000","last_active":"2024-09-10T19:33:24.312111+0000","last_peered":"2024-09-10T19:33:24.312111+0000","last_clean":"2024-09-10T19:33:24.312111+0000","last_became_active":"2024-09-10T19:24:24.910173+0000","last_became_peered":"2024-09-10T19:24:24.910173+0000","last_unstale":"2024-09-10T19:33:24.312111+0000","last_undegraded":"2024-09-10T19:33:24.312111+0000","last_fullsized":"2024-09-10T19:33:24.312111+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:33:06.212952+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:33:06.212952+0000","last_clean_scrub_stamp":"2024-09-10T19:33:06.212952+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:34:24.192124+0000","scrub_duration":0.0014953430000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"376'13","reported_seq":741,"reported_epoch":460,"state":"active+clean","last_fresh":"2024-09-10T19:33:26.004493+0000","last_change":"2024-09-10T19:32:54.273571+0000","last_active":"2024-09-10T19:33:26.004493+0000","last_peered":"2024-09-10T19:33:26.004493+0000","last_clean":"2024-09-10T19:33:26.004493+0000","last_became_active":"2024-09-10T19:27:53.000311+0000","last_became_peered":"2024-09-10T19:27:53.000311+0000","last_unstale":"2024-09-10T19:33:26.004493+0000","last_undegraded":"2024-09-10T19:33:26.004493+0000","last_fullsized":"2024-09-10T19:33:26.004493+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"376'13","last_scrub_stamp":"2024-09-10T19:32:54.273510+0000","last_deep_scrub":"376'13","last_deep_scrub_stamp":"2024-09-10T19:28:17.490297+0000","last_clean_scrub_stamp":"2024-09-10T19:32:54.273510+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:34:18.709551+0000","scrub_duration":0.0020515640000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":491,"reported_epoch":460,"state":"active+clean","last_fresh":"2024-09-10T19:33:25.321194+0000","last_change":"2024-09-10T19:33:25.321140+0000","last_active":"2024-09-10T19:33:25.321194+0000","last_peered":"2024-09-10T19:33:25.321194+0000","last_clean":"2024-09-10T19:33:25.321194+0000","last_became_active":"2024-09-10T19:31:33.335255+0000","last_became_peered":"2024-09-10T19:31:33.335255+0000","last_unstale":"2024-09-10T19:33:25.321194+0000","last_undegraded":"2024-09-10T19:33:25.321194+0000","last_fullsized":"2024-09-10T19:33:25.321194+0000","mapping_epoch":431,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":432,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:33:25.321069+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:33:25.321069+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:34:53.518637+0000","scrub_duration":0.0016760519999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"74'79","reported_seq":303,"reported_epoch":460,"state":"active+clean","last_fresh":"2024-09-10T19:33:23.468743+0000","last_change":"2024-09-10T19:32:43.488435+0000","last_active":"2024-09-10T19:33:23.468743+0000","last_peered":"2024-09-10T19:33:23.468743+0000","last_clean":"2024-09-10T19:33:23.468743+0000","last_became_active":"2024-09-10T19:31:33.338083+0000","last_became_peered":"2024-09-10T19:31:33.338083+0000","last_unstale":"2024-09-10T19:33:23.468743+0000","last_undegraded":"2024-09-10T19:33:23.468743+0000","last_fullsized":"2024-09-10T19:33:23.468743+0000","mapping_epoch":431,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":432,"parent":"0.0","parent_split_bits":4,"last_scrub":"74'79","last_scrub_stamp":"2024-09-10T19:32:43.488362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:08:00.699315+0000","last_clean_scrub_stamp":"2024-09-10T19:32:43.488362+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:34:06.539920+0000","scrub_duration":0.001671467,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"432'69","reported_seq":477,"reported_epoch":460,"state":"active+clean","last_fresh":"2024-09-10T19:33:26.004632+0000","last_change":"2024-09-10T19:32:09.310933+0000","last_active":"2024-09-10T19:33:26.004632+0000","last_peered":"2024-09-10T19:33:26.004632+0000","last_clean":"2024-09-10T19:33:26.004632+0000","last_became_active":"2024-09-10T19:31:33.348744+0000","last_became_peered":"2024-09-10T19:31:33.348744+0000","last_unstale":"2024-09-10T19:33:26.004632+0000","last_undegraded":"2024-09-10T19:33:26.004632+0000","last_fullsized":"2024-09-10T19:33:26.004632+0000","mapping_epoch":431,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":432,"parent":"0.0","parent_split_bits":0,"last_scrub":"432'69","last_scrub_stamp":"2024-09-10T19:32:09.310870+0000","last_deep_scrub":"384'56","last_deep_scrub_stamp":"2024-09-10T19:28:23.457119+0000","last_clean_scrub_stamp":"2024-09-10T19:32:09.310870+0000","objects_scrubbed":10,"log_size":69,"log_dups_size":0,"ondisk_log_size":69,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:33:37.960731+0000","scrub_duration":0.0021013970000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":83,"num_read_kb":68,"num_write":50,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":50,"num_bytes_recovered":1790,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":555,"reported_epoch":461,"state":"active+clean","last_fresh":"2024-09-10T19:33:26.580637+0000","last_change":"2024-09-10T19:33:26.580606+0000","last_active":"2024-09-10T19:33:26.580637+0000","last_peered":"2024-09-10T19:33:26.580637+0000","last_clean":"2024-09-10T19:33:26.580637+0000","last_became_active":"2024-09-10T19:27:53.004220+0000","last_became_peered":"2024-09-10T19:27:53.004220+0000","last_unstale":"2024-09-10T19:33:26.580637+0000","last_undegraded":"2024-09-10T19:33:26.580637+0000","last_fullsized":"2024-09-10T19:33:26.580637+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:33:26.580536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:32:09.420540+0000","last_clean_scrub_stamp":"2024-09-10T19:33:26.580536+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:34:49.718456+0000","scrub_duration":0.001590726,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":471,"reported_epoch":460,"state":"active+clean","last_fresh":"2024-09-10T19:33:24.310083+0000","last_change":"2024-09-10T19:32:01.057536+0000","last_active":"2024-09-10T19:33:24.310083+0000","last_peered":"2024-09-10T19:33:24.310083+0000","last_clean":"2024-09-10T19:33:24.310083+0000","last_became_active":"2024-09-10T19:24:24.892692+0000","last_became_peered":"2024-09-10T19:24:24.892692+0000","last_unstale":"2024-09-10T19:33:24.310083+0000","last_undegraded":"2024-09-10T19:33:24.310083+0000","last_fullsized":"2024-09-10T19:33:24.310083+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:32:01.057482+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:32:01.057482+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:33:28.458880+0000","scrub_duration":0.001292009,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":485,"reported_epoch":460,"state":"active+clean","last_fresh":"2024-09-10T19:33:23.293734+0000","last_change":"2024-09-10T19:32:41.052328+0000","last_active":"2024-09-10T19:33:23.293734+0000","last_peered":"2024-09-10T19:33:23.293734+0000","last_clean":"2024-09-10T19:33:23.293734+0000","last_became_active":"2024-09-10T19:10:13.721744+0000","last_became_peered":"2024-09-10T19:10:13.721744+0000","last_unstale":"2024-09-10T19:33:23.293734+0000","last_undegraded":"2024-09-10T19:33:23.293734+0000","last_fullsized":"2024-09-10T19:33:23.293734+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:32:41.052274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:32:41.052274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:33:51.209533+0000","scrub_duration":0.0012557759999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.2","version":"74'79","reported_seq":583,"reported_epoch":460,"state":"active+clean","last_fresh":"2024-09-10T19:33:23.916134+0000","last_change":"2024-09-10T19:32:23.941209+0000","last_active":"2024-09-10T19:33:23.916134+0000","last_peered":"2024-09-10T19:33:23.916134+0000","last_clean":"2024-09-10T19:33:23.916134+0000","last_became_active":"2024-09-10T19:24:23.895247+0000","last_became_peered":"2024-09-10T19:24:23.895247+0000","last_unstale":"2024-09-10T19:33:23.916134+0000","last_undegraded":"2024-09-10T19:33:23.916134+0000","last_fullsized":"2024-09-10T19:33:23.916134+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":4,"last_scrub":"74'79","last_scrub_stamp":"2024-09-10T19:32:23.941133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:08:00.699315+0000","last_clean_scrub_stamp":"2024-09-10T19:32:23.941133+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:33:52.738059+0000","scrub_duration":0.0015017470000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.7","version":"414'26","reported_seq":260,"reported_epoch":460,"state":"active+clean","last_fresh":"2024-09-10T19:33:24.312153+0000","last_change":"2024-09-10T19:32:59.306088+0000","last_active":"2024-09-10T19:33:24.312153+0000","last_peered":"2024-09-10T19:33:24.312153+0000","last_clean":"2024-09-10T19:33:24.312153+0000","last_became_active":"2024-09-10T19:24:24.894293+0000","last_became_peered":"2024-09-10T19:24:24.894293+0000","last_unstale":"2024-09-10T19:33:24.312153+0000","last_undegraded":"2024-09-10T19:33:24.312153+0000","last_fullsized":"2024-09-10T19:33:24.312153+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"414'26","last_scrub_stamp":"2024-09-10T19:32:59.306018+0000","last_deep_scrub":"82'18","last_deep_scrub_stamp":"2024-09-10T19:15:55.777361+0000","last_clean_scrub_stamp":"2024-09-10T19:32:59.306018+0000","objects_scrubbed":5,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:34:29.066767+0000","scrub_duration":0.0052662430000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":42,"num_read_kb":31,"num_write":24,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"159'6","reported_seq":849,"reported_epoch":461,"state":"active+clean","last_fresh":"2024-09-10T19:33:26.535668+0000","last_change":"2024-09-10T19:32:21.431132+0000","last_active":"2024-09-10T19:33:26.535668+0000","last_peered":"2024-09-10T19:33:26.535668+0000","last_clean":"2024-09-10T19:33:26.535668+0000","last_became_active":"2024-09-10T19:31:33.337696+0000","last_became_peered":"2024-09-10T19:31:33.337696+0000","last_unstale":"2024-09-10T19:33:26.535668+0000","last_undegraded":"2024-09-10T19:33:26.535668+0000","last_fullsized":"2024-09-10T19:33:26.535668+0000","mapping_epoch":431,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":432,"parent":"0.0","parent_split_bits":0,"last_scrub":"159'6","last_scrub_stamp":"2024-09-10T19:32:21.431038+0000","last_deep_scrub":"159'6","last_deep_scrub_stamp":"2024-09-10T19:16:32.649914+0000","last_clean_scrub_stamp":"2024-09-10T19:32:21.431038+0000","objects_scrubbed":1,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:33:30.035389+0000","scrub_duration":0.0019491409999999999,"objects_trimmed":0,"snaptrim_duration":0.00034454499999999997,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":565,"num_read_kb":560,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":1945,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":59,"reported_epoch":461,"state":"active+clean","last_fresh":"2024-09-10T19:33:26.730341+0000","last_change":"2024-09-10T19:33:26.730299+0000","last_active":"2024-09-10T19:33:26.730341+0000","last_peered":"2024-09-10T19:33:26.730341+0000","last_clean":"2024-09-10T19:33:26.730341+0000","last_became_active":"2024-09-10T19:31:33.339737+0000","last_became_peered":"2024-09-10T19:31:33.339737+0000","last_unstale":"2024-09-10T19:33:26.730341+0000","last_undegraded":"2024-09-10T19:33:26.730341+0000","last_fullsized":"2024-09-10T19:33:26.730341+0000","mapping_epoch":431,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":432,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:33:26.730213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:28:21.199065+0000","last_clean_scrub_stamp":"2024-09-10T19:33:26.730213+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:34:54.815372+0000","scrub_duration":0.001426025,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":59,"reported_epoch":460,"state":"active+clean","last_fresh":"2024-09-10T19:33:23.294281+0000","last_change":"2024-09-10T19:33:20.712421+0000","last_active":"2024-09-10T19:33:23.294281+0000","last_peered":"2024-09-10T19:33:23.294281+0000","last_clean":"2024-09-10T19:33:23.294281+0000","last_became_active":"2024-09-10T19:31:33.347689+0000","last_became_peered":"2024-09-10T19:31:33.347689+0000","last_unstale":"2024-09-10T19:33:23.294281+0000","last_undegraded":"2024-09-10T19:33:23.294281+0000","last_fullsized":"2024-09-10T19:33:23.294281+0000","mapping_epoch":431,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":432,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:33:20.712370+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:33:20.712370+0000","last_clean_scrub_stamp":"2024-09-10T19:33:20.712370+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:34:35.235170+0000","scrub_duration":0.0013864369999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":534,"reported_epoch":460,"state":"active+clean","last_fresh":"2024-09-10T19:33:23.916413+0000","last_change":"2024-09-10T19:33:14.818692+0000","last_active":"2024-09-10T19:33:23.916413+0000","last_peered":"2024-09-10T19:33:23.916413+0000","last_clean":"2024-09-10T19:33:23.916413+0000","last_became_active":"2024-09-10T19:27:53.011923+0000","last_became_peered":"2024-09-10T19:27:53.011923+0000","last_unstale":"2024-09-10T19:33:23.916413+0000","last_undegraded":"2024-09-10T19:33:23.916413+0000","last_fullsized":"2024-09-10T19:33:23.916413+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:33:14.818620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:22:01.208272+0000","last_clean_scrub_stamp":"2024-09-10T19:33:14.818620+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:34:36.076319+0000","scrub_duration":0.0016019960000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.3","version":"74'79","reported_seq":238,"reported_epoch":460,"state":"active+clean","last_fresh":"2024-09-10T19:33:24.312173+0000","last_change":"2024-09-10T19:32:05.334159+0000","last_active":"2024-09-10T19:33:24.312173+0000","last_peered":"2024-09-10T19:33:24.312173+0000","last_clean":"2024-09-10T19:33:24.312173+0000","last_became_active":"2024-09-10T19:27:53.016965+0000","last_became_peered":"2024-09-10T19:27:53.016965+0000","last_unstale":"2024-09-10T19:33:24.312173+0000","last_undegraded":"2024-09-10T19:33:24.312173+0000","last_fullsized":"2024-09-10T19:33:24.312173+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":242,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":3,"last_scrub":"74'79","last_scrub_stamp":"2024-09-10T19:32:05.334106+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:08:00.699315+0000","last_clean_scrub_stamp":"2024-09-10T19:32:05.334106+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001553776,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.6","version":"414'42","reported_seq":603,"reported_epoch":460,"state":"active+clean","last_fresh":"2024-09-10T19:33:24.310216+0000","last_change":"2024-09-10T19:32:44.224279+0000","last_active":"2024-09-10T19:33:24.310216+0000","last_peered":"2024-09-10T19:33:24.310216+0000","last_clean":"2024-09-10T19:33:24.310216+0000","last_became_active":"2024-09-10T19:31:33.334775+0000","last_became_peered":"2024-09-10T19:31:33.334775+0000","last_unstale":"2024-09-10T19:33:24.310216+0000","last_undegraded":"2024-09-10T19:33:24.310216+0000","last_fullsized":"2024-09-10T19:33:24.310216+0000","mapping_epoch":431,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":432,"parent":"0.0","parent_split_bits":0,"last_scrub":"414'42","last_scrub_stamp":"2024-09-10T19:32:44.224189+0000","last_deep_scrub":"221'18","last_deep_scrub_stamp":"2024-09-10T19:19:29.606748+0000","last_clean_scrub_stamp":"2024-09-10T19:32:44.224189+0000","objects_scrubbed":6,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:34:04.064176+0000","scrub_duration":0.0016931520000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":53,"num_read_kb":47,"num_write":36,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"417'2","reported_seq":465,"reported_epoch":460,"state":"active+clean","last_fresh":"2024-09-10T19:33:24.309783+0000","last_change":"2024-09-10T19:32:10.020662+0000","last_active":"2024-09-10T19:33:24.309783+0000","last_peered":"2024-09-10T19:33:24.309783+0000","last_clean":"2024-09-10T19:33:24.309783+0000","last_became_active":"2024-09-10T19:24:23.897412+0000","last_became_peered":"2024-09-10T19:24:23.897412+0000","last_unstale":"2024-09-10T19:33:24.309783+0000","last_undegraded":"2024-09-10T19:33:24.309783+0000","last_fullsized":"2024-09-10T19:33:24.309783+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"417'2","last_scrub_stamp":"2024-09-10T19:32:10.020604+0000","last_deep_scrub":"417'2","last_deep_scrub_stamp":"2024-09-10T19:30:39.904339+0000","last_clean_scrub_stamp":"2024-09-10T19:32:10.020604+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:33:38.047182+0000","scrub_duration":0.0023223800000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":549,"reported_epoch":460,"state":"active+clean","last_fresh":"2024-09-10T19:33:23.916089+0000","last_change":"2024-09-10T19:33:09.836643+0000","last_active":"2024-09-10T19:33:23.916089+0000","last_peered":"2024-09-10T19:33:23.916089+0000","last_clean":"2024-09-10T19:33:23.916089+0000","last_became_active":"2024-09-10T19:24:24.896201+0000","last_became_peered":"2024-09-10T19:24:24.896201+0000","last_unstale":"2024-09-10T19:33:23.916089+0000","last_undegraded":"2024-09-10T19:33:23.916089+0000","last_fullsized":"2024-09-10T19:33:23.916089+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:33:09.836575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:31:47.151810+0000","last_clean_scrub_stamp":"2024-09-10T19:33:09.836575+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:34:12.039746+0000","scrub_duration":0.0014646150000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":555,"reported_epoch":460,"state":"active+clean","last_fresh":"2024-09-10T19:33:23.469657+0000","last_change":"2024-09-10T19:33:20.544352+0000","last_active":"2024-09-10T19:33:23.469657+0000","last_peered":"2024-09-10T19:33:23.469657+0000","last_clean":"2024-09-10T19:33:23.469657+0000","last_became_active":"2024-09-10T19:24:24.914680+0000","last_became_peered":"2024-09-10T19:24:24.914680+0000","last_unstale":"2024-09-10T19:33:23.469657+0000","last_undegraded":"2024-09-10T19:33:23.469657+0000","last_fullsized":"2024-09-10T19:33:23.469657+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:33:20.544240+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:33:20.544240+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:34:32.234540+0000","scrub_duration":0.001710488,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":218,"reported_epoch":460,"state":"active+clean","last_fresh":"2024-09-10T19:33:24.311979+0000","last_change":"2024-09-10T19:31:59.247873+0000","last_active":"2024-09-10T19:33:24.311979+0000","last_peered":"2024-09-10T19:33:24.311979+0000","last_clean":"2024-09-10T19:33:24.311979+0000","last_became_active":"2024-09-10T19:24:01.974039+0000","last_became_peered":"2024-09-10T19:24:01.974039+0000","last_unstale":"2024-09-10T19:33:24.311979+0000","last_undegraded":"2024-09-10T19:33:24.311979+0000","last_fullsized":"2024-09-10T19:33:24.311979+0000","mapping_epoch":292,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":293,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:31:59.247819+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:31:59.247819+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.001632283,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"414'45","reported_seq":170,"reported_epoch":460,"state":"active+clean","last_fresh":"2024-09-10T19:33:24.311613+0000","last_change":"2024-09-10T19:33:21.338362+0000","last_active":"2024-09-10T19:33:24.311613+0000","last_peered":"2024-09-10T19:33:24.311613+0000","last_clean":"2024-09-10T19:33:24.311613+0000","last_became_active":"2024-09-10T19:27:53.006290+0000","last_became_peered":"2024-09-10T19:27:53.006290+0000","last_unstale":"2024-09-10T19:33:24.311613+0000","last_undegraded":"2024-09-10T19:33:24.311613+0000","last_fullsized":"2024-09-10T19:33:24.311613+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"414'45","last_scrub_stamp":"2024-09-10T19:33:21.338288+0000","last_deep_scrub":"80'21","last_deep_scrub_stamp":"2024-09-10T19:11:39.836144+0000","last_clean_scrub_stamp":"2024-09-10T19:33:21.338288+0000","objects_scrubbed":7,"log_size":45,"log_dups_size":0,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:34:35.531323+0000","scrub_duration":0.0022233510000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":60,"num_read_kb":50,"num_write":37,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":154,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"446'3","reported_seq":537,"reported_epoch":460,"state":"active+clean","last_fresh":"2024-09-10T19:33:23.294094+0000","last_change":"2024-09-10T19:32:17.358949+0000","last_active":"2024-09-10T19:33:23.294094+0000","last_peered":"2024-09-10T19:33:23.294094+0000","last_clean":"2024-09-10T19:33:23.294094+0000","last_became_active":"2024-09-10T19:27:53.025802+0000","last_became_peered":"2024-09-10T19:27:53.025802+0000","last_unstale":"2024-09-10T19:33:23.294094+0000","last_undegraded":"2024-09-10T19:33:23.294094+0000","last_fullsized":"2024-09-10T19:33:23.294094+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"446'3","last_scrub_stamp":"2024-09-10T19:32:17.358880+0000","last_deep_scrub":"446'3","last_deep_scrub_stamp":"2024-09-10T19:32:17.358880+0000","last_clean_scrub_stamp":"2024-09-10T19:32:17.358880+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:33:33.552976+0000","scrub_duration":0.005695248,"objects_trimmed":0,"snaptrim_duration":0.00075522999999999996,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":531,"reported_epoch":460,"state":"active+clean","last_fresh":"2024-09-10T19:33:23.469266+0000","last_change":"2024-09-10T19:33:09.529149+0000","last_active":"2024-09-10T19:33:23.469266+0000","last_peered":"2024-09-10T19:33:23.469266+0000","last_clean":"2024-09-10T19:33:23.469266+0000","last_became_active":"2024-09-10T19:31:33.349216+0000","last_became_peered":"2024-09-10T19:31:33.349216+0000","last_unstale":"2024-09-10T19:33:23.469266+0000","last_undegraded":"2024-09-10T19:33:23.469266+0000","last_fullsized":"2024-09-10T19:33:23.469266+0000","mapping_epoch":431,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":432,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:33:09.529079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:33:09.529079+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:34:18.773549+0000","scrub_duration":0.0015652789999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"74'79","reported_seq":597,"reported_epoch":460,"state":"active+clean","last_fresh":"2024-09-10T19:33:23.916270+0000","last_change":"2024-09-10T19:33:01.852337+0000","last_active":"2024-09-10T19:33:23.916270+0000","last_peered":"2024-09-10T19:33:23.916270+0000","last_clean":"2024-09-10T19:33:23.916270+0000","last_became_active":"2024-09-10T19:27:53.009367+0000","last_became_peered":"2024-09-10T19:27:53.009367+0000","last_unstale":"2024-09-10T19:33:23.916270+0000","last_undegraded":"2024-09-10T19:33:23.916270+0000","last_fullsized":"2024-09-10T19:33:23.916270+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'79","last_scrub_stamp":"2024-09-10T19:33:01.852266+0000","last_deep_scrub":"74'79","last_deep_scrub_stamp":"2024-09-10T19:31:25.002382+0000","last_clean_scrub_stamp":"2024-09-10T19:33:01.852266+0000","objects_scrubbed":0,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:34:04.920975+0000","scrub_duration":0.0014929170000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":534,"reported_epoch":460,"state":"active+clean","last_fresh":"2024-09-10T19:33:23.916526+0000","last_change":"2024-09-10T19:31:54.175652+0000","last_active":"2024-09-10T19:33:23.916526+0000","last_peered":"2024-09-10T19:33:23.916526+0000","last_clean":"2024-09-10T19:33:23.916526+0000","last_became_active":"2024-09-10T19:31:33.339073+0000","last_became_peered":"2024-09-10T19:31:33.339073+0000","last_unstale":"2024-09-10T19:33:23.916526+0000","last_undegraded":"2024-09-10T19:33:23.916526+0000","last_fullsized":"2024-09-10T19:33:23.916526+0000","mapping_epoch":431,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":432,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:31:54.175551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:31:54.175551+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0015182080000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":495,"reported_epoch":460,"state":"active+clean","last_fresh":"2024-09-10T19:33:23.469734+0000","last_change":"2024-09-10T19:32:02.509629+0000","last_active":"2024-09-10T19:33:23.469734+0000","last_peered":"2024-09-10T19:33:23.469734+0000","last_clean":"2024-09-10T19:33:23.469734+0000","last_became_active":"2024-09-10T19:10:13.720647+0000","last_became_peered":"2024-09-10T19:10:13.720647+0000","last_unstale":"2024-09-10T19:33:23.469734+0000","last_undegraded":"2024-09-10T19:33:23.469734+0000","last_fullsized":"2024-09-10T19:33:23.469734+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:32:02.509565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:28:29.899579+0000","last_clean_scrub_stamp":"2024-09-10T19:32:02.509565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:33:26.850660+0000","scrub_duration":0.0017406069999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.a","version":"414'51","reported_seq":78,"reported_epoch":460,"state":"active+clean","last_fresh":"2024-09-10T19:33:23.294010+0000","last_change":"2024-09-10T19:32:00.568977+0000","last_active":"2024-09-10T19:33:23.294010+0000","last_peered":"2024-09-10T19:33:23.294010+0000","last_clean":"2024-09-10T19:33:23.294010+0000","last_became_active":"2024-09-10T19:31:33.337763+0000","last_became_peered":"2024-09-10T19:31:33.337763+0000","last_unstale":"2024-09-10T19:33:23.294010+0000","last_undegraded":"2024-09-10T19:33:23.294010+0000","last_fullsized":"2024-09-10T19:33:23.294010+0000","mapping_epoch":431,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":432,"parent":"0.0","parent_split_bits":0,"last_scrub":"414'51","last_scrub_stamp":"2024-09-10T19:32:00.568913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:32:00.568913+0000","objects_scrubbed":9,"log_size":51,"log_dups_size":0,"ondisk_log_size":51,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:33:27.261721+0000","scrub_duration":0.0018776369999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":79,"num_read_kb":68,"num_write":52,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":36,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":49,"reported_epoch":460,"state":"active+clean","last_fresh":"2024-09-10T19:33:23.293631+0000","last_change":"2024-09-10T19:33:07.614745+0000","last_active":"2024-09-10T19:33:23.293631+0000","last_peered":"2024-09-10T19:33:23.293631+0000","last_clean":"2024-09-10T19:33:23.293631+0000","last_became_active":"2024-09-10T19:31:33.343515+0000","last_became_peered":"2024-09-10T19:31:33.343515+0000","last_unstale":"2024-09-10T19:33:23.293631+0000","last_undegraded":"2024-09-10T19:33:23.293631+0000","last_fullsized":"2024-09-10T19:33:23.293631+0000","mapping_epoch":431,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":432,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:33:07.614692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:31:25.719972+0000","last_clean_scrub_stamp":"2024-09-10T19:33:07.614692+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:34:25.529285+0000","scrub_duration":0.001709704,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":59,"reported_epoch":460,"state":"active+clean","last_fresh":"2024-09-10T19:33:23.293626+0000","last_change":"2024-09-10T19:33:14.709245+0000","last_active":"2024-09-10T19:33:23.293626+0000","last_peered":"2024-09-10T19:33:23.293626+0000","last_clean":"2024-09-10T19:33:23.293626+0000","last_became_active":"2024-09-10T19:31:33.340592+0000","last_became_peered":"2024-09-10T19:31:33.340592+0000","last_unstale":"2024-09-10T19:33:23.293626+0000","last_undegraded":"2024-09-10T19:33:23.293626+0000","last_fullsized":"2024-09-10T19:33:23.293626+0000","mapping_epoch":431,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":432,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:33:14.709190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:33:14.709190+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:34:39.599665+0000","scrub_duration":0.0013793360000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":555,"reported_epoch":460,"state":"active+clean","last_fresh":"2024-09-10T19:33:23.916064+0000","last_change":"2024-09-10T19:33:06.843999+0000","last_active":"2024-09-10T19:33:23.916064+0000","last_peered":"2024-09-10T19:33:23.916064+0000","last_clean":"2024-09-10T19:33:23.916064+0000","last_became_active":"2024-09-10T19:31:33.337545+0000","last_became_peered":"2024-09-10T19:31:33.337545+0000","last_unstale":"2024-09-10T19:33:23.916064+0000","last_undegraded":"2024-09-10T19:33:23.916064+0000","last_fullsized":"2024-09-10T19:33:23.916064+0000","mapping_epoch":431,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":432,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:33:06.843928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:21:03.910074+0000","last_clean_scrub_stamp":"2024-09-10T19:33:06.843928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:34:32.780150+0000","scrub_duration":0.001549398,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"414'21","reported_seq":554,"reported_epoch":460,"state":"active+clean","last_fresh":"2024-09-10T19:33:24.309655+0000","last_change":"2024-09-10T19:33:03.305872+0000","last_active":"2024-09-10T19:33:24.309655+0000","last_peered":"2024-09-10T19:33:24.309655+0000","last_clean":"2024-09-10T19:33:24.309655+0000","last_became_active":"2024-09-10T19:24:44.892294+0000","last_became_peered":"2024-09-10T19:24:44.892294+0000","last_unstale":"2024-09-10T19:33:24.309655+0000","last_undegraded":"2024-09-10T19:33:24.309655+0000","last_fullsized":"2024-09-10T19:33:24.309655+0000","mapping_epoch":314,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":315,"parent":"0.0","parent_split_bits":0,"last_scrub":"414'21","last_scrub_stamp":"2024-09-10T19:33:03.305818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:33:03.305818+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:34:14.068212+0000","scrub_duration":0.001846271,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":34,"num_read_kb":28,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,4],"acting":[0,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":218,"reported_epoch":460,"state":"active+clean","last_fresh":"2024-09-10T19:33:24.312362+0000","last_change":"2024-09-10T19:32:43.317430+0000","last_active":"2024-09-10T19:33:24.312362+0000","last_peered":"2024-09-10T19:33:24.312362+0000","last_clean":"2024-09-10T19:33:24.312362+0000","last_became_active":"2024-09-10T19:24:01.980348+0000","last_became_peered":"2024-09-10T19:24:01.980348+0000","last_unstale":"2024-09-10T19:33:24.312362+0000","last_undegraded":"2024-09-10T19:33:24.312362+0000","last_fullsized":"2024-09-10T19:33:24.312362+0000","mapping_epoch":292,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":293,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:32:43.317372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:11:44.254513+0000","last_clean_scrub_stamp":"2024-09-10T19:32:43.317372+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:34:07.570635+0000","scrub_duration":0.001876363,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,3],"acting":[1,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":59,"reported_epoch":460,"state":"active+clean","last_fresh":"2024-09-10T19:33:24.681873+0000","last_change":"2024-09-10T19:33:24.681822+0000","last_active":"2024-09-10T19:33:24.681873+0000","last_peered":"2024-09-10T19:33:24.681873+0000","last_clean":"2024-09-10T19:33:24.681873+0000","last_became_active":"2024-09-10T19:31:33.345296+0000","last_became_peered":"2024-09-10T19:31:33.345296+0000","last_unstale":"2024-09-10T19:33:24.681873+0000","last_undegraded":"2024-09-10T19:33:24.681873+0000","last_fullsized":"2024-09-10T19:33:24.681873+0000","mapping_epoch":431,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":432,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:33:24.681769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:33:24.681769+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:34:50.503060+0000","scrub_duration":0.0016144099999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":469,"reported_epoch":460,"state":"active+clean","last_fresh":"2024-09-10T19:33:24.309312+0000","last_change":"2024-09-10T19:33:20.250155+0000","last_active":"2024-09-10T19:33:24.309312+0000","last_peered":"2024-09-10T19:33:24.309312+0000","last_clean":"2024-09-10T19:33:24.309312+0000","last_became_active":"2024-09-10T19:31:33.339964+0000","last_became_peered":"2024-09-10T19:31:33.339964+0000","last_unstale":"2024-09-10T19:33:24.309312+0000","last_undegraded":"2024-09-10T19:33:24.309312+0000","last_fullsized":"2024-09-10T19:33:24.309312+0000","mapping_epoch":431,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":432,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:33:20.250078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:30:33.974134+0000","last_clean_scrub_stamp":"2024-09-10T19:33:20.250078+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:34:43.565662+0000","scrub_duration":0.0022003880000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.8","version":"414'46","reported_seq":619,"reported_epoch":460,"state":"active+clean","last_fresh":"2024-09-10T19:33:23.468792+0000","last_change":"2024-09-10T19:33:19.539310+0000","last_active":"2024-09-10T19:33:23.468792+0000","last_peered":"2024-09-10T19:33:23.468792+0000","last_clean":"2024-09-10T19:33:23.468792+0000","last_became_active":"2024-09-10T19:27:53.004858+0000","last_became_peered":"2024-09-10T19:27:53.004858+0000","last_unstale":"2024-09-10T19:33:23.468792+0000","last_undegraded":"2024-09-10T19:33:23.468792+0000","last_fullsized":"2024-09-10T19:33:23.468792+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"414'46","last_scrub_stamp":"2024-09-10T19:33:19.539238+0000","last_deep_scrub":"220'22","last_deep_scrub_stamp":"2024-09-10T19:19:24.597866+0000","last_clean_scrub_stamp":"2024-09-10T19:33:19.539238+0000","objects_scrubbed":7,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:34:19.828559+0000","scrub_duration":0.0021546320000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":62,"num_read_kb":53,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":59,"reported_epoch":460,"state":"active+clean","last_fresh":"2024-09-10T19:33:23.294468+0000","last_change":"2024-09-10T19:33:09.660425+0000","last_active":"2024-09-10T19:33:23.294468+0000","last_peered":"2024-09-10T19:33:23.294468+0000","last_clean":"2024-09-10T19:33:23.294468+0000","last_became_active":"2024-09-10T19:31:33.347734+0000","last_became_peered":"2024-09-10T19:31:33.347734+0000","last_unstale":"2024-09-10T19:33:23.294468+0000","last_undegraded":"2024-09-10T19:33:23.294468+0000","last_fullsized":"2024-09-10T19:33:23.294468+0000","mapping_epoch":431,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":432,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:33:09.660372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:33:09.660372+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:34:36.542479+0000","scrub_duration":0.0015034549999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.9","version":"376'11","reported_seq":824,"reported_epoch":460,"state":"active+clean","last_fresh":"2024-09-10T19:33:26.004852+0000","last_change":"2024-09-10T19:31:55.163970+0000","last_active":"2024-09-10T19:33:26.004852+0000","last_peered":"2024-09-10T19:33:26.004852+0000","last_clean":"2024-09-10T19:33:26.004852+0000","last_became_active":"2024-09-10T19:27:53.009810+0000","last_became_peered":"2024-09-10T19:27:53.009810+0000","last_unstale":"2024-09-10T19:33:26.004852+0000","last_undegraded":"2024-09-10T19:33:26.004852+0000","last_fullsized":"2024-09-10T19:33:26.004852+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"376'11","last_scrub_stamp":"2024-09-10T19:31:55.163902+0000","last_deep_scrub":"80'9","last_deep_scrub_stamp":"2024-09-10T19:25:29.833271+0000","last_clean_scrub_stamp":"2024-09-10T19:31:55.163902+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0017787429999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":522,"reported_epoch":460,"state":"active+clean","last_fresh":"2024-09-10T19:33:23.469191+0000","last_change":"2024-09-10T19:32:23.388747+0000","last_active":"2024-09-10T19:33:23.469191+0000","last_peered":"2024-09-10T19:33:23.469191+0000","last_clean":"2024-09-10T19:33:23.469191+0000","last_became_active":"2024-09-10T19:27:53.015957+0000","last_became_peered":"2024-09-10T19:27:53.015957+0000","last_unstale":"2024-09-10T19:33:23.469191+0000","last_undegraded":"2024-09-10T19:33:23.469191+0000","last_fullsized":"2024-09-10T19:33:23.469191+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:32:23.388634+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:29:27.408374+0000","last_clean_scrub_stamp":"2024-09-10T19:32:23.388634+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0023981499999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"414'24","reported_seq":593,"reported_epoch":460,"state":"active+clean","last_fresh":"2024-09-10T19:33:23.292687+0000","last_change":"2024-09-10T19:32:42.833945+0000","last_active":"2024-09-10T19:33:23.292687+0000","last_peered":"2024-09-10T19:33:23.292687+0000","last_clean":"2024-09-10T19:33:23.292687+0000","last_became_active":"2024-09-10T19:24:01.974878+0000","last_became_peered":"2024-09-10T19:24:01.974878+0000","last_unstale":"2024-09-10T19:33:23.292687+0000","last_undegraded":"2024-09-10T19:33:23.292687+0000","last_fullsized":"2024-09-10T19:33:23.292687+0000","mapping_epoch":292,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":293,"parent":"0.0","parent_split_bits":0,"last_scrub":"414'24","last_scrub_stamp":"2024-09-10T19:32:42.833876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:32:42.833876+0000","objects_scrubbed":5,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:33:45.734397+0000","scrub_duration":0.0035356340000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.f","version":"59'2","reported_seq":533,"reported_epoch":461,"state":"active+clean","last_fresh":"2024-09-10T19:33:27.063520+0000","last_change":"2024-09-10T19:33:27.063467+0000","last_active":"2024-09-10T19:33:27.063520+0000","last_peered":"2024-09-10T19:33:27.063520+0000","last_clean":"2024-09-10T19:33:27.063520+0000","last_became_active":"2024-09-10T19:24:23.893796+0000","last_became_peered":"2024-09-10T19:24:23.893796+0000","last_unstale":"2024-09-10T19:33:27.063520+0000","last_undegraded":"2024-09-10T19:33:27.063520+0000","last_fullsized":"2024-09-10T19:33:27.063520+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"59'2","last_scrub_stamp":"2024-09-10T19:33:27.063392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:33:27.063392+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:34:46.934809+0000","scrub_duration":0.0019905700000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":14,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,7],"acting":[4,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":49,"reported_epoch":460,"state":"active+clean","last_fresh":"2024-09-10T19:33:23.292866+0000","last_change":"2024-09-10T19:32:33.664513+0000","last_active":"2024-09-10T19:33:23.292866+0000","last_peered":"2024-09-10T19:33:23.292866+0000","last_clean":"2024-09-10T19:33:23.292866+0000","last_became_active":"2024-09-10T19:31:33.344222+0000","last_became_peered":"2024-09-10T19:31:33.344222+0000","last_unstale":"2024-09-10T19:33:23.292866+0000","last_undegraded":"2024-09-10T19:33:23.292866+0000","last_fullsized":"2024-09-10T19:33:23.292866+0000","mapping_epoch":431,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":432,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:32:33.664429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:11:46.587846+0000","last_clean_scrub_stamp":"2024-09-10T19:32:33.664429+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:33:45.166431+0000","scrub_duration":0.001339238,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":522,"reported_epoch":460,"state":"active+clean","last_fresh":"2024-09-10T19:33:24.334473+0000","last_change":"2024-09-10T19:33:24.334412+0000","last_active":"2024-09-10T19:33:24.334473+0000","last_peered":"2024-09-10T19:33:24.334473+0000","last_clean":"2024-09-10T19:33:24.334473+0000","last_became_active":"2024-09-10T19:24:01.971131+0000","last_became_peered":"2024-09-10T19:24:01.971131+0000","last_unstale":"2024-09-10T19:33:24.334473+0000","last_undegraded":"2024-09-10T19:33:24.334473+0000","last_fullsized":"2024-09-10T19:33:24.334473+0000","mapping_epoch":292,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":293,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:33:24.334329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:33:24.334329+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:34:31.938960+0000","scrub_duration":0.00155292,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.16","version":"414'21","reported_seq":508,"reported_epoch":460,"state":"active+clean","last_fresh":"2024-09-10T19:33:24.310135+0000","last_change":"2024-09-10T19:32:51.292762+0000","last_active":"2024-09-10T19:33:24.310135+0000","last_peered":"2024-09-10T19:33:24.310135+0000","last_clean":"2024-09-10T19:33:24.310135+0000","last_became_active":"2024-09-10T19:24:23.901610+0000","last_became_peered":"2024-09-10T19:24:23.901610+0000","last_unstale":"2024-09-10T19:33:24.310135+0000","last_undegraded":"2024-09-10T19:33:24.310135+0000","last_fullsized":"2024-09-10T19:33:24.310135+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"414'21","last_scrub_stamp":"2024-09-10T19:32:51.292702+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:32:51.292702+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:33:56.128089+0000","scrub_duration":0.0019199130000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":34,"num_read_kb":28,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":59,"reported_epoch":460,"state":"active+clean","last_fresh":"2024-09-10T19:33:23.292960+0000","last_change":"2024-09-10T19:33:03.574265+0000","last_active":"2024-09-10T19:33:23.292960+0000","last_peered":"2024-09-10T19:33:23.292960+0000","last_clean":"2024-09-10T19:33:23.292960+0000","last_became_active":"2024-09-10T19:31:33.343673+0000","last_became_peered":"2024-09-10T19:31:33.343673+0000","last_unstale":"2024-09-10T19:33:23.292960+0000","last_undegraded":"2024-09-10T19:33:23.292960+0000","last_fullsized":"2024-09-10T19:33:23.292960+0000","mapping_epoch":431,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":432,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:33:03.574206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:11:41.458523+0000","last_clean_scrub_stamp":"2024-09-10T19:33:03.574206+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:34:06.554523+0000","scrub_duration":0.001247939,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":521,"reported_epoch":460,"state":"active+clean","last_fresh":"2024-09-10T19:33:23.292785+0000","last_change":"2024-09-10T19:33:07.199530+0000","last_active":"2024-09-10T19:33:23.292785+0000","last_peered":"2024-09-10T19:33:23.292785+0000","last_clean":"2024-09-10T19:33:23.292785+0000","last_became_active":"2024-09-10T19:24:01.970325+0000","last_became_peered":"2024-09-10T19:24:01.970325+0000","last_unstale":"2024-09-10T19:33:23.292785+0000","last_undegraded":"2024-09-10T19:33:23.292785+0000","last_fullsized":"2024-09-10T19:33:23.292785+0000","mapping_epoch":292,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":293,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:33:07.199475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:24:09.367654+0000","last_clean_scrub_stamp":"2024-09-10T19:33:07.199475+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:34:31.714857+0000","scrub_duration":0.0015561520000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"66'1","reported_seq":53,"reported_epoch":460,"state":"active+clean","last_fresh":"2024-09-10T19:33:23.293516+0000","last_change":"2024-09-10T19:32:39.551636+0000","last_active":"2024-09-10T19:33:23.293516+0000","last_peered":"2024-09-10T19:33:23.293516+0000","last_clean":"2024-09-10T19:33:23.293516+0000","last_became_active":"2024-09-10T19:31:33.347883+0000","last_became_peered":"2024-09-10T19:31:33.347883+0000","last_unstale":"2024-09-10T19:33:23.293516+0000","last_undegraded":"2024-09-10T19:33:23.293516+0000","last_fullsized":"2024-09-10T19:33:23.293516+0000","mapping_epoch":431,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":432,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'1","last_scrub_stamp":"2024-09-10T19:32:39.551576+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:32:39.551576+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:33:45.602034+0000","scrub_duration":0.0021501459999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":13,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":146,"reported_epoch":460,"state":"active+clean","last_fresh":"2024-09-10T19:33:24.311227+0000","last_change":"2024-09-10T19:33:20.376833+0000","last_active":"2024-09-10T19:33:24.311227+0000","last_peered":"2024-09-10T19:33:24.311227+0000","last_clean":"2024-09-10T19:33:24.311227+0000","last_became_active":"2024-09-10T19:27:53.007487+0000","last_became_peered":"2024-09-10T19:27:53.007487+0000","last_unstale":"2024-09-10T19:33:24.311227+0000","last_undegraded":"2024-09-10T19:33:24.311227+0000","last_fullsized":"2024-09-10T19:33:24.311227+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:33:20.376739+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:33:20.376739+0000","last_clean_scrub_stamp":"2024-09-10T19:33:20.376739+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:34:23.186510+0000","scrub_duration":0.001408587,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"414'18","reported_seq":577,"reported_epoch":460,"state":"active+clean","last_fresh":"2024-09-10T19:33:23.293012+0000","last_change":"2024-09-10T19:33:13.285069+0000","last_active":"2024-09-10T19:33:23.293012+0000","last_peered":"2024-09-10T19:33:23.293012+0000","last_clean":"2024-09-10T19:33:23.293012+0000","last_became_active":"2024-09-10T19:24:01.971492+0000","last_became_peered":"2024-09-10T19:24:01.971492+0000","last_unstale":"2024-09-10T19:33:23.293012+0000","last_undegraded":"2024-09-10T19:33:23.293012+0000","last_fullsized":"2024-09-10T19:33:23.293012+0000","mapping_epoch":292,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":293,"parent":"0.0","parent_split_bits":0,"last_scrub":"414'18","last_scrub_stamp":"2024-09-10T19:33:13.285004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:33:13.285004+0000","objects_scrubbed":3,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:34:24.424743+0000","scrub_duration":0.0021640850000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":27,"num_read_kb":24,"num_write":18,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":524,"reported_epoch":460,"state":"active+clean","last_fresh":"2024-09-10T19:33:23.469508+0000","last_change":"2024-09-10T19:33:21.508895+0000","last_active":"2024-09-10T19:33:23.469508+0000","last_peered":"2024-09-10T19:33:23.469508+0000","last_clean":"2024-09-10T19:33:23.469508+0000","last_became_active":"2024-09-10T19:24:01.978215+0000","last_became_peered":"2024-09-10T19:24:01.978215+0000","last_unstale":"2024-09-10T19:33:23.469508+0000","last_undegraded":"2024-09-10T19:33:23.469508+0000","last_fullsized":"2024-09-10T19:33:23.469508+0000","mapping_epoch":292,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":293,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:33:21.508825+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:33:21.508825+0000","last_clean_scrub_stamp":"2024-09-10T19:33:21.508825+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:34:47.274336+0000","scrub_duration":0.0019394900000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":525,"reported_epoch":460,"state":"active+clean","last_fresh":"2024-09-10T19:33:23.916376+0000","last_change":"2024-09-10T19:32:26.991373+0000","last_active":"2024-09-10T19:33:23.916376+0000","last_peered":"2024-09-10T19:33:23.916376+0000","last_clean":"2024-09-10T19:33:23.916376+0000","last_became_active":"2024-09-10T19:27:53.010120+0000","last_became_peered":"2024-09-10T19:27:53.010120+0000","last_unstale":"2024-09-10T19:33:23.916376+0000","last_undegraded":"2024-09-10T19:33:23.916376+0000","last_fullsized":"2024-09-10T19:33:23.916376+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:32:26.991302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:32:26.991302+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:33:54.243229+0000","scrub_duration":0.001732419,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"414'35","reported_seq":612,"reported_epoch":460,"state":"active+clean","last_fresh":"2024-09-10T19:33:23.293473+0000","last_change":"2024-09-10T19:32:23.292438+0000","last_active":"2024-09-10T19:33:23.293473+0000","last_peered":"2024-09-10T19:33:23.293473+0000","last_clean":"2024-09-10T19:33:23.293473+0000","last_became_active":"2024-09-10T19:24:01.971173+0000","last_became_peered":"2024-09-10T19:24:01.971173+0000","last_unstale":"2024-09-10T19:33:23.293473+0000","last_undegraded":"2024-09-10T19:33:23.293473+0000","last_fullsized":"2024-09-10T19:33:23.293473+0000","mapping_epoch":292,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":293,"parent":"0.0","parent_split_bits":0,"last_scrub":"414'35","last_scrub_stamp":"2024-09-10T19:32:23.292377+0000","last_deep_scrub":"80'15","last_deep_scrub_stamp":"2024-09-10T19:11:41.719310+0000","last_clean_scrub_stamp":"2024-09-10T19:32:23.292377+0000","objects_scrubbed":5,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:33:39.993723+0000","scrub_duration":0.0040992260000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":45,"num_read_kb":40,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":511,"reported_epoch":461,"state":"active+clean","last_fresh":"2024-09-10T19:33:30.500153+0000","last_change":"2024-09-10T19:33:30.500120+0000","last_active":"2024-09-10T19:33:30.500153+0000","last_peered":"2024-09-10T19:33:30.500153+0000","last_clean":"2024-09-10T19:33:30.500153+0000","last_became_active":"2024-09-10T19:10:07.685870+0000","last_became_peered":"2024-09-10T19:10:07.685870+0000","last_unstale":"2024-09-10T19:33:30.500153+0000","last_undegraded":"2024-09-10T19:33:30.500153+0000","last_fullsized":"2024-09-10T19:33:30.500153+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:33:30.500054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:33:30.500054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:34:36.561761+0000","scrub_duration":0.0014094070000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"372'16","reported_seq":850,"reported_epoch":460,"state":"active+clean","last_fresh":"2024-09-10T19:33:26.004842+0000","last_change":"2024-09-10T19:32:41.509286+0000","last_active":"2024-09-10T19:33:26.004842+0000","last_peered":"2024-09-10T19:33:26.004842+0000","last_clean":"2024-09-10T19:33:26.004842+0000","last_became_active":"2024-09-10T19:27:46.703433+0000","last_became_peered":"2024-09-10T19:27:46.703433+0000","last_unstale":"2024-09-10T19:33:26.004842+0000","last_undegraded":"2024-09-10T19:33:26.004842+0000","last_fullsized":"2024-09-10T19:33:26.004842+0000","mapping_epoch":371,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":372,"parent":"0.0","parent_split_bits":0,"last_scrub":"372'16","last_scrub_stamp":"2024-09-10T19:32:41.509222+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:32:41.509222+0000","objects_scrubbed":1,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:33:51.664560+0000","scrub_duration":0.001616609,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,0],"acting":[5,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":455,"reported_epoch":460,"state":"active+clean","last_fresh":"2024-09-10T19:33:24.310155+0000","last_change":"2024-09-10T19:32:40.230900+0000","last_active":"2024-09-10T19:33:24.310155+0000","last_peered":"2024-09-10T19:33:24.310155+0000","last_clean":"2024-09-10T19:33:24.310155+0000","last_became_active":"2024-09-10T19:24:23.900513+0000","last_became_peered":"2024-09-10T19:24:23.900513+0000","last_unstale":"2024-09-10T19:33:24.310155+0000","last_undegraded":"2024-09-10T19:33:24.310155+0000","last_fullsized":"2024-09-10T19:33:24.310155+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:32:40.230850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:32:40.230850+0000","last_clean_scrub_stamp":"2024-09-10T19:32:40.230850+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:33:56.253229+0000","scrub_duration":0.0017157940000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"414'21","reported_seq":573,"reported_epoch":460,"state":"active+clean","last_fresh":"2024-09-10T19:33:23.469506+0000","last_change":"2024-09-10T19:32:30.493699+0000","last_active":"2024-09-10T19:33:23.469506+0000","last_peered":"2024-09-10T19:33:23.469506+0000","last_clean":"2024-09-10T19:33:23.469506+0000","last_became_active":"2024-09-10T19:16:24.859740+0000","last_became_peered":"2024-09-10T19:16:24.859740+0000","last_unstale":"2024-09-10T19:33:23.469506+0000","last_undegraded":"2024-09-10T19:33:23.469506+0000","last_fullsized":"2024-09-10T19:33:23.469506+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"414'21","last_scrub_stamp":"2024-09-10T19:32:30.493636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:32:30.493636+0000","objects_scrubbed":4,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:33:49.084909+0000","scrub_duration":0.0021584120000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":35,"num_read_kb":29,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":49,"reported_epoch":460,"state":"active+clean","last_fresh":"2024-09-10T19:33:23.293692+0000","last_change":"2024-09-10T19:32:55.441239+0000","last_active":"2024-09-10T19:33:23.293692+0000","last_peered":"2024-09-10T19:33:23.293692+0000","last_clean":"2024-09-10T19:33:23.293692+0000","last_became_active":"2024-09-10T19:31:33.341283+0000","last_became_peered":"2024-09-10T19:31:33.341283+0000","last_unstale":"2024-09-10T19:33:23.293692+0000","last_undegraded":"2024-09-10T19:33:23.293692+0000","last_fullsized":"2024-09-10T19:33:23.293692+0000","mapping_epoch":431,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":432,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:32:55.441184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:32:55.441184+0000","last_clean_scrub_stamp":"2024-09-10T19:32:55.441184+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:34:11.325146+0000","scrub_duration":0.001586613,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.14","version":"432'12","reported_seq":828,"reported_epoch":460,"state":"active+clean","last_fresh":"2024-09-10T19:33:26.004650+0000","last_change":"2024-09-10T19:32:30.209134+0000","last_active":"2024-09-10T19:33:26.004650+0000","last_peered":"2024-09-10T19:33:26.004650+0000","last_clean":"2024-09-10T19:33:26.004650+0000","last_became_active":"2024-09-10T19:31:33.341212+0000","last_became_peered":"2024-09-10T19:31:33.341212+0000","last_unstale":"2024-09-10T19:33:26.004650+0000","last_undegraded":"2024-09-10T19:33:26.004650+0000","last_fullsized":"2024-09-10T19:33:26.004650+0000","mapping_epoch":431,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":432,"parent":"0.0","parent_split_bits":0,"last_scrub":"432'12","last_scrub_stamp":"2024-09-10T19:32:30.209076+0000","last_deep_scrub":"297'11","last_deep_scrub_stamp":"2024-09-10T19:24:13.318130+0000","last_clean_scrub_stamp":"2024-09-10T19:32:30.209076+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:33:58.844449+0000","scrub_duration":0.0033330090000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":524,"reported_epoch":460,"state":"active+clean","last_fresh":"2024-09-10T19:33:23.293507+0000","last_change":"2024-09-10T19:32:29.848499+0000","last_active":"2024-09-10T19:33:23.293507+0000","last_peered":"2024-09-10T19:33:23.293507+0000","last_clean":"2024-09-10T19:33:23.293507+0000","last_became_active":"2024-09-10T19:31:33.337440+0000","last_became_peered":"2024-09-10T19:31:33.337440+0000","last_unstale":"2024-09-10T19:33:23.293507+0000","last_undegraded":"2024-09-10T19:33:23.293507+0000","last_fullsized":"2024-09-10T19:33:23.293507+0000","mapping_epoch":431,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":432,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:32:29.848435+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:32:29.848435+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:33:51.895728+0000","scrub_duration":0.001811006,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.12","version":"414'25","reported_seq":321,"reported_epoch":460,"state":"active+clean","last_fresh":"2024-09-10T19:33:24.312008+0000","last_change":"2024-09-10T19:33:19.255892+0000","last_active":"2024-09-10T19:33:24.312008+0000","last_peered":"2024-09-10T19:33:24.312008+0000","last_clean":"2024-09-10T19:33:24.312008+0000","last_became_active":"2024-09-10T19:31:33.342056+0000","last_became_peered":"2024-09-10T19:31:33.342056+0000","last_unstale":"2024-09-10T19:33:24.312008+0000","last_undegraded":"2024-09-10T19:33:24.312008+0000","last_fullsized":"2024-09-10T19:33:24.312008+0000","mapping_epoch":431,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":432,"parent":"0.0","parent_split_bits":0,"last_scrub":"414'25","last_scrub_stamp":"2024-09-10T19:33:19.255826+0000","last_deep_scrub":"79'13","last_deep_scrub_stamp":"2024-09-10T19:11:22.072400+0000","last_clean_scrub_stamp":"2024-09-10T19:33:19.255826+0000","objects_scrubbed":4,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:34:20.132738+0000","scrub_duration":0.001917565,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":34,"num_read_kb":28,"num_write":22,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"461'2","reported_seq":151,"reported_epoch":461,"state":"active+clean","last_fresh":"2024-09-10T19:33:29.381010+0000","last_change":"2024-09-10T19:33:29.380973+0000","last_active":"2024-09-10T19:33:29.381010+0000","last_peered":"2024-09-10T19:33:29.381010+0000","last_clean":"2024-09-10T19:33:29.381010+0000","last_became_active":"2024-09-10T19:27:53.018470+0000","last_became_peered":"2024-09-10T19:27:53.018470+0000","last_unstale":"2024-09-10T19:33:29.381010+0000","last_undegraded":"2024-09-10T19:33:29.381010+0000","last_fullsized":"2024-09-10T19:33:29.381010+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"461'2","last_scrub_stamp":"2024-09-10T19:33:29.380907+0000","last_deep_scrub":"461'2","last_deep_scrub_stamp":"2024-09-10T19:33:29.380907+0000","last_clean_scrub_stamp":"2024-09-10T19:33:29.380907+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:34:34.962607+0000","scrub_duration":0.0027239270000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":362,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":724,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,5],"acting":[6,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":535,"reported_epoch":460,"state":"active+clean","last_fresh":"2024-09-10T19:33:23.293663+0000","last_change":"2024-09-10T19:32:31.177171+0000","last_active":"2024-09-10T19:33:23.293663+0000","last_peered":"2024-09-10T19:33:23.293663+0000","last_clean":"2024-09-10T19:33:23.293663+0000","last_became_active":"2024-09-10T19:24:24.923804+0000","last_became_peered":"2024-09-10T19:24:24.923804+0000","last_unstale":"2024-09-10T19:33:23.293663+0000","last_undegraded":"2024-09-10T19:33:23.293663+0000","last_fullsized":"2024-09-10T19:33:23.293663+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:32:31.177118+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:32:31.177118+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:33:44.531588+0000","scrub_duration":0.001184212,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":474,"reported_epoch":460,"state":"active+clean","last_fresh":"2024-09-10T19:33:24.308905+0000","last_change":"2024-09-10T19:32:50.268020+0000","last_active":"2024-09-10T19:33:24.308905+0000","last_peered":"2024-09-10T19:33:24.308905+0000","last_clean":"2024-09-10T19:33:24.308905+0000","last_became_active":"2024-09-10T19:24:24.916105+0000","last_became_peered":"2024-09-10T19:24:24.916105+0000","last_unstale":"2024-09-10T19:33:24.308905+0000","last_undegraded":"2024-09-10T19:33:24.308905+0000","last_fullsized":"2024-09-10T19:33:24.308905+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:32:50.267968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:25:27.412579+0000","last_clean_scrub_stamp":"2024-09-10T19:32:50.267968+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:34:00.943563+0000","scrub_duration":0.0020030769999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"414'27","reported_seq":639,"reported_epoch":460,"state":"active+clean","last_fresh":"2024-09-10T19:33:23.292949+0000","last_change":"2024-09-10T19:32:19.738922+0000","last_active":"2024-09-10T19:33:23.292949+0000","last_peered":"2024-09-10T19:33:23.292949+0000","last_clean":"2024-09-10T19:33:23.292949+0000","last_became_active":"2024-09-10T19:27:53.016518+0000","last_became_peered":"2024-09-10T19:27:53.016518+0000","last_unstale":"2024-09-10T19:33:23.292949+0000","last_undegraded":"2024-09-10T19:33:23.292949+0000","last_fullsized":"2024-09-10T19:33:23.292949+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"414'27","last_scrub_stamp":"2024-09-10T19:32:19.738852+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:32:19.738852+0000","objects_scrubbed":5,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:33:33.863400+0000","scrub_duration":0.0018747530000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":37,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":242,"reported_epoch":460,"state":"active+clean","last_fresh":"2024-09-10T19:33:24.312331+0000","last_change":"2024-09-10T19:32:38.297715+0000","last_active":"2024-09-10T19:33:24.312331+0000","last_peered":"2024-09-10T19:33:24.312331+0000","last_clean":"2024-09-10T19:33:24.312331+0000","last_became_active":"2024-09-10T19:24:24.909540+0000","last_became_peered":"2024-09-10T19:24:24.909540+0000","last_unstale":"2024-09-10T19:33:24.312331+0000","last_undegraded":"2024-09-10T19:33:24.312331+0000","last_fullsized":"2024-09-10T19:33:24.312331+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:32:38.297639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:32:38.297639+0000","last_clean_scrub_stamp":"2024-09-10T19:32:38.297639+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:33:56.775419+0000","scrub_duration":0.0020595330000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":218,"reported_epoch":460,"state":"active+clean","last_fresh":"2024-09-10T19:33:24.312084+0000","last_change":"2024-09-10T19:32:31.379520+0000","last_active":"2024-09-10T19:33:24.312084+0000","last_peered":"2024-09-10T19:33:24.312084+0000","last_clean":"2024-09-10T19:33:24.312084+0000","last_became_active":"2024-09-10T19:24:01.975520+0000","last_became_peered":"2024-09-10T19:24:01.975520+0000","last_unstale":"2024-09-10T19:33:24.312084+0000","last_undegraded":"2024-09-10T19:33:24.312084+0000","last_fullsized":"2024-09-10T19:33:24.312084+0000","mapping_epoch":292,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":293,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:32:31.379466+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:32:31.379466+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:33:39.186340+0000","scrub_duration":0.001424673,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":502,"reported_epoch":460,"state":"active+clean","last_fresh":"2024-09-10T19:33:23.294437+0000","last_change":"2024-09-10T19:33:06.160552+0000","last_active":"2024-09-10T19:33:23.294437+0000","last_peered":"2024-09-10T19:33:23.294437+0000","last_clean":"2024-09-10T19:33:23.294437+0000","last_became_active":"2024-09-10T19:24:23.907136+0000","last_became_peered":"2024-09-10T19:24:23.907136+0000","last_unstale":"2024-09-10T19:33:23.294437+0000","last_undegraded":"2024-09-10T19:33:23.294437+0000","last_fullsized":"2024-09-10T19:33:23.294437+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:33:06.160473+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:33:06.160473+0000","last_clean_scrub_stamp":"2024-09-10T19:33:06.160473+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:34:13.264507+0000","scrub_duration":0.0015304590000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"414'14","reported_seq":590,"reported_epoch":460,"state":"active+clean","last_fresh":"2024-09-10T19:33:23.294194+0000","last_change":"2024-09-10T19:33:17.354210+0000","last_active":"2024-09-10T19:33:23.294194+0000","last_peered":"2024-09-10T19:33:23.294194+0000","last_clean":"2024-09-10T19:33:23.294194+0000","last_became_active":"2024-09-10T19:27:53.024782+0000","last_became_peered":"2024-09-10T19:27:53.024782+0000","last_unstale":"2024-09-10T19:33:23.294194+0000","last_undegraded":"2024-09-10T19:33:23.294194+0000","last_fullsized":"2024-09-10T19:33:23.294194+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"414'14","last_scrub_stamp":"2024-09-10T19:33:17.354110+0000","last_deep_scrub":"222'6","last_deep_scrub_stamp":"2024-09-10T19:19:30.328841+0000","last_clean_scrub_stamp":"2024-09-10T19:33:17.354110+0000","objects_scrubbed":2,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:34:27.169214+0000","scrub_duration":0.0015824319999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":18,"num_read_kb":16,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":561,"reported_epoch":460,"state":"active+clean","last_fresh":"2024-09-10T19:33:23.469600+0000","last_change":"2024-09-10T19:32:50.571952+0000","last_active":"2024-09-10T19:33:23.469600+0000","last_peered":"2024-09-10T19:33:23.469600+0000","last_clean":"2024-09-10T19:33:23.469600+0000","last_became_active":"2024-09-10T19:31:33.338139+0000","last_became_peered":"2024-09-10T19:31:33.338139+0000","last_unstale":"2024-09-10T19:33:23.469600+0000","last_undegraded":"2024-09-10T19:33:23.469600+0000","last_fullsized":"2024-09-10T19:33:23.469600+0000","mapping_epoch":431,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":432,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:32:50.571844+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:32:50.571844+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:34:15.273985+0000","scrub_duration":0.00161389,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":146,"reported_epoch":460,"state":"active+clean","last_fresh":"2024-09-10T19:33:24.311378+0000","last_change":"2024-09-10T19:33:13.533511+0000","last_active":"2024-09-10T19:33:24.311378+0000","last_peered":"2024-09-10T19:33:24.311378+0000","last_clean":"2024-09-10T19:33:24.311378+0000","last_became_active":"2024-09-10T19:27:53.003591+0000","last_became_peered":"2024-09-10T19:27:53.003591+0000","last_unstale":"2024-09-10T19:33:24.311378+0000","last_undegraded":"2024-09-10T19:33:24.311378+0000","last_fullsized":"2024-09-10T19:33:24.311378+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:33:13.533438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:11:31.837944+0000","last_clean_scrub_stamp":"2024-09-10T19:33:13.533438+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:34:36.324746+0000","scrub_duration":0.001641984,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":535,"reported_epoch":460,"state":"active+clean","last_fresh":"2024-09-10T19:33:23.915990+0000","last_change":"2024-09-10T19:32:34.017145+0000","last_active":"2024-09-10T19:33:23.915990+0000","last_peered":"2024-09-10T19:33:23.915990+0000","last_clean":"2024-09-10T19:33:23.915990+0000","last_became_active":"2024-09-10T19:31:33.342711+0000","last_became_peered":"2024-09-10T19:31:33.342711+0000","last_unstale":"2024-09-10T19:33:23.915990+0000","last_undegraded":"2024-09-10T19:33:23.915990+0000","last_fullsized":"2024-09-10T19:33:23.915990+0000","mapping_epoch":431,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":432,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:32:34.017080+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:32:34.017080+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:33:44.281178+0000","scrub_duration":0.0015294970000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.11","version":"414'27","reported_seq":347,"reported_epoch":460,"state":"active+clean","last_fresh":"2024-09-10T19:33:23.294546+0000","last_change":"2024-09-10T19:32:49.022254+0000","last_active":"2024-09-10T19:33:23.294546+0000","last_peered":"2024-09-10T19:33:23.294546+0000","last_clean":"2024-09-10T19:33:23.294546+0000","last_became_active":"2024-09-10T19:27:53.018213+0000","last_became_peered":"2024-09-10T19:27:53.018213+0000","last_unstale":"2024-09-10T19:33:23.294546+0000","last_undegraded":"2024-09-10T19:33:23.294546+0000","last_fullsized":"2024-09-10T19:33:23.294546+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"414'27","last_scrub_stamp":"2024-09-10T19:32:49.022134+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:08.671214+0000","last_clean_scrub_stamp":"2024-09-10T19:32:49.022134+0000","objects_scrubbed":5,"log_size":27,"log_dups_size":0,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:34:14.333132+0000","scrub_duration":0.0026081020000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":37,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":146,"reported_epoch":460,"state":"active+clean","last_fresh":"2024-09-10T19:33:24.311779+0000","last_change":"2024-09-10T19:32:35.608043+0000","last_active":"2024-09-10T19:33:24.311779+0000","last_peered":"2024-09-10T19:33:24.311779+0000","last_clean":"2024-09-10T19:33:24.311779+0000","last_became_active":"2024-09-10T19:31:33.338443+0000","last_became_peered":"2024-09-10T19:31:33.338443+0000","last_unstale":"2024-09-10T19:33:24.311779+0000","last_undegraded":"2024-09-10T19:33:24.311779+0000","last_fullsized":"2024-09-10T19:33:24.311779+0000","mapping_epoch":431,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":432,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:32:35.607968+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:13:05.876655+0000","last_clean_scrub_stamp":"2024-09-10T19:32:35.607968+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:33:46.380646+0000","scrub_duration":0.0015078699999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":527,"reported_epoch":460,"state":"active+clean","last_fresh":"2024-09-10T19:33:23.915837+0000","last_change":"2024-09-10T19:33:08.807323+0000","last_active":"2024-09-10T19:33:23.915837+0000","last_peered":"2024-09-10T19:33:23.915837+0000","last_clean":"2024-09-10T19:33:23.915837+0000","last_became_active":"2024-09-10T19:27:53.008444+0000","last_became_peered":"2024-09-10T19:27:53.008444+0000","last_unstale":"2024-09-10T19:33:23.915837+0000","last_undegraded":"2024-09-10T19:33:23.915837+0000","last_fullsized":"2024-09-10T19:33:23.915837+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:33:08.807262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:11:57.112092+0000","last_clean_scrub_stamp":"2024-09-10T19:33:08.807262+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:34:30.759357+0000","scrub_duration":0.001404708,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":522,"reported_epoch":460,"state":"active+clean","last_fresh":"2024-09-10T19:33:23.293590+0000","last_change":"2024-09-10T19:33:01.174031+0000","last_active":"2024-09-10T19:33:23.293590+0000","last_peered":"2024-09-10T19:33:23.293590+0000","last_clean":"2024-09-10T19:33:23.293590+0000","last_became_active":"2024-09-10T19:27:53.025650+0000","last_became_peered":"2024-09-10T19:27:53.025650+0000","last_unstale":"2024-09-10T19:33:23.293590+0000","last_undegraded":"2024-09-10T19:33:23.293590+0000","last_fullsized":"2024-09-10T19:33:23.293590+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:33:01.173976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:26:29.959202+0000","last_clean_scrub_stamp":"2024-09-10T19:33:01.173976+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:34:27.056442+0000","scrub_duration":0.0015841900000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"80'2","reported_seq":535,"reported_epoch":460,"state":"active+clean","last_fresh":"2024-09-10T19:33:23.916562+0000","last_change":"2024-09-10T19:32:05.196679+0000","last_active":"2024-09-10T19:33:23.916562+0000","last_peered":"2024-09-10T19:33:23.916562+0000","last_clean":"2024-09-10T19:33:23.916562+0000","last_became_active":"2024-09-10T19:31:33.343250+0000","last_became_peered":"2024-09-10T19:31:33.343250+0000","last_unstale":"2024-09-10T19:33:23.916562+0000","last_undegraded":"2024-09-10T19:33:23.916562+0000","last_fullsized":"2024-09-10T19:33:23.916562+0000","mapping_epoch":431,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":432,"parent":"0.0","parent_split_bits":0,"last_scrub":"80'2","last_scrub_stamp":"2024-09-10T19:32:05.196606+0000","last_deep_scrub":"80'2","last_deep_scrub_stamp":"2024-09-10T19:11:30.272043+0000","last_clean_scrub_stamp":"2024-09-10T19:32:05.196606+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":0.0020235969999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":369,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":738,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":518,"reported_epoch":460,"state":"active+clean","last_fresh":"2024-09-10T19:33:23.468804+0000","last_change":"2024-09-10T19:32:27.449151+0000","last_active":"2024-09-10T19:33:23.468804+0000","last_peered":"2024-09-10T19:33:23.468804+0000","last_clean":"2024-09-10T19:33:23.468804+0000","last_became_active":"2024-09-10T19:24:23.906812+0000","last_became_peered":"2024-09-10T19:24:23.906812+0000","last_unstale":"2024-09-10T19:33:23.468804+0000","last_undegraded":"2024-09-10T19:33:23.468804+0000","last_fullsized":"2024-09-10T19:33:23.468804+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:32:27.449085+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:32:27.449085+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:33:51.226533+0000","scrub_duration":0.001431234,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,0],"acting":[5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1e","version":"414'35","reported_seq":509,"reported_epoch":460,"state":"active+clean","last_fresh":"2024-09-10T19:33:24.310059+0000","last_change":"2024-09-10T19:32:18.102686+0000","last_active":"2024-09-10T19:33:24.310059+0000","last_peered":"2024-09-10T19:33:24.310059+0000","last_clean":"2024-09-10T19:33:24.310059+0000","last_became_active":"2024-09-10T19:24:23.899507+0000","last_became_peered":"2024-09-10T19:24:23.899507+0000","last_unstale":"2024-09-10T19:33:24.310059+0000","last_undegraded":"2024-09-10T19:33:24.310059+0000","last_fullsized":"2024-09-10T19:33:24.310059+0000","mapping_epoch":301,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":302,"parent":"0.0","parent_split_bits":0,"last_scrub":"414'35","last_scrub_stamp":"2024-09-10T19:32:18.102631+0000","last_deep_scrub":"414'35","last_deep_scrub_stamp":"2024-09-10T19:32:18.102631+0000","last_clean_scrub_stamp":"2024-09-10T19:32:18.102631+0000","objects_scrubbed":5,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:33:44.786784+0000","scrub_duration":0.001961431,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":39,"num_write":30,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1f","version":"376'10","reported_seq":207,"reported_epoch":461,"state":"active+clean","last_fresh":"2024-09-10T19:33:26.408491+0000","last_change":"2024-09-10T19:33:26.408431+0000","last_active":"2024-09-10T19:33:26.408491+0000","last_peered":"2024-09-10T19:33:26.408491+0000","last_clean":"2024-09-10T19:33:26.408491+0000","last_became_active":"2024-09-10T19:27:53.003676+0000","last_became_peered":"2024-09-10T19:27:53.003676+0000","last_unstale":"2024-09-10T19:33:26.408491+0000","last_undegraded":"2024-09-10T19:33:26.408491+0000","last_fullsized":"2024-09-10T19:33:26.408491+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"376'10","last_scrub_stamp":"2024-09-10T19:33:26.408355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:33:26.408355+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:34:28.017683+0000","scrub_duration":0.0049084339999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":228,"reported_epoch":460,"state":"active+clean","last_fresh":"2024-09-10T19:33:24.312406+0000","last_change":"2024-09-10T19:33:18.239503+0000","last_active":"2024-09-10T19:33:24.312406+0000","last_peered":"2024-09-10T19:33:24.312406+0000","last_clean":"2024-09-10T19:33:24.312406+0000","last_became_active":"2024-09-10T19:24:01.980391+0000","last_became_peered":"2024-09-10T19:24:01.980391+0000","last_unstale":"2024-09-10T19:33:24.312406+0000","last_undegraded":"2024-09-10T19:33:24.312406+0000","last_fullsized":"2024-09-10T19:33:24.312406+0000","mapping_epoch":292,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":293,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:33:18.239445+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:28:20.514115+0000","last_clean_scrub_stamp":"2024-09-10T19:33:18.239445+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:34:32.561716+0000","scrub_duration":0.0017389860000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":538,"reported_epoch":460,"state":"active+clean","last_fresh":"2024-09-10T19:33:23.294239+0000","last_change":"2024-09-10T19:33:16.328234+0000","last_active":"2024-09-10T19:33:23.294239+0000","last_peered":"2024-09-10T19:33:23.294239+0000","last_clean":"2024-09-10T19:33:23.294239+0000","last_became_active":"2024-09-10T19:27:53.017153+0000","last_became_peered":"2024-09-10T19:27:53.017153+0000","last_unstale":"2024-09-10T19:33:23.294239+0000","last_undegraded":"2024-09-10T19:33:23.294239+0000","last_fullsized":"2024-09-10T19:33:23.294239+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:33:16.328178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:33:16.328178+0000","last_clean_scrub_stamp":"2024-09-10T19:33:16.328178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:34:29.545042+0000","scrub_duration":0.001686089,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,0],"acting":[3,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"414'32","reported_seq":160,"reported_epoch":460,"state":"active+clean","last_fresh":"2024-09-10T19:33:24.311682+0000","last_change":"2024-09-10T19:32:39.649686+0000","last_active":"2024-09-10T19:33:24.311682+0000","last_peered":"2024-09-10T19:33:24.311682+0000","last_clean":"2024-09-10T19:33:24.311682+0000","last_became_active":"2024-09-10T19:27:53.008233+0000","last_became_peered":"2024-09-10T19:27:53.008233+0000","last_unstale":"2024-09-10T19:33:24.311682+0000","last_undegraded":"2024-09-10T19:33:24.311682+0000","last_fullsized":"2024-09-10T19:33:24.311682+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"414'32","last_scrub_stamp":"2024-09-10T19:32:39.649618+0000","last_deep_scrub":"414'32","last_deep_scrub_stamp":"2024-09-10T19:32:39.649618+0000","last_clean_scrub_stamp":"2024-09-10T19:32:39.649618+0000","objects_scrubbed":5,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:33:54.043975+0000","scrub_duration":0.0042512590000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":36,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":456,"reported_epoch":460,"state":"active+clean","last_fresh":"2024-09-10T19:33:24.310012+0000","last_change":"2024-09-10T19:33:18.273197+0000","last_active":"2024-09-10T19:33:24.310012+0000","last_peered":"2024-09-10T19:33:24.310012+0000","last_clean":"2024-09-10T19:33:24.310012+0000","last_became_active":"2024-09-10T19:31:33.334922+0000","last_became_peered":"2024-09-10T19:31:33.334922+0000","last_unstale":"2024-09-10T19:33:24.310012+0000","last_undegraded":"2024-09-10T19:33:24.310012+0000","last_fullsized":"2024-09-10T19:33:24.310012+0000","mapping_epoch":431,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":432,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:33:18.273126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:28:19.464426+0000","last_clean_scrub_stamp":"2024-09-10T19:33:18.273126+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:34:32.331325+0000","scrub_duration":0.001575543,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":534,"reported_epoch":461,"state":"active+clean","last_fresh":"2024-09-10T19:33:30.064725+0000","last_change":"2024-09-10T19:33:30.064664+0000","last_active":"2024-09-10T19:33:30.064725+0000","last_peered":"2024-09-10T19:33:30.064725+0000","last_clean":"2024-09-10T19:33:30.064725+0000","last_became_active":"2024-09-10T19:27:53.011336+0000","last_became_peered":"2024-09-10T19:27:53.011336+0000","last_unstale":"2024-09-10T19:33:30.064725+0000","last_undegraded":"2024-09-10T19:33:30.064725+0000","last_fullsized":"2024-09-10T19:33:30.064725+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:33:30.064593+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:12.693598+0000","last_clean_scrub_stamp":"2024-09-10T19:33:30.064593+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:34:44.493898+0000","scrub_duration":0.0015691170000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":136,"reported_epoch":460,"state":"active+clean","last_fresh":"2024-09-10T19:33:24.311278+0000","last_change":"2024-09-10T19:32:50.572647+0000","last_active":"2024-09-10T19:33:24.311278+0000","last_peered":"2024-09-10T19:33:24.311278+0000","last_clean":"2024-09-10T19:33:24.311278+0000","last_became_active":"2024-09-10T19:27:53.002313+0000","last_became_peered":"2024-09-10T19:27:53.002313+0000","last_unstale":"2024-09-10T19:33:24.311278+0000","last_undegraded":"2024-09-10T19:33:24.311278+0000","last_fullsized":"2024-09-10T19:33:24.311278+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:32:50.572581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:32:50.572581+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:34:08.318842+0000","scrub_duration":0.0016561519999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1c","version":"414'46","reported_seq":75,"reported_epoch":460,"state":"active+clean","last_fresh":"2024-09-10T19:33:23.293697+0000","last_change":"2024-09-10T19:33:10.707829+0000","last_active":"2024-09-10T19:33:23.293697+0000","last_peered":"2024-09-10T19:33:23.293697+0000","last_clean":"2024-09-10T19:33:23.293697+0000","last_became_active":"2024-09-10T19:31:33.344217+0000","last_became_peered":"2024-09-10T19:31:33.344217+0000","last_unstale":"2024-09-10T19:33:23.293697+0000","last_undegraded":"2024-09-10T19:33:23.293697+0000","last_fullsized":"2024-09-10T19:33:23.293697+0000","mapping_epoch":431,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":432,"parent":"0.0","parent_split_bits":0,"last_scrub":"414'46","last_scrub_stamp":"2024-09-10T19:33:10.707736+0000","last_deep_scrub":"78'22","last_deep_scrub_stamp":"2024-09-10T19:11:21.118937+0000","last_clean_scrub_stamp":"2024-09-10T19:33:10.707736+0000","objects_scrubbed":7,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:34:31.395614+0000","scrub_duration":0.0018651609999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":61,"num_read_kb":52,"num_write":40,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":242,"reported_epoch":460,"state":"active+clean","last_fresh":"2024-09-10T19:33:24.312443+0000","last_change":"2024-09-10T19:33:03.280294+0000","last_active":"2024-09-10T19:33:24.312443+0000","last_peered":"2024-09-10T19:33:24.312443+0000","last_clean":"2024-09-10T19:33:24.312443+0000","last_became_active":"2024-09-10T19:24:24.887205+0000","last_became_peered":"2024-09-10T19:24:24.887205+0000","last_unstale":"2024-09-10T19:33:24.312443+0000","last_undegraded":"2024-09-10T19:33:24.312443+0000","last_fullsized":"2024-09-10T19:33:24.312443+0000","mapping_epoch":302,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":303,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:33:03.280233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:10.676876+0000","last_clean_scrub_stamp":"2024-09-10T19:33:03.280233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:34:22.114440+0000","scrub_duration":0.0016609179999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":518,"reported_epoch":460,"state":"active+clean","last_fresh":"2024-09-10T19:33:23.292712+0000","last_change":"2024-09-10T19:32:43.974114+0000","last_active":"2024-09-10T19:33:23.292712+0000","last_peered":"2024-09-10T19:33:23.292712+0000","last_clean":"2024-09-10T19:33:23.292712+0000","last_became_active":"2024-09-10T19:27:53.026591+0000","last_became_peered":"2024-09-10T19:27:53.026591+0000","last_unstale":"2024-09-10T19:33:23.292712+0000","last_undegraded":"2024-09-10T19:33:23.292712+0000","last_fullsized":"2024-09-10T19:33:23.292712+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:32:43.974059+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:32:43.974059+0000","last_clean_scrub_stamp":"2024-09-10T19:32:43.974059+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:33:59.177332+0000","scrub_duration":0.0021937829999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":521,"reported_epoch":460,"state":"active+clean","last_fresh":"2024-09-10T19:33:23.293794+0000","last_change":"2024-09-10T19:32:18.334973+0000","last_active":"2024-09-10T19:33:23.293794+0000","last_peered":"2024-09-10T19:33:23.293794+0000","last_clean":"2024-09-10T19:33:23.293794+0000","last_became_active":"2024-09-10T19:27:53.018726+0000","last_became_peered":"2024-09-10T19:27:53.018726+0000","last_unstale":"2024-09-10T19:33:23.293794+0000","last_undegraded":"2024-09-10T19:33:23.293794+0000","last_fullsized":"2024-09-10T19:33:23.293794+0000","mapping_epoch":374,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":375,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:32:18.334866+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:10:06.664709+0000","last_clean_scrub_stamp":"2024-09-10T19:32:18.334866+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:33:34.896293+0000","scrub_duration":0.001806394,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1d","version":"414'29","reported_seq":586,"reported_epoch":460,"state":"active+clean","last_fresh":"2024-09-10T19:33:23.292703+0000","last_change":"2024-09-10T19:32:27.254562+0000","last_active":"2024-09-10T19:33:23.292703+0000","last_peered":"2024-09-10T19:33:23.292703+0000","last_clean":"2024-09-10T19:33:23.292703+0000","last_became_active":"2024-09-10T19:24:01.976664+0000","last_became_peered":"2024-09-10T19:24:01.976664+0000","last_unstale":"2024-09-10T19:33:23.292703+0000","last_undegraded":"2024-09-10T19:33:23.292703+0000","last_fullsized":"2024-09-10T19:33:23.292703+0000","mapping_epoch":292,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":293,"parent":"0.0","parent_split_bits":0,"last_scrub":"414'29","last_scrub_stamp":"2024-09-10T19:32:27.254505+0000","last_deep_scrub":"250'23","last_deep_scrub_stamp":"2024-09-10T19:20:55.582173+0000","last_clean_scrub_stamp":"2024-09-10T19:32:27.254505+0000","objects_scrubbed":5,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:33:41.635783+0000","scrub_duration":0.001812255,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":26,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":526,"reported_epoch":460,"state":"active+clean","last_fresh":"2024-09-10T19:33:23.293164+0000","last_change":"2024-09-10T19:32:00.743872+0000","last_active":"2024-09-10T19:33:23.293164+0000","last_peered":"2024-09-10T19:33:23.293164+0000","last_clean":"2024-09-10T19:33:23.293164+0000","last_became_active":"2024-09-10T19:31:33.337386+0000","last_became_peered":"2024-09-10T19:31:33.337386+0000","last_unstale":"2024-09-10T19:33:23.293164+0000","last_undegraded":"2024-09-10T19:33:23.293164+0000","last_fullsized":"2024-09-10T19:33:23.293164+0000","mapping_epoch":431,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":432,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T19:32:00.743803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T19:30:39.493257+0000","last_clean_scrub_stamp":"2024-09-10T19:32:00.743803+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-10T19:33:30.335014+0000","scrub_duration":0.001558076,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":382,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":751,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":177,"num_object_clones":0,"num_object_copies":531,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":177,"num_whiteouts":0,"num_read":1541,"num_read_kb":1298,"num_write":989,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":496,"num_bytes_recovered":9644,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1090,"ondisk_log_size":1090,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":565,"num_read_kb":560,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":1983,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":11,"ondisk_log_size":11,"up":9,"acting":9,"num_store_stats":7},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":1328,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":21,"num_read_kb":21,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":724,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":3984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":788064,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":871,"ondisk_log_size":871,"up":33,"acting":33,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":54,"seq":231928234270,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1177564,"kb_used_data":1040,"kb_used_omap":1,"kb_used_meta":1176510,"kb_avail":92567588,"statfs":{"total":95995035648,"available":94789210112,"internally_reserved":0,"allocated":1064960,"data_stored":763572,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":1204746584},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 19:32:36 2024","interfaces":[{"interface":"back","average":{"1min":0.54600000000000004,"5min":0.875,"15min":0.92500000000000004},"min":{"1min":0.24199999999999999,"5min":0.19600000000000001,"15min":0.19600000000000001},"max":{"1min":0.71299999999999997,"5min":4.5220000000000002,"15min":5.0510000000000002},"last":0.627},{"interface":"front","average":{"1min":0.56799999999999995,"5min":0.94399999999999995,"15min":0.82399999999999995},"min":{"1min":0.23499999999999999,"5min":0.23400000000000001,"15min":0.21099999999999999},"max":{"1min":0.747,"5min":3.5089999999999999,"15min":4.7190000000000003},"last":0.67800000000000005}]},{"osd":1,"last update":"Tue Sep 10 19:33:03 2024","interfaces":[{"interface":"back","average":{"1min":0.53900000000000003,"5min":1.0269999999999999,"15min":1.004},"min":{"1min":0.34000000000000002,"5min":0.224,"15min":0.224},"max":{"1min":0.70599999999999996,"5min":12.773999999999999,"15min":12.773999999999999},"last":0.55500000000000005},{"interface":"front","average":{"1min":0.56200000000000006,"5min":1.04,"15min":0.94799999999999995},"min":{"1min":0.38400000000000001,"5min":0.29599999999999999,"15min":0.26400000000000001},"max":{"1min":0.76500000000000001,"5min":12.837999999999999,"15min":12.837999999999999},"last":0.64100000000000001}]},{"osd":2,"last update":"Tue Sep 10 19:33:21 2024","interfaces":[{"interface":"back","average":{"1min":0.53100000000000003,"5min":0.81399999999999995,"15min":0.90800000000000003},"min":{"1min":0.29599999999999999,"5min":0.26600000000000001,"15min":0.23000000000000001},"max":{"1min":0.80400000000000005,"5min":3.3100000000000001,"15min":4.7400000000000002},"last":0.57299999999999995},{"interface":"front","average":{"1min":0.55300000000000005,"5min":0.81499999999999995,"15min":0.90400000000000003},"min":{"1min":0.38800000000000001,"5min":0.29099999999999998,"15min":0.26300000000000001},"max":{"1min":0.754,"5min":6.1059999999999999,"15min":6.1059999999999999},"last":0.72299999999999998}]},{"osd":3,"last update":"Tue Sep 10 19:33:21 2024","interfaces":[{"interface":"back","average":{"1min":0.56299999999999994,"5min":1.2070000000000001,"15min":1.0529999999999999},"min":{"1min":0.34200000000000003,"5min":0.26300000000000001,"15min":0.20200000000000001},"max":{"1min":0.749,"5min":14.515000000000001,"15min":14.515000000000001},"last":0.61299999999999999},{"interface":"front","average":{"1min":0.57399999999999995,"5min":1.097,"15min":1.026},"min":{"1min":0.32500000000000001,"5min":0.32400000000000001,"15min":0.25700000000000001},"max":{"1min":0.80000000000000004,"5min":14.227,"15min":14.227},"last":0.69299999999999995}]},{"osd":4,"last update":"Tue Sep 10 19:33:21 2024","interfaces":[{"interface":"back","average":{"1min":0.437,"5min":0.89000000000000001,"15min":2.528},"min":{"1min":0.311,"5min":0.24299999999999999,"15min":0.222},"max":{"1min":0.56499999999999995,"5min":5.194,"15min":231.923},"last":0.52700000000000002},{"interface":"front","average":{"1min":0.502,"5min":0.85499999999999998,"15min":2.3130000000000002},"min":{"1min":0.34499999999999997,"5min":0.25600000000000001,"15min":0.23100000000000001},"max":{"1min":0.71999999999999997,"5min":6.6029999999999998,"15min":231.322},"last":0.54700000000000004}]},{"osd":5,"last update":"Tue Sep 10 19:33:21 2024","interfaces":[{"interface":"back","average":{"1min":0.5,"5min":1.341,"15min":1.6930000000000001},"min":{"1min":0.375,"5min":0.25800000000000001,"15min":0.24099999999999999},"max":{"1min":0.65000000000000002,"5min":54.441000000000003,"15min":67.382999999999996},"last":0.44400000000000001},{"interface":"front","average":{"1min":0.48999999999999999,"5min":1.633,"15min":2.0739999999999998},"min":{"1min":0.35399999999999998,"5min":0.316,"15min":0.22800000000000001},"max":{"1min":0.78400000000000003,"5min":54.756,"15min":67.75},"last":0.59399999999999997}]},{"osd":6,"last update":"Tue Sep 10 19:33:21 2024","interfaces":[{"interface":"back","average":{"1min":0.54700000000000004,"5min":1.704,"15min":2.4660000000000002},"min":{"1min":0.437,"5min":0.29799999999999999,"15min":0.28000000000000003},"max":{"1min":0.73599999999999999,"5min":47.933999999999997,"15min":181.53800000000001},"last":0.46600000000000003},{"interface":"front","average":{"1min":0.55500000000000005,"5min":1.5409999999999999,"15min":2.2040000000000002},"min":{"1min":0.42099999999999999,"5min":0.34399999999999997,"15min":0.27000000000000002},"max":{"1min":0.76900000000000002,"5min":48.021999999999998,"15min":182.05600000000001},"last":0.48499999999999999}]}]},{"osd":6,"up_from":47,"seq":201863463203,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":899664,"kb_used_data":708,"kb_used_omap":1,"kb_used_meta":898942,"kb_avail":92845488,"statfs":{"total":95995035648,"available":95073779712,"internally_reserved":0,"allocated":724992,"data_stored":435968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":920516952},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 19:32:48 2024","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":2.2210000000000001,"15min":1.9570000000000001},"min":{"1min":0.33300000000000002,"5min":0.186,"15min":0.17899999999999999},"max":{"1min":0.80400000000000005,"5min":127.395,"15min":127.395},"last":0.44700000000000001},{"interface":"front","average":{"1min":0.51300000000000001,"5min":2.254,"15min":1.9970000000000001},"min":{"1min":0.33500000000000002,"5min":0.20599999999999999,"15min":0.20200000000000001},"max":{"1min":0.68000000000000005,"5min":127.547,"15min":127.547},"last":0.45900000000000002}]},{"osd":1,"last update":"Tue Sep 10 19:33:09 2024","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":1.8540000000000001,"15min":1.238},"min":{"1min":0.46200000000000002,"5min":0.20999999999999999,"15min":0.19800000000000001},"max":{"1min":0.77900000000000003,"5min":39.484999999999999,"15min":39.484999999999999},"last":0.374},{"interface":"front","average":{"1min":0.58999999999999997,"5min":1.869,"15min":1.1799999999999999},"min":{"1min":0.41299999999999998,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":0.80300000000000005,"5min":39.472999999999999,"15min":39.472999999999999},"last":0.54000000000000004}]},{"osd":2,"last update":"Tue Sep 10 19:33:20 2024","interfaces":[{"interface":"back","average":{"1min":0.54100000000000004,"5min":0.873,"15min":0.89300000000000002},"min":{"1min":0.35799999999999998,"5min":0.24299999999999999,"15min":0.23799999999999999},"max":{"1min":0.66000000000000003,"5min":4.8330000000000002,"15min":5.9029999999999996},"last":0.39400000000000002},{"interface":"front","average":{"1min":0.53200000000000003,"5min":0.77600000000000002,"15min":0.88100000000000001},"min":{"1min":0.38500000000000001,"5min":0.28899999999999998,"15min":0.25600000000000001},"max":{"1min":0.65200000000000002,"5min":6.4560000000000004,"15min":6.4560000000000004},"last":0.48199999999999998}]},{"osd":3,"last update":"Tue Sep 10 19:33:20 2024","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":1.6859999999999999,"15min":1.3009999999999999},"min":{"1min":0.38400000000000001,"5min":0.30399999999999999,"15min":0.20399999999999999},"max":{"1min":0.749,"5min":42.929000000000002,"15min":42.929000000000002},"last":0.42099999999999999},{"interface":"front","average":{"1min":0.56799999999999995,"5min":1.742,"15min":1.2549999999999999},"min":{"1min":0.41599999999999998,"5min":0.28999999999999998,"15min":0.20499999999999999},"max":{"1min":0.76400000000000001,"5min":43.692999999999998,"15min":43.692999999999998},"last":0.55800000000000005}]},{"osd":4,"last update":"Tue Sep 10 19:33:20 2024","interfaces":[{"interface":"back","average":{"1min":0.46600000000000003,"5min":0.85899999999999999,"15min":1.3700000000000001},"min":{"1min":0.35299999999999998,"5min":0.25700000000000001,"15min":0.22},"max":{"1min":0.60399999999999998,"5min":6.9859999999999998,"15min":35.552},"last":0.434},{"interface":"front","average":{"1min":0.52600000000000002,"5min":0.85499999999999998,"15min":1.369},"min":{"1min":0.35599999999999998,"5min":0.24399999999999999,"15min":0.24099999999999999},"max":{"1min":0.70499999999999996,"5min":4.5069999999999997,"15min":35.192999999999998},"last":0.53000000000000003}]},{"osd":5,"last update":"Tue Sep 10 19:33:20 2024","interfaces":[{"interface":"back","average":{"1min":0.49399999999999999,"5min":0.79900000000000004,"15min":1.3700000000000001},"min":{"1min":0.38300000000000001,"5min":0.29699999999999999,"15min":0.26500000000000001},"max":{"1min":0.63900000000000001,"5min":7.4050000000000002,"15min":48.216000000000001},"last":0.35299999999999998},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.84999999999999998,"15min":1.5369999999999999},"min":{"1min":0.34399999999999997,"5min":0.30399999999999999,"15min":0.254},"max":{"1min":0.72799999999999998,"5min":4.3719999999999999,"15min":50.372999999999998},"last":0.5}]},{"osd":7,"last update":"Tue Sep 10 19:32:48 2024","interfaces":[{"interface":"back","average":{"1min":0.59799999999999998,"5min":0.97599999999999998,"15min":1.542},"min":{"1min":0.42299999999999999,"5min":0.28299999999999997,"15min":0.28100000000000003},"max":{"1min":0.78500000000000003,"5min":7.609,"15min":80.629999999999995},"last":0.51600000000000001},{"interface":"front","average":{"1min":0.63400000000000001,"5min":1.206,"15min":1.7989999999999999},"min":{"1min":0.439,"5min":0.29399999999999998,"15min":0.25700000000000001},"max":{"1min":0.90700000000000003,"5min":6.5659999999999998,"15min":81.088999999999999},"last":0.54900000000000004}]}]},{"osd":5,"up_from":39,"seq":167503724840,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":903392,"kb_used_data":1304,"kb_used_omap":1,"kb_used_meta":902078,"kb_avail":92841760,"statfs":{"total":95995035648,"available":95069962240,"internally_reserved":0,"allocated":1335296,"data_stored":1027476,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":923728216},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 19:32:38 2024","interfaces":[{"interface":"back","average":{"1min":0.502,"5min":0.79800000000000004,"15min":1.024},"min":{"1min":0.30299999999999999,"5min":0.223,"15min":0.187},"max":{"1min":0.75700000000000001,"5min":7.5759999999999996,"15min":7.5759999999999996},"last":0.56299999999999994},{"interface":"front","average":{"1min":0.497,"5min":0.80000000000000004,"15min":1.089},"min":{"1min":0.34100000000000003,"5min":0.20599999999999999,"15min":0.20599999999999999},"max":{"1min":0.65800000000000003,"5min":5.819,"15min":5.819},"last":0.60999999999999999}]},{"osd":1,"last update":"Tue Sep 10 19:32:54 2024","interfaces":[{"interface":"back","average":{"1min":0.51500000000000001,"5min":2.4649999999999999,"15min":1.4690000000000001},"min":{"1min":0.38100000000000001,"5min":0.247,"15min":0.214},"max":{"1min":0.69799999999999995,"5min":54.097999999999999,"15min":54.097999999999999},"last":0.54900000000000004},{"interface":"front","average":{"1min":0.57099999999999995,"5min":2.29,"15min":1.4550000000000001},"min":{"1min":0.39500000000000002,"5min":0.253,"15min":0.221},"max":{"1min":0.73999999999999999,"5min":54.337000000000003,"15min":54.337000000000003},"last":0.66100000000000003}]},{"osd":2,"last update":"Tue Sep 10 19:32:41 2024","interfaces":[{"interface":"back","average":{"1min":0.54300000000000004,"5min":0.73599999999999999,"15min":0.98599999999999999},"min":{"1min":0.34999999999999998,"5min":0.26500000000000001,"15min":0.22900000000000001},"max":{"1min":0.71699999999999997,"5min":7.4489999999999998,"15min":8.6920000000000002},"last":0.46500000000000002},{"interface":"front","average":{"1min":0.54200000000000004,"5min":0.83199999999999996,"15min":0.98499999999999999},"min":{"1min":0.38800000000000001,"5min":0.28399999999999997,"15min":0.21199999999999999},"max":{"1min":0.69599999999999995,"5min":10.045999999999999,"15min":10.045999999999999},"last":0.53300000000000003}]},{"osd":3,"last update":"Tue Sep 10 19:32:41 2024","interfaces":[{"interface":"back","average":{"1min":0.59699999999999998,"5min":2.286,"15min":1.599},"min":{"1min":0.44500000000000001,"5min":0.26600000000000001,"15min":0.19900000000000001},"max":{"1min":0.79100000000000004,"5min":54.832000000000001,"15min":54.832000000000001},"last":0.621},{"interface":"front","average":{"1min":0.57899999999999996,"5min":2.347,"15min":1.613},"min":{"1min":0.46700000000000003,"5min":0.28100000000000003,"15min":0.246},"max":{"1min":0.80600000000000005,"5min":54.850999999999999,"15min":54.850999999999999},"last":0.58299999999999996}]},{"osd":4,"last update":"Tue Sep 10 19:32:41 2024","interfaces":[{"interface":"back","average":{"1min":0.48299999999999998,"5min":1.9970000000000001,"15min":2.101},"min":{"1min":0.33400000000000002,"5min":0.252,"15min":0.252},"max":{"1min":0.65400000000000003,"5min":132.41900000000001,"15min":132.41900000000001},"last":0.42299999999999999},{"interface":"front","average":{"1min":0.57399999999999995,"5min":1.99,"15min":2.2090000000000001},"min":{"1min":0.35099999999999998,"5min":0.29099999999999998,"15min":0.217},"max":{"1min":0.70699999999999996,"5min":132.46700000000001,"15min":132.46700000000001},"last":0.32900000000000001}]},{"osd":6,"last update":"Tue Sep 10 19:33:23 2024","interfaces":[{"interface":"back","average":{"1min":0.58999999999999997,"5min":0.874,"15min":1.6599999999999999},"min":{"1min":0.36699999999999999,"5min":0.28699999999999998,"15min":0.28399999999999997},"max":{"1min":0.80200000000000005,"5min":7.6269999999999998,"15min":111.328},"last":0.39100000000000001},{"interface":"front","average":{"1min":0.56699999999999995,"5min":0.876,"15min":1.6679999999999999},"min":{"1min":0.35399999999999998,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.79000000000000004,"5min":4.1219999999999999,"15min":111.705},"last":0.373}]},{"osd":7,"last update":"Tue Sep 10 19:32:35 2024","interfaces":[{"interface":"back","average":{"1min":0.59899999999999998,"5min":1.028,"15min":1.369},"min":{"1min":0.39200000000000002,"5min":0.29099999999999998,"15min":0.28599999999999998},"max":{"1min":0.77300000000000002,"5min":5.0640000000000001,"15min":6.7869999999999999},"last":0.44700000000000001},{"interface":"front","average":{"1min":0.55000000000000004,"5min":0.98799999999999999,"15min":1.3580000000000001},"min":{"1min":0.371,"5min":0.29899999999999999,"15min":0.27100000000000002},"max":{"1min":0.68400000000000005,"5min":5.6349999999999998,"15min":9.4909999999999997},"last":0.48399999999999999}]}]},{"osd":4,"up_from":32,"seq":137438953772,"num_pgs":55,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":901604,"kb_used_data":728,"kb_used_omap":1,"kb_used_meta":900862,"kb_avail":92843548,"statfs":{"total":95995035648,"available":95071793152,"internally_reserved":0,"allocated":745472,"data_stored":435853,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":922483032},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 19:32:33 2024","interfaces":[{"interface":"back","average":{"1min":0.57099999999999995,"5min":0.76900000000000002,"15min":1.821},"min":{"1min":0.25,"5min":0.20100000000000001,"15min":0.20100000000000001},"max":{"1min":0.85499999999999998,"5min":3.6110000000000002,"15min":39.097000000000001},"last":0.56899999999999995},{"interface":"front","average":{"1min":0.60699999999999998,"5min":0.70199999999999996,"15min":2.0230000000000001},"min":{"1min":0.39400000000000002,"5min":0.24099999999999999,"15min":0.20499999999999999},"max":{"1min":0.93200000000000005,"5min":5.6379999999999999,"15min":39.112000000000002},"last":0.52400000000000002}]},{"osd":1,"last update":"Tue Sep 10 19:32:59 2024","interfaces":[{"interface":"back","average":{"1min":0.56100000000000005,"5min":1.712,"15min":1.177},"min":{"1min":0.36599999999999999,"5min":0.23699999999999999,"15min":0.20599999999999999},"max":{"1min":0.88900000000000001,"5min":75.620000000000005,"15min":75.620000000000005},"last":0.45600000000000002},{"interface":"front","average":{"1min":0.56599999999999995,"5min":1.5249999999999999,"15min":1.042},"min":{"1min":0.42499999999999999,"5min":0.23499999999999999,"15min":0.23400000000000001},"max":{"1min":0.74099999999999999,"5min":75.230999999999995,"15min":75.230999999999995},"last":0.59999999999999998}]},{"osd":2,"last update":"Tue Sep 10 19:33:29 2024","interfaces":[{"interface":"back","average":{"1min":0.55800000000000005,"5min":0.68100000000000005,"15min":0.92700000000000005},"min":{"1min":0.309,"5min":0.25600000000000001,"15min":0.254},"max":{"1min":0.76500000000000001,"5min":5.5609999999999999,"15min":5.5609999999999999},"last":0.623},{"interface":"front","average":{"1min":0.55900000000000005,"5min":0.73199999999999998,"15min":0.93300000000000005},"min":{"1min":0.28599999999999998,"5min":0.28599999999999998,"15min":0.253},"max":{"1min":0.72099999999999997,"5min":3.2120000000000002,"15min":6.7690000000000001},"last":0.72099999999999997}]},{"osd":3,"last update":"Tue Sep 10 19:33:29 2024","interfaces":[{"interface":"back","average":{"1min":0.56100000000000005,"5min":0.82199999999999995,"15min":1.022},"min":{"1min":0.36099999999999999,"5min":0.28599999999999998,"15min":0.20999999999999999},"max":{"1min":0.80700000000000005,"5min":3.1880000000000002,"15min":15.436},"last":0.67400000000000004},{"interface":"front","average":{"1min":0.56000000000000005,"5min":0.89600000000000002,"15min":0.995},"min":{"1min":0.43099999999999999,"5min":0.26600000000000001,"15min":0.246},"max":{"1min":0.82299999999999995,"5min":5.3140000000000001,"15min":14.954000000000001},"last":0.65600000000000003}]},{"osd":5,"last update":"Tue Sep 10 19:32:48 2024","interfaces":[{"interface":"back","average":{"1min":0.48399999999999999,"5min":0.81100000000000005,"15min":2.2810000000000001},"min":{"1min":0.33200000000000002,"5min":0.29899999999999999,"15min":0.224},"max":{"1min":0.65700000000000003,"5min":4.7190000000000003,"15min":227.726},"last":0.68899999999999995},{"interface":"front","average":{"1min":0.51300000000000001,"5min":0.76800000000000002,"15min":2.2909999999999999},"min":{"1min":0.35299999999999998,"5min":0.27400000000000002,"15min":0.20699999999999999},"max":{"1min":0.70599999999999996,"5min":5.6699999999999999,"15min":227.70500000000001},"last":0.75700000000000001}]},{"osd":6,"last update":"Tue Sep 10 19:33:29 2024","interfaces":[{"interface":"back","average":{"1min":0.50800000000000001,"5min":0.72099999999999997,"15min":1.2070000000000001},"min":{"1min":0.34499999999999997,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":0.69899999999999995,"5min":5.8250000000000002,"15min":8.1010000000000009},"last":0.69899999999999995},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.94699999999999995,"15min":1.2849999999999999},"min":{"1min":0.38,"5min":0.254,"15min":0.254},"max":{"1min":0.78300000000000003,"5min":6.0099999999999998,"15min":16.815000000000001},"last":0.73899999999999999}]},{"osd":7,"last update":"Tue Sep 10 19:32:33 2024","interfaces":[{"interface":"back","average":{"1min":0.54600000000000004,"5min":1.2410000000000001,"15min":1.4099999999999999},"min":{"1min":0.38400000000000001,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.71199999999999997,"5min":14.166,"15min":16.853999999999999},"last":0.71199999999999997},{"interface":"front","average":{"1min":0.52200000000000002,"5min":1.214,"15min":1.4350000000000001},"min":{"1min":0.376,"5min":0.25700000000000001,"15min":0.23699999999999999},"max":{"1min":0.67800000000000005,"5min":14.380000000000001,"15min":14.380000000000001},"last":0.77100000000000002}]}]},{"osd":3,"up_from":26,"seq":111669150001,"num_pgs":60,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":1181684,"kb_used_data":1324,"kb_used_omap":1,"kb_used_meta":1180350,"kb_avail":92563468,"statfs":{"total":95995035648,"available":94784991232,"internally_reserved":0,"allocated":1355776,"data_stored":1027685,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":1208678744},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 19:32:43 2024","interfaces":[{"interface":"back","average":{"1min":0.33000000000000002,"5min":3.5640000000000001,"15min":1.579},"min":{"1min":0.19500000000000001,"5min":0.16800000000000001,"15min":0.16800000000000001},"max":{"1min":0.52300000000000002,"5min":117.655,"15min":117.655},"last":0.35699999999999998},{"interface":"front","average":{"1min":0.34100000000000003,"5min":3.5409999999999999,"15min":1.554},"min":{"1min":0.219,"5min":0.185,"15min":0.184},"max":{"1min":0.745,"5min":117.81399999999999,"15min":117.81399999999999},"last":0.20200000000000001}]},{"osd":1,"last update":"Tue Sep 10 19:33:08 2024","interfaces":[{"interface":"back","average":{"1min":0.39200000000000002,"5min":0.876,"15min":0.53100000000000003},"min":{"1min":0.224,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":1.0960000000000001,"5min":5.1779999999999999,"15min":5.1779999999999999},"last":0.28799999999999998},{"interface":"front","average":{"1min":0.48799999999999999,"5min":0.90800000000000003,"15min":0.55000000000000004},"min":{"1min":0.221,"5min":0.19700000000000001,"15min":0.19500000000000001},"max":{"1min":1.873,"5min":4.0919999999999996,"15min":4.4100000000000001},"last":0.32900000000000001}]},{"osd":2,"last update":"Tue Sep 10 19:33:18 2024","interfaces":[{"interface":"back","average":{"1min":0.48899999999999999,"5min":0.60899999999999999,"15min":0.501},"min":{"1min":0.32200000000000001,"5min":0.25800000000000001,"15min":0.17599999999999999},"max":{"1min":1.3300000000000001,"5min":3.7730000000000001,"15min":4.3769999999999998},"last":0.26800000000000002},{"interface":"front","average":{"1min":0.45000000000000001,"5min":0.61799999999999999,"15min":0.495},"min":{"1min":0.27800000000000002,"5min":0.23100000000000001,"15min":0.16400000000000001},"max":{"1min":1.6579999999999999,"5min":4.2990000000000004,"15min":4.2990000000000004},"last":0.308}]},{"osd":4,"last update":"Tue Sep 10 19:32:33 2024","interfaces":[{"interface":"back","average":{"1min":0.56200000000000006,"5min":0.88,"15min":1.262},"min":{"1min":0.39800000000000002,"5min":0.30099999999999999,"15min":0.254},"max":{"1min":0.81399999999999995,"5min":4.4169999999999998,"15min":81.334000000000003},"last":0.34999999999999998},{"interface":"front","average":{"1min":0.58199999999999996,"5min":0.92400000000000004,"15min":1.3560000000000001},"min":{"1min":0.40400000000000003,"5min":0.318,"15min":0.22700000000000001},"max":{"1min":0.81399999999999995,"5min":4.343,"15min":81.381},"last":0.46999999999999997}]},{"osd":5,"last update":"Tue Sep 10 19:32:48 2024","interfaces":[{"interface":"back","average":{"1min":0.58899999999999997,"5min":0.89100000000000001,"15min":1.3740000000000001},"min":{"1min":0.41899999999999998,"5min":0.28000000000000003,"15min":0.27900000000000003},"max":{"1min":0.95999999999999996,"5min":4.4560000000000004,"15min":104.672},"last":0.40300000000000002},{"interface":"front","average":{"1min":0.61399999999999999,"5min":0.98699999999999999,"15min":1.325},"min":{"1min":0.41799999999999998,"5min":0.314,"15min":0.24299999999999999},"max":{"1min":1.159,"5min":3.4900000000000002,"15min":105.971},"last":0.49199999999999999}]},{"osd":6,"last update":"Tue Sep 10 19:33:18 2024","interfaces":[{"interface":"back","average":{"1min":0.59399999999999997,"5min":2.5739999999999998,"15min":2.0590000000000002},"min":{"1min":0.46500000000000002,"5min":0.33400000000000002,"15min":0.23699999999999999},"max":{"1min":1.056,"5min":148.09299999999999,"15min":148.09299999999999},"last":0.379},{"interface":"front","average":{"1min":0.65900000000000003,"5min":2.5499999999999998,"15min":1.9790000000000001},"min":{"1min":0.439,"5min":0.37,"15min":0.26800000000000002},"max":{"1min":1.393,"5min":148.31999999999999,"15min":148.31999999999999},"last":0.41699999999999998}]},{"osd":7,"last update":"Tue Sep 10 19:32:48 2024","interfaces":[{"interface":"back","average":{"1min":0.65600000000000003,"5min":1.151,"15min":1.3979999999999999},"min":{"1min":0.443,"5min":0.33600000000000002,"15min":0.26100000000000001},"max":{"1min":1.361,"5min":6.8639999999999999,"15min":104.749},"last":0.435},{"interface":"front","average":{"1min":0.60199999999999998,"5min":1.0589999999999999,"15min":1.319},"min":{"1min":0.48099999999999998,"5min":0.39200000000000002,"15min":0.26100000000000001},"max":{"1min":1.028,"5min":5.8929999999999998,"15min":104.69499999999999},"last":0.42899999999999999}]}]},{"osd":2,"up_from":18,"seq":77309411638,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":306660,"kb_used_data":984,"kb_used_omap":1,"kb_used_meta":305662,"kb_avail":93438492,"statfs":{"total":95995035648,"available":95681015808,"internally_reserved":0,"allocated":1007616,"data_stored":698971,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":312998232},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 19:32:45 2024","interfaces":[{"interface":"back","average":{"1min":0.34699999999999998,"5min":0.495,"15min":0.46400000000000002},"min":{"1min":0.20899999999999999,"5min":0.17000000000000001,"15min":0.17000000000000001},"max":{"1min":0.63200000000000001,"5min":2.9129999999999998,"15min":3.6299999999999999},"last":0.48899999999999999},{"interface":"front","average":{"1min":0.36299999999999999,"5min":0.54700000000000004,"15min":0.51600000000000001},"min":{"1min":0.26500000000000001,"5min":0.249,"15min":0.19600000000000001},"max":{"1min":0.57199999999999995,"5min":3.2010000000000001,"15min":3.9300000000000002},"last":0.443}]},{"osd":1,"last update":"Tue Sep 10 19:33:00 2024","interfaces":[{"interface":"back","average":{"1min":0.35099999999999998,"5min":3.3700000000000001,"15min":1.3480000000000001},"min":{"1min":0.20000000000000001,"5min":0.193,"15min":0.193},"max":{"1min":0.67600000000000005,"5min":111.398,"15min":111.398},"last":0.38600000000000001},{"interface":"front","average":{"1min":0.40300000000000002,"5min":3.4289999999999998,"15min":1.4059999999999999},"min":{"1min":0.311,"5min":0.245,"15min":0.22500000000000001},"max":{"1min":0.70899999999999996,"5min":111.36,"15min":111.36},"last":0.33800000000000002}]},{"osd":3,"last update":"Tue Sep 10 19:33:25 2024","interfaces":[{"interface":"back","average":{"1min":0.41699999999999998,"5min":0.59699999999999998,"15min":0.70499999999999996},"min":{"1min":0.314,"5min":0.23100000000000001,"15min":0.156},"max":{"1min":0.83499999999999996,"5min":4.3700000000000001,"15min":25.033000000000001},"last":0.46100000000000002},{"interface":"front","average":{"1min":0.34499999999999997,"5min":0.56699999999999995,"15min":0.68999999999999995},"min":{"1min":0.27300000000000002,"5min":0.21099999999999999,"15min":0.17299999999999999},"max":{"1min":0.499,"5min":3.6699999999999999,"15min":24.890999999999998},"last":0.41799999999999998}]},{"osd":4,"last update":"Tue Sep 10 19:32:30 2024","interfaces":[{"interface":"back","average":{"1min":0.63800000000000001,"5min":0.78500000000000003,"15min":0.99099999999999999},"min":{"1min":0.48099999999999998,"5min":0.316,"15min":0.23200000000000001},"max":{"1min":0.87,"5min":4.6399999999999997,"15min":17.137},"last":0.57899999999999996},{"interface":"front","average":{"1min":0.63100000000000001,"5min":0.73799999999999999,"15min":0.96399999999999997},"min":{"1min":0.49399999999999999,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.84999999999999998,"5min":4.6289999999999996,"15min":17.041},"last":0.59099999999999997}]},{"osd":5,"last update":"Tue Sep 10 19:32:54 2024","interfaces":[{"interface":"back","average":{"1min":0.56499999999999995,"5min":0.84599999999999997,"15min":1.248},"min":{"1min":0.41099999999999998,"5min":0.28699999999999998,"15min":0.24099999999999999},"max":{"1min":0.77500000000000002,"5min":5.3029999999999999,"15min":50.848999999999997},"last":0.501},{"interface":"front","average":{"1min":0.63800000000000001,"5min":0.77400000000000002,"15min":1.274},"min":{"1min":0.44800000000000001,"5min":0.307,"15min":0.23899999999999999},"max":{"1min":0.874,"5min":5.0179999999999998,"15min":50.817999999999998},"last":0.55100000000000005}]},{"osd":6,"last update":"Tue Sep 10 19:33:25 2024","interfaces":[{"interface":"back","average":{"1min":0.58599999999999997,"5min":0.82199999999999995,"15min":1.012},"min":{"1min":0.433,"5min":0.34699999999999998,"15min":0.23300000000000001},"max":{"1min":0.75,"5min":4.8650000000000002,"15min":17.859999999999999},"last":0.60199999999999998},{"interface":"front","average":{"1min":0.56699999999999995,"5min":0.77100000000000002,"15min":0.91900000000000004},"min":{"1min":0.42299999999999999,"5min":0.308,"15min":0.25900000000000001},"max":{"1min":0.71799999999999997,"5min":3.819,"15min":17.274000000000001},"last":0.56799999999999995}]},{"osd":7,"last update":"Tue Sep 10 19:32:30 2024","interfaces":[{"interface":"back","average":{"1min":0.623,"5min":0.77200000000000002,"15min":1.022},"min":{"1min":0.42999999999999999,"5min":0.34000000000000002,"15min":0.32400000000000001},"max":{"1min":0.82299999999999995,"5min":5.1900000000000004,"15min":10.413},"last":0.51300000000000001},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.81399999999999995,"15min":1.0009999999999999},"min":{"1min":0.442,"5min":0.29999999999999999,"15min":0.27900000000000003},"max":{"1min":0.754,"5min":5.2039999999999997,"15min":11.047000000000001},"last":0.52800000000000002}]}]},{"osd":1,"up_from":276,"seq":1185410973827,"num_pgs":51,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":310772,"kb_used_data":744,"kb_used_omap":6,"kb_used_meta":310009,"kb_avail":93434380,"statfs":{"total":95995035648,"available":95676805120,"internally_reserved":0,"allocated":761856,"data_stored":436671,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6335,"internal_metadata":317450049},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Sep 10 19:32:45 2024","interfaces":[{"interface":"back","average":{"1min":0.28699999999999998,"5min":0.75800000000000001,"15min":0.58199999999999996},"min":{"1min":0.16400000000000001,"5min":0.16400000000000001,"15min":0.16400000000000001},"max":{"1min":0.40899999999999997,"5min":3.5990000000000002,"15min":3.5990000000000002},"last":0.245},{"interface":"front","average":{"1min":0.308,"5min":0.93200000000000005,"15min":0.59899999999999998},"min":{"1min":0.20100000000000001,"5min":0.192,"15min":0.192},"max":{"1min":0.46700000000000003,"5min":4.6079999999999997,"15min":4.6079999999999997},"last":0.374}]},{"osd":2,"last update":"Tue Sep 10 19:33:10 2024","interfaces":[{"interface":"back","average":{"1min":0.36199999999999999,"5min":0.747,"15min":0.48999999999999999},"min":{"1min":0.245,"5min":0.24099999999999999,"15min":0.19},"max":{"1min":0.60099999999999998,"5min":4.1779999999999999,"15min":4.1779999999999999},"last":0.312},{"interface":"front","average":{"1min":0.374,"5min":0.78800000000000003,"15min":0.51800000000000002},"min":{"1min":0.27100000000000002,"5min":0.22800000000000001,"15min":0.222},"max":{"1min":0.53200000000000003,"5min":4.5810000000000004,"15min":4.5810000000000004},"last":0.32700000000000001}]},{"osd":3,"last update":"Tue Sep 10 19:33:10 2024","interfaces":[{"interface":"back","average":{"1min":0.371,"5min":1.077,"15min":0.60099999999999998},"min":{"1min":0.27800000000000002,"5min":0.23799999999999999,"15min":0.221},"max":{"1min":0.54100000000000004,"5min":11.494999999999999,"15min":11.494999999999999},"last":0.28199999999999997},{"interface":"front","average":{"1min":0.38800000000000001,"5min":1.276,"15min":0.65400000000000003},"min":{"1min":0.26200000000000001,"5min":0.26200000000000001,"15min":0.17599999999999999},"max":{"1min":0.57099999999999995,"5min":21.545000000000002,"15min":21.545000000000002},"last":0.35799999999999998}]},{"osd":4,"last update":"Tue Sep 10 19:33:10 2024","interfaces":[{"interface":"back","average":{"1min":0.50900000000000001,"5min":3.569,"15min":1.6859999999999999},"min":{"1min":0.35599999999999998,"5min":0.28599999999999998,"15min":0.26600000000000001},"max":{"1min":0.65600000000000003,"5min":271.53100000000001,"15min":271.53100000000001},"last":0.60899999999999999},{"interface":"front","average":{"1min":0.53900000000000003,"5min":3.5630000000000002,"15min":1.702},"min":{"1min":0.378,"5min":0.30199999999999999,"15min":0.27600000000000002},"max":{"1min":0.73299999999999998,"5min":271.483,"15min":271.483},"last":0.54600000000000004}]},{"osd":5,"last update":"Tue Sep 10 19:33:10 2024","interfaces":[{"interface":"back","average":{"1min":0.55400000000000005,"5min":3.621,"15min":1.835},"min":{"1min":0.44800000000000001,"5min":0.252,"15min":0.252},"max":{"1min":0.65200000000000002,"5min":212.727,"15min":212.727},"last":0.67900000000000005},{"interface":"front","average":{"1min":0.58599999999999997,"5min":3.613,"15min":2.0409999999999999},"min":{"1min":0.46000000000000002,"5min":0.32100000000000001,"15min":0.313},"max":{"1min":0.71399999999999997,"5min":212.74799999999999,"15min":212.74799999999999},"last":0.64700000000000002}]},{"osd":6,"last update":"Tue Sep 10 19:33:10 2024","interfaces":[{"interface":"back","average":{"1min":0.55500000000000005,"5min":1.1899999999999999,"15min":1.625},"min":{"1min":0.41499999999999998,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.75800000000000001,"5min":7.5270000000000001,"15min":27.379999999999999},"last":0.69999999999999996},{"interface":"front","average":{"1min":0.54500000000000004,"5min":1.117,"15min":1.587},"min":{"1min":0.40699999999999997,"5min":0.33500000000000002,"15min":0.30299999999999999},"max":{"1min":0.73999999999999999,"5min":5.8849999999999998,"15min":26.812999999999999},"last":0.58699999999999997}]},{"osd":7,"last update":"Tue Sep 10 19:33:10 2024","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":3.4729999999999999,"15min":1.829},"min":{"1min":0.46800000000000003,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.75,"5min":242.33099999999999,"15min":242.33099999999999},"last":0.65300000000000002},{"interface":"front","average":{"1min":0.58599999999999997,"5min":3.4689999999999999,"15min":1.831},"min":{"1min":0.436,"5min":0.39000000000000001,"15min":0.33200000000000002},"max":{"1min":0.72399999999999998,"5min":242.31,"15min":242.31},"last":0.71099999999999997}]}]},{"osd":0,"up_from":301,"seq":1292785156207,"num_pgs":51,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":355020,"kb_used_data":712,"kb_used_omap":4,"kb_used_meta":354171,"kb_avail":93390132,"statfs":{"total":95995035648,"available":95631495168,"internally_reserved":0,"allocated":729088,"data_stored":428489,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4602,"internal_metadata":362671622},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Sep 10 19:32:43 2024","interfaces":[{"interface":"back","average":{"1min":0.34599999999999997,"5min":1.004,"15min":0.68799999999999994},"min":{"1min":0.19,"5min":0.19,"15min":0.17499999999999999},"max":{"1min":0.55700000000000005,"5min":6.0369999999999999,"15min":6.0369999999999999},"last":0.45700000000000002},{"interface":"front","average":{"1min":0.372,"5min":0.84799999999999998,"15min":0.59299999999999997},"min":{"1min":0.23200000000000001,"5min":0.188,"15min":0.16900000000000001},"max":{"1min":0.57199999999999995,"5min":3.7909999999999999,"15min":3.7909999999999999},"last":0.41399999999999998}]},{"osd":2,"last update":"Tue Sep 10 19:32:43 2024","interfaces":[{"interface":"back","average":{"1min":0.35399999999999998,"5min":0.58999999999999997,"15min":0.57599999999999996},"min":{"1min":0.20499999999999999,"5min":0.20499999999999999,"15min":0.20399999999999999},"max":{"1min":0.50700000000000001,"5min":4.3230000000000004,"15min":4.3230000000000004},"last":0.47199999999999998},{"interface":"front","average":{"1min":0.38800000000000001,"5min":0.81899999999999995,"15min":0.67200000000000004},"min":{"1min":0.27200000000000002,"5min":0.249,"15min":0.22600000000000001},"max":{"1min":0.62,"5min":5.1079999999999997,"15min":5.1079999999999997},"last":0.376}]},{"osd":3,"last update":"Tue Sep 10 19:32:43 2024","interfaces":[{"interface":"back","average":{"1min":0.38900000000000001,"5min":1.7509999999999999,"15min":0.95699999999999996},"min":{"1min":0.27100000000000002,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.54200000000000004,"5min":89.468999999999994,"15min":89.468999999999994},"last":0.44600000000000001},{"interface":"front","average":{"1min":0.41599999999999998,"5min":1.8480000000000001,"15min":1.0209999999999999},"min":{"1min":0.28399999999999997,"5min":0.216,"15min":0.216},"max":{"1min":0.55300000000000005,"5min":89.272999999999996,"15min":89.272999999999996},"last":0.5}]},{"osd":4,"last update":"Tue Sep 10 19:32:43 2024","interfaces":[{"interface":"back","average":{"1min":0.57299999999999995,"5min":1.0509999999999999,"15min":2.9409999999999998},"min":{"1min":0.375,"5min":0.32700000000000001,"15min":0.25800000000000001},"max":{"1min":0.89700000000000002,"5min":4.9729999999999999,"15min":53.670000000000002},"last":0.60499999999999998},{"interface":"front","average":{"1min":0.60699999999999998,"5min":0.871,"15min":2.8730000000000002},"min":{"1min":0.39200000000000002,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.82499999999999996,"5min":4.0369999999999999,"15min":53.935000000000002},"last":0.68100000000000005}]},{"osd":5,"last update":"Tue Sep 10 19:32:43 2024","interfaces":[{"interface":"back","average":{"1min":0.60999999999999999,"5min":0.88200000000000001,"15min":2.6030000000000002},"min":{"1min":0.374,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":0.82299999999999995,"5min":4.2779999999999996,"15min":53.884999999999998},"last":0.79100000000000004},{"interface":"front","average":{"1min":0.622,"5min":0.95999999999999996,"15min":2.5670000000000002},"min":{"1min":0.40500000000000003,"5min":0.34899999999999998,"15min":0.309},"max":{"1min":0.874,"5min":5.0519999999999996,"15min":53.875999999999998},"last":0.67200000000000004}]},{"osd":6,"last update":"Tue Sep 10 19:32:43 2024","interfaces":[{"interface":"back","average":{"1min":0.625,"5min":1.073,"15min":3.085},"min":{"1min":0.432,"5min":0.311,"15min":0.311},"max":{"1min":0.85999999999999999,"5min":5.1449999999999996,"15min":56.901000000000003},"last":0.63800000000000001},{"interface":"front","average":{"1min":0.625,"5min":1.0649999999999999,"15min":3.044},"min":{"1min":0.39400000000000002,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.89600000000000002,"5min":6.6840000000000002,"15min":55.177},"last":0.80600000000000005}]},{"osd":7,"last update":"Tue Sep 10 19:32:43 2024","interfaces":[{"interface":"back","average":{"1min":0.625,"5min":1,"15min":1.155},"min":{"1min":0.40000000000000002,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.91800000000000004,"5min":6.3959999999999999,"15min":6.3959999999999999},"last":0.76900000000000002},{"interface":"front","average":{"1min":0.64800000000000002,"5min":1.0189999999999999,"15min":1.149},"min":{"1min":0.47399999999999998,"5min":0.375,"15min":0.34000000000000002},"max":{"1min":0.93000000000000005,"5min":5.0810000000000004,"15min":5.0810000000000004},"last":0.64800000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":966,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1236,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1236,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":362,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":2008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1238,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":1788,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":924,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":1144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T19:33:33.258 INFO:tasks.cephadm.ceph_manager.ceph:recovered! 2024-09-10T19:33:33.258 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588 shell --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 -- ceph config rm mgr mgr_debug_aggressive_pg_num_changes 2024-09-10T19:33:33.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:33:33 smithi183 ceph-mon[29736]: pgmap v1075: 142 pgs: 142 active+clean; 582 KiB data, 5.8 GiB used, 709 GiB / 715 GiB avail; 585 B/s rd, 0 op/s 2024-09-10T19:33:33.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:33:33 smithi183 ceph-mon[29736]: 2.a scrub starts 2024-09-10T19:33:33.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:33:33 smithi183 ceph-mon[29736]: 5.5 scrub starts 2024-09-10T19:33:33.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:33:33 smithi183 ceph-mon[29736]: 2.a scrub ok 2024-09-10T19:33:33.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:33:33 smithi183 ceph-mon[29736]: 5.5 scrub ok 2024-09-10T19:33:33.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:33:33 smithi183 ceph-mon[29736]: 6.a scrub starts 2024-09-10T19:33:33.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:33:33 smithi183 ceph-mon[29736]: 6.a scrub ok 2024-09-10T19:33:33.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:33:33 smithi183 ceph-mon[29736]: 6.1c scrub starts 2024-09-10T19:33:33.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:33:33 smithi183 ceph-mon[29736]: 6.1c scrub ok 2024-09-10T19:33:33.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:33:33 smithi183 ceph-mon[41562]: pgmap v1075: 142 pgs: 142 active+clean; 582 KiB data, 5.8 GiB used, 709 GiB / 715 GiB avail; 585 B/s rd, 0 op/s 2024-09-10T19:33:33.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:33:33 smithi183 ceph-mon[41562]: 2.a scrub starts 2024-09-10T19:33:33.432 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:33:33 smithi183 ceph-mon[41562]: 5.5 scrub starts 2024-09-10T19:33:33.433 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:33:33 smithi183 ceph-mon[41562]: 2.a scrub ok 2024-09-10T19:33:33.433 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:33:33 smithi183 ceph-mon[41562]: 5.5 scrub ok 2024-09-10T19:33:33.433 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:33:33 smithi183 ceph-mon[41562]: 6.a scrub starts 2024-09-10T19:33:33.433 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:33:33 smithi183 ceph-mon[41562]: 6.a scrub ok 2024-09-10T19:33:33.433 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:33:33 smithi183 ceph-mon[41562]: 6.1c scrub starts 2024-09-10T19:33:33.433 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:33:33 smithi183 ceph-mon[41562]: 6.1c scrub ok 2024-09-10T19:33:33.507 INFO:teuthology.orchestra.run.smithi183.stderr:Inferring config /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/mon.a/config 2024-09-10T19:33:33.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:33:33 smithi191 ceph-mon[32884]: pgmap v1075: 142 pgs: 142 active+clean; 582 KiB data, 5.8 GiB used, 709 GiB / 715 GiB avail; 585 B/s rd, 0 op/s 2024-09-10T19:33:33.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:33:33 smithi191 ceph-mon[32884]: 2.a scrub starts 2024-09-10T19:33:33.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:33:33 smithi191 ceph-mon[32884]: 5.5 scrub starts 2024-09-10T19:33:33.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:33:33 smithi191 ceph-mon[32884]: 2.a scrub ok 2024-09-10T19:33:33.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:33:33 smithi191 ceph-mon[32884]: 5.5 scrub ok 2024-09-10T19:33:33.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:33:33 smithi191 ceph-mon[32884]: 6.a scrub starts 2024-09-10T19:33:33.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:33:33 smithi191 ceph-mon[32884]: 6.a scrub ok 2024-09-10T19:33:33.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:33:33 smithi191 ceph-mon[32884]: 6.1c scrub starts 2024-09-10T19:33:33.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:33:33 smithi191 ceph-mon[32884]: 6.1c scrub ok 2024-09-10T19:33:34.662 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:33:34 smithi183 ceph-mon[29736]: from='client.16494 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-10T19:33:34.662 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:33:34 smithi183 ceph-mon[29736]: 5.0 scrub starts 2024-09-10T19:33:34.662 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:33:34 smithi183 ceph-mon[29736]: 5.0 scrub ok 2024-09-10T19:33:34.662 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:33:34 smithi183 ceph-mon[29736]: 2.7 scrub starts 2024-09-10T19:33:34.662 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:33:34 smithi183 ceph-mon[29736]: 2.7 scrub ok 2024-09-10T19:33:34.663 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:33:34 smithi183 ceph-mon[41562]: from='client.16494 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-10T19:33:34.663 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:33:34 smithi183 ceph-mon[41562]: 5.0 scrub starts 2024-09-10T19:33:34.663 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:33:34 smithi183 ceph-mon[41562]: 5.0 scrub ok 2024-09-10T19:33:34.663 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:33:34 smithi183 ceph-mon[41562]: 2.7 scrub starts 2024-09-10T19:33:34.663 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:33:34 smithi183 ceph-mon[41562]: 2.7 scrub ok 2024-09-10T19:33:34.715 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:33:34 smithi191 ceph-mon[32884]: from='client.16494 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-10T19:33:34.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:33:34 smithi191 ceph-mon[32884]: 5.0 scrub starts 2024-09-10T19:33:34.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:33:34 smithi191 ceph-mon[32884]: 5.0 scrub ok 2024-09-10T19:33:34.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:33:34 smithi191 ceph-mon[32884]: 2.7 scrub starts 2024-09-10T19:33:34.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:33:34 smithi191 ceph-mon[32884]: 2.7 scrub ok 2024-09-10T19:33:35.641 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:33:35 smithi183 ceph-mon[29736]: pgmap v1076: 142 pgs: 142 active+clean; 582 KiB data, 5.8 GiB used, 709 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-10T19:33:35.641 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:33:35 smithi183 ceph-mon[29736]: from='client.? 172.21.15.183:0/843845047' entity='client.admin' cmd=[{"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"}]: dispatch 2024-09-10T19:33:35.641 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:33:35 smithi183 ceph-mon[29736]: from='client.? ' entity='client.admin' cmd=[{"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"}]: dispatch 2024-09-10T19:33:35.641 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:33:35 smithi183 ceph-mon[29736]: from='client.? ' entity='client.admin' cmd='[{"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"}]': finished 2024-09-10T19:33:35.641 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:33:35 smithi183 ceph-mon[29736]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T19:33:35.642 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Sep 10 19:33:35 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-y[29989]: [10/Sep/2024:19:33:35] ENGINE Bus STOPPING 2024-09-10T19:33:35.642 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:33:35 smithi183 ceph-mon[41562]: pgmap v1076: 142 pgs: 142 active+clean; 582 KiB data, 5.8 GiB used, 709 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-10T19:33:35.642 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:33:35 smithi183 ceph-mon[41562]: from='client.? 172.21.15.183:0/843845047' entity='client.admin' cmd=[{"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"}]: dispatch 2024-09-10T19:33:35.642 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:33:35 smithi183 ceph-mon[41562]: from='client.? ' entity='client.admin' cmd=[{"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"}]: dispatch 2024-09-10T19:33:35.642 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:33:35 smithi183 ceph-mon[41562]: from='client.? ' entity='client.admin' cmd='[{"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"}]': finished 2024-09-10T19:33:35.642 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:33:35 smithi183 ceph-mon[41562]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T19:33:35.715 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:33:35 smithi191 ceph-mon[32884]: pgmap v1076: 142 pgs: 142 active+clean; 582 KiB data, 5.8 GiB used, 709 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-10T19:33:35.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:33:35 smithi191 ceph-mon[32884]: from='client.? 172.21.15.183:0/843845047' entity='client.admin' cmd=[{"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"}]: dispatch 2024-09-10T19:33:35.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:33:35 smithi191 ceph-mon[32884]: from='client.? ' entity='client.admin' cmd=[{"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"}]: dispatch 2024-09-10T19:33:35.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:33:35 smithi191 ceph-mon[32884]: from='client.? ' entity='client.admin' cmd='[{"prefix": "config rm", "who": "mgr", "name": "mgr_debug_aggressive_pg_num_changes"}]': finished 2024-09-10T19:33:35.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:33:35 smithi191 ceph-mon[32884]: from='mgr.14556 172.21.15.183:0/2869626523' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-10T19:33:35.921 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Sep 10 19:33:35 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-y[29989]: [10/Sep/2024:19:33:35] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-09-10T19:33:35.921 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Sep 10 19:33:35 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-y[29989]: [10/Sep/2024:19:33:35] ENGINE Bus STOPPED 2024-09-10T19:33:35.921 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Sep 10 19:33:35 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-y[29989]: [10/Sep/2024:19:33:35] ENGINE Bus STARTING 2024-09-10T19:33:35.921 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Sep 10 19:33:35 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-y[29989]: [10/Sep/2024:19:33:35] ENGINE Serving on http://:::9283 2024-09-10T19:33:35.921 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Sep 10 19:33:35 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-y[29989]: [10/Sep/2024:19:33:35] ENGINE Bus STARTED 2024-09-10T19:33:35.966 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2024-09-10T19:33:35.978 INFO:tasks.cephadm:Teardown begin 2024-09-10T19:33:35.979 DEBUG:teuthology.orchestra.run.smithi183:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-09-10T19:33:36.015 DEBUG:teuthology.orchestra.run.smithi191:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-09-10T19:33:36.042 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2024-09-10T19:33:36.043 DEBUG:teuthology.orchestra.run.smithi183:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-09-10T19:33:36.070 DEBUG:teuthology.orchestra.run.smithi191:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-09-10T19:33:36.098 INFO:tasks.cephadm:Stopping all daemons... 2024-09-10T19:33:36.098 INFO:tasks.cephadm.mon.a:Stopping mon.a... 2024-09-10T19:33:36.099 DEBUG:teuthology.orchestra.run.smithi183:> sudo systemctl stop ceph-320f6df8-6fa7-11ef-bcea-c7b262605968@mon.a 2024-09-10T19:33:36.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:33:36 smithi183 systemd[1]: Stopping Ceph mon.a for 320f6df8-6fa7-11ef-bcea-c7b262605968... 2024-09-10T19:33:36.654 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:33:36 smithi183 ceph-mon[41562]: 6.2 deep-scrub starts 2024-09-10T19:33:36.655 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:33:36 smithi183 ceph-mon[41562]: 6.2 deep-scrub ok 2024-09-10T19:33:36.655 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:33:36 smithi183 ceph-mon[41562]: 6.c scrub starts 2024-09-10T19:33:36.655 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:33:36 smithi183 ceph-mon[41562]: 6.c scrub ok 2024-09-10T19:33:36.655 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:33:36 smithi183 ceph-mon[29736]: 6.2 deep-scrub starts 2024-09-10T19:33:36.655 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:33:36 smithi183 ceph-mon[29736]: 6.2 deep-scrub ok 2024-09-10T19:33:36.655 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:33:36 smithi183 ceph-mon[29736]: 6.c scrub starts 2024-09-10T19:33:36.655 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:33:36 smithi183 ceph-mon[29736]: 6.c scrub ok 2024-09-10T19:33:36.715 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:33:36 smithi191 ceph-mon[32884]: 6.2 deep-scrub starts 2024-09-10T19:33:36.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:33:36 smithi191 ceph-mon[32884]: 6.2 deep-scrub ok 2024-09-10T19:33:36.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:33:36 smithi191 ceph-mon[32884]: 6.c scrub starts 2024-09-10T19:33:36.716 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:33:36 smithi191 ceph-mon[32884]: 6.c scrub ok 2024-09-10T19:33:37.431 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:33:37 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mon-a[29713]: 2024-09-10T19:33:37.007+0000 7f9f2e634640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-09-10T19:33:37.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:33:37 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mon-a[29713]: 2024-09-10T19:33:37.007+0000 7f9f2e634640 -1 mon.a@0(leader) e3 *** Got Signal Terminated *** 2024-09-10T19:33:37.432 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:33:37 smithi183 podman[238615]: 2024-09-10 19:33:37.252068677 +0000 UTC m=+0.381725491 container died fae323eed8e937f6d021ee78cc6ea5e4e5e25230228ac9d0174430e87674bb76 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588, name=ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mon-a, RELEASE=wip-guits-quincy-2024-09-09-1810-quincy-5be5739, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0) 2024-09-10T19:33:37.715 INFO:journalctl@ceph.mgr.x.smithi191.stdout:Sep 10 19:33:37 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-x[34967]: ::ffff:172.21.15.191 - - [10/Sep/2024:19:33:37] "GET /metrics HTTP/1.1" 200 - "" "Prometheus/2.43.0" 2024-09-10T19:33:37.892 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:33:37 smithi183 podman[238615]: 2024-09-10 19:33:37.656330778 +0000 UTC m=+0.785987586 container cleanup fae323eed8e937f6d021ee78cc6ea5e4e5e25230228ac9d0174430e87674bb76 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588, name=ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mon-a, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.license=GPLv2, RELEASE=wip-guits-quincy-2024-09-09-1810-quincy-5be5739, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS) 2024-09-10T19:33:37.892 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:33:37 smithi183 bash[238615]: ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mon-a 2024-09-10T19:33:38.181 INFO:journalctl@ceph.mon.a.smithi183.stdout:Sep 10 19:33:37 smithi183 podman[238739]: 2024-09-10 19:33:37.891269962 +0000 UTC m=+0.635357349 container remove fae323eed8e937f6d021ee78cc6ea5e4e5e25230228ac9d0174430e87674bb76 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588, name=ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mon-a, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, RELEASE=wip-guits-quincy-2024-09-09-1810-quincy-5be5739, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-10T19:33:39.162 DEBUG:teuthology.orchestra.run.smithi183:> sudo pkill -f 'journalctl -f -n 0 -u ceph-320f6df8-6fa7-11ef-bcea-c7b262605968@mon.a.service' 2024-09-10T19:33:39.206 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-10T19:33:39.207 INFO:tasks.cephadm.mon.a:Stopped mon.a 2024-09-10T19:33:39.207 INFO:tasks.cephadm.mon.b:Stopping mon.c... 2024-09-10T19:33:39.207 DEBUG:teuthology.orchestra.run.smithi183:> sudo systemctl stop ceph-320f6df8-6fa7-11ef-bcea-c7b262605968@mon.c 2024-09-10T19:33:39.431 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:33:39 smithi183 systemd[1]: Stopping Ceph mon.c for 320f6df8-6fa7-11ef-bcea-c7b262605968... 2024-09-10T19:33:40.182 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Sep 10 19:33:39 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-y[29989]: [10/Sep/2024:19:33:39] ENGINE Bus STOPPING 2024-09-10T19:33:40.182 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Sep 10 19:33:39 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-y[29989]: [10/Sep/2024:19:33:39] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-09-10T19:33:40.182 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Sep 10 19:33:39 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-y[29989]: [10/Sep/2024:19:33:39] ENGINE Bus STOPPED 2024-09-10T19:33:40.182 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Sep 10 19:33:39 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-y[29989]: [10/Sep/2024:19:33:39] ENGINE Bus STARTING 2024-09-10T19:33:40.182 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Sep 10 19:33:40 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-y[29989]: [10/Sep/2024:19:33:40] ENGINE Serving on http://:::9283 2024-09-10T19:33:40.182 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Sep 10 19:33:40 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-y[29989]: [10/Sep/2024:19:33:40] ENGINE Bus STARTED 2024-09-10T19:33:40.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:33:39 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mon-c[41540]: 2024-09-10T19:33:39.854+0000 7fe4db157640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-09-10T19:33:40.182 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:33:39 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mon-c[41540]: 2024-09-10T19:33:39.854+0000 7fe4db157640 -1 mon.c@2(peon) e3 *** Got Signal Terminated *** 2024-09-10T19:33:40.183 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:33:39 smithi183 podman[239022]: 2024-09-10 19:33:39.916836707 +0000 UTC m=+0.200293362 container died 193339ca1848d4955e928b9db4037cffbdebba48883ee47aee9f02dee955d48f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3bfeaf4417585c821cab9ca76e4d95f390b0e769807122a45463640820b5cd16, name=ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mon-c, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, ceph=True, org.label-schema.schema-version=1.0, RELEASE=wip-guits-quincy-2024-09-09-1810-quincy-5be5739, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux ) 2024-09-10T19:33:40.664 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:33:40 smithi183 podman[239022]: 2024-09-10 19:33:40.394324559 +0000 UTC m=+0.677781240 container cleanup 193339ca1848d4955e928b9db4037cffbdebba48883ee47aee9f02dee955d48f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3bfeaf4417585c821cab9ca76e4d95f390b0e769807122a45463640820b5cd16, name=ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mon-c, GIT_CLEAN=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.schema-version=1.0, RELEASE=wip-guits-quincy-2024-09-09-1810-quincy-5be5739, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, ceph=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD) 2024-09-10T19:33:40.664 INFO:journalctl@ceph.mon.c.smithi183.stdout:Sep 10 19:33:40 smithi183 bash[239022]: ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mon-c 2024-09-10T19:33:40.716 INFO:journalctl@ceph.iscsi.iscsi.a.smithi191.stdout:Sep 10 19:33:40 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-iscsi-iscsi-a[61816]: debug there is no tcmu-runner data available 2024-09-10T19:33:40.813 DEBUG:teuthology.orchestra.run.smithi183:> sudo pkill -f 'journalctl -f -n 0 -u ceph-320f6df8-6fa7-11ef-bcea-c7b262605968@mon.c.service' 2024-09-10T19:33:40.850 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-10T19:33:40.850 INFO:tasks.cephadm.mon.b:Stopped mon.c 2024-09-10T19:33:40.850 INFO:tasks.cephadm.mon.b:Stopping mon.b... 2024-09-10T19:33:40.850 DEBUG:teuthology.orchestra.run.smithi191:> sudo systemctl stop ceph-320f6df8-6fa7-11ef-bcea-c7b262605968@mon.b 2024-09-10T19:33:41.196 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:33:40 smithi191 systemd[1]: Stopping Ceph mon.b for 320f6df8-6fa7-11ef-bcea-c7b262605968... 2024-09-10T19:33:41.465 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:33:41 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mon-b[32849]: 2024-09-10T19:33:41.194+0000 7f5afee57640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-09-10T19:33:41.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:33:41 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mon-b[32849]: 2024-09-10T19:33:41.194+0000 7f5afee57640 -1 mon.b@1(peon) e3 *** Got Signal Terminated *** 2024-09-10T19:33:41.466 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:33:41 smithi191 podman[164852]: 2024-09-10 19:33:41.429172005 +0000 UTC m=+0.366712277 container died d6adad37a6c0f210238ab9bd899b7441d01091fe67611dfcb4d30f7ab1129eb9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3bfeaf4417585c821cab9ca76e4d95f390b0e769807122a45463640820b5cd16, name=ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mon-b, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.license=GPLv2, RELEASE=wip-guits-quincy-2024-09-09-1810-quincy-5be5739) 2024-09-10T19:33:41.681 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Sep 10 19:33:41 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-y[29989]: ::ffff:172.21.15.191 - - [10/Sep/2024:19:33:41] "GET /metrics HTTP/1.1" 200 222377 "" "Prometheus/2.43.0" 2024-09-10T19:33:42.163 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:33:41 smithi191 podman[164852]: 2024-09-10 19:33:41.910087856 +0000 UTC m=+0.847628164 container cleanup d6adad37a6c0f210238ab9bd899b7441d01091fe67611dfcb4d30f7ab1129eb9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3bfeaf4417585c821cab9ca76e4d95f390b0e769807122a45463640820b5cd16, name=ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mon-b, GIT_CLEAN=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.vendor=CentOS, RELEASE=wip-guits-quincy-2024-09-09-1810-quincy-5be5739, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2) 2024-09-10T19:33:42.163 INFO:journalctl@ceph.mon.b.smithi191.stdout:Sep 10 19:33:41 smithi191 bash[164852]: ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mon-b 2024-09-10T19:33:42.345 DEBUG:teuthology.orchestra.run.smithi191:> sudo pkill -f 'journalctl -f -n 0 -u ceph-320f6df8-6fa7-11ef-bcea-c7b262605968@mon.b.service' 2024-09-10T19:33:42.411 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-10T19:33:42.411 INFO:tasks.cephadm.mon.b:Stopped mon.b 2024-09-10T19:33:42.411 INFO:tasks.cephadm.mgr.y:Stopping mgr.y... 2024-09-10T19:33:42.412 DEBUG:teuthology.orchestra.run.smithi183:> sudo systemctl stop ceph-320f6df8-6fa7-11ef-bcea-c7b262605968@mgr.y 2024-09-10T19:33:42.727 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Sep 10 19:33:42 smithi183 systemd[1]: Stopping Ceph mgr.y for 320f6df8-6fa7-11ef-bcea-c7b262605968... 2024-09-10T19:33:43.182 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Sep 10 19:33:42 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-y[29989]: 2024-09-10T19:33:42.724+0000 7f9fe6aef640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mgr -n mgr.y -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-10T19:33:43.182 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Sep 10 19:33:42 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-y[29989]: 2024-09-10T19:33:42.724+0000 7f9fe6aef640 -1 mgr handle_mgr_signal *** Got signal Terminated *** 2024-09-10T19:33:43.182 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Sep 10 19:33:42 smithi183 podman[239162]: 2024-09-10 19:33:42.75696523 +0000 UTC m=+0.153805224 container died 73c5cf4fd854ee8514163d2ca51c956c5145ae3f18ab431c08031417b0232ade (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588, name=ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-y, org.label-schema.schema-version=1.0, RELEASE=wip-guits-quincy-2024-09-09-1810-quincy-5be5739, io.buildah.version=1.37.2, GIT_CLEAN=True, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, ceph=True, maintainer=Guillaume Abrioux , GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD) 2024-09-10T19:33:43.522 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Sep 10 19:33:43 smithi183 podman[239162]: 2024-09-10 19:33:43.507678685 +0000 UTC m=+0.904518673 container cleanup 73c5cf4fd854ee8514163d2ca51c956c5145ae3f18ab431c08031417b0232ade (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:5be57395ea235535910512ae73ba8e7ff25fe588, name=ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-y, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, RELEASE=wip-guits-quincy-2024-09-09-1810-quincy-5be5739, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-10T19:33:43.522 INFO:journalctl@ceph.mgr.y.smithi183.stdout:Sep 10 19:33:43 smithi183 bash[239162]: ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-y 2024-09-10T19:33:43.940 DEBUG:teuthology.orchestra.run.smithi183:> sudo pkill -f 'journalctl -f -n 0 -u ceph-320f6df8-6fa7-11ef-bcea-c7b262605968@mgr.y.service' 2024-09-10T19:33:43.976 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-10T19:33:43.976 INFO:tasks.cephadm.mgr.y:Stopped mgr.y 2024-09-10T19:33:43.976 INFO:tasks.cephadm.mgr.x:Stopping mgr.x... 2024-09-10T19:33:43.977 DEBUG:teuthology.orchestra.run.smithi191:> sudo systemctl stop ceph-320f6df8-6fa7-11ef-bcea-c7b262605968@mgr.x 2024-09-10T19:33:44.282 INFO:journalctl@ceph.mgr.x.smithi191.stdout:Sep 10 19:33:44 smithi191 systemd[1]: Stopping Ceph mgr.x for 320f6df8-6fa7-11ef-bcea-c7b262605968... 2024-09-10T19:33:44.592 INFO:journalctl@ceph.mgr.x.smithi191.stdout:Sep 10 19:33:44 smithi191 podman[164984]: 2024-09-10 19:33:44.280703206 +0000 UTC m=+0.118617639 container died 70400de28ce5b4cb0bc449ee51c920241cb369ae4214f43c28a3d7ce06bc20c2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3bfeaf4417585c821cab9ca76e4d95f390b0e769807122a45463640820b5cd16, name=ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-x, io.buildah.version=1.37.2, ceph=True, org.label-schema.license=GPLv2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, RELEASE=wip-guits-quincy-2024-09-09-1810-quincy-5be5739, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS) 2024-09-10T19:33:44.943 INFO:journalctl@ceph.mgr.x.smithi191.stdout:Sep 10 19:33:44 smithi191 podman[164984]: 2024-09-10 19:33:44.673847318 +0000 UTC m=+0.511761706 container cleanup 70400de28ce5b4cb0bc449ee51c920241cb369ae4214f43c28a3d7ce06bc20c2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3bfeaf4417585c821cab9ca76e4d95f390b0e769807122a45463640820b5cd16, name=ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-x, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, RELEASE=wip-guits-quincy-2024-09-09-1810-quincy-5be5739, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , ceph=True) 2024-09-10T19:33:44.943 INFO:journalctl@ceph.mgr.x.smithi191.stdout:Sep 10 19:33:44 smithi191 bash[164984]: ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-mgr-x 2024-09-10T19:33:45.120 DEBUG:teuthology.orchestra.run.smithi191:> sudo pkill -f 'journalctl -f -n 0 -u ceph-320f6df8-6fa7-11ef-bcea-c7b262605968@mgr.x.service' 2024-09-10T19:33:45.156 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-10T19:33:45.156 INFO:tasks.cephadm.mgr.x:Stopped mgr.x 2024-09-10T19:33:45.156 INFO:tasks.cephadm.osd.0:Stopping osd.0... 2024-09-10T19:33:45.156 DEBUG:teuthology.orchestra.run.smithi183:> sudo systemctl stop ceph-320f6df8-6fa7-11ef-bcea-c7b262605968@osd.0 2024-09-10T19:33:45.449 INFO:journalctl@ceph.osd.0.smithi183.stdout:Sep 10 19:33:45 smithi183 systemd[1]: Stopping Ceph osd.0 for 320f6df8-6fa7-11ef-bcea-c7b262605968... 2024-09-10T19:33:45.931 INFO:journalctl@ceph.osd.0.smithi183.stdout:Sep 10 19:33:45 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd-0[175367]: 2024-09-10T19:33:45.447+0000 7f44f4f07640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-10T19:33:45.931 INFO:journalctl@ceph.osd.0.smithi183.stdout:Sep 10 19:33:45 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd-0[175367]: 2024-09-10T19:33:45.447+0000 7f44f4f07640 -1 osd.0 461 *** Got signal Terminated *** 2024-09-10T19:33:45.931 INFO:journalctl@ceph.osd.0.smithi183.stdout:Sep 10 19:33:45 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd-0[175367]: 2024-09-10T19:33:45.447+0000 7f44f4f07640 -1 osd.0 461 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-10T19:33:51.337 INFO:journalctl@ceph.osd.0.smithi183.stdout:Sep 10 19:33:51 smithi183 podman[239294]: 2024-09-10 19:33:51.051315356 +0000 UTC m=+5.704263521 container died c228d1a8b2b6f4e54923e367c5ec632b01f507707e582a107994c566f8aac7db (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3bfeaf4417585c821cab9ca76e4d95f390b0e769807122a45463640820b5cd16, name=ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd-0, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, RELEASE=wip-guits-quincy-2024-09-09-1810-quincy-5be5739, GIT_BRANCH=HEAD, ceph=True) 2024-09-10T19:33:51.681 INFO:journalctl@ceph.osd.0.smithi183.stdout:Sep 10 19:33:51 smithi183 podman[239294]: 2024-09-10 19:33:51.55344151 +0000 UTC m=+6.206389674 container cleanup c228d1a8b2b6f4e54923e367c5ec632b01f507707e582a107994c566f8aac7db (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3bfeaf4417585c821cab9ca76e4d95f390b0e769807122a45463640820b5cd16, name=ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd-0, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , RELEASE=wip-guits-quincy-2024-09-09-1810-quincy-5be5739, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, GIT_CLEAN=True) 2024-09-10T19:33:51.681 INFO:journalctl@ceph.osd.0.smithi183.stdout:Sep 10 19:33:51 smithi183 bash[239294]: ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd-0 2024-09-10T19:33:52.182 INFO:journalctl@ceph.osd.0.smithi183.stdout:Sep 10 19:33:51 smithi183 podman[239314]: 2024-09-10 19:33:51.797051149 +0000 UTC m=+0.742243742 container remove c228d1a8b2b6f4e54923e367c5ec632b01f507707e582a107994c566f8aac7db (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3bfeaf4417585c821cab9ca76e4d95f390b0e769807122a45463640820b5cd16, name=ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd-0, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_BRANCH=HEAD, GIT_CLEAN=True, RELEASE=wip-guits-quincy-2024-09-09-1810-quincy-5be5739, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-10T19:33:52.501 INFO:journalctl@ceph.osd.0.smithi183.stdout:Sep 10 19:33:52 smithi183 podman[239411]: 2024-09-10 19:33:52.131519121 +0000 UTC m=+0.015605162 image pull 4c55dd4096115e48502f368cff1eab06769f71f030e1692e2611bbc6c6f637e9 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3bfeaf4417585c821cab9ca76e4d95f390b0e769807122a45463640820b5cd16 2024-09-10T19:33:52.502 INFO:journalctl@ceph.osd.0.smithi183.stdout:Sep 10 19:33:52 smithi183 podman[239411]: 2024-09-10 19:33:52.248149623 +0000 UTC m=+0.132235671 container create 1faf0f59b646a19ac08aed3db62fac6f852ae686362820d5e3049091a5bc2c41 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3bfeaf4417585c821cab9ca76e4d95f390b0e769807122a45463640820b5cd16, name=ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd-0-deactivate, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, RELEASE=wip-guits-quincy-2024-09-09-1810-quincy-5be5739, org.label-schema.schema-version=1.0, GIT_CLEAN=True, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2) 2024-09-10T19:33:52.932 INFO:journalctl@ceph.osd.0.smithi183.stdout:Sep 10 19:33:52 smithi183 podman[239411]: 2024-09-10 19:33:52.50065762 +0000 UTC m=+0.384743670 container init 1faf0f59b646a19ac08aed3db62fac6f852ae686362820d5e3049091a5bc2c41 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3bfeaf4417585c821cab9ca76e4d95f390b0e769807122a45463640820b5cd16, name=ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd-0-deactivate, ceph=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_CLEAN=True, RELEASE=wip-guits-quincy-2024-09-09-1810-quincy-5be5739, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=) 2024-09-10T19:33:52.932 INFO:journalctl@ceph.osd.0.smithi183.stdout:Sep 10 19:33:52 smithi183 podman[239411]: 2024-09-10 19:33:52.505578977 +0000 UTC m=+0.389665022 container start 1faf0f59b646a19ac08aed3db62fac6f852ae686362820d5e3049091a5bc2c41 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3bfeaf4417585c821cab9ca76e4d95f390b0e769807122a45463640820b5cd16, name=ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd-0-deactivate, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, GIT_CLEAN=True, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, RELEASE=wip-guits-quincy-2024-09-09-1810-quincy-5be5739, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7) 2024-09-10T19:33:52.932 INFO:journalctl@ceph.osd.0.smithi183.stdout:Sep 10 19:33:52 smithi183 podman[239411]: 2024-09-10 19:33:52.600469372 +0000 UTC m=+0.484555416 container attach 1faf0f59b646a19ac08aed3db62fac6f852ae686362820d5e3049091a5bc2c41 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3bfeaf4417585c821cab9ca76e4d95f390b0e769807122a45463640820b5cd16, name=ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd-0-deactivate, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.license=GPLv2, RELEASE=wip-guits-quincy-2024-09-09-1810-quincy-5be5739, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731) 2024-09-10T19:33:52.932 INFO:journalctl@ceph.osd.0.smithi183.stdout:Sep 10 19:33:52 smithi183 podman[239411]: 2024-09-10 19:33:52.723162299 +0000 UTC m=+0.607248341 container died 1faf0f59b646a19ac08aed3db62fac6f852ae686362820d5e3049091a5bc2c41 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3bfeaf4417585c821cab9ca76e4d95f390b0e769807122a45463640820b5cd16, name=ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd-0-deactivate, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_CLEAN=True, io.buildah.version=1.37.2, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, RELEASE=wip-guits-quincy-2024-09-09-1810-quincy-5be5739) 2024-09-10T19:33:53.586 DEBUG:teuthology.orchestra.run.smithi183:> sudo pkill -f 'journalctl -f -n 0 -u ceph-320f6df8-6fa7-11ef-bcea-c7b262605968@osd.0.service' 2024-09-10T19:33:53.629 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-10T19:33:53.630 INFO:tasks.cephadm.osd.0:Stopped osd.0 2024-09-10T19:33:53.630 INFO:tasks.cephadm.osd.1:Stopping osd.1... 2024-09-10T19:33:53.630 DEBUG:teuthology.orchestra.run.smithi183:> sudo systemctl stop ceph-320f6df8-6fa7-11ef-bcea-c7b262605968@osd.1 2024-09-10T19:33:53.862 INFO:journalctl@ceph.osd.1.smithi183.stdout:Sep 10 19:33:53 smithi183 systemd[1]: Stopping Ceph osd.1 for 320f6df8-6fa7-11ef-bcea-c7b262605968... 2024-09-10T19:33:54.181 INFO:journalctl@ceph.osd.1.smithi183.stdout:Sep 10 19:33:54 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd-1[161468]: 2024-09-10T19:33:54.009+0000 7f70ca930640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-10T19:33:54.182 INFO:journalctl@ceph.osd.1.smithi183.stdout:Sep 10 19:33:54 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd-1[161468]: 2024-09-10T19:33:54.009+0000 7f70ca930640 -1 osd.1 461 *** Got signal Terminated *** 2024-09-10T19:33:54.182 INFO:journalctl@ceph.osd.1.smithi183.stdout:Sep 10 19:33:54 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd-1[161468]: 2024-09-10T19:33:54.009+0000 7f70ca930640 -1 osd.1 461 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-10T19:33:59.681 INFO:journalctl@ceph.osd.1.smithi183.stdout:Sep 10 19:33:59 smithi183 podman[239545]: 2024-09-10 19:33:59.393533879 +0000 UTC m=+5.510657027 container died afd815a009427a6cbc46f247f0533f8b8953e92e817987b3d545be5faf222ca6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3bfeaf4417585c821cab9ca76e4d95f390b0e769807122a45463640820b5cd16, name=ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd-1, org.label-schema.license=GPLv2, RELEASE=wip-guits-quincy-2024-09-09-1810-quincy-5be5739, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, ceph=True, io.buildah.version=1.37.2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7) 2024-09-10T19:33:59.968 INFO:journalctl@ceph.osd.1.smithi183.stdout:Sep 10 19:33:59 smithi183 podman[239545]: 2024-09-10 19:33:59.949250592 +0000 UTC m=+6.066373740 container cleanup afd815a009427a6cbc46f247f0533f8b8953e92e817987b3d545be5faf222ca6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3bfeaf4417585c821cab9ca76e4d95f390b0e769807122a45463640820b5cd16, name=ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd-1, RELEASE=wip-guits-quincy-2024-09-09-1810-quincy-5be5739, io.buildah.version=1.37.2, ceph=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_CLEAN=True) 2024-09-10T19:33:59.968 INFO:journalctl@ceph.osd.1.smithi183.stdout:Sep 10 19:33:59 smithi183 bash[239545]: ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd-1 2024-09-10T19:34:00.566 INFO:journalctl@ceph.osd.1.smithi183.stdout:Sep 10 19:34:00 smithi183 podman[239576]: 2024-09-10 19:34:00.225960774 +0000 UTC m=+0.830719784 container remove afd815a009427a6cbc46f247f0533f8b8953e92e817987b3d545be5faf222ca6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3bfeaf4417585c821cab9ca76e4d95f390b0e769807122a45463640820b5cd16, name=ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd-1, org.label-schema.vendor=CentOS, RELEASE=wip-guits-quincy-2024-09-09-1810-quincy-5be5739, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_CLEAN=True, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux ) 2024-09-10T19:34:00.888 INFO:journalctl@ceph.osd.1.smithi183.stdout:Sep 10 19:34:00 smithi183 podman[239668]: 2024-09-10 19:34:00.468242683 +0000 UTC m=+0.015487214 image pull 4c55dd4096115e48502f368cff1eab06769f71f030e1692e2611bbc6c6f637e9 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3bfeaf4417585c821cab9ca76e4d95f390b0e769807122a45463640820b5cd16 2024-09-10T19:34:00.888 INFO:journalctl@ceph.osd.1.smithi183.stdout:Sep 10 19:34:00 smithi183 podman[239668]: 2024-09-10 19:34:00.568319263 +0000 UTC m=+0.115563781 container create dea6d68d4501abba87ee953669fc7a684a9161e3351f9b53aa55c0487d19d879 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3bfeaf4417585c821cab9ca76e4d95f390b0e769807122a45463640820b5cd16, name=ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd-1-deactivate, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, ceph=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_CLEAN=True, RELEASE=wip-guits-quincy-2024-09-09-1810-quincy-5be5739, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-10T19:34:00.888 INFO:journalctl@ceph.osd.1.smithi183.stdout:Sep 10 19:34:00 smithi183 podman[239668]: 2024-09-10 19:34:00.779005843 +0000 UTC m=+0.326250368 container init dea6d68d4501abba87ee953669fc7a684a9161e3351f9b53aa55c0487d19d879 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3bfeaf4417585c821cab9ca76e4d95f390b0e769807122a45463640820b5cd16, name=ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd-1-deactivate, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, RELEASE=wip-guits-quincy-2024-09-09-1810-quincy-5be5739, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.license=GPLv2) 2024-09-10T19:34:00.888 INFO:journalctl@ceph.osd.1.smithi183.stdout:Sep 10 19:34:00 smithi183 podman[239668]: 2024-09-10 19:34:00.784009919 +0000 UTC m=+0.331254437 container start dea6d68d4501abba87ee953669fc7a684a9161e3351f9b53aa55c0487d19d879 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3bfeaf4417585c821cab9ca76e4d95f390b0e769807122a45463640820b5cd16, name=ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd-1-deactivate, GIT_CLEAN=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=wip-guits-quincy-2024-09-09-1810-quincy-5be5739, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, ceph=True) 2024-09-10T19:34:00.889 INFO:journalctl@ceph.osd.1.smithi183.stdout:Sep 10 19:34:00 smithi183 podman[239668]: 2024-09-10 19:34:00.8871878 +0000 UTC m=+0.434432338 container attach dea6d68d4501abba87ee953669fc7a684a9161e3351f9b53aa55c0487d19d879 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3bfeaf4417585c821cab9ca76e4d95f390b0e769807122a45463640820b5cd16, name=ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd-1-deactivate, GIT_BRANCH=HEAD, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, GIT_CLEAN=True, RELEASE=wip-guits-quincy-2024-09-09-1810-quincy-5be5739, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2) 2024-09-10T19:34:01.181 INFO:journalctl@ceph.osd.1.smithi183.stdout:Sep 10 19:34:00 smithi183 podman[239668]: 2024-09-10 19:34:00.957036813 +0000 UTC m=+0.504281341 container died dea6d68d4501abba87ee953669fc7a684a9161e3351f9b53aa55c0487d19d879 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3bfeaf4417585c821cab9ca76e4d95f390b0e769807122a45463640820b5cd16, name=ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd-1-deactivate, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.vendor=CentOS, RELEASE=wip-guits-quincy-2024-09-09-1810-quincy-5be5739, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-10T19:34:01.714 DEBUG:teuthology.orchestra.run.smithi183:> sudo pkill -f 'journalctl -f -n 0 -u ceph-320f6df8-6fa7-11ef-bcea-c7b262605968@osd.1.service' 2024-09-10T19:34:01.759 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-10T19:34:01.760 INFO:tasks.cephadm.osd.1:Stopped osd.1 2024-09-10T19:34:01.760 INFO:tasks.cephadm.osd.2:Stopping osd.2... 2024-09-10T19:34:01.760 DEBUG:teuthology.orchestra.run.smithi183:> sudo systemctl stop ceph-320f6df8-6fa7-11ef-bcea-c7b262605968@osd.2 2024-09-10T19:34:02.090 INFO:journalctl@ceph.osd.2.smithi183.stdout:Sep 10 19:34:01 smithi183 systemd[1]: Stopping Ceph osd.2 for 320f6df8-6fa7-11ef-bcea-c7b262605968... 2024-09-10T19:34:02.431 INFO:journalctl@ceph.osd.2.smithi183.stdout:Sep 10 19:34:02 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd-2[58691]: 2024-09-10T19:34:02.087+0000 7f395f2c5640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-10T19:34:02.432 INFO:journalctl@ceph.osd.2.smithi183.stdout:Sep 10 19:34:02 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd-2[58691]: 2024-09-10T19:34:02.087+0000 7f395f2c5640 -1 osd.2 461 *** Got signal Terminated *** 2024-09-10T19:34:02.432 INFO:journalctl@ceph.osd.2.smithi183.stdout:Sep 10 19:34:02 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd-2[58691]: 2024-09-10T19:34:02.087+0000 7f395f2c5640 -1 osd.2 461 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-10T19:34:07.931 INFO:journalctl@ceph.osd.2.smithi183.stdout:Sep 10 19:34:07 smithi183 podman[239798]: 2024-09-10 19:34:07.629517872 +0000 UTC m=+5.666906919 container died e335e2a9f798024aac1c7e2bc213c2f515127e12d9b2e698ce33287d5269a6d5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3bfeaf4417585c821cab9ca76e4d95f390b0e769807122a45463640820b5cd16, name=ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd-2, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-guits-quincy-2024-09-09-1810-quincy-5be5739, org.label-schema.build-date=20240731, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.schema-version=1.0, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2) 2024-09-10T19:34:08.354 INFO:journalctl@ceph.osd.2.smithi183.stdout:Sep 10 19:34:08 smithi183 podman[239798]: 2024-09-10 19:34:08.16004697 +0000 UTC m=+6.197436009 container cleanup e335e2a9f798024aac1c7e2bc213c2f515127e12d9b2e698ce33287d5269a6d5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3bfeaf4417585c821cab9ca76e4d95f390b0e769807122a45463640820b5cd16, name=ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd-2, org.label-schema.build-date=20240731, GIT_CLEAN=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=wip-guits-quincy-2024-09-09-1810-quincy-5be5739) 2024-09-10T19:34:08.355 INFO:journalctl@ceph.osd.2.smithi183.stdout:Sep 10 19:34:08 smithi183 bash[239798]: ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd-2 2024-09-10T19:34:08.681 INFO:journalctl@ceph.osd.2.smithi183.stdout:Sep 10 19:34:08 smithi183 podman[239819]: 2024-09-10 19:34:08.353092816 +0000 UTC m=+0.723595743 container remove e335e2a9f798024aac1c7e2bc213c2f515127e12d9b2e698ce33287d5269a6d5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3bfeaf4417585c821cab9ca76e4d95f390b0e769807122a45463640820b5cd16, name=ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd-2, org.label-schema.vendor=CentOS, RELEASE=wip-guits-quincy-2024-09-09-1810-quincy-5be5739, GIT_BRANCH=HEAD, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_CLEAN=True, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-10T19:34:09.107 INFO:journalctl@ceph.osd.2.smithi183.stdout:Sep 10 19:34:08 smithi183 podman[239910]: 2024-09-10 19:34:08.667914337 +0000 UTC m=+0.015957429 image pull 4c55dd4096115e48502f368cff1eab06769f71f030e1692e2611bbc6c6f637e9 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3bfeaf4417585c821cab9ca76e4d95f390b0e769807122a45463640820b5cd16 2024-09-10T19:34:09.108 INFO:journalctl@ceph.osd.2.smithi183.stdout:Sep 10 19:34:08 smithi183 podman[239910]: 2024-09-10 19:34:08.770941886 +0000 UTC m=+0.118984982 container create a9911b99d2b28ee4ff2c77a999e973b3cc32681a586b3c6378e1dd1f5419892b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3bfeaf4417585c821cab9ca76e4d95f390b0e769807122a45463640820b5cd16, name=ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd-2-deactivate, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, RELEASE=wip-guits-quincy-2024-09-09-1810-quincy-5be5739, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-10T19:34:09.108 INFO:journalctl@ceph.osd.2.smithi183.stdout:Sep 10 19:34:08 smithi183 podman[239910]: 2024-09-10 19:34:08.998397219 +0000 UTC m=+0.346440319 container init a9911b99d2b28ee4ff2c77a999e973b3cc32681a586b3c6378e1dd1f5419892b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3bfeaf4417585c821cab9ca76e4d95f390b0e769807122a45463640820b5cd16, name=ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd-2-deactivate, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, io.buildah.version=1.37.2, RELEASE=wip-guits-quincy-2024-09-09-1810-quincy-5be5739, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, GIT_CLEAN=True) 2024-09-10T19:34:09.108 INFO:journalctl@ceph.osd.2.smithi183.stdout:Sep 10 19:34:09 smithi183 podman[239910]: 2024-09-10 19:34:09.003392879 +0000 UTC m=+0.351435975 container start a9911b99d2b28ee4ff2c77a999e973b3cc32681a586b3c6378e1dd1f5419892b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3bfeaf4417585c821cab9ca76e4d95f390b0e769807122a45463640820b5cd16, name=ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd-2-deactivate, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, io.buildah.version=1.37.2, ceph=True, CEPH_POINT_RELEASE=, RELEASE=wip-guits-quincy-2024-09-09-1810-quincy-5be5739, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2) 2024-09-10T19:34:09.108 INFO:journalctl@ceph.osd.2.smithi183.stdout:Sep 10 19:34:09 smithi183 podman[239910]: 2024-09-10 19:34:09.106600168 +0000 UTC m=+0.454643278 container attach a9911b99d2b28ee4ff2c77a999e973b3cc32681a586b3c6378e1dd1f5419892b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3bfeaf4417585c821cab9ca76e4d95f390b0e769807122a45463640820b5cd16, name=ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd-2-deactivate, RELEASE=wip-guits-quincy-2024-09-09-1810-quincy-5be5739, GIT_CLEAN=True, CEPH_POINT_RELEASE=, ceph=True, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2) 2024-09-10T19:34:09.431 INFO:journalctl@ceph.osd.2.smithi183.stdout:Sep 10 19:34:09 smithi183 podman[239910]: 2024-09-10 19:34:09.176868764 +0000 UTC m=+0.524911857 container died a9911b99d2b28ee4ff2c77a999e973b3cc32681a586b3c6378e1dd1f5419892b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3bfeaf4417585c821cab9ca76e4d95f390b0e769807122a45463640820b5cd16, name=ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd-2-deactivate, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, RELEASE=wip-guits-quincy-2024-09-09-1810-quincy-5be5739, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.vendor=CentOS) 2024-09-10T19:34:09.760 DEBUG:teuthology.orchestra.run.smithi183:> sudo pkill -f 'journalctl -f -n 0 -u ceph-320f6df8-6fa7-11ef-bcea-c7b262605968@osd.2.service' 2024-09-10T19:34:09.800 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-10T19:34:09.800 INFO:tasks.cephadm.osd.2:Stopped osd.2 2024-09-10T19:34:09.800 INFO:tasks.cephadm.osd.3:Stopping osd.3... 2024-09-10T19:34:09.800 DEBUG:teuthology.orchestra.run.smithi183:> sudo systemctl stop ceph-320f6df8-6fa7-11ef-bcea-c7b262605968@osd.3 2024-09-10T19:34:10.141 INFO:journalctl@ceph.osd.3.smithi183.stdout:Sep 10 19:34:09 smithi183 systemd[1]: Stopping Ceph osd.3 for 320f6df8-6fa7-11ef-bcea-c7b262605968... 2024-09-10T19:34:10.431 INFO:journalctl@ceph.osd.3.smithi183.stdout:Sep 10 19:34:10 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd-3[65168]: 2024-09-10T19:34:10.138+0000 7f3535326640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-10T19:34:10.432 INFO:journalctl@ceph.osd.3.smithi183.stdout:Sep 10 19:34:10 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd-3[65168]: 2024-09-10T19:34:10.138+0000 7f3535326640 -1 osd.3 461 *** Got signal Terminated *** 2024-09-10T19:34:10.432 INFO:journalctl@ceph.osd.3.smithi183.stdout:Sep 10 19:34:10 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd-3[65168]: 2024-09-10T19:34:10.138+0000 7f3535326640 -1 osd.3 461 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-10T19:34:12.385 INFO:journalctl@ceph.osd.7.smithi191.stdout:Sep 10 19:34:12 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd-7[56482]: 2024-09-10T19:34:12.115+0000 7f9ef7a9a640 -1 osd.7 461 heartbeat_check: no reply from 172.21.15.183:6814 osd.0 since back 2024-09-10T19:33:48.004787+0000 front 2024-09-10T19:33:48.004721+0000 (oldest deadline 2024-09-10T19:34:11.504575+0000) 2024-09-10T19:34:13.401 INFO:journalctl@ceph.osd.7.smithi191.stdout:Sep 10 19:34:13 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd-7[56482]: 2024-09-10T19:34:13.077+0000 7f9ef7a9a640 -1 osd.7 461 heartbeat_check: no reply from 172.21.15.183:6814 osd.0 since back 2024-09-10T19:33:48.004787+0000 front 2024-09-10T19:33:48.004721+0000 (oldest deadline 2024-09-10T19:34:11.504575+0000) 2024-09-10T19:34:13.659 INFO:journalctl@ceph.osd.5.smithi191.stdout:Sep 10 19:34:13 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd-5[44138]: 2024-09-10T19:34:13.531+0000 7f7dce955640 -1 osd.5 461 heartbeat_check: no reply from 172.21.15.183:6814 osd.0 since back 2024-09-10T19:33:48.769215+0000 front 2024-09-10T19:33:48.769237+0000 (oldest deadline 2024-09-10T19:34:12.869339+0000) 2024-09-10T19:34:14.465 INFO:journalctl@ceph.osd.7.smithi191.stdout:Sep 10 19:34:14 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd-7[56482]: 2024-09-10T19:34:14.054+0000 7f9ef7a9a640 -1 osd.7 461 heartbeat_check: no reply from 172.21.15.183:6814 osd.0 since back 2024-09-10T19:33:48.004787+0000 front 2024-09-10T19:33:48.004721+0000 (oldest deadline 2024-09-10T19:34:11.504575+0000) 2024-09-10T19:34:14.949 INFO:journalctl@ceph.osd.5.smithi191.stdout:Sep 10 19:34:14 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd-5[44138]: 2024-09-10T19:34:14.491+0000 7f7dce955640 -1 osd.5 461 heartbeat_check: no reply from 172.21.15.183:6814 osd.0 since back 2024-09-10T19:33:48.769215+0000 front 2024-09-10T19:33:48.769237+0000 (oldest deadline 2024-09-10T19:34:12.869339+0000) 2024-09-10T19:34:15.216 INFO:journalctl@ceph.osd.7.smithi191.stdout:Sep 10 19:34:15 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd-7[56482]: 2024-09-10T19:34:15.034+0000 7f9ef7a9a640 -1 osd.7 461 heartbeat_check: no reply from 172.21.15.183:6814 osd.0 since back 2024-09-10T19:33:48.004787+0000 front 2024-09-10T19:33:48.004721+0000 (oldest deadline 2024-09-10T19:34:11.504575+0000) 2024-09-10T19:34:15.216 INFO:journalctl@ceph.osd.4.smithi191.stdout:Sep 10 19:34:14 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd-4[38197]: 2024-09-10T19:34:14.946+0000 7fb6016fb640 -1 osd.4 461 heartbeat_check: no reply from 172.21.15.183:6814 osd.0 since back 2024-09-10T19:33:49.868845+0000 front 2024-09-10T19:33:49.868952+0000 (oldest deadline 2024-09-10T19:34:13.968709+0000) 2024-09-10T19:34:15.716 INFO:journalctl@ceph.osd.5.smithi191.stdout:Sep 10 19:34:15 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd-5[44138]: 2024-09-10T19:34:15.504+0000 7f7dce955640 -1 osd.5 461 heartbeat_check: no reply from 172.21.15.183:6814 osd.0 since back 2024-09-10T19:33:48.769215+0000 front 2024-09-10T19:33:48.769237+0000 (oldest deadline 2024-09-10T19:34:12.869339+0000) 2024-09-10T19:34:15.716 INFO:journalctl@ceph.osd.6.smithi191.stdout:Sep 10 19:34:15 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd-6[50347]: 2024-09-10T19:34:15.315+0000 7f4ab7287640 -1 osd.6 461 heartbeat_check: no reply from 172.21.15.183:6814 osd.0 since back 2024-09-10T19:33:45.563941+0000 front 2024-09-10T19:33:45.563894+0000 (oldest deadline 2024-09-10T19:34:14.364335+0000) 2024-09-10T19:34:16.122 INFO:journalctl@ceph.osd.3.smithi183.stdout:Sep 10 19:34:15 smithi183 podman[240036]: 2024-09-10 19:34:15.844557504 +0000 UTC m=+5.824478417 container died 373f444e26283f5c705379038cb7ed08c7d58be6072f8a2f43f9d2abc291d68a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3bfeaf4417585c821cab9ca76e4d95f390b0e769807122a45463640820b5cd16, name=ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd-3, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.license=GPLv2, RELEASE=wip-guits-quincy-2024-09-09-1810-quincy-5be5739, GIT_BRANCH=HEAD, ceph=True, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True) 2024-09-10T19:34:16.216 INFO:journalctl@ceph.osd.4.smithi191.stdout:Sep 10 19:34:15 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd-4[38197]: 2024-09-10T19:34:15.905+0000 7fb6016fb640 -1 osd.4 461 heartbeat_check: no reply from 172.21.15.183:6814 osd.0 since back 2024-09-10T19:33:49.868845+0000 front 2024-09-10T19:33:49.868952+0000 (oldest deadline 2024-09-10T19:34:13.968709+0000) 2024-09-10T19:34:16.216 INFO:journalctl@ceph.osd.7.smithi191.stdout:Sep 10 19:34:16 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd-7[56482]: 2024-09-10T19:34:16.010+0000 7f9ef7a9a640 -1 osd.7 461 heartbeat_check: no reply from 172.21.15.183:6814 osd.0 since back 2024-09-10T19:33:48.004787+0000 front 2024-09-10T19:33:48.004721+0000 (oldest deadline 2024-09-10T19:34:11.504575+0000) 2024-09-10T19:34:16.431 INFO:journalctl@ceph.osd.3.smithi183.stdout:Sep 10 19:34:16 smithi183 podman[240036]: 2024-09-10 19:34:16.295457218 +0000 UTC m=+6.275378173 container cleanup 373f444e26283f5c705379038cb7ed08c7d58be6072f8a2f43f9d2abc291d68a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3bfeaf4417585c821cab9ca76e4d95f390b0e769807122a45463640820b5cd16, name=ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd-3, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, RELEASE=wip-guits-quincy-2024-09-09-1810-quincy-5be5739, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_CLEAN=True, io.buildah.version=1.37.2) 2024-09-10T19:34:16.431 INFO:journalctl@ceph.osd.3.smithi183.stdout:Sep 10 19:34:16 smithi183 bash[240036]: ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd-3 2024-09-10T19:34:16.716 INFO:journalctl@ceph.osd.5.smithi191.stdout:Sep 10 19:34:16 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd-5[44138]: 2024-09-10T19:34:16.507+0000 7f7dce955640 -1 osd.5 461 heartbeat_check: no reply from 172.21.15.183:6814 osd.0 since back 2024-09-10T19:33:48.769215+0000 front 2024-09-10T19:33:48.769237+0000 (oldest deadline 2024-09-10T19:34:12.869339+0000) 2024-09-10T19:34:16.716 INFO:journalctl@ceph.osd.6.smithi191.stdout:Sep 10 19:34:16 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd-6[50347]: 2024-09-10T19:34:16.343+0000 7f4ab7287640 -1 osd.6 461 heartbeat_check: no reply from 172.21.15.183:6814 osd.0 since back 2024-09-10T19:33:45.563941+0000 front 2024-09-10T19:33:45.563894+0000 (oldest deadline 2024-09-10T19:34:14.364335+0000) 2024-09-10T19:34:16.914 INFO:journalctl@ceph.osd.3.smithi183.stdout:Sep 10 19:34:16 smithi183 podman[240058]: 2024-09-10 19:34:16.580225045 +0000 UTC m=+0.732452134 container remove 373f444e26283f5c705379038cb7ed08c7d58be6072f8a2f43f9d2abc291d68a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3bfeaf4417585c821cab9ca76e4d95f390b0e769807122a45463640820b5cd16, name=ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd-3, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, maintainer=Guillaume Abrioux , GIT_CLEAN=True, RELEASE=wip-guits-quincy-2024-09-09-1810-quincy-5be5739) 2024-09-10T19:34:17.182 INFO:journalctl@ceph.osd.3.smithi183.stdout:Sep 10 19:34:16 smithi183 podman[240153]: 2024-09-10 19:34:16.816024783 +0000 UTC m=+0.015727524 image pull 4c55dd4096115e48502f368cff1eab06769f71f030e1692e2611bbc6c6f637e9 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3bfeaf4417585c821cab9ca76e4d95f390b0e769807122a45463640820b5cd16 2024-09-10T19:34:17.182 INFO:journalctl@ceph.osd.3.smithi183.stdout:Sep 10 19:34:16 smithi183 podman[240153]: 2024-09-10 19:34:16.92267936 +0000 UTC m=+0.122382101 container create a747c5db6fb9d33315fc13b674d43ac3d36b6a748ce9c13082be7a8206e314ce (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3bfeaf4417585c821cab9ca76e4d95f390b0e769807122a45463640820b5cd16, name=ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd-3-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.build-date=20240731, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.license=GPLv2, GIT_CLEAN=True, maintainer=Guillaume Abrioux , RELEASE=wip-guits-quincy-2024-09-09-1810-quincy-5be5739, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, io.buildah.version=1.37.2) 2024-09-10T19:34:17.182 INFO:journalctl@ceph.osd.3.smithi183.stdout:Sep 10 19:34:17 smithi183 podman[240153]: 2024-09-10 19:34:17.133296146 +0000 UTC m=+0.332998884 container init a747c5db6fb9d33315fc13b674d43ac3d36b6a748ce9c13082be7a8206e314ce (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3bfeaf4417585c821cab9ca76e4d95f390b0e769807122a45463640820b5cd16, name=ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd-3-deactivate, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, ceph=True, CEPH_POINT_RELEASE=, RELEASE=wip-guits-quincy-2024-09-09-1810-quincy-5be5739, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7) 2024-09-10T19:34:17.182 INFO:journalctl@ceph.osd.3.smithi183.stdout:Sep 10 19:34:17 smithi183 podman[240153]: 2024-09-10 19:34:17.138375452 +0000 UTC m=+0.338078192 container start a747c5db6fb9d33315fc13b674d43ac3d36b6a748ce9c13082be7a8206e314ce (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3bfeaf4417585c821cab9ca76e4d95f390b0e769807122a45463640820b5cd16, name=ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd-3-deactivate, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, RELEASE=wip-guits-quincy-2024-09-09-1810-quincy-5be5739, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, ceph=True, maintainer=Guillaume Abrioux ) 2024-09-10T19:34:17.216 INFO:journalctl@ceph.osd.4.smithi191.stdout:Sep 10 19:34:16 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd-4[38197]: 2024-09-10T19:34:16.919+0000 7fb6016fb640 -1 osd.4 461 heartbeat_check: no reply from 172.21.15.183:6814 osd.0 since back 2024-09-10T19:33:49.868845+0000 front 2024-09-10T19:33:49.868952+0000 (oldest deadline 2024-09-10T19:34:13.968709+0000) 2024-09-10T19:34:17.216 INFO:journalctl@ceph.osd.7.smithi191.stdout:Sep 10 19:34:17 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd-7[56482]: 2024-09-10T19:34:17.020+0000 7f9ef7a9a640 -1 osd.7 461 heartbeat_check: no reply from 172.21.15.183:6814 osd.0 since back 2024-09-10T19:33:48.004787+0000 front 2024-09-10T19:33:48.004721+0000 (oldest deadline 2024-09-10T19:34:11.504575+0000) 2024-09-10T19:34:17.661 INFO:journalctl@ceph.osd.3.smithi183.stdout:Sep 10 19:34:17 smithi183 podman[240153]: 2024-09-10 19:34:17.224748305 +0000 UTC m=+0.424451051 container attach a747c5db6fb9d33315fc13b674d43ac3d36b6a748ce9c13082be7a8206e314ce (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3bfeaf4417585c821cab9ca76e4d95f390b0e769807122a45463640820b5cd16, name=ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd-3-deactivate, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_CLEAN=True, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=wip-guits-quincy-2024-09-09-1810-quincy-5be5739, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7) 2024-09-10T19:34:17.662 INFO:journalctl@ceph.osd.3.smithi183.stdout:Sep 10 19:34:17 smithi183 podman[240153]: 2024-09-10 19:34:17.315448201 +0000 UTC m=+0.515150942 container died a747c5db6fb9d33315fc13b674d43ac3d36b6a748ce9c13082be7a8206e314ce (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3bfeaf4417585c821cab9ca76e4d95f390b0e769807122a45463640820b5cd16, name=ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd-3-deactivate, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, ceph=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-guits-quincy-2024-09-09-1810-quincy-5be5739) 2024-09-10T19:34:17.715 INFO:journalctl@ceph.osd.5.smithi191.stdout:Sep 10 19:34:17 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd-5[44138]: 2024-09-10T19:34:17.511+0000 7f7dce955640 -1 osd.5 461 heartbeat_check: no reply from 172.21.15.183:6814 osd.0 since back 2024-09-10T19:33:48.769215+0000 front 2024-09-10T19:33:48.769237+0000 (oldest deadline 2024-09-10T19:34:12.869339+0000) 2024-09-10T19:34:17.716 INFO:journalctl@ceph.osd.6.smithi191.stdout:Sep 10 19:34:17 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd-6[50347]: 2024-09-10T19:34:17.310+0000 7f4ab7287640 -1 osd.6 461 heartbeat_check: no reply from 172.21.15.183:6814 osd.0 since back 2024-09-10T19:33:45.563941+0000 front 2024-09-10T19:33:45.563894+0000 (oldest deadline 2024-09-10T19:34:14.364335+0000) 2024-09-10T19:34:18.058 DEBUG:teuthology.orchestra.run.smithi183:> sudo pkill -f 'journalctl -f -n 0 -u ceph-320f6df8-6fa7-11ef-bcea-c7b262605968@osd.3.service' 2024-09-10T19:34:18.096 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-10T19:34:18.097 INFO:tasks.cephadm.osd.3:Stopped osd.3 2024-09-10T19:34:18.097 INFO:tasks.cephadm.osd.4:Stopping osd.4... 2024-09-10T19:34:18.097 DEBUG:teuthology.orchestra.run.smithi191:> sudo systemctl stop ceph-320f6df8-6fa7-11ef-bcea-c7b262605968@osd.4 2024-09-10T19:34:18.340 INFO:journalctl@ceph.osd.4.smithi191.stdout:Sep 10 19:34:17 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd-4[38197]: 2024-09-10T19:34:17.964+0000 7fb6016fb640 -1 osd.4 461 heartbeat_check: no reply from 172.21.15.183:6814 osd.0 since back 2024-09-10T19:33:49.868845+0000 front 2024-09-10T19:33:49.868952+0000 (oldest deadline 2024-09-10T19:34:13.968709+0000) 2024-09-10T19:34:18.340 INFO:journalctl@ceph.osd.4.smithi191.stdout:Sep 10 19:34:18 smithi191 systemd[1]: Stopping Ceph osd.4 for 320f6df8-6fa7-11ef-bcea-c7b262605968... 2024-09-10T19:34:18.340 INFO:journalctl@ceph.osd.7.smithi191.stdout:Sep 10 19:34:18 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd-7[56482]: 2024-09-10T19:34:18.039+0000 7f9ef7a9a640 -1 osd.7 461 heartbeat_check: no reply from 172.21.15.183:6814 osd.0 since back 2024-09-10T19:33:48.004787+0000 front 2024-09-10T19:33:48.004721+0000 (oldest deadline 2024-09-10T19:34:11.504575+0000) 2024-09-10T19:34:18.716 INFO:journalctl@ceph.osd.5.smithi191.stdout:Sep 10 19:34:18 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd-5[44138]: 2024-09-10T19:34:18.535+0000 7f7dce955640 -1 osd.5 461 heartbeat_check: no reply from 172.21.15.183:6814 osd.0 since back 2024-09-10T19:33:48.769215+0000 front 2024-09-10T19:33:48.769237+0000 (oldest deadline 2024-09-10T19:34:12.869339+0000) 2024-09-10T19:34:18.716 INFO:journalctl@ceph.osd.6.smithi191.stdout:Sep 10 19:34:18 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd-6[50347]: 2024-09-10T19:34:18.337+0000 7f4ab7287640 -1 osd.6 461 heartbeat_check: no reply from 172.21.15.183:6814 osd.0 since back 2024-09-10T19:33:45.563941+0000 front 2024-09-10T19:33:45.563894+0000 (oldest deadline 2024-09-10T19:34:14.364335+0000) 2024-09-10T19:34:18.716 INFO:journalctl@ceph.osd.4.smithi191.stdout:Sep 10 19:34:18 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd-4[38197]: 2024-09-10T19:34:18.384+0000 7fb6058e4640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-10T19:34:18.716 INFO:journalctl@ceph.osd.4.smithi191.stdout:Sep 10 19:34:18 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd-4[38197]: 2024-09-10T19:34:18.384+0000 7fb6058e4640 -1 osd.4 461 *** Got signal Terminated *** 2024-09-10T19:34:18.717 INFO:journalctl@ceph.osd.4.smithi191.stdout:Sep 10 19:34:18 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd-4[38197]: 2024-09-10T19:34:18.384+0000 7fb6058e4640 -1 osd.4 461 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-10T19:34:19.324 INFO:journalctl@ceph.osd.4.smithi191.stdout:Sep 10 19:34:18 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd-4[38197]: 2024-09-10T19:34:18.976+0000 7fb6016fb640 -1 osd.4 461 heartbeat_check: no reply from 172.21.15.183:6814 osd.0 since back 2024-09-10T19:33:49.868845+0000 front 2024-09-10T19:33:49.868952+0000 (oldest deadline 2024-09-10T19:34:13.968709+0000) 2024-09-10T19:34:19.325 INFO:journalctl@ceph.osd.7.smithi191.stdout:Sep 10 19:34:19 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd-7[56482]: 2024-09-10T19:34:19.077+0000 7f9ef7a9a640 -1 osd.7 461 heartbeat_check: no reply from 172.21.15.183:6814 osd.0 since back 2024-09-10T19:33:48.004787+0000 front 2024-09-10T19:33:48.004721+0000 (oldest deadline 2024-09-10T19:34:11.504575+0000) 2024-09-10T19:34:19.715 INFO:journalctl@ceph.osd.5.smithi191.stdout:Sep 10 19:34:19 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd-5[44138]: 2024-09-10T19:34:19.531+0000 7f7dce955640 -1 osd.5 461 heartbeat_check: no reply from 172.21.15.183:6814 osd.0 since back 2024-09-10T19:33:48.769215+0000 front 2024-09-10T19:33:48.769237+0000 (oldest deadline 2024-09-10T19:34:12.869339+0000) 2024-09-10T19:34:19.716 INFO:journalctl@ceph.osd.6.smithi191.stdout:Sep 10 19:34:19 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd-6[50347]: 2024-09-10T19:34:19.322+0000 7f4ab7287640 -1 osd.6 461 heartbeat_check: no reply from 172.21.15.183:6814 osd.0 since back 2024-09-10T19:33:45.563941+0000 front 2024-09-10T19:33:45.563894+0000 (oldest deadline 2024-09-10T19:34:14.364335+0000) 2024-09-10T19:34:20.215 INFO:journalctl@ceph.osd.4.smithi191.stdout:Sep 10 19:34:19 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd-4[38197]: 2024-09-10T19:34:19.960+0000 7fb6016fb640 -1 osd.4 461 heartbeat_check: no reply from 172.21.15.183:6814 osd.0 since back 2024-09-10T19:33:49.868845+0000 front 2024-09-10T19:33:49.868952+0000 (oldest deadline 2024-09-10T19:34:13.968709+0000) 2024-09-10T19:34:20.216 INFO:journalctl@ceph.osd.7.smithi191.stdout:Sep 10 19:34:20 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd-7[56482]: 2024-09-10T19:34:20.055+0000 7f9ef7a9a640 -1 osd.7 461 heartbeat_check: no reply from 172.21.15.183:6814 osd.0 since back 2024-09-10T19:33:48.004787+0000 front 2024-09-10T19:33:48.004721+0000 (oldest deadline 2024-09-10T19:34:11.504575+0000) 2024-09-10T19:34:20.567 INFO:journalctl@ceph.osd.6.smithi191.stdout:Sep 10 19:34:20 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd-6[50347]: 2024-09-10T19:34:20.305+0000 7f4ab7287640 -1 osd.6 461 heartbeat_check: no reply from 172.21.15.183:6814 osd.0 since back 2024-09-10T19:33:45.563941+0000 front 2024-09-10T19:33:45.563894+0000 (oldest deadline 2024-09-10T19:34:14.364335+0000) 2024-09-10T19:34:20.567 INFO:journalctl@ceph.osd.6.smithi191.stdout:Sep 10 19:34:20 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd-6[50347]: 2024-09-10T19:34:20.305+0000 7f4ab7287640 -1 osd.6 461 heartbeat_check: no reply from 172.21.15.183:6806 osd.1 since back 2024-09-10T19:33:54.364736+0000 front 2024-09-10T19:33:54.364820+0000 (oldest deadline 2024-09-10T19:34:20.264938+0000) 2024-09-10T19:34:20.965 INFO:journalctl@ceph.osd.5.smithi191.stdout:Sep 10 19:34:20 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd-5[44138]: 2024-09-10T19:34:20.565+0000 7f7dce955640 -1 osd.5 461 heartbeat_check: no reply from 172.21.15.183:6814 osd.0 since back 2024-09-10T19:33:48.769215+0000 front 2024-09-10T19:33:48.769237+0000 (oldest deadline 2024-09-10T19:34:12.869339+0000) 2024-09-10T19:34:21.345 INFO:journalctl@ceph.osd.4.smithi191.stdout:Sep 10 19:34:20 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd-4[38197]: 2024-09-10T19:34:20.995+0000 7fb6016fb640 -1 osd.4 461 heartbeat_check: no reply from 172.21.15.183:6814 osd.0 since back 2024-09-10T19:33:49.868845+0000 front 2024-09-10T19:33:49.868952+0000 (oldest deadline 2024-09-10T19:34:13.968709+0000) 2024-09-10T19:34:21.346 INFO:journalctl@ceph.osd.7.smithi191.stdout:Sep 10 19:34:21 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd-7[56482]: 2024-09-10T19:34:21.083+0000 7f9ef7a9a640 -1 osd.7 461 heartbeat_check: no reply from 172.21.15.183:6814 osd.0 since back 2024-09-10T19:33:48.004787+0000 front 2024-09-10T19:33:48.004721+0000 (oldest deadline 2024-09-10T19:34:11.504575+0000) 2024-09-10T19:34:21.716 INFO:journalctl@ceph.osd.5.smithi191.stdout:Sep 10 19:34:21 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd-5[44138]: 2024-09-10T19:34:21.551+0000 7f7dce955640 -1 osd.5 461 heartbeat_check: no reply from 172.21.15.183:6814 osd.0 since back 2024-09-10T19:33:48.769215+0000 front 2024-09-10T19:33:48.769237+0000 (oldest deadline 2024-09-10T19:34:12.869339+0000) 2024-09-10T19:34:21.716 INFO:journalctl@ceph.osd.6.smithi191.stdout:Sep 10 19:34:21 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd-6[50347]: 2024-09-10T19:34:21.342+0000 7f4ab7287640 -1 osd.6 461 heartbeat_check: no reply from 172.21.15.183:6814 osd.0 since back 2024-09-10T19:33:45.563941+0000 front 2024-09-10T19:33:45.563894+0000 (oldest deadline 2024-09-10T19:34:14.364335+0000) 2024-09-10T19:34:21.716 INFO:journalctl@ceph.osd.6.smithi191.stdout:Sep 10 19:34:21 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd-6[50347]: 2024-09-10T19:34:21.342+0000 7f4ab7287640 -1 osd.6 461 heartbeat_check: no reply from 172.21.15.183:6806 osd.1 since back 2024-09-10T19:33:54.364736+0000 front 2024-09-10T19:33:54.364820+0000 (oldest deadline 2024-09-10T19:34:20.264938+0000) 2024-09-10T19:34:22.334 INFO:journalctl@ceph.osd.4.smithi191.stdout:Sep 10 19:34:21 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd-4[38197]: 2024-09-10T19:34:21.997+0000 7fb6016fb640 -1 osd.4 461 heartbeat_check: no reply from 172.21.15.183:6814 osd.0 since back 2024-09-10T19:33:49.868845+0000 front 2024-09-10T19:33:49.868952+0000 (oldest deadline 2024-09-10T19:34:13.968709+0000) 2024-09-10T19:34:22.334 INFO:journalctl@ceph.osd.7.smithi191.stdout:Sep 10 19:34:22 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd-7[56482]: 2024-09-10T19:34:22.045+0000 7f9ef7a9a640 -1 osd.7 461 heartbeat_check: no reply from 172.21.15.183:6814 osd.0 since back 2024-09-10T19:33:48.004787+0000 front 2024-09-10T19:33:48.004721+0000 (oldest deadline 2024-09-10T19:34:11.504575+0000) 2024-09-10T19:34:22.589 INFO:journalctl@ceph.osd.6.smithi191.stdout:Sep 10 19:34:22 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd-6[50347]: 2024-09-10T19:34:22.332+0000 7f4ab7287640 -1 osd.6 461 heartbeat_check: no reply from 172.21.15.183:6814 osd.0 since back 2024-09-10T19:33:45.563941+0000 front 2024-09-10T19:33:45.563894+0000 (oldest deadline 2024-09-10T19:34:14.364335+0000) 2024-09-10T19:34:22.589 INFO:journalctl@ceph.osd.6.smithi191.stdout:Sep 10 19:34:22 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd-6[50347]: 2024-09-10T19:34:22.332+0000 7f4ab7287640 -1 osd.6 461 heartbeat_check: no reply from 172.21.15.183:6806 osd.1 since back 2024-09-10T19:33:54.364736+0000 front 2024-09-10T19:33:54.364820+0000 (oldest deadline 2024-09-10T19:34:20.264938+0000) 2024-09-10T19:34:22.965 INFO:journalctl@ceph.osd.5.smithi191.stdout:Sep 10 19:34:22 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd-5[44138]: 2024-09-10T19:34:22.587+0000 7f7dce955640 -1 osd.5 461 heartbeat_check: no reply from 172.21.15.183:6814 osd.0 since back 2024-09-10T19:33:48.769215+0000 front 2024-09-10T19:33:48.769237+0000 (oldest deadline 2024-09-10T19:34:12.869339+0000) 2024-09-10T19:34:22.966 INFO:journalctl@ceph.osd.5.smithi191.stdout:Sep 10 19:34:22 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd-5[44138]: 2024-09-10T19:34:22.587+0000 7f7dce955640 -1 osd.5 461 heartbeat_check: no reply from 172.21.15.183:6806 osd.1 since back 2024-09-10T19:33:56.370264+0000 front 2024-09-10T19:33:56.370209+0000 (oldest deadline 2024-09-10T19:34:22.270225+0000) 2024-09-10T19:34:23.338 INFO:journalctl@ceph.osd.4.smithi191.stdout:Sep 10 19:34:23 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd-4[38197]: 2024-09-10T19:34:23.032+0000 7fb6016fb640 -1 osd.4 461 heartbeat_check: no reply from 172.21.15.183:6814 osd.0 since back 2024-09-10T19:33:49.868845+0000 front 2024-09-10T19:33:49.868952+0000 (oldest deadline 2024-09-10T19:34:13.968709+0000) 2024-09-10T19:34:23.338 INFO:journalctl@ceph.osd.7.smithi191.stdout:Sep 10 19:34:23 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd-7[56482]: 2024-09-10T19:34:23.081+0000 7f9ef7a9a640 -1 osd.7 461 heartbeat_check: no reply from 172.21.15.183:6814 osd.0 since back 2024-09-10T19:33:48.004787+0000 front 2024-09-10T19:33:48.004721+0000 (oldest deadline 2024-09-10T19:34:11.504575+0000) 2024-09-10T19:34:23.614 INFO:journalctl@ceph.osd.6.smithi191.stdout:Sep 10 19:34:23 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd-6[50347]: 2024-09-10T19:34:23.336+0000 7f4ab7287640 -1 osd.6 461 heartbeat_check: no reply from 172.21.15.183:6814 osd.0 since back 2024-09-10T19:33:45.563941+0000 front 2024-09-10T19:33:45.563894+0000 (oldest deadline 2024-09-10T19:34:14.364335+0000) 2024-09-10T19:34:23.614 INFO:journalctl@ceph.osd.6.smithi191.stdout:Sep 10 19:34:23 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd-6[50347]: 2024-09-10T19:34:23.336+0000 7f4ab7287640 -1 osd.6 461 heartbeat_check: no reply from 172.21.15.183:6806 osd.1 since back 2024-09-10T19:33:54.364736+0000 front 2024-09-10T19:33:54.364820+0000 (oldest deadline 2024-09-10T19:34:20.264938+0000) 2024-09-10T19:34:23.924 INFO:journalctl@ceph.osd.5.smithi191.stdout:Sep 10 19:34:23 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd-5[44138]: 2024-09-10T19:34:23.612+0000 7f7dce955640 -1 osd.5 461 heartbeat_check: no reply from 172.21.15.183:6814 osd.0 since back 2024-09-10T19:33:48.769215+0000 front 2024-09-10T19:33:48.769237+0000 (oldest deadline 2024-09-10T19:34:12.869339+0000) 2024-09-10T19:34:23.924 INFO:journalctl@ceph.osd.5.smithi191.stdout:Sep 10 19:34:23 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd-5[44138]: 2024-09-10T19:34:23.612+0000 7f7dce955640 -1 osd.5 461 heartbeat_check: no reply from 172.21.15.183:6806 osd.1 since back 2024-09-10T19:33:56.370264+0000 front 2024-09-10T19:33:56.370209+0000 (oldest deadline 2024-09-10T19:34:22.270225+0000) 2024-09-10T19:34:24.215 INFO:journalctl@ceph.osd.7.smithi191.stdout:Sep 10 19:34:24 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd-7[56482]: 2024-09-10T19:34:24.116+0000 7f9ef7a9a640 -1 osd.7 461 heartbeat_check: no reply from 172.21.15.183:6814 osd.0 since back 2024-09-10T19:33:48.004787+0000 front 2024-09-10T19:33:48.004721+0000 (oldest deadline 2024-09-10T19:34:11.504575+0000) 2024-09-10T19:34:24.216 INFO:journalctl@ceph.osd.7.smithi191.stdout:Sep 10 19:34:24 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd-7[56482]: 2024-09-10T19:34:24.116+0000 7f9ef7a9a640 -1 osd.7 461 heartbeat_check: no reply from 172.21.15.183:6806 osd.1 since back 2024-09-10T19:33:57.906042+0000 front 2024-09-10T19:33:57.906087+0000 (oldest deadline 2024-09-10T19:34:23.205965+0000) 2024-09-10T19:34:24.216 INFO:journalctl@ceph.osd.4.smithi191.stdout:Sep 10 19:34:23 smithi191 podman[165133]: 2024-09-10 19:34:23.923426977 +0000 UTC m=+5.633535584 container died e19799bd64a35d317ab2e4024f73a8fbf4c3113f7d6a204558a282ba953e1691 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3bfeaf4417585c821cab9ca76e4d95f390b0e769807122a45463640820b5cd16, name=ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd-4, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, RELEASE=wip-guits-quincy-2024-09-09-1810-quincy-5be5739, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, GIT_CLEAN=True) 2024-09-10T19:34:24.641 INFO:journalctl@ceph.osd.6.smithi191.stdout:Sep 10 19:34:24 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd-6[50347]: 2024-09-10T19:34:24.375+0000 7f4ab7287640 -1 osd.6 461 heartbeat_check: no reply from 172.21.15.183:6814 osd.0 since back 2024-09-10T19:33:45.563941+0000 front 2024-09-10T19:33:45.563894+0000 (oldest deadline 2024-09-10T19:34:14.364335+0000) 2024-09-10T19:34:24.641 INFO:journalctl@ceph.osd.6.smithi191.stdout:Sep 10 19:34:24 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd-6[50347]: 2024-09-10T19:34:24.375+0000 7f4ab7287640 -1 osd.6 461 heartbeat_check: no reply from 172.21.15.183:6806 osd.1 since back 2024-09-10T19:33:54.364736+0000 front 2024-09-10T19:33:54.364820+0000 (oldest deadline 2024-09-10T19:34:20.264938+0000) 2024-09-10T19:34:24.642 INFO:journalctl@ceph.osd.4.smithi191.stdout:Sep 10 19:34:24 smithi191 podman[165133]: 2024-09-10 19:34:24.333285289 +0000 UTC m=+6.043393882 container cleanup e19799bd64a35d317ab2e4024f73a8fbf4c3113f7d6a204558a282ba953e1691 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3bfeaf4417585c821cab9ca76e4d95f390b0e769807122a45463640820b5cd16, name=ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd-4, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, io.buildah.version=1.37.2, GIT_CLEAN=True, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, RELEASE=wip-guits-quincy-2024-09-09-1810-quincy-5be5739, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0) 2024-09-10T19:34:24.642 INFO:journalctl@ceph.osd.4.smithi191.stdout:Sep 10 19:34:24 smithi191 bash[165133]: ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd-4 2024-09-10T19:34:24.642 INFO:journalctl@ceph.osd.4.smithi191.stdout:Sep 10 19:34:24 smithi191 podman[165154]: 2024-09-10 19:34:24.493051877 +0000 UTC m=+0.559622212 container remove e19799bd64a35d317ab2e4024f73a8fbf4c3113f7d6a204558a282ba953e1691 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3bfeaf4417585c821cab9ca76e4d95f390b0e769807122a45463640820b5cd16, name=ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd-4, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, RELEASE=wip-guits-quincy-2024-09-09-1810-quincy-5be5739, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS) 2024-09-10T19:34:24.896 INFO:journalctl@ceph.osd.5.smithi191.stdout:Sep 10 19:34:24 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd-5[44138]: 2024-09-10T19:34:24.639+0000 7f7dce955640 -1 osd.5 461 heartbeat_check: no reply from 172.21.15.183:6814 osd.0 since back 2024-09-10T19:33:48.769215+0000 front 2024-09-10T19:33:48.769237+0000 (oldest deadline 2024-09-10T19:34:12.869339+0000) 2024-09-10T19:34:24.896 INFO:journalctl@ceph.osd.5.smithi191.stdout:Sep 10 19:34:24 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd-5[44138]: 2024-09-10T19:34:24.639+0000 7f7dce955640 -1 osd.5 461 heartbeat_check: no reply from 172.21.15.183:6806 osd.1 since back 2024-09-10T19:33:56.370264+0000 front 2024-09-10T19:33:56.370209+0000 (oldest deadline 2024-09-10T19:34:22.270225+0000) 2024-09-10T19:34:25.148 INFO:journalctl@ceph.osd.7.smithi191.stdout:Sep 10 19:34:25 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd-7[56482]: 2024-09-10T19:34:25.070+0000 7f9ef7a9a640 -1 osd.7 461 heartbeat_check: no reply from 172.21.15.183:6814 osd.0 since back 2024-09-10T19:33:48.004787+0000 front 2024-09-10T19:33:48.004721+0000 (oldest deadline 2024-09-10T19:34:11.504575+0000) 2024-09-10T19:34:25.148 INFO:journalctl@ceph.osd.7.smithi191.stdout:Sep 10 19:34:25 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd-7[56482]: 2024-09-10T19:34:25.070+0000 7f9ef7a9a640 -1 osd.7 461 heartbeat_check: no reply from 172.21.15.183:6806 osd.1 since back 2024-09-10T19:33:57.906042+0000 front 2024-09-10T19:33:57.906087+0000 (oldest deadline 2024-09-10T19:34:23.205965+0000) 2024-09-10T19:34:25.150 INFO:journalctl@ceph.osd.4.smithi191.stdout:Sep 10 19:34:24 smithi191 podman[165252]: 2024-09-10 19:34:24.797381836 +0000 UTC m=+0.015791110 image pull 4c55dd4096115e48502f368cff1eab06769f71f030e1692e2611bbc6c6f637e9 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3bfeaf4417585c821cab9ca76e4d95f390b0e769807122a45463640820b5cd16 2024-09-10T19:34:25.151 INFO:journalctl@ceph.osd.4.smithi191.stdout:Sep 10 19:34:24 smithi191 podman[165252]: 2024-09-10 19:34:24.902894276 +0000 UTC m=+0.121303580 container create 6359fda96dca04a15d2439f7d4d990c45a28cc192be294e3532553b705e59c21 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3bfeaf4417585c821cab9ca76e4d95f390b0e769807122a45463640820b5cd16, name=ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd-4-deactivate, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, RELEASE=wip-guits-quincy-2024-09-09-1810-quincy-5be5739, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, ceph=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2) 2024-09-10T19:34:25.466 INFO:journalctl@ceph.osd.6.smithi191.stdout:Sep 10 19:34:25 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd-6[50347]: 2024-09-10T19:34:25.363+0000 7f4ab7287640 -1 osd.6 461 heartbeat_check: no reply from 172.21.15.183:6814 osd.0 since back 2024-09-10T19:33:45.563941+0000 front 2024-09-10T19:33:45.563894+0000 (oldest deadline 2024-09-10T19:34:14.364335+0000) 2024-09-10T19:34:25.466 INFO:journalctl@ceph.osd.6.smithi191.stdout:Sep 10 19:34:25 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd-6[50347]: 2024-09-10T19:34:25.363+0000 7f4ab7287640 -1 osd.6 461 heartbeat_check: no reply from 172.21.15.183:6806 osd.1 since back 2024-09-10T19:33:54.364736+0000 front 2024-09-10T19:33:54.364820+0000 (oldest deadline 2024-09-10T19:34:20.264938+0000) 2024-09-10T19:34:25.466 INFO:journalctl@ceph.osd.4.smithi191.stdout:Sep 10 19:34:25 smithi191 podman[165252]: 2024-09-10 19:34:25.147235622 +0000 UTC m=+0.365644892 container init 6359fda96dca04a15d2439f7d4d990c45a28cc192be294e3532553b705e59c21 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3bfeaf4417585c821cab9ca76e4d95f390b0e769807122a45463640820b5cd16, name=ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd-4-deactivate, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, GIT_CLEAN=True, GIT_BRANCH=HEAD, RELEASE=wip-guits-quincy-2024-09-09-1810-quincy-5be5739, ceph=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS) 2024-09-10T19:34:25.466 INFO:journalctl@ceph.osd.4.smithi191.stdout:Sep 10 19:34:25 smithi191 podman[165252]: 2024-09-10 19:34:25.152483488 +0000 UTC m=+0.370892749 container start 6359fda96dca04a15d2439f7d4d990c45a28cc192be294e3532553b705e59c21 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3bfeaf4417585c821cab9ca76e4d95f390b0e769807122a45463640820b5cd16, name=ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd-4-deactivate, org.label-schema.build-date=20240731, RELEASE=wip-guits-quincy-2024-09-09-1810-quincy-5be5739, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, ceph=True, org.label-schema.schema-version=1.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS) 2024-09-10T19:34:25.466 INFO:journalctl@ceph.osd.4.smithi191.stdout:Sep 10 19:34:25 smithi191 podman[165252]: 2024-09-10 19:34:25.280671742 +0000 UTC m=+0.499081026 container attach 6359fda96dca04a15d2439f7d4d990c45a28cc192be294e3532553b705e59c21 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3bfeaf4417585c821cab9ca76e4d95f390b0e769807122a45463640820b5cd16, name=ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd-4-deactivate, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, RELEASE=wip-guits-quincy-2024-09-09-1810-quincy-5be5739, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, ceph=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=) 2024-09-10T19:34:25.835 INFO:journalctl@ceph.osd.5.smithi191.stdout:Sep 10 19:34:25 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd-5[44138]: 2024-09-10T19:34:25.638+0000 7f7dce955640 -1 osd.5 461 heartbeat_check: no reply from 172.21.15.183:6814 osd.0 since back 2024-09-10T19:33:48.769215+0000 front 2024-09-10T19:33:48.769237+0000 (oldest deadline 2024-09-10T19:34:12.869339+0000) 2024-09-10T19:34:25.836 INFO:journalctl@ceph.osd.5.smithi191.stdout:Sep 10 19:34:25 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd-5[44138]: 2024-09-10T19:34:25.638+0000 7f7dce955640 -1 osd.5 461 heartbeat_check: no reply from 172.21.15.183:6806 osd.1 since back 2024-09-10T19:33:56.370264+0000 front 2024-09-10T19:33:56.370209+0000 (oldest deadline 2024-09-10T19:34:22.270225+0000) 2024-09-10T19:34:25.836 INFO:journalctl@ceph.osd.4.smithi191.stdout:Sep 10 19:34:25 smithi191 podman[165252]: 2024-09-10 19:34:25.540355674 +0000 UTC m=+0.758764942 container died 6359fda96dca04a15d2439f7d4d990c45a28cc192be294e3532553b705e59c21 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3bfeaf4417585c821cab9ca76e4d95f390b0e769807122a45463640820b5cd16, name=ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd-4-deactivate, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, RELEASE=wip-guits-quincy-2024-09-09-1810-quincy-5be5739, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.vendor=CentOS, ceph=True) 2024-09-10T19:34:26.215 INFO:journalctl@ceph.osd.7.smithi191.stdout:Sep 10 19:34:26 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd-7[56482]: 2024-09-10T19:34:26.063+0000 7f9ef7a9a640 -1 osd.7 461 heartbeat_check: no reply from 172.21.15.183:6814 osd.0 since back 2024-09-10T19:33:48.004787+0000 front 2024-09-10T19:33:48.004721+0000 (oldest deadline 2024-09-10T19:34:11.504575+0000) 2024-09-10T19:34:26.216 INFO:journalctl@ceph.osd.7.smithi191.stdout:Sep 10 19:34:26 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd-7[56482]: 2024-09-10T19:34:26.063+0000 7f9ef7a9a640 -1 osd.7 461 heartbeat_check: no reply from 172.21.15.183:6806 osd.1 since back 2024-09-10T19:33:57.906042+0000 front 2024-09-10T19:33:57.906087+0000 (oldest deadline 2024-09-10T19:34:23.205965+0000) 2024-09-10T19:34:26.385 DEBUG:teuthology.orchestra.run.smithi191:> sudo pkill -f 'journalctl -f -n 0 -u ceph-320f6df8-6fa7-11ef-bcea-c7b262605968@osd.4.service' 2024-09-10T19:34:26.428 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-10T19:34:26.429 INFO:tasks.cephadm.osd.4:Stopped osd.4 2024-09-10T19:34:26.429 INFO:tasks.cephadm.osd.5:Stopping osd.5... 2024-09-10T19:34:26.429 DEBUG:teuthology.orchestra.run.smithi191:> sudo systemctl stop ceph-320f6df8-6fa7-11ef-bcea-c7b262605968@osd.5 2024-09-10T19:34:26.620 INFO:journalctl@ceph.osd.5.smithi191.stdout:Sep 10 19:34:26 smithi191 systemd[1]: Stopping Ceph osd.5 for 320f6df8-6fa7-11ef-bcea-c7b262605968... 2024-09-10T19:34:26.620 INFO:journalctl@ceph.osd.6.smithi191.stdout:Sep 10 19:34:26 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd-6[50347]: 2024-09-10T19:34:26.396+0000 7f4ab7287640 -1 osd.6 461 heartbeat_check: no reply from 172.21.15.183:6814 osd.0 since back 2024-09-10T19:33:45.563941+0000 front 2024-09-10T19:33:45.563894+0000 (oldest deadline 2024-09-10T19:34:14.364335+0000) 2024-09-10T19:34:26.620 INFO:journalctl@ceph.osd.6.smithi191.stdout:Sep 10 19:34:26 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd-6[50347]: 2024-09-10T19:34:26.396+0000 7f4ab7287640 -1 osd.6 461 heartbeat_check: no reply from 172.21.15.183:6806 osd.1 since back 2024-09-10T19:33:54.364736+0000 front 2024-09-10T19:33:54.364820+0000 (oldest deadline 2024-09-10T19:34:20.264938+0000) 2024-09-10T19:34:26.966 INFO:journalctl@ceph.osd.5.smithi191.stdout:Sep 10 19:34:26 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd-5[44138]: 2024-09-10T19:34:26.617+0000 7f7dce955640 -1 osd.5 461 heartbeat_check: no reply from 172.21.15.183:6814 osd.0 since back 2024-09-10T19:33:48.769215+0000 front 2024-09-10T19:33:48.769237+0000 (oldest deadline 2024-09-10T19:34:12.869339+0000) 2024-09-10T19:34:26.966 INFO:journalctl@ceph.osd.5.smithi191.stdout:Sep 10 19:34:26 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd-5[44138]: 2024-09-10T19:34:26.617+0000 7f7dce955640 -1 osd.5 461 heartbeat_check: no reply from 172.21.15.183:6806 osd.1 since back 2024-09-10T19:33:56.370264+0000 front 2024-09-10T19:33:56.370209+0000 (oldest deadline 2024-09-10T19:34:22.270225+0000) 2024-09-10T19:34:26.966 INFO:journalctl@ceph.osd.5.smithi191.stdout:Sep 10 19:34:26 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd-5[44138]: 2024-09-10T19:34:26.747+0000 7f7dd233d640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-10T19:34:26.966 INFO:journalctl@ceph.osd.5.smithi191.stdout:Sep 10 19:34:26 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd-5[44138]: 2024-09-10T19:34:26.747+0000 7f7dd233d640 -1 osd.5 461 *** Got signal Terminated *** 2024-09-10T19:34:26.967 INFO:journalctl@ceph.osd.5.smithi191.stdout:Sep 10 19:34:26 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd-5[44138]: 2024-09-10T19:34:26.747+0000 7f7dd233d640 -1 osd.5 461 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-10T19:34:27.352 INFO:journalctl@ceph.osd.7.smithi191.stdout:Sep 10 19:34:27 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd-7[56482]: 2024-09-10T19:34:27.097+0000 7f9ef7a9a640 -1 osd.7 461 heartbeat_check: no reply from 172.21.15.183:6814 osd.0 since back 2024-09-10T19:33:48.004787+0000 front 2024-09-10T19:33:48.004721+0000 (oldest deadline 2024-09-10T19:34:11.504575+0000) 2024-09-10T19:34:27.352 INFO:journalctl@ceph.osd.7.smithi191.stdout:Sep 10 19:34:27 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd-7[56482]: 2024-09-10T19:34:27.097+0000 7f9ef7a9a640 -1 osd.7 461 heartbeat_check: no reply from 172.21.15.183:6806 osd.1 since back 2024-09-10T19:33:57.906042+0000 front 2024-09-10T19:33:57.906087+0000 (oldest deadline 2024-09-10T19:34:23.205965+0000) 2024-09-10T19:34:27.651 INFO:journalctl@ceph.osd.6.smithi191.stdout:Sep 10 19:34:27 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd-6[50347]: 2024-09-10T19:34:27.350+0000 7f4ab7287640 -1 osd.6 461 heartbeat_check: no reply from 172.21.15.183:6814 osd.0 since back 2024-09-10T19:33:45.563941+0000 front 2024-09-10T19:33:45.563894+0000 (oldest deadline 2024-09-10T19:34:14.364335+0000) 2024-09-10T19:34:27.651 INFO:journalctl@ceph.osd.6.smithi191.stdout:Sep 10 19:34:27 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd-6[50347]: 2024-09-10T19:34:27.350+0000 7f4ab7287640 -1 osd.6 461 heartbeat_check: no reply from 172.21.15.183:6806 osd.1 since back 2024-09-10T19:33:54.364736+0000 front 2024-09-10T19:33:54.364820+0000 (oldest deadline 2024-09-10T19:34:20.264938+0000) 2024-09-10T19:34:27.966 INFO:journalctl@ceph.osd.5.smithi191.stdout:Sep 10 19:34:27 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd-5[44138]: 2024-09-10T19:34:27.649+0000 7f7dce955640 -1 osd.5 461 heartbeat_check: no reply from 172.21.15.183:6814 osd.0 since back 2024-09-10T19:33:48.769215+0000 front 2024-09-10T19:33:48.769237+0000 (oldest deadline 2024-09-10T19:34:12.869339+0000) 2024-09-10T19:34:27.966 INFO:journalctl@ceph.osd.5.smithi191.stdout:Sep 10 19:34:27 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd-5[44138]: 2024-09-10T19:34:27.649+0000 7f7dce955640 -1 osd.5 461 heartbeat_check: no reply from 172.21.15.183:6806 osd.1 since back 2024-09-10T19:33:56.370264+0000 front 2024-09-10T19:33:56.370209+0000 (oldest deadline 2024-09-10T19:34:22.270225+0000) 2024-09-10T19:34:28.310 INFO:journalctl@ceph.osd.7.smithi191.stdout:Sep 10 19:34:28 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd-7[56482]: 2024-09-10T19:34:28.052+0000 7f9ef7a9a640 -1 osd.7 461 heartbeat_check: no reply from 172.21.15.183:6814 osd.0 since back 2024-09-10T19:33:48.004787+0000 front 2024-09-10T19:33:48.004721+0000 (oldest deadline 2024-09-10T19:34:11.504575+0000) 2024-09-10T19:34:28.310 INFO:journalctl@ceph.osd.7.smithi191.stdout:Sep 10 19:34:28 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd-7[56482]: 2024-09-10T19:34:28.052+0000 7f9ef7a9a640 -1 osd.7 461 heartbeat_check: no reply from 172.21.15.183:6806 osd.1 since back 2024-09-10T19:33:57.906042+0000 front 2024-09-10T19:33:57.906087+0000 (oldest deadline 2024-09-10T19:34:23.205965+0000) 2024-09-10T19:34:28.648 INFO:journalctl@ceph.osd.6.smithi191.stdout:Sep 10 19:34:28 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd-6[50347]: 2024-09-10T19:34:28.308+0000 7f4ab7287640 -1 osd.6 461 heartbeat_check: no reply from 172.21.15.183:6814 osd.0 since back 2024-09-10T19:33:45.563941+0000 front 2024-09-10T19:33:45.563894+0000 (oldest deadline 2024-09-10T19:34:14.364335+0000) 2024-09-10T19:34:28.648 INFO:journalctl@ceph.osd.6.smithi191.stdout:Sep 10 19:34:28 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd-6[50347]: 2024-09-10T19:34:28.308+0000 7f4ab7287640 -1 osd.6 461 heartbeat_check: no reply from 172.21.15.183:6806 osd.1 since back 2024-09-10T19:33:54.364736+0000 front 2024-09-10T19:33:54.364820+0000 (oldest deadline 2024-09-10T19:34:20.264938+0000) 2024-09-10T19:34:28.966 INFO:journalctl@ceph.osd.5.smithi191.stdout:Sep 10 19:34:28 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd-5[44138]: 2024-09-10T19:34:28.646+0000 7f7dce955640 -1 osd.5 461 heartbeat_check: no reply from 172.21.15.183:6814 osd.0 since back 2024-09-10T19:33:48.769215+0000 front 2024-09-10T19:33:48.769237+0000 (oldest deadline 2024-09-10T19:34:12.869339+0000) 2024-09-10T19:34:28.966 INFO:journalctl@ceph.osd.5.smithi191.stdout:Sep 10 19:34:28 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd-5[44138]: 2024-09-10T19:34:28.646+0000 7f7dce955640 -1 osd.5 461 heartbeat_check: no reply from 172.21.15.183:6806 osd.1 since back 2024-09-10T19:33:56.370264+0000 front 2024-09-10T19:33:56.370209+0000 (oldest deadline 2024-09-10T19:34:22.270225+0000) 2024-09-10T19:34:28.966 INFO:journalctl@ceph.osd.5.smithi191.stdout:Sep 10 19:34:28 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd-5[44138]: 2024-09-10T19:34:28.646+0000 7f7dce955640 -1 osd.5 461 heartbeat_check: no reply from 172.21.15.183:6822 osd.2 since back 2024-09-10T19:34:04.471261+0000 front 2024-09-10T19:34:04.471277+0000 (oldest deadline 2024-09-10T19:34:27.971202+0000) 2024-09-10T19:34:29.355 INFO:journalctl@ceph.osd.7.smithi191.stdout:Sep 10 19:34:29 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd-7[56482]: 2024-09-10T19:34:29.102+0000 7f9ef7a9a640 -1 osd.7 461 heartbeat_check: no reply from 172.21.15.183:6814 osd.0 since back 2024-09-10T19:33:48.004787+0000 front 2024-09-10T19:33:48.004721+0000 (oldest deadline 2024-09-10T19:34:11.504575+0000) 2024-09-10T19:34:29.355 INFO:journalctl@ceph.osd.7.smithi191.stdout:Sep 10 19:34:29 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd-7[56482]: 2024-09-10T19:34:29.102+0000 7f9ef7a9a640 -1 osd.7 461 heartbeat_check: no reply from 172.21.15.183:6806 osd.1 since back 2024-09-10T19:33:57.906042+0000 front 2024-09-10T19:33:57.906087+0000 (oldest deadline 2024-09-10T19:34:23.205965+0000) 2024-09-10T19:34:29.613 INFO:journalctl@ceph.osd.6.smithi191.stdout:Sep 10 19:34:29 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd-6[50347]: 2024-09-10T19:34:29.353+0000 7f4ab7287640 -1 osd.6 461 heartbeat_check: no reply from 172.21.15.183:6814 osd.0 since back 2024-09-10T19:33:45.563941+0000 front 2024-09-10T19:33:45.563894+0000 (oldest deadline 2024-09-10T19:34:14.364335+0000) 2024-09-10T19:34:29.613 INFO:journalctl@ceph.osd.6.smithi191.stdout:Sep 10 19:34:29 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd-6[50347]: 2024-09-10T19:34:29.353+0000 7f4ab7287640 -1 osd.6 461 heartbeat_check: no reply from 172.21.15.183:6806 osd.1 since back 2024-09-10T19:33:54.364736+0000 front 2024-09-10T19:33:54.364820+0000 (oldest deadline 2024-09-10T19:34:20.264938+0000) 2024-09-10T19:34:29.965 INFO:journalctl@ceph.osd.5.smithi191.stdout:Sep 10 19:34:29 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd-5[44138]: 2024-09-10T19:34:29.611+0000 7f7dce955640 -1 osd.5 461 heartbeat_check: no reply from 172.21.15.183:6814 osd.0 since back 2024-09-10T19:33:48.769215+0000 front 2024-09-10T19:33:48.769237+0000 (oldest deadline 2024-09-10T19:34:12.869339+0000) 2024-09-10T19:34:29.966 INFO:journalctl@ceph.osd.5.smithi191.stdout:Sep 10 19:34:29 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd-5[44138]: 2024-09-10T19:34:29.611+0000 7f7dce955640 -1 osd.5 461 heartbeat_check: no reply from 172.21.15.183:6806 osd.1 since back 2024-09-10T19:33:56.370264+0000 front 2024-09-10T19:33:56.370209+0000 (oldest deadline 2024-09-10T19:34:22.270225+0000) 2024-09-10T19:34:29.966 INFO:journalctl@ceph.osd.5.smithi191.stdout:Sep 10 19:34:29 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd-5[44138]: 2024-09-10T19:34:29.611+0000 7f7dce955640 -1 osd.5 461 heartbeat_check: no reply from 172.21.15.183:6822 osd.2 since back 2024-09-10T19:34:04.471261+0000 front 2024-09-10T19:34:04.471277+0000 (oldest deadline 2024-09-10T19:34:27.971202+0000) 2024-09-10T19:34:30.466 INFO:journalctl@ceph.osd.6.smithi191.stdout:Sep 10 19:34:30 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd-6[50347]: 2024-09-10T19:34:30.326+0000 7f4ab7287640 -1 osd.6 461 heartbeat_check: no reply from 172.21.15.183:6814 osd.0 since back 2024-09-10T19:33:45.563941+0000 front 2024-09-10T19:33:45.563894+0000 (oldest deadline 2024-09-10T19:34:14.364335+0000) 2024-09-10T19:34:30.466 INFO:journalctl@ceph.osd.6.smithi191.stdout:Sep 10 19:34:30 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd-6[50347]: 2024-09-10T19:34:30.326+0000 7f4ab7287640 -1 osd.6 461 heartbeat_check: no reply from 172.21.15.183:6806 osd.1 since back 2024-09-10T19:33:54.364736+0000 front 2024-09-10T19:33:54.364820+0000 (oldest deadline 2024-09-10T19:34:20.264938+0000) 2024-09-10T19:34:30.466 INFO:journalctl@ceph.osd.7.smithi191.stdout:Sep 10 19:34:30 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd-7[56482]: 2024-09-10T19:34:30.098+0000 7f9ef7a9a640 -1 osd.7 461 heartbeat_check: no reply from 172.21.15.183:6814 osd.0 since back 2024-09-10T19:33:48.004787+0000 front 2024-09-10T19:33:48.004721+0000 (oldest deadline 2024-09-10T19:34:11.504575+0000) 2024-09-10T19:34:30.466 INFO:journalctl@ceph.osd.7.smithi191.stdout:Sep 10 19:34:30 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd-7[56482]: 2024-09-10T19:34:30.098+0000 7f9ef7a9a640 -1 osd.7 461 heartbeat_check: no reply from 172.21.15.183:6806 osd.1 since back 2024-09-10T19:33:57.906042+0000 front 2024-09-10T19:33:57.906087+0000 (oldest deadline 2024-09-10T19:34:23.205965+0000) 2024-09-10T19:34:30.966 INFO:journalctl@ceph.osd.5.smithi191.stdout:Sep 10 19:34:30 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd-5[44138]: 2024-09-10T19:34:30.614+0000 7f7dce955640 -1 osd.5 461 heartbeat_check: no reply from 172.21.15.183:6814 osd.0 since back 2024-09-10T19:33:48.769215+0000 front 2024-09-10T19:33:48.769237+0000 (oldest deadline 2024-09-10T19:34:12.869339+0000) 2024-09-10T19:34:30.966 INFO:journalctl@ceph.osd.5.smithi191.stdout:Sep 10 19:34:30 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd-5[44138]: 2024-09-10T19:34:30.614+0000 7f7dce955640 -1 osd.5 461 heartbeat_check: no reply from 172.21.15.183:6806 osd.1 since back 2024-09-10T19:33:56.370264+0000 front 2024-09-10T19:33:56.370209+0000 (oldest deadline 2024-09-10T19:34:22.270225+0000) 2024-09-10T19:34:30.966 INFO:journalctl@ceph.osd.5.smithi191.stdout:Sep 10 19:34:30 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd-5[44138]: 2024-09-10T19:34:30.614+0000 7f7dce955640 -1 osd.5 461 heartbeat_check: no reply from 172.21.15.183:6822 osd.2 since back 2024-09-10T19:34:04.471261+0000 front 2024-09-10T19:34:04.471277+0000 (oldest deadline 2024-09-10T19:34:27.971202+0000) 2024-09-10T19:34:31.371 INFO:journalctl@ceph.osd.7.smithi191.stdout:Sep 10 19:34:31 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd-7[56482]: 2024-09-10T19:34:31.071+0000 7f9ef7a9a640 -1 osd.7 461 heartbeat_check: no reply from 172.21.15.183:6814 osd.0 since back 2024-09-10T19:33:48.004787+0000 front 2024-09-10T19:33:48.004721+0000 (oldest deadline 2024-09-10T19:34:11.504575+0000) 2024-09-10T19:34:31.371 INFO:journalctl@ceph.osd.7.smithi191.stdout:Sep 10 19:34:31 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd-7[56482]: 2024-09-10T19:34:31.071+0000 7f9ef7a9a640 -1 osd.7 461 heartbeat_check: no reply from 172.21.15.183:6806 osd.1 since back 2024-09-10T19:33:57.906042+0000 front 2024-09-10T19:33:57.906087+0000 (oldest deadline 2024-09-10T19:34:23.205965+0000) 2024-09-10T19:34:31.371 INFO:journalctl@ceph.osd.7.smithi191.stdout:Sep 10 19:34:31 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd-7[56482]: 2024-09-10T19:34:31.071+0000 7f9ef7a9a640 -1 osd.7 461 heartbeat_check: no reply from 172.21.15.183:6822 osd.2 since back 2024-09-10T19:34:04.306980+0000 front 2024-09-10T19:34:04.307045+0000 (oldest deadline 2024-09-10T19:34:30.206889+0000) 2024-09-10T19:34:31.715 INFO:journalctl@ceph.osd.6.smithi191.stdout:Sep 10 19:34:31 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd-6[50347]: 2024-09-10T19:34:31.369+0000 7f4ab7287640 -1 osd.6 461 heartbeat_check: no reply from 172.21.15.183:6814 osd.0 since back 2024-09-10T19:33:45.563941+0000 front 2024-09-10T19:33:45.563894+0000 (oldest deadline 2024-09-10T19:34:14.364335+0000) 2024-09-10T19:34:31.716 INFO:journalctl@ceph.osd.6.smithi191.stdout:Sep 10 19:34:31 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd-6[50347]: 2024-09-10T19:34:31.369+0000 7f4ab7287640 -1 osd.6 461 heartbeat_check: no reply from 172.21.15.183:6806 osd.1 since back 2024-09-10T19:33:54.364736+0000 front 2024-09-10T19:33:54.364820+0000 (oldest deadline 2024-09-10T19:34:20.264938+0000) 2024-09-10T19:34:31.716 INFO:journalctl@ceph.osd.5.smithi191.stdout:Sep 10 19:34:31 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd-5[44138]: 2024-09-10T19:34:31.612+0000 7f7dce955640 -1 osd.5 461 heartbeat_check: no reply from 172.21.15.183:6814 osd.0 since back 2024-09-10T19:33:48.769215+0000 front 2024-09-10T19:33:48.769237+0000 (oldest deadline 2024-09-10T19:34:12.869339+0000) 2024-09-10T19:34:31.716 INFO:journalctl@ceph.osd.5.smithi191.stdout:Sep 10 19:34:31 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd-5[44138]: 2024-09-10T19:34:31.612+0000 7f7dce955640 -1 osd.5 461 heartbeat_check: no reply from 172.21.15.183:6806 osd.1 since back 2024-09-10T19:33:56.370264+0000 front 2024-09-10T19:33:56.370209+0000 (oldest deadline 2024-09-10T19:34:22.270225+0000) 2024-09-10T19:34:31.716 INFO:journalctl@ceph.osd.5.smithi191.stdout:Sep 10 19:34:31 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd-5[44138]: 2024-09-10T19:34:31.612+0000 7f7dce955640 -1 osd.5 461 heartbeat_check: no reply from 172.21.15.183:6822 osd.2 since back 2024-09-10T19:34:04.471261+0000 front 2024-09-10T19:34:04.471277+0000 (oldest deadline 2024-09-10T19:34:27.971202+0000) 2024-09-10T19:34:32.401 INFO:journalctl@ceph.osd.5.smithi191.stdout:Sep 10 19:34:32 smithi191 podman[165380]: 2024-09-10 19:34:32.274032626 +0000 UTC m=+5.623563242 container died adb4c7c20dff03ff8772b1765c715bccf40c48c12a76dea32e20d8f1206941b0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3bfeaf4417585c821cab9ca76e4d95f390b0e769807122a45463640820b5cd16, name=ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd-5, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-guits-quincy-2024-09-09-1810-quincy-5be5739, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7) 2024-09-10T19:34:32.402 INFO:journalctl@ceph.osd.7.smithi191.stdout:Sep 10 19:34:32 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd-7[56482]: 2024-09-10T19:34:32.055+0000 7f9ef7a9a640 -1 osd.7 461 heartbeat_check: no reply from 172.21.15.183:6814 osd.0 since back 2024-09-10T19:33:48.004787+0000 front 2024-09-10T19:33:48.004721+0000 (oldest deadline 2024-09-10T19:34:11.504575+0000) 2024-09-10T19:34:32.402 INFO:journalctl@ceph.osd.7.smithi191.stdout:Sep 10 19:34:32 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd-7[56482]: 2024-09-10T19:34:32.055+0000 7f9ef7a9a640 -1 osd.7 461 heartbeat_check: no reply from 172.21.15.183:6806 osd.1 since back 2024-09-10T19:33:57.906042+0000 front 2024-09-10T19:33:57.906087+0000 (oldest deadline 2024-09-10T19:34:23.205965+0000) 2024-09-10T19:34:32.402 INFO:journalctl@ceph.osd.7.smithi191.stdout:Sep 10 19:34:32 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd-7[56482]: 2024-09-10T19:34:32.055+0000 7f9ef7a9a640 -1 osd.7 461 heartbeat_check: no reply from 172.21.15.183:6822 osd.2 since back 2024-09-10T19:34:04.306980+0000 front 2024-09-10T19:34:04.307045+0000 (oldest deadline 2024-09-10T19:34:30.206889+0000) 2024-09-10T19:34:32.661 INFO:journalctl@ceph.osd.5.smithi191.stdout:Sep 10 19:34:32 smithi191 podman[165380]: 2024-09-10 19:34:32.645417957 +0000 UTC m=+5.994948604 container cleanup adb4c7c20dff03ff8772b1765c715bccf40c48c12a76dea32e20d8f1206941b0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3bfeaf4417585c821cab9ca76e4d95f390b0e769807122a45463640820b5cd16, name=ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd-5, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, io.buildah.version=1.37.2, ceph=True, GIT_BRANCH=HEAD, GIT_CLEAN=True, RELEASE=wip-guits-quincy-2024-09-09-1810-quincy-5be5739, org.label-schema.license=GPLv2) 2024-09-10T19:34:32.661 INFO:journalctl@ceph.osd.5.smithi191.stdout:Sep 10 19:34:32 smithi191 bash[165380]: ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd-5 2024-09-10T19:34:32.661 INFO:journalctl@ceph.osd.6.smithi191.stdout:Sep 10 19:34:32 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd-6[50347]: 2024-09-10T19:34:32.399+0000 7f4ab7287640 -1 osd.6 461 heartbeat_check: no reply from 172.21.15.183:6814 osd.0 since back 2024-09-10T19:33:45.563941+0000 front 2024-09-10T19:33:45.563894+0000 (oldest deadline 2024-09-10T19:34:14.364335+0000) 2024-09-10T19:34:32.661 INFO:journalctl@ceph.osd.6.smithi191.stdout:Sep 10 19:34:32 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd-6[50347]: 2024-09-10T19:34:32.399+0000 7f4ab7287640 -1 osd.6 461 heartbeat_check: no reply from 172.21.15.183:6806 osd.1 since back 2024-09-10T19:33:54.364736+0000 front 2024-09-10T19:33:54.364820+0000 (oldest deadline 2024-09-10T19:34:20.264938+0000) 2024-09-10T19:34:32.661 INFO:journalctl@ceph.osd.6.smithi191.stdout:Sep 10 19:34:32 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd-6[50347]: 2024-09-10T19:34:32.399+0000 7f4ab7287640 -1 osd.6 461 heartbeat_check: no reply from 172.21.15.183:6822 osd.2 since back 2024-09-10T19:34:05.565981+0000 front 2024-09-10T19:34:05.566054+0000 (oldest deadline 2024-09-10T19:34:31.466032+0000) 2024-09-10T19:34:33.215 INFO:journalctl@ceph.osd.5.smithi191.stdout:Sep 10 19:34:32 smithi191 podman[165402]: 2024-09-10 19:34:32.838910628 +0000 UTC m=+0.551673384 container remove adb4c7c20dff03ff8772b1765c715bccf40c48c12a76dea32e20d8f1206941b0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3bfeaf4417585c821cab9ca76e4d95f390b0e769807122a45463640820b5cd16, name=ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd-5, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=wip-guits-quincy-2024-09-09-1810-quincy-5be5739, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, GIT_CLEAN=True, ceph=True, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD) 2024-09-10T19:34:33.216 INFO:journalctl@ceph.osd.7.smithi191.stdout:Sep 10 19:34:33 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd-7[56482]: 2024-09-10T19:34:33.075+0000 7f9ef7a9a640 -1 osd.7 461 heartbeat_check: no reply from 172.21.15.183:6814 osd.0 since back 2024-09-10T19:33:48.004787+0000 front 2024-09-10T19:33:48.004721+0000 (oldest deadline 2024-09-10T19:34:11.504575+0000) 2024-09-10T19:34:33.216 INFO:journalctl@ceph.osd.7.smithi191.stdout:Sep 10 19:34:33 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd-7[56482]: 2024-09-10T19:34:33.075+0000 7f9ef7a9a640 -1 osd.7 461 heartbeat_check: no reply from 172.21.15.183:6806 osd.1 since back 2024-09-10T19:33:57.906042+0000 front 2024-09-10T19:33:57.906087+0000 (oldest deadline 2024-09-10T19:34:23.205965+0000) 2024-09-10T19:34:33.216 INFO:journalctl@ceph.osd.7.smithi191.stdout:Sep 10 19:34:33 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd-7[56482]: 2024-09-10T19:34:33.075+0000 7f9ef7a9a640 -1 osd.7 461 heartbeat_check: no reply from 172.21.15.183:6822 osd.2 since back 2024-09-10T19:34:04.306980+0000 front 2024-09-10T19:34:04.307045+0000 (oldest deadline 2024-09-10T19:34:30.206889+0000) 2024-09-10T19:34:33.493 INFO:journalctl@ceph.osd.6.smithi191.stdout:Sep 10 19:34:33 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd-6[50347]: 2024-09-10T19:34:33.380+0000 7f4ab7287640 -1 osd.6 461 heartbeat_check: no reply from 172.21.15.183:6814 osd.0 since back 2024-09-10T19:33:45.563941+0000 front 2024-09-10T19:33:45.563894+0000 (oldest deadline 2024-09-10T19:34:14.364335+0000) 2024-09-10T19:34:33.493 INFO:journalctl@ceph.osd.6.smithi191.stdout:Sep 10 19:34:33 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd-6[50347]: 2024-09-10T19:34:33.380+0000 7f4ab7287640 -1 osd.6 461 heartbeat_check: no reply from 172.21.15.183:6806 osd.1 since back 2024-09-10T19:33:54.364736+0000 front 2024-09-10T19:33:54.364820+0000 (oldest deadline 2024-09-10T19:34:20.264938+0000) 2024-09-10T19:34:33.493 INFO:journalctl@ceph.osd.6.smithi191.stdout:Sep 10 19:34:33 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd-6[50347]: 2024-09-10T19:34:33.380+0000 7f4ab7287640 -1 osd.6 461 heartbeat_check: no reply from 172.21.15.183:6822 osd.2 since back 2024-09-10T19:34:05.565981+0000 front 2024-09-10T19:34:05.566054+0000 (oldest deadline 2024-09-10T19:34:31.466032+0000) 2024-09-10T19:34:33.493 INFO:journalctl@ceph.osd.5.smithi191.stdout:Sep 10 19:34:33 smithi191 podman[165497]: 2024-09-10 19:34:33.135040763 +0000 UTC m=+0.015543894 image pull 4c55dd4096115e48502f368cff1eab06769f71f030e1692e2611bbc6c6f637e9 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3bfeaf4417585c821cab9ca76e4d95f390b0e769807122a45463640820b5cd16 2024-09-10T19:34:33.494 INFO:journalctl@ceph.osd.5.smithi191.stdout:Sep 10 19:34:33 smithi191 podman[165497]: 2024-09-10 19:34:33.25652158 +0000 UTC m=+0.137024703 container create b70cd5d596cbb14eebc17eee2f2e063150a42d563e9da640110b0bcb1ad96efb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3bfeaf4417585c821cab9ca76e4d95f390b0e769807122a45463640820b5cd16, name=ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd-5-deactivate, GIT_CLEAN=True, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, ceph=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, RELEASE=wip-guits-quincy-2024-09-09-1810-quincy-5be5739, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2) 2024-09-10T19:34:33.966 INFO:journalctl@ceph.osd.5.smithi191.stdout:Sep 10 19:34:33 smithi191 podman[165497]: 2024-09-10 19:34:33.492283318 +0000 UTC m=+0.372786441 container init b70cd5d596cbb14eebc17eee2f2e063150a42d563e9da640110b0bcb1ad96efb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3bfeaf4417585c821cab9ca76e4d95f390b0e769807122a45463640820b5cd16, name=ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd-5-deactivate, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, RELEASE=wip-guits-quincy-2024-09-09-1810-quincy-5be5739, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, ceph=True) 2024-09-10T19:34:33.966 INFO:journalctl@ceph.osd.5.smithi191.stdout:Sep 10 19:34:33 smithi191 podman[165497]: 2024-09-10 19:34:33.497812057 +0000 UTC m=+0.378315183 container start b70cd5d596cbb14eebc17eee2f2e063150a42d563e9da640110b0bcb1ad96efb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3bfeaf4417585c821cab9ca76e4d95f390b0e769807122a45463640820b5cd16, name=ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd-5-deactivate, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.build-date=20240731, RELEASE=wip-guits-quincy-2024-09-09-1810-quincy-5be5739, CEPH_POINT_RELEASE=) 2024-09-10T19:34:33.966 INFO:journalctl@ceph.osd.5.smithi191.stdout:Sep 10 19:34:33 smithi191 podman[165497]: 2024-09-10 19:34:33.6175632 +0000 UTC m=+0.498066332 container attach b70cd5d596cbb14eebc17eee2f2e063150a42d563e9da640110b0bcb1ad96efb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3bfeaf4417585c821cab9ca76e4d95f390b0e769807122a45463640820b5cd16, name=ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd-5-deactivate, RELEASE=wip-guits-quincy-2024-09-09-1810-quincy-5be5739, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0) 2024-09-10T19:34:33.966 INFO:journalctl@ceph.osd.5.smithi191.stdout:Sep 10 19:34:33 smithi191 podman[165497]: 2024-09-10 19:34:33.67424447 +0000 UTC m=+0.554747597 container died b70cd5d596cbb14eebc17eee2f2e063150a42d563e9da640110b0bcb1ad96efb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3bfeaf4417585c821cab9ca76e4d95f390b0e769807122a45463640820b5cd16, name=ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd-5-deactivate, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, ceph=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, CEPH_POINT_RELEASE=, RELEASE=wip-guits-quincy-2024-09-09-1810-quincy-5be5739, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-10T19:34:34.314 INFO:journalctl@ceph.osd.7.smithi191.stdout:Sep 10 19:34:34 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd-7[56482]: 2024-09-10T19:34:34.067+0000 7f9ef7a9a640 -1 osd.7 461 heartbeat_check: no reply from 172.21.15.183:6814 osd.0 since back 2024-09-10T19:33:48.004787+0000 front 2024-09-10T19:33:48.004721+0000 (oldest deadline 2024-09-10T19:34:11.504575+0000) 2024-09-10T19:34:34.314 INFO:journalctl@ceph.osd.7.smithi191.stdout:Sep 10 19:34:34 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd-7[56482]: 2024-09-10T19:34:34.067+0000 7f9ef7a9a640 -1 osd.7 461 heartbeat_check: no reply from 172.21.15.183:6806 osd.1 since back 2024-09-10T19:33:57.906042+0000 front 2024-09-10T19:33:57.906087+0000 (oldest deadline 2024-09-10T19:34:23.205965+0000) 2024-09-10T19:34:34.314 INFO:journalctl@ceph.osd.7.smithi191.stdout:Sep 10 19:34:34 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd-7[56482]: 2024-09-10T19:34:34.067+0000 7f9ef7a9a640 -1 osd.7 461 heartbeat_check: no reply from 172.21.15.183:6822 osd.2 since back 2024-09-10T19:34:04.306980+0000 front 2024-09-10T19:34:04.307045+0000 (oldest deadline 2024-09-10T19:34:30.206889+0000) 2024-09-10T19:34:34.338 DEBUG:teuthology.orchestra.run.smithi191:> sudo pkill -f 'journalctl -f -n 0 -u ceph-320f6df8-6fa7-11ef-bcea-c7b262605968@osd.5.service' 2024-09-10T19:34:34.376 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-10T19:34:34.377 INFO:tasks.cephadm.osd.5:Stopped osd.5 2024-09-10T19:34:34.377 INFO:tasks.cephadm.osd.6:Stopping osd.6... 2024-09-10T19:34:34.377 DEBUG:teuthology.orchestra.run.smithi191:> sudo systemctl stop ceph-320f6df8-6fa7-11ef-bcea-c7b262605968@osd.6 2024-09-10T19:34:34.703 INFO:journalctl@ceph.osd.6.smithi191.stdout:Sep 10 19:34:34 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd-6[50347]: 2024-09-10T19:34:34.425+0000 7f4ab7287640 -1 osd.6 461 heartbeat_check: no reply from 172.21.15.183:6814 osd.0 since back 2024-09-10T19:33:45.563941+0000 front 2024-09-10T19:33:45.563894+0000 (oldest deadline 2024-09-10T19:34:14.364335+0000) 2024-09-10T19:34:34.703 INFO:journalctl@ceph.osd.6.smithi191.stdout:Sep 10 19:34:34 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd-6[50347]: 2024-09-10T19:34:34.425+0000 7f4ab7287640 -1 osd.6 461 heartbeat_check: no reply from 172.21.15.183:6806 osd.1 since back 2024-09-10T19:33:54.364736+0000 front 2024-09-10T19:33:54.364820+0000 (oldest deadline 2024-09-10T19:34:20.264938+0000) 2024-09-10T19:34:34.703 INFO:journalctl@ceph.osd.6.smithi191.stdout:Sep 10 19:34:34 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd-6[50347]: 2024-09-10T19:34:34.425+0000 7f4ab7287640 -1 osd.6 461 heartbeat_check: no reply from 172.21.15.183:6822 osd.2 since back 2024-09-10T19:34:05.565981+0000 front 2024-09-10T19:34:05.566054+0000 (oldest deadline 2024-09-10T19:34:31.466032+0000) 2024-09-10T19:34:34.703 INFO:journalctl@ceph.osd.6.smithi191.stdout:Sep 10 19:34:34 smithi191 systemd[1]: Stopping Ceph osd.6 for 320f6df8-6fa7-11ef-bcea-c7b262605968... 2024-09-10T19:34:34.703 INFO:journalctl@ceph.osd.6.smithi191.stdout:Sep 10 19:34:34 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd-6[50347]: 2024-09-10T19:34:34.701+0000 7f4abbc71640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-10T19:34:34.704 INFO:journalctl@ceph.osd.6.smithi191.stdout:Sep 10 19:34:34 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd-6[50347]: 2024-09-10T19:34:34.701+0000 7f4abbc71640 -1 osd.6 461 *** Got signal Terminated *** 2024-09-10T19:34:34.704 INFO:journalctl@ceph.osd.6.smithi191.stdout:Sep 10 19:34:34 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd-6[50347]: 2024-09-10T19:34:34.701+0000 7f4abbc71640 -1 osd.6 461 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-10T19:34:35.449 INFO:journalctl@ceph.osd.7.smithi191.stdout:Sep 10 19:34:35 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd-7[56482]: 2024-09-10T19:34:35.026+0000 7f9ef7a9a640 -1 osd.7 461 heartbeat_check: no reply from 172.21.15.183:6814 osd.0 since back 2024-09-10T19:33:48.004787+0000 front 2024-09-10T19:33:48.004721+0000 (oldest deadline 2024-09-10T19:34:11.504575+0000) 2024-09-10T19:34:35.450 INFO:journalctl@ceph.osd.7.smithi191.stdout:Sep 10 19:34:35 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd-7[56482]: 2024-09-10T19:34:35.026+0000 7f9ef7a9a640 -1 osd.7 461 heartbeat_check: no reply from 172.21.15.183:6806 osd.1 since back 2024-09-10T19:33:57.906042+0000 front 2024-09-10T19:33:57.906087+0000 (oldest deadline 2024-09-10T19:34:23.205965+0000) 2024-09-10T19:34:35.450 INFO:journalctl@ceph.osd.7.smithi191.stdout:Sep 10 19:34:35 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd-7[56482]: 2024-09-10T19:34:35.026+0000 7f9ef7a9a640 -1 osd.7 461 heartbeat_check: no reply from 172.21.15.183:6822 osd.2 since back 2024-09-10T19:34:04.306980+0000 front 2024-09-10T19:34:04.307045+0000 (oldest deadline 2024-09-10T19:34:30.206889+0000) 2024-09-10T19:34:35.716 INFO:journalctl@ceph.osd.6.smithi191.stdout:Sep 10 19:34:35 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd-6[50347]: 2024-09-10T19:34:35.447+0000 7f4ab7287640 -1 osd.6 461 heartbeat_check: no reply from 172.21.15.183:6814 osd.0 since back 2024-09-10T19:33:45.563941+0000 front 2024-09-10T19:33:45.563894+0000 (oldest deadline 2024-09-10T19:34:14.364335+0000) 2024-09-10T19:34:35.716 INFO:journalctl@ceph.osd.6.smithi191.stdout:Sep 10 19:34:35 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd-6[50347]: 2024-09-10T19:34:35.447+0000 7f4ab7287640 -1 osd.6 461 heartbeat_check: no reply from 172.21.15.183:6806 osd.1 since back 2024-09-10T19:33:54.364736+0000 front 2024-09-10T19:33:54.364820+0000 (oldest deadline 2024-09-10T19:34:20.264938+0000) 2024-09-10T19:34:35.716 INFO:journalctl@ceph.osd.6.smithi191.stdout:Sep 10 19:34:35 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd-6[50347]: 2024-09-10T19:34:35.447+0000 7f4ab7287640 -1 osd.6 461 heartbeat_check: no reply from 172.21.15.183:6822 osd.2 since back 2024-09-10T19:34:05.565981+0000 front 2024-09-10T19:34:05.566054+0000 (oldest deadline 2024-09-10T19:34:31.466032+0000) 2024-09-10T19:34:36.466 INFO:journalctl@ceph.osd.7.smithi191.stdout:Sep 10 19:34:36 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd-7[56482]: 2024-09-10T19:34:36.043+0000 7f9ef7a9a640 -1 osd.7 461 heartbeat_check: no reply from 172.21.15.183:6814 osd.0 since back 2024-09-10T19:33:48.004787+0000 front 2024-09-10T19:33:48.004721+0000 (oldest deadline 2024-09-10T19:34:11.504575+0000) 2024-09-10T19:34:36.466 INFO:journalctl@ceph.osd.7.smithi191.stdout:Sep 10 19:34:36 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd-7[56482]: 2024-09-10T19:34:36.043+0000 7f9ef7a9a640 -1 osd.7 461 heartbeat_check: no reply from 172.21.15.183:6806 osd.1 since back 2024-09-10T19:33:57.906042+0000 front 2024-09-10T19:33:57.906087+0000 (oldest deadline 2024-09-10T19:34:23.205965+0000) 2024-09-10T19:34:36.466 INFO:journalctl@ceph.osd.7.smithi191.stdout:Sep 10 19:34:36 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd-7[56482]: 2024-09-10T19:34:36.043+0000 7f9ef7a9a640 -1 osd.7 461 heartbeat_check: no reply from 172.21.15.183:6822 osd.2 since back 2024-09-10T19:34:04.306980+0000 front 2024-09-10T19:34:04.307045+0000 (oldest deadline 2024-09-10T19:34:30.206889+0000) 2024-09-10T19:34:36.965 INFO:journalctl@ceph.osd.6.smithi191.stdout:Sep 10 19:34:36 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd-6[50347]: 2024-09-10T19:34:36.480+0000 7f4ab7287640 -1 osd.6 461 heartbeat_check: no reply from 172.21.15.183:6814 osd.0 since back 2024-09-10T19:33:45.563941+0000 front 2024-09-10T19:33:45.563894+0000 (oldest deadline 2024-09-10T19:34:14.364335+0000) 2024-09-10T19:34:36.966 INFO:journalctl@ceph.osd.6.smithi191.stdout:Sep 10 19:34:36 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd-6[50347]: 2024-09-10T19:34:36.480+0000 7f4ab7287640 -1 osd.6 461 heartbeat_check: no reply from 172.21.15.183:6806 osd.1 since back 2024-09-10T19:33:54.364736+0000 front 2024-09-10T19:33:54.364820+0000 (oldest deadline 2024-09-10T19:34:20.264938+0000) 2024-09-10T19:34:36.966 INFO:journalctl@ceph.osd.6.smithi191.stdout:Sep 10 19:34:36 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd-6[50347]: 2024-09-10T19:34:36.480+0000 7f4ab7287640 -1 osd.6 461 heartbeat_check: no reply from 172.21.15.183:6822 osd.2 since back 2024-09-10T19:34:05.565981+0000 front 2024-09-10T19:34:05.566054+0000 (oldest deadline 2024-09-10T19:34:31.466032+0000) 2024-09-10T19:34:36.966 INFO:journalctl@ceph.osd.6.smithi191.stdout:Sep 10 19:34:36 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd-6[50347]: 2024-09-10T19:34:36.480+0000 7f4ab7287640 -1 osd.6 461 heartbeat_check: no reply from 172.21.15.183:6830 osd.3 since back 2024-09-10T19:34:13.067522+0000 front 2024-09-10T19:34:13.067559+0000 (oldest deadline 2024-09-10T19:34:35.967470+0000) 2024-09-10T19:34:37.465 INFO:journalctl@ceph.osd.7.smithi191.stdout:Sep 10 19:34:37 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd-7[56482]: 2024-09-10T19:34:37.082+0000 7f9ef7a9a640 -1 osd.7 461 heartbeat_check: no reply from 172.21.15.183:6814 osd.0 since back 2024-09-10T19:33:48.004787+0000 front 2024-09-10T19:33:48.004721+0000 (oldest deadline 2024-09-10T19:34:11.504575+0000) 2024-09-10T19:34:37.466 INFO:journalctl@ceph.osd.7.smithi191.stdout:Sep 10 19:34:37 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd-7[56482]: 2024-09-10T19:34:37.082+0000 7f9ef7a9a640 -1 osd.7 461 heartbeat_check: no reply from 172.21.15.183:6806 osd.1 since back 2024-09-10T19:33:57.906042+0000 front 2024-09-10T19:33:57.906087+0000 (oldest deadline 2024-09-10T19:34:23.205965+0000) 2024-09-10T19:34:37.466 INFO:journalctl@ceph.osd.7.smithi191.stdout:Sep 10 19:34:37 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd-7[56482]: 2024-09-10T19:34:37.082+0000 7f9ef7a9a640 -1 osd.7 461 heartbeat_check: no reply from 172.21.15.183:6822 osd.2 since back 2024-09-10T19:34:04.306980+0000 front 2024-09-10T19:34:04.307045+0000 (oldest deadline 2024-09-10T19:34:30.206889+0000) 2024-09-10T19:34:37.966 INFO:journalctl@ceph.osd.6.smithi191.stdout:Sep 10 19:34:37 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd-6[50347]: 2024-09-10T19:34:37.523+0000 7f4ab7287640 -1 osd.6 461 heartbeat_check: no reply from 172.21.15.183:6814 osd.0 since back 2024-09-10T19:33:45.563941+0000 front 2024-09-10T19:33:45.563894+0000 (oldest deadline 2024-09-10T19:34:14.364335+0000) 2024-09-10T19:34:37.966 INFO:journalctl@ceph.osd.6.smithi191.stdout:Sep 10 19:34:37 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd-6[50347]: 2024-09-10T19:34:37.523+0000 7f4ab7287640 -1 osd.6 461 heartbeat_check: no reply from 172.21.15.183:6806 osd.1 since back 2024-09-10T19:33:54.364736+0000 front 2024-09-10T19:33:54.364820+0000 (oldest deadline 2024-09-10T19:34:20.264938+0000) 2024-09-10T19:34:37.966 INFO:journalctl@ceph.osd.6.smithi191.stdout:Sep 10 19:34:37 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd-6[50347]: 2024-09-10T19:34:37.523+0000 7f4ab7287640 -1 osd.6 461 heartbeat_check: no reply from 172.21.15.183:6822 osd.2 since back 2024-09-10T19:34:05.565981+0000 front 2024-09-10T19:34:05.566054+0000 (oldest deadline 2024-09-10T19:34:31.466032+0000) 2024-09-10T19:34:37.966 INFO:journalctl@ceph.osd.6.smithi191.stdout:Sep 10 19:34:37 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd-6[50347]: 2024-09-10T19:34:37.523+0000 7f4ab7287640 -1 osd.6 461 heartbeat_check: no reply from 172.21.15.183:6830 osd.3 since back 2024-09-10T19:34:13.067522+0000 front 2024-09-10T19:34:13.067559+0000 (oldest deadline 2024-09-10T19:34:35.967470+0000) 2024-09-10T19:34:38.466 INFO:journalctl@ceph.osd.7.smithi191.stdout:Sep 10 19:34:38 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd-7[56482]: 2024-09-10T19:34:38.127+0000 7f9ef7a9a640 -1 osd.7 461 heartbeat_check: no reply from 172.21.15.183:6814 osd.0 since back 2024-09-10T19:33:48.004787+0000 front 2024-09-10T19:33:48.004721+0000 (oldest deadline 2024-09-10T19:34:11.504575+0000) 2024-09-10T19:34:38.466 INFO:journalctl@ceph.osd.7.smithi191.stdout:Sep 10 19:34:38 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd-7[56482]: 2024-09-10T19:34:38.127+0000 7f9ef7a9a640 -1 osd.7 461 heartbeat_check: no reply from 172.21.15.183:6806 osd.1 since back 2024-09-10T19:33:57.906042+0000 front 2024-09-10T19:33:57.906087+0000 (oldest deadline 2024-09-10T19:34:23.205965+0000) 2024-09-10T19:34:38.466 INFO:journalctl@ceph.osd.7.smithi191.stdout:Sep 10 19:34:38 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd-7[56482]: 2024-09-10T19:34:38.127+0000 7f9ef7a9a640 -1 osd.7 461 heartbeat_check: no reply from 172.21.15.183:6822 osd.2 since back 2024-09-10T19:34:04.306980+0000 front 2024-09-10T19:34:04.307045+0000 (oldest deadline 2024-09-10T19:34:30.206889+0000) 2024-09-10T19:34:38.966 INFO:journalctl@ceph.osd.6.smithi191.stdout:Sep 10 19:34:38 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd-6[50347]: 2024-09-10T19:34:38.573+0000 7f4ab7287640 -1 osd.6 461 heartbeat_check: no reply from 172.21.15.183:6814 osd.0 since back 2024-09-10T19:33:45.563941+0000 front 2024-09-10T19:33:45.563894+0000 (oldest deadline 2024-09-10T19:34:14.364335+0000) 2024-09-10T19:34:38.966 INFO:journalctl@ceph.osd.6.smithi191.stdout:Sep 10 19:34:38 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd-6[50347]: 2024-09-10T19:34:38.573+0000 7f4ab7287640 -1 osd.6 461 heartbeat_check: no reply from 172.21.15.183:6806 osd.1 since back 2024-09-10T19:33:54.364736+0000 front 2024-09-10T19:33:54.364820+0000 (oldest deadline 2024-09-10T19:34:20.264938+0000) 2024-09-10T19:34:38.966 INFO:journalctl@ceph.osd.6.smithi191.stdout:Sep 10 19:34:38 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd-6[50347]: 2024-09-10T19:34:38.573+0000 7f4ab7287640 -1 osd.6 461 heartbeat_check: no reply from 172.21.15.183:6822 osd.2 since back 2024-09-10T19:34:05.565981+0000 front 2024-09-10T19:34:05.566054+0000 (oldest deadline 2024-09-10T19:34:31.466032+0000) 2024-09-10T19:34:38.966 INFO:journalctl@ceph.osd.6.smithi191.stdout:Sep 10 19:34:38 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd-6[50347]: 2024-09-10T19:34:38.573+0000 7f4ab7287640 -1 osd.6 461 heartbeat_check: no reply from 172.21.15.183:6830 osd.3 since back 2024-09-10T19:34:13.067522+0000 front 2024-09-10T19:34:13.067559+0000 (oldest deadline 2024-09-10T19:34:35.967470+0000) 2024-09-10T19:34:39.466 INFO:journalctl@ceph.osd.7.smithi191.stdout:Sep 10 19:34:39 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd-7[56482]: 2024-09-10T19:34:39.081+0000 7f9ef7a9a640 -1 osd.7 461 heartbeat_check: no reply from 172.21.15.183:6814 osd.0 since back 2024-09-10T19:33:48.004787+0000 front 2024-09-10T19:33:48.004721+0000 (oldest deadline 2024-09-10T19:34:11.504575+0000) 2024-09-10T19:34:39.466 INFO:journalctl@ceph.osd.7.smithi191.stdout:Sep 10 19:34:39 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd-7[56482]: 2024-09-10T19:34:39.081+0000 7f9ef7a9a640 -1 osd.7 461 heartbeat_check: no reply from 172.21.15.183:6806 osd.1 since back 2024-09-10T19:33:57.906042+0000 front 2024-09-10T19:33:57.906087+0000 (oldest deadline 2024-09-10T19:34:23.205965+0000) 2024-09-10T19:34:39.466 INFO:journalctl@ceph.osd.7.smithi191.stdout:Sep 10 19:34:39 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd-7[56482]: 2024-09-10T19:34:39.081+0000 7f9ef7a9a640 -1 osd.7 461 heartbeat_check: no reply from 172.21.15.183:6822 osd.2 since back 2024-09-10T19:34:04.306980+0000 front 2024-09-10T19:34:04.307045+0000 (oldest deadline 2024-09-10T19:34:30.206889+0000) 2024-09-10T19:34:39.467 INFO:journalctl@ceph.osd.7.smithi191.stdout:Sep 10 19:34:39 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd-7[56482]: 2024-09-10T19:34:39.081+0000 7f9ef7a9a640 -1 osd.7 461 heartbeat_check: no reply from 172.21.15.183:6830 osd.3 since back 2024-09-10T19:34:13.008226+0000 front 2024-09-10T19:34:13.008269+0000 (oldest deadline 2024-09-10T19:34:38.908212+0000) 2024-09-10T19:34:39.965 INFO:journalctl@ceph.osd.6.smithi191.stdout:Sep 10 19:34:39 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd-6[50347]: 2024-09-10T19:34:39.591+0000 7f4ab7287640 -1 osd.6 461 heartbeat_check: no reply from 172.21.15.183:6814 osd.0 since back 2024-09-10T19:33:45.563941+0000 front 2024-09-10T19:33:45.563894+0000 (oldest deadline 2024-09-10T19:34:14.364335+0000) 2024-09-10T19:34:39.966 INFO:journalctl@ceph.osd.6.smithi191.stdout:Sep 10 19:34:39 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd-6[50347]: 2024-09-10T19:34:39.591+0000 7f4ab7287640 -1 osd.6 461 heartbeat_check: no reply from 172.21.15.183:6806 osd.1 since back 2024-09-10T19:33:54.364736+0000 front 2024-09-10T19:33:54.364820+0000 (oldest deadline 2024-09-10T19:34:20.264938+0000) 2024-09-10T19:34:39.966 INFO:journalctl@ceph.osd.6.smithi191.stdout:Sep 10 19:34:39 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd-6[50347]: 2024-09-10T19:34:39.591+0000 7f4ab7287640 -1 osd.6 461 heartbeat_check: no reply from 172.21.15.183:6822 osd.2 since back 2024-09-10T19:34:05.565981+0000 front 2024-09-10T19:34:05.566054+0000 (oldest deadline 2024-09-10T19:34:31.466032+0000) 2024-09-10T19:34:39.966 INFO:journalctl@ceph.osd.6.smithi191.stdout:Sep 10 19:34:39 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd-6[50347]: 2024-09-10T19:34:39.591+0000 7f4ab7287640 -1 osd.6 461 heartbeat_check: no reply from 172.21.15.183:6830 osd.3 since back 2024-09-10T19:34:13.067522+0000 front 2024-09-10T19:34:13.067559+0000 (oldest deadline 2024-09-10T19:34:35.967470+0000) 2024-09-10T19:34:40.426 INFO:journalctl@ceph.osd.6.smithi191.stdout:Sep 10 19:34:40 smithi191 podman[165625]: 2024-09-10 19:34:40.126944345 +0000 UTC m=+5.529943071 container died 86aab1538733c4a670daa481b018734c5c8b3717a86ac1476dfbe7bd44458bad (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3bfeaf4417585c821cab9ca76e4d95f390b0e769807122a45463640820b5cd16, name=ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd-6, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, ceph=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, GIT_CLEAN=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , RELEASE=wip-guits-quincy-2024-09-09-1810-quincy-5be5739, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS) 2024-09-10T19:34:40.426 INFO:journalctl@ceph.osd.7.smithi191.stdout:Sep 10 19:34:40 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd-7[56482]: 2024-09-10T19:34:40.068+0000 7f9ef7a9a640 -1 osd.7 461 heartbeat_check: no reply from 172.21.15.183:6814 osd.0 since back 2024-09-10T19:33:48.004787+0000 front 2024-09-10T19:33:48.004721+0000 (oldest deadline 2024-09-10T19:34:11.504575+0000) 2024-09-10T19:34:40.426 INFO:journalctl@ceph.osd.7.smithi191.stdout:Sep 10 19:34:40 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd-7[56482]: 2024-09-10T19:34:40.068+0000 7f9ef7a9a640 -1 osd.7 461 heartbeat_check: no reply from 172.21.15.183:6806 osd.1 since back 2024-09-10T19:33:57.906042+0000 front 2024-09-10T19:33:57.906087+0000 (oldest deadline 2024-09-10T19:34:23.205965+0000) 2024-09-10T19:34:40.426 INFO:journalctl@ceph.osd.7.smithi191.stdout:Sep 10 19:34:40 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd-7[56482]: 2024-09-10T19:34:40.068+0000 7f9ef7a9a640 -1 osd.7 461 heartbeat_check: no reply from 172.21.15.183:6822 osd.2 since back 2024-09-10T19:34:04.306980+0000 front 2024-09-10T19:34:04.307045+0000 (oldest deadline 2024-09-10T19:34:30.206889+0000) 2024-09-10T19:34:40.426 INFO:journalctl@ceph.osd.7.smithi191.stdout:Sep 10 19:34:40 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd-7[56482]: 2024-09-10T19:34:40.068+0000 7f9ef7a9a640 -1 osd.7 461 heartbeat_check: no reply from 172.21.15.183:6830 osd.3 since back 2024-09-10T19:34:13.008226+0000 front 2024-09-10T19:34:13.008269+0000 (oldest deadline 2024-09-10T19:34:38.908212+0000) 2024-09-10T19:34:40.685 INFO:journalctl@ceph.osd.6.smithi191.stdout:Sep 10 19:34:40 smithi191 podman[165625]: 2024-09-10 19:34:40.515500235 +0000 UTC m=+5.918498960 container cleanup 86aab1538733c4a670daa481b018734c5c8b3717a86ac1476dfbe7bd44458bad (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3bfeaf4417585c821cab9ca76e4d95f390b0e769807122a45463640820b5cd16, name=ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd-6, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-guits-quincy-2024-09-09-1810-quincy-5be5739, CEPH_POINT_RELEASE=, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, GIT_CLEAN=True, ceph=True) 2024-09-10T19:34:40.685 INFO:journalctl@ceph.osd.6.smithi191.stdout:Sep 10 19:34:40 smithi191 bash[165625]: ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd-6 2024-09-10T19:34:40.965 INFO:journalctl@ceph.osd.6.smithi191.stdout:Sep 10 19:34:40 smithi191 podman[165644]: 2024-09-10 19:34:40.683639424 +0000 UTC m=+0.556859665 container remove 86aab1538733c4a670daa481b018734c5c8b3717a86ac1476dfbe7bd44458bad (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3bfeaf4417585c821cab9ca76e4d95f390b0e769807122a45463640820b5cd16, name=ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd-6, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, ceph=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.build-date=20240731, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, RELEASE=wip-guits-quincy-2024-09-09-1810-quincy-5be5739, org.label-schema.license=GPLv2) 2024-09-10T19:34:41.332 INFO:journalctl@ceph.osd.6.smithi191.stdout:Sep 10 19:34:41 smithi191 podman[165737]: 2024-09-10 19:34:40.98519963 +0000 UTC m=+0.015682198 image pull 4c55dd4096115e48502f368cff1eab06769f71f030e1692e2611bbc6c6f637e9 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3bfeaf4417585c821cab9ca76e4d95f390b0e769807122a45463640820b5cd16 2024-09-10T19:34:41.332 INFO:journalctl@ceph.osd.6.smithi191.stdout:Sep 10 19:34:41 smithi191 podman[165737]: 2024-09-10 19:34:41.092814294 +0000 UTC m=+0.123296863 container create 7825890d9c7fa9f59dc83c22aef54fd62c2e577322e14563086a0522b126b4f1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3bfeaf4417585c821cab9ca76e4d95f390b0e769807122a45463640820b5cd16, name=ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd-6-deactivate, io.buildah.version=1.37.2, RELEASE=wip-guits-quincy-2024-09-09-1810-quincy-5be5739, org.label-schema.license=GPLv2, ceph=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_CLEAN=True) 2024-09-10T19:34:41.332 INFO:journalctl@ceph.osd.6.smithi191.stdout:Sep 10 19:34:41 smithi191 podman[165737]: 2024-09-10 19:34:41.328931357 +0000 UTC m=+0.359413927 container init 7825890d9c7fa9f59dc83c22aef54fd62c2e577322e14563086a0522b126b4f1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3bfeaf4417585c821cab9ca76e4d95f390b0e769807122a45463640820b5cd16, name=ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd-6-deactivate, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, ceph=True, GIT_CLEAN=True, RELEASE=wip-guits-quincy-2024-09-09-1810-quincy-5be5739, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-10T19:34:41.332 INFO:journalctl@ceph.osd.7.smithi191.stdout:Sep 10 19:34:41 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd-7[56482]: 2024-09-10T19:34:41.063+0000 7f9ef7a9a640 -1 osd.7 461 heartbeat_check: no reply from 172.21.15.183:6814 osd.0 since back 2024-09-10T19:33:48.004787+0000 front 2024-09-10T19:33:48.004721+0000 (oldest deadline 2024-09-10T19:34:11.504575+0000) 2024-09-10T19:34:41.332 INFO:journalctl@ceph.osd.7.smithi191.stdout:Sep 10 19:34:41 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd-7[56482]: 2024-09-10T19:34:41.063+0000 7f9ef7a9a640 -1 osd.7 461 heartbeat_check: no reply from 172.21.15.183:6806 osd.1 since back 2024-09-10T19:33:57.906042+0000 front 2024-09-10T19:33:57.906087+0000 (oldest deadline 2024-09-10T19:34:23.205965+0000) 2024-09-10T19:34:41.332 INFO:journalctl@ceph.osd.7.smithi191.stdout:Sep 10 19:34:41 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd-7[56482]: 2024-09-10T19:34:41.063+0000 7f9ef7a9a640 -1 osd.7 461 heartbeat_check: no reply from 172.21.15.183:6822 osd.2 since back 2024-09-10T19:34:04.306980+0000 front 2024-09-10T19:34:04.307045+0000 (oldest deadline 2024-09-10T19:34:30.206889+0000) 2024-09-10T19:34:41.333 INFO:journalctl@ceph.osd.7.smithi191.stdout:Sep 10 19:34:41 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd-7[56482]: 2024-09-10T19:34:41.063+0000 7f9ef7a9a640 -1 osd.7 461 heartbeat_check: no reply from 172.21.15.183:6830 osd.3 since back 2024-09-10T19:34:13.008226+0000 front 2024-09-10T19:34:13.008269+0000 (oldest deadline 2024-09-10T19:34:38.908212+0000) 2024-09-10T19:34:41.716 INFO:journalctl@ceph.osd.6.smithi191.stdout:Sep 10 19:34:41 smithi191 podman[165737]: 2024-09-10 19:34:41.333848842 +0000 UTC m=+0.364331414 container start 7825890d9c7fa9f59dc83c22aef54fd62c2e577322e14563086a0522b126b4f1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3bfeaf4417585c821cab9ca76e4d95f390b0e769807122a45463640820b5cd16, name=ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd-6-deactivate, org.label-schema.build-date=20240731, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, RELEASE=wip-guits-quincy-2024-09-09-1810-quincy-5be5739, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, io.buildah.version=1.37.2, ceph=True) 2024-09-10T19:34:41.716 INFO:journalctl@ceph.osd.6.smithi191.stdout:Sep 10 19:34:41 smithi191 podman[165737]: 2024-09-10 19:34:41.437282139 +0000 UTC m=+0.467764723 container attach 7825890d9c7fa9f59dc83c22aef54fd62c2e577322e14563086a0522b126b4f1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3bfeaf4417585c821cab9ca76e4d95f390b0e769807122a45463640820b5cd16, name=ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd-6-deactivate, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, ceph=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, RELEASE=wip-guits-quincy-2024-09-09-1810-quincy-5be5739) 2024-09-10T19:34:41.716 INFO:journalctl@ceph.osd.6.smithi191.stdout:Sep 10 19:34:41 smithi191 podman[165737]: 2024-09-10 19:34:41.505664759 +0000 UTC m=+0.536147328 container died 7825890d9c7fa9f59dc83c22aef54fd62c2e577322e14563086a0522b126b4f1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3bfeaf4417585c821cab9ca76e4d95f390b0e769807122a45463640820b5cd16, name=ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd-6-deactivate, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.schema-version=1.0, GIT_CLEAN=True, RELEASE=wip-guits-quincy-2024-09-09-1810-quincy-5be5739, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731) 2024-09-10T19:34:42.125 INFO:journalctl@ceph.osd.7.smithi191.stdout:Sep 10 19:34:42 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd-7[56482]: 2024-09-10T19:34:42.068+0000 7f9ef7a9a640 -1 osd.7 461 heartbeat_check: no reply from 172.21.15.183:6814 osd.0 since back 2024-09-10T19:33:48.004787+0000 front 2024-09-10T19:33:48.004721+0000 (oldest deadline 2024-09-10T19:34:11.504575+0000) 2024-09-10T19:34:42.125 INFO:journalctl@ceph.osd.7.smithi191.stdout:Sep 10 19:34:42 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd-7[56482]: 2024-09-10T19:34:42.068+0000 7f9ef7a9a640 -1 osd.7 461 heartbeat_check: no reply from 172.21.15.183:6806 osd.1 since back 2024-09-10T19:33:57.906042+0000 front 2024-09-10T19:33:57.906087+0000 (oldest deadline 2024-09-10T19:34:23.205965+0000) 2024-09-10T19:34:42.125 INFO:journalctl@ceph.osd.7.smithi191.stdout:Sep 10 19:34:42 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd-7[56482]: 2024-09-10T19:34:42.068+0000 7f9ef7a9a640 -1 osd.7 461 heartbeat_check: no reply from 172.21.15.183:6822 osd.2 since back 2024-09-10T19:34:04.306980+0000 front 2024-09-10T19:34:04.307045+0000 (oldest deadline 2024-09-10T19:34:30.206889+0000) 2024-09-10T19:34:42.125 INFO:journalctl@ceph.osd.7.smithi191.stdout:Sep 10 19:34:42 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd-7[56482]: 2024-09-10T19:34:42.068+0000 7f9ef7a9a640 -1 osd.7 461 heartbeat_check: no reply from 172.21.15.183:6830 osd.3 since back 2024-09-10T19:34:13.008226+0000 front 2024-09-10T19:34:13.008269+0000 (oldest deadline 2024-09-10T19:34:38.908212+0000) 2024-09-10T19:34:42.147 DEBUG:teuthology.orchestra.run.smithi191:> sudo pkill -f 'journalctl -f -n 0 -u ceph-320f6df8-6fa7-11ef-bcea-c7b262605968@osd.6.service' 2024-09-10T19:34:42.185 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-10T19:34:42.185 INFO:tasks.cephadm.osd.6:Stopped osd.6 2024-09-10T19:34:42.185 INFO:tasks.cephadm.osd.7:Stopping osd.7... 2024-09-10T19:34:42.186 DEBUG:teuthology.orchestra.run.smithi191:> sudo systemctl stop ceph-320f6df8-6fa7-11ef-bcea-c7b262605968@osd.7 2024-09-10T19:34:42.466 INFO:journalctl@ceph.osd.7.smithi191.stdout:Sep 10 19:34:42 smithi191 systemd[1]: Stopping Ceph osd.7 for 320f6df8-6fa7-11ef-bcea-c7b262605968... 2024-09-10T19:34:42.965 INFO:journalctl@ceph.osd.7.smithi191.stdout:Sep 10 19:34:42 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd-7[56482]: 2024-09-10T19:34:42.529+0000 7f9efe488640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-10T19:34:42.966 INFO:journalctl@ceph.osd.7.smithi191.stdout:Sep 10 19:34:42 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd-7[56482]: 2024-09-10T19:34:42.529+0000 7f9efe488640 -1 osd.7 461 *** Got signal Terminated *** 2024-09-10T19:34:42.966 INFO:journalctl@ceph.osd.7.smithi191.stdout:Sep 10 19:34:42 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd-7[56482]: 2024-09-10T19:34:42.529+0000 7f9efe488640 -1 osd.7 461 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-10T19:34:43.466 INFO:journalctl@ceph.osd.7.smithi191.stdout:Sep 10 19:34:43 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd-7[56482]: 2024-09-10T19:34:43.053+0000 7f9ef7a9a640 -1 osd.7 461 heartbeat_check: no reply from 172.21.15.183:6814 osd.0 since back 2024-09-10T19:33:48.004787+0000 front 2024-09-10T19:33:48.004721+0000 (oldest deadline 2024-09-10T19:34:11.504575+0000) 2024-09-10T19:34:43.466 INFO:journalctl@ceph.osd.7.smithi191.stdout:Sep 10 19:34:43 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd-7[56482]: 2024-09-10T19:34:43.053+0000 7f9ef7a9a640 -1 osd.7 461 heartbeat_check: no reply from 172.21.15.183:6806 osd.1 since back 2024-09-10T19:33:57.906042+0000 front 2024-09-10T19:33:57.906087+0000 (oldest deadline 2024-09-10T19:34:23.205965+0000) 2024-09-10T19:34:43.466 INFO:journalctl@ceph.osd.7.smithi191.stdout:Sep 10 19:34:43 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd-7[56482]: 2024-09-10T19:34:43.053+0000 7f9ef7a9a640 -1 osd.7 461 heartbeat_check: no reply from 172.21.15.183:6822 osd.2 since back 2024-09-10T19:34:04.306980+0000 front 2024-09-10T19:34:04.307045+0000 (oldest deadline 2024-09-10T19:34:30.206889+0000) 2024-09-10T19:34:43.466 INFO:journalctl@ceph.osd.7.smithi191.stdout:Sep 10 19:34:43 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd-7[56482]: 2024-09-10T19:34:43.053+0000 7f9ef7a9a640 -1 osd.7 461 heartbeat_check: no reply from 172.21.15.183:6830 osd.3 since back 2024-09-10T19:34:13.008226+0000 front 2024-09-10T19:34:13.008269+0000 (oldest deadline 2024-09-10T19:34:38.908212+0000) 2024-09-10T19:34:44.463 INFO:journalctl@ceph.osd.7.smithi191.stdout:Sep 10 19:34:44 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd-7[56482]: 2024-09-10T19:34:44.080+0000 7f9ef7a9a640 -1 osd.7 461 heartbeat_check: no reply from 172.21.15.183:6814 osd.0 since back 2024-09-10T19:33:48.004787+0000 front 2024-09-10T19:33:48.004721+0000 (oldest deadline 2024-09-10T19:34:11.504575+0000) 2024-09-10T19:34:44.463 INFO:journalctl@ceph.osd.7.smithi191.stdout:Sep 10 19:34:44 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd-7[56482]: 2024-09-10T19:34:44.080+0000 7f9ef7a9a640 -1 osd.7 461 heartbeat_check: no reply from 172.21.15.183:6806 osd.1 since back 2024-09-10T19:33:57.906042+0000 front 2024-09-10T19:33:57.906087+0000 (oldest deadline 2024-09-10T19:34:23.205965+0000) 2024-09-10T19:34:44.463 INFO:journalctl@ceph.osd.7.smithi191.stdout:Sep 10 19:34:44 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd-7[56482]: 2024-09-10T19:34:44.080+0000 7f9ef7a9a640 -1 osd.7 461 heartbeat_check: no reply from 172.21.15.183:6822 osd.2 since back 2024-09-10T19:34:04.306980+0000 front 2024-09-10T19:34:04.307045+0000 (oldest deadline 2024-09-10T19:34:30.206889+0000) 2024-09-10T19:34:44.463 INFO:journalctl@ceph.osd.7.smithi191.stdout:Sep 10 19:34:44 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd-7[56482]: 2024-09-10T19:34:44.080+0000 7f9ef7a9a640 -1 osd.7 461 heartbeat_check: no reply from 172.21.15.183:6830 osd.3 since back 2024-09-10T19:34:13.008226+0000 front 2024-09-10T19:34:13.008269+0000 (oldest deadline 2024-09-10T19:34:38.908212+0000) 2024-09-10T19:34:45.466 INFO:journalctl@ceph.osd.7.smithi191.stdout:Sep 10 19:34:45 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd-7[56482]: 2024-09-10T19:34:45.044+0000 7f9ef7a9a640 -1 osd.7 461 heartbeat_check: no reply from 172.21.15.183:6814 osd.0 since back 2024-09-10T19:33:48.004787+0000 front 2024-09-10T19:33:48.004721+0000 (oldest deadline 2024-09-10T19:34:11.504575+0000) 2024-09-10T19:34:45.466 INFO:journalctl@ceph.osd.7.smithi191.stdout:Sep 10 19:34:45 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd-7[56482]: 2024-09-10T19:34:45.044+0000 7f9ef7a9a640 -1 osd.7 461 heartbeat_check: no reply from 172.21.15.183:6806 osd.1 since back 2024-09-10T19:33:57.906042+0000 front 2024-09-10T19:33:57.906087+0000 (oldest deadline 2024-09-10T19:34:23.205965+0000) 2024-09-10T19:34:45.466 INFO:journalctl@ceph.osd.7.smithi191.stdout:Sep 10 19:34:45 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd-7[56482]: 2024-09-10T19:34:45.044+0000 7f9ef7a9a640 -1 osd.7 461 heartbeat_check: no reply from 172.21.15.183:6822 osd.2 since back 2024-09-10T19:34:04.306980+0000 front 2024-09-10T19:34:04.307045+0000 (oldest deadline 2024-09-10T19:34:30.206889+0000) 2024-09-10T19:34:45.466 INFO:journalctl@ceph.osd.7.smithi191.stdout:Sep 10 19:34:45 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd-7[56482]: 2024-09-10T19:34:45.044+0000 7f9ef7a9a640 -1 osd.7 461 heartbeat_check: no reply from 172.21.15.183:6830 osd.3 since back 2024-09-10T19:34:13.008226+0000 front 2024-09-10T19:34:13.008269+0000 (oldest deadline 2024-09-10T19:34:38.908212+0000) 2024-09-10T19:34:46.466 INFO:journalctl@ceph.osd.7.smithi191.stdout:Sep 10 19:34:45 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd-7[56482]: 2024-09-10T19:34:45.996+0000 7f9ef7a9a640 -1 osd.7 461 heartbeat_check: no reply from 172.21.15.183:6814 osd.0 since back 2024-09-10T19:33:48.004787+0000 front 2024-09-10T19:33:48.004721+0000 (oldest deadline 2024-09-10T19:34:11.504575+0000) 2024-09-10T19:34:46.466 INFO:journalctl@ceph.osd.7.smithi191.stdout:Sep 10 19:34:45 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd-7[56482]: 2024-09-10T19:34:45.996+0000 7f9ef7a9a640 -1 osd.7 461 heartbeat_check: no reply from 172.21.15.183:6806 osd.1 since back 2024-09-10T19:33:57.906042+0000 front 2024-09-10T19:33:57.906087+0000 (oldest deadline 2024-09-10T19:34:23.205965+0000) 2024-09-10T19:34:46.466 INFO:journalctl@ceph.osd.7.smithi191.stdout:Sep 10 19:34:45 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd-7[56482]: 2024-09-10T19:34:45.996+0000 7f9ef7a9a640 -1 osd.7 461 heartbeat_check: no reply from 172.21.15.183:6822 osd.2 since back 2024-09-10T19:34:04.306980+0000 front 2024-09-10T19:34:04.307045+0000 (oldest deadline 2024-09-10T19:34:30.206889+0000) 2024-09-10T19:34:46.466 INFO:journalctl@ceph.osd.7.smithi191.stdout:Sep 10 19:34:45 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd-7[56482]: 2024-09-10T19:34:45.996+0000 7f9ef7a9a640 -1 osd.7 461 heartbeat_check: no reply from 172.21.15.183:6830 osd.3 since back 2024-09-10T19:34:13.008226+0000 front 2024-09-10T19:34:13.008269+0000 (oldest deadline 2024-09-10T19:34:38.908212+0000) 2024-09-10T19:34:47.465 INFO:journalctl@ceph.osd.7.smithi191.stdout:Sep 10 19:34:47 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd-7[56482]: 2024-09-10T19:34:46.999+0000 7f9ef7a9a640 -1 osd.7 461 heartbeat_check: no reply from 172.21.15.183:6814 osd.0 since back 2024-09-10T19:33:48.004787+0000 front 2024-09-10T19:33:48.004721+0000 (oldest deadline 2024-09-10T19:34:11.504575+0000) 2024-09-10T19:34:47.466 INFO:journalctl@ceph.osd.7.smithi191.stdout:Sep 10 19:34:47 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd-7[56482]: 2024-09-10T19:34:46.999+0000 7f9ef7a9a640 -1 osd.7 461 heartbeat_check: no reply from 172.21.15.183:6806 osd.1 since back 2024-09-10T19:33:57.906042+0000 front 2024-09-10T19:33:57.906087+0000 (oldest deadline 2024-09-10T19:34:23.205965+0000) 2024-09-10T19:34:47.466 INFO:journalctl@ceph.osd.7.smithi191.stdout:Sep 10 19:34:47 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd-7[56482]: 2024-09-10T19:34:46.999+0000 7f9ef7a9a640 -1 osd.7 461 heartbeat_check: no reply from 172.21.15.183:6822 osd.2 since back 2024-09-10T19:34:04.306980+0000 front 2024-09-10T19:34:04.307045+0000 (oldest deadline 2024-09-10T19:34:30.206889+0000) 2024-09-10T19:34:47.466 INFO:journalctl@ceph.osd.7.smithi191.stdout:Sep 10 19:34:47 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd-7[56482]: 2024-09-10T19:34:46.999+0000 7f9ef7a9a640 -1 osd.7 461 heartbeat_check: no reply from 172.21.15.183:6830 osd.3 since back 2024-09-10T19:34:13.008226+0000 front 2024-09-10T19:34:13.008269+0000 (oldest deadline 2024-09-10T19:34:38.908212+0000) 2024-09-10T19:34:47.466 INFO:journalctl@ceph.osd.7.smithi191.stdout:Sep 10 19:34:47 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd-7[56482]: 2024-09-10T19:34:46.999+0000 7f9ef7a9a640 -1 osd.7 461 heartbeat_check: no reply from 172.21.15.191:6804 osd.4 since back 2024-09-10T19:34:20.608994+0000 front 2024-09-10T19:34:20.609074+0000 (oldest deadline 2024-09-10T19:34:46.508960+0000) 2024-09-10T19:34:48.321 INFO:journalctl@ceph.osd.7.smithi191.stdout:Sep 10 19:34:48 smithi191 podman[165866]: 2024-09-10 19:34:48.046666907 +0000 UTC m=+5.634024028 container died a476560dfabe752e76f47a698298cbdfecb81352b4d98f4ae2dd251f0e8d4c12 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3bfeaf4417585c821cab9ca76e4d95f390b0e769807122a45463640820b5cd16, name=ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd-7, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , ceph=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=wip-guits-quincy-2024-09-09-1810-quincy-5be5739, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2) 2024-09-10T19:34:48.688 INFO:journalctl@ceph.osd.7.smithi191.stdout:Sep 10 19:34:48 smithi191 podman[165866]: 2024-09-10 19:34:48.42689484 +0000 UTC m=+6.014251951 container cleanup a476560dfabe752e76f47a698298cbdfecb81352b4d98f4ae2dd251f0e8d4c12 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3bfeaf4417585c821cab9ca76e4d95f390b0e769807122a45463640820b5cd16, name=ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd-7, GIT_CLEAN=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-guits-quincy-2024-09-09-1810-quincy-5be5739, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True) 2024-09-10T19:34:48.688 INFO:journalctl@ceph.osd.7.smithi191.stdout:Sep 10 19:34:48 smithi191 bash[165866]: ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd-7 2024-09-10T19:34:48.965 INFO:journalctl@ceph.osd.7.smithi191.stdout:Sep 10 19:34:48 smithi191 podman[165899]: 2024-09-10 19:34:48.686998466 +0000 UTC m=+0.636562933 container remove a476560dfabe752e76f47a698298cbdfecb81352b4d98f4ae2dd251f0e8d4c12 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3bfeaf4417585c821cab9ca76e4d95f390b0e769807122a45463640820b5cd16, name=ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd-7, RELEASE=wip-guits-quincy-2024-09-09-1810-quincy-5be5739, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS) 2024-09-10T19:34:49.291 INFO:journalctl@ceph.osd.7.smithi191.stdout:Sep 10 19:34:49 smithi191 podman[165991]: 2024-09-10 19:34:48.940446942 +0000 UTC m=+0.016152723 image pull 4c55dd4096115e48502f368cff1eab06769f71f030e1692e2611bbc6c6f637e9 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3bfeaf4417585c821cab9ca76e4d95f390b0e769807122a45463640820b5cd16 2024-09-10T19:34:49.291 INFO:journalctl@ceph.osd.7.smithi191.stdout:Sep 10 19:34:49 smithi191 podman[165991]: 2024-09-10 19:34:49.062877148 +0000 UTC m=+0.138582969 container create 2d731b506a83a1aeaa0ff4d842ae1b89d9cf091a19f83e3146fa77da35c32992 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3bfeaf4417585c821cab9ca76e4d95f390b0e769807122a45463640820b5cd16, name=ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd-7-deactivate, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, RELEASE=wip-guits-quincy-2024-09-09-1810-quincy-5be5739, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_CLEAN=True, io.buildah.version=1.37.2) 2024-09-10T19:34:49.291 INFO:journalctl@ceph.osd.7.smithi191.stdout:Sep 10 19:34:49 smithi191 podman[165991]: 2024-09-10 19:34:49.290223266 +0000 UTC m=+0.365929041 container init 2d731b506a83a1aeaa0ff4d842ae1b89d9cf091a19f83e3146fa77da35c32992 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3bfeaf4417585c821cab9ca76e4d95f390b0e769807122a45463640820b5cd16, name=ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd-7-deactivate, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, maintainer=Guillaume Abrioux , GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, RELEASE=wip-guits-quincy-2024-09-09-1810-quincy-5be5739, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD) 2024-09-10T19:34:49.715 INFO:journalctl@ceph.osd.7.smithi191.stdout:Sep 10 19:34:49 smithi191 podman[165991]: 2024-09-10 19:34:49.295186218 +0000 UTC m=+0.370892002 container start 2d731b506a83a1aeaa0ff4d842ae1b89d9cf091a19f83e3146fa77da35c32992 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3bfeaf4417585c821cab9ca76e4d95f390b0e769807122a45463640820b5cd16, name=ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd-7-deactivate, org.label-schema.build-date=20240731, GIT_CLEAN=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , RELEASE=wip-guits-quincy-2024-09-09-1810-quincy-5be5739, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.vendor=CentOS) 2024-09-10T19:34:49.716 INFO:journalctl@ceph.osd.7.smithi191.stdout:Sep 10 19:34:49 smithi191 podman[165991]: 2024-09-10 19:34:49.3901205 +0000 UTC m=+0.465826278 container attach 2d731b506a83a1aeaa0ff4d842ae1b89d9cf091a19f83e3146fa77da35c32992 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3bfeaf4417585c821cab9ca76e4d95f390b0e769807122a45463640820b5cd16, name=ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd-7-deactivate, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, ceph=True, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_CLEAN=True, RELEASE=wip-guits-quincy-2024-09-09-1810-quincy-5be5739, org.label-schema.schema-version=1.0) 2024-09-10T19:34:49.716 INFO:journalctl@ceph.osd.7.smithi191.stdout:Sep 10 19:34:49 smithi191 podman[165991]: 2024-09-10 19:34:49.464765939 +0000 UTC m=+0.540471715 container died 2d731b506a83a1aeaa0ff4d842ae1b89d9cf091a19f83e3146fa77da35c32992 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3bfeaf4417585c821cab9ca76e4d95f390b0e769807122a45463640820b5cd16, name=ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-osd-7-deactivate, RELEASE=wip-guits-quincy-2024-09-09-1810-quincy-5be5739, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_CLEAN=True) 2024-09-10T19:34:50.154 DEBUG:teuthology.orchestra.run.smithi191:> sudo pkill -f 'journalctl -f -n 0 -u ceph-320f6df8-6fa7-11ef-bcea-c7b262605968@osd.7.service' 2024-09-10T19:34:50.195 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-10T19:34:50.195 INFO:tasks.cephadm.osd.7:Stopped osd.7 2024-09-10T19:34:50.196 INFO:tasks.cephadm.ceph.rgw.foo.a:Stopping rgw.foo.a... 2024-09-10T19:34:50.196 DEBUG:teuthology.orchestra.run.smithi183:> sudo systemctl stop ceph-320f6df8-6fa7-11ef-bcea-c7b262605968@rgw.foo.a 2024-09-10T19:34:50.489 INFO:journalctl@ceph.rgw.foo.a.smithi183.stdout:Sep 10 19:34:50 smithi183 systemd[1]: Stopping Ceph rgw.foo.a for 320f6df8-6fa7-11ef-bcea-c7b262605968... 2024-09-10T19:34:50.931 INFO:journalctl@ceph.rgw.foo.a.smithi183.stdout:Sep 10 19:34:50 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-rgw-foo-a[70573]: 2024-09-10T19:34:50.486+0000 7f598affd640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/radosgw -n client.rgw.foo.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-10T19:34:50.931 INFO:journalctl@ceph.rgw.foo.a.smithi183.stdout:Sep 10 19:34:50 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-rgw-foo-a[70573]: 2024-09-10T19:34:50.486+0000 7f599383d7c0 -1 shutting down 2024-09-10T19:35:00.887 INFO:journalctl@ceph.rgw.foo.a.smithi183.stdout:Sep 10 19:35:00 smithi183 bash[240281]: time="2024-09-10T19:35:00Z" level=warning msg="StopSignal SIGTERM failed to stop container ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-rgw-foo-a in 10 seconds, resorting to SIGKILL" 2024-09-10T19:35:00.887 INFO:journalctl@ceph.rgw.foo.a.smithi183.stdout:Sep 10 19:35:00 smithi183 podman[240281]: 2024-09-10 19:35:00.504949478 +0000 UTC m=+10.117738159 container died 13ddc2616ff7061707e71a81e5b7090a646126088d968f8f6d8cbb1b10ad530e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3bfeaf4417585c821cab9ca76e4d95f390b0e769807122a45463640820b5cd16, name=ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-rgw-foo-a, io.buildah.version=1.37.2, ceph=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, RELEASE=wip-guits-quincy-2024-09-09-1810-quincy-5be5739) 2024-09-10T19:35:01.181 INFO:journalctl@ceph.rgw.foo.a.smithi183.stdout:Sep 10 19:35:01 smithi183 podman[240281]: 2024-09-10 19:35:01.100211935 +0000 UTC m=+10.713000608 container cleanup 13ddc2616ff7061707e71a81e5b7090a646126088d968f8f6d8cbb1b10ad530e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3bfeaf4417585c821cab9ca76e4d95f390b0e769807122a45463640820b5cd16, name=ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-rgw-foo-a, RELEASE=wip-guits-quincy-2024-09-09-1810-quincy-5be5739, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, GIT_CLEAN=True, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, ceph=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS) 2024-09-10T19:35:01.181 INFO:journalctl@ceph.rgw.foo.a.smithi183.stdout:Sep 10 19:35:01 smithi183 bash[240281]: ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-rgw-foo-a 2024-09-10T19:35:01.586 DEBUG:teuthology.orchestra.run.smithi183:> sudo pkill -f 'journalctl -f -n 0 -u ceph-320f6df8-6fa7-11ef-bcea-c7b262605968@rgw.foo.a.service' 2024-09-10T19:35:01.624 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-10T19:35:01.625 INFO:tasks.cephadm.ceph.rgw.foo.a:Stopped rgw.foo.a 2024-09-10T19:35:01.625 INFO:tasks.cephadm.prometheus.a:Stopping prometheus.a... 2024-09-10T19:35:01.625 DEBUG:teuthology.orchestra.run.smithi191:> sudo systemctl stop ceph-320f6df8-6fa7-11ef-bcea-c7b262605968@prometheus.a 2024-09-10T19:35:01.915 INFO:journalctl@ceph.prometheus.a.smithi191.stdout:Sep 10 19:35:01 smithi191 systemd[1]: Stopping Ceph prometheus.a for 320f6df8-6fa7-11ef-bcea-c7b262605968... 2024-09-10T19:35:01.931 INFO:journalctl@ceph.alertmanager.a.smithi183.stdout:Sep 10 19:35:01 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-alertmanager-a[80531]: ts=2024-09-10T19:35:01.626Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[0] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi183.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.183:8443: connect: connection refused" 2024-09-10T19:35:01.931 INFO:journalctl@ceph.alertmanager.a.smithi183.stdout:Sep 10 19:35:01 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-alertmanager-a[80531]: ts=2024-09-10T19:35:01.627Z caller=notify.go:732 level=warn component=dispatcher receiver=ceph-dashboard integration=webhook[1] msg="Notify attempt failed, will retry later" attempts=1 err="Post \"https://smithi191.front.sepia.ceph.com:8443/api/prometheus_receiver\": dial tcp 172.21.15.191:8443: connect: connection refused" 2024-09-10T19:35:02.216 INFO:journalctl@ceph.prometheus.a.smithi191.stdout:Sep 10 19:35:01 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-prometheus-a[67174]: ts=2024-09-10T19:35:01.913Z caller=main.go:840 level=warn msg="Received SIGTERM, exiting gracefully..." 2024-09-10T19:35:02.216 INFO:journalctl@ceph.prometheus.a.smithi191.stdout:Sep 10 19:35:01 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-prometheus-a[67174]: ts=2024-09-10T19:35:01.914Z caller=main.go:864 level=info msg="Stopping scrape discovery manager..." 2024-09-10T19:35:02.216 INFO:journalctl@ceph.prometheus.a.smithi191.stdout:Sep 10 19:35:01 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-prometheus-a[67174]: ts=2024-09-10T19:35:01.914Z caller=main.go:878 level=info msg="Stopping notify discovery manager..." 2024-09-10T19:35:02.216 INFO:journalctl@ceph.prometheus.a.smithi191.stdout:Sep 10 19:35:01 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-prometheus-a[67174]: ts=2024-09-10T19:35:01.914Z caller=manager.go:988 level=info component="rule manager" msg="Stopping rule manager..." 2024-09-10T19:35:02.216 INFO:journalctl@ceph.prometheus.a.smithi191.stdout:Sep 10 19:35:01 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-prometheus-a[67174]: ts=2024-09-10T19:35:01.914Z caller=main.go:860 level=info msg="Scrape discovery manager stopped" 2024-09-10T19:35:02.216 INFO:journalctl@ceph.prometheus.a.smithi191.stdout:Sep 10 19:35:01 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-prometheus-a[67174]: ts=2024-09-10T19:35:01.914Z caller=manager.go:998 level=info component="rule manager" msg="Rule manager stopped" 2024-09-10T19:35:02.217 INFO:journalctl@ceph.prometheus.a.smithi191.stdout:Sep 10 19:35:01 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-prometheus-a[67174]: ts=2024-09-10T19:35:01.914Z caller=main.go:874 level=info msg="Notify discovery manager stopped" 2024-09-10T19:35:02.217 INFO:journalctl@ceph.prometheus.a.smithi191.stdout:Sep 10 19:35:01 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-prometheus-a[67174]: ts=2024-09-10T19:35:01.914Z caller=main.go:915 level=info msg="Stopping scrape manager..." 2024-09-10T19:35:02.217 INFO:journalctl@ceph.prometheus.a.smithi191.stdout:Sep 10 19:35:01 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-prometheus-a[67174]: ts=2024-09-10T19:35:01.921Z caller=main.go:907 level=info msg="Scrape manager stopped" 2024-09-10T19:35:02.217 INFO:journalctl@ceph.prometheus.a.smithi191.stdout:Sep 10 19:35:01 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-prometheus-a[67174]: ts=2024-09-10T19:35:01.921Z caller=notifier.go:606 level=info component=notifier msg="Stopping notification manager..." 2024-09-10T19:35:02.217 INFO:journalctl@ceph.prometheus.a.smithi191.stdout:Sep 10 19:35:01 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-prometheus-a[67174]: ts=2024-09-10T19:35:01.921Z caller=main.go:1135 level=info msg="Notifier manager stopped" 2024-09-10T19:35:02.217 INFO:journalctl@ceph.prometheus.a.smithi191.stdout:Sep 10 19:35:01 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-prometheus-a[67174]: ts=2024-09-10T19:35:01.921Z caller=main.go:1147 level=info msg="See you next time!" 2024-09-10T19:35:02.217 INFO:journalctl@ceph.prometheus.a.smithi191.stdout:Sep 10 19:35:01 smithi191 podman[166118]: 2024-09-10 19:35:01.924303559 +0000 UTC m=+0.104620607 container died e9f7fcd20861732a5b312cee595b5fc835c9ca9d5e4eb4b6b509833063076edf (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-10T19:35:02.514 INFO:journalctl@ceph.prometheus.a.smithi191.stdout:Sep 10 19:35:02 smithi191 podman[166118]: 2024-09-10 19:35:02.320270404 +0000 UTC m=+0.500587497 container cleanup e9f7fcd20861732a5b312cee595b5fc835c9ca9d5e4eb4b6b509833063076edf (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-10T19:35:02.514 INFO:journalctl@ceph.prometheus.a.smithi191.stdout:Sep 10 19:35:02 smithi191 bash[166118]: ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-prometheus-a 2024-09-10T19:35:02.734 DEBUG:teuthology.orchestra.run.smithi191:> sudo pkill -f 'journalctl -f -n 0 -u ceph-320f6df8-6fa7-11ef-bcea-c7b262605968@prometheus.a.service' 2024-09-10T19:35:02.771 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-10T19:35:02.771 INFO:tasks.cephadm.prometheus.a:Stopped prometheus.a 2024-09-10T19:35:02.771 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 --force --keep-logs 2024-09-10T19:35:04.727 INFO:journalctl@ceph.alertmanager.a.smithi183.stdout:Sep 10 19:35:04 smithi183 systemd[1]: Stopping Ceph alertmanager.a for 320f6df8-6fa7-11ef-bcea-c7b262605968... 2024-09-10T19:35:05.025 INFO:journalctl@ceph.alertmanager.a.smithi183.stdout:Sep 10 19:35:04 smithi183 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-alertmanager-a[80531]: ts=2024-09-10T19:35:04.724Z caller=main.go:583 level=info msg="Received SIGTERM, exiting gracefully..." 2024-09-10T19:35:05.025 INFO:journalctl@ceph.alertmanager.a.smithi183.stdout:Sep 10 19:35:04 smithi183 podman[240754]: 2024-09-10 19:35:04.766228141 +0000 UTC m=+0.174019509 container died 1693fe6a126991cd9e9bae41ec0e86c2aed52a7bf9b9695f6083f915cbdc7440 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-09-10T19:35:05.400 INFO:journalctl@ceph.alertmanager.a.smithi183.stdout:Sep 10 19:35:05 smithi183 podman[240754]: 2024-09-10 19:35:05.20631925 +0000 UTC m=+0.614110610 container cleanup 1693fe6a126991cd9e9bae41ec0e86c2aed52a7bf9b9695f6083f915cbdc7440 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-09-10T19:35:05.400 INFO:journalctl@ceph.alertmanager.a.smithi183.stdout:Sep 10 19:35:05 smithi183 bash[240754]: ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-alertmanager-a 2024-09-10T19:35:05.678 INFO:journalctl@ceph.alertmanager.a.smithi183.stdout:Sep 10 19:35:05 smithi183 podman[240774]: 2024-09-10 19:35:05.399233847 +0000 UTC m=+0.625848203 container remove 1693fe6a126991cd9e9bae41ec0e86c2aed52a7bf9b9695f6083f915cbdc7440 (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-alertmanager-a, maintainer=The Prometheus Authors ) 2024-09-10T19:35:05.679 INFO:journalctl@ceph.alertmanager.a.smithi183.stdout:Sep 10 19:35:05 smithi183 podman[240774]: 2024-09-10 19:35:05.483609804 +0000 UTC m=+0.710224160 volume remove d4fb3fd11a5f4d7a217c91b4fe698c6bca5f6057348618c30cb85b7187ae5960 2024-09-10T19:35:06.129 INFO:journalctl@ceph.node-exporter.a.smithi183.stdout:Sep 10 19:35:05 smithi183 systemd[1]: Stopping Ceph node-exporter.a for 320f6df8-6fa7-11ef-bcea-c7b262605968... 2024-09-10T19:35:06.129 INFO:journalctl@ceph.alertmanager.a.smithi183.stdout:Sep 10 19:35:05 smithi183 systemd[1]: ceph-320f6df8-6fa7-11ef-bcea-c7b262605968@alertmanager.a.service: Deactivated successfully. 2024-09-10T19:35:06.129 INFO:journalctl@ceph.alertmanager.a.smithi183.stdout:Sep 10 19:35:05 smithi183 systemd[1]: Stopped Ceph alertmanager.a for 320f6df8-6fa7-11ef-bcea-c7b262605968. 2024-09-10T19:35:06.129 INFO:journalctl@ceph.alertmanager.a.smithi183.stdout:Sep 10 19:35:05 smithi183 systemd[1]: ceph-320f6df8-6fa7-11ef-bcea-c7b262605968@alertmanager.a.service: Consumed 1.866s CPU time. 2024-09-10T19:35:06.431 INFO:journalctl@ceph.node-exporter.a.smithi183.stdout:Sep 10 19:35:06 smithi183 podman[240886]: 2024-09-10 19:35:06.124157893 +0000 UTC m=+0.114888987 container died 518c3989dce0713f1a8b2dd14da9174ec8c56045558be06bf3147ed2c8c4f680 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-node-exporter-a, maintainer=The Prometheus Authors ) 2024-09-10T19:35:06.830 INFO:journalctl@ceph.node-exporter.a.smithi183.stdout:Sep 10 19:35:06 smithi183 podman[240886]: 2024-09-10 19:35:06.661222662 +0000 UTC m=+0.651953753 container cleanup 518c3989dce0713f1a8b2dd14da9174ec8c56045558be06bf3147ed2c8c4f680 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-node-exporter-a, maintainer=The Prometheus Authors ) 2024-09-10T19:35:06.830 INFO:journalctl@ceph.node-exporter.a.smithi183.stdout:Sep 10 19:35:06 smithi183 bash[240886]: ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-node-exporter-a 2024-09-10T19:35:07.093 INFO:journalctl@ceph.node-exporter.a.smithi183.stdout:Sep 10 19:35:06 smithi183 podman[240903]: 2024-09-10 19:35:06.82894514 +0000 UTC m=+0.708633994 container remove 518c3989dce0713f1a8b2dd14da9174ec8c56045558be06bf3147ed2c8c4f680 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-node-exporter-a, maintainer=The Prometheus Authors ) 2024-09-10T19:35:07.093 INFO:journalctl@ceph.node-exporter.a.smithi183.stdout:Sep 10 19:35:06 smithi183 systemd[1]: ceph-320f6df8-6fa7-11ef-bcea-c7b262605968@node-exporter.a.service: Main process exited, code=exited, status=143/n/a 2024-09-10T19:35:07.093 INFO:journalctl@ceph.node-exporter.a.smithi183.stdout:Sep 10 19:35:07 smithi183 systemd[1]: ceph-320f6df8-6fa7-11ef-bcea-c7b262605968@node-exporter.a.service: Failed with result 'exit-code'. 2024-09-10T19:35:07.093 INFO:journalctl@ceph.node-exporter.a.smithi183.stdout:Sep 10 19:35:07 smithi183 systemd[1]: Stopped Ceph node-exporter.a for 320f6df8-6fa7-11ef-bcea-c7b262605968. 2024-09-10T19:35:07.093 INFO:journalctl@ceph.node-exporter.a.smithi183.stdout:Sep 10 19:35:07 smithi183 systemd[1]: ceph-320f6df8-6fa7-11ef-bcea-c7b262605968@node-exporter.a.service: Consumed 4.699s CPU time. 2024-09-10T19:35:07.727 DEBUG:teuthology.orchestra.run.smithi191:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 --force --keep-logs 2024-09-10T19:35:09.466 INFO:journalctl@ceph.iscsi.iscsi.a.smithi191.stdout:Sep 10 19:35:09 smithi191 systemd[1]: Stopping Ceph iscsi.iscsi.a for 320f6df8-6fa7-11ef-bcea-c7b262605968... 2024-09-10T19:35:09.466 INFO:journalctl@ceph.iscsi.iscsi.a.smithi191.stdout:Sep 10 19:35:09 smithi191 ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-iscsi-iscsi-a[61816]: debug Shutdown received 2024-09-10T19:35:19.650 INFO:journalctl@ceph.iscsi.iscsi.a.smithi191.stdout:Sep 10 19:35:19 smithi191 bash[166500]: time="2024-09-10T19:35:19Z" level=warning msg="StopSignal SIGTERM failed to stop container ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-iscsi-iscsi-a in 10 seconds, resorting to SIGKILL" 2024-09-10T19:35:19.650 INFO:journalctl@ceph.iscsi.iscsi.a.smithi191.stdout:Sep 10 19:35:19 smithi191 podman[166500]: 2024-09-10 19:35:19.338110731 +0000 UTC m=+10.122608501 container died 63021e878f8c78349c9334cc6961d041522f70a598f9d5b52df2cc5408c49d05 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3bfeaf4417585c821cab9ca76e4d95f390b0e769807122a45463640820b5cd16, name=ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-iscsi-iscsi-a, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, org.label-schema.build-date=20240731, RELEASE=wip-guits-quincy-2024-09-09-1810-quincy-5be5739, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_CLEAN=True) 2024-09-10T19:35:19.913 INFO:journalctl@ceph.iscsi.iscsi.a.smithi191.stdout:Sep 10 19:35:19 smithi191 podman[166500]: 2024-09-10 19:35:19.798401901 +0000 UTC m=+10.582899668 container cleanup 63021e878f8c78349c9334cc6961d041522f70a598f9d5b52df2cc5408c49d05 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3bfeaf4417585c821cab9ca76e4d95f390b0e769807122a45463640820b5cd16, name=ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-iscsi-iscsi-a, GIT_CLEAN=True, org.label-schema.vendor=CentOS, ceph=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, RELEASE=wip-guits-quincy-2024-09-09-1810-quincy-5be5739, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731) 2024-09-10T19:35:19.913 INFO:journalctl@ceph.iscsi.iscsi.a.smithi191.stdout:Sep 10 19:35:19 smithi191 bash[166500]: ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-iscsi-iscsi-a 2024-09-10T19:35:20.215 INFO:journalctl@ceph.iscsi.iscsi.a.smithi191.stdout:Sep 10 19:35:20 smithi191 podman[166523]: 2024-09-10 19:35:20.091700814 +0000 UTC m=+0.753900817 container remove 63021e878f8c78349c9334cc6961d041522f70a598f9d5b52df2cc5408c49d05 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3bfeaf4417585c821cab9ca76e4d95f390b0e769807122a45463640820b5cd16, name=ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-iscsi-iscsi-a, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_CLEAN=True, maintainer=Guillaume Abrioux , RELEASE=wip-guits-quincy-2024-09-09-1810-quincy-5be5739, org.label-schema.vendor=CentOS, ceph=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731) 2024-09-10T19:35:20.216 INFO:journalctl@ceph.iscsi.iscsi.a.smithi191.stdout:Sep 10 19:35:20 smithi191 systemd[1]: ceph-320f6df8-6fa7-11ef-bcea-c7b262605968@iscsi.iscsi.a.service: Main process exited, code=exited, status=137/n/a 2024-09-10T19:35:20.654 INFO:journalctl@ceph.iscsi.iscsi.a.smithi191.stdout:Sep 10 19:35:20 smithi191 podman[166581]: 2024-09-10 19:35:20.332249793 +0000 UTC m=+0.110879119 container died a859d96610538fc85ac9d2cdf1ca966867d8747ed5198589630e4e8e98200703 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3bfeaf4417585c821cab9ca76e4d95f390b0e769807122a45463640820b5cd16, name=ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-iscsi-iscsi-a-tcmu, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, RELEASE=wip-guits-quincy-2024-09-09-1810-quincy-5be5739, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, ceph=True, maintainer=Guillaume Abrioux ) 2024-09-10T19:35:20.916 INFO:journalctl@ceph.iscsi.iscsi.a.smithi191.stdout:Sep 10 19:35:20 smithi191 podman[166581]: 2024-09-10 19:35:20.652504284 +0000 UTC m=+0.431133610 container cleanup a859d96610538fc85ac9d2cdf1ca966867d8747ed5198589630e4e8e98200703 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3bfeaf4417585c821cab9ca76e4d95f390b0e769807122a45463640820b5cd16, name=ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-iscsi-iscsi-a-tcmu, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, GIT_CLEAN=True, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, RELEASE=wip-guits-quincy-2024-09-09-1810-quincy-5be5739, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7) 2024-09-10T19:35:20.916 INFO:journalctl@ceph.iscsi.iscsi.a.smithi191.stdout:Sep 10 19:35:20 smithi191 bash[166581]: ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-iscsi-iscsi-a-tcmu 2024-09-10T19:35:20.917 INFO:journalctl@ceph.iscsi.iscsi.a.smithi191.stdout:Sep 10 19:35:20 smithi191 systemd[1]: ceph-320f6df8-6fa7-11ef-bcea-c7b262605968@iscsi.iscsi.a.service: Failed with result 'exit-code'. 2024-09-10T19:35:20.917 INFO:journalctl@ceph.iscsi.iscsi.a.smithi191.stdout:Sep 10 19:35:20 smithi191 systemd[1]: ceph-320f6df8-6fa7-11ef-bcea-c7b262605968@iscsi.iscsi.a.service: Unit process 61719 (conmon) remains running after unit stopped. 2024-09-10T19:35:20.917 INFO:journalctl@ceph.iscsi.iscsi.a.smithi191.stdout:Sep 10 19:35:20 smithi191 systemd[1]: ceph-320f6df8-6fa7-11ef-bcea-c7b262605968@iscsi.iscsi.a.service: Unit process 166599 (podman) remains running after unit stopped. 2024-09-10T19:35:20.917 INFO:journalctl@ceph.iscsi.iscsi.a.smithi191.stdout:Sep 10 19:35:20 smithi191 systemd[1]: Stopped Ceph iscsi.iscsi.a for 320f6df8-6fa7-11ef-bcea-c7b262605968. 2024-09-10T19:35:20.917 INFO:journalctl@ceph.iscsi.iscsi.a.smithi191.stdout:Sep 10 19:35:20 smithi191 systemd[1]: ceph-320f6df8-6fa7-11ef-bcea-c7b262605968@iscsi.iscsi.a.service: Consumed 3.192s CPU time. 2024-09-10T19:35:21.215 INFO:journalctl@ceph.iscsi.iscsi.a.smithi191.stdout:Sep 10 19:35:20 smithi191 podman[166599]: 2024-09-10 19:35:20.914671028 +0000 UTC m=+0.582719556 container remove a859d96610538fc85ac9d2cdf1ca966867d8747ed5198589630e4e8e98200703 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3bfeaf4417585c821cab9ca76e4d95f390b0e769807122a45463640820b5cd16, name=ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-iscsi-iscsi-a-tcmu, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, RELEASE=wip-guits-quincy-2024-09-09-1810-quincy-5be5739, org.label-schema.vendor=CentOS, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-10T19:35:21.216 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:35:21 smithi191 systemd[1]: Stopping Ceph grafana.a for 320f6df8-6fa7-11ef-bcea-c7b262605968... 2024-09-10T19:35:21.658 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:35:21 smithi191 podman[166725]: 2024-09-10 19:35:21.353205665 +0000 UTC m=+0.123430021 container died 217ef440a5ace4277b60a45237debb4aa0177045ca3690ae58f2684fe8bca300 (image=quay.io/ceph/ceph-grafana:9.4.7, name=ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a, architecture=x86_64, com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com, distribution-scope=public, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.openshift.expose-services=, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, description=Ceph Grafana Container, io.k8s.display-name=Red Hat Universal Base Image 8, release=239.1651231664, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664, maintainer=Paul Cuzner , summary=Grafana Container configured for Ceph mgr/dashboard integration, io.openshift.tags=base rhel8, vcs-type=git, com.redhat.component=ubi8-container, io.buildah.version=1.32.0, build-date=2022-04-29T11:48:11.055897, name=ubi8, version=8.5, vendor=Red Hat, Inc.) 2024-09-10T19:35:21.658 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:35:21 smithi191 podman[166725]: 2024-09-10 19:35:21.657276124 +0000 UTC m=+0.427500479 container cleanup 217ef440a5ace4277b60a45237debb4aa0177045ca3690ae58f2684fe8bca300 (image=quay.io/ceph/ceph-grafana:9.4.7, name=ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, description=Ceph Grafana Container, release=239.1651231664, io.buildah.version=1.32.0, io.openshift.expose-services=, maintainer=Paul Cuzner , architecture=x86_64, com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com, com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664, distribution-scope=public, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.k8s.display-name=Red Hat Universal Base Image 8, version=8.5, name=ubi8, vcs-type=git, vendor=Red Hat, Inc., com.redhat.component=ubi8-container, summary=Grafana Container configured for Ceph mgr/dashboard integration, build-date=2022-04-29T11:48:11.055897, io.openshift.tags=base rhel8) 2024-09-10T19:35:21.965 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:35:21 smithi191 bash[166725]: ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a 2024-09-10T19:35:21.966 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:35:21 smithi191 podman[166745]: 2024-09-10 19:35:21.883645543 +0000 UTC m=+0.529273699 container remove 217ef440a5ace4277b60a45237debb4aa0177045ca3690ae58f2684fe8bca300 (image=quay.io/ceph/ceph-grafana:9.4.7, name=ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-grafana-a, io.k8s.display-name=Red Hat Universal Base Image 8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., maintainer=Paul Cuzner , com.redhat.component=ubi8-container, vendor=Red Hat, Inc., com.redhat.license_terms=https://www.redhat.com/en/about/red-hat-end-user-license-agreements#UBI, name=ubi8, com.redhat.build-host=cpt-1007.osbs.prod.upshift.rdu2.redhat.com, description=Ceph Grafana Container, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.5-239.1651231664, build-date=2022-04-29T11:48:11.055897, io.openshift.expose-services=, vcs-ref=3aadd00326f3dd6cfe65ee31017ab98915fddb56, version=8.5, distribution-scope=public, release=239.1651231664, io.buildah.version=1.32.0, summary=Grafana Container configured for Ceph mgr/dashboard integration, architecture=x86_64, vcs-type=git, io.openshift.tags=base rhel8) 2024-09-10T19:35:21.966 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:35:21 smithi191 systemd[1]: ceph-320f6df8-6fa7-11ef-bcea-c7b262605968@grafana.a.service: Main process exited, code=exited, status=143/n/a 2024-09-10T19:35:22.466 INFO:journalctl@ceph.node-exporter.b.smithi191.stdout:Sep 10 19:35:22 smithi191 systemd[1]: Stopping Ceph node-exporter.b for 320f6df8-6fa7-11ef-bcea-c7b262605968... 2024-09-10T19:35:22.466 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:35:22 smithi191 systemd[1]: ceph-320f6df8-6fa7-11ef-bcea-c7b262605968@grafana.a.service: Failed with result 'exit-code'. 2024-09-10T19:35:22.466 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:35:22 smithi191 systemd[1]: Stopped Ceph grafana.a for 320f6df8-6fa7-11ef-bcea-c7b262605968. 2024-09-10T19:35:22.466 INFO:journalctl@ceph.grafana.a.smithi191.stdout:Sep 10 19:35:22 smithi191 systemd[1]: ceph-320f6df8-6fa7-11ef-bcea-c7b262605968@grafana.a.service: Consumed 13.491s CPU time. 2024-09-10T19:35:22.780 INFO:journalctl@ceph.node-exporter.b.smithi191.stdout:Sep 10 19:35:22 smithi191 podman[166857]: 2024-09-10 19:35:22.466003733 +0000 UTC m=+0.145731125 container died 44646e3eeb19f15281735e23d6f999ef803b33a4eab2191dc67439562f12264a (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-node-exporter-b, maintainer=The Prometheus Authors ) 2024-09-10T19:35:23.056 INFO:journalctl@ceph.node-exporter.b.smithi191.stdout:Sep 10 19:35:22 smithi191 podman[166857]: 2024-09-10 19:35:22.878623842 +0000 UTC m=+0.558351266 container cleanup 44646e3eeb19f15281735e23d6f999ef803b33a4eab2191dc67439562f12264a (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-node-exporter-b, maintainer=The Prometheus Authors ) 2024-09-10T19:35:23.056 INFO:journalctl@ceph.node-exporter.b.smithi191.stdout:Sep 10 19:35:22 smithi191 bash[166857]: ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-node-exporter-b 2024-09-10T19:35:23.309 INFO:journalctl@ceph.node-exporter.b.smithi191.stdout:Sep 10 19:35:23 smithi191 podman[166875]: 2024-09-10 19:35:23.054879449 +0000 UTC m=+0.592817320 container remove 44646e3eeb19f15281735e23d6f999ef803b33a4eab2191dc67439562f12264a (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-320f6df8-6fa7-11ef-bcea-c7b262605968-node-exporter-b, maintainer=The Prometheus Authors ) 2024-09-10T19:35:23.309 INFO:journalctl@ceph.node-exporter.b.smithi191.stdout:Sep 10 19:35:23 smithi191 systemd[1]: ceph-320f6df8-6fa7-11ef-bcea-c7b262605968@node-exporter.b.service: Main process exited, code=exited, status=143/n/a 2024-09-10T19:35:23.309 INFO:journalctl@ceph.node-exporter.b.smithi191.stdout:Sep 10 19:35:23 smithi191 systemd[1]: ceph-320f6df8-6fa7-11ef-bcea-c7b262605968@node-exporter.b.service: Failed with result 'exit-code'. 2024-09-10T19:35:23.309 INFO:journalctl@ceph.node-exporter.b.smithi191.stdout:Sep 10 19:35:23 smithi191 systemd[1]: Stopped Ceph node-exporter.b for 320f6df8-6fa7-11ef-bcea-c7b262605968. 2024-09-10T19:35:23.309 INFO:journalctl@ceph.node-exporter.b.smithi191.stdout:Sep 10 19:35:23 smithi191 systemd[1]: ceph-320f6df8-6fa7-11ef-bcea-c7b262605968@node-exporter.b.service: Consumed 4.582s CPU time. 2024-09-10T19:35:23.975 DEBUG:teuthology.orchestra.run.smithi183:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-09-10T19:35:24.008 DEBUG:teuthology.orchestra.run.smithi191:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-09-10T19:35:24.035 INFO:tasks.cephadm:Archiving crash dumps... 2024-09-10T19:35:24.037 DEBUG:teuthology.misc:Transferring archived files from smithi183:/var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/crash to /home/teuthworker/archive/gabrioux-2024-09-10_17:53:37-orch:cephadm-wip-guits-quincy-2024-09-09-1810-quincy-distro-default-smithi/7900285/remote/smithi183/crash 2024-09-10T19:35:24.037 DEBUG:teuthology.orchestra.run.smithi183:> sudo tar c -f - -C /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/crash -- . 2024-09-10T19:35:24.086 INFO:teuthology.orchestra.run.smithi183.stderr:tar: /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/crash: Cannot open: No such file or directory 2024-09-10T19:35:24.086 INFO:teuthology.orchestra.run.smithi183.stderr:tar: Error is not recoverable: exiting now 2024-09-10T19:35:24.088 DEBUG:teuthology.misc:Transferring archived files from smithi191:/var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/crash to /home/teuthworker/archive/gabrioux-2024-09-10_17:53:37-orch:cephadm-wip-guits-quincy-2024-09-09-1810-quincy-distro-default-smithi/7900285/remote/smithi191/crash 2024-09-10T19:35:24.089 DEBUG:teuthology.orchestra.run.smithi191:> sudo tar c -f - -C /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/crash -- . 2024-09-10T19:35:24.123 INFO:teuthology.orchestra.run.smithi191.stderr:tar: /var/lib/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/crash: Cannot open: No such file or directory 2024-09-10T19:35:24.123 INFO:teuthology.orchestra.run.smithi191.stderr:tar: Error is not recoverable: exiting now 2024-09-10T19:35:24.124 INFO:tasks.cephadm:Checking cluster log for badness... 2024-09-10T19:35:24.124 DEBUG:teuthology.orchestra.run.smithi183:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v 'but it is still running' | egrep -v 'objects unfound and apparently lost' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | head -n 1 2024-09-10T19:35:24.157 INFO:teuthology.orchestra.run.smithi183.stderr:grep: /var/log/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/ceph.log: No such file or directory 2024-09-10T19:35:24.159 INFO:tasks.cephadm:Compressing logs... 2024-09-10T19:35:24.159 DEBUG:teuthology.orchestra.run.smithi183:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-09-10T19:35:24.203 DEBUG:teuthology.orchestra.run.smithi191:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-09-10T19:35:24.254 INFO:teuthology.orchestra.run.smithi183.stderr:find: ‘/var/log/rbd-target-api’: No such file or directory 2024-09-10T19:35:24.255 INFO:teuthology.orchestra.run.smithi183.stderr:gzip -5 --verbose -- /var/log/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/ceph-osd.0.log 2024-09-10T19:35:24.255 INFO:teuthology.orchestra.run.smithi183.stderr:gzip -5 --verbose -- /var/log/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/ceph-osd.1.log 2024-09-10T19:35:24.255 INFO:teuthology.orchestra.run.smithi183.stderr:gzip -5 --verbose -- /var/log/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/ceph-volume.log 2024-09-10T19:35:24.256 INFO:teuthology.orchestra.run.smithi183.stderr:/var/log/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/ceph-mon.c.log 2024-09-10T19:35:24.256 INFO:teuthology.orchestra.run.smithi183.stderr:/var/log/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/ceph-osd.1.log: /var/log/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/ceph-volume.log: gzip -5 --verbose -- /var/log/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/ceph-mon.a.log 2024-09-10T19:35:24.256 INFO:teuthology.orchestra.run.smithi183.stderr:/var/log/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/ceph-mon.c.log: gzip -5 --verbose -- /var/log/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/objectstore_tool.6.log 2024-09-10T19:35:24.258 INFO:teuthology.orchestra.run.smithi183.stderr:/var/log/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/ceph-osd.2.log 2024-09-10T19:35:24.259 INFO:teuthology.orchestra.run.smithi183.stderr:/var/log/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/objectstore_tool.6.log: gzip -5 --verbose -- /var/log/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/ceph-mgr.y.log 2024-09-10T19:35:24.259 INFO:teuthology.orchestra.run.smithi183.stderr:/var/log/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/ceph-client.rgw.foo.a.log 2024-09-10T19:35:24.259 INFO:teuthology.orchestra.run.smithi183.stderr:/var/log/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/ceph-mgr.y.log: gzip -5 --verbose -- /var/log/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/ceph-osd.3.log 2024-09-10T19:35:24.259 INFO:teuthology.orchestra.run.smithi183.stderr:/var/log/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/ceph-client.rgw.foo.a.log: gzip -5 --verbose -- /var/log/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/objectstore_tool.7.log 2024-09-10T19:35:24.259 INFO:teuthology.orchestra.run.smithi183.stderr: 58.6% -- replaced with /var/log/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/ceph-client.rgw.foo.a.log.gz 2024-09-10T19:35:24.259 INFO:teuthology.orchestra.run.smithi183.stderr:/var/log/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/ceph-osd.3.log: gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-09-10T19:35:24.266 INFO:teuthology.orchestra.run.smithi191.stderr:find: ‘/var/log/rbd-target-api’: No such file or directory 2024-09-10T19:35:24.266 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/tcmu-runner.log 2024-09-10T19:35:24.266 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/ceph-osd.7.log 2024-09-10T19:35:24.266 INFO:teuthology.orchestra.run.smithi191.stderr:gzip -5 --verbose -- /var/log/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/ceph-volume.log 2024-09-10T19:35:24.267 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/tcmu-runner.log: /var/log/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/ceph-osd.7.log: gzip -5 --verbose -- /var/log/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/ceph-osd.4.log 2024-09-10T19:35:24.267 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/ceph-volume.log: gzip -5 --verbose -- /var/log/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/ceph-osd.6.log 2024-09-10T19:35:24.268 INFO:teuthology.orchestra.run.smithi191.stderr: 63.5% -- replaced with /var/log/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/tcmu-runner.log.gz 2024-09-10T19:35:24.268 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/ceph-osd.5.log 2024-09-10T19:35:24.269 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/ceph-osd.6.log: gzip -5 --verbose -- /var/log/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/ceph-mgr.x.log 2024-09-10T19:35:24.269 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/ceph-mon.b.log 2024-09-10T19:35:24.269 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-09-10T19:35:24.285 INFO:teuthology.orchestra.run.smithi183.stderr:/var/log/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/objectstore_tool.7.log: /var/log/ceph/cephadm.log: 94.0% -- replaced with /var/log/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/objectstore_tool.6.log.gz 2024-09-10T19:35:24.372 INFO:teuthology.orchestra.run.smithi183.stderr: 95.4% -- replaced with /var/log/ceph/cephadm.log.gz 2024-09-10T19:35:24.383 INFO:teuthology.orchestra.run.smithi191.stderr:/var/log/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/ceph-mon.b.log: /var/log/ceph/cephadm.log: 92.3% -- replaced with /var/log/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/ceph-mgr.x.log.gz 2024-09-10T19:35:24.387 INFO:teuthology.orchestra.run.smithi191.stderr: 94.8% -- replaced with /var/log/ceph/cephadm.log.gz 2024-09-10T19:35:24.503 INFO:teuthology.orchestra.run.smithi191.stderr: 91.2% -- replaced with /var/log/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/ceph-volume.log.gz 2024-09-10T19:35:24.641 INFO:teuthology.orchestra.run.smithi183.stderr: 91.3% -- replaced with /var/log/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/ceph-volume.log.gz 2024-09-10T19:35:25.331 INFO:teuthology.orchestra.run.smithi183.stderr: 89.8% -- replaced with /var/log/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/ceph-mgr.y.log.gz 2024-09-10T19:35:25.456 INFO:teuthology.orchestra.run.smithi183.stderr: 94.6% -- replaced with /var/log/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/objectstore_tool.7.log.gz 2024-09-10T19:35:26.466 INFO:teuthology.orchestra.run.smithi191.stderr: 91.2% -- replaced with /var/log/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/ceph-mon.b.log.gz 2024-09-10T19:35:29.361 INFO:teuthology.orchestra.run.smithi183.stderr: 90.5% -- replaced with /var/log/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/ceph-mon.a.log.gz 2024-09-10T19:35:30.845 INFO:teuthology.orchestra.run.smithi183.stderr: 91.2% -- replaced with /var/log/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/ceph-mon.c.log.gz 2024-09-10T19:36:02.021 INFO:teuthology.orchestra.run.smithi183.stderr: 93.4% -- replaced with /var/log/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/ceph-osd.2.log.gz 2024-09-10T19:37:46.101 INFO:teuthology.orchestra.run.smithi183.stderr: 93.5% -- replaced with /var/log/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/ceph-osd.1.log.gz 2024-09-10T19:38:04.384 INFO:teuthology.orchestra.run.smithi191.stderr: 93.4% -- replaced with /var/log/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/ceph-osd.7.log.gz 2024-09-10T19:38:20.046 INFO:teuthology.orchestra.run.smithi183.stderr: 92.9% -- replaced with /var/log/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/ceph-osd.3.log.gz 2024-09-10T19:38:53.487 INFO:teuthology.orchestra.run.smithi183.stderr: 93.2% -- replaced with /var/log/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/ceph-osd.0.log.gz 2024-09-10T19:38:53.490 INFO:teuthology.orchestra.run.smithi183.stderr: 2024-09-10T19:38:53.490 INFO:teuthology.orchestra.run.smithi183.stderr:real 3m29.273s 2024-09-10T19:38:53.490 INFO:teuthology.orchestra.run.smithi183.stderr:user 4m23.303s 2024-09-10T19:38:53.490 INFO:teuthology.orchestra.run.smithi183.stderr:sys 0m10.916s 2024-09-10T19:40:00.481 INFO:teuthology.orchestra.run.smithi191.stderr: 93.4% -- replaced with /var/log/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/ceph-osd.4.log.gz 2024-09-10T19:43:04.114 INFO:teuthology.orchestra.run.smithi191.stderr: 93.5% -- replaced with /var/log/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/ceph-osd.6.log.gz 2024-09-10T19:43:18.309 INFO:teuthology.orchestra.run.smithi191.stderr: 93.0% -- replaced with /var/log/ceph/320f6df8-6fa7-11ef-bcea-c7b262605968/ceph-osd.5.log.gz 2024-09-10T19:43:18.312 INFO:teuthology.orchestra.run.smithi191.stderr: 2024-09-10T19:43:18.312 INFO:teuthology.orchestra.run.smithi191.stderr:real 7m54.094s 2024-09-10T19:43:18.312 INFO:teuthology.orchestra.run.smithi191.stderr:user 5m37.371s 2024-09-10T19:43:18.312 INFO:teuthology.orchestra.run.smithi191.stderr:sys 0m15.607s 2024-09-10T19:43:18.312 INFO:tasks.cephadm:Archiving logs... 2024-09-10T19:43:18.313 DEBUG:teuthology.misc:Transferring archived files from smithi183:/var/log/ceph to /home/teuthworker/archive/gabrioux-2024-09-10_17:53:37-orch:cephadm-wip-guits-quincy-2024-09-09-1810-quincy-distro-default-smithi/7900285/remote/smithi183/log 2024-09-10T19:43:18.314 DEBUG:teuthology.orchestra.run.smithi183:> sudo tar c -f - -C /var/log/ceph -- . 2024-09-10T19:43:57.350 DEBUG:teuthology.misc:Transferring archived files from smithi191:/var/log/ceph to /home/teuthworker/archive/gabrioux-2024-09-10_17:53:37-orch:cephadm-wip-guits-quincy-2024-09-09-1810-quincy-distro-default-smithi/7900285/remote/smithi191/log 2024-09-10T19:43:57.351 DEBUG:teuthology.orchestra.run.smithi191:> sudo tar c -f - -C /var/log/ceph -- . 2024-09-10T19:44:46.846 INFO:tasks.cephadm:Removing cluster... 2024-09-10T19:44:46.847 DEBUG:teuthology.orchestra.run.smithi183:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 --force 2024-09-10T19:44:47.675 DEBUG:teuthology.orchestra.run.smithi191:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 320f6df8-6fa7-11ef-bcea-c7b262605968 --force 2024-09-10T19:44:48.586 INFO:tasks.cephadm:Removing cephadm ... 2024-09-10T19:44:48.587 DEBUG:teuthology.orchestra.run.smithi183:> rm -rf /home/ubuntu/cephtest/cephadm 2024-09-10T19:44:48.604 DEBUG:teuthology.orchestra.run.smithi191:> rm -rf /home/ubuntu/cephtest/cephadm 2024-09-10T19:44:48.619 INFO:tasks.cephadm:Teardown complete 2024-09-10T19:44:48.620 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-09-10T19:44:48.632 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-09-10T19:44:48.632 DEBUG:teuthology.orchestra.run.smithi183:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-09-10T19:44:48.647 DEBUG:teuthology.orchestra.run.smithi191:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-09-10T19:44:48.760 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-09-10T19:44:48.768 INFO:teuthology.task.clock:Checking final clock skew... 2024-09-10T19:44:48.768 DEBUG:teuthology.orchestra.run.smithi183:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-10T19:44:48.770 DEBUG:teuthology.orchestra.run.smithi191:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-10T19:44:48.783 INFO:teuthology.orchestra.run.smithi183.stderr:bash: line 1: ntpq: command not found 2024-09-10T19:44:48.785 INFO:teuthology.orchestra.run.smithi191.stderr:bash: line 1: ntpq: command not found 2024-09-10T19:44:48.801 INFO:teuthology.orchestra.run.smithi191.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-10T19:44:48.801 INFO:teuthology.orchestra.run.smithi191.stdout:=============================================================================== 2024-09-10T19:44:48.801 INFO:teuthology.orchestra.run.smithi191.stdout:^* hv01.front.sepia.ceph.com 3 6 377 30 -592us[ -598us] +/- 58ms 2024-09-10T19:44:48.801 INFO:teuthology.orchestra.run.smithi191.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 33 +1879us[+1873us] +/- 61ms 2024-09-10T19:44:48.801 INFO:teuthology.orchestra.run.smithi191.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 33 -1608us[-1614us] +/- 74ms 2024-09-10T19:44:48.801 INFO:teuthology.orchestra.run.smithi191.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-09-10T19:44:48.804 INFO:teuthology.orchestra.run.smithi183.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-10T19:44:48.804 INFO:teuthology.orchestra.run.smithi183.stdout:=============================================================================== 2024-09-10T19:44:48.804 INFO:teuthology.orchestra.run.smithi183.stdout:^* hv01.front.sepia.ceph.com 3 6 377 32 -560us[ -520us] +/- 58ms 2024-09-10T19:44:48.804 INFO:teuthology.orchestra.run.smithi183.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 30 +1924us[+1924us] +/- 61ms 2024-09-10T19:44:48.804 INFO:teuthology.orchestra.run.smithi183.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 33 -1571us[-1532us] +/- 74ms 2024-09-10T19:44:48.804 INFO:teuthology.orchestra.run.smithi183.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-09-10T19:44:48.805 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-09-10T19:44:48.813 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-09-10T19:44:48.813 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-09-10T19:44:48.884 DEBUG:teuthology.orchestra.run.smithi183:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-09-10T19:44:49.006 DEBUG:teuthology.orchestra.run.smithi191:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-09-10T19:44:49.080 DEBUG:teuthology.orchestra.run.smithi183:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 2024-09-10T19:44:49.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:44:49.109 DEBUG:teuthology.orchestra.run.smithi191:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 2024-09-10T19:44:49.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:44:49.136 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-09-10T19:44:49.145 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-09-10T19:44:49.154 INFO:teuthology.task.internal:Duration was 2910.321240 seconds 2024-09-10T19:44:49.154 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-09-10T19:44:49.163 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-09-10T19:44:49.163 DEBUG:teuthology.orchestra.run.smithi183:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-09-10T19:44:49.165 DEBUG:teuthology.orchestra.run.smithi191:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-09-10T19:44:49.209 INFO:teuthology.orchestra.run.smithi183.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-10T19:44:49.230 INFO:teuthology.orchestra.run.smithi191.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-10T19:44:49.725 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-09-10T19:44:49.725 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi183.front.sepia.ceph.com 2024-09-10T19:44:49.725 DEBUG:teuthology.orchestra.run.smithi183:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-09-10T19:44:49.746 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi191.front.sepia.ceph.com 2024-09-10T19:44:49.746 DEBUG:teuthology.orchestra.run.smithi191:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-09-10T19:44:49.775 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-09-10T19:44:49.775 DEBUG:teuthology.orchestra.run.smithi183:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-09-10T19:44:49.789 DEBUG:teuthology.orchestra.run.smithi191:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-09-10T19:44:49.842 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-09-10T19:44:49.842 DEBUG:teuthology.orchestra.run.smithi183:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-09-10T19:44:49.856 DEBUG:teuthology.orchestra.run.smithi191:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-09-10T19:44:51.377 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-09-10T19:44:51.386 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-09-10T19:44:51.387 DEBUG:teuthology.orchestra.run.smithi183:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-09-10T19:44:51.415 DEBUG:teuthology.orchestra.run.smithi191:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-09-10T19:44:51.496 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-09-10T19:44:51.506 DEBUG:teuthology.orchestra.run.smithi183:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-09-10T19:44:51.509 DEBUG:teuthology.orchestra.run.smithi191:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-09-10T19:44:51.530 INFO:teuthology.orchestra.run.smithi183.stdout:kernel.core_pattern = core 2024-09-10T19:44:51.539 INFO:teuthology.orchestra.run.smithi191.stdout:kernel.core_pattern = core 2024-09-10T19:44:51.576 DEBUG:teuthology.orchestra.run.smithi183:> test -e /home/ubuntu/cephtest/archive/coredump 2024-09-10T19:44:51.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:44:51.613 DEBUG:teuthology.orchestra.run.smithi191:> test -e /home/ubuntu/cephtest/archive/coredump 2024-09-10T19:44:51.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T19:44:51.632 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-09-10T19:44:51.642 INFO:teuthology.task.internal:Transferring archived files... 2024-09-10T19:44:51.643 DEBUG:teuthology.misc:Transferring archived files from smithi183:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/gabrioux-2024-09-10_17:53:37-orch:cephadm-wip-guits-quincy-2024-09-09-1810-quincy-distro-default-smithi/7900285/remote/smithi183 2024-09-10T19:44:51.644 DEBUG:teuthology.orchestra.run.smithi183:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-09-10T19:44:51.726 DEBUG:teuthology.misc:Transferring archived files from smithi191:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/gabrioux-2024-09-10_17:53:37-orch:cephadm-wip-guits-quincy-2024-09-09-1810-quincy-distro-default-smithi/7900285/remote/smithi191 2024-09-10T19:44:51.727 DEBUG:teuthology.orchestra.run.smithi191:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-09-10T19:44:51.769 INFO:teuthology.task.internal:Removing archive directory... 2024-09-10T19:44:51.770 DEBUG:teuthology.orchestra.run.smithi183:> rm -rf -- /home/ubuntu/cephtest/archive 2024-09-10T19:44:51.772 DEBUG:teuthology.orchestra.run.smithi191:> rm -rf -- /home/ubuntu/cephtest/archive 2024-09-10T19:44:51.818 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-09-10T19:44:51.830 INFO:teuthology.task.internal:Not uploading archives. 2024-09-10T19:44:51.830 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-09-10T19:44:51.839 INFO:teuthology.task.internal:Tidying up after the test... 2024-09-10T19:44:51.839 DEBUG:teuthology.orchestra.run.smithi183:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-09-10T19:44:51.841 DEBUG:teuthology.orchestra.run.smithi191:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-09-10T19:44:51.854 INFO:teuthology.orchestra.run.smithi183.stdout: 265324 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 10 19:44 /home/ubuntu/cephtest 2024-09-10T19:44:51.875 INFO:teuthology.orchestra.run.smithi191.stdout: 266063 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 10 19:44 /home/ubuntu/cephtest 2024-09-10T19:44:51.876 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-09-10T19:44:51.885 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-09-10T19:44:51.906 INFO:teuthology.run:Summary data: description: orch:cephadm/thrash/{0-distro/centos_9.stream_runc 1-start 2-thrash 3-tasks/radosbench fixed-2 msgr/async root} duration: 2910.32124042511 flavor: default owner: scheduled_gabrioux@teuthology success: true 2024-09-10T19:44:51.906 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-09-10T19:44:52.216 INFO:teuthology.run:pass